Help! Cant start BackupExec 12.5 services, after performing an Intelligent Disaster Recovery!

[andre_st]

We just performed a complete Intelligent Disaster Recovery. Everything seemed to go well, the operating system and files are all there. But there seems to be a problem with the SQL-databases (we also have the Backup Exec SQL agent). Therefore I want to do a restore of the SQL user databases once more. But to do that, I need to access BackupExec 12.5 - and there is the problem. The services are not starting up!

Here you have two following error messages retrieved from the event-log

Log Name:      Application
Source:        MSSQLSERVER
Date:          2010-02-19 13:01:52
Event ID:      5123
Task Category: Server
Level:         Error
Keywords:      Classic
User:          MIKROV\AdminSE
Computer:      SVERIGESRV1.valhalla.local
Description:
CREATE FILE encountered operating system error 5(failed to retrieve text for this error. Reason: 15105) while attempting to open or create the physical file 'C:\Program Files\Symantec\Backup Exec\Data\BEDB_dat.mdf'.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="MSSQLSERVER" />
    <EventID Qualifiers="49152">5123</EventID>
    <Level>2</Level>
    <Task>2</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2010-02-19T12:01:52.000Z" />
    <EventRecordID>20071</EventRecordID>
    <Channel>Application</Channel>
    <Computer>SVERIGESRV1.valhalla.local</Computer>
    <Security UserID="S-1-5-21-2131191430-715879785-316617838-3683" />
  </System>
  <EventData>
    <Data>5(failed to retrieve text for this error. Reason: 15105)</Data>
    <Data>C:\Program Files\Symantec\Backup Exec\Data\BEDB_dat.mdf</Data>
    <Binary>03140000100000000C00000053005600450052004900470045005300520056003100000000000000</Binary>
  </EventData>
</Event>

Error 2:

Log Name:      Application
Source:        Backup Exec
Date:          2010-02-19 13:01:52
Event ID:      58068
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SVERIGESRV1.valhalla.local
Description:
The Backup Exec Device and Media Service could not start because the database recovery has failed.  Refer to the database recovery log for details.

 For more information, click the following link:
http://eventlookup.veritas.com/eventlookup/EventLookup.jhtml
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Backup Exec" />
    <EventID Qualifiers="8192">58068</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2010-02-19T12:01:52.000Z" />
    <EventRecordID>20072</EventRecordID>
    <Channel>Application</Channel>
    <Computer>SVERIGESRV1.valhalla.local</Computer>
    <Security />
  </System>
  <EventData>
  </EventData>
</Event>

I tried to repair and recover the database with the BackupExec utility - but it didnt help. Can you please help me as soon as possible!

Regards,

André

[andre_st]

I have found some more information about this issue, in another posts. But I still dont know exactly how to proceed. I found this in the recovery log:

GetDatabaseStatus
OpenFromInitializationString Connection String = Provider=SQLOLEDB.1;Integrated Security=SSPI;Persist Security Info=False;Initial Catalog=master;Data Source=SVERIGESRV1;Locale Identifier=1033;Application Name=BEWS DBUTIL hr=0x0
GetDatabaseStatus status=NOT EXIST
Database BEDB does not exist
File Exist Check: \\SVERIGESRV1\C$\Program Files\Symantec\Backup Exec\Data\BEDB_dat.mdf
File Exist: FALSE
File Exist Check: \\SVERIGESRV1\C$\Program Files\Symantec\Backup Exec\Data\BEDB.bak
File Exist: TRUE
Recover from backup file...
File Exist Check: \\SVERIGESRV1\C$\Program Files\Symantec\Backup Exec\Data\BEDB.bak
File Exist: TRUE
GetDatabaseStatus
OpenFromInitializationString Connection String = Provider=SQLOLEDB.1;Integrated Security=SSPI;Persist Security Info=False;Initial Catalog=master;Data Source=SVERIGESRV1;Locale Identifier=1033;Application Name=BEWS DBUTIL hr=0x0
GetDatabaseStatus status=NOT EXIST
Database BEDB does not exist
Fixing up any database linkage problems for 'BEDB'
Execute command: declare @@stemp as varchar(32)
select @@stemp = isnull(convert(varchar,databasepropertyex('BEDB','status')),'NOT EXIST')
print 'Database status: ' + @@stemp
if @@stemp <> 'ONLINE' and @@stemp <> 'NOT EXIST'
begin
print 'Forcing database detach'
EXEC sp_detach_db 'BEDB'
end

OpenFromInitializationString Connection String = Provider=SQLOLEDB.1;Integrated Security=SSPI;Persist Security Info=False;Initial Catalog=master;Data Source=SVERIGESRV1;Locale Identifier=1033;Application Name=BEWS DBUTIL hr=0x0
(null)
File Exist Check: \\SVERIGESRV1\C$\Program Files\Symantec\Backup Exec\Data\BEDB_dat.mdf
File Exist: FALSE
File Exist Check: \\SVERIGESRV1\C$\Program Files\Symantec\Backup Exec\Data\BEDB_log.ldf
File Exist: FALSE
Create database BEDB
Execute command: create database BEDB on(name=BEDB_dat, filename='C:\Program Files\Symantec\Backup Exec\Data\BEDB_dat.mdf') log on (name=BEDB_log, filename='C:\Program Files\Symantec\Backup Exec\Data\BEDB_log.ldf')
OpenFromInitializationString Connection String = Provider=SQLOLEDB.1;Integrated Security=SSPI;Persist Security Info=False;Initial Catalog=master;Data Source=SVERIGESRV1;Locale Identifier=1033;Application Name=BEWS DBUTIL hr=0x0
(null)
(null)
Execute command failed: 0x80040e14
OS ERROR: 0x80040e14 (-2147217900)
Deinitialize...

-----------------------------
Process completed
02/19/10 13:22:31
Status: DBU_ERROR_DATABASE_CREATE_FAILED

-------
So I now know that the files are missing, but what now?

[Dev_T]

Hello,

Start the SQL Service(BKUPEXEC) on Local system Account.

1  : Stop all BE services.
2  : Start the SQL Service(BKUPEXEC) on Local system Account.
3  : Start the SQL Service(BKUPEXEC) service
4  : start all the BE Services.

Hope this helps...

[andre_st]

Hello Dev T

1: All BE services were allready stopped
2/3.: The SQL services were allready running on local system.
4: The BE services couldnt be started.

I think that these steps wouldnt solve this, as the log-files state that the BackupExec database is missing, and couldnt be restored from the bak-file.. The steps you suggested doesnt involve recreating the BE-database file.

Since I needed to try to restore the database files from the tape, I first needed to be able to launch BackupExec. However, I couldnt start BackupExec because the BackupExec database was missing or corrupt. Therefore I uninstalled BackupExec, and reinstalled it - and thereby creating a new BackupExec database. I was then able to access the tape. Since I didnt know if the SQL master database was corrupt, I thought I might restore all the databases from the tape with the BackupExec SQL Agent. But when I tried that I got the error that It isnt possible to restore the master database when it is in use, and that I had to launch SQL Server in "single mode". Fair enough, I managed to find out how to achieve that - but when the SQL Server was started in single mode, the BackupExec couldnt access the BackupExec database from the SQL Server, meaning that a restore couldnt be performed anyway...

My last resort was to only restore the user database, and hoping that there was no problem with the master database (we only have one user database on the SQL Server) - and luckily that worked! So everything "seems" to be working now, and therefore my guess is that the master database was ok right after the "intelligent disaster recovery". This of course means that we now have a new BackupExec database, so we dont have any information about our previously made backups, but this is probably not a problem for us.

But what I would like to know is what I "should" have done! Surely I shouldnt need to reinstall BackupExec...just to be able to access the tape?
I found some similar posts on the internet, where it often was explained that this issue could occur if BackupExec was looking for the bak-file in the wrong place, and therefore couldnt restore it. But from what I can read from the logfile it "found" the bak-file, but still wasnt able to recover it. As it said in the log:

File Exist Check: \\SVERIGESRV1\C$\Program Files\Symantec\Backup Exec\Data\BEDB.bak
File Exist: TRUE
Recover from backup file...
File Exist Check: \\SVERIGESRV1\C$\Program Files\Symantec\Backup Exec\Data\BEDB.bak
File Exist: TRUE

I wont be able to do some more extensive teseting on this, like doing the whole process over again - as the server is critical for the business. But I need to know how to handle this situation if it arises once more in the future...So please, I need some more insight on this matter!

I also need to know if it is best practice to not have the BackupExec database on the "main" SQL Server (which it is in our case), but rather located on a paralell SQL Server Express? I can imagine for example that if the master database on the SQL Server needs to be restored, then this would be hard to achieve since BackupExec wont run unless it gets access to the SQL Server in the first place...