BE Services stop unexpectedly

[VRT_KMONTGOM]

I need help troubleshooting BE 2010 R2 services that stop unexpectedly.

Symantec support has been unable to provide me troubleshooting steps beyond gathering logs and analyzing, without reporting results.

We are running BE2010 R2 on a Windows 2003 Standard Edition server, HP hardware, with Ultrium 4 tape drive for backup.  Performance on this setup is poor, with throughput that makes backing up 700 Gig impractical in a 24 hour period.  The primary backup of data is a flat folder file system managed by a vendor database. 

I need help troubleshooting, I know this isn't much info, let me see what I can copy from the case record:

We see the following types of errors in Windows Application log:

Fault bucket , type 0
Event Name: APPCRASH
Response: Not available
Cab Id: 0

Problem signature:
P1: beserver.exe
P2: 13.0.4164.106
P3: 4c740f1a
P4: KERNELBASE.dll
P5: 6.1.7600.16385
P6: 4a5bdfe0
P7: e06d7363
P8: 000000000000aa7d
P9:
P10:

 

Analysis symbol:
Rechecking for solution: 0
Report Id: e4b29708-f399-11df-9792-0026552792ee
Report Status: 4

Faulting application name: beserver.exe, version: 13.0.4164.106, time stamp: 0x4c740f1a
Faulting module name: KERNELBASE.dll, version: 6.1.7600.16385, time stamp: 0x4a5bdfe0
Exception code: 0xe06d7363
Fault offset: 0x000000000000aa7d
Faulting process id: 0x%9
Faulting application start time: 0x%10
Faulting application path: %11
Faulting module path: %12
Report Id: %13

Database Exception Context:AutoClearAlerts: sp ClearAutoAlert Error:-536837662: :: -2147467259:Login timeout expired :: DB Error Set 0: native=0xffffffff source=Microsoft OLE DB Provider for SQL Server hr=0x80004005 Login timeout expired

 

Since SQL is not on the server except for the SQL running Backup Exec, we are unsure why this keeps happening.

Any troubleshooting steps you can suggest are appreciated.

[Siddh]

Can you please download Backup exec tool from below URL.

https://www-secure.symantec.com/connect/forums/diagnose-issues-symantec-backup-exec-support-tool-bes...

If there are any errors in report kindly take snapshot in word file and upload the word file in blog.

Also check the event logs on server for any .net or SQL errors/warnings

1) Go to the BE installation directory and then use BEUtility to do a database consistency check and dump the database.  This is to ensure that the BEDB is in good shape.  If there are any errors, use the same utitlity to repair the database.

[pkh]

The primary backup of data is a flat folder file system managed by a vendor database. 

If it is possible, do a test by shutting down the application using this database before backing it up and see what is the result.

Are you using AOFO when you are backing up this database?

[VRT_KMONTGOM]

Hi Siddh and Pkh,  thanks for the comments.

Let me fill in a little more detail and we'll see if there are further suggestions.  I have been troubleshooting this with Symantec support for over 4 weeks, and they dropped the case due to an inability to not do voice contact with me. 

We have dumped logs, submitted them to Symantec, no avail.

I am not using AOFO on the database, as it isn't really a database.  There are millions of image files on this server, which are handled by a database on another server.  This is simply the 'content' and it is not in a true database structure.

Backup exec cannot handle the volume of filenames, so we broke the jobs down into approximately 10 jobs to handle the volume of filenames, which BE seems to handle now.

This morning I found the following error, which I am attempting to correct:

The Backup Exec Server Service did not start. An internal error (4313) occurred in object -1.

So I'm trying to determine the cause of this, and searching symantec and google it indicates that the database is in single user mode, which is causing it to crash.  I have downloaded Microsoft SQL Server Express 2005 management tools but cannot install them, the install fails, is there any other way to get past the single user mode database (first off, why would BE set itself up this way if it is a known problem, and secondly, is there a registry entry that sets this?)

Any suggestions are helpful, I'll try to pull some of the files you mentioned and post if I can.

Ken

[pkh]

I am not using AOFO on the database, as it isn't really a database.  There are millions of image files on this server, which are handled by a database on another server.  This is simply the 'content' and it is not in a true database structure.

You should use AOFO on these files so that BE can take a snapshot of these content files since these are just ordinary files.  Also, try turning of the database on the other server when BE is running.  The database may be holding on to these content files.

[VRT_KMONTGOM]

Thanks for the thoughts, but AOFO isn't reliable and we can't stop the database service.  AOFO reports a million errors as things may have changed with the data files from the initiation of the backup to the time they get verified.  This server seems to seriously underperform when using Backup Exec, as a LTO4 backup runs at around 411 Mb per minute, rather than the 2111 Mb per minute other servers are running at... and the backup process takes over 18 hours to complete, or rather, to kind of complete before the services fail.

The biggest problem I have right now is keeping all of the BE services running consistently, they continue to fail each evening.  This makes the backups impossible to complete.

[VRT_KMONTGOM]

Services continue to crash each evening, here is an example from the event viewer:

Faulting application name: beserver.exe, version: 13.0.4164.107, time stamp: 0x4cc975bc

Faulting module name: KERNELBASE.dll, version: 6.1.7600.16385, time stamp: 0x4a5bdfe0

Exception code: 0xe06d7363

Fault offset: 0x000000000000aa7d

Faulting process id: 0x%9

Faulting application start time: 0x%10

Faulting application path: %11

Faulting module path: %12

Report Id: %13

Any thoughts on this one?

[pkh]

Your problem is due to the fact that the "database" content files are being held or use by the database program.  There is nothing much that BE can do about this, but try to back them up as best as it could.  I hope you realise this is not a happy state of affair, but it is due to your environment and not entirely BE fault.

[VRT_KMONTGOM]

So services crash and jobs stop because it can't backup a single file?

That is not an acceptable answer.  The job should fail, not the entire service crash. If that were the case, every BE user would be screaming that the product is no good.

[pkh]

No.  You are not dealing with one file.  You did say that you have millions of image files to back up and is not using AOFO.

[VRT_KMONTGOM]

Ok, I see your point, but not all files are 'open' or in use.  Only a few of those millions of files are, and on other server BE skips those files. 

Point being, why do the services crash... why don't the jobs just fail?

[pkh]

I can only hazard a guess because I do not get to see the code.  I think it is an accumulation of small things.  It may be able to deal with some held files, but it is not engineered to deal with the volume of your files.  It crashes because your environment is not catered for.