Event ID 34327 - multiple ODBC Errors, Catalog issues

[nPc]

BE 2010 R3

After Hotfix 176937 was installed on April 12th, 2012, restarting BE (technically, BE Services) has resulted in a small cascade (22) of ODBC Access Errors in the Event Log, with IDs: 34327, followed by 34338, one after the other.

A call to Symantec Tech Support helped initially by doing a Repair from the original install files for BE, and then checking to see if we could run Catalog jobs against various B2D and IMG media, which we could.

However, I'm still experiencing failed jobs related, I believe, to errors in the Catalog.  After the initial ODBC communication errors, I had to let the weekend full backups run before I was able to talk to Tech Support and do the Repair.  I don't think good catalog information was recorded during that that weekend, and various errors cascaded during the following week. Because of this I suspect the daily differentials are having a hard time knowing exactly what they are differentializing from.  (Adding to the problems, efforts to obtain clean full backups the following week, the array ran out of disk space, failing several of the same jobs that failed originally. A bit of a mess to clean up.)  Doing one-off full backups of previously failed jobs did fix some of their problems, but this last weekend's suite of full backups resulted in the same e0009543 and e0009578 errors on 6 jobs - seemingly related to VMware "connection" issues, but I'm not so sure.

My main questions are:  Is there a way to clean up these ODBC/Catalog errors?    Do I need to re-catalog all the Disk backup media?   Or will these errors eventually clear out once I have obtained clean full backups?   Will unchecking "Use storage media-based catalogs" clear up the situation, or possibly worsen it? 

One last setting:  Tools | Options | Catalog  has the option "Truncate catalogs after: 8 Weeks"  checked.   Is this wise?

Searching the TechNotes has resulted in very little helpful info, save for this item, which I have done today (although bengine.exe never surpassed 20% CPU utilization), so I'll see tomorrow if that helped or not:  http://www.symantec.com/business/support/index?page=content&id=TECH35390

Many thanks for your assistance and suggestions,

 .
 Neil

____

The first error message is:

Event ID 34327:

Update to catalog index (Catalog index database) failed.
Reason:  [Microsoft][ODBC SQL Server Driver][SQL Server]Cannot insert the value NULL into column 'ImageID', table 'BEDB.dbo.CatFragment'; column does not allow nulls. INSERT fails. cat_RecordSet::ExecuteBulkInsert()
e:\nicobar\5204r\becat\segodbc\seg_odbc.cpp(3113)
sp_sproc_columns CatInsertSetCopyVirtualSet
 

Subsequent instances of Event ID 34327 describe themselves thusly:

Update to catalog index (Catalog index database) failed.
Reason:  [Microsoft][ODBC SQL Server Driver][SQL Server]The INSERT statement conflicted with the FOREIGN KEY constraint "FK_Frag_Image". The conflict occurred in database "BEDB", table "dbo.CatImage", column 'ImageID'. cat_RecordSet::ExecuteBulkInsert()
e:\nicobar\5204r\becat\segodbc\seg_odbc.cpp(3113)
sp_sproc_columns CatInsertSetCopyVirtualSet
 

and

Update to catalog index (Catalog index database) failed.
Reason:  E:\Program Files\Symantec\Backup Exec\Catalogs\SVR-BUEX-05\{5C60BB7F-4B7A-4904-A2D3-24508C228C5D}_674.xml PerformBulkInsert()
e:\nicobar\5204r\becat\catindex\dll\..\catdbupdatethread.cpp(447)
 

These last two repeat each other, while the first message shows up only twice at the beginning of the slew of error messages, then doesn't repeat again.

Event ID 34338 merely announces: Backup Exec Alert: Catalog Error - ODBC access error. Possible lost connection to database or unsuccessful access to catalog index in the database.
 

[pkh]

Unless you are short of space, you should not truncate your catalog.  If you truncate your catalog, you would have to catalog your media before you can restore from them.  This is despite the media's overwrite protection period is not over and the media is not overwritten.

[King_Julien]

Got a similar issue once,  ¿You receive that message each time a job completes, right?

If yes, I will check tomorrow on my job about how I was able to fix it.

[nPc]

pkh -  Ok, thanks.  I'll review space issues vis-a-vis the catalog size.

Julien -- Actually no.  I only find those error message in the Windows Event Log when the BE services start (either upon reboot, or simple restart of services [manually or using the BE Utility]).

[King_Julien]

Hi Neil,

As far as I can found on the KB, this issue is currently under investigation, please take a look at

Getting "ODBC access error. Possible lost connection to database or unsuccessful access to catalog index in the database" on restarting Backup Exec Services

http://www.symantec.com/docs/TECH167225

Meaning that a running a repair of the installation might not help. I think the best for you would be to contact Symantec Technical support having the debug files of the media server so they can do a further investigation, with the event viewer logs (evt format) so they can match exactly what happens exactly on the debug when the error is loggued in the event viewer.

To enable the debug log you just have to modifiy the following registry key:

HKLM\Software\Symantec\Backup Exec\Backup Exec for Windows Servers\Engine\Logging

Then, change the registry entry "CreateDebugLog" from 0 to 1 and restart BE services.

Wait for the error to appears again on the event viewer and then collect the debug files, you will find those files on "x:\program files\symantec\backup exec\logs", the debug logs are named like "SERVERNAME-BENGINExxxx.log", "SERVERNAME-BEREMOTExxxx.log"

Once you get those files, proceed to disable debugging by changing the value for "CreateDebugLog" back to 0, and then restart BE services.

Having all that information, contact Symantec Support and reference to http://www.symantec.com/docs/TECH167225

Hope it helps.

Have a nice day!

[nPc · Accepted solution]

Without investigating the original cause of this problem, Symantec's Tech Support said the way to deal with this is to allow BE to create a new Catalog folder and move forward.  Since the ODBC errors do not appear to be causing any problems with the backup jobs, there is apparently little to worry about.

Somehow, that hotfix introduced errors into the Catalog folder.  But a brand new Catalog folder does not have those errors (I tested this).  It also doesn't have any older catalog information for my media, but c'est la vie, apparently.

The techie said that with any restore job you really want to run a "Catalog Media..." command on the media anyway before the restore, so the only thing I'm actually losing is the ability to view selection options for Restore jobs.

The process to rebuild the Catalog folder, for anyone that doesn't already know, is:

1. Stop all BE services

2. Rename the Catalog folder to "Catalog_old"

3. Restart BE services

-- a new Catalog folder will be created and you can move forward.

-- I suppose you can delete the Catalog_old folder at that point.  I'm not sure there would be a good reason to keep it around.

Thinking about it now, perhaps if I ran a Catalog Media job on all the media used between 4/12 and 4/16 --the period when the original ODBC errors were causing bad or no catalog information to be written to the Catalog folder--, this might all clear up.  Given the time that would take, I doubt I'll actually do that though. 

Onward.