Event ID 34327 - multiple ODBC Errors, Catalog issues
BE 2010 R3
After Hotfix 176937 was installed on April 12th, 2012, restarting BE (technically, BE Services) has resulted in a small cascade (22) of ODBC Access Errors in the Event Log, with IDs: 34327, followed by 34338, one after the other.
A call to Symantec Tech Support helped initially by doing a Repair from the original install files for BE, and then checking to see if we could run Catalog jobs against various B2D and IMG media, which we could.
However, I'm still experiencing failed jobs related, I believe, to errors in the Catalog. After the initial ODBC communication errors, I had to let the weekend full backups run before I was able to talk to Tech Support and do the Repair. I don't think good catalog information was recorded during that that weekend, and various errors cascaded during the following week. Because of this I suspect the daily differentials are having a hard time knowing exactly what they are differentializing from. (Adding to the problems, efforts to obtain clean full backups the following week, the array ran out of disk space, failing several of the same jobs that failed originally. A bit of a mess to clean up.) Doing one-off full backups of previously failed jobs did fix some of their problems, but this last weekend's suite of full backups resulted in the same e0009543 and e0009578 errors on 6 jobs - seemingly related to VMware "connection" issues, but I'm not so sure.
My main questions are: Is there a way to clean up these ODBC/Catalog errors? Do I need to re-catalog all the Disk backup media? Or will these errors eventually clear out once I have obtained clean full backups? Will unchecking "Use storage media-based catalogs" clear up the situation, or possibly worsen it?
One last setting: Tools | Options | Catalog has the option "Truncate catalogs after: 8 Weeks" checked. Is this wise?
Searching the TechNotes has resulted in very little helpful info, save for this item, which I have done today (although bengine.exe never surpassed 20% CPU utilization), so I'll see tomorrow if that helped or not: http://www.symantec.com/business/support/index?page=content&id=TECH35390
Many thanks for your assistance and suggestions,
.
Neil
____
The first error message is:
Event ID 34327:
Update to catalog index (Catalog index database) failed.
Reason: [Microsoft][ODBC SQL Server Driver][SQL Server]Cannot insert the value NULL into column 'ImageID', table 'BEDB.dbo.CatFragment'; column does not allow nulls. INSERT fails. cat_RecordSet::ExecuteBulkInsert()
e:\nicobar\5204r\becat\segodbc\seg_odbc.cpp(3113)
sp_sproc_columns CatInsertSetCopyVirtualSet
Subsequent instances of Event ID 34327 describe themselves thusly:
Update to catalog index (Catalog index database) failed.
Reason: [Microsoft][ODBC SQL Server Driver][SQL Server]The INSERT statement conflicted with the FOREIGN KEY constraint "FK_Frag_Image". The conflict occurred in database "BEDB", table "dbo.CatImage", column 'ImageID'. cat_RecordSet::ExecuteBulkInsert()
e:\nicobar\5204r\becat\segodbc\seg_odbc.cpp(3113)
sp_sproc_columns CatInsertSetCopyVirtualSet
and
Update to catalog index (Catalog index database) failed.
Reason: E:\Program Files\Symantec\Backup Exec\Catalogs\SVR-BUEX-05\{5C60BB7F-4B7A-4904-A2D3-24508C228C5D}_674.xml PerformBulkInsert()
e:\nicobar\5204r\becat\catindex\dll\..\catdbupdatethread.cpp(447)
These last two repeat each other, while the first message shows up only twice at the beginning of the slew of error messages, then doesn't repeat again.
Event ID 34338 merely announces: Backup Exec Alert: Catalog Error - ODBC access error. Possible lost connection to database or unsuccessful access to catalog index in the database.
Without investigating the original cause of this problem, Symantec's Tech Support said the way to deal with this is to allow BE to create a new Catalog folder and move forward. Since the ODBC errors do not appear to be causing any problems with the backup jobs, there is apparently little to worry about.
Somehow, that hotfix introduced errors into the Catalog folder. But a brand new Catalog folder does not have those errors (I tested this). It also doesn't have any older catalog information for my media, but c'est la vie, apparently.
The techie said that with any restore job you really want to run a "Catalog Media..." command on the media anyway before the restore, so the only thing I'm actually losing is the ability to view selection options for Restore jobs.
The process to rebuild the Catalog folder, for anyone that doesn't already know, is:
1. Stop all BE services
2. Rename the Catalog folder to "Catalog_old"
3. Restart BE services
-- a new Catalog folder will be created and you can move forward.
-- I suppose you can delete the Catalog_old folder at that point. I'm not sure there would be a good reason to keep it around.
Thinking about it now, perhaps if I ran a Catalog Media job on all the media used between 4/12 and 4/16 --the period when the original ODBC errors were causing bad or no catalog information to be written to the Catalog folder--, this might all clear up. Given the time that would take, I doubt I'll actually do that though.
Onward.