cancel
Showing results for 
Search instead for 
Did you mean: 

Netbackup database corruption

dozer14
Level 2

Hello,

I inherited a NetBackup set of servers.  One of the media servers is not backing up any files due to the 2074 "Disk volume is down" message.  I have verified that the DD volume is there and useable from a file explorer standpoint.  I am also not able to start the NetBackup Deduplication Engine or Manager.  It failes with Error 1 : Incorrect funtion.  I am guessing in my limited knowledge of NetBackup this is due to the volume it can't see.  The pure disk is showing up when I run the commands to verify this from the CLI.  When I run the spoold DB integrity check it fails with the below entries (C:\Program Files\Veritas\pdde>spoold –trace).  I am not able to find anything online stating how to rebuild the DB or indexes.

December 07 15:12:29 INFO [00000000014D5B90]: Task thread stack size: 0 bytes
December 07 15:12:35 ERR [00000000014D5B90]: -1: _dcHeaderRead: invalid version of container header 826252557
December 07 15:12:35 ERR [00000000014D5B90]: 25032: _storeCheckContainers: failed to read index headerfrom container 148009 (data corrupt)
December 07 15:12:35 ERR [00000000014D5B90]: -1: _dcHeaderRead: invalid version of container header 786686286
December 07 15:12:35 ERR [00000000014D5B90]: 25032: _storeCheckContainers: failed to read index headerfrom container 152657 (data corrupt)
December 07 15:12:35 ERR [00000000014D5B90]: -1: _dcHeaderRead: invalid version of container header 816054650
December 07 15:12:35 ERR [00000000014D5B90]: 25032: _storeCheckContainers: failed to read index headerfrom container 152656 (data corrupt)
December 07 15:12:35 ERR [00000000014D5B90]: 25032: _dcHeaderRead: corrupt dc header, crc32 is 3932161, expected 0
December 07 15:12:35 ERR [00000000014D5B90]: 25032: _storeCheckContainers: failed to read index headerfrom container 152655 (data corrupt)
December 07 15:12:35 ERR [00000000014D5B90]: 25032: _dcHeaderRead: corrupt dc header, crc32 is 3932161, expected 0
December 07 15:12:35 ERR [00000000014D5B90]: 25032: _storeCheckContainers: failed to read index headerfrom container 152654 (data corrupt)
December 07 15:12:35 ERR [00000000014D5B90]: 25032: _dcHeaderRead: corrupt dc header, crc32 is 3932161, expected 0
December 07 15:12:35 ERR [00000000014D5B90]: 25032: _storeCheckContainers: failed to read index headerfrom container 152653 (data corrupt)
December 07 15:12:35 ERR [00000000014D5B90]: -1: _dcHeaderRead: invalid version of container header 19005440
December 07 15:12:35 ERR [00000000014D5B90]: 25032: _storeCheckContainers: failed to read index headerfrom container 155254 (data corrupt)
December 07 15:12:36 ERR [00000000014D5B90]: -1: _dcHeaderRead: invalid version of container header 2982209609
December 07 15:12:36 ERR [00000000014D5B90]: 25032: _storeCheckContainers: failed to read index headerfrom container 168960 (data corrupt)
December 07 15:12:36 ERR [00000000014D5B90]: 25032: _dcHeaderRead: corrupt dc header, crc32 is 300357040, expected 2209462768
December 07 15:12:36 ERR [00000000014D5B90]: 25032: _storeCheckContainers: failed to read index headerfrom container 174198 (data corrupt)
December 07 15:12:36 ERR [00000000014D5B90]: -1: _dcHeaderRead: invalid version of container header 3859960578
December 07 15:12:36 ERR [00000000014D5B90]: 25032: _storeCheckContainers: failed to read index headerfrom container 18720 (data corrupt)
December 07 15:12:36 ERR [00000000014D5B90]: -1: _dcHeaderRead: invalid version of container header 2730215532
December 07 15:12:36 ERR [00000000014D5B90]: 25032: _storeCheckContainers: failed to read index headerfrom container 185344 (data corrupt)
December 07 15:12:36 ERR [00000000014D5B90]: 25032: _dcHeaderRead: corrupt dc header, crc32 is 3932161, expected 0
December 07 15:12:36 ERR [00000000014D5B90]: 25032: _storeCheckContainers: failed to read index headerfrom container 187407 (data corrupt)
December 07 15:12:36 ERR [00000000014D5B90]: 25009: _storeCheckContainers: container 201749 is in a wrong group folder(196)
December 07 15:12:36 ERR [00000000014D5B90]: 25009: DataStore Class: Could not initialize for instance 0
December 07 15:12:36 ERR [0000000000E68D20]: -1: DataStoreManager::init: failed to initialize mount point 0
December 07 15:12:36 INFO [0000000000E68D20]: DataStoreManager::init: joined [thread 00000000014D5B90] for initializing mount point 0

Any help would be greatly appreciated.

Thank you!

8 REPLIES 8

Mouse
Moderator
Moderator
Partner    VIP    Accredited Certified

I guess your next call should be to Support and see if they can do anything

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified
Please tell us what is "DD volume"?
We normally use that abbreviation for DataDomain.
But you seem to be describing MSDP?

If this is MSDP, you need assistance from Veritas Support.
Only problem is that your 7.6 NBU version is no longer supported.

DD - dedupe

I see the confusion on the acronym.  I take it there is nothing to do except upgrade and or call support?

Thanks in advance

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

I doubt that Veritas will take your call while on unsupported version of NBU.

If you know for sure that all MSDP images have been duplicated to alternate storage, it might be easiest to remove and recreate the MSDP pool.

Anyway - PLEASE schedule NBU upgrade as a matter of urgency.

There has been a technote for spoold incorrect function, which did not require a rebuild of the database.

Probably be worth a look.

 

 

The standard questions: Have you checked: 1) What has changed. 2) The manual 3) If there are any tech notes or VOX posts regarding the issue

Thank you for the replies.  It was deemed not necessary to purchase support for this product at some juncture.  With that being said, I agree in that they probably will not accept a support call b/c the version is no longer on the support matrix.  As I am not very familiar with NBU.  Is there a good reference doc that will explain the steps or high level on what to do to remove this media pool and rebuild it/rejoin it?  I believe it has been a few weeks since this one isolated media pool has not backed up anything.  

Any guidance is appreciated.

Thanks!

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

 

 


@dozer14 wrote:

Thank you for the replies.  It was deemed not necessary to purchase support for this product at some juncture. 

 


The problem with a decision like this is that you can compare it with insurance.

It is a grudged expense, especially if one has not claimed for quite a while. 
Murphy says that you WILL suffer some kind of loss at some point. Not having insurance is thus never a good idea.

NetBackup maintenance gives you much more than support with issues being experienced.

It gives you access to bug fixes, access to versions of the software that always include new features (e.g. lots of improvements as far as MSDP is concerned), support for new versions of OS, databases and applications, support for products not having agents previously, etc. 

Hope your management reconsiders...

About removing this MSDP pool - please understand that all backup data on this diskpool will be lost. 
Please always ensure that MSDP data is duplicated to alternate storage (other MSDP at same or other site, cloud, tape, etc.)