cancel
Showing results for 
Search instead for 
Did you mean: 

Catalog Hot Backup error

Sem
Level 3

Hello,

I have NetBackup 6.0 MP6 running on Windows 2003 and I am getting the following error whenever a hot backup of the catalog is run (manual or scheduled).

Error bpdm(pid=6076) db_getIMAGE() failed: no entity was found (227)

I've seen articles about making sure you run a full catalog backup but that doesn't resolve my issue. When I run this policy manually the only option is to run a full, so that's what I choose.

The location that the catalog backups to (and most of the backups that were written to disk) has changed. I've looked in the the Catalog Backup policy and changed the storage unit to one that is valid (it's backing up to a drive on one of our media server).

Is the error telling me that it can't write to the location I've specified? Is there another setting I should have changed in the policy?

Thanks.

1 ACCEPTED SOLUTION

Accepted Solutions

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Seems nbdelete was introduced in NBU 6.5. Sorry...

I'm not sure if the bpimage command will have the desired effect. This is what the Commands manual says:

-prunetir
Prunes the True Image Restore (TIR) information from the specified clients.
The default is all clients.

You could try 'bpimage -allcients -cleanup'.

I feel your best option at this point would be download and run the nbcc utility:

http://www.symantec.com/docs/TECH140582

Open a Support call with Symantec and submit the nbcc output for analysis. They will provide feedback on how to clean up inconsistencies.

Please do not wait too long to get this call logged. You do not want anything to happen to your master server while you do not have a catalog backup...

View solution in original post

16 REPLIES 16

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

I interpret the error as a problem with the source data - i.e. problem with images on the master.

Please enable the following logs:

admin and bpdm on the media server

admin bpbkar and bpdbm on the master

Sem
Level 3

Marianne,

Thank you for your post. I was able to find the location to enable logging on the media and master server. I set bpdm on the media and bpdbm on the master to level 5.

I am unable to locate where the admin and bpbkar settings are, is this done through the command prompt?

I have those log folders already created with some logs, but I'm not sure how to adjust the logging level.

I will post the logs I have.

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

You need to manually create the log directories under ...\netbackup\logs on each of the servers.

Rather drop down the logging level to something like a 3. Level 5 creates massive logs that cannot be opened with normal text editors. I have only ever increased logging to level 5 when Symantec support asked for level 5 logs.

NBU on master needs to be restarted for bpdbm log to take effect.

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

All sorts of image-related errors in bpdbm on master:

03:49:05.261 [5632.5152] <16> db_get_image_info: C:\Program Files\VERITAS\NetBackup\db\images\tulctrx01\1193000000\Windows_Application_Servers_1193805466_INCR is empty, no data retrieved

03:49:05.261 [5632.5152] <16> delete_expired_backups: Bad image header: Windows_Application_Servers_1193805466_INCR, error: file read failed (13)

There are quite a number of TN's that deal with 'bad image headers'. One of them: http://www.symantec.com/docs/TECH60787

 

Found this in media server bpdm log:

06:01:45.843 [6120.4140] <32> bp_sts_open_disk: Invalid STS storage device: D:\unix
06:01:45.843 [6120.4140] <32> delete_image_disk: bp_sts_open_disk failed: error 13
06:01:46.718 [6120.4140] <16> delete_image_disk: system call failed

06:01:46.718 [6120.4140] <2> bpdm: EXITING with status 83

You mentioned the following in your post:

"The location that the catalog backups to .... has changed."

Is D:\unix an STU path that no longer exists? NBU is looking for these images to expire and delete them. What happened to this location? Was the folder renamed or deleted?

This TN might help:

http://www.symantec.com/docs/TECH62005

Sem
Level 3

Yes, D:\unix does not exist anymore. I checked the storage unit groups and don't see any of them pointing to D:\unix.

Our hot catalog backups are failing and that (i believe) is causing our to unix databases to fail when they try to backup.

These changes would have been made close to a year ago but we just recently (within a week or so) started getting these errors.

I'm trying to get them to backup to one of our media servers that has space for their backups.

Where would I find d:\unix listed so that I can change it? Or would that not resolve the issue? (essentially NetBackup 6 is only backing up 2 unix servers).

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

D:\Unix is hard-coded in the image catalog. NBU needs to find it in order to clean up expired images. What have you done with these disk backups on D:\Unix?

Have you had a look at the two TN's that I've posted?

You need to cleanup the bad image headers and deal with the image cleanup. If the disk backups still exist under a new folder name, it will be best to put them back under D:\Unix. That is where NBU needs to find them for image cleanup as well as restores for un-expired backups. If the backups no longer exist, try the nbdelete command that is suggested in the 2nd TN.

Sem
Level 3

I've attached the logs from the master and media server.

Sem
Level 3

Sorry, I had to leave this for a couple of days.

I've looked at the TN's you mentioned but I was concerned that the nbdelete command would expire all backups?

Neither the backup location or the backups exist anymore.

Is it safe to run it or will it delete backups that haven't expired yet? Sorry I'm not very familiar with NetBackup.

Thanks for your help.

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Have a look at the Solution in this thread: https://www-secure.symantec.com/connect/forums/cant-delete-disk-pool

Run this:

/usr/openv/netbackup/bin/admincmd/nbdelete -allvolumes -force 

its a safe command. It just deletes the fragments that are already expired:

From man pages:


The nbdelete command removes all deleted fragments from  the
     disk  volumes  that  are  specified on the command line. The
     -allvolumes option removes the fragments  from  all  volumes

     that   contain   deleted   fragments. 

Sem
Level 3

I've looked in program files\veritas\nebackup\bin\admincmd and I don't see a nbdelete

I've done a search and looked in other directories and can't find nbdelete.

Is it installed by default? Is there a place to download the Windows version from Symantec?

Sem
Level 3

I can run bpimage, would this work the same?

bpimage -prunetir -allcients -cleanup

Is bpimage the windows version of nbdelete?

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Seems nbdelete was introduced in NBU 6.5. Sorry...

I'm not sure if the bpimage command will have the desired effect. This is what the Commands manual says:

-prunetir
Prunes the True Image Restore (TIR) information from the specified clients.
The default is all clients.

You could try 'bpimage -allcients -cleanup'.

I feel your best option at this point would be download and run the nbcc utility:

http://www.symantec.com/docs/TECH140582

Open a Support call with Symantec and submit the nbcc output for analysis. They will provide feedback on how to clean up inconsistencies.

Please do not wait too long to get this call logged. You do not want anything to happen to your master server while you do not have a catalog backup...

Sem
Level 3

Thanks Marianne, we don't have support on NetBackup and Symantec doesn't offer single instance support.

My only option for support are these forums.

Would it be safe to run the bpimage command or could that cause issues? I'm not familiar with TIR, not sure if it's a separately licensed piece or if we even used it.

Andy_Welburn
Level 6

going to suggest, from nbdelete not being available in 6.0 to using the bpimage command. (I stopped myself as some of the T/Ns seemed to intimate that nbdelete was there but I could find nothing in the KB nor Admin Guides specific to that version).

The bpimage -cleanup -allclients is safe - NB schedules this in at regular intervals as part of its housekeeping & it can be run manually if required. I'm doubtful tho' that it will assist in your issue.

TIR is part & parcel of NetBackup - it is a methodology by which you can backup & restore to a specific time taking into account file deletions etc. See the following for a better explanation:

http://www.symantec.com/business/support/index?page=content&id=HOWTO34490

 

Did you try the link Marianne also provided about "bad image headers"?

 

Can you recreate D:\Unix? Doing so may allow the catalog backup complete.....

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

TIR is not separately licensed - it's an attribute that you choose in you backup policy (Collect true image restore information).  So, if you're not using it, you don't need '-prunetir'.

The bpimage command is safe.

Another 'safe command' is:

bpexpdate -deassignempty -force

Please find and delete empty (size zero bytes) image header files (as per TN that I posted on 19 Nov) before you run 'bpexpdate'.

Check admin log on master if command does not complete successfully.

Sem
Level 3

I read through that TN and followed the steps to find the 'zero-byte header files' that the command was failing on.

I found two and removed them (actually i renamed them with _old), only my 3rd try running the command it came back with the same error - "could not update the media list - file read failed" but in the logs I saw this:

23:31:39.797 [6960.1404] <2> initLogging: Opening logger
23:31:39.797 [6960.1404] <2> bpdbm: C:\Program Files\VERITAS\NetBackup\Logs\bpdbm
23:31:39.797 [6960.1404] <2> logconnections: BPDBM ACCEPT FROM 10.99.1.11.2814 TO 10.99.1.5.13724
23:31:39.797 [6960.1404] <2> connected_peer: Connection from host tulbkup03.stgcrude.com, 10.95.1.114, on non-reserved port 2814
23:31:39.797 [6960.1404] <2> ParseConfigExA: Unknown configuration option on line 23: RenameIfExists = 0
23:31:39.797 [6960.1404] <2> ParseConfigExA: Obsolete configuration option on line 122: WAIT_IN_QUEUE = YES
23:31:39.797 [6960.1404] <2> ParseConfigExA: Obsolete configuration option on line 123: QUEUE_ON_ERROR = YES
23:31:39.797 [6960.1404] <2> nb_is_valid_master_server: checking if tulbkup03.stgcrude.com is a valid server

 

That is different than the old messages I was seeing about zero byte header files.

What I copied/pasted is the very end of the log file, it looks like there should be more, is NetBackup tell me what to do next and I'm just not seeing it?