ERR - Unable to process CATALOG_BACKUP cprvngisims OVM1_1328291103_FULL

[Willie_SA]

NetBackup 7.1

UNIX Solaris 10 Master / media server

After server failure due to out of memory condition and jobs failing, the following error ocure when catalog backup run:

02/09/2012 07:43:55 - Error bpbrm (pid=10505) from client cprvnmsnetb: ERR - NBU Catalog Backup API db_getImgBackupList() returned error: 229
02/09/2012 07:43:55 - Error bpbrm (pid=10505) from client cprvnmsnetb: ERR - Unable to process CATALOG_BACKUP cprvngisims OVM1_1328291103_FULL
02/09/2012 07:43:56 - Error bpbrm (pid=10505) from client cprvnmsnetb: ERR - NBU Catalog Backup API db_getImgBackupList() returned error: 229

.

There are about 30 of these entries.

The Image Cleanup did complete successful.

[mph999]

Looks like this might be a 'bad' image ...

02/09/2012 07:43:55 - Error bpbrm (pid=10505) from client cprvnmsnetb: ERR - Unable to process CATALOG_BACKUP cprvngisims OVM1_1328291103_FULL

This file will be in /usr/openv/netbackup/db/images/cprvngisims/1328000000

Can you post the file up here, and also, an ls -al on this file, and the file with the same name but ending in .f

Martin

[Yasuhisa_Ishika]

Try to check catalog consistency by "bpdbm -consistency 2".
By the way, have you already tried restarting your master server?

[Willie_SA]

I did bounce the server and bounce NetBackup.

BUT not since a successfull completion of the Image Cleaning.

Would bouncing NetBackup help?

[Willie_SA]

Hi,

Is this what you need?

Please remember that I only sent you extract from the job log in the Activity Monitor.

There is about 28 of these erros/warnings - it is all the jobs that was running when the server stopped.

root@cprvnmsnetb # pwd
/usr/openv/netbackup/db/images/cprvngisims/1328000000
root@cprvnmsnetb # ls -al
total 12314
drwxr-xr-x   4 root     root        1536 Feb  7 19:09 .
drwxr-xr-x  11 root     root         512 Feb  8 19:20 ..
-rw-r--r--   1 root     root           0 Feb  9 11:16 .lck
-rw-r--r--   1 root     root        2275 Feb  3 17:12 OVM1-SystemState_1328281200_FULL
-rw-------   1 root     root        1675 Feb  3 17:12 OVM1-SystemState_1328281200_FULL.f
-rw-r--r--   1 root     root        3796 Feb  1 21:32 OVM1_1328121983_INCR
-rw-------   1 root     root        1778 Feb  1 21:32 OVM1_1328121983_INCR.f
-rw-r--r--   1 root     root       60666 Feb  2 04:08 OVM1_1328122372_INCR
-rw-------   1 root     root          72 Feb  2 04:08 OVM1_1328122372_INCR.f
-rw-r--r--   1 root     root       16708 Feb  2 02:28 OVM1_1328122829_INCR
-rw-------   1 root     root     1956272 Feb  2 02:28 OVM1_1328122829_INCR.f
-rw-r--r--   1 root     root        1522 Feb  2 20:15 OVM1_1328204819_INCR
-rw-------   1 root     root        2003 Feb  2 20:15 OVM1_1328204819_INCR.f
-rw-r--r--   1 root     root       15795 Feb  2 23:51 OVM1_1328205677_INCR
-rw-------   1 root     root     1956212 Feb  2 23:51 OVM1_1328205677_INCR.f
-rw-r--r--   1 root     root         907 Feb  4 08:46 OVM1_1328283179_FULL
-rw-r--r--   1 root     root       15311 Feb  3 19:45 OVM1_1328283506_FULL
-rw-------   1 root     root     1956212 Feb  3 19:45 OVM1_1328283506_FULL.f
-rw-r--r--   1 root     root       85644 Feb  4 08:46 OVM1_1328289550_FULL
-rw-------   1 root     root          72 Feb  4 08:46 OVM1_1328289550_FULL.f
-rw-r--r--   1 root     root       15660 Feb  3 22:17 OVM1_1328291103_FULL
-rw-------   1 root     root          72 Feb  3 22:17 OVM1_1328291103_FULL.f
-rw-r--r--   1 root     root        4273 Feb  3 20:53 OVM1_1328294315_FULL
-rw-------   1 root     root        2468 Feb  3 20:53 OVM1_1328294315_FULL.f
-rw-r--r--   1 root     root         901 Feb  6 20:59 OVM1_1328552318_INCR
-rw-r--r--   1 root     root        1638 Feb  6 20:50 OVM1_1328552943_INCR
-rw-------   1 root     root        2354 Feb  6 20:50 OVM1_1328552943_INCR.f
-rw-r--r--   1 root     root        1396 Feb  6 20:49 OVM1_1328552944_INCR
-rw-------   1 root     root         507 Feb  6 20:49 OVM1_1328552944_INCR.f
-rw-r--r--   1 root     root        1726 Feb  6 20:59 OVM1_1328552945_INCR
-rw-------   1 root     root       58549 Feb  6 20:59 OVM1_1328552945_INCR.f
-rw-r--r--   1 root     root         927 Feb  6 20:33 OVM1_1328552946_INCR
-rw-------   1 root     root         507 Feb  6 20:33 OVM1_1328552946_INCR.f
-rw-r--r--   1 root     root         900 Feb  7 19:09 OVM1_1328634000_INCR
-rw-r--r--   1 root     root         926 Feb  7 19:03 OVM1_1328634197_INCR
-rw-------   1 root     root         507 Feb  7 19:03 OVM1_1328634197_INCR.f
-rw-r--r--   1 root     root         927 Feb  7 19:07 OVM1_1328634338_INCR
-rw-------   1 root     root         507 Feb  7 19:07 OVM1_1328634338_INCR.f
-rw-r--r--   1 root     root        1626 Feb  7 19:09 OVM1_1328634388_INCR
-rw-------   1 root     root        2354 Feb  7 19:09 OVM1_1328634388_INCR.f
-rw-r--r--   1 root     root        1416 Feb  7 19:09 OVM1_1328634458_INCR
-rw-------   1 root     root       74334 Feb  7 19:09 OVM1_1328634458_INCR.f
drwx------   2 root     root        1536 Feb  4 08:46 catstore
drwxr-xr-x   3 root     root         512 Feb  8 23:17 tmp
root@cprvnmsnetb # ls -al |grep 2911
-rw-r--r--   1 root     root       15660 Feb  3 22:17 OVM1_1328291103_FULL
-rw-------   1 root     root          72 Feb  3 22:17 OVM1_1328291103_FULL.f
root@cprvnmsnetb #
 

[Willie_SA]

Result from catalog check - bpdbm -consistence 2

Against all the files that gave wrning in Catalog backup job.

Bad image header: OVM2_1328642896_INCR, error: events out of sequence - image inconsistency (229)

[Mark_Solutions · Accepted solution]

This tech note, although covering a slightly different situation, covers how to remove corrput images from the catalog:

http://www.symantec.com/docs/TECH60787

Just follow the removal section for all 30 of your corrupt images - these must have all been open when your system crashed so it is unlikely that a repair of any sort is possible

[mph999]

Yep, agree with Mark .. You could remove the one reported by bpdbm -consistency, but then, wen you re-run, it will probably stick on the next.

So, if ...

OVM2_1328642896_INCR

is one of the 30 odd images, I think it would be reasonable to suggest to remove all 30, if YOU are happy with that (final decision is yours ...).

Martin

[Marianne]

Look at it this way - the backups that were running at the time are incomplete, making it impossible to restore from them.

All that is left now is the image cleanup.

[Willie_SA]

Problem solved as per TECH Note.

I moved one image reported with status code 229 in Job Activity Monitor Report from the /usr/openv/netbackup/db/images/?HOSTNAME?/1328000000 to pre defined directory /usr/openv/netbackup/db/images/willie 

Then reran Catalog Backup and that file did not apear in Job Activity Monitor Report.

Then ......

I moved all the other images reported with status code 229 in Job Activity Monitor Report from the /usr/openv/netbackup/db/images/?HOSTNAME?/1328000000 to pre defined directory /usr/openv/netbackup/db/images/willie.

Then ran the Catalog Backup and it complete with Status 0.

I can now go and delete /usr/openv/netbackup/db/images/willie

All OK

Thank you