Error 13 with Vault Job

[jounix]

Unix/Solaris

My vault job stop working over the weekend. I get an error 13. My backups are still running fine. i can still inject/eject tapes. I can telnet from the master and the robotic host to the clients/media servers. I've verified vol pool and groups, clients. Checked the nic settings. Anyone have any suggestions?

01/18/2010 12:50:55 - requesting resource nbup1a.NBVAULT.MAXJOBS
01/18/2010 12:50:55 - requesting resource nbup1a.NBU_POLICY.MAXJOBS.VT_EJECT
01/18/2010 12:50:55 - granted resource  nbup1a.NBVAULT.MAXJOBS
01/18/2010 12:50:55 - granted resource  nbup1a.NBU_POLICY.MAXJOBS.VT_EJECT
01/18/2010 12:50:57 - vault waiting for session ID lock
01/18/2010 12:50:59 - vault session ID lock acquired
01/18/2010 12:50:59 - vault session ID lock released
01/18/2010 12:50:55 - estimated 0 kbytes needed
01/18/2010 12:50:55 - begin Parent Job
01/18/2010 12:50:55 - begin Vault: Start Notify Script
01/18/2010 12:50:56 - started process RUNCMD (pid=23726)
Operation Status: 0
01/18/2010 12:50:56 - end Vault: Start Notify Script; elapsed time 0:00:01
01/18/2010 12:50:56 - begin Vault: Execute Script
01/18/2010 12:50:56 - started process bpbrm (pid=23735)
01/18/2010 12:50:57 - requesting resource nbup1a.VAULT_CREATE_SESSION_ID.LOCK_TLD(1)_ADIC1PZ
01/18/2010 12:50:57 - granted resource  nbup1a.VAULT_CREATE_SESSION_ID.LOCK_TLD(1)_ADIC1PZ
01/18/2010 12:51:10 - vault global lock released
01/18/2010 12:51:10 - end writing
Operation Status: 13
01/18/2010 12:51:10 - end Vault: Execute Script; elapsed time 0:00:14
01/18/2010 12:51:10 - begin Vault: Stop On Error
Operation Status: 0
01/18/2010 12:51:10 - end Vault: Stop On Error; elapsed time 0:00:00
01/18/2010 12:51:10 - begin Vault: End Notify Script
01/18/2010 12:51:10 - started process RUNCMD (pid=23828)
Operation Status: 0
01/18/2010 12:51:10 - end Vault: End Notify Script; elapsed time 0:00:00
Operation Status: 13
01/18/2010 12:51:10 - end Parent Job; elapsed time 0:00:15
file read failed (13)
 

[Bruce999]

I ran into this problem as well and have a resolution for it.  The problem lies with one or more  images that have been backed up and are in your  image catalog that have a consistency issue.

Output a consistency check on the image catalog to find the specific image(s) that are causing file read errors.

UNIX:

/usr/openv/netbackup/bin/bpdbm -consistency 0 > /tmp/bpdbm_out.txt

Windows:

<install_path>\netbackup\bin\bpdbm -consistency 0 > C:\temp\bpdbm_out.txt

***NOTE***

This can take several hours (or more) depending on size of catalog, network, resources avail...etc.  This file can be quite large and you will have to do a search for "error" or "inconsistancy"  "Bad"...you get the idea.

Search the output file once the check has completed.

here is the output from our check that shows the error:

check directory of client <system_name>
________________________________________________

checking image file <WNT-FRI-2200-HQ-TAPE-INF-PROD-HQFILE1-Projects_1266023915_FULL>
checking files file <WNT-FRI-2200-HQ-TAPE-INF-PROD-HQFILE1-Projects_1266023915_FULL.f>
checking image file <WNT-FRI-2200-HQ-TAPE-INF-PROD-HQFILE1-Projects_1266022811_FULL>
checking files file <WNT-FRI-2200-HQ-TAPE-INF-PROD-HQFILE1-Projects_1266022811_FULL.f>
checking image file <WNT-FRI-2200-HQ-TAPE-INF-PROD-HQFILE1-Projects_1266022480_FULL>
checking files file <WNT-FRI-2200-HQ-TAPE-INF-PROD-HQFILE1-Projects_1266022480_FULL.f>
checking image file <WNT-FRI-2200-HQ-TAPE-INF-PROD-HQFILE1-Projects_1266025588_FULL>
checking files file <WNT-FRI-2200-HQ-TAPE-INF-PROD-HQFILE1-Projects_1266025588_FULL.f>
checking image file <WNT-FRI-2200-HQ-TAPE-INF-PROD-HQFILE1-Projects_1266025204_FULL>
checking files file <WNT-FRI-2200-HQ-TAPE-INF-PROD-HQFILE1-Projects_1266025204_FULL.f>
checking image file <WNT-FRI-2200-HQ-TAPE-INF-PROD-HQFILE1-Projects_1266022579_FULL>
checking files file <WNT-FRI-2200-HQ-TAPE-INF-PROD-HQFILE1-Projects_1266022579_FULL.f>
checking image file <WNT-WED-2200-HQ-TAPE-HQFILE1_1266037224_INCR>
Bad image header: WNT-WED-2200-HQ-TAPE-HQFILE1_1266037224_INCR, error: events out of sequence - image inconsistency (229)

you will need to remove (move to a new folder of your choice) the TWO (if one exists with a .f )corresponding entries with the same image ID that are related to the error from your netbackup/db/images/ <system_name> folder.  In this case we moved the following to a temp folder.  We had 6 files with error total

checking image file <WNT-WED-2200-HQ-TAPE-HQFILE1_1266037224_INCR>
Bad image header: WNT-WED-2200-HQ-TAPE-HQFILE1_1266037224_INCR, error: events out of sequence - image inconsistency (229)

once you have removed ALL errored image files...run vault again.  Reports should run as you have set them up.

Hope this helps.