Forum Discussion

10 Replies

  • Can the master server reach all of the media servers on the vnetd (13724) port?  Is the unreachable host identified in the job's detailed status?
  • yes master can reach all of media server via vnetd port

    here backup job status :

    12/16/2009 3:14:10 AM - Info bpdbm(pid=5708) image catalog cleanup         
    12/16/2009 3:14:10 AM - Info bpdbm(pid=5708) deleting images which expire before Wed Dec 16 03:14:10 2009 (1260962050) 
    12/16/2009 3:14:10 AM - Info bpdbm(pid=5708) processing client DS01A         
    12/16/2009 3:14:10 AM - Info bpdbm(pid=5708) processing client DS02A         
    12/16/2009 3:14:10 AM - Info bpdbm(pid=5708) processing client DVWEB02         
    12/16/2009 3:14:10 AM - Info bpdbm(pid=5708) processing client PASCVIFILE01         
    12/16/2009 3:14:10 AM - Info bpdbm(pid=5708) processing client PASCVIFILE02         
    12/16/2009 3:14:10 AM - Info bpdbm(pid=5708) processing client PASCVISQL01         
    12/16/2009 3:14:10 AM - Info bpdbm(pid=5708) processing client PASCVISQL02         
    12/16/2009 3:14:10 AM - Info bpdbm(pid=5708) processing client backup01a         
    12/16/2009 3:14:10 AM - Info bpdbm(pid=5708) processing client backup01a.hanqual.com         
    12/16/2009 3:14:10 AM - Info bpdbm(pid=5708) processing client eft01         
    12/16/2009 3:14:10 AM - Info bpdbm(pid=5708) processing client eft02         
    12/16/2009 3:14:10 AM - Info bpdbm(pid=5708) processing client exch01a         
    12/16/2009 3:14:31 AM - Info bpdbm(pid=5708) processing client file01b         
    12/16/2009 3:14:32 AM - Info bpdbm(pid=5708) processing client file02b         
    12/16/2009 3:14:32 AM - Info bpdbm(pid=5708) processing client file03b         
    12/16/2009 3:14:32 AM - Info bpdbm(pid=5708) processing client file04b         
    12/16/2009 3:14:32 AM - Info bpdbm(pid=5708) processing client fsapps         
    12/16/2009 3:14:41 AM - Info bpdbm(pid=5708) processing client fsarchive         
    12/16/2009 3:14:41 AM - Info bpdbm(pid=5708) processing client fsnonprod         
    12/16/2009 3:16:03 AM - Info bpdbm(pid=5708) processing client fsprod         
    12/16/2009 3:16:54 AM - Info bpdbm(pid=5708) processing client hqmon01a         
    12/16/2009 3:16:54 AM - Info bpdbm(pid=5708) processing client hqmon01b         
    12/16/2009 3:16:54 AM - Info bpdbm(pid=5708) processing client hrws002         
    12/16/2009 3:16:57 AM - Info bpdbm(pid=5708) processing client jtcdspr-ds01         
    12/16/2009 3:17:06 AM - Info bpdbm(pid=5708) processing client jtcdspr-ds01.hanqual.com         
    12/16/2009 3:17:06 AM - Info bpdbm(pid=5708) processing client jtcdspr-ds02         
    12/16/2009 3:17:12 AM - Info bpdbm(pid=5708) processing client mail01a         
    12/16/2009 3:17:12 AM - Info bpdbm(pid=5708) processing client mail01b         
    12/16/2009 3:17:12 AM - Info bpdbm(pid=5708) processing client mail02a         
    12/16/2009 3:17:12 AM - Info bpdbm(pid=5708) processing client mail02b         
    12/16/2009 3:17:12 AM - Info bpdbm(pid=5708) processing client mail05a         
    12/16/2009 3:17:12 AM - Info bpdbm(pid=5708) processing client sec02         
    12/16/2009 3:17:12 AM - Info bpdbm(pid=5708) processing client sec04a         
    12/16/2009 3:17:12 AM - Info bpdbm(pid=5708) processing client sqlsrv04a         
    12/16/2009 3:17:12 AM - Info bpdbm(pid=5708) processing client sqlsrv04b         
    12/16/2009 3:17:12 AM - Info bpdbm(pid=5708) processing client st0006stweb02.hanqual.com         
    12/16/2009 3:17:12 AM - Info bpdbm(pid=5708) processing client stcap01         
    12/16/2009 3:17:12 AM - Info bpdbm(pid=5708) processing client stias01         
    12/16/2009 3:17:13 AM - Info bpdbm(pid=5708) processing client stsqlsrv01         
    12/16/2009 3:17:13 AM - Info bpdbm(pid=5708) processing client stsqlsrv01a         
    12/16/2009 3:17:13 AM - Info bpdbm(pid=5708) processing client stweb02.hanqual.com         
    12/16/2009 3:17:13 AM - Info bpdbm(pid=5708) processing client vsql01a         
    12/16/2009 3:17:13 AM - Info bpdbm(pid=5708) processing client vsql01b         
    12/16/2009 3:17:13 AM - Info bpdbm(pid=5708) processing client vsql02a         
    12/16/2009 3:17:14 AM - Info bpdbm(pid=5708) processing client vsql02b         
    12/16/2009 3:17:14 AM - Info bpdbm(pid=5708) processing client vsql03a         
    12/16/2009 3:17:14 AM - Info bpdbm(pid=5708) processing client vsql03b         
    12/16/2009 3:17:14 AM - Info bpdbm(pid=5708) processing client vsql04a         
    12/16/2009 3:17:14 AM - Info bpdbm(pid=5708) processing client vsql04b         
    12/16/2009 3:17:14 AM - Info bpdbm(pid=5708) deleted 0 expired records, compressed 0, tir removed 0, deleted 0 expired copies
    host is unreachable(47)
  • Indeed it doesn't identify the un-reachable host, but we do get some more information, we get the bpdbm pid.  Now that pid may not be helpful right now if there are no bpdbm logs, at the same time it suggests if we configure the bpdbm logging directory we should be able to trace the log messages associated with the pid.

    Go ahead and create the bpdbm logging directory on the master server, netbackup/logs/bpdbm, once created increase the logging level for bpdbm via the NetBackup administration console -> master server host properties -> logging, or by configuring the 'BPDBM_VERBOSE = 5' within the master's bp.conf file.  Once the verbose levels have been changed, bounce the services on the master or wait for a good time to bounce the services.

    Wait a bit to get another image cleanup job to run, or kick one off by running I believe, bpimage -allclients -cleanup.  Let the job fail, it should fail the same way, then go ahead take note of the new bpdbm PID.  Now go to the netbackup/logs/bpdbm/ directory on the master server, and then grep out the pid from the log file, e.g. 'grep <pid> log.date'.

    Look in the grep output for '<16>' those are usually severe messages, also try looking for the 'host unreachable', or 'status 47', or simply '47'.

    Anything valuable in there?
  • Thanks NBUSE, I created new bpdbm log and wait the next image job runs.
  • We had this recently & it was related to several images that still existed from an old media server that was decommissioned years ago. It only came to light once we upgraded to 6.5.4 (6.5.1 obviously wasn't bothered with them!!). Needed to run a piece of sql supplied by Symantec to clean-up these images.

    Ref: "Problems" with Image Cleanup since upgrade to 6.5.4
  • Please have something meaningful for the support technican if you open a call, such as that verbose bpdbm log.
  • here my bpdbm log :

    00:11:45.074 [7172.2172] <2> set_job_details: Sending jobData jobid (7621)
    00:11:45.074 [7172.2172] <2> send_structure_data: Index 35 Field m_dtLastInfo Value <1261555905>
    00:11:45.074 [7172.2172] <2> set_job_details: Done
    00:11:45.074 [7172.2172] <2> IsCatalogCleanupTerminated: Terminated = 0
    00:11:45.387 [7172.2172] <16> delete_expired_backups: OVsystem(C:\Program Files\Veritas\NetBackup\bin\admincmd\nbdelete.exe -allvolumes -jobid 7621) failed (47)
    00:11:45.387 [7172.2172] <2> IsCatalogCleanupTerminated: Terminated = 0
    00:11:45.387 [7172.2172] <2> set_job_details: Sending jobData jobid (7621)
    00:11:45.387 [7172.2172] <2> send_structure_data: Index 9 Field m_nStatus Value <47>
    00:11:45.387 [7172.2172] <2> send_structure_data: Index 21 Field m_nState Value <3>
    00:11:45.387 [7172.2172] <2> send_structure_data: Index 30 Field m_dtEnded Value <1261555905>
    00:11:45.387 [7172.2172] <2> set_job_details: Done
    00:11:45.387 [7172.2172] <2> job_end_try: Done
    00:11:45.387 [7172.2172] <2> job_monitoring_exex: ACK disconnect
    00:11:45.387 [7172.2172] <2> job_disconnect: Disconnected
    00:11:45.387 [7172.2172] <4> delete_expired_backups: Exiting
    00:14:42.449 [7940.3420] <2> getIDIRSTRUCT: ?
  • Check Master server CPU Usage when Image cleanup is failing with status code 47
  • Check the master server CPU Usage when Image cleanup is failing.