partial successful backups

[somesh_p]

stream discovery is failing with error code 26, which is resulting in partial successful backups. can anyone suggest how to get rid off this issue

12/17/2013 20:28:14 - end Stream Discovery: Stream Discovery; elapsed time 0:00:00

12/17/2013 20:28:14 - begin Stream Discovery: Bare Metal Restore Save

12/17/2013 20:28:14 - started process bpbrm (pid=26779)

Operation Status: 26

[RamNagalla]

you would need to provide more deatils

what is the netbackup version in Master/media & client?

what are the OS versions in Master/media & client?

Does it BMR backup?

does it ever successfull?

Does Mulitstreaming enabled for the  backups?

post the Full detail status 

[SymTerry]

Hello Somesh, 

As metioned by Nagalla, if you could please reply with the detailed status of the failed job and BMR logging. For BMR logging please follow HOWTO45935.

[somesh_p]

Netbackup version is 7.5.0.4 on both master and client
Policy created recently for this client and it was never successful
It does BMR backup. Multistreaming enabled
OS version on both master and client is Solaris 10

Here is the full details

12/17/2013 20:28:13 - Info nbjm (pid=1005) starting backup job (jobid=120) for client trcxb2456bkp, policy trcxb2456bkp_FILES, schedule Weekly_Full

12/17/2013 20:28:13 - Info nbjm (pid=1005) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=120, request id:{C0CE47D8-6759-11E3-9C17-002128001016})

12/17/2013 20:28:13 - requesting resource trcx837-bup-hcart-robot-tld-0

12/17/2013 20:28:13 - requesting resource trcx837-bup.NBU_CLIENT.MAXJOBS.trcxb2456bkp

12/17/2013 20:28:13 - requesting resource trcx837-bup.NBU_POLICY.MAXJOBS.trcxb2456bkp_FILES

12/17/2013 20:28:13 - granted resource trcx837-bup.NBU_CLIENT.MAXJOBS.atrcxb2456bkp

12/17/2013 20:28:13 - granted resource trcx837-bup.NBU_POLICY.MAXJOBS.trcxb2456bkp_FILES

12/17/2013 20:28:13 - granted resource 6701L4

12/17/2013 20:28:13 - granted resource HP.ULTRIUM4-SCSI.001

12/17/2013 20:28:13 - granted resource trcx837-bup-hcart-robot-tld-0

12/17/2013 20:28:13 - estimated 0 kbytes needed

12/17/2013 20:28:13 - begin Parent Job

12/17/2013 20:28:13 - begin Stream Discovery: Start Notify Script

12/17/2013 20:28:13 - Info RUNCMD (pid=26746) started

12/17/2013 20:28:14 - Info RUNCMD (pid=26746) exiting with status: 0

Operation Status: 0

12/17/2013 20:28:14 - end Stream Discovery: Start Notify Script; elapsed time 0:00:01

12/17/2013 20:28:14 - begin Stream Discovery: Stream Discovery

12/17/2013 20:28:14 - Info bpmount (pid=18955) started

12/17/2013 20:28:14 - Info bpmount (pid=18955) exiting with status: 0

Operation Status: 0

12/17/2013 20:28:14 - end Stream Discovery: Stream Discovery; elapsed time 0:00:00

12/17/2013 20:28:14 - begin Stream Discovery: Bare Metal Restore Save

12/17/2013 20:28:14 - started process bpbrm (pid=26779)

Operation Status: 26

12/17/2013 20:28:16 - collecting BMR information

12/17/2013 20:28:16 - connecting

12/17/2013 20:28:16 - connected; connect time: 0:00:00

12/17/2013 20:28:16 - transfering BMR information to the master server

12/17/2013 20:28:16 - connecting

12/17/2013 20:28:16 - connected; connect time: 0:00:00

12/17/2013 20:28:27 - end Stream Discovery: Bare Metal Restore Save; elapsed time 0:00:13

12/17/2013 20:28:27 - begin Stream Discovery: Policy Execution Manager Preprocessed

Operation Status: 0

12/17/2013 20:37:16 - end Stream Discovery: Policy Execution Manager Preprocessed; elapsed time 0:08:49

12/17/2013 20:37:16 - begin Stream Discovery: Validate Image

Operation Status: 0

12/17/2013 20:37:16 - end Stream Discovery: Validate Image; elapsed time 0:00:00

12/17/2013 20:37:16 - begin Stream Discovery: End Notify Script

12/17/2013 20:37:16 - Info RUNCMD (pid=28494) started

12/17/2013 20:37:17 - Info RUNCMD (pid=28494) exiting with status: 0

Operation Status: 0

12/17/2013 20:37:17 - end Stream Discovery: End Notify Script; elapsed time 0:00:01

Operation Status: 1

12/17/2013 20:37:17 - end Parent Job; elapsed time 0:09:04

the requested operation was partially successful (1)

[Marianne]

Is there a firewall between master/media server and client?

Please create bpcd log folder on the client under /usr/opwnv/netbackup/logs.

Post client's bpcd log after next backup failure.

[somesh_p]

There is no firewall between master and client.

no logging is being made to bpcd.

here the logging in bmrsavecfg on client side

bash-3.2# pwd
/usr/openv/logs/bmrsavecfg
bash-3.2# cat 51216-121-3297262346-131223-0000000000.log
0,51216,128,121,1,1387781300244,12668,1,0:,86:Could not do stat on file /usr/openv/netbackup/baremetal/client/data/ClientCustomFiles,44:fileAttribsCommon.cpp:IsFileReadableCommon(),1
0,51216,128,121,2,1387781300294,12668,1,0:,42:Invalid input parameters, exiting, line=50,30:stringManipulation.cpp:Split(),1
0,51216,128,121,3,1387781300303,12668,1,0:,42:Invalid input parameters, exiting, line=50,30:stringManipulation.cpp:Split(),1
0,51216,128,121,4,1387781302709,12668,1,0:,53:Child process pid=12772 exited with non-zero status=1,37:bmrSystemCommon.cpp:BmrSystemCommon(),1
1,51216,128,121,5,1387781302709,12668,1,0:,0:,26:CSaveCfgFile.cpp:execute(),2,(128|u64:66|A74:[ -r /kernel/drv/sg.conf ] && /usr/bin/cat /kernel/drv/sg.conf 2>/dev/null|A16:CSaveCfgFile.cpp|i32:137|i32:1|i32:1|)
0,51216,128,121,6,1387781302717,12668,1,0:,53:Child process pid=12773 exited with non-zero status=1,37:bmrSystemCommon.cpp:BmrSystemCommon(),1
1,51216,128,121,7,1387781302717,12668,1,0:,0:,26:CSaveCfgFile.cpp:execute(),2,(128|u64:66|A61:[ -x /etc/powermt ]&&/etc/powermt display dev=all 2>/dev/null|A16:CSaveCfgFile.cpp|i32:137|i32:1|i32:1|)
0,51216,128,121,8,1387781303066,12668,1,0:,53:Child process pid=12828 exited with non-zero status=1,37:bmrSystemCommon.cpp:BmrSystemCommon(),1
1,51216,128,121,9,1387781303066,12668,1,0:,0:,26:CSaveCfgFile.cpp:execute(),2,(128|u64:66|A49:[ -x /sbin/vxlicrep ]&&/sbin/vxlicrep 2>/dev/null|A16:CSaveCfgFile.cpp|i32:137|i32:1|i32:1|)
0,51216,128,121,10,1387781303074,12668,1,0:,53:Child process pid=12829 exited with non-zero status=1,37:bmrSystemCommon.cpp:BmrSystemCommon(),1
1,51216,128,121,11,1387781303074,12668,1,0:,0:,26:CSaveCfgFile.cpp:execute(),2,(128|u64:66|A89:[ -x /usr/sbin/vxdisk ]&&/usr/sbin/vxdisk -q list  2>/dev/null|awk '$1!="-" { print $1 }'|A16:CSaveCfgFile.cpp|i32:137|i32:1|i32:1|)
0,51216,128,121,12,1387781303082,12668,1,0:,53:Child process pid=12830 exited with non-zero status=1,37:bmrSystemCommon.cpp:BmrSystemCommon(),1
1,51216,128,121,13,1387781303082,12668,1,0:,0:,26:CSaveCfgFile.cpp:execute(),2,(128|u64:66|A79:[ -x /usr/sbin/vxdisk ]&&/usr/sbin/vxdg -q list  2>/dev/null|awk '{ print $1 }'|A16:CSaveCfgFile.cpp|i32:137|i32:1|i32:1|)
0,51216,128,121,14,1387781303089,12668,1,0:,53:Child process pid=12831 exited with non-zero status=1,37:bmrSystemCommon.cpp:BmrSystemCommon(),1
1,51216,128,121,15,1387781303090,12668,1,0:,0:,26:CSaveCfgFile.cpp:execute(),2,(128|u64:66|A58:[ -x /usr/sbin/vxdctl ]&&/usr/sbin/vxdctl list 2>/dev/null|A16:CSaveCfgFile.cpp|i32:137|i32:1|i32:1|)
0,51216,128,121,16,1387781303097,12668,1,0:,53:Child process pid=12832 exited with non-zero status=1,37:bmrSystemCommon.cpp:BmrSystemCommon(),1
1,51216,128,121,17,1387781303097,12668,1,0:,0:,26:CSaveCfgFile.cpp:execute(),2,(128|u64:66|A79:[ -x /usr/sbin/vxdisk ]&&/usr/sbin/vxdg -q list  2>/dev/null|awk '{ print $1 }'|A16:CSaveCfgFile.cpp|i32:137|i32:1|i32:1|)
0,51216,128,121,18,1387781303105,12668,1,0:,53:Child process pid=12833 exited with non-zero status=1,37:bmrSystemCommon.cpp:BmrSystemCommon(),1
1,51216,128,121,19,1387781303105,12668,1,0:,0:,26:CSaveCfgFile.cpp:execute(),2,(128|u64:66|A89:[ -x /usr/sbin/vxdisk ]&&/usr/sbin/vxdisk -q list  2>/dev/null|awk '$1!="-" { print $1 }'|A16:CSaveCfgFile.cpp|i32:137|i32:1|i32:1|)
0,51216,128,121,20,1387781303189,12668,1,0:,53:Child process pid=12834 exited with non-zero status=1,37:bmrSystemCommon.cpp:BmrSystemCommon(),1
1,51216,128,121,21,1387781303189,12668,1,0:,0:,26:CSaveCfgFile.cpp:execute(),2,(128|u64:66|A40:/usr/bin/pkginfo -l VRTSvxvm 2>/dev/null|A16:CSaveCfgFile.cpp|i32:137|i32:1|i32:1|)
0,51216,128,121,22,1387781303273,12668,1,0:,53:Child process pid=12836 exited with non-zero status=1,37:bmrSystemCommon.cpp:BmrSystemCommon(),1
1,51216,128,121,23,1387781303273,12668,1,0:,0:,26:CSaveCfgFile.cpp:execute(),2,(128|u64:66|A40:/usr/bin/pkginfo -l VRTSvxfs 2>/dev/null|A16:CSaveCfgFile.cpp|i32:137|i32:1|i32:1|)
0,51216,128,121,24,1387781303273,12668,1,0:,48:Could not do stat on file /opt/VRTSvcs/bin/hares,46:fileAttribsCommon.cpp:IsFileExecutableCommon(),1
0,51216,121,121,1,1387781303336,12668,1,0:,27:BMROP_IMPORT failed, rc=199,31:CBmrSaveCfg.cpp:sendDiscovery(),1
2,51216,121,121,2,1387781303336,12668,1,0:,0:,0:,2,(27|)
0,51216,121,121,3,1387781303345,12668,1,0:,44:Failed sending the discovery, rc=199, mode=1,23:CBmrSaveCfg::discover(),1
2,51216,121,121,4,1387781303345,12668,1,0:,0:,0:,2,(21|)
0,51216,121,121,5,1387781303345,12668,1,0:,66:The NBU client version is incompatible with the NBU master server.,18:main.cpp:bmrmain(),1
2,51216,121,121,6,1387781303346,12668,1,0:,0:,0:,2,(71|)
 

[Marianne]

Netbackup version is 7.5.0.4 on both master and client

:Failed sending the discovery, rc=199, mode=1,23:CBmrSaveCfg::discover(),1

NBU client version is incompatible with the NBU master server

Maybe you need to disable BMR for now and confirm that comms is fine between the master and this client. Does backup work with BMR de-selected?

If bpcd folder exists on the client and no log file is created, there is 'something' between the client and server preventing comms.

Can you confirm that bpcd and vnetd are both running on the client?

Can you use normal OS commands to communicate with the client using hostnames? (ping, traceroute).

Have you tested forward and reverse name lookup between server and client?
You can use this TN for commands: 

How to verify name resolution for NetBackup (tm) systems, using the "bpclntcmd" command
http://www.symantec.com/docs/TECH27430

When this is working fine from master and from client, do the following to test port connection:

Double-check that bpcd folder exists on client under /usr/openv/netbackup/logs and that vnetd and bpcd processes are running.

Run the following on the master:

bptestbpcd -client <client-name> -verbose -debug

Please post output of command and bpcd log on client.

[somesh_p]

both master and client are running on 7.5.0.4.  bpcd log is attached in file attachments

[somesh_p]

communication is fine. bptestbpcd is running fine.  backup is running fine (with status code 0 ) after unticking BMR in policy attributes. bpcd log is attached

[Marianne]

So, comms is working fine and you have an issue with BMR backup and/or BMR config on the master server..

Maybe start a new discussion for unsuccessful BMR backup?

[Jaime_Vazquez · Accepted solution]

Known issue for BMR enable backups at NBU 7.5.0.4. It is described in the Late Breakling News Bulletin for NBU 7.5.  See this link: http://www.symantec.com/docs/TECH178334 

(ET2991238) <<Fixed in 7.5.0.5>> BMR enabled backup job is partially successful (status code 1) after upgrade to Netbackup 7.5.0.4, Parent job fails reporting status code 26
 http://www.symantec.com/docs/TECH200322

The failing module is "bpbrm".  Best solution is to upgrade Master/Media servers to NBU 7.5.0.5 or later. If not at all possible, request EEB for ET2991238.

[Jaime_Vazquez]

Looking for an update on this matter.  It should have been resolved by the actions already posted.