nbdeployutil Error

[backup-botw]

Anyone ever seen something like this before?

root@masterserver:/usr/openv/netbackup/bin/admincmd ->./nbdeployutil --gather --hoursago=720
NetBackup Deployment Utility, version 7.6.0.2_00.0424
Gathering license deployment information...
  Discovered master server masterserver
Failed to parse tpconfig data from /usr/openv/var/global/reports/20150806_122652_masterserver/tpconfig.out
  Failed gathering for masterserver, /usr/openv/var/global/reports/20150806_122652_masterserver
Gather DONE
Execution time: 13 mins 20 secs

Unable to gather information for the following masters:
  masterserver in directory /usr/openv/var/global/reports/20150806_122652_masterserver

Never had a problem running this before as it has always worked, but I went to run it today and am seeing this problem. Nothing has changed in the environment as far as version being used or anything like that.

[CRZ]

I found exactly one old case with this error reported, and its resolution involved finding that one of the commands nbdeployutil was trying to run had suddenly had its binary replaced with a 0kb touchfile.

This sounds like a long shot to me, but could you take a quick look through your admincmd directory and make sure all the binaries are there and have a nonzero file size?  (`find . -size 0` might be fastest)

[backup-botw]

Beyond strange...

-rw-r--r--   1 root     root           0 Jun 23 09:58 bpclntcmd
-r-xr-xr-x   1 root     bin            0 Jun 23 09:58 bptestbpcd
-rw-r--r--   1 root     root           0 Jun 23 09:58 telnet

[sdo]

You can get NetBackup to report the version of it's own executables using the 'versioninfo' tool, e.g:

[root@rhel64 /]# /usr/openv/netbackup/bin/goodies/support/versioninfo -f /usr/openv/volmgr/bin/tpconfig 
======= /usr/openv/volmgr/bin/tpconfig =======
@(#) NetBackup_7.5.0.7 3042285129

If you don't see any version details for tpconfig on your NetBackup Server, then someone has probably goofed with a CLI command and overwritten the binary executable file.

.

Do you need a hand configuring your login scripts to establish 'path' for NetBackup binaries, so that you never need to be in any of the binaries folders when executing NetBackup CLI commands?  If so, see Martin's super post here:

https://www-secure.symantec.com/connect/articles/making-your-life-easier-netbackup

...which explains many things, and also covers defining 'path' to include the NetBackup binaries.

[backup-botw]

The tpconfig file looks ok from this perspective...

-r-xr-xr-x   1 root     bin      1226176 Apr 24  2014 tpconfig

Thats to say it isnt zeroed out like the others.

I will definitely reference that thread and add these to my path. Actually been meaning to do so and just never seem to get around to it.

[CRZ · Accepted solution]

I think we have a bingo! 

Can you find some known good versions of those binaries and get them replaced?

(Not sure which one is your problem - suspect bptestbpcd)

[backup-botw]

Would it be as easy as just downloading the file from my other master and then uploading it to the broken one?

[sdo]

Ok - so the tpconfig file is not zero bytes - and so contains something... but what is that something?  The file could have been overwritten with text nonsense.

Try a 'file tpconfig', does it report as a valid binary executable?

Try a NetBackup 'versioninfo' on it, does it report expected version?

[backup-botw]

Both look good there...

root@masterserver:/usr/openv/volmgr/bin ->file tpconfig
tpconfig:       ELF 64-bit MSB executable SPARCV9 Version 1, dynamically linked, not stripped

root@masterserver:/usr/openv/netbackup/bin/goodies/support ->./versioninfo -f /usr/openv/volmgr/bin/tpconfig
======= /usr/openv/volmgr/bin/tpconfig =======
@(#) NetBackup_7.6.0.2 1765632197

[CRZ]

So long as the versions of your masters match, yes.  Wouldn't hurt to doublecheck that permissions are the same after they're "restored" as well.

[sdo]

I believe that nbdeployutil is itself an executable, and not a script, so it will be difficult to work out what it's doing.

I'd try fixing/replacing the zero'ed executables first, as you need those for normal day-to-day admin tasks anyway.  Do they have a recent modify time?  Is it possible to work put when they were lost?

Then maybe try nbdeployutil again asfterwards.  Maybe nbdeployutil has a verbose, or logging option?

[backup-botw]

They are both running 7.6.0.2

[backup-botw]

Judging by what I am seeing in the zeroed files I believe it was done on the same day...

-rw-r--r--   1 root     root           0 Jun 23 09:58 bpclntcmd
-r-xr-xr-x   1 root     bin            0 Jun 23 09:58 bptestbpcd
-rw-r--r--   1 root     root           0 Jun 23 09:58 telnet

June 23rd around 9:58AM.

[sdo]

I've just tried corrupting bptestbpcd to see what effect it would have on nbdeployutil...

[root@rhel64 /]# cd tmp

[root@rhel64 tmp]# /usr/openv/netbackup/bin/admincmd/nbdeployutil --gather --output="/tmp/sdo"
NetBackup Deployment Utility, version 7.5.0.7
Gathering license deployment information...
  Discovered master server hserver
  failed bptestbpcd to 4 of 5 clients, for details see: /tmp/sdo/20150807_161816_hserver/nbdeployutil-gather-20150807_161816.log
  Output for hserver at: /tmp/sdo/20150807_161816_hserver    
Gather DONE
Execution time: 36 secs
To create a report for this master server, run one of the following:
  capacity   : nbdeployutil --report --capacity /tmp/sdo/20150807_161816_hserver
  traditional: nbdeployutil --report --traditional /tmp/sdo/20150807_161816_hserver

[root@rhel64 tmp]# cd /usr/openv/netbackup/bin

[root@rhel64 bin]# find . -name bptestbpcd
./admincmd/bptestbpcd

[root@rhel64 bin]# cd admincmd/

[root@rhel64 admincmd]# mv bptestbpcd bptestbpcd.sav

[root@rhel64 admincmd]# touch bptestbpcd

[root@rhel64 admincmd]# cd /tmp

[root@rhel64 tmp]# /usr/openv/netbackup/bin/admincmd/nbdeployutil --gather --output="/tmp/sdo"
NetBackup Deployment Utility, version 7.5.0.7
Gathering license deployment information...
  Discovered master server hserver
`tpconfig` status: 127 (exec failed, Permission denied)
`nbdevquery` status: 127 (exec failed, Permission denied)
`nbemmcmd` status: 127 (exec failed, Permission denied)
`emmcmd_lh` status: 127 (exec failed, Permission denied)
`nbftconfig` status: 127 (exec failed, Permission denied)
`vault` status: 127 (exec failed, Permission denied)
  failed bptestbpcd to 5 of 5 clients, for details see: /tmp/sdo/20150807_172724_hserver/nbdeployutil-gather-20150807_172724.log
  Failed gathering for hserver, /tmp/sdo/20150807_172724_hserver
Gather DONE                           
Execution time: 2 secs                
Unable to gather information for the following masters:
  hserver in directory /tmp/sdo/20150807_172724_hserver

[root@rhel64 tmp]#

...and then putting the bptestbpcd binary file back:

[root@rhel64 tmp]# cd /usr/openv/netbackup/bin/admincmd/

[root@rhel64 admincmd]# rm bptestbpcd
rm: remove regular empty file `bptestbpcd'? y

[root@rhel64 admincmd]# mv bptestbpcd.sav bptestbpcd

[root@rhel64 admincmd]# cd /tmp

[root@rhel64 tmp]# /usr/openv/netbackup/bin/admincmd/nbdeployutil --gather --output="/tmp/sdo"
NetBackup Deployment Utility, version 7.5.0.7
Gathering license deployment information...
  Discovered master server hserver
  failed bptestbpcd to 4 of 5 clients, for details see: /tmp/sdo/20150807_173113_hserver/nbdeployutil-gather-20150807_173113.log
  Output for hserver at: /tmp/sdo/20150807_173113_hserver    
Gather DONE
Execution time: 37 secs
To create a report for this master server, run one of the following:
  capacity   : nbdeployutil --report --capacity /tmp/sdo/20150807_173113_hserver
  traditional: nbdeployutil --report --traditional /tmp/sdo/20150807_173113_hserver

[root@rhel64 tmp]#

...and all is ok again.

[CRZ]

This should already have been resolved by now!

Please replace those missing binaries and you should find that things will start working again.

[backup-botw]

Already done and it is indeed working again. Fixed it and went to lunch.