08-06-2015 10:51 AM
Anyone ever seen something like this before?
root@masterserver:/usr/openv/netbackup/bin/admincmd ->./nbdeployutil --gather --hoursago=720
NetBackup Deployment Utility, version 7.6.0.2_00.0424
Gathering license deployment information...
Discovered master server masterserver
Failed to parse tpconfig data from /usr/openv/var/global/reports/20150806_122652_masterserver/tpconfig.out
Failed gathering for masterserver, /usr/openv/var/global/reports/20150806_122652_masterserver
Gather DONE
Execution time: 13 mins 20 secs
Unable to gather information for the following masters:
masterserver in directory /usr/openv/var/global/reports/20150806_122652_masterserver
Never had a problem running this before as it has always worked, but I went to run it today and am seeing this problem. Nothing has changed in the environment as far as version being used or anything like that.
Solved! Go to Solution.
08-06-2015 02:42 PM
I think we have a bingo!
Can you find some known good versions of those binaries and get them replaced?
(Not sure which one is your problem - suspect bptestbpcd)
08-06-2015 11:31 AM
I found exactly one old case with this error reported, and its resolution involved finding that one of the commands nbdeployutil was trying to run had suddenly had its binary replaced with a 0kb touchfile.
This sounds like a long shot to me, but could you take a quick look through your admincmd directory and make sure all the binaries are there and have a nonzero file size? (`find . -size 0` might be fastest)
08-06-2015 11:51 AM
Beyond strange...
-rw-r--r-- 1 root root 0 Jun 23 09:58 bpclntcmd
-r-xr-xr-x 1 root bin 0 Jun 23 09:58 bptestbpcd
-rw-r--r-- 1 root root 0 Jun 23 09:58 telnet
08-06-2015 11:58 AM
You can get NetBackup to report the version of it's own executables using the 'versioninfo' tool, e.g:
[root@rhel64 /]# /usr/openv/netbackup/bin/goodies/support/versioninfo -f /usr/openv/volmgr/bin/tpconfig ======= /usr/openv/volmgr/bin/tpconfig ======= @(#) NetBackup_7.5.0.7 3042285129
If you don't see any version details for tpconfig on your NetBackup Server, then someone has probably goofed with a CLI command and overwritten the binary executable file.
.
Do you need a hand configuring your login scripts to establish 'path' for NetBackup binaries, so that you never need to be in any of the binaries folders when executing NetBackup CLI commands? If so, see Martin's super post here:
https://www-secure.symantec.com/connect/articles/making-your-life-easier-netbackup
...which explains many things, and also covers defining 'path' to include the NetBackup binaries.
08-06-2015 12:21 PM
The tpconfig file looks ok from this perspective...
-r-xr-xr-x 1 root bin 1226176 Apr 24 2014 tpconfig
Thats to say it isnt zeroed out like the others.
I will definitely reference that thread and add these to my path. Actually been meaning to do so and just never seem to get around to it.
08-06-2015 02:42 PM
I think we have a bingo!
Can you find some known good versions of those binaries and get them replaced?
(Not sure which one is your problem - suspect bptestbpcd)
08-06-2015 02:53 PM
Would it be as easy as just downloading the file from my other master and then uploading it to the broken one?
08-06-2015 02:54 PM
Ok - so the tpconfig file is not zero bytes - and so contains something... but what is that something? The file could have been overwritten with text nonsense.
Try a 'file tpconfig', does it report as a valid binary executable?
Try a NetBackup 'versioninfo' on it, does it report expected version?
08-06-2015 03:05 PM
Both look good there...
root@masterserver:/usr/openv/volmgr/bin ->file tpconfig
tpconfig: ELF 64-bit MSB executable SPARCV9 Version 1, dynamically linked, not stripped
root@masterserver:/usr/openv/netbackup/bin/goodies/support ->./versioninfo -f /usr/openv/volmgr/bin/tpconfig
======= /usr/openv/volmgr/bin/tpconfig =======
@(#) NetBackup_7.6.0.2 1765632197
08-06-2015 03:09 PM
So long as the versions of your masters match, yes. Wouldn't hurt to doublecheck that permissions are the same after they're "restored" as well.
08-06-2015 03:26 PM
I believe that nbdeployutil is itself an executable, and not a script, so it will be difficult to work out what it's doing.
I'd try fixing/replacing the zero'ed executables first, as you need those for normal day-to-day admin tasks anyway. Do they have a recent modify time? Is it possible to work put when they were lost?
Then maybe try nbdeployutil again asfterwards. Maybe nbdeployutil has a verbose, or logging option?
08-07-2015 07:19 AM
They are both running 7.6.0.2
08-07-2015 07:21 AM
Judging by what I am seeing in the zeroed files I believe it was done on the same day...
-rw-r--r-- 1 root root 0 Jun 23 09:58 bpclntcmd
-r-xr-xr-x 1 root bin 0 Jun 23 09:58 bptestbpcd
-rw-r--r-- 1 root root 0 Jun 23 09:58 telnet
June 23rd around 9:58AM.
08-07-2015 09:42 AM
I've just tried corrupting bptestbpcd to see what effect it would have on nbdeployutil...
[root@rhel64 /]# cd tmp [root@rhel64 tmp]# /usr/openv/netbackup/bin/admincmd/nbdeployutil --gather --output="/tmp/sdo" NetBackup Deployment Utility, version 7.5.0.7 Gathering license deployment information... Discovered master server hserver failed bptestbpcd to 4 of 5 clients, for details see: /tmp/sdo/20150807_161816_hserver/nbdeployutil-gather-20150807_161816.log Output for hserver at: /tmp/sdo/20150807_161816_hserver Gather DONE Execution time: 36 secs To create a report for this master server, run one of the following: capacity : nbdeployutil --report --capacity /tmp/sdo/20150807_161816_hserver traditional: nbdeployutil --report --traditional /tmp/sdo/20150807_161816_hserver [root@rhel64 tmp]# cd /usr/openv/netbackup/bin [root@rhel64 bin]# find . -name bptestbpcd ./admincmd/bptestbpcd [root@rhel64 bin]# cd admincmd/ [root@rhel64 admincmd]# mv bptestbpcd bptestbpcd.sav [root@rhel64 admincmd]# touch bptestbpcd [root@rhel64 admincmd]# cd /tmp [root@rhel64 tmp]# /usr/openv/netbackup/bin/admincmd/nbdeployutil --gather --output="/tmp/sdo" NetBackup Deployment Utility, version 7.5.0.7 Gathering license deployment information... Discovered master server hserver `tpconfig` status: 127 (exec failed, Permission denied) `nbdevquery` status: 127 (exec failed, Permission denied) `nbemmcmd` status: 127 (exec failed, Permission denied) `emmcmd_lh` status: 127 (exec failed, Permission denied) `nbftconfig` status: 127 (exec failed, Permission denied) `vault` status: 127 (exec failed, Permission denied) failed bptestbpcd to 5 of 5 clients, for details see: /tmp/sdo/20150807_172724_hserver/nbdeployutil-gather-20150807_172724.log Failed gathering for hserver, /tmp/sdo/20150807_172724_hserver Gather DONE Execution time: 2 secs Unable to gather information for the following masters: hserver in directory /tmp/sdo/20150807_172724_hserver [root@rhel64 tmp]#
...and then putting the bptestbpcd binary file back:
[root@rhel64 tmp]# cd /usr/openv/netbackup/bin/admincmd/ [root@rhel64 admincmd]# rm bptestbpcd rm: remove regular empty file `bptestbpcd'? y [root@rhel64 admincmd]# mv bptestbpcd.sav bptestbpcd [root@rhel64 admincmd]# cd /tmp [root@rhel64 tmp]# /usr/openv/netbackup/bin/admincmd/nbdeployutil --gather --output="/tmp/sdo" NetBackup Deployment Utility, version 7.5.0.7 Gathering license deployment information... Discovered master server hserver failed bptestbpcd to 4 of 5 clients, for details see: /tmp/sdo/20150807_173113_hserver/nbdeployutil-gather-20150807_173113.log Output for hserver at: /tmp/sdo/20150807_173113_hserver Gather DONE Execution time: 37 secs To create a report for this master server, run one of the following: capacity : nbdeployutil --report --capacity /tmp/sdo/20150807_173113_hserver traditional: nbdeployutil --report --traditional /tmp/sdo/20150807_173113_hserver [root@rhel64 tmp]#
...and all is ok again.
08-07-2015 09:44 AM
This should already have been resolved by now!
Please replace those missing binaries and you should find that things will start working again.
08-07-2015 12:00 PM
Already done and it is indeed working again. Fixed it and went to lunch.