08-15-2019 07:01 AM
08-15-2019 07:12 AM
Are you running as root ?
08-15-2019 07:40 AM
Are you running the command on the master server?
Have you checked if all required processes are running on the master?
Please show us ouput of 'bpps -x'.
08-16-2019 01:24 AM
Yes am running as root user and i could see all the requied services are running.
08-16-2019 01:27 AM
@Marianne asked if you are are on the master ?
08-16-2019 01:33 AM
yes am executing the command from Master server only
08-16-2019 01:43 AM
If you are on master, as root, and all services are running, then that report command should work. The fact that it does not work suggests something is quite wrong with your environment. If you are not experienced enough to setup and configure logging and to then dig through the logs on your own then I think your best bet is to open an official support case with Veritas.
08-16-2019 01:45 AM
I was hoping that you would show us output of bpps -x.
We need confirmation that nbstserv is running (along with the rest...)
Status 25 means that connection could not be made to a particular process on the relevant server on a port (normally 1556).
Extract from a very old TN:
This error occurs when a Netbackup process times out while attempting to connect to another Netbackup process.
The initial goal of troubleshooting this issue should be to identify which machines are involved with the failure. For example, if the failure always involves the same Client or Media Server, this is where to concentrate efforts. Once this failure point is identified, then further troubleshooting can be done to identify the specific cause.
So, something wrong with internal comms on the master server.
Have you tried to check nbstserv log on the master server?
08-16-2019 01:52 AM
@Marianne please find the below NBU services currently running.
>:/usr/openv/netbackup/bin:$ ./bpps -x
NB Processes
------------
root 3938 72380 7 04:19 ? 00:00:02 /usr/openv/netbackup/bin/bpdbm
root 4518 72021 0 04:20 ? 00:00:00 /usr/openv/netbackup/bin/bprd
root 4519 72021 0 04:20 ? 00:00:00 /usr/openv/netbackup/bin/bprd
root 4572 72021 0 04:20 ? 00:00:00 /usr/openv/netbackup/bin/bprd
root 4574 72021 0 04:20 ? 00:00:00 /usr/openv/netbackup/bin/bprd
root 4582 72021 0 04:20 ? 00:00:00 /usr/openv/netbackup/bin/bprd
root 4588 72021 0 04:20 ? 00:00:00 /usr/openv/netbackup/bin/bprd
root 4592 72021 0 04:20 ? 00:00:00 /usr/openv/netbackup/bin/bprd
root 4593 72021 0 04:20 ? 00:00:00 /usr/openv/netbackup/bin/bprd
root 4594 72021 0 04:20 ? 00:00:00 /usr/openv/netbackup/bin/bprd
root 4608 72021 0 04:20 ? 00:00:00 /usr/openv/netbackup/bin/bprd
root 4609 72021 0 04:20 ? 00:00:00 /usr/openv/netbackup/bin/bprd
root 4610 72021 0 04:20 ? 00:00:00 /usr/openv/netbackup/bin/bprd
root 4611 72021 0 04:20 ? 00:00:00 /usr/openv/netbackup/bin/bprd
root 4843 72021 0 04:20 ? 00:00:00 /usr/openv/netbackup/bin/bprd
root 4871 72380 99 04:20 ? 00:00:02 /usr/openv/netbackup/bin/bpdbm
root 4910 72021 1 04:20 ? 00:00:00 /usr/openv/netbackup/bin/bprd
root 4971 72021 0 04:20 ? 00:00:00 /usr/openv/netbackup/bin/bprd
root 4997 72380 0 04:20 ? 00:00:00 /usr/openv/netbackup/bin/bpdbm
root 5003 72032 0 04:20 ? 00:00:00 /usr/openv/netbackup/bin/bpcompatd
root 14651 72380 3 00:48 ? 00:06:28 /usr/openv/netbackup/bin/bpdbm
root 71732 1 0 Jul23 ? 00:22:02 /usr/openv/netbackup/bin/bpcd -standalone
root 72021 1 0 Jul23 ? 02:12:33 /usr/openv/netbackup/bin/bprd
root 72032 1 0 Jul23 ? 00:20:16 /usr/openv/netbackup/bin/bpcompatd
root 72193 1 0 Jul23 ? 00:03:41 /usr/openv/netbackup/bin/nbcssc -a NetBackup
root 72380 1 1 Jul23 ? 08:39:05 /usr/openv/netbackup/bin/bpdbm
root 72503 72380 4 Jul23 ? 1-00:56:41 /usr/openv/netbackup/bin/bpjobd
root 77476 1 0 Jul23 ? 00:00:18 /usr/openv/netbackup/bin/private/nbatd -c /usr/openv/var/global/vxss/eab/data
root 77525 1 0 Jul23 ? 01:05:32 /usr/openv/netbackup/bin/private/nbazd
root 77562 1 0 Jul23 ? 00:13:44 /usr/openv/netbackup/bin/nbaudit
root 78404 1 8 Jul23 ? 2-00:54:08 /usr/openv/netbackup/bin/nbdisco
root 78422 1 1 Jul23 ? 06:10:44 /usr/openv/netbackup/bin/nbevtmgr
root 78762 1 10 Jul23 ? 2-09:28:38 /usr/openv/netbackup/bin/nbemm
root 78779 1 7 Jul23 ? 1-20:59:05 /usr/openv/netbackup/bin/nbrb
root 78858 1 8 Jul23 ? 2-00:15:56 /usr/openv/netbackup/bin/nbjm
root 78886 1 1 Jul23 ? 11:06:16 /usr/openv/netbackup/bin/nbpem
root 78931 1 0 Jul23 ? 01:18:49 /usr/openv/netbackup/bin/nbstserv
oot 78988 1 0 Jul23 ? 00:07:13 /usr/openv/netbackup/bin/nbrmms
root 79088 1 1 Jul23 ? 09:42:35 /usr/openv/netbackup/bin/nbsl
root 79140 1 0 Jul23 ? 00:05:52 /usr/openv/netbackup/bin/nbim
root 79222 1 6 Jul23 ? 1-10:56:36 /usr/openv/netbackup/bin/nbars
root 79441 1 0 Jul23 ? 00:00:52 /usr/openv/netbackup/bin/nbvault
root 79482 1 0 Jul23 ? 00:35:44 /usr/openv/netbackup/bin/nbsvcmon
root 79660 72380 0 Jul23 ? 00:21:21 /usr/openv/netbackup/bin/bpdbm
root 79669 78858 0 Jul23 ? 01:00:52 /usr/openv/netbackup/bin/nbproxy dblib nbjm
MM Processes
------------
root 72011 1 0 Jul23 ? 00:01:27 vmd
Shared Veritas Processes
-------------------------
root 77374 1 1 Jul23 ? 11:19:54 /opt/VRTSpbx/bin/pbx_exchange
>:/usr/openv/netbackup/bin:$
08-16-2019 02:11 AM
Not all master server processes are running and there seems to be excessive amount of bprd processes running.
The one that I'm missing immediately in your output is :
NB_dbsrv (Adaptive Server Anywhere - VERITAS_NB - ASA Database service ).
Please stop NBU.
Wait for processes to stop.
Check with bpps that everything has stopped.
Check disk space on the master - NBU will stop the relational database if disk gets full. This is to prevent database corruption.
Check that bprd and bpdbm log folders exist under /usr/openv/netbackup/logs .
If not, create them.
Start NBU.
Verify that all processes are running.
(See https://www.veritas.com/content/support/en_US/article.100002166.html )
If not, look for errors in these logs: bprd, bpdbm and /usr/openv/db/log/server.log.
08-16-2019 02:20 AM
Hello,
other NBU commands (bppllist, bpstulist etc.) are running fine?
Login/Work with NBU Console is OK?
And was nbstlutil ever working?
Sometimes the errors desribed could be caused also by expired certificates/licenses, then also other NBU components would have issues.
regards
Michal
08-16-2019 03:27 AM
@Marianne , we will restart the services once and check.
@Michal_Mikulik1 , bppllist command is working, but i could not execute nbstlutil report command and the same was working till last week.
08-16-2019 03:39 AM
Please check /usr/openv/db/log/server.log to see if there is a reason for the relational database going down.
08-16-2019 04:54 AM
@Marianne , i could see only the below details from server.log file.
I. 08/16 05:45:43. Finished checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2019 05:45
I. 08/16 05:50:18. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 05:50
I. 08/16 05:50:18. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 05:50
I. 08/16 05:56:36. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 05:56
I. 08/16 05:56:36. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 05:56
I. 08/16 06:00:44. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:00
I. 08/16 06:00:44. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:00
I. 08/16 06:05:44. Starting checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2019 06:05
I. 08/16 06:05:44. Finished checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2019 06:05
I. 08/16 06:08:07. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:08
I. 08/16 06:08:07. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:08
I. 08/16 06:11:02. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:11
I. 08/16 06:11:02. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:11
I. 08/16 06:19:39. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:19
I. 08/16 06:19:39. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:19
I. 08/16 06:23:38. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:23
I. 08/16 06:23:39. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:23
I. 08/16 06:25:45. Starting checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2019 06:25
I. 08/16 06:25:45. Finished checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2019 06:25
I. 08/16 06:29:08. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:29
I. 08/16 06:29:08. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:29
I. 08/16 06:33:08. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:33
I. 08/16 06:33:08. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:33
I. 08/16 06:40:16. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:40
I. 08/16 06:40:16. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:40
I. 08/16 06:45:46. Starting checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2019 06:45
I. 08/16 06:45:46. Finished checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2019 06:45
I. 08/16 06:46:14. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:46
I. 08/16 06:46:14. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:46
I. 08/16 06:52:38. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:52
I. 08/16 06:52:38. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:52
I. 08/16 07:00:54. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 07:00
I. 08/16 07:00:54. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 07:00
I. 08/16 07:05:47. Starting checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2019 07:05
I. 08/16 07:05:47. Finished checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2019 07:05
08-16-2019 07:14 AM
@Marianne , please find the below output of server.log.
I. 08/16 05:45:43. Finished checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2019 05:45
I. 08/16 05:50:18. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 05:50
I. 08/16 05:50:18. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 05:50
I. 08/16 05:56:36. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 05:56
I. 08/16 05:56:36. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 05:56
I. 08/16 06:00:44. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:00
I. 08/16 06:00:44. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:00
I. 08/16 06:05:44. Starting checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2019 06:05
I. 08/16 06:05:44. Finished checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2019 06:05
I. 08/16 06:08:07. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:08
I. 08/16 06:08:07. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:08
I. 08/16 06:11:02. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:11
I. 08/16 06:11:02. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:11
I. 08/16 06:19:39. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:19
I. 08/16 06:19:39. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:19
I. 08/16 06:23:38. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:23
I. 08/16 06:23:39. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:23
I. 08/16 06:25:45. Starting checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2019 06:25
I. 08/16 06:25:45. Finished checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2019 06:25
I. 08/16 06:29:08. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:29
I. 08/16 06:29:08. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:29
I. 08/16 06:33:08. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:33
I. 08/16 06:33:08. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:33
I. 08/16 06:40:16. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:40
I. 08/16 06:40:16. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:40
I. 08/16 06:45:46. Starting checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2019 06:45
I. 08/16 06:45:46. Finished checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2019 06:45
I. 08/16 06:46:14. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:46
I. 08/16 06:46:14. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:46
I. 08/16 06:52:38. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:52
I. 08/16 06:52:38. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 06:52
I. 08/16 07:00:54. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 07:00
I. 08/16 07:00:54. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2019 07:00
I. 08/16 07:05:47. Starting checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2019 07:05
I. 08/16 07:05:47. Finished checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2019 07:05
08-16-2019 07:17 AM
@Marianne , I could not see any error messages on server.log file. I have tried the same command"nbstlutil report" and am able to see the output now. This is really strange.
08-16-2019 08:18 AM
1) Well, you definitely were missing the database process, so I wonder if that is running again now. What does this command now show :
bpps -x | grep -i "dbs"
2) Did you restart NetBackup, or reboot the host ?
3) Did someone else restart NetBackup, or reboot the host ?
08-16-2019 08:33 AM
4) What do these commands show :
grep -v "^I" /usr/openv/db/log/server.log grep -v "^I" /usr/openv/db/log/*.slg