04-08-2014 03:25 AM
VCS 5.1 NBU 7.5 Solaris 10 SPARC.
Hi,
I am getting bellow error during the NBU resource group online and it is not starting in all cluster nodes. I did tried exporting environment variables “NB_BPDBM_ALIVE_LOCAL_TIMEOUT=59” and even tried increasing the OnlineRetryLimit=2 in NetBackup types file, but still the resource group is not coming up and it is faulted in all cluster nodes.
“VCS ERROR V-16-2-13066 Agent is calling clean for resource(nbu_server) because the resource is not up even after online completed.”
Anyone suggest me how to solve this issue? Upgrading NBU 7.5 to the latest patch level will solve this issue?
Thanks & Regards,
Shashi Kanth.
04-08-2014 04:03 AM
Please check if NetBackup is operational by manually startup any ode in the cluster, as below.
04-08-2014 04:15 AM
I started netbackup manually my making the IP and mount/dg/volume resources online and freezing the service group.
After that i see nbu_server resource is still OFFLINE, but the NB processes are still running.
# /usr/openv/netbackup/bin/goodies/netbackup start
NetBackup Authentication daemon started.
NetBackup network daemon started.
NetBackup client daemon started.
NetBackup SAN Client Fibre Transport daemon started.
NetBackup will not run without /usr/openv/db/bin/NB_dbsrv running
NetBackup Authorization daemon started.
NetBackup Event Manager started.
NetBackup Audit Manager started.
NetBackup Deduplication Manager started.
NetBackup Deduplication Engine started.
NetBackup Enterprise Media Manager started.
NetBackup Resource Broker started.
Media Manager daemons started.
NetBackup request daemon started.
NetBackup compatibility daemon started.
NetBackup Job Manager started.
NetBackup Policy Execution Manager started.
NetBackup Storage Lifecycle Manager started.
NetBackup Indexing Manager started.
NetBackup Remote Monitoring Management System started.
NetBackup Key Management daemon started.
NetBackup Service Layer started.
NetBackup Agent Request Server started.
NetBackup Bare Metal Restore daemon not started.
NetBackup Vault daemon started.
NetBackup CloudStore Service Container started.
NetBackup Service Monitor started.
NetBackup Bare Metal Restore Boot Server daemon started.
# /opt/VRTS/bin/hares -state nbu_server
#Resource Attribute System Value
nbu_server State node1 OFFLINE
nbu_server State node2 OFFLINE
# /usr/openv/netbackup/bin/bpps
root 28356 1 0 16:38:03 ? 0:00 /usr/openv/netbackup/bin/nbcssc -a NetBackup
root 20327 1 0 11:22:58 ? 0:00 /usr/openv/netbackup/bin/bpcd -standalone
root 28288 1 0 16:37:46 ? 0:00 /usr/openv/netbackup/bin/bpcompatd
root 20323 1 0 11:22:58 ? 0:00 /usr/openv/netbackup/bin/vnetd -standalone
04-08-2014 04:21 AM
I have started NetBackup manually by making IP and dg/volume/mount services online and freezing the nbu_group. After that i see the nbu_server resource group is offline but the netbackup services are still running.
# /usr/openv/netbackup/bin/goodies/netbackup start
NetBackup Authentication daemon started.
NetBackup network daemon started.
NetBackup client daemon started.
NetBackup SAN Client Fibre Transport daemon started.
NetBackup will not run without /usr/openv/db/bin/NB_dbsrv running
NetBackup Authorization daemon started.
NetBackup Event Manager started.
NetBackup Audit Manager started.
NetBackup Deduplication Manager started.
NetBackup Deduplication Engine started.
NetBackup Enterprise Media Manager started.
NetBackup Resource Broker started.
Media Manager daemons started.
NetBackup request daemon started.
NetBackup compatibility daemon started.
NetBackup Job Manager started.
NetBackup Policy Execution Manager started.
NetBackup Storage Lifecycle Manager started.
NetBackup Indexing Manager started.
NetBackup Remote Monitoring Management System started.
NetBackup Key Management daemon started.
NetBackup Service Layer started.
NetBackup Agent Request Server started.
NetBackup Bare Metal Restore daemon not started.
NetBackup Vault daemon started.
NetBackup CloudStore Service Container started.
NetBackup Service Monitor started.
NetBackup Bare Metal Restore Boot Server daemon started.
# /opt/VRTS/bin/hares -state nbu_server
#Resource Attribute System Value
nbu_server State hyi01sehost85 OFFLINE
nbu_server State hyi01sehost87 OFFLINE
# /usr/openv/netbackup/bin/bpps
root 28356 1 0 16:38:03 ? 0:00 /usr/openv/netbackup/bin/nbcssc -a NetBackup
root 20327 1 0 11:22:58 ? 0:00 /usr/openv/netbackup/bin/bpcd -standalone
root 28288 1 0 16:37:46 ? 0:00 /usr/openv/netbackup/bin/bpcompatd
root 20323 1 0 11:22:58 ? 0:00 /usr/openv/netbackup/bin/vnetd -standalone
04-08-2014 04:21 AM
Is this a newly installed cluster that has never worked before?
Or has this worked before and is now failing?
Have you checked disk space on NBU catalog volume?
NBU will shutdown emm if catalog disk gets to 99% full.
If this is a production server that has worked fine previously, do not attempt an upgrade at this stage.
You need to always perform a catalog backup before an upgrade and will not be able to do so with NBU not running.
To troubleshoot, please perform the following steps:
Please post output of 'netbackup start' as well as 'bpps -x' plus these 2 commands:
nbemmcmd -listhosts -verbose
nbemmcmd -getemmserver
If emm does not remain running, post output of emm log:
vxlogview -o 111 -t 00:10:00
(above command will report on last 10 minutes. Ensure -t value includes enough time to catch the last startup attempt.)
Depending on errors and/or output of bpps, we may need to have a look at the other logs as well.
04-08-2014 05:39 AM
When i start manually i see the bellow message.
"NetBackup will not run without /usr/openv/db/bin/NB_dbsrv running"
04-08-2014 01:35 PM
Apologies for only getting back to you now.
Have you checked disk space on the database volume?
Have you verified virtual hostname lookup with 'bpclntcmd -ip <virt-IP>'? and 'bpclntcmd -self'?
Have you checked nbemmcmd commands as per my post above?
And emm log with vxlogview?
Please also check /usr/openv/db/log/server.log for errors and work through this TN to troubleshoot:
http://www.symantec.com/docs/TECH61635
04-11-2014 08:03 AM
Hopefully you have a good catalog backup. If theres db corruption you might be needing it.
There no sgortage of info about the db(s) as mvdb has pointed to.
Jim
04-11-2014 08:14 AM