VCS 5.1 NBU 7.5 Solaris 10 SPARC

[shashi]

VCS 5.1 NBU 7.5 Solaris 10 SPARC.

Hi,

I am getting bellow error during the NBU resource group online and it is not starting in all cluster nodes. I did tried exporting environment variables “NB_BPDBM_ALIVE_LOCAL_TIMEOUT=59” and even tried increasing the OnlineRetryLimit=2 in NetBackup types file, but still the resource group is not coming up and it is faulted in all cluster nodes.

“VCS ERROR V-16-2-13066 Agent is calling clean for resource(nbu_server) because the resource is not up even after online completed.”

Anyone suggest me how to solve this issue? Upgrading NBU 7.5 to the latest patch level will solve this issue?

Thanks & Regards,

Shashi Kanth.

[Yasuhisa_Ishika]

Please check if NetBackup is operational by manually startup any ode in the cluster, as below.

1. Online all the resources but nbu_server resource in the service group.
2. Freeze the service group
3. Startup NetBackup manually by "/usr/openv/netbackup/bin/goodies/netbackup start"
4. Check if nbu_server resource become online. If not, check the NB processes running(or not running) on the node.

[shashi]

I started netbackup manually my making the IP and mount/dg/volume resources online and freezing the service group.

After that i see nbu_server resource is still OFFLINE, but the NB processes are still running.

# /usr/openv/netbackup/bin/goodies/netbackup start
NetBackup Authentication daemon started.
NetBackup network daemon started.
NetBackup client daemon started.
NetBackup SAN Client Fibre Transport daemon started.
NetBackup will not run without /usr/openv/db/bin/NB_dbsrv running
NetBackup Authorization daemon started.
NetBackup Event Manager started.
NetBackup Audit Manager started.
NetBackup Deduplication Manager started.
NetBackup Deduplication Engine started.
NetBackup Enterprise Media Manager started.
NetBackup Resource Broker started.
Media Manager daemons started.
NetBackup request daemon started.
NetBackup compatibility daemon started.
NetBackup Job Manager started.
NetBackup Policy Execution Manager started.
NetBackup Storage Lifecycle Manager started.
NetBackup Indexing Manager started.
NetBackup Remote Monitoring Management System started.
NetBackup Key Management daemon started.
NetBackup Service Layer started.
NetBackup Agent Request Server started.
NetBackup Bare Metal Restore daemon not started.
NetBackup Vault daemon started.
NetBackup CloudStore Service Container started.
NetBackup Service Monitor started.
NetBackup Bare Metal Restore Boot Server daemon started.

# /opt/VRTS/bin/hares -state nbu_server
#Resource    Attribute             System        Value
nbu_server   State                 node1             OFFLINE
nbu_server   State                 node2             OFFLINE

# /usr/openv/netbackup/bin/bpps   
    root 28356     1   0 16:38:03 ?           0:00 /usr/openv/netbackup/bin/nbcssc -a NetBackup
    root 20327     1   0 11:22:58 ?           0:00 /usr/openv/netbackup/bin/bpcd -standalone
    root 28288     1   0 16:37:46 ?           0:00 /usr/openv/netbackup/bin/bpcompatd
    root 20323     1   0 11:22:58 ?           0:00 /usr/openv/netbackup/bin/vnetd -standalone

[shashi]

I have started NetBackup manually by making IP and dg/volume/mount services online and freezing the nbu_group. After that i see the nbu_server resource group is offline but the netbackup services are still running.

# /usr/openv/netbackup/bin/goodies/netbackup start
NetBackup Authentication daemon started.
NetBackup network daemon started.
NetBackup client daemon started.
NetBackup SAN Client Fibre Transport daemon started.
NetBackup will not run without /usr/openv/db/bin/NB_dbsrv running
NetBackup Authorization daemon started.
NetBackup Event Manager started.
NetBackup Audit Manager started.
NetBackup Deduplication Manager started.
NetBackup Deduplication Engine started.
NetBackup Enterprise Media Manager started.
NetBackup Resource Broker started.
Media Manager daemons started.
NetBackup request daemon started.
NetBackup compatibility daemon started.
NetBackup Job Manager started.
NetBackup Policy Execution Manager started.
NetBackup Storage Lifecycle Manager started.
NetBackup Indexing Manager started.
NetBackup Remote Monitoring Management System started.
NetBackup Key Management daemon started.
NetBackup Service Layer started.
NetBackup Agent Request Server started.
NetBackup Bare Metal Restore daemon not started.
NetBackup Vault daemon started.
NetBackup CloudStore Service Container started.
NetBackup Service Monitor started.
NetBackup Bare Metal Restore Boot Server daemon started.

# /opt/VRTS/bin/hares -state nbu_server
#Resource    Attribute             System        Value
nbu_server   State                 hyi01sehost85 OFFLINE
nbu_server   State                 hyi01sehost87 OFFLINE

# /usr/openv/netbackup/bin/bpps   
    root 28356     1   0 16:38:03 ?           0:00 /usr/openv/netbackup/bin/nbcssc -a NetBackup
    root 20327     1   0 11:22:58 ?           0:00 /usr/openv/netbackup/bin/bpcd -standalone
    root 28288     1   0 16:37:46 ?           0:00 /usr/openv/netbackup/bin/bpcompatd
    root 20323     1   0 11:22:58 ?           0:00 /usr/openv/netbackup/bin/vnetd -standalone

[Marianne]

Is this a newly installed cluster that has never worked before?
Or has this worked before and is now failing?
Have you checked disk space on NBU catalog volume? 
NBU will shutdown emm if catalog disk gets to 99% full.

If this is a production server that has worked fine previously, do not attempt an upgrade at this stage.
You need to always perform a catalog backup before an upgrade and will not be able to do so with NBU not running.

To troubleshoot, please perform the following steps:

• Use bpps -x to check for remaining NBU processes. If any, stop them with 'netbackup stop'.
  Leave pbx running.
• Clear the faulted resouces on both nodes.
• Online all resources on one node except for the NetBackup resource (nbu_server)
  (In other words: diskgroup, volume(s), mount point(s) and IP)
• Check that virtual IP resolves to master virtual hostname:
  bpclntcmd -ip <virtual-IP>
• Create log folders (if they don't exist already) on this node under /usr/openv/netbackup/logs:
  bpdbm bprd admin
• Freeze the SG
• Manually start NBU from command line:
  netbackup start
• Carefully check for errors 
• Check processes from another window with 'bpps -x'

Please post output of 'netbackup start' as well as 'bpps -x' plus these 2 commands:
nbemmcmd -listhosts -verbose
nbemmcmd -getemmserver

If emm does not remain running, post output of emm log:
vxlogview -o 111 -t 00:10:00
(above command will report on last 10 minutes. Ensure -t value includes enough time to catch the last startup attempt.)

Depending on errors and/or output of bpps, we may need to have a look at the other logs as well.

[shashi]

When i start manually i see the bellow message.

"NetBackup will not run without /usr/openv/db/bin/NB_dbsrv running"

[Marianne]

Apologies for only getting back to you now.

Have you checked disk space on the database volume?

Have you verified virtual hostname lookup with 'bpclntcmd -ip <virt-IP>'? and 'bpclntcmd -self'?

Have you checked nbemmcmd commands as per my post above? 

And emm log with vxlogview?

Please also check /usr/openv/db/log/server.log for errors and work through this TN to troubleshoot:
http://www.symantec.com/docs/TECH61635 

[jim_dalton]

Hopefully you have a good catalog backup. If theres db corruption you might be needing it.

There no sgortage of info about the db(s) as mvdb has pointed to.

Jim

[Marianne]

No reply in 2 days....

Hopefully this is sorted by now!!!