The failure of recovering NBU Catalog on DR NBU Master server

[Jeffn3u]

Hi there,

I'm getting the following error message when try to recover the DR NBU Master server with using the production NBU master server's online catalog backup.

13:40:10 (17.001) /usr/openv/volmgr/database/
13:40:10 (17.001) Directory /usr/openv/volmgr/database already exists.
13:40:10 (17.001) /usr/openv/netbackup/bp.conf
13:40:10 (17.001) /usr/openv/volmgr/vm.conf
13:40:10 (17.001) /usr/openv/netbackup/vault/sessions
13:40:10 (17.001) Cannot rmdir the directory /usr/openv/netbackup/vault/sessions. Errno = 17: File exists
13:40:10 (17.001) Could not create symlink /usr/openv/netbackup/vault/sessions -> /opt/VRTSnbu/vault/sessions. Errno = 17: File exists
13:40:10 (17.001) /opt/VRTSnbu/vault/sessions/

Although I've try remove the symbolic link and added full permission priviledge but somehow it still couldn't recovery the NBU catalog.

Thanks in advanced.

[Marianne]

Is this a restore using bprecover or a normal filesystem restore?
Are you restoring a clustered master server to non-clustered?

[Jeffn3u]

Hi Marianne,

I managed to recover the DR NBU6.5.5 master server which is running on clustered.

Although the recovering process is successfully shows completed, but unfornately after stop/start netbackup services will not come up as normal.

When try to ping the NBDB, there is an error message of communicating with emm database.

Please help....

[Marianne]

Please give more info about your recovery procedure - Did you run cluster_config after NBU installation? Have you installed the same NBU options/agents (e.g vault) as production master? Have you installed same patches as production master?   Does your DR master have the same hostname/virtual name as production?

Full or partial recovery? Are you using berecover -wizard?

Have you followed the recovery steps as detailed in the Troubleshooting Guide?

It's difficult to even try and provide advice without all the details.

[Jeffn3u]

These are steps & recovery procedure that I've done so far,

1) Fresh installation of DR NBU Master server with maintenance pack 6.5.5,  then ran cluster_config.

root>/var/adm/syslog# cmviewcl

CLUSTER      STATUS      
NBUCL        up          

  NODE         STATUS       STATE       
  NBMSSR02     up           running     
  NBMSSR03     up           running     

    PACKAGE      STATUS       STATE        AUTO_RUN     NODE        
    netbackup    up           running      disabled     NBMSSR03  

2) Configured robotic & device media then perform inventory

3) start catalog recovery wizard gui, using DR_file to perform full online catalog recovery.  

4) Everything is completed without error, then run #nbpemreq -resume_schedule

5) Browser backup id/image, everything is in tagged.

6) Closed NB Java gui, stop/start netbackup services. ==> Netbackup services/daemon NB_svr proxy services will not started.

7) Ping NBDB, it show error communicating with emm database.

Thanks

[Marianne]

Did you install & patch Vault option before catalog recovery?

I have seen emm startup failures when options such as Vault and BMR (present at production site) was not installed.

[Jeffn3u]

After fresh installed of DR NBU master server with all related MP, but NB_dbsvr services is still could not be started.

Do you think is related to licenses issue?

root>/usr/openv/db/bin# ./nbdb_ping NBMSSR01
Database [NBDB] is not available.
root>/usr/openv/db/bin# cat /usr/openv/pack/pack.summary
# DO NOT EDIT THIS FILE !
# * means installed patch was preceded by this patch.
# + means that the installed patch installed this patch as a dependency.
NB_CLT_6.5.3 installed. +NB_6.5.3
NB_6.5.3 installed. *NB_CLT_6.5.3
NB_JAV_6.5.3 installed.
NB_VLT_6.5.3 installed.
NB_SAP_6.5.3 installed. *NB_CLT_6.5.3
NB_CLT_6.5.3.1 installed. *NB_CLT_6.5.3 +NB_6.5.3.1 +NB_JAV_6.5.3.1
NB_6.5.3.1 installed. *NB_6.5.3 *NB_CLT_6.5.3.1
NB_JAV_6.5.3.1 installed. *NB_JAV_6.5.3 *NB_CLT_6.5.3.1
NB_VLT_6.5.5 installed. *NB_VLT_6.5.3
NB_CLT_6.5.5 installed. *NB_CLT_6.5.3.1 +NB_6.5.5 +NB_JAV_6.5.5
NB_6.5.5 installed. *NB_6.5.3.1 *NB_CLT_6.5.5
NB_JAV_6.5.5 installed. *NB_JAV_6.5.3.1 *NB_CLT_6.5.5
root>/usr/openv/db/bin# bpps -x
NB Processes
------------
    root 20542     1  0 01:30:48 ?         0:00 /usr/openv/netbackup/bin/nbstserv
    root 20537 20536  0 01:30:46 ?         0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root 20590     1  0 01:30:52 ?         0:00 /usr/openv/netbackup/bin/nbsl
    root 20522 20521  0 01:30:44 ?         0:00 /usr/openv/netbackup/bin/bpjobd
    root 20608     1  0 01:31:05 ?         0:01 /usr/openv/netbackup/bin/admincmd/bpstsinfo -UPDATE
    root 20524     1  0 01:30:44 ?         0:00 /usr/openv/netbackup/bin/bpcompatd
    root 20492     1  0 01:30:40 ?         0:00 /usr/openv/netbackup/bin/nbevtmgr
    root 20536 20533  0 01:30:46 ?         0:00 sh -c "/usr/openv/netbackup/bin/nbproxy" dblib nbpem
    root 20511     1  0 01:30:43 ?         0:00 /usr/openv/netbackup/bin/bprd
    root 20533     1  0 01:30:46 ?         0:00 /usr/openv/netbackup/bin/nbpem
    root 20547     1  0 01:30:49 ?         0:00 /usr/openv/netbackup/bin/nbrmms
    root 20529     1  0 01:30:45 ?         0:00 /usr/openv/netbackup/bin/nbjm
    root 20598     1  0 01:30:53 ?         0:00 /usr/openv/netbackup/bin/nbvault
    root 20521     1  0 01:30:44 ?         0:00 /usr/openv/netbackup/bin/bpdbm

MM Processes
------------
    root 20535     1  0 01:30:46 ?         0:00 vmd
    root 20504     1  0 01:30:43 ?         0:00 /usr/openv/volmgr/bin/ltid

Shared Symantec Processes
-------------------------
    root   965     1  0  Sep 10  ?         0:05 /opt/VRTSpbx/bin/pbx_exchange

root>/usr/openv/var/global# netbackup stop
stopping the NetBackup Vault daemon
stopping the NetBackup Service Layer
stopping the NetBackup Remote Monitoring Management System
stopping the NetBackup Storage Service Manager
stopping the NetBackup Policy Execution Manager
stopping the NetBackup Job Manager
stopping nbproxy...
stopping the NetBackup request daemon
stopping the NetBackup compatibility daemon
stopping the NetBackup database daemon
stopping the Media Manager device daemon
stopping the Media Manager volume daemon
stopping the NetBackup Event Manager
root>/usr/openv/var/global# netbackup start
NetBackup will not run without /usr/openv/db/bin/NB_dbsrv running
NetBackup Event Manager started.
NetBackup Enterprise Media Manager started.
NetBackup Resource Broker started.
Media Manager daemons started.
NetBackup request daemon started.
NetBackup compatibility daemon started.
NetBackup Job Manager started.
NetBackup Policy Execution Manager started.
NetBackup Storage Service Manager started.
NetBackup Remote Monitoring Management System started.
NetBackup Key Management daemon started.
NetBackup Service Layer started.
NetBackup Vault daemon started.
HPSG
NetBackup Service Monitor started.
root>/usr/openv/var/global#

[Marianne]

It's difficult to guess at this stage... The pack summary file shows that all relevant patches were installed. I have experienced a similar issue where all the patches definately showed successful install, but for some or reason the 'version_vault' file in /usr/openv/share was not updated.

After a re-install of the vault patch (with -p to force install), the version file showed correct info and emm started successfully:

# cat version_vault
NetBackup-VAULT-SOLARIS 6.5.5
 

I have also experienced a situation where /usr/openv/var/global/server.conf for some or other reason had a filesize of zero... A Support engineer picked this up during a Webex session.

Best if you stop all daemons - including pbx, then restart pbx followed by NBU. If emm fails again, check unified logs for emm for last 5 minutes:

(example)

vxlogview -o 111 -a -t 00:05:00

You might want to send output to a file for easier viewing. Hopefully the output will shed light on your particular problem. You might have to increase logging levels if not enough info is displayed in the logs.

In all honesty - there are so many possible reasons for emm to die after startup that you might be better off logging a support call.

[Jeffn3u · Accepted solution]

Hi Marianne,

Finally, I managed to resolve the problem by reducing the parameter buffer sizes on /opt/VRTSnbu/var/global/server.conf file.

The problem occur due to physical memory sizing is different between prod & DR NBU master server.

Thanks for your support.

Regards,
Jeff

[Marianne]

Thanks for the feedback! I have marked your post as solution.