Forum Discussion

Jeffn3u's avatar
Jeffn3u
Level 3
15 years ago
Solved

The failure of recovering NBU Catalog on DR NBU Master server

Hi there,

I'm getting the following error message when try to recover the DR NBU Master server with using the production NBU master server's online catalog backup.

13:40:10 (17.001) /usr/openv/volmgr/database/
13:40:10 (17.001) Directory /usr/openv/volmgr/database already exists.
13:40:10 (17.001) /usr/openv/netbackup/bp.conf
13:40:10 (17.001) /usr/openv/volmgr/vm.conf
13:40:10 (17.001) /usr/openv/netbackup/vault/sessions
13:40:10 (17.001) Cannot rmdir the directory /usr/openv/netbackup/vault/sessions. Errno = 17: File exists
13:40:10 (17.001) Could not create symlink /usr/openv/netbackup/vault/sessions -> /opt/VRTSnbu/vault/sessions. Errno = 17: File exists
13:40:10 (17.001) /opt/VRTSnbu/vault/sessions/

Although I've try remove the symbolic link and added full permission priviledge but somehow it still couldn't recovery the NBU catalog.

Thanks in advanced.
  • Hi Marianne,

    Finally, I managed to resolve the problem by reducing the parameter buffer sizes on /opt/VRTSnbu/var/global/server.conf file.

    The problem occur due to physical memory sizing is different between prod & DR NBU master server.

    Thanks for your support.

    Regards,
    Jeff

9 Replies


  • Is this a restore using bprecover or a normal filesystem restore?
    Are you restoring a clustered master server to non-clustered?
  • Hi Marianne,

    I managed to recover the DR NBU6.5.5 master server which is running on clustered.

    Although the recovering process is successfully shows completed, but unfornately after stop/start netbackup services will not come up as normal.

    When try to ping the NBDB, there is an error message of communicating with emm database.

    Please help....

  • Please give more info about your recovery procedure - Did you run cluster_config after NBU installation? Have you installed the same NBU options/agents (e.g vault) as production master? Have you installed same patches as production master?   Does your DR master have the same hostname/virtual name as production?

    Full or partial recovery? Are you using berecover -wizard?

    Have you followed the recovery steps as detailed in the Troubleshooting Guide?

    It's difficult to even try and provide advice without all the details.

  • These are steps & recovery procedure that I've done so far,

    1) Fresh installation of DR NBU Master server with maintenance pack 6.5.5,  then ran cluster_config.

    root>/var/adm/syslog# cmviewcl

    CLUSTER      STATUS      
    NBUCL        up          
      NODE         STATUS       STATE       
      NBMSSR02     up           running     
      NBMSSR03     up           running     
        PACKAGE      STATUS       STATE        AUTO_RUN     NODE        
        netbackup    up           running      disabled     NBMSSR03  
     
    2) Configured robotic & device media then perform inventory
    3) start catalog recovery wizard gui, using DR_file to perform full online catalog recovery.  
    4) Everything is completed without error, then run #nbpemreq -resume_schedule
    5) Browser backup id/image, everything is in tagged.
    6) Closed NB Java gui, stop/start netbackup services. ==> Netbackup services/daemon NB_svr proxy services will not started.
    7) Ping NBDB, it show error communicating with emm database.
     
    Thanks
  • Did you install & patch Vault option before catalog recovery?

    I have seen emm startup failures when options such as Vault and BMR (present at production site) was not installed.

  • After fresh installed of DR NBU master server with all related MP, but NB_dbsvr services is still could not be started.

    Do you think is related to licenses issue?

    root>/usr/openv/db/bin# ./nbdb_ping NBMSSR01
    Database [NBDB] is not available.
    root>/usr/openv/db/bin# cat /usr/openv/pack/pack.summary
    # DO NOT EDIT THIS FILE !
    # * means installed patch was preceded by this patch.
    # + means that the installed patch installed this patch as a dependency.
    NB_CLT_6.5.3 installed. +NB_6.5.3
    NB_6.5.3 installed. *NB_CLT_6.5.3
    NB_JAV_6.5.3 installed.
    NB_VLT_6.5.3 installed.
    NB_SAP_6.5.3 installed. *NB_CLT_6.5.3
    NB_CLT_6.5.3.1 installed. *NB_CLT_6.5.3 +NB_6.5.3.1 +NB_JAV_6.5.3.1
    NB_6.5.3.1 installed. *NB_6.5.3 *NB_CLT_6.5.3.1
    NB_JAV_6.5.3.1 installed. *NB_JAV_6.5.3 *NB_CLT_6.5.3.1
    NB_VLT_6.5.5 installed. *NB_VLT_6.5.3
    NB_CLT_6.5.5 installed. *NB_CLT_6.5.3.1 +NB_6.5.5 +NB_JAV_6.5.5
    NB_6.5.5 installed. *NB_6.5.3.1 *NB_CLT_6.5.5
    NB_JAV_6.5.5 installed. *NB_JAV_6.5.3.1 *NB_CLT_6.5.5
    root>/usr/openv/db/bin# bpps -x
    NB Processes
    ------------
        root 20542     1  0 01:30:48 ?         0:00 /usr/openv/netbackup/bin/nbstserv
        root 20537 20536  0 01:30:46 ?         0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
        root 20590     1  0 01:30:52 ?         0:00 /usr/openv/netbackup/bin/nbsl
        root 20522 20521  0 01:30:44 ?         0:00 /usr/openv/netbackup/bin/bpjobd
        root 20608     1  0 01:31:05 ?         0:01 /usr/openv/netbackup/bin/admincmd/bpstsinfo -UPDATE
        root 20524     1  0 01:30:44 ?         0:00 /usr/openv/netbackup/bin/bpcompatd
        root 20492     1  0 01:30:40 ?         0:00 /usr/openv/netbackup/bin/nbevtmgr
        root 20536 20533  0 01:30:46 ?         0:00 sh -c "/usr/openv/netbackup/bin/nbproxy" dblib nbpem
        root 20511     1  0 01:30:43 ?         0:00 /usr/openv/netbackup/bin/bprd
        root 20533     1  0 01:30:46 ?         0:00 /usr/openv/netbackup/bin/nbpem
        root 20547     1  0 01:30:49 ?         0:00 /usr/openv/netbackup/bin/nbrmms
        root 20529     1  0 01:30:45 ?         0:00 /usr/openv/netbackup/bin/nbjm
        root 20598     1  0 01:30:53 ?         0:00 /usr/openv/netbackup/bin/nbvault
        root 20521     1  0 01:30:44 ?         0:00 /usr/openv/netbackup/bin/bpdbm


    MM Processes
    ------------
        root 20535     1  0 01:30:46 ?         0:00 vmd
        root 20504     1  0 01:30:43 ?         0:00 /usr/openv/volmgr/bin/ltid

    Shared Symantec Processes
    -------------------------
        root   965     1  0  Sep 10  ?         0:05 /opt/VRTSpbx/bin/pbx_exchange
    root>/usr/openv/var/global# netbackup stop
    stopping the NetBackup Vault daemon
    stopping the NetBackup Service Layer
    stopping the NetBackup Remote Monitoring Management System
    stopping the NetBackup Storage Service Manager
    stopping the NetBackup Policy Execution Manager
    stopping the NetBackup Job Manager
    stopping nbproxy...
    stopping the NetBackup request daemon
    stopping the NetBackup compatibility daemon
    stopping the NetBackup database daemon
    stopping the Media Manager device daemon
    stopping the Media Manager volume daemon
    stopping the NetBackup Event Manager
    root>/usr/openv/var/global# netbackup start
    NetBackup will not run without /usr/openv/db/bin/NB_dbsrv running
    NetBackup Event Manager started.
    NetBackup Enterprise Media Manager started.
    NetBackup Resource Broker started.
    Media Manager daemons started.
    NetBackup request daemon started.
    NetBackup compatibility daemon started.
    NetBackup Job Manager started.
    NetBackup Policy Execution Manager started.
    NetBackup Storage Service Manager started.
    NetBackup Remote Monitoring Management System started.
    NetBackup Key Management daemon started.
    NetBackup Service Layer started.
    NetBackup Vault daemon started.
    HPSG
    NetBackup Service Monitor started.
    root>/usr/openv/var/global#

     
  • It's difficult to guess at this stage... The pack summary file shows that all relevant patches were installed. I have experienced a similar issue where all the patches definately showed successful install, but for some or reason the 'version_vault' file in /usr/openv/share was not updated.

    After a re-install of the vault patch (with -p to force install), the version file showed correct info and emm started successfully:

    # cat version_vault
    NetBackup-VAULT-SOLARIS 6.5.5
     

    I have also experienced a situation where /usr/openv/var/global/server.conf for some or other reason had a filesize of zero... A Support engineer picked this up during a Webex session.

    Best if you stop all daemons - including pbx, then restart pbx followed by NBU. If emm fails again, check unified logs for emm for last 5 minutes:

    (example)

    vxlogview -o 111 -a -t 00:05:00

    You might want to send output to a file for easier viewing. Hopefully the output will shed light on your particular problem. You might have to increase logging levels if not enough info is displayed in the logs.

    In all honesty - there are so many possible reasons for emm to die after startup that you might be better off logging a support call.

     

  • Hi Marianne,

    Finally, I managed to resolve the problem by reducing the parameter buffer sizes on /opt/VRTSnbu/var/global/server.conf file.

    The problem occur due to physical memory sizing is different between prod & DR NBU master server.

    Thanks for your support.

    Regards,
    Jeff