Netbackup 7.0 installation and upgrade issues

Has any solution been found to this because I am running into the same issue and after messing around it looks like it is something with the nbdbms_start_stop command but I can not figure out what.

According to nbdb log on my test Windows host, pingDBServer returning code 12 is valid.
Can you figure out where the stack occurs,  with pstree, truss or so?

Well when I try and run nbdbms_start_stop manually it hangs forever so I am pretty sure the issue is with nbdbms_start_stop but I can not find any log information about what the problem is. I'm fairly new to Solaris so if anyone knows where I can look or log information that would help I can look into it

EDIT:

And I did check the version of solaris I'm using and it is 64bit with the 10/09 update.

nbdbm_start_stop is shell script, and you can figure out more detail point of stack using ptree(sorry , Solaris doesn't have pstree).

# file /usr/openv/netbackup/bin/nbdbms_start_stop
/usr/openv/netbackup/bin/nbdbms_start_stop:     executable shell script

ex)
# ptree
   :
1331  /usr/bin/gnome-terminal
  1344  gnome-pty-helper
  1346  sh
    1404  bash
      23373 /bin/sh /usr/openv/netbackup/bin/nbdbms_start_stop start
        23380 /bin/sh /usr/openv/db/bin/nbdbms_start_server
          23404 /usr/openv/db//bin/dbspawn -f /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/globa

Running shell script with -x option is also useful to track processes.

# sh -x /usr/openv/netbackup/bin/nbdbms_start_stop start
PATH=/usr/sbin:/usr/bin:/usr/openwin/bin:/usr/ucb:/usr/openv/netbackup/bin:/usr/openv/netbackup/bin/admincmd:/usr/openv/netbackup/bin/goodies:/usr/openv/volmgr/bin:/usr/bin:/sbin:/usr/sbin:/bin
NB_DIR=/usr/openv/netbackup
VXDBMS_BIN=/usr/openv/db/bin
VXDBMS_HOME=/usr/openv/db
+ . /usr/openv/db/vxdbms_env.sh
VXDBMS_HOME=/usr/openv/db/
SQLANY=/usr/openv/db/
SQLANY11=/usr/openv/db/
SQLANYSH11=/usr/openv/db//shared
SATMP=/usr/openv/tmp/sqlany
SATMP_UMASK=077
SADIAGDIR=/usr/openv/db//log
SALOGDIR=/usr/openv/db//log
+ export VXDBMS_HOME SQLANY SQLANY11 SQLANYSH11 SATMP SATMP_UMASK SADIAGDIR SALOGDIR
+ echo /usr/sbin:/usr/bin:/usr/openwin/bin:/usr/ucb:/usr/openv/netbackup/bin:/usr/openv/netbackup/bin/admincmd:/usr/openv/netbackup/bin/goodies:/usr/openv/volmgr/bin:/usr/bin:/sbin:/usr/sbin:/bin
+ grep /usr/openv/db//bin
+ [ 1 != 0 ]
PATH=/usr/sbin:/usr/bin:/usr/openwin/bin:/usr/ucb:/usr/openv/netbackup/bin:/usr/openv/netbackup/bin/admincmd:/usr/openv/netbackup/bin/goodies:/usr/openv/volmgr/bin:/usr/bin:/sbin:/usr/sbin:/bin:/usr/openv/db//bin
+ export PATH
+ [ ]
LD_LIBRARY_PATH=/usr/openv/db//lib:/usr/openv/db//bin
+ export LD_LIBRARY_PATH
+ tr -d  \011
+ grep ^EMMSERVER=
+ awk -F= {print $2}
emmserver=nbumastersol
+ repeated_try_on_SuSE10 nbumastersol
+ [ -f /etc/SuSE-release ]
+ return
+ /usr/openv/netbackup/bin/bpclntcmd -is_local_host nbumastersol
+ [ 0 -eq 0 ]
+ /usr/openv/db/bin/nbdbms_start_server
+ exit 0

nbdbms_start_server that is called from nbdbadmin_start_stop is shell script, too.

bash-3.00# sh -x /usr/openv/db/bin/nbdbms_start_server
+ uname -s
ECHO=/usr/bin/echo
PATH=/usr/sbin:/usr/bin:/usr/openwin/bin:/usr/ucb:/usr/openv/netbackup/bin:/usr/openv/netbackup/bin/admincmd:/usr/openv/netbackup/bin/goodies:/usr/openv/volmgr/bin:/usr/bin:/sbin:/usr/sbin:/bin
+ . /usr/openv/db/vxdbms_env.sh
VXDBMS_HOME=/usr/openv/db/
SQLANY=/usr/openv/db/
SQLANY11=/usr/openv/db/
SQLANYSH11=/usr/openv/db//shared
SATMP=/usr/openv/tmp/sqlany
SATMP_UMASK=077
SADIAGDIR=/usr/openv/db//log
SALOGDIR=/usr/openv/db//log
+ export VXDBMS_HOME SQLANY SQLANY11 SQLANYSH11 SATMP SATMP_UMASK SADIAGDIR SALOGDIR
+ echo /usr/sbin:/usr/bin:/usr/openwin/bin:/usr/ucb:/usr/openv/netbackup/bin:/usr/openv/netbackup/bin/admincmd:/usr/openv/netbackup/bin/goodies:/usr/openv/volmgr/bin:/usr/bin:/sbin:/usr/sbin:/bin
+ grep /usr/openv/db//bin
+ [ 1 != 0 ]
PATH=/usr/sbin:/usr/bin:/usr/openwin/bin:/usr/ucb:/usr/openv/netbackup/bin:/usr/openv/netbackup/bin/admincmd:/usr/openv/netbackup/bin/goodies:/usr/openv/volmgr/bin:/usr/bin:/sbin:/usr/sbin:/bin:/usr/openv/db//bin
+ export PATH
+ [ ]
LD_LIBRARY_PATH=/usr/openv/db//lib:/usr/openv/db//bin
+ export LD_LIBRARY_PATH
NBDBMS_CONF_DIR=/usr/openv/var/global
+ [ -f /usr/openv/db//bin/dbsrv9 ]
SATMP=/usr/openv/tmp/sqlany
+ export SATMP
SATMP_UMASK=077
+ export SATMP_UMASK
SALOGDIR=/usr/openv/db//log
+ export SALOGDIR
SADIAGDIR=/usr/openv/db//log
+ export SADIAGDIR
+ cat /usr/openv/db//bin/servername
server_name=NB_nbumastersol
+ [ -f /usr/openv/db//bin/dbsrv9 ]
temp_out_file=/usr/openv/tmp/sqlany/nbdbms_start_server.out
+ rm -f /usr/openv/tmp/sqlany/nbdbms_start_server.out
START=0
STOP=0
STAT=0
FORCE=0
+ cd /usr/openv/db/
+ [ 0 = 0 ]
START=1
+ [ 1 = 1 ]
databases_conf=/usr/openv/var/global/databases.conf
+ [ ! -f /usr/openv/var/global/server.conf ]
+ [ ! -f /usr/openv/var/global/databases.conf ]
command_args="/usr/openv/db//bin/NB_dbsrv" @"/usr/openv/var/global/server.conf"
+ rm -f /usr/openv/var/global/databases.conf.new
+ touch /usr/openv/var/global/databases.conf.new
+ read Line
+ cat /usr/openv/var/global/databases.conf
+ sed s/"//g
+ awk {print $1}
+ echo "/usr/openv/db/data/NBDB.db" -n NBDB
filename=/usr/openv/db/data/NBDB.db
+ [ -s /usr/openv/db/data/NBDB.db ]
+ echo "/usr/openv/db/data/NBDB.db" -n NBDB
+ read Line
+ sed s/"//g
+ awk {print $1}
+ echo "/usr/openv/db/data/BMRDB.db" -n BMRDB
filename=/usr/openv/db/data/BMRDB.db
+ [ -s /usr/openv/db/data/BMRDB.db ]
+ echo "/usr/openv/db/data/BMRDB.db" -n BMRDB
+ read Line
+ mv /usr/openv/var/global/databases.conf.new /usr/openv/var/global/databases.conf
+ get_ServerPort
+ awk {
        for ( field = 1; field <= NF; field++ ) {
                if ( $field ~ /ServerPort/ ) {
                        Elements=split($field, Element, ";")
                        for ( portfield = 1; portfield <= Elements; portfield++ ) {
                                if ( Element[portfield]~ /ServerPort/ ) {
                                        split(Element[portfield], port, "=")
                                        print port[2]
                                }
                        }
                }
        }
} /usr/openv/var/global/server.conf
port=13785)
+ echo 13785)
+ awk -F) {print $1}
port=13785
commlinks=commlinks=shmem,tcpip(host=localhost;port=13785;DoBroadcast=NONE)
+ /usr/openv/db//bin/dbping -q -c ENG=NB_nbumastersol;commlinks=shmem,tcpip(host=localhost;port=13785;DoBroadcast=NONE)
+ [ 1 = 0 ]
command_args="/usr/openv/db//bin/NB_dbsrv" @"/usr/openv/var/global/server.conf" @"/usr/openv/var/global/databases.conf"
+ /usr/bin/echo /usr/openv/db//bin/dbspawn -f "/usr/openv/db//bin/NB_dbsrv" @"/usr/openv/var/global/server.conf" @"/usr/openv/var/global/databases.conf"
+ eval /usr/openv/db//bin/dbspawn -f "/usr/openv/db//bin/NB_dbsrv" @"/usr/openv/var/global/server.conf" @"/usr/openv/var/global/databases.conf"
exit_code=0
+ [ 0 != 0 ]
+ exit 0
bash-3.00#

Hi Folks,

I was able to get over this hump by recreating the test machine (VM). I decided to install Soalris 10 update 8 (X86 - 64bit). Both the fresh install and test upgrade from 6.5.4 went fine without any problems.

Unfortunately, right now, I'm doing the actual upgrade on the production machine and it froze at the part when its starting the daemons after the upgrade :( I have a pro-active ticket with Symantec. Calling them now.....

*********************************************************************************************************************************
Populating the database tables.  This will take some time.

Starting the Media Manager device daemon processes.

Do you want to start the NetBackup bprd process so
backups and restores can be initiated? [y,n] (y)
Starting the NetBackup request daemon process (bprd).
Starting the NetBackup Job Manager.
Starting the NetBackup Policy Execution Manager.
Starting the NetBackup Storage Lifecycle Manager.
Starting the NetBackup Remote Monitoring Management System.
Starting the NetBackup Key Management daemon.
Starting the NetBackup Service Layer.
Starting the NetBackup Agent Request Server.
Starting the NetBackup Bare Metal Restore daemon.
Starting the NetBackup Vault daemon.
Starting the NetBackup Service Monitor.
Starting the NetBackup Bare Metal Restore Boot Server daemon.

*********************************************************************************************************************************

I spoke too soon..... Right after posting my reply earlier, the installation script continued running :) Now on to the equally crucial post upgrade verification and tests....

Thanks for sharing your experience!
Is the time taken to startup in relation to the installation size (i.e image size and/or number of items in EMM database, such as media-id, devices, stu's, etc)?
Let us know how the testing went.

Hi again,

Just a recap. I upgraded our netbackup 6.5.4 running on Solaris 10 u4 (SPARK) to NBU 7.0. The installation was doing fine until it got to the last part. It basically felt it just frozed when it's starting the services/daemons (see below). If I'm not mistaken, it didn't move for a good 10 mins or so.  During my test upgrade, this part was instantaneous which is why i suspected it hanged. But then again, my test nbu didn't have much in the catalog, volume pool, policies, storage unit, etc.But eventually, the installation went on and completed. Btw, I haven't tried stopping/starting (/usr/openv/netbackup/bin/goodies/netbackup stop|start) to justify the delay/frozen-like thingy I had during the upgrade (starting of the daemons).



During my post-activity check, all the drives we have (NDMP and NDMP) were functional. Unlike when I upgraded from 5.1 to 6.5.4 where I had some drive problems after the upgrade.... But this time around things are holding up. I've even rebooted the server and the netbackup services came up without any issues. Also, test backups and restores were fine. The real tests would be later when our daily incremental jobs execute and the same goes for the coming weekend - our weekly full.



****************************************************************************************************
Do you want to start the NetBackup bprd process so
backups and restores can be initiated? [y,n] (y)

Starting the Media Manager device daemon processes.

Do you want to start the NetBackup bprd process so
backups and restores can be initiated? [y,n] (y)
Starting the NetBackup request daemon process (bprd).
Starting the NetBackup Job Manager.
Starting the NetBackup Policy Execution Manager.
Starting the NetBackup Storage Lifecycle Manager.
Starting the NetBackup Remote Monitoring Management System.
Starting the NetBackup Key Management daemon.
Starting the NetBackup Service Layer.
Starting the NetBackup Agent Request Server.
Starting the NetBackup Bare Metal Restore daemon.
Starting the NetBackup Vault daemon.
Starting the NetBackup Service Monitor.
Starting the NetBackup Bare Metal Restore Boot Server daemon. ---> freezing-like moment

processing image candidates for storage unit .........
.
.
.

OpsCenter is the next-generation monitoring, reporting and
administrative solution designed to centrally manage one or
more NetBackup installations from a web browser.  Existing
NetBackup Operations Manager or Veritas Backup Reporter
installations can be upgraded to OpsCenter.

If an OpsCenter server already exists in your environment
or you plan to install one, enter the real hostname of that
OpsCenter server here.  Do not use a virtual name.  If you
do not want this local machine to be an OpsCenter server,
enter NONE.

Enter the OpsCenter server (default: NONE):


NetBackup server installation complete.

************************************************************************************

Thanks.

I wish I could say that the install started working for me as well but even with multiple OS and NBU reinstallation attempts I was never able to get it working on Solaris in a VM

Hi bcostell,

If it helps, the test VM (with vmtools) I used, is running under Vsphere 4.0. On my initial post, that Solaris VM where I encountered the problem was on ESXi 3.5 and another thing with it, when the VM was initially created, Solaris 10 32bit was selected on the VM side but the actual Solaris OS was a 64 bit one. In other words, there was a slight oversight during the VM creation. Although, I am not 100% sure if its the root cause. Thanks.