Netbackup 7.0 installation and upgrade issues

[Denz1217]

Hi Guys,

I'm prepping up to upgrade our existing Netbackup 6.5.4 and decided to do a mock upgrade/installation on our test machine but I'm hitting some issue(s). Please see the details below:

• OS: Solaris 10 update6 - 64-bit (on VMware)
bash-3.00# isainfo -kv
64-bit amd64 kernel modules
bash-3.00# cat /etc/release
Solaris 10 10/08 s10x_u6wos_07b X86
Copyright 2008 Sun Microsystems, Inc.  All Rights Reserved.
Use is subject to license terms.
Assembled 27 October 2008

• 4GB of RAM. Already added the recommended message queue  and semaphore values in /etc/system
When running an upgrade or doing a fresh install, it just gets stuck at a certain point. See below:

*************** When upgrading ***************
x scripts/wdata/dept.dat, 111 bytes, 1 tape blocks
x scripts/wdata/employee.dat, 13225 bytes, 26 tape blocks
x scripts/wdata/fin_code.dat, 186 bytes, 1 tape blocks
x scripts/wdata/fin_data.dat, 1974 bytes, 4 tape blocks
x scripts/wdata/product.dat, 668 bytes, 2 tape blocks
x scripts/wdata/sales_o.dat, 39441 bytes, 78 tape blocks
x scripts/wdata/sales_oi.dat, 53753 bytes, 105 tape blocks
x thirdpartylegal, 0 bytes, 0 tape blocks
x thirdpartylegal/SQLAnywhere11ThirdPartyLegal.pdf, 444340 bytes, 868 tape blocks
x thirdpartylegal/SQLAnywhere11ThirdPartyLegal.txt, 283039 bytes, 553 tape blocks
x tix, 0 bytes, 0 tape blocks
x tix/default.tix, 6008 bytes, 12 tape blocks

Installed SQL Anywhere Version 11.0.1.2279
Installation completed successfully
Database server is NB_lasvmbackup01
NetBackup database already exists in /usr/openv/db/data/NBDB.db --> it get stuck here
***********************************************

  
*************** When doing a fresh install ***************
 x scripts/wdata/dept.dat, 111 bytes, 1 tape blocks
x scripts/wdata/employee.dat, 13225 bytes, 26 tape blocks
x scripts/wdata/fin_code.dat, 186 bytes, 1 tape blocks
x scripts/wdata/fin_data.dat, 1974 bytes, 4 tape blocks
x scripts/wdata/product.dat, 668 bytes, 2 tape blocks
x scripts/wdata/sales_o.dat, 39441 bytes, 78 tape blocks
x scripts/wdata/sales_oi.dat, 53753 bytes, 105 tape blocks
x thirdpartylegal, 0 bytes, 0 tape blocks
x thirdpartylegal/SQLAnywhere11ThirdPartyLegal.pdf, 444340 bytes, 868 tape blocks
x thirdpartylegal/SQLAnywhere11ThirdPartyLegal.txt, 283039 bytes, 553 tape blocks
x tix, 0 bytes, 0 tape blocks
x tix/default.tix, 6008 bytes, 12 tape blocks

Installed SQL Anywhere Version 11.0.1.2279
 
 Installation completed successfully
 Database server is NB_lasvmbackup01 -->it get stuck here
 ***********************************************

The nbdb.log file generated on both instances are identical and the pertinent error message says "NBDBsystem: System call failed with status: 256"

15:01:40.082 [23718] <4> create_nbdb: Entering.
15:01:40.082 [23718] <4> create_nbdb: /usr/openv/db/bin/create_nbdb -force -data /usr/openv/db/data -index /usr/openv/db/data -tlog /usr/openv/db/data -staging /usr/openv/db/staging
15:01:40.083 [23718] <4> create_nbdb: SQLANY = /usr/openv/db
15:01:40.083 [23718] <4> create_nbdb: SQLANY11 = /usr/openv/db
15:01:40.083 [23718] <4> create_nbdb: LD_LIBRARY_PATH = /usr/openv/db/lib
15:01:40.083 [23718] <4> create_nbdb: ASTMP = /usr/openv/tmp/sqlany
15:01:40.083 [23718] <4> create_nbdb: SATMP = /usr/openv/tmp/sqlany
15:01:40.113 [23718] <4> pingDBServer: Entering.
15:01:40.114 [23718] <4> pingDBServer: Running . /usr/openv/db/vxdbms_env.sh && "/usr/openv/db/bin/dbping" -q -c "ENG=NB_lasvmbackup01;LINKS=shmem,tcpip{PORT=13785}"
15:01:45.732 [23718] <16> NBDBsystem: System call failed with status: 256
15:01:45.732 [23718] <16> pingDBServer: Database server [NB_lasvmbackup01] not available.
15:01:45.732 [23718] <4> pingDBServer: Exiting. rc = 12
15:01:45.732 [23718] <4> getExistingDBFile: Entering.
15:01:45.733 [23718] <4> getExistingDBFile: Exiting with rc = 0
15:01:45.733 [23718] <4> create_nbdb: NetBackup database already exists in /usr/openv/db/data/NBDB.db
15:01:45.786 [23718] <4> upgradedb: Entering.
15:01:45.786 [23718] <4> UpdateSettingsinServerConf: Entering.
15:01:45.786 [23718] <4> DetermineMemorySize: Entering.
15:01:45.786 [23718] <4> DetermineMemorySize: Total database file size = 52 MB
15:01:45.786 [23718] <4> DetermineMemorySize: Total memory size = 25 MB
15:01:45.786 [23718] <4> DetermineMemorySize: Exiting.
15:01:45.787 [23718] <4> ReadDbConfInfoMemory: Entering.
15:01:45.787 [23718] <4> ReadDbConfInfoMemory: Exiting with rc = 0
15:01:45.787 [23718] <4> UpdateSettingsinServerConf: No changes or recommendations made for server.conf
15:01:45.787 [23718] <4> UpdateSettingsinServerConf: Exiting with rc = 0
15:01:45.788 [23718] <4> rebuildDBifNecessary: Entering.
15:01:45.788 [23718] <4> rebuildDBifNecessary: Database server is down. Attempting to start the database server.
15:01:45.788 [23718] <4> startDBServer: Entering.
15:01:45.788 [23718] <4> startDBServer: Running "/usr/openv/netbackup/bin/nbdbms_start_stop" start

Any idea on what's causing this? 

THanks and regards,
Dennis

[Brian_Costello]

Has any solution been found to this because I am running into the same issue and after messing around it looks like it is something with the nbdbms_start_stop command but I can not figure out what.

[Yasuhisa_Ishika]

According to nbdb log on my test Windows host, pingDBServer returning code 12 is valid.
Can you figure out where the stack occurs,  with pstree, truss or so?

[Brian_Costello]

Well when I try and run nbdbms_start_stop manually it hangs forever so I am pretty sure the issue is with nbdbms_start_stop but I can not find any log information about what the problem is. I'm fairly new to Solaris so if anyone knows where I can look or log information that would help I can look into it

EDIT:

And I did check the version of solaris I'm using and it is 64bit with the 10/09 update.

[Yasuhisa_Ishika]

nbdbm_start_stop is shell script, and you can figure out more detail point of stack using ptree(sorry , Solaris doesn't have pstree).

# file /usr/openv/netbackup/bin/nbdbms_start_stop
/usr/openv/netbackup/bin/nbdbms_start_stop:     executable shell script

ex)
# ptree
   :
1331  /usr/bin/gnome-terminal
  1344  gnome-pty-helper
  1346  sh
    1404  bash
      23373 /bin/sh /usr/openv/netbackup/bin/nbdbms_start_stop start
        23380 /bin/sh /usr/openv/db/bin/nbdbms_start_server
          23404 /usr/openv/db//bin/dbspawn -f /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/globa

Running shell script with -x option is also useful to track processes.

# sh -x /usr/openv/netbackup/bin/nbdbms_start_stop start
PATH=/usr/sbin:/usr/bin:/usr/openwin/bin:/usr/ucb:/usr/openv/netbackup/bin:/usr/openv/netbackup/bin/admincmd:/usr/openv/netbackup/bin/goodies:/usr/openv/volmgr/bin:/usr/bin:/sbin:/usr/sbin:/bin
NB_DIR=/usr/openv/netbackup
VXDBMS_BIN=/usr/openv/db/bin
VXDBMS_HOME=/usr/openv/db
+ . /usr/openv/db/vxdbms_env.sh
VXDBMS_HOME=/usr/openv/db/
SQLANY=/usr/openv/db/
SQLANY11=/usr/openv/db/
SQLANYSH11=/usr/openv/db//shared
SATMP=/usr/openv/tmp/sqlany
SATMP_UMASK=077
SADIAGDIR=/usr/openv/db//log
SALOGDIR=/usr/openv/db//log
+ export VXDBMS_HOME SQLANY SQLANY11 SQLANYSH11 SATMP SATMP_UMASK SADIAGDIR SALOGDIR
+ echo /usr/sbin:/usr/bin:/usr/openwin/bin:/usr/ucb:/usr/openv/netbackup/bin:/usr/openv/netbackup/bin/admincmd:/usr/openv/netbackup/bin/goodies:/usr/openv/volmgr/bin:/usr/bin:/sbin:/usr/sbin:/bin
+ grep /usr/openv/db//bin
+ [ 1 != 0 ]
PATH=/usr/sbin:/usr/bin:/usr/openwin/bin:/usr/ucb:/usr/openv/netbackup/bin:/usr/openv/netbackup/bin/admincmd:/usr/openv/netbackup/bin/goodies:/usr/openv/volmgr/bin:/usr/bin:/sbin:/usr/sbin:/bin:/usr/openv/db//bin
+ export PATH
+ [ ]
LD_LIBRARY_PATH=/usr/openv/db//lib:/usr/openv/db//bin
+ export LD_LIBRARY_PATH
+ tr -d  \011
+ grep ^EMMSERVER=
+ awk -F= {print $2}
emmserver=nbumastersol
+ repeated_try_on_SuSE10 nbumastersol
+ [ -f /etc/SuSE-release ]
+ return
+ /usr/openv/netbackup/bin/bpclntcmd -is_local_host nbumastersol
+ [ 0 -eq 0 ]
+ /usr/openv/db/bin/nbdbms_start_server
+ exit 0

nbdbms_start_server that is called from nbdbadmin_start_stop is shell script, too.

bash-3.00# sh -x /usr/openv/db/bin/nbdbms_start_server
+ uname -s
ECHO=/usr/bin/echo
PATH=/usr/sbin:/usr/bin:/usr/openwin/bin:/usr/ucb:/usr/openv/netbackup/bin:/usr/openv/netbackup/bin/admincmd:/usr/openv/netbackup/bin/goodies:/usr/openv/volmgr/bin:/usr/bin:/sbin:/usr/sbin:/bin
+ . /usr/openv/db/vxdbms_env.sh
VXDBMS_HOME=/usr/openv/db/
SQLANY=/usr/openv/db/
SQLANY11=/usr/openv/db/
SQLANYSH11=/usr/openv/db//shared
SATMP=/usr/openv/tmp/sqlany
SATMP_UMASK=077
SADIAGDIR=/usr/openv/db//log
SALOGDIR=/usr/openv/db//log
+ export VXDBMS_HOME SQLANY SQLANY11 SQLANYSH11 SATMP SATMP_UMASK SADIAGDIR SALOGDIR
+ echo /usr/sbin:/usr/bin:/usr/openwin/bin:/usr/ucb:/usr/openv/netbackup/bin:/usr/openv/netbackup/bin/admincmd:/usr/openv/netbackup/bin/goodies:/usr/openv/volmgr/bin:/usr/bin:/sbin:/usr/sbin:/bin
+ grep /usr/openv/db//bin
+ [ 1 != 0 ]
PATH=/usr/sbin:/usr/bin:/usr/openwin/bin:/usr/ucb:/usr/openv/netbackup/bin:/usr/openv/netbackup/bin/admincmd:/usr/openv/netbackup/bin/goodies:/usr/openv/volmgr/bin:/usr/bin:/sbin:/usr/sbin:/bin:/usr/openv/db//bin
+ export PATH
+ [ ]
LD_LIBRARY_PATH=/usr/openv/db//lib:/usr/openv/db//bin
+ export LD_LIBRARY_PATH
NBDBMS_CONF_DIR=/usr/openv/var/global
+ [ -f /usr/openv/db//bin/dbsrv9 ]
SATMP=/usr/openv/tmp/sqlany
+ export SATMP
SATMP_UMASK=077
+ export SATMP_UMASK
SALOGDIR=/usr/openv/db//log
+ export SALOGDIR
SADIAGDIR=/usr/openv/db//log
+ export SADIAGDIR
+ cat /usr/openv/db//bin/servername
server_name=NB_nbumastersol
+ [ -f /usr/openv/db//bin/dbsrv9 ]
temp_out_file=/usr/openv/tmp/sqlany/nbdbms_start_server.out
+ rm -f /usr/openv/tmp/sqlany/nbdbms_start_server.out
START=0
STOP=0
STAT=0
FORCE=0
+ cd /usr/openv/db/
+ [ 0 = 0 ]
START=1
+ [ 1 = 1 ]
databases_conf=/usr/openv/var/global/databases.conf
+ [ ! -f /usr/openv/var/global/server.conf ]
+ [ ! -f /usr/openv/var/global/databases.conf ]
command_args="/usr/openv/db//bin/NB_dbsrv" @"/usr/openv/var/global/server.conf"
+ rm -f /usr/openv/var/global/databases.conf.new
+ touch /usr/openv/var/global/databases.conf.new
+ read Line
+ cat /usr/openv/var/global/databases.conf
+ sed s/"//g
+ awk {print $1}
+ echo "/usr/openv/db/data/NBDB.db" -n NBDB
filename=/usr/openv/db/data/NBDB.db
+ [ -s /usr/openv/db/data/NBDB.db ]
+ echo "/usr/openv/db/data/NBDB.db" -n NBDB
+ read Line
+ sed s/"//g
+ awk {print $1}
+ echo "/usr/openv/db/data/BMRDB.db" -n BMRDB
filename=/usr/openv/db/data/BMRDB.db
+ [ -s /usr/openv/db/data/BMRDB.db ]
+ echo "/usr/openv/db/data/BMRDB.db" -n BMRDB
+ read Line
+ mv /usr/openv/var/global/databases.conf.new /usr/openv/var/global/databases.conf
+ get_ServerPort
+ awk {
        for ( field = 1; field <= NF; field++ ) {
                if ( $field ~ /ServerPort/ ) {
                        Elements=split($field, Element, ";")
                        for ( portfield = 1; portfield <= Elements; portfield++ ) {
                                if ( Element[portfield]~ /ServerPort/ ) {
                                        split(Element[portfield], port, "=")
                                        print port[2]
                                }
                        }
                }
        }
} /usr/openv/var/global/server.conf
port=13785)
+ echo 13785)
+ awk -F) {print $1}
port=13785
commlinks=commlinks=shmem,tcpip(host=localhost;port=13785;DoBroadcast=NONE)
+ /usr/openv/db//bin/dbping -q -c ENG=NB_nbumastersol;commlinks=shmem,tcpip(host=localhost;port=13785;DoBroadcast=NONE)
+ [ 1 = 0 ]
command_args="/usr/openv/db//bin/NB_dbsrv" @"/usr/openv/var/global/server.conf" @"/usr/openv/var/global/databases.conf"
+ /usr/bin/echo /usr/openv/db//bin/dbspawn -f "/usr/openv/db//bin/NB_dbsrv" @"/usr/openv/var/global/server.conf" @"/usr/openv/var/global/databases.conf"
+ eval /usr/openv/db//bin/dbspawn -f "/usr/openv/db//bin/NB_dbsrv" @"/usr/openv/var/global/server.conf" @"/usr/openv/var/global/databases.conf"
exit_code=0
+ [ 0 != 0 ]
+ exit 0
bash-3.00#

[Denz1217]

Hi Folks,

I was able to get over this hump by recreating the test machine (VM). I decided to install Soalris 10 update 8 (X86 - 64bit). Both the fresh install and test upgrade from 6.5.4 went fine without any problems.

Unfortunately, right now, I'm doing the actual upgrade on the production machine and it froze at the part when its starting the daemons after the upgrade :( I have a pro-active ticket with Symantec. Calling them now.....

*********************************************************************************************************************************
Populating the database tables.  This will take some time.

Starting the Media Manager device daemon processes.

Do you want to start the NetBackup bprd process so
backups and restores can be initiated? [y,n] (y)
Starting the NetBackup request daemon process (bprd).
Starting the NetBackup Job Manager.
Starting the NetBackup Policy Execution Manager.
Starting the NetBackup Storage Lifecycle Manager.
Starting the NetBackup Remote Monitoring Management System.
Starting the NetBackup Key Management daemon.
Starting the NetBackup Service Layer.
Starting the NetBackup Agent Request Server.
Starting the NetBackup Bare Metal Restore daemon.
Starting the NetBackup Vault daemon.
Starting the NetBackup Service Monitor.
Starting the NetBackup Bare Metal Restore Boot Server daemon.

*********************************************************************************************************************************

[Denz1217]

I spoke too soon..... Right after posting my reply earlier, the installation script continued running :) Now on to the equally crucial post upgrade verification and tests....

[Marianne]

Thanks for sharing your experience!
Is the time taken to startup in relation to the installation size (i.e image size and/or number of items in EMM database, such as media-id, devices, stu's, etc)?
Let us know how the testing went.

[Denz1217]

Hi again,

Just a recap. I upgraded our netbackup 6.5.4 running on Solaris 10 u4 (SPARK) to NBU 7.0. The installation was doing fine until it got to the last part. It basically felt it just frozed when it's starting the services/daemons (see below). If I'm not mistaken, it didn't move for a good 10 mins or so.  During my test upgrade, this part was instantaneous which is why i suspected it hanged. But then again, my test nbu didn't have much in the catalog, volume pool, policies, storage unit, etc.But eventually, the installation went on and completed. Btw, I haven't tried stopping/starting (/usr/openv/netbackup/bin/goodies/netbackup stop|start) to justify the delay/frozen-like thingy I had during the upgrade (starting of the daemons).



During my post-activity check, all the drives we have (NDMP and NDMP) were functional. Unlike when I upgraded from 5.1 to 6.5.4 where I had some drive problems after the upgrade.... But this time around things are holding up. I've even rebooted the server and the netbackup services came up without any issues. Also, test backups and restores were fine. The real tests would be later when our daily incremental jobs execute and the same goes for the coming weekend - our weekly full.



****************************************************************************************************
Do you want to start the NetBackup bprd process so
backups and restores can be initiated? [y,n] (y)

Starting the Media Manager device daemon processes.

Do you want to start the NetBackup bprd process so
backups and restores can be initiated? [y,n] (y)
Starting the NetBackup request daemon process (bprd).
Starting the NetBackup Job Manager.
Starting the NetBackup Policy Execution Manager.
Starting the NetBackup Storage Lifecycle Manager.
Starting the NetBackup Remote Monitoring Management System.
Starting the NetBackup Key Management daemon.
Starting the NetBackup Service Layer.
Starting the NetBackup Agent Request Server.
Starting the NetBackup Bare Metal Restore daemon.
Starting the NetBackup Vault daemon.
Starting the NetBackup Service Monitor.
Starting the NetBackup Bare Metal Restore Boot Server daemon. ---> freezing-like moment

processing image candidates for storage unit .........
.
.
.

OpsCenter is the next-generation monitoring, reporting and
administrative solution designed to centrally manage one or
more NetBackup installations from a web browser.  Existing
NetBackup Operations Manager or Veritas Backup Reporter
installations can be upgraded to OpsCenter.

If an OpsCenter server already exists in your environment
or you plan to install one, enter the real hostname of that
OpsCenter server here.  Do not use a virtual name.  If you
do not want this local machine to be an OpsCenter server,
enter NONE.

Enter the OpsCenter server (default: NONE):


NetBackup server installation complete.

************************************************************************************

Thanks.

[Brian_Costello]

I wish I could say that the install started working for me as well but even with multiple OS and NBU reinstallation attempts I was never able to get it working on Solaris in a VM

[Denz1217]

Hi bcostell,

If it helps, the test VM (with vmtools) I used, is running under Vsphere 4.0. On my initial post, that Solaris VM where I encountered the problem was on ESXi 3.5 and another thing with it, when the VM was initially created, Solaris 10 32bit was selected on the VM side but the actual Solaris OS was a 64 bit one. In other words, there was a slight oversight during the VM creation. Although, I am not 100% sure if its the root cause. Thanks.

[Brian_Costello]

Unfortunately I have pretty much given up on getting Solaris to work within a VM and went with an install of SLES. My VM environment was running 4.0 on an AMD cluster and according to the information from within the OS on the VM it was running the 64bit kernel. I'm not too worried as no one really uses Solaris on anything but a SPARC but it was just a pain for a while trying to get it to work. Thank you for all of the information and I'm glad to hear you got your environment upgraded

[Douglas_A · Accepted solution]

when upgrading from 6.x to 7.x the DB conversion that takes place can take up to 1hour per 100gb Catalog (Symantec support quoted this time)

So be patient when doing your upgrades as to not break the catalog.

[sdo]

Is this time of 1hr per 100GB of catalog documented in a manual or technote anywhere?  Thx.