Symantec Netbackup 6.5.4 to 7.5 Upgrade Failed

[phaul]

Hi Guys,

I have a problem after upgrading symantec netbackup 6.5.4 to NBU netbackup 7.5 for red hat linux 6.2. I've set up 2 clustered master server (nbu 6.5 and update to 6.5.4) with VVR. I readed all netbackup 7.1 and 7.5 release notes, install guide, upgrade guide and clustered master server guide. But after upgrading to nbu 7.5 i received these error:

./netbackup start
NetBackup Authentication daemon started.
NetBackup network daemon started.
NetBackup client daemon started.
NetBackup SAN Client Fibre Transport daemon started.
NetBackup will not run without /usr/openv/db/bin/NB_dbsrv running
NetBackup Authorization daemon started.
NetBackup Event Manager started.
NetBackup Audit Manager started.
NetBackup Deduplication Manager started.
NetBackup Deduplication Engine started.
NetBackup Enterprise Media Manager started.
NetBackup Resource Broker started.
Rebuilding device nodes.
Media Manager daemons started.
NetBackup request daemon started.
NetBackup compatibility daemon started.
NetBackup Job Manager started.
NetBackup Policy Execution Manager started.
NetBackup Storage Lifecycle Manager started.
NetBackup Indexing Manager started.
NetBackup Remote Monitoring Management System started.
NetBackup Key Management daemon started.
NetBackup Service Layer started.
NetBackup Agent Request Server started.
NetBackup Bare Metal Restore daemon started.
NetBackup Vault daemon started.
NetBackup CloudStore Service Container started.
VCS
NetBackup Service Monitor started.
NetBackup Bare Metal Restore Boot Server daemon started.

I tried to start NB_dbsrv but i received these error:

/usr/openv/db/bin/NB_dbsrv: error while loading shared libraries: libdbserv11_r.so: cannot open shared object file: No such file or directory


Please Help.

Thanks.

[revarooo]

Try this:

. /usr/openv/db/vxdbms_env.sh

/usr/openv/db/bin/nbdbms_start_server

Does that work to bring up the database?

[phaul]

I tried this command but I got these error:

./nbdbms_start_server

The server configuration file /usr/openv/var/global/server.conf does not exist
Aborting start server attempt
 

I have My server.conf file but it resides in /opt/VRTSnbu/var/global/server.conf

Thanks.

[revarooo]

ok is /usr/openv/var linked to /opt/VRTSnbu/var ?

ls -ld /usr/openv/var

[phaul]

How will I know if /usr/openv/var linked to /opt/VRTSnbu/var?

[Yasuhisa_Ishika]

ls -ld /usr/openv/var

[Marianne]

Sounds if NBU is not clustered correctly.

Or else rsh was not enabled during upgrade which will result in 'broken' cluster.

bp.conf should contain correct EMM location on shared drive.

Best to log a Sev 1 call with Support to assist with investigation of current state and fixing...

[phaul]

Here's the output:

ls -ld /usr/open/var

drwxr-xr-x. 8 root bin 4096 May 17 01:34 /usr/openv/var

[phaul]

The NBU is clustered correctly. Before I upgraded it to 7.5 it's working correctly. I'm using GCO/VVR and i have no shared disk configured.

Thanks.

[watsons]

When it comes to GCO, you will need to follow upgrade steps as in:

http://www.symantec.com/docs/HOWTO73064

[phaul]

I tried GCO upgrade steps but i doesn't work.

Thanks.

[Marianne]

What does  " but i doesn't work " mean?

Can you share step-by-step actions and where it failed? With what error message?

Please share all of the following:

Your installation logs. Look for install_trace.####  in /usr/openv/tmp.
There should be one for the initial install and one for the upgrade.
Please post as attachments.

bp.conf

Cluster 'response file' :  NBU_RSP in /usr/openv/netbackup/bin/cluster

[phaul]

These are the logs regarding the upgrade of NBU 6.5.4 -> 7.5

Copying RedHat2.6.18 Client Binaries

Terminating NetBackup and/or Media Manager processes.


Installing PBX...
Same version is already installed. Nothing to upgrade for package VRTSpbx
Installation log located here: /var/tmp/installpbx-32432-051812150802.log


Unpacking SYMCnbclt package.
Checking for pre-existing SYMCnbclt package.
Removing pre-existing SYMCnbclt package.
Installing SYMCnbclt package.
Installation of SYMCnbclt was successful.
More details regarding SYMCnbclt can be found in file
/tmp/install_cltpkg_trace.32004 on node1.

Terminating NetBackup and/or Media Manager processes.

Installing NB-Java.

Unpacking SYMCnbjava package.
Checking for pre-existing SYMCnbjava package.
Removing pre-existing SYMCnbjava package.
Installing SYMCnbjava package.
Installation of SYMCnbjava was successful.
More details regarding SYMCnbjava can be found in file
/tmp/install_javapkg_trace.1245 on node1.

Unpacking SYMCnbjre package.
Checking for pre-existing SYMCnbjre package.
Removing pre-existing SYMCnbjre package.
Installing SYMCnbjre package.
Installation of SYMCnbjre was successful.
More details regarding SYMCnbjre can be found in file
/tmp/install_jrepkg_trace.1245 on node1.
+ /bin/cp ./nbj.conf /usr/openv/java

Migrating log files in /usr/openv/logs
Migration of log files complete.

NetBackup Deduplication software is installed, overwriting binaries.
Existing version:
7.0000.0012.0118
+ Extracting PDDE agent package (/usr/openv/netbackup/client/Linux/RedHat2.6.18/pddeagent.tar.gz)...
Using /tmp/pdde_pkg to store temporary package data.
Unpacking SYMCpddea package.
Checking for pre-existing SYMCpddea package.
Removing pre-existing SYMCpddea package.
Installing SYMCpddea package.
+ Extracting PDDE server package (/usr/openv/pddeserver.tar.gz)...
Using /tmp/pdde_pkg to store temporary package data.
Unpacking SYMCpddes package.
Checking for pre-existing SYMCpddes package.
Removing pre-existing SYMCpddes package.
Installing SYMCpddes package.
Keeping existing /usr/openv/lib/ost-plugins/pd.conf
Saving a copy of the uninstall script to /opt/pdde/pddeuninstall.sh
Starting setup for PDDE script
pdregistry.cfg exists.
removing /etc/default/pdde
Done setup for PDDE script
Checking to see if the PDDE configuration needs upgrading
Agent config file path found: {/etc/puredisk/Agent.cfg}
Making changes to /usr/openv/lib/ost-plugins/pd.conf
RESTORE_DECRYPT_LOCAL already in pd.conf
PREFETCH_SIZE already in pd.conf
META_SEGKSIZE already in pd.conf
CLIENT_POLICY_DATE already in pd.conf
Making changes to /usr/openv/lib/ost-plugins/pd.conf
FIBRECHANNEL already in pd.conf

PDDE install finished successfully.
Version now installed:
7.0000.0012.0118
Full PDDE installation log saved to: /var/log/puredisk/2012-05-18_15:09-pdde-install.log


The following automatic startup and shutdown scripts (respectively)
have been installed.  They will cause the NetBackup daemons to be
automatically shut down and restarted each time the system boots.

/etc/rc.d/rc2.d/S77netbackup
/etc/rc.d/rc3.d/S77netbackup
/etc/rc.d/rc5.d/S77netbackup
/etc/rc.d/rc0.d/K01netbackup
/etc/rc.d/rc1.d/K01netbackup
/etc/rc.d/rc6.d/K01netbackup


Installation of Java LiveUpdate agent succeeded.  Refer to file
/tmp/JLU-Log/JavaLiveUpdate-Install.log on node1
for installation details.


Checking LiveUpdate registration for the following products: NB CLT
This may take a few minutes.

Product NB is installed and will be registered.
Product CLT_Linux-PSERIES is installed and will be registered.
Product CLT_Linux-ZSERIES is installed and will be registered.

Updating LiveUpdate registration now...this may take some time.


NetBackup Enterprise Server with all the features enabled is the active NetBackup.
Do you want to continue the installation
with the current active NetBackup license key? [y,n] (y)
All additional keys should be added at this time.
Do you want to add additional license keys now? [y,n] (y) n

Use /usr/openv/netbackup/bin/admincmd/get_license_key
to add, delete or list license keys at a later time.

Installing NetBackup Enterprise Server version: 7.5
Sending SIGHUP to xinetd process.

Reloading configuration:                                   [  OK  ]

Converting STREAMS files.  This may take a few minutes.

STREAMS files conversion is complete.


Successfully updated the session cache parameters.
Starting the NetBackup network daemon.
Starting the NetBackup client daemon.
Starting the NetBackup SAN Client Fibre Transport daemon.

Installed SQL Anywhere Version 11.0.1.2645
Installation completed successfully

AZ database setup complete.

Database server is NB_nbuvcs
NetBackup database already exists in /opt/VRTSnbu/db/data/NBDB.db
Starting the NetBackup database server [NB_nbuvcs].
Stopping the NetBackup database server [NB_nbuvcs].
Rebuilding the NetBackup database [NBDB].  This will take some time.
***** SQL error: Communication error
SQL Anywhere Unload Utility Version 11.0.1.2645
Connecting and initializing
Unloading user and group definitions
Unloading table definitions
Database [NBDB] can't be rebuilt.
Starting the NetBackup database server [NB_nbuvcs].

Unable to create/upgrade the NB database.  Refer to the
log file in /usr/openv/netbackup/logs/nbdb for more
information.  Rerun /usr/openv/netbackup/bin/install_bp
when the problem has been resolved.
 

NBU_RSP contains these file:

#DO NOT DELETE OR EDIT THIS FILE!!!
NBU_GROUP=nbu_group
SHARED_DISK=/opt/VRTSnbu
NODES=node1
PROBE_PROCS=nbevtmgr nbstserv vmd bprd bpdbm nbpem nbjm nbemm nbrb NB_dbsrv
VNAME=nbuvcs
CLUTYPE=VCS

DIR=netbackup mkdir
DIR=netbackup/db mv
DIR=var mkdir
DIR=var/global mv
DIR=volmgr/mkdir
DIR=volmgr/misc mkdir
DIR=volmgr/misc/robotic_db mv

LINK=volmgr/misc/robotic_db
LINK=netbackup/db
LINK=var/global
START_PROCS=NB_dbsrv nbevtmgr nbnos nbemm nbrb ltid vmd bpcompatd nbjm nbpem nbstserv nbrmms nbsl nbvault nbsvcmon bpdbm bprd bptm bpbrmds bpsched bpcd bpversion bpjobd
 nbproxy vltcore acsd tl8cd odld tldcd tl4d tlmd tshd rsmd tlhcd pbx_exchange nbkms
DIR=kms mv
 

Attached file is the nbdb logs. Hope it helps.

Thanks.

[mph999]

Just a general note, not aimed at the OP ...

Just because a cluster is working before the upgrade, does NOT mean it is clustered correctly.

I saw this exact issue the other day ...

Cluster was working, upgrade failed.

Issue was that system was configured witha sym link from /usr/openv/db/data > /opt/VRTSnbu/db/data , which was missing after the install script was run.

Here is an example from a cluster that is correctly configured  ...

/usr/openv/db/data doesn't even exist ...

rdgv240sol22 # ls -al /usr/openv/db/data

/usr/openv/db/data: No such file or directory

The location of the DB is defined in bp.conf ...

VXDBMS_NB_DATA = /opt/VRTSnbu/db/data

Regards,

Martin

[Marianne]

Can we take a step back. In what state was the cluster when you started the upgrade?

Was the entire Service Group offlined or just the NBU resource?

Was the mount point and virtual hostname and IP available and online?

This says to me there were probably more resources offlined than just NBU resource:

***** SQL error: Communication error

[phaul]

The Cluster service group is online when i started the upgrade.

The NBU service group (freeze persistent) is online except the Netbackup resource.

The RVG service group is online.

[Gnur]

Hey Japao,

I came across this issue while upgrading 656 to 75 -> 7501 in a Solaris 10 & VCS 6 cluster. If RSH/RCP was not allowed across the two nodes of the cluster or SSH is blocked, and/or cluster resources were misconfigured or commented out in main.cf, the cluster_config script will have trouble creating symbolic links for both nodes during the run of the cluster_upgrade script (called during cluster_config and can be ran in standalone). NetBackup software developers should consider revamping this script to have more flexibility and work around some of these issues.

The answer is partially found in the NBU_RSP (as indicated by Marianne) in '/usr/openv/netbackup/bin/cluster/':

DIR=netbackup/db
DIR=var/global
DIR=volmgr/misc/robotic_db

A symbolic link needs to be created per each of these paths to the shared disk /opt/VRTSnbu.

Verify on the active node of the cluster that these paths still exist pre 7.5 upgrade:

/opt/VRTSnbu/netbackup

/opt/VRTSnbu/var

/opt/VRTSnbu/volmgr/misc

Of course stop NBU services on both hosts and rename these directories to <dir>.orig (if they were named .orig as symbolic links -proceed to below) and then create links as such (do it for both nodes of the cluster):

$ mv /usr/openv/netbackup/db /usr/openv/netbackup/db.orig

$ mv /usr/openv/var/global /usr/openv/var/global.orig

$ mv /usr/openv/volmgr/misc/robotic_db /usr/openv/volmgr/misc/robotic_db.orig

$ ln -s /opt/VRTSnbu/netbackup/db /usr/openv/netbackup/db

$ ln -s /opt/VRTSnbu/var/global /usr/openv/var/global

$ ln -s /opt/VRTSnbu/volmgr/misc/robotic_db /usr/openv/volmgr/misc/robotic_db

Once links are created -simply cd into each soft link perform a "pwd -P" (Learned this tip from a BCS).

It should indicated that you are traversing the correct link with /opt/VRTSnbu.

Restart nbu and verify that NBDB flag you saw earlier is no longer flagged (bp.start_all)

Hope that will get you through.

[Gnur]

I would recommend patching to NetBackup 7.5.0.1 release for your NBU version prior to migrating the DB -as recommended to us due to a migration bug in the NBU 7.5 base release.

Also refer to the Symantec NetBackup 7.5 Upgrade Guide to perform the metadat migration:

http://www.symantec.com/business/support/index?page=content&id=DOC5198

Best of luck

[phaul]

Thanks for your response.

I tried the steps but when I started the ./bp.start_all i received these error:

[root@node1 bin]# ./bp.start_all
Starting nbatd...
Starting vnetd...
Starting bpcd...
Starting nbftclnt...
NetBackup will not run without /usr/openv/db/bin/NB_dbsrv running.
Starting nbazd...
Starting nbevtmgr...
Starting nbaudit...
Starting spad...
Starting spoold...
Starting nbemm...
Starting nbrb...
Starting ltid...
Starting bprd...
Starting bpcompatd...
Starting nbjm...
Starting nbpem...
Starting nbstserv...
Starting nbim...
Starting nbrmms...
Starting nbkms...
Starting nbsl...
Starting nbars...
Starting bmrd...
Starting nbvault...
Starting nbcssc...
Starting nbsvcmon...
Starting bmrbd...

I tried to start NB_dbsrv but i received these error:

/usr/openv/db/bin/NB_dbsrv: error while loading shared libraries: libdbserv9_r.so: cannot open shared object file: No such file or directory.

Is there any new required RPMS need in Netbackup 7.5?

Thanks

[Marianne]

Is there any new required RPMS need in Netbackup 7.5?

Have you checked the Upgrade Portal and Sort?

Extract from http://www.symantec.com/docs/TECH74584 :

For system requirements, documentation, and compatibility information, please use the Installation and Upgrade checklist found on the SORT website before any upgrade: 

        https://sort.symantec.com/checklist/install