10-24-2013 08:48 AM
Hi.. I'm trying to upgrade NB 7.1.0.3 to 7.5 (then 7.5.0.6) on VCS 5.1 cluster installed on AIX 6.1.
During upgrade installer shows this error message:
Database [NBDB] can't be rebuilt.
Unable to create/upgrade the NB database. Refer to the
log file in /usr/openv/netbackup/logs/nbdb for more
information. Rerun /usr/openv/netbackup/bin/install_bp
when the problem has been resolved.
nbdb.log:
<16> rebuild_db: There are files in the way that cannot be removed. {i.e. NBDB.dbR, BMR.dbR,...}
NBCC and nbdb validation executed before installation showed no errors.
DNS and IP resolution works fine.
Maybe you had similiar issue ??
Solved! Go to Solution.
10-25-2013 01:44 AM
Ok, we've found a solution.
It was a problem with vxdbms.conf file. I don't know why but some entries were missing:
VXDBMS_NB_DATABASE = NBDB
VXDBMS_NB_PASSWORD = 2de0630e88a124027901a7c2f51d973f578efbf4df3d1b4b
AZ_DB_PASSWORD = Jj8mkP3sKTo=
VXDBMS_NB_FULL_KEYWORD = NBDB:2873833:1382602519:F
VXDBMS_NB_INCREMENTAL = NBDB.log.1
We added missing entries and upgrade ended successfully
VXDBMS_NB_SERVER = NB_netbackup
VXDBMS_NB_REMOTE_SERVER = NB_netbackup
VXDBMS_NB_PORT = 13785
VXDBMS_NB_DATABASE = NBDB
VXDBMS_AZ_DATABASE = NBAZDB
VXDBMS_NB_DATA = /netbackup/db/data
VXDBMS_NB_INDEX = /netbackup/db/data
VXDBMS_NB_TLOG = /netbackup/db/data
VXDBMS_NB_STAGING = /netbackup/db/staging
VXDBMS_NB_PASSWORD = 596b2bc5a31344abe6c1749d1684fad3b48892a55cab796c
AZ_DB_PASSWORD = Jj8mkP3sKTo=
VXDBMS_NB_FULL_KEYWORD = NBDB:2874248:1382686920:F
VXDBMS_NB_INCREMENTAL = NBDB.log.1
VXDBMS_AZ_INCREMENTAL = NBAZDB.log.1
10-24-2013 09:38 AM
NBDB.dbR, BMR.dbR does not looks like the regular DB name, it has R added in the end..
ls -l /usr/openv/db/data --> show us what all files that you are seeing in this location..
10-24-2013 10:19 AM
-rwxr-xr-x 1 root system 101 Jul 03 2012 .odbc.ini.az
-r-------- 1 root system 26218496 Jul 03 2012 DARS_DATA.db
-r-------- 1 root system 26218496 Jul 03 2012 DARS_INDEX.db
-r-------- 1 root system 26218496 Oct 24 13:49 DBM_DATA.db
-r-------- 1 root system 26218496 Oct 24 11:15 DBM_INDEX.db
-r-------- 1 root system 44126208 Oct 24 13:49 EMM_DATA.db
-r-------- 1 root system 52727808 Oct 24 13:49 EMM_INDEX.db
-r-------- 1 root system 2433024 Oct 24 15:04 NBAZDB.db
-r-------- 1 root system 4096 Oct 24 15:04 NBAZDB.log
-r-------- 1 root system 3698688 Oct 24 15:04 NBDB.db
-r-------- 1 root system 4096 Oct 24 15:04 NBDB.log
-rw------- 1 root system 212 Oct 24 14:27 vxdbms.conf
10-24-2013 11:08 AM
10-24-2013 10:15 PM
The size of NBDB.log does not look correct to me.. I mean 4096 is more like a default (see the NBAZDB.log) but if you have been running NBU for some time, NBDB.log should have a larger size.
I would think NBDB.log is corrupted, but unsure if it's before or during the upgrade.
To confirm, verify the issue in your /usr/openv/db/log/server.log to check if NBDB.log can be processed during NBDB startup.
If confirm NBDB.log is the culprit, use this technote http://www.symantec.com/docs/TECH173414 to rebuild the transaction log.
IMPORTANT NOTE: Running the above can be dangerous, log a support call if you don't feel comfortable doing that and they can guide you through.
It is also best to get ready with your last catalog backup, in case you need to recover these NBDB from the past.
10-24-2013 10:53 PM
Hi Marek
I apologize for the questions that I'm going to ask as I have great respect for your level of knowledge....
Please share preparation steps before you started with the upgrade.
In what state was NBU service group?
Hopefully just NetBackup resource offlined and not entire service group?
PS:
Hopefully high priority call logged with Support by now?
10-24-2013 11:32 PM
ANything in the server.log ?
The .R files are normal, part of the upgrade process.
10-24-2013 11:37 PM
service group was freezed, NBDB works fine, I can start the database server, nbdb_ping reports alive, server.log shows that database starts without problems.
10-24-2013 11:50 PM
It was a size just after a catalog backup so log was truncated.
10-24-2013 11:54 PM
Before upgrade I checked consistency using NBCC and NBDB structure using nbdb full validation.
10-25-2013 12:10 AM
Not sure if this is applicable to 7.5 as well - please have a look:
10-25-2013 01:44 AM
Ok, we've found a solution.
It was a problem with vxdbms.conf file. I don't know why but some entries were missing:
VXDBMS_NB_DATABASE = NBDB
VXDBMS_NB_PASSWORD = 2de0630e88a124027901a7c2f51d973f578efbf4df3d1b4b
AZ_DB_PASSWORD = Jj8mkP3sKTo=
VXDBMS_NB_FULL_KEYWORD = NBDB:2873833:1382602519:F
VXDBMS_NB_INCREMENTAL = NBDB.log.1
We added missing entries and upgrade ended successfully
VXDBMS_NB_SERVER = NB_netbackup
VXDBMS_NB_REMOTE_SERVER = NB_netbackup
VXDBMS_NB_PORT = 13785
VXDBMS_NB_DATABASE = NBDB
VXDBMS_AZ_DATABASE = NBAZDB
VXDBMS_NB_DATA = /netbackup/db/data
VXDBMS_NB_INDEX = /netbackup/db/data
VXDBMS_NB_TLOG = /netbackup/db/data
VXDBMS_NB_STAGING = /netbackup/db/staging
VXDBMS_NB_PASSWORD = 596b2bc5a31344abe6c1749d1684fad3b48892a55cab796c
AZ_DB_PASSWORD = Jj8mkP3sKTo=
VXDBMS_NB_FULL_KEYWORD = NBDB:2874248:1382686920:F
VXDBMS_NB_INCREMENTAL = NBDB.log.1
VXDBMS_AZ_INCREMENTAL = NBAZDB.log.1
10-25-2013 02:07 AM
Great stuff! Thanks for the feedback!
10-25-2013 02:18 AM
Thanks for your help !
10-25-2013 05:32 AM
So. as I thought then ....