NBDB upgrade from 7.1.0.3 to 7.5 fails

[marekkedzierski]

Hi.. I'm trying to upgrade NB 7.1.0.3 to 7.5 (then 7.5.0.6) on VCS 5.1 cluster installed on AIX 6.1.

During upgrade installer shows this error message:

Database [NBDB] can't be rebuilt.

Unable to create/upgrade the NB database. Refer to the
log file in /usr/openv/netbackup/logs/nbdb for more
information. Rerun /usr/openv/netbackup/bin/install_bp
when the problem has been resolved.

nbdb.log:

<16> rebuild_db: There are files in the way that cannot be removed. {i.e. NBDB.dbR, BMR.dbR,...}

NBCC and nbdb validation executed before installation showed no errors.

DNS and IP resolution works fine.

Maybe you had similiar issue ??

[RamNagalla]

NBDB.dbR, BMR.dbR does not looks like the regular DB name, it has R added in the end..

ls -l /usr/openv/db/data   --> show us what all files that you are seeing in this location..

[marekkedzierski]

-rwxr-xr-x    1 root     system          101 Jul 03 2012  .odbc.ini.az
-r--------    1 root     system     26218496 Jul 03 2012  DARS_DATA.db
-r--------    1 root     system     26218496 Jul 03 2012  DARS_INDEX.db
-r--------    1 root     system     26218496 Oct 24 13:49 DBM_DATA.db
-r--------    1 root     system     26218496 Oct 24 11:15 DBM_INDEX.db
-r--------    1 root     system     44126208 Oct 24 13:49 EMM_DATA.db
-r--------    1 root     system     52727808 Oct 24 13:49 EMM_INDEX.db
-r--------    1 root     system      2433024 Oct 24 15:04 NBAZDB.db
-r--------    1 root     system         4096 Oct 24 15:04 NBAZDB.log
-r--------    1 root     system      3698688 Oct 24 15:04 NBDB.db
-r--------    1 root     system         4096 Oct 24 15:04 NBDB.log
-rw-------    1 root     system          212 Oct 24 14:27 vxdbms.conf

[mph999]

How did I know this issue was going to be an AIX ... I have to go out, at first glance i have no idea what is going on, my first thought is that there is something wrong with one of the config files perhaps, vxdbms.conf, server.conf, bp.conf etc ... But then again, if the system was running previously I would have thought such a problem would stop the DB from starting. Probably best to log a call on this - the system is currently down and so you need some fairly fast resolution. Depending on how broken this is you might need to roll back, so try and have install media for the previous version available, along with the catalog dr files just in case.

[watsons]

The size of NBDB.log does not look correct to me.. I mean 4096 is more like a default (see the NBAZDB.log) but if you have been running NBU for some time, NBDB.log should have a larger size.

I would think NBDB.log is corrupted, but unsure if it's before or during the upgrade.

To confirm, verify the issue in your /usr/openv/db/log/server.log  to check if NBDB.log can be processed during NBDB startup.

If confirm NBDB.log is the culprit, use this technote http://www.symantec.com/docs/TECH173414 to rebuild the transaction log. 

IMPORTANT NOTE: Running the above can be dangerous, log a support call if you don't feel comfortable doing that and they can guide you through.

It is also best to get ready with your last catalog backup, in case you need to recover these NBDB from the past.

[Marianne]

Hi Marek 

I apologize for the questions that I'm going to ask as I have great respect for your level of knowledge....

Please share preparation steps before you started with the upgrade.

In what state was NBU service group?
Hopefully just NetBackup resource offlined and not entire service group?

PS:

Hopefully high priority call logged with Support by now?

[mph999]

ANything in the server.log ?

The .R files are normal, part of the upgrade process.

[marekkedzierski]

service group was freezed, NBDB works fine, I can start the database server, nbdb_ping reports alive, server.log shows that database starts without problems.

[marekkedzierski]

It was a size just after a catalog backup so log was truncated.

[marekkedzierski]

Before upgrade I checked consistency using NBCC and NBDB structure using nbdb full validation.

[Marianne]

Not sure if this is applicable to 7.5 as well - please have a look:

• TECH189078   During installation of 7.5.0.1, 7.5.0.3 or 7.5.0.4, if the <install_path>/openv/db/data...

[marekkedzierski · Accepted solution]

Ok, we've found a solution.

It was a problem with vxdbms.conf file. I don't know why but some entries were missing:

VXDBMS_NB_DATABASE = NBDB
VXDBMS_NB_PASSWORD = 2de0630e88a124027901a7c2f51d973f578efbf4df3d1b4b
AZ_DB_PASSWORD = Jj8mkP3sKTo=
VXDBMS_NB_FULL_KEYWORD = NBDB:2873833:1382602519:F
VXDBMS_NB_INCREMENTAL = NBDB.log.1
 

We added missing entries and upgrade ended successfully

VXDBMS_NB_SERVER = NB_netbackup
VXDBMS_NB_REMOTE_SERVER = NB_netbackup
VXDBMS_NB_PORT = 13785
VXDBMS_NB_DATABASE = NBDB
VXDBMS_AZ_DATABASE = NBAZDB
VXDBMS_NB_DATA = /netbackup/db/data
VXDBMS_NB_INDEX = /netbackup/db/data
VXDBMS_NB_TLOG = /netbackup/db/data
VXDBMS_NB_STAGING = /netbackup/db/staging
VXDBMS_NB_PASSWORD = 596b2bc5a31344abe6c1749d1684fad3b48892a55cab796c
AZ_DB_PASSWORD = Jj8mkP3sKTo=
VXDBMS_NB_FULL_KEYWORD = NBDB:2874248:1382686920:F
VXDBMS_NB_INCREMENTAL = NBDB.log.1
VXDBMS_AZ_INCREMENTAL = NBAZDB.log.1

[Marianne]

Great stuff! Thanks for the feedback!

[marekkedzierski]

Thanks for your help !

[mph999]

So. as I thought then ....