Netbackup 6.5.6 Catalog Backup Errors

Hi,

This doesn't seem to be going as planned. Problems from the start -

C:\Program Files\VERITAS\NetBackup\bin>bpdown

NetBackup 6.5 -- Shutdown Utility

NetBackup software will be shutdown.  This may cause backups to fail.
Do you wish to proceed? (y/n) y
Shutting down services

Shutdown completed successfully.

C:\Program Files\VERITAS\NetBackup\bin>
C:\Program Files\VERITAS\NetBackup\bin>bpdown -f -v

NetBackup 6.5 -- Shutdown Utility

Shutting down services
> NetBackup Service Monitor
> NetBackup Service Monitor -- STOPPED
> NetBackup Storage Lifecycle Manager
> NetBackup Storage Lifecycle Manager -- STOPPED
> NetBackup Key Management Service
> NetBackup Key Management Service -- STOPPED
> BMR Master Service
> BMR Master Service -- STOPPED
> NetBackup Vault Manager
> NetBackup Vault Manager -- STOPPED
> NetBackup Service Layer
> NetBackup Service Layer -- STOPPED
> NetBackup Policy Execution Manager
> NetBackup Policy Execution Manager -- STOPPED
> NetBackup Job Manager
> NetBackup Job Manager -- STOPPED
> NdmpMoverListener
> NetBackup Request Daemon
> NetBackup Request Daemon -- STOPPED
> NetBackup Compatibility Service
> NetBackup Compatibility Service -- STOPPED
> NetBackup Database Manager
> NetBackup Database Manager -- STOPPED
> NetBackup Remote Manager and Monitor Service
> NetBackup Remote Manager and Monitor Service -- STOPPED
> NetBackup Device Manager
> NetBackup Device Manager -- STOPPED
> NetBackup Volume Manager
> NetBackup Volume Manager -- STOPPED
> NetBackup Resource Broker
> NetBackup Resource Broker -- STOPPED
> NetBackup Enterprise Media Manager
> NetBackup Enterprise Media Manager -- STOPPED
> ASANYs_VERITAS_NB
> ASANYs_VERITAS_NB -- STOPPED
> NetBackup Event Manager
> NetBackup Event Manager -- STOPPED
> NetBackup Notification Service
> BMR Boot Service
> NetBackup SAN Client Fibre Transport Service
> NetBackup SAN Client Fibre Transport Service -- STOPPED
> NetBackup INET Daemon
> NetBackup INET Daemon -- STOPPED

Shutdown completed successfully.

C:\Program Files\VERITAS\NetBackup\bin>bpup -e SQLANYs_VERITAS_NB

NetBackup 6.5 -- Startup Utility

NetBackup services are about to be started.
Do you wish to proceed? (y/n) y
Starting services

Start up completed successfully.

C:\Program Files\VERITAS\NetBackup\bin>nbdb_admin NBDB -validate
Adaptive Server Anywhere Validation Utility Version 9.0.2.3886
Database server not found
Database [NBDB] validation failed.

C:\Program Files\VERITAS\NetBackup\bin>nbdb_admin NBDB -validate -full
Adaptive Server Anywhere Validation Utility Version 9.0.2.3886
Database server not found
Database [NBDB] validation failed.

C:\Program Files\VERITAS\NetBackup\bin>nbdb_backup -dbn NBDB -online <C:\tmpdbsym> -truncate_tlog
Access is denied.

Apologies, didn't see you had posted.

SO the NBDB can start ...  

C:\Program Files\VERITAS\NetBackup\bin>bpup -e SQLANYs_VERITAS_NB

NetBackup 6.5 -- Startup Utility

NetBackup services are about to be started.
Do you wish to proceed? (y/n) y
Starting services

Start up completed successfully.

But can't find the NBDB to validate it ...

Well, at least we've probably found why NB can't validate during the catalog backup.

Be interesting to see if nbdb_ping works at this point.  THe DB apperas to have started, but did it stay running ?

C:\Program Files\VERITAS\NetBackup\bin>nbdb_ping
Database [NBDB] is not available.

YIjkes, so the DB started, then stoped.

OK, so restart in and wait a min till it stops (nbdb_ping) then have a peep in the server.log

Eg

C:\Program Files\Veritas\NetBackupDB\server.log

Here is the last few entries from that log file -

I. 10/14 09:10:46. Starting checkpoint of "NBDB" (NBDB.db) at Tue Oct 14 2014 09:10
I. 10/14 09:10:46. Finished checkpoint of "NBDB" (NBDB.db) at Tue Oct 14 2014 09:10
I. 10/14 09:13:13. Starting checkpoint of "BMRDB" (BMRDB.db) at Tue Oct 14 2014 09:13
I. 10/14 09:13:13. Finished checkpoint of "BMRDB" (BMRDB.db) at Tue Oct 14 2014 09:13
I. 10/14 09:30:48. Starting checkpoint of "NBDB" (NBDB.db) at Tue Oct 14 2014 09:30
I. 10/14 09:30:48. Finished checkpoint of "NBDB" (NBDB.db) at Tue Oct 14 2014 09:30
I. 10/14 09:33:14. Starting checkpoint of "BMRDB" (BMRDB.db) at Tue Oct 14 2014 09:33
I. 10/14 09:33:14. Finished checkpoint of "BMRDB" (BMRDB.db) at Tue Oct 14 2014 09:33
I. 10/14 09:50:49. Starting checkpoint of "NBDB" (NBDB.db) at Tue Oct 14 2014 09:50
I. 10/14 09:50:49. Finished checkpoint of "NBDB" (NBDB.db) at Tue Oct 14 2014 09:50
I. 10/14 09:53:15. Starting checkpoint of "BMRDB" (BMRDB.db) at Tue Oct 14 2014 09:53
I. 10/14 09:53:15. Finished checkpoint of "BMRDB" (BMRDB.db) at Tue Oct 14 2014 09:53
I. 10/14 10:10:50. Starting checkpoint of "NBDB" (NBDB.db) at Tue Oct 14 2014 10:10
I. 10/14 10:10:50. Finished checkpoint of "NBDB" (NBDB.db) at Tue Oct 14 2014 10:10
I. 10/14 10:13:16. Starting checkpoint of "BMRDB" (BMRDB.db) at Tue Oct 14 2014 10:13
I. 10/14 10:13:16. Finished checkpoint of "BMRDB" (BMRDB.db) at Tue Oct 14 2014 10:13
I. 10/14 10:30:51. Starting checkpoint of "NBDB" (NBDB.db) at Tue Oct 14 2014 10:30
I. 10/14 10:30:51. Finished checkpoint of "NBDB" (NBDB.db) at Tue Oct 14 2014 10:30
I. 10/14 10:30:57. Starting checkpoint of "BMRDB" (BMRDB.db) at Tue Oct 14 2014 10:30
I. 10/14 10:30:57. Finished checkpoint of "BMRDB" (BMRDB.db) at Tue Oct 14 2014 10:30
I. 10/14 10:30:57. Starting checkpoint of "NBDB" (NBDB.db) at Tue Oct 14 2014 10:30
I. 10/14 10:30:57. Finished checkpoint of "NBDB" (NBDB.db) at Tue Oct 14 2014 10:30
I. 10/14 10:30:57. Database server stopped at Tue Oct 14 2014 10:30
I. 10/15 11:20:22. Adaptive Server Anywhere Network Server Version 9.0.2.3886
I. 10/15 11:20:22.
I. 10/15 11:20:22. Copyright © 1989-2007 Sybase, Inc.
I. 10/15 11:20:22. Portions Copyright © 2002-2007, iAnywhere Solutions, Inc.
I. 10/15 11:20:22. All rights reserved. All unpublished rights reserved.
I. 10/15 11:20:22.  
I. 10/15 11:20:22. This software contains confidential and trade secret information of
I. 10/15 11:20:22. iAnywhere Solutions, Inc.
I. 10/15 11:20:22. Use, duplication or disclosure of the software and documentation
I. 10/15 11:20:22. by the U.S. Government is subject to restrictions set forth in a license
I. 10/15 11:20:22. agreement between the Government and iAnywhere Solutions, Inc. or
I. 10/15 11:20:22. other written agreement specifying the Government's rights to use the
I. 10/15 11:20:22. software and any applicable FAR provisions, for example, FAR 52.227-19.
I. 10/15 11:20:22.  
I. 10/15 11:20:22. iAnywhere Solutions, Inc., One Sybase Drive, Dublin, CA 94568, USA
I. 10/15 11:20:22.
I. 10/15 11:20:22. 4 logical processor(s) on 2 physical processor(s) detected.
I. 10/15 11:20:22. Per-processor licensing model. The server is limited to use 1 processor(s).
I. 10/15 11:20:22. This server is licensed to:
I. 10/15 11:20:22.     Symantec
I. 10/15 11:20:22.     Symantec Inc.
I. 10/15 11:20:22. Running on Windows 2003 Build 3790 Service Pack 2
I. 10/15 11:20:22. 51200K of memory used for caching
I. 10/15 11:20:22. Minimum cache size: 51200K, maximum cache size: 512000K
I. 10/15 11:20:22. Using a maximum page size of 4096 bytes
I. 10/15 11:20:22. Starting database "BMRDB" (C:\Program Files\VERITAS\NetBackupDB\data\BMRDB.db) at Wed Oct 15 2014 11:20
I. 10/15 11:20:22. Starting database "NBDB" (C:\Program Files\VERITAS\NetBackupDB\data\NBDB.db) at Wed Oct 15 2014 11:20
I. 10/15 11:20:22. Opening dbspace 'EMM_DATA' in file 'C:\Program Files\VERITAS\NetBackupDB\data\EMM_DATA.db' for database 'NBDB'
I. 10/15 11:20:22. Opening dbspace 'BMR_DATA' in file 'C:\Program Files\VERITAS\NetBackupDB\data\BMR_DATA.db' for database 'BMRDB'
I. 10/15 11:20:22. Performance warning: Database file "C:\Program Files\VERITAS\NetBackupDB\data\EMM_DATA.db" consists of 2 disk fragments
I. 10/15 11:20:22. Opening dbspace 'EMM_INDEX' in file 'C:\Program Files\VERITAS\NetBackupDB\data\EMM_INDEX.db' for database 'NBDB'
I. 10/15 11:20:22. Opening dbspace 'BMR_INDEX' in file 'C:\Program Files\VERITAS\NetBackupDB\data\BMR_INDEX.db' for database 'BMRDB'
I. 10/15 11:20:22. Transaction log: C:\Program Files\VERITAS\\NetBackupDB\data\NBDB.log
I. 10/15 11:20:22. Transaction log: C:\Program Files\VERITAS\\NetBackupDB\data\BMRDB.log
I. 10/15 11:20:23. Starting checkpoint of "NBDB" (NBDB.db) at Wed Oct 15 2014 11:20
I. 10/15 11:20:23. Starting checkpoint of "BMRDB" (BMRDB.db) at Wed Oct 15 2014 11:20
I. 10/15 11:20:23. Finished checkpoint of "NBDB" (NBDB.db) at Wed Oct 15 2014 11:20
I. 10/15 11:20:23. Finished checkpoint of "BMRDB" (BMRDB.db) at Wed Oct 15 2014 11:20
I. 10/15 11:20:23. Database "NBDB" (NBDB.db) started at Wed Oct 15 2014 11:20
I. 10/15 11:20:23. Database "BMRDB" (BMRDB.db) started at Wed Oct 15 2014 11:20
I. 10/15 11:20:23. Database server started at Wed Oct 15 2014 11:20
I. 10/15 11:20:23. Trying to start SharedMemory link ...
I. 10/15 11:20:23.     SharedMemory link started successfully
I. 10/15 11:20:23. Trying to start TCPIP link ...
I. 10/15 11:20:23. Starting on port 13785
I. 10/15 11:20:23.     TCPIP link started successfully
I. 10/15 11:20:23. Now accepting requests

OK, so the DB isn't stoping, well not according to that log, but it can't be found ...

There was a similar issue after a cluster upgrade (WIn ) where an incorrect registry entry caused the same or a very simlar symptom.  However, you haven;t mentioned this is a cluster (?) and I'm going to hazard a guess you haven't just upgraded ...

I presume after these shutdowns / commands, that NBU at least continues to run ?

I can only come up with three suggestions 

1/ Have a poke through the registry and check that any NBDB path is correct

2/ I think, there can be an issue with SQL tmp files, that can also cause similar symptoms, these use the server's temporary directory as indicated in the registry  - unfortunatley the information on the exact details is sketchy, we're heading a bit more towards Engineering level I think for that, I spoke to a couple of other BL engineers, and only one of them had heard of it, but didn't know exact details.

If I can get time, I don;t mind deleting these tmp files to see what happens, hopefully they will be recreated which would mean that you can do the same - which if we are incredably lucky, might just be the cause.

3/  NBDB / Catalog recovery from before the problem

Personally, I would be seriouly considering a restore - just NBDB - plent of TNs available for this, eg http://www.symantec.com/business/support/index?page=content&id=HOWTO72618

You would then need to run NBCC to mop of the mess that will be made due to the NBDB not knowing about backups taken since the catalog backup that you recover from was run - not a big deal, its easy to do but will require a support call.

If you have a support contract, then if you can upgrade, then I think (I can't promise) that a support call would be accepted to assist with this issue as part of an upgrade.

Oh, did you create the validate log

<install path>\veritas\netbackupdb\VRTS_NBDB_DBVALID_LOGS

I know things didn't go as planned, but is there anything in there ...

Another option, I have just thought of, if the enviroment is small eg. not that many tapes of backups (if on disk) then you could just reinstall and work through importing the tapes to rebuild the catalog information - it's a bit painful, but will give you a working environment.

I'm sorry I can't come up with anything else, I'm completely out of any other ideas,  Debs Wilmot (fellow colleague and forum user) isn't in today (shes in the US and I don't see her online in IM) - but I can drop her a note and see if she can come up with anything else, but I don;t know when shes back, she might be off all week, no idea.

One other thing, make sure your catalog backups are kept, we might need them - to top things off, I'm not going to be about much next week either, though there are pleanty of people on here who can assist with a restore if need be.

Hi,

I am following the TN in your last comment but can only get so far -

C:\Program Files\VERITAS\NetBackup\bin>bpdown

NetBackup 6.5 -- Shutdown Utility

NetBackup software will be shutdown.  This may cause backups to fail.
Do you wish to proceed? (y/n) y
Shutting down services
Calling nbproxyreq to cleanup nbproxy processes.
Creating NBORB...
Initializing NBORB...
Setting Timeouts...
Killing JM proxies...
Killing PEM proxies...
Sleeping for 5 seconds
Shutting down NBORB...
Destroying NBORB...
Exiting nbproxyreq...
Shutdown of one or more services failed.

C:\Program Files\VERITAS\NetBackup\bin>bpdown -f -v

NetBackup 6.5 -- Shutdown Utility

Shutting down services
> NetBackup Service Monitor
> NetBackup Service Monitor -- STOPPED
> NetBackup Storage Lifecycle Manager
> NetBackup Storage Lifecycle Manager -- STOPPED
> NetBackup Key Management Service
> NetBackup Key Management Service -- STOPPED
> BMR Master Service
> BMR Master Service -- STOPPED
> NetBackup Vault Manager
> NetBackup Vault Manager -- STOPPED
> NetBackup Service Layer
> NetBackup Service Layer -- STOPPED
> NetBackup Policy Execution Manager
> NetBackup Policy Execution Manager -- STOPPED
> NetBackup Job Manager
> NetBackup Job Manager -- STOPPED
> NdmpMoverListener
> NetBackup Request Daemon
> NetBackup Request Daemon -- STOPPED
> NetBackup Compatibility Service
> NetBackup Compatibility Service -- STOPPED
> NetBackup Database Manager
> NetBackup Database Manager -- STOPPED
> NetBackup Remote Manager and Monitor Service
> NetBackup Remote Manager and Monitor Service -- STOPPED
> NetBackup Device Manager
> NetBackup Device Manager -- STOPPED
> NetBackup Volume Manager
> NetBackup Volume Manager -- STOPPED
> NetBackup Resource Broker
> NetBackup Resource Broker -- STOPPED
> NetBackup Enterprise Media Manager
> NetBackup Enterprise Media Manager -- STOPPED
> ASANYs_VERITAS_NB
> ASANYs_VERITAS_NB -- STOPPED
> NetBackup Event Manager
> NetBackup Event Manager -- STOPPED
> NetBackup Notification Service
> BMR Boot Service
> NetBackup SAN Client Fibre Transport Service
> NetBackup SAN Client Fibre Transport Service -- STOPPED
> NetBackup INET Daemon
> NetBackup INET Daemon -- STOPPED

Shutdown completed successfully.

C:\Program Files\VERITAS\NetBackup\bin>nbdb_admin -auto_start NONE
Successfully removed NBDB from databases.conf

C:\Program Files\VERITAS\NetBackup\bin>bpup -e SQLANYs_VERITAS_NB

NetBackup 6.5 -- Startup Utility

NetBackup services are about to be started.
Do you wish to proceed? (y/n) y
Starting services

Start up completed successfully.

C:\Program Files\VERITAS\NetBackup\bin>create_nbdb -drop
Database server is NB_OF-UK1-BCKP-03
Ping server failed -- Database server not found.
Ping server failed -- Database server not found.
Database server [NB_OF-UK1-BCKP-03] not available.

Hi,

I am not able to log a call, we have no support for this.

Is Symantec consultancy available and what would the cost be?

We are at a point now where this just needs resolving.

Thanks,