Database [NBDB] is not available - nbdb_ping

[Jaywalker1979]

We have an issue with our netbackup server were the Netbackup Enterprise Media Manager service would not start, upon launching the application a warning appears asking to continue, cancel or repair. Repair doesn't appear to work and you can't manually start the service from services.msc

Initially I thought it may have been an issue with the vxdbms.conf file as it didn't appear where it should. I followed the steps to re create the file using these steps

https://www.veritas.com/support/en_US/article.TECH71702

and now an nbdb_ping returns the error "Database [NBDB] is not available" when trying an nbdb_admin -start command I receive the error "no database information exists in vxdbms.conf."

And I'm still receiving the error about the Netbackup Enterprise Media Manager service.

Any suggestions would be gratefully received. Thanks in advance!

[quebek]

Hey

What is windows evenlog saying? Did you see this https://www.veritas.com/support/en_US/article.HOWTO61176

Can you share your vxdbms.conf file content?

Here is one of mines - NBU 8.0

VXDBMS_NB_SERVER = NB_server_NBU
VXDBMS_NB_REMOTE_SERVER = NB_mediaserver_NBU
VXDBMS_NB_PORT = 13785
VXDBMS_NB_DATABASE = NBDB
VXDBMS_BMR_DATABASE = BMRDB
VXDBMS_AZ_DATABASE = NBAZDB
VXDBMS_NB_DATA = E:\Program Files\Veritas\NetBackupDB\data
VXDBMS_NB_INDEX = E:\Program Files\Veritas\NetBackupDB\data
VXDBMS_NB_TLOG = E:\Program Files\Veritas\NetBackupDB\data
VXDBMS_NB_PASSWORD = some stringssss
AZ_DB_PASSWORD = some stringsss
VXDBMS_ODBC_DRIVER = NB SQL Anywhere
VXDBMS_NB_FULL_KEYWORD = NBDB:1315372:1497348000:F
VXDBMS_NB_INCREMENTAL = NBDB.log.1
VXDBMS_BMR_INCREMENTAL = BMRDB.log.1
VXDBMS_AZ_INCREMENTAL = NBAZDB.log.1
VXDBMS_BACKUP_POLICY = HOT_INX
VXDBMS_BACKUP_SCHEDULE_TYPE = 0
VXDBMS_LAST_CATALOG_BACKUP_TIME = 1497346235

[Marianne]

We need to know why you received the error message in the first place.

A common reason is that the filesystem where NBDB resides has filled up and NBU has shutdown EMM/NBDB in an attempt to prevent corruption.

I am curious about this statement:

"... vxdbms.conf file ...didn't appear where it should... "

Where exactly did you look and where did you find it?
What is the current location and what is the contents?

To know why EMM won't start, do the following from cmd:

Stop and restart NBU:

1. bpdown -v -f   
2. bpps - confirm all services are down. If not try stop again. if that still fails, kill the pid(s). eventually you should see no process
3. Use Services to stop/start Symantec Private Branch Exchange
4. bpup -v -f
5. confirm services are started with bbps

If EMM does not stay up, check the logs:

1. vxlogview -o 111 -t 00:10:00
(Above will display EMM log for the last 10 minutes. Adjust to catch actions since bpup command)
If lots of info is displayed, save output to a text filie in a doc or temp folder, e.g.
vxlogview -o 111 -t 00:10:00 > C:\TEMP\111.txt

2. server.log  (in <install-path>\VERITAS\NetBackupDB\log\)
Open log with Wordpad or Notepad.

[Michael_G_Ander]

nbdb_admin validate full (I think the command is) might tell you if the nbdb database is corrupted.

Have also seen that highly fragmented disk or high response time can cause the nbdb not to start up.

[Jaywalker1979]

Hi,

I forgot to add that when trying to recreate the file the command stops the database server, states it is rebuilding the netbackup database (NBDB)- it may take some time. Then "Database can't be rebuilt". The database server is restarted and then "Exiting from create_nbdb, status[13]."

The content of the vxdbms.conf file is

VXDBMS_NB_SERVER = NB_kmbcsanbackup
VXDBMS_ODBC_DRIVER = NB SQL Anywhere

hope this helps!

[quebek]

Hey

Kinda poor content of this file...

Can you please check this folder if you have copy of this file??<install_dir>\Veritas\NetBackupDB\staging

if there is copy it back to original location, ie <install_dir>\Program Files\Veritas\NetBackupDB\data 

and try to start up... also you can try to copy all staging back to data foder.... last resort.

[Marianne]

Please go through everything in my post and let us know.

Sincerely hope that you have a recent full catalog backup....

[Jaywalker1979]

I have a recent catalog backup (I think, although I can't check for obvious reasons). Is it best to just go with a fresh install and catalog restore?

[quebek]

I would have tried with this data from staging.... more painless ;)

[Marianne]

I want you to go through the steps in my 1st reply.

This will tell us what exactly is wrong with the db or logs.

Catalog recovery is absolute last resort. 
But good to know that you have a backup.
Just in case....

[Jaywalker1979]

Hi

Thanks for getting back to me, I've attached the EMM log. I'll attach the server log as well.

[Marianne]

Please share sequence of events over the last couple of hours?

EMM log only shows events between 13:17 and 13:19. 

Server.log seems to show a system reboot after that and "Database recovery in progress" at 14:49:46

 All seems good at this point: 

I. 06/13 14:49:47. Database server started at Tue Jun 13 2017 14:49
I. 06/13 14:49:47. Trying to start SharedMemory link ...
I. 06/13 14:49:47. SharedMemory link started successfully
I. 06/13 14:49:47. Trying to start TCPIP link ...
I. 06/13 14:49:47. Starting on port 13785
I. 06/13 14:49:47. TCPIP link started successfully
I. 06/13 14:49:47. Now accepting requests

So, please talk to us... tell us what you did at what time with the EMM log for corresponding period.... 

PS.

End of working day for me. Hopefully someone else can assist you from here.

I wanted to ask you to log a Support call, but NBU 7.6.x ran out of support 4 months ago.