Database keeps going down

[NathanNieman]

The last 3 weeks I have had the Netbackup 7.5.0.6 database just stop working.

Today This is the error I had when trying to log into see what was going on.

"Unable to obtail the server list from the Enterprise Meida Manager server   database server is down (93)"

The Master/Medai server are the same server.I have 9GB of space which is low for this server I am guessing it is everything waiting to be backup and log files.

I have no idea where to start looking into this issue.  Like I said this has been happening for about the last 3 weeks and nothing has really changed.

[Will_Restore]

Resolution:
If it is determined that nbpem is not running, it should be restarted.   This can be accomplished from the Services interface on the Windows server.  

After nbpem has restarted, start the EMM database from a command prompt, using the command nbdb_admin -start NBDB.  (This command can be found in <install_path>\VERITAS\NetBackup\bin.)

Article URL http://www.symantec.com/docs/TECH61664

[mph999 · Accepted solution]

I suspect that the disk is filling up enough to cause NBU to shut down the DB to protect it from getting corrupted.

I think it does this somewhere arounf 90% full

M

[NathanNieman]

Ah that could be the issue then.  I will clear more space.

[RamNagalla]

please see if you find any related info inthe server.log file

/usr/openv/db/log/server.log

please post the log if possible

[NathanNieman]

Ok got up to 53GB free we will see what that does.  Thank you.  I will also try the restart command.

[NathanNieman]

Sorry forgot to say this is a windows 2008 server R2.

I don't see a server.log any where.

[RamNagalla]

it should be Installpath\Veritas\NetBackupDB\log\server.log if it is winddows.

[NathanNieman]

Ah found it.  Didn't look there.   I will post if I find anything in the log.

[NathanNieman]

Ok here is what I am seeing that looks Odd to me

I. 01/02 19:14:56. Finished checkpoint of "NBAZDB" (NBAZDB.db) at Thu Jan 02 2014 19:14

I. 01/02 19:27:42. Starting database "utility_db" (utility_db) at Thu Jan 02 2014 19:27

I. 01/02 19:27:42. Database "utility_db" (utility_db) started at Thu Jan 02 2014 19:27

I. 01/02 19:30:52. Connection terminated abnormally

I. 01/02 19:30:52. Disconnected SharedMemory client's AppInfo: IP=131.40.10.88;HOST=UNS40-88;OSUSER=SYSTEM;OS='Windows 2008R2 Build 7601 Service Pack 1';EXE='C:\Program Files\Veritas\NetBackup\bin\nbstserv.exe';PID=0xf10;THREAD=0xf14;VERSION=11.0.1.2960;API=ODBC;TIMEZONEADJUSTMENT=-300

I. 01/02 19:30:52. Connection terminated abnormally

I. 01/02 19:30:52. Disconnected SharedMemory client's AppInfo: IP=131.40.10.88;HOST=UNS40-88;OSUSER=SYSTEM;OS='Windows 2008R2 Build 7601 Service Pack 1';EXE='C:\Program Files\Veritas\NetBackup\bin\nbstserv.exe';PID=0xf10;THREAD=0xf14;VERSION=11.0.1.2960;API=ODBC;TIMEZONEADJUSTMENT=-300

I. 01/02 19:30:52. Connection terminated abnormally

I. 01/02 19:30:52. Disconnected SharedMemory client's AppInfo: IP=131.40.10.88;HOST=UNS40-88;OSUSER=SYSTEM;OS='Windows 2008R2 Build 7601 Service Pack 1';EXE='C:\Program Files\Veritas\NetBackup\bin\nbstserv.exe';PID=0xf10;THREAD=0xf14;VERSION=11.0.1.2960;API=ODBC;TIMEZONEADJUSTMENT=-300

I. 01/02 19:30:52. Connection terminated abnormally

I. 01/02 19:30:52. Disconnected SharedMemory client's AppInfo: IP=131.40.10.88;HOST=UNS40-88;OSUSER=SYSTEM;OS='Windows 2008R2 Build 7601 Service Pack 1';EXE='C:\Program Files\Veritas\NetBackup\bin\nbstserv.exe';PID=0xf10;THREAD=0xf14;VERSION=11.0.1.2960;API=ODBC;TIMEZONEADJUSTMENT=-300

I. 01/02 19:30:52. Connection terminated abnormally

I. 01/02 19:30:52. Disconnected SharedMemory client's AppInfo: IP=131.40.10.88;HOST=UNS40-88;OSUSER=SYSTEM;OS='Windows 2008R2 Build 7601 Service Pack 1';EXE='C:\Program Files\Veritas\NetBackup\bin\nbpem.exe';PID=0x14d0;THREAD=0x157c;VERSION=11.0.1.2960;API=ODBC;TIMEZONEADJUSTMENT=-300

I. 01/02 19:30:52. Connection terminated abnormally

I. 01/02 19:30:52. Disconnected SharedMemory client's AppInfo: IP=131.40.10.88;HOST=UNS40-88;OSUSER=SYSTEM;OS='Windows 2008R2 Build 7601 Service Pack 1';EXE='C:\Program Files\Veritas\NetBackup\bin\bpdbm.exe';PID=0x1a00;THREAD=0xc;VERSION=11.0.1.2960;API=ODBC;TIMEZONEADJUSTMENT=-300

I. 01/02 19:30:52. Connection terminated abnormally

I. 01/02 19:30:52. Disconnected SharedMemory client's AppInfo: IP=131.40.10.88;HOST=UNS40-88;OSUSER=SYSTEM;OS='Windows 2008R2 Build 7601 Service Pack 1';EXE='C:\Program Files\Veritas\NetBackup\bin\bpdbm.exe';PID=0x1e50;THREAD=0x2884;VERSION=11.0.1.2960;API=ODBC;TIMEZONEADJUSTMENT=-300

I. 01/02 19:30:52. Connection terminated abnormally

I. 01/02 19:30:52. Disconnected SharedMemory client's AppInfo: IP=131.40.10.88;HOST=UNS40-88;OSUSER=SYSTEM;OS='Windows 2008R2 Build 7601 Service Pack 1';EXE='C:\Program Files\Veritas\NetBackup\bin\bpdbm.exe';PID=0x2eac;THREAD=0x2b84;VERSION=11.0.1.2960;API=ODBC;TIMEZONEADJUSTMENT=-300

I. 01/02 19:30:52. Connection terminated abnormally

I. 01/02 19:30:52. Disconnected SharedMemory client's AppInfo: IP=131.40.10.88;HOST=UNS40-88;OSUSER=SYSTEM;OS='Windows 2008R2 Build 7601 Service Pack 1';EXE='C:\Program Files\Veritas\NetBackup\bin\bpdbm.exe';PID=0x1590;THREAD=0x1ce8;VERSION=11.0.1.2960;API=ODBC;TIMEZONEADJUSTMENT=-300

I. 01/02 19:30:52. Connection terminated abnormally

I. 01/02 19:30:52. Disconnected SharedMemory client's AppInfo: IP=131.40.10.88;HOST=UNS40-88;OSUSER=SYSTEM;OS='Windows 2008R2 Build 7601 Service Pack 1';EXE='C:\Program Files\Veritas\NetBackup\bin\bpdbm.exe';PID=0x165c;THREAD=0xa6c;VERSION=11.0.1.2960;API=ODBC;TIMEZONEADJUSTMENT=-300

I. 01/02 19:30:52. Connection terminated abnormally

I. 01/02 19:30:52. Disconnected SharedMemory client's AppInfo: IP=131.40.10.88;HOST=UNS40-88;OSUSER=SYSTEM;OS='Windows 2008R2 Build 7601 Service Pack 1';EXE='C:\Program Files\Veritas\NetBackup\bin\bpdbm.exe';PID=0x1e24;THREAD=0x1ac8;VERSION=11.0.1.2960;API=ODBC;TIMEZONEADJUSTMENT=-300

I. 01/02 19:30:52. Connection terminated abnormally

I. 01/02 19:30:52. Disconnected SharedMemory client's AppInfo: IP=131.40.10.88;HOST=UNS40-88;OSUSER=SYSTEM;OS='Windows 2008R2 Build 7601 Service Pack 1';EXE='C:\Program Files\Veritas\NetBackup\bin\bpdbm.exe';PID=0x23b8;THREAD=0xdf4;VERSION=11.0.1.2960;API=ODBC;TIMEZONEADJUSTMENT=-300

I. 01/02 19:30:52. Connection terminated abnormally

I. 01/02 19:30:52. Disconnected SharedMemory client's AppInfo: IP=131.40.10.88;HOST=UNS40-88;OSUSER=SYSTEM;OS='Windows 2008R2 Build 7601 Service Pack 1';EXE='C:\Program Files\Veritas\NetBackup\bin\bpdbm.exe';PID=0xa64;THREAD=0x894;VERSION=11.0.1.2960;API=ODBC;TIMEZONEADJUSTMENT=-300

I. 01/02 19:30:52. Connection terminated abnormally

I. 01/02 19:30:52. Disconnected SharedMemory client's AppInfo: IP=131.40.10.88;HOST=UNS40-88;OSUSER=SYSTEM;OS='Windows 2008R2 Build 7601 Service Pack 1';EXE='C:\Program Files\Veritas\NetBackup\bin\bpdbm.exe';PID=0x29b4;THREAD=0x2bb0;VERSION=11.0.1.2960;API=ODBC;TIMEZONEADJUSTMENT=-300

I. 01/02 19:30:52. Connection terminated abnormally

I. 01/02 19:30:52. Disconnected SharedMemory client's AppInfo: IP=131.40.10.88;HOST=UNS40-88;OSUSER=SYSTEM;OS='Windows 2008R2 Build 7601 Service Pack 1';EXE='C:\Program Files\Veritas\NetBackup\bin\bpdbm.exe';PID=0x2220;THREAD=0x2aec;VERSION=11.0.1.2960;API=ODBC;TIMEZONEADJUSTMENT=-300

I. 01/02 19:30:52. Connection terminated abnormally

I. 01/02 19:30:52. Disconnected SharedMemory client's AppInfo: IP=131.40.10.88;HOST=UNS40-88;OSUSER=SYSTEM;OS='Windows 2008R2 Build 7601 Service Pack 1';EXE='C:\Program Files\Veritas\NetBackup\bin\nbim.exe';PID=0xae4;THREAD=0x758;VERSION=11.0.1.2960;API=ODBC;TIMEZONEADJUSTMENT=-300

I. 01/02 19:30:52. Connection terminated abnormally

I. 01/02 19:30:52. Disconnected SharedMemory client's AppInfo: IP=131.40.10.88;HOST=UNS40-88;OSUSER=SYSTEM;OS='Windows 2008R2 Build 7601 Service Pack 1';EXE='C:\Program Files\Veritas\NetBackup\bin\nbim.exe';PID=0xae4;THREAD=0x758;VERSION=11.0.1.2960;API=ODBC;TIMEZONEADJUSTMENT=-300

I. 01/02 19:30:52. Connection terminated abnormally

I. 01/02 19:30:52. Disconnected SharedMemory client's AppInfo: IP=131.40.10.88;HOST=UNS40-88;OSUSER=SYSTEM;OS='Windows 2008R2 Build 7601 Service Pack 1';EXE='C:\Program Files\Veritas\NetBackup\bin\nbim.exe';PID=0xae4;THREAD=0xca0;VERSION=11.0.1.2960;API=ODBC;TIMEZONEADJUSTMENT=-300

I. 01/02 19:30:52. Connection terminated abnormally

I. 01/02 19:30:52. Disconnected SharedMemory client's AppInfo: IP=131.40.10.88;HOST=UNS40-88;OSUSER=SYSTEM;OS='Windows 2008R2 Build 7601 Service Pack 1';EXE='C:\Program Files\Veritas\NetBackup\bin\nbim.exe';PID=0xae4;THREAD=0xca0;VERSION=11.0.1.2960;API=ODBC;TIMEZONEADJUSTMENT=-300

I. 01/02 19:30:52. Starting checkpoint of "NBDB" (NBDB.db) at Thu Jan 02 2014 19:30

I. 01/02 19:30:53. Finished checkpoint of "NBDB" (NBDB.db) at Thu Jan 02 2014 19:30

I. 01/02 19:30:54. Database "NBDB" (NBDB.db) stopped at Thu Jan 02 2014 19:30

I. 01/02 19:30:54. Starting checkpoint of "NBAZDB" (NBAZDB.db) at Thu Jan 02 2014 19:30

I. 01/02 19:30:54. Finished checkpoint of "NBAZDB" (NBAZDB.db) at Thu Jan 02 2014 19:30

I. 01/02 19:30:54. Database "NBAZDB" (NBAZDB.db) stopped at Thu Jan 02 2014 19:30

I. 01/02 19:30:54. Database "utility_db" (utility_db) stopped at Thu Jan 02 2014 19:30

I. 01/03 08:25:06. Starting database "utility_db" (utility_db) at Fri Jan 03 2014 08:25

I. 01/03 08:25:06. Database "utility_db" (utility_db) started at Fri Jan 03 2014 08:25

I. 01/03 08:25:06. Cache size adjusted to 197812K

I. 01/03 08:25:06. Starting database "NBDB" (C:\Progra

I am guessing it was the 90% of the space used that was causing this issue.

[NathanNieman]

Also seeing this from today

. 01/02 19:30:52. Starting checkpoint of "NBDB" (NBDB.db) at Thu Jan 02 2014 19:30

I. 01/02 19:30:53. Finished checkpoint of "NBDB" (NBDB.db) at Thu Jan 02 2014 19:30

I. 01/02 19:30:54. Database "NBDB" (NBDB.db) stopped at Thu Jan 02 2014 19:30

I. 01/02 19:30:54. Starting checkpoint of "NBAZDB" (NBAZDB.db) at Thu Jan 02 2014 19:30

I. 01/02 19:30:54. Finished checkpoint of "NBAZDB" (NBAZDB.db) at Thu Jan 02 2014 19:30

I. 01/02 19:30:54. Database "NBAZDB" (NBAZDB.db) stopped at Thu Jan 02 2014 19:30

I. 01/02 19:30:54. Database "utility_db" (utility_db) stopped at Thu Jan 02 2014 19:30

I. 01/03 08:25:06. Starting database "utility_db" (utility_db) at Fri Jan 03 2014 08:25

I. 01/03 08:25:06. Database "utility_db" (utility_db) started at Fri Jan 03 2014 08:25

I. 01/03 08:25:06. Cache size adjusted to 197812K

I. 01/03 08:25:06. Starting database "NBDB" (C:\Program Files\Veritas\NetBackupDB\data\NBDB.db) at Fri Jan 03 2014 08:25

I. 01/03 08:25:06. Performance warning: Database file "C:\Program Files\Veritas\NetBackupDB\data\NBDB.db" consists of 42 disk fragments

I. 01/03 08:25:07. Opening dbspace 'EMM_DATA' in file 'C:\Program Files\Veritas\NetBackupDB\data\EMM_DATA.db' for database 'NBDB'

I. 01/03 08:25:07. Performance warning: Database file "C:\Program Files\Veritas\NetBackupDB\data\EMM_DATA.db" consists of 29 disk fragments

I. 01/03 08:25:07. Opening dbspace 'EMM_INDEX' in file 'C:\Program Files\Veritas\NetBackupDB\data\EMM_INDEX.db' for database 'NBDB'

I. 01/03 08:25:07. Performance warning: Database file "C:\Program Files\Veritas\NetBackupDB\data\EMM_INDEX.db" consists of 3 disk fragments

I. 01/03 08:25:07. Opening dbspace 'DBM_DATA' in file 'C:\Program Files\Veritas\NetBackupDB\data\DBM_DATA.db' for database 'NBDB'

I. 01/03 08:25:07. Performance warning: Database file "C:\Program Files\Veritas\NetBackupDB\data\DBM_DATA.db" consists of 106 disk fragments

I. 01/03 08:25:07. Opening dbspace 'DBM_INDEX' in file 'C:\Program Files\Veritas\NetBackupDB\data\DBM_INDEX.db' for database 'NBDB'

I. 01/03 08:25:07. Performance warning: Database file "C:\Program Files\Veritas\NetBackupDB\data\DBM_INDEX.db" consists of 8 disk fragments

I. 01/03 08:25:07. Opening dbspace 'DARS_DATA' in file 'C:\Program Files\Veritas\NetBackupDB\data\DARS_DATA.db' for database 'NBDB'

I. 01/03 08:25:07. Performance warning: Database file "C:\Program Files\Veritas\NetBackupDB\data\DARS_DATA.db" consists of 2 disk fragments

I. 01/03 08:25:07. Opening dbspace 'DARS_INDEX' in file 'C:\Program Files\Veritas\NetBackupDB\data\DARS_INDEX.db' for database 'NBDB'

I. 01/03 08:25:07. Performance warning: Database file "C:\Program Files\Veritas\NetBackupDB\data\DARS_INDEX.db" consists of 2 disk fragments

I. 01/03 08:25:07. Opening dbspace 'SEARCH_DATA' in file 'C:\Program Files\Veritas\NetBackupDB\data\SEARCH_DATA.db' for database 'NBDB'

I. 01/03 08:25:07. Performance warning: Database file "C:\Program Files\Veritas\NetBackupDB\data\SEARCH_DATA.db" consists of 2 disk fragments

I. 01/03 08:25:07. Opening dbspace 'SEARCH_INDEX' in file 'C:\Program Files\Veritas\NetBackupDB\data\SEARCH_INDEX.db' for database 'NBDB'

I. 01/03 08:25:07. Performance warning: Database file "C:\Program Files\Veritas\NetBackupDB\data\SEARCH_INDEX.db" consists of 2 disk fragments

I. 01/03 08:25:07. Opening dbspace 'JOBD_DATA' in file 'C:\Program Files\Veritas\NetBackupDB\data\JOBD_DATA.db' for database 'NBDB'

I. 01/03 08:25:07. Performance warning: Database file "C:\Program Files\Veritas\NetBackupDB\data\JOBD_DATA.db" consists of 2 disk fragments

I. 01/03 08:25:07. Transaction log: NBDB.log

I. 01/03 08:25:07. Starting checkpoint of "NBDB" (NBDB.db) at Fri Jan 03 2014 08:25

I. 01/03 08:25:07. Finished checkpoint of "NBDB" (NBDB.db) at Fri Jan 03 2014 08:25

I. 01/03 08:25:07. Database "NBDB" (NBDB.db) started at Fri Jan 03 2014 08:25

I. 01/03 08:25:07. This database is licensed for use with:

I. 01/03 08:25:07.     Application: CommonRepository

I. 01/03 08:25:07.     Company: VERITAS

I. 01/03 08:25:12. Cache size adjusted to 320200K

I. 01/03 08:25:15. Cache size adjusted to 351700K

I. 01/03 08:25:17. Cache size adjusted to 360788K

[Deb_Wilmot]

I hate to ask - but what can you apply 7.5.0.7?  In that version parallelism (max_query_tasks) has been disabled by default.  This prevents some very bad behavior similar to what you're sseeing.

See  http://www.symantec.com/docs/TECH210414

** the above mentions the paralellism (max_query_tasks) issue in Item 3.

Otherthings I would suggest:

1.  Rebuild your database.

  from NetBackup\bin\ :
   -  Create a directory somewhere

    -  Run nbdb_backup -online /directory/you/created

     *** this is just to make sure that we have a good backup.

    -  Shutdown everything except for the Sybase / dbsrv11 process.

    -  run nbdb_unload -rebuild -verbose

    -  When that's finished run:  nbdb_admin -validate -full

  ** Hopefully that will look good.

If the problem persists:

2.  In C:\Program Files\Veritas\NetBackupDB\conf edit the server.conf file (after saving a copy of course!) and increase the cache size parameters as follows

  -c 500M

   -ch (approximately 30% of physical memory) 

      *** Example - if your physical memory is 16G, set this to 4G :   -ch 4G

  -c 500M

Then stop/restart netbackup.

*** See: http://www.symantec.com/docs/HOWTO67149

If the problem persists:

3.  In <install_path>\NetBackup\var\global add a file named emm.conf and put the following in it:

NUM_DB_CONNECTIONS=21
NUM_DB_BROWSE_CONNECTIONS=20
NUM_ORB_THREADS=31

And recycle.

Monitor the behavior after each change.   I hope that 7.5.0.7 will fix the issue so you won't have to do anything else <fingers crossed>

Good luck

[NathanNieman]

I will try the above and yes we will be update to .7 soon.

So far it seems to be a space issue, and I have been fighting to keep space there.

[revarooo]

One other thing, note this:

. 01/03 08:25:07. Opening dbspace 'DBM_DATA' in file 'C:\Program Files\Veritas\NetBackupDB\data\DBM_DATA.db' for database 'NBDB'

I. 01/03 08:25:07. Performance warning: Database file "C:\Program Files\Veritas\NetBackupDB\data\DBM_DATA.db" consists of 106 disk fragments

Stop NetBackup at some stage and defragment the C: drive too!

Performance will increase.

[NathanNieman]

Space was the issue.  I have since fixed this issue.  But have also rebuild the DB, and did a defrage, bo issues since than you.