Forum Discussion

anash7's avatar
anash7
Level 4
11 years ago

NBDB goes down automatically

Hello All,

 

My master is at NBU 7.5.0.6 and the nbdb goes down automatically. This has become periodic and the db goes down automatically almost once every week. Although I am able to manually recreate the database and get the nbdb up and running fine still I need the find the root cause and fix it permanently. Please help.

Backing Up
Backup and Recovery
NetBackup
Performance
Troubleshooting
Windows Server (2003-2008)
  • sri_vani's avatar
    sri_vani
    11 years ago

    like said in above excellent posts plz verify : \Veritas\NetBackupDB\log\server.log.

    the possible reasons wld be NBDB-corrupted,too large,running out of disk.

     

    1)If you see any "Fatal error: disk write failure /usr/openv/db/data/NBDB.log" in in the /usr/openv/db/log/server.log

    Please verify the filesystem that houses the database size by running df -k

    if it has good space

    2)verify ulimit setting for the filesize

     

    Refernce links:

    http://www.symantec.com/business/support/index?page=content&id=TECH143185

    http://www.symantec.com/business/support/index?page=content&id=TECH75308

     

11 Replies

Sort By
  • Nicolai's avatar
    Nicolai
    Moderator
    11 years ago

    Do you run regular catalog backups ?

    The catalog backup ensure the EMM database log files are truncated.

    http://www.symantec.com/docs/TECH42555

  • Marianne's avatar
    Marianne
    Level 6
    11 years ago

    Check EMM log and server log for errors.
    Use vxlogview to view oid 111.
    server log: \Veritas\NetBackupDB\log\server.log

    One possibility is that NBU disk is running out of space. NBU will stop databases to preven corruption.

    The above should also be logged in Event Viewer System or Application log.

    Also verify AntiVirus software excludes - McAfee is known for terminating NBU services.

  • sri_vani's avatar
    sri_vani
    Level 6
    11 years ago

    like said in above excellent posts plz verify : \Veritas\NetBackupDB\log\server.log.

    the possible reasons wld be NBDB-corrupted,too large,running out of disk.

     

    1)If you see any "Fatal error: disk write failure /usr/openv/db/data/NBDB.log" in in the /usr/openv/db/log/server.log

    Please verify the filesystem that houses the database size by running df -k

    if it has good space

    2)verify ulimit setting for the filesize

     

    Refernce links:

    http://www.symantec.com/business/support/index?page=content&id=TECH143185

    http://www.symantec.com/business/support/index?page=content&id=TECH75308

     

  • Deb_Wilmot's avatar
    Deb_Wilmot
    Level 6
    11 years ago

    Please call support and open a case.  We may need to disable parallelism in Sybase...  We know that this helps stability of NBDB..   There is an eeb we can provide for that. *** Note do not disable parallelism on OpsCenter servers; only NBU.

     

    Deb

  • anash7's avatar
    anash7
    Level 4
    11 years ago

    My catalog backup runs to D drive on the master and the backups initially to E drive, sufficient space is available on both.

     

    The last logs available on the server.log are as belows : No logs for 3 days till the time I worked on the steps for the nbdb_ping resolution.

     

    I. 11/09 01:33:01. Starting database "utility_db" (utility_db) at Sat Nov 09 2013 01:33

    I. 11/09 01:33:01. Database "utility_db" (utility_db) started at Sat Nov 09 2013 01:33

    I. 11/09 01:36:11. Connection terminated abnormally

    I. 11/09 01:36:11. Disconnected SharedMemory client's AppInfo: IP=10.8.2.180;HOST=ABCBKSMDS001;OSUSER=SYSTEM;OS='Windows 2008R2 Build 7601 Service Pack 1';EXE=D:\Veritas\NetBackup\bin\nbstserv.exe;PID=0x12f4;THREAD=0x1e20;VERSION=11.0.1.2960;API=ODBC;TIMEZONEADJUSTMENT=0

    I. 11/09 01:36:11. Connection terminated abnormally

    I. 11/09 01:36:11. Disconnected SharedMemory client's AppInfo: IP=10.8.2.180;HOST=ABCBKSMDS001;OSUSER=SYSTEM;OS='Windows 2008R2 Build 7601 Service Pack 1';EXE=D:\Veritas\NetBackup\bin\nbstserv.exe;PID=0x12f4;THREAD=0x1e20;VERSION=11.0.1.2960;API=ODBC;TIMEZONEADJUSTMENT=0

    I. 11/09 01:36:11. Connection terminated abnormally

    I. 11/09 01:36:11. Disconnected SharedMemory client's AppInfo: IP=10.8.2.180;HOST=ABCBKSMDS001;OSUSER=SYSTEM;OS='Windows 2008R2 Build 7601 Service Pack 1';EXE=D:\Veritas\NetBackup\bin\nbim.exe;PID=0x152c;THREAD=0x1f3c;VERSION=11.0.1.2960;API=ODBC;TIMEZONEADJUSTMENT=0

    I. 11/09 01:36:11. Connection terminated abnormally

    I. 11/09 01:36:11. Disconnected SharedMemory client's AppInfo: IP=10.8.2.180;HOST=ABCBKSMDS001;OSUSER=SYSTEM;OS='Windows 2008R2 Build 7601 Service Pack 1';EXE=D:\Veritas\NetBackup\bin\nbim.exe;PID=0x152c;THREAD=0x1ef0;VERSION=11.0.1.2960;API=ODBC;TIMEZONEADJUSTMENT=0

    I. 11/09 01:36:11. Connection terminated abnormally

    I. 11/09 01:36:11. Disconnected SharedMemory client's AppInfo: IP=10.8.2.180;HOST=ABCBKSMDS001;OSUSER=SYSTEM;OS='Windows 2008R2 Build 7601 Service Pack 1';EXE=D:\Veritas\NetBackup\bin\nbim.exe;PID=0x152c;THREAD=0x1ef0;VERSION=11.0.1.2960;API=ODBC;TIMEZONEADJUSTMENT=0

    I. 11/09 01:36:11. Connection terminated abnormally

    I. 11/09 01:36:11. Disconnected SharedMemory client's AppInfo: IP=10.8.2.180;HOST=ABCBKSMDS001;OSUSER=SYSTEM;OS='Windows 2008R2 Build 7601 Service Pack 1';EXE=D:\Veritas\NetBackup\bin\nbpem.exe;PID=0x77c;THREAD=0x1c18;VERSION=11.0.1.2960;API=ODBC;TIMEZONEADJUSTMENT=0

    I. 11/09 01:36:11. Connection terminated abnormally

    I. 11/09 01:36:11. Disconnected SharedMemory client's AppInfo: IP=10.8.2.180;HOST=ABCBKSMDS001;OSUSER=SYSTEM;OS='Windows 2008R2 Build 7601 Service Pack 1';EXE=D:\Veritas\NetBackup\bin\nbstserv.exe;PID=0x12f4;THREAD=0x1e20;VERSION=11.0.1.2960;API=ODBC;TIMEZONEADJUSTMENT=0

    I. 11/09 01:36:11. Connection terminated abnormally

    I. 11/09 01:36:11. Disconnected SharedMemory client's AppInfo: IP=10.8.2.180;HOST=ABCBKSMDS001;OSUSER=SYSTEM;OS='Windows 2008R2 Build 7601 Service Pack 1';EXE=D:\Veritas\NetBackup\bin\nbstserv.exe;PID=0x12f4;THREAD=0x1e20;VERSION=11.0.1.2960;API=ODBC;TIMEZONEADJUSTMENT=0

    I. 11/09 01:36:11. Connection terminated abnormally

    I. 11/09 01:36:11. Disconnected SharedMemory client's AppInfo: IP=10.8.2.180;HOST=ABCBKSMDS001;OSUSER=SYSTEM;OS='Windows 2008R2 Build 7601 Service Pack 1';EXE=D:\Veritas\NetBackup\bin\nbim.exe;PID=0x152c;THREAD=0x1f3c;VERSION=11.0.1.2960;API=ODBC;TIMEZONEADJUSTMENT=0

    I. 11/09 01:36:11. Starting checkpoint of "NBDB" (NBDB.db) at Sat Nov 09 2013 01:36

    I. 11/09 01:36:11. Finished checkpoint of "NBDB" (NBDB.db) at Sat Nov 09 2013 01:36

    I. 11/09 01:36:11. Database "NBDB" (NBDB.db) stopped at Sat Nov 09 2013 01:36

    I. 11/09 01:36:11. Database "utility_db" (utility_db) stopped at Sat Nov 09 2013 01:36

  • Deb_Wilmot's avatar
    Deb_Wilmot
    Level 6
    11 years ago

    Go to the <install_path>\Veritas\NetbackpDB\conf. 

    Make a copy of the server.conf

     

    Edit the server.conf file and set the -ch value to 30% of physical memory.

    I.e. if you have 16GB of RAM, set the value to 5G

    Recycle NBU.

    If that doesn't help I still would contact support and have a discussion with them about disabling parallelism.

    Feel free to use my name 'in vain'.

     

    Deb

  • anash7's avatar
    anash7
    Level 4
    11 years ago

    The issue got resolved after increasing the file system sizes. Most likely related to space but the file systems never reached 90% while i had the issue.

    Thanks for all your inputs.

  • Marianne's avatar
    Marianne
    Level 6
    11 years ago

    You were saying the following a couple of days ago:

    My catalog backup runs to D drive on the master and the backups initially to E drive, sufficient space is available on both.

    Not sure what exactly you meant, but it looks like the catalog backup is going to D-drive?

    It seems that NBU is also installed on the D-drive. This is probably the reason why drive ran out of space and stopped NBU to prevent corruption. 

    Backup should not be written to same drive letter where NBU is installed.

  • sri_vani's avatar
    sri_vani
    Level 6
    11 years ago

    Glad your issue got fixed.

    we have alerted u on same day on file system space details.

    u can successfully close this thread by Marking solution which helped u most.

    If more than one post helped  u, select request split solution and select all relavent posts.

  • anash7's avatar
    anash7
    Level 4
    11 years ago

    The catalog backup images goes to D drive and the netbackup is installed in D drive.  The  D drive is 500 GB in size and it never crossed 200 Gb , so not sure how it impacted the nbdb. But still I decreased the lower water mark to 10% since then i havent faced the issue.