NB_dbsrv consuming 100% of CPU

The crash of the nbdb_backup is a bit disturbing.  A segmentation violation is due to some sort of memory allocation/deallocation or reference error.  Check the system error logs to see what it is reporting for that.

The NB_dbsrv process performs SQL actions against all of the NBU databases (NBDB, EMMDB, BMRDB). If it is consuming 100% of the CPU then a query action is looping rather badly.  This can cause lock-out conditions against other database connections.  This may be a result of some data corruption internal to the DB   Do a check of the server.log file to see if it displays any sort of DB access errors or problems. Always a good place to start.  The NB_dvsrv process logs it's activity to the file noted in its 'server.conf' file.  The location is specified by the "-o " option. Default location is "/usr/openv/db/log/server.log  ". 

To make a copy of the DB files without the use of the nbdb_backup command:

1. Stop all NBU services:  ../netbackup/bin/rc.kill_all

This will quiesce the databases and all connections to them.

2. Manually copy the contents of the directory to another location.  Based on your initial input:

cd /nbdb_catalog/data
cp * /nbdb_catalog/backup/backup1

Make any changes you want to the "server.conf"  file at this time as well. If something goes awry you can revert back by stopping all sercies and copying back the files to their original location.

3. Restart services:  ./netbackup/bin/rc.start_all

4. Perform the nbdb rebuild/reorganize noted previously.

5. The NB_dbsrv is multi-threaded and by default can have up to 20 open connections.  The "-gn ##" modifies this.  Larger values will help under specific loads.  However, stay below 40.

6. The "-ch ###" sets the high water mark of cached shared memory for the process. I consider the value "-ch 3G" to be a bit too aggressive as it can tie up overall system resources. The information submitted does not indicate memory resources of the server.  I try to limit this to "-ch 1G" under most circumstances.  To see if NB_dbsrv is actively allocating more shared memory, look in the '/usr/openv/db/log/server.log' file.

Let's see how things go after making the changes.