Forum Discussion

sebaquadri's avatar
sebaquadri
Level 3
12 years ago

NB_dbsrv consuming 100% of CPU

Hi all,

 

I'm experiencying the same issue as https://www-secure.symantec.com/connect/forums/heavy-load-emm-db-high-cpu-utilization... with NBU 7.1.0.4 on hpux master server and having a lot of jobs starting but going to queue state showing "Waiting in NetBackup scheduler work queue on server ..." I noted that NB_dbsrv is consuming 100% of CPU... I do not have a big EMM_DATA.db file as you can see below:

ebrbsnp05 >> find /nbdb_catalog -name EMM_DATA.db
/nbdb_catalog/data/EMM_DATA.db
ebrbsnp05 >> ls -l /nbdb_catalog/data/EMM_DATA.db
-rw-------   1 root       sys        51531776 Aug 15 12:26 /nbdb_catalog/data/EMM_DATA.db
ebrbsnp05 >> du -k /nbdb_catalog/data/EMM_DATA.db
50324   /nbdb_catalog/data/EMM_DATA.db
ebrbsnp05 >>

But I'm having 100 active jobs and 140 queued... all of the queued jobs are "Waiting in NetBackup scheduler work queue on server <master server name>"

I have opened a case with Symantec but it looks like they have not a clear understanding about the issue.

Have you finally got a solution for this case? How does the nbdb rebuild/reorganize that mph999 suggested above work? Did this help? Have you tried this?

Let me mention that I tried to perform this rebuild/reorganize yesterday, but I got an error while trying to take a backup of the nbdb vefore to do the first rebuild (I tried to take a backup of the nbdb just to be safe)... see below what I got...

I'm receiving the following error Segmentation fault (core dumped) when trying to perform a backup of the NDBD... could you please give us a hand?

Check below please…

ebrbsnp05 >> /usr/openv/netbackup/bin/nbdbms_start_stop start

ebrbsnp05 >> ../bpps -x

NB Processes

------------

    root 11363     1  0 11:22:32 ?         0:00 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7

 

 

MM Processes

------------

 

 

Shared Symantec Processes

-------------------------

    root 11324     1  0 11:22:27 ?         0:00 /opt/VRTSpbx/bin/pbx_exchange

ebrbsnp05 >> /usr/openv/db/bin/nbdb_ping

Database [NBDB] is alive and well on server [NB_ebrvmsnp01].

ebrbsnp05 >> mkdir /nbdb_catalog/backup

ebrbsnp05 >> /usr/openv/db/bin/nbdb_backup -dbn NBDB -online /nbdb_catalog/backup/backup1

Segmentation fault (core dumped)

ebrbsnp05 >>

 

Symantec is looking into some logs now... could anybody please help? Thanks in advance...

Let me add that I have also rebooted the master server yesterday, but the issue is still there... the reboot did not help. :(

Seba.

7.1.x and Earlier
Backup and Recovery
HP-UX
NetBackup
Troubleshooting
  • Jaime_Vazquez's avatar
    Jaime_Vazquez
    12 years ago

    The crash of the nbdb_backup is a bit disturbing.  A segmentation violation is due to some sort of memory allocation/deallocation or reference error.  Check the system error logs to see what it is reporting for that.

    The NB_dbsrv process performs SQL actions against all of the NBU databases (NBDB, EMMDB, BMRDB). If it is consuming 100% of the CPU then a query action is looping rather badly.  This can cause lock-out conditions against other database connections.  This may be a result of some data corruption internal to the DB   Do a check of the server.log file to see if it displays any sort of DB access errors or problems. Always a good place to start.  The NB_dvsrv process logs it's activity to the file noted in its 'server.conf' file.  The location is specified by the "-o " option. Default location is "/usr/openv/db/log/server.log  ". 

    To make a copy of the DB files without the use of the nbdb_backup command:

    1. Stop all NBU services:  ../netbackup/bin/rc.kill_all

    This will quiesce the databases and all connections to them.

    2. Manually copy the contents of the directory to another location.  Based on your initial input:

    cd /nbdb_catalog/data
    cp * /nbdb_catalog/backup/backup1

    Make any changes you want to the "server.conf"  file at this time as well. If something goes awry you can revert back by stopping all sercies and copying back the files to their original location.

    3. Restart services:  ./netbackup/bin/rc.start_all

    4. Perform the nbdb rebuild/reorganize noted previously.

    5. The NB_dbsrv is multi-threaded and by default can have up to 20 open connections.  The "-gn ##" modifies this.  Larger values will help under specific loads.  However, stay below 40.

    6. The "-ch ###" sets the high water mark of cached shared memory for the process. I consider the value "-ch 3G" to be a bit too aggressive as it can tie up overall system resources. The information submitted does not indicate memory resources of the server.  I try to limit this to "-ch 1G" under most circumstances.  To see if NB_dbsrv is actively allocating more shared memory, look in the '/usr/openv/db/log/server.log' file.

    Let's see how things go after making the changes.

     

  • Jaime_Vazquez's avatar
    Jaime_Vazquez
    Level 6
    12 years ago

    The crash of the nbdb_backup is a bit disturbing.  A segmentation violation is due to some sort of memory allocation/deallocation or reference error.  Check the system error logs to see what it is reporting for that.

    The NB_dbsrv process performs SQL actions against all of the NBU databases (NBDB, EMMDB, BMRDB). If it is consuming 100% of the CPU then a query action is looping rather badly.  This can cause lock-out conditions against other database connections.  This may be a result of some data corruption internal to the DB   Do a check of the server.log file to see if it displays any sort of DB access errors or problems. Always a good place to start.  The NB_dvsrv process logs it's activity to the file noted in its 'server.conf' file.  The location is specified by the "-o " option. Default location is "/usr/openv/db/log/server.log  ". 

    To make a copy of the DB files without the use of the nbdb_backup command:

    1. Stop all NBU services:  ../netbackup/bin/rc.kill_all

    This will quiesce the databases and all connections to them.

    2. Manually copy the contents of the directory to another location.  Based on your initial input:

    cd /nbdb_catalog/data
    cp * /nbdb_catalog/backup/backup1

    Make any changes you want to the "server.conf"  file at this time as well. If something goes awry you can revert back by stopping all sercies and copying back the files to their original location.

    3. Restart services:  ./netbackup/bin/rc.start_all

    4. Perform the nbdb rebuild/reorganize noted previously.

    5. The NB_dbsrv is multi-threaded and by default can have up to 20 open connections.  The "-gn ##" modifies this.  Larger values will help under specific loads.  However, stay below 40.

    6. The "-ch ###" sets the high water mark of cached shared memory for the process. I consider the value "-ch 3G" to be a bit too aggressive as it can tie up overall system resources. The information submitted does not indicate memory resources of the server.  I try to limit this to "-ch 1G" under most circumstances.  To see if NB_dbsrv is actively allocating more shared memory, look in the '/usr/openv/db/log/server.log' file.

    Let's see how things go after making the changes.

     

  • Anonymous's avatar
    Anonymous
    12 years ago

    @Seba

    Before a year we dump an HPUX master server to a AIX system because of memory, CPU and many CORBA problems.

    I suggest you to do the same. Go to an other platform, Linux is a good choice. 

  • sebaquadri's avatar
    sebaquadri
    Level 3
    12 years ago

    Changing from HPUX to AIX or Linux is not an option for me as I work on HP :)

    ......

    @mph999 I've already modifed some files, per symantec advise, and the issue has gone for almost 3 weeks... but last monday the issue was back and today I have 300 queued jobs and 150 active... I'm already using USE_HASH=1 as you mentioned...

    The changes I did 3 weeks ago, when the issue was "fixed" (for 3 weeks) are these:

    ============begin===============
    Install the following  EEB and add USE_HASH=1   in the  /usr/openv/var/global/emm.conf
    =====================================
     
     NetBackup_7.1.0.4  2762882
     
    Problem Description
    Due to the way Sybase processes the query involving backup-id field of EMM_ImageCopy and
    EMM_Image tables, the query takes a long time to execute.
     
    Installed Files
       /usr/openv/netbackup/bin/nbemm
     
    DOWNLOAD LINK: 
    ftp://iosupport:M3Q9r*SI0di7@ftp.entsupport.symantec.com/pub/support/outgoing/04757956/eebinstaller.2762882.1.hpia64
     
    =================================================================================
    =================================================================================
     
    (1)  NBU Config Tuning
    =================================================================================
    =================================================================================
     
     
    1.A.)
    Reduce Master / Media Server socket usage
     
    Move NBU internal VNETD socket connections on master servers to server loopback interface instead of using VNETD daemon  --Add the following line to  /usr/openv/netbackup/bp.conf
           CONNECT_OPTIONS = localhost 1 0 2
     
    No restart needed
     
    =================================================================================
     
    1.B.)
    Master Server
    Add more connections to the EMM database if the environment has 10+ Media Servers  and additional remote admin consoles / other increased backup activity.
     
    STOP NBU on the Master
     
    CREATE file--
          UNIX: /usr/openv/var/global/emm.conf
      
      Add contents--
    NUM_DB_BROWSE_CONNECTIONS=20
    NUM_DB_CONNECTIONS=21
    NUM_ORB_THREADS=35
    USE_HASH=1                 (This line Not needed for 7.5, 7.1 needs EEB's)
                      
     
    REFERENCE:    http://www.symantec.com/docs/TECH57277
    ----------------------------------------------------------------------------------------------
     
    Add more memory and threads to EMM DB due to large environment   (35 media servers).
     
     
    To change , edit file:
           Unix: /usr/openv/var/global/server.conf 
           Make a backup copy of the file before modifying
           
     
     
    CURRENT file settings
    --------------------------------
    # cat -s /usr/openv/var/global/server.conf  -n NB_ebrvmsnp01
       -x tcpip(LocalOnly=YES;ServerPort=13785)  -gp 4096 -gd DBA -gk DBA -gl DBA -ti 0 -c 25M -ch 500M -cl 25M -zl -os 1M  -o /usr/openv/db//log/server.log  -ud 
     
     
    MAKE CHANGES
    ---------------------------------
    Update the following parameters in the file
     
       -ch 500M      to new value    -ch 3G
       -gn 32         ( Add Missing parameter -   Add '-gn 32'  after  '-gl DB '  Afor more DB threads
       -m             ( Add Missing parameter -      Add    ' -m '       after ' -ud ' ) 
                          -m Helps to trim the NBDB.log file, this will be the default in NBU 7.5 
            
     
    AFTER CHANGES
    ---------------------------------
    # cat -s /usr/openv/var/global/server.conf  -n NB_ebrvmsnp01
       -x tcpip(LocalOnly=YES;ServerPort=13785)  -gp 4096 -gd DBA -gk DBA -gl DBA -gn 32 -ti 0 -c 25M -ch 3G -cl 25M -zl -os 1M  -o /usr/openv/db//log/server.log  -ud -m
     
     
    Restart services on the master 
     
     
    REFERENCE:
    http://www.symantec.com/docs/HOTO67149
                   
    =================================================================================
     
    1.C.)
     
     
    Verify minimum MASTER NBRB tuning file is in place
     
    CREATE file if missing--
          UNIX: /usr/openv/var/global/nbrb.conf
          Windows: <install_path>\Veritas\NetBackup\var\global\nbrb.conf
     
    Add contents--
    SECONDS_FOR_EVAL_LOOP_RELEASE = 180
    RESPECT_REQUEST_PRIORITY = 0
    DO_INTERMITTENT_UNLOADS = 1
     
     
    If file exists, do not adjust values to the ones noted above. As they may have been customized for your environment
     REFERENCE:    http://www.symantec.com/docs/TECH57942 
     
       
       
    =================================================================================
    =================================================================================
     OS  TUNING
    =================================================================================
    =================================================================================
     
    2)
    OS TUNING FOR NETBACKUP RESOURCES  -  NBU 6.X  /  7.X
    -------------------------------------
     
     
    2.A)
     
    Default SYSTEM File Descriptors too low for NetBackup Master
    This is a very critical setting for all masters on all OS's
     
     
    # /usr/bin/ulimit -a
    time(seconds)        unlimited
    file(blocks)         unlimited
    data(kbytes)         1048576
    stack(kbytes)        8192
    memory(kbytes)       unlimited
    coredump(blocks)     4194303
    nofiles(descriptors) 2048         <<<<<<<<<<<<< Set to 8192 minimum       
     
     
     
    Reference
     
       Minimum O/S ulimit settings on UNIX platforms 
        http://www.symantec.com/docs/TECH75332 
     
       Insufficient system file descriptors can cause the EMM_DATA.db file to grow very large.
       http://www.symantec.com/docs/TECH168846
     
    =======end====================
    Those settings helped, as I said before... but after 3 weeks the issue is back... :(

    My friends, I'll continue on the new post --> https://www-secure.symantec.com/connect/forums/nbd...

    Please continue on this new post.

    Thanks in advance!

    Seba.

  • sebaquadri's avatar
    sebaquadri
    Level 3
    12 years ago

    the error I got when tried to backup the nbdb was due a bad command syntaxis :) I should not specify a file name, just the path to save the backup... for instance...

    /usr/openv/db/bin/nbdb_backup -dbn NBDB -online /nbdb_catalog/backup/

    this one worked perfect :)

    I'm still having the issue... symantec suggested to make some more tunning but the issue is still there... I did the rebuild and reorganize of the NBDB and I changed the server.conf file and installed the following eeb patch (by symantec suggestion) but the issue has not gone :(

     
    1) removed the "-gn 32" from server.conf
    2) changed -cl 25M and -c 250 for -cl250M and -c250M so now my server.conf file looks like:
     -n NB_ebrvmsnp01
       -x tcpip(LocalOnly=YES;ServerPort=13785)  -gp 4096 -gd DBA -gk DBA -gl DBA -ti 0 -c 250M -ch 3G -cl 250M -zl -os 1M  -o /usr/ope
    nv/db//log/server.log  -ud -m
     
    3) Finally I installed the following eeb patch -> eebinstaller.2925327.1.hpia64
     
    But the issue is still there, NB_dbsrv is taking one CPU to 100% and have 100 queued jobs (100 active jobs, so 200 total)
     
    Do you have any other suggestion? This master server have 4 CPUs and tons of memory... check below...
     
    I cannot believe this is a lack of resources on the master server...
     
    System: ebrbsnp05                                     Fri Aug 16 18:25:19 2013
    Load averages: 0.34, 0.36, 0.40
    286 processes: 204 sleeping, 82 running
    Cpu states:
    CPU   LOAD   USER   NICE    SYS   IDLE  BLOCK  SWAIT   INTR   SSYS
     0    0.20   4.0%   0.0%   1.8%  94.2%   0.0%   0.0%   0.0%   0.0%
     2    0.18  18.5%   0.0%   0.6%  80.9%   0.0%   0.0%   0.0%   0.0%
     4    0.46  40.6%   0.0%   1.0%  58.4%   0.0%   0.0%   0.0%   0.0%
     6    0.54  42.5%   0.0%   0.2%  57.3%   0.0%   0.0%   0.0%   0.0%
    ---   ----  -----  -----  -----  -----  -----  -----  -----  -----
    avg   0.34  26.4%   0.0%   1.0%  72.6%   0.0%   0.0%   0.0%   0.0%
     
    System Page Size: 4Kbytes
    Memory: 2342944K (2010624K) real, 7187024K (6423580K) virtual, 34561896K free  Page# 1/13
     
    CPU TTY    PID USERNAME PRI NI   SIZE    RES STATE    TIME %WCPU  %CPU COMMAND
     0   ?   22393 root     152 20  3330M   156M run    334:55 101.45 101.28 NB_dbsrv
     0   ?   12766 root     128 20 80616K 40724K sleep    2:49  2.55  2.54 bpbkar
     2   ?   22680 root     152 20   224M 72764K run      5:37  1.14  1.14 nbjm
     4   ?   22642 root     152 20   804M   509M run     10:45  0.75  0.75 nbemm
     4   ?    4023 root     168 20 15844K  1332K sleep   14:01  0.52  0.52 utild
     2 pts/0 21332 root     178 20 11092K  1892K run      0:09  0.41  0.41 top
     2   ?   22677 root     154 20   133M 10020K sleep    2:39  0.39  0.39 bpdbm
     2   ?   22678 root     154 20 97084K 16664K sleep    3:28  0.32  0.32 bpjobd
     4   ?    3292 root     152 20 41924K 25228K run      2:44  0.23  0.23 python
     2   ?    1841 root     154 20 10992K  1156K sleep    5:47  0.22  0.21 sendmail:
     0   ?    2812 sfmdb    154 20 37652K  1904K sleep    6:43  0.21  0.21 postgres:
     6   ?    3404 root     152  0 52968K 11676K run      7:19  0.20  0.20 perfd
     4   ?   22645 root     152 20   283M 87228K run      6:59  0.16  0.16 nbrb
     6   ?    2064 cimsrvr  152 20 78608K 23296K run      5:42  0.12  0.12 cimservermain
     6   ?    2070 root     152 20   204M 56384K run      6:35  0.11  0.11 cimprovagt
     4   ?      80 root     152 20 48960K 43520K run     22:43  0.10  0.10 vxfsd
     0   ?    3348 root     127 20 62396K 18436K sleep    2:23  0.09  0.09 scopeux
     4   ?   22183 root     154 20 30324K  2272K sleep    0:38  0.08  0.08 pbx_exchange
     6   ?      82 root     152 20   360K   320K run      4:03  0.07  0.07 pm_schedcpu
     2   ?   19871 ed854663 152 20 25552K  2812K run      0:01  0.06  0.06 sshd:
     2   ?    3356 root     -16 20 38004K 13192K run      4:30  0.05  0.05 midaemon
     
     
    Let me mention that this is a dedicated master server and NOT a media server...
     
    Looking forward to hearing from you...
     
    Seba.
     
  • Jaime_Vazquez's avatar
    Jaime_Vazquez
    Level 6
    12 years ago

    Seba:

    I do not think I said anything about lack of resources for this problem. I tried to explain that a NB_dbsrv connection thread process appears to be just spinning inside a database.  Understand that NB_dbsrv handles connections from the NBU processes to all of the databases. That would be NBDB. EMMDB, and (if configured) BMRDB. NB_dbsrv is multi-threaded and as such will have multiple threads running associated within its process scope.  The "spining" thread, if one exists as I suspect,  will be operating strictly in memory, using shared memory space, so nothing will be stoping it from taking over a CPU time slice. For me that is the best guess.

    Did you look at the file "/usr/openv/db/log/server.log  " to see what is being written to it?  As I said before, the information may be valuable for this.

    Also, when do you see the CPU hit the 100% load value? Is it that way after stopping and then starting NBU processes?

     

  • sebaquadri's avatar
    sebaquadri
    Level 3
    12 years ago

    Hi Jaime,

     

    I do not see anything wrong on this log... I cut the lastest 100 lines (I have 103 queued jobs and 98 active currently)..

    On the other hand we do not have BMRDB as we do not use Bare Metal Restore option... most of our backups are just "standard" "windows" or "sap" or "oracle"...

    ebrbsnp05 >> tail -100 /usr/openv/db/log/server.log
    I. 08/16 14:30:26. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 14:30
    I. 08/16 14:30:26. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 14:30
    I. 08/16 14:43:20. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 14:43
    I. 08/16 14:43:20. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 14:43
    I. 08/16 14:45:27. Starting checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 14:45
    I. 08/16 14:45:27. Finished checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 14:45
    I. 08/16 14:55:05. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 14:55
    I. 08/16 14:55:06. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 14:55
    I. 08/16 15:02:36. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 15:02
    I. 08/16 15:02:36. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 15:02
    I. 08/16 15:05:29. Starting checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 15:05
    I. 08/16 15:05:29. Finished checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 15:05
    I. 08/16 15:10:51. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 15:10
    I. 08/16 15:10:51. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 15:10
    I. 08/16 15:19:28. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 15:19
    I. 08/16 15:19:28. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 15:19
    I. 08/16 15:25:31. Starting checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 15:25
    I. 08/16 15:25:31. Finished checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 15:25
    I. 08/16 15:30:18. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 15:30
    I. 08/16 15:30:18. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 15:30
    I. 08/16 15:45:32. Starting checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 15:45
    I. 08/16 15:45:32. Finished checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 15:45
    I. 08/16 15:45:46. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 15:45
    I. 08/16 15:45:47. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 15:45
    I. 08/16 15:56:01. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 15:56
    I. 08/16 15:56:01. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 15:56
    I. 08/16 16:05:34. Starting checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 16:05
    I. 08/16 16:05:34. Finished checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 16:05
    I. 08/16 16:06:30. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 16:06
    I. 08/16 16:06:31. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 16:06
    I. 08/16 16:12:40. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 16:12
    I. 08/16 16:12:40. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 16:12
    I. 08/16 16:20:33. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 16:20
    I. 08/16 16:20:33. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 16:20
    I. 08/16 16:25:36. Starting checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 16:25
    I. 08/16 16:25:36. Finished checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 16:25
    I. 08/16 16:31:46. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 16:31
    I. 08/16 16:31:47. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 16:31
    I. 08/16 16:43:57. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 16:43
    I. 08/16 16:43:57. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 16:43
    I. 08/16 16:44:01. Starting checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 16:44
    I. 08/16 16:44:01. Finished checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 16:44
    I. 08/16 16:44:01. Starting checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 16:44
    I. 08/16 16:44:01. Finished checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 16:44
    I. 08/16 16:44:01. Starting checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 16:44
    I. 08/16 16:44:01. Finished checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 16:44
    I. 08/16 16:44:01. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 16:44
    I. 08/16 16:44:01. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 16:44
    I. 08/16 16:44:03. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 16:44
    I. 08/16 16:44:03. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 16:44
    I. 08/16 16:44:03. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 16:44
    I. 08/16 16:44:03. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 16:44
    I. 08/16 16:57:17. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 16:57
    I. 08/16 16:57:17. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 16:57
    I. 08/16 17:04:03. Starting checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 17:04
    I. 08/16 17:04:03. Finished checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 17:04
    I. 08/16 17:12:53. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 17:12
    I. 08/16 17:12:54. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 17:12
    I. 08/16 17:24:05. Starting checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 17:24
    I. 08/16 17:24:05. Finished checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 17:24
    I. 08/16 17:26:43. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 17:26
    I. 08/16 17:26:43. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 17:26
    I. 08/16 17:31:04. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 17:31
    I. 08/16 17:31:05. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 17:31
    I. 08/16 17:42:24. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 17:42
    I. 08/16 17:42:24. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 17:42
    I. 08/16 17:44:06. Starting checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 17:44
    I. 08/16 17:44:06. Finished checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 17:44
    I. 08/16 17:56:07. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 17:56
    I. 08/16 17:56:07. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 17:56
    I. 08/16 18:04:08. Starting checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 18:04
    I. 08/16 18:04:08. Finished checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 18:04
    I. 08/16 18:16:52. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 18:16
    I. 08/16 18:16:53. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 18:16
    I. 08/16 18:24:10. Starting checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 18:24
    I. 08/16 18:24:10. Finished checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 18:24
    I. 08/16 18:33:16. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 18:33
    I. 08/16 18:33:16. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 18:33
    I. 08/16 18:43:54. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 18:43
    I. 08/16 18:43:54. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 18:43
    I. 08/16 18:44:12. Starting checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 18:44
    I. 08/16 18:44:12. Finished checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 18:44
    I. 08/16 18:55:13. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 18:55
    I. 08/16 18:55:13. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 18:55
    I. 08/16 19:04:14. Starting checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 19:04
    I. 08/16 19:04:14. Finished checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 19:04
    I. 08/16 19:04:25. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 19:04
    I. 08/16 19:04:25. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 19:04
    I. 08/16 19:16:39. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 19:16
    I. 08/16 19:16:39. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 19:16
    I. 08/16 19:24:15. Starting checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 19:24
    I. 08/16 19:24:15. Finished checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 19:24
    I. 08/16 19:30:20. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 19:30
    I. 08/16 19:30:20. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 19:30
    I. 08/16 19:38:43. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 19:38
    I. 08/16 19:38:43. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 19:38
    I. 08/16 19:44:17. Starting checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 19:44
    I. 08/16 19:44:17. Finished checkpoint of "NBAZDB" (NBAZDB.db) at Fri Aug 16 2013 19:44
    I. 08/16 19:53:42. Starting checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 19:53
    I. 08/16 19:53:42. Finished checkpoint of "NBDB" (NBDB.db) at Fri Aug 16 2013 19:53
    ebrbsnp05 >>
     
     
    Symantec suggested to increase the pfiles on the master and on the media servers...
    changed pfiles to 16264 on the master  (previous value was 8192) and to 8192 on all our media servers (old value was 2048)
     
    But the issue is still there :( no luck yet...
     
    Let me know which other log should I provide to get to the root cause? Thanks in advance...
  • mrmadej's avatar
    mrmadej
    Level 4
    12 years ago

    Hi,

    In my case it was the communication problem. I don't know why but on my master server, after switched the NBU, was running additionaly interface USB0 with assigned IP address. And after shut down this interface and restarted NBU all worked well.
    This was the cause of our high cpu and memory utilization.
     
    Have you problems with Java GUI like me?
     
    Regards
     
    Madej