bpdbm heavy CPU usage - a lot of time in kernel after upgrading to 7.1.0.2

Question

After upgrading to 7.1.0.2, constant 100% or near 100% CPU busy.
&nbsp;
The EEBs that we installed on the master are:
eebinstaller.2529202.1.solaris fixes a problem with incremental backup&nbsp;expired early.&nbsp;(This one has the bpdbm binary.)
eebinstaller.2535076.1.solaris is the NDMP bundle with other fixes.&nbsp;
We didn’t run v7.1.0.2 without the EEBs.&nbsp; We installed the EEBs immediately after upgrading from v7.0.1 (with a few EEBs) to v7.1 then v7.1.0.2 then adding two EEBs.&nbsp;
There isn’t a CPU spike as such – just a constant 100% or near 100% CPU busy.&nbsp;
&nbsp;
load averages: 10.18, 11.91, 12.38&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 10:37:22
182 processes: 172 sleeping, 6 running, 4 on cpu
CPU states:&nbsp;&nbsp;&nbsp;&nbsp; % idle,&nbsp;&nbsp;&nbsp;&nbsp; % user,&nbsp;&nbsp;&nbsp;&nbsp; % kernel,&nbsp;&nbsp;&nbsp;&nbsp; % iowait,&nbsp;&nbsp;&nbsp;&nbsp; % swap
Memory: 16G real, 867M free, 2763M swap in use, 15G swap free
&nbsp;
&nbsp;&nbsp; PID USERNAME LWP PRI NICE&nbsp; SIZE&nbsp;&nbsp; RES STATE&nbsp;&nbsp;&nbsp; TIME&nbsp;&nbsp;&nbsp; CPU COMMAND
&nbsp;23353 root&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp; 22&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp; 0K&nbsp;&nbsp;&nbsp; 0K run&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0:20&nbsp; 9.15% bpdbm
&nbsp;10528 root&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp; 22&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp; 0K&nbsp;&nbsp;&nbsp; 0K run&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 6:38&nbsp; 8.40% bpdbm
&nbsp;23852 root&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp; 31&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp; 0K&nbsp;&nbsp;&nbsp; 0K run&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0:08&nbsp; 5.83% bpdbm
&nbsp;23719 root&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp; 32&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp; 0K&nbsp;&nbsp;&nbsp; 0K run&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0:07&nbsp; 4.66% bpdbm
&nbsp;23815 root&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp; 32&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp; 0K&nbsp;&nbsp;&nbsp; 0K run&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0:04&nbsp; 3.39% bpdbm
&nbsp;24083 root&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp; 22&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp; 0K&nbsp;&nbsp;&nbsp; 0K cpu/2&nbsp;&nbsp;&nbsp; 0:02&nbsp; 2.40% bpdbm
&nbsp;&nbsp; 795 root&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 44&nbsp; 59&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp; 0K&nbsp;&nbsp;&nbsp; 0K sleep&nbsp; 162.1H&nbsp; 2.35% dbsrv11
&nbsp;21310 root&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp; 60&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp; 0K&nbsp;&nbsp;&nbsp; 0K run&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0:13&nbsp; 1.76% bpdbm
&nbsp;&nbsp; 942 root&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 10&nbsp; 59&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp; 0K&nbsp;&nbsp;&nbsp; 0K sleep&nbsp;&nbsp; 54.9H&nbsp; 1.75% nbrb
&nbsp;21313 root&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp; 48&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp; 0K&nbsp;&nbsp;&nbsp; 0K sleep&nbsp;&nbsp;&nbsp; 0:14&nbsp; 1.75% bpdbm
&nbsp;&nbsp; 939 root&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 58&nbsp; 59&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp; 0K&nbsp;&nbsp;&nbsp; 0K sleep&nbsp; 110.1H&nbsp; 1.59% nbemm
&nbsp;21457 root&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp; 59&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp; 0K&nbsp;&nbsp;&nbsp; 0K sleep&nbsp;&nbsp;&nbsp; 0:11&nbsp; 1.51% bpdbm
&nbsp;21390 root&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp; 59&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp; 0K&nbsp;&nbsp;&nbsp; 0K sleep&nbsp;&nbsp;&nbsp; 0:11&nbsp; 1.50% bpdbm
&nbsp;21393 root&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp; 49&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp; 0K&nbsp;&nbsp;&nbsp; 0K sleep&nbsp;&nbsp;&nbsp; 0:11&nbsp; 1.48% bpdbm
&nbsp;21423 root&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp; 54&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp; 0K&nbsp;&nbsp; &nbsp;0K sleep&nbsp;&nbsp;&nbsp; 0:11&nbsp; 1.48% bpdbm
&nbsp;
FYI., we have case opened with Symantec, they recommended us to restart the master server (we are not happy with this)
Please advice someone faced with similar issue / how to troubleshoot this.

revarooo · Answer

Hi rraja,
&nbsp;
Do you have bpdbm logging enabled, with VERBOSE set to high? This may give some answers as to what is going on with bpdbm.
&nbsp;

mkt · Answer

What backup jobs are running during this time?&nbsp; I know mis-configured oracle backups can cause bpdbm to spike as it runs long searches to determine backup ID is valid.&nbsp; %t missing for example can cause this:
Put %t at the end of the backup file name format. NetBackup uses the timestamp
	as part of its search criteria for catalog images. Without this timestamp,
	performance might degrade as the NetBackup catalog grows.
A RMAN cross check can cause this as well.&nbsp; Just some ideas but more details as to what is running would help here.

will_restore · Answer

"OK, turns out the problem was a script that went rogue." albeit an older version running Windows
https://www-secure.symantec.com/connect/forums/multiple-bpdbmexe-processes-using-all-cpu

marianne · Answer

Are there perhaps hung image cleanup jobs in Activity Monitor?
This TN does not mention high CPU, but probably worth a look: http://www.symantec.com/docs/TECH173159

r__ravi_kumar · Answer

If you are using NetBackup with NBAC turned ON , then you will see this. Get in touch with Symantec Suport. The problem has been identified as a NBAC issue and fix is available.

Forum Discussion

bpdbm heavy CPU usage - a lot of time in kernel after upgrading to 7.1.0.2

5 Replies

Related Content

Upgrade tomcat instance

Upgrade Linux Kernel on VCS installed

Help - Upgrade Netbackup for v10.4.0.1

Re: Vsphere 4.1 Windows Flash Backup Problem

8.2 Client Upgrade

Recent Discussions

MS-SharePoint policy restore error (2804) .

How to restore a backup

How to configure RBAC

command: bperror

10 years old netbackup appliance database service down, ssl certification out date