bpdbm heavy CPU usage - a lot of time in kernel after upgrading to 7.1.0.2
After upgrading to 7.1.0.2, constant 100% or near 100% CPU busy.
The EEBs that we installed on the master are:
eebinstaller.2529202.1.solaris fixes a problem with incremental backup expired early. (This one has the bpdbm binary.)
eebinstaller.2535076.1.solaris is the NDMP bundle with other fixes.
We didn’t run v7.1.0.2 without the EEBs. We installed the EEBs immediately after upgrading from v7.0.1 (with a few EEBs) to v7.1 then v7.1.0.2 then adding two EEBs.
There isn’t a CPU spike as such – just a constant 100% or near 100% CPU busy.
load averages: 10.18, 11.91, 12.38 10:37:22
182 processes: 172 sleeping, 6 running, 4 on cpu
CPU states: % idle, % user, % kernel, % iowait, % swap
Memory: 16G real, 867M free, 2763M swap in use, 15G swap free
PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
23353 root 1 22 0 0K 0K run 0:20 9.15% bpdbm
10528 root 1 22 0 0K 0K run 6:38 8.40% bpdbm
23852 root 1 31 0 0K 0K run 0:08 5.83% bpdbm
23719 root 1 32 0 0K 0K run 0:07 4.66% bpdbm
23815 root 1 32 0 0K 0K run 0:04 3.39% bpdbm
24083 root 1 22 0 0K 0K cpu/2 0:02 2.40% bpdbm
795 root 44 59 0 0K 0K sleep 162.1H 2.35% dbsrv11
21310 root 1 60 0 0K 0K run 0:13 1.76% bpdbm
942 root 10 59 0 0K 0K sleep 54.9H 1.75% nbrb
21313 root 1 48 0 0K 0K sleep 0:14 1.75% bpdbm
939 root 58 59 0 0K 0K sleep 110.1H 1.59% nbemm
21457 root 1 59 0 0K 0K sleep 0:11 1.51% bpdbm
21390 root 1 59 0 0K 0K sleep 0:11 1.50% bpdbm
21393 root 1 49 0 0K 0K sleep 0:11 1.48% bpdbm
21423 root 1 54 0 0K 0K sleep 0:11 1.48% bpdbm
FYI., we have case opened with Symantec, they recommended us to restart the master server (we are not happy with this)
Please advice someone faced with similar issue / how to troubleshoot this.