10 dmpdaemons occupying all CPUs

Question

10 dmpdaemons occupying all 4 CPUs but still can connect the server via telnet and excute commands except vx commands..&nbsp; like vxdmpadm, vxdisk, vxdctl...
	
	How can I jump&nbsp;out&nbsp;from this..

gaurav_s · Accepted Answer

Great... reboot most of the time resolves ...&nbsp; :-)

However, looking at output.. I can see many many of vxnotify are running since June 05, also other than normal processes, vxesd &amp; vxvm discovery are running as well...

Just a guess but I agree that issue would be at device discovery layer...

In future, before reboot, you can give a try to restart vxconfigd daemon, but make sure you have taken all cluster/CVM level precautions before doing so ..

If you know what happened on Jun 05, probably can give you some hint..

Gaurav

gaurav_s · Answer

Hello,

Need more information here.... can you paste the output of ps command, would like to see the exact process names running ..

# ps -ef |grep -i vx

--what is version of VxVM you are using ?
--If commands like vxdisk / vxdctl are hanging it would likely because of vxconfigd to be slow, when you say you can't run these commands, do you mean these commands never return or they are taking long time to return ?
--is this a cluster setup or a standalone server ?
-- if this is a standalone server, have you tried restarting vxconfigd daemon ? (vxconfigd -k -x syslog)

Gaurav

keunyoung · Answer

Thank you for your replay..

&nbsp;ps -ef|grep vx
&nbsp;&nbsp;&nbsp; root&nbsp; 622726&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Jun 05&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp; 0:00 /usr/sbin/vxnotify -g vg_fxcfs01
&nbsp;&nbsp;&nbsp; root&nbsp; 729204&nbsp; 794762&nbsp;&nbsp; 0&nbsp;&nbsp; Jun 05&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp; 0:00 vxnotify -f -w 15
&nbsp;&nbsp;&nbsp; root&nbsp; 737396&nbsp; 782472&nbsp;&nbsp; 0&nbsp;&nbsp; Jun 05&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp; 0:00 vxnotify -C -w 15
&nbsp;&nbsp;&nbsp; root&nbsp; 774272&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Jun 05&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp; 0:00 /usr/bin/sh - /usr/lib/vxvm/bin/vxsited root
&nbsp;&nbsp;&nbsp; root&nbsp; 782472&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Jun 05&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp; 0:00 /usr/bin/sh - /usr/lib/vxvm/bin/vxcached root
&nbsp;&nbsp;&nbsp; root&nbsp; 794762&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Jun 05&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp; 0:00 /usr/bin/sh - /usr/lib/vxvm/bin/vxrelocd root
&nbsp;&nbsp;&nbsp; root&nbsp; 831656&nbsp; 774272&nbsp;&nbsp; 0&nbsp;&nbsp; Jun 05&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp; 0:00 vxnotify -m -w 15
&nbsp;&nbsp;&nbsp; root&nbsp; 835744&nbsp; 905456&nbsp;&nbsp; 0&nbsp;&nbsp; Jun 05&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp; 0:00 vxnotify -f -w 15
&nbsp;&nbsp;&nbsp; root&nbsp; 839846&nbsp; 872624&nbsp;&nbsp; 0&nbsp;&nbsp; Jun 05&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp; 0:00 vxnotify -m -w 15
&nbsp;&nbsp;&nbsp; root&nbsp; 848042&nbsp; 868522&nbsp;&nbsp; 0&nbsp;&nbsp; Jun 05&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp; 0:00 vxnotify -C -w 15
&nbsp;&nbsp;&nbsp; root&nbsp; 860330&nbsp; 864424&nbsp;&nbsp; 0&nbsp;&nbsp; Jun 05&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp; 0:00 /opt/VRTS/bin/vxnotify -w 15
&nbsp;&nbsp;&nbsp; root&nbsp; 864424&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Jun 05&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp; 0:06 /opt/VRTSperl/bin/perl /opt/VRTSmh/bin/vxvm-discovery.pl
&nbsp;&nbsp;&nbsp; root&nbsp; 868522&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Jun 05&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp; 0:00 /usr/bin/sh - /usr/lib/vxvm/bin/vxcached root
&nbsp;&nbsp;&nbsp; root&nbsp; 872624&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Jun 05&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp; 0:00 /usr/bin/sh - /usr/lib/vxvm/bin/vxsited root
&nbsp;&nbsp;&nbsp; root&nbsp; 876722&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Jun 05&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp; 0:34 /usr/sbin/vxdclid
&nbsp;&nbsp;&nbsp; root&nbsp; 884926&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Jun 05&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp; 0:08 /opt/VRTSobc/pal33/bin/vxpal -a actionagent -x
&nbsp;&nbsp;&nbsp; root&nbsp; 905456&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Jun 05&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp; 0:00 /usr/bin/sh - /usr/lib/vxvm/bin/vxrelocd root
&nbsp;&nbsp;&nbsp; root&nbsp; 950524&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Jun 05&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp; 0:37 /opt/VRTSob/bin/vxsvc -r /etc/vx/isis/Registry -e
&nbsp;&nbsp;&nbsp; root&nbsp; 979024&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Jun 05&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp; 5:24 /opt/VRTSdbcom/bin/vxdbd -d
&nbsp;&nbsp;&nbsp; root 1028128&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Jun 05&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp; 9:17 /opt/VRTSobc/pal33/bin/vxpal -a StorageAgent -x
&nbsp;&nbsp;&nbsp; root 1065062&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Jun 05&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp; 0:09 /opt/VRTSobc/pal33/bin/vxpal -a gridnode -x
&nbsp;&nbsp;&nbsp; root 1077276&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Jun 05&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp; 0:46 vxconfigd -x syslog -m boot
&nbsp;&nbsp;&nbsp; root 1097764&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Jun 05&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp; 0:21 /sbin/vxesd -x dmp_monitor_fabric=off
&nbsp;&nbsp;&nbsp; root 1212502&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Jun 05&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp; 0:00 /opt/VRTSvxfs/sbin/vxfsckd
&nbsp;&nbsp;&nbsp; root 1339558 1487092&nbsp;&nbsp; 0 05:54:35&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp; 0:00 vxstat -g vg_fxcfs01 -i 5 -c 8640
&nbsp;&nbsp;&nbsp; root 1626130 1446086&nbsp;&nbsp; 0 05:02:01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp; 0:00 vxdisk -o alldgs list
&nbsp;&nbsp;&nbsp; root 1630250 1487092&nbsp;&nbsp; 0 05:54:35&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp; 0:00 vxstat -g vg_isdb02 -i 5 -c 8640
&nbsp;&nbsp;&nbsp; root 1671282 1511584&nbsp;&nbsp; 0 05:03:46&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp; 0:00 vxdisk -o alldgs list
&nbsp;&nbsp;&nbsp; root 1720374 1487092&nbsp;&nbsp; 0 05:54:35&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp; 0:00 vxstat -g vg_oracfs01 -i 5 -c 8640
&nbsp;&nbsp;&nbsp; root 1802464 1487092&nbsp;&nbsp; 0 05:54:35&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp; 0:00 vxstat -g vg_fx2fs01 -i 5 -c 8640
&nbsp;&nbsp;&nbsp; root 1855542 1609842&nbsp;&nbsp; 0 06:43:55&nbsp; pts/0&nbsp; 0:00 grep vx

1. VxVM version is 5.0.3.3(5.0MP3RP3)
2. Yes. I can say any vxdisk or vxdctl commands never return.. "vxdisk -o alldgs list" didn't return in 2 hours even&nbsp;though we have only 70 disks.
3. This is cluster setup

That system(called&nbsp;sysB)&nbsp;was rebooted few hours ago and we found some error messages about some disks for BCV. So we suspect BCV disks cause this problem. At the moment we had this problem, BCV volumes are mounted and then umount all BCV volumes on sysB. About 10 hours later, vx command issued&nbsp;like "vxdisk -o alldgs list" and never returned..

When I met this issue again. I'll post up again.. Thank you for your reply..

Forum Discussion

10 dmpdaemons occupying all CPUs

3 Replies

Related Content

High CPU load on primary server - java process

CPU vulnerability Meltdown

Drive Address 1 is already occupied by drive

NB_dbsrv consuming 100% of CPU

Re: Dedupe Query

Recent Discussions

DataDomain To StoreOnce Migration

VCS ERROR V-16-1-10205 Group Mediation_NFS is faulted on system com1em03

vxprivutil

URGENT disk oublic region recover

super block corrupt