10 dmpdaemons occupying all CPUs

Hello,

Need more information here.... can you paste the output of ps command, would like to see the exact process names running ..

# ps -ef |grep -i vx

--what is version of VxVM you are using ?
--If commands like vxdisk / vxdctl are hanging it would likely because of vxconfigd to be slow, when you say you can't run these commands, do you mean these commands never return or they are taking long time to return ?
--is this a cluster setup or a standalone server ?
-- if this is a standalone server, have you tried restarting vxconfigd daemon ? (vxconfigd -k -x syslog)

Gaurav

Thank you for your replay..

 ps -ef|grep vx
    root  622726       1   0   Jun 05      -  0:00 /usr/sbin/vxnotify -g vg_fxcfs01
    root  729204  794762   0   Jun 05      -  0:00 vxnotify -f -w 15
    root  737396  782472   0   Jun 05      -  0:00 vxnotify -C -w 15
    root  774272       1   0   Jun 05      -  0:00 /usr/bin/sh - /usr/lib/vxvm/bin/vxsited root
    root  782472       1   0   Jun 05      -  0:00 /usr/bin/sh - /usr/lib/vxvm/bin/vxcached root
    root  794762       1   0   Jun 05      -  0:00 /usr/bin/sh - /usr/lib/vxvm/bin/vxrelocd root
    root  831656  774272   0   Jun 05      -  0:00 vxnotify -m -w 15
    root  835744  905456   0   Jun 05      -  0:00 vxnotify -f -w 15
    root  839846  872624   0   Jun 05      -  0:00 vxnotify -m -w 15
    root  848042  868522   0   Jun 05      -  0:00 vxnotify -C -w 15
    root  860330  864424   0   Jun 05      -  0:00 /opt/VRTS/bin/vxnotify -w 15
    root  864424       1   0   Jun 05      -  0:06 /opt/VRTSperl/bin/perl /opt/VRTSmh/bin/vxvm-discovery.pl
    root  868522       1   0   Jun 05      -  0:00 /usr/bin/sh - /usr/lib/vxvm/bin/vxcached root
    root  872624       1   0   Jun 05      -  0:00 /usr/bin/sh - /usr/lib/vxvm/bin/vxsited root
    root  876722       1   0   Jun 05      -  0:34 /usr/sbin/vxdclid
    root  884926       1   0   Jun 05      -  0:08 /opt/VRTSobc/pal33/bin/vxpal -a actionagent -x
    root  905456       1   0   Jun 05      -  0:00 /usr/bin/sh - /usr/lib/vxvm/bin/vxrelocd root
    root  950524       1   0   Jun 05      -  0:37 /opt/VRTSob/bin/vxsvc -r /etc/vx/isis/Registry -e
    root  979024       1   0   Jun 05      -  5:24 /opt/VRTSdbcom/bin/vxdbd -d
    root 1028128       1   0   Jun 05      -  9:17 /opt/VRTSobc/pal33/bin/vxpal -a StorageAgent -x
    root 1065062       1   0   Jun 05      -  0:09 /opt/VRTSobc/pal33/bin/vxpal -a gridnode -x
    root 1077276       1   0   Jun 05      -  0:46 vxconfigd -x syslog -m boot
    root 1097764       1   0   Jun 05      -  0:21 /sbin/vxesd -x dmp_monitor_fabric=off
    root 1212502       1   0   Jun 05      -  0:00 /opt/VRTSvxfs/sbin/vxfsckd
    root 1339558 1487092   0 05:54:35      -  0:00 vxstat -g vg_fxcfs01 -i 5 -c 8640
    root 1626130 1446086   0 05:02:01      -  0:00 vxdisk -o alldgs list
    root 1630250 1487092   0 05:54:35      -  0:00 vxstat -g vg_isdb02 -i 5 -c 8640
    root 1671282 1511584   0 05:03:46      -  0:00 vxdisk -o alldgs list
    root 1720374 1487092   0 05:54:35      -  0:00 vxstat -g vg_oracfs01 -i 5 -c 8640
    root 1802464 1487092   0 05:54:35      -  0:00 vxstat -g vg_fx2fs01 -i 5 -c 8640
    root 1855542 1609842   0 06:43:55  pts/0  0:00 grep vx

1. VxVM version is 5.0.3.3(5.0MP3RP3)
2. Yes. I can say any vxdisk or vxdctl commands never return.. "vxdisk -o alldgs list" didn't return in 2 hours even though we have only 70 disks.
3. This is cluster setup

That system(called sysB) was rebooted few hours ago and we found some error messages about some disks for BCV. So we suspect BCV disks cause this problem. At the moment we had this problem, BCV volumes are mounted and then umount all BCV volumes on sysB. About 10 hours later, vx command issued like "vxdisk -o alldgs list" and never returned..

When I met this issue again. I'll post up again.. Thank you for your reply..