cancel
Showing results for 
Search instead for 
Did you mean: 

10 dmpdaemons occupying all CPUs

keunyoung
Level 3
Partner Accredited

10 dmpdaemons occupying all 4 CPUs but still can connect the server via telnet and excute commands except vx commands..  like vxdmpadm, vxdisk, vxdctl...

How can I jump out from this..

1 ACCEPTED SOLUTION

Accepted Solutions

Gaurav_S
Moderator
Moderator
   VIP    Certified

Great... reboot most of the time resolves ...  :)

However, looking at output.. I can see many many of vxnotify are running since June 05, also other than normal processes, vxesd & vxvm discovery are running as well...

Just a guess but I agree that issue would be at device discovery layer...

In future, before reboot, you can give a try to restart vxconfigd daemon, but make sure you have taken all cluster/CVM level precautions before doing so ..

If you know what happened on Jun 05, probably can give you some hint..

Gaurav


View solution in original post

3 REPLIES 3

Gaurav_S
Moderator
Moderator
   VIP    Certified

Hello,

Need more information here.... can you paste the output of ps command, would like to see the exact process names running ..

# ps -ef |grep -i vx

--what is version of VxVM you are using ?
--If commands like vxdisk / vxdctl are hanging it would likely because of vxconfigd to be slow, when you say you can't run these commands, do you mean these commands never return or they are taking long time to return ?
--is this a cluster setup or a standalone server ?
-- if this is a standalone server, have you tried restarting vxconfigd daemon ? (vxconfigd -k -x syslog)

Gaurav

keunyoung
Level 3
Partner Accredited
Thank you for your replay..

 ps -ef|grep vx
    root  622726       1   0   Jun 05      -  0:00 /usr/sbin/vxnotify -g vg_fxcfs01
    root  729204  794762   0   Jun 05      -  0:00 vxnotify -f -w 15
    root  737396  782472   0   Jun 05      -  0:00 vxnotify -C -w 15
    root  774272       1   0   Jun 05      -  0:00 /usr/bin/sh - /usr/lib/vxvm/bin/vxsited root
    root  782472       1   0   Jun 05      -  0:00 /usr/bin/sh - /usr/lib/vxvm/bin/vxcached root
    root  794762       1   0   Jun 05      -  0:00 /usr/bin/sh - /usr/lib/vxvm/bin/vxrelocd root
    root  831656  774272   0   Jun 05      -  0:00 vxnotify -m -w 15
    root  835744  905456   0   Jun 05      -  0:00 vxnotify -f -w 15
    root  839846  872624   0   Jun 05      -  0:00 vxnotify -m -w 15
    root  848042  868522   0   Jun 05      -  0:00 vxnotify -C -w 15
    root  860330  864424   0   Jun 05      -  0:00 /opt/VRTS/bin/vxnotify -w 15
    root  864424       1   0   Jun 05      -  0:06 /opt/VRTSperl/bin/perl /opt/VRTSmh/bin/vxvm-discovery.pl
    root  868522       1   0   Jun 05      -  0:00 /usr/bin/sh - /usr/lib/vxvm/bin/vxcached root
    root  872624       1   0   Jun 05      -  0:00 /usr/bin/sh - /usr/lib/vxvm/bin/vxsited root
    root  876722       1   0   Jun 05      -  0:34 /usr/sbin/vxdclid
    root  884926       1   0   Jun 05      -  0:08 /opt/VRTSobc/pal33/bin/vxpal -a actionagent -x
    root  905456       1   0   Jun 05      -  0:00 /usr/bin/sh - /usr/lib/vxvm/bin/vxrelocd root
    root  950524       1   0   Jun 05      -  0:37 /opt/VRTSob/bin/vxsvc -r /etc/vx/isis/Registry -e
    root  979024       1   0   Jun 05      -  5:24 /opt/VRTSdbcom/bin/vxdbd -d
    root 1028128       1   0   Jun 05      -  9:17 /opt/VRTSobc/pal33/bin/vxpal -a StorageAgent -x
    root 1065062       1   0   Jun 05      -  0:09 /opt/VRTSobc/pal33/bin/vxpal -a gridnode -x
    root 1077276       1   0   Jun 05      -  0:46 vxconfigd -x syslog -m boot
    root 1097764       1   0   Jun 05      -  0:21 /sbin/vxesd -x dmp_monitor_fabric=off
    root 1212502       1   0   Jun 05      -  0:00 /opt/VRTSvxfs/sbin/vxfsckd
    root 1339558 1487092   0 05:54:35      -  0:00 vxstat -g vg_fxcfs01 -i 5 -c 8640
    root 1626130 1446086   0 05:02:01      -  0:00 vxdisk -o alldgs list
    root 1630250 1487092   0 05:54:35      -  0:00 vxstat -g vg_isdb02 -i 5 -c 8640
    root 1671282 1511584   0 05:03:46      -  0:00 vxdisk -o alldgs list
    root 1720374 1487092   0 05:54:35      -  0:00 vxstat -g vg_oracfs01 -i 5 -c 8640
    root 1802464 1487092   0 05:54:35      -  0:00 vxstat -g vg_fx2fs01 -i 5 -c 8640
    root 1855542 1609842   0 06:43:55  pts/0  0:00 grep vx

1. VxVM version is 5.0.3.3(5.0MP3RP3)
2. Yes. I can say any vxdisk or vxdctl commands never return.. "vxdisk -o alldgs list" didn't return in 2 hours even though we have only 70 disks.
3. This is cluster setup

That system(called sysB) was rebooted few hours ago and we found some error messages about some disks for BCV. So we suspect BCV disks cause this problem. At the moment we had this problem, BCV volumes are mounted and then umount all BCV volumes on sysB. About 10 hours later, vx command issued like "vxdisk -o alldgs list" and never returned..

When I met this issue again. I'll post up again.. Thank you for your reply..

Gaurav_S
Moderator
Moderator
   VIP    Certified

Great... reboot most of the time resolves ...  :)

However, looking at output.. I can see many many of vxnotify are running since June 05, also other than normal processes, vxesd & vxvm discovery are running as well...

Just a guess but I agree that issue would be at device discovery layer...

In future, before reboot, you can give a try to restart vxconfigd daemon, but make sure you have taken all cluster/CVM level precautions before doing so ..

If you know what happened on Jun 05, probably can give you some hint..

Gaurav