Unable to configure multile application for monitoring.
Hello all. I used Application HA 6.1 on windows server 2012 R2. I read this document( http://www.symantec.com/docs/TECH159846 ), then I performed the commands to add an application to an existing configuration. However, commandline put error messesage. How can I resolve this error? --------------- >XprtlC.exe -l https://localhost:5634/vcs/admin/createAppMonHBSG.pl -d params="<Cmd><ID>CreateVMWHBSG</ID><ServiceGroups><Name>SQLServer2012_SG</Name></ServiceGroups></Cmd>" <Cmd><ID>createAppMonHBSG</ID><ReturnCode>11</ReturnCode><Message>[SQLServer2012_SG] Group is not available in configuration. Failed to configure application heartbeat ervice group(s).</Message></Cmd> --------------- Thank you.Solved3.4KViews0likes1Comment
failed to bring up HeartBeat interface with GAB config
Dear Experts, I'm facing an issue here in cluster setup 8+2 I have managed to create 1+1 cluster and when trying to add the 3rd node I got stuck in gabconfig step below is the /var/adm/messages Jun 21 22:49:50 MM-BL2 bnxe: [ID 801725 kern.info] NOTICE: bnxe3: BnxeHwStartCore: Starting hardware (clients None) Jun 21 22:49:51 MM-BL2 bnxe: [ID 801725 kern.info] NOTICE: bnxe3: Phy capabilities (1) (supported 0x00000f50 0x00000000) (link_config 0x01000000 0x00000000) Jun 21 22:49:51 MM-BL2 bnxe: [ID 801725 kern.warning] WARNING: bnxe3: Phy 10000fdx requested but not supported Jun 21 22:49:51 MM-BL2 bnxe: [ID 801725 kern.info] NOTICE: bnxe3: Phy requesting link config of 0x00000006 Jun 21 22:49:51 MM-BL2 bnxe: [ID 801725 kern.info] NOTICE: bnxe3: 10Gb Full Duplex Rx Flow ON Tx Flow ON Link Up Jun 21 22:49:51 MM-BL2 bnxe: [ID 801725 kern.info] NOTICE: bnxe3: BnxeHwStartCore: Hardware started (clients None) Jun 21 22:49:51 MM-BL2 bnxe: [ID 801725 kern.info] NOTICE: bnxe3: BnxeHwStartL2: L2 started (clients L2) Jun 21 22:49:51 MM-BL2 bnxe: [ID 801725 kern.info] NOTICE: bnxe3: BnxeHwStopL2: Stopping L2 (clients L2) Jun 21 22:49:51 MM-BL2 bnxe: [ID 801725 kern.info] NOTICE: bnxe3: BnxeHwStopCore: Stopping hardware (clients None) Jun 21 22:49:51 MM-BL2 bnxe: [ID 801725 kern.info] NOTICE: bnxe3: Link Down Jun 21 22:49:51 MM-BL2 bnxe: [ID 801725 kern.info] NOTICE: bnxe3: BnxeHwStopCore: Hardware stopped (clients None) Jun 21 22:49:51 MM-BL2 bnxe: [ID 801725 kern.info] NOTICE: bnxe3: BnxeHwStopL2: L2 stopped (clients None) Jun 21 22:49:51 MM-BL2 bnxe: [ID 801725 kern.info] NOTICE: bnxe3: BnxeHwStartL2: Starting L2 (clients None) Jun 21 22:49:51 MM-BL2 bnxe: [ID 801725 kern.info] NOTICE: bnxe3: BnxeHwStartCore: Starting hardware (clients None) Jun 21 22:49:52 MM-BL2 bnxe: [ID 801725 kern.info] NOTICE: bnxe3: Phy capabilities (1) (supported 0x00000f50 0x00000000) (link_config 0x01000000 0x00000000) Jun 21 22:49:52 MM-BL2 bnxe: [ID 801725 kern.warning] WARNING: bnxe3: Phy 10000fdx requested but not supported Jun 21 22:49:52 MM-BL2 bnxe: [ID 801725 kern.info] NOTICE: bnxe3: Phy requesting link config of 0x00000006 Jun 21 22:49:52 MM-BL2 bnxe: [ID 801725 kern.info] NOTICE: bnxe3: 10Gb Full Duplex Rx Flow ON Tx Flow ON Link Up Jun 21 22:49:52 MM-BL2 bnxe: [ID 801725 kern.info] NOTICE: bnxe3: BnxeHwStartCore: Hardware started (clients None) Jun 21 22:49:52 MM-BL2 bnxe: [ID 801725 kern.info] NOTICE: bnxe3: BnxeHwStartL2: L2 started (clients L2) Jun 21 22:49:53 MM-BL2 bnxe: [ID 801725 kern.info] NOTICE: bnxe3: BnxeHwStopL2: Stopping L2 (clients L2) Jun 21 22:49:53 MM-BL2 bnxe: [ID 801725 kern.info] NOTICE: bnxe3: BnxeHwStopCore: Stopping hardware (clients None) Jun 21 22:49:53 MM-BL2 bnxe: [ID 801725 kern.info] NOTICE: bnxe3: Link Down Jun 21 22:49:53 MM-BL2 bnxe: [ID 801725 kern.info] NOTICE: bnxe3: BnxeHwStopCore: Hardware stopped (clients None) Jun 21 22:49:53 MM-BL2 bnxe: [ID 801725 kern.info] NOTICE: bnxe3: BnxeHwStopL2: L2 stopped (clients None) knowing that I have the same network profiles interfaces for all blades ( HP Proliant blades ) any ideas how this can be fixed ? BR Mahmoud2KViews0likes1Comment
VVR paused due to network disconnection
Hi all. I have a global cluster with 2 minicluster systems with solaris 10 installed (SPARC) primary: MIVDB01S - 172.22.8.132 secondary: MILDB08S - 10.66.11.148 I stopped the secondary (init 0) for 3 days, and after that I startup the secondary (boot from ok-prompt) and after 1 day I checked for the situation of the replication but: MIVDB01S root vradmin -g datadg printrvg datarvg Replicated Data Set: datarvg Primary: HostName: 172.22.8.132 <localhost> RvgName: datarvg DgName: datadg Secondary: HostName: 10.66.11.148 RvgName: datarvg DgName: datadg vxrlink -g datadg status datarlk Wed Jan 21 09:51:08 2015 VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink datarlk. DCM contains 874432 Kbytes (1%) of the Data Volume(s). vradmin -g datadg repstatus datarvg Replicated Data Set: datarvg Primary: Host name: 172.22.8.132 RVG name: datarvg DG name: datadg RVG state: enabled for I/O Data volumes: 1 VSets: 0 SRL name: srl_vol SRL size: 1.00 G Total secondaries: 1 Secondary: Host name: 10.66.11.148 RVG name: datarvg DG name: datadg Data status: consistent, behind Replication status: paused due to network disconnection (dcm resynchronization) Current mode: asynchronous Logging to: DCM (contains 874432 Kbytes) (SRL protection logging) Timestamp Information: N/A vxprint -Pl | grep flags flags: write enabled attached consistent disconnected asynchronous dcm_logging resync_paused ----- 1st solution MIVDB01S root vradmin -g datadg resync datarvg ----- 2nd solution Stop vradmin on secondary then on primary # /usr/sbin/vxstart_vvr stop Start vradmin on secondary then on primary # /usr/sbin/vxstart_vvr start Can you help me ?Solved6.8KViews0likes8Comments
Network adaptor unavailable
Hi All, After upgrading a cluster server from windows 2008 R1 to windows 2008 R2 the server cannot connect to the network anymore. The adaptor is still there and the heartbeats are working , but the principal network cannot be reached. The server is an HP and the veritas cluster software is 5.1 SP2. Any suggestions how to fix ? Since the portal from Symantec is unavailable to log a case , i need some help ?Solved4.7KViews1like2CommentsAppHA, SRM, VOM VBS Orchestration
Hello All, I'm working through a complex implementation and have hit a couple of snags along the way. Maybe people have worked on verious bits along the way and may be able to contribute to an overall solution The object of this endeavor is to create a flexible environment between 2 regional datacenters that gives us the ability to bring up a complex multi-tiered application pod at either location. Physicals meeting this HA criteria run VCS-Geo Cluster, some VMs run VCS-Geo Cluster with VVR while other VMs run AppHA..using SRM. They are all logically grouped within 2VOM VBS groups. The Home VBS consists of the AppHA VMs, Service Groups from the VCS nodes in the home datacenter while the DR VBS consists of the AppHA VMs and the matching DR geo cluster Service groups. AppHA seems to work as planned. VMs move through SRM to the datacenters and register fine in the SymHA consoles and report through VCenter with current status. My primary hurdle seems to be the control of those AppHA VMs from VOM VBS. The VBS control relies on a control host scan; and when that scan takes upwards of 6 hrs to run and update status. That time is better spent doing a manual startup sequence. Again VOM communication and dynamic updating of the Virtualization environment seem to be an issue. The VCS based elements are not an issue with my belief that they are based on a static infrastructure. I have successfully started up and shut down the home VBS, but once it moves to the opposing Datacenter...VOM can't find the VM and AppHA Consoles cant tie updates to records Has anyone done VBS orchistration of AppHA assets withe SRM handling transfers? VOM 5.0 VMWare 5.1 SRM 5.1 AppHA 6.0 VCS 5.1 SP2\ 6.0.1 Many Thanks, Many thanksSolved1.5KViews0likes1Comment
after mirroring plex, ssh service are not running
hi all, i have some productions server that are using SFHA Cluster. i need to mirroring volume to new storage using. (#vxassist -g DGname mirror VOLname alloc="vxdisk1 vxdisk2 . . . . vxdisk13") after 190 minutes ssh service cannot be access. i aborted the process, because im afraid it will affected another service group and make server get panic and then reboot. if there anyone have some resolve plan like we mirroring in subdisk level?1.1KViews0likes1CommentUnable to integrate guest components on the ApplicationHA console
Environment Two ESXi Servers = Esxi1 and Esxi2 version of Esxi Servers = 5.0 Vcenter OS = Windows2008R2(64bit) SAN = SAN attached on both Esxi Servers HA and DRS = enabled Application HA = 6.0a trial Application HA OS = rhel6.4z (64bit) Hello, i can't deploy a rhel6 ApplicationHA agent directly from the vcenter ApplicationHA Console tab because the package isn't installed. When i wan't to integrate it i can't do it. I think I haven't the correct media to perform this task. On the vcenter OS, i tried to integrate agent guest components D:\Symantec HA\ApplicationHA\Console\installer>CopyInstaller.bat "D:\temp\SYMC_ApplicationHA_6.0a_Linux" Error: Invalid source location. D:\Symantec HA\ApplicationHA\Console\installer>CopyInstaller.bat "D:\temp\VRTS_HA_Agent_Pack_2Q2013_Linux" Error: Invalid source location. I think a dvd2 media/iso is requested but i haven't this one. How to fix it ?Solved1.5KViews0likes1Comment
Application HA clustering has dropped a disk... :-/
Hi, After I completed the Application HA clustering of SQL2008 across two Windows 2008 R2 nodes, I found that the SQL installation'svirtual Backup disk (M: for reference) remained on the first node after I'd initiated a switch to the second node via the High Availability tab in the vSphere Client. The other disks re-attached to the second node ok. On closer inspection in Cluster Explorer on one of the nodes I discovered that the Mount Point for that disk was completely missing! I attempted to manually create the Mount Point and brought the resource online in the cluster on the first node, but attempting the switch operation again failed as the virtual disk failed to re-attach to the second node. How to fix without blatting the clustering / SQL installations? Thanks!Solved2.9KViews0likes10CommentsReccuring unmount of Diskgroups
Hello Experts, After finding that some of our DGs were unmounted, i have executed VCS clear command on resources of DG which were faulted and then made them online after it. Eventually, DGs were mounted and mountpoints were present to OS. After 2-3 days, we have noticed that these DGs were unmounted due to failing resources again. 2013/05/03 05:25:52 VCS ERROR V-16-10001-1004 (EMMDPD03) CVMVxconfigd:???:monitor:Core file saved once already 2013/05/03 05:25:53 VCS INFO V-16-2-13001 (EMMDPD03) Resource(cvm_vxconfigd): Output of the completed operation (monitor) 2013/05/03 05:25:53 VCS ERROR V-16-2-13067 (EMMDPD03) Agent is calling clean for resource(cvm_vxconfigd) because the resource became OFFLINE unexpectedly, on its own. 2013/05/03 05:25:54 VCS INFO V-16-2-13068 (EMMDPD03) Resource(cvm_vxconfigd) - clean completed successfully. 2013/05/03 05:25:54 VCS ERROR V-16-2-13073 (EMMDPD03) Resource(cvm_vxconfigd) became OFFLINE unexpectedly on its own. Agent is restarting (attempt number 1 of 5) the r esource. 2013/05/03 05:25:54 VCS WARNING V-16-10001-1006 (EMMDPD03) CVMVxconfigd:???:online:attempting to start vxconfigd 2013/05/03 05:26:13 VCS NOTICE V-16-2-13076 (EMMDPD03) Agent has successfully restarted resource(cvm_vxconfigd). 2013/05/03 11:59:12 VCS ERROR V-16-10001-1004 (EMMDPD03) CVMVxconfigd:???:monitor:Core file saved once already 2013/05/03 11:59:13 VCS INFO V-16-2-13001 (EMMDPD03) Resource(cvm_vxconfigd): Output of the completed operation (monitor) 2013/05/03 11:59:13 VCS ERROR V-16-2-13067 (EMMDPD03) Agent is calling clean for resource(cvm_vxconfigd) because the resource became OFFLINE unexpectedly, on its own. 2013/05/03 11:59:14 VCS INFO V-16-2-13068 (EMMDPD03) Resource(cvm_vxconfigd) - clean completed successfully. 2013/05/03 11:59:14 VCS ERROR V-16-2-13073 (EMMDPD03) Resource(cvm_vxconfigd) became OFFLINE unexpectedly on its own. Agent is restarting (attempt number 1 of 5) the r esource. 2013/05/03 11:59:14 VCS WARNING V-16-10001-1006 (EMMDPD03) CVMVxconfigd:???:online:attempting to start vxconfigd 2013/05/03 11:59:32 VCS NOTICE V-16-2-13076 (EMMDPD03) Agent has successfully restarted resource(cvm_vxconfigd). 2013/05/03 14:05:48 VCS INFO V-16-1-50135 User root fired command: hagrp -flush ServerGroup1_DG EMMDPD03 from localhost 2013/05/03 14:06:27 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE cfsmount2 EMMDPD03 from localhost 2013/05/03 14:06:41 VCS INFO V-16-1-50135 User root fired command: hagrp -offline ServerGroup1_DG EMMDPD03 from localhost 2013/05/03 14:06:41 VCS NOTICE V-16-1-10167 Initiating manual offline of group ServerGroup1_DG on system EMMDPD03 2013/05/03 14:07:03 VCS INFO V-16-1-50135 User root fired command: hares -clear cvmvoldg2 EMMDPD03 from localhost 2013/05/03 14:07:03 VCS INFO V-16-1-10307 Resource cvmvoldg2 (Owner: unknown, Group: ServerGroup1_DG) is offline on EMMDPD03 (Not initiated by VCS) 2013/05/03 14:07:09 VCS INFO V-16-1-50135 User root fired command: hares -clear cvmvoldg4 EMMDPD03 from localhost 2013/05/03 14:07:09 VCS INFO V-16-1-10307 Resource cvmvoldg4 (Owner: unknown, Group: ora_DG) is offline on EMMDPD03 (Not initiated by VCS) 2013/05/03 14:07:12 VCS INFO V-16-1-50135 User root fired command: hares -clear cvmvoldg5 EMMDPD03 from localhost 2013/05/03 14:07:12 VCS INFO V-16-1-10307 Resource cvmvoldg5 (Owner: unknown, Group: lic_DG) is offline on EMMDPD03 (Not initiated by VCS) 2013/05/03 14:07:15 VCS INFO V-16-1-50135 User root fired command: hares -clear cvmvoldg6 EMMDPD03 from localhost 2013/05/03 14:07:15 VCS INFO V-16-1-10307 Resource cvmvoldg6 (Owner: unknown, Group: lic_DG) is offline on EMMDPD03 (Not initiated by VCS) 2013/05/03 14:20:11 VCS INFO V-16-1-50135 User root fired command: hagrp -flush ServerGroup1_DG EMMDPD03 from localhost 2013/05/03 14:20:12 VCS INFO V-16-1-10306 Resource cvmvoldg2 (Owner: unknown, Group: ServerGroup1_DG) is offline on EMMDPD03 (Previous State = OFFLINE) 2013/05/03 14:28:05 VCS ERROR V-16-2-13027 (EMMDPD05) Resource(Server3) - monitor procedure did not complete within the expected time. 2013/05/03 14:28:09 VCS ERROR V-16-2-13027 (EMMDPD06) Resource(Server5) - monitor procedure did not complete within the expected time. 2013/05/03 14:28:09 VCS ERROR V-16-2-13027 (EMMDPD06) Resource(Server4) - monitor procedure did not complete within the expected time. 2013/05/03 14:28:11 VCS ERROR V-16-2-13027 (EMMDPD03) Resource(cfsmount2) - monitor procedure did not complete within the expected time. 2013/05/03 14:28:11 VCS ERROR V-16-2-13210 (EMMDPD03) Agent is calling clean for resource(cfsmount2) because 1 successive invocations of the monitor procedure did not bash-3.00$ cat /var/VRTSvcs/log/engine_A.log | grep 2013/05/ | more 2013/05/02 06:47:26 VCS ERROR V-16-10001-1004 (EMMDPD03) CVMVxconfigd:???:monitor:Core file saved once already 2013/05/02 06:47:27 VCS INFO V-16-2-13001 (EMMDPD03) Resource(cvm_vxconfigd): Output of the completed operation (monitor) 2013/05/02 06:47:27 VCS ERROR V-16-2-13067 (EMMDPD03) Agent is calling clean for resource(cvm_vxconfigd) because the resource became OFFLINE unexpectedly, on its own. 2013/05/02 06:47:28 VCS INFO V-16-2-13068 (EMMDPD03) Resource(cvm_vxconfigd) - clean completed successfully. 2013/05/02 06:47:28 VCS ERROR V-16-2-13073 (EMMDPD03) Resource(cvm_vxconfigd) became OFFLINE unexpectedly on its own. Agent is restarting (attempt number 1 of 5) the r esource. 2013/05/02 06:47:28 VCS WARNING V-16-10001-1006 (EMMDPD03) CVMVxconfigd:???:online:attempting to start vxconfigd 2013/05/02 06:47:51 VCS NOTICE V-16-2-13076 (EMMDPD03) Agent has successfully restarted resource(cvm_vxconfigd). 2013/05/03 05:25:52 VCS ERROR V-16-10001-1004 (EMMDPD03) CVMVxconfigd:???:monitor:Core file saved once already 2013/05/03 05:25:53 VCS INFO V-16-2-13001 (EMMDPD03) Resource(cvm_vxconfigd): Output of the completed operation (monitor) 2013/05/03 05:25:53 VCS ERROR V-16-2-13067 (EMMDPD03) Agent is calling clean for resource(cvm_vxconfigd) because the resource became OFFLINE unexpectedly, on its own. 2013/05/03 05:25:54 VCS INFO V-16-2-13068 (EMMDPD03) Resource(cvm_vxconfigd) - clean completed successfully. 2013/05/03 05:25:54 VCS ERROR V-16-2-13073 (EMMDPD03) Resource(cvm_vxconfigd) became OFFLINE unexpectedly on its own. Agent is restarting (attempt number 1 of 5) the r esource. 2013/05/03 05:25:54 VCS WARNING V-16-10001-1006 (EMMDPD03) CVMVxconfigd:???:online:attempting to start vxconfigd 2013/05/03 05:26:13 VCS NOTICE V-16-2-13076 (EMMDPD03) Agent has successfully restarted resource(cvm_vxconfigd). 2013/05/03 11:59:12 VCS ERROR V-16-10001-1004 (EMMDPD03) CVMVxconfigd:???:monitor:Core file saved once already 2013/05/03 11:59:13 VCS INFO V-16-2-13001 (EMMDPD03) Resource(cvm_vxconfigd): Output of the completed operation (monitor) 2013/05/03 11:59:13 VCS ERROR V-16-2-13067 (EMMDPD03) Agent is calling clean for resource(cvm_vxconfigd) because the resource became OFFLINE unexpectedly, on its own. 2013/05/03 11:59:14 VCS INFO V-16-2-13068 (EMMDPD03) Resource(cvm_vxconfigd) - clean completed successfully. 2013/05/03 11:59:14 VCS ERROR V-16-2-13073 (EMMDPD03) Resource(cvm_vxconfigd) became OFFLINE unexpectedly on its own. Agent is restarting (attempt number 1 of 5) the r esource. 2013/05/03 11:59:14 VCS WARNING V-16-10001-1006 (EMMDPD03) CVMVxconfigd:???:online:attempting to start vxconfigd 2013/05/03 11:59:32 VCS NOTICE V-16-2-13076 (EMMDPD03) Agent has successfully restarted resource(cvm_vxconfigd). 2013/05/03 14:05:48 VCS INFO V-16-1-50135 User root fired command: hagrp -flush ServerGroup1_DG EMMDPD03 from localhost 2013/05/03 14:06:27 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE cfsmount2 EMMDPD03 from localhost 2013/05/03 14:06:41 VCS INFO V-16-1-50135 User root fired command: hagrp -offline ServerGroup1_DG EMMDPD03 from localhost 2013/05/03 14:06:41 VCS NOTICE V-16-1-10167 Initiating manual offline of group ServerGroup1_DG on system EMMDPD03 2013/05/03 14:07:03 VCS INFO V-16-1-50135 User root fired command: hares -clear cvmvoldg2 EMMDPD03 from localhost 2013/05/03 14:07:03 VCS INFO V-16-1-10307 Resource cvmvoldg2 (Owner: unknown, Group: ServerGroup1_DG) is offline on EMMDPD03 (Not initiated by VCS) 2013/05/03 14:07:09 VCS INFO V-16-1-50135 User root fired command: hares -clear cvmvoldg4 EMMDPD03 from localhost 2013/05/03 14:07:09 VCS INFO V-16-1-10307 Resource cvmvoldg4 (Owner: unknown, Group: ora_DG) is offline on EMMDPD03 (Not initiated by VCS) 2013/05/03 14:07:12 VCS INFO V-16-1-50135 User root fired command: hares -clear cvmvoldg5 EMMDPD03 from localhost 2013/05/03 14:07:12 VCS INFO V-16-1-10307 Resource cvmvoldg5 (Owner: unknown, Group: lic_DG) is offline on EMMDPD03 (Not initiated by VCS) 2013/05/03 14:07:15 VCS INFO V-16-1-50135 User root fired command: hares -clear cvmvoldg6 EMMDPD03 from localhost 2013/05/03 14:07:15 VCS INFO V-16-1-10307 Resource cvmvoldg6 (Owner: unknown, Group: lic_DG) is offline on EMMDPD03 (Not initiated by VCS) 2013/05/03 14:20:11 VCS INFO V-16-1-50135 User root fired command: hagrp -flush ServerGroup1_DG EMMDPD03 from localhost 2013/05/03 14:20:12 VCS INFO V-16-1-10306 Resource cvmvoldg2 (Owner: unknown, Group: ServerGroup1_DG) is offline on EMMDPD03 (Previous State = OFFLINE) 2013/05/03 14:28:05 VCS ERROR V-16-2-13027 (EMMDPD05) Resource(Server3) - monitor procedure did not complete within the expected time. 2013/05/03 14:28:09 VCS ERROR V-16-2-13027 (EMMDPD06) Resource(Server5) - monitor procedure did not complete within the expected time. 2013/05/03 14:28:09 VCS ERROR V-16-2-13027 (EMMDPD06) Resource(Server4) - monitor procedure did not complete within the expected time. 2013/05/03 14:28:11 VCS ERROR V-16-2-13027 (EMMDPD03) Resource(cfsmount2) - monitor procedure did not complete within the expected time. 2013/05/03 14:28:11 VCS ERROR V-16-2-13210 (EMMDPD03) Agent is calling clean for resource(cfsmount2) because 1 successive invocations of the monitor procedure did not complete within the expected time. 2013/05/03 14:28:12 VCS INFO V-16-2-13068 (EMMDPD03) Resource(cfsmount2) - clean completed successfully. 2013/05/03 14:28:17 VCS ERROR V-16-2-13027 (EMMDPD07) Resource(Server1) - monitor procedure did not complete within the expected time. 2013/05/03 14:28:26 VCS ERROR V-16-2-13027 (EMMDPD04) Resource(Server2) - monitor procedure did not complete within the expected time. 2013/05/03 14:28:31 VCS ERROR V-16-2-13027 (EMMDPD04) Resource(fmmweb) - monitor procedure did not complete within the expected time. 2013/05/03 14:29:06 VCS ERROR V-16-2-13027 (EMMDPD07) Resource(licserv) - monitor procedure did not complete within the expected time. 2013/05/03 14:31:51 VCS ERROR V-16-2-13027 (EMMDPD05) Resource(cfsmount2) - monitor procedure did not complete within the expected time. 2013/05/03 14:31:51 VCS ERROR V-16-2-13210 (EMMDPD05) Agent is calling clean for resource(cfsmount2) because 1 successive invocations of the monitor procedure did not complete within the expected time. 2013/05/03 14:31:51 VCS INFO V-16-2-13026 (EMMDPD07) Resource(Server1) - monitor procedure finished successfully after failing to complete within the expected time for (2) consecutive times. 2013/05/03 14:31:52 VCS ERROR V-16-2-13027 (EMMDPD04) Resource(cfsmount2) - monitor procedure did not complete within the expected time. 2013/05/03 14:31:52 VCS ERROR V-16-2-13210 (EMMDPD04) Agent is calling clean for resource(cfsmount2) because 1 successive invocations of the monitor procedure did not complete within the expected time. 2013/05/03 14:31:52 VCS ERROR V-16-2-13027 (EMMDPD06) Resource(cfsmount2) - monitor procedure did not complete within the expected time. 2013/05/03 14:31:52 VCS ERROR V-16-2-13210 (EMMDPD06) Agent is calling clean for resource(cfsmount2) because 1 successive invocations of the monitor procedure did not complete within the expected time. 2013/05/03 14:31:52 VCS ERROR V-16-2-13027 (EMMDPD07) Resource(cfsmount2) - monitor procedure did not complete within the expected time. 2013/05/03 14:31:52 VCS ERROR V-16-2-13210 (EMMDPD07) Agent is calling clean for resource(cfsmount2) because 1 successive invocations of the monitor procedure did not complete within the expected time. 2013/05/03 14:31:52 VCS INFO V-16-2-13026 (EMMDPD04) Resource(fmmweb) - monitor procedure finished successfully after failing to complete within the expected time for (2) consecutive times. 2013/05/03 14:31:52 VCS INFO V-16-2-13068 (EMMDPD05) Resource(cfsmount2) - clean completed successfully. 2013/05/03 14:31:52 VCS INFO V-16-2-13082 (EMMDPD05) Resource(cfsmount2) recovered from fault, on its own. 2013/05/03 14:31:53 VCS INFO V-16-2-13026 (EMMDPD04) Resource(Server2) - monitor procedure finished successfully after failing to complete within the expected time for (2) consecutive times. 2013/05/03 14:31:53 VCS INFO V-16-2-13068 (EMMDPD04) Resource(cfsmount2) - clean completed successfully. 2013/05/03 14:31:53 VCS INFO V-16-2-13026 (EMMDPD06) Resource(Server5) - monitor procedure finished successfully after failing to complete within the expected time for (2) consecutive times. 2013/05/03 14:31:53 VCS INFO V-16-2-13026 (EMMDPD06) Resource(Server4) - monitor procedure finished successfully after failing to complete within the expected time for (2) consecutive times. 2013/05/03 14:31:53 VCS INFO V-16-2-13082 (EMMDPD04) Resource(cfsmount2) recovered from fault, on its own. 2013/05/03 14:31:53 VCS INFO V-16-2-13068 (EMMDPD06) Resource(cfsmount2) - clean completed successfully. 2013/05/03 14:31:53 VCS INFO V-16-2-13082 (EMMDPD06) Resource(cfsmount2) recovered from fault, on its own. 2013/05/03 14:31:53 VCS INFO V-16-2-13068 (EMMDPD07) Resource(cfsmount2) - clean completed successfully. 2013/05/03 14:31:53 VCS INFO V-16-2-13082 (EMMDPD07) Resource(cfsmount2) recovered from fault, on its own. 2013/05/03 14:31:53 VCS INFO V-16-2-13026 (EMMDPD05) Resource(Server3) - monitor procedure finished successfully after failing to complete within the expected time for (2) consecutive times. 2013/05/03 14:32:13 VCS INFO V-16-2-13026 (EMMDPD03) Resource(cfsmount2) - monitor procedure finished successfully after failing to complete within the expected time f or (3) consecutive times. 2013/05/03 14:32:13 VCS INFO V-16-2-13082 (EMMDPD03) Resource(cfsmount2) recovered from fault, on its own. 2013/05/03 14:34:05 VCS ERROR V-16-2-13027 (EMMDPD05) Resource(Server3) - monitor procedure did not complete within the expected time. 2013/05/03 14:34:10 VCS ERROR V-16-2-13027 (EMMDPD06) Resource(Server5) - monitor procedure did not complete within the expected time. 2013/05/03 14:34:10 VCS ERROR V-16-2-13027 (EMMDPD06) Resource(Server4) - monitor procedure did not complete within the expected time. 2013/05/03 14:34:13 VCS ERROR V-16-2-13027 (EMMDPD03) Resource(cfsmount2) - monitor procedure did not complete within the expected time. 2013/05/03 14:34:13 VCS ERROR V-16-2-13210 (EMMDPD03) Agent is calling clean for resource(cfsmount2) because 1 successive invocations of the monitor procedure did not complete within the expected time. 2013/05/03 14:34:14 VCS INFO V-16-2-13068 (EMMDPD03) Resource(cfsmount2) - clean completed successfully. 2013/05/03 14:34:16 VCS ERROR V-16-2-13027 (EMMDPD07) Resource(Server1) - monitor procedure did not complete within the expected time. 2013/05/03 14:34:26 VCS ERROR V-16-2-13027 (EMMDPD04) Resource(Server2) - monitor procedure did not complete within the expected time. 2013/05/03 14:34:30 VCS ERROR V-16-2-13027 (EMMDPD04) Resource(fmmweb) - monitor procedure did not complete within the expected time. 2013/05/03 14:34:40 VCS INFO V-16-1-10305 Resource cfsmount2 (Owner: unknown, Group: ServerGroup1_DG) is offline on EMMDPD03 (VCS initiated) 2013/05/03 14:34:40 VCS NOTICE V-16-1-10446 Group ServerGroup1_DG is offline on system EMMDPD03 2013/05/03 14:34:40 VCS INFO V-16-6-15002 (EMMDPD03) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/nfs_postoffline EMMDPD03 ServerGroup1_DG successfully 2013/05/03 14:34:40 VCS WARNING V-16-0 (EMMDPD03) hatrigger:POSTOFFLINE: EMMDPD03 ServerGroup1_DG 2013/05/03 14:34:41 VCS INFO V-16-6-15002 (EMMDPD03) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/postoffline EMMDPD03 ServerGroup1_DG successfully 2013/05/03 14:38:06 VCS INFO V-16-2-13026 (EMMDPD07) Resource(licserv) - monitor procedure finished successfully after failing to complete within the expected time for (2) consecutive times. 2013/05/03 14:42:43 VCS NOTICE V-16-1-10166 Initiating manual online of group ServerGroup1_DG on system EMMDPD03 2013/05/03 14:42:43 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group ServerGroup1_DG on node EMMDPD03 2013/05/03 14:42:43 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group ServerGroup1_DG on node EMMDPD04 2013/05/03 14:42:43 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group ServerGroup1_DG on node EMMDPD05 2013/05/03 14:42:43 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group ServerGroup1_DG on node EMMDPD06 2013/05/03 14:42:43 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group ServerGroup1_DG on node EMMDPD07 2013/05/03 14:42:43 VCS NOTICE V-16-1-10301 Initiating Online of Resource cvmvoldg2 (Owner: unknown, Group: ServerGroup1_DG) on System EMMDPD03 2013/05/03 14:42:43 VCS WARNING V-16-20007-1074 (EMMDPD03) CVMVolDg:cvmvoldg2:online:setup_vxnotify: old vxnotify of pid 11084 will be killed. my pid is 21744 2013/05/03 14:42:45 VCS INFO V-16-20007-1046 (EMMDPD03) CVMVolDg:cvmvoldg2:online:resource cvmvoldg2 is online 2013/05/03 14:43:17 VCS INFO V-16-1-10298 Resource cvmvoldg2 (Owner: unknown, Group: ServerGroup1_DG) is online on EMMDPD03 (VCS initiated) 2013/05/03 14:43:17 VCS NOTICE V-16-1-10301 Initiating Online of Resource cfsmount2 (Owner: unknown, Group: ServerGroup1_DG) on System EMMDPD03 2013/05/03 14:43:40 VCS INFO V-16-20011-5506 (EMMDPD03) CFSMount:cfsmount2:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,cluster Block Device /d ev/vx/dsk/bgw1dg/vol01 : MountPoint /var/opt/BGw/ServerGroup1 2013/05/03 14:43:41 VCS INFO V-16-1-10298 Resource cfsmount2 (Owner: unknown, Group: ServerGroup1_DG) is online on EMMDPD03 (VCS initiated) 2013/05/03 14:43:41 VCS NOTICE V-16-1-10447 Group ServerGroup1_DG is online on system EMMDPD03 2013/05/03 14:44:44 VCS NOTICE V-16-1-10166 Initiating manual online of group lic_DG on system EMMDPD03 2013/05/03 14:44:44 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group lic_DG on node EMMDPD03 2013/05/03 14:44:44 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group lic_DG on node EMMDPD04 2013/05/03 14:44:44 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group lic_DG on node EMMDPD05 2013/05/03 14:44:44 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group lic_DG on node EMMDPD06 2013/05/03 14:44:44 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group lic_DG on node EMMDPD07 2013/05/03 14:44:44 VCS NOTICE V-16-1-10301 Initiating Online of Resource cvmvoldg5 (Owner: unknown, Group: lic_DG) on System EMMDPD03 2013/05/03 14:44:44 VCS NOTICE V-16-1-10301 Initiating Online of Resource cvmvoldg6 (Owner: unknown, Group: lic_DG) on System EMMDPD03 2013/05/03 14:44:44 VCS WARNING V-16-20007-1074 (EMMDPD03) CVMVolDg:cvmvoldg5:online:setup_vxnotify: old vxnotify of pid 28916 will be killed. my pid is 13614 2013/05/03 14:44:44 VCS WARNING V-16-20007-1074 (EMMDPD03) CVMVolDg:cvmvoldg6:online:setup_vxnotify: old vxnotify of pid 1860 will be killed. my pid is 13654 2013/05/03 14:44:47 VCS INFO V-16-20007-1046 (EMMDPD03) CVMVolDg:cvmvoldg5:online:resource cvmvoldg5 is online 2013/05/03 14:44:47 VCS INFO V-16-20007-1046 (EMMDPD03) CVMVolDg:cvmvoldg6:online:resource cvmvoldg6 is online 2013/05/03 14:45:19 VCS INFO V-16-1-10298 Resource cvmvoldg6 (Owner: unknown, Group: lic_DG) is online on EMMDPD03 (VCS initiated) 2013/05/03 14:45:19 VCS NOTICE V-16-1-10301 Initiating Online of Resource cfsmount6 (Owner: unknown, Group: lic_DG) on System EMMDPD03 2013/05/03 14:45:19 VCS INFO V-16-1-10298 Resource cvmvoldg5 (Owner: unknown, Group: lic_DG) is online on EMMDPD03 (VCS initiated) 2013/05/03 14:45:19 VCS NOTICE V-16-1-10301 Initiating Online of Resource cfsmount5 (Owner: unknown, Group: lic_DG) on System EMMDPD03 2013/05/03 14:45:19 VCS INFO V-16-20011-5506 (EMMDPD03) CFSMount:cfsmount5:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,cluster Block Device /d ev/vx/dsk/lic1dg/vol01 : MountPoint /var/opt/sentinel 2013/05/03 14:45:19 VCS INFO V-16-20011-5506 (EMMDPD03) CFSMount:cfsmount6:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,cluster Block Device /d ev/vx/dsk/fmm1dg/vol01 : MountPoint /var/opt/mediation/fmmdb 2013/05/03 14:45:20 VCS INFO V-16-1-10298 Resource cfsmount6 (Owner: unknown, Group: lic_DG) is online on EMMDPD03 (VCS initiated) 2013/05/03 14:45:20 VCS INFO V-16-1-10298 Resource cfsmount5 (Owner: unknown, Group: lic_DG) is online on EMMDPD03 (VCS initiated) 2013/05/03 14:45:20 VCS NOTICE V-16-1-10447 Group lic_DG is online on system EMMDPD03 2013/05/03 14:45:48 VCS NOTICE V-16-1-10166 Initiating manual online of group ora_DG on system EMMDPD03 2013/05/03 14:45:48 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group ora_DG on node EMMDPD03 2013/05/03 14:45:48 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group ora_DG on node EMMDPD04 2013/05/03 14:45:48 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group ora_DG on node EMMDPD05 2013/05/03 14:45:48 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group ora_DG on node EMMDPD06 2013/05/03 14:45:48 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group ora_DG on node EMMDPD07 2013/05/03 14:45:48 VCS NOTICE V-16-1-10301 Initiating Online of Resource cvmvoldg4 (Owner: unknown, Group: ora_DG) on System EMMDPD03 2013/05/03 14:45:48 VCS WARNING V-16-20007-1074 (EMMDPD03) CVMVolDg:cvmvoldg4:online:setup_vxnotify: old vxnotify of pid 16612 will be killed. my pid is 26115 2013/05/03 14:45:50 VCS INFO V-16-20007-1046 (EMMDPD03) CVMVolDg:cvmvoldg4:online:resource cvmvoldg4 is online 2013/05/03 14:46:22 VCS INFO V-16-1-10298 Resource cvmvoldg4 (Owner: unknown, Group: ora_DG) is online on EMMDPD03 (VCS initiated) 2013/05/03 14:46:22 VCS NOTICE V-16-1-10301 Initiating Online of Resource cfsmount4 (Owner: unknown, Group: ora_DG) on System EMMDPD03 2013/05/03 14:46:22 VCS INFO V-16-20011-5506 (EMMDPD03) CFSMount:cfsmount4:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,cluster Block Device /d ev/vx/dsk/ora1dg/vol01 : MountPoint /var/opt/mediation/ora 2013/05/03 14:46:23 VCS INFO V-16-1-10298 Resource cfsmount4 (Owner: unknown, Group: ora_DG) is online on EMMDPD03 (VCS initiated) 2013/05/03 14:46:23 VCS NOTICE V-16-1-10447 Group ora_DG is online on system EMMDPD03 2013/05/03 15:03:18 VCS ERROR V-16-20007-1017 (EMMDPD03) CVMVolDg:cvmvoldg2:monitor:check_notify_status: can't stabilise vxstat 2013/05/03 15:25:22 VCS ERROR V-16-20007-1017 (EMMDPD03) CVMVolDg:cvmvoldg5:monitor:check_notify_status: can't stabilise vxstat 2013/05/03 15:25:22 VCS ERROR V-16-20007-1017 (EMMDPD03) CVMVolDg:cvmvoldg6:monitor:check_notify_status: can't stabilise vxstat 2013/05/03 15:26:21 VCS ERROR V-16-20007-1017 (EMMDPD03) CVMVolDg:cvmvoldg2:monitor:check_notify_status: can't stabilise vxstat 2013/05/03 15:26:22 VCS ERROR V-16-20007-1017 (EMMDPD03) CVMVolDg:cvmvoldg5:monitor:check_notify_status: can't stabilise vxstat 2013/05/03 15:27:23 VCS ERROR V-16-20007-1017 (EMMDPD03) CVMVolDg:cvmvoldg6:monitor:check_notify_status: can't stabilise vxstat May i ask what is causing it? When checking on the engine logs, resources are failing with out giving a root causeSolved2KViews0likes3Comments