2011/05/03 08:22:16 VCS INFO V-16-1-10307 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is offline on active_node_name (Not initiated by VCS) 2011/05/03 08:27:17 VCS INFO V-16-1-10299 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is online on active_node_name (Not initiated by VCS) 2011/05/03 08:27:17 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group NB653GRP on all nodes 2011/05/03 08:27:17 VCS NOTICE V-16-1-10447 Group NB653GRP is online on system active_node_name 2011/05/03 08:28:19 VCS INFO V-16-1-10307 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is offline on active_node_name (Not initiated by VCS) 2011/05/03 08:33:20 VCS INFO V-16-1-10299 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is online on active_node_name (Not initiated by VCS) 2011/05/03 08:33:20 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group NB653GRP on all nodes 2011/05/03 08:33:20 VCS NOTICE V-16-1-10447 Group NB653GRP is online on system active_node_name 2011/05/03 08:34:22 VCS INFO V-16-1-10307 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is offline on active_node_name (Not initiated by VCS) 2011/05/03 08:39:24 VCS INFO V-16-1-10299 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is online on active_node_name (Not initiated by VCS) 2011/05/03 08:39:24 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group NB653GRP on all nodes 2011/05/03 08:39:24 VCS NOTICE V-16-1-10447 Group NB653GRP is online on system active_node_name 2011/05/03 08:40:26 VCS INFO V-16-1-10307 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is offline on active_node_name (Not initiated by VCS) 2011/05/03 08:44:54 VCS INFO V-16-1-50135 User root fired command: hares - clear NB653GRP_RES_SERVER from localhost 2011/05/03 08:44:54 VCS INFO V-16-1-10307 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is offline on active_node_name (Not initiated by VCS) 2011/05/03 08:45:27 VCS INFO V-16-1-10299 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is online on active_node_name (Not initiated by VCS) 2011/05/03 08:45:27 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group NB653GRP on all nodes 2011/05/03 08:45:27 VCS NOTICE V-16-1-10447 Group NB653GRP is online on system active_node_name 2011/05/03 08:46:29 VCS INFO V-16-1-10307 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is offline on active_node_name (Not initiated by VCS) 2011/05/03 08:51:30 VCS INFO V-16-1-10299 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is online on active_node_name (Not initiated by VCS) 2011/05/03 08:51:30 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group NB653GRP on all nodes 2011/05/03 08:51:30 VCS NOTICE V-16-1-10447 Group NB653GRP is online on system active_node_name 2011/05/03 08:52:31 VCS INFO V-16-1-10307 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is offline on active_node_name (Not initiated by VCS) 2011/05/03 08:57:33 VCS INFO V-16-1-10299 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is online on active_node_name (Not initiated by VCS) 2011/05/03 08:57:33 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group NB653GRP on all nodes 2011/05/03 08:57:33 VCS NOTICE V-16-1-10447 Group NB653GRP is online on system active_node_name 2011/05/03 08:58:33 VCS INFO V-16-1-10307 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is offline on active_node_name (Not initiated by VCS) 2011/05/03 09:03:35 VCS INFO V-16-1-10299 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is online on active_node_name (Not initiated by VCS) 2011/05/03 09:03:35 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group NB653GRP on all nodes 2011/05/03 09:03:35 VCS NOTICE V-16-1-10447 Group NB653GRP is online on system active_node_name 2011/05/03 09:04:36 VCS INFO V-16-1-10307 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is offline on active_node_name (Not initiated by VCS) 2011/05/03 09:06:08 VCS INFO V-16-1-50135 User root fired command: hares - clear NB653GRP_RES_SERVER from localhost 2011/05/03 09:06:08 VCS INFO V-16-1-10307 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is offline on active_node_name (Not initiated by VCS) 2011/05/03 09:09:37 VCS INFO V-16-1-10299 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is online on active_node_name (Not initiated by VCS) 2011/05/03 09:09:37 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group NB653GRP on all nodes 2011/05/03 09:09:37 VCS NOTICE V-16-1-10447 Group NB653GRP is online on system active_node_name 2011/05/03 09:10:38 VCS INFO V-16-1-10307 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is offline on active_node_name (Not initiated by VCS) 2011/05/03 09:15:40 VCS INFO V-16-1-10299 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is online on active_node_name (Not initiated by VCS) 2011/05/03 09:15:40 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group NB653GRP on all nodes 2011/05/03 09:15:40 VCS NOTICE V-16-1-10447 Group NB653GRP is online on system active_node_name 2011/05/03 09:16:40 VCS INFO V-16-1-10307 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is offline on active_node_name (Not initiated by VCS) 2011/05/03 09:17:17 VCS INFO V-16-1-50135 User root fired command: hagrp - unfreeze NB653GRP from localhost 2011/05/03 09:17:35 VCS INFO V-16-1-50135 User root fired command: hagrp - offline NB653GRP active_node_name from localhost 2011/05/03 09:17:35 VCS NOTICE V-16-1-10167 Initiating manual offline of group NB653GRP on system active_node_name 2011/05/03 09:17:35 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NBU_MSTR5_MAESTRO (Owner: unknown, Group: NB653GRP) on System active_node_name 2011/05/03 09:17:35 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NBU_MSTR5_BKUP_IP_MNICB (Owner: unknown, Group: NB653GRP) on System active_node_name 2011/05/03 09:17:35 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NBU_MSTR5_CORP_IP_MNICB (Owner: unknown, Group: NB653GRP) on System active_node_name 2011/05/03 09:17:35 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NBU_MSTR5_NBUCATALOG_MNT (Owner: unknown, Group: NB653GRP) on System active_node_name 2011/05/03 09:17:35 VCS INFO V-16-1-10305 Resource NBU_MSTR5_BKUP_IP_MNICB (Owner: unknown, Group: NB653GRP) is offline on active_node_name (VCS initiated) 2011/05/03 09:17:35 VCS INFO V-16-1-10305 Resource NBU_MSTR5_CORP_IP_MNICB (Owner: unknown, Group: NB653GRP) is offline on active_node_name (VCS initiated) 2011/05/03 09:17:35 VCS INFO V-16-101010 (active_node_name) CISmaestro:???:???:### Entering Offline in NBU_MSTR5_MAESTRO ### 2011/05/03 09:17:47 VCS ERROR V-16-2-13064 (active_node_name) Agent is calling clean for resource(NBU_MSTR5_MAESTRO) because the resource is up even after offline completed. 2011/05/03 09:17:55 VCS INFO V-16-2-13068 (active_node_name) Resource(NBU_MSTR5_MAESTRO) - clean completed successfully. 2011/05/03 09:17:55 VCS ERROR V-16-2-13064 (active_node_name) Agent is calling clean for resource(NBU_MSTR5_NBUCATALOG_MNT) because the resource is up even after offline completed. 2011/05/03 09:17:56 VCS INFO V-16-1-10305 Resource NBU_MSTR5_MAESTRO (Owner: unknown, Group: NB653GRP) is offline on active_node_name (VCS initiated) 2011/05/03 09:17:56 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NBU_MSTR5_MAESTRO_MNT (Owner: unknown, Group: NB653GRP) on System active_node_name 2011/05/03 09:18:01 VCS INFO V-16-1-10305 Resource NBU_MSTR5_MAESTRO_MNT (Owner: unknown, Group: NB653GRP) is offline on active_node_name (VCS initiated) 2011/05/03 09:18:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NBU_MSTR5_MAESTRO_VOL (Owner: unknown, Group: NB653GRP) on System active_node_name 2011/05/03 09:18:03 VCS INFO V-16-1-10305 Resource NBU_MSTR5_MAESTRO_VOL (Owner: unknown, Group: NB653GRP) is offline on active_node_name (VCS initiated) 2011/05/03 09:18:05 VCS INFO V-16-10001-5524 (active_node_name) Mount:NBU_MSTR5_NBUCATALOG_MNT:clean:Forced unmount with -o force option of umount. 2011/05/03 09:18:05 VCS INFO V-16-2-13001 (active_node_name) Resource(NBU_MSTR5_NBUCATALOG_MNT): Output of the completed operation (clean) 2011/05/03 09:18:05 VCS INFO V-16-2-13068 (active_node_name) Resource(NBU_MSTR5_NBUCATALOG_MNT) - clean completed successfully. 2011/05/03 09:18:05 VCS INFO V-16-1-10305 Resource NBU_MSTR5_NBUCATALOG_MNT (Owner: unknown, Group: NB653GRP) is offline on active_node_name (VCS initiated) 2011/05/03 09:18:05 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NBU_MSTR5_NBUCATALOG_VOL (Owner: unknown, Group: NB653GRP) on System active_node_name 2011/05/03 09:18:07 VCS INFO V-16-1-10305 Resource NBU_MSTR5_NBUCATALOG_VOL (Owner: unknown, Group: NB653GRP) is offline on active_node_name (VCS initiated) 2011/05/03 09:18:07 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NBU_MSTR5_DG (Owner: unknown, Group: NB653GRP) on System active_node_name 2011/05/03 09:18:36 VCS INFO V-16-1-10305 Resource NBU_MSTR5_DG (Owner: unknown, Group: NB653GRP) is offline on active_node_name (VCS initiated) 2011/05/03 09:18:36 VCS NOTICE V-16-1-10446 Group NB653GRP is offline on system active_node_name 2011/05/03 09:22:53 VCS INFO V-16-1-50135 User root fired command: hares - clear NB653GRP_RES_SERVER active_node_name from localhost 2011/05/03 09:22:53 VCS INFO V-16-1-10307 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is offline on active_node_name (Not initiated by VCS) 2011/05/03 09:23:07 VCS INFO V-16-1-50135 User root fired command: hagrp - online NB653GRP active_node_name from localhost 2011/05/03 09:23:07 VCS NOTICE V-16-1-10166 Initiating manual online of group NB653GRP on system active_node_name 2011/05/03 09:23:07 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group NB653GRP on all nodes 2011/05/03 09:23:07 VCS NOTICE V-16-1-10301 Initiating Online of Resource NBU_MSTR5_DG (Owner: unknown, Group: NB653GRP) on System active_node_name 2011/05/03 09:23:07 VCS NOTICE V-16-1-10301 Initiating Online of Resource NBU_MSTR5_BKUP_IP_MNICB (Owner: unknown, Group: NB653GRP) on System active_node_name 2011/05/03 09:23:07 VCS NOTICE V-16-1-10301 Initiating Online of Resource NBU_MSTR5_CORP_IP_MNICB (Owner: unknown, Group: NB653GRP) on System active_node_name 2011/05/03 09:23:07 VCS INFO V-16-1-10298 Resource NBU_MSTR5_BKUP_IP_MNICB (Owner: unknown, Group: NB653GRP) is online on active_node_name (VCS initiated) 2011/05/03 09:23:07 VCS INFO V-16-1-10298 Resource NBU_MSTR5_CORP_IP_MNICB (Owner: unknown, Group: NB653GRP) is online on active_node_name (VCS initiated) 2011/05/03 09:23:09 VCS WARNING V-16-10001-1014 (active_node_name) DiskGroup:NBU_MSTR5_DG:online:Diskgroups will be imported without reservations 2011/05/03 09:23:18 VCS INFO V-16-1-50135 User root fired command: hagrp - unfreeze NB653GRP from localhost 2011/05/03 09:23:18 VCS INFO V-16-1-50135 User root fired command: hagrp - unfreeze NB653GRP from localhost 2011/05/03 09:23:19 VCS NOTICE V-16-10001-1009 (active_node_name) DiskGroup:NBU_MSTR5_DG:online:vxdg import succeeded on Disk Group NB653GRP1 2011/05/03 09:23:20 VCS NOTICE V-16-10001-1010 (active_node_name) DiskGroup:NBU_MSTR5_DG:online:Volumes in Disk Group NB653GRP1 are started. Any mirrors are updated in background 2011/05/03 09:23:22 VCS INFO V-16-1-10298 Resource NBU_MSTR5_DG (Owner: unknown, Group: NB653GRP) is online on active_node_name (VCS initiated) 2011/05/03 09:23:22 VCS NOTICE V-16-1-10301 Initiating Online of Resource NBU_MSTR5_NBUCATALOG_VOL (Owner: unknown, Group: NB653GRP) on System active_node_name 2011/05/03 09:23:22 VCS NOTICE V-16-1-10301 Initiating Online of Resource NBU_MSTR5_MAESTRO_VOL (Owner: unknown, Group: NB653GRP) on System active_node_name 2011/05/03 09:23:23 VCS NOTICE V-16-10001-11501 (active_node_name) Volume:NBU_MSTR5_NBUCATALOG_VOL:online:Volume opt-nbucatalog is started. Any mirrors are updated in background 2011/05/03 09:23:24 VCS INFO V-16-1-10298 Resource NBU_MSTR5_NBUCATALOG_VOL (Owner: unknown, Group: NB653GRP) is online on active_node_name (VCS initiated) 2011/05/03 09:23:24 VCS NOTICE V-16-1-10301 Initiating Online of Resource NBU_MSTR5_NBUCATALOG_MNT (Owner: unknown, Group: NB653GRP) on System active_node_name 2011/05/03 09:23:24 VCS NOTICE V-16-10001-11501 (active_node_name) Volume:NBU_MSTR5_MAESTRO_VOL:online:Volume opt-prod-maestro is started. Any mirrors are updated in background 2011/05/03 09:23:25 VCS INFO V-16-1-10298 Resource NBU_MSTR5_MAESTRO_VOL (Owner: unknown, Group: NB653GRP) is online on active_node_name (VCS initiated) 2011/05/03 09:23:25 VCS NOTICE V-16-1-10301 Initiating Online of Resource NBU_MSTR5_MAESTRO_MNT (Owner: unknown, Group: NB653GRP) on System active_node_name 2011/05/03 09:23:30 VCS INFO V-16-1-10298 Resource NBU_MSTR5_NBUCATALOG_MNT (Owner: unknown, Group: NB653GRP) is online on active_node_name (VCS initiated) 2011/05/03 09:23:30 VCS INFO V-16-1-10298 Resource NBU_MSTR5_MAESTRO_MNT (Owner: unknown, Group: NB653GRP) is online on active_node_name (VCS initiated) 2011/05/03 09:23:30 VCS NOTICE V-16-1-10301 Initiating Online of Resource NBU_MSTR5_MAESTRO (Owner: unknown, Group: NB653GRP) on System active_node_name 2011/05/03 09:24:14 VCS INFO V-16-1-10298 Resource NBU_MSTR5_MAESTRO (Owner: unknown, Group: NB653GRP) is online on active_node_name (VCS initiated) 2011/05/03 09:24:14 VCS NOTICE V-16-1-10301 Initiating Online of Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) on System active_node_name 2011/05/03 09:24:36 VCS INFO V-16-2-13001 (active_node_name) Resource(NB653GRP_RES_SERVER): Output of the completed operation (online) 2011/05/03 09:24:40 VCS INFO V-16-1-10298 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is online on active_node_name (VCS initiated) 2011/05/03 09:24:40 VCS NOTICE V-16-1-10447 Group NB653GRP is online on system active_node_name 2011/05/03 09:25:01 VCS INFO V-16-1-50135 User root fired command: hagrp - freeze NB653GRP from localhost 2011/05/04 07:05:58 VCS INFO V-16-1-10307 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is offline on active_node_name (Not initiated by VCS) 2011/05/04 07:10:59 VCS INFO V-16-1-10299 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is online on active_node_name (Not initiated by VCS) 2011/05/04 07:10:59 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group NB653GRP on all nodes 2011/05/04 07:10:59 VCS NOTICE V-16-1-10447 Group NB653GRP is online on system active_node_name 2011/05/04 07:12:01 VCS INFO V-16-1-10307 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is offline on active_node_name (Not initiated by VCS) 2011/05/04 07:17:02 VCS INFO V-16-1-10299 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is online on active_node_name (Not initiated by VCS) 2011/05/04 07:17:02 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group NB653GRP on all nodes 2011/05/04 07:17:02 VCS NOTICE V-16-1-10447 Group NB653GRP is online on system active_node_name 2011/05/04 07:18:04 VCS INFO V-16-1-10307 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is offline on active_node_name (Not initiated by VCS) 2011/05/04 07:23:06 VCS INFO V-16-1-10299 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is online on active_node_name (Not initiated by VCS) 2011/05/04 07:23:06 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group NB653GRP on all nodes 2011/05/04 07:23:06 VCS NOTICE V-16-1-10447 Group NB653GRP is online on system active_node_name 2011/05/04 07:24:06 VCS INFO V-16-1-10307 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is offline on active_node_name (Not initiated by VCS) 2011/05/04 07:29:08 VCS INFO V-16-1-10299 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is online on active_node_name (Not initiated by VCS) 2011/05/04 07:29:08 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group NB653GRP on all nodes 2011/05/04 07:29:08 VCS NOTICE V-16-1-10447 Group NB653GRP is online on system active_node_name 2011/05/04 07:30:09 VCS INFO V-16-1-10307 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is offline on active_node_name (Not initiated by VCS) 2011/05/04 07:34:41 VCS INFO V-16-1-50135 User root fired command: hagrp - online NB653GRP active_node_name from localhost 2011/05/04 07:35:11 VCS INFO V-16-1-10299 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is online on active_node_name (Not initiated by VCS) 2011/05/04 07:35:11 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group NB653GRP on all nodes 2011/05/04 07:35:11 VCS NOTICE V-16-1-10447 Group NB653GRP is online on system active_node_name 2011/05/04 07:36:11 VCS INFO V-16-1-10307 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is offline on active_node_name (Not initiated by VCS) 2011/05/04 07:41:13 VCS INFO V-16-1-10299 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is online on active_node_name (Not initiated by VCS) 2011/05/04 07:41:13 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group NB653GRP on all nodes 2011/05/04 07:41:13 VCS NOTICE V-16-1-10447 Group NB653GRP is online on system active_node_name 2011/05/04 07:42:13 VCS INFO V-16-1-10307 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is offline on active_node_name (Not initiated by VCS) 2011/05/04 07:43:17 VCS INFO V-16-1-50135 User root fired command: hagrp - unfreeze NB653GRP from localhost 2011/05/04 07:43:31 VCS INFO V-16-1-50135 User root fired command: hagrp - offline NB653GRP active_node_name from localhost 2011/05/04 07:43:31 VCS NOTICE V-16-1-10167 Initiating manual offline of group NB653GRP on system active_node_name 2011/05/04 07:43:31 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NBU_MSTR5_MAESTRO (Owner: unknown, Group: NB653GRP) on System active_node_name 2011/05/04 07:43:31 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NBU_MSTR5_BKUP_IP_MNICB (Owner: unknown, Group: NB653GRP) on System active_node_name 2011/05/04 07:43:31 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NBU_MSTR5_CORP_IP_MNICB (Owner: unknown, Group: NB653GRP) on System active_node_name 2011/05/04 07:43:31 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NBU_MSTR5_NBUCATALOG_MNT (Owner: unknown, Group: NB653GRP) on System active_node_name 2011/05/04 07:43:31 VCS INFO V-16-1-10305 Resource NBU_MSTR5_BKUP_IP_MNICB (Owner: unknown, Group: NB653GRP) is offline on active_node_name (VCS initiated) 2011/05/04 07:43:31 VCS INFO V-16-1-10305 Resource NBU_MSTR5_CORP_IP_MNICB (Owner: unknown, Group: NB653GRP) is offline on active_node_name (VCS initiated) 2011/05/04 07:43:31 VCS INFO V-16-101010 (active_node_name) CISmaestro:???:???:### Entering Offline in NBU_MSTR5_MAESTRO ### 2011/05/04 07:43:44 VCS INFO V-16-1-10305 Resource NBU_MSTR5_MAESTRO (Owner: unknown, Group: NB653GRP) is offline on active_node_name (VCS initiated) 2011/05/04 07:43:44 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NBU_MSTR5_MAESTRO_MNT (Owner: unknown, Group: NB653GRP) on System active_node_name 2011/05/04 07:43:53 VCS ERROR V-16-2-13064 (active_node_name) Agent is calling clean for resource(NBU_MSTR5_NBUCATALOG_MNT) because the resource is up even after offline completed. 2011/05/04 07:44:01 VCS INFO V-16-10001-5513 (active_node_name) Mount:NBU_MSTR5_MAESTRO_MNT:offline:Unmounted /dev/vx/dsk/NB653GRP1/opt- prod-maestro using SIGKILL 2011/05/04 07:44:02 VCS INFO V-16-1-10305 Resource NBU_MSTR5_MAESTRO_MNT (Owner: unknown, Group: NB653GRP) is offline on active_node_name (VCS initiated) 2011/05/04 07:44:02 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NBU_MSTR5_MAESTRO_VOL (Owner: unknown, Group: NB653GRP) on System active_node_name 2011/05/04 07:44:03 VCS INFO V-16-10001-5524 (active_node_name) Mount:NBU_MSTR5_NBUCATALOG_MNT:clean:Forced unmount with -o force option of umount. 2011/05/04 07:44:03 VCS INFO V-16-2-13001 (active_node_name) Resource(NBU_MSTR5_NBUCATALOG_MNT): Output of the completed operation (clean) 2011/05/04 07:44:03 VCS INFO V-16-2-13068 (active_node_name) Resource(NBU_MSTR5_NBUCATALOG_MNT) - clean completed successfully. 2011/05/04 07:44:03 VCS INFO V-16-1-10305 Resource NBU_MSTR5_NBUCATALOG_MNT (Owner: unknown, Group: NB653GRP) is offline on active_node_name (VCS initiated) 2011/05/04 07:44:03 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NBU_MSTR5_NBUCATALOG_VOL (Owner: unknown, Group: NB653GRP) on System active_node_name 2011/05/04 07:44:04 VCS INFO V-16-1-10305 Resource NBU_MSTR5_MAESTRO_VOL (Owner: unknown, Group: NB653GRP) is offline on active_node_name (VCS initiated) 2011/05/04 07:44:05 VCS INFO V-16-1-10305 Resource NBU_MSTR5_NBUCATALOG_VOL (Owner: unknown, Group: NB653GRP) is offline on active_node_name (VCS initiated) 2011/05/04 07:44:05 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NBU_MSTR5_DG (Owner: unknown, Group: NB653GRP) on System active_node_name 2011/05/04 07:44:30 VCS INFO V-16-1-10305 Resource NBU_MSTR5_DG (Owner: unknown, Group: NB653GRP) is offline on active_node_name (VCS initiated) 2011/05/04 07:44:30 VCS NOTICE V-16-1-10446 Group NB653GRP is offline on system active_node_name 2011/05/04 07:52:58 VCS INFO V-16-1-50135 User root fired command: hagrp - clear NB653GRP from localhost 2011/05/04 07:52:58 VCS INFO V-16-1-10307 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is offline on active_node_name (Not initiated by VCS) 2011/05/04 07:53:20 VCS INFO V-16-1-50135 User root fired command: hares - clear NB653GRP_RES_SERVER active_node_name from localhost 2011/05/04 07:53:25 VCS INFO V-16-1-50135 User root fired command: hares - online NB653GRP_RES_SERVER active_node_name from localhost 2011/05/04 07:53:25 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group NB653GRP on all nodes 2011/05/04 07:53:25 VCS NOTICE V-16-1-10301 Initiating Online of Resource NBU_MSTR5_BKUP_IP_MNICB (Owner: unknown, Group: NB653GRP) on System active_node_name 2011/05/04 07:53:25 VCS NOTICE V-16-1-10301 Initiating Online of Resource NBU_MSTR5_CORP_IP_MNICB (Owner: unknown, Group: NB653GRP) on System active_node_name 2011/05/04 07:53:25 VCS NOTICE V-16-1-10301 Initiating Online of Resource NBU_MSTR5_DG (Owner: unknown, Group: NB653GRP) on System active_node_name 2011/05/04 07:53:26 VCS INFO V-16-1-10298 Resource NBU_MSTR5_CORP_IP_MNICB (Owner: unknown, Group: NB653GRP) is online on active_node_name (VCS initiated) 2011/05/04 07:53:26 VCS INFO V-16-1-10298 Resource NBU_MSTR5_BKUP_IP_MNICB (Owner: unknown, Group: NB653GRP) is online on active_node_name (VCS initiated) 2011/05/04 07:53:28 VCS WARNING V-16-10001-1014 (active_node_name) DiskGroup:NBU_MSTR5_DG:online:Diskgroups will be imported without reservations 2011/05/04 07:53:34 VCS INFO V-16-1-50135 User root fired command: hagrp - online NB653GRP active_node_name from localhost 2011/05/04 07:53:34 VCS NOTICE V-16-1-10166 Initiating manual online of group NB653GRP on system active_node_name 2011/05/04 07:53:34 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group NB653GRP on all nodes 2011/05/04 07:53:37 VCS NOTICE V-16-10001-1009 (active_node_name) DiskGroup:NBU_MSTR5_DG:online:vxdg import succeeded on Disk Group NB653GRP1 2011/05/04 07:53:37 VCS NOTICE V-16-10001-1010 (active_node_name) DiskGroup:NBU_MSTR5_DG:online:Volumes in Disk Group NB653GRP1 are started. Any mirrors are updated in background 2011/05/04 07:53:40 VCS INFO V-16-1-10298 Resource NBU_MSTR5_DG (Owner: unknown, Group: NB653GRP) is online on active_node_name (VCS initiated) 2011/05/04 07:53:40 VCS NOTICE V-16-1-10301 Initiating Online of Resource NBU_MSTR5_NBUCATALOG_VOL (Owner: unknown, Group: NB653GRP) on System active_node_name 2011/05/04 07:53:40 VCS NOTICE V-16-1-10301 Initiating Online of Resource NBU_MSTR5_MAESTRO_VOL (Owner: unknown, Group: NB653GRP) on System active_node_name 2011/05/04 07:53:40 VCS NOTICE V-16-10001-11501 (active_node_name) Volume:NBU_MSTR5_NBUCATALOG_VOL:online:Volume opt-nbucatalog is started. Any mirrors are updated in background 2011/05/04 07:53:41 VCS INFO V-16-1-10298 Resource NBU_MSTR5_NBUCATALOG_VOL (Owner: unknown, Group: NB653GRP) is online on active_node_name (VCS initiated) 2011/05/04 07:53:41 VCS NOTICE V-16-1-10301 Initiating Online of Resource NBU_MSTR5_NBUCATALOG_MNT (Owner: unknown, Group: NB653GRP) on System active_node_name 2011/05/04 07:53:42 VCS NOTICE V-16-10001-11501 (active_node_name) Volume:NBU_MSTR5_MAESTRO_VOL:online:Volume opt-prod-maestro is started. Any mirrors are updated in background 2011/05/04 07:53:43 VCS INFO V-16-1-10298 Resource NBU_MSTR5_MAESTRO_VOL (Owner: unknown, Group: NB653GRP) is online on active_node_name (VCS initiated) 2011/05/04 07:53:43 VCS NOTICE V-16-1-10301 Initiating Online of Resource NBU_MSTR5_MAESTRO_MNT (Owner: unknown, Group: NB653GRP) on System active_node_name 2011/05/04 07:53:49 VCS INFO V-16-1-10298 Resource NBU_MSTR5_NBUCATALOG_MNT (Owner: unknown, Group: NB653GRP) is online on active_node_name (VCS initiated) 2011/05/04 07:53:49 VCS INFO V-16-1-10298 Resource NBU_MSTR5_MAESTRO_MNT (Owner: unknown, Group: NB653GRP) is online on active_node_name (VCS initiated) 2011/05/04 07:53:49 VCS NOTICE V-16-1-10301 Initiating Online of Resource NBU_MSTR5_MAESTRO (Owner: unknown, Group: NB653GRP) on System active_node_name 2011/05/04 07:54:33 VCS INFO V-16-1-10298 Resource NBU_MSTR5_MAESTRO (Owner: unknown, Group: NB653GRP) is online on active_node_name (VCS initiated) 2011/05/04 07:54:33 VCS NOTICE V-16-1-10301 Initiating Online of Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) on System active_node_name 2011/05/04 07:54:51 VCS INFO V-16-2-13001 (active_node_name) Resource(NB653GRP_RES_SERVER): Output of the completed operation (online) 2011/05/04 07:54:52 VCS INFO V-16-1-10298 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is online on active_node_name (VCS initiated) 2011/05/04 07:54:52 VCS NOTICE V-16-1-10447 Group NB653GRP is online on system active_node_name 2011/05/04 07:55:10 VCS INFO V-16-1-50135 User root fired command: hagrp - freeze NB653GRP from localhost 2011/05/04 07:55:58 VCS INFO V-16-1-50135 User root fired command: hagrp - freeze NB653GRP from localhost 2011/05/04 11:11:04 VCS INFO V-16-1-10307 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is offline on active_node_name (Not initiated by VCS) 2011/05/04 11:16:06 VCS INFO V-16-1-10299 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is online on active_node_name (Not initiated by VCS) 2011/05/04 11:16:06 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group NB653GRP on all nodes 2011/05/04 11:16:06 VCS NOTICE V-16-1-10447 Group NB653GRP is online on system active_node_name 2011/05/04 11:17:06 VCS INFO V-16-1-10307 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is offline on active_node_name (Not initiated by VCS) 2011/05/04 11:22:08 VCS INFO V-16-1-10299 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is online on active_node_name (Not initiated by VCS) 2011/05/04 11:22:08 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group NB653GRP on all nodes 2011/05/04 11:22:08 VCS NOTICE V-16-1-10447 Group NB653GRP is online on system active_node_name 2011/05/04 11:23:09 VCS INFO V-16-1-10307 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is offline on active_node_name (Not initiated by VCS) 2011/05/04 11:28:10 VCS INFO V-16-1-10299 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is online on active_node_name (Not initiated by VCS) 2011/05/04 11:28:10 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group NB653GRP on all nodes 2011/05/04 11:28:10 VCS NOTICE V-16-1-10447 Group NB653GRP is online on system active_node_name 2011/05/04 11:29:11 VCS INFO V-16-1-10307 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is offline on active_node_name (Not initiated by VCS) 2011/05/04 11:34:12 VCS INFO V-16-1-10299 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is online on active_node_name (Not initiated by VCS) 2011/05/04 11:34:12 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group NB653GRP on all nodes 2011/05/04 11:34:12 VCS NOTICE V-16-1-10447 Group NB653GRP is online on system active_node_name 2011/05/04 11:35:14 VCS INFO V-16-1-10307 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is offline on active_node_name (Not initiated by VCS) 2011/05/04 11:40:15 VCS INFO V-16-1-50135 User root fired command: hagrp - unfreeze NB653GRP from localhost 2011/05/04 11:40:15 VCS INFO V-16-1-10299 Resource NB653GRP_RES_SERVER (Owner: unknown, Group: NB653GRP) is online on active_node_name (Not initiated by VCS) 2011/05/04 11:40:15 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group NB653GRP on all nodes 2011/05/04 11:40:15 VCS NOTICE V-16-1-10447 Group NB653GRP is online on system active_node_name 2011/05/04 11:41:00 VCS INFO V-16-1-50135 User root fired command: hares - clear NB653GRP_RES_SERVER from localhost 2011/05/04 11:41:16 VCS ERROR V-16-2-13067 (active_node_name) Agent is calling clean for resource(NB653GRP_RES_SERVER) because the resource became OFFLINE unexpectedly, on its own. 2011/05/04 11:43:50 VCS INFO V-16-2-13001 (active_node_name) Resource(NB653GRP_RES_SERVER): Output of the completed operation (clean) 2011/05/04 11:43:50 VCS INFO V-16-2-13068 (active_node_name) Resource(NB653GRP_RES_SERVER) - clean completed successfully. 2011/05/04 11:43:50 VCS ERROR V-16-2-13073 (active_node_name) Resource(NB653GRP_RES_SERVER) became OFFLINE unexpectedly on its own. Agent is restarting (attempt number 1 of 2) the resource. 2011/05/04 11:44:03 VCS INFO V-16-2-13001 (active_node_name) Resource(NB653GRP_RES_SERVER): Output of the completed operation (online) 2011/05/04 11:44:05 VCS NOTICE V-16-2-13076 (active_node_name) Agent has successfully restarted resource(NB653GRP_RES_SERVER).