#
# Log Name:     engine
# System:       app01.local
# SysInfo:      Linux:app01.local,#1 SMP Tue Jan 29 11:47:41 EST 2013,2.6.32-358.el6.x86_64,x86_64
# Created:      2013/08/02 11:40:36 
#

2013/08/02 11:55:59 VCS INFO V-16-1-50135 User root fired command: hagrp -switch app-rg  app02  from localhost
2013/08/02 11:55:59 VCS NOTICE V-16-1-10208 Initiating switch of group app-rg from system app01 to system app02
2013/08/02 11:55:59 VCS NOTICE V-16-1-10300 Initiating Offline of Resource nfs-restr_U (Owner: Unspecified, Group: app-rg) on System app01
2013/08/02 11:56:00 VCS INFO V-16-1-10305 Resource nfs-restr_U (Owner: Unspecified, Group: app-rg) is offline on app01 (VCS initiated)
2013/08/02 11:56:00 VCS NOTICE V-16-1-10300 Initiating Offline of Resource app-rg-app (Owner: Unspecified, Group: app-rg) on System app01
2013/08/02 11:56:58 VCS ERROR V-16-10031-7029 (app01) NFS:NFS-group:monitor:Daemon [rpc.nfsd] is not running.
2013/08/02 11:56:58 VCS INFO V-16-10031-7030 (app01) NFS:NFS-group:monitor:Group might be going online or offline. Waiting for the operation to complete
2013/08/02 11:57:29 VCS INFO V-16-10031-504 (app01) Application:app-rg-app:offline:Executed /opt/OV/misc/app/ha/appharg.ovpl as user root
2013/08/02 11:57:33 VCS INFO V-16-1-10305 Resource app-rg-app (Owner: Unspecified, Group: app-rg) is offline on app01 (VCS initiated)
2013/08/02 11:57:33 VCS NOTICE V-16-1-10300 Initiating Offline of Resource app-rg-ip (Owner: Unspecified, Group: app-rg) on System app01
2013/08/02 11:57:34 VCS INFO V-16-1-10305 Resource app-rg-ip (Owner: Unspecified, Group: app-rg) is offline on app01 (VCS initiated)
2013/08/02 11:57:34 VCS NOTICE V-16-1-10300 Initiating Offline of Resource share_export_F (Owner: Unspecified, Group: app-rg) on System app01
2013/08/02 11:57:34 VCS NOTICE V-16-1-10300 Initiating Offline of Resource share_export_L (Owner: Unspecified, Group: app-rg) on System app01
2013/08/02 11:57:40 VCS INFO V-16-1-10305 Resource share_export_F (Owner: Unspecified, Group: app-rg) is offline on app01 (VCS initiated)
2013/08/02 11:57:42 VCS INFO V-16-1-10305 Resource share_export_L (Owner: Unspecified, Group: app-rg) is offline on app01 (VCS initiated)
2013/08/02 11:57:42 VCS NOTICE V-16-1-10300 Initiating Offline of Resource nfs-restr_L (Owner: Unspecified, Group: app-rg) on System app01
2013/08/02 11:57:48 VCS INFO V-16-1-10305 Resource nfs-restr_L (Owner: Unspecified, Group: app-rg) is offline on app01 (VCS initiated)
2013/08/02 11:57:48 VCS NOTICE V-16-1-10300 Initiating Offline of Resource lockinfo-mnt (Owner: Unspecified, Group: app-rg) on System app01
2013/08/02 11:57:48 VCS NOTICE V-16-1-10300 Initiating Offline of Resource app-rg-mount (Owner: Unspecified, Group: app-rg) on System app01
2013/08/02 11:57:49 VCS INFO V-16-1-10305 Resource lockinfo-mnt (Owner: Unspecified, Group: app-rg) is offline on app01 (VCS initiated)
2013/08/02 11:57:49 VCS NOTICE V-16-1-10300 Initiating Offline of Resource app-rvg-lock (Owner: Unspecified, Group: app-rg) on System app01
2013/08/02 11:57:50 VCS INFO V-16-1-10305 Resource app-rg-mount (Owner: Unspecified, Group: app-rg) is offline on app01 (VCS initiated)
2013/08/02 11:57:50 VCS NOTICE V-16-1-10300 Initiating Offline of Resource app-rvg-prim (Owner: Unspecified, Group: app-rg) on System app01
2013/08/02 11:57:53 VCS INFO V-16-1-10305 Resource app-rvg-lock (Owner: Unspecified, Group: app-rg) is offline on app01 (VCS initiated)
2013/08/02 11:57:58 VCS INFO V-16-1-10305 Resource app-rvg-prim (Owner: Unspecified, Group: app-rg) is offline on app01 (VCS initiated)
2013/08/02 11:57:58 VCS NOTICE V-16-1-10446 Group app-rg is offline on system app01
2013/08/02 11:57:58 VCS NOTICE V-16-1-10301 Initiating Online of Resource app-rvg-lock (Owner: Unspecified, Group: app-rg) on System app02
2013/08/02 11:57:58 VCS NOTICE V-16-1-10301 Initiating Online of Resource app-rvg-prim (Owner: Unspecified, Group: app-rg) on System app02
2013/08/02 11:58:05 VCS INFO V-16-20084-20 (app02) RVGPrimary:app-rvg-lock:online:The RVG hr_rvg_lock migrated successfully.
2013/08/02 11:58:05 VCS INFO V-16-20084-73 (app02) RVGPrimary:app-rvg-lock:online:Difference based synchronization is not needed for any of the additional secondaries.
2013/08/02 11:58:07 VCS INFO V-16-1-10298 Resource app-rvg-lock (Owner: Unspecified, Group: app-rg) is online on app02 (VCS initiated)
2013/08/02 11:58:07 VCS NOTICE V-16-1-10301 Initiating Online of Resource lockinfo-mnt (Owner: Unspecified, Group: app-rg) on System app02
2013/08/02 11:58:11 VCS INFO V-16-1-10298 Resource lockinfo-mnt (Owner: Unspecified, Group: app-rg) is online on app02 (VCS initiated)
2013/08/02 11:58:14 VCS INFO V-16-20084-20 (app02) RVGPrimary:app-rvg-prim:online:The RVG hr_rvg migrated successfully.
2013/08/02 11:58:14 VCS INFO V-16-20084-73 (app02) RVGPrimary:app-rvg-prim:online:Difference based synchronization is not needed for any of the additional secondaries.
2013/08/02 11:58:16 VCS INFO V-16-1-10298 Resource app-rvg-prim (Owner: Unspecified, Group: app-rg) is online on app02 (VCS initiated)
2013/08/02 11:58:16 VCS NOTICE V-16-1-10301 Initiating Online of Resource app-rg-mount (Owner: Unspecified, Group: app-rg) on System app02
2013/08/02 11:58:19 VCS INFO V-16-1-10298 Resource app-rg-mount (Owner: Unspecified, Group: app-rg) is online on app02 (VCS initiated)
2013/08/02 11:58:19 VCS NOTICE V-16-1-10301 Initiating Online of Resource nfs-restr_L (Owner: Unspecified, Group: app-rg) on System app02
2013/08/02 11:58:22 VCS INFO V-16-1-10298 Resource nfs-restr_L (Owner: Unspecified, Group: app-rg) is online on app02 (VCS initiated)
2013/08/02 11:58:22 VCS NOTICE V-16-1-10301 Initiating Online of Resource share_export_L (Owner: Unspecified, Group: app-rg) on System app02
2013/08/02 11:58:22 VCS NOTICE V-16-1-10301 Initiating Online of Resource share_export_F (Owner: Unspecified, Group: app-rg) on System app02
2013/08/02 11:58:26 VCS INFO V-16-1-10298 Resource share_export_F (Owner: Unspecified, Group: app-rg) is online on app02 (VCS initiated)
2013/08/02 11:58:26 VCS INFO V-16-1-10298 Resource share_export_L (Owner: Unspecified, Group: app-rg) is online on app02 (VCS initiated)
2013/08/02 11:58:26 VCS NOTICE V-16-1-10301 Initiating Online of Resource app-rg-ip (Owner: Unspecified, Group: app-rg) on System app02
2013/08/02 11:58:32 VCS ERROR V-16-10031-7029 (app02) NFS:NFS-group:monitor:Daemon [rpc.statd] is not running.
2013/08/02 11:58:32 VCS INFO V-16-10031-7030 (app02) NFS:NFS-group:monitor:Group might be going online or offline. Waiting for the operation to complete
2013/08/02 11:58:38 VCS INFO V-16-1-10298 Resource app-rg-ip (Owner: Unspecified, Group: app-rg) is online on app02 (VCS initiated)
2013/08/02 11:58:38 VCS NOTICE V-16-1-10301 Initiating Online of Resource app-rg-app (Owner: Unspecified, Group: app-rg) on System app02
2013/08/02 11:59:32 VCS ERROR V-16-10031-7029 (app02) NFS:NFS-group:monitor:Daemon [rpc.statd] is not running.
2013/08/02 11:59:32 VCS INFO V-16-10031-7030 (app02) NFS:NFS-group:monitor:Group might be going online or offline. Waiting for the operation to complete
2013/08/02 12:00:32 VCS ERROR V-16-10031-7029 (app02) NFS:NFS-group:monitor:Daemon [rpc.statd] is not running.
2013/08/02 12:00:32 VCS INFO V-16-10031-7030 (app02) NFS:NFS-group:monitor:Group might be going online or offline. Waiting for the operation to complete
2013/08/02 12:01:32 VCS ERROR V-16-10031-7029 (app02) NFS:NFS-group:monitor:Daemon [rpc.statd] is not running.
2013/08/02 12:01:33 VCS ERROR V-16-2-13067 (app02) Agent is calling clean for resource(NFS-group) because the resource became OFFLINE unexpectedly, on its own.
2013/08/02 12:01:33 VCS WARNING V-16-10031-7002 (app02) NFS:NFS-group:clean:Sending SIGKILL signal to nfsd (15208 15209 15210 15211 15212 15213 15214 15215 15216 15217 15218 15219 15220 15221 15222 15223 15224 15225 15226 15227 15228 15229 15230 15231 15232 15233 15234 15235 15236 15237).
2013/08/02 12:01:35 VCS INFO V-16-2-13716 (app02) Resource(NFS-group): Output of the completed operation (clean) 
==============================================
mount: block device nfsd is write-protected, mounting read-only
mount: cannot mount block device nfsd read-only
fs.nfs.nlm_grace_period = 90
==============================================

2013/08/02 12:01:35 VCS INFO V-16-2-13068 (app02) Resource(NFS-group) - clean completed successfully.
2013/08/02 12:01:35 VCS ERROR V-16-2-13073 (app02) Resource(NFS-group) became OFFLINE unexpectedly on its own. Agent is restarting (attempt number 1 of 1) the resource.
2013/08/02 12:01:35 VCS WARNING V-16-10031-7002 (app02) NFS:NFS-group:online:Sending SIGKILL signal to nfsd (17330 17331 17332 17333 17334 17335 17336 17337 17338 17339 17340 17341 17342 17343 17344 17345 17346 17347 17348 17349 17350 17351 17352 17353 17354 17355 17356 17357 17358 17359).
2013/08/02 12:01:37 VCS INFO V-16-2-13716 (app02) Resource(NFS-group): Output of the completed operation (online) 
==============================================
mount: block device nfsd is write-protected, mounting read-only
mount: cannot mount block device nfsd read-only
fs.nfs.nlm_grace_period = 90
==============================================

2013/08/02 12:01:40 VCS NOTICE V-16-2-13076 (app02) Agent has successfully restarted resource(NFS-group).
2013/08/02 12:01:40 VCS INFO V-16-1-55031 Resource NFS-group in online state received recurring online message on system app02
2013/08/02 12:03:48 VCS INFO V-16-10031-504 (app02) Application:app-rg-app:online:Executed /opt/OV/misc/app/ha/appharg.ovpl as user root
2013/08/02 12:03:49 VCS INFO V-16-2-13716 (app02) Resource(app-rg-app): Output of the completed operation (online) 
==============================================
Copying files from shared disk.
==============================================

2013/08/02 12:03:54 VCS INFO V-16-1-10298 Resource app-rg-app (Owner: Unspecified, Group: app-rg) is online on app02 (VCS initiated)
2013/08/02 12:03:54 VCS NOTICE V-16-1-10301 Initiating Online of Resource nfs-restr_U (Owner: Unspecified, Group: app-rg) on System app02
2013/08/02 12:03:58 VCS INFO V-16-1-10298 Resource nfs-restr_U (Owner: Unspecified, Group: app-rg) is online on app02 (VCS initiated)
2013/08/02 12:03:58 VCS NOTICE V-16-1-10447 Group app-rg is online on system app02