05-03-2013 11:23 AM
Hi All.
The VRTSvxfs-6.0.300.000-SLES11 patch install failed while applying the SF 6.0.3 patches with installmr on top of SF 6.0.1. The RPM verification passes (rpm -V), but the VRTSvxfs patch log shows "Failed to create /dev/vxportal" and "ERROR: No appropriate VxFS drivers found that can be loaded." A check of the kernel modules showed that vxportal and vxfs are not loaded. A reboot was performed to load the missing kernel modules, and VxFS now functions without any further issues.
The number of servers affected was 4 out of ~ 50.
Q: Has anyone else experienced this issue on SLES 11.1 when applying 6.0.3 patches ?
Q: Any pointers on where to look/check on the next time that it occurs ?
Here are some specifics...
Installmr Error
Performing SF preinstall tasks ........................................................................................ Done Installing VRTSperl patch ............................................................................................. Done Installing VRTSspt patch .............................................................................................. Done Installing VRTSvxvm patch ............................................................................................. Done Installing VRTSvxfs patch ........................................................................................... Failed Installing VRTSdbed patch ............................................................................................. Done Installing VRTSodm patch .............................................................................................. Done Installing VRTSsfcpi601 patch ......................................................................................... Done Performing SF postinstall tasks ....................................................................................... Done Veritas Storage Foundation Install did not complete successfully VRTSvxfs patch failed to install on HHHHH Refer to the installer logs for details and correct the settings. Continue with the installer again, or try to install the packages/patches manually.
VRTSvxfs patch log
# cat opt/VRTS/install/logs/installmr-201305011327LbS/install.patch.VRTSvxfs.SVBMN14GLBSM4A Preparing packages for installation... VRTSvxfs-6.0.300.000-SLES11 WARNING: No modules found for 2.6.32.59-0.7-default, using compatible modules for 2.6.32.12-0.7-default. Failed to create /dev/vxportal ERROR: No appropriate VxFS drivers found that can be loaded. See VxFS documentation for the list of supported platforms.
Platform
# rpm -qa *-release sles-release-11.1-1.206 # uname -r 2.6.32.59-0.7-default # uname -m x86_64
Full installmr
# ./installmr -responsefile ../installmr_SF603_patchupgrade
Logs are being written to /var/tmp/installmr-201305011327LbS while installmr is in progress
Verifying systems: 100%
Estimated time remaining: (mm:ss) 0:00 8 of 8
Checking system communication ......................................................................................... Done
Checking release compatibility ........................................................................................ Done
Checking installed product ............................................................................................ Done
Checking prerequisite patches and rpms ................................................................................ Done
Checking platform version ............................................................................................. Done
Checking file system free space ....................................................................................... Done
Checking product licensing ............................................................................................ Done
Performing product prechecks .......................................................................................... Done
System verification checks completed successfully
The following notes were discovered on the systems:
CPI NOTE V-9-30-1309 SF version 6.0.100.000 will be upgraded to SF version 6.0.300.000 on HHHHH
The following Veritas Storage Foundation patches will be installed on all systems:
Patch Rpm
VRTSperl-5.14.2.8 VRTSperl
VRTSsfcpi601-6.0.300.0 VRTSsfcpi601
VRTSspt-6.0.300.0 VRTSspt
VRTSvxvm-6.0.300.0 VRTSvxvm
VRTSvxfs-6.0.300.0 VRTSvxfs
VRTSdbed-6.0.300.0 VRTSdbed
VRTSodm-6.0.100.100 VRTSodm
Logs are being written to /var/tmp/installmr-201305011327LbS while installmr is in progress
Stopping SF: 100%
Estimated time remaining: (mm:ss) 0:00 18 of 18
Performing SF prestop tasks ........................................................................................... Done
Stopping vxsvc ........................................................................................................ Done
Stopping vxodm ........................................................................................................ Done
Stopping vxdbd ........................................................................................................ Done
Stopping fdd .......................................................................................................... Done
Stopping vxportal ..................................................................................................... Done
Stopping vxfs ......................................................................................................... Done
Stopping sfmh-discovery ............................................................................................... Done
Stopping vxdclid ...................................................................................................... Done
Stopping vxconfigbackupd .............................................................................................. Done
Stopping vxsited ...................................................................................................... Done
Stopping vxrelocd ..................................................................................................... Done
Stopping vxnotify ..................................................................................................... Done
Stopping vxesd ........................................................................................................ Done
Stopping vxconfigd .................................................................................................... Done
Stopping vxspec ....................................................................................................... Done
Stopping vxio ......................................................................................................... Done
Stopping vxdmp ........................................................................................................ Done
Veritas Storage Foundation Shutdown completed successfully
Logs are being written to /var/tmp/installmr-201305011327LbS while installmr is in progress
Installing SF: 100%
Estimated time remaining: (mm:ss) 0:00 9 of 9
Performing SF preinstall tasks ........................................................................................ Done
Installing VRTSperl patch ............................................................................................. Done
Installing VRTSspt patch .............................................................................................. Done
Installing VRTSvxvm patch ............................................................................................. Done
Installing VRTSvxfs patch ........................................................................................... Failed
Installing VRTSdbed patch ............................................................................................. Done
Installing VRTSodm patch .............................................................................................. Done
Installing VRTSsfcpi601 patch ......................................................................................... Done
Performing SF postinstall tasks ....................................................................................... Done
Veritas Storage Foundation Install did not complete successfully
VRTSvxfs patch failed to install on HHHHH
Refer to the installer logs for details and correct the settings. Continue with the installer again, or try to install the
packages/patches manually.
installmr log files and summary file are saved at:
/opt/VRTS/install/logs/installmr-201305011327LbS
RPM List After 6.0.3 patch
# rpm -qa VRTS\* | sort | cat -b 1 VRTSaslapm-6.0.100.000-GA_SLES11 2 VRTSdbed-6.0.300.000-GA_SLES 3 VRTSfsadv-6.0.100.000-GA_SLES11 4 VRTSfssdk-6.0.100.000-GA_SLES11 5 VRTSlvmconv-6.0.100.000-GA_SLES11 6 VRTSob-3.4.553-0 7 VRTSodm-6.0.100.100-SLES11 8 VRTSperl-5.14.2.8-SLES11 9 VRTSsfcpi601-6.0.300.000-GA_GENERIC 10 VRTSsfmh-5.0.196.0-0 11 VRTSspt-6.0.300.000-GA 12 VRTSvlic-3.02.61.004-0 13 VRTSvxfs-6.0.300.000-SLES11 14 VRTSvxvm-6.0.300.000-SLES11
Kernel Modules Before Reboot
# lsmod | grep vx vxspec 3622 2 vxio 3924994 1 vxspec vxdmp 423852 482 vxspec,vxio
Kernel Modules After Reboot
# lsmod | grep vx vxodm 216428 1 vxspec 3622 6 vxio 3924994 46 vxspec vxdmp 423852 482 vxspec,vxio vxportal 6695 0 fdd 55690 2 vxodm vxfs 3651117 49 vxportal,fdd
~~~~
end
05-06-2013 02:56 AM
Hi kwachtler,
It seems you met the same issue with RHEL 6.4.
Pls check on public technote: http://www.symantec.com/docs/TECH203099
Maybe Symantec is working on fix of the issue, but I didn't find available solution currently from public info.
To open a tech support case from Symantec is better to work you out the situation.
05-06-2013 05:19 AM
Hi kwachtler,
By any chance do the nodes with failure have relatively more number of luns/dmp paths? Incase you're doing more of these upgrades & encounter this again, instead of a reboot can you try starting vxfs manually using :/etc/vx/vxfs-startup and share the result?
05-07-2013 08:53 AM
Thank you for comments above. In response...
1. Yes, similar to RHEL 6.4 issue, but this may be 1st report of this issue on SLES11, so wanted to raise it here.
2. Symantec support case has been opened, but at a lower priority, since workaround exists (reboot).
3. Yes, all 4 systems affected have a high number of LUN/paths, so that could be a significant clue. These systems are non-clustered, so just basic Storage Foundation.
2 systems @ 200 LUNs, 1600 paths (8 paths per LUN)
2 systems @ 240 LUNs , 960 paths (4 paths per LUN)
I am waiting for Symantec TS to weigh-in via the support case.
Thank You all.
05-09-2013 12:16 PM
at 6.0.3, /etc/vx/vxfs-startup script should have the implementation to recreate /dev/vxportal if it failed for some reason (you may want to cat the script to doublecheck) . Instead of reboot, retrying to start vxfs by executing this script might be a better w/around than reboot. In some earlier instances it was seen that /dev/udevd (thats reponsible for creating devices under /dev was busy updating dmp devices (in case of large number of devices) ..that is was not able to service the request for creating /dev/vxportal -probably a timing issue. Good that you already opened a support case for a detailed analysis here.