VxFS Performance problem after HPUX/Veritas upgrade
An an old machine - model "9000/800/rp4440 " (4 PA-RISC 8800 processors (1000 MHz, 64 MB)) with 16GB we have upgraded HPUX from 11.23 to 11.31 - Veritas from 3.5 to 4.1. Now, from time to time we have system with very high response time. Basically, all normal operations (login, directory listing, ... ) take very, very long time, even though the CPU usage during that period is very low. Also, memory system is occuped only 50%, pagging is not used at all, I/O seems not to be overloaded too. Sometimes this situation is solved after some period of time without intervention, but during working hour machine has to be restarted, because we can't wait too much such situation during the day. One of measureable issues we have detecetd is slow execution of lstat64 system call on vxfs partitions/mount points. We used TUSC to perform a 'll' command on a directory with 15k files. NFS mount points are not affected. For example: 06:31:46 [ls -l /test ]{2035012} (0.000054) lstat64("/test/FILE_00249395_20130301063957.Z", 0x77ff0468) = 0 st_dev: 64 0x000003 st_ino: 42172 st_mode: S_IFREG|0640 st_nlink: 1 st_rdev: 0 0x000000 st_size: 0 st_blksize: 8192 st_blocks: 0 st_uid: 0 st_gid: 3 st_aclv: 0 st_acl: 0 st_fstype: 10 st_atime: Thu Feb 26 15:29:17 2015 st_mtime: Thu Feb 26 15:29:17 2015 st_ctime: Thu Feb 26 15:29:17 2015 06:32:29 [ls -l /test ]{2035012} lstat64(0x40001888, 0x77ff0468) [running] 06:32:29 [ls -l /test ]{2035012} (42.357834) lstat64("/test/FILE_00249396_20130301064257.Z", 0x77ff0468) = 0 st_dev: 64 0x000003 st_ino: 42173 st_mode: S_IFREG|0640 st_nlink: 1 st_rdev: 0 0x000000 st_size: 0 st_blksize: 8192 st_blocks: 0 st_uid: 0 st_gid: 3 st_aclv: 0 st_acl: 0 st_fstype: 10 st_atime: Thu Feb 26 15:29:17 2015 st_mtime: Thu Feb 26 15:29:17 2015 st_ctime: Thu Feb 26 15:29:17 2015 06:32:29 [ls -l /test ]{2035012} (0.000063) lstat64("/test/FILE_00249397_20130301064557.Z", 0x77ff0468) = 0 As you can notice, there is a 42seconds of delay during the execution lstat64 system call. This is observed on a local vxfs drive (Veritas 4.1 is installed). The whole directory listing 'll' sometimes takes 15min, sometimes even 20min. When system is fresh reboted it takes few seconds. Just to note. System did not change in hardware or softvare after the upgrade. All issues staretd after the upgrade. I don't know if someone had similar issues after the upgrade to 11.31 (Veritas 4.1). If you can get a hint what should be checked I would appreaciate very much.
ORA-600 [ksfd_odmcrt8], [ODM ERROR V-41-4-1-105-22 Invalid argument]
hi.. when we open db (Oracle 11.2.0.4 2node RAC in HP), we got the error... Sun Jun 01 23:19:24 2014 ALTER SYSTEM SET local_listener=' (ADDRESS=(PROTOCOL=TCP)(HOST=xxx.xxx.xxx.xxx)(PORT=1521))' SCOPE=MEMORY SID='MNP001'; ALTER DATABASE MOUNT /* db agent *//* {1:15272:181} */ This instance was first to mount Errors in file /oracle/MNP/saptrace/diag/rdbms/mnp/MNP001/trace/MNP001_ckpt_230.trc (incident=432209): ORA-00600: internal error code, arguments: [ksfd_odmopn2], [ODM ERROR V-41-4-2-313-22 Invalid argument], [], [], [], [], [], [], [], [], [], [] Incident details in: /oracle/MNP/saptrace/diag/rdbms/mnp/MNP001/incident/incdir_432209/MNP001_ckpt_230_i432209.trc [PROD spncpd01:/#] mount -p ....................... /dev/vg00/lvol10 /maxigent50 vxfs ioerror=mwdisable,largefiles,qio,delaylog,dev=4000000a 0 0 /dev/vg00/lvol11 /SIMPANA vxfs ioerror=mwdisable,largefiles,qio,delaylog,dev=4000000b 0 0 xx.xx.89.167:/snctrans /usr/sap/trans nfs rsize=32768,wsize=32768,NFSv3,dev=4000002 0 0 xx.xx.89.167:/sncmnp /sapmnt/MNP nfs rsize=32768,wsize=32768,NFSv3,dev=4000003 0 0 xx.xx.89.167:/sapimage /sapcd nfs rsize=32768,wsize=32768,NFSv3,dev=4000004 0 0 -hosts /net autofs ignore,indirect,nosuid,soft,nobrowse,dev=4000005 0 0 /dev/odm /dev/odm odm smartsync,dev=4000006 0 0 /dev/vx/dsk/ncpdg1/mnpomnp /oracle/MNP vxfs ioerror=mdisable,largefiles,qio,cluster,crw,delaylog,dev=20059d8 0 0 /dev/vx/dsk/ncpdg1/mnpstage /oracle/stage/112_64 vxfs ioerror=mdisable,largefiles,qio,cluster,crw,delaylog,dev=20059d9 0 0 /dev/vx/dsk/ncpdg1/mnpcli /oracle/client vxfs ioerror=mdisable,largefiles,qio,cluster,crw,delaylog,dev=20059da 0 0 /dev/vx/dsk/ncpdg3/mnpocr1 /oracle/OCR1 vxfs ioerror=mdisable,largefiles,qio,cluster,crw,delaylog,dev=2009858 0 0 /dev/vx/dsk/ncpdg3/mnpvote1 /oracle/VOTE1 vxfs ioerror=mdisable,largefiles,qio,cluster,crw,delaylog,dev=2009859 0 0 /dev/vx/dsk/ncpdg4/mnpocr2 /oracle/OCR2 vxfs ioerror=mdisable,largefiles,qio,cluster,crw,delaylog,dev=200c350 0 0 /dev/vx/dsk/ncpdg4/mnpvote2 /oracle/VOTE2 vxfs ioerror=mdisable,largefiles,qio,cluster,crw,delaylog,dev=200c351 0 0 /dev/vx/dsk/ncpdg5/mnpocr3 /oracle/OCR3 vxfs ioerror=mdisable,largefiles,qio,cluster,crw,delaylog,dev=200e678 0 0 /dev/vx/dsk/ncpdg5/mnpvote3 /oracle/VOTE3 vxfs ioerror=mdisable,largefiles,qio,cluster,crw,delaylog,dev=200e679 0 0 /dev/vx/dsk/ncpdg1/mnploga /oracle/MNP/origlogA vxfs ioerror=mdisable,largefiles,qio,cluster,crw,delaylog,dev=20059db 0 0 /dev/vx/dsk/ncpdg1/mnplogb /oracle/MNP/origlogB vxfs ioerror=mdisable,largefiles,qio,cluster,crw,delaylog,dev=20059dc 0 0 /dev/vx/dsk/ncpdg1/mnpmloga /oracle/MNP/mirrlogA vxfs ioerror=mdisable,largefiles,qio,cluster,crw,delaylog,dev=20059dd 0 0 <--- no using CIO so, we can not use veritas odm library... could you teach me, how we can use veritas odm library ??? ( Is there any issue in veritas / Oracle ?? ) Thanks~~~~problem with starting vxconfigd
Hello! i have a problem with starting vxconfigd daemon # # vxconfigd VxVM vxconfigd ERROR V-5-1-8561 ddl_fill_dmp_info: Path 0x1f021000 not found in devlist FAKE_ENCLR_SNO: Error 0 VxVM vxconfigd ERROR V-5-1-1589 enable failed: System error occurred in the client Error 0; aborting First was problem with free space in / file system. Somebody made # mv /sbin/fs/vxfs4.1 /opt; ln -s /opt/vxfs4.1 /sbin/fs/vxfs4.1 After reboot - server didn't start made > hpux /last/vmunix after starting did # rm /sbin/fs/vxfs4.1; mv /opt/vxfs4.1 /sbin/fs/vxfs4.1 # shutdown -r But now we can't to mount some directories by vxfs What can i do at this moment? Thank you.