VxFS Performance problem after HPUX/Veritas upgrade

An an old machine  - model "9000/800/rp4440  " (4 PA-RISC 8800 processors (1000 MHz, 64 MB)) with 16GB we have upgraded HPUX from 11.23 to 11.31 - Veritas from 3.5 to 4.1. Now, from time to time we have system with very high response time. Basically, all normal operations (login, directory listing, ... ) take very, very long time, even though the CPU usage during that period is very low. Also, memory system is occuped only 50%, pagging is not used at all, I/O  seems not to be overloaded too. Sometimes this situation is solved after some period of time without intervention, but during working hour machine has to be restarted, because we can't wait too much such situation during the day.

One of measureable issues we have detecetd is slow execution of lstat64 system call on vxfs partitions/mount points. We used TUSC to perform a 'll' command on a directory with 15k files. NFS mount points are not affected.

 

For example:

06:31:46 [ls -l /test ]{2035012} (0.000054) lstat64("/test/FILE_00249395_20130301063957.Z", 0x77ff0468) = 0
                            st_dev: 64 0x000003
                            st_ino: 42172
                           st_mode: S_IFREG|0640
                          st_nlink: 1
                           st_rdev: 0 0x000000
                           st_size: 0
                        st_blksize: 8192
                         st_blocks: 0
                            st_uid: 0
                            st_gid: 3
                           st_aclv: 0
                            st_acl: 0
                         st_fstype: 10
                          st_atime: Thu Feb 26 15:29:17 2015
                          st_mtime: Thu Feb 26 15:29:17 2015
                          st_ctime: Thu Feb 26 15:29:17 2015
06:32:29 [ls -l /test ]{2035012} lstat64(0x40001888, 0x77ff0468) [running]
06:32:29 [ls -l /test ]{2035012} (42.357834) lstat64("/test/FILE_00249396_20130301064257.Z", 0x77ff0468) = 0
                            st_dev: 64 0x000003
                            st_ino: 42173
                           st_mode: S_IFREG|0640
                          st_nlink: 1
                           st_rdev: 0 0x000000
                           st_size: 0
                        st_blksize: 8192
                         st_blocks: 0
                            st_uid: 0
                            st_gid: 3
                           st_aclv: 0
                            st_acl: 0
                         st_fstype: 10
                          st_atime: Thu Feb 26 15:29:17 2015
                          st_mtime: Thu Feb 26 15:29:17 2015
                          st_ctime: Thu Feb 26 15:29:17 2015
06:32:29 [ls -l /test ]{2035012} (0.000063) lstat64("/test/FILE_00249397_20130301064557.Z", 0x77ff0468) = 0

 

As you can notice, there is a 42seconds of delay during the execution lstat64 system call. This is observed on a local vxfs drive (Veritas 4.1 is installed). The whole directory listing 'll' sometimes takes 15min, sometimes even 20min. When system is fresh reboted it takes few seconds.

 

Just to note. System did not change in hardware or softvare after the upgrade. All issues staretd after the upgrade. I don't know if someone had similar issues after the upgrade to 11.31 (Veritas 4.1). If you can get a hint what should be checked I would appreaciate very much.