Problem with /usr filesystem usage after upgrade to 7.6 on AIX Media Servers

[marekkedzierski]

Hi, I've got a strange issue on AIX media servers after upgrade to 7.6. Filesystem /usr gets full because of inodes not released properly (some files are not deleted properly). Have you experienced similar issues ?

[Nicolai]

If a file is in use when deleted, the space is not released before the process using the file releases it - normal a process exit.

Excatly what files are deleted in /usr - it normal a static file system.

[marekkedzierski]

We don't know which files are not deleted properly. We can't get filename from inode. We tried ncheck command but it doesn't show filename.

Server reboot resolves this issue but it's not a solution ;)) This issue exists on many AIX media servers after upgrade to 7.6.0.4 so it's probably a bug..

[revarooo]

I highly doubt this is a NBU issue. It's down to the OS to delete files and release inodes.

[Nicolai]

Stopping Netbackup media server processes should be enough to release the inodes if netbackup still has them open.

Can you try that ?

[Nicolai]

Maybe worth a try ....

http://www.aixhealthcheck.com/blog?id=37

[mph999]

ls -il should show the inode numbers against the files, well it does for most Unix/ Linux unless AIX 'works' differently (which is quite possible).

[marekkedzierski]

I've found this script but it doesn't find anything..

[marekkedzierski]

Unfortunately we are not able to find which files are not released. We've got inodes numbers but "fuser -dV /usr" and "lsof | grep deleted" doesn't show them. We stopped services on one AIX test server and free space on /usr stopped decreasing. On active media servers we lose 1GB per day on /usr (DF shows free space is decreasing but DU doesn't show occupied space increasing). We can free space on filesystem only by reboot...

How can I move a suppor case to next support line ???

[Marianne]

How can I move a suppor case to next support line ???

Email your Support engineer and ask that the call be escalated.
Follow up with a phone call, ask for the Duty Manager and ask for escalation.

Hopefully you have logged a call with IBM as well? 

[marekkedzierski]

I have to call Duty Manager... 

We logged a call with IBM but they said it's a problem with application ;)

[marekkedzierski]

Free space can be recovered using slibclean command without any reboots and unmounts :)

[Nicolai]

slibclean is described here:

http://www-01.ibm.com/support/knowledgecenter/ssw_aix_53/com.ibm.aix.cmds/doc/aixcmds5/slibclean.htm?cp=ssw_aix_53%2F1-2-0-18-45

[marekkedzierski]

We tested in and it works. It's probably a bug of nbtelesched which isn't releasing shared memory.

[mph999]

I'm looking at this issue at the moment.

I think yes, nbtelemetry runs approximately every 15 minutes regardless of joining the NetBackup Product Improvement Program and previous cases has been seen with the same symptoms on AIX after an upgrade.

[mph999 · Accepted solution]

Resolution is marked for 7.6.1 - looking at the eTrack it looks too complex to release as an EEB.

Workaround is to rename or change permissions on nbtelesched & nbtelemetry.

[marekkedzierski]

Thanks :)

[youjie_yu]

Hi all,

I' ve too got this issue on AIX 6100-08,media servers, NBU version is  7.6.0.4. the /usr filesystem is filling up 100%, I had to enlarge /usr,but it still grow at 10MB/20mins.