Backup are failing with 11 error(System Call failed)

[Stiva]

Hi All,

From last few days I am observing that the backups are failing for some policies with errro code 11 at one specific time. But after some time when I restart the backup. They are getting successfully completed.

Please find error:

06/19/2014 19:20:27 - Error bptm (pid=24055) system call failed - Resource temporarily unavailable (at tmmpx.c.1507)

Netbackup Master server version 6.5 OS - Solaris.

Please help to resolve this issue. I found there are so many reasons for error code 11 failure. but not able to decide exact resolution.

Thank You.

[Marianne]

Status 11 means that something went wrong at OS level. 
This means that it is difficult to trace from NBU point of view.

The 1st thing to check is system resources - disk space, memory, swap, shared memory.

Check NBU All Log Entries report as well as /var/adm/messages for clues.

PS:
Are you aware that NBU 6.5 ran out of support almost 2 years ago?

[Linette_V]

Hi Stiva,

As you've already found this can happen for lots of different reasons. The most common theme tends to be system resource issues. This can be due to an overloaded NIC, or more commonly not enough inodes/file descriptors.

Whilst the issue is manifesting differently here, this is a clear KM article for file descriptor tuning: http://www.symantec.com/docs/TECH168846

You may also find this useful: http://www.symantec.com/docs/TECH62633

Give tuning a go, and see if they still fail. If so it would be worth monitoring network utilization at the time of failure.

Good luck!

[Stiva]

Thanks to reply :)

@Marianne : I am aware that 6.5 is out of service.

I will check NBU logs and update.

[Marianne]

Not only NBU logs - OS logs as well. 
Important to understand that some OS call is failing.

I was hoping that you would get back to us today with section of /var/adm/messages that covers the time frame when status 11 errors were seen.

Same with All Log Entries report - select the master server and the start and end time a bit before (+- 1 hour) the status 11 up to about 1 hour after the error was seen.

[Stiva]

Hi Marianne,

I am observing that from last 3 days failures are not coming. I will observe it for some more days and will get back to you.

Thank you for your help.

[Marianne · Accepted solution]

Well, it has been 6 days since you posted.

So, All Log Entries report and messages file need to be checked for the time that the error was seen.

I have never seen a Status 11 where NOTHING was logged in messages or NBU All Log Entries.

Because bptm reported the error, you need to check logs for the media server that was used at the time.

[Stiva]

Hi Marianne,

We found the issue. Yes,There was a issue with media server itself.

Media server was not rebooted from last 3.7 Years. Real memory was 14GB on the server. The kernel was using 98% which leaves almost nothing for programs, program data, and I/O caching.

We rebooted the server and applied latest patch on it. Now its up and running fine.

Thank you Marianne & L. Voller for your valuable comments. :)

[Marianne]

So, this means that nothing has been updated on this (and probably other servers) in almost 4 years?

No OS patches, no NBU updates?

Please shedule updates/upgrades as a matter of urgency. Especially NBU.
If anything more serious goes wrong, you will be left without any support.

Glad all is fine (for now).

[Stiva]

No, With only two media servers. It was a very old setup. Previously it was working on shared library. Now we are using dedicated one. We are in the process to upgrade NBU to 7.1.

Let's hope It will work fine for 1/2 months...

[Marianne]

Please skip NBU 7.1 and upgrade straight to 7.5 or 7.6.