Status Code 40 after Backup running for 15 Minutes - Everytime

I just run a backup with no port being blocked between the Client and Master. I still get the same result ! 

Backup just STOP at the same juncture. 

Try to configure the TCP keelalive parameter on the master/media and client. Even with no ports blocked the firewall will close inactive tcp sessions.

Refer to this technote of how to configure the settings on AIX:

http://www.symantec.com/docs/TECH125896

You must configure the tcp keepalive parameter to send keep alive packages faster than 15 minutes.

I too had this problem in my environment.

Schedule backus will start and after a certain period it will go to Incomplete State with Error Code 40.

And with in minutes it will automatically resume and the jobs is getting completed successfully.

We have contact network team.

For checking the network settings. Some Buffer size etc I dont know in specific.

Hi Nicolai, 

The article you show me is about communication between Master and Media whereas my case the Master/Media is the same server. Is this still applicable ? 

Please ask firewall team to confirm the timeout on the firewall.

Status 40 says the connection is broken outside of NBU - therefore not a NBU timeout or failure.

Please ensure that you have bpbrm and bptm logs on the master/media server.

This will confirm if data was received from client in a continuous stream when network failure occurred.

Always best to involve firewall team in situations like these - they need to monitor the port connections while the backup is running.

YES

It is not a Netbackup specific settings - it a OS setting that apply to all tcp sessions made.

You must configure the idle timeout on the master and media and client.

Marriane,

Do you mean that even thought their no restriction on the traffic between the master/media and the client thru the firewall, firewall timeout still applies ? 

I'm asking this because the last round of test I actually ask the firewall admin to let any traffic to pass from client to master/media. 

I am no firewall expert - just years of sleepless nights at customer sites troubleshooting connection issues.
I have learned that firewalls also have timeout settings, over and above port connection rules.

The only way to troubleshoot these things is to get the firewall admins on site to monitor connections/traffic while the backup is running.
If firewall and network admins are in different teams, get network admin involved as well.

According to the FIREWALL vendor, the TCP timeout at the firewall is set to 24 hours. 

If the firewall guys are sure of that then it is likely to be the keep alive settings as mentioned earlier but could you also check the following:

1. Do you have the client read timeout on the Media Servers also set higher than 15 minutes

2. Do you have correct host name resolution through the firewall - can you ping the client from the Master and Media Severs and the Master and Media from the clients (I am thinking you have the client connect / read timeouts set to 15 minutes and that they are not always resolving correctly - so do you use hosts files or rely on DNS)

3. Does the client have multiple network interfaces? if so than again hosts files may help

Not sure of exactly how the settings are done on AIX but on other systems i use these keep alive settings:

# echo 510 > /proc/sys/net/ipv4/tcp_keepalive_time
# echo 3 > /proc/sys/net/ipv4/tcp_keepalive_intvl
# echo 3 > /proc/sys/net/ipv4/tcp_keepalive_probes

Let us know the answer to my questions - hope this helps