ALL DATABASE backups failing with Error 41 ( Network Connection timeout )

[Nayab_Rasool]

Hi All,

All Database Backups failed with error 41 in our environment , Don't knwo what is the reason according to my work mate this was fixed after rebooting the server last time . Please help me to proceed on this.

Thanks,

Nayab

[Ashish-Sharma]

Hi,

Check this artical

STATUS CODE: 41 "Network connection timed out"

http://www.symantec.com/business/support/index?page=content&id=TECH37297

[Marianne]

You need to give us more info: NBU version on master, media and database clients OS on master, media and database clients Which type of databases? Using db agents? Reboot of server means network drivers and services are reloaded. Best to find the cause. Create all relevant logs: On master: bprd On media server: bpbrm and bptm On client: bpcd and relevant db agent logs (most use dbclient) with write permission for db user on agent logs. Follow NBU process flow to check logs.

[Nayab_Rasool]

NBU Version Master, Media and Clients :- 7.1.0.4

OS Master , Media , Clients :- WINDOWS 2008 R2 SP1

Database :- MY SQL

Yes Using DB agents 

[Marianne]

Using MySQL agent from Zmanda?

Which server reboot resolves the problem? Master? Media? Db Client?
You need to concentrate troubleshooting on the server that needs the reboot.

Are filesystem backups working fine?

Have you tried to increase Client Connect and Client Read Timeouts on Master and Media servers?
The default is 5 minutes for both. Large database backups normally need more. We change it to 1800 (30 minutes) when we see timeout errors.

Have you created log folders as per my post above?
Use your Db agent manual for a list of logs on the client (in addition to bpcd).

Please post all text in Details tab of one of the failed jobs. 

[Yasuhisa_Ishika]

Did you(or your colleague) make any change in DNS or hosts file at that time? NetBackup caches result of name resolving. Change to DNS and hosts file may not take effect for a while after changing. Possible scenario is: 1. you tried to backup while the hostname is resolved to wrong IP address. 2. You corrected DNS entry or hosts file so that the host name is resolved to correct IP address. 3. Change in DNS or hosts file did not take effect until the chache bocame expired. 4. You rebooted the system. 5.Cache was expired before you got success in retry.

[Nayab_Rasool]

Hi All, 

Thanks a lot for all you responses and i have checked with my SQL team regarding this i came to know they have enabled KASPERSKY ANTI VIRUS REAL TIME SCANNING few days back ,this might be the reason for these backups to fail ??

Thanks,

Nayab

[Marianne]

You still have not given us enough info (see questions in my previous post).

We really can't say. If the AV is causing this, normal filesystem backups should also be affected.

[Nayab_Rasool]

Yes yes all backups are failing with Error 41 :( 

[Nayab_Rasool]

NBU version on master, media and database clients

All Servers NBU :- 7.1.0.4 

OS on master, media and database clients

All :- WINDOWS 2008 R2 SP1

Which type of databases?

SQL

Using db agents?

I can see the backup type as MS-WINDOWS backup

Reboot of server means network drivers and services are reloaded.

Reboot of CLIENT machine fixed the issue

Best to find the cause. Create all relevant logs:
On master: bprd
On media server: bpbrm and bptm
On client: bpcd and relevant db agent logs (most use dblient) with write permission for db user on agent logs.

All logs are in place already 

[Marianne]

Can you see if bpcd log on the client is written to?

If not, AV software is probably blocking ports.

If logs are written, incoming connections are allowed.
Check bpcd log for errors. 
Check if client is attempting to connect back.

[Nayab_Rasool]

This is how the backup is failing :( 

[Marianne]

Looks like Client Read Timeout rather than Connect Timeout.

We see Connecting, then Connected, followed by bpbkar32 starting up.

This says to me that the client is trying to generate data stream but has not been able to do so within the 5 minute timeout period. You can increase the timeout on the media server, but better to fix the problem on the client.

You need to see what is happening on the client when the backup is started. Check AV software: Is it scanning each file? Is it preventing files from being sent?

Check bpbkar log on client.

[Nayab_Rasool]

Sorry i din find bpbkar and bpfis logs in LOGS , What would be the next step

[Marianne]

Log folders do not exist by default. 

You need to CREATE them as per my post 2 days ago (at the time we were under the impression that you were performing database agent backups).

[Nayab_Rasool]

Hi Please find the bpcd log

[Marianne]

PLEASE read my post of earlier today (after you posted the screenshot) - connection is actually successful. This means we no longer need bpcd. We now need to troubleshoot the backup stream.

Looks like Client Read Timeout rather than Connect Timeout.

We see Connecting, then Connected, followed by bpbkar32 starting up.

This says to me that the client is trying to generate data stream but has not been able to do so within the 5 minute timeout period. You can increase the timeout on the media server, but better to fix the problem on the client.

You need to see what is happening on the client when the backup is started. Check AV software: Is it scanning each file? Is it preventing files from being sent?

Check bpbkar log on client.

[Nayab_Rasool]

Sorry Maria, 

The above log file is Bpbkar log file could you please check it and let me know your findings. As i have created the Bpbkar log folder today and started a backup manually.

Thanks,

Nayab

[TomerG]

Regarding "If the AV is causing this, normal filesystem backups should also be affected.", that may normally be true, however because of AV scanning the delay for the client read might be enough to throw it over that 5 minute limit (or whatever it is set to), whereas a file system backup may not have the same delay, especially since later the job details look like a client read timeout. Interested to see the outcome

[Marianne]

This turned out to be MS-Windows policy only....

See below.