12-10-2012 10:49 PM
Hi All,
All Database Backups failed with error 41 in our environment , Don't knwo what is the reason according to my work mate this was fixed after rebooting the server last time . Please help me to proceed on this.
Thanks,
Nayab
Solved! Go to Solution.
12-17-2012 12:45 AM
Client is simply taking too long to generate backup stream:
Backup starts at 5:59:40.
Only at 6:07:27 does it start to evaluate local drive letters.
At 6:15:19 client is ready to 'START BACKUP'.
By this time, the Client Read Timeout on the media server has been exceeded and we see this message:
<16> dtcp_read: TCP - failure: recv socket (648) (TCP 10054: Connection reset by peer)
As per my 2 previous posts - you need to see what is happening on the client when the backup kicks off:
This says to me that the client is trying to generate data stream but has not been able to do so within the 5 minute timeout period. You can increase the timeout on the media server, but better to fix the problem on the client.
You need to see what is happening on the client when the backup is started. Check AV software: Is it scanning each file? Is it preventing files from being sent?
12-10-2012 10:53 PM
Hi,
Check this artical
STATUS CODE: 41 "Network connection timed out"
http://www.symantec.com/business/support/index?page=content&id=TECH37297
12-10-2012 11:19 PM
You need to give us more info: NBU version on master, media and database clients OS on master, media and database clients Which type of databases? Using db agents? Reboot of server means network drivers and services are reloaded. Best to find the cause. Create all relevant logs: On master: bprd On media server: bpbrm and bptm On client: bpcd and relevant db agent logs (most use dbclient) with write permission for db user on agent logs. Follow NBU process flow to check logs.
12-11-2012 01:29 AM
NBU Version Master, Media and Clients :- 7.1.0.4
OS Master , Media , Clients :- WINDOWS 2008 R2 SP1
Database :- MY SQL
Yes Using DB agents
12-11-2012 01:47 AM
Using MySQL agent from Zmanda?
Which server reboot resolves the problem? Master? Media? Db Client?
You need to concentrate troubleshooting on the server that needs the reboot.
Are filesystem backups working fine?
Have you tried to increase Client Connect and Client Read Timeouts on Master and Media servers?
The default is 5 minutes for both. Large database backups normally need more. We change it to 1800 (30 minutes) when we see timeout errors.
Have you created log folders as per my post above?
Use your Db agent manual for a list of logs on the client (in addition to bpcd).
Please post all text in Details tab of one of the failed jobs.
12-11-2012 02:26 PM
12-12-2012 09:46 PM
Hi All,
Thanks a lot for all you responses and i have checked with my SQL team regarding this i came to know they have enabled KASPERSKY ANTI VIRUS REAL TIME SCANNING few days back ,this might be the reason for these backups to fail ??
Thanks,
Nayab
12-12-2012 09:57 PM
You still have not given us enough info (see questions in my previous post).
We really can't say. If the AV is causing this, normal filesystem backups should also be affected.
12-12-2012 10:33 PM
Yes yes all backups are failing with Error 41 :(
12-12-2012 11:08 PM
NBU version on master, media and database clients
All Servers NBU :- 7.1.0.4
OS on master, media and database clients
All :- WINDOWS 2008 R2 SP1
Which type of databases?
SQL
Using db agents?
I can see the backup type as MS-WINDOWS backup
Reboot of server means network drivers and services are reloaded.
Reboot of CLIENT machine fixed the issue
Best to find the cause. Create all relevant logs:
On master: bprd
On media server: bpbrm and bptm
On client: bpcd and relevant db agent logs (most use dblient) with write permission for db user on agent logs.
All logs are in place already
12-12-2012 11:29 PM
Can you see if bpcd log on the client is written to?
If not, AV software is probably blocking ports.
If logs are written, incoming connections are allowed.
Check bpcd log for errors.
Check if client is attempting to connect back.
12-12-2012 11:32 PM
This is how the backup is failing :(
12-13-2012 12:14 AM
Looks like Client Read Timeout rather than Connect Timeout.
We see Connecting, then Connected, followed by bpbkar32 starting up.
This says to me that the client is trying to generate data stream but has not been able to do so within the 5 minute timeout period. You can increase the timeout on the media server, but better to fix the problem on the client.
You need to see what is happening on the client when the backup is started. Check AV software: Is it scanning each file? Is it preventing files from being sent?
Check bpbkar log on client.
12-13-2012 04:53 AM
Sorry i din find bpbkar and bpfis logs in LOGS , What would be the next step
12-13-2012 05:05 AM
Log folders do not exist by default.
You need to CREATE them as per my post 2 days ago (at the time we were under the impression that you were performing database agent backups).
12-13-2012 05:08 AM
Hi Please find the bpcd log
12-13-2012 05:23 AM
PLEASE read my post of earlier today (after you posted the screenshot) - connection is actually successful. This means we no longer need bpcd. We now need to troubleshoot the backup stream.
Looks like Client Read Timeout rather than Connect Timeout.
We see Connecting, then Connected, followed by bpbkar32 starting up.
This says to me that the client is trying to generate data stream but has not been able to do so within the 5 minute timeout period. You can increase the timeout on the media server, but better to fix the problem on the client.
You need to see what is happening on the client when the backup is started. Check AV software: Is it scanning each file? Is it preventing files from being sent?
Check bpbkar log on client.
12-13-2012 05:50 AM
Sorry Maria,
The above log file is Bpbkar log file could you please check it and let me know your findings. As i have created the Bpbkar log folder today and started a backup manually.
Thanks,
Nayab
12-13-2012 01:53 PM
Regarding "If the AV is causing this, normal filesystem backups should also be affected.", that may normally be true, however because of AV scanning the delay for the client read might be enough to throw it over that 5 minute limit (or whatever it is set to), whereas a file system backup may not have the same delay, especially since later the job details look like a client read timeout. Interested to see the outcome
12-13-2012 09:48 PM