01-31-2012 06:51 AM
Good morning to all and some :)
I'm having some issues in receiving an Error 13 for three of our Exchange servers and one Blackberry Enterprise Server when backing up ALL_LOCAL_DRIVES. I'm using NetBackup 6.5.6.
I was wondering if something was wrong with the way I have the policies setup for each server and was wondering if someone could possibly give me some advice on how to set them up if they are indeed incorrect. Here is the current setup.
For the three Exchange servers, the backup selections are: ALL_LOCAL_DRIVES and Shadow Copy Components:\. That always error's out with the Error 13: Error bptm(pid=584) socket operation failed - 10054 (at child.c.1294)
1/30/2012 7:23:11 PM - Error bptm(pid=584) unable to perform read from client socket, connection may have been broken
1/30/2012 7:23:16 PM - Error bpbrm(pid=3004) could not send server status message
1/30/2012 7:23:18 PM - end writing; write time: 00:33:37
file read failed(13)
For the Blackberry Exchange server, the backup selections are: System_State:\ and ALL_LOCAL_DRIVES. This backup always error's out with the Error 13: Error bptm(pid=7244) socket operation failed - 10054 (at child.c.1294)
1/30/2012 9:54:46 PM - Error bptm(pid=7244) unable to perform read from client socket, connection may have been broken
1/30/2012 9:54:51 PM - Error bpbrm(pid=8140) could not send server status message
1/30/2012 9:54:56 PM - end writing; write time: 00:33:50
file read failed(13)
Same error 13 as the Exchange servers. We do not have any network latency or bandwidth problems as I'm able to complete differential and full backup's on about 30 other servers with no problem.
Any ideas? Any suggestions, advice, comments is greatly appreciated. I can provide more information if need be.
Thanks to all!
Y
01-31-2012 07:12 AM
10054 tends to be a network issue but could just be a disconnect
The thing that stands out is that you have ALL_LOCAL_DRIVES and Shadow Copy Components in the policy file list
The ALL_LOCAL_DRIVES directive includes the Shadow Copy Components so you dont need both and if you are running multiple streams it may be trying to backup (and VSS Snapshot) the Shadow Copy twice at the same time
Remove the Shadow Copy Components from the selection list and, assuming they are Windows 2003 Servers get the latest VSS rollup package as well and check the System and Application and event logs to see what is reported at the time of the failure
One other thing to check - are these virtualised?
01-31-2012 07:41 AM
We meet again Mark! :)
Thanks for replying back. Hopefully you can solve this one :)
I will remove the Shadow Copy Components from the policy as I was not aware (obviously lol) that ALL_LOCAL_DRIVES directive contained the SCC. I'm not running multiple streams for these policies for the three Exchange servers and BES.
Yes, all three of them are Windows 2003 servers...no, they are not virutalized as well.
I will check the logs and see what I can do about getting the latest VSS rollup package.
Thanks! Will let you know the outcome.
Y
01-31-2012 07:48 AM
Yvonne
If it helps the latest roll up package for VSS for Windows 2003 should be this one:
http://support.microsoft.com/kb/833167
The issue may be caused by the servers being Exchange based so causing VSS issues
To pin it down further you can setup logging on the clients to give us more information - just create the following directories on the clients and once you have a failure post the logs on here for checking:
<install path>\program files\veritas\netbackup\logs\bpcd\
<install path>\program files\veritas\netbackup\logs\bpfis\
<install path>\program files\veritas\netbackup\logs\bpbkar\
01-31-2012 09:23 AM
Well, I applied the VSS update rollup package to the NetBackup master server, rebooted and enabled the logs on one of the Exchange servers. I didn't do it for the other two Exchange servers and BES as the Error 13's are all the same. Still received an Error 13 after the full backup finished.
I will attach the logs as attachments. I created the bpfis directory for logging per your recommendation, but nothing showed up in that folder after the job finished with the Error 13.
If you wouldn't mind, would you please take a look and let me know if you see anything out of the ordinary?
Much appreciated.
Thanks,
Y
01-31-2012 09:29 AM
Yvonne
The rollup package is for your exchange and BES servers rather than the Master Server.
And on that note you have this in the bobkar log:
8:29:30.021 AM: [6660.948] <16> VssSnapshot::V_VssVolumeSnapshot(): INF - VSS Volume Set prepare failed
Followed by this:
9:03:00.096 AM: [6660.948] <2> ov_log::V_GlobalLog: ERR - BEDS_AttachToDLE():FS_AttachToDLE() DeviceName:'System?State' BackupReason:0x400 Failed! (0xE000FEC9:A failure occurred accessing the Writer metadata.
9:03:00.096 AM: [6660.948] <2> tar_base::V_vTarMsgW: ERR - Unable to backup System State or Shadow Copy. Please check the state of VSS and associated Writers.9:03:00.096 AM: [6660.948] <4> dos_backup::tfs_scanstart: INF - Backup Special Objects request AFTER System State:
So go to a command line and run: vssadmin list writers
If any show as unstable or failed then apply the rollup package and reboot (the Exchange and BES Servers that is) - as I suspected the 10054 error is a red herring - this is all VSS related
01-31-2012 09:34 AM
Hi Mark,
This is what I got after going to cmd -> vssadmin list writers
C:\Documents and Settings\silvaya>vssadmin list writers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001 Microsoft Corp.
Writer name: 'System Writer'
Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
Writer Instance Id: {68787137-8027-465b-873d-a8678b339e2b}
State: [1] Stable
Last error: No error
Writer name: 'MSDEWriter'
Writer Id: {f8544ac1-0611-4fa5-b04b-f7ee00b03277}
Writer Instance Id: {ba37fb33-be99-4da8-b736-0b7bd22bfdfc}
State: [1] Stable
Last error: No error
Writer name: 'Microsoft Exchange Writer'
Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
Writer Instance Id: {4498dd43-c5a3-4b9b-a584-6d2fe20e6728}
State: [1] Stable
Last error: No error
Writer name: 'Event Log Writer'
Writer Id: {eee8c692-67ed-4250-8d86-390603070d00}
Writer Instance Id: {7796fb63-9bb8-4529-9f92-6bd5bb59668b}
State: [1] Stable
Last error: No error
Writer name: 'COM+ REGDB Writer'
Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
Writer Instance Id: {6c7df179-f62c-4b5c-ab83-54911ec229b3}
State: [1] Stable
Last error: No error
Writer name: 'Registry Writer'
Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
Writer Instance Id: {78a1e3ee-0013-4d11-b4f8-8358a5b42b88}
State: [1] Stable
Last error: No error
Writer name: 'IIS Metabase Writer'
Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}
Writer Instance Id: {d14edc70-5463-44f5-b0de-0a7a328b3bcb}
State: [1] Stable
Last error: No error
Writer name: 'BITS Writer'
Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}
Writer Instance Id: {25b9c28b-0028-43a9-bf54-5a02305efeda}
State: [1] Stable
Last error: No error
Writer name: 'WMI Writer'
Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
Writer Instance Id: {f50cc9b2-ec31-437a-b7a9-9b10f9ab47fe}
State: [1] Stable
Last error: No error
No errors or failures =\
01-31-2012 10:09 AM
Hi Yvonne
Since Exchange is backed up in a separate policy - have you excluded the Exchange files from normal MS-Windows-NT policy?
01-31-2012 01:13 PM
Hi Marianne,
The Exchange is backed up in a separate policy; however for the MS-Windows-NT policy for the Exchange and BES servers, I just have ALL_LOCAL_DRIVES and System State:\.
Having the ALL_LOCAL_DRIVES policy for the servers, I understand, contains the Exchange files. I'll have to dig and see specifically which Exchange files need to be excluded from the MS-Windows-NT policy backup.
Error bpbrm(pid=7624) from client hostxxxx: ERR - Unable to backup System State or Shadow Copy. Please check the state of VSS and associated Writers.ERR - Unable to backup System State or Shadow Copy. Please check the state of VSS and associated Writers.INF - Estimate:-1 -1
1/31/2012 11:21:43 AM - Error bpbrm(pid=7624) from client hostxxxx: ERR - Unable to backup System State or Shadow Copy. Please check the state of VSS and associated Writers.INF - EXIT STATUS 69: invalid filelist specification
1/31/2012 11:21:43 AM - Error bptm(pid=4888) socket operation failed - 10054 (at child.c.1294)
1/31/2012 11:21:43 AM - Error bptm(pid=4888) unable to perform read from client socket, connection may have been broken
1/31/2012 11:21:47 AM - Error bpbrm(pid=7624) could not send server status message
1/31/2012 11:21:49 AM - end writing; write time: 00:58:14
file read failed(13)
02-01-2012 01:58 AM
Yvonne
Marianne makes a good point about excluding the Exchange files (you can always set a wildcard of *.edb and *.stm which should sort it pretty much but niced to exclude the actual exchange database directories)
I would still suggest applying the rollup package to the Exchange Servers which may also clear this error for you - especially as the VSS wroiters all report being in a stable state
02-01-2012 09:17 AM
Where in NetBackup do I exclude these Exchange database files? I see I can do it in the client area, but then that means the backup's for the Exchange mailboxes will fail...
02-01-2012 09:59 AM
Exchange backups will not fail because the agent is backing up the contents of the databases - not filenames.
If you want to be 100% sure, exclude the data files for the filesystem policy name.
02-01-2012 10:08 AM
Reading again through previous post:
Have you removed System State and/or System State from Backup Selection yet?
Have you created bpfis folder on the Exchange and BlackBerry servers?
Please enable multiple data streams - better to have one stream failing and troubleshoot only that while the rest of the drive letters will hopefully go through.