Error 13 for Exchange Servers & Blackberry Enterprise Server

[yvonne_ewry]

Good morning to all and some :)

I'm having some issues in receiving an Error 13 for three of our Exchange servers and one Blackberry Enterprise Server when backing up ALL_LOCAL_DRIVES.  I'm using NetBackup 6.5.6.

I was wondering if something was wrong with the way I have the policies setup for each server and was wondering if someone could possibly give me some advice on how to set them up if they are indeed incorrect.  Here is the current setup.

For the three Exchange servers, the backup selections are:  ALL_LOCAL_DRIVES and Shadow Copy Components:\.  That always error's out with the Error 13:  Error bptm(pid=584) socket operation failed - 10054 (at child.c.1294)     
1/30/2012 7:23:11 PM - Error bptm(pid=584) unable to perform read from client socket, connection may have been broken
1/30/2012 7:23:16 PM - Error bpbrm(pid=3004) could not send server status message      
1/30/2012 7:23:18 PM - end writing; write time: 00:33:37
file read failed(13)

For the Blackberry Exchange server, the backup selections are:  System_State:\ and ALL_LOCAL_DRIVES.  This backup always error's out with the Error 13:   Error bptm(pid=7244) socket operation failed - 10054 (at child.c.1294)     
1/30/2012 9:54:46 PM - Error bptm(pid=7244) unable to perform read from client socket, connection may have been broken
1/30/2012 9:54:51 PM - Error bpbrm(pid=8140) could not send server status message      
1/30/2012 9:54:56 PM - end writing; write time: 00:33:50
file read failed(13)

Same error 13 as the Exchange servers.  We do not have any network latency or bandwidth problems as I'm able to complete differential and full backup's on about 30 other servers with no problem.

Any ideas?  Any suggestions, advice, comments is greatly appreciated.  I can provide more information if need be.

Thanks to all!

Y

[Mark_Solutions]

10054 tends to be a network issue but could just be a disconnect

The thing that stands out is that you have ALL_LOCAL_DRIVES and Shadow Copy Components in the policy file list

The ALL_LOCAL_DRIVES directive includes the Shadow Copy Components so you dont need both and if you are running multiple streams it may be trying to backup (and VSS Snapshot) the Shadow Copy twice at the same time

Remove the Shadow Copy Components from the selection list and, assuming they are Windows 2003 Servers get the latest VSS rollup package as well and check the System and Application and event logs to see what is reported at the time of the failure

One other thing to check - are these virtualised?

[yvonne_ewry]

We meet again Mark!  :)

Thanks for replying back.  Hopefully you can solve this one :)

I will remove the Shadow Copy Components from the policy as I was not aware (obviously lol) that ALL_LOCAL_DRIVES directive contained the SCC.  I'm not running multiple streams for these policies for the three Exchange servers and BES.

Yes, all three of them are Windows 2003 servers...no, they are not virutalized as well.

I will check the logs and see what I can do about getting the latest VSS rollup package.

Thanks!  Will let you know the outcome.

Y

[Mark_Solutions]

Yvonne

If it helps the latest roll up package for VSS for Windows 2003 should be this one:

http://support.microsoft.com/kb/833167

The issue may be caused by the servers being Exchange based so causing VSS issues

To pin it down further you can setup logging on the clients to give us more information - just create the following directories on the clients and once you have a failure post the logs on here for checking:

<install path>\program files\veritas\netbackup\logs\bpcd\
<install path>\program files\veritas\netbackup\logs\bpfis\
<install path>\program files\veritas\netbackup\logs\bpbkar\

[yvonne_ewry]

Well, I applied the VSS update rollup package to the NetBackup master server, rebooted and enabled the logs on one of the Exchange servers.  I didn't do it for the other two Exchange servers and BES as the Error 13's are all the same.  Still received an Error 13 after the full backup finished. 

I will attach the logs as attachments.  I created the bpfis directory for logging per your recommendation, but nothing showed up in that folder after the job finished with the Error 13.

If you wouldn't mind, would you please take a look and let me know if you see anything out of the ordinary?

Much appreciated.

Thanks,

Y

[Mark_Solutions]

Yvonne

The rollup package is for your exchange and BES servers rather than the Master Server.

And on that note you have this in the bobkar log:

8:29:30.021 AM: [6660.948] <16> VssSnapshot::V_VssVolumeSnapshot(): INF - VSS Volume Set prepare failed

Followed by this:

9:03:00.096 AM: [6660.948] <2> ov_log::V_GlobalLog: ERR - BEDS_AttachToDLE():FS_AttachToDLE() DeviceName:'System?State' BackupReason:0x400 Failed! (0xE000FEC9:A failure occurred accessing the Writer metadata.

9:03:00.096 AM: [6660.948] <2> tar_base::V_vTarMsgW: ERR - Unable to backup System State or Shadow Copy. Please check the state of VSS and associated Writers.9:03:00.096 AM: [6660.948] <4> dos_backup::tfs_scanstart: INF - Backup Special Objects request AFTER System State:

So go to a command line and run: vssadmin list writers

If any show as unstable or failed then apply the rollup package and reboot (the Exchange and BES Servers that is) - as I suspected the 10054 error is a red herring - this is all VSS related

[yvonne_ewry]

Hi Mark,

This is what I got after going to cmd -> vssadmin list writers

C:\Documents and Settings\silvaya>vssadmin list writers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001 Microsoft Corp.

Writer name: 'System Writer'
   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Instance Id: {68787137-8027-465b-873d-a8678b339e2b}
   State: [1] Stable
   Last error: No error

Writer name: 'MSDEWriter'
   Writer Id: {f8544ac1-0611-4fa5-b04b-f7ee00b03277}
   Writer Instance Id: {ba37fb33-be99-4da8-b736-0b7bd22bfdfc}
   State: [1] Stable
   Last error: No error

Writer name: 'Microsoft Exchange Writer'
   Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
   Writer Instance Id: {4498dd43-c5a3-4b9b-a584-6d2fe20e6728}
   State: [1] Stable
   Last error: No error

Writer name: 'Event Log Writer'
   Writer Id: {eee8c692-67ed-4250-8d86-390603070d00}
   Writer Instance Id: {7796fb63-9bb8-4529-9f92-6bd5bb59668b}
   State: [1] Stable
   Last error: No error

Writer name: 'COM+ REGDB Writer'
   Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
   Writer Instance Id: {6c7df179-f62c-4b5c-ab83-54911ec229b3}
   State: [1] Stable
   Last error: No error

Writer name: 'Registry Writer'
   Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
   Writer Instance Id: {78a1e3ee-0013-4d11-b4f8-8358a5b42b88}
   State: [1] Stable
   Last error: No error

Writer name: 'IIS Metabase Writer'
   Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}
   Writer Instance Id: {d14edc70-5463-44f5-b0de-0a7a328b3bcb}
   State: [1] Stable
   Last error: No error

Writer name: 'BITS Writer'
   Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}
   Writer Instance Id: {25b9c28b-0028-43a9-bf54-5a02305efeda}
   State: [1] Stable
   Last error: No error

Writer name: 'WMI Writer'
   Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
   Writer Instance Id: {f50cc9b2-ec31-437a-b7a9-9b10f9ab47fe}
   State: [1] Stable
   Last error: No error

No errors or failures =\

[Marianne]

Hi Yvonne

Since Exchange is backed up in a separate policy - have you excluded the Exchange files from normal MS-Windows-NT policy?

[yvonne_ewry]

Hi Marianne,

The Exchange is backed up in a separate policy; however for the MS-Windows-NT policy for the Exchange and BES servers, I just have ALL_LOCAL_DRIVES and System State:\.

Having the ALL_LOCAL_DRIVES policy for the servers, I understand, contains the Exchange files.  I'll have to dig and see specifically which Exchange files need to be excluded from the MS-Windows-NT policy backup.

Error bpbrm(pid=7624) from client hostxxxx: ERR - Unable to backup System State or Shadow Copy. Please check the state of VSS and associated Writers.ERR - Unable to backup System State or Shadow Copy. Please check the state of VSS and associated Writers.INF - Estimate:-1 -1
1/31/2012 11:21:43 AM - Error bpbrm(pid=7624) from client hostxxxx: ERR - Unable to backup System State or Shadow Copy. Please check the state of VSS and associated Writers.INF - EXIT STATUS 69: invalid filelist specification
1/31/2012 11:21:43 AM - Error bptm(pid=4888) socket operation failed - 10054 (at child.c.1294)     
1/31/2012 11:21:43 AM - Error bptm(pid=4888) unable to perform read from client socket, connection may have been broken
1/31/2012 11:21:47 AM - Error bpbrm(pid=7624) could not send server status message      
1/31/2012 11:21:49 AM - end writing; write time: 00:58:14
file read failed(13)

[Mark_Solutions]

Yvonne

Marianne makes a good point about excluding the Exchange files (you can always set a wildcard of *.edb and *.stm which should sort it pretty much but niced to exclude the actual exchange database directories)

I would still suggest applying the rollup package to the Exchange Servers which may also clear this error for you - especially as the VSS wroiters all report being in a stable state

[yvonne_ewry]

Where in NetBackup do I exclude these Exchange database files?  I see I can do it in the client area, but then that means the backup's for the Exchange mailboxes will fail...

[Marianne]

Exchange backups will not fail because the agent is backing up the contents of the databases - not filenames.

If you want to be 100% sure, exclude the data files for the filesystem policy name.

[Marianne]

Reading again through previous post:

Have you removed System State and/or System State from Backup Selection yet?

Have you created bpfis folder on the Exchange and BlackBerry servers?

Please enable multiple data streams - better to have one stream failing and troubleshoot only that while the rest of the drive letters will hopefully go through.