Exchange Backup Failing 'file read failed(13)'

[Ultra317]

Our Netbackup 7.5.0.5 Exchange backups have started failing with 'file read failed(13)' errors.  Both servers (Exchange and Netbackup) are running on Server 2008 R2.  The mailstores (database & logs) are residing on a Netapp FAS2040 with SnapDrive managing snapshots which are created frequently without error.  The Netbackup policy itself is set to perform snapshot backups using VSS Provider Type 1 (System), without a DAG database source, has been backing up without any problems until some Windows updates including SP3 for Exchange 2010 were installed.

The job fails as follows:

28/11/2014 10:32:33 - Info nbjm(pid=5940) starting backup job (jobid=186119) for client exchange, policy EXCHANGE, schedule Full_Disc 
28/11/2014 10:32:33 - estimated 0 Kbytes needed
28/11/2014 10:32:33 - Info nbjm(pid=5940) started backup (backupid=exchange_1417170753) job for client exchange, policy EXCHANGE, schedule Full_Disc on storage unit NexsanDSU
28/11/2014 10:32:34 - started process bpbrm (484288)
28/11/2014 10:32:40 - Info bpbrm(pid=484288) exchange is the host to backup data from    
28/11/2014 10:32:40 - Info bpbrm(pid=484288) reading file list from client       
28/11/2014 10:32:40 - connecting
28/11/2014 10:32:43 - Info bpbrm(pid=484288) starting bpbkar32 on client        
28/11/2014 10:32:43 - connected; connect time: 00:00:03
28/11/2014 10:33:27 - Info bpbkar32(pid=11548) Backup started          
28/11/2014 10:33:27 - Info bptm(pid=484816) start           
28/11/2014 10:33:28 - Info bptm(pid=484816) using 262144 data buffer size       
28/11/2014 10:33:28 - Info bptm(pid=484816) setting receive network buffer to 1049600 bytes     
28/11/2014 10:33:28 - Info bptm(pid=484816) using 30 data buffers        
28/11/2014 10:33:29 - Info bptm(pid=484816) start backup          
28/11/2014 10:33:30 - Info bptm(pid=484816) backup child process is pid 482596.479660      
28/11/2014 10:33:30 - Info bptm(pid=482596) start           
28/11/2014 10:33:30 - begin writing
28/11/2014 11:07:53 - Error bpbrm(pid=484288) from client exchange: ERR - Terminating backup.     
28/11/2014 11:07:53 - Error bpbrm(pid=484288) from client exchange: ERR - failure reading file: Microsoft Information Store:\1GB Limit Mailbox Database\Logs_1417170539 (BEDS 0x0: )
28/11/2014 11:08:03 - Error bpbrm(pid=484288) could not send server status message      
28/11/2014 11:08:04 - Error bpbrm(pid=484288) cannot send mail to ...       
28/11/2014 11:08:05 - Info bpbkar32(pid=11548) done. status: 13: file read failed      
28/11/2014 11:08:05 - end writing; write time: 00:34:35
file read failed(13)

I have checked the VSS Writers on Exchange and all are showing as stable with no errors.  The Event Viewer on Exchange shows the following events just before the Netbackup job fails:

Log Name:      Application
Source:        MSExchangeIS
Date:          28/11/2014 11:06:35
Event ID:      9606
Task Category: Exchange VSS Writer
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      exchange
Description:
Exchange VSS Writer (instance 8315289c-83fc-4e18-95c6-2f9e76f0f821) has prepared for backup successfully.

Log Name:      Application
Source:        SnapManager for Exchange
Date:          28/11/2014 11:06:36
Event ID:      200
Task Category: Backup
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      exchange
Description:
SnapManager VSS asynchronous DoSnapshotSet operation started.

Log Name:      Application
Source:        ESE
Date:          28/11/2014 11:06:39
Event ID:      2007
Task Category: ShadowCopy
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      exchange
Description:
Information Store (6236) Shadow copy instance 3 aborted.

Log Name:      Application
Source:        MSExchangeIS
Date:          28/11/2014 11:06:39
Event ID:      9609
Task Category: Exchange VSS Writer
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      exchange
Description:
Exchange VSS Writer (instance 8315289c-83fc-4e18-95c6-2f9e76f0f821:3) failed with error code -2403 when preparing for Snapshot.

I appreciate this is probably an Exchange issue rather than a Netbackup one, but if anyone has an ideas as to why it is failing I'd be grateful for some help.

I have attached some log files, let me know if there is any other information that may help.

Thanks

[Marianne]

The important log that is missing here is bpbkar on the Exchange server.

You can see that bpbkar reported the error:

Info bpbkar32(pid=11548) done. status: 13: file read failed 

Oh, and you need to log a call with Microsoft about the VSS errors.
This is probably the cause of the failure. 

[Ultra317]

Thanks Marianne

I ran the job again and collected the bpbkar log (see attached) which contains 'ERR - invalid STRM header'. After a little searching I found out this may be related to SnapManager for Exchange, where it changes the log location after creating a snapshot to a softlink that Netbackup cannot follow. It recommended removing the Netapp SnapManager for Exchange, which I don't think is an option.

Could this be the main cause of the failure or shall I continue down the VSS error route?

Thanks

[ramwbp]

I am also seeing the same issue. Any solution yet?