NDMP restore failing with media error status 85

[Dollypee]

Netbackup : 7.5.0.5

Master server: Win2k3R2

Media server: Win2k8R2

Problem: NDMP restore is consitently failing with media write error EXIT STATUS 85. The tricky aspect of this error is that, customer confirmed he could see the restored files but from activity monitor and BAR, restore consistently failing with media write errors.

Please see below logs from activity monitor:

8/14/2013 4:05:47 PM - begin Restore
8/14/2013 4:05:50 PM - restoring image XXXXXXX_1374872410
8/14/2013 4:05:50 PM - requesting resource @aaaag
8/14/2013 4:05:50 PM - granted resource MediaID=@aaaag;DiskVxxxxxxxxxx
8/14/2013 4:05:51 PM - Info bprd(pid=41860) Restoring from copy 1 of image created 07/26/13 17:00:10   
8/14/2013 4:05:54 PM - Info bpbrm(pid=5280) 172.24.191.61 is the host to restore to     
8/14/2013 4:05:54 PM - Info bpbrm(pid=5280) reading file list from client       
8/14/2013 4:05:56 PM - connecting
8/14/2013 4:05:56 PM - Info bpbrm(pid=5280) starting bptm          
8/14/2013 4:05:57 PM - Info ndmpagent(pid=7252) Restore started          
8/14/2013 4:05:57 PM - connected; connect time: 00:00:01
8/14/2013 4:05:57 PM - Info bptm(pid=1768) start           
8/14/2013 4:05:57 PM - started process bptm (1768)
8/14/2013 4:05:57 PM - Info bpdm(pid=1768) reading backup image         
8/14/2013 4:06:04 PM - begin reading
8/14/2013 4:06:06 PM - Info ndmpagent(pid=7252) INF - Restoring NDMP files from /__VOLUME__/xxxxxxxxxxxxx 
8/14/2013 4:06:08 PM - Info ndmpagent(pid=7252) INF - Restoring NDMP files from [See line above] to /__VOLUME__/xxxxxxxxxxxxxxxxxx 
8/14/2013 4:06:13 PM - Info ndmpagent(pid=7252) DAR enabled          
8/14/2013 4:06:17 PM - Info ndmpagent(pid=7252) 172.24.191.61: NDMP(371545): Starting (destination end) Recovery of files to /abcd  
8/14/2013 4:06:21 PM - Critical bptm(pid=1768) sts_read_image failed: error 2060046 bytes Read = 0plugin error   
8/14/2013 4:06:21 PM - Critical bptm(pid=1768) image read failed: error 2060046: plugin error     
8/14/2013 4:06:22 PM - Error bptm(pid=1768) cannot read image from disk, Invalid argument     
8/14/2013 4:06:22 PM - Error ndmpagent(pid=7252) NDMP restore failed from path /abcd      
8/14/2013 4:06:24 PM - Info ndmpagent(pid=7252) done. status: 150         
8/14/2013 4:06:24 PM - Info bptm(pid=1768) EXITING with status 85 <----------       
8/14/2013 4:06:26 PM - Critical bpbrm(pid=5280) unexpected termination of client 172.24.191.61       
8/14/2013 4:06:26 PM - Info ndmpagent(pid=7252) done. status: 85: media read error      
8/14/2013 4:06:26 PM - Error bpbrm(pid=5280) client restore EXIT STATUS 85: media read error    
8/14/2013 4:06:26 PM - restored image xxxxxxx_1374872410 - (media read error(85)); restore time 00:00:36
8/14/2013 4:06:30 PM - end Restore; elapsed time: 00:00:43
media read error(85)

[Yasuhisa_Ishika]

What type of storage unit and storages you used? Please tell us model and version of both backup source and backup destination.

[Wiriadi_Wangsa]

Hi Dolly,

Also, you might want to check the following logs in media server: bpbrm, bpdm, bptm and ndmpagent.

The PIDs for each process are mentioned in the job detail, so they would be a snap to find. 

Cheers.

[Dollypee]

Yasuhisa Ishikawa: Am using datadomain DD990 as my storage units.And my backup source and destination is on Hitachi.

[Marianne]

The only other reference to 'error 2060046 plugin error'  that I could find was this unresolved forum post that seemed to have been caused by a specific DD plugin version:
https://www-secure.symantec.com/connect/forums/status-code-83-when-using-datadomain-ost-plugin-2211

You may want to check the Data Domain logs as well.

[Nicolai]

Do you relly have a  share/volume calld /abcd ? Possible configuation typo ?

8/14/2013 4:06:17 PM - Info ndmpagent(pid=7252) 172.24.191.61: NDMP(371545): Starting (destination end) Recovery of files to /abcd  

8/14/2013 4:06:21 PM - Critical bptm(pid=1768) sts_read_image failed: error 2060046 bytes Read = 0plugin error   
8/14/2013 4:06:21 PM - Critical bptm(pid=1768) image read failed: error 2060046: plugin error     
8/14/2013 4:06:22 PM - Error bptm(pid=1768) cannot read image from disk, Invalid argument     
8/14/2013 4:06:22 PM - Error ndmpagent(pid=7252) NDMP restore failed from path /abcd

[Dollypee]

Hi Nichola, ..please ignore /abcd. I had replaced the actual share name with /abcd for confidential reasons. But the logs remain as it is with the exemption of the share name.  

Thanks.

[Dollypee]

<16> 162242:bptm:3500:crppms1vmmprd24.nychhc.org: [DAC:2E80] ddp_read() failed Offset 342283913728, BytesToRead 1049088, BytesRead 0 Err: 5008-Invalid Input
<16> 162242:bptm:3500:crppms1vmmprd24.nychhc.org: D:\Program Files\Veritas\NetBackup\bin\\ost-plugins\libstspiDataDomain.dll:stspi_read_image STS_EPLUGIN [DDErrNo = 5008 (invalid argument)]

<32> bp_sts_read_image_non_512: sts_read_image failed: error 2060046 bytes Read = 0

Hello team, I also encountered this error today after trying to troubleshoot this porblem:

[smurphy]

Try temporarily disabling DAR to see if it makes a difference.  It's in the Host Properties of the master server, I forget which section but it is a checkbox labeled "Use direct access recovery for NDMP restores".
This was more common with 6.5.x media servers (regardless of the version of the master) but is still worth a try.
Otherwise, gather the logs described here:

http://www.symantec.com/docs/TECH56492

and consider opening a Support case.

[watsons]

The error message looks like the plugin error between DataDomain & the NDMP data read.

To narrow down the cause, try switching the storage unit to be a BasicDisk or disk and see if that works fine - this can tell if the plugin to the DataDomain was having problem.

In addition to the ndmp logs, consider also enabling DebugLevel=6 \netbackup\logs\nbrmms\  (In unified logs, the OID is 222)  for more details.

[Dollypee]

What is weird here is that, there is no issue with every other NDMPs in the same environment using the same storage unit which is the same datadomain as well as same OST version. backup and restore is good except this particular ndmp in question. Smurphy....I have case opened already with both symantec & datadomain. I have generated same logs to symantec and they found above errors on the logs. Right now, symantec is waiting on datadomain to provide answers to thier findings. As for datadomain, they could not find specific problem on the support bundles they had requested for. They intend raising the debug level to see if they can find any further errors.

Watson, thanks for the idea...am going to test this out asap on the policy and see how it goes.

Thank you all for all the support.

[smurphy]

Ok, understood.
I would still recommend talking to your Symantec tech about disabling DAR as a test.

[Dollypee]

Smurphy...Disabling DAR did not resolve the problem..

[Will_Restore]

[DDErrNo = 5008 (invalid argument)] in your log above sure looks like an Data Domain problem to me.  Sorry I don't have a fix other than to escalate case with EMC.