Backup failing with error code media write error (84)

[dmendhekar]

21/5/2015 5:04:04 PM - Info nbjm(pid=4052) starting backup job (jobid=378) for client mhpappfp02-bk, policy FP02-DD_CompanyFolder, schedule Full
21/5/2015 5:04:04 PM - Info nbjm(pid=4052) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=378, request id:{B71F1724-F9E1-4939-92B7-27F2A0B0A528})
21/5/2015 5:04:04 PM - requesting resource FP02
21/5/2015 5:04:04 PM - requesting resource mhpappbk04.healthgrp.com.sg.NBU_CLIENT.MAXJOBS.mhpappfp02-bk
21/5/2015 5:04:04 PM - requesting resource mhpappbk04.healthgrp.com.sg.NBU_POLICY.MAXJOBS.FP02-DD_CompanyFolder
21/5/2015 5:04:04 PM - granted resource mhpappbk04.healthgrp.com.sg.NBU_CLIENT.MAXJOBS.mhpappfp02-bk
21/5/2015 5:04:04 PM - granted resource mhpappbk04.healthgrp.com.sg.NBU_POLICY.MAXJOBS.FP02-DD_CompanyFolder
21/5/2015 5:04:04 PM - granted resource MediaID=@aaaao;Path=\\172.10.10.2\nbu\fp02-bk;MediaServer=mhpappbk04.healthgrp.com.sg
21/5/2015 5:04:04 PM - granted resource FP02
21/5/2015 5:04:05 PM - estimated 0 Kbytes needed
21/5/2015 5:04:05 PM - Info nbjm(pid=4052) started backup (backupid=mhpappfp02-bk_1432199045) job for client mhpappfp02-bk, policy FP02-DD_CompanyFolder, schedule Full on storage unit FP02
21/5/2015 5:04:05 PM - started process bpbrm (4172)
21/5/2015 5:04:06 PM - Info bpbrm(pid=4172) mhpappfp02-bk is the host to backup data from
21/5/2015 5:04:06 PM - Info bpbrm(pid=4172) reading file list for client
21/5/2015 5:04:06 PM - connecting
21/5/2015 5:04:07 PM - Info bpbrm(pid=4172) starting bpbkar32 on client
21/5/2015 5:04:07 PM - connected; connect time: 0:00:01
21/5/2015 5:04:08 PM - Info bpbkar32(pid=2652) Backup started
21/5/2015 5:04:08 PM - Info bpbkar32(pid=2652) change time comparison:<disabled>
21/5/2015 5:04:08 PM - Info bpbkar32(pid=2652) archive bit processing:<enabled>
21/5/2015 5:04:08 PM - Info bptm(pid=4428) start
21/5/2015 5:04:08 PM - Info bpbkar32(pid=2652) not using change journal data for <D:\Company_Folder\HIP>: not enabled
21/5/2015 5:04:09 PM - Info bptm(pid=4428) using 262144 data buffer size
21/5/2015 5:04:09 PM - Info bptm(pid=4428) setting receive network buffer to 1049600 bytes
21/5/2015 5:04:09 PM - Info bptm(pid=4428) using 30 data buffers
21/5/2015 5:04:09 PM - Info bptm(pid=4428) start backup
21/5/2015 5:04:09 PM - Info bptm(pid=4428) backup child process is pid 6916.5464
21/5/2015 5:04:09 PM - begin writing
21/5/2015 5:04:10 PM - Info bptm(pid=6916) start
21/5/2015 5:44:45 PM - Info bpbkar32(pid=2652) 4998 entries sent to bpdbm
21/5/2015 6:05:16 PM - Info bpbkar32(pid=2652) 9998 entries sent to bpdbm
21/5/2015 6:50:47 PM - Info bpbkar32(pid=2652) 14998 entries sent to bpdbm
21/5/2015 8:12:03 PM - Critical bptm(pid=4428) image write failed: error 2060017: system call failed
21/5/2015 8:15:05 PM - Critical bptm(pid=4428) sts_close_handle failed: 2060017 system call failed
21/5/2015 8:16:01 PM - Error bptm(pid=4428) cannot write image to disk, Invalid argument
21/5/2015 8:16:01 PM - Info bptm(pid=4428) EXITING with status 84 <----------
21/5/2015 8:16:05 PM - Error bpbrm(pid=4172) cannot send mail to boonhui.er@mohh.com.sg
21/5/2015 8:16:05 PM - Info bpbkar32(pid=2652) done. status: 84: media write error
21/5/2015 8:16:05 PM - end writing; write time: 3:11:56
media write error (84)

=========================================================================

Storage Dell DR400 is configured as a basic disk.

=========================================================================

 policy type is MS-Windows, Storage unit is FP02 which is a DELL DR 4000 storage.

=========================================================================

Errors in bptm logs:

16:04:04.648 [6308.2080] <16> 361:bptm:6308:mhpappbk04.healthgrp.com.sg: STSImageManager: write() failed on image file \\172.10.10.2\nbu\fp02-bk\mhpappfp02-bk_1432194317_C1_F1.1432194317.img (error=64, Unknown error)
16:04:04.648 [6308.2080] <16> 361:bptm:6308:mhpappbk04.healthgrp.com.sg: STSImageManager: Caught exception in STSIMGImageManager::WriteImage, FILE=../STSIMGImage.cpp, FUNCTION=STSIMGImage::Write, LINE=812, ERROR=STS_ESYS
16:04:04.648 [6308.2080] <32> do_write_image: sts_write_image failed: byteswritten = 0, len = 262144
16:04:04.648 [6308.2080] <32> do_write_image: sts_write_image failed: error 2060017 (0 262144 0)
16:04:04.648 [6308.2080] <2> set_job_details: Tfile (361): LOG 1432195444 32 bptm 6308 image write failed: error 2060017: system call failed
16:04:04.648 [6308.2080] <2> send_job_file: job ID 361, ftype = 3 msg len = 82, msg = LOG 1432195444 32 bptm 6308 image write failed: error 2060017: system call failed
16:04:04.648 [6308.2080] <32> write_data: image write failed: error 2060017:
16:04:04.648 [6308.2080] <2> bp_sts_close_image: 00000000020AB560 0 <mhpappfp02-bk_1432194317_C1_F1 0 1> 1 4 0 1 1 0 complete flag (8)(M_FRAG_COMPLETE ), close type (1)(STS_CLOSEF_IH_COMPLETE ) 11
16:06:52.115 [6308.2080] <16> 361:bptm:6308:mhpappbk04.healthgrp.com.sg: STSImageManager: Caught exception in STSIMGImageManager::CloseImage, FILE=../STSIMGImage.cpp, FUNCTION=STSIMGImage::Close, LINE=684, ERROR=STS_ESYS
16:06:52.115 [6308.2080] <2> set_job_details: Tfile (361): LOG 1432195612 32 bptm 6308 sts_close_handle failed: 2060017 system call failed
16:06:52.115 [6308.2080] <2> send_job_file: job ID 361, ftype = 3 msg len = 80, msg = LOG 1432195612 32 bptm 6308 sts_close_handle failed: 2060017 system call failed

[watsons]

Netbackup version? 

Try a different storage unit on a different storage server to see if it works. This can tell easily if the Dell storage is having disk issue. You should see a log entry in event viewer if there is a disk problem.

Otherwise, check bptm logs.

[mph999]

Did this every work, any changes recently ?

Does it always fail, or intermittant  - do other backups to this storage work ?

Does it always fail at the same point ?

This is the key line I think :

16:04:04.648 [6308.2080] <16> 361:bptm:6308:mhpappbk04.healthgrp.com.sg: STSImageManager: write() failed on image file \\172.10.10.2\nbu\fp02-bk\mhpappfp02-bk_1432194317_C1_F1.1432194317.img (error=64, Unknown error)

error=64 is a Windows error

From:

The specified network name is no longer available.

https://msdn.microsoft.com/en-us/library/windows/desktop/ms681382(v=vs.85).aspx

ERROR_NETNAME_DELETED

64 (0x40)

The specified network name is no longer available

So it seems you have a network error /issue, not a NetBackup issue.  Firstly, I would disable any threat protection software (antivirus etc) just as a test to see if that is the cause.  Sometimes disabling anti-virus isn't enough, and it has to be completely uninstalled to eliminate it from being a cause.

Failing the above, I'd suggest speaking with your Network team.

[dmendhekar]

Did this ever work, any changes recently ?

No changes. This is a new environment.

Does it always fail, or intermittant  - do other backups to this storage work ?

Yes other backup works for this storage. Issue is only with three folders which is having almost 4 TB data.

Does it always fail at the same point ?

Yes it is failing at the same point.

I have attched the extracted bptm logs.

NBU Master/Media server: 7.6.1.1 Win2012R2

Client  mhpappfp02-bk Os: Win 2012 R2 Standard 7.6.1.1

[Marianne · Accepted solution]

As per Martin's post : this looks like a network error. You need to involve network admins. Issues like these are difficult to troubleshoot because the storage is remote network location. NBU is reporting the error, not causing it.

[dmendhekar]

Thank You Marianne. I will check with my Network team to look into this issue.