cancel
Showing results for 
Search instead for 
Did you mean: 

Error bptm (pid=214108) cannot write image to disk, media close failed with status 2060019

dugga
Level 4

08/21/2017 18:09:20 - Critical bptm (pid=25890) sts_close_handle failed: 2060019 error occurred on network socket
08/21/2017 18:09:20 - Error bptm (pid=25890) cannot write image to disk, media close failed with status 2060019

I get these errors on different clients at time, and whenever i re-run the backup it runs successfully.

I don't see a technote which explains a permanent fix for this/ an explanation on why 2060019 occurs

Can somebody help to fix such errors from further occurences.

 

 

Netbackup Master Server : 7.7.3

Media Server ( Netbackup 5220 appliance : 2.6.0.3)

 

11 REPLIES 11

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Google found the same error here: 

https://www.veritas.com/connect/forums/after-upgrading-appliance-2602-backups-fail-intermittantly-er...

and 

https://vox.veritas.com/t5/NetBackup/sts-close-handle-failed-2060019-error-occurred-on-network-socke...

Which then leads to this TN: https://www.veritas.com/support/en_US/article.TECH224031

So, the issue was introduced by upgrade/installation of 2.6.0.x:

A change introduced in NetBackup appliance version 2.6.0.x disabled the kernel parameter tcp_timestamps.

.....

Workaround:
Until the affected NetBackup 52x0 Appliance can be patched to version 2.6.1, re-enable tcp_timestamps to resolve the issue:

PLEASE upgrade your appliance to match master server software (2.7.3).
All Appliance and NBU versions up to 7.6.x (2.6.x) ran out of support 6 months ago.

 

Thanks for the update @Marianne.

I have changed the tcp_timestamps as a temporary workaround , the technotes talks about status code 24, the error which we are getting are 84 and 87 which are also related to socket errors : media close error, media write errors.

The upgrade is under plan, also we are in an out of support version, so bit afraid on who would support if we encounter issues during upgrade

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

You never showed us all text for failed backup, right? All you gave us was error 2060019.

Hello,

 

If you have purchased supported version license and have support contract with Veritas.

You can open case with Support team for assistance. Veritas Support team will help you incase of any upgrade failure.

Regards,

I'm sorry, i didnt post because this was one common error message in all those 3 status messages

File system backup - linux

Critical bptm (pid=25890) Storage Server Error: (Storage server: PureDisk: nbu_app1_sg ) mtstrm_close_write_channel: Fatal error occured in Multi-Threaded Agent: Close Write Channel command failed: Cr_ErrnoException: Timed out after waiting 180s for the push threads to complete (cnt=1) V-454-96
Critical bptm (pid=25890) sts_close_handle failed: 2060019 error occurred on network socket
Error bptm (pid=25890) cannot write image to disk, media close failed with status 2060019
Info bptm (pid=25890) EXITING with status 87 <----------
Info nbu_app1_sg (pid=25890) StorageServer=PureDisk: nbu_app1_sg; Report=PDDO Stats (multi-threaded stream used) for (nbu_app1_sg): scanned: 59023 KB, CR sent: 588 KB, CR sent over FC: 0 KB, dedup: 99.0%, cache disabled
Info bpbkar (pid=25297) done. status: 87: media close error
end writing; write time: 0:07:16
job 3348315 was restarted as job 3349557
media close error (87)

 

 

NDMP Backup

Info ndmpagent (pid=6025) Netapp11-r: DUMP: Sat Aug 19 06:21:23 2017 : We have written 408290539 KB.
Critical bptm (pid=6026) Storage Server Error: (Storage server: PureDisk:nbu_app1_sg) mtstrm_write_segment: Fatal error occured in Multi-Threaded Agent: Timed out after waiting 1200s to send data to mtstrmd on stream /Netapp11-r/NDMP_BDC_Netapp11/Netapp11-r_1503082512_C1_F16.img V-454-95
Critical bptm (pid=6026) image write failed: error 2060019: error occurred on network socket
Critical bptm (pid=6026) sts_close_handle failed: 2060011 offset invalid for object or context
Error bptm (pid=6026) cannot write image to disk, Invalid argument
Error ndmpagent (pid=6025) NDMP backup failed, path = /vol/Home
Error ndmpagent (pid=6025) Netapp11-r: DUMP: Write to socket failed
Error ndmpagent (pid=6025) Netapp11-r: DUMP: DUMP IS ABORTED
Info ndmpagent (pid=6025) Netapp11-r: DUMP: Deleting "/vol/Home/../snapshot_for_backup.256558" snapshot.
Error ndmpagent (pid=6025) Netapp11-r: DATA: Operation terminated (for /vol/Home).
Info bptm (pid=6026) EXITING with status 84 <----------
Info nbu_app1_sg (pid=6026) StorageServer=PureDisk:nbu_app1_sg; Report=PDDO Stats (multi-threaded stream used) for (nbu_app1_sg): scanned: 408332175 KB, CR sent: 1529226 KB, CR sent over FC: 0 KB, dedup: 99.6%, cache hits: 2303 (0.1%), rebased: 1212 (0.1%)
Info ndmpagent (pid=0) done. status: 84: media write error
end writing; write time: 10:12:28
media write error (84)

 

VMware Backup

Info bpbkar (pid=5283) 1071162 entries sent to bpdbm
Critical bptm (pid=5284) Storage Server Error: (Storage server: PureDisk:nbu_app1_sg) mtstrm_write_segment: Fatal error occured in Multi-Threaded Agent: Timed out after waiting 1200s to send data to mtstrmd on stream /CL00110v/VMware_Policy/CL00110v_1503114421_C1_F7.img V-454-95
Critical bptm (pid=5284) image write failed: error 2060019: error occurred on network socket
Error bptm (pid=5284) cannot write image to disk, Invalid argument
Info bptm (pid=5284) EXITING with status 84 <----------
Error bpbrm (pid=5276) from client CL00110v: ERR - bpbkar exiting because backup is aborting
Critical bpbrm (pid=5276) from client CL00110v: FTL - cleanup() failed, status 40
Info nbu_app1_sg (pid=5284) StorageServer=PureDisk:nbu_app1_sg; Report=PDDO Stats (multi-threaded stream used) for (nbu_app1_sg): scanned: 157018066 KB, CR sent: 35903487 KB, CR sent over FC: 0 KB, dedup: 77.1%, cache hits: 148005 (10.8%), rebased: 277683 (20.2%)
Info bpbkar (pid=0) done. status: 84: media write error
end writing; write time: 3:25:18
media write error (84)

Oracle Backup

Info dbclient (pid=27001802) dbclient(pid=27001802) wrote first buffer(size=262144)
Critical bptm (pid=24919) Storage Server Error: (Storage server: PureDisk:nbu_app1_bdc) mtstrm_write_segment: Fatal error occured in Multi-Threaded Agent: Timed out after waiting 1200s to send data to mtstrmd on stream /prodora/Oracle_Prod/prodora_1503203777_C1_F8.img V-454-95
Critical bptm (pid=24919) image write failed: error 2060019: error occurred on network socket
Error bptm (pid=24919) cannot write image to disk, Invalid argument
Info dbclient (pid=27001802) done. status: 6
Info bptm (pid=24919) EXITING with status 84 <----------
Info nbu_app1_sg (pid=24919) StorageServer=PureDisk:nbu_app1_bdc; Report=PDDO Stats (multi-threaded stream used) for (nbu_app1_bdc): scanned: 198964235 KB, CR sent: 116149641 KB, CR sent over FC: 0 KB, dedup: 41.6%, cache hits: 0 (0.0%)
Info dbclient (pid=27001802) done. status: 84: media write error
end writing; write time: 4:00:54
media write error (84)

@Tousif  : We only have the standard support license, as we are in an out-of-support version we need to purchase and extended support, our management has not agreed to it yet.

Thiago_Ribeiro
Moderator
Moderator
Partner    VIP    Accredited

Hi @dugga,

According to those logs that you posted I saw..Did you change/increase the timed out value of Multi-Threaded Agent ?

Critical bptm (pid=24919) Storage Server Error: (Storage server: PureDisk:nbu_app1_bdc) mtstrm_write_segment: Fatal error occured in Multi-Threaded Agent: Timed out after waiting 1200s to send data to mtstrmd 

 

Regards,

 

Thiago

Yes the following timeout values were changed. still no luck, backups are failing with 84, 97 errors  ( media close error, media write error)
On client changed the client read timeout to 37656 which was set to 300
On media servers ( NBU 5220 appliance)   changed  timeout value from 800 to 1800 in following file:
/usr/openv/netbackup/db/config/DPS_PROXYDEFAULTRECVTMO
Also created it on the master server.

How shall I change the timeout  value of Multi-Threaded Agent.?

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

PLEASE upgrade your appliance as a matter of urgency to match master server software (2.7.3).

Other than support from Veritas, you also get new features and bug fixes.

When the appliance is at a supported level and the issue persists, you can log a call with Veritas support to investigate.

Yes we are recommending our management for the same, but a date is not yet fixed as the upgrade involves cost, also few weeks back we got a maintenance window to perform the upgrade but we couldn't even initiate the upgrade, as when we rebooted the appliance, it reported CRC errors , then the MSDP pool was showing down on master server, later the DNS cache was cleared and the pool was up, so we arent sure if the CRC errors till persists on the pool, if so it will not allow to perform the upgrade.

Until then management is saying to find a temporary fix/workaround.

:(

 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

The problem with the unsupported Appliance is that you are going to find it very difficult to find a temporary fix or workaround.

Members on this forum have suggested a few things to the best of our abilities.
If none of that has helped, you are really on your own. 

You may want to show your management this discussion.

PS
I'm not sure why you believe that upgrade involves cost.
The fact that your master is on a currently supported NBU version should entitle you to free upgrades that includes the Appliance software.
With active support, Veritas Support will be able to assist with the upgrade and look at issues that might prevent successful upgrade.
Simply log the call as 'upgrade assistance'.