cancel
Showing results for 
Search instead for 
Did you mean: 

NDMP media write error(84) while writting to Data Domain

MateuszK
Level 3

Hello,

Hardware/software info:

Netbackup Server Linux Redhat using Netbackup version 7.7.3 (master as media)

Client: os version "NetApp Release 8.2.4P6 7-Mode"

DD6300  OS 6.0.1.10-561375

It is my first post so please forgive me mistakes. I have an issue with NDMP backup. The problem occures only with montlhy schedule wchich is using accelerator forced rescan option. It was working properly long time and a few days ago strated to fail with error code 84 media write error. It starts to write data and after some time fails. The backups goes directly to DD via DDboost.

I do not have an access into client so I need to focuse on master server.

anyone has faced similar issue?

Job details:

...
01/09/2018 09:07:41 - Info ndmpagent (pid=17596) HOST1: DUMP: Tue Jan  9 09:04:10 2018 : We have written 688907814 KB.
01/09/2018 09:12:41 - Info ndmpagent (pid=17596) HOST1: DUMP: Tue Jan  9 09:09:10 2018 : We have written 692824322 KB.
01/09/2018 09:21:27 - Critical bptm (pid=17598) include image error, bytesw(53248) not equal to wlen(524288)
01/09/2018 09:21:27 - Critical bptm (pid=17598) image write failed: error 2060022: software error
01/09/2018 09:21:38 - Error bptm (pid=17598) cannot write image to disk, Invalid argument
01/09/2018 09:21:38 - Info ndmpagent (pid=17596) Received ABORT request from bptm
01/09/2018 09:21:38 - Error ndmpagent (pid=17596) NDMP backup failed, path = /vol/Vol_docs
01/09/2018 09:21:38 - Error ndmpagent (pid=17596) HOST1: DUMP: Write to socket failed
01/09/2018 09:21:38 - Error ndmpagent (pid=17596) HOST1: DUMP: DUMP IS ABORTED
01/09/2018 09:21:39 - Error ndmpagent (pid=17596) HOST1: DATA: Operation terminated (for /vol/Vol_docs).
01/09/2018 09:21:39 - Info bptm (pid=17598) EXITING with status 84 <----------
01/09/2018 09:21:39 - Error bpbrm (pid=17590) could not send server status message
01/09/2018 09:21:39 - Info ndmpagent (pid=0) done. status: 84: media write error
01/09/2018 09:21:39 - end writing; write time: 19:34:29
media write error  (84)

BPTM logs:

09:21:27.548 [17598] <32> do_include_image: include image error, bytesw(53248) not equal to wlen(524288)
09:21:27.549 [17598] <32> write_data: image write failed: error 2060022:
09:21:27.560 [17598] <8> vnet_get_user_credential_path: [vnet_vxss.c:1474] status 35 0x23
09:21:27.560 [17598] <8> vnet_check_user_certificate: [vnet_vxss_helper.c:3728] vnet_get_user_credential_path failed 35 0x23
09:21:27.560 [17598] <2> ConnectionCache::connectAndCache: Acquiring new connection for host MASTER_SERVER_NBU, query type 161
09:21:27.561 [17598] <2> logconnections: BPDBM CONNECT FROM 172.16.57.1.57136 TO 172.16.57.1.13721 fd = 25
09:21:27.566 [17598] <2> db_end: Need to collect reply
09:21:38.459 [17598] <16> write_data: cannot write image to disk, Invalid argument
09:21:38.459 [17598] <4> write_backup: Calling close_all_ft_pipes
09:21:38.459 [17598] <2> NdmpAgentSession_abort_by_index[0]: aborting...
09:21:38.505 [17598] <2> NdmpAgentSession[0]: ndmp_xm_wait_for_abort_complete: Waiting for ABORT COMPLETE...
09:21:39.249 [17598] <2> bptm: EXITING with status 84 <----------
Connecting to host "HOST1" as user "ndmp"...
Waiting for connect notification message...
Opening session--attempting with NDMP protocol version 4...
Opening session--successful with NDMP protocol version 4
  host supports MD5 authentication
Getting MD5 challenge from host...
Logging in using MD5 method...
Host info is:
  host name "HOST1"
  os type "NetApp"
  os version "NetApp Release 8.2.4P6 7-Mode"
  host id "0536982947"
Login was successful
Host supports LOCAL backup/restore
Host supports 3-way backup/restore
1 ACCEPTED SOLUTION

Accepted Solutions

Nicolai
Moderator
Moderator
Partner    VIP   

Hi @MateuszK

I am not sure Data Domain is the one to blame, since it looks like to be a issue sending block to disk (bptm process).

I would take a look at the Netapp syslog (Think that is the "event log show" command) to check for further error messages.

Best Regards

Nicolai

View solution in original post

6 REPLIES 6

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

The error is more likely to do with an issue on DD, rather than NBU or NetApp.

This looks like a problem with backup image on the DD:

09:21:27.548 [17598] <32> do_include_image: include image error, bytesw(53248) not equal to wlen(524288)
09:21:27.549 [17598] <32> write_data: image write failed: error 2060022:

 

You need to check logs on the DD for any clues. Or send those 2 lines to your DD support team.
Also check the bptm entries leading up to the <32> critical errors to see if there is any additional info.

Marianne,

Thank you. I will create case with EMC because I do not see any clues in DD logs.

Nicolai
Moderator
Moderator
Partner    VIP   

Hi @MateuszK

I am not sure Data Domain is the one to blame, since it looks like to be a issue sending block to disk (bptm process).

I would take a look at the Netapp syslog (Think that is the "event log show" command) to check for further error messages.

Best Regards

Nicolai

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Please note that bptm is merely reporting the error that it received from DD via the OST plugin.

Error 20600xx is always an error reported by the OST plugin.

Here is the problem : 

... image error, bytesw(53248) not equal to wlen(524288)


The problem is with the image size on DD. 

Thank you everyone for the involvement.

I have worked with EMC and there was no problems on DD. I have also incresed verbosity on logs in Netbackup and the was no issues too.

The problem was with the NetApp storage. I do not have any logs and do not know what it was exacly but the storage team fixed something and backup is running properly now.

Nicolai
Moderator
Moderator
Partner    VIP   

Could you ask them what they changed ?

It would be a great way to finish off the thread with a root cause 

Best Regards

Nicolai