NDMP Backups fail with status 84

[Siddesh]

Hello Guys,

I'm facing issue with NDMP backups wherein backups fails after backing up 300Gb- 500Gb.

Also observed that the backup tends to complete when done via a media manager storage unit.

But when backup via NDMP storage unit it fails.

**********Job Log****************

04/07/2012 23:13:06 - requesting resource dakcmstnetbkp-hcart2-robot-tld-2-EVS3KMILL
04/07/2012 23:13:06 - requesting resource dakcmstnetbkp.NBU_CLIENT.MAXJOBS.EVS1CHAND
04/07/2012 23:13:06 - requesting resource dakcmstnetbkp.NBU_POLICY.MAXJOBS.DAKC_D_W_M_NDMP_EVS1CHAND
04/07/2012 23:19:01 - Waiting for scan drive stop IBM.ULT3580-TD5.002, Media server: dakcmstnetbkp
04/07/2012 23:19:03 - granted resource  dakcmstnetbkp.NBU_CLIENT.MAXJOBS.EVS1CHAND
04/07/2012 23:19:03 - granted resource  dakcmstnetbkp.NBU_POLICY.MAXJOBS.DAKC_D_W_M_NDMP_EVS1CHAND
04/07/2012 23:19:03 - granted resource  DW0077
04/07/2012 23:19:03 - granted resource  IBM.ULT3580-TD5.002
04/07/2012 23:19:03 - granted resource  dakcmstnetbkp-hcart2-robot-tld-2-EVS3KMILL
04/07/2012 23:19:03 - estimated 1999256737 kbytes needed
04/07/2012 23:19:04 - started process bpbrm (pid=21166)
04/07/2012 23:19:04 - connecting
04/07/2012 23:19:04 - connected; connect time: 0:00:00
04/07/2012 23:19:09 - mounting DW0077
04/07/2012 23:19:36 - mounted DW0077; mount time: 0:00:27
04/07/2012 23:19:36 - positioning DW0077 to file 1
04/07/2012 23:19:43 - positioned DW0077; position time: 0:00:07
04/07/2012 23:19:43 - begin writing
04/08/2012 03:33:39 - Error ndmpagent (pid=21182) MOVER_HALTED media write error - reason = 5 (NDMP_MOVER_HALT_MEDIA_ERROR)
04/08/2012 03:33:39 - Error ndmpagent (pid=21182) NDMP backup failed, path = /__VOLUME__/FS2-CHAND/
04/08/2012 03:33:39 - Error bptm (pid=21183) cannot write image to media id DW0077, drive index 11, NDMP_HALT_MEDIA_ERROR occurred
04/08/2012 03:33:46 - end writing; write time: 4:14:03
media write error (84)
************************

The other NDMP backup too fails; but with status 99

**********Job Log****************

04/11/2012 14:54:45 - requesting resource dakcmstnetbkp-hcart3-robot-tld-1
04/11/2012 14:54:45 - requesting resource dakcmstnetbkp.NBU_CLIENT.MAXJOBS.EVS3KMILL
04/11/2012 14:54:45 - requesting resource dakcmstnetbkp.NBU_POLICY.MAXJOBS.DAKC_D_W_M_NDMP_EVS3KMILL
04/11/2012 14:57:53 - granted resource  dakcmstnetbkp.NBU_CLIENT.MAXJOBS.EVS3KMILL
04/11/2012 14:57:53 - granted resource  dakcmstnetbkp.NBU_POLICY.MAXJOBS.DAKC_D_W_M_NDMP_EVS3KMILL
04/11/2012 14:57:53 - granted resource  310272
04/11/2012 14:57:53 - granted resource  HP.ULTRIUM3-SCSI.010
04/11/2012 14:57:53 - granted resource  dakcmstnetbkp-hcart3-robot-tld-1
04/11/2012 14:57:54 - estimated 959655937 kbytes needed
04/11/2012 14:57:54 - started process bpbrm (pid=25870)
04/11/2012 14:57:54 - connecting
04/11/2012 14:57:55 - connected; connect time: 0:00:00
04/11/2012 14:57:58 - mounting 310272
04/11/2012 14:58:36 - mounted 310272; mount time: 0:00:38
04/11/2012 14:58:37 - positioning 310272 to file 1
04/11/2012 14:58:47 - positioned 310272; position time: 0:00:10
04/11/2012 14:58:47 - begin writing
04/12/2012 03:37:01 - Error bpbrm (pid=25870) socket read failed: errno = 62 - Timer expired
04/12/2012 03:37:01 - Error ndmpagent (pid=25887) ndmp_data_get_state_failed, status = 18 (NDMP_XDR_DECODE_ERR)
04/12/2012 03:37:01 - Error ndmpagent (pid=25887) NDMP backup failed, path = /__VOLUME__/FS1-KM
04/12/2012 03:37:02 - Error ndmpagent (pid=25887) ndmp_data_get_state_failed, status = 12 (NDMP_EOF_ERR)
04/12/2012 03:38:07 - end writing; write time: 12:39:20
NDMP backup failure (99)

********************************************

Setup: Master Server (Solaris 10)

Netbackup for NDMP host is the master server(dakcmstnetbkp) itself.

Let me know if any other logs that need to be checked.

Regards,

Sid

[watsons]

1. Error 84:

Try using another tape (other than DW0077) for backup. See if it fails.

It might be a tape or tape drive issue, just run a few times to determine if it is.

2. Error 99:

There is a "timer expired" msg which sounds like timeout issue. Try increasing your Client Read Timeout to higher value to retry.

It is also helpful to get ndmp log from filer side to check.

Logs to gather: bpbrm, bptm, ndmpagent, ndmp (last 2 are unified logs, with OID 134 & 151 respectively).

[Yogesh9881]

how many time you faced this error (is it with same tape drive or tape media)

BPTM logs will help you 

[Siddesh]

Hi Guys,

I have tried the backup multiple times with different tape and drive; but still the issue persists.

Have logged a call with Syamtec TSE; and have uploaded the requested logs.

Also the NAS team has planned for a upgardation activity on the NAS Box by this week.

Will keep you posted...

[Mark_Solutions]

I have seen this issue in the past and caused by a corrupt file

Do a vxlogview against oid 134

Look for NDMP_LOG_WARNING (or error) in the log file

This may pin point the file that is actually corrupt

Hope this helps