Job Failing with NDMP backup failure (99)

[amartin]

We have a cummulative incremental job that just started failing over the weekend that has previously ran successfully for a long period of time.  I thought that it might be a bad tape so I removed that tape and the problem is still occuring.  Here is the log of the latest run.  There haven't been any changes made to the system.  We are running Netbackup 6.5.4.  Any assistance with this would be greatly appreciated.

5/14/2012 9:52:16 AM - requesting resource ndmp
5/14/2012 9:52:16 AM - requesting resource indybkup.NBU_CLIENT.MAXJOBS.g8_server_2
5/14/2012 9:52:16 AM - requesting resource indybkup.NBU_POLICY.MAXJOBS.ndmp3
5/14/2012 9:52:16 AM - granted resource indybkup.NBU_CLIENT.MAXJOBS.g8_server_2
5/14/2012 9:52:16 AM - granted resource indybkup.NBU_POLICY.MAXJOBS.ndmp3
5/14/2012 9:52:16 AM - granted resource 000437
5/14/2012 9:52:16 AM - granted resource HP.ULTRIUM3-SCSI.000
5/14/2012 9:52:16 AM - granted resource ndmp
5/14/2012 9:52:17 AM - estimated 179491597 kbytes needed
5/14/2012 9:52:17 AM - started process bpbrm (3424)
5/14/2012 9:52:17 AM - connecting
5/14/2012 9:52:17 AM - connected; connect time: 00:00:00
5/14/2012 9:52:20 AM - mounting 000437
5/14/2012 9:53:03 AM - mounted; mount time: 00:00:43
5/14/2012 9:53:07 AM - positioning 000437 to file 2
5/14/2012 9:54:01 AM - positioned 000437; position time: 00:00:54
5/14/2012 9:54:01 AM - begin writing
5/14/2012 10:02:17 AM - Error ndmpagent(pid=2612) connection 0xc75f58 ndmp_message_process_one_failed, status = NDMP_XDR_DECODE_ERR       
5/14/2012 10:02:17 AM - Error ndmpagent(pid=2612) eof is set - connection 0xc75f58       
5/14/2012 10:02:17 AM - Error ndmpagent(pid=2612) terminated by parent process         
5/14/2012 10:02:17 AM - Error ndmpagent(pid=2612) ndmp_data_get_state_failed, status = 12 (NDMP_EOF_ERR)        
5/14/2012 10:02:17 AM - Error ndmpagent(pid=2612) NDMP backup failed, path = /root_vdm_1/Databases       
5/14/2012 10:02:17 AM - Error ndmpagent(pid=2612) ndmp_data_get_state_failed, status = 12 (NDMP_EOF_ERR)        
5/14/2012 10:02:38 AM - end writing; write time: 00:08:37
NDMP backup failure(99)

[Will_Restore]

braheem states he "was able to resolve this issue by disabling jumbo frames on the NIC"

https://www-secure.symantec.com/connect/forums/ndmpxdrdecodeerr

[amartin]

I already have that disabled.

[Yasuhisa_Ishika]

Are there any clue/logs on the target NDMP mover?

[Yogesh9881]

post below command output,

bppllist <ndmp_policy> -L

also let us know what NAS box u r using ?

[V4]

most of time NDMP jobs fails with status 99 especially if path found are incorrect. NDMP path are case sensitive and it even doesn't acceps Space at the end or at beginning . NBU Doesn't removes those wildcards automatically.

post ndmpd log which can help in identifying issue

[amartin]

This job started working again last night with making no changes.  I talked with the support engineer this morning regarding the case I opened for this issue and we are going to leave the case open for a couple of days to make sure it doesn't come back.

I will post an update with any new information.  Thanks for all of the suggestions.

[watsons]

Sometimes when there is no change in Netbackup setting, but NDMP job get successful one day and failing the other day, it might be something on the NDMP host. Can be ndmpd was being restarted or timeout. As Yasuhisa pointed out, best to check ndmp side of logs (ndmpd + messages)