NDMP Backup failure Status code 24 or 99

[GulzarShaikhAUS]

Dear Experts,

After upgrading to Netbackup 7.1.0.4 the NDMP backups are failing with error code 24 or 199 intermittantly. Anyone has any experiecen with the same.

7/15/2012 12:42:34 PM - Info bpbrm(pid=11116) fileserver is the host to backup data from    
7/15/2012 12:42:34 PM - Info bpbrm(pid=11116) reading file list from client       
7/15/2012 12:42:34 PM - Info bpbrm(pid=11116) starting ndmpagent on client        
7/15/2012 12:42:34 PM - Info ndmpagent(pid=8452) Backup started          
7/15/2012 12:42:34 PM - Info nbjm(pid=7632) starting backup job (jobid=93874) for client fileserver, policy FILESERVER_NDMP, schedule Incremental 
7/15/2012 12:42:34 PM - Info nbjm(pid=7632) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=93874, request id:{8B131C75-01E3-48BC-A3DE-100C5E3280B7}) 
7/15/2012 12:42:34 PM - requesting resource pmmnbu01-hcart-robot-tld-0-fileserver
7/15/2012 12:42:34 PM - requesting resource pmmnbu01.NBU_CLIENT.MAXJOBS.fileserver
7/15/2012 12:42:34 PM - requesting resource pmmnbu01.NBU_POLICY.MAXJOBS.FILESERVER_NDMP
7/15/2012 12:42:34 PM - granted resource pmmnbu01.NBU_CLIENT.MAXJOBS.fileserver
7/15/2012 12:42:34 PM - granted resource pmmnbu01.NBU_POLICY.MAXJOBS.FILESERVER_NDMP
7/15/2012 12:42:34 PM - granted resource 000033
7/15/2012 12:42:34 PM - granted resource HP.ULTRIUM4-SCSI.001
7/15/2012 12:42:34 PM - granted resource pmmnbu01-hcart-robot-tld-0-fileserver
7/15/2012 12:42:34 PM - estimated 21662769 Kbytes needed
7/15/2012 12:42:34 PM - Info nbjm(pid=7632) started backup job for client fileserver, policy FILESERVER_NDMP, schedule Incremental on storage unit pmmnbu01-hcart-robot-tld-0-fileserver
7/15/2012 12:42:34 PM - started process bpbrm (11116)
7/15/2012 12:42:34 PM - connecting
7/15/2012 12:42:34 PM - connected; connect time: 00:00:00
7/15/2012 12:42:35 PM - Info bptm(pid=8152) start           
7/15/2012 12:42:43 PM - Info bptm(pid=8152) using 30 data buffers        
7/15/2012 12:42:43 PM - Info bptm(pid=8152) using 131072 data buffer size       
7/15/2012 12:42:43 PM - Info bptm(pid=8152) start backup          
7/15/2012 12:42:43 PM - Info bptm(pid=8152) Waiting for mount of media id 000033 (copy 1) on server pmmnbu01.
7/15/2012 12:42:43 PM - mounting 000033
7/15/2012 12:45:35 PM - Info bptm(pid=8152) media id 000033 mounted on drive index 1, drivepath nrst0a, drivename HP.ULTRIUM4-SCSI.001, copy 1
7/15/2012 12:45:35 PM - mounted; mount time: 00:02:52
7/15/2012 12:45:47 PM - positioning 000033 to file 3
7/15/2012 12:48:29 PM - Error bptm(pid=8152) io_ioctl_ndmp (MTFSF) failed on media id 000033, drive index 1, return code 18 (NDMP_XDR_DECODE_ERR) (bptm.c.7042)
7/15/2012 12:48:29 PM - Error ndmpagent(pid=8452) connection 0x1abdc60 ndmp_message_process_one_failed, status = NDMP_ILLEGAL_STATE_ERR      
7/15/2012 12:48:29 PM - Error ndmpagent(pid=8452) eof is set - connection 0x1abdc60      
7/15/2012 12:48:29 PM - Error ndmpagent(pid=8452) NDMP backup failed, path = UNKNOWN      
7/15/2012 12:50:01 PM - positioned 000033; position time: 00:04:14
7/15/2012 12:50:01 PM - begin writing
7/15/2012 12:50:20 PM - Info bptm(pid=8152) EXITING with status 24 <----------       
7/15/2012 12:50:20 PM - end writing; write time: 00:00:19
7/15/2012 12:50:25 PM - Info ndmpagent(pid=0) done. status: 24: socket write failed      
socket write failed(24)

===================================================================================

7/15/2012 1:12:25 PM - Info bpbrm(pid=8196) fileserver is the host to backup data from    
7/15/2012 1:12:25 PM - Info bpbrm(pid=8196) reading file list from client       
7/15/2012 1:12:25 PM - Info bpbrm(pid=8196) starting ndmpagent on client        
7/15/2012 1:12:25 PM - Info ndmpagent(pid=9224) Backup started          
7/15/2012 1:12:25 PM - Info bptm(pid=5872) start           
7/15/2012 1:12:25 PM - Info nbjm(pid=7632) starting backup job (jobid=93876) for client fileserver, policy FILESERVER_NDMP, schedule test_Diff 
7/15/2012 1:12:25 PM - Info nbjm(pid=7632) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=93876, request id:{E600DB0C-620C-40C1-BD9A-6BA77FA82661}) 
7/15/2012 1:12:25 PM - requesting resource pmmnbu01-hcart-robot-tld-0-fileserver
7/15/2012 1:12:25 PM - requesting resource pmmnbu01.NBU_CLIENT.MAXJOBS.fileserver
7/15/2012 1:12:25 PM - requesting resource pmmnbu01.NBU_POLICY.MAXJOBS.FILESERVER_NDMP
7/15/2012 1:12:25 PM - granted resource pmmnbu01.NBU_CLIENT.MAXJOBS.fileserver
7/15/2012 1:12:25 PM - granted resource pmmnbu01.NBU_POLICY.MAXJOBS.FILESERVER_NDMP
7/15/2012 1:12:25 PM - granted resource 000029
7/15/2012 1:12:25 PM - granted resource HP.ULTRIUM4-SCSI.001
7/15/2012 1:12:25 PM - granted resource pmmnbu01-hcart-robot-tld-0-fileserver
7/15/2012 1:12:25 PM - estimated 0 Kbytes needed
7/15/2012 1:12:25 PM - Info nbjm(pid=7632) started backup job for client fileserver, policy FILESERVER_NDMP, schedule test_Diff on storage unit pmmnbu01-hcart-robot-tld-0-fileserver
7/15/2012 1:12:25 PM - started process bpbrm (8196)
7/15/2012 1:12:25 PM - connecting
7/15/2012 1:12:25 PM - connected; connect time: 00:00:00
7/15/2012 1:12:26 PM - Info bptm(pid=5872) using 30 data buffers        
7/15/2012 1:12:26 PM - Info bptm(pid=5872) using 131072 data buffer size       
7/15/2012 1:12:26 PM - Info bptm(pid=5872) start backup          
7/15/2012 1:12:26 PM - Info bptm(pid=5872) Waiting for mount of media id 000029 (copy 1) on server pmmnbu01.
7/15/2012 1:12:26 PM - mounting 000029
7/15/2012 1:15:24 PM - Info bptm(pid=5872) media id 000029 mounted on drive index 1, drivepath nrst0a, drivename HP.ULTRIUM4-SCSI.001, copy 1
7/15/2012 1:15:24 PM - mounted; mount time: 00:02:58
7/15/2012 1:15:33 PM - positioning 000029 to file 1
7/15/2012 1:15:54 PM - positioned 000029; position time: 00:00:21
7/15/2012 1:15:54 PM - begin writing
7/15/2012 1:18:46 PM - Error ndmpagent(pid=9224) ndmp_mover_get_state failed, status = 12 (NDMP_EOF_ERR)      
7/15/2012 1:18:46 PM - Error ndmpagent(pid=9224) NDMP backup failed, path = UNKNOWN      
7/15/2012 1:18:46 PM - Error ndmpagent(pid=9224) ndmp_mover_get_state failed, status = 12 (NDMP_EOF_ERR)      
7/15/2012 1:18:46 PM - Error ndmpagent(pid=9224) connection 0x1badc60 ndmp_message_process_one_failed, status = NDMP_ILLEGAL_STATE_ERR      
7/15/2012 1:18:46 PM - Error ndmpagent(pid=9224) eof is set - connection 0x1badc60      
7/15/2012 1:18:46 PM - Error bptm(pid=5872) io_ioctl_ndmp (MTBSF) failed on media id 000029, drive index 1, return code 12 (NDMP_EOF_ERR) (bptm.c.8763)
7/15/2012 1:18:51 PM - Info bptm(pid=5872) EXITING with status 99 <----------       
7/15/2012 1:18:51 PM - end writing; write time: 00:02:57
7/15/2012 1:18:56 PM - Info ndmpagent(pid=0) done. status: 99: NDMP backup failure      
NDMP backup failure(99)

[honey_jack]

The following TechNote indicates that you can get this if none of the files have changed since the last backup.

STATUS CODE: 99 "NDMP backup failure" occurs when backing up a NDMP client.

NetBackup error messages and status codes

http://www.symantec.com/business/support/index?page=content&id=HOWTO50775

https://www-secure.symantec.com/connect/forums/ndmp-backup-failure99

[GulzarShaikhAUS]

Hi Honey_Jack,

Thanks a lot for such a good information. Normaly I had observed that the INCR backups are only victims. Morever, when I tested it, I ran the backups back to back so there is a high chance of no file being changed. Some of the INCR backups fail on weekends as there are no users to make changes...

Still is there any communication from NetApp filer which we can trace which tells us that no files were changed. Its very important if you want to convince customer. Also this looks very wiered and Symantec should take care of this issue.

[watsons]

Error 24 is connection timeout error

Error 99 is more generic and normally we have to go check the job details and logs to find out what exactly is the NDMP-type of error, here in your details it shows NDMP_ILLEGAL_STATE_ERR.

A possible cause maybe: http://www.symantec.com/docs/TECH86723

Suggest to check more details on ndmpagent (134) & ndmp (151) logs, as well as those from the NDMP host.

If there was no change at all and only happen right after upgrade, could be a Netbackup issue may want to log a support call.

[V4]

similar issue was heard. where master /media was upgraded but NDMP agent was not. Is this the same case with you.

[GulzarShaikhAUS · Accepted solution]

Hi All,

Finally after log calls with Backline and NetApp support we found that the issue was from Filer side. The option ip.fastpath.enable was set to ON which causes packets to retransmitt and also casued resets. Disabling the option resolved the issue.

Though a question still remains is why it all started after the upgrade?

Regards

[V4]

thanx never heard about this config parameter is to be changed at filer level except for NDMP to be set on ON status.. thanx for sharing info