NDMP backup failure(99)

[mickk30]

I am running into a problem when backing up from a NAS using NDMP.  If I run the job as a full backup of the volume it work’s.  However if I change that policy to an Incremental backup then I get the following error.

11/03/2009 15:14:24 - Error ndmpagent(pid=5276) NDMP backup failed, path = /vol/mk      

11/03/2009 15:14:25 - Error bptm(pid=3924) none of the NDMP backups for client NAS001.prod completed successfully  

11/03/2009 15:14:37 - end writing; write time: 00:00:20

NDMP backup failure(99)

Netbackup – 6.5.3

NAS = IBM 3600 (re-branded Netapp

[Andy_Welburn]

The following TechNote indicates that you can get this if none of the files have changed since the last backup.

STATUS CODE: 99 "NDMP backup failure" occurs when backing up a NDMP client.

Could that be the case with yours?

[mickk30]

Just to ensure that data was updated, I copied in another 30 GB onto the CIFS share.  Got the same error when I run the Inc Backup.  I checked the logs and there does not seem to be any other relevant messages -

,51216,134,134,35,1236788379028,2320,1104,0:,89:NDMP_LOG_NORMAL 0 DUMP: creating "/vol/mk/../snapshot_for_backup.1218" snapshot.,11:CtnLogMsgCB,1

0,51216,134,134,36,1236788379575,2320,1104,0:,85:NDMP_LOG_NORMAL 0 DUMP: /etc/dumpdates does not exist, try performing a level 0 dump.,11:CtnLogMsgCB,1

0,51216,134,134,37,1236788379950,2320,1104,0:,50:NDMP_LOG_NORMAL 0 DUMP: Dump failed to initialize.,11:CtnLogMsgCB,1

0,51216,134,134,38,1236788379950,2320,1104,0:,39:NDMP_LOG_NORMAL 0 DUMP: DUMP IS ABORTED,11:CtnLogMsgCB,1

0,51216,134,134,39,1236788379950,2320,1104,0:,89:NDMP_LOG_NORMAL 0 DUMP: Deleting "/vol/mk/../snapshot_for_backup.1218" snapshot.,11:CtnLogMsgCB,1

1,51216,134,134,40,1236788380122,2320,1104,0:,0:,35:NdmpBackupManager::NotifyDataHalted,2,(33|A1:3|A29:NDMP_DATA_HALT_INTERNAL_ERROR|)

1,51216,134,134,41,1236788380122,2320,1104,0:,0:,36:NdmpBackupManager::UpdateTotalKbytes,4,(24|u32:0|u64:0|u64:0|)

2,51216,134,134,42,1236788380122,2320,1104,0:,0:,0:,2,(32|A16:/vol/LonUserHome|)

0,51216,134,134,43,1236788380169,2320,1104,0:,31:NDMP_LOG_NORMAL 0 Dump aborted.,11:CtnLogMsgCB,1

1,51216,134,134,44,1236788380559,2320,1104,0:,0:,35:XmServerControl::SendControlMessage,3,(135|i32:3|i32:7|A6:FAILED|A0:|)

1,51216,134,134,45,1236788380575,2320,1104,0:,0:,35:XmServerControl::SendControlMessage,3,(137|i32:3|i32:0|A0:|i32:0|)

0,51216,134,134,46,1236788380575,2320,1104,0:,14:Read all paths,32:NdmpManager::GetOperationsStatus,1

0,51216,134,134,47,1236788380575,2320,1104,0:,46:Operations to do = 1 Successful operations = 0,32:NdmpManager::GetOperationsStatus,1

1,51216,134,134,48,1236788393183,2320,1104,0:,0:,12:MainShutdown,2,(7|A2:99|u32:2320|u32:1104|)

16:18:04.407 [2320.1104] <2> ParseConfigExA: Unknown configuration option on line 54: RenameIfExists = 0
16:19:38.263 [2320.1104] <2> nb_bind_on_port_addr: set socket option SO_EXCLUSIVEADDRUSE
16:19:40.122 [2320.1104] <2> ParseConfigExA: Unknown configuration option on line 54: RenameIfExists = 0
16:19:40.122 [2320.1104] <2> init_cache: vnet_hosts.c.990: host_cache_size: 200 0x000000c8
16:19:40.122 [2320.1104] <2> init_cache: vnet_hosts.c.991: cache_time: 3600 0x00000e10
16:19:40.122 [2320.1104] <2> init_cache: vnet_hosts.c.1003: host_failed_cache_size: 40 0x00000028
16:19:40.122 [2320.1104] <2> init_cache: vnet_hosts.c.1004: cache_time: 3600 0x00000e10
16:19:40.137 [2320.1104] <2> init_cache: vnet_hosts.c.990: host_cache_size: 200 0x000000c8
16:19:40.137 [2320.1104] <2> init_cache: vnet_hosts.c.991: cache_time: 3600 0x00000e10
16:19:40.137 [2320.1104] <2> init_cache: vnet_hosts.c.1003: host_failed_cache_size: 40 0x00000028
16:19:40.137 [2320.1104] <2> init_cache: vnet_hosts.c.1004: cache_time: 3600 0x00000e10
16:19:40.153 [2320.1104] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2043: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
16:19:40.153 [2320.1104] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2057: service: bpjobd
16:19:40.153 [2320.1104] <2> job_connect: SO_KEEPALIVE set on socket 1756 for client ad2zgbnts001
16:19:40.153 [2320.1104] <2> logconnections: BPJOBD CONNECT FROM 10.190.10.11.4627 TO 10.190.10.11.13724
16:19:40.153 [2320.1104] <2> job_authenticate_connection: ignoring VxSS authentication check for now...
16:19:40.153 [2320.1104] <2> job_connect: Connected to the host ad2zgbnts001 contype 10 jobid <99196> socket <1756>
16:19:40.153 [2320.1104] <2> job_connect: Connected on port 4627
16:19:40.169 [2320.1104] <2> job_monitoring_exex: ACK disconnect
16:19:40.169 [2320.1104] <2> job_disconnect: Disconnected
16:19:40.169 [2320.1104] <2> db_error_add_to_file: dberrorq.c:midnite = 1236729600

[schmaustech]

Try putting a slash on the end of your backup selection (IE. /vol/mk/)

Regards,

Benjamin Schmaus

[Darren_Dunham]

The logs you posted show that the machine wasn't able to get dump rolling at all.  I find it odd that /etc/dumpdates (actually /vol/vol0/etc/dumpdates) doesn't exist if you've already done a full.  That could be part of the problem.

This doesn't appear to be a Netbackup or a Netbackup configuration problem at all.  It's almost certainly an issue internal to the NDMP host.

You could probably reproduce it just by running a level 1 "dump" to null on the filer.  That would confirm it's not related to Netbackup (or even NDMP).

Logs seem like after the "creating snapshot" it took almost 10 minutes before it logged the dumpdates problem.  Can you check during a backup to see if the snapshot is created properly at that time?  Not sure what else to suggest here.

--

Darren

[Douglas_A]

Error 99 usualy means the Backup target no longer exists or has a different name i would begin there. If you fined that the target is correct and exists run the verify cmd

set_ndmp_attr -verify <servername>

make sure it can login and is working properly.

[Darren_Dunham]

The logs that are posted above show that the target exists and that the NBU server can communicate and log in properly to the filer.  The error must be internal to the NDMP server.

--

Darren

[schmaustech]

I actually have a case open for a status 99 with Netapp and their ndmpdlog shows an IO error which they claim is caused by my VTL and not the filer, however I never see any errors on the VTL side nor does bptm show any error.  It is like a communication issue within the filer.

Good luck getting Netapp to track this one down.

Regards,

Benjamin Schmaus