Random NDMP Backup failures

[PWSteve]

I keep on getting failed NDMP Backups from Netapps, it will backup correctly occasionally but then fails the next few times, I have been going through the logs and have noticed that when I get a failed backup the NDMP Log is very long and it only writes a few KB every 5 minutes and fails when it has exceeded maximum run time 24hrs

NDMP Log Message: DUMP: creating "/vol/VMWare_VHD_Datastores_3/../snapshot_for_backup.52431" snapshot.
NDMP Log Message: DUMP: Using Full Volume Dump
NDMP Log Message: DUMP: Date of this level 0 dump: Fri Dec 12 09:48:59 2014.
NDMP Log Message: DUMP: Date of last level 0 dump: the epoch.
NDMP Log Message: DUMP: Dumping /vol/VMWare_VHD_Datastores_3 to NDMP connection
NDMP Log Message: DUMP: mapping (Pass I)[regular files]
NDMP Log Message: DUMP: mapping (Pass II)[directories]
NDMP Log Message: DUMP: estimated 633249236 KB.
NDMP Log Message: DUMP: dumping (Pass III) [directories]
NDMP Log Message: DUMP: Fri Dec 12 09:54:47 2014 : We have written 172 KB.
NDMP Log Message: DUMP: dumping (Pass IV) [regular files]
NDMP Log Message: DUMP: Fri Dec 12 09:59:48 2014 : We have written 303 KB.
NDMP Log Message: DUMP: Fri Dec 12 10:04:50 2014 : We have written 440 KB.
NDMP Log Message: DUMP: Fri Dec 12 10:09:55 2014 : We have written 576 KB.
NDMP Log Message: DUMP: Fri Dec 12 10:14:58 2014 : We have written 712 KB.
NDMP Log Message: DUMP: Fri Dec 12 10:20:06 2014 : We have written 849 KB.
NDMP Log Message: DUMP: Fri Dec 12 10:25:11 2014 : We have written 985 KB.
NDMP Log Message: DUMP: Fri Dec 12 10:30:16 2014 : We have written 1121 KB.
NDMP Log Message: DUMP: Fri Dec 12 10:35:21 2014 : We have written 1257 KB.
NDMP Log Message: DUMP: Fri Dec 12 10:40:27 2014 : We have written 1394 KB.
NDMP Log Message: DUMP: Fri Dec 12 10:45:34 2014 : We have written 1530 KB.
NDMP Log Message: DUMP: Fri Dec 12 10:50:39 2014 : We have written 1666 KB.
NDMP Log Message: DUMP: Fri Dec 12 10:55:45 2014 : We have written 1802 KB.
NDMP Log Message: DUMP: Fri Dec 12 11:00:53 2014 : We have written 1939 KB.
NDMP Log Message: DUMP: Fri Dec 12 11:05:58 2014 : We have written 2075 KB.
NDMP Log Message: DUMP: Fri Dec 12 11:11:00 2014 : We have written 2211 KB.
NDMP Log Message: DUMP: Fri Dec 12 11:16:08 2014 : We have written 2347 KB.
NDMP Log Message: DUMP: Fri Dec 12 11:21:16 2014 : We have written 2484 KB.
NDMP Log Message: DUMP: Fri Dec 12 11:26:20 2014 : We have written 2620 KB.
NDMP Log Message: DUMP: Fri Dec 12 11:31:28 2014 : We have written 2756 KB.
NDMP Log Message: DUMP: Fri Dec 12 11:36:36 2014 : We have written 2893 KB.
NDMP Log Message: DUMP: Fri Dec 12 11:41:39 2014 : We have written 3029 KB.
NDMP Log Message: DUMP: Fri Dec 12 11:46:46 2014 : We have written 3165 KB.
NDMP Log Message: DUMP: Fri Dec 12 11:51:51 2014 : We have written 3301 KB.
NDMP Log Message: DUMP: Fri Dec 12 11:56:59 2014 : We have written 3438 KB.
NDMP Log Message: DUMP: Fri Dec 12 12:02:03 2014 : We have written 3574 KB.
NDMP Log Message: DUMP: Fri Dec 12 12:07:11 2014 : We have written 3710 KB.
NDMP Log Message: DUMP: Fri Dec 12 12:12:16 2014 : We have written 3846 KB.
NDMP Log Message: DUMP: Fri Dec 12 12:17:24 2014 : We have written 3983 KB.
NDMP Log Message: DUMP: Fri Dec 12 12:22:31 2014 : We have written 4119 KB.
NDMP Log Message: DUMP: Fri Dec 12 12:27:35 2014 : We have written 4255 KB.
NDMP Log Message: DUMP: Fri Dec 12 12:32:39 2014 : We have written 4391 KB.
NDMP Log Message: DUMP: Fri Dec 12 12:37:40 2014 : We have written 4524 KB.
NDMP Log Message: DUMP: Fri Dec 12 12:42:44 2014 : We have written 4660 KB.
NDMP Log Message: DUMP: Fri Dec 12 12:47:49 2014 : We have written 4796 KB.
NDMP Log Message: DUMP: Fri Dec 12 12:52:50 2014 : We have written 4929 KB.
NDMP Log Message: DUMP: Fri Dec 12 12:57:53 2014 : We have written 5065 KB.
NDMP Log Message: DUMP: Fri Dec 12 13:03:01 2014 : We have written 5201 KB.
NDMP Log Message: DUMP: Fri Dec 12 13:08:05 2014 : We have written 5337 KB.
NDMP Log Message: DUMP: Fri Dec 12 13:13:12 2014 : We have written 5474 KB.
NDMP Log Message: DUMP: Fri Dec 12 13:18:17 2014 : We have written 5610 KB.
NDMP Log Message: DUMP: Fri Dec 12 13:23:24 2014 : We have written 5746 KB.
NDMP Log Message: DUMP: Fri Dec 12 13:28:28 2014 : We have written 5882 KB.
NDMP Log Message: DUMP: Fri Dec 12 13:33:36 2014 : We have written 6019 KB.
NDMP Log Message: DUMP: Fri Dec 12 13:38:44 2014 : We have written 6155 KB.
NDMP Log Message: DUMP: Fri Dec 12 13:43:48 2014 : We have written 6291 KB.
NDMP Log Message: DUMP: Fri Dec 12 13:48:53 2014 : We have written 6427 KB.
NDMP Log Message: DUMP: Fri Dec 12 13:53:53 2014 : We have written 6560 KB.
NDMP Log Message: DUMP: Fri Dec 12 13:58:57 2014 : We have written 6696 KB.
NDMP Log Message: DUMP: Fri Dec 12 14:04:00 2014 : We have written 6832 KB.
NDMP Log Message: DUMP: Fri Dec 12 14:09:07 2014 : We have written 6968 KB.
NDMP Log Message: DUMP: Fri Dec 12 14:14:14 2014 : We have written 7105 KB.
NDMP Log Message: DUMP: Fri Dec 12 14:19:17 2014 : We have written 7241 KB.
NDMP Log Message: DUMP: Fri Dec 12 14:24:24 2014 : We have written 7377 KB.
NDMP Log Message: DUMP: Fri Dec 12 14:29:31 2014 : We have written 7514 KB.
NDMP Log Message: DUMP: Fri Dec 12 14:34:36 2014 : We have written 7650 KB.
NDMP Log Message: DUMP: Fri Dec 12 14:39:42 2014 : We have written 7786 KB.
NDMP Log Message: DUMP: Fri Dec 12 14:44:47 2014 : We have written 7922 KB.
NDMP Log Message: DUMP: Fri Dec 12 14:49:55 2014 : We have written 8059 KB.
NDMP Log Message: DUMP: Fri Dec 12 14:54:58 2014 : We have written 8195 KB.
NDMP Log Message: DUMP: Fri Dec 12 15:00:05 2014 : We have written 8331 KB.
NDMP Log Message: DUMP: Fri Dec 12 15:05:09 2014 : We have written 8467 KB.
NDMP Log Message: DUMP: Fri Dec 12 15:10:16 2014 : We have written 8604 KB.
NDMP Log Message: DUMP: Fri Dec 12 15:15:23 2014 : We have written 8740 KB.
NDMP Log Message: DUMP: Fri Dec 12 15:20:27 2014 : We have written 8876 KB.
NDMP Log Message: DUMP: Fri Dec 12 15:25:30 2014 : We have written 9012 KB.
NDMP Log Message: DUMP: Fri Dec 12 15:30:31 2014 : We have written 9145 KB.
NDMP Log Message: DUMP: Fri Dec 12 15:35:36 2014 : We have written 9281 KB.
NDMP Log Message: DUMP: Fri Dec 12 15:40:39 2014 : We have written 9417 KB.
NDMP Log Message: DUMP: Fri Dec 12 15:45:41 2014 : We have written 9550 KB.
NDMP Log Message: DUMP: Fri Dec 12 15:50:46 2014 : We have written 9686 KB.
NDMP Log Message: DUMP: Fri Dec 12 15:55:52 2014 : We have written 9822 KB.
NDMP Log Message: DUMP: Fri Dec 12 16:00:56 2014 : We have written 9958 KB.
NDMP Log Message: DUMP: Fri Dec 12 16:06:04 2014 : We have written 10095 KB.
NDMP Log Message: DUMP: Fri Dec 12 16:11:08 2014 : We have written 10231 KB.
NDMP Log Message: DUMP: Fri Dec 12 16:16:15 2014 : We have written 10367 KB.
NDMP Log Message: DUMP: Fri Dec 12 16:21:19 2014 : We have written 10503 KB.
NDMP Log Message: DUMP: Fri Dec 12 16:26:26 2014 : We have written 10640 KB.
NDMP Log Message: DUMP: Fri Dec 12 16:31:34 2014 : We have written 10776 KB.
NDMP Log Message: DUMP: Fri Dec 12 16:36:39 2014 : We have written 10912 KB.
NDMP Log Message: DUMP: Fri Dec 12 16:41:42 2014 : We have written 11048 KB.
NDMP Log Message: DUMP: Fri Dec 12 16:46:43 2014 : We have written 11181 KB.
NDMP Log Message: DUMP: Fri Dec 12 16:51:48 2014 : We have written 11317 KB.
NDMP Log Message: DUMP: Fri Dec 12 16:56:52 2014 : We have written 11453 KB.
NDMP Log Message: DUMP: Fri Dec 12 17:02:00 2014 : We have written 11589 KB.
NDMP Log Message: DUMP: Fri Dec 12 17:07:08 2014 : We have written 11726 KB.
NDMP Log Message: DUMP: Fri Dec 12 17:12:12 2014 : We have written 11862 KB.
NDMP Log Message: DUMP: Fri Dec 12 17:17:18 2014 : We have written 11998 KB.
NDMP Log Message: DUMP: Fri Dec 12 17:22:26 2014 : We have written 12135 KB.
NDMP Log Message: DUMP: Fri Dec 12 17:27:29 2014 : We have written 12271 KB.
NDMP Log Message: DUMP: Fri Dec 12 17:32:37 2014 : We have written 12407 KB.
NDMP Log Message: DUMP: Fri Dec 12 17:37:41 2014 : We have written 12543 KB.
NDMP Log Message: DUMP: Fri Dec 12 17:42:48 2014 : We have written 12680 KB.
NDMP Log Message: DUMP: Fri Dec 12 17:47:53 2014 : We have written 12816 KB.
NDMP Log Message: DUMP: Fri Dec 12 17:53:01 2014 : We have written 12952 KB.
NDMP Log Message: DUMP: Fri Dec 12 17:58:04 2014 : We have written 13088 KB.
NDMP Log Message: DUMP: Fri Dec 12 18:03:05 2014 : We have written 13221 KB.
NDMP Log Message: DUMP: Fri Dec 12 18:08:09 2014 : We have written 13357 KB.
NDMP Log Message: DUMP: Fri Dec 12 18:13:12 2014 : We have written 13493 KB.
NDMP Log Message: DUMP: Fri Dec 12 18:18:19 2014 : We have written 13629 KB.
NDMP Log Message: DUMP: Fri Dec 12 18:23:26 2014 : We have written 13766 KB.
NDMP Log Message: DUMP: Fri Dec 12 18:28:30 2014 : We have written 13902 KB.
NDMP Log Message: DUMP: Fri Dec 12 18:33:38 2014 : We have written 14038 KB.
NDMP Log Message: DUMP: Fri Dec 12 18:38:43 2014 : We have written 14175 KB.
NDMP Log Message: DUMP: Fri Dec 12 18:43:47 2014 : We have written 14311 KB.
NDMP Log Message: DUMP: Fri Dec 12 18:48:51 2014 : We have written 14447 KB.
NDMP Log Message: DUMP: Fri Dec 12 18:53:59 2014 : We have written 14583 KB.
NDMP Log Message: DUMP: Fri Dec 12 18:59:06 2014 : We have written 14720 KB.
NDMP Log Message: DUMP: Fri Dec 12 19:04:09 2014 : We have written 14856 KB.
NDMP Log Message: DUMP: Fri Dec 12 19:09:14 2014 : We have written 14992 KB.
NDMP Log Message: DUMP: Fri Dec 12 19:14:17 2014 : We have written 15128 KB.
NDMP Log Message: DUMP: Fri Dec 12 19:19:24 2014 : We have written 15265 KB.
NDMP Log Message: DUMP: Fri Dec 12 19:24:27 2014 : We have written 15401 KB.
NDMP Log Message: DUMP: Fri Dec 12 19:29:33 2014 : We have written 15537 KB.
NDMP Log Message: DUMP: Fri Dec 12 19:34:37 2014 : We have written 15673 KB.
NDMP Log Message: DUMP: Fri Dec 12 19:39:38 2014 : We have written 15806 KB.
NDMP Log Message: DUMP: Fri Dec 12 19:44:41 2014 : We have written 15942 KB.
NDMP Log Message: DUMP: Fri Dec 12 19:49:46 2014 : We have written 16078 KB.
NDMP Log Message: DUMP: Fri Dec 12 19:54:47 2014 : We have written 16211 KB.
NDMP Log Message: DUMP: Fri Dec 12 19:59:50 2014 : We have written 16347 KB.
NDMP Log Message: DUMP: Fri Dec 12 20:04:54 2014 : We have written 16483 KB.
NDMP Log Message: DUMP: Fri Dec 12 20:09:59 2014 : We have written 16619 KB.
NDMP Log Message: DUMP: Fri Dec 12 20:15:06 2014 : We have written 16756 KB.
NDMP Log Message: DUMP: Fri Dec 12 20:20:08 2014 : We have written 16892 KB.
NDMP Log Message: DUMP: Fri Dec 12 20:25:15 2014 : We have written 17028 KB.
NDMP Log Message: DUMP: Fri Dec 12 20:30:18 2014 : We have written 17164 KB.
NDMP Log Message: DUMP: Fri Dec 12 20:35:26 2014 : We have written 17301 KB.
NDMP Log Message: DUMP: Fri Dec 12 20:40:27 2014 : We have written 17437 KB.
NDMP Log Message: DUMP: Fri Dec 12 20:45:35 2014 : We have written 17573 KB.
NDMP Log Message: DUMP: Fri Dec 12 20:50:39 2014 : We have written 17709 KB.
NDMP Log Message: DUMP: Fri Dec 12 20:55:39 2014 : We have written 17842 KB.
NDMP Log Message: DUMP: Fri Dec 12 21:00:44 2014 : We have written 17978 KB.
NDMP Log Message: DUMP: Fri Dec 12 21:05:49 2014 : We have written 18114 KB.
NDMP Log Message: DUMP: Fri Dec 12 21:10:55 2014 : We have written 18250 KB.
NDMP Log Message: DUMP: Fri Dec 12 21:16:03 2014 : We have written 18387 KB.
NDMP Log Message: DUMP: Fri Dec 12 21:21:07 2014 : We have written 18523 KB.
NDMP Log Message: DUMP: Fri Dec 12 21:26:14 2014 : We have written 18659 KB.
NDMP Log Message: DUMP: Fri Dec 12 21:31:22 2014 : We have written 18796 KB.
NDMP Log Message: DUMP: Fri Dec 12 21:36:26 2014 : We have written 18932 KB.
NDMP Log Message: DUMP: Fri Dec 12 21:41:30 2014 : We have written 19068 KB.
NDMP Log Message: DUMP: Fri Dec 12 21:46:38 2014 : We have written 19204 KB.
NDMP Log Message: DUMP: Fri Dec 12 21:51:44 2014 : We have written 19341 KB.
NDMP Log Message: DUMP: Fri Dec 12 21:56:46 2014 : We have written 19477 KB.
NDMP Log Message: DUMP: Fri Dec 12 22:01:54 2014 : We have written 19613 KB.
NDMP Log Message: DUMP: Fri Dec 12 22:06:58 2014 : We have written 19749 KB.
NDMP Log Message: DUMP: Fri Dec 12 22:12:05 2014 : We have written 19886 KB.
NDMP Log Message: DUMP: Fri Dec 12 22:17:13 2014 : We have written 20022 KB.
NDMP Log Message: DUMP: Fri Dec 12 22:22:17 2014 : We have written 20158 KB.
NDMP Log Message: DUMP: Fri Dec 12 22:27:21 2014 : We have written 20294 KB.
NDMP Log Message: DUMP: Fri Dec 12 22:32:22 2014 : We have written 20427 KB.
NDMP Log Message: DUMP: Fri Dec 12 22:37:26 2014 : We have written 20563 KB.
NDMP Log Message: DUMP: Fri Dec 12 22:42:30 2014 : We have written 20699 KB.
NDMP Log Message: DUMP: Fri Dec 12 22:47:30 2014 : We have written 20832 KB.
NDMP Log Message: DUMP: Fri Dec 12 22:52:34 2014 : We have written 20968 KB.
NDMP Log Message: DUMP: Fri Dec 12 22:57:39 2014 : We have written 21104 KB.
NDMP Log Message: DUMP: job aborted by NDMP
NDMP Log Message: DUMP: DUMP IS ABORTED
NDMP Log Message: DUMP: Deleting "/vol/VMWare_VHD_Datastores_3/../snapshot_for_backup.52431" snapshot.
NDMP Log Message: DATA: Backup terminated (for /vol/VMWare_VHD_Datastores_3).
NDMP Data Halted: Aborted
NDMP Log Message: Mover encountered internal socket error.
NDMP Mover Halted: Internal Error

================  NDMP ENVIRONMENT  ================
BASE_DATE=-1
PATHNAME_SEPARATOR=/
NDMP_VERSION=4
DATA_BLOCK_SIZE=64
HIST=Y
NO_ACLS=N
UPDATE=Y
DUMP_DATE=-1
LEVEL=0
FILESYSTEM=/vol/VMWare_VHD_Datastores_3
===============  NDMP ENVIRONMENT END  ===============

And a successful back up it writes a lot of data every 5 Mins and completes in 14hrs

NDMP Log Message: DUMP: creating "/vol/VMWare_VHD_Datastores_3/../snapshot_for_backup.62249" snapshot.
NDMP Log Message: DUMP: Using Full Volume Dump
NDMP Log Message: DUMP: Date of this level 0 dump: Fri Dec 26 10:16:57 2014.
NDMP Log Message: DUMP: Date of last level 0 dump: the epoch.
NDMP Log Message: DUMP: Dumping /vol/VMWare_VHD_Datastores_3 to NDMP connection
NDMP Log Message: DUMP: mapping (Pass I)[regular files]
NDMP Log Message: DUMP: mapping (Pass II)[directories]
NDMP Log Message: DUMP: estimated 633249236 KB.
NDMP Log Message: DUMP: dumping (Pass III) [directories]
NDMP Log Message: DUMP: dumping (Pass IV) [regular files]
NDMP Log Message: DUMP: Fri Dec 26 10:22:46 2014 : We have written 17134002 KB.
NDMP Log Message: DUMP: Fri Dec 26 10:27:46 2014 : We have written 35195485 KB.
NDMP Log Message: DUMP: Fri Dec 26 10:32:46 2014 : We have written 55623975 KB.
NDMP Log Message: DUMP: Fri Dec 26 10:37:46 2014 : We have written 74222742 KB.
NDMP Log Message: DUMP: Fri Dec 26 10:42:46 2014 : We have written 91114178 KB.
NDMP Log Message: DUMP: Fri Dec 26 10:47:46 2014 : We have written 110177674 KB.
NDMP Log Message: DUMP: Fri Dec 26 10:52:46 2014 : We have written 127397239 KB.
NDMP Log Message: DUMP: Fri Dec 26 10:57:46 2014 : We have written 144357555 KB.
NDMP Log Message: DUMP: Fri Dec 26 11:02:46 2014 : We have written 165400097 KB.
NDMP Log Message: DUMP: Fri Dec 26 11:07:46 2014 : We have written 182318554 KB.
NDMP Log Message: DUMP: Fri Dec 26 11:12:46 2014 : We have written 197248170 KB.
NDMP Log Message: DUMP: Fri Dec 26 11:17:46 2014 : We have written 213217548 KB.
NDMP Log Message: DUMP: Fri Dec 26 11:22:46 2014 : We have written 232916506 KB.
NDMP Log Message: DUMP: Fri Dec 26 11:27:46 2014 : We have written 251085611 KB.
NDMP Log Message: DUMP: Fri Dec 26 11:32:47 2014 : We have written 270700375 KB.
NDMP Log Message: DUMP: Fri Dec 26 11:37:47 2014 : We have written 288223986 KB.
NDMP Log Message: DUMP: Fri Dec 26 11:42:47 2014 : We have written 306096148 KB.
NDMP Log Message: DUMP: Fri Dec 26 11:47:47 2014 : We have written 322593407 KB.
NDMP Log Message: DUMP: Fri Dec 26 11:52:47 2014 : We have written 338163558 KB.
NDMP Log Message: DUMP: Fri Dec 26 11:57:47 2014 : We have written 355240342 KB.
NDMP Log Message: DUMP: Fri Dec 26 12:02:47 2014 : We have written 373542427 KB.
NDMP Log Message: DUMP: Fri Dec 26 12:07:47 2014 : We have written 390758318 KB.
NDMP Log Message: DUMP: Fri Dec 26 12:12:47 2014 : We have written 408288311 KB.
NDMP Log Message: DUMP: Fri Dec 26 12:17:47 2014 : We have written 429518840 KB.
NDMP Log Message: DUMP: Fri Dec 26 12:22:47 2014 : We have written 450956986 KB.
NDMP Log Message: DUMP: Fri Dec 26 12:27:47 2014 : We have written 472913799 KB.
NDMP Log Message: DUMP: Fri Dec 26 12:32:47 2014 : We have written 495918302 KB.
NDMP Log Message: DUMP: Fri Dec 26 12:37:47 2014 : We have written 516109040 KB.
NDMP Log Message: DUMP: Fri Dec 26 12:42:47 2014 : We have written 537708298 KB.
NDMP Log Message: DUMP: Fri Dec 26 12:47:47 2014 : We have written 560321037 KB.
NDMP Log Message: DUMP: Fri Dec 26 12:52:47 2014 : We have written 580289213 KB.
NDMP Log Message: DUMP: Fri Dec 26 12:57:47 2014 : We have written 597857628 KB.
NDMP Log Message: DUMP: Fri Dec 26 13:02:47 2014 : We have written 620910497 KB.
NDMP Log Message: ENHANCED_DAR_ENABLED is 'T'
NDMP Log Message: ACL_START is '645910372352'
NDMP Log Message: DUMP: dumping (Pass V) [ACLs]
NDMP Log Message: DUMP: 630771855 KB
NDMP Log Message: DUMP: DUMP IS DONE
NDMP Log Message: DUMP: Deleting "/vol/VMWare_VHD_Datastores_3/../snapshot_for_backup.62249" snapshot.
NDMP Log Message: Mover: Data connection closed by foreign host.
NDMP Data Halted: Succesful
NDMP Mover Halted: Connect Closed

================  NDMP ENVIRONMENT  ================
ACL_START=645910372352
ENHANCED_DAR_ENABLED=T
BASE_DATE=-1
PATHNAME_SEPARATOR=/
NDMP_VERSION=4
DATA_BLOCK_SIZE=64
HIST=Y
NO_ACLS=N
UPDATE=Y
DUMP_DATE=-1
LEVEL=0
FILESYSTEM=/vol/VMWare_VHD_Datastores_3
===============  NDMP ENVIRONMENT END  ===============

Issue happens randomly on our netapp NDMP backups any ideas as to how to solve this issue

[Larry_Fine]

What version of NetApp ONTAP?

[PWSteve]

We're running version 8.1 in 7-mode

(due to upgrade in the next few months)

[Colin_Weaver]

Not really your issue but we do not recommend backing up VMs with the NDMP option because there will be no GRT capability. if running VMware (which you appear to be, although VHD is confusing name if vmware) then a more efficient solution to optimize Backup and Restore capability would be SAN Transport with backup storage devices connected to the media server.