Restore of NDMP fails

[maitakeboy]

I have had an ongoing issue with my Backup Exec 2010 R3 and my NetApp SAN CIFS. I believe, but am not certain, that it was coincident with the upgrading of the NetApp controller firmware. All of a sudden, I could not restore any backed up NDMP files! The backups were good, but I kept getting restore errors. After going round and round with Symantec and NetApp support, I was able to figure out myself a workaround: I could restore directories, but not individual files. Which worked fine until I had to restore a 20Mb Word doc that was in a 150GB directory! So I'm back hoping that in the interim, something has been found or some new eyes might see this and have a clue. 

Here is the error I receive:

Completed status: Failed
Final error: 0xe000130a - The NDMP server returned a restore error.
See the NDMP Log for more information.
Final error category: Resource Errors

For additional information regarding this error refer to link V-79-57344-4874

Here is the Backup Exec NDMP Log

NDMP Log Message: RESTORE: Tue Jul 15 16:08:31 2014 : We have read 420330 KB from the backup.
NDMP Log Message: RESTORE: Tue Jul 15 16:13:31 2014 : We have read 722153 KB from the backup.
NDMP Log Message: RESTORE: Tue Jul 15 16:18:31 2014 : We have read 943678 KB from the backup.
NDMP Log Message: RESTORE: Tue Jul 15 16:23:31 2014 : We have read 1057125 KB from the backup.
NDMP Log Message: RESTORE: Tue Jul 15 16:28:31 2014 : We have read 1167314 KB from the backup.
NDMP Log Message: RESTORE: Tue Jul 15 16:33:31 2014 : We have read 1267869 KB from the backup.
NDMP Log Message: RESTORE: Tue Jul 15 16:38:31 2014 : We have read 1409057 KB from the backup.
NDMP Log Message: RESTORE: Tue Jul 15 16:43:31 2014 : We have read 1532971 KB from the backup.
NDMP Log Message: RESTORE: Tue Jul 15 16:48:31 2014 : We have read 1657610 KB from the backup.
NDMP Log Message: RESTORE: Tue Jul 15 16:53:32 2014 : We have read 1767226 KB from the backup.
NDMP Log Message: RESTORE: /vol/CIFS/NetAdmin/Restore/BOXFILESTORAGELIST.DOC/: Path could not be created.
NDMP Log Message: RESTORE: /vol/CIFS/NetAdmin/Restore/DirectionsLVOffice.doc/: Path could not be created.
NDMP Log Message: RESTORE: /vol/CIFS/NetAdmin/Restore/LHVTRANSMITTAL1.doc/: Path could not be created.
NDMP Log Message: RESTORE: /vol/CIFS/NetAdmin/Restore/Proposal - Fed Ex.doc/: Path could not be created.
NDMP Log Message: RESTORE: /vol/CIFS/NetAdmin/Restore/RFIR_STAMP_MOCK_UP.pdf/: Path could not be created.
NDMP Log Message: RESTORE: /vol/CIFS/NetAdmin/Restore/Seismic Table.doc/: Path could not be created.
NDMP Log Message: RESTORE: /vol/CIFS/NetAdmin/Restore/SPECISSU.DOC/: Path could not be created.
NDMP Log Message: RESTORE: /vol/CIFS/NetAdmin/Restore/SPECKEY.DOC/: Path could not be created.
NDMP Log Message: RESTORE: /vol/CIFS/NetAdmin/Restore/2-10-webinar.doc/: Path could not be created.
NDMP Log Message: RESTORE: /vol/CIFS/NetAdmin/Restore/aisc webinar.doc/: Path could not be created.
NDMP Log Message: RESTORE: /vol/CIFS/NetAdmin/Restore/cooper hospital.doc/: Path could not be created.
NDMP Log Message: RESTORE: No files were created.
NDMP Log Message: RESTORE: RESTORE IS ABORTED
NDMP Data Halted: Internal Error
NDMP Log Message: Dump aborted.
NDMP Log Message: Mover: Closing data connection.
NDMP Mover Halted: Connect Closed


================  NDMP ENVIRONMENT  ================
DMP_NAME=NDMP
DIRECT=Y
VERBOSE=N
DEBUG=N
LIST=N
SMTAPE_BREAK_MIRROR=N
LIST_QTREES=N
EXTRACT=Y
RESTART=N
IGNORE_QTREES=N
MULTI_SUBTREE_NAMES=engineer
secure-c
clerical
typdet
approachDB
MAH_MP
CAD
projects
NetAdmin
RECOVER_FULL_PATHS=N
FILESYSTEM=/vol/CIFS
EXTRACT_ACL=Y
DATA_BLOCK_SIZE=64
SMTAPE_DELETE_SNAPSHOT=N
REPLICATE=N
TYPE=dump
RECURSIVE=Y
HIST=Y
EXCLUDE=*.bak, *.plt, *.tmp, *.mp3, *.mpeg
LEVEL=0
NDMP_UNICODE_FH=N
NDMP_VERSION=4
FORCE=N
DUMP_DATE=-1
ACL_START=2863959305216
RENAME_SUPPORTED=Y
UPDATE=Y
NOWRITE=N
NON_QUOTA_TREE=N
ENHANCED_DAR_ENABLED=T
BASE_DATE=-1
NO_ACLS=N
===============  NDMP ENVIRONMENT END  ===============

Instead of writing the files, it creates directories with the files names! You can see that the real crux is the Path Could Not Be Created in the NDMP log, right after the "path" whcih includes the filename with the curious  /: appended. Why would Backup Exec try to create a directory instead of restoring the file?

Any help would be greatly appreciated.

Thanks!

[VJware]

What is the version of ONTAP ? Is the version of ONTAP which you are using is supported as per the HCL ?

Hardware Compatibility List - http://www.symantec.com/business/support/index?page=content&id=TECH137050

EDIT**

This was seen in OnTap 7.3.7. I would recommend you to contact NetApp and confirm the same.

[maitakeboy]

VJware,

Thanks for your response!

Yes it is. I have been using this version of ONTAP and BE for years. I am using 7.3.3. I have a new service request in to NetApp and will report here whatever I find. Do you happen to have a link to the info on 7.3.7 that you mention?

Thanks again for your help!

[henny71]

HI..I am having NDMP issue as well with restoring files back to Netapp where the orginal was located.

Ontap 7.3.7

Netbackup 7.1 Master server is Linux - RedHat

Fiber channel connection.

Process goes and reads the tape and starts putting on NetApp then fails.

It writes a directory with the name of the file but no contents.

Seems like your isssue was similar, how did you fix and do you have the info for the tech or bug report?

Thanks.

[maitakeboy]

I never got a resolution to this issue. I had to implement a workaround, which I found, which was that I could not select files, only directories. So if I wanted to restore a single file, I had to restore the entire directory which contained the file. Obviously, I redirected the restore, then brabbed the file and put it in its proper directory. A pain-in-the-ass, but at least I can do restores.

[Larry_Fine]

I found a decent write up about this issue here.

http://www.pickysysadmin.ca/2012/09/12/symantec-backup-exec-ndmp-backups-fail-the-verify-stage-when-backing-up-from-a-netapp/

It appears to be a version specific NetApp issue.  I hope this helps you.

[henny71]

maitakeboy and Larry Fine,

Thank you for the info/links and suggestions.

I retried the restore today selecting the entire directory and not just the one file I needed and changed the path to a directory I setup for the restore and it worked!

I went back and selected the entire directory and deselected the files I did not want and the restore failed.  I thought it would allow me to do this much but I was wrong.  Entire directory or nothing.

Thank you for replying your help came faster than the tech support we paid for.

Now a decision if we will need to jump to the P1 patch or go with this solution for awhile.

Thanks again.

[maurijo]

Check if you have "File history" enabled on your backup job NDMP options.

This can cause the issue with only being able to restore the entire directory instead of single files.

[henny71]

I do not see "File History" as an option on the Attributes for the policy I have.  Is this a setting on the filer or on NBU?

[maurijo]

NBU ? I thought this was Backup Exec question as stated in the first post.

I don't have a backup exec 2010 installation right now but in 2012 it is in the job properties under NDMP.

Maybe this can help u:

http://www.symantec.com/business/support/index?page=content&id=HOWTO23316

Otherwise just google: backup exec 2010 ndmp file history.