NDMP Error - hmmmm

[systematic92]

We've recently upgraded to BE2014. Some of our jobs are immediately failing whilst 4 are running with the following error:

NDMP Log Message: DUMP: Maximum number of allowed dumps/restores (4) in progress.
NDMP Log Message: NDMP_OP_ERRORCODE is 'EMAXBACKUPS'
NDMP Log Message: DUMP: Dump failed to initialize.
NDMP Log Message: DUMP: DUMP IS ABORTED
NDMP Log Message: DATA: Operation terminated: EVENT: INTERNAL ERROR (for /vol/EXMBX_DB_05)
NDMP Data Halted: Internal Error
NDMP Mover Halted: Connect Closed

Any ideas what is going on? 

We have 7 drives across 2 libraries backing up from multiple NetApp controllers that are running Ontap 8.2.2

The strange part is that 5 backups had started fine. One job completed and the the next scheduled job failed with the above error within seconds, the next one after started and failed within seconds, the third and fourth the same and then the fifth one starts fine......

I'd be grateful for your help.

[lmosla]

Are you fully updated to SP 1?  What other errors are reported in the job log and windows events?

also since this started after the upgrade to BE 2014 try running a repair of the BE install through Programs and Features.

[systematic92]

I dont think that is the issue. The issue is to do with the maximum numer of concurrent streams from NetApp. I stumbled upon this article:

https://www-secure.symantec.com/connect/forums/netapp-filer-nas-and-ndmp-backup

That is 1 years old.

How can I get around this please?

[Steve-Young]

Did you upgrade Backup Exec and your ONTap version around the same time?

Thet NDMP error is being returned by the filer.

Ihttps://library.netapp.com/ecmdocs/ECMP1368865/html/frameset.html

This bug was fixed in 8.2.2 http://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=755925

Haved you contacted netapp ?

[systematic92]

Hi Steve,

My understanding is this -

Netapp was upraded to 8.2.2 and all our backups (when using 2010R3) had failed. The team tried to get them going again but were unsuccessful and put forward a case to the buisness to upgrade to BE 2014. This upgrade took place 1 day after NetApp was upgraded to 8.2.2.

You mentioned that this bug was fixed in 8.2.2 however, that is the version we have in play.

NetApp have not been contacted as of yet.

On a side note - cheeky question - Should AOFO be enabled for our backup jobs? All of them are configured to backup data to tape from the NetApp filers. Thanks,

[systematic92]

By the way lads, I figured it out. The issue was a limitation in RAM on the Filers.

The maximum number of concurrent dump streams you can have from any one NetApp controller is dependant on the RAM.

Less than 16GB of ram equates to 4 concurrent streams. 16 - 24GB equates to 16 concurrents and 24+ equates to 32 streams.

We currently have FAS3160's with 8GB of RAM installed in each controller. We need to up this to 16GB to benefit from 16 concurrent streams.

LAST ISSUE:

Should AOFO be enabled for NDMP backsup from NetApp to tape? The reason I ask is because I have 3 jobs currently been running for 14 hours with a rate of 0.00 MB/per min with 1kb byte count. Yes you read that correctly 1kb byte count. I understand the data on the volumes contains millions of files however, I have spilt the job up to no avail. AOFO is currently disabled. I wonder if enabling this will prove to be advatnageous?

Thanks!

[pkh]

Turning on AOF will have no effect because the OS cannot snapshot the NAS

[systematic92]

I dont get it, prior to upgrading to 2014, thee very jobs were bcking up to tape fine. They were taking 5 hours to start given the sheer amount of data but they were starting. Something must have been enabled OR disabled in 2010 that isnt enabled or disabled in 2014.

any ideas PKH?

ACL and file history is enabled. We need ACL but not sure about File History. In laymans terms what does it do?

annything else i can explore would be greatly appreciated.