NDMP backups running very slow

[rk1074]

Hello guys,

Of late we are facing performance issue with our NDMP backups. we have two NAS filer in our datacenter ( Netapp 3250 and 2040). data is snapvaulted from ther first one to the second one and then it is moved to tapes from 2040 through netbackup.

the issue is that the speed of the backup from filer to tape is extremely slow (~2-4 MBPS). we are using LTO5 tape drives and are conected via FC.

Master server is a Linux VMwith NBu 7.5

kindly suggest and let me know if any other info is required

[GV89]

Could you please let us know the CPU utilization of filer 2040..??

what is the rate of data replication from 3250 to 2040..??

what is the total size of data that u are backing up daily...??

[rk1074]

hello GV89,

for the first 2 question i need to get in touch with our storage guys.. will come back with the answer..

we have around 9 TB of data that needs to be backed up daily within 20 hours of backup window.

[Mark_Solutions]

Has it suddenly gone slow or is it a new setup and it has always run at that speed?

It helps us to know as much as possible in case something has changed or if it has never been right

Thanks

[rk1074]

Hi Mark,

This is an entirely new setup. Backups are running slow from the start of the operations.

[Mark_Solutions]

If it has always been like that (and that is VERY slow) then something is clearly wrong

You can setup some logging ndmpagent, ndmp, nbrb, nbjm but this sounds moe like somethin internal to the filer

You need to have a chat with your filer admins to see what information you can get from them in relation to how long the snapvault actually takes and any messages that are beingr raised on the filer during the backup process

Have you seen anything in the actual job log that causes any concern?

Maybe you could show us the policy settings - bppllist policyname -U and also the text from the detailed tab on the job itself.

Thanks

[GV89]

Did u get any rply from storage team.....???

[watsons]

You mentioned about having Snapvault in your environment, do you also have SnapMirror configured between the 2 NAS? 

I have heard about SnapMirror setup, if being too frequent (such as every 15min), can cause slowness when backup window kicks in.

[Jim-90]

You could always load a tape into a tape drive with robtest and do a native dump to the tape from the filer command line and see if there's any difference.  

Another is to do a native NetApp dump to NUL that will test how fast you can suck data off the filer. 

[GV89]

Its seem that u are using local NDMP....

In ur enviroment

You have NDMP storage unit and not media manager storage unit. The data is directly dumped to filer from  tape, as it is a local tape drive connected to  the filer.Hence netbackup has a very little role to play. Netbackup can only initiate backup and send request to filer for further processing

1) Stop the replication(Snavaulting) for a time being and perform tape backup to check the speed when snapvaulting is not enabled.

2) try use dump command and chck the performance

[rk1074]

@Mark

please find attached the policy detail and the job detail of one of the completed stream

[rk1074]

@watsons

We have snapmirror setup within the same filer for the HA (thats what I get from the storage guy :( .. ) and it shud not be affecting the tape backup as we are dealing with F2 only

Snavaulting is done every hour from F1 to F2

@Jim-90

Yes we tried the the dump after loading the tape but it failed with the below error:

>dump 0uf nrst0a /vol/tempvol_s01

DUMP: creating "/vol/tempvol_s01/../snapshot_for_backup.565" snapshot.

DUMP: Using Full Volume Dump

DUMP: Dumping tape file 1 on nrst0a

DUMP: open of nrst0a failed.

DUMP:  "Yes" or "No"? yes

DUMP: open of nrst0a failed.

DUMP: Retry the open: ("yes" or "no") no

DUMP: Could not initialize media.

DUMP: Dump failed to initialize.

DUMP: DUMP IS ABORTED

DUMP: Deleting "/vol/tempvol_s01/../snapshot_for_backup.565" snapshot.

please check

[watsons]

You should start engaging Netapp support when using dump still gives you error. It's probably not a Netbackup issue.

[Will_Restore]

DUMP: Could not initialize media.

manually mount a tape in the drive and try again

https://communities.netapp.com/thread/21887

[GV89 · Accepted solution]

Hi backup guys,

You need to make sure that if your filer is heavily utilized than snapvaulting and backups should not be occur simultaneously this could make your backups slow.

and before running dump command you need to mount a tape manually.

Also Please check with network team to resolve the same.

[rk1074]

I believe the cause of the issue lies in the bottleneck at the filer end . Our backups happens to be running at the time when the snapvaulting is also running .

[Jim-90]

An interesting test would be a DUMP to NUL when snapvaulting is happening and when it is not.

Any reference(s) to not doing NDMP backups and snapvualting at the same time or do we have to rely upon forum folklore?