09-19-2013 08:35 AM
Hello guys,
Of late we are facing performance issue with our NDMP backups. we have two NAS filer in our datacenter ( Netapp 3250 and 2040). data is snapvaulted from ther first one to the second one and then it is moved to tapes from 2040 through netbackup.
the issue is that the speed of the backup from filer to tape is extremely slow (~2-4 MBPS). we are using LTO5 tape drives and are conected via FC.
Master server is a Linux VMwith NBu 7.5
kindly suggest and let me know if any other info is required
Solved! Go to Solution.
09-25-2013 11:28 AM
Hi backup guys,
You need to make sure that if your filer is heavily utilized than snapvaulting and backups should not be occur simultaneously this could make your backups slow.
and before running dump command you need to mount a tape manually.
Also Please check with network team to resolve the same.
09-19-2013 08:42 AM
Could you please let us know the CPU utilization of filer 2040..??
what is the rate of data replication from 3250 to 2040..??
what is the total size of data that u are backing up daily...??
09-19-2013 09:07 AM
hello GV89,
for the first 2 question i need to get in touch with our storage guys.. will come back with the answer..
we have around 9 TB of data that needs to be backed up daily within 20 hours of backup window.
09-19-2013 09:13 AM
Has it suddenly gone slow or is it a new setup and it has always run at that speed?
It helps us to know as much as possible in case something has changed or if it has never been right
Thanks
09-20-2013 03:50 AM
Hi Mark,
This is an entirely new setup. Backups are running slow from the start of the operations.
09-20-2013 04:02 AM
If it has always been like that (and that is VERY slow) then something is clearly wrong
You can setup some logging ndmpagent, ndmp, nbrb, nbjm but this sounds moe like somethin internal to the filer
You need to have a chat with your filer admins to see what information you can get from them in relation to how long the snapvault actually takes and any messages that are beingr raised on the filer during the backup process
Have you seen anything in the actual job log that causes any concern?
Maybe you could show us the policy settings - bppllist policyname -U and also the text from the detailed tab on the job itself.
Thanks
09-20-2013 04:03 AM
Did u get any rply from storage team.....???
09-20-2013 04:53 AM
You mentioned about having Snapvault in your environment, do you also have SnapMirror configured between the 2 NAS?
I have heard about SnapMirror setup, if being too frequent (such as every 15min), can cause slowness when backup window kicks in.
09-20-2013 05:35 AM
You could always load a tape into a tape drive with robtest and do a native dump to the tape from the filer command line and see if there's any difference.
Another is to do a native NetApp dump to NUL that will test how fast you can suck data off the filer.
09-20-2013 06:41 AM
Its seem that u are using local NDMP....
In ur enviroment
You have NDMP storage unit and not media manager storage unit. The data is directly dumped to filer from tape, as it is a local tape drive connected to the filer.Hence netbackup has a very little role to play. Netbackup can only initiate backup and send request to filer for further processing
1) Stop the replication(Snavaulting) for a time being and perform tape backup to check the speed when snapvaulting is not enabled.
2) try use dump command and chck the performance
09-20-2013 06:43 AM
@Mark
please find attached the policy detail and the job detail of one of the completed stream
09-20-2013 06:50 AM
@watsons
We have snapmirror setup within the same filer for the HA (thats what I get from the storage guy :( .. ) and it shud not be affecting the tape backup as we are dealing with F2 only
Snavaulting is done every hour from F1 to F2
@Jim-90
Yes we tried the the dump after loading the tape but it failed with the below error:
>dump 0uf nrst0a /vol/tempvol_s01
DUMP: creating "/vol/tempvol_s01/../snapshot_for_backup.565" snapshot.
DUMP: Using Full Volume Dump
DUMP: Dumping tape file 1 on nrst0a
DUMP: open of nrst0a failed.
DUMP: "Yes" or "No"? yes
DUMP: open of nrst0a failed.
DUMP: Retry the open: ("yes" or "no") no
DUMP: Could not initialize media.
DUMP: Dump failed to initialize.
DUMP: DUMP IS ABORTED
DUMP: Deleting "/vol/tempvol_s01/../snapshot_for_backup.565" snapshot.
please check
09-21-2013 04:01 AM
You should start engaging Netapp support when using dump still gives you error. It's probably not a Netbackup issue.
09-23-2013 10:20 AM
DUMP: Could not initialize media.
manually mount a tape in the drive and try again
09-25-2013 11:28 AM
Hi backup guys,
You need to make sure that if your filer is heavily utilized than snapvaulting and backups should not be occur simultaneously this could make your backups slow.
and before running dump command you need to mount a tape manually.
Also Please check with network team to resolve the same.
09-25-2013 11:37 AM
I believe the cause of the issue lies in the bottleneck at the filer end . Our backups happens to be running at the time when the snapvaulting is also running .
09-25-2013 03:11 PM
An interesting test would be a DUMP to NUL when snapvaulting is happening and when it is not.
Any reference(s) to not doing NDMP backups and snapvualting at the same time or do we have to rely upon forum folklore?