07-09-2019 05:42 AM
Hyper-V VM 2016 file server backing up to disk via BackupExec 20.4
Differential gets rates between 8mb per min to 30 mb/min
Full backup gets average rates of about 2000 mb/min
Our other file server (same setup) gets about 60-140 mb/min diff but the same rate with full.
At the moment the backup to disk job for the first server is taking around 13 hours to backup 27GB. The full job takes around 19 hours to backup 1.7TB.
The other wierd thing that may be just down to compression or something is that when I look at the backup sets on the backup disk I see that the size of the differentials are reported as 274gb (ave-ish) whereas the job history reports 27gb etc....
Not sure where to start troubleshooting this
07-09-2019 05:50 AM
Is it just the latest diif which is affected?
07-09-2019 05:58 AM - edited 07-09-2019 06:02 AM
Thanks for the reply. My answers below and apologies if any of them are not relevant!
All the diffs are slow from Tuesday to Thursday. None of them finish until late the next day.
All of our backups use VSS so I think there are snapshots happening (limited knowledge on this bit).
This backup in particular starts at midnight when the school is closed - there should be no files open anywhere as all the school PCs shut down at 20:00. No applications run from this server - it's just profiles and user data.
I'll take a look at AV activity but as the server performs fine as a file server (I get no complaints during the day of slow performance etc) I haven't thought of that as a reason. Are there any particular folders etc that should be excluded on the target server as far as backups are concerned?
I can't look at the specific job as it's still running :rolleyes: but the other file server is set to no compression or encryption so I would assume this to be the case with the problem server.
<Edit> the job just finished so I checked it - software compression is set on both the full and diff backups for this server.
07-09-2019 06:46 AM
I'd add the entire Backup Exec folder into the AV exceptions list. C:\Program Files\Veritas\BackupExec or wherever it's installed on your server. Also, it may be worth checking what time the last scan was executed to see if coincides with the backup window.
As for the snapshots, I was thinking of the Hyper-V snapshots. Is your backup server a physical or VM?
07-09-2019 07:04 AM - edited 07-09-2019 07:06 AM
The backup server is physical and is also the media server.
The file sever is a Hyper-V vm
A lot of the symantec subfolders were already excluded from AV on the target and the backup server but I have added the entire c:\program files\symantec\ and c:\program files (x86)\symantec\ to the exclusions.
Meant to add that there is a scheduled AV scan that runs every day at 16:00. No idea on when it finishes though as the task scheduler log has the task starting and finishing all within a minute. When I checked the scan it was on pause and I have no idea if this is supposed to be the case or not. The task has (0x1) as it's last completion result.
07-09-2019 07:14 AM
Do you have your BE server backing up the Hyper-V host or do you have any Hyper-V snapshots scheduled? I'm just wondering how you're protecting the VM data (.vhd's etc) and whether this is conflicting in any way.
Presumably you've added your file server to BE as a Windows server with the RAWS agent and are just backing up folder and files?
07-09-2019 07:22 AM
We backup all three of our Hyper-V hosts. The host with the file server currently on it starts at 03:00 and takes about 30 minutes to run (diff) so it does run at the same time as the file server but only for 30 minutes.
I think we do just normal file and folder backups of the VMs but not too sure.
The only snapshots I can see happening are VSS for open files etc... Again - not too au fait with this bit..
07-09-2019 07:33 AM
So if BE is backing up you Hyper-V hosts and the file server, I presume all these jobs are running at different times? This will avoid any conflicts.
The VSS service will enable temporary snapshotting of open files to allow them to be backed up in a consistent state. You can change the options here using the Advanced Open File menu item in the job definition. By default it's set to Automatic - Allow VSS to select the snapshot provider. For Windows machines, this is usually better set to: System - Use Microsoft Shadow Copy provider.
What is your disk storage? Is it physically separate to the storage for the VMs?
07-09-2019 07:38 AM
The Host and it's file server do back up at the same time but the host only takes 30 minutes to back up. It may be possible to change this but there will always be one or two VMs that clash with the host backup.
Disk storage for the backup server is separate to the SAN for the VMs
07-09-2019 07:41 AM
If you check the disk storage, how many simultaneous writes are allowed?
Also, check the max file size is set to 4 GB
07-09-2019 07:51 AM
OK - Concurrent writes is set to 16
Max file size is set to 50GB
07-09-2019 07:56 AM
Presumably you have a dedicated directory on the disk storage for your file server backups? If you do, I'd change the number of writes on this to 2 or maybe even 1.
Drop the max file size to 4 GB as well, this will make the writes to the storage smaller and could help speed things up.
07-09-2019 08:01 AM
Please have a look at this article:
This behavior is because of the way Incremental/Differential backup is designed to work in Hyper-V environment.
The time taken on the Hyper-V host, to determine and calculate the changes made with respect to the previous backups will vary depending on amount of changes made to the virtual machines. Hence it is not necessary that there will also be a time gain upon running Incremental/Differential backups.
07-09-2019 08:09 AM
OK - I see what you're saying. Problem is we have 19 jobs that run between 18:45 and 03:00 so there will always be more than 2 jobs running at the same time. I'd have to look into it further to work out the max number of simultaneous jobs (and it's nearly home time!) but I worry that limiting writes to 2 concurrent would have a negative impact.
Will change max size to 4GB though and see what happens overnight.
Thanks so far!
07-09-2019 08:31 AM
Ah yeah, well maybe keep it above 2 then!
Let us know how you get on.
07-09-2019 06:50 PM
With bigger size disks, it is not advisable to have a 4GB limit. You would end up with a lot of small files and more disk fragmentation. Also, you are paying for the OS having to constantly allocate new files.
A bigger disk limit is recommended.
Concurrency is the number of writes that BE can do to the disk storage device and this will affect the number of jobs that you can run concurrently. Restriciting concurrency to 2 means that you can only run 2 jobs concurrently if they are writing to the same disk storage.