Sudden severe performance drop

[Adam_Stadnick]

I have a strange issue and Veritas support has proven to be unable to figure it out. I am hoping one of you will have a brainwave and give me something to start with.

We are running Backup Exec on a VM. We are using ESXi 6 and this VM is the only one on the host. We have three hosts total, backup set for all servers is about 350GB.

About a month ago I had Veritas support assist me with a compatibility issue with our tape drive, and starting that night and every night since our backup speed has dropped from ~4000MB/min to 300/MB/min. This means our backups run for 19 hours plus, and are stll running during the day when the system is in use.

No other changes were made to any network or VM resources when this problem started, and at the prompting of Veritas I have tried everything else I can think of. We have tried with and without GRT, we have tried Agent vs VMWare image backups, we have created a new storage volume on the SAN, we have upgraded from FP3 to FP4. Nothing has worked. Their most recent suggestion is to delete our SAN volume and expose the LUN directly to the server, but Backup Exec was working fine with a virtual disk for months previously. 

I hate to think there's a problem on the network or on one of our hosts that I can't find, as everything else is running perfectly well.

We did have one strange issue where a drive failed on one of the hosts, and after replacing it the backup performance for that host shot back up to normal for one run, then dropped again the next night.

Ever since we have tried doing the fixes suggested by Veritas we also have issues where the Windows agents don't connect or can't be updated, this never happened before.

Am I right in thinking I should just erase the software and do a fresh installation? Maybe even erase the VM (there's nothing else on it)?

Any other ideas would be welcome. I've been working with them on this for weeks and have had no success.

[Larry_Fine]

What is the connection to your tape drive?  Generally, BE isn't supported on VMs with SAS or FC tape drives or libraries.

[Gurvinder]

I have seen sometimes disabling NIC chimney, RSS, netdma help. Also compare the Nic of each VM,if they are the same (vm settings) https://support.microsoft.com/en-in/kb/951037

[Adam_Stadnick]

Larry, Our connection to the tape drive is SAS, but I see that I failed to mention that this is disk performance only - our duplication to tape is actually quite fast, no problems there. 

Gurv, I'm willing to look at those settings (all the NICs are identical however, all of our VMs are from the same templates, the only changes aside from CPU/RAM allocation are what software and features are installed), but again we didn't change anything on the setup of any VM or host when this happened. I have a hard time believing that they would all spontaneously change such a setting across the board. 

[Gurvinder]

What is job rate on a few GB backup selected from BE server to Disk and then backup the same local data to tape. Lets see what is the difference in job rate when local data is backed up and then take it forward from there.

[CraigV]

SAS/SCSI/FC are not supported when BE runs on a VM at all.

Connections to VMFS-based volumes for B2D/Dedupe backups and iSCSI connections to disk and tape are. You're running an unsupported configuration and if you log a call with Veritas they are not obliged to assist you at all.

If that VM is the only VM on the host, why not convert the VM to a dedicated and physical backup server instead? Which also means you get support...

Thanks!

[Adam_Stadnick]

Gurv, the backup rate on small backups to disk is normal whether it is from the server's own files or from another server across the network to disk. It appears that only full system backups have this problem. If I recall correctly the transfer rate was in the same neighborhood as previous backup jobs, around 4000MB/min. Tape backups are consistently 8-9000MB/min even after the problem started. The slowest one I have is 7,892.72 for duplication to tape.

CraigV, I inherited this setup from the previous admin and have not wanted to make a change to the server on that scale without a very good reason. We are also expecting to have other VMs on that host shortly so using it as a physical server is not an option right now. In any case, it worked perfectly fine for at least a few months since I came onboard, and Veritas has not mentioned anything about the server being virtualized as a possible source of problems - and yes, I did mention it to to them up front. No idea why that is, not looking to argue the point, just adding information. This is the first I've heard of there being a problem doing things this way, although I don't understand how it could suddenly change so drastically from good performance to bad.

[RobinMau]

Try a further process of elimination. Can you backup the backup exec server itself to the storage you're having an issue with? Create a test VM on the same host and try back that up. 

Run the Veritas Assist tool to do a health check.

Did you and Veritas forget to turn debug mode off during your troubleshooting previously?

[Adam_Stadnick]

Backing up the BE server to the same storage runs at blistering fast speeds, approaching 10GB/min. To me, this means the storage is fine, and the connection to the SAN is fine as the server is running off the same LUN as the backup storage location. This leaves us with a network problem or a configuration problem, right?

Veritas Quick Assist shows several issues but they don't make any sense. For example, it claims the logon account I am using is not a member of Domain Admins (it is), UDP port 445 is not open (it is), UAC is on (it's not) and so on. There are also several errors relating to deduplication but we have not set up deduplication yet.

I believe debug mode is off and I have asked Veritas repeatedly to make sure it is off. The debug monitor is not running and I don't see any evidence that it's on. Is there some non-obvious place where it could be enabled?

[Adam_Stadnick]

Okay, this is weird. If I select a VM from the list and back it up, it runs at the original speed! If I select a host and pick the same VM with the same settings from the host it runs slow.

What.

[Adam_Stadnick]

And now backing it up from the host is running normally again as well. It actually ran faster than Veeam using the same methods.