I have an environment with:
I´ve set STU01 (Open Storage) to use MediaServer01 as the media server. Backup operation appeared to start and run as it should be using vmware agent (could see a great throughput of 85MB/s on job details). However the vm MediaServer01 (entire vm) shuts down at the end of the backup.
An interesting point is that the virtual disk becomes unavailable after shutdown. Have to reattach it to the vm MediaServer01 and turn it on again.
Tried both, hotadd and nbd transports, with the same results.
Note: When setting the STU01 to use MasterServer01 as the media server, backups of vms runs succesfully.
Does anybody here have any suggestion to solve this strange problem?
It is curious issue as NetBackup does not touch VM configuration while backup.
Do you deploy any VM aware tool in your environment that monitor VM load and try to move vmdk from one VM to another?
Thanks for your prompt reply.
Agree with you that it´s a curious issue. :(
In this environment, no monitoring tool is available.
MediaServer01 vm has 8GB of RAM, more than enough to handle the amount of jobs (one at a time).
I will increase the RAM to 16GB, rerun a backup and see if problem persists.
If you have any other tip, please let me know. :)
I have recently been investigating the hotadd transport option and found this - not sure if it is relevant, but the symtpoms do match. This is from the VMWare VDDK release notes:
Under known issues and workarounds:
Do not use HotAdd mode to back up the backup proxy
After using HotAdd transport mode to back up the virtual machine that serves as the backup proxy, the backup completes successfully, but during cleanup, regular virtual disk is mistakenly removed along with HotAdded disk. This will be fixed in the next release. If you mistakenly use HotAdd mode, you will have to run the vSphere Client, select the proxy VM, edit settings, and re-add the “Hard Disk” VMDK file from storage.
While it is technically supportable to run the media server in a virtual machine environment, the paradigm doesn't really fit... VM's are great for a number of guests which have bursts of activity, or do not generate a substantial amount of I/O... Media Servers have sustained periods (their whole backup window) of intense activity, generating lots of I/O...
Disagree, and for a small environment, it may be the best option.
We've been migrating our large NBU environment from physical to virtual where able (master and medias) The only place we haven't been able to do it is where we still have tape in our main facility. We have close to 30 media servers, in 10 different data centers.
We have yet to run into an issue where the VMware folks were down our throats due to excessive I/O, memory bursting, etc. and we've had virtual media's for at least the past 3 years, and with our latest cutover to the SQL agent, have close to 40000 jobs running daily.
It works, and works well as long as you tune properly.
And a virtual media server (or at least a client) is the only way to do hot add - don't see a waya round that!
So based on the thread so far it looks like it just need the media server (the VMware backup host) excluding from the policy and maybe that will sort it
It would be nice to have a Symantec Tech hop in, but when we were having issues with one of our media servers, Symantec ran benchmarks using both physical and virtual media servers, and the virtual media server sustained a higher I/O rate than the physical.
We have all of our SQL servers as virtual and with our change rate (which is extreme), never had a performance issue.
You seem to be the exception to the rule. Your ESXi farm is obviously quite large and resource heavy. That's a perfect world. You're quite fortunate. However this isn't the case for most environments. The key word here is "Small" environment. I/O tends to be an issue in larger environments. I would also like to point out that there have been issues regarding deduplication using a Virtual Media Server in any size environment with CPU ultilization and I/O waits degrading performance. I've spoken to many of my contacts within Symantec and the concensus has been Virtual Media Servers are not recommended. Sure it will work. After all a VM is just another computer, right? So why not? The sharing of resources however, tends to restrict its ability to perform in most environments. And depending on how you have your resource allocation dynamics configured in VMware you could lose CPU and Memory resources to a higher priority VM in an instant making things even worse. Not likley a virtual media server is configured with dynamic cpu/mem but its certainly not impossible to think there are some environments out there configured that way.
I wish Symantec would get their stories straight. I passed along your note to 3 engineers and 3 product managers in Roseville and heard two different answers (they probably compared notes) one along your line, and another towards mine. The only real consensus between the two was that backups are generally being run 'off-hours' and during non-business hour windows.
Yes, we are quite fortunate in our main environment in that we have the resources to do what we need (as well as a 10 gig network to leverage in most places) but in talking to folks at Vision and at the Customer Forum, we've found that a number of folks are doing the same. Again, proper tuning can alleviate many of the concerns, and using Resource Limits in NetBackup help reduce the strain on a VMware environment.
Without completely hijacking this thread, I'd be interested in any documentation surrounding dedupe with a VM Media Server. Do you have something you can share? We use both MSDP and OST.