Performance issues on SQL Server due to Netbackup

[miamikk]

In one of our client environment, we have Netbackup 7.5.0.5 performing backups of the whole SQL Server VM using VMWare type backup (like snapshot as per our Intel tech description). When these backups are runnings, the system kinda freezes up IO activity and its impacting SQL Server.

Since I don't have any knowledge (I am a SQL guy), I would like to know there are any best practice recommendations from Symantec for performing backups on SQL Server. If these type of snapshot backups are not recommended and instead use netbackup agent to perform windows type backup ?

Any advise on Netbackup related to SQL would be really appreciated.

[RamNagalla]

what is the size of the VMDK that you have for the SQL VM? total VM Disks size?

did you try that in off hours, i mean when you dont have much load on the server?

what is the VMhardware version? and Vcenter/ESXi verions?

[miamikk]

The SQL Server is not that big (the total size of disk usage is about 100GB). The backups occur after hours but the databases are used around the clock and the downtime (basically when disks freeze up during the backups and SQL becomes unresponsive) is unacceptable.

I have no knowledge about VMhardware or ESXI verions. I am just a SQL guy. 

What I am looking for is if these type of VMDK backups are not recommended for SQL Server which in most environments are used 24/7. 

[RamNagalla]

to be open.. i did have this issue.. where i have turned off the VMware backups on those machines and used the backups native netbackup agent backups ...

but i am sure there is something that we can tune to get rid of this issue.. generally when snapshot is happing it will create a redo logs..  all the I/O are taken by redo logs as long as the Snaphhost is active, so tuning this might help to get this issue resolved.. but I am not sure how to do that.. but VMware team might have some more inside on it...

let me know to if you find the way..  

[MortenSeeberg]

You would have to troubleshoot this with the VMware guys as well. I have seen this happen when VMware creates the snapshot, and its usually related to I/O performance on VMware storage. VMware triggers a Microsoft VSS snapshot of the server, and VSS contacts MS-SQL and asks it to suspend for the time the snapshots is created. This time can vary depending on the load and speed of the storage sub-system.

Moving the SQL server to faster storage could help the issue. Also I have experienced the the problem is smaller with vSphere 5.1 and forward, because it handles I/O pausing much better than previous versions.

Another way of avoiding the issue is to stop taking VMware backups, and backing up the server with standard agent backup for both OS and SQL, but that might not be the way forward in your setup, although it would give you the option of running transaction log backups.

[rookie11]

try to run only backup of smallest SQL database then check if same problem exist on server

[miamikk]

I have asked the Intel guys to provide additional information on and any possible tuning to overcome this issue. The alternate is to go back to standard netbackup agent performing the disk backups.

[huanglao2002]

Are you use lan or san for backup?

[Mark_Solutions]

Are you using a VMware policy type and actually backing up the SQL databases during the backup or just backing up the operating system (i.e. vmdk) and backing up the databases using the agent (or via SQL dumps)?

If you are not specifically backing up SQL during the VMware backup the issue is being caused by VMware tools calling on VSS to snapshot the server and its applications - so it tries to quiesce SQL as well as the rest of the system,

If you back up SQL using a different method then you need to tell VMware tools not to snashot applications during the backup

This is done by adding a line to the tools.conf file as per option B in this tech note:

http://www.symantec.com/docs/TECH141518

See if this helps