01-21-2016 08:25 AM
Environment:
NBU 7.7.1 Master
NBU 7.7.1 Media
RHEL 5.6 O/S
VMWare 5.5
Client with error RHEL7 running vm-open-tools
Hi,
I am having an issue with the snapshot backups of a RHEL7 VM client that is running vm-open-tools (VMWare ones not available), the snapshot fails to quiecence the drives, if I run a non quiecence this works fine, the client does have a CIFS mount mapped to it from the OS
Looking in the NBU logs I dont really see that much, well nothing that I can make out, in the client /var/log I see the following: error
Jan 21 11:08:11 lonbfbvmwwing4 kernel: CIFS VFS: cifs_mount failed w/return code= -13
Jan 21 11:08:11 lonbfbvmwwing4 vmsvc[710]: [ warning] [vmbackup] Error freezing filesystems.
Jan 21 11:08:24 lonbfbvmwwing4 vmsvc[710]: [ warning] [vmbackup] Error freezing filesystems.
I have looked at the CIFS mount on the client and this looks Ok however it is a HDI mountpoint.
Was wondering if this could be a VSS issue but I am unable to run an advanced install of the vm-open-tools to check if this option is selected, if anyone has any experience with vm-open-tools it would be appreciated.
I am able to take a quiecence backup usning VM directly but not via NetBackup
Any help with the above would be appreciated.
Kev
Solved! Go to Solution.
01-21-2016 09:53 PM
01-21-2016 08:44 AM
HI Kevin,
when you perform a snapshot with NBU what is the error from the VC showing as?
01-21-2016 08:54 AM
Hi Doug,
In the vSphere client window I get the following error message
An error occurred while taking a snapshot: msg.snapshot.error-QUIESCINGERROR.
Kev
01-21-2016 09:29 AM
I think you need the symcquiesce utility installed. It supposed to work with vmware tools to quiesce the machine.
Read about it here: https://www.veritas.com/support/en_US/article.HOWTO70978
In the requirements section it says that it needs VMWare tools, not sure if it will work with the open tools you mentioned.
01-21-2016 09:57 AM
Original log contains what looks like permissions issue
kernel: CIFS VFS: cifs_mount failed w/return code= -13
make sure the mount is really valid and accessible
01-21-2016 10:31 AM
Hi Kevin,
Silly question but have you tried running a quiesced snapshot with the CIFS share unmounted?
This certainly does sound more like a VMware issue rather than something that Netbackup is failing to do.
01-21-2016 10:33 AM
Hi Areznik, just installed the SYMCquiesce but same error.
Hi Will, I am able to access the mount point with out any issues
Just wondering if it is something to do with the vm-open-tools.....
01-21-2016 11:04 AM
The plot thickens, just found this in the SYMCquiesce log file
Build version: SYMCquiesce 1.0.0-003 Tue May 13 17:42:53 IST 2014
Stats - Thu Jan 21 18:26:02 2016
Freeze of volume [/mysql-data] returned status [-1]
Thaw of volume [/mysql-data] returned status [-1]
Looks like it may be the mysql area
Kev
01-21-2016 09:53 PM
01-22-2016 12:50 AM
Thanks Marianne,
This is a new server for an application being bought into the company, I will speak to the server owners and find out how they will be managing the MySql DB backups.
I will be moving this over to a standard client rather than a VM policy based backup
Thanks to everyone who has helped with this issue.
Kev
01-22-2016 12:53 PM
Apologies Kevin, I don't mean to teach you how to suck eggs... the above is perhaps more for our newer, and perhaps less experienced members, who may stumble across this topic... in how to think "enterprise" backup... after all, this product is named: NetBackup Enterprise Server.
And the above behavioural characteristics at the VM layer is probably still true for any backup product... until you get into the realms of advanced storage array integration.
01-22-2016 12:57 PM
my 2 cents...
.
Recap – the general styles of backup are:
A) Plain file system agent ONLY (skipping raw database files - but securing a copy of dump/export/datapump/RMAN-on-disk).
B) Plain file system agent PLUS database agent (skipping raw database files).
C) VM style, either “whole VM with no file cataloguing” or “whole VM with file cataloguing” ONLY.
D) VM style, either “whole VM with no file cataloguing” or “whole VM with file cataloguing” PLUS leveraged database agent.
(although there are others - e.g. flashbackup, and other complex snapshot based technologies)...
.
The rest of my text is pertinent to item C) above.
.
The issues with backup style C) above, for VM style backups of medium to large database servers… are:
.
Having said all that. Then, if your VMs contain small non-volatile databases (which are doing their own dumps/exports/data-pumps/RMAN)... AND you are taking your VM style backup AFTER these database application dumps/exports/datapumps/RMAN have occured, then there really is no problem doing VM style backups of said small virtual database servers - as long as you remember to delete/discard the restored raw database file before attempting recovery from the dump/export/datapump/RMAN backup which should have also been captured within the VM style backup (as long as the backup ran after said dumps had completed).
In summary, the problem gets worse as the virtualised database servers get both larger and/or more volatile - with posisbly huge amounts of wasted IO, and resource, at multiple and various different times and stages, for files which will very likely be useless upon restore. In which case, it really does become a very good idea indeed to use either backup type A) or backup type B) above... or a backup of type D) above (but even type "D" above will still accrue lots of "snapshot related" IO). The only way to avoid VM level snapshot files growing very large during backups is to use backup type A) or B) above.
01-25-2016 10:07 AM
Great writeup sdo, im bookmarking this for later. I think you should repost this as a blog for more visibility.
One thing that always frustrates me with these backups is that you often see servers built by people unaware of this problem, and they will put the active DB and the dumps on the C:\ drive with the OS. Now if I want to backup the OS with snapshots, I have no choice but to snap the DB as well, and end up running into a lot of the problems you described. AFAIK excluding *.mdf and *.ldf doesnt really help, as these files still end up getting snapped. Do you know of any other clever solutions for this kind of scenario, other than asking the server owners to move their stuff to a different drive?
01-25-2016 09:31 PM
01-26-2016 02:32 AM
SDO, no apology needed, great write up and certainly a reference document that should be blogged
Kev