NetBackup 7.1 SYMCquiesce issue!

[elanmbx]

Working toward getting our Linux VMware guests integrated into our vStorage API backup configuration.

I have a test guest (64-bit Red Hat 5.4) on which I installed the 64-bit version of the SYMCquiesce utility:

Tried a test backup and checked /opt/SYMCquiesce/logs/052011.log and the only thing that got logged:

"Unable to allocate shared memory"

So I uninstalled and installed the 32-bit version of the SYMCquiesce utility.  Which at least throws errors as if it's *trying* to quiesce filesystems:

 Build version: SYMCquiesce 1.0.0-001 Tue Nov  2 00:32:51 IST 2010

Stats - Fri May 20 12:38:16 2011


Freeze of volume [/] returned status [-1]

Thaw of volume [/] returned status [-1]

Freeze of volume [/var] returned status [-1]

Thaw of volume [/var] returned status [-1]

Freeze of volume [/boot] returned status [-1]

Thaw of volume [/boot] returned status [-1]

Freeze of volume [/kickstart] returned status [-1]

Thaw of volume [/kickstart] returned status [-1] 

Does anyone have any insight into why the 64-bit version of this utility is throwing the "unable to allocate shared memory" message?

[elanmbx]

Gah!  I removed the 32-bit version and reinstalled the 64-bit version and now I get what appears to be a success:

 # cat 052011.log 


Build version: SYMCquiesce 1.0.0-001 Tue Nov  2 00:32:52 IST 2010

Stats - Fri May 20 12:53:10 2011


Freeze of volume [/] returned status [0]

Thaw of volume [/] returned status [0]

Freeze of volume [/var] returned status [0]

Thaw of volume [/var] returned status [0]

Freeze of volume [/boot] returned status [0]

Thaw of volume [/boot] returned status [0]

Freeze of volume [/kickstart] returned status [0]

Thaw of volume [/kickstart] returned status [0] 

This really makes me uneasy with regards to filesystem quiescence on Linux guests... any insight anyone might have would be greatly appreciated!

[oschistad]

This probably won't help with your unease, but we have seen several instances of Linux VMs actually crashing due to the freeze binary from SYMCquiesce.

What's even more worrying is that there does not seem to be any predictability for these freezes - one particular VM tends to crash every time the scheduled backup runs, but usually works when we do a manual backup later in the day. I suspect that the backup window coincides with a high-IO period for that particular VM, but it is certainly not good to have production hosts that crash every time you back them up.

I am also very disappointed with the documentation for the product. Through my own testing I have discovered that Linux guests in VMware must run the same build of VMware tools as the ESX before the pre-freeze-scripts get invoked - if its tools are older then no attempt is made to run this script. This is not documented anywhere at Symantec or VMware.

And it does look like you need to have SYMCquiesce running on Linux guests for reliable backup - VFM error 3 are very common for non-quiesced ext3 filesystems it seems.

[elanmbx]

Interesting - we have had some Linux-VM freeze issues with just a few guests.  We've pulled them from teh vStorage policy, but think it may have been due to the *really* old Redhat version that was running.  We've patched the systems to current and I'm going to try putting them back in the vStorage policy to see if that has taken care of the issue.  Out of curiosity - what patch levels have your Linux VMs been?  Current?

Thanks for the input!

[oschistad]

Well, these VMs are all RHEL 5.6 and mostly up to date with security fixes. We also have RHEL 4 guests, but since the SYMCquiesce binary is linked against a more recent glibc version it cannot be installed on these.

Our workaround has been to create a separate policy for these clients, that does not perform a file-level backup (ie Full VM backup instead of Mapped Full VM backup), and then install the regular netbackup client and perform a file-level backup through that as well.

This works but is cumbersome compared to just having one job per datacenter pull in all VMs tagged with a flag, and back them up both image- and file-level.