CentOS Virtual Machine File-level Backups Failing

[mlukach99]

We have been running into an issue with our CentOS file-level virtual machine backups.

First I'll explain our environment and then the problem at hand.

NetBackup Version:  7.6.0.2

Master Server: SLES 11.3

Two media servers:  Both CentOS 6.5

Two NetBackup 5230 Appliances.

Guest OS affected:  We've seen this in several varieties of CentOS but specifically 6.5 is our main concern now.

File system format: ext4

In our VMWare policies we check the option to "Enable File Recovey from VM backup" on the VMWare tab and this causes backups for our CentOS machines to sit there and hang for around 15-45 minutes and timeout with a status 40 network connection broken.  We've seen the amount of time attributed to the client read timeout on the particular appliance that this tries to back up on.

After working a case with support it appeared the solution was to have the SYMCQuiesce tool installed on all guests.  This happened to be true for the single guest we tested with but on subsequent tests with different guests we are not having any success with file-level backups.  Even with the SYMCQuiesce tool installed we aren't having successful backups with the Enable File Recovery option checked.

I'm looking for information to see what others are doing, if anything to remedy this issue.  We have hundres of CentOS guests and we are currently doing full backups on these every night.

I can provide any further detail that is necessary.

Thank you.

[RamNagalla]

what about the Vceneter , ESXI and VM hardware versions..?

does the appliance acting as backup host?

[mlukach99]

Thanks for the reply.  

vCenter version = 5.1  

All ESX hosts version = 5.1 Update 2.  

VM Hardware Version = 9

We have our appliances as the backup host yes.  

[mlukach99]

We have since upgraded our environment to version 7.6.1.1 all across the master, media servers, and appliances and we continue to have issues with file-level backups on CentOS virtual machines.

Anyone else out there experiencing this?  The backups all hang and typically end with a status 40 after anywhere from one to two hours.  No files are ever backed up, it appears to just hang and hang and hang and then finally times out.

Thanks for any help.

[sdo]

Apologies, but are you implying that it used to work before the upgrade(s)?

.

I can't see CentOS listed as a supported OS for file level recovery from a VM backup, see page 17 of:

DOCUMENTATION: Support for NetBackup 7.x in virtual environments...

https://support.symantec.com/en_US/article.TECH127089.html

And I haven't found anything in the 7.6 SCL to say that CentOS is treated as RHEL.

[mlukach99]

No this has never worked consistently in our environment.  We actaully do have a few random CentOS servers that actually do backup successfully with file-level recovery checked so I do know it does work in some cases, however the bulk do not work.  

While I understand CentOS is not formally supported per the support matrix you have linked I'm really just seeing if there are other customers that have run into this problem and what if anything they've done to fix it. Even some of the support technicians I've worked with have stated that if it works in RHEL it really should work in CentOS.  

I'll probably open a new case but just looking on the forums here for any extra info.  

[sdo]

The only other things that I can think of (right now) checking is:

1) Whether any other file system types, other than ext4, are present within the VMs that fail backups.

2) Whether the ext4 file systems are 32-bit or 64-bit - if I remember correctly, ext4 can 32-bit or 64-bit, but I could be wrong on this - if there are two versions - then are different versions used across your VMs?

3) Is VMtools up to date within all of the VMs?

4) This one is a guess... Does SYMCquiesce come in different versions for different versions of NetBackup?  If so, is it up to date?

5) Is the VM hardware level consistent across all of the VMs?  (e.g. vmx-07/8/9/10 or whatever it is supposed to be).

[sdo]

I have a question re consistency of behaviour which could become a steer to one's future efforts...

...the backups of VMs that do work, do they consistently work all of the time, or do they have intermittent backup failures even though nothing is apparently changing within the estate (hypervisor, guest, storage, LAN and SAN)?