Missing files/folders from a VMware backup

[Hamza_H]

Hello !

We are facing a problem when we are trying to restore individual files/folders from a Vmware backup.

the backup finishes with no problem (status code = 0), no error or warning messages in detailed status on both backup and snapshot.

But we can't find some files/folders on both consoles (JAVA & BAR)

The problem arrived just after the upgrade to 8.1.2 .

We have another policy (standard) that backs up the same server and the folder missing in the VMware backup is present in the standard backup.

The folder is on an xfs FS and other folders (on the same partition and path) are presents.

The folder's size is 3TB.

on vxms logs I have some messages like :

02/27/2019 07:58:11 : excludeUnallocated:RvpUnixFileSystemMgr.cpp:1595 <INFO> : Excluding clusters on /home/alfdata

The folder is on /home/alfdata => /dev/mapper/data2/alfdata

XFS is supported on NetBackup 8.1.2 , but I can't understand why we have this issue?

Anyone can please help?

In advance Thanks !

[Lowell_Palecek]

>The problem arrived just after the upgrade to 8.1.2.

This can't be true, since single file restore support for XFS began in 8.1.2.

That said, I suggest you open a support case, because XFS support is new and the issue is likely to require someone in Veritas crawling through your vxms logs.

> The folder is on an xfs FS and other folders (on the same partition and path) are present.

To make sure I understand, there are some folders on the same XFS file system that do appear when you browse the image, correct?

Please provide your VCenter and VDDK versions, too.

[Hamza_H]

Hello @Lowell_Palecek ,

Thank you for your response,

That's what I thought too, the support of XFS is new, so it still probably has some problems..

And yes there are some folders on the same XFS file system that do appear when you browse the image,

The esx version is 5.5.0 and the vmx is vmx-10.

and yes I'm considering to open a new support case, but I preferred to do some digging first and share it with you..

[Marianne]

@Hamza_H 

You may be onto something with the vxms log entries that you found.
Although, it seems to be a notice (INFO) that unused blocks are being excluded.

The process of 'excluding unused and deleted blocks' is described in this TN:  https://www.veritas.com/support/en_US/article.100011492

With the option to Exclude unused and deleted blocks turned on, NetBackup has to examine the filesystem and find which blocks are in use. In addition, with this option enabled, NetBackup will perform reads of contiguous used blocks of data (extents).

So, maybe this option does not 'play well' with fragmented XFS fs (yet).

Suggestion:
Disable 'Exclude unused and deleted blocks' option and check results after next backup. 

Submit existing vxms log and new one (with exclude option disabled) when you log Support call with Veritas. 

[Hamza_H]

Hello @Marianne,

Your suggestion seems correct, I'd like to disable the option "Exclude unused and deleted blocks" and give it a try,

Can you please show me how can I disable it?

Thanks !

[Hamza_H]

Is it the Exclude deleted blocks in the VMware tab?

[Marianne]

I assume so, yes. I was just Googl'ing the messages you saw in vxms log. 

I just had a look in the NBU for VMware Admin Guide and saw the following:

NetBackup for VMware: notes on Linux virtual machines
The following notes apply to virtual machines with Linux guest operating systems:
■ NetBackup cannot exclude unused or deleted blocks from the backup if the
virtual machine is configured with software RAID volumes. The policy's Exclude
deleted blocks option is not supported.

So, it seems that you need to disable the option in any case. 

[Hamza_H]

Hello All,

Just a reply to make this topic updated,

After long investigations and discussions with Veritas Support, it turned out that the problem is already been worked on another Etrack : 3970781.

"We can see in ET3970781 that the change required to fix this issue is not yet available. Also the change required to resolve this issue is not a small one and may take some time to be ready."

Looking at the stat -f and ls -li  output  (of the missing folders on the Vm)  it seems that the problem is related to the inode values being greater than the maximum signed 32-bit integer, which is 2,147,483,648.

As mentionned, the problem is not resolved yet, so I will come back again as soon as I get an update (EEB or ETRACK).

Regards

[Lowell_Palecek · Accepted solution]

Did you execute ls -li to check that you have i-node values greater than 2^31 - 1?

The ETrack you reference (3970781) is targeted for a future release after 8.2. For NetBackup 8.2, see the sibling ETrack ET 3977870. The issue is fixed for 8.2. (The reason there are two fixes is that the issue was discovered and fixed only very recently.)

I am drafting a TechNote for the problem this evening. I'll post the reference here when it's ready.

[Lowell_Palecek · Accepted solution]

https://www.veritas.com/support/en_US/article.100045622.html

The ET number I put in the TechNote is the parent to the two ETracks mentioned in the preceding post, as well as a third ET for a support escalation on the topic.

[Hamza_H]

Hello @Lowell_Palecek ,

Thanks a lot for your update, I have marked your reply as a solution because it describes the exact problem that I have and the Technote is gonna simplfy things to persons who may face the same problem.

For your questions, yes I have sent the output of the command ls -li to the support veritas and they highlighted that the missing folder has inodes values greater than the maximum :

2149371168 drwxr-xr-x 13 alfresco alfresco 138 Jan 2 11:50 missingfolder

Thanks again for your help

and thanks to @Marianne too