Netbackup backs up 0 bytes until files are 96 hours old

[DennisGauge]

Netbackup 7.6.1.2

Recently we discovered that Netbackup is not backing up specific files properly until they are 96 hours hold. These files have a .AMA extension, and are generic binary data.

On the day the file is created, Netbackup backs it up with zero errors, but shows that it with a size of 0 bytes. The files definitely have data in them and are absolutely NOT 0 bytes on the disk.

These 0 byte files cannot be restored, and generate a "2800" error code in Netbackup, which I've never seen before.

This continues through the next three days. The files are 0 bytes, until the fourth day. On the fourth day, the file is backed up properly, and Netbackup shows its proper size.

There are other files in the same directory, but they are not affected. Only files with .AMA extensions behave this way. Multiple directories across two different hosts at two different sites with two completely different Netbackup installations are behaving this way. Both hosts are for the same application.

Has anybody run into this?

[DennisGauge]

Some additional information:

The hard link count on these files is two, until the file is 96 hours old:

-rw-rw-r-- 2 etcs etcs 1232896 Nov 4 03:20 FILE1.AMA

[Michael_G_Ander]

Think the bpbkar log at high verbosity is you best bet at figuring out what is going on.

My guess is that they are locked by some process other than Netbackup for the first 96 hours.

I would try fuser on one of the "0" byte files and of course talking with the person reponsible for the application using these files.

[P_Kesavan]

Hi ,

If the file that you are backing up is related to database , during the backups open DB files will be skipped and will not be covered in backup.

How is your backups completing is it partial or successful ?

[DennisGauge]

The bpbkar logs show nothing out of the ordinary. Backups complete 100% successfully.

fuser shows nothing on the files.

As for your standard questions:

1) Nothing has changed. Investigating it appears that it has been messed up since day one. The application owner has scripts to scan the backups daily to see if their data is backed and they said nothing until recently, so clearly they have not been doing their verification.

2) I can't find a "Netbackup backs up files as 0 bytes for the first 96 hours of their existence" section in the manual.

3) This problem is very hard to describe, and if you don't describe it exactly the same way as it's recorded in the knowledgebase, you get nothing, or you get totally unrelated information. Believe me, I've tried.

[Michael_G_Ander]

You say that bpkar logs does not show anything, you might have to crank up verbose to 99 and implement bpbkar_tr(ace) if it is still called that. Watch disk space closely when doing this as these settings will very large logs.

If that does not give anything you will have to create support case at Veritas if you havn't already.

Agree that support & VOX search could be a lot better, usually end using google with site:veritas.com when I have the more unusual problems

[Genericus]

Test to see if the files are locked - can you copy them? Can you edit them? Can you move them?

(Please be carefull not to damage them!)

Can you quesce the database, or backup during an outage to see if they backup successfully?

From the google - 

The AMA file extension is related to ArcGIS, a geographic information system from ESRI.

An *.ama file is an ArcMap file

[Arkya]

Is there any way you can trace wheher the files was used by any other application during the backup failure..and when the backup was completed.

[DennisGauge · Accepted solution]

Thank you all for your suggestions. Unfortunately none of them panned out.

We opened a case with Veritas Netbackup support. The astute Veritas technician noticed that the "hard link count" on the 0 byte files was 2, indicating that the 0 byte files were the second hard link to the data.

According to the Veritas technician, Netbackup only backs up the data from the first hard link. The second hard link is backed up as a placeholder, and shows as 0 bytes.

The data is being backed up fine, just in a different directory. Now we have to figure out why it is taking 96 hours for the system to process these files. It should not be taking that long.