11-07-2016 11:22 AM
Netbackup 7.6.1.2
Recently we discovered that Netbackup is not backing up specific files properly until they are 96 hours hold. These files have a .AMA extension, and are generic binary data.
On the day the file is created, Netbackup backs it up with zero errors, but shows that it with a size of 0 bytes. The files definitely have data in them and are absolutely NOT 0 bytes on the disk.
These 0 byte files cannot be restored, and generate a "2800" error code in Netbackup, which I've never seen before.
This continues through the next three days. The files are 0 bytes, until the fourth day. On the fourth day, the file is backed up properly, and Netbackup shows its proper size.
There are other files in the same directory, but they are not affected. Only files with .AMA extensions behave this way. Multiple directories across two different hosts at two different sites with two completely different Netbackup installations are behaving this way. Both hosts are for the same application.
Has anybody run into this?
Solved! Go to Solution.
11-09-2016 01:14 PM
Thank you all for your suggestions. Unfortunately none of them panned out.
We opened a case with Veritas Netbackup support. The astute Veritas technician noticed that the "hard link count" on the 0 byte files was 2, indicating that the 0 byte files were the second hard link to the data.
According to the Veritas technician, Netbackup only backs up the data from the first hard link. The second hard link is backed up as a placeholder, and shows as 0 bytes.
The data is being backed up fine, just in a different directory. Now we have to figure out why it is taking 96 hours for the system to process these files. It should not be taking that long.
11-07-2016 11:39 AM
11-07-2016 11:59 AM
Think the bpbkar log at high verbosity is you best bet at figuring out what is going on.
My guess is that they are locked by some process other than Netbackup for the first 96 hours.
I would try fuser on one of the "0" byte files and of course talking with the person reponsible for the application using these files.
11-07-2016 12:11 PM
Hi ,
If the file that you are backing up is related to database , during the backups open DB files will be skipped and will not be covered in backup.
How is your backups completing is it partial or successful ?
11-07-2016 12:24 PM
The bpbkar logs show nothing out of the ordinary. Backups complete 100% successfully.
fuser shows nothing on the files.
As for your standard questions:
1) Nothing has changed. Investigating it appears that it has been messed up since day one. The application owner has scripts to scan the backups daily to see if their data is backed and they said nothing until recently, so clearly they have not been doing their verification.
2) I can't find a "Netbackup backs up files as 0 bytes for the first 96 hours of their existence" section in the manual.
3) This problem is very hard to describe, and if you don't describe it exactly the same way as it's recorded in the knowledgebase, you get nothing, or you get totally unrelated information. Believe me, I've tried.
11-07-2016 01:20 PM - edited 11-07-2016 01:31 PM
You say that bpkar logs does not show anything, you might have to crank up verbose to 99 and implement bpbkar_tr(ace) if it is still called that. Watch disk space closely when doing this as these settings will very large logs.
If that does not give anything you will have to create support case at Veritas if you havn't already.
Agree that support & VOX search could be a lot better, usually end using google with site:veritas.com when I have the more unusual problems
11-09-2016 11:53 AM
Test to see if the files are locked - can you copy them? Can you edit them? Can you move them?
(Please be carefull not to damage them!)
Can you quesce the database, or backup during an outage to see if they backup successfully?
From the google -
The AMA file extension is related to ArcGIS, a geographic information system from ESRI.
An *.ama file is an ArcMap file
11-09-2016 12:08 PM
Is there any way you can trace wheher the files was used by any other application during the backup failure..and when the backup was completed.
11-09-2016 01:14 PM
Thank you all for your suggestions. Unfortunately none of them panned out.
We opened a case with Veritas Netbackup support. The astute Veritas technician noticed that the "hard link count" on the 0 byte files was 2, indicating that the 0 byte files were the second hard link to the data.
According to the Veritas technician, Netbackup only backs up the data from the first hard link. The second hard link is backed up as a placeholder, and shows as 0 bytes.
The data is being backed up fine, just in a different directory. Now we have to figure out why it is taking 96 hours for the system to process these files. It should not be taking that long.