storagefilewatch not updating watchfile

[tim_d_jordan]

I have this alert atm in vault

There are 1989037 savesets that have not yet been backed up or replicated for Vault Store 'FileArchive'

My backups were indeed sucessful and the storage service has been updated and the stores and indexes are not in backup mode.

When I open the watchfile table in the EVFileArchive DB I see 1989037 rows with ItemSecured=False

I dtraced the storagefilewatched process

I see lots of the following message

10704752 07:56:07.830  [3628] (StorageFileWatch) <1284> EV:L The Collector for Partition 1706224C379580C4DA099D85C6851FC2D1q10000Vault1.ledcor.net is going to Collect file E:\FA_04\2012\01-29\4\150\4150EA9C3887529285F1A959DB158021.DVS

 but not a lot else.

Storage never reduces and files are not collected.

Any help would be appreciated.

Tim

[JesusWept3]

OK so some questions

1. What storage are you using to store your EV Partitions? (regular NTFS, SAN, NetApp, Centera etc)?
2. Are you using the ArchiveBitTrigger.txt or partition XML files or just using regular backups to remove the archive bits?
3. What Backup Software are you using?
4. Is the backup software removing the Archive Bit from the files?
5. What type of backup are you doing? full? incremental?
6. Are you ever restarting the storage service or just taking it in and out of backup mode?
7. How many vault stores do you have and are they all in the same vault store group with sharing enabled?

[tim_d_jordan]

1. What storage are you using to store your EV Partitions? (regular NTFS, SAN, NetApp, Centera etc)?

iscsi NTFS

 2. Are you using the ArchiveBitTrigger.txt or partition XML files or just using regular backups to remove the archive bits?
Using Netbackup 7 integration with archive bits set on the partitions

3. What Backup Software are you using?

Netbackup 7
4. Is the backup software removing the Archive Bit from the files?

Yes
5. What type of backup are you doing? full? incremental?

Full
6. Are you ever restarting the storage service or just taking it in and out of backup mode?

Yes, I have also Dtraced it and I can see it walking the partition

7. How many vault stores do you have and are they all in the same vault store group with sharing enabled?

All of the ones that are having the issue at present are in the filearchive group. There are 5 file archive partitions. This has started happening (i.e. this all worked before) on the partition 4 which rolled over onto partition 5

[Chris_Warren]

Just a thought, are we sure that the new Partition 5 had been included in the backup job?

[tim_d_jordan]

Thanks, this is using the NBU7 integration with the EV_OPEN_PARTITION directive and the vault store names specified.

The files are being committed to tape, I can see the file count in the NBU job. this is more a vault problem than a backup one, as vault is refusing to admit it.

[JesusWept3]

Well literally all EV does is check for the Archive Bit on each item to make sure its been backed up.
but where it becomes tricky is if you have anything archived that spans multiple vault stores and it gets shared out, that other shared piece needs to be backed up and secured also.

So for instance lets say you have a DVS file on server1 and it has attachments that have already been archived on Server2, the item on Server1 might be backed up fine, but each time EV checks to see if its really been backed up, it will check to see if the items (The DVSSP and DVSCC) have also been backed up.

If Server2's items have not had the Archive bit removed, then it will show the item on Server1 as not being secured.

I would be interested in seeing a dtrace of StorageFileWatch from start up, and i'd probably change the time of the collections run so that when you start up it doesnt immediately attempt to collect everything and make extra noise within the dtrace

All the dtrace above is telling you is part of the collections process, not the backup process

[tim_d_jordan]

I see these is the dtrace when it starts

8797 14:05:27.316 [6136] (StorageFileWatch) <10216> EV:L CVSDBAccessor: penConnection Information: Created a new ADOContext and DataAccess Object for Provider=SQLOLEDB;Server=701-0212;Database=EVFileArchive;Integrated Security=SSPI connection string.
8798 14:05:27.316 [6136] (StorageFileWatch) <10216> EV:L {CVSDBAccessor: penConnection} (Exit) Status: [Success]
8799 14:05:27.316 [6136] (StorageFileWatch) <10216> EV:L {CWatchFileScanRequestBase::GetWatchFileEnumBatchSize} (Entry)
8800 14:05:27.316 [6136] (StorageFileWatch) <10216> EV:L CWatchFileScanRequestBase::GetWatchFileEnumBatchSize Warning: Could not read the chunk size for Watch file batch enumeration. VaultStoreIdentity = [2] and PartitionIdentity = [3].
8801 14:05:27.316 [6136] (StorageFileWatch) <10216> EV:L {CWatchFileScanRequestBase::GetWatchFileEnumBatchSize} (Exit)
8802 14:05:27.316 [6136] (StorageFileWatch) <10216> EV:L {CWatchFileScanRequestBase::GetMaxEntryInSISPartIdCache} (Entry)
8803 14:05:27.316 [6136] (StorageFileWatch) <10216> EV:L CWatchFileScanRequestBase::GetMaxEntryInSISPartIdCache Warning: Could not read the max entry count for sis part id cache. ValueUsed is [100000] VaultStoreIdentity = [2] and PartitionIdentity = [3].
8804 14:05:27.316 [6136] (StorageFileWatch) <10216> EV:L {CWatchFileScanRequestBase::GetMaxEntryInSISPartIdCache} (Exit)
8805 14:05:27.331 [6136] (StorageFileWatch) <10216> EV:L {CVSDBAccessor::GetUnsecuredSavesetsCount} (Entry) PartitionIdentity = [3]
8806 14:05:27.331 [6136] (StorageFileWatch) <10216> EV:L CVSDBAccessor: penConnection Information: An ADOContext Object already exists for Provider=SQLOLEDB;Server=701-0212;Database=EVFileArchive;Integrated Security=SSPI connection string.

[John_Chisari]

That dtrace snippet is OK, just a question - what are you collection settings set for?

What could possibly be happening is that you are collecting files before SFW has had a chance to check to see if the files (DVS, DVSSP) are backed up and marking them as True in the watchfile table.

Suggestion here is to use the trigger file mechanism to clear the backlog.

[KevinZW]

As JW2 has mentioned it would be good to see a dtrace of StorageFileWatch, specifically to see if there is an (Exit) recorded for it:

{CVSDBAccessor::GetUnsecuredSavesetsCount} (Entry) PartitionIdentity = [3]

i.e there should be a dtrace entry a few lines after the (Entry) line above like this:

{CVSDBAccessor::GetUnsecuredSavesetsCount} (Exit) Status: [Success]

[tim_d_jordan]

I only seem to see those entries for my mailarchive ones

I do not see any for my filearchive store.

Is it possible that there are just too many entries for this to deal with?

Tim

[tim_d_jordan]

Still no progress on this.

I tried manually changing one of the entries in sql to backupcomplete=1 and it still didn't do anything.

Two things I have noticed

1. if I look at the partition properties and go to the backup tab the last scan dates are not updating for the filearchive partitions but they are for the mailarchive ones

2. the mailarchives ones were created without acls but the newer file archive ones were created with acls.

Tim

[JesusWept3]

Never just set backup complete to 1 because that poor EV will start to attempt to delete those journal archive entries and it will generate an error for each one because of the dependency on the records in the watch file table

[tim_d_jordan]

I am trying to make the process described here happen

http://www.symantec.com/business/support/index?page=content&id=TECH68881&actp=search&viewlocale=en_US&searchid=1296201086039

I am having a hard time with support. This has been open since Christmas in some guise or another and nothing seems to clear these tables down.

[JesusWept3]

tim
If you want I can look at the DTrace for you, i used to be on backline for EV in the US for several years, so if you want, stop the storage service and set a DTrace for StorageFileWatch and then start the storage service and then let it run for 30 mins or so, then exit the dtrace.

Afterwards, just PM me and i'll have a look at the dtrace for you.

I actually wrote more about the backup changes in EV8 here:
http://news.support.veritas.com/connect/pt-br/articles/changes-backup-procedures-enterprise-vault-8

[tim_d_jordan · Accepted solution]

This was a bug in 9.03

http://www.symantec.com/business/support/index?page=content&pmv=print&impressions=&viewlocale=&id=TE...