Forum Discussion

tim_d_jordan's avatar
tim_d_jordan
Level 4
13 years ago

storagefilewatch not updating watchfile

I have this alert atm in vault

There are 1989037 savesets that have not yet been backed up or replicated for Vault Store 'FileArchive'

My backups were indeed sucessful and the storage service has been updated and the stores and indexes are not in backup mode.

When I open the watchfile table in the EVFileArchive DB I see 1989037 rows with ItemSecured=False

I dtraced the storagefilewatched process

I see lots of the following message

10704752 07:56:07.830  [3628] (StorageFileWatch) <1284> EV:L The Collector for Partition 1706224C379580C4DA099D85C6851FC2D1q10000Vault1.ledcor.net is going to Collect file E:\FA_04\2012\01-29\4\150\4150EA9C3887529285F1A959DB158021.DVS

 but not a lot else.

Storage never reduces and files are not collected.

Any help would be appreciated.

 

Tim

Enterprise Vault
Error messages
Information Governance
  • JesusWept3's avatar
    JesusWept3
    Level 6
    13 years ago

    OK so some questions

    1. What storage are you using to store your EV Partitions? (regular NTFS, SAN, NetApp, Centera etc)?
    2. Are you using the ArchiveBitTrigger.txt or partition XML files or just using regular backups to remove the archive bits?
    3. What Backup Software are you using?
    4. Is the backup software removing the Archive Bit from the files?
    5. What type of backup are you doing? full? incremental?
    6. Are you ever restarting the storage service or just taking it in and out of backup mode?
    7. How many vault stores do you have and are they all in the same vault store group with sharing enabled?

  • tim_d_jordan's avatar
    tim_d_jordan
    Level 4
    13 years ago

    1. What storage are you using to store your EV Partitions? (regular NTFS, SAN, NetApp, Centera etc)?

    iscsi NTFS


     2. Are you using the ArchiveBitTrigger.txt or partition XML files or just using regular backups to remove the archive bits?
    Using Netbackup 7 integration with archive bits set on the partitions

    3. What Backup Software are you using?

    Netbackup 7
    4. Is the backup software removing the Archive Bit from the files?

    Yes
    5. What type of backup are you doing? full? incremental?

    Full
    6. Are you ever restarting the storage service or just taking it in and out of backup mode?

    Yes, I have also Dtraced it and I can see it walking the partition


    7. How many vault stores do you have and are they all in the same vault store group with sharing enabled?

    All of the ones that are having the issue at present are in the filearchive group. There are 5 file archive partitions. This has started happening (i.e. this all worked before) on the partition 4 which rolled over onto partition 5

  • Chris_Warren's avatar
    Chris_Warren
    Level 5
    13 years ago

    Just a thought, are we sure that the new Partition 5 had been included in the backup job?

  • tim_d_jordan's avatar
    tim_d_jordan
    Level 4
    13 years ago

    Thanks, this is using the NBU7 integration with the EV_OPEN_PARTITION directive and the vault store names specified.

    The files are being committed to tape, I can see the file count in the NBU job. this is more a vault problem than a backup one, as vault is refusing to admit it.

  • JesusWept3's avatar
    JesusWept3
    Level 6
    13 years ago

    Well literally all EV does is check for the Archive Bit on each item to make sure its been backed up.
    but where it becomes tricky is if you have anything archived that spans multiple vault stores and it gets shared out, that other shared piece needs to be backed up and secured also.

    So for instance lets say you have a DVS file on server1 and it has attachments that have already been archived on Server2, the item on Server1 might be backed up fine, but each time EV checks to see if its really been backed up, it will check to see if the items (The DVSSP and DVSCC) have also been backed up.

    If Server2's items have not had the Archive bit removed, then it will show the item on Server1 as not being secured.

    I would be interested in seeing a dtrace of StorageFileWatch from start up, and i'd probably change the time of the collections run so that when you start up it doesnt immediately attempt to collect everything and make extra noise within the dtrace

    All the dtrace above is telling you is part of the collections process, not the backup process

  • tim_d_jordan's avatar
    tim_d_jordan
    Level 4
    13 years ago
    I see these is the dtrace when it starts
     
     
    8797 14:05:27.316 [6136] (StorageFileWatch) <10216> EV:L CVSDBAccessor: penConnection Information: Created a new ADOContext and DataAccess Object for Provider=SQLOLEDB;Server=701-0212;Database=EVFileArchive;Integrated Security=SSPI connection string.
    8798 14:05:27.316 [6136] (StorageFileWatch) <10216> EV:L {CVSDBAccessor: penConnection} (Exit) Status: [Success]
    8799 14:05:27.316 [6136] (StorageFileWatch) <10216> EV:L {CWatchFileScanRequestBase::GetWatchFileEnumBatchSize} (Entry)
    8800 14:05:27.316 [6136] (StorageFileWatch) <10216> EV:L CWatchFileScanRequestBase::GetWatchFileEnumBatchSize Warning: Could not read the chunk size for Watch file batch enumeration. VaultStoreIdentity = [2] and PartitionIdentity = [3].
    8801 14:05:27.316 [6136] (StorageFileWatch) <10216> EV:L {CWatchFileScanRequestBase::GetWatchFileEnumBatchSize} (Exit)
    8802 14:05:27.316 [6136] (StorageFileWatch) <10216> EV:L {CWatchFileScanRequestBase::GetMaxEntryInSISPartIdCache} (Entry)
    8803 14:05:27.316 [6136] (StorageFileWatch) <10216> EV:L CWatchFileScanRequestBase::GetMaxEntryInSISPartIdCache Warning: Could not read the max entry count for sis part id cache. ValueUsed is [100000] VaultStoreIdentity = [2] and PartitionIdentity = [3].
    8804 14:05:27.316 [6136] (StorageFileWatch) <10216> EV:L {CWatchFileScanRequestBase::GetMaxEntryInSISPartIdCache} (Exit)
    8805 14:05:27.331 [6136] (StorageFileWatch) <10216> EV:L {CVSDBAccessor::GetUnsecuredSavesetsCount} (Entry) PartitionIdentity = [3]
    8806 14:05:27.331 [6136] (StorageFileWatch) <10216> EV:L CVSDBAccessor: penConnection Information: An ADOContext Object already exists for Provider=SQLOLEDB;Server=701-0212;Database=EVFileArchive;Integrated Security=SSPI connection string.
  • John_Chisari's avatar
    John_Chisari
    Level 6
    13 years ago

    That dtrace snippet is OK, just a question - what are you collection settings set for?

    What could possibly be happening is that you are collecting files before SFW has had a chance to check to see if the files (DVS, DVSSP) are backed up and marking them as True in the watchfile table.

    Suggestion here is to use the trigger file mechanism to clear the backlog.

  • KevinZW's avatar
    KevinZW
    Level 2
    13 years ago

    As JW2 has mentioned it would be good to see a dtrace of StorageFileWatch, specifically to see if there is an (Exit) recorded for it:

     

    {CVSDBAccessor::GetUnsecuredSavesetsCount} (Entry) PartitionIdentity = [3]

    i.e there should be a dtrace entry a few lines after the (Entry) line above like this:

    {CVSDBAccessor::GetUnsecuredSavesetsCount} (Exit) Status: [Success]

  • tim_d_jordan's avatar
    tim_d_jordan
    Level 4
    13 years ago

    I only seem to see those entries for my mailarchive ones

    I do not see any for my filearchive store.

    Is it possible that there are just too many entries for this to deal with?

     

    Tim

  • tim_d_jordan's avatar
    tim_d_jordan
    Level 4
    13 years ago

    Still no progress on this.

    I tried manually changing one of the entries in sql to backupcomplete=1 and it still didn't do anything.

    Two things I have noticed

    1. if I look at the partition properties and go to the backup tab the last scan dates are not updating for the filearchive partitions but they are for the mailarchive ones

    2. the mailarchives ones were created without acls but the newer file archive ones were created with acls.

     

    Tim