Forum Discussion

Thehound's avatar
Thehound
Level 3
10 years ago

IgnoreArchiveBitTrigger - Should it clear the Items Awaiting Back Up?

Hello Everybody,

This is for EV8.0.5. The company uses backup software that doesn't reset the archive bit, as a result there are scripts that create a blank text file called IgnoreArchiveBitTrigger.txt in each of the Vault Partitions. It has been set up as per the Symantec guides, and in the main has been working as expected without a problem.

Recently however it has started to act up a bit. There were problems with the backups which are now resolved (the backups were taking about 9 hours to complete, so we now only backup items less than 2 years old by excluding the drives with the older data), but everyday I get a warning that :

There are 35263 savesets that have not yet been backed up or replicated for Vault Store 'MailboxVS04'. There is a build-up of savesets that Enterprise Vault cannot confirm have been backed up or replicated.

Example symptoms: archive-pending items remain in mailboxes, leading to calls to your Help Desk from users; store sizes do not decrease, even though items have been archived.

There may be a problem with backups or replication

 

My understanding was that when the partition comes out of Backup Mode the StorageFileWatch process looks first for an XML file and if that is not present next looks for the IgnoreArchiveBitTrigger.txt - if it finds this then it renames it .old and makes a note of the date modified - it then assumes that anything up to that point is safely backed up and changes the Pending Items to full Archived Items.

So if that is accurate then I should be able to essentially fool EV into thinking a back up has taken place successfully simply by putting the partition into Backup Mode - deleting the IgnoreArchiveBitTrigger.old and creating a new IgnoreArchiveBitTrigger.txt - finally set the partition back to normal - it will read the modified date of the .txt rename it as .old and then anything up to that point will change from Pending to Archived.

I have a number of mailboxes, all archived by the same EV server that have items pending for 8 weeks - they have a one month policy - that is EV will attempt to archive anything older than one month, so having items outstanding for 8 weeks is a bit of a problem. MSMQ is empty each morning, so things aren't just piling up there.

I know they have been backed up as I have a daily report from the backup team, the company uses Tivoli TSM for backups, and I see the start/finish times and the word "Success" in large red letters each morning. This is good and makes me feel warm and safe.

But those items are still pending. The users are complaining and harmony in the workplace is not being achieved. So I thought it should be an easy fix, set the IgnoreArchiveBitTrigger file, cycle through BackupMode and Hey Presto everyone is happy. EV thinks a backup has happened so it changes all those Pending Icons to Archived and that is the end of the tale.

I have run a DTrace from the Console selecting Items Remain Pending as the nature of the problem, this returned an awful lot of information but nothing I would call useful (to me at any rate). While running the DTrace I did a Run Now for that mailbox, also did a Sync and for good measure knocked the Backup Mode on and off again. I can see this all happening and can see the inthe DTrace where it looks for the IgnoreArchiveBitTrigger file, that is found ok. Just it never switches the pending items to done.

So does anyone know why this little ploy doesn't work? What am I missing? Something fundamental? Where does the error get it's information? Once I reset the BitTrigger file how is it managing to tell me that there are 35000 items awaiting backup? Surely it should be well and truly fooled into thinking it just had a backup?

 

Thanks.

 

 

 

Basics
Best Practice
Configuring
Enterprise Vault
Error messages
Information Governance
Troubleshooting
Windows Server (2003-2008)
  • dcVAST's avatar
    dcVAST
    Level 5
    10 years ago

    How many partitions are there for MailboxVS04? Could there be unsecured items in some old closed partitions? Might be worth it to remove all trigger files in all partitions (IgnoreArchiveBitTrigger.txt or .old and PartitionSecuredNotification.xml or .old if they exist) and place new IgnoreArchiveBitTrigger.txt files.

    Have you restarted the Storage Service recently and are you actually seeing the triggers renamed to .old with the current date?

  • AndrewB's avatar
    AndrewB
    Moderator
    10 years ago

    i've seen fragmentation in vault store databases lead to this issue. worth checking and making sure you're regularly performing maintenance on your sql databases. if all else fails, support can work with you to manually go in and mark the items as backed up.

  • Thehound's avatar
    Thehound
    Level 3
    10 years ago

    Hello,

    Well lets see -

    MailboxVS04 has 3 closed partitions and 1 open one plus a rollover ready to go. The script that takes it out of Backup Mode recreates the IgnoreArchiveBitTrigger across all the partitions, open - closed and rollover each morning after the backup finishes and before the archiving run starts.

    All servers were rebooted recently - during last week.

    If I manually run the scheduled tasks that controls the post backup script I can see the IgnoreArchiveBitTrigger file gets renamed from .txt to .old

    Haven't checked anything on the SQL box, I'll make sure the maintenance tasks are running and are doing a defrag.

    This morning the properties for Mailbox VS04 have :

    Last Item secured 25/08/2015 08:37:16

    Last Scan:
    Scan State: Completed
    Last Scan Started 25/08/2015 07:57:28
    Last Scan Finished 25/08/2015 08:37:21

    Summary
    Unsecured Items : 472
    secured Items : 9198
    Total Items 9670

    Unsecured Items
    Unsecured items in local partition 387
    Unsecured SIS parts in local partition : 0
    Unsecured SIS parts in other partitions : 85

     

    And I have the same status alert :

    There are 9921 savesets that have not yet been backed up or replicated for Vault Store 'MailboxVS04'.

     

    I'll start with the SQL defrag and see if that helps. Puzzled by the above messages though, if there are 9921 savesets not backed up, is that different from Unsecured Items?

     

    Thanks for your replies.

     

  • dcVAST's avatar
    dcVAST
    Level 5
    10 years ago

    So it went down from 35k to 9k. You may just need to look at changing your monitoring thresholds along with when your pre and post scripts and being kicked off.

    You're likely post-processing older messages but new ones are continuing to be archived.

    Also see: https://www-secure.symantec.com/connect/blogs/sql-maintenance-enterprise-vault

     

    Here are some SQL queries to help you monitor in realtime if you are kicking off the scheduled tasks:

     

    /* Run these queries against their respective database (EnterpriseVaultDirectory and Vault Store Database */

     

    USE EnterpriseVaultDirectory

     SELECT     

         CE.ComputerName,

         VSE.VaultStoreName,

         VSE.VaultStoreEntryId,

         CASE BackupFlag

             WHEN 0 THEN 'Never'

             WHEN 1 THEN 'Immediately After Archive'

             WHEN 2 THEN 'After Backup'

         END as 'RemoveSafetyCopy',

         BackupModeRefCount

     FROM

         ComputerEntry CE

         left  join StorageServiceEntry SSE on CE.ComputerEntryId = SSE.ComputerEntryId

         left join VaultStoreEntry VSE on SSE.ServiceEntryId = VSE.StorageServiceEntryId

     ORDER BY ComputerName

     

    Use EVVSMailboxVaultStore_2 /* <<<<<<<< Enter your vault store DB name */

    select count (*) from dbo.JournalArchive where Backupcomplete = '0'

     

    Use EVVSMailboxVaultStore_2 /* <<<<<<<< Enter your vault store DB name */

    select count (*) from dbo.WatchFile where itemsecured = '0'

     

    /*Index*/

     

    Use EVVSMailboxVaultStore_2 /* <<<<<<<< Enter your vault store DB name */

    select count (*) from dbo.JournalArchive where IndexCommited = '0'

     

    USE EnterpriseVaultDirectory

    SELECT * FROM View_IndexRootPath WHERE BackupMode = 1

  • Thehound's avatar
    Thehound
    Level 3
    10 years ago

    Thanks for your advice on this. This morning I have no warnings about items remaining unsecured, but those Pending Icons are still there, so the backup has completed - the IgnoreArchiveBitTrigger file has been written, but the final step of changing the Pending Icons still hasn't happened.

    I've had a chat with the SQL team and the maintenance was happening weekly, so we've now set that to happen each night - which will hopeuflly help going forward.

     

    With the Pending Icons, they should change pretty much straight away once the store comes out of Backup mode - shouldn't the Storage File Watch kick in and start changing them at that point?

    My thinking is that for some reason these are now being ignored. Not sure why, but they certainly aren't being touched by the gentle hands of EV. So would reverting them back be a good plan? Change the Pending Shortcut Timeout value to 0 on the Properties - Advanced tab and run the Archiving task in Report Mode to reset all the items that are pending, then let things go back to normal and those older items should be picked up in the next scheduled archive run.

     

     

  • dcVAST's avatar
    dcVAST
    Level 5
    10 years ago

    Great. Yeah you can set PendingShortcutTimeout to 0 and run in Report mode; or just leave it at a vaule of like 5 or 10 to clear any items stuck pending.

    If you run the SQL query do you get counts for backupcomplete?

  • Thehound's avatar
    Thehound
    Level 3
    10 years ago

    Coming back to this old thread just in case anyone has any bright ideas they'll share...

     

    So to re-cap.

    There are 7 EV servers archiving 7 Exchange servers.

    Archiving is working and there are no problems at that level.

    The backups work, we spent some time ensuring this to be correct and all EV servers and all SQL databases relating to EV are backed up successfully every night.

    SQL Maintenance runs once per week.

    The backup system (Tivoli) does not set the Archive Bit - instead we use a Trigger File (IgnoreArchiveBitTrigger.txt/old)

    Storage File Watch changes the .txt to .old when Backup Mode is cycled.

    We use Pre and Post backup scripts to handle the trigger file, these have been written and re-written and tested and re-tested many times. They do as it says in the docs - put in backupmode, remove *.old, write IgnoreArchiveBitTrigger.txt to each Vault Store partition, move out of backup mode.

    I've manually done each task from the script. Set BackupMode - navigate to folder containing the Vault Store partition - delete *.old - create text file called IgnoreArchiveBitTrigger.txt - taken out of Backup Mode and watched as the file extension is changed.

     

    All that works exactly as it should. But I still have a partition that claims to have not been backed up for days, and has tens of thousands of items awaiting backup. As a result items remain pending in users Outlook and they complain about their quotas filling up. I get Event ID 41261 and 41008 telling me that no backup has taken place.

    I found a number of useful articles on this forum, and folowed them. Ran these couple of queries :

    Use EVVaultStoreDB
    Select * from JournalArchive where BackupComplete = '0'

    Use EVVaultStoreDB
    Select * from JournalArchive where IndexCommited = '0'

     

    There were a few items from the Index query and tens of thousands in the backup query. Which fits in with the error, but my undestanding is that the value in the error comes from looking at the BackUpComplete field anyway, so that would be even stranger if they were different.

    So what I know is that : Archiving works, Indexing works, retrieval and searching work, the backups work, the triggerfile is written by script and amended by StorageFileWatch process, SQL doesn't need defragmentation or any other maintenance, SQL permissions have all been checked, this process is fine on 6 other servers - just one has a problem.

    What I cannot understand is what is going wrong for one server/partition. As the file name changes from .txt to .old then I am sure the FileWatch process is running. It seems to me that the missing part is updating SQL to let it know the items it lists as BackupComplete = 0 should be BackupComplete = 1.

    I did, for a moment or two, think about updating BackupComplete to 1, but my cautionary nature/cowardice stops me changing things along those lines.

    I've looked at the Backup tab on the properties of the VaultStore, which is now set to scan every 15 minutes - it takes 12 to run the scan according to Event Viewer. It also tells me that there are no items in other SIS partitions that need securing and the last item it secured was 4 days ago.

     

    So my questions are :

    How do I get these items secured and get the pending icon out of users Outlook?

    As we use the trigger file EV has no idea whatsoever about backups, all it does it look at the Date Modified of the text file and from there it assumes anything before that date is backed up, so in a test environment I have played with this, and sure enough by going into Backup Mode and manually creating the Trigger files, then out of backup mode I see items changed from pending to fully archived - so in testing it works as expected, and indeed it works as expected on 6 production servers - but one of them is changing the txt/old file but then doesn't appear to update SQL. So what am I missing or what might be broken?

     

    Any suggestions or ideas happily received. I've a feeling this is going to be a support case, as I've been through so many articles and all the usual things that people suggest aren't working, but don't let that put you off if you have an idea, I'm completely stumped and would welcome any idea - even if it's just "step away from the machine and sit quietly in a dark room for a while"

     

    Thanks for reading.

     

     

     


     

     

     

  • dcVAST's avatar
    dcVAST
    Level 5
    10 years ago

    I've seen instances in the past where backupcomplete was flipped manually and then execute uspd_journals. What type of storage are you using? Maybe snag a dtrace as you clear backup mode and the triggers are renamed. Do you have Pending Shortcut Timeout set?