Pending items - Changed the status of remove safety copies

[r3v0ke]

I have two doubts, here in my environment ALL the archived items are pending, the storage keeps growing, since the original items are not being deleted. At first the problem was (I guess) that the properties to "Remove safety copies" were at "After backup" - and I wasn't backing up anything at all

My backup solution (NetBackup) is temporarily disabled, so to test Enterprise Vault 10.0.3, I changed the status of the "Remove safety copies" to "Immediately after archive". But there was no effect, the items keep being archived in my vaults, but they are not being deleted at all.

After read this KB http://www.symantec.com/business/support/index?page=content&id=TECH66202 I noticed the following:

Important Note:  When making the decision to alter the method of removing safety copies from "After backup" to "Immediately after archive" it is necessary to disable the schedule for the Task (Mailbox Archiving) or set the Task for Report Mode (Journal Archiving). This will confirm that additional items are not added to these tables.  Once both of these tables are finished processing all outstanding items, it is then safe to change this functionality. If this option is changed with outstanding items in the WatchFile table, items associated with these entries will not complete the Post Process and be locked in a Pending State.  This can also cause items from being able to be post processed and pending items to build up.

Ok so I have more thousands and thousand of archived items that ARE LOCKED ON PENDING STATE ? How to solve that ?

My other question is what exactly I need to backup up to archive the items correctly. I know that I need to backup up the tables, and already configured that in NetBackup, just need to run the backup. But this will solve my problem ? Backup just the SQL tables ? Or I need to backup the Vault Store Partion ?

Thanks in advance

[JesusWept3]

OK So when an item is archived it goes to two tables in the vault store database
JournalArchive and WatchFile

JournalArchive will list each item and has a column for BackupComplete and IndexCommitted
WatchFile lists the exact location of each DVS file awaiting backup

When Storage starts or comes in and out of backup mode, it lists all items in the JournalArchive table that have backup complete 0, it then links it to the watchfile table

StorageFileWatch process then goes and examines each of those items listed in the watchfile to determine whether the Archive Bit has been removed, if the archive bit no longer exists, then the item is considered backed up, the record is then deleted from WatchFile table and BackupComplete is set to 1 in the JournalArchive table.

If you are on storage such as CIFS or NetApp that do not support Archive Bits, you can use the Secured Notification file or the IgnoreArchiveBitTrigger.txt file, that sits in the root of the Vault Store Partition, StorageFileWatch looks at the *CREATED* Date of the file, and any files older than the IgnoreArchiveBitTrigger.txt are counted as being backed up

Also depending on how your vault store sharing is set up, you may find that a shared part of an email in another vault store partition hasn't been backed up, and that is why its saying the email hasn't been backed up.

Note that changing Immediately After Archive and After Backup are for items going forward, they do not work retroactively on items already in the watchfile table

You have two choices:
1. Set the Vault Store Partitions to use a trigger file and place in the IgnoreArchiveBitTrigger.txt
2. Cancel Pending all the items, switch the vault store to immediately after archive, rearchive the items (This will not cause duplciates)

You can read more about the trigger file here:
http://www.symantec.com/business/support/index?page=content&id=TECH35610

Also i wrote an article about changes to backup procedures in EV8 that covers this in more depth:
https://www-secure.symantec.com/connect/articles/changes-backup-procedures-enterprise-vault-8

 

[r3v0ke]

Thanks for the fast reply JesusWept3

Ok I did the following:

1. Set the Vault Store partitions to use a trigger file and created the IgnoreArchiveBitTrigger.txt on each Vault Partion folder root

2. Cancel Pending all the items using the following method:

- Open the VAC and browse to the Site,Policies, Exchange, Mailbox section

- Right click on the Mailbox Policy and select the Advanced Tab

- Ensure you are viewing the Archiving General settings and change the Pending Shortcut Timeout value to 0.

- Select 'Ok' to save the policy changes and then run a Synchronization job on the appropriate Mailbox Archiving task(s) to which the policy applied

- Open the Vault Administration Console (VAC)

- Right click to select Run Now on the appropriate Archiving Service/Task

- Select Report mode and the mailbox or mailboxes against which you wish to run the PendingShortcutTimeout functionality.

3. Restarted the Storage Service.

What I need to do now ? Guess I should run some task or something like that...

[JesusWept3]

Sorry i should have been clearer about that, that should have been an either/or situation, not both combined
Simply changing the backup strategy to immediately after archive, telling it to look for the trigger file, creating the file and then restarting storage should have been enough

Performing a cancel pending will reset all those items from Pending shortcuts to regular emails again, you will have to let archiving go back out and archive those items again, turning it in to a pending item, and then it will be post processed and turned in to a shortcut.

In a production environment though, you may want to consider making sure netbackup is correctly configured to remove the archive bit and not doing immediately after archive as that does introduce the possiblility of data loss.

So for instance if you have a succesful backup at 1am, and then 9am users come in, they then archive a bunch of items and your EV Storage then goes down, you can succesfully restore items from the 1am backup, but now you have shortcuts in peoples mailboxes that don't exist on the storage anymore.

You would have to pull from an exchange backup from the night before to get those items back.
And thats only if the user didn't archive something they received that morning, so if it was received at 8am and archived at the same time, then ev storage went down, you wouldn't have a record of it on EV backups, nor would you have it on exchange backups and your only hope would be that the item exists in the Exchange Transaction logs


 

[r3v0ke]

In Addition, when NetBackup is back I plan to use "Use archive attribute", create a task to create the txt file while I could use my NetBackup have no point....

Here is what I'm backing up with NetBackup, today I added more tables, but do I need to add the the partitions folders as well ?

[r3v0ke]

Ok, so lets clear this and possibly mark as a solved problem.

If I run the task for Archive and Shortcut in the Exchange Server that I want, with the trigger file and all that things that we commented earlier ok, the space occupied on Exchange will fall ? My items will be archived ? Even those items that were pending will be deleted in the Exchange ? Not only the new ones ?

[JesusWept3 · Accepted solution]

OK So ....

1. If you're backup strategy on the vault store is set to "Immediately After Archive" then you don't need to use the trigger file to make items to go from pending to shortcuts immediately.

2. If you did a cancel pending on all the shortcuts, they will have reverted to regular email, you just have to do a Run Now against the mailbox, archive only is fine, you don't need to shortcut processing. Once those items have been archived they will turn in to a shortcut immediately

3. If you are using the trigger file and your vault store is set to After Backup, then you would perform your archiving run, create the trigger file and then either restart storage or go in and out of backup mode

4. Your exchange databases will not automatically shrink, most space saved in Exchange will remain as white space, you will have to do maintainance, offline maintanance will reclaim the most space... that being said, Exchange will just re-use the white space that exists, so shrinking the EDB's isn't necessarily a good thing as you will save stress on your disks by not having to keep growing the EDB files

Also if you are using Exchange 2010, make sure you are on the latest service packs and hotfixes that the EV compatibility supports for your version of EV, as when Exchange 2010 first came out, there were a lot of issues around space not being freed up by exchange and there was a lot of back and forth between symc and msft about why exactly this was.

I believe those issues have all been fixed.
At the very least you should see the majority of the mailboxes reported sizes drop.

But bare in mind, the space savings you will commonly see will be from items with attachments, a simple message saying "Hello" will be a bigger shortcut than the message originally was (since a lot of EV attributes are added, you may also have a banner saying the message was archived etc)

And lastly as far as space savings go, typically the biggest folders in most peoples mailboxes are the calendars, make sure to look at the policies to see if stripping the attachments and replacing them with links will work better if you haven't looked at or considered this already


 

[r3v0ke]

Ok, so If I check the disk properties on Exchange I won't see any difference ?

How do I know that Enterprise Vault is really working ?

Here is a printscreen of the environment, is that right ? (ps.: Most of the critical status are backups)

Could you please answer my question about NetBackup too ? That one with the printscreen

[JesusWept3]

How can you tell if EV is really working? Well you can use the Usage.asp reports to show how many items are being archived and how big the archive is, so if you see you have a 1.4GB archive but the mailbox is 200MB, then you can safely assume that EV has saved you 900MB from exchange. But the white space and maintanance would be an issue with or without Enterprise Vault, as a deletion from the users mailbox will not immediately free up any space. As for the screenshot, the immediately after archive is ok as long as you accept the risks associated with it, the trigger file is ok if your backup software can't remove the archive bit from the files or you are on a file system that does not support the archive bit attribute

[TonySterling]

For your NBU questions, if you haven't added the vault store partitions you will need to add them.  Unless you are using the NBU EV Agent, which it doesn't seem like you are.

[r3v0ke]

Thanks for the replies guys

TonySterling: How can I add de NBU EV Agent to the EV server ? I thought that the usual Agent was the right one. Besides the databases, I must backup the Vault Partitions to EV understand that I'm backing up everything or just the tables ?

JesusWept3: Sorry but I didn't understand you calculus... where I can see this report that you are saying ? You mean the Mail Archiving task report ?

[JesusWept3]

Http://yourEVServer/EnterpriseVault/usage.asp

[TonySterling]

I believe you would have to purchase the EV Agent and install it.  You need to back up the databases, vault store partitions, and Indexes locations.

[r3v0ke]

TonySterling: Got it! Thanks!

JesusWept3: Lets try an example

Here is one line of the report that you said:

Total Item / Total Size(MB)

20,494

2,866

This is total of the archive of this user right ? If I look on his mailbox, what kind of impact I must see ? I mean, not all items are archived, since the policy just stores mails with one year or older - so it is an incomplete view...

The impact is on the mailboxes, is that right ?

[r3v0ke]

I re-ran all the checks and I still got this message:

It is more than 2 days since backup or replication was detected for one or more open partitions.

Open partitions with unsecured items scanned: 1
Total open partitions: 2

List of partitions:
    Mail Archiving Ptn4
 
 
 Items in mailboxes will remain in an archive-pending state.
 
 

[JesusWept3]

What kind of impact will you see? Depends on what kind of items they are, whether they had attachments or not, how the shortcuts are set, are you limiting it to a number of characters, are you putting in the banner? are you telling it to leave the shortcut intact etc So in the situation you are in right now, if you have 20,000 items archived totally 2.8GB but the items remain pending, then most likely the mailbox won't be any smaller until the post processing is done and they're turned from pending items in to shortcuts. But once that does happen, you should hopefully see a major shift. I believe the average size of a shortcut is about 4kb in size 1. You have 20,494 Items 2. Your archive is 2,866MB (2,934,784KB) in Size 3. Average Item Size is 143.20KB (2,934,784 / 20,494) 3. Average Shortcut size is 4KB 4. Shortcuts should take up 80.05MB (81,976KB) 5. Space savings should be 2,785.94MB (2,852,808KB)

[r3v0ke]

Got it! Thanks!

Any suggestions for the problem of the items still being pending ?

It is more than 2 days since backup or replication was detected for one or more open partitions.

Open partitions with unsecured items scanned: 1
Total open partitions: 2

List of partitions:
    Mail Archiving Ptn4
 
 
 Items in mailboxes will remain in an archive-pending state.

[GertjanA]

Perhaps I missed it, but are you setting/clearing backup mode on the stores? Are you using the powershell commands to do so? Are you perhaps setting the backupmode on the site-level? Can you try to set it on VaultStore level? Are you using the triggerfile, or is your backup software able to change the attribute?

If using triggerfile, are you writing a NEW one after backup, then clearing backup mode?

Thanks.