cancel
Showing results for 
Search instead for 
Did you mean: 

EV - Vault Partitions Scan for backup success, has stopped

benq
Level 4

Hi,

i have EV 8.0 and get the error reported in EV Admin Console - Status checks: "It is more than 1 days since some partitions were scanned to check for newly-archived items that have not been backed up or replicated."
(as described : http://seer.entsupport.symantec.com/docs/312348.htm )

we have employed the SQL backups as per http://seer.entsupport.symantec.com/docs/322715.htm
we also have the EV server being backed-up over the network via Symantec Backup Exec 12.0 (all drives where EV and associated d/b are installed, as well as the System State and Shadow Copy Components).
these backups are reported as successful from within SQL Server 2005 (locally), and Backup Exec (from remote server)
I believe the archive flag is being reset, as we're not getting any errors regarding 'no backups done since...' etc. in EV

when i observe the stats from within EV Admin Console > Vault Store Groups > Express Vault Store Group > Express Vault Store > Express Vault Store Ptn1 >properties:Backup:
"Last item secured: 20/08/2010
"Last scan started: 20/08/2010
"Unsecured items found in last scan: 187
"Items secured in last scan: 279981

(this is taken today, 23/08/2010)

i manually initiated the scan a few days ago (20/08/2010) by restarting the Storage Service, and this cleared the counters (and the error), this is because the last reported scan was about 10 days earlier

what is causing the scan not to run automatically ? can i set this manually or can i check this setting somewhere from within the EV Admin Console?

thanks
Ben

thanks

1 ACCEPTED SOLUTION

Accepted Solutions

TonySterling
Moderator
Moderator
Partner    VIP    Accredited Certified

You can check to see if you back up job is calling the powershell scripts that put EV in backup mode.

Have you seen this?  http://support.veritas.com/docs/312327

View solution in original post

18 REPLIES 18

benq
Level 4
see screenshots attached

TonySterling
Moderator
Moderator
Partner    VIP    Accredited Certified

Are you using the EV agent for BE? 
Is BE setting Partitions in Backup Mode when you are backing up?

benq
Level 4
Hi Tony,
no EV agent for BE, just 'standard' Backup2Disk-backup via Remote Agent for Windows Systems

how can i check whether BE is setting the partitions to backup mode, during backup? are there logs or do I need to manually log in to the EV Admin Console during time of BE backup, and check the backup mode of the partition then?

thanks
Ben

TonySterling
Moderator
Moderator
Partner    VIP    Accredited Certified

You can check to see if you back up job is calling the powershell scripts that put EV in backup mode.

Have you seen this?  http://support.veritas.com/docs/312327

benq
Level 4
thanks very much Tony.  I've incorporated those scripts into my nightly SQL backup, to put the vault in backup mode and then take it out of backup mode- post successful vault backup.


benq
Level 4
Hi Tony,

wondering if you can assist with what seems to be an associated problem with backups...

i have archiving policy set so that the savesets are "removed after backup"

i notice that there are a number of items in different user mailboxes which remain in the 'This item is being archived by Enterprise Vault..." -state

i've read through some discussions, where you've also contributed eg:
http://aka-community.symantec.com/connect/forums/pending-archive-two-weeks

and associated KB's: http://seer.entsupport.symantec.com/docs/316761.htm &
http://seer.entsupport.symantec.com/docs/286728.htm


and as described, i ran the SQL queries
SELECT COUNT(*) FROM JournalArchive WHERE Indexcommited = '0'
SELECT COUNT(*) FROM JournalArchive WHERE BackupComplete = '0'

the result for Indexcommited = 0
the result for BackupComplete = 1340

[EDIT- i read from http://www.symantec.com/connect/articles/changes-backup-procedures-enterprise-vault-8

When the item is set to after backup, it is watched in two locations in the Vault Store Database.
One in the JournalArchive table with a BackupComplete flag set to 1, and one in a table called Watchfile, which contains a full path to the Archived item.

Every item listed in the watchfile table will be scanned upon Storage Service start or every 12 hours.
The StorageFileWatch process will go through each item listed in the WatchFile table and determine whether the item has been backed up or not. It determines this by looking at whether the archive bit exists on the file or not. If the archive bit has been removed, then Enterprise Vault removes that particular entry from the watchfile table, and then in the journalArchive table, it changes the “BackupComplete” column from a 0 to a 1
]

does the changing of the BackupMode of the Vault Stores (as you pointed out in your previous post, and now i've implented) NOT change the BackupComplete flag from 0 back to 1?

what process would change this ?

[EDIT 2 - i manually restarted the Vault Storage Service, which then ran the archive process and created full shortcuts for about 900 of the 1340 items. [i presume the remainder of items were due to archiving running -after- the last backup

Question, when is the 12hr cycle started?  "...The Storage service performs the check when it starts and then every 12 hours."
ie do i read this as 12hrs of the last time the Storage Service was restarted?

so imlpementing a backup strategy, with this in mind, could be as follows:
01:00am - SQL Backup
02:00am - EV server backup via remote BUEXEC file-to-disk
03:00am - (providing backup @2:00am finished) - have Storage Service run.  this cuold be done by restarting the Storage Service at 3pm the day before??
03:30am - schedule EV archive of mailboxes to run

thanks
Ben

TonySterling
Moderator
Moderator
Partner    VIP    Accredited Certified

you should backup AFTER your archive run.  Doing it the way you suggest means you will have pending items until the next back up run the following night.

so the flow is this:

Archive Maiboxes
Put EV in backup mode
Backup EV and SQL
remove EV from backup mode

benq
Level 4
thanks for the clarification Tony.
i've put that in place.  will check it out over the next couple of days

can you confirm whether, when the storage crawler service runs, does it complete processing of any archived items that were hanging in the "This item is being archived by Enterprise Vault..." -state, into EV shotcuts

ie is the trigger for completing archive of items, the storage crawler service (at every 12 hr interval)
or, is the trigger to archive immediately when the vault store/partition is brought out of backup mode

eg in this scenario...

23:00 - Archive Maiboxes
02:00 - Put EV in backup mode
03:00 - Backup EV and SQL
04:00 - remove EV from backup mode
05:00 - Storage Service 12hr interval runs
...
08:00 -[users manually runs EV archive on items in Outlook]
17:00 - storage crawler 12hr interval ticks over and runs the service
> At this point, are the 08:00am archived files processed from 'pending' state to 'archived'. 
> Or are they only archived at 04:00am the next morning when the vault store is brought out of backup mode?

TonySterling
Moderator
Moderator
Partner    VIP    Accredited Certified

They will only be switched from pending to shortcuts when they vault store partition is backed up at night.

benq
Level 4
Hi Tony,

i checked this morning after yesterday's changes and it appears the archiving isnt completing the shortcut creation

as shown in the log table,
>the archiving job ran against both my Exchange servers (as shown in the table below at times 22:00:25 and 23:00:15)
>the backup mode is then set at 02:00:05,
>then various databases are backed up between 02:00:05 and 02:00:18 (we only have a handful of trial users at the moment, hence the small backup times). 
>the backup mode is then cleared at 02:00:20.

this indicates that the correct running order completed successfully overnight.

So I run this SQL query, with the following results:
SELECT COUNT(*) FROM JournalArchive WHERE Indexcommited = '0'     (result=0)
SELECT COUNT(*) FROM JournalArchive WHERE BackupComplete = '0'    (result=1189)

to me, this indicates that the 'new' msgs that were vaulted during last night's run have not been marked as being backed up.  and in practice, i checked one of the user's mailboxes that we had manually archived items yesterday and it's still in the 'this item is being archived byu Enterprise Vault..." pre-processing state

can you help explain?


08/26/2010 02:00:20 Enterprise Vault  Successfully cleared Backup Mode from all the Vault Stores that are managed by the Storage Service on this computer.
08/26/2010 02:00:20 Enterprise Vault  StorageDelete threads restarted successfully.
08/26/2010 02:00:20 Enterprise Vault  Request thread pool (InstanceId = [1]) stopped. Total threads stopped = [8].
08/26/2010 02:00:18 Backup Database backed up. Database: EnterpriseVaultDirectory
08/26/2010 02:00:18 Backup Database backed up. Database: EnterpriseVaultFSAReporting
08/26/2010 02:00:17 Backup Database backed up. Database: EVVSGExpressVaultStoreGroup_1_1
08/26/2010 02:00:14 Backup Database backed up. Database: EnterpriseVaultMonitoring
08/26/2010 02:00:13 Backup Database backed up. Database: EVVSExpressVaultStore_1
08/26/2010 02:00:05 Backup Database backed up. Database: EVVSITFilesVaultStore_2
08/26/2010 02:00:05 Enterprise Vault  Request thread pool (InstanceId = [1]) started with maximum thread count = [8].
08/26/2010 02:00:05 Enterprise Vault  Successfully set Backup Mode on all the Vault Stores that are managed by the Storage Service on this computer.
08/26/2010 02:00:05 Enterprise Vault  StorageDelete threads restarted successfully.
08/26/2010 02:00:05 Enterprise Vault  Request thread pool (InstanceId = [1]) stopped. Total threads stopped = [8].
08/26/2010 00:00:13 Enterprise Vault  The Exchange mailbox provisioning task has completed.
08/26/2010 00:00:11 Enterprise Vault  The Exchange mailbox provisioning task has started.
08/26/2010 00:00:11 Enterprise Vault  Completed a synchronization run for task: File System Archiving Task_Synchronization.
08/26/2010 00:00:11 Enterprise Vault  Starting a synchronization run for task: File System Archiving Task_Synchronization.
08/26/2010 00:00:05 Enterprise Vault  Starting synchronization run of all mailboxes on Exchange server MELEXC01
08/26/2010 00:00:04 Enterprise Vault  Starting synchronization run of all mailboxes on Exchange server SYDEXC01
08/25/2010 23:00:15 Enterprise Vault  Start to process the Exchange system MELEXC01.
08/25/2010 22:00:25 Enterprise Vault  Start to process the Exchange system SYDEXC01.
 

TonySterling
Moderator
Moderator
Partner    VIP    Accredited Certified

You say the databases are backed up, what about the vault store partition where the dvs files are?  Those files need to be backed up and their archive bit removed unless you are using a trigger file.

benq
Level 4
Hi Tony,
the backup is via BE 12.5 'backup-to-disk' from a network server, using the backup method: "incremental  - using archive bit (reset archive bit)".  this backup includes all of c: drive (there is only this c: drive partition on the EV server), which includes the dir where the Vault store dvs files are kept.

i've checked the backups have been successful for the past 10days and beyond....

any ideas?

cheers
Ben

benq
Level 4

you can see in the two pics below, in BE the two DVS files from last night (08-25)  have been backed-up (reset archive bit)

the third pic shows the archive bit having been removed for that file on the EV server

is it the Storage Service that then runs every 12hrs (from the last time the service was restarted) that will then 'detect' that these DVS files have been backed up and then should finalise the processing into shortcuts?
if so, it seems that it's failing to run at it's allotted 12hr cycle

benq
Level 4
...gave it a day and it seems to be sorting itself out.  the majority of items in pending state have been converted to shortcuts.
however there are still about a few hundred items this morning that havent been converted to shortcuts, why?- i cant understand.  Based on the archive& backup flow, there should be no outstanding items awaiting in the pending state when checking first thing in the morning (before users have chance to manually archive items).

ben_quach
Level 2

just revisiting this thread...

i believe i've sorted out the order, just wanted to see if someone can confirm; and if confirmed, wanted to question the logic...

Based on Tony's reply posts above (24 & 25 Aug 2010), this then is my process flow:

10:00pm -   Archive Maiboxes
02:00:05am -   Put EV in backup mode
02:00:08 - 02:00:21am -   Backup EV and SQL (.BAK copied locally to EV server)
02:00:23am -   remove EV from backup mode
03:00am -   Backup of entire EV server -including- directory where vault store dvs files are,a nd SQl .BAK backup sets.  Reset archive bit during backup.

what im finding however is that the Last Scan Started (checking in Vault Store Groups > Express Vault Store group > Express Vault Store > properties / Backup-tab) is at 2:00:23AM.  Im assuming this corresponds to the timestamp at 02:00:23am when backup mode is removed from EV... thus the scan takes place before the backup is done for vault stores at 03:00am.

my issue is this...

i set the EV server backup at 03:00am after the SQL backups are done, to hae a copy of the most recent d/b backup also.  I'm finding that because of this timing the scan that runs at 2am does not recognise that the vault stores have been backed up, hence does not secure the mail items (until eventually the next day's scheduled run)

if i swap the schedules of the 02:00am SQL backup and 03:00am EV server backup, then that means the daily backup fo the EV Server will be without hte latest SQl .BAK backup set.

what can i put in place to ensure all mail items are secured nightly?  i can only think of adding another job to trigger the putting and removing of backup mode at say 4:00am, which will then force another scan and secure items.  is this feasible?

thanks for your time.

TonySterling
Moderator
Moderator
Partner    VIP    Accredited Certified

Look again at what I wrote:

Archive Maiboxes
Put EV in backup mode
Backup EV and SQL
remove EV from backup mode

 

Remove from backup mode should be the LAST thing you do:

 

So your schedule should look more like this:

10:00pm -   Archive Maiboxes
02:00:05am -   Put EV in backup mode
02:00:08 - 02:00:21am -   Backup EV and SQL (.BAK copied locally to EV server)

03:00am -   Backup of entire EV server -including- directory where vault store dvs files are,a nd SQl .BAK backup sets.  Reset archive bit during backup

*****am -   remove EV from backup mode

ben_quach
Level 2

it's so obvious, it's scary.

thanks Tony.

TonySterling
Moderator
Moderator
Partner    VIP    Accredited Certified

No worries, happens to us all. 

If you think you got your answer mark a post as the solution please. :)