Files are being collected but not migrated

[SSR1]

Hello.

I have a problem with EV 10.0.3 and error 41008. Files are being collected but not migrated to Secondary Storage. New items awaiting backup or replication.

We have backup of vault partition on BackUp Exec 2010 R3, and we are using BackUp Exec migrator.

On Vault Store Properties is set remove safety coppies immediately after archive.

SQL Database is backuped daily by ArcServe BackUp.

Could you tell me what we are doing wrong because we have lot of savesets that have not yet been backed up or replicated for Vault Store.

When I open WatchFile table I see that all items have ItemSecured set to False not to True.

DTrace log is in attachment.

[Rob_Wilcox1]

What is the dtrace of?  (I mean I see a storage restart somewhere in there... amongst other things)

Why are there three DTRACEs?

[SSR1]

Three files for three trace options: storageServer, storageFileWatch, StorageCrawler. Here is the tracelog for storagefilewatch only in the attachment. Log is divided into 3 parts,  because of 1 log file can only be 100mb.

[SSR1]

Has anyone such problem? I have no way to solve it.

[Rob_Wilcox1]

You need to run a DTRACE of the collection/migration process.  I didn't see any start of the collection process in the traces you provided.. 

My question was really relating to what are you DTRACEing, not in terms of which components, but what operation did you perform afterwards?  I also would expect ONE DTRACE.. not 3.

Does your partition use a trigger file, or the archive bit?  Is ArcServe successfully handling whichever option you have chosen?

[SSR1]

Partions use the archive bit. ArcServe is succesfully handling whichever option I have chosen. Now I'm running DTRACE of the collection issue. I didn't find collection\migration process in dtrace options. There are only collection issue and migration issue separately.

[Rob_Wilcox1]

<snip>

Partions use the archive bit. ArcServe is succesfully handling whichever option I have chosen

</snip>

How are you verifying that?

Do you have system status warning messages about the number of items awaitng backup? Do you have EVOM where you can see the number of items awaiting back, or even the simple Vault Store Group reports show the information, I think.

[SSR1]

Yes I have messages in EV about items awaiting backup or replication and that there are some number savesets that have not yet been backed up or replicated for VaultStore. And such message for each partition.

[Rob_Wilcox1]

Then ArcServe is not correctly processing the backup.

Either it's not resetting the archive bit, or can't.

If it can't then you'd need to use one of the trigger file mechanisms.  If it can, but isn't, you'd need to investigate the ArcServe side of things to see why that is.

Once that's fixed...  the collections, and migration should then be fine. (I think)

[SSR1]

And one thing, I cannot see any tasks of migration in Back Up Exec, where backup exec agent migration  is set to back up to tape. ArcServe is backs up only SQL database of EV and shrink logs. 

[Rob_Wilcox1 · Accepted solution]

Oh okay, didn't see that in the original post about using ArcServe for part of it, and Backup Exec for part of it.

So the general process is:

* Set backup mode on vault stores

* Set backup mode on index locations

* Do SQL backups

* Do backups of index locations and vault store partitions (*1)

* Clear backup mode on index locations

* Clear backup mode on Vault Stores

*1 -> Some people only backup active, open, partitions, and closed partitions at different frequencies/schedules.

So.. the bit about the number of items awaiting backup - if you're seeing that (and it sounds like you are) comes from the backup of vault store partitions.  If you're using the Archive Bit, then perhaps that's not being cleared by the backup software.

[SSR1]

Rob, will collection migrate into the second storage after backup or it wil remain on local disk? Just I have not enough  local disk space, where EV is.

[Rob_Wilcox1]

It will migrate after backup... depending on your collection/migration policy settings on the properties of the Vault Store Partition(s).

[SSR1]

ok thanks

[SSR1]

Rob I made backup with clear archive bit, but after backup migration didn't begin. Backup ended with exceptions: the Enterprise Vault services cannot create the PartitionSecuredNotification.xml trigger file for the backup of an Enterprise Vault Agent partition. Ensure that the credentials have the correct privileges and that there is sufficient free space on the partition.

I made fix with this article http://www.symantec.com/business/support/index?page=content&id=TECH127866 

but result was the same - migration didn't begin and the same error about creation of the PartitionSecuredNotification.xml trigger file

[Rob_Wilcox1]

Okay so backups now work correctly?

But nothing is still migrated?

Have you got a screenshot of your migration tab on the Vault Store partition?

[SSR1]

yeah backups work correctly, but nothing is still migrated. Screeshot is in the attachment

[Rob_Wilcox1]

So the next question then is whether you're getting CAB files created?

IF you're not .. then do a run now on the collector whilst DTRACE'ing StorageFileWatch.  You will see storagefilewatch processing each of the files and folders on your partition, and you'll see reasons why the CAB files are not being created.

IF you are getting CABs, and the issue now is that they're not being sent to your migrator you need to a DTRACE of StorageFileWatch

eg for the migration you should see:

413757     12:43:40.421     [3544]     (StorageFileWatch)     <5612>     EV:M     The Migrator (EV) for Partition 17571A7774EEE064785F636FAA3E1FA151q10000ev1a.EV.Local is about to attempt to Migrate (copy) file C:\Enterprise Vault Stores\vs1 Ptn1\2012\03-30\9\Collection1.CAB to \\W2K3SRV\remote\7571A7774EEE064785F636FAA3E1FA15\2012\03-30\9\Collection1.CAB
413763     12:43:40.827     [3544]     (StorageFileWatch)     <5612>     EV:M     The Collector for Partition 17571A7774EEE064785F636FAA3E1FA151q10000ev1a.EV.Local successfully migrated (copied) and verified file C:\Enterprise Vault Stores\vs1 Ptn1\2012\03-30\9\Collection1.CAB

[SSR1]

Rob can't see CAB files, only DVS. Tracelogs in the attachments. There are log divided into  4 parts by up to 100 MB.

[Rob_Wilcox1]

Okay you're not capturing this issue correctly from what I can tell.

When the collector starts you should see:

413606 12:43:40.219  [3544] (StorageFileWatch) <5612> EV:M Starting a Collector/Migrator run for Partition 17571A7774EEE064785F636FAA3E1FA151q10000ev1a.EV.Local

413607 12:43:40.265  [3544] (StorageFileWatch) <5612> EV~I Event ID: 6976 A Partition's Collector/Migrator run has started. |Partition Root Path: C:\Enterprise Vault Stores\vs1 Ptn1 |

413608 12:43:40.265  [3544] (StorageFileWatch) <5612> EV:M Collector is compacting sparse Collections for Partition 17571A7774EEE064785F636FAA3E1FA151q10000ev1a.EV.Local

...

413618 12:43:40.281  [3544] (StorageFileWatch) <5612> EV:M Collector is walking the directory tree for Partition 17571A7774EEE064785F636FAA3E1FA151q10000ev1a.EV.Local

That's not in any of your log files.

When the collector finishes you should see:

486813 12:44:29.327  [3544] (StorageFileWatch) <5612> EV:M Finished the Collector/Migrator run for Partition 17571A7774EEE064785F636FAA3E1FA151q10000ev1a.EV.Local

486814 12:44:29.327  [3544] (StorageFileWatch) <5612> EV~I Event ID: 7134 A Partition's Collector/Migrator run has finished. |Some files were not collected because they have not been backed up. |Partition Root Path: C:\Enterprise Vault Stores\vs1 Ptn1 |Archived items: 0 |Files added to collections: 0 |Collections: 0 |Files migrated to secondary storage: 69 |Time taken for the run [H:M:S]: 0:00:49 |

That appears in two of your files:

1236990 16:59:41.943  [5588] (StorageFileWatch) <576> EV:M Finished the Collector/Migrator run for Partition 13313B05877F0504284AF29633C4A04141q10000vault.msk.gubbank.ru

1236991 16:59:41.976  [5588] (StorageFileWatch) <576> EV~I Event ID: 6917 A Partition's Collector/Migrator run has finished. |Partition Root Path: D:\EVStores\UserBoxChel Ptn 16 |Archived items: 0 |Files added to collections: 0 |Collections: 0 |Files migrated to secondary storage: 0 |Time taken for the run [H:M:S]: 0:06:57 |

And

2728329 17:00:52.412  [5588] (StorageFileWatch) <6316> EV:M Finished the Collector/Migrator run for Partition 16ED4CD6C42F695449105884249C4E6701q10000vault.msk.gubbank.ru

But there is an awful lot of:

---- 17:00:54.200  [11708] (DTrace) <15036> EV~X ***** Trace buffer overflow. Trace entries lost: 2728323 to 2728325 *****

Are running this trace, and having it running on the screen as well?