Highlighted

Items that have been deleted from Vault Store but have not yet been deleted from the indexes

Hello together, 

We get Event ID: 41022

There are XXXXX items that have been deleted from Vault Store 'VaultStore01' but have not yet been deleted from the indexes.

There may be a problem with Enterprise Vault indexing.

 

This VaultStore is only for Journaling and it contains one journal archive. We had some problems with the storage queue a few weeks ago and we get these messages since then.

The number of items mentioned does not change. I've taken a look at the journaldelete table. All the entrys have DeletionReason state 4 which isn't mentioned under http://www.symantec.com/docs/HOWTO42225

Synchronizing the Index Volumes for this archive didn't help. We haven't done a rebuild because these archive currently contains about 6 million items. 

 

Enterprise Vault 11.0.1 CHF1 on Windows Server 2008 R2. 

Does anyone know what else we could try? 

 

 

PMCS GmbH & Co. KG - A Serviceware Company
www.serviceware.de
1 Solution

Accepted Solutions
Accepted Solution!

Just to add to the

Just to add to the discussion, deletion reason 4 is 'System Deleted', so at this point I would say that you assumption is correct Marcde09.

I would hazard a guess and say that the IDPartition/PartitionID of those items in the JournalDelete table is '-1'?

 

If the values are -1 for IDPartition/PartitionID (cant remember the exact phrasing right now) it means that you are 100% correct, it was never stored to a partition and EV backed out of the operation, turned the items to normal items again then moved the DB entries to the JournalDelete table.

 

There is a problem with a particular version of EV where if the value of the IDPartition column is '-1' they stay in that table, will need to wait until I am back in the office tomorrow to see if 11.0.1 CHF1 is one of them.

Items in the JournalDelete table are cleared ONLY if the item meets certain criteria, if they do not meet ALL of those criteria they will stay in that table. So they can stay in the table for a variety of reasons, it may well be worth getting a case opened with us so that we can work our way through them with you and ensure the items are cleared out of that table correctly.

 

View solution in original post

14 Replies

what was the storage queue

what was the storage queue issue?

are your indexes 32 or 64bit or mix?

do you know what deleted the items from EV?

Hi Marc, How about creating a

Hi Marc,

How about creating a new Journal Archive, start using that, then sync the old indexes?

That might cause some disruption in your E-discovery searches, but would not affect new items being journaled.

 

Regards. Gertjan

Hello AndrewB,  as my first

Hello AndrewB, 

as my first reply didn't get published, here is my answer: 

it started with elements stuck in pending state in the journaling mailbox. After restarting the services the task was working as expected but the "old" elements still were stuck in pending state. 

So we analyzed the EV eventlog and saw that we got access denied messages to the storage queue location. 

That was the night we first got event 41022.

The following events were shown:

"Event 29010 A problem was encountered while processing a saveset in the Storage queue.

Details: The Vault Service account does not have access to the storage queue location. "

"Event 29060 Could not process an item from the Storage Queue and there was a problem copying the item to the failed item folder."

"Event 29059 Could not process an item from the Storage Queue because the queue file does not exist. "

All these events mentioned the same transaction id. We found out that the id mentioned under the first events was the one of the oldest pending item in the journaling mailbox. 

We checked the location of the dvs file and it doesn't exist (anymore?!)

After a bit of investigation we found out that a backup setting of the enterprise vault server may be the root cause of our problem. After changing the settings the problems didn't reoccur. 

We finally changed the pending shortcut timeout to 0 to get rid of the stuck items in pending state. 

 

All Indexes are 64bit. 

I think Enterprise Vault deleted the items. I can imagine the following:

1) Enterprise Vault pre processed the elements

2) The elements got placed in the storage queue location and following on the storage

3) The elements got indexed 

4) Our backup problem occured 

5) Enterprise Vault tries to post process the element / the storage queue location 

6) Enterpise Vault is not able to process and abandoned the element 

7) Element gets deleted from storage and has to be deleted from index. 

 

Please correct me if I am completely wrong. 

 

 

PMCS GmbH & Co. KG - A Serviceware Company
www.serviceware.de

Hi Marc, I ment sync, which

Hi Marc,

I ment sync, which is the initial try to get indexes in sync. You might (if you have a new archive) perform a rebuild anyway.

You might have a look too here: http://www.symantec.com/docs/HOWTO58947

That has topics on Indexing (for EV10, but applicable to EV11 too) That might help in assisting.

 

As sidenote, can you check this forumpost of mine:

https://www-secure.symantec.com/connect/forums/ev11-indexing-running-behind-check

If you have that offending MS patch applied, it might be worth a try to install the update update.

Regards. Gertjan

Hello Andy and Gertjan,  this

Hello Andy and Gertjan, 

this morning i wrote an comprehensive reply to AndrewB. It said that my reply has to get veriefied. That was around about 3 h ago. Right now I cannot see my reply and it's status. Do you know why? 

 

@Gertjan, we haven't thought about it until now but we already synchronized the volumes without success - Did you mean rebuild? 

 

Edit:

interestingly I was able to post a second reply to the post of AndrewB

PMCS GmbH & Co. KG - A Serviceware Company
www.serviceware.de

Hi Marc, You write: 4) Our

Hi Marc,

You write:

4) Our backup problem occured 

5) Enterprise Vault tries to post process the element / the storage queue location 

6) Enterpise Vault is not able to process and abandoned the element 

7) Element gets deleted from storage and has to be deleted from index. 

for 6/7 - the archived items will sit in the storage-queue. they will only be removed after EV SUCCESFULLY stored the items in the archive.

If EV would not do delete items if it cannot succesfully archive them. That means you risk dataloss, which then makes Journal Archiving useless. It might be worth opening a support call, to have them verify what is wrong.

 

Regards. Gertjan

Hi Gertjan,  I think I didn't

Hi Gertjan, 

I think I didn't explain it correctly. You are right. So for our situation it means that probably the elements never got stored successfully to the archive (thats why they still were in the storage queue location)

I understand what you mean but we were able to verfiy that the elements (Which transaction IDs were mentioned in the above events) were stuck in pending state. I think that we are fine here. 

We solved this but trying to find a way to get around event 41022.

I will try to check the information mentioned in your second post. Will give some feedback. 

PMCS GmbH & Co. KG - A Serviceware Company
www.serviceware.de

because external elements

because external elements were involved which interfered with the archiving process it might be worth running EVSVR on that dataset to get a deeper idea of the missing elements.

Accepted Solution!

Just to add to the

Just to add to the discussion, deletion reason 4 is 'System Deleted', so at this point I would say that you assumption is correct Marcde09.

I would hazard a guess and say that the IDPartition/PartitionID of those items in the JournalDelete table is '-1'?

 

If the values are -1 for IDPartition/PartitionID (cant remember the exact phrasing right now) it means that you are 100% correct, it was never stored to a partition and EV backed out of the operation, turned the items to normal items again then moved the DB entries to the JournalDelete table.

 

There is a problem with a particular version of EV where if the value of the IDPartition column is '-1' they stay in that table, will need to wait until I am back in the office tomorrow to see if 11.0.1 CHF1 is one of them.

Items in the JournalDelete table are cleared ONLY if the item meets certain criteria, if they do not meet ALL of those criteria they will stay in that table. So they can stay in the table for a variety of reasons, it may well be worth getting a case opened with us so that we can work our way through them with you and ensure the items are cleared out of that table correctly.

 

View solution in original post

Hello guys, please excuse my

Hello guys,

please excuse my late response. Ben I think you are right. The column idpartition for all of these entrys shows a -1.

Have you been able to check if this enterprise vault version is affected?

When I open a case. Could you work on this? 

 

Thanks 

Marc 

PMCS GmbH & Co. KG - A Serviceware Company
www.serviceware.de

Hi Marc (?), Yes that

Hi Marc (?),

 

Yes that shouldn't be a problem at all, if you log it and ask for it to be assigned to me, or log it via the Portal and put a Case Note on it asking for the same, it should make its way to me.

If you also let me know the case ref here I will go grab it if I haven't already been made aware of it.

 

Cheers

Ben

 

Hi Ben,  I've created a case

Hi Ben, 

I've created a case for this. It's case 09509330. 

 

Thanks

Marc 

PMCS GmbH & Co. KG - A Serviceware Company
www.serviceware.de

Re: Items that have been deleted from Vault Store but have not yet been deleted from the indexes

Hi, I believe this is the article you need for this issue. https://www.veritas.com/support/en_US/article.TECH227713

Re: Items that have been deleted from Vault Store but have not yet been deleted from the indexes

Hello all, 

together with Veritas we confirmed that all these entrys had DeletionReason = 4 (System Delete) and IdPartition = -1 (from my understanding it means that the item has never been stored on the storage). 

To make this sure we ran a sql query to compare the entrys from the journaldelete table with those of the saveset table. 

 

At leat we deleted these entrys from the table.

I will mark the comment from Ben as solution. 

 

 

PMCS GmbH & Co. KG - A Serviceware Company
www.serviceware.de