cancel
Showing results for 
Search instead for 
Did you mean: 

41022 Deleted items waiting to be deleted from indexes; EV 10.0.1

MikeAtGamestop
Level 3

For several weeks now, we've been getting a 41022 alert in the status console, saying that, "There are 31394 items that have been deleted from the Vault Store 'EVMBXVS01' but have not yet been deleted from the indexes."

No stores are in backup mode, the database compatibility level is correct, the storage service is running, and there are no failed index volumes. A review of the event log returned no indexing errors or failures of any type. In fact, the only index events I see are the normal start/stop/summary events.

That item count, 31394, never changes. It's like something is stuck. I've got a case open with support, but so far they've come up with nothing.

Any ideas? Since there are no errors in the event logs, I'm not too concerned yet, but I'd like to know where this alert is coming from. Where does that item count come from? Is this the result of a database query, or something in MSMQ? The server is rebooted monthly, or more often if I'm troubleshooting an issue.

Thanks,
Mike

 

25 REPLIES 25

JesusWept3
Level 6
Partner Accredited Certified

OK this may be a bit obscure but....
Can you delete C:\Program Files (x86)\EnterpriseVault\MRIndexServer.dat
Restart the Enterprise Vault Indexing Service and try the rebuild of the index again?

Otherwise what i would suggest is tell support about this thread and have them advance it to ADV support or Backline support, though be aware they may suggest doing an upgrade to EV10 SP3, not that I can see this has been a known issue or fixed in future releases, but lets say it is a bug and you wanted a hotfix and not a work around, they could only give it for EV10 SP3 and not below

https://www.linkedin.com/in/alex-allen-turl-07370146

JesusWept3
Level 6
Partner Accredited Certified
Actually thinking about this more Did you at ever point restore that database per chance? I'm wondering that maybe items were purged for the user and the directory database got rid of the older indexes and then a vault store was restored which put back items records that were previously deleted but because the index volume was removed from directory, it can't find the volume anymore to reindex those items
https://www.linkedin.com/in/alex-allen-turl-07370146

JesusWept3
Level 6
Partner Accredited Certified

To determine this, run this query first , and see if the database was ever restored at any point, this may give a false positive though if you restored the database if you migrated from one SQL Server to another....


SELECT [rs].[destination_database_name], 
[rs].[restore_date], 
[bs].[backup_start_date], 
[bs].[backup_finish_date], 
[bs].[database_name] as [source_database_name], 
[bmf].[physical_device_name] as [backup_file_used_for_restore]
FROM msdb..restorehistory rs
INNER JOIN msdb..backupset bs
ON [rs].[backup_set_id] = [bs].[backup_set_id]
INNER JOIN msdb..backupmediafamily bmf 
ON [bs].[media_set_id] = [bmf].[media_set_id] 
ORDER BY [rs].[restore_date] DESC


And then run this query to see the date range of items that were deleted
 

SELECT DISTINCT(A.ArchiveName) "Archive Name",
       MIN(JD.DeletionDate) "First Deletion Date",
       MAX(JD.DeletionDate) "Last Deletion Date"
FROM   yourVaultStore.dbo.JournalDelete JD,
       yourVaultStore.dbo.ArchivePoint AP,
       EnterpriseVaultDirectory.dbo.Root R,
       EnterpriseVaultdirectory.dbo.Archive A
WHERE  JD.ArchivePointIdentity = AP.ArchivePointIdentity
  AND  AP.ArchivePointId = R.VaultEntryId
  AND  R.RootIdentity = A.RootIdentity
  AND  JD.IndexCommitted = 0
GROUP BY A.ArchiveName
ORDER BY A.ArchiveName
https://www.linkedin.com/in/alex-allen-turl-07370146

MikeAtGamestop
Level 3

We've never done any SQL restore that involved the EV databases that I'm aware of, but we have a separate team that handles the SQL servers.

The first query returned several records for the DAConfiguration database, but nothing for the mailbox store.

The second query returned:

Archive Name  First Deletion Date  Last Deletion Date
User1 2012-12-07 02:22:55.037 2012-12-07 02:24:31.090
User2 2012-12-10 04:50:51.883 2012-12-10 04:51:12.377
User3 2012-10-25 04:07:51.170 2012-10-25 04:15:34.860
User4 2012-12-15 04:59:43.553 2012-12-15 05:05:36.397
User5 2013-02-10 04:33:04.797 2013-02-10 04:39:28.890

I do have a case open with support to work on several alerts in the console, but we haven't focused on this particular one so far. I'll refer them to this thread when I get back from vacation at the end of the month. smiley

 

Thank you very much for your help and advice. I appreciate it very much.

Mike

 

MikeAtGamestop
Level 3

Interesting. I see that Symantec just posted a new Technote, which sounds very similar to this problem.

http://www.symantec.com/docs/TECH203681

Same error code and message that I saw in the dtrace.

 

JesusWept3
Level 6
Partner Accredited Certified

and it was created today!!!!
good job :)

so best thing to do, tell the symc person about that technote, quote the etrack, ask them to put you on the etrack as well, they will give you the work around and then you can consider it closed.

 

https://www.linkedin.com/in/alex-allen-turl-07370146