EV10 - Indexing Problem with one element

[Simon_B_]

Hello folks,

I have an interesting problem with the 64Bit Indexing Engine in EV10.

It seems that there is a mail element which is preventing the index from updating correctly.

Although the index itself is showing healthy this specific element and all elements which were archived after this are not added to the index, even if rebuilding the whole index.

Interestingly enough when searching for the element in the users mailbox is shows up as correctly converted to a shortcut, also the element including the attachments can be opened without issues. Even a manual retsore and archive is possible.

This leads me to the conclusion that the problem solely lies with indexing.

In EV9 and before there was the so called poison pilling which skipped an element after it failed to index 3 times. It seems as if 64Bit Indexing does not have this functionality or it does not kick in here as indexing tries to index this element over and over and fails every time.

Debug Information:

Dtrace:

84843 09:57:56.368 [13128] (StorageCrawler) <15292> EV:H CEVByteStreamMemoryFileImpl::UseFileStreamNow Swapping to using a file stream

84844 09:57:56.369 [13128] (StorageCrawler) <15292> EV:H CEVByteStreamMemoryFileImpl::UseFileStreamNow Stream created on C:\Users\evault\AppData\Local\Temp\EV$9DCC.tmp

..

86034 09:59:50.198 [13128] (StorageCrawler) <15292> EV:L CSequentialStream::CopyTo (CEVByteStream::CopyTo) Buffer Size:64K Read:2031.813MB (2130509824 bytes) Written:2031.750MB (2130444288 bytes) hr=The compound file is too large for the current implementation  (0x80030111) Elapsed:114333.7ms

86035 09:59:50.198 [13128] (StorageCrawler) <15292> EV:H CEVByteStream::CopyTo _com_error exception. hr=The compound file is too large for the current implementation  (0x80030111)

86036 09:59:50.198 [13128] (StorageCrawler) <15292> EV:H CSavesetOnIStg2::put_ConvertedContent _com_error exception. hr=The compound file is too large for the current implementation  (0x80030111)

86037 09:59:50.199 [13128] (StorageCrawler) <15292> EV:H {CSavesetOnIStg2::put_ConvertedContent} (Exit) Status: [The compound file is too large for the current implementation  (0x80030111)]

86038 09:59:50.199 [13128] (StorageCrawler) <15292> EV:H CSaveset2::put_ConvertedContent _com_error exception. hr=The compound file is too large for the current implementation  (0x80030111)

86039 09:59:50.199 [13128] (StorageCrawler) <15292> EV:H {CSaveset2::put_ConvertedContent} (Exit) Status: [The compound file is too large for the current implementation  (0x80030111)]

..

86345 09:59:50.585 [13128] (StorageCrawler) <15292> EV:H CSisPartConstructorBase::LoadSisPartConvertedContent _com_error exception. hr=The compound file is too large for the current implementation  (0x80030111)

86346 09:59:50.585 [13128] (StorageCrawler) <15292> EV:H {CSisPartConstructorBase::LoadSisPartConvertedContent} (Exit) Status: [The compound file is too large for the current implementation  (0x80030111)]

86347 09:59:50.585 [13128] (StorageCrawler) <15292> EV:H CSisPartConstructorBase::OnAttachmentOrDocument _com_error exception. hr=The compound file is too large for the current implementation  (0x80030111)

86348 09:59:50.585 [13128] (StorageCrawler) <15292> EV:H {CSisPartConstructorBase::OnAttachmentOrDocument} (Exit) Status: [The compound file is too large for the current implementation  (0x80030111)]

86349 09:59:50.585 [13128] (StorageCrawler) <15292> EV:H CSisPartConstructorBase::OnAttachment _com_error exception. hr=The compound file is too large for the current implementation  (0x80030111)

86350 09:59:50.585 [13128] (StorageCrawler) <15292> EV:H {CSisPartConstructorBase::OnAttachment} (Exit) Status: [The compound file is too large for the current implementation  (0x80030111)]

86351 09:59:50.585 [13128] (StorageCrawler) <15292> EV:H {CMAPIMsgTraverser::ProcessAttachment} (Exit) Status: [The compound file is too large for the current implementation  (0x80030111)]

86352 09:59:50.585 [13128] (StorageCrawler) <15292> EV:H {CMAPIMsgTraverser::ProcessMessage} (Exit) Status: [The compound file is too large for the current implementation  (0x80030111)]

86353 09:59:50.585 [13128] (StorageCrawler) <15292> EV:H {CMAPIMsgTraverser::Traverse} (Exit) Status: [The compound file is too large for the current implementation  (0x80030111)]

86354 09:59:50.585 [13128] (StorageCrawler) <15292> EV:L {CMAPIMsgTraverser::LiberateMessage} (Entry)

86355 09:59:50.755 [13128] (StorageCrawler) <15292> EV:L {CMAPIMsgTraverser::LiberateMessage} (Exit) Status: [Success]

86356 09:59:50.755 [13128] (StorageCrawler) <15292> EV:H {CMsgTraverserControl::Traverse} (Exit) Status: [The compound file is too large for the current implementation  (0x80030111)]

86357 09:59:50.755 [13128] (StorageCrawler) <15292> EV:H CSisPartConstructorBase::LoadSisParts _com_error exception. hr=The compound file is too large for the current implementation  (0x80030111)

86358 09:59:50.755 [13128] (StorageCrawler) <15292> EV:H {CSisPartConstructorBase::LoadSisParts} (Exit) Status: [The compound file is too large for the current implementation  (0x80030111)]

86359 09:59:50.755 [13128] (StorageCrawler) <15292> EV:H CSisPartConstructorBase::Build _com_error exception. hr=The compound file is too large for the current implementation  (0x80030111)

86360 09:59:50.755 [13128] (StorageCrawler) <15292> EV:H {CSisPartConstructorBase::Build} (Exit) Status: [The compound file is too large for the current implementation  (0x80030111)]

86361 09:59:50.755 [13128] (StorageCrawler) <15292> EV:H {CVault::GetSaveset} (Exit) Status: [The compound file is too large for the current implementation  (0x80030111)]

86362 09:59:50.755 [13128] (StorageCrawler) <15292> EV:L {CVault::GetItemProperties} (Entry)

86363 09:59:50.755 [13128] (StorageCrawler) <15292> EV:H {CVault::GetItemProperties}|Selector: D02041B015CFBC2329EA5631907F8E21

86364 09:59:50.755 [13128] (StorageCrawler) <15292> EV:H CSavesetIdentifier::ExtractTransactionId Invalid saveset identifier: D02041B015CFBC2329EA5631907F8E21

86365 09:59:50.759 [13128] (StorageCrawler) <15292> EV:L CVaultStoreDB::GetSavesetProperties Information: No SISParts found for saveset having TID:'D02041B0-15CF-BC23-29EA-5631907F8E21'.

86366 09:59:50.759 [13128] (StorageCrawler) <15292> EV:L {CVault::ReadPartitionEntry} (Entry)

86367 09:59:50.759 [13128] (StorageCrawler) <15292> EV:M {CVault::ReadPartitionEntry}|Id: 5

86369 09:59:50.759 [13128] (StorageCrawler) <15292> EV:L CBaseDirectoryServiceWrapper::CreateDirectoryService() - Entry [m_nNumTries = 40]

86370 09:59:50.761 [13128] (StorageCrawler) <15292> EV:L CBaseDirectoryServiceWrapper::CreateDirectoryService() - Successfully communicated with an EV Directory Service on the local machine

86371 09:59:50.765 [13128] (StorageCrawler) <15292> EV:L CDirectoryVaultObject::GetAttributeListFromType Unknown type - view_VSEntry_VSGroup_OpenPartition

86372 09:59:50.765 [13128] (StorageCrawler) <15292> EV:M CPartitionProperties::CPartitionProperties - vsId = [1], partitionId = [5]

86374 09:59:50.765 [13128] (StorageCrawler) <15292> EV:L CBaseDirectoryServiceWrapper::CreateDirectoryService() - Entry [m_nNumTries = 40]

86375 09:59:50.766 [13128] (StorageCrawler) <15292> EV:L CBaseDirectoryServiceWrapper::CreateDirectoryService() - Successfully communicated with an EV Directory Service on the local machine

86376 09:59:50.780 [13128] (StorageCrawler) <15292> EV:L {CVaultStorePartitionCache::LoadEntry:#493} Successfully loaded partition info for Vault Store [14F3C061CC65B614898B78DFF82BC01BB1210000EVSite] from Directory

86377 09:59:50.780 [13128] (StorageCrawler) <15292> EV:L {CVaultStorePartitionCache::ReadEntry:#135} Successfully read partition info for Vault Store [14F3C061CC65B614898B78DFF82BC01BB1210000EVSite]

86378 09:59:50.780 [13128] (StorageCrawler) <15292> EV:M {CVault::ReadPartitionEntry}|Partition Id:5VaultStore:14F3C061CC65B614898B78DFF82BC01BB1210000EVSite HR=Success  (0)

86379 09:59:50.780 [13128] (StorageCrawler) <15292> EV:L {CVault::ReadPartitionEntry} (Exit) Status: [Success]

86380 09:59:50.780 [13128] (StorageCrawler) <15292> EV:M CSSASCache::ReadEntry Id:1C243C9B7C414CA4A9E613FB791F8FA841e10000EVSite hr:0

86381 09:59:50.780 [13128] (StorageCrawler) <15292> EV:M CRetentionCategoryIdentityCache::ReadEntry Id:2 SiteId:16CDB6D00D8762A41AED6117620DEC7141d10000EVSite hr:0

86382 09:59:50.780 [13128] (StorageCrawler) <15292> EV:M {CVault::GetItemProperties}|Sel:D02041B015CFBC2329EA5631907F8E21 Seq No:70014 Partition ID:5 Store Id: CollectionId:False

86383 09:59:50.780 [13128] (StorageCrawler) <15292> EV:L {CVault::GetItemProperties} (Exit) Status: [Success]

86384 09:59:50.780 [13128] (StorageCrawler) <15292> EV:H CVault::HandleVaultError CVault setup ErrorId : [0xC0041AE2], CLogErr EventId : [0xC0041AE2]

86385 09:59:50.780 [13128] (StorageCrawler) <15292> EV:M CEventLog::RegisterEventFilterForThread - registered thread filter callback [TID: 15292]

86386 09:59:50.780 [13128] (StorageCrawler) <15292> EV:L CEventBuffer::Unregister - unregistered EventBuffer filter callback [TID: 15292]

86387 09:59:50.786 [13128] (StorageCrawler) <15292> EV~E Event ID: 6882 Unable to complete retrieval request | |Reason:|          The compound file is too large for the current implementation  (0x80030111) |Vault:              *sanitized* |Vault Id:           104F3D058F255164782654243C55D8A141110000EVSite |Item Id:            201202028211954~201201121743470000~Z~D02041B015CFBC2329EA5631907F8E21 |Reference:          Vault/RI |Extended Item Info: 104F3D058F255164782654243C55D8A141110000EVSite+201202028211954~201201121743470000~Z~D02041B015CFBC2329EA5631907F8E21+5++0+2+40941.82291666666400000+154101+70014++65948+877+15133767+0 |Supplementary Info: | |                     |

DumpSaveset:

2012-03-29 13:06:35  Starting DumpSaveset operations

2012-03-29 13:06:35  Getting recombined DVS file

2012-03-29 13:17:00  ERROR: Failed to get recombined saveset. Error: %s

2012-03-29 13:17:00  Unspecified error  (0x80004005)

2012-03-29 13:17:00                      Event Output: Unable to complete retrieval request      Reason:           The compound file is too large for the current implementation  (0x80030111)   Vault:              *sanitized*   Vault Id:           13CC8E318CD2A7643BE3CBF288097794B1110000EVSite   Item Id:            201202028211954~201201121743470000~Z~D02041B015CFBC2329EA5631907F8E21   Reference:          Vault/RI   Extended Item Info:    Supplementary Info:                                (0xc0041ae2)

Any ideas how to troubleshoot/fix this? 

[MarkBarefoot]

This HF Technote may help you, http://www.symantec.com/docs/TECH182076

This was identified a while back and looking as some internal defect notes this issue was replicated internally, and also resolved using the HF.

[Simon_B_]

Hi Mark,

thanks for the fast reply, I'll test with the new hotfix and let you know the result.

Regards

Simon

[Simon_B_]

Hi Mark,

unfortunately this HF does not resolve the issue. When rebuilding the index after installing it the issue occurs again.

Any further ideas? Maybe an idea on how to remove this single element from the archive or the indexing queue?

[MarkBarefoot]

Let me see if I can find anything more...

[Ed_Whyatt]

Hi Simon,

Do you have the EVIndexVolumesProcessor trace that matches the storagecrawler trace above?

One thing I'd suggest is to run an Index Synchronize task against the volume with trace against the two processes mentioned above.

The sync task will only attempt to add the missing item(s).

Another question - if you've managed to retrieve the item from the archive, could you advise us on the make-up of the item?

Language - # of recipients - attachment types, sizes etc.

The trace implies the combined saveset is too large - how large is it and how many saveset parts are there?

Cheers

Ed

[Simon_B_]

Hi Ed,

thanks for following up on this. I'll perform a Sync with a Dtrace which includes EVIndexVolumesProcessor.

Regarding the Mail Element in Question: I have not managed to identify anything which is outstanding regarding this mail. It contains a normal charset, and has only a small number of recipients. There were less than five attachments, and the total attachment size was around 5MB - so pretty normal.

An interesting observation is: Although the element is smaller than 10MB one of the DVSCC which is recalled when performing a dumpsaveset is 145MB in size.

I'll check the element again to provide more accurate details regarding the message.

[Tremaine]

Hi Simon

From the sounds of things I would recommend opening a support case and provide the details/trace as per Ed's post above along with the Saveset as well so we can see whether we can reproduce the issue and troubleshoot it further. If however you can ascertain that there are a very large number of recipients on the message then you may want to set the recipientmax regkey as per http://www.symantec.com/business/support/index?page=content&id=HOWTO74999

[Simon_B_]

I've collected the traces, but as Ghost said I'll probably log a support case for this.

If any of you want to look through a part of the trace showing the error - its attached to this case.

Apart from logging the case, do you have any suggestions how to remove this single element? You cannot delete it through AE or the search as it does not turn up in the index. Configuring Hard Delete and deleting the shortcut in Outlook might work - but this might leave the element in the indexing queue. 

[AMPearce]

We've developed the exact same "The compound file is too large for the current implementation" error. I'll log a support call also.

Simon, was the issue resolved successfully?

[dAncer]

I've got the same problem. Maybe this is the solution:

http://www.symantec.com/business/support/index?page=content&id=TECH191942

[AMPearce]

Yes, that's the same steps we followed to workround the problem. Will still need to rename them back at some point and re-index of a proper patch is in place.