Move between EV sites fails (0x80010108)

[Jakob]

Hi together,

i´m in the process of migrate mass of mail archives to a new EV surrounding.

The old System is a single EV 9.0.2 single Server based on Windows 2003 with a dedicated SQL Database and an NTFS Vault Store.

The new System and new EV Site is a single Server with EV 10.0.4 CH3 based on Windows Server 2008 R2 with a dedicated SQL Datbase and a NTFS Vault Store.

The first few hundred archives i´ve already migrated. I had different errors which result from a data corruption in the past in some Collections. We know about it and live with this possible data loss. So we confgured different exceptions in the move Config:
 

<add key="NonFatalHexCodes" value="0x80040306,0x8000ffff,0x80070002,0x80004005,0x80004002,0xc0041bdf,0xc0041b0f,0x80030050,0xc0041aa7,0x8004011b,0x80070057,0x80040308,0xc00418e7,0x800706be"/>

In the last weeks one error comes up again and again for which i didn´t found the reason. In most cases the move proceeds after reporting this error, but it leads to big delay of the planned end of the migration. In the last days only 1-3 Migrations where finished. A value of 10-15 is normal.

The error is logged in the EV Application log with the EventID 41226. The message is:

An error was encountered while processing an item. Enterprise Vault will try again later unless the retry count has been reached.

Item: B0DC5432437A9F83D6BA104C6A2CAD61

Archive: 192DFA2D80055544D831B4781A451A7F51110000XXXVAULT

Exception: Source item saveset ID: 201208134878717~201207131343240000~Z~B0DC5432437A9F83D6BA104C6A2CAD61. CM API error code: 0x80040306. CM API error: An internal failure occurred. Internal Error: 'The object invoked has disconnected from its clients.  (0x80010108)'.. Inner storage error code: 0x80010108. Inner storage error description: The object invoked has disconnected from its clients. .   at KVS.EnterpriseVault.Interop.IItem.Insert()
   at KVS.EnterpriseVault.MoveArchiveTask.ItemProcessor.CopyItem(QueuedSourceItem sourceitem, IContentManagementAPI4 cmAPImaThread, String& srcSSID, String& destSSID).

I don´t won to exclude the Error Code 0x80010108 in the Move Config because i don´t know the reason for it and i don´t want to have more data loss than needed.

In the Dtrace of the move i don´t find a more precise description of the error:

47466    09:52:16.082     [688]    (EVMoveArchiveTask)    <9888>    EV:M    CContentManagementAPI::AccessSimpleStoreT Setting DCOM security
47467    09:52:16.098     [688]    (EVMoveArchiveTask)    <9888>    EV:L    CContentManagementAPI::TransformAuthenticationIdentity AuthenticationTypeIdentity is WindowsUser and no Identity set - defaulting to '.' (current user)
47503    09:52:19.442     [688]    (EVMoveArchiveTask)    <4244>    EV:L    CAuthHelper::Reset Cancel registration? True CancelId: 0
47504    09:52:19.442     [688]    (EVMoveArchiveTask)    <4244>    EV:M    Error accessing SimpleStore. Attempt:1 EntryId:10AB5D5075477AA439ADBFD2BBDCD59BF1110000NEWSITE|hr=The object invoked has disconnected from its clients.  (0x80010108)
47505    09:52:19.442     [688]    (EVMoveArchiveTask)    <4244>    EV:M    CItem::Insert Elapsed time 104647.4ms
47506    09:52:19.458     [688]    (EVMoveArchiveTask)    <4244>    EV:L    CStorageProxyBase::FindStorageComputer reading computer name from cache for entryid=10AB5D5075477AA439ADBFD2BBDCD59BF1110000NEWSITE
47507    09:52:19.458     [688]    (EVMoveArchiveTask)    <4244>    EV:L    CStorageProxyBase::FindStorageComputer reading computer name from cache for entryid=10AB5D5075477AA439ADBFD2BBDCD59BF1110000NEWSITE
47508    09:52:19.458     [688]    (EVMoveArchiveTask)    <4244>    EV:M    VaultCoCreateInstanceEx: IsLocalMachineExtendedCheck returned false for server [NEWSITE.ONE4ALL.LAN]  as the IP Address matched that of the local machine
47509    09:52:19.473     [688]    (EVMoveArchiveTask)    <4244>    EV:M    CStorageProxyBase::_GetStorageObject - Setting DCOM security
47510    09:52:19.473     [688]    (EVMoveArchiveTask)    <4244>    EV:M    CContentManagementAPI::AccessSimpleStoreT Setting DCOM security
47511    09:52:19.473     [688]    (EVMoveArchiveTask)    <4244>    EV:L    CContentManagementAPI::TransformAuthenticationIdentity AuthenticationTypeIdentity is WindowsUser and no Identity set - defaulting to '.' (current user)
47665    09:52:25.942     [688]    (EVMoveArchiveTask)    <8668>    EV:L    CAuthHelper::Reset Cancel registration? True CancelId: 0
47666    09:52:25.942     [688]    (EVMoveArchiveTask)    <8668>    EV:M    Error accessing SimpleStore. Attempt:1 EntryId:10AB5D5075477AA439ADBFD2BBDCD59BF1110000NEWSITE|hr=The object invoked has disconnected from its clients.  (0x80010108)
47667    09:52:25.942     [688]    (EVMoveArchiveTask)    <8668>    EV:M    CItem::Insert Elapsed time 111695.3ms
47668    09:52:25.958     [688]    (EVMoveArchiveTask)    <8668>    EV:L    CStorageProxyBase::FindStorageComputer reading computer name from cache for entryid=10AB5D5075477AA439ADBFD2BBDCD59BF1110000NEWSITE
47669    09:52:25.958     [688]    (EVMoveArchiveTask)    <8668>    EV:L    CStorageProxyBase::FindStorageComputer reading computer name from cache for entryid=10AB5D5075477AA439ADBFD2BBDCD59BF1110000NEWSITE
47670    09:52:25.958     [688]    (EVMoveArchiveTask)    <8668>    EV:M    VaultCoCreateInstanceEx: IsLocalMachineExtendedCheck returned false for server [NEWSITE.DOMAIN.LAN]  as the IP Address matched that of the local machine
47671    09:52:25.973     [688]    (EVMoveArchiveTask)    <8668>    EV:M    CStorageProxyBase::_GetStorageObject - Setting DCOM security
47672    09:52:25.973     [688]    (EVMoveArchiveTask)    <8668>    EV:M    CContentManagementAPI::AccessSimpleStoreT Setting DCOM security
47673    09:52:25.973     [688]    (EVMoveArchiveTask)    <8668>    EV:L    CContentManagementAPI::TransformAuthenticationIdentity AuthenticationTypeIdentity is WindowsUser and no Identity set - defaulting to '.' (current user)
47674    09:52:26.520     [688]    (EVMoveArchiveTask)    <12156>    EV-L    {EnumerateAndProcessArchive.WaitForThreads} Wait Timeout
47676    09:52:26.520     [688]    (EVMoveArchiveTask)    <12156>    EV:L    CBaseDirectoryServiceWrapper::CreateDirectoryService() - Entry [m_nNumTries = 40]
47677    09:52:26.520     [688]    (EVMoveArchiveTask)    <12156>    EV:L    CBaseDirectoryServiceWrapper::CreateDirectoryService() - Successfully communicated with an EV Directory Service on the local machine
48209    09:52:56.537     [688]    (EVMoveArchiveTask)    <12156>    EV-L    {EnumerateAndProcessArchive.WaitForThreads} Wait Timeout
48211    09:52:56.537     [688]    (EVMoveArchiveTask)    <12156>    EV:L    CBaseDirectoryServiceWrapper::CreateDirectoryService() - Entry [m_nNumTries = 40]
48212    09:52:56.537     [688]    (EVMoveArchiveTask)    <12156>    EV:L    CBaseDirectoryServiceWrapper::CreateDirectoryService() - Successfully communicated with an EV Directory Service on the local machine
48271    09:53:03.600     [688]    (EVMoveArchiveTask)    <9840>    EV:L    CAuthHelper::Reset Cancel registration? True CancelId: 0
48272    09:53:03.600     [688]    (EVMoveArchiveTask)    <9840>    EV:M    Error accessing SimpleStore. Attempt:2 EntryId:10AB5D5075477AA439ADBFD2BBDCD59BF1110000NEWSITE|hr=The object invoked has disconnected from its clients.  (0x80010108)
48273    09:53:03.600     [688]    (EVMoveArchiveTask)    <9840>    EV:M    CItem::Insert Elapsed time 68409.1ms
48274    09:53:03.615     [688]    (EVMoveArchiveTask)    <9840>    EV:H    CItem::Insert - HRXEX fn trace : Error 0x80010108, .\Item.cpp [lines {484,518,611,615,619,697}], built Mar 14 10:56:28 2011.
48275    09:53:03.615     [688]    (EVMoveArchiveTask)    <9840>    EV:L    CAuthHelper::Reset Cancel registration? True CancelId: 0
48276    09:53:03.615     [688]    (EVMoveArchiveTask)    <9840>    EV:M    CItem::Insert com_error raised Error 0x80010108 (err=80010108)

I already tried to reduce the ammount of parallel moves and tasks but also with the smalest values the error appeares. I didn´t found any errors on the depending File and SQL Server also. McAffee is deactivated on the EV Server

The error already appeard in the past, but not very often.

Any ideas?

Jakob

[Jakob]

Hey,

...no ideas? It´s very hard to move all the Archives (ca. 1700 remain) with this error. 

Every Idea which could lead to the solution would be great.

Jakob

[GabeV]

Hi Jakob,

According with these three lines in the dtrace, it's taking over a minute to insert an item on three different threads:

47505    09:52:19.442     [688]    (EVMoveArchiveTask)    <4244>    EV:M    CItem::Insert Elapsed time 104647.4ms
47667    09:52:25.942     [688]    (EVMoveArchiveTask)    <8668>    EV:M    CItem::Insert Elapsed time 111695.3ms
48273    09:53:03.600     [688]    (EVMoveArchiveTask)    <9840>    EV:M    CItem::Insert Elapsed time 68409.1ms

Take a look at the SQL server to see if there are any timeouts occurring for any reason (SQL Profiler maybe). Since you have the archive ID, you can check the ArchiveFolderView to rule out any issue in the folder structure:

SELECT * FROM EnterpriseVaultDirectory.dbo.ArchiveFolderView WHERE ArchiveVEID = '192DFA2D80055544D831B4781A451A7F51110000XXXVAULT'

You mentioned that the error already appeard in the past, but was it fixed? Also, I suggest you to open a case with support for further assistance.

I hope this helps.

[GertjanA]

Spend some money on ArchiveShuttle from QUADROtech. Saves you time, and sorrow. Gives you audit and logs to prove all is ok.

<end commercial break>

I'm moving about 1800 archives from one site to the other too using M.A. As time was not an issue, this was choosen. I have no errors (yet), but do know that SQL needs to be well maintained. If you have ev and sql on one box, then that might be the issue. Make sure you don't overstretch the moving server (by editing the threads and number of processes)

[Jakob]

Hi Gabe,

thank you for the hint. On the first view i don´t see performance related Errors on the dedicated SQL Servers.

So I have to do some further analysis - From which SQL Server do you think comes this error? Source or destination.

The error comes up in the past only in individual cases. Because it proceeds - i ignored them.

@Gertjan:

at the beginning of the move we decided to use the built in move functions. In generall it works fine even if it is not very comfortable.

The SQL Servers seperated Servers and i´ve already adjusted the threads and processes to a minimum.

Regards

Jakob

[Jakob]

I´ve checked the maintenance Plans on both SQL Servers with the following links:

http://www.symantec.com/business/support/index?page=content&id=TECH74666#Creating_a_database_maintenance_plan_for_EV__CA__and_DA_SQL_databases

and

http://www.symantec.com/business/support/index?page=content&id=TECH168905

The maintenance Plans are working normaly.

Another Application on the Source SQL Server is running fine - so it doesnt seems to be a general performance issue of the QSL Server and the results of the queries from the both named articles didn´t show any abnormalities..

On the destination SQL Server it also seems to be right and the results of the queries from the both named articles didn´t show any abnormalities.

[Jakob]

@Gabe: The Query you posted to check the folders delivers a valid result. I can see different folders there...

Any more ideas?

[Jakob · Accepted solution]

Hi together,

I´ve opened a ticket at Symantec Support and we´ve figured out that there was a DCOM-related issue on the destination EV-System.

We didn´t found out what exactly was the error, because after a restart of the destination EV Server the issue was gone. I hope it will not come up again.

thank you for your help.

Regards

Jakob