Solved: Index errors after upgrade to 10.0.3 - Page 2

Sarah_Seftel1 · ‎03-14-2013

Hi

Upgrade of EV from 9.0.2 to 10.0.3 in microsoft cluster (active passive).
Upgrade completed succesfully.

Index administration task created with upgrade to all 32bit indexes.

Upgrade is not working (stays inactive), but it looks like system is having indexing problems, as events are full with the following:

Log Name: Symantec Enterprise Vault

Source: Enterprise Vault

Date: 14/03/2013 13:40:13

Event ID: 40966

Task Category: Index Admin Service

Level: Error

Keywords: Classic

User: N/A

Computer: Server

Description:

A program fault has raised an exception.

Exception: Failed to get subtask for subtask ID 1DB7E29DCE8D64FEA8701D5EFB860EAED1013b00vaultserver

Diagnostic:

Type: System.ServiceModel.FaultException`1[[Symantec.EnterpriseVault.Indexing.Common.CommunicationFault, EVSharedManagedInterfaces, Version=10.0.3.0, Culture=neutral, PublicKeyToken=26c5e2ccf2b9267c]]

Reference: An error occurred while invoking Void <StopSubTask>b__f().

Error Context : Stopping sub task for TaskEntryId 17DC08710A3B705409EFC2C9200093CFE1013a00vaultserver and SubTaskEntryId 1DB7E29DCE8D64FEA8701D5EFB860EAED1013b00vaultserver

Command Line: "E:\Enterprise Vault\EVIndexAdminService.exe" -EntryID:1EC5115667B050B40B71EFC0504B51FD71710000vaultserver

Application Domain: EVIndexAdminService.exe

Process Id: 8836

Thread Id: 18288

Stack Trace:

Server stack trace:

at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)

at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)

at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)

at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:

at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)

at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)

at Symantec.EnterpriseVault.Indexing.IndexVolumesProcessor.WebService.ITaskProcessorWebService.StopProcessingSubtask(String taskId, String subtaskId, StopReason reasonForStop)

at Symantec.EnterpriseVault.Indexing.Admin.IndexAdminImpl.Stop64BitSubTasks(String taskEntryId, String subTaskEntryId, StopReason reasonForStop)

at Symantec.EnterpriseVault.Indexing.Admin.EVIndexAdminUtils.InvokeHelper(Object COMInterface, Action act, String context, LoggingMode loggingMode, Boolean rethrow)

-----------------------------------------------------------------------------

Log Name: Symantec Enterprise Vault

Source: Enterprise Vault

Date: 14/03/2013 13:44:02

Event ID: 7182

Task Category: Web Application (WP)

Level: Error

Keywords: Classic

User: N/A

Computer: Server

Description:

Index Search failed: Unspecified error (0x80004005)

Index: 1EB6DC7DA00ED2447BA21981FFB2D908D1110000vaultserver/VolumeSet:866

Internal reference: SRCI

------------------------------------------------------------------------------

Upgrade went smooth with no special issues, all steps performed by the book, including moving the IndexMetaData to shared location.

Any ideas?

Also attached dtrace of the following:

EVIndexAdminService

EVIndexVolumesProcessor

IndexBroker

IndexServer

Thanks,

Sarah.

James_Drake · ‎04-08-2013

ok sounds like a good idea. I built this server a week ago and it had all standard and critical updates applied from Windows Update. The server is virtual running on an ESXi 5.0 platform with the version of VMware tools that shipped with v 5.0.

SQL 2008 R2 SP2 is installed on the same virtual server.

EV 10.0.3 has both the cumulative hotfix 1 installed and the other addtional one (fix for Exchange 2007 - 2013 mailbox moves)

Windows 2008 R2 SP1 is installed

Outlook 2007 SP3 is the MAPI provider

The list of Windows hotfixes that are installed are as follows

(generated by powershell

HotFixId
--------
KB981391
KB981392
KB977236
KB981111
KB977238
982861
KB977239
KB2670838
KB2592687
KB981390
KB2425227
KB2506014
KB2506212
KB2506928
KB2509553
KB2511455
KB2515325
KB2530548
KB2533552
KB2536275
KB2536276
KB2541014
KB2544893
KB2545698
KB2547666
KB2552343
KB2560656
KB2563227
KB2564958
KB2570947
KB2574819
KB2584146
KB2585542
KB2603229
KB2604115
KB2607047
KB2608658
KB2618451
KB2620704
KB2620712
KB2621440
KB2631813
KB2636573
KB2640148
KB2643719
KB2644615
KB2645640
KB2647753
KB2653956
KB2654428
KB2656356
KB2659262
KB2660075
KB2661254
KB2667402
KB2676562
KB2685811
KB2685813
KB2685939
KB2690533
KB2691442
KB2698365
KB2699779
KB2705219
KB2709630
KB2709981
KB2712808
KB2718704
KB2719857
KB2726535
KB2729094
KB2729452
KB2732059
KB2736233
KB2739159
KB2742599
KB2743555
KB2749655
KB2750841
KB2753842
KB2757638
KB2758857
KB2761217
KB2763523
KB2765809
KB2769369
KB2770660
KB2778344
KB2779562
KB2785220
KB2786081
KB2786400
KB2789645
KB2790113
KB2790655
KB2791765
KB2797052
KB2799494
KB976902

GertjanA · ‎04-08-2013

Although I am on an EV9 environment, I also am on 2008r2sp2.

A few weeks ago we had an issue, where we found that after a patch-weekend, after the reboot, an additional DotNet patch was installed. That patch also required a reboot, but was not listed anywhere. I assume you have also already rebooted this server, but it might be worth a check to see if there is something in the dotnet folder, that has a later date than the last reboot.

I hope you be able to find the issue soon. These are really difficult to troubleshoot and find...

Regards. Gertjan

James_Drake · ‎04-08-2013

The server has been rebooted about 20 times since the initial installation and only 10 minutes ago. Unfortunately this is not the fix.

Sarah_Seftel1 · ‎04-08-2013

KB2425227

KB2446710

KB2484033

KB2488113

KB2492386

KB2497640

KB2503658

KB2503665

KB2505438

KB2506014

KB2506212

KB2506223

KB2506928

KB2507618

KB2508272

KB2508429

KB2509553

KB2510531

KB2511250

KB2511455

KB2512715

KB2515325

KB2518869

KB2522422

KB2522766

KB2524375

KB2528357

KB2529073

KB2533552

KB2534366

KB2536275

KB2536276

KB2539635

KB2541014

KB2544521

KB2544893

KB2545698

KB2547666

KB2552343

KB2556532

KB2560656

KB2562937

KB2563227

KB2564958

KB2567680

KB2570791

KB2570947

KB2572077

KB2586448

KB2588516

KB2598845

KB2603229

KB2607576

KB2616676

KB2617657

KB2620704

KB2633952

KB2641690

KB2794119

KB976902

KB976932

KB982018

will do the compare later...

Sarah_Seftel1 · ‎04-08-2013

yes, servers were rebooted so many times I stopped counting...

Sarah_Seftel1 · ‎04-08-2013

Another customer with same issue:

HotFixID

--------

KB2536276

KB2552343

KB2560656

KB2564958

KB2570947

KB2584146

KB2585542

KB2604115

KB2618451

KB2620704

KB2620712

KB2621440

KB2631813

KB2633952

KB2643719

KB2644615

KB2645640

KB2653956

KB2654428

KB2655992

KB2656356

KB2656411

KB2658846

KB2659262

KB2661254

KB2667402

KB2676562

KB2685939

KB2690533

KB2691442

KB2698365

KB2705219

KB2706045

KB2712808

KB2718704

KB2719033

KB2729452

KB2736422

KB2742599

KB2743555

KB2749655

KB2753842

KB2756921

KB2757638

KB2758857

KB2765809

KB2769369

KB2770660

KB2785220

KB2786081

KB976902

James_Drake · ‎04-08-2013

I have done a comparison but unless I have done something wrong there are only a few similarities between all 3 hotfix lists. I used beyond compare, using my list as the master, comparing it against both of yours and then comparing both outputs against each other.

KB2536276
KB2552343
KB2560656
KB2564958
KB2570947
KB2620704
KB976902

What operating system and service pack were your servers built with?

Sarah_Seftel1 · ‎04-09-2013

just got the following from Symantec.

http://www.symantec.com/business/support/index?page=content&id=TECH203789

going to try...

James_Drake · ‎04-09-2013

We are already running cumulative hotfix 1 and it hasn't helped us. Let me know if it helps your installations. However, given the situation, the article doesn't fit the problem anyway because nothing is indexed at all, even items archived 10 seconds ago. To be sure, I reapplied the hotfix, it is still not working for us.

Have you got any customers with 10.0.3 indexing succesfully? The last time I did an install of v10 it was 10.0.2 and that was a clean installation, indexing was working ok then.

Sarah_Seftel1 · ‎04-09-2013

OK, my customers servers:

First:

Microsoft Windows Server 2008 R2 Enterprise, on two nodes with microsoft failover cluster.

Second:

Microsoft Windows Server 2008 R2 Enterprise, one server, not clustered.

will do the hotfix tomorrow...

Sarah_Seftel1 · ‎04-09-2013

As for your question, these are the only 2 upgrades I did to 10.0.3...

the 10.0.2 ones are ok, only 10.0.3 are not working in indexing issue :(

James_Drake · ‎04-09-2013

Has anyone else had indexing working against EV 10.0.3? I need to resolve this by the end of this week, so whatever it takes, even a complete destruction and rebuild of all indexes.

Sarah_Seftel1 · ‎04-09-2013

James,

Believe me that I need this to be resolved ASAP as well, as one customer is an insurance company and the other is a hospital, and they all need to search what's archived

And yet - my guess is that even if you will try to rebuild indexes, it Will only rebuild the 32bit ones, and will do nothing on 64bit ones.

My systems are archiving, but are not indexing anything since the upgrade.

It is now escalated to engineering, hoping for the best, but realistic, as I faced big issues like that before, and it takes time to point the exact cause

I wish I was able to "downgrade" the system to 10.0.2...

Rob_Wilcox1 · ‎04-09-2013

Very odd. I've not got an 'upgraded' lab, but my straight install of 10.0.3 appears to be indexing things fine.

Working for cloudficient.com

James_Drake · ‎04-09-2013

We actually have a test lab here in which I initially performed a basic upgrade test to mirror the live upgrade. Having managed to get access to it again, I found that the indexer was not upgraded with the cumulative hotfix nor has the operating system had any patches applied. It is Windows 2008 R2 SP1 base release. Only features and software to enable to upgrade to EV 10.0.3 were installed.

The indexing works fine in the lab. This now leads me to believe that the problem is either the cumulative hotfix, a Windows update, or a combination of both. I can upgrade the test lab with the cumulative hotfix and see if it breaks it, but I cannot update Windows as internet access from the lab is virtually impossible.

Sarah_Seftel1 · ‎04-09-2013

a reminder:

I didn't install the cumulative hotfix provided by Symantec yet, so we can narrow it down maybe to a windows update.

I wonder what will happene to the server is I uninstall the updates that you found in common for the 3 servers.

Sarah.

James_Drake · ‎04-09-2013

That would be a good step, however our system is live so unfortunately I cannot perform the uninstall. If you are in a position to do that, then it would be appreciated.

Sarah_Seftel1 · ‎04-09-2013

My systems are live too, but one of the customers is in a cluster mode...

I can uninstall it from the passive node, reboot - and fail over the system to it after reboot.

Will check with my customer tomorrow, at this point - nothing to loose...

Yet · ‎04-10-2013

I'm following this up as I'm planning to upgrade few sites from 9.0.2 to 10.0.3 due to requirement for Outlook 2013 & Windows 8. With all these issues, I'm having 2nd thought now...

I highly appreciate if you post latest update on your saga.

I believe lots of us is looking after this post.

Goodluck.

James_Drake · ‎04-10-2013

As soon as we have updates or a resolution we will update this thread. I too have another customer in the next 4 weeks. In that case it will be a completely new install of EV, but if this isn't resolved by then I will be installing 10.0.2.

VOX

Index errors after upgrade to 10.0.3