Clear-IndexLocationBackUpMode : Could not contact Enterprise Vault server

[AKL]

Hello

I am running Enterprise vault 9.0.1 environment and have configured a Powershell script that controls Backup mode for all vault stores and index locations from Directory Server. There are total of 5 Enterprise vault servers in same site.

Issue is that the script is able to put everything in backup mode except Index location of one of Enterprise vault servers.  So all vault stores and all index location mode can be put into backup mode except one. Below is error received:

Clear-IndexLocationBackUpMode : Could not contact Enterprise Vault server 'EV2
'. Check that this server is available and try again.
At G:\scripts\Post-Backup-A.ps1:30 char:30
+ Clear-IndexLocationBackupMode <<<<  -EVServerName EV2
    + CategoryInfo          : ResourceUnavailable: (Symantec.Enterp...ationBac
   kUpMode:ClearIndexLocationBackUpMode) [Clear-IndexLocationBackUpMode], COM
  Exception
    + FullyQualifiedErrorId : STORAGE_E_ERROR_CONNECTING_STORAGE_SERVER,Symant
   ec.EnterpriseVault.PowerShell.Commands.ClearIndexLocationBackUpMode
Clear-IndexLocationBackUpMode : Failed to read indexing service info
At G:\scripts\Post-Backup-A.ps1:30 char:30
+ Clear-IndexLocationBackupMode <<<<  -EVServerName EV2
    + CategoryInfo          : ObjectNotFound: (Symantec.Enterp...ationBackUpMo
   de:ClearIndexLocationBackUpMode) [Clear-IndexLocationBackUpMode], Director
  yServiceReadRecordException
    + FullyQualifiedErrorId : DirectoryRecordNotFound,Symantec.EnterpriseVault
   .PowerShell.Commands.ClearIndexLocationBackUpMode

I have tried IISreset on EV2 server and when I run same command locally from EV2 - it completes fine. It was working fine a while ago - trying to understand what changed which could have broke this.

[GertjanA]

Did you apply any Windows patches to this system?

Check if WMI is being blocked, or perhaps a firewall setting. If the script runs on the local machine, try running the specific command from another machine for this non-working machine.

Can you query the service (sc query \\servername\anyservicename) and get results?

[LCT]

can you manually set and clear the backup mode for the index location on the affected EV server via the admin console or running the script on the local affected machine? As Gertjan pointed out, it seems that the powershell script is executed on the remote machine and perhaps a firewall is blocking it....check the firewall settings on the affected EV server.

Hope that helps.

[AKL]

Thanks guys but I can confirm it is not firewall because:

1. Script is run from server EV1 - EV1 and EV2 are both on same LAN.

2. Windows firewall was off on both servers.

3. I stopped and disabled Windows firewall service on both servers with no luck.

4. It is able to put vault store managed by storage service of EV2 into backup mode - but not index location.

Windows patches are same on all EV servers - so I am not sure about that one either - I can dig into it but there are just lot of patches to go through for same, and again point 4 holds valid for this one as well, if it was patches - I believe vault store wont have gone into backup mode as well right?

[LCT]

check the SiteID registry on your EV server2 from this technote?

http://www.symantec.com/business/support/index?page=content&id=TECH63099

i know the problem is not the same but worth a check as this tecnote resolved a very similar issue to yours. Also, do you have problem puttung ev ino backup mode or clearing? as the error from your problem descrition says it has problem clearing? can you confirm if manually putting the index location and clearing works on that machine?

hope that helps.

[AKL]

Hello CTEV

Issue is with both side - either putting index location in backup mode OR clearing out backup mode. And its confiend to index locations only, vault store behaves as it should.

Also I checked the SiteID registry as suggested - it is same on all five EV servers. Thanks for technote though, can be useful somewhere down the road :)

[Nathan_Clark_2]

ok lets see what the issue is just dtrace Powershell when running the cmdlet.

[AKL]

Attached is the complete DTRace logs for Powershell while running the same script couple times using bat and PS1 both.

Below are some lines i have been focusing on:

548 12:18:01.061  [6388] (StorageFileWatch) <6060> EV~I Event ID: 7103 Watch file partition scan restarted for all the partitions on a VaultStore as it is exiting the backup mode. | Vault Store Name: ReprocessTest | Vault Store EntryID: 14ABDABB2D58FF5478E2620D9991A5CBF1210000EVSITE1 |
549 12:18:01.108  [6388] (StorageFileWatch) <3040> EV~E Event ID: 6842 A Collector encountered an error. |Reason: The system cannot find the path specified.  [0x80070003] |PartitionEntryID: 14C18BE743D84F24A9E7C409094F2A8371q10000EVSITE1 |Method: CCollectorEMCCentera::CheckTemporaryFiles |
550 12:18:01.108  [7588] (StorageDelete) <804> EV~I Event ID: 4230 Request thread pool (InstanceId = [1]) stopped. Total threads stopped = [8]. |
551 12:18:01.108  [6388] (StorageFileWatch) <3040> EV~E Event ID: 6842 A Collector encountered an error. |Reason: The system cannot find the path specified.  [0x80070003] |PartitionEntryID: 14C18BE743D84F24A9E7C409094F2A8371q10000EVSITE1 |Method: CCollectorEMCCentera::HouseKeepTemporaryFiles |
552 12:18:07.217  [7588] (StorageDelete) <7188> EV~I Event ID: 7050 StorageDelete threads restarted successfully. |
553 12:18:07.217  [7588] (StorageDelete) <7428> EV~I Event ID: 4229 Request thread pool (InstanceId = [1]) started with maximum thread count = [8]. |
554 12:18:07.233  [9036] (StorageManagement) <7156> EV~I Event ID: 7079 Successfully cleared Backup Mode from Vault Store. |Vault Store Name: ReprocessTest |Vault Store EntryID: 14ABDABB2D58FF5478E2620D9991A5CBF1210000EVSITE1 |
 

634 12:18:07.952  [3684] (powershell) <9312> EV:M VaultCoCreateInstanceEx: IsLocalMachineExtendedCheck returned false for server [EV2]  as the IP Address matched that of the local machine
635 12:18:07.968  [3684] (powershell) <9312> EV:L CDCOMSecurityHelper::SetClientBlanket Authentication Level=6 Impersonation Level=2 Capabilities=32 HRESULT=Success  [0]
636 12:18:08.077  [3684] (powershell) <Pipeline Execution Thread:9312> EV-H {IndexLocationBackupModeTasks.ProcessRecord} Exception: Failed to read indexing service info Info: Diag: Type:KVS.EnterpriseVault.Diagnostics.DirectoryServiceReadRecordException ST:   at Symantec.EnterpriseVault.PowerShell.Commands.IndexLocationBackupModeTasks.LoadIndexingServicesByName(EVObjectType evObjectType)|   at Symantec.EnterpriseVault.PowerShell.Commands.IndexLocationBackupModeTasks.ProcessRecord()
637 12:18:08.077  [3684] (powershell) <Pipeline Execution Thread:9312> EV-L {ClearIndexLocationBackUpMode} Completed
638 12:18:08.077  [3684] (powershell) <Pipeline Execution Thread:9312> EV-H {IndexLocationBackupModeTasks.EndProcessing} Completed with errors
639 12:18:08.077  [3684] (powershell) <Pipeline Execution Thread:9312> EV-L {ClearIndexLocationBackUpMode} enter
640 12:18:08.108  [3684] (powershell) <9312> EV:M VaultCoCreateInstanceEx: IsLocalMachineExtendedCheck returned false for server [EV4]  as the IP Address matched that of the local machine
641 12:18:08.108  [3684] (powershell) <9312> EV:L CDCOMSecurityHelper::SetClientBlanket Authentication Level=6 Impersonation Level=2 Capabilities=32 HRESULT=Success  [0]

Can this issue be related to the virtual machines running on same VM host?

[Nathan_Clark_2]

its failing to create a COM object running in the Admin service on the remote server with acess denied:

Unable to get site id Info: Diag:HRESULT: 80070005 Type:System.Runtime.InteropServices.COMException ST

I assume this VSA account has access on the other server, check DCOM perms etc, also check the system log on EV1 for further detail.

[AKL]

Basically all 5 enterprise vault servers are identical with Vault service account having identical full access on all of them.

I didnt see any error/warning thrown into System logs when I run the script - I ran it 6 times to confirm same.

Should I be looking somewhere in particular? (Sorry if I sound like a rookie...)

[Paul_Grimshaw]

First I think you should narrow down the issue.

Can you run the command locally on EV2 and see if it works or not. If that does not even work then you can say the issue is specifically on the box. If it does work then that points to some sort of occurance over the wires. Does Get-IndexLocationBackupMode work? I suspect you get the same problem.

I can reproduce your exact problem by disabling DCOM on the EV server so if you are saying that all permissions are correct then check DCOM communications between the server you are running the command from and EV2. There is a test app you can use between servers:-

http://support.microsoft.com/kb/259011

[AKL]

Thanks for response Paul.

Can you run the command locally on EV2 and see if it works or not - It works fine, right now this is the only way am able to clear index location on that server for now.

Does Get-IndexLocationBackupMode work? - Nope, your suspect was right - this doesn't works either.

check DCOM communications between the server, Below are results from DCOM Test:

From EV1:

tstsrvr.exe
Server: Waiting
Server: IClassFactory:CreateInstance
Server: IStream:Read
Server: IStream:Write
Server: Done

From EV2:

TstClnt.exe EV1
Client: Creating Instance...0.2045 seconds
Client: Reading data...0.1588 seconds
Client: Writing data...0.0080 seconds
Client: Done

[Nathan_Clark_2]

ok can you run this obviously changing REMOTESERVER for you remote server:

PS 32-bit Powershell running as VSA prompt> [Activator]::CreateInstance([Type]::GetTypeFromCLSID([Guid]'{0002DF01-0000-0000-C000-000000000046}',"REMOTESERVER"))

should work

and then this:

PS 32-bit Powershell running as VSA prompt> [Activator]::CreateInstance([Type]::GetTypeFromCLSID([Guid]'{2D198DCA-C1F8-49F2-9A7E-169016F562F7}',"REMOTESERVER"))

This should fail.

If so check perms as per:

http://www.stuffthatjustworks.com/SearchView.aspx?q=UnauthorizedAccessException

For the 'Enterprise Vault Admin Service' DCOM application.

[AKL]

Sorry - but above didnt work as expected.

[Activator]::CreateInstance([Type]::GetTypeFromCLSID([Guid]'{0002DF01-0000-0000-C000-000000000046}',"EV2"))  - Returned Internet Explorer Com Object

[Activator]::CreateInstance([Type]::GetTypeFromCLSID([Guid]'{2D198DCA-C1F8-49F2-9A7E-169016F562F7}',"EV2")) - Didn't fail - Just returned System Com Object - I checked, there is no CLID used in this cmdlet on EV2 server so that's why it couldnt return the result.

I changed the permissions as per article - without any luck - things are still same.

Also I do not see any error in any category of event log after running the Get-IndexLocationBackupMode cmdlet - Any means - searched all categories of event logs for any error - None.

[Nathan_Clark_2 · Accepted solution]

AKL rather than continuing troubleshooting this on this thread, I have sent you a mail.