EV 10 Problem

[Sox]

Hello to all,

I have major problem with my Enterprise Vault 10, or to be more precise, with backup of my enterprise vault.

We backup our Enterprise vault with HP data protector; SQL and file system, and with version 6 until now.

With that version everything was fine, after backup was done, arhcive bit of backued up files was reseted, Vault knows that and continue to work ok.

Recently, before few days, we upgraded HP data protector to version 8, along with backup clients, and suddenly Enterprise vault start to notify two critical errors from status in EV console:

1. Status 41261 - Check for success of backup and replication of newly-archived data

It is more than 2 days since backup or replication was detected for one or more open partitions

2. Status 41008 - New items awaiting backup or replication

There are 19405 savesets that have not yet been backed up or replicated for Vault Store 'Name of store'.

Also, our clients says that they cannot arhive mails.

Sounds like to me, that EV no longer knows when HP do backups.

On HP backup server, script that making backup say to EV server to put vault partitions out of backup mode when backup is done, but EV still hold partitions in backup mode.

Any help would be appreciated.

This is really major problem so I'm asking for solution... If there's any.

[TonySterling]

You REALLY need to get with the backup team to figure out why the backup scripts are not be called correctly to remove EV from backup mode.

[Sox]

Since we cannot put vault stores and index places into backup mode via standard backup scripts beacuse of involvement of vss service, can we configure additional post scripts after every backup that will put vault stores and index places in backup mode?

[A_J1]

Hello Sox,

I would suggest first to Isolate the issue is with the scripts(i.e run by your Backup Application) or with Enterprise Vault.

To confirm everything is ok with Enterprise Vault.

- Go the Vault Admin console and right click on Site Node and set backupmode on all vaultstore.

- Refer the below screenshot

- Once successfully done, Clear enterprise Vault stores from backup mode.

- Confirm both operations were successfully.

Now once confirmed from the above steps that the issue is not with EV.

As you asked you can use the following script for Set-EV-BackupMode and Clear-EV-Backupmode for Vaultstores and index locations.

SET BackupMode

C:\Windows\SysWOW64\WindowsPowerShell\v1.0\PowerShell -psconsolefile "C:\Program Files (x86)\Enterprise Vault\EVShell.psc1" -command "& {Set-VaultStoreBackupMode -Name 'Name of the Vaultstore Group' -EVServerName evserver1 -EVObjectType VaultStoreGroup}"

C:\Windows\SysWOW64\WindowsPowerShell\v1.0\PowerShell -psconsolefile "C:\Program Files (x86)\Enterprise Vault\EVShell.psc1" -command "& {Set-IndexLocationBackupMode -EVServerName evserver1 -EVSiteName 'EV Lab'}"

CLEAR BackupMode

C:\Windows\SysWOW64\WindowsPowerShell\v1.0\PowerShell -psconsolefile "C:\Program Files (x86)\Enterprise Vault\EVShell.psc1" -command "& {Clear-VaultStoreBackupMode -Name 'Name of the Vaultstore Group' -EVServerName evserver1 -EVObjectType VaultStoreGroup}"

C:\Windows\SysWOW64\WindowsPowerShell\v1.0\PowerShell -psconsolefile "C:\Program Files (x86)\Enterprise Vault\EVShell.psc1" -command "& {Clear-IndexLocationBackupMode -EVServerName evserver1 -EVSiteName 'EV Lab'}"

You can use the above PowerShell commands to create a batch file and schedule the Set-Backupmode before the Backup Schedule and Clear-Backupmode script after the backup is complete.

Note: In case you have an issue with Enterprise Vault unable to set/clear backup mode then we need to troubleshoot..

I hope this helps !!!

Regards,

A_J

[Sox]

Thank you A_J...

I was thinking to do just like that.

Thank You very much on Your answer.

I will keep you posted...

[A_J1]

My pleasure Sox,

You can also refer the below article on generating powershell commands for you environment.

Generating PowerShell backup commands for your environment

http://www.symantec.com/docs/HOWTO37806

Download the Generating PowerShell Backup Commands guide below:

http://www.symantec.com/docs/TECH75694

[Sox]

Hello A_J,

I put my vault store partitions and index locations out of backup mode, but I still have those two errors:

Even after I re-run all checks?

Is it indicates that backup was not passed correctly?

Beacuse I'm using backup software that obviusely doesn't reset archive bit after backup, can I somehow start using trigger files?

[Sox]

Sorry if picture is not uploaded:

Errors are:

41261 - Check for success of backup and replication of newly-archived data (Source: Enterprise Vault)

41008 - New items awaiting backup or replication (Source: Exchange Store)

[A_J1 · Accepted solution]

Hello Sox,

So we can say that the issue is not with Enterprise Vault

You asked: Beacuse I'm using backup software that obviusely doesn't reset archive bit after backup, can I somehow start using trigger files?

==> Make sure trigger file is selected on all partition

Your script should be as follows.

SET BackupMode

C:\Windows\SysWOW64\WindowsPowerShell\v1.0\PowerShell -psconsolefile "C:\Program Files (x86)\Enterprise Vault\EVShell.psc1" -command "& {Set-VaultStoreBackupMode -Name 'Name of the Vaultstore Group' -EVServerName evserver1 -EVObjectType VaultStoreGroup}"

C:\Windows\SysWOW64\WindowsPowerShell\v1.0\PowerShell -psconsolefile "C:\Program Files (x86)\Enterprise Vault\EVShell.psc1" -command "& {Set-IndexLocationBackupMode -EVServerName evserver1 -EVSiteName 'EV Lab'}"

del "C:\EVStorage\Express Vault Store\Express Vault Store Ptn1\ignorearchivebittrigger.old"

Note: You have to specify the above command for all of your partition

(The above statement in bold will deleted the "ignorearchivebittrigger.old" file from the partition)

CLEAR BackupMode

C:\Windows\SysWOW64\WindowsPowerShell\v1.0\PowerShell -psconsolefile "C:\Program Files (x86)\Enterprise Vault\EVShell.psc1" -command "& {Clear-VaultStoreBackupMode -Name 'Name of the Vaultstore Group' -EVServerName evserver1 -EVObjectType VaultStoreGroup}"

C:\Windows\SysWOW64\WindowsPowerShell\v1.0\PowerShell -psconsolefile "C:\Program Files (x86)\Enterprise Vault\EVShell.psc1" -command "& {Clear-IndexLocationBackupMode -EVServerName evserver1 -EVSiteName 'EV Lab'}"

ECHO > "E:\Enterprise Vault Stores\MailArchiving Ptn1\ignorearchivebittrigger.txt"

Note: You have to specify the above command for all of your partition

(The above statement in bold will create the "ignorearchivebittrigger.txt" file in the partition)

You can use the above PowerShell commands to create a batch file and schedule the Set-Backupmode before the Backup Schedule and Clear-Backupmode script after the backup is complete.

I hope this helps !!!

[Sox]

Thank You A_J for being so helpful...

Ofcourse, I will schedule those two bat scritps and see if it will be ok.

Kindest regards

[Sox]

We've setup those pre and post scripts, and I will let you know tomorow how it goes...

Thank You again A_J.

[A_J1]

My Pleasure Sox...

[A_J1]

Hello Sox,

Did everything worked ok ??

[Sox]

Hello A_J,

I was just checking situation.

Must tell You that scritps worked perfectly, but only with one small problem...

After all process my store was still in backup mode, and after I manually put it out, and do re-check, everything is green :) (finally)

I will check with my backup guys maybe, if they can remodify script so that tomorow we don't get that error again..

And I will let You know.

Thank You very much...

You were really really helpful...

[Sox]

I have just one last question:

My guys told me that in scripts log they can see this errors:

C:\Windows\system32>C:\Windows\SysWow64\WindowsPowerShell\v1.0\powershell -psconsolefile "C:\Program Files (x86)\Enterprise Vault\EVShell.psc1" -command "& {Set-VaultStoreBackupMode -Name 'Name of the Vaultstore Group' -EVServerName evserver1 -EVObjectType VaultStoreGroup}"

Set-VaultStoreBackupMode : Could not contact Enterprise Vault server 'evserver1

'. Check that this server is available and try again.

At line:1 char:28

+ & {Set-VaultStoreBackupMode <<<<  -Name 'Name of the Vaultstore Group' -EVSer

verName evserver1 -EVObjectType VaultStoreGroup}

    + CategoryInfo          : ResourceUnavailable: (Symantec.Enterp...StoreBac

   kUpMode:SetVaultStoreBackUpMode) [Set-VaultStoreBackupMode], COMException

    + FullyQualifiedErrorId : STORAGE_E_ERROR_CONNECTING_STORAGE_SERVER,Symant

   ec.EnterpriseVault.PowerShell.Commands.SetVaultStoreBackUpMode

Set-VaultStoreBackupMode : Unable to get site id

At line:1 char:28

+ & {Set-VaultStoreBackupMode <<<<  -Name 'Name of the Vaultstore Group' -EVSer

verName evserver1 -EVObjectType VaultStoreGroup}

    + CategoryInfo          : ObjectNotFound: (Symantec.Enterp...StoreBackUpMo

   de:SetVaultStoreBackUpMode) [Set-VaultStoreBackupMode], COMException

    + FullyQualifiedErrorId : STORAGE_E_UNKNOWN_ERROR,Symantec.EnterpriseVault

   .PowerShell.Commands.SetVaultStoreBackUpMode

C:\Windows\system32>C:\Windows\SysWow64\WindowsPowerShell\v1.0\powershell -psconsolefile "C:\Program Files (x86)\Enterprise Vault\EVShell.psc1" -command "& {Set-IndexLocationBackupMode -EVServerName evserver1 -EVSiteName 'EVSite'}"

Set-IndexLocationBackupMode : Failed to read indexlocation info, error:The RPC

server is unavailable. (Exception from HRESULT: 0x800706BA)

At line:1 char:31

+ & {Set-IndexLocationBackupMode <<<<  -EVServerName evserver1 -EVSiteName 'EVS

ite'}

    + CategoryInfo          : NotSpecified: (:) [Set-IndexLocationBackupMode],

    DirectoryServiceReadRecordException

    + FullyQualifiedErrorId : KVS.EnterpriseVault.Diagnostics.DirectoryService

   ReadRecordException,Symantec.EnterpriseVault.PowerShell.Commands.SetIndexL 

  ocationBackUpMode

I prosume that they didn't enter valid EV server name in script. Am I right?

[A_J1]

My Pleasure Sox,

You can run the Set-backup script manually on the EV Server.

Then run the Clear-Backup script and check if it clears the backupmode, If not then we need to find out what the problem.

[A_J1]

Yes, Sox.

Follwing parameters you need to pass which are bold in the in the following script.

C:\Windows\SysWow64\WindowsPowerShell\v1.0\powershell -psconsolefile "C:\Program Files (x86)\Enterprise Vault\EVShell.psc1" -command "& {Set-VaultStoreBackupMode -Name 'Name of the Vaultstore Group' -EVServerName 'name of the evserver' -EVObjectType VaultStoreGroup}"

And for Index location you need to pass the following parameters.

C:\Windows\system32>C:\Windows\SysWow64\WindowsPowerShell\v1.0\powershell -psconsolefile "C:\Program Files (x86)\Enterprise Vault\EVShell.psc1" -command "& {Set-IndexLocationBackupMode -EVServerName 'EV Server Name' -EVSiteName 'EVSiteName'}"

You can refer to the below article in generating powershell commands for you EV Environment.

Generating PowerShell backup commands for your environment

http://www.symantec.com/docs/HOWTO37806

Download the Generating PowerShell Backup Commands guide below:

http://www.symantec.com/docs/TECH75694

I hope this helps !!!

[Sox]

This helps a lot...

We are making progress thanks to You... :)

Ok... I will direct my backup guys to fix scripts and we will wait until tomorow morning to see if it was ok.

Again, thank You very much...

I will keep You posted.

[Sox]

Hello A_J,

Just to inform You that we solved problem with scripts that You provide us, but we made a little modifications regarding our infrastructure.

Now, backup works, as well as putting EV to backup mode, and out of backup mode.

Thank You again,

Wish You the best...

Kindest regards,

SOX