NetBackup 7.5.0.5 and Enterprise Vault 9.0.4 Exchange 2010 - Backup Indexes failed

[oleksii_volkov]

Hello

We have NetBackup 7.5.0.5 server on Linux RH (on master and one media server) and Enterprise Vault 9.0.4 for Exchange 2010 in MCS Windows 2008 R2 cluster. Backup of SEV indexes failed with error - 2. All other SEV backups (partitions, FP DB - passed normally). During backup error appeared in SEV application log

7264 - Abnormal error occurred 

Error:     Exception occurred.  (0x80020009) 

Reference: CCache::SetIndexLocationBackupMode 

Index:      

Info:      EM/e 

We've cleared backup mode from all indexes and rerun job - but job still fail. BPresolver log from SEV server in attach.  

Please help.

Best Regards,

Stanislav

[Mark_Solutions]

It does imply that the index was already in backup mode - have you checked for anything else that may be putting it into backup mode - bpstart_notify script or more than one policy with the same directive, scheduled task etc? Take a look at the EV Server to see if you can pin down when it goes in and out of backup mode

[oleksii_volkov]

Hello

I've checked several times. Before INDEX Backup Job - all indexes are on "clean - non backup state". During manual INDEX backup job -  all indexes going to backup state and after job failed - several index remain in backup state, several in "clean - non backup state". No other task are running at the same time on NB.

I've pinned new bpresolver and dtrace.

Best Regards,

Stanislav 

[Mark_Solutions]

Do you have the bpbkar log file from the client - may need high logging level to get what we need but if you have it as it is lets take a look

[oleksii_volkov]

Yes. I have - please see in attach. But from first look - it include info from my night jobs (nothing from INDEX backup - which I've run in ~ 1300)

[Mark_Solutions]

Was that from the EV Server itself? It is the client that runs the backup process that creates its bpbkar log

[oleksii_volkov]

Yes. Its from SEV server - Active Node in MCS cluster.

[Mark_Solutions]

OK - not sure how it did a backup and yet did not write to its bpbkar log file! A bit lost at this point as to where the backup actually ran - try upping the logging level but I cannot see how it ran a backup at 1300 but logged nothing - does it have a bpcd log file?

[oleksii_volkov]

BPCD - yes. In attach. But in bpbkar - nothing about 13.00 task.

[Mark_Solutions]

Was something still running at 1800? bpresolver is still processing things - just wondering if there are orphaned processes causing your issue and keep putting EV into backup mode?

[oleksii_volkov]

at 18.00 I have a job with FINGERPRINT DB backup. Passed normally

[Will_Restore · Accepted solution]

from your original bpresolver log:

12:42:15.815 [6648.7284] <2> DebugLog(): INF - CEVBridgeHelper::ModifyEVEntityBackupModeByEntityID - EVBridge Instance FAILED to update the entity (1777A253AB957E84CBE064C399C20038B1810000AR-SEVCLRS-01) of Site (163CC21C33399F349A5FC8BEB1968520D1d10000AR-SEVCLRS-01). Error: For the Archiving Option, an error occurred with the Enterprise Vault Directory API. Check the Enterprise Vault event log in the Windows Event Viewer for details.!!

13:18:09.481 [8852.5584] <2> DebugLog(): INF - CEVBridgeHelper::ModifyEVEntityBackupModeByEntityID - EVBridge Instance FAILED to update the entity (1777A253AB957E84CBE064C399C20038B1810000AR-SEVCLRS-01) of Site (163CC21C33399F349A5FC8BEB1968520D1d10000AR-SEVCLRS-01). Error: For the Archiving Option, an error occurred with the Enterprise Vault Directory API. Check the Enterprise Vault event log in the Windows Event Viewer for details.!!

13:24:56.154 [8808.9012] <2> DebugLog(): INF - CEVBridgeHelper::ModifyEVEntityBackupModeByEntityID - EVBridge Instance FAILED to update the entity (1777A253AB957E84CBE064C399C20038B1810000AR-SEVCLRS-01) of Site (163CC21C33399F349A5FC8BEB1968520D1d10000AR-SEVCLRS-01). Error: For the Archiving Option, an error occurred with the Enterprise Vault Directory API. Check the Enterprise Vault event log in the Windows Event Viewer for details.!!

So what does Event Viewer have to say?

Each of these errors occurred while trying to set backupmode on Index 8.

Try to set & clear backupmode manually. Is it successful?

[oleksii_volkov]

Hello

Manually set & clear backupmode isn't successful (Index service errot). So we start comparing contents of index 8 folder with query from SQL DB. I've found one extra folder. After we've deleted this folder & restarted indexing service backup going well. Thank you.

Best Regards,

Stanislav

[Mark_Solutions]

wr spotted this in your logs - i think he should really have been marked as the solution as he guided you there Thanks

[oleksii_volkov]

Marked wr message as a solution. Thank you.