EV11.0.1 event 13404 A Schema inconsistency error was detected while accessing the Vault Database 'EnterpriseVaultDirectory'

[GertjanA]

Hello,

We upgraded to EV11.0.1 last saturday. I just had this event on all EV-servers. there is no column RetainArchivehistory anymore.

Anyone running EV11.0.1 seen this too perhaps?

Log Name:      Symantec Enterprise Vault
Source:        Enterprise Vault
Date:          2/18/2015 12:25:41 PM
Event ID:      13404
Task Category: Directory Service
Level:         Error
Keywords:      Classic
User:          N/A
Computer:     servername
Description:
A Schema inconsistency error was detected while accessing the Vault Database 'EnterpriseVaultDirectory' (Internal reference: {CADODataAccess::ExecuteSQLCommand} [.\ADODataAccess.cpp, lines {1317,1319,1338,1370}, built Nov 19 15:44:06 2014]):

Description: 

Invalid column name 'RetainArchiveHistory'.
 
 
SQL Command:
 SELECT SiteEntryId,SiteName,DefaultFilterId,DefaultPublicFolderFilterId,DefaultRetentionCategoryId,DefaultWebAppURL,DefaultArchivingSchedule,DefaultIndexingLevel,SiteDescription,MessageOfTheDay,AutoEnableMailboxes,AutoEnableInitSuspend,SiteAdminNote,AdminMinVersion,UsersCanDeleteItems,DeleteExpiredItems,DeleteProtected,DefaultPublicFolderSchedule,DefaultExpirySchedule,DesktopDoubleClickView,ExpiryDateBasis,ExpiryReportMode,SiteAccountDistinguishedName,SiteAccountMailboxAlias,SiteDirectoryEntryId,SiteVaultLimitSize,SiteVaultLimitOperation,IndexMode,AllClientsAtSiteVersion,DefaultPFRetentionCategoryId,SPSExcludeFileTypes,SPSItemSize,SiteVaultNotifySize,SiteVaultNotifyOperation,QuotaNotificationFreq,CreateStructuredArchives,EnableCodeMonitoring,PSTHoldingDirectory,PSTHoldingLimit,DumpsterEnabled,DumpsterPeriod,RetainArchiveHistory,DaysBeforeDeletingIndexingSubTasks,IndexSnippetLength,IndexAttachmentSnippet,EnableFacetedResults,ServicePort,OnlyUpdateRetIfLonger FROM view_site ORDER BY SiteName
 
 
Additional Microsoft supplied information:
 
Source:       Microsoft OLE DB Provider for SQL Server
Number:       0x80040e14
SQL State:    42S22
Native Error: 00000207
HRESULT  0x80040e14

[JesusWept3]

thats quite impressive!
Whats the schema version?
Wonder if running the database upgrader would run again against it?

[GabeV]

Hi GertjanA,

It seems like something went wrong with the database upgrade. Run the EVDatabaseUpgrader.exe located in the Enterprise Vault installation directory. Also, don't forget to backup all the Enterprise Vault databases before running this tool.

Running Database Upgrader
http://www.symantec.com/docs/HOWTO109498

The Enterprise Vault Database Upgrader tool is used to update the Enterprise Vault directory database, and the vault store and vault store group databases to the latest schema version. Running the tool once updates all the databases across all Enterprise Vault servers in the directory.

The databases are updated in the following order:

    1. Directory database
    2. Vault store databases (processed in parallel)
    3. Vault store group databases (processed in parallel)

The tool generates a report file. By default, this is saved to the following location:
    C:\Program Files (x86)\Enterprise Vault\Reports\

Before you run the tool, stop Enterprise Vault services on all Enterprise Vault servers, and ensure there are no active connections to the databases, for example Microsoft SQL Server Management Studio.

On successful completion of the tool the Enterprise Vault services can then be started.

The tool should be run with elevated command prompt privileges, and only one instance of the tool should be run at a time across all Enterprise Vault servers.

The tool does not require any command line parameters.

See the Upgrade Instructions for full details.

[GertjanA]

@Gabe, I'll try that tomorrow.

We're already running for 3 days (sunday, monday, tuesday), so could the upgrade have gone wrong? I have opened a case, but the engineer said he did not see any collation issues....

When back in office tomorrow, I'll verify. Be advised this error only occurred today for the first time, and was logged 5 times in a row. Until 17:00 CET, no new appearance.

It is only doing a select, but I find it a little scary.

I know that the Retain Archive History setting has moved somewhere else.

I assume I cannot just target the ' directory db' using the upgrader? I am not sure I can shutdown EV during the day.

@JW how would I get the schema version?

[GertjanA]

I've done some checking on the logfiles of the upgrader, and on the eventlogs on the servers. I also verified the Schema versions.

Schema version:
Directory DB = 11.48.0.160.0
VaultStores = 11.32.9.45.0

which seems to be correct according to Sym documentation.

We ran DBupgrader, it failed first time on 1 store, and thus on 1 FP database.

Second time same.

We than ran Storage service on server hosting specified VS database, worked fine. VS and VSG updated correct. Below snips from the logfiles and registry.

Could it be that this is a select coming from a report which was not correctly upgraded? I'll wait for same time today, see what happens.

SNIPPETS:

[GertjanA]

removed snippets, had company info.

[GertjanA]

Anonymized snippets:

D:\>cd "Program Files (x86)"

D:\Program Files (x86)>cd "Enterprise Vault"

D:\Program Files (x86)\Enterprise Vault>EVDatabaseUpgrader.exe
[10:49:44.684] Enterprise Vault database upgrader tool started
[10:49:45.793] Started database upgrade. Database:[EnterpriseVaultDirectory] Server:[SQLSERVER\INSTANCE] InitialSchemaVersion:[10.125.5.137.2].
[10:50:24.016] Completed database upgrade. Database:[EnterpriseVaultDirectory] Server:[SQLSERVER\INSTANCE] UpgradedSchemaVersion:[11.48.0.160.0] TimeElapsed:[00:00:37].
[10:50:24.110] Starting update of [6] vault store databases
[10:50:24.219] EXCEPTION: An error occurred attempting to perform a security check.Failed to check whether user [DOMAIN\VSA] is DBOwner. Could not connect to SQL database: [VS1] at SQL Server: [SQLSERVER\INSTANCE]
[10:50:24.329] Started database upgrade. Database:[VS2] Server:[SQLSERVER\INSTANCE] InitialSchemaVersion:[10.33.5.37.1].
[10:50:24.329] Started database upgrade. Database:[VS3] Server:[SQLSERVER\INSTANCE] InitialSchemaVersion:[10.33.5.37.1].
[10:50:24.329] Started database upgrade. Database:[VS4] Server:[SQLSERVER\INSTANCE] InitialSchemaVersion:[10.33.5.37.1].
[10:50:24.329] Started database upgrade. Database:[VS5] Server:[SQLSERVER\INSTANCE] InitialSchemaVersion:[10.33.5.37.1].
[10:50:24.344] Started database upgrade. Database:[VS6] Server:[SQLSERVER\INSTANCE] InitialSchemaVersion:[10.33.5.37.1].
[10:50:34.287] Completed database upgrade. Database:[VS4]Server:[SQLSERVER\INSTANCE] UpgradedSchemaVersion:[11.32.9.45.0] TimeElapsed:[00:00:09].
[10:51:40.320] Completed database upgrade. Database:[VS6] Server:[SQLSERVER\INSTANCE] UpgradedSchemaVersion:[11.32.9.45.0] TimeElapsed:[00:01:15].
[10:52:25.289] Completed database upgrade. Database:[VS3] Server:[SQLSERVER\INSTANCE] UpgradedSchemaVersion:[11.32.9.45.0]TimeElapsed:[00:02:00].
[10:52:49.706] Completed database upgrade. Database:[VS5]Server:[SQLSERVER\INSTANCE] UpgradedSchemaVersion:[11.32.9.45.0] TimeElapsed:[00:02:25].
[10:55:50.897] Completed database upgrade. Database:[VS2] Server:[SQLSERVER\INSTANCE] UpgradedSchemaVersion:[11.32.9.45.0] TimeElapsed:[00:05:26].
[10:55:50.897] It took [00:05:26] to successfully process [5] of [6] vault store databases[10:55:50.897] Failed to update [1] vault store databases:
[10:55:50.897]     Database: [VS7] SQL server: [SQLSERVER\INSTANCE] State: [Failed_PermissionsError]
[10:55:50.960] Vault store group database [VSG1][SQLSERVER\INSTANCE] will not be updated due to failed Vault store updates:
[10:55:50.960]     Vault store database: [VS7] SQL server: [SQLSERVER\INSTANCE] State: [Failed_PermissionsError]
[10:55:50.975] Starting update of [2] vault store group databases
[10:55:51.178] Started database upgrade. Database:[VSG2] Server:[SQLSERVER\INSTANCE] InitialSchemaVersion:[9.5.0.2.0].
[10:55:51.178] Started database upgrade. Database:[VSG3]Server:[SQLSERVER\INSTANCE] InitialSchemaVersion:[9.5.0.2.0].
[10:56:03.215] Completed database upgrade. Database:[VSG2] Server:[SQLSERVER\INSTANCE] UpgradedSchemaVersion:[9.10.4.4.0] TimeElapsed:[00:00:12].
[10:56:07.305] Completed database upgrade. Database:[VSG3] Server:[SQLSERVER\INSTANCE] UpgradedSchemaVersion:[9.10.4.4.0] TimeElapsed:[00:00:16].
[10:56:07.305] It took [00:00:16] to successfully process [2] of [2] vault store group databases
[10:56:07.337] One or more database updates failed.

Successfully updated:
  Directory database
  [5] Vault store databases
  [2] Vault store group databases

Failed update:
  [1] Vault store databases:
    Name: VS7
    SQL server: SQLSERVER\INSTANCE
    Update state: Failed_PermissionsError

  [1] Vault store group databases:
    Name: VSG1
    SQL server: SQLSERVER\INSTANCE
    Update state: Failed_Blocked

Refer to previous event log errors for details.
[10:56:07.337] Time elapsed: [00:06:22]

===========
2ND ATTEMPT:
===========

D:\Program Files (x86)\Enterprise Vault>EVDatabaseUpgrader.exe
[11:12:33.363] Enterprise Vault database upgrader tool started
[11:12:33.504] [EnterpriseVaultDirectory] Current database version up to date [11.48.0.160.0]
[11:12:33.582] Starting update of [6] vault store databases
[11:12:33.660] EXCEPTION: An error occurred attempting to perform a security check.Failed to check whether user [DOMAIN\VSA] is DBOwner. Could not connect to SQL database: [VS1] at SQL Server: [SQLSERVER\INSTANCE]
[11:12:33.675] [VS5] Current database version up to date [11.32.9.45.0]
[11:12:33.675] [VS6] Current database version up to date [11.32.9.45.0]
[11:12:33.675] [VS2] Current database version up to date [11.32.9.45.0]
[11:12:33.675] [VS3] Current database version up to date [11.32.9.45.0]
[11:12:33.691] [VS4] Current database version up to date [11.32.9.45.0]
[11:12:33.691] It took [00:00:00] to successfully process [5] of [6] vault store databases
[11:12:33.691] Failed to update [1] vault store databases:
[11:12:33.691]     Database: [VS7] SQL server: [SQLSERVER\INSTANCE] State: [Failed_PermissionsError]
[11:12:33.722] Vault store group database [VSG1][SQLSERVER\INSTANCE] will not be updated due to failed Vault store updates:
[11:12:33.722]     Vault store database: [VS7] SQL server: [SQLSERVER\INSTANCE] State: [Failed_PermissionsError]
[11:12:33.769] Starting update of [2] vault store group databases
[11:12:33.831] [VSG2] Current database version up to date [9.10.4.4.0]
[11:12:33.831] [VSG3] Current database version up to date [9.10.4.4.0]
[11:12:33.847] It took [00:00:00] to successfully process [2] of [2] vault store group databases
[11:12:33.862] One or more database updates failed.

Update not required:
  Directory database
  [5] Vault store databases
  [2] Vault store group databases

Failed update:
  [1] Vault store databases:
    Name: VS7
    SQL server: SQLSERVER\INSTANCE
    Update state: Failed_PermissionsError

  [1] Vault store group databases:
    Name: VSG1
    SQL server: SQLSERVER\INSTANCE
    Update state: Failed_Blocked

Refer to previous event log errors for details.
[11:12:33.862] Time elapsed: [00:00:00]

[GertjanA · Accepted solution]

Hmm. This might have been a false alarm. The error has not resurfaced until today.

Closing this entry.