Failed to determine whether the item can be deleted. - event id 6283

[mjscreen1]

Getting many occurence of event errors -

eg:
Log Name:      Symantec Enterprise Vault
Source:        Enterprise Vault
Date:          22/12/2011 10:28:14
Event ID:      6283
Task Category: Storage Online
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      xxxxx
Description:
Failed to determine whether the item can be deleted.
201006297846556~201005240820540000~Z~E059A84CB65EE180E2DC3A3C71BD5441

We are running 9.03.1164
Enterprise Vault for Exchange

nothing is building up on queues and no users reports of issues come through

drawing a blank when looking for this on internet
any suggestions - thank you

[LCT]

my questions would be, are you allowing users to delete their archived items from vault? Probably need to enable dtrace and let it run for a few minutes to capture the info. dtrace the Storageonlineopns, archiving task and storagedelete processes. attach the dtrace would great. 

[JesusWept3]

Actually if you're using 9 sp3, use backtrace

[mjscreen1]

Thank you - I have turned on backtrace with an include of 6283

will await detail of the trace - cheers M

[Percy_Vere]

Is storage expiry running? Check your site properties.

[JesusWept3]

OK So looking more in to this, the place where its most likely failing from is either Archive Explorer or Search.asp doing a Delete or a Move To Mailbox (in AE)

Can you check your IIS Logs looking deleteo2k.asp and Moveo2k.asp and check for a status 500?
This will tell you who is attempting to move or delete the item.

So what you want to DTrace is
W3WP and StorageOnlineOpns , thats all you need.

[mjscreen1]

I have turned on the dtrace for StorageOnlineOpns

here is an portion:

.....

83513 10:24:37.410  [5156] (StorageOnlineOpns) <5360> EV:M {CSecurityWrapper::LoadAzStoreCache:#1385} Loading authorization store cache from [C:\Program Files (x86)\Enterprise Vault\EvAzStore_Svc.xml]...
83514 10:24:37.410  [5156] (StorageOnlineOpns) <5360> EV:M {CSecurityWrapper::LoadAzStoreCache:#1399} Waiting for file lock for exclusive access...
83515 10:24:37.410  [5156] (StorageOnlineOpns) <5360> EV:M {CSecurityWrapper::LoadAzStoreCache:#1402} Acquired file lock for exclusive access.
83516 10:24:37.472  [5156] (StorageOnlineOpns) <5360> EV:M {CSecurityWrapper::LoadAzStoreCache:#1437} Role access checks are enabled.
83517 10:24:37.472  [5156] (StorageOnlineOpns) <5360> EV:M CSecurityWrapper::CommonRoleAccessCheck : Com Exception HRESULT 0x800710dd
83518 10:24:37.472  [5156] (StorageOnlineOpns) <5360> EV:M CSecurityWrapper::CheckRoleAccess : Com Exception HRESULT 0x800710dd
83519 10:24:37.472  [5156] (StorageOnlineOpns) <5360> EV:M CClientIdentity::HasRequiredRoleBasedAccess - An error was returned:0x800710DD
83520 10:24:37.472  [5156] (StorageOnlineOpns) <5360> EV:H {CClientIdentity::CheckAccess(vSIDS)} (Exit) Status: [Exception occurred.  (0x80020009)]
83521 10:24:37.472  [5156] (StorageOnlineOpns) <5360> EV:H {CClientIdentity::CheckAccess} (Exit) Status: [Exception occurred.  (0x80020009)]
83522 10:24:37.472  [5156] (StorageOnlineOpns) <5360> EV:H {CStorageOnline::CheckCallersVaultAccess} (Exit) Status: [Exception occurred.  (0x80020009)]
83523 10:24:37.472  [5156] (StorageOnlineOpns) <5360> EV:L {CStore::~CStore} (Entry)
83524 10:24:37.472  [5156] (StorageOnlineOpns) <5360> EV:L {CStore::CloseVault} (Entry)
83525 10:24:37.472  [5156] (StorageOnlineOpns) <5360> EV:L {CVault::~CVault} (Entry)
83526 10:24:37.472  [5156] (StorageOnlineOpns) <5360> EV:H {CVault::~CVault} (Exit) Status: [Exception occurred.  (0x80020009)]
83527 10:24:37.472  [5156] (StorageOnlineOpns) <5360> EV:L {CExclusiveUseThread::EndExclusiveUse} (Entry)
83528 10:24:37.472  [5156] (StorageOnlineOpns) <5360> EV:H {CExclusiveUseThread::EndExclusiveUse} (Exit) Status: [Exception occurred.  (0x80020009)]
83529 10:24:37.472  [5156] (StorageOnlineOpns) <5360> EV:H {CStore::CloseVault} (Exit) Status: [Exception occurred.  (0x80020009)]
83530 10:24:37.472  [5156] (StorageOnlineOpns) <5360> EV:L {CStore::CloseIndexerNotifier} (Entry)
83531 10:24:37.472  [5156] (StorageOnlineOpns) <5360> EV:H {CStore::CloseIndexerNotifier} (Exit) Status: [Exception occurred.  (0x80020009)]
83532 10:24:37.472  [5156] (StorageOnlineOpns) <5360> EV:L {CStore::CloseDatabase} (Entry)
83533 10:24:37.472  [5156] (StorageOnlineOpns) <5360> EV:L SingletonRegistry: US '0X14A7CD8TransactionContext'
83534 10:24:37.472  [5156] (StorageOnlineOpns) <5360> EV:L SingletonRegistry: US '0X14A7CD81'
83535 10:24:37.472  [5156] (StorageOnlineOpns) <5360> EV:L SingletonRegistry: US '0X14A7CD8'
83536 10:24:37.472  [5156] (StorageOnlineOpns) <5360> EV:H {CStore::CloseDatabase} (Exit) Status: [Exception occurred.  (0x80020009)]
83537 10:24:37.472  [5156] (StorageOnlineOpns) <5360> EV:L {CStreamerObjectCache::ReleaseObjectCacheOnThread} (Entry) Thread Id=5360
83538 10:24:37.472  [5156] (StorageOnlineOpns) <5360> EV:L {CStreamerObjectCache::ReleaseObjectCacheOnThread:#277} The streamer and callback object caches for this thread are already empty.
83539 10:24:37.472  [5156] (StorageOnlineOpns) <5360> EV:L {CStreamerObjectCache::ReleaseObjectCacheOnThread} (Exit)
83540 10:24:37.472  [5156] (StorageOnlineOpns) <5360> EV:H {CStore::~CStore} (Exit) Status: [Exception occurred.  (0x80020009)]
83541 10:24:37.472  [5156] (StorageOnlineOpns) <5360> EV:L {CStorageSession::~CStorageSession} (Entry)
83542 10:24:37.472  [5156] (StorageOnlineOpns) <5360> EV:H {CStorageSession::~CStorageSession} (Exit) Status: [Exception occurred.  (0x80020009)]
83543 10:24:37.472  [5156] (StorageOnlineOpns) <5360> EV~E Event ID: 6283 Failed to determine whether the item can be deleted. |201109075357693~201107261045190000~Z~010F653730397FB1DEF886D7C42EC981 |
83544 10:24:37.472  [5156] (StorageOnlineOpns) <5360> EV:H {CStorageOnline::CanItemBeDeleted} (Exit) Status: [Catastrophic failure  (0x8000ffff)]
83545 10:24:37.472  [5156] (StorageOnlineOpns) <5360> EV:L MSTA2 Deallocating busy thread: 5360
......

============

- a quick google has yielded nothing of note  from the above...

any thoughts ?

 

[LCT]

I could be wrong, but it seems that the item(s) no longer exists in storage therefore EV cannot determine whether to delete it or not.

Saveset ID:

201109075357693~201107261045190000~Z~010F653730397FB1DEF886D7C42EC981

Has there been any disaster recovery of your DBs or/and your storage i.e. has data been restored from backup etc.? Is the Event ID reoccuring for the same item every time?

Are you able to confirm that the item exists in storage?

Run a SQL query against your vault store;

select * from saveset

where idtransaction = '010F6537-3039-7FB1-DEF8-86D7C42EC981'

The result should list the IdPartition number and also an IdDateTime, from this you open the PartitionEntry table in the EnterpriseVaultDirectory database and it will tell you which partition path it belongs to from the IdPartition, you can use the IdDateTime to trace the items or do a search for it. If you use Collection then you will have to search within cab/zip file etc.

Alternative you can use EVSVR to verify the item from the partition that it belongs to, this process will take a lot longer than the above.

Hope that helps.

[JesusWept3 · Accepted solution]

The error is from Authorization Manager (Roles Based Administration) via this line
 

83518 10:24:37.472  [5156] (StorageOnlineOpns) <5360> EV:M CSecurityWrapper::CheckRoleAccess : Com Exception HRESULT 0x800710dd

The error it is receiving is

0x800710dd = The Operation Identifier Is Not Valid

If you search on this elsewhere in regards to Enterprise Vault, you see it more commonly with backup scripts via the PowerShell where the user does not have the Backup Admin Role

Do you see this error anywhere in your Event Logs?
 

Event Type: Error
Event Source: Enterprise Vault
Event Category: Directory Service
Event ID: 8591

Description:
Error Launching EvAzUtil utility in order to upgrade the roles-based administration database. Error Code: '7'

You may want to take a look at
http://www.symantec.com/business/support/index?page=content&id=TECH138994

If you don't see that error, then it may be worth doing a reset of the Roles Based Administration DB
To do this, perform the following steps

1. Log on to the Enterprise Vault Server
2. Open up a RegEdit
3. If using a 32-bit (x86) OS then use HKEY_LOCAL_MACHINE\Software\KVS\Enterprise Vault\Admin
4. If using x64 then HKEY_LOCAL_MACHINE\Software\WOW6432Node\KVS\Enterprise Vault\Admin
5. Create a new String called "ResetAuthorizationStore" and give it a value of 1
6. Open the Vault Admin Console
7. Right click "Directory" on the left hand side and you now have a "Reset Authorization Store"

If you do this, you will have to make sure to set up any backup roles and PST roles you may have set in the past

[LCT]

I have seen lots of

Error Launching EvAzUtil utility in order to upgrade the roles-based administration database. Error Code: '7'

they usually occur after an upgrade from EV2007 to EV or/and EV8 to EV9.

Have you done an upgrade to EV9.0.3 recently and then seeing these errors?

I saw the line earlier

EV:M CClientIdentity::HasRequiredRoleBasedAccess - An error was returned:0x800710DD

but I just thought you might have logged in as a user other than the VSA. If you are getting 8591 then I think JW2 is right and you need to resolve that issue first.

[JesusWept3]

Chau,
If you actually look at it it causes the cascading effect
 

[5156] (StorageOnlineOpns) <5360> EV:M CClientIdentity::HasRequiredRoleBasedAccess - An error was returned:0x800710DD
[5156] (StorageOnlineOpns) <5360> EV:H {CClientIdentity::CheckAccess(vSIDS)} (Exit) Status: [Exception occurred.  (0x80020009)]

So the first part it is loads up the AzStore for the RBA, and then gives the "Operation Identifier Is Not Valid"
Which actually suggests that the role its looking to determine, doesn't actually exist.

Then the CheckAccessVSID's that called the the RBA check fails out with the Exception Occured (0x80020009), then that passes back to its caller that Exception occured, and so on and so forth until the originating call fails with that error.

[LCT]

Cool. Thanks for that explanation JW2.

[mjscreen1]

A note to say thank you -

yes - we were seeing:

Event Type: Error
Event Source: Enterprise Vault
Event Category: Directory Service
Event ID: 8591

Description:
Error Launching EvAzUtil utility in order to upgrade the roles-based administration database. Error Code: '7'

so have run through procedure
http://www.symantec.com/business/support/index?page=content&id=TECH138994

have stopped and restarted Enterprise Vault directory service - the event id 8591 does not appear
Will see if this has the effect of stopping the:

Failed to determine whether the item can be deleted. - event id 6283

once again - thank you