IndexCommit = 0 produces items for 1 user

[Sani_B]

Hi,

Originally I started to investigate a problem items stuck in "waiting for bu".

So I ended up checking all kinds of things and also ran the query to see the indexing situation. I got the main problem solved but now nothing in the VAC is alarming about anything etc but there are Items as a result of this query:

SELECT A.ArchiveName "Archive Name",

CE.ComputerName "Index Server",

(IRP.IndexRootPath + '\' + IV.FolderName) "Folder Path",

COUNT(JA.IndexCommited) "Items Awaiting Index",

IV.Failed "Index Failed",

IV.ReadOnly "Index ReadOnly",

IV.Offline "Index Offline",

IV.Rebuilding "Index Rebuilding",

IV.WorkPending "Work Pending"

FROM EnterpriseVaultDirectory.dbo.Archive A,

EnterpriseVaultDirectory.dbo.Root R,

EnterpriseVaultDirectory.dbo.IndexVolume IV,

EnterpriseVaultDirectory.dbo.IndexRootPathEntry IRP,

EnterpriseVaultDirectory.dbo.IndexingServiceEntry ISE,

EnterpriseVaultDirectory.dbo.ComputerEntry CE,

EVVSVaultStore_1.dbo.ArchivePoint AP,

EVVSVaultStore_1.dbo.JournalArchive JA

WHERE JA.ArchivePointIdentity = AP.ArchivePointIdentity

AND AP.ArchivePointId = R.VaultEntryId

AND R.RootIdentity = A.RootIdentity

AND R.RootIdentity = IV.RootIdentity

AND IV.IndexRootPathEntryId = IRP.IndexRootPathEntryId

AND IRP.IndexServiceEntryId = ISE.ServiceEntryId

AND ISE.ComputerEntryId = CE.ComputerEntryId

AND JA.IndexCommited = 0

GROUP BY A.ArchiveName, CE.ComputerName,

(IRP.IndexRootPath + '\' + IV.FolderName),

IV.Failed, IV.ReadOnly, IV.Offline,

IV.Rebuilding, IV.WorkPending

ORDER BY "Items Awaiting Index" DESC

Archive Name	Index Server	Folder Path	Items Awaiting Index	Index Failed	Index ReadOnly	Index Offline	Index Rebuilding	Work Pending
Username	EVarchive.company.com	E:\Index\index2\190F47D836DDB1840A29C361F9B782BD9_1502	1283	0	0	0	0	1

 I have verified, synchronized and eventually even rebuilt the index of this user and still this remains. And the number is groving so it looks like the newly archived ites also go to this queue. How do I force this to work and not be pending?

The ev version is 10.0.4 / OS win server 2012 R2 standard

As I said, from the console it looks like the index is fine etc but this tells a different story... any suggestions?

Sani B.

[GertjanA]

Hi Sani,

You are aware that 2012R2 standard is not (yet) supported for 10.0.4? compatibility guide shows P(ending).

Anyway.

I think you have 2 options.

1 - Note down the exact indexlocation (foldername). Goto Indexing, start a rebuild of that specific archive's index. When the rebuild is started, stop it again. Then move the content of the indexlocation to somewhere else. Restart the rebuild. See if that fixes it.

or:

2 - Disable the user for archiving. Export the archive to PST. Remove the shortcuts from the mailbox. Zap the mailbox. Run provisioning. Sync the mailbox. Enable the user manually again, create a NEW archive. Import the PST into the new archive with creating the shortcuts etc.

When ok, wait a few days, delete the old archive.

[Sani_B]

Actually I wasn't aware of that! I didn't put this EV server together just managing it now... Thanks!

I'll see if your suggestions work - thank you Mr. G.

Sani B.

[Sani_B]

Update:

Suggested method 1 does not solve the problem.

[Ben_Watts]

I wouldnt worry about option 2 as that is using a hammer to crack a nut, you may need to go down that route but best to find out why those items are not Indexing.

I would check to see what the vlaue is for 'Invalid' column for the items in the JA table for that Archive, see if they are set to 1 or 0.

If your backend storage is Centera, or WORM, then we would be trying, but failing, to delete the item from storage and then from the table if the Invalid column is set to 1.

If all the items are set to 0 in the Invalid column I would find out the TransactionId of the earliest item in the table for that archive, do a dumpsaveset of it and find out if it is a bit of a strange item or not.

Whats the IndexSeqNum on those items, do we have any items that are higher IndexSeqNums in the Archive already? (I would assume not as there appears to be 1 item that is failing to Index and the IndexService appears to be hung up on, creating a block for the other items in there)

[Sani_B]

There's 1283 items and the IndexSeqNum seem to be growing with each item... There is no column "invalid" in the table JounalArchive - perhaps somewhere else? And what would that be?

[Sani_B]

WatcFile does have "itemInvalid" column but watch file is empty.

[Ben_Watts]

Apologies, link the two and see if the items in JA have 1 in the WF.ItemInvalid column.

[Ben_Watts]

Ah ok, in that case follow my other suggestions above.

[Sani_B]

no the WF is completely empty so no invalid items either...

[Ben_Watts]

If you run the below, its your query but modified slightly, run it now, then wait for 10 - 15 mins and run it again, if the IndexSeqNo does not change then you know nothing is getting Indexed for this Volume.

SELECT TOP 10
A.ArchiveName "Archive Name",
CE.ComputerName "Index Server",
(IRP.IndexRootPath + '\' + IV.FolderName) "Folder Path",
IndexSeqNo,
TransactionID,
IV.Failed "Index Failed",
IV.ReadOnly "Index ReadOnly",
IV.Offline "Index Offline",
IV.Rebuilding "Index Rebuilding",
IV.WorkPending "Work Pending"

FROM EnterpriseVaultDirectory.dbo.Archive A,
EnterpriseVaultDirectory.dbo.Root R,
EnterpriseVaultDirectory.dbo.IndexVolume IV,
EnterpriseVaultDirectory.dbo.IndexRootPathEntry IRP,
EnterpriseVaultDirectory.dbo.IndexingServiceEntry ISE,
EnterpriseVaultDirectory.dbo.ComputerEntry CE,
ArchivePoint AP,
JournalArchive JA

WHERE JA.ArchivePointIdentity = AP.ArchivePointIdentity
AND AP.ArchivePointId = R.VaultEntryId
AND R.RootIdentity = A.RootIdentity
AND R.RootIdentity = IV.RootIdentity
AND IV.IndexRootPathEntryId = IRP.IndexRootPathEntryId
AND IRP.IndexServiceEntryId = ISE.ServiceEntryId
AND ISE.ComputerEntryId = CE.ComputerEntryId
AND JA.IndexCommited = 0
AND (IRP.IndexRootPath + '\' + IV.FolderName) = 'E:\Index\index2\190F47D836DDB1840A29C361F9B782BD9_1502'

GROUP BY JA.IndexSeqNo, 
TransactionID,
A.ArchiveName, CE.ComputerName,
(IRP.IndexRootPath + '\' + IV.FolderName),
IV.Failed, IV.ReadOnly, IV.Offline,
IV.Rebuilding, IV.WorkPending

ORDER BY IndexSeqNo ASC

If the ISN is not changing, do a DS against the TransactionID for the very first item and see if it dumps successfully.

[Sani_B]

Numbers not changing so have to continue tomorrow and try the DS.

Thanks Ben! You've been very helpful :)

[Ben_Watts]

Not a problem, that is what this forum is great for, there are a lot of people more than willing to help out.

[Sani_B]

Saw the DS and it looks like an ordinary msg, nothing suspicious etc. Now what...? Because of the items pending/waiting to be indexed - it looks like the archive cannot be searched at all... it says "The Vault that you are searching is being updated"

[Sani_B]

Actually scratch that - can be searched but it does complain about being updated and obviously it "didn't find anything" as the item I tried to find is not indexed....

[Ben_Watts]

Ok, small hammer to crack a nut but think you are at that time now.

I would:-

1) Stop and clear any Indexing Jobs for this Archive

2) Stop the EV Indexing Service

3) Find all Index Volumes for this archive that are causing issues and rename/delete them

4) Go to the indexingmetadata folder location, go into the Search-Collections folder and run a search for the first part of the Index Folder name, for example:-  190F47D836DDB1840A29C361F9B782BD9  -  Rename/Delete any and all folders that return with a result

5) Find the Repository.xml file in the IndexMetaData location and rename/delete this

6) Restart the EV Indexing Service

EV will recreate the Repository.xml, it will also to a search for all Index Volumes that need updating and will automatically update/sync them.

As you have then removed any and all possible corruption locations for that index volume it should resync everything ok, if not then it is something with the Items themselves and then we are onto Dtrace to find out why we are not furthering the Indexing.

Of course you could do it the other way around, Dtrace first and investigate and then the above options if the Dtrace points to it, which is what I would do if it was a case with us, but the top option above will more than likely be the quicker of the two options to a fix. If Dtrace points out something else wrong with the item we are attempting to Index then we can manually tell EV the item has been indexed and see if we continue with the items below it.

[Sani_B]

I'll do the dtrace first - what components should I trace for this?

[Ben_Watts]

EVIndexAdminService and EVIndexVolumesProcessor.

Stop Indexing Service.

Start Dtrace with those processes above and log to a file

Open the log file for Dtrace whilst it is still running and do a search for the Index Volume in question.

Keep the Dtrace going until you get a hit for your search, then once you get a hit leave Dtrace for another 5 mins.

Stop Dtrace and have a look over the log, it should be attempting to sync the volume and pull back the next ISN in the JA table, follow that thread to see what is happening with that item and why we arent indexing it correctly and moving onto the next one.

[Sani_B]

Ok took the dtrace - don't know how to read it and can't really post it here cause there's too much of specific client info to block... But it does say the index of this user quite often in the log...

And the "small hammer" ended up to right where it started, same index range and Journal Archive still has those same items waiting for the indexing...

What would happen if I simply delete those items from the sql and then rebuild the index...?

[Ben_Watts]

You would PROBABLY be in the same situaton again as EV would clear out the items from the Index, then attempt to recall and reindex them again, if we are having issues indexing the items now it would be the same at that point.

Would you be happy uploading it to a box account, I can have a quick look over if for you?

If you are happy with Box then use the below link, it only allows UPLOADING and does not allow downloading, the only person able to view it will be myself:-

https://symantec.box.com/signup/collablink/d_3912795497/11dbbbba2e6095

If you arent happy with the above then I would recommend opening a case with us and we can work through it with you.