Evault indexes consuming lot of disk space

[jaykayk]

I am finding that my Evault indexes are taking up a lot of space. They are up to 45GB at the moment.
I have seen a kb relating to moving the indexes to another drive, but that's just sweeping it under the carpet-I'll have the same problem on the the new drive.
Ideally I would like to truncate the indexes if possible, or reduce their size in some way.

Note that my indexing level is set to medium, which should take up 8% of the size of the vault stores. My stores take up 290GB, which should only translate to 23GB.
I wonder if a re-index would help?

[TonySterling]

Are you sure the indexes have always been set to medium?  Also, how are you getting your Vault Store size?  Indexes are based on original file size and does not take Single Insancing in to consideration so you will need to use the original file size to calculate how big the indexes should be.

To answer your question, there is no way to truncate the indexes nor would you want to.  You could run Storage Expiry to delete older archived items. 

[jaykayk]

The index level has always been set to medium yes. I got the store size using treesize and explorer.

Some background history:

the drive that holds the stores used to have the block size incorrectly set to 64k. We changed it to 8k a few weeks ago, and so our stores took up 300GB, instead of about 500GB. Would this affect the indexes? And what's the best way to resolve it?

[Michael_Bilsbor]

yes block size would certainly have an affect since it means files take up greater amount of space than they really should.  Check the EV performance guide as to the ideal block size (I can't remember off the top of my head) but obvioulsy you are still way way way over what you'd expect to have and that simply cannot be explained by block size.

have you checked with the directories to see something else hasn't been placed there, perhaps it used to have a vault store in there or perhaps there are some old very large trace files?

Mike

[TonySterling]

It would help to know the original file size, not just size on disk, to see how far off you are.

What version of EV are you and what modules are you archiving?  Exchange, Filesystem, etc..

Do you have Reporting installed?

[GertjanA]

Hi jaykayk,

Do you by any change do journal archiving, for legal/security other reasons? I have seen EV-users tend to leave that out of scope, as it does not affect users directly. Most installations I have seen run full indexing on the journalarchiving..

Can you verify that?

[WiTSend]

Actually 45GB for indexes with a VS size of 290 is within a normal range.  Remember that the 8% number is of the original data, not the archived data.  I would move my indexes to a storage platfor with a 4k sector size since the indexes are made up of a large number of fairly small file and you'll have fewer partial sectors that way.

[JesusWept3]

Just to add on to what Tony said, not only is it original Sizes but the size of the physical DVS files don't really relate to how big the indexes will be.

For instance say you have a 10MB word document sent to 1000 people.
EV Stores this as 1 x 10MB DVSSP file, 1 x 10MB DVSCC file and 1000 x 3KB DVS files.

Without sharing, you'd be looking at 10GB of file usage.
With Sharing you'd be looking at at 23MB (1 x DVSSP, 1 x DVSCCC and 1000 x DVS files)

So OSIS has saved you 9.98GB of files.
However, when you index it, you are looking at each of those 1000 users has to add it to index.

So Brief is 3%, Medium is 8% and Full is 13%
a 10MB email then becomes

3% of 10MB is 307KB
8% of 10MB is 819KB
12% of 10MB is 1.19MB

so if we then have 1000 users all indexing that same item it would be

1000 x 3% indexing = 300MB
1000 x 8% indexing = 799MB
1000 x 12% indexing = 1.16GB

So in this scenario, we have 23MB of shared files from 1 email, and then 799MB of indexes

[Andy_Joyce_VERI]

>>For instance say you have a 10MB word document sent to 1000 people.
>>EV Stores this as 1 x 10MB DVSSP file, 1 x 10MB DVSCC file and 1000 x 3KB DVS files.

This is not quite correct.  You would generally assume the DVSSP file is compressed to some degree; if it's Office 2007 upwards then it won't be much because those files are already natively compressed, but earlier versions of Word you'd expect about 50% compression.  Text files compress more, images etc. less.

The DVSCC file is usually about 5% of the size of DVSSP file - again mileage will vary.  This is true for Word docs but if your file is an image or an audio file, for example, then obviously there is less indexible content to store.

And DVS files are an average of 16KB for Exchange and Domino messages, but less for FSA and SharePoint (about 4KB).

A

[JesusWept3 · Accepted solution]

apologies, you are correct.
if you have a 10MB Word Doc that compresses down from 10MB to 5MB, and then the DVSCC being 5% of the DVSSP so thats 256KB, and then 16k per shortcut

You are looking at
1 x 5MB DVSSP, 1 x 256KB DVSCC, and then 1000 x 16KB DVS files

Then that is 20.7MB instead of the 23MB.

The DVSCC can become bigger if its converting to html,
For instance if you have something with say 7 spaces, in html this is represented as  
So seven spaces becomes '       ' which equates to 35B more than the original 7 spaces.

Also you could have in your email something such as a 10MB ZIP file, which when ZLIB attempts to compress, you will either get no compression ratio or a larger item, but the INSO converters will still index the contents of, so then you get to the point where you could have 10MB DVSSP files from a 10MB attachment.

[jaykayk]

Thanks for all the replies guys. I have never had this many prompt replies on a dedicated support forum.

I do have exchange journalling enabled. How do I check if indexing is enabled on the journalarchiving? However, we are having issues with journalling recently, where there's a backlog of mail in the Exchange journal mailbox. A reboot of the E vault server usually fixes it.

My exchange version is 2007 SP3. Evault is 8.0.3.1845. Am I missing any patches that are recommended? I would like to patch, rather than pay to upgrade to version 9.0 or 10.0.
 

[jaykayk]

On second thoughts, it looks like an upgrade to version 9.0 makes sense. We are moving to Exchange 2010 in a few months.

Is there an upgrade path from 8.0 SP3 to 9.0?

[JesusWept3]

Yup, you can move directly from any SP of EV8 to any SP of EV9 no problems
As for indexing on journal, it will be there automatically, i don't think any mailbox archiving supports defered indexing like FSA does

As for the backlog, thats a seperate issue, it can depend alot on storage as well, most commonly i see backlogs more with centeras than anything else, most other backlogs are usually caused by exchange connectivity issues, like seeing a lot of 3305 and 3310's in the event log saying the task is going to sleep because of one reason or another

[JesusWept3]

you'd have to contact your sales person, honestly though, i don't think that EV is one of those products you really want to lose support for, especially since very few hotfixes and services will be released for EV8 as the concentration is mainly for EV9 and EV10 now

[jaykayk]

Thanks JesusWept, A restart of all the Evault services always fixes the backlog issue. I'm not seeing any errors on my exchange box.

I can see from my licensing portal that I can upgrade my 8.0 license to 9.0. Where do I download the actual software from?

[AndrewB]

http://fileconnect.symantec.com

choose your languange and then plug in your serial number from your license (slf) file