Over 100Tb of PST files

[maudib]

We have purchased a company and they have a 13-14 months worth of full backups.  I’ve been tasked with converting it all to .pst files and ingesting it into Enterprise Vault.  We have the Ontrack Extract tools to extract the data convert it to .pst. 

The problem is we have 107 Tb to ingest I’ve calculated that it will be 10 years to do based on the speed of ingesting over the past several months.

I was thinking that because the files probably don’t change much from day to day, that I could simply merge/de-dupe the pst and then ingest it.  But I haven’t been able to find a good scriptable tool to use.   

They had about 3000 users so I don’t want to manually merge the files. 

Does anyone know of scriptable pst merge/de-dupe tool?  Or is there an easier way to accomplish this?

Thanks 

[JesusWept3 · Accepted solution]

I think you should look at something like PSTFlightDeck by GlassHouse or talk to Symantec Professional services, i mean thats one hell of a project, and there are lots of factors to consider really

are you looking to import the folder structure of the PST files? if so you would need to archive it to a Mailbox Archive, but you'd need actual archives to exist.

What do the messages in the PST files look like? there are tools that do imports that generate MS Journal Envelopes, and the only way to have them process correctly is via the JournalTask, if you use the PST Import Wizard or some such, you will have the P1 message intact which could severely impact searching (The envelope would be treated as the email, and the actual email would be treated as an attachment etc)

 

[AndrewB]

Hi Maudib, another product to consider might be TransVault. Like the others, it will parse/process all your PSTs and migrate to EV with all the usual benefits around speed, auditing, selective filtering, etc. It's able to push the limits of the EV API and achieve speeds much greater than PST import wizard.

If you're interested, please feel free to PM me and I can arrange for more information.

[mashles]

You can also run multiple instances of the PST wizard simultaneously, a single instance is only using 1 import thread even if you queue multiple PST files. On a server dedicated to doing the PST imports (if its going to a vault store also on that server and dedicated to the PST import archives) then 5 threads is reasonable.

You could also think about setting up multiple temporary import servers each with a vault store to import the data, then just switch those vault stores over to your other EV servers (process for this varies depending on the storage type) once the imports are complete. That way you can run say 2-3 servers each doing 5 threads, giving you at least 20-30gb/hour, and probably more with decent spec servers.

Theres still quite a bit of overhead on doing it like this though with manual imports, so if you can get the OK to use an 3rd party tool that will probably be easier.

[Rob_Wilcox1]

I think you'd have to write a tool to do that, or pay someone to do it... it's not an easy task at all.

[maudib]

Thanks for the advice, I've contacted Symantec Professional Services on the 'how to'.  We do have a couple of TransVault servers for another project so those may come in useful again. 

[FreKac2]

Would be interesting to hear how it goes once you've started.

[BarneyHaye]

maudib,

Advice may be coming a bit late, but if you have TransVault Servers on site, ask your migration partner to get some eval licences for PST to EV so you can test things out for yourself.  Our experience is that you could expect at least 10 GB per hour per Vault Store, but a partner in France is currently doing an EV to EV migration for a financial institution and reporting an aggregated 86 GB per hour!  Your mileage will undoubtedly vary, but pulling data from PSTs is one of the fastest connections for TransVault.

These figures refer to uncompressed data.

Barney

TVS