Enteprise Vault - Error logs, corrupted data, data recovery - advice required

[AdamBut]

Hi There

I'm completely new to this forum so please bear with me but I really could do with some help and advice.

I'm trying to help out a family friend who is currently involved in a dispute with a business he used to work for. As part of the dispute resolution process the business is required to produce emails between certain date ranges. The business has come back and said that it is not able to provide copies of  the emails etc as the data for the date ranges is 'corrputed' and non-recoverable.

The date ranges in question were from 2 or 3 years ago and at the time the business was using Enterprise Vault for archiving of emails etc. The business was generating circa 35k emails per day and some of the date ranges are over a week so circa 250k emails gone missing at a time!! They apparently have backup tapes stored offsite but are saying that the data on these will also be corrupted.

I am not a techie and this is genuinely new ground for me. I know my way around a computer (just!) but have got completely lost when trying to search through the tech support information and guides regarding Enterprise Vault.

I'm really hoping someone here can help answer some questions, (in layman's terms if possible). Basically what I'm trying to find out is:

• Can data become corruptedc during the archiving process when using EV?
• If data is either corrupted before or during the archiving process does EV generate some kind of fault / error log, (including dates & times)?
• Being as teh data in question relates to information from a few years back is there any way that EV would store such fault logs and could the system be asked to generate a report of dates / times when such data corruption occurred
• Is there any way the data / email info could be recovered?

I'm sure there's a few more questions but I would be really grateful if someone 'in the know' could just provide me with some answers / advice or at least point me in the direction of some relevant links / resources which I could then pass on to someone who's a bit more technically minded.

Really appreciate any help anyone could provide

Thanks

[LCT]

• Can data become corruptedc during the archiving process when using EV?

It is possible, but very small chances

• If data is either corrupted before or during the archiving process does EV generate some kind of fault / error log, (including dates & times)?

If emails are corrupted then EV will not archive those emails, during archiving Ev will throw errors and reject those corrupted emails.

• Being as teh data in question relates to information from a few years back is there any way that EV would store such fault logs and could the system be asked to generate a report of dates / times when such data corruption occurred

If they had configured Enterprise Vault reporting and it had been installed and configured from the beginning the they you could try and running report against date range and

• Is there any way the data / email info could be recovered?

If the backup data are intact then they can be recovered by restoring the ENTIRE EV environment from back up. You will need expert(s) such as a Symantec Solutions Partner to do this.

I would strongly recommend that you or your friend get in contact with Symantec or their Solutions Partner for 'proper' advice.

[AdamBut]

Hi LCT

Thanks so much for your advice, that's really helpful and appreciate your input. Will recommend he get in touch with Symantec.

Thanks again

[LCT]

You're welcome and good luck with the above!!!

[JesusWept3]

Actually your best bet to determine corruption is to run EVSVR in a full verification mode, and then look at the files it comes back with as being corrupt.

Then you can look on your backups for as far back as they go for those specific files, you dont need to restore the entire environment

EVSVR will also give you a general idea of the date/time the items were sent/received and the date they were archived, however if corruption happened after that date in time, EV would never have any knowledge of it.

For instance if you had a virus, or even a virus scanner causing issues (which does happen), nothing in the EV logs would show that.

As for the reporting. the only thing that will hold that information would be the event viewer and dtrace.
DTrace would however need to be running at the time against the specific task or service, and the chances of that are small.

All you would get from reporting is pretty much Server health, but nothing saying "this item was corrupted etc"

And on top of that, Enterprise Vault pretty much archives everything in its original state, so its garbage in, garbage out in some circumstances.

So for instance, if i sent a corrupt Excel spreadsheet, it wouldn't be corrupt in terms of how Exchange, Outlook and EV see's it.... for an item to be corrupt and EV not archive it, there would need to be something physically wrong with the MAPI Messages and the message properties

[AdamBut]

Hi!

Thanks for taking the time to reply, I really appreciate it. Reassuring to know there's people out there who know about this kind of thing!

Cheers

[LCT]

Ooops...I was under the assumptions that the people "The date ranges in question were from 2 or 3 years ago and at the time the business was using Enterprise Vault for archiving of emails etc." meaning they don't run Enterprise vault anymore? hence I suggested restore the entire enterprise vault or I may misunderstood that sentence.

[AdamBut]

I believe the business is still using Enterprise Vault today

[AdamBut]

Hi there guys

Hope you don't mind but just wanted to ask a follow up question regarding this situation.

With regard to the 'missing emails' / corrupted data that the business in question can't find, the part that seems really strange is that the missing data falls into blocks of five days. So for example there is a week of emails in the archive then the following week all emails are missing / "corrupted", then 2 further weeks of all emails then another week of emails missing and so on.

In total there are 40 days of emails missing out of 120 days, (approx 1 million emails) Is there a logical explanation of how this could happen other than data being completely corrupted or deleted or whatever?

My friend is just trying to get his head round it and and get a simple explanation of how this could happen. And if data loss happens on this sort of scale is there any kind of traceability / reporting that EV generates which could be tied to these gaps / losses.

Again, really appreciate any input or thoughts you might have.

Cheers

[JesusWept3 · Accepted solution]

OK so i guess the question is, how are you determining corruption and how are you determining missing items? Are you using EVSVR to determine this?

Also how long ago were these emails archived and when did they notice them become missing or corrupt?
Generally if you are seeing missing or corrupt emails on that wide a scale, it would suggest something programatically going through and deleting or changing them (such as virus scanner)

Again though if a program or utility is going in to the physical data stores and altering or changing the metadata, then there really is no way for Enterprise Vault to know about it or be aware of it, and if it was logging to the point where you could see in the event logs "this item has become corrupt", you would hope that the admins looking after the machine would have seen it and gotten a case with symantec open to stop the issue from occuring on more items

[AdamBut]

Thanks for coming back to me.

The simple answer to the first couple of questions is that I don't know. Basically my frind usd to work for the business in question and there is a dispute, (which I won't go into) and there is are certain files / emails which have evidence to support his dispute.

The business has been asked to provide copies / disclose all emails between end of 2007 and end of 2008. They have apparently gone back through the archives (using Enterprise Vault) and pulled out all the emails between those date ranges.

They have then come back to my friend and said that within that period there are significant gaps (i.e 5 day blocks / weeks) where they have been unable to retrieve ANY emails / files because quote "the data is corrupted". Basically my friend knows the approximate dates of when the relevant emails are that would support his dispute and surprise surprise the the dates of those emails happen to be in the same date ranges where there are the data gaps.

The business is saying that it has only just found out about this corrupt data since it was asked to go back through the archives and search between 2007 and 2008 as part of the disclosure process. So they are climing they have only just been made aware of the missing / corrupt data.

Clearly my friend believes that the data has been deleted but the business would obviously deny that. So, what he knows and wants to know is:

• The business was and still is using EV
• They are saying that it is not possible to retrieve the information
• Is it actually possible to lose such amounts of data completely?
• Has the data been deleted?
• If so, would something like EV record if / when that was done?
• If it is a case of the fact that data was corrupted, can EV at least produce some kind of report which would correlate with the dates of the missing data?
• Is there any way of recovering the lost data?

Just in case you're wondering why I'm getting so involved, the family friend is my father in law and I owe him one :)

Does that help?

[ia01]

Vault store partitions are excluded from Antivirus scan? Which should be excluded if not already in place.

Check if anything from vault store is in Antivirus quarantine folder?

[Dead-Data]

Hi There,

As regards the why and wherefore of 2 weeks being available, then 1 week not, etc. 

This could well be due to the company using some sort of "round robin" system to distributed newly ingressed data across storage devices. 

So quite feasibly, if one of those devices had a problem they were unaware of / had not noticed, then those items might be unindexed / stored incorrectly or not stored at all. 

Obviously not ideal for a company with such volumes of data, but not inconceivable. 

There's plenty of EV consultants around here / out there who could independently assess the health of the system in question.

Whether the nature of the litigation justifies this is down to your friend and their legal advisors to determine. 

Generally a defendant who is unable to produce all requested ediscovery evidence is viewed unfavourably but the courts.

Hope this helps.