cancel
Showing results for 
Search instead for 
Did you mean: 

Deleting Journal Entries before a specific date

Not applicable

Hi

 

Before I looked after EV the installer setup retention periods of everything retained forever. This policy applied to both archives and the journal.

Now some three years on the business is discussing changing this. We have been asked to reduce the journal to 12 months. That's OK because i have created a new Journal policy of 12 months.

The problem is the old Journal entries. I can't modify the old retention policy because it applies to both the Journal and Archives. No one has yet decided on the retention period for the archives but it will be different from the Journal.

So I want to delete ALL Journal entries before 1st January 2009. I cannot change the retention policy at this time. Is there a way to do this or a way to retrospectively alter the retention policy for that data.

 

Current EV version 9.0.

 

Looking forward to some suggestions.

20 REPLIES 20

Liam_Finn1
Level 6
Employee Accredited Certified

You could turn off the setting on the Archives except the Journal that deletes expired items then modify the present retention category to 12 months and run the expiry.

 

This will delete the items from the Journal but leave the data in the other archives

 

The setting  is in the VAC under Archives

 

Select the archives and under the advanced tab in the archive you have a tick box under storage expiry. Un-select this tick box and apply.

 

Do this for any archive you wish to not be deleted.

 

Next modify your retention category to 12 months and do a manual run of storage expiry

 

Remember to set the retention back to keep forever before you chance the setting back on the archives to delete

 

Mohawk_Marvin
Level 6
Partner

What about a little SQL hacking? That way you could set the journals to your newly created retention category.

 

You could change all savesets associated with that Archive to have a new retention category?

 

Something like

Update Saveset

set RetentionCategory = '4'

Where VaultIdentity = '1'

 

So to get the new retention category ID open up the Directory DB and look under the RetentionCategoryEntry table, there is a RetentionCategoryIdentity column.

 

for the VaultIdentity open up the Vault Store DB, and under the Vault table, the column VaultID is the one with the Archive name, so make sure you get the correct ID, you can copy that line out and check the advanced tab of the archive to make sure it correlates?

 

I owuld suggest you conduct some tests in your own labs, and have a lot of back ups of the DBs prior to trying this.

 

There is no way SYMC are going to support this mind you :)

MichelZ
Level 6
Partner Accredited Certified

I'd advice you against that, as the retention category and stuff is not only saved in SQL, but in the saveset on disk, too.

This can lead to a great deal of confusion once you have to use EVSVR and stuff like that, as the entries are then gonna be mismatched.

 

Use at your own risk...

 

Scanner001's idea seems to be a "nicer" solution, because that would be entirely supported as far as I can tell. (However, you should still backup your environment before you do anything like this)  laugh


cloudficient - EV Migration, creators of EVComplete.

Liam_Finn1
Level 6
Employee Accredited Certified

This is very dangerous and if it goes wrong you are on your own.

 

If you are going to do something like this get Symantec Professional Services to work with you to ensure it goes as hoped

 

Changing a retention category in a vault retroactively is presently not supported

Not applicable

 

Yes I can see that. However I would have to change all 8,466 archives individually for this method to work. That would take a while.

 

Anything less cumbersome?

 

Steve

Liam_Finn1
Level 6
Employee Accredited Certified

You can change this in SQL if you prefer

 

In the VaultstoreDirectory database in the table VaultStoreEntry you need to modify the DeleteExpiredItems from 1 to 0

MichelZ
Level 6
Partner Accredited Certified

Ask support if you can do that in bulk in SQL. (should be a no-brainer for them)

 

Cheers


cloudficient - EV Migration, creators of EVComplete.

Not applicable

If I limited the size of the JOURNAL below the current size would it delete the earliest entries?

 

Steve

MichelZ
Level 6
Partner Accredited Certified

I don't think so, it would just stop archiving. (not sure if you are able to limit a journal archive anyway..)


cloudficient - EV Migration, creators of EVComplete.

Liam_Finn1
Level 6
Employee Accredited Certified

It would not delete. The only way to preform a delete is through storage expiry or manually selecting items out of Archive Explorer

Liam_Finn1
Level 6
Employee Accredited Certified

Something like this can do the update for you

USE EnterpriseVaultDirectory

UPDATE VaultStoreEntry

SET DeleteExpiredItems = '0'

WHERE DeleteExpiredItems ='1'

 

Please test it in your lab before using it in a live environment

Liam_Finn1
Level 6
Employee Accredited Certified

Just another note.

 

Once you run this it will set all to do not delete.

 

After this is run you should, using the VAC, manually set the journal archives to delete so their data will be erased according to the expiry process.

Not applicable

Thanks for the ideas. It's obvious that there is not an easy way of changing retention periods on archived items.

This could be a nice feature for the future.

Meanwhile I can see that I am going to have to approach this VERY carefully.

 

Steve

Liam_Finn1
Level 6
Employee Accredited Certified

Not sure if you found a solution but if you have please mark the appropriate post as solved

 

If we can be of any further help please let us know

Not applicable

I was having a look at PST exporting the Journal and set it to delete. Unfortunately the utility wont use dates and delete! <agggh!!>

 

Doing global changes to retention leaves me nervous. We dont have a backup. Well we do but it's really relying on Centera replication and I cant backup 11TB of data before I try some of solutions here.

 

What EV needs is a way of correcting retention policy already in place or shall I say the flexibility to change things after installation. There are too many settings that seem to have huge problems when you want to alter them later. Retention being one of them. I havwe one policy for archives and Journal and now the journal policy for the organisation has changed. I can start a new policy now but can't backwards apply it to the whole Journal. I understand all this legal compliance stuff but sometimes you just need functionality.

 

I am still looking for a workable solution.

 

Steve

Not applicable

 

Nope. Changes the SQL fine. The users still have a tciked b ox saying 'expire'.

 

Dont have a "lab" to test this I am afraid. Does anyone! In the UK public sector there is barely enough money to by production kit!

 

Steve

JesusWept3
Level 6
Partner Accredited Certified

Once you change the SQL close the VAC and restart the Directory service as its all cached anyway, but to be honest, you may want to look at consultancy if your situation is this precarious.

Like Michele said, if you change the Retention categories in the SQL Database, if you ever do a repair, it will set all the retentions back based on whats stored in the saveset.

But really i think Liam has hit the nail right on the head straight off

1. Change all the archives except for the journal archives to Not Expire items
2. Change the main retention category to 12 months
3. Run Expiry and delete all the old journal items
4. Make sure that you correct the situation by having multiple retention categories applied to multiple things such as 1 retention for archiving and one for journaling which is more suitable
5. change the users to allow for expiry

It's straight forward and simple and is really the correct way to do things

However there is one thing that i'm not sure of based on what you've written thus far, but is this Centera is compliance mode? because if it is, and you've set it to hold the items forever and it has a time to live of 9999 years, then forget it, the items aren't going anywhere

https://www.linkedin.com/in/alex-allen-turl-07370146

Not applicable

In a sense the real issue is that one retention policy was established for the whole site. We need to un-glue the Journal from the regular archive.

In the office we have been throwing around ideas and what we may try is building a new site completely with one server and moving the journal to it. We can then impose site wide policy that only effects the Journal. It sounds bizarre but it might get us out of the jam.

I think there is a fundamental problem with EV insofar as it tries to make it difficult to delete material in the archive. This is quite right from a legal and regulatory standpoint but it's absolutely annoying from a storage mangement perspective where you want to move things around or where policies change - as in our case. I think EV has to think more about the rigid nature of the choices you make at install and maybe devise a more flexible approach.

 

Steve

JesusWept3
Level 6
Partner Accredited Certified

Well to be fair to Enterprise Vault you have all the freedom in the world to set things up before archiving a single item, and as far as journaling goes, you can do custom filters and other third party add ons to journal items with different retention categories depending on who received the message or if it has a kind of attachment or keywords etc etc etc

It's why i'd advise anyone who is setting up EV to make sure they have their needs and future needs clearly mapped out and bring in consultancy as a misconfiguration at the beginning, especially with a compliance device such as a Centera, can cause major headaches in the future.

The idea though of seperating out the sites to me is a little extreme not only because of the sharing of items, but because you can potentially break compliance, for instance you cannot import a PST file through the VAC to a journal mailbox, you have to do it through EVPM.

However if you import the PST files, you lose everything that the journal task does, for instance Journaling will go through and expand distribution lists and what not to determine who receieved that email, but with a PST import it will only store the Distribution List

So for instance lets say i send an email to DL-IT-OPS, once its sent journaling will do the following

1. Open the email and expand out DL-IT-OPS
2. It finds that its been sent to UserA, UserB, UserC and UserD
3. It then adds that to the saveset and to the index
4. If i search for an email receieved by UserB, this email will show up

If I was to import the PST in to the archive and i search for email received by UserB, it will not show this email as being received, but if i search DL-IT-OPS then it will show that that distribution list got it, but not it's members

Also if i do have things like custom journaling and filters, this will be negated by a PST Import.

So the solution then sounds like you should import the PST files in to a journal mailbox and let the journal task take care of it. however you can then end up getting false positives

For instance lets say
2009 - DL-IT-OPS has UserA, UserB, UserC and UserD
2010 - DL-IT-OPS has UserA, USerB, UserD, and UserE

You then get one false negative and one false positive
Because EV will take the message sent to DL-IT-OPS, expand the distribution list and it says the recipients are A, B, D and E

So if you search for emails receieved by UserE, it will show this message, even though when it was sent, the user was never a part of the Distribution List

If you search for emails receieved by UserC, it will not show the message, even though the user did receieve it, they are no longer part of the distribution group so won't be registered as such.


Another thing also is that when you export items to a PST file from a journal, it no longer has the envelope message, which contains a list of all the recipients that got the email, including anyone who has been BCC'd

So what Enterprise Vault will do is see that the envelope shows it was sent to
UserA
UserB
UserC
UserD

It then opens up the main message and see's that it was sent to
UserA
UserB
UserC

This means that UserD is now an "Undisclosed recipient" and is essentially a BCC recipient, which is useful if you want to find out if anyone has been sending email "secretly" to someone who should not be getting this.


I really think scanners original idea of changing the retention, changing people not to expire, then expire the items and change the retention category going forward is the simplest and easiest solution that you have.

But again, if your centera is in compliance none of this makes a difference

https://www.linkedin.com/in/alex-allen-turl-07370146