Whilst there is quite a large section in the Setting up Exchange Server Archiving document for EV 8, I think there is still some pieces missing from it (at the moment). One of the important areas I've looked into recently is how to write a custom filter to assign a different retention category to mails with a particular message class.
At first sight this isn't achievable. But in fact it is. This article explains how to do it.
Registry setup for filtering
This particular part is documented in the Setting up Exchange Server Archiving document, but to reiterate you have to create the following :
Essentially thats a key called "External Filtering" another key called "Mailbox" (because we're going to do our manipulation with the mailbox archiving), then a string value. The name of the value is 1, and the value itself is EnterpriseVault.CustomFilter
Files needed
Next you need to put the following in C:\Program Files\Enterprise Vault\Custom Filter Rules :
Default Filter Rules.xml
<?xml version="1.0"?>
<RULE_SET xmlns="x-schema:ruleset schema.xdr">
<RULE NAME="MBX DIFF_RET_CAT" ACTION="ARCHIVE_ITEM" CONTENTCATEGORY="MsgClassTest" RETENTION="60Day">
<NAMEDPROP TAG="MSGCLASS" INCLUDES="ANY">
<PROP VALUE="IPM.Note.Microsoft.Voicemail.UM.Ca" />
</NAMEDPROP>
</RULE>
</RULE_SET>
Custom Properties.xml
<?xml version="1.0" encoding="UTF-16"?>
<CUSTOMPROPERTYMETADATA xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="customproperties.xsd">
<CONTENTCATEGORIES DEFAULT="MsgClassTest">
<CONTENTCATEGORY NAME="MsgClassTest">
<INDEXEDPROPERTIES>
<PROPERTY TAG="MSGCLASS"/>
</INDEXEDPROPERTIES>
</CONTENTCATEGORY>
</CONTENTCATEGORIES>
<CUSTOMPROPERTIES>
<NAMESPACE TYPE="MAPI">
<PROPERTY TAG="MSGCLASS" NAME="0x001A" />
</NAMESPACE>
</CUSTOMPROPERTIES>
<PRESENTATION>
<APPLICATION NAME="search.asp" LOCALE="1033">
<FIELDGROUPS>
<FIELDGROUP LABEL="Content Category">
<FIELD TAG="MSGCLASS" LABEL="Message Class" CATEGORY="MsgClassTest"/>
</FIELDGROUP>
</FIELDGROUPS>
<AVAILABLECATEGORIES>
<AVAILABLECATEGORY CONTENTCATEGORY="MsgClassTest" LABEL="Message Class Test"/>
</AVAILABLECATEGORIES>
</APPLICATION>
</PRESENTATION>
</CUSTOMPROPERTYMETADATA>
Pecularities
The files need to be unicode !
Fortunately, for once (!), the message class testing isn't case sensitive, so IPM.Note.Microsoft.VoIcEmAiL.UM.CA will match the rule.
In action
To test this out what I did was sent a message to my test user, and then using Outlook Spy I edited the message class to be IPM.Note.Microsoft.Voicemail.UM.CA. I also sent a second test message, and left the message classs alone.
Now I do a run now of the archiving task, and wait a few minutes.
You can then search for the items using Browser Search. Note you'll have an extra bit in browser search, like my screenshot below :
If you change the Retention Category drop down, and select your 60 day policy you'll see the test message was archived with that retention category. Or just do a blanket search, and look at the returned results to see the following :
The proof is in the DTRACE
You're looking for a couple of different things here. First of all, does the filter load correctly?
1167927 06:23:38.027 [6860] (ArchiveTask) <17472> EV~I Event ID: 45329 External Filter 'EnterpriseVault.CustomFilter' initialising... |
1167950 06:23:38.308 [6860] (ArchiveTask) <17472> EV-M {CustomPropertiesDefinition} Loading Custom Properties from file:\C:\PROGRAM FILES\ENTERPRISE VAULT\Custom Filter Rules\Custom Properties.xml
1167951 06:23:38.308 [6860] (ArchiveTask) <17472> EV-L {CustomPropertiesDefinition} Loading Custom Property definitions...
1167952 06:23:38.324 [6860] (ArchiveTask) <17472> EV-L {CustomPropertiesDefinition} Adding property MSGCLASS [namespace=]
1167953 06:23:38.324 [6860] (ArchiveTask) <17472> EV-L {CustomPropertiesDefinition} Adding content categories...
1167954 06:23:38.324 [6860] (ArchiveTask) <17472> EV-L {CustomPropertiesDefinition} Adding category MsgClassTest
1167955 06:23:38.324 [6860] (ArchiveTask) <17472> EV-L {CustomPropertiesDefinition} Default Category = MsgClassTest
1167956 06:23:38.339 [6860] (ArchiveTask) <17472> EV-L {CustomPropertiesDefinition} Adding presentation applications...
1167957 06:23:38.339 [6860] (ArchiveTask) <17472> EV-L {CustomPropertiesDefinition} Adding application search.asp (Locale='1033')
1167958 06:23:38.339 [6860] (ArchiveTask) <17472> EV:M [CustomXMLFilter] Setting DEFAULT Content Category to [MsgClassTest]
1167959 06:23:38.339 [6860] (ArchiveTask) <17472> EV:M Adding External Filter 'EnterpriseVault.CustomFilter' to the list for processing|
1167960 06:23:38.339 [6860] (ArchiveTask) <17472> EV:M Successfully added External Filter 'EnterpriseVault.CustomFilter'|Calling Initialize
1167961 06:23:38.339 [6860] (ArchiveTask) <17472> EV:M [CustomXMLFilter] Custom Filter initialized on thread.
1167962 06:23:38.339 [6860] (ArchiveTask) <17472> EV:M CEVFilterController::CreateFilterObject() (Exit) |Success [0] |
1167963 06:23:38.339 [6860] (ArchiveTask) <17472> EV:M CEVFilterController::InitializeFiltersFromRegistry - MoveOnFilterFailure RegKey: [0x00000000]
1167964 06:23:38.339 [6860] (ArchiveTask) <17472> EV:M CEVFilterController::InitializeFiltersFromRegistry() (Exit) |Success [0] |
1167965 06:23:38.339 [6860] (ArchiveTask) <17472> EV:M Successfully enabled external filtering
And usually if it doesn't your archiving task will fail.
Secondly look for how the message was evaluated as follows :
1171158 06:23:49.996 [6860] (ArchiveTask) <17472> EV:H [CustomXMLFilter] Custom Filter processing message 'Mail test 1'
1171159 06:23:49.996 [6860] (ArchiveTask) <17472> EV:L [CustomRules][CRuleSet] CRuleSet::HasRuleEvaluationOfType - RuleEvaluationType [4] [EXISTS] in RuleEvaluationTypes [5],
1171160 06:23:49.996 [6860] (ArchiveTask) <17472> EV:H [CustomXMLFilter] RuleDataXML is ''
1171161 06:23:49.996 [6860] (ArchiveTask) <17472> EV:H [CustomRules][CRuleSet] Getting rule data...
1171162 06:23:50.058 [6860] (ArchiveTask) <17472> EV:L [CustomRules][CRule] CRule::HasRuleClauseType - Rule clause type [32] [EXISTS] in [32]
1171163 06:23:50.058 [6860] (ArchiveTask) <17472> EV:M [CustomXMLFilter] MakeUniqueRuleDataNodes : Transforming Ruledata to remove redundancy.
1171164 06:23:50.058 [6860] (ArchiveTask) <17472> EV:H [CustomXMLFilter] New RuleDataXML is now '<?xml version="1.0" encoding="UTF-16"?><RULE_DATA><DATATYPE NAME="NAMEDPROPERTIES"><DATA NAME="TAG"><VALUE>MSGCLASS</VALUE></DATA></DATATYPE></RULE_DATA>'
1171165 06:23:50.058 [6860] (ArchiveTask) <17472> EV:L [CustomXMLFilter] GetMessageNamedProperties - XML RULE Data ='<?xml version="1.0" encoding="UTF-16"?><RULE_DATA><DATATYPE NAME="NAMEDPROPERTIES"><DATA NAME="TAG"><VALUE>MSGCLASS</VALUE></DATA></DATATYPE></RULE_DATA>'
1171166 06:23:50.058 [6860] (ArchiveTask) <17472> EV:L [CustomXMLFilter] GetMessageNamedProperties - Found '1' nodes of named properties in Rule data XML
1171167 06:23:50.058 [6860] (ArchiveTask) <17472> EV:L [CustomXMLFilter] GetMessageNamedProperties - Getting Tag ='MSGCLASS' from custom properties
1171168 06:23:50.058 [6860] (ArchiveTask) <17472> EV:M CEVFilterController::get_MessageClass - Entering routine
1171169 06:23:50.058 [6860] (ArchiveTask) <17472> EV:M CEVFilterController::get_MessageClass - Returning 'Original Message Class' = IPm.Note.Microsoft.Voicemail.UM.CA
1171170 06:23:50.058 [6860] (ArchiveTask) <17472> EV:L [CustomXMLFilter] Custom tag 'MSGCLASS' and name '0x001A', set to IPm.Note.Microsoft.Voicemail.UM.CA
1171171 06:23:50.058 [6860] (ArchiveTask) <17472> EV:L [CustomXMLFilter] Adding property 'PR_MESSAGE_CLASS (0x001a)' to Items XML. [tag='MSGCLASS', value='IPm.Note.Microsoft.Voicemail.UM.CA']
1171172 06:23:50.058 [6860] (ArchiveTask) <17472> EV:L [CustomRules][CRule] Evaluating item against MBX DIFF_RET_CAT rule...
1171173 06:23:50.058 [6860] (ArchiveTask) <17472> EV:L [CustomRules][CNamedPropClause] testing against ANY of 1 NamedProps
1171174 06:23:50.058 [6860] (ArchiveTask) <17472> EV:L [CustomRules][CNamedPropClause] : ipm.note.microsoft.voicemail.um.ca MATCHED ipm.note.microsoft.voicemail.um.ca
1171175 06:23:50.058 [6860] (ArchiveTask) <17472> EV:L [CustomRules][CNamedPropClause] match with test ' 'ipm.note.microsoft.voicemail.um.ca''
1171176 06:23:50.058 [6860] (ArchiveTask) <17472> EV:L [CustomRules][CNamedPropClause] Named prop clause: MSGCLASS MATCHED ANY PROP Values
1171177 06:23:50.058 [6860] (ArchiveTask) <17472> EV:L [CustomRules][CRule] Finished evaluating item against MBX DIFF_RET_CAT rule; matches
1171178 06:23:50.058 [6860] (ArchiveTask) <17472> EV:M CRule::GetRuleProperties - m_RuleProps is not NULL
1171179 06:23:50.058 [6860] (ArchiveTask) <17472> EV:M [CustomXMLFilter] Reading MBX DIFF_RET_CAT rule properties...
1171180 06:23:50.058 [6860] (ArchiveTask) <17472> EV:L [CustomRules][CRuleSet] CRuleSet::HasRuleEvaluationOfType - RuleEvaluationType [2] [DOES NOT EXIST] in RuleEvaluationTypes [5],
1171181 06:23:50.074 [6860] (ArchiveTask) <17472> EV:M [CustomXMLFilter] Setting recognised ACTION to [1]
1171182 06:23:50.074 [6860] (ArchiveTask) <17472> EV:M [CustomXMLFilter] Setting message content category to [MsgClassTest]
1171183 06:23:50.074 [6860] (ArchiveTask) <17472> EV:M CEVFilterController::get_MessageClass - Entering routine
1171184 06:23:50.074 [6860] (ArchiveTask) <17472> EV:M CEVFilterController::get_MessageClass - Returning 'Original Message Class' = IPm.Note.Microsoft.Voicemail.UM.CA
1171185 06:23:50.074 [6860] (ArchiveTask) <17472> EV:L [CustomXMLFilter] Custom tag 'MSGCLASS' and name '0x001A', set to IPm.Note.Microsoft.Voicemail.UM.CA
1171186 06:23:50.074 [6860] (ArchiveTask) <17472> EV:L [CustomXMLFilter] Adding Custom index property set 'MsgClassTest'...
1171187 06:23:50.074 [6860] (ArchiveTask) <17472> EV:L [CustomXMLFilter] Adding property 'PR_MESSAGE_CLASS (0x001a)' to index property set 'MsgClassTest' [tag='MSGCLASS', value='IPm.Note.Microsoft.Voicemail.UM.CA']
1171188 06:23:50.074 [6860] (ArchiveTask) <17472> EV:M [CustomXMLFilter] Setting retention category to [167A06CB31E01744F8500E3D54FC80BEC1b10000evsite]
1171189 06:23:50.074 [6860] (ArchiveTask) <17472> EV:M Returning IndexedPropertiesSet = <?xml version="1.0" encoding="UTF-16"?>|<ARCHIVED_ITEM xmlns:o="urn:kvsplc-com:archived_item" version="1.0"><MSG><PROPSETLIST><PROPSET NAME="MsgClassTest" SEARCH="y" RESULTS="n"><PROP NAME="MSGCLASS"><VALUE>IPm.Note.Microsoft.Voicemail.UM.CA</VALUE></PROP></PROPSET></PROPSETLIST></MSG></ARCHIVED_ITEM>|
1171190 06:23:50.074 [6860] (ArchiveTask) <17472> EV:M Returning Create Shortcut = TRUE
1171191 06:23:50.074 [6860] (ArchiveTask) <17472> EV:M Returning Delete Original = TRUE
1171192 06:23:50.074 [6860] (ArchiveTask) <17472> EV:M Returning Vault Id = 1E5850B2EA77101459FCD56CBC4D3A5871110000evsite
1171193 06:23:50.074 [6860] (ArchiveTask) <17472> EV:M Returning Retention Category = 167A06CB31E01744F8500E3D54FC80BEC1b10000evsite
1171194 06:23:50.074 [6860] (ArchiveTask) <17472> EV:M Returning Action = 1
1171195 06:23:50.074 [6860] (ArchiveTask) <17472> EV:L CEVFilterController::FilteringCompleted() (Entry) |
1171196 06:23:50.074 [6860] (ArchiveTask) <17472> EV:M CEVFilterController::FilteringCompleted() (Exit) |Success [0] |
1171197 06:23:50.074 [6860] (ArchiveTask) <17472> EV:L CEVFilterController::InitializeCurrentItem() (Entry) |
1171198 06:23:50.074 [6860] (ArchiveTask) <17472> EV:M CEVFilterController::InitializeCurrentItem() (Exit) |Success [0] |
1171199 06:23:50.074 [6860] (ArchiveTask) <17472> EV:M CEVFilterController::ProcessAllFilters() (Exit) |Success [0] |
1171200 06:23:50.089 [6860] (ArchiveTask) <17472> EV:M EF: Item will be archived|Mailbox: /o=EV Training/ou=First Administrative Group/cn=Recipients/cn=TCowper|Folder: ?Inbox|Message: Mail test 1
1171201 06:23:50.089 [6860] (ArchiveTask) <17472> EV:L CArchivingAgent::ExternalFiltering() (Exit) |Success [0] |
1171202 06:23:50.089 [6860] (ArchiveTask) <17472> EV:M CArchivingAgent::ProcessItemInternal - After call to ExternalFiltering. RetentionCategory[167A06CB31E01744F8500E3D54FC80BEC1b10000evsite] ArchiveId[1E5850B2EA77101459FCD56CBC4D3A5871110000evsite] ContainingArchiveId[1DF2DFF131A9AFB4EB0B493648330C02B1110000evsite] IndexedPropertiesSet[<?xml version="1.0" encoding="UTF-16"?>|<ARCHIVED_ITEM xmlns:o="urn:kvsplc-com:archived_item" version="1.0"><MSG><PROPSETLIST><PROPSET NAME="MsgClassTest" SEARCH="y" RESULTS="n"><PROP NAME="MSGCLASS"><VALUE>IPm.Note.Microsoft.Voicemail.UM.CA</VALUE></PROP></PROPSET></PROPSETLIST></MSG></ARCHIVED_ITEM>|] MessageModified[FALSE] RetryCount[0] [0x00000000]
And this one is the second message with a normal message class :
1171769 06:23:51.214 [6860] (ArchiveTask) <17472> EV:H [CustomXMLFilter] Custom Filter processing message 'Mail test 2'
1171770 06:23:51.214 [6860] (ArchiveTask) <17472> EV:L [CustomRules][CRuleSet] CRuleSet::HasRuleEvaluationOfType - RuleEvaluationType [4] [EXISTS] in RuleEvaluationTypes [5],
1171771 06:23:51.214 [6860] (ArchiveTask) <17472> EV:H [CustomXMLFilter] RuleDataXML is '<?xml version="1.0" encoding="UTF-16"?><RULE_DATA><DATATYPE NAME="NAMEDPROPERTIES"><DATA NAME="TAG"><VALUE>MSGCLASS</VALUE></DATA></DATATYPE></RULE_DATA>'
1171772 06:23:51.214 [6860] (ArchiveTask) <17472> EV:L [CustomXMLFilter] GetMessageNamedProperties - XML RULE Data ='<?xml version="1.0" encoding="UTF-16"?><RULE_DATA><DATATYPE NAME="NAMEDPROPERTIES"><DATA NAME="TAG"><VALUE>MSGCLASS</VALUE></DATA></DATATYPE></RULE_DATA>'
1171773 06:23:51.214 [6860] (ArchiveTask) <17472> EV:L [CustomXMLFilter] GetMessageNamedProperties - Found '1' nodes of named properties in Rule data XML
1171774 06:23:51.214 [6860] (ArchiveTask) <17472> EV:L [CustomXMLFilter] GetMessageNamedProperties - Getting Tag ='MSGCLASS' from custom properties
1171775 06:23:51.214 [6860] (ArchiveTask) <17472> EV:M CEVFilterController::get_MessageClass - Entering routine
1171776 06:23:51.214 [6860] (ArchiveTask) <17472> EV:M CEVFilterController::get_MessageClass - Returning 'Original Message Class' = IPM.Note
1171777 06:23:51.214 [6860] (ArchiveTask) <17472> EV:L [CustomXMLFilter] Custom tag 'MSGCLASS' and name '0x001A', set to IPM.Note
1171778 06:23:51.214 [6860] (ArchiveTask) <17472> EV:L [CustomXMLFilter] Adding property 'PR_MESSAGE_CLASS (0x001a)' to Items XML. [tag='MSGCLASS', value='IPM.Note']
1171779 06:23:51.214 [6860] (ArchiveTask) <17472> EV:L [CustomRules][CRule] Evaluating item against MBX DIFF_RET_CAT rule...
1171780 06:23:51.214 [6860] (ArchiveTask) <17472> EV:L [CustomRules][CNamedPropClause] testing against ANY of 1 NamedProps
1171781 06:23:51.214 [6860] (ArchiveTask) <17472> EV:L [CustomRules][CNamedPropClause] : ipm.note.microsoft.voicemail.um.ca DID NOT MATCH ipm.note
1171782 06:23:51.214 [6860] (ArchiveTask) <17472> EV:L [CustomRules][CNamedPropClause] Named prop clause: MSGCLASS DID NOT MATCH ANY PROP Values
1171783 06:23:51.214 [6860] (ArchiveTask) <17472> EV:L [CustomRules][CRule] Finished evaluating item against MBX DIFF_RET_CAT rule; does not match
1171784 06:23:51.214 [6860] (ArchiveTask) <17472> EV:M [CustomXMLFilter] Defaulting to ACTION ARCHIVE_ITEM
1171785 06:23:51.214 [6860] (ArchiveTask) <17472> EV:L [CustomRules][CRuleSet] CRuleSet::HasRuleEvaluationOfType - RuleEvaluationType [2] [DOES NOT EXIST] in RuleEvaluationTypes [5],
1171786 06:23:51.214 [6860] (ArchiveTask) <17472> EV:M [CustomXMLFilter] Setting message content category to DEFAULT [MsgClassTest]
1171787 06:23:51.214 [6860] (ArchiveTask) <17472> EV:M CEVFilterController::get_MessageClass - Entering routine
1171788 06:23:51.214 [6860] (ArchiveTask) <17472> EV:M CEVFilterController::get_MessageClass - Returning 'Original Message Class' = IPM.Note
1171789 06:23:51.214 [6860] (ArchiveTask) <17472> EV:L [CustomXMLFilter] Custom tag 'MSGCLASS' and name '0x001A', set to IPM.Note
1171790 06:23:51.214 [6860] (ArchiveTask) <17472> EV:L [CustomXMLFilter] Adding Custom index property set 'MsgClassTest'...
1171791 06:23:51.214 [6860] (ArchiveTask) <17472> EV:L [CustomXMLFilter] Adding property 'PR_MESSAGE_CLASS (0x001a)' to index property set 'MsgClassTest' [tag='MSGCLASS', value='IPM.Note']
1171792 06:23:51.214 [6860] (ArchiveTask) <17472> EV:M Returning IndexedPropertiesSet = <?xml version="1.0" encoding="UTF-16"?>|<ARCHIVED_ITEM xmlns:o="urn:kvsplc-com:archived_item" version="1.0"><MSG><PROPSETLIST><PROPSET NAME="MsgClassTest" SEARCH="y" RESULTS="n"><PROP NAME="MSGCLASS"><VALUE>IPM.Note</VALUE></PROP></PROPSET></PROPSETLIST></MSG></ARCHIVED_ITEM>|
1171793 06:23:51.214 [6860] (ArchiveTask) <17472> EV:M Returning Create Shortcut = TRUE
1171794 06:23:51.214 [6860] (ArchiveTask) <17472> EV:M Returning Delete Original = TRUE
1171795 06:23:51.214 [6860] (ArchiveTask) <17472> EV:M Returning Vault Id = 1E5850B2EA77101459FCD56CBC4D3A5871110000evsite
1171796 06:23:51.214 [6860] (ArchiveTask) <17472> EV:M Returning Retention Category = 1417053BF457E9048AD2408BE1AF991851b10000evsite
1171797 06:23:51.214 [6860] (ArchiveTask) <17472> EV:M Returning Action = 1
1171798 06:23:51.214 [6860] (ArchiveTask) <17472> EV:L CEVFilterController::FilteringCompleted() (Entry) |
1171799 06:23:51.230 [6860] (ArchiveTask) <17472> EV:M CEVFilterController::FilteringCompleted() (Exit) |Success [0] |
1171800 06:23:51.230 [6860] (ArchiveTask) <17472> EV:L CEVFilterController::InitializeCurrentItem() (Entry) |
1171801 06:23:51.230 [6860] (ArchiveTask) <17472> EV:M CEVFilterController::InitializeCurrentItem() (Exit) |Success [0] |
1171802 06:23:51.230 [6860] (ArchiveTask) <17472> EV:M CEVFilterController::ProcessAllFilters() (Exit) |Success [0] |
1171803 06:23:51.230 [6860] (ArchiveTask) <17472> EV:M EF: Item will be archived|Mailbox: /o=EV Training/ou=First Administrative Group/cn=Recipients/cn=TCowper|Folder: ?Inbox|Message: Mail test 2
1171804 06:23:51.230 [6860] (ArchiveTask) <17472> EV:L CArchivingAgent::ExternalFiltering() (Exit) |Success [0] |
1171805 06:23:51.230 [6860] (ArchiveTask) <17472> EV:M CArchivingAgent::ProcessItemInternal - After call to ExternalFiltering. RetentionCategory[1417053BF457E9048AD2408BE1AF991851b10000evsite] ArchiveId[1E5850B2EA77101459FCD56CBC4D3A5871110000evsite] ContainingArchiveId[1DF2DFF131A9AFB4EB0B493648330C02B1110000evsite] IndexedPropertiesSet[<?xml version="1.0" encoding="UTF-16"?>|<ARCHIVED_ITEM xmlns:o="urn:kvsplc-com:archived_item" version="1.0"><MSG><PROPSETLIST><PROPSET NAME="MsgClassTest" SEARCH="y" RESULTS="n"><PROP NAME="MSGCLASS"><VALUE>IPM.Note</VALUE></PROP></PROPSET></PROPSETLIST></MSG></ARCHIVED_ITEM>|] MessageModified[FALSE] RetryCount[0] [0x00000000]
Other Notes
Changes to the custom properties.xml file affecting the search.asp page don't take effect until you restart IIS Admin on the EV server.
Changes to the custom filter.xml file don't take effect until you restart the archiving task in the VAC
I hope that helps people who have been struggling with this, and I'm trying to get the description of this added to the documentation.