EV90SP2 - provisioning issue event 41110

[GertjanA]

Hello all,

Following an upgrade from 80SP5 to 90SP2, I get an error when provisioning is being run. The error is 41110. The event is:

Event Type: Error
Event Source: Enterprise Vault
Event Category: Exchange Provisioning Task
Event ID: 41110
Date:  10/10/2011
Time:  9:10:39 AM
User:  N/A
Computer: SPFRMAS10010
Description:
The Exchange mailbox provisioning task failed during provisioning group processing. The task has stopped.

Task: Exchange Provisioning Task for europe.intranet Domain: europe.intranet Provisioning group: <none> Group member: <none>

Error: Data is Null. This method or property cannot be called on Null values.

Trace:    at System.Data.SqlClient.SqlBuffer.get_String()
   at System.Data.SqlClient.SqlDataReader.GetString(Int32 i)
   at KVS.EnterpriseVault.ExchangePolicySync.ExchangePolicySynchronization.CheckEnabledMbxs()
   at KVS.EnterpriseVault.ExchangePolicySync.ExchangePolicySynchronization.RunSynchronization(ExchangePolicySyncTaskEntry taskEntry, CheckControlEventsDelegate checkControlEvents, Boolean reportMode)

The dtrace of the provisioning shows:

13202 09:15:33.816  [7780] (EvExchangePolicySyncTask) <7288> EV-M {DirectoryDBAccess.MapDNSAliasToSQLServer} DNS Alias: EVIDFR.EUROPE.INTRANET ==> SQL server: spfrmas10010,65001 Cached? True
13203 09:15:33.832  [7780] (EvExchangePolicySyncTask) <7288> EV-H {ExchangePolicySynchronization.HandleFatalException} Fatal exception raised while processing provisioning group [<none>] / [<none>] - Data is Null. This method or property cannot be called on Null values.. Stack trace:    at System.Data.SqlClient.SqlBuffer.get_String()|   at System.Data.SqlClient.SqlDataReader.GetString(Int32 i)|   at KVS.EnterpriseVault.ExchangePolicySync.ExchangePolicySynchronization.CheckEnabledMbxs()|   at KVS.EnterpriseVault.ExchangePolicySync.ExchangePolicySynchronization.RunSynchronization(ExchangePolicySyncTaskEntry taskEntry, CheckControlEventsDelegate checkControlEvents, Boolean reportMode)
13204 09:15:33.832  [7780] (EvExchangePolicySyncTask) <7288> EV~E |Event ID: 41110 The Exchange mailbox provisioning task failed during provisioning group processing. The task has stopped.|Task: Exchange Provisioning Task for europe.intranet|Domain: europe.intranet|Provisioning group: <none>|Group member: <none>|Error: Data is Null. This method or property cannot be called on Null values.|Trace:    at System.Data.SqlClient.SqlBuffer.get_String()|   at System.Data.SqlClient.SqlDataReader.GetString(Int32 i)|   at KVS.EnterpriseVault.ExchangePolicySync.ExchangePolicySynchronization.CheckEnabledMbxs()|   at KVS.EnterpriseVault.ExchangePolicySync.ExchangePolicySynchronization.RunSynchronization(ExchangePolicySyncTaskEntry taskEntry, CheckControlEventsDelegate checkControlEvents, Boolean reportMode)
13205 09:15:33.847  [7780] (EvExchangePolicySyncTask) <7288> EV-M {ExchangePolicySynchronization.WaitForWorkerThreads} Waiting for worker threads to finish processing
13206 09:15:33.847  [7780] (EvExchangePolicySyncTask) <7288> EV-M {ExchangePolicySynchronization.WaitForWorkerThreads} Queue is empty, setting WorkerStopEvent
13207 09:15:33.847  [7780] (EvExchangePolicySyncTask) <7288> EV-M {ExchangePolicySynchronization.WaitForWorkerThreads} All worker threads finished processing
13208 09:15:33.847  [7780] (EvExchangePolicySyncTask) <7288> EV~I |Event ID: 41104 The Exchange mailbox provisioning task has completed.|Task: Exchange Provisioning Task for europe.intranet|Domain: europe.intranet
13209 09:15:33.847  [7780] (EvExchangePolicySyncTask) <7288> EV-H {ExchangePolicySynchronization.RunSynchronization} Leaving

There are 2 provisioning groups, targeting 1 OU each. There are users in the OU's. As far as I see in the event and the trace, it looks like there is a leftover provisioning group in the database? |Provisioning group: <none>|Group member: <none>|Error: Data is Null. This method or property cannot be called on Null values.

Where are the provisioning groups stored in the databases? Can I check that, and remove erroneous errors? Or should I recreate (delete/create) the provisioning groups/provisioning task?

any help/advice appreciated.

Regards,

Gertjan

[GertjanA]

I recreated provisioning groups, and provisioning task, but same issue remains.

[TonySterling]

Hey mate,

I just saw this in the Can you solve these section.  Do you still need help on this?

Tony

[GertjanA]

Hello Tony,

Thanks, forgot about this one myself.

The issue still exists. It looks like the provisioning processes all users, then fails on something.

I have doublechecked of anything has changed, but it hasn't. If you by any chance know what to do next, I''d be gratefull...

Greetz.

Gertjan

[AndrewB]

gertjan, i presume you saw this:

"Fatal exception raised while processing provisioning group [<none>] / [<none>] - Data is Null. This method or property cannot be called on Null values."

was a provisioning group created and deleted recently perhaps? maybe you can check in sql for orphaned references or something.

[GertjanA]

Hello Andrew,

Not as far as I know. Do you happen to know which tables to check?

I'm a little lost here.

[TonySterling]

You can run this

Select * from PolicyTargetGroup

The display name is a bit to the right in the results.

[GertjanA]

I've ran the query, and it shows 2 lines.

we have 2 pg's and these are in here.

error remains.

[Natalia_B_]

Hi Gertjan,

we had the same issue with one customer. The cause was an invalid exchange server under exchange targets.

Natalia

[GertjanA]

Hello Natalia,

Thanks. Good point. Unfortunately, this is not the issue. It does point to a possible issue though. The 2 servers I am targeting are in different OU's, and it might be that the rights are incorrect. I'm investigating. Thanks again!

[KarlW · Accepted solution]

Hi Gertjan,

I've taken a look at the issue and have hopefully figured out the issue.

The provisioning task is trying to report on mailboxes that appear enabled (MbxArchivingState=1) but are not connected to an archiving task.

A new column was added in EV9.0 to the ExchangeMailboxStore table - normally this would get populated by the provisioning task after upgrade.  However mailbox stores that no longer have any active (existing) or targeted mailboxes will not get updated.  When provisioning runs it will generate this error.

Do you have access to the SQL Server and Directory DB?  If so run the following:

 select * from ExchangeMailboxStore 

If any rows come back with a NULL ServerName column then this could be the issue.  It will depend if any mailboxes are connected to this mailbox store.  Run the following query:

select EME.LegacyMbxDN, EMS.Name, EMS.ServerName from ExchangeMailboxEntry EME
 inner join ExchangeMailboxStore EMS
 on EME.MbxStoreIdentity = EMS.MbxStoreIdentity
 inner join ExchangeServerEntry ESE 
 on EMS.ExchangeServerIdentity = ESE.ExchangeServerIdentity
 left join ArchivingRetrievalTask ART 
 on ESE.ExchangeServerEntryId = ART.ExchangeServerEntryId 
where
 MbxArchivingState = 1 AND 
 ExchangeMbxType = 1 AND 
 ART.ExchangeServerEntryId IS NULL

If any rows come back with a NULL value for any column this will generate the provisioning task error.  To resolve the issue run the following on the affected mailbox stores:

 update ExchangeMailboxStore Set ServerName = '<blank>' where ServerName is NULL

This will resolve the exception and the provisioning report will record the server name as <blank>.  This shouldn't impact any other behaviour.  

If the mailbox store becomes active again (for instance a mailbox against it is provisioned) the provisioning task will re-synchronize the correct ServerName value.  If this is for a mailbox store that no longer exists then leaving the ServerName as 'blank' should be OK.

Let me know if this resolves the issue - if so I'll get it written up into a TechNote (I assume you couldn't find an existing one).

Cheers

Karl

[MarkBarefoot]

Good spot Karl. I took a look but didn't know about the change in DB. I very much doubt there is a Technote for this either.

[GertjanA]

10 points for Karl!

1st query returned 20 rows. 2nd query returned 152 rows. Ran 3rd query. Ran provisioning. No failure anymore!

I now do see in the report:

Enabled mailboxes on Exchange Servers that do not have a mailbox archiving task:

Mailbox: legacyDN, Exchange mailbox database: Mbx Store 2 (ExchangeServerName), Exchange Server: <blank>

Info: Removed mailbox info, changed to LegacyDN.Removed ExchangeServerName, changed to ExchangeServerName.

Should I delete the entries in the ExchangeMailboxEntry, then run provisioning again to clear the errorneous entries? This specific environment does not have that many rows in it.

[KarlW]

Before deleting the entries I would check that they are legacy (or deleted) data - for example do the users or mailbox stores still exist?

If the mailbox stores still exist you could get the distinguished name for the Exchange server from AD and populate the server names on the store table.

If the mailboxes still exist should they be being archived?  If so you'll need to add an archive task for the Exchange server they are on.

I would be careful deleting rows as if the mailboxes no longer exist you will lose the tie between mailbox and archive.  Another option is to update the MbxArchivingState to 2 for these entries - this is another workaround for the original error (and means you don't need to lose the entries from the DB).

Thanks

Karl