41447 - There has been no mailbox synchronization with Exchange since...

[EVSpinner]

Hi I have a really stubborn 41447 in my VAC - I just can't shift it

41447 - There has been no mailbox synchronization with Exchange since etc etc

The number of days increments

It's definitely not a simple ReRun checks

If I purge the queue, delete the task associated the Exchange in question, delete the MSMQ parts associated with this now deleted task, recreate the task and run a synchronise all, messages populate the new MSMQ A7 for the new task, clear down, rerun checks and the alert goes away

Leave it to sync automatically over a few days and the problem returns

I can see the A7 calls in the queues right now, just build daily, so I know the task initiates the sync, and the call to sync gets as far as the MSMQ's but they just won't process through

MSMQ service restarts - no effect

EV service restarts - no effect

Message are going through other MSMQ's, A1 and A2 for example

I've DTraced the ArchiveTask and left it running during automatic and manual syncs - There's nothing obvious that jumps out, but if anyone knows specific things to look for please let me know

Are there any other DTrace componants I should be targetting?

Has anyone got any ideas what I could try next?

[MassimoVagli]

Hello,

Have you seen this forum post?

https://www-secure.symantec.com/connect/forums/error-dag

Massimo

[EVSpinner]

Progress (I hope)

On advice of a colleague I deleted the entire A4 queue, restarted the Admin service and refreshed the queues - movement immediately (scheduled archiving set to run from 6-6)

I will update again tomorrow, but at the moment it looks like a messages stuck in the A4, prevented previous queues - A5 & 6 -  from clearing having the knock on effect of preventing the A7 from clearing. That looks like what happened, but if anyone can critique my logic or correct me where I'm wrong I'd appreciate the education 

[EVSpinner]

Cheers Massimo, I should have mentioned - Exchange is 2007, so DAG's not the prob

[EVSpinner]

This problem seems to be all about the MSMQ's and passage of messages through the queues

I've read through the Admin guide, the process diagrams and I'm still stumped

If I start with clear queues (except the A5) and initiate a sync all, the A7 queue populates and might do 10 before stalling. It will just sit there not going down, or up, all the time the A6 queue starts to build.

If I restart the Task Controller, I get movement again

Reriun the checks after a few process in the A7 and expect the 41447 to disappear. It doesn't

DTrace Archive Task produces a massive log that is hard to go through without knowing what I'm looking for

I do some keyword searches around Sync, 41447, A7 etc, and nothing shows up

Basically I need some advice on how I can better troubleshoot why these queues are so sticky

Any ideas? 

[MassimoVagli]

Hello EVSpinner, 

I think it's better if you open a new case with Symantec support.

Besides analyzing the provisioning task reports together with multiple EVExchangePolicySyncTask dtraces (provisioning task) would help.

FYI here you can find the explanation of the MSMQ flow https://www-secure.symantec.com/connect/forums/explanation-ev-queues 

Massimo

[EVSpinner]

According to the following TechNote the Archive Task is authoritative for synchronization not the policysynctask

Why would this be related to provisioning or the provisioning task? It's initiation of mailbox synchronisation via archiving task

 http://www.symantec.com/docs/TECH129595

List of DTrace processes and decriptions of each

ArchiveService\ArchiveTask

Mailbox archiving, sync etc. (Processes mailbox and user requests)

EvExchangePolicySyncTask

EV Exchange Provisioning Task

[Merv]

Spinner, empty your A7 queue, then do a manual sync of 1 mbx dtrace agentclientbroker and if you see mapi error codes follow the usual mapi troubleshooting steps like leftover mapi profiles, connection to system mailbox, exchange permissions and connectivity issues to GC /CAS VIP 

http://www.symantec.com/business/support/index?page=content&id=TECH35774

[Merv]

On second thought if there are no mapi issues and your issue is the other A q's it would be different steps

e.g. Items in A4 queue stuck or storage archive queue ? That would be storage service issues..- issues doing a manual archive or restore ? Dtrace storagearchive and storageserver

A6 stuck? That's moved items..you may try  changing the mailbox arching policy to not process move items just to rule that out

Are there any other EV events /errors that may be related?

Anything stuck in your admin queues since you've been lots of purging?

Common causes of items being stuck or slow to process from storage archive queues -  vault store SQL DBs perf issues - Any sql blocking and long wait times?is connectivity to Storage and partitions all good? 

[EVSpinner]

"empty your A7 queue"

done.

I'm pretty sure it's not the sync itself or the A7 queue itself that's the issue. It's the queues up ahead not clearing down. 'A' queues are FIFO right, so the preceding queue won't clear until the one ahead completes

"then do a manual sync of 1 mbx dtrace agentclientbroker and if you see mapi error codes follow the usual mapi troubleshooting steps like leftover mapi profiles, connection to system mailbox, exchange permissions and connectivity issues to GC /CAS VIP"

Interesting point - I'll look into that. Also worth noting some inconsistent behaviour. At one point I was initiating full sync and the A7 would increment by one. Later (after all service restart including MSMQ services, I initiated the all mailbox sync and the A7 incremenetd by several thousand - the same number as there were archives/mailboxes. Weird

Items in A4 queue stuck or storage archive queue ? 

No

"That would be storage service issues..- issues doing a manual archive or restore ? Dtrace storagearchive and storageserver" 

I don't have any reports of retrieval/restore or archiving issues, but noted

"A6 stuck? That's moved items..you may try  changing the mailbox arching policy to not process move items just to rule that out" 

Again interesting. I'll try this and rule it out also

"Are there any other EV events /errors that may be related?" 

No

"Anything stuck in your admin queues since you've been lots of purging?" 

No

"Common causes of items being stuck or slow to process from storage archive queues -  vault store SQL DBs perf issues - Any sql blocking and long wait times?"

I'll keep an eye on it, but nothing has been stickign in this queue that I observed 

"is connectivity to Storage and partitions all good?" 

No complaints

Other points of interest. There are two Exchanges and 2 tasks. this problem only affects one Exchange and task. Yes, I've recreated the task already. It makes the problem go away (ie, Sync initiates, A7 populates, and clears down, rerun VAC checks and the 41447 goes away). Leave it 24 hours and it returns

The only other thing I did different yesterday and to look at the archiving schedule, backup schedule and sit the Sync schedule to sit in betwen these two, removing the mornign sync all together

[Merv]

"Other points of interest. There are two Exchanges and 2 tasks. this problem only affects one Exchange and task. Yes, I've recreated the task already. It makes the problem go away (ie, Sync initiates, A7 populates, and clears down, rerun VAC checks and the 41447 goes away). Leave it 24 hours and it returns

The only other thing I did different yesterday and to look at the archiving schedule, backup schedule and sit the Sync schedule to sit in betwen these two, removing the mornign sync all together"

There is no sync schedule only provisioning task schedule where you can schedule  twice a day. The sync schedule starts as part of the archiving task schedule. When archiving  starts all targeted mbx are synced( updating EV policy to hidden msg.)you will all the individual mbx usernames if you expand the messages in the A7 queue. This happens along with populating the A5 queue. Provisioning does not make any changes to the A Q just runs against AD to match up users to the policy and update EV Directory exchange  mailboxentry table.common misconception with provisioning which is NOT actual enabling or syncing of the EV policy to the hidden message.. That is done by the archiving task and not provisioning task.

Big big clue - the problem is isolated to just 1 exchange server...and problem is the scheduled automatically archiving for that server is not happening? But if you manually run it looks to be working. I've seen obvious config gotchas like no daily schedule for archiving or just having it set to run in report mode or tasked stopped. Another interesting one is DAG or passive server environment with no mounted mbx DBs..I.e all the active DB's are on the other exch servers...so obviously nothing is going to happen to that one passive server. 

[EVSpinner]

This one is going to Symantec so I'll have a solution thereafter. Please do not lock this thread until I've provided the solution 

[EVSpinner]

Update:  I have continued to tackle this by documenting whats going on in the MSMQ's. This is the battleground. Also rather than looking at it as a problem, I'm coming down more on the side of the fence of configuration. Queues being First In First Out the queues up ahead aren't clearing down, so A7 is not proceeding - if anyone can correct any errors in my understand you have an avid listerner

My A6 and A5 are populating but not clearing, then going up again, and processing some more - dymanic but never emptying. I am looking to increase the number fo Exchange threads and reducing the number of items archived per run to try and mitigate this

Again, any criticism or errors in my logic, I am very open to be corrected

[Merv]

Spinner, Ok besides the 41447 which is indicative that there was no A7 messages (mbx sync) processed in the last 24 hours you should be checking your daily exchange scheduled archiving reports and status? How many mbx were successfully archived daily? Any not processed? Strange that you are not getting any errors...and what is your archiving schedule like? 12 hours a day? 7 days a week?