Probles with A queues A5 A6 and A7

[James_Slack]

Hi,

I have 2 servers and looking in the Message Queuing I can see a backlog of messages in there.

Server 1

A5 = 4240

A6 = 10329

A7 = 419

Server 2

A5 = 1341

A6 = 6786

A7 = 202

I have done some basic things to see if they will go down, change the schedule, stop archiving for a bit, reboot etc. Nothing. It seems this lot is stuck in the queues.

My questions are this.

1) What are each queue used for?

2) How do I clear down each queue and would we lose data doing so?

[James_Slack]

Sorry for lack of info here, had someone waiting for something over my shoulder during writing the above.

This is Vault 10 on 2008R2. Archiving seems to be happening OK even though there seems to be items stuck in these queues.

The reason I found these is I keep getting failed items when I am importing PSTs. This may be unrelated to that issue, but I found it and I guess it's not right.

[JesusWept3 · Accepted solution]

A7 = scheduled mailbox synchronization A6 = moved shortcut processing (updates the folder and retention categories when Items are moved) A5 = scheduled mailbox archiving Have a look on your SQL server and see if your CPU is through the roof Moved item processing can be quite intensive and if you don't do regular maintanance on your SQL server, ie index rebuilds, then you can cause major slow downs on SQL etc which slow everything down

[James_Slack]

Argh. Page just swalloed my reply!

SQL is fine, 0% CPU with the odd 1% / 2% blip. Memory is fine too.

Looking at the A5 queues I see a mix of live users, deleted users (that I need to tidy up I guess) and "deprovisioned" users (who should all be live users as far as EV is concerned.

A6 probably shows similar but of all the ones I checked they were live users

A7 shows a mix of deleted, deprov, disabled and probably some live users (from memory having just lost my lengthy reply).

What do I do next please?

[EdLacey]

You can quite safely purge the A5 queues which will then give the other 2 queues a chance to purge themselves.

As your A5 queues are not clear at the end of a scheduled archiving task you need to tune the relevant tasks.

1. Expand the archiving window during the week

2. Can you run the archiving tasks all day during the weekend?

3. Decrease the amount of items archived per pass if you can't do either of the above (default is 1000)

4. On the Advanced tab of the Storage Service is the Archive Processes value still the default of 5?

[James_Slack]

Hi,

The archiving schedule was actually set to 24x7.

I have only just today (on advice I found searching before posting here) changed my schedule to stop for 15 minutes every 4 hours.

I'll just look up the other settings now...

[James_Slack]

Hi,

3) It is still at the default of 1000 - is this OK?

4) This is also at the default of 5.

Thanks

[James_Slack]

Is there a way to drain these queues down?

Like if I was to change the archive schedule to not archive new items or something?

Since this morning the

A5s have gone down a bit

A6s have gone up a bit

A7s have gone up 1 each.

[EdLacey · Accepted solution]

If your server has the recommended rather than the minimum amount of CPU's for the version of EV you're running and they're not maxed out during an archive task then definitely increase Archive Processes but be cautious. First, try increasing the number to equal the amount of CPU's (I'll assume 8) then monitor performance during an archive run. If CPU's/Memory aren't maxed out try increasing to 10 then 12.

Have a read of the relevant Performance Guide for your version of EV

http://www.symantec.com/docs/DOC2200

If your EV server is virtualised take a look at this best practise guide

http://www.symantec.com/docs/TECH180094

[James_Slack]

Hi,

We are nowhere near maxing out the CPU, we average 8% with tiny bursts up to 20%.

I'll have a read through the doc and whack it up and see if it helps.

Thanks

[James_Slack]

Thanks, the guide says up it and if you have 4 cores then you should be on at least 10. I started with 10 and then tried 12, it's making little to no difference (I have restarted the services).

Is there an upper limit or is this purely a hardware limit.

Can I set 20? 30?

It's 2x Intel Xeon X5650s (6 cores / 12 threads per server @ 2.66 GHZ) so a total of 12 cores / 24 threads.

The box has 48GB of ram of which we have 11gb free right now.

Thanks

[WiTSend]

One of the 1st things I would do is change the archiving schedules to a given non-working hour timeslot each day.  If you are sized correctly you shoudl be able to do all daily archiving in an 4-8 hour window per day.  That would free up the resources to perform the other tasks. 

Generally I run the archiving window from 1 hour after the normal workday end to midnight.  Then I schedule the backups of EV 1 hour after archiving ends.  I.e...  archiving window 7pm-12am ,  backup 1am-4am window.  All other processes can clear the queues with the available cycles.

[EdLacey]

Upper limit is 25.

When you say you tried 12 and it made little to no difference do you mean you weren't seeing the A5 queue go down any quicker?

[James_Slack]

2 problems with that WiT Send.

1) We are a 24x7 Global business. There is no after hours and no quiet period.

2) The backup job is run within a separate system and we are unable to control when it starts (starts when there is a gap in the quue of about 400 servers being backed up). Though to be clear, we only backup the open partition. All of the closed stuff I deal with manually.

[James_Slack]

EdLacey, I could not see the queues clearing faster and there was certainly no boost in CPU utilization.

I've just rebooted the server and will leave it over night and see how it looks in the morning.

Thanks

[James_Slack]

Actually Ed, it does seem the A5 queues are going down a bit quicker now thanks. Will see how it looks tomorrow morning.

A6 and A7 remain about the same.

[WiTSend]

Regardless of the global nature of the operations (mine is as well) it works very well to confine the archiving window to a 6-8 hour window and allow cycles for the other processes.  In my experience this methodology will give the most efficient performance to EV.

[James_Slack]

Hi Wit Send, This was how we had our archiving (8 hour window) before but I was told by the consultant who came in that this is no longer the case with EV10. This is why we had it set to 24x7.

I have now dropped it down to have 15 minutes of stoppage every 4 hours.

I think I am going to close this case and log a new one specifically for the A6 and A7 queues.

Since making Ed's change the A5 queue is dropping now and while it is not empty yet, I expect it will be soon enough.