02-25-2014 01:49 AM
Hi,
I have 2 servers and looking in the Message Queuing I can see a backlog of messages in there.
Server 1
A5 = 4240
A6 = 10329
A7 = 419
Server 2
A5 = 1341
A6 = 6786
A7 = 202
I have done some basic things to see if they will go down, change the schedule, stop archiving for a bit, reboot etc. Nothing. It seems this lot is stuck in the queues.
My questions are this.
1) What are each queue used for?
2) How do I clear down each queue and would we lose data doing so?
Solved! Go to Solution.
02-25-2014 04:44 AM
02-25-2014 06:44 AM
If your server has the recommended rather than the minimum amount of CPU's for the version of EV you're running and they're not maxed out during an archive task then definitely increase Archive Processes but be cautious. First, try increasing the number to equal the amount of CPU's (I'll assume 8) then monitor performance during an archive run. If CPU's/Memory aren't maxed out try increasing to 10 then 12.
Have a read of the relevant Performance Guide for your version of EV
http://www.symantec.com/docs/DOC2200
If your EV server is virtualised take a look at this best practise guide
http://www.symantec.com/docs/TECH180094
02-25-2014 03:20 AM
Sorry for lack of info here, had someone waiting for something over my shoulder during writing the above.
This is Vault 10 on 2008R2. Archiving seems to be happening OK even though there seems to be items stuck in these queues.
The reason I found these is I keep getting failed items when I am importing PSTs. This may be unrelated to that issue, but I found it and I guess it's not right.
02-25-2014 04:44 AM
02-25-2014 05:12 AM
Argh. Page just swalloed my reply!
SQL is fine, 0% CPU with the odd 1% / 2% blip. Memory is fine too.
Looking at the A5 queues I see a mix of live users, deleted users (that I need to tidy up I guess) and "deprovisioned" users (who should all be live users as far as EV is concerned.
A6 probably shows similar but of all the ones I checked they were live users
A7 shows a mix of deleted, deprov, disabled and probably some live users (from memory having just lost my lengthy reply).
What do I do next please?
02-25-2014 06:15 AM
You can quite safely purge the A5 queues which will then give the other 2 queues a chance to purge themselves.
As your A5 queues are not clear at the end of a scheduled archiving task you need to tune the relevant tasks.
1. Expand the archiving window during the week
2. Can you run the archiving tasks all day during the weekend?
3. Decrease the amount of items archived per pass if you can't do either of the above (default is 1000)
4. On the Advanced tab of the Storage Service is the Archive Processes value still the default of 5?
02-25-2014 06:29 AM
Hi,
The archiving schedule was actually set to 24x7.
I have only just today (on advice I found searching before posting here) changed my schedule to stop for 15 minutes every 4 hours.
I'll just look up the other settings now...
02-25-2014 06:33 AM
Hi,
3) It is still at the default of 1000 - is this OK?
4) This is also at the default of 5.
Thanks
02-25-2014 06:36 AM
Is there a way to drain these queues down?
Like if I was to change the archive schedule to not archive new items or something?
Since this morning the
A5s have gone down a bit
A6s have gone up a bit
A7s have gone up 1 each.
02-25-2014 06:44 AM
If your server has the recommended rather than the minimum amount of CPU's for the version of EV you're running and they're not maxed out during an archive task then definitely increase Archive Processes but be cautious. First, try increasing the number to equal the amount of CPU's (I'll assume 8) then monitor performance during an archive run. If CPU's/Memory aren't maxed out try increasing to 10 then 12.
Have a read of the relevant Performance Guide for your version of EV
http://www.symantec.com/docs/DOC2200
If your EV server is virtualised take a look at this best practise guide
http://www.symantec.com/docs/TECH180094
02-25-2014 06:49 AM
Hi,
We are nowhere near maxing out the CPU, we average 8% with tiny bursts up to 20%.
I'll have a read through the doc and whack it up and see if it helps.
Thanks
02-25-2014 07:38 AM
Thanks, the guide says up it and if you have 4 cores then you should be on at least 10. I started with 10 and then tried 12, it's making little to no difference (I have restarted the services).
Is there an upper limit or is this purely a hardware limit.
Can I set 20? 30?
It's 2x Intel Xeon X5650s (6 cores / 12 threads per server @ 2.66 GHZ) so a total of 12 cores / 24 threads.
The box has 48GB of ram of which we have 11gb free right now.
Thanks
02-25-2014 09:01 AM
One of the 1st things I would do is change the archiving schedules to a given non-working hour timeslot each day. If you are sized correctly you shoudl be able to do all daily archiving in an 4-8 hour window per day. That would free up the resources to perform the other tasks.
Generally I run the archiving window from 1 hour after the normal workday end to midnight. Then I schedule the backups of EV 1 hour after archiving ends. I.e... archiving window 7pm-12am , backup 1am-4am window. All other processes can clear the queues with the available cycles.
02-25-2014 09:06 AM
Upper limit is 25.
When you say you tried 12 and it made little to no difference do you mean you weren't seeing the A5 queue go down any quicker?
02-25-2014 09:09 AM
2 problems with that WiT Send.
1) We are a 24x7 Global business. There is no after hours and no quiet period.
2) The backup job is run within a separate system and we are unable to control when it starts (starts when there is a gap in the quue of about 400 servers being backed up). Though to be clear, we only backup the open partition. All of the closed stuff I deal with manually.
02-25-2014 09:15 AM
EdLacey, I could not see the queues clearing faster and there was certainly no boost in CPU utilization.
I've just rebooted the server and will leave it over night and see how it looks in the morning.
Thanks
02-25-2014 09:29 AM
Actually Ed, it does seem the A5 queues are going down a bit quicker now thanks. Will see how it looks tomorrow morning.
A6 and A7 remain about the same.
02-25-2014 11:09 AM
Regardless of the global nature of the operations (mine is as well) it works very well to confine the archiving window to a 6-8 hour window and allow cycles for the other processes. In my experience this methodology will give the most efficient performance to EV.
02-26-2014 01:26 AM
Hi Wit Send, This was how we had our archiving (8 hour window) before but I was told by the consultant who came in that this is no longer the case with EV10. This is why we had it set to 24x7.
I have now dropped it down to have 15 minutes of stoppage every 4 hours.
I think I am going to close this case and log a new one specifically for the A6 and A7 queues.
Since making Ed's change the A5 queue is dropping now and while it is not empty yet, I expect it will be soon enough.