Storage Service Keeps Stopping

[junecastillote]

We have Enterprise Vault 8.0.3.1845 running on Windows 2003 SP2 Enterprise.

The issue we are seeing is exactly the one described in this Article: http://www.symantec.com/business/support/index?page=content&id=tech57598

"Enterprise Vault (EV) Storage and Task Controller services stop because Message Queue (MQ) messages cannot be sent"

According to the article the resolution is to increase the MSMQ storage limit. We've tried to double the value (from 2GB previously to 4GB) and was able to start the Storage Service.. until it stopped again when the storage folder is filled with 4GB data again.

In this case, are we just suppose to keep increasing the storage limit or should we be looking at something else like what is causing the queue to just keep growing and filling up the storage folder? Any recommendations as to what we should consider checking in to?

Also to mention, there have been no successful backups for atleast 2 months now and trying to backup (using BackupExec) just fails. Could this have been contributing to the problem at hand?

[GertjanA]

Hello,

Sounds like you have some issues there.

First, you are aware that in july Windows2003 is no longer officially supported by Microsoft?

I would do the following:

Set the archiving tasks in report mode, restart them. Enlarge the MSMQ memory to 10GB. Verify you have enough free space on the storage locations you use. Verify your MSMQ is not containing tons of items. (via ServerManagement on 2003 I believe, look at the 'private queues'). Most likely you see many items in A5, and possible the Storage queue.

Make a succesfull full backup. If necessary shutdown EV, and do a granular backup (partition by partition), but DO backup. Having no backups is really frightening :-). If you have set your archiving policy to only create shortcuts after backup, you might also see issues in Exchange. If you use the triggerfile, make sure it is set properly. If necessary, create a new VaultStore partition, and close the current ones. Verify your index location is ok in regards to free diskspace.

After backup, give EV time to process the backedup items. That might mean no archiving for a weekend, but let EV do it's work. Once you're back to normal, start archiving again.

Make sure your SQL maintenance is running ok, without issues, and your databases are also backedup regularly.

Upgrade to EV11.0.1 as soon as possible (you will have to go through 9 and 10), so you can use Windows 2012 as OS, and possibly also upgrade your SQL server to SQL2012.

Good luck :)

[junecastillote]

Yes, this server is due for refresh in the coming months, but until then we'll have to manage this one :D

Thank you for the information we will try these.

Since you mentioned the Private Queue, I am attaching a screencap of the current Private Queue, and maybe you can tell me what you think of this figure.

[GertjanA]

Wow! 604224 items to restore?

I am not sure you can purge this queue.

You need to call support, to verify it is possible to purge this queue.

In MSMQ, it should be possible to expand to the content of this queue, and verify the messages in there. Perhaps that gives an indication to which mailbox this is a restore for. If you found it, go to user, and ask wtf he is doing such a massive restore.

I would now:

Set tasks in report mode, restart them. Stop and start the Storage and shopping service. Monitor eventlog to see if events pop up in regards to this Storage Restore/shopping. If the queue drops, let it continue until (close to) empty. When done, restart the tasks in normal mode.

If the queue stays at the 600.000+ level, I would purge it,

Stop EV-services, make sure all EV-processes are also ended. Rightclick queue, purge. when purged, refresh MSMQ, and if the messages are in an 'admin' queue also purge that.

Then start services. Monitor eventlog.

Monitor MSMQ to see it works as expected...

[AndrewB]

have you increased the msmq limits and restarted the EV services? you might also want to temporarily increase the number of restore threads until the backlog in your queue is cleared.