Intermittent archived item retrieval failure

OK So when its talking about Not Enough Storage Is Available, its really talking not enough memory
So when it happens, open up a quick task manager, have it show handles and it have it show memory too (including non paged pool) and then see if anythings taking excessive amounts of memory or file handles

so for instance, you might just be seeing a slow mapi leak in the archive task, and when you restart it, its clearing that memory and such which is why it starts working again

Now, if it does happen to be a memory leak, its very doubtful you will get a hotfix, and will probably be given a workaround, such as restarting the tasks once a day, or restartallmapitasks type registry keys to try and stem the problem.

You're quite right. 

I had looked at system handles at one point and forgotten about it completely. 

About 2 hrs after a services restart, a process called something like storageopenns.exe (forgotten the name and not in a position to check at the minute unfortunately) was up around 16000 handles. I'd normally be suspicious of anything over 5000 so did mean to look at it closer. 

I googled it at the time and it was a process called during the retrieval process, meant to check further if it was a likely cause or just a symptom during the backup window. 

Also compared the same process on another site and it was down around 600, but outside of their backup window. 

Caved in and logged on to check, process is StorageOnlineOpns.exe (32bit).

It's up to 23000 handles now & 2946K non-paged pool memory.

Gave the archiving task a quick restart, but didn't affect it, I'd guess going by the name I need to restart the storage service, but thats going to impact the backup window which possibly has another 2 days to run!

hmmm so storageonlineopns is used for opening items from the vault
things that will invoke it are:
PST Exports through the VAC
PST Exports through Discovery Accelerator
Vault Cache builds from clients
Opening items through Browser Search and Archive Explorer
Opening items through shortcuts in the users mailbox
Index updates and rebuilds
Move Archive from one vault store to another store or to another ev site

So i guess the question is do you use Vault Cache and Virtual Vault in your environment?
Do you have any index rebuilds going on?
Do you use Discovery Accelerator in your environment?
What is the EV Server used for ? mailbox archiving? Journaling? both?
Are you doing any move archive operations?

No Index rebuilds or move archive operations.

Don't use Vault Cache, Virtual Vault or Discovery Accelerator.

This EV server is for Exchange mailbox archiving and it also has a SharePoint archviing task, but the SharePoint Task is configured not to run against the site schedule or it's own schedule.

Journal archiving is completed via a 2nd EV server.

There would be almost no users online currently, so the only thing that should be actively hitting EV would be the backups.

The email archive would be around 2.5TB currently, partitions have been closed in the past, but I couldn't honestly say how big the current open partition is.

Something else being looked at is the requirement for leavers archives, there would be considerable benefit to taking them out of EV and taking a one off backup of exported PSTs. It would reduce both the size and the volume of small items that are killing the backup currently, unless we magic up a better way to improve the backups that is

Backup wasn't a full, so it's finished already.

StorageOnlineOpns.exe is still high as far as resources go.

23,801 handles

2946K NP Pool (closest is down at 430K)

Server itself is under almost no load, 2% CPU & 3GB used from 16GB total

Reports from users that they were starting to have issues again, nothing in the Event Logs this time (possibly caught it earlier this time?).

Restart of the services and all is grand again, StorageOnlineOpns.exe is down to 1544 handles and 184K non-paged pool memory.

I'll see about getting a call logged now

Symantec support recreated the IIS EnterpriseVault directory.

Keeping an eye on it now, if the fault re-occurs I've to confirm if the site becomes inaccessible as well and any error it presents.