Forum Discussion

Francis-T's avatar
Francis-T
Level 4
12 years ago

Intermittent archived item retrieval failure

Hi,

 

Have an intermittent issue that seems to be occuring a few times a week, no services stop, but a  restart of the services or server gets things running.

Seems to be 3 Enterprise Vault events (all with a Task Category of Web Application WP) on the EV server that tie in with when archive items start to fail.

In order they are;

 

AutoStorageOnline error.

Reason: Not enough storage is available to process this command. [0x80070008]

Reference: [CTSOL]

 

AutoStorageOnline error.

Reason: Not enough storage is available to process this command. [0x80070008]

Reference: [CO]

 

Unable to fetch item from "EVSERVER1.domain.wan".

Reason: Not enough storage is available to process this command. [0x80070008]

Saveset Id: 201302090437169~201301101307220000~Z~C123407DDFF16F5881C3C3D29B10AAC1

Archive Name: user.name

Archive Folder Path: ?Inbox

Reference: [GOAFS]

 

We found a couple of KB's that talk about free space, which isn't an issue on this server, and also about the amount of items in %TEMP% locations, but they've been cleared down as well.

Environment is 2 x EV 8.0.5.1048 servers. 1 for the journal and 1 for the user archives, this issue is on the server used for user archives.

OS is 2008 SP2 x64.

There are other issues relating to backups and the time they are taking (2-3 days for a full at present), still an ongoing piece of work to address this.

Exchange is 2007 SP3 RU6 on a 2003 R2 x64 cluster

SQL is 2005 on a 2003 R2 x64 cluster

Beyond the couple of KB's that didn't get us anywhere, now at a bit of a dead end for what to look at next.

Hoping to get a DTrace when trying to retrieve items the next time it happens and look to log a support call as well if I'm no getting anywhere.

 

Anything else I should be looking at, or anything obvious I've missed?

 

Cheers

 

Enterprise Vault
Error messages
Information Governance
Troubleshooting
  • Francis-T's avatar
    Francis-T
    12 years ago

    Morning Ryan,

     

    Had almost forgotten about this thread, still having intermittent issues with EV though, the previous fixes did nothing.

    Do believe Symantec Support have now given us the root cause of the issue though, and it does tie in with your fix;

    http://www.symantec.com/docs/TECH155407

    Do you have a 64bit SharePoint web front end that is archived by EV?

    I was getting hung up on the EV backup, but now that Symantec highlighted the KB and SharePoint interaction, that would explain our issue.

    We've been monitoring the handles of storageonlineopns.exe proactively and getting alerts when it hits 20,000, that way we can restart the service and keep EV running before users start to have issues.

    In the meantime we're just confirming the trend of it's handles spiking during the SharePoint backup, but ultimately it does look like we'll need to upgrade.

  • Ryan_Seymour's avatar
    Ryan_Seymour
    Level 2
    12 years ago

    Hi Francis-T

    Has this issue been resolved since the recreation of the IIS EnterpriseVault Directory? We have the exact same issue on our EV FSA Server with the retrieval of Placeholders. I am interested to know so that we can do the same on this server and see if it fixes the problem before we go the route of logging a call with Symantec Support.

    Many thanks,

     

  • Francis-T's avatar
    Francis-T
    Level 4
    12 years ago

    Unfortunately not Ryan.

    It does appear to have improved it slightly though, previously it would happen every few days, after the recreation of the EnterpriseVault directory we got 6 days before it reoccurred.

    In that time we could see StorageOnlineOpns.exe resources slowly creeping up as far as handles and non-paged pool.

    Reached 24,000 handles and 2,900K NP Pool before it started failing.

    At that time, the EVsite was still accessible, restoring an item from the vault back to the mailbox worked (as opposed to just double clicking an item to view it in the vault).

    EV searches also worked.

    Archive Explorer lists items, but gives an error when retrieving.

    Managed to get a DTrace as well which is now in the hands of Symantec support to hopefully give them some indication of what is going on.

  • Ryan_Seymour's avatar
    Ryan_Seymour
    Level 2
    12 years ago

    Hi Francis-T

    To add my experience to this, we upgraded from 9.0.2 to 10.0.3 on the weekend of the 9th and since then we have not had so much as a spike on the storageonlineopns.exe process on our FSA EV server. This appears to have fixed it but we are only 1 week in and have had this length of up time in the past. However we would have seen the process spike to thousands of handles and 700 threads so I am very optimistic at this point.

    Not sure what level of support you have with your EV vendor and if this upgrade is available to you but maybe chat to the guys that are handling your support ticket and see what they think?

  • Francis-T's avatar
    Francis-T
    Level 4
    12 years ago

    Morning Ryan,

     

    Had almost forgotten about this thread, still having intermittent issues with EV though, the previous fixes did nothing.

    Do believe Symantec Support have now given us the root cause of the issue though, and it does tie in with your fix;

    http://www.symantec.com/docs/TECH155407

    Do you have a 64bit SharePoint web front end that is archived by EV?

    I was getting hung up on the EV backup, but now that Symantec highlighted the KB and SharePoint interaction, that would explain our issue.

    We've been monitoring the handles of storageonlineopns.exe proactively and getting alerts when it hits 20,000, that way we can restart the service and keep EV running before users start to have issues.

    In the meantime we're just confirming the trend of it's handles spiking during the SharePoint backup, but ultimately it does look like we'll need to upgrade.

  • John_Santana's avatar
    John_Santana
    Level 6
    12 years ago

    I was facing this issue as well but with FSA agent on my file server, the fix was to re-register the ASP.dll and then restart IIS service.

    hope that helps.

  • Rob_Wilcox1's avatar
    Rob_Wilcox1
    Level 6
    12 years ago

    That particular issue relating to 64 bit SharePoint was fixed quite some versions ago, so as with anything it's good to upgrade to the latest (or nearly the latest) versions of the product.

  • Ryan_Seymour's avatar
    Ryan_Seymour
    Level 2
    12 years ago

    Hi All

    The article mentioned here by Francis-T explains the issue we have seen 100%. Our FSA Server is also running our SharePoint Archiving. Our SharePoint Farm is running on 2010 and our 2 front ends are 64 bit servers. I am certain that had we come across that article earlier and applied the patch mentioned to the relative servers on our previous 9.0.2 version of EV it would have resolve the memory leak we were seeing with the StorageOnlineOpns.exe process.

    Thanks to all for thier input here. It has been extremely valuable and I hope this post helps others out there that may come across the same issue in time.