Set/Clear backup mode on EV 9.03 failing - items awaiting backup

[Itegral]

Can you please help me progress….

 I have migrated a production user with 13GB archives (between 13-18GB) and archive items are 194,140. When we put the Vault Store in backup mode (GUI) it works OK but to clear just goes for ever and we had to kill the process and restart the server/services. 

I have noticed that StorageDelete process is being failed in the event log and found the following TECH;#

The Enterprise Vault (EV) Storage Delete processes may fail with an 'Out of Memory' condition when starting. 

|

http://www.symantec.com/docs/TECH175979

How to fix it? The TECH doesn’t tell the resolution. 

Thanks,

Ausaf

[JesusWept3]

If you read the technote it shows that it's a known issue (bug) that willl be resolved at some point, but as of yet due to its low priority they haven't committed to getting it into any particular service pac Your best bet is calling in a case, refer them to that technote and etrack, have them add you to the etrack , more people that have the issue the more they will look at it, also the etracks may mention updated workarounds not mentioned in the technote Also subscribe to the technote so if it gets updated you will get emailed the info

[jim_leggett]

At this time there does not appear to be a workaround for the issue.

However, I would recommend that you are on EV 9.0 SP3 so you may be able to get on the front end of or assist with any verification of any fixes coming out for it. Or you can do as JW2 suggested and subscribe to the TN and you will be updated when a fully verified fix is available. Either way I would recommend calling into support and getting the issue documented.

[beta_dodo]

Hi,

It's not clear from your comment if you are actually seeing an out of memory error in your event log.  Are you?

That tech note states that the out of memory is related to a large number of archives.  Is that what you have?

Perhaps post number of archives and details of the errors would be useful.

Mike

[Itegral]

When I run the report on Vault Store, I can see 161,063 saveset waiting to be backed up.

However, the backup tab on the partition is not telling me any unsecured items.

Putting in the backup mode goes OK but clearing the backup mode takes very long and I can see Storage File Watch errors in Event ID 6848 i.e. the collector manager is not running. Check the Storage Servive is running and no previous errors have been logged. The Storage service may require a restart (0xc0041ac1).

Also, there is another Event ID: 13345 - an error was detected while accessing the vautt database .....

[Itegral]

I still have items awaiting backup. Is there any hotfix for Archived items remained in a 'pending' state situation. I know a hotfix is available for 9.02 but how about 9.03?

When putting the Vault Stores in backup mode, I receive the event 6659 ;

Storage Manage Process (1)  -  Terminating process

PID: 3124 

The process has failed to shutdown normally and is being terminated 

For more information, see Help and Support Center at http://evevent.symantec.com/rosetta/showevent.asp?EvtID=6659

--------------------------------------------------------------------------------------------------------------------------------------

Event: 13345

An error was detected while accessing the Vault Database 'Provider=SQLOLEDB;Server=LONS001xxx.xx.xxx\EV;Database=EVVSMailVaultStore_20;

Integrated Security=SSPI' (Internal reference: {CADODataAccess::ResetParameters} [.\ADODataAccess.cpp, lines {2300,2303,2309},

built Nov 11 21:02:49 2011]), error code <0x800a0e7f>. 

SQL Command:

-------------------------------------------------------------------------------------------------------------------------------------

Event: 4228

An error occurred in thread pool used for processing requests.

Further error information: Timeout/wait failure occurred while waiting for [4] threads to finish execution.

Internal reference: EVCommon::EVThreading::CThreadPool::WaitForCompletion/WaitForMultipleObjects error 

For more information, see Help and Support Center at http://evevent.symantec.com/rosetta/showevent.asp?EvtID=4228

------------------------------------------------------------------------------------------------------------------------------------

[John_Chisari]

Looks like you are getting a different issue to the technote you are specifying.

Hotfixes are usually rolled up into the next service pack, but it depends on the timing of when the hotfix is available and when the service pack is released.

as above, would suggest calling into support and having them look further into the issue.

[Itegral]

correct John - I disabled all the archiving tasks and stop them as well.

This, in my opinion, released some resources for the StorageDelete process. I didnt see the above StorageDelete process failing error later but putting the Vault Stores in to back up mode takes long and I am getting the above three events. Clearning the backup mode takes longer i.e. can even take some hours....

Also, restarting the storage service is a slow process and getting event ID: 4234

[John_Chisari]

Few hours, nice.

ok, would suggest you gather a dtrace of all the storage processes, StorageManagement, StorageDelete, StorageArchive etc and logging your case.  You might also need to take a SQL profiler whilst it is occuring.

[Itegral]

with support now...

[Itegral]

 I disabled all the archiving tasks and stop them as well.

This, in my opinion, released some resources for the StorageDelete process. I didnt see the above StorageDelete process failing error later but putting the Vault Stores in to back up mode takes long.

Please see the attached dtrace logs and events screen-shots.

[Itegral]

i have been asked to generate dtrace logs which I have done and sent it out to Symantec.

The support I spoke with didnt seem to have very indepth knowledge of the EV product.

The dtrace logs are attached here-with for you guys to please assist - thanks.

Also attached are the event logs appeared when I tried clearning the backup mode. I dont think the Storage Watch File process is doing its work when putting in backup mode and clearing the backup mode.

[Prone2Typos]

Not sure if it helps but the following appears to be an excerpt of your log showing the issue:

57740    18:30:40.292     [7296]    (StorageFileWatch)    <4196>    EV:L    {EVCommon::EVThreading::CThreadPool::GetRequestCountForFilter} (Entry)
57741    18:30:40.292     [7296]    (StorageFileWatch)    <4196>    EV:L    {EVCommon::EVThreading::CThreadPool::GetRequestCountForFilter} (Exit)
57742    18:30:40.292     [7296]    (StorageFileWatch)    <4196>    EV~E    Event ID: 4234 Thread pool (InstanceId = [1]) could not stop [1] requests. Total requests attempted to be stopped = [1]. |
57743    18:30:40.292     [7296]    (StorageFileWatch)    <4196>    EV:L    {EVCommon::EVThreading::CThreadPool::StopRequests} (Exit)
57744    18:30:40.292     [7296]    (StorageFileWatch)    <4196>    EV:L    {EVCommon::EVThreading::CThreadPool::WaitForCompletion} (Entry)
64942    18:40:40.339     [7296]    (StorageFileWatch)    <4196>    EV~E    Event ID: 4228 An error occurred in thread pool used for processing requests. |Further error information: Timeout/wait failure occurred while waiting for [4] threads to finish execution. |Internal reference: EVCommon::EVThreading::CThreadPool::WaitForCompletion/WaitForMultipleObjects error |
64943    18:40:40.339     [7296]    (StorageFileWatch)    <4196>    EV:H    EVCommon::EVThreading::CThreadPool::WaitForCompletion Error: Timeout/wait failure occurred while waiting for [4] threads to finish execution.
64944    18:40:40.339     [7296]    (StorageFileWatch)    <4196>    EV:L    {EVCommon::EVThreading::CThreadPool::KillThreads} (Entry)
64945    18:40:40.339     [7296]    (StorageFileWatch)    <4196>    EV:H    EVCommon::EVThreading::CThreadPool::KillThreads Information: Terminated a thread.
64946    18:40:40.339     [7296]    (StorageFileWatch)    <4196>    EV:L    {EVCommon::EVThreading::CThreadPool::KillThreads} (Exit)
64947    18:40:40.339     [7296]    (StorageFileWatch)    <4196>    EV:L    {EVCommon::EVThreading::CThreadPool::WaitForCompletion} (Exit)
64948    18:40:40.339     [7296]    (StorageFileWatch)    <4196>    EV:L    {CVSDBAccessor::CloseConnection} (Entry)
64949    18:40:40.339     [7296]    (StorageFileWatch)    <4196>    EV:H    {CADODataAccess::DeleteParameter} HRXEX fn trace : Error [0x800a0e7f], ref(1): [DP1], [.\ADODataAccess.cpp, lines {2320,2324}, built Nov 11 21:02:49 2011].
64950    18:40:40.339     [7296]    (StorageFileWatch)    <4196>    EV~E    Event ID: 13345 An error was detected while accessing the Vault Database 'Provider=SQLOLEDB;Server=LONS001I0061.bnppi.priv\EV;Database=EVVSMailVaultStore_20;Integrated Security=SSPI' (Internal reference: {CADODataAccess::DeleteParameter} [.\ADODataAccess.cpp, lines {2320,2324}, built Nov 11 21:02:49 2011]), error code <0x800a0e7f>.| |SQL Command:| N/A |
64951    18:40:40.339     [7296]    (StorageFileWatch)    <4196>    EV:H    {CADODataAccess::ResetParameters} HRXEX fn trace : Error [0x800a0e7f], ref(1): [RP2], [.\ADODataAccess.cpp, lines {2300,2303,2309}, built Nov 11 21:02:49 2011].
64952    18:40:40.339     [7296]    (StorageFileWatch)    <4196>    EV~E    Event ID: 13345 An error was detected while accessing the Vault Database 'Provider=SQLOLEDB;Server=LONS001I0061.bnppi.priv\EV;Database=EVVSMailVaultStore_20;Integrated Security=SSPI' (Internal reference: {CADODataAccess::ResetParameters} [.\ADODataAccess.cpp, lines {2300,2303,2309}, built Nov 11 21:02:49 2011]), error code <0x800a0e7f>.| |SQL Command:| N/A |

The code appears to be on teh DB side of things

http://msdn.microsoft.com/en-us/library/ee275033%28v=bts.10%29.aspx

and

http://msdn.microsoft.com/en-us/library/windows/desktop/ms677004%28v=vs.85%29.aspx

I have not seen it before from memory but it states:

Operation cannot be performed while executing asynchronously.

I would state that you will likely need a profiler and this will have to be further investigated by support

[Itegral · Accepted solution]

support wasnt sure where the issues lies as they suspect fingerprint database could be at fault.

I ran the repair and it went ok.

[allanh1]

Hi,

We are getting the same issue since the upgrade to 9.0.3 and support cant seem to wortk the answer.

When you say " i ran the repair and it went ok" what repair did you run?  Did this resolve the issue completly?

Thanks

[Nathan_Clark_2]

This should sort it:

http://www.symantec.com/business/support/index?page=content&id=TECH180841