Event 7109 - Problem with Savesets not found by StorageFileWatch.exe

[GertjanA]

EV80SP2

Loads of errors 7109:
Event Type: Warning
Event Source: Enterprise Vault
Event Category: Storage File Watch
Event ID: 7109
Date:  1-12-2009
Time:  12:44:56
User:  N/A
Computer: vaultservername
Description:
An invalid saveset was encountered while doing watch file scan. This archive operation will be attempted to be cancelled. Following further information is available.
 Partition Name = MBX41x3 Ptn4
 VaultStoreEntryId = 13C9685678F71B64CB22179B468460A3E1210000DMS
 TransactionID = D00D6BDFA7072EA764909E74EC5FBEB1
 Invalid Reason = File 'F:\EVStorage\VaultStore\MBX41x4 Ptn4\2009\10-16\D\00D\D00D6BDFA7072EA764909E74EC5FBEB1~B3~871FD70F~00~1.DVSSP' not found for SISPartIdentity [181847]

File indeed does not exist, there is however a file called the same, having extention archdvssp

This causes the storagefilewatch.exe to run up to 1.7GB memory usage. I stopped the services. Could evsvr help out?
Backup's (if any good left)
Or just add more memory, and keep my fingers crossed?

Thanks.

Gertjan

[Michael_Bilsbor]

hiya,
what makes you think the high memory usage is related to this issue (by the way that amount of memory doesn't sound right at all.....)

 

[GertjanA]

Hi Mike,

hunch, and prove (sort of)

Event Category: Storage File Watch

What happens is the following
backupprogram runs cmd file that places EV-index+store in backupmode.
backup
backupprogram runs cmdfile to first set triggerfile in partitions, then takes ev out of backupmode. (check to see if backup succesfull or not before placing triggerfile)

When EV comes out of backupmode, collect/migrate starts, as does some other stuff. What happens on this 1 server (the other 14 run fine) is that storagefilwatch.exe starts fine. It rattles away, doing it's job (I see the number of items awaiting backup going down from 1.947.333 to 1.932.250 in an hour)
Then, it starts logging the 7109 errors. (many really quick) 
After about 10 minutes, I see StorageFileWatch.exe running at up to a max of 96%cpu time, consuming more and more memory.
Additionally, there is an event logged (not at hand sorry), which indicates a virtual memory error (something with unable to proces request due to vm being low)
Restarting storageservice helps for a shortwhile. I need the SFW.exe to finish. My other 14 servers have less than 25.000 items awaiting backup (normal average operation between backup and archiving window). This one server sits on over 1.9 million.
I do not know what happened, but suspect an issue with backup somewhere. Due to the amount of events, I cannot go back in the eventlog anymore. I do have a case (also, number not at hand)
I now have added 2GB more to the server (running now at 4GB RAM). Hopefully it cleans out tonight. Backup has been disabled, tasks are in report mode. I fear I need to run EVSVR, but the manual is not too clear.

More tomorrow.

Thanks.

[GertjanA]

Hi Mike.

If you can speed up, please do

Thanks.

[GertjanA]

The answer of collections enabled is what I was hoping for, I have come across this issue a few times.
 
I have a couple of SQL scripts which can be run to identify if it is the same issue. The scripts have changed slightly recently so I need to confirm I am sending the most updated ones to you, but at the moment I am copying data to a new laptop so will be tomorrow morning, I hope this is OK. Also I feel it would be good to allow the items awaiting to reduce so we are only left with the entries causing the 7109 warnings.
 
Below is brief explanation of what may have occurred
 
Some SIS parts are not marked as secured in Fingerprint DB. However they are removed from WatchSISPartFile in VaultStoreDB. Later if another Saveset shares with those SIS parts, StorageFileWatch assumes they (shared SIS part) are unsecured. Later on SIS part gets collected (as it is actually secured, parent saveset for that SIS part is secured). When StorageFileWatch runs and looks for this SIS parts it can't locate it as it is in collection and hence WatchFile raises that warning.


The engineer will send me the scripts as soon as he finds them ;)
Glad to see this is indeed an issue....

[GertjanA]

The events logged are
Event Type:    Error
Event Source:    Enterprise Vault
Event Category:    Storage File Watch
Event ID:    6265
Date:        1-12-2009
Time:        17:10:00
User:        N/A
Computer:    vaultservername
Description:
Failed to open the IArchivingAgentUpdate interface to the Enterprise Vault ExchangeArchivingAgent Class COM component
Reason: Not enough storage is available to complete this operation.  [0x8007000e]

For more information, see Help and Support Center at http://evevent.symantec.com/rosetta/showevent.asp

Event Type:    Error
Event Source:    Enterprise Vault
Event Category:    Admin Service
Event ID:    4183
Date:        1-12-2009
Time:        17:22:37
User:        N/A
Computer:    vaultservername
Description:
There have been more than 10000 Errors in the last 7200 seconds, of which 90% or more are from Enterprise Vault.

 Enterprise Vault will now shut down.

For more information, see Help and Support Center at http://evevent.symantec.com/rosetta/showevent.asp

[GertjanA · Accepted solution]

Support gave me to registry keys, one called Critical, 1 called Warning. They basically stop EV from monitoring the eventlog, and keep on working.

On the affected server, I had tasks running in report mode for almost 12 days. That gave ev the time needed to process the 2.2 million items that were alledgely in not backed up. I saw StorageFileWatch running for several hours at high cpu and memory usage, but luckily the number did drop. There is still reports of the 7109 in eventlog, but at least the number is down.

[A_Z_rcher]

Hi GertjanA

Did the support give you some information when this will be fixed? (I reported this with SP2 already and I was told it will be fixed with SP3)
Monitoring the log is one thing, having many events 7109 mentioning data loss is another..

How to get rid of these annoying events?
Do we have to be concerned about the event?

regards