fsautility bulk recall crashes file server (?)
I've done several small successful tests of the EV 9.0.2 fsautility bulk recall to my file servers. Last night I tried a rather large one (approx 300GB). It ran for nearly 11 hours until around 9AM my file server locked up and became nonresponsive. Everyone lost access to their mapped drives. The backup that was still running failed. The bulk recall failed. I'm not sure if the file server lock up caused the bulk recall failure or if the bulk recall caused the file server lockup.
What I do know is the file server looked to lock up before the bulk recall failed b/c my backup was the first thing to go.
I had a couple of 41145 ("input string not in the correct formtat") and 40966 ("object reference not set to an instance of an object") events on my EV server. I had one strange event on the file server that appeared nearly 20 minutes after the errors on the EV server:
Event Type: Warning Event Source: Srv Event Category: None Event ID: 2027 Date: 1/31/2012 Time: 9:21:17 AM User: N/A Computer: CHH-FS2 Description: The server has detected a potential Denial-of-Service attack caused by consuming all the work-items. Some connections were disconnected to protect against this. If this is not the case, please raise the MaxWorkItems for the server or disable DoS detection. This event will not be logged again for 24 hours.
I was wondering if any registry tweaks need to be made to the file server before running a large bulk recall. When we were running 8.0.x, we used a powershell script that Symantec TS provided us that basically did what fsautility bulk recall does. The documentation the provided me gave me a list of registry keys to update on the target file server prior to running it. Is it possible/likley I need to make the same changes now?
The following needs to be set in the registry on the file server under HKLM\SOFTWARE\Wow6432Node\KVS\Enterprise Vault\FSA\PlaceholderService so that the recalls can be processed more quickly and not be limited.
BypassRecalLimitsForAdmins set to 1
RecallLimitMaxRecalls set to 99 decimal
RecallLimitTimeInterval set to 1
RevertItemsToPlaceholders set to 0
Also for every file listed in the resulting XML file from the fsautility.exe run, the error message was
Error in determining whether the file is a placeholder - Failure
Parameters for my fsautility run were
C:\Program Files\Enterprise Vault>FSAUtility.exe -b -s \\<server>\X$\<folder> -recurse
hmm, very interesting. It's not something that I know a vast amount about, unfortunately. I'd suggest contacting Symantec Support to discuss OS versions and the like, and whether or not there is anything that can be done to work around the problem that you faced.