Enterprise Vault offline cache only headers being downloaded

[eeitocr]

Hi,

I am having an issue with Enterprise Vault whereby only headers are being downloaded to the vault cache on all my desktops and the content is not being downloaded.

BITS is enabled on the desktop PCs, wondering does the BITS service also need to be started on the EV server - currently the BITS service is set to manual and not started on the EV server?

Thanks.

[Ben_Heymink]

BITS isn't required on the server; the service is only used client side to pull down a file from a remote server. Can you get a client log of the client when a sync is attempted?

Also, I assume your Desktop policy is actually set to download content to the client machines?

[Rob_Wilcox1]

Also worth checking to make sure the size constraints aren't being exceeded.

[eeitocr]

Hi Ben,

Thanks for your quick response.

Yes, the desktop policy is set to download content, please find client log below.

[eeitocr]

C:\Program Files\Support Tools>bitsadmin /list /verbose

BITSADMIN version 2.0 [ 6.6.2600.2180 ]
BITS administration utility.
(C) Copyright 2000-2004 Microsoft Corp.

GUID: {F87258DE-69BE-459F-AC36-40620D3AC990} DISPLAY: EV: 2012-01-01 to 2012-03-
31
TYPE: DOWNLOAD STATE: SUSPENDED OWNER: Domain\Test User
PRIORITY: LOW FILES: 0 / 0 BYTES: 0 / 0
CREATION TIME: 23/04/2012 09:20:52 MODIFICATION TIME: 23/04/2012 09:20:52
COMPLETION TIME: UNKNOWN ACL FLAGS:
NOTIFY INTERFACE: UNREGISTERED NOTIFICATION FLAGS: 3
RETRY DELAY: 300 NO PROGRESS TIMEOUT: 1209600 ERROR COUNT: 0
PROXY USAGE: PRECONFIG PROXY LIST: NULL PROXY BYPASS LIST: NULL
DESCRIPTION: 122F2297D3B38BF4F92550D305A7E9F151110000Our EV Server
JOB FILES:
NOTIFICATION COMMAND LINE: none

Listed 1 job(s).

C:\Program Files\Support Tools>bitsadmin /resume {F87258DE-69BE-459F-AC36-40620D
3AC990}

BITSADMIN version 2.0 [ 6.6.2600.2180 ]
BITS administration utility.
(C) Copyright 2000-2004 Microsoft Corp.

Unable to resume job - 0x80200003
There are no files attached to this job. Attach files to the job, and then try a
gain.

C:\Program Files\Support Tools>

[eeitocr]

Hi Rob - don't think there is an issue with size constraints - have posted log files.

[Ben_Heymink]

Looks like the client is still waiting to get a 'slot' to download content to the client. The server operates a slot-based mechanism to restrict the number of clients concurrently building/downloading Content to Vault Cache.

So either the server is at full capacity handling download requests for other clients, or the server piece that handles these requests is offline. Since you mention this seems to apply to all clients, my first step would be to ensure that I have the pre-requisite setup on my server; is your cache location configured?

If it is, Dtrace EVMonitoring.exe and look for lines prefixed with 'CCRM'; this is a component that should report if you server is not configured correctly for Vault Cache. Hope this helps!

[eeitocr]

Hi Ben,

Please find attached additional log information - there are numerous RETRY logic started reason '302'
events in the log and no content has been downloaded?

24/04/2012 05:39:57.136[1172]: CONTENT:BUILD: Slot retrying
24/04/2012 05:44:57.127[1172]: CONTENT:DWNL:WEB: Acquiring INITIAL slot base URL 'http://Our EV Server/EnterpriseVault'
24/04/2012 05:44:57.128[1172]: CONTENT:DWNL:WEB: Acquiring INITIAL slot with page 'GetSlotWithServer.aspx' for DBID '1' from '2012-01-01' to '2012-03-31'
24/04/2012 05:44:57.443[1172]: CONTENT:BUILD: We are NOT using the proxy infrastructure
24/04/2012 05:44:57.443[1172]: CONTENT:BUILD: Slot acquired '1E14DD43EFE9DDD4FAB36B8CD171E185F1p10000Our EV Server'
24/04/2012 05:44:57.444[1172]: CONTENT:DWNL: JobURL:http://Our EV Server/EnterpriseVault/DownloadContent.aspx?JobId=1E14DD43EFE9DDD4FAB36B8CD171E185F1p10000Our EV Server
24/04/2012 05:49:57.703[1172]: CONTENT:BUILD: JobId '1E14DD43EFE9DDD4FAB36B8CD171E185F1p10000Our EV Server' BUILT containing '2' items
24/04/2012 05:50:02.226[1172]: CONTENT:BUILD: Completed INITIAL Job filename '2012_01_03_0001.db' state '7'
24/04/2012 05:50:02.226[1172]: CONTENT: RETRY logic started reason '302'
24/04/2012 05:57:44.714[1172]: CONTENT:BUILD:
24/04/2012 05:57:44.714[1172]: CONTENT:BUILD: Content cache entire download started.
24/04/2012 05:57:44.715[1172]: CONTENT:BUILD:

[Ben_Heymink]

It's important to note that seeing 'Retry' in the log is not cause for concern, this is normal operting procedure to retry downloads/calls to the Enterprise Vault server.

Exactly how are you determining that no content has been downloaded? If you go to the user AppData->Local->KVS->Enterprise Vault folder on the client machine and open the folder in there (It will be named something like '41FE580E602B2D458015B88FD619B959', do you see any Data base files with dates as thier file names? If so, this is your Content Cache being downloaded in the background; it could just be that due to your server load/configuration, that it is taking a long time to complete.

If nothing is in there, what does the Add-in UI describe when you attempt a sync? Is it currently synchronising? I'm talking about the following dialog:

If there is an error on this dialog, visit the Vault Cache Properties dialog in Outlook and examine the 'Details' tab; more information may be found here. If none of this helps, then post a complete client log of an affected client, from startup, through initiating a manual sync, and after being left to run for a few minutes.

[eeitocr]

Hi Ben,

Thanks for all your help on this - log file attached.

[Ben_Heymink]

Wowzers. In the future, can you attach log files as a file? Makes searching through them somewhat easier ;)

Also, can you answer my previous questions about exactly how are you determining that no content has been downloaded? If you go to the user AppData->Local->KVS->Enterprise Vault folder on the client machine and open the folder in there (It will be named something like '41FE580E602B2D458015B88FD619B959', do you see any Data base files with dates as thier file names? If so, this is your Content Cache being downloaded in the background; it could just be that due to your server load/configuration, that it is taking a long time to complete.

If nothing is in there, what does the Add-in UI describe when you attempt a sync?

[eeitocr]

Hi Ben,

There are two DB files in the AppData->Local->KVS->Enterprise Vault folder but no additional DB files have been downloaded and the laptop has been constantly connected to the network for 3-4 weeks.

[Ben_Heymink]

Ok, So the next step is to see what is happeneing on the server. DTrace EVMonitoring.exe, save the trace, and search for lines prefixed with "CCRM:". 

From what I can tell from the client log, the server is repeatedly turning away the client. A DTrace of EVmonitoring.exe should tell us why. If that is not feasible, raise the client logging level to maximum so that we can see the HTML response coming back from calls to the server, this will indicate the reason for the continual retrying. After raising the log level, the client will trace out, in detail, responses from the server:

HDR: LoadXMLResponse is <?xml version='1.0' encoding='utf-8'?><MDCSyncResponse hr='0' msg='SUCCESS'><GetSyncSlotResponse SyncSlot='b1efa890-f8bf-4b85-8f91-e83f6b846582'></GetSyncSlotResponse></MDCSyncResponse>

[eeitocr]

Hi Ben,

We use a registry key 'DisablePST=1' to prevent our users from creating PST files in Outlook, have seen some articles stating that this registry key could cause a problem with the offline vault cache - could this registry key be the cause of our issues?

Will get the traces to you later today.

Thanks again for all your help.

[Ben_Heymink]

DisablePST should be OK, its DisablePSTGrow that can cause issues. Regardless, if that had been the case, you would have seen different errors in your client log. 

[eeitocr]

Hi ben,

We are seeing the following in the client log file with maximum tracing selected:-

25/04/2012 10:59:09.395[1120]: Body is <?xml version='1.0' encoding='utf-8'?><ENTERPRISEVAULT UsingProxy='0'><RETRY Reason='CCRM Full' WaitTime='300000'/></ENTERPRISEVAULT>

25/04/2012 10:59:09.397[1120]: CONTENT:DWNL:WEB: XML is '<?xml version='1.0' encoding='utf-8'?><ENTERPRISEVAULT UsingProxy='0'><RETRY Reason='CCRM Full' WaitTime='300000'/></ENTERPRISEVAULT>

Hope to have the DTrace for you shortly.

[eeitocr]

Hi Ben,

Please see extract from DTrace - this message is repeated throughout the DTrace file for all users attempting to connect to the EV server?

369 14:59:15.295  [4968] (EVMonitoring) <47928> EV:L ClientAuthHelperImpl::ValAuthString AuthToken:OuR EV Server L1J6***** ==> User:Domain\Test.User (hr=Success  (0))
370 14:59:15.295  [4968] (EVMonitoring) <47928> EV-H {ContentCacheRequestManager.StartBuild} CCRM: Incoming Build request from: Domain\Test.User
371 14:59:15.295  [4968] (EVMonitoring) <47928> EV-H {ContentCacheRequestManager.GenerateJob} CCRM: No free ContentCacheAssemblers
372 14:59:15.295  [4968] (EVMonitoring) <47928> EV-H {ContentCacheRequestManager.StartBuild} CCRM: Turning away client Domain\Test.User. (The CCRM is running at full capacity)

[Ben_Heymink · Accepted solution]

So nothing to worry about - you're server is simply at full capacity building other user's vault cache files. If your server has the capacity, you *could* increase the number of concurrent users the server will attempt to 'handle' via an advanced Vault Cache server property (it's on your server properties dialog in the Vault Administration Console, right under where you set up the Cache location) - but I advise against this unless you know what you are doing. The time taken to setup EVERY users vault cache will vary greatly depending on how many users, the size of thier archives, network bandwidth etc, so it can take a long time! Remember, it is essentially replicating each archive to each client machine!! 

Hopefully this answers your original issue? If so, don't forget to mark a soloution to help other who may stumble onto this thread.

[eeitocr]

Ben - thanks for all your help.

[Ben_Heymink]

No problem. We're always looking at improving how we can relay information back to an end-user/administrator around situations like this, so I'll keep this one in mind!