Customer's background tasks stuck in starting

[Jolaine_Y]

CA 10.0.4.1060 On 1 of our 2 CA servers the all of the customer's task status is showing as "Starting". The service status for these customers shows as "Stopped" in the CA client. I can't find any errors in the Event logs or IIS logs that give any sort of clue as to why the tasks aren't starting. I've tried using dtrace to find some answers but I can't see anything that reveals the source of the problem. I've done IIS resets, restarted the Accelerator Manager Service, the server has even been rebooted. We did install an application called NeverFail on both of the CA servers around the time this started. It's been disabled since. Has anyone seen this before? Maybe give me some direction on where to look next? I'm puzzled as to why it's happening on one of the CA servers but not the other.

[AndrewB]

i googled neverfail and it looks like some sort of HA solution. how do you have two CA servers? can you describe your setup? what about the databases?

[Jolaine_Y]

In our production datacenter we have 2 CA servers that have about 10 customers each. There are 2 SQL 2012 servers that are used for EV, CA, and DA. The CA customer databases are spread evenly across the 2 database servers.

Neverfail makes a clone (essentially a snapshot) of the virtual machine and synchronizes the data from the primary to the clone. In the event of a failure you can switch over to the clone with little to no downtime. For the production CA servers we never got past installing Neverfail because it kept failing to create a quiesced snapshot. It was around that time we noticed this issue and decided to stop doing any more work with Neverfail until we got this sorted out. Neverfail is still installed on the server but its services are stopped and disabled. FWIW Neverfail is working on our test EV, CA, DA servers without any problems and dozens of other production application servers.

I guess I'm just trying to figure out what's happening in the background when the "Customer's Background tasks" option is selected in the evbaadmin webpage that would cause the status to show as "starting" and not change to started. I need some place to begin and none of the logs I've looked at really give much direction. It's also worth mentioning that aside from the status showing "Starting" in evbaadmin and the Service status showing Stopped in the CA client there doesn't seem to be any problem. Searches are running, random sampling is working, we can create new searches.. I figure those things wouldn't work if the background tasks weren't running....

[Kenneth_Adams]

Hello, Jolaine;

Have you worked through this issue?  If not, please look in the CA configuration database's tblCustomer table for the status of the CA customers showing as having the service stopped and relate the StatusID shown to the tblStatus table in the config db to see what that statue actually means.  Also look in Task Manager on the CA server to see if you have 1 instance of the AcceleratorService.exe process for each CA Customer on that server plus 1 for the Remoting option (i.e., 10 CA Customers would have 11 instances of AcceleratorService.exe process running).

If you don't see the right number of AcceleratorService.exe processes, the you have some CA customers that are not running their background tasks, which also means they won't be running any searches.

If you find the tblCustomer table shows these customers are actually running, then you've encountered an ASP.NET cache refresh issue that can be addressed by implementing the DisableLazyContentPropogation registry value per TECH124694.

Let us know how things are going and what you find with the information above if you're still having the problem.  We'll do what we can to get you running again.

[Jolaine_Y]

For the customer's that have the background task showing as "Starting" the tblCustomer shows ATStatus = 20 and AS_Status = 23 I'm guessing that ATStatus 21 means "Running" because that's what I see for the customer's that I'm not having a problem with. The number of processes looks good. There are 12 AcceleratorService.exe processes and 1 AcceleratorManager.exe (we have 11 customer's on the "problem server"). Do you think the DisableLazyContentPropogation registry setting help us? Thanks Ken.

[Nups]

You may have already done this but have you unchecked the Customer background task(right click on the customer in EVBAAdmin page) and re-checked it just to see if that makes a difference in the status showing up in Client and EVBAAdmin page.

[Jolaine_Y]

Yep, tried that. No change.

[Jolaine_Y · Accepted solution]

We upgraded to EV 11.0.1 over the weekend and haven't seen this behavior since. Could be a coincidence.. not sure.