SSR 2013 R2 getting stuck constantly

[jrauch]

Hi all,

I'm having an issue with backups.

I'm using SSR 2013 R2 for my backup solution. I've got it installed on a Windows Server 2012 Essentials system. I've got it scheduled to run an incremental every night at 1AM and a full on Sunday at noon. I've noticed the backup will kind of just stall and not work. The weird thing is that after the backup completes, as far as I can tell, I'm seeing an Error 1000 Faulting Applcation "RunTask". From then on SSR 2013 is pretty much locked up and these RunTask errors are constant. Everything will fix itself when I restart the server.

Troubleshooting so far:
I've uninstalled and reinstalled SSR 2013 R2, changed schedules, changed VSS schedules, looked into seeing if SQL was a culprit, I'm not finding anything. Nothing popping out in Event Viewer either, just the runtask.exe error.

First off, what is the RunTask.exe application doing and would it interfere with backups?

Any ideas?

[criley]

Are you sure there is nothing in the Windows application event log? It sounds like a process is crashing which would normally be logged there.

Can you also confirm the exact version of SSR you have (check in Help/About for version details).

If you could provide a screenshot of the error, it would be helpful as well.

[jrauch]

Version is 11.1.4.55331

I'm looking through the logs now. At 1:00AM Symantec starts the backup.
A bunch of SQL stuff, I/O is frozen, then I/O resumed.

Some ESENT warnings:
"DFSRs (1940) \\.\C:\System Volume Information\DFSR\database_CC12_5F3C_125F_2AAA\dfsr.db: A request to write to the file "\\.\C:\System Volume Information\DFSR\database_CC12_5F3C_125F_2AAA\dfsr.db" at offset 0 (0x0000000000000000) for 8192 (0x00002000) bytes succeeded, but took an abnormally long time (15 seconds) to be serviced by the OS. In addition, 0 other I/O requests to this file have also taken an abnormally long time to be serviced since the last message regarding this problem was posted 172812 seconds ago. This problem is likely due to faulty hardware. Please contact your hardware vendor for further assistance diagnosing the problem."

Symantec messages that it completed successfully
A bunch of Security-SPP messages that the service is starting, then one that it stopped.

Then RunTask error. Every ten minutes RunTask.exe faulting application and a bunch of Software Protection service completions.

Meanwhile Symantec is down at the bottom chilling like this:

I can open Symantec and view everything, if I try to run a new backup it says recovery point from same job still in progress.

[criley]

As the backup starts at 1am and the RunTask.exe error is not seen until 2:32am, I'm not convinced that is the cause of the issue you are seeing.

Can you export the application event log to .txt format and attach it here? Also, please provide the following log file:

C:\ProgramData\Symantec\Symantec System Recovery\Logs\Symantec System Recovery.log.txt

RunTask.exe is a Microsoft process so you may need to speak to them to understand why it's failing.

[jrauch]

Here you are.

[criley]

When was the last time you saw the issue with SSR? I ask because it looks like the backup was successful on the 27th:

2/27/2016 1:00:22 AM High Priority Info: Info 6C8F1F65: The drive-based backup job, Drive Backup of Recovery (*:\), (*:\), (C:\), has been started automatically. 0x00 (Symantec System Recovery)
2/27/2016 1:08:39 AM High Priority Info: Info 6C8F1FF2: Drive Backup of Recovery (*:\), (*:\), (C:\), the scheduled drive-based incremental backup job is completed successfully. 0x00 (Symantec System Recovery)
2/27/2016 1:09:31 AM Medium Priority Info: Info 6C8F1F5F: Created offsite copy [Offsite B]\CHMSERVER_C_Drive003_i006.iv2i.

[jrauch]

27th was the last day it worked.

I saw the issue a couple more times before this, so I tried a bunch of the troubleshooting steps I mentioned above.

That's why this is weird it's showing it completed but, now it's stuck at 1% Creating Recovery Point.

Last time it did this was the 20th. I restarted the server Monday. Worked Tuesday to Sat. Stopped working again. Haven't been able to restart the server again yet.

Did it not fully complete on Sat? I can see it stopping at 1:09 then the 28th it says it's still in progress.

[criley]

It was successful on the 20th, yes ..

2/20/2016 1:00:11 AM High Priority Info: Info 6C8F1F65: The drive-based backup job, Drive Backup of Recovery (*:\), (*:\), (C:\), has been started automatically. 0x00 (Symantec System Recovery)
2/20/2016 1:13:17 AM High Priority Warning:  0x800706BA (Symantec System Recovery)
2/20/2016 1:13:17 AM High Priority Info: Info 6C8F1FF2: Drive Backup of Recovery (*:\), (*:\), (C:\), the scheduled drive-based incremental backup job is completed successfully. 0x00 

I don't spot anything obvious that could be causing this so I think a support case may be needed as more detailed log analysis will be needed.

However, you may want to contact Microsoft first to try and get the issue with RunTask.exe fixed. It would be good to eliminate this as a possible cause..

[jrauch]

Lol, well we've got a dilemma, I'm talking to Microsoft now and they keep telling me RunTask is your problem. Gah!

[criley]

Oh, really?

Did they elaborate on that with any detail/evidence of why they think it's our problem?

[jrauch · Accepted solution]

Finally figured it out.

I ended up doing the whole thing from scratch to see what I missed and eliminated things I thought might be causing problems.

I had Shadow Copies/Previous Versions running on the server at noon everday. Disabled that.
I then removed all of my backup jobs, offsites, everything.
I started a new one, and in the options -> Advanced made sure "perform full VSS backup" was checked.
I let it run and haven't seen a problem since.

I suspect VSS and Symantec don't get along sometimes and the error log didn't point us in the right direction.