Backup Exec 2010 R2 Job Stalling, But No Errors

[Orangoutang]

Hi Everyone,

I have a problem with a new installation of Backup Exec 2010 R2 that I'm hoping someone will be able to help with. Basically, during any backup job (backing up to tape or disk, it doesn't matter), the job will randomly freeze and it will stop backing up. From the console it appears as if the job is still running, but the job rate slowly gets lower and lower. It's like it's getting stuck on a file, and then can't carry on with the rest of the job so just sits there waiting for something? There are no Alerts in the Backup Exec console, and nothing of any relavence in Event Viewer. It happens randomly when backing up from any of our 7 windows based servers, and the file it gets stuck on is different from night to night. The backup server is Windows 2008 R2, running on an IBM x3650, with latest version of Backup Exec 2010 R2. All of our agents have also been updated to the new versions. The server itself isn't running anything out of the ordinary - just the customised IBM drivers which have never caused a problem in the past. I also tried disabling the Advanced Open File Option from the job, disabled the firewall and antivirus (McAfee 8.7i), but it still does the same thing. It might be worth me mentioning that when this happens, I can't even cancel the job - I have to restart the services to get the job to register as failed.

Any suggestions would be greatly appreciated.

[CraigV]

Hi there,

I have had this happen to me with a tape drive that was busy failing. Are you running your backups to tape, and if so, what make/model are you using?
Best bet would be to get hold of your manufacturer's utility for that device and run diagnostics tests against it to make sure it is still OK.

Laters!

[Orangoutang]

Hi CraigV,

Thanks for the suggestion. I've tried backing up to various devices including a backup-to-disk folder and an HP tape drive. I still get the problem regardless of the backup device used.

Thanks anyway,
All the best.

[Bhill4375]

Hi, I have an almost identical problem as what you described and was wondering whether you've reported this problem to Symantec and if so, what was their solution?  I have an IBM x3650 server with Windows Server 2008 R2 OS and Backup Exec 2010 SP1 and am experiencing intermittent stalling of the backup job on a remote IBM x346 Windows Server 2003 server.  The remote server also has the latest Backup Exec 2010 remote agent.  Similar to what you reported, it stalls on different files on the remote server and will hang on that same file until I either stop the Backup Exec services or restart the remote server.  This problem is also intermittent because sometimes when I run the backup, it completes without hanging up.  There are no existing problems on the remote server and it's not being accessed when the backup runs.  If you resolve this problem or do any additional testing, can you please let me know?  Thank you!

[Orangoutang]

Hi Bhill4375,

Thanks for your response, it's almost good to know that we're not the only one with the problem. Some of our remote servers are IBM x346 servers, so our situation is virtually identical, although it stalls when backing up from x3650 servers as well. Leads me to think it could be driver related? I made some configuration changes to our server last night, so if it proves successful, I will certainly let you know. If the problem occurs again, I will contact Symantec for advice and post any solution here.

Thanks again,
All the best.

[Sush---]

Hello there,

   Can you remove (unplug) the tape drive from the server and then try the backups on Backup to Disk folder. We have seen that if there is any fault in the tape drive then backups goes to queued state even when they are targetted to Backup to Disk folder,

SO to isolate the issue I would appreciate if you can try the step of removing the tape drive and taking the backups to the Backup to Disk.

Thanks,

-Sush...

[Orangoutang]

Hello Sush,

Thanks for your reply. I can confirm that the problem still occurs when backing up to a Backup to Disk folder, even when the tape drive is not  physically connected to the server.

Thanks for all your help.
Orangoutang

[Mahesh_Roja]

Any warning and errors in event logs, if anything is there please post the error logs.

[CraigV]

...the poster mentioned No in the original post...

[Adam_Garrett]

I think my problem might be similar to yours.   I upgraded from Backup Exec 2010 to Backup Exec 2010 R2 and ever since my backup hangs at the last item to be backed up.  I thought it might be the particular server, but it did not matter what order you changed backups around.  It still seems to fail at the last item.   Cannot cancel the job (forever stuck on "cancel pending") and cannot restart services.  Only way to clear problem is to reboot server.   Big problem on a production server.   I have submitted a Critical case (#412-877-936) and have been waiting all this week for a call back.   :(

[Orangoutang]

Hi Adam,

Thanks for your post - it does sound very similar to the issue I'm having. On Thursday last week I reinstalled the OS on our backup server from scratch and since then I've had 2 successful backups and no failures. I've not installed anything else as yet - no antivirus or anything, just the OS (Windows Server 2008 R2) and Backup Exec 2010 R2. I also removed a redundant PCI Express network card that was no longer connected to anything. The successful backups could, however, just be a coincidence as we've had runs of successful backups before only to end up with the same problem a few days later. I guess only time will tell, and I understand that not everyone will be in the position to spontaneously reinstall the OS.

Thanks again,
Orangoutang

[Turtle5277]

Appear to have the same problem here backing up VMware servers.  Backup job hangs in the running state without any progress on the last item for a server.  When I try to cancel, the confirmation window appears, but the job will not change state and remains showing running.  I believe last week when I tried to restart the Backup Exec services, it just hung and therefore had to reboot the server.  Recently upgraded to Backup Exec 2010 R2 on a Windows 2008 X64 server.  This server has 2 LTO tape libraries with 2 tape drives/library.  Each drive has its own backup job running against it (4 jobs).  So far, only the backup job that backups the VMware servers has experienced this problem.

[Orangoutang]

Hi Everyone,

I just thought I'd confirm that our problem with this seems to have been solved. As I said in a previous post, I reinstalled the OS and removed a redundant PCI Express network card that was no longer connected to anything. I'm guessing that our problem was caused by a software issue, but am unable to confirm that. I would still be very interested to hear from anyone that has a similar issue, as there is obviously an outstanding problem that Symantec need to get to the bottom of.

Thanks again to everyone who has responded so far.

Orangoutang.

[jshwank]

I upgraded our 12.5 over to 2010 R2 about 3 weeks ago. BE is running on top of Windows Server 2003 and after the upgrade all backup jobs were working fine for the first 2 weeks. Then for the last week all of our daily jobs have been hanging up with no errors or alerts. Only way to cancel the job is to either stop all the services (Job Engine takes a couple tries in order to get it to stop) or reboot the server. I can run manual jobs fine, I created a new scheduled job and the same thing happens. I've tried different tapes and they all result in the same issue. Our tape library is an HP StorageWorks 1/8 Autoloader (LTO3). Tried rebooting the server and tape library, but again the same issue. BE should be up to date as we installed Hotfix 358289 and Hotfix 140360.

Anyone get a resolution to this issue yet? Unfortunately, I'm thinking a reinstall is probably in order shortly.

Thanks!

[Quad]

Usually when i've seen a job hang and the throughput counts down it means there is a connectivity issue and either there is a loss of communication between media server and device being backed up, or media server and the backup to disk remote location or tape drive.

You might also see that the agent has crashed usually if you can't cancel the job that is the case and we used to get this alot on Apple MAC servers when backing up files in a path over 255 characters.

Things you could try

If you have windows firewall enabled, disable it (on by default in windows7 and server 2008)

Modern network cards have all kinds of extra features to priotise traffic through the card etc which really aren't necessary and cause more issues than they fix especially with backup exec.  Turn off anything that is an additional feature on the network cards being used in particular Receive Side Scaling and any Checksum Offloads, this will improve the stability of the network card.

If you have antivirus running on your backup media server or the backup to disk remote location disable it as it can interfere with backup exec when backing up.

[villeah]

Hi,

Just want to add my name on the list. My jobs stalls when trying to run VMware backups to deduplication folder. Weird thing is that another job still runs smoothly while another job is stalled. I receive no error messages, and job will stay in runnning state until I restart services.

I have HP Proliant DL180G6 running Windows Server 2008 R2 64-bit. Backup Exec version is 2010 R2. Any help would be appreciated since I've lost my faith on Symantec support.

[Turtle5277]

Actually tracked problem down to 3 virtual servers that are causing the hang.  It was not the server that the backup job stuck on, but the next server in the backup sequence that has the issue.  Still investigating cause of the problem on these 3 servers, but currently suspect problems between VSS and McAfee.

[Hodgkinsons]

Hi,

We're having similar issues with version 2010R2 our backup jobs stall over night, after emmitting certain folders it just stall at other locations. To Cancel the job we have to restart the services then end process on the BE engine service in task manager, no need to restart the machine though. if left for approx 7 hrs the job does complete!! not sure what's going on, has anyone been able to resolve their issues yet?

Thanks

[stephan_vanheld]

We have the same problem. The backup of our file server (around 3 TB) stops somewhere between 70 and 400 GB, usually on a large file.

[OpeSys]

I'm getting this exact problem.

i need to shrink my exchange database but can't because the job stalls before completing.

it's really a huge issue.

Anyone figure anything out yet?