Two jobs will not run, all other jobs are running fine.

[Erik_L_]

We have a backup server that has 12-15 jobs on it from all different servers on our network.  Everything was working fine (well fine enough, I always have issues with the backups but nothing I can't normally handle), but yesterday there were two jobs that would not run.  I have Backup Exec version 12.5.2213.133.  Both jobs are Disk-to-Disk.

Here is what I know so far:

1. The job starts, then goes into "Queued."
2. The job cannot be cancelled once it is started, until the Job Engine service is stopped
3. In order to stop the service, you have to wait ~5 minutes because the Job Engine service is frozen and it takes a while after initiating the stop on it for it to actually stop.
4. Both jobs are backing up the same data on the same server, one is an incremental backup and one is a full backup.
5. Rebuilding the job does not work
6. Using a newly created "Backup-to-disk Folder" does not work.
7. Applying all LiveUpdate hotfixes does not work
8. The .bkf file that is "In-Use" while the job is Queued is only 2KB large.
9. After the job is cancelled, that bkf file continues to stay in use until services are cycled.
10. Rebuilding the catalog index does not work (CatRebuildIndex.exe -r) and only takes ~10 seconds to complete
11. Rebooting the server does not work
12. When the job is queued, it has a computer with a green "Pause" button Icon in the State column on the Job Monitor tab.  I am not sure if that is normal but I think that when a regular job runs it just has the computer icon without the green pause picture on it.
13. The job is not on hold
14. When I go to "Pause" the server and "Unpause" it.  It will give me an error: Unable to pause XXX, but the server does go into a paused state.  When I go to Unpause it, it will give me the error: Unable to pause XXX, but the server does come out of a paused state.

If there are any other questions or information you need, ask and I will do my best to answer.

Thanks,

Erik

[CraigV]

Hi,

Do these jobs run at the same time?

[Erik_L_]

They are not scheduled too, but since one was in the queue for 33 hours, by the next day they were both stuck.

I have not tried to run them at the same time in a while, I was just doing all my testing with the Full backup.

[pkh]

Both jobs are backing up the same data on the same server, one is an incremental backup and one is a full backup.

In this case, you have a deadlock situation.  Both jobs will try to hold the same resources and wait for each other.  You should trouble-shoot why the first job waited for 33 hours.  That is the cause of your problem

[AmolB]

As per point # 14 you are not able to pause the server, what is the status of the utility

jobs like inventory or erase do they complete successfully or they remain in queued status.

[Erik_L_]

Amol,  Utility jobs work just fine.  I ran erase to erase the 2KB bkf file, it worked fine, I also ran an Inventory job afterwords and that went fine.

pkh, A deadlock situation is not happening, the jobs have been working fine for years.  Yes, the situation where the first one is stuck in queue for 33 hours IS the actual issue, and that is the issue I am trying to get help for.

I think it must be an issue with our file server (the server that the jobs are backing up), because all of our other jobs are running fine.  Came in this morning and all the jobs overnight ran just fine, but the same incremental job as before is stuck in queue.

[Erik_L_ · Accepted solution]

Fixed the issue by rebooting the File Server.