Highlighted

Backup Exec Job stuck in Queue

My environment: Windows Server 2012 and Backup Exec 2014, but the problem and solution should apply to recent versions of BE.

Wed night, backup jobs ran as scheduled and I tested restores from disk and from tape that same day.  Came in Thur morning and all backup jobs were queued and ready to run but none of them had started to wite.  (BTW I am writing to a disk device defined as a local folder and then duplicating the data to tape in the same job.  I have about 10 jobs backing up VMs, files and databases on several servers.)

Symtoms:

(1) There was an error: Unable to initialize and communicate with a device .  . .

(2) Anyplace in the BE interface where a storage device was prompted, if the control was selected, it would almost immediately be de-selected.  A drop-down list would expand and immediately collapse making it diffiicult to select/ change a device.  Applied to editing a job, defining a storage device, etc.

(3) In the Storage tab - (The Device in question) - Job History, the screen would flash the information as if it were in a refresh loop.  Other devices would display job history as expected.

I worked with level 1 support and was elevated to level 2.  Level 2 did a DB repair and I completed a software repair via Add/Remove programs. The problem was not fixed by the end of the day. 

When I arrived on Friday, I created a second disk device and added it to a new disk pool and that device showed normally on the screen, meaning no refresh loops.  Before I could run a test backup job to that device, Symantec called back and, through Webex, we continued debugging the issue. 

After a while, we talked about the specific device being corrupt.  We tried to remove the disk device and received a message "Catastrophic Error."

The tech shutdown BE services and then he dropped into PowerShell (command line) and started a SQL session using "OSQL."  He queried the device to get a four digit device ID and then he removed the device using the device ID. 

He restarted the services and restarted the GUI and added a disk device with the same information as the old device.  He then added the device to the original disk pool and we ran a test backup job. 

All was well with the world and Backup Exec.

 

Hope this info helps! 

**  Important: The BE DB repair did NOT report or fix the problem and (I assume) Symantec supplied OSQL tool was used to modify the BE database. **

I do not remember the statements, I.E. tables and columns, entered, but they were standard SQL select and delete.

 

Jim

 

7 Replies
Highlighted

My environment - recently

My environment - recently upgraded from Backup Exec 2012 to 2014 via clean installation on Server 2008 R2 Standard.

I'm having these *exact* symptons on my main 3TB disk storage - the flashing drop-down menus, the job history view stuck in a refresh loop, the jobs which queue or hang on 'loading media'.

I'm grateful to have found this article so that I can manipulate SQL to delete the disk storage and recreate it. HOWEVER, this is the second time this has happened in as many weeks?  All of these backup sets are eventually duplicated to tape, but it's going to start to really hurt if I have to blow away the disk after a few months?  I never had one issue under BE 2012 that caused me to have to recreate the disk.

I'm hoping this makes to the known bug or cause stage, because at this point all I can think to do is try 2015 (we're under maintenance), or go back to 2012?

If anybody has any insight it would be much appreciated.

 

Thanks, -Joe

 

Highlighted

You should try BE 15. Do

You should try BE 15. Do avoid BE 2012
Highlighted

Thanks.  I disliked 2012

Thanks.  I disliked 2012 because of the wholesale interface change, but once i got my jobs dialed-in I found it to be very reliable.

In case anyone else is following this, the actual steps I used to remove the problematic disk storage (it always returns "catastrpohic failure" whether using GUI or CLI) are here,

https://www-secure.symantec.com/connect/forums/how-do-i-remove-disc-storage#comment-8411861


(for BE2014, instead download Microsoft SQL Server 2008 R2 Management Studio Express,
https://www.microsoft.com/en-us/download/details.aspx?id=22985)

Thanks, -Joe

Highlighted

Update:  I didn't format or

Update:  I didn't format or delete the original disk/volume after removing fromm SQL, nor did I inventory and catalog it when BE found it after servcie restart. All of the backup sets were still there, and I was able to duplicate the ones in question to tape.

If this remains the case next time (not having to completely "blow away" the existing disk storage), the issue is more annoying than critical.

 

Thanks, -Joe

 

 

Highlighted

Updated:  Aug 31, 2015

Updated:  Aug 31, 2015 (previusly posted under jsparksa)

The problem reocurred over the weekend and I was able to fix it myself.

 

My environment: Windows Server 2012 and Backup Exec 2014, but the problem and solution should apply to recent versions of BE.

On Friday night, the weekend backup jobs started and they were Queued and did not leave the queue:

Symtoms:

(1) There was an error: "Unable to initialize and communicate with a device .  . ."

(2) Anyplace in the BE interface where a storage device was prompted, if the control was selected, it would almost immediately be de-selected.  A drop-down list would expand and immediately collapse making it diffiicult to select/ change a device.  Also applied to editing a job, defining a storage device, etc.

(3) In the Storage tab - (The Device in question) - Job History, the screen would flash the information as if it were in a refresh loop.  Other devices would display job history as expected.

Tried to remove the disk device and received a message "Catastrophic Error."

 

Here is the "Fix"

(1) stop BE services

(2) using your favorite SQL editior, navigate to your BE instance and enter the commands:

         (My BE DB is Microsoft SQL, and I used SQL Studio Manager)

     Select * from dbo.devices

           You will see a list of all BE devices.  Find your device by name and get the coresponding DeviceID.

    Delete from dbo.devices where DeviceID = xxxx        (where xxxx is the DeviceID)

(3) Restart BE services

(4) In the BE Console, add the device  via Configure Storage.

    a. reuse the same name for the device

    b. use a different device name                              (continued below)

(5) Add the device to any "pools" where it previously existed.

 

 

In My Case, all of my backup jobs use a disk pool or a tape pool for storage, and the disk and tape devices are defined in the pools.  I did NOT have to do any of the steps below.

If your jobs are configured to use a device directly, I.E.  NOT a pool,  there is more work to be done after device is added back to the console. (The newly added device will likely receive a different DeviceID than the previous device even with the same name.)

 You will have to edit each job and each stage within a job where the device name was used.  If you reused the device name, simply edit the job and save it and the job will pick up the new DeviceID.

If you have added the device with a different name, edit each job / stage and change the device in the Storage section.

All is well with the world and Backup Exec.

Hope this info helps! 

**  Important: The BE DB repair did NOT report or fix the problem.

 

Jim

Highlighted

I just upgraded from 2012 to

I just upgraded from 2012 to 2014 and then to 2015 FP1.  Ran into precisely the same issue for two external HDs.  Your fix (via removing the device from the SQL database and the letting BE re-discover it) worked EXACTLY as advertised!   Thanks much, mate!

If this recurs however, I will open a case with Symantec as having a tech work on it might provide Symantec with information and clues that they may find helpful in creating a fix for the problem (I assume that because there is no fix, they haven't clearly defined the cause of the problem).   Workarounds are okay, but a fix will save everyone time.

Thanks again, Joe.

Cheers.

Richard

Highlighted

Hi please assist this problem

Hi please assist this problem is been happening always every week now : Backup Exec 2014 Job stuck in Queue, the msg I received is  Active Queued not running I  dont why .