Duplicate job waiting for media in other tape drive.

[mcheer]

Hi,

We have a problem with Backup Exec 2010 R2 (SP1 and all hotfixes) with a library and 2 tape drives. After investigating slow backups, I discovered that Backup Exec is frequently selecting media that is currently being used in the other tape drive. Therefore, one tape backup will remain queued until it can pull the tape from the other drive. Often, the other drive will then end up doing the same. This means we seldom get both drives writing at the same time, despite recyclable media (in the same set) being available in the library. How such broken logic even features in a backup application is beyond my comprehension?!?! Nevertheless, I welcome any ideas on this issue...

Many thanks.

[Kiran_Bandi]

Backup Exec is frequently selecting media that is currently being used in the other tape drive.

How many jobs are running at that time? If only one then BE will not use two drives paralelly. If you are running two jobs targetted towards the same media set, then BE will use both the drives for writing the backups. (1 for each job, if you have LEO license installed).

Regards..

[mcheer]

How many jobs are running at that time? If only one then BE will not use two drives paralelly. If you are running two jobs targetted towards the same media set, then BE will use both the drives for writing the backups. (1 for each job, if you have LEO license installed).

There are multiple B2D jobs running and usually 2 duplicate jobs. There are 3 media sets being used (daily, weekly and monthly) and one of each will run in parallel. It's when 2 jobs are backing up to the same media set that one will frequently stop and wait for the tape in the other drive. This isn't 100% of the time and will sometimes backup concurrently to different tapes in the same set. Basically, Backup Exec is selecting media that is currently in use, instead of available media in the tape library... surely this isn't expected behaviour?

[Kiran_Bandi]

surely this isn't expected behaviour?

Yes...

Any partitions configured in your robotic library?

[mcheer]

No partitions are configured...

Unfortunately, I've made quite a few changes/updates in the last week in an attempt to improve performance. I don't know if this problem has always been happening or whether it's a new problem? Some of the recent changes:

* Installed SP1 and all available hostfixes.

* Updated tape library and drive firmware. Updated tape drivers.

* Migrated B2D drives from RAID5 -> RAID0. Updated controller and disk firmware(s). Updated Controller driver.

* Changed policies to use a Device Pool, instead of "IBM Library"... I'm suspicious of this one. Will try changing back.

Cheers!

[pkh]

When you use device pool, the job will use the first available device in the pool.  I think you should target you job to the individual tape drives, not even the library.  For example, if your library is IBM1 and your tape drives are IBM2 and IBM3.  You should target your jobs to IBM2 or IBM3.  This will make sure that they use the specific tape drive.  However, if the other tape drive is free, the job will still not use it.

[mcheer]

Hi,

Selecting a device isn't the problem (I think); it's selecting the media. I would expect BE to select an appropriate AVAILABLE media, not one that is currently in use. Surely it has enough smarts to know that a tape is being used somewhere else, so try another one?!

Anyway, I'll continue to try some things to resolve the issue. I've just resolved another problem with setting device priorities by repairing the BE database (another thread), so maybe this will also help?

I've also been considering separating daily and weekly backups onto separate tape drives (as you mentioned above). However, I would like the daily backups to start using the other tape drive once the weekly ones are complete (hence why I was testing device priorities). The idea behind this was to try and reduce tape changes/delays as weekly/daily backup tapes were switched. Hopefully, I'll get there in the end!

[mcheer]

This is getting unbearable now... everything I try and do with this product results in another problem. I've tried setting up partitions to get better control of what media BE is attemping to use and now discover a job stuck in queued state... for NO APPARENT REASON. Is it possible to find out WHY it is queued??

There are 2 partitions with a single drive assigned to each and the (policy-based) jobs are targetting a relevant partition. Each partition has recyclable media in it that is assigned to the appropriate media set. One Duplicate job to tape is currently running, while another duplicate job is queued... using a different policy, a different partition (and tape drive) and with recyclable media available...

Please help if you have any ideas on how to resolve these issues! My experience to date has yielded a backup application with flawed logic and an inability to fulfil it's basic function... and I'm rapidly losing patience. To anticipate some questions:

* Yes, LEO is available and concurrent backup to tape does occur on occasions.

* Yes, the backup jobs are targetting the correct device... in this case, "IBM Library [00xx..00yy]

* Yes, there is a tape drive assigned to each of these partitions; i.e. IBM DRVx.

* Yes, there is recyclable media available; i.e. tapes with appendable space available.

I really do appreciate any assistance and I apologise for my rant!

[pkh]

Are the 2 jobs duplicating from the same B2D folder?  If so, then it could be that the resource contention is at the source not at the library end.

[mcheer]

Haha - I didn't anticipate that question! I've currently separated backups (partition, drive, B2D folder, media set) into weekly and daily. So a different B2D folder was the source for each backup. At present, all B2D backups are complete and only duplicates remain.

Just after my rant, the running backup stopped and the other started... then a few ran concurrently. I thought my rant was hasty, however, I notice that it has happened again. I'm fairly confident that the queued backup will start once the other (using the other drive) has completed... even though there doesn't appear to be anything shared by the 2 jobs. I'm investigating further now... is there a log file or debugging mode I can use to see exactly what decisions or logic is occurring?

Many thanks.

[mcheer]

Yep - and as predicted, the "Weekly" duplicate backup completed and the "Daily" began running. Of course, the next "Weekly" is now queued...

[pkh]

What happens when you target the tape drives rather than the partitions?  Do the jobs run concurrently?

[Colin_Weaver]

It is possible it relates to this:

http://www.symantec.com/docs/TECH155880

In which case a generally available Hotfix will be available soon.

If it does not relate to this then we are likely to need a formal support case opening so that we can gatehr in depth logs and configuration details to isolate the cause.

[mcheer]

Hi Colin,

Thank you... that gives me hope! So far, this looks like the most likely answer. I've only just inherited the backup environment, so I wasn't sure if this problem only started occurring after I updated to R2, however, I suspected. I understand this is an R2-only "bug"?

I've checked the policies and although we backup Hyper-V, SQL, Exchange and AD; GRT seems to be disabled in most cases? Nevertheless, some of the duplicate backups are sourced from folders containing GRT backups, or virtual machine backups at least (I've yet to separate the GRT backups onto a separate B2D folder - my next task). Certainly, the issue doesn't always appear and 2 duplicate jobs will sometimes run concurrently.

I'll enable debugging this evening and check the debug log for the error message described in TECH155880.

Many thanks!

[mcheer · Accepted solution]

Hi All,

Just letting you know that the issue was resolved after updating to 2010 R3. In fact, I'm pretty impressed with R3...

Thanks for all help!