BExec 2010 tape utilization and job queueing

[RSMi_Cleghorn]

I have what I think is a pretty standard setup.  We have a tape library with 2 drives, we backup 15 or so servers using a selection list per server and a single policy which does the scheduling etc.

We have recently upgraded our tape libraries from LTO4 to LTO5, so we have finished filling out our new tape rotation and are running through the tapes for the second time.

All the backup jobs are scheduled to run at the same time (as you would expect when they're all running off the same policy) and I use priority settings to make sure the critical servers get backed up first.  The policy is set to append to media if possible and overwrite if necessary (this is according to best practice for multiple tape setups, right?).  There is a nominal overwrite protection of 6 hours on the tapes.

Here is the situation: I end up with 2 jobs queued, waiting for a new tape.  Both tapes in the daily rotation are full.  Most backups miss their schedule some small amount of data gets written to tape.  What seems to happen is that the tapes have had some data written to them and they are both (just for example) 75% full from the first run through of the tapes in the rotation.  On the second run, the policy decides to append to the tape, runs through the last 25% and then starts looking for a new tape.  Since both tapes available are now full, it will sit and queue forever.

Now, I know there are a couple of ways that I can manually work around this problem but I've gone to considerable trouble to setup our backup jobs according to Symantec best practice and I'm wondering if I'm not understanding something or if I'm in some kind of very unusual situation where best practice just doesn't produce reliable backups.

tl;dr summary; best practice backups starting with 3/4 full tapes and set to append/overwrite will fill up the tape and queue.  What's up with that?

[Sush---]

Hello there,

    When the tape gets full in the middle of the backup it will not overwrite the same tape immediately.

It will always look for an overwritable tape. So just ensure that you have another tape in the library which is overwritable.

Thanks,

-Sush...

[RSMi_Cleghorn]

So, what happens next time around when that tape fills up?  I just need to keep adding tapes to my backup rotation until I'm back to one tape per server?  What happens when one of those tapes fills up?

This is an unsatisfactory solution.

[Colin_Weaver]

You use overwritee protection settings to eventually allow tapes in the library to be re-used.

Also if you onoy want to use one tape per server then do not start the jobs as appends.

[Sush---]

You have mentioned that you have overwrite protection period set to 6 hours. So you can use any tape which has expired the overwrite protection period of 6 hours.

Thanks,

-Sush...

[RSMi_Cleghorn]

I don't understand your point.  You're saying I should turn overwrite protection off?  I'm pretty sure I can guarantee the outcome will be the same since I have had jobs which have been queued long enough for the overwrite protection to expire and they don't start writing to the start of the tape.

[RSMi_Cleghorn]

That's trivially true but doesn't address the problem in any significant way that I can see.

[RSMi_Cleghorn]

Bump

[RSMi_Cleghorn]

Bump

[Jaydeep_S · Accepted solution]

By what you have mentioned, the only way I can see to make this work is set your first 2 jobs to Overwrite media and the following jobs to append.

This would ensure that the first 2 jobs which I am assuming have been set to go to 2 seperate tape drives will begin with an overwrite operation and would leave enough space for the following jobs to Append. In this the only issue that I can see is if any of those tapes get Full, then yes you will need a new media to be written to. This is because, if a job starts to append to a tape which gets full, BE cannot overwrite the same tape even if the OP is set to none, hence new tape is required.

Note - The job that overrites and the jobs that follow-up with Append need to use the same Media Set.

[RSMi_Cleghorn]

Yep, that's one of the work-arounds I thought of.  I would need an entirely seperate policy for the first 2 jobs and schedule them for just before the other policy starts.  Which sees to be creating a heavy management overhead and would also ensure that I have to maintain that setup for the entire life of the backup solution.

The other would be to have an erase job which runs just before the schedule starts but scheduling erase jobs gives me shivers.

Either way, as you mention there are implications if we need to expand to an additional tape in the normal rotation.

It sounds very much like there's no good answer.

:(

[RSMi_Cleghorn]

I'll mark this as a solution because I think it's as close as I'm going to get.  If anyone can come up with something better, feel free to continue the discussion.

[Jaydeep_S]

To get a bit more specific about this, does this happen after you have replaced the tapes. Cause by what I am assuming is 6 hrs of OP is not much. Now if you are recycling last weeks tapes to be used now, ideally they have OP which is expired and should show as overwritable media (in blue text). If this is the case then the only operation permitted on this tapes is "Overwrite". Could you confirm if you are seeing the media this way?

Also, could you go to Tools -> Media management and see if the Media overwrite protection level is set to atleast partial or Full and not None. Additionally, the Media overwrite Options are set to "Overwrite recyclable media contained in the target mdia set before overwriting scratch"

What I am trying to point out here is if the above settings are correct and if your jobs are set to "Append to Media, overwrite if no appendable media isavailable" then the first job will always Overwrite the Expired Media. Could you confirm if this is happening correctly.

[RSMi_Cleghorn]

Confirmed.  All available media is showing as "Overwritable", media overwrite protection level is "Partial", media overwrite options is set to "Overwrite scratch media before overwriting recyclable media" and Policy is set to "Append to media, overwrite if no appendable media is available".

I have just confirmed that for the media in the drive right now, I will confirm the behaviour tomorrow morning.

[RSMi_Cleghorn]

And behaviour confirmed.  We've managed to compress just over 1TB into the last few hundred GB free on the two tapes.  Both tapes are now full, the first 2 jobs are queued and BE is busily swapping the tapes in and out of the drives.

I am going to have to cancel all my backups (or wait for them to miss their schedule) and hope no one needs any data from last night.

Also, these two jobs will sit queued forever.  If I'm not paying attention they will kick off again when I change the tapes today and if I'm unlucky they will also fill tonights tapes so I will have no backups for 2 days.

The reason I'm upset about this is that I'm not trying to do anything complicated.  I have to be close to the dead centre of the use case for BE.  I've set up the backup jobs as recommended by Symantec (and indeed in the way BE2012 will force me to do) and I'm unable to get a successful backup.

I've seen this comment so many times on the forums but I have to repeat it.  As a long time user of Backup Exec (I too fondly remember the Veritas days) I am deeply disappointed in the way this product has been managed.  Support, whilst technically knowledgable, is almost impossible to contact.  I shudder to imagine having to contact Symantec Support in a DR situation where a 24 to 48 hour delay in first contact would be business threatening.  Rather than improving over the years and version iterations, it has gotten progressively worse.  11 was better than 12 was better than 2010 is better than 2012.

We are in the process of re-evaluating our backup strategy and technology selection and BE is not even in the running because it is simply incapable of reliably producing a backup.

[Jaydeep_S]

"And behaviour confirmed.  We've managed to compress just over 1TB into the last few hundred GB free on the two tapes.  Both tapes are now full, the first 2 jobs are queued and BE is busily swapping the tapes in and out of the drives."

I am not sure what you meant by this. Did the first 2 jobs overwrite the tapes or appended to them? If they did overwrite then you probabely are backing up more information than what the tapes can hold. As mentioned in your original post as this is a robotic library, you could provision for more tapes to be used by the jobs allowing them to complete.

If the tapes are not being overwritten, then there is something wrong and needs to be investigated further. This would imply that BE is not honoring the OP and AP set by the media set by appending to a overwritable media. If this is true, please open a support ticket with Symantec and have the issue investigated.

[RSMi_Cleghorn]

The first 2 jobs appended to the tapes.

I will speak to support.