BE2010 R3 Issue - Tapes reverting to old media sets after being put into Scratch

[CraigL]

Hi Everyone -

Having a puzzling, intermittent issue with Backup Exec 2010 R3 that I'm hoping someone can shed some light on.

When we load tapes into the attached loader (approximately every 3 weeks), we take all freshly-loaded tapes and move them back into the Scratch media set.  We simply go into Media -> Media Location -> Online Media -> Select All Tapes -> Associate with Media Set -> Associate with [Scratch Media] -> Yes to All.  This finishes successfully and we see each tape in the loader has the scratch icon and says 'Scratch Media' under Media Set.  Pretty simple stuff.

Here's where it gets odd -- every second or third time we do this..sometime during the week, some of the tapes revert back to their original media sets.  This causes some administrative headache, as BE will, for example, use 2 or 3 tapes for 'Daily' class jobs when all of the daily jobs for the week can easily fit on a single tape.  On weeks where the media sets do not revert, everything works perfectly.  We pull our two tapes at the end of the week, scan the loader in BE (so the two pulled ones disapper), and don't look at the console until the following Monday when we do the same dance again. 

A number of months back, we opened a ticket with Symantec and more or less, got nowhere.  The support person we worked with simply had us install a few patches from LiveUpdate and hope for the best.   Five or six weeks went by and we thought the issue was resolved, but sure enough, we hit it again.  We've just worked around it now for a couple months and unfortunately, experienced it again this morning, which is what prompted this post.

We were running BE 12.5 for quite some time and never experienced this issue -- we're certain this only began once we upgraded to 2010 R3 last year. 

Here's the vitals of our setup:

Backup Exec 2010 R3

Updates: Service Pack 2, Hotfix 180429, Hotfix 176937, Hotfix 191248 and Hotfix 194471

OS: Windows 2003 Standard R2 w/ SP2, mostly patched to-date

Has anyone experienced anything like this?

Any help would be greatly appreciated.

Thanks in advance,

Craig

[Larry_Fine]

I suspect that something isn't quite right in the BE database?  But I am no expert on that.

It shouldn't be needed (as your process is theoretically correct), but might you try running an erase job on the freshly imported tapes instead of just putting them in the scratch media set?

I would take a look at your audit log and see if you see anything weird or relevant in there (Tools | Audit Log | change the dropdown to "Device & Media" | sort by date/time | export to a text file for review and search for the bar codes of interest)

[CraigL]

Hi Larry -

Thanks for your reply.

I checked the Audit Log and found something interesting.  The log shows when we moved 5 out of the 8 tapes we loaded last week into the Scratch set, but not the other 3.  This leads me to believe the GUI is improperly reporting successful moves.  We're selecting every single tape and moving them all at once, so I am 100% positive this is not a 'user error' type thing :)

You're right -- we shouldn't have to run an erase job on the tapes once they move into scratch.  I'll put that in the "last ditched-effort" pile :)  (Upgrading to BE2012 isn't even in there!)

Not sure where to go from here......

-Craig

[Larry_Fine]

I checked the Audit Log and found something interesting.  The log shows when we moved 5 out of the 8 tapes we loaded last week into the Scratch set, but not the other 3.  This leads me to believe the GUI is improperly reporting successful moves.

Weird, as I would expect the GUI to move all or none.  Maybe the GUI is not refreshing properly?  did you try hitting F5 or close/open the GUI to see if the GUI really shows all 8 tapes in scratch?  The audit log showing something different than the GUI would concern me and may be worth open a support case for.

I would suggest starting SGmon and then practice moving 8 tapes back & forth between "Scratch" and "Retired" (or a real media set if you feel safe).  Maybe you can catch the error or something interesting in SGmon?  I have never run into this.

[pkh]

When a tape is used by a job, the tape will be placed in the media set that is targeted by the job.  This is by design.  When your tape is moved to a media set, check your job log to see that it has been used by the job.  If it is used, then there is nothing wrong.

When you place a tape in the scratch media set, it is meant to used by any job which needs it.

As to the situation where more tapes are used than necessary, check that your jobs specify append and that your AP is set long enough for the jobs to append to the tape.  You might want to read my article below on why jobs are not appending to tapes.

https://www-secure.symantec.com/connect/articles/why-my-job-not-appending-data-my-tapes

Also, you might want to read the media management section of the Admin Guide which can be found in the BE installation directory.

[CraigL]

Yep, I'm familiar with what normal and expected operation is.  Tape appending works perfectly most of the time (there are a grand total of four jobs..this is a very small setup) so we know that is not the issue.

Again, this only began once we upgraded to 2010 R3.

[CraigL]

I agree, it should be all-or-nothing, or at least generate some kind of error/warning if an issue does arise.

What we're going to do is this:  The next couple of times we load tapes, we're going to do it exactly as-is, check the audit log each time while SGmon is running -- thank you for that tip.  At some point, we're going to catch it.. could take a month or so, but that should at least give us some more evidence to go to support with.

Larry, thanks a ton for your help on this.  I'll report back once we have more info....

-Craig

[Biker_Dude]

There is also a Configure logging button in the lower right hand corner.  You can tweak the output to your liking if it will help to hone in on the problem.

[pkh]

You might want to go to the BE installation directory and use BEUtility to repair the BEDB.

[CraigL]

Just did that.  We'll see if that helps.....

Thanks for the tip.

-Craig

[CraigL]

OK.. loaded tapes today and moved all of them into the scratch pool -- see attached pic!

I had SGMon running and got a trace of all of the events in case the issue arises again.

We will see what happens...

-Craig

[CraigL]

Well, it did it again.

We ran a single job since loading tapes, and four tapes were allocated to two different Pools.  Audit log says only one tape was allocated, which is correct.

Going to re-open a ticket with Symantec now and see what they say...

-Craig

[CraigL]

UPDATE:

I opened a ticket w/ support, who tried telling me that when you move tapes in and out of scratch "this happens sometimes" and "your jobs must be using the tapes".  Obviously, these were not acceptable responses so I asked for the case to be escalated.  More logs and screen shots were taken, and I'm hoping to hear back in the next day or two.

More as it develops.....

-Craig