DLM and BE2014-SP1

[Marcel_CH]

Dear all,

we are running Windows 2012-R2 and BE2014-SP1.

There seems to be a critical bug where DLM is marking medias as "expired" an then moving them to "retired media" as acknowledged here:

http://www.symantec.com/docs/TECH225365

That is all fine but to me it seems there is more behind it.

I already posted in the thread here,

https://www-secure.symantec.com/connect/forums/tapes-randomly-disappear-slots-backup-exec-2014

but it seems to me that not all of the tragedy behind DLM gets it's deserved attention.

On Friday I put 8 tapes. I manually scratch them all. The Full Backup runs, and on monday some tapes of that full backup set, again, have dissapeared. If I scan the Slots, where I know I put tapes, they re-appear but blue, overwritable, scratch. Only if I go then and check the Full-Backup log, those tapes have been in use for this Full-Backup.

And it happens to us, that DLM will "expire" medias of Backups that are on the bank. Meaning we cannot get to our data, unless we re-catalog the whole set.

So we can't browse through backup-sets to see if we have the Data backed up, which set to get from the bank in order to restore data for a user.

 Luckily this weekend for example, it didn't happen. DLM messed up the tapes before the Full-Backup ran, so I only hat to intervene and scan the tapes in the "lost slots". But it could be, that in one week DLM will take medias used for this weekends Full-backup and "expire" them.

So my word of caution would be;

always check the media used against the audit log to make sure that there will not be valuable data overwritten once you scan slots where "lost medias" are located.

Keep an eye on your Backup-Sets by checking your Full-Backups through a simple browse through an restore job to check if all of the backups are still present and calculate an catalog in case you need data.

To me, this seems a serious bug.

It affects our Backups:

- lost data because of "expired" medias that are re-used even if they contained data

- restores turn into an painful experience because all of the sudden, medias in the Set dissapear because the are being "expired"

P.S. I really miss the ability to see in the restore section, which medias has been used for the backup job

[MASIAlbert]

Marcel_CH, regarding "On Friday I put 8 tapes. I manually scratch them all. The Full Backup runs, and on monday some tapes of that full backup set, again, have dissapeared. If I scan the Slots, where I know I put tapes, they re-appear but blue, overwritable, scratch. Only if I go then and check the Full-Backup log, those tapes have been in use for this Full-Backup."

Can you confirm that if you perform a catalog operation on the re-appeared blue, overwritable scratch tapes in question, they will still be usable by a restore operation?

[Marcel_CH]

Hey MASIAlbert,

I have not had the Time to confirm that a re-catalog will get me access to the "lost" Data.

Sorry. 

But as mentioned in 

https://www-secure.symantec.com/connect/forums/tapes-randomly-disappear-slots-backup-exec-2014

from Colin

A tape will always need a backup operation to overwrite it in order to erase the date, as such just going expired won't erase the data, but running a backup job after the expired status occurs might.

So my approach now is, to identify the Tapes used for the Full-Backup as soon as the Full-Backup is done and remove them from the Library and put them already in my Backup-Set Box with physical write protection activated.

FYI: I opened a case regarding this issue and I'm currently in contact with 3rd lvl to have a look at the problem

[Marcel_CH]

Whoever might be interested this is the Hotfix we installed now;

http://www.symantec.com/business/support/index?page=content&id=TECH225092

It sounds promising, but we'll only see for sure if those "DLM - Deleted expired media" stop popping up and messing our Backups.

[Marcel_CH]

The Hotfix didn't help.

On Monday we realized, that DLM deleted the fist of the Medias used in the Full Backup about 7 hours after it was written.

Another WebEx Session has shown that we didn't put some wrong settings but that we have a serious bug here apparently.

So the Problem is that DLM is affecting Tapes, which it souldn't. It should only work with B2D medias.

Now the engineers have a look at the Database from us and the logs to check what is going on.

If this proves to be the problem we think it is, I am seriously disapointed that this feature could go live like this.

Go check in the restores, if you don't have holes in your Backup chains. For us, when one Tape in the Full Backup is "expired" the Full backup will not show up in the Restore Selection anymore (probably only after DB-Maintenance).

[Marcel_CH]

So, "Backline" couldn't help us. Referring to http://www.symantec.com/docs/TECH225365.

The mentioned Workaround, off course, does not help. I know that because I do scratch the Medias manually when I load them on Friday for the Full-Backup since I realized there is something fishy goning on. Maybe 2 and a half months back.

What might help, is to Lable the medias. This is going to be the workaround for me until I hear from "Engineering".

When I checked the case this morning on the "My Symantec" Page, it showed that we were escalated again on the 28th of november. Next Customre Contact Date is the 17th of December.

I'm really starting to wonder if I'm the only one that has those severe problems with DLM. One would expect a Bug this size beeing taken seriously and with the utmost effort to solve it.

[MASIAlbert]

No, you're not the only one.  I have not been responding because Symantec has not made any progress to my knowledge.  So there is no change to talk about.

[Marcel_CH]

Very sad to hear (both that I am not the only one, and that Symantec has not made any progress).

Thought I keep the thread updated in case it's interesting information for others.

[JimHinMD]

I have recently migrated to BUExec2014 and have noticed this behavior as well.  Never had it in 2010 or 2012.  I can't quite determine yet but it doesn't appear to be affecting "live" tapes.  I have simply seen where all of my "Scratch Media" get turned into Backup Exec and Windows NT media randomly.  All the tape details arer wiped and no longer has associations to media sets.  This has occurred 4-5 times over the past 7-8 weeks and I finally opened a case.  Doesn't appear as though they can pinpoint the cuase.

[MASIAlbert]

You should be careful and keep watch.  At first, I didn't think it was affecting "live" tapes as well.  But then a couple times it deleted a tape that had just been filled with backup data from a job from the previous day.

[Marcel_CH]

I have received a link from Engineering to install BE1786.1093RSP2_x64bit which should address the problem. 

Could only do it now. Install was ok. Now all I can do is to keep fingers crossed and wait to see it actually solves the DLM problem. 

Running End of Year Backup this Friday and will still manually expire and Lable the involved Tapes.

[Figleaves]

Hi Marcel_CH

Is there an update to this as i am experiencing the same problem?

Thanks,

[JimHinMD]

I applied the 1786.1093 SP2 patch back on 12/23/14 in hope to resolve the strange tape behavior.  Unfortunately, this past weekend I just expereinced my first issue with live tapes.  2 jobs that were running the verify process from a Duplicate disk to tape job simply couldn't find a some of the tapes it had just written to a few hours prior.  These jobs ended with an "Invalid Physical Volume Library Media Identifier" message.  I found the tapes by manually looking at the details of each empty slot, reinventoried, and a backup set is now associated though the expiration date is 2062 and the tape information is all gone.  Trying to catalog now to see what happens.

[StefanWR]

Hi All, we do have a similar looking problem and I believe it is based on the same bug described by Marcel_CH.

We have a 80 slot library and sometimes after a backup run there seems to be some tapes not beeing marked with the correct Media set. I have not seen the symptom of missing tapes that only were visible after a scan of the library. But after some Backup jobs media was labled as 'Scratch Media'. In addition to that, in the slot view of the library, these tapes were missing an entry in the column 'Capacity'. Normally Scratch Media has an entry like '5MB used of 745 GB' (LTO4 in our case). If I do a catalog of these tapes they are set to the media set 'Backup Exec- and Windows NT-Backup-Media'. But if I would not do a catalog on these tapes, they could have easily been overwritten by accident, which already happened to me.

I agree to Marcel, this is a serious bug and I will raise a ticket with Symantec as well to increase the pressure.

Best regards, Stefan

[Marcel_CH]

Dear all

Today I saw the dreaded DLM Message again. It still affects Tape medias and it just shouldn't. Even if I was apparently lucky this time and it happened before my Backups ran.

Not beeing a BE-Tech, I'd say that the issue is still existing.

This whole matter is taking a freaking long time to be resolved as we are not talking about a "nice to have feature" but an Backup Data threatening Bug!

Best regards,

Marcel

[MASIAlbert]

Symantec contacted me and said SP2 would fix the issue.  Can you confirm that you have SP2 installed and it does not fix the issue?

See email below:

Hello Albert,

I would like to follow up with you regarding your case status (07505878). Your issue has been resolved in the latest service pack (SP2). Please update to the latest version SP2 for Backup Exec 2014 and verify that the issue is resolved.

"-Data Lifecycle Management (DLM) may treat non expired tape media as expired, which makes it possible for them to be overwritten by another backup job. (TECH225365)"

SP2 Release Notes:

http://www.symantec.com/business/support/index?page=content&id=TECH226778

SP2 Downloads and Landing Page:

http://www.symantec.com/business/support/index?page=content&id=TECH225093

Have a great day,

John Klann

Senior Technical Support Engineer

Backup Exec Technical Support, Symantec Corporation www.Symantec.com

[JimHinMD]

Albert.. per my post from 1/5/15, I have already applied 1786.1093 SP2 patch back on 12/23/14 and have still seen tapes randonly moving to expired and then being deleted.  The latest step that Symantec wants me to perform (which I haven't done yet) is to run a repair of BUExec 2014 then reapply SP2.  ???

[MASIAlbert]

Sigh.  OK, well.  I guess I have to jump through the hoops...

[Marcel_CH]

Hey Albert

Sorry to be the bearer of bad news.

Yes, I have SP2 (14.1 Rev 1786) installed since before Christmas break. Since 2nd of January until today DLM decided to retire 12 LTO Tapes (LTO5 and LTO6 mixed) in my case. 

Regards,

Marcel

[Marcel_CH]

Realize I put my reply in a bad spot.

It is NOT fixed yet apparently.

Regards,

Marcel

P.S. and as a side note, I will not follow this "occupational therapy" proposal of repairing BE and re-apply SP2. Not until someone tells me exactly, why I should do that. This is inviting other problems that I have no patience of fixing afterwards.