Error - Not appendable (Media full)

[Dan-E]

I am using Backup Exec 2012, latest updates on Windows server 2008. I am doing nightly backups on a rotated set of 10 tapes (1 per business day). I am not concerned with overwrite protection whatsoever, all I want is for the backups to be written to tape - appended if there is enough space, or if not, to overwrite the old data, which will always be 2 weeks old due to the time it takes for a full rotation (1 fortnight).

However, I am finding that when my backup jobs get a tape that is full or close to full, instead of overwriting the old data my backups are spontaneously failing with the errors:

Described against the tape in the Media area:

• Overwrite Protection Period: Overwriteable
• Append Period: Not appendable (Media full) 

Described in the job log against the failed job:

No appendable media could be mounted.

Switching to overwrite operation on scratch media.

When these errors occur, the job ejects the tape typically in the middle of the night and halts the job. Here are my relevant settings:

Media set properties:

• Overwrite protection period: 1 Hour
• Append period: None

Other:

• Settings > Storage > Media overwrite protection level: None + No prompting
• Settings > Storage > Media overwrite options: Overwrite recyclable media ...

This is baffling, because according to my settings, it should overwrite recyclable media, not scratch media, but you can see in the job failure log that when it decides appending is off the table it goes straight for scratch media. Additionally, there is effectively no overwrite protection whatsoever, and even if there was, the old data on this tape is a month old, so I can't see how this is an OPP related error.

Also, I have 3 different media sets associated with different tape drives, and they are all suffering from this issue. Also I should note that I am new to this position and while I was being trained up by the prior guys, this issue didn't seem to be happening, so I'm not certain but perhaps they were manually erasing the tapes when they got full, but theoretically that shouldn't be neccessary anyway.

Anybody got any clues what's going on?

[CraigV]

Hi,

BE would overwrite an entire tape, not sections of it containing data that is past the OPP settings.

As a matter of interest, what are your Append/OPP settings?

Thanks!

[Siddhant_Saini]

Hello Dan,

There are two things which I think could cause it: 

1. Settings > Storage > Media overwrite protection level: None + No prompting  > All the tapes would be overwritten and not appended. Refer to http://www.symantec.com/docs/HOWTO22311

2. When a job is spaned on multiple media, only the first media would be appended rest all would be overwritten(As tape can only be read linearly). So make sure all the media except the first is overwritable.

[pkh]

When you set your OPL to None, then any OPP and AP is not valid.  On top of this, there are numerous past discussions which described strange behaviour when OPL is set to None.  You should always set the OPL to Partial or Full to prevent accidental overwriting of tapes, especially if you are using a tape library.

[Dan-E]

1.  Well, I can assure you that they are definitely being appended to, at least until they are full and start to fail as described in the scenario above. I can see the old backups on the tapes, they usually hold about 2 full backups worth and then overflow during the 3rd backup session written to them.

2. I'm not sure how spanning works in Backup Exec, but you might be onto something here because I have noticed that sometimes the backup job will actually get say 30% through before it gives up and ejects the tape saying it is full. I assumed that what it would (should?) do is rewind the tape once it is at capacity, and start overwriting data from the beginning (which would be the oldest backups) so long as the data it is overwriting is outside the bounds set by the OPP. Am I wrong about this assumption?

[Dan-E]

OK, I didn't know that, but nevertheless whenever a fresh tape goes in, nothing on it is within the OPP because the most recent backup is at least 1 fortnight old, and I have no OPP at all.

The Append/OPP settings are contained in my original post:

Media set properties:

• Overwrite protection period: 1 Hour
• Append period: None

But another guy pointed out that these settings are apparently irrelevant because in my Backup Exec settings I have:

• Settings > Storage > Media overwrite protection level: None + No prompting

...which overrides the OPP, or at least that's my understanding.

[Siddhant_Saini]

Why don't we set the 'Media Overwrite Protection Level' to Partial and then test things out? 

Backup Exec or no other application would do the work of Appending and Overwriting the same tape in the same backup job(the rewind and write thing won't work). For the next tape, the tape has to be overwritable. Backup Exec would not be able to append to the second tape. 

[pkh]

Why don't we set the 'Media Overwrite Protection Level' to Partial and then test things out? 

Good suggestion, which is why I suggested it earlier.

[Dan-E]

Alright I'll give it a go during the backups this week, if the problem doesn't present over the next fortnight, I'd call it a solution.

On a seperate note, if this works, that would imply a bug with the Settings > Storage > Media overwrite protection level: None + No prompting option.

I have now changed it to:

Settings > Storage > Media overwrite protection level: Partial + No prompting

[pkh]

On a seperate note, if this works, that would imply a bug with the Settings > Storage > Media overwrite protection level: None + No prompting option.

No.  I don't think so.  You are expecting BE to obey your OPP and AP settings and yet you have turn them off by setting the OPL to None.

[Dan-E]

No, in my scenario it doesn't matter if it obeys my OPP/AP settings or not, since they are effectively set to "write over anything ASAP" anyway.

If anything changing the OPL from none to partial would theoretically make it more likely to fail, because then the OPP/AP restrictions come into play and I would (at minimum) have a 1 hour OPP restriction, whereas currently I have (theoretically) absolutely no restrictions as to when the tape can be overwritten due to the OPL being set to None.

But it's not working out that way...

[pkh]

... but you are appending to the tapes, not overwriting them.

[Dan-E]

True, but nevertheless, changing OPL from none to partial should still make no difference with appending, since OPL is an overwrite protection which does not affect append operations.

And as of this morning, that seems to be the case :(

(as in, nothing has changed...)

This happened overnight to two of the tape drives.

The alert:

The job:

The tape:

The tape error:

[Dan-E]

OK So here are the screenshots from the last post in order, hopefully.

The last screenshot is interesting - basically telling me that it can't overwrite the media because it's allocated and protected (assumingly due to OPP), but you can see that the OPP time is well past by looking at the system time. But it still refuses to write to the tape, even when I manually re-insert it.

[pkh]

As far as I can ascertain from your screenshots, your job starts at around 1400-1500 hrs. on 31/3 and your tape is write-protected until 2300 hrs on 31/3.  Hence, your problem.

[Dan-E]

That explains nothing. As you can see from the screenshots, the job runs to 44% completion before it ejects the tape. When the backup begins there is no write protection or OPP in place, since the last backup operation is about a fortnight old, hence, the 1 hour OPP does not apply & the backup operation starts.

The reason there is OPP up until 2300 on 31/3 is because that OPP is an hour from the last time data was successfully written to the tape (which would have been at about 44% through the backup, when the tape reached capacity). Don't forget my OPP is set to an hour.

All this aside, I think I now know what is going on! Turns out that shortly after I took this position, the other two guys left the organisation and another guy joined. He didn't like the way the backups were working and create a new set of backups. It took a few weeks after this when the eject problems started to occur.

My condensed understanding of the problem now is as follows:

I have tape drive FatBoy

I have seperate backup jobs Server_1, Server_2, Server_3, they are queued up to start at 5:00pm, 7:00pm, 9:00pm respectively each night. Each of the backup jobs is set to Overwrite + Append, with the destination as Fatboy. So a non-full tape goes into the drive, Server_1 starts at 5:00pm, and fills the tape when it is 55% through at 5:30pm. It spits the tape because the tape is full, and not the OPP but BE internal 'smarts' prevents it from writing over the media that has been used in the current job (and of course, you probably wouldn't want it to do this, since you would lose whatever data had just been written to it). This is despite the fact that my OPL is set to None, and/or despite the fact that my OPP has expired - hence it is an internal feature of the software not to do this, not a user-changable parameter. Also, despite the fact that the 1 hour OPP expires, I am still unable to insert the same tape back into the drive, so BE must have some smarts to refuse to overwrite a tape that has formed part of the current backup operation, and you can't force it to do otherwise.

So at this point, the only way that Server_1 can continue is when an entirely different non-full tape gets inserted, with no OPP in place. Alternatively, if I cancel Server_1 the next morning at 8:30am, the 1 hr OPP (if OPL enabled) on the current tape has well and truly expired, and the next job Server_2 will happily kick off and write over the current tape after manual re-insertion, since Server_2 is a different backup than Server_1 and doesn't know or care about what media Server_1 used. And as long as there's still space, Server_3 will go ahead and append it's data after Server_2 is done.

Therefore, my current theory is that the old backups were setup such that the Server_1 job was not set to Overwrite+Append, but to Overwrite. And the subsequent jobs Server_2 & Server_3 were set to Overwrite+Append. So when the nightly backups kick off, the first job overwrites the tape from scratch, and the subsequent jobs that target the same tape append data to that backup, this is something the new guy has obviously overlooked. And after talking to him, that seems highly likely.

So I've changed the backup regime now, we'll see how it goes. But I'm quietly confident. 

[pkh]

despite the fact that the 1 hour OPP expires, I am still unable to insert the same tape back into the drive, so BE must have some smarts to refuse to overwrite a tape that has formed part of the current backup operation, and you can't force it to do otherwise.

No.  This is not true.  If you do re-insert the ejected tape, BE will overwrite the tape.  This is why there is a warning from BE when you set short OPP.

The ejection behaviour is by design and is to save you the effort to eject the tape before inserting another overwritable tape.  If your OPP is short and has expired, BE will overwrite the tape if you accidentally push the tape and re-insert it.

[Dan-E]

I don't know what you want me to say. I hallucinated the whole thing?

I have just told you that did not happen and provided you with screenshots with the associated error messages. Do you think that I am making this up as some sort of bizarre ruse to confuse you?

If I put the tape back in, BE immediately ejects it, even if OPL is set to none. Even if OPL is set to Partial, and the OPP has expired by half a day.. So actually, it IS true, at least in my implementation. I guess the question then becomes is my implementation broken?

Can we get an actual Symantec person in here to provide some insight??

[Colin_Weaver · Accepted solution]

Ok I am not going to read all your exact points as this thread is too long. However I will explain a few key points about how it works

1) If your tapes are both overwritable and appendable when a job starts and if the job is set to use append at start then it will append to the tape. Either change your job start settings or your media set append period to avoid this.

2) As soon as new data is written to a tape (either an append or an overwrite) it will move the overwrite protection back to whatever the media set states. In other words the overwrite period is not fixed to the first overwrite like the append period is but will move.

3) The overwrite protection period is always calculated from the last write to the tape (so the end of a job not the beginning)

4) If a job fills a tape in the middle then the tape will be ejected (from a single/stand alone tape drive) or returned to a slot ( for a library ) and a new overwritable tape will be requested. Note for an eject in a stand alone drive, the cartridge will still be resting physically in the drive but will be in an ejected state that will require someone to take action to change.

5) If an overwritable tape cannot be found (no tape in drive for stand alone drive or no overwritable tape in any library slots) then you will receive an insert media alert and the job will sit waiting.

6) We can only append to the first tape used by a job, subsequent tapes chosen as each tape fills up MUST be overwrittten

7) An overwrite of any media is a complete overwrite/erase of the media no matter how many backup sets are contained in the media - in other words you cannot overwrite just the first backup set held on a tape.

7a) I suspect (but have not tested this) that because of point 7, we block the overwriting of a tape that was appended to at the start of the job, that ended up filling the tape, to avoid a "swallow your own tail scenario" and in effect corrupting your own backup. I would expect this block to occur even if Overwrite Protection is set to "None" on the media server although as stated this might need a test to validate.

8) Whether or not a tape can be overwritten depends on it's protection timings when the backup job starts. When you come in the next morning and realize your overnight backup job is awaiting a tape, the status of the tape at 09:00 that morning will not reflect the status at 20:00 the night before and will mean that wrong facts are being used to understand whether or not you had overwritable media available.

9) Once a job has decided it cannot use a tape it will not suddenly start using that tape just because the job has sat waiting for long enough for the tape to become overwriteable. It is possible (again I have not tested this) that acknowledging the alert in the BE console after the tape has become overwritable would use the tape or the job might continue to ask for a different tape.

10) Whilst the admin guide is a bit overwhelming, the one chapter I recommend all customers using tapes spends time to understand is the one relating to overwriting and append on tape - which in the BE 2012 Admin Guide is Chapter 13, and the section entitled "About tape and disk cartidge media". Pages 371 and 372 contain a couple of very useful graphics and page 378/9 provides a breakdown of how overwriteable media is chosen

11) Backup Exec will move media from one media set to another if the overwriteable media is found in a different set (see the admin guide info in point 10 for why this happens) Understanding your media management and/or partitioning the libraries can help control this.

Some general recommendations

a) Do not set Overwrite protection to none, this is a recipe for overwriting media you need too early.

b) Do not overwrite your only existing backup set before you have completely finished, successfully, creating the next backup set of the same data - if something happens duriong the backup job activity to create the second backup set you will have no backup at all rather than still having one backup to restore from. In other words set you overwrite protection to at least one day if doing daily full backups and longer than a week if doing incremental sets with one weekly full.

c) If you have too much data for one tape and are using a stand alone drive then invest in new hardware (preferably a library)

d) Again if using a stand alone drive, and primarily if appending, understand when your tape will fill up and plan to change your tape before a job is likely to fill it completely. Do not try to completely fill it and you will then avoid a job waiting all night for the next tape.

e) If you know you need to do a critical restore from tape, then slide the write protect tabs across on the tape itself. Then if something distracts you before the restore has been actioned, no accidental overwrites can occur dure to a backup job that has run.

f) If your library supports barcodes then use them, but make sure you have the correct format barcodes for your hardware

g) Keep a record out side of Backup Exec of which tapes were used on which days including whether the tapes have been sent off-site. If you don't do this and something happens to your media server you won't know which tapes are most critical to bring back and work with for a restore. Which means your DR restore will take longer.

h) We recommend not using Infinite append on a media sets as appending for ever creates an infinitely long media family that can cause issues restoring depending on the history of the media sequence within the family. A job that starts by overwriting the first tape starts a new family, a job that spans to a second tape increases the size of the existing family.

i) Do spend time to look at the blogs relating to media handling written by Symantec Connect members - as some of the members have been using versions of Backup Exec for years and have day-today ongoing experience of setting up various scenarios, their "real-world" advice on how media sets, overwriting, and appending etc work is usually a very good supplement to our own documentation.

[CraigV]

Thanks for the insight Colin!