Vault fail with error 306

[Mastana420]

Hi

We usually have Vault run on mondays, but for some reason i had to run this on wednesday this week. It only copies the full backups which runs on Fridays, so i changed the amount of days it needs to go back in order to get theese full backups. I added 3 additional days to it. I had around 63 empty tapes in scratch before i started the Vault. Today morning i saw the Vault job, and it had failed with error code 306. I did some research and figured that it failed because there were no media left in scratch. Usually vault does not use more then 20-25 Tapes pr vault session, so wonder why it used all 63 of them still wanting more.

Only reason i come up with is that i might have screwed it up when editing amount of days it goes back to get hold of all the full backups. It was set to 3 days and 6 hours, so i changed it to 6 days and 6 hours (basically added 3 days). So it might have gotten some full backups even from thursday. We usually run full backups only on fridays, but we might have some full backups running even on thursday.

When trying to eject the tapes of this particular Vault session, it only shows 3 Tapes, meaning that it never managed to end the jobs on rest of the 60 tapes. I ran the following command on some of the media ID which i had in scratch prior to start of Vault session: nbemmcmd -listmedia -mediaid <media_ID> and here is the output: (notice that i have removed some of the sesnsitive data from this log)

NBEMMCMD, Version: 7.6.1.2
====================================================================
Media GUID:                     d82d47e0-360a-11e5-8000-e7571c8337b0
Media ID:                       100805
Partner:                        -
Media Type:                     HCART
Volume Group:                   002_00000_ACS
Application:                    Netbackup
Media Flags:                    1
Description:                    ---
Barcode:                        100805
Partner Barcode:                --------
Last Write Host:                < removed >
Created:                        07/29/2015 13:59
Time Assigned:                  07/29/2015 15:36
First Mount:                    07/29/2015 15:44
Last Mount:                     07/29/2015 15:44
Volume Expiration:              -
Data Expiration:                -
Last Written:                   -
Last Read:                      -
Robot Type:                     ACS
Robot Control Host:             NONE
Robot Number:                   0
AcsAcs:                         0
AcsLsm:                         2
Cleanings Remaining:            -
Number of Mounts:               1
Maximum Mounts Allowed:         0
Media Status:                   FROZEN
Kilobytes:                      0
Images:                         0
Valid Images:                   0
Retention Period:               8
Number of Restores:             0
Optical Header Size Bytes:      1024
Optical Sector Size Bytes:      0
Optical Partition Size Bytes:   0
Last Header Offset:             0
Adamm Guid:                     00000000-0000-0000-0000-000000000000
Rsm Guid:                       00000000-0000-0000-0000-000000000000
Origin Host:                    NONE
Master Host:                    <removed>
Server Group:                   NO_SHARING_GROUP
Upgrade Conflicts Flag:
Pool Number:                    6
Volume Pool:                    Offsite
Previous Pool Name:             scratch
Vault Flags:                    -
Vault Container:                -
Vault Name:                     -
Vault Slot:                     -
Session ID:                     -
Date Vaulted:                   -
Return Date:                    -
Media on Hold:                  0
====================================================================
 

Apart from thoose 3 tapes which it had written successfully to, rest of 60 media ID returns with same result, basically having no valid image on them and in a Frozen state. Should i simply use the bpexpdate command to start reusing them?

And i will appreciate if someone could explain why this session used so many tapes, could it be the days i ammended in the Vault policy?

[GulzarShaikhAUS]

The one you mentioned in the command is frozen. May be the same might have happened with some other medias and you ran out of medias. Just take a look at the medias frozen and you can check the date and time of last mount which will help you in understanding about the medias used for that perticular day.

Alternatively you can check the session log located under /netbackup/vault/sessions/ and bptm log for that session. 

[GulzarShaikhAUS · Accepted solution]

One more thing is if the medias are FROZEN , first try to determine why those medias are frozen, if these are new medias or old one where atleast one successful backup has happened.. If  you think all is OK you can unfreeze the medias using bpmedia -unfreeze -m <mediaID> command and retry a tape operation to see if the media is freezing again.

[Mastana420]

All of the media were frozen, and what i did is to use the bpexpdate command to get them unforzen and delete anything on them. All 60 of them are new tapes, which have never been used before. Let me get back to you regarding why they got in to frozen state by looking in the log you mentioned. I thought everything got screwed up because of the days i edited in the Vault profile.

[Mastana420]

I have looked at the following log: /Netbackup/Logs/bptm

Its a big log so i tried using search command to find anything mentioning Frozen or the job ID i have from Activity monitor, but seems like nothing there. I dont know what verbose level this log has been set to but it should have some information about the reason.

[Mastana420]

Oki seems like it is freezing the media because of the following error:

Freezing media id, it contains ANSI-format data and cannot be used for backups

Remember theese are brand new tapes just arrived from Oracle. Dont know if this information is relevant.

[Marianne · Accepted solution]

I am surprized that you are seeing this problem only now...

This has always been a problem with pre-labelled tapes received from STK/SUN/Oracle - their tapes have been pre-initialized with ANSI headers.

You need to add
ALLOW_MEDIA_OVERWRITE = ANSI 
to bp.conf on all media servers attached to ACSLS (or use Host Properties -> Media Servers -> Media)

You then need to use bpmedia command (NOT bpexpdate) to unfreeze the frozen media.

[GulzarShaikhAUS]

one of the reason a new media would freeze is wrong media type. Just see if the medias are of correct type and can be used with the tape library you have.A common mistake is tryig to LTO5 medias with LTO4 drives. In such cases you can also see errors on your tape library.

You can test a simple backup running to the tape library. Created a test volume pool and move a media in it and direct the backup to it. While doing so first delete existing bptm log or rename it. It will be good if there are no backups/restore/duplication running at that time. Once you have captured the log send it across. ALso make sure verbosity is set to 5.

[GulzarShaikhAUS]

Got the updated late so ended posting above response. A mentioned by Marianne you can add the entry and test a backup. It should be OK.

[Mastana420]

Marianne does it require restart of media servers?

SymGuy-IT: I am gonna give you suggestion a go aswell by creating a new test pool, moving 1 or 2 tapes in question to that pool, creating and directing job to that pool. See without change, and then after that do as suggested by Marianne on adding the overwrite option, and see the result just to be on the safer side.

[Marianne]

No restart.

Simply add the entry.

[Mastana420]

Ok, i have now completed the test, and editing the ANSI settings on media server helped. Now i have edited all the media servers in my enviroment so they are allowed to format ANSI tapes. Will wait until monday before i close this thread.

Wonder why we have never used this option before, and how tapes worked straight out of the box???
Does it have to do with kind of order we made, we ordered the following:
 

StorageTek T10000 T2 tape cartridge: initialization option, T10000D format, ASCII (for factory installation)

Another question is, will this change cause any problems to previous backups that have been taken??

[Marianne]

I don't know about the various options on the Oracle price list, but it seems that the 'ASCII format' may mean the same as ANSI format.

As long as the OVERWRITE entry on all media servers, you don't have to worry about new tapes in future.

This does not affect any previously written tapes - NBU writes NBU header when it labels the tape on a first write.

The ALLOW ... OVERWRITE entry is to allow NBU to overwrite the 'non-NBU' ANSI header.

To learn more about this feature, see: http://www.symantec.com/docs/HOWTO105720 

[Mastana420]

Oki, seems like it did the trick. Now Vault has been running sins monday morning. Thanks alot for your help, and credit goes to Symguy-IT and Marianne.

[GulzarShaikhAUS]

Good to know it helped.. You can mark the post for solution..

[Mastana420]

Already submitted a spliit Solution which is pending :)