cancel
Showing results for 
Search instead for 
Did you mean: 

Status 96 - tapes getting FROZEN

Ishac
Level 3

Hello Marianne,

I am having a similar issue.

 

tape drive , control = TLD , Ready = NO

Below is my output for nbrbutil -dump

D:\Program Files\Veritas\NetBackup\bin\admincmd>nbrbutil -dump

Allocation Requests
(AllocationRequestSeq )


Allocations
(AllocationSeq )

MDS allocations in EMM:

Please advise

***** SUBJECT EDITED to reflect real issue  *****

14 REPLIES 14

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified
@Ishac

I have move your comment in https://vox.veritas.com/t5/NetBackup/problem-tape-drive-control-TLD-Ready-NO/td-p/872482 to a new discussion.

Please start from scratch and explain what exactly the problem is?

Hello Marianne,

My device monitor currently shows the Drives are up but not ready. 

This is causing my backups to fail

1: (96) unable to allocate new media for backup, storage unit has none available

Mar 13, 2020 11:01:28 AM - begin Duplicate
Mar 13, 2020 11:01:29 AM - requesting resource  srabkup01dal-hcart-robot-tld-0
Mar 13, 2020 11:01:30 AM - Error nbjm (pid=15860) NBU status: 96, EMM status: No media is available
Mar 13, 2020 11:01:30 AM - Error nbjm (pid=15860) NBU status: 96, EMM status: No media is available
Mar 13, 2020 11:01:51 AM - end Duplicate; elapsed time 0:00:23
unable to allocate new media for backup, storage unit has none available  (96)

I have enough tapes in my scratch pool but Netbackup wont move the tapes to the drives. I am not sure why.

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified
Up, not ready is not a problem. It simply means that no tape is loaded.

You need to troubleshoot status 96.
Show us output of available_media and the SLP config.

002005 4 0 03/13/2020 13:18 N/A hcart 0 0 0 N/A N/A FROZEN 0 002006 4 0 03/13/2020 13:18 N/A hcart 0 0 0 N/A N/A FROZEN 0 002009 3 0 03/13/2020 13:18 N/A hcart 0 0 0 N/A N/A FROZEN 0 002012 3 0 03/13/2020 13:18 N/A hcart 0 0 0 N/A N/A FROZEN 0 002013 4 0 03/13/2020 13:18 N/A hcart 0 0 0 N/A N/A FROZEN 0 002014 3 0 03/13/2020 13:18 N/A hcart 0 0 0 N/A N/A FROZEN 0 002015 4 0 03/13/2020 13:18 N/A hcart 0 0 0 N/A N/A FROZEN 0 002016 4 0 03/13/2020 13:18 N/A hcart 0 0 0 N/A N/A FROZEN 0 002017 4 0 03/13/2020 13:18 N/A hcart 0 0 0 N/A N/A FROZEN 0 002018 3 0 03/13/2020 13:18 N/A hcart 0 0 0 N/A N/A FROZEN 0 002019 4 0 03/13/2020 13:18 N/A hcart 0 0 0 N/A N/A FROZEN 0 002020 4 0 03/13/2020 13:18 N/A hcart 0 0 0 N/A N/A FROZEN 0 002022 3 0 03/13/2020 13:18 N/A hcart 0 0 0 N/A N/A FROZEN 0 002030 8 21 10/31/2019 13:46 10/31/2019 15:56 hcart 943197308 0 21 09/20/2020 16:49 N/A FULL 0 002037 8 20 09/21/2019 22:45 09/22/2019 01:12 hcart 951539359 0 20 09/13/2020 00:32 N/A FULL 0 SIS262 3 0 03/13/2020 13:18 N/A hcart 0 0 0 N/A N/A FROZEN 0 SIS294 4 0 03/13/2020 13:18 N/A hcart 0 0 0 N/A N/A FROZEN 0 SIS299 3 0 03/13/2020 13:18 N/A hcart 0 0 0 N/A N/A FROZEN 0 D:\Program Files\Veritas\NetBackup\bin\admincmd> The tapes get frozen for some reason. I have tried to debug issue but have failed.

 

000196 8 18 08/09/2019 23:00 08/11/2019 22:00 hcart 878943808 0
14 08/09/2020 00:00 N/A FULL
0

000197 3 0 03/13/2020 13:18 N/A hcart 0 0
0 N/A N/A FROZEN
0

000198 4 9 03/10/2020 20:00 03/12/2020 21:07 hcart 131593097 0
9 05/13/2020 21:07 N/A
0

000199 3 0 03/13/2020 13:18 N/A hcart 0 0
0 N/A N/A FROZEN
0

000221 3 0 03/13/2020 13:18 N/A hcart 0 0
0 N/A N/A FROZEN
0

000232 8 27 10/31/2019 15:56 10/31/2019 19:47 hcart 1223241516 0
27 09/21/2020 00:00 N/A FULL
0

000241 4 0 03/13/2020 13:18 N/A hcart 0 0
0 N/A N/A FROZEN
0

002001 8 21 09/22/2019 06:37 10/31/2019 13:46 hcart 983261056 0
15 09/20/2020 16:49 N/A FULL
0

002002 8 35 09/22/2019 04:35 09/22/2019 06:37 hcart 886995751 0
35 09/07/2020 00:33 N/A FULL
0

002005 4 0 03/13/2020 13:18 N/A hcart 0 0
0 N/A N/A FROZEN
0

002006 4 0 03/13/2020 13:18 N/A hcart 0 0
0 N/A N/A FROZEN
0

002009 3 0 03/13/2020 13:18 N/A hcart 0 0
0 N/A N/A FROZEN
0

002012 3 0 03/13/2020 13:18 N/A hcart 0 0
0 N/A N/A FROZEN
0

002013 4 0 03/13/2020 13:18 N/A hcart 0 0
0 N/A N/A FROZEN
0

002014 3 0 03/13/2020 13:18 N/A hcart 0 0
0 N/A N/A FROZEN
0

002015 4 0 03/13/2020 13:18 N/A hcart 0 0
0 N/A N/A FROZEN
0

002016 4 0 03/13/2020 13:18 N/A hcart 0 0
0 N/A N/A FROZEN
0

002017 4 0 03/13/2020 13:18 N/A hcart 0 0
0 N/A N/A FROZEN
0

002018 3 0 03/13/2020 13:18 N/A hcart 0 0
0 N/A N/A FROZEN
0

002019 4 0 03/13/2020 13:18 N/A hcart 0 0
0 N/A N/A FROZEN
0

002020 4 0 03/13/2020 13:18 N/A hcart 0 0
0 N/A N/A FROZEN
0

002022 3 0 03/13/2020 13:18 N/A hcart 0 0
0 N/A N/A FROZEN
0

002030 8 21 10/31/2019 13:46 10/31/2019 15:56 hcart 943197308 0
21 09/20/2020 16:49 N/A FULL
0

002037 8 20 09/21/2019 22:45 09/22/2019 01:12 hcart 951539359 0
20 09/13/2020 00:32 N/A FULL
0

SIS262 3 0 03/13/2020 13:18 N/A hcart 0 0
0 N/A N/A FROZEN
0

SIS294 4 0 03/13/2020 13:18 N/A hcart 0 0
0 N/A N/A FROZEN
0

SIS299 3 0 03/13/2020 13:18 N/A hcart 0 0
0 N/A N/A FROZEN
0

I have tapes that just freeze. I am not sure why. I have tried to debug but failed.

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified
How have you tried to troubleshoot frozen tapes?
Were there any errors in Activity monitor when the tapes were used today and frozen?
Do you have bptm log?
Have you checked tape logs report?

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

@Ishac 

Please help us to help you. 
Quite a number of tapes got FROZEN on 03/13/2020 at 13:18.

Have you checked for errors in Activity Monitor around this time?
There should be some information in Job Details.
Maybe new tapes with non-Veritas headers?

quebek
Moderator
Moderator
   VIP    Certified

Hello

In order to see why tapes were frozen I would run this on master server

bperror -media -hoursago 240 |findstr /i freez

for windows or for linux based master server

bperror -media -hoursago 240|grep -i freez

You can also review bptm log on media server for the same.

Hello,

Attached is the screenshot from bperror. I have unfreezed the tapes and but issue is still happening. Once it freezes one tape, it freezes the next and then all the tapes.

Thanks.

Ishac

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

@Ishac , have you read the error messages and tried to figure out how this happened?

All of the tapes were frozen because of 'incorrect media found in drive' , e.g.
Found 002012, expected 2012L4.

This says to me that something changed in your environment.
Maybe a new tape library at some point in time?
Looking at above media id's, my guess is that the full 8-digit barcode is 002012L4.

NBU will by default add the last 6 digits as media id, unless a backup admin changes this. So, initially the tapes were added with 1st 6 characters of the barcode (or the other way round if I look at @mph999 's post.... )
Tapes were used for backups and 002012 was written on the tape as internal label. Then, something changed in your environment and the same tapes were added as new tapes. The same tape is now added with last 6 characters as 2012L4.
So, mismatch between internal label and media id.

It's going to be a lengthy process to fix this. You need to figure out when this happened, how many media-id's in the environment have valid images with 1st 6 digits and how many with last 6 (ending in L4).

Could you please run available_media again and save output to a text file?
e.g. available_media > C:\temp\media.txt.
(If C:\temp\ exists)

Run this command as well (from volmgr\bin):
vmcheckxxx -rn 0 -rt tld > C:\temp\robot.txt
(I hope I have the command correct, it is after hours and I'm using my mobile phone. )

Upload media.txt and robot.txt.
We will try to help you fix this problem based on the output.

mph999
Level 6
Employee Accredited

Found 002012, expected 2012L4

.... so the media ID of the tape loaded is 002012, but the media ID written to the header of the tape is 2012L4

As Marianne mentions, something has chaged, as a guess the tapes have been moved to a new/ different library, or the library has been replaced, had some work done on it (firmware upgrade), had a part replaced etc ...

The barcode NBU displays (not media ID ) is controlled 100% by the library.  The tape should have a 8 digit barcode on the label:

002012L4

The library can display all 8 characters, or may be configured to display a selection of them eg. 002012 or 2012L4 or maybe even 02012L (but lets not go there ,,,).   NBU does NOT control the barcode - easiest way to see what the barcodes are, is to run robtest, then command 's s'. (show slots).  

The media ID is derived from the barcode, by default NBU takes the last 6 characters, so if the library is displaying all 8 characters it would be 2012L4.  If the library is only displaying the first 6 characters, it would be well, the same - 002012 - you get the idea.

NBU identically a tape as new, and will add it into the config, if the bearcode is different.

002012L4

002012

2012L4 

... would all be considered different tapes, even though quite clearly they are the same

If you did have two different barcodes.

AA1234L4 and BB1234L4 and used the last 6 digits as the media ID you would have a problem, as NBU would see two different tapes, but the media ID would work out to be the same - 1234L4 for both, you would get a duplicate media ID error.

Going back to this:

Found 002012, expected 2012L4

I cannot tell what the barcode of the tape was/ is (as displayed by the robot), nor if you have the tape displayed twice with to different barcodes eg. 002012L4 and 2012L4 - would need to see vmquery -a output .

But ... the tape originally had (remember the robot displays what characters in the barcode we see) ....

002012L4 or 2012L4 and the media ID generation rule was either set to 3:4:5:6:7:8 or was default which equals the same, so the media ended up with media ID 2012L4 - this was written to the tape header when it was labelled.

Since that point, 'another tape' has been added that has media ID 002012 - we know this is the same tape, but in media ID terms 002012 and 2012L4 are different.  When the tape 002012 is loaded, NBU reads the tape header and discovers in fact, it is 2012L4 - hence the error (a safety feature to prevent the wrong tape being overwritten).

So, since the tape was originally written to (which could have been on a different system, and they were moved to a different environment / robot ) the barcode as displayed to NBU has changed - I don't know what to, but it is different than it was originally, as an educated guess it went from 2012L4 to 002012  ...

I don't think it went to 002012L4 - this would have given the same media ID again, 2012L4 - not the issue you have.

I think therefore the barcode changed to 002012 - this would give the media ID 002012 and the tape would be added without getting a duplicate media ID error.

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

@Ishac 

Is there anything else that we can do to try and assist?