Netbackup Tape Full Erase

UIDAI_Backup · ‎06-24-2019

Hi Team,

I am trying to expire a lto5 tape, but it's not clearing fully.

I tried long erase also but no use, after writing 200gb or 300gb tape is showing Full

Please sugest us on this issue, we are unable to use some tapes like this.

Thanks in Advance.

RiaanBadenhorst · ‎06-24-2019

Hi,

The process of expiry involves the removal of the entries in the catalog and marking the tape as available in the EMM. It does not do an erase. The media is not even required to be in the library when you do it, because it doesn't get touched. All you need to do is expire the tape, erasure is not required unless you need to do it for some audit, or other, reasons. If you simply want the tape to be available, expiry will do.

from cli bpexpdate -m 'mediaID' -d 0

UIDAI_Backup · ‎06-30-2019

Hi Riaan,

Thanks for the reply, we have did the bpexpdate cmd tape is expiring but tape is showing full after 300gb. Capacity of tape is 1.5tb.

Please check the attached screenshot and give us a solution.

Marianne · ‎07-01-2019

@UIDAI_Backup

Please show us output of this command for one of the problematic Media-ids:

nbemmcmd -listmedia -mediaid <media-id>

Also have a look at the Solution by @mph999 in this post:

Tape capacity is determined by the tape drive itself. At some point near the physical end of the tape there is a 'special mark' that is detected by the drive firmware, and this sets a 'flag' in the tape driver. When NBU tries to send another block of data the tape driver refuses to accept it (well, the OS does ...) and it passes a message back to NBU that the tape is full, and we need a new one. So, without this message, NBU would try and write to the same tape for ever.

Reasons for low capacity are usually that the data isn't as compressible as you think it is. Other causes are bad firmware and even faulty hardware.

Another related post: https://vox.veritas.com/t5/NetBackup/tape-full/td-p/826801

Handy NetBackup Links

mph999 · ‎07-01-2019

The above quote from Marianne is correct. In summary - NBU has no understanding of tape capcaity and only knows a tape is full, when it is told by the operating/ system tape driver. THerefore the issue is outside of NBU, and is either a firmware, tape driver or hardware fault.

mph999 · ‎07-01-2019

From Technote: https://www.veritas.com/content/support/en_US/article.100014480.htm

Tapes not reaching capacity

Symptom:

Less data is written to the the tape than expected.

For example, only 300 GB of Data is written to a 400 GB capacity tape

NetBackup passes data to the OS, one block at a time, to be written to the tape drive. NetBackup has no concept of tape capacity. In theory, it would keep writing to the same tape "forever".

The tape drive firmware detects when the tape physically passes the logical end-of-tape. It then sets a 'flag' in the tape driver. There is still enough physical space on the tape for the current block to be written, so this completes successfully. NetBackup then tries to send the next block of data but the tape driver refuses, as the 'tape full' flag is set. The driver passes this 'tape full' message to the OS, which then passes it to NetBackup. Only once this has happened will Netbackup request the tape be changed.

Common causes of this issue are tape drive firmware, or faulty hardware.

There are no settings in NetBackup that control tape capacity. Your hardware vendor or OS support should be able to help to find the root cause.

UIDAI_Backup · ‎07-02-2019

Hi Marianne/Martin,

Thanks for your inputs, Here is the o/p for nbemmcmd

As you mentioned 'special mark' detected by drive firmware, how to rectify it. Shall i call to library vendor or anyother option.

Tape-1:

C:\Program Files\Veritas\NetBackup\bin\admincmd>nbemmcmd.exe -listmedia -mediaid i23819
NBEMMCMD, Version: 8.1
====================================================================
Media GUID: 04bae117-7dd4-481d-a657-efdf8d483232
Media ID: I23819
Partner: -
Media Type: HCART2
Volume Group: 000_00000_TLD
Application: Netbackup
Media Flags: 1
Description: Added by Media Manager
Barcode: UI23819L
Partner Barcode: --------
Last Write Host: hbdc1symsmsmed2
Created: 06/04/2019 13:47
Time Assigned: 06/05/2019 10:00
First Mount: 06/04/2019 20:00
Last Mount: 06/05/2019 10:38
Volume Expiration: -
Data Expiration: 07/06/2019 10:00
Last Written: 06/05/2019 10:00
Last Read: -
Robot Type: TLD
Robot Control Host: HDCSYMBKPCLUS
Robot Number: 0
Slot: 774
Side/Face: -
Cleanings Remaining: -
Number of Mounts: 4
Maximum Mounts Allowed: 0
Media Status: FULL
Kilobytes: 36366016
Images: 1
Valid Images: 1
Retention Period: 3
Number of Restores: 0
Optical Header Size Bytes: 1024
Optical Sector Size Bytes: 0
Optical Partition Size Bytes: 0
Last Header Offset: 0
Adamm Guid: 00000000-0000-0000-0000-000000000000
Rsm Guid: 00000000-0000-0000-0000-000000000000
Origin Host: NONE
Master Host: HDCSYMBKPCLUS
Server Group: NO_SHARING_GROUP
Upgrade Conflicts Flag:
Pool Number: 61
Volume Pool: NECPOOL_Monthly
Previous Pool Name: Scratch
Vault Flags: -
Vault Container: -
Vault Name: -
Vault Slot: -
Session ID: -
Date Vaulted: -
Return Date: -
Media on Hold: 0
====================================================================
Command completed successfully.

C:\Program Files\Veritas\NetBackup\bin\admincmd>

Tape-2:

C:\Program Files\Veritas\NetBackup\bin\admincmd>nbemmcmd.exe -listmedia -mediaid i23949
NBEMMCMD, Version: 8.1
====================================================================
Media GUID: 560ca1eb-7e78-4ed7-9087-4341b1451a92
Media ID: I23949
Partner: -
Media Type: HCART2
Volume Group: 000_00000_TLD
Application: Netbackup
Media Flags: 1
Description: Added by Media Manager
Barcode: UI23949L
Partner Barcode: --------
Last Write Host: hbdc1symsmsmed2
Created: 05/22/2019 19:40
Time Assigned: 06/05/2019 10:00
First Mount: 05/22/2019 19:58
Last Mount: 06/05/2019 10:28
Volume Expiration: -
Data Expiration: 07/06/2019 10:00
Last Written: 06/05/2019 10:00
Last Read: -
Robot Type: TLD
Robot Control Host: HDCSYMBKPCLUS
Robot Number: 0
Slot: 89
Side/Face: -
Cleanings Remaining: -
Number of Mounts: 7
Maximum Mounts Allowed: 0
Media Status: FULL
Kilobytes: 35021760
Images: 2
Valid Images: 2
Retention Period: 3
Number of Restores: 0
Optical Header Size Bytes: 1024
Optical Sector Size Bytes: 0
Optical Partition Size Bytes: 0
Last Header Offset: 0
Adamm Guid: 00000000-0000-0000-0000-000000000000
Rsm Guid: 00000000-0000-0000-0000-000000000000
Origin Host: NONE
Master Host: HDCSYMBKPCLUS
Server Group: NO_SHARING_GROUP
Upgrade Conflicts Flag:
Pool Number: 61
Volume Pool: NECPOOL_Monthly
Previous Pool Name: Scratch
Vault Flags: -
Vault Container: -
Vault Name: -
Vault Slot: -
Session ID: -
Date Vaulted: -
Return Date: -
Media on Hold: 0
====================================================================
Command completed successfully.

C:\Program Files\Veritas\NetBackup\bin\admincmd>

Marianne · ‎07-02-2019

@UIDAI_Backup

Thanks for the output - I have edited your post to remove domain name.

Before you log a call with your library or tape vendor, you may want to dig a bit deeper to find out if the problem is with a specific media server or particular tape drive.

We can see from the output that the same media server wrote these backups - hbdc1symsmsmed2.
Next thing is to check which tape drive(s) was used for these backups.
If the jobs are still in Activity Monitor, you should be able to see the drive name(s).
These details will also be in Media Logs report or All Log entries (if Days to Keep Logs is at default of 28 days).

Another place will be bptm logs on the media server - I suggest level 3 logging (although Support engineers will always ask for level 5 - wink-wink @mph999 ).

If you can pinpoint the media server and tape drive(s), compare tape driver installed on media server with drivers on other media servers.
Compare firmware on tape drive(s) with other tape drives.
Hopefully you will be able to pinpoint the issue to the problematic tape driver and/or firmware.

Handy NetBackup Links

Marianne · ‎07-03-2019

@UIDAI_Backup

I found a reply that you posted in the VOX SPAM Quarantine.

I tried to mark it as 'not spam', but then received a message saying
'Request Entity Too Large'.

So, I don't know if you maybe tried to attach a very large log file.

In all honesty, we do not have the capacity to analyze very large logs in this forum.
Level 5 logs should be uploaded to Veritas as part of a Support call.

Could you maybe copy out the lines that belong to the same PID in (square brackets [ ] ) that resulted in reduced capacity? You should be able to upload the reduced text file.

Handy NetBackup Links

UIDAI_Backup · ‎07-06-2019

Hi Marianne,

Sorry for late reply, Please check the below logs for other tape I23132

C:\Program Files\Veritas\NetBackup\bin\admincmd>nbemmcmd.exe -listmedia -mediaid i23132
NBEMMCMD, Version: 8.1
====================================================================
Media GUID: 887e6e84-8f05-49a7-855b-2d7f6344676d
Media ID: I23132
Partner: -
Media Type: HCART2
Volume Group: ---
Application: Netbackup
Media Flags: 1
Description: Added by Media Manager
Barcode: UI23132L
Partner Barcode: --------
Last Write Host: hbdc1symsmsmed2.cidr.gov.in
Created: 06/18/2019 14:32
Time Assigned: 06/21/2019 11:11
First Mount: 06/20/2019 13:34
Last Mount: 06/21/2019 12:22
Volume Expiration: -
Data Expiration: 07/22/2019 11:11
Last Written: 06/21/2019 11:11
Last Read: -
Robot Type: NONE
Robot Control Host: -
Robot Number: -
Slot: -
Side/Face: -
Cleanings Remaining: -
Number of Mounts: 8
Maximum Mounts Allowed: 0
Media Status: FULL MPX
Kilobytes: 17998720
Images: 1
Valid Images: 1
Retention Period: 3
Number of Restores: 0
Optical Header Size Bytes: 1024
Optical Sector Size Bytes: 0
Optical Partition Size Bytes: 0
Last Header Offset: 0
Adamm Guid: 00000000-0000-0000-0000-000000000000
Rsm Guid: 00000000-0000-0000-0000-000000000000
Origin Host: NONE
Master Host: HDCSYMBKPCLUS
Server Group: NO_SHARING_GROUP
Upgrade Conflicts Flag:
Pool Number: 5
Volume Pool: SMSPLPOOL
Previous Pool Name: SMSPLPOOL
Vault Flags: -
Vault Container: -
Vault Name: -
Vault Slot: -
Session ID: -
Date Vaulted: -
Return Date: -
Media on Hold: 0
====================================================================
Command completed successfully.

C:\Program Files\Veritas\NetBackup\bin\admincmd>

###Media Logs#####

Wed Jun 19 14:54:32 IST 2019 hbdc1symsmsmed2.cidr.gov.in HBDCSMLAPP005 550312 Warning media id I23132 load operation reported an error
Wed Jun 19 14:54:33 IST 2019 hbdc1symsmsmed2.cidr.gov.in 0 Warning TapeAlert Code: 0x03, Type: Warning, Flag: HARD ERROR, from drive Drive001 (index 29), Media Id I23132
Wed Jun 19 14:54:33 IST 2019 hbdc1symsmsmed2.cidr.gov.in 0 Error TapeAlert Code: 0x04, Type: Critical, Flag: MEDIA, from drive Drive001 (index 29), Media Id I23132
Wed Jun 19 14:54:34 IST 2019 hbdc1symsmsmed2.cidr.gov.in 0 Warning TapeAlert Code: 0x27, Type: Warning, Flag: DIAGNOSTICS REQ., from drive Drive001 (index 29), Media Id I23132
Wed Jun 19 14:54:34 IST 2019 hbdc1symsmsmed2.cidr.gov.in 0 Error TapeAlert Code: 0x37, Type: Critical, Flag: LOADING FAILURE, from drive Drive001 (index 29), Media Id I23132
Wed Jun 19 14:55:01 IST 2019 hbdc1symsmsmed2.cidr.gov.in HBDCNECBDB002 550753 Info begin writing backup id HBDCNECBDB002_1560936162, copy 1, fragment 1, to media id 14562L on drive Drive034 (index 12)
Wed Jun 19 15:09:21 IST 2019 hbdc1symsmsmed2.cidr.gov.in 0 Error error unloading media, TpErrno = Robot operation failed
Wed Jun 19 15:09:21 IST 2019 hbdc1symsmsmed2.cidr.gov.in 0 Warning TapeAlert Code: 0x03, Type: Warning, Flag: HARD ERROR, from drive Drive022 (index 31), Media Id
Wed Jun 19 15:09:22 IST 2019 hbdc1symsmsmed2.cidr.gov.in 0 Error TapeAlert Code: 0x04, Type: Critical, Flag: MEDIA, from drive Drive022 (index 31), Media Id
Wed Jun 19 15:09:22 IST 2019 hbdc1symsmsmed2.cidr.gov.in 0 Warning TapeAlert Code: 0x27, Type: Warning, Flag: DIAGNOSTICS REQ., from drive Drive022 (index 31), Media Id
Wed Jun 19 15:09:22 IST 2019 hbdc1symsmsmed2.cidr.gov.in 0 Error TapeAlert Code: 0x37, Type: Critical, Flag: LOADING FAILURE, from drive Drive022 (index 31), Media Id
Wed Jun 19 15:15:43 IST 2019 hbdc1symsmsmed2.cidr.gov.in 0 Error error unloading media, TpErrno = Robot operation failed
Wed Jun 19 15:15:43 IST 2019 hbdc1symsmsmed2.cidr.gov.in 0 Warning TapeAlert Code: 0x03, Type: Warning, Flag: HARD ERROR, from drive Drive001 (index 29), Media Id I23132
Wed Jun 19 15:15:43 IST 2019 hbdc1symsmsmed2.cidr.gov.in 0 Error TapeAlert Code: 0x04, Type: Critical, Flag: MEDIA, from drive Drive001 (index 29), Media Id I23132
Wed Jun 19 15:15:44 IST 2019 hbdc1symsmsmed2.cidr.gov.in 0 Warning TapeAlert Code: 0x27, Type: Warning, Flag: DIAGNOSTICS REQ., from drive Drive001 (index 29), Media Id I23132
Wed Jun 19 15:15:44 IST 2019 hbdc1symsmsmed2.cidr.gov.in 0 Error TapeAlert Code: 0x37, Type: Critical, Flag: LOADING FAILURE, from drive Drive001 (index 29), Media Id I23132
Wed Jun 19 15:18:02 IST 2019 hbdc1symsmsmed2.cidr.gov.in 0 Warning TapeAlert Code: 0x03, Type: Warning, Flag: HARD ERROR, from drive Drive001 (index 29), Media Id
Wed Jun 19 15:18:02 IST 2019 hbdc1symsmsmed2.cidr.gov.in 0 Error TapeAlert Code: 0x04, Type: Critical, Flag: MEDIA, from drive Drive001 (index 29), Media Id
Wed Jun 19 15:18:02 IST 2019 hbdc1symsmsmed2.cidr.gov.in 0 Warning TapeAlert Code: 0x27, Type: Warning, Flag: DIAGNOSTICS REQ., from drive Drive001 (index 29), Media Id
Wed Jun 19 15:18:03 IST 2019 hbdc1symsmsmed2.cidr.gov.in 0 Error TapeAlert Code: 0x37, Type: Critical, Flag: LOADING FAILURE, from drive Drive001 (index 29), Media Id
Wed Jun 19 15:30:37 IST 2019 hbdc1symsmsmed2.cidr.gov.in HBDCSMLAPP005 550312 Error error requesting media, TpErrno = Robot operation failed

mph999 · ‎07-07-2019

Wed Jun 19 14:54:32 IST 2019 hbdc1symsmsmed2.cidr.gov.in HBDCSMLAPP005 550312 Warning media id I23132 load operation reported an error
Wed Jun 19 14:54:33 IST 2019 hbdc1symsmsmed2.cidr.gov.in 0 Warning TapeAlert Code: 0x03, Type: Warning, Flag: HARD ERROR, from drive Drive001 (index 29), Media Id I23132
Wed Jun 19 14:54:33 IST 2019 hbdc1symsmsmed2.cidr.gov.in 0 Error TapeAlert Code: 0x04, Type: Critical, Flag: MEDIA, from drive Drive001 (index 29), Media Id I23132
Wed Jun 19 14:54:34 IST 2019 hbdc1symsmsmed2.cidr.gov.in 0 Warning TapeAlert Code: 0x27, Type: Warning, Flag: DIAGNOSTICS REQ., from drive Drive001 (index 29), Media Id I23132
Wed Jun 19 14:54:34 IST 2019 hbdc1symsmsmed2.cidr.gov.in 0 Error TapeAlert Code: 0x37, Type: Critical, Flag: LOADING FAILURE, from drive Drive001 (index 29), Media Id I23132
Wed Jun 19 15:15:43 IST 2019 hbdc1symsmsmed2.cidr.gov.in 0 Warning TapeAlert Code: 0x03, Type: Warning, Flag: HARD ERROR, from drive Drive001 (index 29), Media Id I23132
Wed Jun 19 15:15:43 IST 2019 hbdc1symsmsmed2.cidr.gov.in 0 Error TapeAlert Code: 0x04, Type: Critical, Flag: MEDIA, from drive Drive001 (index 29), Media Id I23132
Wed Jun 19 15:15:44 IST 2019 hbdc1symsmsmed2.cidr.gov.in 0 Warning TapeAlert Code: 0x27, Type: Warning, Flag: DIAGNOSTICS REQ., from drive Drive001 (index 29), Media Id I23132
Wed Jun 19 15:15:44 IST 2019 hbdc1symsmsmed2.cidr.gov.in 0 Error TapeAlert Code: 0x37, Type: Critical, Flag: LOADING FAILURE, from drive Drive001 (index 29), Media Id I23132

It would appear that drive001 might have an issue ... Tapealerts are sent from the drive firmware.
Certainly for these errors you need to speak with the hardware vendor and show them this email.

NetBackup cannot cause tapealerts.

UIDAI_Backup · ‎07-11-2019

Hi Marianne,

As per martin's post drive is not working properly, So shall i proceed to contact with vendor or else to check anything from our side.

Thank you very much for your support.

mph999 · ‎07-12-2019

I'd contact the vendor and get the drive fixed first. Hopefully that will resolve all the issues, but if not, we can then look at anything else that is left.