LTO5 Capacity with NetBackup 7.1.0.1

[75gary]

System Configuration:

Symantec NetBackup v7.1.0.1 1-master/2-media servers

Tape Library:  Quantum Scalar i500 with LTO5 tapes/tape drive

Issue:

Running out of tapes fast; just put into production in Nov 2011 and already consumed over 400 tapes; with half the tapes averaging only 40gb of data.

Would this be a hardware or software mis-configuration?  To only write 40gb on something that can do 1,500-3000GB!

[Mark_Solutions]

Anything else we should know about in your setup?

Encryption / compression set on policy attributes tab

Vault being used to suspend and eject tapes

All tapes either active, MPX or Full in the Status column and not Frozen (look or add the Media Status column in the Media View)

Thanks

[Nicolai]

Do you write data to a VTL first ?.

If the logical volume size in the VTL is defined to 40G and later staged to phycial tape, this could explain it.

[75gary]

I tried the bpimagelist command but got error - pasting below.

From NBU Console, I ran the "Images on Tape" report --- Attached a spreadsheet.

c:\Program Files\Veritas\NetBackup\bin\admincmd>bpimagelist -mlist -m server-name
bpimagelist: unrecognized option -mlist
USAGE: bpimagelist [-media] [-l|-L|-U|-idonly] [-tape]
           [-d mm/dd/yyyy HH:MM:SS] [-e mm/dd/yyyy HH:MM:SS] [-hoursago hours]
           [-keyword "keyword phrase"]
           [-client client_name] [-server server_name]
           [-backupid backup_id] [ [-option option_name] ... ]
           [-policy policy_name] [-pt policy_type]
           [-rl retention_level]
           [-sl sched_label] [-st sched_type]
           [-class_id data_classification_id_guid] [-stl_complete]
           [-stl_incomplete] [-stl_name storage_service_name]
           [-M master_server...] [-v]
           [-force]

c:\Program Files\Veritas\NetBackup\bin\admincmd>

[75gary]

Yes, I write data to a VTL first.

I should also point out that prior to Nov 2011; I wrote data to a HP-EML Library, LTO3 and I never went below 400gb; averaged as high as 800gb.

So I don't know how writing data first to a VTL would affect it?

[mph999]

Netbackup has no understanding of tape capacity. A tape is only marked full when a flag is set in the OS tapedriver, by the firmware on the tape drive. This flag is passed to netbackup, which then loads a new tape. The question here is to ask if bpmedialist -m is showing the tapes as Full Martin

[75gary]

I checked a couple of tapes and they marked it as Suspended.

c:\Program Files\Veritas\NetBackup\bin\admincmd>bpmedialist -m 500483
Server Host = master_server

 id     rl  images   allocated        last updated      density  kbytes restores

           vimages   expiration       last read         <------- STATUS ------->

--------------------------------------------------------------------------------

500483   3     11   01/13/2012 03:40  01/13/2012 06:09  hcart2   247827988     0

               11   02/13/2012 03:12        N/A         SUSPENDED

[Will_Restore]

~236Gb 

Are you vaulting daily?  That would account for SUSPENDED volume less than tape capacity. 

[Anonymous]

Query your tape media by running bpimagelist -mlist and see what how many images are written on your selection of tapes. And verify the space taken up.

bpmedialist -mlist -m EZX926

 id     rl  images   allocated        last updated      density  kbytes restores

           vimages   expiration       last read         <------- STATUS ------->

--------------------------------------------------------------------------------

EZX926   1    355   11/26/2011 02:30  01/13/2012 05:04   hcart  2634632263     0

              MPX   01/27/2012 05:04        N/A

Another thought:

Post the output of one of your tapes with this command

vmquery -m <mediaid> -X

You havent got tapes set to maximum mount =1 or a low value?

[J_H_Is_gone]

maybe you other people will get a clue from this

if the vtl tape is set to a size of 40g (as you want keep them small )  [ 75gary please tell us the size of your vtl tapes]

when you vault does it do one to one - meaning one vlt tape to one physical tape?

can vault append tapes? - meaning can it dup more than 1 vtl to 1 physical tape?

[mph999]

Perhaps need to check a few more tapes but it looks like the tape is vaulted as suspended, this could increase tape usage. Martin

[mph999]

Vault does not duplicate by tape, it duplicates by image. Martin

[Mark_Solutions]

75gary

It does look from what you have said that Vault is the most likely culprit here as I suspected in my first reply.

If vault is being used to duplicate from the VTL to the "real" tape library then you need to take a look at how it is configured if it is not doing very much before it ejects tapes and suspends them - either that or the Vault Schedule allows it to run many times during the day / night

Let us know your config - if vault is not being used then perhaps you have a scheduled script running that does your duplications?

[75gary]

Additional information:

 How does a "hcart Density" factor into this?

If there was a mis-configuration, would it have an effect on how much data is written to tape?

For instance, the LTO3 is hcart2 density and LTO5 is hcart3 Density.

[Mark_Solutions]

The only settings in NetBackup that will affect the amount of data written to tape are when you use NetBackup software compression or encryption

Nothing else will affect it apart from the tape being suspended - which your tapes are - so we need to know why they are suspended and in general only vault or a command being run manually will cause that

Run an All Log Entries for the last 24 hours and post it on here so that we try and spot what is happening

[mph999]

No setting in netbackup can affect the amount of data on a tape. The density has no technical meaning, a tape is only marked full when the OS indicates it is full. Martin

Edited to correct regarding Marks comment below.

Fair point, what I meant, was that for a given amount of data sent (theerfore either compressed by NBU or not) - the compress/ uncomressed capacity of a tape cannot be affected by NBU.

Martin

[75gary]

A lot of support feedback points to hardware issues.  Hardware vendor states their logs look clean.... time to dig deeper into the hardware vendor!

Attaching the 'all log entries' from this past weekend when i wrote to tape.  Logs were only for a few hours; had to delete names so its public info, etc.

[Yasuhisa_Ishika]

I'm looking for FULL tapes in "Images on Tape" list you posted. But it does not show which tape is really FULL.Can you post output of "bpmedialist -U"? By the way, tape usage on you site is as below.

Retention   TBs kept    Tapes
----------------------------
1month       64 TB      295

1year        19 TB       12

2month        3 TB       57

2week         1 TB        1

3month      111 TB       72
----------------------------
Total       198 TB      437

1month data spent a lot of tapes, but there are no "fragment 2" here. That is, there are no tapes in FULL state. I suspect that was caused by operation. As wrobbins mentioned, someone suspended tapes so that new tapes are allocated for next backup.

[mph999]

Vault could be the cause, if the tapes are gettig ejected before they are full, or if it is suspending them after each sesion.

I wrote this (below) for another post, might as well add it in here.  Not attempting to 'dodge' blame, but this shows why NetBackup has no control over how much data is written to tape.  However, this is outside of Vault - Vault could be causing your issue.

If the tapes that are 'not full' but should be are all offsite/ and or suspended, this would be an indication that Vault is send non'full tapes away.

It is difficult to stop Vault doing this, as it will use how-ever many drives it has to duplicate images.  However, you can try setting the number of partially full media  on the vault volume group, or reducing the number of write drives available.

Here are the other details:

Technically, NBU knows nothing about tape capaicty - the densities, hcart, hcart2 etc ... have no technical meaning, they are just a label.

It would be better to replace the densities with colours, green, red, blue - a green tape will ONLY go into a green drive and so on.  For example, I have a 4mm drive on one server, I can call that hcart3 and the tapes hcart3 , it will work fine, but the capacity of the tape remains the same.

Aside of not understanding tape capacity, NBU  does NOT actually write data to tapes directly.  It passes the data to the operating system, and requests it is written to tape using block size x.  NetBackup then relies on the operating system to complete the task.  (This is the reason that in more cases than not, tape drive issues are not the fault of NetBackup).

The data is sent from NetBackup to the operating system, a block at a time - at some point the EOT marker is detected by the tape drive - there is sufficient physical tape left to finish writing that block, but at this point, the firmware causes a 'tape full' flag to be set in the tape driver.

NetBackup then sends the next block of data, at this time, the tape driver (via the os) refuses the data, and a 'tape full' message is passed back to NetBackup.  At this time, a new tape is loaded.

Martin

[Omar_Villa]

Besides the increase of speeds and size on the new LTO-5 drives they support partitioning, have you check this feature on your drives? the drive supports 2 partitions but I cannot find the default sizes, check with your vendor and firmware and drivers versions, probably you have a bug with this.

http://www.lto.org/About/faq.html

Regards