NBU media doesn't fill up

[Drama]

Hi all.

We are having an issue on our NBU 6.5.6 platform. (Master, media an client are 6.5.6)

We have a policy that runs an oracle application schedule for archive. The job runs every hour, and the retention applied it one month (Multiplexing =20)

Today we found out that this policy in using our medias and filling them with onty 7727KB of data.

The total of medias in this situtation is 32, we thought that it was possible that a media was being used each day, but there's no relation between retention period and the number of medias "not filled"

We have also checked there is no mount limit, and no storage unit restriction.

The most curious thing is that the problem is happening only with this policy an this client, because we have an identical policy with the same configuration and other client that is working fine.

I'll try to sumarize the secuence with an example:

1 - Archive job executes, backing up 3MB in media S6B001 - Hour 1:00PM

2 - Archive job executes, backing up 5MB in media S6B001 - Hour 2:00PM

3 - Archive job executes, backing up 6MB in media S6B001 - Hour 3:00PM

4 - Archive job executes, backing up 7727KB in media S6B111 - Hour 4:00PM <------ PROBLEM

5 - Archive job executes, backing up 6MB in media S6B001 - Hour 5:00PM

6 - Archive job executes, backing up 6MB in media S6B001 - Hour 6:00PM

.

.

and so on....

Media S6B111 filled with 7727KB won't be used again...WHY? No frozen, no drive error, each job uses a diferent drive...I'm so lost :)

Thanks in advance!!!

[Rajesh_s1]

As per your description i came to know that "S6B001" media is using every hourly but "S6B111" is not using by other policies , please correct me if i am wrong. 

Can you pleas let us know which type of drive you are using and what's its capacity ?

If only one client policy is using this media mean there maight be difference in retention between other polices , one possibility is on the master server host properties by default "Allow multiple retention per media" will be unchecked. 

[mph999]

If you run bpmedialist -m S6B111

Is it showing as Full ?

Thanks,

Martin

[Marianne]

In addition to bpmedialist in Martin's post, run reports like 'availabe_media' from cmd (in netbackup\bin\goodies dir) as well as GUI reports like Tapes Written and Tapes List report (same as bpmedialist) on a daily basis to determine/monitor patterns.

Tape list (bpmedialist)  for S6B001 and S6B111 should point out what is 'different' about these 2 tapes.

[Drama]

Hello and thaks for responses.

Continue in the same situation. Another cartrigde was allocated tonight and written only with 7727 KB. At least the expiration of the first written medias is working fine and we're keeping the level in 30-32 occupied.

The media is SDLT3 and drives are SDLT600.  Native capacity 300GB, compressed 600GB.

We have UNchecked the attribute "Allow multiple retention per media " but retention is 1 month, standard for us, and there are another clients that uses the "good" media as well as the client with the problem.

All medias appear as Active Multiplexed.

I paste the output of bpmedialist and available_media:

S6S883 --> Bad media

S6S872 --> Media working fine

First for bad media:

sudo $PWD/bpmedialist -m s6s883
Server Host = bckflo03

 id     rl  images   allocated        last updated      density  kbytes restores
           vimages   expiration       last read         <------- STATUS ------->
--------------------------------------------------------------------------------
S6S883   3      1   10/12/2011 03:15  10/12/2011 03:15    dlt3        7727     0
              MPX   11/12/2011 02:15        N/A

sudo $PWD/available_media |grep S6S883
S6S883  DLT3     TLD      2       355    -       3     7727     ACTIVE/MPX

--------------------------------------------------

--------------------------------------------------

Now "good" media:

sudo $PWD/bpmedialist -m s6s872
Server Host = bckflo03

 id     rl  images   allocated        last updated      density  kbytes restores
           vimages   expiration       last read         <------- STATUS ------->
--------------------------------------------------------------------------------
S6S872   3    424   10/19/2011 15:11  10/28/2011 12:11    dlt3   142448029     0
              MPX   11/28/2011 11:16        N/A
 

 sudo $PWD/available_media |grep S6S872
S6S872  DLT3     TLD      2       266    -       3     142448029        ACTIVE/MPX

I see no difference between them :(

Is there any chance that RMAN has a problem that causes this unusual use of the tape.

Thanks in advance!!!

[Will_Restore]

please post output for media S6B001 S6B111  instead of S6S883 S6S872  

[Omar_Villa]

If you see the expiration times are different u can enable the allow multiple retentions option and get your tapes full but issue comes with expirations. Better can be eject tapes daily.

[Drama]

Hello again.

Medias S6B001 and S6B111 are already expired, so they don't apply to this issue.

I took S6S883 and S6S872 as example of a bad and good tape use respectively.

Expiration time is different because last written time is also different, I think important data here is retention level and it's the same for the two cartridges.

So, the point in the example with this two tapes, is to show how one of them is only used 7727 KB while the other is used and filled normally.

We opened a support request and we're waiting till monday to continue the case.

Thanks.

[Marianne]

What is MPX level set to in Policy schedule? Double-check Application Backup Schedule. If more than one job is due and MPX = 1 and more that one tape drive is available, one job will go to one drive using one piece of media and the next job will use another drive with a different piece of media.

If media usage is that important to you, consider implementing  Maximum number of partially full media in Volume Pool properties. (See NBU 6.5 Admin Guide I for UNIX/Linux http://www.symantec.com/docs/TECH52801 ).

[Drama]

Hello Marianne.

Actually the MPX level for this schedule is set up to 20. We have 18 SDLT drives in the robotic library and in peak hours the average of available drives is 8. As you can see we have an over sized robotic platform at the moment.

Nonetheless, we can easily fix up the problem, because we have also bigs STUs available, but we really want to know why this strange behavior.

And what is more strange is that we have another 3 policies exactly like the problematic one, even schedules, and they're behaving well. :S

Thanks!!

[Marianne]

We can see from bpmedialist output that S6S883 was used once and and never appended to. For some or other reason this tapes does not seem to be considered for more backups.

Volume (hardware) expiration perhaps?

Please run nbemmcmd -listmedia -mediaid S6S883

The default for Volume expiration is 'Never'.

[Drama]

I paste the output of nbemmcmd command for the two medias of the example.

Regarding to harware expiration, media managing is made only by software, even eject processes.

Thanks in advance for your time!

sudo $PWD/nbemmcmd -listmedia -mediaid S6S883

NBEMMCMD, Version:6.5.6

====================================================================

Media GUID:                     38b19cbc-5dd8-11df-8000-ec7e07d1f564

Media ID:                       S6S883

Partner:                        -

Media Type:                     DLT3

Volume Group:                   00_002_TLD

Application:                    Netbackup

Media Flags:                    1

Description:                    ---

Barcode:                        S6S883

Partner Barcode:                --------

Last Write Host:                bckflo03

Created:                        05/12/2010 13:08

Time Assigned:                  10/12/2011 03:15

First Mount:                    05/12/2010 14:28

Last Mount:                     10/12/2011 03:16

Volume Expiration:              -

Data Expiration:                11/12/2011 02:15

Last Written:                   10/12/2011 03:15

Last Read:                      -

Robot Type:                     TLD

Robot Control Host:             backup00

Robot Number:                   2

Slot:                           355

Side/Face:                      -

Cleanings Remaining:            -

Number of Mounts:               34

Maximum Mounts Allowed:         0

Media Status:                   MPX

Kilobytes:                      7727

Images:                         1

Valid Images:                   1

Retention Period:               3

Number of Restores:             0

Optical Header Size Bytes:      1024

Optical Sector Size Bytes:      0

Optical Partition Size Bytes:   0

Last Header Offset:             35

Adamm Guid:                     00000000-0000-0000-0000-000000000000

Rsm Guid:                       00000000-0000-0000-0000-000000000000

Origin Host:                    NONE

Master Host:                    backup00

Server Group:                   NO_SHARING_GROUP

Upgrade Conflicts Flag:

Pool Number:                    22

Volume Pool:                    DISK_STAGING

Previous Pool Name:             SCRATCH

Vault Flags:                    -

Vault Container:                -

Vault Name:                     -

Vault Slot:                     -

Session ID:                     -

Date Vaulted:                   -

Return Date:                    -

====================================================================

Command completed successfully.

[admbck@bckm01] $ sudo $PWD/nbemmcmd -listmedia -mediaid S6S872

NBEMMCMD, Version:6.5.6

====================================================================

Media GUID:                     382980fc-5dd8-11df-8000-ec7e07d1f564

Media ID:                       S6S872

Partner:                        -

Media Type:                     DLT3

Volume Group:                   00_002_TLD

Application:                    Netbackup

Media Flags:                    1

Description:                    ---

Barcode:                        S6S872

Partner Barcode:                --------

Last Write Host:                bckflo03

Created:                        05/12/2010 13:08

Time Assigned:                  10/19/2011 15:11

First Mount:                    05/12/2010 13:51

Last Mount:                     10/31/2011 09:11

Volume Expiration:              -

Data Expiration:                12/01/2011 09:17

Last Written:                   10/31/2011 09:17

Last Read:                      -

Robot Type:                     TLD

Robot Control Host:             backup00

Robot Number:                   2

Slot:                           266

Side/Face:                      -

Cleanings Remaining:            -

Number of Mounts:               165

Maximum Mounts Allowed:         0

Media Status:                   MPX

Kilobytes:                      493585295

Images:                         1134

Valid Images:                   1134

Retention Period:               3

Number of Restores:             0

Optical Header Size Bytes:      1024

Optical Sector Size Bytes:      0

Optical Partition Size Bytes:   0

Last Header Offset:             1932939

Adamm Guid:                     00000000-0000-0000-0000-000000000000

Rsm Guid:                       00000000-0000-0000-0000-000000000000

Origin Host:                    NONE

Master Host:                    backup00

Server Group:                   NO_SHARING_GROUP

Upgrade Conflicts Flag:

Pool Number:                    22

Volume Pool:                    DISK_STAGING

Previous Pool Name:             SCRATCH

Vault Flags:                    -

Vault Container:                -

Vault Name:                     -

Vault Slot:                     -

Session ID:                     -

Date Vaulted:                   -

Return Date:                    -

====================================================================

Command completed successfully.

[Omar_Villa]

i just notice from your source post:

We have a policy that runs an oracle application schedule for archive. The job runs every hour, and the retention applied it one month (Multiplexing =20)""" 

Multiplexing is way to high, try to reduce that to 4, 20 can make the restore unstable and also will take a lot of time to get it done.

[Marianne]

I cannot see anything obvious that prevents S6S883 from being used.

I'm curious to see if Symantec Support can find a reason for this....

[Drama]

Ok, thanks anyway.

I will let you know if we get any solution from Symantec Support.

Thanks!!!

[KEELIN_HART_2]

We appear to be having a similar issue, whereby the media for our archive logs (these are in a seperate volume pool) are all being used, but never filled.

[Marianne]

Consider implementing  Maximum number of partially full media in Volume Pool properties. (See NBU 6.5 Admin Guide I for UNIX/Linux http://www.symantec.com/docs/TECH52801 ).

[KEELIN_HART_2]

Thanks for that Marianne. I wasn't aware of this new option. Although I thought the default behaviour of NetBackup (pre 6.x anyway) was to fill active media before resorting to available media?

[Drama]

Still no notice from Symantec Support, they asked us for verbose 5 logs and we don't have enough space in our filesystem at the moment. We are expecting for a new SAN disk assignation to continue this case.

Regards.