Basic Disk Storage unit group does not migrate the storage units automatically

[Krishna_Sriniva]

We have been using Storage Unit Group which has four Basic Disk Storage Units configured and all the storage units have high water mark level set to 89% still the storage units are getting 100% full and the storage unit doesnt have any effect .

[Krishna_Sriniva]

/usr/openv/netbackup/bin/admincmd/bpstulist -g -U

Label: muttley-hcart-robot-tld-0
Storage Unit Type: Media Manager
Host Connection: muttley
Number of Drives: 2
On Demand Only: no
Max MPX/drive: 4
Density: hcart - 1/2 Inch Cartridge
Robot Type/Number: TLD / 0
Max Fragment Size: 1048575 MB

Label: nbu02
Storage Unit Type: Disk
Storage Unit Subtype: Basic (1)
Host Connection: muttley
Concurrent Jobs: 4
On Demand Only: no
Max MPX: 1
Path: "/nbu02"
Max Fragment Size: 524287 MB
Stage data: yes
Block Sharing: no
High Water Mark: 89
Low Water Mark: 70
Ok On Root: no

Label: nbu03
Storage Unit Type: Disk
Storage Unit Subtype: Basic (1)
Host Connection: muttley
Concurrent Jobs: 2
On Demand Only: yes
Max MPX: 1
Path: "/nbu03"
Max Fragment Size: 524287 MB
Stage data: yes
Block Sharing: no
High Water Mark: 89
Low Water Mark: 70
Ok On Root: no

Label: nbu04
Storage Unit Type: Disk
Storage Unit Subtype: Basic (1)
Host Connection: muttley
Concurrent Jobs: 4
On Demand Only: no
Max MPX: 1
Path: "/nbu04"
Max Fragment Size: 524287 MB
Stage data: yes
Block Sharing: no
High Water Mark: 89
Low Water Mark: 70
Ok On Root: no

Label: nbu05
Storage Unit Type: Disk
Storage Unit Subtype: Basic (1)
Host Connection: muttley
Concurrent Jobs: 4
On Demand Only: no
Max MPX: 1
Path: "/nbu05"
Max Fragment Size: 524287 MB
Stage data: yes
Block Sharing: no
High Water Mark: 89
Low Water Mark: 70
Ok On Root: no

Label: Catalog_Backup
Storage Unit Type: Disk
Storage Unit Subtype: Basic (1)
Host Connection: muttley
Concurrent Jobs: 1
On Demand Only: yes
Max MPX: 1
Path: "/Catalogbackup"
Max Fragment Size: 524287 MB
Stage data: no
Block Sharing: no
High Water Mark: 90
Low Water Mark: 80
Ok On Root: no

Label: real1
Storage Unit Type: Disk
Storage Unit Subtype: Basic (1)
Host Connection: muttley
Concurrent Jobs: 4
On Demand Only: yes
Max MPX: 1
Path: "/nbudisk/real1"
Max Fragment Size: 524287 MB
Stage data: yes
Block Sharing: no
High Water Mark: 89
Low Water Mark: 70
Ok On Root: no

Label: real2
Storage Unit Type: Disk
Storage Unit Subtype: Basic (1)
Host Connection: muttley
Concurrent Jobs: 4
On Demand Only: no
Max MPX: 1
Path: "/nbudisk/real2"
Max Fragment Size: 524287 MB
Stage data: yes
Block Sharing: no
High Water Mark: 89
Low Water Mark: 70
Ok On Root: no

Label: real3
Storage Unit Type: Disk
Storage Unit Subtype: Basic (1)
Host Connection: muttley
Concurrent Jobs: 2
On Demand Only: yes
Max MPX: 1
Path: "/nbudisk/real3"
Max Fragment Size: 524287 MB
Stage data: yes
Block Sharing: no
High Water Mark: 89
Low Water Mark: 70
Ok On Root: no

Label: real4
Storage Unit Type: Disk
Storage Unit Subtype: Basic (1)
Host Connection: muttley
Concurrent Jobs: 4
On Demand Only: no
Max MPX: 1
Path: "/nbudisk/real4"
Max Fragment Size: 524287 MB
Stage data: yes
Block Sharing: no
High Water Mark: 89
Low Water Mark: 70
Ok On Root: no

Label: nbu01
Storage Unit Type: Disk
Storage Unit Subtype: Basic (1)
Host Connection: muttley
Concurrent Jobs: 4
On Demand Only: yes
Max MPX: 1
Path: "/nbu01"
Max Fragment Size: 524287 MB
Stage data: yes
Block Sharing: no
High Water Mark: 89
Low Water Mark: 70
Ok On Root: no

Label: nbu06
Storage Unit Type: Disk
Storage Unit Subtype: Basic (1)
Host Connection: muttley
Concurrent Jobs: 4
On Demand Only: yes
Max MPX: 1
Path: "/nbu06"
Max Fragment Size: 524287 MB
Stage data: yes
Block Sharing: no
High Water Mark: 89
Low Water Mark: 70
Ok On Root: no

Label: indiasrc
Storage Unit Type: Disk
Storage Unit Subtype: Basic (1)
Host Connection: muttley
Concurrent Jobs: 1
On Demand Only: yes
Max MPX: 1
Path: "/indiasrc"
Max Fragment Size: 524288 MB
Stage data: no
Block Sharing: no
High Water Mark: 98
Low Water Mark: 80
Ok On Root: no

Label: nbu11
Storage Unit Type: Disk
Storage Unit Subtype: Basic (1)
Host Connection: muttley
Concurrent Jobs: 2
On Demand Only: yes
Max MPX: 1
Path: "/nbu11"
Max Fragment Size: 524287 MB
Stage data: yes
Block Sharing: no
High Water Mark: 89
Low Water Mark: 70
Ok On Root: no

Label: real11
Storage Unit Type: Disk
Storage Unit Subtype: Basic (1)
Host Connection: muttley
Concurrent Jobs: 4
On Demand Only: yes
Max MPX: 1
Path: "/nbudisk/real11"
Max Fragment Size: 524287 MB
Stage data: yes
Block Sharing: no
High Water Mark: 89
Low Water Mark: 70
Ok On Root: no
__GROUPS__
prod-full 1 nbu03 nbu11 nbu02 nbu01 nbu06 nbu04 nbu05
test-sg 1
prod-real 1 real3 real11 real1 real2 real4

[Marianne]

The storage unit output does not include staging details.

Have you added a disk staging schedule to each of the disk storage units?

Water marks can only be applied when a staging schedule is associated with the DSU and staging has been successful.

*** EDIT ***

I see some of your DSU's has 
Stage data: yes
But some has a 'no' here.

Please check schedules in the DSU config - how often does it run? 

Can you see successful staging jobs on a regular basis in Activity Monitor?

One more thing:

All the DSU's under /nbudisk/ should be different mount points/filesystems.

If they are directories, high and low water marks cannot be applied correctly by NBU.

[Krishna_Sriniva]

Hi ,

We have the staging schedule configured for all the Storage Units and also they are all individual mountpoints.

[RiaanBadenhorst]

Silly question but it the schedule open for it to stage to tape?

[Krishna_Sriniva]

Hi ,

we are currently using the ones with staging yes only others created earlier for testing.

Storage unit group is created using prioritized .

[Marianne]

Please show us details of you staging schedules?

bpschedule -L -label <schedule-name>

(Repeat command for each staging schedule)

Can you see successful staging jobs on a regular basis in Activity Monitor?

Also double-check your STUs - you have 2 that says:
Stage data: no
 

[Nicolai]

Is /nbudisk/ one disk or are real1-4 separate mount points ?

It not recommended to use one file system because there is no logic in the DSSU to detect other DSSU's filling the file system concurrent. That logic is in place for the AdvancedDisk type.

[Krishna_Sriniva]

Hi ,

All the Filesystems are zfs file systems.

pool1/nbu01            3.5T   2.3T   1.2T    67%    /nbu01
pool2/nbu02            5.4T   3.5T   1.9T    65%    /nbu02
pool3/nbu03            6.0T   5.6T   433G    93%    /nbu03
pool4/nbu04            3.5T   2.4T   1.0T    70%    /nbu04
pool4/nbu05            2.0T   1.7T   309G    85%    /nbu05
pool1/nbu06            1.5T   502G   1.0T    33%    /nbu06
pool1/real1            2.8T   1.2T   1.5T    44%    /nbudisk/real1
pool2/real2            3.2T   1.8T   1.4T    57%    /nbudisk/real2
pool3/real3            3.5T   2.5T   665G    80%    /nbudisk/real3
pool4/real4            3.0T   2.0T   982G    69%    /nbudisk/real4
 

[Krishna_Sriniva]

Hi Marianna,

We have stage data= no for only catalog disk and for another test basic disk ,

I can see the duplication jobs running fine . My concern is that storage unit group is not leveraging to another STU in the group when the first one reaches high water mark of 89%.

[RiaanBadenhorst · Accepted solution]

You should move the advanced disk storage units so you can use Media server load balancing in your STUG.

0. The Media server load balancing option indicates that NetBackup select a storage unit based on a capacity-managed approach. In this way, NetBackup avoids sending jobs to busy media servers.

   If a storage unit is unavailable, NetBackup examines the next storage unit until it finds one that is available.

   The selection is based on the following factors:

   • The rank of the media server.
     NetBackup considers the number of processes that are running on each CPU along with the memory thresholds on each server to determine the rank of a media server. If the free memory drops below a determined threshold, or if the number of running processes per CPU rises over a determined threshold, then the overall rank of the media server drops.

   • The number of jobs on the media server.
     NetBackup considers the number of scheduled jobs on each media server.

   • ■ Whether the media server has enough disk space to accommodate the estimated size of the image. (Physical and virtual tapes ignore this requirement.) NetBackup estimates the size of any of the new or any current jobs on each media server. It then determines whether the jobs fit on a given volume. NetBackup estimates the amount of space that the job may require, based on previous backup history. If no history is available, the high water mark for the storage unit serves as a guide.

     Media server load balancing cannot be selected for a storage unit group that includes a BasicDisk storage unit. Also, a BasicDisk storage unit cannot be included in an existing storage unit group with Media server load balancing enabled.

[Marianne]

I am looking at what the manual says about stug's:

■ Prioritized. Choose the first storage unit in the list that is not busy, down, or out of media.

■ Failover. Choose the first storage unit in the list that is not down or out of media.

■ Round Robin. Choose the least recently selected storage unit in the list.

■ Media server load balancing. Choose a storage unit based on a capacity-managed approach.

Symantec recommends the Media server load balancing criteria for disk staging storage units within a storage unit group.

See “Media server load balancing” on page 535.

[Krishna_Sriniva]

For basic disks , can you elaborate on what best setting can we use.

[Marianne]

Let us have another look at your opening post:

 all the storage units have high water mark level set to 89% still the storage units are getting 100% full 

I notice that you forgot to mention or select your NBU version.
The reason why version and patch level is important is because older versions of NBU had bugs as far as cleanup is concerned.

Ensure all of the following log folders exist on server muttley:
admin   (to log actual staging)
bptm    (read and write operations during backup and duplication)
bpdm   (disk cleanup based on successful duplication and HWM)

The above logs will help to troubleshoot staging and cleanup.

[Krishna_Sriniva]

Currently , we are using 7.1.0.3 version and all the mentioned logs are updating on server muttley.

[Krishna_Sriniva]

@ Riann,

Yes the staging schedule is open to stage the images to tape.

[Marianne]

For basic disks , can you elaborate on what best setting can we use.

As per the manual - Media server load balancing criteria for disk staging storage units within a storage unit group  is best.

One more thing - I remember a White Paper that explains why zfs is not a good choice for capacity managed disk storage. Something about the OS not detecting or reporting disk usage correctly. Will see if I can find it.

[Krishna_Sriniva]

Hi ,

I have referred to one of the technotes as below which mentions as media server load balancing is not applicable for basic disk storage units.

http://www.symantec.com/business/support/index?page=content&id=TECH87559

Please share me any doc regarding ZFS pools usage being not well for netbackup

[Yasuhisa_Ishika]

NetBackup 7.5 Administrator's Guide Volume I states as below:

Media server load balancing cannot be selected for a storage unit group that includes a BasicDisk storage unit.

[Krishna_Sriniva]

Yes Yasuhisa,

Need Assistance on the same as all our storage units are of basic disk . Which option wil be more appropriate as currently we are using prioritized which doesnt have much effect.