Disk Volume is Down: Status 2074 - Can't be UP

[Bnj_V]

Hello,

The error I've encountered is already common. I've read lot of forums and discussions about and I applied the solutions suggested.

But my drives are still down. May I ask for any suggestions for resolution please. (Note:Veritas NBU 6.5) 

Thanks in advance.

==============================================

Job Activity details:

Aug 11, 2015 2:59:55 PM - requesting resource zfs-p1nbu-incremental
Aug 11, 2015 2:59:55 PM - requesting resource PBCKSP01.NBU_CLIENT.MAXJOBS.pnet.ph.ds.amkor.com
Aug 11, 2015 2:59:55 PM - requesting resource PBCKSP01.NBU_POLICY.MAXJOBS.Windows_ATPNET
Aug 11, 2015 2:59:55 PM - Error nbjm (pid=15478) NBU status: 2074, EMM status: Disk volume is down
Disk volume is down  (2074)

===============================================

bperror:

1439276395 1 4 16 PBCKSP01 174136 174136 0 pnet.ph.ds.****.com nbpem backup of client pnet.ph.ds.****.com exited with status 2074 (Disk volume is down)
1439276506 1 1536 8 psk01 0 0 0 *NULL* nbemm Volume disk_windows-amds:Internal_16 is down (not ok on root)  
1439276506 1 1536 8 psk01 0 0 0 *NULL* nbemm Volume disk_windows-barcode:Internal_16 is down (not ok on root)  
1439276506 1 1536 8 psk01 0 0 0 *NULL* nbemm Volume disk_windows-cimapps07:Internal_16 is down (not ok on root)  
1439277889 1 1536 8 psk01 0 0 0 *NULL* nbemm Volume Lotus_Notes_Migration:Internal_16 is down (not ok on root)  
1439277889 1 1536 8 psk01 0 0 0 *NULL* nbemm Volume disk_windows-notes:Internal_16 is down (not ok on root)  
1439280113 1 1536 8 psk01 0 0 0 *NULL* nbemm Volume disk_Windows_Payroll_Full:Internal_16 is down (not ok on root)  
1439281496 1 1536 8 psk01 0 0 0 *NULL* nbemm Volume zfs-p1nbu-incremental:Internal_16 is down (not ok on root)

================================================

/usr/openv/netbackup/bin/admincmd/nbdevquery -listdv -stype BasicDisk -U

Disk Pool Name : BMR-DISK Disk Type : BasicDisk Disk Volume Name : Internal_16 Disk Media ID : @aaaab Total Capacity (GB) : 1000.00 Free Space (GB) : 498.15 Use% : 50 Status : UP Flag : ReadOnWrite Flag : AdminUp Flag : InternalUp Num Read Mounts : 0 Num Write Mounts : 1 Cur Read Streams : 0 Cur Write Streams : 0

Disk Pool Name : BMR-atpxisp01n1 Disk Type : BasicDisk Disk Volume Name : Internal_16 Disk Media ID : @aaaae Total Capacity (GB) : 1000.00 Free Space (GB) : 498.15 Use% : 50 Status : UP Flag : ReadOnWrite Flag : AdminUp Flag : InternalUp Num Read Mounts : 0 Num Write Mounts : 1 Cur Read Streams : 0 Cur Write Streams : 0

Disk Pool Name : bmr_atpxi03 Disk Type : BasicDisk Disk Volume Name : Internal_16 Disk Media ID : @aaaad Total Capacity (GB) : 1000.00 Free Space (GB) : 498.15 Use% : 50 Status : UP Flag : ReadOnWrite Flag : AdminUp Flag : InternalUp Num Read Mounts : 0 Num Write Mounts : 1 Cur Read Streams : 0 Cur Write Streams : 0

Disk Pool Name : BMR-atpxisp01n2 Disk Type : BasicDisk Disk Volume Name : Internal_16 Disk Media ID : @aaaaf Total Capacity (GB) : 1000.00 Free Space (GB) : 498.15 Use% : 50 Status : UP Flag : ReadOnWrite Flag : AdminUp Flag : InternalUp Num Read Mounts : 0 Num Write Mounts : 1 Cur Read Streams : 0 Cur Write Streams : 0

Disk Pool Name : disk_Windows_Payroll_Full Disk Type : BasicDisk Disk Volume Name : Internal_16 Disk Media ID : @aaaam Total Capacity (GB) : 2048.00 Free Space (GB) : 1739.63 Use% : 15 Status : DOWN Flag : ReadOnWrite Flag : AdminUp Flag : InternalDown Num Read Mounts : 0 Num Write Mounts : 1 Cur Read Streams : 0 Cur Write Streams : 0

Disk Pool Name : StripTest-Disk Disk Type : BasicDisk Disk Volume Name : Internal_16 Disk Media ID : @aaaah Total Capacity (GB) : 1000.00 Free Space (GB) : 498.15 Use% : 50 Status : UP Flag : ReadOnWrite Flag : AdminUp Flag : InternalUp Num Read Mounts : 0 Num Write Mounts : 1 Cur Read Streams : 0 Cur Write Streams : 0

Disk Pool Name : disk_windows-barcode Disk Type : BasicDisk Disk Volume Name : Internal_16 Disk Media ID : @aaaai Total Capacity (GB) : 2048.00 Free Space (GB) : 1739.63 Use% : 15 Status : DOWN Flag : ReadOnWrite Flag : AdminUp Flag : InternalDown Num Read Mounts : 0 Num Write Mounts : 1 Cur Read Streams : 0 Cur Write Streams : 0

Disk Pool Name : disk_windows-cimapps07 Disk Type : BasicDisk Disk Volume Name : Internal_16 Disk Media ID : @aaaaj Total Capacity (GB) : 2048.00 Free Space (GB) : 1739.63 Use% : 15 Status : DOWN Flag : ReadOnWrite Flag : AdminUp Flag : InternalDown Num Read Mounts : 0 Num Write Mounts : 1 Cur Read Streams : 0 Cur Write Streams : 0

Disk Pool Name : disk_windows-cimapps04 Disk Type : BasicDisk Disk Volume Name : Internal_16 Disk Media ID : @aaaak Total Capacity (GB) : 2048.00 Free Space (GB) : 1739.63 Use% : 15 Status : DOWN Flag : ReadOnWrite Flag : AdminUp Flag : InternalDown Num Read Mounts : 0 Num Write Mounts : 1 Cur Read Streams : 0 Cur Write Streams : 0

Disk Pool Name : disk_windows-amds Disk Type : BasicDisk Disk Volume Name : Internal_16 Disk Media ID : @aaaal Total Capacity (GB) : 2048.00 Free Space (GB) : 1739.63 Use% : 15 Status : DOWN Flag : ReadOnWrite Flag : AdminUp Flag : InternalDown Num Read Mounts : 0 Num Write Mounts : 1 Cur Read Streams : 0 Cur Write Streams : 0

Disk Pool Name : zfs-p1nbu-incremental Disk Type : BasicDisk Disk Volume Name : Internal_16 Disk Media ID : @aaaaq Total Capacity (GB) : 19.70 Free Space (GB) : 2.91 Use% : 85 Status : DOWN Flag : ReadOnWrite Flag : AdminUp Flag : InternalDown Num Read Mounts : 0 Num Write Mounts : 1 Cur Read Streams : 0 Cur Write Streams : 0

Disk Pool Name : Lotus_Notes_Migration Disk Type : BasicDisk Disk Volume Name : Internal_16 Disk Media ID : @aaaar Total Capacity (GB) : 19.70 Free Space (GB) : 2.84 Use% : 85 Status : DOWN Flag : ReadOnWrite Flag : AdminUp Flag : InternalDown Num Read Mounts : 0 Num Write Mounts : 1 Cur Read Streams : 0 Cur Write Streams : 0

Disk Pool Name : disk_windows-notes Disk Type : BasicDisk Disk Volume Name : Internal_16 Disk Media ID : @aaaas Total Capacity (GB) : 19.70 Free Space (GB) : 2.84 Use% : 85 Status : DOWN Flag : ReadOnWrite Flag : AdminUp Flag : InternalDown Num Read Mounts : 0 Num Write Mounts : 1 Cur Read Streams : 0 Cur Write Streams : 0

================================================

I tried runing this command to all the down disk pool name:

./nbdevconfig -changestate -stype BasicDisk -dp [disk-pol-name] -dv Internal_16 -state UP

successfully changed the state of disk volume. But when running nbdevquery again.. the status are still DOWN(Same as above)

[Bnj_V]

Sample drive:

root@TPBACKUPSP01 # /usr/openv/netbackup/bin/admincmd/nbdevquery -listdv -stype BasicDisk -dp zfs-p1nbu-incremental -U
Disk Pool Name      : zfs-p1nbu-incremental
Disk Type           : BasicDisk
Disk Volume Name    : Internal_16
Disk Media ID       : @aaaaq
Total Capacity (GB) : 19.70
Free Space (GB)     : 2.91
Use%                : 85
Status              : DOWN
Flag                : ReadOnWrite
Flag                : AdminUp
Flag                : InternalDown
Num Read Mounts     : 0
Num Write Mounts    : 1
Cur Read Streams    : 0
Cur Write Streams   : 0

[RiaanBadenhorst · Accepted solution]

PLease post df -h of the server, or show the file system layout where these volumes are lying.

"1439281496 1 1536 8 psk01 0 0 0 *NULL* nbemm Volume zfs-p1nbu-incremental:Internal_16 is down (not ok on root)"

Is the basic disk configured on the / file system?

[Marianne]

Please show us output of 'df -h' and 'bpstulist -L'

The path specified in the STU must be on a different filesystem (as per STU config)

[Bnj_V]

Thanks you for the comment.

Here's the output. 

Here's the output:

bpstulist -L

Label:                atpsk01-hcart3-robot-tld-0-atpnas01-3020

Storage Unit Type:    NDMP

Host Connection:      atpsk01

Number of Drives:     2

On Demand Only:       no

Max MPX/drive:        1

Density:              hcart3 - 1/2 Inch Cartridge 3

Robot Type/Number:    TLD / 0

Max Fragment Size:    1048576 MB

NDMP attach host:     atpnas01-3020

Label:                atpsk01-hcart3-robot-tld-0-atpnas02-3020

Storage Unit Type:    NDMP

Host Connection:      atpsk01

Number of Drives:     4

On Demand Only:       no

Max MPX/drive:        1

Density:              hcart3 - 1/2 Inch Cartridge 3

Robot Type/Number:    TLD / 0

Max Fragment Size:    1048576 MB

NDMP attach host:     atpnas02-3020

Label:                atpsk01-hcart3-robot-tld-2

Storage Unit Type:    Media Manager

Host Connection:      atpsk01

Number of Drives:     4

On Demand Only:       no

Max MPX/drive:        1

Density:              hcart3 - 1/2 Inch Cartridge 3

Robot Type/Number:    TLD / 2

Max Fragment Size:    1048576 MB

Label:                atpsk01-hcart3

Storage Unit Type:    Media Manager

Host Connection:      atpsk01

Number of Drives:     4

On Demand Only:       no

Max MPX/drive:        1

Density:              hcart3 - 1/2 Inch Cartridge 3

Robot Type:           (not robotic)

Max Fragment Size:    1048576 MB

Label:                BMR-DISK

Storage Unit Type:    Disk

Storage Unit Subtype: Basic

Host Connection:      atpsk01

Concurrent Jobs:      1

On Demand Only:       yes

Max MPX:              1

Path:                 "/bmr"

Max Fragment Size:    524288 MB

Stage data:           no

Block Sharing:        no

High Water Mark:      98

Low Water Mark:       80

Ok On Root:           no

Label:                BMR-atpxisp01n1

Storage Unit Type:    Disk

Storage Unit Subtype: Basic

Host Connection:      atpsk01

Concurrent Jobs:      1

On Demand Only:       yes

Max MPX:              1

Path:                 "/bmr/atpxisp01n1"

Max Fragment Size:    524288 MB

Stage data:           no

Block Sharing:        no

High Water Mark:      98

Low Water Mark:       80

Ok On Root:           no

Label:                bmr_atpxi03

Storage Unit Type:    Disk

Storage Unit Subtype: Basic

Host Connection:      atpsk01

Concurrent Jobs:      1

On Demand Only:       yes

Max MPX:              1

Path:                 "/bmr/atpxi03"

Max Fragment Size:    524288 MB

Stage data:           no

Block Sharing:        no

High Water Mark:      98

Low Water Mark:       80

Ok On Root:           no

Label:                BMR-atpxisp01n2

Storage Unit Type:    Disk

Storage Unit Subtype: Basic

Host Connection:      atpsk01

Concurrent Jobs:      1

On Demand Only:       yes

Max MPX:              1

Path:                 "/bmr/atpxisp01n2"

Max Fragment Size:    524288 MB

Stage data:           no

Block Sharing:        no

High Water Mark:      98

Low Water Mark:       80

Ok On Root:           no

Label:                disk_Windows_Payroll_Full

Storage Unit Type:    Disk

Storage Unit Subtype: Basic

Host Connection:      atpsk01

Concurrent Jobs:      1

On Demand Only:       yes

Max MPX:              1

Path:                 "/p3_nbu_copy/Windows_Payroll_Full"

Max Fragment Size:    524288 MB

Stage data:           no

Block Sharing:        no

High Water Mark:      98

Low Water Mark:       80

Ok On Root:           no

Label:                StripTest-Disk

Storage Unit Type:    Disk

Storage Unit Subtype: Basic

Host Connection:      atpsk01

Concurrent Jobs:      1

On Demand Only:       yes

Max MPX:              1

Path:                 "/bmr/StripTest"

Max Fragment Size:    524288 MB

Stage data:           no

Block Sharing:        no

High Water Mark:      98

Low Water Mark:       80

Ok On Root:           no

Label:                disk_windows-barcode

Storage Unit Type:    Disk

Storage Unit Subtype: Basic

Host Connection:      atpsk01

Concurrent Jobs:      1

On Demand Only:       yes

Max MPX:              1

Path:                 "/p3_nbu_copy/windows-barcode"

Max Fragment Size:    524288 MB

Stage data:           no

Block Sharing:        no

High Water Mark:      98

Low Water Mark:       80

Ok On Root:           no

Label:                disk_windows-cimapps07

Storage Unit Type:    Disk

Storage Unit Subtype: Basic

Host Connection:      atpsk01

Concurrent Jobs:      1

On Demand Only:       yes

Max MPX:              1

Path:                 "/p3_nbu_copy/windows-cimapps07"

Max Fragment Size:    524288 MB

Stage data:           no

Block Sharing:        no

High Water Mark:      98

Low Water Mark:       80

Ok On Root:           no

Label:                disk_windows-cimapps04

Storage Unit Type:    Disk

Storage Unit Subtype: Basic

Host Connection:      atpsk01

Concurrent Jobs:      1

On Demand Only:       yes

Max MPX:              1

Path:                 "/p3_nbu_copy/windows-cimapps04"

Max Fragment Size:    524288 MB

Stage data:           no

Block Sharing:        no

High Water Mark:      98

Low Water Mark:       80

Ok On Root:           no

Label:                disk_windows-amds

Storage Unit Type:    Disk

Storage Unit Subtype: Basic

Host Connection:      atpsk01

Concurrent Jobs:      1

On Demand Only:       yes

Max MPX:              1

Path:                 "/p3_nbu_copy/windows-amds"

Max Fragment Size:    524288 MB

Stage data:           no

Block Sharing:        no

High Water Mark:      98

Low Water Mark:       80

Ok On Root:           no

Label:                zfs-p1nbu-incremental

Storage Unit Type:    Disk

Storage Unit Subtype: Basic

Host Connection:      atpsk01

Concurrent Jobs:      4

On Demand Only:       yes

Max MPX:              1

Path:                 "/zfs-p1nbu/incremental/"

Max Fragment Size:    524288 MB

Stage data:           no

Block Sharing:        no

High Water Mark:      98

Low Water Mark:       80

Ok On Root:           no

Label:                Lotus_Notes_Migration

Storage Unit Type:    Disk

Storage Unit Subtype: Basic

Host Connection:      atpsk01

Concurrent Jobs:      4

On Demand Only:       yes

Max MPX:              1

Path:                 "/zfs-p1nbu/incremental/Lotus_Notes"

Max Fragment Size:    524288 MB

Stage data:           no

Block Sharing:        no

High Water Mark:      98

Low Water Mark:       80

Ok On Root:           no

Label:                disk_windows-notes

Storage Unit Type:    Disk

Storage Unit Subtype: Basic

Host Connection:      atpsk01

Concurrent Jobs:      4

On Demand Only:       yes

Max MPX:              1

Path:                 "/zfs-p1nbu/incremental"

Max Fragment Size:    524288 MB

Stage data:           no

Block Sharing:        no

High Water Mark:      98

Low Water Mark:       80

Ok On Root:           no

root@ATPBACKUPSP01 # 

=====================================

root@ATPBACKUPSP01 # df -h

Filesystem             size   used  avail capacity  Mounted on

/dev/dsk/c0t0d0s0       20G   9.2G    10G    48%    /

/devices                 0K     0K     0K     0%    /devices

ctfs                     0K     0K     0K     0%    /system/contract

proc                     0K     0K     0K     0%    /proc

mnttab                   0K     0K     0K     0%    /etc/mnttab

swap                    13G   1.5M    13G     1%    /etc/svc/volatile

objfs                    0K     0K     0K     0%    /system/object

/platform/sun4u-us3/lib/libc_psr/libc_psr_hwcap1.so.1

                        20G   9.2G    10G    48%    /platform/sun4u-us3/lib/libc_psr.so.1

/platform/sun4u-us3/lib/sparcv9/libc_psr/libc_psr_hwcap1.so.1

                        20G   9.2G    10G    48%    /platform/sun4u-us3/lib/sparcv9/libc_psr.so.1

fd                       0K     0K     0K     0%    /dev/fd

/dev/dsk/c0t0d0s3       15G   720M    14G     5%    /var

swap                    13G   176K    13G     1%    /tmp

swap                    13G    64K    13G     1%    /var/run

/dev/dsk/c0t0d0s4       15G    12G   2.3G    85%    /opt

/dev/dsk/c0t0d0s5      7.9G   4.4G   3.4G    57%    /export/home

atpnas07-3020P:/vol/fv_atp3backupsp01

                       2.8T   1.6T   1.2T    58%    /p1_nbu_copy

192.168.1.1:/vol/fv_bmr

                      1000G   502G   498G    51%    /bmr

=================================================

 cat /etc/vfstab

#device         device          mount           FS      fsck    mount   mount

#to mount       to fsck         point           type    pass    at boot options

#

fd      -       /dev/fd fd      -       no      -

/proc   -       /proc   proc    -       no      -

/dev/dsk/c0t0d0s1       -       -       swap    -       no      -

/dev/dsk/c0t0d0s0       /dev/rdsk/c0t0d0s0      /       ufs     1       no      -

/dev/dsk/c0t0d0s3       /dev/rdsk/c0t0d0s3      /var    ufs     1       no      -

/dev/dsk/c0t0d0s5       /dev/rdsk/c0t0d0s5      /export/home    ufs     2       yes     -

/dev/dsk/c0t0d0s4       /dev/rdsk/c0t0d0s4      /opt    ufs     2       yes     -

192.168.1.1:/vol/fv_bmr -       /bmr    nfs     -       yes     -

#atpnas03-r200:/vol/fv_p1nbu/atpbackupsp01    -       /p3_nbu_copy    nfs     -       yes     -

atpnas07-3020P:/vol/fv_atp3backupsp01   -       /p1_nbu_copy    nfs     -       yes     -

/devices        -       /devices        devfs   -       no      -

ctfs    -       /system/contract        ctfs    -       no      -

objfs   -       /system/object  objfs   -       no      -

swap    -       /tmp    tmpfs   -       yes     -      

192.168.1.1:/vol/vol1 -         /root/testmount nfs     -       yes      - 

atpnas6040-n2:/vol/fv_restore   -       /restore        nfs     -       yes     -

[RiaanBadenhorst]

Sorry, not to familiar with zfs. Where is /zfs-p1nbu/incremental/ located, i don't see in the df -h output

[Marianne]

There is also no mount point for /p3_nbu_copy.

[Bnj_V · Accepted solution]

Thanks Riaan and Marianne!

/zfs-p1nbu/incremental/  and /p3_nbu_copy is the cause of my problem. no mount points found on df -h.

Those are shared paths from our NetApp Filers.our previous storage admin did not informed us that those paths were removed/not used already. So I have to remove now the policies on those affected clients.

[RiaanBadenhorst]

Ok, update the policies to use a different stu.