Forum Discussion

Bnj_V's avatar
Bnj_V
Level 3
9 years ago

Disk Volume is Down: Status 2074 - Can't be UP

Hello,

The error I've encountered is already common. I've read lot of forums and discussions about and I applied the solutions suggested.

But my drives are still down. May I ask for any suggestions for resolution please. (Note:Veritas NBU 6.5) 

Thanks in advance.

==============================================

Job Activity details:

Aug 11, 2015 2:59:55 PM - requesting resource zfs-p1nbu-incremental
Aug 11, 2015 2:59:55 PM - requesting resource PBCKSP01.NBU_CLIENT.MAXJOBS.pnet.ph.ds.amkor.com
Aug 11, 2015 2:59:55 PM - requesting resource PBCKSP01.NBU_POLICY.MAXJOBS.Windows_ATPNET
Aug 11, 2015 2:59:55 PM - Error nbjm (pid=15478) NBU status: 2074, EMM status: Disk volume is down
Disk volume is down  (2074)

===============================================

bperror:

1439276395 1 4 16 PBCKSP01 174136 174136 0 pnet.ph.ds.****.com nbpem backup of client pnet.ph.ds.****.com exited with status 2074 (Disk volume is down)
1439276506 1 1536 8 psk01 0 0 0 *NULL* nbemm Volume disk_windows-amds:Internal_16 is down (not ok on root)  
1439276506 1 1536 8 psk01 0 0 0 *NULL* nbemm Volume disk_windows-barcode:Internal_16 is down (not ok on root)  
1439276506 1 1536 8 psk01 0 0 0 *NULL* nbemm Volume disk_windows-cimapps07:Internal_16 is down (not ok on root)  
1439277889 1 1536 8 psk01 0 0 0 *NULL* nbemm Volume Lotus_Notes_Migration:Internal_16 is down (not ok on root)  
1439277889 1 1536 8 psk01 0 0 0 *NULL* nbemm Volume disk_windows-notes:Internal_16 is down (not ok on root)  
1439280113 1 1536 8 psk01 0 0 0 *NULL* nbemm Volume disk_Windows_Payroll_Full:Internal_16 is down (not ok on root)  
1439281496 1 1536 8 psk01 0 0 0 *NULL* nbemm Volume zfs-p1nbu-incremental:Internal_16 is down (not ok on root)

================================================

/usr/openv/netbackup/bin/admincmd/nbdevquery -listdv -stype BasicDisk -U

Disk Pool Name : BMR-DISK Disk Type : BasicDisk Disk Volume Name : Internal_16 Disk Media ID : @aaaab Total Capacity (GB) : 1000.00 Free Space (GB) : 498.15 Use% : 50 Status : UP Flag : ReadOnWrite Flag : AdminUp Flag : InternalUp Num Read Mounts : 0 Num Write Mounts : 1 Cur Read Streams : 0 Cur Write Streams : 0

Disk Pool Name : BMR-atpxisp01n1 Disk Type : BasicDisk Disk Volume Name : Internal_16 Disk Media ID : @aaaae Total Capacity (GB) : 1000.00 Free Space (GB) : 498.15 Use% : 50 Status : UP Flag : ReadOnWrite Flag : AdminUp Flag : InternalUp Num Read Mounts : 0 Num Write Mounts : 1 Cur Read Streams : 0 Cur Write Streams : 0

Disk Pool Name : bmr_atpxi03 Disk Type : BasicDisk Disk Volume Name : Internal_16 Disk Media ID : @aaaad Total Capacity (GB) : 1000.00 Free Space (GB) : 498.15 Use% : 50 Status : UP Flag : ReadOnWrite Flag : AdminUp Flag : InternalUp Num Read Mounts : 0 Num Write Mounts : 1 Cur Read Streams : 0 Cur Write Streams : 0

Disk Pool Name : BMR-atpxisp01n2 Disk Type : BasicDisk Disk Volume Name : Internal_16 Disk Media ID : @aaaaf Total Capacity (GB) : 1000.00 Free Space (GB) : 498.15 Use% : 50 Status : UP Flag : ReadOnWrite Flag : AdminUp Flag : InternalUp Num Read Mounts : 0 Num Write Mounts : 1 Cur Read Streams : 0 Cur Write Streams : 0

Disk Pool Name : disk_Windows_Payroll_Full Disk Type : BasicDisk Disk Volume Name : Internal_16 Disk Media ID : @aaaam Total Capacity (GB) : 2048.00 Free Space (GB) : 1739.63 Use% : 15 Status : DOWN Flag : ReadOnWrite Flag : AdminUp Flag : InternalDown Num Read Mounts : 0 Num Write Mounts : 1 Cur Read Streams : 0 Cur Write Streams : 0

Disk Pool Name : StripTest-Disk Disk Type : BasicDisk Disk Volume Name : Internal_16 Disk Media ID : @aaaah Total Capacity (GB) : 1000.00 Free Space (GB) : 498.15 Use% : 50 Status : UP Flag : ReadOnWrite Flag : AdminUp Flag : InternalUp Num Read Mounts : 0 Num Write Mounts : 1 Cur Read Streams : 0 Cur Write Streams : 0

Disk Pool Name : disk_windows-barcode Disk Type : BasicDisk Disk Volume Name : Internal_16 Disk Media ID : @aaaai Total Capacity (GB) : 2048.00 Free Space (GB) : 1739.63 Use% : 15 Status : DOWN Flag : ReadOnWrite Flag : AdminUp Flag : InternalDown Num Read Mounts : 0 Num Write Mounts : 1 Cur Read Streams : 0 Cur Write Streams : 0

Disk Pool Name : disk_windows-cimapps07 Disk Type : BasicDisk Disk Volume Name : Internal_16 Disk Media ID : @aaaaj Total Capacity (GB) : 2048.00 Free Space (GB) : 1739.63 Use% : 15 Status : DOWN Flag : ReadOnWrite Flag : AdminUp Flag : InternalDown Num Read Mounts : 0 Num Write Mounts : 1 Cur Read Streams : 0 Cur Write Streams : 0

Disk Pool Name : disk_windows-cimapps04 Disk Type : BasicDisk Disk Volume Name : Internal_16 Disk Media ID : @aaaak Total Capacity (GB) : 2048.00 Free Space (GB) : 1739.63 Use% : 15 Status : DOWN Flag : ReadOnWrite Flag : AdminUp Flag : InternalDown Num Read Mounts : 0 Num Write Mounts : 1 Cur Read Streams : 0 Cur Write Streams : 0

Disk Pool Name : disk_windows-amds Disk Type : BasicDisk Disk Volume Name : Internal_16 Disk Media ID : @aaaal Total Capacity (GB) : 2048.00 Free Space (GB) : 1739.63 Use% : 15 Status : DOWN Flag : ReadOnWrite Flag : AdminUp Flag : InternalDown Num Read Mounts : 0 Num Write Mounts : 1 Cur Read Streams : 0 Cur Write Streams : 0

Disk Pool Name : zfs-p1nbu-incremental Disk Type : BasicDisk Disk Volume Name : Internal_16 Disk Media ID : @aaaaq Total Capacity (GB) : 19.70 Free Space (GB) : 2.91 Use% : 85 Status : DOWN Flag : ReadOnWrite Flag : AdminUp Flag : InternalDown Num Read Mounts : 0 Num Write Mounts : 1 Cur Read Streams : 0 Cur Write Streams : 0

Disk Pool Name : Lotus_Notes_Migration Disk Type : BasicDisk Disk Volume Name : Internal_16 Disk Media ID : @aaaar Total Capacity (GB) : 19.70 Free Space (GB) : 2.84 Use% : 85 Status : DOWN Flag : ReadOnWrite Flag : AdminUp Flag : InternalDown Num Read Mounts : 0 Num Write Mounts : 1 Cur Read Streams : 0 Cur Write Streams : 0

Disk Pool Name : disk_windows-notes Disk Type : BasicDisk Disk Volume Name : Internal_16 Disk Media ID : @aaaas Total Capacity (GB) : 19.70 Free Space (GB) : 2.84 Use% : 85 Status : DOWN Flag : ReadOnWrite Flag : AdminUp Flag : InternalDown Num Read Mounts : 0 Num Write Mounts : 1 Cur Read Streams : 0 Cur Write Streams : 0

================================================

I tried runing this command to all the down disk pool name:

./nbdevconfig -changestate -stype BasicDisk -dp [disk-pol-name] -dv Internal_16 -state UP

successfully changed the state of disk volume. But when running nbdevquery again.. the status are still DOWN(Same as above)

  • PLease post df -h of the server, or show the file system layout where these volumes are lying.

    "1439281496 1 1536 8 psk01 0 0 0 *NULL* nbemm Volume zfs-p1nbu-incremental:Internal_16 is down (not ok on root)"

    Is the basic disk configured on the / file system?

  • Thanks Riaan and Marianne!

    /zfs-p1nbu/incremental/  and /p3_nbu_copy is the cause of my problem. no mount points found on df -h.

    Those are shared paths from our NetApp Filers.our previous storage admin did not informed us that those paths were removed/not used already. So I have to remove now the policies on those affected clients.

     

8 Replies

  • PLease post df -h of the server, or show the file system layout where these volumes are lying.

    "1439281496 1 1536 8 psk01 0 0 0 *NULL* nbemm Volume zfs-p1nbu-incremental:Internal_16 is down (not ok on root)"

    Is the basic disk configured on the / file system?

  • Sample drive:

    root@TPBACKUPSP01 # /usr/openv/netbackup/bin/admincmd/nbdevquery -listdv -stype BasicDisk -dp zfs-p1nbu-incremental -U
    Disk Pool Name      : zfs-p1nbu-incremental
    Disk Type           : BasicDisk
    Disk Volume Name    : Internal_16
    Disk Media ID       : @aaaaq
    Total Capacity (GB) : 19.70
    Free Space (GB)     : 2.91
    Use%                : 85
    Status              : DOWN
    Flag                : ReadOnWrite
    Flag                : AdminUp
    Flag                : InternalDown
    Num Read Mounts     : 0
    Num Write Mounts    : 1
    Cur Read Streams    : 0
    Cur Write Streams   : 0

  • Please show us output of 'df -h' and 'bpstulist -L'

    The path specified in the STU must be on a different filesystem (as per STU config)

  • Thanks you for the comment.

    Here's the output. 

    Here's the output:
     
    bpstulist -L
     
    Label:                atpsk01-hcart3-robot-tld-0-atpnas01-3020
    Storage Unit Type:    NDMP
    Host Connection:      atpsk01
    Number of Drives:     2
    On Demand Only:       no
    Max MPX/drive:        1
    Density:              hcart3 - 1/2 Inch Cartridge 3
    Robot Type/Number:    TLD / 0
    Max Fragment Size:    1048576 MB
    NDMP attach host:     atpnas01-3020
     
    Label:                atpsk01-hcart3-robot-tld-0-atpnas02-3020
    Storage Unit Type:    NDMP
    Host Connection:      atpsk01
    Number of Drives:     4
    On Demand Only:       no
    Max MPX/drive:        1
    Density:              hcart3 - 1/2 Inch Cartridge 3
    Robot Type/Number:    TLD / 0
    Max Fragment Size:    1048576 MB
    NDMP attach host:     atpnas02-3020
     
    Label:                atpsk01-hcart3-robot-tld-2
    Storage Unit Type:    Media Manager
    Host Connection:      atpsk01
    Number of Drives:     4
    On Demand Only:       no
    Max MPX/drive:        1
    Density:              hcart3 - 1/2 Inch Cartridge 3
    Robot Type/Number:    TLD / 2
    Max Fragment Size:    1048576 MB
     
    Label:                atpsk01-hcart3
    Storage Unit Type:    Media Manager
    Host Connection:      atpsk01
    Number of Drives:     4
    On Demand Only:       no
    Max MPX/drive:        1
    Density:              hcart3 - 1/2 Inch Cartridge 3
    Robot Type:           (not robotic)
    Max Fragment Size:    1048576 MB
     
    Label:                BMR-DISK
    Storage Unit Type:    Disk
    Storage Unit Subtype: Basic
    Host Connection:      atpsk01
    Concurrent Jobs:      1
    On Demand Only:       yes
    Max MPX:              1
    Path:                 "/bmr"
    Max Fragment Size:    524288 MB
    Stage data:           no
    Block Sharing:        no
    High Water Mark:      98
    Low Water Mark:       80
    Ok On Root:           no
     
    Label:                BMR-atpxisp01n1
    Storage Unit Type:    Disk
    Storage Unit Subtype: Basic
    Host Connection:      atpsk01
    Concurrent Jobs:      1
    On Demand Only:       yes
    Max MPX:              1
    Path:                 "/bmr/atpxisp01n1"
    Max Fragment Size:    524288 MB
    Stage data:           no
    Block Sharing:        no
    High Water Mark:      98
    Low Water Mark:       80
    Ok On Root:           no
     
    Label:                bmr_atpxi03
    Storage Unit Type:    Disk
    Storage Unit Subtype: Basic
    Host Connection:      atpsk01
    Concurrent Jobs:      1
    On Demand Only:       yes
    Max MPX:              1
    Path:                 "/bmr/atpxi03"
    Max Fragment Size:    524288 MB
    Stage data:           no
    Block Sharing:        no
    High Water Mark:      98
    Low Water Mark:       80
    Ok On Root:           no
     
    Label:                BMR-atpxisp01n2
    Storage Unit Type:    Disk
    Storage Unit Subtype: Basic
    Host Connection:      atpsk01
    Concurrent Jobs:      1
    On Demand Only:       yes
    Max MPX:              1
    Path:                 "/bmr/atpxisp01n2"
    Max Fragment Size:    524288 MB
    Stage data:           no
    Block Sharing:        no
    High Water Mark:      98
    Low Water Mark:       80
    Ok On Root:           no
     
    Label:                disk_Windows_Payroll_Full
    Storage Unit Type:    Disk
    Storage Unit Subtype: Basic
    Host Connection:      atpsk01
    Concurrent Jobs:      1
    On Demand Only:       yes
    Max MPX:              1
    Path:                 "/p3_nbu_copy/Windows_Payroll_Full"
    Max Fragment Size:    524288 MB
    Stage data:           no
    Block Sharing:        no
    High Water Mark:      98
    Low Water Mark:       80
    Ok On Root:           no
     
    Label:                StripTest-Disk
    Storage Unit Type:    Disk
    Storage Unit Subtype: Basic
    Host Connection:      atpsk01
    Concurrent Jobs:      1
    On Demand Only:       yes
    Max MPX:              1
    Path:                 "/bmr/StripTest"
    Max Fragment Size:    524288 MB
    Stage data:           no
    Block Sharing:        no
    High Water Mark:      98
    Low Water Mark:       80
    Ok On Root:           no
     
    Label:                disk_windows-barcode
    Storage Unit Type:    Disk
    Storage Unit Subtype: Basic
    Host Connection:      atpsk01
    Concurrent Jobs:      1
    On Demand Only:       yes
    Max MPX:              1
    Path:                 "/p3_nbu_copy/windows-barcode"
    Max Fragment Size:    524288 MB
    Stage data:           no
    Block Sharing:        no
    High Water Mark:      98
    Low Water Mark:       80
    Ok On Root:           no
     
    Label:                disk_windows-cimapps07
    Storage Unit Type:    Disk
    Storage Unit Subtype: Basic
    Host Connection:      atpsk01
    Concurrent Jobs:      1
    On Demand Only:       yes
    Max MPX:              1
    Path:                 "/p3_nbu_copy/windows-cimapps07"
    Max Fragment Size:    524288 MB
    Stage data:           no
    Block Sharing:        no
    High Water Mark:      98
    Low Water Mark:       80
    Ok On Root:           no
     
    Label:                disk_windows-cimapps04
    Storage Unit Type:    Disk
    Storage Unit Subtype: Basic
    Host Connection:      atpsk01
    Concurrent Jobs:      1
    On Demand Only:       yes
    Max MPX:              1
    Path:                 "/p3_nbu_copy/windows-cimapps04"
    Max Fragment Size:    524288 MB
    Stage data:           no
    Block Sharing:        no
    High Water Mark:      98
    Low Water Mark:       80
    Ok On Root:           no
     
    Label:                disk_windows-amds
    Storage Unit Type:    Disk
    Storage Unit Subtype: Basic
    Host Connection:      atpsk01
    Concurrent Jobs:      1
    On Demand Only:       yes
    Max MPX:              1
    Path:                 "/p3_nbu_copy/windows-amds"
    Max Fragment Size:    524288 MB
    Stage data:           no
    Block Sharing:        no
    High Water Mark:      98
    Low Water Mark:       80
    Ok On Root:           no
     
    Label:                zfs-p1nbu-incremental
    Storage Unit Type:    Disk
    Storage Unit Subtype: Basic
    Host Connection:      atpsk01
    Concurrent Jobs:      4
    On Demand Only:       yes
    Max MPX:              1
    Path:                 "/zfs-p1nbu/incremental/"
    Max Fragment Size:    524288 MB
    Stage data:           no
    Block Sharing:        no
    High Water Mark:      98
    Low Water Mark:       80
    Ok On Root:           no
     
    Label:                Lotus_Notes_Migration
    Storage Unit Type:    Disk
    Storage Unit Subtype: Basic
    Host Connection:      atpsk01
    Concurrent Jobs:      4
    On Demand Only:       yes
    Max MPX:              1
    Path:                 "/zfs-p1nbu/incremental/Lotus_Notes"
    Max Fragment Size:    524288 MB
    Stage data:           no
    Block Sharing:        no
    High Water Mark:      98
    Low Water Mark:       80
    Ok On Root:           no
     
    Label:                disk_windows-notes
    Storage Unit Type:    Disk
    Storage Unit Subtype: Basic
    Host Connection:      atpsk01
    Concurrent Jobs:      4
    On Demand Only:       yes
    Max MPX:              1
    Path:                 "/zfs-p1nbu/incremental"
    Max Fragment Size:    524288 MB
    Stage data:           no
    Block Sharing:        no
    High Water Mark:      98
    Low Water Mark:       80
    Ok On Root:           no
    root@ATPBACKUPSP01 # 
    =====================================
    root@ATPBACKUPSP01 # df -h
    Filesystem             size   used  avail capacity  Mounted on
    /dev/dsk/c0t0d0s0       20G   9.2G    10G    48%    /
    /devices                 0K     0K     0K     0%    /devices
    ctfs                     0K     0K     0K     0%    /system/contract
    proc                     0K     0K     0K     0%    /proc
    mnttab                   0K     0K     0K     0%    /etc/mnttab
    swap                    13G   1.5M    13G     1%    /etc/svc/volatile
    objfs                    0K     0K     0K     0%    /system/object
    /platform/sun4u-us3/lib/libc_psr/libc_psr_hwcap1.so.1
                            20G   9.2G    10G    48%    /platform/sun4u-us3/lib/libc_psr.so.1
    /platform/sun4u-us3/lib/sparcv9/libc_psr/libc_psr_hwcap1.so.1
                            20G   9.2G    10G    48%    /platform/sun4u-us3/lib/sparcv9/libc_psr.so.1
    fd                       0K     0K     0K     0%    /dev/fd
    /dev/dsk/c0t0d0s3       15G   720M    14G     5%    /var
    swap                    13G   176K    13G     1%    /tmp
    swap                    13G    64K    13G     1%    /var/run
    /dev/dsk/c0t0d0s4       15G    12G   2.3G    85%    /opt
    /dev/dsk/c0t0d0s5      7.9G   4.4G   3.4G    57%    /export/home
    atpnas07-3020P:/vol/fv_atp3backupsp01
                           2.8T   1.6T   1.2T    58%    /p1_nbu_copy
    192.168.1.1:/vol/fv_bmr
                          1000G   502G   498G    51%    /bmr
    =================================================
     cat /etc/vfstab
    #device         device          mount           FS      fsck    mount   mount
    #to mount       to fsck         point           type    pass    at boot options
    #
    fd      -       /dev/fd fd      -       no      -
    /proc   -       /proc   proc    -       no      -
    /dev/dsk/c0t0d0s1       -       -       swap    -       no      -
    /dev/dsk/c0t0d0s0       /dev/rdsk/c0t0d0s0      /       ufs     1       no      -
    /dev/dsk/c0t0d0s3       /dev/rdsk/c0t0d0s3      /var    ufs     1       no      -
    /dev/dsk/c0t0d0s5       /dev/rdsk/c0t0d0s5      /export/home    ufs     2       yes     -
    /dev/dsk/c0t0d0s4       /dev/rdsk/c0t0d0s4      /opt    ufs     2       yes     -
    192.168.1.1:/vol/fv_bmr -       /bmr    nfs     -       yes     -
    #atpnas03-r200:/vol/fv_p1nbu/atpbackupsp01    -       /p3_nbu_copy    nfs     -       yes     -
    atpnas07-3020P:/vol/fv_atp3backupsp01   -       /p1_nbu_copy    nfs     -       yes     -
    /devices        -       /devices        devfs   -       no      -
    ctfs    -       /system/contract        ctfs    -       no      -
    objfs   -       /system/object  objfs   -       no      -
    swap    -       /tmp    tmpfs   -       yes     -      
    192.168.1.1:/vol/vol1 -         /root/testmount nfs     -       yes      - 
    atpnas6040-n2:/vol/fv_restore   -       /restore        nfs     -       yes     -
     
  • Sorry, not to familiar with zfs. Where is /zfs-p1nbu/incremental/ located, i don't see in the df -h output

  • Thanks Riaan and Marianne!

    /zfs-p1nbu/incremental/  and /p3_nbu_copy is the cause of my problem. no mount points found on df -h.

    Those are shared paths from our NetApp Filers.our previous storage admin did not informed us that those paths were removed/not used already. So I have to remove now the policies on those affected clients.