Disk volume is down even when mounted

[Sandeepkumar]

Hi 

When I trigger a backup it is failing immediately with 2074 disk volume is down. I have verified that the logical volume is mounted.It is a basic disk.

When I run the below command it is shwoing me the below error for some of the disks.

nbdevquery -listdv -stype BasicDisk -U

Disk Pool Name : msg132c9full
Disk Type : BasicDisk
Disk Volume Name : Internal_16
Disk Media ID : @aaaiQ
Total Capacity (GB) : 6101.73
Free Space (GB) : 76.69
Use% : 98
Status : DOWN
Flag : ReadOnWrite
Flag : AdminUp
Flag : InternalDown
Num Read Mounts : 0
Num Write Mounts : 1
Cur Read Streams : 0
Cur Write Streams : 0
Num Repl Sources : 0
Num Repl Targets : 0

What does this mean and how to clear the flag and up the disk

[RiaanBadenhorst]

Probably a comms issue (if the diskpool is not on the master)

Try to run

nbdevconfig -changestate -stype BasicDisk -dp msg132c9full -state RESET

[Sandeepkumar]

It did not work out.

Diskpool is on media server.

What else can i check only few are in that state

[Marianne]

Verify comms with media server:

nbemmcmd -listhosts -verbose 
(Check output related to this media server)

bptestbpcd -host <media-server-name> -debug -verbose

[Sandeepkumar]

I have pasted the outputs.

Machine name is nbumedia.nic.in.

[root@nbumaster ~]# bptestbpcd -host nbumedia.nic.in -debug -verbose
12:06:02.135 [17333] <2> bptestbpcd: VERBOSE = 0
12:06:02.137 [17333] <2> vnet_pbxConnect: pbxConnectEx Succeeded
12:06:02.137 [17333] <2> logconnections: BPCD CONNECT FROM 10.103.2.61.50644 TO 10.103.2.62.1556 fd = 3
12:06:02.138 [17333] <2> vnet_pbxConnect: pbxConnectEx Succeeded
12:06:02.139 [17333] <8> do_pbx_service: [vnet_connect.c:2152] via PBX VNETD CONNECT FROM 10.103.2.61.52446 TO 10.103.2.62.1556 fd = 4
12:06:02.139 [17333] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:443] VN_REQUEST_CONNECT_FORWARD_SOCKET 10 0xa
12:06:02.179 [17333] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:460] ipc_string /usr/openv/var/tmp/vnet-22578497508566997469000000037-dELBFA
12:06:02.261 [17333] <2> bpcr_get_version_rqst: bpcd version: 07600000
1 1 1
10.103.2.61:50644 -> 10.103.2.62:1556
10.103.2.61:52446 -> 10.103.2.62:1556
12:06:02.281 [17333] <2> bpcr_get_peername_rqst: Server peername length = 16
12:06:02.281 [17333] <2> bpcr_get_hostname_rqst: Server hostname length = 15
12:06:02.282 [17333] <2> bpcr_get_clientname_rqst: Server clientname length = 15
12:06:02.282 [17333] <2> bpcr_get_version_rqst: bpcd version: 07600000
12:06:02.282 [17333] <2> bpcr_get_platform_rqst: Server platform length = 17
12:06:02.282 [17333] <2> bpcr_get_version_rqst: bpcd version: 07600000
12:06:02.285 [17333] <2> bpcr_patch_version_rqst: theRest == > <
12:06:02.286 [17333] <2> bpcr_get_version_rqst: bpcd version: 07600000
12:06:02.326 [17333] <2> bpcr_patch_version_rqst: theRest == > <
12:06:02.327 [17333] <2> bpcr_get_version_rqst: bpcd version: 07600000
PEER_NAME = nbumaster.nic.in
HOST_NAME = nbumedia.nic.in
CLIENT_NAME = nbumedia.nic.in
VERSION = 0x07600000
PLATFORM = linuxR_x86_2.6.18
PATCH_VERSION = 7.6.0.1
SERVER_PATCH_VERSION = 7.6.0.1
MASTER_SERVER = nbumaster
EMM_SERVER = nbumaster.nic.in
NB_MACHINE_TYPE = MEDIA_SERVER
<2>bptestbpcd: EXIT status = 0
12:06:02.342 [17333] <2> bptestbpcd: EXIT status = 0

---------------------------------------------------------------------------------------

[root@nbumaster ~]# nbemmcmd -listhosts -verbose
NBEMMCMD, Version: 7.6.0.1
The following hosts were found:
nbumaster
MachineName = "nbumaster"
FQName = "nbumaster.nic.in"
MachineDescription = ""
MachineNbuType = server (6)
nbumaster
ClusterName = ""
MachineName = "nbumaster"
FQName = "nbumaster.nic.in"
GlobalDriveSeed = "VEND:#.:PROD:#.:IDX"
LocalDriveSeed = ""
MachineDescription = ""
MachineFlags = 0x65
MachineNbuType = master (3)
MachineState = active for disk jobs (12)
NetBackupVersion = 7.6.0.1 (760100)
OperatingSystem = linux (16)
ScanAbility = 5
nbumedia.nic.in
ClusterName = ""
MachineName = "nbumedia.nic.in"
FQName = "nbumedia.nic.in"
LocalDriveSeed = ""
MachineDescription = ""
MachineFlags = 0x65
MachineNbuType = media (1)
MachineState = active for disk jobs (12)
MasterServerName = "nbumaster"
NetBackupVersion = 7.6.0.1 (760100)
OperatingSystem = linux (16)
ScanAbility = 5
dr-msgbe3
MachineName = "dr-msgbe3"
FQName = "dr-msgbe3"
MachineDescription = ""
MachineNbuType = client (8)
dr-msgbe4
MachineName = "dr-msgbe4"
FQName = "dr-msgbe4"
MachineDescription = ""
MachineNbuType = client (8)
nullstsv
MachineName = "nullstsv"
FQName = "nullstsv"
MachineDescription = ""
MachineFlags = 0x2
MachineNbuType = ndmp (2) (storage_server)
nbumedia2.nic.in
ClusterName = ""
MachineName = "nbumedia2.nic.in"
FQName = "nbumedia2.nic.in"
LocalDriveSeed = ""
MachineDescription = ""
MachineFlags = 0x65
MachineNbuType = media (1)
MachineState = active for disk jobs (12)
MasterServerName = "nbumaster"
NetBackupVersion = 7.6.0.1 (760100)
OperatingSystem = linux (16)
ScanAbility = 5
hyd-msgbe1
MachineName = "hyd-msgbe1"
FQName = "hyd-msgbe1"
MachineDescription = ""
MachineNbuType = client (8)
hyd-msgbe2
MachineName = "hyd-msgbe2"
FQName = "hyd-msgbe2"
MachineDescription = ""
MachineNbuType = client (8)
hyd-msgbe3
MachineName = "hyd-msgbe3"
FQName = "hyd-msgbe3"
MachineDescription = ""
MachineNbuType = client (8)
hyd-msgbe4
MachineName = "hyd-msgbe4"
FQName = "hyd-msgbe4"
MachineDescription = ""
MachineNbuType = client (8)
Command completed successfully.

[Marianne]

So, comms is fine and media server is "active for disk jobs ".

I am wondering about this:
Use% : 98

Default HWM is 98%. So, the disk will be considered as full and not considered for backup jobs.
Maybe the reason why NBU marks it Down?

[Sandeepkumar]

should i change the value and check if it is working..??

[Marianne]

You need to cleanup or add space.

Staging disk backups to alternate storage (e.g. tape) will trigger cleanup after successful duplication or else natural image expiration. 

[Sandeepkumar]

Hi marianne

Thanks for the reply.

I have observed that nbrmms service has not started while netbackup services restarted.

I have restarted nbu services now the volumes are UP and backups are running perfectly.