(2016) Disk Storage Server is Down

[shahriar_sadm]

Hi Dears,

I have a problem with backups on DataDomain system, All backups failed with status 2016 (without any change on system). 

./nbdevquery -liststs -stype DataDomain -U
Storage Server : DD990-1
Storage Server Type : DataDomain
Storage Type : Formatted Disk, Network Attached
State : DOWN
Flag : OpenStorage
Flag : CopyExtents
Flag : AdminUp
Flag : InternalDown
Flag : LifeCycle
Flag : CapacityMgmt
Flag : FragmentImages
Flag : Cpr
Flag : FT-Transfer
Flag : OptimizedImage

********************************

./nbdevquery -listdp -stype DataDomain -U
Disk Pool Name : DD990-DiskPool
Disk Pool Id : DD990-DiskPool
Disk Type : DataDomain
Status : UP
Flag : Patchwork
Flag : Visible
Flag : OpenStorage
Flag : SingleStorageServer
Flag : CopyExtents
Flag : AdminUp
Flag : InternalUp
Flag : LifeCycle
Flag : CapacityMgmt
Flag : FragmentImages
Flag : Cpr
Flag : FT-Transfer
Flag : OptimizedImage
Raw Size (GB) : 522426.35
Usable Size (GB) : 522426.35
Num Volumes : 1
High Watermark : 98
Low Watermark : 80
Max IO Streams : -1
Comment :
Storage Server : DD990-1 (DOWN)

********************

./nbdevquery -listdv -stype DataDomain -U
Disk Pool Name : DD990-DiskPool
Disk Type : DataDomain
Disk Volume Name : ddboost-stu
Disk Media ID : @aaaar
Total Capacity (GB) : 522426.35
Free Space (GB) : 296119.36
Use% : 43
Status : DOWN
Flag : ReadOnWrite
Flag : AdminDown
Flag : InternalUp
Num Read Mounts : 0
Num Write Mounts : 1
Cur Read Streams : 0
Cur Write Streams : 0
Num Repl Sources : 0
Num Repl Targets : 0

*********************

./nbdevconfig -changestate -stype DataDomain -dp DD990-DiskPool -dv ddboost-stu -state UP
successfully changed the state of disk volume

(but not happend)

================

Please Help! 

Thanks

[watsons]

Try this as well:

./nbdevconfig -changestate -stype DataDomain -dp DD990-DiskPool -dv ddboost-stu -state RESET

If it didn't stay up, you might want to check the DataDomain plugin to make sure it's OK - or if version is supported.

[Marianne]

At my previous company, we had a customer where this started happening regularly. The DD was simply not responding to NBU. The only way to get it to work again was to reboot the DD.
(They eventually replaced the DDs with different type of OST storage.)

I suggest you start investigation on the DD.

[shahriar_sadm · Accepted solution]

Hi dears and thanks for replies

After my request to change one client attached to DD, Infrastructure team detach one operational client from network switch, as I learned all of clients added to DD must be alive but disk storage server going down, I uncked this client on storage server section in NBU and storage server worked well.

Thanks

[Marianne]

Clients should NOT be added as Storage Servers.
Unless you install the client as (San) media server.