Disk storage server is down(2106)

[MickBoosh]

Hi,

I've started getting the Disk storage server is down(2106) error. I'm having trouble trying to trouble shoot it. We have a staging area attached via fibre. There ar no windows event log errors for the time it went down. 

I've looked in bpdm and bptm but I can't find anything obvious in there.

Windows 2003 R2 x86

NBU 7.5.0.3

02/02/2013 13:08:23 - Info nbjm(pid=5848) starting backup job (jobid=4311605) for client XXXXXXXn, policy SQL_Backup_TransactionLogsNew, schedule Default-Application-Backup  

02/02/2013 13:08:23 - Info nbjm(pid=5848) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=4311605, request id:{B0C094C4-7A4A-44A9-B021-A983E41A9522})  

02/02/2013 13:08:23 - requesting resource Disk_Staging

02/02/2013 13:08:23 - requesting resource [MASTER].NBU_CLIENT.MAXJOBS.xxxxxxxxxn

02/02/2013 13:08:23 - requesting resource [MASTER].NBU_POLICY.MAXJOBS.SQL_Backup_TransactionLogsNew

02/02/2013 13:08:23 - awaiting resource Disk_Staging - Maximum job count has been reached for the storage unit

02/02/2013 13:10:23 - Info nbjm(pid=5848) Waiting in NetBackup scheduler work queue on server xxxx    

02/02/2013 13:14:23 - Info nbjm(pid=5848) Waiting in NetBackup scheduler work queue on server xxxx    

02/02/2013 13:18:23 - Info nbjm(pid=5848) Waiting in NetBackup scheduler work queue on server xxxx    

02/02/2013 13:20:23 - Info nbjm(pid=5848) Waiting in NetBackup scheduler work queue on server xxxx    

02/02/2013 13:24:23 - Info nbjm(pid=5848) Waiting in NetBackup scheduler work queue on server xxxx    

02/02/2013 13:26:23 - Info nbjm(pid=5848) Waiting in NetBackup scheduler work queue on server xxxx    

02/02/2013 13:30:23 - Info nbjm(pid=5848) Waiting in NetBackup scheduler work queue on server xxxx    

02/02/2013 13:32:23 - Info nbjm(pid=5848) Waiting in NetBackup scheduler work queue on server xxxx    

02/02/2013 13:34:23 - Info nbjm(pid=5848) Waiting in NetBackup scheduler work queue on server xxxx    

02/02/2013 13:36:23 - Info nbjm(pid=5848) Waiting in NetBackup scheduler work queue on server xxxx    

02/02/2013 13:38:23 - Info nbjm(pid=5848) Waiting in NetBackup scheduler work queue on server xxxx    

02/02/2013 13:41:04 - Error nbjm(pid=5848) NBU status: 2106, EMM status: Storage Server is down or unavailable  

Disk storage server is down(2106)

[RamNagalla]

hi,

what kind of storage server you are using?

Pure disk or Data domain?

let us know the output of below file

Install path\veritas\netbackup\bin\nbdevquery -listdp -stype <storagetype -U

[Marianne]

Please tell us more about your Disk_Staging storage server.

32-bit W2003? This means Advanced Disk, right? Not Basic Disk?
Configured on Master or on a media server?

Extract from Troubleshooter:

Verify that all media servers configured for the storage server can communicate with the storage server. The bpstsinfo command queries the storage server periodically, so you can use the bpstsinfo log on the media server.

Please create bpstsinfo folder under .../netbackup/logs

Disclaimer: I cannot find any information on whether bpstsinfo is a legacy or unified log. I am hoping that creating legacy type log folder will work.

[MickBoosh]

I'm not sure we are using a storage server, I've never heard of this before. It's just a basic disk. That command says no disk pools found. Confused!

[RamNagalla]

please provide the nbemmcmd -listhost -verobose  for the media server which is in issue.

and also, if that is the basic disk,

just log into the server , go to the baisc disk directory and try to create one text file, and see how it works.

if creating text file fails, than you need to work with your windows again, and storage team to fix that.

[MickBoosh]

It is intermittent I must add. The errors started on the 02/02/2013 and only lasted an hour. No errrors since then, but we have had them for a few weeks intermittently. It's configured on a master server.

NBEMMCMD, Version: 7.5.0.3

The following hosts were found:

[master]

        MachineName = "[master]"

        FQName = "[master].[domain]"

        MachineDescription = ""

        MachineNbuType = server (6)

[master]

        ClusterName = ""

        MachineName = "[master]"

        FQName = "[master].[domain]"

        GlobalDriveSeed = "VEND:#.:PROD:#.:IDX"

        LocalDriveSeed = ""

        MachineDescription = ""

        MachineFlags = 0xb7

        MachineNbuType = master (3)

        MachineState = active for tape and disk jobs (14)

        NetBackupVersion = 7.5.0.3 (750300)

        OperatingSystem = windows (11)

        ScanAbility = 5

stasbkp

        ClusterName = ""

        MachineName = "[media]"

        FQName = "[media].[domain]"

        LocalDriveSeed = ""

        MachineDescription = ""

        MachineFlags = 0x17

        MachineNbuType = media (1)

        MachineState = active for tape and disk jobs (14)

        MasterServerName = "stas75"

        NetBackupVersion = 7.5.0.3 (750300)

        OperatingSystem = windows (11)

        ScanAbility = 5

Command completed successfully.

[RamNagalla]

hi,

if its working fine now, you are good to go..

if you get this error in future, its better to practice, to start from the OS disk level,as its basic disk.

check if the disk is accssible from the OS level or not, 

then try to check the  the disk status. 

make sure your disk is not reaching High disk I/O performance. 

[MickBoosh]

It is working now yes, but as I've had this problem for the last two weeks, it will happen again. It usually happens through the night so I will be unable to check.

I have suspected it is when the disk is being highly utilised actually.

Thanks for your help

[Mark_Solutions · Accepted solution]

This may be a timeout caused, as you say, by the servers being very busy.

It may be a lack of network ports or it may just be a communications timeout whilst checking the disks.

Add the following file (just a flat file with no file extention - so caraful that you are viewing extentions and it doesnt get created as a text file) to the Master and Media Servers:

<install path>\veritas\netbackup\db\config\DPS_PROXYDEFAULTRECVTMO

Open it up in notepad and put the value of 800 in it.

This just gives disk status checks a better timeout value which usually helps when things are very busy

[MickBoosh]

Thanks I will try that.