ec2009

[Shekaib]

Hi All,

I am using a Master(V7.1) and a media (V6.5)..Backup for the media server which is being taken is failing with ec2009.

12/03/2012 22:25:12 - requesting resource asprd434-ebr_lto
12/03/2012 22:25:12 - requesting resource asprd212.aldc.att.com.NBU_CLIENT.MAXJOBS.asprd434-ebr.aldc.att.com
12/03/2012 22:25:12 - requesting resource asprd212.aldc.att.com.NBU_POLICY.MAXJOBS.asprd434
12/03/2012 22:25:17 - Error nbjm (pid=5547) NBU status: 2009, EMM status: All compatible drive paths are down, but media is available
All compatible drive paths are down but media is available  (2009)
 

I have restarted the services and also restarted PBX but still getting the same error.

Any suggestions.

[Marianne]

All compatible drive paths are down

Which OS on Media server?

Troubleshooting needs to start at OS level.
We can provide troubleshooting steps once we have more info.
 

[Andy_Welburn]

Closest I can get:

http://www.symantec.com/business/support/index?page=content&id=TECH177880

[Shekaib]

@Marianne..OS of media server os Solaris 10..

[Marianne]

Add VERBOSE entry to /usr/openv/volmgr/vm.conf on the media server.

Restart NBU on media server.

Try to UP the drives.

If drives go DOWN again, check /var/adm/messages for reason.

[Shekaib]

output for vm.conf

cat vm.conf
VERBOSE
MM_SERVER_NAME = asprd434-ebr.aldc.att.com
ACS_TCP_RPCSERVICE
ACS_CSI_HOSTPORT = aacsls01.aldc.att.com 30031
ACS_SSI_INET_PORT = aacsls01.aldc.att.com 30031
 

Please check and also this is ACS tape library.

[Marianne]

Good!

What does 'vmoprcmd -d' on the media server show?

Have you checked /var/adm/messages for errors?

[Shekaib]

more messages | grep error
Dec  3 01:01:08 asprd434 automountd[4101]: [ID 784820 daemon.error] server cstadmin.sbc.com not responding
Dec  3 11:49:10 asprd434 vmd[10909]: [ID 715111 daemon.error] volume daemon terminating because it received a signal (15)
Dec  3 11:49:10 asprd434 vmd[10909]: [ID 164182 daemon.error] terminating - daemon terminated (7)
Dec  3 11:52:16 asprd434 tldd[9163]: [ID 886097 daemon.error] daemon terminating because no devices are configured, tldcd has not been started
Dec  4 09:23:07 asprd434 acsd[5290]: [ID 578017 daemon.error] ACS(0) [5290] timed out after waiting 901 seconds for ready, drive (0,8,1,8)
Dec  4 09:24:03 asprd434 ltid[9125]: [ID 926221 daemon.error] Operator/EMM server has DOWN'ed drive A00_LI001_LS8_D08_L3 (device 3)

vmoprcmd -d ds

                                  DRIVE STATUS

Drv Type   Control  User      Label  RecMID  ExtMID  Ready   Wr.Enbl.  ReqId
  0 hcart3   ACS                -                     No       -         0
  1 hcart3   ACS                -                     No       -         0
  2 hcart    ACS                -                     No       -         0
  3 hcart    ACS                -                     No       -         0

 

[Marianne]

"grep error" does not help to get the reason for drives being DOWN'ed. Anyhow, drives are currently up/ACS control. Are backups still failing? If so, use nbrbutil command on the master to reset Media server.

[RamNagalla]

hi,

does this drives are shared?

do you have 2 different denistity of Drives, Hcart3 and Hcart?

please post the outputs of  /usr/openv/volmgr/bin/scan

/usr/openv/volmgr/bin/vmoprcmd -d

/usr/openv/volmgr/bin/tpconfig -d

/usr/openv/volmgr/bin/tpautoconf -t

[Shekaib]

i did nbrbutil -resetMediaServer  name

restarted backups and they again failed with same ec.

2/04/2012 21:28:41 - requesting resource asprd434-ebr_lto
12/04/2012 21:28:41 - requesting resource asprd212.aldc.att.com.NBU_CLIENT.MAXJOBS.asprd434-ebr.aldc.att.com
12/04/2012 21:28:41 - requesting resource asprd212.aldc.att.com.NBU_POLICY.MAXJOBS.asprd434
12/04/2012 21:28:59 - Error nbjm (pid=5547) NBU status: 2009, EMM status: All compatible drive paths are down, but media is available
All compatible drive paths are down but media is available  (2009

[Marianne]

Are these 4 ACS drives the only devices on the media server?

How many Storage Units configured for this media server?

We see the following error in /var/adm/messages:

Dec  3 11:52:16 asprd434 tldd[9163]: [ID 886097 daemon.error] daemon terminating because no devices are configured, tldcd has not been started

This indicates that there should be a direct-attached TLD robot with drive(s) on this media server.

What does 'bpstulist -label asprd434-ebr_lto -U' show?

[Shekaib]

bpstulist -label asprd434-ebr_lto -U

Label:                asprd434-ebr_lto
Storage Unit Type:    Media Manager
Host Connection:      asprd434-ebr.aldc.att.com
Number of Drives:     2
On Demand Only:       no
Max MPX/drive:        4
Density:              hcart - 1/2 Inch Cartridge
Robot Type/Number:    ACS / 0
Max Fragment Size:    2048 MB

[Marianne]

Is master/EMM server aware of the UP state of the drives?

What does the following command on Master show?

vmoprcmd -h asprd434-ebr.aldc.att.com -d ds

Please also post 'MDS allocations' section of 'nbrbutil -dump'. 

[Shekaib]

vmoprcmd -h asprd434-ebr.aldc.att.com -d ds

                                  DRIVE STATUS

Drv Type   Control  User      Label  RecMID  ExtMID  Ready   Wr.Enbl.  ReqId
  0 hcart3   ACS                -                     No       -         0
  1 hcart3   ACS                -                     No       -         0
  2 hcart  DOWN-ACS             -                     No       -         0
  3 hcart    ACS                -                     No       -         0

when i try to reset and up the drive..

 vmoprcmd -reset 2
Unknown drive name
 vmoprcmd -up 2
Invalid Drive Number
 

[Marianne]

Are you issuing -reset and -up from master on from media server?

If master, remember to add -h <hostname>.

You need to check messages file to see what is wrong with drive(s). 
Find DOWN in messages file, then look at the lines above that to see what caused this.

This could be part of the problem:

 timed out after waiting 901 seconds for ready, drive (0,8,1,8)

Some issue between OS and device?

Try to troubleshoot outside of NBU - mount tape in drive using ACSLS.

Check status from OS:

mt -f /dev/rmt/## status

[Shekaib]

thanks for reminding..i gave cmd from media server and it worked and drive also came up..

Problem which i think is in the tape mounting...

2/05/2012 18:45:56 - requesting resource asprd434-ebr_lto
12/05/2012 18:45:56 - requesting resource asprd212.aldc.att.com.NBU_CLIENT.MAXJOBS.asprd434-ebr.aldc.att.com
12/05/2012 18:45:56 - requesting resource asprd212.aldc.att.com.NBU_POLICY.MAXJOBS.asprd434
12/05/2012 18:46:04 - granted resource  asprd212.aldc.att.com.NBU_CLIENT.MAXJOBS.asprd434-ebr.aldc.att.com
12/05/2012 18:46:04 - granted resource  asprd212.aldc.att.com.NBU_POLICY.MAXJOBS.asprd434
12/05/2012 18:46:04 - granted resource  GX0467
12/05/2012 18:46:04 - granted resource  A00_LI001_LS8_D08_L3
12/05/2012 18:46:04 - granted resource  asprd434-ebr_lto
12/05/2012 18:46:05 - started process bpbrm (pid=19317)
12/05/2012 18:46:06 - connecting
12/05/2012 18:46:06 - connected; connect time: 0:00:00
12/05/2012 18:47:07 - mounting GX0467

generally it fails after this.

Try to troubleshoot outside of NBU - mount tape in drive using ACSLS--for this we did try to mount and unmount tape..and it worked fine.

[Marianne]

Did you confirm that OS is seeing tape mount with 'mt -f /dev/rmt/xx status'?

[RamNagalla]

Shekaib,

did you ever have the succuess full backup with this Drives?

Did you set the access control in the ACLSLS servers?

do you have 4 Drives connected in OS level?

[Shekaib]

 mt -f /dev/rmt/0cbn status
/dev/rmt/0cbn: write protected or reserved.