V-76-58645-585: Failed to reserve a majority of disks in cluster dynamic disk group

Hi Zahid,
   It is normal to see the SFW disks show as unreadable once they have been added to a Dynamic Disk Group in VEA.

A couple of questions/things to try which may help narrow down the cause of this:

1. Can you import the DG as a secondary/non-clustered? You can check this box after choosing to import the DG in VEA, or run the import command with the -d switch: vxdg -g SQL-DiskGroup -d -C import? I also added -C (clear host ID) in this example just to be safe.

If this imports successfully, then we will know that we have access to the disks, but it would appear these disks aren't allowing a SCSI reservation to be placed on them.

2. Is this SAN attached storage, or at least storage attached via SCSI to the server? If the disks are IDE or are connected via  a PERC card, they will not support SCSI reservations and will fail to import as a clustered DG, but will be successful as a secondary DG.

3. Are you using SCSI-2 or SCSI-3 reservations? I have seen this issue occur in cases where SFW has been configured to use SCSI-3, but the array is not configured to accept SCSI-3, therefore they fail. We have a utility that can be used to help ensure the SCSI reservation process works properly (SCSI-2 or SCSI3), described below.

4. You can download our scscicmd utility (scscmd.zip) from ftp.veritas.com/pub/support/vmtools and try running some SCSI commands (i.e. SCSI2/3 tests to confirm they support SCSI, and even try to place SCSI reservations on them).

Below is some information on this tool and how to run it (sorry, it is quite lengthy as it has example output included for each command:

The SCSIcmd utility has been enhanced to now support operations on both SCSI-2 and SCSI-3 reserved disks. All the original options of SCSIcmd still exist which can be seen by either running SCSIcmd /?, or by reviewing Technote 288646 which is included in the 'Related Documents' section below; however, with the new SCSIcmd commands, there should be very little need to use the older options.

SCSIcmd download location: ftp://ftp.veritas.com/pub/support/vmtools
Filename: SCSIcmd.zip

Before we begin:
The syntax for all commands discussed in this article are as follows: scsicmd -d<harddisk#> -s<scsi_command>

Syntax Example: scsicmd -d14 -sscsi_reserved_status

This command will run a SCSI reservation status check on Harddisk14

Note: In regards to this syntax, here are some further points to consider:

- Make sure to include the -s before the command (i.e. -sscsi3_test) which means all commands will start with two S's. Failure to do so will result in an error being displayed: ERROR: No such SCSI Opcode: 0xc from SCSI opcode table.

- To perform an action across all disks in the system, you can specify ALL with the -d switch. Example: scsicmd -dALL -sscsi_reserved_status

- You can also specify multiple disks for the command. For example: scsicmd -d3,4,5,7,9,11,12 -sscsi_reserved_status.

As stated above, the new SCSIcmd utility now supports SCSI-3, and has been enhanced to perform SCSI-2 operations more effectively then before. This Technote will cover the commands that are most likely to be run to assist in reservation issue cases. Section 1 will provide commands for both SCSI-2 and SCSI3, Section II will provide SCSI-2 specific commands, and Section III will provide SCSI-3 specific commands.

I. Commands for both SCSI-2 and SCSI-3

scsi_reserved_status
This option provides the current SCSI reservation status (SCSI-2 or SCSI-3) of the disk(s) specified. This is a good command to run to determine if there is any type of reservation on the disk which might be causing access issues.

Syntax Example: scsicmd -d14 -sscsi_reserved_status

Below are some example outputs of what will be seen with this command depending on the reservation type and status:

SCSI-2 Output when there is a reservation on the disk:

Harddisk14

Scsi Address
------------------
Length : 0x8
PortNumber : 0x2
PathId : 0x1
TargetId : 0x2
Lun : 0x45

Harddisk14 is currently SCSI-2 reserved by SCSI Address (0x2, 0x1, 0x0, 0x1) of the computer: MTV2850-07.

SCSI-3 Output when there is a reservation on the disk

Harddisk14

Scsi Address
------------------
Length : 0x8
PortNumber : 0x2
PathId : 0x1
TargetId : 0x2
Lun : 0x45

Harddisk 14 has SCSI-3 key(s) registered.
SCSI3 registered keys on harddisk14
Key: 7856341278563412
Key: 7856341278563412
Harddisk14 is currently under SCSI-3 reservation
SCSI3 reservation holder key on harddisk14
Key: 7856341278563412
Harddisk14 is currently being SCSI-3 reserved on the computer: MTV2850-07.

SCSI2 or SCSI3 output when there is no reservation on the disk:

Harddisk14

Scsi Address
------------------
Length : 0x8
PortNumber : 0x2
PathId : 0x1
TargetId : 0x2
Lun : 0x45

Harddisk14 of the computer: MTV2850-07 is not under any SCSI reservation.


II. Commands specific to SCSI-2

scsi2_reserve
This performs a SCSI-2 reservation. This can be run to ensure a SCSI-2 reservation can be placed on a disk. One example would be in an environment where SCSI-2 is being used, but the Clustered Disk Group fails to come online with an error stating it failed to reserve a Majority of the Disks.

Syntax Example: scsicmd -d14 -sscsi2_reserve

Example of the output of a successful reservation:

Harddisk12

Scsi Address
------------------
Length : 0x8
PortNumber : 0x3
PathId : 0x1
TargetId : 0x1
Lun : 0x7

Successfully issued SCSI-2 reservation on harddisk12 through SCSI Address (0x3, 0x1, 0x1, 0x7) of the computer: MTV2850-07

Checking the SCSI-2 reservation status on harddisk12.....
Harddisk12 is being SCSI-2 reserved.

scsi2_release
This option performs a SCSI-2 release reservation. This can be used in instances where there may be a stuck reservation on a disk that needs to be cleared in order for the current system to import the Disk Group.

Syntax Example: scsicmd -d14 -sscsi2_release

Example of the output of a successful reservation release on Harddisk14:

Harddisk14

Scsi Address
------------------
Length : 0x8
PortNumber : 0x3
PathId : 0x1
TargetId : 0x1
Lun : 0x7

Successfully issued SCSI-2 release on harddisk14 through SCSI Address (0x3, 0x1, 0x1, 0x7) of the computer: MTV2850-07

scsi2_clear
This option performs SCSI-2 bus reset/break reservation. This will not only break the reservation on the specified harddisk(s), but will also issue a SCSI BUS reset down the path(s) of the disk(s) specified. This should only be run when all disks or Disk Groups are affected on a server. For example, if a server has 2 Disk Groups and only one is having an issue and the other one is imported and properly running, issuing this command could cause the cluster resource to determine that there is a reservation issue on the healthy disk and issue a resource fault. When performing a BUS reset, SFW will place another reservation on all imported disks, but if a cluster monitor cycle occurs prior to this, it will fault. This is more likely to occur in an MSCS cluster as it is checking reservation status every 5 seconds (where VCS only checks every 60 seconds).

Syntax Example: scsicmd -d14 -sscsi2_clear

Example of a successful SCSI-2 clear command:

Checking for disk(s) that have the common Scsi Port number and Scsi Path Id....
Harddisk14

Scsi Address
------------------
Length : 0x8
PortNumber : 0x3
PathId : 0x1
TargetId : 0x1
Lun : 0x7

Successfully issued bus reset to ScsiPort3, PathId1.


III. Commands specific to SCSI-3

scsi3_reserve
This command performs a SCSI-3 reservation with a predefined SCSI-3 key (1234567812345678)

Syntax Example: scsicmd -d14 -sscsi3_reserve

Example of the output of a successful SCSI-3 reservation being placed on the disk:

Harddisk14

Scsi Address
------------------
Length : 0x8
PortNumber : 0x3
PathId : 0x1
TargetId : 0x1
Lun : 0x7

Successfully performed SCSI-3 reservation on harddisk14.

scsi3_release
This command performs a SCSI-3 reservation release. This is only used to remove the reservation key from a specific disk down a specific path (remember, all disks share the same key in SCSI-3). To remove a reservation completely from a disk (removing both the registrant and register key), use the scsi3_clear option instead which is explained further below.

Syntax Example: scsicmd -d14 -sscsi3_release

Example of the output of a successful reservation:

Harddisk14

Scsi Address
------------------
Length : 0x8
PortNumber : 0x3
PathId : 0x1
TargetId : 0x1
Lun : 0x7

Harddisk14 has the current SCSI-3 reservation holder:
7856341278563412

Successfully performed SCSI-3 release reservation on harddisk14

scsi3_clear
This option clears the SCSI-3 key and SCSI-3 reservation. This is the option that should be used when trying to clear a reservation on a disk that is SCSI-3 reserved. This should be used instead of the scsi3_release command as the release command doesn't completely clear the reservation key.

Syntax Example: scsicmd -d14 -sscsi3_clear

Example of the output of a successful clear:

Harddisk14

Scsi Address
------------------
Length : 0x8
PortNumber : 0x3
PathId : 0x1
TargetId : 0x1
Lun : 0x7

Successfully performed SCSI-3 clear operation on harddisk14

scsi3_test
This option tests a disk to determine if it supports SCSI-3. This is a very useful command to run when a customer is considering moving to SCSI-2 to SCSI-3. For example, when running Active/Active in a cluster, SCSI-3 is required and we need to confirm that the disk/array supports this.

Syntax Example: scsicmd -d14 -sscsi3_test

Example of the output of a successful SCSI-3 reservation test:

Test #1: Clean up any SCSI-3 keys left on harddisk14
Test #2: Perform SCSI-3 Register action on harddisk14 with SCSI-3 key: 7856341278563412
Test #3: Perform SCSI-3 reservation action on harddisk14 with SCSI-3 key: 7856341278563412
Test #4: Perform SCSI-3 release action on harddisk14 with SCSI-3 key: 7856341278563412
Test #5: Perform a part of SCSI-3 preempt action test on harddisk14.
Test #6: Perform SCSI-3 Register Ignore Existing action on harddisk14.
Test #7: Perform SCSI-3 Unregister Service Action on harddisk14.
Test #8: Perform SCSI-3 Clear Service Action on harddisk14.
SCSI-3 PERSISTENT OPERATION TESTS: All tests are PASSED on harddisk14.

Hi rhanley,

I have the same issue with SCSI-3 reservation. Storage is ST6140/DS4700 with 7.50.x.x firmware, connected to the Windows 2003 server host with Emulex LPe11002 HBA (newest driver 2.20.6 and newest firmware 2.82a3)

I used utility suggested by you and here the output:

 

C:\SCSIcmd\SCSIcmd>scsicmd.exe -d4 -sscsi3_test

 

********************** SCSI-3 SUPPORT TEST  ***********************************
* The SCSI-3 SUPPORT TEST performs a set of SCSI-3 PR on disk(s) specified in *
* -d option.                                                                  *
* SCSICMD uses the predefined '1234567812345678' key in SCSI-3 support test.  *
*                                                                             *
* Make sure that the testing shouldn't be under the cluster resource.         *
*                                                                             *
* If SFW DMP (5.0 DDI-3/5.1 GA/5.1 DDI-1 or a newer DDI version) hasn't been  *
* installed nor claim the disk, make sure that there is only one HBA path     *
* connected to the disk(s) from the testing host before running SCSICMD tool. *
*******************************************************************************

Harddisk4

Scsi Address
------------------
  Length     : 0x8
  PortNumber : 0x3
  PathId     : 0x0
  TargetId   : 0x2
  Lun        : 0x1

******** PERFORM SCSI-3 PR OPERATION TESTS ON Harddisk4   *************

===>Test #1:  Clean up any SCSI-3 keys left on harddisk4

Test #1 - **** PASSED.  There is no SCSI-3 key left on the disk: #4.

===>  Test #2:  Perform SCSI-3 Register action on harddisk4 with SCSI-3 key: 7856341278563412

Test #2 - Successfully issued SCSI-3 Register action.
Test #2 - Checking...There should be a registered key on harddisk4
                SCSI3 registered keys on harddisk4
                        Key: 7856341278563412
                        Key: 7856341278563412
                        Key: 7856341278563412
                        Key: 7856341278563412
Test #2 - *** PASSED.  Successfully performed SCSI-3 Register Service Action.

===>  Test #3:  Perform SCSI-3 reservation action on harddisk4 with SCSI-3 key: 7856341278563412

Test #3 - Successfully issued SCSI-3 reservation action.
Test #3 - Checking...Reading the SCSI-3 reservation holder
                SCSI3 reservation holder key on harddisk4
                        Key: 7856341278563412
Test #3 - *** PASSED.  Successfully performed SCSI-3 Reservation Action.

==>   Test #4:  Perform SCSI-3 release action on harddisk4 with SCSI-3 key: 7856341278563412

Test #4 - Successfully issued SCSI-3 release action.
Test #4 - Checking...There should be none SCSI-3 reservation holder left on the disk.
Test #4 - *** Failed.  There is a SCSI-3 holder left on harddisk4.

 

 

C:\SCSIcmd\SCSIcmd>scsicmd.exe -d4 -sscsi3_release

 

************************* SCSI-3 RELEASE OPERATION ****************************
* The SCSI-3 RESERVATION performs SCSI-3 release on disk(s) given in the      *
*  -d option.                                                                 *
*                                                                             *
* If SFW DMP (5.0 DDI-3/5.1 GA/5.1 DDI-1 or a newer DDI version) hasn't been  *
* installed nor claim the disk, make sure that there is only one HBA path     *
* connected to the disk(s) from the testing host before running SCSICMD tool. *
*******************************************************************************

Harddisk4

Scsi Address
------------------
  Length     : 0x8
  PortNumber : 0x3
  PathId     : 0x0
  TargetId   : 0x2
  Lun        : 0x1

Harddisk4 has the current SCSI-3 reservation holder:
0000000000000000

          *****  PGR OUT <== RELEASE operation on harddisk4  *****

Scsi Address
------------------
  Length     : 0x8
  PortNumber : 0x3
  PathId     : 0x0
  TargetId   : 0x2
  Lun        : 0x1

Retry flag: 0
psptdwb->sptd.ScsiStatus: 0x0
Error: 170  The requested resource is in use.

ERROR: Unable to perform SCSI-3 release action on harddisk4.
       The disk may be been under a cluster resource or
       has been SCSI-3 reserved by another HBA iniator/host.

 

C:\SCSIcmd\SCSIcmd>

 

So import as dynamic disk group works fine, but as cluster disk group does not work.

I appreciate any ideas,
Thanks in advance