Template:TroubleshootingHAStoragePools

SAS JBOD Cabling Diagrams

Important Cabling Rules to Follow

• Rule #1: Never connect the same HBA to the same disk enclosure twice.
• Rule #2: Never connect the same SAS expander to the same HBA twice. (see Rule #1)
• Rule #3: Every server must be connected to each Disk Enclosure twice (but not with the same HBA, see Rule #1)

Cabling Guidelines

• Use Broadcom 9300/9400/9500 HBAs or OEM variants of these HBAs. HPE servers connected to HPE JBODs/enclosures should use the standard HPE OEM HBAs.
• The same cabling rules apply to all enclosure models from all manufacturers.
• All Dell, HPE, WD/HGST, Seagate and Lenovo JBOD units come with dual SAS expanders as standard.
• Although most JBODs have labels for IN/OUT ports some have IN/OUT ports that may be used interchangeably (see vendor documentation).
• Supermicro offers Disk Enclosures / JBOD enclosure models with a single SAS/SATA expander (E1/E1C models) which should be avoided for HA configurations. Be sure to use enclosure models with dual SAS expanders, these units have “E2/E2C” in the model number.
• Each port from a given HBA must connect to separate JBOD. It is ok to have unused HBA ports, they may be used to connect to more JBOD units in the future.
• Incorrect cabling will cause IO fencing and performance problems so it is important to double check.
• Cascading JBODs should be avoided to reduce complexity.

Troubleshooting for HA Storage Pools

Expanding shortly with tips and suggestions on troubleshooting issues with Storage Pool HA Group setup issues...

Disks are not visible on all systems

If you don't see the storage pool name label on the pool's disk devices on both systems use this checklist to verify possible connection or configuration problems.

• Make sure there is connectivity from both QuantaStor Systems used in the Site Cluster to the back-end shared JBOD for the storage pool or the back-end SAN.
• Verify FC/SAS cables are securely seated and link activity lights are active (FC).
• All drives used to create the storage pool must be SAS, FC, or iSCSI devices that support multi-port and SCSI3 Persistent Reservations.
• SAS JBOD should have at least two SAS Expansion ports. Having a JBOD with 3 or more expansions ports and Redundant SAS Expander/Environment Service Modules(ESM) is prefferred.
• SAS JBOD cables should be within standard SAS cable lengths (less than 15 meters) of the SAS HBA's installed in the QuantaStor Systems.
• Faster devices such as SSDs and 15K RPM platter disk should be placed in a separate enclosure from the NL-SAS disks to ensure best performance.
• Cluster-in-a-Box solutions provide these hardware requirements in a single HA QuantaStor System and are recommended for small configurations (less than 25x drives)

The qs-util devicemap can be helpful when troubleshooting this issue.

Useful Tools

qs-iofence devstatus

The qs-iofence utility is helpful for the diagnosis and troubleshooting of SCSI reservations on disks used by HA Storage Pools.

# qs-iofence devstatus

Displays a report showing all SCSI3 PGR reservations for devices on the system. This can be helpful when zfs pools will not import and "scsi reservation conflict" error messages are observed in the syslog.log system log.

qs-util devicemap

The qs-util devicemap utility is helpful for checking via the CLI what disks are present on the system. This can be extremely helpful when troubleshooting disk presentation issues as the output shows the Disk ID, and Serial Numbers, which can then be compared between nodes.

HA-CIB-305-3-27-A:~# qs-util devicemap | sort
...
/dev/sdb	/dev/disk/by-id/scsi-35000c5008e742b13, SEAGATE, ST1200MM0017, S3L26RL80000M603QCVF,
/dev/sdc	/dev/disk/by-id/scsi-35000c5008e73ecc7, SEAGATE, ST1200MM0017, S3L26RWZ0000M604EFKC,
/dev/sdd	/dev/disk/by-id/scsi-35000c5008e65f387, SEAGATE, ST1200MM0017, S3L26JPA0000M604EDAH,
/dev/sde	/dev/disk/by-id/scsi-35000c5008e75a71b, SEAGATE, ST1200MM0017, S3L26Q9N0000M604W6MB,
...

HA-CIB-305-3-27-B:~# qs-util devicemap | sort
...
/dev/sdb	/dev/disk/by-id/scsi-35000c5008e742b13, SEAGATE, ST1200MM0017, S3L26RL80000M603QCVF,
/dev/sdc	/dev/disk/by-id/scsi-35000c5008e73ecc7, SEAGATE, ST1200MM0017, S3L26RWZ0000M604EFKC,
/dev/sdd	/dev/disk/by-id/scsi-35000c5008e65f387, SEAGATE, ST1200MM0017, S3L26JPA0000M604EDAH,
/dev/sde	/dev/disk/by-id/scsi-35000c5008e75a71b, SEAGATE, ST1200MM0017, S3L26Q9N0000M604W6MB,
...

• Note that while many SAS JBODs will typically produce consistent assignment of /dev/sdXX lettering, FC/iSCSI attached storage will typically vary due to variation in how Storage Arrays respond to bus probes. The important parts to match between nodes are the /dev/disk/by-id/.... values, and the Serial Numbers in the final column.