Template:TroubleshootingHAStoragePools

SAS JBOD Cabling Diagrams

Important Cabling Rules to Follow

Rule #1: Never connect the same HBA to the same disk enclosure twice.  
Rule #2: Never connect the same SAS expander to the same HBA twice.  (see Rule #1)
Rule #3: Every server must be connected to each Disk Enclosure twice (but not with the same HBA, see Rule #1)

Cabling Guidelines

Use Broadcom/Avago/LSI HBAs or OEM versions of these HBAs in your QuantaStor appliances.  These are standard with all major manufacturers.  HPE is an exception, there HBAs are Adaptec based.  HPE HBAs must be used with HPE JBODs.
The same cabling rules apply to all enclosure models from all manufacturers.  
All Dell, HPE, HGST, and Lenovo JBOD units we know of come with dual SAS expanders as standard.
Although most JBODs have labels for IN/OUT ports note that these are largely for simplified documentation and that IN/OUT ports may be used interchangeably.  Ie. If a JBOD has 2x IN and 2x OUT ports, they can all be used to provide direct connectivity to the server side HBAs.
Note that SuperMicro offers Disk Enclosures / JBOD enclosure models with a single SAS/SATA expander (E1 models) which should be avoided.  Be sure to purchase a model with dual SAS expanders, these units have “E2” in the model number.  The 12Gb SAS variety are preferred.
Each port from a given HBA must connect to separate JBOD.   It is ok to have unused HBA ports, they may be used to connect to more JBOD units in the future.  
Incorrect cabling will cause IO fencing and performance problems so it is important to double check.
Cascading JBODs is ok but should be avoided with SSD configurations as it will reduce performance.  

Troubleshooting for HA Storage Pools

Expanding shortly with tips and suggestions on troubleshooting issues with Storage Pool HA Group setup issues...

Disks are not visible on all systems

If you don't see the storage pool name label on the pool's disk devices on both systems use this checklist to verify possible connection or configuration problems.

• Make sure there is connectivity from both QuantaStor Systems used in the Site Cluster to the back-end shared JBOD for the storage pool or the back-end SAN.
• Verify FC/SAS cables are securely seated and link activity lights are active (FC).
• All drives used to create the storage pool must be SAS, FC, or iSCSI devices that support multi-port and SCSI3 Persistent Reservations.
• SAS JBOD should have at least two SAS Expansion ports. Having a JBOD with 3 or more expansions ports and Redundant SAS Expander/Environment Service Modules(ESM) is prefferred.
• SAS JBOD cables should be within standard SAS cable lengths (less than 15 meters) of the SAS HBA's installed in the QuantaStor Systems.
• Faster devices such as SSDs and 15K RPM platter disk should be placed in a separate enclosure from the NL-SAS disks to ensure best performance.
• Cluster-in-a-Box solutions provide these hardware requirements in a single HA QuantaStor System and are recommended for small configurations (less than 25x drives)

The qs-util devicemap can be helpful when troubleshooting this issue.

Useful Tools

qs-iofence devstatus

The qs-iofence utility is helpful for the diagnosis and troubleshooting of SCSI reservations on disks used by HA Storage Pools.

# qs-iofence devstatus

Displays a report showing all SCSI3 PGR reservations for devices on the system. This can be helpful when zfs pools will not import and "scsi reservation conflict" error messages are observed in the syslog.log system log.

qs-util devicemap

The qs-util devicemap utility is helpful for checking via the CLI what disks are present on the system. This can be extremely helpful when troubleshooting disk presentation issues as the output shows the Disk ID, and Serial Numbers, which can then be compared between nodes.

HA-CIB-305-3-27-A:~# qs-util devicemap | sort
...
/dev/sdb	/dev/disk/by-id/scsi-35000c5008e742b13, SEAGATE, ST1200MM0017, S3L26RL80000M603QCVF,
/dev/sdc	/dev/disk/by-id/scsi-35000c5008e73ecc7, SEAGATE, ST1200MM0017, S3L26RWZ0000M604EFKC,
/dev/sdd	/dev/disk/by-id/scsi-35000c5008e65f387, SEAGATE, ST1200MM0017, S3L26JPA0000M604EDAH,
/dev/sde	/dev/disk/by-id/scsi-35000c5008e75a71b, SEAGATE, ST1200MM0017, S3L26Q9N0000M604W6MB,
...

HA-CIB-305-3-27-B:~# qs-util devicemap | sort
...
/dev/sdb	/dev/disk/by-id/scsi-35000c5008e742b13, SEAGATE, ST1200MM0017, S3L26RL80000M603QCVF,
/dev/sdc	/dev/disk/by-id/scsi-35000c5008e73ecc7, SEAGATE, ST1200MM0017, S3L26RWZ0000M604EFKC,
/dev/sdd	/dev/disk/by-id/scsi-35000c5008e65f387, SEAGATE, ST1200MM0017, S3L26JPA0000M604EDAH,
/dev/sde	/dev/disk/by-id/scsi-35000c5008e75a71b, SEAGATE, ST1200MM0017, S3L26Q9N0000M604W6MB,
...

• Note that while many SAS JBODs will typically produce consistent assignment of /dev/sdXX lettering, FC/iSCSI attached storage will typically vary due to variation in how Storage Arrays respond to bus probes. The important parts to match between nodes are the /dev/disk/by-id/.... values, and the Serial Numbers in the final column.