HA Cluster Setup (external SAN)
A SAN-attached scale-up high-availability cluster puts a pair of QuantaStor appliances in front of block storage delivered from somewhere else -- an external array, or a second tier of QuantaStor appliances -- over Fibre Channel, iSCSI or NVMe-oF. The front-end appliances build a ZFS storage pool from the LUNs they are given and serve it to clients over every protocol, with automatic failover between them. This page covers the part that is specific to that back end: presenting the LUNs so that both front-end appliances see them, and the checks that behave differently when the shared storage arrives over a fabric.
Everything above the shared storage -- the site cluster, the HA group, the virtual interface, failover and I/O fencing -- is identical to the JBOD-attached build and is documented on HA Cluster Setup (JBODs). Recovery from a failure in a running cluster is on Scale-up HA Storage Pool Troubleshooting.
| Section | Purpose |
|---|---|
| What the SAN-attached design is | Front-end and back-end roles, and when to choose this over a JBOD |
| Hardware requirements | Front-end, back-end and fabric |
| Presenting the back-end storage | Host entries, software adapters, FC target mode |
| Building the HA pool | What is common with the JBOD build, and the one check that is not |
| Provisioning to clients | Shares, volumes and host assignment on the front end |
| Troubleshooting | SAN-specific diagnostics, and where the recovery procedures live |
| Command line reference | The back-end connectivity commands |
What the SAN-attached design is
A scale-up HA cluster needs at least two appliances, and the pool's disks cannot live inside either of them -- a surviving appliance has to be able to import the same pool. The simplest way to satisfy that is a shared SAS enclosure, which is what HA Cluster Setup (JBODs) covers. The alternative is to have the block storage arrive over a fabric.
In this design the two QuantaStor appliances act as a gateway to storage that lives elsewhere. They are the front-end or controller nodes: they hold the ZFS pool, the HA group, the virtual interfaces, and every client-facing protocol. Whatever supplies the LUNs is the back end. That can be a third-party array, or it can be QuantaStor appliances configured as data nodes, each with its own local pools and storage volumes, which is the tiered arrangement the diagrams below describe.
Choose this over a JBOD when the back-end storage already exists, when the capacity has to sit further away than a SAS cable reaches, or when the back end is doing something a JBOD cannot -- its own RAID, its own tiering, its own scale.
Two consequences follow from the storage arriving over a fabric rather than a SAS cable, and both matter:
- Every LUN in the pool must be presented to both front-end appliances. This is the same rule as the JBOD build, and QuantaStor enforces it the same way: creating the HA group verifies every pool device on the secondary appliance and refuses if any is missing.
- The LUNs must support SCSI-3 persistent reservations. I/O fencing is what guarantees only one appliance imports the pool, and it is enforced by the device rather than by QuantaStor. A back end that does not honour persistent reservations cannot host a highly-available pool. See I/O fencing and SCSI-3 persistent reservations.
QuantaStor has been tested with third-party arrays including NetApp and HPE MSA, as well as with QuantaStor itself as the back end. Contact support@osnexus.com for the current hardware compatibility list, or to have an array added to it.
Hardware requirements
- 2x QuantaStor appliances as front-end controller nodes.
- Back-end storage presenting block LUNs. Where that is QuantaStor, 2x or more appliances as data nodes with local SAS or SATA disk.
- A high-performance fabric between them -- FC, iSCSI or NVMe-oF.
- At least two network interfaces per front-end appliance on separate networks, for the two heartbeat rings of the site cluster.
- Mirrored boot devices in each appliance, 480GB or larger SSD or NVMe media, on a hardware RAID controller. A boot device that fills up is the most common cause of local database corruption.
Separate the front-end and back-end networks. Client traffic and the traffic between the controller and data nodes have different characteristics and different failure consequences, and mixing them makes both harder to reason about. Where the back end is iSCSI, disable iSCSI access on the management and replication ports so that back-end traffic cannot land on them:
qs network-port-modify --port=eth0 --iscsi-enable=false
qs network-port-modify also carries --nvmeof-enable for NVMe-oF TCP, --mtu for jumbo frames, and --auto-tune for RX/TX buffer sizing. The same settings are in the WUI; see Network Ports.
Back-end data nodes
Where the back end is QuantaStor, configure each data node as an ordinary appliance:
- Build one or more storage pools from its local disk. A pool of 10 to 20 drives is a good size; use several pools per appliance rather than one very large one.
- Create one storage volume per pool, sized to the pool.
- SAS disk is preferable, but enterprise SATA is usable here -- the dual-port requirement applies to the front-end pool's devices, not to the disks behind a back-end array.
- HBAs or hardware RAID controllers are both acceptable on a data node.
See Storage Pools and Storage Volumes for the pool and volume work itself, which is not HA-specific.
Presenting the back-end storage
The goal is that both front-end appliances see the same set of LUNs as physical disks. What that takes depends on the transport.
Where the back end is QuantaStor, the mechanism is a Host entry per front-end appliance, carrying that appliance's initiator identifiers, with the back-end storage volumes assigned to those hosts. Get the front-end appliance's iSCSI initiator name from the appliance itself:
# qs-util iscsiiqn INFO: iSCSI IQN: iqn.2009-15.com.osnexus:01:7531003303be
Then, on the back end, qs host-add creates the host and qs host-initiator-add adds an identifier to it -- --iqn takes an iSCSI IQN, an NVMe NQN or an FC WWPN. Assign every back-end volume to every front-end host, so that both front-end appliances see every LUN.
iSCSI and NVMe-oF: software adapters
On each front-end appliance, a software adapter establishes and maintains the connection to the back end. It logs in to the targets and surfaces their LUNs in the Physical Disks section, where they can be used to build a pool like any other device.
The dialog asks for the Portal IP Address of the back end and states what it will do: "All LUNs assigned to the initiator IQN/NQN of the system will be imported." Under Software Adapter Settings the port and CHAP credentials are optional -- the port defaults to 3260 for iSCSI and 4420 for NVMe. A CHAP username without a password, or a password without a username, is rejected; supply both or neither.
From the command line:
qs sw-controller-add --storage-system=qs-node1 --name=backend-1 \ --ip-address=10.10.20.5 --sw-controller-type=iscsi
qs sw-controller-add takes --sw-controller-type of iscsi (the default), nvme-tcp or nvme-rdma, an optional --port, and optional --chap-user and --chap-pass. Add one adapter per front-end appliance per back-end portal.
Automatic configuration for a QuantaStor back end
Where both ends are QuantaStor and in the same grid, all of the above can be done in one operation. Select the back-end volumes, select the front-end appliances to receive them, and QuantaStor creates the host entries, the volume assignments and the software adapters itself.
The dialog is explicit about where it stops: "Automates connectivity of iSCSI storage between systems for the setup of tiered highly-available storage pools. First select the volumes to be mapped to the designated front-end systems, then select all the front-end systems which will receive the volumes. Pool creation and cluster setup is not automatic." So it gets you as far as the LUNs being visible on both front-end appliances; the pool, the site cluster and the HA group are still yours to create.
qs sw-controller-autoconfig --system-list=qs-node1,qs-node2 --volume-list=backend-vol-1,backend-vol-2
qs sw-controller-autoconfig takes --flags=dry-run-only, which reports what it would create without creating it. Use that first on a grid where the host entries may already exist.
Fibre Channel
For an FC back end, create the Host entries with the WWPNs of the front-end FC ports that will face the back end, and assign the back-end volumes to them. There is no software adapter for FC -- the LUNs appear once zoning and assignment allow it, and qs disk-scan or a rescan of the controller picks them up.
Two things about FC are worth knowing before you cable anything.
QuantaStor presents FC target LUNs only from QLogic adapters. Target mode is implemented against the QLogic qla2xxx driver, and QuantaStor enables target mode automatically on ports that support it. A back-end QuantaStor appliance therefore needs QLogic FC cards to serve LUNs at all; the front end, which is only an initiator on the back-end fabric, does not. qs fc-remote-port-list shows which remote WWNs the appliance's FC target ports can actually see, which is the quickest way to tell whether zoning and cabling are right before you go looking for LUNs.
A card that must both serve and initiate may need dual mode set explicitly. A back-end appliance whose FC ports both present LUNs and initiate to something else has to have initiator and target mode active on the same port. qs-util enabledualmode writes qlini_mode=dual into /etc/modprobe.d/qla2xxx.conf for both the qla2xxx and qla2xxx_scst module names, and qs-util disabledualmode writes qlini_mode=exclusive to return the port to target-only. Either takes effect on the next boot. Read the port's current state from the kernel with cat /sys/class/fc_host/host<n>/device/scsi_host/host<n>/active_mode, which prints Initiator, Target when both are live. Dual-mode FC configuration covers doing this by hand without a reboot.
Direct point-to-point cabling between front end and back end avoids the cost of a switch in a small configuration. Where you do use a fabric, use a topology that gives you two independent paths from each front-end appliance to each back-end port, and zone it so that both front-end appliances can reach every LUN. A single-path fabric will build a working pool and then fail the first time a switch is serviced.
Managing a software adapter
Once an adapter exists, its own context menu carries the operations you need while troubleshooting:
| Item | Command | Purpose |
|---|---|---|
| Scan for Targets | qs sw-controller-scan | Re-run discovery against the portal |
| Target Login | qs sw-controller-target-login | Log in to one or more discovered targets |
| Target Logout | qs sw-controller-target-logout | Log out, which removes the LUNs from this appliance |
| Remove Software Adapter | qs sw-controller-remove | Delete the adapter entirely |
| Auto-config Software iSCSI Adapters | qs sw-controller-autoconfig | Offered on iSCSI adapters only |
qs sw-controller-list lists the adapters, qs sw-controller-target-list the targets they have discovered, and qs sw-disk-session-list the live sessions. Target Logout takes the LUNs away from the appliance, so do not run it against an adapter carrying a pool that is imported there.
Building the HA pool
From the point where both front-end appliances can see the LUNs, the build is the same as the JBOD-attached one and is documented once, there:
- First boot. Install the current release on every appliance, apply a licence key to each one (each key is unique to its appliance), set a static address rather than leaving DHCP in place, and set the DNS and NTP servers. See Storage System.
- Grid. Both front-end appliances -- and the back-end appliances, where they are QuantaStor -- join one storage grid. See Grid Configuration.
- Site cluster. Create it across the two front-end appliances only; back-end data nodes are left out. Each ring needs the same network present on both appliances -- ring 1 on the first network, ring 2 on a second, separate one -- so that a single switch cannot take the heartbeat with it. See Site Cluster Setup.
- Storage pool. Create it on one front-end appliance from the physical disks that have arrived from the back end. QuantaStor analyses them and stripes across back-end sources where it can, so that the loss of one back-end node does not stop the pool. The HA-specific device rules are under Before you build the HA group; the pool itself is covered on Storage Pools.
- HA group. Create it for the pool. If any device is not reachable from both appliances the create refuses and names the devices. See Creating the storage pool HA group and What the create checks, and what it rejects.
- HA virtual interface. Every client must reach the pool through this address rather than through an appliance's own. The parent port you choose must exist under the same name on the failover appliance -- the interface attaches to the port of that name there -- so name your ports consistently across the pair. See The HA virtual interface and Cluster VIFs.
- Activate, then test failover in both directions before the cluster carries production data. See Failover.
The one check that behaves differently with FC
Enabling high availability on a pool switches its Fibre Channel presentation into ALUA mode, and that drops every active FC client session. QuantaStor refuses the HA group create rather than doing it silently:
Enabling high-availability on Storage Pool '<pool>' will drop all active FC sessions as the pool will switch to FC ALUA communication mode. Use the 'force' option to enable this or disconnect all active sessions before enabling FC ALUA mode.
This only bites when a pool has already been serving FC clients before it was made highly available, which is a common order of events when an existing appliance is being converted into an HA pair. Either disconnect the FC clients first, or pass --flags=force to qs ha-group-create and accept the session drop. Confirm the resulting ALUA state with qs-util alua.
Provisioning to clients
Provisioning on the front end is ordinary QuantaStor work, with one rule that is not optional:
- Create Network Shares for SMB and NFS, and Storage Volumes for iSCSI, FC and NVMe-oF, on the HA pool.
- Create a Host entry for each client that needs block storage, carrying its IQN, NQN or WWPN, and assign volumes to it.
- Point every client at the pool's HA virtual interface, not at an appliance address. Block clients are protected from getting this wrong, because iSCSI access is restricted to the HA virtual interfaces. NFS and SMB clients are not, and a share mounted over a management address works perfectly until the first failover. Unexpected power-off of an appliance covers how to find clients that have done this.
Troubleshooting
Recovery procedures for a running cluster are on Scale-up HA Storage Pool Troubleshooting, which opens with a symptom-to-procedure table. Failed media, a lost back-end node, a lost front-end appliance, a pool that will not start, split-brain, boot media failure and reinstall, and ransomware are all covered there, and none of it depends on how the shared storage is attached.
A few diagnostics are specific to a SAN back end.
LUNs missing on one front-end appliance. The two appliances must see the same devices, and qs-util devicemap on each of them is the direct comparison -- the /dev/sdX letters will differ and that is expected, but the /dev/disk/by-id paths and the serial numbers must match. FC and iSCSI attached storage varies its sdX assignment far more than a SAS enclosure does, because arrays differ in how they answer bus probes. Check the adapter's session state with qs sw-controller-list and re-run Scan for Targets, and confirm the back end has the front-end appliance's IQN, NQN or WWPN in a Host entry with every volume assigned to it. See Disks not visible on both appliances for the full checklist.
Fencing not working. qs-iofence devstatus reports the reservation on every device. A LUN that reports NOT-SUPPORTED does not implement persistent reservations, which for a back-end array is a hardware or firmware limitation rather than a configuration error -- that array cannot host a highly-available pool. See qs-iofence devstatus.
FC diagnostics. qs-util alua prints the ALUA state of each FC target LUN, qs-util listfcclients lists the connected FC clients, and qs-util issuelip issues a loop initialization primitive on every FC port to force rediscovery. That last one disrupts FC traffic while it runs; do not use it on a cluster serving production FC clients.
iSCSI initiator diagnostics. qs-util iscsiiqn prints the appliance's own initiator name, qs-util iscsidiscover <ip> discovers targets at an address, and qs-util iscsirelogin <ip> logs back in to previously established targets. Prefer the software adapter operations above for anything QuantaStor is managing; these are for confirming that the network and the target are answering at all.
Command line reference
| Command | Purpose |
|---|---|
| qs sw-controller-add | Add an iSCSI or NVMe-oF software adapter |
| qs sw-controller-autoconfig | Create the hosts, assignments and adapters for a QuantaStor back end in one step |
| qs sw-controller-list | List the software adapters |
| qs sw-controller-get | Detail for one adapter |
| qs sw-controller-scan | Re-run target discovery |
| qs sw-controller-target-list | List discovered targets |
| qs sw-controller-target-login | Log in to targets |
| qs sw-controller-target-logout | Log out of targets |
| qs sw-controller-remove | Remove an adapter |
| qs sw-disk-session-list | List live adapter sessions |
| qs host-add | Create a host entry on the back end |
| qs host-initiator-add | Add an IQN, NQN or WWPN to a host entry |
| qs fc-remote-port-list | List the remote FC WWNs visible to the appliance's FC target ports |
| qs network-port-modify | Address, MTU, and iSCSI or NVMe-oF access per port |
| qs disk-scan | Rescan for newly presented devices |
| qs disk-list | List physical disks with the pool each belongs to |
The HA group and virtual interface commands are on HA Cluster Setup (JBODs).
Related pages
- HA Cluster Setup (JBODs) -- the same design with a shared SAS enclosure, and the authority for everything above the shared storage
- Scale-up HA Storage Pool Troubleshooting -- recovery procedures for every scale-up HA failure scenario
- Site Cluster Setup -- heartbeat rings, cluster membership, standby and maintenance mode
- Cluster VIFs -- the virtual interface dialog and every use case
- Storage Pools -- creating and managing the pool itself
- Storage Volumes -- volumes on the back-end data nodes and on the front-end pool
- Physical Disks/Devices -- verifying that the presented LUNs arrived on both appliances
- Multipath Configuration -- multipath naming and path counts for fabric-attached devices
- Hardware Controllers & Enclosures -- where software adapters are created and managed
- Dual-mode FC configuration -- putting a QLogic FC port into initiator and target mode at once
- Network Ports -- port addressing, MTU and per-port protocol access
- Grid Configuration -- building the storage grid
Also worth watching: QuantaStor 6: Deployment of Highly Available (HA) Storage Pool on Seagate Exos (19:02).
Verified against QuantaStor 6.9.0.