HA Cluster Setup (JBODs)
A scale-up high-availability cluster is a pair (or trio) of QuantaStor appliances cabled to the same SAS JBOD, running a ZFS storage pool that either appliance can import. If the appliance hosting the pool fails, the pool, its virtual IP address, and every share and volume on it move to a surviving appliance automatically. This page covers the JBOD-attached build: hardware, cabling, the HA group and its virtual interface, failover, and I/O fencing. For the same design with a third-party SAN behind it instead of a JBOD, see HA Cluster Setup (external SAN).
| Section | Purpose |
|---|---|
| What a scale-up HA cluster is | The design, and when to choose it over scale-out |
| Hardware requirements | HBAs, media, and what "shared" has to mean |
| Cabling | Rules and per-enclosure-count diagrams |
| Before you build the HA group | Grid, site cluster, network, and the pool |
| Creating the storage pool HA group | The dialog, every field, and what the create checks reject |
| The HA virtual interface | The address clients use, and how it is named |
| Activating and deactivating the group | Turning automatic failover on and off |
| Failover | Deliberate and triggered, what happens in what order, how long it takes |
| I/O fencing and SCSI-3 persistent reservations | How QuantaStor guarantees one owner, and how to read the reservations |
| Maintenance | Standby and maintenance mode |
| Troubleshooting | A pool that will not import, a group that will not form, and where the recovery procedures live |
| Command line reference | Every HA group command |
What a scale-up HA cluster is
QuantaStor offers two independent routes to high availability, and they are not variations on one another.
Scale-up puts the storage outside the appliances, in a shared SAS JBOD, and makes the appliances interchangeable front ends for it. A ZFS storage pool is built from disks in the enclosure and imported by exactly one appliance at a time. Redundancy against disk failure comes from the pool's RAID layout; redundancy against appliance failure comes from the second appliance being able to import the same pool. This is the right choice when you want the capacity efficiency and the feature set of ZFS -- compression, snapshots, RAIDZ2 and RAIDZ3 -- with node-level availability on top, and when the whole cluster fits in one rack.
Scale-out replicates data across independent appliances with no shared enclosure, and is documented under Scale-out Block Setup (ceph), Scale-out File Setup (ceph) and Scale-out Object Setup (ceph). Choose it when you need to grow past what one JBOD chain can hold, or when you cannot rely on a shared enclosure.
The rest of this page is about the scale-up case.
Three QuantaStor objects make it work, and they are created in this order:
- A site cluster -- the corosync and pacemaker layer that carries the heartbeat between appliances and decides when one has gone away. One site cluster serves any number of pools. See Site Cluster Setup.
- A storage pool HA group -- the object that ties one ZFS pool to the appliances allowed to import it, holds the failover policies, and is the thing you act on to fail a pool over.
- One or more HA virtual interfaces -- the IP addresses clients connect to, which move with the pool.
Hardware requirements
Every disk in the pool must be in the shared enclosure. That includes cache, log and hot-spare devices. A pool that draws even one device from a disk internal to one of the appliances cannot be made highly available, because the surviving appliance would be unable to import it. QuantaStor enforces this: creating the HA group verifies that every one of the pool's devices is visible on the secondary appliance, and refuses the create if any are missing.
Connect the enclosure with HBAs, not RAID controllers. Scale-up pools are HBA-only. ZFS needs unmediated access to the drives, and I/O fencing needs to issue SCSI-3 persistent reservation commands straight to them, which a RAID controller presenting virtual drives does not permit. Use Broadcom 9300, 9400 or 9500 series HBAs or the OEM equivalents; HPE servers attached to HPE enclosures should use the HPE OEM HBAs. A hardware RAID controller is still the right choice for the mirrored boot devices, which are internal to each appliance and are not part of any pool.
Pool media must be dual-ported and must support persistent reservations. In practice that means dual-port SAS, NL-SAS or dual-port NVMe. SATA drives are not supported for scale-up HA pools and QuantaStor rejects them by name at HA group creation:
Storage Pool contains one or more SATA disk devices including '<device>'. Storage Pool High-Availability feature requires that all disks devices be dual-port SAS, dual-port NVMe, or FC devices.
Single-ported media in a shared enclosure is a subtler failure, because the pool will build and run. QuantaStor detects it and raises a multipath configuration problem alert against the pool; Multipath Configuration documents that alert and why it is a correctness problem rather than a warning to acknowledge.
What "dual-connected" actually requires. Each disk's two SAS ports are wired to different places, and what you wire them to decides what survives. In the common two-appliance, one-JBOD build, each disk sends one port to each appliance: the sharing is at the appliance level, each appliance sees a single path to each disk, and a disk showing one path in multipath -ll is expected rather than a fault. Path-level redundancy on top of that needs a second HBA connection from each appliance into a second SAS expander in the enclosure, which is why enclosures with dual expanders are the ones to buy. Multipath Configuration covers how QuantaStor names and stacks these devices.
Minimum configuration
- 2x QuantaStor appliances acting as storage pool controllers
- 1x or more SAS JBOD cabled to both appliances
- 2x to 100x dual-port SAS HDDs or SSDs for pool storage, all of them in the shared enclosure
- 1x hardware RAID controller per appliance for mirrored boot devices
- 2x boot devices per appliance; 480GB or larger SSD or NVMe media is recommended, because a boot device that fills up is the most common cause of local database corruption
Storage bridge bay systems
A cluster-in-a-box or SBB (storage bridge bay) chassis holds two hot-swap servers and a JBOD in a single 2U unit, and QuantaStor supports the SuperMicro SBB models. Everything on this page applies unchanged; the cabling is internal to the chassis. Contact your OSNEXUS reseller or sdr@osnexus.com for hardware options.
Cabling
Incorrect cabling is the single most common cause of I/O fencing and performance problems in a scale-up cluster, and the symptoms rarely point at the cables. Check the wiring against the diagrams before building anything.
- The same rules apply to every enclosure model from every manufacturer.
- All Dell, HPE, WD/HGST, Seagate and Lenovo JBOD units ship with dual SAS expanders.
- SuperMicro sells single-expander enclosures (E1/E1C models) which should not be used for HA. Buy the dual-expander models, which carry E2/E2C in the model number.
- Some enclosures label their ports IN and OUT but accept either; check the vendor documentation.
- Avoid cascading JBODs. It adds failure modes for no benefit.
- Keep SAS cables within the 5 metre standard limit, or use optical SAS cables for longer runs.
Three rules govern which HBA port goes where:
- Never connect the same HBA to the same disk enclosure twice.
- Never connect the same SAS expander to the same HBA twice.
- Every appliance connected to eight or fewer enclosures must be connected to each enclosure twice, using different HBAs.
Unused HBA ports are fine and can be used to attach more enclosures later.
Scale-up cabling diagrams
2x appliances, 1x disk chassis
2x appliances, 2x disk chassis
2x appliances, 3x disk chassis
2x appliances, 4x disk chassis
2x appliances, 5x disk chassis
2x appliances, 6x disk chassis
2x appliances, 7x disk chassis
2x appliances, 8x disk chassis
2x appliances, 9x disk chassis
2x appliances, 10x disk chassis
2x appliances, 11x disk chassis
2x appliances, 12x disk chassis
Single-appliance expansion cabling
These layouts attach expansion chassis to one appliance. They are not HA configurations -- there is no second appliance to fail over to -- and are included here because the cabling rules are the same.
1x appliance, 1x expansion chassis
1x appliance, 2x expansion chassis
1x appliance, 3x expansion chassis
Before you build the HA group
Four things have to be in place first, in this order.
Licence both appliances. Each appliance needs its own unique Gold, Platinum or Cloud Edition key. See License Management.
Put both appliances in one storage grid. An HA group can only span grid members. Grid creation takes under a minute; see Grid Configuration.
Configure the networks. Set static addresses on every port you intend to use, set DNS and NTP, and keep the heartbeat networks separate from client traffic. The site cluster requires that the port carrying a heartbeat ring has the same interface name on every appliance, so plan the naming before you cable. Network Ports covers port configuration; Site Cluster Setup covers the heartbeat requirements in detail.
Create the site cluster, with two heartbeat rings. The site cluster is what detects an appliance going away. A single-ring cluster is fragile enough that ordinary network maintenance can trigger a failover, so configure a second ring on a separate subnet -- a direct crossover cable between the two appliances is ideal, because it survives a top-of-rack switch outage. The full procedure is on Site Cluster Setup; do not build the HA group until the site cluster reports every member online.
Then create the pool. A scale-up HA pool is created exactly like any other ZFS pool -- see Storage Pools -- with one constraint: select only disks from the shared enclosure. Before you create it, confirm in the Physical Disks section that the same disks appear with the same SCSI IDs and serial numbers on both appliances. That shared, identical naming is what makes the pool importable on either side, and if it is missing the HA group create will fail. Create the pool on the appliance you intend to be its primary, although that is a convention rather than a requirement.
For parity layouts, use at least double parity. RAIDZ2 or RAIDZ3 leaves the pool with error-correction capability while a failed device is being replaced, and -- unlike RAID10 -- cannot start on half its devices, which removes even the theoretical possibility of a split-brain.
Creating the storage pool HA group
The storage pool HA group associates one pool with the appliances allowed to import it, carries the failover policies, and owns the pool's virtual interfaces. It is the object you activate, deactivate and fail over.
Existing groups are listed on the Storage Pool HA Groups tab in the centre pane, which shows each group's state, the appliance currently hosting it, and its connectivity and link-state policies.
General tab
- Name -- prefilled as the pool name with
-ha-groupappended, so a pool namedpool-2producespool-2-ha-group. Keep the pool name in it; the group is what you will be looking for during an incident. - Description -- optional.
- Storage Pool -- the pool to protect. Only ZFS pools are listed, and a pool that already belongs to a group is rejected on OK.
- Export Timeout (seconds) -- default 50. This is how long the appliance losing the pool is given to export it during a failover. If it overruns, the acquiring appliance stops waiting and takes the devices preemptively, which gets the pool back into service but leaves the exporting appliance needing a reboot to clear its I/O stack. Raise it for a pool with many volumes and shares, where a clean export legitimately takes longer than a smaller one.
- Primary System -- read-only, and set to the appliance the selected pool is currently imported on.
- Secondary System -- the appliance that will take the pool. Only members of the same site cluster as the primary are offered.
- Tertiary System -- an optional third failover target, gated behind its own checkbox. The checkbox stays greyed out unless the site cluster has at least three members, which is why it is disabled on a two-appliance cluster.
- Enable SCSI3-PR Distributed Locking -- Auto, Enabled or Disabled, defaulting to Enabled for a new group. See Distributed locking for clustered clients below.
- Force -- unticked by default. Ticking it relaxes the device connectivity check from "every pool device is visible on the target" to "a majority of them are". It also allows the group to be created while FC sessions are active, which enabling HA would otherwise drop as the pool switches to FC ALUA mode. Leave it off for a first build: a device that is not visible on the secondary is a cabling fault to fix, not a check to bypass.
Connectivity tab
This tab decides when QuantaStor should fail a pool over preemptively -- that is, while the hosting appliance is still alive but has lost the connectivity that makes it useful.
- Enforce SMB HA VIF Access and Enforce NFS HA VIF Access -- off by default. Each restricts SMB and NFS access for the pool's shares to the networks the pool has virtual interfaces on. Turn them on when you need to be certain clients cannot reach the shares through an appliance-local address, because a client that connects that way keeps working right up until a failover and then does not come back.
- Ethernet Port Link State Policy -- default Failover if ALL Ports are Link-Down. The other settings are any port down, a majority of ports down, or disabled. "All ports down" is the conservative default; a single flapping port should not move a pool.
- FC Port Link State Policy -- the same choices, applied to Fibre Channel target ports, and with the same default. It only applies to pools that actually export volumes over FC.
- Enable Client Connectivity Checks -- off by default. Ticking it enables the two radio buttons and the Verify Connectivity button below, all of which are inert until it is on. With it enabled, QuantaStor pings the client addresses you list and fails the pool over when they stop answering, on either the ALL specified IPs are unresponsive or MAJORITY of specified IPs are unresponsive policy. The addresses must be genuine remote clients: an address belonging to a system in the same grid is rejected, since pinging your own grid tells you nothing about client reachability.
- Verify Connectivity -- pings the listed addresses now and reports how many answered, so you can confirm the list before relying on it.
From the command line
qs ha-group-create --pool=pool-2 --sys-secondary=qs-node2 \
--export-timeout=50 --enable-cluster-pr=enabled
Use qs ha-group-create to create the group and qs ha-group-modify to change it afterwards. The CLI takes the same settings under the names --client-connectivity-check-policy, --port-linkstate-policy, --fc-port-linkstate-policy, --verify-client-ips, --enforce-smb-allowed and --enforce-nfs-allowed. Two differences from the dialog are worth knowing: --sys-primary is settable rather than read-only, and --enable-cluster-pr defaults to auto where the dialog defaults a new group to enabled.
What the create checks, and what it rejects
The create is a long sequence of preconditions, and each one produces a distinct message. Reading the message saves guessing:
- The pool is not ZFS, or already belongs to another HA group.
- The chosen appliances are not all in the same site cluster. A group can still be created when no appliance is in a site cluster at all -- a clusterless group -- but it cannot carry a virtual interface and cannot fail over automatically, so it is only useful as a way to move a pool by hand.
- Either appliance has I/O fencing disabled:
Storage system '<name>' has I/O fencing disabled, this must be re-enabled before an HA group may be created. - The two appliances' system UUIDs share their first six hex digits. The SCSI reservation key is built from those digits, so identical prefixes would make it impossible to tell which appliance holds the pool.
- Any pool device is a SATA disk.
- Any pool device is not visible on the secondary appliance:
Unable to verify access to '<n>' devices (<scsi ids>) on secondary storage system for storage pool '<pool>'. - The devices do not support the persistent reservations that fencing requires -- QuantaStor registers a key on each one as part of the create, and fails if it cannot.
- Any grid member is running a service version older than the minimum the HA group code requires.
The HA virtual interface
Every client must reach the pool through the HA virtual interface, not through an appliance's own address. The virtual interface is a cluster resource that pacemaker moves with the pool, so a client pointed at it finds its data wherever the pool has landed. A client pointed at an appliance-local address works perfectly until the first failover and then does not come back, and that is by far the most common cause of "the failover worked but my clients did not recover".
An HA virtual interface is a cluster VIF with the scale-up pool use case, bound to an HA group. Cluster VIFs owns the dialog and the full set of use cases; create it there:
Three things are specific to the scale-up case:
- It needs an IP address of its own, on the client network, not in use anywhere else, and distinct from both appliances' addresses on that network. In practice each HA virtual interface consumes three addresses on its subnet: its own, plus one per appliance on the parent port.
- The parent port name is checked across the whole site cluster, not only on the appliance you selected, because a virtual interface that cannot follow the pool to the secondary appliance defeats the point.
- QuantaStor writes location constraints so the interface is only ever placed on the group's primary, secondary and tertiary appliances.
The interface can also be created from the HA group side, which is what the CLI does:
qs ha-interface-create --ha-group=pool-2-ha-group --parent-port=bond0.100 \
--ip-address=192.168.0.124 --netmask=255.255.0.0 --iscsi-enable=true --nvmeof-enable=true
See qs ha-interface-create, and qs ha-interface-list to check the result. --convert-vif turns an existing appliance-local virtual IP into an HA interface, and the matching --convert-to-vif on qs ha-interface-delete turns it back -- which is the supported way to keep an address alive while the paired appliance is being rebuilt.
How the interface is named
A scale-up HA interface is named for its parent port plus a generated tag, in the form <parent>:<tag>. The tag begins with ha, continues with the leading hex digits of the pool GUID, and ends with an index number. A pool whose GUID starts d786e0b7 on parent port bond0.100 produces the tag had70 and the interface bond0.100:had70. That is how a scale-up interface is told apart from a site VIF, which uses an sv tag instead.
Linux caps an interface name at 15 characters, so the length of the parent port name decides how many GUID digits fit -- and a long parent name can leave no room at all. If the create fails with a name-length error, either rename the parent interface or switch the appliance to eth-prefix naming on the Storage System Modify dialog.
Activating and deactivating the group
A group must be activated before automatic failover will happen. Until then the group exists and can be failed over by hand, but the site cluster will not move the pool on its own.
Activating restores the interface location constraints and places the pool's resource group on the appliance currently hosting it. Deactivating disables the failover policies without deleting anything. Two behaviours are worth knowing before you deactivate:
- A group cannot be activated while its site cluster is in maintenance mode. Exit maintenance mode first.
- While a group is deactivated, a manual failover to a different appliance is refused if the group still has virtual interfaces attached:
Cannot execute HA pool failover to another system while HA Group '<name>' is deactivated and has one or more HA interfaces.Re-activate the group, or remove its interfaces, to move the pool.
Use qs ha-group-activate and qs ha-group-deactivate from the command line.
Failover
Automatic failover
Once the group is activated, the site cluster monitors appliance health across every heartbeat ring, and QuantaStor's own checks watch the things a heartbeat cannot see. A failover is triggered when:
- An appliance stops answering on all heartbeat rings.
- The small write test QuantaStor runs against each pool every few seconds fails to complete. This is what catches a lost SAS path or a failed JBOD I/O controller, neither of which stops the appliance answering its heartbeat.
- Network ports fail according to the Ethernet or FC link-state policy on the group.
- The configured client addresses stop answering, if client connectivity checks are enabled.
The settle time on the group -- 60 seconds by default -- is a cooldown that prevents a second failover from being triggered immediately after one completes.
Deliberate failover
Move a pool by hand to test the cluster, or to empty an appliance before working on it.
On the General tab, HA Failover Group selects the group, Storage Pool and HA Group Status are shown read-only, From System is the appliance currently hosting the pool, and To System is where it is going. The other appliance in the group is preselected. Selecting the appliance the pool is already on is allowed and prompts for confirmation -- see A pool that will not import for why you would.
Advanced Settings carries two preliminary health checks. Import Health Checks is ticked by default and reviews health statuses on the target appliance that would stop it receiving the pool. Export Health Checks is unticked by default and reviews statuses on the source appliance that would stop it exporting cleanly. Either finding a critical status aborts the failover before anything moves, and names the checkbox to clear if you want to proceed anyway.
From the command line, both checks are explicit arguments:
qs ha-group-failover --ha-group=pool-2-ha-group --storage-system=qs-node2 \
--import-health-checks=true --export-health-checks=false
See qs ha-group-failover. The command runs synchronously and returns the group object when the failover completes.
What happens, in order
A failover is driven from the appliance receiving the pool, and runs in this sequence:
- Device information is rescanned and the receiving appliance verifies it can see the pool's devices.
- The pool's ALUA state is moved to standby, if ALUA is in use.
- The HA virtual interfaces are moved to the receiving appliance as a pacemaker resource group. A group with no virtual interfaces still fails over, with a warning; only the pool moves.
- The pool is exported on the appliance that had it, within the export timeout.
- SCSI-3 reservations on the pool's devices are preempted, giving the receiving appliance sole write access. If it cannot take ownership the failover stops here rather than importing on top of another appliance's reservation.
- LUKS devices are opened, for an encrypted pool.
- The pool is imported and activated.
- ALUA state is updated and an FC LIP is issued so Fibre Channel clients rescan their paths.
- SMB and NFS configuration, share namespaces and firewall rules are regenerated for the pool on its new host.
What clients see
The dialog warns that a failover "can take upwards of 30 seconds or more for larger configurations", and that is the right expectation to set. On a small pool -- four SAS disks, 576GB, RAIDZ2, no client load -- a failover measured end to end took 44 to 45 seconds, with the change of ownership visible in qs pool-list about 25 seconds in and the remaining time spent on share, ALUA and firewall reconfiguration. A pool with many volumes and shares takes longer, and the export stage is usually what grows.
For the duration, the pool's storage is unavailable: the virtual interface has moved but the pool behind it has not yet imported. Clients connected through the HA virtual interface see a stall and then recover, which is what iSCSI, NFS and SMB timeouts exist for; clients connected to an appliance-local address see the connection go away and do not recover. Applications with short storage timeouts may need those timeouts raised.
Two operational points follow from that:
- Test failover before the cluster carries production data, in both directions, and confirm the pool lands healthy each time.
- Fail back deliberately. After an appliance is repaired and rejoins, pools do not return on their own. If you run two pools, one on each appliance, you have to move one back by hand.
I/O fencing and SCSI-3 persistent reservations
Fencing is what makes the guarantee that only one appliance can write to the pool at a time, and it is enforced by the drives themselves rather than by the software. It is the scale-up equivalent of the ARP probing documented on High-availability VIF Management: both exist to stop two appliances claiming the same resource, one for addresses and one for disks.
When an appliance imports an HA pool it places a SCSI-3 persistent reservation of type WERO (write exclusive, registrants only) on every device in the pool. The registration key encodes who holds it and what for. It has the form 0xffaaaaaaffbbbbbb, where the ffs are separators, aaaaaa is the first six hex digits of the appliance's system UUID, and bbbbbb is the first six of the storage pool's UUID. On NVMe devices the middle separator is a single f and the appliance portion is truncated to five digits, because NVMe registration keys are shorter.
Read the current state with qs-iofence devstatus, which lists every device, its serial number, the keys registered on it, the key holding the reservation, and the reservation type:
# qs-iofence devstatus /dev/sdm 6XP2F4JX0000B2359Y4A (0xff753100ffd786e0) [0xff753100ffd786e0] <WERO> /dev/sdn 6SE46WWK0000B146RNKP (0xff753100ffd786e0) [0xff753100ffd786e0] <WERO> /dev/sdo PFVVJUYE (0xff753100ffd786e0) [0xff753100ffd786e0] <WERO> /dev/sdp 6SE31S1V0000B130JCEV (0xff753100ffd786e0) [0xff753100ffd786e0] <WERO>
Every device in the pool should carry the same key, and that key's appliance portion should match the appliance the pool is imported on. Run the command on both appliances: they see the same reservations, because the reservation lives on the drive. After a failover the appliance portion changes and the pool portion does not, which is the quickest confirmation that fencing followed the pool rather than being left behind. A device that shows no reservation while the pool is running, or one whose key names the wrong appliance, is a fencing problem -- resolve it by failing the pool over, including to the appliance it is already on.
QuantaStor also surfaces this in the Physical Disks section: a black underline on a device icon means the device is correctly fenced to the appliance running the pool, and a red underline means it is not.
Fencing is why cabling rules matter. If a cabling change ever isolated one enclosure to each appliance, reservations placed by one appliance would not reach the other's disks, which is the only realistic route to a split-brain -- and only for mirrored or 2d+2p and 3d+3p layouts, since a RAIDZ2 or RAIDZ3 pool cannot start on half its devices at all.
Distributed locking for clustered clients
Enable SCSI3-PR Distributed Locking on the HA group is a different thing from the fencing above, and it is easy to conflate them. Fencing is QuantaStor arbitrating which appliance owns the disks. Distributed locking is about the reservations clients place on QuantaStor's storage volumes: with it enabled, those client-side reservations are shared across every appliance in the HA cluster through the distributed lock manager, so a reservation a client holds survives a failover instead of being lost with the appliance that recorded it.
Turn it on for Windows Server Failover Clustering, including Hyper-V cluster shared volumes and SQL Server failover cluster instances, which depend on persistent reservations to arbitrate between cluster nodes. QuantaStor configures the DLM on each appliance itself; there is nothing to install.
The setting has three values. Enabled and Disabled are explicit. Auto means "keep doing whatever this pool is already doing", which is what a group created before this option existed, or by an automation client that does not send the field, will report -- Auto is not the same as off. Turning it off on a running pool is disruptive: reservations stop being shared, so any reservation a client is holding is lost at the next failover, and the dialog asks you to confirm that.
Maintenance
To work on one appliance without dismantling the cluster, put it in standby mode. A standby appliance stays in the site cluster but will not host resources: anything it is running moves to a healthy appliance, and it will not receive a failover. It has an auto-activating form, which returns to active once the appliance is healthy or after a reboot, and a manual form, which stays until you clear it. See Configure Member Standby for the settings and Site Cluster Setup for how standby differs from cluster-wide maintenance mode.
From the command line, qs site-cluster-set-standby-mode takes --standby-mode=active, standby-auto-activate or standby-manual-activate.
Deleting an HA group is the opposite of maintenance and worth stating plainly: it disables automatic failover and removes every virtual interface attached to the group. The pool stays online on whichever appliance is hosting it, but it stops being highly available, and the addresses clients were using disappear.
Troubleshooting
This section covers the two problems specific to building a JBOD-attached cluster. Recovery from a failure in a cluster that is already running -- failed media, a lost enclosure, a lost appliance, a pool that will not start, split-brain, boot media loss, ransomware -- is on Scale-up HA Storage Pool Troubleshooting, which has a symptom-to-procedure table at the top.
An HA group that will not form
Work through What the create checks, and what it rejects above -- the create names the reason it refused, and almost every failure is one of those checks. The two that account for most first builds are a device that is not visible on the secondary appliance, and appliances that are not both in the same site cluster.
For the device visibility case, compare the two appliances directly. qs-util devicemap prints each device with its /dev/disk/by-id path, vendor, model and serial number; run it on both appliances and sort the output. The /dev/sdX letters will differ and that does not matter -- the by-id paths and the serial numbers are what must match. qs disk-list shows the same information with the pool each disk belongs to.
A pool that will not import
Two situations look identical and have different fixes.
The pool is on an appliance but not running. Most often the HA group is deactivated, or has no virtual interfaces, or the interface addresses are in use elsewhere on the network. Check the group's state first. If the group is deliberately deactivated, run a manual failover from the appliance the pool is already on to that same appliance. That is more than starting the pool: it re-runs the fencing sequence and places the virtual interfaces, which a plain pool start does not do.
The enclosures were powered on after the appliances. Nothing will import, because the devices were not present at boot. Power the JBODs on, wait for them to come up, and then run the same failover-to-itself against each affected pool. In the Storage Pools section the tree label tells you which appliance a pool belongs to -- it reads pool-2 (on: qs-node2) -- so you know which appliance to target.
If the appliance that had the pool needs a reboot, it will say so. An unclean export -- one that overran the export timeout, or a pull of the SAS cables -- leaves in-flight writes in the I/O stack and a pool state that cannot be cleared any other way. Reboot it, let it rejoin the grid, and check its state detail in Properties before failing anything back to it.
Recovery from a failure
Every recovery procedure for a running cluster is on Scale-up HA Storage Pool Troubleshooting: device replacement and hot-spare policy, more failed devices than the layout tolerates, the wrong drive pulled, pool I/O failure, disconnected SAS cables, devices not visible on both appliances, enclosure controller failure, enclosure and appliance power loss, multiple enclosure failures at different times, ungraceful reboot, re-integrating a repaired appliance, top-of-rack switch loss, a pool reported missing or corrupted, an HA pool that will not start, split-brain, boot media failure and reinstall, ransomware, and the qs-iofence and qs-util diagnostics. Those procedures apply whether the shared storage is a JBOD or a SAN, so they are not repeated here.
Command line reference
| Command | Purpose |
|---|---|
| qs ha-group-create | Create an HA group for a pool |
| qs ha-group-modify | Change nodes, policies, timeouts or distributed locking |
| qs ha-group-list | List HA groups and their state |
| qs ha-group-get | Full detail for one group, including its interfaces and fenced device serials |
| qs ha-group-activate | Enable automatic failover |
| qs ha-group-deactivate | Disable automatic failover |
| qs ha-group-failover | Move a pool to another appliance |
| qs ha-group-delete | Remove the group and its virtual interfaces |
| qs ha-group-get-health-status | Report stored failover health statuses for a pool |
| qs ha-interface-create | Add an HA virtual interface to a group |
| qs ha-interface-list | List HA virtual interfaces |
| qs ha-interface-get | Detail for one HA virtual interface |
| qs ha-interface-delete | Remove one, optionally converting it to a local virtual IP |
| qs site-cluster-set-standby-mode | Put an appliance into or out of standby |
Related pages
- Scale-up HA Storage Pool Troubleshooting -- recovery procedures for every scale-up HA failure scenario
- HA Cluster Setup (external SAN) -- the same design with SAN-attached storage behind it
- Site Cluster Setup -- heartbeat rings, cluster membership, maintenance mode
- Cluster VIFs -- the virtual interface dialog and every use case
- High-availability VIF Management -- how cluster VIFs are managed and probed
- Configure Member Standby -- standby settings for one appliance
- Multipath Configuration -- multipath device naming, path counts, and the single-ported media alert
- Hardware Controllers & Enclosures -- HBAs, enclosures, slot numbering and drive identify
- Storage Pools -- creating and managing the pool itself
- Physical Disks/Devices -- verifying disk visibility and serial numbers
- Grid Configuration -- building the storage grid
- Network Ports -- port addressing and naming
Verified against QuantaStor 6.9.0.