Qadmin: /* Is there a single point of failure in a QuantaStor HA pair? */

2026-10-27T20:25:32Z

Is there a single point of failure in a QuantaStor HA pair?

← Older revision		Revision as of 20:25, 27 October 2026
Line 105:		Line 105:
	=== Is there a single point of failure in a QuantaStor HA pair? ===		=== Is there a single point of failure in a QuantaStor HA pair? ===

	~~Not~~ when it is built as described here. Head nodes are paired in an HA group, disks are dual-pathed, heartbeat runs over two independent rings, client networks are bonded across switches, and the pool can be laid out to survive an enclosure loss. I/O fencing keeps two nodes from importing the same pool.		No, not when properly configured as it is built and described here. Head nodes are paired in an HA group, disks are dual-pathed, heartbeat runs over two independent rings, client networks are bonded across switches, and the pool can be laid out to survive an enclosure loss. I/O fencing along with additional guards keeps two nodes from importing the same pool.

	=== How long does a failover take? ===		=== How long does a failover take? ===

Qadmin: /* The failover sequence */

2026-10-27T20:23:41Z

The failover sequence

← Older revision		Revision as of 20:23, 27 October 2026
Line 49:		Line 49:
	# '''Trigger.''' The partner node loses heartbeat with the active node, a port link-state or client connectivity policy fires on the active node, or an administrator requests a move.		# '''Trigger.''' The partner node loses heartbeat with the active node, a port link-state or client connectivity policy fires on the active node, or an administrator requests a move.
	# '''Health checks.''' For a planned move the dialog can review failover health status on both sides: whether the source can export the pool cleanly and whether the target can receive it.		# '''Health checks.''' For a planned move the dialog can review failover health status on both sides: whether the source can export the pool cleanly and whether the target can receive it.
			# '''Move the VIFs.''' The pool's HA VIFs come up on the matching ports on the new owner, so iSCSI, FC, NFS and SMB clients reconnect to the same addresses.
	# '''Export from the current owner.''' If the current owner is still running, it stops serving the pool, takes down the pool's VIFs and exports the pool. The export timeout limits how long this step may take.		# '''Export from the current owner.''' If the current owner is still running, it stops serving the pool, takes down the pool's VIFs and exports the pool. The export timeout limits how long this step may take.
	# '''Take ownership of the devices.''' The receiving node checks the I/O fencing state of the pool's devices and places its own SCSI reservations on them. If the export did not finish within the timeout, or the old owner is down, the receiving node takes these locks preemptively. A node that was preempted while still running needs a reboot to clear its state.		# '''Take ownership of the devices.''' The receiving node checks the I/O fencing state of the pool's devices and places its own SCSI reservations on them. If the export did not finish within the timeout, or the old owner is down, the receiving node takes these locks preemptively. A node that was preempted while still running needs a reboot to clear its state.
	# '''Import and bring services online.''' The receiving node imports the pool and brings its storage volumes and network shares back online.		# '''Import and bring services online.''' The receiving node imports the pool and brings its storage volumes and network shares back online.
	~~# '''Move the VIFs.''' The pool's HA VIFs come up on the matching ports on the new owner, so iSCSI, FC, NFS and SMB clients reconnect to the same addresses.~~		# '''Settle.''' The settle time keeps the group from bouncing the pool again before the new owner has stabilized.
	# '''Settle.''' The settle time keeps the group from bouncing the pool again before the new owner has ~~stabilised~~.

	Fencing is what makes the sequence safe. Because the pool's devices carry the owner's reservations, a node that has lost contact with the cluster cannot keep writing to a pool that its partner has already taken over.		Fencing is what makes the sequence safe. Because the pool's devices carry the owner's reservations, a node that has lost contact with the cluster cannot keep writing to a pool that its partner has already taken over.

Qadmin: osn-seo-utilities: ha-failover-sequence-overview @ cdc16019f12c (approved in the portal)

2026-10-27T20:15:04Z

osn-seo-utilities: ha-failover-sequence-overview @ cdc16019f12c (approved in the portal)

New page

''By Steve Umbehocker, CTO, OSNexus · Updated October 27, 2026''

A QuantaStor high-availability (HA) storage pools are OpenZFS based and required shared storage. Share back-end storage for HA pools can be delivered to the cluster in many ways including via SAS JBODs, SAS JBOFs, NVMe-oF JBOFs, FC/iSCSI SANs and via internal dual-port NVMe within specialized Storage Bridge Bay (SBB) dual-node servers which contain a two server nodes and a share NVMe backplane. The failover sequence is the ordered set of steps QuantaStor runs to move that pool, its volumes, its shares and its client IP addresses from one head node to another, whether the move is planned or forced by a failure and this page summarizes that sequence and how the architecture covers controller failure, path failure, network failure and enclosure failure, so you can check that no single component takes the storage offline.

== Why it matters ==

Every IT department needs to have a solid understanding of what happens when something breaks. For our scale-up HA cluster design we cover each layer separately:

* '''Head node (controller) failure:''' another node must take over the pool automatically.
* '''Path failure:''' a failed cable, HBA or enclosure I/O module must not take disks away from the active node.
* '''Network failure:''' clients must reach the pool on the same IP address after a takeover, and a node that has lost its client networks should hand the pool off.
* '''Enclosure (shelf) failure:''' the pool layout must tolerate the loss of a whole shelf where the design calls for it.
* '''Split-brain:''' two nodes must never import the same pool at the same time.

QuantaStor answers each one with a specific mechanism, summarised in the table below and described in the rest of this guide.

{| class="wikitable"
! Failure !! What protects the data path !! Result
|-
| Head node crash or power loss || Site cluster heartbeat plus an active HA group || Secondary node imports the pool and takes over its virtual IPs
|-
| Loss of client-facing network ports || Ethernet and FC port link-state policies || Preemptive failover to the partner node
|-
| Clients unreachable from the active node || Optional client connectivity checks || Preemptive failover when all, or a majority, of the listed hosts stop responding
|-
| Single SAS cable, HBA or I/O module || Dual-path cabling with multipath IO || I/O continues on the surviving path; no failover needed
|-
| Heartbeat network failure || Two independent cluster rings on separate networks || Cluster communication continues on the other ring
|-
| Whole enclosure || Enclosure-redundant pool layout || Pool stays online in a degraded state
|-
| Both nodes trying to own the pool || I/O fencing with SCSI reservations on the pool devices || Only one node can hold and import the pool
|}

== How it works ==

Three building blocks take part in every failover.

'''The site cluster.''' The nodes that can own a pool are members of a site cluster, which carries heartbeat and cluster-management traffic over one or more cluster rings. The recommended design uses two independent rings on separate networks, so losing one switch or one subnet doesn't look like a node failure. See [[Site Cluster Setup]] for creating the cluster and adding rings. An HA group that has no associated site cluster can still be moved by hand, but it cannot fail over automatically when a system fails.

'''The HA failover group.''' A [[Create Storage Pool High-Availability Group|Storage Pool High-Availability Group]] ties one pool to a primary node and a secondary node, with an optional tertiary node. It holds the failover policies: the Ethernet and FC port link-state policies, the optional client connectivity checks, the export timeout and a settle time that acts as a cooldown before another failover can be triggered. Automatic failover runs only while the group is activated; see [[Set Failover Group Activation Mode]].

'''HA virtual interfaces.''' Clients never connect to a node's own address. They connect to [[Storage Pool HA Failover Interface Create|HA virtual interfaces]] (HA VIFs) that belong to the pool. Each VIF is bound to a parent port by name, for example <code><nowiki>eth2</nowiki></code>, and on failover it comes up on the port with the same name on the node that now owns the pool. You can also restrict SMB and NFS access to the networks the pool's VIFs are on.

=== The failover sequence ===

Whether it starts from a failure or from an administrator, a failover runs in this order:

# '''Trigger.''' The partner node loses heartbeat with the active node, a port link-state or client connectivity policy fires on the active node, or an administrator requests a move.
# '''Health checks.''' For a planned move the dialog can review failover health status on both sides: whether the source can export the pool cleanly and whether the target can receive it.
# '''Export from the current owner.''' If the current owner is still running, it stops serving the pool, takes down the pool's VIFs and exports the pool. The export timeout limits how long this step may take.
# '''Take ownership of the devices.''' The receiving node checks the I/O fencing state of the pool's devices and places its own SCSI reservations on them. If the export did not finish within the timeout, or the old owner is down, the receiving node takes these locks preemptively. A node that was preempted while still running needs a reboot to clear its state.
# '''Import and bring services online.''' The receiving node imports the pool and brings its storage volumes and network shares back online.
# '''Move the VIFs.''' The pool's HA VIFs come up on the matching ports on the new owner, so iSCSI, FC, NFS and SMB clients reconnect to the same addresses.
# '''Settle.''' The settle time keeps the group from bouncing the pool again before the new owner has stabilised.

Fencing is what makes the sequence safe. Because the pool's devices carry the owner's reservations, a node that has lost contact with the cluster cannot keep writing to a pool that its partner has already taken over.

A failover can take 30 seconds or more on larger configurations. Size client-side timeouts to ride through that window. iSCSI and FC initiators should use multipathing, and Windows hosts should follow [[Windows MPIO for Failover Clusters and Hyper-V]].

== Design and sizing ==

'''Controllers and back-end paths.''' For a shared-JBOD design, cable every enclosure to both head nodes and give each node two paths to every disk, through two HBAs or two enclosure I/O modules. [[Multipath IO Configuration]] covers the host side. With dual paths, losing one cable, HBA port or I/O module is handled by multipathing on the active node, and the pool stays where it is. The full build is in [[HA Cluster Setup (JBODs)]]. If the back end is FC or iSCSI LUNs from an external array, the same model applies; see [[HA Cluster Setup (external SAN)]]. Enabling HA on a pool exported over FC switches its FC targets to ALUA mode, which drops active FC sessions once, so do it in a maintenance window.

'''Enclosure (shelf) failure.''' Dual paths protect against a failed component inside a shelf. Surviving the loss of the entire shelf depends on the pool layout. Each RAID group (vdev) must have its disks spread across enclosures so that losing one enclosure takes out no more disks per group than its parity or mirroring can absorb. The storage pool grid reports this as '''Enclosure Redundancy''', with values such as Verified, Partial or Not Redundant, so you can confirm the layout before you go live. [[Hardware Controllers & Enclosures]] shows enclosure and controller health.

'''Networks.''' Bond the client-facing and heartbeat ports across two switches; see [[Changing Network Bonding Mode]]. Put the two cluster rings on separate networks, for example 192.0.2.0/24 for ring 0 and 198.51.100.0/24 for ring 1, and place client VIFs on their own network, such as 203.0.113.0/24.

'''Clustered client reservations.''' Windows Server failover clusters and Hyper-V use SCSI-3 persistent reservations from the client side. With [[Clustered SCSI-3 Persistent Reservations]] enabled on the HA group, those client reservations are shared across the HA cluster and survive a pool failover. If you disable the feature, any reservation a client holds is lost at the next failover.

== Setting it up ==

The reference pages above walk through each dialog. The overall order is: create the site cluster with two rings, create the pool on the shared devices, create the HA group, add HA VIFs, set the failover policies, then activate the group. From the CLI the same sequence looks like this (node names, pool name and addresses are examples):

<pre>qs site-cluster-create --name=site1 --system-ip-list=192.0.2.11,192.0.2.12
qs ha-group-create --pool=pool1 --name=ha-group1 --sys-primary=qs-node-a --sys-secondary=qs-node-b --port-linkstate-policy=all-linkdown --fc-port-linkstate-policy=all-linkdown
qs ha-interface-create --ha-group=ha-group1 --parent-port=eth2 --ip-address=203.0.113.50
qs ha-group-activate --ha-group=ha-group1</pre>

The Connectivity tab of the HA group dialog is where you decide what counts as a failure on the network side. Link-state policies can fail over when all, any or a majority of the pool's ports go link-down, or ignore link state. Client connectivity checks ping a list of client IP addresses and trigger a preemptive failover when all or a majority of them stop responding.

[[File:Guide-ha-failover-sequence-overview-ha-failover-connectivity-policies.png|frame|center|The Connectivity tab of the HA group dialog: the Ethernet and FC Port Link State policies, the SMB and NFS HA VIF access options, and the client connectivity check list that can trigger a preemptive failover]]

== Operating and testing ==

Test failover before production and after any hardware change. A planned failover uses the same sequence as an automatic one, so it is the most realistic test available. Run it from the web UI ([[Storage Pool Ha Failover Group Failover]]) or from the CLI:

<pre>qs ha-group-check-health-status --pool-list=pool1
qs ha-group-failover --ha-group=ha-group1 --storage-system=qs-node-b --import-health-checks=true --export-health-checks=true</pre>

[[File:Guide-ha-failover-sequence-overview-ha-failover-manual-failover.png|frame|center|The Execute High-Availability Pool Failover dialog: check the HA Group Status, the From System and To System fields, and the warning that larger configurations can take 30 seconds or more]]

A test plan for an acceptance or RFP response usually covers:

* A planned failover and failback while clients run I/O.
* Powering off the active node to confirm unattended takeover.
* Pulling one SAS cable or I/O module to confirm multipathing keeps the pool in place.
* Unplugging one heartbeat network to confirm the second ring keeps the cluster intact.
* Pulling the client-facing links on the active node to confirm the link-state policy fails the pool over.

The manual failover dialog has a force option that completes the move even if the pool doesn't export cleanly. Use it only when the source node is known to be unhealthy, because that node will need a reboot afterwards. Configure [[Call-home / Alerting]] so failover events reach your team, and keep [[Scale-up HA Storage Pool Troubleshooting]] at hand for recovery steps.

== FAQ ==

=== Is there a single point of failure in a QuantaStor HA pair? ===

Not when it is built as described here. Head nodes are paired in an HA group, disks are dual-pathed, heartbeat runs over two independent rings, client networks are bonded across switches, and the pool can be laid out to survive an enclosure loss. I/O fencing keeps two nodes from importing the same pool.

=== How long does a failover take? ===

It depends on the size of the configuration. The product warns that larger configurations can take 30 seconds or more, so set client-side timeouts and multipath settings to ride through that window. Measure it on your own hardware with a planned failover before going live.

=== What happens if the failed node comes back? ===

It does not reclaim the pool on its own. The surviving node holds the reservations on the pool devices, and the settle time stops the group from moving the pool again straight away. You fail the pool back with a planned failover once the node is healthy.

=== Can a network problem trigger failover even if the node is still running? ===

Yes. The Ethernet and FC port link-state policies and the optional client connectivity checks can move the pool to the partner node preemptively when the active node can no longer serve clients, even though it is still running.

=== Do Windows and Hyper-V clusters keep their reservations through a failover? ===

Yes, when clustered SCSI-3 persistent reservation support is enabled on the HA group. The reservations are shared across the HA cluster, so the new owner presents the same reservation state to the clients.

----
''Part of the [[Guides:Index|QuantaStor Guides]] series. For reference documentation, see the [[Main Page|QuantaStor documentation]].''

[[Category:Guides]]

Guides:HA Failover Sequence Overview for QuantaStor Storage Pools - Revision history

Qadmin: /* Is there a single point of failure in a QuantaStor HA pair? */

Qadmin: /* The failover sequence */

Qadmin: osn-seo-utilities: ha-failover-sequence-overview @ cdc16019f12c (approved in the portal)