HA Cluster Setup (JBODs) - Revision history

Qadmin: Add 'What determines failover time': assigned-only SCSI targets, concurrent multi-pool moves, no wait on a silent donor, deferred late devices, busy-share handling, encrypted fail-back (DLM/cluster-PR hardening work, QSTOR-12343 and follow-ups)

2026-09-28T20:30:32Z

Add 'What determines failover time': assigned-only SCSI targets, concurrent multi-pool moves, no wait on a silent donor, deferred late devices, busy-share handling, encrypted fail-back (DLM/cluster-PR hardening work, QSTOR-12343 and follow-ups)

@@ Line 320: / Line 320: @@
 # ALUA state is updated and an FC LIP is issued so Fibre Channel clients rescan their paths.
 # SMB and NFS configuration, share namespaces and firewall rules are regenerated for the pool on its new host.
 === What clients see ===

Qadmin: Correct the HA group PR setting: label is Clustered SCSI3-PR Reservations, Enabled/Disabled, default Disabled (was documented as Auto/Enabled/Disabled defaulting to Enabled); add Allow Reboot / Lock Recovery; move detail to Clustered SCSI-3 Persistent Reservations

2026-09-28T16:32:31Z

Correct the HA group PR setting: label is Clustered SCSI3-PR Reservations, Enabled/Disabled, default Disabled (was documented as Auto/Enabled/Disabled defaulting to Enabled); add Allow Reboot / Lock Recovery; move detail to Clustered SCSI-3 Persistent Reservations

← Older revision		Revision as of 16:32, 28 September 2026
Line 189:		Line 189:
	* '''Secondary System''' -- the appliance that will take the pool. Only members of the same site cluster as the primary are offered.		* '''Secondary System''' -- the appliance that will take the pool. Only members of the same site cluster as the primary are offered.
	* '''Tertiary System''' -- an optional third failover target, gated behind its own checkbox. The checkbox stays greyed out unless the site cluster has at least three members, which is why it is disabled on a two-appliance cluster.		* '''Tertiary System''' -- an optional third failover target, gated behind its own checkbox. The checkbox stays greyed out unless the site cluster has at least three members, which is why it is disabled on a two-appliance cluster.
	* '''~~Enable~~ SCSI3-PR ~~Distributed Locking~~''' -- ~~'''Auto''',~~ '''Enabled''' or '''Disabled''', defaulting to '''~~Enabled~~''' for a new group. ~~See~~ [[#Distributed locking for clustered clients\|Distributed locking for clustered clients]] below.		* '''Clustered SCSI3-PR Reservations''' -- '''Enabled''' or '''Disabled''', defaulting to '''Disabled''' for a new group. Turn it on for Windows failover clusters; see [[#Distributed locking for clustered clients\|Distributed locking for clustered clients]] below.
			* '''Allow Reboot / Lock Recovery''' -- unticked by default. Lets an appliance that has been fenced away from a pool restart itself to recover; see [[Clustered SCSI-3 Persistent Reservations#Allow Reboot / Lock Recovery\|Allow Reboot / Lock Recovery]].
	* '''Force''' -- unticked by default. Ticking it relaxes the device connectivity check from "every pool device is visible on the target" to "a majority of them are". It also allows the group to be created while FC sessions are active, which enabling HA would otherwise drop as the pool switches to FC ALUA mode. Leave it off for a first build: a device that is not visible on the secondary is a cabling fault to fix, not a check to bypass.		* '''Force''' -- unticked by default. Ticking it relaxes the device connectivity check from "every pool device is visible on the target" to "a majority of them are". It also allows the group to be created while FC sessions are active, which enabling HA would otherwise drop as the pool switches to FC ALUA mode. Leave it off for a first build: a device that is not visible on the secondary is a cabling fault to fix, not a check to bypass.

Line 355:		Line 356:
	=== Distributed locking for clustered clients ===		=== Distributed locking for clustered clients ===

	'''~~Enable~~ SCSI3-PR ~~Distributed Locking~~''' on the HA group is ~~a different thing~~ from the fencing above~~, and~~ it ~~is easy to conflate them. Fencing is QuantaStor arbitrating which appliance owns the disks. Distributed locking is about~~ the reservations '''clients''' place on ~~QuantaStor~~'s storage volumes~~: with it enabled, those client-side reservations are shared~~ across ~~every appliance in~~ the HA ~~cluster through the distributed lock manager~~, so ~~a reservation a client holds survives a failover instead of being lost with the appliance that recorded it.~~		'''Clustered SCSI3-PR Reservations''' on the HA group is separate from the fencing above: it shares the reservations '''clients''' place on the pool's storage volumes across the HA group, so Windows Server Failover Clustering, Hyper-V Cluster Shared Volumes and SQL Server Failover Cluster Instances keep their reservations through a failover. It defaults to Disabled; see [[Clustered SCSI-3 Persistent Reservations]].

	~~Turn it on for~~ Windows Server Failover Clustering, ~~including~~ Hyper-V ~~cluster shared volumes~~ and SQL Server failover ~~cluster instances, which depend on persistent reservations to arbitrate between cluster nodes~~. ~~QuantaStor configures the DLM on each appliance itself; there is nothing~~ to ~~install.~~

	~~The setting has three values. '''Enabled''' and '''~~Disabled''' are explicit. '''Auto''' means "keep doing whatever this pool is already doing", which is what a group created before this option existed, or by an automation client that does not send the field, will report -- Auto is not the same as off. Turning it off on a running pool is disruptive: reservations stop being shared, so any reservation a client is holding is lost at the next failover, and the dialog asks you to confirm that.

	== Maintenance ==		== Maintenance ==

Qadmin: Replace the TroubleshootingHAStoragePools transclusion with a link to the new Scale-up HA Storage Pool Troubleshooting page; keep the two build-specific troubleshooting sections (QSTOR-12352)

2026-09-04T05:19:12Z

Replace the TroubleshootingHAStoragePools transclusion with a link to the new Scale-up HA Storage Pool Troubleshooting page; keep the two build-specific troubleshooting sections (QSTOR-12352)

← Older revision		Revision as of 05:19, 4 September 2026
Line 27:		Line 27:
	\| [[#Maintenance\|Maintenance]] \|\| Standby and maintenance mode		\| [[#Maintenance\|Maintenance]] \|\| Standby and maintenance mode
	\|-		\|-
	\| [[#Troubleshooting\|Troubleshooting]] \|\| A pool that will not import, a group that will not form		\| [[#Troubleshooting\|Troubleshooting]] \|\| A pool that will not import, a group that will not form, and where the recovery procedures live
	\|-		\|-
	\| [[#Command line reference\|Command line reference]] \|\| Every HA group command		\| [[#Command line reference\|Command line reference]] \|\| Every HA group command
Line 372:		Line 372:

	== Troubleshooting ==		== Troubleshooting ==

			This section covers the two problems specific to building a JBOD-attached cluster. Recovery from a failure in a cluster that is already running -- failed media, a lost enclosure, a lost appliance, a pool that will not start, split-brain, boot media loss, ransomware -- is on '''[[Scale-up HA Storage Pool Troubleshooting]]''', which has a symptom-to-procedure table at the top.

	=== An HA group that will not form ===		=== An HA group that will not form ===
Line 389:		Line 391:
	If the appliance that had the pool needs a reboot, it will say so. An unclean export -- one that overran the export timeout, or a pull of the SAS cables -- leaves in-flight writes in the I/O stack and a pool state that cannot be cleared any other way. Reboot it, let it rejoin the grid, and check its state detail in '''Properties''' before failing anything back to it.		If the appliance that had the pool needs a reboot, it will say so. An unclean export -- one that overran the export timeout, or a pull of the SAS cables -- leaves in-flight writes in the I/O stack and a pool state that cannot be cleared any other way. Reboot it, let it rejoin the grid, and check its state detail in '''Properties''' before failing anything back to it.

	~~The recovery procedures for hardware failures -- device replacement, enclosure power loss, split-brain, boot media loss -- are collected below.~~		=== Recovery from a failure ===

	{{~~Template:TroubleshootingHAStoragePools~~}}		Every recovery procedure for a running cluster is on '''[[Scale-up HA Storage Pool Troubleshooting]]''': device replacement and hot-spare policy, more failed devices than the layout tolerates, the wrong drive pulled, pool I/O failure, disconnected SAS cables, devices not visible on both appliances, enclosure controller failure, enclosure and appliance power loss, multiple enclosure failures at different times, ungraceful reboot, re-integrating a repaired appliance, top-of-rack switch loss, a pool reported missing or corrupted, an HA pool that will not start, split-brain, boot media failure and reinstall, ransomware, and the {{Code\|1=qs-iofence}} and {{Code\|1=qs-util}} diagnostics. Those procedures apply whether the shared storage is a JBOD or a SAN, so they are not repeated here.

	== Command line reference ==		== Command line reference ==
Line 429:		Line 431:
	== Related pages ==		== Related pages ==

			* [[Scale-up HA Storage Pool Troubleshooting]] -- recovery procedures for every scale-up HA failure scenario
	* [[HA Cluster Setup (external SAN)]] -- the same design with SAN-attached storage behind it		* [[HA Cluster Setup (external SAN)]] -- the same design with SAN-attached storage behind it
	* [[Site Cluster Setup]] -- heartbeat rings, cluster membership, maintenance mode		* [[Site Cluster Setup]] -- heartbeat rings, cluster membership, maintenance mode

Qadmin: Rewrite from the product: verified HA group and failover dialogs, HA VIF naming, I/O fencing and SCSI-3 PR, measured failover timings, corrected failover sequence, troubleshooting; site cluster and pool creation now linked rather than transcluded (QSTOR-12352)

2026-09-04T03:53:35Z

Rewrite from the product: verified HA group and failover dialogs, HA VIF naming, I/O fencing and SCSI-3 PR, measured failover timings, corrected failover sequence, troubleshooting; site cluster and pool creation now linked rather than transcluded (QSTOR-12352)

Show changes

Qadmin: Remove version-era references: dropped Category:QuantaStor6; removed 1 QuantaStor 5 video link(s) -- all customers run QuantaStor 6 or newer (wiki-docs rule 13, QSTOR-12352)

2026-09-03T03:15:13Z

Remove version-era references: dropped Category:QuantaStor6; removed 1 QuantaStor 5 video link(s) -- all customers run QuantaStor 6 or newer (wiki-docs rule 13, QSTOR-12352)

← Older revision		Revision as of 03:15, 3 September 2026
Line 32:		Line 32:



	~~''' OSNEXUS Videos '''~~
	* [[Image:youtube_icon.png\|50px\|link=https://www.youtube.com/watch?v=cofeLchc8JU]] [https://www.youtube.com/watch?v=cofeLchc8JU Covers QuantaStor 5 Storage Pool and High-availability Setup. (8:43)]

	== High Availability Shared JBOD/ZFS based SAN/NAS Storage Layout ==		== High Availability Shared JBOD/ZFS based SAN/NAS Storage Layout ==
Line 159:		Line 156:

	[[Category:QuantaStor Guide]]		[[Category:QuantaStor Guide]]
	~~[[Category:QuantaStor6]]~~

Qadmin at 07:01, 30 September 2025

2025-09-30T07:01:56Z

← Older revision		Revision as of 07:01, 30 September 2025
Line 1:		Line 1:
	[[~~index.php?title=~~Category:~~Admin guide~~]]		[[Category:admin_guide]]
	__TOC__		__TOC__
	<!-- Removed Overview header, replaced with Introduction and reduced to H2 for TOC alignment/navigation -->		<!-- Removed Overview header, replaced with Introduction and reduced to H2 for TOC alignment/navigation -->

Qadmin: /* Cabling Diagrams / Guidelines */ Add new cabling diagrams for 9-12 JBODs in scale-up

2025-06-02T21:27:56Z

Cabling Diagrams / Guidelines: Add new cabling diagrams for 9-12 JBODs in scale-up

← Older revision		Revision as of 21:27, 2 June 2025
Line 89:		Line 89:
	[[File:jbod2x6.png\|QuantaStor HA cluster connectivity to 4x Disk Chassis via 2x HBAs (JBODs)\|800px\|alt=QuantaStor HA cluster connectivity to 6x Disk Chassis via 2x HBAs (JBODs)]]		[[File:jbod2x6.png\|QuantaStor HA cluster connectivity to 4x Disk Chassis via 2x HBAs (JBODs)\|800px\|alt=QuantaStor HA cluster connectivity to 6x Disk Chassis via 2x HBAs (JBODs)]]

	==== 2x ~~Quantastor~~ Servers with 7x Disk Chassis ====		==== 2x QuantaStor Servers with 7x Disk Chassis ====
	[[File:Jbod2x7.png\|alt=QuantaStor HA cluster connectivity to 7x Disk Chassis via 2x HBAs (JBODs)\|800x800px\|QuantaStor HA cluster connectivity to 4x Disk Chassis via 2x HBAs (JBODs)]]		[[File:Jbod2x7.png\|alt=QuantaStor HA cluster connectivity to 7x Disk Chassis via 2x HBAs (JBODs)\|800x800px\|QuantaStor HA cluster connectivity to 4x Disk Chassis via 2x HBAs (JBODs)]]

	==== 2x ~~Quantastor~~ Servers with 8x Disk Chassis ====		==== 2x QuantaStor Servers with 8x Disk Chassis ====
	[[File:Jbod2x8.png\|alt=QuantaStor HA cluster connectivity to 8x Disk Chassis via 2x HBAs (JBODs)\|800x800px\|QuantaStor HA cluster connectivity to 4x Disk Chassis via 2x HBAs (JBODs)]]		[[File:Jbod2x8.png\|alt=QuantaStor HA cluster connectivity to 8x Disk Chassis via 2x HBAs (JBODs)\|800x800px\|QuantaStor HA cluster connectivity to 4x Disk Chassis via 2x HBAs (JBODs)]]

			==== 2x QuantaStor Servers with 9x Disk Chassis ====
			[[File:OSN-WiringDiagram2x9.png\|alt=QuantaStor HA cluster connectivity to 9x Disk Chassis via 3x HBAs (JBODs)\|800x800px\|QuantaStor HA cluster connectivity to 4x Disk Chassis via 2x HBAs (JBODs)]]

			==== 2x QuantaStor Servers with 10x Disk Chassis ====
			[[File:Jbod2x10.png\|alt=QuantaStor HA cluster connectivity to 10x Disk Chassis via 3x HBAs (JBODs)\|800x800px\|QuantaStor HA cluster connectivity to 4x Disk Chassis via 2x HBAs (JBODs)]]

			==== 2x QuantaStor Servers with 11x Disk Chassis ====
			[[File:Jbod2x11.png\|alt=QuantaStor HA cluster connectivity to 9x Disk Chassis via 3x HBAs (JBODs)\|800x800px\|QuantaStor HA cluster connectivity to 4x Disk Chassis via 2x HBAs (JBODs)]]

			==== 2x QuantaStor Servers with 12x Disk Chassis ====
			[[File:Jbod2x12.png\|alt=QuantaStor HA cluster connectivity to 9x Disk Chassis via 3x HBAs (JBODs)\|800x800px\|QuantaStor HA cluster connectivity to 4x Disk Chassis via 2x HBAs (JBODs)]]

	=== Scale-out JBOD Cabling Diagrams ===		=== Scale-out JBOD Cabling Diagrams ===
Line 102:		Line 114:
	==== 1x QuantaStor Server with 2x Expansion Chassis ====		==== 1x QuantaStor Server with 2x Expansion Chassis ====
	[[File:Jbod1x2.png\|alt=QuantaStor HA cluster connectivity to 8x Disk Chassis via 2x HBAs (JBODs)\|481x481px\|QuantaStor HA cluster connectivity to 4x Disk Chassis via 2x HBAs (JBODs)]]		[[File:Jbod1x2.png\|alt=QuantaStor HA cluster connectivity to 8x Disk Chassis via 2x HBAs (JBODs)\|481x481px\|QuantaStor HA cluster connectivity to 4x Disk Chassis via 2x HBAs (JBODs)]]

			==== 1x QuantaStor Server with 3x Expansion Chassis ====
			[[File:Jbod1x3.png\|alt=QuantaStor HA cluster connectivity to 8x Disk Chassis via 2x HBAs (JBODs)\|397x397px\|QuantaStor HA cluster connectivity to 4x Disk Chassis via 2x HBAs (JBODs)]]

	== Setup Process ==		== Setup Process ==

Qadmin: /* Important Cabling Rules to Follow */ Added exception to cabling rules for scale out

2025-05-30T20:58:52Z

Important Cabling Rules to Follow: Added exception to cabling rules for scale out

← Older revision		Revision as of 20:58, 30 May 2025
Line 63:		Line 63:
	* Cascading JBODs should be avoided to reduce complexity.		* Cascading JBODs should be avoided to reduce complexity.

	=== Important Cabling Rules to Follow ===		=== Important Cabling Rules to Follow (Scale-up only) ===

	* Rule #1: Never connect the same HBA to the same disk enclosure twice.		* Rule #1: Never connect the same HBA to the same disk enclosure twice.

Qadmin: /* Important Cabling Rules to Follow */ Added exception to double connection rule for 9 or more JBODs

2025-05-30T20:54:08Z

Important Cabling Rules to Follow: Added exception to double connection rule for 9 or more JBODs

← Older revision		Revision as of 20:54, 30 May 2025
Line 67:		Line 67:
	* Rule #1: Never connect the same HBA to the same disk enclosure twice.		* Rule #1: Never connect the same HBA to the same disk enclosure twice.
	* Rule #2: Never connect the same SAS expander to the same HBA twice. (see Rule #1)		* Rule #2: Never connect the same SAS expander to the same HBA twice. (see Rule #1)
	* Rule #3: Every server must be connected to each Disk Enclosure twice (but not with the same HBA, see Rule #1)		* Rule #3: Every server (excluding those connected to 9 or more enclosures) must be connected to each Disk Enclosure twice (but not with the same HBA, see Rule #1)

	=== Scale-up SAS JBOD Cabling Diagrams ===		=== Scale-up SAS JBOD Cabling Diagrams ===

Qadmin: /* SAS JBOD Cabling Diagrams */ Reformat headings and subheadings, add 1x1 and 1x2 Ceph cabling diagrams

2025-05-30T20:43:59Z

SAS JBOD Cabling Diagrams: Reformat headings and subheadings, add 1x1 and 1x2 Ceph cabling diagrams

← Older revision		Revision as of 20:43, 30 May 2025
Line 69:		Line 69:
	* Rule #3: Every server must be connected to each Disk Enclosure twice (but not with the same HBA, see Rule #1)		* Rule #3: Every server must be connected to each Disk Enclosure twice (but not with the same HBA, see Rule #1)

	=== SAS JBOD Cabling Diagrams ===		=== Scale-up SAS JBOD Cabling Diagrams ===

	=== 2x QuantaStor Servers with 1x Disk Chassis ===		==== 2x QuantaStor Servers with 1x Disk Chassis ====
	[[File:jbod2x1.png\|QuantaStor HA cluster connectivity to 1x Disk Chassis (JBODs)\|800px]]		[[File:jbod2x1.png\|QuantaStor HA cluster connectivity to 1x Disk Chassis (JBODs)\|800px]]

	=== 2x QuantaStor Servers with 2x Disk Chassis ===		==== 2x QuantaStor Servers with 2x Disk Chassis ====
	[[File:jbod2x2.png\|QuantaStor HA cluster connectivity to 2x Disk Chassis (JBODs)\|800px]]		[[File:jbod2x2.png\|QuantaStor HA cluster connectivity to 2x Disk Chassis (JBODs)\|800px]]

	=== 2x QuantaStor Servers with 3x Disk Chassis ===		==== 2x QuantaStor Servers with 3x Disk Chassis ====
	[[File:jbod2x3.png\|QuantaStor HA cluster connectivity to 4x Disk Chassis via 2x HBAs (JBODs)\|800px]]\		[[File:jbod2x3.png\|QuantaStor HA cluster connectivity to 4x Disk Chassis via 2x HBAs (JBODs)\|800px]]\

	=== 2x QuantaStor Servers with 4x Disk Chassis ===		==== 2x QuantaStor Servers with 4x Disk Chassis ====
	[[File:jbod2x4.png\|QuantaStor HA cluster connectivity to 4x Disk Chassis via 2x HBAs (JBODs)\|800px]]		[[File:jbod2x4.png\|QuantaStor HA cluster connectivity to 4x Disk Chassis via 2x HBAs (JBODs)\|800px]]

	=== 2x QuantaStor Servers with 5x Disk Chassis ===		==== 2x QuantaStor Servers with 5x Disk Chassis ====
	[[File:jbod2x5.png\|QuantaStor HA cluster connectivity to 4x Disk Chassis via 2x HBAs (JBODs)\|800px\|alt=QuantaStor HA cluster connectivity to 5x Disk Chassis via 2x HBAs (JBODs)]]		[[File:jbod2x5.png\|QuantaStor HA cluster connectivity to 4x Disk Chassis via 2x HBAs (JBODs)\|800px\|alt=QuantaStor HA cluster connectivity to 5x Disk Chassis via 2x HBAs (JBODs)]]

	=== 2x QuantaStor Servers with 6x Disk Chassis ===		==== 2x QuantaStor Servers with 6x Disk Chassis ====
	[[File:jbod2x6.png\|QuantaStor HA cluster connectivity to 4x Disk Chassis via 2x HBAs (JBODs)\|800px\|alt=QuantaStor HA cluster connectivity to 6x Disk Chassis via 2x HBAs (JBODs)]]		[[File:jbod2x6.png\|QuantaStor HA cluster connectivity to 4x Disk Chassis via 2x HBAs (JBODs)\|800px\|alt=QuantaStor HA cluster connectivity to 6x Disk Chassis via 2x HBAs (JBODs)]]

	=== 2x Quantastor Servers with 7x Disk Chassis ===		==== 2x Quantastor Servers with 7x Disk Chassis ====
	[[File:Jbod2x7.png\|alt=QuantaStor HA cluster connectivity to 7x Disk Chassis via 2x HBAs (JBODs)\|800x800px\|QuantaStor HA cluster connectivity to 4x Disk Chassis via 2x HBAs (JBODs)]]		[[File:Jbod2x7.png\|alt=QuantaStor HA cluster connectivity to 7x Disk Chassis via 2x HBAs (JBODs)\|800x800px\|QuantaStor HA cluster connectivity to 4x Disk Chassis via 2x HBAs (JBODs)]]

	=== 2x Quantastor Servers with 8x Disk Chassis ===		==== 2x Quantastor Servers with 8x Disk Chassis ====
	[[File:Jbod2x8.png\|alt=QuantaStor HA cluster connectivity to 8x Disk Chassis via 2x HBAs (JBODs)\|800x800px\|QuantaStor HA cluster connectivity to 4x Disk Chassis via 2x HBAs (JBODs)]]		[[File:Jbod2x8.png\|alt=QuantaStor HA cluster connectivity to 8x Disk Chassis via 2x HBAs (JBODs)\|800x800px\|QuantaStor HA cluster connectivity to 4x Disk Chassis via 2x HBAs (JBODs)]]

			=== Scale-out JBOD Cabling Diagrams ===

			==== 1x QuantaStor Server with 1x Expansion Chassis ====
			[[File:Jbod1x1.png\|alt=QuantaStor HA cluster connectivity to 2x Expansion Chassis via 1x HBAs (JBODs)\|481x481px\|QuantaStor HA cluster connectivity to 4x Disk Chassis via 2x HBAs (JBODs)]]

			==== 1x QuantaStor Server with 2x Expansion Chassis ====
			[[File:Jbod1x2.png\|alt=QuantaStor HA cluster connectivity to 8x Disk Chassis via 2x HBAs (JBODs)\|481x481px\|QuantaStor HA cluster connectivity to 4x Disk Chassis via 2x HBAs (JBODs)]]

	== Setup Process ==		== Setup Process ==