HA Cluster Setup (external SAN): Difference between revisions

From OSNEXUS Online Documentation Site
Jump to navigation Jump to search
← Older edit
VisualWikitext
mNo edit summary
m Rewrite to house style: verified software-adapter and auto-config paths for iSCSI/NVMe-oF back ends, the FC ALUA session check, per-port protocol access; replace the troubleshooting transclusion with a link to Scale-up HA Storage Pool Troubleshooting (QSTOR-12352)
 
(37 intermediate revisions by the same user not shown)
Line 1: Line 1:
[[Category:admin_guide]]
[[Category:admin_guide]]
__TOC__
__TOC__
<!-- Removed Overview header, replaced with Introduction and reduced to H2 for TOC alignment/navigation -->


== Introduction ==
A SAN-attached scale-up high-availability cluster puts a pair of QuantaStor appliances in front of block storage delivered from somewhere else -- an external array, or a second tier of QuantaStor appliances -- over Fibre Channel, iSCSI or NVMe-oF. The front-end appliances build a ZFS storage pool from the LUNs they are given and serve it to clients over every protocol, with automatic failover between them. This page covers the part that is specific to that back end: presenting the LUNs so that both front-end appliances see them, and the checks that behave differently when the shared storage arrives over a fabric.
QuantaStor's clustered storage pool configurations ensure high-availability (HA) in the event of a node outage or storage connectivity to the active node is lost.  From a hardware perspective a QuantaStor deployment of one or more clustered storage pools requires at least two QuantaStor appliances so that automatic fail-over can occur in the event that the active appliance goes offline.  Another requirement of these configurations is that the disk storage devices for the highly-available pool cannot be in either of the server units.  Rather the storage must come from an external source which can be as simple as using one or more SAS JBOD enclosures or for higher performance configurations could be a separate SAN delivering block storage (LUNs) to the QuantaStor front-end appliances over FC (preferred) or iSCSI. This guide is focused on how to setup QuantaStor with a FC or iSCSI SAN back-end, for information on how to setup with a JBOD SAS enclosure back-end see the [[Clustered_HA_SAN/NAS_with_JBOD_back-end|guide for that here]].


[[File:qs4_clustered_san.png|600px]]
Everything above the shared storage -- the site cluster, the HA group, the virtual interface, failover and I/O fencing -- is identical to the JBOD-attached build and is documented on '''[[HA Cluster Setup (JBODs)]]'''. Recovery from a failure in a running cluster is on '''[[Scale-up HA Storage Pool Troubleshooting]]'''.


=== Setup and Configuration Overview ===
{| class="wikitable"
<!-- Using 'Overview' here instead of 'Summary' will clarify this is not a perform-these-steps section -->
! Section !! Purpose
* Install both QuantaStor Appliances with the most recent release of QuantaStor.
|-
** Apply the QuantaStor Gold, Platinum or Cloud Edition license key to each appliance. Each key must be unique.
| [[#What the SAN-attached design is|What the SAN-attached design is]] || Front-end and back-end roles, and when to choose this over a JBOD
** Create a Grid and join both QuantaStor Appliances to the grid.
|-
| [[#Hardware requirements|Hardware requirements]] || Front-end, back-end and fabric
|-
| [[#Presenting the back-end storage|Presenting the back-end storage]] || Host entries, software adapters, FC target mode
|-
| [[#Building the HA pool|Building the HA pool]] || What is common with the JBOD build, and the one check that is not
|-
| [[#Provisioning to clients|Provisioning to clients]] || Shares, volumes and host assignment on the front end
|-
| [[#Troubleshooting|Troubleshooting]] || SAN-specific diagnostics, and where the recovery procedures live
|-
| [[#Command line reference|Command line reference]] || The back-end connectivity commands
|}


* Configure SAS Hardware and Verify connectivity of SAS cables from HBAs to JBOD
== What the SAN-attached design is ==
** LSI SAS HBA's must be configured with the 'Boot Support' MPT BIOS option configured to 'OS only' mode.
** Verify that the SAS disks installed in both systems appear to both head nodes in the WebUI Hardware Enclosures and Controllers section.


* Create Network and Cluster heartbeat configuration and Verify network connectivity
[[File:qs4_clustered_san.png|thumb|right|322px|Two front-end appliances sharing block storage delivered from a back-end SAN.]]
** At least two NICs are required for a HA Cluster configuration with separate networks


* Create a Storage Pool
A scale-up HA cluster needs at least two appliances, and the pool's disks cannot live inside either of them -- a surviving appliance has to be able to import the same pool. The simplest way to satisfy that is a shared SAS enclosure, which is what [[HA Cluster Setup (JBODs)]] covers. The alternative is to have the block storage arrive over a fabric.
** Use only drives from shared storage


* Create Storage Pool HA Group
In this design the two QuantaStor appliances act as a '''gateway''' to storage that lives elsewhere. They are the '''front-end''' or '''controller''' nodes: they hold the ZFS pool, the HA group, the virtual interfaces, and every client-facing protocol. Whatever supplies the LUNs is the '''back end'''. That can be a third-party array, or it can be QuantaStor appliances configured as '''data''' nodes, each with its own local pools and storage volumes, which is the tiered arrangement the diagrams below describe.
* Create one or more Storage Pool HA Virtual Interfaces
* Activate the Storage Pool
* Test failover


== High Availability Tiered SAN/ZFS based SAN/NAS Storage Layout ==
Choose this over a JBOD when the back-end storage already exists, when the capacity has to sit further away than a SAS cable reaches, or when the back end is doing something a JBOD cannot -- its own RAID, its own tiering, its own scale.


In this configuration the QuantaStor front-end controller appliances are acting as a ''gateway'' to the storage in the SANs on the back-end.  QuantaStor has been tested with NetApp and HP MSA 3rd party SANs as back-end storage as well as with QuantaStor SDS as a back-end SAN.  Please contact support@osnexus.com for the latest HCL for 3rd party SAN support or to get additional SAN support added to the HCL.
Two consequences follow from the storage arriving over a fabric rather than a SAS cable, and both matter:


[[File:qs_clustered_san_minimum_hardware.png|640px]]
* '''Every LUN in the pool must be presented to both front-end appliances.''' This is the same rule as the JBOD build, and QuantaStor enforces it the same way: creating the HA group verifies every pool device on the secondary appliance and refuses if any is missing.
* '''The LUNs must support SCSI-3 persistent reservations.''' I/O fencing is what guarantees only one appliance imports the pool, and it is enforced by the device rather than by QuantaStor. A back end that does not honour persistent reservations cannot host a highly-available pool. See [[HA Cluster Setup (JBODs)#I/O fencing and SCSI-3 persistent reservations|I/O fencing and SCSI-3 persistent reservations]].


=== Minimum Hardware Requirements ===
QuantaStor has been tested with third-party arrays including NetApp and HPE MSA, as well as with QuantaStor itself as the back end. Contact support@osnexus.com for the current hardware compatibility list, or to have an array added to it.


* 2x QuantaStor appliances which will be configured as front-end ''controller nodes''
== Hardware requirements ==
* 2x (or more) QuantaStor appliance configured as back-end ''data nodes'' with SAS or SATA disk
* High-performance SAN (FC/iSCSI) connectivity between front-end ''controller nodes'' and back-end ''data nodes''


== Setup Process ==
[[File:qs_clustered_san_minimum_hardware.png|thumb|right|415px|Minimum layout: two front-end controller nodes, two or more back-end data nodes, and a fabric between them.]]


==== All Appliances ====
* 2x QuantaStor appliances as front-end '''controller nodes'''.
* Login to the QuantaStor web management interface on each appliance
* Back-end storage presenting block LUNs. Where that is QuantaStor, 2x or more appliances as '''data nodes''' with local SAS or SATA disk.
* Add your license keys, one unique key for each appliance
* A high-performance fabric between them -- FC, iSCSI or NVMe-oF.
* Setup static IP addresses on each appliance (DHCP is the default and should only be used to get the appliance initially setup)
* At least two network interfaces per front-end appliance on separate networks, for the two heartbeat rings of the site cluster.
* Right-click on the storage system and set the DNS IP address (eg 8.8.8.8), and your NTP server IP address
* Mirrored boot devices in each appliance, 480GB or larger SSD or NVMe media, on a hardware RAID controller. A boot device that fills up is the most common cause of local database corruption.


==== Back-end Appliances (Data Nodes) ====
Separate the front-end and back-end networks. Client traffic and the traffic between the controller and data nodes have different characteristics and different failure consequences, and mixing them makes both harder to reason about. Where the back end is iSCSI, disable iSCSI access on the management and replication ports so that back-end traffic cannot land on them:
* Setup each back-end ''data node'' appliance as per basic appliance configuration with one or more storage pools, each with one storage volume per pool.
** Ideal pool size is 10 to 20 drives, you may need to create multiple pools per back-end appliance.
** SAS is recommended but enterprise SATA drives can also be used
** HBA(s) or Hardware RAID controller(s) can be used for storage connectivity


[[File:qs4_ha_backend.png|640px]]
<pre style="font-size: smaller">
qs network-port-modify --port=eth0 --iscsi-enable=false
</pre>


==== Front-end Appliances (Controller Nodes) ====
[[QuantaStor CLI Command Reference#network-port-modify|qs network-port-modify]] also carries {{Code|1=--nvmeof-enable}} for NVMe-oF TCP, {{Code|1=--mtu}} for jumbo frames, and {{Code|1=--auto-tune}} for RX/TX buffer sizing. The same settings are in the WUI; see [[Network Ports]].
* Connectivity between the front-end and back-end nodes can be FC or iSCSI


[[File:qs4_ha_frontend.png|1024px]]
{{Navigation|Storage Management &rarr; Storage Systems ''(section)'' &rarr; Network Port ''(select + right-click)'' &rarr; Modify Network Port...}}


===== FC SAN Back-end Configuration =====
=== Back-end data nodes ===
* Create ''Host'' entries, one for each front-end appliance and add the WWPN of each of the FC ports on the front-end appliances which will be used for intercommunication between the front-end and back-end nodes.
* Direct point-to-point physical cabling connections can be made in smaller configurations to avoid the cost of an FC switch.  Here's a guide that can help with some [http://blog.osnexus.com/2012/03/20/understanding-fc-fabric-configuration-5-paragraphs/ advice on FC zone setup] for larger configurations using a back-end fabric.
* If you're using a FC switch you should use a fabric topology that will give you fault tolerance.
* Back-end appliances must use Qlogic QLE 8Gb or 16Gb FC cards as QuantaStor can only present Storage Volumes as FC target LUNs via Qlogic cards.


===== iSCSI SAN Back-end Configuration =====
[[File:qs4_ha_backend.png|thumb|right|436px|A back-end data node: local pools, one storage volume per pool, presented to the front end.]]
* It is highly recommended to separate the networks for the front-end (client communication) vs back-end (communicate between the control and data appliances).
* If the back-end appliances are in the same grid with the front-end appliances, add Host entries with the IQN (or FC WWPN) for each of the front-end appliances.  These Host entries will be used to assign Storage Volumes from the back-end appliances to the front-end appliances so that the storage can be aggregated.
* Assign all the Storage Volumes on the back-end appliances to the Host entries for the front-end appliances. 
* To establish iSCSI connectivity to the Storage Volumes on the back-end nodes, create a ''Software iSCSI Adapter'' by going to the Hardware Controllers & Enclosures section and adding a iSCSI Adapter.  This will take care of logging into and accessing the back-end storage.  Again, back-end storage appliances must assign their Storage Volumes to all the ''Hosts'' for the front-end nodes with their associated iSCSI IQNs.
* Right-click to Modify Network Port of each port of the back-end appliances.  Disable client iSCSI access on the slow 1GbE ports or other (management) ports where iSCSI traffic should not flow.  (If you have 10GbE be sure to disable iSCSI access on the slower 1GbE ports used for management access and/or remote-replication.)


[[File:qs4_iscsi_share.png|1024px]]
Where the back end is QuantaStor, configure each data node as an ordinary appliance:


==== HA Network Setup ====
* Build one or more storage pools from its local disk. A pool of 10 to 20 drives is a good size; use several pools per appliance rather than one very large one.
* Make sure that eth0 is on the same network on both appliances
* Create one storage volume per pool, sized to the pool.
* Make sure that eth1 is on the same but separate network from eth0 on both appliances
* SAS disk is preferable, but enterprise SATA is usable here -- the dual-port requirement applies to the front-end pool's devices, not to the disks behind a back-end array.
* Create the ''Site Cluster'' with Ring 1 on the first network and Ring 2 on the second network, both front-end nodes should be in the ''Site Cluster'', back-end nodes can be left out.  This establishes a redundant (dual ring) heartbeat between the front-end appliances which will be used to detect hardware problems which in turn will trigger a failover of the pool to the passive node.
* HBAs or hardware RAID controllers are both acceptable on a data node.
==== HA Storage Pool Setup ====
* Create a ''Storage Pool'' on the first front-end appliance (ZFS based) using the ''physical disks'' which have arrived from the back-end appliances.
** QuantaStor will automatically analyze the disks from the back-end appliances and stripe across the appliances to ensure proper fault-tolerance across the back-end nodes.
* Create a ''Storage Pool HA Group'' for the pool created in the previous step.  If the storage is not accessible to both appliances, it will block you from creating the group.
* Create a Storage Pool Virtual Interface for the Storage Pool HA Group.  All NFS/iSCSI access to the pool must be through the Virtual Interface IP address to ensure highly-available access to the storage for the clients.
* Enable the Storage Pool HA Group.  Automatic Storage Pool fail-over to the passive node will now occur if the active node is disabled or heartbeat between the nodes is lost.
* Test pool failover, right-click on the Storage Pool HA Group and choose 'Manual Failover' to fail-over the pool to another node.


==== Standard Storage Provisioning ====
See [[Storage Pools]] and [[Storage Volumes]] for the pool and volume work itself, which is not HA-specific.
* Create one or more ''Network Shares'' (CIFS/NFS) and ''Storage Volumes'' (iSCSI/FC)
* Create one or more ''Host'' entries with the iSCSI initiator IQN or FC WWPN of your client hosts/servers that will be accessing block storage.
* Assign Storage Volumes to client ''Host'' entries created in the previous step to enable iSCSI/FC access to Storage Volumes.


=== Diagram of Front-end Appliance Configuration ===
== Presenting the back-end storage ==


[[File:qs_ha_frontend_detail.png|800px]]
[[File:qs4_ha_frontend.png|thumb|right|635px|The front-end appliances import the back-end volumes as physical disks and build one pool across them.]]


=== Diagram of Back-end Appliance Configuration ===
The goal is that both front-end appliances see the same set of LUNs as physical disks. What that takes depends on the transport.


[[File:qs_ha_backend_detail.png|800px]]
Where the back end is QuantaStor, the mechanism is a '''Host''' entry per front-end appliance, carrying that appliance's initiator identifiers, with the back-end storage volumes assigned to those hosts. Get the front-end appliance's iSCSI initiator name from the appliance itself:


=== Diagram of Completed Configuration ===
<pre style="font-size: smaller">
# qs-util iscsiiqn
INFO: iSCSI IQN: iqn.2009-15.com.osnexus:01:7531003303be
</pre>


[[File:qs_ha_frontend_backend_overview.png|800px]]
Then, on the back end, [[QuantaStor CLI Command Reference#host-add|qs host-add]] creates the host and [[QuantaStor CLI Command Reference#host-initiator-add|qs host-initiator-add]] adds an identifier to it -- {{Code|1=--iqn}} takes an iSCSI IQN, an NVMe NQN or an FC WWPN. Assign every back-end volume to every front-end host, so that both front-end appliances see every LUN.


=== iSCSI and NVMe-oF: software adapters ===


{{Template:ClusteredHACommon}}
[[File:Add Software Adapter Web 6.jpg|thumb|right|512px|Adding a software adapter for an iSCSI or NVMe-oF back end.]]


{{Template:ClusteredHAPoolCreation}}
On each front-end appliance, a '''software adapter''' establishes and maintains the connection to the back end. It logs in to the targets and surfaces their LUNs in the '''Physical Disks''' section, where they can be used to build a pool like any other device.


{{Template:StoragePoolHAGroupConfiguration}}
{{Navigation|Storage Management &rarr; Controllers & Enclosures ''(section)'' &rarr; Hardware Controller ''(select + right-click)'' &rarr; Add Software Adapter...}}


{{Template:StoragePoolHAGroupFailover}}
The dialog asks for the '''Portal IP Address''' of the back end and states what it will do: "All LUNs assigned to the initiator IQN/NQN of the system will be imported." Under '''Software Adapter Settings''' the port and CHAP credentials are optional -- the port defaults to 3260 for iSCSI and 4420 for NVMe. A CHAP username without a password, or a password without a username, is rejected; supply both or neither.


{{Template:TroubleshootingHAStoragePools}}
From the command line:


<pre style="font-size: smaller">
qs sw-controller-add --storage-system=qs-node1 --name=backend-1 \
  --ip-address=10.10.20.5 --sw-controller-type=iscsi
</pre>
[[QuantaStor CLI Command Reference#sw-controller-add|qs sw-controller-add]] takes {{Code|1=--sw-controller-type}} of {{Code|1=iscsi}} (the default), {{Code|1=nvme-tcp}} or {{Code|1=nvme-rdma}}, an optional {{Code|1=--port}}, and optional {{Code|1=--chap-user}} and {{Code|1=--chap-pass}}. Add one adapter per front-end appliance per back-end portal.
=== Automatic configuration for a QuantaStor back end ===
Where both ends are QuantaStor and in the same grid, all of the above can be done in one operation. Select the back-end volumes, select the front-end appliances to receive them, and QuantaStor creates the host entries, the volume assignments and the software adapters itself.
{{Navigation|Storage Management &rarr; Controllers & Enclosures ''(section)'' &rarr; Hardware Controller ''(select + right-click)'' &rarr; Auto-config Software iSCSI Adapters...}}
The dialog is explicit about where it stops: "Automates connectivity of iSCSI storage between systems for the setup of tiered highly-available storage pools. First select the volumes to be mapped to the designated front-end systems, then select all the front-end systems which will receive the volumes. Pool creation and cluster setup is not automatic." So it gets you as far as the LUNs being visible on both front-end appliances; the pool, the site cluster and the HA group are still yours to create.
<pre style="font-size: smaller">
qs sw-controller-autoconfig --system-list=qs-node1,qs-node2 --volume-list=backend-vol-1,backend-vol-2
</pre>
[[QuantaStor CLI Command Reference#sw-controller-autoconfig|qs sw-controller-autoconfig]] takes {{Code|1=--flags=dry-run-only}}, which reports what it would create without creating it. Use that first on a grid where the host entries may already exist.
=== Fibre Channel ===
For an FC back end, create the '''Host''' entries with the WWPNs of the front-end FC ports that will face the back end, and assign the back-end volumes to them. There is no software adapter for FC -- the LUNs appear once zoning and assignment allow it, and [[QuantaStor CLI Command Reference#disk-scan|qs disk-scan]] or a rescan of the controller picks them up.
Two things about FC are worth knowing before you cable anything.
'''QuantaStor presents FC target LUNs only from QLogic adapters.''' Target mode is implemented against the QLogic {{Code|1=qla2xxx}} driver, and QuantaStor enables target mode automatically on ports that support it. A back-end QuantaStor appliance therefore needs QLogic FC cards to serve LUNs at all; the front end, which is only an initiator on the back-end fabric, does not. [[QuantaStor CLI Command Reference#fc-remote-port-list|qs fc-remote-port-list]] shows which remote WWNs the appliance's FC target ports can actually see, which is the quickest way to tell whether zoning and cabling are right before you go looking for LUNs.
'''A card that must both serve and initiate may need dual mode set explicitly.''' A back-end appliance whose FC ports both present LUNs and initiate to something else has to have initiator and target mode active on the same port. {{Code|1=qs-util enabledualmode}} writes {{Code|1=qlini_mode=dual}} into {{Code|1=/etc/modprobe.d/qla2xxx.conf}} for both the {{Code|1=qla2xxx}} and {{Code|1=qla2xxx_scst}} module names, and {{Code|1=qs-util disabledualmode}} writes {{Code|1=qlini_mode=exclusive}} to return the port to target-only. Either takes effect on the next boot. Read the port's current state from the kernel with {{Code|1=cat /sys/class/fc_host/host&lt;n&gt;/device/scsi_host/host&lt;n&gt;/active_mode}}, which prints {{Code|1=Initiator, Target}} when both are live. [[Dual-mode FC configuration]] covers doing this by hand without a reboot.
Direct point-to-point cabling between front end and back end avoids the cost of a switch in a small configuration. Where you do use a fabric, use a topology that gives you two independent paths from each front-end appliance to each back-end port, and zone it so that both front-end appliances can reach every LUN. A single-path fabric will build a working pool and then fail the first time a switch is serviced.
=== Managing a software adapter ===
Once an adapter exists, its own context menu carries the operations you need while troubleshooting:
{{Navigation|Storage Management &rarr; Controllers & Enclosures ''(section)'' &rarr; Software Adapter ''(select + right-click)''}}
{| class="wikitable"
! Item !! Command !! Purpose
|-
| Scan for Targets || [[QuantaStor CLI Command Reference#sw-controller-scan|qs sw-controller-scan]] || Re-run discovery against the portal
|-
| Target Login || [[QuantaStor CLI Command Reference#sw-controller-target-login|qs sw-controller-target-login]] || Log in to one or more discovered targets
|-
| Target Logout || [[QuantaStor CLI Command Reference#sw-controller-target-logout|qs sw-controller-target-logout]] || Log out, which removes the LUNs from this appliance
|-
| Remove Software Adapter || [[QuantaStor CLI Command Reference#sw-controller-remove|qs sw-controller-remove]] || Delete the adapter entirely
|-
| Auto-config Software iSCSI Adapters || [[QuantaStor CLI Command Reference#sw-controller-autoconfig|qs sw-controller-autoconfig]] || Offered on iSCSI adapters only
|}
[[QuantaStor CLI Command Reference#sw-controller-list|qs sw-controller-list]] lists the adapters, [[QuantaStor CLI Command Reference#sw-controller-target-list|qs sw-controller-target-list]] the targets they have discovered, and [[QuantaStor CLI Command Reference#sw-disk-session-list|qs sw-disk-session-list]] the live sessions. '''Target Logout''' takes the LUNs away from the appliance, so do not run it against an adapter carrying a pool that is imported there.
== Building the HA pool ==
[[File:qs_ha_frontend_detail.png|thumb|right|743px|Front-end appliance detail: the imported LUNs, the pool built across them, and the client-facing protocols.]]
From the point where both front-end appliances can see the LUNs, the build is the same as the JBOD-attached one and is documented once, there:
# '''First boot.''' Install the current release on every appliance, apply a licence key to each one (each key is unique to its appliance), set a static address rather than leaving DHCP in place, and set the DNS and NTP servers. See [[Storage System]].
# '''Grid.''' Both front-end appliances -- and the back-end appliances, where they are QuantaStor -- join one storage grid. See [[Grid Configuration]].
# '''Site cluster.''' Create it across the two '''front-end''' appliances only; back-end data nodes are left out. Each ring needs the same network present on both appliances -- ring 1 on the first network, ring 2 on a second, separate one -- so that a single switch cannot take the heartbeat with it. See [[Site Cluster Setup]].
# '''Storage pool.''' Create it on one front-end appliance from the physical disks that have arrived from the back end. QuantaStor analyses them and stripes across back-end sources where it can, so that the loss of one back-end node does not stop the pool. The HA-specific device rules are under [[HA Cluster Setup (JBODs)#Before you build the HA group|Before you build the HA group]]; the pool itself is covered on [[Storage Pools]].
# '''HA group.''' Create it for the pool. If any device is not reachable from both appliances the create refuses and names the devices. See [[HA Cluster Setup (JBODs)#Creating the storage pool HA group|Creating the storage pool HA group]] and [[HA Cluster Setup (JBODs)#What the create checks, and what it rejects|What the create checks, and what it rejects]].
# '''HA virtual interface.''' Every client must reach the pool through this address rather than through an appliance's own. The parent port you choose must exist under the same name on the failover appliance -- the interface attaches to the port of that name there -- so name your ports consistently across the pair. See [[HA Cluster Setup (JBODs)#The HA virtual interface|The HA virtual interface]] and [[Cluster VIFs]].
# '''Activate, then test failover in both directions''' before the cluster carries production data. See [[HA Cluster Setup (JBODs)#Failover|Failover]].
=== The one check that behaves differently with FC ===
Enabling high availability on a pool switches its Fibre Channel presentation into ALUA mode, and that drops every active FC client session. QuantaStor refuses the HA group create rather than doing it silently:
{{Code|1=Enabling high-availability on Storage Pool '&lt;pool&gt;' will drop all active FC sessions as the pool will switch to FC ALUA communication mode.  Use the 'force' option to enable this or disconnect all active sessions before enabling FC ALUA mode.}}
This only bites when a pool has already been serving FC clients before it was made highly available, which is a common order of events when an existing appliance is being converted into an HA pair. Either disconnect the FC clients first, or pass {{Code|1=--flags=force}} to [[QuantaStor CLI Command Reference#ha-group-create|qs ha-group-create]] and accept the session drop. Confirm the resulting ALUA state with {{Code|1=qs-util alua}}.
== Provisioning to clients ==
[[File:qs_ha_frontend_backend_overview.png|thumb|right|425px|The completed configuration: clients to the front-end virtual interfaces, front end to back end over the fabric.]]
Provisioning on the front end is ordinary QuantaStor work, with one rule that is not optional:
* Create '''Network Shares''' for SMB and NFS, and '''Storage Volumes''' for iSCSI, FC and NVMe-oF, on the HA pool.
* Create a '''Host''' entry for each client that needs block storage, carrying its IQN, NQN or WWPN, and assign volumes to it.
* '''Point every client at the pool's HA virtual interface, not at an appliance address.''' Block clients are protected from getting this wrong, because iSCSI access is restricted to the HA virtual interfaces. NFS and SMB clients are not, and a share mounted over a management address works perfectly until the first failover. [[Scale-up HA Storage Pool Troubleshooting#Unexpected power-off of an appliance|Unexpected power-off of an appliance]] covers how to find clients that have done this.
== Troubleshooting ==
[[File:qs_ha_backend_detail.png|thumb|right|747px|Back-end data node detail: local pools and the volumes presented to the front end.]]
'''Recovery procedures for a running cluster are on [[Scale-up HA Storage Pool Troubleshooting]]''', which opens with a symptom-to-procedure table. Failed media, a lost back-end node, a lost front-end appliance, a pool that will not start, split-brain, boot media failure and reinstall, and ransomware are all covered there, and none of it depends on how the shared storage is attached.
A few diagnostics are specific to a SAN back end.
'''LUNs missing on one front-end appliance.''' The two appliances must see the same devices, and {{Code|1=qs-util devicemap}} on each of them is the direct comparison -- the {{Code|1=/dev/sdX}} letters will differ and that is expected, but the {{Code|1=/dev/disk/by-id}} paths and the serial numbers must match. FC and iSCSI attached storage varies its {{Code|1=sdX}} assignment far more than a SAS enclosure does, because arrays differ in how they answer bus probes. Check the adapter's session state with {{Code|1=qs sw-controller-list}} and re-run '''Scan for Targets''', and confirm the back end has the front-end appliance's IQN, NQN or WWPN in a Host entry with every volume assigned to it. See [[Scale-up HA Storage Pool Troubleshooting#Disks not visible on both appliances|Disks not visible on both appliances]] for the full checklist.
'''Fencing not working.''' {{Code|1=qs-iofence devstatus}} reports the reservation on every device. A LUN that reports {{Code|1=NOT-SUPPORTED}} does not implement persistent reservations, which for a back-end array is a hardware or firmware limitation rather than a configuration error -- that array cannot host a highly-available pool. See [[Scale-up HA Storage Pool Troubleshooting#qs-iofence devstatus|qs-iofence devstatus]].
'''FC diagnostics.''' {{Code|1=qs-util alua}} prints the ALUA state of each FC target LUN, {{Code|1=qs-util listfcclients}} lists the connected FC clients, and {{Code|1=qs-util issuelip}} issues a loop initialization primitive on every FC port to force rediscovery. That last one disrupts FC traffic while it runs; do not use it on a cluster serving production FC clients.
'''iSCSI initiator diagnostics.''' {{Code|1=qs-util iscsiiqn}} prints the appliance's own initiator name, {{Code|1=qs-util iscsidiscover &lt;ip&gt;}} discovers targets at an address, and {{Code|1=qs-util iscsirelogin &lt;ip&gt;}} logs back in to previously established targets. Prefer the software adapter operations above for anything QuantaStor is managing; these are for confirming that the network and the target are answering at all.
== Command line reference ==
{| class="wikitable"
! Command !! Purpose
|-
| [[QuantaStor CLI Command Reference#sw-controller-add|qs sw-controller-add]] || Add an iSCSI or NVMe-oF software adapter
|-
| [[QuantaStor CLI Command Reference#sw-controller-autoconfig|qs sw-controller-autoconfig]] || Create the hosts, assignments and adapters for a QuantaStor back end in one step
|-
| [[QuantaStor CLI Command Reference#sw-controller-list|qs sw-controller-list]] || List the software adapters
|-
| [[QuantaStor CLI Command Reference#sw-controller-get|qs sw-controller-get]] || Detail for one adapter
|-
| [[QuantaStor CLI Command Reference#sw-controller-scan|qs sw-controller-scan]] || Re-run target discovery
|-
| [[QuantaStor CLI Command Reference#sw-controller-target-list|qs sw-controller-target-list]] || List discovered targets
|-
| [[QuantaStor CLI Command Reference#sw-controller-target-login|qs sw-controller-target-login]] || Log in to targets
|-
| [[QuantaStor CLI Command Reference#sw-controller-target-logout|qs sw-controller-target-logout]] || Log out of targets
|-
| [[QuantaStor CLI Command Reference#sw-controller-remove|qs sw-controller-remove]] || Remove an adapter
|-
| [[QuantaStor CLI Command Reference#sw-disk-session-list|qs sw-disk-session-list]] || List live adapter sessions
|-
| [[QuantaStor CLI Command Reference#host-add|qs host-add]] || Create a host entry on the back end
|-
| [[QuantaStor CLI Command Reference#host-initiator-add|qs host-initiator-add]] || Add an IQN, NQN or WWPN to a host entry
|-
| [[QuantaStor CLI Command Reference#fc-remote-port-list|qs fc-remote-port-list]] || List the remote FC WWNs visible to the appliance's FC target ports
|-
| [[QuantaStor CLI Command Reference#network-port-modify|qs network-port-modify]] || Address, MTU, and iSCSI or NVMe-oF access per port
|-
| [[QuantaStor CLI Command Reference#disk-scan|qs disk-scan]] || Rescan for newly presented devices
|-
| [[QuantaStor CLI Command Reference#disk-list|qs disk-list]] || List physical disks with the pool each belongs to
|}
The HA group and virtual interface commands are on [[HA Cluster Setup (JBODs)#Command line reference|HA Cluster Setup (JBODs)]].
== Related pages ==
* [[HA Cluster Setup (JBODs)]] -- the same design with a shared SAS enclosure, and the authority for everything above the shared storage
* [[Scale-up HA Storage Pool Troubleshooting]] -- recovery procedures for every scale-up HA failure scenario
* [[Site Cluster Setup]] -- heartbeat rings, cluster membership, standby and maintenance mode
* [[Cluster VIFs]] -- the virtual interface dialog and every use case
* [[Storage Pools]] -- creating and managing the pool itself
* [[Storage Volumes]] -- volumes on the back-end data nodes and on the front-end pool
* [[Physical Disks/Devices]] -- verifying that the presented LUNs arrived on both appliances
* [[Multipath Configuration]] -- multipath naming and path counts for fabric-attached devices
* [[Hardware Controllers & Enclosures]] -- where software adapters are created and managed
* [[Dual-mode FC configuration]] -- putting a QLogic FC port into initiator and target mode at once
* [[Network Ports]] -- port addressing, MTU and per-port protocol access
* [[Grid Configuration]] -- building the storage grid
Also worth watching: [https://www.youtube.com/watch?v=WNCEjgboAQI QuantaStor 6: Deployment of Highly Available (HA) Storage Pool on Seagate Exos] (19:02).


[[Category:QuantaStor Guide]]
[[Category:QuantaStor Guide]]
[[Category:QuantaStor4]]
 
----
<small>''Verified against QuantaStor 6.9.0.''</small>

Latest revision as of 05:28, 4 September 2026

A SAN-attached scale-up high-availability cluster puts a pair of QuantaStor appliances in front of block storage delivered from somewhere else -- an external array, or a second tier of QuantaStor appliances -- over Fibre Channel, iSCSI or NVMe-oF. The front-end appliances build a ZFS storage pool from the LUNs they are given and serve it to clients over every protocol, with automatic failover between them. This page covers the part that is specific to that back end: presenting the LUNs so that both front-end appliances see them, and the checks that behave differently when the shared storage arrives over a fabric.

Everything above the shared storage -- the site cluster, the HA group, the virtual interface, failover and I/O fencing -- is identical to the JBOD-attached build and is documented on HA Cluster Setup (JBODs). Recovery from a failure in a running cluster is on Scale-up HA Storage Pool Troubleshooting.

Section Purpose
What the SAN-attached design is Front-end and back-end roles, and when to choose this over a JBOD
Hardware requirements Front-end, back-end and fabric
Presenting the back-end storage Host entries, software adapters, FC target mode
Building the HA pool What is common with the JBOD build, and the one check that is not
Provisioning to clients Shares, volumes and host assignment on the front end
Troubleshooting SAN-specific diagnostics, and where the recovery procedures live
Command line reference The back-end connectivity commands

What the SAN-attached design is

Two front-end appliances sharing block storage delivered from a back-end SAN.

A scale-up HA cluster needs at least two appliances, and the pool's disks cannot live inside either of them -- a surviving appliance has to be able to import the same pool. The simplest way to satisfy that is a shared SAS enclosure, which is what HA Cluster Setup (JBODs) covers. The alternative is to have the block storage arrive over a fabric.

In this design the two QuantaStor appliances act as a gateway to storage that lives elsewhere. They are the front-end or controller nodes: they hold the ZFS pool, the HA group, the virtual interfaces, and every client-facing protocol. Whatever supplies the LUNs is the back end. That can be a third-party array, or it can be QuantaStor appliances configured as data nodes, each with its own local pools and storage volumes, which is the tiered arrangement the diagrams below describe.

Choose this over a JBOD when the back-end storage already exists, when the capacity has to sit further away than a SAS cable reaches, or when the back end is doing something a JBOD cannot -- its own RAID, its own tiering, its own scale.

Two consequences follow from the storage arriving over a fabric rather than a SAS cable, and both matter:

  • Every LUN in the pool must be presented to both front-end appliances. This is the same rule as the JBOD build, and QuantaStor enforces it the same way: creating the HA group verifies every pool device on the secondary appliance and refuses if any is missing.
  • The LUNs must support SCSI-3 persistent reservations. I/O fencing is what guarantees only one appliance imports the pool, and it is enforced by the device rather than by QuantaStor. A back end that does not honour persistent reservations cannot host a highly-available pool. See I/O fencing and SCSI-3 persistent reservations.

QuantaStor has been tested with third-party arrays including NetApp and HPE MSA, as well as with QuantaStor itself as the back end. Contact support@osnexus.com for the current hardware compatibility list, or to have an array added to it.

Hardware requirements

Minimum layout: two front-end controller nodes, two or more back-end data nodes, and a fabric between them.
  • 2x QuantaStor appliances as front-end controller nodes.
  • Back-end storage presenting block LUNs. Where that is QuantaStor, 2x or more appliances as data nodes with local SAS or SATA disk.
  • A high-performance fabric between them -- FC, iSCSI or NVMe-oF.
  • At least two network interfaces per front-end appliance on separate networks, for the two heartbeat rings of the site cluster.
  • Mirrored boot devices in each appliance, 480GB or larger SSD or NVMe media, on a hardware RAID controller. A boot device that fills up is the most common cause of local database corruption.

Separate the front-end and back-end networks. Client traffic and the traffic between the controller and data nodes have different characteristics and different failure consequences, and mixing them makes both harder to reason about. Where the back end is iSCSI, disable iSCSI access on the management and replication ports so that back-end traffic cannot land on them: 

qs network-port-modify --port=eth0 --iscsi-enable=false

qs network-port-modify also carries --nvmeof-enable for NVMe-oF TCP, --mtu for jumbo frames, and --auto-tune for RX/TX buffer sizing. The same settings are in the WUI; see Network Ports.

Navigation: Storage Management → Storage Systems (section) → Network Port (select + right-click) → Modify Network Port...

Back-end data nodes

A back-end data node: local pools, one storage volume per pool, presented to the front end.

Where the back end is QuantaStor, configure each data node as an ordinary appliance:

  • Build one or more storage pools from its local disk. A pool of 10 to 20 drives is a good size; use several pools per appliance rather than one very large one.
  • Create one storage volume per pool, sized to the pool.
  • SAS disk is preferable, but enterprise SATA is usable here -- the dual-port requirement applies to the front-end pool's devices, not to the disks behind a back-end array.
  • HBAs or hardware RAID controllers are both acceptable on a data node.

See Storage Pools and Storage Volumes for the pool and volume work itself, which is not HA-specific.

Presenting the back-end storage

The front-end appliances import the back-end volumes as physical disks and build one pool across them.

The goal is that both front-end appliances see the same set of LUNs as physical disks. What that takes depends on the transport.

Where the back end is QuantaStor, the mechanism is a Host entry per front-end appliance, carrying that appliance's initiator identifiers, with the back-end storage volumes assigned to those hosts. Get the front-end appliance's iSCSI initiator name from the appliance itself: 

# qs-util iscsiiqn
INFO: iSCSI IQN: iqn.2009-15.com.osnexus:01:7531003303be

Then, on the back end, qs host-add creates the host and qs host-initiator-add adds an identifier to it -- --iqn takes an iSCSI IQN, an NVMe NQN or an FC WWPN. Assign every back-end volume to every front-end host, so that both front-end appliances see every LUN.

iSCSI and NVMe-oF: software adapters

Adding a software adapter for an iSCSI or NVMe-oF back end.

On each front-end appliance, a software adapter establishes and maintains the connection to the back end. It logs in to the targets and surfaces their LUNs in the Physical Disks section, where they can be used to build a pool like any other device.

Navigation: Storage Management → Controllers & Enclosures (section) → Hardware Controller (select + right-click) → Add Software Adapter...

The dialog asks for the Portal IP Address of the back end and states what it will do: "All LUNs assigned to the initiator IQN/NQN of the system will be imported." Under Software Adapter Settings the port and CHAP credentials are optional -- the port defaults to 3260 for iSCSI and 4420 for NVMe. A CHAP username without a password, or a password without a username, is rejected; supply both or neither.

From the command line: 

qs sw-controller-add --storage-system=qs-node1 --name=backend-1 \
   --ip-address=10.10.20.5 --sw-controller-type=iscsi

qs sw-controller-add takes --sw-controller-type of iscsi (the default), nvme-tcp or nvme-rdma, an optional --port, and optional --chap-user and --chap-pass. Add one adapter per front-end appliance per back-end portal.

Automatic configuration for a QuantaStor back end

Where both ends are QuantaStor and in the same grid, all of the above can be done in one operation. Select the back-end volumes, select the front-end appliances to receive them, and QuantaStor creates the host entries, the volume assignments and the software adapters itself.

Navigation: Storage Management → Controllers & Enclosures (section) → Hardware Controller (select + right-click) → Auto-config Software iSCSI Adapters...

The dialog is explicit about where it stops: "Automates connectivity of iSCSI storage between systems for the setup of tiered highly-available storage pools. First select the volumes to be mapped to the designated front-end systems, then select all the front-end systems which will receive the volumes. Pool creation and cluster setup is not automatic." So it gets you as far as the LUNs being visible on both front-end appliances; the pool, the site cluster and the HA group are still yours to create. 

qs sw-controller-autoconfig --system-list=qs-node1,qs-node2 --volume-list=backend-vol-1,backend-vol-2

qs sw-controller-autoconfig takes --flags=dry-run-only, which reports what it would create without creating it. Use that first on a grid where the host entries may already exist.

Fibre Channel

For an FC back end, create the Host entries with the WWPNs of the front-end FC ports that will face the back end, and assign the back-end volumes to them. There is no software adapter for FC -- the LUNs appear once zoning and assignment allow it, and qs disk-scan or a rescan of the controller picks them up.

Two things about FC are worth knowing before you cable anything.

QuantaStor presents FC target LUNs only from QLogic adapters. Target mode is implemented against the QLogic qla2xxx driver, and QuantaStor enables target mode automatically on ports that support it. A back-end QuantaStor appliance therefore needs QLogic FC cards to serve LUNs at all; the front end, which is only an initiator on the back-end fabric, does not. qs fc-remote-port-list shows which remote WWNs the appliance's FC target ports can actually see, which is the quickest way to tell whether zoning and cabling are right before you go looking for LUNs.

A card that must both serve and initiate may need dual mode set explicitly. A back-end appliance whose FC ports both present LUNs and initiate to something else has to have initiator and target mode active on the same port. qs-util enabledualmode writes qlini_mode=dual into /etc/modprobe.d/qla2xxx.conf for both the qla2xxx and qla2xxx_scst module names, and qs-util disabledualmode writes qlini_mode=exclusive to return the port to target-only. Either takes effect on the next boot. Read the port's current state from the kernel with cat /sys/class/fc_host/host<n>/device/scsi_host/host<n>/active_mode, which prints Initiator, Target when both are live. Dual-mode FC configuration covers doing this by hand without a reboot.

Direct point-to-point cabling between front end and back end avoids the cost of a switch in a small configuration. Where you do use a fabric, use a topology that gives you two independent paths from each front-end appliance to each back-end port, and zone it so that both front-end appliances can reach every LUN. A single-path fabric will build a working pool and then fail the first time a switch is serviced.

Managing a software adapter

Once an adapter exists, its own context menu carries the operations you need while troubleshooting:

Navigation: Storage Management → Controllers & Enclosures (section) → Software Adapter (select + right-click)
Item Command Purpose
Scan for Targets qs sw-controller-scan Re-run discovery against the portal
Target Login qs sw-controller-target-login Log in to one or more discovered targets
Target Logout qs sw-controller-target-logout Log out, which removes the LUNs from this appliance
Remove Software Adapter qs sw-controller-remove Delete the adapter entirely
Auto-config Software iSCSI Adapters qs sw-controller-autoconfig Offered on iSCSI adapters only

qs sw-controller-list lists the adapters, qs sw-controller-target-list the targets they have discovered, and qs sw-disk-session-list the live sessions. Target Logout takes the LUNs away from the appliance, so do not run it against an adapter carrying a pool that is imported there.

Building the HA pool

Front-end appliance detail: the imported LUNs, the pool built across them, and the client-facing protocols.

From the point where both front-end appliances can see the LUNs, the build is the same as the JBOD-attached one and is documented once, there:

  1. First boot. Install the current release on every appliance, apply a licence key to each one (each key is unique to its appliance), set a static address rather than leaving DHCP in place, and set the DNS and NTP servers. See Storage System.
  2. Grid. Both front-end appliances -- and the back-end appliances, where they are QuantaStor -- join one storage grid. See Grid Configuration.
  3. Site cluster. Create it across the two front-end appliances only; back-end data nodes are left out. Each ring needs the same network present on both appliances -- ring 1 on the first network, ring 2 on a second, separate one -- so that a single switch cannot take the heartbeat with it. See Site Cluster Setup.
  4. Storage pool. Create it on one front-end appliance from the physical disks that have arrived from the back end. QuantaStor analyses them and stripes across back-end sources where it can, so that the loss of one back-end node does not stop the pool. The HA-specific device rules are under Before you build the HA group; the pool itself is covered on Storage Pools.
  5. HA group. Create it for the pool. If any device is not reachable from both appliances the create refuses and names the devices. See Creating the storage pool HA group and What the create checks, and what it rejects.
  6. HA virtual interface. Every client must reach the pool through this address rather than through an appliance's own. The parent port you choose must exist under the same name on the failover appliance -- the interface attaches to the port of that name there -- so name your ports consistently across the pair. See The HA virtual interface and Cluster VIFs.
  7. Activate, then test failover in both directions before the cluster carries production data. See Failover.

The one check that behaves differently with FC

Enabling high availability on a pool switches its Fibre Channel presentation into ALUA mode, and that drops every active FC client session. QuantaStor refuses the HA group create rather than doing it silently:

Enabling high-availability on Storage Pool '<pool>' will drop all active FC sessions as the pool will switch to FC ALUA communication mode. Use the 'force' option to enable this or disconnect all active sessions before enabling FC ALUA mode.

This only bites when a pool has already been serving FC clients before it was made highly available, which is a common order of events when an existing appliance is being converted into an HA pair. Either disconnect the FC clients first, or pass --flags=force to qs ha-group-create and accept the session drop. Confirm the resulting ALUA state with qs-util alua.

Provisioning to clients

The completed configuration: clients to the front-end virtual interfaces, front end to back end over the fabric.

Provisioning on the front end is ordinary QuantaStor work, with one rule that is not optional:

  • Create Network Shares for SMB and NFS, and Storage Volumes for iSCSI, FC and NVMe-oF, on the HA pool.
  • Create a Host entry for each client that needs block storage, carrying its IQN, NQN or WWPN, and assign volumes to it.
  • Point every client at the pool's HA virtual interface, not at an appliance address. Block clients are protected from getting this wrong, because iSCSI access is restricted to the HA virtual interfaces. NFS and SMB clients are not, and a share mounted over a management address works perfectly until the first failover. Unexpected power-off of an appliance covers how to find clients that have done this.

Troubleshooting

Back-end data node detail: local pools and the volumes presented to the front end.

Recovery procedures for a running cluster are on Scale-up HA Storage Pool Troubleshooting, which opens with a symptom-to-procedure table. Failed media, a lost back-end node, a lost front-end appliance, a pool that will not start, split-brain, boot media failure and reinstall, and ransomware are all covered there, and none of it depends on how the shared storage is attached.

A few diagnostics are specific to a SAN back end.

LUNs missing on one front-end appliance. The two appliances must see the same devices, and qs-util devicemap on each of them is the direct comparison -- the /dev/sdX letters will differ and that is expected, but the /dev/disk/by-id paths and the serial numbers must match. FC and iSCSI attached storage varies its sdX assignment far more than a SAS enclosure does, because arrays differ in how they answer bus probes. Check the adapter's session state with qs sw-controller-list and re-run Scan for Targets, and confirm the back end has the front-end appliance's IQN, NQN or WWPN in a Host entry with every volume assigned to it. See Disks not visible on both appliances for the full checklist.

Fencing not working. qs-iofence devstatus reports the reservation on every device. A LUN that reports NOT-SUPPORTED does not implement persistent reservations, which for a back-end array is a hardware or firmware limitation rather than a configuration error -- that array cannot host a highly-available pool. See qs-iofence devstatus.

FC diagnostics. qs-util alua prints the ALUA state of each FC target LUN, qs-util listfcclients lists the connected FC clients, and qs-util issuelip issues a loop initialization primitive on every FC port to force rediscovery. That last one disrupts FC traffic while it runs; do not use it on a cluster serving production FC clients.

iSCSI initiator diagnostics. qs-util iscsiiqn prints the appliance's own initiator name, qs-util iscsidiscover <ip> discovers targets at an address, and qs-util iscsirelogin <ip> logs back in to previously established targets. Prefer the software adapter operations above for anything QuantaStor is managing; these are for confirming that the network and the target are answering at all.

Command line reference

Command Purpose
qs sw-controller-add Add an iSCSI or NVMe-oF software adapter
qs sw-controller-autoconfig Create the hosts, assignments and adapters for a QuantaStor back end in one step
qs sw-controller-list List the software adapters
qs sw-controller-get Detail for one adapter
qs sw-controller-scan Re-run target discovery
qs sw-controller-target-list List discovered targets
qs sw-controller-target-login Log in to targets
qs sw-controller-target-logout Log out of targets
qs sw-controller-remove Remove an adapter
qs sw-disk-session-list List live adapter sessions
qs host-add Create a host entry on the back end
qs host-initiator-add Add an IQN, NQN or WWPN to a host entry
qs fc-remote-port-list List the remote FC WWNs visible to the appliance's FC target ports
qs network-port-modify Address, MTU, and iSCSI or NVMe-oF access per port
qs disk-scan Rescan for newly presented devices
qs disk-list List physical disks with the pool each belongs to

The HA group and virtual interface commands are on HA Cluster Setup (JBODs).

Related pages

Also worth watching: QuantaStor 6: Deployment of Highly Available (HA) Storage Pool on Seagate Exos (19:02).

Verified against QuantaStor 6.9.0.

Retrieved from "https://wiki.osnexus.com/index.php?title=HA_Cluster_Setup_(external_SAN)&oldid=28068"