OSNEXUS Online Documentation Site - User contributions [en]

Guides:Index

2026-10-08T21:00:07Z

Qadmin: osn-seo-utilities: regenerate index

''Practical, in-depth guides to designing and running QuantaStor storage. Each guide links to the reference documentation for the features it covers.''

== Guides ==

* '''[[Guides:Asynchronous Remote Replication and Disaster Recovery Failover|Asynchronous Remote Replication and Disaster Recovery Failover]]''' — Asynchronous remote replication keeps a second, mountable copy of your volumes and shares at another site, updated on a schedule by sending only the blocks that changed.
* '''[[Ceph over ZFS|Ceph over ZFS]]''' — [[Category:admin_guide]] Ceph over ZFS runs Ceph OSDs on ZFS Storage Volumes (zvols) instead of directly on Physical Disks.
* '''[[Guides:Encryption at Rest and FIPS Mode for On-Premises Storage|Encryption at Rest and FIPS Mode for On-Premises Storage]]''' — Encryption at rest protects the data on a storage pool's media, so drives that leave the data center, whether failed, stolen or decommissioned, can't be read without the keys.
* '''[[Guides:Enterprise Shared Storage for Proxmox VE Clusters|Enterprise Shared Storage for Proxmox VE Clusters]]''' — Shared storage for a Proxmox VE cluster is a block or file store that every node can reach, so virtual machines and containers can migrate between nodes and restart elsewhere when a node fails.
* '''[[Guides:Fibre Channel and iSCSI Block Storage with ALUA Multipathing|Fibre Channel and iSCSI Block Storage with ALUA Multipathing]]''' — QuantaStor presents storage volumes as block LUNs over Fibre Channel, through QLogic host bus adapters running in target mode, and over iSCSI, and it reports each path's state to the host with ALUA (Asymmetric Logical Unit Access).
* '''[[HA Cluster Setup (JBODs)|HA Cluster Setup (JBODs)]]''' — [[Category:admin_guide]] ''_TOC_''
* '''[[High-availability VIF Management|High-availability VIF Management]]''' — Cluster virtual interfaces (VIFs) are floating IP addresses that move between the appliances of a site cluster so that a service address stays reachable when the appliance hosting it fails.
* '''[[ISCSI Target Configuration|ISCSI Target Configuration]]''' — QuantaStor presents each Storage Volume to iSCSI clients as its own target, with its own IQN, on the network ports you allow for iSCSI.
* '''[[Guides:Multi-Admin Approval for Destructive Storage Operations|Multi-Admin Approval for Destructive Storage Operations]]''' — Multi-admin approval is a two-person rule for data-destructive operations: when it is enabled, a delete of a protected object type is held as a pending request until a set number of administrators approve it.
* '''[[Network Ports|Network Ports]]''' — [[Category:admin_guide]]
* '''[[Network Ports|Network Ports]]''' — [[Category:admin_guide]]
* '''[[Guides:Nightly Database Analysis from iSCSI Snapshots|Nightly Database Analysis from iSCSI Snapshots]]''' — Offline database analysis means running reports, audits and heavy queries against a point-in-time copy of production data on a separate server, so that the production database never sees the load.
* '''[[Snapshot Schedules|Snapshot Schedules]]''' — [[Category:admin_guide]] A '''Snapshot Schedule''' takes point-in-time snapshots of a chosen set of [[Storage Volumes|Storage Volumes]] and [[Network Shares|Network Shares]] on a repeating schedule, and rotates them so each volume and share keeps a moving window of recovery points.
* '''[[Guides:Tiering On-Premises Object Storage to AWS S3 with Lifecycle Policies|Tiering On-Premises Object Storage to AWS S3 with Lifecycle Policies]]''' — Cloud tiering moves older objects from a local S3 bucket to a bucket at AWS while the bucket keeps serving the same namespace.

[[Category:Guides]]

Guides:Enterprise Shared Storage for Proxmox VE Clusters

2026-10-08T21:00:03Z

Qadmin: osn-seo-utilities: proxmox-shared-storage @ 2579ba917a3f (approved in the portal)

''By Steve Umbehocker, CTO, OSNexus · Updated October 3, 2026''

Shared storage for a Proxmox VE cluster is a block or file store that every node can reach, so virtual machines and containers can migrate between nodes and restart elsewhere when a node fails. QuantaStor provides that storage from a central appliance or HA cluster, and its open-source Proxmox VE storage plugin adds a native <code><nowiki>quantastor</nowiki></code> storage type that provisions one iSCSI volume per Proxmox disk, with snapshots, templates, clones, resize and migration driven from the standard Proxmox tools. This guide covers the shared storage options, how the plugin works, and a walkthrough from storage pool to first VM disk.

== Why it matters ==

Proxmox VE runs well on local disks, but local disks tie each guest to one node. Live migration, HA restart and a single place to manage capacity all need storage every node can see. Teams moving off VMware hit the same issue in reverse: a vSAN cluster combined compute and storage in the same hosts, and the replacement hypervisor needs somewhere to put the data.

External shared storage separates those concerns. Compute nodes can be added, upgraded or reinstalled without touching the data, and capacity grows by adding drives to the storage tier rather than buying matching hypervisor hosts. The storage tier also brings its own data services: snapshots, replication to a second site, encryption and role-based administration.

== How it works ==

Proxmox VE has several built-in ways to consume shared storage (see the Proxmox [https://pve.proxmox.com/wiki/Storage storage documentation]). QuantaStor can serve the common ones:

{| class="wikitable"
! Option !! What Proxmox sees !! QuantaStor side !! Best fit
|-
| QuantaStor storage plugin (iSCSI) || A <code><nowiki>quantastor</nowiki></code> storage; one block device per disk || Scale-up [[Storage Pools|storage pool]], one [[Storage Volumes|storage volume]] per disk || VM and LXC disks with per-disk snapshots and clones
|-
| NFS || A shared directory for disk images, ISOs and backups || [[NFS Configuration|NFS share]] on a storage pool || ISO libraries, backups, file-based disk images
|-
| Ceph RBD || Proxmox's own RBD storage type || [[Scale-out Block Setup (ceph)|Scale-out block storage]] on a Ceph cluster || Scale-out deployments across many storage nodes
|}

For the RBD option, Proxmox creates images directly with its own Ceph client. QuantaStor recognizes block devices that were created outside its own management by Proxmox, OpenStack or Kubernetes and cleans up their volume objects automatically, so the grid view stays consistent with the cluster. Ceph block storage needs its own Ceph pool, separate from any pool that holds a CephFS file system, and the web manager only offers compatible pools when you create block storage.

The rest of this guide focuses on the plugin, because it gives Proxmox the most direct control over each disk. It separates management traffic from storage traffic:

* '''Control plane.''' Create, delete, snapshot, clone, resize and access-control operations are REST calls to the [[QuantaStor REST API Reference Guide|QuantaStor REST API]] over HTTPS on port 8153, authenticated with HTTP Basic credentials.
* '''Data plane.''' Guest I/O runs over iSCSI on port 3260. The plugin drives the standard <code><nowiki>open-iscsi</nowiki></code> initiator with <code><nowiki>iscsiadm</nowiki></code>: it discovers the target, logs in and waits for the device before handing it to Proxmox.

Each storage volume is exported through its own iSCSI target at LUN 0, and the plugin addresses it by a stable <code><nowiki>/dev/disk/by-path/...-lun-0</nowiki></code> path rather than a <code><nowiki>/dev/sdX</nowiki></code> name. QEMU attaches the raw block device for VM disks; for containers, Proxmox formats the same device with ext4 and mounts it as the root filesystem.

On first activation, the plugin registers each Proxmox node as a [[Hosts and Host Groups|host]] using the initiator IQN from <code><nowiki>/etc/iscsi/initiatorname.iscsi</nowiki></code>. Before a node uses a volume, the plugin assigns that host to the volume's access control list, and it removes the assignment when the volume is released. A volume is visible only to the nodes actively using it.

Volume names follow the Proxmox convention, so mapping a disk to its volume needs no lookup table:

{| class="wikitable"
! Type !! Name pattern !! Example
|-
| VM or container disk || <code><nowiki>vm-<vmid>-disk-<N></nowiki></code> || <code><nowiki>vm-100-disk-0</nowiki></code>
|-
| Template base disk || <code><nowiki>base-<vmid>-disk-<N></nowiki></code> || <code><nowiki>base-100-disk-0</nowiki></code>
|-
| Template snapshot || <code><nowiki>template-base-<vmid>-disk-<N></nowiki></code> || <code><nowiki>template-base-100-disk-0</nowiki></code>
|-
| Snapshot || <code><nowiki><volume>_<snapshot></nowiki></code> || <code><nowiki>vm-100-disk-0_snap1</nowiki></code>
|}

The plugin does not patch or replace Proxmox source files, so Proxmox package upgrades do not break it. It is distributed as a Debian package from the [https://github.com/OSNEXUS/pve-quantastor-plugin pve-quantastor-plugin repository].

== Design and sizing ==

Plan the cluster around these requirements from the [[Proxmox Storage Plugin|plugin reference]]:

{| class="wikitable"
! Item !! Requirement
|-
| Proxmox VE || 9.1.x and 9.2.x, which can be mixed in one cluster. 8.4.x and older do not load the plugin, so keep 8.x nodes out of the cluster.
|-
| Storage || A storage pool to provision from, on a current QuantaStor release
|-
| Network || Every Proxmox node reaches the appliance on TCP 8153 (REST API) and TCP 3260 (iSCSI)
|-
| QuantaStor configuration || None beyond the pool: the plugin creates volumes, host entries and ACL assignments
|}

For production clusters, put the pool in a [[Create Storage Pool High-Availability Group|high-availability failover group]] so a storage controller failure doesn't take every VM down with it. The plugin's configuration options cover HA pools and SSL; see the plugin reference for the exact settings. The grid below shows scale-up ZFS pools next to scale-out Ceph block, file and object pools on the same grid, with the HA group controls in the toolbar.

[[File:Guide-proxmox-shared-storage-proxmox-storage-pools.png|frame|center|The Storage Pools grid: scale-up (ZFS) pools that can back the Proxmox plugin, scale-out block pools for Ceph RBD, and the Storage Pool HA Resource Group toolbar used to make a pool highly available]]

Keep the iSCSI data path on a dedicated storage network where possible. The REST API traffic is light, but guest I/O follows every disk.

== Setting it up ==

=== 1. Create a least-privilege API user ===

The <code><nowiki>admin</nowiki></code> account works, but credentials stored on the Proxmox cluster should carry only what the plugin needs. Create a [[Role Create|role]] under '''Security → Role → Create''', then a [[User Add|user]] with that role under '''Security → User → Add'''. The role needs view access to everything plus these write operations:

* '''Storage Volume''': create, delete, modify, resize, createSnapshot, deleteSnapshot, rollback, clone
* '''Storage Volume ACL''': add, remove
* '''Host''': add, addInitiator, modify, remove, assignVolume, unassignVolume

[[File:Guide-proxmox-shared-storage-proxmox-create-role-dialog.png|frame|center|The Create Role dialog: filter the Object Type list, tick the rows the plugin needs, then use Apply Permission Scope to Selected to give them System scope]]

Or create both in one pass with the [[QuantaStor CLI Command Reference|QuantaStor CLI]]:

<pre>qs role-add --name=pve-plugin-role --desc="Role for the Proxmox VE storage plugin" \
--permissions="*:view:system,\
StorageVolume:create:system,StorageVolume:delete:system,StorageVolume:modify:system,\
StorageVolume:resize:system,StorageVolume:createSnapshot:system,StorageVolume:deleteSnapshot:system,\
StorageVolume:rollback:system,StorageVolume:clone:system,\
StorageVolumeAcl:add:system,StorageVolumeAcl:remove:system,\
Host:add:system,Host:addInitiator:system,Host:modify:system,Host:remove:system,\
Host:assignVolume:system,Host:unassignVolume:system"

qs user-add --name=pve-plugin --password=<password> --role=pve-plugin-role</pre>

=== 2. Check connectivity from each Proxmox node ===

Before installing anything, confirm every node can reach both ports on the appliance (192.0.2.10 here):

<pre>nc -zv 192.0.2.10 8153
nc -zv 192.0.2.10 3260
cat /etc/iscsi/initiatorname.iscsi</pre>

=== 3. Install the plugin and add the storage ===

Install the <code><nowiki>pve-storage-quantastor</nowiki></code> package from the [https://github.com/OSNEXUS/pve-quantastor-plugin/releases releases page] on every node in the cluster. The <code><nowiki>quantastor</nowiki></code> type then appears under '''Datacenter → Storage → Add''' in the Proxmox web interface, alongside the built-in types. Enter the appliance address, the storage pool and the API user created in step 1. The plugin reference lists every configuration option, including the HA pool and SSL settings.

== Operating and testing ==

Once the storage is added, it works with the standard <code><nowiki>pvesm</nowiki></code>, <code><nowiki>qm</nowiki></code> and <code><nowiki>pct</nowiki></code> tooling. A quick check from any node:

<pre>pvesm status
qm create 9001 --name test-vm --memory 2048 --scsi0 <storage-id>:8
iscsiadm -m session</pre>

The new disk appears in QuantaStor as a storage volume named <code><nowiki>vm-9001-disk-0</nowiki></code>, and the node that owns the VM appears as a host assigned to it. Migrate the VM to another node and the assignment moves with it: the volume is released from the old node and assigned to the new one.

Snapshots, templates, clones, resize and migration all run through the plugin; the Usage section of the [[Proxmox Storage Plugin|plugin reference]] describes how each behaves. Each Proxmox disk is a regular storage volume, so the volume-level tools on the QuantaStor side also apply: [[Snapshot Schedules|snapshot schedules]] for point-in-time copies outside Proxmox, and [[Remote-replication (DR)|remote replication]] to a second site for disaster recovery. For the replication design, see [[Guides:Asynchronous Remote Replication and Disaster Recovery Failover|Asynchronous Remote Replication and Disaster Recovery Failover]].

== FAQ ==

=== Which Proxmox VE versions does the QuantaStor plugin support? ===

Proxmox VE 9.1.x and 9.2.x, and both can run in the same cluster. On 9.2 the plugin logs a cosmetic message about implementing an older storage API. Proxmox VE 8.4 and older do not load the plugin, so any storage operation routed to an 8.x node fails.

=== Does the plugin work for LXC containers as well as VMs? ===

Yes. Container root filesystems run on QuantaStor volumes: Proxmox formats the iSCSI block device with ext4 and mounts it as the container root, using the same provisioning and access control as VM disks.

=== Do I need to configure iSCSI targets or host entries on QuantaStor first? ===

No. Apart from the storage pool, the plugin creates everything itself: a volume and dedicated target per disk, a host entry per Proxmox node from its initiator IQN, and the ACL assignments that make a volume visible only to the node using it.

=== Can QuantaStor replace vSAN after moving from VMware to Proxmox? ===

It replaces vSAN's role as shared storage with an external tier. Proxmox nodes use QuantaStor over iSCSI through the plugin, over NFS, or over Ceph RBD for scale-out designs, and the storage tier adds HA pools, snapshots and remote replication of its own. Compute and storage then scale independently instead of in matched hypervisor hosts.

=== Should I use the plugin, NFS or Ceph RBD? ===

Use the plugin for VM and container disks when you want one volume per disk with storage-side snapshots, clones and per-node access control. Use NFS for ISO libraries, backups and file-based images. Choose Ceph RBD when the deployment needs to scale out across many storage nodes.

----
''Part of the [[Guides:Index|QuantaStor Guides]] series. For reference documentation, see the [[Main Page|QuantaStor documentation]].''

[[Category:Guides]]

File:Guide-proxmox-shared-storage-proxmox-create-role-dialog.png

2026-10-08T21:00:03Z

Qadmin: osn-seo-utilities: screenshot for Guides (proxmox-shared-storage)

osn-seo-utilities: screenshot for Guides (proxmox-shared-storage)

File:Guide-proxmox-shared-storage-proxmox-storage-pools.png

2026-10-08T21:00:03Z

Qadmin: osn-seo-utilities: screenshot for Guides (proxmox-shared-storage)

osn-seo-utilities: screenshot for Guides (proxmox-shared-storage)

Windows MPIO for Failover Clusters and Hyper-V

2026-10-08T11:12:54Z

Qadmin: Add custom path recovery (UseCustomPathRecoveryTime Enabled, 40 s) to the recommended MPIO settings; iSCSI notes; validated over FC and iSCSI

[[Category:admin_guide]]

This page describes the recommended Microsoft Multipath I/O (MPIO) configuration for Windows Server hosts that use QuantaStor storage volumes as shared cluster disks, including Windows Server Failover Clustering, Cluster Shared Volumes, Hyper-V clusters and SQL Server Failover Cluster Instances. It applies to Fibre Channel and iSCSI access to a QuantaStor HA pool, and it is the client-side companion to [[Clustered SCSI-3 Persistent Reservations]].

'''Summary.''' Install MPIO, let the Microsoft DSM claim QuantaStor devices, and apply the recommended MPIO timer settings below on every cluster node. With this configuration, an HA failover or an appliance reboot is a brief, transparent pause for the cluster: cluster disks stay online, Cluster Shared Volumes keep running, and Hyper-V virtual machines continue without interruption. The configuration was validated on Windows Server 2022 Hyper-V clusters connected to a QuantaStor HA pair over both Fibre Channel and iSCSI, including a live migration of the cluster's storage from iSCSI to Fibre Channel.

{| class="wikitable"
! Section !! Purpose
|-
| [[#How the recommended settings help|How the recommended settings help]] || How MPIO carries cluster disks through an HA failover
|-
| [[#Installing MPIO and claiming QuantaStor devices|Installing MPIO and claiming QuantaStor devices]] || The MPIO feature and the QuantaStor hardware ID
|-
| [[#Recommended MPIO settings|Recommended MPIO settings]] || The timer values, and how to apply and verify them
|-
| [[#Paths and load balancing|Paths and load balancing]] || What a correctly configured disk looks like
|-
| [[#After maintenance on a QuantaStor appliance|After maintenance on a QuantaStor appliance]] || Confirming all paths are in use before moving pools back
|-
| [[#Troubleshooting|Troubleshooting]] || Quick checks if a disk or path does not look as expected
|}

== How the recommended settings help ==

During an HA failover, the pool moves from one QuantaStor appliance to its partner. For a short interval, while the receiving appliance imports the pool, I/O is held rather than completed. With Fibre Channel and ALUA, QuantaStor keeps the standby paths through the partner appliance present throughout, so Windows sees the paths change state and continues on the new active paths, typically within well under a minute.

With iSCSI connected through the pool's HA virtual interface, each disk has a single session that moves with the virtual interface: the session reconnects to the partner appliance once it has imported the pool. MPIO's custom path recovery returns that path to service as soon as the session is back, so the disk continues on the same path.

MPIO decides how long to hold I/O and keep a disk while its paths change. The Windows defaults are tuned for general-purpose storage and allow only a short window (20 seconds before an unreachable disk is released, 60 seconds per I/O). The recommended settings extend that window so it comfortably covers an HA failover, including a slower one, for example when a busy pool takes longer to export. The cluster then experiences the failover as a short pause, and Cluster Shared Volumes and virtual machines carry on as normal.

Together with [[Clustered SCSI-3 Persistent Reservations]], which keeps the cluster's disk reservations in place across the failover, these settings give Windows clusters a seamless experience on QuantaStor HA pools.

== Installing MPIO and claiming QuantaStor devices ==

Do this on every cluster node, before presenting QuantaStor volumes to it.

'''1. Install the Multipath I/O feature''' (a reboot completes the installation):

<pre style="font-size: smaller">
Install-WindowsFeature -Name Multipath-IO
</pre>

'''2. Add the QuantaStor hardware ID to the Microsoft DSM''' so that MPIO combines the paths to each QuantaStor volume into a single disk:

<pre style="font-size: smaller">
New-MSDSMSupportedHW -VendorId OSNEXUS -ProductId QUANTASTOR
Get-MSDSMSupportedHW | Where-Object VendorId -match OSNEXUS
</pre>

The PowerShell cmdlet pads the vendor and product IDs to their SCSI field widths for you. If you prefer the MPIO control panel or {{Code|1=mpclaim}}, use the string {{Code|1=OSNEXUS QUANTASTOR}} followed by exactly six trailing spaces, as described in [[Multipath IO Configuration#Configuring Microsoft MPIO|Configuring Microsoft MPIO]].

'''3. Reboot''' the node so MPIO claims the devices. Each QuantaStor volume then appears once in Disk Management and in {{Code|1=Get-Disk}}.

For iSCSI, connect one session per path in the iSCSI Initiator with '''Enable multi-path''' selected, using addresses on separate subnets; see [[ISCSI Initiator Setup]] and [[Multipath IO Configuration]]. For an HA pool, connect to the pool's HA virtual interfaces so the sessions follow the pool when it fails over.

== Recommended MPIO settings ==

{| class="wikitable"
! Setting !! Windows default !! Recommended !! What it does
|-
| PDORemovePeriod || 20 || '''240''' || Seconds MPIO keeps a disk available while its paths are changing. 240 seconds comfortably covers an HA failover.
|-
| DiskTimeoutValue || 60 || '''100''' || Seconds Windows allows for each I/O. 100 is the largest value {{Code|1=Set-MPIOSetting}} accepts.
|-
| PathVerificationState || Disabled || '''Enabled''' || MPIO checks every path periodically, so it picks up path changes and returning paths promptly.
|-
| PathVerificationPeriod || 30 || 30 || Seconds between path checks.
|-
| RetryCount || 3 || 3 || Times MPIO retries an I/O on a path.
|-
| RetryInterval || 1 || 1 || Seconds between those retries.
|-
| UseCustomPathRecoveryTime || Disabled || '''Enabled''' || MPIO retries a path that has become unavailable, so the path returns to service on its own once it is reachable again, for example when an iSCSI session reconnects after a failover.
|-
| CustomPathRecoveryTime || 40 || 40 || Seconds between those path recovery attempts.
|}

Apply them in an elevated PowerShell session on each cluster node:

<pre style="font-size: smaller">
Set-MPIOSetting -NewPathVerificationState Enabled -NewPathVerificationPeriod 30 -NewPDORemovePeriod 240 -NewRetryCount 3 -NewRetryInterval 1 -NewDiskTimeout 100 -CustomPathRecovery Enabled -NewPathRecoveryInterval 40
</pre>

'''Reboot the node to activate the settings.''' In a running cluster, work through the nodes one at a time: drain the node (Pause, Drain Roles in Failover Cluster Manager, or {{Code|1=Suspend-ClusterNode -Drain}}), reboot it, resume it, and continue with the next node. The cluster stays available throughout.

Confirm the result with {{Code|1=Get-MPIOSetting}}, which shows the active values:

<pre style="font-size: smaller">
PS C:\> Get-MPIOSetting

PathVerificationState : Enabled
PathVerificationPeriod : 30
PDORemovePeriod : 240
RetryCount : 3
RetryInterval : 1
UseCustomPathRecoveryTime : Enabled
CustomPathRecoveryTime : 40
DiskTimeoutValue : 100
</pre>

{{Code|1=Get-MPIOSetting}} is the best place to check these values, because {{Code|1=Set-MPIOSetting}} stores them through WMI rather than in the MPIO registry key. {{Code|1=Set-MPIOSetting}} validates all of its parameters together, so if it reports an error (for example a {{Code|1=-NewDiskTimeout}} above 100), correct the value and run the command again.

The Fibre Channel HBA driver parameters can stay at the vendor defaults; this configuration was validated with the QLogic Windows driver at its defaults.

The Microsoft iSCSI Initiator settings can also stay at their defaults. The initiator's {{Code|1=LinkDownTime}} (15 seconds by default) is how long it waits for a session to reconnect before it reports the path as unavailable; during a failover that is longer than this, custom path recovery is what brings the path back when the session reconnects. The configuration was validated with the iSCSI Initiator at its defaults.

== Paths and load balancing ==

The Microsoft DSM's default load-balance policy works well with QuantaStor and needs no change. QuantaStor reports ALUA path states for HA pools: paths through the appliance that owns the pool are Active/Optimized and paths through the partner appliance are Standby. The Microsoft DSM's default for ALUA storage, Round Robin With Subset, spreads I/O across the Active/Optimized paths and moves to the other set when the pool fails over.

View a disk's paths with {{Code|1=mpclaim}}:

<pre style="font-size: smaller">
mpclaim -s -d # list MPIO disks
mpclaim -s -d 1 # paths and their states for MPIO disk 1
</pre>

A host with two HBA ports zoned to both appliances of an HA pair shows four paths per disk: two Active/Optimized and two Standby. After a failover the two sets swap roles, and the path count stays the same.

An iSCSI disk connected through a pool's HA virtual interface shows one Active/Optimized path per session. It has no Standby path, because the session itself follows the virtual interface to the partner appliance during a failover.

== After maintenance on a QuantaStor appliance ==

After an appliance has been rebooted or its Fibre Channel ports have been offline, a quick rescan on the Windows nodes makes sure all of its paths are back in use. Doing this before moving a pool back to that appliance ensures every node has its full set of paths ready for the move.

<ol>
<li>Wait until the appliance is fully up, with its Storage Pools showing a Normal state in the web interface. QuantaStor brings an appliance's Fibre Channel target ports online once its LUNs are mapped and its ALUA states are set.</li>
<li>On every cluster node, rescan storage:
<pre style="font-size: smaller">
Update-HostStorageCache
pnputil /scan-devices
</pre>
A {{Code|1=rescan}} in {{Code|1=diskpart}} does the same if you prefer it.</li>
<li>Confirm with {{Code|1=mpclaim -s -d <n>}} that every QuantaStor disk shows its full path count.</li>
<li>Move the pool back.</li>
</ol>

== Troubleshooting ==

'''A cluster disk did not stay online through a failover.''' Run {{Code|1=Get-MPIOSetting}} on every node and confirm the recommended values are active, including {{Code|1=UseCustomPathRecoveryTime : Enabled}}; they take effect after a reboot. Also confirm that [[Clustered SCSI-3 Persistent Reservations]] is Enabled on the pool's HA group. To bring the disk back, run {{Code|1=Update-HostStorageCache}} and a {{Code|1=diskpart}} rescan on each node, then bring the cluster disk online in Failover Cluster Manager ({{Code|1=Start-ClusterResource}}).

'''A node no longer lists a disk after a failover, or the disk has no disk number.''' Rescan on that node as described above. If the disk is still missing, drain the node, restart it, and resume it; the disk returns with its paths and the node rejoins its Cluster Shared Volumes in direct mode. Then confirm {{Code|1=UseCustomPathRecoveryTime : Enabled}} on every node.

'''A disk shows fewer paths than expected, or Standby paths only.''' Rescan as described in [[#After maintenance on a QuantaStor appliance|After maintenance on a QuantaStor appliance]]. Also check that both of the host's HBA ports are zoned to both appliances, and that both of its WWPNs are listed under the host in QuantaStor.

'''A QuantaStor volume appears more than once in Disk Management.''' MPIO has not yet claimed the device. Check {{Code|1=Get-MSDSMSupportedHW}} for the OSNEXUS QUANTASTOR entry, add it if needed, and reboot.

'''Fibre Channel port mode on the appliances.''' Keep the appliances' Fibre Channel ports in their default target-only mode for HA pools serving Windows clusters. Dual initiator and target mode ({{Code|1=qs-util enabledualmode}}) is intended for diagnostics.

== Related pages ==

* [[Clustered SCSI-3 Persistent Reservations]] -- the HA group setting that keeps Windows cluster reservations in place across failovers
* [[Multipath IO Configuration]] -- MPIO and Linux multipath basics for QuantaStor volumes
* [[ISCSI Initiator Setup]] -- connecting iSCSI initiators
* [[Fibre Channel Target Port Management]] -- presenting volumes over Fibre Channel
* [[Hosts and Host Groups]] -- host records, initiators, and Host Groups for clusters
* [[HA Cluster Setup (JBODs)]] -- HA groups and failover

----

Guides:Index

2026-10-08T00:00:18Z

Qadmin: osn-seo-utilities: regenerate index

''Practical, in-depth guides to designing and running QuantaStor storage. Each guide links to the reference documentation for the features it covers.''

== Guides ==

* '''[[Guides:Asynchronous Remote Replication and Disaster Recovery Failover|Asynchronous Remote Replication and Disaster Recovery Failover]]''' — Asynchronous remote replication keeps a second, mountable copy of your volumes and shares at another site, updated on a schedule by sending only the blocks that changed.
* '''[[Ceph over ZFS|Ceph over ZFS]]''' — [[Category:admin_guide]] Ceph over ZFS runs Ceph OSDs on ZFS Storage Volumes (zvols) instead of directly on Physical Disks.
* '''[[Guides:Encryption at Rest and FIPS Mode for On-Premises Storage|Encryption at Rest and FIPS Mode for On-Premises Storage]]''' — Encryption at rest protects the data on a storage pool's media, so drives that leave the data center, whether failed, stolen or decommissioned, can't be read without the keys.
* '''[[Guides:Fibre Channel and iSCSI Block Storage with ALUA Multipathing|Fibre Channel and iSCSI Block Storage with ALUA Multipathing]]''' — QuantaStor presents storage volumes as block LUNs over Fibre Channel, through QLogic host bus adapters running in target mode, and over iSCSI, and it reports each path's state to the host with ALUA (Asymmetric Logical Unit Access).
* '''[[HA Cluster Setup (JBODs)|HA Cluster Setup (JBODs)]]''' — [[Category:admin_guide]] ''_TOC_''
* '''[[High-availability VIF Management|High-availability VIF Management]]''' — Cluster virtual interfaces (VIFs) are floating IP addresses that move between the appliances of a site cluster so that a service address stays reachable when the appliance hosting it fails.
* '''[[ISCSI Target Configuration|ISCSI Target Configuration]]''' — QuantaStor presents each Storage Volume to iSCSI clients as its own target, with its own IQN, on the network ports you allow for iSCSI.
* '''[[Guides:Multi-Admin Approval for Destructive Storage Operations|Multi-Admin Approval for Destructive Storage Operations]]''' — Multi-admin approval is a two-person rule for data-destructive operations: when it is enabled, a delete of a protected object type is held as a pending request until a set number of administrators approve it.
* '''[[Network Ports|Network Ports]]''' — [[Category:admin_guide]]
* '''[[Network Ports|Network Ports]]''' — [[Category:admin_guide]]
* '''[[Guides:Nightly Database Analysis from iSCSI Snapshots|Nightly Database Analysis from iSCSI Snapshots]]''' — Offline database analysis means running reports, audits and heavy queries against a point-in-time copy of production data on a separate server, so that the production database never sees the load.
* '''[[Snapshot Schedules|Snapshot Schedules]]''' — [[Category:admin_guide]] A '''Snapshot Schedule''' takes point-in-time snapshots of a chosen set of [[Storage Volumes|Storage Volumes]] and [[Network Shares|Network Shares]] on a repeating schedule, and rotates them so each volume and share keeps a moving window of recovery points.
* '''[[Guides:Tiering On-Premises Object Storage to AWS S3 with Lifecycle Policies|Tiering On-Premises Object Storage to AWS S3 with Lifecycle Policies]]''' — Cloud tiering moves older objects from a local S3 bucket to a bucket at AWS while the bucket keeps serving the same namespace.

[[Category:Guides]]

Snapshot Schedules

2026-10-08T00:00:14Z

Qadmin: osn-seo-utilities: docs-triggering-a-snapshot-schedule-on-demand-snaps @ 2a0350bcdfcb (approved in the portal)

[[Category:admin_guide]]
A '''Snapshot Schedule''' takes point-in-time snapshots of a chosen set of [[Storage Volumes|Storage Volumes]] and [[Network Shares|Network Shares]] on a repeating schedule, and rotates them so each volume and share keeps a moving window of recovery points. Snapshots are space-efficient and instant, so a schedule that runs every few hours costs little more than the data that changes between runs.

A schedule keeps two kinds of snapshot: a small set of '''short term''' snapshots that follow the schedule directly, and an optional set of '''long term''' snapshots promoted out of them and held for hours, days, weeks, months or quarters. This page owns that retention model; [[Backup Policies]] and [[Remote-replication (DR)]] present the same controls for their own schedules and refer back here.

{| class="wikitable"
! Section !! Covers
|-
| [[#What a Run Does|What a Run Does]] || The sequence of a single schedule run, snapshot naming, and what gets skipped
|-
| [[#Creating a Snapshot Schedule|Creating a Snapshot Schedule]] || Every tab of the Create dialog, field by field
|-
| [[#How Retention Works|How Retention Works]] || Short term rotation, long term promotion, retention tags and expiration dates
|-
| [[#Keeping a Snapshot Beyond Its Retention Window|Keeping a Snapshot Beyond Its Retention Window]] || Holds, and what renaming a snapshot does
|-
| [[#Changing Which Volumes and Shares a Schedule Covers|Changing Which Volumes and Shares a Schedule Covers]] || The Update Selections dialog and the CLI add/remove commands
|-
| [[#Modifying a Schedule|Modifying a Schedule]] || What the Modify dialog can and cannot change
|-
| [[#Enabling, Disabling and Triggering|Enabling, Disabling and Triggering]] || Suspending a schedule and resuming it
|-
| [[#Triggering a Snapshot Schedule On Demand|Triggering a Snapshot Schedule On Demand]] || Running a schedule now from the web interface, the CLI or the REST API, and waiting for its snapshots
|-
| [[#Reclaiming Orphaned Snapshots|Reclaiming Orphaned Snapshots]] || Adopting snapshots left behind by a deleted schedule
|-
| [[#Deleting a Schedule|Deleting a Schedule]] || What happens to the snapshots it created
|-
| [[#Managing Snapshot Schedules from the CLI|Managing Snapshot Schedules from the CLI]] || The equivalent <code>qs</code> commands
|-
| [[#Troubleshooting|Troubleshooting]] || Snapshots not appearing, snapshots not expiring, pool filling up
|}

{{Navigation|Storage Management → Schedules ''(section)'' → Snapshot Schedules ''(tab)'' → Snapshot Schedule ''(toolbar group)''}}

The '''Snapshot Schedule''' toolbar group holds four buttons -- Create, Modify, Delete and Update Selections -- and the '''Snapshot Schedules''' tab in the center pane lists the schedules with their status, scheduled hours, days, and short-term snapshot count.

[[File:snapsched_list.png|thumb|right|800px|The Snapshot Schedule toolbar group and the Snapshot Schedules tab.]]

'''Enable''', '''Disable''' and '''Trigger''' are not on the toolbar. Right-click a schedule in the grid to reach them.

[[File:snapsched_context_menu.png|thumb|right|700px|Right-clicking a schedule adds Enable, Disable and Trigger to the four toolbar operations.]]

You can also start a schedule from the object it will cover: right-click a '''Storage Volume''' or a '''Network Share''' and choose '''Create Snapshot Schedule'''.

== What a Run Does ==

Each time a schedule fires it works through the same sequence:

# Existing snapshots of each member are examined, long-term retention rules are applied, and any snapshot that is now surplus is queued for deletion. This runs as a '''Delete Storage Volume Schedule Expired Snapshot(s)''' or '''Delete Network Share Schedule Expired Snapshot(s)''' task, so expiry is visible in the Tasks pane.
# {{Code|1=/var/opt/osnexus/custom/schedule-prestart.sh}} is run if it exists, with <code>--name=<schedule name> --id=<schedule id></code>. See [[#Custom scripts|Custom scripts]] below.
# New snapshots are created for every member that was not skipped.

All ZFS members that live in the same [[Storage Pools|Storage Pool]] are snapshotted by a single ZFS command, so they share one point in time. That matters when a database or an application spans several volumes or shares in a pool: the set is crash-consistent with itself rather than seconds apart. '''Disable Atomicity''' on the Advanced Settings tab breaks that guarantee up into batches, and is only worth using on very large schedules -- see below.

=== Snapshot naming ===

Snapshot names are derived from the UTC time the run started, not local time:

{| class="wikitable"
! Member !! Snapshot name
|-
| Network Share || <code><share>@GMT-2026.09.03-04.19.59</code>
|-
| Storage Volume || <code><volume>_GMT20260903_041959</code>
|}

The <code>@GMT-</code> form is the name Windows expects for '''Previous Versions''', which is why share snapshots use it. Both forms are what the schedule looks for when it decides which snapshots it owns, so they are worth recognising in the snapshot list.

=== What a run skips ===

* '''Members owned by another appliance.''' A schedule object is visible grid-wide, but a run only snapshots the volumes and shares owned by the appliance the schedule runs on. Members that belong to another grid member are skipped.
* '''Members used by another schedule in the last 10 seconds.''' If a volume or share was snapshotted by any snapshot or replication schedule within the previous 10 seconds it is left out of this run and an alert is raised naming the resource and the schedule. Two schedules covering the same object at the same hour will interfere with each other this way, so stagger them with '''Offset'''.
* '''Demoted shares.''' A share whose name carries <code>_demoted_</code> or <code>_demotedclone_</code> -- the result of promoting a snapshot -- is skipped, with an alert raised at most once a day.
* '''A run that overlaps the previous one.''' A scheduled run is skipped rather than queued while work from the previous run for the same schedule is still active.
* '''Everything, if the schedule is disabled.''' A disabled schedule neither creates snapshots nor expires them.

A schedule that has no volumes and no shares left disables itself the next time it runs.

== Creating a Snapshot Schedule ==

{{Navigation|Storage Management → Schedules ''(section)'' → Snapshot Schedule ''(toolbar group)'' → Create}}

The dialog is a five-tab wizard; '''Next''' and '''Previous''' walk the tabs and '''OK''' is accepted from any of them. The dialog refuses to open at all if the grid has no storage volumes and no network shares.

=== General ===

[[File:snapsched_create_general.png|thumb|right|800px|The General tab of the Create Snapshot Schedule dialog.]]

* '''Schedule Name''' -- pre-filled with the next free <code>snapshot-sched-''N''</code>. Names may contain letters, digits and the characters <code>- _ .</code> only, and must be unique across the grid.
* '''Description''' -- free text, optional.
* '''Start date''' -- defaults to today. A calendar schedule ignores any trigger point earlier than this date, so set it forward to arm a schedule ahead of time.
* '''Enabled''' -- on by default. Clear it to create the schedule without arming it.

The Snapshot Schedules grid has a '''Resource Group''' column, but there is no resource group control on this dialog; assign one with the CLI's <code>--resource-group</code> argument.

=== Schedule Interval ===

[[File:snapsched_create_interval.png|thumb|right|800px|The Schedule Interval tab with the default Monday-to-Friday, every-four-hours calendar schedule. The timer interval fieldset is greyed out while a calendar schedule is selected.]]

'''Schedule Type''' picks between two mutually exclusive models, and the fieldset for the one you did not pick is greyed out:

* '''Use day/hour selections''' (default) -- a calendar schedule. Tick the days and the hours; the schedule runs at each ticked hour on each ticked day. The default is '''Monday through Friday''' at '''12 AM, 4 AM, 8 AM, 12 PM, 4 PM and 8 PM'''. '''All Days''' and '''All Hours''' toggle every checkbox in their row at once.
* '''Use timer interval''' -- a rest interval instead of fixed times. The next run starts the given number of minutes '''after the previous run finishes''', so runs cannot overlap and the effective period is the interval plus however long a run takes. The minimum is '''3''' minutes and a smaller value is reset to 3; the field defaults to 30 and the slider covers 3 to 360 minutes.

[[File:snapsched_create_timer.png|thumb|right|800px|Selecting Use timer interval greys out the day and hour selections and enables the interval field.]]

==== Offset ====

'''Offset''' shifts every calendar trigger point by 0 to 59 minutes past the hour. It is the control to reach for when several schedules, or a schedule and a replication schedule, would otherwise fire on the same hour and skip each other's members: give each one a different offset and they no longer collide.

The hour checkboxes relabel themselves to the offset minute as you move the slider, so the tab always shows the times the schedule will actually run.

[[File:snapsched_offset.png|thumb|right|796px|With Offset set to 30 the hour checkboxes relabel to 12:30AM, 4:30AM and so on.]]

Offset applies to calendar schedules only; it has no effect on a timer interval schedule.

=== Select Volumes & Shares ===

[[File:snapsched_create_sources.png|thumb|right|800px|The Select Volumes & Shares tab. Only ZFS and CephFS shares are offered; the two cloud shares on this appliance are not listed.]]

Two inner tabs, '''Network Shares''' and '''Storage Volumes''', each hold a dual list: available objects on the left, the schedule's members on the right, moved with the arrow buttons or by dragging. '''At least one volume or share must be selected''' or OK is refused.

Each list has its own filter bar:

* the '''Storage Pool''' combo narrows the list to one pool;
* the text field takes one or more comma-separated terms and matches them against the object name -- prefix a term with <code>pool:</code> to filter by pool name or ID instead, for example <code>pool:pool-1,pool:pool-2</code>;
* '''Search''' applies the filter and '''Reset''' clears both the text and the pool selection.

'''Hide Nested Shares''', on the Network Shares tab, is ticked by default and hides nested shares so the list shows only top-level shares.

What the lists offer is deliberately narrow:

* '''Network Shares''' -- ZFS and CephFS shares only. '''Cloud''' shares, share aliases and sub-shares are not listed and cannot be added to a schedule.
* '''Storage Volumes''' -- volumes that are not themselves snapshots and are not cloud-backup volumes.
* Snapshots already created by a schedule are excluded from both lists, so you cannot build a schedule that snapshots another schedule's output by accident.

A volume or share may belong to several schedules at once -- for instance an hourly schedule for a short window and a daily schedule holding a longer history. Each schedule counts and expires only the snapshots it created itself.

=== Snapshot Settings ===

[[File:snapsched_create_retention.png|thumb|right|800px|The Snapshot Settings tab, with the default retention counts and the computed total.]]

* '''Long Term Snapshot Retention Settings''' -- how many hourly, daily, weekly, monthly and quarterly recovery points to hold. Each field defaults to '''2''' and accepts up to 1000. '''Suggested Defaults''' puts all five back to 2, and '''Clear''' sets all five to 0, turning long-term retention off.
* '''Max Short Term Snapshots''' -- how many schedule-driven snapshots to keep in the rotating window. The default is '''3'''; the service limit is 1000.
* '''Max Total Source Snapshots''' -- the sum of the five long-term counts and '''Max Short Term Snapshots''', recomputed as you type. It is the ceiling on how many snapshots of one member this schedule will hold, and the number to sanity-check against pool free space before you commit.

Long-term retention applies to ZFS and CephFS members, which is every member a schedule can have.

=== Advanced Settings ===

[[File:snapsched_create_advanced.png|thumb|right|800px|The Advanced Settings tab of the Create dialog. The Modify dialog offers the same options minus Force, plus Old Schedule ID.]]

* '''Recursively snapshot nested shares''' -- snapshots each selected share together with the shares nested beneath it, in the same operation. It has no effect on storage volumes.
* '''Reclaim Orphaned Snapshots''' -- adopts snapshots that another schedule created, so this schedule counts and expires them. See [[#Reclaiming Orphaned Snapshots|Reclaiming Orphaned Snapshots]].
* '''Disable Atomicity''' -- snapshots the members in batches rather than as one transaction per pool. A single transaction covering hundreds of datasets is a long transaction; batching keeps each one short at the cost of the members no longer sharing an identical point in time. The batch size is 16 by default and is set with <code>non_atomic_snapshot_batch_size</code> in the <code>[schedule_manager]</code> section of {{Code|1=/etc/quantastor.conf}}.
* '''Force''' -- lets you add a replication checkpoint to the schedule. Without it, selecting a checkpoint object is refused, because the next run of the replication schedule that owns the checkpoint will delete any snapshot taken of it. Force exists for the cases where that is acceptable; it is on the Create dialog only.

== How Retention Works ==

Both retention mechanisms act only on snapshots this schedule created, identified by the schedule ID stamped on each snapshot when it was taken.

'''Short term retention''' is the rotating window. After each run the schedule counts the snapshots it owns and marks the oldest surplus ones for deletion until only '''Max Short Term Snapshots''' untagged snapshots remain.

'''Long term retention''' does not create anything. After each run the schedule sorts the snapshots it owns oldest to newest and, for each of the five rules independently, keeps the oldest one and then the next one that is at least the rule's spacing later, up to the count you configured; when it has more than the count it drops the oldest one it was holding. Snapshots picked by any rule are tagged and are then exempt from the rotating window.

{| class="wikitable"
! Rule !! Minimum spacing between kept snapshots !! A count of 4 gives you
|-
| Hourly || 1 hour || 4 snapshots spanning the last 4 hours
|-
| Daily || 1 day || 4 snapshots spanning the last 4 days
|-
| Weekly || 7 days || 4 snapshots spanning the last month
|-
| Monthly || 30 days || 4 snapshots spanning the last 4 months
|-
| Quarterly || 90 days || 4 snapshots spanning the last year
|}

Spacing is matched with a five-minute tolerance, so a schedule that drifts slightly still satisfies a rule.

The practical consequence is that '''long-term retention can only promote a snapshot that the schedule already took'''. A quarterly count of 4 on a schedule that only ran for a week gives you one tagged snapshot, not four -- the other three slots have nothing to promote. Set the calendar or interval so that a snapshot exists at roughly the spacing each rule needs.

=== Retention tags and expiration dates ===

Each snapshot carries the tags it earned and a projected expiration date, both shown in the snapshot list under its parent volume or share.

[[File:snapsched_snapshot_retention.png|thumb|right|800px|The snapshot list for a share, showing which schedule owns each snapshot, the retention tags it earned, and its projected expiration.]]

* '''Schedule''' -- the schedule that created the snapshot. A raw GUID shown in red means the schedule no longer exists, so nothing is rotating that snapshot any more.
* '''Retention Tags''' -- <code>delta</code> for a snapshot held by the rotating window, and <code>hourly</code>, <code>daily</code>, <code>weekly</code>, <code>monthly</code> or <code>quarterly</code> for each long-term rule holding it. A snapshot commonly carries several. A snapshot with no tags at all is the next one out.
* '''Expiration''' -- the date the longest-held tag stops covering the snapshot, computed from its creation time. <code>N/A</code> means no long-term tag applies. It is a projection for planning, not a timer: the snapshot is removed when a run finds it surplus, not when this date passes.

The window each tag projects is the retention count multiplied by the rule's period, so 2 quarterlies gives a snapshot roughly 269 days.

== Keeping a Snapshot Beyond Its Retention Window ==

Two mechanisms take a snapshot out of a schedule's reach. They behave differently, and only one of them works for both object types.

'''Put a hold on it.''' A snapshot with at least one hold is excluded from schedule cleanup entirely -- it is not counted and not deleted, whatever the retention settings say -- and it also cannot be deleted by hand until the hold is released. This is the reliable option, and the only one that works for share snapshots.

<pre style="font-size: smaller">
qs share-hold-add --share=<share snapshot> --hold-tag=keep-for-audit
qs volume-hold-add --volume=<volume snapshot> --hold-tag=keep-for-audit
</pre>

Use <code>[[QuantaStor CLI Command Reference#share-hold-remove|qs share-hold-remove]]</code> and <code>[[QuantaStor CLI Command Reference#volume-hold-remove|qs volume-hold-remove]]</code> to release one, and <code>qs share-list --snapshots-with-holds-only=true</code> or <code>qs volume-list --snapshots-with-holds-only=true</code> to list what is currently held. The web interface offers the same operations as '''Add Hold''' and '''Remove Hold''' on a snapshot.

'''Rename it -- storage volume snapshots only.''' Renaming a storage volume snapshot clears its schedule ownership, so the schedule stops counting it and will never expire it. The snapshot survives indefinitely and is yours to delete. Renaming a '''share''' snapshot does not do this: the QuantaStor object is renamed but the underlying ZFS snapshot name and the schedule ownership are unchanged, so the schedule still owns it. Use a hold for shares.

Setting a description on a snapshot does not protect it.

== Changing Which Volumes and Shares a Schedule Covers ==

{{Navigation|Storage Management → Schedules ''(section)'' → Snapshot Schedule ''(toolbar group)'' → Update Selections}}

Membership is not on the Modify dialog. '''Update Selections''' is the only dialog that changes it, and the same dialog is what the '''Update Selections''' right-click item opens -- it is titled '''Update Selections for Snapshot Schedule''' and was called Add/Remove Volumes in earlier releases.

[[File:snapsched_update_selections.png|thumb|right|800px|Update Selections, with the schedule's current members already on the right and the running Total Available and Total Selected counts.]]

Pick the schedule at the top and the two dual lists load with its current members already on the right. What you leave on the right when you click OK '''becomes''' the membership -- the dialog replaces the whole set rather than adding to it, so anything you move back to the left is removed from the schedule. As on the Create dialog, at least one volume or share must remain selected.

Two details differ from the Create dialog:

* '''Hide Nested Shares''' starts unticked if the schedule already contains a nested share, so its existing members are visible.
* The '''Storage Volumes''' list is restricted to ZFS and Ceph RBD volumes.

Removing a member from a schedule does not delete the snapshots it already has; they stay, and stop being rotated. From the CLI, <code>[[QuantaStor CLI Command Reference#snap-schedule-add|qs snap-schedule-add]]</code> and <code>[[QuantaStor CLI Command Reference#snap-schedule-remove|qs snap-schedule-remove]]</code> change membership incrementally instead of replacing it:

<pre style="font-size: smaller">
qs snap-schedule-add --schedule=nightly --volume-list=vol3,vol4
qs snap-schedule-remove --schedule=nightly --share-list=share2
qs snap-schedule-assoc-list --schedule=nightly
</pre>

== Modifying a Schedule ==

{{Navigation|Storage Management → Schedules ''(section)'' → Snapshot Schedule ''(toolbar group)'' → Modify}}

The Modify dialog carries the same '''General''', '''Schedule Interval''', '''Snapshot Settings''' and '''Advanced Settings''' tabs as Create, with a schedule chooser at the top of the General tab. There is no '''Select Volumes & Shares''' tab and no '''Force''' checkbox, and the Advanced Settings tab gains '''Old Schedule ID'''.

[[File:snapsched_modify_advanced.png|thumb|right|800px|The Modify dialog's Advanced Settings tab. Old Schedule ID stays greyed out until Reclaim Orphaned Snapshots is ticked.]]

Reducing '''Max Short Term Snapshots''' or clearing the long-term counts takes effect on the next run, which will delete whatever is now surplus. Changing the schedule type, days, hours, interval or offset re-plans the schedule immediately.

== Enabling, Disabling and Triggering ==

{{Navigation|Storage Management → Schedules ''(section)'' → Snapshot Schedules ''(tab)'' → Snapshot Schedule ''(select + right-click)'' → Enable Snapshot Schedule / Disable Snapshot Schedule}}

The Enable and Disable dialogs are each a single schedule chooser.

'''Disabling''' a schedule stops both halves of its work: no new snapshots and no expiry. Existing snapshots are untouched and start accumulating no further, which makes disable the safe way to freeze a history in place while you investigate something. Enabling it again resumes both.

<pre style="font-size: smaller">
qs snap-schedule-disable --schedule=nightly
qs snap-schedule-enable --schedule=nightly
</pre>

Triggering has its own section, below.

== Triggering a Snapshot Schedule On Demand ==

{{Navigation|Storage Management → Schedules ''(section)'' → Snapshot Schedules ''(tab)'' → Snapshot Schedule ''(select + right-click)'' → Trigger Snapshot Schedule}}

Triggering runs a schedule once, now, in addition to its scheduled runs. Use it when you need a current set of recovery points before something risky -- an upgrade, a bulk import, a schema change -- or a fresh, crash-consistent copy of a group of volumes and shares to clone for testing or analysis. Because the run is the schedule's own, the snapshots carry its ID, are named and tagged like any other run, and are rotated out by its retention settings in due course; you do not have to remember to clean them up.

=== The Trigger Snapshot Schedule dialog ===

* '''Snapshot Schedule''' -- the schedule to run, pre-selected when you open the dialog from a right-clicked schedule.
* '''Schedule Information''' -- read-only: the schedule's '''State''', and how many '''Storage Volume(s)''' and '''Network Share(s)''' it covers. Check these before you click OK; a schedule whose state is not normal, or whose counts are not what you expect, will not give you the snapshots you are after.

Click '''OK''' to queue the run. If the grid has no snapshot schedules the dialog does not open, and reports ''There are no snapshot schedules available to trigger.''

=== What a triggered run does ===

A triggered run is an ordinary run started early, so everything in [[#What a Run Does|What a Run Does]] applies. Points that matter when you trigger by hand:

* '''It expires before it snapshots.''' The run applies retention first, so on a schedule already holding '''Max Short Term Snapshots''' untagged snapshots, a trigger rotates the oldest one out. If you are about to trigger to protect a particular older snapshot, put a hold on it first -- see [[#Keeping a Snapshot Beyond Its Retention Window|Keeping a Snapshot Beyond Its Retention Window]].
* '''A disabled schedule does nothing.''' The request is accepted and the '''Trigger Snapshot Schedule''' task completes successfully, but no snapshot is taken and nothing is expired. Enable the schedule first.
* '''A schedule with no members disables itself''' when triggered, as it would on a scheduled run.
* '''It is not debounced at schedule level.''' A scheduled run is skipped if the same schedule fired within the last 10 seconds; a manual trigger always runs. The per-member skips listed under [[#What a run skips|What a run skips]] still apply.
* '''Repeated clicks do not stack.''' If a trigger for the schedule is already waiting to be picked up, a second request is folded into it rather than queuing another run.
* '''It runs on one appliance.''' The request is forwarded to the grid member that owns the schedule's first storage volume (or, with no volumes, its first network share), and that appliance's run snapshots only the members it owns. Keep the volumes and shares you need captured together in one schedule on one appliance -- ideally in one pool, so they also share one point in time.

=== Triggering from the CLI ===

<code>[[QuantaStor CLI Command Reference#snap-schedule-trigger|qs snap-schedule-trigger]]</code> (alias <code>sch-trigger</code>) takes one argument, <code>--schedule</code>, which accepts the schedule's name or its ID:

<pre style="font-size: smaller">
qs snap-schedule-trigger --schedule=nightly
</pre>

=== Triggering from the REST API ===

The REST method is <code>snapshotScheduleTrigger</code>, with the same <code>schedule</code> parameter (name or ID) and the standard <code>flags</code> parameter. The response returns the schedule object and the task. With only a user name after <code>-u</code>, curl prompts for the account's credential rather than leaving it in your shell history. See [[QuantaStor REST API Reference Guide]] for authentication and the JSON-RPC form.

<pre style="font-size: smaller">
curl -k -u <user> "https://<appliance>:8153/qstorapi/snapshotScheduleTrigger?schedule=nightly&flags=0"
</pre>

=== Waiting for the new snapshots ===

'''The trigger returns before the snapshots exist.''' The '''Trigger Snapshot Schedule''' task -- and therefore the CLI command and the REST call -- completes as soon as the request has been handed to the schedule manager. The run itself starts moments later, takes the snapshots, and rescans the pool, and only then do the new snapshots appear in the snapshot list. A script that triggers a schedule and then clones, mounts or replicates the result has to wait for them.

Because snapshot names carry the UTC time the run took them (see [[#Snapshot naming|Snapshot naming]]), the reliable test is a name with a timestamp at or after the moment you triggered. Do not wait for the snapshot '''count''' to rise: the run expires surplus snapshots first, so on a schedule at its limit the count stays the same.

Run on the appliance that owns the volume, this records the UTC time, triggers the schedule, and waits up to five minutes for a snapshot of <code>vol1</code> taken at or after that time:

<pre style="font-size: smaller">
start=$(date -u +%Y%m%d_%H%M%S)
qs snap-schedule-trigger --schedule=nightly
for i in $(seq 60); do
qs volume-list | grep -o 'vol1_GMT[0-9]\{8\}_[0-9]\{6\}' | sed 's/.*_GMT//' \
| awk -v s="$start" '$0 >= s {f=1} END {exit !f}' && break
sleep 5
done
</pre>

For a network share, list snapshots with <code>qs share-list --include-snapshots=true</code>, match <code>share1@GMT-</code> followed by the timestamp, and record the start time as <code>date -u +%Y.%m.%d-%H.%M.%S</code> so the two compare as text in the same way.

Three things to keep in mind:

* '''Run it where the clock agrees.''' The comparison uses the local clock against the appliance's. On the appliance itself they are the same clock; from another host, make sure both are synchronised.
* '''Always bound the wait.''' A disabled schedule, a member skipped because another schedule just used it, or a full pool all produce no snapshot, and an unbounded loop would wait for ever. If the wait runs out, work through [[#No snapshots are being created|No snapshots are being created]].
* '''Wait for every member you need.''' Members in one pool are snapshotted together, but repeat the check for each volume or share your next step depends on.

== Reclaiming Orphaned Snapshots ==

Delete a schedule and its snapshots stay behind with its ID still stamped on them. Nothing rotates them any more, and they show that ID as a red GUID in the '''Schedule''' column of the snapshot list. On a busy share those orphans accumulate and consume pool space indefinitely.

'''Reclaim Orphaned Snapshots''', on the Advanced Settings tab of both the Create and Modify dialogs, hands them to the schedule you are creating or modifying. When you click OK, each of the schedule's volumes and shares is scanned for snapshots that are stamped with '''some other''' schedule's ID, and each one found is re-stamped with this schedule's ID -- in the database and in the <code>quantastor:scheduleid</code> and <code>quantastor:scheduletype</code> properties on the dataset itself. From then on they are counted by this schedule's retention rules and rotated out by it like its own snapshots.

Two limits are worth knowing:

* Only snapshots taken by a snapshot schedule or by hand are adopted. Snapshots created by a [[Remote-replication (DR)|replication schedule]] or a [[Backup Policies|backup policy]] are left alone, so reclaiming cannot break a replication relationship.
* The scan covers the schedule's own members only. Add the volumes and shares whose orphans you want adopted before you tick the box.

'''Old Schedule ID''', on the Modify dialog, narrows the scan from "any other schedule" to one specific schedule. It stays greyed out until '''Reclaim Orphaned Snapshots''' is ticked, and the value must be a UUID. Use it when a share carries orphans from more than one retired schedule and you only want one set adopted -- the red GUID in the snapshot list is the value to paste in.

== Deleting a Schedule ==

{{Navigation|Storage Management → Schedules ''(section)'' → Snapshot Schedule ''(toolbar group)'' → Delete}}

Deleting a schedule removes the schedule object only. '''Every snapshot it created remains''', and nothing expires them from that point on -- they become the orphans described above. If you are replacing a schedule rather than retiring it, create the replacement with '''Reclaim Orphaned Snapshots''' ticked so the existing history keeps rotating.

== Custom scripts ==

Scripts placed in {{Code|1=/var/opt/osnexus/custom/}} are run at fixed points if they exist. Snapshot schedules use one:

{| class="wikitable"
! Script !! When it runs
|-
| <code>schedule-prestart.sh</code> || At the start of every schedule run, before any snapshot is taken
|}

Each hook has two forms. <code>schedule-prestart.sh</code> runs synchronously and blocks the run until it exits, under a five-minute timeout; <code>schedule-prestart-async.sh</code> is backgrounded and nothing waits for it. Both are called with <code>--name=<schedule name> --id=<schedule id></code>, so one script can serve every schedule and branch on the name.

== Where the Snapshots Appear ==

Schedule-created snapshots are listed under the volume or share they were taken from, not under the schedule. In the web interface, expand a '''Network Share''' or '''Storage Volume''' in the tree, or select it and use the '''Snapshots''' tab in its detail panel.

From the CLI, share snapshots have to be asked for explicitly, while volume snapshots are always included:

<pre style="font-size: smaller">
qs share-list --include-snapshots=true
qs volume-list
</pre>

Recovering data out of a snapshot -- browsing it over NFS or SMB, restoring, rolling back or cloning it -- is covered on [[Network Shares]] and [[Storage Volumes]].

== Managing Snapshot Schedules from the CLI ==

{| class="wikitable"
! Command !! Alias !! Purpose
|-
| <code>[[QuantaStor CLI Command Reference#snap-schedule-create|snap-schedule-create]]</code> || <code>sch-create</code> || Create a schedule
|-
| <code>[[QuantaStor CLI Command Reference#snap-schedule-modify|snap-schedule-modify]]</code> || <code>sch-modify</code> || Change a schedule's settings
|-
| <code>[[QuantaStor CLI Command Reference#snap-schedule-delete|snap-schedule-delete]]</code> || <code>sch-delete</code> || Delete a schedule; its snapshots are kept
|-
| <code>[[QuantaStor CLI Command Reference#snap-schedule-enable|snap-schedule-enable]]</code> || <code>sch-enable</code> || Enable a schedule
|-
| <code>[[QuantaStor CLI Command Reference#snap-schedule-disable|snap-schedule-disable]]</code> || <code>sch-disable</code> || Disable a schedule
|-
| <code>[[QuantaStor CLI Command Reference#snap-schedule-trigger|snap-schedule-trigger]]</code> || <code>sch-trigger</code> || Run a schedule now; returns before the snapshots exist
|-
| <code>[[QuantaStor CLI Command Reference#snap-schedule-list|snap-schedule-list]]</code> || <code>sch-list</code> || List all schedules
|-
| <code>[[QuantaStor CLI Command Reference#snap-schedule-get|snap-schedule-get]]</code> || <code>sch-get</code> || Show one schedule in full
|-
| <code>[[QuantaStor CLI Command Reference#snap-schedule-add|snap-schedule-add]]</code> || <code>sch-add</code> || Add volumes or shares to a schedule
|-
| <code>[[QuantaStor CLI Command Reference#snap-schedule-remove|snap-schedule-remove]]</code> || <code>sch-remove</code> || Remove volumes or shares from a schedule
|-
| <code>[[QuantaStor CLI Command Reference#snap-schedule-assoc-list|snap-schedule-assoc-list]]</code> || <code>scha-list</code> || List a schedule's members
|}

A calendar schedule, weekdays at 2 AM and 6 PM, thirty minutes past the hour, keeping four rotating snapshots plus a week of dailies and a month of weeklies:

<pre style="font-size: smaller">
qs snap-schedule-create --name=nightly \
--days=mon,tue,wed,thu,fri --hours=2am,6pm --offset-minutes=30 \
--volume-list=vol1,vol2 --share-list=share1 \
--max-snaps=4 --rc-dailies=7 --rc-weeklies=4
</pre>

An interval schedule that snapshots one share every 30 minutes and keeps the last twelve:

<pre style="font-size: smaller">
qs snap-schedule-create --name=frequent \
--schedule-type=interval --interval=30 \
--share-list=share1 --max-snaps=12
</pre>

Adopting the orphans on a share into an existing schedule:

<pre style="font-size: smaller">
qs snap-schedule-modify --schedule=nightly --reclaim-old-snapshots=true
</pre>

Arguments worth knowing: <code>--schedule-type</code> takes <code>calendar</code> or <code>interval</code>; <code>--enabled</code> takes <code>true</code> or <code>false</code>; <code>--include-nested-shares</code>, <code>--disable-atomicity</code> and <code>--reclaim-old-snapshots</code> are the CLI names for the Advanced Settings checkboxes; and <code>--resource-group</code> assigns a resource group, which the dialogs do not offer.

== Troubleshooting ==

=== No snapshots are being created ===

# Check the schedule is enabled -- <code>qs snap-schedule-get --schedule=<name></code> reports '''Is Enabled'''. A schedule left with no members disables itself, so an unexpectedly disabled schedule is often an empty one. A trigger on a disabled schedule succeeds as a task but takes no snapshots.
# Check it still has members -- <code>qs snap-schedule-assoc-list --schedule=<name></code>.
# Check the members are owned by this appliance. A schedule run skips volumes and shares that belong to another grid member.
# Check for a collision. If another snapshot or replication schedule covers the same objects on the same hour, one of them will find its members busy and skip them; the alert names the resource and the schedule. Separate them with '''Offset'''.
# Check the pool has free space. A snapshot of a full pool cannot be created.
# The service log records every trigger decision: <code>journalctl -u quantastor --since "1 hour ago" | grep -i sched</code>.

=== Snapshots are not being expired ===

# Check '''Retention Tags''' on the snapshot. A long-term tag exempts it from the rotating window until its rule stops holding it.
# Check for a hold. A held snapshot is excluded from schedule cleanup entirely; <code>qs share-list --snapshots-with-holds-only=true</code> lists the held share snapshots.
# Check whether the snapshot has snapshots or clones of its own. A snapshot with children is never auto-deleted; remove the child first.
# Check the '''Schedule''' column. A red GUID means the owning schedule was deleted and nothing is rotating that snapshot -- see [[#Reclaiming Orphaned Snapshots|Reclaiming Orphaned Snapshots]].
# For a storage volume snapshot, check it has not been renamed. Renaming clears schedule ownership by design.

=== The pool is filling up with snapshots ===

A snapshot holds on to the blocks that have been overwritten since it was taken, so cost tracks write volume, not the size of the data. Work down from '''Max Total Source Snapshots''': it is the number of snapshots of each member the schedule will hold at once, and the honest input to a capacity estimate.

# Reduce '''Max Short Term Snapshots''', or '''Clear''' the long-term counts you are not using.
# Lengthen the interval, or drop hours from the calendar selection.
# Release holds on snapshots you no longer need to keep.
# Watch the '''Snapshot Used''' figure per share and volume, and set capacity alerts on the pool -- see [[Storage Pools]].

== Related pages ==

* [[Storage Volumes]] -- provisioning volumes, and browsing, cloning and reverting their snapshots
* [[Network Shares]] -- provisioning shares, nested shares, and snapshot access over NFS and SMB
* [[Storage Pools]] -- the pool the snapshots consume space from
* [[Backup Policies]] -- file-level backup to and from a remote export, with the same schedule and retention controls
* [[Remote-replication (DR)]] -- replicating volumes and shares to another QuantaStor system on a schedule
* [[QuantaStor REST API Reference Guide]] -- the <code>snapshotScheduleTrigger</code> method and the rest of the API

----
''Verified against QuantaStor 6.9.0.''

+ Admin Guide Overview

2026-10-08T00:00:04Z

Qadmin: osn-seo-utilities: list Ceph over ZFS under Scale-out Cluster Configuration (docs-ceph-over-zfs-zfs-draid-pool-reserved)

[[Category:admin_guide]]
The Administrator Guide provides documentation on how to configure all aspects of QuantaStor systems and storage grids, with a primary focus on the web user interface (WUI) and a secondary focus on how to automate using the QuantaStor CLI. In contrast the [[+ User Guide Overview|User Guide]] focuses on using QuantaStor storage and accessing it from various operating systems, and on the client-side configuration steps that involves.

It is written for IT administrators setting up or maintaining a system or grid, and for anyone wanting a deeper understanding of how the platform works. Each section below groups related topics, with a one-line summary of what each page covers so you can find the right one without opening several. If you are new to the interface, start with [[Using the Web Interface]]; if you are bringing up a new appliance, start with '''System & Grid'''. For the commands referenced throughout, see the [[+ CLI Guide Overview|CLI Guide]] and the [[QuantaStor CLI Command Reference]].

== Using the Web Interface ==

The web management interface itself: its screen regions, its nine main tabs, and the navigation conventions this guide uses to describe it.

{| class="wikitable"
! Topic !! Description
|-
| [[Using the Web Interface]] || Logging in, every region of the screen, the nine main tabs, toolbar groups and the overflow chevron, right-click menus, and how to read a '''Navigation''' breadcrumb.
|}

== System & Grid ==

Bringing up an appliance, joining it to a grid, licensing it, and recovering its configuration.

{| class="wikitable"
! Topic !! Description
|-
| [[Storage System]] || A single QuantaStor appliance, physical or virtual, and the settings on its Modify dialog.
|-
| [[Storage System Optimization]] || System tunables controlling cache behavior and Storage Pool I/O, and how the tunables file works.
|-
| [[Grid Configuration]] || Combining multiple systems into one Storage Grid managed as a single unit.
|-
| [[License Management]] || Applying and managing the license every system requires before the software can be used.
|-
| [[Recovery Manager]] || Restoring the internal configuration database from one of its automatic backups.
|}

== Hardware Configuration ==

Network interfaces, disks, controllers and third-party enclosures -- the physical layer beneath a Storage Pool.

{| class="wikitable"
! Topic !! Description
|-
| [[Network Ports]] || The Ethernet interfaces used for management and storage access, plus bonding, VLANs, virtual ports and static routes.
|-
| [[Physical Disks/Devices]] || Identifying, scanning, formatting and importing the physical disks a system can see.
|-
| [[Hardware Controllers & Enclosures]] || QuantaStor's integration modules for the major HBA and RAID controller models.
|-
| [[Multipath Configuration]] || Redundant SAS paths to dual-ported media, which QuantaStor configures automatically where it can.
|-
| [[External Systems Configuration]] || Disk enclosures and arrays from other vendors, used as the media behind a Storage Pool.
|-
| style="padding-left: 2.5em;" | [[Western Digital (WD) Data24 Configuration|Western Digital (WD) Data24]] || An NVMe-oF JBOF attached over RDMA (RoCE), including driver and lossless-network setup.
|-
| style="padding-left: 2.5em;" | [[Seagate Corvault Configuration|Seagate Corvault]] || A SAS-attached array that does its own RAID and presents large logical volumes.
|}

== Storage Provisioning ==

Turning physical media into pools, then into the block, file and object storage that clients consume.

{| class="wikitable"
! Topic !! Description
|-
| [[Storage Pools]] || Aggregating one or more disks into a pool of fault-tolerant storage.
|-
| [[Provisioning Tiers]] || Grouping pools together so automated frameworks such as OpenStack Cinder can provision from a tier.
|-
| [[Storage Volumes]] || Block devices (LUNs) accessed over iSCSI, Fibre Channel or Infiniband/SRP.
|-
| style="padding-left: 2.5em;" | [[Clustered SCSI-3 Persistent Reservations]] || Sharing client reservations across an HA group so Hyper-V, WSFC and SQL Server clusters survive a failover.
|-
| [[Fibre Channel Target Port Management]] || Exposing volumes as LUNs to Fibre Channel initiator hosts alongside iSCSI.
|-
| [[Network Shares]] || NAS access to a pool over NFSv3, NFSv4, SMB2 and SMB3.
|-
| [[NFS Configuration]] || The two NFS server implementations, and which one applies to a given storage architecture.
|-
| [[Multi-protocol File Locking]] || Locking on shares used by NFS and SMB clients at once, the setting that makes it coherent, and its limits.
|-
| [[Cloud Containers / NAS Gateway]] || Mapping an object-storage bucket to a Network Share so it can be accessed as files.
|-
| [[ISCSI Target Configuration]] || How QuantaStor presents storage volumes over iSCSI - target IQNs, portals and network ports, discovery, and what an initiator sees after a volume is assigned to a host.
|}

== Security, Alerting & Upgrades ==

Controlling who can do what, learning when a system needs attention, and keeping it patched.

{| class="wikitable"
! Topic !! Description
|-
| [[Security Configuration]] || Users, roles and RBAC, multi-factor authentication, certificates, the firewall and audit logging.
|-
| [[Call-home / Alerting]] || How a grid notifies you when systems need attention, including the ITSM integrations.
|-
| [[Upgrade Manager]] || Kernel, driver, security and core QuantaStor package updates.
|-
| [[Send System Log Report]] || Collecting the configuration and log bundle OSNEXUS support asks for on a new ticket.
|-
| [[FIPS Mode]] || Running the appliance with the FIPS 140-2 validated cryptographic module, and verifying it is active.
|-
| [[Encryption Bypass]] || Why QuantaStor does not encrypt devices that arrays such as Seagate Corvault, Seagate Exos E/EP and Dell PowerVault already encrypt in hardware, and how to add new vendor and model combinations as they are released. Applies to scale-up Storage Pools and scale-out OSDs alike.
|}

== High-availability VIF Management ==

The cluster heartbeat layer and the floating service addresses it manages, for both scale-out (Ceph) and scale-up (ZFS) storage clusters within a grid. Named for the main tab these are managed from.

{| class="wikitable"
! Topic !! Description
|-
| [[High-availability VIF Management]] || How cluster VIFs work: the grid, site cluster, heartbeat ring, HA group and VIF hierarchy, what triggers a failover and how long one takes, what blocks one, moving a VIF deliberately, and location constraints.
|-
| [[Site Cluster Setup]] || Creating a site cluster and its heartbeat rings, adding and removing members, and the corosync and pacemaker configuration it puts in place.
|-
| [[Cluster VIFs]] || Adding a cluster virtual interface and choosing its use case -- grid primary, scale-up storage pool, scale-out storage pool -- and what each type follows.
|-
| [[Configure Member Standby]] || Setting a node into standby moves all VIF resources and pools off it, so maintenance can be done without risk of a pool or resource moving onto it while the work is in progress.
|}

== Snapshots & Replication ==

Point-in-time copies, scheduled backups, and getting data to a second site.

{| class="wikitable"
! Topic !! Description
|-
| [[Snapshot Schedules]] || Automating space-efficient point-in-time snapshots of volumes and shares.
|-
| [[Backup Policies]] || Backing up any NFS or CIFS share on your network onto a QuantaStor system.
|-
| [[Remote-replication (DR)]] || Replicating both Storage Volumes (SAN) and Network Shares (NAS) to a remote system.
|-
| [[Bucket Synchronization Policies]] || Keeping object storage buckets in step across sites.
|}

== Scale-out Cluster Configuration ==

Scale-out Ceph cluster setup and configuration.

{| class="wikitable"
! Topic !! Description
|-
| [[Scale-out Block Setup (ceph)]] || Scale-out SAN built on Ceph.
|-
| [[Scale-out File Setup (ceph)]] || Scale-out NAS with access over native CephFS, SMB and NFS.
|-
| [[Scale-out Object Setup (ceph)]] || Scale-out object storage over the S3-compatible REST protocols.
|-
| [[Ceph over ZFS]] || Reserve a ZFS (DRAID) storage pool as the backing store for Ceph OSDs, with zvols used as the OSD devices.
|}

== Scale-up Cluster Configuration ==

Highly available scale-up pool cluster configuration.

{| class="wikitable"
! Topic !! Description
|-
| [[HA Cluster Setup (JBODs)]] || Clustered pools on shared SAS JBODs, so a node or path outage does not take storage offline.
|-
| [[HA Cluster Setup (external SAN)]] || The same clustered pool configuration where the media comes from an external SAN.
|}

== Optimization ==

Matching pool behavior to your workload, and watching how the grid performs.

{| class="wikitable"
! Topic !! Description
|-
| [[Performance Tuning]] || Pool I/O profiles, which set read-ahead, request queue depth and the I/O scheduler.
|-
| [[Performance Monitoring]] || The Grid Dashboard and the health and performance views built on it.
|}

== System Internals ==

How the platform is put together, for deeper troubleshooting and for extending it.

{| class="wikitable"
! Topic !! Description
|-
| [[QuantaStor systemd Services]] || The internal services and the systemd units under <code>/lib/systemd/system</code> that manage them.
|-
| [[QuantaStor Configuration Files]] || The configuration files that let engineering and support customize platform behavior.
|-
| [[QuantaStor Shell Utilities]] || The <code>qs</code> CLI plus the <code>qs-*</code> helpers such as <code>qs-iostat</code>, <code>qs-sendlogs</code> and <code>qs-upgrade</code>.
|-
| [[QuantaStor Custom Scripting / Application Extensions]] || Script call-outs for integrating QuantaStor with custom applications.
|}

----
''Verified against QuantaStor 6.9.0.''

Ceph over ZFS

2026-10-08T00:00:04Z

Qadmin: osn-seo-utilities: docs-ceph-over-zfs-zfs-draid-pool-reserved @ d389deda0b81 (approved in the portal)

[[Category:admin_guide]]
Ceph over ZFS runs Ceph OSDs on ZFS Storage Volumes (zvols) instead of directly on Physical Disks. There are a number of benefits of this architecture but the biggest is the double layer durability localizes the rebuild from failed drives so that the Ceph cluster doesn't need to absorb that load and production workload impact. The setup is simple, on each node you provision a Scale-up (ZFS) Storage Pool, ideally in a declustered RAID (dRAID) layout when you have 40+ drives per node. These pools become the backing store for Ceph OSDs, which become one-to-one mapped to Storage Volumes (zvols). So after the pool is created you'll batch create something like 16x, 24x, or 32x Storage Volumes from each pool. We call this the P:V or physical-to-virtual ratio and usually you'll want to aim for anything from 2:1 to 4:1. A 3:1 ratio means that for a node with 90x HDDs one would make that into a pool and then provision 30x (90:30 = 3:1) Storage Volumes to be turned into OSDs. This reduction also reduces the number of OSD processes running on each node and the amount of RAM you need per system. As an example with all 90x HDDs getting directly turned into OSDs we'd allocate 4GB to 6GB of RAM per OSD which amounts to 360GB to 540GB of RAM. In contrast, with a P:V of 3:1 we are producing 30x OSDs rather than 90x OSDs for roughly the same capacity. 30x OSDs at 8GB RAM per OSD is just 240GB of RAM so we're roughly cutting the RAM requirements in half. The same goes for the required CPU power.. we'd normally allocate about 1 to 1.5GHz per HDD based OSD. That's 90GHz to 135GHz if using the 90x HDDs directly as OSDs. Allocating 2GHz for each of the 30x zvol based OSDs puts us at just 60GHz. Granted, with the second filesystem (ZFS) at work underneath Ceph you'll need to reserve CPU for it and RAM for its ARC layer so the reduction can be somewhat offset but you should still see an overall reduction of 25% in CPU and RAM requirements.

{| class="wikitable"
! Section !! Purpose
|-
| [[#How it works|How it works]] || The Use Case setting and what it changes
|-
| [[#Requirements|Requirements]] || What must be in place before you start
|-
| [[#Step 1: Create a Storage Pool reserved for Ceph OSDs|Step 1]] || Reserve a ZFS pool for Ceph OSDs
|-
| [[#Step 2: Create the zvols|Step 2]] || Carve the zvols that back the OSDs
|-
| [[#Step 3: Create the OSDs|Step 3]] || Build one OSD on each zvol
|-
| [[#Operating notes|Operating notes]] || Boot order, LVM visibility and missing OSDs
|}

__TOC__

== How it works ==

A ZFS Storage Pool carries a '''Use Case''' that is set when the pool is created and cannot be changed afterwards:

{| class="wikitable"
! Use Case !! Meaning
|-
| Default || The pool provisions Storage Volumes and Network Shares as usual.
|-
| Ceph OSD || The pool is reserved for zvols that back Ceph OSDs. Every Storage Volume created in it inherits the Ceph OSD use case.
|}

A Storage Volume with the Ceph OSD use case is consumed directly by the Ceph OSD running on it. QuantaStor never presents it to hosts over iSCSI, FC or NVMe-oF: assigning one to a host fails with the message that the volume ''is reserved as a Ceph OSD backing device''.

The setup has three steps, each covered below:

# Create a ZFS Storage Pool with the Use Case set to '''Ceph OSD'''.
# Create the zvols in that pool, one per OSD you want.
# Create the OSDs from the zvols in the Ceph cluster.

The Storage Pools list has a '''Use Case''' column, so you can tell a Default pool from one reserved for Ceph OSDs at a glance.

== Requirements ==

* The pool must be a ZFS pool. QuantaStor refuses the Ceph OSD use case for any other pool type, and refuses a Ceph OSD zvol in a non-ZFS pool.
* The Storage System that owns the pool must be a member of the Ceph cluster you create the OSDs in, because that system creates the OSD on its own zvol.
* The pool must be started when you create the OSDs.
* Each zvol backs exactly one OSD. A snapshot, or a volume that is assigned to hosts, cannot back an OSD.

See [[Scale-out Block Setup (ceph)]] or [[Scale-out Object Setup (ceph)]] for creating the Ceph cluster itself.

== Step 1: Create a Storage Pool reserved for Ceph OSDs ==

{{Navigation|Storage Management → Storage Pools → Create ''(toolbar)'' → Advanced Settings ''(tab)''}}

[[File:Docs-docs-ceph-over-zfs-zfs-draid-pool-reserved-pool-use-case.png|thumb|right|800px|The Use Case list on the Advanced Settings tab of Create Storage Pool.]]

Create the pool as described on [[Storage Pools]]: name it, pick the disks on the '''General''' tab and choose a '''RAID Type'''. The DRAID layouts suit this use case because the OSD layer is designed around a DRAID pool of HDDs (see [[#How QuantaStor treats a zvol-backed OSD|How QuantaStor treats a zvol-backed OSD]]).

On the '''Advanced Settings''' tab, set '''Use Case''' to '''Ceph OSD'''. The other options on the tab (I/O Profile, Enable Compression, Enable SSD Auto Trim, Block Size Offset (ashift)) work as they do for any pool.

The Use Case cannot be changed once the pool exists, so set it here.

From the CLI, use <code>[[QuantaStor CLI Command Reference#pool-create|qs pool-create]]</code> with <code>--pool-use-case=ceph-zvol-osd</code>:

<pre style="font-size: smaller">
qs pool-create --name=osd-pool-1 --disk-list=<disk1>,<disk2>,... --raid-type=DRAID2_1S --pool-use-case=ceph-zvol-osd
</pre>

The DRAID layouts the CLI accepts are DRAID, DRAID1_1S, DRAID1_2S, DRAID1_3S, DRAID2_1S, DRAID2_2S, DRAID2_3S, DRAID3_1S, DRAID3_2S and DRAID3_3S.

== Step 2: Create the zvols ==

{{Navigation|Storage Management → Storage Volumes → Create ''(toolbar)''}}

Select the reserved pool as the '''Storage Pool''' on the '''General Settings''' tab, then set a '''Size''' for each zvol. Each zvol becomes one OSD, so the size you choose here is the OSD's capacity.

On the '''Advanced Settings''' tab:

* '''Use Case''' shows '''Ceph OSD''' and is locked, because the pool forces its use case onto every volume it holds.
* '''% Reserved''' defaults to '''100%'''. A zvol in a Ceph OSD pool is fully reserved by default because if the pool underneath runs out of space, every OSD on it starts failing writes at once. You can still lower it deliberately.
* '''Block Size''' sets the zvol's block size. When the OSD is created, QuantaStor sets Bluestore's minimum allocation size to match it, within a range of 4K to 64K.
* '''Batch Create''' creates several zvols of the same size in one step.

From the CLI, use <code>[[QuantaStor CLI Command Reference#volume-create|qs volume-create]]</code>. The CLI default for <code>--percent-reserved</code> is 0, so pass 100 explicitly to match the web interface:

<pre style="font-size: smaller">
qs volume-create --name=osd-vol --size=4T --pool=osd-pool-1 --count=6 --percent-reserved=100
</pre>

=== A single OSD zvol in a Default pool ===

You can also reserve an individual zvol in a Default pool: set '''Use Case''' to '''Ceph OSD''' on the '''Advanced Settings''' tab of Create Storage Volume, or pass <code>--volume-use-case=ceph-zvol-osd</code> to <code>qs volume-create</code>. Only that zvol is set aside for Ceph; the pool's other volumes stay available to hosts as usual. On a Ceph block pool there is nothing to choose, and the field stays at Default.

== Step 3: Create the OSDs ==

{{Navigation|Scale-out Storage Configuration → Data & Journal Devices → Create OSDs & Journals ''(toolbar)'' → Logical Devices ''(tab)''}}

The Create OSDs dialog has two tabs of available devices: '''Physical Devices''' for unused disks and '''Logical Devices''' for zvols. The Logical Devices tab lists every zvol that has the Ceph OSD use case, is healthy, is not a snapshot and does not already back an OSD. Move the zvols you want into '''Data/OSD Devices (HDDs or SSDs)''' and click '''OK'''. You can combine zvols and physical disks in one create. See [[Create Ceph OSD]] for the journal and optimization options in the rest of the dialog.

If a node has no unused physical disks but does have eligible zvols, the dialog still opens; it closes with ''No unused devices were found to create new OSDs or Journal Groups with'' only when neither tab has anything to offer.

From the CLI, use <code>[[QuantaStor CLI Command Reference#ceph-osd-multi-create|qs ceph-osd-multi-create]]</code> with <code>--volume-list</code>, which you can combine with <code>--physical-disk-list</code>:

<pre style="font-size: smaller">
qs ceph-osd-multi-create --ceph-cluster=<cluster> --volume-list=osd-vol-1,osd-vol-2,osd-vol-3
</pre>

=== How QuantaStor treats a zvol-backed OSD ===

* A zvol is new and has no partition table to clear, so QuantaStor skips the quick-format it runs on physical disks.
* For WAL/DB placement, a zvol counts as spinning media, because it is only as fast as the HDD pool behind it. Journal Group devices on SSD therefore apply to it as they do to an HDD.
* Once the OSD is up, QuantaStor renames its zvol to {{Code|1=vol-<system name>-osd.<n>}}, so the volume and the OSD it carries line up in the web interface and the CLI. If the rename fails, the OSD is unaffected and the zvol keeps its old name.

== Operating notes ==

=== Boot order ===

The OSDs cannot start until their pool is imported. At boot, QuantaStor imports every pool reserved for Ceph OSDs, and every Default pool that holds a Ceph OSD zvol, before the Ceph OSD services start. A pool that fails to import does not stop the rest of the boot; Ceph reports the OSDs on it as down.

Two cases are skipped:

* A pool with auto-start disabled is not imported, and the OSDs on it do not start.
* A pool managed by a Storage Pool HA failover group is left to the HA manager rather than imported at boot.

=== LVM visibility ===

Ceph builds each OSD as an LVM volume on its device, so LVM must be able to see the zvol. By default, the {{Code|1=global_filter}} in {{Code|1=/etc/lvm/lvm.conf}} hides every zvol from LVM so the appliance never scans a volume handed out to a client. QuantaStor adds accept rules for Ceph OSD zvols only: one rule for each pool reserved for Ceph OSDs, and one per zvol for a Ceph OSD zvol in a Default pool, so that pool's other zvols stay hidden. It updates the rules when a pool is created or started, including on another node after an HA failover, and when a Ceph OSD zvol is created or deleted. Other entries in the filter are left alone.

=== Missing OSDs ===

If the pool behind an OSD is not imported, the OSD shows as missing. QuantaStor keeps the OSD's link to its zvol, so it reconnects the OSD to the zvol once the pool is imported again.

== Related pages ==

* [[Storage Pools]]
* [[Storage Volumes]]
* [[Create Ceph OSD]]
* [[Delete Ceph OSD]]
* [[Scale-out Block Setup (ceph)]]
* [[Scale-out Object Setup (ceph)]]
* [[Designing Systems]]
* [[QuantaStor CLI Command Reference]]

----
''Verified against QuantaStor 6.9.0.''

File:Docs-docs-ceph-over-zfs-zfs-draid-pool-reserved-pool-use-case.png

2026-10-08T00:00:04Z

Qadmin: osn-seo-utilities: screenshot for Ceph over ZFS (docs-ceph-over-zfs-zfs-draid-pool-reserved)

osn-seo-utilities: screenshot for Ceph over ZFS (docs-ceph-over-zfs-zfs-draid-pool-reserved)

Guides:Index

2026-10-07T20:00:23Z

Qadmin: osn-seo-utilities: regenerate index

''Practical, in-depth guides to designing and running QuantaStor storage. Each guide links to the reference documentation for the features it covers.''

== Guides ==

* '''[[Guides:Asynchronous Remote Replication and Disaster Recovery Failover|Asynchronous Remote Replication and Disaster Recovery Failover]]''' — Asynchronous remote replication keeps a second, mountable copy of your volumes and shares at another site, updated on a schedule by sending only the blocks that changed.
* '''[[Guides:Encryption at Rest and FIPS Mode for On-Premises Storage|Encryption at Rest and FIPS Mode for On-Premises Storage]]''' — Encryption at rest protects the data on a storage pool's media, so drives that leave the data center, whether failed, stolen or decommissioned, can't be read without the keys.
* '''[[Guides:Fibre Channel and iSCSI Block Storage with ALUA Multipathing|Fibre Channel and iSCSI Block Storage with ALUA Multipathing]]''' — QuantaStor presents storage volumes as block LUNs over Fibre Channel, through QLogic host bus adapters running in target mode, and over iSCSI, and it reports each path's state to the host with ALUA (Asymmetric Logical Unit Access).
* '''[[HA Cluster Setup (JBODs)|HA Cluster Setup (JBODs)]]''' — [[Category:admin_guide]] ''_TOC_''
* '''[[High-availability VIF Management|High-availability VIF Management]]''' — Cluster virtual interfaces (VIFs) are floating IP addresses that move between the appliances of a site cluster so that a service address stays reachable when the appliance hosting it fails.
* '''[[ISCSI Target Configuration|ISCSI Target Configuration]]''' — QuantaStor presents each Storage Volume to iSCSI clients as its own target, with its own IQN, on the network ports you allow for iSCSI.
* '''[[Guides:Multi-Admin Approval for Destructive Storage Operations|Multi-Admin Approval for Destructive Storage Operations]]''' — Multi-admin approval is a two-person rule for data-destructive operations: when it is enabled, a delete of a protected object type is held as a pending request until a set number of administrators approve it.
* '''[[Network Ports|Network Ports]]''' — [[Category:admin_guide]]
* '''[[Network Ports|Network Ports]]''' — [[Category:admin_guide]]
* '''[[Guides:Nightly Database Analysis from iSCSI Snapshots|Nightly Database Analysis from iSCSI Snapshots]]''' — Offline database analysis means running reports, audits and heavy queries against a point-in-time copy of production data on a separate server, so that the production database never sees the load.
* '''[[Guides:Tiering On-Premises Object Storage to AWS S3 with Lifecycle Policies|Tiering On-Premises Object Storage to AWS S3 with Lifecycle Policies]]''' — Cloud tiering moves older objects from a local S3 bucket to a bucket at AWS while the bucket keeps serving the same namespace.

[[Category:Guides]]

High-availability VIF Management

2026-10-07T20:00:19Z

Qadmin: osn-seo-utilities: docs-active-directory-spn-failover-with-the-ha @ b83457fda786 (approved in the portal)

Cluster virtual interfaces (VIFs) are floating IP addresses that move between the appliances of a site cluster so that a service address stays reachable when the appliance hosting it fails. This page covers the cluster-level behaviour of those addresses -- what makes one fail over, how to move one deliberately, and how to control which appliances it is allowed to run on. Both scale-up (ZFS HA pool) and scale-out (Ceph) configurations use them, and no special hardware is required.

For virtual interfaces at the network port level -- creating a plain second address on a port, bonds, VLANs, MTU and static routes -- see [[Network Ports]]. That page owns the port; this page owns the cluster behaviour layered on top of it. For adding a cluster VIF and for the four use case types, see [[Cluster VIFs]].

{| class="wikitable"
! Section !! Purpose
|-
| [[#Grid, site cluster, heartbeat ring, HA group and VIF|Grid, site cluster, heartbeat ring, HA group and VIF]] || How the objects nest, and which one a VIF actually belongs to
|-
| [[#Where cluster VIFs are managed|Where cluster VIFs are managed]] || The High-availability VIF Management tab
|-
| [[#Use cases and creating a VIF|Use cases and creating a VIF]] || Pointer to [[Cluster VIFs]], which owns adding a VIF and the four use case types
|-
| [[#How failover works|How failover works]] || What triggers a move, and how long it takes
|-
| [[#What blocks a failover|What blocks a failover]] || Three checks that will refuse to start a VIF, and what they mean
|-
| [[#Moving a VIF deliberately|Moving a VIF deliberately]] || Planned relocation, and why it is not the same as a failover
|-
| [[#Location constraints|Location constraints]] || Pinning a VIF to, or away from, particular appliances
|-
| [[#Standby mode and maintenance mode|Standby mode and maintenance mode]] || Evacuating one node versus freezing the whole cluster
|-
| [[#What happens to client connections|What happens to client connections]] || What a client sees during a move, and giving a VIF name its own Active Directory (Kerberos) identity for SMB
|-
| [[#Scale-up and scale-out differences|Scale-up and scale-out differences]] || Where the two configurations diverge
|-
| [[#Removing a cluster VIF|Removing a cluster VIF]] || Deletion, and converting back to a local address
|-
| [[#Troubleshooting|Troubleshooting]] || When the Web UI, the cluster and the clients disagree
|}

== Grid, site cluster, heartbeat ring, HA group and VIF ==

This hierarchy is the thing readers most often get wrong, so it is worth stating plainly. Each layer sits on the one before it, and a VIF is at the top.

A '''storage grid''' is the management layer. It is the set of appliances you administer together through one interface, and it says nothing about high availability -- see [[Grid Configuration]]. Grid membership is a prerequisite: a site cluster can only be built from appliances that are already grid members.

A '''site cluster''' is the high-availability layer. It is a group of appliances at one location -- the location can span buildings, but the members are normally in close proximity, because they have to share a network. A site cluster is what runs the cluster software (corosync and pacemaker) that decides where a floating resource lives. A grid can contain several site clusters, and an appliance can belong to at most one. Site clusters are covered on [[Site Cluster Setup]].

A '''cluster heartbeat ring''' is how the members of a site cluster tell whether each other are alive. It is a network path, not an address that serves clients: each member contributes one port, and every port in a ring must be on the same subnet. A site cluster has at least one ring and at most two, and two rings on separate networks is the recommended configuration, so that losing one network does not look like losing a node. The first ring is created for you when you create the site cluster.

A '''storage pool HA failover group''' is the scale-up layer. It ties a ZFS pool to the appliances that are allowed to import it, so the pool itself can fail over. It exists only in scale-up configurations; a Ceph cluster provides its own redundancy and has no HA group. See [[HA Cluster Setup (JBODs)]] and [[HA Cluster Setup (external SAN)]].

A '''cluster virtual interface''' is the address clients talk to. It belongs to a site cluster, is attached to a named port on one member at a time, and is the layer that makes a service address survive the loss of an appliance. Its '''use case''' is what ties it to the resource it has to follow -- an HA failover group for a ZFS pool, a Ceph cluster for scale-out storage, or nothing at all.

The two things worth taking away: a heartbeat ring carries cluster gossip and a VIF carries client traffic, and they are separate; and a VIF's parent is the site cluster, not the pool, even when its whole purpose is to follow a pool.

== Where cluster VIFs are managed ==

[[File:havif_overview.png|thumb|right|800px|The High-availability VIF Management tab, with a site cluster selected. The Site Cluster Members grid shows each member's standby mode, and the Cluster Heartbeat Ring Ports grid shows which address each member contributes to the ring.]]

{{Navigation|High-availability VIF Management}}

Cluster VIFs and the site clusters they belong to have their own top-level tab. The tree on the left has two sections:

* '''Site Clusters''' -- the site clusters in the grid, each expanding to show its heartbeat rings. Selecting a site cluster shows its members and its ring ports, and the toolbar switches to the site cluster operations.
* '''Site Cluster Virtual Interfaces''' -- the cluster VIFs, grouped by site cluster. Selecting this section switches the toolbar to '''Virtual Interface Management''' with '''Add Cluster VIF''', '''Modify Cluster VIF''' and '''Remove Cluster VIF'''.

[[File:havif_vif_list.png|thumb|right|800px|The Site Cluster Virtual Interfaces section. Note that Managed By and Started On differ: the VIF object is owned by one appliance but the address is currently running on another. Move is only on the right-click menu.]]

Two columns in the Site Cluster Virtual Interfaces grid are easy to confuse. '''Managed By''' is the appliance that owns the VIF configuration object; '''Started On''' is the appliance where the address is actually up right now. They differ whenever the VIF has moved, and it is '''Started On''' that tells you where client traffic is going.

'''Move Site Cluster VIF''' is on the right-click menu only -- there is no toolbar button for it. Right-clicking a VIF gives '''Move Site Cluster VIF...''', '''Remove Site Cluster VIF...''', '''Modify Site Cluster VIF...''' and '''Properties...'''.

The grid below lists the '''Site Cluster Member Location Constraints''' for the selected VIF -- one row per member, with the weight that decides how strongly the VIF prefers that member.

== Use cases and creating a VIF ==

A cluster VIF's '''use case''' is what ties it to the resource its address has to follow, and it is chosen once, when the VIF is created. [[Cluster VIFs]] owns both topics: it covers adding a VIF field by field, in the dialog and from the command line, and each of the four use case types in depth -- what each one follows, what gates it, and what constraints it carries.

== How failover works ==

Underneath, each cluster VIF is a pacemaker resource. QuantaStor creates it as an {{Code|1=IPaddr2}} resource named after the VIF's tag, with the address, the netmask as a CIDR prefix, the parent port as the NIC, and a '''monitor that runs every 10 seconds'''. You can see it on any member with {{Code|1=crm_mon -1}}:

<pre style="font-size: smaller">
Active Resources:
* gm (ocf:heartbeat:IPaddr2): Started qs-node-111
</pre>

Three things move a VIF automatically:

* '''The monitor fails.''' The 10-second monitor finds the address is no longer correctly configured on the host, and pacemaker restarts the resource -- on another member if it cannot start locally.
* '''The appliance leaves the cluster.''' The remaining members stop hearing it on the heartbeat rings and pacemaker re-places its resources. How fast that is detected is governed by the corosync token settings; the shipped configuration uses a 3000 ms token with 10 retransmits before loss, so a lost member is normally declared within a few seconds. {{Code|1=/etc/corosync/corosync.conf}} on the appliance is authoritative -- it is managed by QuantaStor and carries a warning header saying so.
* '''A member is put into standby.''' See [[#Standby mode and maintenance mode|Standby mode and maintenance mode]].

Where it goes is decided by the [[#Location constraints|location constraints]], which give each member a score, subject to the checks in the next section.

Two cluster-wide settings are worth knowing because they explain behaviour that otherwise looks wrong. QuantaStor sets '''stonith-enabled to false''', so pacemaker does not fence a node it has lost contact with -- the duplicate-address check described below is what protects against two appliances answering for one address instead. And it sets '''no-quorum-policy to ignore''', so a member that finds itself in a minority partition does not stop its resources on its own.

=== How long a failover takes ===

Measured on a lab site cluster by pinging the VIF at 200 ms intervals from a third appliance and counting the gap. These are observations on idle test systems, not a specification:

{| class="wikitable"
! What was done !! Address unreachable for
|-
| Deliberate move, 3-node scale-up cluster || 2.50 s
|-
| Member put into standby while hosting the VIF || 2.49 s
|-
| Cluster services restarted on the hosting member || 2.68 s
|-
| Location weight raised above the stickiness threshold || 3.60 s
|-
| Deliberate move, 4-node scale-out cluster || 3.11 s
|-
| Cluster services restarted on the hosting member, scale-out || 3.12 s
|}

So a planned relocation and an unplanned one cost about the same, in the region of two to three seconds, and a scale-out VIF is slightly slower because the resource agent checks that the Ceph filesystem is actually mounted before it will start the address. What is '''not''' represented here is the abrupt loss of a whole appliance -- losing power, for instance. That adds the time corosync needs to declare the member gone, on top of the figures above.

== What blocks a failover ==

QuantaStor ships its own version of the {{Code|1=IPaddr2}} resource agent, installed over the distribution's copy at {{Code|1=/usr/lib/ocf/resource.d/heartbeat/IPaddr2}} (the original is kept alongside it as {{Code|1=IPaddr2.backup}}). It adds preflight checks that refuse to bring a floating address up in situations where doing so would make things worse. When one of them refuses, the reason appears in the failed actions list in {{Code|1=crm_mon -1}} and in the task's error detail, so it is worth recognising them.

'''The address is still answering somewhere else.''' Before claiming the address, the agent flushes the ARP cache and pings it. If anything replies, it waits and tries again, and if it still replies it refuses:

<pre style="font-size: smaller">
Failed Resource Actions:
* gm start on qs-node-110 returned 'error' because 'IP address is still in use
on other node, failover blocked.' at Thu Sep 3 10:45:41 2026 after 7.607s
</pre>

This is the protection that stands in for fencing, and it is easy to trigger accidentally. If an appliance loses cluster communication but keeps running -- a heartbeat network problem rather than a crash -- it goes on holding its floating addresses while the surviving members conclude it has gone. The survivors then try to take the addresses over, find them still answering, and correctly decline. The VIF stops being managed anywhere until the stranded appliance rejoins or is shut down, but it does '''not''' go offline: it is still up on the appliance that never lost it. In a lab reproduction of exactly this -- cluster communication killed on the hosting member while the appliance itself stayed up -- the VIF was pinged continuously throughout and lost no packets at all, while the other two members logged the message above.

The remedy is to restore cluster communication on the stranded member. <code>[[QuantaStor CLI Command Reference#site-cluster-restart-services|qs site-cluster-restart-services]] --storage-system=<system></code> restarts corosync and pacemaker on one appliance, which brings it back into the cluster; the address is released and placed properly, and the failed actions clear on their own. Note that this restart will relocate any VIF the appliance is currently hosting.

'''The QuantaStor service is not running.''' The agent refuses to start a VIF on an appliance whose core service is stopped, with {{Code|1=QuantaStor service is stopped, failover blocked.}} An address that works but has no service behind it is worse than an address that has moved elsewhere.

'''The appliance is in manual standby.''' The agent checks for the manual-standby marker file independently of pacemaker's own standby flag, and refuses with a message telling you to take the node out of standby. The duplication is deliberate: pacemaker's flag can be cleared underneath QuantaStor when the cluster stack restarts, and without the second check a node you had deliberately parked could quietly start hosting resources again.

'''For scale-out VIFs, the storage has to be present.''' A file VIF will not start on an appliance unless the CephFS pool is mounted and its export is mounted there; an object VIF will not start unless the RADOS gateway is running and configured. This is what keeps a scale-out service address from landing somewhere that cannot serve it. The requirement is recorded per VIF in {{Code|1=/var/opt/osnexus/quantastor/clustervif_<port>_<tag>.uses}}, which is written to every member of the site cluster:

<pre style="font-size: smaller">
cephfs_id="7f69711e-de3e-fdd9-4ecd-5262201d1a56"
cephfs_name="cephfs-pool-1"
use_case_obj_id=c06b4f30-8c65-6dac-7421-239daedbe3bb
use_scaleout_filepool=true
</pre>

== Moving a VIF deliberately ==

[[File:havif_move.png|thumb|right|500px|Move Site Cluster Virtual Interface. Current System is read-only; pick the destination in Move to System.]]

{{Navigation|High-availability VIF Management → Site Cluster Virtual Interfaces → Site Cluster VIF ''(select + right-click)'' → Move Site Cluster VIF...}}

A deliberate move relocates a VIF to a member you choose -- before taking an appliance down for maintenance, or to rebalance which appliance is serving which address. The dialog asks for the '''Site Cluster''', the '''Cluster Virtual Interface''', and the '''Move to System'''; '''Current System''' is shown for reference and cannot be edited. There is no toolbar button, only the right-click menu.

From the command line, with <code>[[QuantaStor CLI Command Reference#site-vif-move|qs site-vif-move]]</code>:

<pre style="font-size: smaller">
qs site-vif-move --vif-resource=<vif-id> --move-to-system=qs-node-111
</pre>

A move is not a failover, and the difference matters:

* '''A move does not leave a preference behind.''' Pacemaker's own move mechanism works by pinning the resource with a temporary constraint; QuantaStor clears that constraint once the move completes. Checking the location constraints before and after a move shows them unchanged, so the VIF remains free to fail over normally afterwards. This is why you should move a VIF rather than reach for pacemaker directly -- doing it by hand leaves a pin that quietly prevents failover.
* '''A move will not override a pin.''' If the destination's location weight is None, the move is rejected before anything happens, with a message naming the fix: {{Code|1=Cannot move Site Cluster virtual network interface '<vif>' to system '<system>': that system's location constraint for this interface is set to 'None', which pins the interface to never run there. Set the location constraint for '<system>' to Low, Medium, or High before moving the interface.}}
* '''A move will not work while the cluster is frozen.''' In maintenance mode the attempt fails, because pacemaker is not managing the resource at all.

The destination must be a member of the same site cluster. Moving a VIF to an appliance that is merely in the same grid is not possible, and is not a meaningful request -- the cluster software has no presence there.

== Location constraints ==

[[File:havif_add_constraints.png|thumb|right|536px|Location constraints with Automatic cleared. Each member gets a weight, and the row has to be ticked as well as weighted.]]

Location constraints are how you say which appliances a VIF prefers, and which it must never run on. Each member of the site cluster gets a weight, which becomes the pacemaker score for that member -- the higher the score, the stronger the preference.

The Web UI offers four weights:

{| class="wikitable"
! Weight !! Score !! Effect
|-
| '''None''' || −INFINITY || The VIF will never run on this member. Automatic failover will not place it here, and a deliberate move to it is rejected.
|-
| '''Low''' || 100 || Eligible, least preferred
|-
| '''Medium''' || 200 || Eligible
|-
| '''High''' || 300 || Eligible, most preferred
|}

'''Automatic Location Constraints''' is ticked by default and is the right choice unless you have a specific reason to pin. What it computes depends on the use case: for Grid Primary and Other it gives '''every''' member of the site cluster an equal weight of 100; for scale-up it weights the HA failover group's primary, secondary and tertiary appliances at 100; for scale-out it weights every Ceph cluster member at 100. Any remaining site cluster member is set to None. Note that automatic mode is '''flat''' -- it does not rank the members it selects, so it expresses "these are eligible" rather than "prefer this one".

Clearing the checkbox enables the grid. Two things are needed per member, not one: '''set the weight and tick the row'''. The dialog reads the ticked rows, so a weight set on an unticked row is not submitted, and clearing Automatic without ticking anything fails validation with a message telling you to select and configure the node weights.

At least one member must have a weight above None. The appliance whose port you chose on the Virtual Interface tab must be one of them -- a VIF cannot be created pinned away from the port it is being attached to.

From the command line, weights are given to <code>[[QuantaStor CLI Command Reference#site-vif-modify|qs site-vif-modify]]</code> as {{Code|1=<system>:<weight>}} pairs, and any non-negative number is accepted rather than only the four the Web UI offers:

<pre style="font-size: smaller">
qs site-vif-modify --vif-resource=<vif-id> \
--location-config=qs-node-110:300,qs-node-111:200,qs-node-112:0
</pre>

Read them back with <code>[[QuantaStor CLI Command Reference#site-vif-location-constraint-list|qs site-vif-location-constraint-list]]</code>, or for one member with <code>[[QuantaStor CLI Command Reference#site-vif-location-constraint-get|qs site-vif-location-constraint-get]] --vif-resource=<vif> --storage-system=<system></code>. A weight given for one member leaves the others as they are, so you can adjust a single appliance without restating the whole set.

=== Raising a weight does not usually move a running VIF ===

This is the least obvious thing on this page, and it looks like a bug when you meet it. QuantaStor sets pacemaker's '''resource stickiness to 1000''' on every site cluster, which is a preference for leaving a running resource where it is. All four Web UI weights are well below that, so '''changing weights through the Web UI will not relocate a VIF that is already running.''' It changes where the VIF will land the next time it has to be placed.

Verified on a running VIF: with the hosting member at Medium (200) and another member raised to High (300), the VIF stayed put. Only when a weight was set above 1000 from the command line did it relocate. If you want a running VIF on a particular appliance now, move it -- do not raise its weight and wait.

The corresponding useful case is '''None''', which does take effect immediately, because −INFINITY beats stickiness. Setting a member to None will push a VIF off it.

== Standby mode and maintenance mode ==

Both take a site cluster out of normal operation, and they do close to opposite things to VIFs. Choosing the wrong one is the most consequential mistake in this area.

'''Standby mode is per appliance, and it evacuates.''' Putting a member into standby moves its resources to a healthy member and stops it receiving any more, while the rest of the cluster keeps protecting itself normally. This is what you want before working on one appliance. In a lab measurement, a member hosting a VIF was put into standby and the address moved to another member with 2.49 s of unreachability. Standby has three states -- Active, Standby Auto Activation and Standby Manual Activation -- and [[Configure Member Standby]] covers the dialog and the difference between them.

'''Maintenance mode is per site cluster, and it freezes.''' It tells pacemaker to stop managing resources across the whole site cluster. Nothing moves, nothing is monitored, and nothing recovers. Existing VIFs stay exactly where they are and keep serving traffic; they simply stop being protected. The Web UI warns about this when you enter it, and the effect is visible in both {{Code|1=crm_mon -1}} and the VIF's own state:

<pre style="font-size: smaller">
*** Resource management is DISABLED ***
The cluster will not attempt to start, stop or recover services

Active Resources:
* gm (ocf:heartbeat:IPaddr2): Started qs-node-111 (unmanaged)
</pre>

Verified live: while the site cluster was in maintenance mode the VIF reported state Warning with '''Is Unmanaged''' true, stayed on its member, and an attempted move failed. Taking the cluster out of maintenance mode returned it to Normal without moving it.

So maintenance mode suppresses cluster activity and alerting for planned work across the whole cluster -- and while it is on, '''an appliance failure will not fail anything over.''' Keep the window short, and use standby mode instead if you only need to work on one appliance.

{{Navigation|High-availability VIF Management → Site Clusters → Site Cluster ''(select + right-click)'' → Enter Maintenance Mode...}}

From the command line, with <code>[[QuantaStor CLI Command Reference#site-cluster-toggle-maintenance-mode|qs site-cluster-toggle-maintenance-mode]]</code> and <code>[[QuantaStor CLI Command Reference#site-cluster-set-standby-mode|qs site-cluster-set-standby-mode]]</code>:

<pre style="font-size: smaller">
qs site-cluster-toggle-maintenance-mode --site=site-cluster-1 --enable-maintenance-mode=true
qs site-cluster-set-standby-mode --site=site-cluster-1 --storage-system=qs-node-111 \
--standby-mode=standby-manual-activate
</pre>

The standby modes are named <code>active</code>, <code>standby-auto-activate</code> and <code>standby-manual-activate</code> on the command line.

== What happens to client connections ==

A VIF move is not a graceful handover. The address is removed from one appliance and added to another, so '''every TCP connection bound to that address is broken''' and clients have to reconnect. Nothing drains sessions, logs initiators out, or unexports a share first.

What QuantaStor does do is announce the new location quickly: as soon as the address comes up, the resource agent broadcasts a burst of gratuitous ARP -- five packets at 200 ms intervals by default -- so switches and clients on the segment learn the new MAC without waiting for their caches to expire. That is why the measured outage is a couple of seconds rather than minutes.

How much a client notices depends entirely on the client. Recovery is the client's business, not the appliance's, and it varies by protocol and by client configuration -- a hard NFS mount and an iSCSI initiator with a short timeout behave very differently across the same two-second outage. Test your own clients against a deliberate move before relying on the behaviour in production; a planned move is the cheapest way to find out what a real failover will look like. For what clients are connecting to, see [[NFS Configuration]] and [[Network Shares]].

Note that the '''iSCSI Portal''' and '''NVMeoF Portal''' flags on the VIF are what make the floating address usable as a target portal at all. Without them the address moves, but block initiators were never pointed at it.

The practical consequence is that a move is cheap but not free. Schedule one the way you would schedule a brief service restart, and prefer moving a VIF deliberately at a quiet moment over letting a failure move it at a busy one.

Deleting a VIF has the same effect on connections as a move, without the reconnect target -- the Remove dialog says so explicitly.

=== SMB clients and Active Directory: giving a VIF name its own Kerberos identity ===

When the appliances are joined to Active Directory (see [[Active Directory Configuration]]), SMB clients that connect by a VIF's host name rather than by an appliance's own name need one extra piece of setup. A VIF is a floating IP address with a DNS name and no computer account of its own, so Active Directory has no {{Code|1=cifs/<vif-name>}} service principal name (SPN) to issue a Kerberos service ticket against. In a domain where NTLM is disabled, a client connecting to {{Code|1=\\<vif-name>\<share>}} therefore cannot authenticate at all.

The fix is to give the VIF name its own AD computer object with {{Code|1=cifs/}} SPNs, and put that object's keys in {{Code|1=/etc/krb5.keytab}} on '''every appliance the VIF can run on''' -- for a scale-up VIF, every member of the HA failover group. '''Nothing happens in Active Directory at failover.''' Every eligible appliance holds the VIF's keys all the time, so whichever one currently hosts the address can answer for its name. An appliance that is missing the keys fails Kerberos authentication by the VIF name whenever the VIF lands on it, so the keys have to be merged everywhere, not just on the current owner.

Two pieces make this work:

* '''The domain join prepares Samba.''' Joining the domain sets {{Code|1=kerberos method = dedicated keytab}} and {{Code|1=dedicated keytab file = /etc/krb5.keytab}} in {{Code|1=/etc/samba/smb.conf}}. With the default {{Code|1=system keytab}} method, smbd keeps only the principals that match the appliance's own names and silently ignores a VIF's entries. Leaving the domain removes both settings and deletes {{Code|1=/etc/krb5.keytab}}.
* '''A script creates and distributes the VIF's identity.''' {{Code|1=/opt/osnexus/quantastor/bin/qs_vif_kerberos_setup.sh}} creates the AD computer object, exports its keytab, and merges it into {{Code|1=/etc/krb5.keytab}}. Nothing in the Web UI or the {{Code|1=qs}} CLI does this; run the script as root on an appliance.

==== Before you start ====

* Every appliance the VIF can run on is joined to the domain.
* {{Code|1=msktutil}} and {{Code|1=krb5-user}} are installed. They are recommended rather than required packages, so they can be missing; the script stops and names whichever one is absent.
* You have an AD account that can create computer objects. A machine account cannot do it, because AD refuses an SPN whose host part is not the account's own name.
* The VIF name is 15 characters or fewer -- the NetBIOS/sAMAccountName limit.
* Clients can resolve the VIF's FQDN to the VIF's IP address. Create a DNS A record, or an entry in the clients' hosts files. The '''FQDN''' field on the cluster VIF (see [[Cluster VIFs]]) adds the name to {{Code|1=/etc/hosts}} on every appliance in the grid so that internal services, Kerberos included, can resolve it, but it does not publish anything to your clients' DNS.

==== Creating the VIF's identity ====

Run {{Code|1=create}} on any one appliance and list the others with {{Code|1=--nodes}}:

<pre style="font-size: smaller">
/opt/osnexus/quantastor/bin/qs_vif_kerberos_setup.sh create --vif-name sharevif \
--ad-admin Administrator --nodes qs-node-111,qs-node-112
</pre>

The script prompts for the AD account's password, or reads it from the {{Code|1=QS_AD_ADMIN_PASSWORD}} environment variable. It never accepts the password as an argument, because the command line is visible in the process table. Then it:

# Reads the realm from the appliance's Samba configuration, so you do not retype it.
# Checks that AD does not already have an account named {{Code|1=<vif-name>$}}, and stops before changing anything if it does (see below).
# Creates the computer object with two SPNs, {{Code|1=cifs/<vif-name>}} and {{Code|1=cifs/<fqdn>}}. The FQDN defaults to the VIF name followed by the realm in lower case; set {{Code|1=--fqdn}} if the name clients use is different. Only AES128 and AES256 keys are issued; RC4 is left out because it is deprecated and AD may refuse it.
# Merges the keytab into the local {{Code|1=/etc/krb5.keytab}}, keeping a timestamped backup, and asks smbd to reload its configuration rather than restarting it, so live SMB sessions are not dropped.
# Verifies the local appliance (see below).
# Copies the keytab to each appliance in {{Code|1=--nodes}} over root SSH, merges it there and deletes the copy.

QuantaStor appliances do not have root SSH trust between them by default, so the copy step usually cannot finish on its own. When it cannot, the script keeps the exported keytab and prints the exact {{Code|1=scp}} and {{Code|1=merge}} commands to run for each remaining appliance. Run them, then delete the keytab file -- it holds the VIF account's keys. Without {{Code|1=--nodes}}, only the local appliance is updated and the same instructions are printed.

On an appliance where you copied the keytab by hand:

<pre style="font-size: smaller">
/opt/osnexus/quantastor/bin/qs_vif_kerberos_setup.sh merge --keytab /tmp/qs_vif_keytab.XXXXXX
</pre>

A merge is safe to repeat: if every entry is already present it does nothing.

==== Verifying every appliance ====

Run {{Code|1=verify}} on each appliance the VIF can run on:

<pre style="font-size: smaller">
/opt/osnexus/quantastor/bin/qs_vif_kerberos_setup.sh verify --vif-name sharevif
</pre>

It checks three things, and fails if any of them is wrong:

* The appliance's keytab holds {{Code|1=cifs/<vif-name>}} entries.
* The keys are the ones Active Directory currently issues tickets with. It asks the KDC for a service ticket for each SPN, authenticating as the appliance's own machine account, and checks that the keytab can decrypt it. This catches keys that are present but out of date, which counting entries does not.
* Samba is using {{Code|1=kerberos method = dedicated keytab}}. If it is not, the VIF's entries are ignored and SMB by the VIF name fails with {{Code|1=NT_STATUS_LOGON_FAILURE}}. Leave the domain and join it again so the join sets the method, then repeat {{Code|1=create --rotate}}, because leaving deletes the keytab.

If verify reports stale keys straight after a {{Code|1=create}}, the new computer object may not have replicated to every domain controller yet. Wait a minute and run it again.

==== Rotating, or recovering after a domain leave ====

Running {{Code|1=create}} for a name AD already has would change that account's keys, and every appliance still holding the old keys would immediately start rejecting tickets for the VIF name. So the script refuses, and changes nothing. If the appliances already hold the keys there is nothing to do; confirm with {{Code|1=verify}}.

You need new keys when an appliance has lost them -- leaving the domain deletes {{Code|1=/etc/krb5.keytab}} outright -- or when you rotate them deliberately. Re-run {{Code|1=create}} with {{Code|1=--rotate}}, and list '''every''' other appliance the VIF can run on in {{Code|1=--nodes}}: rotating replaces the keys in AD, so an appliance that does not merge the new keytab fails Kerberos authentication by the VIF name until it does. When a merge brings in newer keys, the script prints a warning that names each changed principal.

After any domain leave and re-join, run {{Code|1=verify}} on that appliance.

==== What QuantaStor refreshes automatically ====

The VIF's keys belong to its own AD computer object and do not change unless you rotate them. The appliance's '''own''' machine account is different: Active Directory rotates its password, and with the dedicated keytab method Samba does not rewrite the appliance's own keytab entries when that happens, so access by the appliance's own host name could fail. QuantaStor refreshes those entries itself, using the machine account credentials, without an administrator password:

* once a day,
* on the first pass after the QuantaStor service starts, so an appliance that was down through a rotation catches up straight away,
* whenever storage pools are imported, and during an HA pool failover on the appliance taking the pool over, so the node that has just picked up the VIF re-checks its own keytab rather than waiting for the daily cycle.

The refresh adds or updates only the appliance's own principals and leaves the VIF's entries untouched. If it fails, QuantaStor raises an alert saying that SMB access by the appliance's own host name may fail after the next machine password rotation. An appliance joined in SSSD-only mode skips the refresh and logs that it did so.

== Scale-up and scale-out differences ==

The VIF mechanism is the same in both configurations. What differs is what the address follows and where it is allowed to go.

{| class="wikitable"
! !! Scale-up (ZFS HA pool) !! Scale-out (Ceph)
|-
| Use case || Storage Pool (Scale-up HA) || Storage Pool (Scale-out), with config type Object, File or Block
|-
| Associated object || A storage pool HA failover group || A Ceph cluster
|-
| Eligible appliances (automatic) || The HA group's primary, secondary and tertiary || Every Ceph cluster member
|-
| Prerequisite on every eligible appliance || A port of the same name, online, on every site cluster member || The Ceph service the config type names must be running and its storage mounted
|-
| Relationship to the data || The address follows the pool; both move together || The data is already distributed. The address is a service endpoint, and moving it moves no data
|-
| What a failover costs || Pool import time on the new appliance, plus the address move || The address move only
|}

The important asymmetry is the last row. In scale-up, a VIF failover is usually part of a pool failover, and the pool import dominates the time -- the VIF is the small part. In scale-out there is nothing to import, so the numbers in [[#How long a failover takes|How long a failover takes]] are close to the whole story.

Scale-up VIFs are ordered and colocated with the pool they belong to, which is what keeps the address and the pool on the same appliance. They also appear with an {{Code|1=:ha}} tag rather than {{Code|1=:sv}}, and they are created from the HA failover group; see [[Storage Pool HA Failover Interface Create]], [[HA Cluster Setup (JBODs)]] and [[HA Cluster Setup (external SAN)]].

For scale-out, note that a site cluster is a '''prerequisite''', not an optional extra: without one there is no cluster software to run the resource, so a Ceph cluster on its own cannot have a floating address. Set up the site cluster across the Ceph members first. See [[Scale-out File Setup (ceph)]], [[Scale-out Block Setup (ceph)]] and [[Scale-out Object Setup (ceph)]].

Cluster VIFs are also the preferred way to present Ceph iSCSI, because routing all the SCSI reservation traffic for a target through one floating address keeps it going through a single target instance.

If a VIF's address is used to activate replication schedules, arrival of the VIF on an appliance is what activates them there; see [[Remote-replication (DR)]].

== Removing a cluster VIF ==

{{Navigation|High-availability VIF Management → Site Cluster Virtual Interfaces → Remove Cluster VIF ''(toolbar)''}}

Removing a cluster VIF deletes the pacemaker resource and takes the address down. The dialog warns that active connections on the interface will be dropped. It also offers '''Convert cluster VIF resource to local virtual IP''', which is meant to keep the address in service as an ordinary virtual interface on the appliance it was last running on. It currently does not: the address is dropped, as with a plain removal, so do not rely on it to keep an address in service.

<pre style="font-size: smaller">
qs site-vif-delete --vif-resource=<vif-id>
qs site-vif-delete --vif-resource=<vif-id> --convert-to-vif=true
</pre>

Remove the VIFs before deleting the site cluster that owns them. <code>[[QuantaStor CLI Command Reference#site-cluster-delete|qs site-cluster-delete]]</code> refuses while VIFs still reference the site cluster, and says so, precisely to stop a site cluster teardown taking a service address off the network unannounced.

If the VIF had its own Active Directory identity, its computer object stays in AD after the VIF is removed; QuantaStor does not delete it.

== Troubleshooting ==

'''The Web UI and the cluster disagree about a VIF.''' If the VIF list shows a state that {{Code|1=crm_mon -1}} contradicts, run a site cluster rescan. It re-reads the live cluster configuration and rebuilds QuantaStor's view of the rings, the VIFs and their location constraints from it, treating pacemaker as authoritative. It does not stop or restart anything.

The command, <code>[[QuantaStor CLI Command Reference#site-cluster-rescan|qs site-cluster-rescan]]</code>, takes no arguments and acts on the site cluster the appliance you run it on belongs to, so run it on a member:

<pre style="font-size: smaller">
qs site-cluster-rescan
</pre>

'''A VIF reports MISSING.''' QuantaStor has a VIF object but pacemaker has no resource for it. A rescan will reconcile the record.

'''A VIF will not start anywhere.''' Read the failed actions in {{Code|1=crm_mon -1}} first -- one of the checks in [[#What blocks a failover|What blocks a failover]] will normally name the cause, and each names its own remedy. The most common is the duplicate-address check firing because an appliance that lost cluster communication is still holding the address.

'''A VIF is stuck and reports unmanaged.''' The site cluster is in maintenance mode. Exit maintenance mode and the VIF is managed again.

'''A VIF will not move to a particular appliance.''' Check its location constraint on that appliance. None pins it away permanently, and the error message says so.

'''SMB by the VIF name works on one appliance but fails after a failover.''' The appliance now hosting the VIF does not hold the VIF's current keys. Run {{Code|1=qs_vif_kerberos_setup.sh verify}} there; see [[#SMB clients and Active Directory: giving a VIF name its own Kerberos identity|SMB clients and Active Directory]].

== Command line reference ==

{| class="wikitable"
! Command !! Purpose
|-
| <code>[[QuantaStor CLI Command Reference#site-vif-create|qs site-vif-create]]</code> || Create a cluster VIF, optionally converting an existing local one
|-
| <code>[[QuantaStor CLI Command Reference#site-vif-list|qs site-vif-list]]</code> / <code>[[QuantaStor CLI Command Reference#site-vif-get|qs site-vif-get]]</code> || List cluster VIFs, or show one in detail
|-
| <code>[[QuantaStor CLI Command Reference#site-vif-modify|qs site-vif-modify]]</code> || Change the description or the location constraints
|-
| <code>[[QuantaStor CLI Command Reference#site-vif-move|qs site-vif-move]]</code> || Move a VIF to another member deliberately
|-
| <code>[[QuantaStor CLI Command Reference#site-vif-delete|qs site-vif-delete]]</code> || Remove a VIF, optionally converting it to a local address
|-
| <code>[[QuantaStor CLI Command Reference#site-vif-location-constraint-list|qs site-vif-location-constraint-list]]</code> || List the weights for every VIF
|-
| <code>[[QuantaStor CLI Command Reference#site-vif-location-constraint-get|qs site-vif-location-constraint-get]]</code> || Show one member's weight for one VIF
|-
| <code>[[QuantaStor CLI Command Reference#site-cluster-set-standby-mode|qs site-cluster-set-standby-mode]]</code> || Put one member into or out of standby
|-
| <code>[[QuantaStor CLI Command Reference#site-cluster-toggle-maintenance-mode|qs site-cluster-toggle-maintenance-mode]]</code> || Freeze or unfreeze the whole site cluster
|-
| <code>[[QuantaStor CLI Command Reference#site-cluster-restart-services|qs site-cluster-restart-services]]</code> || Restart corosync and pacemaker on one appliance
|-
| <code>[[QuantaStor CLI Command Reference#site-cluster-rescan|qs site-cluster-rescan]]</code> || Rebuild QuantaStor's view from the live cluster configuration
|}

Site cluster and heartbeat ring commands are covered on [[Site Cluster Setup]].

== Related pages ==

* [[Cluster VIFs]] -- adding a cluster VIF, and the four use case types in depth
* [[Site Cluster Setup]] -- the site cluster object, its members and its heartbeat rings
* [[Network Ports]] -- virtual interfaces, bonds, VLANs and static routes at the port level
* [[Grid Configuration]] -- the management grid a site cluster is built from
* [[Configure Member Standby]] -- the standby mode dialog and its three states
* [[HA Cluster Setup (JBODs)]] and [[HA Cluster Setup (external SAN)]] -- scale-up HA pool setup
* [[Storage Pool HA Failover Interface Create]] -- creating a VIF from an HA failover group
* [[Scale-out File Setup (ceph)]], [[Scale-out Block Setup (ceph)]], [[Scale-out Object Setup (ceph)]] -- scale-out configurations
* [[NFS Configuration]] and [[Network Shares]] -- what clients connect to through a VIF
* [[Active Directory Configuration]] -- joining the appliances to Active Directory
* [[Remote-replication (DR)]] -- replication schedules activated by a VIF
* [[QuantaStor CLI Command Reference]] -- full argument lists for the commands above

[[Category:admin_guide]]

----
''Verified against QuantaStor 6.9.0.''

HA Cluster Setup (JBODs)

2026-10-07T20:00:14Z

Qadmin: osn-seo-utilities: docs-manual-planned-pool-failover @ d066481c2606 (approved in the portal)

[[Category:admin_guide]]
__TOC__

A scale-up high-availability cluster is a pair (or trio) of QuantaStor appliances cabled to the same SAS JBOD, running a ZFS storage pool that either appliance can import. If the appliance hosting the pool fails, the pool, its virtual IP address, and every share and volume on it move to a surviving appliance automatically. This page covers the JBOD-attached build: hardware, cabling, the HA group and its virtual interface, failover, and I/O fencing. For the same design with a third-party SAN behind it instead of a JBOD, see [[HA Cluster Setup (external SAN)]].

{| class="wikitable"
! Section !! Purpose
|-
| [[#What a scale-up HA cluster is|What a scale-up HA cluster is]] || The design, and when to choose it over scale-out
|-
| [[#Hardware requirements|Hardware requirements]] || HBAs, media, and what "shared" has to mean
|-
| [[#Cabling|Cabling]] || Rules and per-enclosure-count diagrams
|-
| [[#Before you build the HA group|Before you build the HA group]] || Grid, site cluster, network, and the pool
|-
| [[#Creating the storage pool HA group|Creating the storage pool HA group]] || The dialog, every field, and what the create checks reject
|-
| [[#The HA virtual interface|The HA virtual interface]] || The address clients use, and how it is named
|-
| [[#Activating and deactivating the group|Activating and deactivating the group]] || Turning automatic failover on and off
|-
| [[#Failover|Failover]] || Automatic and triggered, what happens in what order, how long it takes
|-
| [[#Manual (planned) pool failover|Manual (planned) pool failover]] || Moving a pool by hand: the dialog, the health checks, the CLI, and what refuses a move
|-
| [[#I/O fencing and SCSI-3 persistent reservations|I/O fencing and SCSI-3 persistent reservations]] || How QuantaStor guarantees one owner, and how to read the reservations
|-
| [[#Maintenance|Maintenance]] || Standby and maintenance mode
|-
| [[#Troubleshooting|Troubleshooting]] || A pool that will not import, a group that will not form, and where the recovery procedures live
|-
| [[#Command line reference|Command line reference]] || Every HA group command
|}

== What a scale-up HA cluster is ==

[[File:qs4_clustered_jbod.png|thumb|right|318px|Two appliances sharing one JBOD. The pool lives on the disks in the enclosure, not in either server.]]

QuantaStor offers two independent routes to high availability, and they are not variations on one another.

'''Scale-up''' puts the storage outside the appliances, in a shared SAS JBOD, and makes the appliances interchangeable front ends for it. A ZFS storage pool is built from disks in the enclosure and imported by exactly one appliance at a time. Redundancy against disk failure comes from the pool's RAID layout; redundancy against appliance failure comes from the second appliance being able to import the same pool. This is the right choice when you want the capacity efficiency and the feature set of ZFS -- compression, snapshots, RAIDZ2 and RAIDZ3 -- with node-level availability on top, and when the whole cluster fits in one rack.

'''Scale-out''' replicates data across independent appliances with no shared enclosure, and is documented under [[Scale-out Block Setup (ceph)]], [[Scale-out File Setup (ceph)]] and [[Scale-out Object Setup (ceph)]]. Choose it when you need to grow past what one JBOD chain can hold, or when you cannot rely on a shared enclosure.

The rest of this page is about the scale-up case.

Three QuantaStor objects make it work, and they are created in this order:

* A '''site cluster''' -- the corosync and pacemaker layer that carries the heartbeat between appliances and decides when one has gone away. One site cluster serves any number of pools. See [[Site Cluster Setup]].
* A '''storage pool HA group''' -- the object that ties one ZFS pool to the appliances allowed to import it, holds the failover policies, and is the thing you act on to fail a pool over.
* One or more '''HA virtual interfaces''' -- the IP addresses clients connect to, which move with the pool.

== Hardware requirements ==

[[File:qs_clustered_jbod_minimum_hardware.png|thumb|right|419px|Minimum layout: two appliances, one shared JBOD, pool disks only in the enclosure.]]

'''Every disk in the pool must be in the shared enclosure.''' That includes cache, log and hot-spare devices. A pool that draws even one device from a disk internal to one of the appliances cannot be made highly available, because the surviving appliance would be unable to import it. QuantaStor enforces this: creating the HA group verifies that every one of the pool's devices is visible on the secondary appliance, and refuses the create if any are missing.

'''Connect the enclosure with HBAs, not RAID controllers.''' Scale-up pools are HBA-only. ZFS needs unmediated access to the drives, and I/O fencing needs to issue SCSI-3 persistent reservation commands straight to them, which a RAID controller presenting virtual drives does not permit. Use Broadcom 9300, 9400 or 9500 series HBAs or the OEM equivalents; HPE servers attached to HPE enclosures should use the HPE OEM HBAs. A hardware RAID controller is still the right choice for the mirrored boot devices, which are internal to each appliance and are not part of any pool.

'''Pool media must be dual-ported and must support persistent reservations.''' In practice that means dual-port SAS, NL-SAS or dual-port NVMe. SATA drives are not supported for scale-up HA pools and QuantaStor rejects them by name at HA group creation:

{{Code|1=Storage Pool contains one or more SATA disk devices including '<device>'. Storage Pool High-Availability feature requires that all disks devices be dual-port SAS, dual-port NVMe, or FC devices.}}

Single-ported media in a shared enclosure is a subtler failure, because the pool will build and run. QuantaStor detects it and raises a multipath configuration problem alert against the pool; [[Multipath Configuration]] documents that alert and why it is a correctness problem rather than a warning to acknowledge.

'''What "dual-connected" actually requires.''' Each disk's two SAS ports are wired to different places, and what you wire them to decides what survives. In the common two-appliance, one-JBOD build, each disk sends one port to each appliance: the sharing is at the appliance level, each appliance sees a single path to each disk, and a disk showing one path in {{Code|1=multipath -ll}} is expected rather than a fault. Path-level redundancy on top of that needs a second HBA connection from each appliance into a second SAS expander in the enclosure, which is why enclosures with dual expanders are the ones to buy. [[Multipath Configuration]] covers how QuantaStor names and stacks these devices.

=== Minimum configuration ===

* 2x QuantaStor appliances acting as storage pool controllers
* 1x or more SAS JBOD cabled to both appliances
* 2x to 100x dual-port SAS HDDs or SSDs for pool storage, all of them in the shared enclosure
* 1x hardware RAID controller per appliance for mirrored boot devices
* 2x boot devices per appliance; 480GB or larger SSD or NVMe media is recommended, because a boot device that fills up is the most common cause of local database corruption

=== Storage bridge bay systems ===

A cluster-in-a-box or SBB (storage bridge bay) chassis holds two hot-swap servers and a JBOD in a single 2U unit, and QuantaStor supports the SuperMicro SBB models. Everything on this page applies unchanged; the cabling is internal to the chassis. Contact your OSNEXUS reseller or sdr@osnexus.com for hardware options.

== Cabling ==

Incorrect cabling is the single most common cause of I/O fencing and performance problems in a scale-up cluster, and the symptoms rarely point at the cables. Check the wiring against the diagrams before building anything.

* The same rules apply to every enclosure model from every manufacturer.
* All Dell, HPE, WD/HGST, Seagate and Lenovo JBOD units ship with dual SAS expanders.
* SuperMicro sells single-expander enclosures (E1/E1C models) which should not be used for HA. Buy the dual-expander models, which carry E2/E2C in the model number.
* Some enclosures label their ports IN and OUT but accept either; check the vendor documentation.
* Avoid cascading JBODs. It adds failure modes for no benefit.
* Keep SAS cables within the 5 metre standard limit, or use optical SAS cables for longer runs.

Three rules govern which HBA port goes where:

# Never connect the same HBA to the same disk enclosure twice.
# Never connect the same SAS expander to the same HBA twice.
# Every appliance connected to eight or fewer enclosures must be connected to each enclosure twice, using different HBAs.

Unused HBA ports are fine and can be used to attach more enclosures later.

=== Scale-up cabling diagrams ===

==== 2x appliances, 1x disk chassis ====
[[File:jbod2x1.png|thumb|right|505px|QuantaStor HA cluster connectivity to 1x disk chassis]]

==== 2x appliances, 2x disk chassis ====
[[File:jbod2x2.png|thumb|right|506px|QuantaStor HA cluster connectivity to 2x disk chassis]]

==== 2x appliances, 3x disk chassis ====
[[File:jbod2x3.png|thumb|right|505px|QuantaStor HA cluster connectivity to 3x disk chassis]]

==== 2x appliances, 4x disk chassis ====
[[File:jbod2x4.png|thumb|right|505px|QuantaStor HA cluster connectivity to 4x disk chassis]]

==== 2x appliances, 5x disk chassis ====
[[File:jbod2x5.png|thumb|right|520px|QuantaStor HA cluster connectivity to 5x disk chassis]]

==== 2x appliances, 6x disk chassis ====
[[File:jbod2x6.png|thumb|right|520px|QuantaStor HA cluster connectivity to 6x disk chassis]]

==== 2x appliances, 7x disk chassis ====
[[File:Jbod2x7.png|thumb|right|700px|QuantaStor HA cluster connectivity to 7x disk chassis]]

==== 2x appliances, 8x disk chassis ====
[[File:Jbod2x8.png|thumb|right|710px|QuantaStor HA cluster connectivity to 8x disk chassis]]

==== 2x appliances, 9x disk chassis ====
[[File:OSN-WiringDiagram2x9.png|thumb|right|800px|QuantaStor HA cluster connectivity to 9x disk chassis]]

==== 2x appliances, 10x disk chassis ====
[[File:Jbod2x10.png|thumb|right|800px|QuantaStor HA cluster connectivity to 10x disk chassis]]

==== 2x appliances, 11x disk chassis ====
[[File:Jbod2x11.png|thumb|right|800px|QuantaStor HA cluster connectivity to 11x disk chassis]]

==== 2x appliances, 12x disk chassis ====
[[File:Jbod2x12.png|thumb|right|800px|QuantaStor HA cluster connectivity to 12x disk chassis]]

=== Single-appliance expansion cabling ===

These layouts attach expansion chassis to one appliance. They are not HA configurations -- there is no second appliance to fail over to -- and are included here because the cabling rules are the same.

==== 1x appliance, 1x expansion chassis ====
[[File:Jbod1x1.png|thumb|right|165px|QuantaStor connectivity to 1x expansion chassis]]

==== 1x appliance, 2x expansion chassis ====
[[File:Jbod1x2.png|thumb|right|185px|QuantaStor connectivity to 2x expansion chassis]]

==== 1x appliance, 3x expansion chassis ====
[[File:Jbod1x3.png|thumb|right|280px|QuantaStor connectivity to 3x expansion chassis]]

== Before you build the HA group ==

Four things have to be in place first, in this order.

'''Licence both appliances.''' Each appliance needs its own unique Gold, Platinum or Cloud Edition key. See [[License Management]].

'''Put both appliances in one storage grid.''' An HA group can only span grid members. Grid creation takes under a minute; see [[Grid Configuration]].

'''Configure the networks.''' Set static addresses on every port you intend to use, set DNS and NTP, and keep the heartbeat networks separate from client traffic. The site cluster requires that the port carrying a heartbeat ring has the '''same interface name''' on every appliance, so plan the naming before you cable. [[Network Ports]] covers port configuration; [[Site Cluster Setup]] covers the heartbeat requirements in detail.

'''Create the site cluster, with two heartbeat rings.''' The site cluster is what detects an appliance going away. A single-ring cluster is fragile enough that ordinary network maintenance can trigger a failover, so configure a second ring on a separate subnet -- a direct crossover cable between the two appliances is ideal, because it survives a top-of-rack switch outage. The full procedure is on [[Site Cluster Setup]]; do not build the HA group until the site cluster reports every member online.

'''Then create the pool.''' A scale-up HA pool is created exactly like any other ZFS pool -- see [[Storage Pools]] -- with one constraint: select only disks from the shared enclosure. Before you create it, confirm in the '''Physical Disks''' section that the same disks appear with the same SCSI IDs and serial numbers on both appliances. That shared, identical naming is what makes the pool importable on either side, and if it is missing the HA group create will fail. Create the pool on the appliance you intend to be its primary, although that is a convention rather than a requirement.

{{Navigation|Storage Management → Physical Disks ''(section)''}}

For parity layouts, use at least double parity. RAIDZ2 or RAIDZ3 leaves the pool with error-correction capability while a failed device is being replaced, and -- unlike RAID10 -- cannot start on half its devices, which removes even the theoretical possibility of a split-brain.

== Creating the storage pool HA group ==

[[File:hajbod_ha_toolbar.png|thumb|right|358px|The Storage Pool HA Resource Group toolbar group, under Storage Management.]]

The storage pool HA group associates one pool with the appliances allowed to import it, carries the failover policies, and owns the pool's virtual interfaces. It is the object you activate, deactivate and fail over.

{{Navigation|Storage Management → Storage Pools ''(section)'' → ''select the pool'' → Storage Pool HA Resource Group ''(toolbar group)'' → Create Group}}

Existing groups are listed on the '''Storage Pool HA Groups''' tab in the centre pane, which shows each group's state, the appliance currently hosting it, and its connectivity and link-state policies.

[[File:hajbod_ha_groups_tab.png|thumb|right|800px|The Storage Pool HA Groups tab lists each group, its state, and the appliance hosting it.]]

=== General tab ===

[[File:hajbod_ha_group_create.png|thumb|right|470px|Creating an HA group. Tertiary System is greyed out because the site cluster has only two members.]]

* '''Name''' -- prefilled as the pool name with {{Code|1=-ha-group}} appended, so a pool named {{Code|1=pool-2}} produces {{Code|1=pool-2-ha-group}}. Keep the pool name in it; the group is what you will be looking for during an incident.
* '''Description''' -- optional.
* '''Storage Pool''' -- the pool to protect. Only ZFS pools are listed, and a pool that already belongs to a group is rejected on OK.
* '''Export Timeout (seconds)''' -- default 50. This is how long the appliance losing the pool is given to export it during a failover. If it overruns, the acquiring appliance stops waiting and takes the devices preemptively, which gets the pool back into service but leaves the exporting appliance needing a reboot to clear its I/O stack. Raise it for a pool with many volumes and shares, where a clean export legitimately takes longer than a smaller one.
* '''Primary System''' -- read-only, and set to the appliance the selected pool is currently imported on.
* '''Secondary System''' -- the appliance that will take the pool. Only members of the same site cluster as the primary are offered.
* '''Tertiary System''' -- an optional third failover target, gated behind its own checkbox. The checkbox stays greyed out unless the site cluster has at least three members, which is why it is disabled on a two-appliance cluster.
* '''Clustered SCSI3-PR Reservations''' -- '''Enabled''' or '''Disabled''', defaulting to '''Disabled''' for a new group. Turn it on for Windows failover clusters; see [[#Distributed locking for clustered clients|Distributed locking for clustered clients]] below.
* '''Allow Reboot / Lock Recovery''' -- unticked by default. Lets an appliance that has been fenced away from a pool restart itself to recover; see [[Clustered SCSI-3 Persistent Reservations#Allow Reboot / Lock Recovery|Allow Reboot / Lock Recovery]].
* '''Force''' -- unticked by default. Ticking it relaxes the device connectivity check from "every pool device is visible on the target" to "a majority of them are". It also allows the group to be created while FC sessions are active, which enabling HA would otherwise drop as the pool switches to FC ALUA mode. Leave it off for a first build: a device that is not visible on the secondary is a cabling fault to fix, not a check to bypass.

=== Connectivity tab ===

[[File:hajbod_ha_group_conn.png|thumb|right|470px|The Connectivity tab. The client-IP controls stay inert until Enable Client Connectivity Checks is ticked.]]

This tab decides when QuantaStor should fail a pool over '''preemptively''' -- that is, while the hosting appliance is still alive but has lost the connectivity that makes it useful.

* '''Enforce SMB HA VIF Access''' and '''Enforce NFS HA VIF Access''' -- off by default. Each restricts SMB and NFS access for the pool's shares to the networks the pool has virtual interfaces on. Turn them on when you need to be certain clients cannot reach the shares through an appliance-local address, because a client that connects that way keeps working right up until a failover and then does not come back.
* '''Ethernet Port Link State Policy''' -- default '''Failover if ALL Ports are Link-Down'''. The other settings are any port down, a majority of ports down, or disabled. "All ports down" is the conservative default; a single flapping port should not move a pool.
* '''FC Port Link State Policy''' -- the same choices, applied to Fibre Channel target ports, and with the same default. It only applies to pools that actually export volumes over FC.
* '''Enable Client Connectivity Checks''' -- off by default. Ticking it enables the two radio buttons and the '''Verify Connectivity''' button below, all of which are inert until it is on. With it enabled, QuantaStor pings the client addresses you list and fails the pool over when they stop answering, on either the '''ALL specified IPs are unresponsive''' or '''MAJORITY of specified IPs are unresponsive''' policy. The addresses must be genuine remote clients: an address belonging to a system in the same grid is rejected, since pinging your own grid tells you nothing about client reachability.
* '''Verify Connectivity''' -- pings the listed addresses now and reports how many answered, so you can confirm the list before relying on it.

=== From the command line ===

<pre style="font-size: smaller">
qs ha-group-create --pool=pool-2 --sys-secondary=qs-node2 \
--export-timeout=50 --enable-cluster-pr=enabled
</pre>

Use [[QuantaStor CLI Command Reference#ha-group-create|qs ha-group-create]] to create the group and [[QuantaStor CLI Command Reference#ha-group-modify|qs ha-group-modify]] to change it afterwards. The CLI takes the same settings under the names {{Code|1=--client-connectivity-check-policy}}, {{Code|1=--port-linkstate-policy}}, {{Code|1=--fc-port-linkstate-policy}}, {{Code|1=--verify-client-ips}}, {{Code|1=--enforce-smb-allowed}} and {{Code|1=--enforce-nfs-allowed}}. Two differences from the dialog are worth knowing: {{Code|1=--sys-primary}} is settable rather than read-only, and {{Code|1=--enable-cluster-pr}} defaults to {{Code|1=auto}} where the dialog defaults a new group to {{Code|1=enabled}}.

=== What the create checks, and what it rejects ===

The create is a long sequence of preconditions, and each one produces a distinct message. Reading the message saves guessing:

* The pool is not ZFS, or already belongs to another HA group.
* The chosen appliances are not all in the same site cluster. A group can still be created when no appliance is in a site cluster at all -- a '''clusterless''' group -- but it cannot carry a virtual interface and cannot fail over automatically, so it is only useful as a way to move a pool by hand.
* Either appliance has I/O fencing disabled: {{Code|1=Storage system '<name>' has I/O fencing disabled, this must be re-enabled before an HA group may be created.}}
* The two appliances' system UUIDs share their first six hex digits. The SCSI reservation key is built from those digits, so identical prefixes would make it impossible to tell which appliance holds the pool.
* Any pool device is a SATA disk.
* Any pool device is not visible on the secondary appliance: {{Code|1=Unable to verify access to '<n>' devices (<scsi ids>) on secondary storage system for storage pool '<pool>'.}}
* The devices do not support the persistent reservations that fencing requires -- QuantaStor registers a key on each one as part of the create, and fails if it cannot.
* Any grid member is running a service version older than the minimum the HA group code requires.

== The HA virtual interface ==

[[File:hajbod_ha_vif_row.png|thumb|right|800px|The pool's virtual interface, shown with the Scale-up Pool use case and bound to its HA group.]]

'''Every client must reach the pool through the HA virtual interface, not through an appliance's own address.''' The virtual interface is a cluster resource that pacemaker moves with the pool, so a client pointed at it finds its data wherever the pool has landed. A client pointed at an appliance-local address works perfectly until the first failover and then does not come back, and that is by far the most common cause of "the failover worked but my clients did not recover".

An HA virtual interface is a cluster VIF with the scale-up pool use case, bound to an HA group. [[Cluster VIFs]] owns the dialog and the full set of use cases; create it there:

{{Navigation|High-availability VIF Management → Site Cluster Virtual Interfaces ''(section)'' → Add Cluster VIF ''(toolbar)''}}

Three things are specific to the scale-up case:

* It needs an IP address of its own, on the client network, not in use anywhere else, and distinct from both appliances' addresses on that network. In practice each HA virtual interface consumes three addresses on its subnet: its own, plus one per appliance on the parent port.
* The parent port name is checked across the whole site cluster, not only on the appliance you selected, because a virtual interface that cannot follow the pool to the secondary appliance defeats the point.
* QuantaStor writes location constraints so the interface is only ever placed on the group's primary, secondary and tertiary appliances.

The interface can also be created from the HA group side, which is what the CLI does:

<pre style="font-size: smaller">
qs ha-interface-create --ha-group=pool-2-ha-group --parent-port=bond0.100 \
--ip-address=192.168.0.124 --netmask=255.255.0.0 --iscsi-enable=true --nvmeof-enable=true
</pre>

See [[QuantaStor CLI Command Reference#ha-interface-create|qs ha-interface-create]], and [[QuantaStor CLI Command Reference#ha-interface-list|qs ha-interface-list]] to check the result. {{Code|1=--convert-vif}} turns an existing appliance-local virtual IP into an HA interface, and the matching {{Code|1=--convert-to-vif}} on [[QuantaStor CLI Command Reference#ha-interface-delete|qs ha-interface-delete]] turns it back -- which is the supported way to keep an address alive while the paired appliance is being rebuilt.

=== How the interface is named ===

A scale-up HA interface is named for its parent port plus a generated tag, in the form {{Code|1=<parent>:<tag>}}. The tag begins with {{Code|1=ha}}, continues with the leading hex digits of the '''pool''' GUID, and ends with an index number. A pool whose GUID starts {{Code|1=d786e0b7}} on parent port {{Code|1=bond0.100}} produces the tag {{Code|1=had70}} and the interface {{Code|1=bond0.100:had70}}. That is how a scale-up interface is told apart from a site VIF, which uses an {{Code|1=sv}} tag instead.

Linux caps an interface name at 15 characters, so the length of the parent port name decides how many GUID digits fit -- and a long parent name can leave no room at all. If the create fails with a name-length error, either rename the parent interface or switch the appliance to {{Code|1=eth}}-prefix naming on the '''Storage System Modify''' dialog.

== Activating and deactivating the group ==

A group must be '''activated''' before automatic failover will happen. Until then the group exists and can be failed over by hand, but the site cluster will not move the pool on its own.

{{Navigation|Storage Management → Storage Pools ''(section)'' → ''select the pool'' → Storage Pool HA Resource Group ''(toolbar group)'' → Activate Group}}

Activating restores the interface location constraints and places the pool's resource group on the appliance currently hosting it. Deactivating disables the failover policies without deleting anything. Two behaviours are worth knowing before you deactivate:

* A group cannot be activated while its site cluster is in maintenance mode. Exit maintenance mode first.
* While a group is deactivated, a manual failover to a '''different''' appliance is refused if the group still has virtual interfaces attached: {{Code|1=Cannot execute HA pool failover to another system while HA Group '<name>' is deactivated and has one or more HA interfaces.}} Re-activate the group, or remove its interfaces, to move the pool.

Use [[QuantaStor CLI Command Reference#ha-group-activate|qs ha-group-activate]] and [[QuantaStor CLI Command Reference#ha-group-deactivate|qs ha-group-deactivate]] from the command line.

== Failover ==

=== Automatic failover ===

Once the group is activated, the site cluster monitors appliance health across every heartbeat ring, and QuantaStor's own checks watch the things a heartbeat cannot see. A failover is triggered when:

* An appliance stops answering on all heartbeat rings.
* The small write test QuantaStor runs against each pool every few seconds fails to complete. This is what catches a lost SAS path or a failed JBOD I/O controller, neither of which stops the appliance answering its heartbeat.
* Network ports fail according to the Ethernet or FC link-state policy on the group.
* The configured client addresses stop answering, if client connectivity checks are enabled.

The '''settle time''' on the group -- 60 seconds by default -- is a cooldown that prevents a second failover from being triggered immediately after one completes.

To move a pool yourself rather than wait for a trigger, see [[#Manual (planned) pool failover|Manual (planned) pool failover]] below. Both kinds of failover run the same sequence once they start.

=== What happens, in order ===

A failover is driven from the appliance receiving the pool, and runs in this sequence:

# Device information is rescanned and the receiving appliance verifies it can see the pool's devices.
# The pool's ALUA state is moved to standby, if ALUA is in use.
# The HA virtual interfaces are moved to the receiving appliance as a pacemaker resource group. A group with no virtual interfaces still fails over, with a warning; only the pool moves.
# The pool is exported on the appliance that had it, within the export timeout.
# SCSI-3 reservations on the pool's devices are preempted, giving the receiving appliance sole write access. If it cannot take ownership the failover stops here rather than importing on top of another appliance's reservation.
# LUKS devices are opened, for an encrypted pool.
# The pool is imported and activated.
# ALUA state is updated and an FC LIP is issued so Fibre Channel clients rescan their paths.
# SMB and NFS configuration, share namespaces and firewall rules are regenerated for the pool on its new host.

=== What determines failover time ===

Most of a failover's duration goes to the steps above that scale with the pool: building SCSI targets for its volumes, exporting and importing it, and reaching agreement with the cluster manager. QuantaStor keeps each of them to the minimum the pool needs.

* '''Only volumes a host can reach get a SCSI target.''' The receiving appliance creates targets and devices for the volumes assigned to a host or host group and skips the rest, so a pool holding many unassigned volumes -- replication destinations, clones kept for later, volumes that back nothing -- fails over as quickly as one without them. Removing the assignments of volumes no host uses any more shortens failover.
* '''Several pools move at the same time.''' When more than one pool fails over at once, as when an appliance fails, each pool's move holds the cluster lock only long enough to submit the move, so the pools proceed side by side instead of queuing behind one another. Each move also touches only its own cluster placement rule, so concurrent failovers do not disturb each other. This is where an appliance carrying several pools gains the most.
* '''A silent appliance is not waited for.''' If the appliance giving up the pool stops responding -- powered off, crashed, or partway through a reboot -- the receiving appliance does not wait out the export timeout. It goes straight to preempting the reservations and importing, which is safe because the reservations fence the old owner off the disks. A reboot of the active appliance therefore fails over about as quickly as a power loss.
* '''Late devices do not hold up the rest of the pool.''' If a volume's device has not appeared yet when the pool is imported, its target is created on a later pass while the other volumes are brought online.
* '''Busy shares do not stall the export.''' The appliance giving up the pool stops sharing its NFS and SMB exports and closes open handles before unmounting. If SMB clients still hold the mounts busy, it briefly blocks the SMB ports, and only as a last resort restarts Samba, so in the usual case SMB sessions to pools that are not moving stay connected.
* '''Encrypted pools failing back skip key derivation''' when the receiving appliance still has the devices open from before.

[[Clustered SCSI-3 Persistent Reservations|Clustered SCSI-3 persistent reservations]] add a step: the receiving appliance joins each volume's lock space before serving it. That cost is one reason the feature is off by default; enable it for HA groups that serve Windows failover clusters.

=== What clients see ===

The failover dialog warns that a failover "can take upwards of 30 seconds or more for larger configurations", and that is the right expectation to set. On a small pool -- four SAS disks, 576GB, RAIDZ2, no client load -- a failover measured end to end took '''44 to 45 seconds''', with the change of ownership visible in {{Code|1=qs pool-list}} about 25 seconds in and the remaining time spent on share, ALUA and firewall reconfiguration. A pool with many volumes and shares takes longer, and the export stage is usually what grows.

For the duration, the pool's storage is unavailable: the virtual interface has moved but the pool behind it has not yet imported. Clients connected through the HA virtual interface see a stall and then recover, which is what iSCSI, NFS and SMB timeouts exist for; clients connected to an appliance-local address see the connection go away and do not recover. Applications with short storage timeouts may need those timeouts raised.

Two operational points follow from that:

* '''Test failover before the cluster carries production data''', in both directions, and confirm the pool lands healthy each time.
* '''Fail back deliberately.''' After an appliance is repaired and rejoins, pools do not return on their own. If you run two pools, one on each appliance, you have to move one back by hand.

== Manual (planned) pool failover ==


A manual failover moves one pool, together with its HA virtual interfaces, to another appliance in its HA group when you choose to, rather than when a failure triggers it. Use it to:

* test the cluster in both directions before it carries production data;
* move a pool off an appliance before you work on it, or fail it back once the appliance is repaired and has rejoined;
* re-run fencing and virtual interface placement for a pool that is not running, by failing it over to the appliance it is already on -- see [[#A pool that will not import|A pool that will not import]].

To empty a whole appliance of every pool it hosts, standby mode is usually simpler than failing each pool over in turn; see [[#Maintenance|Maintenance]].

Once it starts, a manual failover runs the same sequence as an automatic one -- see [[#What happens, in order|What happens, in order]] -- and clients see the same stall, described in [[#What clients see|What clients see]]. Plan it for a time when a pause of 30 seconds or more in storage I/O is acceptable.

=== Starting a manual failover ===

The dialog is reachable from the toolbar or from the pool's right-click menu:

{{Navigation|Storage Management → Storage Pools ''(section)'' → ''select the pool'' → Storage Pool HA Resource Group ''(toolbar group)'' → Manual Failover}}

{{Navigation|Storage Management → Storage Pools ''(section)'' → Storage Pool ''(select + right-click)'' → Execute Storage Pool Failover...}}

The right-click item appears only on pools that belong to an HA group. Both open the '''Execute High-Availability Pool Failover''' dialog. If no HA group exists in the grid, the dialog closes with "No HA groups are available to failover."

=== General tab ===

[[File:hajbod_ha_failover.png|thumb|right|450px|The failover dialog. From System is read-only; To System offers the group's other appliances.]]

* '''HA Failover Group''' -- the group to fail over. When you open the dialog from a pool, the pool's group is preselected; you can pick any other group in the grid here instead.
* '''Storage Pool''' and '''HA Group Status''' -- read-only, under '''High-Availability Failover Group Information'''. They show the pool the group protects and the group's current state. Check the state before you continue: a deactivated group with virtual interfaces cannot be moved to another appliance (see [[#What refuses a manual failover|What refuses a manual failover]]).
* '''From System''' -- read-only, under '''Execute Manual Pool Failover'''. The appliance the pool is imported on now.
* '''To System''' -- the appliance that receives the pool. Only the group's primary, secondary and tertiary appliances are offered, and if the preselected appliance is the one already hosting the pool, the dialog moves the selection to another member. On a two-appliance group that is the other appliance.

Selecting the appliance the pool is already on is allowed. On OK, the dialog asks "The same system was selected for the source and target. Are you sure you want to execute the failover operation?" -- answer Yes only when you mean to re-run the failover sequence in place.

=== Advanced Settings tab ===

[[File:hajbod_ha_failover_adv.png|thumb|right|450px|Advanced Settings. Import checks are on by default, export checks are off.]]

The '''Preliminary Health Checks''' group holds two checkboxes. Both read the failover health statuses QuantaStor records against the HA group for each appliance, before anything moves:

* '''Import Health Checks''' -- ticked by default. Reviews the statuses recorded for the '''target''' appliance that would prevent it receiving the pool.
* '''Export Health Checks''' -- unticked by default. Reviews the statuses recorded for the '''source''' appliance that would prevent it exporting the pool cleanly.

If either check finds a status of severity Critical, the failover stops before anything moves, raises a Critical alert for each status found, and reports, for example:

{{Code|1=1 CRITICAL import health status events found, including '<status title>'. Deselect the 'Import Health Checks' option to continue with failover and ignore these critical health statuses. See alerts for more details.}}

Read the alert before you clear the checkbox: a Critical import status names a condition on the target appliance that would prevent it receiving the pool. If an export is not clean, the receiving appliance still preempts the reservations and imports the pool, and the source appliance is left needing a reboot -- see '''Export Timeout''' under [[#General tab|General tab]].

To see the statuses before you start, run [[QuantaStor CLI Command Reference#ha-group-get-health-status|qs ha-group-get-health-status]] {{Code|1=--pool-list=<pool>}}. To refresh them first, run {{Code|1=qs ha-group-check-health-status --pool-list=<pool>}}, which runs the failover health checks on every appliance in the pool's HA group and returns the results.

=== From the command line ===

<pre style="font-size: smaller">
qs ha-group-failover --ha-group=pool-2-ha-group --storage-system=qs-node2 \
--import-health-checks=true --export-health-checks=false
</pre>

[[QuantaStor CLI Command Reference#ha-group-failover|qs ha-group-failover]] takes four required arguments: {{Code|1=--ha-group}} (name or UUID of the group), {{Code|1=--storage-system}} (the appliance that receives the pool), and {{Code|1=--import-health-checks}} and {{Code|1=--export-health-checks}}, each {{Code|1=true}} or {{Code|1=false}}. The health checks have no default on the command line, so state both. You can run the command from any grid member; QuantaStor hands the request to the receiving appliance, which drives the failover. Without {{Code|1=--flags=async}} the command waits and returns the group object when the failover completes.

One difference from the dialog matters. A failover started from the web interface is always sent as a '''forced''' manual failover. From the CLI it is forced only if you add {{Code|1=--flags=force}}, and without it the receiving appliance applies an extra safety check: if none of the pool's HA virtual interfaces were up on it when the failover started, and another appliance still holds or is registered on the reservations of the pool's devices, the failover stops before taking the devices and raises an alert:

{{Code|1=Stopping failover of storage pool '<pool>': no HA virtual interfaces for this pool were up on the local system when the failover started, and its devices are reserved or registered by another node [<status>]. Taking the devices now would fence the node that is currently serving the pool. Re-run as a manual failover with the force flag to override.}}

That check exists so a failover aimed at the wrong appliance cannot fence off one that is still serving the pool. Confirm which appliance should own the pool -- {{Code|1=qs-iofence devstatus}}, described under [[#I/O fencing and SCSI-3 persistent reservations|I/O fencing and SCSI-3 persistent reservations]], shows who holds the reservations -- then re-run with {{Code|1=--flags=force}} if the move is still what you want. The check does not stop a normal failover: once the receiving appliance has asked the current owner to export the pool, the reservation left on the devices is treated as the old owner's and is preempted.

=== What refuses a manual failover ===

Each of these stops the failover with its own message, and none of them moves anything:

* '''The group is deactivated and has virtual interfaces''', and the target is a different appliance. Re-activate the group, or remove its interfaces; see [[#Activating and deactivating the group|Activating and deactivating the group]].
* '''A failover is already running for the group:''' {{Code|1=Another HA Failover operation is already active for this Storage Pool HA Group '<name>'.}} Wait for the running task to finish.
* '''The pool is encrypted with a user passphrase that the cluster does not hold:''' {{Code|1=User passphrase encrypted storage pool cannot be safely failed over in it's current state. Start pool '<pool>' using the user defined encryption passphrase.}} Start the pool with its passphrase first, so the receiving appliance can open the devices.
* '''The receiving appliance failed an earlier export of this pool''' and still has a reservation conflict on its devices. The appliance is marked as requiring a reboot, and the failover reports that the box must be rebooted before the pool can be re-activated on it. Reboot it, then retry.
* '''A preliminary health check found a Critical status''', as described above.
* '''The receiving appliance cannot see enough of the pool's devices.''' The connectivity check that opens every failover fails, and the import does not start. Compare device visibility on both appliances as described in [[#An HA group that will not form|An HA group that will not form]].

If the failover gets as far as the import and the import fails, the error names the reason where it can: too few of the pool's devices visible on the receiving appliance, devices still reserved by another appliance (with its reservation key), or devices that are known to the appliance but did not answer, such as on an offline path.

== I/O fencing and SCSI-3 persistent reservations ==

Fencing is what makes the guarantee that only one appliance can write to the pool at a time, and it is enforced by the drives themselves rather than by the software. It is the scale-up equivalent of the ARP probing documented on [[High-availability VIF Management]]: both exist to stop two appliances claiming the same resource, one for addresses and one for disks.

When an appliance imports an HA pool it places a SCSI-3 persistent reservation of type '''WERO''' (write exclusive, registrants only) on every device in the pool. The registration key encodes who holds it and what for. It has the form {{Code|1=0xffaaaaaaffbbbbbb}}, where the {{Code|1=ff}}s are separators, {{Code|1=aaaaaa}} is the first six hex digits of the appliance's system UUID, and {{Code|1=bbbbbb}} is the first six of the storage pool's UUID. On NVMe devices the middle separator is a single {{Code|1=f}} and the appliance portion is truncated to five digits, because NVMe registration keys are shorter.

Read the current state with {{Code|1=qs-iofence devstatus}}, which lists every device, its serial number, the keys registered on it, the key holding the reservation, and the reservation type:

<pre style="font-size: smaller">
# qs-iofence devstatus
/dev/sdm 6XP2F4JX0000B2359Y4A (0xff753100ffd786e0) [0xff753100ffd786e0] <WERO>
/dev/sdn 6SE46WWK0000B146RNKP (0xff753100ffd786e0) [0xff753100ffd786e0] <WERO>
/dev/sdo PFVVJUYE (0xff753100ffd786e0) [0xff753100ffd786e0] <WERO>
/dev/sdp 6SE31S1V0000B130JCEV (0xff753100ffd786e0) [0xff753100ffd786e0] <WERO>
</pre>

Every device in the pool should carry the same key, and that key's appliance portion should match the appliance the pool is imported on. Run the command on both appliances: they see the same reservations, because the reservation lives on the drive. After a failover the appliance portion changes and the pool portion does not, which is the quickest confirmation that fencing followed the pool rather than being left behind. A device that shows no reservation while the pool is running, or one whose key names the wrong appliance, is a fencing problem -- resolve it by failing the pool over, including to the appliance it is already on.

QuantaStor also surfaces this in the '''Physical Disks''' section: a black underline on a device icon means the device is correctly fenced to the appliance running the pool, and a red underline means it is not.

Fencing is why cabling rules matter. If a cabling change ever isolated one enclosure to each appliance, reservations placed by one appliance would not reach the other's disks, which is the only realistic route to a split-brain -- and only for mirrored or 2d+2p and 3d+3p layouts, since a RAIDZ2 or RAIDZ3 pool cannot start on half its devices at all.

=== Distributed locking for clustered clients ===

'''Clustered SCSI3-PR Reservations''' on the HA group is separate from the fencing above: it shares the reservations '''clients''' place on the pool's storage volumes across the HA group, so Windows Server Failover Clustering, Hyper-V Cluster Shared Volumes and SQL Server Failover Cluster Instances keep their reservations through a failover. It defaults to Disabled; see [[Clustered SCSI-3 Persistent Reservations]].

== Maintenance ==

To work on one appliance without dismantling the cluster, put it in '''standby mode'''. A standby appliance stays in the site cluster but will not host resources: anything it is running moves to a healthy appliance, and it will not receive a failover. It has an auto-activating form, which returns to active once the appliance is healthy or after a reboot, and a manual form, which stays until you clear it. See [[Configure Member Standby]] for the settings and [[Site Cluster Setup]] for how standby differs from cluster-wide maintenance mode.

{{Navigation|High-availability VIF Management → Site Clusters ''(section)'' → Site Cluster ''(select + right-click)'' → Configure Member Standby...}}

From the command line, [[QuantaStor CLI Command Reference#site-cluster-set-standby-mode|qs site-cluster-set-standby-mode]] takes {{Code|1=--standby-mode=active}}, {{Code|1=standby-auto-activate}} or {{Code|1=standby-manual-activate}}.

Deleting an HA group is the opposite of maintenance and worth stating plainly: it disables automatic failover and removes every virtual interface attached to the group. The pool stays online on whichever appliance is hosting it, but it stops being highly available, and the addresses clients were using disappear.

== Troubleshooting ==

This section covers the two problems specific to building a JBOD-attached cluster. Recovery from a failure in a cluster that is already running -- failed media, a lost enclosure, a lost appliance, a pool that will not start, split-brain, boot media loss, ransomware -- is on '''[[Scale-up HA Storage Pool Troubleshooting]]''', which has a symptom-to-procedure table at the top.

=== An HA group that will not form ===

Work through [[#What the create checks, and what it rejects|What the create checks, and what it rejects]] above -- the create names the reason it refused, and almost every failure is one of those checks. The two that account for most first builds are a device that is not visible on the secondary appliance, and appliances that are not both in the same site cluster.

For the device visibility case, compare the two appliances directly. {{Code|1=qs-util devicemap}} prints each device with its {{Code|1=/dev/disk/by-id}} path, vendor, model and serial number; run it on both appliances and sort the output. The {{Code|1=/dev/sdX}} letters will differ and that does not matter -- the by-id paths and the serial numbers are what must match. [[QuantaStor CLI Command Reference#disk-list|qs disk-list]] shows the same information with the pool each disk belongs to.

=== A pool that will not import ===

Two situations look identical and have different fixes.

'''The pool is on an appliance but not running.''' Most often the HA group is deactivated, or has no virtual interfaces, or the interface addresses are in use elsewhere on the network. Check the group's state first. If the group is deliberately deactivated, run a manual failover from the appliance the pool is already on '''to that same appliance'''. That is more than starting the pool: it re-runs the fencing sequence and places the virtual interfaces, which a plain pool start does not do.

'''The enclosures were powered on after the appliances.''' Nothing will import, because the devices were not present at boot. Power the JBODs on, wait for them to come up, and then run the same failover-to-itself against each affected pool. In the '''Storage Pools''' section the tree label tells you which appliance a pool belongs to -- it reads {{Code|1=pool-2 (on: qs-node2)}} -- so you know which appliance to target.

If the appliance that had the pool needs a reboot, it will say so. An unclean export -- one that overran the export timeout, or a pull of the SAS cables -- leaves in-flight writes in the I/O stack and a pool state that cannot be cleared any other way. Reboot it, let it rejoin the grid, and check its state detail in '''Properties''' before failing anything back to it.

=== Recovery from a failure ===

Every recovery procedure for a running cluster is on '''[[Scale-up HA Storage Pool Troubleshooting]]''': device replacement and hot-spare policy, more failed devices than the layout tolerates, the wrong drive pulled, pool I/O failure, disconnected SAS cables, devices not visible on both appliances, enclosure controller failure, enclosure and appliance power loss, multiple enclosure failures at different times, ungraceful reboot, re-integrating a repaired appliance, top-of-rack switch loss, a pool reported missing or corrupted, an HA pool that will not start, split-brain, boot media failure and reinstall, ransomware, and the {{Code|1=qs-iofence}} and {{Code|1=qs-util}} diagnostics. Those procedures apply whether the shared storage is a JBOD or a SAN, so they are not repeated here.

== Command line reference ==

{| class="wikitable"
! Command !! Purpose
|-
| [[QuantaStor CLI Command Reference#ha-group-create|qs ha-group-create]] || Create an HA group for a pool
|-
| [[QuantaStor CLI Command Reference#ha-group-modify|qs ha-group-modify]] || Change nodes, policies, timeouts or distributed locking
|-
| [[QuantaStor CLI Command Reference#ha-group-list|qs ha-group-list]] || List HA groups and their state
|-
| [[QuantaStor CLI Command Reference#ha-group-get|qs ha-group-get]] || Full detail for one group, including its interfaces and fenced device serials
|-
| [[QuantaStor CLI Command Reference#ha-group-activate|qs ha-group-activate]] || Enable automatic failover
|-
| [[QuantaStor CLI Command Reference#ha-group-deactivate|qs ha-group-deactivate]] || Disable automatic failover
|-
| [[QuantaStor CLI Command Reference#ha-group-failover|qs ha-group-failover]] || Move a pool to another appliance
|-
| [[QuantaStor CLI Command Reference#ha-group-delete|qs ha-group-delete]] || Remove the group and its virtual interfaces
|-
| [[QuantaStor CLI Command Reference#ha-group-get-health-status|qs ha-group-get-health-status]] || Report stored failover health statuses for a pool
|-
| qs ha-group-check-health-status || Run the failover health checks now on every appliance in a pool's HA group
|-
| [[QuantaStor CLI Command Reference#ha-interface-create|qs ha-interface-create]] || Add an HA virtual interface to a group
|-
| [[QuantaStor CLI Command Reference#ha-interface-list|qs ha-interface-list]] || List HA virtual interfaces
|-
| [[QuantaStor CLI Command Reference#ha-interface-get|qs ha-interface-get]] || Detail for one HA virtual interface
|-
| [[QuantaStor CLI Command Reference#ha-interface-delete|qs ha-interface-delete]] || Remove one, optionally converting it to a local virtual IP
|-
| [[QuantaStor CLI Command Reference#site-cluster-set-standby-mode|qs site-cluster-set-standby-mode]] || Put an appliance into or out of standby
|}

== Related pages ==

* [[Scale-up HA Storage Pool Troubleshooting]] -- recovery procedures for every scale-up HA failure scenario
* [[HA Cluster Setup (external SAN)]] -- the same design with SAN-attached storage behind it
* [[Site Cluster Setup]] -- heartbeat rings, cluster membership, maintenance mode
* [[Cluster VIFs]] -- the virtual interface dialog and every use case
* [[High-availability VIF Management]] -- how cluster VIFs are managed and probed
* [[Configure Member Standby]] -- standby settings for one appliance
* [[Multipath Configuration]] -- multipath device naming, path counts, and the single-ported media alert
* [[Hardware Controllers & Enclosures]] -- HBAs, enclosures, slot numbering and drive identify
* [[Storage Pools]] -- creating and managing the pool itself
* [[Physical Disks/Devices]] -- verifying disk visibility and serial numbers
* [[Grid Configuration]] -- building the storage grid
* [[Network Ports]] -- port addressing and naming

----
''Verified against QuantaStor 6.9.0.''

Guides:Fibre Channel and iSCSI Block Storage with ALUA Multipathing

2026-10-07T20:00:04Z

Qadmin: osn-seo-utilities: fc-iscsi-alua @ f86c646a4cec (approved in the portal)

''By Steve Umbehocker, CTO, OSNexus · Updated October 7, 2026''

QuantaStor presents storage volumes as block LUNs over Fibre Channel, through QLogic host bus adapters running in target mode, and over iSCSI, and it reports each path's state to the host with ALUA (Asymmetric Logical Unit Access). On a high-availability storage pool, the node that owns the pool serves a volume's active paths, and the partner node serves standby paths to the same logical unit, so host multipath software always knows which paths to use and switches over cleanly when the pool fails over. This guide explains how those path states behave through a failover, how to design and set up FC and iSCSI access, and what to configure on Linux, VMware and Windows hosts, including Hyper-V failover clusters with Cluster Shared Volumes.

== Why it matters ==

Most virtualization and database estates still run on block storage, and most of those hosts expect two things from the array: more than one path to every LUN, and an honest answer about which of those paths are optimized. Without ALUA, a host either sends I/O down paths that cannot serve it or waits for timeouts to learn that a path is dead. With ALUA, the array tells the host which paths are active and which are standby, and the host's multipath driver (dm-multipath with <code><nowiki>scsi_dh_alua</nowiki></code> on Linux, the in-box Microsoft DSM on Windows, the native multipathing plug-in on VMware ESXi) follows those states.

Clustered hosts add a second requirement. A Windows Server Failover Cluster, Hyper-V Cluster Shared Volumes and other shared-disk clusters arbitrate ownership with SCSI-3 persistent reservations, and those reservations must stay valid when the storage itself fails over from one controller to the other. QuantaStor covers both: ALUA path states for every FC and iSCSI LUN on an HA pool, and clustered SCSI-3 persistent reservations that survive an HA failover.

== How it works ==

=== Fibre Channel target ports ===

QuantaStor's fabric manager scans the FC transport about once a minute and creates an FC target port object for each QLogic port it finds. Any port that is target-mode capable is switched into target mode automatically unless you have explicitly set it to initiator mode, so a dedicated storage appliance needs no per-port configuration. The ports appear in the FC Ports tab of each Storage System, with State, Link State, Status and Mode columns; <code><nowiki>Link Up - F_Port</nowiki></code> in the Link State column means the port negotiated a point-to-point link to a fabric switch, which is what you want. [[Fibre Channel Target Port Management]] covers each column, target and initiator mode, LIPs and the CLI.

Zoning is your job on the switch: QuantaStor cannot configure it. Zone each host initiator WWPN with the target WWPNs on both nodes of the HA pair, so every host sees paths through both controllers.

=== Active and standby paths on an HA pool ===

Each volume on an HA pool is one logical unit with one identity, visible through the target ports of both nodes:

{| class="wikitable"
! Node !! Device behind the LUN !! ALUA state reported !! What the host does
|-
| Pool owner || The volume itself || Active || Sends I/O on these paths
|-
| Partner node || A standby placeholder device with the same identity || Standby || Holds the paths in reserve
|}

The standby device advertises the same thin-provisioning support, UNMAP limits and block geometry as the volume, so a host sees one consistent logical unit no matter which node's path it probes first. That matters for hosts that cache those properties per LUN, such as Windows trim and alignment handling, VMware VAAI UNMAP and dm-multipath discard limits.

During an HA failover the old owner moves its LUNs onto standby devices rather than removing them, and the new owner switches its target group to active only once every LUN is backed by the real volume. Hosts keep the same set of paths throughout; only the states change. A volume's relative target ID also stays the same across failovers, so hosts that track LUNs by target port identity keep their mapping.

After a node restarts, its FC target ports are enabled only once every exported volume is mapped and its ALUA state is in place. When a former pool owner reboots and comes back as the partner, its ports open as soon as its standby devices are ready. Initiators therefore never log in to a port that has no LUNs or unsettled path states behind it. Deleting an HA group removes the partner node's standby devices, ALUA device groups and FC LUN maps, so hosts do not keep standby paths to a node that can no longer serve the pool.

=== Clustered SCSI-3 persistent reservations ===

With clustered reservations enabled on the HA group, both nodes share reservation state through DLM (the Linux distributed lock manager) integrated with the SCST target driver. A reservation a cluster member takes through one node is honored on the other, and it survives a failover in either direction. Volumes enter clustered reservation mode only once they are assigned to a host, so unassigned volumes use no cluster locking resources. See [[Clustered SCSI-3 Persistent Reservations]].

== Design and sizing ==

* '''Pool layout.''' Put block volumes on a storage pool in an HA failover group ([[Create Storage Pool High-Availability Group]]). Only HA pools have a partner node to serve standby paths.
* '''Paths.''' Give each host at least two initiator ports, each zoned (FC) or routed (iSCSI) to both nodes. Two host ports and two target ports per node gives four paths per LUN to each node: active through the owner, standby through the partner.
* '''Port roles.''' Keep the appliance's FC ports in target mode. Use dual mode (<code><nowiki>qs-util enabledualmode</nowiki></code>, then a reboot) only when the appliance must also consume FC storage from another array; [[Dual-mode FC configuration]] describes the procedure.
* '''iSCSI addressing.''' Point iSCSI discovery at an HA virtual interface, which moves with the pool ([[High-availability VIF Management]]).
* '''Block size.''' Choose the volume block size at creation time; it cannot be changed later. For VMware VMFS datastores use 8K to 64K, with 64K recommended for general virtualization workloads and 8K worth considering for databases ([[VMware Configuration]], [[Storage Volumes]]).
* '''Clustered hosts.''' Enable clustered SCSI-3 reservations on any HA group that serves Hyper-V CSVs, WSFC disks or other shared-disk clusters.

== Setting it up ==

# '''Check the FC ports.''' In Storage Management, select a Storage System and open the FC Ports tab. Every port you have cabled should show Mode <code><nowiki>Target</nowiki></code>, Status <code><nowiki>Online</nowiki></code> and a <code><nowiki>Link Up - F_Port</nowiki></code> link state. Ideally you'll see the ports in dual Target+Initiator mode but if they are in just Initiator mode they'll need to be transitioned to Target mode before they can be used to present Storage Volumes as FC LUNs from QuantaStor.

# '''Create the HA group''' for the pool, choose the primary and secondary systems, and set '''Clustered SCSI3-PR Reservations''' if the pool will serve a host cluster. Enabling or disabling it later from Modify Group brings every device of the pool into line on both nodes ([[Modify Storage Pool High-Availability Group]]).

[[File:Guide-fc-iscsi-alua-fc-iscsi-alua-ha-group-pr.png|frame|center|The Modify Storage Pool High-Availability Group dialog: the Clustered SCSI3-PR Reservations setting sits below the primary and secondary system choices]]

# '''Register each host.''' In Hosts & Portal Groups, choose Add, set the operating system type, and enter the host's iSCSI IQN or FC WWPN. For FC, Select Initiator lets you pick the WWPN from a list instead of typing it. Group the members of a host cluster in a host group so they get identical assignments ([[Hosts and Host Groups]]).

[[File:Guide-fc-iscsi-alua-fc-iscsi-alua-add-host.png|frame|center|The Add Host dialog: one host entry can carry an iSCSI IQN, an FC WWPN or an NVMe-oF NQN as its initiator]]

# '''Assign volumes''' to the host or host group. An assignment maps the volume as a LUN through the target ports of both nodes, active on the owner and standby on the partner.

[[File:Guide-fc-iscsi-alua-fc-iscsi-alua-hosts.png|frame|center|The Hosts view: each registered host lists its initiator identifiers on the left and its storage volume assignments on the right]]

# '''Connect the hosts.'''
** ''Linux:'' set up the iSCSI initiator ([[ISCSI Initiator Setup]]) or rescan the FC HBAs, then configure dm-multipath with ALUA path grouping ([[Multipath IO Configuration]]).
** ''VMware ESXi:'' add the software iSCSI adapter, add the HA virtual interface address under Dynamic Discovery, and rescan storage. The volume appears as an "OSNEXUS iSCSI Disk"; match it to the EUI shown in the volume's properties in QuantaStor, then create the VMFS datastore.
** ''Windows and Hyper-V:'' enable MPIO with the in-box Microsoft DSM, claim the QuantaStor disks, bring them online on one node, and add them to the failover cluster as Cluster Shared Volumes.

== Operating and testing ==

Test the paths before production. On a Linux host, <code><nowiki>multipath -ll</nowiki></code> should show two path groups per LUN: one whose paths go to the pool owner as active, and one whose paths go to the partner as standby (Windows MPIO and the ESXi path list show the same split). Then trigger a manual failover from Storage Pool HA Groups with Manual Failover ([[Storage Pool Ha Failover Group Failover]]) and watch the two groups swap states while I/O continues.

For clustered hosts, test failover under real load. QuantaStor has been validated with two-host Windows Server 2022 Hyper-V failover clusters running Cluster Shared Volumes with SCSI-3 persistent reservations on FC HA pools, through manual and automatic failovers, live migration during a failover, and host drain and reboot.

If a LUN does not appear on a host, check in this order: zoning on the switch, the FC port's Link State, the host's initiator entry in QuantaStor, and the volume assignment. Issuing a LIP from the FC Ports tab forces the fabric to rediscover the ports.

== FAQ ==

=== Which software-defined storage platforms support Fibre Channel and iSCSI with ALUA multipathing? ===

QuantaStor does. It presents volumes over FC through QLogic HBAs in target mode and over iSCSI, and it reports active and standby ALUA path states for every LUN on an HA storage pool. Hosts use their standard multipath drivers (dm-multipath, Windows MPIO with the Microsoft DSM, VMware native multipathing) with no vendor plug-in.

=== What storage can I use for a Hyper-V failover cluster with Cluster Shared Volumes? ===

Any FC or iSCSI LUN that supports SCSI-3 persistent reservations and keeps them through a controller failover. On QuantaStor that is a volume on an HA pool whose HA group has clustered SCSI-3 reservations enabled. Two-host Windows Server 2022 Hyper-V clusters with CSVs have been validated on FC HA pools, including live migration during a failover.

=== What happens to host paths during an HA failover? ===

Nothing is removed. The old owner's paths change from active to standby, the new owner's paths change from standby to active once every LUN is backed by the volume, and the host's multipath driver moves I/O accordingly. Persistent reservations held by clustered hosts stay in place.

=== Do I need to enable target mode on each FC port? ===

No. Any target-capable QLogic port without an explicit initiator-mode setting is put into target mode automatically within about a minute. Use the Enable FC Initiator Mode action only for ports you want to use as initiators.

=== Can I change a volume's block size after creating it? ===

No. The block size is fixed when the volume is created. To change it, create a new volume with the right size and move the data, for example by live-migrating VMs to a new VMware datastore. You can grow the size of the Storage Volume at any time but the underlying block size is fixed at creation, 64K by default.

----
''Part of the [[Guides:Index|QuantaStor Guides]] series. For reference documentation, see the [[Main Page|QuantaStor documentation]].''

[[Category:Guides]]

File:Guide-fc-iscsi-alua-fc-iscsi-alua-hosts.png

2026-10-07T20:00:04Z

Qadmin: osn-seo-utilities: screenshot for Guides (fc-iscsi-alua)

osn-seo-utilities: screenshot for Guides (fc-iscsi-alua)

File:Guide-fc-iscsi-alua-fc-iscsi-alua-add-host.png

2026-10-07T20:00:03Z

Qadmin: osn-seo-utilities: screenshot for Guides (fc-iscsi-alua)

osn-seo-utilities: screenshot for Guides (fc-iscsi-alua)

File:Guide-fc-iscsi-alua-fc-iscsi-alua-ha-group-pr.png

2026-10-07T20:00:03Z

Qadmin: osn-seo-utilities: screenshot for Guides (fc-iscsi-alua)

osn-seo-utilities: screenshot for Guides (fc-iscsi-alua)

Guides:Index

2026-10-07T05:45:21Z

Qadmin: osn-seo-utilities: regenerate index

Network Ports

2026-10-07T05:45:18Z

Qadmin: osn-seo-utilities: docs-arp-flux-protection-for-multi-homed-ha @ d54c6794ab44 (approved in the portal)

[[Category:admin_guide]]

Network ports (also called target ports or network interfaces) are the Ethernet interfaces through which a QuantaStor system is managed and through which it serves storage. This page covers how to address a port, how to combine ports into a bond, how to layer VLAN and virtual interfaces on top of a port, how to control which protocols each port carries, how QuantaStor keeps ARP answers on the right port, how to add static routes, and which TCP and UDP ports and external sites QuantaStor needs so you can lock the network down without breaking it.

Client hosts reach [[Storage Volumes]] over iSCSI and NVMe-oF/TCP through a network port, authorized users reach [[Network Shares]] over NFS and SMB through a network port, and object storage buckets are reached over S3 through a network port. Every one of those paths depends on the port configuration described here, so it is worth getting right before you provision storage.

{| class="wikitable"
! Section !! Purpose
|-
| [[#Port types and naming|Port types and naming]] || How to tell a physical port, a bond, a VLAN, a VIF and a cluster VIF apart by name.
|-
| [[#Where network ports are managed|Where network ports are managed]] || The tabs, toolbar and right-click menu that hold every port operation.
|-
| [[#Static vs DHCP assigned IP addresses|Static vs DHCP assigned IP addresses]] || Why static addressing is required in practice, and the one-time conversion on a fresh install.
|-
| [[#Modifying network port settings|Modifying network port settings]] || Every field on the General Settings tab of Modify Network Port.
|-
| [[#Firewall settings per port|Firewall settings per port]] || Allowing and blocking individual protocols on one port.
|-
| [[#NIC bonding and trunking|NIC bonding and trunking]] || Combining ports for throughput and fault tolerance, and which bonding mode to pick.
|-
| [[#VLAN interfaces|VLAN interfaces]] || Tagged virtual ports on a physical or bonded port.
|-
| [[#Virtual interfaces|Virtual interfaces]] || Extra IP addresses on one port.
|-
| [[#High-availability and cluster virtual interfaces|High-availability and cluster virtual interfaces]] || Floating IPs that move with a pool, a site or the grid.
|-
| [[#ARP Flux Protection|ARP Flux Protection]] || How QuantaStor stops a multi-homed system answering ARP on the wrong port, and the ARP Filtering policy.
|-
| [[#Renaming a network port|Renaming a network port]] || Giving a port a stable, meaningful name.
|-
| [[#Online, offline, restart and rescan|Online, offline, restart and rescan]] || Bringing a port up or down and picking up out-of-band changes.
|-
| [[#Static route management|Static route management]] || Routing specific networks through a gateway other than the default.
|-
| [[#Choosing which ports carry which traffic|Choosing which ports carry which traffic]] || Separating management traffic from storage traffic.
|-
| [[#TCP and UDP ports QuantaStor uses|TCP and UDP ports QuantaStor uses]] || Every port the appliance listens on, why, and which ones the Firewall tab can block.
|-
| [[#External sites QuantaStor connects to|External sites QuantaStor connects to]] || Outbound connections for licensing, upgrades, support logs, alerting and replication.
|-
| [[#Locking down the network|Locking down the network]] || What you can safely block, and what has to stay open.
|-
| [[#Verifying connectivity|Verifying connectivity]] || The Network Check report.
|-
| [[#Command line reference|Command line reference]] || The <code>qs</code> commands that cover everything on this page.
|}

== Port types and naming ==

QuantaStor names every port after the interface the kernel presents, and uses two delimiters to indicate that a port is layered on top of another one. Reading the name tells you what a port is:

{| class="wikitable"
! Name !! Type !! Notes
|-
| <code>ens192</code>, <code>eno1</code>, <code>enp3s0f0</code> || Physical port || The predictable name assigned by the kernel. Rename it if you want something more meaningful -- see [[#Renaming a network port|Renaming a network port]].
|-
| <code>bond0</code> || Bonded port || Two or more physical ports combined. The member ports become slaves and stop carrying their own IP address.
|-
| <code>ens192.56</code> || VLAN interface || A period separates the parent port name from the VLAN tag. A VLAN on a bond looks like <code>bond0.56</code>.
|-
| <code>ens192:1</code> || Virtual interface (VIF) || A colon followed by an index, allocated from 1 upwards. An extra IP address on the parent port.
|-
| <code>ens192:ha345680</code> || HA pool virtual interface || Floats with a [[Storage Pools|storage pool]] when the pool fails over.
|-
| <code>ens192:sv...</code> || Site cluster virtual interface || Floats between the appliances of a site cluster.
|-
| <code>ens192:gm</code> || Grid management virtual interface || The single floating IP a storage grid is managed through. iSCSI and NVMe-oF are permanently disabled on it, because a management IP that moves on failover is not a safe place to export block storage from.
|-
| <code>ens192 (S)</code> || Manually configured secondary port || An address that QuantaStor discovered but does not own. Management operations are refused on these.
|}

Interface names are limited to 15 characters by the kernel, and the tag QuantaStor appends when it creates a cluster virtual interface has to fit inside that budget. A long parent port name is therefore not merely untidy -- it will eventually block cluster VIF creation, so keep renamed ports short.

== Where network ports are managed ==

[[File:nwport_ports_tab.png|thumb|right|800px|The Network Ports tab lists every port on every system in the grid, grouped by storage system. The Network Port toolbar group above it holds the create, delete and state operations.]]

{{Navigation|Storage Management → Storage Systems → ''select a Storage System'' → Network Ports ''(tab)''}}

Select the '''Storage Systems''' section in the tree on the left, then the '''Network Ports''' tab in the centre of the screen. The tab lists every port on every system in the grid, grouped by storage system, with its state, link state, link speed, IP address, subnet mask, whether it is an iSCSI or NVMe-oF portal, and its vendor, model, MAC address, byte counters and MTU. The '''Network Static Routes''' tab beside it lists the static routes on those same ports.

The '''Network Port''' toolbar group at the top of the screen carries Modify, Create Virtual Port, Create VLAN Port, Create Bonded Port, the matching Delete operations, Online, Offline and Rescan All.

[[File:nwport_context_menu.png|thumb|right|237px|Right-clicking a port in the Network Ports tab reaches every port operation, including the three that are not on the toolbar: Rename, Restart, and Create/Delete Static Route.]]

Three operations are reachable only by right-clicking a port, not from the toolbar: '''Rename Network Port...''', '''Restart Network Port...''' and '''Create Static Route...''' / '''Delete Static Route...'''. Right-click a port either in the tree on the left or in the Network Ports tab.

For grid-wide settings such as the DNS servers and search domain that apply to all ports, use '''Modify Grid Network Settings''' instead -- the '''Modify Grid''' button in the '''Storage System Grid''' toolbar group. It writes one DNS search suffix, DNS server list and NTP server list to every system you tick, overwriting whatever those systems had. The per-system equivalents are the [[Storage System#DNS Servers|DNS Servers]] and [[Storage System#NTP Servers|NTP Servers]] tabs of Storage System Modify.

== Static vs DHCP assigned IP addresses ==

QuantaStor supports both statically assigned IP addresses and dynamically assigned '''D'''ynamic '''H'''ost '''C'''onfiguration '''P'''rotocol ('''DHCP''') addresses. After installation the first port is typically configured by DHCP and the rest are disabled, so that the appliance is reachable for initial setup.

Reconfigure every port with a static IP address before you provision storage. A DHCP lease can change, and a storage system whose address moves takes its iSCSI targets, NFS exports and SMB shares with it. The appliance also has to be able to boot and serve storage when the DHCP server is unreachable. The only defensible use of DHCP is a reservation that always hands the same address to the same MAC, and even then a static address is simpler to reason about.

There is a second, one-time reason to set a static management address first. A freshly installed system still has the installer's netplan configuration active. The first time you modify any network port, QuantaStor converts that configuration into {{Code|1=/etc/network/interfaces}}, backs up the previous file as {{Code|1=/etc/network/interfaces-netplan-convert-<timestamp>.backup}}, renames each {{Code|1=/etc/netplan/*.yaml}} file to <code>.bak</code>, and disables the netplan renderer. All interfaces are torn down and brought back during that conversion, so a port still on DHCP can come back with a different address and the management session drops. QuantaStor guards against this: while any port is in that state it is flagged with a warning whose detail begins "Netplan", and the Modify Network Port dialog refuses any change other than to <code>static</code>, with the message ''"Be sure to configure a static management network port before modifying network ports."''

Set a static address on your management port first, then work through the remaining DHCP ports until they all have static IPs assigned. '''Reboot afterwards and confirm the configuration, which is what the accompanying alert asks for.''' This is an important step after an initial QuantaStor OS installation as this allows the system to cleanup network configuration issues left behind by the text mode installer.

== Modifying network port settings ==

[[File:nwport_modify_general.png|thumb|right|550px|The General Settings tab of Modify Network Port. The Static IP Configuration Settings fieldset is only enabled when Config Type is set to static.]]

{{Navigation|Storage Management → Storage Systems → Network Port ''(select + right-click)'' → Modify Network Port...}}

The dialog is also on the '''Modify''' button in the Network Port toolbar group. It has two tabs, '''General Settings''' and '''Firewall'''.

=== General Settings ===

{| class="wikitable"
! Field !! Description
|-
| '''Storage System''' || The system whose port you are configuring. Changing it reloads the Network Port list.
|-
| '''Network Port''' || The port to modify. Every port on the selected system is offered, including bonds, VLANs and virtual interfaces.
|-
| '''Config Type''' || <code>static</code>, <code>dhcp</code> or <code>disabled</code>. Only <code>static</code> enables the Static IP Configuration Settings fieldset below. Choosing <code>disabled</code> prompts for confirmation and then removes the port's stanza from {{Code|1=/etc/network/interfaces}} entirely, so the port stops carrying traffic and does not come back after a reboot. A port that is a bond slave reports as <code>disabled</code> and cannot be changed here -- delete the bond first. Cluster virtual interfaces are locked to <code>static</code> because the cluster software owns their addresses.
|-
| '''Description''' || An optional note recorded against the port. Use it to record what the port is for; there is no other place to do that.
|-
| '''iSCSI Portal''' || Makes this port's IP address an allowed iSCSI portal. Clearing it removes the address from the allowed-portal list of every iSCSI target on the system, so initiators can no longer log in through it. Existing sessions are not torn down -- the change takes effect on the next login. If no port at all is left iSCSI-enabled, QuantaStor restricts iSCSI to loopback rather than leaving it open.
|-
| '''NVMeoF Portal''' || Makes this port's IP address an NVMe-oF/TCP listener on port 4420. Unlike the iSCSI setting, clearing it removes the listener, which does drop connections through that address.
|-
| '''Lossless Networking''' || Configures RoCEv2 with priority flow control on the port. Only selectable on adapters that support it -- in practice Mellanox/NVIDIA ConnectX cards with RDMA enabled -- and greyed out everywhere else. It sets DSCP-based priority trust, enables PFC on priority 3 only, turns global pause off, sizes the receive buffers for the adapter model, and records the port in {{Code|1=/var/opt/osnexus/quantastor/conf/qs_losslessNetworking.conf}}. QuantaStor verifies the result and reverts the change if verification fails. Enable it only where the switch fabric is configured for the same priority flow control scheme; a one-sided configuration performs worse than plain Ethernet.
|-
| '''Optimize hardware RX/TX buffer settings for throughput''' || Enlarges the adapter's receive and transmit ring buffers. QuantaStor records the current sizes so it can put them back, then raises both rings to the largest power-of-two multiple of 128 that still fits under the adapter's hardware maximum -- roughly half the maximum, which improves large-block and replication throughput without asking the driver for a size it will reject. Settings are recorded in {{Code|1=/var/opt/osnexus/quantastor/conf/qs_porttunings.conf}} and re-applied every five minutes. Available on physical ports only: a bond, VLAN or virtual interface has no hardware ring of its own, so enable it on the member ports instead.
|}

The '''Static IP Configuration Settings''' fieldset holds the addressing:

{| class="wikitable"
! Field !! Description
|-
| '''IP Address''' || The static address for the port. Each port needs a unique address; QuantaStor rejects an address already recorded on another port. If the address is part of a Ceph cluster configuration you have to disable the port before the address can be changed.
|-
| '''Subnet Mask''' || The mask for the port's network, such as <code>255.255.0.0</code>.
|-
| '''Gateway''' || Optional. QuantaStor checks that the gateway is on the same subnet as the port address and refuses it otherwise. Configure a gateway on one port only. QuantaStor gives every static port the same route metric, so a second gateway-bearing port ends up competing for the default route rather than adding a fallback.
|-
| '''MTU''' and '''Jumbo Frames''' || The maximum transmission unit, 1492 to 64000. The '''Jumbo Frames''' button toggles between 9000 and 1500 and relabels itself '''Default Frames''' once jumbo is set. Set 9000 on all 10GbE and faster interfaces. If you move away from 1500, the switch ports and the client hosts on that network have to match -- a mismatched MTU produces intermittent stalls rather than a clean failure. The field is disabled for VLAN and virtual interfaces, which inherit the parent port's MTU; changing the parent cascades the new MTU to all of its children. Setting a VLAN MTU above its parent's is capped to the parent's value with an alert.
|}

Changing the IP address of a port that has static routes configured is flagged: the routes are bound to the old address and client connections through them can drop. Remove the routes first.

Click '''Apply''' to commit the change and keep the dialog open, or '''OK''' to commit and close.

== Firewall settings per port ==

[[File:nwport_modify_firewall.png|thumb|right|550px|The Firewall tab sets each protocol to Allow, Block or Inherit for this port only.]]

{{Navigation|Storage Management → Storage Systems → Network Port ''(select + right-click)'' → Modify Network Port... → Firewall ''(tab)''}}

The '''Firewall''' tab lists each service QuantaStor knows how to filter, with its description and a '''Status''' setting. The three settings are what makes this useful, and the distinction is easy to miss:

* '''Inherit''' -- follow the system-wide firewall setting. This is the default for every service on every port.
* '''Block''' -- drop traffic for this service arriving at this port's IP address, even though the system-wide setting allows it.
* '''Allow''' -- accept traffic for this service at this port's IP address, even though the system-wide setting blocks it.

So the system level sets the policy and the port level carries the exceptions. The usual pattern is to block everything you do not use at the system level -- see [[Firewall Management]] -- and then Allow the handful of protocols each port is actually meant to serve.

Two things are worth knowing before you rely on it. The rules match on the port's '''destination IP address''', not on the interface, so a per-port rule really follows the address; that is deliberate, and it is why the masks are also recorded against HA and site virtual interfaces so the rule follows a floating address to whichever system holds it. And because QuantaStor reports a service as blocked whenever any DROP rule exists for it, blocking a protocol on one port raises an alert saying that service is blocked system-wide, even though only that one address is affected.

The service list, the TCP and UDP ports behind each entry, and the bit each one occupies come from {{Code|1=/opt/osnexus/quantastor/conf/qs_services_firewall.conf}} on the appliance. The rules themselves are iptables rules in QuantaStor's own chains (<code>QUANTASTOR-INPUT</code>, <code>QUANTASTOR-FORWARD</code>, <code>QUANTASTOR-OUTPUT</code>), written out to {{Code|1=/var/opt/osnexus/quantastor/iptables.qsfirewallrules}} so they survive a restart, and kept out of the shared chains so the firewall can be rebuilt without disturbing other rules on the system. QuantaStor re-applies them on change and again every 20 minutes.

Changing a setting that makes a service unreachable prompts for confirmation first. For the TCP and UDP ports behind each entry, and the ports the Firewall tab does not cover, see [[#TCP and UDP ports QuantaStor uses|TCP and UDP ports QuantaStor uses]] below.

The same firewall settings can be driven from the CLI with <code>[[QuantaStor CLI Command Reference#network-port-modify|qs network-port-modify]] --protocol-block=</code> and <code>--protocol-allow=</code>, each taking a comma-separated protocol list, with <code>--protocol-block-reset</code> and <code>--protocol-allow-reset</code> to clear a list. Prefix a protocol with a tilde (<code>~</code>) to remove it from a list rather than add it.

== NIC bonding and trunking ==

[[File:nwport_bond_create.png|thumb|right|550px|Create Bonded Port. At least two ports must be selected in the grid; clear Hide Configured Ports to bond a port that already has an IP address.]]

{{Navigation|Storage Management → Storage Systems → Network Port → Create Bonded Port ''(toolbar)''}}

Bonding, also called trunking or teaming, combines several physical ports into one logical port to improve throughput and to survive the loss of a cable, a switch port or a NIC. The bond takes the IP address; the member ports become slaves and no longer carry addresses of their own.

For every mode except LACP, all of the member ports must be connected to the same switch. LACP works across switches -- which is what makes it the only mode that survives a whole switch going away -- but it requires the switch ports to be configured for LACP as well. QuantaStor surfaces switch-side mistakes on the port itself, reporting an invalid partner MAC address or a partner that is not advertising LACP support, so check the port state after creating an LACP bond rather than assuming it came up.

The dialog fields are the same addressing fields as Modify Network Port, plus:

{| class="wikitable"
! Field !! Description
|-
| '''Bond Mode''' || The bonding policy. See the table below.
|-
| '''Select the Network Ports to be Bonded''' || Select at least two ports. The grid lists each candidate port's name, IP address, vendor and model.
|-
| '''Hide Configured Ports''' || Selected by default, hiding ports that already have an IP address so you do not accidentally bond your management port. Clear it to bond a configured port; QuantaStor warns that the port's existing IP configuration will be removed and that the bond will only have the address you entered.
|-
| '''Force''' || Proceeds without the confirmation prompt when a selected port already has an IP configuration.
|}

The modes QuantaStor offers, with the switch requirement each one carries:

{| class="wikitable"
! Mode !! Kernel mode !! Switch !! Behaviour
|-
| Link Aggr Ctrl Protocol (LACP layer2) || <code>802.3ad</code>, hash <code>layer2</code> || Managed, LACP configured || Balances by source and destination MAC. Spans switches. Recommended default.
|-
| Link Aggr Ctrl Protocol (LACP layer2+3) || <code>802.3ad</code>, hash <code>layer2+3</code> || Managed, LACP configured || Adds the IP addresses to the hash, which spreads traffic better when many clients sit behind one router MAC.
|-
| Link Aggr Ctrl Protocol (LACP layer3+4) || <code>802.3ad</code>, hash <code>layer3+4</code> || Managed, LACP configured || Adds the TCP/UDP ports to the hash, so a single client with many connections can use more than one member link.
|-
| Round Robin (balance-rr) || <code>balance-rr</code> || Etherchannel, managed || Sends packets in sequence across the members. Highest single-stream throughput, but it reorders packets and does not span switches.
|-
| Adaptive Transmit Load Balancing (balance-tlb) || <code>balance-tlb</code> || Unmanaged || Balances outbound traffic only; inbound arrives on one member. Needs no switch configuration.
|-
| Adaptive Load Balancing (balance-alb) || <code>balance-alb</code> || Unmanaged || As balance-tlb, plus inbound balancing via ARP negotiation. Needs no switch configuration.
|-
| Balance XOR (balance-xor) || <code>balance-xor</code> || Etherchannel, managed || Picks a member by hashing the MAC addresses. Static, so both ends must agree.
|-
| Active-Backup (active-backup) || <code>active-backup</code> || Unmanaged || One member carries all traffic; the others stand by. No throughput gain, and the simplest mode to get right.
|}

For fault tolerance that survives a switch outage, LACP is the mode to use. Where the switches are unmanaged and cannot be configured, use Active-Backup or one of the adaptive load balancing modes.

The '''Bond Mode''' combo in the Create Bonded Port dialog is pre-selected from the storage system's own default bonding policy, which QuantaStor sets on first startup and records in {{Code|1=/etc/modprobe.d/bonding.conf}}. On the 6.9.0 system this page was verified against that default was LACP layer 2. Check yours with <code>[[QuantaStor CLI Command Reference#system-get|qs system-get]]</code> and look at the '''Bond Mode''' field, and change it with <code>[[QuantaStor CLI Command Reference#system-modify|qs system-modify]] --bond-mode=<mode></code>. QuantaStor sets <code>miimon=100</code> on every bond so link failures are detected within a tenth of a second, and allows a maximum of four bonds per system.

To change the mode of a bond that already exists, open '''Modify Network Port''' on the bond and use the '''Bond Port Advanced Settings...''' button, which appears only when the selected port is a bond. The mode cannot be changed while the bond is carrying a virtual interface or an HA virtual interface -- remove those first. If a member port drops out of a bond, QuantaStor re-enslaves it automatically, at most once an hour per member.

Delete a bond with '''Delete Bonded Port''' in the toolbar, or <code>[[QuantaStor CLI Command Reference#bonded-interface-delete|qs bond-delete]] --port=<bond></code>. The member ports return to being unconfigured physical ports.

== VLAN interfaces ==

[[File:nwport_vlan_create.png|thumb|right|396px|Create VLAN Interface. The MTU is read-only because a VLAN inherits it from the parent port.]]

{{Navigation|Storage Management → Storage Systems → Network Port → Create VLAN Port ''(toolbar)''}}

A '''VLAN''' ('''V'''irtual '''LAN''') interface is a virtual port that tags its traffic with a VLAN ID, letting one physical or bonded port serve several isolated networks. VLAN interfaces can be created and deleted at any time.

{| class="wikitable"
! Field !! Description
|-
| '''Storage System''' || The system to create the VLAN interface on.
|-
| '''Parent Port''' || The physical or bonded port to tag on. A VLAN cannot be created on another VLAN, nor on a virtual interface, so those are filtered out of the list.
|-
| '''IP Address''', '''Subnet Mask''', '''Gateway''' || The addressing for the VLAN interface, on the network that VLAN reaches.
|-
| '''Description''' || An optional note. Recording which network the tag corresponds to pays for itself.
|-
| '''MTU''' || Read-only. A VLAN inherits the parent port's MTU; change it on the parent.
|-
| '''VLAN ID''' || The 802.1Q tag, 1 to 4094 (4095 is reserved). Defaults to 5.
|-
| '''VLAN QoS''' || The 802.1p Class of Service user priority to record for the interface, 0 (lowest, the default) to 7 (highest).
|}

The new port is named after its parent with a period and the tag, so VLAN 56 on <code>ens192</code> becomes <code>ens192.56</code> and VLAN 56 on <code>bond0</code> becomes <code>bond0.56</code>. Delete a VLAN interface with '''Delete VLAN Port''' in the toolbar or <code>[[QuantaStor CLI Command Reference#vlan-interface-delete|qs vlan-delete]] --port=<port></code>. A port that has a VLAN child cannot be disabled or renamed until the VLAN is removed.

== Virtual interfaces ==

[[File:nwport_vif_create.png|thumb|right|500px|Create Virtual Interface. A VIF adds a second address to an existing port.]]

{{Navigation|Storage Management → Storage Systems → Network Port → Create Virtual Port ''(toolbar)''}}

A virtual interface (VIF) gives a physical or bonded port an additional IP address, so one port can be present on several networks or carry several addresses on the same network. VIFs can be created and deleted at any time, and cannot be layered on top of another VIF.

The fields are '''Storage System''', '''Parent Port''', '''IP Address''', '''Description''', '''Subnet Mask''', '''Gateway''', and a read-only '''MTU''' inherited from the parent port. QuantaStor names the new interface after the parent with a colon and the next free index, so the first VIF on <code>ens192</code> is <code>ens192:1</code>.

Two constraints are not obvious from the dialog. A VIF cannot be created on a bond running in Active-Backup mode. And DHCP is not valid on a virtual interface -- a VIF is always statically addressed.

Delete a VIF with '''Delete Virtual Port''' in the toolbar or <code>[[QuantaStor CLI Command Reference#virtual-interface-delete|qs vif-delete]] --port=<port></code>.

== High-availability and cluster virtual interfaces ==

QuantaStor also has virtual interfaces that float between systems. You will see these with <code>:ha</code>, <code>:sv</code> or <code>:gm</code> in the name, such as <code>ens192:ha345680</code>.

* '''HA pool virtual interfaces''' (<code>:ha</code>) are associated with a specific [[Storage Pools|storage pool]] and move with the pool when it fails over to another system. A pool can have several; each additional one increments the last digit of the port name, so they run <code>:ha345680</code>, <code>:ha345681</code>, <code>:ha345682</code> and so on. Create them by right-clicking an HA failover group in the Storage Pools section and choosing '''Add Cluster VIF...''', or with <code>[[QuantaStor CLI Command Reference#ha-interface-create|qs hai-create]] --ha-group=<group> --parent-port=<port> --ip-address=<ip></code>. A cluster heartbeat must be configured between the systems that manage the pool before an HA VIF can be created; see [[Setup Guide for Clustered HA Storage Pools]] and [[Storage Pool HA Failover Interface Create]].
* '''Site cluster virtual interfaces''' (<code>:sv</code>) float between the appliances of a site so that a service address stays available. See [[High-availability VIF Management]].
* The '''grid management virtual interface''' (<code>:gm</code>) is the single floating address a storage grid is managed through. There is exactly one per site cluster, and iSCSI and NVMe-oF are permanently disabled on it.

These are all managed from the '''[[High-availability VIF Management]]''' main tab, and by the cluster software rather than by the network port configuration -- that page covers use cases, failover, deliberate moves and location constraints. That is why the Modify Network Port dialog locks their Config Type to <code>static</code>, disables the addressing fields, and refuses IP, MTU and bond mode changes -- and why static routes cannot be created on them. Change a floating address through the HA or site VIF dialogs, not through Modify Network Port.

A floating address only works if the switches believe the system that currently holds it, which is what ARP flux protection, below, is for.

== ARP Flux Protection ==

A system with more than one network port -- every HA node and every Ceph node, and any system with a separate management and storage network -- is '''multi-homed'''. With the Linux kernel's default ARP settings, any port on a multi-homed system will answer an ARP request for any address configured anywhere on the system, and the system may use any of its own addresses as the source of the ARP requests it sends. The switches then learn the wrong MAC address and switch port for an address, and traffic meant for the storage network arrives on the management port, or the other way round. This is known as '''ARP flux'''.

ARP flux matters most for the floating addresses described above. When an HA or cluster virtual interface moves, the system taking it over sends a gratuitous ARP so that the switches and clients update their tables. If the switches have already learned the wrong port for that address, or if another port answers for it, clients keep sending to the old location and the failover looks like an outage. On a bond carrying HA virtual interfaces the effect is worse, because the address also has to be learned through the right member link.

QuantaStor guards against this with two separate mechanisms. One is always on; the other is the '''ARP Filtering''' policy in Storage System Modify.

=== Always-on ARP flux protection ===

When the QuantaStor service starts, it sets four kernel parameters:

{| class="wikitable"
! Parameter !! Value !! Effect
|-
| <code>net.ipv4.conf.all.arp_ignore</code> || <code>1</code> || Answer an ARP request only when the requested address is configured on the port the request arrived on.
|-
| <code>net.ipv4.conf.default.arp_ignore</code> || <code>1</code> || The same, for ports created after the service started.
|-
| <code>net.ipv4.conf.all.arp_announce</code> || <code>2</code> || Always use the best local address for the destination as the source of an ARP request, never an address that belongs to another port.
|-
| <code>net.ipv4.conf.default.arp_announce</code> || <code>2</code> || The same, for ports created after the service started.
|}

The kernel applies the larger of the <code>all</code> value and a port's own value, so the <code>all</code> settings cover every existing port and the <code>default</code> settings cover bonds, VLANs and virtual interfaces created later. QuantaStor applies the values immediately and also writes them to {{Code|1=/etc/sysctl.d/70-quantastor-arp.conf}}, so they survive a reboot and an upgrade takes effect without one. The file is managed by QuantaStor; do not edit it, because QuantaStor rewrites it on the next service start.

There is no setting for this in the web interface. It needs none: with these values a port still answers every ARP request for its own addresses, so single-homed systems behave exactly as before.

To check the live values on an appliance:

<pre style="font-size: smaller">
sysctl net.ipv4.conf.all.arp_ignore net.ipv4.conf.all.arp_announce
cat /etc/sysctl.d/70-quantastor-arp.conf
</pre>

==== Turning it off ====

Turn ARP flux protection off only if your network design depends on the kernel's default ARP behaviour, for example a host-based load balancing scheme that expects any port to answer for any address. Create the touch file and restart the QuantaStor service:

<pre style="font-size: smaller">
touch /var/opt/osnexus/quantastor/touchfiles/tf_arp_flux_protection.disable
systemctl restart quantastor
</pre>

On the next start QuantaStor deletes {{Code|1=/etc/sysctl.d/70-quantastor-arp.conf}} and sets all four parameters back to the kernel default of <code>0</code>. It reverts them only if that file exists, so values you set by hand with no QuantaStor file present are left alone. To turn the protection back on, delete the touch file and restart the service again. The setting is per system: on an HA pair or a cluster, do the same on every member, or the members will answer ARP differently. See [[QuantaStor Touch Files]] for touch files in general.

Setting <code>disable_arp_logic=true</code> in the <code>[system]</code> section of {{Code|1=/etc/quantastor.conf}} has the same effect, and also stops QuantaStor from managing the ARP Filtering policy below.

=== ARP Filtering policy ===

[[File:Docs-arp-flux-protection-for-multi-homed-ha-arp-filtering.png|thumb|right|590px|The ARP Filtering fieldset on the Network Settings tab of Storage System Modify. Status shows whether filtering is currently in effect, here Disabled under the Auto policy on a system with no bond or virtual port.]]

{{Navigation|Storage Management → Storage Systems → ''select a Storage System'' → Modify ''(toolbar)'' → Network Settings ''(tab)''}}

The '''ARP Filtering''' fieldset on the '''Network Settings''' tab of [[Storage System Modify]] controls a second kernel parameter, <code>arp_filter</code>. With <code>arp_filter</code> on, the kernel answers an ARP request on a port only if it would route traffic for the requester out of that same port. It adds to the always-on protection when two ports sit on the same subnet, which is where <code>arp_ignore</code> alone cannot decide which port should answer.

{| class="wikitable"
! Field !! Description
|-
| '''ARP Policy''' || '''Auto''' (the default), '''Enabled''' or '''Disabled'''. '''Auto''' turns filtering on when the system has a bonded port or any other virtual port, and off otherwise. '''Enabled''' and '''Disabled''' force it on or off regardless of the ports.
|-
| '''Status''' || Read-only. Whether ARP filtering is in effect right now, read from the running kernel: '''Enabled''' or '''Disabled'''.
|}

QuantaStor writes the setting to {{Code|1=/etc/sysctl.conf}} as <code>net.ipv4.conf.all.arp_filter</code> so it survives a reboot, applies it to the running kernel, and, if the value changed, flushes the system's ARP table so neighbours are relearned through the right ports.

Under '''Auto''', QuantaStor re-evaluates the setting when the service starts, when you change the policy, and when a bond is deleted. Creating a bond turns filtering on immediately. Creating a VLAN or virtual interface does not, by itself, re-evaluate the policy; the change is picked up the next time the service starts. If you are building a multi-homed configuration and want filtering on from the first port, set the policy to '''Enabled'''.

Leave the policy on '''Auto''' unless you have a specific reason not to. Use '''Enabled''' on a system that has two or more physical ports on the same network without a bond. Use '''Disabled''' only if filtering interferes with a routing design that deliberately sends replies out of a different port from the one the request arrived on.

== Renaming a network port ==

[[File:nwport_rename.png|thumb|right|550px|Rename Network Port. The Port Information fieldset identifies the port you are about to rename.]]

{{Navigation|Storage Management → Storage Systems → Network Port ''(select + right-click)'' → Rename Network Port...}}

Predictable kernel names such as <code>enp3s0f1</code> say where the adapter is but nothing about what it does. Renaming a port gives it a name that survives hardware changes and reads clearly in the port list.

Enter the new name in '''New Name'''; the '''Port Information''' fieldset below shows the selected port's UUID, IP address, MAC address and state so you can confirm you have the right one. QuantaStor writes a systemd link file at {{Code|1=/etc/systemd/network/10-<newname>.link}} that matches the port by MAC address, renames the running interface, and rewrites the port's stanza in {{Code|1=/etc/network/interfaces}}.

The dialog's own guidance is worth following: prefix your management port with <code>mgmt</code>, and avoid the <code>eth</code> prefix, which collides with the names kernel drivers hand out. Interfaces are conventionally numbered from zero, as in <code>eno0</code>, <code>eno1</code>. Keep the name under 15 characters -- see [[#Port types and naming|Port types and naming]]. The name is lowercased for you.

A port cannot be renamed while it is a bond slave, while it has VLAN or virtual interface children, or while the system is using legacy <code>ethN</code> naming mode. Remove the bond or the children first, or switch back to predictable naming in [[Storage System Modify|Modify Storage System]]. A reboot may be needed for the rename to take effect everywhere, and QuantaStor raises an alert saying so.

The CLI equivalent is <code>[[QuantaStor CLI Command Reference#network-port-rename|qs np-rename]] --port=<port> --name=<newname></code>.

== Online, offline, restart and rescan ==

Four state operations sit alongside the configuration dialogs:

{| class="wikitable"
! Operation !! Where !! What it does
|-
| '''Online''' || Network Port toolbar; right-click → Online Network Port... || Brings the port up. The CLI equivalent is <code>[[QuantaStor CLI Command Reference#network-port-enable|qs np-enable]] --port=<port></code>.
|-
| '''Offline''' || Network Port toolbar; right-click → Offline Network Port... || Takes the port down without discarding its configuration, so Online brings it back as it was. Static routes on the port go down with it and come back when it is brought back up. The CLI equivalent is <code>[[QuantaStor CLI Command Reference#network-port-disable|qs np-disable]] --port=<port></code>.
|-
| '''Restart Network Port...''' || Right-click only || Cycles the port so a changed setting takes effect. The CLI equivalent is <code>[[QuantaStor CLI Command Reference#network-port-restart|qs np-restart]] --port=<port></code>.
|-
| '''Rescan All''' || Network Port toolbar; right-click → Rescan All Network Ports... || Discovers newly added ports and picks up configuration made outside QuantaStor. The CLI equivalent is <code>[[QuantaStor CLI Command Reference#network-port-rescan|qs np-rescan]]</code>.
|}

Offlining a port that has virtual interfaces attached takes the virtual interfaces down with it. Setting Config Type to <code>disabled</code> in Modify Network Port is not the same thing as taking the port offline: offline is temporary, whereas <code>disabled</code> removes the configuration and does not survive a reboot.

Ports QuantaStor discovers but that you do not want cluttering the list can be hidden with <code>[[QuantaStor CLI Command Reference#network-port-hide|qs np-hide]] --port=<port> --hide=true</code>.

QuantaStor also repairs drift on its own. Discovery re-applies the recorded MTU, subnet mask and gateway when the live configuration diverges from the configuration file, at most once every six hours per correction, and marks the port with a warning telling you to use the modify operation rather than editing the system configuration directly.

== Static route management ==

A static route customizes traffic flow so that a specific network is reached through a gateway other than the default. This is how you reach a network that the default gateway cannot route to, whether that is across a VLAN, through a VPN tunnel, or over a dedicated replication link. Each route has two parts: the gateway on the local network through which the remote network is reachable, and the address and mask of the remote network itself.

QuantaStor's static routes are per port, which gives granular control over which interface carries traffic for which network.

=== Creating a static route ===

[[File:nwport_static_route_create.png|thumb|right|500px|Create Static Route. The gateway must be on the port's own network; the destination must be given in bind address form.]]

{{Navigation|Storage Management → Storage Systems → Network Port ''(select + right-click)'' → Create Static Route...}}

The dialog is also reachable by right-clicking inside the '''Network Static Routes''' tab.

{| class="wikitable"
! Field !! Description
|-
| '''Storage System''' || The system to add the route to.
|-
| '''Parent Port''' || The port the route is bound to. Floating cluster interfaces are filtered out -- routes cannot be created on them.
|-
| '''Gateway IP Address''' || The gateway on the port's own network through which the destination network is reached.
|-
| '''Destination IP Address''' || The network to route to, in bind address form. QuantaStor checks the address against the mask and rejects anything that is not a network address, telling you the correct form to use.
|-
| '''Destination Subnet Mask''' || The mask of the destination network.
|}

For example, on a system whose <code>ens192</code> port is <code>10.0.8.70/24</code> and which has a gateway at <code>10.0.8.1</code> that can reach the <code>192.168.0.0/16</code> network, enter a '''Gateway IP Address''' of <code>10.0.8.1</code>, a '''Destination IP Address''' of <code>192.168.0.0</code> and a '''Destination Subnet Mask''' of <code>255.255.0.0</code>. All traffic from that port destined for <code>192.168.0.0/16</code> is then sent via <code>10.0.8.1</code>. Note that the gateway has to be on the port's own network -- QuantaStor applies the route immediately and refuses a gateway the port cannot reach.

Routes are activated the moment they are created, and QuantaStor refuses a gateway it cannot reach rather than recording a route that will never work. A duplicate destination network and mask on the same system is also refused.

[[File:nwport_static_routes_tab.png|thumb|right|800px|The Network Static Routes tab lists each route with its parent port, gateway, and destination network and mask.]]

Route management is integrated with the platform rather than bolted on. Each route is written to an executable script at {{Code|1=/etc/network/<port>.staticroutes}}, and QuantaStor installs hooks in {{Code|1=/etc/network/if-up.d}} and {{Code|1=/etc/network/if-down.d}} that run it. The routes therefore enable and disable correctly when a port is brought up or down with <code>ifup</code> and <code>ifdown</code> from an SSH session, not just from the web interface. QuantaStor treats those scripts as the source of truth and imports routes it finds in them, so a route added by hand is picked up rather than overwritten. We recommend using the <code>qs</code> CLI or the web management interface all the same, so the routes are recorded as objects you can see in the Network Static Routes tab.

The CLI equivalent is <code>[[QuantaStor CLI Command Reference#static-route-create|qs sr-create]] --port=<port> --destination-ip-address=<network> --destination-netmask=<mask> --gateway-ip-address=<gateway></code>:

<pre style="font-size: smaller">
qs sr-create --port=ens192 --destination-ip-address=192.168.0.0 \
--destination-netmask=255.255.0.0 --gateway-ip-address=10.0.8.1
qs sr-list
</pre>

=== Deleting a static route ===

{{Navigation|Storage Management → Storage Systems → Network Static Routes ''(tab)'' → ''select a route'' → Delete Static Route... ''(select + right-click)''}}

Select the '''Network Static Routes''' tab in the centre of the screen, right-click the route, and choose '''Delete Static Route...'''. Alternatively, expand the port in the '''Storage Systems''' section on the left, then right-click the route.

Routes can be created and deleted at any time. Deleting one withdraws it immediately. If a port is disabled or taken offline, the routes associated with it are disabled with it.

The CLI equivalent is <code>[[QuantaStor CLI Command Reference#static-route-delete|qs sr-delete]] --route-id=<uuid></code>, where the UUID comes from <code>[[QuantaStor CLI Command Reference#static-route-list|qs sr-list]]</code>.

== Choosing which ports carry which traffic ==

QuantaStor works with all major 10/25/40/56/100GbE network interface cards (NICs) from Intel, Mellanox/NVIDIA, Broadcom, Cisco, Dell, HPE and Supermicro. Most systems have a mix of speeds, and the configuration that matters is which port carries which kind of traffic.

Clear '''iSCSI Portal''' and '''NVMeoF Portal''' on the slower 1GbE ports. That stops initiators from discovering and logging in through them, and reserves those ports for management traffic. A 1GbE port that answers an iSCSI discovery is a port some client will eventually run its I/O over.

Put the 1GbE management ports on a separate network from the ports serving NAS and SAN traffic. If management and storage share a network, the system can choose the 1GbE path for storage traffic and the resulting performance problem is hard to see from either end. Separate networks also keep [[#ARP Flux Protection|ARP flux protection]] effective, because each address is then reachable through exactly one port.

Configure the gateway on the management port only. Use the '''Firewall''' tab to block the protocols each port is not meant to serve, and the table in [[#TCP and UDP ports QuantaStor uses|TCP and UDP ports QuantaStor uses]] to see which TCP and UDP ports each service uses.

== TCP and UDP ports QuantaStor uses ==

Most organizations close every port a product does not need, to keep the attack surface small. This section lists every port a QuantaStor appliance listens on, what it is for, and which entry on the per-port [[#Firewall settings per port|Firewall]] tab controls it, so you can tell what can be closed and what has to stay open for the system to run normally. A dash in the Firewall tab column means the per-port firewall has no entry for that port; control it on your network firewall instead.

QuantaStor does not drop traffic by default. Its own iptables rules accept the ports it needs and drop only what you block on the Firewall tab, so on an appliance you have not locked down, every service below is reachable on every port that has an address.

=== Management and grid ===

{| class="wikitable"
! Port !! Protocol !! Service !! Firewall tab entry !! Notes
|-
| 443 || TCP || Web management interface (HTTPS) || -- || The address you browse to. Keep it open from the networks your administrators work from.
|-
| 80 || TCP || Web management interface (HTTP) || QS Web Management || Redirects to HTTPS on 443 by default. This is safe to close as you'll always be using HTTPS but the redirect is there for convenience.
|-
| 8080 || TCP || Web management interface (HTTPS, alternate port) || QS Web Management || Serves the same interface over HTTPS on a second port. This is available as a backup in case a service or firewall is setup to block it. On example where this can be handy is if the object storage gateways is set to bind to 443 rather than 7480. Safe to close it unless something depends on it.
|-
| 8153 || TCP || [[QuantaStor REST API Reference Guide|REST API]] (HTTPS) || QS REST API || Needed by scripts, automation, plugins and integrations that call the REST API. Safe to close if you're not using the QuantaStor REST API in scripts, QuantaStor Proxmox plug-in or the QuantaStor Python Client library. The 'qs' CLI does not use this interface, it goes direct into the service API on 5151.
|-
| 5151 || TCP || QuantaStor core service (TLS) || -- || Every appliance in a [[Grid Configuration|storage grid]] talks to the others on this port, so it must be open between all grid members in both directions. The <code>qs</code> CLI also uses it when pointed at a remote system.
|-
| 22 || TCP || SSH || -- || Console access for administrators and OSNEXUS support, and the transport for encrypted remote replication (see below). Restrict it to administrator networks and to the other appliances you replicate with.
|}

The core service also listens on TCP 5152, bound to the loopback address only. It needs no rule on your network firewall.

=== Storage protocols ===

{| class="wikitable"
! Port !! Protocol !! Service !! Firewall tab entry !! Notes
|-
| 860, 3260 || TCP || iSCSI target || iSCSI Target || Initiator access to [[Storage Volumes]]. Clearing '''iSCSI Portal''' on a port also stops initiators logging in through it.
|-
| 4420 || TCP || NVMe-oF/TCP target || NVMeoF Target || Host access to [[Storage Volumes]] over [[NVMe-oF Target Configuration|NVMe-oF]]. Clearing '''NVMeoF Portal''' removes the listener on that address.
|-
| 2049, 111 || TCP and UDP || NFSv3 and NFSv4 || NFS || Client access to [[Network Shares]] over NFS. Port 111 is the RPC portmapper.
|-
| 2249 || TCP || NFS Ganesha || NFS Ganesha || The alternate NFS port used for scale-out (Ceph) file shares.
|-
| 137-139, 389, 445, 901 || TCP || SMB || SMB || Client access to [[Network Shares]] over SMB.
|}

=== Scale-out, clustering and replication ===

{| class="wikitable"
! Port !! Protocol !! Service !! Firewall tab entry !! Notes
|-
| 3300, 6789 || TCP || Ceph monitors || Ceph (6789 only) || Between the members of a [[Scale-out Block Setup (ceph)|Ceph cluster]], and from Ceph clients. The Ceph entry on the Firewall tab does not include 3300.
|-
| 6800-7100 || TCP || Ceph OSD and manager daemons || Ceph || Between the members of a Ceph cluster.
|-
| 7480, 7481 || TCP || Ceph object gateway (S3) || Ceph (7480 only) || S3 access to [[Ceph Object Storage|object storage]] buckets. A second gateway instance listens on 7481.
|-
| 5303-5332, 5403-5412 || TCP and UDP || Cluster heartbeat (Corosync) || -- || Between the appliances of an [[Setup Guide for Clustered HA Storage Pools|HA cluster]] or site cluster. QuantaStor also accepts multicast, IGMP and ICMP for the heartbeat. Blocking any of these splits the cluster.
|-
| 22 || TCP || Remote replication, encrypted || -- || The default. The source appliance sends the replication stream over SSH to the target appliance.
|-
| 20000-24999 || TCP || Remote replication, unencrypted || -- || Only for a replication link whose '''Encryption''' box is cleared. The target opens a receiver on a random port in this range for each transfer, and the source still uses SSH on 22 to start it. Open this range from source to target if you use unencrypted links.
|}

For [[Remote-replication (DR)|remote replication]] both appliances must be in the same grid, so TCP 5151 between them is needed as well as the replication transport.

=== Monitoring ===

{| class="wikitable"
! Port !! Protocol !! Service !! Firewall tab entry !! Notes
|-
| 3000 || TCP || [[Grafana Integration|Grafana]] || Grafana || Dashboards.
|-
| 9090, 9093, 9100, 9283 || TCP || [[Grafana Ceph Dashboard & Prometheus Integration|Prometheus]] || Prometheus || Prometheus server, Alertmanager, node exporter and the Ceph manager exporter.
|-
| 8888 || TCP || [[Chronograf Integration|Chronograf]] || Chronograf || Real-time statistics dashboard.
|-
| 161 || UDP || SNMP agent || -- || Only when the [[SNMP Agent Setup|SNMP agent]] is in use.
|}

The ports behind each Firewall tab entry are read from {{Code|1=/opt/osnexus/quantastor/conf/qs_services_firewall.conf}} on the appliance, and the static accept rules from the QuantaStor iptables service. Check those files on your own system if the table above and your appliance ever disagree.

== External sites QuantaStor connects to ==

These are the outbound connections an appliance makes. If your firewall restricts outbound traffic, allow the ones for the features you use. Every one of them is optional for serving storage: an appliance with no outbound access at all still serves its pools, shares and volumes, but cannot be activated online, upgraded from the public repositories or send logs to support.

{| class="wikitable"
! Purpose !! Destination !! Port !! Notes
|-
| Online license activation || <code>licensing.osnexus.com</code> || TCP 443 (HTTPS) || Used when you activate a license online and when the system reports license status. Without it, activate offline instead -- see [[Storage System Activate License Offline]] and [[License Management]].
|-
| Upgrades (QuantaStor packages) || <code>packages.osnexus.com</code> || TCP 80 (HTTP) || The QuantaStor package repository the [[Upgrade Manager]] installs from.
|-
| Upgrades and [[Security Updates|security updates]] (operating system) || <code>archive.ubuntu.com</code>, <code>us.archive.ubuntu.com</code>, <code>security.ubuntu.com</code> || TCP 80 (HTTP) || The Ubuntu package mirrors for the base operating system. Where appliances have no Internet access, use a [[QuantaStor Local Update Mirror|local update mirror]] instead.
|-
| [[Send Support Log Files|Sending support logs]] || <code>jira.osnexus.com</code> || TCP 21 (FTP, passive) || The first upload attempt. The log bundle is encrypted before it leaves the system.
|-
| Sending support logs (fallback) || <code>services.osnexus.com</code> || TCP 443 (HTTPS) || Used when FTP is unreachable. When the system has an HTTP proxy configured, QuantaStor skips FTP and uploads over HTTPS through the proxy.
|-
| Sending support logs (collection list) || <code>qstor-downloads.s3.amazonaws.com</code> || TCP 80 (HTTP) || Fetches the latest list of what to collect before building the bundle.
|-
| Email alerts || Your SMTP server || The port you configure || Set in the [[Storage System Alert Manager|Alert Manager]]. See [[Call-home / Alerting]].
|-
| Alert integrations || The service's own endpoint, for example [[Slack Integration|Slack]], [[PagerDuty Integration|PagerDuty]] or [[OpsGenie Integration|OpsGenie]] || TCP 443 (HTTPS) || Only for the integrations you configure.
|-
| Time synchronization || Your NTP servers, or <code>ntp.ubuntu.com</code> when none are set || UDP 123 || Set NTP servers per system or for the whole grid -- see [[#Where network ports are managed|Where network ports are managed]].
|-
| Name resolution || Your DNS servers || UDP and TCP 53 || Every external site on this list is reached by name.
|-
| Encryption key server || Your KMIP key server || TCP 5696 by default || Only when a [[Create Key Server Profile|key server profile]] is configured.
|-
| Cloud backup and cloud containers || The cloud provider's endpoint, such as <code>s3.amazonaws.com</code> || TCP 443 (HTTPS) || Only for the [[Cloud Containers / NAS Gateway|cloud containers]] and [[Backup Policies|backup policies]] you configure.
|-
| Remote replication || The other appliance's address || TCP 22, TCP 5151, and TCP 20000-24999 for unencrypted links || See [[#Scale-out, clustering and replication|Scale-out, clustering and replication]] above.
|}

== Locking down the network ==

Use two layers. The per-port [[#Firewall settings per port|Firewall]] tab and [[Firewall Management]] block services by destination address on the appliance itself; your network firewall handles everything those settings have no entry for, and all outbound traffic.

* '''Keep open between grid members:''' TCP 5151. Add TCP 22 between systems that replicate to each other, the Corosync ranges between HA and site cluster members, and the Ceph ports between scale-out cluster members.
* '''Keep open from administrator networks:''' TCP 443 for the web interface, TCP 22 if you use SSH, and TCP 8153 if anything calls the REST API.
* '''Keep open from clients:''' only the storage protocols they use -- iSCSI, NVMe-oF, NFS, SMB or S3 -- and only on the ports that serve them.
* '''Safe to close if unused:''' TCP 80 and 8080, the monitoring ports, NFS Ganesha, and every storage protocol you do not serve.

Block per port rather than system-wide where you can. A storage port that only serves iSCSI should block NFS, SMB and the web interface; a management port should block the storage protocols. After locking down, run the [[#Verifying connectivity|Network Check]] and send a test alert to confirm nothing you rely on was cut off.

== Verifying connectivity ==

{{Navigation|Storage Management → Storage Systems → ''select a Storage System'' → Health Checker ''(toolbar)''}}

The Network Check report tests reachability from each of the system's ports, including whether jumbo frames get through end to end, and reports the round-trip time for each destination. Run it after changing an MTU, since an MTU mismatch is exactly the kind of fault that leaves a port looking healthy while large transfers stall. See [[Network Check View]].

The CLI equivalent is <code>[[QuantaStor CLI Command Reference#network-check|qs net-check]]</code>:

<pre style="font-size: smaller">
qs net-check
</pre>

== Command line reference ==

Every operation on this page has a CLI equivalent. Add <code>--verbose</code> to any command to see its full argument list.

{| class="wikitable"
! Command !! Purpose
|-
| <code>[[QuantaStor CLI Command Reference#network-port-list|qs np-list]]</code> / <code>[[QuantaStor CLI Command Reference#network-port-get|qs np-get]]</code> || List all ports, or show one in detail.
|-
| <code>[[QuantaStor CLI Command Reference#network-port-modify|qs np-modify]]</code> || Change addressing, MTU, portal flags, autotuning, lossless networking and the per-port firewall.
|-
| <code>[[QuantaStor CLI Command Reference#network-port-enable|qs np-enable]]</code> / <code>[[QuantaStor CLI Command Reference#network-port-disable|qs np-disable]]</code> || Bring a port online or offline.
|-
| <code>[[QuantaStor CLI Command Reference#network-port-restart|qs np-restart]]</code> / <code>[[QuantaStor CLI Command Reference#network-port-rescan|qs np-rescan]]</code> || Restart one port, or rediscover all ports.
|-
| <code>[[QuantaStor CLI Command Reference#network-port-rename|qs np-rename]]</code> / <code>[[QuantaStor CLI Command Reference#network-port-hide|qs np-hide]]</code> || Rename a port, or hide it from the interface.
|-
| <code>[[QuantaStor CLI Command Reference#bonded-interface-create|qs bond-create]]</code> / <code>[[QuantaStor CLI Command Reference#bonded-interface-delete|qs bond-delete]]</code> || Create and delete bonded ports.
|-
| <code>[[QuantaStor CLI Command Reference#vlan-interface-create|qs vlan-create]]</code> / <code>[[QuantaStor CLI Command Reference#vlan-interface-delete|qs vlan-delete]]</code> || Create and delete VLAN interfaces.
|-
| <code>[[QuantaStor CLI Command Reference#virtual-interface-create|qs vif-create]]</code> / <code>[[QuantaStor CLI Command Reference#virtual-interface-delete|qs vif-delete]]</code> || Create and delete virtual interfaces.
|-
| <code>[[QuantaStor CLI Command Reference#static-route-create|qs sr-create]]</code> / <code>[[QuantaStor CLI Command Reference#static-route-delete|qs sr-delete]]</code> / <code>[[QuantaStor CLI Command Reference#static-route-list|qs sr-list]]</code> / <code>[[QuantaStor CLI Command Reference#static-route-get|qs sr-get]]</code> || Manage static routes.
|-
| <code>[[QuantaStor CLI Command Reference#ha-interface-create|qs hai-create]]</code> / <code>[[QuantaStor CLI Command Reference#site-vif-create|qs svr-create]]</code> || Create HA pool and site cluster virtual interfaces.
|-
| <code>[[QuantaStor CLI Command Reference#network-check|qs net-check]]</code> || Run the network reachability and jumbo frame report.
|}

Note that the <code>--port-type</code> argument of <code>[[QuantaStor CLI Command Reference#network-port-modify|qs np-modify]]</code> documents only <code>static</code> and <code>dhcp</code>. To remove a port's configuration entirely, use the '''Config Type''' setting of <code>disabled</code> in the web interface.

Set the ARP Filtering policy in the web interface, as described in [[#ARP Filtering policy|ARP Filtering policy]].

== Related pages ==

* [[Storage System#DNS Servers|Storage System]] -- the DNS, NTP and network settings on the Storage System Modify dialog
* [[Storage System Modify]] -- the Network Settings tab, which holds the ARP Filtering policy
* [[Network Port Usage]] -- the older stand-alone list of TCP and UDP ports
* [[Firewall Management]] -- system-level firewall settings, which the per-port settings inherit from
* [[Storage System]] -- the storage system object these ports belong to
* [[Storage Pools]] -- pools whose HA failover groups own the floating virtual interfaces
* [[Network Shares]] -- NAS shares served over the ports configured here
* [[Storage Volumes]] -- SAN volumes served over the iSCSI and NVMe-oF portals configured here
* [[Setup Guide for Clustered HA Storage Pools]] -- cluster heartbeat and HA virtual interface setup
* [[Remote-replication (DR)]] -- replication between appliances
* [[QuantaStor Local Update Mirror]] -- upgrading appliances that have no Internet access
* [[High-availability VIF Management]] -- how floating cluster addresses fail over
* [[QuantaStor Touch Files]] -- touch files, including the one that turns ARP flux protection off
* [[Network Check View]] -- the network reachability report
* [[QuantaStor CLI Command Reference]] -- full argument lists for every command named on this page

----
''Verified against QuantaStor 6.9.0.''

File:Docs-arp-flux-protection-for-multi-homed-ha-arp-filtering.png

2026-10-07T05:45:18Z

Qadmin: osn-seo-utilities: screenshot for Network Ports (docs-arp-flux-protection-for-multi-homed-ha)

osn-seo-utilities: screenshot for Network Ports (docs-arp-flux-protection-for-multi-homed-ha)

+ Admin Guide Overview

2026-10-07T05:45:12Z

Qadmin: osn-seo-utilities: list ISCSI Target Configuration under Storage Provisioning (docs-iscsi-target-presentation-of-storage-volumes-t)

[[Category:admin_guide]]
The Administrator Guide provides documentation on how to configure all aspects of QuantaStor systems and storage grids, with a primary focus on the web user interface (WUI) and a secondary focus on how to automate using the QuantaStor CLI. In contrast the [[+ User Guide Overview|User Guide]] focuses on using QuantaStor storage and accessing it from various operating systems, and on the client-side configuration steps that involves.

It is written for IT administrators setting up or maintaining a system or grid, and for anyone wanting a deeper understanding of how the platform works. Each section below groups related topics, with a one-line summary of what each page covers so you can find the right one without opening several. If you are new to the interface, start with [[Using the Web Interface]]; if you are bringing up a new appliance, start with '''System & Grid'''. For the commands referenced throughout, see the [[+ CLI Guide Overview|CLI Guide]] and the [[QuantaStor CLI Command Reference]].

== Using the Web Interface ==

The web management interface itself: its screen regions, its nine main tabs, and the navigation conventions this guide uses to describe it.

{| class="wikitable"
! Topic !! Description
|-
| [[Using the Web Interface]] || Logging in, every region of the screen, the nine main tabs, toolbar groups and the overflow chevron, right-click menus, and how to read a '''Navigation''' breadcrumb.
|}

== System & Grid ==

Bringing up an appliance, joining it to a grid, licensing it, and recovering its configuration.

{| class="wikitable"
! Topic !! Description
|-
| [[Storage System]] || A single QuantaStor appliance, physical or virtual, and the settings on its Modify dialog.
|-
| [[Storage System Optimization]] || System tunables controlling cache behavior and Storage Pool I/O, and how the tunables file works.
|-
| [[Grid Configuration]] || Combining multiple systems into one Storage Grid managed as a single unit.
|-
| [[License Management]] || Applying and managing the license every system requires before the software can be used.
|-
| [[Recovery Manager]] || Restoring the internal configuration database from one of its automatic backups.
|}

== Hardware Configuration ==

Network interfaces, disks, controllers and third-party enclosures -- the physical layer beneath a Storage Pool.

{| class="wikitable"
! Topic !! Description
|-
| [[Network Ports]] || The Ethernet interfaces used for management and storage access, plus bonding, VLANs, virtual ports and static routes.
|-
| [[Physical Disks/Devices]] || Identifying, scanning, formatting and importing the physical disks a system can see.
|-
| [[Hardware Controllers & Enclosures]] || QuantaStor's integration modules for the major HBA and RAID controller models.
|-
| [[Multipath Configuration]] || Redundant SAS paths to dual-ported media, which QuantaStor configures automatically where it can.
|-
| [[External Systems Configuration]] || Disk enclosures and arrays from other vendors, used as the media behind a Storage Pool.
|-
| style="padding-left: 2.5em;" | [[Western Digital (WD) Data24 Configuration|Western Digital (WD) Data24]] || An NVMe-oF JBOF attached over RDMA (RoCE), including driver and lossless-network setup.
|-
| style="padding-left: 2.5em;" | [[Seagate Corvault Configuration|Seagate Corvault]] || A SAS-attached array that does its own RAID and presents large logical volumes.
|}

== Storage Provisioning ==

Turning physical media into pools, then into the block, file and object storage that clients consume.

{| class="wikitable"
! Topic !! Description
|-
| [[Storage Pools]] || Aggregating one or more disks into a pool of fault-tolerant storage.
|-
| [[Provisioning Tiers]] || Grouping pools together so automated frameworks such as OpenStack Cinder can provision from a tier.
|-
| [[Storage Volumes]] || Block devices (LUNs) accessed over iSCSI, Fibre Channel or Infiniband/SRP.
|-
| style="padding-left: 2.5em;" | [[Clustered SCSI-3 Persistent Reservations]] || Sharing client reservations across an HA group so Hyper-V, WSFC and SQL Server clusters survive a failover.
|-
| [[Fibre Channel Target Port Management]] || Exposing volumes as LUNs to Fibre Channel initiator hosts alongside iSCSI.
|-
| [[Network Shares]] || NAS access to a pool over NFSv3, NFSv4, SMB2 and SMB3.
|-
| [[NFS Configuration]] || The two NFS server implementations, and which one applies to a given storage architecture.
|-
| [[Multi-protocol File Locking]] || Locking on shares used by NFS and SMB clients at once, the setting that makes it coherent, and its limits.
|-
| [[Cloud Containers / NAS Gateway]] || Mapping an object-storage bucket to a Network Share so it can be accessed as files.
|-
| [[ISCSI Target Configuration]] || How QuantaStor presents storage volumes over iSCSI - target IQNs, portals and network ports, discovery, and what an initiator sees after a volume is assigned to a host.
|}

== Security, Alerting & Upgrades ==

Controlling who can do what, learning when a system needs attention, and keeping it patched.

{| class="wikitable"
! Topic !! Description
|-
| [[Security Configuration]] || Users, roles and RBAC, multi-factor authentication, certificates, the firewall and audit logging.
|-
| [[Call-home / Alerting]] || How a grid notifies you when systems need attention, including the ITSM integrations.
|-
| [[Upgrade Manager]] || Kernel, driver, security and core QuantaStor package updates.
|-
| [[Send System Log Report]] || Collecting the configuration and log bundle OSNEXUS support asks for on a new ticket.
|-
| [[FIPS Mode]] || Running the appliance with the FIPS 140-2 validated cryptographic module, and verifying it is active.
|-
| [[Encryption Bypass]] || Why QuantaStor does not encrypt devices that arrays such as Seagate Corvault, Seagate Exos E/EP and Dell PowerVault already encrypt in hardware, and how to add new vendor and model combinations as they are released. Applies to scale-up Storage Pools and scale-out OSDs alike.
|}

== High-availability VIF Management ==

The cluster heartbeat layer and the floating service addresses it manages, for both scale-out (Ceph) and scale-up (ZFS) storage clusters within a grid. Named for the main tab these are managed from.

{| class="wikitable"
! Topic !! Description
|-
| [[High-availability VIF Management]] || How cluster VIFs work: the grid, site cluster, heartbeat ring, HA group and VIF hierarchy, what triggers a failover and how long one takes, what blocks one, moving a VIF deliberately, and location constraints.
|-
| [[Site Cluster Setup]] || Creating a site cluster and its heartbeat rings, adding and removing members, and the corosync and pacemaker configuration it puts in place.
|-
| [[Cluster VIFs]] || Adding a cluster virtual interface and choosing its use case -- grid primary, scale-up storage pool, scale-out storage pool -- and what each type follows.
|-
| [[Configure Member Standby]] || Setting a node into standby moves all VIF resources and pools off it, so maintenance can be done without risk of a pool or resource moving onto it while the work is in progress.
|}

== Snapshots & Replication ==

Point-in-time copies, scheduled backups, and getting data to a second site.

{| class="wikitable"
! Topic !! Description
|-
| [[Snapshot Schedules]] || Automating space-efficient point-in-time snapshots of volumes and shares.
|-
| [[Backup Policies]] || Backing up any NFS or CIFS share on your network onto a QuantaStor system.
|-
| [[Remote-replication (DR)]] || Replicating both Storage Volumes (SAN) and Network Shares (NAS) to a remote system.
|-
| [[Bucket Synchronization Policies]] || Keeping object storage buckets in step across sites.
|}

== Scale-out Cluster Configuration ==

Scale-out Ceph cluster setup and configuration.

{| class="wikitable"
! Topic !! Description
|-
| [[Scale-out Block Setup (ceph)]] || Scale-out SAN built on Ceph.
|-
| [[Scale-out File Setup (ceph)]] || Scale-out NAS with access over native CephFS, SMB and NFS.
|-
| [[Scale-out Object Setup (ceph)]] || Scale-out object storage over the S3-compatible REST protocols.
|}

== Scale-up Cluster Configuration ==

Highly available scale-up pool cluster configuration.

{| class="wikitable"
! Topic !! Description
|-
| [[HA Cluster Setup (JBODs)]] || Clustered pools on shared SAS JBODs, so a node or path outage does not take storage offline.
|-
| [[HA Cluster Setup (external SAN)]] || The same clustered pool configuration where the media comes from an external SAN.
|}

== Optimization ==

Matching pool behavior to your workload, and watching how the grid performs.

{| class="wikitable"
! Topic !! Description
|-
| [[Performance Tuning]] || Pool I/O profiles, which set read-ahead, request queue depth and the I/O scheduler.
|-
| [[Performance Monitoring]] || The Grid Dashboard and the health and performance views built on it.
|}

== System Internals ==

How the platform is put together, for deeper troubleshooting and for extending it.

{| class="wikitable"
! Topic !! Description
|-
| [[QuantaStor systemd Services]] || The internal services and the systemd units under <code>/lib/systemd/system</code> that manage them.
|-
| [[QuantaStor Configuration Files]] || The configuration files that let engineering and support customize platform behavior.
|-
| [[QuantaStor Shell Utilities]] || The <code>qs</code> CLI plus the <code>qs-*</code> helpers such as <code>qs-iostat</code>, <code>qs-sendlogs</code> and <code>qs-upgrade</code>.
|-
| [[QuantaStor Custom Scripting / Application Extensions]] || Script call-outs for integrating QuantaStor with custom applications.
|}

----
''Verified against QuantaStor 6.9.0.''

ISCSI Target Configuration

2026-10-07T05:45:12Z

Qadmin: osn-seo-utilities: docs-iscsi-target-presentation-of-storage-volumes-t @ a4968c7b29ee (approved in the portal)

QuantaStor presents each Storage Volume to iSCSI clients as its own target, with its own IQN, on the network ports you allow for iSCSI. This page covers how the target IQN is formed, which addresses a target listens on, how assigning a volume to a host opens the target to that host's initiator, how CHAP is applied, and what the initiator sees once it logs in.

__TOC__

{| class="wikitable"
! Section !! Purpose
|-
| [[#One target per Storage Volume|One target per Storage Volume]] || The target IQN format and why every iSCSI volume is LUN 0
|-
| [[#Portals: which network ports serve iSCSI|Portals: which network ports serve iSCSI]] || The iSCSI Portal setting on a network port, TCP port 3260, HA VIFs and Portal Groups
|-
| [[#Hosts and initiator IQNs|Hosts and initiator IQNs]] || Registering a client by its initiator IQN
|-
| [[#Assigning a Storage Volume to a host|Assigning a Storage Volume to a host]] || What an assignment programs on the target
|-
| [[#Discovery and login|Discovery and login]] || What a client needs to find and log in to a target
|-
| [[#CHAP authentication|CHAP authentication]] || Per-volume, per-user and Resource Group CHAP
|-
| [[#What the initiator sees|What the initiator sees]] || Vendor and product strings, device identifiers
|-
| [[#iSCSI sessions|iSCSI sessions]] || Viewing and dropping active sessions
|-
| [[#Troubleshooting|Troubleshooting]] || A volume that does not appear on the client
|-
| [[#CLI reference|CLI reference]] || The commands used on this page
|}

== One target per Storage Volume ==

QuantaStor uses the SCST iSCSI target. Every Storage Volume gets its own iSCSI target, and assigning the volume to more hosts adds initiators to that same target rather than creating new ones. Because each volume has a unique IQN, the volume is always presented at '''LUN 0''' of its target. The LUN number you can set on a Storage Volume, labelled FC LUN, applies only to Fibre Channel; see [[Fibre Channel Target Port Management]].

A target IQN has three parts after the fixed prefix:

<pre style="font-size: smaller">
iqn.2009-10.com.osnexus:9c30f734-0d4edacd5d071f74:volume-vmwaretest.001
| | |
| | +-- Storage Volume name (underscores become dots)
| +-- first 16 hex digits of the Storage Volume ID
+-- first 8 hex digits of the Storage Pool ID
</pre>

The pool part is what ties a target to its pool. QuantaStor uses it to decide which portals a target is offered on, which is how a volume in an HA pool follows the pool's virtual interface (see below). If a volume is in a namespace, the namespace is inserted before the pool part.

The IQN is shown in the IQN row of the Storage Volume's Properties panel and in the Target IQN/WWN column of the Storage Volumes grid. It is also returned by <code>[[QuantaStor CLI Command Reference#volume-get|qs volume-get]] --volume=<volume></code>.

The IQN format is controlled by the {{Code|1=[iqn]}} section of {{Code|1=/etc/quantastor.conf}}. Leave it at the defaults: initiators that are already logged in identify the volume by its IQN, so a change affects every existing client.

== Portals: which network ports serve iSCSI ==

A portal is an IP address and TCP port on which a target accepts logins. QuantaStor's iSCSI target listens on TCP port '''3260'''. If a firewall sits between the clients and the appliance, it must allow that port; see [[Firewall Configuration]].

=== The iSCSI Portal setting on a network port ===

{{Navigation|Storage Management → Storage Systems ''(section)'' → ''select a Storage System'' → Network Ports ''(tab)'' → ''select a port'' ''(select + right-click)'' → Modify Network Port}}

Each network port has an '''iSCSI Portal''' checkbox in the Modify Network Port dialog. The dialog's tooltip describes it: ''by checking this option this interface becomes an allowed portal for the protocol.'' A target is offered only on ports that have iSCSI enabled and a valid IP address. Clear the checkbox on management or replication ports to keep block traffic off them. The checkbox is not available on the grid management virtual interface. [[Network Ports]] covers the rest of the dialog.

From the CLI, use <code>[[QuantaStor CLI Command Reference#network-port-modify|qs network-port-modify]] --port=<port> --iscsi-enable=true</code>. The Properties panel for a port shows the current setting in the iSCSI Enabled? row.

If no port on the appliance has iSCSI enabled, QuantaStor binds the targets to the loopback address only, which blocks all external iSCSI access.

=== HA pools and virtual interfaces ===

For a volume in a high-availability Storage Pool, QuantaStor offers the target '''only''' on the pool's HA virtual interfaces (VIFs), not on the appliance's physical port addresses. The VIF moves with the pool on failover, so a client that logged in through it reconnects to the node that now owns the pool. Point initiators at the VIF address, never at a node's own IP. Configure the VIF in [[High-availability VIF Management]], and enable iSCSI on it when you create it.

Volumes in a scale-out (Ceph) pool are offered on the site cluster VIFs by default; see [[Cluster VIFs]].

=== Portal Groups ===

A Portal Group restricts a set of Storage Volumes to a specific set of cluster VIFs and FC ports, for example to keep one group of clients on one storage network. Portal Groups live in the Hosts & Portal Groups section. A Portal Group belongs to one Storage Pool, and a Storage Volume can be in at most one Portal Group. Create one with <code>[[QuantaStor CLI Command Reference#portal-group-create|qs portal-group-create]] --name=<name> --pool=<pool></code>, or list the portals a pool can use with <code>[[QuantaStor CLI Command Reference#pool-portals-list|qs pool-portals-list]] --pool=<pool></code>.

Resource Group network settings narrow the allowed portals further for volumes that belong to a tenant; see [[Create Resource Group]].

== Hosts and initiator IQNs ==

[[File:Docs-iscsi-target-host-add.png|thumb|right|600px|The Add Host dialog. Choose iSCSI Initiator (IQN) and paste the client's initiator IQN.]]

{{Navigation|Storage Management → Hosts & Portal Groups ''(section)'' → Host ''(toolbar group)'' → Add ''(toolbar)''}}

QuantaStor grants access by initiator, so each client must exist as a host with its initiator IQN. Copy the IQN from the client: on Linux it is in {{Code|1=/etc/iscsi/initiatorname.iscsi}}; on VMware ESXi it is shown on the software iSCSI adapter (see [[VMware Configuration]]); on Windows it is on the Configuration tab of the iSCSI Initiator.

The Add Host dialog has these fields:

{| class="wikitable"
! Field !! Notes
|-
| '''Host Name''' || A name for the host in QuantaStor. It does not have to match the client's hostname. Letters, digits and {{Code|1=- _ .}} only.
|-
| '''Description''' || Optional.
|-
| '''Operating System Type''' || Windows, Mac OS X, Linux, Solaris, AIX, HP-UX, VMware, XenServer or Other. Windows is the default.
|-
| '''Initiator''' || One of '''iSCSI Initiator (IQN)''', '''FC Initiator WWPN''' or '''NVMeoF Initiator (NQN)'''. For iSCSI, select the first option and enter the IQN, for example {{Code|1=iqn.1991-05.com.example:iscsihost-03fo1500}}.
|}

The dialog takes one initiator. A client with several initiators, or a cluster node that should be reachable under one host entry, gets the others through '''Add Initiator''' on the Host toolbar group, or <code>[[QuantaStor CLI Command Reference#host-initiator-add|qs host-initiator-add]] --host=<host> --iqn=<iqn></code>. Only entries that begin with {{Code|1=iqn.}} are used for iSCSI access; WWPNs and NQNs are used by the FC and NVMe-oF targets.

To give a group of clients, such as the nodes of a hypervisor cluster, the same volumes, put the hosts in a Host Group and assign volumes to the group. [[Hosts and Host Groups]] covers hosts, host groups and their dialogs in full.

The CLI equivalent of the dialog is <code>[[QuantaStor CLI Command Reference#host-add|qs host-add]] --hostname=<name> --iqn=<iqn> --host-type=linux</code>.

== Assigning a Storage Volume to a host ==

[[File:Docs-iscsi-target-volume-assign.png|thumb|right|438px|The Assign/Unassign Storage Volume dialog. Tick the hosts, or switch to the Host Groups tab, then click OK.]]

{{Navigation|Storage Management → Storage Volumes ''(section)'' → ''select a Storage Volume'' → Assign ''(toolbar)''}}

The same dialog is on the right-click menu of a Storage Volume as '''Assign/Unassign Host Access...'''. To work from the other direction, select a host and use '''Assign''' on the Host toolbar group, or '''Assign Volumes...''' on its right-click menu, to choose several volumes for one host.

The dialog lists every host and host group. Ticked entries have access to the volume; clearing a tick removes it. The '''Release/free unused storage volume LUN number(s)''' checkbox concerns FC LUN numbers only and has no effect on iSCSI, where the LUN is always 0.

When you click OK, QuantaStor updates the volume's target:

# The target gets an access list (an SCST initiator group) containing the iSCSI IQNs of every assigned host and every host in an assigned host group. New initiators are added before removed ones are taken out.
# The volume's CHAP settings are applied to the target.
# The target is enabled, on the allowed portals described above.

'''A target is enabled only while it has at least one initiator.''' An unassigned volume, or one assigned only to hosts that have no iSCSI IQN, has no active iSCSI target, and no client can log in to it.

From the CLI: <code>[[QuantaStor CLI Command Reference#volume-assign|qs volume-assign]] --volume=<volume> --host-list=<host></code>, reversed with <code>[[QuantaStor CLI Command Reference#volume-unassign|qs volume-unassign]]</code>. <code>[[QuantaStor CLI Command Reference#volume-assign-list|qs volume-assign-list]] --host=<host></code> lists what a host can reach.

== Discovery and login ==

QuantaStor does not configure iSNS. Clients find targets by SendTargets discovery against a portal address:

* For a volume in a standalone pool, use the IP address of a network port with iSCSI enabled.
* For a volume in an HA pool, use the pool's HA VIF address.
* For a volume in a scale-out pool, use a site cluster VIF address.

On a Linux client with open-iscsi, discover and log in like this:

<pre style="font-size: smaller">
iscsiadm -m discovery -t sendtargets -p 10.0.20.50:3260
iscsiadm -m node -T iqn.2009-10.com.osnexus:9c30f734-0d4edacd5d071f74:vol1 -p 10.0.20.50:3260 --login
</pre>

A client discovers a target only on the portals that the target is allowed on. To reach a volume over several networks for multipathing, enable iSCSI on a port in each network and log in once per portal; see [[Multipath Configuration]]. Client-side setup for Linux and Windows is covered in [[ISCSI Initiator Setup]], and for ESXi in [[VMware Configuration]].

== CHAP authentication ==

{{Navigation|Storage Management → Storage Volumes ''(section)'' → ''select a Storage Volume'' → Modify ''(toolbar)'' → Security Settings ''(tab)''}}

CHAP is set per Storage Volume, so it applies to the volume's target and to every host assigned to it. Hosts have no CHAP settings. The Security Settings tab of the Storage Volume Create and Modify dialogs has a '''CHAP Authentication & Policy Settings''' group with three options:

{| class="wikitable"
! Option !! Credentials used
|-
| '''Use User's default CHAP user/pass if available''' || The default CHAP username and password of the QuantaStor user who owns the volume.
|-
| '''Use Resource Group default CHAP user/pass''' || The CHAP credentials of the volume's Resource Group, for multi-tenant setups.
|-
| '''Use Volume specific CHAP user/pass specified below''' || The '''CHAP Username''' and '''CHAP Password''' entered in the dialog.
|}

The username may not contain spaces. The password must be 12 to 16 alphanumeric characters. CHAP is disabled by default. Once it is enabled, the client must be configured with the same username and password before it can log in, so set the client first or expect existing sessions to fail at their next login.

From the CLI, use <code>[[QuantaStor CLI Command Reference#volume-modify|qs volume-modify]] --volume=<volume> --chap-policy=target --chap-user=<user> --chap-pass=<password></code>. The other {{Code|1=--chap-policy}} values are {{Code|1=user-defaults}}, {{Code|1=cloud-defaults}} (the Resource Group credentials) and {{Code|1=disabled}}.

QuantaStor configures one-way CHAP, where the target authenticates the initiator. Mutual CHAP is off.

== What the initiator sees ==

After login, the client sees one disk per target at LUN 0, with:

{| class="wikitable"
! Property !! Value
|-
| SCSI vendor || {{Code|1=OSNEXUS}}
|-
| SCSI product || {{Code|1=QUANTASTOR}}
|-
| Device identification || An NAA type 6 identifier, {{Code|1=62000000}} followed by a descriptor built from the Storage Volume ID and the Storage System ID, plus a T10 vendor ID and a unit serial number built from the same descriptor
|-
| Size || The Storage Volume's size. A resize is visible after the client rescans the device.
|}

The identifiers come from the Storage Volume's ID rather than its name, and they are the same on every portal the target is offered on. That is what lets a multipath driver recognise the paths through each portal as one device. The Properties panel of a Storage Volume also shows a VMware EUI identifier, which is how [[VMware Configuration]] matches a datastore device to its volume.

Per-volume iSCSI tuning, such as burst lengths and queued commands, comes from the volume's IO profile rather than from appliance-wide settings; see [[Storage Volumes]].

== iSCSI sessions ==

{{Navigation|Storage Management → Storage Volumes ''(section)'' → ''select a Storage Volume'' → Sessions ''(tab)''}}

The Sessions tab below the Storage Volumes grid lists the active iSCSI sessions for the selected volume: Storage System, Session, State, Storage Volume, Initiator IQN/WWN, Initiator IP, Target IQN/WWN, Reads, Writes and Created. It is the quickest way to confirm that a client actually logged in, and from which address.

To end a session, right-click it and choose '''Drop Session'''; see [[Session Drop]]. The CLI help recommends removing the host's assignment instead, because dropping a session does not take away the host's access.

From the CLI: <code>[[QuantaStor CLI Command Reference#volume-session-list|qs volume-session-list]] --volume=<volume></code> and <code>[[QuantaStor CLI Command Reference#volume-session-close|qs volume-session-close]] --volume=<volume> --session-list=<session></code>.

== Troubleshooting ==

'''Discovery returns no targets.'''
* Check that the volume is assigned to a host, and that the host entry has the client's exact initiator IQN. A typo in the IQN leaves the target with no matching initiator.
* Check that the address you discover against is on a port with iSCSI Portal enabled, or, for an HA pool, is the pool's VIF.
* Check that TCP port 3260 is open between the client and the appliance.

'''Discovery works but login fails.'''
* CHAP mismatch: compare the volume's CHAP settings with the client's.
* The client is using a node IP for a volume in an HA pool. Use the VIF.

'''The client logs in but sees no disk.'''
* Rescan the client's iSCSI adapter. On ESXi, rescan storage after every new assignment.

'''A volume disappears from a client.'''
* Check whether the assignment was removed, the host's initiator IQN changed, or the port's iSCSI Portal setting cleared.

== CLI reference ==

{| class="wikitable"
! Command !! Purpose
|-
| <code>[[QuantaStor CLI Command Reference#host-add|qs host-add]]</code> || Add a host with its initiator IQN
|-
| <code>[[QuantaStor CLI Command Reference#host-initiator-add|qs host-initiator-add]]</code> || Add another initiator IQN to a host
|-
| <code>[[QuantaStor CLI Command Reference#volume-assign|qs volume-assign]]</code> || Give hosts access to a Storage Volume
|-
| <code>[[QuantaStor CLI Command Reference#volume-unassign|qs volume-unassign]]</code> || Remove access
|-
| <code>[[QuantaStor CLI Command Reference#volume-assign-list|qs volume-assign-list]]</code> || List volume-to-host assignments
|-
| <code>[[QuantaStor CLI Command Reference#volume-modify|qs volume-modify]]</code> || Set the CHAP policy and credentials
|-
| <code>[[QuantaStor CLI Command Reference#network-port-modify|qs network-port-modify]]</code> || Enable or disable iSCSI on a port
|-
| <code>[[QuantaStor CLI Command Reference#portal-group-create|qs portal-group-create]]</code> || Restrict volumes to a set of VIFs and FC ports
|-
| <code>[[QuantaStor CLI Command Reference#volume-session-list|qs volume-session-list]]</code> || List active iSCSI sessions
|-
| <code>[[QuantaStor CLI Command Reference#volume-session-close|qs volume-session-close]]</code> || Close an iSCSI session
|}

== Related pages ==

* [[Hosts and Host Groups]]
* [[Storage Volumes]]
* [[Network Ports]]
* [[High-availability VIF Management]]
* [[ISCSI Initiator Setup]]
* [[Multipath Configuration]]
* [[VMware Configuration]]
* [[Fibre Channel Target Port Management]]
* [[NVMe-oF Target Configuration]]
* [[QuantaStor CLI Command Reference]]

[[Category:admin_guide]]

----
''Verified against QuantaStor 6.9.0.''

File:Docs-iscsi-target-volume-assign.png

2026-10-07T05:45:12Z

Qadmin: osn-seo-utilities: screenshot for ISCSI Target Configuration (docs-iscsi-target-presentation-of-storage-volumes-t)

osn-seo-utilities: screenshot for ISCSI Target Configuration (docs-iscsi-target-presentation-of-storage-volumes-t)

File:Docs-iscsi-target-host-add.png

2026-10-07T05:45:12Z

Qadmin: osn-seo-utilities: screenshot for ISCSI Target Configuration (docs-iscsi-target-presentation-of-storage-volumes-t)

osn-seo-utilities: screenshot for ISCSI Target Configuration (docs-iscsi-target-presentation-of-storage-volumes-t)

Guides:Nightly Database Analysis from iSCSI Snapshots

2026-10-07T05:45:08Z

Qadmin: osn-seo-utilities: db-analysis-iscsi-snapshot @ cf7d5275edd5 (approved in the portal)

''By Steve Umbehocker, CTO, OSNexus · Updated October 3, 2026''

Offline database analysis means running reports, audits and heavy queries against a point-in-time copy of production data on a separate server, so that the production database never sees the load. QuantaStor makes that copy with a snapshot schedule that snapshots every database volume in a pool at the same instant, presents those snapshots to an analysis server over iSCSI, and rotates old copies out on its own. This guide builds the whole nightly loop with the <code><nowiki>qs</nowiki></code> CLI and a short bash script that triggers the schedule, waits for the new snapshots, swaps the old mounts for new ones and verifies them.

== Why it matters ==

Nightly reporting, data-quality checks and ad-hoc analysis all compete with production I/O when they run against the live database. Copying databases to a reporting server every night is slow and doubles the capacity bill. A snapshot avoids both: it is created near-instantly, records metadata rather than copying blocks, and consumes pool space only for blocks that change afterwards.

A snapshot schedule adds two things a hand-rolled snapshot script lacks. All ZFS members of a schedule that live in the same [[Storage Pools|storage pool]] are snapshotted by a single ZFS command, so a database spread across data, log and index volumes is captured as one crash-consistent set rather than several copies taken seconds apart. The schedule also owns rotation, so you never write cleanup code that deletes the wrong snapshot.

== How it works ==

The moving parts on the appliance side are all documented reference features:

* '''[[Snapshot Schedules]]''' take the snapshots, name them <code><nowiki><volume>_GMT<YYYYMMDD>_<HHMMSS></nowiki></code> from the UTC start time of the run, and keep only '''Max Short Term Snapshots''' of each volume in the rotating window.
* '''Triggering''' runs a schedule immediately, in addition to its timed runs. From the CLI it is <code><nowiki>qs snap-schedule-trigger</nowiki></code>, and over the [[QuantaStor REST API Reference Guide|REST API]] it is the <code><nowiki>snapshotScheduleTrigger</nowiki></code> call. Triggering a disabled schedule does nothing, so the schedule must stay enabled.
* '''Lazy cloning.''' A schedule snapshot sits in the Offline state with no iSCSI target until it is needed. Assigning it to a host materializes it: the clone is created, the target is registered and the state moves to Normal. Each volume, snapshots included, gets its own target IQN and presents at LUN 0 (see [[Storage Volumes]]).
* '''Host assignment''' is the LUN masking that lets only the analysis server see the snapshot ([[Hosts and Host Groups]]).

On the analysis server, open-iscsi logs in to the snapshot's target and the filesystem is mounted like any local disk. Because the snapshot is a writable clone (Read/Write is the default access mode for snapshots), the database engine or the filesystem can replay its journal on mount as it would after a power loss. Those writes land in the clone and never touch the production volume.

== Design and sizing ==

Put every volume of one database, or every database you want captured at the same instant, in the '''same storage pool''' and the '''same schedule'''. Atomicity holds per pool, and '''Disable Atomicity''' on the schedule's Advanced Settings tab breaks it into batches, so leave it off for this use. A schedule run also snapshots only members owned by the appliance it runs on.

The snapshots are crash-consistent. If your database needs a cleaner point, the schedule runs <code><nowiki>/var/opt/osnexus/custom/schedule-prestart.sh</nowiki></code> on the appliance at the start of every run, before any snapshot is taken, with <code><nowiki>--name=<schedule name> --id=<schedule id></nowiki></code>. It blocks the run for up to five minutes, which is enough to ask the database to checkpoint.

Retention for an analysis schedule is short. You only need the snapshot being analyzed and the one about to replace it:

{| class="wikitable"
! Setting !! Suggested value !! Reason
|-
| Max Short Term Snapshots (<code><nowiki>--max-snaps</nowiki></code>) || 3 || The mounted snapshot is never the oldest one in the window
|-
| Hourly to quarterly retention (<code><nowiki>--rc-*</nowiki></code>) || 0 || Long-term tags exempt snapshots from rotation, holding space you don't need
|-
| Schedule type || Calendar, one hour per day || The schedule must be enabled for triggers to work, so give it a harmless timed run
|-
| Offset || Different from any replication schedule on the same volumes || A member snapshotted by another schedule within the last 10 seconds is skipped
|}

Capacity cost tracks write volume, not database size: each snapshot holds the blocks overwritten since it was taken, and the mounted clone adds whatever the analysis server writes. Check '''Max Total Source Snapshots''' on the Snapshot Settings tab against pool free space before you commit.

[[File:Guide-db-analysis-iscsi-snapshot-db-snap-schedule-retention.png|frame|center|The Snapshot Settings tab: clear the long-term counts and keep Max Short Term Snapshots small, and watch the Max Total Source Snapshots figure]]

== Setting it up ==

'''1. Create the schedule.''' In the web interface, go to '''Storage Management → Schedules → Snapshot Schedule → Create''', or use the CLI. This schedule runs at 1 AM every day, keeps three snapshots per volume and no long-term copies:

<pre>qs snap-schedule-create --name=db-nightly \
--days=mon,tue,wed,thu,fri,sat,sun --hours=1am \
--volume-list=db-sales,db-finance,db-hr \
--max-snaps=3 --rc-hourly=0 --rc-dailies=0 --rc-weeklies=0 \
--rc-monthlies=0 --rc-quarterlies=0</pre>

[[File:Guide-db-analysis-iscsi-snapshot-db-snap-schedule-interval.png|frame|center|The Schedule Interval tab: tick the days and the single hour the schedule should fire on its own]]

To add another database later, use <code><nowiki>qs snap-schedule-add --schedule=db-nightly --volume-list=db-ops</nowiki></code> or the '''Update Selections''' dialog ([[Snapshot Schedule Add/Remove Volumes]]). <code><nowiki>qs snap-schedule-assoc-list --schedule=db-nightly</nowiki></code> confirms the members.

'''2. Register the analysis server.''' Read its IQN from <code><nowiki>/etc/iscsi/initiatorname.iscsi</nowiki></code> (install <code><nowiki>open-iscsi</nowiki></code> or <code><nowiki>iscsi-initiator-utils</nowiki></code> if it is missing) and add a Host with that IQN, as described in [[ISCSI Initiator Setup]]:

<pre>qs host-add --hostname=analytics1 --host-type=linux \
--iqn=iqn.1994-05.com.redhat:5a66857c49dc</pre>

[[File:Guide-db-analysis-iscsi-snapshot-db-snap-add-host.png|frame|center|The Add Host dialog: set Operating System Type to Linux and paste the server's IQN into iSCSI Initiator]]

The appliance port at 192.0.2.10 in the examples below must have iSCSI enabled, or discovery returns no portals.

'''3. Install the refresh script''' on the analysis server. It needs the <code><nowiki>qs</nowiki></code> CLI pointed at the appliance (the <code><nowiki>QS_SERVER</nowiki></code> variable or <code><nowiki>~/.qs.cnf</nowiki></code>, per the [[QuantaStor CLI Command Reference|CLI reference]]), plus <code><nowiki>jq</nowiki></code> and <code><nowiki>iscsiadm</nowiki></code>:

<pre>#!/bin/bash
set -euo pipefail # usage: refresh-db-snaps.sh [--use-existing-snaps]

SCHEDULE=db-nightly
HOST=analytics1 # Host object holding this server's IQN
PORTAL=192.0.2.10 # iSCSI-enabled port on the appliance
VOLUMES="db-sales db-finance db-hr" # source volumes; each mounts at $MNT/<volume>
MNT=/mnt/dbsnap
PART=-part1 # set to "" if the filesystem uses the whole disk
STATE=/var/lib/dbsnap # which snapshot is mounted where
WAIT=600 # seconds to wait for new snapshots

USE_EXISTING=0
[ "${1:-}" = "--use-existing-snaps" ] && USE_EXISTING=1
mkdir -p "$STATE"

latest() { # names are <volume>_GMTYYYYMMDD_HHMMSS (UTC): last sorted is newest
qs volume-list --json | jq -r '.. | objects | .name? // empty' \
| grep -E "^${1}_GMT[0-9]{8}_[0-9]{6}\$" | sort -u | tail -1 || true
}

declare -A NEW OLD
if [ "$USE_EXISTING" = 1 ]; then
for v in $VOLUMES; do NEW[$v]=$(latest "$v"); done
else
for v in $VOLUMES; do OLD[$v]=$(latest "$v"); done
qs snap-schedule-trigger --schedule="$SCHEDULE"
deadline=$((SECONDS + WAIT))
for v in $VOLUMES; do
while NEW[$v]=$(latest "$v"); [ "${NEW[$v]}" = "${OLD[$v]}" ]; do
[ "$SECONDS" -lt "$deadline" ] || { echo "no new snapshot of $v" >&2; exit 1; }
sleep 10
done
done
fi

changed=0 # remount only if some volume has a newer snapshot than the one mounted
for v in $VOLUMES; do
[ -n "${NEW[$v]}" ] || { echo "no snapshot of $v yet" >&2; exit 1; }
[ "${NEW[$v]}" = "$(cat "$STATE/$v" 2>/dev/null)" ] || changed=1
done
[ "$changed" = 1 ] || { echo "snapshots unchanged, mounts left in place"; exit 0; }

for v in $VOLUMES; do
# Detach the previous snapshot: unmount, log out, remove the assignment.
if [ -f "$STATE/$v" ]; then
old=$(cat "$STATE/$v"); oldiqn=$(cat "$STATE/$v.iqn")
if mountpoint -q "$MNT/$v"; then umount "$MNT/$v"; fi
iscsiadm -m node -T "$oldiqn" -p "$PORTAL" --logout || true
iscsiadm -m node -T "$oldiqn" -p "$PORTAL" -o delete || true
qs volume-unassign --volume="$old" --host-list="$HOST" || true
rm -f "$STATE/$v" "$STATE/$v.iqn"
fi

# Attach the new one. Assigning it materializes the lazy clone and its target.
snap=${NEW[$v]}
qs volume-assign --volume="$snap" --host-list="$HOST"
iqn=$(qs volume-get --volume="$snap" --json \
| jq -r '.. | objects | .iqn? // empty' | head -1)
iscsiadm -m discovery -t sendtargets -p "$PORTAL" >/dev/null
iscsiadm -m node -T "$iqn" -p "$PORTAL" --login
dev=/dev/disk/by-path/ip-$PORTAL:3260-iscsi-$iqn-lun-0$PART
for _ in $(seq 30); do [ -b "$dev" ] && break; sleep 1; done
mkdir -p "$MNT/$v"
mount "$dev" "$MNT/$v"

# Verify: mounted and not empty, then record it for the next run.
mountpoint -q "$MNT/$v" && [ -n "$(ls -A "$MNT/$v")" ] \
|| { echo "mount check failed for $snap" >&2; exit 1; }
echo "$snap" > "$STATE/$v"; echo "$iqn" > "$STATE/$v.iqn"
echo "$v -> $snap mounted at $MNT/$v"
done</pre>

<code><nowiki>set -e</nowiki></code> stops the script if an unmount fails, which is what you want when a report still has files open: the old snapshot stays mounted and nothing is half-swapped.

== Operating and testing ==

Choose one of two timing models:

* '''Script-driven.''' Cron runs <code><nowiki>refresh-db-snaps.sh</nowiki></code> when you want the copy, for example after the nightly ETL load finishes. The trigger makes the snapshots, and the script waits up to <code><nowiki>WAIT</nowiki></code> seconds for a new name to appear on every volume.
* '''Schedule-driven.''' The schedule's own 1 AM run makes the snapshots and cron runs <code><nowiki>refresh-db-snaps.sh --use-existing-snaps</nowiki></code> at 1:30 AM. The script compares the newest snapshot names with what it last mounted and exits without touching the mounts if nothing is new. This is also the safe way to re-run the script by hand during the day.

[[File:Guide-db-analysis-iscsi-snapshot-db-snap-host-assignments.png|frame|center|The Hosts view: the analysis server's initiator IQN, and in Storage Volume Assignments, the volumes currently assigned to it]]

To test, run the script by hand once and check four things:

# <code><nowiki>qs volume-list</nowiki></code> shows a new <code><nowiki>db-sales_GMT...</nowiki></code> snapshot for each volume, all with the same timestamp.
# <code><nowiki>qs volume-assign-list --host=analytics1</nowiki></code> lists exactly one snapshot per source volume.
# <code><nowiki>iscsiadm -m session</nowiki></code> shows one session per snapshot target.
# The database engine starts against <code><nowiki>/mnt/dbsnap</nowiki></code> and your reports return yesterday's numbers.

Run it a second time with <code><nowiki>--use-existing-snaps</nowiki></code> and confirm it reports the snapshots as unchanged. If no new snapshot ever appears, <code><nowiki>qs snap-schedule-get --schedule=db-nightly</nowiki></code> shows whether the schedule is enabled. A schedule left with no members disables itself.

Old snapshots need no cleanup code. Once the script has unassigned a snapshot ([[Remove Storage Assignment]] is the equivalent in the web interface), the schedule rotates it out when it falls outside the window. To keep one copy for an audit, put a hold on it with <code><nowiki>qs volume-hold-add --volume=<snapshot> --hold-tag=keep-for-audit</nowiki></code>. A held snapshot is excluded from schedule cleanup until the hold is released.

== FAQ ==

=== Is a snapshot taken by the schedule consistent across several database volumes? ===

Yes, for volumes in the same storage pool. The schedule snapshots all of its ZFS members in a pool with a single ZFS command, so they share one point in time and are crash-consistent with each other. Volumes in different pools, or a schedule with Disable Atomicity ticked, don't get that guarantee.

=== Does running analysis on the snapshot slow down the production database? ===

The analysis reads come from the snapshot's clone, which shares unchanged blocks with the production volume in the same pool. The analysis server's queries therefore still use the pool's disks, but none of the work runs inside the production database engine, and writes to the clone never reach the production volume. If pool I/O is a concern, schedule the refresh and heavy reports outside peak hours.

=== What does --use-existing-snaps do? ===

It skips the <code><nowiki>snap-schedule-trigger</nowiki></code> call and works with snapshots the schedule has already made. If the newest snapshot of every volume is the one already mounted, the script exits without unmounting anything. Otherwise it swaps in the newer snapshots exactly as a normal run would.

=== Can I trigger the schedule over the REST API instead of the CLI? ===

Yes. The <code><nowiki>snapshotScheduleTrigger</nowiki></code> call takes the schedule name or ID and signals the schedule manager to run it immediately, the same as <code><nowiki>qs snap-schedule-trigger</nowiki></code>. The CLI is the simpler choice for a bash script, and the API suits tools that already speak JSON-RPC.

=== How much pool space do the snapshots use? ===

Each snapshot holds only the blocks overwritten in production since it was taken, plus anything the analysis server writes to its clone. With Max Short Term Snapshots at 3 and no long-term retention, you hold roughly three days of change per volume.

----
''Part of the [[Guides:Index|QuantaStor Guides]] series. For reference documentation, see the [[Main Page|QuantaStor documentation]].''

[[Category:Guides]]

File:Guide-db-analysis-iscsi-snapshot-db-snap-host-assignments.png

2026-10-07T05:45:07Z

Qadmin: osn-seo-utilities: screenshot for Guides (db-analysis-iscsi-snapshot)

osn-seo-utilities: screenshot for Guides (db-analysis-iscsi-snapshot)

File:Guide-db-analysis-iscsi-snapshot-db-snap-add-host.png

2026-10-07T05:45:07Z

Qadmin: osn-seo-utilities: screenshot for Guides (db-analysis-iscsi-snapshot)

osn-seo-utilities: screenshot for Guides (db-analysis-iscsi-snapshot)

File:Guide-db-analysis-iscsi-snapshot-db-snap-schedule-interval.png

2026-10-07T05:45:07Z

Qadmin: osn-seo-utilities: screenshot for Guides (db-analysis-iscsi-snapshot)

osn-seo-utilities: screenshot for Guides (db-analysis-iscsi-snapshot)

File:Guide-db-analysis-iscsi-snapshot-db-snap-schedule-retention.png

2026-10-07T05:45:07Z

Qadmin: osn-seo-utilities: screenshot for Guides (db-analysis-iscsi-snapshot)

osn-seo-utilities: screenshot for Guides (db-analysis-iscsi-snapshot)

Guides:Encryption at Rest and FIPS Mode for On-Premises Storage

2026-10-07T05:45:04Z

Qadmin: osn-seo-utilities: fips-encryption @ 9d6065261ad2 (approved in the portal)

''By Steve Umbehocker, CTO, OSNexus · Updated October 3, 2026''

Encryption at rest protects the data on a storage pool's media, so drives that leave the data center, whether failed, stolen or decommissioned, can't be read without the keys. QuantaStor encrypts storage pools with AES-256, either in software through Linux LUKS with AES-NI acceleration or in hardware on Opal 2 self-encrypting NVMe drives, with keys held on the appliance, protected by a passphrase or stored on a KMIP key server. A separate FIPS mode switches the appliance's cryptography to the FIPS 140-2 validated OSNEXUS Crypto Library and self-tests it at every start.

== Why it matters ==

Federal, defense, healthcare and financial buyers usually face two separate requirements that are easy to mix up:

* '''Data at rest must be encrypted.''' Lost drives, RMA returns and decommissioned servers must not expose data.
* '''The cryptography must be FIPS validated.''' Many government contracts and agency policies require cryptographic modules validated under the NIST Cryptographic Module Validation Program (CMVP), and an auditor will ask for the certificate number.

Encryption alone covers the first requirement. To cover the second, the module doing the encryption needs a certificate, and that decides which of the options below fits. The [[End-to-end Security]] overview places encryption at rest alongside the other controls agencies typically review: encryption on the wire, role-based access control, password policy and data shredding.

== How it works ==

=== Software encryption ===

[[Software Encryption|Software encryption]] uses the Linux LUKS key management system, so it works on any media type: HDD, SATA/SAS SSD or NVMe. The AES-NI instructions in modern server processors do the encryption. It is applied when the pool is created, because the underlying devices themselves are encrypted. An encrypted pool shows a lock icon in the web interface. When you grow the pool, new devices are encrypted automatically before they're added.

=== Hardware encryption (self-encrypting drives) ===

[[Hardware Encryption|Hardware encryption]] moves the work onto the drive. QuantaStor supports Opal 2 compliant NVMe self-encrypting drives (SEDs), both FIPS and non-FIPS models. For HDD-based SED encryption, the recommended approach is a Seagate Corvault system, which handles encryption in the external enclosure. [[Physical Disks/Devices]] reports each drive's SED capability and status. The older RAID-controller-based encryption is deprecated; HBA (IT) mode controllers are the current standard.

=== Key protection ===

The '''Encryption''' tab of the Create Storage Pool dialog offers three ways to protect a pool's keys:

{| class="wikitable"
! Key protection !! Where the key lives !! Behavior
|-
| Encrypt with No Passphrase || On the appliance || Pool starts automatically at boot; supported in HA groups
|-
| Encrypt with User Defined Passphrase || On the appliance, locked by the passphrase || Pool starts only after an administrator enters the passphrase; not supported in HA groups
|-
| Encrypt and Store on Key Server || On a KMIP key server || Keys are managed centrally through a key server profile
|}

KMIP is the standard protocol for talking to enterprise key managers. You add a [[Create Key Server Profile|key server profile]], then select it when you create a pool. You can also move a pool's locally stored keys onto the key server later by modifying the pool and selecting the profile.

[[File:Guide-fips-encryption-fips-encryption-create-pool-software.png|frame|center|The Encryption tab of Create Storage Pool with Software Encryption selected: AES-256 for every option, the three Key Protection choices, and the passphrase and Key Server Profile fields below them]]

=== FIPS mode ===

[[FIPS Mode|FIPS mode]] controls which cryptographic library the appliance uses. QuantaStor ships two complete OpenSSL library sets and switches between them:

{| class="wikitable"
! Mode !! OpenSSL !! Crypto library
|-
| Non-FIPS (default) || 3.5.7 || Non-FIPS build of the OSNEXUS Crypto Library
|-
| FIPS || 3.1.2 with the OpenSSL FIPS provider || OSNEXUS Crypto Library (FIPS build)
|}

In FIPS mode the appliance runs power-on self-tests and integrity checks, and the QuantaStor service won't start if they fail. The FIPS path stays on OpenSSL 3.1.2 on purpose: the FIPS provider is tied to the OpenSSL release it was built and validated against, so it is pinned separately from the general runtime.

== Design and sizing ==

=== Certified versus compliant ===

Be precise about what is validated. The '''OSNEXUS Crypto Library holds FIPS 140-2 Level 1 validation, [https://csrc.nist.gov/projects/cryptographic-module-validation-program/certificate/4185 NIST CMVP certificate 4185]''', with a public [https://csrc.nist.gov/CSRC/media/projects/cryptographic-module-validation-program/documents/security-policies/140sp4185.pdf non-proprietary security policy]. QuantaStor 6 and 7 have '''not''' been through FIPS 140-3 certification. The crypto library is maintained against current FIPS 140-3 compliant OpenSSL releases, but compliant isn't the same as certified.

Where a deployment requires FIPS certification, the documented recommendation is '''self-encrypting drives with full-disk encryption on FIPS 140-3 validated media'''. That puts the certified cryptographic boundary at the drive, where the media vendor holds the certificate. Running FIPS mode on the appliance as well is still worthwhile: it limits management cryptography to validated algorithms and self-tests them at every start, which satisfies many internal security policies.

=== Choosing an approach ===

{| class="wikitable"
! Requirement !! Recommended configuration
|-
| Encrypt at rest, no formal certification || Software encryption, no passphrase or KMIP
|-
| Certificate required for data at rest || FIPS 140-3 SED media with hardware encryption, plus FIPS mode
|-
| Keys held off the appliance under central control || Software encryption with a KMIP key server profile
|-
| Pool must not start without a person present || Software encryption with a user-defined passphrase (not with HA)
|-
| HA failover between two controllers || Software encryption without a passphrase, or SEDs
|}

=== Performance ===

Software encryption typically reduces a pool's performance by about 15%, and by as much as 30% depending on the CPU and the number of devices. Hardware encryption avoids that CPU cost, so if your media is SED Opal compliant, it's the better option. Plan software-encrypted pools with that headroom in mind, especially all-NVMe pools, where the CPU is more likely to be the bottleneck.

== Setting it up ==

=== Create an encrypted pool ===

# Go to '''Storage Management → Storage Pools → Create'''. See [[Storage Pools]] for the General tab settings.
# On the '''Encryption''' tab, select '''Software Encryption''' or '''Hardware Encryption'''.
# Under '''Key Protection''', choose no passphrase, a user-defined passphrase, or '''Encrypt and Store on Key Server''' with a key server profile.
# Finish creating the pool. Encryption can't be added after the pool is created.

=== Back up the keys ===

Immediately after you create the pool, use '''Storage Pool Encryption → Export Keys''' ([[Storage Pool Export Encryption Keys]]) and keep the backup somewhere secure. If the boot media is lost, you reinstall QuantaStor on new boot devices, run '''Import Keys''' and start the pool. The same ribbon group holds '''Re-Key Pool'''.

[[File:Guide-fips-encryption-fips-encryption-pool-encryption-ribbon.png|frame|center|The Storage Pools view with the Storage Pool Encryption group on the ribbon: Import Keys, Export Keys and Re-Key Pool sit next to the pool and HA group actions]]

=== Enable FIPS mode ===

FIPS mode is set per appliance, from the console as root. A grid has no grid-wide switch, so repeat this on every appliance (see [[Grid Configuration]]). Connect over SSH as <code><nowiki>qadmin</nowiki></code>, then run:

<pre>sudo qs-util enablefips
sudo reboot</pre>

<code><nowiki>enablefips</nowiki></code> repoints the library symlinks and restarts the service, reporting <code><nowiki>FIPS mode enabled in the storage system. Restarting QuantaStor Service</nowiki></code>. The reboot is required. Several integrity checks run only at system startup, and the FIPS state doesn't settle until they have. Restarting the service isn't a substitute.

=== HA and the rest of the security stack ===

Encrypted pools work in [[Create Storage Pool High-Availability Group|storage pool HA groups]] like unencrypted pools, provided they have no passphrase. For data on the wire, use SMB3 encryption, [[IPSec|IPsec]] and HTTPS, and CHAP authentication for iSCSI. Grid management and replication traffic use SSL/TLS 1.2 with strong ciphers; see [[Custom SSL TLS Security]]. Administrative access is covered by [[Security Configuration]] (role-based access control) and the [[Password Policy set|password policy]], which supports HIPAA and CJIS requirements.

== Operating and testing ==

Check the FIPS state after the reboot, not before:

<pre>qs system-get | grep -i fips
sudo /opt/osnexus/quantastor/bin/osn_fipscheck validate-fips
tail -f /var/log/qs/qs_crypto.log</pre>

On an appliance in FIPS mode, <code><nowiki>qs system-get</nowiki></code> reports:

<pre>FIPS State: Verified
FIPS State Detail: FIPS 140-2 security validation checks succeeded, FIPS mode enabled.</pre>

<code><nowiki>osn_fipscheck validate-fips</nowiki></code> confirms that FIPS mode is enabled and the module passed its self-tests, returning 0 on success and 1 or greater on failure. A healthy start logs <code><nowiki>Start up self-tests completed successfully</nowiki></code> and the DRBG in use (<code><nowiki>CTR-DRBG</nowiki></code>) in <code><nowiki>qs_crypto.log</nowiki></code>. In the web interface, the '''FIPS''' and '''FIPS State Detail''' fields appear in the storage system's Properties panel.

To confirm which OpenSSL the appliance actually uses, don't run <code><nowiki>openssl version</nowiki></code>, which reports the operating system's copy. Read the active library instead:

<pre>strings /opt/osnexus/common/lib/active/libcrypto.so.3 | grep -oE 'OpenSSL 3\.[0-9]+\.[0-9]+' | sort -u</pre>

Routine operations on an encrypted pool work the same as on a plain one: adding devices, replacing a faulted drive, and adding write log, cache or spare devices to a passphrase-protected pool. After a reboot, a passphrase-protected pool stays stopped until an administrator starts it with the passphrase from the web interface or the CLI. To change a passphrase, remove it and apply a new one.

When a pool is retired, delete it using a data shredding option: the 4-pass DoD 5220.22-M (section 8-306) procedure, the 4-pass NNSA Policy Letter NAP-14.1-C procedure or the US Army AR380-19 method. Shredding runs at the block level, concurrently across all disks in the pool, and the encryption keys can be shredded too.

== FAQ ==

=== Is QuantaStor FIPS 140 validated? ===

The OSNEXUS Crypto Library, which QuantaStor uses in FIPS mode, holds FIPS 140-2 Level 1 validation under NIST CMVP certificate 4185. QuantaStor 6 and 7 haven't been through FIPS 140-3 certification. Where a 140-3 certificate is required for data at rest, use FIPS 140-3 validated self-encrypting drives, where the drive vendor holds the certificate.

=== Is it suitable for US government and defense deployments? ===

It provides the controls these environments usually review: AES-256 encryption at rest, a FIPS mode that uses a validated crypto library, support for FIPS SED media, encryption on the wire with SMB3, IPsec and TLS 1.2, role-based access control, password policy that supports HIPAA and CJIS requirements, and DoD 5220.22-M, NNSA and AR380-19 data shredding. Whether a given deployment passes accreditation depends on the agency's specific control set, so map these controls to it.

=== Can I encrypt an existing pool? ===

No. Software encryption encrypts the underlying devices, so it has to be selected when the pool is created. To encrypt existing data, create a new encrypted pool and move the data onto it.

=== What happens if I lose the boot drives? ===

If you exported the pool keys, reinstall on new boot media, import the keys and start the pool. Without the key backup, a software-encrypted pool can't be recovered, so export the keys as soon as you create the pool.

=== Does encryption work with high availability? ===

Yes. Encrypted pools fail over between controllers like unencrypted pools, provided they have no passphrase. A passphrase would stop the pool from starting automatically on the surviving controller.

----
''Part of the [[Guides:Index|QuantaStor Guides]] series. For reference documentation, see the [[Main Page|QuantaStor documentation]].''

[[Category:Guides]]

File:Guide-fips-encryption-fips-encryption-pool-encryption-ribbon.png

2026-10-07T05:45:04Z

Qadmin: osn-seo-utilities: screenshot for Guides (fips-encryption)

osn-seo-utilities: screenshot for Guides (fips-encryption)

File:Guide-fips-encryption-fips-encryption-create-pool-software.png

2026-10-07T05:45:04Z

Qadmin: osn-seo-utilities: screenshot for Guides (fips-encryption)

osn-seo-utilities: screenshot for Guides (fips-encryption)

Guides:Index

2026-10-07T05:00:08Z

Qadmin: osn-seo-utilities: regenerate index

Network Ports

2026-10-07T05:00:03Z

Qadmin: osn-seo-utilities: quantastor-network-port-usage @ 5caca23311c7 (approved in the portal)

[[Category:admin_guide]]

Network ports (also called target ports or network interfaces) are the Ethernet interfaces through which a QuantaStor system is managed and through which it serves storage. This page covers how to address a port, how to combine ports into a bond, how to layer VLAN and virtual interfaces on top of a port, how to control which protocols each port carries, how to add static routes, and which TCP and UDP ports and external sites QuantaStor needs so you can lock the network down without breaking it.

Client hosts reach [[Storage Volumes]] over iSCSI and NVMe-oF/TCP through a network port, authorized users reach [[Network Shares]] over NFS and SMB through a network port, and object storage buckets are reached over S3 through a network port. Every one of those paths depends on the port configuration described here, so it is worth getting right before you provision storage.

{| class="wikitable"
! Section !! Purpose
|-
| [[#Port types and naming|Port types and naming]] || How to tell a physical port, a bond, a VLAN, a VIF and a cluster VIF apart by name.
|-
| [[#Where network ports are managed|Where network ports are managed]] || The tabs, toolbar and right-click menu that hold every port operation.
|-
| [[#Static vs DHCP assigned IP addresses|Static vs DHCP assigned IP addresses]] || Why static addressing is required in practice, and the one-time conversion on a fresh install.
|-
| [[#Modifying network port settings|Modifying network port settings]] || Every field on the General Settings tab of Modify Network Port.
|-
| [[#Firewall settings per port|Firewall settings per port]] || Allowing and blocking individual protocols on one port.
|-
| [[#NIC bonding and trunking|NIC bonding and trunking]] || Combining ports for throughput and fault tolerance, and which bonding mode to pick.
|-
| [[#VLAN interfaces|VLAN interfaces]] || Tagged virtual ports on a physical or bonded port.
|-
| [[#Virtual interfaces|Virtual interfaces]] || Extra IP addresses on one port.
|-
| [[#High-availability and cluster virtual interfaces|High-availability and cluster virtual interfaces]] || Floating IPs that move with a pool, a site or the grid.
|-
| [[#Renaming a network port|Renaming a network port]] || Giving a port a stable, meaningful name.
|-
| [[#Online, offline, restart and rescan|Online, offline, restart and rescan]] || Bringing a port up or down and picking up out-of-band changes.
|-
| [[#Static route management|Static route management]] || Routing specific networks through a gateway other than the default.
|-
| [[#Choosing which ports carry which traffic|Choosing which ports carry which traffic]] || Separating management traffic from storage traffic.
|-
| [[#TCP and UDP ports QuantaStor uses|TCP and UDP ports QuantaStor uses]] || Every port the appliance listens on, why, and which ones the Firewall tab can block.
|-
| [[#External sites QuantaStor connects to|External sites QuantaStor connects to]] || Outbound connections for licensing, upgrades, support logs, alerting and replication.
|-
| [[#Locking down the network|Locking down the network]] || What you can safely block, and what has to stay open.
|-
| [[#Verifying connectivity|Verifying connectivity]] || The Network Check report.
|-
| [[#Command line reference|Command line reference]] || The <code>qs</code> commands that cover everything on this page.
|}

== Port types and naming ==

QuantaStor names every port after the interface the kernel presents, and uses two delimiters to indicate that a port is layered on top of another one. Reading the name tells you what a port is:

{| class="wikitable"
! Name !! Type !! Notes
|-
| <code>ens192</code>, <code>eno1</code>, <code>enp3s0f0</code> || Physical port || The predictable name assigned by the kernel. Rename it if you want something more meaningful -- see [[#Renaming a network port|Renaming a network port]].
|-
| <code>bond0</code> || Bonded port || Two or more physical ports combined. The member ports become slaves and stop carrying their own IP address.
|-
| <code>ens192.56</code> || VLAN interface || A period separates the parent port name from the VLAN tag. A VLAN on a bond looks like <code>bond0.56</code>.
|-
| <code>ens192:1</code> || Virtual interface (VIF) || A colon followed by an index, allocated from 1 upwards. An extra IP address on the parent port.
|-
| <code>ens192:ha345680</code> || HA pool virtual interface || Floats with a [[Storage Pools|storage pool]] when the pool fails over.
|-
| <code>ens192:sv...</code> || Site cluster virtual interface || Floats between the appliances of a site cluster.
|-
| <code>ens192:gm</code> || Grid management virtual interface || The single floating IP a storage grid is managed through. iSCSI and NVMe-oF are permanently disabled on it, because a management IP that moves on failover is not a safe place to export block storage from.
|-
| <code>ens192 (S)</code> || Manually configured secondary port || An address that QuantaStor discovered but does not own. Management operations are refused on these.
|}

Interface names are limited to 15 characters by the kernel, and the tag QuantaStor appends when it creates a cluster virtual interface has to fit inside that budget. A long parent port name is therefore not merely untidy -- it will eventually block cluster VIF creation, so keep renamed ports short.

== Where network ports are managed ==

[[File:nwport_ports_tab.png|thumb|right|800px|The Network Ports tab lists every port on every system in the grid, grouped by storage system. The Network Port toolbar group above it holds the create, delete and state operations.]]

{{Navigation|Storage Management → Storage Systems → ''select a Storage System'' → Network Ports ''(tab)''}}

Select the '''Storage Systems''' section in the tree on the left, then the '''Network Ports''' tab in the centre of the screen. The tab lists every port on every system in the grid, grouped by storage system, with its state, link state, link speed, IP address, subnet mask, whether it is an iSCSI or NVMe-oF portal, and its vendor, model, MAC address, byte counters and MTU. The '''Network Static Routes''' tab beside it lists the static routes on those same ports.

The '''Network Port''' toolbar group at the top of the screen carries Modify, Create Virtual Port, Create VLAN Port, Create Bonded Port, the matching Delete operations, Online, Offline and Rescan All.

[[File:nwport_context_menu.png|thumb|right|237px|Right-clicking a port in the Network Ports tab reaches every port operation, including the three that are not on the toolbar: Rename, Restart, and Create/Delete Static Route.]]

Three operations are reachable only by right-clicking a port, not from the toolbar: '''Rename Network Port...''', '''Restart Network Port...''' and '''Create Static Route...''' / '''Delete Static Route...'''. Right-click a port either in the tree on the left or in the Network Ports tab.

For grid-wide settings such as the DNS servers and search domain that apply to all ports, use '''Modify Grid Network Settings''' instead -- the '''Modify Grid''' button in the '''Storage System Grid''' toolbar group. It writes one DNS search suffix, DNS server list and NTP server list to every system you tick, overwriting whatever those systems had. The per-system equivalents are the [[Storage System#DNS Servers|DNS Servers]] and [[Storage System#NTP Servers|NTP Servers]] tabs of Storage System Modify.

== Static vs DHCP assigned IP addresses ==

QuantaStor supports both statically assigned IP addresses and dynamically assigned '''D'''ynamic '''H'''ost '''C'''onfiguration '''P'''rotocol ('''DHCP''') addresses. After installation the first port is typically configured by DHCP and the rest are disabled, so that the appliance is reachable for initial setup.

Reconfigure every port with a static IP address before you provision storage. A DHCP lease can change, and a storage system whose address moves takes its iSCSI targets, NFS exports and SMB shares with it. The appliance also has to be able to boot and serve storage when the DHCP server is unreachable. The only defensible use of DHCP is a reservation that always hands the same address to the same MAC, and even then a static address is simpler to reason about.

There is a second, one-time reason to set a static management address first. A freshly installed system still has the installer's netplan configuration active. The first time you modify any network port, QuantaStor converts that configuration into {{Code|1=/etc/network/interfaces}}, backs up the previous file as {{Code|1=/etc/network/interfaces-netplan-convert-<timestamp>.backup}}, renames each {{Code|1=/etc/netplan/*.yaml}} file to <code>.bak</code>, and disables the netplan renderer. All interfaces are torn down and brought back during that conversion, so a port still on DHCP can come back with a different address and the management session drops. QuantaStor guards against this: while any port is in that state it is flagged with a warning whose detail begins "Netplan", and the Modify Network Port dialog refuses any change other than to <code>static</code>, with the message ''"Be sure to configure a static management network port before modifying network ports."''

Set a static address on your management port first, then work through the remaining DHCP ports until they all have static IPs assigned. '''Reboot afterwards and confirm the configuration, which is what the accompanying alert asks for.''' This is an important step after an initial QuantaStor OS installation as this allows the system to cleanup network configuration issues left behind by the text mode installer.

== Modifying network port settings ==

[[File:nwport_modify_general.png|thumb|right|550px|The General Settings tab of Modify Network Port. The Static IP Configuration Settings fieldset is only enabled when Config Type is set to static.]]

{{Navigation|Storage Management → Storage Systems → Network Port ''(select + right-click)'' → Modify Network Port...}}

The dialog is also on the '''Modify''' button in the Network Port toolbar group. It has two tabs, '''General Settings''' and '''Firewall'''.

=== General Settings ===

{| class="wikitable"
! Field !! Description
|-
| '''Storage System''' || The system whose port you are configuring. Changing it reloads the Network Port list.
|-
| '''Network Port''' || The port to modify. Every port on the selected system is offered, including bonds, VLANs and virtual interfaces.
|-
| '''Config Type''' || <code>static</code>, <code>dhcp</code> or <code>disabled</code>. Only <code>static</code> enables the Static IP Configuration Settings fieldset below. Choosing <code>disabled</code> prompts for confirmation and then removes the port's stanza from {{Code|1=/etc/network/interfaces}} entirely, so the port stops carrying traffic and does not come back after a reboot. A port that is a bond slave reports as <code>disabled</code> and cannot be changed here -- delete the bond first. Cluster virtual interfaces are locked to <code>static</code> because the cluster software owns their addresses.
|-
| '''Description''' || An optional note recorded against the port. Use it to record what the port is for; there is no other place to do that.
|-
| '''iSCSI Portal''' || Makes this port's IP address an allowed iSCSI portal. Clearing it removes the address from the allowed-portal list of every iSCSI target on the system, so initiators can no longer log in through it. Existing sessions are not torn down -- the change takes effect on the next login. If no port at all is left iSCSI-enabled, QuantaStor restricts iSCSI to loopback rather than leaving it open.
|-
| '''NVMeoF Portal''' || Makes this port's IP address an NVMe-oF/TCP listener on port 4420. Unlike the iSCSI setting, clearing it removes the listener, which does drop connections through that address.
|-
| '''Lossless Networking''' || Configures RoCEv2 with priority flow control on the port. Only selectable on adapters that support it -- in practice Mellanox/NVIDIA ConnectX cards with RDMA enabled -- and greyed out everywhere else. It sets DSCP-based priority trust, enables PFC on priority 3 only, turns global pause off, sizes the receive buffers for the adapter model, and records the port in {{Code|1=/var/opt/osnexus/quantastor/conf/qs_losslessNetworking.conf}}. QuantaStor verifies the result and reverts the change if verification fails. Enable it only where the switch fabric is configured for the same priority flow control scheme; a one-sided configuration performs worse than plain Ethernet.
|-
| '''Optimize hardware RX/TX buffer settings for throughput''' || Enlarges the adapter's receive and transmit ring buffers. QuantaStor records the current sizes so it can put them back, then raises both rings to the largest power-of-two multiple of 128 that still fits under the adapter's hardware maximum -- roughly half the maximum, which improves large-block and replication throughput without asking the driver for a size it will reject. Settings are recorded in {{Code|1=/var/opt/osnexus/quantastor/conf/qs_porttunings.conf}} and re-applied every five minutes. Available on physical ports only: a bond, VLAN or virtual interface has no hardware ring of its own, so enable it on the member ports instead.
|}

The '''Static IP Configuration Settings''' fieldset holds the addressing:

{| class="wikitable"
! Field !! Description
|-
| '''IP Address''' || The static address for the port. Each port needs a unique address; QuantaStor rejects an address already recorded on another port. If the address is part of a Ceph cluster configuration you have to disable the port before the address can be changed.
|-
| '''Subnet Mask''' || The mask for the port's network, such as <code>255.255.0.0</code>.
|-
| '''Gateway''' || Optional. QuantaStor checks that the gateway is on the same subnet as the port address and refuses it otherwise. Configure a gateway on one port only. QuantaStor gives every static port the same route metric, so a second gateway-bearing port ends up competing for the default route rather than adding a fallback.
|-
| '''MTU''' and '''Jumbo Frames''' || The maximum transmission unit, 1492 to 64000. The '''Jumbo Frames''' button toggles between 9000 and 1500 and relabels itself '''Default Frames''' once jumbo is set. Set 9000 on all 10GbE and faster interfaces. If you move away from 1500, the switch ports and the client hosts on that network have to match -- a mismatched MTU produces intermittent stalls rather than a clean failure. The field is disabled for VLAN and virtual interfaces, which inherit the parent port's MTU; changing the parent cascades the new MTU to all of its children. Setting a VLAN MTU above its parent's is capped to the parent's value with an alert.
|}

Changing the IP address of a port that has static routes configured is flagged: the routes are bound to the old address and client connections through them can drop. Remove the routes first.

Click '''Apply''' to commit the change and keep the dialog open, or '''OK''' to commit and close.

== Firewall settings per port ==

[[File:nwport_modify_firewall.png|thumb|right|550px|The Firewall tab sets each protocol to Allow, Block or Inherit for this port only.]]

{{Navigation|Storage Management → Storage Systems → Network Port ''(select + right-click)'' → Modify Network Port... → Firewall ''(tab)''}}

The '''Firewall''' tab lists each service QuantaStor knows how to filter, with its description and a '''Status''' setting. The three settings are what makes this useful, and the distinction is easy to miss:

* '''Inherit''' -- follow the system-wide firewall setting. This is the default for every service on every port.
* '''Block''' -- drop traffic for this service arriving at this port's IP address, even though the system-wide setting allows it.
* '''Allow''' -- accept traffic for this service at this port's IP address, even though the system-wide setting blocks it.

So the system level sets the policy and the port level carries the exceptions. The usual pattern is to block everything you do not use at the system level -- see [[Firewall Management]] -- and then Allow the handful of protocols each port is actually meant to serve.

Two things are worth knowing before you rely on it. The rules match on the port's '''destination IP address''', not on the interface, so a per-port rule really follows the address; that is deliberate, and it is why the masks are also recorded against HA and site virtual interfaces so the rule follows a floating address to whichever system holds it. And because QuantaStor reports a service as blocked whenever any DROP rule exists for it, blocking a protocol on one port raises an alert saying that service is blocked system-wide, even though only that one address is affected.

The service list, the TCP and UDP ports behind each entry, and the bit each one occupies come from {{Code|1=/opt/osnexus/quantastor/conf/qs_services_firewall.conf}} on the appliance. The rules themselves are iptables rules in QuantaStor's own chains (<code>QUANTASTOR-INPUT</code>, <code>QUANTASTOR-FORWARD</code>, <code>QUANTASTOR-OUTPUT</code>), written out to {{Code|1=/var/opt/osnexus/quantastor/iptables.qsfirewallrules}} so they survive a restart, and kept out of the shared chains so the firewall can be rebuilt without disturbing other rules on the system. QuantaStor re-applies them on change and again every 20 minutes.

Changing a setting that makes a service unreachable prompts for confirmation first. For the TCP and UDP ports behind each entry, and the ports the Firewall tab does not cover, see [[#TCP and UDP ports QuantaStor uses|TCP and UDP ports QuantaStor uses]] below.

The same firewall settings can be driven from the CLI with <code>[[QuantaStor CLI Command Reference#network-port-modify|qs network-port-modify]] --protocol-block=</code> and <code>--protocol-allow=</code>, each taking a comma-separated protocol list, with <code>--protocol-block-reset</code> and <code>--protocol-allow-reset</code> to clear a list. Prefix a protocol with a tilde (<code>~</code>) to remove it from a list rather than add it.

== NIC bonding and trunking ==

[[File:nwport_bond_create.png|thumb|right|550px|Create Bonded Port. At least two ports must be selected in the grid; clear Hide Configured Ports to bond a port that already has an IP address.]]

{{Navigation|Storage Management → Storage Systems → Network Port → Create Bonded Port ''(toolbar)''}}

Bonding, also called trunking or teaming, combines several physical ports into one logical port to improve throughput and to survive the loss of a cable, a switch port or a NIC. The bond takes the IP address; the member ports become slaves and no longer carry addresses of their own.

For every mode except LACP, all of the member ports must be connected to the same switch. LACP works across switches -- which is what makes it the only mode that survives a whole switch going away -- but it requires the switch ports to be configured for LACP as well. QuantaStor surfaces switch-side mistakes on the port itself, reporting an invalid partner MAC address or a partner that is not advertising LACP support, so check the port state after creating an LACP bond rather than assuming it came up.

The dialog fields are the same addressing fields as Modify Network Port, plus:

{| class="wikitable"
! Field !! Description
|-
| '''Bond Mode''' || The bonding policy. See the table below.
|-
| '''Select the Network Ports to be Bonded''' || Select at least two ports. The grid lists each candidate port's name, IP address, vendor and model.
|-
| '''Hide Configured Ports''' || Selected by default, hiding ports that already have an IP address so you do not accidentally bond your management port. Clear it to bond a configured port; QuantaStor warns that the port's existing IP configuration will be removed and that the bond will only have the address you entered.
|-
| '''Force''' || Proceeds without the confirmation prompt when a selected port already has an IP configuration.
|}

The modes QuantaStor offers, with the switch requirement each one carries:

{| class="wikitable"
! Mode !! Kernel mode !! Switch !! Behaviour
|-
| Link Aggr Ctrl Protocol (LACP layer2) || <code>802.3ad</code>, hash <code>layer2</code> || Managed, LACP configured || Balances by source and destination MAC. Spans switches. Recommended default.
|-
| Link Aggr Ctrl Protocol (LACP layer2+3) || <code>802.3ad</code>, hash <code>layer2+3</code> || Managed, LACP configured || Adds the IP addresses to the hash, which spreads traffic better when many clients sit behind one router MAC.
|-
| Link Aggr Ctrl Protocol (LACP layer3+4) || <code>802.3ad</code>, hash <code>layer3+4</code> || Managed, LACP configured || Adds the TCP/UDP ports to the hash, so a single client with many connections can use more than one member link.
|-
| Round Robin (balance-rr) || <code>balance-rr</code> || Etherchannel, managed || Sends packets in sequence across the members. Highest single-stream throughput, but it reorders packets and does not span switches.
|-
| Adaptive Transmit Load Balancing (balance-tlb) || <code>balance-tlb</code> || Unmanaged || Balances outbound traffic only; inbound arrives on one member. Needs no switch configuration.
|-
| Adaptive Load Balancing (balance-alb) || <code>balance-alb</code> || Unmanaged || As balance-tlb, plus inbound balancing via ARP negotiation. Needs no switch configuration.
|-
| Balance XOR (balance-xor) || <code>balance-xor</code> || Etherchannel, managed || Picks a member by hashing the MAC addresses. Static, so both ends must agree.
|-
| Active-Backup (active-backup) || <code>active-backup</code> || Unmanaged || One member carries all traffic; the others stand by. No throughput gain, and the simplest mode to get right.
|}

For fault tolerance that survives a switch outage, LACP is the mode to use. Where the switches are unmanaged and cannot be configured, use Active-Backup or one of the adaptive load balancing modes.

The '''Bond Mode''' combo in the Create Bonded Port dialog is pre-selected from the storage system's own default bonding policy, which QuantaStor sets on first startup and records in {{Code|1=/etc/modprobe.d/bonding.conf}}. On the 6.9.0 system this page was verified against that default was LACP layer 2. Check yours with <code>[[QuantaStor CLI Command Reference#system-get|qs system-get]]</code> and look at the '''Bond Mode''' field, and change it with <code>[[QuantaStor CLI Command Reference#system-modify|qs system-modify]] --bond-mode=<mode></code>. QuantaStor sets <code>miimon=100</code> on every bond so link failures are detected within a tenth of a second, and allows a maximum of four bonds per system.

To change the mode of a bond that already exists, open '''Modify Network Port''' on the bond and use the '''Bond Port Advanced Settings...''' button, which appears only when the selected port is a bond. The mode cannot be changed while the bond is carrying a virtual interface or an HA virtual interface -- remove those first. If a member port drops out of a bond, QuantaStor re-enslaves it automatically, at most once an hour per member.

Delete a bond with '''Delete Bonded Port''' in the toolbar, or <code>[[QuantaStor CLI Command Reference#bonded-interface-delete|qs bond-delete]] --port=<bond></code>. The member ports return to being unconfigured physical ports.

== VLAN interfaces ==

[[File:nwport_vlan_create.png|thumb|right|396px|Create VLAN Interface. The MTU is read-only because a VLAN inherits it from the parent port.]]

{{Navigation|Storage Management → Storage Systems → Network Port → Create VLAN Port ''(toolbar)''}}

A '''VLAN''' ('''V'''irtual '''LAN''') interface is a virtual port that tags its traffic with a VLAN ID, letting one physical or bonded port serve several isolated networks. VLAN interfaces can be created and deleted at any time.

{| class="wikitable"
! Field !! Description
|-
| '''Storage System''' || The system to create the VLAN interface on.
|-
| '''Parent Port''' || The physical or bonded port to tag on. A VLAN cannot be created on another VLAN, nor on a virtual interface, so those are filtered out of the list.
|-
| '''IP Address''', '''Subnet Mask''', '''Gateway''' || The addressing for the VLAN interface, on the network that VLAN reaches.
|-
| '''Description''' || An optional note. Recording which network the tag corresponds to pays for itself.
|-
| '''MTU''' || Read-only. A VLAN inherits the parent port's MTU; change it on the parent.
|-
| '''VLAN ID''' || The 802.1Q tag, 1 to 4094 (4095 is reserved). Defaults to 5.
|-
| '''VLAN QoS''' || The 802.1p Class of Service user priority to record for the interface, 0 (lowest, the default) to 7 (highest).
|}

The new port is named after its parent with a period and the tag, so VLAN 56 on <code>ens192</code> becomes <code>ens192.56</code> and VLAN 56 on <code>bond0</code> becomes <code>bond0.56</code>. Delete a VLAN interface with '''Delete VLAN Port''' in the toolbar or <code>[[QuantaStor CLI Command Reference#vlan-interface-delete|qs vlan-delete]] --port=<port></code>. A port that has a VLAN child cannot be disabled or renamed until the VLAN is removed.

== Virtual interfaces ==

[[File:nwport_vif_create.png|thumb|right|500px|Create Virtual Interface. A VIF adds a second address to an existing port.]]

{{Navigation|Storage Management → Storage Systems → Network Port → Create Virtual Port ''(toolbar)''}}

A virtual interface (VIF) gives a physical or bonded port an additional IP address, so one port can be present on several networks or carry several addresses on the same network. VIFs can be created and deleted at any time, and cannot be layered on top of another VIF.

The fields are '''Storage System''', '''Parent Port''', '''IP Address''', '''Description''', '''Subnet Mask''', '''Gateway''', and a read-only '''MTU''' inherited from the parent port. QuantaStor names the new interface after the parent with a colon and the next free index, so the first VIF on <code>ens192</code> is <code>ens192:1</code>.

Two constraints are not obvious from the dialog. A VIF cannot be created on a bond running in Active-Backup mode. And DHCP is not valid on a virtual interface -- a VIF is always statically addressed.

Delete a VIF with '''Delete Virtual Port''' in the toolbar or <code>[[QuantaStor CLI Command Reference#virtual-interface-delete|qs vif-delete]] --port=<port></code>.

== High-availability and cluster virtual interfaces ==

QuantaStor also has virtual interfaces that float between systems. You will see these with <code>:ha</code>, <code>:sv</code> or <code>:gm</code> in the name, such as <code>ens192:ha345680</code>.

* '''HA pool virtual interfaces''' (<code>:ha</code>) are associated with a specific [[Storage Pools|storage pool]] and move with the pool when it fails over to another system. A pool can have several; each additional one increments the last digit of the port name, so they run <code>:ha345680</code>, <code>:ha345681</code>, <code>:ha345682</code> and so on. Create them by right-clicking an HA failover group in the Storage Pools section and choosing '''Add Cluster VIF...''', or with <code>[[QuantaStor CLI Command Reference#ha-interface-create|qs hai-create]] --ha-group=<group> --parent-port=<port> --ip-address=<ip></code>. A cluster heartbeat must be configured between the systems that manage the pool before an HA VIF can be created; see [[Setup Guide for Clustered HA Storage Pools]] and [[Storage Pool HA Failover Interface Create]].
* '''Site cluster virtual interfaces''' (<code>:sv</code>) float between the appliances of a site so that a service address stays available. See [[High-availability VIF Management]].
* The '''grid management virtual interface''' (<code>:gm</code>) is the single floating address a storage grid is managed through. There is exactly one per site cluster, and iSCSI and NVMe-oF are permanently disabled on it.

These are all managed from the '''[[High-availability VIF Management]]''' main tab, and by the cluster software rather than by the network port configuration -- that page covers use cases, failover, deliberate moves and location constraints. That is why the Modify Network Port dialog locks their Config Type to <code>static</code>, disables the addressing fields, and refuses IP, MTU and bond mode changes -- and why static routes cannot be created on them. Change a floating address through the HA or site VIF dialogs, not through Modify Network Port.

QuantaStor also enables ARP filtering automatically once a bonded or virtual port exists, because several addresses on the same network answering for each other's ARP requests otherwise sends traffic to the wrong interface.

== Renaming a network port ==

[[File:nwport_rename.png|thumb|right|550px|Rename Network Port. The Port Information fieldset identifies the port you are about to rename.]]

{{Navigation|Storage Management → Storage Systems → Network Port ''(select + right-click)'' → Rename Network Port...}}

Predictable kernel names such as <code>enp3s0f1</code> say where the adapter is but nothing about what it does. Renaming a port gives it a name that survives hardware changes and reads clearly in the port list.

Enter the new name in '''New Name'''; the '''Port Information''' fieldset below shows the selected port's UUID, IP address, MAC address and state so you can confirm you have the right one. QuantaStor writes a systemd link file at {{Code|1=/etc/systemd/network/10-<newname>.link}} that matches the port by MAC address, renames the running interface, and rewrites the port's stanza in {{Code|1=/etc/network/interfaces}}.

The dialog's own guidance is worth following: prefix your management port with <code>mgmt</code>, and avoid the <code>eth</code> prefix, which collides with the names kernel drivers hand out. Interfaces are conventionally numbered from zero, as in <code>eno0</code>, <code>eno1</code>. Keep the name under 15 characters -- see [[#Port types and naming|Port types and naming]]. The name is lowercased for you.

A port cannot be renamed while it is a bond slave, while it has VLAN or virtual interface children, or while the system is using legacy <code>ethN</code> naming mode. Remove the bond or the children first, or switch back to predictable naming in [[Storage System Modify|Modify Storage System]]. A reboot may be needed for the rename to take effect everywhere, and QuantaStor raises an alert saying so.

The CLI equivalent is <code>[[QuantaStor CLI Command Reference#network-port-rename|qs np-rename]] --port=<port> --name=<newname></code>.

== Online, offline, restart and rescan ==

Four state operations sit alongside the configuration dialogs:

{| class="wikitable"
! Operation !! Where !! What it does
|-
| '''Online''' || Network Port toolbar; right-click → Online Network Port... || Brings the port up. The CLI equivalent is <code>[[QuantaStor CLI Command Reference#network-port-enable|qs np-enable]] --port=<port></code>.
|-
| '''Offline''' || Network Port toolbar; right-click → Offline Network Port... || Takes the port down without discarding its configuration, so Online brings it back as it was. Static routes on the port go down with it and come back when it is brought back up. The CLI equivalent is <code>[[QuantaStor CLI Command Reference#network-port-disable|qs np-disable]] --port=<port></code>.
|-
| '''Restart Network Port...''' || Right-click only || Cycles the port so a changed setting takes effect. The CLI equivalent is <code>[[QuantaStor CLI Command Reference#network-port-restart|qs np-restart]] --port=<port></code>.
|-
| '''Rescan All''' || Network Port toolbar; right-click → Rescan All Network Ports... || Discovers newly added ports and picks up configuration made outside QuantaStor. The CLI equivalent is <code>[[QuantaStor CLI Command Reference#network-port-rescan|qs np-rescan]]</code>.
|}

Offlining a port that has virtual interfaces attached takes the virtual interfaces down with it. Setting Config Type to <code>disabled</code> in Modify Network Port is not the same thing as taking the port offline: offline is temporary, whereas <code>disabled</code> removes the configuration and does not survive a reboot.

Ports QuantaStor discovers but that you do not want cluttering the list can be hidden with <code>[[QuantaStor CLI Command Reference#network-port-hide|qs np-hide]] --port=<port> --hide=true</code>.

QuantaStor also repairs drift on its own. Discovery re-applies the recorded MTU, subnet mask and gateway when the live configuration diverges from the configuration file, at most once every six hours per correction, and marks the port with a warning telling you to use the modify operation rather than editing the system configuration directly.

== Static route management ==

A static route customizes traffic flow so that a specific network is reached through a gateway other than the default. This is how you reach a network that the default gateway cannot route to, whether that is across a VLAN, through a VPN tunnel, or over a dedicated replication link. Each route has two parts: the gateway on the local network through which the remote network is reachable, and the address and mask of the remote network itself.

QuantaStor's static routes are per port, which gives granular control over which interface carries traffic for which network.

=== Creating a static route ===

[[File:nwport_static_route_create.png|thumb|right|500px|Create Static Route. The gateway must be on the port's own network; the destination must be given in bind address form.]]

{{Navigation|Storage Management → Storage Systems → Network Port ''(select + right-click)'' → Create Static Route...}}

The dialog is also reachable by right-clicking inside the '''Network Static Routes''' tab.

{| class="wikitable"
! Field !! Description
|-
| '''Storage System''' || The system to add the route to.
|-
| '''Parent Port''' || The port the route is bound to. Floating cluster interfaces are filtered out -- routes cannot be created on them.
|-
| '''Gateway IP Address''' || The gateway on the port's own network through which the destination network is reached.
|-
| '''Destination IP Address''' || The network to route to, in bind address form. QuantaStor checks the address against the mask and rejects anything that is not a network address, telling you the correct form to use.
|-
| '''Destination Subnet Mask''' || The mask of the destination network.
|}

For example, on a system whose <code>ens192</code> port is <code>10.0.8.70/24</code> and which has a gateway at <code>10.0.8.1</code> that can reach the <code>192.168.0.0/16</code> network, enter a '''Gateway IP Address''' of <code>10.0.8.1</code>, a '''Destination IP Address''' of <code>192.168.0.0</code> and a '''Destination Subnet Mask''' of <code>255.255.0.0</code>. All traffic from that port destined for <code>192.168.0.0/16</code> is then sent via <code>10.0.8.1</code>. Note that the gateway has to be on the port's own network -- QuantaStor applies the route immediately and refuses a gateway the port cannot reach.

Routes are activated the moment they are created, and QuantaStor refuses a gateway it cannot reach rather than recording a route that will never work. A duplicate destination network and mask on the same system is also refused.

[[File:nwport_static_routes_tab.png|thumb|right|800px|The Network Static Routes tab lists each route with its parent port, gateway, and destination network and mask.]]

Route management is integrated with the platform rather than bolted on. Each route is written to an executable script at {{Code|1=/etc/network/<port>.staticroutes}}, and QuantaStor installs hooks in {{Code|1=/etc/network/if-up.d}} and {{Code|1=/etc/network/if-down.d}} that run it. The routes therefore enable and disable correctly when a port is brought up or down with <code>ifup</code> and <code>ifdown</code> from an SSH session, not just from the web interface. QuantaStor treats those scripts as the source of truth and imports routes it finds in them, so a route added by hand is picked up rather than overwritten. We recommend using the <code>qs</code> CLI or the web management interface all the same, so the routes are recorded as objects you can see in the Network Static Routes tab.

The CLI equivalent is <code>[[QuantaStor CLI Command Reference#static-route-create|qs sr-create]] --port=<port> --destination-ip-address=<network> --destination-netmask=<mask> --gateway-ip-address=<gateway></code>:

<pre style="font-size: smaller">
qs sr-create --port=ens192 --destination-ip-address=192.168.0.0 \
--destination-netmask=255.255.0.0 --gateway-ip-address=10.0.8.1
qs sr-list
</pre>

=== Deleting a static route ===

{{Navigation|Storage Management → Storage Systems → Network Static Routes ''(tab)'' → ''select a route'' → Delete Static Route... ''(select + right-click)''}}

Select the '''Network Static Routes''' tab in the centre of the screen, right-click the route, and choose '''Delete Static Route...'''. Alternatively, expand the port in the '''Storage Systems''' section on the left, then right-click the route.

Routes can be created and deleted at any time. Deleting one withdraws it immediately. If a port is disabled or taken offline, the routes associated with it are disabled with it.

The CLI equivalent is <code>[[QuantaStor CLI Command Reference#static-route-delete|qs sr-delete]] --route-id=<uuid></code>, where the UUID comes from <code>[[QuantaStor CLI Command Reference#static-route-list|qs sr-list]]</code>.

== Choosing which ports carry which traffic ==

QuantaStor works with all major 10/25/40/56/100GbE network interface cards (NICs) from Intel, Mellanox/NVIDIA, Broadcom, Cisco, Dell, HPE and Supermicro. Most systems have a mix of speeds, and the configuration that matters is which port carries which kind of traffic.

Clear '''iSCSI Portal''' and '''NVMeoF Portal''' on the slower 1GbE ports. That stops initiators from discovering and logging in through them, and reserves those ports for management traffic. A 1GbE port that answers an iSCSI discovery is a port some client will eventually run its I/O over.

Put the 1GbE management ports on a separate network from the ports serving NAS and SAN traffic. If management and storage share a network, the system can choose the 1GbE path for storage traffic and the resulting performance problem is hard to see from either end.

Configure the gateway on the management port only. Use the '''Firewall''' tab to block the protocols each port is not meant to serve, and the table in [[#TCP and UDP ports QuantaStor uses|TCP and UDP ports QuantaStor uses]] to see which TCP and UDP ports each service uses.

== TCP and UDP ports QuantaStor uses ==

Most organizations close every port a product does not need, to keep the attack surface small. This section lists every port a QuantaStor appliance listens on, what it is for, and which entry on the per-port [[#Firewall settings per port|Firewall]] tab controls it, so you can tell what can be closed and what has to stay open for the system to run normally. A dash in the Firewall tab column means the per-port firewall has no entry for that port; control it on your network firewall instead.

QuantaStor does not drop traffic by default. Its own iptables rules accept the ports it needs and drop only what you block on the Firewall tab, so on an appliance you have not locked down, every service below is reachable on every port that has an address.

=== Management and grid ===

{| class="wikitable"
! Port !! Protocol !! Service !! Firewall tab entry !! Notes
|-
| 443 || TCP || Web management interface (HTTPS) || -- || The address you browse to. Keep it open from the networks your administrators work from.
|-
| 80 || TCP || Web management interface (HTTP) || QS Web Management || Redirects to HTTPS on 443 by default. Close it if your browsers always use HTTPS.
|-
| 8080 || TCP || Web management interface (HTTPS, alternate port) || QS Web Management || Serves the same interface over HTTPS on a second port. Close it unless something depends on it.
|-
| 8153 || TCP || [[QuantaStor REST API Reference Guide|REST API]] (HTTPS) || QS REST API || Needed by scripts, automation, plugins and integrations that call the REST API. Close it if nothing does.
|-
| 5151 || TCP || QuantaStor core service (TLS) || -- || Every appliance in a [[Grid Configuration|storage grid]] talks to the others on this port, so it must be open between all grid members in both directions. The <code>qs</code> CLI also uses it when pointed at a remote system.
|-
| 22 || TCP || SSH || -- || Console access for administrators and OSNEXUS support, and the transport for encrypted remote replication (see below). Restrict it to administrator networks and to the other appliances you replicate with.
|}

The core service also listens on TCP 5152, bound to the loopback address only. It needs no rule on your network firewall.

=== Storage protocols ===

{| class="wikitable"
! Port !! Protocol !! Service !! Firewall tab entry !! Notes
|-
| 860, 3260 || TCP || iSCSI target || iSCSI Target || Initiator access to [[Storage Volumes]]. Clearing '''iSCSI Portal''' on a port also stops initiators logging in through it.
|-
| 4420 || TCP || NVMe-oF/TCP target || NVMeoF Target || Host access to [[Storage Volumes]] over [[NVMe-oF Target Configuration|NVMe-oF]]. Clearing '''NVMeoF Portal''' removes the listener on that address.
|-
| 2049, 111 || TCP and UDP || NFSv3 and NFSv4 || NFS || Client access to [[Network Shares]] over NFS. Port 111 is the RPC portmapper.
|-
| 2249 || TCP || NFS Ganesha || NFS Ganesha || The alternate NFS port used for scale-out (Ceph) file shares.
|-
| 137-139, 389, 445, 901 || TCP || SMB || SMB || Client access to [[Network Shares]] over SMB.
|}

=== Scale-out, clustering and replication ===

{| class="wikitable"
! Port !! Protocol !! Service !! Firewall tab entry !! Notes
|-
| 3300, 6789 || TCP || Ceph monitors || Ceph (6789 only) || Between the members of a [[Scale-out Block Setup (ceph)|Ceph cluster]], and from Ceph clients. The Ceph entry on the Firewall tab does not include 3300.
|-
| 6800-7100 || TCP || Ceph OSD and manager daemons || Ceph || Between the members of a Ceph cluster.
|-
| 7480, 7481 || TCP || Ceph object gateway (S3) || Ceph (7480 only) || S3 access to [[Ceph Object Storage|object storage]] buckets. A second gateway instance listens on 7481.
|-
| 5303-5332, 5403-5412 || TCP and UDP || Cluster heartbeat (Corosync) || -- || Between the appliances of an [[Setup Guide for Clustered HA Storage Pools|HA cluster]] or site cluster. QuantaStor also accepts multicast, IGMP and ICMP for the heartbeat. Blocking any of these splits the cluster.
|-
| 22 || TCP || Remote replication, encrypted || -- || The default. The source appliance sends the replication stream over SSH to the target appliance.
|-
| 20000-24999 || TCP || Remote replication, unencrypted || -- || Only for a replication link whose '''Encryption''' box is cleared. The target opens a receiver on a random port in this range for each transfer, and the source still uses SSH on 22 to start it. Open this range from source to target if you use unencrypted links.
|}

For [[Remote-replication (DR)|remote replication]] both appliances must be in the same grid, so TCP 5151 between them is needed as well as the replication transport.

=== Monitoring ===

{| class="wikitable"
! Port !! Protocol !! Service !! Firewall tab entry !! Notes
|-
| 3000 || TCP || [[Grafana Integration|Grafana]] || Grafana || Dashboards.
|-
| 9090, 9093, 9100, 9283 || TCP || [[Grafana Ceph Dashboard & Prometheus Integration|Prometheus]] || Prometheus || Prometheus server, Alertmanager, node exporter and the Ceph manager exporter.
|-
| 8888 || TCP || [[Chronograf Integration|Chronograf]] || Chronograf || Real-time statistics dashboard.
|-
| 161 || UDP || SNMP agent || -- || Only when the [[SNMP Agent Setup|SNMP agent]] is in use.
|}

The ports behind each Firewall tab entry are read from {{Code|1=/opt/osnexus/quantastor/conf/qs_services_firewall.conf}} on the appliance, and the static accept rules from the QuantaStor iptables service. Check those files on your own system if the table above and your appliance ever disagree.

== External sites QuantaStor connects to ==

These are the outbound connections an appliance makes. If your firewall restricts outbound traffic, allow the ones for the features you use. Every one of them is optional for serving storage: an appliance with no outbound access at all still serves its pools, shares and volumes, but cannot be activated online, upgraded from the public repositories or send logs to support.

{| class="wikitable"
! Purpose !! Destination !! Port !! Notes
|-
| Online license activation || <code>licensing.osnexus.com</code> || TCP 443 (HTTPS) || Used when you activate a license online and when the system reports license status. Without it, activate offline instead -- see [[Storage System Activate License Offline]] and [[License Management]].
|-
| Upgrades (QuantaStor packages) || <code>packages.osnexus.com</code> || TCP 80 (HTTP) || The QuantaStor package repository the [[Upgrade Manager]] installs from.
|-
| Upgrades and [[Security Updates|security updates]] (operating system) || <code>archive.ubuntu.com</code>, <code>us.archive.ubuntu.com</code>, <code>security.ubuntu.com</code> || TCP 80 (HTTP) || The Ubuntu package mirrors for the base operating system. Where appliances have no Internet access, use a [[QuantaStor Local Update Mirror|local update mirror]] instead.
|-
| [[Send Support Log Files|Sending support logs]] || <code>jira.osnexus.com</code> || TCP 21 (FTP, passive) || The first upload attempt. The log bundle is encrypted before it leaves the system.
|-
| Sending support logs (fallback) || <code>services.osnexus.com</code> || TCP 443 (HTTPS) || Used when FTP is unreachable. When the system has an HTTP proxy configured, QuantaStor skips FTP and uploads over HTTPS through the proxy.
|-
| Sending support logs (collection list) || <code>qstor-downloads.s3.amazonaws.com</code> || TCP 80 (HTTP) || Fetches the latest list of what to collect before building the bundle.
|-
| Email alerts || Your SMTP server || The port you configure || Set in the [[Storage System Alert Manager|Alert Manager]]. See [[Call-home / Alerting]].
|-
| Alert integrations || The service's own endpoint, for example [[Slack Integration|Slack]], [[PagerDuty Integration|PagerDuty]] or [[OpsGenie Integration|OpsGenie]] || TCP 443 (HTTPS) || Only for the integrations you configure.
|-
| Time synchronization || Your NTP servers, or <code>ntp.ubuntu.com</code> when none are set || UDP 123 || Set NTP servers per system or for the whole grid -- see [[#Where network ports are managed|Where network ports are managed]].
|-
| Name resolution || Your DNS servers || UDP and TCP 53 || Every external site on this list is reached by name.
|-
| Encryption key server || Your KMIP key server || TCP 5696 by default || Only when a [[Create Key Server Profile|key server profile]] is configured.
|-
| Cloud backup and cloud containers || The cloud provider's endpoint, such as <code>s3.amazonaws.com</code> || TCP 443 (HTTPS) || Only for the [[Cloud Containers / NAS Gateway|cloud containers]] and [[Backup Policies|backup policies]] you configure.
|-
| Remote replication || The other appliance's address || TCP 22, TCP 5151, and TCP 20000-24999 for unencrypted links || See [[#Scale-out, clustering and replication|Scale-out, clustering and replication]] above.
|}

== Locking down the network ==

Use two layers. The per-port [[#Firewall settings per port|Firewall]] tab and [[Firewall Management]] block services by destination address on the appliance itself; your network firewall handles everything those settings have no entry for, and all outbound traffic.

* '''Keep open between grid members:''' TCP 5151. Add TCP 22 between systems that replicate to each other, the Corosync ranges between HA and site cluster members, and the Ceph ports between scale-out cluster members.
* '''Keep open from administrator networks:''' TCP 443 for the web interface, TCP 22 if you use SSH, and TCP 8153 if anything calls the REST API.
* '''Keep open from clients:''' only the storage protocols they use -- iSCSI, NVMe-oF, NFS, SMB or S3 -- and only on the ports that serve them.
* '''Safe to close if unused:''' TCP 80 and 8080, the monitoring ports, NFS Ganesha, and every storage protocol you do not serve.

Block per port rather than system-wide where you can. A storage port that only serves iSCSI should block NFS, SMB and the web interface; a management port should block the storage protocols. After locking down, run the [[#Verifying connectivity|Network Check]] and send a test alert to confirm nothing you rely on was cut off.

== Verifying connectivity ==

{{Navigation|Storage Management → Storage Systems → ''select a Storage System'' → Health Checker ''(toolbar)''}}

The Network Check report tests reachability from each of the system's ports, including whether jumbo frames get through end to end, and reports the round-trip time for each destination. Run it after changing an MTU, since an MTU mismatch is exactly the kind of fault that leaves a port looking healthy while large transfers stall. See [[Network Check View]].

The CLI equivalent is <code>[[QuantaStor CLI Command Reference#network-check|qs net-check]]</code>:

<pre style="font-size: smaller">
qs net-check
</pre>

== Command line reference ==

Every operation on this page has a CLI equivalent. Add <code>--verbose</code> to any command to see its full argument list.

{| class="wikitable"
! Command !! Purpose
|-
| <code>[[QuantaStor CLI Command Reference#network-port-list|qs np-list]]</code> / <code>[[QuantaStor CLI Command Reference#network-port-get|qs np-get]]</code> || List all ports, or show one in detail.
|-
| <code>[[QuantaStor CLI Command Reference#network-port-modify|qs np-modify]]</code> || Change addressing, MTU, portal flags, autotuning, lossless networking and the per-port firewall.
|-
| <code>[[QuantaStor CLI Command Reference#network-port-enable|qs np-enable]]</code> / <code>[[QuantaStor CLI Command Reference#network-port-disable|qs np-disable]]</code> || Bring a port online or offline.
|-
| <code>[[QuantaStor CLI Command Reference#network-port-restart|qs np-restart]]</code> / <code>[[QuantaStor CLI Command Reference#network-port-rescan|qs np-rescan]]</code> || Restart one port, or rediscover all ports.
|-
| <code>[[QuantaStor CLI Command Reference#network-port-rename|qs np-rename]]</code> / <code>[[QuantaStor CLI Command Reference#network-port-hide|qs np-hide]]</code> || Rename a port, or hide it from the interface.
|-
| <code>[[QuantaStor CLI Command Reference#bonded-interface-create|qs bond-create]]</code> / <code>[[QuantaStor CLI Command Reference#bonded-interface-delete|qs bond-delete]]</code> || Create and delete bonded ports.
|-
| <code>[[QuantaStor CLI Command Reference#vlan-interface-create|qs vlan-create]]</code> / <code>[[QuantaStor CLI Command Reference#vlan-interface-delete|qs vlan-delete]]</code> || Create and delete VLAN interfaces.
|-
| <code>[[QuantaStor CLI Command Reference#virtual-interface-create|qs vif-create]]</code> / <code>[[QuantaStor CLI Command Reference#virtual-interface-delete|qs vif-delete]]</code> || Create and delete virtual interfaces.
|-
| <code>[[QuantaStor CLI Command Reference#static-route-create|qs sr-create]]</code> / <code>[[QuantaStor CLI Command Reference#static-route-delete|qs sr-delete]]</code> / <code>[[QuantaStor CLI Command Reference#static-route-list|qs sr-list]]</code> / <code>[[QuantaStor CLI Command Reference#static-route-get|qs sr-get]]</code> || Manage static routes.
|-
| <code>[[QuantaStor CLI Command Reference#ha-interface-create|qs hai-create]]</code> / <code>[[QuantaStor CLI Command Reference#site-vif-create|qs svr-create]]</code> || Create HA pool and site cluster virtual interfaces.
|-
| <code>[[QuantaStor CLI Command Reference#network-check|qs net-check]]</code> || Run the network reachability and jumbo frame report.
|}

Note that the <code>--port-type</code> argument of <code>[[QuantaStor CLI Command Reference#network-port-modify|qs np-modify]]</code> documents only <code>static</code> and <code>dhcp</code>. To remove a port's configuration entirely, use the '''Config Type''' setting of <code>disabled</code> in the web interface.

== Related pages ==

* [[Storage System#DNS Servers|Storage System]] -- the DNS, NTP and network settings on the Storage System Modify dialog
* [[Network Port Usage]] -- the older stand-alone list of TCP and UDP ports
* [[Firewall Management]] -- system-level firewall settings, which the per-port settings inherit from
* [[Storage System]] -- the storage system object these ports belong to
* [[Storage Pools]] -- pools whose HA failover groups own the floating virtual interfaces
* [[Network Shares]] -- NAS shares served over the ports configured here
* [[Storage Volumes]] -- SAN volumes served over the iSCSI and NVMe-oF portals configured here
* [[Setup Guide for Clustered HA Storage Pools]] -- cluster heartbeat and HA virtual interface setup
* [[Remote-replication (DR)]] -- replication between appliances
* [[QuantaStor Local Update Mirror]] -- upgrading appliances that have no Internet access
* [[Network Check View]] -- the network reachability report
* [[QuantaStor CLI Command Reference]] -- full argument lists for every command named on this page

----
''Verified against QuantaStor 6.9.0.''

Clustered SCSI-3 Persistent Reservations

2026-10-06T16:02:05Z

Qadmin: Reword the Windows MPIO links

[[Category:admin_guide]]

Clustered SCSI-3 persistent reservations let the reservations a client cluster places on QuantaStor [[Storage Volumes]] survive an HA failover. Windows Server Failover Clustering -- including Hyper-V with Cluster Shared Volumes and SQL Server Failover Cluster Instances -- depends on them. This page explains why those clusters need the feature, how to turn it on for an HA pool, and what QuantaStor does when you do.

'''The feature is off by default. If you present HA pool volumes to a Windows failover cluster, turn it on in the HA group's Modify Group dialog before the cluster starts using the disks.'''

{| class="wikitable"
! Section !! Purpose
|-
| [[#Why Windows failover clusters need it|Why Windows failover clusters need it]] || What persistent reservations do for WSFC, and what goes wrong without this feature
|-
| [[#How it works|How it works]] || How reservations are shared between the appliances of an HA group
|-
| [[#Requirements|Requirements]] || Pool type, protocols, operating system, release and cluster prerequisites
|-
| [[#Enabling clustered reservations|Enabling clustered reservations]] || The HA group setting, in the web interface and from the CLI
|-
| [[#Setting up a Hyper-V or WSFC cluster|Setting up a Hyper-V or WSFC cluster]] || End-to-end walkthrough from HA group to Cluster Shared Volume
|-
| [[#When to leave it disabled|When to leave it disabled]] || What the feature costs, and why it is not on by default
|-
| [[#Disabling clustered reservations|Disabling clustered reservations]] || What turning it off does to a running cluster
|-
| [[#Allow Reboot / Lock Recovery|Allow Reboot / Lock Recovery]] || The companion setting in the same dialog
|-
| [[#Checking the state on an appliance|Checking the state on an appliance]] || Confirming every volume is sharing its reservations
|-
| [[#Troubleshooting|Troubleshooting]] || When the setting is on but reservations are not shared
|}

== Why Windows failover clusters need it ==

A Windows Server Failover Cluster arbitrates ownership of its shared disks with SCSI-3 persistent reservations (PR). Each cluster node registers a key with every clustered LUN, and the node that owns a disk holds a reservation on it. When a node stops responding, the surviving nodes remove its registration, which fences it off the disk so it cannot keep writing to storage the cluster has handed to someone else. Cluster Shared Volumes, where every node reads and writes the same LUN at once, rely on this to keep a failed node away from the volume. SQL Server Failover Cluster Instances use the same cluster disks and depend on it in the same way.

Because the whole cluster depends on reservations behaving correctly, Microsoft's '''Validate a Configuration''' wizard tests them. Its storage tests register, reserve, preempt and release keys against every candidate disk, and a disk that fails the SCSI-3 persistent reservation test cannot be used as a supported cluster disk.

A reservation is state held by the SCSI target, not data on the disk. On a QuantaStor HA pool the target that recorded the reservation is the appliance that currently owns the pool. Without clustered reservations, that record exists only on that appliance. When the pool fails over, the receiving appliance starts serving the same LUNs with an empty reservation table: every key the Windows nodes registered, and the reservation that says which node owns each disk, is gone. The pool came back, but from the cluster's point of view the disks no longer carry the registrations it placed on them, and it treats them as failed. Cluster disks go offline and Cluster Shared Volumes stop, which is exactly the outage an HA pool is meant to prevent.

With '''Clustered SCSI3-PR Reservations''' enabled, every appliance in the HA group holds the same reservation state for every volume in the pool, so the appliance that takes the pool over already knows every registration and reservation, and the Windows cluster sees no change.

This is a different thing from the persistent reservations QuantaStor places on the pool's own disks. Those are I/O fencing: QuantaStor arbitrating which appliance may import the pool, described in [[HA Cluster Setup (JBODs)#I/O fencing and SCSI-3 persistent reservations|I/O fencing and SCSI-3 persistent reservations]]. Clustered reservations concern the reservations that '''clients''' place on the storage volumes QuantaStor exports, and the two are independent.

== How it works ==

QuantaStor shares reservation state between the appliances of an HA group through the Linux distributed lock manager (DLM), which runs as a resource of the site cluster's Corosync and Pacemaker stack.

* Every exported storage volume in the pool gets its own DLM lockspace, named after the volume's SCSI device identifier. The reservation state lives in that lockspace rather than in a single appliance's memory.
* The appliance that owns the pool serves the volume and is a member of its lockspace.
* Every other appliance in the HA group -- the secondary, and the tertiary if the group has one -- holds a placeholder device with the same identity, which keeps it a member of the same lockspace. It carries no data; its job is to hold a copy of the reservation state.
* During a failover the appliances swap roles device by device, so at every moment at least one appliance is holding each volume's reservations. The appliance taking over joins with the state already in place.

Setting the HA group to Enabled is the whole of the configuration. QuantaStor installs the DLM, creates and starts the Pacemaker DLM resource, and switches the pool's volumes into clustered mode on every appliance in the group. Appliances that did not run the operation converge on their own shortly afterwards, and an appliance that was down or rebuilt catches up when it returns. Volumes created in the pool later pick the setting up when they are created.

== Requirements ==

* '''QuantaStor scale-up Storage Pool (ZFS) protected by an HA group.''' Clustered reservations are a property of the HA group, so a pool without one cannot use them. See [[HA Cluster Setup (JBODs)]] and [[HA Cluster Setup (external SAN)]].
* '''iSCSI or Fibre Channel.''' The feature applies to storage volumes exported over iSCSI and Fibre Channel. It does not apply to NVMe over Fabrics.
* '''QuantaStor 6.9 or newer''' The DLM and SCSI target driver changes the feature depends on are built for standard QuantaStor installs on 6.9 and newer and is not yet available on RHEL based configurations. Non-Ubuntu based QuantaStor configurations will refuse to put volumes into clustered mode rather than risk the target stack.
* '''Every appliance in the HA group at QuantaStor 6.9.0 or newer, and rebooted onto its current target driver.''' Enabling the setting is refused, naming the appliance, if any member of the group is on an older release, or if an appliance has had new target drivers installed but has not rebooted since. Complete the upgrade on every node, including the reboot, before turning the feature on.
* '''A healthy site cluster.''' The appliances must be members of a [[Site Cluster Setup|site cluster]] with Corosync and Pacemaker running. QuantaStor gives the Corosync cluster a name if it does not have one, because the DLM cannot start without it; this is done across the cluster under maintenance mode, and needs no action from you.

The operating system on the Windows side needs nothing beyond what a failover cluster already requires: multipath I/O configured for the iSCSI or Fibre Channel paths to the QuantaStor appliances, and the cluster nodes' initiators registered as hosts on QuantaStor. For the recommended MPIO configuration, which keeps cluster disks online and Cluster Shared Volumes running smoothly through a failover, see [[Windows MPIO for Failover Clusters and Hyper-V]].

== Enabling clustered reservations ==

{{Navigation|Storage Management → Storage Pools ''(section)'' → ''select the pool'' → Storage Pool HA Resource Group ''(toolbar group)'' → Modify Group}}

[[File:hagroup_modify_clusterpr.png|thumb|right|470px|The General tab of Modify Storage Pool High-Availability Group. Clustered SCSI3-PR Reservations shows Disabled, the default for a new HA group; Allow Reboot / Lock Recovery sits directly below it.]]

On the '''General''' tab of the '''Modify Storage Pool High-Availability Group''' dialog, set '''Clustered SCSI3-PR Reservations''' to '''Enabled''' and click '''OK'''.

The same field is on the Create Group dialog, so you can enable it when you first create the HA group. On both dialogs it offers '''Enabled''' and '''Disabled''', and a new HA group starts at '''Disabled'''.

The change applies to the running pool. No failover, service restart or reboot is needed: existing volumes are switched into clustered mode in place, and the other appliances in the group converge within a short time. The Modify Group task reports "Enabling SCSI3-PR distributed locking" while it works. On an appliance that has never had the DLM installed, the first enable includes the package install and takes longer.

If the DLM cannot be configured -- the site cluster is not running, for example -- the HA group is still created or modified, because it is useful without clustered reservations. QuantaStor records the reason in the service log -- and, on a create, raises a warning on the group -- and keeps retrying on its own, and the volumes take up clustered mode once the condition is cleared.

'''Enable it before the Windows cluster takes reservations on the volumes.''' The walkthrough below does it first for that reason.

=== From the command line ===

[[QuantaStor CLI Command Reference#ha-group-modify|qs ha-group-modify]] and [[QuantaStor CLI Command Reference#ha-group-create|qs ha-group-create]] take {{Code|1=--enable-cluster-pr}}:

<pre style="font-size: smaller">
qs ha-group-modify --ha-group=pool-1-ha-group --enable-cluster-pr=enabled
</pre>

It accepts {{Code|1=enabled}}, {{Code|1=disabled}} and {{Code|1=auto}}, and {{Code|1=auto}} is what you get when you leave the argument out. On a modify, {{Code|1=auto}} leaves the current setting unchanged, so a command that changes some other property of the group never switches clustered reservations off as a side effect. On a create, {{Code|1=auto}} resolves to disabled unless the pool's volumes are already in clustered mode. Pass {{Code|1=enabled}} explicitly when you want the feature.

== Setting up a Hyper-V or WSFC cluster ==

This is the order to follow when presenting QuantaStor volumes to a new Windows Server Failover Cluster, whether for Hyper-V Cluster Shared Volumes or for a SQL Server Failover Cluster Instance.

=== 1. Enable clustered reservations on the HA group ===

Set '''Clustered SCSI3-PR Reservations''' to '''Enabled''' on the pool's HA group, as described in [[#Enabling clustered reservations|Enabling clustered reservations]]. Do this first, so that every reservation the cluster ever takes is shared between the appliances from the start.

=== 2. Create the volumes ===

Create the volumes the cluster will use in the HA pool -- see [[Storage Volumes#Creating a Storage Volume|Creating a Storage Volume]]. A small volume for the cluster's disk witness, if you intend to use one, belongs in the same pool.

=== 3. Present the volumes to every cluster node ===

Add each Windows cluster node as a host with its iSCSI IQN or Fibre Channel WWPNs, put all of the nodes in one Host Group, and assign the volumes to that Host Group. [[Hosts and Host Groups#Host Groups|Host Groups]] covers why a group assignment is the right tool for a cluster; for Fibre Channel, [[Fibre Channel Target Port Management#Assigning a storage volume over FC|Assigning a storage volume over FC]] covers zoning and WWPNs.

For iSCSI, have each node log in to the pool's HA virtual interface rather than an appliance's own address, so its sessions follow the pool when it fails over. For Fibre Channel, configure multipath on each node; QuantaStor advertises the paths through the owning appliance as active and the others as standby. For the best results, set up MPIO with the recommended settings on every node as described in [[Windows MPIO for Failover Clusters and Hyper-V]]; a reboot of each node activates them.

=== 4. Bring the disks online on one node ===

On one cluster node, rescan storage, bring the new disks online, initialize them and create the volumes you intend to use, in the usual way for disks that will become cluster disks. The other nodes see the same LUNs through the host group and need no preparation of their own.

=== 5. Run cluster validation ===

Run the '''Validate a Configuration''' wizard in Failover Cluster Manager, or the {{Code|1=Test-Cluster}} PowerShell cmdlet, against all of the nodes and include the storage tests. The SCSI-3 persistent reservation test should pass on every QuantaStor disk.

If you want proof that reservations survive a failover before going into production, fail the pool over to the other appliance with '''Manual Failover''' (see [[HA Cluster Setup (JBODs)#Deliberate failover|Deliberate failover]]) and run the storage validation again, or confirm in Failover Cluster Manager that the cluster disks stayed online throughout.

=== 6. Add the disks to the cluster and to Cluster Shared Volumes ===

Add the validated disks to the cluster as cluster disks, then add the ones Hyper-V will use to Cluster Shared Volumes. For a SQL Server Failover Cluster Instance, select the cluster disks during SQL Server's failover cluster setup instead.

== When to leave it disabled ==

Leave the setting Disabled on HA groups whose clients never take SCSI-3 reservations -- VMware vSphere datastores, Linux hosts, file shares, and single Windows servers. The feature is off by default because it has a cost:

* '''One DLM lockspace for every exported volume''', held on every appliance in the HA group, including the ones not serving the pool. A pool whose clients never reserve a LUN gains nothing from that.
* '''A dependency on the cluster's lock manager.''' A volume in clustered mode relies on the DLM being able to form and recover its lockspace. QuantaStor gates every step on the DLM being verified healthy on the appliance, precisely because a lockspace that cannot be joined or recovered can stall the appliance's SCSI target. That is a risk worth taking for a Windows cluster that needs the feature, and not one to take by default.

== Disabling clustered reservations ==

Set '''Clustered SCSI3-PR Reservations''' back to '''Disabled''' in Modify Group. The dialog asks you to confirm, and means it:

<blockquote>''Disabling Clustered Persistent Reservations is disruptive to connected clustered clients such as Windows Server with Hyper-V. Reservations are no longer shared across the HA cluster, so any reservation a client is holding will be LOST the next time this pool fails over. Are you sure you want to do this?''</blockquote>

The pool's volumes are taken back out of clustered mode straight away but note that the Pacemaker DLM resource itself is left in place as HA groups can be recreated as part of maintenance operations.

Deleting the HA group does '''not''' turn the feature off. Removing an HA group is a routine step in HA pool maintenance, and tearing the DLM down with it would drop the reservations a running Windows cluster is holding, so the lockspaces and the devices holding them are deliberately left as they are. Removing the DLM configuration entirely is a support operation; contact OSNEXUS support if you need it.

== Allow Reboot / Lock Recovery ==

The General tab carries a second, related checkbox: '''Allow Reboot / Lock Recovery'''. It is off by default, and it applies to HA pools whether or not clustered reservations are enabled.

It exists for one situation. When an appliance has been fenced away from a pool -- its peer took the pool over and preempted its reservations on the pool's disks -- the pool on the fenced appliance goes SUSPENDED or FAILED, and ZFS cannot bring a suspended pool back without a reboot. Until that reboot the appliance serves nothing, and it cannot always be shut down cleanly either, because I/O already submitted to the suspended pool never completes. Left alone, the appliance sits in that state until an administrator restarts it by hand.

Ticking the checkbox allows the appliance to restart itself to recover. QuantaStor does not otherwise reboot appliances on its own, so the restart is hedged with three conditions, all of which must hold:

# at least one pool imported on the appliance is SUSPENDED or FAILED;
# every such pool's HA group has '''Allow Reboot / Lock Recovery''' ticked -- one group that has not opted in vetoes the restart;
# the appliance holds no other healthy pool, so an appliance that is still serving storage is never restarted.

When the conditions hold, QuantaStor raises an alert and starts a three-minute countdown task before restarting. Cancel the task to stop the restart; the veto lasts until the appliance next reboots. The conditions are checked again just before the restart, so failing a pool back to the appliance during the countdown also stops it. The appliance does not re-import the pool when it comes back -- the pool belongs to its peer by then -- which is why the restart cannot loop.

From the command line the equivalent is {{Code|1=--allow-lock-recovery-reboot=enabled}} on [[QuantaStor CLI Command Reference#ha-group-modify|qs ha-group-modify]] or [[QuantaStor CLI Command Reference#ha-group-create|qs ha-group-create]]; as with {{Code|1=--enable-cluster-pr}}, leaving it out leaves the setting unchanged.

== Checking the state on an appliance ==

Two read-only commands, run as root (with {{Code|1=sudo}}) from a shell on an appliance in the HA group, show whether reservations are being shared.

{{Code|1=qs-scstdlm check}} reports whether the appliance's distributed lock manager is ready, and if not, the first thing standing in the way:

<pre style="font-size: smaller">
sudo qs-scstdlm check
</pre>

A ready appliance prints lines beginning {{Code|1=READY:}}. A line beginning {{Code|1=NOTREADY:}} names the problem -- the DLM daemon not installed, the Pacemaker DLM resource not created, Corosync with no cluster name, and so on. On an HA group that has never had the feature enabled it reports the following, which is expected until you enable the setting and is not a fault:

<pre style="font-size: smaller">
NOTREADY: the Pacemaker 'dlm' resource has not been created in the cluster.
</pre>

{{Code|1=qs-scstdlm inspect}} lists every SCSI target device on the appliance with its clustered-mode flag, whether the appliance has joined that device's lockspace, and the persistent reservations the device currently holds:

<pre style="font-size: smaller">
sudo qs-scstdlm inspect
sudo qs-scstdlm inspect --no-prs
</pre>

Add {{Code|1=--no-prs}} to list the devices without their reservations, which keeps the output short on an appliance exporting many volumes.

In the {{Code|1=CLM}} column, {{Code|1=1}} means the device is in clustered mode. The {{Code|1=LOCKSPACE}} column should read {{Code|1=joined}} for every such device. {{Code|1=NOT-JOINED}} alongside {{Code|1=CLM}} 1 means that volume's reservations are not being shared and would not survive a failover. An appliance with no volumes in clustered mode prints {{Code|1=(no SCST devices with a cluster_mode attribute on this node)}}. On the appliance that does not own the pool, the devices are the placeholders described in [[#How it works|How it works]] and show no backing file. Run it on both appliances: after the Windows cluster has taken its reservations, both should list the same registrations for each volume.

== Troubleshooting ==

'''Enabling is refused with a version error.''' The message names the appliance holding it up. Either it is on a release older than 6.9.0, or it has had new target drivers installed and has not rebooted onto them. Finish upgrading that appliance, including the reboot, and try again.

'''The setting is Enabled but volumes are not in clustered mode.''' Run {{Code|1=qs-scstdlm check}} on each appliance. QuantaStor holds volumes out of clustered mode until the appliance is running a supported platform, is a member of a site cluster, has Corosync and Pacemaker running with a named cluster, and has a running DLM daemon, and it records which condition is missing in the service log. Once the condition clears, the volumes switch over on their own.

'''Validation fails the SCSI-3 persistent reservation test.''' Confirm the setting is Enabled on the pool's HA group, and that {{Code|1=qs-scstdlm inspect}} shows every volume in clustered mode and joined on both appliances. A disk presented from a pool that is not in an HA group, or through an appliance's own address rather than the HA virtual interface, is not covered by the feature.

'''Cluster disks went offline after a failover.''' Check whether the setting was Enabled at the time. Reservations taken while it was Disabled are not shared, and are lost when the pool moves. Enable the setting, then bring the cluster disks back online in Failover Cluster Manager.

== Related pages ==

* [[Storage Volumes]] -- creating volumes, assigning them to hosts, and the multi-host caution
* [[Hosts and Host Groups]] -- host records, initiators, and Host Groups for clusters
* [[Fibre Channel Target Port Management]] -- presenting volumes over Fibre Channel
* [[HA Cluster Setup (JBODs)]] -- HA groups, failover, and I/O fencing on the pool's own disks
* [[HA Cluster Setup (external SAN)]] -- HA pools on LUNs from a back-end array
* [[Site Cluster Setup]] -- the Corosync and Pacemaker cluster the DLM runs on
* [[Windows MPIO for Failover Clusters and Hyper-V]] -- recommended multipathing setup and tuning for Windows cluster disks, Cluster Shared Volumes and Hyper-V
* [[QuantaStor CLI Command Reference]] -- full argument lists for the commands above

----
''Verified against QuantaStor 6.9.0.''

Windows MPIO for Failover Clusters and Hyper-V

2026-10-06T16:02:04Z

Qadmin: Reword as the recommended configuration

[[Category:admin_guide]]

This page describes the recommended Microsoft Multipath I/O (MPIO) configuration for Windows Server hosts that use QuantaStor storage volumes as shared cluster disks, including Windows Server Failover Clustering, Cluster Shared Volumes, Hyper-V clusters and SQL Server Failover Cluster Instances. It applies to Fibre Channel and iSCSI access to a QuantaStor HA pool, and it is the client-side companion to [[Clustered SCSI-3 Persistent Reservations]].

'''Summary.''' Install MPIO, let the Microsoft DSM claim QuantaStor devices, and apply the recommended MPIO timer settings below on every cluster node. With this configuration, an HA failover or an appliance reboot is a brief, transparent pause for the cluster: cluster disks stay online, Cluster Shared Volumes keep running, and Hyper-V virtual machines continue without interruption. The configuration was validated on Windows Server 2022 Hyper-V clusters connected over Fibre Channel to a QuantaStor HA pair.

{| class="wikitable"
! Section !! Purpose
|-
| [[#How the recommended settings help|How the recommended settings help]] || How MPIO carries cluster disks through an HA failover
|-
| [[#Installing MPIO and claiming QuantaStor devices|Installing MPIO and claiming QuantaStor devices]] || The MPIO feature and the QuantaStor hardware ID
|-
| [[#Recommended MPIO settings|Recommended MPIO settings]] || The timer values, and how to apply and verify them
|-
| [[#Paths and load balancing|Paths and load balancing]] || What a correctly configured disk looks like
|-
| [[#After maintenance on a QuantaStor appliance|After maintenance on a QuantaStor appliance]] || Confirming all paths are in use before moving pools back
|-
| [[#Troubleshooting|Troubleshooting]] || Quick checks if a disk or path does not look as expected
|}

== How the recommended settings help ==

During an HA failover, the pool moves from one QuantaStor appliance to its partner. For a short interval, while the receiving appliance imports the pool, I/O is held rather than completed. With Fibre Channel and ALUA, QuantaStor keeps the standby paths through the partner appliance present throughout, so Windows sees the paths change state and continues on the new active paths, typically within well under a minute.

MPIO decides how long to hold I/O and keep a disk while its paths change. The Windows defaults are tuned for general-purpose storage and allow only a short window (20 seconds before an unreachable disk is released, 60 seconds per I/O). The recommended settings extend that window so it comfortably covers an HA failover, including a slower one, for example when a busy pool takes longer to export. The cluster then experiences the failover as a short pause, and Cluster Shared Volumes and virtual machines carry on as normal.

Together with [[Clustered SCSI-3 Persistent Reservations]], which keeps the cluster's disk reservations in place across the failover, these settings give Windows clusters a seamless experience on QuantaStor HA pools.

== Installing MPIO and claiming QuantaStor devices ==

Do this on every cluster node, before presenting QuantaStor volumes to it.

'''1. Install the Multipath I/O feature''' (a reboot completes the installation):

<pre style="font-size: smaller">
Install-WindowsFeature -Name Multipath-IO
</pre>

'''2. Add the QuantaStor hardware ID to the Microsoft DSM''' so that MPIO combines the paths to each QuantaStor volume into a single disk:

<pre style="font-size: smaller">
New-MSDSMSupportedHW -VendorId OSNEXUS -ProductId QUANTASTOR
Get-MSDSMSupportedHW | Where-Object VendorId -match OSNEXUS
</pre>

The PowerShell cmdlet pads the vendor and product IDs to their SCSI field widths for you. If you prefer the MPIO control panel or {{Code|1=mpclaim}}, use the string {{Code|1=OSNEXUS QUANTASTOR}} followed by exactly six trailing spaces, as described in [[Multipath IO Configuration#Configuring Microsoft MPIO|Configuring Microsoft MPIO]].

'''3. Reboot''' the node so MPIO claims the devices. Each QuantaStor volume then appears once in Disk Management and in {{Code|1=Get-Disk}}.

For iSCSI, connect one session per path in the iSCSI Initiator with '''Enable multi-path''' selected, using addresses on separate subnets; see [[ISCSI Initiator Setup]] and [[Multipath IO Configuration]]. For an HA pool, connect to the pool's HA virtual interfaces so the sessions follow the pool when it fails over.

== Recommended MPIO settings ==

{| class="wikitable"
! Setting !! Windows default !! Recommended !! What it does
|-
| PDORemovePeriod || 20 || '''240''' || Seconds MPIO keeps a disk available while its paths are changing. 240 seconds comfortably covers an HA failover.
|-
| DiskTimeoutValue || 60 || '''100''' || Seconds Windows allows for each I/O. 100 is the largest value {{Code|1=Set-MPIOSetting}} accepts.
|-
| PathVerificationState || Disabled || '''Enabled''' || MPIO checks every path periodically, so it picks up path changes and returning paths promptly.
|-
| PathVerificationPeriod || 30 || 30 || Seconds between path checks.
|-
| RetryCount || 3 || 3 || Times MPIO retries an I/O on a path.
|-
| RetryInterval || 1 || 1 || Seconds between those retries.
|}

Apply them in an elevated PowerShell session on each cluster node:

<pre style="font-size: smaller">
Set-MPIOSetting -NewPathVerificationState Enabled -NewPathVerificationPeriod 30 -NewPDORemovePeriod 240 -NewRetryCount 3 -NewRetryInterval 1 -NewDiskTimeout 100
</pre>

'''Reboot the node to activate the settings.''' In a running cluster, work through the nodes one at a time: drain the node (Pause, Drain Roles in Failover Cluster Manager, or {{Code|1=Suspend-ClusterNode -Drain}}), reboot it, resume it, and continue with the next node. The cluster stays available throughout.

Confirm the result with {{Code|1=Get-MPIOSetting}}, which shows the active values:

<pre style="font-size: smaller">
PS C:\> Get-MPIOSetting

PathVerificationState : Enabled
PathVerificationPeriod : 30
PDORemovePeriod : 240
RetryCount : 3
RetryInterval : 1
UseCustomPathRecoveryTime : Disabled
CustomPathRecoveryTime : 40
DiskTimeoutValue : 100
</pre>

{{Code|1=Get-MPIOSetting}} is the best place to check these values, because {{Code|1=Set-MPIOSetting}} stores them through WMI rather than in the MPIO registry key. {{Code|1=Set-MPIOSetting}} validates all of its parameters together, so if it reports an error (for example a {{Code|1=-NewDiskTimeout}} above 100), correct the value and run the command again.

The Fibre Channel HBA driver parameters can stay at the vendor defaults; this configuration was validated with the QLogic Windows driver at its defaults.

== Paths and load balancing ==

The Microsoft DSM's default load-balance policy works well with QuantaStor and needs no change. QuantaStor reports ALUA path states for HA pools: paths through the appliance that owns the pool are Active/Optimized and paths through the partner appliance are Standby. The Microsoft DSM's default for ALUA storage, Round Robin With Subset, spreads I/O across the Active/Optimized paths and moves to the other set when the pool fails over.

View a disk's paths with {{Code|1=mpclaim}}:

<pre style="font-size: smaller">
mpclaim -s -d # list MPIO disks
mpclaim -s -d 1 # paths and their states for MPIO disk 1
</pre>

A host with two HBA ports zoned to both appliances of an HA pair shows four paths per disk: two Active/Optimized and two Standby. After a failover the two sets swap roles, and the path count stays the same.

== After maintenance on a QuantaStor appliance ==

After an appliance has been rebooted or its Fibre Channel ports have been offline, a quick rescan on the Windows nodes makes sure all of its paths are back in use. Doing this before moving a pool back to that appliance ensures every node has its full set of paths ready for the move.

<ol>
<li>Wait until the appliance is fully up, with its Storage Pools showing a Normal state in the web interface. QuantaStor brings an appliance's Fibre Channel target ports online once its LUNs are mapped and its ALUA states are set.</li>
<li>On every cluster node, rescan storage:
<pre style="font-size: smaller">
Update-HostStorageCache
pnputil /scan-devices
</pre>
A {{Code|1=rescan}} in {{Code|1=diskpart}} does the same if you prefer it.</li>
<li>Confirm with {{Code|1=mpclaim -s -d <n>}} that every QuantaStor disk shows its full path count.</li>
<li>Move the pool back.</li>
</ol>

== Troubleshooting ==

'''A cluster disk did not stay online through a failover.''' Run {{Code|1=Get-MPIOSetting}} on every node and confirm the recommended values are active; they take effect after a reboot. Also confirm that [[Clustered SCSI-3 Persistent Reservations]] is Enabled on the pool's HA group. To bring the disk back, run {{Code|1=Update-HostStorageCache}} and a {{Code|1=diskpart}} rescan on each node, then bring the cluster disk online in Failover Cluster Manager ({{Code|1=Start-ClusterResource}}).

'''A disk shows fewer paths than expected, or Standby paths only.''' Rescan as described in [[#After maintenance on a QuantaStor appliance|After maintenance on a QuantaStor appliance]]. Also check that both of the host's HBA ports are zoned to both appliances, and that both of its WWPNs are listed under the host in QuantaStor.

'''A QuantaStor volume appears more than once in Disk Management.''' MPIO has not yet claimed the device. Check {{Code|1=Get-MSDSMSupportedHW}} for the OSNEXUS QUANTASTOR entry, add it if needed, and reboot.

'''Fibre Channel port mode on the appliances.''' Keep the appliances' Fibre Channel ports in their default target-only mode for HA pools serving Windows clusters. Dual initiator and target mode ({{Code|1=qs-util enabledualmode}}) is intended for diagnostics.

== Related pages ==

* [[Clustered SCSI-3 Persistent Reservations]] -- the HA group setting that keeps Windows cluster reservations in place across failovers
* [[Multipath IO Configuration]] -- MPIO and Linux multipath basics for QuantaStor volumes
* [[ISCSI Initiator Setup]] -- connecting iSCSI initiators
* [[Fibre Channel Target Port Management]] -- presenting volumes over Fibre Channel
* [[Hosts and Host Groups]] -- host records, initiators, and Host Groups for clusters
* [[HA Cluster Setup (JBODs)]] -- HA groups and failover

----

Multipath IO Configuration

2026-10-06T15:58:54Z

Qadmin: Point to the Windows MPIO page for failover clusters and Hyper-V

== Understanding MPIO with QuantaStor ==
QuantaStor supports round-robin style active/active multipathing using Microsoft's MPIO and via Linux using the device-mapper multipath (dmmp) driver. Active/active multipath support not only boosts performance, it also gives you the benefit of automatic path failover when a network connection is lost to one of your NICs on the host (initiator side) or the storage system (target side).

iSCSI Multipathing should be configured using IP's on different network subnets/vlans to ensure proper communication of the paths. Using the same subnet with multiple IP's will not provide a proper multipathing config. QuantaStor will present out iSCSI communication on any iSCSI enabled interface by default, there is no additional configuration required inside QuantaStor to enable path(s) for iSCSI Multipathing other than configuring the network interface with an appropriate IP address and subnet.

e.g. the below example networking configuration would provide two network paths between the QuantaStor and the Host connecting to the QuantaStor.

<pre>
QuantaStor server IP configuration

10.0.1.121/24
10.0.2.121/24

Host Client IP Configuration

10.0.1.125/24
10.0.2.125/24
</pre>

Once the Network configuration is correct, the appropriate Multipathing configuration will need to be performed on the Host Client side, the rest of this section details how to configure Multipathing for the most popular client operating systems.

For Microsoft MPIO configuration you'll need to add the QuantaStor vendor / model information so that Microsoft's default DSM (device specific module) can detect QuantaStor volumes and automatically enable multipathing for them thereby combining multiple paths to your QuantaStor Storage System in a single device as seen in the Windows Disk Manager.

For Linux, you'll need to first install dmmp for your Linux distribution, and then modify the /etc/multipath.conf file to include an additional stanza so that the system properly identifies the QuantaStor devices and uses the correct multi-path device identification and path management rules.

<pre>
Note: QuantaStor supports SCSI-3 Persistent Reservations so you can use it with Microsoft's Failover Clustering / Hyper-V and other cluster solutions that require SCSI-2 or SCSI-3 reservations.
</pre>

=== Configuring Microsoft MPIO ===
''For Windows Server failover clusters, Cluster Shared Volumes and Hyper-V on QuantaStor HA pools, see [[Windows MPIO for Failover Clusters and Hyper-V]] for the recommended MPIO setup and timer tuning; the steps below cover only adding the QuantaStor device string.''

The first step in configuring MPIO on Windows Server is to navigate to MPIO properties. (Note that desktop versions of Windows such as XP, Vista, and Windows 7/8 do not support MPIO, you must be using Windows Server 2003/2008/2012) The easiest way to get to the MPIO configuration screen is to type MPIO in the ''Start'' menu search bar. It can also be found in the Control Panel under "MPIO".

[[File:mpio.png|frame|center|The string you will want to add for QuantaStor is "OSNEXUS QUANTASTOR      ". ]]

Once you're at the MPIO dialog, select the "MPIO Devices" tab and then click the "Add" button. The string you will need to add for QuantaStor is exactly:

<pre>
OSNEXUS QUANTASTOR
^ ^^^^^^
</pre>

Note that the '^' characters are provided here so that you can see where the spaces are. Note also that these are all capital letters, '''with one space between the words, and exactly six trailing spaces after QUANTASTOR.''' If you don't have exactly (6) trailing spaces then the MPIO driver won't recognize the QuantaStor devices and you'll end up with multiple instances of the same disk in Windows Disk Manager.
=== VMWare Special Note ===

In the case that VMWare is used, no changes to multipathing are required. VMWare works with iSCSI and FC multipathing by default.

=== Configuring Linux Device-Mapper Multipath (DMMP) ===

After you have device mapper installed on your Linux distribution, you'll need to login to your system as root and then add this section to your ''/etc/multipath.conf'' file for CentOS:

<pre>
device {
vendor "OSNEXUS"
product "QUANTASTOR"
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
path_grouping_policy multibus
failback immediate
path_selector "round-robin 0"
rr_weight uniform
rr_min_io 100
path_checker tur
}
</pre>

On Ubuntu/Debian the 'getuid_callout' part is a little different so use this version for Ubuntu:

<pre>
device {
vendor "OSNEXUS"
product "QUANTASTOR"
getuid_callout "/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/%n"
path_grouping_policy multibus
failback immediate
path_selector "round-robin 0"
rr_weight uniform
rr_min_io 100
path_checker tur
}
</pre>

In the case that a Fibre Channel card is in use, ALUA will need to be enabled. The multipath device entry will have these properties:

<pre>
device {
vendor "OSNEXUS"
product "QUANTASTOR"
getuid_callout "/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/%n"
prio alua
hardware_handler "1 alua"
path_grouping_policy group_by_prio
failback immediate
no_path_retry 10
rr_min_io 100
path_checker tur
rr_weight uniform
}

</pre>

Please Note: Starting with the 0.5 release of the linux multipath tools, an additional ''property "(ID_SCSI_VPD|ID_WWN|ID_SERIAL)"'' definition needs to be added to the definitions in the blacklist_exceptions portion of the multipath.conf file.

<pre>
blacklist_exceptions {
property "(ID_SCSI_VPD|ID_WWN|ID_SERIAL)"
device {
vendor "OSNEXUS"
product "QUANTASTOR"

}
}
</pre>

Once the file is updated you'll want to restart the multipath service like so:

<pre>
service multipathd restart
</pre>

Once you have it all configured and you're logging into the iSCSI target / storage volume with your iSCSI initiator you'll want to run 'multipath -l' to list the paths and you should see something like this:

<pre>
14f534e455855530038323061623737393365306365643738dm-2 OSNEXUS,QUANTASTOR
[size=512M][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
\_ 120:0:0:0 sdg 8:96 [active][undef]
\_ 119:0:0:0 sdf 8:80 [active][undef]
14f534e455855530038323061623737393735393231656332dm-1 OSNEXUS,QUANTASTOR
[size=4.0G][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
\_ 118:0:0:0 sde 8:64 [active][undef]
\_ 117:0:0:0 sdc 8:32 [active][undef]
</pre>

Remember, you'll need to make sure that you mount your filesystems using the correct unique device ID under ''/dev/mapper/'' so that the multipath driver is utilized. If you mount your filesystem from /dev/sdXX you'll have completely bypassed the multi-path driver and will not get the performance an fail-over capabilities.

=== RHEL 8.6 ===

Some users noticed that /sbin/scsi_id command did not exist in Redhat 8.6, but the/lib/udev/scsi_id command did, as in Ubuntu/Debian. They suspected that when they booted their system, it was trying to execute “scsi_id” to an FC attached volume that was not supposed to be there, which prevented it from booting.

As a result, they configured multipathing like the below for volumes that are FC attached to RHEL 8.6:

<pre>
device {
vendor "OSNEXUS"
product "QUANTASTOR"
getuid_callout "/lib/udev/scsi_id -g -u -s /block/%n"
prio alua
hardware_handler "1 alua"
path_grouping_policy group_by_prio
failback immediate
no_path_retry 10
rr_min_io 100
path_checker tur
rr_weight uniform

}
</pre>

Clustered SCSI-3 Persistent Reservations

2026-10-06T15:58:54Z

Qadmin: Link the recommended Windows MPIO setup and tuning page

[[Category:admin_guide]]

Clustered SCSI-3 persistent reservations let the reservations a client cluster places on QuantaStor [[Storage Volumes]] survive an HA failover. Windows Server Failover Clustering -- including Hyper-V with Cluster Shared Volumes and SQL Server Failover Cluster Instances -- depends on them. This page explains why those clusters need the feature, how to turn it on for an HA pool, and what QuantaStor does when you do.

'''The feature is off by default. If you present HA pool volumes to a Windows failover cluster, turn it on in the HA group's Modify Group dialog before the cluster starts using the disks.'''

{| class="wikitable"
! Section !! Purpose
|-
| [[#Why Windows failover clusters need it|Why Windows failover clusters need it]] || What persistent reservations do for WSFC, and what goes wrong without this feature
|-
| [[#How it works|How it works]] || How reservations are shared between the appliances of an HA group
|-
| [[#Requirements|Requirements]] || Pool type, protocols, operating system, release and cluster prerequisites
|-
| [[#Enabling clustered reservations|Enabling clustered reservations]] || The HA group setting, in the web interface and from the CLI
|-
| [[#Setting up a Hyper-V or WSFC cluster|Setting up a Hyper-V or WSFC cluster]] || End-to-end walkthrough from HA group to Cluster Shared Volume
|-
| [[#When to leave it disabled|When to leave it disabled]] || What the feature costs, and why it is not on by default
|-
| [[#Disabling clustered reservations|Disabling clustered reservations]] || What turning it off does to a running cluster
|-
| [[#Allow Reboot / Lock Recovery|Allow Reboot / Lock Recovery]] || The companion setting in the same dialog
|-
| [[#Checking the state on an appliance|Checking the state on an appliance]] || Confirming every volume is sharing its reservations
|-
| [[#Troubleshooting|Troubleshooting]] || When the setting is on but reservations are not shared
|}

== Why Windows failover clusters need it ==

A Windows Server Failover Cluster arbitrates ownership of its shared disks with SCSI-3 persistent reservations (PR). Each cluster node registers a key with every clustered LUN, and the node that owns a disk holds a reservation on it. When a node stops responding, the surviving nodes remove its registration, which fences it off the disk so it cannot keep writing to storage the cluster has handed to someone else. Cluster Shared Volumes, where every node reads and writes the same LUN at once, rely on this to keep a failed node away from the volume. SQL Server Failover Cluster Instances use the same cluster disks and depend on it in the same way.

Because the whole cluster depends on reservations behaving correctly, Microsoft's '''Validate a Configuration''' wizard tests them. Its storage tests register, reserve, preempt and release keys against every candidate disk, and a disk that fails the SCSI-3 persistent reservation test cannot be used as a supported cluster disk.

A reservation is state held by the SCSI target, not data on the disk. On a QuantaStor HA pool the target that recorded the reservation is the appliance that currently owns the pool. Without clustered reservations, that record exists only on that appliance. When the pool fails over, the receiving appliance starts serving the same LUNs with an empty reservation table: every key the Windows nodes registered, and the reservation that says which node owns each disk, is gone. The pool came back, but from the cluster's point of view the disks no longer carry the registrations it placed on them, and it treats them as failed. Cluster disks go offline and Cluster Shared Volumes stop, which is exactly the outage an HA pool is meant to prevent.

With '''Clustered SCSI3-PR Reservations''' enabled, every appliance in the HA group holds the same reservation state for every volume in the pool, so the appliance that takes the pool over already knows every registration and reservation, and the Windows cluster sees no change.

This is a different thing from the persistent reservations QuantaStor places on the pool's own disks. Those are I/O fencing: QuantaStor arbitrating which appliance may import the pool, described in [[HA Cluster Setup (JBODs)#I/O fencing and SCSI-3 persistent reservations|I/O fencing and SCSI-3 persistent reservations]]. Clustered reservations concern the reservations that '''clients''' place on the storage volumes QuantaStor exports, and the two are independent.

== How it works ==

QuantaStor shares reservation state between the appliances of an HA group through the Linux distributed lock manager (DLM), which runs as a resource of the site cluster's Corosync and Pacemaker stack.

* Every exported storage volume in the pool gets its own DLM lockspace, named after the volume's SCSI device identifier. The reservation state lives in that lockspace rather than in a single appliance's memory.
* The appliance that owns the pool serves the volume and is a member of its lockspace.
* Every other appliance in the HA group -- the secondary, and the tertiary if the group has one -- holds a placeholder device with the same identity, which keeps it a member of the same lockspace. It carries no data; its job is to hold a copy of the reservation state.
* During a failover the appliances swap roles device by device, so at every moment at least one appliance is holding each volume's reservations. The appliance taking over joins with the state already in place.

Setting the HA group to Enabled is the whole of the configuration. QuantaStor installs the DLM, creates and starts the Pacemaker DLM resource, and switches the pool's volumes into clustered mode on every appliance in the group. Appliances that did not run the operation converge on their own shortly afterwards, and an appliance that was down or rebuilt catches up when it returns. Volumes created in the pool later pick the setting up when they are created.

== Requirements ==

* '''QuantaStor scale-up Storage Pool (ZFS) protected by an HA group.''' Clustered reservations are a property of the HA group, so a pool without one cannot use them. See [[HA Cluster Setup (JBODs)]] and [[HA Cluster Setup (external SAN)]].
* '''iSCSI or Fibre Channel.''' The feature applies to storage volumes exported over iSCSI and Fibre Channel. It does not apply to NVMe over Fabrics.
* '''QuantaStor 6.9 or newer''' The DLM and SCSI target driver changes the feature depends on are built for standard QuantaStor installs on 6.9 and newer and is not yet available on RHEL based configurations. Non-Ubuntu based QuantaStor configurations will refuse to put volumes into clustered mode rather than risk the target stack.
* '''Every appliance in the HA group at QuantaStor 6.9.0 or newer, and rebooted onto its current target driver.''' Enabling the setting is refused, naming the appliance, if any member of the group is on an older release, or if an appliance has had new target drivers installed but has not rebooted since. Complete the upgrade on every node, including the reboot, before turning the feature on.
* '''A healthy site cluster.''' The appliances must be members of a [[Site Cluster Setup|site cluster]] with Corosync and Pacemaker running. QuantaStor gives the Corosync cluster a name if it does not have one, because the DLM cannot start without it; this is done across the cluster under maintenance mode, and needs no action from you.

The operating system on the Windows side needs nothing beyond what a failover cluster already requires: multipath I/O configured for the iSCSI or Fibre Channel paths to the QuantaStor appliances, and the cluster nodes' initiators registered as hosts on QuantaStor. For the recommended MPIO setup and timer tuning, so that cluster disks ride through a failover instead of being removed, see [[Windows MPIO for Failover Clusters and Hyper-V]].

== Enabling clustered reservations ==

{{Navigation|Storage Management → Storage Pools ''(section)'' → ''select the pool'' → Storage Pool HA Resource Group ''(toolbar group)'' → Modify Group}}

[[File:hagroup_modify_clusterpr.png|thumb|right|470px|The General tab of Modify Storage Pool High-Availability Group. Clustered SCSI3-PR Reservations shows Disabled, the default for a new HA group; Allow Reboot / Lock Recovery sits directly below it.]]

On the '''General''' tab of the '''Modify Storage Pool High-Availability Group''' dialog, set '''Clustered SCSI3-PR Reservations''' to '''Enabled''' and click '''OK'''.

The same field is on the Create Group dialog, so you can enable it when you first create the HA group. On both dialogs it offers '''Enabled''' and '''Disabled''', and a new HA group starts at '''Disabled'''.

The change applies to the running pool. No failover, service restart or reboot is needed: existing volumes are switched into clustered mode in place, and the other appliances in the group converge within a short time. The Modify Group task reports "Enabling SCSI3-PR distributed locking" while it works. On an appliance that has never had the DLM installed, the first enable includes the package install and takes longer.

If the DLM cannot be configured -- the site cluster is not running, for example -- the HA group is still created or modified, because it is useful without clustered reservations. QuantaStor records the reason in the service log -- and, on a create, raises a warning on the group -- and keeps retrying on its own, and the volumes take up clustered mode once the condition is cleared.

'''Enable it before the Windows cluster takes reservations on the volumes.''' The walkthrough below does it first for that reason.

=== From the command line ===

[[QuantaStor CLI Command Reference#ha-group-modify|qs ha-group-modify]] and [[QuantaStor CLI Command Reference#ha-group-create|qs ha-group-create]] take {{Code|1=--enable-cluster-pr}}:

<pre style="font-size: smaller">
qs ha-group-modify --ha-group=pool-1-ha-group --enable-cluster-pr=enabled
</pre>

It accepts {{Code|1=enabled}}, {{Code|1=disabled}} and {{Code|1=auto}}, and {{Code|1=auto}} is what you get when you leave the argument out. On a modify, {{Code|1=auto}} leaves the current setting unchanged, so a command that changes some other property of the group never switches clustered reservations off as a side effect. On a create, {{Code|1=auto}} resolves to disabled unless the pool's volumes are already in clustered mode. Pass {{Code|1=enabled}} explicitly when you want the feature.

== Setting up a Hyper-V or WSFC cluster ==

This is the order to follow when presenting QuantaStor volumes to a new Windows Server Failover Cluster, whether for Hyper-V Cluster Shared Volumes or for a SQL Server Failover Cluster Instance.

=== 1. Enable clustered reservations on the HA group ===

Set '''Clustered SCSI3-PR Reservations''' to '''Enabled''' on the pool's HA group, as described in [[#Enabling clustered reservations|Enabling clustered reservations]]. Do this first, so that every reservation the cluster ever takes is shared between the appliances from the start.

=== 2. Create the volumes ===

Create the volumes the cluster will use in the HA pool -- see [[Storage Volumes#Creating a Storage Volume|Creating a Storage Volume]]. A small volume for the cluster's disk witness, if you intend to use one, belongs in the same pool.

=== 3. Present the volumes to every cluster node ===

Add each Windows cluster node as a host with its iSCSI IQN or Fibre Channel WWPNs, put all of the nodes in one Host Group, and assign the volumes to that Host Group. [[Hosts and Host Groups#Host Groups|Host Groups]] covers why a group assignment is the right tool for a cluster; for Fibre Channel, [[Fibre Channel Target Port Management#Assigning a storage volume over FC|Assigning a storage volume over FC]] covers zoning and WWPNs.

For iSCSI, have each node log in to the pool's HA virtual interface rather than an appliance's own address, so its sessions follow the pool when it fails over. For Fibre Channel, configure multipath on each node; QuantaStor advertises the paths through the owning appliance as active and the others as standby. Before going further, set up MPIO and apply the recommended timer settings on every node as described in [[Windows MPIO for Failover Clusters and Hyper-V]]; the settings need a reboot of each node.

=== 4. Bring the disks online on one node ===

On one cluster node, rescan storage, bring the new disks online, initialize them and create the volumes you intend to use, in the usual way for disks that will become cluster disks. The other nodes see the same LUNs through the host group and need no preparation of their own.

=== 5. Run cluster validation ===

Run the '''Validate a Configuration''' wizard in Failover Cluster Manager, or the {{Code|1=Test-Cluster}} PowerShell cmdlet, against all of the nodes and include the storage tests. The SCSI-3 persistent reservation test should pass on every QuantaStor disk.

If you want proof that reservations survive a failover before going into production, fail the pool over to the other appliance with '''Manual Failover''' (see [[HA Cluster Setup (JBODs)#Deliberate failover|Deliberate failover]]) and run the storage validation again, or confirm in Failover Cluster Manager that the cluster disks stayed online throughout.

=== 6. Add the disks to the cluster and to Cluster Shared Volumes ===

Add the validated disks to the cluster as cluster disks, then add the ones Hyper-V will use to Cluster Shared Volumes. For a SQL Server Failover Cluster Instance, select the cluster disks during SQL Server's failover cluster setup instead.

== When to leave it disabled ==

Leave the setting Disabled on HA groups whose clients never take SCSI-3 reservations -- VMware vSphere datastores, Linux hosts, file shares, and single Windows servers. The feature is off by default because it has a cost:

* '''One DLM lockspace for every exported volume''', held on every appliance in the HA group, including the ones not serving the pool. A pool whose clients never reserve a LUN gains nothing from that.
* '''A dependency on the cluster's lock manager.''' A volume in clustered mode relies on the DLM being able to form and recover its lockspace. QuantaStor gates every step on the DLM being verified healthy on the appliance, precisely because a lockspace that cannot be joined or recovered can stall the appliance's SCSI target. That is a risk worth taking for a Windows cluster that needs the feature, and not one to take by default.

== Disabling clustered reservations ==

Set '''Clustered SCSI3-PR Reservations''' back to '''Disabled''' in Modify Group. The dialog asks you to confirm, and means it:

<blockquote>''Disabling Clustered Persistent Reservations is disruptive to connected clustered clients such as Windows Server with Hyper-V. Reservations are no longer shared across the HA cluster, so any reservation a client is holding will be LOST the next time this pool fails over. Are you sure you want to do this?''</blockquote>

The pool's volumes are taken back out of clustered mode straight away but note that the Pacemaker DLM resource itself is left in place as HA groups can be recreated as part of maintenance operations.

Deleting the HA group does '''not''' turn the feature off. Removing an HA group is a routine step in HA pool maintenance, and tearing the DLM down with it would drop the reservations a running Windows cluster is holding, so the lockspaces and the devices holding them are deliberately left as they are. Removing the DLM configuration entirely is a support operation; contact OSNEXUS support if you need it.

== Allow Reboot / Lock Recovery ==

The General tab carries a second, related checkbox: '''Allow Reboot / Lock Recovery'''. It is off by default, and it applies to HA pools whether or not clustered reservations are enabled.

It exists for one situation. When an appliance has been fenced away from a pool -- its peer took the pool over and preempted its reservations on the pool's disks -- the pool on the fenced appliance goes SUSPENDED or FAILED, and ZFS cannot bring a suspended pool back without a reboot. Until that reboot the appliance serves nothing, and it cannot always be shut down cleanly either, because I/O already submitted to the suspended pool never completes. Left alone, the appliance sits in that state until an administrator restarts it by hand.

Ticking the checkbox allows the appliance to restart itself to recover. QuantaStor does not otherwise reboot appliances on its own, so the restart is hedged with three conditions, all of which must hold:

# at least one pool imported on the appliance is SUSPENDED or FAILED;
# every such pool's HA group has '''Allow Reboot / Lock Recovery''' ticked -- one group that has not opted in vetoes the restart;
# the appliance holds no other healthy pool, so an appliance that is still serving storage is never restarted.

When the conditions hold, QuantaStor raises an alert and starts a three-minute countdown task before restarting. Cancel the task to stop the restart; the veto lasts until the appliance next reboots. The conditions are checked again just before the restart, so failing a pool back to the appliance during the countdown also stops it. The appliance does not re-import the pool when it comes back -- the pool belongs to its peer by then -- which is why the restart cannot loop.

From the command line the equivalent is {{Code|1=--allow-lock-recovery-reboot=enabled}} on [[QuantaStor CLI Command Reference#ha-group-modify|qs ha-group-modify]] or [[QuantaStor CLI Command Reference#ha-group-create|qs ha-group-create]]; as with {{Code|1=--enable-cluster-pr}}, leaving it out leaves the setting unchanged.

== Checking the state on an appliance ==

Two read-only commands, run as root (with {{Code|1=sudo}}) from a shell on an appliance in the HA group, show whether reservations are being shared.

{{Code|1=qs-scstdlm check}} reports whether the appliance's distributed lock manager is ready, and if not, the first thing standing in the way:

<pre style="font-size: smaller">
sudo qs-scstdlm check
</pre>

A ready appliance prints lines beginning {{Code|1=READY:}}. A line beginning {{Code|1=NOTREADY:}} names the problem -- the DLM daemon not installed, the Pacemaker DLM resource not created, Corosync with no cluster name, and so on. On an HA group that has never had the feature enabled it reports the following, which is expected until you enable the setting and is not a fault:

<pre style="font-size: smaller">
NOTREADY: the Pacemaker 'dlm' resource has not been created in the cluster.
</pre>

{{Code|1=qs-scstdlm inspect}} lists every SCSI target device on the appliance with its clustered-mode flag, whether the appliance has joined that device's lockspace, and the persistent reservations the device currently holds:

<pre style="font-size: smaller">
sudo qs-scstdlm inspect
sudo qs-scstdlm inspect --no-prs
</pre>

Add {{Code|1=--no-prs}} to list the devices without their reservations, which keeps the output short on an appliance exporting many volumes.

In the {{Code|1=CLM}} column, {{Code|1=1}} means the device is in clustered mode. The {{Code|1=LOCKSPACE}} column should read {{Code|1=joined}} for every such device. {{Code|1=NOT-JOINED}} alongside {{Code|1=CLM}} 1 means that volume's reservations are not being shared and would not survive a failover. An appliance with no volumes in clustered mode prints {{Code|1=(no SCST devices with a cluster_mode attribute on this node)}}. On the appliance that does not own the pool, the devices are the placeholders described in [[#How it works|How it works]] and show no backing file. Run it on both appliances: after the Windows cluster has taken its reservations, both should list the same registrations for each volume.

== Troubleshooting ==

'''Enabling is refused with a version error.''' The message names the appliance holding it up. Either it is on a release older than 6.9.0, or it has had new target drivers installed and has not rebooted onto them. Finish upgrading that appliance, including the reboot, and try again.

'''The setting is Enabled but volumes are not in clustered mode.''' Run {{Code|1=qs-scstdlm check}} on each appliance. QuantaStor holds volumes out of clustered mode until the appliance is running a supported platform, is a member of a site cluster, has Corosync and Pacemaker running with a named cluster, and has a running DLM daemon, and it records which condition is missing in the service log. Once the condition clears, the volumes switch over on their own.

'''Validation fails the SCSI-3 persistent reservation test.''' Confirm the setting is Enabled on the pool's HA group, and that {{Code|1=qs-scstdlm inspect}} shows every volume in clustered mode and joined on both appliances. A disk presented from a pool that is not in an HA group, or through an appliance's own address rather than the HA virtual interface, is not covered by the feature.

'''Cluster disks went offline after a failover.''' Check whether the setting was Enabled at the time. Reservations taken while it was Disabled are not shared, and are lost when the pool moves. Enable the setting, then bring the cluster disks back online in Failover Cluster Manager.

== Related pages ==

* [[Storage Volumes]] -- creating volumes, assigning them to hosts, and the multi-host caution
* [[Hosts and Host Groups]] -- host records, initiators, and Host Groups for clusters
* [[Fibre Channel Target Port Management]] -- presenting volumes over Fibre Channel
* [[HA Cluster Setup (JBODs)]] -- HA groups, failover, and I/O fencing on the pool's own disks
* [[HA Cluster Setup (external SAN)]] -- HA pools on LUNs from a back-end array
* [[Site Cluster Setup]] -- the Corosync and Pacemaker cluster the DLM runs on
* [[Windows MPIO for Failover Clusters and Hyper-V]] -- recommended multipathing setup and tuning for Windows cluster disks, Cluster Shared Volumes and Hyper-V
* [[QuantaStor CLI Command Reference]] -- full argument lists for the commands above

----
''Verified against QuantaStor 6.9.0.''

Windows MPIO for Failover Clusters and Hyper-V

2026-10-06T15:58:54Z

Qadmin: Recommended Windows MPIO setup and timer tuning for failover clusters, CSV and Hyper-V on QuantaStor HA pools

[[Category:admin_guide]]

This page covers the recommended Microsoft Multipath I/O (MPIO) setup and tuning for Windows Server hosts that use QuantaStor storage volumes as shared cluster disks: Windows Server Failover Clustering, Cluster Shared Volumes, Hyper-V clusters and SQL Server Failover Cluster Instances. It applies to Fibre Channel and iSCSI access to a QuantaStor HA pool, and it is the client-side companion to [[Clustered SCSI-3 Persistent Reservations]].

'''Summary.''' Install MPIO, let the Microsoft DSM claim QuantaStor devices, and raise three MPIO timers above their Windows defaults so that a disk survives a QuantaStor HA failover or an appliance reboot without Windows removing it. The settings were validated on Windows Server 2022 Hyper-V clusters connected over Fibre Channel to a QuantaStor HA pair.

{| class="wikitable"
! Section !! Purpose
|-
| [[#Why the defaults are not enough|Why the defaults are not enough]] || What Windows does when every path to a disk is briefly unavailable
|-
| [[#Installing MPIO and claiming QuantaStor devices|Installing MPIO and claiming QuantaStor devices]] || The MPIO feature and the QuantaStor hardware ID
|-
| [[#Recommended MPIO settings|Recommended MPIO settings]] || The timer values, and how to apply and verify them
|-
| [[#Paths and load balancing|Paths and load balancing]] || What a correctly configured disk looks like
|-
| [[#After maintenance on a QuantaStor appliance|After maintenance on a QuantaStor appliance]] || Checking that a rebooted appliance's paths are back before failing pools to it
|-
| [[#Troubleshooting|Troubleshooting]] || Disks that go offline during a failover, and missing paths
|}

== Why the defaults are not enough ==

During an HA failover the pool moves from one QuantaStor appliance to the other. For a short time no path to a volume can complete I/O: the departing appliance has stopped serving the pool and the receiving appliance has not yet finished importing it. With Fibre Channel and ALUA, QuantaStor keeps the standby paths through the partner appliance present throughout, so Windows sees the paths change state rather than disappear; the I/O stall is typically well under a minute.

Windows decides how long to wait with three MPIO settings. With the defaults, MPIO deletes the disk 20 seconds after its last path stops working (PDORemovePeriod), and each I/O times out after 60 seconds (DiskTimeoutValue). A failover that runs longer than that, for example because the pool export runs up to its configured Export Timeout, makes Windows remove the disk. For a cluster disk that means the disk goes Failed or Offline, a Cluster Shared Volume stops, and Hyper-V virtual machines lose their storage. Windows does not bring a removed disk back on its own when the pool comes back: an administrator has to rescan storage and bring the cluster disk online again.

Raising the timers makes Windows hold I/O and keep the disk while the pool moves, so the failover appears to the cluster as a pause rather than a disk failure.

== Installing MPIO and claiming QuantaStor devices ==

Do this on every cluster node, before presenting QuantaStor volumes to it.

'''1. Install the Multipath I/O feature''' (requires a reboot):

<pre style="font-size: smaller">
Install-WindowsFeature -Name Multipath-IO
</pre>

'''2. Add the QuantaStor hardware ID to the Microsoft DSM''' so that MPIO combines the paths to each QuantaStor volume into one disk:

<pre style="font-size: smaller">
New-MSDSMSupportedHW -VendorId OSNEXUS -ProductId QUANTASTOR
Get-MSDSMSupportedHW | Where-Object VendorId -match OSNEXUS
</pre>

The PowerShell cmdlet pads the vendor and product IDs to their SCSI field widths. If you use the MPIO control panel or {{Code|1=mpclaim}} instead, the string must be exactly {{Code|1=OSNEXUS QUANTASTOR}} followed by six trailing spaces, as described in [[Multipath IO Configuration#Configuring Microsoft MPIO|Configuring Microsoft MPIO]]. Without this entry each path appears as a separate disk.

'''3. Reboot''' the node so MPIO claims the devices, then confirm in Disk Management or with {{Code|1=Get-Disk}} that each QuantaStor volume appears once.

For iSCSI, also connect one session per path in the iSCSI Initiator with '''Enable multi-path''' selected, using addresses on separate subnets; see [[ISCSI Initiator Setup]] and [[Multipath IO Configuration]]. For an HA pool, connect to the pool's HA virtual interfaces rather than an appliance's own addresses, so the sessions follow the pool when it fails over.

== Recommended MPIO settings ==

{| class="wikitable"
! Setting !! Windows default !! Recommended !! Effect
|-
| PDORemovePeriod || 20 || '''240''' || Seconds MPIO keeps a disk after its last path has failed before removing it. Must be longer than the slowest failover you expect.
|-
| DiskTimeoutValue || 60 || '''100''' || Seconds before Windows times out an individual I/O. 100 is the largest value {{Code|1=Set-MPIOSetting}} accepts.
|-
| PathVerificationState || Disabled || '''Enabled''' || MPIO periodically tests every path, so a failed path is detected and a recovered path is put back into use without waiting for I/O to hit it.
|-
| PathVerificationPeriod || 30 || 30 || Seconds between path tests. The default is fine.
|-
| RetryCount || 3 || 3 || Times MPIO retries a failed I/O on a path. Leave at the default.
|-
| RetryInterval || 1 || 1 || Seconds between those retries. Leave at the default.
|}

Apply them in an elevated PowerShell on each cluster node:

<pre style="font-size: smaller">
Set-MPIOSetting -NewPathVerificationState Enabled -NewPathVerificationPeriod 30 -NewPDORemovePeriod 240 -NewRetryCount 3 -NewRetryInterval 1 -NewDiskTimeout 100
</pre>

'''Reboot the node for the settings to take effect.''' In a running cluster, drain one node at a time (Pause, Drain Roles in Failover Cluster Manager, or {{Code|1=Suspend-ClusterNode -Drain}}), reboot it, resume it, and move on to the next.

Verify with {{Code|1=Get-MPIOSetting}}, which is the authoritative view; the cmdlet stores the values through WMI, so reading the MPIO registry key directly can be misleading:

<pre style="font-size: smaller">
PS C:\> Get-MPIOSetting

PathVerificationState : Enabled
PathVerificationPeriod : 30
PDORemovePeriod : 240
RetryCount : 3
RetryInterval : 1
UseCustomPathRecoveryTime : Disabled
CustomPathRecoveryTime : 40
DiskTimeoutValue : 100
</pre>

If {{Code|1=Set-MPIOSetting}} reports a parameter error, for example a {{Code|1=-NewDiskTimeout}} above 100, none of the values in that command are applied. Check the output for errors and run {{Code|1=Get-MPIOSetting}} afterwards.

Leave the Fibre Channel HBA driver parameters at the vendor defaults; these settings were validated with the QLogic Windows driver at its defaults.

== Paths and load balancing ==

Leave the Microsoft DSM load-balance policy at its default. QuantaStor reports ALUA path states for HA pools: paths through the appliance that owns the pool are Active/Optimized and paths through the partner appliance are Standby. The Microsoft DSM's default for ALUA storage, Round Robin With Subset, sends I/O across the Active/Optimized paths only and switches to the other set when the states change.

Check a disk's paths with {{Code|1=mpclaim}}:

<pre style="font-size: smaller">
mpclaim -s -d # list MPIO disks
mpclaim -s -d 1 # paths and their states for MPIO disk 1
</pre>

A host with two HBA ports zoned to both appliances of an HA pair shows four paths per disk: two Active/Optimized and two Standby. After a failover, the states swap between the two appliances, and the number of paths stays the same.

== After maintenance on a QuantaStor appliance ==

When a QuantaStor appliance reboots or its Fibre Channel ports go down, Windows marks its paths as failed. When the appliance returns, Windows does not always add its paths back on its own. A disk then keeps running on the partner appliance's paths, but a pool failed back to the returning appliance would find that host with standby paths only.

After any appliance reboot, and before failing a pool back to it:

<ol>
<li>On every cluster node, rescan storage:
<pre style="font-size: smaller">
Update-HostStorageCache
pnputil /scan-devices
</pre>
If paths are still missing, run {{Code|1=rescan}} in {{Code|1=diskpart}}.</li>
<li>Check with {{Code|1=mpclaim -s -d <n>}} that every QuantaStor disk shows its full path count again.</li>
<li>Only then fail the pool back.</li>
</ol>

QuantaStor holds an appliance's Fibre Channel target ports disabled during startup until its LUNs are mapped and ALUA states are set, so wait until the appliance is fully up, with its Storage Pools showing a Normal state in the web interface, before rescanning.

== Troubleshooting ==

'''A cluster disk went Failed or Offline during a failover, and the disk disappeared from Disk Management.''' MPIO removed the disk because the failover outlasted PDORemovePeriod. Check {{Code|1=Get-MPIOSetting}} on every node; the values above must be in effect, which requires a reboot after setting them. To recover, run {{Code|1=Update-HostStorageCache}} and a {{Code|1=diskpart}} rescan on each node, then bring the cluster disk online in Failover Cluster Manager ({{Code|1=Start-ClusterResource}}).

'''Cluster disks went offline after a failover, but the disks are still present in Windows.''' Check that [[Clustered SCSI-3 Persistent Reservations]] is Enabled on the pool's HA group. Without it the reservations the cluster placed on the disks are lost when the pool moves, and the cluster treats the disks as failed.

'''A disk shows only Standby paths, or fewer paths than expected.''' One appliance's paths were not re-added after it rebooted. Follow [[#After maintenance on a QuantaStor appliance|After maintenance on a QuantaStor appliance]]. Also confirm that both of the host's HBA ports are zoned to both appliances, and that both of its WWPNs are listed under the host in QuantaStor.

'''Each QuantaStor volume appears more than once in Disk Management.''' MPIO has not claimed the devices. Check {{Code|1=Get-MSDSMSupportedHW}} for the OSNEXUS QUANTASTOR entry, add it if missing, and reboot.

'''Keep the appliances' Fibre Channel ports in the default target-only mode.''' Dual initiator and target mode ({{Code|1=qs-util enabledualmode}}) is a diagnostic setting. On an appliance that serves an HA pool, it can log out host sessions whenever the fabric changes, for example while the partner appliance reboots, which shows up on Windows as long I/O stalls.

== Related pages ==

* [[Clustered SCSI-3 Persistent Reservations]] -- the HA group setting Windows failover clusters need
* [[Multipath IO Configuration]] -- MPIO and Linux multipath basics for QuantaStor volumes
* [[ISCSI Initiator Setup]] -- connecting iSCSI initiators
* [[Fibre Channel Target Port Management]] -- presenting volumes over Fibre Channel
* [[Hosts and Host Groups]] -- host records, initiators, and Host Groups for clusters
* [[HA Cluster Setup (JBODs)]] -- HA groups and failover

----

Guides:Index

2026-10-03T02:45:06Z

Qadmin: osn-seo-utilities: regenerate index

Guides:Multi-Admin Approval for Destructive Storage Operations

2026-10-03T02:45:02Z

Qadmin: osn-seo-utilities: multi-admin-approval @ d95f434838dc (approved in the portal)

''By Steve Umbehocker, CTO, OSNexus · Updated October 3, 2026''

Multi-admin approval is a two-person rule for data-destructive operations: when it is enabled, a delete of a protected object type is held as a pending request until a set number of administrators approve it. QuantaStor applies it grid-wide from the Security Manager, covering deletes of storage volumes, network shares, buckets, scale-up and scale-out pools, object storage classes and Ceph clusters. Approvers work from the Multi-admin Approval toolbar in the web interface, and held deletes issued from the CLI report how many approvals they have so far.

== Why it matters ==

Most storage-side ransomware and insider attacks end with the same step: someone with valid administrator credentials deletes the pools, volumes or buckets that hold the data, and with them the snapshots that would have allowed recovery. Password policy and [[Multi-Factor Authentication Manager|multi-factor authentication]] make it harder to steal those credentials, but once an attacker holds them, a single account can still destroy everything.

Multi-admin approval removes that single point of failure. A stolen or misused account can request a delete, but the delete does not start until other administrators agree. The same control catches honest mistakes, such as a pool deleted on the wrong system or a share removed during the wrong change window.

For regulated environments, it fits alongside the controls described in [[Security Configuration]]: password and lockout policy with a '''Suggested Defaults''' preset aimed at standards such as HIPAA, CJIS and NIST 800-53 / 800-171, role based access control over every management operation, LDAP single sign-on for administrators, and always-on [[Audit Logging|audit logging]] of every management operation, login attempt and authorization decision. Agencies and contractors that need separation of duties for destructive actions can enforce it on the storage itself instead of relying on a change-control process alone.

== How it works ==

The workflow has three parts: a policy, a pending request and the approval votes.

# '''Policy.''' In '''Security Manager''', the '''Multi-admin Approvals''' tab turns the feature on, sets the minimum number of approvals and the expiry time, and selects which delete operations require approval. The policy applies to the whole storage grid.
# '''Pending request.''' When an administrator deletes an object whose type is selected, QuantaStor records a pending multi-admin approval request instead of starting the delete task. The request carries the object name, type and ID, its status, the required and current approval counts, the request originator and an expiry time.
# '''Approval votes.''' Other administrators open '''Security → Multi-admin Approval → Approve''', which "casts an approval vote for the associated data-destructive operation", or '''Reject''' to refuse it. Once the current count reaches the required count, the delete proceeds. If the request is not approved before it expires, the delete does not run.

From the CLI, a held delete reports its approval status, for example <code><nowiki>1 of 3 approvals met</nowiki></code>, rather than returning an empty task, so a script or operator can see exactly why the delete has not started.

[[File:Guide-multi-admin-approval-multi-admin-approval-security-manager.png|frame|center|The Multi-admin Approvals tab of Security Manager: the Enable checkbox, Minimum Approvals and Hours Until Auto-expiration, and the list of delete operations that can require approval]]

=== Operations that can require approval ===

{| class="wikitable"
! Required approval type !! Reference
|-
| Storage Volume : Delete || [[Storage Volume Delete]]
|-
| Network Share : Delete || [[Network Share Delete]]
|-
| Bucket : Delete || [[Object Bucket Delete]]
|-
| Scale-up Storage Pool : Delete || [[Storage Pools]]
|-
| Scale-out Block Pool : Delete || [[Storage Pools]]
|-
| Scale-out File Pool : Delete || [[Storage Pools]]
|-
| Scale-out Object Pool : Delete || [[Storage Pools]]
|-
| Object Storage Class : Delete ||
|-
| Ceph Cluster : Delete ||
|}

Each type is a separate checkbox, and '''Select All''' and '''Clear All''' set them in one step.

== Design and sizing ==

Three settings define the policy. Choose them with your team size and on-call coverage in mind.

{| class="wikitable"
! Setting !! What it controls !! Guidance
|-
| Minimum Approvals || Approvals a pending delete needs before it runs (2 by default in the dialog) || 2 suits most teams. Use a higher number only if that many administrators can realistically be reached during a change window.
|-
| Hours Until Auto-expiration || How long a request waits for approval (48 hours by default in the dialog) || Long enough to span a weekend change window, short enough that stale requests don't linger. A value of 0 means requests never expire.
|-
| Required Approval Types || Which delete operations are held || Select all pool, cluster and bucket types at a minimum, because those deletes remove the most data in one step.
|}

Some practical points:

* '''Give every administrator a named account.''' Approval only means something if each vote comes from a different person. Shared logins such as a single <code><nowiki>admin</nowiki></code> account defeat the purpose. Map named accounts to roles, or use LDAP single sign-on so administrator groups are managed in your directory.
* '''Decide who may approve.''' Approving and rejecting are ordinary RBAC permissions on the <code><nowiki>PendingOperationRequest</nowiki></code> object type (<code><nowiki>view</nowiki></code>, <code><nowiki>create</nowiki></code>, <code><nowiki>approve</nowiki></code>, <code><nowiki>reject</nowiki></code>, <code><nowiki>clear</nowiki></code>). Grant <code><nowiki>approve</nowiki></code> and <code><nowiki>reject</nowiki></code> to the roles that should vote, and leave them off roles such as System Monitor.
* '''Protect the approvers.''' Turn on multi-factor authentication for every account that holds the approve permission, so a single stolen password cannot both request and approve.
* '''Keep snapshots in the plan.''' Approval stops a malicious delete, but it does nothing against data that is encrypted or overwritten in place. Pair it with [[Snapshot Schedules|snapshot schedules]] so there is a clean recovery point when data inside a volume or share is damaged.

[[File:Guide-multi-admin-approval-multi-admin-approval-role-permissions.png|frame|center|Update Role Permissions filtered to the PendingOperationRequest operations, where approve and reject rights are granted to a role. The Administrator role already has this permission but you can create new roles that are limited to a subset of operations including PendingOperationRequest approval and rejection.]]

== Setting it up ==

# Create a named account for each administrator and assign roles under '''Security → Management Users''' and '''Management Roles'''. In '''Add/Remove Permissions''' for each approving role, confirm <code><nowiki>PendingOperationRequest</nowiki></code> <code><nowiki>approve</nowiki></code> and <code><nowiki>reject</nowiki></code> are granted at the scope you need.
# Enable multi-factor authentication for those accounts from '''Multi-Factor Auth Manager'''.
# Open '''Security → Management Users → Security Manager''' (toolbar) and select the '''Multi-admin Approvals''' tab.
# Tick '''Enable Multi-admin Approvals'''.
# Set '''Minimum Approvals''' and '''Hours Until Auto-expiration'''.
# Under '''Required Approval Types''', tick the delete operations to protect, or use '''Select All'''.
# Click '''OK'''. Changes made only on this tab save without the logout and password-expiry confirmation that password-policy changes trigger, so you can enable the feature during working hours.

To roll it out without blocking routine work, start with the operations that destroy the most data at once (pools, Ceph clusters, object storage classes and buckets), run with them for a few weeks, then add volume and share deletes once the team is used to the approval step. Volume and share cleanup is more frequent, so plan who covers approvals before you protect those types.

== Operating and testing ==

Test the policy before relying on it. On a non-production object, for example a scratch volume <code><nowiki>vol-test1</nowiki></code> in <code><nowiki>pool1</nowiki></code>:

# As one administrator, delete <code><nowiki>vol-test1</nowiki></code>. The delete should not start. From the CLI, the response shows the approval status, such as <code><nowiki>1 of 2 approvals met</nowiki></code>.
# As a second administrator, open '''Security → Multi-admin Approval → Approve'''. The '''Pending Multi-admin Approval Requests''' list shows the object name, object type, required and current counts, the originator and the expiry time.
# Select the request and approve it. When the required count is reached, the delete task runs and appears in the Tasks pane.
# Repeat with a second scratch object and use '''Reject''' to confirm a refused delete never runs.

If nothing is waiting, the Approve dialog reports that there are no pending operation requests to approve.

Day to day, treat an unexpected pending request as a security event: check who originated it, and reject it if it isn't tied to an approved change. Every request, approval and delete is recorded by [[Audit Logging|audit logging]], which gives you the trail auditors ask for. For scripted cleanup jobs, check the CLI approval status so the job reports a held delete instead of treating it as a failure. Command syntax is in the [[QuantaStor CLI Command Reference]].

== FAQ ==

=== Which operations does multi-admin approval cover? ===

It covers deletes: storage volumes, network shares, buckets, scale-up storage pools, scale-out block, file and object pools, object storage classes and Ceph clusters. You choose which of these require approval on the Multi-admin Approvals tab of Security Manager. Other management operations are governed by role based access control as usual.

=== Does multi-admin approval protect against ransomware? ===

It protects against the destructive step attackers use once they hold administrator credentials: deleting pools, volumes or buckets so there is nothing left to recover from. It does not stop data from being encrypted inside a volume or share, so combine it with snapshot schedules, replication schedules, multi-factor authentication and audit logging.

=== What happens if nobody approves a request? ===

The delete never runs. The request expires after the configured '''Hours Until Auto-expiration''', and the object stays in place. Setting the expiry to 0 keeps requests open until they are approved or rejected.

=== Is multi-admin approval suitable for government and regulated deployments? ===

It enforces separation of duties for data destruction on the storage system itself, which supports audit and access-control requirements in frameworks such as NIST 800-53 and 800-171. It works alongside QuantaStor's password and lockout policy presets, RBAC, multi-factor authentication, LDAP single sign-on and always-on audit logging, all described in Security Configuration.

=== Can an administrator approve their own delete? ===

Treat approval as needing different people: give each administrator a named account, grant the approve permission only to roles that should vote, and protect those accounts with multi-factor authentication. Check the Request Originator column before approving a request.

----
''Part of the [[Guides:Index|QuantaStor Guides]] series. For reference documentation, see the [[Main Page|QuantaStor documentation]].''

[[Category:Guides]]

File:Guide-multi-admin-approval-multi-admin-approval-role-permissions.png

2026-10-03T02:45:02Z

Qadmin: osn-seo-utilities: screenshot for Guides (multi-admin-approval)

osn-seo-utilities: screenshot for Guides (multi-admin-approval)

File:Guide-multi-admin-approval-multi-admin-approval-security-manager.png

2026-10-03T02:45:02Z

Qadmin: osn-seo-utilities: screenshot for Guides (multi-admin-approval)

osn-seo-utilities: screenshot for Guides (multi-admin-approval)

Guides:Index

2026-10-03T02:30:07Z

Qadmin: osn-seo-utilities: regenerate index

Guides:Asynchronous Remote Replication and Disaster Recovery Failover

2026-10-03T02:30:03Z

Qadmin: osn-seo-utilities: remote-replication-dr @ f2ee090c4281 (approved in the portal)

''By Steve Umbehocker, CTO, OSNexus · Updated October 3, 2026''

Asynchronous remote replication keeps a second, mountable copy of your volumes and shares at another site, updated on a schedule by sending only the blocks that changed. QuantaStor replicates Storage Volumes and Network Shares between Storage Pools on systems in the same grid. When the primary site is lost, you activate the replica checkpoints at the DR site, and later you can fail back with or without the changes made.

== Why it matters ==

Snapshots protect against mistakes, but they live in the same pool as the data, so they go down with the pool, the system or the site. Replication puts a full copy somewhere else. The three scheduled data-protection features answer different questions:

{| class="wikitable"
! Feature !! Where the copy lands !! Use it for
|-
| [[Snapshot Schedules]] || Destination is the same Storage Pool || Fast local recovery points, undoing a bad change
|-
| Remote replication || Destination is a Storage Pool on another system within the grid || Surviving the loss of a system, pool or site
|-
| [[Backup Policies]] || Destination is cloud object storage or an an external NFS/SMB source || Provides great flexibility and you can do inbound or outbound policies and autotiering but it doesn't have the block-level incremental efficiency of the other methods
|}

A replication schedule already snapshots its sources and keeps its own retention on both sides, so you rarely need a separate Snapshot Schedule on the same volume or share.

== How it works ==

Replication is snapshot-driven ZFS send/receive between two pools. The full reference is [[Remote-replication (DR)]]. The moving parts:

* '''Grid membership.''' Both systems must belong to the same storage grid. A grid can span sites, so replicating to a remote site means joining the remote appliances to this grid. See [[Grid Configuration]].
* '''Storage System Replication Link.''' A trust relationship and data path between two systems. Creating one generates a dedicated SSH key pair, registers each side's public key on the other, and pins replication traffic to a fixed IP address on each end. Links come in pairs: A to B creates B to A as well.
* '''Replication schedule.''' Names a link (which sets the direction), a destination pool, the volumes and shares, when to run, and how many snapshots to keep on each side. The system that owns the destination pool owns and runs the schedule.
* '''Replica checkpoint.''' For each source, the destination pool holds a real volume or share named with a <code><nowiki>_chkpnt</nowiki></code> suffix (<code><nowiki>backups</nowiki></code> becomes <code><nowiki>backups_chkpnt</nowiki></code>), plus timestamped snapshots that serve as retained recovery points.
* '''Replica association.''' The persistent source-to-checkpoint relationship, where live status and sync times are recorded.

Only the first run is a full copy. After that, each run snapshots the source, finds the newest GMT-stamped snapshot that both source and checkpoint have, and sends the difference. The pools don't need to match in size, layout, disk type or hardware, and the remote-replication feature must be included in the license.

On a Ceph scale-out cluster, Ceph RBD pools can be a destination only when the remote system belongs to the same Ceph cluster as the source. Network Shares on scale-out CephFS replicate to a second Ceph cluster through CephFS snapshot mirroring instead; see [[Scale-out File Replication]].

== Design and sizing ==

'''RPO.''' On the Schedule Interval tab, a timer interval is the rest time between the ''end'' of one run and the start of the next: 3 minutes minimum, 30 by default, up to 360. Your worst-case data loss is about one interval plus the duration of a run, so the real lever is how long a run takes. A run never starts while the previous one is still working. For a fixed timetable, the day/hour grid gives a calendar schedule, with an offset of 0–59 minutes to stagger several schedules. Frequent small runs usually transfer less each time and keep the DR copy closer than one big nightly run.

'''Bandwidth.''' Each link has a Bandwidth Limit in MB/sec (the dialog suggests 200; entering 0 means the service default of 100). The limit is shared evenly across the streams currently running on that link and rebalanced as streams finish, so four concurrent replications on a 200 MB/sec link get 50 MB/sec each. Size the limit against your daily change rate. This is plain arithmetic, not a measured benchmark:

{| class="wikitable"
! Changed data per run !! Time to send at 100 MB/sec !! Time to send at 200 MB/sec
|-
| 10 GB || about 1.7 minutes || about 50 seconds
|-
| 100 GB || about 17 minutes || about 8.5 minutes
|-
| 1 TB || about 2.8 hours || about 1.4 hours
|}

The first full copy is the long one. Seeding with a one-time replica (below) before you enable a tight schedule keeps the initial copy from colliding with production hours.

'''Transport.''' Encryption is on by default and tunnels the stream over SSH. Turning it off sends the stream over a plain mbuffer TCP session, which is faster on a trusted private link and offers no confidentiality. Compression adds lz4 only when the source dataset isn't already compressed.

'''Retention.''' The Snapshot Settings tab keeps 3 short-term snapshots by default (the recommended value) for computing deltas, plus long-term hourly, daily, weekly, monthly and quarterly counts set separately for source and checkpoint. Keep enough snapshots that both sides always share one; otherwise the next run falls back to a full copy.

'''HA pools.''' Each run re-selects its link based on which systems currently own the source and destination pools. With an HA pool on systems A and B replicating to one on C and D, create all four links (A–C, A–D, B–C, B–D) so replication keeps running after a failover on either end.

== Setting it up ==

# Join both systems to one grid ([[Grid Configuration]]) and create a destination pool ([[Storage Pools]]).
# Create the link: '''Remote Replication → Storage System Replication Links → Replication Link → Create'''. Choose the storage system and a fixed IP address for each side. Floating cluster VIF addresses are filtered out. For a cloud system with a public address, the far side can use '''External IP Address''', which is pre-filled from the system's external hostname (set with <code><nowiki>qs system-modify --ext-hostname</nowiki></code>, see the [[QuantaStor CLI Command Reference]]).

[[File:Guide-remote-replication-dr-remote-replication-dr-replication-link-create.png|frame|center|The Create Storage System Replication Link dialog, with an existing link pair (primary site to DR site) behind it: pick a system and fixed IP address for each end, then check the Bandwidth Limit and the Encryption box, which is on by default]]

# Create the schedule: '''Remote Replication → Volume & Share Replication Schedules → Replication Schedule → Create'''. On the General tab, pick the link (this sets the direction) and the Remote Pool. Then set the interval, select volumes and shares (selecting a parent share and one nested inside it is rejected), set retention, and review Advanced Settings. Leave '''Enable resumable replication''' and '''Enable target checkpoint recovery''' on, as they are by default.
# To seed a destination or copy one volume on demand without a schedule, use [[Create Volume Replica]]. The diff-copy option sends changes against an existing checkpoint; full copy creates a new one.
# Interval schedules start shortly after you create them. To run any schedule immediately:

<pre>qs replication-schedule-trigger --schedule=nightly-dr
qs replica-assoc-list
qs replica-report-summary-list</pre>

== Operating and testing ==

Watch the '''Volume & Share Replica Associations''' section. Each source shows its status (Synchronizing, Synchronized, Sync Failed (Resumable), Skipped and so on), elapsed time, and when the last sync started and completed. Each run also writes a summary report with bytes transferred and average speed, which is the number to check against your RPO math.

[[File:Guide-remote-replication-dr-remote-replication-dr-replica-associations.png|frame|center|The Remote Replica Associations grid after a scheduled run: a volume and a share, both Synchronized, with sync start and completion times and their _chkpnt targets on the DR system]]

[[File:Guide-remote-replication-dr-remote-replication-dr-replication-report.png|frame|center|The Replication Report for the schedule: a full first sync of 1.29 GB, then incremental runs that send only the changed blocks, plus one run skipped because the previous one was still active]]

If a run is interrupted (the WAN drops, a node reboots), resumable replication keeps a ZFS resume token and the next run continues from where it stopped. A failed run raises an alert, at most one per hour per schedule and source; see [[Call-home / Alerting]] for sending alerts off the appliance. A run is skipped, with its reason shown on the schedule, if a link is missing or not Normal, if a pool is unhealthy, or if a destination HA failover is in progress.

'''Failover.''' '''Activate Checkpoints''' on the schedule toolbar is the promotion step. It disables the schedule, marks the checkpoints as Active Replica Checkpoints, brings checkpoint shares online and makes checkpoint volumes available to map to hosts. It can also create share aliases named after the sources, so clients reach <code><nowiki>backups</nowiki></code> rather than <code><nowiki>backups_chkpnt</nowiki></code>. Optional <code><nowiki>dr-prefailover</nowiki></code> and <code><nowiki>dr-postfailover</nowiki></code> scripts in <code><nowiki>/var/opt/osnexus/custom/</nowiki></code> handle site-specific steps such as repointing DNS. For hands-off failover, the Automatic Activation tab activates checkpoints when a Site Cluster VIF ([[Site Cluster Create]]) moves to the destination node.

[[File:Guide-remote-replication-dr-remote-replication-dr-replication-schedule-toolbar.png|frame|center|A replication schedule that runs every 15 minutes and keeps 10 checkpoints. Its toolbar has Activate Checkpoints, Deactivate Checkpoints and Rollback for failover and failback]]

'''Failback.''' After a test, disconnect DR-site clients, then use '''Deactivate Checkpoints''' with '''Re-enable Replication Schedule''' ticked; the next run overwrites the checkpoints from the source. If the DR site took writes you need to keep, use '''Rollback''' first (<code><nowiki>qs replication-schedule-trigger-rollback --schedule=<name></nowiki></code>). It sends the checkpoint's changes back and overwrites the original source. Then deactivate and re-enable.

'''Test it regularly.''' A DR test is activate, mount and verify from a DR-site client, then deactivate. Any client session on a checkpoint marks it active again on its own, and while any checkpoint is active the schedule stays blocked. A test that ends with clients still attached quietly stops replication, so check the schedule's state afterwards.

== FAQ ==

=== Does QuantaStor support asynchronous replication with DR failover between sites? ===

Yes. Replication schedules copy Storage Volumes and Network Shares asynchronously and incrementally to a Storage Pool on another grid member, which can be at another site. Failover is the Activate Checkpoints step, either manual or automatic on a Site Cluster VIF move. Failback either discards the DR-site changes or rolls them back to the source first.

=== What RPO can I get? ===

The shortest timer interval is 3 minutes, measured from the end of one run to the start of the next. Your effective RPO is that interval plus however long a run takes, which depends on change rate and the link's bandwidth limit. Check the average speed in the replication reports to see what you're actually achieving.

=== Do the source and destination systems need identical hardware? ===

No. The pools don't need to match in size, layout, disk type or hardware. Both systems do need to be in the same grid and licensed for remote replication. Ceph RBD destinations must be in the same Ceph cluster as the source.

=== Can one source replicate to more than one site? ===

Yes. Create one schedule per destination (N-way), or replicate from the DR site's checkpoint to a third site (cascading). On the second and later schedules, enable source snapshot reuse so all destinations share a common snapshot.

=== Will replication overwrite changes made at the DR site? ===

Not while a checkpoint is active. An active checkpoint blocks the schedule, and any iSCSI, FC, NFS or SMB client session marks a checkpoint active automatically. Overwriting happens only after you deactivate, so roll back first if you want to keep DR-site writes.

----
''Part of the [[Guides:Index|QuantaStor Guides]] series. For reference documentation, see the [[Main Page|QuantaStor documentation]].''

[[Category:Guides]]

File:Guide-remote-replication-dr-remote-replication-dr-replication-schedule-toolbar.png

2026-10-03T02:30:03Z

Qadmin: osn-seo-utilities: screenshot for Guides (remote-replication-dr)

osn-seo-utilities: screenshot for Guides (remote-replication-dr)

File:Guide-remote-replication-dr-remote-replication-dr-replication-report.png

2026-10-03T02:30:03Z

Qadmin: osn-seo-utilities: screenshot for Guides (remote-replication-dr)

osn-seo-utilities: screenshot for Guides (remote-replication-dr)

File:Guide-remote-replication-dr-remote-replication-dr-replica-associations.png

2026-10-03T02:30:03Z

Qadmin: osn-seo-utilities: screenshot for Guides (remote-replication-dr)

osn-seo-utilities: screenshot for Guides (remote-replication-dr)

File:Guide-remote-replication-dr-remote-replication-dr-replication-link-create.png

2026-10-03T02:30:03Z

Qadmin: osn-seo-utilities: screenshot for Guides (remote-replication-dr)

osn-seo-utilities: screenshot for Guides (remote-replication-dr)

Guides:Index

2026-10-02T23:45:06Z

Qadmin: osn-seo-utilities: regenerate index

''Practical, in-depth guides to designing and running QuantaStor storage. Each guide links to the reference documentation for the features it covers.''

== Guides ==

* '''[[Guides:Tiering On-Premises Object Storage to AWS S3 with Lifecycle Policies|Tiering On-Premises Object Storage to AWS S3 with Lifecycle Policies]]''' — Cloud tiering moves older objects from a local S3 bucket to a bucket at AWS while the bucket keeps serving the same namespace.

[[Category:Guides]]