Multipath Configuration
Multipathing gives a storage system more than one physical route to the same disk, and presents those routes to the operating system as a single device. This page covers how QuantaStor decides which devices are multipath capable, how it names the resulting devices, how to read a multipath device's paths, and how multipathing underpins scale-up high-availability pools built on shared SAS JBODs.
QuantaStor uses the Linux device-mapper multipath stack (DMMP), managed by the multipathd service and configured through /etc/multipath.conf. You do not normally configure any of this by hand -- QuantaStor detects multipath-capable media and configures it automatically.
| Section | Purpose |
|---|---|
| Which devices are multipath capable | How QuantaStor tests a device for multiple ports |
| Reading a multipath device name | The dm-uuid-mpath- and dm-name-enc- naming, and the WWID
|
| Paths and the device stack | How a multipath device relates to the paths beneath it |
| Automatic multipath configuration | The Multipath Config Mode setting and when autoconfiguration runs |
| Multi-path Device Configurator | Reviewing and editing the configured vendor/product list |
| Rescanning after a hardware change | Picking up newly attached multipath media |
| The multipath.conf file | Why this file must not be hand-edited |
| Multipathing and scale-up HA pools | Shared JBODs, I/O fencing, and the configuration alert |
| Troubleshooting | Fewer paths than expected, and no multipath devices at all |
Which devices are multipath capable
QuantaStor does not guess from the vendor name. For a SCSI device (SAS, Fibre Channel or iSCSI) it reads the device's own SCSI INQUIRY response and tests the MultiP bit -- bit 4 of INQUIRY byte 6 -- which the device sets when it has more than one port. For an NVMe device it reads the CMIC field reported by nvme id-ctrl, which describes the controller's multi-path and namespace-sharing capability.
You can read the same bit the service reads:
python3 -c 'b=open("/sys/block/sdb/device/inquiry","rb").read(); print(bool(b[6] & 0x10))'
Three rules narrow the result further:
- ATA devices are never autoconfigured for multipathing. A SATA disk is single-ported, so a device reporting a vendor or product ID of
ATAis skipped. This is why an internal SATA boot device stays a plainsddevice while the SAS media around it becomes multipathed. - Devices already at a higher level of the stack are skipped. A device that already has child paths, or whose name contains
dm-, is not itself a candidate. - Where the INQUIRY answer is inconclusive, QuantaStor falls back to I/O fencing support. If the device's multipath capability cannot be determined, the device is treated as multipath capable only if it supports the SCSI persistent reservations that I/O fencing needs. The two capabilities travel together in practice, because both are properties of shared, multi-ported enterprise media.
Devices known to be unsuitable are excluded outright regardless of what they report -- GRAID and Pliops accelerator control devices, and the QuantaStor RAM disk, which device-mapper would otherwise claim exclusively and break.
A separate exclusion list, /etc/quantastor/qs_device_blocklist.conf, keeps non-storage SCSI devices such as iDRAC virtual floppies and USB virtual disks out of the disk inventory altogether. See QuantaStor Configuration Files.
If your media is behind a hardware RAID controller rather than an HBA, the controller presents each logical drive as a single device and handles its own internal redundancy, so there is nothing for DMMP to aggregate and no multipath configuration to do. Multipathing as described on this page applies to HBA-attached SAS, Fibre Channel and iSCSI media.
Reading a multipath device name
A multipath device is named for the WWID of the disk it represents, not for the enclosure slot or the kernel device letter. That is the point of it: the name is stable across reboots, across cable moves, and -- importantly for high availability -- identical on every appliance that can see the disk.
The names you will see in disk listings and as storage pool members take one of two forms:
| Name | What it is |
|---|---|
dm-uuid-mpath-3<wwn> |
The multipath device itself |
dm-name-enc-dm-uuid-mpath-3<wwn> |
An encrypted device layered on top of that multipath device |
sd<x> |
A single path -- one physical route to the disk |
The enc- element is easy to misread as part of the multipath naming. It is not: it marks a LUKS encryption layer stacked on the multipath device below it. A name of the form dm-name-enc-dm-uuid-mpath-35000c5003af293d3 is read from the inside out -- the multipath device for WWID 35000c5003af293d3, with encryption on top. On such a device QuantaStor reports Is Encrypted: true and an encryption cipher; on the multipath device beneath it, both are absent. The encrypted device also reports very slightly less capacity than the multipath device under it, because the LUKS header consumes the difference.
The WWID
In 35000c5003af293d3 the leading 3 is the SCSI name-designator prefix added by scsi_id, and the remaining 16 hex digits are the disk's NAA World Wide Name. The disk's two SAS port addresses are adjacent to that WWN, differing only in the final digit -- a drive with WWID 35000c5003af293d3 reports SAS port addresses 5000c5003af293d1 and 5000c5003af293d2. Both ports resolve to the one WWID, which is exactly what lets device-mapper recognise two routes as one disk.
QuantaStor surfaces the WWID as the SCSI ID of a Physical Disk, and the port addresses as SAS Address Port0 and SAS Address Port1. See Physical Disks/Devices.
Aliases
The shipped configuration sets user_friendly_names yes, so multipathd also assigns each device a short alias -- mpatha, mpathb and so on -- and records the alias-to-WWID mapping in /etc/multipath/bindings. That file carries its own warning that it is maintained automatically.
Do not use the mpath<x> alias to identify a disk. Aliases are assigned per appliance in first-discovery order and are local to that appliance's bindings file, so the same alias can refer to different disks on two nodes that share a JBOD. QuantaStor itself never relies on them; it addresses devices by dm-uuid-mpath-<wwid>. Use the WWID whenever you need to identify a shared disk, and in any script.
Which link QuantaStor prefers
A single device is reachable through several symlinks under /dev/disk/, and QuantaStor has to pick one canonical path per device. /etc/quantastor/qs_device_link.conf assigns a weight to each link prefix and excludes some outright, which is what makes the choice deterministic -- an encrypted multipath link outranks a plain multipath link, which outranks a raw SCSI link. This file is listed in QuantaStor Configuration Files; like multipath.conf it is managed by QuantaStor and should not be hand-edited.
Paths and the device stack
A multipath device sits above one or more path devices, and QuantaStor models the relationship in both directions on the Physical Disk object:
- Multipath Parent Disk ID points up the stack, from a path to the device that aggregates it.
- The path list points down, from a device to the paths beneath it.
For an encrypted, multipathed disk that gives a three-level chain -- the sd path names the multipath device as its parent, and the multipath device names the encryption device as its parent. The encryption device, at the top, has no parent. SMART data is read from the leaf path device and reported on whichever device is on top, so health information does not disappear behind the stack.
Only the top of each stack is offered as usable storage. A device with a multipath parent is treated as in use, so the Physical Disks grid and the Create Storage Pool disk list show one row per usable device -- the encryption device where there is one, otherwise the multipath device -- and never the individual sd paths. This is why a system with 17 shared disks presents 17 selectable devices rather than one per path, and why storage pool members appear under their dm- names.
To see the paths themselves and their individual states, use the multipath tools directly on the appliance:
multipath -ll
Each device is reported with its alias and WWID, then one line per path group and one line per path, ending in that path's state -- for example active ready running. The path selector, path grouping policy and priority in effect are shown alongside.
The values that govern path behaviour come from the shipped configuration and the multipath tools' own defaults rather than from any QuantaStor setting. On a current release the effective defaults include a service-time 0 path selector, a failover path grouping policy, a tur (Test Unit Ready) path checker, manual failback, and a 5-second polling interval. Per-array overrides for a number of SAN products ship in the devices section of multipath.conf. Report the values actually in force with:
multipathd show config
(Note: these defaults track the multipath-tools package and may change between releases; treat the output of the command above as authoritative rather than the values quoted here.)
Path loss compared with device loss
The two failures are not the same and are not reported the same way.
- Losing a path while at least one path survives is handled inside device-mapper. The multipath device keeps its name and stays available, I/O continues over the remaining path, and any storage pool built on it is unaffected. The loss is visible as a path state change in
multipath -ll; the QuantaStor Physical Disk object continues to represent the device as a whole. - Losing the last path removes access to the disk. The multipath device has no route to serve I/O, and the failure surfaces the way any disk failure does -- as a degraded storage pool. Recovery is a hardware matter: restore the route, then rescan.
Because a multipath device is named for its WWID and not for its paths, restoring a path does not rename anything, and a pool does not need to be re-imported after a cable is reseated.
Automatic multipath configuration
Each storage system carries a Multipath Config Mode setting, and it is Enabled by default.
While it is enabled, QuantaStor scans the attached media for multipath-capable devices and adds an entry for each distinct vendor and product ID it finds to /etc/multipath.conf, then refreshes the multipath configuration and rescans for disks. This runs at service startup and again during a forced disk scan, and it only ever adds entries that are missing -- if nothing is new, nothing is written.
The practical effect is that attaching a supported SAS JBOD requires no multipath configuration at all. The models present in the enclosure appear in the configured list on their own, and their disks appear as multipath devices.
Setting the mode to Disabled stops that scanning. QuantaStor will also try to clear the configured entries at that point, but it refuses to do so while storage pools are present on the appliance -- clearing multipath configuration out from under a live pool would strand it. Remove or migrate the pools first if you genuinely need a cleared configuration.
The equivalent CLI setting is on qs system-modify --storage-system=<system> --multipath-autoconfig=<enabled|disabled>.
Note that creating a Ceph cluster disables multipath autoconfiguration on the participating appliance, because Ceph OSDs manage their own devices.
Multi-path Device Configurator
This dialog shows which vendor and product IDs are currently configured for multipath access, and lets you add or remove them. It is the supported way to change that list.
The same dialog is on the toolbar as Multipath Config in the Physical Disk group.
Storage System selects the appliance whose configuration you are editing. Multipath configuration is per appliance, not grid-wide, because it describes the media physically attached to that appliance. Changing the selection reloads the grid and rescans.
The SCSI Device Multipath Vendor/Product Identifiers grid lists one row per configured entry, with its SCSI Vendor ID and SCSI Product ID. Both fields are regular expressions rather than literal strings, which is how single entries such as EMC:.* cover a whole product family. The shipped list already contains entries for common SAN arrays -- 3PARdata, DataCore, Dell, EMC, HITACHI, HP, IBM, Intel and NetApp among them -- alongside OSNEXUS:QUANTASTOR for QuantaStor's own targets. Entries added by autoconfiguration for locally attached media appear in the same list.
The controls below the grid work on that list:
- Scan re-detects the multipath-capable models on the selected appliance and loads them into the combo box beside Add. The scan reports what the hardware supports; it does not change any configuration.
- Add adds the model currently shown in the combo box to the grid.
- Remove Selected removes the rows whose checkboxes are ticked from the grid.
- Clear Changes discards your edits and reloads the grid from the configuration file as it currently stands.
The grid is the new configuration, not a list of changes. On OK the dialog replaces the configured entry list with exactly the rows shown, so a row you removed from the grid is removed from multipath.conf. That is worth pausing over before clicking OK on a system with pools: removing the entry that covers a model already in use withdraws multipath support from disks that are carrying data. Cancel writes nothing.
After the change is applied QuantaStor refreshes the multipath configuration and rescans for disks automatically, so there is no separate step to make the result visible.
Because both fields are regular expressions, matching a family of models means writing a regular expression and not a shell-style glob -- EG0300.*, with the dot, rather than EG0300*. QuantaStor additionally validates any entry you add: it must name both a vendor and a product, and where it contains a * that character may appear only at the end of the half it is in, at most once. HP:EG0300.* and EMC:.* are accepted; *:LUN and H*P:LUN are rejected.
Rescanning after a hardware change
After attaching new media, run a disk scan.
The dialog selects which storage systems to scan and offers one multipath-specific option, Rescan Multipath Configuration, which is checked by default. Leaving it checked makes the scan a full multipath refresh: QuantaStor re-runs autoconfiguration for newly detected models, flushes multipath maps that are not in use, restarts multipathd, re-triggers udev, and drops disk records that no longer resolve to hardware. Clearing it limits the operation to an ordinary bus rescan.
Because that refresh restarts the multipath service, prefer to run it during a maintenance window on an appliance with active pools. The flush itself only removes maps that are not in use, so devices carrying pools are not torn down, but a service restart is still a disruption worth scheduling.
The CLI equivalent is qs disk-scan. See Physical Disks/Devices for the rest of the dialog.
The multipath.conf file
/etc/multipath.conf is a QuantaStor-managed file. QuantaStor ships it, registers it in its configuration-file registry, and rewrites its blacklist_exceptions section whenever multipath entries are added or removed -- through autoconfiguration, through the Multi-path Device Configurator, or through the CLI. When it rewrites that section it reformats it, so hand-made formatting there does not survive.
Do not hand-edit this file. Edit the configured device list through the dialog or the CLI instead. Two things make hand-editing more dangerous here than the general rule suggests:
- The file has no in-file warning. Unlike some other QuantaStor-managed files it begins straight at its first configuration block, with no header saying it is generated. There is nothing in the file to tell an administrator that their changes are at risk.
- One of its blacklist rules is load-bearing for ZFS. The shipped
blacklistsection excludeszddevice nodes, which are ZFS volumes. If that exclusion is removed, multipath claims ZFS volumes and locks them, so storage volumes cannot be deleted. QuantaStor guards against this by checking for the rule at startup and stoppingmultipathdoutright if it is missing -- which presents as every multipath device disappearing at once.
The overall structure is worth understanding even though you should not edit it. The blacklist section excludes everything -- it contains a device entry matching any vendor and any product -- and blacklist_exceptions then allows specific vendor/product pairs back in. Multipathing is therefore opt-in per device model, which is why a model that is not in the exception list produces no multipath device no matter how many paths it has. That single design decision explains most multipath surprises on this platform.
For the full list of files QuantaStor manages, see QuantaStor Configuration Files.
Multipathing and scale-up HA pools
Multipathing is not primarily a performance feature on this platform. Its main job is to make a shared SAS JBOD safe to build a highly available storage pool on.
In a scale-up HA configuration, two appliances are cabled to the same JBOD and each disk's two SAS ports are wired one to each appliance. Because a multipath device is named for the disk's WWID, both appliances name the same disk identically, and a pool created on either appliance imports cleanly on the other without any device renaming. That shared, stable naming is what makes failover possible. See HA Cluster Setup (JBODs) for the full procedure, and HA Cluster Setup (external SAN) for the SAN-attached equivalent.
Two consequences follow that are worth knowing before you build one.
Single-ported media must not be used in a shared JBOD. QuantaStor actively checks for this: when it finds a disk that is reachable from more than one appliance, is a member of a shared pool, and does not support multipathing, it raises a multipath configuration problem alert against the pool -- "Storage pool '<name>' contains one or more SCSI disk devices which do not support multipathing and should not be used in a Storage Pool High-Availability configuration" -- and logs the offending disk by SCSI ID with a recommendation to remove it from the enclosure. Treat that alert as a correctness problem, not a warning to acknowledge: a single-ported disk in a shared JBOD cannot be fenced reliably, so the HA guarantees do not hold for that pool.
Multipath capability and I/O fencing capability go together. Both depend on the same class of media, and where a device's multipath capability is inconclusive QuantaStor decides it by asking whether the device supports the persistent reservations that fencing requires. If you are choosing drives for an HA JBOD, dual-ported SAS media with persistent-reservation support is the requirement, and the two settings sit side by side on the Storage System Modify dialog for that reason.
One point of topology often mistaken for a fault: in this configuration each appliance normally has exactly one path to each shared disk, because each of the disk's two ports goes to a different appliance. A device showing a single path in multipath -ll on a shared-JBOD HA cluster is expected, and redundancy is delivered at the appliance level by the HA group rather than at the path level. Dual paths from a single appliance require either dual HBA connections to the enclosure or a SAN fabric with more than one route.
CLI reference
Four commands manage the configured multipath device list. All four accept an optional --storage-system to target a grid member other than the one you are connected to, and default to the local appliance.
| Command | Purpose |
|---|---|
qs disk-multipath-config-list |
Lists the vendor/product entries currently configured |
qs disk-multipath-config-scan |
Reports the multipath-capable models detected on the hardware |
qs disk-multipath-config-add --device-tag-list=<list> |
Adds entries |
qs disk-multipath-config-remove --device-tag-list=<list> |
Removes entries |
A device tag is a single vendor:product string, and --device-tag-list takes a comma-separated list of them:
qs disk-multipath-config-scan qs disk-multipath-config-add --device-tag-list=HGST:HUC101818CS4200,TOSHIBA:MBF2600RC qs disk-multipath-config-list
Unlike the dialog, add and remove are incremental -- they change only the entries you name. Both refresh the multipath configuration and rescan for disks when they finish. A remove of an entry that is not configured is skipped rather than treated as an error, and the command reports only the entries it actually removed.
To turn autoconfiguration on or off, use qs system-modify --storage-system=<system> --multipath-autoconfig=<enabled|disabled>.
Troubleshooting
A device shows fewer paths than expected
Work down this list in order.
- Confirm what the topology should be. On a shared-JBOD HA pair, one path per appliance per disk is correct -- see Multipathing and scale-up HA pools. Only expect two paths from one appliance if you have cabled two routes from that appliance to the enclosure.
- Check the disk reports two ports. In the Physical Disk properties, a dual-ported disk populates both SAS Address Port0 and SAS Address Port1. If only one is populated, the disk is single-ported and no amount of configuration will produce a second path -- and if it is in a shared JBOD, it should not be.
- Check the path states.
multipath -lllists each path with its state. A path present but notactive ready runningis a live fault -- a failing cable, expander port or drive port -- rather than a configuration problem. - Look for connectivity errors on the drive. QuantaStor surfaces the SMART health assessment on the Physical Disk, and a high non-medium error count there is reported explicitly as indicating possible connectivity problems. A drive accumulating those on one port is a good candidate for the missing path.
- Rescan. If a route was restored, run a disk scan with Rescan Multipath Configuration checked so the map is rebuilt.
No multipath devices appear at all
- Check
multipathdis running --systemctl status multipathd. If it is stopped, check whether thezdblacklist rule is still present in/etc/multipath.conf; QuantaStor deliberately stops the service when that rule is missing, and every multipath device disappears as a result. See The multipath.conf file. - Check the model is configured. Open the Multi-path Device Configurator and click Scan. If the model appears in the scan results but not in the grid, multipathing is not enabled for it -- add it. If it does not appear in the scan results either, the device is not reporting multipath capability, so check the INQUIRY MultiP bit as shown in Which devices are multipath capable.
- Check Multipath Config Mode is Enabled on the storage system, so that newly attached models are picked up automatically.
- Check the device is not blocklisted in
/etc/quantastor/qs_device_blocklist.conf.
Deprecated option warnings from the multipath tools
The multipath tools log a warning for each use of the deprecated getuid_callout option in the per-array devices entries of the shipped multipath.conf, on every invocation:
/etc/multipath.conf line 124: ignoring deprecated option "getuid_callout"
The option is ignored, and the tools derive the device identifier themselves. The warnings are noise rather than a fault, and they do not indicate a misconfiguration you need to correct -- in particular, do not edit the file to silence them.
Related pages
- Physical Disks/Devices -- the Physical Disk object, its properties, and the disk scan
- Hardware Controllers & Enclosures -- controllers, enclosures, slot mapping and physically identifying a drive
- Storage Pools -- creating and managing pools on multipath devices
- HA Cluster Setup (JBODs) -- scale-up high availability on a shared SAS JBOD
- HA Cluster Setup (external SAN) -- scale-up high availability on SAN-attached storage
- QuantaStor Configuration Files -- the files QuantaStor manages, including
multipath.conf,qs_device_link.confandqs_device_blocklist.conf - QuantaStor CLI Command Reference -- full argument lists for the commands on this page
Verified against QuantaStor 6.9.0.