Proxmox Storage Plugin: Difference between revisions

From OSNEXUS Online Documentation Site
Jump to navigation Jump to search
m Credentials section: state the storage property without spelling the full password-file path; exact path remains in Troubleshooting where it is needed
m Plugin 1.1.1: install with apt (multipath-tools dependency), no manual multipath config, least-privilege role adds PendingOperationRequest:create, separate networks / ARP note for multipath
 
(One intermediate revision by the same user not shown)
Line 13: Line 13:
| [[#Installation|Installation]] || Installing, upgrading, and removing the plugin package
| [[#Installation|Installation]] || Installing, upgrading, and removing the plugin package
|-
|-
| [[#Configuration|Configuration]] || Adding the storage, all configuration options, HA pools, SSL
| [[#Configuration|Configuration]] || Adding the storage, all configuration options, API tokens, HA pools, multipath, SSL
|-
|-
| [[#Container (LXC) support|Container (LXC) support]] || Running container root filesystems on QuantaStor volumes
| [[#Container (LXC) support|Container (LXC) support]] || Running container root filesystems on QuantaStor volumes
Line 48: Line 48:
* '''Data plane''' -- guest I/O flows over iSCSI (port 3260) between the Proxmox node's initiator and the QuantaStor target. The plugin drives the standard {{Code|1=open-iscsi}} initiator with {{Code|1=iscsiadm}}: it discovers the target, logs in, and waits for the device to appear before handing it to Proxmox.
* '''Data plane''' -- guest I/O flows over iSCSI (port 3260) between the Proxmox node's initiator and the QuantaStor target. The plugin drives the standard {{Code|1=open-iscsi}} initiator with {{Code|1=iscsiadm}}: it discovers the target, logs in, and waits for the device to appear before handing it to Proxmox.


Each storage volume is exported through its own dedicated iSCSI target, always presented at LUN 0. The plugin resolves devices through stable {{Code|1=/dev/disk/by-path/ip-<portal>:<port>-iscsi-<target-iqn>-lun-0}} paths rather than ephemeral {{Code|1=/dev/sdX}} names. QEMU attaches the raw block device directly for VM disks; for containers, Proxmox formats the same block device with ext4 and mounts it as the container root filesystem.
Each storage volume is exported through its own dedicated iSCSI target, always presented at LUN 0. The plugin resolves devices through stable {{Code|1=/dev/disk/by-path/ip-<portal>:<port>-iscsi-<target-iqn>-lun-0}} paths rather than ephemeral {{Code|1=/dev/sdX}} names; with [[#Multipath|multipath]] enabled, it logs in on every portal instead and resolves the volume's device-mapper multipath device. QEMU attaches the raw block device directly for VM disks; for containers, Proxmox formats the same block device with ext4 and mounts it as the container root filesystem.


On first activation the plugin registers each Proxmox node with QuantaStor as an iSCSI host entry, using the node's initiator IQN from {{Code|1=/etc/iscsi/initiatorname.iscsi}}. Before a volume is used on a node, the plugin assigns that host to the volume's access control list; when the volume is released, the assignment is removed. Volumes are therefore only visible to the nodes actively using them.
On first activation the plugin registers each Proxmox node with QuantaStor as an iSCSI host entry, using the node's initiator IQN from {{Code|1=/etc/iscsi/initiatorname.iscsi}}. Before a volume is used on a node, the plugin assigns that host to the volume's access control list; when the volume is released, the assignment is removed. Volumes are therefore only visible to the nodes actively using them.
Line 86: Line 86:
=== QuantaStor requirements ===
=== QuantaStor requirements ===


* A QuantaStor system with a [[Storage Pools|storage pool]] to provision from. Use a current QuantaStor release; older builds refuse to delete freshly created snapshots for several minutes and can wedge volumes on template conversion (see [[#Troubleshooting|Troubleshooting]], err=493).
* A QuantaStor system with a [[Storage Pools|storage pool]] to provision from. QuantaStor 6.9 is the recommended release; builds without its fixes refuse to delete freshly created snapshots for several minutes and can wedge volumes on template conversion (see [[#Troubleshooting|Troubleshooting]], err=493).
* Network reachability from every Proxmox node to the appliance on TCP 8153 (REST API) and TCP 3260 (iSCSI).
* Network reachability from every Proxmox node to the appliance on TCP 8153 (REST API) and TCP 3260 (iSCSI).
* No configuration is required on the QuantaStor side beyond the pool: the plugin creates volumes, host entries, and access control assignments itself.
* No configuration is required on the QuantaStor side beyond the pool: the plugin creates volumes, host entries, and access control assignments itself.
Line 92: Line 92:
=== API credentials ===
=== API credentials ===


The plugin authenticates to the QuantaStor REST API with a username and password. The {{Code|1=admin}} account works, but we recommend creating a dedicated user so the credentials stored on the Proxmox cluster carry only the permissions the plugin needs.
The plugin authenticates to the QuantaStor REST API with a username and password, or with an application token in the same two fields (see [[#Authenticating with an API token|Authenticating with an API token]]). The {{Code|1=admin}} account works, but we recommend creating a dedicated user so the credentials stored on the Proxmox cluster carry only the permissions the plugin needs.


[[File:pve_qs_role_create.png|thumb|right|570px|The Create Role dialog: filter the object types, select the rows, and apply the System permission scope to the selected operations.]]
[[File:pve_qs_role_create.png|thumb|right|570px|The Create Role dialog: filter the object types, select the rows, and apply the System permission scope to the selected operations.]]
Line 107: Line 107:
* '''Storage Volume ACL''' -- add, remove
* '''Storage Volume ACL''' -- add, remove
* '''Host''' -- add, addInitiator, modify, remove, assignVolume, unassignVolume
* '''Host''' -- add, addInitiator, modify, remove, assignVolume, unassignVolume
* '''Pending Operation Request''' -- create (deletes are submitted as a pending operation, so a role without it cannot delete volumes or snapshots)


The role can be built in the web interface, or in one command with <code>[[QuantaStor CLI Command Reference#role-add|qs role-add]]</code> followed by <code>[[QuantaStor CLI Command Reference#user-add|qs user-add]]</code>:
The role can be built in the web interface, or in one command with <code>[[QuantaStor CLI Command Reference#role-add|qs role-add]]</code> followed by <code>[[QuantaStor CLI Command Reference#user-add|qs user-add]]</code>:
Line 118: Line 119:
StorageVolumeAcl:add:system,StorageVolumeAcl:remove:system,\
StorageVolumeAcl:add:system,StorageVolumeAcl:remove:system,\
Host:add:system,Host:addInitiator:system,Host:modify:system,Host:remove:system,\
Host:add:system,Host:addInitiator:system,Host:modify:system,Host:remove:system,\
Host:assignVolume:system,Host:unassignVolume:system"
Host:assignVolume:system,Host:unassignVolume:system,\
PendingOperationRequest:create:system"


qs user-add --name=pve-plugin --password=<password> --role=pve-plugin-role
qs user-add --name=pve-plugin --password=<password> --role=pve-plugin-role
Line 130: Line 132:


<pre style="font-size: smaller">
<pre style="font-size: smaller">
dpkg -i pve-storage-quantastor_1.0.0-1_all.deb
apt install ./pve-storage-quantastor_1.1.1-1_all.deb
</pre>
</pre>
Use {{Code|1=apt}} rather than {{Code|1=dpkg -i}}: it also installs the packages the plugin depends on ({{Code|1=open-iscsi}}, {{Code|1=multipath-tools}}). {{Code|1=dpkg -i}} does not, and on a node that lacks one it stops with "dependency problems - leaving unconfigured"; run {{Code|1=apt -f install}} to finish.


The package automatically enables the iSCSI initiator service ({{Code|1=iscsid}}), registers the storage type with the Proxmox services, and adds the QuantaStor panel to the web interface. Refresh the browser after installing and the '''QuantaStor''' entry appears in the Add Storage menu.
The package automatically enables the iSCSI initiator service ({{Code|1=iscsid}}), registers the storage type with the Proxmox services, and adds the QuantaStor panel to the web interface. Refresh the browser after installing and the '''QuantaStor''' entry appears in the Add Storage menu.
Line 139: Line 143:
=== Upgrades ===
=== Upgrades ===


Install the new package version with {{Code|1=dpkg -i}} on each node. Existing storage definitions, running guests, and iSCSI sessions are unaffected; the Proxmox services are restarted automatically by the package.
Install the new package version with {{Code|1=apt install ./<package>.deb}} on each node. From 1.0.0 or later, install the current package directly; there are no intermediate versions to step through and no migration steps. Existing storage definitions, running guests, and iSCSI sessions are unaffected; the Proxmox services are restarted automatically by the package.
 
'''Downgrading needs one step first if multipath is on.''' Older releases do not know the {{Code|1=multipath}} option. Before installing a 1.0.x package over 1.1.0 or later, stop the guests on every storage that uses multipath and turn it off:
 
<pre style="font-size: smaller">
pvesm set <storage-id> --delete multipath
</pre>


=== Uninstalling ===
=== Uninstalling ===
Line 166: Line 176:
| Username || QuantaStor API user.
| Username || QuantaStor API user.
|-
|-
| Password || QuantaStor API password. Stored in a root-only file on the cluster filesystem, never in {{Code|1=storage.cfg}}.
| Password || QuantaStor API password, or {{Code|1=QSAPPTKN:<token>}} for an application token. Stored in a root-only file on the cluster filesystem, never in {{Code|1=storage.cfg}}.
|-
|-
| Pool || QuantaStor storage pool to provision volumes from. Type a pool name, or click the magnifier button to connect to the appliance and pick from its active pools.
| Pool || QuantaStor storage pool to provision volumes from. Type a pool name, or click the magnifier button to connect to the appliance and pick from its active pools.
Line 175: Line 185:
|-
|-
| iSCSI Portal ''(Advanced)'' || Portal address for iSCSI logins. Defaults to the API Host when left blank.
| iSCSI Portal ''(Advanced)'' || Portal address for iSCSI logins. Defaults to the API Host when left blank.
|-
| Multipath ''(Advanced)'' || Log in on every portal QuantaStor advertises and use the multipath device. Off by default; see [[#Multipath|Multipath]] for the prerequisites.
|-
|-
| SSL Verify ''(Advanced)'' || Verify the appliance's TLS certificate against the node's system CA store. Leave off for self-signed certificates.
| SSL Verify ''(Advanced)'' || Verify the appliance's TLS certificate against the node's system CA store. Leave off for self-signed certificates.
Line 226: Line 238:
|-
|-
| {{Code|1=api_port}} || no || 8153 || QuantaStor REST API port
| {{Code|1=api_port}} || no || 8153 || QuantaStor REST API port
|-
| {{Code|1=multipath}} || no || 0 || Use every advertised portal through DM-MPIO (see [[#Multipath|Multipath]])
|-
|-
| {{Code|1=ssl_verify}} || no || 0 || Verify the appliance TLS certificate (see [[#SSL verification|SSL verification]])
| {{Code|1=ssl_verify}} || no || 0 || Verify the appliance TLS certificate (see [[#SSL verification|SSL verification]])
Line 247: Line 261:


If the stored password is missing or empty on a node, every plugin operation on that node fails with an explicit error naming the affected file and the {{Code|1=pvesm set}} command that repairs it.
If the stored password is missing or empty on a node, every plugin operation on that node fails with an explicit error naming the affected file and the {{Code|1=pvesm set}} command that repairs it.
=== Authenticating with an API token ===
Instead of a user's password, the plugin can authenticate with a QuantaStor user application token, so the Proxmox cluster never holds a login that also works in the web interface. No plugin option is involved: enter the token in the same two fields.
# On the appliance, create a token for the account the plugin uses -- ideally the dedicated user from [[#API credentials|API credentials]]: <pre style="font-size: smaller">qs user-app-token-add --user=pve-plugin --name=pve-cluster</pre> The result shows an '''app key''' (a UUID, used as the username) and a 64-character '''access token''' (used as the password). Copy the access token now; it is not shown again.
# Add or edit the storage with '''Username''' set to the app key and '''Password''' set to {{Code|1=QSAPPTKN:}} followed by the access token. The token name also works in place of the app key.
Everything else is unchanged: the pool scan, {{Code|1=pvesm add}}, and every volume operation run under the token, and the credential is stored exactly like a password. A token inherits the role of its user, so pair it with a least-privilege user rather than {{Code|1=admin}}, and do not mark it read-only -- the plugin creates and deletes volumes. To rotate a token, delete it with {{Code|1=qs user-app-token-delete}}, create a new one, and run {{Code|1=pvesm set <storage-id> --password 'QSAPPTKN:<new token>'}}.


=== SSL verification ===
=== SSL verification ===
Line 277: Line 300:


Nothing else changes -- the plugin does not need to be told the pool is HA, and no extra options are required. During a controller failover, guests pause rather than fail; see [[#Known limitations|Known limitations]] for the measured behavior and the one initiator setting that must not be changed.
Nothing else changes -- the plugin does not need to be told the pool is HA, and no extra options are required. During a controller failover, guests pause rather than fail; see [[#Known limitations|Known limitations]] for the measured behavior and the one initiator setting that must not be changed.
=== Multipath ===
When QuantaStor serves iSCSI on more than one network port, the storage can use every portal at once and keep guest I/O running when one path fails. Multipath is off by default.
With it on, the plugin logs each volume in on every portal QuantaStor advertises for it, builds a device-mapper multipath map for that one volume, and gives guests the map ({{Code|1=/dev/disk/by-id/dm-uuid-mpath-<wwid>}}) instead of a single-path device. Deactivating, deleting, rolling back, templating, and cloning a volume tear down or bring up every path, and an online resize rescans every path and grows the map.
Nothing needs setting up on the nodes. The plugin package depends on {{Code|1=multipath-tools}} and installs this device section for QuantaStor volumes as {{Code|1=/etc/multipath/conf.d/quantastor.conf}}:
<pre style="font-size: smaller">
devices {
    device {
        vendor              "OSNEXUS"
        product              "QUANTASTOR"
        path_grouping_policy "multibus"
        path_checker        "tur"
        failback            "immediate"
        no_path_retry        24
    }
}
</pre>
Keep the {{Code|1=no_path_retry}} setting if you edit the file (local edits are kept across upgrades). multipathd shortens each iSCSI session's recovery timeout to 5 seconds so that a dead path is dropped quickly; without queueing, a moment when '''every''' path is down -- an HA pool failover, a switch reboot -- fails guest I/O instead of pausing it. With this setting the map holds I/O for up to about two minutes until a path returns. With the Debian defaults, multipathd only claims the volumes the plugin registers for storages with multipath on, never local disks or other storage, so the section has no effect on storages without multipath. A copy of this section added to {{Code|1=/etc/multipath.conf}} for plugin 1.1.0 is now redundant and can be removed.
Turn multipath on with the '''Multipath''' checkbox in the dialog's advanced options, {{Code|1=--multipath 1}} on {{Code|1=pvesm add}}, or on an existing storage:
<pre style="font-size: smaller">
pvesm set <storage-id> --multipath 1
</pre>
Change the setting only while no guest on that storage is running. Then start a guest and confirm that its volume has one map with a path per portal, and that QEMU opened the map:
<pre style="font-size: smaller">
multipath -ll
qm showcmd <vmid> | grep -o 'dm-uuid-mpath-[0-9a-f]*'
</pre>
Expect an {{Code|1=OSNEXUS,QUANTASTOR}} map with one {{Code|1=active ready}} line per portal. When a path fails, I/O pauses for about 15 seconds while the path is failed out, then continues on the remaining paths; the path is reinstated automatically when it returns.
'''Give each path its own network''' -- separate subnets, ideally separate switches or VLANs -- so one failure takes out one path. If a node's iSCSI NICs share one layer-2 segment, Linux can answer ARP for one NIC's address on the other, and losing one NIC can then interrupt both paths: guests keep running, but I/O pauses until the path recovers (measured 43 s with two NICs on one segment). On such nodes set {{Code|1=net.ipv4.conf.all.arp_ignore = 1}} and {{Code|1=net.ipv4.conf.all.arp_announce = 2}} in a file under {{Code|1=/etc/sysctl.d/}} and run {{Code|1=sysctl --system}}; the same failure then keeps one path (measured 15 s pause).
QuantaStor advertises every port that has iSCSI enabled, including ports on networks the Proxmox nodes are not on. The plugin checks each advertised portal with a 3-second TCP connection and skips the ones that do not answer, logging a "not reachable on TCP, skipped" warning, rather than stalling activation for about two minutes per unreachable portal. To keep the warning out of the log, enable iSCSI only on the ports the Proxmox nodes use: <code>[[QuantaStor CLI Command Reference#network-port-modify|qs np-modify]] --port=<port> --iscsi-enable=false</code>. A path that is down when a volume is activated joins the next time the volume is activated.


=== Verifying the setup ===
=== Verifying the setup ===
Line 305: Line 370:
* '''Container templates ({{Code|1=vztmpl}}) cannot be stored on this storage''' -- template tarballs need a file-based storage such as {{Code|1=local}}. Create containers from a template on {{Code|1=local}} onto QuantaStor storage. This mirrors Ceph RBD, where template storage is likewise not possible on block storage.
* '''Container templates ({{Code|1=vztmpl}}) cannot be stored on this storage''' -- template tarballs need a file-based storage such as {{Code|1=local}}. Create containers from a template on {{Code|1=local}} onto QuantaStor storage. This mirrors Ceph RBD, where template storage is likewise not possible on block storage.
* '''Containers cannot live-migrate''' -- migration is restart mode, a Proxmox-wide constraint for containers on block storage (the same applies to RBD and LVM-thin).
* '''Containers cannot live-migrate''' -- migration is restart mode, a Proxmox-wide constraint for containers on block storage (the same applies to RBD and LVM-thin).
* '''Proxmox HA fencing with mounted container filesystems has not been explicitly validated.''' Proxmox's configuration locking prevents two nodes from mounting the same LUN during normal operation, but fence-triggered recovery of a node running containers on QuantaStor storage has not been exercised.
* '''Proxmox HA fencing with mounted container filesystems is safe under Proxmox's default watchdog fencing.''' Validated on a two-node cluster with a corosync QDevice by partitioning the node running a container from the cluster while its storage path stayed reachable: the partitioned node lost quorum, its watchdog reset it, and only then did the surviving node attach and mount the volume, with no window in which both nodes wrote to it. The plugin provides no storage-level exclusion, so this rests on Proxmox's watchdog timing -- do not shorten the watchdog or fence-delay defaults.


== Usage ==
== Usage ==
Line 339: Line 404:
# A password containing shell metacharacters was mangled by an unquoted {{Code|1=pvesm add}} -- re-set it single-quoted.
# A password containing shell metacharacters was mangled by an unquoted {{Code|1=pvesm add}} -- re-set it single-quoted.


'''{{Code|1=Failed to delete storage volume ... [err=493]}}''' on destroy or snapshot delete -- the backing volume was busy when the appliance tried to destroy it. Current QuantaStor releases eliminate the known causes; on older appliance builds, a delete of a freshly created snapshot can be refused for a few minutes. If a failed snapshot delete leaves the guest locked, unlock it ({{Code|1=qm unlock <vmid>}} or {{Code|1=pct unlock <ctid>}}) and retry after a few minutes. A persistently failing delete on a current release warrants a support ticket with {{Code|1=/var/log/qs/qs_service.log}} from around the failure.
'''{{Code|1=Failed to delete storage volume ... [err=493]}}''' on destroy or snapshot delete -- the backing volume was busy when the appliance tried to destroy it. Current QuantaStor releases eliminate the known causes; on older appliance builds, a delete of a freshly created snapshot can be refused for a few minutes. The plugin retries a refused snapshot delete for about three minutes, and the task log shows each attempt. If it still fails, the guest is left locked: unlock it ({{Code|1=qm unlock <vmid>}} or {{Code|1=pct unlock <ctid>}}) and retry after a few minutes. A persistently failing delete on a current release warrants a support ticket with {{Code|1=/var/log/qs/qs_service.log}} from around the failure.


'''Container fails to start with {{Code|1=run_buffer: ... Script exited with status 255}}''' -- check that the storage still advertises the Container content type. Proxmox's pre-start hook refuses to start a container whose root storage lacks {{Code|1=rootdir}}, and the real error is swallowed. Restore with {{Code|1=pvesm set <storage-id> --content images,rootdir}}.
'''Container fails to start with {{Code|1=run_buffer: ... Script exited with status 255}}''' -- check that the storage still advertises the Container content type. Proxmox's pre-start hook refuses to start a container whose root storage lacks {{Code|1=rootdir}}, and the real error is swallowed. Restore with {{Code|1=pvesm set <storage-id> --content images,rootdir}}.
Line 353: Line 418:
== Known limitations ==
== Known limitations ==


* '''QuantaStor HA pool failover pauses guest I/O for roughly 30-45 seconds.''' When an HA pool moves between controllers, guests on that storage block until the pool is imported on the other controller, then continue with no errors and no data loss -- measured guest-visible pauses were 26-34 seconds, independent of how many volumes the pool serves. Guests and their filesystems ride through it without tuning: the initiator holds queued I/O while the session recovers, so the guest never sees an error. '''Do not lower {{Code|1=node.session.timeo.replacement_timeout}}''' in {{Code|1=/etc/iscsi/iscsid.conf}}: its 120-second default is the budget that makes the pause survivable, and multipath tuning advice that shortens it does not apply here because there is no alternate path. This is QuantaStor pool failover, separate from Proxmox's own HA -- for Proxmox HA fencing see [[#Container (LXC) support|Container (LXC) support]].
* '''QuantaStor HA pool failover pauses guest I/O for roughly 30-45 seconds.''' When an HA pool moves between controllers, guests on that storage block until the pool is imported on the other controller, then continue with no errors and no data loss -- measured guest-visible pauses were 26-34 seconds, independent of how many volumes the pool serves. Guests and their filesystems ride through it without tuning: the initiator holds queued I/O while the session recovers, so the guest never sees an error. '''Do not lower {{Code|1=node.session.timeo.replacement_timeout}}''' in {{Code|1=/etc/iscsi/iscsid.conf}}: its 120-second default is the budget that makes the pause survivable, and multipath tuning advice that shortens it does not apply here because there is no alternate path. On a storage with [[#Multipath|multipath]] on, multipathd does shorten it, and the {{Code|1=no_path_retry}} queueing carries the pause instead. This is QuantaStor pool failover, separate from Proxmox's own HA -- for Proxmox HA fencing see [[#Container (LXC) support|Container (LXC) support]].
* '''Snapshot rollback''' is limited to the most recent snapshot and requires the guest stopped; rollback can wait up to a few minutes for the appliance to release the volume (see [[#Usage|Usage]]).
* '''Snapshot rollback''' is limited to the most recent snapshot and requires the guest stopped; rollback can wait up to a few minutes for the appliance to release the volume (see [[#Usage|Usage]]).
* '''Volumes cannot be shrunk''', only grown.
* '''Volumes cannot be shrunk''', only grown.
Line 374: Line 439:


----
----
<small>''Plugin release 1.0.0. Verified against QuantaStor 7.0.0.''</small>
<small>''Plugin release 1.1.1. Verified against QuantaStor 6.9.0.''</small>

Latest revision as of 20:31, 20 October 2026

The QuantaStor Proxmox VE storage plugin integrates QuantaStor as a native iSCSI block storage backend for Proxmox VE virtual machines and LXC containers. This page covers installing the plugin, configuring a QuantaStor-backed storage in Proxmox, and the operational behavior to expect.

Section Purpose
Overview What the plugin is and where to get it
Architecture How Proxmox, the plugin, and the appliance interact
Requirements Supported Proxmox VE versions, QuantaStor requirements, credentials
Installation Installing, upgrading, and removing the plugin package
Configuration Adding the storage, all configuration options, API tokens, HA pools, multipath, SSL
Container (LXC) support Running container root filesystems on QuantaStor volumes
Usage Snapshots, templates, clones, resize, and migration behavior
Troubleshooting The most common errors and their fixes
Known limitations Constraints to know before deploying
Reporting issues Where to report bugs and ask for help

Overview

The plugin registers a first-class storage type named quantastor in Proxmox VE. It appears in the Proxmox web interface under Datacenter → Storage → Add like any built-in storage type, and it works with the standard pvesm, qm, and pct tooling. It does not patch or replace any Proxmox source files, so Proxmox package upgrades do not break it.

Each Proxmox disk is a dedicated QuantaStor storage volume exported over iSCSI. The plugin provisions volumes through the QuantaStor REST API and logs into their iSCSI targets on the Proxmox node, handing Proxmox a stable local block device. Virtual machine disks and container root filesystems are both supported.

The plugin is open source and distributed as a Debian package:

The README in the repository carries the full module reference and developer documentation; this page is the operator-facing guide.

Architecture

QuantaStor Proxmox VE plugin architecture: the control plane provisions volumes over the REST API while the data plane serves guest I/O over iSCSI.

The plugin separates management traffic from storage traffic:

  • Control plane -- all provisioning operations (create, delete, snapshot, clone, resize, access control) are REST calls to the QuantaStor API over HTTPS on port 8153, authenticated with HTTP Basic credentials.
  • Data plane -- guest I/O flows over iSCSI (port 3260) between the Proxmox node's initiator and the QuantaStor target. The plugin drives the standard open-iscsi initiator with iscsiadm: it discovers the target, logs in, and waits for the device to appear before handing it to Proxmox.

Each storage volume is exported through its own dedicated iSCSI target, always presented at LUN 0. The plugin resolves devices through stable /dev/disk/by-path/ip-<portal>:<port>-iscsi-<target-iqn>-lun-0 paths rather than ephemeral /dev/sdX names; with multipath enabled, it logs in on every portal instead and resolves the volume's device-mapper multipath device. QEMU attaches the raw block device directly for VM disks; for containers, Proxmox formats the same block device with ext4 and mounts it as the container root filesystem.

On first activation the plugin registers each Proxmox node with QuantaStor as an iSCSI host entry, using the node's initiator IQN from /etc/iscsi/initiatorname.iscsi. Before a volume is used on a node, the plugin assigns that host to the volume's access control list; when the volume is released, the assignment is removed. Volumes are therefore only visible to the nodes actively using them.

Volume naming

Volumes on the QuantaStor pool follow the standard Proxmox naming convention, so the mapping between a Proxmox disk and its QuantaStor volume is direct:

Type Name pattern Example
VM or container disk vm-<vmid>-disk-<N> vm-100-disk-0
Template base disk base-<vmid>-disk-<N> base-100-disk-0
Template snapshot template-base-<vmid>-disk-<N> template-base-100-disk-0
Snapshot <volume>_<snapshot> vm-100-disk-0_snap1

Requirements

Supported Proxmox VE versions

Proxmox VE version Supported Notes
9.2.x Yes Loads with a cosmetic "implementing an older storage API" log message; see Known limitations
9.1.x Yes Loads silently
8.4.x and older No The plugin does not load; the quantastor type will not appear

Nodes running 9.1 and 9.2 can be mixed in one cluster and storage operations work transparently across both. Do not include 8.x nodes -- any storage operation routed to one (for example a migration target) fails because the storage type is not registered there.

QuantaStor requirements

  • A QuantaStor system with a storage pool to provision from. QuantaStor 6.9 is the recommended release; builds without its fixes refuse to delete freshly created snapshots for several minutes and can wedge volumes on template conversion (see Troubleshooting, err=493).
  • Network reachability from every Proxmox node to the appliance on TCP 8153 (REST API) and TCP 3260 (iSCSI).
  • No configuration is required on the QuantaStor side beyond the pool: the plugin creates volumes, host entries, and access control assignments itself.

API credentials

The plugin authenticates to the QuantaStor REST API with a username and password, or with an application token in the same two fields (see Authenticating with an API token). The admin account works, but we recommend creating a dedicated user so the credentials stored on the Proxmox cluster carry only the permissions the plugin needs.

The Create Role dialog: filter the object types, select the rows, and apply the System permission scope to the selected operations.

Create a role that grants view access to everything plus the specific write operations the plugin performs, then create a user with that role (see User Add).

Navigation: Security → Role → Create (toolbar)
Navigation: Security → User → Add (toolbar)

The write operations the plugin needs are:

  • Storage Volume -- create, delete, modify, resize, createSnapshot, deleteSnapshot, rollback, clone
  • Storage Volume ACL -- add, remove
  • Host -- add, addInitiator, modify, remove, assignVolume, unassignVolume
  • Pending Operation Request -- create (deletes are submitted as a pending operation, so a role without it cannot delete volumes or snapshots)

The role can be built in the web interface, or in one command with qs role-add followed by qs user-add:

qs role-add --name=pve-plugin-role --desc="Role for the Proxmox VE storage plugin" \
  --permissions="*:view:system,\
StorageVolume:create:system,StorageVolume:delete:system,StorageVolume:modify:system,\
StorageVolume:resize:system,StorageVolume:createSnapshot:system,StorageVolume:deleteSnapshot:system,\
StorageVolume:rollback:system,StorageVolume:clone:system,\
StorageVolumeAcl:add:system,StorageVolumeAcl:remove:system,\
Host:add:system,Host:addInitiator:system,Host:modify:system,Host:remove:system,\
Host:assignVolume:system,Host:unassignVolume:system,\
PendingOperationRequest:create:system"

qs user-add --name=pve-plugin --password=<password> --role=pve-plugin-role

The full storage lifecycle -- storage activation, volume create, iSCSI access control, snapshot create and delete, and volume delete -- works under a user with exactly this permission set.

Installation

Install the plugin package on every node in the cluster. A node without the plugin cannot activate the storage, which breaks migration to that node and leaves the storage inactive there.

apt install ./pve-storage-quantastor_1.1.1-1_all.deb

Use apt rather than dpkg -i: it also installs the packages the plugin depends on (open-iscsi, multipath-tools). dpkg -i does not, and on a node that lacks one it stops with "dependency problems - leaving unconfigured"; run apt -f install to finish.

The package automatically enables the iSCSI initiator service (iscsid), registers the storage type with the Proxmox services, and adds the QuantaStor panel to the web interface. Refresh the browser after installing and the QuantaStor entry appears in the Add Storage menu.

The package installs dpkg triggers that re-apply the web interface integration whenever the pve-manager or pve-storage packages are upgraded, so routine Proxmox updates need no plugin reinstall.

Upgrades

Install the new package version with apt install ./<package>.deb on each node. From 1.0.0 or later, install the current package directly; there are no intermediate versions to step through and no migration steps. Existing storage definitions, running guests, and iSCSI sessions are unaffected; the Proxmox services are restarted automatically by the package.

Downgrading needs one step first if multipath is on. Older releases do not know the multipath option. Before installing a 1.0.x package over 1.1.0 or later, stop the guests on every storage that uses multipath and turn it off:

pvesm set <storage-id> --delete multipath

Uninstalling

apt remove pve-storage-quantastor

Removing the package does not remove storage entries from /etc/pve/storage.cfg. Delete or disable QuantaStor storage entries first if the plugin is being removed permanently; otherwise those entries error once the plugin is gone.

Configuration

Adding the storage in the web interface

Navigation: Datacenter → Storage → Add (button) → QuantaStor
The Add: QuantaStor dialog. The magnifier button next to Pool scans the appliance and lists its active pools.
Field Description
ID Proxmox storage identifier. Auto-fills with the first free qs-storage-N; edit as needed.
API Host QuantaStor appliance IP address or hostname. For an HA pool, use the failover group's virtual interface -- see QuantaStor HA storage pools.
Username QuantaStor API user.
Password QuantaStor API password, or QSAPPTKN:<token> for an application token. Stored in a root-only file on the cluster filesystem, never in storage.cfg.
Pool QuantaStor storage pool to provision volumes from. Type a pool name, or click the magnifier button to connect to the appliance and pick from its active pools.
Content Disk image for VM disks, Container for LXC root filesystems. Both are selected by default.
Nodes Restrict the storage to specific Proxmox nodes; leave empty for all nodes.
iSCSI Portal (Advanced) Portal address for iSCSI logins. Defaults to the API Host when left blank.
Multipath (Advanced) Log in on every portal QuantaStor advertises and use the multipath device. Off by default; see Multipath for the prerequisites.
SSL Verify (Advanced) Verify the appliance's TLS certificate against the node's system CA store. Leave off for self-signed certificates.

API Host, Username, and Pool are fixed once the storage is created. The other fields can be changed later through the Edit dialog or pvesm set.

The same dialog with Advanced ticked: the iSCSI portal defaults to the API Host, and SSL verification is off for self-signed certificates.

Adding the storage from the command line

pvesm add quantastor qs-storage-1 \
  --api_host 10.0.0.20 \
  --username admin \
  --password '<password>' \
  --pool_id <pool-name-or-uuid> \
  --content images,rootdir

Single-quote the password so the shell does not interpret special characters ($, !, backticks).

The resulting /etc/pve/storage.cfg entry contains no password:

quantastor: qs-storage-1
        api_host 10.0.0.20
        pool_id <pool-name-or-uuid>
        username admin
        content images,rootdir
        shared 1
        ssl_verify 0

All options:

Option Required Default Description
api_host yes -- QuantaStor appliance IP or hostname (HA pools: the failover group virtual interface)
username yes -- QuantaStor API user
password on create -- QuantaStor API password; stored outside storage.cfg
pool_id yes -- QuantaStor pool name or UUID
content no images images (VM disks), rootdir (containers), or both
portal no api_host iSCSI portal address when it differs from the API endpoint
api_port no 8153 QuantaStor REST API port
multipath no 0 Use every advertised portal through DM-MPIO (see Multipath)
ssl_verify no 0 Verify the appliance TLS certificate (see SSL verification)
sparse no 1 Thin provisioning; set 0 for thick (see below)
shared no 1 Set automatically on create; marks the storage cluster-shared so migration works
nodes no all Restrict to specific Proxmox nodes

Change settings with pvesm set (or the Edit dialog), not by removing and re-adding the storage. The remove path deletes the stored password file cluster-wide, and a re-add that fails to capture the password leaves every operation failing until the password is set again.

Thin and thick provisioning

Volumes are thin provisioned by default: they consume pool space as data is written, not their full size at creation. Set sparse 0 on the storage for thick provisioning, where each volume reserves 100% of its size in the pool at creation time. Note this default is the opposite of Proxmox's built-in ZFS storage types, where sparse defaults to off.

How the plugin stores credentials

The API password is never written to storage.cfg. It is kept in a root-only file under the cluster filesystem's private area (/etc/pve/priv), following the same convention Proxmox uses for CIFS and Proxmox Backup Server credentials, and it replicates to every node automatically.

If the stored password is missing or empty on a node, every plugin operation on that node fails with an explicit error naming the affected file and the pvesm set command that repairs it.

Authenticating with an API token

Instead of a user's password, the plugin can authenticate with a QuantaStor user application token, so the Proxmox cluster never holds a login that also works in the web interface. No plugin option is involved: enter the token in the same two fields.

  1. On the appliance, create a token for the account the plugin uses -- ideally the dedicated user from API credentials:
    qs user-app-token-add --user=pve-plugin --name=pve-cluster
    The result shows an app key (a UUID, used as the username) and a 64-character access token (used as the password). Copy the access token now; it is not shown again.
  2. Add or edit the storage with Username set to the app key and Password set to QSAPPTKN: followed by the access token. The token name also works in place of the app key.

Everything else is unchanged: the pool scan, pvesm add, and every volume operation run under the token, and the credential is stored exactly like a password. A token inherits the role of its user, so pair it with a least-privilege user rather than admin, and do not mark it read-only -- the plugin creates and deletes volumes. To rotate a token, delete it with qs user-app-token-delete, create a new one, and run pvesm set <storage-id> --password 'QSAPPTKN:<new token>'.

SSL verification

By default (ssl_verify 0) the plugin accepts the appliance's certificate without verification, which suits the self-signed certificate QuantaStor ships with. To enable verification, install the QuantaStor CA certificate into the system trust store on every Proxmox node, then enable the option:

cp qs-ca.crt /usr/local/share/ca-certificates/
update-ca-certificates
pvesm set <storage-id> --ssl_verify 1

The value configured as API Host must appear in the certificate's Subject Alternative Name list -- verification checks the hostname as well as the trust chain, so a certificate without the configured IP address or hostname in its SAN entries fails even with the CA trusted.

QuantaStor HA storage pools

If the pool is a QuantaStor HA storage pool -- one owned by a storage pool high-availability group that fails over between two controllers -- configure the storage against the failover group's virtual interface, not against either controller's own address:

quantastor: qs-ha
        api_host 10.0.0.20        # the failover group's floating VIF
        portal 10.0.0.20          # the same VIF
        pool_id my-ha-pool
        username admin
        content images,rootdir
        shared 1

Both api_host and portal must be that virtual interface (see High-availability VIF Management). The address moves with the pool on failover, so the REST endpoint and the iSCSI portal follow the active controller on their own. Pointed at a controller's fixed address instead, the storage stops working the moment the pool moves to the other controller.

Nothing else changes -- the plugin does not need to be told the pool is HA, and no extra options are required. During a controller failover, guests pause rather than fail; see Known limitations for the measured behavior and the one initiator setting that must not be changed.

Multipath

When QuantaStor serves iSCSI on more than one network port, the storage can use every portal at once and keep guest I/O running when one path fails. Multipath is off by default.

With it on, the plugin logs each volume in on every portal QuantaStor advertises for it, builds a device-mapper multipath map for that one volume, and gives guests the map (/dev/disk/by-id/dm-uuid-mpath-<wwid>) instead of a single-path device. Deactivating, deleting, rolling back, templating, and cloning a volume tear down or bring up every path, and an online resize rescans every path and grows the map.

Nothing needs setting up on the nodes. The plugin package depends on multipath-tools and installs this device section for QuantaStor volumes as /etc/multipath/conf.d/quantastor.conf:

devices {
    device {
        vendor               "OSNEXUS"
        product              "QUANTASTOR"
        path_grouping_policy "multibus"
        path_checker         "tur"
        failback             "immediate"
        no_path_retry        24
    }
}

Keep the no_path_retry setting if you edit the file (local edits are kept across upgrades). multipathd shortens each iSCSI session's recovery timeout to 5 seconds so that a dead path is dropped quickly; without queueing, a moment when every path is down -- an HA pool failover, a switch reboot -- fails guest I/O instead of pausing it. With this setting the map holds I/O for up to about two minutes until a path returns. With the Debian defaults, multipathd only claims the volumes the plugin registers for storages with multipath on, never local disks or other storage, so the section has no effect on storages without multipath. A copy of this section added to /etc/multipath.conf for plugin 1.1.0 is now redundant and can be removed.

Turn multipath on with the Multipath checkbox in the dialog's advanced options, --multipath 1 on pvesm add, or on an existing storage:

pvesm set <storage-id> --multipath 1

Change the setting only while no guest on that storage is running. Then start a guest and confirm that its volume has one map with a path per portal, and that QEMU opened the map:

multipath -ll
qm showcmd <vmid> | grep -o 'dm-uuid-mpath-[0-9a-f]*'

Expect an OSNEXUS,QUANTASTOR map with one active ready line per portal. When a path fails, I/O pauses for about 15 seconds while the path is failed out, then continues on the remaining paths; the path is reinstated automatically when it returns.

Give each path its own network -- separate subnets, ideally separate switches or VLANs -- so one failure takes out one path. If a node's iSCSI NICs share one layer-2 segment, Linux can answer ARP for one NIC's address on the other, and losing one NIC can then interrupt both paths: guests keep running, but I/O pauses until the path recovers (measured 43 s with two NICs on one segment). On such nodes set net.ipv4.conf.all.arp_ignore = 1 and net.ipv4.conf.all.arp_announce = 2 in a file under /etc/sysctl.d/ and run sysctl --system; the same failure then keeps one path (measured 15 s pause).

QuantaStor advertises every port that has iSCSI enabled, including ports on networks the Proxmox nodes are not on. The plugin checks each advertised portal with a 3-second TCP connection and skips the ones that do not answer, logging a "not reachable on TCP, skipped" warning, rather than stalling activation for about two minutes per unreachable portal. To keep the warning out of the log, enable iSCSI only on the ports the Proxmox nodes use: qs np-modify --port=<port> --iscsi-enable=false. A path that is down when a volume is activated joins the next time the volume is activated.

Verifying the setup

The password file should exist on every cluster node, and the storage should activate:

ls -la /etc/pve/priv/storage/
pvesm status --storage <storage-id>

Expect the <storage-id>.pw file present on each node and the storage reported active with sizes populated. On the QuantaStor side, each Proxmox node appears as a host entry once the storage has been activated there -- verify with qs host-list or in the web interface under Hosts.

Container (LXC) support

LXC containers can place their root filesystem (and additional mount points) on QuantaStor storage. Enable the Container content type on the storage -- tick it in the dialog, or:

pvesm set <storage-id> --content images,rootdir

A container volume is the same raw iSCSI LUN as a VM disk. Proxmox's container stack formats the LUN with ext4, mounts it, and grows the filesystem on resize -- the plugin's job is only to provide the block device, the same model Proxmox uses for containers on Ceph RBD and LVM-thin storage.

The full container lifecycle is supported: create, start, stop, and destroy (privileged and unprivileged), snapshots of running containers (filesystems are frozen first, so snapshots are consistent), snapshot rollback (container stopped), online and offline resize, templates with linked or full clones, restart-mode migration between nodes, extra mount points (one dedicated LUN per mount point), and pct fstrim (freed space returns to the thin volume through iSCSI UNMAP).

Container limitations:

  • Container templates (vztmpl) cannot be stored on this storage -- template tarballs need a file-based storage such as local. Create containers from a template on local onto QuantaStor storage. This mirrors Ceph RBD, where template storage is likewise not possible on block storage.
  • Containers cannot live-migrate -- migration is restart mode, a Proxmox-wide constraint for containers on block storage (the same applies to RBD and LVM-thin).
  • Proxmox HA fencing with mounted container filesystems is safe under Proxmox's default watchdog fencing. Validated on a two-node cluster with a corosync QDevice by partitioning the node running a container from the cluster while its storage path stayed reachable: the partitioned node lost quorum, its watchdog reset it, and only then did the surviving node attach and mount the volume, with no window in which both nodes wrote to it. The plugin provides no storage-level exclusion, so this rests on Proxmox's watchdog timing -- do not shorten the watchdog or fence-delay defaults.

Usage

VM disks on QuantaStor storage behave like disks on any shared Proxmox block storage: create them when creating a guest or add them later, and the plugin provisions the backing volume on the pool automatically. The operations below have behavior worth knowing about.

Snapshots and rollback

Snapshots are QuantaStor volume snapshots taken through the API. Snapshots of running containers freeze the filesystem first, so they are filesystem-consistent.

Rollback requires the guest to be stopped, and only the most recent snapshot can be rolled back to (Proxmox enforces the stop; the plugin enforces the ordering). A rollback issued right after stopping a guest can take up to a few minutes: the appliance's session tracking releases the volume on a cycle of up to three minutes after the initiator disconnects, and the plugin waits for that release before asking the appliance to roll back. This wait is normal, not a hang.

Templates and clones

Converting a guest to a template renames its volumes to base-* and takes a template snapshot; clones are created from that snapshot on the appliance, so creating a clone does not copy the full volume up front. Destroy clones before destroying their template.

Resizing

Disks can be grown online -- the volume grows on the appliance, the plugin refreshes the kernel's view of the LUN, and the guest sees the new size without a restart. Shrinking is not supported.

Migration

Live and offline VM migration between nodes works over the shared storage: the same volume is activated on the destination and released on the source, with no data copied. Every node involved needs the plugin installed and the storage active -- see Troubleshooting if a migration fails.

Troubleshooting

The README troubleshooting section carries the full decision trees; these are the most common cases.

Authentication check failed ... [err=26] -- QuantaStor rejected the request's credentials. Check, in order:

  1. The password file exists and is non-empty on the node running the operation: ls -la /etc/pve/priv/storage/<storage-id>.pw. If missing, repair with pvesm set <storage-id> --password '<password>' from any node.
  2. The credentials are actually valid -- test with curl, which bypasses the plugin: curl -k "https://<api_host>:8153/qstorapi/storagePoolGet?storagePool=<pool>" -u admin:'<password>'
  3. A password containing shell metacharacters was mangled by an unquoted pvesm add -- re-set it single-quoted.

Failed to delete storage volume ... [err=493] on destroy or snapshot delete -- the backing volume was busy when the appliance tried to destroy it. Current QuantaStor releases eliminate the known causes; on older appliance builds, a delete of a freshly created snapshot can be refused for a few minutes. The plugin retries a refused snapshot delete for about three minutes, and the task log shows each attempt. If it still fails, the guest is left locked: unlock it (qm unlock <vmid> or pct unlock <ctid>) and retry after a few minutes. A persistently failing delete on a current release warrants a support ticket with /var/log/qs/qs_service.log from around the failure.

Container fails to start with run_buffer: ... Script exited with status 255 -- check that the storage still advertises the Container content type. Proxmox's pre-start hook refuses to start a container whose root storage lacks rootdir, and the real error is swallowed. Restore with pvesm set <storage-id> --content images,rootdir.

Storage shows active on some nodes and inactive on others -- the inactive node's error says which case it is: a "no password configured" message means the password file is missing there (repair as in err=26 above); a network or SSL error means that node cannot reach the API host on port 8153 -- check firewalls and routing, not the plugin.

Migration or qm move-disk to another node fails -- verify on the destination node that the plugin package is installed, pvesm status shows the storage, and the password file exists. If the plugin and storage are healthy, check for QEMU version skew between the nodes (pin the VM's machine type to the older node's version) and confirm the storage has shared 1 set.

Plugin ... is implementing an older storage API, an upgrade is recommended in the logs -- cosmetic; see Known limitations.

Plugin errors surface in the Proxmox task log of the operation that failed, and in the journal of the PVE service that ran it (journalctl -u pvedaemon, journalctl -u pvestatd).

Known limitations

  • QuantaStor HA pool failover pauses guest I/O for roughly 30-45 seconds. When an HA pool moves between controllers, guests on that storage block until the pool is imported on the other controller, then continue with no errors and no data loss -- measured guest-visible pauses were 26-34 seconds, independent of how many volumes the pool serves. Guests and their filesystems ride through it without tuning: the initiator holds queued I/O while the session recovers, so the guest never sees an error. Do not lower node.session.timeo.replacement_timeout in /etc/iscsi/iscsid.conf: its 120-second default is the budget that makes the pause survivable, and multipath tuning advice that shortens it does not apply here because there is no alternate path. On a storage with multipath on, multipathd does shorten it, and the no_path_retry queueing carries the pause instead. This is QuantaStor pool failover, separate from Proxmox's own HA -- for Proxmox HA fencing see Container (LXC) support.
  • Snapshot rollback is limited to the most recent snapshot and requires the guest stopped; rollback can wait up to a few minutes for the appliance to release the volume (see Usage).
  • Volumes cannot be shrunk, only grown.
  • All volumes present at iSCSI LUN 0 on their dedicated target; the lun field the QuantaStor API reports on volume objects is an internal value and does not reflect the wire LUN.
  • The "older storage API" log message on Proxmox VE 9.2 is cosmetic. The plugin declares storage API version 13 so that one build supports both 9.1 and 9.2; PVE 9.2 logs a recommendation to upgrade at every plugin scan. Behavior is identical on both versions. If the log noise is a problem, filter it with an rsyslog rule: :msg, contains, "implementing an older storage API" stop in /etc/rsyslog.d/.
  • Proxmox storage replication is not supported (pvesr); QuantaStor's own remote replication can protect the pool instead.

Reporting issues

Report bugs and feature requests through GitHub Issues on the plugin repository, or by email to eng-proxmox@osnexus.com. Include the Proxmox VE version, the plugin version, and the failing task's log output.

Related pages


Plugin release 1.1.1. Verified against QuantaStor 6.9.0.