Proxmox Storage Plugin: Difference between revisions

From OSNEXUS Online Documentation Site
Jump to navigation Jump to search
m Rewrite for plugin 1.0.0: native 'quantastor' storage type replaces the patch-based alpha; adds container support, HA pool guidance, verified least-privilege role recipe, new architecture diagram
Line 1: Line 1:
[[Category:integration_guide]]
[[Category:integration_guide]]
== Proxmox Storage Plugins ==
The QuantaStor Proxmox VE storage plugin integrates QuantaStor as a native iSCSI block storage backend for Proxmox VE virtual machines and LXC containers. This page covers installing the plugin, configuring a QuantaStor-backed storage in Proxmox, and the operational behavior to expect.
A Proxmox Storage Plugin is a Perl module that integrates an external storage system into Proxmox VE so it can be managed natively through the Proxmox UI and CLI.


* Storage plugins allow Proxmox to:
{| class="wikitable"
* Create, delete, and resize VM disks
! Section !! Purpose
* Report storage capacity and usage
|-
* Manage snapshots (where supported)
| [[#Overview|Overview]] || What the plugin is and where to get it
* Coordinate locks and concurrency across a cluster
|-
| [[#Architecture|Architecture]] || How Proxmox, the plugin, and the appliance interact
|-
| [[#Requirements|Requirements]] || Supported Proxmox VE versions, QuantaStor requirements, credentials
|-
| [[#Installation|Installation]] || Installing, upgrading, and removing the plugin package
|-
| [[#Configuration|Configuration]] || Adding the storage, all configuration options, HA pools, SSL
|-
| [[#Container (LXC) support|Container (LXC) support]] || Running container root filesystems on QuantaStor volumes
|-
| [[#Usage|Usage]] || Snapshots, templates, clones, resize, and migration behavior
|-
| [[#Troubleshooting|Troubleshooting]] || The most common errors and their fixes
|-
| [[#Known limitations|Known limitations]] || Constraints to know before deploying
|-
| [[#Reporting issues|Reporting issues]] || Where to report bugs and ask for help
|}


From Proxmox’s perspective, the plugin is the storage backend.
== Overview ==


== QuantaStor PVE Plugin ==
The plugin registers a first-class storage type named {{Code|1=quantastor}} in Proxmox VE. It appears in the Proxmox web interface under '''Datacenter → Storage → Add''' like any built-in storage type, and it works with the standard {{Code|1=pvesm}}, {{Code|1=qm}}, and {{Code|1=pct}} tooling. It does not patch or replace any Proxmox source files, so Proxmox package upgrades do not break it.
Github Repo: [https://github.com/OSNEXUS/pve-quantastor-plugin here]


=== Overview ===
Each Proxmox disk is a dedicated QuantaStor storage volume exported over iSCSI. The plugin provisions volumes through the QuantaStor REST API and logs into their iSCSI targets on the Proxmox node, handing Proxmox a stable local block device. Virtual machine disks and container root filesystems are both supported.


==== What the plugin does ====
The plugin is open source and distributed as a Debian package:
The QuantaStor Proxmox plugin uses ZFS over ISCSi protocol to provide block storage for PVE using QuantaStor APIs.


==== Supported Proxmox versions ====
* GitHub repository: [https://github.com/OSNEXUS/pve-quantastor-plugin OSNEXUS/pve-quantastor-plugin]
The plug-in supports PVE versions 8.4 and 9.1. Installation script allows a “patch” mode and a “full copy” mode. After an upgrade to a PVE system, users will need to reinstall the plugin. Major Minor version updates may break installation and will require developers to update the repository to generate the latest patch files.
* Releases: [https://github.com/OSNEXUS/pve-quantastor-plugin/releases pve-storage-quantastor .deb packages]


==== Supported backends ====
The [https://github.com/OSNEXUS/pve-quantastor-plugin/blob/master/README.md README] in the repository carries the full module reference and developer documentation; this page is the operator-facing guide.
The current version of the plugin extends the functionality of the ZFS over iSCSI storage configuration by adding another option for ISCSi provider. QuantaStor RestAPI calls are then used to manage block storage provisioning.


==== Current maturity ====
== Architecture ==
Plugin is currently in pre-release phase


=== Architecture ===
[[File:pve_plugin_architecture.png|thumb|right|800px|QuantaStor Proxmox VE plugin architecture: the control plane provisions volumes over the REST API while the data plane serves guest I/O over iSCSI.]]
==== How Proxmox interacts with the plugin ====
After installing the plugin, users can create ZFS over ISCSi storage configurations using QuantaStor as the ISCSi provider. Proxmox will then use the plugin’s QuantaStor API functionality to facilitate ISCSi connection to Proxmox node or cluster. QuantaStor creates and wires storage objects, hands Proxmox a block device and then steps out of the way. <br>
[[File:Base-diagram-proxmox-qs-storage-plugin.JPG|800px]] <br>
* '''Proxmox VE Layer''' (''Orange'') - User interface and storage API
* '''Plugin Layer''' (''Blue'') - QuantaStor plugin
** ZFSPlugin.pm (modified to support QuantaStor)
** QuantaStorPlugin.pm (LUN command handler)
* '''QuantaStor Layer''' (''Green'') - Storage management system
* '''Storage Layer''' (''Yellow'') - Actual ZFS pools and block devices
* '''VM Layer''' (''Purple'') - Virtual machines using the storage


==== External dependencies ====
The plugin separates management traffic from storage traffic:
After installation of the plugin on all nodes, there should be no other external dependencies required. You will need a QuantaStor storage server with a ZFS pool. See QuantaStor documentation for installation and storage pool setup.


=== Requirements ===
* '''Control plane''' -- all provisioning operations (create, delete, snapshot, clone, resize, access control) are REST calls to the QuantaStor API over HTTPS on port 8153, authenticated with HTTP Basic credentials.
==== Network requirements ====
* '''Data plane''' -- guest I/O flows over iSCSI (port 3260) between the Proxmox node's initiator and the QuantaStor target. The plugin drives the standard {{Code|1=open-iscsi}} initiator with {{Code|1=iscsiadm}}: it discovers the target, logs in, and waits for the device to appear before handing it to Proxmox.
A stable network connection is needed between Proxmox and QuantaStor nodes to ensure API call reliability and statistics gathering.


==== Permissions / API credentials ====
Each storage volume is exported through its own dedicated iSCSI target, always presented at LUN 0. The plugin resolves devices through stable {{Code|1=/dev/disk/by-path/ip-<portal>:<port>-iscsi-<target-iqn>-lun-0}} paths rather than ephemeral {{Code|1=/dev/sdX}} names. QEMU attaches the raw block device directly for VM disks; for containers, Proxmox formats the same block device with ext4 and mounts it as the container root filesystem.
Recommended to create a proxmox user/role in QuantaStor so you don’t have to use the admin credentials which are saved on the Proxmox systems. The user needs permissions that will allow it to provision hosts initiators, storage volumes, snapshots, and clones. Additionally at minimum view permissions for storage pools. <br>
[[File:Role-create-storage-pool.JPG|400px]] <br>
Copy over system monitor permission for viewing and then apply 'System' permissions for 'storagePool' operations. <br>
[[File:Role-create-storage-volume.JPG|400px]] <br>
Apply 'System' permissions for 'storageVolume' operations. <br>
[[File:Role-create-host.JPG|400px]] <br>
Apply 'System' permissions for 'host' operations. <br>
[[File:User-create-pve-plugin-user.JPG|400px]] <br>
Create a new user using the plugin role.<br>


=== Installation ===
On first activation the plugin registers each Proxmox node with QuantaStor as an iSCSI host entry, using the node's initiator IQN from {{Code|1=/etc/iscsi/initiatorname.iscsi}}. Before a volume is used on a node, the plugin assigns that host to the volume's access control list; when the volume is released, the assignment is removed. Volumes are therefore only visible to the nodes actively using them.
[https://github.com/OSNEXUS/pve-quantastor-plugin/blob/master/README.md QuantaStor PVE Plugin README] <br>
Installation instructions can be found in the README on the github repository. Plugin needs to be installed on all nodes in the cluster or storage configurations will break. Support for 2 installation modes: patch mode and full copy mode. Using patch mode is a more gentle approach that will add QuantaStor plugin patches to the existing PVE source files and requires the ‘patch’ cli dependency. Full copy mode will fully overwrite the PVE source files (for developement).


=== Upgrade ===
=== Volume naming ===
To upgrade the QuantaStor Proxmox VE storage plugin, re-run the installation script on each Proxmox node. This applies to both patch-level updates and full upgrades.


During a Proxmox VE upgrade, existing QuantaStor storage configurations may temporarily lose management functionality until the installation script is executed again. Reinstalling the plugin restores full integration. <br>
Volumes on the QuantaStor pool follow the standard Proxmox naming convention, so the mapping between a Proxmox disk and its QuantaStor volume is direct:


Currently supported Proxmox VE versions:
{| class="wikitable"
* 8.4.x
! Type !! Name pattern !! Example
* 9.1.x
|-
| VM or container disk || {{Code|1=vm-<vmid>-disk-<N>}} || {{Code|1=vm-100-disk-0}}
|-
| Template base disk || {{Code|1=base-<vmid>-disk-<N>}} || {{Code|1=base-100-disk-0}}
|-
| Template snapshot || {{Code|1=template-base-<vmid>-disk-<N>}} || {{Code|1=template-base-100-disk-0}}
|-
| Snapshot || {{Code|1=<volume>_<snapshot>}} || {{Code|1=vm-100-disk-0_snap1}}
|}


=== Uninstall / Rollback ===
== Requirements ==
The installation script supports a rollback mechanism using the <code>--rollback</code> option. When executed, the script restores previously backed-up Proxmox source files, effectively removing the plugin and returning the system to its pre-installation state.


A file-based backup of modified Proxmox source files is automatically created during each installation or upgrade to ensure safe rollback between releases.
=== Supported Proxmox VE versions ===


=== Configuration ===
{| class="wikitable"
==== Storage definitions in Proxmox ====
! Proxmox VE version !! Supported !! Notes
[[File:Base-storage-config-create-dialog.JPG|400px]]
|-
| 9.2.x || Yes || Loads with a cosmetic "implementing an older storage API" log message; see [[#Known limitations|Known limitations]]
|-
| 9.1.x || Yes || Loads silently
|-
| 8.4.x and older || No || The plugin does not load; the {{Code|1=quantastor}} type will not appear
|}


* '''ID''' – Storage configuration identifier.
Nodes running 9.1 and 9.2 can be mixed in one cluster and storage operations work transparently across both. Do not include 8.x nodes -- any storage operation routed to one (for example a migration target) fails because the storage type is not registered there.
* '''Portal''' – IPv4 iSCSI portal address.
* '''Pool''' – QuantaStor ZFS pool used to provision storage volumes.
* '''Block Size''' – Block size (in KB) used when creating ZFS volumes.
* '''Target''' – Required to be non-empty; currently not used by the plugin.
* '''QS Username''' – QuantaStor API user name.
* '''iSCSI Provider''' – Must be set to '''QuantaStor API''' to enable the QuantaStor storage plugin.
* '''Thin provision''' –
** <code>sparse=1</code>: Thin-provisioned volume with 0% space reserved.
** <code>sparse=0</code>: Thick-provisioned volume with 100% space reserved.
* '''QS Host''' – QuantaStor API endpoint. When SSL is enabled, the configured CA root certificate must include this value (IP address or hostname) in its SAN list.
* '''QS Password''' – QuantaStor API password (confirmation required).


==== Enable SSL Verification ====
=== QuantaStor requirements ===
To enable SSL verification for QuantaStor API communication, the QuantaStor Root CA certificate must be installed on each Proxmox node in the cluster.


Copy the QuantaStor Root CA certificate to the following location:
* A QuantaStor system with a [[Storage Pools|storage pool]] to provision from. Use a current QuantaStor release; older builds refuse to delete freshly created snapshots for several minutes and can wedge volumes on template conversion (see [[#Troubleshooting|Troubleshooting]], err=493).
<code>/etc/ssl/certs/qs-ca-certificates.crt</code>
* Network reachability from every Proxmox node to the appliance on TCP 8153 (REST API) and TCP 3260 (iSCSI).
* No configuration is required on the QuantaStor side beyond the pool: the plugin creates volumes, host entries, and access control assignments itself.


After copying the certificate, update the system certificate store to ensure the CA is trusted by the operating system.
=== API credentials ===


Proper SSL certificate configuration is required for successful API communication. The value specified in '''QS Host''' must match a Subject Alternative Name (SAN) entry in the QuantaStor API server certificate. If the API endpoint (IPv4 address or hostname) is not present in the SAN list, SSL verification will fail and API calls will be rejected.
The plugin authenticates to the QuantaStor REST API with a username and password. The {{Code|1=admin}} account works, but we recommend creating a dedicated user so the credentials stored on the Proxmox cluster carry only the permissions the plugin needs.


==== Example <code>storage.cfg</code> entry ====
[[File:pve_qs_role_create.png|thumb|right|570px|The Create Role dialog: filter the object types, select the rows, and apply the System permission scope to the selected operations.]]
<pre>
 
zfs: qs-shared-storage
Create a role that grants view access to everything plus the specific write operations the plugin performs, then create a user with that role (see [[User Add]]).
        blocksize 4k
 
        iscsiprovider quantastor
{{Navigation|Security &rarr; Role &rarr; Create ''(toolbar)''}}
        pool qs-52fa39c2-64b0-1830-5e80-050f469bb21d
 
        portal 10.0.26.31
{{Navigation|Security &rarr; User &rarr; Add ''(toolbar)''}}
        target qs-iscsi
 
        content images
The write operations the plugin needs are:
        nowritecache 1
 
        qs_apiv4_host 10.0.26.31
* '''Storage Volume''' -- create, delete, modify, resize, createSnapshot, deleteSnapshot, rollback, clone
        qs_password password
* '''Storage Volume ACL''' -- add, remove
        qs_user admin
* '''Host''' -- add, addInitiator, modify, remove, assignVolume, unassignVolume
        sparse 0
 
        zfs-base-path /dev/zvol
The role can be built in the web interface, or in one command with <code>[[QuantaStor CLI Command Reference#role-add|qs role-add]]</code> followed by <code>[[QuantaStor CLI Command Reference#user-add|qs user-add]]</code>:
 
<pre style="font-size: smaller">
qs role-add --name=pve-plugin-role --desc="Role for the Proxmox VE storage plugin" \
  --permissions="*:view:system,\
StorageVolume:create:system,StorageVolume:delete:system,StorageVolume:modify:system,\
StorageVolume:resize:system,StorageVolume:createSnapshot:system,StorageVolume:deleteSnapshot:system,\
StorageVolume:rollback:system,StorageVolume:clone:system,\
StorageVolumeAcl:add:system,StorageVolumeAcl:remove:system,\
Host:add:system,Host:addInitiator:system,Host:modify:system,Host:remove:system,\
Host:assignVolume:system,Host:unassignVolume:system"
 
qs user-add --name=pve-plugin --password=<password> --role=pve-plugin-role
</pre>
 
The full storage lifecycle -- storage activation, volume create, iSCSI access control, snapshot create and delete, and volume delete -- works under a user with exactly this permission set.
 
== Installation ==
 
Install the plugin package on '''every node in the cluster'''. A node without the plugin cannot activate the storage, which breaks migration to that node and leaves the storage inactive there.
 
<pre style="font-size: smaller">
dpkg -i pve-storage-quantastor_1.0.0-1_all.deb
</pre>
 
The package automatically enables the iSCSI initiator service ({{Code|1=iscsid}}), registers the storage type with the Proxmox services, and adds the QuantaStor panel to the web interface. Refresh the browser after installing and the '''QuantaStor''' entry appears in the Add Storage menu.
 
The package installs dpkg triggers that re-apply the web interface integration whenever the {{Code|1=pve-manager}} or {{Code|1=pve-storage}} packages are upgraded, so routine Proxmox updates need no plugin reinstall.
 
=== Upgrades ===
 
Install the new package version with {{Code|1=dpkg -i}} on each node. Existing storage definitions, running guests, and iSCSI sessions are unaffected; the Proxmox services are restarted automatically by the package.
 
=== Uninstalling ===
 
<pre style="font-size: smaller">
apt remove pve-storage-quantastor
</pre>
 
Removing the package does not remove storage entries from {{Code|1=/etc/pve/storage.cfg}}. Delete or disable QuantaStor storage entries first if the plugin is being removed permanently; otherwise those entries error once the plugin is gone.
 
== Configuration ==
 
=== Adding the storage in the web interface ===
 
{{Navigation|Datacenter &rarr; Storage &rarr; Add ''(button)'' &rarr; QuantaStor}}
 
[[File:pve_qs_add_storage.png|thumb|right|600px|The Add: QuantaStor dialog. The magnifier button next to Pool scans the appliance and lists its active pools.]]
 
{| class="wikitable"
! Field !! Description
|-
| ID || Proxmox storage identifier. Auto-fills with the first free {{Code|1=qs-storage-N}}; edit as needed.
|-
| API Host || QuantaStor appliance IP address or hostname. For an HA pool, use the failover group's virtual interface -- see [[#QuantaStor HA storage pools|QuantaStor HA storage pools]].
|-
| Username || QuantaStor API user.
|-
| Password || QuantaStor API password. Stored in a root-only file on the cluster filesystem, never in {{Code|1=storage.cfg}}.
|-
| Pool || QuantaStor storage pool to provision volumes from. Type a pool name, or click the magnifier button to connect to the appliance and pick from its active pools.
|-
| Content || {{Code|1=Disk image}} for VM disks, {{Code|1=Container}} for LXC root filesystems. Both are selected by default.
|-
| Nodes || Restrict the storage to specific Proxmox nodes; leave empty for all nodes.
|-
| iSCSI Portal ''(Advanced)'' || Portal address for iSCSI logins. Defaults to the API Host when left blank.
|-
| SSL Verify ''(Advanced)'' || Verify the appliance's TLS certificate against the node's system CA store. Leave off for self-signed certificates.
|}
 
'''API Host''', '''Username''', and '''Pool''' are fixed once the storage is created. The other fields can be changed later through the Edit dialog or {{Code|1=pvesm set}}.
 
[[File:pve_qs_add_storage_advanced.png|thumb|right|600px|The same dialog with Advanced ticked: the iSCSI portal defaults to the API Host, and SSL verification is off for self-signed certificates.]]
 
=== Adding the storage from the command line ===
 
<pre style="font-size: smaller">
pvesm add quantastor qs-storage-1 \
  --api_host 10.0.0.20 \
  --username admin \
  --password '<password>' \
  --pool_id <pool-name-or-uuid> \
  --content images,rootdir
</pre>
</pre>


==== Common misconfigurations ====
Single-quote the password so the shell does not interpret special characters ({{Code|1=$}}, {{Code|1=!}}, backticks).
* Invalid or non-existent QuantaStor storage pool (e.g. <code>qs-&lt;uuid&gt;</code>).
 
* iSCSI portal or QuantaStor API host unreachable due to network or firewall issues.
The resulting {{Code|1=/etc/pve/storage.cfg}} entry contains no password:
* Incorrect QuantaStor API credentials.
* Empty '''Target''' field or target value containing whitespace.
* SSL verification failures caused by missing or incorrect SAN entries for the configured '''QS Host'''.


==== Validation commands ====
<pre style="font-size: smaller">
You can use the <code>pvesm</code> storage CLI to see the status of your storage configuration
quantastor: qs-storage-1
<pre>
         api_host 10.0.0.20
pvesm status
        pool_id <pool-name-or-uuid>
Name                Type    Status    Total (KiB)      Used (KiB) Available (KiB)        %
        username admin
local                dir    active        14173644         5316720        8115140  37.51%
         content images,rootdir
local-lvm        lvmthin    active        12406784              0       12406784    0.00%
         shared 1
qs-storage          zfs    active      236716032         1074352      235641680    0.45%
         ssl_verify 0
qs-storage-2         zfs    active      236716032         1074352      235641680    0.45%
</pre>
</pre>
From QuantaStor you can verify that a host has been created for each node in your PVE cluster.
<pre>
qs host-list


Name                IP Address      Host OS Type 
All options:
-----------------------------------------------------
 
pve-0-proxmox-host                    0             
{| class="wikitable"
pve-1-proxmox-host                    0 
! Option !! Required !! Default !! Description
|-
| {{Code|1=api_host}} || yes || -- || QuantaStor appliance IP or hostname (HA pools: the failover group virtual interface)
|-
| {{Code|1=username}} || yes || -- || QuantaStor API user
|-
| {{Code|1=password}} || on create || -- || QuantaStor API password; stored outside {{Code|1=storage.cfg}}
|-
| {{Code|1=pool_id}} || yes || -- || QuantaStor pool name or UUID
|-
| {{Code|1=content}} || no || {{Code|1=images}} || {{Code|1=images}} (VM disks), {{Code|1=rootdir}} (containers), or both
|-
| {{Code|1=portal}} || no || {{Code|1=api_host}} || iSCSI portal address when it differs from the API endpoint
|-
| {{Code|1=api_port}} || no || 8153 || QuantaStor REST API port
|-
| {{Code|1=ssl_verify}} || no || 0 || Verify the appliance TLS certificate (see [[#SSL verification|SSL verification]])
|-
| {{Code|1=sparse}} || no || 1 || Thin provisioning; set {{Code|1=0}} for thick (see below)
|-
| {{Code|1=shared}} || no || 1 || Set automatically on create; marks the storage cluster-shared so migration works
|-
| {{Code|1=nodes}} || no || all || Restrict to specific Proxmox nodes
|}
 
'''Change settings with {{Code|1=pvesm set}} (or the Edit dialog), not by removing and re-adding the storage.''' The remove path deletes the stored password file cluster-wide, and a re-add that fails to capture the password leaves every operation failing until the password is set again.
 
=== Thin and thick provisioning ===
 
Volumes are '''thin provisioned by default''': they consume pool space as data is written, not their full size at creation. Set {{Code|1=sparse 0}} on the storage for thick provisioning, where each volume reserves 100% of its size in the pool at creation time. Note this default is the opposite of Proxmox's built-in ZFS storage types, where {{Code|1=sparse}} defaults to off.
 
=== How the plugin stores credentials ===
 
The API password is never written to {{Code|1=storage.cfg}}. The plugin stores it in {{Code|1=/etc/pve/priv/storage/<storage-id>.pw}} (mode 0600), the same convention Proxmox uses for CIFS and Proxmox Backup Server credentials. Because {{Code|1=/etc/pve}} is the pmxcfs cluster filesystem, the file replicates to every node automatically.
 
If that file is missing or empty on a node, every plugin operation on that node fails with an explicit error naming the file and the {{Code|1=pvesm set}} command that repairs it.
 
=== SSL verification ===
 
By default ({{Code|1=ssl_verify 0}}) the plugin accepts the appliance's certificate without verification, which suits the self-signed certificate QuantaStor ships with. To enable verification, install the QuantaStor CA certificate into the system trust store on '''every''' Proxmox node, then enable the option:
 
<pre style="font-size: smaller">
cp qs-ca.crt /usr/local/share/ca-certificates/
update-ca-certificates
pvesm set <storage-id> --ssl_verify 1
</pre>
 
The value configured as '''API Host''' must appear in the certificate's Subject Alternative Name list -- verification checks the hostname as well as the trust chain, so a certificate without the configured IP address or hostname in its SAN entries fails even with the CA trusted.
 
=== QuantaStor HA storage pools ===
 
If the pool is a QuantaStor HA storage pool -- one owned by a [[Create Storage Pool High-Availability Group|storage pool high-availability group]] that fails over between two controllers -- configure the storage against the failover group's '''virtual interface''', not against either controller's own address:
 
<pre style="font-size: smaller">
quantastor: qs-ha
        api_host 10.0.0.20        # the failover group's floating VIF
        portal 10.0.0.20          # the same VIF
        pool_id my-ha-pool
        username admin
        content images,rootdir
        shared 1
</pre>
 
Both {{Code|1=api_host}} and {{Code|1=portal}} must be that virtual interface (see [[High-availability VIF Management]]). The address moves with the pool on failover, so the REST endpoint and the iSCSI portal follow the active controller on their own. Pointed at a controller's fixed address instead, the storage stops working the moment the pool moves to the other controller.
 
Nothing else changes -- the plugin does not need to be told the pool is HA, and no extra options are required. During a controller failover, guests pause rather than fail; see [[#Known limitations|Known limitations]] for the measured behavior and the one initiator setting that must not be changed.
 
=== Verifying the setup ===
 
The password file should exist on every cluster node, and the storage should activate:
 
<pre style="font-size: smaller">
ls -la /etc/pve/priv/storage/
pvesm status --storage <storage-id>
</pre>
 
Expect the {{Code|1=<storage-id>.pw}} file present on each node and the storage reported {{Code|1=active}} with sizes populated. On the QuantaStor side, each Proxmox node appears as a host entry once the storage has been activated there -- verify with <code>[[QuantaStor CLI Command Reference#host-list|qs host-list]]</code> or in the web interface under '''Hosts'''.
 
== Container (LXC) support ==
 
LXC containers can place their root filesystem (and additional mount points) on QuantaStor storage. Enable the '''Container''' content type on the storage -- tick it in the dialog, or:
 
<pre style="font-size: smaller">
pvesm set <storage-id> --content images,rootdir
</pre>
</pre>


=== Usage ===
A container volume is the same raw iSCSI LUN as a VM disk. Proxmox's container stack formats the LUN with ext4, mounts it, and grows the filesystem on resize -- the plugin's job is only to provide the block device, the same model Proxmox uses for containers on Ceph RBD and LVM-thin storage.


==== Creating VM disks ====
The full container lifecycle is supported: create, start, stop, and destroy (privileged and unprivileged), snapshots of running containers (filesystems are frozen first, so snapshots are consistent), snapshot rollback (container stopped), online and offline resize, templates with linked or full clones, restart-mode migration between nodes, extra mount points (one dedicated LUN per mount point), and {{Code|1=pct fstrim}} (freed space returns to the thin volume through iSCSI UNMAP).
VM disks can be created during VM creation or added to an existing VM. When using QuantaStor-backed storage, disk provisioning behavior (thin vs thick) is controlled by the storage configuration.


[[File:Base-create-vm-disk-dialog.JPG|400px]]
Container limitations:


==== VM to Template ====
* '''Container templates ({{Code|1=vztmpl}}) cannot be stored on this storage''' -- template tarballs need a file-based storage such as {{Code|1=local}}. Create containers from a template on {{Code|1=local}} onto QuantaStor storage. This mirrors Ceph RBD, where template storage is likewise not possible on block storage.
Virtual machines stored on QuantaStor-backed storage can be converted to templates using standard Proxmox workflows. Template creation does not immediately consume additional storage capacity.<br>
* '''Containers cannot live-migrate''' -- migration is restart mode, a Proxmox-wide constraint for containers on block storage (the same applies to RBD and LVM-thin).
[[File:Base-qs-volume-view-template-and-base-clone.JPG|400px]]
* '''Proxmox HA fencing with mounted container filesystems has not been explicitly validated.''' Proxmox's configuration locking prevents two nodes from mounting the same LUN during normal operation, but fence-triggered recovery of a node running containers on QuantaStor storage has not been exercised.


==== Cloning and Snapshots ====
== Usage ==
Cloning and snapshot operations are supported for VM disks provisioned on QuantaStor storage. These operations leverage ZFS snapshot and clone functionality on the QuantaStor backend.


[[File:Base-clone-dialog.JPG|400px]]
VM disks on QuantaStor storage behave like disks on any shared Proxmox block storage: create them when creating a guest or add them later, and the plugin provisions the backing volume on the pool automatically. The operations below have behavior worth knowing about.


==== Moving and Resizing VM disks ====
=== Snapshots and rollback ===
Disk resize operations are supported and will update the backing ZFS volume size on the QuantaStor system. Disk move operations between QuantaStor-backed storage and other storage types are supported, subject to available capacity and network performance.


==== Reassign Hard Disk Owner ====
Snapshots are QuantaStor volume snapshots taken through the API. Snapshots of running containers freeze the filesystem first, so they are filesystem-consistent.
Disk ownership reassignment is supported for QuantaStor-backed volumes and may be required after cloning, restoring, or manual disk operations.


==== Creating Containers (CTs) ====
Rollback requires the guest to be stopped, and only the most recent snapshot can be rolled back to (Proxmox enforces the stop; the plugin enforces the ordering). A rollback issued right after stopping a guest can take up to a few minutes: the appliance's session tracking releases the volume on a cycle of up to three minutes after the initiator disconnects, and the plugin waits for that release before asking the appliance to roll back. This wait is normal, not a hang.
Support for LXC containers on QuantaStor-backed storage has **not been tested**. Users attempting to provision containers should proceed with caution and validate behavior before using in production.


==== VM Migration ====
=== Templates and clones ===
Live and offline VM migration is supported when the destination node has access to the same QuantaStor-backed storage configuration.


==== Limitations and Unsupported Operations ====
Converting a guest to a template renames its volumes to {{Code|1=base-*}} and takes a template snapshot; clones are created from that snapshot on the appliance, so creating a clone does not copy the full volume up front. Destroy clones before destroying their template.
Certain operations may be restricted or behave differently due to the ZFS over iSCSI architecture. Refer to this section before performing large-scale migrations or automation.


=== Troubleshooting ===
=== Resizing ===


==== Logging locations ====
Disks can be grown online -- the volume grows on the appliance, the plugin refreshes the kernel's view of the LUN, and the guest sees the new size without a restart. Shrinking is not supported.
By default, logging is disabled. To enable debug logging:


* Open the QuantaStor plugin Perl module: 
=== Migration ===
  <code>/usr/share/perl5/PVE/Storage/LunCmd/QuantaStorPlugin.pm</code>
* Set the debug variable to 1: 
  <code>our $QS_DEBUG = 1;</code>
* Save the file and restart the Proxmox service (see Installation instructions).


Once enabled, logs will be written to:
Live and offline VM migration between nodes works over the shared storage: the same volume is activated on the destination and released on the source, with no data copied. Every node involved needs the plugin installed and the storage active -- see [[#Troubleshooting|Troubleshooting]] if a migration fails.
<code>/var/log/pve-quantastor-plugin.log</code> on the node where the plugin is installed.


Logging provides detailed information about API calls, storage operations, and potential errors, which is useful for troubleshooting configuration or connectivity issues.
== Troubleshooting ==


==== Common errors ====
The [https://github.com/OSNEXUS/pve-quantastor-plugin/blob/master/README.md#troubleshooting README troubleshooting section] carries the full decision trees; these are the most common cases.


* '''Permission denied - invalid PVE ticket (401)'''
'''{{Code|1=Authentication check failed ... [err=26]}}''' -- QuantaStor rejected the request's credentials. Check, in order:
  *Cause:* Plugin commands require a valid Proxmox session. 
  *Fix:* Ensure the plugin is installed on **all nodes** in the cluster. If using snapshots or reverting VMs, make sure sessions are refreshed. Restart the node if necessary.


* '''Unable to verify host / SSL errors'''
# The password file exists and is non-empty on the node running the operation: {{Code|1=ls -la /etc/pve/priv/storage/<storage-id>.pw}}. If missing, repair with {{Code|1=pvesm set <storage-id> --password '<password>'}} from any node.
  *Cause:* SSL verification fails if the QuantaStor API endpoint is not present in the root CA certificate’s Subject Alternative Name (SAN) list.
# The credentials are actually valid -- test with curl, which bypasses the plugin: <code>curl -k "https://<api_host>:8153/qstorapi/storagePoolGet?storagePool=<pool>" -u admin:'<password>'</code>
  *Fix:* Ensure the QuantaStor Root CA is installed on all nodes:
# A password containing shell metacharacters was mangled by an unquoted {{Code|1=pvesm add}} -- re-set it single-quoted.
    <code>/etc/ssl/certs/qs-ca-certificates.crt</code>
    Update the system certificate store. The <code>QS Host</code> must match a SAN entry (IPv4 or hostname).


* '''Storage locked / command timed out'''
'''{{Code|1=Failed to delete storage volume ... [err=493]}}''' on destroy or snapshot delete -- the backing volume was busy when the appliance tried to destroy it. Current QuantaStor releases eliminate the known causes; on older appliance builds, a delete of a freshly created snapshot can be refused for a few minutes. If a failed snapshot delete leaves the guest locked, unlock it ({{Code|1=qm unlock <vmid>}} or {{Code|1=pct unlock <ctid>}}) and retry after a few minutes. A persistently failing delete on a current release warrants a support ticket with {{Code|1=/var/log/qs/qs_service.log}} from around the failure.
  *Cause:* Proxmox sometimes locks storage during operations such as VM creation or migration.
  *Fix:* Do **not manually delete lock files**. Wait for operations to finish or check for hung processes. Restart the node if locks persist. Ensure ZFS or iSCSI targets are healthy and reachable.


* '''Portal or storage pool unreachable'''
'''Container fails to start with {{Code|1=run_buffer: ... Script exited with status 255}}''' -- check that the storage still advertises the Container content type. Proxmox's pre-start hook refuses to start a container whose root storage lacks {{Code|1=rootdir}}, and the real error is swallowed. Restore with {{Code|1=pvesm set <storage-id> --content images,rootdir}}.
  *Cause:* Network issues or incorrect iSCSI portal/IP configuration.
  *Fix:* Verify network connectivity between Proxmox nodes and QuantaStor. Ensure the pool exists and is online in QuantaStor.


* '''Invalid QuantaStor credentials'''
'''Storage shows {{Code|1=active}} on some nodes and {{Code|1=inactive}} on others''' -- the inactive node's error says which case it is: a "no password configured" message means the password file is missing there (repair as in err=26 above); a network or SSL error means that node cannot reach the API host on port 8153 -- check firewalls and routing, not the plugin.
  *Cause:* Username/password mismatch or insufficient API permissions. 
  *Fix:* Confirm the API user exists in QuantaStor with permissions to manage hosts, volumes, snapshots, and clones. Avoid using admin credentials directly if possible.


* '''Low reported usage / thin-provisioned volumes'''
'''Migration or {{Code|1=qm move-disk}} to another node fails''' -- verify on the destination node that the plugin package is installed, {{Code|1=pvesm status}} shows the storage, and the password file exists. If the plugin and storage are healthy, check for QEMU version skew between the nodes (pin the VM's machine type to the older node's version) and confirm the storage has {{Code|1=shared 1}} set.
  *Cause:* Thin-provisioned volumes only consume physical storage as data is written.
  *Fix:* This is expected behavior; usage will increase as VMs write data. Thick-provisioned volumes show reserved storage immediately.


==== Additional tips ====
'''{{Code|1=Plugin ... is implementing an older storage API, an upgrade is recommended}}''' in the logs -- cosmetic; see [[#Known limitations|Known limitations]].
* Restart Proxmox services after installing or updating the plugin.
* Verify that ZFS pools and iSCSI targets are healthy in QuantaStor before creating VMs.
* Use the debug log for tracking API requests and responses when troubleshooting errors. 
* For SSL issues, test connectivity using: 
  <code>openssl s_client -connect &lt;QS Host&gt;:8153 -showcerts</code> 
  to confirm the certificate chain is trusted.


=== Roadmap ===
Plugin errors surface in the Proxmox task log of the operation that failed, and in the journal of the PVE service that ran it ({{Code|1=journalctl -u pvedaemon}}, {{Code|1=journalctl -u pvestatd}}).


==== Planned features ====
== Known limitations ==
* '''Standalone QuantaStor plugin''' – Allow the plugin to operate independently without patching core PVE files. 
* '''Support for Proxmox VE Replication''' – Enable replication of VMs and storage volumes using QuantaStor storage backend. 
* '''QuantaStor API tokens''' – Move from password-based authentication to secure API token usage. 
* '''CHAP support''' – Implement CHAP authentication for iSCSI targets to improve security.


==== Known gaps ====
* '''QuantaStor HA pool failover pauses guest I/O for roughly 30-45 seconds.''' When an HA pool moves between controllers, guests on that storage block until the pool is imported on the other controller, then continue with no errors and no data loss -- measured guest-visible pauses were 26-34 seconds, independent of how many volumes the pool serves. Guests and their filesystems ride through it without tuning: the initiator holds queued I/O while the session recovers, so the guest never sees an error. '''Do not lower {{Code|1=node.session.timeo.replacement_timeout}}''' in {{Code|1=/etc/iscsi/iscsid.conf}}: its 120-second default is the budget that makes the pause survivable, and multipath tuning advice that shortens it does not apply here because there is no alternate path. This is QuantaStor pool failover, separate from Proxmox's own HA -- for Proxmox HA fencing see [[#Container (LXC) support|Container (LXC) support]].
* '''Proxmox HA and Replication''' – High Availability and cluster replication with QuantaStor-managed volumes are not yet fully supported. Use caution when deploying mission-critical workloads in a HA cluster. QuantaStor storage plugin does not support Proxmox native HA and replication, however QuantaStor has its own replication and HA systems for ZFS storage pools.
* '''Snapshot rollback''' is limited to the most recent snapshot and requires the guest stopped; rollback can wait up to a few minutes for the appliance to release the volume (see [[#Usage|Usage]]).
* '''Volumes cannot be shrunk''', only grown.
* '''All volumes present at iSCSI LUN 0''' on their dedicated target; the {{Code|1=lun}} field the QuantaStor API reports on volume objects is an internal value and does not reflect the wire LUN.
* '''The "older storage API" log message on Proxmox VE 9.2 is cosmetic.''' The plugin declares storage API version 13 so that one build supports both 9.1 and 9.2; PVE 9.2 logs a recommendation to upgrade at every plugin scan. Behavior is identical on both versions. If the log noise is a problem, filter it with an rsyslog rule: {{Code|1=:msg, contains, "implementing an older storage API" stop}} in {{Code|1=/etc/rsyslog.d/}}.
* '''Proxmox storage replication is not supported''' ({{Code|1=pvesr}}); QuantaStor's own [[Remote-replication (DR)|remote replication]] can protect the pool instead.


=== Support & Contributing ===
== Reporting issues ==


==== Reporting Issues ====
Report bugs and feature requests through [https://github.com/OSNEXUS/pve-quantastor-plugin/issues GitHub Issues] on the plugin repository, or by email to {{Code|1=eng-proxmox@osnexus.com}}. Include the Proxmox VE version, the plugin version, and the failing task's log output.
The QuantaStor Proxmox Storage Plugin is an actively developed project and requires ongoing maintenance to remain compatible with new Proxmox VE releases.


Issues, bugs, and feature requests can be reported through the following public channels:
== Related pages ==


* '''GitHub Issues''' – The preferred method for reporting bugs, requesting enhancements, and tracking development work. 
* [[Storage Pools]]
Please submit issues through the project’s GitHub repository and include relevant details such as Proxmox version, plugin version, error messages, and logs when available.
* [[Storage Volumes]]
* [[Create Storage Pool High-Availability Group]]
* [[High-availability VIF Management]]
* [[ISCSI Initiator Setup]]
* [[User Add]]


* '''OSNEXUS Engineering Email''' – Public support and feedback are also accepted via email at: <code>eng-proxmox@osnexus.com</code>
----
 
<small>''Plugin release 1.0.0. Verified against QuantaStor 7.0.0.''</small>
To help us diagnose issues efficiently, please include clear reproduction steps, environment details, and any relevant log output (if plugin logging is enabled).

Revision as of 19:43, 4 September 2026

The QuantaStor Proxmox VE storage plugin integrates QuantaStor as a native iSCSI block storage backend for Proxmox VE virtual machines and LXC containers. This page covers installing the plugin, configuring a QuantaStor-backed storage in Proxmox, and the operational behavior to expect.

Section Purpose
Overview What the plugin is and where to get it
Architecture How Proxmox, the plugin, and the appliance interact
Requirements Supported Proxmox VE versions, QuantaStor requirements, credentials
Installation Installing, upgrading, and removing the plugin package
Configuration Adding the storage, all configuration options, HA pools, SSL
Container (LXC) support Running container root filesystems on QuantaStor volumes
Usage Snapshots, templates, clones, resize, and migration behavior
Troubleshooting The most common errors and their fixes
Known limitations Constraints to know before deploying
Reporting issues Where to report bugs and ask for help

Overview

The plugin registers a first-class storage type named quantastor in Proxmox VE. It appears in the Proxmox web interface under Datacenter → Storage → Add like any built-in storage type, and it works with the standard pvesm, qm, and pct tooling. It does not patch or replace any Proxmox source files, so Proxmox package upgrades do not break it.

Each Proxmox disk is a dedicated QuantaStor storage volume exported over iSCSI. The plugin provisions volumes through the QuantaStor REST API and logs into their iSCSI targets on the Proxmox node, handing Proxmox a stable local block device. Virtual machine disks and container root filesystems are both supported.

The plugin is open source and distributed as a Debian package:

The README in the repository carries the full module reference and developer documentation; this page is the operator-facing guide.

Architecture

QuantaStor Proxmox VE plugin architecture: the control plane provisions volumes over the REST API while the data plane serves guest I/O over iSCSI.

The plugin separates management traffic from storage traffic:

  • Control plane -- all provisioning operations (create, delete, snapshot, clone, resize, access control) are REST calls to the QuantaStor API over HTTPS on port 8153, authenticated with HTTP Basic credentials.
  • Data plane -- guest I/O flows over iSCSI (port 3260) between the Proxmox node's initiator and the QuantaStor target. The plugin drives the standard open-iscsi initiator with iscsiadm: it discovers the target, logs in, and waits for the device to appear before handing it to Proxmox.

Each storage volume is exported through its own dedicated iSCSI target, always presented at LUN 0. The plugin resolves devices through stable /dev/disk/by-path/ip-<portal>:<port>-iscsi-<target-iqn>-lun-0 paths rather than ephemeral /dev/sdX names. QEMU attaches the raw block device directly for VM disks; for containers, Proxmox formats the same block device with ext4 and mounts it as the container root filesystem.

On first activation the plugin registers each Proxmox node with QuantaStor as an iSCSI host entry, using the node's initiator IQN from /etc/iscsi/initiatorname.iscsi. Before a volume is used on a node, the plugin assigns that host to the volume's access control list; when the volume is released, the assignment is removed. Volumes are therefore only visible to the nodes actively using them.

Volume naming

Volumes on the QuantaStor pool follow the standard Proxmox naming convention, so the mapping between a Proxmox disk and its QuantaStor volume is direct:

Type Name pattern Example
VM or container disk vm-<vmid>-disk-<N> vm-100-disk-0
Template base disk base-<vmid>-disk-<N> base-100-disk-0
Template snapshot template-base-<vmid>-disk-<N> template-base-100-disk-0
Snapshot <volume>_<snapshot> vm-100-disk-0_snap1

Requirements

Supported Proxmox VE versions

Proxmox VE version Supported Notes
9.2.x Yes Loads with a cosmetic "implementing an older storage API" log message; see Known limitations
9.1.x Yes Loads silently
8.4.x and older No The plugin does not load; the quantastor type will not appear

Nodes running 9.1 and 9.2 can be mixed in one cluster and storage operations work transparently across both. Do not include 8.x nodes -- any storage operation routed to one (for example a migration target) fails because the storage type is not registered there.

QuantaStor requirements

  • A QuantaStor system with a storage pool to provision from. Use a current QuantaStor release; older builds refuse to delete freshly created snapshots for several minutes and can wedge volumes on template conversion (see Troubleshooting, err=493).
  • Network reachability from every Proxmox node to the appliance on TCP 8153 (REST API) and TCP 3260 (iSCSI).
  • No configuration is required on the QuantaStor side beyond the pool: the plugin creates volumes, host entries, and access control assignments itself.

API credentials

The plugin authenticates to the QuantaStor REST API with a username and password. The admin account works, but we recommend creating a dedicated user so the credentials stored on the Proxmox cluster carry only the permissions the plugin needs.

The Create Role dialog: filter the object types, select the rows, and apply the System permission scope to the selected operations.

Create a role that grants view access to everything plus the specific write operations the plugin performs, then create a user with that role (see User Add).

Navigation: Security → Role → Create (toolbar)
Navigation: Security → User → Add (toolbar)

The write operations the plugin needs are:

  • Storage Volume -- create, delete, modify, resize, createSnapshot, deleteSnapshot, rollback, clone
  • Storage Volume ACL -- add, remove
  • Host -- add, addInitiator, modify, remove, assignVolume, unassignVolume

The role can be built in the web interface, or in one command with qs role-add followed by qs user-add:

qs role-add --name=pve-plugin-role --desc="Role for the Proxmox VE storage plugin" \
  --permissions="*:view:system,\
StorageVolume:create:system,StorageVolume:delete:system,StorageVolume:modify:system,\
StorageVolume:resize:system,StorageVolume:createSnapshot:system,StorageVolume:deleteSnapshot:system,\
StorageVolume:rollback:system,StorageVolume:clone:system,\
StorageVolumeAcl:add:system,StorageVolumeAcl:remove:system,\
Host:add:system,Host:addInitiator:system,Host:modify:system,Host:remove:system,\
Host:assignVolume:system,Host:unassignVolume:system"

qs user-add --name=pve-plugin --password=<password> --role=pve-plugin-role

The full storage lifecycle -- storage activation, volume create, iSCSI access control, snapshot create and delete, and volume delete -- works under a user with exactly this permission set.

Installation

Install the plugin package on every node in the cluster. A node without the plugin cannot activate the storage, which breaks migration to that node and leaves the storage inactive there.

dpkg -i pve-storage-quantastor_1.0.0-1_all.deb

The package automatically enables the iSCSI initiator service (iscsid), registers the storage type with the Proxmox services, and adds the QuantaStor panel to the web interface. Refresh the browser after installing and the QuantaStor entry appears in the Add Storage menu.

The package installs dpkg triggers that re-apply the web interface integration whenever the pve-manager or pve-storage packages are upgraded, so routine Proxmox updates need no plugin reinstall.

Upgrades

Install the new package version with dpkg -i on each node. Existing storage definitions, running guests, and iSCSI sessions are unaffected; the Proxmox services are restarted automatically by the package.

Uninstalling

apt remove pve-storage-quantastor

Removing the package does not remove storage entries from /etc/pve/storage.cfg. Delete or disable QuantaStor storage entries first if the plugin is being removed permanently; otherwise those entries error once the plugin is gone.

Configuration

Adding the storage in the web interface

Navigation: Datacenter → Storage → Add (button) → QuantaStor
The Add: QuantaStor dialog. The magnifier button next to Pool scans the appliance and lists its active pools.
Field Description
ID Proxmox storage identifier. Auto-fills with the first free qs-storage-N; edit as needed.
API Host QuantaStor appliance IP address or hostname. For an HA pool, use the failover group's virtual interface -- see QuantaStor HA storage pools.
Username QuantaStor API user.
Password QuantaStor API password. Stored in a root-only file on the cluster filesystem, never in storage.cfg.
Pool QuantaStor storage pool to provision volumes from. Type a pool name, or click the magnifier button to connect to the appliance and pick from its active pools.
Content Disk image for VM disks, Container for LXC root filesystems. Both are selected by default.
Nodes Restrict the storage to specific Proxmox nodes; leave empty for all nodes.
iSCSI Portal (Advanced) Portal address for iSCSI logins. Defaults to the API Host when left blank.
SSL Verify (Advanced) Verify the appliance's TLS certificate against the node's system CA store. Leave off for self-signed certificates.

API Host, Username, and Pool are fixed once the storage is created. The other fields can be changed later through the Edit dialog or pvesm set.

The same dialog with Advanced ticked: the iSCSI portal defaults to the API Host, and SSL verification is off for self-signed certificates.

Adding the storage from the command line

pvesm add quantastor qs-storage-1 \
  --api_host 10.0.0.20 \
  --username admin \
  --password '<password>' \
  --pool_id <pool-name-or-uuid> \
  --content images,rootdir

Single-quote the password so the shell does not interpret special characters ($, !, backticks).

The resulting /etc/pve/storage.cfg entry contains no password:

quantastor: qs-storage-1
        api_host 10.0.0.20
        pool_id <pool-name-or-uuid>
        username admin
        content images,rootdir
        shared 1
        ssl_verify 0

All options:

Option Required Default Description
api_host yes -- QuantaStor appliance IP or hostname (HA pools: the failover group virtual interface)
username yes -- QuantaStor API user
password on create -- QuantaStor API password; stored outside storage.cfg
pool_id yes -- QuantaStor pool name or UUID
content no images images (VM disks), rootdir (containers), or both
portal no api_host iSCSI portal address when it differs from the API endpoint
api_port no 8153 QuantaStor REST API port
ssl_verify no 0 Verify the appliance TLS certificate (see SSL verification)
sparse no 1 Thin provisioning; set 0 for thick (see below)
shared no 1 Set automatically on create; marks the storage cluster-shared so migration works
nodes no all Restrict to specific Proxmox nodes

Change settings with pvesm set (or the Edit dialog), not by removing and re-adding the storage. The remove path deletes the stored password file cluster-wide, and a re-add that fails to capture the password leaves every operation failing until the password is set again.

Thin and thick provisioning

Volumes are thin provisioned by default: they consume pool space as data is written, not their full size at creation. Set sparse 0 on the storage for thick provisioning, where each volume reserves 100% of its size in the pool at creation time. Note this default is the opposite of Proxmox's built-in ZFS storage types, where sparse defaults to off.

How the plugin stores credentials

The API password is never written to storage.cfg. The plugin stores it in /etc/pve/priv/storage/<storage-id>.pw (mode 0600), the same convention Proxmox uses for CIFS and Proxmox Backup Server credentials. Because /etc/pve is the pmxcfs cluster filesystem, the file replicates to every node automatically.

If that file is missing or empty on a node, every plugin operation on that node fails with an explicit error naming the file and the pvesm set command that repairs it.

SSL verification

By default (ssl_verify 0) the plugin accepts the appliance's certificate without verification, which suits the self-signed certificate QuantaStor ships with. To enable verification, install the QuantaStor CA certificate into the system trust store on every Proxmox node, then enable the option:

cp qs-ca.crt /usr/local/share/ca-certificates/
update-ca-certificates
pvesm set <storage-id> --ssl_verify 1

The value configured as API Host must appear in the certificate's Subject Alternative Name list -- verification checks the hostname as well as the trust chain, so a certificate without the configured IP address or hostname in its SAN entries fails even with the CA trusted.

QuantaStor HA storage pools

If the pool is a QuantaStor HA storage pool -- one owned by a storage pool high-availability group that fails over between two controllers -- configure the storage against the failover group's virtual interface, not against either controller's own address:

quantastor: qs-ha
        api_host 10.0.0.20        # the failover group's floating VIF
        portal 10.0.0.20          # the same VIF
        pool_id my-ha-pool
        username admin
        content images,rootdir
        shared 1

Both api_host and portal must be that virtual interface (see High-availability VIF Management). The address moves with the pool on failover, so the REST endpoint and the iSCSI portal follow the active controller on their own. Pointed at a controller's fixed address instead, the storage stops working the moment the pool moves to the other controller.

Nothing else changes -- the plugin does not need to be told the pool is HA, and no extra options are required. During a controller failover, guests pause rather than fail; see Known limitations for the measured behavior and the one initiator setting that must not be changed.

Verifying the setup

The password file should exist on every cluster node, and the storage should activate:

ls -la /etc/pve/priv/storage/
pvesm status --storage <storage-id>

Expect the <storage-id>.pw file present on each node and the storage reported active with sizes populated. On the QuantaStor side, each Proxmox node appears as a host entry once the storage has been activated there -- verify with qs host-list or in the web interface under Hosts.

Container (LXC) support

LXC containers can place their root filesystem (and additional mount points) on QuantaStor storage. Enable the Container content type on the storage -- tick it in the dialog, or:

pvesm set <storage-id> --content images,rootdir

A container volume is the same raw iSCSI LUN as a VM disk. Proxmox's container stack formats the LUN with ext4, mounts it, and grows the filesystem on resize -- the plugin's job is only to provide the block device, the same model Proxmox uses for containers on Ceph RBD and LVM-thin storage.

The full container lifecycle is supported: create, start, stop, and destroy (privileged and unprivileged), snapshots of running containers (filesystems are frozen first, so snapshots are consistent), snapshot rollback (container stopped), online and offline resize, templates with linked or full clones, restart-mode migration between nodes, extra mount points (one dedicated LUN per mount point), and pct fstrim (freed space returns to the thin volume through iSCSI UNMAP).

Container limitations:

  • Container templates (vztmpl) cannot be stored on this storage -- template tarballs need a file-based storage such as local. Create containers from a template on local onto QuantaStor storage. This mirrors Ceph RBD, where template storage is likewise not possible on block storage.
  • Containers cannot live-migrate -- migration is restart mode, a Proxmox-wide constraint for containers on block storage (the same applies to RBD and LVM-thin).
  • Proxmox HA fencing with mounted container filesystems has not been explicitly validated. Proxmox's configuration locking prevents two nodes from mounting the same LUN during normal operation, but fence-triggered recovery of a node running containers on QuantaStor storage has not been exercised.

Usage

VM disks on QuantaStor storage behave like disks on any shared Proxmox block storage: create them when creating a guest or add them later, and the plugin provisions the backing volume on the pool automatically. The operations below have behavior worth knowing about.

Snapshots and rollback

Snapshots are QuantaStor volume snapshots taken through the API. Snapshots of running containers freeze the filesystem first, so they are filesystem-consistent.

Rollback requires the guest to be stopped, and only the most recent snapshot can be rolled back to (Proxmox enforces the stop; the plugin enforces the ordering). A rollback issued right after stopping a guest can take up to a few minutes: the appliance's session tracking releases the volume on a cycle of up to three minutes after the initiator disconnects, and the plugin waits for that release before asking the appliance to roll back. This wait is normal, not a hang.

Templates and clones

Converting a guest to a template renames its volumes to base-* and takes a template snapshot; clones are created from that snapshot on the appliance, so creating a clone does not copy the full volume up front. Destroy clones before destroying their template.

Resizing

Disks can be grown online -- the volume grows on the appliance, the plugin refreshes the kernel's view of the LUN, and the guest sees the new size without a restart. Shrinking is not supported.

Migration

Live and offline VM migration between nodes works over the shared storage: the same volume is activated on the destination and released on the source, with no data copied. Every node involved needs the plugin installed and the storage active -- see Troubleshooting if a migration fails.

Troubleshooting

The README troubleshooting section carries the full decision trees; these are the most common cases.

Authentication check failed ... [err=26] -- QuantaStor rejected the request's credentials. Check, in order:

  1. The password file exists and is non-empty on the node running the operation: ls -la /etc/pve/priv/storage/<storage-id>.pw. If missing, repair with pvesm set <storage-id> --password '<password>' from any node.
  2. The credentials are actually valid -- test with curl, which bypasses the plugin: curl -k "https://<api_host>:8153/qstorapi/storagePoolGet?storagePool=<pool>" -u admin:'<password>'
  3. A password containing shell metacharacters was mangled by an unquoted pvesm add -- re-set it single-quoted.

Failed to delete storage volume ... [err=493] on destroy or snapshot delete -- the backing volume was busy when the appliance tried to destroy it. Current QuantaStor releases eliminate the known causes; on older appliance builds, a delete of a freshly created snapshot can be refused for a few minutes. If a failed snapshot delete leaves the guest locked, unlock it (qm unlock <vmid> or pct unlock <ctid>) and retry after a few minutes. A persistently failing delete on a current release warrants a support ticket with /var/log/qs/qs_service.log from around the failure.

Container fails to start with run_buffer: ... Script exited with status 255 -- check that the storage still advertises the Container content type. Proxmox's pre-start hook refuses to start a container whose root storage lacks rootdir, and the real error is swallowed. Restore with pvesm set <storage-id> --content images,rootdir.

Storage shows active on some nodes and inactive on others -- the inactive node's error says which case it is: a "no password configured" message means the password file is missing there (repair as in err=26 above); a network or SSL error means that node cannot reach the API host on port 8153 -- check firewalls and routing, not the plugin.

Migration or qm move-disk to another node fails -- verify on the destination node that the plugin package is installed, pvesm status shows the storage, and the password file exists. If the plugin and storage are healthy, check for QEMU version skew between the nodes (pin the VM's machine type to the older node's version) and confirm the storage has shared 1 set.

Plugin ... is implementing an older storage API, an upgrade is recommended in the logs -- cosmetic; see Known limitations.

Plugin errors surface in the Proxmox task log of the operation that failed, and in the journal of the PVE service that ran it (journalctl -u pvedaemon, journalctl -u pvestatd).

Known limitations

  • QuantaStor HA pool failover pauses guest I/O for roughly 30-45 seconds. When an HA pool moves between controllers, guests on that storage block until the pool is imported on the other controller, then continue with no errors and no data loss -- measured guest-visible pauses were 26-34 seconds, independent of how many volumes the pool serves. Guests and their filesystems ride through it without tuning: the initiator holds queued I/O while the session recovers, so the guest never sees an error. Do not lower node.session.timeo.replacement_timeout in /etc/iscsi/iscsid.conf: its 120-second default is the budget that makes the pause survivable, and multipath tuning advice that shortens it does not apply here because there is no alternate path. This is QuantaStor pool failover, separate from Proxmox's own HA -- for Proxmox HA fencing see Container (LXC) support.
  • Snapshot rollback is limited to the most recent snapshot and requires the guest stopped; rollback can wait up to a few minutes for the appliance to release the volume (see Usage).
  • Volumes cannot be shrunk, only grown.
  • All volumes present at iSCSI LUN 0 on their dedicated target; the lun field the QuantaStor API reports on volume objects is an internal value and does not reflect the wire LUN.
  • The "older storage API" log message on Proxmox VE 9.2 is cosmetic. The plugin declares storage API version 13 so that one build supports both 9.1 and 9.2; PVE 9.2 logs a recommendation to upgrade at every plugin scan. Behavior is identical on both versions. If the log noise is a problem, filter it with an rsyslog rule: :msg, contains, "implementing an older storage API" stop in /etc/rsyslog.d/.
  • Proxmox storage replication is not supported (pvesr); QuantaStor's own remote replication can protect the pool instead.

Reporting issues

Report bugs and feature requests through GitHub Issues on the plugin repository, or by email to eng-proxmox@osnexus.com. Include the Proxmox VE version, the plugin version, and the failing task's log output.

Related pages


Plugin release 1.0.0. Verified against QuantaStor 7.0.0.