Scale-out Block Setup (ceph)

Scale-out block storage presents Ceph RBD images as QuantaStor Storage Volumes, reachable over iSCSI, Fibre Channel and NVMe-oF through every node of the Ceph cluster at once. This page covers the whole path: building the Ceph cluster, creating OSDs and journal groups, creating a block Storage Pool, provisioning volumes from it, and growing, monitoring and repairing the result.

Scale-out block is one of three things a Ceph cluster can serve. For file, see Scale-out File Setup (ceph); for S3 object storage, see Scale-out Object Setup (ceph). One cluster can serve all three at the same time from the same OSDs, and the Scaling Factor described below is how you divide the placement groups between them.

Section	Purpose
How the pieces fit together	Grid, cluster, OSD, journal group, pool, volume -- and which owns what.
Where the controls are	The Scale-out Storage Configuration tab, its three tree sections and its toolbar groups.
Before you start	Node count, per-node CPU and RAM minimums, and the network layout.
Creating the Ceph cluster	Every field on the three tabs, and the two settings that cannot be changed later.
Creating OSDs and journal groups	The Create OSDs & Journals dialog, its five policy checkboxes, and what Auto Config does.
Journal groups in detail	WAL and MDB devices, sizing, mirroring, migration and what can be removed.
Creating the block Storage Pool	Data pool layout, erasure coding, the Scaling Factor and the Force checkbox.
How the placement group count is chosen	The formula, why it stops scaling below 32 OSDs, and why it is permanent.
What an erasure-coded block pool actually creates	Two Ceph pools, not one, and which one the RBD lives in.
Provisioning Storage Volumes	The Ceph-specific fields on Create Storage Volume, and pool namespaces.
Presenting volumes to hosts	Multipathing across cluster members, and direct RBD access.
Growing the cluster	Adding members, adding OSDs, and when to add monitors.
Removing OSDs safely	Why delete is refused, and the reweight-first procedure.
Replacing a failed device	Replace Prepare, Replace, and the drain they perform.
Maintenance mode	What it sets in Ceph, and why it can appear on its own.
Monitoring cluster health	The dashboard, health events, and the three fullness ratios.
CLI command reference	Every `qs` command for scale-out block, and the ones with no dialog.
Troubleshooting	Clock skew, stranded journal groups, slow BlueStore operations, unreachable LUNs.

How the pieces fit together

Six objects stack up, and each one has to exist before the next can be created:

Object	What it is	Created by
Storage Grid	The management domain. Every system that will join a Ceph cluster must first be a grid member.	Grid Configuration
Ceph Cluster	A set of at least three grid members running Ceph together, with its own monitors, front-end and back-end networks.	Create Cluster
OSD	One Ceph Object Storage Daemon per data device. Holds the data.	Create OSDs & Journals
Journal Group	A mirrored pair of flash devices on one member, from which WAL and metadata-DB devices are carved for OSDs on that member.	Create OSDs & Journals, or Create Journal Group
Scale-out Storage Pool	A Ceph pool with a redundancy layout and a placement group count. The interface labels a block pool Scale-out Block.	Create Block Storage Pool
Storage Volume	An RBD image inside the pool, presented to hosts over iSCSI, FC or NVMe-oF.	Create (Storage Volume)

The grid and the cluster are separate constructs, and the distinction matters when you plan. A grid can hold systems that are in no Ceph cluster at all, and a system can belong to only one Ceph cluster. Cluster membership is what binds a system's data devices, monitors and network ports into the Ceph configuration; grid membership only makes it manageable from the same interface.

Ceph itself is documented at length upstream; this page does not restate it. What it does cover is every decision QuantaStor asks you to make, and what QuantaStor does with the answer.

Where the controls are

Navigation: Scale-out Storage Configuration

Everything for scale-out storage lives under the Scale-out Storage Configuration main tab. The tree on the left has three accordion sections, and the toolbar changes with the section you select:

Tree section	Toolbar groups	Detail tabs
Scale-out Storage Clusters	Scale-out Cluster Management (Create Cluster, Add Member, Modify Cluster, Adjust Clocks, Delete Cluster, Remove Member, Cluster Import) and Service Management (Add Monitor, Add MDS, Create Load Balancer, Add Object Gateway, Modify Object Gateway, Remove Monitor, Remove MDS, Delete Load Balancer, Remove Object Gateway)	Health Details, Cluster Members, Cluster Monitors, Object Gateways, File System Metadata Server, Load Balancer
Data & Journal Devices	Data & Journal Devices (Create OSDs & Journals, Create Journal Group, Modify OSD, Replace Prepare OSD, Multi-Delete OSD, Delete Journal Group, Service OSD, Replace OSD)	Data Devices/OSDs, Journal Groups, Journal Devices
Scale-out Storage Pools	EC Profiles & Rules, Block Storage (Create Block Storage Pool, Create Ceph Pool Namespace, Delete Block Storage Pool, Delete Ceph Pool Namespace), Object Storage, and File Storage behind the » overflow button at the right-hand end of the toolbar	Scale-out Storage Pools, Ceph File System, EC Profiles, Redundancy Rules, Object Pool Groups

Two things are worth knowing before you go looking for a button. There is no Modify Block Storage Pool button -- the Block Storage group offers only create and delete, and the modify dialog is reached by right-clicking the pool. And the File Storage group is normally hidden in the toolbar overflow, so it looks absent until you click ».

The dashboard above the detail tabs is the same on every section. Its seven tiles are Health (the raw Ceph health string), OSDs Up, MONs in Quorum, Pools Online, Object Gateways, Metadata Servers and Load Balancers. Below them, Placement Groups Health breaks the cluster's PGs into Unknown, Error, Warning, Busy and Normal, and Cluster Storage Capacity shows raw free against raw utilized -- raw, not usable, so a replica-3 pool consumes three times what its volumes hold.

Before you start

Set the systems up as ordinary appliances first, then build the cluster on top. Specifically:

Install and license each system, and set its DNS and NTP servers. See Storage System. NTP matters more here than anywhere else in the product -- Ceph monitors will not form a stable quorum across systems whose clocks disagree, and clock skew is the single most common cause of a cluster that forms and then reports a warning.
Configure the network ports so that the same port name carries the same role on every system: ethN on the front-end (client) network on all nodes, ethM on the back-end (cluster) network on all nodes. Mixing roles across nodes on the same port name is what makes a cluster hard to reason about later. See Network Ports.
Join the systems into one grid, from the system that will be the grid master. See Grid Configuration.

Only then create the Ceph cluster.

What the product enforces

Three systems minimum. Create Cluster does not open unless at least three grid members are available -- that is, not already in some other Ceph cluster. It closes immediately with Insufficient number of available storage systems detected. A minimum of three storage systems are required to create a Ceph Cluster. Three is the floor because Ceph needs three monitors to hold quorum.

Per-member hardware minimums, checked at create time. Each prospective member must have at least 2 CPU cores and at least 4 GB of RAM; the service refuses the create otherwise, and its own message names 16 GB as the practical minimum: "System '%s' has '%s' GB RAM, a minimum of 4GB of RAM is required (16GB minimum recommended). Cannot use system as a Ceph Cluster member until upgraded." Budget roughly 2 GB of RAM and a fraction of a core per OSD on top of that -- a 60-OSD node is a large dual-socket machine with well over 100 GB of RAM, not a 4 GB one.

All selected ports of one role must be on one subnet. The cluster create collects the CIDR of every interface you pick and fails if more than one appears: "More than 1 network addresses were specified from the given list of IP addresses." This is the error you get when you pick client interfaces from mixed subnets.

Hardware RAID under OSDs

Hardware RAID can be used as the storage abstraction an OSD is built on, and QuantaStor manages and monitors RAID controllers, but we do not recommend it except in specialised and hyper-scale configurations. It has a place in very large clusters and on servers whose CPU and RAM cannot support one OSD per drive: combining drives into RAID sets cuts the OSD count, and with it the RAM and CPU each node needs. The trade is that the cluster loses visibility of individual drives and CRUSH loses the granularity it uses to place data. Where hardware RAID is used, disk failures are handled inside the controller and never reach the cluster state -- which is the extra resiliency it buys. See Hardware Controllers & Enclosures.

Front-end and back-end networks

Ceph distinguishes the front-end (public) network, which clients and gateways use, from the back-end (cluster) network, which OSDs use for replication and recovery. QuantaStor asks for both when the cluster is created and for each member added afterwards.

Separating them isolates recovery traffic from client traffic, which is worth doing on 10 or 25 GbE. On 100 GbE and faster it is increasingly common -- and simpler -- to put both roles on the same subnet, and QuantaStor supports that: point the front-end and back-end selections at the same network. Whichever you choose, keep it uniform across the cluster.

Creating the Ceph cluster

Navigation: Scale-out Storage Configuration → Scale-out Storage Clusters → Scale-out Cluster Management → Create Cluster (toolbar)

Create Ceph Cluster Configuration is a three-tab wizard.

General asks for the substance of the cluster:

Name -- defaults to cluster-<n>. This is the QuantaStor display name; Ceph's own cluster identity is a separate FSID that QuantaStor generates and shows as the cluster's Custom ID.
Description -- free text.
Select Ceph Cluster Interfaces -- a table of every eligible grid member with its client (front-end) and back-end interface selection. Systems already in a Ceph cluster are filtered out of this list, which is why the dialog refuses to open once all your members are consumed.
Enable Compression -- checked by default, and create-time only. The dialog's own Advanced Settings header says so: "It is important to enable the compression option when the scale-out cluster is initially created as this option cannot be modified after deployment." There is no compression control in Modify Cluster. Leaving it on is the right default: it is what allows QuantaStor to apply BlueStore compression to the pools it creates for you.
Enable Encryption -- unchecked by default. Ticking it enables the Encryption tab, which is otherwise disabled.

Encryption offers Software Encryption (dmcrypt/LUKS, the default mode) or Hardware Encryption (self-encrypting drives), and a Key Server Profile when keys are to be held off the appliance. Encryption is a cluster-wide property applied to every OSD built in the cluster; there is no per-OSD encryption control anywhere in the interface, and WAL and metadata-DB devices share their OSD's key. Where some devices already encrypt themselves and you do not want to double-encrypt them, see Encryption Bypass. For how QuantaStor's software encryption and its key material work generally, see Software Encryption.

Advanced Settings carries the two OSD failure timers and a Domain Suffix field. The timers are described under Monitoring cluster health below.

QuantaStor deploys three Ceph monitors automatically as part of the create, on the first three members. You do not choose them.

The create is a long, staged operation and the task timeout scales with the cluster size -- five minutes for clusters under twelve nodes, and fifteen seconds per node above that. Watch it in the Tasks pane rather than assuming it has finished.

Modifying the cluster afterwards

Navigation: Scale-out Storage Configuration → Scale-out Storage Clusters → Scale-out Cluster Management → Modify Cluster (toolbar)

Modify Ceph Cluster has two tabs. General takes a new Name and Description, and shows an Encryption Key Settings fieldset -- Encrypt and Store Locally, Encrypt and Store on Key Server and a Key Server Profile picker -- which stays greyed out unless the cluster was created with encryption enabled. Advanced Settings holds the tunables covered under Monitoring cluster health.

Note what is not here: the member list, the network interfaces, and compression. Members are changed with Add Member and Remove Member, interfaces per member when it is added, and compression not at all.

Importing an existing cluster

Cluster Import adopts a Ceph cluster that already exists on the systems into QuantaStor's management. It is the path for taking over a cluster built outside QuantaStor, or re-adopting one after a management-database loss. See Ceph Cluster Import.

Creating OSDs and journal groups

Navigation: Scale-out Storage Configuration → Data & Journal Devices → Data & Journal Devices → Create OSDs & Journals (toolbar)

This one dialog builds both the OSDs and, if needed, the journal groups they will draw their WAL and metadata-DB devices from. Devices are moved from the available list on the left into one of two lists on the right using the arrow buttons between them.

The left-hand list has two tabs. Physical Devices lists unused physical disks across the cluster's members. Logical Devices lists ZFS zvols that were reserved as Ceph OSD backing devices -- volumes created with the Ceph OSD use case, which lets you build OSDs on top of a scale-up pool. Zvols may only become data devices; moving one towards the journal list is refused with "Logical Devices (zvols) can only be used as Data/OSD devices, not for Journal Groups. Journal Groups must be created from SSD or NVMe Physical Disks."

The Data/OSD Devices tab

The OSD Optimization via Journal Groups fieldset decides what journal devices the new OSDs get. All five controls are enabled when the dialog opens:

Control	Default	Effect
Auto-create Write-ahead Log (WAL) devices + size	checked, 2 GB	Carves a WAL device per new OSD. The size combo has exactly one entry, 2 GB, and the service caps the value there in any case.
Auto-create Meta-data Offload (MDB) devices + size	checked, 300 GB	Carves a BlueStore metadata-DB device per new OSD. The size combo offers 3, 6, 30, 60, 90, 180, 300, 480, 600, 900, 1200, 1500, 1800 and 2100 GB. The service floors the value at 3 GB.
Allow provisioning from pre-existing Journal Groups	checked	Lets the new OSDs take their WAL and MDB devices from journal groups that already exist, instead of requiring new journal devices in the same operation. This is the checkbox you need when adding OSDs to a system whose journal groups are already built.
Block creation of external WAL/MDB for SSD based OSDs	checked	Suppresses external journal devices for OSDs whose data device is already flash. Its tooltip is a warning against turning it off: "Unselecting the 'Block creation of external WAL/MDB for SSD based OSDs' is not recommended for SSD based OSDs."
Enable Dynamic Incremental Re-weighting	unchecked	Writes `osd_crush_initial_weight = 0` into the cluster configuration so new OSDs join at weight zero and are weighted up gradually, rather than attracting data the moment they appear. Tooltip: "Optimizes the newly created OSD weights".

The size choices are per OSD, and the journal group has to hold them all. With HDD data devices and a mirrored pair of flash devices per node, the arithmetic that matters is osds_per_node x (mdb_size + 2 GB) against the usable size of the journal group -- exceed it and OSD creation fails for want of journal space.

The Journal Group Devices tab

The Journal Group Creation Policy fieldset has two checkboxes, both checked by default:

Auto-create software RAID1 protected Journal Groups -- pairs the selected flash devices into mirrored groups. Leave this on. A journal group is a single point of failure for every OSD drawing WAL and MDB devices from it, so an unmirrored group turns one SSD failure into the loss of every OSD on that node.
Block RAID1 mirroring of mixed size SSD devices -- refuses to pair devices of different sizes. Its tooltip: "Unselecting the 'Block RAID1 mirroring of mixed size SSD devices' is not recommended for production environments. Uncheck for test environments only."

What Auto Config actually does

Auto Config is not a hint; it applies a specific algorithm, and it is worth knowing so you can tell whether its answer is the one you want. In order, it:

Refuses to run if anything is already selected -- "Cannot auto-config when some OSDs devices or Journal Group devices have already been selected." Clear both right-hand lists first.
Moves every Logical Device (zvol) to the Data/OSD list, since a zvol can be nothing else.
Finds the most common exact device size among the available physical disks and marks every disk of that size as a data device.
Finds the most common size again among what is left, and treats that group as the journal-group candidates.
Requires journal candidates to be strictly faster media than the data devices -- NVMe or PMEM behind SSD OSDs, flash behind HDD OSDs. If nothing faster exists it stops with "No suitable SSD media could be found for new Journal Group creation that is faster than the selected devices for new OSDs." If the data devices are already SSD and the candidates are not faster, it turns the WAL and MDB checkboxes off and tells you why: "No Journal Group SSD devices were selected because the OSDs were detected to be SSD media."
Forces an even journal-device count per node when mirroring is on, dropping the last candidate on any node with an odd count. A node with only one candidate produces "Unable to locate available pair of SSD devices for a new Journal Group on Ceph member systems ..."
Requires journal groups on every node that got OSDs. If they cannot be found everywhere it deselects everything with "Not enough devices are available to select as Journal Groups on all systems".
Sizes the MDB combo down to fit. It walks the size list from largest to smallest and picks the largest size where osds_per_node x (mdb_size + 2 GB) fits within the journal device and the MDB is smaller than half the data device. If even the smallest does not fit it selects 3 GB and warns that the configuration may not be ideal.

Always review what Auto Config selected before pressing OK. Its own tooltip says as much: "Auto Configure tries to create best possible configuration from the available disk, after which the configuration should always be reviewed before selecting OK."

What happens on OK

The operation runs as one task tree in a fixed order: journal groups are created first, one task per mirror pair, and only then are the OSDs created against them. So a single OK press can produce a batch of Create Ceph Journal Group tasks followed by Create Ceph Storage OSD and Create Ceph Journal Device tasks. Expect a few minutes, and expect the OSDs to appear in the grid before they are all fully up.

Adding OSDs to a cluster with existing journal groups is the same dialog with Allow provisioning from pre-existing Journal Groups left checked and no devices moved to the journal list.

Journal groups in detail

A Journal Group is a mirrored pair of flash devices on one cluster member. From it QuantaStor provisions two kinds of Journal Device, attached to individual OSDs:

Write-Ahead Log (WAL) devices absorb BlueStore's write-ahead log. An OSD can acknowledge a write once it has landed on low-latency flash, and the data reaches the slow device lazily. WAL devices are 2 GB.
Meta-data Database (MDB) devices hold BlueStore's metadata (RocksDB). They lift both read and write performance, because BlueStore's small metadata updates are exactly the workload an HDD is worst at.

The two are distinguished by size rather than by a flag: a journal device at or below 2.5 GiB is a WAL device and is named wal-lv-<id>; anything larger is a metadata-DB device named mdb-lv-<id>.

Underneath, a journal group is a single LVM volume group spanning both flash devices, and each journal device is an LVM RAID1 logical volume inside it. So the group's usable capacity is half the raw capacity of the pair, and every WAL and MDB device is individually mirrored across the two disks. Nothing here is mdraid.

Journal groups are recommended for HDD OSDs and not for SSD OSDs. With platter data devices, an external WAL and MDB device on flash is the single largest performance win available. With flash data devices the WAL and metadata live in a reserved slice of the OSD's own device, and pushing them onto separate flash usually buys nothing -- which is why Block creation of external WAL/MDB for SSD based OSDs is checked by default. Because journal groups sustain a high sustained write load for years, build them from datacenter-grade / enterprise flash only, and prefer NVMe.

Creating a journal group on its own

Navigation: Scale-out Storage Configuration → Data & Journal Devices → Data & Journal Devices → Create Journal Group (toolbar)

Ceph Journal Group Create takes a Ceph Cluster Member, a Primary Disk, a Use RAID1 Mirroring checkbox (checked by default) and a Secondary Disk. The secondary field is enabled only while mirroring is checked. With mirroring on, the two disks must be different devices of the same size -- otherwise you get "Ceph Journal Group must consist of two different devices with the same size."

Moving a journal device: Migrate Journal

Migrate Journal moves an existing WAL or metadata-DB device from one journal group to another journal group on the same node. It is the supported way to empty a journal group -- for instance to retire the flash pair underneath it -- because it relocates the journal device without disturbing the OSD that depends on it.

The migration copies the device's contents, moves the LVM tags, and rewrites the BlueStore label so the OSD looks for its journal in the new place. While it runs, QuantaStor puts the whole cluster into the same maintenance state described under Maintenance mode -- noout, norecover, nobackfill and nodown -- and clears it afterwards. That is worth knowing: a cluster can report itself in maintenance mode because of a journal migration nobody thought of as maintenance.

What can and cannot be removed

This is different from the scale-up case, where a ZFS metadata or dedup offload group can never be removed. Ceph journal devices are removable, with two different rules by granularity:

An individual WAL or MDB device cannot be deleted while an OSD is using it, and there is no force override. The attempt fails with "Ceph Journal Device '<name>' is associated with OSD '<osd>'. Only orphaned Journal Devices can be deleted." Accordingly, the Ceph Journal Device Delete dialog only lists devices with no OSD attached, and its header says so: "Delete unused Ceph journal devices one at a time from the selected cluster. ... Only '[Available]' journal devices are eligible to be deleted manually."
A whole journal group can be deleted, and Force will push past the in-use check. Without Force the delete is refused with "Ceph Journal Group delete operation failed. Please specify the FORCE flag to complete this operation." With Force it removes every journal logical volume in the group, releases the volume group, and returns both disks to available -- and it does not evacuate anything first.

So do not use Force on a journal group delete to get rid of a group that OSDs are still using. Migrate each journal device off it first, then delete the empty group. The Delete Journal Group dialog does not filter in-use groups out of its picker, so nothing in the interface will stop you selecting the wrong one.

Deleting an OSD is the other way journal devices go away. When an OSD is deleted its WAL and MDB logical volumes are removed and their space returns to the journal group's volume group, so the group does not accumulate orphans in the normal case.

When a journal disk fails

QuantaStor raises three alerts for journal groups: Ceph Journal Group Degraded (warning), Ceph Journal Group Missing (error), and Ceph Journal Group Repaired when it recovers -- "Ceph Journal Group '<name>' has returned to the healthy state with '<n>' attached devices."

The mirror is what keeps the OSDs running through a single flash failure. A device that comes back is refreshed automatically. A permanently failed journal disk has no automated rebuild onto a replacement -- the hot-spare mechanism covers failed OSDs, not failed journal-group members -- so plan on migrating the journal devices to another group, or rebuilding the group and its OSDs, with support's help if the cluster is large.

Creating the block Storage Pool

Navigation: Scale-out Storage Configuration → Scale-out Storage Pools → Block Storage → Create Block Storage Pool (toolbar)

Block, file and object pools are three separate dialogs reached from three separate toolbar groups; Create Block Storage Pool is the block one and it fixes the pool's use case as RBD. The dialog is one page:

Ceph Cluster -- which cluster to create the pool in.
Name -- defaults to block-pool-<n>. Names containing .rgw., _data or _metadata are rejected, because QuantaStor reserves those suffixes for the pools it derives for object and file storage.
Description -- free text.
Data Pool Layout → Type -- the redundancy layout. Four choices are offered: Erasure, Replica-3 (Triple-Mirror), Replica-4 and Replica-5. Replica-3 is the default, and there is no Replica-2 option -- two copies is not offered in the interface at all, because losing a second device while the first is still rebuilding loses data.
Set Manually / Data Chunks (K) / Code Chunks (M) -- enabled only for an Erasure layout, and selected by default when it is. K defaults to 2 (range 2-20) and M to 2 (range 1-7). The tooltips are Ceph's own definitions: K is "the number of data chunks, i.e. the number of chunks in which the original object is divided" and M "the number of coding chunks ... If there are 2 coding chunks, it means 2 OSDs can be out without losing data."
Use Profile -- the alternative to setting K and M by hand: pick a pre-defined or custom EC profile instead. Selecting this radio enables the profile picker and greys out the K and M controls.
Scaling Factor (%) -- defaults to 50, range 5 to 200. This is the placement-group allocation, and it is the one field on the dialog whose meaning is not obvious. See How the placement group count is chosen.
Force (Required for Erasure M=1) -- unchecked. Only needed to push past the refusal of a single coding chunk.

QuantaStor does not choose K and M for you on this dialog. A member-count rule exists elsewhere in the product -- it seeds the sliders on the Create Ceph EC Profile dialog -- but the block pool dialog defaults to Replica-3, and if you switch to Erasure the chunk counts sit at a static 2 and 2. Choose them deliberately.

The rules the service enforces on the layout:

Layout	Requirement	What you get if you do not meet it
Any pool	At least 2 OSDs in the cluster	"Ceph Cluster '%s' has '%lu' OSD's configured, which is less than the minimum replica '2' required for pool creation."
Replica-N	At least N cluster member systems	"Specified replica count requires at least '%u' Ceph Cluster member systems"
Erasure	At least K+M member systems, and at least three members for any erasure layout	"Specified values for erasure coded pool are invalid ... k='%d' and m='%d' requires at least '%d' members in the ceph cluster"
Erasure with M=1	Force flag	A warning that a single coding block "has a high probability of some data loss over the life span of the pool and is not recommended for any production deployment"

The member-count rule, rather than an OSD-count rule, is what usually bites: an erasure layout of K=4 + M=2 needs six cluster members, not six OSDs, because QuantaStor builds the EC profile with a host-level failure domain so that no two chunks of an object land on the same server.

Redundancy rules and EC profiles

The failure domain and the media class come from a CRUSH rule (for replicated pools) or an EC profile (for erasure pools), and both can be created explicitly from the EC Profiles & Rules toolbar group when the defaults are not what you want:

Create Ceph Redundancy/CRUSH Rule takes a Device Class (HDD, SSD or NVME) and a Bucket Type -- the failure domain -- of OSD, Host, Chassis, Rack or Datacenter, defaulting to Host. Only replicated rules can be created here; the Type control is fixed at Replicated.
Create Ceph EC Profile takes K and M, a Failure Domain (OSD, Host, Rack or Datacenter, defaulting to Host), a Plug-in (only jerasure is offered) a Technique (defaulting to reed_sol_van) and a Device Class. Note that where a technique other than reed_sol_van or cauchy_orig is chosen, M must be exactly 2.

If you create none of these, QuantaStor still builds sensible defaults. After OSDs are created it pre-creates one replicated CRUSH rule per device class present, rooted at default with a host failure domain -- replicated_hdd, replicated_ssd, replicated_nvme as applicable, plus _osd variants -- and assigns a new replicated pool the fastest one available. An erasure pool gets a generated profile named for its parameters, always with a host failure domain.

Rack, chassis and datacenter failure domains are only reached by naming them in a rule or profile explicitly. The default everywhere -- the pre-created replicated rules and every generated erasure profile -- is a host failure domain, and nothing derives a wider domain from how the grid is organised into sites or racks. So if your redundancy has to survive a rack or site outage, create the rule or profile that names that failure domain and select it on the pool; it will not happen by itself.

Modifying and deleting a pool

Navigation: Scale-out Storage Configuration → Scale-out Storage Pools → Scale-out Storage Pool (select + right-click) → Modify

Modify Redundancy (CRUSH) Rule Selection -- reached by right-clicking the pool, since there is no toolbar button -- changes the Name, Description, Pool Layout and Redundancy Rule. Pool Type is displayed and permanently read-only, and the layout combo is disabled for an erasure pool: a replicated pool can be moved between Replica-3, 4 and 5, but a pool cannot be converted between replicated and erasure.

Delete Block Storage Pool shows a read-only Storage Pool Information panel -- Scale-out Type, Data Pool Type, Storage Class, Object Count and Space Utilized -- and a Force (required when Scale-out Storage Pool is in use) checkbox. For a block pool the confirmation is explicit that this is not recoverable: "WARNING: All Scale-out Storage Volumes (RBDs) contained in the selected Storage Pool will be destroyed." For an erasure-coded pool, remember the second delete described under What an erasure-coded block pool actually creates.

How the placement group count is chosen

Placement groups are the layer between a Ceph pool and its OSDs; each PG is effectively one stripe across as many OSDs as the layout requires. Their number is set when the pool is created, and on QuantaStor it stays set.

You do not enter a PG count. You enter a Scaling Factor, and QuantaStor computes the count. The factor is a statement about how much of the cluster this pool is expected to use, and the tooltip is the clearest description of it:

"Scaling Factor adjusts the number of placement groups allocated for a given Storage Pool / use case. Where 100% implies that all placement groups will be allocated for a single use case. For Ceph Clusters, that will be used for multiple use cases (eg 70% for object storage pools and 30% NAS/CephFS storage pools), the scaling factor should be reduced accordingly with the sum for all use cases amounting to 100%."

So on a cluster that serves only scale-out block, use 100. On a cluster shared between block and file, split the budget -- and the default of 50 is a deliberately conservative half-share, not a recommendation for a single-use cluster.

The count itself comes out as:

pgs_per_osd     = average OSD size / 100 GiB, clamped to the range 40..100
osd_multiplier  = number of OSDs of the relevant media class, but never less than 32
divisor         = replica count            for a replicated pool
                = max(K + M, 3)            for an erasure pool

pg_count = round to a power of two ( pgs_per_osd x osd_multiplier / divisor x scaling_factor / 100 )

Three consequences are worth planning around:

The PG count does not vary with OSD count below 32 OSDs. The multiplier is floored at 32, so a 6-OSD cluster and a 30-OSD cluster of the same drive size get the same PG count for the same layout.
A Scaling Factor above 100 buys almost nothing. The effective percentage is clamped at 100 in the service, so values in the 100-200 part of the slider have no effect on a plain block pool -- even though the dialog raises a confirmation implying they will over-allocate.
The PG autoscaler is off. QuantaStor sets pg_autoscale_mode off on every pool it creates, and the shipped Ceph defaults carry osd pool default pg autoscale mode = off as well. Nothing will re-tune the count later, so the Scaling Factor you choose at create time is the one you keep unless someone changes it by hand.

A worked example, measured on a four-node cluster with sixteen 10 GiB SSD OSDs: a Replica-3 block pool at the default Scaling Factor of 50 came out at 256 placement groups, size 3 min_size 2, on the replicated_ssd CRUSH rule, with autoscale_mode off and the rbd application tag. The OSD count did not enter the arithmetic -- sixteen is below the floor of 32.

What an erasure-coded block pool actually creates

An erasure layout is more space-efficient than replication, but RBD cannot keep an image's own metadata in an erasure-coded pool. So when you create an erasure-coded block pool, QuantaStor creates two Ceph pools:

<name>_ec_data -- the erasure-coded data pool, with allow_ec_overwrites enabled.
<name> -- a companion replicated pool, always 3 copies with a minimum of 2, at half the scaling factor you asked for.

RBD images are created in the replicated pool with the erasure pool as their --data-pool: metadata in the replicated pool, bulk data erasure-coded. Both rows appear in the Scale-out Storage Pools grid, which is why an erasure block pool looks like two pools with one name between them. When you provision a Storage Volume you select the pool with the plain name, not the _ec_data one.

Measured on the same four-node cluster, K=2 + M=2 produced an _ec_data pool at size 4 min_size 3 with 128 PGs and the companion replicated pool at size 3 min_size 2 with 128 PGs.

Deleting an erasure-coded block pool takes two operations. Deleting the pool by its plain name removes only the replicated half; the _ec_data pool is left behind, still holding its placement groups, and has to be deleted separately. After deleting an erasure-coded block pool, check the Scale-out Storage Pools grid for a leftover <name>_ec_data row and delete that too. Tracked as QSTOR-12460.

Provisioning Storage Volumes

Navigation: Storage Management → Storage Volumes → Storage Volume → Create (toolbar)

Provisioning from a scale-out block pool uses the same Create Storage Volume dialog as a scale-up pool, and Storage Volumes documents it field by field. Pick the block pool in the Storage Pool combo -- the Storage Pool Information panel then reads Pool Type: Scale-out Block -- and the dialog adapts.

What differs on a Ceph pool:

Block Size is the RBD object size, not a ZFS block size, and it defaults to 4M. Set it deliberately: too small an object size produces an enormous object count and a LUN whose queue limits are derived from it, which is a well-known way to make a volume unusably slow.
Stripe Size and Stripe Count are enabled only for a Ceph pool and only at create time. They split each RBD object into that many chunks of that size. Both must be powers of two, must be set together, and their product may not exceed the block size. auto is the default and the right answer unless you have a specific reason.
Namespace becomes available when the selected pool has Ceph pool namespaces defined -- see below.
Enable Direct RBD Mapping/Access appears only for a Ceph pool -- see Presenting volumes to hosts.
% Reserved, Compression, Sync Policy and the two Cache Policy settings are greyed out. They are ZFS pool properties; an RBD image is always thin-provisioned and there is no RBD equivalent of a ZFS reservation.

RBD images are created in format 2 with the layering, exclusive-lock, object-map and fast-diff features enabled. Snapshots, clones, rollback and resize all work, with the differences Storage Volumes describes -- notably that an RBD image can be grown but never shrunk, and that a snapshot of a snapshot is not supported on Ceph.

Pool namespaces

Navigation: Scale-out Storage Configuration → Scale-out Storage Pools → Block Storage → Create Ceph Pool Namespace (toolbar)

A Ceph Pool Namespace partitions one RBD pool into isolated name spaces, so that volumes for different tenants or workloads live in the same pool without seeing each other. Namespaces apply to block pools only; the create dialog's pool picker lists nothing but RBD pools. Once a namespace exists it becomes selectable in the Namespace combo on Create Storage Volume. Namespaces are also the natural unit to scope a Ceph client keyring to, when handing direct RBD access to a tenant.

Presenting volumes to hosts

Assignment is the same as for any Storage Volume -- add the initiator as a Host with its IQN, then assign the volume to it -- and Storage Volumes covers the dialogs. What is specific to scale-out block is how the volume is reachable.

Every member of the Ceph cluster exports the volume. When a volume is assigned, each cluster member maps the RBD locally as a /dev/rbdN device and registers the same iSCSI target IQN for it. Verified on a four-node cluster: all four nodes showed the identical target name and their own mapping of the image. So an initiator that logs in to the portal IP of several nodes gets several paths to one LUN, and multipathing on the client handles a node outage.

This is the point of scale-out block, and getting it wrong costs you the availability you built the cluster for. Configure the initiator -- VMware, Windows, Linux -- to log in to a target port IP on each system in the Ceph cluster and to run multipath over the result. An initiator connected to a single node has no path left when that node reboots, however healthy Ceph is underneath. See Multipath Configuration for the client-side settings.

A discovery against one node returns only the targets for volumes assigned to that initiator's IQN, so an initiator sees its own LUNs and nothing else.

For Fibre Channel, the same volume is presented through the FC target ports of each member; see Fibre Channel Target Port Management.

Direct RBD access

Enable Direct RBD Mapping/Access on the Advanced Settings tab of Create Storage Volume hands the image to the client to map itself with the RBD client software instead of exporting it as a LUN. Its tooltip states the consequence plainly: "Enables remote hosts and clients to use RBD mapping via the RBD client software to access the block storage device. This also permanently disables access via iSCSI, FC, or NVMeoF protocols to this RBD based Storage Volume."

Two things to understand before ticking it. It is create-time only and permanent -- there is no such control in Modify Storage Volume, and a clone of a direct-RBD volume is itself unmapped. And a direct-RBD volume never appears in the host assignment dialogs at all, because there is nothing to assign; the client authenticates to Ceph with a keyring instead. Use Create Ceph Client Keyring to issue one and Export Ceph Client Keyring / Export Ceph Client Conf to hand the client its credentials and cluster configuration.

Growing the cluster

A Ceph cluster grows by adding OSDs to existing members, or by adding members. Neither requires downtime, and both trigger a rebalance.

Adding OSDs. Use Create OSDs & Journals as described above, leaving Allow provisioning from pre-existing Journal Groups checked if you are not adding new journal devices with them. Add OSDs in comparable numbers across members so the rebalance spreads evenly; adding a large batch to one node and none to the others just moves data onto that node.

Adding a member. First add the system to the grid, then:

Navigation: Scale-out Storage Configuration → Scale-out Storage Clusters → Scale-out Cluster Management → Add Member (toolbar)

Add Member asks for the Ceph Cluster, the Storage System, and its Client and Backend interfaces. QuantaStor pre-selects the interfaces from their IP addresses -- check them rather than accepting them, and if they are empty the system does not yet have valid addresses on both networks. Then create OSDs on the new member.

The hardware need not be identical to the existing members, but keeping it comparable -- and the new OSDs the same size as the old ones -- is what lets the rebalance distribute data evenly.

Adding monitors. The three monitors created with the cluster are enough up to roughly ten to sixteen members. Beyond that, go to five for extra fault tolerance. What decides it is not really the node count but the failure domains: if the cluster spans racks, put a monitor in each rack. Three monitors in one rack means a rack power outage takes storage access down with it. Use Add Monitor and Remove Monitor in the Service Management toolbar group.

Monitor counts step 1, 3, 5, 7 -- an even number never helps quorum. Monitor removal is protected: at three or fewer healthy monitors it requires the force flag, at one it is refused outright, and unless forced Ceph must agree the daemon is safe to stop. That check, not anything on the member-removal path, is what protects your quorum, so do not force your way past it.

Removing OSDs safely

An OSD can be removed at any time and its data will be rebuilt across the rest of the cluster. Doing it in the wrong order is how a healthy cluster becomes an unavailable one.

QuantaStor will not delete an OSD while any Ceph pool exists unless you force it. The refusal is explicit: "There are underlying ceph pools using OSDs. Please use 'force' flag if you still intend to delete OSDs". That is the reason the Force (required when Ceph OSD is in use) checkbox exists on the delete dialog -- in practice it is required for every delete on a cluster that is actually serving storage.

Force skips the drain. A plain OSD delete reweights the OSD down to zero in graded steps first; a forced delete goes straight to marking it out and destroying it. Since force is required whenever pools exist, the drain effectively never happens on a production cluster unless you do it yourself. So:

Check there is room. Removing 5 of 30 OSDs raises utilization on the rest by roughly a fifth. If the remaining OSDs cannot absorb it, expand first and retire afterwards.
Check redundancy. In multi-site and rack-aware configurations, confirm the removal cannot drop a pool below the copies it needs to stay writable.
Reweight to zero and wait. qs osd-rw --ceph-cluster=<cluster> --osd-weight-list=<osd>:0 --flags=force, or the Reweight OSDs dialog. Reducing a weight requires the force flag and says why: "OSD weight reduction is a dangerous action and may cause cluster unavailability. Use the force flag to continue or contact support for further assistance." QuantaStor also runs two pre-flight checks against a simulated CRUSH map, refusing a reweight that would leave too many PGs per OSD or fail to satisfy the CRUSH rules.
Wait for the rebalance to finish -- the OSD's placement group count drops to zero and the cluster returns to active+clean.
Then delete it, with Force.

Navigation: Scale-out Storage Configuration → Data & Journal Devices → Data & Journal Devices → Multi-Delete OSD (toolbar)

Ceph OSD Multi-Delete takes a Ceph Cluster and Ceph Cluster Member, then a tick-list of that member's OSDs, and pre-selects any that are in an error, warning or offline state -- convenient for clearing out failed devices, but check the selection before pressing OK. The confirmation lists the OSD names.

Deleting an OSD removes its WAL and metadata-DB devices, releases that space back into the journal group, and returns the data disk to available so it can be reused. Verified end to end on a live cluster: reweight to zero, force delete, disk back to available, then a new OSD built on the same disk drawing fresh WAL and MDB devices from the existing journal group.

Replacing a failed device

Replacing a drive is a two-step operation with its own pair of dialogs, and it is the only path that genuinely drains an OSD before taking it out.

Replace Prepare OSD -- select one OSD. QuantaStor waits for Ceph to agree the daemon is safe to stop, stops and masks it, marks it out, waits for Ceph to agree the data is safe to destroy, and then destroys the OSD while keeping its ID reserved. Its confirmation says what it does: "Replace Prepare will drain the Ceph OSD of all data. Are you sure you want to prepare the selected OSD for replacement?" This waits as long as it takes -- polling with a 24-hour no-progress timeout -- so run it and let it work.
Replace OSD -- select the prepared OSD and the replacement disk. The dialog lists only OSDs that are ready to replace. The replacement must be at least as large as the original, or the Force checkbox is required: "Force flag needed when replacement disk is smaller than the current OSD disk."

Two checkboxes on Replace OSD are worth understanding:

Replace using same disk -- reuses the original device after clearing its previous LVM configuration. Use it when the drive itself is fine and the OSD is not. Ticking it disables the disk picker.
Recreate Journals -- rebuilds the WAL and MDB devices instead of reusing them. "Required for encrypted OSDs with journal devices." On an encrypted cluster the dialog forces this on and makes it read-only, because the journal devices share the OSD's encryption key.

If no replacement disks are available at all, the dialog disables the disk picker and forces both checkboxes on -- the only remaining option is to rebuild on the same device.

For failed media generally, use Replace Prepare rather than delete. It removes the device from the cluster gracefully and leaves the cluster ready to accept its replacement.

Maintenance mode

Maintenance mode tells Ceph to stop reacting to things going away, which is what you want while a node is deliberately down. Enter Ceph Maintenance Mode sets four Ceph flags -- nodown, noout, nobackfill and norecover -- and Exit Ceph Maintenance Mode clears them. Without them, rebooting a node for ten minutes starts a full rebalance that has to be undone when it comes back.

QuantaStor reports the cluster as being in maintenance mode only when all four flags are present. Two practical consequences: setting some of them by hand at the console will not show up as maintenance mode in the interface, and a journal migration -- which sets the same four flags for its own duration -- makes the cluster report itself in maintenance mode while it runs.

Monitoring cluster health

The Health tile carries Ceph's own health string, and the Health Details tab below the dashboard lists the individual Ceph health checks behind it, with a severity per row. qs cc-health-event-list returns the same rows, with the Ceph check name -- BLUESTORE_SLOW_OP_ALERT and the like -- which is the string to search Ceph's documentation for.

The Placement Groups Health ring is the one to watch. It aggregates every PG of every pool in the cluster. A node or a set of OSDs going away turns part of it orange or red; hover a segment for detail on which OSDs are involved. Because the ring covers all pools, a problem in a file or object pool shows up here too.

Cluster Storage Capacity is raw. A replica-3 pool consumes three times what its volumes hold, and an erasure K+M pool consumes (K+M)/K times. Read pool-level usable space from the Scale-out Storage Pools grid instead.

The three fullness ratios

On Modify Cluster → Advanced Settings, the Drive Freespace Threshold Settings fieldset holds the three thresholds Ceph uses as an OSD fills up:

Setting	Default	What happens at the threshold
OSD Near Full Ratio	85%	"Cluster storage percentage at which a warning will be generated to indicate the cluster is near full."
OSD Backfill Full Ratio	90%	"Cluster storage percentage at which cluster blocks backfills from starting."
OSD Full Ratio	95%	"Cluster storage percentage at which cluster is full and clients are stopped from writing data"

They must stay in ascending order -- near-full below backfill-full below full. The sliders accept each value independently with no cross-field check, so an out-of-order set is only rejected when the task runs.

Treat near-full as the point to act, not the point to note. Once an OSD passes the backfill-full ratio the cluster can no longer rebalance onto it, which removes exactly the mechanism you would rely on to recover from a device failure.

The two OSD grace periods

The same tab carries the Device Monitoring Settings fieldset: OSD Down Grace Period (seconds), the time a monitor waits without hearing from an OSD before marking it down, and OSD Out Grace Period (seconds), the further time a down OSD is left alone before being marked out and its data rebuilt elsewhere. The out period is the more consequential of the two: too short and a node reboot triggers a full rebalance, too long and a genuinely dead device leaves its placement groups degraded.

(Note: the values these fields display on a cluster where they have never been set are the dialog's own, not the cluster's -- an unset timer leaves Ceph's default in place. Read the effective values from the cluster itself if it matters, and see QSTOR-12458.)

Adjust Clocks in the Scale-out Cluster Management group re-synchronizes the clocks of every member, for when clock skew has already been reported.

CLI command reference

Every operation above is scriptable. The commands below are the scale-out block subset; QuantaStor CLI Command Reference carries the full argument lists.

Command	Purpose
`qs cc-create`, `qs cc-modify`, `qs cc-delete`, `qs cc-import`	Create, modify, delete and import a Ceph cluster.
`qs cc-list`, `qs cc-get`, `qs ccm-list`, `qs ccm-get`	Enumerate clusters and members.
`qs cc-amn`, `qs cc-rmn`	Add and remove cluster members.
`qs cc-health-event-list`, `qs cc-srv-event-list`, `qs cc-srv-event-delete`	Health checks and daemon crash events.
`qs cc-fcs`	Re-synchronize member clocks (the Adjust Clocks button).
`qs cc-set-coord`	Set the cluster's coordinator node, which drives cluster-wide tasks.
`qs cmon-add`, `qs cmon-remove`, `qs cmon-list`	Monitor management.
`qs osd-multi-create`	Create OSDs and, if needed, their journal groups.
`qs osd-list`, `qs osd-get`, `qs osd-modify`	Enumerate and describe OSDs.
`qs osd-rw`	Reweight OSDs. Takes `--osd-weight-list=<osd>:<percent>` pairs; reducing a weight needs `--flags=force`.
`qs osd-delete`	Delete one or more OSDs. Needs `--flags=force` while any pool exists.
`qs osd-rep-prep`, `qs osd-rep`	Drain and replace a device.
`qs osd-serv-up`, `qs osd-identify`	Start, stop or restart the OSD daemon; blink the drive LED.
`qs osd-krep`	Replace the encryption key of encrypted OSDs.
`qs cjg-create`, `qs cjg-delete`, `qs cjg-list`	Journal group management.
`qs cj-list`, `qs cj-delete`, `qs cj-migrate`	Journal device management. Delete only works on an orphaned device.
`qs cpool-create`, `qs cpool-modify`, `qs cpool-delete`, `qs cpool-list`	Pool management. Pass `--use-case=block` explicitly for a block pool.
`qs cpns-create`, `qs cpns-delete`, `qs cpns-list`	Pool namespaces.
`qs ecp-create`, `qs ecp-delete`, `qs ecp-list`	Erasure-coding profiles.
`qs ccr-create-rep`, `qs ccr-create-era`, `qs ccr-delete`, `qs ccr-list`	Redundancy (CRUSH) rules.
`qs cckr-create`, `qs cckr-export`, `qs ccc-export`	Issue and export credentials for direct RBD clients.
`qs volume-create`, `qs volume-assign`, `qs host-add`	Provision an RBD volume and present it to a host.

Creating a pool and a volume on it from the console looks like this:

qs cpool-create --name=block-pool-1 --ceph-cluster=ceph-cluster-1 \
    --use-case=block --max-replicas=3 --scaling-factor=100

qs volume-create --name=vol1 --size=2T --pool=block-pool-1 --blocksize-kb=4096

qs host-add --hostname=esx-01 --iqn=iqn.1998-01.com.vmware:esx-01
qs volume-assign --volume=vol1 --host-list=esx-01

Set --blocksize-kb explicitly when creating a volume in a scale-out block pool. On a Ceph pool that value is the RBD object size, and 4096 (4 MiB) is the size the web interface selects for you.

Setting the cluster coordinator has no dialog and is CLI or API only: qs cc-set-coord. Conversely the CLI exposes pool options the dialog does not, including --min-replicas, --cache-tier and --compression-algo.

Troubleshooting

Clock skew

The most common warning on a newly built cluster. Monitors need closely agreeing clocks, and the causes are almost always one of: no NTP server configured on one or more systems, NTP blocked by a firewall, only one NTP server configured so its outage causes drift, or the NTP servers themselves being unreachable. Check every member's NTP configuration -- see Storage System -- and use Adjust Clocks to re-synchronize once the configuration is right. Configuring NTP and time zone for the whole grid at once is quicker than per system; see Grid Configuration.

A member's OSDs all go offline together

Look at the network before the storage. All nodes must carry the same networks on the same port names -- if one node has the back-end subnet on a different interface than its peers, that node's OSDs cannot replicate. Verify the cabling and each node's port configuration under Network Ports. Ceph restores OSD status and rebalances on its own once connectivity is back.

A device has failed

Use Replace Prepare OSD followed by Replace OSD, not delete. Replace Prepare is the path that drains the OSD and hands the cluster a graceful replacement.

A whole node has failed

Replace the server. If the data devices themselves are undamaged, move them into the replacement chassis -- the OSDs can then be recovered rather than rebuilt from scratch, which on a large node is the difference between minutes and days of backfill. If the node and all its OSDs are lost, contact OSNEXUS support for help removing the node and its OSDs cleanly; a replacement node can be added to the cluster at any time.

OSD creation failed part way through

A multi-OSD create builds journal groups first and OSDs afterwards, and a failure in the later stage does not currently unwind the journal groups it already created. Check the Journal Groups tab for groups with no journal devices in them, and the Journal Devices tab for devices marked [Available]. Remove the stranded ones with Delete Journal Group and Ceph Journal Device Delete before retrying, or the disks will not be offered as available again. Tracked as QSTOR-12450.

"OSD(s) experiencing slow operations in BlueStore"

A BlueStore health check, not a QuantaStor one, and it points at the media rather than the cluster. On HDD OSDs without external WAL and MDB devices it is the expected symptom -- see Journal groups in detail. Otherwise look for a failing device, and check whether the journal group behind the affected OSDs is degraded.

A LUN is unreachable after a node outage

Almost always a client-side multipath problem rather than a cluster one: the initiator was logged in to one node only. Check that it has sessions to a target port on every cluster member, and see Multipath Configuration. Confirm the cluster itself is healthy on the dashboard first -- if placement groups are not active+clean, fix that before looking at the client.

Console-level Ceph configuration

Everything on this page is available through the web interface, the CLI and the REST API. Custom Ceph settings and hand-written CRUSH map changes can be made at the console for special configurations, but we recommend contacting OSNEXUS support or pre-sales engineering before making any major changes -- particularly to a CRUSH map, where a mistake redistributes every object in the cluster. QuantaStor writes ceph.conf itself and it should not be hand-edited; for the configuration files QuantaStor does expose, see QuantaStor Configuration Files.

Related pages

Scale-out File Setup (ceph) -- scale-out NAS on the same cluster, using CephFS
Scale-out Object Setup (ceph) -- S3 object storage on the same cluster
Storage Volumes -- provisioning, snapshots, clones and host assignment for volumes
Storage Pools -- the scale-up (ZFS) Storage Pool, for comparison
Grid Configuration -- creating the grid and adding systems, the prerequisite for a Ceph cluster
Network Ports -- front-end and back-end port configuration, bonding and VLANs
Storage System -- per-system setup including DNS and NTP
Multipath Configuration -- client-side multipathing across cluster members
Fibre Channel Target Port Management -- presenting volumes over FC
Physical Disks/Devices -- disk discovery and per-disk operations
Encryption Bypass -- excluding self-encrypting media from cluster encryption
Ceph Cluster Import -- adopting an existing Ceph cluster
Performance Monitoring -- watching the cluster under load
Call-home / Alerting -- where cluster health alerts are delivered
QuantaStor Configuration Files -- the configuration files QuantaStor ships and how overrides resolve
QuantaStor CLI Command Reference -- full argument lists for every command above

Verified against QuantaStor 6.9.0.