Ceph OSDs on ZFS Storage Pools
This page explains how to run Ceph OSDs on ZFS. You build a ZFS Storage Pool on each node (dRAID layouts included), carve Storage Volumes (zvols) from it, and use those zvols instead of whole Physical Disks as the backing devices for the OSDs of a scale-out cluster. Ceph then sees one OSD per zvol, and ZFS provides the disk-level RAID, distributed spares and rebuilds underneath.
How it fits together
A normal Ceph OSD in QuantaStor consumes one whole Physical Disk. A ZFS-backed OSD consumes one Storage Volume instead:
- A ZFS Storage Pool on a cluster member aggregates that node's disks, for example in a distributed-parity (dRAID) layout.
- Storage Volumes (zvols) in that pool are reserved for Ceph through their Use Case setting, Ceph OSD.
- The OSD create dialog offers those zvols on a separate Logical Devices tab, and QuantaStor creates one BlueStore OSD on each.
The rest of the Ceph configuration is unchanged. Cluster setup, Ceph pools, CRUSH rules and the block, file and object services all work the same way as with disk-backed OSDs; see Scale-out Block Setup (ceph), Scale-out File Setup (ceph) and Scale-out Object Setup (ceph).
Requirements:
- The node that owns the zvol must be a member of the Ceph cluster the OSD is created in.
- The pool must be a ZFS (scale-up) pool, and it must be started.
- Each zvol backs exactly one OSD.
Step 1: Create the ZFS Storage Pool
Create the pool on the General tab as you would any ZFS pool: pick the Storage System, the disks and a RAID Type. Then set Use Case on the Advanced Settings tab.
Use Case
| Value | Effect |
|---|---|
| Default | A general-purpose pool. Storage Volumes and Network Shares are provisioned as usual. You can still reserve individual zvols in it for Ceph (see Step 2). |
| Ceph OSD | Reserves the whole pool for OSD backing zvols. Every Storage Volume created in it takes the Ceph OSD use case automatically, and none of them can be presented to hosts over iSCSI, FC or NVMe-oF. |
The Ceph OSD use case is offered only for ZFS pools, because zvols are a ZFS construct. The Use Case column in the Storage Pools list shows which pools are reserved.
The other Advanced Settings (I/O Profile, Enable Compression, Enable SSD Auto Trim, Block Size Offset (ashift)) work as they do for any pool; see Storage Pools.
dRAID layouts
The RAID Type list offers distributed RAID once enough unused disks are selectable on the Storage System. dRAID spreads data, parity and spare capacity across every disk in the group, so a rebuild after a disk failure reads from and writes to all of the remaining disks rather than one replacement drive.
| RAID Type | Parity | Distributed spares | Shown when at least |
|---|---|---|---|
| Distributed Single-parity (DRAID1 1VS / 2VS / 3VS) | 1 | 1, 2 or 3 | 10 disks are available |
| Distributed Double-parity (DRAID2 1VS / 2VS / 3VS) | 2 | 1, 2 or 3 | 16 disks are available |
| Distributed Triple-parity (DRAID3 1VS / 2VS / 3VS) | 3 | 1, 2 or 3 | 20 disks are available |
The non-distributed layouts (RAID10, RAIDZ1, RAIDZ2, RAIDZ3, RAID0) can also back OSD zvols. For layout guidance, see Designing Systems.
From the CLI, set the use case with --pool-use-case on qs pool-create. The dRAID layouts are named DRAID1_1S through DRAID3_3S for --raid-type:
qs pool-create --name=osdpool1 --disk-list=[unused] --raid-type=DRAID2_2S --pool-use-case=ceph-zvol-osd
Step 2: Create the OSD backing zvols
Create one Storage Volume for each OSD you want on the node. On the Advanced Settings tab, set Use Case:
- In a pool reserved for Ceph OSD, the field is already set to Ceph OSD and cannot be changed.
- In a Default ZFS pool, choose Ceph OSD to reserve this zvol as an OSD backing device. Other volumes in the pool remain ordinary volumes.
- On a Ceph (scale-out) pool, the field stays at Default and is disabled.
Use the Batch Create count on the same tab to create several zvols of the same size in one step.
Choose the Block Size with care. It cannot be changed after the volume is created, and QuantaStor matches the OSD's BlueStore allocation unit (bluestore_min_alloc_size) to it when the OSD is created. The allocation unit is held between 4K and 64K: a 4K or smaller block size gives 4K, and anything 64K or larger gives 64K.
The use case is fixed once the volume exists. A zvol reserved as a Ceph OSD backing device:
- cannot be assigned to a host. The assignment fails with a message saying the device is consumed directly by the Ceph OSD running on it.
- cannot be snapshotted or cloned once an OSD is running on it, because the copy would carry a duplicate of the OSD's LVM physical volume.
From the CLI, use --volume-use-case, --count and --blocksize-kb on qs volume-create. In a pool created with --pool-use-case=ceph-zvol-osd, the volumes inherit the use case and you can leave --volume-use-case out:
qs volume-create --name=osdvol --size=4T --pool=osdpool1 --count=8 --blocksize-kb=64 --volume-use-case=ceph-zvol-osd
Step 3: Create the OSDs
The Create OSDs & Journals dialog lists Physical Disks on one tab and zvols on a Logical Devices tab. A zvol appears on the Logical Devices tab when it:
- carries the Ceph OSD use case,
- is in the Normal state,
- is not a snapshot, and
- does not already back an OSD.
Move the zvols you want into the Data/OSD list. Auto Config moves every listed zvol into the Data/OSD list before it chooses Physical Disks for OSDs and Journal Groups. You can mix Physical Disks and zvols in a single create.
Zvols can only be OSD data devices. If you try to move one into the Journal Groups list, the dialog refuses with Logical Devices (zvols) can only be used as Data/OSD devices, not for Journal Groups. Journal Groups must be built from SSD or NVMe Physical Disks.
When QuantaStor decides where to put an OSD's WAL and metadata (DB), it treats a zvol as spinning media. With the dialog's Auto-create Write-ahead Log (WAL) devices and Auto-create Meta-data Offload (MDB) devices options enabled, a zvol-backed OSD gets WAL and MDB devices from SSD/NVMe Journal Groups, just as an HDD-backed OSD does. For how Journal Groups are sized and used, see Designing Systems.
From the CLI, pass the zvols to qs ceph-osd-multi-create with --volume-list. It can be combined with --physical-disk-list:
qs ceph-osd-multi-create --ceph-cluster=cluster1 --volume-list=osdvol_1,osdvol_2,osdvol_3
When the create task starts, QuantaStor checks every zvol. It rejects a zvol that:
- lacks the Ceph OSD use case,
- is a snapshot,
- is assigned to a host,
- already backs an OSD, or is listed twice,
- is not on a ZFS pool,
- is on a pool that is not started, or
- is owned by a Storage System that is not a member of the cluster.
What happens on create
- A zvol is a freshly provisioned device, so QuantaStor skips the quick-format it applies to Physical Disks.
- QuantaStor creates a BlueStore OSD on the zvol with the cluster's OSD encryption and compression settings, and sets the allocation unit from the zvol's block size.
- After the OSD is up, QuantaStor renames the zvol to
vol-<storage system>-osd.<N>, so the zvol and its OSD line up by name in the web interface and the CLI. If the rename fails, the OSD is unaffected and the zvol keeps its old name. - In the scale-out tree, a ZFS-backed OSD has its own icon, which tells it apart from a disk-backed OSD.
Startup and pool availability
A ZFS-backed OSD can only start once its zvol exists, so QuantaStor imports these pools early in boot, before the Ceph OSD services start and also handles unlocking SED encrypted devices at this time. Stopping or exporting a pool takes all of its zvols offline, along with every OSD on them. Plan pool maintenance as you would for a node outage: Ceph marks those OSDs down and recovers from the replicas or erasure-coded chunks held on other nodes.
Related pages
- Storage Pools
- Storage Volumes
- Scale-out Block Setup (ceph)
- Scale-out File Setup (ceph)
- Scale-out Object Setup (ceph)
- Designing Systems
- QuantaStor CLI Command Reference
Verified against QuantaStor 6.9.0.