Qadmin: osn-seo-utilities: docs-ceph-over-zfs-zfs-draid-pool-reserved @ d389deda0b81 (approved in the portal)

2026-10-08T00:00:04Z

osn-seo-utilities: docs-ceph-over-zfs-zfs-draid-pool-reserved @ d389deda0b81 (approved in the portal)

New page

[[Category:admin_guide]]
Ceph over ZFS runs Ceph OSDs on ZFS Storage Volumes (zvols) instead of directly on Physical Disks. There are a number of benefits of this architecture but the biggest is the double layer durability localizes the rebuild from failed drives so that the Ceph cluster doesn't need to absorb that load and production workload impact. The setup is simple, on each node you provision a Scale-up (ZFS) Storage Pool, ideally in a declustered RAID (dRAID) layout when you have 40+ drives per node. These pools become the backing store for Ceph OSDs, which become one-to-one mapped to Storage Volumes (zvols). So after the pool is created you'll batch create something like 16x, 24x, or 32x Storage Volumes from each pool. We call this the P:V or physical-to-virtual ratio and usually you'll want to aim for anything from 2:1 to 4:1. A 3:1 ratio means that for a node with 90x HDDs one would make that into a pool and then provision 30x (90:30 = 3:1) Storage Volumes to be turned into OSDs. This reduction also reduces the number of OSD processes running on each node and the amount of RAM you need per system. As an example with all 90x HDDs getting directly turned into OSDs we'd allocate 4GB to 6GB of RAM per OSD which amounts to 360GB to 540GB of RAM. In contrast, with a P:V of 3:1 we are producing 30x OSDs rather than 90x OSDs for roughly the same capacity. 30x OSDs at 8GB RAM per OSD is just 240GB of RAM so we're roughly cutting the RAM requirements in half. The same goes for the required CPU power.. we'd normally allocate about 1 to 1.5GHz per HDD based OSD. That's 90GHz to 135GHz if using the 90x HDDs directly as OSDs. Allocating 2GHz for each of the 30x zvol based OSDs puts us at just 60GHz. Granted, with the second filesystem (ZFS) at work underneath Ceph you'll need to reserve CPU for it and RAM for its ARC layer so the reduction can be somewhat offset but you should still see an overall reduction of 25% in CPU and RAM requirements.

{| class="wikitable"
! Section !! Purpose
|-
| [[#How it works|How it works]] || The Use Case setting and what it changes
|-
| [[#Requirements|Requirements]] || What must be in place before you start
|-
| [[#Step 1: Create a Storage Pool reserved for Ceph OSDs|Step 1]] || Reserve a ZFS pool for Ceph OSDs
|-
| [[#Step 2: Create the zvols|Step 2]] || Carve the zvols that back the OSDs
|-
| [[#Step 3: Create the OSDs|Step 3]] || Build one OSD on each zvol
|-
| [[#Operating notes|Operating notes]] || Boot order, LVM visibility and missing OSDs
|}

__TOC__

== How it works ==

A ZFS Storage Pool carries a '''Use Case''' that is set when the pool is created and cannot be changed afterwards:

{| class="wikitable"
! Use Case !! Meaning
|-
| Default || The pool provisions Storage Volumes and Network Shares as usual.
|-
| Ceph OSD || The pool is reserved for zvols that back Ceph OSDs. Every Storage Volume created in it inherits the Ceph OSD use case.
|}

A Storage Volume with the Ceph OSD use case is consumed directly by the Ceph OSD running on it. QuantaStor never presents it to hosts over iSCSI, FC or NVMe-oF: assigning one to a host fails with the message that the volume ''is reserved as a Ceph OSD backing device''.

The setup has three steps, each covered below:

# Create a ZFS Storage Pool with the Use Case set to '''Ceph OSD'''.
# Create the zvols in that pool, one per OSD you want.
# Create the OSDs from the zvols in the Ceph cluster.

The Storage Pools list has a '''Use Case''' column, so you can tell a Default pool from one reserved for Ceph OSDs at a glance.

== Requirements ==

* The pool must be a ZFS pool. QuantaStor refuses the Ceph OSD use case for any other pool type, and refuses a Ceph OSD zvol in a non-ZFS pool.
* The Storage System that owns the pool must be a member of the Ceph cluster you create the OSDs in, because that system creates the OSD on its own zvol.
* The pool must be started when you create the OSDs.
* Each zvol backs exactly one OSD. A snapshot, or a volume that is assigned to hosts, cannot back an OSD.

See [[Scale-out Block Setup (ceph)]] or [[Scale-out Object Setup (ceph)]] for creating the Ceph cluster itself.

== Step 1: Create a Storage Pool reserved for Ceph OSDs ==

{{Navigation|Storage Management → Storage Pools → Create ''(toolbar)'' → Advanced Settings ''(tab)''}}

[[File:Docs-docs-ceph-over-zfs-zfs-draid-pool-reserved-pool-use-case.png|thumb|right|800px|The Use Case list on the Advanced Settings tab of Create Storage Pool.]]

Create the pool as described on [[Storage Pools]]: name it, pick the disks on the '''General''' tab and choose a '''RAID Type'''. The DRAID layouts suit this use case because the OSD layer is designed around a DRAID pool of HDDs (see [[#How QuantaStor treats a zvol-backed OSD|How QuantaStor treats a zvol-backed OSD]]).

On the '''Advanced Settings''' tab, set '''Use Case''' to '''Ceph OSD'''. The other options on the tab (I/O Profile, Enable Compression, Enable SSD Auto Trim, Block Size Offset (ashift)) work as they do for any pool.

The Use Case cannot be changed once the pool exists, so set it here.

From the CLI, use <code>[[QuantaStor CLI Command Reference#pool-create|qs pool-create]]</code> with <code>--pool-use-case=ceph-zvol-osd</code>:

<pre style="font-size: smaller">
qs pool-create --name=osd-pool-1 --disk-list=<disk1>,<disk2>,... --raid-type=DRAID2_1S --pool-use-case=ceph-zvol-osd
</pre>

The DRAID layouts the CLI accepts are DRAID, DRAID1_1S, DRAID1_2S, DRAID1_3S, DRAID2_1S, DRAID2_2S, DRAID2_3S, DRAID3_1S, DRAID3_2S and DRAID3_3S.

== Step 2: Create the zvols ==

{{Navigation|Storage Management → Storage Volumes → Create ''(toolbar)''}}

Select the reserved pool as the '''Storage Pool''' on the '''General Settings''' tab, then set a '''Size''' for each zvol. Each zvol becomes one OSD, so the size you choose here is the OSD's capacity.

On the '''Advanced Settings''' tab:

* '''Use Case''' shows '''Ceph OSD''' and is locked, because the pool forces its use case onto every volume it holds.
* '''% Reserved''' defaults to '''100%'''. A zvol in a Ceph OSD pool is fully reserved by default because if the pool underneath runs out of space, every OSD on it starts failing writes at once. You can still lower it deliberately.
* '''Block Size''' sets the zvol's block size. When the OSD is created, QuantaStor sets Bluestore's minimum allocation size to match it, within a range of 4K to 64K.
* '''Batch Create''' creates several zvols of the same size in one step.

From the CLI, use <code>[[QuantaStor CLI Command Reference#volume-create|qs volume-create]]</code>. The CLI default for <code>--percent-reserved</code> is 0, so pass 100 explicitly to match the web interface:

<pre style="font-size: smaller">
qs volume-create --name=osd-vol --size=4T --pool=osd-pool-1 --count=6 --percent-reserved=100
</pre>

=== A single OSD zvol in a Default pool ===

You can also reserve an individual zvol in a Default pool: set '''Use Case''' to '''Ceph OSD''' on the '''Advanced Settings''' tab of Create Storage Volume, or pass <code>--volume-use-case=ceph-zvol-osd</code> to <code>qs volume-create</code>. Only that zvol is set aside for Ceph; the pool's other volumes stay available to hosts as usual. On a Ceph block pool there is nothing to choose, and the field stays at Default.

== Step 3: Create the OSDs ==

{{Navigation|Scale-out Storage Configuration → Data & Journal Devices → Create OSDs & Journals ''(toolbar)'' → Logical Devices ''(tab)''}}

The Create OSDs dialog has two tabs of available devices: '''Physical Devices''' for unused disks and '''Logical Devices''' for zvols. The Logical Devices tab lists every zvol that has the Ceph OSD use case, is healthy, is not a snapshot and does not already back an OSD. Move the zvols you want into '''Data/OSD Devices (HDDs or SSDs)''' and click '''OK'''. You can combine zvols and physical disks in one create. See [[Create Ceph OSD]] for the journal and optimization options in the rest of the dialog.

If a node has no unused physical disks but does have eligible zvols, the dialog still opens; it closes with ''No unused devices were found to create new OSDs or Journal Groups with'' only when neither tab has anything to offer.

From the CLI, use <code>[[QuantaStor CLI Command Reference#ceph-osd-multi-create|qs ceph-osd-multi-create]]</code> with <code>--volume-list</code>, which you can combine with <code>--physical-disk-list</code>:

<pre style="font-size: smaller">
qs ceph-osd-multi-create --ceph-cluster=<cluster> --volume-list=osd-vol-1,osd-vol-2,osd-vol-3
</pre>

=== How QuantaStor treats a zvol-backed OSD ===

* A zvol is new and has no partition table to clear, so QuantaStor skips the quick-format it runs on physical disks.
* For WAL/DB placement, a zvol counts as spinning media, because it is only as fast as the HDD pool behind it. Journal Group devices on SSD therefore apply to it as they do to an HDD.
* Once the OSD is up, QuantaStor renames its zvol to {{Code|1=vol-<system name>-osd.<n>}}, so the volume and the OSD it carries line up in the web interface and the CLI. If the rename fails, the OSD is unaffected and the zvol keeps its old name.

== Operating notes ==

=== Boot order ===

The OSDs cannot start until their pool is imported. At boot, QuantaStor imports every pool reserved for Ceph OSDs, and every Default pool that holds a Ceph OSD zvol, before the Ceph OSD services start. A pool that fails to import does not stop the rest of the boot; Ceph reports the OSDs on it as down.

Two cases are skipped:

* A pool with auto-start disabled is not imported, and the OSDs on it do not start.
* A pool managed by a Storage Pool HA failover group is left to the HA manager rather than imported at boot.

=== LVM visibility ===

Ceph builds each OSD as an LVM volume on its device, so LVM must be able to see the zvol. By default, the {{Code|1=global_filter}} in {{Code|1=/etc/lvm/lvm.conf}} hides every zvol from LVM so the appliance never scans a volume handed out to a client. QuantaStor adds accept rules for Ceph OSD zvols only: one rule for each pool reserved for Ceph OSDs, and one per zvol for a Ceph OSD zvol in a Default pool, so that pool's other zvols stay hidden. It updates the rules when a pool is created or started, including on another node after an HA failover, and when a Ceph OSD zvol is created or deleted. Other entries in the filter are left alone.

=== Missing OSDs ===

If the pool behind an OSD is not imported, the OSD shows as missing. QuantaStor keeps the OSD's link to its zvol, so it reconnects the OSD to the zvol once the pool is imported again.

== Related pages ==

* [[Storage Pools]]
* [[Storage Volumes]]
* [[Create Ceph OSD]]
* [[Delete Ceph OSD]]
* [[Scale-out Block Setup (ceph)]]
* [[Scale-out Object Setup (ceph)]]
* [[Designing Systems]]
* [[QuantaStor CLI Command Reference]]

----
<small>''Verified against QuantaStor 6.9.0.''</small>

Ceph over ZFS - Revision history

Qadmin: osn-seo-utilities: docs-ceph-over-zfs-zfs-draid-pool-reserved @ d389deda0b81 (approved in the portal)