Scale-out File Setup (ceph)
Scale-out file storage in QuantaStor is a CephFS file system presented as a Storage Pool, from which you provision Network Shares that clients reach over SMB, NFS or the native CephFS client. This page covers when to choose it over a scale-up ZFS pool, how the metadata servers and the metadata and data pools fit together, how to create the pool and shares on it, and which share features are unavailable on CephFS.
| Section | Purpose |
|---|---|
| When to choose scale-out file storage | How a CephFS pool differs from a scale-up ZFS pool, and which workloads suit each |
| How a CephFS file system is put together | Metadata servers, the metadata pool, the data pools, and where the file system is mounted |
| Deploying metadata servers | How many MDS instances you need and how to add them |
| Creating the file Storage Pool | The Ceph File System Create dialog, field by field, and what it provisions |
| Adding a second data pool | Mixing replicated and erasure-coded layouts in one file system |
| Modifying and deleting the pool | What can be changed after creation, and what cannot |
| Creating a Network Share on a scale-out pool | How the share dialog changes on a CephFS pool |
| What a scale-out share cannot do | The features that are ZFS-only, and what replaces them |
| Quotas and Snapshots | How both are implemented on CephFS |
| NFS, SMB and native CephFS access | Client connectivity for each protocol |
| Monitoring | Pool and MDS health, and MDS failover behaviour |
| CLI reference | The commands that manage file systems, data pools and metadata servers |
When to choose scale-out file storage
A scale-up pool is ZFS on the disks of a single storage system. It is the faster and more capable of the two for a single-system workload, and it carries the whole of QuantaStor's share feature set. Its ceiling is the system it lives on: capacity, throughput and availability are bounded by one appliance and its HA partner.
A scale-out pool is a CephFS file system spread across every system in a Ceph Cluster. Capacity and throughput grow by adding systems rather than by replacing them, a single namespace can exceed what one appliance can hold, and there is no failover pair to plan because the data is already distributed. What you give up is a set of share features that are implemented in ZFS and have no CephFS equivalent.
| Scale-up (ZFS) | Scale-out (CephFS) | |
|---|---|---|
| Storage | Disks of one storage system | OSDs across every system in the Ceph Cluster |
| A share is | A ZFS dataset with its own properties | A directory in the Ceph file system |
| Grows by | Adding disks or replacing the system | Adding systems to the cluster |
| Redundancy | RAID / mirroring within the pool, plus an HA failover pair | Replicas or erasure coding across systems; no failover pair to configure |
| Minimum systems | 1 (2 for HA) | 3, so the Ceph monitors can hold quorum |
| NFS server | The Linux kernel NFS server | NFS-Ganesha -- see NFS Configuration |
| Share features | All of them | See What a scale-out share cannot do |
| Native client | None; SMB and NFS only | CephFS kernel and FUSE clients, which talk to the OSDs directly |
Choose scale-out when the namespace has to outgrow one appliance, when clients can use the native CephFS client for throughput that SMB and NFS cannot reach, or when erasure coding across systems is the right cost model for a large archive. Choose scale-up when you need the ZFS share features -- replication, cloning, encryption, immutability, per-user quotas -- or when the data fits comfortably on one system.
A single storage system can host both a ZFS pool and a CephFS pool, with one exception that is easy to overlook: a node serves NFS from scale-up pools or from scale-out pools, but not both. CephFS does not work correctly re-exported through the kernel NFS server, so scale-out shares are served by NFS-Ganesha instead, and selecting one server masks the other. NFS Configuration owns that topic and explains the trade-off in full; plan each node as a scale-up NAS or a scale-out NAS and set its NFS Server Type to match.
How a CephFS file system is put together
A CephFS file system is three things: one metadata pool, one or more data pools, and a set of metadata server (MDS) daemons.
- The metadata pool holds the directory tree, file names, inodes and permissions. QuantaStor names it
<name>_metadataand always creates it replicated, whatever layout you choose for the data, because metadata is small, latency-sensitive and read constantly. - The data pool holds file contents. QuantaStor names the first one
<name>_dataand creates it with the layout you select -- replicated or erasure-coded. A file system can carry several data pools with different layouts; see Adding a second data pool. - The metadata servers serve metadata to clients out of the metadata pool. They hold no data of their own, so an MDS is a cache and a coordinator rather than a store, and losing one costs nothing but the time another takes to pick up its rank.
Placement groups are divided between the two pools in a 1:4 ratio, metadata to data, because the metadata pool is far smaller. On a cluster of 16 OSDs at the default Scaling Factor of 50%, that produced a 64-PG metadata pool and a 256-PG data pool.
QuantaStor creates a Storage Pool object with the same name as the file system, mounts the file system on every member of the Ceph Cluster at /mnt/storage-pools/qs-<pool-id>, and bind-mounts that path to /export/<pool-name> so the export paths match the scale-up convention. Each Network Share is a directory directly beneath the mount point. It also raises the CephFS maximum file size to 32 TiB; the Ceph default is far lower.
Deploying metadata servers
An MDS must exist before you can create the file system. The Ceph File System Create dialog checks for one and offers to open this dialog if it finds none, and the service refuses the operation outright if no MDS is in an up:active or up:standby state.
Select the Ceph Cluster, tick the members that should run a metadata server, and click OK. Members that already have one are filtered out of the grid, so the list only ever offers somewhere new to put one. QuantaStor names each daemon mds.<hostname> and sizes its metadata cache automatically from the free memory on the host -- 2 GiB is the floor, and the value it settled on shows in the Cache Limit column of the File System Metadata Server tab.
How many do you need? The dialog's own guidance is the right answer: at least one per cluster, and two or more for performance and high availability. One MDS is enough to serve the file system, but it is a single point of interruption -- while it is down no client can resolve a path. Each additional daemon becomes a standby that can take over a rank. Three metadata servers on a three- or four-system cluster gives one active and two standbys, which is a sound default.
More than one MDS can also be made active at once, which shares the directory tree between them and is worth doing only when a single MDS is genuinely the bottleneck. Neither the active nor the standby count may exceed the number of systems in the cluster, and their sum may not exceed it either.
From the CLI:
qs ceph-metadata-server-add --ceph-cluster=ceph-cluster-1 \
--ceph-cluster-member-list=qs-node-1,qs-node-2,qs-node-3
qs ceph-metadata-server-list
The commands are qs ceph-metadata-server-add, qs ceph-metadata-server-list and qs ceph-metadata-server-remove.
Creating the file Storage Pool
The File Storage toolbar group is the last of four in this section, so on a narrow window it is hidden behind the overflow chevron at the right-hand end of the toolbar rather than shown inline. It holds Create File Storage Pool, Modify File Storage Pool, Delete File Storage Pool, Add File Data Pool, Add MDS and Remove MDS.
A Ceph Cluster supports one scale-out file Storage Pool, and a second one is not a supported configuration. Different layouts within a cluster are handled by adding data pools to that one file system -- see Adding a second data pool -- not by creating a second file system. A second file system would also compete for the same metadata servers: each one claims an MDS rank of its own, so on a cluster with three metadata servers a second file system leaves only one standby, and a third leaves none.
The dialog will not let you get ahead of the prerequisites. With no OSDs in the cluster it offers to open the Create OSDs dialog and closes; with fewer than two OSDs it stops with an error; with no metadata server it offers to open the Add MDS dialog and closes.
General tab
| Field | Notes |
|---|---|
| Ceph Cluster | Which cluster to create the file system in. |
| Name | Names the file system, the Storage Pool object and both Ceph pools. Defaults to file-pool-<n>. Alphanumerics and -_. only.
|
| Description | Free text. The only field on this tab you can change afterwards. |
| Type | The data pool layout: Erasure, Replica-3 (Triple-Mirror) (the default), Replica-4 or Replica-5. Replicated layouts are the faster choice and suit active workloads; erasure coding is more space-efficient and suits backup and archive. |
| Set Manually → Data Chunks (K) / Code Chunks (M) | Enabled only when Type is Erasure. QuantaStor creates an erasure-coding profile for the K and M you pick whose CRUSH failure domain is the host, so each of the K+M chunks lands on a different storage system. That is what sets the floor on cluster size: a K8+M2 layout needs at least 10 systems, and 11 is better so the cluster can still recover a chunk after losing one. |
| Use Profile | Enabled only when Type is Erasure. Selects an existing erasure-coding profile instead of building one from K and M -- use this when the profile needs a failure domain other than the host, such as a rack or a site. Profiles are created from the EC Profiles & Rules toolbar group. |
| Scaling Factor (%) | How much of the cluster's placement group budget this file system should claim, defaulting to 50%. Set it to 100 if the cluster will only ever serve scale-out file storage; if it will also carry block or object pools, set each pool to roughly the share of cluster capacity it is expected to use. The PG count can be tuned later. |
| Force (Required for Erasure M=1) | Overrides the validation that rejects a layout with too little redundancy. An M=1 erasure-coded pool cannot recover from a failure during recovery, so it is refused unless you insist. |
Advanced Settings tab
The two MDS fields on this tab are health-check minimums, not a deployment request. They do not start or stop any daemon. They tell Ceph how many active and standby metadata servers the file system expects, and Ceph raises a health warning when the expectation is not met.
| Field | Default | Notes |
|---|---|---|
| Disable MDS Instance Minimums | ticked | Ticked, no minimum is enforced and any number of MDS instances is acceptable. Clear it to set an explicit expectation. |
| Minimum number of active MDS service instances | 0 | The number of MDS ranks the file system runs. Left at zero, Ceph's own default of one active MDS applies. |
| Minimum number of standby MDS service instances | 0 | How many standbys must be available for the cluster to be considered healthy. Zero disables the check. |
| Auto Configure NFS Settings | ticked | Switches each cluster member to the scale-out NFS server so the new shares are exportable. See below. |
Worth knowing: clearing Disable MDS Instance Minimums enables the two fields but leaves both at zero, and zero means "no minimum". Type the numbers you want -- do not assume clearing the checkbox filled them in.
The effect is easy to see. On a three-MDS cluster running one active and two standbys, setting the standby minimum to 2 and then stopping one MDS produced exactly this:
[WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available
have 1; want 1 more
With the minimum left at zero the same outage raises nothing at all. Set the minimums if you want to be told when the cluster has run out of spare metadata servers; leave them alone if you do not.
Auto Configure NFS Settings is what makes the new shares reachable over NFS. On each cluster member it checks for ZFS Storage Pools, and:
- if the member has none, it switches that node to the scale-out NFS server -- NFS-Ganesha takes port 2049 with NFSv3 and NFSv4 enabled, and the kernel NFS server is stopped, disabled and masked;
- if the member has a ZFS pool, it changes nothing and raises the alert "SCALE-UP pool(s) present on local system. Not configuring SCALE-OUT NFS settings."
That second case is not a failure -- it is the constraint in NFS Configuration enforced from the Ceph side, protecting the ZFS shares that node is already exporting. If you leave the box ticked on a mixed node, expect the alert and configure that node's NFS Server Type deliberately.
What creating the pool actually does
In order: it creates the erasure-coding profile if one is needed, creates the _metadata and _data Ceph pools and tags both with the cephfs application, runs ceph fs new to bind them into a file system, sets the active MDS count and the standby expectation, raises the maximum file size to 32 TiB, optionally reconfigures NFS, creates the QuantaStor Storage Pool object, and mounts the file system on every cluster member.
From the CLI the equivalent is qs ceph-filesystem-create:
qs ceph-filesystem-create --name=file-pool-1 --ceph-cluster=ceph-cluster-1 \
--pool-type=replicated --max-replicas=3 --scaling-factor=100 --auto-config-nfs=true
The CLI exposes two things the dialog does not. --min-replicas sets the number of replicas that must be writable for the pool to accept writes -- the dialog always uses 2. --auto-config-nfs must be given explicitly, because unlike the dialog checkbox it is not on by default.
Adding a second data pool
One file system can carry several data pools with different layouts, and each Network Share can be pointed at whichever one suits it. A replicated pool for active project directories and an erasure-coded pool for an archive directory, in one namespace, is the usual reason to do this.
The layout controls are the same as on the create dialog. The one addition is Data Pool Suffix: the new pool is named <filesystem>_data_<suffix>, and the suffix defaults to the layout you chose, so a replicated addition becomes <filesystem>_data_replica. Tick Custom Suffix to name it yourself. The Data Pool Name line previews the result before you commit.
qs ceph-filesystem-pool-add --ceph-filesystem=file-pool-1 --pool-type=erasure \
--data-pool-suffix=archive --k=8 --m=2 --scaling-factor=25
To move a share onto a different data pool, use qs share-data-pool-layout-modify, or the Sub Pool field on the share dialogs:
qs share-data-pool-layout-modify --share=share1 --new-sub-pool=file-pool-1_data_archive
This sets the directory's data layout, so new files written to the share land in the new pool. Files already there stay where they are: --recursive-contents is rejected with "Recursive mode not currently supported". To move existing content, see Optimizing directory placement.
Modifying and deleting the pool
Only the description and the two MDS minimums can be changed after creation. The name, the data pool layout, the K and M values and the Scaling Factor are all fixed at create time, so choose the layout for the workload the pool will actually carry. If the layout turns out wrong, add a data pool with the right one and re-point the shares rather than rebuilding.
Deleting the file system destroys the metadata pool, every data pool and all the data in them. QuantaStor refuses the delete while Network Shares exist on the file system unless you pass the force flag:
qs ceph-filesystem-delete --ceph-filesystem=file-pool-1 qs ceph-filesystem-list qs ceph-filesystem-get --ceph-filesystem=file-pool-1
Shares on a scale-out pool are created from the same dialog as shares on a scale-up pool, documented in full in Network Shares. Selecting a CephFS pool changes what the dialog offers.
What appears only for a CephFS pool:
- Sub Pool -- which data pool inside the file system backs this share. With one data pool there is nothing to choose; with several this is where you choose.
- Enable File Quota → Number of files -- a cap on the number of files rather than the space they use. This is a CephFS capability and the service rejects it on any other pool type.
- Pool Type in the read-only Storage Pool Settings summary reads Scale-out File.
What changes:
- Enable Usage Tracking is off by default, where a scale-up share has it on. Per-user and per-group collection on CephFS walks the directory tree rather than reading counters out of the filesystem, so it is opt-in and sampled far less often. There is a global switch that has to be on as well -- see Usage tracking.
- Excludes Snapshot Usage is greyed out and cleared. CephFS quotas do not make the distinction.
- Nested Share Options is greyed out. Nested shares, aliases and sub-shares are ZFS-only.
Once created, the share is a directory under the pool mount point, exported over SMB from that path and over NFS by NFS-Ganesha.
This is the part of scale-out worth reading before you commit to it. The features below are implemented in ZFS and have no CephFS equivalent, so the dialog disables them and the service refuses them. Network Shares carries the full per-field matrix; the list here is what it means for a scale-out deployment.
| Unavailable | What to use instead |
|---|---|
| Rollback to a snapshot, and Restore from one | Copy the files you need back out of the snapshot directory. The service says so itself: "You must use the object versioning or manually restore files using snapshot content in /.snap sub-directories." |
| Clone of a share or a snapshot | Nothing equivalent. Copy the data if you need a writable second copy. |
| Replication to another system, and Promote | Not available for CephFS shares. The redundancy story for scale-out is the cluster itself, spread across systems and, with a suitable CRUSH failure domain, across racks or sites. |
| Share encryption -- load, unload, passphrase, key import and export | Not available. Encrypt at rest below the OSDs, or in the application. |
| Per-user and per-group quotas | Only the whole-share space quota and file quota, both below. |
| Immutability / WORM and Anomaly (ransomware) Detection | Both fieldsets are disabled on the Security Settings tab. Use snapshots plus a Snapshot Schedule with retention as the recovery path. |
| NFS Security Mode (Kerberos) | Forced to inherit and disabled. Every Ganesha export is generated with sectype = sys, so Kerberos is unavailable on scale-out shares.
|
| Aliases, sub-shares and nested shares | Not available. Create separate shares. |
| Global Namespaces | A CephFS share cannot be added to a namespace. |
| Compression, Sync Policy, Cache Policy, Record Size, Small Block Offload, Read Only, Case Sensitivity | All ZFS dataset properties. The whole Advanced Settings tab greys out. Compression on scale-out is a property of the underlying Ceph pool, not the share. |
| Snapshot of a snapshot | Snapshot the parent share again. |
| Backup Policies using CDP or Auto-copy | Not supported with a CephFS source. |
Two things point the other way, and are scale-out only: the file quota, and the Sub Pool choice of data layout per share.
The share Dashboard is also reduced for a CephFS share -- the User Usage and Group Usage views are available when usage tracking is on, and the Capacity, Trends, NFS, SMB and Replication views are not.
Long-term retention rules on a Snapshot Schedule do apply to CephFS shares as well as ZFS ones.
Quotas
Two quotas apply to a scale-out share, and both are set on the General tab of the share dialogs or with qs share-modify:
qs share-modify --share=share1 --space-quota=10GB --file-quota=100000
QuantaStor applies them as CephFS extended attributes on the share's directory -- ceph.quota.max_bytes and ceph.quota.max_files -- so Ceph enforces them wherever the share is accessed from, including the native client. Read them back with getfattr on the share directory, or from qs share-get.
Three differences from a scale-up share are worth planning around:
- There is no "quota excludes snapshots" option. The checkbox is greyed out. Space held by snapshots counts toward the quota.
- The file quota is CephFS-only. It is the one quota a scale-out share has that a scale-up share does not.
- Neither NFS server reports quotas to clients. The Ganesha configuration QuantaStor ships sets
enable_rquota = false, noted in the file as not supported on CephFS, soquotaon a client tells the user nothing. Read the quota from the WUI or the CLI.
Per-user and per-group quotas are not available; the service rejects them with "User/group quotas are only supported on Network Shares provisioned from ZFS based Storage Pools."
Snapshots
A CephFS share snapshot is a directory under the share's hidden .snap directory. Snapshotting share1 as snap1 creates share1/.snap/snap1, a complete point-in-time view of the share taken instantly and costing only the blocks that change afterwards.
qs share-snapshot --share=share1 --name=snap1
CephFS snapshots are read-only, at the filesystem level. Writing into one fails with Read-only file system regardless of permissions, which is why rollback, restore and clone are unavailable and why the recovery procedure is to copy files out of the snapshot. This is a real difference from a scale-up share, where a snapshot can be made writable.
The snapshot directory is reachable from a CephFS client and from an NFS client at <share>/.snap/<snapshot>. Over SMB the snapshots are presented as Windows Previous Versions -- QuantaStor generates the share with vfs objects = shadow_copy2 pointed at a .qsnaps link into .snap, so a user can restore an earlier copy of a file from the Explorer property sheet without an administrator.
Snapshot Schedules work on CephFS shares, and taking scheduled snapshots is the main recovery mechanism a scale-out share has -- there is no replication and no immutability to fall back on. Long-term retention rules apply to CephFS shares as well as ZFS ones.
NFS access
Scale-out shares are exported by NFS-Ganesha, not by the kernel NFS server, because CephFS does not work correctly re-exported through nfsd. NFS Configuration owns the subject -- how to select the server, what differs between the two, and the per-client options Ganesha does not implement. The scale-out specifics are:
- Ganesha reads the file system through its CEPH FSAL, so it talks to the OSDs directly rather than through the node's own mount.
- The path a client mounts is
/<share>, not/export/<share>as it is for a scale-up share. - With the node in scale-out mode Ganesha owns port 2049 and serves NFSv3 and NFSv4. In scale-up mode it is moved to port 2249 in NFSv4-only mode so the kernel server can keep 2049.
- NFSv3 byte-range locking (NLM) and RQUOTA are both disabled in the shipped configuration.
- Kerberos is not available -- every generated export is
sectype = sys.
A worked mount, from a node in scale-out mode:
mkdir -p /mnt/share1 mount -t nfs -o vers=4 qs-node-1:/share1 /mnt/share1
QuantaStor regenerates the EXPORT sections of /etc/ganesha/ganesha.conf on every change, so put hand additions in /etc/ganesha/ganesha.conf.custom instead.
SMB access
SMB access to a scale-out share goes through the same Samba path as a scale-up share -- there is no CephFS-specific SMB stack. QuantaStor writes an ordinary stanza into /etc/samba/smb.conf pointing at the share's directory under the pool mount point, and Samba serves it from the node's own CephFS mount. Everything on the CIFS/SMB Settings and File Masks & Permission Settings tabs applies, as does Active Directory integration; Network Shares covers all of it.
Two scale-out details:
- Previous Versions works, by way of
shadow_copy2over the.snapdirectory as described above. - Because Samba reads the file system through the node's mount, an SMB client's throughput is bounded by that one node. A client that needs the full cluster bandwidth should use the native CephFS client instead.
Native CephFS client access
SMB and NFS are point-to-point: every byte passes through the storage system the client connected to. The native CephFS clients talk to the metadata servers and the OSDs directly, so a client reads and writes across the whole cluster at once. For throughput-bound Linux clients this is the reason to run scale-out file storage at all.
Getting the client configuration and a keyring
A client needs two files in /etc/ceph: a cluster configuration naming the monitors, and a keyring. QuantaStor generates both.
qs ceph-client-conf-export --ceph-cluster=ceph-cluster-1 --output-file=/tmp/ceph.conf qs ceph-client-keyring-create --name=share1-client --client-key-type=cephfs --resource-list=share1 qs ceph-client-keyring-export --ceph-client=share1-client
qs ceph-client-conf-export writes a minimal client configuration -- cephx required, and the monitor addresses -- to /etc/ceph/qs-ceph.conf unless you name another file. qs ceph-client-keyring-create creates a Ceph client scoped to the shares you list, and qs ceph-client-keyring-export writes it to /etc/ceph/ceph.client.<name>.keyring. Copy both files to the client.
For a client scoped to one share the generated capabilities look like this:
[client.share1-client] key = aAbBcCdDeEfF0123456789aAbBcCdDeEfF012345 caps mds = "allow r, allow rw path=/share1" caps mon = "allow r" caps osd = "allow rw pool=file-pool-1_data"
Read that capability set carefully: it is a write restriction, not a confinement. allow r on the metadata server lets the client mount the file system root and list every share in it; only writes are limited to its own path. If a client must not see the names of other shares, do not give it a keyring for this file system.
Use the client.admin keyring only for administration. It has full access to every share in the file system, and on a client system whose users have root it is equivalent to handing them the whole namespace.
Kernel client mount
The kernel client is the faster of the two and is the one to use unless something rules it out. Two syntaxes work:
mkdir -p /mnt/share1 mount -t ceph share1-client@<fsid>.file-pool-1=/share1 /mnt/share1
The newer form above names the client, the cluster FSID, the file system and the path within it; it reads the monitor addresses from /etc/ceph/ceph.conf. Get the FSID with ceph fsid on any cluster member. The older monitor-address form is still accepted:
mount -t ceph qs-node-1:6789,qs-node-2:6789,qs-node-3:6789:/share1 /mnt/share1 -o name=share1-client
Mount :/ instead of :/share1 to mount the whole file system, with every share visible as a directory beneath it.
FUSE client mount
The FUSE client runs in user space, so it needs no matching kernel module and is the fallback when the client's kernel is older than the cluster:
mkdir -p /mnt/share1 ceph-fuse -n client.share1-client -r /share1 /mnt/share1
It also reads /etc/ceph/ceph.conf for the monitors. Unmount it with fusermount -u.
Custom mount options on the storage systems
QuantaStor mounts the file system on the cluster members themselves with the kernel client, using a command template in /opt/osnexus/quantastor/conf/qs_cephfs.conf:
[global]
#kernel mount mode
mountCommand=mount -t ceph {OPTIONS} :/ {POOLMOUNT} -o name=admin
{POOLMOUNT} is substituted with the pool's mount path. To change how the storage systems mount the file system -- to add a mount option, or to switch the node to the FUSE client by commenting the line out -- edit mountCommand itself; there is no separate options key to set. As with every file in the registry, place an edited copy in /var/opt/osnexus/quantastor/conf/ so a package upgrade does not overwrite it. See QuantaStor Configuration Files.
This file controls only how the storage systems mount the file system. It has no bearing on how a client mounts it.
Monitoring
The Ceph File System tab under Scale-out Storage Pools is the pool-level view. Size and Free Space are the total across every data pool in the file system, so adding a data pool increases both; Meta Data Size and Meta Utilized Size report the metadata pool separately -- watch that one, because a metadata pool that fills stops the file system even when the data pools have room. Active MDS Count and Standby MDS Count show the file system's expectations, matching the two minimums on the Modify dialog.
The Ceph Cluster dashboard carries a Metadata Servers tile alongside the OSD, monitor and pool tiles, and the File System Metadata Server tab lists each daemon with its Status -- Ceph's own state string, up:active or up:standby -- the system it runs on, and its metadata cache limit.
MDS failover
Metadata server failover is handled by Ceph and needs no configuration. When the daemon holding a rank stops, a standby claims the rank and replays the metadata journal; clients block for the duration and then continue. On a healthy cluster this takes seconds. Stopping the active MDS on a three-MDS cluster moved rank 0 to a standby and left one standby remaining, with no client error.
Two behaviours to expect:
- Failover is not failback. When the original daemon comes back it rejoins as a standby. The rank stays where it moved to. There is nothing to correct.
- An outage with the minimums at zero is silent. Losing a standby raises no warning unless you have set a standby minimum, which is the reason to set one -- see Advanced Settings tab.
Cluster health
QuantaStor surfaces Ceph's own health checks as Ceph Cluster health events, under their Ceph names, so the MDS ones read as they do in ceph health detail:
| Check | Meaning |
|---|---|
MDS_INSUFFICIENT_STANDBY |
Fewer standby metadata servers are available than the standby minimum you set. |
MDS_UP_LESS_THAN_MAX |
Fewer MDS ranks are active than the active count asks for -- usually not enough daemons deployed. |
RECENT_CRASH |
A daemon crashed recently and the crash report has not been archived. |
Route these to a destination through the Alert Manager rather than watching the dashboard.
Optimizing directory placement
qs-cephfsutil analyses a directory's file-size distribution and moves files between data pools -- which is how you migrate existing content after re-pointing a share's Sub Pool, since the re-point itself only affects new files. It filters by size and age and has a dry-run mode. QuantaStor Shell Utilities documents it.
Managing scale-out file storage from the CLI
| Command | Purpose |
|---|---|
qs ceph-filesystem-create |
Create the file system, its pools and the Storage Pool object |
qs ceph-filesystem-list / qs ceph-filesystem-get |
List file systems, or report one in full including its pool associations and MDS counts |
qs ceph-filesystem-modify |
Change the description and the MDS minimums |
qs ceph-filesystem-pool-add |
Add a data pool with a different layout |
qs ceph-filesystem-delete |
Destroy the file system and every pool in it |
qs ceph-metadata-server-add / qs ceph-metadata-server-remove |
Deploy or remove MDS daemons |
qs ceph-metadata-server-list |
List MDS daemons and their state |
qs ceph-pool-list |
Show the metadata and data pools with their PG counts, layout and replica settings |
qs share-create / qs share-modify |
Create and modify shares, including both quotas |
qs share-data-pool-layout-modify |
Point a share at a different data pool |
qs share-snapshot |
Snapshot a share |
qs ceph-client-keyring-create / qs ceph-client-keyring-export |
Create and export a scoped keyring for a native CephFS client |
qs ceph-client-conf-export |
Export a client cluster configuration file |
Troubleshooting
The Create File Storage Pool dialog closes and offers another dialog instead. It found no OSDs, or no metadata server. Take the offer -- the prerequisite has to be met first either way.
Creating the file system fails with "No Metadata server(s) found configured". No MDS exists on the cluster, or none is in an up:active or up:standby state. Check the File System Metadata Server tab and add one.
A client mount fails with "no mds (Metadata Server) is up ... or you may not be authorized". That message covers authentication failures as well as MDS outages, and authentication is the more likely cause. Check that the keyring for the client name you passed exists in /etc/ceph on the client, and that the MDS tab shows a daemon up:active before looking any further.
NFS clients cannot reach a scale-out share. Check which NFS server owns port 2049 on that node. If the node is in scale-up mode, Ganesha is on 2249 in NFSv4-only mode and an NFSv3 client will fail outright. NFS Configuration covers the switch.
Enabling scale-out NFS raised "SCALE-UP pool(s) present on local system". That node has a ZFS pool, so QuantaStor left its NFS configuration alone rather than take the ZFS shares offline. Decide which architecture that node serves and set its NFS Server Type deliberately.
Writing into a snapshot fails with "Read-only file system". Expected -- CephFS snapshots are read-only. Copy the files you need out of .snap.
A share's new files are still landing in the old data pool. Re-pointing the Sub Pool changes the layout for new files only, and only for files created after the change. Existing files stay put; move them with qs-cephfsutil.
The metadata pool is filling up. It is sized from the Scaling Factor at create time and grows with the number of files rather than their size, so a file system with very many small files can exhaust it while the data pools look empty. Watch Meta Utilized Size on the Ceph File System tab.
Ceph Terminology & Concepts
This section will introduce Ceph terms and concepts to familiarize oneself with to become more proficient with Ceph Cluster administration in QuantaStor. This discussion will address general concepts surrounding Ceph. To implement Ceph in QuantaStor it is recommended to see Getting Started to quickly implement Ceph. Getting Started can be found at Storage Management --> Storage System --> Storage System --> Getting Started (toolbar).
Getting Started in Administration guide.
Further Information: Introduction to Ceph.
Ceph Cluster
A Ceph Cluster is a group of three or more systems that have been clustered together using the Ceph storage technology. Ceph requires a minimum of three nodes to create a cluster so that quorum may be established across the Ceph Monitors. Wikipedia Quorum (distributed computing).
In QuantaStor based Ceph configurations, QuantaStor systems must first be combined into a Storage Grid. After the Storage Grid is formed one more more Ceph Clusters may be created within the Storage Grid. In the example above the Storage Grid is comprised of a single Ceph Cluster. When the Ceph Cluster is initially created QuantaStor automatically deploys 3x Ceph Monitors within the new Ceph Cluster.
Ceph Monitor
The Ceph Monitors form a Paxos The Part-Time Parliament cluster for the management of cluster membership, configuration information, and state. Paxos is an algorithm (developed by Leslie Lamport in the late 80s) which uses a three-phase consensus protocol to ensure that cluster updates can be done in a fault-tolerant timely fashion even in the event of a node outage or node that is acting improperly. Ceph uses the algorithm so that the membership, configuration and state information is updated safely across the cluster in an efficient manner. Since the algorithm requires a quorum of nodes to agree on any given change an odd number of systems (three or more) are required for any given Ceph cluster deployment.
During initial Ceph cluster creation, QuantaStor will configure the first three systems to have active Ceph Monitor services. Configurations with more than 16 nodes should add two additional monitors. This can be done through the QuantaStor web user interface in the Scale-out Storage Configuration section.
In a Ceph cluster with 3x monitors a minimum of 2x monitors must be online at all times. If only one monitor (or none) are running then storage access will is automatically disabled until quorum among monitors may be reestablished. In larger clusters with 5x monitors then 3x monitors must be online at all times to maintain quorum and storage accessibility.
Ceph Object Storage Daemon / OSD
Navigation: Scale-out Storage Configuration --> Data & Journal Devices --> Data & Journal Devices --> Create OSDs & Journals (toolbar)
The Ceph Object Storage Daemon, known as the OSD, is a daemon process that reads and writes data and generally maps 1-to-1 to a HDD or a SSD device. OSD devices may be used by multiple Storage Pools so after the OSDs are added one may allocate pools for file, block, and object storage which all use the available OSDs in the cluster to store their data.
QuantaStor Scale-out SAN with Ceph deployments must have at least 3x OSDs per system, making 9x OSDs total the minimum number OSDs. QuantaStor 5 and newer versions use the BlueStore OSD storage back-end. For additional BlueStore information see, New in Luminous: BlueStore.
N.B., for ease of use there is an Auto Config button that will optimize selection of available devices.
Journal Groups
Journal Groups are used to boost the performance of OSDs. Each Journal Group can provide a performance boost for 5x to 30x OSD devices depending on the speed of the storage media used to create a given Journal Group. Journal Groups are typically created using a pair of SSDs which QuantaStor combines into software RAID1 mirror. Once created Journal Groups provide high performance, low latency, storage from which Ceph Journal Devices may be provisioned and attached to new OSDs to boost performance. Because Journal Groups must sustain high write loads over a period of years only datacenter (DC) grade / enterprise grade flash media should be used to create them. Journal Groups can be created using all types of flash storage media including NVMe, PMEM, SATA SSD, or SAS SSD.
Journal Devices are provisioned from Journal Groups. Journal Devices come in two types, Write-Ahead-Log (WAL) devices and Meta-data DB (MDB) devices. QuantaStor automatically provisions WAL devices to be 2GB in size and MDB devices can be 3GB, 30GB (default), or 300GB in size.
Journal Groups are not required but are highly recommended when creating HDD based OSDs. With SSD based OSDs it is not recommended to assign them external WAL and MDB devices from Journal Groups. Rather the MDB and WAL storage for SSDs will be allocated out of a small portion of the underlying OSD data device. With platter/HDD based OSDs we highly recommend the creation of Journal Groups so that each OSD can have both an external WAL device and a external MDB device.
NVMe and 3D XPoint flash storage media are the best storage types for creating Journal Groups due to their high throughput and IOPS performance. We recommend allocating 100MB/sec and 32GB of capacity for each HDD based OSD. For example, a system with 60x HDD based OSDs would require 60x100MB/sec or 6000MB/sec of Journal Group throughput. If NVMe devices are selected that can do 2000MB/sec then three Journal Groups will be required and a total of 6x NVMe SSDs (3x RAID1 Journal Groups). Capacity wise 60x HDDs will require 60x32GB of storage for all the WAL and MDB devices to be created or what amounts to 1.92TB of provisionable Journal Group capacity. One possible design to meet both the performance and the capacity requirements would be to make the 3x Journal Groups using a total of 6x 800GB NVMe devices with a 3x DWPD endurance.
Write-Ahead Log (WAL) Journal Devices
WAL Journal Devices are provisioned from Journal Groups and are then attached to new OSDs when they are created. WAL devices accelerate write performance. When a write request is received by an OSD it is able to write the data to low-latency stable flash media very quickly to complete the write. Data can then be written lazily to the HDD as time allows without risk of losing data due to a sudden system power outage.
Meta-data Database (MDB) Journal Devices
MDB Journal Devices effectively boost both read and write performance as they contain all the Bluestore filesystem metadata. Rather than having to write small blocks of metadata to HDDs which have low IOPS performance and external MDB device on flash media can sustain high IOPS loads and in turn greatly boosts performance.
Hardware RAID
Although Hardware RAID may be used in Ceph Clusters as an underlying storage abstraction for OSDs it is generally not recommended. It does have applications in very large Ceph clusters (ie. 1000s of OSDs) and with clusters comprised of servers with limited RAM and CPU core count. Roughly speaking each OSD requires approximately 2GB of RAM and a 1GHz fractional CPU core. A server with 60x HDD based OSDs will require a large dual-processor configuration and 192GB of RAM. By combining disks using HW RAID these requirements are reduced 5:1. QuantaStor does have integrated hardware RAID management and monitoring to manage configurations that use hardware RAID. But again, we do not recommend the use of HW RAID except in specialized configurations and in hyper-scale configurations.
Ceph Placement Groups (PGs)
Ceph Pools do not write data directly to OSDs, rather there is an abstraction layer between each Ceph Pool and the OSDs comprised of Placement Groups, PGs. Each PG can be thought of as a logical stripe across a group of OSDs. Ceph Pools created with a replica=2 storage layout will have PGs that each reference 2x OSDs. Similarly a Ceph Pool with an erasure-coding layout of K8+2M would have PGs that each span 10x OSDs. When creating new File, Block, or Object Storage Pools with QuantaStor you have control over the number of PGs to be created using the Scaling Factor option. If a given Ceph Cluster is to be used for a single type of storage such as File or Object then one would set the Scaling Factor to 100%. If it is expected that a given Ceph Cluster will be used for 30% Object storage and 70% File storage then those Storage Pools should be allocated with those Scaling Factors respectively. Your choice for the Scaling Factor for any given pool should be a best guess. The PG count can be adjusted later to provide better optimization of storage distribution and balancing across the OSDs in the future if required.
Ceph CRUSH Maps and Resource Domains
Ceph supports the ability to organize placement groups, which provide data mirroring across OSDs, so that high-availability and fault-tolerance can be maintained even in the event of a rack or site outage. By defining failure-domains, such as a Rack of systems, a Site, or Building, a map can be created so that Placement Groups are intelligently laid out to ensure high-availability despite the outage of one or more failure-domains, depending on the level of redundancy.
This intelligent map is called the Ceph CRUSH map (Controlled Replication Under Scalable Hashing), standing for Controlled, Scalable, Decentralized Placement of Replicated Data, and it defines how to mirror data in the Ceph cluster to ensure optimal performance and availability.
Creating CRUSH maps manually can be a complex process, so QuantaStor creates and configures CRUSH maps automatically, saving a large degree of administrative overhead. To facilitate automatic CRUSH map management, detail regarding where each QuantaStor system is deployed must be provided. This is done by creating a tree of Resource Domains via the WebUI (or via CLI/REST APIs) to organize the systems in a given QuantaStor Grid into Racks, Sites, and Buildings. QuantaStor uses this information to automatically generate an optimal CRUSH map when pools are provisioned, ensuring optimal performance and high-availability.
Custom CRUSH map changes can still be made to adjust the map after the pool(s) are created and OSNEXUS provides consulting services to meet special requirements. Resource Domains are a QuantaStor construct so you will not find mention of them in general Ceph documentation, but they map closely to the CRUSH bucket hierarchy.
For additional information see, CRUSH MAPS
Related pages
- Network Shares -- creating, modifying and exporting shares, and the full per-field feature matrix
- NFS Configuration -- the two NFS server implementations, and why a node serves one architecture or the other
- Snapshot Schedules -- scheduled snapshots and retention, which apply to CephFS shares
- Storage Pools -- scale-up (ZFS) Storage Pools
- Scale-out Block Setup (ceph) -- scale-out block storage on the same Ceph Cluster
- Scale-out Object Setup (ceph) -- scale-out object storage on the same Ceph Cluster
- QuantaStor Configuration Files -- the configuration file registry, including
qs_cephfs.conf - QuantaStor Shell Utilities --
qs-cephfsutiland the other appliance utilities - Alert Manager -- routing cluster health events to a destination
- QuantaStor CLI Command Reference -- full argument lists for every command named here
Verified against QuantaStor 6.9.0.