Scale-out File Replication
Scale-out file replication copies Network Shares from a CephFS file system in one Ceph Cluster to a CephFS file system in a second Ceph Cluster, using Ceph's own snapshot mirroring. This page covers how it differs from scale-up (ZFS) replication, the mirroring service and mirror peer it depends on, the replication schedule that drives it, how to monitor it, and what it does not do -- in particular, there is no built-in failover or failback.
For replicating Storage Volumes and shares on scale-up (ZFS) pools, see Remote-replication (DR). The two mechanisms share the Create Replication Schedule dialog and nothing else.
| Section | Purpose |
|---|---|
| How it differs from scale-up replication | The mechanism, and what does and does not carry over from ZFS replication |
| Requirements | What must be in place before you start |
| Setup at a glance | The three objects, in the order you create them |
| Mirroring services | The cephfs-mirror daemon that moves the data |
| Mirror peers | The relationship between a local and a remote file system |
| Scale-out replication schedules | What the schedule does, and how the dialog changes for scale-out |
| Monitoring | States, alerts, and what is not reported |
| Recovery and failback | What to expect when you need the copy |
| Removing replication | Deleting schedules, peers and services, and in which order |
| CLI reference | The commands for services, peers and schedules |
How it differs from scale-up replication
A scale-out replication schedule has no data mover of its own. Each time the schedule runs, QuantaStor takes a snapshot of every share on it. The cephfs-mirror daemon on the source cluster notices each new snapshot of a mirrored directory and copies it to the remote file system on its own. QuantaStor registers each share's directory (/<share name>) with Ceph for mirroring while the share is on a scale-out schedule, and unregisters it when it is not.
| Scale-up (ZFS) replication | Scale-out (CephFS) replication | |
|---|---|---|
| Replicates | Storage Volumes and Network Shares on ZFS pools | Network Shares on a CephFS file system only |
| Moved by | QuantaStor replication tasks, ZFS send/receive over SSH | The Ceph cephfs-mirror daemon on the source cluster |
| Trust relationship | A Storage System Replication Link between two systems | A mirroring service on the source cluster, plus a mirror peer from one file system to another |
| Destination | A Storage Pool on the remote system | A CephFS file system in another Ceph Cluster |
| Direction | Set by the link you choose | One way, from the peer's local file system to its remote file system |
| On the destination | A replica checkpoint per volume or share, imported into QuantaStor | The mirrored directory and its snapshots; no checkpoint and no Network Share are created |
| Destination retention | Its own retention counts | Follows the source; the destination fields are greyed out |
| Failover | Checkpoint activation, manual or automatic | None |
| Replication reports | Per-run summary and entry reports | None; the Replica Reports tab is disabled |
Requirements
- Two Ceph Clusters, each with at least one CephFS file system, managed from the same QuantaStor grid. The peer dialog lists only clusters and file systems the grid knows about, and the peer is bootstrapped by coordinating the two clusters' coordinator nodes. The two clusters must be different clusters; a peer from a cluster to itself is refused. See Scale-out File Setup (ceph) for creating the file systems.
- Network reachability from the source cluster to the remote cluster. The mirror daemon on the source connects to the remote cluster's monitors and OSDs to write the copy.
- A mirroring service on the source cluster. The remote cluster does not need one for replication in this direction.
- The Network Shares to replicate all on one CephFS file system. One schedule mirrors one file system, over one peer.
Setup at a glance
- Create a Mirroring Service on the source Ceph Cluster. This starts the cephfs-mirror daemon.
- Create a Mirror Peer from the source file system to the file system on the remote cluster.
- Create a Replication Schedule, choose Scale out replication, select the peer as the Replica Destination, and select the shares.
Each step is described below. The Create Mirror Peer dialog offers to launch step 1 for you if the cluster you select has no mirroring service.
Mirroring services
A mirroring service is the cephfs-mirror daemon running on one storage system in a Ceph Cluster. It is what copies the snapshots, so replication stops whenever no mirroring service on the source cluster is running.
- Ceph Cluster -- the cluster to run the daemon in. Use the source cluster, the one whose file system you are replicating from.
- Run On Storage System -- the cluster member that hosts the daemon. Each storage system can run one mirroring service per cluster.
On OK, QuantaStor enables the Ceph mirroring manager module, creates the client.mirror Ceph user and its keyring, and enables and starts cephfs-mirror@mirror.service on the chosen system. The service is named cephfs-mirror.<storage system>.
The cluster's mirroring services are listed on the Replication Service tab of the Ceph Cluster, with their State, Daemon Status (Running or Stopped) and Storage System. The same tab's right-click menu offers Create Mirroring Service and Delete Mirroring Service.
Mirror peers
A mirror peer connects one CephFS file system in the local cluster to one CephFS file system in a remote cluster. It is the scale-out counterpart of a replication link: a scale-out replication schedule names a peer as its destination.
- Ceph Cluster and Ceph File System -- the source: the cluster and file system whose shares are replicated.
- Remote Ceph Cluster and Remote Ceph File System -- the destination. The remote list never offers the cluster selected as the source.
- Peer Name -- read-only. QuantaStor names the peer
<local file system>-to-<remote file system>, and the field previews it.
If the grid has fewer than two qualifying clusters, the dialog closes with A mirror peer requires two Ceph Clusters, each with at least one Ceph File System. If the selected source cluster has no mirroring service, the dialog shows a warning and offers to open Create Mirroring Service; creating the peer without one fails.
On OK, QuantaStor enables mirroring on the source file system, creates a client.mirror_remote user on the remote cluster and generates a bootstrap token there, imports that token on the source, and records the peer UUID Ceph assigns. Only one peer can exist between a given pair of file systems. Any scale-out replication schedule already waiting on this pair starts mirroring its shares once the peer exists.
A peer is one way. Replicating in the other direction would need a mirroring service on the other cluster and a second peer created from that side; that configuration has not been tested. Contact OSNEXUS support before attempting it.
Scale-out replication schedules
A scale-out replication schedule decides when snapshots are taken, which shares are mirrored, and how many snapshots the source keeps. The mirror daemon decides when each snapshot actually crosses to the remote cluster, so the copy on the destination lags the schedule by however long the transfer takes.
The dialog is the one described in Replication schedules. Choosing Scale out replication, mirror CephFS Network Share snapshots to a peered Ceph File System on the General tab changes it as follows:
- Replica Destination lists mirror peers instead of replication links. The remote file system of the peer you pick is the destination, so Remote Pool is disabled.
- Select Volumes & Shares offers only the CephFS shares on the peer's local file system. Storage Volumes cannot be selected, and the nested share options are hidden.
- Snapshot Settings -- the source-side retention counts apply as usual. The destination (checkpoint) counts are greyed out and follow the source values, because the daemon copies the source snapshots as they are.
- Automatic Activation and Advanced Settings are disabled. Both are ZFS replication features.
- Schedule Interval works exactly as for scale-up replication.
If the grid has links but no peers, the scale-out choice is disabled; if it has peers but no links, the dialog opens on scale-out. With neither, it closes with There is nothing to replicate to.
Rules the service enforces, whichever way the schedule is created:
- A schedule cannot mix CephFS shares with ZFS shares.
- Every share on a scale-out schedule must be on the same CephFS file system.
- A mirror peer must exist from that file system to the destination file system.
- The type is fixed once the schedule exists. Modify shows the same disabled tabs. Shares added with Update Selections are registered for mirroring straight away, so they are copied from their next snapshot; the directories of shares taken off the schedule are unregistered within a few minutes.
Each run creates one snapshot per share, described Auto-generated by scale-out file replication schedule '<name>'. There is no consistency group: CephFS snapshots are per directory, so shares on the same schedule are snapshotted independently. A Snapshot Schedule is not needed on the same shares; mirroring follows replication schedules only.
Monitoring
- Mirror peer state -- Normal while Ceph still has the peer configured on the file system, Missing if it has been removed outside QuantaStor. A missing peer raises a CephFS Mirroring Peer Missing alert (severity Error); replication has stopped, and the fix is to delete and recreate the peer.
- Unmanaged peers -- peers configured on a file system directly in Ceph, which QuantaStor has no record of, raise the same alert at most once a day.
- Mirroring service Daemon Status -- Running or Stopped. A stopped daemon raises a CephFS Mirroring Daemon Down alert (severity Error), and replication is stopped until the daemon runs again.
- Schedule state -- a schedule whose peer has been deleted goes to Warning with The CephFS mirroring peer for this replication schedule no longer exists, cannot run. and skips its runs rather than piling up snapshots that would never be copied. The warning clears on the next run after the peer is back.
QuantaStor does not report transfer progress, lag or throughput for scale-out replication. There are no replication tasks, so nothing appears in the Tasks pane beyond the share snapshots, and the schedule's Replica Reports tab is disabled. See Call-home / Alerting to deliver the alerts off the appliance.
Recovery and failback
Scale-out file replication has no built-in failover, promote or reverse replication. The destination file system holds the mirrored directories and their snapshots, but QuantaStor does not create Network Shares for them, has no equivalent of activating a checkpoint, and cannot reverse the direction of a peer.
To recover from the copy, or to send changes made at the destination back to the source, use another mechanism to copy the data back. Setting up a mirroring service and peer in the opposite direction is untested; contact OSNEXUS support before attempting it. Plan the recovery procedure before you need it, since nothing in the product runs it for you.
Removing replication
Remove things in the reverse order you created them: the schedule, then the peer, then the service.
- Replication schedule -- deleting it unregisters its shares' directories for mirroring, unless another scale-out schedule still covers them.
- Mirror peer -- Navigation: Remote Replication → Mirror Peers (section) → select a peer → Mirror Peers (toolbar group) → DeleteDeleting a peer stops replication to the remote file system; snapshots already copied there are not removed. When it is the file system's last peer, QuantaStor also unregisters every mirrored directory and disables mirroring on the file system. Force removes the local record even if the remote cluster is unreachable.
- Mirroring service -- Navigation: Scale-out Storage Configuration → Scale-out Storage Clusters → select a cluster → Service Management → Delete Mirroring Service (toolbar)This stops and removes the daemon on that system. Deleting the cluster's last mirroring service is refused while mirror peers exist, unless you tick Force; forcing it leaves the peers in place with nothing to service them. The
mirroringmanager module stays enabled.
Deleting a CephFS file system that still has mirror peers is refused unless forced; a forced delete removes the peers with it.
Managing scale-out file replication from the CLI
The mirroring service and peer commands are the ceph-file-replication-* group. Each accepts a name or ID wherever it asks for an object.
qs cfrs-create --ceph-cluster=<source cluster> --storage-system=<member to run the daemon on> qs cfrs-list --ceph-cluster-id=<cluster> qs cfrs-get --ceph-file-replication-service=<service> qs cfrs-delete --ceph-file-replication-service=<service> qs cfrp-create --ceph-cluster-id=<source cluster> --ceph-filesystem-id=<source file system> --remote-ceph-cluster-id=<remote cluster> --remote-ceph-filesystem-id=<remote file system> qs cfrp-list --ceph-cluster-id=<cluster> --ceph-filesystem-id=<file system> qs cfrp-get --ceph-file-replication-peer=<peer> qs cfrp-delete --ceph-file-replication-peer=<peer>
For cfrs-create, both arguments are optional: the cluster defaults to the local one, and the storage system to the cluster's coordinator node. Note that cfrs-create takes --ceph-cluster where the other commands take --ceph-cluster-id. The delete commands do not accept a force flag; to force a peer or service delete, use Force in the web interface dialog.
A scale-out schedule is created with the ordinary qs rsch-create. Name the remote CephFS file system as the target pool and list CephFS shares only; QuantaStor recognises the schedule as scale-out from the shares, and no link is named:
qs rsch-create --name=projects-dr --target-pool=<remote file system> --share-list=<share1>,<share2> --schedule-type=interval --interval=60 --rc-src-hourly=24 --rc-src-dailies=7
Related pages
- Remote-replication (DR) -- scale-up (ZFS) replication, and the Create Replication Schedule dialog in full
- Scale-out File Setup (ceph) -- creating the CephFS file systems and shares that get replicated
- Network Shares -- creating and managing shares
- Snapshot Schedules -- local scheduled snapshots, which do not drive mirroring
- Grid Configuration -- managing both clusters from one grid
- Call-home / Alerting -- delivering the mirroring alerts
- QuantaStor CLI Command Reference -- full argument lists for the commands above
Verified against QuantaStor 6.9.0.