Remote-replication (DR): Difference between revisions

From OSNEXUS Online Documentation Site
Jump to navigation Jump to search
m Remove version-era references: removed 1 QuantaStor 5 video link(s) -- all customers run QuantaStor 6 or newer (wiki-docs rule 13, QSTOR-12352)
m Full rewrite from the product: link/schedule dialogs tab by tab with defaults and enablement rules, checkpoint model, delta vs full sync mechanism, failover/failback, failure handling, monitoring. Corrects minimum interval (3 not 5 min), checkpoint numbering suffix, blocked-schedule state, and removes unverifiable performance claims and the unreachable link-refresh procedure (QSTOR-12352)
Line 1: Line 1:
[[Category:admin_guide]]
[[Category:Admin guide]]


=== Volume & Share Remote-Replication (DR Setup) Overview ===
Remote replication copies [[Storage Volumes|Storage Volumes]] and [[Network Shares|Network Shares]] from a Storage Pool on one QuantaStor system to a Storage Pool on another, asynchronously and block-level incrementally, so that a second site holds a usable copy you can bring online if the first site is lost. This page covers the trust relationship the two systems need, the replication schedules that drive the copying, the replica checkpoints that appear at the destination, and how failover and failback work.


QuantaStor supports remote-replication of both Storage Volumes (SAN) and Network Shares (NAS).  Remote-replication is typically setup with a Remote-Replication Schedule but can also be done as a one-time replication without a schedule.  Remote-replication is done asynchronously and is block-level incremental for both Storage Volumes and Network Shares.  Block-level incremental means that only the changes (aka deltas) made to the source volumes and shares are replicated to the destination.  This makes efficient use of WAN bandwidth and enables near-CDP type functionality replication schedules can be set to replicate ever couple minutes to keep the destination copies in close lock-step with the sources.
Everything on this page lives under the '''Remote Replication''' tab.


==== Full Copy vs Incremental Replication ====
{| class="wikitable"
! Section !! Purpose
|-
| [[#Choosing between replication, snapshots and backups|Choosing between replication, snapshots and backups]] || Which of the three features solves your problem
|-
| [[#Requirements|Requirements]] || What must be in place before you start
|-
| [[#Storage System Replication Links|Storage System Replication Links]] || The trust relationship and data path between two systems
|-
| [[#Replication schedules|Replication schedules]] || The dialog that drives repeating replication, tab by tab
|-
| [[#Replica checkpoints|Replica checkpoints]] || What is created on the destination, and delta versus full sync
|-
| [[#One-time replicas|One-time replicas]] || Replicating once without a schedule
|-
| [[#DR failover and failback|DR failover and failback]] || Activating checkpoints, and getting back to normal
|-
| [[#When a replication run fails|When a replication run fails]] || Resumable replication, blocked schedules, alerts
|-
| [[#Monitoring replication|Monitoring replication]] || Where state, history and throughput are visible
|}


All replication starts with a full copy of the source to the destination and this can take some time depending on the speed of the network and the amount of data to be transferred.  After the initial full copy all subsequent replication operations are delta based so only the changes (block level incremental) are transferred.  Replication supports compression and encryption options.
== Choosing between replication, snapshots and backups ==


==== Cascading & N-way Replication ====
QuantaStor has three scheduled data-protection features and they solve different problems. Pick by where the copy has to land and what form it has to take.


Remote-replication of any set of Network Shares and Storage Volumes can be setup to replicate to multiple destinations, this is called N-way replication.  For example, a Network Share 'datastore1' can be setup to be replicated to destination pool-2 and pool-3. This is done by setting up two replication schedules, one to replicate to 'pool-2' and a second to replicate 'datastore1' to destination 'pool-3' where the destination pools can be on any QuantaStor systems within the Storage Grid. 
{| class="wikitable"
! Feature !! Where the copy lands !! What the copy is !! Use it for
|-
| [[Snapshot Schedules]] || The same Storage Pool as the source || A point-in-time snapshot of the volume or share || Fast local recovery points; undoing a bad change; a source for off-host backup
|-
| '''Remote replication''' (this page) || A Storage Pool on another system in the grid || A full, mountable copy of the volume or share, plus its own snapshots || Disaster recovery; surviving the loss of a system, pool or site
|-
| [[Backup Policies]] || A Network Share on a QuantaStor system, from an external NFS/SMB source || Files copied in by an rsync-based backup job || Protecting data that lives outside QuantaStor
|}


Cascading replication may also be setup where a given source like 'datastore1' is first replicated to one destination like 'pool-2' from which it is then replicated to destination 'pool-3'.  Cascading replication reduces the load on the primary storage as the replication is done once to a secondary system and then from there the same data is sent to a tertiary system without any load on the primary.  Cascading replication is setup by creating multiple Replication Schedules and may cascade multiple times from system to system.
Two consequences are worth spelling out.


=== Minimum Requirements for Remote-replication ===
* '''A replication schedule already makes snapshots, so you rarely want a Snapshot Schedule on the same volume or share as well.''' Every replication run snapshots the source, and the schedule's own retention settings decide how many source-side and destination-side snapshots are kept. Adding a [[Snapshot Schedules|Snapshot Schedule]] on top duplicates that work.
* '''Replication is not a backup of a file server that is not QuantaStor.''' If the data you want to protect is on a third-party NAS, that is what [[Backup Policies|Backup Policies]] are for.


* 2x QuantaStor storage systems within a QuantaStor Grid each with a ''Storage Pool''.
== Requirements ==
* Storage pools do not need to be the same size or and the hardware and disk types on the systems can be asymmetrical (non-matching pool size and hardware configurations).
* Replication may be cascaded across many systems from pool to pool.
* Replication may be configured to be N-way, replicating from one-to-many or many-to-one system.
* Replication is incremental/delta based so only the changes are sent and only for actual data blocks, empty space is not transmitted.
* Replication is supported for both Storage Volumes and Network Shares.
* Replication interval may be set to as low as 5 minutes for interval based schedule configurations or scheduled to run at specific hours on specific days.
* All data is AES 256 encrypted on the wire and leverages AES-NI chipset features to accelerate encryption/decryption performance by roughly 8x.  Typical performance overhead due to encryption is 20%.
* Source and destination Storage Pools must be of the same underlying filesystem type (ZFS to ZFS, Ceph to Ceph).
* Replication is not yet supported for Network Shares in scale-out CephFS based Storage Pools.  See ''Backup Policies'' for an alternative approach.


=== Setup Process Overview ===
* '''Two QuantaStor systems that are members of the same storage grid.''' A replication link can only be created between grid members; QuantaStor rejects anything else with {{Code|1=Specified remote storage system is not a member of the grid. Be sure to add the appliance as a grid member first.}} See [[Grid Configuration]] for joining systems into a grid. A grid can span sites, so "replicate to a remote site" means "join the remote appliances to this grid", not "link to a separate grid" -- there is no separate remote-grid replication setup in the interface.
* '''A [[Storage Pools|Storage Pool]] on each system.''' The pools do not need to match in size, layout, disk type or hardware.
* '''A license with the remote-replication feature.''' Without it, creating a link fails with {{Code|1=Remote replication is not licensed for this storage system.}} Trial and cloud-edition licenses always include it.
* '''A destination pool of a compatible type.''' The Remote Pool list in the schedule dialog only offers pools on the remote system, and it never offers XFS, CephFS or Ceph object pools. Ceph RBD pools are offered only when the remote system belongs to the same Ceph cluster as the source. Network Shares in scale-out CephFS pools cannot be replicated; use [[Backup Policies]] for those.
* '''TCP reachability between the two addresses you choose for the link.''' Replication data moves over SSH (or, on an unencrypted link, over an mbuffer TCP session) between the two link addresses.


# Select the 'Remote Replication' tab and choose '''Create''' from the 'Replication Link' toolbar. This will exchange keys between the two systems so that a replication schedule can be created.  You can create an unlimited number of links.  The link also stores information about the ports to be used for remote-replication traffic.
== Setup at a glance ==
'''Navigation:''' Remote Replication --> Storage System Replication Links  --> Replication Link --> Create ''(toolbar)''


# Select the 'Volume & Share Replication Schedules' section in the left pane and choose '''Create''' in the 'Replication Schedule' toolbar to bring up the dialog to [[Remote-replication_/_Disaster_Recovery_Setup#Creating_a_Storage_System_Link_for_Replication | create a new remote replication schedule.]]
[[File:osn_dr_workflow.png|thumb|right|697px|A completed configuration. The two systems are joined by a Storage System Replication Link pair; the schedule on the destination system drives the transfer; the destination holds a checkpoint for each replicated volume and share.]]
## Select the replication link that will indicate the direction of replication.
## Select the storage pool on the destination system where the replicated shares and volumes will reside
## Select the times of day or interval at which replication will be run
## Select the volumes and shares to be replicated
## Click '''OK''' to create the schedule
# Interval based replication schedules start momentarily after creation else one may test the schedule by selecting a Replication Schedule and right clicking. Then clicking '''Trigger Replication Schedule...''' to start it immediately.


=== Diagram of Completed Configuration ===
# Join both systems to the same storage grid, if they are not already. See [[Grid Configuration]].
# Create a [[Storage Pools|Storage Pool]] on the destination system to hold the replicas.
# Create a '''Storage System Replication Link''' between the two systems. This exchanges keys and fixes the addresses replication traffic uses. There is no limit on how many links a system may have.
# Create a '''Replication Schedule''' naming that link, the destination pool, the volumes and shares to replicate, and when to run.
# Interval schedules start on their own shortly afterwards. To test a calendar schedule immediately, trigger it by hand.


[[File:osn_dr_workflow.png|700px]]
Each step is described in the sections below.


== Storage System Replication Links ==
== Storage System Replication Links ==


The first step in setting up remote-replication is to establish a ''Storage System Link'' between two systems in the storage grid. One must have at least two nodes (Storage Systems) configured into a QuantaStor storage grid in order to setup remote replication.  QuantaStor's storage grid communication mechanism connects systems (nodes) together so that they can share information, coordinate activities like remote-replication and high-availability features, while simplifying automation and management operations. After one has setup a storage grid of two or more systems, Storage System Links may be created.  The Storage System Link represents a low level security key exchange between the two nodes so that they may transmit data between pools and the link also specifies which network interface should be used for the transmission of data across the link.
Before anything can be replicated, the two systems need a '''Storage System Replication Link'''. Creating one generates an SSH key pair dedicated to that pair of systems, registers each system's public key on the other, and records which IP address on each side carries replication traffic. Links are created in pairs: making a link from A to B also establishes the reverse link on B, and deleting either one deletes both.


=== Bandwidth Throttling ===
The link is what QuantaStor uses to move data. It is not what records replication progress -- that is the replica association (see [[#Monitoring replication|Monitoring replication]]) -- so links can be deleted and recreated without disturbing existing replication schedules.


WAN links are often limited in bandwidth in a range between 2MB-60MBytes/sec for on-premises deployments and 20MBytes-100MBytes/sec and higher in datacenters depending on the service provider.  QuantaStor does automatic load balancing of replication activities to limit the impact to active workloads and to limit the use of your available WAN or LAN bandwidth.  The limitation of bandwidth is set in the ''Storage System Replication Link'' which can be created and modified in the top section under the Remote Replication tab.
=== Creating a link ===


=== Creating a Storage System Link for Replication ===
[[File:remoterep_link_create.png|thumb|right|500px|The Create Storage System Replication Link dialog. Both ends must already be members of the same storage grid.]]


[[File:Create Replication Link Web.jpg|thumb|512px|To open the dialog choose the 'Remote Replication' tab then choose '''Create''' from the 'Replication Link' toolbar. This dialog can also be accessed via a right click in the "Storage System Replication Links" left pane and choosing '''Create Storage Replication Link...'''.]]
{{Navigation|Remote Replication → Storage System Replication Links → Replication Link → Create ''(toolbar)''}}


Creation of the Storage System Link may be done through the QuantaStor Manager web interface by selecting the 'Remote Replication' tab, and then choosing the '''Create''' button from the 'Storage System Link' in the toolbar.  Select the IP address on each system to be utilized for communication of remote replication network traffic.
'''Link System (A)''' and '''Link System (B)''' each take a Storage System from the grid and an address on it:


'''Navigation:''' Remote Replication --> Storage System Replication Links --> Replication Link --> Create ''(toolbar)''
* '''Storage System''' -- the grid member. Selecting a system in (A) auto-selects a different system in (B), so the dialog opens ready to create a system-to-system link.
* '''IP Address''' -- a drop-down of the addresses actually configured on that system's network ports. Floating (cluster VIF) addresses and ports without a valid IP address are filtered out, so a link is always pinned to a fixed address rather than one that can move.
* '''External IP Address''' -- a free-text alternative, enabled by selecting its radio button. It is pre-filled from the system's external hostname when one is set (<code>[[QuantaStor CLI Command Reference#system-modify|qs system-modify]] --ext-hostname</code>), which is how cloud deployments record a public address.


Replication for two nodes.
'''The address you give for Link System (A) must be one that is configured on a port of that system.''' QuantaStor validates it and fails with {{Code|1=Invalid storage system link specification, IP address '&lt;addr&gt;' is not configured on any network port for system '&lt;name&gt;'.}} otherwise. In practice this means the External IP Address field is only usable for the far side of the link.


[[File:Remote Replication two Nodes.jpg|712px]]
Below the two fieldsets:


Once the links have been created they'll appear as two separate directional links in the web user interface as shown in the above screenshot.
* '''Bandwidth Limit''' -- see [[#Bandwidth throttling|Bandwidth throttling]] below. The field is pre-filled with 200 MB/sec; entering 0 tells the service to use its own default of 100 MB/sec.
* '''Description''' -- free text.
* '''Encryption''' -- on by default. See [[#What the link does with encryption and compression|What the link does with encryption and compression]].
* '''Compression''' -- off by default.


=== Configuring Storage System Links for High-Availability Configurations ===
Selecting the same Storage System for both (A) and (B) creates a '''local link''', used to replicate between two pools on one appliance. Encryption and Compression grey out in that case, because there is no network hop to protect or compress.


Storage System Links are bi-directional and deletion of a link in one direction will automatically delete the link in the reverse direction.  Remote-replication links which maintain the replication status information between network shares and storage volumes are unaffected by the deletion of storage systems link but if no valid storage system link is available when a remote replication schedule is activated then an alert notice will be raised.  The HA failover system is designed to work in tandem with the DR system so in the event that a ''Storage Pool'' is manually or automatically failed-over to another system the scheduler will automatically select and use the appropriate Storage System Link pair to replicate between the designated pools.  Note though, that one must establish all the necessary storage system links so that remote replication may continue uninterrupted.  For example, if one has systems A & B configured with an HA pool which has volumes replicating to a HA pool managed by systems C & D then for Storage System Link pairs must be setup.  Specifically A <--> C, A <--> D, B <--> C, B <--> D so that no matter how the source and destination pools are moved between node pairs that the remote replication schedule will be able to continue replication normally.
=== What the link does with encryption and compression ===


=== Modifying a Storage System Link ===
The two checkboxes select the transport, not just a flag:


Use the modify dialog to adjust the replication schedule as needed. Activation of the schedule by the schedule manager within QuantaStor will continue normally per the new settings automatically at the next activation point.
* '''Encryption on''' (the default) tunnels the ZFS send stream over SSH to {{Code|1=root}} on the remote link address, using the key pair created with the link (kept in {{Code|1=/var/opt/osnexus/quantastor/replication-keys/}}). The appliance's shipped SSH client configuration lists {{Code|1=aes256-gcm@openssh.com}} first, so AES-256 is what two QuantaStor systems negotiate. ''(Note: due to the frequency of security updates this may be out of date and {{Code|1=/etc/ssh/ssh_config}} on the appliance should be referenced.)''
* '''Encryption off''' sends the stream over a plain TCP session between {{Code|1=mbuffer}} processes instead. It is faster on a trusted, private link and offers no confidentiality.
* '''Compression on''' inserts {{Code|1=lz4}} compression into the send pipeline -- but only when the source dataset is not already compressed. Replicating a compressed pool gains nothing from the option, so QuantaStor skips it.


'''Navigation:''' Remote Replication --> Storage System Replication Links --> Replication Link --> Modify ''(toolbar)''
Before each run, QuantaStor pre-checks that the key file exists and that it can reach the remote appliance over SSH. A failed pre-check reports {{Code|1=Replication pre-check failed to verify SSH tunnel access to remote appliance}}, which points at firewall rules or a wrong link address rather than at the data itself.


[[File:Modify Remote Replication Link Web.jpg|512px|To access this dialog either right click on a Storage System Link from either the left or center pane or press the '''Modify Storage System Link...''' button in the "Storage System Link" toolbar.]]
=== Bandwidth throttling ===


=== Deleting a Storage System Link ===
The '''Bandwidth Limit''' on the link caps total replication throughput for everything using that link, in MB/sec. QuantaStor enforces it by rate-limiting each replication process and '''re-dividing the link's limit evenly across the streams that are currently running''': four concurrent replications on a 200 MB/sec link get 50 MB/sec each, and when three of them finish the survivor is raised back to 200 MB/sec. Rebalancing happens whenever the set of running replications changes, so you do not have to plan around how many volumes are in a schedule.
[[File:Delete Remote Replication Link Web.jpg|512px|thumb|To access the dialog either right click on "Storage System Link" in the left or center pane and click '''Delete Storage System Replication Link...''', else '''Delete''' from the "Replication Link" toolbar.]]


To remove a storage system link simply right-click on the link within the web user interface, ''Tree View'',  and choose '''Delete Storage System Replication Link..'''.
Change it at any time with '''Modify''' on the link; the new value is applied to running replications at the next rebalance.


Deletion of a storage system replication link will also delete the reverse direction link as they are added and removed as pairs.  If the IP addresses or other configuration settings on a given set of systems has changed one may delete the links and recreate them without having to adjust the replication schedules.  The ''schedules automatically select the currently available link'' necessary for a given replication task.
{{Navigation|Remote Replication &rarr; Storage System Replication Links &rarr; Replication Link &rarr; Modify ''(toolbar)''}}


'''Navigation:''' Remote Replication --> Storage System Replication Links --> Replication Link --> Delete ''(toolbar)''
The Modify dialog is the same form as Create with the link selected, and it shows the link's current limit rather than the 200 MB/sec default.


== Creating a one-time Remote Replicas ==
=== Link state ===


Once a storage system link pair has been established replication features for the replication of volumes and network shares between the systems are made available.  Instant replication of volumes and shares is accessible by right-clicking on a given volume or share to be replicated, then choose '''Create Remote Replica...''' from the menu.  Creating a remote replica is much like creating a local clone, only the data is being copied over to a storage pool in a remote storage system. 
A link's '''State''' column tracks the state of the remote grid member, re-checked about once a minute:


When selecting replication options, first select the destination storage system to replicate too (only systems which have established and online storage system links will be displayed). The destination storage pool within that system should be utilized to hold the remote replica.  
* '''Normal''' -- the remote system is a healthy grid member.
* '''Offline''' -- the remote system is a grid member but is not in the Normal state. The replica associations using the link are marked offline with it, and schedules using it are skipped.
* '''Warning''' -- the grid holds no information about the remote system ID at all, so the link points nowhere.


If the volume or share had been previously replicated then an incremental ''diff'' replication is optimal. One can force a complete replication as well and these will be numbered at the destination with a suffix like ''.1_chkpnt'', ''.2_chkpnt'' and so on.  Incremental replication from that point will show the destination check-point with a GMT based timestamp in the name of the check-point.
A link returns to Normal on its own when the remote system does. There is nothing to reset by hand.


== Remote Replication Schedules (Disaster Recovery Failover) ==
=== Deleting a link ===


Remote replication schedules provide a mechanism for automatic replication of changes to volumes & shares to destination Storage Pool.  On the destination Storage Pool the copied volumes and shares are referred to as checkpoints and are easily identified by their _chkpnt suffix.  Replication schedules run on a delay timer which indicates the delay time between replications or a fixed schedule.  If a calendar based replication schedule is still running when the schedule is to be run again it will delay until the next interval time.  Interval based replication schedules have a set amount of delay time between each completed replication and will not activate again until the last transfer has completed plus the specified delay time has elapsed.
{{Navigation|Remote Replication &rarr; Storage System Replication Links &rarr; Replication Link &rarr; Delete ''(toolbar)''}}


=== Creating Remote Replication Schedules ===
Deleting a link disconnects it on the remote system, removes the key pair for that system pair, and deletes the reverse-direction link. Existing replica associations survive. If the addresses on a pair of systems change, delete the links and recreate them -- schedules pick up the new link automatically, as described next.


To create a schedule navigate to the 'Volume & Share Replication Schedules' section of the tree view after selecting the 'Remote Replication' tab at the top of the screen.  Right-click on the section header and choose 'Create Schedule...'.  Alternatively, from the 'Replication Schedule' toolbar select '''Create''' as is shown below.
=== Links in high-availability configurations ===


'''Navigation:''' Remote Replication --> Volume & Share Replication Schedules --> Replication Schedule --> Create ''(toolbar)''
'''Every replication run re-selects its link.''' At each activation the scheduler looks through the local system's links for one whose source and remote system IDs match the systems that currently own the source and destination pools, and repairs the schedule's stored link reference if it has changed. That is what lets replication survive an HA pool failover on either end -- but only if a link exists for the pair of systems the pools have moved to.


So with an HA pool on systems A and B replicating to an HA pool on systems C and D, create all four links: A to C, A to D, B to C, B to D. Any missing combination shows up as a skipped run with the status {{Code|1=Storage System Replication Link lookup failed.}} A run is also skipped when the matching link exists but is not in the Normal state.


[[File:Create Rplctn Schedule Web.jpg|512px|frame|To access this dialog either click '''Create''' from the 'Replication Schedule' toolbar or right click on left or center panes for 'Volume & Share Replication Schedules' and click on '''Create Schedule...".]]
Replication is likewise skipped while an HA failover of the destination pool is in progress, with the status {{Code|1=Failover in progress on the target pool}}.


Besides selection of the volumes and/or shares to be replicated you must select the number of snapshot checkpoints to be maintained on the local and remote systems as shown below.  You can use these snapshots for off-host backup and other data recovery purposes as well so there is no need to have a Snapshot Schedule which would be redundant with the snapshots which will be crated by your replication schedule.  If you choose a 'Max Delta Points' of 5 then up to 5 snapshot checkpoints will be retained. If for example you were replicating nightly at 1 a.m. each day of the week from Monday to Friday then you will have a week's worth of snapshots as data recovery points.  If you are replicating 4 times each day and need a week of snapshots then you would need 5x4 or a 'Max Delta Points' setting of 20.
== Replication schedules ==


[[File:Create Rep Schd - Snap Settings.jpg|512px]]
A replication schedule is the normal way to run remote replication. It names a link, a destination pool, a set of volumes and shares, when to run, and how many snapshots to keep on each side.


=== Interval vs Calendar Schedules ===
[[File:remoterep_tab.png|thumb|right|800px|The Remote Replication tab with the Volume & Share Replication Schedules section selected. The centre pane shows the schedule's report chart, the schedule grid, and the volumes and shares it replicates.]]
[[File:Replication_Schd_-_Schd_Intv.jpg|512px|thumb|Schedule intervals between replication activity.]]


Replication can be done on a schedule at specific times and days of the week or can be done continuously based upon an interval.   The interval represents the amount of time between replication activities where no replication is done.  The system automatically detects active schedules and will not active a given schedule again while any volumes or shares are still being replicated.  As such the replication schedule interval is ''more of a delay interval'' between replications and can be safely set to a low value.  Since only the changes are replicated it is often better to replicate more frequently to replicate throughout the day versus replicating all at once at the end of the day.
'''The schedule object is owned and run by the system that owns the destination pool''', not by the source system. This is visible in the Storage System column of the task log and in <code>[[QuantaStor CLI Command Reference#replication-schedule-get|qs replication-schedule-get]]</code>, and it is why the destination system must be up for a schedule to be created, modified or triggered.


'''Navigation:''' Remote Replication --> Volume & Share Replication Schedules --> Replication Schedule --> Create ''(toolbar)''
{{Navigation|Remote Replication &rarr; Volume & Share Replication Schedules &rarr; Replication Schedule &rarr; Create ''(toolbar)''}}


=== Modifying Remote Replication Schedules ===
The Create Replication Schedule dialog is a six-tab wizard. '''Previous''' and '''Next''' walk the tabs; '''OK''' is available from any tab.


Schedules may be adjusted at any time by right-clicking on the schedule then choosing ''Modify Schedule..'' to make changes.
=== General tab ===


'''Navigation:''' Remote Replication --> Storage System Replication Links --> Replication Link --> Modify ''(toolbar)''
[[File:remoterep_sched_general.png|thumb|right|640px|General tab. Replica Destination is the replication link, which fixes the direction of replication.]]


[[File:Mdfy Replication Schedule - Web.jpg|1024px]]
* '''Policy Name''' -- pre-filled with a generated name such as {{Code|1=replication-sched-1}}. Names may contain letters, digits and {{Code|1=-_.}} only.
* '''Description''' -- free text.
* '''Replica Destination''' -- the Storage System Replication Link to use. '''This choice sets the direction of replication''': the link {{Code|1=A --> B}} replicates from A to B. Because links exist in both directions, both appear in the list. Selecting a link that is offline clears the Remote Pool list.
* '''Remote Pool''' -- the destination Storage Pool, labelled with its system: {{Code|1=pool-name [system-name]}}. Only pools on the link's remote system appear, subject to the type rules in [[#Requirements|Requirements]].
* '''Start date''' and '''Enabled''' -- when the schedule becomes eligible to run, and whether it is active. Enabled is checked by default.


== Managing Replication Checkpoints ==
If no replication link exists at all, the dialog closes immediately with ''There are no remote storage systems to replicate your volumes. Add a link to a remote storage system first.''


=== Volume Checkpoints ===
=== Schedule Interval tab ===


Checkpoint volumes have the suffix ''_chkpnt'' typically followed by a ''GMT'' timestamp. At the completion of each replication cycle the check-point volume parent which has no ''GMT'' timestamp suffix will contain all the latest information from the last transfer. In the event of a failure of a primary node one need only access a given check-point ''Storage Volume'' and it will switch to the ''Active Replica Checkpoint'' stage.  This indicates that the check-point volume may have been used or written to by users so care should be taken to rollback that information using ''Replica Rollback...'' to the primary side as part of the DR fail-back process to preserve any changes.
[[File:remoterep_sched_interval.png|thumb|right|640px|Schedule Interval tab. The day/hour grid and the timer interval are mutually exclusive; the unselected one greys out.]]


=== Network Share Checkpoints ===
'''Schedule Type''' picks between two mutually exclusive modes, and the group you did not pick is disabled:


Network share check-points also contain the ''_chkpnt'' suffix just as with ''Storage Volume'' replica check-points and this can create some challenges if users are expecting to find their data under a share with the same name.  For example, a share named ''backups'' will have the name ''backups_chkpnt'' at the destination / DR site system. There is an easy way to resolve this through the use of Network Share Aliases. Simply right-click on the check-point for the ''Network Share'' (eg. backups_chkpnt) and choose ''Create Alias/Sub-share...'' which allows one to assign alternate names to a share as can be seen below. After the alias is created as ''backups'' the share will be accessible both as ''backups'' and ''backups_chkpnt''.  If users find the dual-naming confusing then the ''browsable'' setting may be adjusted on the ''backups_chkpnt'' to hide it from users using the ''Modify Share'' dialog.
* '''Use day/hour selections''' (default) -- a calendar schedule. Tick the days of the week and the hours of the day at which replication should start. The default selection is Monday to Friday at 12 AM, 4 AM, 8 AM, 12 PM, 4 PM and 8 PM. '''All Days''' and '''All Hours''' toggle a whole row of checkboxes. '''Offset''' delays every trigger by 0-59 minutes, so a 30-minute offset on a 1 AM and 4 AM schedule fires at 1:30 AM and 4:30 AM -- useful for staggering several schedules that would otherwise all start on the hour.
* '''Use timer interval''' -- a rest interval, in minutes, between the '''end''' of one run and the start of the next. The minimum is 3 minutes and the slider runs to 360; the default is 30. Entering less than 3 resets the field and raises a warning.


'''Navigation:''' Storage Management --> Volume & Share Replication Schedules --> Create Alias... ''(rightclick)''
The distinction matters: an interval schedule measures the gap after a run completes, so a low interval is safe -- it cannot pile runs on top of each other. QuantaStor never starts a run for a schedule whose previous run is still working; a calendar trigger that arrives while the schedule is busy is skipped rather than queued.


[[File:Create Network Share Alias.jpg|512px]]
Because only changed blocks are sent, replicating often through the day generally transfers less at each run and keeps the destination closer to the source than one large nightly run.


=== Manual DR Failover - Activating Checkpoints ===
'''Report Retention Count''' at the bottom of the tab sets how many replication summary reports to keep for this schedule. The default is 30; a value of 0 is replaced with 30 by the service.


[[File:qs_activate_checkpoints.png|600px|thumb|Choose 'Activate Checkpoints' to activate the use of _chkpnt Volumes and Shares at the DR site.]]
=== Select Volumes & Shares tab ===


To activate the Storage Volume and Network Share checkpoints associated with a given Remote Replication Schedule, simply right-click on the schedule in the 'Volume & Share Replication Schedules' section and choose 'Activate Checkpoints...'.  From that dialog one can select the specific subset of checkpoints to be activated or all the checkpoints. If selected the 'Auto-create Network Share Aliases' option will create aliases for the '_chkpnt' Network Shares so that they may be referenced at the DR site using the same names as the source/origin share names without the '_chkpnt' suffix.
[[File:remoterep_sched_select.png|thumb|right|640px|Select Volumes & Shares tab. The Network Shares and Storage Volumes sub-tabs each have their own selection list; a schedule can carry both.]]


=== Automatic DR Failover - Activating Checkpoints via VIF ===
Two sub-tabs, '''Network Shares''' and '''Storage Volumes''', each with a dual list: available on the left, selected on the right. Move items with the arrow buttons or by dragging. A Storage Pool filter, a text search and '''Hide Nested Shares''' narrow the left-hand list.


[[File:Mod Rplctn Schd - Auto Activation.jpg|600px|thumb|Choose 'Enable Automatic Checkpoint Activation' then a Site Cluster VIF to activate the schedule.]]
At least one volume or share must be selected, from either sub-tab; a schedule can replicate volumes and shares together. Selecting both a parent share and a share nested inside it is rejected -- replicate the parent (with '''Recursively replicate nested shares''' on the Advanced Settings tab) or the children, not both.


In some environments it may be preferable to have the DR site checkpoints activate automatically in the event of an outage of a given primary site.  This may be easily setup by creating a 'Site Cluster' in the 'High-availability VIF Management' section.  The Site Cluster must spans the source and destination systems so that when the source side system goes offline the VIF will automatically move to the destination side.  After the 'Site Cluster' is created one will next create a Site VIF (virtual network interface).  When the VIF becomes active at the destination side it will automatically activate the Remote Replication Schedule's checkpoints just as if it was done manually using the Activate Checkpoints dialog.  Once the VIF has been created within the Site Cluster choose the 'Modify Remote Replication Schedule' option in the WUI and then select the 'Automatic Activation' tab.  Within there select the 'Enable Automatic Checkpoint Activation' option and then choose the VIF to activate.
Changing the selection later is a separate dialog, '''Update Selections''', rather than part of Modify:


=== Resolving DR Failover - Deactivating Checkpoints ===
{{Navigation|Remote Replication &rarr; Volume & Share Replication Schedules &rarr; Replication Schedule &rarr; Update Selections ''(toolbar)''}}


After a DR site failover has been done by 'Activating Checkpoints' they must be deactivated before replication may resume and overwrite changes to the destination _chkpnts.
Active replication tasks finish before the new selection takes effect at the next activation.
This can be easily done using the 'Deactivate Checkpoints...' dialog by right-clicking on the replication schedule and then choosing that option from the pop-up menu. To automatically activate and resume normal replication after deactivating the checkpoints select the 'Re-enable Replication Schedule' option within the dialog.  Alternatively one may use the 'Enable Schedule' dialog to re-enable the schedule as a second step.


=== Manual Activation/Deactivation of Checkpoints ===
=== Snapshot Settings tab ===


Via the 'Modify Network Share' and 'Modify Storage Volume' dialogs there is a check-box option to toggle the 'Active Replica Checkpoint' flag.  This option may only be toggled on shares and volumes that are checkpoints with the _chkpnt suffix.
[[File:remoterep_sched_snapshots.png|thumb|right|640px|Snapshot Settings tab. The Source column controls snapshots kept on the source volume or share; the Checkpoint column controls snapshots kept on the destination checkpoint.]]


=== Session based Activation of Checkpoint Status ===
Replication is driven by snapshots, and this tab decides how many are kept on each side. Two independent groups:


If a given destination side Storage Volume or Network Share checkpoint is accessed via iSCSI or SMB respectively, QuantaStor will automatically mark the given checkpoint as an active checkpoint.  This ensures any changes to the destination are not lost.   The replication scheduler in QuantaStor does a check before starting any new replication to ensure the destination checkpoints are not active and if detected as active the Replication Schedule is automatically transitioned to the ''Offline'' state.
'''Short Term Snapshot Retention Settings''' -- '''Max Short Term Snapshots''' is the number of rotating snapshots used to compute deltas. Each run snapshots the source, sends the difference against the newest snapshot the destination also has, and the oldest short-term snapshot is removed once the count is exceeded. The default is 3, which is also the recommended value; the maximum is 1024. The service raises anything below 2 to 2.


=== Resolving a DR Site Failover ===
'''Long Term Snapshot Retention Settings''' -- additional recovery points promoted out of the short-term snapshots and kept for longer, with an independent count for the source and for the destination checkpoint:


Once a given set of checkpoints have been marked as 'Active Checkpoints' these may be used as part of a Disaster Recovery (DR) failover strategy for any amount of time.  To transition back to normal operation of the Replication Schedule to send data from the sources to the destination checkpoints one will choose one of two options.
{| class="wikitable"
! Rule !! What a count of ''n'' retains !! Source default !! Checkpoint default
|-
| Hourly || ''n'' snapshots spanning the last ''n'' hours || 2 || 4
|-
| Daily || ''n'' snapshots spanning the last ''n'' days, at least a day apart || 2 || 4
|-
| Weekly || ''n'' snapshots at least 7 days apart || 2 || 4
|-
| Monthly || ''n'' snapshots at least 30 days apart || 0 || 2
|-
| Quarterly || ''n'' snapshots at least 90 days apart || 0 || 2
|}


==== Option 1: Throw out the destination side changes & resume replication ====
'''Suggested Defaults''' restores the values in the table; '''Clear''' sets every count to 0. Each field caps at 1000.


In the process of testing one's DR site failover it'll be common to activate the DR site checkpoints periodically and then simply re-activate the Remote Replication schedule.  The easiest way to do this is to use the 'Deactivate Checkpoints' dialog as it makes it easy to deactivate all the checkpoints at once and re-enable the Replication Schedule in a single operation.
The two read-only lines at the bottom, '''Max Total Source Snapshots''' and '''Max Total Checkpoint Snapshots''', add the long-term counts to Max Short Term Snapshots and update as you type. They are the number to watch: this is how many snapshots the schedule can accumulate per volume or share on each side.


[[File:Deactivate Chkpts for Replication Schd.jpg|512px]]
=== Automatic Activation tab ===


==== Option 2: Send checkpoint side changes back to the source side, then resume replication from source to destination ====
'''Enable Automatic Checkpoint Activation''' ties DR failover to a Site Cluster virtual interface. '''Activate on VIF arrival''' selects the VIF, and is disabled until the checkbox is ticked. Both are off and empty by default. See [[#Automatic failover on VIF arrival|Automatic failover on VIF arrival]] for what happens when it fires.
[[File:Rollback Strg Rep - Web.jpg|512px|thumb|Sending the changes made to the DR site checkpoints back to the source side.]]


The process of sending the changes made to the DR site checkpoints back to the source side is called 'Rollback from Replica...' in QuantaStor and is done from the 'Volume & Share Replica Associations' section.  Within that section one must select the link between a given source and destination and then choose 'Rollback from Replica...' in the pop-up menu.  This must be done for each share and volume to be rolled back to the source side. '''Once the rollback activity is completed be sure''' to do a 'Rescan Storage Pools...' from the Storage Management-->Storage Pools section on each source side system.  Once all the rollback replication activity has completed the source side will have a GMT snapshot with the same timestamp as the destination side.  In effect rollback is done by taking a snapshot of the destination side changes and then sending these delta based incremental changes back to the source side volume/share.  Use caution with this as the source side will be an identical copy of the destination after rollback so any changes made to the source side will be overwritten.
=== Advanced Settings tab ===


After the data has been sent (rolled-back from DR site) use the steps from Option 1 to resume regular replication.
[[File:remoterep_sched_advanced.png|thumb|right|640px|Advanced Settings tab. Enable target checkpoint recovery and Enable resumable replication are on by default; the latter depends on the destination pool.]]
 
Seven checkboxes. Two are on by default and five are off.
 
* '''Enable source snapshot reuse (use with N-way and cascading A->B->C replication)''' -- off. Replicates from the most recent existing source snapshot instead of taking a new one. Turn it on for the second and subsequent schedules when one source replicates to several destinations, or in a cascade, so that all destinations end up with a common replica checkpoint to work from. If the schedule finds no suitable existing snapshot it fails rather than silently taking a new one.
* '''Enable target checkpoint recovery (unchecking will perform full replication)''' -- '''on'''. Keeps replication incremental by finding and reusing the existing {{Code|1=_chkpnt}} at the destination even when the replica association between source and checkpoint has been deleted. Unchecking it asks for a full replication, and the dialog raises a confirmation prompt before accepting that.
* '''Recursively replicate nested shares''' -- off. Includes shares nested under a selected share. It has no effect on Storage Volumes, and it disables resumable replication.
* '''Enable resumable replication''' -- '''on''' when the selected Remote Pool supports it, and disabled and cleared otherwise. Lets an interrupted run pick up where it stopped rather than resending from the beginning; see [[#When a replication run fails|When a replication run fails]]. It is greyed out while '''Recursively replicate nested shares''' is on, and greyed out until a Remote Pool is selected.
* '''Copy source share settings to the checkpoint network share''' -- off. On a '''full''' replication of a Network Share, copies the source share's settings onto the checkpoint share, so the DR copy is exported the way the original was. Shares only; ignored for volumes, and not applied on incremental runs.
* '''Reclaim Orphaned Snapshots''' -- off. Adopts replication snapshots left behind by a schedule that has since been deleted, so the new schedule counts and rotates them instead of leaving them to accumulate. In the '''Modify''' dialog this checkbox enables a '''Reclaim Schedule with ID''' field, where you paste the old schedule's ID -- it appears in the schedule column of the property view for a share or volume snapshot.
* '''Force''' -- off. Cascading replication is meant to be run with source snapshot reuse; Force overrides that and creates new snapshots on the checkpoints instead.
 
=== N-way and cascading replication ===
 
A source can be replicated to more than one destination by creating more than one schedule for it. Two shapes are common:
 
* '''N-way''' -- a share {{Code|1=datastore1}} replicates to {{Code|1=pool-2}} on one system and to {{Code|1=pool-3}} on another. Create two schedules, each with the same source and a different link and destination pool. Many-to-one works the same way: several schedules on different sources can share a destination pool.
* '''Cascading (A to B to C)''' -- {{Code|1=datastore1}} replicates from A to B, and the checkpoint on B is itself the source of a second schedule from B to C. Cascading keeps the extra transfer off the primary system: A sends once, and B fans it out. It can be chained further.
 
'''On the second and subsequent schedules of an N-way or cascading configuration, tick '''Enable source snapshot reuse''' on the Advanced Settings tab.''' Without it each schedule takes its own new source snapshot, and the destinations end up with no snapshot in common -- which is what future incremental replication between them depends on. With it, the second schedule replicates from the snapshot the first one already made.
 
QuantaStor recognises a cascade by the {{Code|1=_chkpnt}} in the source name and does not double the suffix, so the second hop's checkpoint keeps the same name rather than becoming {{Code|1=datastore1_chkpnt_chkpnt}}.
 
=== Modifying, enabling and disabling a schedule ===
 
{{Navigation|Remote Replication &rarr; Volume & Share Replication Schedules &rarr; Replication Schedule &rarr; Modify ''(toolbar)''}}
 
Modify offers five tabs -- the same as Create without '''Select Volumes & Shares''', which is handled by '''Update Selections''' -- and its Advanced Settings tab adds the '''Reclaim Schedule with ID''' field described above. The link and destination pool of an existing schedule cannot be changed; create a new schedule to replicate somewhere else. Changed settings take effect at the next activation.
 
'''Enable''' and '''Disable''' turn a schedule on and off without deleting it. A disabled schedule shows a state of Offline. '''Delete''' removes the schedule and leaves the snapshots and checkpoints it created in place.
 
=== Running a schedule now ===
 
An interval schedule starts shortly after it is created. To run any schedule immediately, use <code>[[QuantaStor CLI Command Reference#replication-schedule-trigger|qs replication-schedule-trigger]] --schedule=&lt;name&gt;</code>. The command takes one-off overrides of three of the Advanced Settings options -- {{Code|1=--reuse-source-snapshot}}, {{Code|1=--reuse-target-chkpnt}} and {{Code|1=--copy-share-settings}} -- without changing the schedule itself:
 
<pre style="font-size: smaller">
qs replication-schedule-trigger --schedule=nightly-dr
qs replication-schedule-trigger --schedule=nightly-dr --reuse-target-chkpnt=true
</pre>
 
Recovering a repaired association is the common reason to pass {{Code|1=--reuse-target-chkpnt=true}} by hand: it reconnects a source to the best-matching checkpoint on the destination pool instead of starting a fresh full copy.
 
== Replica checkpoints ==
 
=== What appears on the destination ===
 
For each replicated source, QuantaStor creates one object on the destination pool named after the source with a {{Code|1=_chkpnt}} suffix -- {{Code|1=backups}} becomes {{Code|1=backups_chkpnt}}. If that name is already taken it appends {{Code|1=.1}}, {{Code|1=.2}} and so on, though current releases prefer to reuse the existing {{Code|1=_chkpnt}} for a full resync rather than create a numbered one.
 
The checkpoint is a real, mountable volume or share, not an archive format. Alongside it sit the replicated snapshots, timestamped in GMT:
 
<pre style="font-size: smaller">
source volume      doctest-repl-vol
source snapshot    doctest-repl-vol_GMT20260903_045938
checkpoint volume  doctest-repl-vol_chkpnt
checkpoint snapshot doctest-repl-vol_chkpnt_GMT20260903_045938
 
source share        doctest-repl-share
checkpoint share    doctest-repl-share_chkpnt
checkpoint snapshot doctest-repl-share_chkpnt@GMT-2026.09.03-04.59.38
</pre>
 
The suffix-free checkpoint always holds the state of the most recently completed run. The timestamped snapshots beside it are the retained recovery points, governed by the Snapshot Settings tab.
 
'''A share checkpoint is not exported and not active until you activate it.''' QuantaStor deliberately leaves {{Code|1=_chkpnt}} shares offline during pool discovery so that a DR copy is not accidentally serving clients.
 
Because the checkpoint share carries the {{Code|1=_chkpnt}} suffix, users looking for {{Code|1=backups}} at the DR site will not find it. Give the checkpoint a Network Share alias named after the source and it is reachable under both names; '''Activate Checkpoints''' can create those aliases for you. See [[Network Shares]] for aliases in general.
 
{{Navigation|Storage Management &rarr; Network Shares &rarr; ''a checkpoint share'' ''(select + right-click)'' &rarr; Create Alias/Sub-share...}}
 
Once the alias exists the data is reachable as both {{Code|1=backups}} and {{Code|1=backups_chkpnt}}. If users find the two names confusing, clear '''browseable''' on the {{Code|1=_chkpnt}} share in '''Modify Network Share''' so only the alias is advertised over SMB.
 
=== Delta sync versus full sync ===
 
Only the first run of a source is a full copy. Every run after that sends only changed blocks.
 
The choice is made per run, and it is mechanical: QuantaStor looks for the '''newest GMT-stamped snapshot that exists on both the source and the destination checkpoint''' and sends the difference from there. If no such common snapshot exists -- the first run, a checkpoint with no snapshots yet, or a source whose only snapshot is the one just taken -- the run sends everything.
 
This is why the Snapshot Settings retention counts matter operationally. Retaining too few snapshots on either side can leave no common point between two runs, and the next run falls back to a full copy.
 
A full copy is forced deliberately in two places: clearing '''Enable target checkpoint recovery''' on a schedule, and choosing '''Create a new replica on the target storage system. (full copy)''' in the one-time replica dialog.
 
=== The Active Replica Checkpoint state ===
 
A checkpoint carries a flag, '''Active Replica Checkpoint''', which means "somebody may have written to this copy". It exists to stop replication from overwriting DR-site changes.
 
'''It is set three ways:'''
 
# '''Explicitly''', by the '''Activate Checkpoints''' dialog or <code>[[QuantaStor CLI Command Reference#replication-schedule-activate-checkpoints|qs replication-schedule-activate-checkpoints]]</code>.
# '''Automatically on client access.''' An iSCSI or Fibre Channel login to a checkpoint volume, or an NFS/SMB client on an active checkpoint share, marks it active on its own. This is a safety net: it means data written to a DR copy that someone mounted without going through the failover dialog is still protected.
# '''By hand''', with the Active Replica Checkpoint checkbox in the '''Modify Storage Volume''' and '''Modify Network Share''' dialogs. The checkbox only applies to objects that are checkpoints.
 
'''While any checkpoint in a schedule is active, the schedule will not run.''' The scheduler checks before every activation, and a schedule that is blocked moves to a Warning state with a status such as {{Code|1=Schedule blocked due to active replication checkpoint(s) 1 shares (including backups_chkpnt)}}, and records a Skipped entry in its replication report. Every checkpoint in the schedule has to be deactivated before replication resumes.
 
For volumes the flag is stored on the dataset itself, as the ZFS user property {{Code|1=quantastor:isactivecheckpoint}}, so it survives a pool export and import.
 
== One-time replicas ==
 
[[File:remoterep_replica_create.png|thumb|right|617px|Create Remote Volume Replica. Replica Name and Description are disabled unless the full-copy option is selected.]]
 
Replicating once, without creating a schedule, is useful for seeding a destination or for moving a copy of a volume to another system on demand.
 
{{Navigation|Remote Replication &rarr; Storage System Replication Links &rarr; Remote Replication &rarr; Create Volume Replica ''(toolbar)''}}
 
{{Navigation|Remote Replication &rarr; Storage System Replication Links &rarr; Remote Replication &rarr; Create Share Replica ''(toolbar)''}}
 
* '''Storage Volume''' / '''Network Share''' -- the source.
* '''System Link''' -- the replication link, which again fixes the direction.
* '''Remote Storage Pool''' -- the destination pool on the link's remote system.
* '''Replicate over a previous replica target if available (diff copy).''' -- the default. Sends only the changes against an existing checkpoint.
* '''Create a new replica on the target storage system. (full copy)''' -- sends everything to a new checkpoint.
* '''Replica Name''' and '''Description''' under Optional Settings -- '''both are disabled while the diff-copy option is selected''' and are enabled by selecting full copy, which is the case where a new destination object is named.
 
The '''Stop Volume Replication''' and '''Resume Volume Replication''' toolbar buttons act on a replica association rather than on a source, and correspond to <code>[[QuantaStor CLI Command Reference#replica-assoc-stop|qs replica-assoc-stop]]</code> and <code>[[QuantaStor CLI Command Reference#replica-assoc-sync|qs replica-assoc-sync]]</code>. There is no CLI equivalent of the one-time replica dialogs themselves.
 
== DR failover and failback ==
 
=== Manual failover: activating checkpoints ===
 
[[File:remoterep_activate_chkpnt.png|thumb|right|640px|Activate Checkpoints. Selecting nothing activates every volume and share in the schedule.]]
 
{{Navigation|Remote Replication &rarr; Volume & Share Replication Schedules &rarr; Replication Schedule &rarr; Activate Checkpoints ''(toolbar)''}}
 
Activating checkpoints is the DR promotion step. It does five things:
 
# Runs the {{Code|1=dr-prefailover}} custom script, if one is installed.
# '''Disables the schedule''', so nothing can overwrite the destination copy.
# '''Marks the selected checkpoints as Active Replica Checkpoints.''' Checkpoint shares are brought online and become accessible; checkpoint volumes become available to map to hosts. Leaving both selection lists empty activates everything in the schedule.
# Activates any existing aliases on the checkpoint shares, and -- with '''Auto-create Network Share Aliases''' ticked -- creates an alias named after the source share for each checkpoint that does not already have one, so clients can reach the DR copy under the original name. The option is off by default. Cloud containers backing auto-tiered checkpoint shares are mounted so tiered-out files stay readable.
# Rewrites the Samba configuration and the NFS exports file, then runs the {{Code|1=dr-postfailover}} custom script.
 
The two hook scripts live in {{Code|1=/var/opt/osnexus/custom/}} and are called with {{Code|1=--schedule=&lt;schedule-id&gt;}}. QuantaStor runs {{Code|1=dr-prefailover.sh}} and {{Code|1=dr-postfailover.sh}} synchronously with a five-minute timeout, and {{Code|1=dr-prefailover-async.sh}} and {{Code|1=dr-postfailover-async.sh}} in the background. They are the place to put site-specific promotion steps such as repointing DNS or restarting an application.
 
What activation does '''not''' do is change the checkpoint's name or turn it into the source of the replication relationship. The copy stays a checkpoint; the schedule stays pointed the same way. Failback is the separate step described below.
 
=== Automatic failover on VIF arrival ===
 
Set up on the schedule's '''Automatic Activation''' tab, this activates a schedule's checkpoints when a Site Cluster virtual interface moves to the node that owns the destination pool -- which is what happens when the primary site goes offline. Create the Site Cluster and its VIF under [[High-availability VIF Management]] first, spanning the source and destination systems, then select that VIF in the schedule.
 
When the VIF arrives, QuantaStor raises an alert and runs the same activation task as the dialog, always for '''every''' volume and share in the schedule and always with alias auto-creation on.
 
'''An automatic activation is skipped if the schedule is disabled''', and that case raises its own alert rather than failing quietly. If you disable a schedule for maintenance, remember that its automatic failover is disabled with it.
 
Node standby state interacts with this indirectly: [[Configure Member Standby|Standby Mode]] controls which nodes a cluster VIF and its pools are allowed to move to, so a destination node left in Standby Manual Activation will not receive the VIF and the checkpoints will not activate. There is no direct relationship between standby mode and a replication schedule.
 
=== Failback, option 1: discard the DR-site changes ===
 
This is the normal path after a DR test, where nothing at the DR site needs keeping.
 
{{Navigation|Remote Replication &rarr; Volume & Share Replication Schedules &rarr; Replication Schedule &rarr; Deactivate Checkpoints ''(toolbar)''}}
 
Deactivating clears the Active Replica Checkpoint flag on the selected checkpoints and takes checkpoint shares back offline. Tick '''Re-enable Replication Schedule''' to re-enable the schedule in the same operation, or use '''Enable''' afterwards. The next run overwrites the checkpoints from the source, discarding anything written at the DR site.
 
'''Disconnect the DR-site clients before you deactivate.''' A live iSCSI, FC, NFS or SMB session on a checkpoint re-marks it as an Active Replica Checkpoint by itself, so a schedule re-enabled with clients still attached is blocked again at its next activation.
 
=== Failback, option 2: send the DR-site changes back first ===
 
[[File:remoterep_rollback.png|thumb|right|800px|Rollback from Replication Schedule. The contents of the rollback target -- the original source -- are overwritten by the checkpoint.]]
 
'''Rollback''' reverses the direction for one transfer: it snapshots the changed checkpoint and sends the delta back to the original source volume or share. Use it when the DR site has taken real writes that must be preserved.
 
{{Navigation|Remote Replication &rarr; Volume & Share Replication Schedules &rarr; Replication Schedule &rarr; Rollback ''(toolbar)''}}
 
Select the volumes and shares to roll back, per schedule. A single association can be rolled back on its own from the Volume & Share Replica Associations section, which is right-click only:
 
{{Navigation|Remote Replication &rarr; Volume & Share Replica Associations &rarr; ''an association'' ''(select + right-click)'' &rarr; Rollback from Replica...}}
 
On the CLI the equivalents are <code>[[QuantaStor CLI Command Reference#replication-schedule-trigger-rollback|qs replication-schedule-trigger-rollback]] --schedule=&lt;name&gt;</code> and <code>[[QuantaStor CLI Command Reference#replica-assoc-rollback|qs replica-assoc-rollback]] --replica-assoc=&lt;name&gt;</code>; the per-association form requires {{Code|1=--flags=force}}.
 
'''The rollback target is overwritten.''' Anything written on the source side since the last successful replication is lost, because rollback makes the source an identical copy of the checkpoint. The dialog states this in its own warning; treat it as accurate.
 
When the rollback completes, the source has a GMT snapshot with the same timestamp as the checkpoint, which is the common point the next forward replication uses. Run '''Rescan Storage Pools''' on the source system afterwards so the new snapshots are picked up in its inventory -- the association object is owned by the source system, so some of the destination's updates are not visible there until a rescan.
 
Then follow option 1 -- deactivate the checkpoints and re-enable the schedule -- to resume forward replication.
 
== When a replication run fails ==
 
A run can fail part-way through a large transfer for the usual reasons: the WAN drops, the destination pool fills, a node reboots. What happens next depends on '''Enable resumable replication'''.
 
'''With resumable replication on''', the destination keeps a ZFS resume token for the partial stream, and the next run continues from where it stopped instead of resending everything. The association reports the status '''Sync Failed (Resumable)''' in the meantime.
 
'''QuantaStor validates the token before trusting it.''' If the snapshot the interrupted send started from no longer exists -- typically because retention rotated it away in the interim -- the token cannot be used. QuantaStor clears it, raises an alert reading {{Code|1=Target '<name>' has an invalid resume token. Restarting replication from a new snapshot.}}, and starts again from a new snapshot. Creating the touch file {{Code|1=/var/opt/osnexus/quantastor/touchfiles/tf_resume_token_cleanup.disable}} suppresses the automatic cleanup, in which case the alert instead asks you to contact support; do that only when support asks you to.
 
'''With resumable replication off''', a failed run is simply retried from the last common snapshot at the next activation, resending whatever the interrupted run had transferred.
 
Either way, a failed run raises an alert -- at most one per hour per schedule and source -- and records a failed entry in the schedule's replication report. See [[Alert Manager]] for routing those alerts.
 
Runs are also '''skipped''' rather than failed for conditions the scheduler detects up front, each with its own status string on the schedule:
 
{| class="wikitable"
! Status !! Meaning
|-
| {{Code|1=Schedule blocked due to active replication checkpoint(s) ...}} || A checkpoint is marked as an Active Replica Checkpoint. Deactivate it.
|-
| {{Code|1=Storage System Replication Link lookup failed.}} || No link exists between the systems that currently own the source and destination pools.
|-
| {{Code|1=Storage system link lookup found '<link>' but it is not healthy.}} || The link exists but is not in the Normal state -- see [[#Link state|Link state]].
|-
| {{Code|1=storage pool <name> is not in the 'NORMAL' healthy state}} || A source or destination pool is unhealthy -- for example an encrypted pool that has not been started.
|-
| {{Code|1=Failover in progress on the target pool}} || An HA failover of the destination pool is under way.
|-
| {{Code|1=Schedule is busy}} || The previous run has not finished.
|}
 
== Monitoring replication ==
 
=== Replica associations ===
 
A '''replica association''' is the persistent relationship between one source and one checkpoint, and it is where live replication state lives. Associations are listed in the '''Volume & Share Replica Associations''' section, grouped by schedule.
 
[[File:remoterep_assoc.png|thumb|right|800px|The Remote Replica Associations grid, grouped by replication schedule. Status, elapsed time and the sync timestamps are per source.]]
 
{{Navigation|Remote Replication &rarr; Volume & Share Replica Associations}}
 
The grid shows the source and its size, the source system, the status, elapsed time, the start and completion time of the last sync, and the target, its size and its system. On the CLI the same information comes from <code>[[QuantaStor CLI Command Reference#replica-assoc-list|qs replica-assoc-list]]</code> and <code>[[QuantaStor CLI Command Reference#replica-assoc-get|qs replica-assoc-get]] --replica-assoc=&lt;name&gt;</code>. Association names read like a path, which makes the direction unambiguous:
 
<pre style="font-size: smaller">
qs-node-110://doctest-repl-vol --> qs-node-111://doctest-repl-vol_chkpnt
</pre>
 
The Status column uses this vocabulary:
 
{| class="wikitable"
! Status !! Meaning
|-
| Preparing, Initializing || The run is being set up; the transfer has not started moving data yet
|-
| Synchronizing, Resynchronizing || Data is moving
|-
| Synchronized || The last run completed and the checkpoint matches the source snapshot
|-
| Stopped/Paused || Replication was stopped
|-
| Init Failed, Sync Failed, Resync Failed || The run failed; see the alert and the report entry for the reason
|-
| Sync Failed (Resumable) || The run failed and left a resume token, so the next run continues from that point
|-
| Partial Sync || Some sources in the run succeeded and others did not
|-
| Skipped || The scheduler declined to start the run; the schedule's status says why
|-
| User Cancelled || The task was stopped from the Tasks pane
|}
 
Selecting a link in the '''Storage System Replication Links''' section gives two further centre-pane tabs, '''Replication Sources''' and '''Replication Targets''', each named for the system it describes. They split the same association data by direction, which is the quickest way to see what a given system is sending versus receiving.
 
=== Replication reports ===
 
Each run of a schedule produces a '''summary report''' with one '''report entry''' per volume or share. Select a schedule and the '''Replication Report''' tab in the centre pane shows a dashboard: a column chart that can be switched between '''Tasks Summary''' and '''Total Amount Transferred''', over the retained history. The button beside them generates an HTML report, which can be opened in a new browser tab, shown in a dialog, or downloaded.
 
On the CLI:
 
<pre style="font-size: smaller">
qs replica-report-summary-list
qs replica-report-entry-list
</pre>
 
The summary carries the run's status, task count, start and end time, the number of completed and failed transfers, total bytes transferred and average speed. Each entry carries the same for one source, plus the estimated transfer size, so a long run can be compared against its estimate while it is in progress.
 
<pre style="font-size: smaller">
Name                                  Status        Task Count  Completed  Failed  Total transferred  Avg speed/sec
doctest-repl-sched_GMT20260903_045937  Synchronized  2          2          0      71KiB              35KiB
</pre>
 
How many summaries are kept is the '''Report Retention Count''' on the Schedule Interval tab. Delete one with <code>[[QuantaStor CLI Command Reference#replica-report-summary-delete|qs replica-report-summary-delete]] --report-summary=&lt;name&gt;</code>, which also deletes its entries.
 
=== Tasks and alerts ===
 
Every replication run appears in the Tasks pane as an ''Initiating Remote-Replication'' task on the source system, with a per-transfer description and progress, followed by an ''Importing Replica(s)'' task on the destination as it takes the new checkpoint and snapshots into inventory. Replication failures, invalid resume tokens and blocked automatic activations all raise alerts; see [[Call-home / Alerting]] for delivering them off the appliance.
 
== Related pages ==
 
* [[Storage Volumes]] -- the SAN objects that get replicated
* [[Network Shares]] -- the NAS objects that get replicated, and share aliases
* [[Storage Pools]] -- source and destination pools
* [[Grid Configuration]] -- joining systems into the grid that replication requires
* [[Snapshot Schedules]] -- local point-in-time recovery points
* [[Backup Policies]] -- pulling data in from non-QuantaStor sources
* [[High-availability VIF Management]] -- Site Clusters and the VIFs that drive automatic activation
* [[Configure Member Standby]] -- per-node standby state, which governs where a cluster VIF may move
* [[QuantaStor CLI Command Reference]] -- full argument lists for the commands above
 
----
<small>''Verified against QuantaStor 6.9.0.''</small>

Revision as of 05:35, 3 September 2026


Remote replication copies Storage Volumes and Network Shares from a Storage Pool on one QuantaStor system to a Storage Pool on another, asynchronously and block-level incrementally, so that a second site holds a usable copy you can bring online if the first site is lost. This page covers the trust relationship the two systems need, the replication schedules that drive the copying, the replica checkpoints that appear at the destination, and how failover and failback work.

Everything on this page lives under the Remote Replication tab.

Section Purpose
Choosing between replication, snapshots and backups Which of the three features solves your problem
Requirements What must be in place before you start
Storage System Replication Links The trust relationship and data path between two systems
Replication schedules The dialog that drives repeating replication, tab by tab
Replica checkpoints What is created on the destination, and delta versus full sync
One-time replicas Replicating once without a schedule
DR failover and failback Activating checkpoints, and getting back to normal
When a replication run fails Resumable replication, blocked schedules, alerts
Monitoring replication Where state, history and throughput are visible

Choosing between replication, snapshots and backups

QuantaStor has three scheduled data-protection features and they solve different problems. Pick by where the copy has to land and what form it has to take.

Feature Where the copy lands What the copy is Use it for
Snapshot Schedules The same Storage Pool as the source A point-in-time snapshot of the volume or share Fast local recovery points; undoing a bad change; a source for off-host backup
Remote replication (this page) A Storage Pool on another system in the grid A full, mountable copy of the volume or share, plus its own snapshots Disaster recovery; surviving the loss of a system, pool or site
Backup Policies A Network Share on a QuantaStor system, from an external NFS/SMB source Files copied in by an rsync-based backup job Protecting data that lives outside QuantaStor

Two consequences are worth spelling out.

  • A replication schedule already makes snapshots, so you rarely want a Snapshot Schedule on the same volume or share as well. Every replication run snapshots the source, and the schedule's own retention settings decide how many source-side and destination-side snapshots are kept. Adding a Snapshot Schedule on top duplicates that work.
  • Replication is not a backup of a file server that is not QuantaStor. If the data you want to protect is on a third-party NAS, that is what Backup Policies are for.

Requirements

  • Two QuantaStor systems that are members of the same storage grid. A replication link can only be created between grid members; QuantaStor rejects anything else with Specified remote storage system is not a member of the grid. Be sure to add the appliance as a grid member first. See Grid Configuration for joining systems into a grid. A grid can span sites, so "replicate to a remote site" means "join the remote appliances to this grid", not "link to a separate grid" -- there is no separate remote-grid replication setup in the interface.
  • A Storage Pool on each system. The pools do not need to match in size, layout, disk type or hardware.
  • A license with the remote-replication feature. Without it, creating a link fails with Remote replication is not licensed for this storage system. Trial and cloud-edition licenses always include it.
  • A destination pool of a compatible type. The Remote Pool list in the schedule dialog only offers pools on the remote system, and it never offers XFS, CephFS or Ceph object pools. Ceph RBD pools are offered only when the remote system belongs to the same Ceph cluster as the source. Network Shares in scale-out CephFS pools cannot be replicated; use Backup Policies for those.
  • TCP reachability between the two addresses you choose for the link. Replication data moves over SSH (or, on an unencrypted link, over an mbuffer TCP session) between the two link addresses.

Setup at a glance

A completed configuration. The two systems are joined by a Storage System Replication Link pair; the schedule on the destination system drives the transfer; the destination holds a checkpoint for each replicated volume and share.
  1. Join both systems to the same storage grid, if they are not already. See Grid Configuration.
  2. Create a Storage Pool on the destination system to hold the replicas.
  3. Create a Storage System Replication Link between the two systems. This exchanges keys and fixes the addresses replication traffic uses. There is no limit on how many links a system may have.
  4. Create a Replication Schedule naming that link, the destination pool, the volumes and shares to replicate, and when to run.
  5. Interval schedules start on their own shortly afterwards. To test a calendar schedule immediately, trigger it by hand.

Each step is described in the sections below.

Storage System Replication Links

Before anything can be replicated, the two systems need a Storage System Replication Link. Creating one generates an SSH key pair dedicated to that pair of systems, registers each system's public key on the other, and records which IP address on each side carries replication traffic. Links are created in pairs: making a link from A to B also establishes the reverse link on B, and deleting either one deletes both.

The link is what QuantaStor uses to move data. It is not what records replication progress -- that is the replica association (see Monitoring replication) -- so links can be deleted and recreated without disturbing existing replication schedules.

Creating a link

The Create Storage System Replication Link dialog. Both ends must already be members of the same storage grid.
Navigation: Remote Replication → Storage System Replication Links → Replication Link → Create (toolbar)

Link System (A) and Link System (B) each take a Storage System from the grid and an address on it:

  • Storage System -- the grid member. Selecting a system in (A) auto-selects a different system in (B), so the dialog opens ready to create a system-to-system link.
  • IP Address -- a drop-down of the addresses actually configured on that system's network ports. Floating (cluster VIF) addresses and ports without a valid IP address are filtered out, so a link is always pinned to a fixed address rather than one that can move.
  • External IP Address -- a free-text alternative, enabled by selecting its radio button. It is pre-filled from the system's external hostname when one is set (qs system-modify --ext-hostname), which is how cloud deployments record a public address.

The address you give for Link System (A) must be one that is configured on a port of that system. QuantaStor validates it and fails with Invalid storage system link specification, IP address '<addr>' is not configured on any network port for system '<name>'. otherwise. In practice this means the External IP Address field is only usable for the far side of the link.

Below the two fieldsets:

Selecting the same Storage System for both (A) and (B) creates a local link, used to replicate between two pools on one appliance. Encryption and Compression grey out in that case, because there is no network hop to protect or compress.

What the link does with encryption and compression

The two checkboxes select the transport, not just a flag:

  • Encryption on (the default) tunnels the ZFS send stream over SSH to root on the remote link address, using the key pair created with the link (kept in /var/opt/osnexus/quantastor/replication-keys/). The appliance's shipped SSH client configuration lists aes256-gcm@openssh.com first, so AES-256 is what two QuantaStor systems negotiate. (Note: due to the frequency of security updates this may be out of date and /etc/ssh/ssh_config on the appliance should be referenced.)
  • Encryption off sends the stream over a plain TCP session between mbuffer processes instead. It is faster on a trusted, private link and offers no confidentiality.
  • Compression on inserts lz4 compression into the send pipeline -- but only when the source dataset is not already compressed. Replicating a compressed pool gains nothing from the option, so QuantaStor skips it.

Before each run, QuantaStor pre-checks that the key file exists and that it can reach the remote appliance over SSH. A failed pre-check reports Replication pre-check failed to verify SSH tunnel access to remote appliance, which points at firewall rules or a wrong link address rather than at the data itself.

Bandwidth throttling

The Bandwidth Limit on the link caps total replication throughput for everything using that link, in MB/sec. QuantaStor enforces it by rate-limiting each replication process and re-dividing the link's limit evenly across the streams that are currently running: four concurrent replications on a 200 MB/sec link get 50 MB/sec each, and when three of them finish the survivor is raised back to 200 MB/sec. Rebalancing happens whenever the set of running replications changes, so you do not have to plan around how many volumes are in a schedule.

Change it at any time with Modify on the link; the new value is applied to running replications at the next rebalance.

Navigation: Remote Replication → Storage System Replication Links → Replication Link → Modify (toolbar)

The Modify dialog is the same form as Create with the link selected, and it shows the link's current limit rather than the 200 MB/sec default.

Link state

A link's State column tracks the state of the remote grid member, re-checked about once a minute:

  • Normal -- the remote system is a healthy grid member.
  • Offline -- the remote system is a grid member but is not in the Normal state. The replica associations using the link are marked offline with it, and schedules using it are skipped.
  • Warning -- the grid holds no information about the remote system ID at all, so the link points nowhere.

A link returns to Normal on its own when the remote system does. There is nothing to reset by hand.

Deleting a link

Navigation: Remote Replication → Storage System Replication Links → Replication Link → Delete (toolbar)

Deleting a link disconnects it on the remote system, removes the key pair for that system pair, and deletes the reverse-direction link. Existing replica associations survive. If the addresses on a pair of systems change, delete the links and recreate them -- schedules pick up the new link automatically, as described next.

Links in high-availability configurations

Every replication run re-selects its link. At each activation the scheduler looks through the local system's links for one whose source and remote system IDs match the systems that currently own the source and destination pools, and repairs the schedule's stored link reference if it has changed. That is what lets replication survive an HA pool failover on either end -- but only if a link exists for the pair of systems the pools have moved to.

So with an HA pool on systems A and B replicating to an HA pool on systems C and D, create all four links: A to C, A to D, B to C, B to D. Any missing combination shows up as a skipped run with the status Storage System Replication Link lookup failed. A run is also skipped when the matching link exists but is not in the Normal state.

Replication is likewise skipped while an HA failover of the destination pool is in progress, with the status Failover in progress on the target pool.

Replication schedules

A replication schedule is the normal way to run remote replication. It names a link, a destination pool, a set of volumes and shares, when to run, and how many snapshots to keep on each side.

The Remote Replication tab with the Volume & Share Replication Schedules section selected. The centre pane shows the schedule's report chart, the schedule grid, and the volumes and shares it replicates.

The schedule object is owned and run by the system that owns the destination pool, not by the source system. This is visible in the Storage System column of the task log and in qs replication-schedule-get, and it is why the destination system must be up for a schedule to be created, modified or triggered.

Navigation: Remote Replication → Volume & Share Replication Schedules → Replication Schedule → Create (toolbar)

The Create Replication Schedule dialog is a six-tab wizard. Previous and Next walk the tabs; OK is available from any tab.

General tab

General tab. Replica Destination is the replication link, which fixes the direction of replication.
  • Policy Name -- pre-filled with a generated name such as replication-sched-1. Names may contain letters, digits and -_. only.
  • Description -- free text.
  • Replica Destination -- the Storage System Replication Link to use. This choice sets the direction of replication: the link A --> B replicates from A to B. Because links exist in both directions, both appear in the list. Selecting a link that is offline clears the Remote Pool list.
  • Remote Pool -- the destination Storage Pool, labelled with its system: pool-name [system-name]. Only pools on the link's remote system appear, subject to the type rules in Requirements.
  • Start date and Enabled -- when the schedule becomes eligible to run, and whether it is active. Enabled is checked by default.

If no replication link exists at all, the dialog closes immediately with There are no remote storage systems to replicate your volumes. Add a link to a remote storage system first.

Schedule Interval tab

Schedule Interval tab. The day/hour grid and the timer interval are mutually exclusive; the unselected one greys out.

Schedule Type picks between two mutually exclusive modes, and the group you did not pick is disabled:

  • Use day/hour selections (default) -- a calendar schedule. Tick the days of the week and the hours of the day at which replication should start. The default selection is Monday to Friday at 12 AM, 4 AM, 8 AM, 12 PM, 4 PM and 8 PM. All Days and All Hours toggle a whole row of checkboxes. Offset delays every trigger by 0-59 minutes, so a 30-minute offset on a 1 AM and 4 AM schedule fires at 1:30 AM and 4:30 AM -- useful for staggering several schedules that would otherwise all start on the hour.
  • Use timer interval -- a rest interval, in minutes, between the end of one run and the start of the next. The minimum is 3 minutes and the slider runs to 360; the default is 30. Entering less than 3 resets the field and raises a warning.

The distinction matters: an interval schedule measures the gap after a run completes, so a low interval is safe -- it cannot pile runs on top of each other. QuantaStor never starts a run for a schedule whose previous run is still working; a calendar trigger that arrives while the schedule is busy is skipped rather than queued.

Because only changed blocks are sent, replicating often through the day generally transfers less at each run and keeps the destination closer to the source than one large nightly run.

Report Retention Count at the bottom of the tab sets how many replication summary reports to keep for this schedule. The default is 30; a value of 0 is replaced with 30 by the service.

Select Volumes & Shares tab

Select Volumes & Shares tab. The Network Shares and Storage Volumes sub-tabs each have their own selection list; a schedule can carry both.

Two sub-tabs, Network Shares and Storage Volumes, each with a dual list: available on the left, selected on the right. Move items with the arrow buttons or by dragging. A Storage Pool filter, a text search and Hide Nested Shares narrow the left-hand list.

At least one volume or share must be selected, from either sub-tab; a schedule can replicate volumes and shares together. Selecting both a parent share and a share nested inside it is rejected -- replicate the parent (with Recursively replicate nested shares on the Advanced Settings tab) or the children, not both.

Changing the selection later is a separate dialog, Update Selections, rather than part of Modify:

Navigation: Remote Replication → Volume & Share Replication Schedules → Replication Schedule → Update Selections (toolbar)

Active replication tasks finish before the new selection takes effect at the next activation.

Snapshot Settings tab

Snapshot Settings tab. The Source column controls snapshots kept on the source volume or share; the Checkpoint column controls snapshots kept on the destination checkpoint.

Replication is driven by snapshots, and this tab decides how many are kept on each side. Two independent groups:

Short Term Snapshot Retention Settings -- Max Short Term Snapshots is the number of rotating snapshots used to compute deltas. Each run snapshots the source, sends the difference against the newest snapshot the destination also has, and the oldest short-term snapshot is removed once the count is exceeded. The default is 3, which is also the recommended value; the maximum is 1024. The service raises anything below 2 to 2.

Long Term Snapshot Retention Settings -- additional recovery points promoted out of the short-term snapshots and kept for longer, with an independent count for the source and for the destination checkpoint:

Rule What a count of n retains Source default Checkpoint default
Hourly n snapshots spanning the last n hours 2 4
Daily n snapshots spanning the last n days, at least a day apart 2 4
Weekly n snapshots at least 7 days apart 2 4
Monthly n snapshots at least 30 days apart 0 2
Quarterly n snapshots at least 90 days apart 0 2

Suggested Defaults restores the values in the table; Clear sets every count to 0. Each field caps at 1000.

The two read-only lines at the bottom, Max Total Source Snapshots and Max Total Checkpoint Snapshots, add the long-term counts to Max Short Term Snapshots and update as you type. They are the number to watch: this is how many snapshots the schedule can accumulate per volume or share on each side.

Automatic Activation tab

Enable Automatic Checkpoint Activation ties DR failover to a Site Cluster virtual interface. Activate on VIF arrival selects the VIF, and is disabled until the checkbox is ticked. Both are off and empty by default. See Automatic failover on VIF arrival for what happens when it fires.

Advanced Settings tab

Advanced Settings tab. Enable target checkpoint recovery and Enable resumable replication are on by default; the latter depends on the destination pool.

Seven checkboxes. Two are on by default and five are off.

  • Enable source snapshot reuse (use with N-way and cascading A->B->C replication) -- off. Replicates from the most recent existing source snapshot instead of taking a new one. Turn it on for the second and subsequent schedules when one source replicates to several destinations, or in a cascade, so that all destinations end up with a common replica checkpoint to work from. If the schedule finds no suitable existing snapshot it fails rather than silently taking a new one.
  • Enable target checkpoint recovery (unchecking will perform full replication) -- on. Keeps replication incremental by finding and reusing the existing _chkpnt at the destination even when the replica association between source and checkpoint has been deleted. Unchecking it asks for a full replication, and the dialog raises a confirmation prompt before accepting that.
  • Recursively replicate nested shares -- off. Includes shares nested under a selected share. It has no effect on Storage Volumes, and it disables resumable replication.
  • Enable resumable replication -- on when the selected Remote Pool supports it, and disabled and cleared otherwise. Lets an interrupted run pick up where it stopped rather than resending from the beginning; see When a replication run fails. It is greyed out while Recursively replicate nested shares is on, and greyed out until a Remote Pool is selected.
  • Copy source share settings to the checkpoint network share -- off. On a full replication of a Network Share, copies the source share's settings onto the checkpoint share, so the DR copy is exported the way the original was. Shares only; ignored for volumes, and not applied on incremental runs.
  • Reclaim Orphaned Snapshots -- off. Adopts replication snapshots left behind by a schedule that has since been deleted, so the new schedule counts and rotates them instead of leaving them to accumulate. In the Modify dialog this checkbox enables a Reclaim Schedule with ID field, where you paste the old schedule's ID -- it appears in the schedule column of the property view for a share or volume snapshot.
  • Force -- off. Cascading replication is meant to be run with source snapshot reuse; Force overrides that and creates new snapshots on the checkpoints instead.

N-way and cascading replication

A source can be replicated to more than one destination by creating more than one schedule for it. Two shapes are common:

  • N-way -- a share datastore1 replicates to pool-2 on one system and to pool-3 on another. Create two schedules, each with the same source and a different link and destination pool. Many-to-one works the same way: several schedules on different sources can share a destination pool.
  • Cascading (A to B to C) -- datastore1 replicates from A to B, and the checkpoint on B is itself the source of a second schedule from B to C. Cascading keeps the extra transfer off the primary system: A sends once, and B fans it out. It can be chained further.

On the second and subsequent schedules of an N-way or cascading configuration, tick Enable source snapshot reuse on the Advanced Settings tab. Without it each schedule takes its own new source snapshot, and the destinations end up with no snapshot in common -- which is what future incremental replication between them depends on. With it, the second schedule replicates from the snapshot the first one already made.

QuantaStor recognises a cascade by the _chkpnt in the source name and does not double the suffix, so the second hop's checkpoint keeps the same name rather than becoming datastore1_chkpnt_chkpnt.

Modifying, enabling and disabling a schedule

Navigation: Remote Replication → Volume & Share Replication Schedules → Replication Schedule → Modify (toolbar)

Modify offers five tabs -- the same as Create without Select Volumes & Shares, which is handled by Update Selections -- and its Advanced Settings tab adds the Reclaim Schedule with ID field described above. The link and destination pool of an existing schedule cannot be changed; create a new schedule to replicate somewhere else. Changed settings take effect at the next activation.

Enable and Disable turn a schedule on and off without deleting it. A disabled schedule shows a state of Offline. Delete removes the schedule and leaves the snapshots and checkpoints it created in place.

Running a schedule now

An interval schedule starts shortly after it is created. To run any schedule immediately, use qs replication-schedule-trigger --schedule=<name>. The command takes one-off overrides of three of the Advanced Settings options -- --reuse-source-snapshot, --reuse-target-chkpnt and --copy-share-settings -- without changing the schedule itself:

qs replication-schedule-trigger --schedule=nightly-dr
qs replication-schedule-trigger --schedule=nightly-dr --reuse-target-chkpnt=true

Recovering a repaired association is the common reason to pass --reuse-target-chkpnt=true by hand: it reconnects a source to the best-matching checkpoint on the destination pool instead of starting a fresh full copy.

Replica checkpoints

What appears on the destination

For each replicated source, QuantaStor creates one object on the destination pool named after the source with a _chkpnt suffix -- backups becomes backups_chkpnt. If that name is already taken it appends .1, .2 and so on, though current releases prefer to reuse the existing _chkpnt for a full resync rather than create a numbered one.

The checkpoint is a real, mountable volume or share, not an archive format. Alongside it sit the replicated snapshots, timestamped in GMT:

source volume       doctest-repl-vol
source snapshot     doctest-repl-vol_GMT20260903_045938
checkpoint volume   doctest-repl-vol_chkpnt
checkpoint snapshot doctest-repl-vol_chkpnt_GMT20260903_045938

source share        doctest-repl-share
checkpoint share    doctest-repl-share_chkpnt
checkpoint snapshot doctest-repl-share_chkpnt@GMT-2026.09.03-04.59.38

The suffix-free checkpoint always holds the state of the most recently completed run. The timestamped snapshots beside it are the retained recovery points, governed by the Snapshot Settings tab.

A share checkpoint is not exported and not active until you activate it. QuantaStor deliberately leaves _chkpnt shares offline during pool discovery so that a DR copy is not accidentally serving clients.

Because the checkpoint share carries the _chkpnt suffix, users looking for backups at the DR site will not find it. Give the checkpoint a Network Share alias named after the source and it is reachable under both names; Activate Checkpoints can create those aliases for you. See Network Shares for aliases in general.

Navigation: Storage Management → Network Shares → a checkpoint share (select + right-click) → Create Alias/Sub-share...

Once the alias exists the data is reachable as both backups and backups_chkpnt. If users find the two names confusing, clear browseable on the _chkpnt share in Modify Network Share so only the alias is advertised over SMB.

Delta sync versus full sync

Only the first run of a source is a full copy. Every run after that sends only changed blocks.

The choice is made per run, and it is mechanical: QuantaStor looks for the newest GMT-stamped snapshot that exists on both the source and the destination checkpoint and sends the difference from there. If no such common snapshot exists -- the first run, a checkpoint with no snapshots yet, or a source whose only snapshot is the one just taken -- the run sends everything.

This is why the Snapshot Settings retention counts matter operationally. Retaining too few snapshots on either side can leave no common point between two runs, and the next run falls back to a full copy.

A full copy is forced deliberately in two places: clearing Enable target checkpoint recovery on a schedule, and choosing Create a new replica on the target storage system. (full copy) in the one-time replica dialog.

The Active Replica Checkpoint state

A checkpoint carries a flag, Active Replica Checkpoint, which means "somebody may have written to this copy". It exists to stop replication from overwriting DR-site changes.

It is set three ways:

  1. Explicitly, by the Activate Checkpoints dialog or qs replication-schedule-activate-checkpoints.
  2. Automatically on client access. An iSCSI or Fibre Channel login to a checkpoint volume, or an NFS/SMB client on an active checkpoint share, marks it active on its own. This is a safety net: it means data written to a DR copy that someone mounted without going through the failover dialog is still protected.
  3. By hand, with the Active Replica Checkpoint checkbox in the Modify Storage Volume and Modify Network Share dialogs. The checkbox only applies to objects that are checkpoints.

While any checkpoint in a schedule is active, the schedule will not run. The scheduler checks before every activation, and a schedule that is blocked moves to a Warning state with a status such as Schedule blocked due to active replication checkpoint(s) 1 shares (including backups_chkpnt), and records a Skipped entry in its replication report. Every checkpoint in the schedule has to be deactivated before replication resumes.

For volumes the flag is stored on the dataset itself, as the ZFS user property quantastor:isactivecheckpoint, so it survives a pool export and import.

One-time replicas

Create Remote Volume Replica. Replica Name and Description are disabled unless the full-copy option is selected.

Replicating once, without creating a schedule, is useful for seeding a destination or for moving a copy of a volume to another system on demand.

Navigation: Remote Replication → Storage System Replication Links → Remote Replication → Create Volume Replica (toolbar)
Navigation: Remote Replication → Storage System Replication Links → Remote Replication → Create Share Replica (toolbar)
  • Storage Volume / Network Share -- the source.
  • System Link -- the replication link, which again fixes the direction.
  • Remote Storage Pool -- the destination pool on the link's remote system.
  • Replicate over a previous replica target if available (diff copy). -- the default. Sends only the changes against an existing checkpoint.
  • Create a new replica on the target storage system. (full copy) -- sends everything to a new checkpoint.
  • Replica Name and Description under Optional Settings -- both are disabled while the diff-copy option is selected and are enabled by selecting full copy, which is the case where a new destination object is named.

The Stop Volume Replication and Resume Volume Replication toolbar buttons act on a replica association rather than on a source, and correspond to qs replica-assoc-stop and qs replica-assoc-sync. There is no CLI equivalent of the one-time replica dialogs themselves.

DR failover and failback

Manual failover: activating checkpoints

Activate Checkpoints. Selecting nothing activates every volume and share in the schedule.
Navigation: Remote Replication → Volume & Share Replication Schedules → Replication Schedule → Activate Checkpoints (toolbar)

Activating checkpoints is the DR promotion step. It does five things:

  1. Runs the dr-prefailover custom script, if one is installed.
  2. Disables the schedule, so nothing can overwrite the destination copy.
  3. Marks the selected checkpoints as Active Replica Checkpoints. Checkpoint shares are brought online and become accessible; checkpoint volumes become available to map to hosts. Leaving both selection lists empty activates everything in the schedule.
  4. Activates any existing aliases on the checkpoint shares, and -- with Auto-create Network Share Aliases ticked -- creates an alias named after the source share for each checkpoint that does not already have one, so clients can reach the DR copy under the original name. The option is off by default. Cloud containers backing auto-tiered checkpoint shares are mounted so tiered-out files stay readable.
  5. Rewrites the Samba configuration and the NFS exports file, then runs the dr-postfailover custom script.

The two hook scripts live in /var/opt/osnexus/custom/ and are called with --schedule=<schedule-id>. QuantaStor runs dr-prefailover.sh and dr-postfailover.sh synchronously with a five-minute timeout, and dr-prefailover-async.sh and dr-postfailover-async.sh in the background. They are the place to put site-specific promotion steps such as repointing DNS or restarting an application.

What activation does not do is change the checkpoint's name or turn it into the source of the replication relationship. The copy stays a checkpoint; the schedule stays pointed the same way. Failback is the separate step described below.

Automatic failover on VIF arrival

Set up on the schedule's Automatic Activation tab, this activates a schedule's checkpoints when a Site Cluster virtual interface moves to the node that owns the destination pool -- which is what happens when the primary site goes offline. Create the Site Cluster and its VIF under High-availability VIF Management first, spanning the source and destination systems, then select that VIF in the schedule.

When the VIF arrives, QuantaStor raises an alert and runs the same activation task as the dialog, always for every volume and share in the schedule and always with alias auto-creation on.

An automatic activation is skipped if the schedule is disabled, and that case raises its own alert rather than failing quietly. If you disable a schedule for maintenance, remember that its automatic failover is disabled with it.

Node standby state interacts with this indirectly: Standby Mode controls which nodes a cluster VIF and its pools are allowed to move to, so a destination node left in Standby Manual Activation will not receive the VIF and the checkpoints will not activate. There is no direct relationship between standby mode and a replication schedule.

Failback, option 1: discard the DR-site changes

This is the normal path after a DR test, where nothing at the DR site needs keeping.

Navigation: Remote Replication → Volume & Share Replication Schedules → Replication Schedule → Deactivate Checkpoints (toolbar)

Deactivating clears the Active Replica Checkpoint flag on the selected checkpoints and takes checkpoint shares back offline. Tick Re-enable Replication Schedule to re-enable the schedule in the same operation, or use Enable afterwards. The next run overwrites the checkpoints from the source, discarding anything written at the DR site.

Disconnect the DR-site clients before you deactivate. A live iSCSI, FC, NFS or SMB session on a checkpoint re-marks it as an Active Replica Checkpoint by itself, so a schedule re-enabled with clients still attached is blocked again at its next activation.

Failback, option 2: send the DR-site changes back first

Rollback from Replication Schedule. The contents of the rollback target -- the original source -- are overwritten by the checkpoint.

Rollback reverses the direction for one transfer: it snapshots the changed checkpoint and sends the delta back to the original source volume or share. Use it when the DR site has taken real writes that must be preserved.

Navigation: Remote Replication → Volume & Share Replication Schedules → Replication Schedule → Rollback (toolbar)

Select the volumes and shares to roll back, per schedule. A single association can be rolled back on its own from the Volume & Share Replica Associations section, which is right-click only:

Navigation: Remote Replication → Volume & Share Replica Associations → an association (select + right-click) → Rollback from Replica...

On the CLI the equivalents are qs replication-schedule-trigger-rollback --schedule=<name> and qs replica-assoc-rollback --replica-assoc=<name>; the per-association form requires --flags=force.

The rollback target is overwritten. Anything written on the source side since the last successful replication is lost, because rollback makes the source an identical copy of the checkpoint. The dialog states this in its own warning; treat it as accurate.

When the rollback completes, the source has a GMT snapshot with the same timestamp as the checkpoint, which is the common point the next forward replication uses. Run Rescan Storage Pools on the source system afterwards so the new snapshots are picked up in its inventory -- the association object is owned by the source system, so some of the destination's updates are not visible there until a rescan.

Then follow option 1 -- deactivate the checkpoints and re-enable the schedule -- to resume forward replication.

When a replication run fails

A run can fail part-way through a large transfer for the usual reasons: the WAN drops, the destination pool fills, a node reboots. What happens next depends on Enable resumable replication.

With resumable replication on, the destination keeps a ZFS resume token for the partial stream, and the next run continues from where it stopped instead of resending everything. The association reports the status Sync Failed (Resumable) in the meantime.

QuantaStor validates the token before trusting it. If the snapshot the interrupted send started from no longer exists -- typically because retention rotated it away in the interim -- the token cannot be used. QuantaStor clears it, raises an alert reading Target '<name>' has an invalid resume token. Restarting replication from a new snapshot., and starts again from a new snapshot. Creating the touch file /var/opt/osnexus/quantastor/touchfiles/tf_resume_token_cleanup.disable suppresses the automatic cleanup, in which case the alert instead asks you to contact support; do that only when support asks you to.

With resumable replication off, a failed run is simply retried from the last common snapshot at the next activation, resending whatever the interrupted run had transferred.

Either way, a failed run raises an alert -- at most one per hour per schedule and source -- and records a failed entry in the schedule's replication report. See Alert Manager for routing those alerts.

Runs are also skipped rather than failed for conditions the scheduler detects up front, each with its own status string on the schedule:

Status Meaning
Schedule blocked due to active replication checkpoint(s) ... A checkpoint is marked as an Active Replica Checkpoint. Deactivate it.
Storage System Replication Link lookup failed. No link exists between the systems that currently own the source and destination pools.
Storage system link lookup found '<link>' but it is not healthy. The link exists but is not in the Normal state -- see Link state.
storage pool <name> is not in the 'NORMAL' healthy state A source or destination pool is unhealthy -- for example an encrypted pool that has not been started.
Failover in progress on the target pool An HA failover of the destination pool is under way.
Schedule is busy The previous run has not finished.

Monitoring replication

Replica associations

A replica association is the persistent relationship between one source and one checkpoint, and it is where live replication state lives. Associations are listed in the Volume & Share Replica Associations section, grouped by schedule.

The Remote Replica Associations grid, grouped by replication schedule. Status, elapsed time and the sync timestamps are per source.
Navigation: Remote Replication → Volume & Share Replica Associations

The grid shows the source and its size, the source system, the status, elapsed time, the start and completion time of the last sync, and the target, its size and its system. On the CLI the same information comes from qs replica-assoc-list and qs replica-assoc-get --replica-assoc=<name>. Association names read like a path, which makes the direction unambiguous:

qs-node-110://doctest-repl-vol --> qs-node-111://doctest-repl-vol_chkpnt

The Status column uses this vocabulary:

Status Meaning
Preparing, Initializing The run is being set up; the transfer has not started moving data yet
Synchronizing, Resynchronizing Data is moving
Synchronized The last run completed and the checkpoint matches the source snapshot
Stopped/Paused Replication was stopped
Init Failed, Sync Failed, Resync Failed The run failed; see the alert and the report entry for the reason
Sync Failed (Resumable) The run failed and left a resume token, so the next run continues from that point
Partial Sync Some sources in the run succeeded and others did not
Skipped The scheduler declined to start the run; the schedule's status says why
User Cancelled The task was stopped from the Tasks pane

Selecting a link in the Storage System Replication Links section gives two further centre-pane tabs, Replication Sources and Replication Targets, each named for the system it describes. They split the same association data by direction, which is the quickest way to see what a given system is sending versus receiving.

Replication reports

Each run of a schedule produces a summary report with one report entry per volume or share. Select a schedule and the Replication Report tab in the centre pane shows a dashboard: a column chart that can be switched between Tasks Summary and Total Amount Transferred, over the retained history. The button beside them generates an HTML report, which can be opened in a new browser tab, shown in a dialog, or downloaded.

On the CLI:

qs replica-report-summary-list
qs replica-report-entry-list

The summary carries the run's status, task count, start and end time, the number of completed and failed transfers, total bytes transferred and average speed. Each entry carries the same for one source, plus the estimated transfer size, so a long run can be compared against its estimate while it is in progress.

Name                                   Status        Task Count  Completed  Failed  Total transferred  Avg speed/sec
doctest-repl-sched_GMT20260903_045937  Synchronized  2           2          0       71KiB              35KiB

How many summaries are kept is the Report Retention Count on the Schedule Interval tab. Delete one with qs replica-report-summary-delete --report-summary=<name>, which also deletes its entries.

Tasks and alerts

Every replication run appears in the Tasks pane as an Initiating Remote-Replication task on the source system, with a per-transfer description and progress, followed by an Importing Replica(s) task on the destination as it takes the new checkpoint and snapshots into inventory. Replication failures, invalid resume tokens and blocked automatic activations all raise alerts; see Call-home / Alerting for delivering them off the appliance.

Related pages


Verified against QuantaStor 6.9.0.