Backup Policies
A Backup Policy copies or moves file data between a QuantaStor Network Share and a remote data source, on a schedule. The remote source can be an SMB or NFS export on another server or NAS filer, or a Cloud Container backed by S3-compatible object storage. Policies transfer data in either direction, so the same mechanism covers pulling a filer onto QuantaStor, pushing a share out to the cloud, and tiering cold files off a share to object storage.
Scanning and copying are parallelized, so filers with 100 million or more files can be scanned for changes in a practical amount of time. Each run creates a Backup Job object you can monitor while it works.
| Section | Covers |
|---|---|
| Backup Modes and Transfer Direction | Copy, move and auto-tier; inbound versus outbound transfers |
| Creating a Backup Policy | Every tab of the Create dialog, field by field |
| Modifying a Backup Policy | What can and cannot be changed after creation |
| Running a Policy on Demand | Triggering a backup outside its schedule |
| Enabling and Disabling a Policy | Suspending a policy without deleting it |
| Deleting a Backup Policy | Removing a policy, and what happens to the data |
| Monitoring Backup Jobs | The Backup Jobs tab and the backup logs |
| Managing Backup Policies from the CLI | The equivalent qs commands
|
The Backup Policy toolbar group holds all six operations -- Create, Modify, Delete, Run, Enable and Disable -- and the Backup Policies & Jobs tab in the center pane lists the policies and the jobs they have produced.
You can also create a policy from the share it will be attached to: right-click a Network Share and choose Create Backup Policy.
Backup Modes and Transfer Direction
Mode and direction are chosen independently on the Policy Settings tab, and together they determine what the policy does. Getting these two right matters more than any other setting on the dialog, because they decide whether source files survive the transfer.
| Backup Mode | Effect |
|---|---|
| Copy Files | Files are copied and the source copy is left in place. This is the default and the safe choice. |
| Move Files | Files are copied and then removed from the source. Use this to migrate data off a filer, not to back it up. |
| Auto-tier Files | Files are moved to the destination and replaced at the source with a stub, so they still appear in the share and can still be opened. Selecting this forces Outbound, and is intended for tiering cold data to a Cloud Container. |
| Backup Direction | Data flows |
|---|---|
| Inbound Data Transfer | Network Share ← Remote Storage Export. QuantaStor pulls from the remote source. This is the default. |
| Outbound Data Transfer | Network Share → Remote Storage Export. QuantaStor pushes to the remote target. |
Enable Continuous Data Protection is only available for Outbound policies; it is greyed out while Inbound is selected. It copies files to the destination as they change rather than waiting for the scheduled run, which reduces the amount of data each scheduled run has to move. This is important for Network Shares with millions of files as it allows one to run the Backup Policy jobs less frequently as the files will already be copied to the destination. The Backup Policy still needs to run in order to catch anything that may have been missed as a verification pass.
Creating a Backup Policy
The dialog has six tabs. Network Share and the remote export are the only required selections; everything else has a working default.
General
- Network Share -- the QuantaStor share the policy is attached to. For an inbound policy this is the destination; for an outbound policy it is the source. Only one share can be selected, and the share type matters: long-term snapshot retention is unavailable for cloud type shares.
- Policy Name -- pre-filled with a generated name such as
backup-policy-1. The name is also used as the destination subdirectory when Store files to policy specific subdirectory on destination is enabled, so choose something meaningful. - Description -- free text for your own reference.
- Enable Policy -- checked by default; the policy becomes active as soon as it is created. Clear it to create the policy in a disabled state and start it later.
Policy Settings
Backup Mode and Backup Direction are covered in Backup Modes and Transfer Direction above. The rest of the tab identifies the remote side of the transfer.
Remote Storage Export selects the protocol, and the fields below it change to match:
- SMB -- an SMB3 or SMB2.1 share. Requires Username and Password.
- NFS -- an NFSv3 or NFSv4 export. No credentials are used; access is controlled by the export's own host permissions.
- Cloud Container -- a Cloud Container already configured on this system, pointing at S3-compatible or other blob storage. You select the container rather than entering a hostname.
For SMB and NFS:
- Hostname / IP Address -- the remote NAS filer or server.
- Username / Password -- SMB only. For a domain, enter the username as
DOMAIN/username. The username itself cannot contain a%character. - Scan -- contacts the host and enumerates its exports. Nothing appears in the list below until you run it.
- Select Share/Export -- the specific export to transfer, populated by Scan.
Enter the credentials in the Username and Password fields. Do not put them in the Hostname / IP Address field -- the dialog assembles the internal //user%password@host form for you. The Policy Summary box at the bottom restates the resulting transfer in a sentence, and is the quickest way to confirm the direction is what you intended before clicking OK.
Schedule Interval
Schedule Type picks between two mutually exclusive models:
- Use day/hour selections (default) -- a calendar schedule. Select the days and the hours; the policy runs at each selected hour on each selected day. The default selects Monday through Friday at 12 AM, 4 AM, 8 AM, 12 PM, 4 PM and 8 PM. All Days and All Hours select everything at once, and Offset delays each trigger by up to 59 minutes, so a 30 minute offset turns a 1 AM trigger into 1:30 AM.
- Use timer interval -- a rest interval instead of fixed times. The policy waits the specified number of minutes after a run completes before starting the next one, so runs never overlap. The minimum is 3 minutes.
A scheduled run is skipped rather than queued if the previous run for the same policy is still active.
Snapshot Settings
Each run can leave a snapshot of the destination share behind as a recovery checkpoint. Short term snapshots follow the schedule; long term ones are promoted from the short-term set and kept longer.
- Long Term Snapshot Retention Settings -- how many hourly, daily, weekly, monthly and quarterly checkpoints to keep. Each defaults to 2. A count of 7 dailies keeps 7 snapshots spanning the last week, with at least a day between them. Suggested Defaults fills in a sensible spread and Clear empties all five.
- Max Short Term Snapshots -- the schedule-driven snapshots to retain, 3 by default. Once the limit is reached the oldest is removed before a new one is created. To keep one permanently, rename it or set a description on it.
- Max Total Source Snapshots -- the computed total of the above, shown for reference.
These settings apply to ZFS and CephFS shares only. The whole tab is disabled when the selected Network Share is a cloud type share.
File Selection Settings
This tab decides which files are transferred and which are removed from the destination.
Purge Options
- Purge Frequency -- when the destination is reconciled against the source. Never Purge Files From Backup is the default, and it never deletes anything. The alternatives are Purge Expired/Deleted Files Immediately After Backup, ...Daily and ...Weekly. Purging is a separate scan from the backup itself, so running it nightly rather than after every job is usually the efficient choice.
- Purge Files Older Than -- the retention period in days. Files older than this are removed from the destination on the next purge.
- Backup Concurrency -- the number of parallel scan and copy streams. The default is Parallelized Backup (12 streams); the options are Serialized Backup (one stream) and 6, 12, 24, 32 or Highly Parallelized Backup (64). More streams help most on filers with very large file counts. This field is disabled when purging is set to never, because there is nothing to reconcile.
Be careful when purging is enabled. A purge removes files from the destination that are no longer present on the source. If you attach such a policy to a share that already holds unrelated data, that data will be deleted. With purging on, a Copy Files inbound policy maintains a mirror of the source, not an accumulating archive.
File Age Options
- Minimum Age Threshold -- files newer than this are skipped. Useful to avoid copying files still being written.
- Maximum Age Threshold -- files older than this are skipped.
Setting both to 0, or leaving both unchecked, transfers files of any age. Setting one or both defines an age window. Which timestamp is used is controlled by Include file access time-stamp in file age checks on the Advanced Settings tab; without it, age is based on creation and modification time only.
File Size Options
- Minimum Size Threshold -- files smaller than this are skipped.
- Maximum Size Threshold -- files larger than this are skipped.
- Minimum Size for Auto-tiering -- only stub files larger than this. Applies to Auto-tier Files mode only and is greyed out otherwise.
Filtering Options
- 'Include' File Filtering with a Match Pattern -- transfer only files matching the pattern, for example
/subdir/*. - 'Exclude' File Filtering with a Match Pattern -- transfer everything except files matching the pattern, for example
*.txt.
Using both filters at once is not recommended; if you do, make sure the two patterns cannot contradict each other.
Advanced Settings
- Start date -- the date the schedule becomes active. Defaults to today, so the policy begins at its next scheduled slot.
- Store files to policy specific subdirectory on destination -- unchecked by default. Leave it unchecked and the policy writes into the root of the destination, which makes an exact mirror and limits the share to a single policy. Check it and each policy writes into its own subdirectory, which is what allows several policies to share one Network Share. For an inbound policy the subdirectory is named after the policy automatically; for an outbound policy it is the Destination Subdirectory field below.
- Destination Subdirectory -- only enabled with the option above; required when it is.
- Maintain a log of each backup -- checked by default. See Monitoring Backup Jobs for where the logs are written.
- Include file access time-stamp in file age checks -- also considers access time when evaluating the age thresholds, so recently-read but unmodified files count as recent.
- Stub Creation Policy -- Create stubs nightly or Create stubs on every backup. Applies to Auto-tier Files mode only and is greyed out otherwise. It does not affect files copied by Continuous Data Protection.
- Reclaim Orphaned Snapshots -- adopts snapshots left behind by a previously deleted schedule so they are counted against this policy's retention rules instead of accumulating.
Modifying a Backup Policy
Modify presents the same six tabs, pre-filled from the selected policy. Schedule, filters, thresholds, purge behavior and credentials can all be changed on an existing policy, and the change applies from the next scheduled run.
The Network Share the policy is attached to is fixed at creation. To retarget a policy at a different share, delete it and create a new one.
Running a Policy on Demand
Run starts a Backup Job for the selected policy immediately, without waiting for the schedule and without altering it. Use it to test a new policy or to catch up after a source outage. The toolbar button is labelled Run; the dialog it opens is titled Trigger Backup Policy.
Enabling and Disabling a Policy
Disable stops a policy from running on its schedule while keeping the policy and its history intact; Enable resumes it. Prefer this over deleting a policy you expect to use again -- for example while a source filer is down for maintenance.
Deleting a Backup Policy
Deleting a policy removes the policy and the Backup Jobs associated with it. It does not delete any backed-up data -- the files already transferred stay on the destination share. A policy can also be deleted by right-clicking it and choosing the delete option.
Monitoring Backup Jobs
Each run creates a Backup Job object. Select the Network Share the policy is attached to and open the Backup Jobs tab in the center pane to watch progress, or use the Backup Policies & Jobs tab under Schedules to see jobs across all policies.
With Maintain a log of each backup enabled, each job writes a log on the QuantaStor system under:
/var/log/qs/backup-log/POLICY-NAME/
The backup phase writes a .changelog file and the purge phase a .purgelog file. Each log lists the full path of every file the job handled, which makes them usable as input to tape backup software such as IBM TSM.
Managing Backup Policies from the CLI
Every operation above has a CLI equivalent, which is the practical way to create policies in bulk. The full argument list for each is in the QuantaStor CLI Command Reference.
| Command | Purpose |
|---|---|
qs backup-policy-create |
Create a policy |
qs backup-policy-modify |
Change an existing policy |
qs backup-policy-list |
List policies |
qs backup-policy-get |
Show one policy in detail |
qs backup-policy-trigger |
Start a backup job now |
qs backup-policy-enable |
Resume a disabled policy |
qs backup-policy-disable |
Suspend a policy |
qs backup-policy-delete |
Delete a policy |
qs backup-job-list |
List backup jobs |
qs backup-job-get |
Show one job |
qs backup-job-cancel |
Cancel a running job |
A minimal inbound policy pulling an NFS export onto an existing share:
qs backup-policy-create --name=nightly-archive --network-share=archive \ --remote-hostname=10.0.10.50 --remote-export-type=nfs \ --remote-export-path=/export/projects --policy-type=copy-inbound
The same for an SMB source, with credentials passed as separate arguments:
qs backup-policy-create --name=filer-backup --network-share=archive \ --remote-hostname=10.0.10.60 --remote-export-type=smb \ --smb-username=DOMAIN/backupsvc --smb-password=aAbBcCdDeEfF0123 \ --policy-type=copy-inbound --purge-policy=daily --retain-period=60
Note that the CLI and the web interface do not share every default. --scan-threads defaults to 5 from the CLI while the dialog defaults to 12 streams, --remote-export-type defaults to nfs while the dialog pre-selects SMB, and --policy-type defaults to copy-inbound in both. Set the values you want explicitly in scripts rather than relying on either default.
Data Mover Utility / pwalk
The scanning and copying is performed by pwalk, an open source command line utility extended by OSNEXUS and shipped with QuantaStor. It is what makes the parallel scan possible, and it can be used directly for ad-hoc copies and scans -- see the pwalk page.
Related pages
- Network Shares -- creating the share a policy attaches to
- Cloud Containers / NAS Gateway -- configuring a Cloud Container as a policy target
- Snapshot Schedules -- scheduled snapshots independent of a backup policy
- Remote-replication (DR) -- replicating whole shares and volumes between QuantaStor systems
- Pwalk utility -- the underlying scan and copy tool
- Storage Pools -- the pool the destination share is provisioned from
Verified against QuantaStor 6.9.0.