Backup Policies: Difference between revisions

From OSNEXUS Online Documentation Site
Jump to navigation Jump to search
← Older editNewer edit →
VisualWikitext
mNo edit summary
m Rewrite to the documentation guidelines: document all six Create/Modify tabs field by field, add the undocumented Move/Auto-tier modes, outbound transfers, Cloud Container sources, Schedule, Snapshot Settings, File Age/Size/Filter options, CDP, stubs and the Run/Enable/Disable operations; correct the backup log path, the concurrency default and the SMB credential instructions; new screenshots; CLI section verified against a live system (QSTOR-12350)
Line 1: Line 1:
[[Category:admin_guide]]
[[Category:admin_guide]]
A '''Backup Policy''' copies or moves file data between a QuantaStor [[Network Shares|Network Share]] and a remote data source, on a schedule. The remote source can be an SMB or NFS export on another server or NAS filer, or a [[Cloud Containers / NAS Gateway|Cloud Container]] backed by S3-compatible object storage. Policies transfer data in either direction, so the same mechanism covers pulling a filer onto QuantaStor, pushing a share out to the cloud, and tiering cold files off a share to object storage.


== Overview ==
Scanning and copying are parallelized, so filers with 100 million or more files can be scanned for changes in a practical amount of time. Each run creates a '''Backup Job''' object you can monitor while it works.


Within QuantaStor you can create ''backup policies'' where data from any NFS or CIFS share on your network can be automatically backed up for you to your QuantaStor system.  To create a ''Backup Policy'' simply right-click on the ''Network Share'' where you want the data to be backed up to and choose the 'Create Backup Policy...' option from the pop-up menu.
{| class="wikitable"
Backup policies will do a CIFS/NFS mount of the specified NAS share on your network locally to the system in order to access the data to be archived.  When the backup starts it creates a Backup Job object which you will see in the web interface and you can see the progress of any given ''Backup Job'' by monitoring it in the ''Backup Jobs'' tab in the center-pane of the web interface after you select the ''Network Share'' to which the backup policy is attached.
! Section !! Covers
|-
| [[#Backup Modes and Transfer Direction|Backup Modes and Transfer Direction]] || Copy, move and auto-tier; inbound versus outbound transfers
|-
| [[#Creating a Backup Policy|Creating a Backup Policy]] || Every tab of the Create dialog, field by field
|-
| [[#Modifying a Backup Policy|Modifying a Backup Policy]] || What can and cannot be changed after creation
|-
| [[#Running a Policy on Demand|Running a Policy on Demand]] || Triggering a backup outside its schedule
|-
| [[#Enabling and Disabling a Policy|Enabling and Disabling a Policy]] || Suspending a policy without deleting it
|-
| [[#Deleting a Backup Policy|Deleting a Backup Policy]] || Removing a policy, and what happens to the data
|-
| [[#Monitoring Backup Jobs|Monitoring Backup Jobs]] || The Backup Jobs tab and the backup logs
|-
| [[#Managing Backup Policies from the CLI|Managing Backup Policies from the CLI]] || The equivalent <code>qs</code> commands
|}


[[File:Create Backup Policy.jpg|800px|Right-click on a share, then select ''Create Backup Policy...'']]
{{Navigation|Storage Management &rarr; Schedules ''(section)'' &rarr; Backup Policies &amp; Jobs ''(tab)'' &rarr; Backup Policy ''(toolbar group)''}}


== [https://wiki.osnexus.com/index.php?title=Create_Backup_Policy Creating Backup Policies] ==
The '''Backup Policy''' toolbar group holds all six operations -- Create, Modify, Delete, Run, Enable and Disable -- and the '''Backup Policies &amp; Jobs''' tab in the center pane lists the policies and the jobs they have produced.


Backup policies in QuantaStor support heavy parallelism so that very large NAS filers with 100m+ files can be easily scanned for changes. The default level of parallelism is 32 concurrent scan+copy threads but this can be reduced or increased to 64 concurrent threads.
[[File:bkuppol_grid.png|thumb|right|800px|The Backup Policy toolbar group and the Backup Policies &amp; Jobs tab.]]


'''Navigation:''' Storage Management --> Schedules --> Backup Policy --> Create ''(toolbar)''
You can also create a policy from the share it will be attached to: right-click a '''Network Share''' and choose '''Create Backup Policy'''.


==== Network Share ====
== Backup Modes and Transfer Direction ==


This is where you indicate where you want the data to be backed up to on your QuantaStor systems. With QuantaStor backup policies your data is copied from a NAS share on the network to a ''Network Share'' on your QuantaStor system.
Mode and direction are chosen independently on the '''Policy Settings''' tab, and together they determine what the policy does. Getting these two right matters more than any other setting on the dialog, because they decide whether source files survive the transfer.


==== Policy Name ====
{| class="wikitable"
! Backup Mode !! Effect
|-
| '''Copy Files''' || Files are copied and the source copy is left in place. This is the default and the safe choice.
|-
| '''Move Files''' || Files are copied and then '''removed from the source'''. Use this to migrate data off a filer, not to back it up.
|-
| '''Auto-tier Files''' || Files are moved to the destination and replaced at the source with a stub, so they still appear in the share and can still be opened. Selecting this forces '''Outbound''', and is intended for tiering cold data to a Cloud Container.
|}


This is a friendly name for your ''Backup Policy''.  If you are going to have multiple policies doing backups to the same ''Network Share'' then each policy will be associated with a directory with the name of the policy. For example, if your share is called "media-backups" and you have a policy called "project1" and a policy called "project2" then there will be sub-directories under the "media-backups" share for "project1" and "project2".  In order to support multiple policies per ''Network Share'' you must select, in the ''Advanced Settings'' tab, the option which says 'Store files to policy specific subdirectory on destination'.  If that is not selected then only one policy can be associated with the network share and the backups will go into the root of the share to form an exact mirror copy.
{| class="wikitable"
! Backup Direction !! Data flows
|-
| '''Inbound Data Transfer''' || Network Share &larr; Remote Storage Export. QuantaStor pulls from the remote source. This is the default.
|-
| '''Outbound Data Transfer''' || Network Share &rarr; Remote Storage Export. QuantaStor pushes to the remote target.
|}


[[File:Bkup Policy - Gnrl.jpg|512px|Backup Policies use parallelized scanning and copy techniques to efficiently backup (copy) or migrate (move) data between Network Shares and remote data sources. Long Term Retention Rules only apply to ZFS and CephFS share types.]]
'''Enable Continuous Data Protection''' is only available for '''Outbound''' policies; it is greyed out while '''Inbound''' is selected. It copies files to the destination as they change rather than waiting for the scheduled run, which reduces the amount of data each scheduled run has to move.


[[File:Bkup Policy - Adv Settings.jpg|512px]]
== Creating a Backup Policy ==


==== Selecting the Backup Source ====
{{Navigation|Storage Management &rarr; Schedules ''(section)'' &rarr; Backup Policy ''(toolbar group)'' &rarr; Create}}


Under the ''Policy Settings'' tab the section which says '''Hostname / IP Address:''' enter the IP address of the NAS filer or server which is sharing the NAS folder you want to backup.  For NFS or SMB shares you should enter the IP address and press the '''Scan''' button.  If NFS shares are found they'll show up in the CIFS/NFS Export: list.  For CIFS share backups you'll need to enter the network path to the share in a special format starting with double forward slashes like so:  '''//username%password@ipaddress'''.  For example, you might scan for shares on a filer located at 10.10.5.5 using the SMB credentials of 'admin' and password 'password123' using this path: '''//admin%password123@10.10.5.5'''.  In AD environments you can also include the domain in the SMB path like so '''//DOMAIN/username%password@ipaddress'''.
The dialog has six tabs. '''Network Share''' and the remote export are the only required selections; everything else has a working default.


[[File:Create Backup Policy - Poly Set.jpg|512x512px]]
=== General ===


==== Policy Type ====
[[File:bkuppol_general.png|thumb|right|700px|The General tab: the Network Share the policy is attached to, and the policy name.]]


You can indicate that you want the backup policy to backup everything by selecting 'Backup All Files' or you can do a 'Sliding Window Backup'. For backing up data from huge filers with 100m+ files it is sometimes useful to only backup and maintain a ''sliding window'' of the most recently modified or created files. If you set the ''Retention Period'' to 60 days then all files that have been created or modified within the last 60 days will be retained. Files that are older than that will be purged from the backup folder.
* '''Network Share''' -- the QuantaStor share the policy is attached to. For an inbound policy this is the destination; for an outbound policy it is the source. Only one share can be selected, and the share type matters: long-term snapshot retention is unavailable for '''cloud''' type shares.
* '''Policy Name''' -- pre-filled with a generated name such as <code>backup-policy-1</code>. The name is also used as the destination subdirectory when '''Store files to policy specific subdirectory on destination''' is enabled, so choose something meaningful.
* '''Description''' -- free text for your own reference.
* '''Enable Policy''' -- checked by default; the policy becomes active as soon as it is created. Clear it to create the policy in a disabled state and start it later.


Be careful with the ''Backup All Files'' mode.  If you have a Purge Policy enabled it will remove any files from the ''network share'' which were not found on the source NAS share that's being backed up.  If you attached such a backup policy to an existing share which has data on it, the purge policy will remove any data/files that exists in your QuantaStor Network Share which is not on the source NAS share on the remote filer.  So use caution with this as ''Backup All Files'' really means ''maintain a mirror copy of the remote NAS share''.
=== Policy Settings ===


==== Purge Policy ====
[[File:bkuppol_policy_settings.png|thumb|right|700px|The Policy Settings tab: backup mode, transfer direction, and the remote storage export.]]


Backup policies may run many times per day to quickly backup new and modified files.  A scan to determine what needs purging is typically less important so it is more efficient to run it nightly rather than with each and every backup job.  For the ''Sliding Window'' policies the purge phase will throw out any files that are older than the retention period. For the ''Backup All Files'' policies there is a comparison that is done and any files that are no longer present in the NAS source share are removed from the backup.  The Purge Policy can also be set to 'Never Purge Files From Backup' which will backup files to your Network Share but never remove them.
Backup Mode and Backup Direction are covered in '''[[#Backup Modes and Transfer Direction|Backup Modes and Transfer Direction]]''' above. The rest of the tab identifies the remote side of the transfer.


[[File:Bkup Policy - File Sel.jpg|512x512px]]
'''Remote Storage Export''' selects the protocol, and the fields below it change to match:


==== Backup Logs ====
* '''SMB''' -- an SMB3 or SMB2.1 share. Requires '''Username''' and '''Password'''.
* '''NFS''' -- an NFSv3 or NFSv4 export. No credentials are used; access is controlled by the export's own host permissions.
* '''Cloud Container''' -- a [[Cloud Containers / NAS Gateway|Cloud Container]] already configured on this system, pointing at S3-compatible or other blob storage. You select the container rather than entering a hostname.


If you select 'Maintain a log of each backup' from the '''Advanced Settings''' tab then a backup log file will be written out after each backup.  Backup logs can be found on your QuantaStor system in the /var/log/backup-log/POLICY-NAME directory.  The purge process produces a log with the .purgelog suffix and the backup process produces a log with the .changelog suffix.
For SMB and NFS:


== [https://wiki.osnexus.com/index.php?title=Backup_Policy_Delete Deleting Backup Policies] ==
* '''Hostname / IP Address''' -- the remote NAS filer or server.
* '''Username''' / '''Password''' -- SMB only. For a domain, enter the username as <code>DOMAIN/username</code>. The username itself cannot contain a <code>%</code> character.
* '''Scan''' -- contacts the host and enumerates its exports. Nothing appears in the list below until you run it.
* '''Select Share/Export''' -- the specific export to transfer, populated by '''Scan'''.


To delete an existing ''Backup Policy'' simply right-click on the policy and choose the delete option from the menu. Deleting a backup policy does not delete any data, just the policy itself and any ''Backup Jobs'' associated with it.
'''Enter the credentials in the Username and Password fields.''' Do not put them in the '''Hostname / IP Address''' field -- the dialog assembles the internal <code>//user%password@host</code> form for you. The '''Policy Summary''' box at the bottom restates the resulting transfer in a sentence, and is the quickest way to confirm the direction is what you intended before clicking OK.
 
=== Schedule Interval ===
 
[[File:bkuppol_schedule.png|thumb|right|700px|The Schedule Interval tab, with the default Monday-Friday, every-four-hours calendar schedule.]]
 
'''Schedule Type''' picks between two mutually exclusive models:
 
* '''Use day/hour selections''' (default) -- a calendar schedule. Select the days and the hours; the policy runs at each selected hour on each selected day. The default selects '''Monday through Friday''' at '''12 AM, 4 AM, 8 AM, 12 PM, 4 PM and 8 PM'''. '''All Days''' and '''All Hours''' select everything at once, and '''Offset''' delays each trigger by up to 59 minutes, so a 30 minute offset turns a 1 AM trigger into 1:30 AM.
* '''Use timer interval''' -- a rest interval instead of fixed times. The policy waits the specified number of minutes '''after a run completes''' before starting the next one, so runs never overlap. The minimum is 3 minutes.
 
A scheduled run is skipped rather than queued if the previous run for the same policy is still active.
 
=== Snapshot Settings ===
 
[[File:bkuppol_snapshots.png|thumb|right|700px|The Snapshot Settings tab. These controls are greyed out for cloud type shares.]]
 
Each run can leave a snapshot of the destination share behind as a recovery checkpoint. '''Short term''' snapshots follow the schedule; '''long term''' ones are promoted from the short-term set and kept longer.
 
* '''Long Term Snapshot Retention Settings''' -- how many hourly, daily, weekly, monthly and quarterly checkpoints to keep. Each defaults to '''2'''. A count of 7 dailies keeps 7 snapshots spanning the last week, with at least a day between them. '''Suggested Defaults''' fills in a sensible spread and '''Clear''' empties all five.
* '''Max Short Term Snapshots''' -- the schedule-driven snapshots to retain, '''3''' by default. Once the limit is reached the oldest is removed before a new one is created. To keep one permanently, rename it or set a description on it.
* '''Max Total Source Snapshots''' -- the computed total of the above, shown for reference.
 
'''These settings apply to ZFS and CephFS shares only.''' The whole tab is disabled when the selected Network Share is a '''cloud''' type share.
 
=== File Selection Settings ===
 
[[File:bkuppol_file_selection.png|thumb|right|700px|The File Selection Settings tab: purge behavior, age and size thresholds, and filename filters.]]
 
This tab decides which files are transferred and which are removed from the destination.
 
'''Purge Options'''
 
* '''Purge Frequency''' -- when the destination is reconciled against the source. '''Never Purge Files From Backup''' is the default, and it never deletes anything. The alternatives are '''Purge Expired/Deleted Files Immediately After Backup''', '''...Daily''' and '''...Weekly'''. Purging is a separate scan from the backup itself, so running it nightly rather than after every job is usually the efficient choice.
* '''Purge Files Older Than''' -- the retention period in days. Files older than this are removed from the destination on the next purge.
* '''Backup Concurrency''' -- the number of parallel scan and copy streams. The default is '''Parallelized Backup (12 streams)'''; the options are '''Serialized Backup''' (one stream) and '''6''', '''12''', '''24''', '''32''' or '''Highly Parallelized Backup (64)'''. More streams help most on filers with very large file counts. This field is disabled when purging is set to never, because there is nothing to reconcile.
 
'''Be careful when purging is enabled.''' A purge removes files from the destination that are no longer present on the source. If you attach such a policy to a share that already holds unrelated data, that data will be deleted. With purging on, a '''Copy Files''' inbound policy maintains a mirror of the source, not an accumulating archive.
 
'''File Age Options'''
 
* '''Minimum Age Threshold''' -- files newer than this are skipped. Useful to avoid copying files still being written.
* '''Maximum Age Threshold''' -- files older than this are skipped.
 
Setting both to 0, or leaving both unchecked, transfers files of any age. Setting one or both defines an age window. Which timestamp is used is controlled by '''Include file access time-stamp in file age checks''' on the '''Advanced Settings''' tab; without it, age is based on creation and modification time only.
 
'''File Size Options'''
 
* '''Minimum Size Threshold''' -- files smaller than this are skipped.
* '''Maximum Size Threshold''' -- files larger than this are skipped.
* '''Minimum Size for Auto-tiering''' -- only stub files larger than this. Applies to '''Auto-tier Files''' mode only and is greyed out otherwise.
 
'''Filtering Options'''
 
* '''&#39;Include&#39; File Filtering''' with a '''Match Pattern''' -- transfer only files matching the pattern, for example <code>/subdir/*</code>.
* '''&#39;Exclude&#39; File Filtering''' with a '''Match Pattern''' -- transfer everything except files matching the pattern, for example <code>*.txt</code>.
 
Using both filters at once is '''not recommended'''; if you do, make sure the two patterns cannot contradict each other.
 
=== Advanced Settings ===
 
[[File:bkuppol_advanced.png|thumb|right|700px|The Advanced Settings tab.]]
 
* '''Start date''' -- the date the schedule becomes active. Defaults to today, so the policy begins at its next scheduled slot.
* '''Store files to policy specific subdirectory on destination''' -- '''unchecked by default'''. Leave it unchecked and the policy writes into the '''root''' of the destination, which makes an exact mirror and limits the share to a single policy. Check it and each policy writes into its own subdirectory, which is what allows '''several policies to share one Network Share'''. For an inbound policy the subdirectory is named after the policy automatically; for an outbound policy it is the '''Destination Subdirectory''' field below.
* '''Destination Subdirectory''' -- only enabled with the option above; required when it is.
* '''Maintain a log of each backup''' -- '''checked by default'''. See '''[[#Monitoring Backup Jobs|Monitoring Backup Jobs]]''' for where the logs are written.
* '''Include file access time-stamp in file age checks''' -- also considers access time when evaluating the age thresholds, so recently-read but unmodified files count as recent.
* '''Stub Creation Policy''' -- '''Create stubs nightly''' or '''Create stubs on every backup'''. Applies to '''Auto-tier Files''' mode only and is greyed out otherwise. It does not affect files copied by Continuous Data Protection.
* '''Reclaim Orphaned Snapshots''' -- adopts snapshots left behind by a previously deleted schedule so they are counted against this policy's retention rules instead of accumulating.
 
== Modifying a Backup Policy ==
 
{{Navigation|Storage Management &rarr; Schedules ''(section)'' &rarr; Backup Policy ''(toolbar group)'' &rarr; Modify}}
 
[[File:bkuppol_modify.png|thumb|right|700px|The Modify Backup Policy dialog, pre-filled from the selected policy.]]
 
Modify presents the same six tabs, pre-filled from the selected policy. Schedule, filters, thresholds, purge behavior and credentials can all be changed on an existing policy, and the change applies from the next scheduled run.
 
The '''Network Share''' the policy is attached to is fixed at creation. To retarget a policy at a different share, delete it and create a new one.
 
== Running a Policy on Demand ==
 
{{Navigation|Storage Management &rarr; Schedules ''(section)'' &rarr; Backup Policy ''(toolbar group)'' &rarr; Run}}
 
[[File:bkuppol_run.png|thumb|right|700px|Run starts a backup job immediately, outside the schedule.]]
 
'''Run''' starts a Backup Job for the selected policy immediately, without waiting for the schedule and without altering it. Use it to test a new policy or to catch up after a source outage. The toolbar button is labelled '''Run'''; the dialog it opens is titled '''Trigger Backup Policy'''.
 
== Enabling and Disabling a Policy ==
 
{{Navigation|Storage Management &rarr; Schedules ''(section)'' &rarr; Backup Policy ''(toolbar group)'' &rarr; Enable / Disable}}
 
'''Disable''' stops a policy from running on its schedule while keeping the policy and its history intact; '''Enable''' resumes it. Prefer this over deleting a policy you expect to use again -- for example while a source filer is down for maintenance.
 
== Deleting a Backup Policy ==
 
{{Navigation|Storage Management &rarr; Schedules ''(section)'' &rarr; Backup Policy ''(toolbar group)'' &rarr; Delete}}
 
[[File:bkuppol_delete.png|thumb|right|700px|Deleting a policy removes the policy and its job history, not the data.]]
 
Deleting a policy removes the policy and the '''Backup Jobs''' associated with it. '''It does not delete any backed-up data''' -- the files already transferred stay on the destination share. A policy can also be deleted by right-clicking it and choosing the delete option.
 
== Monitoring Backup Jobs ==
 
Each run creates a '''Backup Job''' object. Select the '''Network Share''' the policy is attached to and open the '''Backup Jobs''' tab in the center pane to watch progress, or use the '''Backup Policies &amp; Jobs''' tab under '''Schedules''' to see jobs across all policies.
 
With '''Maintain a log of each backup''' enabled, each job writes a log on the QuantaStor system under:
 
<pre style="font-size: smaller">
/var/log/qs/backup-log/POLICY-NAME/
</pre>
 
The backup phase writes a <code>.changelog</code> file and the purge phase a <code>.purgelog</code> file. Each log lists the full path of every file the job handled, which makes them usable as input to tape backup software such as IBM TSM.
 
== Managing Backup Policies from the CLI ==
 
Every operation above has a CLI equivalent, which is the practical way to create policies in bulk. The full argument list for each is in the [[QuantaStor CLI Command Reference]].
 
{| class="wikitable"
! Command !! Purpose
|-
| [[QuantaStor CLI Command Reference#backup-policy-create|<code>qs backup-policy-create</code>]] || Create a policy
|-
| [[QuantaStor CLI Command Reference#backup-policy-modify|<code>qs backup-policy-modify</code>]] || Change an existing policy
|-
| [[QuantaStor CLI Command Reference#backup-policy-list|<code>qs backup-policy-list</code>]] || List policies
|-
| [[QuantaStor CLI Command Reference#backup-policy-get|<code>qs backup-policy-get</code>]] || Show one policy in detail
|-
| [[QuantaStor CLI Command Reference#backup-policy-trigger|<code>qs backup-policy-trigger</code>]] || Start a backup job now
|-
| [[QuantaStor CLI Command Reference#backup-policy-enable|<code>qs backup-policy-enable</code>]] || Resume a disabled policy
|-
| [[QuantaStor CLI Command Reference#backup-policy-disable|<code>qs backup-policy-disable</code>]] || Suspend a policy
|-
| [[QuantaStor CLI Command Reference#backup-policy-delete|<code>qs backup-policy-delete</code>]] || Delete a policy
|-
| [[QuantaStor CLI Command Reference#backup-job-list|<code>qs backup-job-list</code>]] || List backup jobs
|-
| [[QuantaStor CLI Command Reference#backup-job-get|<code>qs backup-job-get</code>]] || Show one job
|-
| [[QuantaStor CLI Command Reference#backup-job-cancel|<code>qs backup-job-cancel</code>]] || Cancel a running job
|}
 
A minimal inbound policy pulling an NFS export onto an existing share:
 
<pre style="font-size: smaller">
qs backup-policy-create --name=nightly-archive --network-share=archive \
  --remote-hostname=10.0.10.50 --remote-export-type=nfs \
  --remote-export-path=/export/projects --policy-type=copy-inbound
</pre>
 
The same for an SMB source, with credentials passed as separate arguments:
 
<pre style="font-size: smaller">
qs backup-policy-create --name=filer-backup --network-share=archive \
  --remote-hostname=10.0.10.60 --remote-export-type=smb \
  --smb-username=DOMAIN/backupsvc --smb-password=aAbBcCdDeEfF0123 \
  --policy-type=copy-inbound --purge-policy=daily --retain-period=60
</pre>
 
Note that the CLI and the web interface do not share every default. <code>--scan-threads</code> defaults to '''5''' from the CLI while the dialog defaults to '''12''' streams, <code>--remote-export-type</code> defaults to '''nfs''' while the dialog pre-selects '''SMB''', and <code>--policy-type</code> defaults to '''copy-inbound''' in both. Set the values you want explicitly in scripts rather than relying on either default.


== Data Mover Utility / pwalk ==
== Data Mover Utility / pwalk ==


pwalk is a open source command line utility extended by OSNEXUS and included with QuantaStor and used for concurrent file copy/movement and scanning, more information is available [[Pwalk utility|here]].
The scanning and copying is performed by '''pwalk''', an open source command line utility extended by OSNEXUS and shipped with QuantaStor. It is what makes the parallel scan possible, and it can be used directly for ad-hoc copies and scans -- see [[Pwalk utility|the pwalk page]].
 
== Related pages ==
 
* [[Network Shares]] -- creating the share a policy attaches to
* [[Cloud Containers / NAS Gateway]] -- configuring a Cloud Container as a policy target
* [[Snapshot Schedules]] -- scheduled snapshots independent of a backup policy
* [[Remote-replication (DR)]] -- replicating whole shares and volumes between QuantaStor systems
* [[Pwalk utility]] -- the underlying scan and copy tool
* [[Storage Pools]] -- the pool the destination share is provisioned from
 
----
<small>''Verified against QuantaStor 6.9.0.''</small>

Revision as of 02:32, 3 September 2026

A Backup Policy copies or moves file data between a QuantaStor Network Share and a remote data source, on a schedule. The remote source can be an SMB or NFS export on another server or NAS filer, or a Cloud Container backed by S3-compatible object storage. Policies transfer data in either direction, so the same mechanism covers pulling a filer onto QuantaStor, pushing a share out to the cloud, and tiering cold files off a share to object storage.

Scanning and copying are parallelized, so filers with 100 million or more files can be scanned for changes in a practical amount of time. Each run creates a Backup Job object you can monitor while it works.

Section Covers
Backup Modes and Transfer Direction Copy, move and auto-tier; inbound versus outbound transfers
Creating a Backup Policy Every tab of the Create dialog, field by field
Modifying a Backup Policy What can and cannot be changed after creation
Running a Policy on Demand Triggering a backup outside its schedule
Enabling and Disabling a Policy Suspending a policy without deleting it
Deleting a Backup Policy Removing a policy, and what happens to the data
Monitoring Backup Jobs The Backup Jobs tab and the backup logs
Managing Backup Policies from the CLI The equivalent qs commands
Navigation: Storage Management → Schedules (section) → Backup Policies & Jobs (tab) → Backup Policy (toolbar group)

The Backup Policy toolbar group holds all six operations -- Create, Modify, Delete, Run, Enable and Disable -- and the Backup Policies & Jobs tab in the center pane lists the policies and the jobs they have produced.

The Backup Policy toolbar group and the Backup Policies & Jobs tab.

You can also create a policy from the share it will be attached to: right-click a Network Share and choose Create Backup Policy.

Backup Modes and Transfer Direction

Mode and direction are chosen independently on the Policy Settings tab, and together they determine what the policy does. Getting these two right matters more than any other setting on the dialog, because they decide whether source files survive the transfer.

Backup Mode Effect
Copy Files Files are copied and the source copy is left in place. This is the default and the safe choice.
Move Files Files are copied and then removed from the source. Use this to migrate data off a filer, not to back it up.
Auto-tier Files Files are moved to the destination and replaced at the source with a stub, so they still appear in the share and can still be opened. Selecting this forces Outbound, and is intended for tiering cold data to a Cloud Container.
Backup Direction Data flows
Inbound Data Transfer Network Share ← Remote Storage Export. QuantaStor pulls from the remote source. This is the default.
Outbound Data Transfer Network Share → Remote Storage Export. QuantaStor pushes to the remote target.

Enable Continuous Data Protection is only available for Outbound policies; it is greyed out while Inbound is selected. It copies files to the destination as they change rather than waiting for the scheduled run, which reduces the amount of data each scheduled run has to move.

Creating a Backup Policy

Navigation: Storage Management → Schedules (section) → Backup Policy (toolbar group) → Create

The dialog has six tabs. Network Share and the remote export are the only required selections; everything else has a working default.

General

The General tab: the Network Share the policy is attached to, and the policy name.
  • Network Share -- the QuantaStor share the policy is attached to. For an inbound policy this is the destination; for an outbound policy it is the source. Only one share can be selected, and the share type matters: long-term snapshot retention is unavailable for cloud type shares.
  • Policy Name -- pre-filled with a generated name such as backup-policy-1. The name is also used as the destination subdirectory when Store files to policy specific subdirectory on destination is enabled, so choose something meaningful.
  • Description -- free text for your own reference.
  • Enable Policy -- checked by default; the policy becomes active as soon as it is created. Clear it to create the policy in a disabled state and start it later.

Policy Settings

The Policy Settings tab: backup mode, transfer direction, and the remote storage export.

Backup Mode and Backup Direction are covered in Backup Modes and Transfer Direction above. The rest of the tab identifies the remote side of the transfer.

Remote Storage Export selects the protocol, and the fields below it change to match:

  • SMB -- an SMB3 or SMB2.1 share. Requires Username and Password.
  • NFS -- an NFSv3 or NFSv4 export. No credentials are used; access is controlled by the export's own host permissions.
  • Cloud Container -- a Cloud Container already configured on this system, pointing at S3-compatible or other blob storage. You select the container rather than entering a hostname.

For SMB and NFS:

  • Hostname / IP Address -- the remote NAS filer or server.
  • Username / Password -- SMB only. For a domain, enter the username as DOMAIN/username. The username itself cannot contain a % character.
  • Scan -- contacts the host and enumerates its exports. Nothing appears in the list below until you run it.
  • Select Share/Export -- the specific export to transfer, populated by Scan.

Enter the credentials in the Username and Password fields. Do not put them in the Hostname / IP Address field -- the dialog assembles the internal //user%password@host form for you. The Policy Summary box at the bottom restates the resulting transfer in a sentence, and is the quickest way to confirm the direction is what you intended before clicking OK.

Schedule Interval

The Schedule Interval tab, with the default Monday-Friday, every-four-hours calendar schedule.

Schedule Type picks between two mutually exclusive models:

  • Use day/hour selections (default) -- a calendar schedule. Select the days and the hours; the policy runs at each selected hour on each selected day. The default selects Monday through Friday at 12 AM, 4 AM, 8 AM, 12 PM, 4 PM and 8 PM. All Days and All Hours select everything at once, and Offset delays each trigger by up to 59 minutes, so a 30 minute offset turns a 1 AM trigger into 1:30 AM.
  • Use timer interval -- a rest interval instead of fixed times. The policy waits the specified number of minutes after a run completes before starting the next one, so runs never overlap. The minimum is 3 minutes.

A scheduled run is skipped rather than queued if the previous run for the same policy is still active.

Snapshot Settings

The Snapshot Settings tab. These controls are greyed out for cloud type shares.

Each run can leave a snapshot of the destination share behind as a recovery checkpoint. Short term snapshots follow the schedule; long term ones are promoted from the short-term set and kept longer.

  • Long Term Snapshot Retention Settings -- how many hourly, daily, weekly, monthly and quarterly checkpoints to keep. Each defaults to 2. A count of 7 dailies keeps 7 snapshots spanning the last week, with at least a day between them. Suggested Defaults fills in a sensible spread and Clear empties all five.
  • Max Short Term Snapshots -- the schedule-driven snapshots to retain, 3 by default. Once the limit is reached the oldest is removed before a new one is created. To keep one permanently, rename it or set a description on it.
  • Max Total Source Snapshots -- the computed total of the above, shown for reference.

These settings apply to ZFS and CephFS shares only. The whole tab is disabled when the selected Network Share is a cloud type share.

File Selection Settings

The File Selection Settings tab: purge behavior, age and size thresholds, and filename filters.

This tab decides which files are transferred and which are removed from the destination.

Purge Options

  • Purge Frequency -- when the destination is reconciled against the source. Never Purge Files From Backup is the default, and it never deletes anything. The alternatives are Purge Expired/Deleted Files Immediately After Backup, ...Daily and ...Weekly. Purging is a separate scan from the backup itself, so running it nightly rather than after every job is usually the efficient choice.
  • Purge Files Older Than -- the retention period in days. Files older than this are removed from the destination on the next purge.
  • Backup Concurrency -- the number of parallel scan and copy streams. The default is Parallelized Backup (12 streams); the options are Serialized Backup (one stream) and 6, 12, 24, 32 or Highly Parallelized Backup (64). More streams help most on filers with very large file counts. This field is disabled when purging is set to never, because there is nothing to reconcile.

Be careful when purging is enabled. A purge removes files from the destination that are no longer present on the source. If you attach such a policy to a share that already holds unrelated data, that data will be deleted. With purging on, a Copy Files inbound policy maintains a mirror of the source, not an accumulating archive.

File Age Options

  • Minimum Age Threshold -- files newer than this are skipped. Useful to avoid copying files still being written.
  • Maximum Age Threshold -- files older than this are skipped.

Setting both to 0, or leaving both unchecked, transfers files of any age. Setting one or both defines an age window. Which timestamp is used is controlled by Include file access time-stamp in file age checks on the Advanced Settings tab; without it, age is based on creation and modification time only.

File Size Options

  • Minimum Size Threshold -- files smaller than this are skipped.
  • Maximum Size Threshold -- files larger than this are skipped.
  • Minimum Size for Auto-tiering -- only stub files larger than this. Applies to Auto-tier Files mode only and is greyed out otherwise.

Filtering Options

  • 'Include' File Filtering with a Match Pattern -- transfer only files matching the pattern, for example /subdir/*.
  • 'Exclude' File Filtering with a Match Pattern -- transfer everything except files matching the pattern, for example *.txt.

Using both filters at once is not recommended; if you do, make sure the two patterns cannot contradict each other.

Advanced Settings

The Advanced Settings tab.
  • Start date -- the date the schedule becomes active. Defaults to today, so the policy begins at its next scheduled slot.
  • Store files to policy specific subdirectory on destination -- unchecked by default. Leave it unchecked and the policy writes into the root of the destination, which makes an exact mirror and limits the share to a single policy. Check it and each policy writes into its own subdirectory, which is what allows several policies to share one Network Share. For an inbound policy the subdirectory is named after the policy automatically; for an outbound policy it is the Destination Subdirectory field below.
  • Destination Subdirectory -- only enabled with the option above; required when it is.
  • Maintain a log of each backup -- checked by default. See Monitoring Backup Jobs for where the logs are written.
  • Include file access time-stamp in file age checks -- also considers access time when evaluating the age thresholds, so recently-read but unmodified files count as recent.
  • Stub Creation Policy -- Create stubs nightly or Create stubs on every backup. Applies to Auto-tier Files mode only and is greyed out otherwise. It does not affect files copied by Continuous Data Protection.
  • Reclaim Orphaned Snapshots -- adopts snapshots left behind by a previously deleted schedule so they are counted against this policy's retention rules instead of accumulating.

Modifying a Backup Policy

Navigation: Storage Management → Schedules (section) → Backup Policy (toolbar group) → Modify
The Modify Backup Policy dialog, pre-filled from the selected policy.

Modify presents the same six tabs, pre-filled from the selected policy. Schedule, filters, thresholds, purge behavior and credentials can all be changed on an existing policy, and the change applies from the next scheduled run.

The Network Share the policy is attached to is fixed at creation. To retarget a policy at a different share, delete it and create a new one.

Running a Policy on Demand

Navigation: Storage Management → Schedules (section) → Backup Policy (toolbar group) → Run
Run starts a backup job immediately, outside the schedule.

Run starts a Backup Job for the selected policy immediately, without waiting for the schedule and without altering it. Use it to test a new policy or to catch up after a source outage. The toolbar button is labelled Run; the dialog it opens is titled Trigger Backup Policy.

Enabling and Disabling a Policy

Navigation: Storage Management → Schedules (section) → Backup Policy (toolbar group) → Enable / Disable

Disable stops a policy from running on its schedule while keeping the policy and its history intact; Enable resumes it. Prefer this over deleting a policy you expect to use again -- for example while a source filer is down for maintenance.

Deleting a Backup Policy

Navigation: Storage Management → Schedules (section) → Backup Policy (toolbar group) → Delete
Deleting a policy removes the policy and its job history, not the data.

Deleting a policy removes the policy and the Backup Jobs associated with it. It does not delete any backed-up data -- the files already transferred stay on the destination share. A policy can also be deleted by right-clicking it and choosing the delete option.

Monitoring Backup Jobs

Each run creates a Backup Job object. Select the Network Share the policy is attached to and open the Backup Jobs tab in the center pane to watch progress, or use the Backup Policies & Jobs tab under Schedules to see jobs across all policies.

With Maintain a log of each backup enabled, each job writes a log on the QuantaStor system under: 

/var/log/qs/backup-log/POLICY-NAME/

The backup phase writes a .changelog file and the purge phase a .purgelog file. Each log lists the full path of every file the job handled, which makes them usable as input to tape backup software such as IBM TSM.

Managing Backup Policies from the CLI

Every operation above has a CLI equivalent, which is the practical way to create policies in bulk. The full argument list for each is in the QuantaStor CLI Command Reference.

Command Purpose
qs backup-policy-create Create a policy
qs backup-policy-modify Change an existing policy
qs backup-policy-list List policies
qs backup-policy-get Show one policy in detail
qs backup-policy-trigger Start a backup job now
qs backup-policy-enable Resume a disabled policy
qs backup-policy-disable Suspend a policy
qs backup-policy-delete Delete a policy
qs backup-job-list List backup jobs
qs backup-job-get Show one job
qs backup-job-cancel Cancel a running job

A minimal inbound policy pulling an NFS export onto an existing share: 

qs backup-policy-create --name=nightly-archive --network-share=archive \
  --remote-hostname=10.0.10.50 --remote-export-type=nfs \
  --remote-export-path=/export/projects --policy-type=copy-inbound

The same for an SMB source, with credentials passed as separate arguments: 

qs backup-policy-create --name=filer-backup --network-share=archive \
  --remote-hostname=10.0.10.60 --remote-export-type=smb \
  --smb-username=DOMAIN/backupsvc --smb-password=aAbBcCdDeEfF0123 \
  --policy-type=copy-inbound --purge-policy=daily --retain-period=60

Note that the CLI and the web interface do not share every default. --scan-threads defaults to 5 from the CLI while the dialog defaults to 12 streams, --remote-export-type defaults to nfs while the dialog pre-selects SMB, and --policy-type defaults to copy-inbound in both. Set the values you want explicitly in scripts rather than relying on either default.

Data Mover Utility / pwalk

The scanning and copying is performed by pwalk, an open source command line utility extended by OSNEXUS and shipped with QuantaStor. It is what makes the parallel scan possible, and it can be used directly for ad-hoc copies and scans -- see the pwalk page.

Related pages

Verified against QuantaStor 6.9.0.

Retrieved from "https://wiki.osnexus.com/index.php?title=Backup_Policies&oldid=27474"