Backup Policies
Contents
Backup Policy Management
Within QuantaStor you can create backup policies where data from any NFS or CIFS share on your network can be automatically backed up for you to your QuantaStor appliance. To create a backup policy simply right-click on the Network Share where you want the data to be backed up to and choose the 'Create Backup Policy..' option from the pop-up menu. Backup policies will do a CIFS/NFS mount of the specified NAS share on your network locally to the appliance in order to access the data to be archived. When the backup starts it creates a Backup Job object which you will see in the web interface and you can see the progress of any given backup job by monitoring it in the Backup Jobs tab in the center-pane of the web interface after you select the Network Share to which the backup policy is attached.
Creating Backup Policies
Backup policies in QuantaStor support heavy parallelism so that very large NAS filers with 100m+ files can be easily scanned for changes. The default level of parallelism is 32 concurrent scan+copy threads but this can be reduced or increased to 64 concurrent threads.
This is where you indicate where you want the data to be backed up to on your QuantaStor appliance. With QuantaStor backup policies your data is copied from a NAS share on the network to a network share on your QuantaStor appliance.
Policy Name
This is a friendly name for your backup policy. If you are going to have multiple policies doing backups to the same network share then each policy will be associated with a directory with the name of the policy. For example, if your share is called media-backups and you have a policy called 'project1' and a policy called 'project2' then there will be sub-directories under the media-backups share for project1 and project2. In order to support multiple policies per Network Share you must select the option which says Backup files to policy specific subdirectory. If that is not selected then only one policy can be associated with the network share and the backups will go into the root of the share to form an exact mirror copy.
Selecting the Backup Source
In the section which says Hostname / IP Address: enter the IP address of the NAS filer or server which is sharing the NAS folder you want to backup. For NFS shares you should enter the IP address and press the Scan button. If NFS shares are found they'll show up in the CIFS/NFS Export: list. For CIFS share backups you'll need to enter the network path to the share in a special format starting with double forward slashes like so: //username%password@ipaddress. For example, you might scan for shares on a filer located at 10.10.5.5 using the SMB credentials of 'admin' and password 'password123' using this path: //admin%password123@10.10.5.5. In AD environments you can also include the domain in the SMB path like so //DOMAIN/username%password@ipaddress.
Policy Type
You can indicate that you want the backup policy to backup everything by selecting 'Backup All Files' or you can do a 'Sliding Window Backup'. For backing up data from huge filers with 100m+ files it is sometimes useful to only backup and maintain a sliding window of the most recently modified or created files. If you set the Retention Period to 60 days then all files that have been created or modified within the last 60 days will be retained. Files that are older than that will be purged from the backup folder.
Be careful with the Backup All Files mode. If you have a Purge Policy enabled it will remove any files from the network share which were not found on the source NAS share that's being backed up. If you attached such a backup policy to an existing share which has data on it, the purge policy will remove any data/files that exists in your QuantaStor Network Share which is not on the source NAS share on the remote filer. So use caution with this as Backup All Files really means maintain a mirror copy of the remote NAS share.
Purge Policy
Backup policies may run many times per day to quickly backup new and modified files. A scan to determine what needs purging is typically less important so it is more efficient to run it nightly rather than with each and every backup job. For the Sliding Window policies the purge phase will throw out any files that are older than the retention period. For the Backup All Files policies there is a comparison that is done and any files that are no longer present in the NAS source share are removed from the backup. The Purge Policy can also be set to 'Never delete files' which will backup files to your Network Share but never remove them.
Backup Logs
If you select 'Maintain a log of each backup' then a backup log file will be written out after each backup. Backup logs can be found on your QuantaStor appliance in the /var/log/backup-log/POLICY-NAME directory. The purge process produces a log with the .purgelog suffix and the backup process produces a log with the .changelog suffix.
pwalk
pwalk is a open source command line utility included with QuantaStor (see /usr/bin/pwalk). It was originally written by John Dey to work as a parallelized version of the 'du -a' unix utility which would be suitable for scanning filesystems with 100s of millions of files. It was then reworked at OSNEXUS to support backups, sliding window backups, additional output formats, etc. If you type 'pwalk' by itself at the QuantaStor ssh or console window you'll see the following usage page / documentation. The pwalk utility has three modes, 'walk' which does a parallelized crawl of a directory, 'copy' which does a backup from a SOURCEDIR to a specified --targetdir, and 'purge' mode which removes files in the PURGEDIR which are not found in the --comparedir. In general you would never need to use pwalk directly but the documentation is provided here to support special use cases like custom backup or replication cron jobs.
pwalk version 3.1 Oct 22nd 2013 - John F Dey john@fuzzdog.com, OSNEXUS, eng@osnexus.com
Usage :
pwalk --help --version
Common Args :
--dryrun : use this to test commands
without making any changes to the system
--maxthreads=N : indicates the number of threads (default=32)
--nototals : disables printing of totals after the scan
--dots : prints a dot and total every 1000 files scanned.
--quiet : no chatter, speeds up the scan.
--nosnap : Ignore directories with name .snapshot
--debug : Verbose debug spam
Output Format : CSV
Fields : DateStamp,"inode","filename","fileExtension","UID",
"GID","st_size","st_blocks","st_mode","atime",
"mtime","ctime","File Count","Directory Size"
Walk Usage :
pwalk SOURCEDIR
Command Args :
SOURCEDIR : Fully qualified path to the directory to walk
Copy/Backup Usage :
pwalk --targetdir=TARGETDIR SOURCEDIR
pwalk --retain=30 --targetdir=TARGETDIR SOURCEDIR
Command Args :
--targetdir : copy files to specified TARGETDIR
--atime : copy if access time change (default=no atime)
--backuplog=LOGFILE : log all files that were copied.
--status=STATUSFILE : write periodic status updates to specified file
--retain : copy if file ctime or mtime within retention period
specified in days. eg: --retain=60
--nomtime : ignore mtime (default=use mtime)
SOURCEDIR : Fully qualified path to the directory to walk
Delete/Purge Usage :
pwalk --purge [--force] --comparedir=COMPAREDIR PURGEDIR
pwalk --purge [--force] --retain=N PURGEDIR
Command Args :
--comparedir : compare against this dir but dont touch any files
in it. comparedir is usually the SOURCEDIR from
a prior copy/sync stage.
--purge : !!WARNING!! this deletes files older than the
retain period -OR- if retain is not specified
--comparedir is required. The comparedir is
compared against the specified dir and any files
not found in the comparedir are purged.
--force : !NOTE! default is a *dry-run* for purge, you must
specify --force option to actually purge files
--atime : keep if access time within retain period
--retain : keep if file ctime or mtime within retention period
specified in days. eg: --retain=60
OSNEXUS modified version of the C source code for pwalk is available here pwalk.c. The original version is available here.
