Recovery Manager
The Recovery Manager creates, downloads, uploads and restores configuration bundles. A configuration bundle is a single .tgz file holding a Storage System's internal configuration database (osn.db) together with the configuration files around it: encryption keys, NFS and SMB settings, Ceph keyrings, network customizations, certificates, time settings and more. Use a bundle to bring a system back after its boot device is replaced or its OS is reinstalled, or to pass the identity and configuration of a retired server to the new server that replaces it.
The Recovery Manager is also on the right-click menu of a Storage System. Every operation in it is also available from the command line.
| Section | Purpose |
|---|---|
| Configuration Bundles | Generate, download, upload and delete bundles |
| Generate and download a bundle | Step by step: take a bundle of any grid member and save it off the system |
| Restore Options | Choose which categories a restore writes, and what each one holds |
| Upload and restore a bundle | Step by step: upload a saved bundle and restore all of it or selected categories |
| What happens during a restore | The checks, the pre-restore backup, the service restart and rollback |
| Roll back a restore | Undo a restore with the pre-restore bundle |
| Command line | The qs config-bundle-* commands and a reinstall walk-through
|
| Cautions | Node identity, network changes and maintenance windows |
| Pool database backups | The hourly database copies kept on each Storage Pool |
Your data does not depend on a bundle. Storage Pools, and the volumes and shares within them, carry their own metadata and come back when the pool is imported. The configuration database holds the logical objects layered on top -- hosts, users, schedules, share and volume settings -- which is why a pool imported into a freshly installed system comes back with its volumes intact but without the objects that referenced them. A bundle restores those, plus the files the database does not hold, such as the encryption keys an encrypted pool needs to unlock.
Bundles are stored on the system's boot device, in /var/opt/osnexus/quantastor/configbundles. A bundle left there is lost with the boot device it is meant to recover from, so download every bundle you intend to keep and store it somewhere else. QuantaStor creates bundles only when you ask for one, so generate a new bundle after significant configuration changes.
Configuration Bundles
The Configuration Bundles tab works on one Storage System at a time. Choose it in the Storage System list; the list is preselected when you open the dialog from a Storage System's right-click menu. The table below it lists the bundles on that system, newest first:
| Column | Shows |
|---|---|
| Bundle | The bundle's file name. |
| Type | generated for a bundle you created, prerestore for the backup QuantaStor takes before every restore, legacy for a bundle from an older release that has no manifest, and invalid for a file that failed validation and cannot be restored. |
| Files | The number of configuration files in the bundle. |
| Size | The size of the bundle file. |
| Created | When the bundle was created. |
The buttons above the table act on the selected Storage System or on the bundle you tick in the table:
| Button | What it does |
|---|---|
| Generate Bundle | Captures every category of the current configuration into a new bundle named qs_config_bundle_YYYYMMDD-HHMMSS.tgz.
|
| Restore | Restores the ticked bundle, using the categories selected on the Restore Options tab. A confirmation names the bundle and the system. See What happens during a restore. |
| Download | Saves the ticked bundle to your computer through the browser. |
| Delete | Deletes the ticked bundle from the system after a confirmation. |
| Refresh | Reloads the list. |
| Upload Bundle... | Uploads a bundle file from your computer to the selected system so that you can restore it. The bundle keeps its file name, with -2, -3 and so on added if a bundle of that name is already there. A file whose name is not a bundle name is stored as qs_config_bundle_YYYYMMDD-HHMMSS_imported.tgz.
|
The dialog's own button is Close; nothing happens until you press one of the buttons on this tab.
Uploads and downloads are limited to 100 MB per bundle. A bundle is validated when it is uploaded and again before it is restored: it must be a readable .tgz, every file must match the checksum recorded in its manifest, and it may only contain files that a restore is allowed to write. Anything else in the file is refused, and a bundle that fails validation shows as invalid and cannot be restored.
A bundle holds secrets. It contains the configuration database and the encryption keys and salts of encrypted pools. Downloading a bundle therefore needs the same permission as modifying the Storage System, and you should store a downloaded bundle as carefully as the keys themselves.
Generate and download a bundle
A bundle is always of one node: it captures that node's database and configuration files, and it is stored on that node's boot device. You do not need to log in to each node to take one. The Storage System list offers every member of the grid, and whichever node you are logged in to passes the request on to the system you choose, so you can generate and download the bundles of the whole grid from one browser session.
- Open the Recovery Manager from any node in the grid.
- In the Storage System list, choose the node to back up. Opened from a Storage System's right-click menu, the list starts on that system.
- Click Generate Bundle. The new bundle appears at the top of the table as type generated, with every category captured.
- Tick the bundle in the table and click Download. The browser saves it under the bundle's own file name,
qs_config_bundle_YYYYMMDD-HHMMSS.tgz, in its usual download location. - Move the file to storage that does not depend on this system -- not one of its own Storage Pools or shares, since those may be unavailable when you need the bundle.
- Repeat from step 2 for each node you want to protect. Rename the downloaded files if you need to tell nodes apart later; an upload keeps any file name that is a bundle name, and stores any other name as an imported bundle.
The file name records when the bundle was taken but not which node it came from, which is why renaming the downloads matters on a grid with several members. A download needs the Storage System modify permission, because the bundle carries the node's encryption keys.
Restore Options
The Restore Options tab chooses which parts of the bundle a restore writes. Every category is ticked when the dialog opens, which restores everything the bundle holds. Select All and Clear All tick or clear the whole list; at least one category must be ticked to restore.
Restoring a subset is useful when only part of a system needs to go back -- for example the SMB configuration after a bad edit, or the encryption keys of a reinstalled node -- without replacing the rest of its configuration.
| Restore Options label | CLI name | What it captures and restores |
|---|---|---|
| Grid database (osn.db) | database |
The QuantaStor configuration database: the node's identity and its logical objects. |
| Encryption (crypttab, keys, salts) | encryption |
/etc/crypttab, and the keys, salts and pool metadata of software- and SED-encrypted pools on ZFS and Ceph. LUKS header backups are left out to keep bundles small.
|
| NFS exports and NFS-Ganesha | nfs |
Custom export rules in /etc/exports.custom, and the NFS-Ganesha configuration used for CephFS and bucket shares.
|
| SMB / Samba | smb |
/etc/samba/smb.conf and /etc/samba/smb.globals.
|
| Kerberos / directory services | directory |
/etc/krb5.conf and /etc/krb5.default.
|
| Ceph configuration and keyrings | ceph |
Everything in /etc/ceph.
|
| Static routes, port filters, proxy and firewall | network |
Per-port static routes, port filtering and lossless networking settings, proxy configuration and firewall rule customizations. |
| Multipath and udev rules | storagedevices |
/etc/multipath.conf and the NVMe-oF udev rules.
|
| SSL certificates and system use notice | certificates |
The QuantaStor SSL certificates and the system use notification text. |
| Load balancer (haproxy, nginx) | loadbalancer |
Custom files in /etc/haproxy/conf.d and /etc/nginx/conf.d.
|
| Conf overrides, touch-files, alert filters, support and OEM files | system |
Override files in /var/opt/osnexus/quantastor/conf, touch-files, alert filters, support contact information and OEM branding files.
|
| Time zone and NTP | time |
The time zone and the chrony or ntpd configuration. |
The full list of captured paths is in /opt/osnexus/quantastor/conf/qs_config_exporter.conf on the appliance. A restore only ever writes paths that this file covers. Performance metrics (Telegraf and InfluxDB data) are not captured; they are far too large for a bundle.
If you request a category the bundle has nothing for -- encryption on a system without encrypted pools, for example -- that category is skipped and the restore carries on; the restore task's description lists the categories it skipped. A bundle taken on an older release stays restorable: files it holds that the current release no longer covers are skipped and logged rather than written.
Put lasting SMB global options in smb.globals. QuantaStor rewrites smb.conf from its database whenever a share changes, so hand edits to its [global] section do not last. Options in /etc/samba/smb.globals are appended to [global] every time, and the bundle captures that file too.
Upload and restore a bundle
To restore a bundle you downloaded earlier, upload it to the node it belongs to and restore it there. The selected node is always the one the restore changes.
- Open the Recovery Manager and choose the node to restore in the Storage System list. After a reinstall, this is the freshly installed system.
- Click Upload Bundle... and pick the
.tgzfile. The upload starts as soon as you choose the file. A file over 100 MB is refused before anything is sent. When the upload has been validated, the bundle appears in the table. - Tick the bundle. If its type is invalid, it cannot be restored; the error message says why.
- Open the Restore Options tab. To restore everything, leave every category ticked. To restore selectively, click Clear All and tick only the categories you want -- for example SMB / Samba alone to undo a bad SMB change, or Encryption (crypttab, keys, salts) alone to unlock encrypted pools on a reinstalled node.
- Return to the Configuration Bundles tab, check that the bundle is still ticked, and click Restore.
- Confirm. The confirmation names the bundle and the system it is about to change. The dialog closes and the restore runs as a Restore Configuration Bundle task.
- Wait for the service restart, reload the browser, then check for the System Configuration Restore Completed alert. Reboot the system when it is convenient.
The pre-restore backup in step 6 is taken automatically: before writing anything, QuantaStor generates a bundle of the complete current configuration, every category regardless of which ones you ticked. It appears in the table as type prerestore, and it is what you restore to undo the change -- see Roll back a restore.
When the database is part of the restore, QuantaStor raises a System Configuration Rollback Started alert naming the bundle being restored and the pre-restore bundle it took. Whenever the restore needs the service to restart, a Storage System Core Services Restarting alert follows; the service restarts within about a minute and the web interface disconnects until it is back.
What happens during a restore
- Checks. The bundle is validated. If the database is being restored, it must pass an integrity check and must not belong to another member of the grid the system is in, because two grid nodes with the same ID break the grid. A restore is refused while an earlier one is still being applied.
- Pre-restore backup. QuantaStor generates a bundle of the complete current configuration, named
..._prerestore.tgz, before it changes anything. Restore that bundle to undo the restore. - Files. The selected categories are written. Files that already match the bundle are left alone. The time service is restarted if its configuration changed.
- Database and service restart. The database is staged, and within about a minute the QuantaStor service stops, swaps in the restored database and starts again. The previous database is kept as
osn.db.prerestore. Reload the browser once the service is back. - Result. A System Configuration Restore Completed alert confirms success. If the service does not come up on the restored database within ten minutes, or comes up on a different database, QuantaStor puts the previous database back, keeps the rejected one as
osn.db.failedrestoreand raises System Configuration Restore Failed. The other configuration files stay restored in that case; restore the pre-restore bundle to undo them. See Alert Manager for alert delivery.
A reboot of the system is recommended once the restore completes, so that every restored setting takes effect.
When the database is restored, QuantaStor re-creates the Samba users from it. If the SMB category was restored too, the restored smb.conf is kept rather than regenerated. If the network category was restored with the database, QuantaStor also reapplies the network configuration stored in the database -- see the cautions below.
Roll back a restore
Every restore leaves a prerestore bundle behind, holding the full configuration as it was just before the restore. To undo the restore:
- Open the Recovery Manager on the same system and find the prerestore bundle created at the time of the restore. Its name ends in
_prerestore.tgz. - On the Restore Options tab, leave every category ticked. The pre-restore bundle holds every category, so a full restore of it puts back everything the earlier restore changed, whichever categories that restore used.
- Tick the bundle on the Configuration Bundles tab, click Restore and confirm.
Rolling back is itself a restore, so it takes a pre-restore bundle of its own first. If you roll back by mistake, restore that newer bundle.
The pre-restore bundle is stored on the boot device like any other bundle. Download it too if you may need it after the boot device is replaced. Delete pre-restore bundles you no longer need; QuantaStor keeps them until you do.
Command line
Six qs commands drive the same operations as the dialog. Each takes an optional --storage-system argument naming the system to act on, so you can manage the bundles of any grid member from one node; without it, the command acts on the system you are connected to.
| Command | Short form | Purpose |
|---|---|---|
qs config-bundle-generate [--categories=<list>] |
cb-generate |
Create a bundle. |
qs config-bundle-list |
cb-list |
List the bundles on a system. |
qs config-bundle-export --config-bundle=<bundle> [--output-file=<path>] |
cb-export |
Download a bundle to a local file. |
qs config-bundle-import --input-file=<path> |
cb-import |
Upload a local bundle file to a system. |
qs config-bundle-restore --config-bundle=<bundle> [--categories=<list>] |
cb-restore |
Restore a bundle. |
qs config-bundle-delete --config-bundle=<bundle> |
cb-delete |
Delete a bundle. |
--config-bundle takes the bundle's file name or ID, as shown by config-bundle-list. config-bundle-export writes the file readable by your user only; without --output-file it saves it under the bundle's own name in the current directory. config-bundle-import reads the file on the machine where you run the command and uploads it.
--categories takes the CLI names from the Restore Options table, comma separated. Leave it out, or pass all, for every category. A plain list selects only those categories; a name prefixed with ~ leaves that category out of the full set:
qs config-bundle-restore --config-bundle=qs_config_bundle_20261102-091500.tgz --categories=encryption,smb qs config-bundle-restore --config-bundle=qs_config_bundle_20261102-091500.tgz --categories=~network
The first restores only the encryption keys and the SMB configuration; the second restores everything except the network files.
Recovering a reinstalled system
Generate and download a bundle while the system is healthy:
qs config-bundle-generate --storage-system=qs-node1 qs config-bundle-list --storage-system=qs-node1 qs config-bundle-export --storage-system=qs-node1 --config-bundle=qs_config_bundle_20261102-091500.tgz --output-file=/root/qs-node1-bundle.tgz
Copy the file off the system. After the boot device is replaced and QuantaStor reinstalled, copy the file back to the system and restore it:
qs config-bundle-import --input-file=/root/qs-node1-bundle.tgz qs config-bundle-list qs config-bundle-restore --config-bundle=qs-node1-bundle.tgz
The uploaded file's name is not a bundle name in this example, so config-bundle-import stores it as qs_config_bundle_YYYYMMDD-HHMMSS_imported.tgz; use the name config-bundle-list shows. Watch for the System Configuration Restore Completed alert, then reboot.
Cautions
Restore a node's bundle only onto that node or onto the server replacing it. The database carries the node's unique ID. QuantaStor refuses a database that belongs to another member of the same grid, but it cannot see a retired server that is still running elsewhere -- power the old server off, or remove it from the grid, before its replacement takes over its identity.
Network configuration recovery can cut off your own session. If the restored network settings differ from the current ones, your connection drops when the address changes. Where ports may become unreachable, run the restore from the console, or untick the network category.
The restore replaces the running configuration and restarts the QuantaStor service. Schedule it as maintenance. If you are unsure which bundle is right, contact OSNEXUS support at support@osnexus.com before restoring rather than working through them by trial.
Pool database backups
Separately from configuration bundles, QuantaStor copies its configuration database to every online Storage Pool, keeping rotating hourly and daily copies in a .qsbackups directory in the pool. The number of copies of each is set by max_db_backups in /opt/osnexus/quantastor/conf/quantastor.conf (default 5, maximum 100; 0 turns pool backups off). A backup failure raises a Configuration DB Backup Failure alert.
These copies hold the database only, not the encryption keys or other files a bundle carries, and the Recovery Manager does not list them. They are a last resort for a system that has no bundle. To use one, stop the service, move the live database and its -wal and -shm files aside, copy the backup into place and start the service:
systemctl stop quantastor cd /var/opt/osnexus/quantastor mv osn.db osn.db.old; mv -f osn.db-wal osn.db-wal.old; mv -f osn.db-shm osn.db-shm.old cp <backup-file> osn.db systemctl start quantastor
We recommend contacting OSNEXUS support before restoring a database this way.
Related pages
- Storage System -- system-level configuration and the Modify dialog
- Upgrade Manager -- includes a re-install / boot drive swap procedure
- Alert Manager -- restore and backup alerts
- Software Encryption -- encrypted pools, whose keys a bundle carries
- QuantaStor Touch Files -- the touch-files captured by the system category
Verified against QuantaStor 7.0.0.