Storage System Optimization

The Storage System Optimization dialog allows one to adjust system tunings (tunables) that control cache behavior and Storage Pool I/O, so performance can be more closely matched to a specific hardware configuration and workload.

Navigation: Storage Management → Storage Systems → right-click a Storage System → Storage System Optimization...

Note this dialog is reached from the right-click context menu on a Storage System, not from the toolbar. Note also that most of the tuning settings (all those based on zfs params) only effect Scale-up (ZFS-based) pools and not Scale-out (Ceph-based) pools.

Every setting is applied live to the running system and most settings do not require a reboot (driver changes to the SPL excepted). All changes are persisted so they survive reboots and upgrades. Values are per Storage System, so in a grid each system is tuned independently. The Storage System selector at the top of the dialog switches which system you are editing. To apply a common configuration to each system save your custom configuration as a Optimization Profile first, then apply it to other systems.

The six tabs group the tunables by what they affect:

• General Settings -- resilver, replication prefetch, cleanup, metaslabs, read-ahead (11 settings)
• Cache Settings -- ARC size, write throttle and flush behaviour, compression, prefetch (7)
• Pool Settings -- per-device queue depths, I/O aggregation, scrub queue (9)
• Network Settings -- TX/RX queue length and device backlog (2)
• Volume Settings -- Storage Volume sync I/O mode (1)
• Driver Settings -- SPL kernel memory cache (3)

See also Optimization Profiles and Extending the configuration files.

Before you change anything

The shipped defaults are appropriate for the large majority of deployments. The tunables here are the ones that matter for specific, identifiable problems (resilver taking too long, write latency spikes under load, a replication window overrunning). Changing them speculatively is more likely to reduce performance and create problems rather than help so we recommend any changes here be done with the assistance of the support team.

Recommended approach:

• Establish a baseline first. Use the Performance and Cache Stats (ARC) views on the Storage System dashboard to see what the system is actually doing before you change anything.
• Change one group of related settings at a time, then measure again under a representative workload.
• Save a named profile before you start experimenting, so you can get back to a known state -- see Optimization Profiles.
• Revert All returns the dialog to the values currently stored in the database, which is the quickest way out of a half-finished experiment. Note it reverts the workspace -- you still need OK or Apply to make that stick.

If you are unsure whether a setting applies to your situation, contact OSNEXUS support rather than guessing; several of these interact.

Optimization Profiles

A profile is a named collection of tunable values that can be applied to a Storage System in one step. Profiles make it practical to keep a tuning recipe consistent across a fleet, and to get back to a known configuration.

The Optimization Profiles controls at the top of the dialog are:

• Apply Profile -- copies the selected profile's values into the dialog. Nothing is committed until you press OK or Apply, so you can review what a profile will change before accepting it.
• Save Profile... -- saves the values currently in the dialog as a new named profile.
• Delete Profile -- deletes the selected profile. This only removes the profile; it does not change the settings of any system that had it applied.
• Revert All -- resets the dialog back to the values stored in the database for this system.

Two profiles ship with QuantaStor:

• Default -- every tunable at its shipped default. Applying it resets the whole system optimization configuration. This profile is regenerated from the current defaults each time the QuantaStor service starts, so it always reflects the running release, including tunables added in a newer version.
• Seagate Exos Protect Optimized -- tuning for large scale-up deployments layered over Seagate ADAPT hardware RAID.

Profiles can also be managed from the CLI -- see the QuantaStor CLI Command Reference for each command's full argument list:

qs tunable-profile-list
qs tunable-profile-get --tunable-profile=<name>
qs tunable-profile-create --name=<name> ...
qs tunable-profile-modify --tunable-profile=<name> ...
qs tunable-profile-apply --tunable-profile=<name> --storage-system=<system>
qs tunable-profile-delete --tunable-profile=<name>

Individual tunables can be read and set directly. Note tunable-set takes the value as part of --tunable in key:value form -- there is no separate --value argument -- and accepts a comma separated list to set several at once:

qs tunable-list
qs tunable-get --tunable=<sst_name>
qs tunable-set --tunable=<sst_name>:<value>
qs tunable-set --tunable=sst_cache_size:80,sst_write_flush_rate_sec:5

tunable-set also takes --tunable-option=reset to return tunables to their defaults, and --storage-system to target a system other than the local one.

Settings Reference

The dialog groups the tunables into six tabs. The kernel parameter each setting writes is given in parentheses in its description, or listed under each tab.

General Settings

Storage System Optimization, General Settings tab.

Setting	Range	Default	What it does
Resilver Priority (msec/TXG)	500 – 8000	3000	Higher settings indicates more time should be dedicated to resilvering between transaction groups (zfs_resilver_min_time_ms).
Remote Replication Prefetch (MB)	4 – 128 MB	50 MB	Improves performance of replication send operations by prefetching data from disk (units in MB). Reducing this may reduce latency for systems replicating during production hours. (zfs_pd_bytes_max)
Cleanup Priority (% of TXG)	5 – 90	30	Controls the maximum amount of dirty blocks to be freed as a percentage of each transaction group. Increase to give high priority to cleanup operations like file deletion.
Limit Active Async Writes	5 – 50	30	Limit active asynchronous writes based on dirty percent. This can help with write latency and resilver performance when set to a value between 10 and 30.
Transaction Group (TXG) Commit Timeout	5 – 90	10	Percentage of dirty data at which the transaction group commit timeout is reduced to 1 second to speed up commits and reduce latency. Default is 10% which means that once the dirty data reaches 10% of RAM the system will start aggressively committing transaction groups to keep latency low.
Pool Metaslab Shift	28 – 38	34	Controls the size of metaslabs used for space allocation. Larger metaslabs can improve performance on large pools but may lead to fragmentation and reduced performance over time if set too high.
Pool Metaslab Count Limit	1024 – 262144	131072	Limits the number of metaslabs that can be allocated for a single IO operation. Increasing this can improve performance for large IO operations on large pools.
Pool Metaslab LBA Weighting	on / off	on	Enabling LBA weighting allows the allocator to prefer lower LBAs for better performance. This can be beneficial for some workloads but may lead to increased fragmentation over time.
Metaslab Aliquot	64 – 32768 KB	1024 KB	Controls the size of metaslab aliquots used for space allocation. Larger aliquots can improve performance on large pools but may lead to fragmentation and reduced performance over time if set too high.
Read-Ahead Distance Max (MB)	8 – 512 MB	64 MB	Maximum distance in bytes that the ZFS prefetcher will read ahead. Increasing this can improve performance for sequential read workloads.
Read-Ahead Distance Min (MB)	0 – 128 MB	4 MB	Minimum distance in bytes that the ZFS prefetcher will read ahead. Reducing this can improve performance for workloads with smaller sequential read patterns.

Kernel parameters set by this tab:

/sys/module/zfs/parameters/zfs_resilver_min_time_ms
/sys/module/zfs/parameters/zfs_pd_bytes_max
/sys/module/zfs/parameters/zfs_per_txg_dirty_frees_percent
/sys/module/zfs/parameters/zfs_vdev_async_write_active_min_dirty_percent
/sys/module/zfs/parameters/zfs_commit_timeout_pct
/sys/module/zfs/parameters/zfs_vdev_max_ms_shift
/sys/module/zfs/parameters/zfs_vdev_ms_count_limit
/sys/module/zfs/parameters/metaslab_lba_weighting_enabled
/sys/module/zfs/parameters/metaslab_aliquot
/sys/module/zfs/parameters/zfetch_max_distance
/sys/module/zfs/parameters/zfetch_min_distance

Cache Settings

Storage System Optimization, Cache Settings tab.

Setting	Range	Default	What it does
Cache Size (% of RAM)	30 – 90	70	Sets the percentage of the system's RAM available as in-memory read cache for use by all Storage Pools.
Write Throttle Limit (% of RAM)	3 – 20	10	Percentage of system RAM allocated for dirty data which after exeeded halts further writes until enough has flushed to free up more space in RAM. (controls zfs_dirty_data_max).
Write Flush Threshold (% of Write Queue RAM)	5 – 80	20	Percentage of write queue filled at which transaction group syncing is ensured. Reducing this can improve IOPS.(zfs_dirty_data_sync_percent)
Write Flush Threshold (sec)	1 – 10	5	Once the threshold has been reached a new transaction group will be started. Reducing this can improve IOPS.(zfs_txg_timeout)
Async Write - Min Active I/Os	1 – 5	2	Minimum asynchronous write I/Os active to each VDEV (zfs_vdev_async_write_min_active). Lower values generally improve latency on rotational media and hurt resilver performance.
Cache Compression	on / off	on	Compresses ARC data to increase the effective size of the in-memory read cache for Storage Pools.(zfs_compressed_arc_enabled)
Prefetch Disable	on / off	off	In cases where the IO patterns is predominantly random reads performance may be improved by disabling prefetch.(zfs_prefetch_disable)

Kernel parameters set by this tab:

/sys/module/zfs/parameters/zfs_arc_max
/sys/module/zfs/parameters/zfs_dirty_data_max
/sys/module/zfs/parameters/zfs_dirty_data_sync_percent
/sys/module/zfs/parameters/zfs_txg_timeout
/sys/module/zfs/parameters/zfs_vdev_async_write_min_active
/sys/module/zfs/parameters/zfs_compressed_arc_enabled
/sys/module/zfs/parameters/zfs_prefetch_disable

Pool Settings

Storage System Optimization, Pool Settings tab.

Setting	Range	Default	What it does
Aggregate Per-Device Active I/O Limit	256 – 4096	2000	Maximum number of IO operations active per VDEV. Acts as a global cap of the sum of all sync/async read/write IOs and scrub/resilver classes of IO.
Read Queue Depth (sync I/O)	4 – 128	32	Maximum number of synchronous read IO operations active per VDEV. (zfs_vdev_sync_read_max_active)
Write Queue Depth (sync I/O)	4 – 128	32	Maximum number of synchronous direct write IO operations active per VDEV. (zfs_vdev_sync_write_max_active)
Read Queue Depth (async I/O)	4 – 128	32	Maximum number of asynchronous read IO operations active per VDEV. (zfs_vdev_async_read_max_active)
Write Queue Depth (async I/O)	2 – 128	32	Maximum asynchronous write I/Os active to each VDEV (zfs_vdev_async_write_max_active). Reducing max-active can improve latency under contention, but it can also lower throughput.
I/O Aggregation Limit (KB)	128 – 32768 KB	128 KB	Sets the VDEV upper bound on I/O coalescence/aggregation for a stripe of data. Increasing this value to 1M or larger may increase throughput for sequential I/O workloads. (zfs_vdev_aggregation_limit)
Scrub Queue Depth - Max Active I/Os	1 – 32	3	Maximum scrub or scan read I/Os active to each VDEV (zfs_vdev_scrub_max_active). Increasing this can speed up scrub and scan operations, but may increase impact on client workloads.
Scrub Queue Depth - Min Active I/Os	1 – 8	1	Minimum scrub or scan read I/Os active to each VDEV (zfs_vdev_scrub_min_active). Lower values reduce impact on production I/O, while higher values can improve scrub progress when the pool is busy.
I/O Allocator Default Queue Depth	2 – 1024	32	Default queue depth for each VDEV IO allocator. Higher values allow for better coalescing of sequential writes before sending them to the disk, but can increase transaction commit times. (zfs_vdev_def_queue_depth)

Kernel parameters set by this tab:

/sys/module/zfs/parameters/zfs_vdev_max_active
/sys/module/zfs/parameters/zfs_vdev_sync_read_max_active
/sys/module/zfs/parameters/zfs_vdev_sync_write_max_active
/sys/module/zfs/parameters/zfs_vdev_async_read_max_active
/sys/module/zfs/parameters/zfs_vdev_async_write_max_active
/sys/module/zfs/parameters/zfs_vdev_aggregation_limit
/sys/module/zfs/parameters/zfs_vdev_scrub_max_active
/sys/module/zfs/parameters/zfs_vdev_scrub_min_active
/sys/module/zfs/parameters/zfs_vdev_def_queue_depth

Network Settings

Storage System Optimization, Network Settings tab.

Setting	Range	Default	What it does
Network TX/RX Queue Length	1000 – 20000	5000	Sets the transmit and receive network queue length for high-speed network ports (10GbE and faster). Values like 5000 and higher are recommended for systems with 10GbE and faster networking. (txqueuelen)
Network Device Max Backlog	128 – 1000000	1000	Maximum number of packets allowed to queue on the input side when the interface receives data faster than the kernel can process it. (netdev_max_backlog)

Kernel parameters set by this tab:

ip.link.txqueuelen
/proc/sys/net/core/netdev_max_backlog

Volume Settings

Storage System Optimization, Volume Settings tab.

Setting	Range	Default	What it does
Storage Volume sync IO mode	on / off	on	Storage Volume (ZVOL) sync mode can be ON(1) or OFF(0), default is ON.

Kernel parameters set by this tab:

/sys/module/zfs/parameters/zvol_request_sync

Driver Settings

Storage System Optimization, Driver Settings tab.

Setting	Range	Default	What it does
KMEM Cache Threads	1 – 16	4	Number of threads used for KMEM cache operations. Increasing this can improve performance for workloads with high levels of concurrent allocations and frees.
KMEM Cache Objects Per Slab	8 – 128	8	Number of objects per slab in the KMEM cache. Increasing this can improve performance for workloads with many small allocations.
KMEM Cache Max Size (MB)	32 – 2048 MB	32 MB	Maximum size of the KMEM cache in megabytes. Increasing this can improve performance for workloads with high memory usage.

Kernel parameters set by this tab:

/sys/module/spl/parameters/spl_kmem_cache_kmem_threads
/sys/module/spl/parameters/spl_kmem_cache_obj_per_slab
/sys/module/spl/parameters/spl_kmem_cache_max_size

Deprecated tunables

Two tunables have been retired and no longer appear in the dialog:

• sst_resilver_prio -- superseded by Resilver Priority (msec/TXG)
• sst_write_flush_rate_mb -- superseded by Write Flush Threshold (% of Write Queue RAM)

They are still present in the configuration file marked param=deprecated so that older saved profiles referencing them load without error. Do not use them in new profiles.

Extending and customizing the configuration files

The tunable set and the shipped profiles are both driven by plain text configuration files on each Storage System, so a site can add tunables that QuantaStor does not expose out of the box and define its own profiles.

Both files live in:

/opt/osnexus/quantastor/conf/

These files are replaced on upgrade. Keep a copy of any local additions outside that directory and re-apply them after upgrading, or the changes will be lost. In both files a line is treated as a comment only when # is the first character of the line -- an indented # is not a comment.

Changes take effect when the QuantaStor service restarts:

systemctl restart quantastor

qs_systemtunables.conf -- defining what appears in the dialog

Each tunable is one section. The section name is the tunable's identifier and must begin with sst_. A complete example:

[sst_cache_size]
tab=cache_settings
param=/sys/module/zfs/parameters/zfs_arc_max
title="Cache Size (% of RAM)"
desc="Sets the maximum size of the ARC read cache as a percentage of system RAM."
type=percentage
min=30
max=90
default=70

The keys are:

Key	Purpose
tab	Which tab the setting appears under. One of general_settings, cache_settings, pool_settings, network_settings, volume_settings, driver_settings. A tab name that is not one of these will not render.
param	The kernel parameter to write, given as a full path such as /sys/module/zfs/parameters/zfs_arc_max. Set param=deprecated to retire a tunable while keeping old profiles loadable.
title	The label shown in the dialog. Quote it.
desc	The help text shown for the setting. Quote it. Naming the underlying kernel parameter in the text is the existing convention and worth following.
type	range for a numeric slider, percentage for a slider expressed as a percentage, boolean for a checkbox.
min / max	The permitted range. The dialog will not offer values outside it, and a profile value outside it is rejected -- see the warning below.
default	The shipped default, and the value the Default profile restores.
param_units / display_units	Used when the kernel parameter's unit differs from the unit you want shown. For example param_units=B with display_units=MB lets an administrator work in megabytes while the kernel parameter is written in bytes.

Because the built-in Default profile is regenerated from these default values on every service start, a tunable you add here automatically becomes part of Default without any further work.

qs_systemprofiles.conf -- defining profiles

Each profile is one section, with a display name, a description, and a list of tunable values:

[my_site_nfs_tuning]
name=Site NFS Tuning
description=Tuning for NFS-heavy workloads on our 60-bay shelves.
tunables="zfs_arc_max:80,zfs_txg_timeout:5,spl_kmem_cache_kmem_threads:8"

The tunables value is a comma separated list of key:value pairs. Four key forms are accepted:

• the tunable's sst_ section name, for example sst_cache_size
• a bare ZFS parameter name -- anything beginning zfs_, zfetch_, metaslab_, zio_ or vdev_ -- resolved under /sys/module/zfs/parameters/
• a bare SPL parameter name beginning spl_, resolved under /sys/module/spl/parameters/
• a full parameter path beginning /

A value carrying a B suffix is treated as a byte count and converted into the tunable's display units, which is why the shipped Seagate profile can write zfs_vdev_aggregation_limit:33554432B for a setting the dialog presents in KB.

A value outside the tunable's min/max is skipped, not clamped. The profile still applies, but that one setting is silently left alone apart from a warning in the service log:

Skipping tunable '<name>' for profile '<profile>', value '<n>' is outside of range (<min> : <max>)

If a profile does not appear to take full effect, check /var/log/qs/qs_service.log for that message before assuming the tunable itself is broken.

Two further rules are worth knowing:

• Do not add a tunables line to a profile marked populate_defaults=true. That flag marks the built-in Default profile, whose tunable set is generated automatically from every tunable's default value on each service start.
• A profile you have modified is not overwritten by the file. If a profile of the same name already exists and has been edited, the definition in the configuration file is not re-applied over it. The Default profile is the exception and is always refreshed.

The easiest way to author a custom profile is to set the values you want in the dialog, press Save Profile..., and then read the result back with qs tunable-profile-get. That gives you a known-good set of values to copy into the configuration file for deployment across a fleet.

Related pages

• Storage System -- the Storage System Modify dialog and the rest of the system-level configuration
• Storage Pools -- pool-level tuning, which is separate from these system-wide tunables

---

Verified against QuantaStor 6.9.0.