Storage System Optimization

From OSNEXUS Online Documentation Site
Jump to navigation Jump to search


The Storage System Optimization dialog allows one to adjust system tunings (tunables) that control cache behavior and Storage Pool I/O, so performance can be more closely matched to a specific hardware configuration and workload.

Navigation: Storage Management → Storage Systems → right-click a Storage System → Storage System Optimization...

Note this dialog is reached from the right-click context menu on a Storage System, not from the toolbar. Note also that most of the tuning settings (all those based on zfs params) only effect Scale-up (ZFS-based) pools and not Scale-out (Ceph-based) pools.

Every setting is applied live to the running system and most settings do not require a reboot (driver changes to the SPL excepted). All changes are persisted so they survive reboots and upgrades. Values are per Storage System, so in a grid each system is tuned independently. The Storage System selector at the top of the dialog switches which system you are editing. To apply a common configuration to each system save your custom configuration as a Optimization Profile first, then apply it to other systems.

The six tabs group the tunables by what they affect:

See also Optimization Profiles and Extending the configuration files.

Before you change anything

The shipped defaults are appropriate for the large majority of deployments. The tunables here are the ones that matter for specific, identifiable problems (resilver taking too long, write latency spikes under load, a replication window overrunning). Changing them speculatively is more likely to reduce performance and create problems rather than help so we recommend any changes here be done with the assistance of the support team.

Recommended approach:

  • Establish a baseline first. Use the Performance and Cache Stats (ARC) views on the Storage System dashboard to see what the system is actually doing before you change anything.
  • Change one group of related settings at a time, then measure again under a representative workload.
  • Save a named profile before you start experimenting, so you can get back to a known state -- see Optimization Profiles.
  • Revert All returns the dialog to the values currently stored in the database, which is the quickest way out of a half-finished experiment. Note it reverts the workspace -- you still need OK or Apply to make that stick.

If you are unsure whether a setting applies to your situation, contact OSNEXUS support rather than guessing; several of these interact.

Optimization Profiles

A profile is a named collection of tunable values that can be applied to a Storage System in one step. Profiles make it practical to keep a tuning recipe consistent across a fleet, and to get back to a known configuration.

The Optimization Profiles controls at the top of the dialog are:

  • Apply Profile -- copies the selected profile's values into the dialog. Nothing is committed until you press OK or Apply, so you can review what a profile will change before accepting it.
  • Save Profile... -- saves the values currently in the dialog as a new named profile.
  • Delete Profile -- deletes the selected profile. This only removes the profile; it does not change the settings of any system that had it applied.
  • Revert All -- resets the dialog back to the values stored in the database for this system.

Two profiles ship with QuantaStor:

  • Default -- every tunable at its shipped default. Applying it resets the whole system optimization configuration. This profile is regenerated from the current defaults each time the QuantaStor service starts, so it always reflects the running release, including tunables added in a newer version.
  • Seagate Exos Protect Optimized -- tuning for large scale-up deployments layered over Seagate ADAPT hardware RAID.

Profiles can also be managed from the CLI -- see the QuantaStor CLI Command Reference for each command's full argument list: 

qs tunable-profile-list
qs tunable-profile-get --tunable-profile=<name>
qs tunable-profile-create --name=<name> ...
qs tunable-profile-modify --tunable-profile=<name> ...
qs tunable-profile-apply --tunable-profile=<name> --storage-system=<system>
qs tunable-profile-delete --tunable-profile=<name>

Individual tunables can be read and set directly. Note tunable-set takes the value as part of --tunable in key:value form -- there is no separate --value argument -- and accepts a comma separated list to set several at once: 

qs tunable-list
qs tunable-get --tunable=<sst_name>
qs tunable-set --tunable=<sst_name>:<value>
qs tunable-set --tunable=sst_cache_size:80,sst_write_flush_rate_sec:5

tunable-set also takes --tunable-option=reset to return tunables to their defaults, and --storage-system to target a system other than the local one.

Settings Reference

Storage System Optimization, General Settings tab.

Every setting the dialog offers is documented on qs_systemtunables.conf, the configuration file the tunable set is loaded from: one table per tab, giving each setting's identifier, its accepted range, its shipped default, its help text, and the kernel parameter it writes. That page also covers the file's format, the two retired tunables, and the override path that keeps a local change to the tunable set across an upgrade.

Extending and customizing the configuration files

The tunable set and the shipped profiles are both driven by plain text configuration files on each Storage System, so a site can add tunables that QuantaStor does not expose out of the box and define its own profiles.

Both files live in: 

/opt/osnexus/quantastor/conf/

These files are replaced on upgrade. Keep a copy of any local additions outside that directory and re-apply them after upgrading, or the changes will be lost. In both files a line is treated as a comment only when # is the first character of the line -- an indented # is not a comment.

Changes take effect when the QuantaStor service restarts: 

systemctl restart quantastor

qs_systemtunables.conf -- defining what appears in the dialog

Adding a tunable, retiring one, or changing a range is done in qs_systemtunables.conf. See qs_systemtunables.conf for its keys, its validation rules, and the override location that survives an upgrade. A tunable added there automatically becomes part of the built-in Default profile, because that profile is regenerated from every tunable's default value on each service start.

qs_systemprofiles.conf -- defining profiles

Each profile is one section, with a display name, a description, and a list of tunable values: 

[my_site_nfs_tuning]
name=Site NFS Tuning
description=Tuning for NFS-heavy workloads on our 60-bay shelves.
tunables="zfs_arc_max:80,zfs_txg_timeout:5,spl_kmem_cache_kmem_threads:8"

The tunables value is a comma separated list of key:value pairs. Four key forms are accepted:

  • the tunable's sst_ section name, for example sst_cache_size
  • a bare ZFS parameter name -- anything beginning zfs_, zfetch_, metaslab_, zio_ or vdev_ -- resolved under /sys/module/zfs/parameters/
  • a bare SPL parameter name beginning spl_, resolved under /sys/module/spl/parameters/
  • a full parameter path beginning /

A value carrying a B suffix is treated as a byte count and converted into the tunable's display units, which is why the shipped Seagate profile can write zfs_vdev_aggregation_limit:33554432B for a setting the dialog presents in KB.

A value outside the tunable's min/max is skipped, not clamped. The profile still applies, but that one setting is silently left alone apart from a warning in the service log: 

Skipping tunable '<name>' for profile '<profile>', value '<n>' is outside of range (<min> : <max>)

If a profile does not appear to take full effect, check /var/log/qs/qs_service.log for that message before assuming the tunable itself is broken.

Two further rules are worth knowing:

  • Do not add a tunables line to a profile marked populate_defaults=true. That flag marks the built-in Default profile, whose tunable set is generated automatically from every tunable's default value on each service start.
  • A profile you have modified is not overwritten by the file. If a profile of the same name already exists and has been edited, the definition in the configuration file is not re-applied over it. The Default profile is the exception and is always refreshed.

The easiest way to author a custom profile is to set the values you want in the dialog, press Save Profile..., and then read the result back with qs tunable-profile-get. That gives you a known-good set of values to copy into the configuration file for deployment across a fleet.

Related pages

  • Storage System -- the Storage System Modify dialog and the rest of the system-level configuration
  • Storage Pools -- pool-level tuning, which is separate from these system-wide tunables

Verified against QuantaStor 6.9.0.

Retrieved from "https://wiki.osnexus.com/index.php?title=Storage_System_Optimization&oldid=27669"