Performance Tuning - Revision history

Qadmin at 18:50, 17 April 2023

2023-04-17T18:50:09Z

← Older revision		Revision as of 18:50, 17 April 2023
Line 45:		Line 45:
	=== Profile Read-Ahead ===		=== Profile Read-Ahead ===
	The read_ahead_kb represents the amount of additional data that should be read after fulfilling a given read request. For example, if there's a read-request for 4KB and the read_ahead_kb is set to 64, then an additional 64KB will be read into the cache after the base 4KB request has been met. Why read this additional data? It counteracts the rotational latency problems inherent in spinning disk / hard disk drives. A 7200 RPM hard drive rotates 120 times per second, or roughly once every 8ms. That may sound fast, but take an example where you're reading records from a database and only gathering 4KB with each IO read request (one read per rotation). Done serially that would produce a throughput of a mere 480K/sec. This is why read-ahead and request queue depth are so important to getting good performance with spinning disk. With SSD, there are no mechanical rotational latency issues so the SSD profile uses a small 4k read-ahead.		The read_ahead_kb represents the amount of additional data that should be read after fulfilling a given read request. For example, if there's a read-request for 4KB and the read_ahead_kb is set to 64, then an additional 64KB will be read into the cache after the base 4KB request has been met. Why read this additional data? It counteracts the rotational latency problems inherent in spinning disk / hard disk drives. A 7200 RPM hard drive rotates 120 times per second, or roughly once every 8ms. That may sound fast, but take an example where you're reading records from a database and only gathering 4KB with each IO read request (one read per rotation). Done serially that would produce a throughput of a mere 480K/sec. This is why read-ahead and request queue depth are so important to getting good performance with spinning disk. With SSD, there are no mechanical rotational latency issues so the SSD profile uses a small 4k read-ahead.

	~~=== Profile Chunk Size ===~~
	This parameter only applies to XFS based storage pools. QuantaStor layers XFS storage pools over MDADM based software RAID and the chunk_size_kb is the chunk size used for the underlying md software RAID array that is created.

	=== Profile IO Scheduler ===		=== Profile IO Scheduler ===

Qadmin: /* ZFS Performance Tuning */

2020-05-14T16:11:23Z

ZFS Performance Tuning

← Older revision		Revision as of 16:11, 14 May 2020
Line 201:		Line 201:
	<pre>		<pre>
	qs-util setzfsarcmax 80		qs-util setzfsarcmax 80
	qs-util ~~setzfsarcmax~~ 50		qs-util setzfsarcmin 50
	</pre>		</pre>

Qadmin: /* Storage Pool IO Tuning Overview */

2019-05-02T16:32:43Z

Storage Pool IO Tuning Overview

← Older revision		Revision as of 16:32, 2 May 2019
Line 2:		Line 2:
	= Storage Pool IO Tuning Overview =		= Storage Pool IO Tuning Overview =

	Storage pools in QuantaStor have associated IO profiles which control parameters like read-ahead, request queue depth, and the IO scheduler. QuantaStor comes with a number of IO profiles to match common workloads but it is possible to create your own IO profiles to tune for a specific ~~system~~. In general, we recommend going with one of the defaults but for performance tuning purposes you can add additional profiles to the /etc/qs_io_profiles.conf file. Here are the IO profiles for the default and default SSD configurations.		Storage pools in QuantaStor have associated IO profiles which control parameters like read-ahead, request queue depth, and the IO scheduler. QuantaStor comes with a number of IO profiles to match common workloads but it is possible to create your own IO profiles to tune for a specific application. In general, we recommend going with one of the defaults but for performance tuning purposes you can add additional profiles to the /etc/qs_io_profiles.conf file. Here are the IO profiles for the default and default SSD configurations.

	<pre>		<pre>

Qadmin: /* Storage Pool IO Tuning Overview */

2019-05-02T16:24:02Z

Storage Pool IO Tuning Overview

← Older revision		Revision as of 16:24, 2 May 2019
Line 2:		Line 2:
	= Storage Pool IO Tuning Overview =		= Storage Pool IO Tuning Overview =

	Storage pools in QuantaStor have associated IO profiles which control parameters like read-ahead, request queue depth, and the IO scheduler. QuantaStor comes with a number of IO profiles to match common workloads but it is possible to create your own IO profiles to tune for a specific ~~application~~. In general, we recommend going with one of the defaults but for performance tuning purposes you can add additional profiles to the /etc/qs_io_profiles.conf file. Here are the IO profiles for the default and default SSD configurations.		Storage pools in QuantaStor have associated IO profiles which control parameters like read-ahead, request queue depth, and the IO scheduler. QuantaStor comes with a number of IO profiles to match common workloads but it is possible to create your own IO profiles to tune for a specific system. In general, we recommend going with one of the defaults but for performance tuning purposes you can add additional profiles to the /etc/qs_io_profiles.conf file. Here are the IO profiles for the default and default SSD configurations.

	<pre>		<pre>
	[default]		[default]
	name=Default		name=Default
	description=Optimizes for general purpose server ~~application~~ workloads		description=Optimizes for general purpose server system workloads
	nr_requests=2048		nr_requests=2048
	read_ahead_kb=256		read_ahead_kb=256
Line 51:		Line 51:
	=== Profile IO Scheduler ===		=== Profile IO Scheduler ===

	The IO scheduler represents the elevator algorithm used for [http://en.wikipedia.org/wiki/Deadline_scheduler scheduling] I/O operations. For storage ~~applications~~ the two best schedulers are the 'deadline' and the 'noop' scheduler. We find that deadline is best for HDDs and that the noop scheduler is best for SSDs.		The IO scheduler represents the elevator algorithm used for [http://en.wikipedia.org/wiki/Deadline_scheduler scheduling] I/O operations. For storage systems the two best schedulers are the 'deadline' and the 'noop' scheduler. We find that deadline is best for HDDs and that the noop scheduler is best for SSDs.
	The fifo_batch setting is related to the use of the 'deadline' scheduler. More information can be found [http://en.wikipedia.org/wiki/Deadline_scheduler here].		The fifo_batch setting is related to the use of the 'deadline' scheduler. More information can be found [http://en.wikipedia.org/wiki/Deadline_scheduler here].

Qadmin at 14:22, 17 April 2019

2019-04-17T14:22:20Z

← Older revision		Revision as of 14:22, 17 April 2019
Line 41:		Line 41:

	=== Profile Disk I/O Queue Depth (nr_requests) ===		=== Profile Disk I/O Queue Depth (nr_requests) ===
	The nr_requests represents the IO queue size. This is an important variable to change on the host/initiator side as well if you're connecting to your QuantaStor ~~appliance~~ via [http://www.monperrus.net/martin/scheduler+queue+size+and+resilience+to+heavy+IO iSCSI from a Linux based server]. In QuantaStor when you set the nr_requests the core service applies this setting to all disks in the pool by adjusting parameters in /sys/block/ part of the sys filesystem.		The nr_requests represents the IO queue size. This is an important variable to change on the host/initiator side as well if you're connecting to your QuantaStor system via [http://www.monperrus.net/martin/scheduler+queue+size+and+resilience+to+heavy+IO iSCSI from a Linux based server]. In QuantaStor when you set the nr_requests the core service applies this setting to all disks in the pool by adjusting parameters in /sys/block/ part of the sys filesystem.

	=== Profile Read-Ahead ===		=== Profile Read-Ahead ===
Line 75:		Line 75:
	====Windows 2008 R2 host====		====Windows 2008 R2 host====
	* four 10GbE ports		* four 10GbE ports
	====QuantaStor Storage ~~Appliance~~====		====QuantaStor Storage System====
	* four 10GbE ports		* four 10GbE ports
	* LSI 9750-8i Controller		* LSI 9750-8i Controller
Line 198:		Line 198:
	=== ZFS Performance Tuning ===		=== ZFS Performance Tuning ===

	One of the most common tuning tasks that is done for ZFS is to set the size of the ARC cache. If your system has less than 10GB of RAM you should just use the default but if you have 32GB or more then it is a good idea to increase the size of the ARC cache to make maximum use of the available RAM for your storage ~~appliance~~. Before you set the tuning parameters you should run 'top' to verify how much RAM you have in the system. Next, run this command to set the amount of RAM to some percentage of the available RAM. For example to set the ARC cache to use a maximum of 80% of the available RAM, and a minimum of 50% of the available RAM in the system, run these, then reboot:		One of the most common tuning tasks that is done for ZFS is to set the size of the ARC cache. If your system has less than 10GB of RAM you should just use the default but if you have 32GB or more then it is a good idea to increase the size of the ARC cache to make maximum use of the available RAM for your storage system. Before you set the tuning parameters you should run 'top' to verify how much RAM you have in the system. Next, run this command to set the amount of RAM to some percentage of the available RAM. For example to set the ARC cache to use a maximum of 80% of the available RAM, and a minimum of 50% of the available RAM in the system, run these, then reboot:
	<pre>		<pre>
	qs-util setzfsarcmax 80		qs-util setzfsarcmax 80

Qadmin at 17:49, 21 March 2019

2019-03-21T17:49:03Z

Show changes

Qadmin at 02:12, 21 March 2019

2019-03-21T02:12:00Z

← Older revision		Revision as of 02:12, 21 March 2019
Line 1:		Line 1:
	~~== IO Tuning ==~~		[[Category:admin_guide]]

	=== ZFS Performance Tuning ===		=== ZFS Performance Tuning ===

Qadmin: Created page with "== IO Tuning == === ZFS Performance Tuning === One of the most common tuning tasks that is done for ZFS is to set the size of the ARC cache. If your system has less than 10..."

2019-03-21T02:11:35Z

Created page with "== IO Tuning == === ZFS Performance Tuning === One of the most common tuning tasks that is done for ZFS is to set the size of the ARC cache. If your system has less than 10..."

New page

== IO Tuning ==

=== ZFS Performance Tuning ===

One of the most common tuning tasks that is done for ZFS is to set the size of the ARC cache. If your system has less than 10GB of RAM you should just use the default but if you have 32GB or more then it is a good idea to increase the size of the ARC cache to make maximum use of the available RAM for your storage appliance. Before you set the tuning parameters you should run 'top' to verify how much RAM you have in the system. Next, run this command to set the amount of RAM to some percentage of the available RAM. For example to set the ARC cache to use a maximum of 80% of the available RAM, and a minimum of 50% of the available RAM in the system, run these, then reboot:
<pre>
qs-util setzfsarcmax 80
qs-util setzfsarcmax 50
</pre>

Example:
<pre>
sudo -i
qs-util setzfsarcmax 80
INFO: Updating max ARC cache size to 80% of total RAM 1994 MB in /etc/modprobe.d/zfs.conf to: 1672478720 bytes (1595 MB)
qs-util setzfsarcmin 50
INFO: Updating min ARC cache size to 50% of total RAM 1994 MB in /etc/modprobe.d/zfs.conf to: 1045430272 bytes (997 MB)
</pre>

To see how many cache hits you are getting you can monitor the ARC cache while the system is under load with the qs-iostat command:

<pre>
qs-iostat -a

Name Data
---------------------------------------------
hits 237841
misses 1463
c_min 4194304
c_max 520984576
size 16169912
l2_hits 19839653
l2_misses 74509
l2_read_bytes 256980043
l2_write_bytes 1056398
l2_cksum_bad 0
l2_size 9999875
l2_hdr_size 233044
arc_meta_used 4763064
arc_meta_limit 390738432
arc_meta_max 5713208

ZFS Intent Log (ZIL) / writeback cache statistics

Name Data
---------------------------------------------
zil_commit_count 876
zil_commit_writer_count 495
zil_itx_count 857
</pre>

A description of the different metrics for ARC, L2ARC and ZIL are below.

<pre>
hits = the number of client read requests that were found in the ARC
misses = the number of client read requests were not found in the ARC
c_min = the minimum size of the ARC allocated in the system memory.
c_max = the maximum size of the ARC that can be allocated in the system memory.
size = = the current ARC size
l2_hits = the number of client read requests that were found in the L2ARC
ls_misses = the number of client read requests were not found in the L2ARC
ls_read_bytes = The number of bytes read from the L2ARC ssd devices.
l2_write_bytes = The number of bytes written to the L2ARC ssd devices.
l2_chksum_bad = The number of checksums that failed the check on an SSD (a number of these occurring on the L2ARC usually indicates a fault for a SSD device that needs to be replaced)
l2_size = the current L2ARC size
l2_hdr_size = The size of the L2ARC reference headers that are present in ARC Metadata
arc_meta_used = The amount of ARC memory used for Metadata
arc_meta_limit = The maximum limit for the ARC Metadata
arc_meta_max = The maximum value that the ARC Metadata has achieved on this system

zil_commit_count = How many ZIL commits have occurred since bootup
zil_commit_writer_count = How many ZIL writers were used since bootup
zil_itx_count = the number of indirect transaction groups that have occurred sinc bootup
</pre>

=== Pool Performance Profiles ===

Read-ahead and request queue size adjustments can help tune your storage pool for certain workloads. You can also create new storage pool IO profiles by editing the /etc/qs_io_profiles.conf file. The default profile looks like this and you can duplicate it and edit it to customize it.

<pre>
[default]
name=Default
description=Optimizes for general purpose server application workloads
nr_requests=2048
read_ahead_kb=256
fifo_batch=16
chunk_size_kb=128
scheduler=deadline
</pre>

If you edit the profiles configuration file be sure to restart the management service with 'service quantastor restart' so that your new profile is discovered and is available in the web interface.

=== Storage Pool Tuning Parameters ===

QuantaStor has a number of tunable parameters in the /etc/quantastor.conf file that can be adjusted to better match the needs of your application. That said, we've spent a considerable amount of time tuning the system to efficiently support a broad set of application types so we do not recommend adjusting these settings unless you are a highly skilled Linux administrator.
The default contents of the /etc/quantastor.conf configuration file are as follows:
<pre>
[device]
nr_requests=2048
scheduler=deadline
read_ahead_kb=512

[mdadm]
chunk_size_kb=256
parity_layout=left-symmetric
</pre>

There are tunable settings for device parameters which are applied to the storage media (SSD/SATA/SAS), as well as settings like the MD device array chunk-size and parity configuration settings used with XFS based storage pools. These configuration settings are read from the configuration file dynamically each time one of the settings is needed so there's no need to restart the quantastor service. Simply edit the file and the changes will be applied to the next operation that utilizes them. For example, if you adjust the chunk_size_kb setting for mdadm then the next time a storage pool is created it will use the new chunk size. Other tunable settings like the device settings will automatically be applied within a minute or so of your changes because the system periodically checks the disk configuration and updates it to match the tunable settings.
Also, you can delete the quantastor.conf file and it will automatically use the defaults that you see listed above.