Guides:Estimating Performance for Scale-up ZFS Storage Pools
By Steve Umbehocker, CTO, OSNexus · Updated October 31, 2026
A scale-up storage pools in QuantaStor are OpenZFS based and are owned by one server at a time (active/passive) but both nodes can be (active/active) when there are two or more pools by balancing them across the HA cluster pair. A rough estimate of a pool's performance can be done easily by following a few rules outlined in this article. Factors including pool's layout, the media type and interface (NVMe, SAS SSD, NL-SAS, SATA), the number of devices in the pool, CPU performance and more all contribute to the result. This guide gives those rules for all-flash pools (IOPS and throughput) and for hybrid pools, which use HDDs for capacity and flash for metadata offload and write logs. It also covers how to size the flash in a hybrid pool and how to check an estimate once the system is built. The figures are planning estimates for sizing, not guarantees. Validate them against your own workload before you go into production.
Why it matters
A storage design has to meet the performance needs of the workloads and users it serves. If a system turns out undersized for its use case, the shortfall shows up as slow applications, backup windows that overrun and constant complaints, and it is expensive to fix after the hardware is racked. A rough but grounded estimate at design time lets you choose the media, device count, pool count and CPU with confidence, and gives you a number to test against at acceptance.
How it works
Performance in a ZFS pool scales with the number of vdevs (the RAID groups that make up the pool), up to the point where the CPU and the single pool become the limit. QuantaStor builds pools from vdevs of a chosen layout. See Storage Pools for the layouts and the pool creation dialog.
- All-flash pools are limited mainly by vdev count, CPU clock speed and, for throughput, client concurrency.
- Hybrid pools are limited by the mechanical latency of the HDDs. Throughput scales well with spindle count, but small random IOPS stay low unless flash write-log devices absorb synchronous writes.
- Metadata offload devices (a ZFS special vdev) keep file system metadata, and optionally small blocks, on flash so HDDs only serve large data blocks. These are not needed in all-flash pools but can help support the longevity of large block QLC/PLC media when using TLC flash media to absorb the small-block IO metadata operations.
- Write log devices (SLOG) take the ZFS intent log off the HDDs, which raises synchronous write IOPS. Again these are generally only needed with HDD based pools and we generally do not add these to HDD based pools unless there will be workloads like VMs and databases that have SYNC_IO operations.
Design and sizing
All-flash pools
OSNexus configures all-flash pools as 4d+2p (RAIDZ2 vdevs of six devices). This gives good usable capacity, survives two device failures per vdev, and performs well even for small-block IO. Measured results with SAS SSDs:
| SAS SSDs in one pool | 4d+2p vdevs | Approximate IOPS |
|---|---|---|
| 6 | 1 | 120K |
| 12 | 2 | 240K |
| 18 | 3 | about 240K |
| 24 | 4 | about 240K |
A single pool levels off at about 240K IOPS after two vdevs. To get more from 24 SSDs, split them into two pools of 12 and run each pool on a separate controller (node) of an HA cluster pair. Each node then serves its own pool at about 240K IOPS, for about 480K IOPS combined. Each pool gets its own failover group, so either node can take over both pools if its peer goes down; see Create Storage Pool High-Availability Group. For NVMe based pools you can increase the above performance numbers by at least 20%.
CPU: for all-flash scale-up designs, choose processors at 3 GHz or faster. Clock speed helps all-flash performance more than core count does, but plan on at least 16 cores.
Throughput depends heavily on how much concurrency the clients bring. With high concurrency, expect about 12 GB/s per all-flash pool. With moderate concurrency, plan on about 6 GB/s per pool. On Linux NFS clients the nconnect mount option opens several TCP connections per mount, which is the easiest way to raise concurrency from a single host.
Hybrid pools
For HDD-based pools, estimate throughput per drive. OSNexus testing shows about 50 to 100 MB/s per HDD in a pool. The exact figure varies with the layout type, file size and the share's record size. On large pools, start with a large record size such as 1M or 4M and test to see which works best for your configuration.
Hybrid throughput estimate = number of HDDs x 50-100 MB/s Example: pool with 60 HDDs low 60 x 50 MB/s = 3.0 GB/s high 60 x 100 MB/s = 6.0 GB/s average 4.5 GB/s
This rule of thumb holds up to very large pools, beyond 10 PB.
OSNexus doesn't normally give IOPS estimates for hybrid pools, because HDD mechanical latency makes it less predictable and keeps them at about 2K to 5K IOPS. Flash write-log devices raise that number and adds predictabilty:
| Hybrid pool configuration | Approximate IOPS |
|---|---|
| HDDs only | 2K-5K |
| HDDs + one pair of write log devices | about 10K |
| HDDs + two pairs of write log devices | about 15K |
Write log devices absorb a constant stream of writes. Use flash with high write performance, and pick either higher-capacity media or mixed-use media so the log devices don't wear out early.
Sizing metadata offload flash
Add flash (SSD or NVMe) equal to at least 1% of the pool's raw capacity as triple-mirrored metadata offload devices.
100 x 20 TB HDDs = 2,000 TB raw 1% target = 20 TB of flash 3 x 7.68 TB flash devices = 23 TB, as one triple-mirrored metadata offload group
Each extra group of three flash devices adds room for small files through the Small Block Offload setting. That setting routes writes at or below a chosen block size to the metadata offload devices; the product recommends a value between 8K and 32K.
Putting it together
| Design | Estimate |
|---|---|
| 12 SAS SSDs, one pool, 4d+2p | about 240K IOPS; 6-12 GB/s depending on concurrency |
| 24 SAS SSDs, two pools across an HA pair | about 480K IOPS combined |
| 60 HDDs, hybrid | 3-6 GB/s (about 4.5 GB/s average) |
| 100 x 20 TB HDDs, hybrid | 5-10 GB/s; at least 20 TB of metadata offload flash |
Setting it up
- Create the pool with the planned layout. In Storage Pools, create the pool with RAIDZ2 vdevs of six devices for all-flash, or the layout you chose for HDDs. See Storage Pools.
- Add the flash devices. In the Add Log/Cache/Metadata Device(s) to Storage Pool dialog, add three flash devices per metadata offload group. On hybrid pools that need more IOPS, add pairs of flash devices as write log devices.
- Set Small Block Offload on the pool if you added extra metadata offload groups for small files.
- Set the share record size. On large hybrid pools, create shares with a 1M or 4M record size; see Network Shares.
- For all-flash HA designs, create one pool per node and a failover group for each pool, so each node normally serves one pool.
On the client side, mount NFS shares with several connections to raise concurrency. Export and client access settings are covered in NFS Configuration.
mount -t nfs -o nconnect=8 192.0.2.10:/export/share1 /mnt/share1
Operating and testing
Treat the estimate as a hypothesis and test it before production:
- Run a benchmark with the block size, read/write mix and concurrency of the real workload, from as many clients as production will have. A single client with one stream will show a fraction of the pool's capability. See Performance Testing.
- On hybrid pools, try more than one record size (for example 1M and 4M) on test shares and keep the one that performs best.
- Raise client concurrency (more clients, more
nconnectconnections, more parallel jobs) until throughput stops climbing. That plateau is the pool's practical ceiling. - Watch the write log devices' wear over time and replace them before they reach their endurance limit.
- For tuning beyond layout and record size, see Performance Tuning.
If measured numbers are well below the estimate, check concurrency first, then CPU clock speed, then whether metadata offload and write log devices are present and healthy.
FAQ
How do I estimate the throughput of an HDD-based pool?
Multiply the number of HDDs by 50 MB/s for a low estimate and by 100 MB/s for a high one. A 60-drive pool comes out at 3 to 6 GB/s, with about 4.5 GB/s as a reasonable planning figure. The rule holds for pools beyond 10 PB.
Why doesn't adding more SSDs to one pool increase IOPS?
In OSNexus testing, a 4d+2p all-flash pool reaches about 240K IOPS with 12 SAS SSDs and stays near that level at 18 and 24 SSDs. To go further, split the devices into two pools and run one on each node of an HA pair, which gives about 480K IOPS combined.
How much flash does a hybrid pool need for metadata offload?
At least 1% of the pool's raw capacity, configured as triple-mirrored metadata offload devices. A 2,000 TB raw pool needs about 20 TB, which three 7.68 TB devices provide. Extra groups of three add room for small files.
Do write log (SLOG) devices help hybrid pool performance?
Yes, for IOPS. A hybrid pool without them delivers about 2K to 5K IOPS. One pair of write log devices raises that to about 10K and a second pair to about 15K. Use high-endurance or mixed-use flash.
Which matters more for all-flash pools, CPU clock speed or core count?
Clock speed. Choose processors at 3 GHz or faster for all-flash scale-up systems, and still plan on at least 16 cores.
Part of the QuantaStor Guides series. For reference documentation, see the QuantaStor documentation.