Designing Systems - Revision history

Qadmin: /* DRAID / Ideal for Backup and Archive */

2024-09-03T19:36:09Z

DRAID / Ideal for Backup and Archive

← Older revision		Revision as of 19:36, 3 September 2024
Line 59:		Line 59:

	===== DRAID / Ideal for Backup and Archive =====		===== DRAID / Ideal for Backup and Archive =====
	QuantaStor supports [https://openzfs.github.io/openzfs-docs/Basic%20Concepts/dRAID%20Howto.html dRAID] which is a distributed RAID feature of OpenZFS. dRAID addresses the slow RAID rebuild speeds one can see with longer stripe RAIDZ2/RAIDZ3 layouts and has some large file performance benefits. dRAID groups use 'virtual hot-spares' to auto-heal / repair in the event of a device failure and QuantaStor requires one or two virtual spares per DRAID group. DRAID is ideal when one has multiple large JBODs as one can dedicate each JBOD to be a DRAID VDEV within a larger pool and have both higher durability and higher rebuild speeds.		QuantaStor supports [https://openzfs.github.io/openzfs-docs/Basic%20Concepts/dRAID%20Howto.html dRAID] which is a distributed RAID feature of OpenZFS. dRAID addresses the slow RAID rebuild speeds one can see with longer stripe RAIDZ2/RAIDZ3 layouts and has some large file performance benefits. dRAID groups use 'virtual hot-spares' to auto-heal / repair in the event of a device failure and QuantaStor requires one or two virtual spares per DRAID group. DRAID is ideal when one has multiple large JBODs as one can dedicate each JBOD to be a DRAID VDEV within a larger pool and have both higher durability and higher rebuild speeds. QuantaStor also integrates with the Seagate Corvault which has technology (ADAPT) similar to OpenZFS dRAID but it is built into the smart-JBOD hardware itself. For configurations with less than 90x devices we typically recommend a simple RAIDZ3 8d+3p layout but when the design gets into multiple JBODs with hundreds of devices there are advantages to using dRAID with internal Z3 rather than Z3 directly.

	== Narrow By Use Case ==		== Narrow By Use Case ==

Qadmin: /* RAIDZ1/Z2/Z3/erasure-coding / Ideal for Media and Archive */

2024-09-03T19:33:25Z

RAIDZ1/Z2/Z3/erasure-coding / Ideal for Media and Archive

← Older revision		Revision as of 19:33, 3 September 2024
Line 57:		Line 57:
	===== RAIDZ1/Z2/Z3/erasure-coding / Ideal for Media and Archive =====		===== RAIDZ1/Z2/Z3/erasure-coding / Ideal for Media and Archive =====
	RAIDZ2 employs double parity (P and Q) so that a Storage Pool may sustain two simultaneous disk failures per stripe set with no data loss. RAIDZ2 with 4 data + 2 parity (6 devices per VDEV) will provide the best rebuild speed. RAIDZ3 with 8 data + 3 parity (11 devices per VDEV) will provide higher fault-tolerance and a greater usable to raw capacity ratio (72.7%) but the rebuild speed will not be as fast. Use the RAIDZ2 configuration for active backup/archive workloads and the RAIDZ3 layout for cold archive configurations where rebuild speed is less important. Avoid the use of RAIDZ1 as it can only sustain the loss of one hard drive per RAID set (VDEV). Ceph configurations also support parity based data fault-tolerance but the technology is referred to as erasure-coding. Erasure coding spans all the OSDs (data devices) in a given Ceph cluster so that one or more server can fail with no downtime and no loss of read write access. Erasure coding uses a terminology of "data blocks (K)" and "coding blocks (M)" with a notation of K+M (vs D+P for RAID). The RAIDZ3 layout with 8 data + 3 parity has an erasure coding equivalent of 8K+3M. Whereas RAID configurations require just a pair of servers the sum of the K+M with erasure coding modes is generally the minimal server count. So a QuantaStor Ceph cluster setup with 8K+3M erasure coding for an S3 object storage configuration would typically have a minimum of 11 servers. This would allow for 3x servers to fail with no downtime.		RAIDZ2 employs double parity (P and Q) so that a Storage Pool may sustain two simultaneous disk failures per stripe set with no data loss. RAIDZ2 with 4 data + 2 parity (6 devices per VDEV) will provide the best rebuild speed. RAIDZ3 with 8 data + 3 parity (11 devices per VDEV) will provide higher fault-tolerance and a greater usable to raw capacity ratio (72.7%) but the rebuild speed will not be as fast. Use the RAIDZ2 configuration for active backup/archive workloads and the RAIDZ3 layout for cold archive configurations where rebuild speed is less important. Avoid the use of RAIDZ1 as it can only sustain the loss of one hard drive per RAID set (VDEV). Ceph configurations also support parity based data fault-tolerance but the technology is referred to as erasure-coding. Erasure coding spans all the OSDs (data devices) in a given Ceph cluster so that one or more server can fail with no downtime and no loss of read write access. Erasure coding uses a terminology of "data blocks (K)" and "coding blocks (M)" with a notation of K+M (vs D+P for RAID). The RAIDZ3 layout with 8 data + 3 parity has an erasure coding equivalent of 8K+3M. Whereas RAID configurations require just a pair of servers the sum of the K+M with erasure coding modes is generally the minimal server count. So a QuantaStor Ceph cluster setup with 8K+3M erasure coding for an S3 object storage configuration would typically have a minimum of 11 servers. This would allow for 3x servers to fail with no downtime.

			===== DRAID / Ideal for Backup and Archive =====
			QuantaStor supports [https://openzfs.github.io/openzfs-docs/Basic%20Concepts/dRAID%20Howto.html dRAID] which is a distributed RAID feature of OpenZFS. dRAID addresses the slow RAID rebuild speeds one can see with longer stripe RAIDZ2/RAIDZ3 layouts and has some large file performance benefits. dRAID groups use 'virtual hot-spares' to auto-heal / repair in the event of a device failure and QuantaStor requires one or two virtual spares per DRAID group. DRAID is ideal when one has multiple large JBODs as one can dedicate each JBOD to be a DRAID VDEV within a larger pool and have both higher durability and higher rebuild speeds.

	== Narrow By Use Case ==		== Narrow By Use Case ==

Qadmin: /* Hardware RAID */

2024-09-03T19:28:27Z

Hardware RAID

← Older revision		Revision as of 19:28, 3 September 2024
Line 48:		Line 48:

	===== Hardware RAID =====		===== Hardware RAID =====
	The use of hardware RAID should be limited to the QuantaStor OS boot device. Do not use hardware RAID for scale-up or scale-out Storage Pools. If you ~~have~~ a hardware RAID controller use the pass-thru option to make the media accessible without applying RAID at the hardware controller level.		The use of hardware RAID should be limited to the QuantaStor OS boot device and even there we recommend using software RAID1 with m.2 media on the motherboard. Do not use hardware RAID for scale-up or scale-out Storage Pools, the only exception being special configurations such as those that use the Seagate Corvault. If you must use a hardware RAID controller use the pass-thru option (HBA pass-thru mode) to make the media accessible without applying RAID at the hardware controller level.

	===== RAID10 / Ideal for Virtualization, Databases, Email Server, VDI and other high IOPS workloads =====		===== RAID10 / Ideal for Virtualization, Databases, Email Server, VDI and other high IOPS workloads =====

Qadmin: /* Hardware RAID */

2021-06-30T23:06:49Z

Hardware RAID

← Older revision		Revision as of 23:06, 30 June 2021
Line 48:		Line 48:

	===== Hardware RAID =====		===== Hardware RAID =====
	~~We only recommend the~~ use of hardware RAID ~~for~~ the OS boot device ~~and/~~or ~~for Gluster based configurations~~. ~~Gluster configurations benefit from the~~ RAID controller~~'s write-back cache but~~ the ~~controller must be configured with a super~~-~~capacitor~~ to ~~ensure there is no data loss in~~ the ~~event of a sudden power outage~~.		The use of hardware RAID should be limited to the QuantaStor OS boot device. Do not use hardware RAID for scale-up or scale-out Storage Pools. If you have a hardware RAID controller use the pass-thru option to make the media accessible without applying RAID at the hardware controller level.

	===== RAID10 / Ideal for Virtualization, Databases, Email Server, VDI and other high IOPS workloads =====		===== RAID10 / Ideal for Virtualization, Databases, Email Server, VDI and other high IOPS workloads =====

Qadmin: /* File Storage Pools (NFS/SMB) */

2021-06-30T23:04:58Z

File Storage Pools (NFS/SMB)

← Older revision		Revision as of 23:04, 30 June 2021
Line 24:		Line 24:
	There are both scale-out and scale-up options for NFS/SMB file storage. For most workloads the scale-up configurations are often the best option as they're lower cost in terms of hardware. For data lakes, HPC, and other large scale use cases where the datasets need scalability to 10PB upwards to 100PB scale-out NAS is the better option.		There are both scale-out and scale-up options for NFS/SMB file storage. For most workloads the scale-up configurations are often the best option as they're lower cost in terms of hardware. For data lakes, HPC, and other large scale use cases where the datasets need scalability to 10PB upwards to 100PB scale-out NAS is the better option.

	* Scale-up ZFS based ~~storage pools~~ are the most feature rich and offer the highest performance for NFS/CIFS access. Scale-up Storage Pools are suitable for a broad spectrum of use cases. [[https://www.osnexus.com/zfs-designer?usable=1000&jbod_min=3&jbod_max=4&compression=20&overhead=20&ram_ratio=1&layout=raidz2_4_2&drive_type=8TB '''example design''']]		* Scale-up (ZFS based) Storage Pools are the most feature rich and offer the highest performance for NFS/CIFS access. Scale-up Storage Pools are suitable for a broad spectrum of use cases. [[https://www.osnexus.com/zfs-designer?usable=1000&jbod_min=3&jbod_max=4&compression=20&overhead=20&ram_ratio=1&layout=raidz2_4_2&drive_type=8TB '''example design''']]
			* Scale-out (CephFS based) Storage Pools should generally follow the same guidelines recommended above for S3 object storage configurations.

	==== Block Storage Pools (iSCSI/FC) ====		==== Block Storage Pools (iSCSI/FC) ====

Qadmin: /* File Storage Pools (NFS/SMB) */

2021-06-30T23:03:35Z

File Storage Pools (NFS/SMB)

← Older revision		Revision as of 23:03, 30 June 2021
Line 22:		Line 22:
	==== File Storage Pools (NFS/SMB) ====		==== File Storage Pools (NFS/SMB) ====

	There are ~~one choice~~ for ~~building File~~ storage ~~pools~~, ~~ZFS~~.		There are both scale-out and scale-up options for NFS/SMB file storage. For most workloads the scale-up configurations are often the best option as they're lower cost in terms of hardware. For data lakes, HPC, and other large scale use cases where the datasets need scalability to 10PB upwards to 100PB scale-out NAS is the better option.
	* ZFS based storage pools are the most feature rich and offer the highest performance for NFS/CIFS access. ~~ZFS based pools~~ are ~~also usable and adjustable~~ for a broad spectrum of use cases ~~including virtualization and databases~~. [[https://www.osnexus.com/zfs-designer?usable=1000&jbod_min=3&jbod_max=4&compression=20&overhead=20&ram_ratio=1&layout=raidz2_4_2&drive_type=8TB '''example design''']]
			* Scale-up ZFS based storage pools are the most feature rich and offer the highest performance for NFS/CIFS access. Scale-up Storage Pools are suitable for a broad spectrum of use cases. [[https://www.osnexus.com/zfs-designer?usable=1000&jbod_min=3&jbod_max=4&compression=20&overhead=20&ram_ratio=1&layout=raidz2_4_2&drive_type=8TB '''example design''']]

	==== Block Storage Pools (iSCSI/FC) ====		==== Block Storage Pools (iSCSI/FC) ====

Qadmin: /* Object Storage Pools (S3 Compatible) */

2021-06-30T22:59:52Z

Object Storage Pools (S3 Compatible)

← Older revision		Revision as of 22:59, 30 June 2021
Line 16:		Line 16:
	General guideline is to have at least 60MB/sec to 100MB/sec SSD write throughput per HDD data device that is added to the cluster. As an example, a scale-out QuantaStor cluster with 60x HDDs per server will require between 3600MB/sec and 6000MB/sec of SSD write performance. SSDs must be added in pairs as QuantaStor does a software RAID1 mirror of the media used in ''Journal Groups''. To meet 6000MB/sec one could select 6x SSDs that have a rated write performance of 2000MB/sec. The 6x SSDs will form into 3x mirrored ''Journal Groups'' when the media is added to the cluster. The 60x OSDs that are created using the 60x HDDs will each be allocated slices/partitions from the ''Journal Groups'' to be used as ''Write-ahead Log'' and ''Metadata DB Offload''.		General guideline is to have at least 60MB/sec to 100MB/sec SSD write throughput per HDD data device that is added to the cluster. As an example, a scale-out QuantaStor cluster with 60x HDDs per server will require between 3600MB/sec and 6000MB/sec of SSD write performance. SSDs must be added in pairs as QuantaStor does a software RAID1 mirror of the media used in ''Journal Groups''. To meet 6000MB/sec one could select 6x SSDs that have a rated write performance of 2000MB/sec. The 6x SSDs will form into 3x mirrored ''Journal Groups'' when the media is added to the cluster. The 60x OSDs that are created using the 60x HDDs will each be allocated slices/partitions from the ''Journal Groups'' to be used as ''Write-ahead Log'' and ''Metadata DB Offload''.

	[[https://www.osnexus.com/ceph-designer?usable=4249&layout=erasure_code_4_2&ssd_count=4&compression=20&overhead=10&drive_type=~~12TB~~&server_type=svr_wd_4u60_8 '''example design''']]		One additional SSD should be added to each QuantaStor server in order to have some number of SSD based OSDs to be used for the object storage bucket index pool. This is important for fast bucket operations and fast re-sharding of the bucket index as buckets grow over time.

			[[https://www.osnexus.com/ceph-designer?usable=4249&layout=erasure_code_4_2&ssd_count=4&compression=20&overhead=10&drive_type=18TB&server_type=svr_wd_4u60_8&journal_type=3840&journal_count=7 '''example design''']]

	==== File Storage Pools (NFS/SMB) ====		==== File Storage Pools (NFS/SMB) ====

Qadmin: /* Narrow By Protocol Type */

2021-06-30T22:56:19Z

Narrow By Protocol Type

← Older revision		Revision as of 22:56, 30 June 2021
Line 8:		Line 8:
	== Narrow By Protocol Type ==		== Narrow By Protocol Type ==

	==== Object Storage Pools (S3~~/SWIFT~~) ====		==== Object Storage Pools (S3 Compatible) ====

	QuantaStor delivers object storage support ~~via both~~ S3 ~~and SWIFT~~ using integrated Ceph technology. All object storage configurations require a minimum configuration of a ~~3x system node grid each~~ with at least 3x disks ~~or logical disk drives~~ per system. ~~(For non~~-~~production use one may setup Ceph in a single-node configuration) A typical system will have 12x to 60x disk drives per system~~. NVMe media is ~~used~~ as ~~a journal and~~ it ~~is recommended~~ to ~~have one~~ NVMe ~~device for each 12-20 disk devices~~. ~~For example, a 2U12 bay server would require 1x NVMe device,~~ and ~~a 4U60 would require 3x or 4x~~ NVMe ~~devices~~. ~~With high-performance Intel Optane~~ media ~~the ratio can be higher with one Optane journal device for each 20x-30x HDD devices (OSDs)~~. ~~The general goal~~ is to have at least 100MB/sec ~~of NVMe~~ write throughput ~~available~~ per device. So an ~~NVMe device~~ with ~~2000MB~~/sec ~~sequential~~ write performance ~~can provide~~ write ~~journal space for up~~ to ~~20x~~ HDDs ~~(OSDs) maximum~~. [[https://www.osnexus.com/ceph-designer?usable=4249&layout=erasure_code_4_2&ssd_count=4&compression=20&overhead=10&drive_type=12TB&server_type=svr_wd_4u60_8 '''example design''']]		QuantaStor delivers object storage support that is S3 Compatible using it's integrated Ceph technology. All object storage configurations require a minimum configuration of a 4x QuantaStor systems with at least 4x disks per system and 2x SSDs per system to boost write performance. All-flash configurations do not require separate additional dedicated SSDs for write logging.

			NVMe media is preferred as it can support 12x to 20x HDDs with NVMe PCIe 3.0 media and up to 40x HDDs with newer NVMe PCIe 4.0 media.

			General guideline is to have at least 60MB/sec to 100MB/sec SSD write throughput per HDD data device that is added to the cluster. As an example, a scale-out QuantaStor cluster with 60x HDDs per server will require between 3600MB/sec and 6000MB/sec of SSD write performance. SSDs must be added in pairs as QuantaStor does a software RAID1 mirror of the media used in ''Journal Groups''. To meet 6000MB/sec one could select 6x SSDs that have a rated write performance of 2000MB/sec. The 6x SSDs will form into 3x mirrored ''Journal Groups'' when the media is added to the cluster. The 60x OSDs that are created using the 60x HDDs will each be allocated slices/partitions from the ''Journal Groups'' to be used as ''Write-ahead Log'' and ''Metadata DB Offload''.

			[[https://www.osnexus.com/ceph-designer?usable=4249&layout=erasure_code_4_2&ssd_count=4&compression=20&overhead=10&drive_type=12TB&server_type=svr_wd_4u60_8 '''example design''']]

	==== File Storage Pools (NFS/SMB) ====		==== File Storage Pools (NFS/SMB) ====

Qadmin: /* SSD Write Log and Read Caching */

2021-06-30T22:47:27Z

SSD Write Log and Read Caching

← Older revision		Revision as of 22:47, 30 June 2021
Line 169:		Line 169:
	As you can see from the table above, the higher the write load on the system the more free space is required in order to maintain consistent performance.		As you can see from the table above, the higher the write load on the system the more free space is required in order to maintain consistent performance.

	== ~~SSD Write~~ Log ~~and Read Caching~~ ==		== Scale-up Storage Pool Cache & Log Device Performance Optimizations ==

	When you select ZFS as the ''Storage Pool'' type in QuantaStor you gain access to additional capabilities like SSD write log and read cache features. These will boost write IOPS for most workloads and the read cache is especially helpful for playback applications. To add SSDs to a Storage Pool simply right-click on the pool in the web interface and choose 'Add Cache Devices..' from the pop-up menu. Note that these SSDs used for caching must be dedicated to a specific Storage Pool and cannot be assigned to multiple pools at the same time.		When you select ZFS as the ''Storage Pool'' type in QuantaStor you gain access to additional capabilities like SSD write log and read cache features. These will boost write IOPS for most workloads and the read cache is especially helpful for playback applications. To add SSDs to a Storage Pool simply right-click on the pool in the web interface and choose 'Add Cache Devices..' from the pop-up menu. Note that these SSDs used for caching must be dedicated to a specific Storage Pool and cannot be assigned to multiple pools at the same time.

Qadmin: /* Considerations with Scale-up Clusters (ZFS based) */

2021-06-30T22:46:08Z

Considerations with Scale-up Clusters (ZFS based)

← Older revision		Revision as of 22:46, 30 June 2021
Line 164:		Line 164:
	\| style="text-align: center;" \| 10%		\| style="text-align: center;" \| 10%
	\| style="text-align: center;" \| 90%		\| style="text-align: center;" \| 90%
	\| style="text-align: center;" \| 35%		\| style="text-align: center;" \| 25%
	\|}		\|}