Template:SetupStep CephRequirements - Revision history

Qadmin: /* Requirements Before Getting Started */

2020-07-28T07:08:19Z

Requirements Before Getting Started

← Older revision		Revision as of 07:08, 28 July 2020
Line 7:		Line 7:
	** 4x or more SSDs/HDDs for OSD data devices per server		** 4x or more SSDs/HDDs for OSD data devices per server
	** 2x SSDs per server for Journal Groups (only required for HDD based OSD systems)		** 2x SSDs per server for Journal Groups (only required for HDD based OSD systems)


	=== Front-end / Back-end Network Configuration ===		=== Front-end / Back-end Network Configuration ===

Qadmin: /* Requirements Before Getting Started */

2020-07-28T07:08:10Z

Requirements Before Getting Started

← Older revision		Revision as of 07:08, 28 July 2020
Line 1:		Line 1:
	== Requirements Before Getting Started ==		== Requirements Before Getting Started ==

	To achieve quorum a minimum of three systems are required. The storage provided by the system can be SAS or SATA HDD or SSDs but a minimum of 1x SSD is required for use as a journal (write log) device in each system.

	* 3x QuantaStor servers minimum		* 3x QuantaStor servers minimum
	** Dual Intel ~~Xeon~~ or AMD ~~Opteron~~ CPUs		** Dual Intel or AMD CPUs
	** ~~128~~ GB RAM		** 96 GB RAM
	** 1x to 6x 375GB or larger high write endurance NVMe/NVRAM/Optane device for use as Journal Device
	** 5x to 100x HDDs or SSD for data storage per system
	** 2x SSDs in hardware RAID1 (boot/system)		** 2x SSDs in hardware RAID1 (boot/system)
	** ~~NTP~~ / ~~Time Synchronization~~		** 4x or more SSDs/HDDs for OSD data devices per server
	** ~~Separate Networks~~ for ~~iSCSI and Ceph communication~~		** 2x SSDs per server for Journal Groups (only required for HDD based OSD systems)


	=== Front-end / Back-end Network Configuration ===		=== Front-end / Back-end Network Configuration ===

	~~Networking~~ for ~~scale-out file and block storage deployments~~ use a separate front-end and back-end ~~network~~ to ~~separate the client communication~~ to the front-end network ~~ports (S3/BlueStore, iSCSI/RBD) from the inter-node Ceph communication on~~ the back-end. ~~This not only boosts performance,~~ it ~~increases the fault-tolerance, reliability and maintainability of the Ceph cluster.~~		Network configuration for Ceph Clusters may use separate front-end and back-end networks. Depending on the protocols to be used it can sometimes be easier and simpler to configure a given Ceph Cluster with a single subnet for both the front-end/public network and the back-end/cluster network communication. With 100GbE and faster networks it is more common to deploy a single network configured for all cluster traffic.

	~~All nodes should have one or~~ more ~~ports designated as front-end ports and assigned IP address and subnet masks specifically for client access. One can have multiple physical, virtual IPs, and VLANs used on the front-end network~~ to ~~enable~~ a ~~variety of clients to access the storage. The back-end~~ network ~~ports should~~ all ~~be physical ports but it is not required~~.

	~~A basic configuration Ceph network configuration:~~
	[[File:qs4_front_back_network_ports.png\|1024px]]		[[File:qs4_front_back_network_ports.png\|1024px]]

Qadmin: /* Front-end / Back-end Network Configuration */

2019-10-10T22:20:30Z

Front-end / Back-end Network Configuration

← Older revision		Revision as of 22:20, 10 October 2019
Line 14:		Line 14:
	=== Front-end / Back-end Network Configuration ===		=== Front-end / Back-end Network Configuration ===

	Networking for scale-out file and block storage deployments use a separate front-end and back-end network to separate the client communication to the front-end network ports (S3/~~SWIFT~~, iSCSI/RBD) from the inter-node Ceph communication on the back-end. This not only boosts performance, it increases the fault-tolerance, reliability and maintainability of the Ceph cluster.		Networking for scale-out file and block storage deployments use a separate front-end and back-end network to separate the client communication to the front-end network ports (S3/BlueStore, iSCSI/RBD) from the inter-node Ceph communication on the back-end. This not only boosts performance, it increases the fault-tolerance, reliability and maintainability of the Ceph cluster.

	All nodes should have one or more ports designated as front-end ports and assigned IP address and subnet masks specifically for client access. One can have multiple physical, virtual IPs, and VLANs used on the front-end network to enable a variety of clients to access the storage. The back-end network ports should all be physical ports but it is not required.		All nodes should have one or more ports designated as front-end ports and assigned IP address and subnet masks specifically for client access. One can have multiple physical, virtual IPs, and VLANs used on the front-end network to enable a variety of clients to access the storage. The back-end network ports should all be physical ports but it is not required.

Qadmin: /* Requirements Before Getting Started */

2019-06-04T17:52:18Z

Requirements Before Getting Started

← Older revision		Revision as of 17:52, 4 June 2019
Line 1:		Line 1:
	== Requirements Before Getting Started ==		== Requirements Before Getting Started ==

	To achieve quorum a minimum of three systems are required. The storage provided by the system can be SAS or SATA HDD or SSDs but a minimum of 1x SSD is required for use as a journal (write log) device in each system~~. Systems must use a hardware RAID controller for QuantaStor boot/system devices and we recommend using a hardware RAID controller for the storage pools as well~~.		To achieve quorum a minimum of three systems are required. The storage provided by the system can be SAS or SATA HDD or SSDs but a minimum of 1x SSD is required for use as a journal (write log) device in each system.

	* 3x QuantaStor servers minimum		* 3x QuantaStor servers minimum
	** Dual Intel Xeon or AMD Opteron CPUs		** Dual Intel Xeon or AMD Opteron CPUs
	** 128 GB RAM		** 128 GB RAM
	** 1x 375GB or larger high write endurance ~~SSD/~~NVMe/NVRAM/Optane device for use as Journal Device		** 1x to 6x 375GB or larger high write endurance NVMe/NVRAM/Optane device for use as Journal Device
	** 5x to 100x HDDs or SSD for data storage per system		** 5x to 100x HDDs or SSD for data storage per system
			** 2x SSDs in hardware RAID1 (boot/system)
	** NTP / Time Synchronization		** NTP / Time Synchronization
	** Separate Networks for iSCSI and Ceph communication		** Separate Networks for iSCSI and Ceph communication

Qadmin: /* Requirements Before Getting Started */

2019-06-04T17:29:41Z

Requirements Before Getting Started

← Older revision		Revision as of 17:29, 4 June 2019
Line 3:		Line 3:
	To achieve quorum a minimum of three systems are required. The storage provided by the system can be SAS or SATA HDD or SSDs but a minimum of 1x SSD is required for use as a journal (write log) device in each system. Systems must use a hardware RAID controller for QuantaStor boot/system devices and we recommend using a hardware RAID controller for the storage pools as well.		To achieve quorum a minimum of three systems are required. The storage provided by the system can be SAS or SATA HDD or SSDs but a minimum of 1x SSD is required for use as a journal (write log) device in each system. Systems must use a hardware RAID controller for QuantaStor boot/system devices and we recommend using a hardware RAID controller for the storage pools as well.

	* 3x QuantaStor servers minimum ~~(up to 64x Systems)~~		* 3x QuantaStor servers minimum
	** Dual Intel Xeon or AMD Opteron ~~CPU~~		** Dual Intel Xeon or AMD Opteron CPUs
	** 128 GB RAM		** 128 GB RAM
	** 1x 375GB or larger high write endurance SSD/NVMe/NVRAM/Optane device for use as Journal Device		** 1x 375GB or larger high write endurance SSD/NVMe/NVRAM/Optane device for use as Journal Device

Qadmin: /* Requirements Before Getting Started */

2019-06-04T17:29:18Z

Requirements Before Getting Started

← Older revision		Revision as of 17:29, 4 June 2019
Line 3:		Line 3:
	To achieve quorum a minimum of three systems are required. The storage provided by the system can be SAS or SATA HDD or SSDs but a minimum of 1x SSD is required for use as a journal (write log) device in each system. Systems must use a hardware RAID controller for QuantaStor boot/system devices and we recommend using a hardware RAID controller for the storage pools as well.		To achieve quorum a minimum of three systems are required. The storage provided by the system can be SAS or SATA HDD or SSDs but a minimum of 1x SSD is required for use as a journal (write log) device in each system. Systems must use a hardware RAID controller for QuantaStor boot/system devices and we recommend using a hardware RAID controller for the storage pools as well.

	* 3x QuantaStor ~~Storage Systems~~ minimum (up to 64x Systems)		* 3x QuantaStor servers minimum (up to 64x Systems)
	~~:Per System:~~
	** Dual Intel Xeon or AMD Opteron CPU		** Dual Intel Xeon or AMD Opteron CPU
	** 128 GB RAM		** 128 GB RAM

Qadmin: /* Hardware RAID Configuration */

2019-06-04T17:28:42Z

Hardware RAID Configuration

@@ Line 11: / Line 11: @@
 ** NTP / Time Synchronization
 ** Separate Networks for iSCSI and Ceph communication
 === Front-end / Back-end Network Configuration ===

Qadmin: /* Requirements Before Getting Started */

2019-06-04T17:28:31Z

Requirements Before Getting Started

← Older revision		Revision as of 17:28, 4 June 2019
Line 5:		Line 5:
	* 3x QuantaStor Storage Systems minimum (up to 64x Systems)		* 3x QuantaStor Storage Systems minimum (up to 64x Systems)
	:Per System:		:Per System:
	** Intel Xeon or AMD Opteron CPU		** Dual Intel Xeon or AMD Opteron CPU
	** 64 GB RAM		** 128 GB RAM
	** 1x ~~200GB~~ or larger high write endurance SSD/NVMe/NVRAM device for use as ~~Journal Device~~		** 1x 375GB or larger high write endurance SSD/NVMe/NVRAM/Optane device for use as Journal Device
	~~:: 4x Object Storage Daemons per Journal Device, maximum of 8x OSDs per~~ Journal Device
	** 5x to 100x HDDs or SSD for data storage per system		** 5x to 100x HDDs or SSD for data storage per system
	** 1x hardware RAID controller for OSDs (SAS HBA can also be used but RAID is faster)
	** NTP / Time Synchronization		** NTP / Time Synchronization
	** Separate Networks for iSCSI and Ceph communication		** Separate Networks for iSCSI and Ceph communication

Qadmin: /* Front-end / Back-end Network Configuration */

2019-05-10T21:02:19Z

Front-end / Back-end Network Configuration

← Older revision		Revision as of 21:02, 10 May 2019
Line 33:		Line 33:
	Networking for scale-out file and block storage deployments use a separate front-end and back-end network to separate the client communication to the front-end network ports (S3/SWIFT, iSCSI/RBD) from the inter-node Ceph communication on the back-end. This not only boosts performance, it increases the fault-tolerance, reliability and maintainability of the Ceph cluster.		Networking for scale-out file and block storage deployments use a separate front-end and back-end network to separate the client communication to the front-end network ports (S3/SWIFT, iSCSI/RBD) from the inter-node Ceph communication on the back-end. This not only boosts performance, it increases the fault-tolerance, reliability and maintainability of the Ceph cluster.

	All nodes should have one or more ports designated as front-end ports and assigned IP address and subnet masks specifically for client access. ~~For all nodes one or more ports should be designated as the front-end ports and assigned appropriate IP addresses and subnets to enable client access.~~ One can have multiple physical, virtual IPs, and VLANs used on the front-end network to enable a variety of clients to access the storage. The back-end network ports should all be physical ports but it is not required.		All nodes should have one or more ports designated as front-end ports and assigned IP address and subnet masks specifically for client access. One can have multiple physical, virtual IPs, and VLANs used on the front-end network to enable a variety of clients to access the storage. The back-end network ports should all be physical ports but it is not required.

	A basic configuration Ceph network configuration:		A basic configuration Ceph network configuration:
	[[File:qs4_front_back_network_ports.png\|1024px]]		[[File:qs4_front_back_network_ports.png\|1024px]]

Qadmin at 16:53, 2 May 2019

2019-05-02T16:53:53Z

← Older revision		Revision as of 16:53, 2 May 2019
Line 1:		Line 1:
	== Requirements Before Getting Started ==		== Requirements Before Getting Started ==

	To achieve quorum a minimum of three ~~appliances~~ are required. The storage in the ~~appliance~~ can be SAS or SATA HDD or SSDs but a minimum of 1x SSD is required for use as a journal (write log) device in each ~~appliance~~. ~~Appliances~~ must use a hardware RAID controller for QuantaStor boot/system devices and we recommend using a hardware RAID controller for the storage pools as well.		To achieve quorum a minimum of three systems are required. The storage provided by the system can be SAS or SATA HDD or SSDs but a minimum of 1x SSD is required for use as a journal (write log) device in each system. Systems must use a hardware RAID controller for QuantaStor boot/system devices and we recommend using a hardware RAID controller for the storage pools as well.

	* 3x QuantaStor Storage ~~Appliances~~ minimum (up to 64x ~~Appliances~~)		* 3x QuantaStor Storage Systems minimum (up to 64x Systems)
	:Per ~~Appliance~~:		:Per System:
	** Intel Xeon or AMD Opteron CPU		** Intel Xeon or AMD Opteron CPU
	** 64 GB RAM		** 64 GB RAM
	** 1x 200GB or larger high write endurance SSD/NVMe/NVRAM device for use as Journal Device		** 1x 200GB or larger high write endurance SSD/NVMe/NVRAM device for use as Journal Device
	:: 4x Object Storage Daemons per Journal Device, maximum of 8x OSDs per Journal Device		:: 4x Object Storage Daemons per Journal Device, maximum of 8x OSDs per Journal Device
	** 5x to 100x HDDs or SSD for data storage per ~~appliance~~		** 5x to 100x HDDs or SSD for data storage per system
	** 1x hardware RAID controller for OSDs (SAS HBA can also be used but RAID is faster)		** 1x hardware RAID controller for OSDs (SAS HBA can also be used but RAID is faster)
	** NTP / Time Synchronization		** NTP / Time Synchronization
Line 24:		Line 24:
	* When the storage is fault-tolerant one need only maintain two (2) copies of the data (instead of 3x) so the storage efficiency is 40% usable vs. the standard Ceph mode of operation which is only 33% usable. That's a 20% increase in usable capacity.		* When the storage is fault-tolerant one need only maintain two (2) copies of the data (instead of 3x) so the storage efficiency is 40% usable vs. the standard Ceph mode of operation which is only 33% usable. That's a 20% increase in usable capacity.
	* RAID controllers bring with them 1GB of NVRAM write-back cache which greatly boosts the performance of OSDs and Journal devices. (Be sure that the card has the CacheVault/MaxCache supercapacitor which is required to protect the write cache).		* RAID controllers bring with them 1GB of NVRAM write-back cache which greatly boosts the performance of OSDs and Journal devices. (Be sure that the card has the CacheVault/MaxCache supercapacitor which is required to protect the write cache).
	* Reduces the OSD and placement group count by 5x or more which allows the cluster to scale that much bigger and with reduced complexity (a cluster with 100,000 PGs use less RAM per ~~appliance~~ and is easier for Ceph monitors to manage than 500,000 PGs).		* Reduces the OSD and placement group count by 5x or more which allows the cluster to scale that much bigger and with reduced complexity (a cluster with 100,000 PGs use less RAM per system and is easier for Ceph monitors to manage than 500,000 PGs).

	Besides the nominal extra cost associated with a SATA/SAS RAID Controller vs a SATA/SAS HBA we see few benefits and many drawbacks to using HBAs with Ceph. Some in the Ceph community prefer to let bad OSD devices fail in-place never to be replaced as a maintenance strategy. Our preference is to replace bad HDDs to maintain 100% of the designed capacity for the life of the cluster. Hardware RAID makes that especially easy and QuantaStor has integrated management for all major RAID controller models (and HBAs) via the Web UI (and CLI/REST). QuantaStor's web UI also has an enclosure management view so that it's easy to identify which drive is bad and where it is located in the server chassis.		Besides the nominal extra cost associated with a SATA/SAS RAID Controller vs a SATA/SAS HBA we see few benefits and many drawbacks to using HBAs with Ceph. Some in the Ceph community prefer to let bad OSD devices fail in-place never to be replaced as a maintenance strategy. Our preference is to replace bad HDDs to maintain 100% of the designed capacity for the life of the cluster. Hardware RAID makes that especially easy and QuantaStor has integrated management for all major RAID controller models (and HBAs) via the Web UI (and CLI/REST). QuantaStor's web UI also has an enclosure management view so that it's easy to identify which drive is bad and where it is located in the server chassis.