Template:About Ceph: Difference between revisions

From OSNEXUS Online Documentation Site
Jump to navigation Jump to search
VisualWikitext
Created page with "== Ceph Cluster == A ceph cluster is a group of three or more appliances that have been clustered together using the ceph storage technology. Ceph requires a minimum of thre..."
 
 
(89 intermediate revisions by the same user not shown)
Line 1: Line 1:
== Ceph Cluster ==
= Ceph Terminology & Concepts =


A ceph cluster is a group of three or more appliances that have been clustered together using the ceph storage technology. Ceph requires a minimum of three nodes to create a cluster in order to establish a quorum[https://en.wikipedia.org/wiki/Quorum_(distributed_computing)].
This section will introduce Ceph terms and concepts to familiarize oneself with to become more proficient with Ceph Cluster administration in QuantaStor. This discussion will address general concepts surrounding Ceph. To implement Ceph in QuantaStor it is recommended to see '''Getting Started''' to quickly implement Ceph. '''Getting Started''' can be found at Storage Management --> Storage System --> Storage System --> Getting Started (toolbar).


In QuantaStor, Appliances must first be Grid members before they can be added to, or create, a Ceph cluster. In the above diagram, the QuantaStor Grid is also the Ceph Cluster. Note that when the Ceph Cluster is initially created there is no storage associated with it (OSDs), only monitors.
[https://wiki.osnexus.com/index.php?title=%2B_Getting_Started_Overview Getting Started in Administration guide.]


== Ceph Monitor ==
[https://docs.ceph.com/en/latest/start/intro/ Further Information: Introduction to Ceph].


The Ceph Monitors form a [https://en.wikipedia.org/wiki/Paxos_%28computer_science%29 paxos] [http://research.microsoft.com/en-us/um/people/lamport/pubs/pubs.html#lamport-paxos part-time parliment] cluster for the management of cluster membership, configuration information, and state.  Paxos is an algorithm (developed by Leslie Lamport in the late 80s) which uses a three-phase consensus protocol to ensure that cluster updates can be done in a fault-tolerant timely fashion even in the event of a node outage or node that is acting improperly.  Ceph uses the algorithm so that the membership, configuration and state information is updated safely across the cluster in an efficient manner.  Since the algorithm requires a quorum of nodes to agree on any given change an odd number of appliances (three or more) are required for any given Ceph cluster deployment. 
== Ceph Cluster ==


During initial Ceph cluster creation, QuantaStor will configure the first three Appliances to have active Ceph Monitor services.  Configurations with more than 16 nodes should add at least two additional monitors.  On configurations with more than 16 nodes two additional monitors should be setup can this can be done through the QuantaStor WebUI in the '''Scale-out Block & Object''' section.   
A Ceph Cluster is a group of three or more systems that have been clustered together using the Ceph storage technology.  Ceph requires a minimum of three nodes to create a cluster so that quorum may be established across the Ceph Monitors.  [https://en.wikipedia.org/wiki/Quorum_(distributed_computing) Wikipedia Quorum (distributed computing)].


Monitors startup automatically when the appliance starts and the status and health of monitors is monitored by QuantaStor and displayed in the WebUI.  A minimum of two ceph monitors must be online at all times so in a three node configuration two of the three appliances must be online for the storage to be online and available.
In QuantaStor based Ceph configurations, QuantaStor systems must first be combined into a Storage Grid.  After the Storage Grid is formed one more more Ceph Clusters may be created within the Storage Grid.  In the example above the Storage Grid is comprised of a single Ceph Cluster.  When the Ceph Cluster is initially created QuantaStor automatically deploys 3x Ceph Monitors within the new Ceph Cluster.


When a cluster is initially created, QuantaStor configures the first three appliances to have active Ceph Monitor services running. 
== Ceph Monitor ==


== Ceph Object Storage Daemon / OSD ==
The Ceph Monitors form a [https://en.wikipedia.org/wiki/Paxos_%28computer_science%29 Paxos] [http://research.microsoft.com/en-us/um/people/lamport/pubs/pubs.html#lamport-paxos The Part-Time Parliament] cluster for the management of cluster membership, configuration information, and state.  Paxos is an algorithm (developed by Leslie Lamport in the late 80s) which uses a three-phase consensus protocol to ensure that cluster updates can be done in a fault-tolerant timely fashion even in the event of a node outage or node that is acting improperly.  Ceph uses the algorithm so that the membership, configuration and state information is updated safely across the cluster in an efficient manner.  Since the algorithm requires a quorum of nodes to agree on any given change an odd number of systems (three or more) are required for any given Ceph cluster deployment. 


The Ceph '''Object Storage Daemon''', known as the OSD, is a daemon process that reads and writes data, representing the actual data storage containers.  When a client writes data to a Ceph based iSCSI/RBD block device, or via the S3/SWIFT gateway, the data is spread out across the OSDs in the cluster automatically.
During initial Ceph cluster creation, QuantaStor will configure the first three systems to have active Ceph Monitor services.  Configurations with more than 16 nodes should add two additional monitors.  This can be done through the QuantaStor web user interface in the '''Scale-out Storage Configuration''' section.   


QuantaStor Scale-out SAN with Ceph deployments must have at least 3x OSDs per appliance, making 9x OSDs total the minimum number of Daemons.  Each OSD is attached to one XFS-based QuantaStor Storage Pool.  QuantaStor requires the use of XFS Storage Pools for use as Ceph OSDs due to extended attribute requirements. Each OSD is also assigned one Journal Device. 
In a Ceph cluster with 3x monitors a minimum of 2x monitors must be online at all times.  If only one monitor (or none) are running then storage access will is automatically disabled until quorum among monitors may be reestablished.  In larger clusters with 5x monitors then 3x monitors must be online at all times to maintain quorum and storage accessibility.


Because the creation of OSDs, the underlying Storage Pools for them, and their associated Journal devices is a multi-step process QuantaStor has a '''Multi-OSD Create''' configuration dialog which does all of these configuration steps for an entire cluster in a single dialog. This makes it easy to setup even hyper-scale Ceph deployments in minutes.
== Ceph Object Storage Daemon / OSD ==
[[File:Create OSDs & Journals - OSD Devices.jpg|712px]] [[File:Create OSDs & Journals - Journal Devices.jpg.jpg|712px]]


== Ceph Journals and Journal Devices ==
'''Navigation:''' Scale-out Storage Configuration --> Data & Journal Devices --> Data & Journal Devices --> Create OSDs & Journals ''(toolbar)''


[[File:qs4_ceph_journal_osd.png|600px|right]]
The Ceph '''Object Storage Daemon''', known as the OSD, is a daemon process that reads and writes data and generally maps 1-to-1 to a HDD or a SSD device.  OSD devices may be used by multiple Storage Pools so after the OSDs are added one may allocate pools for file, block, and object storage which all use the available OSDs in the cluster to store their data.


It is important to note that, in ceph, writes are never cached.  This is good because it ensures that every write is written to stable media (the disk devices) before Ceph acknowledges to a client that the write is complete.  This prevents corruption in the event of a power outage because data that is being written out is only complete once it is on stable media. The cluster will work around the bad node until it comes back online and re-synchronizes with the cluster.
QuantaStor Scale-out SAN with Ceph deployments must have at least 3x OSDs per system, making 9x OSDs total the minimum number OSDs.  QuantaStor 5 and newer versions use the BlueStore OSD storage back-end.  For additional BlueStore information see, [https://ceph.io/community/new-luminous-bluestore/ New in Luminous: BlueStore.]


The trade-off to never caching is a loss of write performance. HDDs are slow due to rotational latency and seek times for spinning disk are high. The solution is to log every write to fast persistent solid state media (SSD, NVMe, X Point, NVDIMM, etc), called a '''journal device'''.
'''N.B.,''' for ease of use there is an Auto Config button that will optimize selection of available devices.


Using a fast '''journal device''' allows Ceph to initially write data to the journal, returning a "write complete" to the client much faster, even though the data has not yet been written to the slower HDDs.  By retaining the data on the journal until it can be written to the HDDs, the data can be recovered automatically in the event of a power outage.  Ceph retains a copy of the data in RAM and uses that to write lazily to the HDD.  This means that the journal device is only used as a write log and will never be read from unless a recovery scenario is encountered.
== Journal Groups ==


Because the '''journal device''' will encounter high, sustained write-pressure, Datacenter grade or Enterprise grade SSDs must be used for Ceph journal devices. Desktop SSD devices are fast for a few seconds, but their write performance drops significantly.  Additionally they wear out very quickly.  As a result OSNEXUS will not certify the use of desktop SSDs in any production deployment of any kind.  By example, we tested with a popular desktop SSD device which produces 600MB/sec but performance quickly dropped to just 30MB/sec after just a few seconds of sustained write load.
'''Journal Groups''' are used to boost the performance of OSDs. Each '''Journal Group''' can provide a performance boost for 5x to 30x OSD devices depending on the speed of the storage media used to create a given Journal Group.   Journal Groups are typically created using a pair of SSDs which QuantaStor combines into software RAID1 mirror.  Once created Journal Groups provide high performance, low latency, storage from which Ceph Journal Devices may be provisioned and attached to new OSDs to boost performance. Because '''Journal Groups''' must sustain high write loads over a period of years '''only datacenter (DC) grade / enterprise grade flash media''' should be used to create them.
Journal Groups can be created using all types of flash storage media including NVMe, PMEM, SATA SSD, or SAS SSD.


QuantaStor's WebUI will allow any block device in the Physical Disks section to be selected as a Journal Device, but only Enterprise SSD, NVMe or PCI SSD devices with high performance and write-endurance should be used.
Journal Devices are provisioned from Journal Groups.  Journal Devices come in two types, Write-Ahead-Log (WAL) devices and Meta-data DB (MDB) devices.  QuantaStor automatically provisions WAL devices to be 2GB in size and MDB devices can be 3GB, 30GB (default), or 300GB in size.


Once selected to be a Journal Device, QuantaStor will automatically partition it up into 8 journal partitions.  One journal is required for each Object Storage Daemon (OSD), meaning each Journal Device can support up to 8 OSDs.  For an appliance with 20x OSDs one would want at least 3x SSDs, but in reality the use of additional SSDs would be recommended in order to ensure even distribution of load across Journal Devices.
Journal Groups are not required but are highly recommended when creating HDD based OSDs.  With SSD based OSDs it is not recommended to assign them external WAL and MDB devices from Journal Groups.  Rather the MDB and WAL storage for SSDs will be allocated out of a small portion of the underlying OSD data device.  With platter/HDD based OSDs we highly recommend the creation of Journal Groups so that each OSD can have both an external WAL device and a external MDB device.


A hardware RAID controller can be used to combine multiple SSDs (using RAID5) to make a high-performance fault-tolerant journal.  We recommend using PCI SSD, NVMe devices, or multiple SSDs in RAID5 or a pair in RAID1.  A dedicated RAID controller can also be used to create a RAID5 based SSD journal devices to further boost performance.  Again, each of these logical or physical devices is sliced up into 8x journal partitions so that up to 8x OSDs can be supported per Journal Device.
NVMe and 3D XPoint flash storage media are the best storage types for creating Journal Groups due to their high throughput and IOPS performance.  We recommend allocating 100MB/sec and 32GB of capacity for each HDD based OSD.  For example, a system with 60x HDD based OSDs would require 60x100MB/sec or 6000MB/sec of Journal Group throughput.  If NVMe devices are selected that can do 2000MB/sec then three Journal Groups will be required and a total of 6x NVMe SSDs (3x RAID1 Journal Groups).  Capacity wise 60x HDDs will require 60x32GB of storage for all the WAL and MDB devices to be created or what amounts to 1.92TB of provisionable Journal Group capacity.  One possible design to meet both the performance and the capacity requirements would be to make the 3x Journal Groups using a total of 6x 800GB NVMe devices with a 3x DWPD endurance.


== Placement Group / PG ==
[[File:ceph_journal_group.png|512px|thumb|Journal Devices (WAL and MDB) are provisioned from Journal Groups]]


Ceph uses Placement Groups to implement mirroring (or erasure coding) of data across OSDs according to the configured replica count for a given Ceph Pool.
=== Write-Ahead Log (WAL) Journal Devices ===
WAL '''Journal Devices''' are provisioned from '''Journal Groups''' and are then attached to new OSDs when they are created.  WAL devices accelerate write performance.  When a write request is received by an OSD it is able to write the data to low-latency stable flash media very quickly to complete the write.  Data can then be written lazily to the HDD as time allows without risk of losing data due to a sudden system power outage.  


The user specifies how many copies of the data must be maintained by the Ceph Pool during creation to ensure a level of high-availability and fault-tolerance, usually 2 copies when using hardware RAID or 3 copies when no disk-level RAID is present.  Ceph in turn creates a series of Placement Groups as directed by QuantaStor to be associated with the Ceph Pool. 
=== Meta-data Database (MDB) Journal Devices ===


One way to think of the placement groups is as logical mini-mirrors in a RAID10 configuration.  Each placement group is either a two-way, three-way or 4-way mirror across 2, 3, or 4 OSDs respectively.  Because the number of OSDs will grow over the life of the cluster, QuantaStor allocates a large number of PGs for each Ceph Pool to evenly distribute data across the OSDs and accommodate future expansion as OSDs are added.  In this way Ceph can very efficiently re-organize and re-balance PGs to mirror across new OSDs as they are added.
MDB '''Journal Devices''' effectively boost both read and write performance as they contain all the Bluestore filesystem metadata.  Rather than having to write small blocks of metadata to HDDs which have low IOPS performance and external MDB device on flash media can sustain high IOPS loads and in turn greatly boosts performance.


The PG count stays fixed as OSDs are added but a maintenance command can be run to increase the PG count for a Ceph Pool if the PG count gets low relative to the number of OSDs in the Pool.  In general the PG count should be roughly 10x to 100x higher than the OSD count for a given Ceph Pool.
== Hardware RAID ==
Although Hardware RAID may be used in Ceph Clusters as an underlying storage abstraction for OSDs '''it is generally not recommended'''.  It does have applications in very large Ceph clusters (ie. 1000s of OSDs) and with clusters comprised of servers with limited RAM and CPU core count.  Roughly speaking each OSD requires approximately 2GB of RAM and a 1GHz fractional CPU core.  A server with 60x HDD based OSDs will require a large dual-processor configuration and 192GB of RAM.  By combining disks using HW RAID these requirements are reduced 5:1.  QuantaStor does have integrated hardware RAID management and monitoring to manage configurations that use hardware RAID.  But again, '''we do not recommend the use of HW RAID except in specialized configurations and in hyper-scale configurations'''.


Similar to RAID10 technology, a PG can become degraded if one or more copies is offline.  Ceph is designed to keep running in a degraded state when copies are lost, so whole systems can go offline without any disruption to clients accessing the cluster.  Ceph also automatically repairs and updates the offline PGs once the offline OSDs come back online online and if the offline appliance doesn't come back online in a reasonable amount of time the cluster will auto heal itself by adjusting the PGs, swapping out the offline OSDs with good online OSDs.  In this way a cluster will automatically heal a Ceph Pool back to 100% automatically (ie, return to full/complete copy count). 
== Ceph Placement Groups (PGs) ==
 
Also, if an OSD is explicitly removed, the PGs referencing it are re-balanced and re-organized across the remaining OSDs to recover the system back to 100% health on the remaining OSDs.
 
== Object Storage Group (QuantaStor specific) ==
 
S3/SWIFT object storage gateways require the creation and management of several Ceph Pools, which together represent a region+zone for the storage of objects and buckets.  QuantaStor groups all the Ceph Pools used to manage a given object storage configuration into a Object Storage Group.  QuantaStor also automatically deploys and manages Ceph S3/SWIFT Object Gateways on all appliances in the cluster that were selected as gateway nodes when the Object Storage Group was created.  Additional gateways can be deployed on new or existing nodes at any time. 
 
Note that Object Storage Groups are a QuantaStor construct so you won't find documentation about it in general Ceph documentation.
 
=== User Object Access Entries ===
 
Access to object storage via S3 and SWIFT requires a Access Key and a Secret Key just like with Amazon S3 storage.  Each User Object Access Entry is a Access Key + Secret Key pair which is associated with a Ceph Cluster and Object Storage Group.  You must allocate at least one User Object Access Entry to read/write buckets and objects to an Object Storage Group via the Ceph S3/SWIFT Gateway.


Ceph Pools do not write data directly to OSDs, rather there is an abstraction layer between each Ceph Pool and the OSDs comprised of '''Placement Groups, PGs'''.  Each PG can be thought of as a logical stripe across a group of OSDs.  Ceph Pools created with a replica=2 storage layout will have PGs that each reference 2x OSDs.  Similarly a Ceph Pool with an erasure-coding layout of K8+2M would have PGs that each span 10x OSDs. 
When creating new File, Block, or Object Storage Pools with QuantaStor you have control over the number of PGs to be created using the Scaling Factor option.  If a given Ceph Cluster is to be used for a single type of storage such as File or Object then one would set the Scaling Factor to 100%.  If it is expected that a given Ceph Cluster will be used for 30% Object storage and 70% File storage then those Storage Pools should be allocated with those Scaling Factors respectively.  Your choice for the Scaling Factor for any given pool should be a best guess.  The PG count can be adjusted later to provide better optimization of storage distribution and balancing across the OSDs in the future if required.


== Ceph CRUSH Maps and Resource Domains ==
== Ceph CRUSH Maps and Resource Domains ==


Ceph supports the ability to organize placement groups, which provide data mirroring across OSDs, so that high-availability and fault-tolerance can be maintained even in the event of a rack or site outage.  By defining ''failure-domains'', such as a Rack of appliances, a Site, or Building, a map can be created so that Placement Groups are intelligently laid out to ensure high-availability despite the outage of one or more failure-domains, depending on the level of redundancy.
Ceph supports the ability to organize placement groups, which provide data mirroring across OSDs, so that high-availability and fault-tolerance can be maintained even in the event of a rack or site outage.  By defining ''failure-domains'', such as a Rack of systems, a Site, or Building, a map can be created so that Placement Groups are intelligently laid out to ensure high-availability despite the outage of one or more failure-domains, depending on the level of redundancy.


This intelligent map is called the '''Ceph CRUSH map''', standing for Controlled, Scalable, Decentralized Placement of Replicated Data, and it defines how to mirror data in the Ceph cluster to ensure optimal performance and availability.
This intelligent map is called the '''Ceph CRUSH map''' ('''C'''ontrolled '''R'''eplication '''U'''nder '''S'''calable '''H'''ashing), standing for Controlled, Scalable, Decentralized Placement of Replicated Data, and it defines how to mirror data in the Ceph cluster to ensure optimal performance and availability.


Creating CRUSH maps manually can be a complex process, so QuantaStor creates and configures CRUSH maps automatically, saving a large degree of administrative overhead.  To facilitate automatic CRUSH map management, detail regarding where each QuantaStor appliance is deployed must be provided.  This is done by creating a tree of ''Resource Domains'' via the WebUI (or via CLI/REST APIs) to organize the Appliances in a given QuantaStor Grid into Racks, Sites, and Buildings.  QuantaStor uses this information to automatically generate an optimal CRUSH map when pools are provisioned, ensuring optimal performance and high-availability.   
Creating CRUSH maps manually can be a complex process, so QuantaStor creates and configures CRUSH maps automatically, saving a large degree of administrative overhead.  To facilitate automatic CRUSH map management, detail regarding where each QuantaStor system is deployed must be provided.  This is done by creating a tree of ''Resource Domains'' via the WebUI (or via CLI/REST APIs) to organize the systems in a given QuantaStor Grid into Racks, Sites, and Buildings.  QuantaStor uses this information to automatically generate an optimal CRUSH map when pools are provisioned, ensuring optimal performance and high-availability.   


Custom CRUSH map changes can still be made to adjust the map after the pool(s) are created and OSNEXUS provides consulting services to meet special requirements.  Resource Domains are a QuantaStor construct so you will not find mention of them in general Ceph documentation, but they map closely to the CRUSH bucket hierarchy.
Custom CRUSH map changes can still be made to adjust the map after the pool(s) are created and OSNEXUS provides consulting services to meet special requirements.  Resource Domains are a QuantaStor construct so you will not find mention of them in general Ceph documentation, but they map closely to the CRUSH bucket hierarchy.
For additional information see, [https://docs.ceph.com/en/quincy/rados/operations/crush-map/#crush-maps CRUSH MAPS]

Latest revision as of 20:43, 16 January 2024

Ceph Terminology & Concepts

This section will introduce Ceph terms and concepts to familiarize oneself with to become more proficient with Ceph Cluster administration in QuantaStor. This discussion will address general concepts surrounding Ceph. To implement Ceph in QuantaStor it is recommended to see Getting Started to quickly implement Ceph. Getting Started can be found at Storage Management --> Storage System --> Storage System --> Getting Started (toolbar).

Getting Started in Administration guide.

Further Information: Introduction to Ceph.

Ceph Cluster

A Ceph Cluster is a group of three or more systems that have been clustered together using the Ceph storage technology. Ceph requires a minimum of three nodes to create a cluster so that quorum may be established across the Ceph Monitors. Wikipedia Quorum (distributed computing).

In QuantaStor based Ceph configurations, QuantaStor systems must first be combined into a Storage Grid. After the Storage Grid is formed one more more Ceph Clusters may be created within the Storage Grid. In the example above the Storage Grid is comprised of a single Ceph Cluster. When the Ceph Cluster is initially created QuantaStor automatically deploys 3x Ceph Monitors within the new Ceph Cluster.

Ceph Monitor

The Ceph Monitors form a Paxos The Part-Time Parliament cluster for the management of cluster membership, configuration information, and state. Paxos is an algorithm (developed by Leslie Lamport in the late 80s) which uses a three-phase consensus protocol to ensure that cluster updates can be done in a fault-tolerant timely fashion even in the event of a node outage or node that is acting improperly. Ceph uses the algorithm so that the membership, configuration and state information is updated safely across the cluster in an efficient manner. Since the algorithm requires a quorum of nodes to agree on any given change an odd number of systems (three or more) are required for any given Ceph cluster deployment.

During initial Ceph cluster creation, QuantaStor will configure the first three systems to have active Ceph Monitor services. Configurations with more than 16 nodes should add two additional monitors. This can be done through the QuantaStor web user interface in the Scale-out Storage Configuration section.

In a Ceph cluster with 3x monitors a minimum of 2x monitors must be online at all times. If only one monitor (or none) are running then storage access will is automatically disabled until quorum among monitors may be reestablished. In larger clusters with 5x monitors then 3x monitors must be online at all times to maintain quorum and storage accessibility.

Ceph Object Storage Daemon / OSD

Navigation: Scale-out Storage Configuration --> Data & Journal Devices --> Data & Journal Devices --> Create OSDs & Journals (toolbar)

The Ceph Object Storage Daemon, known as the OSD, is a daemon process that reads and writes data and generally maps 1-to-1 to a HDD or a SSD device. OSD devices may be used by multiple Storage Pools so after the OSDs are added one may allocate pools for file, block, and object storage which all use the available OSDs in the cluster to store their data.

QuantaStor Scale-out SAN with Ceph deployments must have at least 3x OSDs per system, making 9x OSDs total the minimum number OSDs. QuantaStor 5 and newer versions use the BlueStore OSD storage back-end. For additional BlueStore information see, New in Luminous: BlueStore.

N.B., for ease of use there is an Auto Config button that will optimize selection of available devices.

Journal Groups

Journal Groups are used to boost the performance of OSDs. Each Journal Group can provide a performance boost for 5x to 30x OSD devices depending on the speed of the storage media used to create a given Journal Group. Journal Groups are typically created using a pair of SSDs which QuantaStor combines into software RAID1 mirror. Once created Journal Groups provide high performance, low latency, storage from which Ceph Journal Devices may be provisioned and attached to new OSDs to boost performance. Because Journal Groups must sustain high write loads over a period of years only datacenter (DC) grade / enterprise grade flash media should be used to create them. Journal Groups can be created using all types of flash storage media including NVMe, PMEM, SATA SSD, or SAS SSD.

Journal Devices are provisioned from Journal Groups. Journal Devices come in two types, Write-Ahead-Log (WAL) devices and Meta-data DB (MDB) devices. QuantaStor automatically provisions WAL devices to be 2GB in size and MDB devices can be 3GB, 30GB (default), or 300GB in size.

Journal Groups are not required but are highly recommended when creating HDD based OSDs. With SSD based OSDs it is not recommended to assign them external WAL and MDB devices from Journal Groups. Rather the MDB and WAL storage for SSDs will be allocated out of a small portion of the underlying OSD data device. With platter/HDD based OSDs we highly recommend the creation of Journal Groups so that each OSD can have both an external WAL device and a external MDB device.

NVMe and 3D XPoint flash storage media are the best storage types for creating Journal Groups due to their high throughput and IOPS performance. We recommend allocating 100MB/sec and 32GB of capacity for each HDD based OSD. For example, a system with 60x HDD based OSDs would require 60x100MB/sec or 6000MB/sec of Journal Group throughput. If NVMe devices are selected that can do 2000MB/sec then three Journal Groups will be required and a total of 6x NVMe SSDs (3x RAID1 Journal Groups). Capacity wise 60x HDDs will require 60x32GB of storage for all the WAL and MDB devices to be created or what amounts to 1.92TB of provisionable Journal Group capacity. One possible design to meet both the performance and the capacity requirements would be to make the 3x Journal Groups using a total of 6x 800GB NVMe devices with a 3x DWPD endurance.

Journal Devices (WAL and MDB) are provisioned from Journal Groups

Write-Ahead Log (WAL) Journal Devices

WAL Journal Devices are provisioned from Journal Groups and are then attached to new OSDs when they are created. WAL devices accelerate write performance. When a write request is received by an OSD it is able to write the data to low-latency stable flash media very quickly to complete the write. Data can then be written lazily to the HDD as time allows without risk of losing data due to a sudden system power outage.

Meta-data Database (MDB) Journal Devices

MDB Journal Devices effectively boost both read and write performance as they contain all the Bluestore filesystem metadata. Rather than having to write small blocks of metadata to HDDs which have low IOPS performance and external MDB device on flash media can sustain high IOPS loads and in turn greatly boosts performance.

Hardware RAID

Although Hardware RAID may be used in Ceph Clusters as an underlying storage abstraction for OSDs it is generally not recommended. It does have applications in very large Ceph clusters (ie. 1000s of OSDs) and with clusters comprised of servers with limited RAM and CPU core count. Roughly speaking each OSD requires approximately 2GB of RAM and a 1GHz fractional CPU core. A server with 60x HDD based OSDs will require a large dual-processor configuration and 192GB of RAM. By combining disks using HW RAID these requirements are reduced 5:1. QuantaStor does have integrated hardware RAID management and monitoring to manage configurations that use hardware RAID. But again, we do not recommend the use of HW RAID except in specialized configurations and in hyper-scale configurations.

Ceph Placement Groups (PGs)

Ceph Pools do not write data directly to OSDs, rather there is an abstraction layer between each Ceph Pool and the OSDs comprised of Placement Groups, PGs. Each PG can be thought of as a logical stripe across a group of OSDs. Ceph Pools created with a replica=2 storage layout will have PGs that each reference 2x OSDs. Similarly a Ceph Pool with an erasure-coding layout of K8+2M would have PGs that each span 10x OSDs. When creating new File, Block, or Object Storage Pools with QuantaStor you have control over the number of PGs to be created using the Scaling Factor option. If a given Ceph Cluster is to be used for a single type of storage such as File or Object then one would set the Scaling Factor to 100%. If it is expected that a given Ceph Cluster will be used for 30% Object storage and 70% File storage then those Storage Pools should be allocated with those Scaling Factors respectively. Your choice for the Scaling Factor for any given pool should be a best guess. The PG count can be adjusted later to provide better optimization of storage distribution and balancing across the OSDs in the future if required.

Ceph CRUSH Maps and Resource Domains

Ceph supports the ability to organize placement groups, which provide data mirroring across OSDs, so that high-availability and fault-tolerance can be maintained even in the event of a rack or site outage. By defining failure-domains, such as a Rack of systems, a Site, or Building, a map can be created so that Placement Groups are intelligently laid out to ensure high-availability despite the outage of one or more failure-domains, depending on the level of redundancy.

This intelligent map is called the Ceph CRUSH map (Controlled Replication Under Scalable Hashing), standing for Controlled, Scalable, Decentralized Placement of Replicated Data, and it defines how to mirror data in the Ceph cluster to ensure optimal performance and availability.

Creating CRUSH maps manually can be a complex process, so QuantaStor creates and configures CRUSH maps automatically, saving a large degree of administrative overhead. To facilitate automatic CRUSH map management, detail regarding where each QuantaStor system is deployed must be provided. This is done by creating a tree of Resource Domains via the WebUI (or via CLI/REST APIs) to organize the systems in a given QuantaStor Grid into Racks, Sites, and Buildings. QuantaStor uses this information to automatically generate an optimal CRUSH map when pools are provisioned, ensuring optimal performance and high-availability.

Custom CRUSH map changes can still be made to adjust the map after the pool(s) are created and OSNEXUS provides consulting services to meet special requirements. Resource Domains are a QuantaStor construct so you will not find mention of them in general Ceph documentation, but they map closely to the CRUSH bucket hierarchy.

For additional information see, CRUSH MAPS

Retrieved from "https://wiki.osnexus.com/index.php?title=Template:About_Ceph&oldid=24420"