Template:About Ceph
Ceph Cluster
A ceph cluster is a group of three or more systems that have been clustered together using the ceph storage technology. Ceph requires a minimum of three nodes to create a cluster which in turn establishes a quorum, Wikipedia Quorum (distributed computing).
In QuantaStor, systems must first be Grid members before they can be added to, or create, a Ceph cluster. In the above diagram, the QuantaStor Grid is also the Ceph Cluster. Note that when the Ceph Cluster is initially created there is no storage associated with it (OSDs), only monitors.
Ceph Monitor
The Ceph Monitors form a paxos part-time parliment cluster for the management of cluster membership, configuration information, and state. Paxos is an algorithm (developed by Leslie Lamport in the late 80s) which uses a three-phase consensus protocol to ensure that cluster updates can be done in a fault-tolerant timely fashion even in the event of a node outage or node that is acting improperly. Ceph uses the algorithm so that the membership, configuration and state information is updated safely across the cluster in an efficient manner. Since the algorithm requires a quorum of nodes to agree on any given change an odd number of systems (three or more) are required for any given Ceph cluster deployment.
During initial Ceph cluster creation, QuantaStor will configure the first three systems to have active Ceph Monitor services. Configurations with more than 16 nodes should add at least two additional monitors. On configurations with more than 16 nodes two additional monitors should be setup. This can be done through the QuantaStor WebUI in the Scale-out Storage Configuration section.
Monitors startup automatically when the system starts. The status and health of monitors is monitored by QuantaStor, then displayed in the WebUI. A minimum of two ceph monitors must be online at all times. As an example, in a three node configuration two of the three systems must be online for the storage to be online and available.
When a cluster is initially created, QuantaStor configures the first three systems to have active Ceph Monitor services running.
Ceph Object Storage Daemon / OSD
The Ceph Object Storage Daemon, known as the OSD, is a daemon process that reads and writes data, representing the actual data storage containers. When a client writes data to a Ceph based iSCSI/RBD block device, or via the S3 gateway, the data is spread out across the OSDs in the cluster automatically.
QuantaStor Scale-out SAN with Ceph deployments must have at least 3x OSDs per system, making 9x OSDs total the minimum number of Daemons. Each OSD is attached to one BlueStore-based QuantaStor Storage Pool. QuantaStor requires the use of BlueStore Storage Pools for use as Ceph OSDs due to extended attribute requirements. Each OSD is also assigned one Journal Device.
Because the creation of OSDs, the underlying Storage Pools for them, and their associated Journal devices is a multi-step process QuantaStor has a Multi-Create configuration dialog which does all of these configuration steps for an entire cluster in a single dialog. This makes it easy to setup even hyper-scale Ceph deployments in minutes.
For additional BlueStore information see, New in Luminous: BlueStore.
Ceph Journals and Journal Devices
It is important to note that part of Ceph's design is to never cache writes. This is good and important because it ensures that every write is written to stable media (disk or SSD media) before Ceph acknowledges to a client that the write is complete. This applies to all writes irrespective of whether file, block, or object storage is configured. This design feature prevents corruption in the event of a power outage because the write transaction is only complete once the data is on stable media with redundancy. In the event of a system failure the cluster will automatically work around the bad node (essentially its collection of OSDs) until it comes back online and re-synchronizes with the cluster.
The trade-off to never caching writes is a loss of write performance, especially with spinning media. Hard-drives are slow due to rotational latency and seek times for spinning disk are high. The solution is log writes to very fast persistent solid state media (SSD, NVMe, XPoint, NVDIMM, etc). This write log is called a journal device and sometime a WAL device. (Technically in the Ceph architecture these are separate things but in practice the same media is used for both.)
Using a fast journal device allows Ceph to initially write data to the journal, returning a "write complete" to the client much much faster. Even though the data has not yet been written to the slower HDDs at that stage the data is on stable media so in the event of a power outage the log is automatically used to recover the in-flight writes. Ceph retains a copy of the data in RAM and uses that to write lazily to the HDD. This means that the journal device is only used as a write log and will never be read from unless a recovery scenario is encountered.
Because the journal device will encounter high, sustained write-pressure, Datacenter grade or Enterprise grade SSDs must be used for Ceph journal devices. NVMe and Optane based flash storage makes for the best journal devices. (Note: Desktop grade SSD devices generally do not have the necessary sustained write performance nor the endurance required to be used as a log device so they're unsuitable. As such OSNEXUS will not certify the use of any desktop media in any production deployment of any kind.)
Avoid Hardware RAID
As of QuantaStor 5 we no longer recommend the use of hardware RAID with Ceph configurations. It made sense to use it up until the Ceph Jewel release where the underlying storage used the FileStore layout (XFS based) because the hardware RAID helped improve journal performance. As of QuantaStor 5, the BlueStore layout is used exclusively so the use of hardware RAID is no longer needed or recommended.
Placement Group / PG
Ceph uses Placement Groups, PGs, to implement mirroring (or erasure coding) of data across OSDs according to the configured replica count for a given Ceph Pool.
The user specifies how many copies of the data must be maintained by the Ceph Pool during creation to ensure a level of high-availability and fault-tolerance, usually 2 copies when using hardware RAID or 3 copies when no disk-level RAID is present. Ceph in turn creates a series of Placement Groups as directed by QuantaStor to be associated with the Ceph Pool.
One way to think of the placement groups is as logical mini-mirrors in a RAID10 configuration. Each placement group is either a two-way, three-way or 4-way mirror across 2, 3, or 4 OSDs respectively. Because the number of OSDs will grow over the life of the cluster, QuantaStor allocates a large number of PGs for each Ceph Pool to evenly distribute data across the OSDs and accommodate future expansion as OSDs are added. In this way Ceph can very efficiently re-organize and re-balance PGs to mirror across new OSDs as they are added.
The PG count stays fixed as OSDs are added but a maintenance command can be run to increase the PG count for a Ceph Pool if the PG count gets low relative to the number of OSDs in the Pool. In general the PG count should be roughly 10x to 100x higher than the OSD count for a given Ceph Pool.
Similar to RAID10 technology, a PG can become degraded if one or more copies is offline. Ceph is designed to keep running in a degraded state when copies are lost, so whole systems can go offline without any disruption to clients accessing the cluster. Ceph also automatically repairs and updates the offline PGs once the offline OSDs come back online online and if the offline system doesn't come back online in a reasonable amount of time the cluster will auto heal itself by adjusting the PGs, swapping out the offline OSDs with good online OSDs. In this way a cluster will automatically heal a Ceph Pool back to 100% automatically (ie, return to full/complete copy count).
Also, if an OSD is explicitly removed, the PGs referencing it are re-balanced and re-organized across the remaining OSDs to recover the system back to 100% health on the remaining OSDs.
Object Storage Zone
S3 object storage gateways require the creation and management of several Ceph Pools, which together represent a region+zone for the storage of objects and buckets. QuantaStor groups all the Ceph Pools used to manage a given object storage configuration into a Object Storage Group or Zone. QuantaStor also automatically deploys and manages Ceph S3 Object Gateways on all systems in the cluster that were selected as gateway nodes when the Object Storage Group was created. Additional gateways can be deployed on new or existing nodes at any time via the web UI, cli or REST API.
For additional information see Wikipedia, Ceph (software)
User Object Access Entries
Access to object storage via S3 requires a Access Key and a Secret Key just as with Amazon S3 storage. Each User Object Access Entry is an Access Key + Secret Key pair which is associated with a Ceph Cluster and Object Storage Group. You must allocate at least one User Object Access Entry to read/write buckets and objects to an Object Storage Group via the Ceph S3 Gateway.
Ceph CRUSH Maps and Resource Domains
Ceph supports the ability to organize placement groups, which provide data mirroring across OSDs, so that high-availability and fault-tolerance can be maintained even in the event of a rack or site outage. By defining failure-domains, such as a Rack of systems, a Site, or Building, a map can be created so that Placement Groups are intelligently laid out to ensure high-availability despite the outage of one or more failure-domains, depending on the level of redundancy.
This intelligent map is called the Ceph CRUSH map, standing for Controlled, Scalable, Decentralized Placement of Replicated Data, and it defines how to mirror data in the Ceph cluster to ensure optimal performance and availability.
Creating CRUSH maps manually can be a complex process, so QuantaStor creates and configures CRUSH maps automatically, saving a large degree of administrative overhead. To facilitate automatic CRUSH map management, detail regarding where each QuantaStor system is deployed must be provided. This is done by creating a tree of Resource Domains via the WebUI (or via CLI/REST APIs) to organize the systems in a given QuantaStor Grid into Racks, Sites, and Buildings. QuantaStor uses this information to automatically generate an optimal CRUSH map when pools are provisioned, ensuring optimal performance and high-availability.
Custom CRUSH map changes can still be made to adjust the map after the pool(s) are created and OSNEXUS provides consulting services to meet special requirements. Resource Domains are a QuantaStor construct so you will not find mention of them in general Ceph documentation, but they map closely to the CRUSH bucket hierarchy.
For additional information see, CRUSH MAPS