Template:SetupStep CephRequirements

From OSNEXUS Online Documentation Site
Revision as of 16:53, 2 May 2019 by Qadmin (talk | contribs)
Jump to navigation Jump to search

Requirements Before Getting Started

To achieve quorum a minimum of three systems are required. The storage provided by the system can be SAS or SATA HDD or SSDs but a minimum of 1x SSD is required for use as a journal (write log) device in each system. Systems must use a hardware RAID controller for QuantaStor boot/system devices and we recommend using a hardware RAID controller for the storage pools as well.

  • 3x QuantaStor Storage Systems minimum (up to 64x Systems)
Per System:
    • Intel Xeon or AMD Opteron CPU
    • 64 GB RAM
    • 1x 200GB or larger high write endurance SSD/NVMe/NVRAM device for use as Journal Device
4x Object Storage Daemons per Journal Device, maximum of 8x OSDs per Journal Device
    • 5x to 100x HDDs or SSD for data storage per system
    • 1x hardware RAID controller for OSDs (SAS HBA can also be used but RAID is faster)
    • NTP / Time Synchronization
    • Separate Networks for iSCSI and Ceph communication

Hardware RAID Configuration

Ceph has the ability to repair itself in the event of the loss of one or more OSDs which can be provisioned one-to-one with each Storage Pool which is one-to-one with a HDD. In QuantaStor deployments we support this style of deployment but our reference configurations always use hardware RAID to combine disks in to 5x disk RAID5 groups for several reasons:

  • Disk failures have no impact on network load since they're repaired using a hot-spare device associated with the RAID controller
  • Journals can be made fault-tolerant and easily maintained by configuring them into RAID1 or RAID5 units.
  • Multiple DC grade SSDs can be combined to make ultra high-performance and high-endurance Journal Devices and the sequential nature of the journal writes lend well to the write patterns for RAID5.
  • Disk drives are easy to replace with no knowledge of Ceph required. Simply remove the bad drive identified by the RED LED and replace it with a good drive. The RAID controller will absorb the new drive and automatically start repairing the degraded array.
  • When the storage is fault-tolerant one need only maintain two (2) copies of the data (instead of 3x) so the storage efficiency is 40% usable vs. the standard Ceph mode of operation which is only 33% usable. That's a 20% increase in usable capacity.
  • RAID controllers bring with them 1GB of NVRAM write-back cache which greatly boosts the performance of OSDs and Journal devices. (Be sure that the card has the CacheVault/MaxCache supercapacitor which is required to protect the write cache).
  • Reduces the OSD and placement group count by 5x or more which allows the cluster to scale that much bigger and with reduced complexity (a cluster with 100,000 PGs use less RAM per system and is easier for Ceph monitors to manage than 500,000 PGs).

Besides the nominal extra cost associated with a SATA/SAS RAID Controller vs a SATA/SAS HBA we see few benefits and many drawbacks to using HBAs with Ceph. Some in the Ceph community prefer to let bad OSD devices fail in-place never to be replaced as a maintenance strategy. Our preference is to replace bad HDDs to maintain 100% of the designed capacity for the life of the cluster. Hardware RAID makes that especially easy and QuantaStor has integrated management for all major RAID controller models (and HBAs) via the Web UI (and CLI/REST). QuantaStor's web UI also has an enclosure management view so that it's easy to identify which drive is bad and where it is located in the server chassis.


Front-end / Back-end Network Configuration

Networking for scale-out file and block storage deployments use a separate front-end and back-end network to separate the client communication to the front-end network ports (S3/SWIFT, iSCSI/RBD) from the inter-node Ceph communication on the back-end. This not only boosts performance, it increases the fault-tolerance, reliability and maintainability of the Ceph cluster.

All nodes should have one or more ports designated as front-end ports and assigned IP address and subnet masks specifically for client access. For all nodes one or more ports should be designated as the front-end ports and assigned appropriate IP addresses and subnets to enable client access. One can have multiple physical, virtual IPs, and VLANs used on the front-end network to enable a variety of clients to access the storage. The back-end network ports should all be physical ports but it is not required.

A basic configuration Ceph network configuration:

Retrieved from "https://wiki.osnexus.com/index.php?title=Template:SetupStep_CephRequirements&oldid=13054"