Custom Object Gateway SSl/TLS Certificate

When configuring QuantaStor scale-out clusters with an object storage zone it is important to setup a SSL/TLS certificate to associate with the S3 gateway (CephRGW) so that clients connecting are able to validate the certificate against either an internal or public certificate authority (CA). This article goes into the procedure for making a SSL certificate using OpenSSL. In the configuration of the openssl setting one will set a series of 'Alt Names' or 'Subject Alt Names' which identifies all the FQDNs for all the ports on all the QuantaStor systems that make up the cluster and any floating cluster wide FQDNs that my be used to access the S3 gateways as well as any IP addresses used to access the S3 gateways. If you are pre-planning a deployment we highly recommend gather this information and generating the certificates ahead of time.

S3 Certificate CSR generation and signing using Windows AD/Certificate Authority

Prepare Certs Directory

Start by creating a empty directory to store the certificates to be generated.

mkdir s3_adtest_osnexus_net_crt
cd s3_adtest_osnexus_net_crt

Create OpenSSL Configuration File

Next create a file named openssl.cnf that contains the proper values for your site, and the Subject Alt Names you wish to have match FQDN to IP mappings in your DNS. Here's an example openssl configuration file (save as openssl.cnf), you can start with this one and edit the values to match your organization. Be sure to change all the values under [ alt_names ] and under [ req_distinguished_name ] to match the settings for your organization :

[ req ]
default_bits       = 2048
prompt             = no
default_md         = sha256
distinguished_name = req_distinguished_name
req_extensions     = req_ext
x509_extensions    = v3_req

[ req_distinguished_name ]
C  = US
ST = Washington
L  = Bellevue
O  = OSNEXUS Development
OU = s3 services

[ req_ext ]
subjectAltName = @alt_names

[ v3_req ]
subjectAltName = @alt_names

[ alt_names ]
DNS.1   = s3.adtest.osnexus.net
DNS.2   = cg-631f-201.adtest.osnexus.net
DNS.3   = cg-631f-202.adtest.osnexus.net
DNS.4   = cg-631f-203.adtest.osnexus.net
DNS.5   = vif205.adtest.osnexus.net
DNS.6   = vif206.adtest.osnexus.net
DNS.7   = vif207.adtest.osnexus.net
IP.1    = 10.0.12.201
IP.2    = 10.0.12.202
IP.3    = 10.0.12.203
IP.4    = 10.0.12.205
IP.5    = 10.0.12.206
IP.6    = 10.0.12.207

Generate Server Key

This will generate a new server key we'll use to make a CSR from in the next step.

openssl genrsa -out s3_adtest_osnexus_net.key 2048

Generate Certificate Signing Request (CSR)

Here we use the server key with the openssl.cnf configuration file to generate a CSR:

openssl req -new -out s3_adtest_osnexus_net.csr -key s3_adtest_osnexus_net.key -config openssl.cnf

Verify / Validate CSR

Next review the generated CSR to ensure it accurately contains the correct settings:

openssl req -in s3_adtest_osnexus_net.csr -noout -text

Submit CSR to Certificate Authority

Next take the CSR and submit it to your Certificate Authority. Refer to the Certificate Authorities documentation for signing the CSR and generating a server cert and CA chain. For Windows AD/Certificate Authority, install and deploy the Windows Certificate Authority role with the Certificate Authority web portal options. This should give you a new path to submit CSRs to under http://<AD Server IP>/certsrv

Follow the onscreen prompts on the Windows Active Directory (AD) Certificate Authority page to submit the CSR and request a signed certificate. When the certificate is available for download you can download it in DER format (X. 509 digital certificate encoded in binary) or in Base64 format. If you download it in DER format we can convert it after the fact.

If the .cer file is already in base64 encoded PEM format, you can simply rename it to .pem or proceed to the next step.

Converting DER certificates to Base64 pem Format

This shows how to take a DER formatted files (certnew.p7b & certnew.cer) from your Certificate Authority to produce a certificate chain (cert_chain.pem) and certificate files in PEM format for use in the S3 Gateway:

openssl pkcs7 -print_certs -inform der -in certnew.p7b -out cert_chain.pem

Next we'll .cer file from the produced certificate (certnew.cer) to convert it to PEM format:

openssl x509 -inform der -in certnew.cer -out certnew.pem

Next we'll rename the certificate authority chain file to a clearer name:

cp cert_chain.pem adtest_osnexus_net.pem

Next ensure your private key is now in PEM format, it should look like this in a text editor with BEGIN and END blocks showing the key in Base64 format:

-----BEGIN PRIVATE KEY-----
(Your private key here)
-----END PRIVATE KEY-----

DER to PEM Format Conversion (optional)

If your private key is in DER format, convert it to PEM using the following command:

openssl rsa -inform DER -outform PEM -in derformat.key -out s3_adtest_osnexus_net.key

Replace `derformat.key` with the path to your DER-formatted private key and `s3_adtest_osnexus_net.key` with the desired output filename for the PEM-formatted private key.

Concatenate the Private Key, Certificate, and Chain

You want to have the private key at the beginning of your PEM file followed by your certificate and then the certificate chain. Use the `cat` command to concatenate them in this order:

cat s3_adtest_osnexus_net.key certnew.pem cert_chain.pem > s3_adtest_osnexus_net.pem

Replace `myprivatekey.pem` with your private key file, `certnew.pem` with your certificate file, and `cert_chain.pem` with your certificate chain file. The combined file `combined_with_key.pem` will contain all three components.

Verify the Contents

You can check the contents of the final PEM file to ensure all parts are included correctly:

openssl x509 -in s3_adtest_osnexus_net.pem -text -noout

This command will display the certificate details. You won't be able to view the private key this way (and you shouldn't be able to), but you can open the PEM file in a text editor to ensure the private key and certificates are present.

Applying Cert at Creation of S3 Gateways

When creating new S3 Gateways in QuantaStor there's an option to provide a PEM file at that time and there you'll want to specify the 's3_adtest_osnexus_net.pem' file you've generated.

Applying Cert to Existing S3 Gateways

Copy your certificate to /etc/ceph/ceph_<HOSTNAME>_radosgw.pem where <HOSTNAME> needs to be replace with the local host hostname which you can also get by running 'hostname' then restart the S3 gateway (radosgw) services with this command:

systemctl restart ceph-radosgw@*.service

Updating Clients

On your S3 client systems/hosts insert the CA's cert chain file if not already present. For example, on a Ubuntu 20.04 based system you can use the following commands to add the cert chain:

cp adtest_osnexus_net.pem /usr/local/share/ca-certificates/
update-ca-certificates

Note, you may need to break the cert file up into separate files depending on the length of the cert chain.

Test the Certificate

Finally, it is important to test the certificate from a client to verify it via openssl to ensure there are no configuration errors or other issues. The curl and openssl utilities for both good for this:

curl https://<FQDN of server>:<port>

Additionally if your client has openssl installed you can use it to print out the certificate from the S3 gateway, be sure to replace <FQDN of server> with your fqdn like s3.example.com and the <port> is typically 8580 which is the default load balancer port number on QuantaStor systems:

echo -n | openssl s_client -showcerts -connect <FQDN of server>:<port>