Software Encryption

QuantaStor supports both software and hardware encryption. Software encryption leverages the Linux based LUKS key management system which is hardware independent, supports a broad set of encryption algorithms, and provides flexible configuration options for key location. For hardware based encryption you must use a LSI MegaRAID or equivalent OEM version (IBM, Dell, SuperMicro, etc) hardware RAID controller with the SafeStore license key applied along with one or more enterprise SED SAS drives.

QuantaStor uses the LUKS (Linux Unified Key Setup) system for key management but also comes with tools to greatly simplify the configuration and setup of encryption. Specifically, the qs-util CLI utility comes with a series of additional commands to encrypt disk devices including cryptformat, cryptopen, cryptclose, cryptdestroy, and cryptswap. There's also a devicemap command which will scan for and display a list of devices available on the system.

  Device Encryption Commands
    qs-util cryptformat <device> [keyfile]  : Enrypts the specified device using LUKS format, generates key if needed.
    qs-util cryptopen <device> [/path/to/keyfile.key] [updatecrypttab] : Opens the specified LUKS encrypted device.
    qs-util cryptopenall             : Opens all encrypted devices using keys in /etc/cryptconf/keys
    qs-util cryptclose <device>      : Closes the specified LUKS encrypted device.
    qs-util cryptcloseall            : Closes all LUKS encrypted devices.
    qs-util cryptdestroy <device>    : Closes the LUKS device and deletes the keys and header backups.
    qs-util cryptswap <device>       : Enables swap device encryption, updates /etc/fstab and /etc/crypttab.
    qs-util crypttabrepair           : Recreates a new /etc/crypttab by trying all keys with all LUKs devices

Summary of Software Encryption Setup Procedure

1. qs-util cryptswap

2. qs-util devicemap

• Shows a list of the devices on the system

3. qs-util cryptformat <device>

• Formats the specified device with an encryption header and sets up /etc/crypttab to automatically open the device at boot time

4. Use the Scan for Disks.. command and your encrypted (dm-name-enc-) device(s) will appear in the QuantaStor web management interface.

• There is a known issue in that you have to logout out and log back in to the web interface after you run the Scan for Disks. This will cleanup the list of disks.

5. Create a new storage pool using your encrypted device(s)

Note also that if you are using SSD caching that you must also cryptformat your SSD read/write cache devices before using them with an encrypted pool. If you don't you will be creating a security hole.

Setup: Selecting Drives for Encryption

The first step in setting up software encryption is selecting the drives to be encrypted. Use the 'qs-util devicemap' command and you'll see a list of devices that will look something like this:

/dev/sdj        /dev/disk/by-id/scsi-350000393a8c96130, TOSHIBA, MK1001TRKB, Y1M0A01HFM16
/dev/sdv        /dev/disk/by-id/scsi-350000393a8c960c4, TOSHIBA, MK1001TRKB, Y1M0A011FM16
/dev/sdt        /dev/disk/by-id/scsi-350000393a8c960b4, TOSHIBA, MK1001TRKB, Y1M0A00XFM16
/dev/sdu        /dev/disk/by-id/scsi-350000393a8c960a4, TOSHIBA, MK1001TRKB, Y1M0A00TFM16
/dev/sdr        /dev/disk/by-id/scsi-350000393a8c960bc, TOSHIBA, MK1001TRKB, Y1M0A00ZFM16

Make sure that you are connected to the QuantaStor web management interface so that you can be sure that you're selecting the correct disks. NOTE: You cannot encrypt drives that already have data on them. You must setup encryption on the drives before you create the storage pool on top.

Setup: Formatting Drives with Encryption Header

Use the qs-util cryptformat <device> command to encrypt devices. WARNING: Any data on these drives will be lost as cryptformat will be imprinting the device with an encryption header. For example, to encrypt the devices noted in the previous section, one would run these commands:

qs-util cryptformat /dev/sdj
qs-util cryptformat /dev/sdv
qs-util cryptformat /dev/sdt
qs-util cryptformat /dev/sdu
qs-util cryptformat /dev/sdr

The cryptformat command does several things:

1. Generates a new key (1MB) for the device using /dev/urandom and stores it in /etc/cryptconf/keys/
2. Formats (luksFormat) the device with the new encryption header using the generated key (default is AES 256-bit encryption)
3. Makes a backup of the encryption header and stores it in /etc/cryptconf/headers/
4. Updates the /etc/crypttab configuration file so that the appliance automatically opens the device at boot time
5. Opens (luksOpen) the device using the generated key so that you can start using it

Note that even though we've specified the devices by their short names (/dev/sdj) that the utility automatically looks up the correct persistent device name with the scsi- prefix so that the encryption driver can locate the correct device even if the device lettering changes after a reboot.

Hardware Accelerated Encryption (AES-NI)

QuantaStor supports Intel's AES-NI hardware accelerated encryption by default. In testing we found AES-NI performance on a basic server to be around 1.1GB/sec when enabled (the default) and the same encrypted device with the AES-NI drivers removed dropped to 145MB/sec. So AES-NI boosts the performance of encrypted storage pools by about 7.5x.

There are no special steps you need to do activate support for AES-NI and you can run this command to ensure that the drivers are properly loaded:

lsmod | grep aesni_intel

Should should see output which includes the aesni_intel driver like this:

aesni_intel           172032  2
aes_x86_64             20480  1 aesni_intel
ablk_helper            16384  5 aesni_intel,twofish_avx_x86_64,serpent_avx2,serpent_avx_x86_64,serpent_sse2_x86_64
cryptd                 24576  4 aesni_intel,ghash_clmulni_intel,ablk_helper
lrw                    16384  6 aesni_intel,twofish_avx_x86_64,twofish_x86_64_3way,serpent_avx2,serpent_avx_x86_64,serpent_sse2_x86_64
glue_helper            16384  6 aesni_intel,twofish_avx_x86_64,twofish_x86_64_3way,serpent_avx2,serpent_avx_x86_64,serpent_sse2_x86_64

If you don't see the aesni_intel driver loaded then you may be using hardware that doesn't have AES-NI support or has it disabled in the BIOS. For more information on Intel and AMD processors that support AES-NI please see the Wikipedia article on it here.

Setup: Custom Key Generation (Optional)

You can use the lower-level cryptsetup commands like 'cryptsetup luksFormat' to prepare and open your devices using your own custom generated keys and decryption script. That is ok, but you must follow the naming convention where devices named /dev/disk/by-id/scsi- are given the same name with the enc- prefix added. For example, the encrypted target name for device /dev/disk/by-id/scsi-35001517bb282023a must be set to enc-scsi-35001517bb282023a. This is automatically setup for you in the /etc/crypttab file when you use the 'qs-util cryptformat' command. Note also that you don't need to use the /etc/crypttab file to open your devices at boot time. You can have a custom script that is run from /etc/rc.local or you can have a script that requires a passphrase in order to unlock the keys and open the devices.

Setup: Backing up your Encryption Keys

You should immediately make a backup of your encryption keys and headers to someplace secure (NOTE: You can use a tool like Filezilla to sftp into the appliance to get the keys from /etc/cryptconf). You can make an encrypted backup of your key files and headers using this command:

tar -cj /etc/cryptconf/ | openssl des3 -salt > mykeys.tar.enc

You can then decrypt it later using this command:

cat mykeys.tar.enc | openssl des3 -d -salt | tar -xvj

You'll see output that looks like this:

enter des-ede3-cbc decryption password:
etc/cryptconf/
etc/cryptconf/keys/
etc/cryptconf/keys/enc-scsi-350000393a8c960c8.key
etc/cryptconf/keys/enc-scsi-35001517bb282023a.key
etc/cryptconf/keys/enc-scsi-35001517bb2820591.key
etc/cryptconf/headers/
etc/cryptconf/headers/enc-scsi-35001517bb2820591.header
etc/cryptconf/headers/enc-scsi-350000393a8c960c8.header
etc/cryptconf/headers/enc-scsi-35001517bb282023a.header

Setup: Securing your Encryption Keys

The qs-util cryptformat command places your keys on the appliance boot/system drive in /etc/cryptconf. On most systems this is a vulnerable place to have the keys as the boot drives can be removed from most servers with the data drives. If the boot devices have physical locks on them so that the devices containing the keys cannot be stolen or if the boot devices are on the interior of the storage appliance chassis where they cannot be easily removed or accessed then that may be sufficient for your needs. That said, in most cases you'll want to copy the generated keys to someplace safer and shred the local copies. Some strategies you might use include:

• Connect a small capacity USB stick/media to the inside of the server and have the /etc/fstab setup to mount the device to /etc/cryptconf. Now when you cryptformat devices the keys are stored on removable media which cannot be taken from the server without physical access to open the chassis to take out the USB device.
• Connect a small capacity USB stick/media to the outside of the server and have the /etc/fstab setup to mount the device to /etc/cryptconf. Now when you cryptformat devices the keys are stored on removable media which you can remove from the appliance after it has been booted and the devices have been opened.
• Copy the keys to a Key Server and have the appliance reference the keys via a custom script which calls 'cryptsetup luksOpen'. The process for setting this up will depend on your key server software, etc.
• Put the keys on an secure NFS/CIFS share which is only accessible to the appliance and which mounts the share to /etc/cryptconf automatically at boot-time. In the event that the hardware is stolen the keys are not in the appliance so the data cannot be accessed/decrypted.
• After the system starts, manually run a script that decrypts your key backups mykeys.tar.enc which requires a pass-phrase and places the keys into /tmp. The script should then use cryptsetup luksOpen to open each device using the decrypted keys and then delete the decrypted keys from /tmp automatically. Then run 'qs pool-start poolname' to start the encrypted storage pool to expose access to the encrypted resources.

Whatever strategy you choose to secure the keys, you'll want to make sure you have backup copies and you'll want to be sure to adjust the /etc/crypttab file accordingly if you change the location of the keys. The /etc/crypttab is configured automatically when you use the 'qs-util cryptformat <device>' command and it'll look something like this:

# <target name> <source device>         <key file>      <options>
enc-scsi-35001517bb282023a      /dev/disk/by-id/scsi-35001517bb282023a  /etc/cryptconf/keys/enc-scsi-35001517bb282023a.key      luks
enc-scsi-35001517bb2820591      /dev/disk/by-id/scsi-35001517bb2820591  /etc/cryptconf/keys/enc-scsi-35001517bb2820591.key      luks
swap    /dev/disk/by-id/scsi-SATA_ST32000641AS_9WM288LS-part5   /dev/urandom    swap,cipher=aes-cbc-essiv:sha256,size=256

If you have copied the keys to a new location such as /mykeys you'll need to update the paths in the /etc/crypttab to replace /etc/cryptconf/keys/ with /mykeys/. Note also that you'll want to shred the old keys after you have copied them to the new location and make backups. You can do that securely with the shred command like so:

shred /etc/cryptconf/keys/enc-scsi-35001517bb282023a.key

which will ensure there are no remnants of the old key left on the boot drive.

Setup: Swap Device Encryption

QuantaStor appliances use swap devices which can temporary contain a cache of unencrypted data that was in memory. This is a security risk which is easily resolved by making your swap device encrypted like so:

qs-util cryptswap

This command updates the /etc/fstab and /etc/crypttab so that the system will have an encrypted swap device after the next reboot. Note that it generates a new key every time the system boots using a randomly generated key.

NOTE: You will see an entry in the /etc/crypttab that looks like this:

swap    /dev/disk/by-id/scsi-SATA_ST32000641AS_9WM288LS-part5   /dev/urandom    swap,cipher=aes-cbc-essiv:sha256,size=256

You will also see an updated entry in the /etc/fstab for your swap device that looks like this:

/dev/mapper/swap swap swap defaults 0 0

Setup: Scan for Devices & Create Storage Pool(s)

At this point your encrypted devices are all setup. If you have restarted the system then you'll see new devices in the Physical Disks section of the web management interface with device names prefixed with 'dm-name-enc-'. These are your encrypted devices you can now create a storage pool from. If you haven't rebooted after completing the above steps you can find your new encryption formatted devices by using the 'Scan for Disks..' command in the web management interface. You can access this by right-clicking on the Physical Disks section header.

Setup: Verifying Encrypted Storage Pool(s)

After you have created your encrypted storage pool(s) it is recommended that you reboot to make sure that the swap device is encrypted and that your storage pools have come back online automatically. You can check the swap device by running the swapon command like so:

swapon -s<pre>

The output of which should look like this with the path to the swap device under /dev/mapper/swap:

<pre>
Filename                                Type            Size    Used    Priority
/dev/mapper/swap                        partition       6281212 0       -1

Next, look at the encrypted storage pool in the web interface and you should see that all of the devices for your pool should start with the prefix of dm-name-enc-. If the pool is offline try starting the pool using 'Start Pool..'. For the pool to start you must have already opened the encrypted devices as per the naming convention noted above.