Western Digital (WD) Data24 Configuration

WD Data24 NVMe-oF Setup with QuantaStor

This guide describes how to configure a WD Data24 NVMe-oF JBOF with QuantaStor using RDMA (RoCE) transport in a highly-available scale-up configuration.

Prerequisites

• Latest QuantaStor installed on two QuantaStor servers for HA (WD Data24 42xx), one QuantaStor server for non-HA (WD Data24 41xx)
• WD Data24 41xx or 42xx installed and configured with all devices visible from all ports
• Dedicated 100GbE links between each QuantaStor node and the Data24 (see diagram)
• Lossless Ethernet configuration supported by the network adapters and switches

Cabling

The cabling for the Western Digital Data24 is a little different from how we cable SAS JBOFs. Where with SAS JBOFs we crossover so that each server is connected to separate IO Modules (IOMs) with the Data24 we need to keep the server connections all to the same IOM. In this diagram we show server 1 connecting to IOM-A and server 2 connecting to IOM-B. Two connections is sufficient and you can even get away with a single cable connection per server. Three or four connections could be done but they won't yield more performance due to filesystem overhead so we recommend two cables per server, again, to the same IOM.

Driver Installation

Before configuring networking or NVMe-oF connectivity, install the Mellanox OFED drivers using the QuantaStor-supplied script.

OFED Driver Installation

Be sure to login to each QuantaStor system via SSH then change to root user before running the script like so:

sudo -i
/opt/osnexus/quantastor/bin/mellanox-ofed-install.sh

Reboot

After driver installation a system reboot is required. After the reboot the NIC port names will change to the OFED naming scheme.

Network Configuration

Clearing DHCP Settings

After initial installation of the QuantaStor operating system you'll often find that many of the network ports are configured incorrectly with DHCP. It's very important to go through all these ports and use the Modify Network Interface/Port dialog to change all these unused ports to either 'disabled' or 'static' and assign and IP if you're setting it to static.

Configuring Lossless Networking

After the reboot the OFED drivers that support Lossless Ethernet networking will be active. We can now configure static IPs on the 100GbE ports directly connected to the Data24(s).

Open Modify Interface / Port in the QuantaStor GUI. Configure static IP addresses on the 100GbE ports connected to the WD Data24. In the same dialog, enable Lossless Networking on those ports.

Repeat this configuration on both QuantaStor nodes in the HA configuration.

• Note - These ports are dedicated exclusively for NVMe-oF connectivity to the WD Data24. They should not be used for client data access (NFS/SMB/iSCSI).

Reboot

After making the DHCP changes and/or enabling Lossless Ethernet it is important to reboot the systems before proceeding.

WD Data24 Configuration

Shared Mode Setup

This is most easily done through the WD Data24 web user interface. After logging in navigate to the section for the IO modules then click on the slider for 'Shared' mode access on all IO modules. This makes it so that all the NVMe devices are accessible from all ports.

Enabling Device Reservations

The following script should be written to a wd-set-devres.sh script and then run in order to enable device reservations (aka IO fencing / persistent reservations) which is needed in order for QuantaStor to use the system in a scale-up HA cluster. For scale-out configurations this feature is not needed.

#!/bin/bash

if [ "${#}" -gt 1 -a "${1}" == "-p" ]; then
    PSWD="${2}"
    shift 2
else
    read -r -s -p "Enter password: " PSWD
    echo
fi

IP=$1
VAL=${2-true}

NAME=$(curl -i -u admin:"$PSWD" -X GET http://"$IP"/Storage/Devices/ 2>/dev/null | grep -oP '(openflex-|of)data24-([0-9]{4}-)?us(alp|cos)[0-9]{5}[a-z]{2}[0-9]{4}' | head -1)
# NAME=$(curl -i -u admin:admin -X GET http://"$IP"/Storage/Devices/ 2>/dev/null | grep -oP '(openflex-|of)data24-([0-9]{4}-)?us(alp|cos)[0-9]{5}[a-z]{2}[0-9]{4}' | head -1)

# echo $NAME $VAL

for i in {1..6}
do
    echo Adapter $i Setting DeviceReservationEnabled to ${VAL}
    ETAG=$(curl -i -u admin:"$PSWD" -X GET http://"$IP"/Storage/Devices/"$NAME"/Adapters/"$i"/ 2>/dev/null | grep -oP 'Etag: \K([a-z0-9]+)' | head -1)
    curl -i -u admin:"$PSWD" -H "Content-Type: application/json" -H "If-Match: $ETAG" -X PUT --data "{ \"DeviceReservationEnabled\": ${VAL}}" http://"$IP"/Storage/Devices/"$NAME"/Adapters/"$i"/ 2>/dev/null

    # sleep 1
done

exit 0

To run this command after the script has been created:

# chmod 755 wd-set-devres.sh
# ./wd-set-devres.sh <IOM IP Address>

The default username and password are both 'admin' for the Data24.

NVMe-oF Software Adapter Creation

Navigate to Enclosures & Controllers. Create a new Software Adapter. Select:

1. Mode: NVMe-oF
2. Transport: RDMA (not TCP)

Once the adapter logs into the WD Data24, new devices should appear under Physical Disks.

If the devices do not appear:

Click Rescan Disks. Verify connectivity and retry.

If you're having issues with the NVMe login note that you can customize the NVMeoF configuration file settings located here:

/opt/osnexus/quantastor/conf/qs_nvme_initiator.conf

It contains optional settings that you can apply to the connection options. With the WD Data24 hardware the IO queue settings must be set to 8 so we have defaults to take care of that.

[global]
nvme_connect_options=--nr-io-queues=8

[openflex_data24_gen4]
nqn_match=nqn.1992-05.com.wdc.ofdata24-4
nvme_connect_options=--nr-io-queues=8

[openflex_fseries_gen1]
nqn_match=nqn.1992-05.com.wdc.openflex-f
nvme_connect_options=--nr-io-queues=7

[openflex_data24_gen1]
nqn_match=nqn.1992-05.com.wdc.openflex-data24
nvme_connect_options=--nr-io-queues=8

HA Node Configuration

Repeat all steps above on the second HA node:

1. nVidia/Mellanox OFED driver installation
2. Network configuration
3. Software Adapter creation

Once both nodes see the same NVMe devices, continue configuring the system as a standard QuantaStor scale-up HA deployment.

Port Naming and HA Considerations

With OFED drivers installed, NIC ports use long device names which may be inconvenient for HA Virtual Interface (VIF) attachment.

It is recommended to:

Rename the interfaces to shorter logical names, or

Bond the interfaces and use the bond device for HA VIF assignment

This simplifies management and avoids HA configuration limitations related to long interface names.

Important Design Notes

Ports connected to the WD Data24 are strictly for NVMe-oF JBOF access.

Separate 100GbE ports should be used for client access protocols including:

• NFS
• SMB
• iSCSI

If your Data24s are connected through switches we do not recommend mixing storage front-end traffic with NVMe-oF back-end traffic on the name NICs within the QuantaStor. Configure dedicated NIC(s) for your storage connectivity to the Data24 and front-end NICs for everything else.

Summary

This configuration enables QuantaStor to access the WD Data24 as a high-performance NVMe-oF backend using RDMA, providing a fully redundant, scale-up HA architecture suitable for high-bandwidth and low-latency workloads.