Cluster VIFs
A cluster VIF is a floating IP address owned by a site cluster rather than by one appliance, so a service address survives the loss of the appliance serving it. This page covers adding one and, in depth, the four use case types -- what each one makes the address follow, and the constraints each one carries.
For the cluster behaviour of a VIF once it exists -- how failover is triggered, how long it takes, moving one deliberately, location constraints, standby and maintenance mode -- see High-availability VIF Management. For plain virtual interfaces, bonds, VLANs and MTU at the network port level, see Network Ports, which owns the port itself.
| Section | Purpose |
|---|---|
| What a cluster VIF is | How it differs from an ordinary virtual interface on a port |
| Before you add one | Three things to have in place first |
| Adding a cluster VIF | Every field on every tab of the Add dialog |
| Choosing the use case | The four types, and why some are greyed out |
| Grid Primary | The grid management address -- and grid mastership |
| Storage Pool (Scale-up HA) | Follows a ZFS pool through its HA failover group |
| Storage Pool (Scale-out) | A Ceph file, block or object service address |
| Other | A floating address with nothing to follow |
| Converting a local virtual IP | Promoting an existing address instead of recreating it |
| Interface names and tags | What the tag means and why it can be truncated |
| Deleting a cluster VIF | The Remove dialog and its options |
| When a VIF will not come up | Create-time failures and what each one means |
What a cluster VIF is
An ordinary virtual interface is a second address on a network port of one appliance. It is configured on that appliance, it exists only there, and if the appliance goes down the address goes with it. Network Ports covers those.
A cluster VIF is an address that belongs to a site cluster. Underneath, the cluster software runs it as a resource that can be started on any member the VIF is permitted to run on, so exactly one member holds the address at a time and the cluster moves it when that member fails. The address is attached to a port by name -- on failover it attaches to the port with the same name on whichever member takes over, which is why that name has to exist everywhere the VIF is allowed to run.
Three consequences follow from the address belonging to the cluster rather than to a port:
- You cannot edit it from Network Ports. A cluster VIF appears there as a port, but the cluster software owns its configuration, so address, MTU and bond changes are refused.
- It has a use case. That is the field with no counterpart on an ordinary virtual interface, and it is the subject of most of this page.
- It has location constraints -- a per-member weight saying where it prefers to run. Those are covered on High-availability VIF Management.
Before you add one
A site cluster must exist, and the appliances must already be members of it. A cluster VIF has no meaning without one; there is no cluster software to run it. See Site Cluster Setup.
The parent port name must exist on every member the VIF may run on. The VIF remembers a port name, not a port. If two members call their interface ens192 and a third calls its equivalent something else, the VIF cannot run on the third.
The address must be genuinely free. QuantaStor checks that no port anywhere in the grid already has the address, and pings it once before claiming it, but a host that happens to be powered off passes both checks and collides later. Confirm the address is unallocated on your network before you commit it.
Adding a cluster VIF
The dialog is a three-step wizard. General holds the use case and its associated object; Virtual Interface holds the addressing and the port; Location Constraints holds the per-member weights. Use Next and Previous to step through them, or click the tabs directly.
The Virtual Interface tab
- Site -- the site cluster the VIF will belong to. Changing it re-reads the member list, so set it before anything else.
- IP Address -- the floating address. It is pre-filled with the local network prefix as a typing convenience, not a suggestion of a free address.
- Description -- optional free text, stored on the VIF.
- Subnet Mask -- pre-filled from the local network.
- FQDN -- optional. When set, the name is added to
/etc/hostson every appliance in the storage grid so that internal services, Kerberos among them, can refer to the VIF by name. It must be unique and must not be a host name already used by one of the appliances. - iSCSI Portal and NVMeoF Portal -- whether the floating address is an allowed portal for those protocols. Both are ticked by default in the web interface. They are forced off and greyed for the Grid Primary use case.
- Convert local virtual IP to cluster VIF resource -- see Converting a local virtual IP.
- Select the target port interface to attach to -- pick the Storage System, then a port from the grid below it. The grid lists only that appliance's plain physical ports: bond slaves, VLANs and existing virtual interfaces are filtered out.
The command line omits the two portal flags rather than defaulting them on. Where the dialog ticks iSCSI Portal and NVMeoF Portal for you, qs site-vif-create leaves both false unless you pass --iscsi-enable=true and --nvmeof-enable=true. Pass them explicitly if you are scripting a VIF that block initiators will connect to.
The Location Constraints tab
Automatic Location Constraints is ticked by default, and what it computes depends on the use case -- each use case section below says what you get. Clearing the checkbox lets you set the weights by hand. The weights, what they mean, and why raising one does not usually move a running VIF are covered on High-availability VIF Management.
From the command line
qs site-vif-create --site=site-cluster-1 --parent-port=ens192 \
--ip-address=10.0.8.240 --netmask=255.255.0.0 \
--vif-use-case=other --iscsi-enable=true --nvmeof-enable=true \
--description="service address for pool-1"
qs site-vif-create also takes --fqdn, --mtu, --mac-address, --gateway, --usecase-obj-id, --location-config and --convert-vif. The use case is given by name: grid-primary, ha, scaleout-file, scaleout-block, scaleout-obj or other. ha is the default -- the scale-up use case -- so a create that omits --vif-use-case asks for a scale-up VIF, and fails with Given use case 'scaleup_pool' requires associated use case object ID. unless you also pass --usecase-obj-id. Always state the use case explicitly.
Every site-vif command identifies a VIF by its ID, not its name. Take the ID from qs site-vif-list; the displayed name is of the form 10.0.8.240 (svb19c00) and is not accepted as an identifier, and neither the bare address nor the tag resolves.
qs site-vif-list qs site-vif-get --vif-resource=7fbcb839-96c3-7f79-9031-0c8b467862b2
Choosing the use case
The use case tells the cluster software which resource the address has to follow. It is the one field with no equivalent on an ordinary virtual interface, and it decides three things: what object the VIF is tied to, which members it is eligible to run on by default, and what has to be true on a member before the address is allowed to start there.
The web interface offers four:
| Use case | Follows | Associated object | Selectable when | CLI value |
|---|---|---|---|---|
| Grid Primary | The grid master role | none | always | grid-primary
|
| Storage Pool (Scale-up HA) | A ZFS pool, through its HA failover group | a storage pool HA failover group | an HA failover group exists | ha
|
| Storage Pool (Scale-out) | A Ceph service -- object, file or block | a Ceph cluster, plus a config type | Ceph pools exist | scaleout-obj, scaleout-file, scaleout-block
|
| Other | nothing | none | always | other
|
A greyed-out option is a statement about your grid, not a fault. The two Storage Pool options are gated on what the grid actually contains, because each needs an object to attach to. Create the HA failover group or the Ceph pool first and the option becomes selectable. If neither is available, QuantaStor preselects Other; if an HA failover group exists it preselects Storage Pool (Scale-up HA).
The use case is set once, at creation. Neither the Modify dialog nor qs site-vif-modify can change it -- both offer only the description and the location constraints. To change a VIF's use case, delete it and create it again.
QuantaStor records the chosen use case in a small file on every member of the site cluster, at /var/opt/osnexus/quantastor/clustervif_<parent-port>_<tag>.uses. Reading it is the quickest way to confirm what a VIF was created as:
# cat /var/opt/osnexus/quantastor/clustervif_ens192_gm.uses use_gridprimary=true # cat /var/opt/osnexus/quantastor/clustervif_ens192_svb19c00.uses use_other=true
The other keys are use_scaleup_hapool, use_scaleout_filepool, use_scaleout_blockpool and use_scaleout_objpool. Where a use case has an associated object, the file also carries a use_case_obj_id line.
Grid Primary
Grid Primary is the floating address the storage grid is administered through. Point browsers and automation at it rather than at one appliance's management address, and management access survives the loss of that appliance.
Where the Grid Primary VIF runs is where the grid master runs. This is the most surprising property of the use case and the reason to think before moving one. Moving the VIF moves grid mastership with it. Verified on a three-appliance grid: with the VIF on qs-node-110, qs grid-get reported that appliance as Master Node; moving the VIF to qs-node-111 changed the reported master to qs-node-111 within seconds, the tree relabelled it, and moving the VIF back returned mastership to qs-node-110.
The mechanism is a cluster check that runs about once a minute on every member: an appliance that finds the grid management VIF running locally, and that is not already the master, elects itself. So the VIF is the token that carries the role, and moving it is the supported way to hand the role over.
Grid membership is not affected. Only the master role moves; the grid keeps every member it had. Nothing about moving a Grid Primary VIF adds or removes an appliance from the grid.
Three more rules apply to this use case, and all three are enforced:
- One per site cluster. A second attempt fails with
Grid virtual interface already exists for Site Cluster '<name>'. - iSCSI and NVMe-oF are permanently off. Selecting the use case clears and disables both portal checkboxes, and the service forces them false on the stored object regardless of what was requested. A grid management address is not a storage portal.
- The tag is always
gm, not a generated one, so the interface is<port>:gmon whichever appliance is hosting it. That is what makes it recognisable inip addroutput and in the cluster resource list.
With automatic location constraints, every member of the site cluster is made eligible with an equal weight of 100.
qs site-vif-create --site=site-cluster-1 --parent-port=ens192 \
--ip-address=10.0.8.241 --netmask=255.255.0.0 --vif-use-case=grid-primary
Storage Pool (Scale-up HA)
A scale-up VIF is the address clients use to reach a ZFS pool, and it follows that pool as the pool fails over. The associated object is a storage pool HA failover group -- the object that ties a pool to the appliances allowed to import it -- chosen from the HA Failover Group list in the Scale-up Use Case fieldset on the General tab. Because the address and the pool move together, a client that reconnects after a failover finds its data on the appliance the address landed on.
This is the use case to choose for NFS and SMB service addresses on an HA pool, and for iSCSI and NVMe-oF portals on one.
Three constraints are specific to it:
- An HA failover group must exist, or the radio is greyed out. Setting up the pool and its group comes first: see HA Cluster Setup (JBODs) and HA Cluster Setup (external SAN).
- The group must belong to a site cluster. A clusterless HA group cannot carry a VIF, and the dialog says so:
The selected HA Group has no associated Site Clusters. Clusterless HA Groups cannot have HA VIFs. - The parent port name is checked across the whole site cluster before the VIF is created, rather than only on the appliance you selected. This is stricter than the other use cases, and it is the right check -- a pool VIF that cannot follow its pool to the secondary appliance defeats the purpose.
With automatic location constraints, the HA group's primary, secondary and tertiary appliances are made eligible with a weight of 100 each, and every other member of the site cluster is set to 0 so the VIF is never placed where the pool cannot be imported.
A scale-up VIF is created as a storage pool HA failover interface rather than as a plain site VIF, and carries an :ha tag instead of an :sv one. It can also be created from the HA failover group itself rather than from this dialog; see Storage Pool HA Failover Interface Create.
qs site-vif-create --site=site-cluster-1 --parent-port=ens192 \
--ip-address=10.0.8.245 --netmask=255.255.0.0 \
--vif-use-case=ha --usecase-obj-id=ha-group-1 \
--iscsi-enable=true --nvmeof-enable=true
Storage Pool (Scale-out)
A scale-out VIF is a service endpoint for a Ceph cluster. Unlike the scale-up case there is no data to move with it: Ceph has already distributed the data, so the VIF is purely an address that stays reachable, and moving it moves nothing but the address.
Two fields in the Scale-out Use Case fieldset define it:
- Ceph Cluster -- the associated object.
- Config Type -- Object, File or Block. This is not cosmetic. It selects which Ceph service the address is for, and each type carries its own precondition, checked at create time and again every time the address is started on an appliance.
| Config type | For | Selectable when | Will not start on an appliance unless |
|---|---|---|---|
| Object | S3 and Swift access through the RADOS gateways | a RADOS gateway and an object pool group exist | the RADOS gateway is running and configured there |
| File | CephFS access | a CephFS filesystem exists | the CephFS pool and its export are mounted there |
| Block | RBD access, including Ceph iSCSI | Ceph pools exist | -- |
That last column is the substance of the type. A scale-out VIF is not just an address that floats -- it is an address that refuses to land where the service behind it is not actually running, so a client that reaches it always reaches a working endpoint. It is also why a scale-out VIF takes slightly longer to fail over than a scale-up one: the resource checks the storage before claiming the address.
If the Ceph cluster has none of the resources a type needs, the create is refused rather than silently producing a VIF that can never start -- with a message naming the missing piece, such as no CephFS pool for the File type or no RADOS gateway for the Object type.
With automatic location constraints, every Ceph cluster member that is also a site cluster member is made eligible with weight 100, and other site cluster members are set to 0. For this to describe the cluster properly, all Ceph members need to be in the site cluster -- if some are not, set the weights by hand instead.
Block: a cluster VIF is the preferred way to present Ceph iSCSI. Routing a target's traffic through one floating address keeps all of it going through a single target instance, which is what SCSI reservations need.
Setting the Ceph cluster up comes first: see Scale-out Object Setup (ceph), Scale-out File Setup (ceph) and Scale-out Block Setup (ceph).
qs site-vif-create --site=site-cluster-1 --parent-port=ens192 \
--ip-address=10.0.8.246 --netmask=255.255.0.0 \
--vif-use-case=scaleout-file --usecase-obj-id=ceph-cluster-1
--usecase-obj-id may be omitted for the three scale-out types if the appliance belongs to exactly one Ceph cluster, in which case that cluster is used.
Other
Other is a floating address with no resource to follow. It has no associated object, no start-time precondition beyond the cluster software running, and with automatic location constraints every member of the site cluster is eligible at an equal weight of 100. It is treated exactly like Grid Primary for placement purposes, and differs in that it carries no special role and no restriction on the portal flags.
Choose it when an address needs to stay reachable across an appliance failure but is not tied to a storage pool or a Ceph service -- a stable endpoint for monitoring, for an administrative service, or for a client configuration you do not want to have to change. It is also the honest choice for an address whose purpose is not one of the other three: nothing is lost by using it, because the other use cases add constraints rather than capability.
Other is what QuantaStor preselects when neither Storage Pool option is available, so on a grid with no HA failover group and no Ceph it is the only type you can create besides Grid Primary.
Converting a local virtual IP
An address that already exists as an ordinary virtual interface on one appliance can be promoted to a cluster VIF in place, rather than being deleted and recreated. That matters when the address is in use: clients keep the address they have, and it becomes highly available.
Ticking Convert local virtual IP to cluster VIF resource on the Virtual Interface tab clears and disables IP Address and Subnet Mask, because both come from the interface being converted rather than from you.
From the command line, pass the existing virtual interface as the parent port and add --convert-vif=true:
qs network-port-list
qs site-vif-create --site=site-cluster-1 --parent-port=ens192:1 \
--ip-address=10.0.8.242 --netmask=255.255.0.0 \
--vif-use-case=other --convert-vif=true
Two things about the result are worth knowing:
- The converted VIF's parent is the virtual interface, not the physical port. Its parent interface name stays
ens192:1, and the address keeps that label on whichever appliance is hosting it rather than taking an:svlabel. Failover works normally; the address simply appears asens192:1on the new appliance. - The tag is shortened to fit. A longer parent port name leaves less room within the kernel's interface name limit, so a VIF converted from
ens192:1gets a shorter tag than the same VIF created onens192would.
Interface names and tags
Every cluster VIF gets a tag, which is both the name of its cluster resource and the label the address carries on the host:
| Use case | Tag | Interface on the host |
|---|---|---|
| Grid Primary | always gm |
ens192:gm
|
| Scale-up HA | begins ha |
ens192:ha<...>
|
| Scale-out and Other | sv plus a fragment of the site cluster ID and a sequence number |
ens192:svb19c00
|
The tag is generated, not chosen, and it is unique within the site cluster. It is worth recognising because it appears in three places a reader will meet: the VIF's displayed name, <ip> (<tag>); the interface label in ip addr output; and the resource name in the cluster resource list.
The Linux interface name limit is 15 characters, and the tag is truncated to fit. The name on the host is the parent port name, a colon, and the tag, so a long parent port name leaves fewer characters for the tag. ens192 leaves eight, giving ens192:svb19c00; ens192:1 leaves six, giving a six-character tag. If nothing can be made to fit, the create fails rather than producing a truncated interface name.
Deleting a cluster VIF
Removing a cluster VIF deletes its cluster resource and takes the address off the network. The dialog warns that active connections on the interface will be dropped -- there is no drain and no graceful handover, so treat it the way you would treat a brief service outage.
Choosing a VIF from Cluster Virtual Interface fills in a read-only Cluster VIF Information panel showing its site, address, subnet mask, MTU, parent interface name, use case and use case object. Check the use case there before you remove anything; it is the only place in the interface that shows a VIF's full identity next to the delete button.
The dialog also offers Force, which proceeds when the cluster resource cannot be removed cleanly, and Convert cluster VIF resource to local virtual IP, intended to keep the address in service as an ordinary virtual interface instead of removing it. Do not rely on the conversion option: on 6.9.0 it takes the address off the network instead of re-creating it locally. Remove the VIF normally and add the local address back with qs virtual-interface-create if you need it.
qs site-vif-delete --vif-resource=7fbcb839-96c3-7f79-9031-0c8b467862b2
Remove a site cluster's VIFs before deleting the site cluster itself. qs site-cluster-delete refuses while VIFs still reference it, precisely so that dismantling a cluster cannot take a service address off the network unannounced.
When a VIF will not come up
These are the failures that happen at create time. For a VIF that exists but will not start or move, see High-availability VIF Management.
IP '<ip>' is pingable and may already be in use by another system. The create pings the address first and refuses if anything answers. Either the address is genuinely taken, or a previous attempt left it configured somewhere -- check for it on the site cluster members before assuming the former.
Interface '<name>' already exists with IP address '<ip>'. A port object in the grid already holds the address. This is the message you get after a failed create that left a port record behind. List the ports with qs network-port-list, and remove a leftover virtual interface with qs virtual-interface-delete --port=<id>.
Failed to create Site Cluster resource for virtual interface ... The cluster software could not be asked to create the resource. The usual cause is that the site cluster is not settled -- corosync and pacemaker restarting, a member rejoining, or the cluster being reconfigured. Confirm the site cluster reports a quorate partition with all members online, then try again.
The create failed but the address is up anyway. A create that fails after the cluster resource was made can leave the resource in place with no VIF object in QuantaStor. qs site-cluster-rescan rebuilds QuantaStor's view from the live cluster configuration and recovers the VIF as an object. Note that a VIF recovered this way comes back with no use case, because the use case is stored alongside the resource and is removed when the failed create rolls back. Delete it and create it again rather than leaving it in that state.
The use case you want is greyed out. That is the gating described in Choosing the use case -- the grid does not yet contain the object the use case needs.
Related pages
- High-availability VIF Management -- failover behaviour, moving a VIF, location constraints, standby and maintenance mode
- Site Cluster Setup -- creating the site cluster a VIF belongs to
- Network Ports -- ordinary virtual interfaces, bonds, VLANs, MTU and static routes
- Grid Configuration -- the storage grid and its master node
- HA Cluster Setup (JBODs) and HA Cluster Setup (external SAN) -- scale-up HA pool setup
- Storage Pool HA Failover Interface Create -- creating a scale-up VIF from the HA failover group
- Scale-out Object Setup (ceph), Scale-out File Setup (ceph), Scale-out Block Setup (ceph) -- scale-out configurations
- Configure Member Standby -- evacuating one appliance before maintenance
Verified against QuantaStor 6.9.0.