High-availability VIF Management
Cluster virtual interfaces (VIFs) are floating IP addresses that move between the appliances of a site cluster so that a service address stays reachable when the appliance hosting it fails. This page covers the cluster-level behaviour of those addresses -- what makes one fail over, how to move one deliberately, and how to control which appliances it is allowed to run on. Both scale-up (ZFS HA pool) and scale-out (Ceph) configurations use them, and no special hardware is required.
For virtual interfaces at the network port level -- creating a plain second address on a port, bonds, VLANs, MTU and static routes -- see Network Ports. That page owns the port; this page owns the cluster behaviour layered on top of it. For adding a cluster VIF and for the four use case types, see Cluster VIFs.
| Section | Purpose |
|---|---|
| Grid, site cluster, heartbeat ring, HA group and VIF | How the objects nest, and which one a VIF actually belongs to |
| Where cluster VIFs are managed | The High-availability VIF Management tab |
| Use cases and creating a VIF | Pointer to Cluster VIFs, which owns adding a VIF and the four use case types |
| How failover works | What triggers a move, and how long it takes |
| What blocks a failover | Three checks that will refuse to start a VIF, and what they mean |
| Moving a VIF deliberately | Planned relocation, and why it is not the same as a failover |
| Location constraints | Pinning a VIF to, or away from, particular appliances |
| Standby mode and maintenance mode | Evacuating one node versus freezing the whole cluster |
| What happens to client connections | What a client sees during a move |
| Scale-up and scale-out differences | Where the two configurations diverge |
| Removing a cluster VIF | Deletion, and converting back to a local address |
Grid, site cluster, heartbeat ring, HA group and VIF
This hierarchy is the thing readers most often get wrong, so it is worth stating plainly. Each layer sits on the one before it, and a VIF is at the top.
A storage grid is the management layer. It is the set of appliances you administer together through one interface, and it says nothing about high availability -- see Grid Configuration. Grid membership is a prerequisite: a site cluster can only be built from appliances that are already grid members.
A site cluster is the high-availability layer. It is a group of appliances at one location -- the location can span buildings, but the members are normally in close proximity, because they have to share a network. A site cluster is what runs the cluster software (corosync and pacemaker) that decides where a floating resource lives. A grid can contain several site clusters, and an appliance can belong to at most one. Site clusters are covered on Site Cluster Setup.
A cluster heartbeat ring is how the members of a site cluster tell whether each other are alive. It is a network path, not an address that serves clients: each member contributes one port, and every port in a ring must be on the same subnet. A site cluster has at least one ring and at most two, and two rings on separate networks is the recommended configuration, so that losing one network does not look like losing a node. The first ring is created for you when you create the site cluster.
A storage pool HA failover group is the scale-up layer. It ties a ZFS pool to the appliances that are allowed to import it, so the pool itself can fail over. It exists only in scale-up configurations; a Ceph cluster provides its own redundancy and has no HA group. See HA Cluster Setup (JBODs) and HA Cluster Setup (external SAN).
A cluster virtual interface is the address clients talk to. It belongs to a site cluster, is attached to a named port on one member at a time, and is the layer that makes a service address survive the loss of an appliance. Its use case is what ties it to the resource it has to follow -- an HA failover group for a ZFS pool, a Ceph cluster for scale-out storage, or nothing at all.
The two things worth taking away: a heartbeat ring carries cluster gossip and a VIF carries client traffic, and they are separate; and a VIF's parent is the site cluster, not the pool, even when its whole purpose is to follow a pool.
Where cluster VIFs are managed
Cluster VIFs and the site clusters they belong to have their own top-level tab. The tree on the left has two sections:
- Site Clusters -- the site clusters in the grid, each expanding to show its heartbeat rings. Selecting a site cluster shows its members and its ring ports, and the toolbar switches to the site cluster operations.
- Site Cluster Virtual Interfaces -- the cluster VIFs, grouped by site cluster. Selecting this section switches the toolbar to Virtual Interface Management with Add Cluster VIF, Modify Cluster VIF and Remove Cluster VIF.
Two columns in the Site Cluster Virtual Interfaces grid are easy to confuse. Managed By is the appliance that owns the VIF configuration object; Started On is the appliance where the address is actually up right now. They differ whenever the VIF has moved, and it is Started On that tells you where client traffic is going.
Move Site Cluster VIF is on the right-click menu only -- there is no toolbar button for it. Right-clicking a VIF gives Move Site Cluster VIF..., Remove Site Cluster VIF..., Modify Site Cluster VIF... and Properties....
The grid below lists the Site Cluster Member Location Constraints for the selected VIF -- one row per member, with the weight that decides how strongly the VIF prefers that member.
Use cases and creating a VIF
A cluster VIF's use case is what ties it to the resource its address has to follow, and it is chosen once, when the VIF is created. Cluster VIFs owns both topics: it covers adding a VIF field by field, in the dialog and from the command line, and each of the four use case types in depth -- what each one follows, what gates it, and what constraints it carries.
How failover works
Underneath, each cluster VIF is a pacemaker resource. QuantaStor creates it as an IPaddr2 resource named after the VIF's tag, with the address, the netmask as a CIDR prefix, the parent port as the NIC, and a monitor that runs every 10 seconds. You can see it on any member with crm_mon -1:
Active Resources: * gm (ocf:heartbeat:IPaddr2): Started qs-node-111
Three things move a VIF automatically:
- The monitor fails. The 10-second monitor finds the address is no longer correctly configured on the host, and pacemaker restarts the resource -- on another member if it cannot start locally.
- The appliance leaves the cluster. The remaining members stop hearing it on the heartbeat rings and pacemaker re-places its resources. How fast that is detected is governed by the corosync token settings; the shipped configuration uses a 3000 ms token with 10 retransmits before loss, so a lost member is normally declared within a few seconds.
/etc/corosync/corosync.confon the appliance is authoritative -- it is managed by QuantaStor and carries a warning header saying so. - A member is put into standby. See Standby mode and maintenance mode.
Where it goes is decided by the location constraints, which give each member a score, subject to the checks in the next section.
Two cluster-wide settings are worth knowing because they explain behaviour that otherwise looks wrong. QuantaStor sets stonith-enabled to false, so pacemaker does not fence a node it has lost contact with -- the duplicate-address check described below is what protects against two appliances answering for one address instead. And it sets no-quorum-policy to ignore, so a member that finds itself in a minority partition does not stop its resources on its own.
How long a failover takes
Measured on a lab site cluster by pinging the VIF at 200 ms intervals from a third appliance and counting the gap. These are observations on idle test systems, not a specification:
| What was done | Address unreachable for |
|---|---|
| Deliberate move, 3-node scale-up cluster | 2.50 s |
| Member put into standby while hosting the VIF | 2.49 s |
| Cluster services restarted on the hosting member | 2.68 s |
| Location weight raised above the stickiness threshold | 3.60 s |
| Deliberate move, 4-node scale-out cluster | 3.11 s |
| Cluster services restarted on the hosting member, scale-out | 3.12 s |
So a planned relocation and an unplanned one cost about the same, in the region of two to three seconds, and a scale-out VIF is slightly slower because the resource agent checks that the Ceph filesystem is actually mounted before it will start the address. What is not represented here is the abrupt loss of a whole appliance -- losing power, for instance. That adds the time corosync needs to declare the member gone, on top of the figures above.
What blocks a failover
QuantaStor ships its own version of the IPaddr2 resource agent, installed over the distribution's copy at /usr/lib/ocf/resource.d/heartbeat/IPaddr2 (the original is kept alongside it as IPaddr2.backup). It adds preflight checks that refuse to bring a floating address up in situations where doing so would make things worse. When one of them refuses, the reason appears in the failed actions list in crm_mon -1 and in the task's error detail, so it is worth recognising them.
The address is still answering somewhere else. Before claiming the address, the agent flushes the ARP cache and pings it. If anything replies, it waits and tries again, and if it still replies it refuses:
Failed Resource Actions:
* gm start on qs-node-110 returned 'error' because 'IP address is still in use
on other node, failover blocked.' at Thu Sep 3 10:45:41 2026 after 7.607s
This is the protection that stands in for fencing, and it is easy to trigger accidentally. If an appliance loses cluster communication but keeps running -- a heartbeat network problem rather than a crash -- it goes on holding its floating addresses while the surviving members conclude it has gone. The survivors then try to take the addresses over, find them still answering, and correctly decline. The VIF stops being managed anywhere until the stranded appliance rejoins or is shut down, but it does not go offline: it is still up on the appliance that never lost it. In a lab reproduction of exactly this -- cluster communication killed on the hosting member while the appliance itself stayed up -- the VIF was pinged continuously throughout and lost no packets at all, while the other two members logged the message above.
The remedy is to restore cluster communication on the stranded member. qs site-cluster-restart-services --storage-system=<system> restarts corosync and pacemaker on one appliance, which brings it back into the cluster; the address is released and placed properly, and the failed actions clear on their own. Note that this restart will relocate any VIF the appliance is currently hosting.
The QuantaStor service is not running. The agent refuses to start a VIF on an appliance whose core service is stopped, with QuantaStor service is stopped, failover blocked. An address that works but has no service behind it is worse than an address that has moved elsewhere.
The appliance is in manual standby. The agent checks for the manual-standby marker file independently of pacemaker's own standby flag, and refuses with a message telling you to take the node out of standby. The duplication is deliberate: pacemaker's flag can be cleared underneath QuantaStor when the cluster stack restarts, and without the second check a node you had deliberately parked could quietly start hosting resources again.
For scale-out VIFs, the storage has to be present. A file VIF will not start on an appliance unless the CephFS pool is mounted and its export is mounted there; an object VIF will not start unless the RADOS gateway is running and configured. This is what keeps a scale-out service address from landing somewhere that cannot serve it. The requirement is recorded per VIF in /var/opt/osnexus/quantastor/clustervif_<port>_<tag>.uses, which is written to every member of the site cluster:
cephfs_id="7f69711e-de3e-fdd9-4ecd-5262201d1a56" cephfs_name="cephfs-pool-1" use_case_obj_id=c06b4f30-8c65-6dac-7421-239daedbe3bb use_scaleout_filepool=true
Moving a VIF deliberately
A deliberate move relocates a VIF to a member you choose -- before taking an appliance down for maintenance, or to rebalance which appliance is serving which address. The dialog asks for the Site Cluster, the Cluster Virtual Interface, and the Move to System; Current System is shown for reference and cannot be edited. There is no toolbar button, only the right-click menu.
From the command line:
qs site-vif-move --vif-resource=<vif-id> --move-to-system=qs-node-111
A move is not a failover, and the difference matters:
- A move does not leave a preference behind. Pacemaker's own move mechanism works by pinning the resource with a temporary constraint; QuantaStor clears that constraint once the move completes. Checking the location constraints before and after a move shows them unchanged, so the VIF remains free to fail over normally afterwards. This is why you should move a VIF rather than reach for pacemaker directly -- doing it by hand leaves a pin that quietly prevents failover.
- A move will not override a pin. If the destination's location weight is None, the move is rejected before anything happens, with a message naming the fix:
Cannot move Site Cluster virtual network interface '<vif>' to system '<system>': that system's location constraint for this interface is set to 'None', which pins the interface to never run there. Set the location constraint for '<system>' to Low, Medium, or High before moving the interface. - A move will not work while the cluster is frozen. In maintenance mode the attempt fails, because pacemaker is not managing the resource at all.
The destination must be a member of the same site cluster. Moving a VIF to an appliance that is merely in the same grid is not possible, and is not a meaningful request -- the cluster software has no presence there.
Location constraints
Location constraints are how you say which appliances a VIF prefers, and which it must never run on. Each member of the site cluster gets a weight, which becomes the pacemaker score for that member -- the higher the score, the stronger the preference.
The Web UI offers four weights:
| Weight | Score | Effect |
|---|---|---|
| None | −INFINITY | The VIF will never run on this member. Automatic failover will not place it here, and a deliberate move to it is rejected. |
| Low | 100 | Eligible, least preferred |
| Medium | 200 | Eligible |
| High | 300 | Eligible, most preferred |
Automatic Location Constraints is ticked by default and is the right choice unless you have a specific reason to pin. What it computes depends on the use case: for Grid Primary and Other it gives every member of the site cluster an equal weight of 100; for scale-up it weights the HA failover group's primary, secondary and tertiary appliances at 100; for scale-out it weights every Ceph cluster member at 100. Any remaining site cluster member is set to None. Note that automatic mode is flat -- it does not rank the members it selects, so it expresses "these are eligible" rather than "prefer this one".
Clearing the checkbox enables the grid. Two things are needed per member, not one: set the weight and tick the row. The dialog reads the ticked rows, so a weight set on an unticked row is not submitted, and clearing Automatic without ticking anything fails validation with a message telling you to select and configure the node weights.
At least one member must have a weight above None. The appliance whose port you chose on the Virtual Interface tab must be one of them -- a VIF cannot be created pinned away from the port it is being attached to.
From the command line, weights are given as <system>:<weight> pairs, and any non-negative number is accepted rather than only the four the Web UI offers:
qs site-vif-modify --vif-resource=<vif-id> \
--location-config=qs-node-110:300,qs-node-111:200,qs-node-112:0
Read them back with qs site-vif-location-constraint-list, or for one member with qs site-vif-location-constraint-get --vif-resource=<vif> --storage-system=<system>. A weight given for one member leaves the others as they are, so you can adjust a single appliance without restating the whole set.
Raising a weight does not usually move a running VIF
This is the least obvious thing on this page, and it looks like a bug when you meet it. QuantaStor sets pacemaker's resource stickiness to 1000 on every site cluster, which is a preference for leaving a running resource where it is. All four Web UI weights are well below that, so changing weights through the Web UI will not relocate a VIF that is already running. It changes where the VIF will land the next time it has to be placed.
Verified on a running VIF: with the hosting member at Medium (200) and another member raised to High (300), the VIF stayed put. Only when a weight was set above 1000 from the command line did it relocate. If you want a running VIF on a particular appliance now, move it -- do not raise its weight and wait.
The corresponding useful case is None, which does take effect immediately, because −INFINITY beats stickiness. Setting a member to None will push a VIF off it.
Standby mode and maintenance mode
Both take a site cluster out of normal operation, and they do close to opposite things to VIFs. Choosing the wrong one is the most consequential mistake in this area.
Standby mode is per appliance, and it evacuates. Putting a member into standby moves its resources to a healthy member and stops it receiving any more, while the rest of the cluster keeps protecting itself normally. This is what you want before working on one appliance. In a lab measurement, a member hosting a VIF was put into standby and the address moved to another member with 2.49 s of unreachability. Standby has three states -- Active, Standby Auto Activation and Standby Manual Activation -- and Configure Member Standby covers the dialog and the difference between them.
Maintenance mode is per site cluster, and it freezes. It tells pacemaker to stop managing resources across the whole site cluster. Nothing moves, nothing is monitored, and nothing recovers. Existing VIFs stay exactly where they are and keep serving traffic; they simply stop being protected. The Web UI warns about this when you enter it, and the effect is visible in both crm_mon -1 and the VIF's own state:
*** Resource management is DISABLED *** The cluster will not attempt to start, stop or recover services Active Resources: * gm (ocf:heartbeat:IPaddr2): Started qs-node-111 (unmanaged)
Verified live: while the site cluster was in maintenance mode the VIF reported state Warning with Is Unmanaged true, stayed on its member, and an attempted move failed. Taking the cluster out of maintenance mode returned it to Normal without moving it.
So maintenance mode suppresses cluster activity and alerting for planned work across the whole cluster -- and while it is on, an appliance failure will not fail anything over. Keep the window short, and use standby mode instead if you only need to work on one appliance.
From the command line:
qs site-cluster-toggle-maintenance-mode --site=site-cluster-1 --enable-maintenance-mode=true
qs site-cluster-set-standby-mode --site=site-cluster-1 --storage-system=qs-node-111 \
--standby-mode=standby-manual-activate
The standby modes are named active, standby-auto-activate and standby-manual-activate on the command line.
What happens to client connections
A VIF move is not a graceful handover. The address is removed from one appliance and added to another, so every TCP connection bound to that address is broken and clients have to reconnect. Nothing drains sessions, logs initiators out, or unexports a share first.
What QuantaStor does do is announce the new location quickly: as soon as the address comes up, the resource agent broadcasts a burst of gratuitous ARP -- five packets at 200 ms intervals by default -- so switches and clients on the segment learn the new MAC without waiting for their caches to expire. That is why the measured outage is a couple of seconds rather than minutes.
How much a client notices depends entirely on the client:
Recovery is the client's business, not the appliance's, and it varies by protocol and by client configuration -- a hard NFS mount and an iSCSI initiator with a short timeout behave very differently across the same two-second outage. Test your own clients against a deliberate move before relying on the behaviour in production; a planned move is the cheapest way to find out what a real failover will look like. For what clients are connecting to, see NFS Configuration and Network Shares.
Note that the iSCSI Portal and NVMeoF Portal flags on the VIF are what make the floating address usable as a target portal at all. Without them the address moves, but block initiators were never pointed at it.
The practical consequence is that a move is cheap but not free. Schedule one the way you would schedule a brief service restart, and prefer moving a VIF deliberately at a quiet moment over letting a failure move it at a busy one.
Deleting a VIF has the same effect on connections as a move, without the reconnect target -- the Remove dialog says so explicitly.
Scale-up and scale-out differences
The VIF mechanism is the same in both configurations. What differs is what the address follows and where it is allowed to go.
| Scale-up (ZFS HA pool) | Scale-out (Ceph) | |
|---|---|---|
| Use case | Storage Pool (Scale-up HA) | Storage Pool (Scale-out), with config type Object, File or Block |
| Associated object | A storage pool HA failover group | A Ceph cluster |
| Eligible appliances (automatic) | The HA group's primary, secondary and tertiary | Every Ceph cluster member |
| Prerequisite on every eligible appliance | A port of the same name, online, on every site cluster member | The Ceph service the config type names must be running and its storage mounted |
| Relationship to the data | The address follows the pool; both move together | The data is already distributed. The address is a service endpoint, and moving it moves no data |
| What a failover costs | Pool import time on the new appliance, plus the address move | The address move only |
The important asymmetry is the last row. In scale-up, a VIF failover is usually part of a pool failover, and the pool import dominates the time -- the VIF is the small part. In scale-out there is nothing to import, so the numbers in How long a failover takes are close to the whole story.
Scale-up VIFs are ordered and colocated with the pool they belong to, which is what keeps the address and the pool on the same appliance. They also appear with an :ha tag rather than :sv, and they are created from the HA failover group; see Storage Pool HA Failover Interface Create, HA Cluster Setup (JBODs) and HA Cluster Setup (external SAN).
For scale-out, note that a site cluster is a prerequisite, not an optional extra: without one there is no cluster software to run the resource, so a Ceph cluster on its own cannot have a floating address. Set up the site cluster across the Ceph members first. See Scale-out File Setup (ceph), Scale-out Block Setup (ceph) and Scale-out Object Setup (ceph).
Cluster VIFs are also the preferred way to present Ceph iSCSI, because routing all the SCSI reservation traffic for a target through one floating address keeps it going through a single target instance.
If a VIF's address is used to activate replication schedules, arrival of the VIF on an appliance is what activates them there; see Remote-replication (DR).
Removing a cluster VIF
Removing a cluster VIF deletes the pacemaker resource and takes the address down. The dialog warns that active connections on the interface will be dropped, and offers Convert cluster VIF resource to local virtual IP, which is documented as keeping the address in service as an ordinary virtual interface, but does not -- the address is dropped (QSTOR-12508), so do not rely on it on the appliance it was last running on instead of removing it entirely. That is the option to use when you are dismantling a site cluster but still need the address.
qs site-vif-delete --vif-resource=<vif-id> qs site-vif-delete --vif-resource=<vif-id> --convert-to-vif=true
Remove the VIFs before deleting the site cluster that owns them. qs site-cluster-delete refuses while VIFs still reference the site cluster, and says so, precisely to stop a site cluster teardown taking a service address off the network unannounced.
Troubleshooting
The Web UI and the cluster disagree about a VIF. If the VIF list shows a state that crm_mon -1 contradicts, run a site cluster rescan. It re-reads the live cluster configuration and rebuilds QuantaStor's view of the rings, the VIFs and their location constraints from it, treating pacemaker as authoritative. It does not stop or restart anything.
The command takes no arguments and acts on the site cluster the appliance you run it on belongs to, so run it on a member:
qs site-cluster-rescan
A VIF reports MISSING. QuantaStor has a VIF object but pacemaker has no resource for it. A rescan will reconcile the record.
A VIF will not start anywhere. Read the failed actions in crm_mon -1 first -- one of the checks in What blocks a failover will normally name the cause, and each names its own remedy. The most common is the duplicate-address check firing because an appliance that lost cluster communication is still holding the address.
A VIF is stuck and reports unmanaged. The site cluster is in maintenance mode. Exit maintenance mode and the VIF is managed again.
A VIF will not move to a particular appliance. Check its location constraint on that appliance. None pins it away permanently, and the error message says so.
Command line reference
| Command | Purpose |
|---|---|
qs site-vif-create |
Create a cluster VIF, optionally converting an existing local one |
qs site-vif-list / qs site-vif-get |
List cluster VIFs, or show one in detail |
qs site-vif-modify |
Change the description or the location constraints |
qs site-vif-move |
Move a VIF to another member deliberately |
qs site-vif-delete |
Remove a VIF, optionally converting it to a local address |
qs site-vif-location-constraint-list |
List the weights for every VIF |
qs site-vif-location-constraint-get |
Show one member's weight for one VIF |
qs site-cluster-set-standby-mode |
Put one member into or out of standby |
qs site-cluster-toggle-maintenance-mode |
Freeze or unfreeze the whole site cluster |
qs site-cluster-restart-services |
Restart corosync and pacemaker on one appliance |
qs site-cluster-rescan |
Rebuild QuantaStor's view from the live cluster configuration |
Site cluster and heartbeat ring commands are covered on Site Cluster Setup.
Related pages
- Cluster VIFs -- adding a cluster VIF, and the four use case types in depth
- Site Cluster Setup -- the site cluster object, its members and its heartbeat rings
- Network Ports -- virtual interfaces, bonds, VLANs and static routes at the port level
- Grid Configuration -- the management grid a site cluster is built from
- Configure Member Standby -- the standby mode dialog and its three states
- HA Cluster Setup (JBODs) and HA Cluster Setup (external SAN) -- scale-up HA pool setup
- Storage Pool HA Failover Interface Create -- creating a VIF from an HA failover group
- Scale-out File Setup (ceph), Scale-out Block Setup (ceph), Scale-out Object Setup (ceph) -- scale-out configurations
- NFS Configuration and Network Shares -- what clients connect to through a VIF
- Remote-replication (DR) -- replication schedules activated by a VIF
- QuantaStor CLI Command Reference -- full argument lists for the commands above
Verified against QuantaStor 6.9.0.