High-availability VIF Management: Difference between revisions

From OSNEXUS Online Documentation Site
Jump to navigation Jump to search
← Older edit
VisualWikitext
m Point the site cluster reference at the new Site Cluster Setup page, which now exists (QSTOR-12352)
m osn-seo-utilities: docs-active-directory-spn-failover-with-the-ha @ b83457fda786 (approved in the portal)
 
(One intermediate revision by the same user not shown)
Line 1: Line 1:
Cluster virtual interfaces (VIFs) are floating IP addresses that move between the appliances of a site cluster so that a service address stays reachable when the appliance hosting it fails. This page covers the cluster-level behaviour of those addresses -- what makes one fail over, how to move one deliberately, and how to control which appliances it is allowed to run on. Both scale-up (ZFS HA pool) and scale-out (Ceph) configurations use them, and no special hardware is required.
Cluster virtual interfaces (VIFs) are floating IP addresses that move between the appliances of a site cluster so that a service address stays reachable when the appliance hosting it fails. This page covers the cluster-level behaviour of those addresses -- what makes one fail over, how to move one deliberately, and how to control which appliances it is allowed to run on. Both scale-up (ZFS HA pool) and scale-out (Ceph) configurations use them, and no special hardware is required.


For virtual interfaces at the network port level -- creating a plain second address on a port, bonds, VLANs, MTU and static routes -- see [[Network Ports]]. That page owns the port; this page owns the cluster behaviour layered on top of it.
For virtual interfaces at the network port level -- creating a plain second address on a port, bonds, VLANs, MTU and static routes -- see [[Network Ports]]. That page owns the port; this page owns the cluster behaviour layered on top of it. For adding a cluster VIF and for the four use case types, see [[Cluster VIFs]].


{| class="wikitable"
{| class="wikitable"
Line 10: Line 10:
| [[#Where cluster VIFs are managed|Where cluster VIFs are managed]] || The High-availability VIF Management tab
| [[#Where cluster VIFs are managed|Where cluster VIFs are managed]] || The High-availability VIF Management tab
|-
|-
| [[#Use cases|Use cases]] || What a VIF is for, and why some options are greyed out
| [[#Use cases and creating a VIF|Use cases and creating a VIF]] || Pointer to [[Cluster VIFs]], which owns adding a VIF and the four use case types
|-
| [[#Creating a cluster VIF|Creating a cluster VIF]] || Every field in the create dialog
|-
|-
| [[#How failover works|How failover works]] || What triggers a move, and how long it takes
| [[#How failover works|How failover works]] || What triggers a move, and how long it takes
Line 24: Line 22:
| [[#Standby mode and maintenance mode|Standby mode and maintenance mode]] || Evacuating one node versus freezing the whole cluster
| [[#Standby mode and maintenance mode|Standby mode and maintenance mode]] || Evacuating one node versus freezing the whole cluster
|-
|-
| [[#What happens to client connections|What happens to client connections]] || What a client sees during a move
| [[#What happens to client connections|What happens to client connections]] || What a client sees during a move, and giving a VIF name its own Active Directory (Kerberos) identity for SMB
|-
|-
| [[#Scale-up and scale-out differences|Scale-up and scale-out differences]] || Where the two configurations diverge
| [[#Scale-up and scale-out differences|Scale-up and scale-out differences]] || Where the two configurations diverge
|-
|-
| [[#Removing a cluster VIF|Removing a cluster VIF]] || Deletion, and converting back to a local address
| [[#Removing a cluster VIF|Removing a cluster VIF]] || Deletion, and converting back to a local address
|-
| [[#Troubleshooting|Troubleshooting]] || When the Web UI, the cluster and the clients disagree
|}
|}


Line 66: Line 66:
The grid below lists the '''Site Cluster Member Location Constraints''' for the selected VIF -- one row per member, with the weight that decides how strongly the VIF prefers that member.
The grid below lists the '''Site Cluster Member Location Constraints''' for the selected VIF -- one row per member, with the weight that decides how strongly the VIF prefers that member.


== Use cases ==
== Use cases and creating a VIF ==


[[File:havif_add_usecase.png|thumb|right|536px|The use case tab. Both Storage Pool options are greyed out here because this grid has no HA failover group and no Ceph pools, so QuantaStor has selected Other.]]
A cluster VIF's '''use case''' is what ties it to the resource its address has to follow, and it is chosen once, when the VIF is created. [[Cluster VIFs]] owns both topics: it covers adding a VIF field by field, in the dialog and from the command line, and each of the four use case types in depth -- what each one follows, what gates it, and what constraints it carries.
 
A VIF's use case tells the cluster software which resource the address has to follow, and it is set once at creation. The Web UI offers four:
 
{| class="wikitable"
! Use case !! What it is for !! Associated object !! Available when
|-
| '''Grid Primary''' || The single floating address the storage grid is managed through || none || always
|-
| '''Storage Pool (Scale-up HA)''' || Follows a ZFS pool as it fails over between appliances || an HA failover group || an HA failover group exists
|-
| '''Storage Pool (Scale-out)''' || A service address for Ceph file, block or object storage || a Ceph cluster, plus a config type of Object, File or Block || Ceph pools exist
|-
| '''Other''' || A floating address with no resource to follow || none || always
|}
 
'''The two Storage Pool options are gated on what exists in the grid.''' Scale-up is selectable only once an HA failover group has been created, and scale-out only once Ceph pools exist. Within scale-out, the config type is gated further: '''File''' needs a CephFS filesystem, '''Block''' needs Ceph pools, and '''Object''' needs both a RADOS gateway and an object pool group. If none of the gated options are available, QuantaStor preselects '''Other'''. A greyed-out option is therefore a statement about your grid, not a fault -- create the pool or HA group first and the option becomes selectable.
 
Two things are specific to '''Grid Primary'''. There can be only one per site cluster; a second attempt fails with {{Code|1=Grid virtual interface already exists for Site Cluster '<name>'.}} And iSCSI and NVMe-oF are permanently disabled on it -- the checkboxes are cleared and greyed as soon as you select the use case, and target port modify refuses to turn them back on later.
 
'''Where the Grid Primary VIF runs is where the grid master runs.''' Moving a Grid Primary VIF to another appliance moves grid mastership with it: after moving one from {{Code|1=qs-node-110}} to {{Code|1=qs-node-111}}, {{Code|1=qs system-list}} reported {{Code|1=qs-node-111}} as the grid primary and the tree header relabelled it {{Code|1=(grid master)}}. That is the point of the use case, and it is worth knowing before you move one.
 
The use case cannot be changed after creation. Neither the Modify dialog nor <code>[[QuantaStor CLI Command Reference#site-vif-modify|qs site-vif-modify]]</code> offers it; to change a VIF's use case, delete it and create it again.
 
== Creating a cluster VIF ==
 
[[File:havif_add_general.png|thumb|right|536px|The Virtual Interface tab. The IP address and subnet mask are pre-filled from the local network, and the port grid lists only the plain physical ports of the selected appliance.]]
 
{{Navigation|High-availability VIF Management &rarr; Site Cluster Virtual Interfaces &rarr; Add Cluster VIF ''(toolbar)''}}
 
The dialog is a three-step wizard: '''General''' (the use case, above), '''Virtual Interface''', and '''Location Constraints'''.
 
The '''Virtual Interface''' tab carries the addressing:
 
* '''Site''' -- the site cluster the VIF will belong to.
* '''IP Address''' -- the floating address. It is pre-filled with the local network prefix as a convenience, not a suggestion of a free address. '''Verify the address is unused before you commit it.''' QuantaStor checks that no port in the grid already has it and pings it once, but on a busy subnet a host that is merely powered off will pass both checks and collide later.
* '''Description''' -- optional free text.
* '''Subnet Mask''' -- defaults to the parent port's mask.
* '''FQDN''' -- optional. When set, the name is added to {{Code|1=/etc/hosts}} on every appliance in the grid, so internal services -- Kerberos among them -- can refer to the VIF by name. It must not be a name already used by one of the appliances.
* '''iSCSI Portal''' and '''NVMeoF Portal''' -- whether the floating address is an allowed portal for those protocols. Both are ticked by default for every use case except Grid Primary, where they are forced off. The setting is stored on the VIF, not the port, so it survives a move.
* '''Convert local virtual IP to cluster VIF''' -- takes an existing plain virtual interface and turns it into a cluster VIF instead of making you delete and recreate it. Ticking it clears and disables the address fields, because the address comes from the interface you are converting; the port grid then lists existing virtual interfaces rather than physical ports.
* '''Select the target port interface to attach to''' -- pick the appliance and the port. On failover the VIF attaches to the port '''with the same name''' on whichever appliance takes it, so the name has to exist everywhere the VIF is allowed to run.
 
That last point is the one that catches people out. The VIF does not remember a port on one machine; it remembers a port '''name'''. If {{Code|1=ens192}} exists on two members and the third calls its equivalent something else, the VIF cannot run on the third.
 
The equivalent from the command line is <code>[[QuantaStor CLI Command Reference#site-vif-create|qs site-vif-create]] --site=&lt;site&gt; --parent-port=&lt;port&gt; --ip-address=&lt;ip&gt;</code>, with optional <code>--netmask</code>, <code>--fqdn</code>, <code>--vif-use-case</code>, <code>--usecase-obj-id</code>, <code>--iscsi-enable</code>, <code>--nvmeof-enable</code>, <code>--location-config</code> and <code>--convert-vif</code>.
 
<pre style="font-size: smaller">
qs site-vif-create --site=site-cluster-1 --parent-port=ens192 \
    --ip-address=10.0.8.240 --netmask=255.255.0.0 --vif-use-case=grid-primary
</pre>
 
Once created, the address appears on the host as an alias of the parent port, labelled with the VIF's tag -- {{Code|1=ens192:gm}} for a Grid Primary VIF, {{Code|1=ens192:sv<nnnnnn><nn>}} for the others. Because the label is part of the interface name and the kernel limits that name to 15 characters, a long parent port name can leave no room for the tag; the create fails with an explicit message if so.
 
The VIF also shows up in [[Network Ports]] as a port of type Virtual. It cannot be edited there: the cluster software owns it, so Modify Network Port locks the config type to static and refuses address, MTU and bond changes.


== How failover works ==
== How failover works ==
Line 202: Line 148:
A deliberate move relocates a VIF to a member you choose -- before taking an appliance down for maintenance, or to rebalance which appliance is serving which address. The dialog asks for the '''Site Cluster''', the '''Cluster Virtual Interface''', and the '''Move to System'''; '''Current System''' is shown for reference and cannot be edited. There is no toolbar button, only the right-click menu.
A deliberate move relocates a VIF to a member you choose -- before taking an appliance down for maintenance, or to rebalance which appliance is serving which address. The dialog asks for the '''Site Cluster''', the '''Cluster Virtual Interface''', and the '''Move to System'''; '''Current System''' is shown for reference and cannot be edited. There is no toolbar button, only the right-click menu.


From the command line:
From the command line, with <code>[[QuantaStor CLI Command Reference#site-vif-move|qs site-vif-move]]</code>:


<pre style="font-size: smaller">
<pre style="font-size: smaller">
qs site-vif-move --vif-resource=10.0.8.240 --move-to-system=qs-node-111
qs site-vif-move --vif-resource=<vif-id> --move-to-system=qs-node-111
</pre>
</pre>


Line 242: Line 188:
At least one member must have a weight above None. The appliance whose port you chose on the Virtual Interface tab must be one of them -- a VIF cannot be created pinned away from the port it is being attached to.
At least one member must have a weight above None. The appliance whose port you chose on the Virtual Interface tab must be one of them -- a VIF cannot be created pinned away from the port it is being attached to.


From the command line, weights are given as {{Code|1=<system>:<weight>}} pairs, and any non-negative number is accepted rather than only the four the Web UI offers:
From the command line, weights are given to <code>[[QuantaStor CLI Command Reference#site-vif-modify|qs site-vif-modify]]</code> as {{Code|1=<system>:<weight>}} pairs, and any non-negative number is accepted rather than only the four the Web UI offers:


<pre style="font-size: smaller">
<pre style="font-size: smaller">
qs site-vif-modify --vif-resource=10.0.8.240 \
qs site-vif-modify --vif-resource=<vif-id> \
     --location-config=qs-node-110:300,qs-node-111:200,qs-node-112:0
     --location-config=qs-node-110:300,qs-node-111:200,qs-node-112:0
</pre>
</pre>
Line 281: Line 227:
{{Navigation|High-availability VIF Management &rarr; Site Clusters &rarr; Site Cluster ''(select + right-click)'' &rarr; Enter Maintenance Mode...}}
{{Navigation|High-availability VIF Management &rarr; Site Clusters &rarr; Site Cluster ''(select + right-click)'' &rarr; Enter Maintenance Mode...}}


From the command line:
From the command line, with <code>[[QuantaStor CLI Command Reference#site-cluster-toggle-maintenance-mode|qs site-cluster-toggle-maintenance-mode]]</code> and <code>[[QuantaStor CLI Command Reference#site-cluster-set-standby-mode|qs site-cluster-set-standby-mode]]</code>:


<pre style="font-size: smaller">
<pre style="font-size: smaller">
Line 297: Line 243:
What QuantaStor does do is announce the new location quickly: as soon as the address comes up, the resource agent broadcasts a burst of gratuitous ARP -- five packets at 200 ms intervals by default -- so switches and clients on the segment learn the new MAC without waiting for their caches to expire. That is why the measured outage is a couple of seconds rather than minutes.
What QuantaStor does do is announce the new location quickly: as soon as the address comes up, the resource agent broadcasts a burst of gratuitous ARP -- five packets at 200 ms intervals by default -- so switches and clients on the segment learn the new MAC without waiting for their caches to expire. That is why the measured outage is a couple of seconds rather than minutes.


How much a client notices depends entirely on the client:
How much a client notices depends entirely on the client. Recovery is the client's business, not the appliance's, and it varies by protocol and by client configuration -- a hard NFS mount and an iSCSI initiator with a short timeout behave very differently across the same two-second outage. Test your own clients against a deliberate move before relying on the behaviour in production; a planned move is the cheapest way to find out what a real failover will look like. For what clients are connecting to, see [[NFS Configuration]] and [[Network Shares]].
 
Recovery is the client's business, not the appliance's, and it varies by protocol and by client configuration -- a hard NFS mount and an iSCSI initiator with a short timeout behave very differently across the same two-second outage. Test your own clients against a deliberate move before relying on the behaviour in production; a planned move is the cheapest way to find out what a real failover will look like. For what clients are connecting to, see [[NFS Configuration]] and [[Network Shares]].


Note that the '''iSCSI Portal''' and '''NVMeoF Portal''' flags on the VIF are what make the floating address usable as a target portal at all. Without them the address moves, but block initiators were never pointed at it.
Note that the '''iSCSI Portal''' and '''NVMeoF Portal''' flags on the VIF are what make the floating address usable as a target portal at all. Without them the address moves, but block initiators were never pointed at it.
Line 306: Line 250:


Deleting a VIF has the same effect on connections as a move, without the reconnect target -- the Remove dialog says so explicitly.
Deleting a VIF has the same effect on connections as a move, without the reconnect target -- the Remove dialog says so explicitly.
=== SMB clients and Active Directory: giving a VIF name its own Kerberos identity ===
When the appliances are joined to Active Directory (see [[Active Directory Configuration]]), SMB clients that connect by a VIF's host name rather than by an appliance's own name need one extra piece of setup. A VIF is a floating IP address with a DNS name and no computer account of its own, so Active Directory has no {{Code|1=cifs/<vif-name>}} service principal name (SPN) to issue a Kerberos service ticket against. In a domain where NTLM is disabled, a client connecting to {{Code|1=\\<vif-name>\<share>}} therefore cannot authenticate at all.
The fix is to give the VIF name its own AD computer object with {{Code|1=cifs/}} SPNs, and put that object's keys in {{Code|1=/etc/krb5.keytab}} on '''every appliance the VIF can run on''' -- for a scale-up VIF, every member of the HA failover group. '''Nothing happens in Active Directory at failover.''' Every eligible appliance holds the VIF's keys all the time, so whichever one currently hosts the address can answer for its name. An appliance that is missing the keys fails Kerberos authentication by the VIF name whenever the VIF lands on it, so the keys have to be merged everywhere, not just on the current owner.
Two pieces make this work:
* '''The domain join prepares Samba.''' Joining the domain sets {{Code|1=kerberos method = dedicated keytab}} and {{Code|1=dedicated keytab file = /etc/krb5.keytab}} in {{Code|1=/etc/samba/smb.conf}}. With the default {{Code|1=system keytab}} method, smbd keeps only the principals that match the appliance's own names and silently ignores a VIF's entries. Leaving the domain removes both settings and deletes {{Code|1=/etc/krb5.keytab}}.
* '''A script creates and distributes the VIF's identity.''' {{Code|1=/opt/osnexus/quantastor/bin/qs_vif_kerberos_setup.sh}} creates the AD computer object, exports its keytab, and merges it into {{Code|1=/etc/krb5.keytab}}. Nothing in the Web UI or the {{Code|1=qs}} CLI does this; run the script as root on an appliance.
==== Before you start ====
* Every appliance the VIF can run on is joined to the domain.
* {{Code|1=msktutil}} and {{Code|1=krb5-user}} are installed. They are recommended rather than required packages, so they can be missing; the script stops and names whichever one is absent.
* You have an AD account that can create computer objects. A machine account cannot do it, because AD refuses an SPN whose host part is not the account's own name.
* The VIF name is 15 characters or fewer -- the NetBIOS/sAMAccountName limit.
* Clients can resolve the VIF's FQDN to the VIF's IP address. Create a DNS A record, or an entry in the clients' hosts files. The '''FQDN''' field on the cluster VIF (see [[Cluster VIFs]]) adds the name to {{Code|1=/etc/hosts}} on every appliance in the grid so that internal services, Kerberos included, can resolve it, but it does not publish anything to your clients' DNS.
==== Creating the VIF's identity ====
Run {{Code|1=create}} on any one appliance and list the others with {{Code|1=--nodes}}:
<pre style="font-size: smaller">
/opt/osnexus/quantastor/bin/qs_vif_kerberos_setup.sh create --vif-name sharevif \
    --ad-admin Administrator --nodes qs-node-111,qs-node-112
</pre>
The script prompts for the AD account's password, or reads it from the {{Code|1=QS_AD_ADMIN_PASSWORD}} environment variable. It never accepts the password as an argument, because the command line is visible in the process table. Then it:
# Reads the realm from the appliance's Samba configuration, so you do not retype it.
# Checks that AD does not already have an account named {{Code|1=<vif-name>$}}, and stops before changing anything if it does (see below).
# Creates the computer object with two SPNs, {{Code|1=cifs/<vif-name>}} and {{Code|1=cifs/<fqdn>}}. The FQDN defaults to the VIF name followed by the realm in lower case; set {{Code|1=--fqdn}} if the name clients use is different. Only AES128 and AES256 keys are issued; RC4 is left out because it is deprecated and AD may refuse it.
# Merges the keytab into the local {{Code|1=/etc/krb5.keytab}}, keeping a timestamped backup, and asks smbd to reload its configuration rather than restarting it, so live SMB sessions are not dropped.
# Verifies the local appliance (see below).
# Copies the keytab to each appliance in {{Code|1=--nodes}} over root SSH, merges it there and deletes the copy.
QuantaStor appliances do not have root SSH trust between them by default, so the copy step usually cannot finish on its own. When it cannot, the script keeps the exported keytab and prints the exact {{Code|1=scp}} and {{Code|1=merge}} commands to run for each remaining appliance. Run them, then delete the keytab file -- it holds the VIF account's keys. Without {{Code|1=--nodes}}, only the local appliance is updated and the same instructions are printed.
On an appliance where you copied the keytab by hand:
<pre style="font-size: smaller">
/opt/osnexus/quantastor/bin/qs_vif_kerberos_setup.sh merge --keytab /tmp/qs_vif_keytab.XXXXXX
</pre>
A merge is safe to repeat: if every entry is already present it does nothing.
==== Verifying every appliance ====
Run {{Code|1=verify}} on each appliance the VIF can run on:
<pre style="font-size: smaller">
/opt/osnexus/quantastor/bin/qs_vif_kerberos_setup.sh verify --vif-name sharevif
</pre>
It checks three things, and fails if any of them is wrong:
* The appliance's keytab holds {{Code|1=cifs/<vif-name>}} entries.
* The keys are the ones Active Directory currently issues tickets with. It asks the KDC for a service ticket for each SPN, authenticating as the appliance's own machine account, and checks that the keytab can decrypt it. This catches keys that are present but out of date, which counting entries does not.
* Samba is using {{Code|1=kerberos method = dedicated keytab}}. If it is not, the VIF's entries are ignored and SMB by the VIF name fails with {{Code|1=NT_STATUS_LOGON_FAILURE}}. Leave the domain and join it again so the join sets the method, then repeat {{Code|1=create --rotate}}, because leaving deletes the keytab.
If verify reports stale keys straight after a {{Code|1=create}}, the new computer object may not have replicated to every domain controller yet. Wait a minute and run it again.
==== Rotating, or recovering after a domain leave ====
Running {{Code|1=create}} for a name AD already has would change that account's keys, and every appliance still holding the old keys would immediately start rejecting tickets for the VIF name. So the script refuses, and changes nothing. If the appliances already hold the keys there is nothing to do; confirm with {{Code|1=verify}}.
You need new keys when an appliance has lost them -- leaving the domain deletes {{Code|1=/etc/krb5.keytab}} outright -- or when you rotate them deliberately. Re-run {{Code|1=create}} with {{Code|1=--rotate}}, and list '''every''' other appliance the VIF can run on in {{Code|1=--nodes}}: rotating replaces the keys in AD, so an appliance that does not merge the new keytab fails Kerberos authentication by the VIF name until it does. When a merge brings in newer keys, the script prints a warning that names each changed principal.
After any domain leave and re-join, run {{Code|1=verify}} on that appliance.
==== What QuantaStor refreshes automatically ====
The VIF's keys belong to its own AD computer object and do not change unless you rotate them. The appliance's '''own''' machine account is different: Active Directory rotates its password, and with the dedicated keytab method Samba does not rewrite the appliance's own keytab entries when that happens, so access by the appliance's own host name could fail. QuantaStor refreshes those entries itself, using the machine account credentials, without an administrator password:
* once a day,
* on the first pass after the QuantaStor service starts, so an appliance that was down through a rotation catches up straight away,
* whenever storage pools are imported, and during an HA pool failover on the appliance taking the pool over, so the node that has just picked up the VIF re-checks its own keytab rather than waiting for the daily cycle.
The refresh adds or updates only the appliance's own principals and leaves the VIF's entries untouched. If it fails, QuantaStor raises an alert saying that SMB access by the appliance's own host name may fail after the next machine password rotation. An appliance joined in SSSD-only mode skips the refresh and logs that it did so.


== Scale-up and scale-out differences ==
== Scale-up and scale-out differences ==
Line 341: Line 366:
{{Navigation|High-availability VIF Management &rarr; Site Cluster Virtual Interfaces &rarr; Remove Cluster VIF ''(toolbar)''}}
{{Navigation|High-availability VIF Management &rarr; Site Cluster Virtual Interfaces &rarr; Remove Cluster VIF ''(toolbar)''}}


Removing a cluster VIF deletes the pacemaker resource and takes the address down. The dialog warns that active connections on the interface will be dropped, and offers '''Convert cluster VIF resource to local virtual IP''', which keeps the address in service as an ordinary virtual interface on the appliance it was last running on instead of removing it entirely. That is the option to use when you are dismantling a site cluster but still need the address.
Removing a cluster VIF deletes the pacemaker resource and takes the address down. The dialog warns that active connections on the interface will be dropped. It also offers '''Convert cluster VIF resource to local virtual IP''', which is meant to keep the address in service as an ordinary virtual interface on the appliance it was last running on. It currently does not: the address is dropped, as with a plain removal, so do not rely on it to keep an address in service.


<pre style="font-size: smaller">
<pre style="font-size: smaller">
qs site-vif-delete --vif-resource=10.0.8.240
qs site-vif-delete --vif-resource=<vif-id>
qs site-vif-delete --vif-resource=10.0.8.240 --convert-to-vif=true
qs site-vif-delete --vif-resource=<vif-id> --convert-to-vif=true
</pre>
</pre>


Remove the VIFs before deleting the site cluster that owns them. <code>[[QuantaStor CLI Command Reference#site-cluster-delete|qs site-cluster-delete]]</code> refuses while VIFs still reference the site cluster, and says so, precisely to stop a site cluster teardown taking a service address off the network unannounced.
Remove the VIFs before deleting the site cluster that owns them. <code>[[QuantaStor CLI Command Reference#site-cluster-delete|qs site-cluster-delete]]</code> refuses while VIFs still reference the site cluster, and says so, precisely to stop a site cluster teardown taking a service address off the network unannounced.
If the VIF had its own Active Directory identity, its computer object stays in AD after the VIF is removed; QuantaStor does not delete it.


== Troubleshooting ==
== Troubleshooting ==
Line 354: Line 381:
'''The Web UI and the cluster disagree about a VIF.''' If the VIF list shows a state that {{Code|1=crm_mon -1}} contradicts, run a site cluster rescan. It re-reads the live cluster configuration and rebuilds QuantaStor's view of the rings, the VIFs and their location constraints from it, treating pacemaker as authoritative. It does not stop or restart anything.
'''The Web UI and the cluster disagree about a VIF.''' If the VIF list shows a state that {{Code|1=crm_mon -1}} contradicts, run a site cluster rescan. It re-reads the live cluster configuration and rebuilds QuantaStor's view of the rings, the VIFs and their location constraints from it, treating pacemaker as authoritative. It does not stop or restart anything.


The command takes no arguments and acts on the site cluster the appliance you run it on belongs to, so run it on a member:
The command, <code>[[QuantaStor CLI Command Reference#site-cluster-rescan|qs site-cluster-rescan]]</code>, takes no arguments and acts on the site cluster the appliance you run it on belongs to, so run it on a member:


<pre style="font-size: smaller">
<pre style="font-size: smaller">
Line 367: Line 394:


'''A VIF will not move to a particular appliance.''' Check its location constraint on that appliance. None pins it away permanently, and the error message says so.
'''A VIF will not move to a particular appliance.''' Check its location constraint on that appliance. None pins it away permanently, and the error message says so.
'''SMB by the VIF name works on one appliance but fails after a failover.''' The appliance now hosting the VIF does not hold the VIF's current keys. Run {{Code|1=qs_vif_kerberos_setup.sh verify}} there; see [[#SMB clients and Active Directory: giving a VIF name its own Kerberos identity|SMB clients and Active Directory]].


== Command line reference ==
== Command line reference ==
Line 400: Line 429:
== Related pages ==
== Related pages ==


* [[Cluster VIFs]] -- adding a cluster VIF, and the four use case types in depth
* [[Site Cluster Setup]] -- the site cluster object, its members and its heartbeat rings
* [[Site Cluster Setup]] -- the site cluster object, its members and its heartbeat rings
* [[Network Ports]] -- virtual interfaces, bonds, VLANs and static routes at the port level
* [[Network Ports]] -- virtual interfaces, bonds, VLANs and static routes at the port level
Line 408: Line 438:
* [[Scale-out File Setup (ceph)]], [[Scale-out Block Setup (ceph)]], [[Scale-out Object Setup (ceph)]] -- scale-out configurations
* [[Scale-out File Setup (ceph)]], [[Scale-out Block Setup (ceph)]], [[Scale-out Object Setup (ceph)]] -- scale-out configurations
* [[NFS Configuration]] and [[Network Shares]] -- what clients connect to through a VIF
* [[NFS Configuration]] and [[Network Shares]] -- what clients connect to through a VIF
* [[Active Directory Configuration]] -- joining the appliances to Active Directory
* [[Remote-replication (DR)]] -- replication schedules activated by a VIF
* [[Remote-replication (DR)]] -- replication schedules activated by a VIF
* [[QuantaStor CLI Command Reference]] -- full argument lists for the commands above
* [[QuantaStor CLI Command Reference]] -- full argument lists for the commands above

Latest revision as of 20:00, 7 October 2026

Cluster virtual interfaces (VIFs) are floating IP addresses that move between the appliances of a site cluster so that a service address stays reachable when the appliance hosting it fails. This page covers the cluster-level behaviour of those addresses -- what makes one fail over, how to move one deliberately, and how to control which appliances it is allowed to run on. Both scale-up (ZFS HA pool) and scale-out (Ceph) configurations use them, and no special hardware is required.

For virtual interfaces at the network port level -- creating a plain second address on a port, bonds, VLANs, MTU and static routes -- see Network Ports. That page owns the port; this page owns the cluster behaviour layered on top of it. For adding a cluster VIF and for the four use case types, see Cluster VIFs.

Section Purpose
Grid, site cluster, heartbeat ring, HA group and VIF How the objects nest, and which one a VIF actually belongs to
Where cluster VIFs are managed The High-availability VIF Management tab
Use cases and creating a VIF Pointer to Cluster VIFs, which owns adding a VIF and the four use case types
How failover works What triggers a move, and how long it takes
What blocks a failover Three checks that will refuse to start a VIF, and what they mean
Moving a VIF deliberately Planned relocation, and why it is not the same as a failover
Location constraints Pinning a VIF to, or away from, particular appliances
Standby mode and maintenance mode Evacuating one node versus freezing the whole cluster
What happens to client connections What a client sees during a move, and giving a VIF name its own Active Directory (Kerberos) identity for SMB
Scale-up and scale-out differences Where the two configurations diverge
Removing a cluster VIF Deletion, and converting back to a local address
Troubleshooting When the Web UI, the cluster and the clients disagree

Grid, site cluster, heartbeat ring, HA group and VIF

This hierarchy is the thing readers most often get wrong, so it is worth stating plainly. Each layer sits on the one before it, and a VIF is at the top.

A storage grid is the management layer. It is the set of appliances you administer together through one interface, and it says nothing about high availability -- see Grid Configuration. Grid membership is a prerequisite: a site cluster can only be built from appliances that are already grid members.

A site cluster is the high-availability layer. It is a group of appliances at one location -- the location can span buildings, but the members are normally in close proximity, because they have to share a network. A site cluster is what runs the cluster software (corosync and pacemaker) that decides where a floating resource lives. A grid can contain several site clusters, and an appliance can belong to at most one. Site clusters are covered on Site Cluster Setup.

A cluster heartbeat ring is how the members of a site cluster tell whether each other are alive. It is a network path, not an address that serves clients: each member contributes one port, and every port in a ring must be on the same subnet. A site cluster has at least one ring and at most two, and two rings on separate networks is the recommended configuration, so that losing one network does not look like losing a node. The first ring is created for you when you create the site cluster.

A storage pool HA failover group is the scale-up layer. It ties a ZFS pool to the appliances that are allowed to import it, so the pool itself can fail over. It exists only in scale-up configurations; a Ceph cluster provides its own redundancy and has no HA group. See HA Cluster Setup (JBODs) and HA Cluster Setup (external SAN).

A cluster virtual interface is the address clients talk to. It belongs to a site cluster, is attached to a named port on one member at a time, and is the layer that makes a service address survive the loss of an appliance. Its use case is what ties it to the resource it has to follow -- an HA failover group for a ZFS pool, a Ceph cluster for scale-out storage, or nothing at all.

The two things worth taking away: a heartbeat ring carries cluster gossip and a VIF carries client traffic, and they are separate; and a VIF's parent is the site cluster, not the pool, even when its whole purpose is to follow a pool.

Where cluster VIFs are managed

The High-availability VIF Management tab, with a site cluster selected. The Site Cluster Members grid shows each member's standby mode, and the Cluster Heartbeat Ring Ports grid shows which address each member contributes to the ring.
Navigation: High-availability VIF Management

Cluster VIFs and the site clusters they belong to have their own top-level tab. The tree on the left has two sections:

  • Site Clusters -- the site clusters in the grid, each expanding to show its heartbeat rings. Selecting a site cluster shows its members and its ring ports, and the toolbar switches to the site cluster operations.
  • Site Cluster Virtual Interfaces -- the cluster VIFs, grouped by site cluster. Selecting this section switches the toolbar to Virtual Interface Management with Add Cluster VIF, Modify Cluster VIF and Remove Cluster VIF.
The Site Cluster Virtual Interfaces section. Note that Managed By and Started On differ: the VIF object is owned by one appliance but the address is currently running on another. Move is only on the right-click menu.

Two columns in the Site Cluster Virtual Interfaces grid are easy to confuse. Managed By is the appliance that owns the VIF configuration object; Started On is the appliance where the address is actually up right now. They differ whenever the VIF has moved, and it is Started On that tells you where client traffic is going.

Move Site Cluster VIF is on the right-click menu only -- there is no toolbar button for it. Right-clicking a VIF gives Move Site Cluster VIF..., Remove Site Cluster VIF..., Modify Site Cluster VIF... and Properties....

The grid below lists the Site Cluster Member Location Constraints for the selected VIF -- one row per member, with the weight that decides how strongly the VIF prefers that member.

Use cases and creating a VIF

A cluster VIF's use case is what ties it to the resource its address has to follow, and it is chosen once, when the VIF is created. Cluster VIFs owns both topics: it covers adding a VIF field by field, in the dialog and from the command line, and each of the four use case types in depth -- what each one follows, what gates it, and what constraints it carries.

How failover works

Underneath, each cluster VIF is a pacemaker resource. QuantaStor creates it as an IPaddr2 resource named after the VIF's tag, with the address, the netmask as a CIDR prefix, the parent port as the NIC, and a monitor that runs every 10 seconds. You can see it on any member with crm_mon -1: 

Active Resources:
  * gm	(ocf:heartbeat:IPaddr2):	 Started qs-node-111

Three things move a VIF automatically:

  • The monitor fails. The 10-second monitor finds the address is no longer correctly configured on the host, and pacemaker restarts the resource -- on another member if it cannot start locally.
  • The appliance leaves the cluster. The remaining members stop hearing it on the heartbeat rings and pacemaker re-places its resources. How fast that is detected is governed by the corosync token settings; the shipped configuration uses a 3000 ms token with 10 retransmits before loss, so a lost member is normally declared within a few seconds. /etc/corosync/corosync.conf on the appliance is authoritative -- it is managed by QuantaStor and carries a warning header saying so.
  • A member is put into standby. See Standby mode and maintenance mode.

Where it goes is decided by the location constraints, which give each member a score, subject to the checks in the next section.

Two cluster-wide settings are worth knowing because they explain behaviour that otherwise looks wrong. QuantaStor sets stonith-enabled to false, so pacemaker does not fence a node it has lost contact with -- the duplicate-address check described below is what protects against two appliances answering for one address instead. And it sets no-quorum-policy to ignore, so a member that finds itself in a minority partition does not stop its resources on its own.

How long a failover takes

Measured on a lab site cluster by pinging the VIF at 200 ms intervals from a third appliance and counting the gap. These are observations on idle test systems, not a specification:

What was done Address unreachable for
Deliberate move, 3-node scale-up cluster 2.50 s
Member put into standby while hosting the VIF 2.49 s
Cluster services restarted on the hosting member 2.68 s
Location weight raised above the stickiness threshold 3.60 s
Deliberate move, 4-node scale-out cluster 3.11 s
Cluster services restarted on the hosting member, scale-out 3.12 s

So a planned relocation and an unplanned one cost about the same, in the region of two to three seconds, and a scale-out VIF is slightly slower because the resource agent checks that the Ceph filesystem is actually mounted before it will start the address. What is not represented here is the abrupt loss of a whole appliance -- losing power, for instance. That adds the time corosync needs to declare the member gone, on top of the figures above.

What blocks a failover

QuantaStor ships its own version of the IPaddr2 resource agent, installed over the distribution's copy at /usr/lib/ocf/resource.d/heartbeat/IPaddr2 (the original is kept alongside it as IPaddr2.backup). It adds preflight checks that refuse to bring a floating address up in situations where doing so would make things worse. When one of them refuses, the reason appears in the failed actions list in crm_mon -1 and in the task's error detail, so it is worth recognising them.

The address is still answering somewhere else. Before claiming the address, the agent flushes the ARP cache and pings it. If anything replies, it waits and tries again, and if it still replies it refuses: 

Failed Resource Actions:
  * gm start on qs-node-110 returned 'error' because 'IP address is still in use
    on other node, failover blocked.' at Thu Sep  3 10:45:41 2026 after 7.607s

This is the protection that stands in for fencing, and it is easy to trigger accidentally. If an appliance loses cluster communication but keeps running -- a heartbeat network problem rather than a crash -- it goes on holding its floating addresses while the surviving members conclude it has gone. The survivors then try to take the addresses over, find them still answering, and correctly decline. The VIF stops being managed anywhere until the stranded appliance rejoins or is shut down, but it does not go offline: it is still up on the appliance that never lost it. In a lab reproduction of exactly this -- cluster communication killed on the hosting member while the appliance itself stayed up -- the VIF was pinged continuously throughout and lost no packets at all, while the other two members logged the message above.

The remedy is to restore cluster communication on the stranded member. qs site-cluster-restart-services --storage-system=<system> restarts corosync and pacemaker on one appliance, which brings it back into the cluster; the address is released and placed properly, and the failed actions clear on their own. Note that this restart will relocate any VIF the appliance is currently hosting.

The QuantaStor service is not running. The agent refuses to start a VIF on an appliance whose core service is stopped, with QuantaStor service is stopped, failover blocked. An address that works but has no service behind it is worse than an address that has moved elsewhere.

The appliance is in manual standby. The agent checks for the manual-standby marker file independently of pacemaker's own standby flag, and refuses with a message telling you to take the node out of standby. The duplication is deliberate: pacemaker's flag can be cleared underneath QuantaStor when the cluster stack restarts, and without the second check a node you had deliberately parked could quietly start hosting resources again.

For scale-out VIFs, the storage has to be present. A file VIF will not start on an appliance unless the CephFS pool is mounted and its export is mounted there; an object VIF will not start unless the RADOS gateway is running and configured. This is what keeps a scale-out service address from landing somewhere that cannot serve it. The requirement is recorded per VIF in /var/opt/osnexus/quantastor/clustervif_<port>_<tag>.uses, which is written to every member of the site cluster: 

cephfs_id="7f69711e-de3e-fdd9-4ecd-5262201d1a56"
cephfs_name="cephfs-pool-1"
use_case_obj_id=c06b4f30-8c65-6dac-7421-239daedbe3bb
use_scaleout_filepool=true

Moving a VIF deliberately

Move Site Cluster Virtual Interface. Current System is read-only; pick the destination in Move to System.
Navigation: High-availability VIF Management → Site Cluster Virtual Interfaces → Site Cluster VIF (select + right-click) → Move Site Cluster VIF...

A deliberate move relocates a VIF to a member you choose -- before taking an appliance down for maintenance, or to rebalance which appliance is serving which address. The dialog asks for the Site Cluster, the Cluster Virtual Interface, and the Move to System; Current System is shown for reference and cannot be edited. There is no toolbar button, only the right-click menu.

From the command line, with qs site-vif-move: 

qs site-vif-move --vif-resource=<vif-id> --move-to-system=qs-node-111

A move is not a failover, and the difference matters:

  • A move does not leave a preference behind. Pacemaker's own move mechanism works by pinning the resource with a temporary constraint; QuantaStor clears that constraint once the move completes. Checking the location constraints before and after a move shows them unchanged, so the VIF remains free to fail over normally afterwards. This is why you should move a VIF rather than reach for pacemaker directly -- doing it by hand leaves a pin that quietly prevents failover.
  • A move will not override a pin. If the destination's location weight is None, the move is rejected before anything happens, with a message naming the fix: Cannot move Site Cluster virtual network interface '<vif>' to system '<system>': that system's location constraint for this interface is set to 'None', which pins the interface to never run there. Set the location constraint for '<system>' to Low, Medium, or High before moving the interface.
  • A move will not work while the cluster is frozen. In maintenance mode the attempt fails, because pacemaker is not managing the resource at all.

The destination must be a member of the same site cluster. Moving a VIF to an appliance that is merely in the same grid is not possible, and is not a meaningful request -- the cluster software has no presence there.

Location constraints

Location constraints with Automatic cleared. Each member gets a weight, and the row has to be ticked as well as weighted.

Location constraints are how you say which appliances a VIF prefers, and which it must never run on. Each member of the site cluster gets a weight, which becomes the pacemaker score for that member -- the higher the score, the stronger the preference.

The Web UI offers four weights:

Weight Score Effect
None −INFINITY The VIF will never run on this member. Automatic failover will not place it here, and a deliberate move to it is rejected.
Low 100 Eligible, least preferred
Medium 200 Eligible
High 300 Eligible, most preferred

Automatic Location Constraints is ticked by default and is the right choice unless you have a specific reason to pin. What it computes depends on the use case: for Grid Primary and Other it gives every member of the site cluster an equal weight of 100; for scale-up it weights the HA failover group's primary, secondary and tertiary appliances at 100; for scale-out it weights every Ceph cluster member at 100. Any remaining site cluster member is set to None. Note that automatic mode is flat -- it does not rank the members it selects, so it expresses "these are eligible" rather than "prefer this one".

Clearing the checkbox enables the grid. Two things are needed per member, not one: set the weight and tick the row. The dialog reads the ticked rows, so a weight set on an unticked row is not submitted, and clearing Automatic without ticking anything fails validation with a message telling you to select and configure the node weights.

At least one member must have a weight above None. The appliance whose port you chose on the Virtual Interface tab must be one of them -- a VIF cannot be created pinned away from the port it is being attached to.

From the command line, weights are given to qs site-vif-modify as <system>:<weight> pairs, and any non-negative number is accepted rather than only the four the Web UI offers: 

qs site-vif-modify --vif-resource=<vif-id> \
    --location-config=qs-node-110:300,qs-node-111:200,qs-node-112:0

Read them back with qs site-vif-location-constraint-list, or for one member with qs site-vif-location-constraint-get --vif-resource=<vif> --storage-system=<system>. A weight given for one member leaves the others as they are, so you can adjust a single appliance without restating the whole set.

Raising a weight does not usually move a running VIF

This is the least obvious thing on this page, and it looks like a bug when you meet it. QuantaStor sets pacemaker's resource stickiness to 1000 on every site cluster, which is a preference for leaving a running resource where it is. All four Web UI weights are well below that, so changing weights through the Web UI will not relocate a VIF that is already running. It changes where the VIF will land the next time it has to be placed.

Verified on a running VIF: with the hosting member at Medium (200) and another member raised to High (300), the VIF stayed put. Only when a weight was set above 1000 from the command line did it relocate. If you want a running VIF on a particular appliance now, move it -- do not raise its weight and wait.

The corresponding useful case is None, which does take effect immediately, because −INFINITY beats stickiness. Setting a member to None will push a VIF off it.

Standby mode and maintenance mode

Both take a site cluster out of normal operation, and they do close to opposite things to VIFs. Choosing the wrong one is the most consequential mistake in this area.

Standby mode is per appliance, and it evacuates. Putting a member into standby moves its resources to a healthy member and stops it receiving any more, while the rest of the cluster keeps protecting itself normally. This is what you want before working on one appliance. In a lab measurement, a member hosting a VIF was put into standby and the address moved to another member with 2.49 s of unreachability. Standby has three states -- Active, Standby Auto Activation and Standby Manual Activation -- and Configure Member Standby covers the dialog and the difference between them.

Maintenance mode is per site cluster, and it freezes. It tells pacemaker to stop managing resources across the whole site cluster. Nothing moves, nothing is monitored, and nothing recovers. Existing VIFs stay exactly where they are and keep serving traffic; they simply stop being protected. The Web UI warns about this when you enter it, and the effect is visible in both crm_mon -1 and the VIF's own state: 

              *** Resource management is DISABLED ***
  The cluster will not attempt to start, stop or recover services

Active Resources:
  * gm	(ocf:heartbeat:IPaddr2):	 Started qs-node-111 (unmanaged)

Verified live: while the site cluster was in maintenance mode the VIF reported state Warning with Is Unmanaged true, stayed on its member, and an attempted move failed. Taking the cluster out of maintenance mode returned it to Normal without moving it.

So maintenance mode suppresses cluster activity and alerting for planned work across the whole cluster -- and while it is on, an appliance failure will not fail anything over. Keep the window short, and use standby mode instead if you only need to work on one appliance.

Navigation: High-availability VIF Management → Site Clusters → Site Cluster (select + right-click) → Enter Maintenance Mode...

From the command line, with qs site-cluster-toggle-maintenance-mode and qs site-cluster-set-standby-mode: 

qs site-cluster-toggle-maintenance-mode --site=site-cluster-1 --enable-maintenance-mode=true
qs site-cluster-set-standby-mode --site=site-cluster-1 --storage-system=qs-node-111 \
    --standby-mode=standby-manual-activate

The standby modes are named active, standby-auto-activate and standby-manual-activate on the command line.

What happens to client connections

A VIF move is not a graceful handover. The address is removed from one appliance and added to another, so every TCP connection bound to that address is broken and clients have to reconnect. Nothing drains sessions, logs initiators out, or unexports a share first.

What QuantaStor does do is announce the new location quickly: as soon as the address comes up, the resource agent broadcasts a burst of gratuitous ARP -- five packets at 200 ms intervals by default -- so switches and clients on the segment learn the new MAC without waiting for their caches to expire. That is why the measured outage is a couple of seconds rather than minutes.

How much a client notices depends entirely on the client. Recovery is the client's business, not the appliance's, and it varies by protocol and by client configuration -- a hard NFS mount and an iSCSI initiator with a short timeout behave very differently across the same two-second outage. Test your own clients against a deliberate move before relying on the behaviour in production; a planned move is the cheapest way to find out what a real failover will look like. For what clients are connecting to, see NFS Configuration and Network Shares.

Note that the iSCSI Portal and NVMeoF Portal flags on the VIF are what make the floating address usable as a target portal at all. Without them the address moves, but block initiators were never pointed at it.

The practical consequence is that a move is cheap but not free. Schedule one the way you would schedule a brief service restart, and prefer moving a VIF deliberately at a quiet moment over letting a failure move it at a busy one.

Deleting a VIF has the same effect on connections as a move, without the reconnect target -- the Remove dialog says so explicitly.

SMB clients and Active Directory: giving a VIF name its own Kerberos identity

When the appliances are joined to Active Directory (see Active Directory Configuration), SMB clients that connect by a VIF's host name rather than by an appliance's own name need one extra piece of setup. A VIF is a floating IP address with a DNS name and no computer account of its own, so Active Directory has no cifs/<vif-name> service principal name (SPN) to issue a Kerberos service ticket against. In a domain where NTLM is disabled, a client connecting to \\<vif-name>\<share> therefore cannot authenticate at all.

The fix is to give the VIF name its own AD computer object with cifs/ SPNs, and put that object's keys in /etc/krb5.keytab on every appliance the VIF can run on -- for a scale-up VIF, every member of the HA failover group. Nothing happens in Active Directory at failover. Every eligible appliance holds the VIF's keys all the time, so whichever one currently hosts the address can answer for its name. An appliance that is missing the keys fails Kerberos authentication by the VIF name whenever the VIF lands on it, so the keys have to be merged everywhere, not just on the current owner.

Two pieces make this work:

  • The domain join prepares Samba. Joining the domain sets kerberos method = dedicated keytab and dedicated keytab file = /etc/krb5.keytab in /etc/samba/smb.conf. With the default system keytab method, smbd keeps only the principals that match the appliance's own names and silently ignores a VIF's entries. Leaving the domain removes both settings and deletes /etc/krb5.keytab.
  • A script creates and distributes the VIF's identity. /opt/osnexus/quantastor/bin/qs_vif_kerberos_setup.sh creates the AD computer object, exports its keytab, and merges it into /etc/krb5.keytab. Nothing in the Web UI or the qs CLI does this; run the script as root on an appliance.

Before you start

  • Every appliance the VIF can run on is joined to the domain.
  • msktutil and krb5-user are installed. They are recommended rather than required packages, so they can be missing; the script stops and names whichever one is absent.
  • You have an AD account that can create computer objects. A machine account cannot do it, because AD refuses an SPN whose host part is not the account's own name.
  • The VIF name is 15 characters or fewer -- the NetBIOS/sAMAccountName limit.
  • Clients can resolve the VIF's FQDN to the VIF's IP address. Create a DNS A record, or an entry in the clients' hosts files. The FQDN field on the cluster VIF (see Cluster VIFs) adds the name to /etc/hosts on every appliance in the grid so that internal services, Kerberos included, can resolve it, but it does not publish anything to your clients' DNS.

Creating the VIF's identity

Run create on any one appliance and list the others with --nodes: 

/opt/osnexus/quantastor/bin/qs_vif_kerberos_setup.sh create --vif-name sharevif \
    --ad-admin Administrator --nodes qs-node-111,qs-node-112

The script prompts for the AD account's password, or reads it from the QS_AD_ADMIN_PASSWORD environment variable. It never accepts the password as an argument, because the command line is visible in the process table. Then it:

  1. Reads the realm from the appliance's Samba configuration, so you do not retype it.
  2. Checks that AD does not already have an account named <vif-name>$, and stops before changing anything if it does (see below).
  3. Creates the computer object with two SPNs, cifs/<vif-name> and cifs/<fqdn>. The FQDN defaults to the VIF name followed by the realm in lower case; set --fqdn if the name clients use is different. Only AES128 and AES256 keys are issued; RC4 is left out because it is deprecated and AD may refuse it.
  4. Merges the keytab into the local /etc/krb5.keytab, keeping a timestamped backup, and asks smbd to reload its configuration rather than restarting it, so live SMB sessions are not dropped.
  5. Verifies the local appliance (see below).
  6. Copies the keytab to each appliance in --nodes over root SSH, merges it there and deletes the copy.

QuantaStor appliances do not have root SSH trust between them by default, so the copy step usually cannot finish on its own. When it cannot, the script keeps the exported keytab and prints the exact scp and merge commands to run for each remaining appliance. Run them, then delete the keytab file -- it holds the VIF account's keys. Without --nodes, only the local appliance is updated and the same instructions are printed.

On an appliance where you copied the keytab by hand: 

/opt/osnexus/quantastor/bin/qs_vif_kerberos_setup.sh merge --keytab /tmp/qs_vif_keytab.XXXXXX

A merge is safe to repeat: if every entry is already present it does nothing.

Verifying every appliance

Run verify on each appliance the VIF can run on: 

/opt/osnexus/quantastor/bin/qs_vif_kerberos_setup.sh verify --vif-name sharevif

It checks three things, and fails if any of them is wrong:

  • The appliance's keytab holds cifs/<vif-name> entries.
  • The keys are the ones Active Directory currently issues tickets with. It asks the KDC for a service ticket for each SPN, authenticating as the appliance's own machine account, and checks that the keytab can decrypt it. This catches keys that are present but out of date, which counting entries does not.
  • Samba is using kerberos method = dedicated keytab. If it is not, the VIF's entries are ignored and SMB by the VIF name fails with NT_STATUS_LOGON_FAILURE. Leave the domain and join it again so the join sets the method, then repeat create --rotate, because leaving deletes the keytab.

If verify reports stale keys straight after a create, the new computer object may not have replicated to every domain controller yet. Wait a minute and run it again.

Rotating, or recovering after a domain leave

Running create for a name AD already has would change that account's keys, and every appliance still holding the old keys would immediately start rejecting tickets for the VIF name. So the script refuses, and changes nothing. If the appliances already hold the keys there is nothing to do; confirm with verify.

You need new keys when an appliance has lost them -- leaving the domain deletes /etc/krb5.keytab outright -- or when you rotate them deliberately. Re-run create with --rotate, and list every other appliance the VIF can run on in --nodes: rotating replaces the keys in AD, so an appliance that does not merge the new keytab fails Kerberos authentication by the VIF name until it does. When a merge brings in newer keys, the script prints a warning that names each changed principal.

After any domain leave and re-join, run verify on that appliance.

What QuantaStor refreshes automatically

The VIF's keys belong to its own AD computer object and do not change unless you rotate them. The appliance's own machine account is different: Active Directory rotates its password, and with the dedicated keytab method Samba does not rewrite the appliance's own keytab entries when that happens, so access by the appliance's own host name could fail. QuantaStor refreshes those entries itself, using the machine account credentials, without an administrator password:

  • once a day,
  • on the first pass after the QuantaStor service starts, so an appliance that was down through a rotation catches up straight away,
  • whenever storage pools are imported, and during an HA pool failover on the appliance taking the pool over, so the node that has just picked up the VIF re-checks its own keytab rather than waiting for the daily cycle.

The refresh adds or updates only the appliance's own principals and leaves the VIF's entries untouched. If it fails, QuantaStor raises an alert saying that SMB access by the appliance's own host name may fail after the next machine password rotation. An appliance joined in SSSD-only mode skips the refresh and logs that it did so.

Scale-up and scale-out differences

The VIF mechanism is the same in both configurations. What differs is what the address follows and where it is allowed to go.

Scale-up (ZFS HA pool) Scale-out (Ceph)
Use case Storage Pool (Scale-up HA) Storage Pool (Scale-out), with config type Object, File or Block
Associated object A storage pool HA failover group A Ceph cluster
Eligible appliances (automatic) The HA group's primary, secondary and tertiary Every Ceph cluster member
Prerequisite on every eligible appliance A port of the same name, online, on every site cluster member The Ceph service the config type names must be running and its storage mounted
Relationship to the data The address follows the pool; both move together The data is already distributed. The address is a service endpoint, and moving it moves no data
What a failover costs Pool import time on the new appliance, plus the address move The address move only

The important asymmetry is the last row. In scale-up, a VIF failover is usually part of a pool failover, and the pool import dominates the time -- the VIF is the small part. In scale-out there is nothing to import, so the numbers in How long a failover takes are close to the whole story.

Scale-up VIFs are ordered and colocated with the pool they belong to, which is what keeps the address and the pool on the same appliance. They also appear with an :ha tag rather than :sv, and they are created from the HA failover group; see Storage Pool HA Failover Interface Create, HA Cluster Setup (JBODs) and HA Cluster Setup (external SAN).

For scale-out, note that a site cluster is a prerequisite, not an optional extra: without one there is no cluster software to run the resource, so a Ceph cluster on its own cannot have a floating address. Set up the site cluster across the Ceph members first. See Scale-out File Setup (ceph), Scale-out Block Setup (ceph) and Scale-out Object Setup (ceph).

Cluster VIFs are also the preferred way to present Ceph iSCSI, because routing all the SCSI reservation traffic for a target through one floating address keeps it going through a single target instance.

If a VIF's address is used to activate replication schedules, arrival of the VIF on an appliance is what activates them there; see Remote-replication (DR).

Removing a cluster VIF

Navigation: High-availability VIF Management → Site Cluster Virtual Interfaces → Remove Cluster VIF (toolbar)

Removing a cluster VIF deletes the pacemaker resource and takes the address down. The dialog warns that active connections on the interface will be dropped. It also offers Convert cluster VIF resource to local virtual IP, which is meant to keep the address in service as an ordinary virtual interface on the appliance it was last running on. It currently does not: the address is dropped, as with a plain removal, so do not rely on it to keep an address in service. 

qs site-vif-delete --vif-resource=<vif-id>
qs site-vif-delete --vif-resource=<vif-id> --convert-to-vif=true

Remove the VIFs before deleting the site cluster that owns them. qs site-cluster-delete refuses while VIFs still reference the site cluster, and says so, precisely to stop a site cluster teardown taking a service address off the network unannounced.

If the VIF had its own Active Directory identity, its computer object stays in AD after the VIF is removed; QuantaStor does not delete it.

Troubleshooting

The Web UI and the cluster disagree about a VIF. If the VIF list shows a state that crm_mon -1 contradicts, run a site cluster rescan. It re-reads the live cluster configuration and rebuilds QuantaStor's view of the rings, the VIFs and their location constraints from it, treating pacemaker as authoritative. It does not stop or restart anything.

The command, qs site-cluster-rescan, takes no arguments and acts on the site cluster the appliance you run it on belongs to, so run it on a member: 

qs site-cluster-rescan

A VIF reports MISSING. QuantaStor has a VIF object but pacemaker has no resource for it. A rescan will reconcile the record.

A VIF will not start anywhere. Read the failed actions in crm_mon -1 first -- one of the checks in What blocks a failover will normally name the cause, and each names its own remedy. The most common is the duplicate-address check firing because an appliance that lost cluster communication is still holding the address.

A VIF is stuck and reports unmanaged. The site cluster is in maintenance mode. Exit maintenance mode and the VIF is managed again.

A VIF will not move to a particular appliance. Check its location constraint on that appliance. None pins it away permanently, and the error message says so.

SMB by the VIF name works on one appliance but fails after a failover. The appliance now hosting the VIF does not hold the VIF's current keys. Run qs_vif_kerberos_setup.sh verify there; see SMB clients and Active Directory.

Command line reference

Command Purpose
qs site-vif-create Create a cluster VIF, optionally converting an existing local one
qs site-vif-list / qs site-vif-get List cluster VIFs, or show one in detail
qs site-vif-modify Change the description or the location constraints
qs site-vif-move Move a VIF to another member deliberately
qs site-vif-delete Remove a VIF, optionally converting it to a local address
qs site-vif-location-constraint-list List the weights for every VIF
qs site-vif-location-constraint-get Show one member's weight for one VIF
qs site-cluster-set-standby-mode Put one member into or out of standby
qs site-cluster-toggle-maintenance-mode Freeze or unfreeze the whole site cluster
qs site-cluster-restart-services Restart corosync and pacemaker on one appliance
qs site-cluster-rescan Rebuild QuantaStor's view from the live cluster configuration

Site cluster and heartbeat ring commands are covered on Site Cluster Setup.

Related pages

Verified against QuantaStor 6.9.0.

Retrieved from "https://wiki.osnexus.com/index.php?title=High-availability_VIF_Management&oldid=28227"