Grid Configuration
A Storage Grid joins several QuantaStor systems so you manage them as one. Log in to any member and you administer the whole grid from that session -- create a pool on another system, replicate between members, move a policy -- without connecting to each appliance separately. Grid membership is also the prerequisite for features that span systems, including remote replication and the clustered pool configurations.
A system belongs to at most one grid at a time. A grid can hold any number of systems, and within it any number of storage clusters, each with their own pools.
| Section | Covers |
|---|---|
| The Grid Master | The coordinating member, what it does, and what happens if it is lost |
| Creating a Grid | Turning a standalone system into a one-member grid |
| Adding a System | Joining a system, and what merges when it joins |
| Modify Grid Network Settings | Writing one DNS and NTP configuration to several members at once |
| Modify Grid Time Zone Settings | Setting the time zone across the grid, and the field to check first |
| Renaming a Grid | The grid's own name and description |
| Changing the Grid Master | Electing a different coordinator |
| Removing a System | Taking a member out, including Ceph members |
| Deleting a Grid | Dissolving the grid, and what survives |
| Managing a Grid from the CLI | The twelve qs grid commands
|
Six grid operations live in the Storage System Grid toolbar group: Create Grid, Modify Grid, Add System, Delete Grid, Set Grid Master and Remove System. Two more are grid-scope but are not on that toolbar -- Modify Grid Network Settings and Modify Grid Time Zone Settings are reached by right-clicking the Storage Grid node at the top of the Storage Systems tree. The tree header shows the grid name in brackets, and the Storage Systems list marks the coordinating member (grid master).
The Grid Master
One member is the grid master: it coordinates configuration state across the grid, acting as the conduit through which members learn about each other's objects. The overhead is small -- this is a coordination role, not a data path, and storage I/O never flows through it.
The same role appears under four different names, which is worth knowing when reading dialogs and command output:
| Where | What it is called |
|---|---|
| Storage Systems list | (grid master) beside the member name |
| Toolbar button | Set Grid Master |
| Set Grid Master dialog | "primary coordinator system" |
qs grid-get output |
Master Node ID
|
You do not need to connect to the grid master to manage the grid -- any member gives you the whole grid. Management operations may feel more responsive when run against the master, because operations that need coordination do not have to be relayed.
If the grid master is lost, storage keeps serving but grid management degrades unless you have a Grid VIF configured which will automatically elect another node as the new grid master. Pools stay online and clients keep their connections, because the master is not in the data path. Management operations that require coordination will fail until a new master is elected -- see Changing the Grid Master.
Creating a Grid
Run this on the system that will be the first member. It becomes a one-member grid and that system becomes the initial grid master. Supply a Name for the grid, and optionally a description.
Names may contain alphanumeric characters and the characters -, _ and .
Creating the grid changes nothing about the system's storage. It is a management-plane operation: pools, volumes and shares are untouched.
Adding a System
- Storage System IP Address -- the system to join. Reach it from the system you are running this on.
- Admin Username / Admin Password -- credentials on the system being added, not on the grid you are adding it to. QuantaStor authenticates against the remote system to establish the join, so these are its own local administrator credentials.
Run this from a member of the grid you want to grow.
The system being added must not already belong to another grid. If it does, the join is refused with "This node is already a member of grid '<name>', cannot join grid '<name>'".
Forcing a join of an already-gridded system merges the two grids. The force flag overrides that guard, and the operation then becomes a merge -- every system in the other grid is brought in, not just the one you named. That is a much larger change than adding one appliance, and it is the same operation as qs grid-merge. If a merge is what you want, use grid-merge, where the intent is explicit.
What merges when a system joins
Joining merges management objects across the grid: users, user groups and roles all become grid-wide, so an account created on one member is usable on all of them.
The merge keeps the superset -- nothing is deleted. Users, user groups and roles are merged such that objects are added or updated but never removed, and matching is by object identity rather than by name. Two accounts that happen to share a name, created independently on two systems, therefore both survive as separate accounts. There is no name-based precedence and nothing is discarded.
The one exception is the admin account. Duplicate admin accounts are reconciled by name: the grid master's copy is kept and the others are removed. The practical consequence is the one to plan for -- after joining, the grid master's admin password becomes the admin password on the newly added system. If you are joining a system whose admin password differs, expect to use the master's password afterwards.
(Note: this section describes the merge behaviour of the shipping release. If you rely on account layout across a join, verify the result on a test grid before doing it in production.)
Modify Grid Network Settings
Modify Grid Network Settings writes one DNS and NTP configuration to several grid members in a single operation, so a grid comes up with the same resolvers, the same search domain and the same time sources on every system. It is a grid-scope operation and it is not on the Storage System Grid toolbar: right-click the Storage Grid node, which is the top of the Storage Systems tree and carries the grid's name. It is also Step 3 -- Update DNS & NTP on the Storage Grid Setup card of Getting Started, which is where it fits in a new deployment -- after the members have joined and before anything that depends on name resolution or accurate time.
The dialog says what it is about to do, in bold, above the fields: "The selected storage systems settings will be overwritten with the configuration below." Take it literally -- see What it replaces.
Which systems it applies to
The Select Storage System(s) list carries one tick box per system and shows, beside each name, that system's current DNS Servers, NTP Servers and Search Suffix. Read those columns first: they are what you are about to replace.
- Nothing is ticked when the dialog opens. Tick the systems you want to change. Clicking OK with none ticked is refused with "One or more storage systems must be selected."
- Only systems in the Normal state are listed. A member that is offline, disconnected or in any other state does not appear in the list, cannot be ticked, and is left exactly as it is -- no operation is queued for it and nothing reports that it was skipped. Bring it back to Normal and run the dialog again for that system.
- An unticked system is not touched. QuantaStor issues one modify operation per ticked system and none at all for the others, so ticking a subset is a supported way to standardise some members while leaving others on their own settings.
Fields
The fields are pre-filled from the first system in the list, which is sorted by name -- not from the grid master, and not from the system you are logged in to. On a grid whose members already differ, the values you see are one member's, so check them before you click OK.
DNS Servers
- IP Address with Add and Remove -- the DNS servers to set. IPv4 addresses only; a host name or an IPv6 address is rejected. At most three servers can be set on a system that currently has three or fewer; a system that already carries more than three keeps that allowance.
- At least one DNS server is required. Clicking OK with an empty list is refused with "It is required to have at least 1 DNS server configured."
- Search Suffix -- one or more DNS search domains, separated by spaces. This is what a short name is resolved against. There is no separate domain suffix field -- the search domain replaces it, and the value a system reports as its domain suffix is the first search domain.
NTP Servers
- NTP Servers with Add and Remove -- the time servers to set, given as host names or addresses.
- The up and down arrows set the order, which is the order the servers are written into the time service configuration.
- An empty NTP list is ignored rather than applied. Removing every entry and clicking OK leaves each ticked system's existing time configuration alone. The dialog cannot be used to clear NTP.
What it replaces
For each ticked system the values in the dialog replace that system's own values. Nothing is merged, and nothing the system had is preserved:
- DNS servers -- the system's list becomes exactly the list in the dialog. A server the system had that is not in the list is removed.
- Search Suffix -- replaced with the contents of the field.
- NTP servers -- the server entries in the time service configuration are replaced with the list in the dialog, in the order shown. Lines in that file that are not server entries are left alone.
Nothing else about the system changes. The dialog re-sends every other Storage System Modify setting -- name, description, firewall, proxy, bonding, ARP filtering and the rest -- unchanged, per system.
The practical consequence is the one to plan for: if DNS, the search domain or NTP has been set per system, ticking that system here discards what it had. Tick only the systems you intend to standardise.
A setting that is already at the value you are applying is skipped, so re-applying a system's existing values does nothing to it.
Where the settings land, and when they take effect
You do not have to restart anything. For each ticked system:
- The DNS servers and search domains are written to
/etc/systemd/resolved.conf.d/quantastor-static.confand the local resolver is restarted. - The NTP servers are written to the time service configuration (
/etc/chrony/chrony.conf), the time service is restarted, and the clock is stepped to the new source immediately rather than being allowed to drift into line.
Each ticked system gets its own Modify System Properties task, so the task list is where per-system progress and per-system failures show up. The dialog closes as soon as you click OK and does not report the results itself, so check the task list rather than assuming a clean run -- particularly on a grid where a member has been unreliable.
The per-system equivalents
DNS and NTP are also configurable one system at a time, on the DNS Servers and NTP Servers tabs of Storage System Modify. Storage System owns those two settings, and covers why working name resolution and accurate time are prerequisites rather than niceties. Use the per-system tabs when one member needs to differ; use this dialog when the whole grid should agree.
These are system settings, not port settings. Network Ports owns per-port configuration -- addresses, bonds, VLAN ports and static routes -- and does not carry DNS or NTP.
From the CLI
There is no grid-wide command; run qs system-modify once per system, which is the same operation the dialog issues for each ticked system.
qs system-modify --storage-system=qs-node-110 --dns-servers=10.0.0.1 --search=example.com \
--ntp-servers=ntp.ubuntu.com,0.ubuntu.pool.ntp.org
Arguments you leave out keep the system's current value, so --dns-servers on its own changes only the DNS servers. The replace semantics are the same as the dialog's: the list you pass becomes the whole list.
Modify Grid Time Zone Settings
Modify Grid Time Zone Settings sets the system time zone on several grid members at once. Like Modify Grid Network Settings it is a grid-scope operation reached by right-clicking the Storage Grid node rather than from the toolbar, and it is Step 4 -- Update Time Zone on the Storage Grid Setup card of Getting Started.
This is the only place in the web interface where the time zone is set. There is no time zone field on Storage System Modify, so setting the time zone on a single system means opening this dialog and ticking only that system.
Which systems it applies to
The Select Storage System(s) list carries one tick box per system and shows each system's current Time Zone beside its name.
- Every listed system is ticked when the dialog opens -- the opposite of Modify Grid Network Settings, which starts with nothing ticked. Untick the systems you do not want to change before you click OK.
- Only systems in the Normal state are listed, and a system that does not appear is left as it is.
- An unticked system is not touched. One operation is queued per ticked system and none for the others.
Time Zone Settings
- Time Zone -- the zone to apply. The list is read from the system time zone table,
/usr/share/zoneinfo/zone.tab, which holds a little over four hundred zones named Region/City. The field accepts typing as well as selection, so a zone can be found by typing part of its name. - Country Code -- read-only. The two-letter country code the time zone table records for the selected zone.
- Comment -- read-only. The note the time zone table carries for the selected zone, which is how zones within one country are told apart ("New Zealand time - McMurdo, South Pole", for instance). Most zones have no comment.
Confirm the Time Zone field before you click OK
The Time Zone field does not always open on the systems' current zone. It can only preselect a zone that exists in the system time zone table, and it falls back to the first entry in that table when the current zone is not there. A QuantaStor system ships set to Etc/UTC, which the table does not list, so on a default grid the field opens on Europe/Andorra while every system is still ticked -- and clicking OK in that state sets the whole grid to Europe/Andorra.
So read the field, not the list, before committing: the Time Zone column shows what each system has now, and the Time Zone field shows what all the ticked systems are about to get. For the same reason UTC cannot be selected here, nor with qs timezone-set -- the time zone table carries no UTC entry.
What it changes
For each ticked system the operation sets the system time zone and nothing else. The clock is not moved: the same instant is simply reported in a different zone. Each ticked system gets its own Update System Time Zone task, and as with Modify Grid Network Settings the dialog closes on OK without reporting per-system results, so confirm the outcome in the task list.
Setting it correctly per site matters more than the log timestamps suggest. Schedules are evaluated in each system's own local time zone, so a snapshot or replication schedule set to run at 02:00 runs at 02:00 local time on the system that owns it. Two members in different zones running the same schedule therefore fire at different absolute times -- deliberate across a geographically distributed grid, and a surprise when the time zones differ by accident.
From the CLI
qs timezone-list prints the same table the dialog offers, with the country code and comment for each zone, and qs timezone-set applies one to a system. A zone that is not in the table is rejected.
qs timezone-list qs timezone-set --storage-system=qs-node-110 --timezone=America/Los_Angeles
Without --storage-system the zone is set on the system you are connected to. As with the network settings there is no grid-wide command -- run it once per system.
Renaming a Grid
Modify Grid changes the grid's Name and Description and nothing else. The name is what appears in the tree header, so it is worth setting to something that identifies the site or purpose rather than leaving a generated value.
The grid name is separate from the member system names. Renaming the grid does not rename any system, and renaming a system does not change the grid name -- see Storage System for renaming a member.
Changing the Grid Master
- Storage System -- the member to elect as the primary coordinator for synchronizing grid configuration state.
- Force -- promotes the selected member even when the normal election cannot complete.
Use this to move the coordinator role deliberately -- before taking the current master down for maintenance, for instance -- and to recover grid management after a master is lost.
When Force is needed. A normal election coordinates with the existing master and, in a grid with a grid-management virtual interface, moves that interface to the new master. If that step fails -- which is what happens when the member cannot reach the current master -- the operation is refused and nothing changes. Selecting Force makes QuantaStor promote the member directly instead, without that coordination.
So the ordinary reason to reach for Force is the one that matters most: the current master is unreachable and you need to promote a surviving member to get grid management back. It is not a retry switch for an election that failed for some other reason.
Because a forced promotion skips coordination with the existing master, only use it when you are confident the old master is genuinely down. Forcing a promotion while the old master is still running and merely unreachable from this member leaves two systems believing they coordinate the grid and will take a minute or two for the nodes to sort.
Removing a System
- Storage System -- the member to remove.
- Force -- required when the selected system is a Ceph cluster member. Without it the operation is refused with "The selected System is a Ceph cluster member and the Force option must be selected to remove." The guard exists because removing a Ceph member from the grid separates it from the cluster it participates in; requiring Force makes that deliberate rather than accidental.
A removed system becomes standalone again and keeps its own pools, volumes and shares. It also keeps the grid-derived admin password if its admin account was reconciled during the join -- removal does not restore the password it had beforehand.
Before removing a member, move or retire anything that depends on the grid: replication relationships to other members, and cluster membership if the system participates in a clustered pool or a Ceph cluster.
Deleting a Grid
Deleting the grid dissolves the management relationship and returns every member to standalone operation.
Storage is not affected. Pools, volumes and shares stay where they are and stay online -- this removes grid membership, not data. What you lose is single-pane management and anything that depended on membership, so replication relationships between former members and any cross-system cluster configuration must be dealt with first.
Managing a Grid from the CLI
Twelve commands cover the grid. The full argument list for each is in the QuantaStor CLI Command Reference.
| Command | Purpose |
|---|---|
qs grid-create |
Create a grid on this system |
qs grid-get |
Show the grid, its master, and its members |
qs grid-modify |
Change the grid name or description |
qs grid-add |
Add a system by IP address |
qs grid-remove |
Remove a member |
qs grid-set-master |
Elect a different grid master |
qs grid-delete |
Dissolve the grid |
qs grid-merge |
Merge another grid into this one |
qs grid-split |
Split members out into a separate grid |
qs grid-assoc-list |
List grid associations |
qs grid-assoc-get |
Show one system's grid association |
qs grid-send-supportlogs |
Collect and send support logs for the whole grid |
Building a three-member grid from the first system:
qs grid-create --name=qs-grid qs grid-add --node-ipaddress=10.0.8.111 --node-username=admin --node-password=aAbBcCdD qs grid-add --node-ipaddress=10.0.8.112 --node-username=admin --node-password=aAbBcCdD
Confirming the result -- grid-get reports the grid and its Master Node ID, and system-list shows the members:
qs grid-get qs system-list
grid-merge and grid-split have no toolbar equivalent and are the way to reorganise existing grids -- merging two grids into one, or splitting members out into their own. Both change membership for several systems at once, so plan them against a test grid first.
Troubleshooting
A member shows offline. Check that grid communication reaches it: the members must be able to talk to each other on the grid port, and a firewall rule or a changed IP address on the management interface will break that. See Network Ports for the port configuration and the Use Preferred Grid Port IP setting on Storage System Modify, which pins grid traffic to a chosen interface on systems with several networks.
A member shows a warning that will not clear. Some changes park a system in a warning state until it is restarted. If a member reports that a restart is required, the state persists until the reboot happens -- it is not a transient condition that clears itself.
Management operations fail but storage is fine. This is the signature of a lost or unreachable grid master, since the master coordinates configuration but carries no data. Elect a new master with Set Grid Master.
Related pages
- Storage System -- individual member configuration, renaming, the per-system DNS and NTP tabs, and the preferred grid port setting
- Configure Member Standby -- taking a member out of service without removing it from the grid
- License Management -- licensing across grid members
- Network Ports -- the network configuration grid communication depends on
- Remote-replication (DR) -- replication between grid members
- HA Cluster Setup (JBODs) -- clustered pools within a grid
- Security Configuration -- users and roles, which become grid-wide on join
Verified against QuantaStor 6.9.0.