Template:TroubleshootingHAStoragePools - Revision history

Qadmin: Deprecate: content rewritten and verified as Scale-up HA Storage Pool Troubleshooting; template is now orphaned (QSTOR-12352)

2026-09-04T05:29:21Z

Deprecate: content rewritten and verified as Scale-up HA Storage Pool Troubleshooting; template is now orphaned (QSTOR-12352)

← Older revision		Revision as of 05:29, 4 September 2026
Line 1:		Line 1:
			<noinclude>
			''This template is deprecated and is no longer transcluded by any page. Its content has been rewritten, verified against QuantaStor 6.9.0 and corrected as '''[[Scale-up HA Storage Pool Troubleshooting]]''', which is the page to edit and the page to link. The text below is retained only so the original wording remains findable; do not transclude it and do not update it. (QSTOR-12352)''
			</noinclude>
	== Recovery and Resolution of Failure Scenarios ==		== Recovery and Resolution of Failure Scenarios ==

Qadmin: /* Recovery from Device Failures */

2024-10-02T04:32:02Z

Recovery from Device Failures

← Older revision		Revision as of 04:32, 2 October 2024
Line 3:		Line 3:
	=== Recovery from Device Failures ===		=== Recovery from Device Failures ===

	HDDs (and many SSDs) have a typical annual failure rate (AFR) of 1.5% so replacing bad media is just a common eventuality in maintaining a healthy Storage System. For both scale-up and scale-out cluster configurations we always require parity based pools to have at least two coding blocks. For scale-up configurations this means RAIDZ2 or RAIDZ3 and for scale-out using erasure-coding that means a k+m where m is >= 2. Single parity layouts ~~are~~ ok for test environments but not for production workloads as they are not durable enough. With 2, 3 or more parity bits per stripe a single device failure will leave the effected pool running in a degraded state with additional parity information to still do error correction to deal with bad sectors or a second concurrent device failure.		HDDs (and many SSDs) have a typical annual failure rate (AFR) of 1.5% so replacing bad media is just a common eventuality in maintaining a healthy Storage System. For both scale-up and scale-out cluster configurations we always require parity based pools to have at least two parity devices (coding blocks) per stripe (VDEV or placement-group). For scale-up configurations this means RAIDZ2 or RAIDZ3 and for scale-out using erasure-coding that means a K+M (data blocks + coding blocks) where m is greater than or equal to two (>=2). Single parity layouts may be ok for some test environments but not for production workloads as they are not durable enough. With 2, 3 or more parity bits per stripe a single device failure will leave the effected pool running in a degraded state with additional parity information to still do error correction to deal with bad sectors or a second concurrent device failure which is critically important.

	In a scale-up configuration simply add a new HDD or SSD that is equal or greater in capacity to the failed device to the system i an available free slot, then mark the device as a hot-spare. If there are no free slots available then first remove the failed device and ~~replace it with~~ the new ~~good~~ device. After that mark the new device as a hot-spare. Once marked as a hot-spare the new device will be automatically utilized to repair the degraded pool. There are two ways to mark a device as a hot-spare, in the Physical Disks section as a 'Universal Hot Spare' that can be used to repair any pool ~~and~~ in the Storage Pools section to assign the new device as a hot-spare for a specific pool. ~~If you have~~ more than one pool ~~and some hot-~~spares ~~in the system then mark those~~ as Universal Hot-Spares so that they're able to used to repair any pool in the HA cluster. If you are explicitly replacing a bad drive for a pool then simply right click on the degraded pool and add the new device as a hot-spare using the 'Recover / Add Hot-spare' dialog.		To recover from a device failure in a scale-up configuration simply add a new HDD or SSD that is equal to or greater in capacity than the failed device to the system with the device failure. Use an available free slot if one is available, then mark the device as a hot-spare. If there are no free slots available then first remove the failed device and then place the new replacement device into the same slot where the bad device was located. After that mark the new device as a hot-spare. Once marked as a hot-spare the new device will be automatically be utilized to repair the degraded pool.

			There are two ways to mark a device as a hot-spare, in the Physical Disks section as a 'Universal Hot Spare' that can be used to make a spare as 'universal' meaning it may be used to repair any pool. The second option is in the Storage Pools section where one may assign the new spare device as a hot-spare for a specific pool, simply right-click on the degraded pool then use the 'Recover Pool / Add Hot-Spare..' dialog.

			After setting up a new system where more than one pool has been created we recommend marking all spares as Universal Hot-Spares so that they're able to used by QuatnaStor to repair any pool in the HA cluster, this is done via the Physical Disks section. If you are explicitly replacing a bad drive for a pool then a direct assignment is preferred, simply right-click on the degraded pool and add the new device as a hot-spare using the 'Recover Pool / Add Hot-spare' dialog.

	It is important to remove bad media as soon as possible as some media can behave poorly and cause bus resets and other things that can impact normal operations of the JBOD. Some brands of JBODs are more/less impacted by failed media but our general recommendation is to remove bad media once a pool has been repaired.		It is important to remove bad media as soon as possible as some media can behave poorly and cause bus resets and other things that can impact normal operations of the JBOD. Some brands of JBODs are more/less impacted by failed media but our general recommendation is to remove bad media once a pool has been repaired.

	If there are multiple bad devices the repairs will be done one at a time. QuantaStor has logic to intelligently select hot-spares by media type, jbod location, and capacity. As such, marking a SSD as a hot-spare will only use the device to repair a failed SSD and similarly a HDD will only be used to repair a pool with a failed HDD. It will also ensure that the HDD is large enough and will have preferential selection to using a hot-spare that is in the same chassis as the failed device so that any enclosure level redundancy is maintained.		If there are multiple bad devices the repairs will be done one at a time. QuantaStor has logic to intelligently select hot-spares by media type, jbod location, and capacity. As such, marking a SSD as a hot-spare will only use the device to repair a failed SSD and similarly a HDD will only be used to repair a pool with a failed HDD. It will also ensure that the HDD is large enough and will have preferential selection to using a hot-spare that is in the same chassis as the failed device so that any enclosure level redundancy is maintained. In the event that multiple VDEVs are degraded the hot-spare will automatically be used to repair the VDEV with the most degradation.

	=== Recovery from Pool I/O Failure ===		=== Recovery from Pool I/O Failure ===

Qadmin: /* Recognize and recover from HA split-brain events */

2024-10-02T04:24:30Z

Recognize and recover from HA split-brain events

← Older revision		Revision as of 04:24, 2 October 2024
Line 91:		Line 91:
	=== Recognize and recover from HA split-brain events ===		=== Recognize and recover from HA split-brain events ===

	It is ~~near impossible~~ to get QuantaStor to split-brain. A split-brain is where a mirrored pool can be split in two halves ~~and~~ both head nodes are running the half pool ~~as effectively a RAID0 pool at the same time~~. At this ~~point both pools are valid~~ pools ~~but could have different clients and files being written to them. Recovery from a scenario like this involves media reformatting of~~ a pool ~~deemed to be~~ the ~~'bad' pool and re-silvering the media back to the 'good' pool. Pools using RAIDZ2 (eg 4d+2p) or RAIDZ3 (8d+3p) will not start~~ with ~~50% of~~ the ~~media so they cannot get into traditional split-brain scenarios at all~~.		To start, is very very difficult to get QuantaStor into a split-brain configuration but in this section we'll outline how one may attempt to do so. A split-brain is where a mirrored pool can be split in two halves such that both head nodes are successfully running the half pool devices. In this mode one has split a given storage pool into two pools, typically a RAID10 pool that is now running as two separate RAID0 pools at the same time with the same name on two separate nodes.

	* QuantaStor prevents split brain by IO ~~fencing (SCSI3~~-~~PR) all~~ media ~~devices in a given storage~~ pool (~~dual-port SAS and dual-port NVMe~~)		In a split-brain scenario these two separate head nodes are running two separate pools that are now diverged from each other such that merging them back together can be a time consuming and IO intensive process. Typical recovery from a scenario like this where there have been changes made to both pool involves reformatting one of the two pool's devices, the pool deemed to be the 'bad' pool, and then re-silvering the media back to the 'good' pool. Pools using RAIDZ2 (eg 4d+2p) or RAIDZ3 (8d+3p) cannot not start with 50% of the media so they cannot get into traditional split-brain scenarios at all. These are some of the ways that QuantaStor prevents split brain from ever happening:
	* QuantaStor prevents split brain ~~by only allowing a pool to run on~~ the ~~system which has the HA VIFs~~
	* QuantaStor prevents split brain ~~by only starting pools where enough devices are available for it to start in a degraded state~~

	~~Given~~ the above the only way to cause a split-brain ~~would be~~ to use RAID10 and then carefully manipulate JBOD access ~~and a disabling of~~ the ~~HA group so that the pool is able~~ to ~~start on two~~ separate ~~servers as RAID0~~. For RAIDZ2/RAIDZ3 ~~pools this would~~ not ~~work~~ as a ~~majority~~ of the ~~media is needed in order for~~ the ~~pool~~ to ~~start and so~~ the ~~pool can never be started on~~ two ~~servers at once due~~ to IO fencing. ~~There are still ways to create problems here though but~~ it ~~would require running~~ the pool ~~across multiple~~ JBODs and then running the pool in a ~~degraded state~~ with ~~different groups of JBODs disabled at different times before~~ the pool ~~could return~~ to a ~~healthy state~~. ~~Not an easy scenario~~ to ~~recreate~~ and the ~~fix would~~ be to ~~start~~ the pool with ~~the last known set of devices that were successfully imported~~. For example, ~~if you have~~ a pool that spans JBODs A, B, C such that the pool can run in a degraded state with any two JBODs online and JBOD-C is powered off then the JBOD-C is powered ON and JBOD-B is powered off then the pool will stop. This is because JBODs A+B have the most recent pool information and they need to be powered on with JBOD-C so that the pool can rebuild/resilver. The filesystem (OpenZFS) ~~will detect~~ scenarios ~~that~~ are ~~invalid like A+B then A+C with no repair~~ in~~-between~~ and will stop the pool. The fix is to start the pool again with all three JBODs A+B+C and let C resilver to get the pool back to full health. Then one could disable JBOD-B and the pool would run degraded with A+C. Going direct from a degraded A+B to A+C ~~would~~ not work ~~and~~ again ~~would~~ result in the pool stopping and not ~~importing even after a reboot~~.		* QuantaStor prevents split brain by IO fencing (SCSI3-PR) all media devices in a given storage pool (dual-port SAS and dual-port NVMe), this ensures that only one node at a time can start the pool with the media. To bypass this one would need to use RAID10 then through a series of cabling changes isolate each JBOD to a separate QuantaStor head node. This would require significant intentional physical reconfiguration of the hardware.
			* QuantaStor prevents split brain by only allowing a pool to run on the system which has all the the HA virtual interfaces (VIFs) which move as a cluster resource group. Only the node with the VIFs are allowed to start the pool but in the re-cabling scenario described above with a RAID10 pool it is possible to failover a RAID10 pool from one node to another where on node-1 JBOD1 is being used and node-2 JBOD2 is being used.
			* QuantaStor prevents split brain by only starting pools where enough devices are available for it to start in a degraded state. Again, this still leaves a RAID10 pool somewhat vulnerable if the devices were perfectly split between two JBODs.

			In summary, the only way to cause a split-brain is to use a RAID10 pool layout and then carefully manipulate JBOD access through recabling to isolate the JBODs to separate nodes. For pools using parity based RAID such as RAIDZ1/RAIDZ2/RAIDZ3 it is not possible to have two different versions of the same pool running as any given pool cannot start with 50% of the devices. There are a couple of exceptions to this which are pools created using 2d+2p RAIDZ2 and 3d+3p RAIDZ3 layouts. Now that we understand the narrow scenarios which can lead to a split brain what are the ways to avoid the possibility of it ever happening.

			* When using RAID10 with enclosure redundancy across two JBODs make sure you have two paths to both controller head nodes to both JBODs, this ensures the IO fencing will reach all devices at all times
			* When using RAID10 you can opt to use a single JBOD or more than two JBODs.
			* If the above cannot be ensured then one could avoid using RAID10, 2d+2p and 3d+3p storage pool layouts. In most cases we recommend RAIDZ2 4d+2p for virtualization and databases rather than RAID10 as it provides more capacity, better durability and eliminates even the rare potential for split-brain.

			=== Recovery from multiple JBOD failures ===

			If too many JBODs fail (powered off, etc) then the pool will stop. The best way to correct that is to power the JBODs on and then reboot the node that was running the pool before the JBOD failure. If multiple pools were effected then it is best to just cold boot the whole cluster. In a pool with an enclosure redundant layout such as a RAIDZ2 4d+2p pool layout spanning three enclosures JBOD-A, JBOD-B and JBOD-C then the pool will continue to run if any one JBOD fails. After a JBOD failure the JBOD should be powered back on and then one should manually failover the pool back to the node it is already on (for example if the pool is on node-1, then failover to node-1 where it already lives) to force a revalidation of all connectivity. QuantaStor has logic to automatically re-fence the media after a JBOD restart to ensure it is locked to the node where the pool is running. You can verify the IO fencing by checking in the Physical Disks section and look for a black underline device icon across all the media that makes up a pool. If you see a red-underline that indicates a IO fencing issue which can be resolved through a failover to the node the pool is on or to the paired node.

			Another scenario to consider with multiple JBODs is what if one has multiple JBOD failures but at different times. For example, with a pool that spans JBODs A, B, and C such that the pool can run in a degraded state with any two JBODs online and JBOD-C is powered off then the JBOD-C is powered ON and JBOD-B is powered off before the pool has resilvered (repaired) then the pool will stop. This is because JBODs A+B have the most recent pool information and they need to be powered on with JBOD-C so that the pool devices on JBOD-C can rebuild/resilver before the devices in JBOD-C may provide additional durability to the pool. The filesystem (we use OpenZFS with scale-up HA clusters) detects these scenarios where the media are at different points in time transactionally and will prevent the pool from starting and/or will stop the pool in such a scenario. The fix is to start the pool again with all three JBODs A+B+C and let C resilver to get the pool back to full health. Then one could disable JBOD-B and the pool would run degraded with A+C. Going direct from a degraded A+B to degraded A+C will not work, again this will result in the pool automatically stopping and/or not starting.

	=== Resolving Disks Not Visible on both HA nodes / Storage Systems ===		=== Resolving Disks Not Visible on both HA nodes / Storage Systems ===

Qadmin: /* Recognize and recover from HA split-brain events */

2024-10-02T03:52:15Z

Recognize and recover from HA split-brain events

← Older revision		Revision as of 03:52, 2 October 2024
Line 98:		Line 98:

	Given the above the only way to cause a split-brain would be to use RAID10 and then carefully manipulate JBOD access and a disabling of the HA group so that the pool is able to start on two separate servers as RAID0. For RAIDZ2/RAIDZ3 pools this would not work as a majority of the media is needed in order for the pool to start and so the pool can never be started on two servers at once due to IO fencing. There are still ways to create problems here though but it would require running the pool across multiple JBODs and then running the pool in a degraded state with different groups of JBODs disabled at different times before the pool could return to a healthy state. Not an easy scenario to recreate and the fix would be to start the pool with the last known set of devices that were successfully imported. For example, if you have a pool that spans JBODs A, B, C such that the pool can run in a degraded state with any two JBODs online and JBOD-C is powered off then the JBOD-C is powered ON and JBOD-B is powered off then the pool will stop. This is because JBODs A+B have the most recent pool information and they need to be powered on with JBOD-C so that the pool can rebuild/resilver. The filesystem (OpenZFS) will detect scenarios that are invalid like A+B then A+C with no repair in-between and will stop the pool. The fix is to start the pool again with all three JBODs A+B+C and let C resilver to get the pool back to full health. Then one could disable JBOD-B and the pool would run degraded with A+C. Going direct from a degraded A+B to A+C would not work and again would result in the pool stopping and not importing even after a reboot.		Given the above the only way to cause a split-brain would be to use RAID10 and then carefully manipulate JBOD access and a disabling of the HA group so that the pool is able to start on two separate servers as RAID0. For RAIDZ2/RAIDZ3 pools this would not work as a majority of the media is needed in order for the pool to start and so the pool can never be started on two servers at once due to IO fencing. There are still ways to create problems here though but it would require running the pool across multiple JBODs and then running the pool in a degraded state with different groups of JBODs disabled at different times before the pool could return to a healthy state. Not an easy scenario to recreate and the fix would be to start the pool with the last known set of devices that were successfully imported. For example, if you have a pool that spans JBODs A, B, C such that the pool can run in a degraded state with any two JBODs online and JBOD-C is powered off then the JBOD-C is powered ON and JBOD-B is powered off then the pool will stop. This is because JBODs A+B have the most recent pool information and they need to be powered on with JBOD-C so that the pool can rebuild/resilver. The filesystem (OpenZFS) will detect scenarios that are invalid like A+B then A+C with no repair in-between and will stop the pool. The fix is to start the pool again with all three JBODs A+B+C and let C resilver to get the pool back to full health. Then one could disable JBOD-B and the pool would run degraded with A+C. Going direct from a degraded A+B to A+C would not work and again would result in the pool stopping and not importing even after a reboot.


	=== Resolving Disks Not Visible on both HA nodes / Storage Systems ===		=== Resolving Disks Not Visible on both HA nodes / Storage Systems ===

Qadmin: /* Troubleshooting for HA Storage Pools */

2024-10-02T03:51:59Z

Troubleshooting for HA Storage Pools

← Older revision		Revision as of 03:51, 2 October 2024
Line 99:		Line 99:
	Given the above the only way to cause a split-brain would be to use RAID10 and then carefully manipulate JBOD access and a disabling of the HA group so that the pool is able to start on two separate servers as RAID0. For RAIDZ2/RAIDZ3 pools this would not work as a majority of the media is needed in order for the pool to start and so the pool can never be started on two servers at once due to IO fencing. There are still ways to create problems here though but it would require running the pool across multiple JBODs and then running the pool in a degraded state with different groups of JBODs disabled at different times before the pool could return to a healthy state. Not an easy scenario to recreate and the fix would be to start the pool with the last known set of devices that were successfully imported. For example, if you have a pool that spans JBODs A, B, C such that the pool can run in a degraded state with any two JBODs online and JBOD-C is powered off then the JBOD-C is powered ON and JBOD-B is powered off then the pool will stop. This is because JBODs A+B have the most recent pool information and they need to be powered on with JBOD-C so that the pool can rebuild/resilver. The filesystem (OpenZFS) will detect scenarios that are invalid like A+B then A+C with no repair in-between and will stop the pool. The fix is to start the pool again with all three JBODs A+B+C and let C resilver to get the pool back to full health. Then one could disable JBOD-B and the pool would run degraded with A+C. Going direct from a degraded A+B to A+C would not work and again would result in the pool stopping and not importing even after a reboot.		Given the above the only way to cause a split-brain would be to use RAID10 and then carefully manipulate JBOD access and a disabling of the HA group so that the pool is able to start on two separate servers as RAID0. For RAIDZ2/RAIDZ3 pools this would not work as a majority of the media is needed in order for the pool to start and so the pool can never be started on two servers at once due to IO fencing. There are still ways to create problems here though but it would require running the pool across multiple JBODs and then running the pool in a degraded state with different groups of JBODs disabled at different times before the pool could return to a healthy state. Not an easy scenario to recreate and the fix would be to start the pool with the last known set of devices that were successfully imported. For example, if you have a pool that spans JBODs A, B, C such that the pool can run in a degraded state with any two JBODs online and JBOD-C is powered off then the JBOD-C is powered ON and JBOD-B is powered off then the pool will stop. This is because JBODs A+B have the most recent pool information and they need to be powered on with JBOD-C so that the pool can rebuild/resilver. The filesystem (OpenZFS) will detect scenarios that are invalid like A+B then A+C with no repair in-between and will stop the pool. The fix is to start the pool again with all three JBODs A+B+C and let C resilver to get the pool back to full health. Then one could disable JBOD-B and the pool would run degraded with A+C. Going direct from a degraded A+B to A+C would not work and again would result in the pool stopping and not importing even after a reboot.

	~~== Troubleshooting for HA Storage Pools ==~~

	~~''Expanding shortly with tips and suggestions~~ on ~~troubleshooting issues with~~ Storage ~~Pool HA Group setup issues...''~~		=== Resolving Disks Not Visible on both HA nodes / Storage Systems ===

	~~=== Disks are not visible on all systems ===~~		If you don't see the Storage Pool name label on the pool's disk devices under the Physical Disks section on both Storage Systems use this checklist to verify possible connection or configuration problems.
			* Verify physical connectivity from both QuantaStor nodes/systems used in the Site Cluster to the back-end shared JBOD for the storage pool or the back-end SAN.
	If you don't see the ~~storage pool~~ name label on the pool's disk devices on both ~~systems~~ use this checklist to verify possible connection or configuration problems.		* Verify SAS (or FC) cables connected to backend storage are securely seated and link activity lights are active
	* ~~Make sure there is~~ connectivity from both QuantaStor ~~Systems~~ used in the Site Cluster to the back-end shared JBOD for the storage pool or the back-end SAN.		* Verify that all devices that make up the pool are dual-port SAS, dual-port NVMe, FC, or iSCSI devices that support multi-port and Persistent Reservations (NVMe dual-port, SAS dual-port, and NL-SAS dual-port media are all supported for use in scale-up HA clusters).
	* Verify FC~~/SAS~~ cables are securely seated and link activity lights are active ~~(FC).~~		* Verify SAS JBODs have at least two SAS expansion ports and each QuantaStor server has two HBAs. JBODs/JBOFs with 3 or more expansions ports and redundant SAS Expander/Environment Service Modules(ESM) are preferred.
	* ~~All drives used to create~~ the ~~storage~~ pool ~~must be~~ SAS, FC, or iSCSI devices that support multi-port and ~~SCSI3~~ Persistent Reservations.		* Verify SAS JBOD cables are within standard SAS cable lengths (less than 5 meters) of the SAS HBA's installed in the QuantaStor Systems. Use optical SAS cables for longer distances.
	* SAS ~~JBOD should~~ have at least two SAS ~~Expansion~~ ports. ~~Having a JBOD~~ with 3 or more expansions ports and ~~Redundant~~ SAS Expander/Environment Service Modules(ESM) ~~is prefferred~~.		* Note: '''qs-util devicemap''' utility may be helpful when troubleshooting connectivity issues as well as '''qs-iofence devstatus'''
	* SAS JBOD cables ~~should be~~ within standard SAS cable lengths (less than 15 meters) of the SAS HBA's installed in the QuantaStor Systems.
	* Faster devices such as SSDs and 15K RPM platter disk should be placed in a separate enclosure from the NL-SAS ~~disks to ensure best performance~~.
	* ~~Cluster-in-a-Box solutions provide these hardware requirements in a single HA QuantaStor System and are recommended for small configurations (less than 25x drives)~~

	~~The~~ '''qs-util devicemap''' ~~can~~ be helpful when troubleshooting ~~this issue.~~

	=== Useful Tools ===		=== Useful Tools ===

Qadmin: /* Recognize and recover from HA split-brain events */

2024-10-01T06:44:55Z

Recognize and recover from HA split-brain events

← Older revision		Revision as of 06:44, 1 October 2024
Line 91:		Line 91:
	=== Recognize and recover from HA split-brain events ===		=== Recognize and recover from HA split-brain events ===

	It is near impossible to get QuantaStor to split-brain. A split-brain is where a mirrored pool can be split in two halves and both head nodes are running the half pool as effectively a RAID0 pool at the same time. At this point both pools are valid pools but could have different clients and files being written to them. Recovery from a scenario like this involves media reformatting of a pool deemed to be the 'bad' pool and re-silvering the media back to the 'good' pool. Pools using RAIDZ2 (eg 4d+2p) or RAIDZ3 (8d+3p) will not start with 50% of the media so ~~it's even more difficult to cause a~~ split brain ~~with these~~.		It is near impossible to get QuantaStor to split-brain. A split-brain is where a mirrored pool can be split in two halves and both head nodes are running the half pool as effectively a RAID0 pool at the same time. At this point both pools are valid pools but could have different clients and files being written to them. Recovery from a scenario like this involves media reformatting of a pool deemed to be the 'bad' pool and re-silvering the media back to the 'good' pool. Pools using RAIDZ2 (eg 4d+2p) or RAIDZ3 (8d+3p) will not start with 50% of the media so they cannot get into traditional split-brain scenarios at all.

	* QuantaStor prevents split brain by IO fencing (SCSI3-PR) all media devices in a given storage pool (dual-port SAS and dual-port NVMe)		* QuantaStor prevents split brain by IO fencing (SCSI3-PR) all media devices in a given storage pool (dual-port SAS and dual-port NVMe)
Line 97:		Line 97:
	* QuantaStor prevents split brain by only starting pools where enough devices are available for it to start in a degraded state		* QuantaStor prevents split brain by only starting pools where enough devices are available for it to start in a degraded state

	Given the above the only way to cause a split-brain would be to use RAID10 and then carefully manipulate JBOD access and a disabling of the HA group so that the pool is able to start on two separate servers as RAID0. For RAIDZ2/RAIDZ3 pools this would not work as a majority of the media is needed in order for the pool to start and so the pool can never be started on two servers at once due to IO fencing.		Given the above the only way to cause a split-brain would be to use RAID10 and then carefully manipulate JBOD access and a disabling of the HA group so that the pool is able to start on two separate servers as RAID0. For RAIDZ2/RAIDZ3 pools this would not work as a majority of the media is needed in order for the pool to start and so the pool can never be started on two servers at once due to IO fencing. There are still ways to create problems here though but it would require running the pool across multiple JBODs and then running the pool in a degraded state with different groups of JBODs disabled at different times before the pool could return to a healthy state. Not an easy scenario to recreate and the fix would be to start the pool with the last known set of devices that were successfully imported. For example, if you have a pool that spans JBODs A, B, C such that the pool can run in a degraded state with any two JBODs online and JBOD-C is powered off then the JBOD-C is powered ON and JBOD-B is powered off then the pool will stop. This is because JBODs A+B have the most recent pool information and they need to be powered on with JBOD-C so that the pool can rebuild/resilver. The filesystem (OpenZFS) will detect scenarios that are invalid like A+B then A+C with no repair in-between and will stop the pool. The fix is to start the pool again with all three JBODs A+B+C and let C resilver to get the pool back to full health. Then one could disable JBOD-B and the pool would run degraded with A+C. Going direct from a degraded A+B to A+C would not work and again would result in the pool stopping and not importing even after a reboot.

	== Troubleshooting for HA Storage Pools ==		== Troubleshooting for HA Storage Pools ==

Qadmin: /* Recover from OSNexus boot media failure/software corruption/reinstall */

2024-10-01T06:31:46Z

Recover from OSNexus boot media failure/software corruption/reinstall

← Older revision		Revision as of 06:31, 1 October 2024
Line 73:		Line 73:
	This is not something to be overly concerned about, it's all fixable. First remove any excess or large log files from /var/log if you see or suspect that that boot media is simply 100% full. If the boot media has failed then you'll want to replace it with new media and reinstall QuantaStor. The QuantaStor OS boot media does not impact the health of the Storage Pools.		This is not something to be overly concerned about, it's all fixable. First remove any excess or large log files from /var/log if you see or suspect that that boot media is simply 100% full. If the boot media has failed then you'll want to replace it with new media and reinstall QuantaStor. The QuantaStor OS boot media does not impact the health of the Storage Pools.

	* NOTE: Encrypted pools do keep the pool encryption keys on the boot media (AES key wrapped) so they should be exported/saved after the pool is created (see Export Pool Encryption Key Metadata). Assuming the second system still has healthy media or you have a backup of your pool encryption keys then there's no risk to reinstalling QuantaStor onto new or existing boot media. ~~When re-installing we recommend using the latest version of QuantaStor.~~		* NOTE: Encrypted pools do keep the pool encryption keys on the boot media (AES key wrapped) so they should be exported/saved after the pool is created (see Export Pool Encryption Key Metadata). Assuming the second system still has healthy media or you have a backup of your pool encryption keys then there's no risk to reinstalling QuantaStor onto new or existing boot media.

	QuantaStor can be reinstalled from scratch on the the impacted node and then the node can be added back into the storage grid.		When re-installing we recommend using the latest version of QuantaStor. QuantaStor can be reinstalled from scratch on the the impacted node and then the node can be added back into the storage grid. The Site Cluster and associated HA VIFs for the pool will typically need to be re-created. The procedure here is to temporarily change the HA VIFs to local VIFs on the healthy node (see Delete HA Cluster VIF dialog), then after the paired node has been re-added to the grid and the Site Cluster recreated then the local VIF(s) for the pool can be converted back to a HA VIF(s) (see Create HA VIF dialog).
	~~Note that the~~ Site Cluster and associated HA VIFs for the pool will typically need to be re-created. The procedure here is to temporarily change the HA VIFs to local VIFs on the healthy node, then after the paired node has been re-added to the grid and the Site Cluster recreated then the local VIF(s) for the pool can be converted back to a HA VIF(s).

	In a single-node configuration you'll need to use the 'Storage Pool Import..' dialog to re-import the storage pool. It will automatically recover all the metadata associated with the pool including Hosts, Host Groups, Storage Volume ACLs, Network Shares and their configuration settings and more.		In a single-node configuration you'll need to use the 'Storage Pool Import..' dialog to re-import the storage pool. It will automatically recover all the metadata associated with the pool including Hosts, Host Groups, Storage Volume ACLs, Network Shares and their configuration settings and more.

Qadmin: /* Recover from OSNexus boot media failure/software corruption/reinstall */

2024-10-01T06:29:51Z

Recover from OSNexus boot media failure/software corruption/reinstall

← Older revision		Revision as of 06:29, 1 October 2024
Line 73:		Line 73:
	This is not something to be overly concerned about, it's all fixable. First remove any excess or large log files from /var/log if you see or suspect that that boot media is simply 100% full. If the boot media has failed then you'll want to replace it with new media and reinstall QuantaStor. The QuantaStor OS boot media does not impact the health of the Storage Pools.		This is not something to be overly concerned about, it's all fixable. First remove any excess or large log files from /var/log if you see or suspect that that boot media is simply 100% full. If the boot media has failed then you'll want to replace it with new media and reinstall QuantaStor. The QuantaStor OS boot media does not impact the health of the Storage Pools.

	** NOTE: Encrypted pools do keep the pool encryption keys on the boot media (AES key wrapped) so they should be exported/saved after the pool is created (see Export Pool Encryption Key Metadata). Assuming the second system still has healthy media or you have a backup of your pool encryption keys then there's no risk to reinstalling QuantaStor onto new or existing boot media. When re-installing we recommend using the latest version of QuantaStor.		* NOTE: Encrypted pools do keep the pool encryption keys on the boot media (AES key wrapped) so they should be exported/saved after the pool is created (see Export Pool Encryption Key Metadata). Assuming the second system still has healthy media or you have a backup of your pool encryption keys then there's no risk to reinstalling QuantaStor onto new or existing boot media. When re-installing we recommend using the latest version of QuantaStor.

	QuantaStor can be reinstalled from scratch on the the impacted node and then the node can be added back into the storage grid.		QuantaStor can be reinstalled from scratch on the the impacted node and then the node can be added back into the storage grid.

Qadmin: /* Recover from controller failure on JBOD */

2024-10-01T06:29:25Z

Recover from controller failure on JBOD

@@ Line 45: / Line 45: @@
 === Recover from controller failure on JBOD ===
-QuantaStor Storage Systems are generally configured with multi-path connectivity to all attached JBODs with each path coming from a different HBA.
+QuantaStor Storage Systems are generally configured with multi-path connectivity to all attached JBODs with each path coming from a different HBA.  In such configurations the pool will continue to run and the number of paths will be simply reduced from 2 to 1 which in some scenarios can reduce performance, especially with SSD JBOFs.   If the connectivity to the JBOF is single path and the JBOD IO controller fails this will cause the pool to automatically failover to the paired HA node (passive node).  This trigger is typically caused by a failure of the write IO test which runs every few seconds.  If the write test fails a HA failover of the pool is triggered immediately.
-. Recover from unexpected power-off TOR switch.
+The heartbeat mechanism in a HA cluster pair is maintained through what's called a 'Site Cluster'.  Each Site Cluster is typically configured with two 'Cluster Rings' so that there is redundancy to the heartbeat in case a top-of-rack (TOR) switch is disabled, failed or powered off.  In cases where there are spare 1GbE ports it is recommended to use a simple direct connect (crossover cable) between two nodes of an HA pair and to use that as the second cluster ring.  In this way there's always a heartbeat available that will work even in the event all TOR switches are disabled.  For those familiar with corosync+pacemaker technologies, QuantaStor uses these with a custom QuantaStor specific HA failover agent to manage the IP failover which does additional checks and activities like ARP flushing.
-. Re-integrate head node to HA cluster after hardware failure/power-off.
-. Recover from ungraceful head node reboot.
+=== Re-integrate head node to HA cluster after hardware failure/power-off ===
-. Recover from corrupted storage pool.
-. Recover from OSNexus software corruption/reinstall, and ensure data integrity/availability.
+To re-integrate a head node to a HA cluster after a power-off simply just power it back on.  It will re-join the storage grid and will do checks to validate its readiness to accept an HA failover.   Check in the Storage System section to verify the health status of the Storage System and if you see it in a 'Warning' or 'Error' state look in the 'Properties' section for more details under 'State Detail' as it will have a detailed description of the issue.  For example, if the system is powered back on but the SAS cables to the JBOD are disconnected it will indicate a 'Warning' state with details indicating it is not ready to accept a HA failover due to media accessibility issues.  Simply resolve the connectivity issues and the 'Warning' state will automatically clear.
-. Recover from ransomware: contain damage and restore from known-safe snapshot.
-. Remediate incorrectly-replaced hard drive.
+=== Recover from ungraceful head node reboot ===
-. Remediate local replication failure.
-. Replace and rebuild failed hard drives.
+Simply wait for the reboot to finish, the system will resync with the grid and will be ready to accept pool failovers.  If you have two or more pools you will need to manually push one of the pools back to the newly rebooted node as it will not fail-back automatically.
-. Recognize and recover from HA split-brain events.
 == Troubleshooting for HA Storage Pools ==

Qadmin: /* Recover from unexpected power-off of Storage System (HA node) */

2024-10-01T05:09:41Z

Recover from unexpected power-off of Storage System (HA node)

← Older revision		Revision as of 05:09, 1 October 2024
Line 37:		Line 37:

	When a system is powered off (and for any HA failover scenario) the virtual interfaces (VIFs) attached to the pool will automatically failover to the passive node to restore storage access to the pool for all protocols (NFS, SMB, iSCSI, FC, NVMeoF).		When a system is powered off (and for any HA failover scenario) the virtual interfaces (VIFs) attached to the pool will automatically failover to the passive node to restore storage access to the pool for all protocols (NFS, SMB, iSCSI, FC, NVMeoF).
	If you find that one or more clients did not properly recover then you ~~may~~ have clients connecting to Network Shares of the Storage Pool via local IPs on the system rather than via the HA VIFs. There is a 'Client Connectivity Checker' dialog that is accessible from the Storage Systems section, simply right-click on a system to access it and it will enable one to check which IP addresses all clients are using and will report errors for incorrect IP address usage. This generally does not happen with block storage as iSCSI access is limited to just the HA VIFs so we're able to ensure correct connectivity. The same is not the case with NAS protocols like NFS/SMB where clients can be mis-configured to access the storage via local IPs that do not move with the HA VIFs attached to the pool.		If you find that one or more clients did not properly recover after the failover then you most likely have clients connecting to Network Shares of the Storage Pool via local IPs on the system rather than via the HA VIFs associated with the pool. There is a 'Client Connectivity Checker' dialog that is accessible from the Storage Systems section, simply right-click on a system to access it and it will enable one to check which IP addresses all clients are using and will report errors for incorrect IP address usage. This generally does not happen with block storage as iSCSI access is limited to just the HA VIFs so we're able to ensure correct connectivity. The same is not the case with NAS protocols like NFS/SMB where clients can be mis-configured to access the storage via local IPs that do not move with the HA VIFs attached to the pool.

	=== Recover from unexpected power-off of JBOD ===		=== Recover from unexpected power-off of JBOD ===