Scale-out File Setup (ceph) - Revision history

Qadmin: Correct three statements that CephFS shares cannot be replicated (scale-out file replication now exists); link multi-admin approval for file pool deletes

2026-09-28T16:43:16Z

Correct three statements that CephFS shares cannot be replicated (scale-out file replication now exists); link multi-admin approval for file pool deletes

Qadmin: Rewrite from the product on 6.9.0: MDS deployment and failover, Ceph File System Create/Modify/Add File Data Pool dialogs, CephFS share constraints, quotas as ceph.quota xattrs, .snap snapshots, Ganesha/SMB/native client access, qs_cephfs.conf, monitoring, verified CLI (QSTOR-12352)

2026-09-03T08:45:20Z

Rewrite from the product on 6.9.0: MDS deployment and failover, Ceph File System Create/Modify/Add File Data Pool dialogs, CephFS share constraints, quotas as ceph.quota xattrs, .snap snapshots, Ganesha/SMB/native client access, qs_cephfs.conf, monitoring, verified CLI (QSTOR-12352)

Show changes

Qadmin: Remove version-era references: removed 1 QuantaStor 5 video link(s) -- all customers run QuantaStor 6 or newer (wiki-docs rule 13, QSTOR-12352)

2026-09-03T03:15:13Z

Remove version-era references: removed 1 QuantaStor 5 video link(s) -- all customers run QuantaStor 6 or newer (wiki-docs rule 13, QSTOR-12352)

← Older revision		Revision as of 03:15, 3 September 2026
Line 8:		Line 8:
	QuantaStor with Ceph is a highly-available and elastic SDS platform that enables scaling object storage environments from a small 3x system configuration to hyper-scale storage grid configurations with 100s of systems. Within a QuantaStor storage grid, up to 20x Ceph Clusters may be managed and automated. QuantaStor's Storage Grid management technology provides single pane of management for all the systems and API, Web UI, and CLI access is available on all systems at the same time. QuantaStor's powerful configuration, monitoring and management features elevate Ceph to an enterprise platform making it easy to setup and maintain large configurations with ease. The following guide covers how to setup scale-out NAS storage, monitor, and maintain it.		QuantaStor with Ceph is a highly-available and elastic SDS platform that enables scaling object storage environments from a small 3x system configuration to hyper-scale storage grid configurations with 100s of systems. Within a QuantaStor storage grid, up to 20x Ceph Clusters may be managed and automated. QuantaStor's Storage Grid management technology provides single pane of management for all the systems and API, Web UI, and CLI access is available on all systems at the same time. QuantaStor's powerful configuration, monitoring and management features elevate Ceph to an enterprise platform making it easy to setup and maintain large configurations with ease. The following guide covers how to setup scale-out NAS storage, monitor, and maintain it.

	~~[[Image:youtube_icon.png\|50px\|link=https://www.youtube.com/watch?v=f9nX9_oOgO0]] [https://www.youtube.com/watch?v=f9nX9_oOgO0 Training Video: QuantaStor 5 Scale-out NAS with CephFS]~~

	{{Template:About_Ceph}}		{{Template:About_Ceph}}

Qadmin: /* Creating Scale-out NAS Storage Pools */

2023-12-07T19:10:52Z

Creating Scale-out NAS Storage Pools

← Older revision		Revision as of 19:10, 7 December 2023
Line 24:		Line 24:
	To create a new File Storage Pool navigate to the '''Scale-out Storage Configuration''' tab, then choose the '''Scale-out Storage Pools''' section. In the ''File Storage'' toolbar press the button for '''Create Storage Pool (NAS/CephFS)''' to bring up the dialog to create a new pool for scale-out NAS storage.		To create a new File Storage Pool navigate to the '''Scale-out Storage Configuration''' tab, then choose the '''Scale-out Storage Pools''' section. In the ''File Storage'' toolbar press the button for '''Create Storage Pool (NAS/CephFS)''' to bring up the dialog to create a new pool for scale-out NAS storage.

	Next choose a name for the pool such as '''file-pool-1''' and then choose a '''Data Pool Layout'''. For workloads that require high performance it is generally best to choose a replica based pool layout. For backup and archive use cases choose an erasure-coding layout. Note ~~that~~ the layout '''cannot be changed later''' so be sure to choose the correct one for your given use case. Note also that the layout can dictate the minimum number of servers required for your Ceph Cluster. For example, if erasure-coding with a K=8 (data blocks) + M=2 (coding blocks) is selected then 8+2 = 10x servers are required at minimum and 11x (one extra beyond the minimum) would be recommended. In contrast a pool with a replica=2 data layout would technically only require a cluster of just 2x servers but as a practical matter QuantaStor requires at least 3x Storage Systems per Ceph Cluster so that quorum may be achieved across the monitors.		Next choose a name for the pool such as '''file-pool-1''' and then choose a '''Data Pool Layout'''. For workloads that require high performance it is generally best to choose a replica based pool layout. For backup and archive use cases choose an erasure-coding layout. Note, the layout '''cannot be changed later''' so be sure to choose the correct one for your given use case. Note also that the layout can dictate the minimum number of servers required for your Ceph Cluster. For example, if erasure-coding with a K=8 (data blocks) + M=2 (coding blocks) is selected then 8+2 = 10x servers are required at minimum and 11x (one extra beyond the minimum) would be recommended. In contrast a pool with a replica=2 data layout would technically only require a cluster of just 2x servers but as a practical matter QuantaStor requires at least 3x Storage Systems per Ceph Cluster so that quorum may be achieved across the monitors.

	= Client Connectivity to Scale-out NAS Storage Pools =		= Client Connectivity to Scale-out NAS Storage Pools =

Qadmin: /* Creating Scale-out NAS Storage Pools */

2023-12-07T19:10:18Z

Creating Scale-out NAS Storage Pools

← Older revision		Revision as of 19:10, 7 December 2023
Line 18:		Line 18:
	Scale-out NAS in QuantaStor uses the CephFS based scale-out filesystem technology. Each Ceph Cluster can only have a single Scale-out NAS (CephFS) based Storage Pool but it is possible to have multiple internal data pools with different layouts. For example one could assign one directory to use a replica=2 data pool and another directory to use a erasure coded (eg K8+M2) data pool.		Scale-out NAS in QuantaStor uses the CephFS based scale-out filesystem technology. Each Ceph Cluster can only have a single Scale-out NAS (CephFS) based Storage Pool but it is possible to have multiple internal data pools with different layouts. For example one could assign one directory to use a replica=2 data pool and another directory to use a erasure coded (eg K8+M2) data pool.

	[[File:~~cephfs_create~~.~~png~~\|512px\|thumb\|Select the Data Pool Layout for a given Scale-out NAS Storage Pool based on the intended use-case]]		[[File:Ceph File System Create - Gen.jpg\|512px\|thumb\|Select the Data Pool Layout for a given Scale-out NAS Storage Pool based on the intended use-case]]

	'''Navigation:''' Scale-out Storage Configuration --> Scale-out Storage Pools --> File Storage --> Create File Storage Pool ''(toolbar)''		'''Navigation:''' Scale-out Storage Configuration --> Scale-out Storage Pools --> File Storage --> Create File Storage Pool ''(toolbar)''
Line 24:		Line 24:
	To create a new File Storage Pool navigate to the '''Scale-out Storage Configuration''' tab, then choose the '''Scale-out Storage Pools''' section. In the ''File Storage'' toolbar press the button for '''Create Storage Pool (NAS/CephFS)''' to bring up the dialog to create a new pool for scale-out NAS storage.		To create a new File Storage Pool navigate to the '''Scale-out Storage Configuration''' tab, then choose the '''Scale-out Storage Pools''' section. In the ''File Storage'' toolbar press the button for '''Create Storage Pool (NAS/CephFS)''' to bring up the dialog to create a new pool for scale-out NAS storage.

	Next choose a name for the pool such as '''~~pool1~~''' and then choose a '''Data Pool Layout'''. For workloads that require high performance it is generally best to choose a replica based pool layout. For backup and archive use cases choose an erasure-coding layout. Note that the layout cannot be changed later so be sure to choose the correct one for your given use case. Note also that the layout can dictate the minimum number of servers required for your Ceph Cluster. For example, if erasure-coding with a K=8 (data blocks) + M=2 (coding blocks) is selected then 8+2 = 10x servers are required at minimum and 11x (one extra beyond the minimum) would be recommended. In contrast a pool with a replica=2 data layout would technically only require a cluster of just 2x servers but as a practical matter QuantaStor requires at least 3x Storage Systems per Ceph Cluster so that quorum may be achieved across the monitors.		Next choose a name for the pool such as '''file-pool-1''' and then choose a '''Data Pool Layout'''. For workloads that require high performance it is generally best to choose a replica based pool layout. For backup and archive use cases choose an erasure-coding layout. Note that the layout '''cannot be changed later''' so be sure to choose the correct one for your given use case. Note also that the layout can dictate the minimum number of servers required for your Ceph Cluster. For example, if erasure-coding with a K=8 (data blocks) + M=2 (coding blocks) is selected then 8+2 = 10x servers are required at minimum and 11x (one extra beyond the minimum) would be recommended. In contrast a pool with a replica=2 data layout would technically only require a cluster of just 2x servers but as a practical matter QuantaStor requires at least 3x Storage Systems per Ceph Cluster so that quorum may be achieved across the monitors.

	= Client Connectivity to Scale-out NAS Storage Pools =		= Client Connectivity to Scale-out NAS Storage Pools =

Qadmin: /* Creating Scale-out NAS Storage Pools */

2023-12-07T19:04:56Z

Creating Scale-out NAS Storage Pools

← Older revision		Revision as of 19:04, 7 December 2023
Line 20:		Line 20:
	[[File:cephfs_create.png\|512px\|thumb\|Select the Data Pool Layout for a given Scale-out NAS Storage Pool based on the intended use-case]]		[[File:cephfs_create.png\|512px\|thumb\|Select the Data Pool Layout for a given Scale-out NAS Storage Pool based on the intended use-case]]

	To create a new File Storage Pool navigate to the '''Scale-out Storage Configuration''' tab, then choose the '''Scale-out Storage Pools''' section. In the toolbar press the button for '''Create Storage Pool (NAS/CephFS)''' to bring up the dialog to create a new pool for scale-out NAS storage.		'''Navigation:''' Scale-out Storage Configuration --> Scale-out Storage Pools --> File Storage --> Create File Storage Pool ''(toolbar)''

			To create a new File Storage Pool navigate to the '''Scale-out Storage Configuration''' tab, then choose the '''Scale-out Storage Pools''' section. In the ''File Storage'' toolbar press the button for '''Create Storage Pool (NAS/CephFS)''' to bring up the dialog to create a new pool for scale-out NAS storage.

	Next choose a name for the pool such as '''pool1''' and then choose a '''Data Pool Layout'''. For workloads that require high performance it is generally best to choose a replica based pool layout. For backup and archive use cases choose an erasure-coding layout. Note that the layout cannot be changed later so be sure to choose the correct one for your given use case. Note also that the layout can dictate the minimum number of servers required for your Ceph Cluster. For example, if erasure-coding with a K=8 (data blocks) + M=2 (coding blocks) is selected then 8+2 = 10x servers are required at minimum and 11x (one extra beyond the minimum) would be recommended. In contrast a pool with a replica=2 data layout would technically only require a cluster of just 2x servers but as a practical matter QuantaStor requires at least 3x Storage Systems per Ceph Cluster so that quorum may be achieved across the monitors.		Next choose a name for the pool such as '''pool1''' and then choose a '''Data Pool Layout'''. For workloads that require high performance it is generally best to choose a replica based pool layout. For backup and archive use cases choose an erasure-coding layout. Note that the layout cannot be changed later so be sure to choose the correct one for your given use case. Note also that the layout can dictate the minimum number of servers required for your Ceph Cluster. For example, if erasure-coding with a K=8 (data blocks) + M=2 (coding blocks) is selected then 8+2 = 10x servers are required at minimum and 11x (one extra beyond the minimum) would be recommended. In contrast a pool with a replica=2 data layout would technically only require a cluster of just 2x servers but as a practical matter QuantaStor requires at least 3x Storage Systems per Ceph Cluster so that quorum may be achieved across the monitors.

Qadmin at 19:43, 17 August 2022

2022-08-17T19:43:11Z

← Older revision		Revision as of 19:43, 17 August 2022
Line 11:		Line 11:

	{{Template:About_Ceph}}		{{Template:About_Ceph}}

			{{Template:Scale-out File Pool Setup (Ceph FS)}}

	= Creating Scale-out NAS Storage Pools =		= Creating Scale-out NAS Storage Pools =

Qadmin: /* Introduction to QuantaStor Scale-out NAS using CephFS technology */

2020-08-19T18:52:32Z

Introduction to QuantaStor Scale-out NAS using CephFS technology

← Older revision		Revision as of 18:52, 19 August 2020
Line 9:		Line 9:

	[[Image:youtube_icon.png\|50px\|link=https://www.youtube.com/watch?v=f9nX9_oOgO0]] [https://www.youtube.com/watch?v=f9nX9_oOgO0 Training Video: QuantaStor 5 Scale-out NAS with CephFS]		[[Image:youtube_icon.png\|50px\|link=https://www.youtube.com/watch?v=f9nX9_oOgO0]] [https://www.youtube.com/watch?v=f9nX9_oOgO0 Training Video: QuantaStor 5 Scale-out NAS with CephFS]


	{{Template:About_Ceph}}		{{Template:About_Ceph}}

Qadmin: /* Introduction to QuantaStor Scale-out NAS using CephFS technology */

2020-08-19T18:52:06Z

Introduction to QuantaStor Scale-out NAS using CephFS technology

← Older revision		Revision as of 18:52, 19 August 2020
Line 7:		Line 7:

	QuantaStor with Ceph is a highly-available and elastic SDS platform that enables scaling object storage environments from a small 3x system configuration to hyper-scale storage grid configurations with 100s of systems. Within a QuantaStor storage grid, up to 20x Ceph Clusters may be managed and automated. QuantaStor's Storage Grid management technology provides single pane of management for all the systems and API, Web UI, and CLI access is available on all systems at the same time. QuantaStor's powerful configuration, monitoring and management features elevate Ceph to an enterprise platform making it easy to setup and maintain large configurations with ease. The following guide covers how to setup scale-out NAS storage, monitor, and maintain it.		QuantaStor with Ceph is a highly-available and elastic SDS platform that enables scaling object storage environments from a small 3x system configuration to hyper-scale storage grid configurations with 100s of systems. Within a QuantaStor storage grid, up to 20x Ceph Clusters may be managed and automated. QuantaStor's Storage Grid management technology provides single pane of management for all the systems and API, Web UI, and CLI access is available on all systems at the same time. QuantaStor's powerful configuration, monitoring and management features elevate Ceph to an enterprise platform making it easy to setup and maintain large configurations with ease. The following guide covers how to setup scale-out NAS storage, monitor, and maintain it.

			[[Image:youtube_icon.png\|50px\|link=https://www.youtube.com/watch?v=f9nX9_oOgO0]] [https://www.youtube.com/watch?v=f9nX9_oOgO0 Training Video: QuantaStor 5 Scale-out NAS with CephFS]


	{{Template:About_Ceph}}		{{Template:About_Ceph}}

Qadmin: /* User Specific Folder Access */

2020-07-28T09:13:03Z

User Specific Folder Access

← Older revision		Revision as of 09:13, 28 July 2020
Line 118:		Line 118:
	caps osd = "allow rw pool=pool1cfs_data"		caps osd = "allow rw pool=pool1cfs_data"
	</pre>		</pre>


	Additional documentation with examples is available [https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2/html/ceph_file_system_guide_technology_preview/mounting_and_unmounting_ceph_file_systems here at the RedHat documentation web site.]

	Next login to the CephFS client system and do the same process for mounting as outlined above but replace client.admin with client.share1. For example, assuming the above commands were run on the 10.0.8.60 system one could scp to get the keyring for share1 like so:		Next login to the CephFS client system and do the same process for mounting as outlined above but replace client.admin with client.share1. For example, assuming the above commands were run on the 10.0.8.60 system one could scp to get the keyring for share1 like so:
Line 133:		Line 130:
	ceph-fuse -n client.share1 -m 10.0.8.60:6789,10.0.8.61:6789,10.0.8.62:6789 -r /share1 /mnt/share1		ceph-fuse -n client.share1 -m 10.0.8.60:6789,10.0.8.61:6789,10.0.8.62:6789 -r /share1 /mnt/share1
	</pre>		</pre>


			Note, additional documentation on CephFS mount options with examples is available [https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2/html/ceph_file_system_guide_technology_preview/mounting_and_unmounting_ceph_file_systems here at the RedHat documentation web site.]

← Older revision		Revision as of 16:43, 28 September 2026
Line 56:		Line 56:
	\|}		\|}

	Choose scale-out when the namespace has to outgrow one appliance, when clients can use the native CephFS client for throughput that SMB and NFS cannot reach, or when erasure coding across systems is the right cost model for a large archive. Choose scale-up when you need the ZFS share features -- ~~replication,~~ cloning, encryption, immutability, per-user quotas -- or when the data fits comfortably on one system.		Choose scale-out when the namespace has to outgrow one appliance, when clients can use the native CephFS client for throughput that SMB and NFS cannot reach, or when erasure coding across systems is the right cost model for a large archive. Choose scale-up when you need the ZFS share features -- cloning, encryption, immutability, per-user quotas, replication with failover -- or when the data fits comfortably on one system.

	A single storage system can host both a ZFS pool and a CephFS pool, with one exception that is easy to overlook: '''a node serves NFS from scale-up pools or from scale-out pools, but not both'''. CephFS does not work correctly re-exported through the kernel NFS server, so scale-out shares are served by NFS-Ganesha instead, and selecting one server masks the other. [[NFS Configuration]] owns that topic and explains the trade-off in full; plan each node as a scale-up NAS or a scale-out NAS and set its NFS Server Type to match.		A single storage system can host both a ZFS pool and a CephFS pool, with one exception that is easy to overlook: '''a node serves NFS from scale-up pools or from scale-out pools, but not both'''. CephFS does not work correctly re-exported through the kernel NFS server, so scale-out shares are served by NFS-Ganesha instead, and selecting one server masks the other. [[NFS Configuration]] owns that topic and explains the trade-off in full; plan each node as a scale-up NAS or a scale-out NAS and set its NFS Server Type to match.
Line 212:		Line 212:
	'''Only the description and the two MDS minimums can be changed after creation.''' The name, the data pool layout, the K and M values and the Scaling Factor are all fixed at create time, so choose the layout for the workload the pool will actually carry. If the layout turns out wrong, add a data pool with the right one and re-point the shares rather than rebuilding.		'''Only the description and the two MDS minimums can be changed after creation.''' The name, the data pool layout, the K and M values and the Scaling Factor are all fixed at create time, so choose the layout for the workload the pool will actually carry. If the layout turns out wrong, add a data pool with the right one and re-point the shares rather than rebuilding.

	Deleting the file system destroys the metadata pool, every data pool and all the data in them. QuantaStor refuses the delete while Network Shares exist on the file system unless you pass the force flag:		Deleting the file system destroys the metadata pool, every data pool and all the data in them. Where [[Security Configuration#Multi-admin Approvals\|multi-admin approval]] covers file pool deletes, the delete is held until the required number of administrators approve it. QuantaStor refuses the delete while Network Shares exist on the file system unless you pass the force flag:

	<pre style="font-size: smaller">		<pre style="font-size: smaller">
Line 256:		Line 256:
	\| '''Clone''' of a share or a snapshot \|\| Nothing equivalent. Copy the data if you need a writable second copy.		\| '''Clone''' of a share or a snapshot \|\| Nothing equivalent. Copy the data if you need a writable second copy.
	\|-		\|-
	\| '''Replication''' to another system, and '''Promote''' \|\| ~~Not~~ available for CephFS shares. ~~The~~ redundancy story ~~for scale-out~~ is the cluster itself, spread across systems and, with a suitable CRUSH failure domain, across racks or sites.		\| '''Replication''' to another system, and '''Promote''' \|\| ZFS replication and Promote are not available for CephFS shares. A CephFS share can be replicated to a file system in a second Ceph Cluster by snapshot mirroring, without failover or promote; see [[Scale-out File Replication]]. Within one cluster, the redundancy story is the cluster itself, spread across systems and, with a suitable CRUSH failure domain, across racks or sites.
	\|-		\|-
	\| '''Share encryption''' -- load, unload, passphrase, key import and export \|\| Not available. Encrypt at rest below the OSDs, or in the application.		\| '''Share encryption''' -- load, unload, passphrase, key import and export \|\| Not available. Encrypt at rest below the OSDs, or in the application.
Line 313:		Line 313:
	The snapshot directory is reachable from a CephFS client and from an NFS client at {{Code\|1=<share>/.snap/<snapshot>}}. Over SMB the snapshots are presented as Windows '''Previous Versions''' -- QuantaStor generates the share with {{Code\|1=vfs objects = shadow_copy2}} pointed at a {{Code\|1=.qsnaps}} link into {{Code\|1=.snap}}, so a user can restore an earlier copy of a file from the Explorer property sheet without an administrator.		The snapshot directory is reachable from a CephFS client and from an NFS client at {{Code\|1=<share>/.snap/<snapshot>}}. Over SMB the snapshots are presented as Windows '''Previous Versions''' -- QuantaStor generates the share with {{Code\|1=vfs objects = shadow_copy2}} pointed at a {{Code\|1=.qsnaps}} link into {{Code\|1=.snap}}, so a user can restore an earlier copy of a file from the Explorer property sheet without an administrator.

	[[Snapshot Schedules]] work on CephFS shares, and taking scheduled snapshots is the main recovery mechanism a scale-out share has -- there is ~~no replication and~~ no immutability to fall back on. Long-term retention rules apply to CephFS shares as well as ZFS ones.		[[Snapshot Schedules]] work on CephFS shares, and taking scheduled snapshots is the main recovery mechanism a scale-out share has -- there is no immutability to fall back on, and [[Scale-out File Replication\|scale-out file replication]] to a second cluster has no built-in failover. Long-term retention rules apply to CephFS shares as well as ZFS ones.

	== NFS access ==		== NFS access ==
Line 511:		Line 511:
	* [[NFS Configuration]] -- the two NFS server implementations, and why a node serves one architecture or the other		* [[NFS Configuration]] -- the two NFS server implementations, and why a node serves one architecture or the other
	* [[Snapshot Schedules]] -- scheduled snapshots and retention, which apply to CephFS shares		* [[Snapshot Schedules]] -- scheduled snapshots and retention, which apply to CephFS shares
			* [[Scale-out File Replication]] -- replicating CephFS shares to a second Ceph Cluster
	* [[Storage Pools]] -- scale-up (ZFS) Storage Pools		* [[Storage Pools]] -- scale-up (ZFS) Storage Pools
	* [[Scale-out Block Setup (ceph)]] -- scale-out block storage on the same Ceph Cluster		* [[Scale-out Block Setup (ceph)]] -- scale-out block storage on the same Ceph Cluster