Cloudera(tm) Hadoop Integration Guide - Revision history

Qadmin: /* Summary */

2023-03-02T18:56:50Z

Summary

← Older revision		Revision as of 18:56, 2 March 2023
Line 368:		Line 368:
	=== Summary ===		=== Summary ===
	That's the basics of getting Hadoop running with QuantaStor. Now you're ready to deploy		That's the basics of getting Hadoop running with QuantaStor. Now you're ready to deploy
	[https://docs.cloudera.com/documentation/enterprise/latest/topics/installation.html Cloudera Installation Guide]		See: [https://docs.cloudera.com/documentation/enterprise/latest/topics/installation.html Cloudera Installation Guide]
	if you haven't done so already.		if you haven't done so already.

	We are looking forward to automating some of the above configuration steps and adding deeper integration features to monitor CDH in future releases. If you have specific needs, ideas, or would like to share your feedback please write us at engineering@osnexus.com.		We are looking forward to automating some of the above configuration steps and adding deeper integration features to monitor CDH in future releases. If you have specific needs, ideas, or would like to share your feedback please write us at engineering@osnexus.com.

Qadmin: /* Summary */

2023-03-02T18:56:03Z

Summary

← Older revision		Revision as of 18:56, 2 March 2023
Line 367:		Line 367:

	=== Summary ===		=== Summary ===
	That's the basics of getting Hadoop running with QuantaStor. Now you're ready to deploy [https://~~ccp~~.cloudera.com/~~display~~/~~CDH4DOC~~/~~CDH4+~~Installation~~#CDH4Installation-Step4%3ADeployCDHandInstallComponents CDH and install components~~] if you haven't done so already.		That's the basics of getting Hadoop running with QuantaStor. Now you're ready to deploy
			[https://docs.cloudera.com/documentation/enterprise/latest/topics/installation.html Cloudera Installation Guide]
			if you haven't done so already.

	We are looking forward to automating some of the above configuration steps and adding deeper integration features to monitor CDH in future releases. If you have specific needs, ideas, or would like to share your feedback please write us at engineering@osnexus.com.		We are looking forward to automating some of the above configuration steps and adding deeper integration features to monitor CDH in future releases. If you have specific needs, ideas, or would like to share your feedback please write us at engineering@osnexus.com.

Qadmin: /* Phase 4. Setting Configuration Parameters */

2023-03-01T23:00:20Z

Phase 4. Setting Configuration Parameters

← Older revision		Revision as of 23:00, 1 March 2023
Line 291:		Line 291:

	<br>		<br>
	[[File:~~CDH~~-~~Install-Fig-3-05d~~.jpg\| Figure 4-01d]]		[[File:Enable Avid - General.jpg\| Figure 4-01d]]
	<br>		<br>
	Figure 4-01d		Figure 4-01d

Qadmin: /* Phase 4. Setting Configuration Parameters */

2023-03-01T22:56:44Z

Phase 4. Setting Configuration Parameters

← Older revision		Revision as of 22:56, 1 March 2023
Line 286:		Line 286:

	<br>		<br>
	[[File:~~CDH~~-~~Install-Fig-3-05c~~.jpg\| Figure 4-01c]]		[[File:Create Storage Pool - Web.jpg\| Figure 4-01c]]
	<br>		<br>
	Figure 4-~~01b~~		Figure 4-01c

	<br>		<br>

Qadmin at 18:00, 3 May 2019

2019-05-03T18:00:52Z

← Older revision		Revision as of 18:00, 3 May 2019
Line 1:		Line 1:
	[[Category:integration_guide]]		[[Category:integration_guide]]
	This integration guide is focused on configuring your QuantaStor storage grid as [http://www.cloudera.com/content/cloudera/en/home.html Cloudera® Hadoop™] data cluster for better Hadoop performance with less hardware. Note that you can still use your QuantaStor system normally as SAN/NAS ~~appliance~~ with the Hadoop services installed.		This integration guide is focused on configuring your QuantaStor storage grid as [http://www.cloudera.com/content/cloudera/en/home.html Cloudera® Hadoop™] data cluster for better Hadoop performance with less hardware. Note that you can still use your QuantaStor system normally as SAN/NAS system with the Hadoop services installed.

	Setting up [http://www.cloudera.com/content/cloudera/en/home.html Cloudera® Hadoop™] within your [http://www.osnexus.com QuantaStor®] storage ~~appliance~~ is very similar to the steps you would take to setup CDH4 with a standard Ubuntu™ 12.04 / Precise. This is because [http://www.osnexus.com/trynow-iso QuantaStor v3] is built on top of Ubuntu Server.		Setting up [http://www.cloudera.com/content/cloudera/en/home.html Cloudera® Hadoop™] within your [http://www.osnexus.com QuantaStor®] storage system is very similar to the steps you would take to setup CDH4 with a standard Ubuntu™ 12.04 / Precise. This is because [http://www.osnexus.com/trynow-iso QuantaStor v3] is built on top of Ubuntu Server.

	However, there are some important differences and this guide covers all that. Also note that if you're following instructions from the Cloudera web site for steps outside of this How-To be sure refer to the sections regarding Ubuntu Server v12.04/Precise.		However, there are some important differences and this guide covers all that. Also note that if you're following instructions from the Cloudera web site for steps outside of this How-To be sure refer to the sections regarding Ubuntu Server v12.04/Precise.
Line 10:		Line 10:
	Note also that on the Hadoop Manager node, the Hadoop management functionality will impose additional resource costs on the system. For example, when testing the installation using [http://www.osnexus.com QuantaStor®] nodes based on Virtual Machines, the Manager node required at least 10 GB of memory.		Note also that on the Hadoop Manager node, the Hadoop management functionality will impose additional resource costs on the system. For example, when testing the installation using [http://www.osnexus.com QuantaStor®] nodes based on Virtual Machines, the Manager node required at least 10 GB of memory.

	To get started you'll need to login to your QuantaStor v3 storage ~~appliance~~(s) using SSH or via the console. Note that all of the commands shown in this How-To guide should be run as ''root'' so be sure to run 'sudo -i' to get super-user privileges before you begin.		To get started you'll need to login to your QuantaStor v3 storage system(s) using SSH or via the console. Note that all of the commands shown in this How-To guide should be run as ''root'' so be sure to run 'sudo -i' to get super-user privileges before you begin.

	Last, if you don't yet have a QuantaStor v3 storage ~~appliance~~ setup, you can get the [http://www.osnexus.com/trynow-iso CD-ROM ISO and a license key here].		Last, if you don't yet have a QuantaStor v3 storage system setup, you can get the [http://www.osnexus.com/trynow-iso CD-ROM ISO and a license key here].
	<br>		<br>
	<br>		<br>

Qadmin at 18:13, 21 March 2019

2019-03-21T18:13:10Z

← Older revision		Revision as of 18:13, 21 March 2019
Line 1:		Line 1:
			[[Category:integration_guide]]
	This integration guide is focused on configuring your QuantaStor storage grid as [http://www.cloudera.com/content/cloudera/en/home.html Cloudera® Hadoop™] data cluster for better Hadoop performance with less hardware. Note that you can still use your QuantaStor system normally as SAN/NAS appliance with the Hadoop services installed.		This integration guide is focused on configuring your QuantaStor storage grid as [http://www.cloudera.com/content/cloudera/en/home.html Cloudera® Hadoop™] data cluster for better Hadoop performance with less hardware. Note that you can still use your QuantaStor system normally as SAN/NAS appliance with the Hadoop services installed.

Qadmin at 22:46, 7 April 2016

2016-04-07T22:46:56Z

← Older revision		Revision as of 22:46, 7 April 2016
Line 374:		Line 374:

	''Cloudera is a registered trademark of Cloudera Corporation, Hadoop is a registered trademark of the Apache Foundation, Ubuntu is a registered trademark of Canonical, and QuantaStor is a registered trademark of OS NEXUS Corporation.''		''Cloudera is a registered trademark of Cloudera Corporation, Hadoop is a registered trademark of the Apache Foundation, Ubuntu is a registered trademark of Canonical, and QuantaStor is a registered trademark of OS NEXUS Corporation.''

			[[Category:QuantaStor_Guide]]

Qadmin: /* Phase 4. Setting Configuration Parameters */

2013-10-23T00:42:06Z

Phase 4. Setting Configuration Parameters

← Older revision		Revision as of 00:42, 23 October 2013
Line 294:		Line 294:
	Figure 4-01d		Figure 4-01d

			<br>
	Having created Storage Pools, and Network Shares on those pools, the directories where these data destinations should reside would be according to the following examples:		Having created Storage Pools, and Network Shares on those pools, the directories where these data destinations should reside would be according to the following examples:
	<br>		<br>

Qadmin: /* Phase 4. Setting Configuration Parameters */

2013-10-23T00:41:47Z

Phase 4. Setting Configuration Parameters

@@ Line 279: / Line 279: @@
 <br>
-These configuration options bear special attention on QuantaStor installations - to intent here is to specifically setup Hadoop to use the storage on the QuantaStor Storage Pools. By default, the Hadoop installations create top-level directories on the nodes' root filesystem, such as '/dfs' and '/mapred'.
+These configuration options bear special attention on QuantaStor installations - to intent here is to specifically setup Hadoop to use the storage on the QuantaStor Storage Pools. By default, the Hadoop installations create top-level directories on the nodes' root filesystem, such as '/dfs' and '/mapred'. So part of the pre-installation setup of the QuantaStor systems includes, on each node in the cluster, the creation of a Storage Pool and a Network Share on that Pool to function as the filesystem root for the Hadoop data store within the Pool.
 Having created Storage Pools, and Network Shares on those pools, the directories where these data destinations should reside would be according to the following examples:

Qadmin: /* Phase 4. Setting Configuration Parameters */

2013-10-14T23:13:26Z

Phase 4. Setting Configuration Parameters

← Older revision		Revision as of 23:13, 14 October 2013
Line 314:		Line 314:
	<pre>		<pre>
	$ mkdir /export/osn-grid2a-pool1-share1/dfs		$ mkdir /export/osn-grid2a-pool1-share1/dfs
			$ ln -s /export/osn-grid2a-pool1-share1/dfs /dfs
	$ mkdir /export/osn-grid2a-pool1-share1/mapred		$ mkdir /export/osn-grid2a-pool1-share1/mapred
	~~$ ln -s /export/osn-grid2a-pool1-share1/dfs /dfs~~
	$ ln -s /export/osn-grid2a-pool1-share1/mapred /mapred		$ ln -s /export/osn-grid2a-pool1-share1/mapred /mapred
	</pre>		</pre>