Difference between revisions of "QuantaStor Monitoring & Cloud Metrics Integration Guide"

From OSNEXUS Online Documentation Site
Jump to: navigation, search
m (Setup)
m (SNMP Agent Configuration)
 
(17 intermediate revisions by the same user not shown)
Line 1: Line 1:
  
== SNMP Agent Configuration ==
 
QuantaStor v3.10.1 and newer comes with an SNMP agent so that you can remotely monitor your system via SNMP and get notified of system alerts via the SNMP trap mechanism.
 
 
=== SNMP MIB ===
 
 
The full [https://s3.amazonaws.com/qstor-downloads/mibs/QUANTASTOR-SYS-STATS.mib SNMP MIB for QuantaStor] can be found [https://s3.amazonaws.com/qstor-downloads/mibs/QUANTASTOR-SYS-STATS.mib here].
 
 
=== qs-util SNMP Utility Commands ===
 
 
The qs-util command line utility has a number of helper commands to make enabling SNMP and verifying the configuration easier.  Here's a list of those commands, you can also run 'qs-util' at the console to see a full list of these commands.  Note that you must run many of these commands as root so be sure to do a 'sudo -i' before running them.
 
 
<pre>
 
  SNMP Commands
 
    qs-util snmpenable              : Configures the SNMP agent to startup automatically at system startup.
 
    qs-util snmpdisable              : Configures the SNMP agent to not start automatically at system startup (default).
 
    qs-util snmpactivate            : Turns on the SNMP agent
 
    qs-util snmprestart              : Restarts the SNMP service and agent
 
    qs-util snmpwalkall              : Walks the entire SNMP mib
 
    qs-util snmpwalkvolumes          : Walks the volumes via the SNMP mib
 
    qs-util snmpwalkalerts          : Walks the alerts via the SNMP mib
 
    qs-util snmpmib                  : Displays the contents of the SNMP mib
 
</pre>
 
 
=== Enabling the SNMP Agent ===
 
By default the QuantaStor SNMP agent is turned off but you can enable it at the console with a couple of commands:
 
 
<pre>
 
sudo qs-util snmpenable
 
sudo qs-util snmpactivate
 
</pre>
 
 
The snmpenable command sets up the appliance so that the SNMP agent will start automatically when the appliance boots up.  The snmpactivate command will startup the snmpd and qs_snmpagent services.  You must also install the snmp package which contains the snmpwalk and snmpget utilities you can use for testing the agent.
 
 
<pre>
 
sudo apt-get install snmp
 
</pre>
 
 
=== Configuring the SNMP Agent user account ===
 
 
You must edit the /etc/snmp/snmpd.conf configuration file to contain the plain text username and password for the account that will be used for communication between the SNMP agent and the QuantaStor core services.  We recommend creating a 'snmpuser' account with the 'System Monitor' role so that even if someone gets the plain text password for the SNMP agent they still cannot make configuration changes to the appliance.
 
If you are not logged into the web management interface you can create a new management user account at the command line like so:
 
 
<pre>
 
qs user-add snmpuser snmppass "System Monitor" server=localhost,admin,password
 
</pre>
 
 
In the /etc/snmp/snmpd.conf file you will see lines in there that look like this:
 
 
<pre>
 
createUser snmpuser MD5 snmppass DES
 
group nmsGroup usm snmpuser
 
</pre>
 
 
Edit 'nano /etc/snmp/snmpd.conf' those to match the new user account username and password you gave in the previous step. For example, replace 'snmpuser' with the username of the account you created via the QuantaStor manager web interface, and replace 'snmppass' with the password you gave to that account.  When the SNMP agent starts up, it will use the credentials for the first createUser entry in the snmpd.conf file for all communication with the QuantaStor service.  So even if you have multiple createUser entries in the snmpd.conf file like "admin" but the first createUser entry is "snmpuser" then "snmpuser" credentials are used for all the SNMP agent to qs_service communication.
 
 
Now it is time to restart the SNMP daemon and agent like so:
 
<pre>
 
sudo qs-util snmprestart
 
</pre>
 
 
=== Testing the SNMP Agent ===
 
 
Now that you have the SNMP agent enabled with an account associated with it, now it's time to test it to make sure it is working.  To do this, use the qs-util commands for doing an SNMP walk, for example:
 
 
<pre>
 
qs-util snmpwalkvolumes snmpuser snmppass
 
qs-util snmpwalkalerts snmpuser snmppass
 
qs-util snmpwalkall snmpuser snmppass
 
</pre>
 
 
Alternatively you can run a snmpwalk like so:
 
 
<pre>
 
snmpwalk -v 3 -u snmpuser -a MD5 -A snmppass -x DES -X "snmppass" -l authPriv localhost QUANTASTOR-SYS-STATS::storageVolume
 
</pre>
 
 
Be sure to replace snmpuser and snmppass with the user account you setup and specified in the /etc/snmp/snmpd.conf configuration file.  If you're not able to get any data from the snmpwalk commands, try running a simple qs command to verify that the credentials are correct for the account like so:
 
<pre>
 
qs alert-list server=localhost,snmpuser,snmppass
 
</pre>
 
If that doesn't work then either the quantastor service is not running (service quantastor start) or the user account username or password isn't correct.
 
 
=== Configuring SNMP Agent Trap Settings ===
 
The alerts within QuantaStor have a severity of error, warning or informational and via the /etc/qs_snmptrapd.conf configuration file you can turn off these categories of alerts to fit your needs.  In general you should not ever ignore error messages but it may be handy to disable informational alerts in some cases.  Here's the default contents of the /etc/qs_snmptrapd.conf file.  Note that if you delete it, the SNMP agent will automatically re-create it for you with the defaults:
 
 
<pre>
 
poll-interval=120
 
ignore-error-alerts=false
 
ignore-warn-alerts=false
 
ignore-info-alerts=false
 
</pre>
 
 
If you make any changes to this file, be sure to restart the agent like so.
 
<pre>
 
service snmpagent restart
 
</pre>
 
Or you can restart both the agent and SNMP service like so:
 
<pre>
 
qs-util snmprestart
 
</pre>
 
 
=== Testing SNMP Trap Settings ===
 
By default the SNMP agent only pushes out traps every 120 seconds so you will have to wait awhile for the trap to be generated after you raise a test alert. QuantaStor only raises traps for Alert objects, so anything that you see in the Alert status bar in the web interface or see in 'qs alert-list' will be sent out as traps.  Traps are only sent a single time and the agent keeps track of what alerts have been sent by writing the alert UUIDs to '/var/log/qs_snmpraisedtraps.dat'.  If you delete that file then all the alerts will be raised again after the agent restarts.  To generate a test alert which will be converted into an SNMP trap use this command:
 
 
<pre>
 
qs alert-raise --message="snmp test message" --alert-severity=warning --server=localhost,admin,password
 
</pre>
 
 
After you create the test alert you can then look in the log to see if it has been raised:
 
 
<pre>
 
qs-showlog -snmp
 
</pre>
 
 
An easier way to do that is to leave the log open with a 'tail -f /var/log/qs_snmpagent.log' then hit Ctrl-C to stop monitoring the log once you see the trap generated.  By default the /etc/snmp/snmpd.conf file is configure to only raise traps to the local host.  To raise traps outside of the local host you'll need to add additional lines to the snmpd.conf file like this:
 
 
<pre>
 
trap2sink 127.0.0.1 public
 
trap2sink 192.168.10.123 public
 
trap2sink 10.10.50.134 public
 
</pre>
 
 
You can also monitor traps using the snmptrapd utility like so:
 
 
<pre>
 
snmptrapd -P -F "%02.2h:%02.2j TRAP%w.%q from %A %v %W\n"
 
</pre>
 
 
== PagerDuty ==
 
 
PagerDuty is an alarm aggregation and dispatching service for system administrators and support teams. It collects alerts from your monitoring tools, gives you an overall view of all of your monitoring alarms, and alerts an on duty engineer if there's a problem.
 
 
Quantastor can be setup to trigger PagerDuty alerts when Quantastor encounters an alert that is of severity "Error", "Warning", or "Critical". Getting setup only requires a few simple steps (internet connection required).
 
 
=== Adding a New Service in PagerDuty ===
 
 
{|
 
|-
 
|
 
[[File:pagerDutySetup1.png|400px]]
 
|
 
After logging into your PagerDuty account click on the "Services" tab along the top.
 
From here click on the "Add New Service" button.
 
 
This service is what all of the Quantastor alerts will be kept under. This will keep the alerts separate from the other programs that may be sending their alerts to PagerDuty.
 
|-
 
|
 
[[File:pagerDutySetup2.png|400px]]
 
|
 
For the "Service Name" field I would recommend something that describes the box or grid that is being monitored. Also make sure to select "Generic API System" under service type. Quantastor uses PagerDuty's API to post the alert to PagerDuty. After everything is set click "Add Service".
 
|-
 
|
 
[[File:pagerDutySetup3.png|400px]]
 
|
 
Everything on the PagerDuty side should now be setup. Copy the "Service API Key" and set it aside. This key is the input parameter to tell Quantastor where to post the alert.
 
|}
 
 
=== Adding PagerDuty to Quantastor ===
 
 
{|
 
|-
 
|
 
[[File:pagerduty3.png|400px]]
 
|
 
Open the web interface for the Quantastor system. Right click on the storage system or grid, and select "Alert Manager".
 
|-
 
|
 
[[File:pagerduty2.png|400px]]
 
|
 
In the text box titled "PagerDuty.com Service Key" paste the service key from before. Then click on "Apply".
 
|-
 
|
 
[[File:pagerduty1.png|400px]]
 
|
 
To test if the system is working select generate test alert. Make sure to select a severity level of "Error", "Warning", or "Critical" and then click okay. If everything is setup correctly a test alert should now be generated and sent to PagerDuty.
 
|}
 
 
 
=== Example Alerts ===
 
 
When Quantastor sends an alert to PagerDuty it also sends a list of details to make solving the issue easier. These details include:
 
* The serial number of the system
 
* The startup time of the system
 
* The location
 
* The title of the alert
 
* The version of the Quantastor service
 
* The time at which the alert was sent
 
* The name of the system
 
* The id of the system
 
* The current firmware version
 
* The severity of the alert
 
 
 
{|
 
|-
 
|
 
[[File:pagerduty5.png|400px]]
 
|
 
[[File:pagerduty4.png|400px]]
 
|}
 
 
== Librato Metrics ==
 
 
Metrics takes away the headaches of traditional server based monitoring solutions that take time to set up, require investments in hardware and take effort to maintain. Metrics is delivered as a service so you don't have to worry about storage, reliability, redundancy, or scalability. 
 
 
=== Setup for Librato Metrics ===
 
 
{|
 
|-
 
|
 
[[File:MetricsAccount.png|700px]]
 
|
 
To post data to Librato Metrics you first must have a Librato Metrics account, which can be created through their website at https://metrics.librato.com. Next you will want to go to your account settings page. This is where you will find your username (email used to create the account) and your API token. This token will be used to post data. At this screen you can do other things such as change your password, or generate a new API token.
 
|-
 
|
 
[[File:ApiTokenSettings.png|700px]]
 
|
 
When you create the API token, make sure that it is set to "Full Access". This will allow us to create the different Instruments and Dashboards.
 
|-
 
|
 
[[File:MetricsSettings.png|700px]]
 
|
 
The next step is to configure Quantastor to post data to Librato Metrics using the same API token. Right click on the storage system you wish to post data, and select the Librato Metrics settings. In the dialog that appears set your username as the email you use to log into Librato Metrics. Paste the token from the Librato Metrics site into the token field. The post interval allows you to change how often Quantastor will send data to Librato Metrics. The default value is 60 seconds. Click "OK", and Quantastor should begin posting data.
 
|}
 
 
=== Viewing the Metrics ===
 
 
To view the data you will first sign into your Librato Metrics account. After signing in click on the "Metrics" tab along the top. This will bring you to a list of all the metrics that have been posted to your account. Quantastor uses a naming convention of:
 
"<storage system/grid name> - <gauge name>"
 
 
Quantastor creates the following gauges:
 
 
{|
 
|- valign="top"
 
|
 
Metrics
 
|- valign="top"
 
|
 
* CPU Load Average
 
* Storage Pool Free Space
 
* Storage Pool Reads Per Sec
 
* Storage Pool Read kB Per Sec
 
* Storage Pool Writes Per Sec
 
* Storage Pool Write kB Per Sec
 
|- valign="top"
 
|
 
Instruments
 
|- valign="top"
 
|
 
* Storage Pool Read:Write
 
* Storage Pool Read:Write kBps
 
|}
 
 
=== Examples ===
 
 
{|
 
|-
 
|
 
The picture on the left shows an example of a gauge Metric. This graph is the CPU load averages Metric. In the top right corner of the graph you can change the window of time that is currently being viewed.
 
 
To the right of that is an example of an Instrument. An Instrument is a combination of of different Metrics. In this Instrument the Storage Pool Read kBps and Write kBps have been combined into one graph.
 
|-
 
|
 
[[File:metricsCPU.png|500px]]  [[File:MetricsInstrument.png|500px]]
 
 
 
 
[[File:librato_dashboard.png|1000px]]
 
|}
 
 
 
== CopperEgg ==
 
 
CopperEgg is a service that helps you monitor all of your servers with ease. Out of the box monitoring of CPU, processes, load, memory, swap, disk IO, and the ability to send any custom data you desire. CopperEgg also allows you to set custom thresholds for alerting, which allows you to be notified right when a problem happens.
 
 
=== Setup ===
 
 
The setup is very straight forward for CopperEgg. It is as simple as running a single command the the server you would like to monitor.
 
 
Step 1:
 
* Select Linux from the list of operating systems
 
 
Step 2:
 
* Copy and paste the command to be run on your QuantaStor system
 
 
Your system should now be sending basic monitoring statistics
 
 
 
[[File:CoppereggInstall.png|500px]] [[File:CoppereggInstallInstructions.png|500px]]
 
 
=== Example Screenshots ===
 
 
[[File:CoppereggServerView1.png|500px]]
 
 
[[File:CoppereggServerView2.png|500px]]
 
 
== Nagios Integration / Support ==
 
 
This article has some good detail on setting up [http://askubuntu.com/questions/145518/how-do-i-install-nagios Nagios] but the installation requires running just a couple of commands:
 
 
<pre>
 
sudo apt-get update
 
sudo apt-get install -y nagios3
 
</pre>
 
 
When installing Nagios for use with QuantaStor note that you must adjust the default port number for apache to something other than port 80 which conflicts with the QuantaStor web management service. For more information on changing the apache port numbers, please see this [http://www.cyberciti.biz/faq/linux-apache2-change-default-port-ipbinding/ article] which has more detail. To change the port numbers edit '/etc/apache2/ports.conf' and modify the default port number of 80 something like to 8001 and 443 to 4431.  Finally, restart apache with 'service apache2 restart'.
 
 
After the port number has been changed you can then access Nagios via your web browser at the new port number like so:
 
 
<pre>
 
http://your-appliance-ip-address:8001/nagios3/
 
</pre>
 
 
== Zabbix Integration / Support ==
 
 
To enable the Zabbix agent directly within your QuantaStor appliance you'll need to install the agent as per the Zabbix documentation on how to install into Ubuntu Server 12.04 (Precise) which can be found [https://www.zabbix.com/documentation/2.0/manual/installation/install_from_packages here].
 
 
Here is a quick summary of the commands to run as detailed on the [https://www.zabbix.com/documentation/2.2/manual/installation/install#installing_zabbix_daemons Zabbix web site]:
 
<pre>
 
sudo -i
 
wget http://repo.zabbix.com/zabbix/2.0/ubuntu/pool/main/z/zabbix-release/zabbix-release_2.0-1precise_all.deb
 
dpkg -i zabbix-release_2.0-1precise_all.deb
 
apt-get update
 
apt-get install zabbix-server-mysql zabbix-frontend-php
 
</pre>
 
 
Note that Zabbix uses the apache2 web server for its web management interface.  Apache uses port 80 by default which conflicts with the Tomcat service QuantaStor uses for its web management interface.  As such, you must edit the /etc/apache2/ports.conf file to change the default port numbers.  For example you can change 80 to 8001 and 443 to 4431, then restart the apache service with 'service apache2 restart'. 
 
This will eliminate the port conflict with the QuantaStor manager web interface.  For more information on changing the apache port numbers, please see this [http://www.cyberciti.biz/faq/linux-apache2-change-default-port-ipbinding/ article] which has more detail.
 
 
After the port number has been changed you can then access Nagios via your web browser at the new port number like so:
 
 
<pre>
 
http://your-appliance-ip-address:8001/zabbix/
 
</pre>
 

Latest revision as of 12:23, 12 February 2016