QuantaStor Monitoring & Cloud Metrics Integration Guide

From OSNEXUS Online Documentation Site
Revision as of 12:29, 28 October 2014 by Qadmin (Talk | contribs)

Jump to: navigation, search

SNMP Agent Configuration

QuantaStor v3.10.1 and newer comes with an SNMP agent so that you can remotely monitor your system via SNMP and get notified of system alerts via the SNMP trap mechanism.

SNMP MIB

The full SNMP MIB for QuantaStor can be found here.

qs-util SNMP Utility Commands

The qs-util command line utility has a number of helper commands to make enabling SNMP and verifying the configuration easier. Here's a list of those commands, you can also run 'qs-util' at the console to see a full list of these commands. Note that you must run many of these commands as root so be sure to do a 'sudo -i' before running them.

  SNMP Commands
    qs-util snmpenable               : Configures the SNMP agent to startup automatically at system startup.
    qs-util snmpdisable              : Configures the SNMP agent to not start automatically at system startup (default).
    qs-util snmpactivate             : Turns on the SNMP agent
    qs-util snmprestart              : Restarts the SNMP service and agent
    qs-util snmpwalkall              : Walks the entire SNMP mib
    qs-util snmpwalkvolumes          : Walks the volumes via the SNMP mib
    qs-util snmpwalkalerts           : Walks the alerts via the SNMP mib
    qs-util snmpmib                  : Displays the contents of the SNMP mib

Enabling the SNMP Agent

By default the QuantaStor SNMP agent is turned off but you can enable it at the console with a couple of commands:

sudo qs-util snmpenable
sudo qs-util snmpactivate

The snmpenable command sets up the appliance so that the SNMP agent will start automatically when the appliance boots up. The snmpactivate command will startup the snmpd and qs_snmpagent services. You must also install the snmp package which contains the snmpwalk and snmpget utilities you can use for testing the agent.

sudo apt-get install snmp

Configuring the SNMP Agent user account

You must edit the /etc/snmp/snmpd.conf configuration file to contain the plain text username and password for the account that will be used for communication between the SNMP agent and the QuantaStor core services. We recommend creating a 'snmpuser' account with the 'System Monitor' role so that even if someone gets the plain text password for the SNMP agent they still cannot make configuration changes to the appliance. If you are not logged into the web management interface you can create a new management user account at the command line like so:

qs user-add snmpuser snmppass "System Monitor" server=localhost,admin,password

In the /etc/snmp/snmpd.conf file you will see lines in there that look like this:

createUser snmpuser MD5 snmppass DES
group nmsGroup usm snmpuser

Edit 'nano /etc/snmp/snmpd.conf' those to match the new user account username and password you gave in the previous step. For example, replace 'snmpuser' with the username of the account you created via the QuantaStor manager web interface, and replace 'snmppass' with the password you gave to that account. When the SNMP agent starts up, it will use the credentials for the first createUser entry in the snmpd.conf file for all communication with the QuantaStor service. So even if you have multiple createUser entries in the snmpd.conf file like "admin" but the first createUser entry is "snmpuser" then "snmpuser" credentials are used for all the SNMP agent to qs_service communication.

Now it is time to restart the SNMP daemon and agent like so:

sudo qs-util snmprestart

Testing the SNMP Agent

Now that you have the SNMP agent enabled with an account associated with it, now it's time to test it to make sure it is working. To do this, use the qs-util commands for doing an SNMP walk, for example:

 
qs-util snmpwalkvolumes snmpuser snmppass
qs-util snmpwalkalerts snmpuser snmppass
qs-util snmpwalkall snmpuser snmppass

Alternatively you can run a snmpwalk like so:

snmpwalk -v 3 -u snmpuser -a MD5 -A snmppass -x DES -X "snmppass" -l authPriv localhost QUANTASTOR-SYS-STATS::storageVolume

Be sure to replace snmpuser and snmppass with the user account you setup and specified in the /etc/snmp/snmpd.conf configuration file. If you're not able to get any data from the snmpwalk commands, try running a simple qs command to verify that the credentials are correct for the account like so:

qs alert-list server=localhost,snmpuser,snmppass

If that doesn't work then either the quantastor service is not running (service quantastor start) or the user account username or password isn't correct.

Configuring SNMP Agent Trap Settings

The alerts within QuantaStor have a severity of error, warning or informational and via the /etc/qs_snmptrapd.conf configuration file you can turn off these categories of alerts to fit your needs. In general you should not ever ignore error messages but it may be handy to disable informational alerts in some cases. Here's the default contents of the /etc/qs_snmptrapd.conf file. Note that if you delete it, the SNMP agent will automatically re-create it for you with the defaults:

poll-interval=120
ignore-error-alerts=false
ignore-warn-alerts=false
ignore-info-alerts=false

If you make any changes to this file, be sure to restart the agent like so.

service snmpagent restart

Or you can restart both the agent and SNMP service like so:

qs-util snmprestart

Testing SNMP Trap Settings

By default the SNMP agent only pushes out traps every 120 seconds so you will have to wait awhile for the trap to be generated after you raise a test alert. QuantaStor only raises traps for Alert objects, so anything that you see in the Alert status bar in the web interface or see in 'qs alert-list' will be sent out as traps. Traps are only sent a single time and the agent keeps track of what alerts have been sent by writing the alert UUIDs to '/var/log/qs_snmpraisedtraps.dat'. If you delete that file then all the alerts will be raised again after the agent restarts. To generate a test alert which will be converted into an SNMP trap use this command:

qs alert-raise --message="snmp test message" --alert-severity=warning --server=localhost,admin,password

After you create the test alert you can then look in the log to see if it has been raised:

qs-showlog -snmp

An easier way to do that is to leave the log open with a 'tail -f /var/log/qs_snmpagent.log' then hit Ctrl-C to stop monitoring the log once you see the trap generated. By default the /etc/snmp/snmpd.conf file is configure to only raise traps to the local host. To raise traps outside of the local host you'll need to add additional lines to the snmpd.conf file like this:

trap2sink 127.0.0.1 public
trap2sink 192.168.10.123 public
trap2sink 10.10.50.134 public

You can also monitor traps using the snmptrapd utility like so:

snmptrapd -P -F "%02.2h:%02.2j TRAP%w.%q from %A %v %W\n"

PagerDuty

PagerDuty is an alarm aggregation and dispatching service for system administrators and support teams. It collects alerts from your monitoring tools, gives you an overall view of all of your monitoring alarms, and alerts an on duty engineer if there's a problem.

Quantastor can be setup to trigger PagerDuty alerts when Quantastor encounters an alert that is of severity "Error", "Warning", or "Critical". Getting setup only requires a few simple steps (internet connection required).

Adding a New Service in PagerDuty

PagerDutySetup1.png

After logging into your PagerDuty account click on the "Services" tab along the top. From here click on the "Add New Service" button.

This service is what all of the Quantastor alerts will be kept under. This will keep the alerts separate from the other programs that may be sending their alerts to PagerDuty.

PagerDutySetup2.png

For the "Service Name" field I would recommend something that describes the box or grid that is being monitored. Also make sure to select "Generic API System" under service type. Quantastor uses PagerDuty's API to post the alert to PagerDuty. After everything is set click "Add Service".

PagerDutySetup3.png

Everything on the PagerDuty side should now be setup. Copy the "Service API Key" and set it aside. This key is the input parameter to tell Quantastor where to post the alert.

Adding PagerDuty to Quantastor

Pagerduty3.png

Open the web interface for the Quantastor system. Right click on the storage system or grid, and select "Alert Manager".

Pagerduty2.png

In the text box titled "PagerDuty.com Service Key" paste the service key from before. Then click on "Apply".

Pagerduty1.png

To test if the system is working select generate test alert. Make sure to select a severity level of "Error", "Warning", or "Critical" and then click okay. If everything is setup correctly a test alert should now be generated and sent to PagerDuty.


Example Alerts

When Quantastor sends an alert to PagerDuty it also sends a list of details to make solving the issue easier. These details include:

  • The serial number of the system
  • The startup time of the system
  • The location
  • The title of the alert
  • The version of the Quantastor service
  • The time at which the alert was sent
  • The name of the system
  • The id of the system
  • The current firmware version
  • The severity of the alert


Pagerduty5.png

Pagerduty4.png

Librato Metrics

Metrics takes away the headaches of traditional server based monitoring solutions that take time to set up, require investments in hardware and take effort to maintain. Metrics is delivered as a service so you don't have to worry about storage, reliability, redundancy, or scalability.

Setup for Librato Metrics

MetricsAccount.png

To post data to Librato Metrics you first must have a Librato Metrics account, which can be created through their website at https://metrics.librato.com. Next you will want to go to your account settings page. This is where you will find your username (email used to create the account) and your API token. This token will be used to post data. At this screen you can do other things such as change your password, or generate a new API token.

ApiTokenSettings.png

When you create the API token, make sure that it is set to "Full Access". This will allow us to create the different Instruments and Dashboards.

MetricsSettings.png

The next step is to configure Quantastor to post data to Librato Metrics using the same API token. Right click on the storage system you wish to post data, and select the Librato Metrics settings. In the dialog that appears set your username as the email you use to log into Librato Metrics. Paste the token from the Librato Metrics site into the token field. The post interval allows you to change how often Quantastor will send data to Librato Metrics. The default value is 60 seconds. Click "OK", and Quantastor should begin posting data.

Viewing the Metrics

To view the data you will first sign into your Librato Metrics account. After signing in click on the "Metrics" tab along the top. This will bring you to a list of all the metrics that have been posted to your account. Quantastor uses a naming convention of: "<storage system/grid name> - <gauge name>"

Quantastor creates the following gauges:

Metrics

  • CPU Load Average
  • Storage Pool Free Space
  • Storage Pool Reads Per Sec
  • Storage Pool Read kB Per Sec
  • Storage Pool Writes Per Sec
  • Storage Pool Write kB Per Sec

Instruments

  • Storage Pool Read:Write
  • Storage Pool Read:Write kBps

Examples

The picture on the left shows an example of a gauge Metric. This graph is the CPU load averages Metric. In the top right corner of the graph you can change the window of time that is currently being viewed.

To the right of that is an example of an Instrument. An Instrument is a combination of of different Metrics. In this Instrument the Storage Pool Read kBps and Write kBps have been combined into one graph.

MetricsCPU.png MetricsInstrument.png


Librato dashboard.png


CopperEgg

CopperEgg is a service that helps you monitor all of your QuantaStor appliances and servers with ease. Out of the box monitoring of CPU, processes, load, memory, swap, disk IO, and the ability to send any custom data you desire. CopperEgg also allows you to set custom thresholds for alerting, which allows you to be notified right when a problem happens.

Setup

The setup is very straight forward for CopperEgg. It is as simple as running a single command the the server you would like to monitor.

Step 1:

  • Select Linux from the list of operating systems (the QuantaStor platform is Ubuntu/Debian Linux based)

Step 2:

  • Copy and paste the command to be run on your QuantaStor system

Your system should now be sending basic monitoring statistics to the CopperEgg cloud. Please allow a minute for updates/stats to start registering at CopperEgg.


CoppereggInstall.png CoppereggInstallInstructions.png

Example Screenshots

CoppereggServerView1.png CoppereggServerView2.png

Nagios Integration / Support

This article has some good detail on setting up Nagios but the installation requires running just a couple of commands:

sudo apt-get update
sudo apt-get install -y nagios3

When installing Nagios for use with QuantaStor note that you must adjust the default port number for apache to something other than port 80 which conflicts with the QuantaStor web management service. For more information on changing the apache port numbers, please see this article which has more detail. To change the port numbers edit '/etc/apache2/ports.conf' and modify the default port number of 80 something like to 8001 and 443 to 4431. Finally, restart apache with 'service apache2 restart'.

After the port number has been changed you can then access Nagios via your web browser at the new port number like so:

http://your-appliance-ip-address:8001/nagios3/

Zabbix Integration / Support

To enable the Zabbix agent directly within your QuantaStor appliance you'll need to install the agent as per the Zabbix documentation on how to install into Ubuntu Server 12.04 (Precise) which can be found here.

Here is a quick summary of the commands to run as detailed on the Zabbix web site:

sudo -i
wget http://repo.zabbix.com/zabbix/2.0/ubuntu/pool/main/z/zabbix-release/zabbix-release_2.0-1precise_all.deb
dpkg -i zabbix-release_2.0-1precise_all.deb
apt-get update
apt-get install zabbix-server-mysql zabbix-frontend-php

Note that Zabbix uses the apache2 web server for its web management interface. Apache uses port 80 by default which conflicts with the Tomcat service QuantaStor uses for its web management interface. As such, you must edit the /etc/apache2/ports.conf file to change the default port numbers. For example you can change 80 to 8001 and 443 to 4431, then restart the apache service with 'service apache2 restart'. This will eliminate the port conflict with the QuantaStor manager web interface. For more information on changing the apache port numbers, please see this article which has more detail.

After the port number has been changed you can then access Nagios via your web browser at the new port number like so:

http://your-appliance-ip-address:8001/zabbix/