Difference between revisions of "QuantaStor Monitoring & Cloud Metrics Integration Guide"

From OSNEXUS Online Documentation Site
Jump to: navigation, search
m (Internal SAS Device Multi-path Configuration)
m (Samba v4 / SMB3 Support)
Line 337: Line 337:
 
http://your-appliance-ip-address:8001/zabbix/
 
http://your-appliance-ip-address:8001/zabbix/
 
</pre>
 
</pre>
 
== Samba v4 / SMB3 Support ==
 
 
QuantaStor versions 3.8.2 and newer have support for Samba v4 but an additional configuration step is required to upgrade your system from the default Samba server (Samba v3.6.3) to Samba v4.  The command you need to run as root at the console/SSH is:
 
 
<pre>
 
sudo samba4-install
 
</pre>
 
 
It will ask you a few questions about your Active Directory configuration.  Your answers might look similar to these (note you must use the default 'dc' mode, we do not yet support the other modes). Note also that you must provide a strong password for the domain 'Administrator password' or the script will fail and you'll need to retry using the procedure outlined below.
 
 
<pre>
 
Realm [UNASSIGNED-DOMAIN]: osnexus.net
 
Domain [osnexus]:
 
Server Role (dc, member, standalone) [dc]:
 
DNS backend (SAMBA_INTERNAL, BIND9_FLATFILE, BIND9_DLZ, NONE) [SAMBA_INTERNAL]:
 
DNS forwarder IP address (write 'none' to disable forwarding) [192.168.0.1]: none
 
Administrator password:
 
Retype password:
 
</pre>
 
 
 
If you make a mistake and need to reconfigure the AD configuration settings just re-run the installer and it will prompt you again to enter the AD configuration settings.  In some cases you will have to uninstall samba4, and cleanup the remnants of the failed install, then try again like so:
 
 
<pre>
 
sudo -i
 
apt-get remove samba4
 
rm -rf /opt/samba4
 
samba4-install
 
</pre>
 
 
As of 12/19/2013 we only support the default 'dc' mode and have not yet complete testing of the other modes, namely 'standalone' and 'member'.  After the installation completes you can run this command to verify that the samba4 services are running:
 
 
<pre>
 
service samba4 status
 
smbstatus -V
 
</pre>
 
 
Starting in QuantaStor v3.9 the samba4-install script will turn off the enforcement of strong passwords but you can manually adjust it meet your company's security requirements by running this command.  For strong passwords you'd want a minimum password length of 10 with the complexity requirement turned 'on' rather than 'off'.  Note also that any existing user 'local' user accounts will need to have their passwords re-applied when you upgrade to Samba4, but that does not apply to AD accounts.  If you have strong passwords enabled and a given user has a password that is not strong left over from a prior config then it will block the login when they attempt to access it from their Windows host. 
 
 
<pre>
 
samba-tool domain passwordsettings set --min-pwd-length=1 --complexity=off
 
</pre>
 
 
If you have any questions please feel free to contact us at support (at) osnexus.com or via the Community Support Forum.
 
  
 
== Custom Scripting / Application Extensions ==
 
== Custom Scripting / Application Extensions ==

Revision as of 16:35, 25 August 2014

SNMP Agent Configuration

QuantaStor v3.10.1 and newer comes with an SNMP agent so that you can remotely monitor your system via SNMP and get notified of system alerts via the SNMP trap mechanism.

SNMP MIB

The full SNMP MIB for QuantaStor can be found here.

qs-util SNMP Utility Commands

The qs-util command line utility has a number of helper commands to make enabling SNMP and verifying the configuration easier. Here's a list of those commands, you can also run 'qs-util' at the console to see a full list of these commands. Note that you must run many of these commands as root so be sure to do a 'sudo -i' before running them.

  SNMP Commands
    qs-util snmpenable               : Configures the SNMP agent to startup automatically at system startup.
    qs-util snmpdisable              : Configures the SNMP agent to not start automatically at system startup (default).
    qs-util snmpactivate             : Turns on the SNMP agent
    qs-util snmprestart              : Restarts the SNMP service and agent
    qs-util snmpwalkall              : Walks the entire SNMP mib
    qs-util snmpwalkvolumes          : Walks the volumes via the SNMP mib
    qs-util snmpwalkalerts           : Walks the alerts via the SNMP mib
    qs-util snmpmib                  : Displays the contents of the SNMP mib

Enabling the SNMP Agent

By default the QuantaStor SNMP agent is turned off but you can enable it at the console with a couple of commands:

sudo qs-util snmpenable
sudo qs-util snmpactivate

The snmpenable command sets up the appliance so that the SNMP agent will start automatically when the appliance boots up. The snmpactivate command will startup the snmpd and qs_snmpagent services. You must also install the snmp package which contains the snmpwalk and snmpget utilities you can use for testing the agent.

sudo apt-get install snmp

Configuring the SNMP Agent user account

You must edit the /etc/snmp/snmpd.conf configuration file to contain the plain text username and password for the account that will be used for communication between the SNMP agent and the QuantaStor core services. We recommend creating a 'snmpuser' account with the 'System Monitor' role so that even if someone gets the plain text password for the SNMP agent they still cannot make configuration changes to the appliance. If you are not logged into the web management interface you can create a new management user account at the command line like so:

qs user-add snmpuser snmppass "System Monitor" server=localhost,admin,password

In the /etc/snmp/snmpd.conf file you will see lines in there that look like this:

createUser snmpuser MD5 snmppass DES
group nmsGroup usm snmpuser

Edit 'nano /etc/snmp/snmpd.conf' those to match the new user account username and password you gave in the previous step. For example, replace 'snmpuser' with the username of the account you created via the QuantaStor manager web interface, and replace 'snmppass' with the password you gave to that account. When the SNMP agent starts up, it will use the credentials for the first createUser entry in the snmpd.conf file for all communication with the QuantaStor service. So even if you have multiple createUser entries in the snmpd.conf file like "admin" but the first createUser entry is "snmpuser" then "snmpuser" credentials are used for all the SNMP agent to qs_service communication.

Now it is time to restart the SNMP daemon and agent like so:

sudo qs-util snmprestart

Testing the SNMP Agent

Now that you have the SNMP agent enabled with an account associated with it, now it's time to test it to make sure it is working. To do this, use the qs-util commands for doing an SNMP walk, for example:

 
qs-util snmpwalkvolumes snmpuser snmppass
qs-util snmpwalkalerts snmpuser snmppass
qs-util snmpwalkall snmpuser snmppass

Alternatively you can run a snmpwalk like so:

snmpwalk -v 3 -u snmpuser -a MD5 -A snmppass -x DES -X "snmppass" -l authPriv localhost QUANTASTOR-SYS-STATS::storageVolume

Be sure to replace snmpuser and snmppass with the user account you setup and specified in the /etc/snmp/snmpd.conf configuration file. If you're not able to get any data from the snmpwalk commands, try running a simple qs command to verify that the credentials are correct for the account like so:

qs alert-list server=localhost,snmpuser,snmppass

If that doesn't work then either the quantastor service is not running (service quantastor start) or the user account username or password isn't correct.

Configuring SNMP Agent Trap Settings

The alerts within QuantaStor have a severity of error, warning or informational and via the /etc/qs_snmptrapd.conf configuration file you can turn off these categories of alerts to fit your needs. In general you should not ever ignore error messages but it may be handy to disable informational alerts in some cases. Here's the default contents of the /etc/qs_snmptrapd.conf file. Note that if you delete it, the SNMP agent will automatically re-create it for you with the defaults:

poll-interval=120
ignore-error-alerts=false
ignore-warn-alerts=false
ignore-info-alerts=false

If you make any changes to this file, be sure to restart the agent like so.

service snmpagent restart

Or you can restart both the agent and SNMP service like so:

qs-util snmprestart

Testing SNMP Trap Settings

By default the SNMP agent only pushes out traps every 120 seconds so you will have to wait awhile for the trap to be generated after you raise a test alert. QuantaStor only raises traps for Alert objects, so anything that you see in the Alert status bar in the web interface or see in 'qs alert-list' will be sent out as traps. Traps are only sent a single time and the agent keeps track of what alerts have been sent by writing the alert UUIDs to '/var/log/qs_snmpraisedtraps.dat'. If you delete that file then all the alerts will be raised again after the agent restarts. To generate a test alert which will be converted into an SNMP trap use this command:

qs alert-raise --message="snmp test message" --alert-severity=warning --server=localhost,admin,password

After you create the test alert you can then look in the log to see if it has been raised:

qs-showlog -snmp

An easier way to do that is to leave the log open with a 'tail -f /var/log/qs_snmpagent.log' then hit Ctrl-C to stop monitoring the log once you see the trap generated. By default the /etc/snmp/snmpd.conf file is configure to only raise traps to the local host. To raise traps outside of the local host you'll need to add additional lines to the snmpd.conf file like this:

trap2sink 127.0.0.1 public
trap2sink 192.168.10.123 public
trap2sink 10.10.50.134 public

You can also monitor traps using the snmptrapd utility like so:

snmptrapd -P -F "%02.2h:%02.2j TRAP%w.%q from %A %v %W\n"

PagerDuty

PagerDuty is an alarm aggregation and dispatching service for system administrators and support teams. It collects alerts from your monitoring tools, gives you an overall view of all of your monitoring alarms, and alerts an on duty engineer if there's a problem.

Quantastor can be setup to trigger PagerDuty alerts when Quantastor encounters an alert that is of severity "Error", "Warning", or "Critical". Getting setup only requires a few simple steps (internet connection required).

Adding a New Service in PagerDuty

PagerDutySetup1.png

After logging into your PagerDuty account click on the "Services" tab along the top. From here click on the "Add New Service" button.

This service is what all of the Quantastor alerts will be kept under. This will keep the alerts separate from the other programs that may be sending their alerts to PagerDuty.

PagerDutySetup2.png

For the "Service Name" field I would recommend something that describes the box or grid that is being monitored. Also make sure to select "Generic API System" under service type. Quantastor uses PagerDuty's API to post the alert to PagerDuty. After everything is set click "Add Service".

PagerDutySetup3.png

Everything on the PagerDuty side should now be setup. Copy the "Service API Key" and set it aside. This key is the input parameter to tell Quantastor where to post the alert.

Adding PagerDuty to Quantastor

Pagerduty3.png

Open the web interface for the Quantastor system. Right click on the storage system or grid, and select "Alert Manager".

Pagerduty2.png

In the text box titled "PagerDuty.com Service Key" paste the service key from before. Then click on "Apply".

Pagerduty1.png

To test if the system is working select generate test alert. Make sure to select a severity level of "Error", "Warning", or "Critical" and then click okay. If everything is setup correctly a test alert should now be generated and sent to PagerDuty.


Example Alerts

When Quantastor sends an alert to PagerDuty it also sends a list of details to make solving the issue easier. These details include:

  • The serial number of the system
  • The startup time of the system
  • The location
  • The title of the alert
  • The version of the Quantastor service
  • The time at which the alert was sent
  • The name of the system
  • The id of the system
  • The current firmware version
  • The severity of the alert


Pagerduty5.png

Pagerduty4.png

Librato Metrics

Metrics takes away the headaches of traditional server based monitoring solutions that take time to set up, require investments in hardware and take effort to maintain. Metrics is delivered as a service so you don't have to worry about storage, reliability, redundancy, or scalability.

Setup for Librato Metrics

MetricsAccount.png

To post data to Librato Metrics you first must have a Librato Metrics account, which can be created through their website at https://metrics.librato.com. Next you will want to go to your account settings page. This is where you will find your username (email used to create the account) and your API token. This token will be used to post data. At this screen you can do other things such as change your password, or generate a new API token.

ApiTokenSettings.png

When you create the API token, make sure that it is set to "Full Access". This will allow us to create the different Instruments and Dashboards.

MetricsSettings.png

The next step is to configure Quantastor to post data to Librato Metrics using the same API token. Right click on the storage system you wish to post data, and select the Librato Metrics settings. In the dialog that appears set your username as the email you use to log into Librato Metrics. Paste the token from the Librato Metrics site into the token field. The post interval allows you to change how often Quantastor will send data to Librato Metrics. The default value is 60 seconds. Click "OK", and Quantastor should begin posting data.

Viewing the Metrics

To view the data you will first sign into your Librato Metrics account. After signing in click on the "Metrics" tab along the top. This will bring you to a list of all the metrics that have been posted to your account. Quantastor uses a naming convention of: "<storage system/grid name> - <gauge name>"

Quantastor creates the following gauges:

Metrics

  • CPU Load Average
  • Storage Pool Free Space
  • Storage Pool Reads Per Sec
  • Storage Pool Read kB Per Sec
  • Storage Pool Writes Per Sec
  • Storage Pool Write kB Per Sec

Instruments

  • Storage Pool Read:Write
  • Storage Pool Read:Write kBps

Examples

The picture on the left shows an example of a gauge Metric. This graph is the CPU load averages Metric. In the top right corner of the graph you can change the window of time that is currently being viewed.

To the right of that is an example of an Instrument. An Instrument is a combination of of different Metrics. In this Instrument the Storage Pool Read kBps and Write kBps have been combined into one graph.

MetricsCPU.png MetricsInstrument.png

Gladinet Enterprise Configuration (Secure Private Dropbox-like Solution)

Gladinet Enterprise is Dropbox-like software that allows you to store the data on a local SAN/NAS appliance like your QuantaStor SDS appliance. Gladinet provides remote secure access to folder and files for your users via an "M:" drive (default). Gladinet works much like Dropbox(tm) and adds advanced features like encryption on-the-wire and at rest, user management, and team folders. With all the data stored securely in your datacenter on your QuantaStor appliance(s) you can also ensure physical security of the data and deploy appliances for high security deployments to meet government standards like HIPAA compliance.

QuantaStor Configuration

Setup up QuantaStor to be used with Gladinet was very simple. Here are the steps to getting everything setup:

  1. Create a user account to be used by Gladinet via the QuantaStor web management interface under the Users & Groups section. In the example below you can see that we created a user named 'gladinet' in QuantaStor and then we use that for configuring Gladinet.
  2. Create a network share in the storage pool you would like to use.
    1. When you create the share, be sure to check the Enable CIFS/SMB Access option.
    2. In the CIFS/SMB Advanced Settings section click on the 'None' setting for the gladinet or other user account that you created and set it as a Valid User.
  3. Finally, verify that you can access the network share using this user account from your Windows host before configuring it in Gladinet. After you have verified connectivity you can disconnect from the share in Windows.

Navigation to Attach Local Storage

This part of the guide just covers the configuration of the storage so we've skipped a few of the other steps that come before this if you're installing Gladinet Enterprise from scratch. When you see this storage configuration screen, please follow these steps to complete the configuration of Gladinet for use with your QuantaStor appliance. If you've already installed Gladinet you'll need to navigate to the Attache Local Storage section. Note, the title Attach Local Storage shown in Gladinet is something of a misnomer as this section covers connecting to NAS storage and local filesystems.

  1. Navigate to the Management Console
  2. Select Collaboration from the left hand menu
  3. Select the Storage Manager tab from the top menu
  4. Click on Attach Local Storage link in the upper right

AttachLocalStorage.png

Creating Storage Attachment

Once the Attach Local Storage dialog appears, follow these steps to connect to your QuantaStor network share.

  1. The Root Folder Name is an arbitrary friendly name by which the storage share will be referred to in Gladinet.
  2. For the Local Storage Location provide the full SMB path to the CIFS share. In the example my network share was named test when we created it in the QuantaStor appliance so here we refer to it via the IP address of the QuantaStor appliance and the share name like so \\hostname\sharename.
  3. For the Username enter the QuantaStor user that was given access to the share, in this example we created a user in the QuantaStor appliance named gladinet. Also make sure to put a '\' in front so it doesn't use your local Windows server's domain as part of the username.
  4. The password is the password to the QuantaStor user account; in this example it is the password to the gladinet user.
  5. Make sure to select both check-boxes and then click Create. QuantaStor is Linux based so Gladinet needs this information to properly interface with Samba based shares. Also, we want all access to the QuantaStor appliance to flow through the gladinet user account we're specifying here.

AttachDialog.png

Nagios Integration / Support

This article has some good detail on setting up Nagios but the installation requires running just a couple of commands:

sudo apt-get update
sudo apt-get install -y nagios3

When installing Nagios for use with QuantaStor note that you must adjust the default port number for apache to something other than port 80 which conflicts with the QuantaStor web management service. For more information on changing the apache port numbers, please see this article which has more detail. To change the port numbers edit '/etc/apache2/ports.conf' and modify the default port number of 80 something like to 8001 and 443 to 4431. Finally, restart apache with 'service apache2 restart'.

After the port number has been changed you can then access Nagios via your web browser at the new port number like so:

http://your-appliance-ip-address:8001/nagios3/

Zabbix Integration / Support

To enable the Zabbix agent directly within your QuantaStor appliance you'll need to install the agent as per the Zabbix documentation on how to install into Ubuntu Server 12.04 (Precise) which can be found here.

Here is a quick summary of the commands to run as detailed on the Zabbix web site:

sudo -i
wget http://repo.zabbix.com/zabbix/2.0/ubuntu/pool/main/z/zabbix-release/zabbix-release_2.0-1precise_all.deb
dpkg -i zabbix-release_2.0-1precise_all.deb
apt-get update
apt-get install zabbix-server-mysql zabbix-frontend-php

Note that Zabbix uses the apache2 web server for its web management interface. Apache uses port 80 by default which conflicts with the Tomcat service QuantaStor uses for its web management interface. As such, you must edit the /etc/apache2/ports.conf file to change the default port numbers. For example you can change 80 to 8001 and 443 to 4431, then restart the apache service with 'service apache2 restart'. This will eliminate the port conflict with the QuantaStor manager web interface. For more information on changing the apache port numbers, please see this article which has more detail.

After the port number has been changed you can then access Nagios via your web browser at the new port number like so:

http://your-appliance-ip-address:8001/zabbix/

Custom Scripting / Application Extensions

QuantaStor has script call-outs which you can use to extend the functionality of the appliance for integration with custom applications. For example, you may have an application which needs to be notified before or after a storage pool starts or stops. Or you may have need to call a script before an automated snapshot policy starts in order to quiesce applications.

Security Issues

Scripts are called from the root user account so you must be careful to not allow anyone but the root user to have write access to create files under /var/opt/osnexus/custom. By default the scripts directory has permissions '755'. Your scripts should be configured with file permissions using the command 'chmod 755 scriptname.sh' to prevent non-root user accounts from modifying the scripts. Additionally, if you have sensitive information like a plain text password in your custom script be sure to set the permissions to 700 rather than 755 so only the root user account can read the script.

Timeouts

Scripts must complete within 120 seconds; scripts taking longer are automatically terminated.

Where to install custom scripts

Custom script call-outs are hard-wired to specific file names and must be placed in the custom scripts directory '/var/opt/osnexus/custom' within your QuantaStor appliance. If you have a grid of appliances you'll need to install your script onto all of the appliances.

Custom Scripts Directory:

/var/opt/osnexus/custom

Storage System Custom Scripts

Scripts related to the startup / shutdown of the appliance.

system-poststart.sh

The system poststart script is only called one time when the system boots up. If the management services are restarted it will check against the timestamp in /var/opt/osnexus/quantastor/qs_lastreboot an only call the system-poststart.sh script if it has changed. If you want your poststart script to run every time the management service is restarted you can just delete the qs_lastreboot file in your script.

system-prestop.sh

Called when the user initiates a shutdown or a restart via the web management interface (or CLI). Note that if the admin bypasses the normal shutdown procedure and restarts the appliance at the console using 'reboot' or 'shutdown -P now' or similar command your script won't get called.

Storage Pool Custom Scripts

If you have custom applications running within the appliance which need to attach/detach from the pool or specific directories within a given storage pool these scripts may be helpful to you.

pool-poststart.sh

Called just after a storage pool is started. The UUID of the pool is provided as an input arguement to the script as '--pool=<POOLUUID>'. You can use 'qs pool-get <POOLUUID> --server=localhost,admin,password --xml' to get more detail about the storage pool from within your script. The --xml flag is optional, and you'll need to provide the correct admin password.

pool-prestop.sh

Called just before the pool is stopped.

Snapshot & Replication Schedule Custom Scripts

schedule-prestart.sh

Called just before the a snapshot or replication schedule is triggered / executed. This script is helpful for calling over to applications like databases to tell it to flush writes to prepare for the database to have a snapshot image taken of it. Snapshots are atomic but snapshots taken of multiple volumes or network shares are not atomic as a group. That's where this script can help guide an application spanning multiple Storage Volumes (LUNs) to flush and briefly quiesce IO to give you atomicity across volume snapshots for your application.