Zabbix Integration

Zabbix is an enterprise-class open source monitoring and observability platform. QuantaStor 6.9 and newer integrate with Zabbix via an alert handler tied into the QuantaStor Alert Manager, which pushes each alert into Zabbix over the Zabbix API. Setup is a matter of generating an API token in Zabbix, combining it with your Zabbix URL, and adding the result to the QuantaStor Alert Manager screen as a new Alert Endpoint. For QuantaStor 6.8 and earlier, see QuantaStor 6.8 and earlier.

Requirements: QuantaStor 6.9 or newer and Zabbix 7.0 or newer, with the Zabbix front end reachable over HTTPS from every storage system in the grid. If you don't have a Zabbix server available you can stand up a free 3-day trial instance at cloud.zabbix.com using the nano tier.

Step 1 - Generate an API Token

Login to Zabbix and navigate to:

Users -> API tokens -> Create API token

Name the token quantastor-itsm and choose the user to associate it with. We recommend leaving Set expiration date and time un-checked so that alerts are not missed due to token expiration.

The token is displayed only once, so be sure to copy it before leaving the page.

The user the token is tied to needs rights to create hosts and, for the one-time setup in Step 3, to create host groups and import templates. A Super admin user is the simplest choice.

Step 2 - Assemble the Endpoint URL

The endpoint URL that QuantaStor needs is the base URL of your Zabbix front end, followed by a '#' character, followed by the API token from Step 1:

https://<zabbix-host>/#<API-token>

With Zabbix Cloud the front end is at the root of the host name, so the URL will look like this:

https://qs-example.zabbix.cloud/#b7f3a91c2d6e480fa5c81e37d94b06af2c15e8d073a4bf6295cd10e8473f92

With a self-hosted Zabbix server the front end is usually served under /zabbix (the Zabbix nginx package serves it at the root of the host instead), so the URL will look like this:

https://zabbix.example.com/zabbix#b7f3a91c2d6e480fa5c81e37d94b06af2c15e8d073a4bf6295cd10e8473f92

QuantaStor appends the API path (api_jsonrpc.php) to the URL automatically, so there's no need to include it.

The Alert Manager accepts only https:// URLs, and QuantaStor checks the Zabbix server's certificate. If the certificate was issued by an internal or self-signed certificate authority, add that CA certificate to the trusted store on every storage system in the grid:

sudo cp <ca-certificate>.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates

Step 3 - Import the QuantaStor Template

Run the alert handler once with the --setup option on any storage system in the grid, passing the URL assembled in Step 2:

/opt/osnexus/quantastor/bin/qs_alerthandler_zabbix.py --setup --url 'https://qs-example.zabbix.cloud/#b7f3a91c2d6e480fa5c81e37d94b06af2c15e8d073a4bf6295cd10e8473f92'

Note the single quotes around the URL, they're required so that the shell doesn't treat the '#' as the start of a comment.

This imports the QuantaStor by Trapper template (items, triggers and a per-host dashboard) and creates a grid-wide QuantaStor Alerts dashboard under Dashboards in Zabbix.

This step is optional in the sense that alerts will still be delivered without it, but until the template is imported they arrive only as a raw JSON payload with no triggers, no entries under Monitoring -> Problems, and no dashboard, so we recommend always running it.

Step 4 - Add the Endpoint to QuantaStor

Login to your QuantaStor storage grid and then select the Alert Manager button in the toolbar to bring up the dialog for adding alert endpoints. On the second page select Zabbix from the dropdown list of ITSM integrations and then input the URL endpoint you assembled in Step 2.

Step 5 - Apply Changes and Test

Apply your changes to add the new endpoint, then use the Generate Test Alert button to verify that everything is configured correctly and that alerts are making it into Zabbix. Note that "INFO" level alerts are not sent to the ITSM modules, so be sure to generate a "WARNING" or "ERROR" level alert for testing.

The first alert from a given storage system creates its host in Zabbix, and the Zabbix server only picks up that new configuration on its next cache reload (up to 60 seconds). The alert handler retries until the alert lands, so the first test alert may take a minute or so to appear.

A test alert on the QuantaStor Alerts dashboard.

Viewing Alerts in Zabbix

Alerts can be reviewed in three places:

• Dashboards -> QuantaStor Alerts - alert counts by severity plus the full alert list across every storage system in the grid.
• Monitoring -> Problems - the standard Zabbix problem view, filtered however you like.
• The QuantaStor alerts dashboard on an individual host - the last alert received from that one storage system.

The QuantaStor Alerts dashboard, with alert counts by severity, the alerts received, and the latest alert from each storage system.

QuantaStor alerts are one-shot events with no automatic recovery condition, so a problem stays open until it is closed by hand. To dismiss an alert use the Update link to the right of it and select Close problem.

What the Integration Creates in Zabbix

Object	Name	Notes
Host group	QuantaStor	Holds every storage system reporting into Zabbix
Host	One per storage system	Named by the storage system's IP address, with the system name as the visible name
Template	QuantaStor by Trapper	Linked automatically to each host created by the handler
Dashboard	QuantaStor Alerts	Grid-wide alert dashboard, created by --setup
Template dashboard	QuantaStor alerts	Per-host view of the most recent alert

Each alert is pushed as a JSON document into the quantastor.alert trapper item, and the template derives the individual fields (title, description, severity, alert type, storage system, timestamp) from it. QuantaStor alert severities map to Zabbix trigger priorities as follows:

QuantaStor severity	Zabbix priority
Critical	Disaster
Error	High
Warning	Warning
Info	not forwarded

QuantaStor 6.8 and earlier

QuantaStor 6.8 and earlier do not include the Zabbix alert handler. Instead, install a small custom alert handler on each storage system that sends every warning, error and critical alert to Zabbix with zabbix_sender, and optionally the Zabbix agent 2 for operating system metrics such as CPU, memory and disk usage. Both connect from the storage system to the Zabbix server or proxy on port 10051, so Zabbix does not need to reach the storage system. QuantaStor runs every executable .py file in /var/opt/osnexus/alerthandlers for each new alert; the qs_alerthandler_sample.py.disabled file in that directory shows the arguments it passes.

If your monitoring policy allows Zabbix to poll the storage system, you can also read QuantaStor objects over SNMP; see SNMP Agent Setup.

Create the host and alert item in Zabbix

Login to Zabbix and navigate to:

Data collection -> Hosts -> Create host

Set Host name to the storage system's host name exactly as the hostname command prints it on that system, and enter QuantaStor in Host groups to create the group. If you plan to install the Zabbix agent 2 (see below), add the Linux by Zabbix agent active template. The host needs no interface. Click Add.

Open the new host's Items list and click Create item:

Field	Value
Name	QuantaStor alert
Type	Zabbix trapper
Key	quantastor.alert
Type of information	Text

Open the host's Triggers list and create one trigger for each alert severity. In each, set PROBLEM event generation mode to Multiple, OK event generation to None, and select Allow manual close:

Name	Severity	Expression
QuantaStor critical alert on {HOST.NAME}: {ITEM.LASTVALUE}	Disaster	find(/<host-name>/quantastor.alert,,"regexp","^critical:")=1
QuantaStor error alert on {HOST.NAME}: {ITEM.LASTVALUE}	High	find(/<host-name>/quantastor.alert,,"regexp","^error:")=1
QuantaStor warning alert on {HOST.NAME}: {ITEM.LASTVALUE}	Warning	find(/<host-name>/quantastor.alert,,"regexp","^warning:")=1

Replace <host-name> with the host name from the first step. QuantaStor alerts have no recovery condition, so each problem stays open until you close it from Monitoring -> Problems.

Install the alert handler on the storage system

Login to the storage system over SSH and add the Zabbix 7.0 package repository, then install zabbix_sender and the Zabbix agent 2:

wget -O /tmp/zabbix-release.deb https://repo.zabbix.com/zabbix/7.0/ubuntu/pool/main/z/zabbix-release/zabbix-release_latest_7.0+ubuntu$(lsb_release -rs)_all.deb
sudo dpkg -i /tmp/zabbix-release.deb
sudo apt update
sudo apt install -y zabbix-sender zabbix-agent2

Leave out zabbix-agent2 if you do not want operating system metrics.

Create the alert handler:

sudo tee /var/opt/osnexus/alerthandlers/qs_alerthandler_zabbix_sender.py > /dev/null <<'EOF'
#!/usr/bin/python3
# QuantaStor custom alert handler: sends each warning, error and critical alert to
# the Zabbix trapper item "quantastor.alert" with zabbix_sender, as one line of text:
#   <severity>: <title> - <description>
# Settings are read from /etc/qs_zabbix_sender.conf:
#   server=<Zabbix server or proxy>
#   port=10051
#   host=<host name of this storage system in Zabbix>
import argparse
import subprocess
import sys

CONF = "/etc/qs_zabbix_sender.conf"

def read_conf():
    conf = {"port": "10051"}
    with open(CONF) as f:
        for line in f:
            if "=" in line and not line.lstrip().startswith("#"):
                key, value = line.split("=", 1)
                conf[key.strip()] = value.strip()
    for key in ("server", "host"):
        if not conf.get(key):
            print(f"Err: {CONF} has no {key}= line")
            sys.exit(1)
    return conf

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--severity", default="")
    parser.add_argument("--title", default="")
    parser.add_argument("--description", default="")
    parser.add_argument("--is_close", action="store_true")
    opts, _ = parser.parse_known_args()  # QuantaStor passes more arguments than this handler uses
    if opts.is_close:
        return
    conf = read_conf()
    value = f"{opts.severity}: {opts.title} - {opts.description}"
    result = subprocess.run(["zabbix_sender", "-z", conf["server"], "-p", conf["port"], "-s", conf["host"],
                             "-k", "quantastor.alert", "-o", value], capture_output=True, text=True, timeout=60)
    if result.returncode:  # zabbix_sender exits non-zero unless the server accepted the value
        print(f"Err: zabbix_sender exit {result.returncode}: {result.stdout.strip()} {result.stderr.strip()}")
        sys.exit(1)

if __name__ == "__main__":
    main()
EOF
sudo chmod 0755 /var/opt/osnexus/alerthandlers/qs_alerthandler_zabbix_sender.py

Tell the handler where to send alerts, replacing <zabbix-server> with the address of your Zabbix server or proxy:

printf 'server=<zabbix-server>\nport=10051\nhost=%s\n' "$(hostname)" | sudo tee /etc/qs_zabbix_sender.conf

The host= value must match the Host name you gave the host in Zabbix.

Configure the Zabbix agent 2 (optional)

Configure the agent for active checks only, so it sends its data to Zabbix and does not listen for connections, then restart it:

sudo sed -i -e 's/^Server=.*/Server=/' -e 's/^ServerActive=.*/ServerActive=<zabbix-server>/' -e "s/^Hostname=.*/Hostname=$(hostname)/" /etc/zabbix/zabbix_agent2.conf
sudo systemctl restart zabbix-agent2

The Linux by Zabbix agent active template on the host starts collecting within a few minutes.

Test

Raise a test alert, replacing <admin-password> with the password of the QuantaStor admin user:

qs alert-raise --title="Zabbix test" --message="Test alert from QuantaStor" --alert-severity=warning --server=localhost,admin,<admin-password>

The alert appears within a few seconds under Monitoring -> Problems as QuantaStor warning alert on <host-name>: warning: [122] Zabbix test - Test alert from QuantaStor. Info level alerts are not sent. If nothing arrives, check /var/log/qs/qs_service.log for a line containing Err: zabbix_sender exit 2 and failed: 1, which means the Zabbix server rejected the value. The usual cause is a host= value that does not match the host name in Zabbix.

Moving to QuantaStor 6.9

After upgrading to QuantaStor 6.9, set up the built-in handler with the steps at the top of this page. It creates its own host for each storage system, so first delete /var/opt/osnexus/alerthandlers/qs_alerthandler_zabbix_sender.py on each storage system to stop alerts arriving twice, then delete the hosts you created for 6.8 from Zabbix.