Call-home / Alerting

From OSNEXUS Online Documentation Site
Jump to navigation Jump to search


QuantaStor's Alert Manager is the central place to configure how a Storage Grid notifies you when its systems need attention -- media replacement, capacity exhaustion, service failures -- and to set the capacity thresholds that trigger those notifications.

Alerts are delivered through every configured mechanism, so a system with both email and PagerDuty configured sends each event to both. The available mechanisms are:

Navigation: Storage Management → select a Storage System → Alert Manager (toolbar)

Note the Alert Manager toolbar button is inert until a Storage System is selected in the tree.

All alerts are also written to the audit log -- see Audit Logging.

Email Alerts

Alert Manager, Email Alerts tab.
Navigation: Alert Manager → Email Alerts (tab)

Email SMTP Server Settings

These settings route alerts through your mail server.

  • SMTP Server Address -- the address of an accessible SMTP relay or mail server, for example mail.example.com.
  • SMTP Port -- leave as Auto to use the default port for the selected connection security, or set an explicit port when your relay listens elsewhere.
  • SMTP Connection Security -- None, STARTTLS, or SSL/TLS. Prefer STARTTLS or SSL/TLS; None sends credentials and alert content unencrypted.
  • SMTP User / SMTP Password -- credentials for an account with permission to relay through that server. Leave blank for a relay that accepts unauthenticated mail from the storage network.

Email Sender/Recipient

  • Sender Email Address -- the address that appears in the From line. Make it unique and identifiable per storage grid, so an administrator can tell at a glance which system an alert came from.
  • Recipient Email Address -- an address or distribution list for the administrators monitoring this grid. This address receives every severity level.

To route different severities to different people, leave the grid-wide recipient for the catch-all mailbox and configure per-user alert subscriptions instead, on the Add User or Modify User dialog. Each user can subscribe to any combination of Critical, Error, Warning, and Info. See Add User and Modify User.

Append Alerts to Syslog

Ticking this uses the logger tool to append every alert to /var/log/syslog as well, which is useful when a syslog collector is already aggregating the estate.

ITSM Integrations

Alert Manager, ITSM Integrations tab.

IT Service Management (ITSM) modules deliver alerts to a service provider via a webhook URL or service token, so storage alerts land in the same queue as the rest of your infrastructure.

Navigation: Alert Manager → ITSM Integrations (tab)

To add an integration, choose the Module, paste the Webhook URL (or service token) that the provider issued, and press Add. The grid lists the configured integrations; select a row and press Remove Selected to delete one. Several integrations can be configured at once, and all of them receive every alert.

The ? button beside the Module selector opens the OSNEXUS documentation page for the selected module.

Info level alerts are not forwarded to ITSM modules. QuantaStor deliberately withholds Info severity alerts from the webhook modules, so only Warning, Error, and Critical alerts reach your ITSM provider. This is a QuantaStor-side restriction, not a provider setting, and it matters when testing: an Info level test alert will never appear in your ITSM tool and the integration will look broken when it is working correctly. Info level alerts are still delivered by email and recorded in the audit log.

Supported Modules

The Module selector, listing the supported ITSM platforms.
Module Integration notes
AlertOps Inbound Integration
Dynatrace Events V2 Ingest API
Freshservice Webhook API
Google Chat Google Chat channel webhook
Mattermost Mattermost channel webhook
Microsoft Teams Teams channel webhook
OpsGenie Atlassian OpsGenie V2 Alerts API
PagerDuty Events V2 API
ServiceNow Lightstep Generic Webhook API
Slack Slack channel webhook
SolarWinds Solarwinds Service Desk API
Splunk On-Call Webhook API (formerly VictorOps)
Squadcast Webhook API
XMatters Webhook API
Zabbix Monitoring via the Zabbix platform

The module definitions live on each system at:

/opt/osnexus/quantastor/conf/qs_alerthandlers.conf

Additional modules may be present but disabled by default. A module is enabled by uncommenting its stanza in that file and restarting the QuantaStor service; the handler scripts for every module ship regardless, so no reinstall is needed. Note the configuration reader only treats # at the start of a line as a comment.

For the full per-provider setup walkthroughs see the Integration Guide.

Capacity Alert Thresholds

Alert Manager, Capacity Alert Thresholds tab -- nine thresholds across three resource types.

Capacity alerts fire when a Storage Pool, a quota-limited Network Share, or a quota-limited Object user crosses one of three thresholds.

Navigation: Alert Manager → Capacity Alert Thresholds (tab)

All nine values are expressed as "% remaining", not % full. A Pool Low-space Warning of 30 means the alert fires when the pool has 30% free space left -- that is, when it is 70% full.

The three severities escalate:

  • Warning -- an early notice that free space is getting low.
  • Urgent -- a follow-up reminder after the warning level was crossed.
  • Critical -- action should be taken immediately.

Thresholds are set independently for three resource types:

Resource Warning Urgent Critical
Storage Pool low free space 30% remaining 10% remaining 5% remaining
Share Quota low space 20% remaining 10% remaining 5% remaining
Object Quota low space 20% remaining 10% remaining 5% remaining

(The values above are the shipped defaults; each has a slider and a numeric field.)

Share and Object quota pressure is usually resolved by raising the quota. Storage Pools need real capacity added, so act early: expand a pool at around 30% remaining to maintain performance. Pool performance can degrade once utilization passes 90% (10% remaining), depending on fragmentation, so the Urgent and Critical thresholds are deliberately late-stage warnings rather than planning tools.

Alert Types

Alert Manager, Alert Types tab. Each alert type carries an SNMP ID and can be disabled or paused.

QuantaStor defines 289 alert types. Each has a numeric SNMP ID, a Title, a Status, and a Pause Duration.

Navigation: Alert Manager → Alert Types (tab)

This tab is how you silence a persistent alert while maintenance is in progress, without disabling alerting as a whole. Set a type's Status to one of:

  • Enabled -- the default.
  • Disabled -- the alert type never fires. It stays off until you re-enable it.
  • Pause (1 day), Pause (1 week), Pause (1 month), Pause (1 quarter) -- the alert type is silenced for that period and then resumes automatically when the pause window ends.

Prefer a Pause over Disabled for maintenance work: a paused type comes back on its own, whereas a disabled one stays off until somebody remembers it.

With 289 types the list is long, so use the filter: click the Title column header, tick Filters, and type the text to match.

The SNMP ID is the identifier the alert carries when delivered by SNMP trap. An individual alert's SNMP ID is also visible in its properties, by selecting the alert in the Alerts tab at the bottom of the interface.

SNMP

Alert Manager, SNMP tab.

Alerts can be delivered as SNMP traps.

Navigation: Alert Manager → SNMP (tab)

Tick Enable SNMP Agent, then supply the Username and Password under SNMP Credential Settings to require authenticated access.

Full agent configuration, including trap destinations and the MIB, is covered under SNMP Agent Setup.

Generating Test Alerts

Send User Generated Alert dialog -- a message and a severity.

The Generate Test Alert button is available from every tab of the Alert Manager. It raises a real alert through every configured mechanism, which is the quickest way to confirm that SMTP settings, ITSM webhooks, and SNMP are actually working.

IMPORTANT: save your settings with OK or Apply before sending a test alert. An unsaved SMTP server address is not used for the test.

The dialog takes a Message and a Severity. Send the test at "Warning" or higher -- see the note below on Info level alerts.

See Send User Generated Alert for details.

Generating Alerts

QuantaStor systems raise alerts at four severity levels: Critical, Error, Warning, and Info. These are the levels users subscribe to individually in their alert subscriptions.

Note the delivery mechanisms do not all carry every level: email, syslog, and the audit log receive all four, but the ITSM webhook modules do not receive Info -- see ITSM Integrations.

A single physical event commonly produces several alerts. A disk that needs replacing, for example, produces a Warning for the disk itself plus Error level events from the controller, each of which raises its own alert.

To raise an alert deliberately -- for a test, or from a script -- use the CLI:

qs alert-raise --message="Scheduled maintenance window starting" --severity=warning

...or the Generate Test Alert button described above.

Custom Alert Handlers

When none of the built-in mechanisms fit, you can supply your own handler. The per-vendor handlers that ship with QuantaStor are ordinary scripts in:

/opt/osnexus/quantastor/bin/qs_alerthandler_*.py

Any of them works as a worked example of the interface -- they read the alert from QuantaStor and post it onward. Handler registration is defined in:

/opt/osnexus/quantastor/conf/qs_alerthandlers.conf

Audit Logging

Every management operation, alert included, is recorded in the audit log. Audit logging is on by default on all QuantaStor systems and cannot be disabled.

/var/log/qs/qs_audit.log

The file is NIST compliant CEE JSON, one object per line, so a log aggregator or SIEM can ingest it without custom parsing. A helper for reading and summarizing it is installed at:

/opt/osnexus/quantastor/bin/qs_audit.py

For the wider security picture -- password policy, RBAC, and per-user alert subscriptions -- see Security Configuration.

Managing Alerts from the CLI

The alerting subsystem is scriptable. The available commands are:

Command Purpose
qs alert-list List current alerts
qs alert-get Show one alert, including its SNMP ID
qs alert-raise Raise an alert (test, or from a script)
qs alert-clear Clear a specific alert
qs alert-clear-all Clear all alerts
qs alert-config-get Read the Alert Manager configuration, including capacity thresholds
qs alert-config-set Write the Alert Manager configuration
qs alert-type-list List all alert types with their SNMP IDs
qs alert-type-get Show one alert type, including its status and pause state

Run any command with --verbose to see its full argument list, or qs help --min for the complete command list. Each command above links to its full entry in the QuantaStor CLI Command Reference.

Sending logs to support

Support log collection is separate from alerting and is covered on its own page:

  • Sending Logs to Support -- collecting and uploading system logs, the PII scrubbing that makes it GDPR compliant, and the procedure for appliances with no outbound access.

Verified against QuantaStor 6.9.0.