Datadog Integration

QuantaStor can send its alerts to Datadog as Datadog events, through the Datadog module of the Alert Manager. This page covers creating the Datadog API key, adding the endpoint in QuantaStor, testing it, and building Datadog monitors on the severity and system tags that every event carries.

Section	Purpose
How it works	What is sent, where, and what is not sent
Step 1: Create a Datadog API key	The key the endpoint URL carries
Step 2: Add the Datadog endpoint	The Alert Manager ITSM Integrations tab
Step 3: Send a test alert	Confirm events arrive in Datadog
Step 4: Create Datadog monitors	Turn QuantaStor events into notifications
Endpoint URL format and Datadog sites	The URL forms accepted and the site each host selects
Event contents	Title, body, host and date of each event
Severity mapping and tags	How QuantaStor severities and tags appear in Datadog
Configuring from the CLI	Adding, removing and testing the endpoint with qs
Troubleshooting	No events, 403 errors, missing Info alerts

How it works

Each QuantaStor alert of Warning, Error or Critical severity is posted directly from the appliance to the Datadog Events API as a Datadog event. No Datadog Agent is needed on the appliance. Every event carries tags that name the product, the severity, the storage system and the grid, so you can filter the Datadog Event Explorer and build monitors on them.

Three behaviours to know before you start:

Info alerts are never sent. QuantaStor sends only Warning, Error and Critical alerts to ITSM modules, Datadog included. A test at Info level produces nothing in Datadog, which makes a correctly configured endpoint look broken.
Clearing an alert sends nothing. Datadog events cannot be closed, so dismissing or clearing an alert in QuantaStor has no counterpart in Datadog. Use monitor recovery logic in Datadog if you need one.
Repeats roll up. Repeats of the same alert type on the same storage system share an aggregation key, so Datadog groups them together rather than listing each one separately.

The appliance needs outbound HTTPS access to the Datadog API host for your site. If the appliance is configured to use an HTTPS proxy, the module sends through it.

Step 1: Create a Datadog API key

In Datadog, open Organization Settings → API Keys and create a key for QuantaStor, or copy an existing one. Use an API key, not an Application key: the Events API accepts only API keys.

Note which Datadog site your organization is on (US1, US3, US5, EU1, AP1, AP2 or US1-FED). The key works only against that site's API host; see Endpoint URL format and Datadog sites.

Step 2: Add the Datadog endpoint

Navigation: Storage Management → Storage System (toolbar group) → Alert Manager (toolbar) → ITSM Integrations (tab)

Alert Manager, ITSM Integrations tab with a Datadog endpoint added. Choose the module, paste the URL into Webhook URL, then click Add.

The ITSM Integrations tab lists every configured endpoint, one row per module and URL. To add Datadog:

Open the Alert Manager and select the ITSM Integrations tab.
In Module, select Datadog. The ? button beside it opens the setup page for the selected module.
In Webhook URL, paste the Datadog API host followed by # and your API key, for example https://api.datadoghq.com#aAbBcCdDeEfF0123456789aAbBcCdDeE.
Click Add. The endpoint appears in the list.
Click OK or Apply to save the configuration.

The Module drop-down lists every ITSM module the appliance supports, Datadog among them.

To remove an endpoint, tick it in the list, click Remove Selected, then click OK or Apply.

QuantaStor sends the API key to Datadog in the DD-API-KEY request header, never in the request URL, and masks it as ******** in the service log.

Step 3: Send a test alert

Send User Generated Alert, opened from Generate Test Alert. Severity offers Warning, Error and Critical.

Click Generate Test Alert at the bottom of the Alert Manager. The Send User Generated Alert dialog opens with two fields:

Message -- the text of the test alert.
Severity -- Warning, Error or Critical. Info is not offered, because Info alerts never reach ITSM modules.

Click OK, then open the Datadog Event Explorer and search for product:quantastor. The test event appears within a few seconds, with the host set to the storage system name.

If nothing arrives, see Troubleshooting.

Step 4: Create Datadog monitors

Events on their own notify nobody. To page an on-call engineer, create a Datadog Event Monitor for each severity you want to act on. For example:

Monitor	Query	Condition
QuantaStor critical alerts	`product:quantastor severity:critical`	count > 0
QuantaStor error alerts	`product:quantastor severity:error`	count > 0
QuantaStor warnings	`product:quantastor severity:warning`	count > 0

Filter on the severity tag rather than on the Datadog event status: Datadog has no Critical status, so Error and Critical alerts both arrive with status Error, and only the tag keeps them apart. Add system:<name> or grid:<name> to a query to scope a monitor to one storage system or one grid, and route each monitor to the team or channel that owns it.

Endpoint URL format and Datadog sites

The Webhook URL takes either of two forms:

Form	Example
API host, `#`, API key	`https://api.datadoghq.com#aAbBcCdDeEfF0123456789aAbBcCdDeE`
Datadog's own Events API URL	`https://api.datadoghq.com/api/v1/events?api_key=aAbBcCdDeEfF0123456789aAbBcCdDeE`

Both send the key in a request header; the second form exists so you can paste the URL Datadog's documentation gives.

The host selects the Datadog site:

Datadog site	API host
US1	`api.datadoghq.com`
US3	`api.us3.datadoghq.com`
US5	`api.us5.datadoghq.com`
EU1	`api.datadoghq.eu`
AP1	`api.ap1.datadoghq.com`
AP2	`api.ap2.datadoghq.com`
US1-FED	`api.ddog-gov.com`

If you paste a site or web interface host instead, such as datadoghq.eu or app.datadoghq.com, QuantaStor maps it to the matching API host automatically.

Event contents

Each QuantaStor alert becomes one Datadog event:

Event field	Content
Title	The alert title, prefixed with the alert type number in brackets, for example `[122] ...`
Text	The alert description, followed by Severity, Alert Time, Alert Type, Storage System (linked to its IP address), Storage System ID, Storage Grid, Location, Serial Number, Service Version, Firmware Version, Kernel Version, System Startup and Alert ID
Host	The storage system name
Date	The time QuantaStor raised the alert

Severity mapping and tags

QuantaStor severity	Datadog event status	Severity tag
Info	not sent	--
Warning	Warn	`severity:warning`
Error	Error	`severity:error`
Critical	Error	`severity:critical`

Every event carries these tags:

Tag	Value
`product:quantastor`	Always present; use it to select all QuantaStor events
`severity:<level>`	`warning`, `error` or `critical`
`system:<name>`	The storage system that raised the alert
`system_id:<id>`	The storage system's unique ID
`grid:<name>`	The storage grid the system belongs to
`alert_type_id:<id>`	The alert type number, the same one shown in the title
`location:<location>`	The storage system's location, only when one is set

Datadog adds source:my_apps itself.

Configuring from the CLI

Use qs alert-config-set with --endpoints to add the endpoint. Each entry is module:url; separate several entries with commas.

qs alert-config-set --endpoints="datadog:https://api.datadoghq.com#aAbBcCdDeEfF0123456789aAbBcCdDeE"

To remove the endpoint, give the same entry with a ~ prefix:

qs alert-config-set --endpoints="~datadog:https://api.datadoghq.com#aAbBcCdDeEfF0123456789aAbBcCdDeE"

Send a test alert with qs alert-raise. Its --alert-severity defaults to info, which never reaches Datadog, so always set it to warning, error or critical:

qs alert-raise --title="Datadog test" --message="Testing the Datadog endpoint" --alert-severity=warning

Troubleshooting

Symptom	Cause and fix
No event arrives after a test	Check the test severity first: Info alerts are never sent. Then confirm the appliance can reach the Datadog API host over HTTPS, directly or through its configured proxy.
Datadog returns 403	The API key usually belongs to a different Datadog site than the URL host. Use the API host for the site your organization is on.
Datadog rejects the key	Make sure you used an API key, not an Application key.
An alert cleared in QuantaStor still shows in Datadog	Expected. QuantaStor sends nothing when an alert is cleared, and Datadog events cannot be closed.
Error and Critical alerts look the same in Datadog	Datadog has no Critical status. Filter on the `severity` tag instead.

To see what QuantaStor sent, check the service log at /var/log/qs/qs_service.log on the appliance; the API key appears there masked.

Related pages

Verified against QuantaStor 7.0.0.