Audit Logging

From OSNEXUS Online Documentation Site
Revision as of 20:46, 28 September 2026 by Qadmin (talk | contribs) (New page: RFC 5424 tamper-evident audit logging (QuantaStor 7.0). Record format, integrity chain and qs audit-log-verify verified on a branch build; sample records are real, anonymized)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search


QuantaStor records every management operation, login attempt and authorization decision in a tamper-evident audit log on each appliance. This page covers what is logged, the RFC 5424 record format, the configuration settings, how the signature chain works and how to verify it, forwarding the log to a SIEM, and the tools for reading it.

Note: The RFC 5424 record format, failed-login auditing and the tamper-evidence signatures are available starting with QuantaStor 7.0. Earlier releases write the audit log as JSON lines with no signatures; upgrade every appliance to 7.0 or newer before relying on the features described here.

Section Purpose
What is logged The record classes (MSGIDs) and what produces each one
Record format The RFC 5424 header, priority, and the structured-data elements
Syslog facility settings Changing the facility codes written into each record
Credential redaction Which argument values are masked before they reach the log
Log file location and rotation Where the log lives and how it is rotated
Tamper evidence The HMAC-SHA256 chain, the signing key, and verification results
Forwarding to a SIEM Sending records off the appliance over RFC 5425 TLS
Audit Log Analyzer Browsing recent records in the web interface
Audit logging from the CLI qs audit-log-list, qs audit-log-verify and qs-audit

What is logged

Audit logging is always on. Every API, CLI and web interface operation passes through the QuantaStor core service, which writes one record per event to the audit log of the appliance that handled it. Each record carries an RFC 5424 MSGID naming its class:

MSGID Facility Written when
api audit Any management API call is made, including list and get calls. The record names the API and every argument passed to it (see Credential redaction). It is written after the caller authenticates and before the authorization check, so a call that is later refused still has an api record followed by an authz record.
auth auth A login succeeds or fails. Failures include an unknown user name, a wrong password, a login refused because the account is locked, and directory (LDAP/AD) authentication failures. Successful logins are recorded for operations that change something; successful authentication for list and get calls is not recorded, to keep polling out of the log.
authz auth An authenticated user is refused an operation by role based access control, including resource group restrictions.
account audit The password policy acts on an account: auto-locking an idle account, marking a password expired, locking or removing an expired temporary or emergency account.
task audit A task changes state. Repeated identical updates are written once.
alert audit An alert is raised. Repeated identical alerts are written once.
event audit Anything else the service audits, such as service startup.

Facility here means which of the two configurable facility codes the record uses; see Syslog facility settings.

Failed logins on polled APIs are rate-limited. Grid peers, the SNMP agent and monitoring pollers call a small set of internal APIs in a loop, and a misconfigured poller with a stale password would otherwise flood the log. For those internal APIs QuantaStor writes at most one wrong-password or locked-account record per user, client address and API every 5 minutes, and appends a count to the next record, for example (12 similar failures not logged in the previous 5 minutes). A stale session token presented to one of those internal APIs is not recorded at all: grid peers and pollers present one routinely after a service restart and then refresh it. Failed logins to every other API, including a bad token, are recorded individually.

Record format

Each record is one line in the RFC 5424 syslog format:

<PRI>1 TIMESTAMP HOSTNAME APP-NAME PROCID MSGID [STRUCTURED-DATA...] MSG
Field Content
PRI Facility × 8 + severity, per RFC 5424. With the default facilities an informational API record is <110>, a successful login <86>, and a failed login or authorization refusal <83>.
TIMESTAMP RFC 3339 in UTC with microseconds, for example 2026-09-22T18:56:29.412305Z
HOSTNAME The appliance host name
APP-NAME quantastor
PROCID Process ID of the core service
MSGID The record class, see What is logged
MSG A human-readable description. User names in it are wrapped in [[ ]] so the log-bundle scrubber can find them. The client IP address is not repeated in MSG; it is in the clientIp parameter.

Severity

Severity Code Used for
Critical 2 Critical alerts
Error 3 Failed logins, locked-account rejections, authorization refusals, error alerts
Warning 4 Failed tasks, warning alerts
Notice 5 Service events logged at the security level
Informational 6 API calls, successful logins, tasks, informational alerts and events

Structured data

The standard origin and meta elements lead every record, so a record stands on its own once forwarded or rotated. The QuantaStor elements are private SD-IDs under the OSNEXUS IANA Private Enterprise Number 39324, the same number the QuantaStor SNMP MIB is registered under.

Element Parameters Present on
origin software (QuantaStor), swVersion, enterpriseId (39324) every record
meta sequenceId -- increments per record, restarts at 1 when the service restarts every record
qsAudit@39324 eventId (unique identifier of the record), user, clientIp, thread; outcome (success or failure) on auth and authz records; method (the API name) on api records; function on auth, authz, account and event records every record
qsApi@39324 One parameter per API argument, named after the argument. Empty arguments are recorded too, so the call can be reconstructed. api
qsTask@39324 taskId, sysOrigin, state, progress, operation, objectType, name; the task description is in MSG task
qsAlert@39324 alertId, objectId, objectType, severity, title, type, storageSystem; the alert description is in MSG alert
qsIntegrity@39324 line, prev, kid, hash -- see Tamper evidence. Always the last element. every record written while the signing key is available

Parameters with empty values are omitted, except in qsApi@39324 and qsIntegrity@39324. An argument value longer than 4096 characters, or a MSG longer than 16384 characters, is cut and ends in ...[truncated].

A signed API record looks like this (hash values shortened, line breaks added for readability; the file holds one record per line):

<110>1 2026-09-28T20:24:28.775961Z qs-node-185 quantastor 6856 api
  [origin software="QuantaStor" swVersion="7.0.0" enterpriseId="39324"]
  [meta sequenceId="5"]
  [qsAudit@39324 eventId="66d6a937-1487-e566-c8f7-6b4a6e6d6757" user="admin" clientIp="192.0.2.10" thread="757fb6c0" method="osn__auditLogVerify"]
  [qsApi@39324 storageSystem="" logFile="" entryCount="5" flags="0"]
  [qsIntegrity@39324 line="50" prev="dc17fb...4950" kid="47708162" hash="3c12e6...d8f7"]
  API osn__auditLogVerify invoked

A failed login by an unknown user name, the record that follows it in the same log (its prev is the hash above):

<83>1 2026-09-28T20:36:05.279123Z qs-node-185 quantastor 6856 auth
  [origin software="QuantaStor" swVersion="7.0.0" enterpriseId="39324"]
  [meta sequenceId="6"]
  [qsAudit@39324 eventId="6d03dea5-1fe1-7a87-491d-dfac3e821535" user="jdoe" clientIp="192.0.2.10" thread="757fb6c0" outcome="failure" function="authenticate"]
  [qsIntegrity@39324 line="51" prev="3c12e6...d8f7" kid="47708162" hash="a66f50...ec26"]
  Failed to authenticate user [[]] via [[jdoe]] attempting to access 'osn__storageSystemEnum', invalid user name.

Syslog facility settings

Two settings in the [system] section of /etc/quantastor.conf choose the facility code written into each record's PRI:

Setting Default Applies to
audit_log_auth_facility 10 (authpriv) auth and authz records
audit_log_facility 13 (log audit) every other record

Both ship commented out, so the defaults apply. An upgraded appliance keeps its existing /etc/quantastor.conf, so the lines may be absent there; add them to the [system] section to change a facility. Values are RFC 5424 facility codes, 0 to 23; a value above 23 is ignored and the default is used. Change them only if your collector routes on facility and expects something else -- for example 4 (auth) for authentication records, or one of 16 to 23 (local0 to local7) to put all QuantaStor records in their own stream. To change them, uncomment and edit the lines, then restart the QuantaStor service; the values are read when the service starts.

The facility only changes the PRI value in the log file and in anything forwarded from it. QuantaStor writes the audit log directly and does not send records to the local syslog daemon.

Credential redaction

QuantaStor masks secret values before an API record is written. An argument's value is replaced with ******** when its name:

  • ends in password, passphrase, secret or secretkey, or contains privatekey, ignoring case, or
  • is one of appKey, appToken, authRequestCode, clientKey, integrationKey, keyData or newIntegrationKey.

The match is on the end of the name, so policy settings such as minDaysBetweenPasswordChange are still recorded. An empty value stays empty, so the log still shows whether a secret was supplied. The same masking applies to the service debug trace, and failed-login records never contain the password that was tried.

This matters because the audit log is readable by every local account on the appliance.

Separately, when you send a log bundle to support, the bundle's copy of the audit log has its user and clientIp values replaced with scrubbed. See Send System Log Report.

Log file location and rotation

Each appliance writes its own log:

/var/log/qs/qs_audit.log

In a grid, each appliance writes the records for the operations it handled, so collecting or verifying the audit trail means doing it on every appliance.

QuantaStor rotates the log with logrotate using /etc/logrotate.d/qs_audit_logrotate.conf:

Setting Value Effect
size 5G rotate when the log exceeds 5 GB
rotate 20 keep 20 rotated files, then delete the oldest
copytruncate copy the log, then truncate it in place, so the file a reader has open never changes
compress, delaycompress gzip rotated files, except the most recent one
missingok no error if the log does not exist

Rotated files are qs_audit.log.1 (uncompressed), then qs_audit.log.2.gz through qs_audit.log.20.gz. The package installer rewrites this file on every upgrade, so a local change to it does not survive an upgrade. If you need to keep audit records for longer than 20 rotations, forward them off the appliance.

After an upgrade to 7.0, rotated files from before the upgrade still hold the older JSON-line records. The Audit Log Analyzer, qs audit-log-list and qs-audit read both formats; qs audit-log-verify reports the older lines as UNSIGNED.

Tamper evidence

Every record carries a qsIntegrity@39324 element that signs it and links it to the record before it:

Parameter Content
line A counter that increases by one per record. It keeps counting across rotation and service restarts.
prev The hash of the previous record, or genesis for the first record of a chain
kid A short identifier of the key that signed the record (8 hex digits derived from the key)
hash HMAC-SHA256 of the whole record, computed with hash set to empty

Changing any byte of a record breaks its hash. Deleting, inserting or reordering records breaks the line sequence and the prev link. Because the verifier continues the chain from each record's stored values, damage stays localized: one altered record shows as one tampered record, not as everything after it.

The signing key

The key is a 256-bit value from the FIPS random number generator, stored as 64 hex characters in:

/var/opt/osnexus/quantastor/qs_audit_hmac.key

The service creates it with mode 0600 (readable by root only) the first time it writes an audit record and finds no key. Each appliance has its own key, so each appliance's log can only be verified on that appliance.

If the key file exists but cannot be read or is not a 256-bit hex key, the service writes records unsigned and logs Audit log records will not be signed with the reason in the service log.

Verifying the log

Run qs audit-log-verify (see below). It checks every record in the file and prints each listed record prefixed with its result, then a summary:

Result Meaning
VERIFIED The record's hash is correct and it follows the previous record in the chain
TAMPERED(reason) The record fails a check. The reason is one of: record contents do not match its hash; chain broken, the previous record was altered or replaced; N record(s) missing before this one; out of order or duplicated, line X follows line Y; not a well-formed RFC 5424 record
UNSIGNED The record has no signature: an older JSON-line record from before the upgrade, or a record written while the key was unavailable
UNVERIFIABLE(reason) The record was signed with a different key than the one the appliance holds now (signed with key K, current key is K2), so it cannot be checked

When you verify the live log, QuantaStor also compares the end of the file with the last line number it wrote. If records are missing from the end, it adds a final line such as TAMPERED(3 record(s) missing from the end of the file, the service has written through line 1237).

The summary ends with an overall result:

qs_audit.log: 1237 record(s), 1237 verified, 0 tampered, 0 unsigned, 0 unverifiable; signed lines 1 to 1237. RESULT: VERIFIED

RESULT: TAMPERED means at least one record failed. RESULT: VERIFIED means no record failed and at least one verified. RESULT: NO SIGNED RECORDS means nothing in the file could be checked. Read the counts as well as the result: RESULT: VERIFIED does not exclude UNSIGNED or UNVERIFIABLE records, and a file that should be fully signed but has either deserves investigation.

What the signatures do not protect against

The chain makes tampering detectable by anyone who does not hold the key. Know its limits before you rely on it:

  • Root can re-sign. Anyone with root access to the appliance can read the key and rewrite the log with a valid chain. The protection against a root-level attacker is a copy of the records off the appliance, made as they are written -- see Forwarding to a SIEM.
  • Records removed from the start of a file are not detected. The verifier accepts the first record of a file as the anchor of its chain, so cutting the oldest records out of a file leaves a file that verifies.
  • Each file is verified on its own. The chain does run across rotation -- the first record of qs_audit.log carries the prev hash of the last record of qs_audit.log.1, and its line is one higher -- but qs audit-log-verify does not check that link. To check it, compare the line values at the boundary yourself.
  • Truncation of the end of a file is detected only for the live log, and only until the service restarts. The service remembers the last line number it wrote; after a restart it continues from whatever the file ends with. Records cut from the end of a rotated file are not detected.
  • Deleting the key or the log starts a new chain. If the key file is removed, the service creates a new key: records signed with the old key become UNVERIFIABLE and new records start again at line 1. If the log and its rotations are removed, the chain restarts at line 1 and the new file verifies.
  • Reinstalling the appliance loses the key unless you keep a copy of /var/opt/osnexus/quantastor/qs_audit_hmac.key. Anyone holding a copy can re-sign the log, so protect it as you would any other secret.

Forwarding to a SIEM

QuantaStor ships an example rsyslog configuration that forwards the audit log to a remote collector. It is not active: you install and edit it yourself, on every appliance.

/opt/osnexus/quantastor/conf/qs_audit_rsyslog_forward.conf.example

The example uses rsyslog's file input to follow /var/log/qs/qs_audit.log and forwards each line unchanged: the records are already RFC 5424, so it does not wrap them in a second syslog header. The transport is RFC 5425 syslog over TLS on port 6514 with RFC 6587 octet-counted framing, and a disk-assisted queue holds records while the collector is unreachable and keeps them across an rsyslog restart. Because the log is truncated in place on rotation, the file input is configured to reopen it when that happens.

To enable forwarding on an appliance:

  1. Copy the example into place: cp /opt/osnexus/quantastor/conf/qs_audit_rsyslog_forward.conf.example /etc/rsyslog.d/60-qs-audit-forward.conf
  2. Edit /etc/rsyslog.d/60-qs-audit-forward.conf: set Target and StreamDriverPermittedPeers to your collector's host name, and DefaultNetstreamDriverCAFile to the CA certificate that signed the collector's certificate.
  3. Install rsyslog's TLS module, package rsyslog-gnutls. It is not in the QuantaStor package repository that appliances are configured to use, so apt-get install rsyslog-gnutls finds no candidate on a standard appliance; contact OSNEXUS support to obtain it, or forward over plain TCP to a collector on a trusted network as described below.
  4. Restart rsyslog: systemctl restart rsyslog
  5. Repeat on every appliance in the grid.

If your collector does not accept TLS, remove the StreamDriver lines and set Port="514" for plain TCP. Use Protocol="udp" only if the collector cannot accept TCP: UDP receivers may truncate records longer than the path MTU (RFC 5426), and a long API record easily exceeds it.

The collector receives the records with their qsIntegrity@39324 elements intact. It cannot check the HMAC without the appliance's key, but a forwarded copy is the reference you compare the appliance's log against if you suspect it was rewritten.

Audit Log Analyzer

Navigation: Storage Management → Storage Systems → Storage System (select + right-click) → Audit Log Analyzer

The Audit Log Analyzer lists the most recent audit records of one appliance. It is a quick view for recent activity, not a search tool: it reads the last 20 lines of the log and then applies the filters, so a filter shows only matches among those 20 records. For more history, use qs audit-log-list or qs-audit. The analyzer does not check signatures; use qs audit-log-verify for that.

The filter area at the top has:

  • Storage System -- the appliance whose log to read. Selecting a different system reloads the list.
  • Audit User Filter -- limit the list to one user, or <all users>. Selecting a user reloads the list.
  • Audit Entry Types -- check boxes for APIs, Tasks, Alerts and Events, all checked by default. auth, authz, account and event records are listed under Events. Changing a check box does not reload the list; click Rescan.

Rescan, in the header of the Analysis Results pane, reloads the list. The list shows User ID, Type, Status, Client IP Address, Time Stamp and Message for each record, and the User ID, Type, Status, Time Stamp and Message column menus have filters for narrowing the list further.

Selecting a record shows its details on the right:

  • API Arg Details -- for an API record, each argument and its value, then the Method.
  • Task Details -- for a task record: Start Time, Status, User, Storage System and Detail.
  • Alert Details -- for an alert record: Time Stamp, Storage System, Alert and Description.

Viewing the audit log requires permission to modify Storage Systems.

Audit logging from the CLI

Both qs commands require permission to modify Storage Systems, the same as the Audit Log Analyzer. Each takes --storage-system to act on another appliance in the grid; the request is redirected to that appliance, which reads its own log.

qs audit-log-list

qs audit-log-list (short form qs audit-list) lists audit records:

Argument Meaning
--max-entries How many of the most recent log lines to read. Default 100, maximum 10000.
--storage-system The appliance whose log to read. Default: the appliance you are connected to.
--audit-user Only records for this user, given by user ID.
--audit-entry-types Only records of this type: api, task, alert or message. message covers auth, authz, account and event records.

As in the Audit Log Analyzer, the filters are applied to the lines read, so --max-entries=1000 --audit-entry-types=message returns the matching records among the last 1000 lines, not the last 1000 matching records.

qs audit-log-verify

qs audit-log-verify (short form qs audit-verify) checks the signature chain of one log file, as described in Verifying the log:

Argument Meaning
--log-file The file in /var/log/qs to verify: qs_audit.log (the default) or a rotation such as qs_audit.log.1 or qs_audit.log.2.gz. Only these names are accepted; the command does not read other files.
--max-entries How many of the last records to print. Default 100, maximum 10000. Every record is checked regardless; records that fail are always printed as well, up to 1000 of them.
--storage-system The appliance whose log to verify. Default: the appliance you are connected to.
qs audit-log-verify
qs audit-log-verify --log-file=qs_audit.log.3.gz --max-entries=10

The output is plain text: one line per listed record, the result first and the raw record after it, then a blank line and the summary. Scripts can test the summary for RESULT: VERIFIED, RESULT: TAMPERED or RESULT: NO SIGNED RECORDS.

To verify an appliance's whole audit trail, run the command once for qs_audit.log and once for each rotated file.

qs-audit

The qs-audit tool, run in an SSH session on the appliance, reads the log file directly and prints records as JSON with colour highlighting. It is the quickest way to search further back than the Audit Log Analyzer, and it can turn a recorded API call back into the CLI command or REST call that reproduces it.

qs-audit -n 50              # the last 50 API records
qs-audit -v -n 50           # the last 50 records of every type
qs-audit -f                 # follow the log as records are written
qs-audit --filter=pool-create --filter-apis -c   # pool-create calls, with the equivalent CLI call
qs-audit --file=/var/log/qs/qs_audit.log.1 -a    # every API record in a rotated file

By default qs-audit shows API records only; add -v to include logins, tasks, alerts and events. Other options: --raw prints the log lines unconverted, -u and -j print an equivalent curl call, and --no-color turns off highlighting. Run qs-audit --help for the full list.

qs-showlog -audit prints the last 200 records the same way, and qs-showlog -audit -f follows the log.

Related pages


Verified against QuantaStor 7.0.0.