QuantaStor Custom Scripting / Application Extensions: Difference between revisions
m Rewrite from the product: all 24 service call-outs enumerated from the code with verified arguments, sync/async and 5m timeout semantics observed live, root/cwd/env of the call-out environment documented, exit codes shown to be logged-not-acted-on; corrects the claim that the directory ships 0755 (it ships 0777 - QSTOR-12443), that system-prestop fires (it has no call site - QSTOR-12445) and that system-poststart runs only once per boot (it runs every service start); flags schedule-presnap/po... |
|||
| Line 1: | Line 1: | ||
[[Category:admin_guide]] | [[Category:admin_guide]] | ||
QuantaStor has script call-outs which you can use to extend the functionality of the system for integration with custom applications. A call-out is a shell script you place at a fixed path on the appliance; QuantaStor runs it as root at a defined point in a storage operation -- before a pool starts, after a share is created, around a snapshot schedule, either side of an HA failover -- and passes it the identity of the object being operated on. | |||
Because these scripts run as root and are triggered by ordinary storage activity, read [[#Securing the call-out directory|Securing the call-out directory]] before you install your first one. | |||
{| class="wikitable" | |||
! Section !! Purpose | |||
|- | |||
| [[#How call-outs work|How call-outs work]] || Naming, synchronous versus asynchronous, the environment scripts run in, what happens to a non-zero exit | |||
|- | |||
| [[#Securing the call-out directory|Securing the call-out directory]] || Ownership and permissions, and why a writable call-out directory is a root escalation path | |||
|- | |||
| [[#Storage System call-outs|Storage System call-outs]] || Service start, chassis beacon, system use notification | |||
|- | |||
| [[#Network port call-outs|Network port call-outs]] || Either side of bringing an interface up | |||
|- | |||
| [[#Storage Pool call-outs|Storage Pool call-outs]] || Start, stop, export, HA failover, scrub | |||
|- | |||
| [[#Network Share call-outs|Network Share call-outs]] || Share create and modify | |||
|- | |||
| [[#Snapshot and replication schedule call-outs|Snapshot and replication schedule call-outs]] || Quiesce and thaw around scheduled snapshots and replication | |||
|- | |||
| [[#DR failover call-outs|DR failover call-outs]] || Either side of Activate Checkpoints | |||
|- | |||
| [[#Software adapter call-outs|Software adapter call-outs]] || Either side of iSCSI and NVMe-oF logins | |||
|- | |||
| [[#Other extension points|Other extension points]] || Backup policy post-job handler, alternate pool import script, alert handlers | |||
|- | |||
| [[#Calling the QuantaStor API from a call-out|Calling the QuantaStor API from a call-out]] || Getting object detail back out of QuantaStor from inside your script | |||
|- | |||
| [[#Example script|Example script]] || A starting-point script that parses the arguments | |||
|- | |||
| [[#Troubleshooting a call-out that is not firing|Troubleshooting a call-out that is not firing]] || The log to read and the three things that are usually wrong | |||
|- | |||
| [[#Call-outs and upgrades|Call-outs and upgrades]] || What survives a package upgrade and the one name that does not | |||
|} | |||
== | == How call-outs work == | ||
=== Where scripts go, and what they are called === | |||
Every call-out lives in one directory and its file name is fixed. QuantaStor looks for the exact name; there is no registration step and no configuration file. | |||
<pre style="font-size: smaller"> | |||
= | |||
/var/opt/osnexus/custom | /var/opt/osnexus/custom | ||
</pre> | </pre> | ||
Each call-out has two possible forms, and the suffix decides how it runs: | |||
{| class="wikitable" | |||
! Form !! How it runs | |||
|- | |||
| {{Code|1=<hook>.sh}} || '''Synchronously.''' The operation that triggered it waits for the script to finish. Wrapped in {{Code|1=/usr/bin/timeout -k 15s 5m}}, so the script gets SIGTERM after five minutes and SIGKILL fifteen seconds after that. | |||
|- | |||
| {{Code|1=<hook>-async.sh}} || '''In the background.''' The operation does not wait, there is no timeout, and the script can outlive the operation entirely. | |||
|} | |||
If both forms exist QuantaStor runs both, starting the asynchronous one first and then the synchronous one. Use the asynchronous form for anything slow, and the synchronous form only when the work genuinely has to complete before the operation continues. | |||
A synchronous script blocks for its full runtime. A {{Code|1=poolscrub-prestart.sh}} that sleeps for 25 seconds makes {{Code|1=[[QuantaStor CLI Command Reference#pool-scrub-start|qs pool-scrub-start]]}} take 27 seconds instead of about one. Under HA failover and pool start, that delay is added to your recovery time, so keep synchronous scripts short. | |||
=== | === The environment scripts run in === | ||
Call-outs are executed by the QuantaStor service, so they inherit its environment and not a login shell's: | |||
==== / | {| class="wikitable" | ||
! !! Value | |||
|- | |||
| User || {{Code|1=root}} (uid 0), always | |||
|- | |||
| Working directory || {{Code|1=/opt/osnexus/quantastor/bin}} | |||
|- | |||
| {{Code|1=PATH}} || {{Code|1=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin}} | |||
|- | |||
| {{Code|1=HOME}}, {{Code|1=USER}} || empty | |||
|- | |||
| {{Code|1=TERM}} || {{Code|1=dumb}} | |||
|- | |||
| {{Code|1=LANG}} || {{Code|1=C.UTF-8}} | |||
|} | |||
There are only nine environment variables in total and none of your shell profile is loaded, so use absolute paths for every command and file your script touches, and do not rely on {{Code|1=$HOME}} or on anything a login would have set. Note in particular that {{Code|1=$HOME}} is an empty string, which breaks any tool configured through a path built from it. | |||
=== | === Exit codes are logged, not acted on === | ||
A non-zero exit from a synchronous call-out is recorded as a warning together with the script's standard error, and '''the operation continues regardless'''. A call-out cannot veto a pool start, a failover or a snapshot. A script that exits 42 during a pool scrub start produces this and the scrub proceeds: | |||
= | <pre style="font-size: smaller"> | ||
WARN custom_script_manager <the script's stderr, line by line> | |||
WARN custom_script_manager Custom script '/var/opt/osnexus/custom/poolscrub-prestart.sh' completed with error '42'. | |||
</pre> | |||
The practical consequence is that a broken call-out is invisible from the web interface and from the task list. Nothing turns red. The only place it shows up is the service log -- see [[#Troubleshooting a call-out that is not firing|Troubleshooting a call-out that is not firing]]. | |||
== | === Grid systems === | ||
A call-out runs on the node that is performing the operation, using that node's copy of the script. In a grid, install your scripts on '''every''' node that could own the object, or the call-out will fire on some failovers and pool moves and not others. | |||
=== | === Turning call-outs off === | ||
Call-outs are on whenever the directory exists. To disable the mechanism entirely -- for a support investigation, say -- the QuantaStor service accepts a {{Code|1=disable-custom-scripts-manager}} startup argument. We recommend contacting support before changing service startup arguments. | |||
== | == Securing the call-out directory == | ||
'''The call-out directory is a root execution path.''' Anything in it named after a hook is run as root the next time an ordinary, unprivileged storage event occurs -- a pool starting, a network port coming up, a snapshot schedule triggering, a share being created. Write access to that directory is therefore equivalent to root on the appliance, and it needs to be treated the way you would treat {{Code|1=/etc/sudoers.d}}. | |||
= | '''Check the directory permissions before you install anything.''' Current releases ship {{Code|1=/var/opt/osnexus/custom}} '''world-writable (mode 0777)''', which means any local account can plant a script there and have it run as root. This is a packaging defect, tracked as QSTOR-12443. Repair it, and repair the sibling directories that have the same problem: | ||
<pre style="font-size: smaller"> | |||
chown root:root /var/log/qs /var/opt/osnexus/custom /var/opt/osnexus/alerthandlers /var/opt/osnexus/quantastor | |||
chmod 755 /var/log/qs /var/opt/osnexus/custom /var/opt/osnexus/alerthandlers /var/opt/osnexus/quantastor | |||
</pre> | |||
Then set each script to mode 755 and root ownership, so that no non-root account can modify a script that root is going to execute: | |||
<pre style="font-size: smaller"> | |||
chown root:root /var/opt/osnexus/custom/*.sh | |||
chmod 755 /var/opt/osnexus/custom/*.sh | |||
</pre> | |||
If a script contains sensitive information -- a plain text password, an API token, a private key path -- use '''700''' rather than 755 so only root can read it. The service runs as root, so 700 does not stop the call-out from working. Better still, keep the secret in a separate root-only file and read it from the script. | |||
Two further points worth making: | |||
* '''Hardening survives an upgrade.''' A directory mode you set by hand is preserved across a package upgrade, so this is a one-time repair per appliance rather than something to redo each release. | |||
* '''Anyone who can write a call-out owns the appliance.''' That includes any service account with write access to the directory and any process running as a non-root user that can reach it. If you are auditing privilege escalation paths on a QuantaStor system, this directory belongs on the list along with the alert handler directory. | |||
See [[Security Configuration]] for the appliance's wider security controls. | |||
== Storage | == Storage System call-outs == | ||
{| class="wikitable" | |||
the | ! Script !! Fires !! Arguments | ||
|- | |||
| {{Code|1=system-poststart.sh}} || Every time the QuantaStor service finishes starting || {{Code|1=--firstboot}}, only when the appliance itself has rebooted since the previous start; otherwise no arguments | |||
|- | |||
| {{Code|1=system-ident-start.sh}} || Before the Storage System beacon task begins its LED blink and audible beep pattern || none | |||
|- | |||
| {{Code|1=system-ident-stop.sh}} || After the beacon task completes || none | |||
|- | |||
| {{Code|1=security-message-update.sh}} || When a non-empty System Use Notification is applied || {{Code|1=--message="TEXT"}} | |||
|} | |||
=== system-poststart.sh === | |||
''' | This script runs on '''every''' service start, including a plain service restart with no reboot. What changes between the two cases is the argument: QuantaStor compares the current boot time against the stamp it keeps at {{Code|1=/var/opt/osnexus/quantastor/qs_lastreboot}} and passes {{Code|1=--firstboot}} only when the appliance has actually rebooted. Branch on that argument if you want work that happens once per boot rather than once per service start. | ||
== | {{Code|1=system-poststart-async.sh}} is '''reserved by the product''' and is not available to use. QuantaStor installs its own Ceph OSD activation script at that name as a symlink, and the package replaces whatever is there on every install and upgrade, silently. Use the synchronous {{Code|1=system-poststart.sh}} instead, or start your own background work from it. This collision is tracked as QSTOR-12444. | ||
=== system-prestop.sh === | |||
'''This call-out does not currently fire.''' The hook is implemented in the service but nothing invokes it, so a script installed at this name never runs on a shutdown or restart, whether initiated from the web interface, the CLI or the console. Tracked as QSTOR-12445. If you need work done before a controlled shutdown, drive it from your own orchestration rather than from this call-out. | |||
=== security-message-update.sh === | |||
Called with the System Use Notification text configured under [[Security Configuration]], so you can mirror the same banner into other subsystems -- a GNOME classification banner, for example, or a login screen you manage yourself. The text is sanitized before it reaches your script: double quotes become single quotes, backslashes become forward slashes, and non-ASCII characters are removed. The call-out is not run when the notification is cleared; QuantaStor removes {{Code|1=/etc/issue.notice}} instead. | |||
== Network port call-outs == | |||
==== | {| class="wikitable" | ||
! Script !! Fires !! Arguments | |||
|- | |||
| {{Code|1=port-prestart.sh}} || Immediately before QuantaStor brings a port up || {{Code|1=--name=PORTNAME}} | |||
|- | |||
| {{Code|1=port-poststart.sh}} || After the port is up and its configuration has been reconciled || {{Code|1=--name=PORTNAME}} | |||
|} | |||
{{Code|1=port-prestart.sh}} can fire more than once for a single port activation. When the port is a bond, QuantaStor cycles the slave interfaces first and calls the script around that step as well as around the bond itself. Write the script so that running it twice in a row is harmless. | |||
== Storage Pool | == Storage Pool call-outs == | ||
These are the call-outs to use when an application on the appliance has to attach to or detach from a pool, or from a directory inside one, in step with the pool's lifecycle. All of them take the pool's UUID. | |||
{| class="wikitable" | |||
! Script !! Fires !! Arguments | |||
|- | |||
| {{Code|1=pool-poststart.sh}} || After a pool has been started || {{Code|1=--pool=POOLUUID}} | |||
|- | |||
| {{Code|1=pool-prestop.sh}} || Before a pool is stopped || {{Code|1=--pool=POOLUUID}} | |||
|- | |||
| {{Code|1=pool-preexport.sh}} || Before a pool export begins || {{Code|1=--pool=POOLUUID}} | |||
|- | |||
| {{Code|1=pool-postexport.sh}} || After the export finishes, whether or not it succeeded || {{Code|1=--pool=POOLUUID}} | |||
|- | |||
| {{Code|1=pool-prefailover.sh}} || Before an HA failover of the pool starts || {{Code|1=--pool=POOLUUID}} | |||
|- | |||
| {{Code|1=pool-postfailover.sh}} || After the failover completes, whether or not it succeeded || {{Code|1=--pool=POOLUUID}} | |||
|- | |||
| {{Code|1=poolscrub-prestart.sh}} || Before a pool scrub is started || {{Code|1=--name=POOLNAME --id=POOLUUID}} | |||
|- | |||
| {{Code|1=poolscrub-poststart.sh}} || Just after the scrub has been started || {{Code|1=--name=POOLNAME --id=POOLUUID}} | |||
|} | |||
Points worth knowing: | |||
* '''The post script always runs.''' Export and failover are paired so that the {{Code|1=post}} call-out fires even when the operation throws partway through. Write {{Code|1=pool-postexport.sh}} and {{Code|1=pool-postfailover.sh}} as cleanup that must be safe on both the success and failure paths, and do not treat being called as proof the operation worked -- query the pool state if you need to know. | |||
* '''{{Code|1=poolscrub-poststart.sh}} fires when the scrub has been ''started'', not finished.''' A scrub runs for hours; the call-out returns as soon as the scrub is under way. There is no scrub-completion call-out. | |||
* '''{{Code|1=pool-poststart.sh}} also covers CephFS pools''', firing when the pool is mounted on its owning node. {{Code|1=pool-prestop.sh}} fires for every pool stop regardless of pool type. The scrub call-outs are ZFS-only, because a scrub request against a CephFS pool is rejected before they are reached -- Ceph scrubbing is driven by policy on the underlying OSDs instead. | |||
* Keep {{Code|1=pool-prefailover.sh}} and {{Code|1=pool-poststart.sh}} fast if they are synchronous. Both sit directly in the HA recovery path -- see [[HA Cluster Setup (JBODs)]]. | |||
== Network Share | == Network Share call-outs == | ||
{| class="wikitable" | |||
! Script !! Fires !! Arguments | |||
|- | |||
| {{Code|1=share-postcreate.sh}} || After a new Network Share has been created and its client access entries written || {{Code|1=--share-path=PATH --share-name=NAME --share-id=SHAREUUID --pool=POOLUUID}} | |||
|- | |||
| {{Code|1=share-postmodify.sh}} || After a Network Share has been modified || {{Code|1=--share-path=PATH --share-name=NAME --share-id=SHAREUUID --pool=POOLUUID}} | |||
|} | |||
The usual reason to use these is to stamp house defaults onto a share that the dialog does not cover -- POSIX permissions, ACLs, extended attributes, an SELinux label, a directory skeleton. {{Code|1=--share-path}} is the full filesystem path, so the script can act on it directly. See [[Network Shares]] for the share settings themselves. | |||
== Snapshot and replication schedule call-outs == | |||
These exist so that a database or application can be quiesced before a scheduled snapshot and released afterwards. QuantaStor takes the snapshots of all the Storage Volumes and Network Shares in a schedule as a consistency group rather than one at a time, so the quiesce window your script has to hold open is short. | |||
{| class="wikitable" | |||
! Script !! Fires !! Arguments | |||
|- | |||
| {{Code|1=schedule-prestart.sh}} || When a snapshot or replication schedule begins a run, before any per-member work || {{Code|1=--name=SCHEDULE_NAME --id=SCHEDULE_ID}} | |||
|- | |||
| {{Code|1=schedule-presnap.sh}} || After all snapshot tasks are staged and immediately before they are released as a group || {{Code|1=--name=SCHEDULE_NAME --id=SCHEDULE_ID}} | |||
|- | |||
| {{Code|1=schedule-postsnap.sh}} || After every snapshot in the group has been taken, before replication transfers begin || {{Code|1=--name=SCHEDULE_NAME --id=SCHEDULE_ID}} | |||
|} | |||
= | {{Code|1=schedule-prestart.sh}} fires for both [[Snapshot Schedules|snapshot schedules]] and [[Remote-replication (DR)|replication schedules]] and is the reliable one of the three. It is called early enough to do preparation work, but not tightly enough around the snapshot itself to serve as a freeze point. | ||
'''{{Code|1=schedule-presnap.sh}} and {{Code|1=schedule-postsnap.sh}} do not fire for schedules whose members are on a ZFS pool.''' The snapshot schedule takes a separate code path for ZFS volumes and shares and returns before reaching the point where these two call-outs are invoked. Because virtually all snapshot schedules contain ZFS members, in practice these hooks are not usable today. This is tracked as QSTOR-12378. Do your quiesce work from {{Code|1=schedule-prestart.sh}} until it is resolved, and be aware that the window between {{Code|1=schedule-prestart.sh}} and the snapshots being taken is wider than the presnap window would have been. | |||
Both call-outs are checked for existence before QuantaStor attempts them, so a schedule run logs nothing about them if you have not installed them. | |||
== DR failover call-outs == | |||
== | {| class="wikitable" | ||
! Script !! Fires !! Arguments | |||
|- | |||
| {{Code|1=dr-prefailover.sh}} || At the start of the Activate Checkpoints task on a replication schedule || {{Code|1=--schedule=SCHEDULE_ID}} | |||
|- | |||
| {{Code|1=dr-postfailover.sh}} || After the checkpoints are activated, the aliases are created and the CIFS and NFS configurations have been rewritten || {{Code|1=--schedule=SCHEDULE_ID}} | |||
|} | |||
Activating checkpoints is what turns a DR site's replicas into live, servable shares and volumes, so {{Code|1=dr-postfailover.sh}} is the point at which to repoint an application, update DNS, or start services that depend on the DR copy. See [[Remote-replication (DR)]] for the failover procedure these bracket. | |||
= | Both are subject to the five-minute synchronous timeout, which matters more here than elsewhere: a DR failover is exactly when someone is watching the clock. Use the {{Code|1=-async.sh}} form for anything that involves waiting on an external system. | ||
== Software adapter call-outs == | |||
QuantaStor's software adapter feature connects the appliance to other systems as an initiator over iSCSI and NVMe-oF (TCP and RDMA). These call-outs bracket the login step so that anything that has to be orchestrated alongside the connection can be. | |||
==== | {| class="wikitable" | ||
! Script !! Fires !! Arguments | |||
|- | |||
| {{Code|1=swadapter-preconnect.sh}} || Before logging in to the adapter's target devices || {{Code|1=--adapter=ADAPTERNAME --id=ADAPTERUUID --ip-address=IPADDRESS}} | |||
|- | |||
| {{Code|1=swadapter-postconnect.sh}} || After the logins have been issued || {{Code|1=--adapter=ADAPTERNAME --id=ADAPTERUUID --ip-address=IPADDRESS}} | |||
|} | |||
These fire on a boot-time login, on a login you force, and on a repair pass that finds a target to repair -- not on every adapter scan. A steady-state system with all targets connected calls neither script. | |||
== Other extension points == | |||
= | Three more places let you supply your own code. They do not use the call-out mechanism described above -- different directory conventions, different argument styles, no {{Code|1=-async}} variants -- so treat them separately. | ||
=== Backup policy post-job handler === | |||
A [[Backup Policies|Backup Policy]] job runs your script after the transfer completes, if it exists: | |||
<pre style="font-size: smaller"> | |||
/var/opt/osnexus/custom/post-backupjob.sh | |||
</pre> | |||
It is invoked by the backup job itself rather than by the service, with short-form arguments and no timeout wrapper: | |||
{| class="wikitable" | |||
! Argument !! Meaning | |||
|- | |||
| {{Code|1=-p POLICYNAME}} || Name of the Backup Policy as it appears in the web interface | |||
|- | |||
| {{Code|1=-b BACKUPDIR}} || Directory the backup was written to | |||
|- | |||
| {{Code|1=-m SOURCEMOUNTDIR}} || Read-only source directory that was backed up | |||
|- | |||
| {{Code|1=-s SUBPATH}} || Sub-path within the remote storage mount | |||
|- | |||
| {{Code|1=-l BACKUPLOG}} || List of files that were backed up; passed only when the policy is configured to create a backup log | |||
|} | |||
= | A working example ships on every appliance and is the best starting point -- copy it, drop the {{Code|1=.example}} suffix, and replace the body: | ||
<pre style="font-size: smaller"> | |||
/var/opt/osnexus/custom/post-backupjob.sh.example | |||
</pre> | |||
Its second worked example is genuinely useful in its own right: it prunes the empty directories a file-selection filter leaves behind in the backup target, guarding the operation with a hold file so a recursive {{Code|1=rmdir -p}} cannot delete the backup directory itself. | |||
=== Alternate pool import script === | |||
== | If a script exists at the path below, QuantaStor calls it '''instead of''' running its own {{Code|1=zpool import}} when starting a ZFS pool, passing the zpool name and adding {{Code|1=-f}} when the import is being forced: | ||
<pre style="font-size: smaller"> | |||
/var/opt/osnexus/custom/qs_poolimport.sh | |||
</pre> | |||
This is an override, not a call-out: get it wrong and the pool does not import. It exists for unusual device-path and multipath situations, and it bypasses the cache-file and device-path handling QuantaStor normally applies. We recommend contacting support before using it. | |||
=== Alert handlers === | |||
Alerts have their own extension mechanism -- a handler script in {{Code|1=/var/opt/osnexus/alerthandlers}}, registered through a configuration file, with per-vendor examples shipped as working reference implementations. It is documented under [[Call-home / Alerting#Custom Alert Handlers|Custom Alert Handlers]] on the [[Call-home / Alerting]] page. The security guidance in [[#Securing the call-out directory|Securing the call-out directory]] applies to that directory too, and for the same reason. | |||
== | == Calling the QuantaStor API from a call-out == | ||
Most call-outs are handed a UUID rather than the object itself, so a useful script generally starts by asking QuantaStor for the detail. The {{Code|1=qs}} CLI is installed on the appliance and works from inside a call-out: | |||
<pre style="font-size: smaller"> | |||
qs pool-get --pool=$POOLID --server=127.0.0.1,admin,PASSWORD --json | |||
</pre> | |||
Note the argument form: it is {{Code|1=[[QuantaStor CLI Command Reference#pool-get|qs pool-get]] --pool=<id>}}, not a positional UUID. Every command's exact argument list is in the [[QuantaStor CLI Command Reference]], and {{Code|1=qs help --min}} lists the full command set on the appliance itself. {{Code|1=--server}} and {{Code|1=--json}} are global to the CLI rather than arguments of a particular command, so they can be added to any of them. | |||
Three things to get right when scripting the CLI: | |||
* '''Supply credentials explicitly.''' With no server argument the CLI falls back to the default administrator account, which will not work on a system whose administrator password has been changed. Pass a comma-separated address, user and password with no spaces, or export the equivalent as {{Code|1=QS_SERVER}} inside the script. Do not rely on {{Code|1=~/.qs.cnf}} -- {{Code|1=HOME}} is empty in the call-out environment. | |||
* '''Any script holding a password must be mode 700.''' See [[#Securing the call-out directory|Securing the call-out directory]]. | |||
* '''Ask for JSON or XML output for anything you parse.''' The default table output is formatted for people and its column widths change. | |||
For integration from outside the appliance, QuantaStor exposes the same operations over REST; see the [[REST API Reference Guide]]. Call-outs and the REST API complement each other -- the call-out tells your application that something happened, and the API lets it find out what. [[QuantaStor Shell Utilities]] covers the {{Code|1=qs-}} helper commands that are often more convenient than a raw CLI call. | |||
== Example script == | |||
A starting point that parses the two most common argument styles and does nothing else. Copy it to the call-out name you need, make it root-owned and mode 755, and put your work where the comment is. | |||
<pre style="font-size: smaller"> | |||
#!/usr/bin/env bash | |||
# QuantaStor custom call-out skeleton. | |||
# Environment note: runs as root, cwd /opt/osnexus/quantastor/bin, no HOME, | |||
# minimal PATH. Use absolute paths. | |||
= | set -u | ||
LOGFILE=/var/log/qs/my-callout.log | |||
POOLID="" | |||
SCHEDULE_NAME="" | |||
SCHEDULE_ID="" | |||
while [ $# -gt 0 ]; do | while [ $# -gt 0 ]; do | ||
case "$1" in | case "$1" in | ||
--pool | --pool=*) POOLID="${1#*=}" ;; | ||
--id=*) SCHEDULE_ID="${1#*=}" ;; | |||
--schedule=*) SCHEDULE_ID="${1#*=}" ;; | |||
--name=*) SCHEDULE_NAME="${1#*=}" ;; | |||
-- | *) echo "ignoring unrecognized argument '$1'" >&2 ;; | ||
*) | |||
esac | esac | ||
shift | shift | ||
done | done | ||
{ | |||
echo "pool | echo "[$(/bin/date -Is)] called as $0" | ||
echo " pool=$POOLID name=$SCHEDULE_NAME id=$SCHEDULE_ID" | |||
} >> "$LOGFILE" 2>&1 | |||
## | ## Put your custom work here. Keep it short if this is the synchronous form: | ||
## the operation that called you is waiting, and you have five minutes. | |||
exit 0 | exit 0 | ||
</pre> | |||
Ignoring an unrecognized argument rather than failing on it is deliberate. QuantaStor has added arguments to call-outs across releases, and a script that exits non-zero on an unexpected one produces a warning in the service log on every single invocation. | |||
== Troubleshooting a call-out that is not firing == | |||
'''Start with the service log.''' Every attempt is recorded, successful or not, tagged {{Code|1=custom_script_manager}}: | |||
<pre style="font-size: smaller"> | |||
grep custom_script_manager /var/log/qs/qs_service.log | |||
</pre> | |||
A successful synchronous run looks like this -- note that the command QuantaStor actually ran is quoted in full, including the {{Code|1=timeout}} wrapper and every argument, which is the fastest way to confirm what your script was handed: | |||
<pre style="font-size: smaller"> | |||
INFO custom_script_manager Running backgrounded custom script: '/usr/bin/timeout -k 15s 5m /var/opt/osnexus/custom/schedule-prestart.sh --name=nightly --id=67f0dc8a-...' | |||
INFO custom_script_manager Custom script '/var/opt/osnexus/custom/schedule-prestart.sh' completed. | |||
</pre> | </pre> | ||
The "backgrounded" wording is used for both forms and is wrong for the synchronous one (QSTOR-12446); tell them apart by the {{Code|1=timeout}} prefix, which only the synchronous form has, or by the trailing {{Code|1=&}}, which only the asynchronous form has. | |||
Then work through the usual causes: | |||
{| class="wikitable" | |||
! Symptom in the log !! Cause !! Fix | |||
|- | |||
| {{Code|1=completed with error '126'}} and {{Code|1=Permission denied}} || The file is not executable || {{Code|1=chmod 755 <script>}} | |||
|- | |||
| {{Code|1=completed with error '127'}} and {{Code|1=No such file or directory}}, but the file is plainly there || Windows line endings, so the interpreter named on the shebang line has a trailing carriage return; or a shebang pointing at an interpreter that is not installed || {{Code|1=dos2unix <script>}}, and check the shebang path exists | |||
|- | |||
| No line for the script at all || QuantaStor never found the file. Most often the name is wrong -- a missing {{Code|1=.sh}}, a hyphen where an underscore was typed, or the file left in place with a {{Code|1=.example}} or {{Code|1=.txt}} suffix || Compare the name against this page, character for character | |||
|- | |||
| No line for the script, name confirmed correct || The event you expected did not actually reach that call-out. Several hooks are narrower than they look -- see the notes in each section || Trigger the operation deliberately from the CLI and watch the log live | |||
|- | |||
| The script runs but behaves differently than from your shell || The call-out environment, not your login environment || Test with {{Code|1=env -i /var/opt/osnexus/custom/<script> <args>}} from {{Code|1=/opt/osnexus/quantastor/bin}} as root | |||
|} | |||
A missing script is a normal, expected condition -- QuantaStor logs it at debug level and carries on -- so do not read the absence of an error as evidence that your script ran. | |||
The most reliable way to prove a call-out fires at all is a stub that appends its arguments and a timestamp to a file, installed at the hook name, then triggering the operation from the CLI and reading the file. Once that works, replace the stub with the real thing. | |||
== Call-outs and upgrades == | |||
=== | Call-out scripts live under {{Code|1=/var/opt}} and are not owned by any package, so '''a QuantaStor upgrade leaves them in place''', along with any directory permissions you have tightened. Only two files in the directory belong to the product: {{Code|1=post-backupjob.sh.example}} and the {{Code|1=system-poststart-async.sh}} symlink. | ||
Two things to do after an upgrade all the same: | |||
# | * '''Re-check {{Code|1=system-poststart-async.sh}}''' if you were using that name. The package replaces it on every install; see [[#system-poststart.sh|system-poststart.sh]] above. | ||
* '''Re-read this page's argument lists.''' Arguments have been added to call-outs between releases, and a script written against an older set still runs -- it just ignores information it could be using. | |||
== Related pages == | |||
# | * [[Call-home / Alerting]] -- alert delivery, ITSM integrations, and the [[Call-home / Alerting#Custom Alert Handlers|custom alert handler]] mechanism | ||
* [[Snapshot Schedules]] -- the schedules that drive {{Code|1=schedule-prestart.sh}} | |||
* [[Remote-replication (DR)]] -- replication schedules and the Activate Checkpoints failover that the {{Code|1=dr-}} call-outs bracket | |||
* [[HA Cluster Setup (JBODs)]] -- HA failover, where the {{Code|1=pool-prefailover.sh}} and {{Code|1=pool-postfailover.sh}} call-outs run | |||
* [[Backup Policies]] -- the policies that invoke {{Code|1=post-backupjob.sh}} | |||
* [[Network Shares]] -- share settings that {{Code|1=share-postcreate.sh}} is typically used to extend | |||
* [[Storage Pools]] -- pool start, stop, export and scrub operations | |||
* [[Security Configuration]] -- appliance security controls, including the System Use Notification passed to {{Code|1=security-message-update.sh}} | |||
* [[QuantaStor Shell Utilities]] -- the {{Code|1=qs-}} helper commands available to a call-out | |||
* [[QuantaStor CLI Command Reference]] -- full argument lists for every {{Code|1=qs}} command | |||
* [[REST API Reference Guide]] -- the same operations over REST, for integration from outside the appliance | |||
</ | ---- | ||
<small>''Verified against QuantaStor 6.9.0.''</small> | |||
Revision as of 07:44, 3 September 2026
QuantaStor has script call-outs which you can use to extend the functionality of the system for integration with custom applications. A call-out is a shell script you place at a fixed path on the appliance; QuantaStor runs it as root at a defined point in a storage operation -- before a pool starts, after a share is created, around a snapshot schedule, either side of an HA failover -- and passes it the identity of the object being operated on.
Because these scripts run as root and are triggered by ordinary storage activity, read Securing the call-out directory before you install your first one.
| Section | Purpose |
|---|---|
| How call-outs work | Naming, synchronous versus asynchronous, the environment scripts run in, what happens to a non-zero exit |
| Securing the call-out directory | Ownership and permissions, and why a writable call-out directory is a root escalation path |
| Storage System call-outs | Service start, chassis beacon, system use notification |
| Network port call-outs | Either side of bringing an interface up |
| Storage Pool call-outs | Start, stop, export, HA failover, scrub |
| Network Share call-outs | Share create and modify |
| Snapshot and replication schedule call-outs | Quiesce and thaw around scheduled snapshots and replication |
| DR failover call-outs | Either side of Activate Checkpoints |
| Software adapter call-outs | Either side of iSCSI and NVMe-oF logins |
| Other extension points | Backup policy post-job handler, alternate pool import script, alert handlers |
| Calling the QuantaStor API from a call-out | Getting object detail back out of QuantaStor from inside your script |
| Example script | A starting-point script that parses the arguments |
| Troubleshooting a call-out that is not firing | The log to read and the three things that are usually wrong |
| Call-outs and upgrades | What survives a package upgrade and the one name that does not |
How call-outs work
Where scripts go, and what they are called
Every call-out lives in one directory and its file name is fixed. QuantaStor looks for the exact name; there is no registration step and no configuration file.
/var/opt/osnexus/custom
Each call-out has two possible forms, and the suffix decides how it runs:
| Form | How it runs |
|---|---|
<hook>.sh |
Synchronously. The operation that triggered it waits for the script to finish. Wrapped in /usr/bin/timeout -k 15s 5m, so the script gets SIGTERM after five minutes and SIGKILL fifteen seconds after that.
|
<hook>-async.sh |
In the background. The operation does not wait, there is no timeout, and the script can outlive the operation entirely. |
If both forms exist QuantaStor runs both, starting the asynchronous one first and then the synchronous one. Use the asynchronous form for anything slow, and the synchronous form only when the work genuinely has to complete before the operation continues.
A synchronous script blocks for its full runtime. A poolscrub-prestart.sh that sleeps for 25 seconds makes qs pool-scrub-start take 27 seconds instead of about one. Under HA failover and pool start, that delay is added to your recovery time, so keep synchronous scripts short.
The environment scripts run in
Call-outs are executed by the QuantaStor service, so they inherit its environment and not a login shell's:
| Value | |
|---|---|
| User | root (uid 0), always
|
| Working directory | /opt/osnexus/quantastor/bin
|
PATH |
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
|
HOME, USER |
empty |
TERM |
dumb
|
LANG |
C.UTF-8
|
There are only nine environment variables in total and none of your shell profile is loaded, so use absolute paths for every command and file your script touches, and do not rely on $HOME or on anything a login would have set. Note in particular that $HOME is an empty string, which breaks any tool configured through a path built from it.
Exit codes are logged, not acted on
A non-zero exit from a synchronous call-out is recorded as a warning together with the script's standard error, and the operation continues regardless. A call-out cannot veto a pool start, a failover or a snapshot. A script that exits 42 during a pool scrub start produces this and the scrub proceeds:
WARN custom_script_manager <the script's stderr, line by line> WARN custom_script_manager Custom script '/var/opt/osnexus/custom/poolscrub-prestart.sh' completed with error '42'.
The practical consequence is that a broken call-out is invisible from the web interface and from the task list. Nothing turns red. The only place it shows up is the service log -- see Troubleshooting a call-out that is not firing.
Grid systems
A call-out runs on the node that is performing the operation, using that node's copy of the script. In a grid, install your scripts on every node that could own the object, or the call-out will fire on some failovers and pool moves and not others.
Turning call-outs off
Call-outs are on whenever the directory exists. To disable the mechanism entirely -- for a support investigation, say -- the QuantaStor service accepts a disable-custom-scripts-manager startup argument. We recommend contacting support before changing service startup arguments.
Securing the call-out directory
The call-out directory is a root execution path. Anything in it named after a hook is run as root the next time an ordinary, unprivileged storage event occurs -- a pool starting, a network port coming up, a snapshot schedule triggering, a share being created. Write access to that directory is therefore equivalent to root on the appliance, and it needs to be treated the way you would treat /etc/sudoers.d.
Check the directory permissions before you install anything. Current releases ship /var/opt/osnexus/custom world-writable (mode 0777), which means any local account can plant a script there and have it run as root. This is a packaging defect, tracked as QSTOR-12443. Repair it, and repair the sibling directories that have the same problem:
chown root:root /var/log/qs /var/opt/osnexus/custom /var/opt/osnexus/alerthandlers /var/opt/osnexus/quantastor chmod 755 /var/log/qs /var/opt/osnexus/custom /var/opt/osnexus/alerthandlers /var/opt/osnexus/quantastor
Then set each script to mode 755 and root ownership, so that no non-root account can modify a script that root is going to execute:
chown root:root /var/opt/osnexus/custom/*.sh chmod 755 /var/opt/osnexus/custom/*.sh
If a script contains sensitive information -- a plain text password, an API token, a private key path -- use 700 rather than 755 so only root can read it. The service runs as root, so 700 does not stop the call-out from working. Better still, keep the secret in a separate root-only file and read it from the script.
Two further points worth making:
- Hardening survives an upgrade. A directory mode you set by hand is preserved across a package upgrade, so this is a one-time repair per appliance rather than something to redo each release.
- Anyone who can write a call-out owns the appliance. That includes any service account with write access to the directory and any process running as a non-root user that can reach it. If you are auditing privilege escalation paths on a QuantaStor system, this directory belongs on the list along with the alert handler directory.
See Security Configuration for the appliance's wider security controls.
Storage System call-outs
| Script | Fires | Arguments |
|---|---|---|
system-poststart.sh |
Every time the QuantaStor service finishes starting | --firstboot, only when the appliance itself has rebooted since the previous start; otherwise no arguments
|
system-ident-start.sh |
Before the Storage System beacon task begins its LED blink and audible beep pattern | none |
system-ident-stop.sh |
After the beacon task completes | none |
security-message-update.sh |
When a non-empty System Use Notification is applied | --message="TEXT"
|
system-poststart.sh
This script runs on every service start, including a plain service restart with no reboot. What changes between the two cases is the argument: QuantaStor compares the current boot time against the stamp it keeps at /var/opt/osnexus/quantastor/qs_lastreboot and passes --firstboot only when the appliance has actually rebooted. Branch on that argument if you want work that happens once per boot rather than once per service start.
system-poststart-async.sh is reserved by the product and is not available to use. QuantaStor installs its own Ceph OSD activation script at that name as a symlink, and the package replaces whatever is there on every install and upgrade, silently. Use the synchronous system-poststart.sh instead, or start your own background work from it. This collision is tracked as QSTOR-12444.
system-prestop.sh
This call-out does not currently fire. The hook is implemented in the service but nothing invokes it, so a script installed at this name never runs on a shutdown or restart, whether initiated from the web interface, the CLI or the console. Tracked as QSTOR-12445. If you need work done before a controlled shutdown, drive it from your own orchestration rather than from this call-out.
security-message-update.sh
Called with the System Use Notification text configured under Security Configuration, so you can mirror the same banner into other subsystems -- a GNOME classification banner, for example, or a login screen you manage yourself. The text is sanitized before it reaches your script: double quotes become single quotes, backslashes become forward slashes, and non-ASCII characters are removed. The call-out is not run when the notification is cleared; QuantaStor removes /etc/issue.notice instead.
Network port call-outs
| Script | Fires | Arguments |
|---|---|---|
port-prestart.sh |
Immediately before QuantaStor brings a port up | --name=PORTNAME
|
port-poststart.sh |
After the port is up and its configuration has been reconciled | --name=PORTNAME
|
port-prestart.sh can fire more than once for a single port activation. When the port is a bond, QuantaStor cycles the slave interfaces first and calls the script around that step as well as around the bond itself. Write the script so that running it twice in a row is harmless.
Storage Pool call-outs
These are the call-outs to use when an application on the appliance has to attach to or detach from a pool, or from a directory inside one, in step with the pool's lifecycle. All of them take the pool's UUID.
| Script | Fires | Arguments |
|---|---|---|
pool-poststart.sh |
After a pool has been started | --pool=POOLUUID
|
pool-prestop.sh |
Before a pool is stopped | --pool=POOLUUID
|
pool-preexport.sh |
Before a pool export begins | --pool=POOLUUID
|
pool-postexport.sh |
After the export finishes, whether or not it succeeded | --pool=POOLUUID
|
pool-prefailover.sh |
Before an HA failover of the pool starts | --pool=POOLUUID
|
pool-postfailover.sh |
After the failover completes, whether or not it succeeded | --pool=POOLUUID
|
poolscrub-prestart.sh |
Before a pool scrub is started | --name=POOLNAME --id=POOLUUID
|
poolscrub-poststart.sh |
Just after the scrub has been started | --name=POOLNAME --id=POOLUUID
|
Points worth knowing:
- The post script always runs. Export and failover are paired so that the
postcall-out fires even when the operation throws partway through. Writepool-postexport.shandpool-postfailover.shas cleanup that must be safe on both the success and failure paths, and do not treat being called as proof the operation worked -- query the pool state if you need to know. poolscrub-poststart.shfires when the scrub has been started, not finished. A scrub runs for hours; the call-out returns as soon as the scrub is under way. There is no scrub-completion call-out.pool-poststart.shalso covers CephFS pools, firing when the pool is mounted on its owning node.pool-prestop.shfires for every pool stop regardless of pool type. The scrub call-outs are ZFS-only, because a scrub request against a CephFS pool is rejected before they are reached -- Ceph scrubbing is driven by policy on the underlying OSDs instead.- Keep
pool-prefailover.shandpool-poststart.shfast if they are synchronous. Both sit directly in the HA recovery path -- see HA Cluster Setup (JBODs).
| Script | Fires | Arguments |
|---|---|---|
share-postcreate.sh |
After a new Network Share has been created and its client access entries written | --share-path=PATH --share-name=NAME --share-id=SHAREUUID --pool=POOLUUID
|
share-postmodify.sh |
After a Network Share has been modified | --share-path=PATH --share-name=NAME --share-id=SHAREUUID --pool=POOLUUID
|
The usual reason to use these is to stamp house defaults onto a share that the dialog does not cover -- POSIX permissions, ACLs, extended attributes, an SELinux label, a directory skeleton. --share-path is the full filesystem path, so the script can act on it directly. See Network Shares for the share settings themselves.
Snapshot and replication schedule call-outs
These exist so that a database or application can be quiesced before a scheduled snapshot and released afterwards. QuantaStor takes the snapshots of all the Storage Volumes and Network Shares in a schedule as a consistency group rather than one at a time, so the quiesce window your script has to hold open is short.
| Script | Fires | Arguments |
|---|---|---|
schedule-prestart.sh |
When a snapshot or replication schedule begins a run, before any per-member work | --name=SCHEDULE_NAME --id=SCHEDULE_ID
|
schedule-presnap.sh |
After all snapshot tasks are staged and immediately before they are released as a group | --name=SCHEDULE_NAME --id=SCHEDULE_ID
|
schedule-postsnap.sh |
After every snapshot in the group has been taken, before replication transfers begin | --name=SCHEDULE_NAME --id=SCHEDULE_ID
|
schedule-prestart.sh fires for both snapshot schedules and replication schedules and is the reliable one of the three. It is called early enough to do preparation work, but not tightly enough around the snapshot itself to serve as a freeze point.
schedule-presnap.sh and schedule-postsnap.sh do not fire for schedules whose members are on a ZFS pool. The snapshot schedule takes a separate code path for ZFS volumes and shares and returns before reaching the point where these two call-outs are invoked. Because virtually all snapshot schedules contain ZFS members, in practice these hooks are not usable today. This is tracked as QSTOR-12378. Do your quiesce work from schedule-prestart.sh until it is resolved, and be aware that the window between schedule-prestart.sh and the snapshots being taken is wider than the presnap window would have been.
Both call-outs are checked for existence before QuantaStor attempts them, so a schedule run logs nothing about them if you have not installed them.
DR failover call-outs
| Script | Fires | Arguments |
|---|---|---|
dr-prefailover.sh |
At the start of the Activate Checkpoints task on a replication schedule | --schedule=SCHEDULE_ID
|
dr-postfailover.sh |
After the checkpoints are activated, the aliases are created and the CIFS and NFS configurations have been rewritten | --schedule=SCHEDULE_ID
|
Activating checkpoints is what turns a DR site's replicas into live, servable shares and volumes, so dr-postfailover.sh is the point at which to repoint an application, update DNS, or start services that depend on the DR copy. See Remote-replication (DR) for the failover procedure these bracket.
Both are subject to the five-minute synchronous timeout, which matters more here than elsewhere: a DR failover is exactly when someone is watching the clock. Use the -async.sh form for anything that involves waiting on an external system.
Software adapter call-outs
QuantaStor's software adapter feature connects the appliance to other systems as an initiator over iSCSI and NVMe-oF (TCP and RDMA). These call-outs bracket the login step so that anything that has to be orchestrated alongside the connection can be.
| Script | Fires | Arguments |
|---|---|---|
swadapter-preconnect.sh |
Before logging in to the adapter's target devices | --adapter=ADAPTERNAME --id=ADAPTERUUID --ip-address=IPADDRESS
|
swadapter-postconnect.sh |
After the logins have been issued | --adapter=ADAPTERNAME --id=ADAPTERUUID --ip-address=IPADDRESS
|
These fire on a boot-time login, on a login you force, and on a repair pass that finds a target to repair -- not on every adapter scan. A steady-state system with all targets connected calls neither script.
Other extension points
Three more places let you supply your own code. They do not use the call-out mechanism described above -- different directory conventions, different argument styles, no -async variants -- so treat them separately.
Backup policy post-job handler
A Backup Policy job runs your script after the transfer completes, if it exists:
/var/opt/osnexus/custom/post-backupjob.sh
It is invoked by the backup job itself rather than by the service, with short-form arguments and no timeout wrapper:
| Argument | Meaning |
|---|---|
-p POLICYNAME |
Name of the Backup Policy as it appears in the web interface |
-b BACKUPDIR |
Directory the backup was written to |
-m SOURCEMOUNTDIR |
Read-only source directory that was backed up |
-s SUBPATH |
Sub-path within the remote storage mount |
-l BACKUPLOG |
List of files that were backed up; passed only when the policy is configured to create a backup log |
A working example ships on every appliance and is the best starting point -- copy it, drop the .example suffix, and replace the body:
/var/opt/osnexus/custom/post-backupjob.sh.example
Its second worked example is genuinely useful in its own right: it prunes the empty directories a file-selection filter leaves behind in the backup target, guarding the operation with a hold file so a recursive rmdir -p cannot delete the backup directory itself.
Alternate pool import script
If a script exists at the path below, QuantaStor calls it instead of running its own zpool import when starting a ZFS pool, passing the zpool name and adding -f when the import is being forced:
/var/opt/osnexus/custom/qs_poolimport.sh
This is an override, not a call-out: get it wrong and the pool does not import. It exists for unusual device-path and multipath situations, and it bypasses the cache-file and device-path handling QuantaStor normally applies. We recommend contacting support before using it.
Alert handlers
Alerts have their own extension mechanism -- a handler script in /var/opt/osnexus/alerthandlers, registered through a configuration file, with per-vendor examples shipped as working reference implementations. It is documented under Custom Alert Handlers on the Call-home / Alerting page. The security guidance in Securing the call-out directory applies to that directory too, and for the same reason.
Calling the QuantaStor API from a call-out
Most call-outs are handed a UUID rather than the object itself, so a useful script generally starts by asking QuantaStor for the detail. The qs CLI is installed on the appliance and works from inside a call-out:
qs pool-get --pool=$POOLID --server=127.0.0.1,admin,PASSWORD --json
Note the argument form: it is qs pool-get --pool=<id>, not a positional UUID. Every command's exact argument list is in the QuantaStor CLI Command Reference, and qs help --min lists the full command set on the appliance itself. --server and --json are global to the CLI rather than arguments of a particular command, so they can be added to any of them.
Three things to get right when scripting the CLI:
- Supply credentials explicitly. With no server argument the CLI falls back to the default administrator account, which will not work on a system whose administrator password has been changed. Pass a comma-separated address, user and password with no spaces, or export the equivalent as
QS_SERVERinside the script. Do not rely on~/.qs.cnf--HOMEis empty in the call-out environment. - Any script holding a password must be mode 700. See Securing the call-out directory.
- Ask for JSON or XML output for anything you parse. The default table output is formatted for people and its column widths change.
For integration from outside the appliance, QuantaStor exposes the same operations over REST; see the REST API Reference Guide. Call-outs and the REST API complement each other -- the call-out tells your application that something happened, and the API lets it find out what. QuantaStor Shell Utilities covers the qs- helper commands that are often more convenient than a raw CLI call.
Example script
A starting point that parses the two most common argument styles and does nothing else. Copy it to the call-out name you need, make it root-owned and mode 755, and put your work where the comment is.
#!/usr/bin/env bash
# QuantaStor custom call-out skeleton.
# Environment note: runs as root, cwd /opt/osnexus/quantastor/bin, no HOME,
# minimal PATH. Use absolute paths.
set -u
LOGFILE=/var/log/qs/my-callout.log
POOLID=""
SCHEDULE_NAME=""
SCHEDULE_ID=""
while [ $# -gt 0 ]; do
case "$1" in
--pool=*) POOLID="${1#*=}" ;;
--id=*) SCHEDULE_ID="${1#*=}" ;;
--schedule=*) SCHEDULE_ID="${1#*=}" ;;
--name=*) SCHEDULE_NAME="${1#*=}" ;;
*) echo "ignoring unrecognized argument '$1'" >&2 ;;
esac
shift
done
{
echo "[$(/bin/date -Is)] called as $0"
echo " pool=$POOLID name=$SCHEDULE_NAME id=$SCHEDULE_ID"
} >> "$LOGFILE" 2>&1
## Put your custom work here. Keep it short if this is the synchronous form:
## the operation that called you is waiting, and you have five minutes.
exit 0
Ignoring an unrecognized argument rather than failing on it is deliberate. QuantaStor has added arguments to call-outs across releases, and a script that exits non-zero on an unexpected one produces a warning in the service log on every single invocation.
Troubleshooting a call-out that is not firing
Start with the service log. Every attempt is recorded, successful or not, tagged custom_script_manager:
grep custom_script_manager /var/log/qs/qs_service.log
A successful synchronous run looks like this -- note that the command QuantaStor actually ran is quoted in full, including the timeout wrapper and every argument, which is the fastest way to confirm what your script was handed:
INFO custom_script_manager Running backgrounded custom script: '/usr/bin/timeout -k 15s 5m /var/opt/osnexus/custom/schedule-prestart.sh --name=nightly --id=67f0dc8a-...' INFO custom_script_manager Custom script '/var/opt/osnexus/custom/schedule-prestart.sh' completed.
The "backgrounded" wording is used for both forms and is wrong for the synchronous one (QSTOR-12446); tell them apart by the timeout prefix, which only the synchronous form has, or by the trailing &, which only the asynchronous form has.
Then work through the usual causes:
| Symptom in the log | Cause | Fix |
|---|---|---|
completed with error '126' and Permission denied |
The file is not executable | chmod 755 <script>
|
completed with error '127' and No such file or directory, but the file is plainly there |
Windows line endings, so the interpreter named on the shebang line has a trailing carriage return; or a shebang pointing at an interpreter that is not installed | dos2unix <script>, and check the shebang path exists
|
| No line for the script at all | QuantaStor never found the file. Most often the name is wrong -- a missing .sh, a hyphen where an underscore was typed, or the file left in place with a .example or .txt suffix |
Compare the name against this page, character for character |
| No line for the script, name confirmed correct | The event you expected did not actually reach that call-out. Several hooks are narrower than they look -- see the notes in each section | Trigger the operation deliberately from the CLI and watch the log live |
| The script runs but behaves differently than from your shell | The call-out environment, not your login environment | Test with env -i /var/opt/osnexus/custom/<script> <args> from /opt/osnexus/quantastor/bin as root
|
A missing script is a normal, expected condition -- QuantaStor logs it at debug level and carries on -- so do not read the absence of an error as evidence that your script ran.
The most reliable way to prove a call-out fires at all is a stub that appends its arguments and a timestamp to a file, installed at the hook name, then triggering the operation from the CLI and reading the file. Once that works, replace the stub with the real thing.
Call-outs and upgrades
Call-out scripts live under /var/opt and are not owned by any package, so a QuantaStor upgrade leaves them in place, along with any directory permissions you have tightened. Only two files in the directory belong to the product: post-backupjob.sh.example and the system-poststart-async.sh symlink.
Two things to do after an upgrade all the same:
- Re-check
system-poststart-async.shif you were using that name. The package replaces it on every install; see system-poststart.sh above. - Re-read this page's argument lists. Arguments have been added to call-outs between releases, and a script written against an older set still runs -- it just ignores information it could be using.
Related pages
- Call-home / Alerting -- alert delivery, ITSM integrations, and the custom alert handler mechanism
- Snapshot Schedules -- the schedules that drive
schedule-prestart.sh - Remote-replication (DR) -- replication schedules and the Activate Checkpoints failover that the
dr-call-outs bracket - HA Cluster Setup (JBODs) -- HA failover, where the
pool-prefailover.shandpool-postfailover.shcall-outs run - Backup Policies -- the policies that invoke
post-backupjob.sh - Network Shares -- share settings that
share-postcreate.shis typically used to extend - Storage Pools -- pool start, stop, export and scrub operations
- Security Configuration -- appliance security controls, including the System Use Notification passed to
security-message-update.sh - QuantaStor Shell Utilities -- the
qs-helper commands available to a call-out - QuantaStor CLI Command Reference -- full argument lists for every
qscommand - REST API Reference Guide -- the same operations over REST, for integration from outside the appliance
Verified against QuantaStor 6.9.0.