QuantaStor Shell Utilities

From OSNEXUS Online Documentation Site
Jump to: navigation, search

Internal to every QuantaStor system is the QuantaStor CLI utility 'qs' which has all the same command functionality as the QuantaStor web management interface (WUI) but there are also other helper utilities for support and triage activities like log collection. This section will go over the basic description of each of these utilities.

qs

This is the main QuantaStor management CLI which is available for download for Linux and Windows from the downloads page. Full documentation for the qs CLI is available on here here. The qs CLI can be used to automate applying licenses to systems, creating snapshots, provisioning file, block and object storage and even automating complete configuration of new clusters. It's especially useful for MSPs/CSPs where frequent configuration and setup of new QuantaStor systems needs to be automated.

qs-ceph

This internal utility helps with gathering stats on mons, pools, osds, and general cluster health. It's a low level utility that in turn calls ceph CLIs. The 'qs-ceph cleanup' command can be used to completely wipe a Ceph Cluster configuration from a system but this should only be used with great caution and only by OSNexus support.

qs-distupgrade

Regular upgrades of QuantaStor systems do not do an upgrade of the underlying distro. This is because a ditro upgrade will upgrade various drivers and other packages that could lead to a longer maintenance window for the upgrade. As such upgrades of the underlying Linux distro should only be done when an adequate maintenance window has been reserved and one should coordinate with OSNexus support before doing a distro upgrade so that your system logs can be reviewed before and after. This script runs the distro upgrade itself and is customized to handle the various platforms QuantaStor is available on.

qs-iostat

This is a helper script which is an overlay over other Linux utilities for looking at device IO stats, ARC stats, and CPU stats. It can be used as part of a hardware triage effort to see what's happening with the hardware performance wise.

qs-sendlogs

This is the main log collection and sending script and you can run it with no args to collect logs and have those sent automatically to the OSNexus log collection server. If your system is deployed in a dark-site then you should use the --no_upload=1 option so that the logs are not attempted to be uploaded when the system doesn't have direct internet access. The --scrub=1 option will remove all personally identifiable information from the logs to make them GDPR compliant. When you use the Send Logs dialog from within the QuantaStor WUI it is using this same script to send logs and by default it does a --scrub=1 to ensure all logs sent to OSNexus are pre-scrubed.

qs-sslcert

This is a helper utility to get information about the SSL certs on the system that QuantaStor is using for inter-node and web communication. All certificates can be replaced and this script helps with that process. See the Security Configuration section for more detail on how to change the certificates in your QuantaStor storage grid.

qs-statsdb

QuantaStor uses InfluxDB to store real-time stats which are then presented in dashboards within various parts of the QuantaStor web management interface (WUI). Custom dashboards can be created in Grafana and this is also documented in the wiki. This qs-statsdb is a simple utility that makes inquiries into InfluxDB so that one can see some of the data. It is good to triaging issues with the dashboards or stats data collection issues.

qs-upgrade

This is the main product upgrade script which is used to upgrade QuantaStor using the package servers at packages.osnexus.com. When one does an upgrade via the QuantaStor WUI it in turn uses this script to drive the upgrade process. QuantaStor commercial packages are packaged in .rpm and .deb platform and use standard yum/apt tools to drive the upgrade process. This script is doing operations like updating the package catalog and running the upgrade of all the various packages and security package updates for the platform.

qs-version

Prints the current version of QuantaStor running on the system for the CLI, Service, WUI, and then gets the version information for all the nodes of the grid the local system is a member of.

qs-zpoolscrub

This is a deprecated script that can be used to run a periodic scrub of a pool. It has been replaced by the Maintenance Schedule which can be created in the Schedules section in the QuantaStor WUI.

qs-crm

QuantaStor systems with Site Clusters setup will have cluster VIFs which underneath are being managed by Corosync and Pacemaker services within the platform. This utility provides commands like 'qs-crm status' which provide some overall health information at a glance. Helpful for triaging issues with cluster floating VIFs.

qs-iofence

QuantaStor does IO fencing to protect highly-available ZFS based Storage Pools. Our IO fencing logic goes through this CLI which can communicate with and do IO-fencing operations with SAS, iSCSI, FC, NVMe, and NVMeoF media. Use 'qs-iofence devstatus' to see the IO fencing status of all devices on the system.

qs-kernelupgrade

This script is used to do kernel upgrades. Most upgrades just use the qs-upgrade script but if the option is selected to also do a kernel upgrade this script is used to upgrade the system to the latest kernel.

qs-showlog

This command is used to show information stored in the log files contained in /var/log/qs/ on a QuantaStor system.

qs-showlog is a helper script for printing various QuantaStor service logs

Usage:

  Service Log:
    qs-showlog -a                  : Dump QuantaStor service log to the console (cat /var/log/qs/qs_service.log)
    qs-showlog -f                  : View service log (tail -f /var/log/qs/qs_service.log)
    qs-showlog -e                  : Print just the errors and warnings in the qs_service.log.
  Service Internals:
    qs-showlog -audit              : View the QuantaStor service audit log (tail /var/log/qs/qs_audit.log)
    qs-showlog -api                : View API command log (tail /var/log/qs/qs_apicall.log)
    qs-showlog -boot               : View the QuantaStor service boot log (tail /var/log/qs/qs_boot.log)
    qs-showlog -crypto             : View crypto log traces (tail /var/log/qs/qs_crypto.log)
    qs-showlog -event [--clear]    : View grid events being processed and generated by the QuantaStor service (tail /var/log/qs/qs_event.log)
    qs-showlog -execcmd [--clear]  : View commands being run by the core service (tail /var/log/qs/qs_exec.log)
    qs-showlog -gs [--clear]       : View grid stats (tail /var/log/qs/qs_gridstats.log)
    qs-showlog -running            : View currently running commands (tail /var/log/qs/qs_service.log)
    qs-showlog -snmp               : View the QuantaStor SNMP agent log (tail /var/log/qs/qs_snmpagent.log)
    qs-showlog -scst [--clear]     : VIew SCST sysfs commands (tail /var/log/qs/qs_scst.log)
    qs-showlog -vols [--clear]     : View iSCSI/FC volume sessions (tail /var/log/qs/qs_volsession.log)

qs-sslcertgen

same as qs-sslcert

qs-status

Provides a quick summary report of all the services running within the system.

[core service]
          quantastor:     active (running) since Wed 2021-11-10 17:09:32 PST; 22h ago

[internal services]
       qs-snmpagentd:     active (running) since Wed 2021-11-10 17:09:30 PST; 22h ago
        qs-zfseventd:     active (running) since Wed 2021-11-10 17:09:29 PST; 22h ago
         qs-jsonrpcd:     active (running) since Wed 2021-11-10 17:09:29 PST; 22h ago
           qs-statsd:     active (running) since Wed 2021-11-10 17:13:33 PST; 22h ago
          qs-scriptd:     active (running) since Wed 2021-11-10 17:09:30 PST; 22h ago
        qs-sso-authd:     active (running) since Wed 2021-11-10 17:09:30 PST; 22h ago

[web services]
               nginx:     active (running) since Wed 2021-11-10 17:09:34 PST; 22h ago
              tomcat:     active (running) since Wed 2021-11-10 17:09:38 PST; 22h ago

[protocol services]
        iscsi-target:     active (running) since Fri 2021-11-05 15:07:57 PDT; 6 days ago
          nfs-server:     active (exited) since Wed 2021-11-10 17:22:29 PST; 22h ago
         nfs-ganesha:     active (running) since Fri 2021-11-05 15:07:55 PDT; 6 days ago
                smbd:     active (running) since Wed 2021-11-10 17:22:27 PST; 22h ago
                nmbd:     active (running) since Fri 2021-11-05 15:07:55 PDT; 6 days ago
             winbind:     active (running) since Wed 2021-11-10 17:22:27 PST; 22h ago

qs-util

This utility has various commands for the support team to do triage and information gathering, it's for internal use only.

qs-zconvert

This command is used to import 3rd party OpenZFS storage pools. QuantaStor has a naming convention for the pool (qs-UUID) and the zvols (UUID) so 3rd party pools must go through a renaming process to prepare the pool for use and management by QuantaStor. So that none of the naming information is lost QuantaStor keeps the friendly name for the pool and zvol(s) and stores these as custom ZFS properties on the datasets. This command doesn't need to be used directly and can be accessed via the Storage Pool Import process that's available in the QuantaStor WUI and available in the CLI (qs pool-import) and REST API.

qs-zvolutil

On systems pre-2015 some users deployed QuantaStor on top of hardware RAID. This is not recommended as the hardware RAID layer can change the data out from underneath the filesystem. If the ZFS based filesystem is at RAID0 it will still detect this bit-rot and corruption as the checksums will not match but with RAID0 at the ZFS layer there's no ability to auto-repair the pool. This qs-zvolutil is a utility that can be used to zero out unreadable blocks within a zvol on a RAID0 device to assist with recovering the Storage Volume. It should only be used by OSNexus support.