@andrian.yablonskyy/thub-coordinator 1.1.17 → 1.1.18

package/views/help/_client-machines.pug

@@ -16,6 +16,7 @@
             li A physical machine (a mini-PC, NUC or Raspberry-class ARM64 board) next to the bench.
             li USB for the DUT's debugger (ST-Link), USB-serial adapters (UART) and DUT USB. A #[strong powered USB hub] is recommended.
             li #[code stlink-tools] and/or #[code openocd], #[code usbutils] (for the dashboard's USB scan).
+            li To power-cycle boards: a hub with per-port power switching and #[code uhubctl] (#[a(href="#usb-power") USB port power]).
             li Docker only if jobs run containers in their #[code --command].
             li One Client instance per board. One machine can drive several boards (up to 8 ST-Links, UARTs and USB devices per instance).
     .col-md-6
@@ -85,3 +86,5 @@
     li Whatever tools your jobs' commands use (git, docker, a flasher) are installed. Their credentials come with each job as #[code --env] (#[a(href="#git") Using git], #[a(href="#docker") Using Docker]), not from the host.
     li Under systemd, the service can only write to its state directory. Jobs write to #[code $THUB_WORK_DIR], not to #[code ~].
     li Optional: a nightly reboot from the runner card's #[strong Reboot] tab (cron, host-local time). It waits for running jobs.
+    li Optional (HW): boards on switchable USB ports, so jobs and their owners can power-cycle them (#[a(href="#usb-power") USB port power]).
+    li Optional: boards on a smart socket or PDU outlet that jobs switch themselves. Install the tools they call (#[code curl], #[code snmp], …) and give each instance its device (#[a(href="#lab-devices") Smart sockets, PDUs, devices]).

package/views/help/_client-setup.pug

@@ -124,3 +124,5 @@
     thub-client restart
     thub-client --config ~/.config/thub/dut1.json status    # a specific instance
     thub-client udev --print                                # the udev rules this config generates
+    thub-client power status                                # USB port power (hw-devices.usbPower)
+    thub-client power reset --port 2 --delay 3              # power-cycle one board, 3 s off

package/views/help/_docker.pug

@@ -41,6 +41,35 @@
         td: code --device /dev/thub/dut1-uart
         td HW: gives the container the DUT's UART (#[code /dev/thub/dut1-uart]). Use #[code --privileged] only if you must.
 
+  h3.h6 Registry logins, per registry
+  p.small.
+    Always #[code docker login <registry> -u <user> --password-stdin], with the secret piped in from #[code --env]
+    (never #[code -p]). Only the user and password differ. For cloud registries, mint the short-lived token in the CI step
+    and pass only that.
+  .table-responsive
+    table.table.table-sm.small.align-middle
+      thead
+        tr
+          th Registry
+          th User
+          th Password / token
+      tbody
+        each row in [['Your own (registry:2, Harbor)', 'a robot or service account', 'its password or token'], ['JFrog Artifactory', 'a service user', 'an access token (see Artifact storage)'], ['GitHub (ghcr.io)', 'any GitHub user name', 'a PAT with read:packages, or the workflow\'s GITHUB_TOKEN'], ['GitLab', 'gitlab-ci-token', 'the job\'s CI_JOB_TOKEN (or a deploy token)'], ['Docker Hub', 'your user', 'a personal access token'], ['AWS ECR', 'AWS', 'aws ecr get-login-password (12 h)'], ['Google Artifact Registry', 'oauth2accesstoken', 'gcloud auth print-access-token (1 h)'], ['Azure Container Registry', 'a service principal or token name', 'its secret or token password']]
+          tr
+            td= row[0]
+            td: code= row[1]
+            td= row[2]
+  +code('AWS ECR: token minted in the CI step, never AWS credentials in the job').
+    # CI step with AWS access (e.g. OIDC → an IAM role that can pull from ECR)
+    export ECR_PASSWORD=$(aws ecr get-login-password --region eu-central-1)
+    thub run --type sw --env ECR_PASSWORD --env REG=123456789012.dkr.ecr.eu-central-1.amazonaws.com \
+      --command 'echo "$ECR_PASSWORD" | docker login "$REG" -u AWS --password-stdin &&
+                 docker run --rm --user "$(id -u):$(id -g)" -v "$THUB_WORK_DIR/src:/work" -w /work "$REG/test-runner:1.4" ./run-tests.sh' \
+      --wait
+  p.small.
+    A registry with a private-CA certificate needs #[code /etc/docker/certs.d/<registry>[:port]/ca.crt] on the Client host
+    (plus #[code client.cert] / #[code client.key] there if it asks for a client certificate). Never use #[code insecure-registries].
+
   h3.h6 Clone and test inside a container, with a deploy key and a registry login from the job
   p.small.
     Everything comes from the Agent as #[code --env]: the registry and its credentials, and the private key the

package/views/help/_env.pug

@@ -39,6 +39,9 @@
       ['JOB_ARG / JOB_ARG_<n>', '--arg', 'all, space-separated / each (also "$@")'],
       ['JOB_TIMEOUT', '--timeout', 'seconds'],
       ['JOB_PRIORITY', '--priority', '0–100'],
+      ['JOB_POWER_ON_START', '--power-on-start', 'on, off or reset (HW, when given)'],
+      ['JOB_POWER_ON_END', '--power-on-end', 'on, off or reset (HW, when given)'],
+      ['JOB_POWER_RESET_DELAY', '--power-reset-delay', 'seconds (when given)'],
       ['JOB_META_<KEY>', '--meta key=value', 'the value'],
       ['<NAME>', '--env NAME=value', 'your own variables, under the names you chose']
     ]

package/views/help/_git.pug

@@ -18,19 +18,32 @@
     The token appears only inside the command's environment. To keep it out of #[code .git/config] as well, pass it as a header:
     #[code git -c http.extraHeader="Authorization: Bearer $GH_TOKEN" clone …].
 
-  h3.h6 SSH with a key from the job
-  +code('Private key as --env (base64), used for this job only').
-    export DEPLOY_KEY_B64="$(base64 < ~/.ssh/thub_deploy | tr -d '\n')"
-    thub run --type hw --label board:nucleo-f401re \
-      --env DEPLOY_KEY_B64 \
-      --command 'umask 077 && echo "$DEPLOY_KEY_B64" | base64 -d > "$THUB_WORK_DIR/key" &&
-                 GIT_SSH_COMMAND="ssh -i $THUB_WORK_DIR/key -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new" \
-                   git clone --depth 1 git@github.com:yourorg/firmware-tests.git src && cd src &&
+  h3.h6 SSH with a deploy key and a pinned host key
+  p.small.
+    Create a key pair for TestHub only, without a passphrase (#[code ssh-keygen -t ed25519 -N '' -f thub_deploy]), and add the
+    #[strong public] key as a read-only deploy key: GitHub, Settings → Deploy keys; GitLab, Settings → Repository → Deploy keys;
+    Bitbucket, Repository settings → Access keys. The private key goes with the job as #[code --env]. Pin the server's
+    host key too: fetch it once with #[code ssh-keyscan], check it against the fingerprints your git host publishes,
+    and store both in your CI's secret store.
+  +code('Deploy key and known_hosts from --env, used for this job only').
+    # once, then store both in your CI's secret store:
+    ssh-keyscan github.com > known_hosts            # GitLab: gitlab.com; Bitbucket: bitbucket.org; your server: -p <port> host
+    export GIT_KEY="$(cat thub_deploy)" GIT_KNOWN_HOSTS="$(cat known_hosts)"
+
+    thub run --type hw --label board:nucleo-f401re --env GIT_KEY --env GIT_KNOWN_HOSTS \
+      --command 'umask 077 &&
+                 printf "%s\n" "$GIT_KEY" > "$THUB_WORK_DIR/id" && printf "%s\n" "$GIT_KNOWN_HOSTS" > "$THUB_WORK_DIR/known_hosts" &&
+                 export GIT_SSH_COMMAND="ssh -i $THUB_WORK_DIR/id -o IdentitiesOnly=yes -o UserKnownHostsFile=$THUB_WORK_DIR/known_hosts -o StrictHostKeyChecking=yes" &&
+                 git clone --depth 1 --recurse-submodules --shallow-submodules git@github.com:yourorg/firmware-tests.git src && cd src &&
                  ./ci/test.sh' \
       --wait
-  p.small.
-    The job directory, key included, is deleted when the job ends. Add the public key as a read-only deploy key: GitHub,
-    Repository → Settings → Deploy keys; GitLab, Settings → Repository → Deploy keys; Bitbucket, Repository settings → Access keys.
+  ul.small
+    li #[code GIT_SSH_COMMAND] is exported, so #[code git fetch] and submodules on the same host use it as well.
+    li The key and #[code known_hosts] are written with #[code umask 077] into the job's directory and deleted with it.
+    li #[code StrictHostKeyChecking=yes] refuses an unknown host. #[code accept-new] protects nothing here: every job starts with an empty #[code known_hosts].
+    li Another port (Bitbucket Server, Gitea): #[code ssh://git@git.example.com:7999/proj/repo.git] and #[code ssh-keyscan -p 7999 git.example.com].
+    li To keep the key off the disk, load it into #[code ssh-agent]: #[code eval "$(ssh-agent -s)" && printf "%s\n" "$GIT_KEY" | ssh-add -].
+    li A key in the service user's #[code ~/.ssh] needs no #[code --env], but every job on that host can then use it.
 
   p.small.
     To clone #[em inside a container] with a key passed the same way (held by #[code ssh-agent], never written to disk), see

package/views/help/_lab-devices.pug

@@ -0,0 +1,154 @@
++section('lab-devices', 'Smart sockets, PDUs and other lab devices', 'plug')
+  p.
+    TestHub has one power control built in: #[a(href="#usb-power") USB port power] with uhubctl. Anything else in the lab
+    is driven by #[strong the job itself]: a smart socket (Shelly, Tasmota, TP-Link Kasa, or any socket Home Assistant
+    controls), a PDU outlet (APC, Raritan, …), a USB relay board, a bench supply. The job's #[code --command], or a script
+    from its repository, calls the device with its own tools. The Client needs no configuration for it.
+
+  .table-responsive
+    table.table.table-sm.small.align-middle
+      thead
+        tr
+          th(style="width: 18%")
+          th USB port power (built in)
+          th Socket, PDU or other device
+      tbody
+        tr
+          th Configured in
+          td The Client's #[code hw-devices.usbPower]
+          td The job's command or script, plus a variable naming the bench's device
+        tr
+          th Start / end of job
+          td #[code --power-on-start] / #[code --power-on-end]. The end action runs even on cancel
+          td The script's own steps; on cancel, only as reliable as its trap (below)
+        tr
+          th While the job runs
+          td #[code thub power reset <jobId>] (owner)
+          td Whenever the script decides. It can't be triggered from outside
+        tr
+          th Credentials
+          td None
+          td #[code --env], or the Client host's environment
+
+  h3.h6 Before you start
+  ul.small
+    li #[strong The command runs on the Client host]: the device must be reachable from there (the lab network), never through the Coordinator.
+    li #[strong The tools must be on the Client host]: #[code curl], #[code snmpset] (#[code sudo apt install snmp]), #[code kasa] (#[code pipx install python-kasa]), …
+    li #[strong Credentials come with the job] as #[code --env], so they're masked and dropped when the job ends.
+    li
+      | #[strong Tell each Client which device is its bench's.] Jobs inherit the Client service's environment, so add a
+      | systemd drop-in per instance. Every job on that Client can read it, so put addresses there, not secrets.
+      | Alternatively, pin jobs with #[code --client] and map #[code $JOB_CLIENT] in your script.
+      +code('Client host').
+        sudo systemctl edit thub-client@dut1
+        #   [Service]
+        #   Environment=BENCH_POWER=shelly:10.0.20.11
+        sudo systemctl restart thub-client@dut1
+        systemctl show thub-client@dut1 -p Environment     # check
+
+  h3.h6 One command per device
+  p.small.
+    Each example switches the device off, waits, switches it on, then tests. API details vary between models and firmware,
+    so check your device's manual.
+  +code('Shelly Gen2/Plus/Pro (RPC; digest auth, user admin)').
+    thub run --type hw --label board:nucleo-f401re --env SHELLY_PASSWORD \
+      --command 'S="http://$BENCH_IP/rpc/Switch.Set?id=0"
+                 curl -fsS --digest -u "admin:$SHELLY_PASSWORD" "$S&on=false" && sleep 2 &&
+                 curl -fsS --digest -u "admin:$SHELLY_PASSWORD" "$S&on=true" && sleep 3 && ./ci/test.sh' --wait
+    # Gen1: http://$BENCH_IP/relay/0?turn=off|on (basic auth: curl -u user:password)
+  +code('Tasmota (Power2 … for multi-relay devices)').
+    thub run --type hw … --env TASMOTA_PASSWORD \
+      --command 'T="http://$BENCH_IP/cm?user=admin&password=$TASMOTA_PASSWORD&cmnd=Power%20"
+                 curl -fsS "${T}Off" && sleep 2 && curl -fsS "${T}On" && sleep 3 && ./ci/test.sh'
+  +code('Home Assistant (any socket it controls; a long-lived access token)').
+    thub run --type hw … --env HA_URL=http://ha.lab:8123 --env HA_TOKEN \
+      --command 'ha() { curl -fsS -X POST -H "Authorization: Bearer $HA_TOKEN" -H "Content-Type: application/json" \
+                             -d "{\"entity_id\":\"$BENCH_SWITCH\"}" "$HA_URL/api/services/switch/turn_$1"; }
+                 ha off && sleep 2 && ha on && sleep 3 && ./ci/test.sh'
+  +code('TP-Link Kasa / Tapo (python-kasa; newer firmware needs --username/--password)').
+    thub run --type hw … --command 'kasa --host "$BENCH_IP" off && sleep 2 && kasa --host "$BENCH_IP" on && sleep 3 && ./ci/test.sh'
+  +code('APC switched PDU (SNMP: 1 on, 2 off, 3 reboot; older AP79xx: sPDUOutletCtl …1.4.4.2.1.3.<outlet>)').
+    thub run --type hw … --env PDU_COMMUNITY \
+      --command 'snmpset -v1 -c "$PDU_COMMUNITY" "$PDU_HOST" 1.3.6.1.4.1.318.1.1.12.3.3.1.1.4.$PDU_OUTLET i 3 && sleep 15 && ./ci/test.sh'
+    # state: snmpget -v1 -c "$PDU_COMMUNITY" "$PDU_HOST" 1.3.6.1.4.1.318.1.1.12.3.5.1.1.4.$PDU_OUTLET   (1 on, 2 off)
+  p.small.
+    Anything else with a command line or a network interface works the same way: an SSH-managed PDU
+    (#[code ssh pdu1 "olOff 5"]), a USB relay board (#[code usbrelay]), a bench supply over SCPI
+    (#[code echo 'OUTP OFF' | nc psu1.lab 5025]).
+
+  h3.h6 A power script in the test repository
+  p.small.
+    Once several jobs need it, keep the device logic in the repository the job clones. A single script can cover every
+    kind of device in the lab, keyed by #[code $BENCH_POWER]:
+  +code('ci/power.sh').
+    #!/bin/sh
+    # ci/power.sh on|off|reset [off-seconds] — switch this bench's socket or PDU outlet.
+    # BENCH_POWER: shelly:10.0.20.11 | tasmota:10.0.20.12 | ha:switch.bench1 | apc:pdu1.lab:5
+    # Credentials (--env): POWER_PASSWORD (Shelly, Tasmota), HA_URL + HA_TOKEN, PDU_COMMUNITY.
+    set -eu
+    : "${BENCH_POWER:?BENCH_POWER is not set for this Client}"
+    kind=${BENCH_POWER%%:*} target=${BENCH_POWER#*:}
+
+    switch() {   # $1: on | off
+      case $kind in
+        shelly)
+          url="http://$target/rpc/Switch.Set?id=0&on=$([ "$1" = on ] && echo true || echo false)"
+          if [ -n "${POWER_PASSWORD:-}" ]; then curl -fsS --digest -u "admin:$POWER_PASSWORD" "$url"; else curl -fsS "$url"; fi ;;
+        tasmota)
+          curl -fsS "http://$target/cm?user=admin&password=${POWER_PASSWORD:-}&cmnd=Power%20$1" ;;
+        ha)
+          curl -fsS -X POST -H "Authorization: Bearer $HA_TOKEN" -H 'Content-Type: application/json' \
+            -d "{\"entity_id\":\"$target\"}" "$HA_URL/api/services/switch/turn_$1" ;;
+        apc)
+          snmpset -v1 -c "$PDU_COMMUNITY" "${target%:*}" "1.3.6.1.4.1.318.1.1.12.3.3.1.1.4.${target##*:}" \
+            i "$([ "$1" = on ] && echo 1 || echo 2)" ;;
+        *)
+          echo "power.sh: unknown device kind '$kind' in BENCH_POWER" >&2; exit 2 ;;
+      esac >/dev/null
+      echo "power: $BENCH_POWER $1"
+    }
+
+    case ${1:-} in
+      on|off) switch "$1" ;;
+      reset)  switch off; sleep "${2:-1}"; switch on ;;
+      *)      echo "usage: power.sh on|off|reset [off-seconds]" >&2; exit 2 ;;
+    esac
+  +code('Use it from the job').
+    thub run --type hw --label board:nucleo-f401re \
+      --download-file "$IMAGE_URL" --env GH_TOKEN --env POWER_PASSWORD \
+      --command 'git clone --depth 1 "https://x-access-token:$GH_TOKEN@github.com/yourorg/firmware-tests.git" src && cd src &&
+                 ci/power.sh reset 2 && sleep 3 &&
+                 st-flash --reset write "$THUB_DOWNLOAD_1" 0x08000000 && ./ci/test.sh' --wait
+  +code('Or download the script instead of cloning a repository').
+    thub run --type hw … --download-file https://artifactory.example.com/lab/tools/power.sh \
+      --command 'sh "$THUB_DOWNLOAD_1" reset 2 && ./run-tests.sh'
+
+  h3.h6 Switching it off again, also when the job is canceled
+  p.small.
+    When a job is canceled, times out or its Client is stopped, the Client sends #[code SIGTERM] to the job's
+    #[strong shell only], then #[code SIGKILL] 10 s later. To power the board off whatever happens:
+  ul.small
+    li Start the wrapper with #[code exec], so the wrapper is the shell that receives the signal.
+    li Run the tests in the background and #[code wait] for them. A shell runs a trap only after its current foreground command ends.
+    li Finish the cleanup within 10 s. A #[code SIGKILL] can't be trapped.
+  +code('ci/run.sh — cold-boot, flash, test, always power off').
+    #!/bin/sh
+    set -u
+    cd "$(dirname "$0")/.."
+    test_pid=
+    off() { ci/power.sh off || echo "power: could not switch $BENCH_POWER off" >&2; }
+    trap '[ -n "$test_pid" ] && kill "$test_pid" 2>/dev/null; off; exit 143' TERM INT
+
+    ci/power.sh reset 2 || exit 2
+    sleep 3
+    st-flash --reset write "$THUB_DOWNLOAD_1" 0x08000000 || { off; exit 2; }
+    ./ci/test.sh "$@" & test_pid=$!
+    wait "$test_pid"; rc=$?
+    off
+    exit "$rc"
+  +code('Started with exec').
+    thub run --type hw … --command 'git clone … src && cd src && exec ci/run.sh' --wait
+  +note.
+    For a guarantee independent of the job, also set a timer on the device as a dead-man switch: Shelly's
+    #[code toggle_after] (#[code Switch.Set?id=0&amp;on=true&amp;toggle_after=3600] switches it off again after an hour) or Tasmota's
+    #[code PulseTime]. A failing #[code power.sh] step fails the job, since the command's exit code is the verdict.

package/views/help/_storage.pug

@@ -0,0 +1,218 @@
++section('storage', 'Artifact storage and certificates: Artifactory, S3, Google Drive, (S)FTP', 'cloud-arrow-up')
+  p.
+    TestHub stores no files. A job's #[strong inputs] come from wherever their URLs point, and its #[strong outputs]
+    (reports, binaries, core dumps) go wherever its command uploads them. The patterns are the same for every kind of storage:
+  .table-responsive
+    table.table.table-sm.small.align-middle
+      thead
+        tr
+          th(style="width: 22%") Direction
+          th How
+          th(style="width: 30%") Credentials
+      tbody
+        tr
+          td Input, anonymous HTTP(S)
+          td #[code --download-file <url>]: fetched by the Client before the command (#[code $THUB_DOWNLOAD_1] …)
+          td None. A plain GET with no headers
+        tr
+          td Input, signed URL
+          td #[code --download-file "<pre-signed URL>"] (S3, GCS, Azure SAS, Artifactory signed URLs)
+          td In the URL. It shows on the job page, so keep its expiry short
+        tr
+          td Input, private
+          td Fetched in #[code --command] (#[code curl], #[code aws], #[code rclone], …)
+          td #[code --env NAME]: masked, dropped when the job ends
+        tr
+          td Input, client certificate
+          td Fetched in #[code --command] with #[code curl --cert] (#[code --download-file] can't present one)
+          td The certificate and key as #[code --env]
+        tr
+          td Output
+          td Uploaded in #[code --command], then listed in #[code $THUB_ARTIFACTS_FILE] so it shows on the job page
+          td #[code --env NAME]
+  ul.small
+    li The tools run on the #[strong Client host], which must reach the storage. Install #[code awscli], #[code rclone], … there, or run them in a container.
+    li Keep secrets off the command line (visible in #[code ps]): #[code curl -K -] reads them from standard input, and #[code aws] and #[code rclone] read them from the environment.
+    li Artifact links must be #[code http(s)]: an #[code s3://] or #[code sftp://] link is dropped from the list.
+    li Keep the verdict: run the tests, save #[code rc=$?], upload, then #[code exit $rc].
+    li Name outputs after the job (#[code …/$THUB_JOB_ID/…]) or a CI id, so a CI step can find them.
+
+  h3.h6 Helper: list what was published
+  p.small The examples below call #[code ci/artifacts.sh] from the tests repository, and assume the command has cloned it.
+  +code('ci/artifacts.sh').
+    #!/bin/sh
+    # ci/artifacts.sh name link [file] — append one entry to $THUB_ARTIFACTS_FILE (§7.3)
+    name=$1 link=$2 file=${3:-}
+    size=null; [ -n "$file" ] && size=$(wc -c < "$file" | tr -d ' ')
+    entry=$(printf '{"name":"%s","size":%s,"link":"%s","timestamp":%s}' "$name" "$size" "$link" "$(date +%s)")
+    if [ -s "$THUB_ARTIFACTS_FILE" ]; then   # plain shell, not sed: signed URLs contain & and |
+      list=$(cat "$THUB_ARTIFACTS_FILE")
+      printf '%s,%s]\n' "${list%]}" "$entry" > "$THUB_ARTIFACTS_FILE"
+    else
+      printf '[%s]\n' "$entry" > "$THUB_ARTIFACTS_FILE"
+    fi
+
+  h3.h6 HTTPS certificates: private CAs and client certificates
+  p.small.
+    #[strong A private CA.] #[code --download-file] is fetched by the Client's Node.js process, which doesn't read the
+    system CA store, so the download fails with #[code SELF_SIGNED_CERT_IN_CHAIN]. Give the Client the CA with
+    #[code NODE_EXTRA_CA_CERTS], once per host, in a drop-in for the template unit, which applies to every instance.
+    #[code update-ca-certificates] covers #[code curl] and #[code git] in jobs, and Docker reads
+    #[code /etc/docker/certs.d/<registry>/ca.crt]. Never switch verification off in a job.
+  +code('Client host').
+    sudo cp lab-ca.crt /usr/local/share/ca-certificates/ && sudo update-ca-certificates   # curl, git, wget, apt…
+    sudo systemctl edit thub-client@.service          # every instance on this host
+    #   [Service]
+    #   Environment=NODE_EXTRA_CA_CERTS=/usr/local/share/ca-certificates/lab-ca.crt
+    sudo systemctl restart 'thub-client@*'
+  p.small.
+    #[strong A client certificate (mutual TLS).] #[code --download-file] can't present one. Fetch the file in
+    #[code --command] with #[code curl --cert], with the certificate and key passed as #[code --env] and written
+    (#[code umask 077]) to the job's directory, which is deleted with the job. For PKCS#12, use
+    #[code curl --cert-type P12 --cert file.p12:password]. One certificate for the whole lab can instead live on the host
+    (readable by the service user only), named in a drop-in: every job on that host can then use it.
+  +code('mTLS download in the command').
+    export CLIENT_CERT="$(cat thub-ci.crt)" CLIENT_KEY="$(cat thub-ci.key)"     # PEM, from your CI's secret store
+    thub run --type hw --env CLIENT_CERT --env CLIENT_KEY \
+      --command 'umask 077 && printf "%s\n" "$CLIENT_CERT" > "$THUB_WORK_DIR/c.pem" && printf "%s\n" "$CLIENT_KEY" > "$THUB_WORK_DIR/k.pem" &&
+                 curl -fsSL --cert "$THUB_WORK_DIR/c.pem" --key "$THUB_WORK_DIR/k.pem" -o app.bin https://artifactory.lab/fw-local/app/1.4.0/app.bin &&
+                 st-flash --reset write app.bin 0x08000000 && ./ci/test.sh' --wait
+
+  h3.h6 JFrog Artifactory: service authentication
+  .table-responsive
+    table.table.table-sm.small.align-middle
+      thead
+        tr
+          th Credential
+          th Sent as
+          th Notes
+      tbody
+        tr
+          td #[strong Access token] (scoped, expiring)
+          td: code Authorization: Bearer <token>
+          td Recommended. A service user with read access to inputs and deploy access to outputs.
+        tr
+          td Reference token
+          td: code Authorization: Bearer <token>
+          td Short form of an access token, for CI variables with a length limit.
+        tr
+          td Identity token / password
+          td: code -u svc-thub:<token>
+          td For tools that only speak basic auth: #[code docker login], #[code pip].
+        tr
+          td API key
+          td: code X-JFrog-Art-Api
+          td Deprecated and turned off in current versions. Migrate.
+        tr
+          td Anonymous read
+          td —
+          td Public repositories only. Then #[code --download-file] is enough.
+  p.small.
+    Best: a #[strong short-lived token per run], minted in the CI step from a service token, so a leaked token expires on its own.
+    On GitHub Actions, JFrog's OIDC integration (#[code jfrog/setup-jfrog-cli] with an #[code oidc-provider-name]) needs no stored secret at all.
+  +code('CI step: a one-hour token for the job').
+    # CI step: ART_SERVICE_TOKEN is a CI secret for the service user svc-thub
+    export ART_TOKEN=$(curl -fsS -H "Authorization: Bearer $ART_SERVICE_TOKEN" -X POST \
+      -d scope=applied-permissions/user -d expires_in=3600 \
+      https://artifactory.example.com/access/api/v1/tokens | jq -r .access_token)
+    thub run --type hw --env ART_TOKEN --command '…' --wait
+  +code('curl: download, test, upload with checksum, list').
+    thub run --type hw --env ART_TOKEN \
+      --command 'ART=https://artifactory.example.com/artifactory
+                 H() { printf "header = \"Authorization: Bearer %s\"\n" "$ART_TOKEN"; }     # via curl -K -, so not visible in ps
+                 H | curl -fsSL -K - -o app.bin "$ART/fw-local/app/1.4.0-42/app.bin" &&
+                 st-flash --reset write app.bin 0x08000000 && ./ci/test.sh; rc=$?
+                 url="$ART/qa-local/$THUB_JOB_ID/junit.xml"
+                 H | curl -fsS -K - -H "X-Checksum-Sha256: $(sha256sum results/junit.xml | cut -d" " -f1)" -T results/junit.xml "$url" &&
+                   sh ci/artifacts.sh junit.xml "$url" results/junit.xml
+                 exit $rc' --wait
+  +code('JFrog CLI, configured from --env, nothing left in the home directory').
+    export JF_ACCESS_TOKEN="$ART_TOKEN"
+    thub run --type sw --env JF_URL=https://artifactory.example.com --env JF_ACCESS_TOKEN \
+      --command 'export JFROG_CLI_HOME_DIR="$THUB_WORK_DIR/.jfrog" CI=true
+                 jf rt dl "fw-local/app/1.4.0-42/*.bin" downloads/ --flat && ./ci/test.sh; rc=$?
+                 jf rt u "results/*.xml" "qa-local/$THUB_JOB_ID/" --flat
+                 exit $rc' --wait
+  +code('Signed URL (Enterprise+): --download-file with nothing else').
+    IMAGE_URL=$(curl -fsS -H "Authorization: Bearer $ART_SERVICE_TOKEN" -H "Content-Type: application/json" -X POST \
+      -d '{"repo_path":"/fw-local/app/1.4.0-42/app.bin","valid_for_secs":3600}' \
+      https://artifactory.example.com/artifactory/api/signed/url)
+    thub run --type hw --download-file "$IMAGE_URL" --command 'st-flash --reset write "$THUB_DOWNLOAD_1" 0x08000000 && ./ci/test.sh' --wait
+  +code('The same token for Docker, PyPI and npm repositories').
+    # Docker registry (see §7.2): the token as the password
+    echo "$ART_TOKEN" | docker login artifactory.example.com -u svc-thub --password-stdin
+    docker pull artifactory.example.com/docker-local/team/test-runner:1.4
+    # PyPI remote/virtual repository
+    pip install -r requirements.txt --index-url "https://svc-thub:$ART_TOKEN@artifactory.example.com/artifactory/api/pypi/pypi/simple"
+    # npm: a project-local .npmrc in the work directory, not ~/.npmrc
+    printf '//artifactory.example.com/artifactory/api/npm/npm/:_authToken=%s\n' "$ART_TOKEN" > .npmrc
+    npm ci --registry https://artifactory.example.com/artifactory/api/npm/npm/
+  p.small.
+    Nexus raw repositories take #[code curl -u user:password --upload-file]. GitLab's generic packages take #[code curl -H "JOB-TOKEN: $CI_JOB_TOKEN" --upload-file].
+
+  h3.h6 AWS S3 (and MinIO, Ceph, Cloudflare R2, Wasabi)
+  p.small.
+    Pre-sign inputs in CI, so the job needs no AWS credentials. Pass short-lived session credentials from the CI step's role
+    (OIDC) for outputs. A pre-signed link lasts at most 7 days. For S3-compatible storage, add #[code --endpoint-url] to #[code aws].
+  +code('Inputs: pre-signed in the CI step').
+    # in the CI step, which has AWS access (e.g. GitHub/GitLab OIDC → an IAM role)
+    IMAGE_URL=$(aws s3 presign "s3://fw-builds/app/$SHA/app.bin" --expires-in 3600)
+    thub run --type hw --download-file "$IMAGE_URL" --command 'st-flash --reset write "$THUB_DOWNLOAD_1" 0x08000000 && ./ci/test.sh' --wait
+  +code('Outputs: aws s3 cp with session credentials').
+    thub run --type sw --env AWS_ACCESS_KEY_ID --env AWS_SECRET_ACCESS_KEY --env AWS_SESSION_TOKEN --env AWS_DEFAULT_REGION=eu-central-1 \
+      --command 'aws s3 cp "s3://fw-builds/app/1.4.0/app.elf" . && ./ci/test.sh; rc=$?
+                 aws s3 cp --recursive results/ "s3://qa-results/$THUB_JOB_ID/" &&
+                 sh ci/artifacts.sh report.html "$(aws s3 presign "s3://qa-results/$THUB_JOB_ID/report.html" --expires-in 604800)"
+                 exit $rc' --wait
+
+  h3.h6 Google Drive
+  p.small.
+    Use #[a(href="https://rclone.org/drive/" target="_blank" rel="noopener") rclone] with a #[strong service account] that's a
+    member of a #[strong shared drive] (service accounts have no storage of their own). rclone takes its configuration from
+    environment variables. #[code rclone link] makes a file readable by anyone with the link; leave it out for private reports.
+    Public #[code uc?export=download] links work with #[code --download-file] only for small files.
+  +code('rclone with a service account').
+    # GDRIVE_SA: the service account's JSON key, as one line (a CI secret)
+    thub run --type sw --env GDRIVE_SA --env RCLONE_CONFIG_GD_TYPE=drive --env RCLONE_CONFIG_GD_SCOPE=drive \
+      --env RCLONE_CONFIG_GD_TEAM_DRIVE=0ABCdEfGhIjKlUk9PVA \
+      --command 'export RCLONE_CONFIG_GD_SERVICE_ACCOUNT_CREDENTIALS="$GDRIVE_SA"
+                 rclone copyto "gd:firmware/1.4.0/app.bin" app.bin && ./ci/test.sh app.bin; rc=$?
+                 rclone copy results/ "gd:qa/$THUB_JOB_ID/" &&
+                 sh ci/artifacts.sh report.html "$(rclone link "gd:qa/$THUB_JOB_ID/report.html")"
+                 exit $rc' --wait
+  +code('Or the Drive API with curl and an OAuth token from the CI step').
+    curl -fsSL -H "Authorization: Bearer $GDRIVE_TOKEN" -o app.bin "https://www.googleapis.com/drive/v3/files/$FILE_ID?alt=media&supportsAllDrives=true"
+    curl -fsS -H "Authorization: Bearer $GDRIVE_TOKEN" \
+      -F "metadata={\"name\":\"report.html\",\"parents\":[\"$FOLDER_ID\"]};type=application/json" -F "file=@results/report.html" \
+      "https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart&supportsAllDrives=true"
+
+  h3.h6 FTP, FTPS and SFTP
+  p.small.
+    #[code curl] speaks all three. Prefer SFTP or FTPS (#[code --ssl-reqd]); plain FTP sends the password in clear text. Pin the
+    SFTP host key with #[code --hostpubsha256] (from #[code ssh-keyscan host | ssh-keygen -lf -], without #[code SHA256:]).
+    FTP and SFTP links aren't #[code http(s)], so list the server's HTTPS view, if it has one.
+  +code('SFTP download and upload (FTPS and key auth in the comments)').
+    thub run --type hw --env SFTP_USER --env SFTP_PASSWORD --env SFTP_HOSTKEY \
+      --command 'auth() { printf "user = \"%s:%s\"\n" "$SFTP_USER" "$SFTP_PASSWORD"; }   # read by curl -K -, so not visible in ps
+                 auth | curl -fsS -K - --hostpubsha256 "$SFTP_HOSTKEY" -o app.bin "sftp://files.lab/fw/1.4.0/app.bin" &&
+                 st-flash --reset write app.bin 0x08000000 && ./ci/test.sh; rc=$?
+                 auth | curl -fsS -K - --hostpubsha256 "$SFTP_HOSTKEY" --ftp-create-dirs -T results/junit.xml "sftp://files.lab/qa/$THUB_JOB_ID/"
+                 exit $rc' --wait
+    # FTPS:         auth | curl -fsS -K - --ssl-reqd --ftp-create-dirs -T results/junit.xml "ftp://files.lab/qa/$THUB_JOB_ID/"
+    # SSH key auth: printf '%s\n' "$SFTP_KEY" > "$THUB_WORK_DIR/.key" && chmod 600 "$THUB_WORK_DIR/.key" &&
+    #               curl --key "$THUB_WORK_DIR/.key" -u "$SFTP_USER:" --hostpubsha256 "$SFTP_HOSTKEY" …
+
+  h3.h6 Any other HTTP storage: curl and your own authentication
+  +code('curl authentication patterns').
+    H() { printf 'header = "%s"\n' "$1"; }                                     # headers via curl -K -, kept out of ps
+    H "Authorization: Bearer $API_TOKEN" | curl -fsSL -K - -o app.bin "$URL"   # bearer token
+    H "X-API-Key: $API_KEY"              | curl -fsSL -K - -o app.bin "$URL"   # API-key header
+    printf 'user = "%s:%s"\n' "$USER_NAME" "$USER_PASS" | curl -fsSL -K - -o app.bin "$URL"   # basic auth
+    curl -fsSL --netrc-file "$THUB_WORK_DIR/.netrc" -o app.bin "$URL"         # a .netrc the command wrote from --env values
+    curl -fsSL --cert "$THUB_WORK_DIR/c.pem" --key "$THUB_WORK_DIR/k.pem" -o app.bin "$URL"   # mutual TLS
+    # OAuth2 client credentials: get a token, then use it as a bearer token
+    TOKEN=$(curl -fsS -d grant_type=client_credentials -u "$CLIENT_ID:$CLIENT_SECRET" https://auth.example.com/oauth/token | jq -r .access_token)
+    # uploads: PUT (-T file), or a form POST (-F "file=@results/report.html")
+  p.small.mb-0.
+    Once the patterns settle, move them into #[code ci/fetch.sh] / #[code ci/publish.sh] in the tests repository, so every job's
+    #[code --command] stays one line. The CI examples in #[a(href="#ci") CI/CD integration] use this storage to bring JUnit reports back.

package/views/help/_troubleshooting.pug

@@ -9,10 +9,18 @@
       ['The job is rejected with `422`', 'No registered Client can ever match its type, labels, group or client. Compare `thub resources` with your `--type` and `--label`s; your group (set on the dashboard, see `thub whoami`) must have a Client of that kind.'],
       ['The job stays `QUEUED`', 'Matching Clients are all `BUSY`, locked (`local`), in `MAINTENANCE` or offline, or the job is pinned with `--client`. It times out after `--timeout`.'],
       ['`ERROR` during prepare', 'A download failed; the job log names the URL and the reason.'],
+      ['`Download failed … (SELF_SIGNED_CERT_IN_CHAIN)` or `UNABLE_TO_GET_ISSUER_CERT_LOCALLY`', 'The server\'s certificate is from a CA the Client doesn\'t trust. Node.js ignores the system CA store: add `Environment=NODE_EXTRA_CA_CERTS=/path/ca.crt` with `sudo systemctl edit thub-client@.service` and restart the Clients. A download that needs a client certificate can\'t use `--download-file` at all: fetch it with `curl --cert` in `--command` (Artifact storage and certificates).'],
+      ['`Host key verification failed` on a git clone', 'The job\'s `known_hosts` has no key for that server (or a different one). Pass the output of `ssh-keyscan <host>` as `--env GIT_KNOWN_HOSTS` and use it with `UserKnownHostsFile` (Using git). `Permission denied (publickey)` means the host is fine, but the deploy key isn\'t on that repository.'],
+      ['`docker login`: unauthorized', 'Check the user the registry expects: `AWS` for ECR, `oauth2accesstoken` for Google, `gitlab-ci-token` with `CI_JOB_TOKEN` for GitLab (Using Docker). ECR and Google tokens expire after 12 h and 1 h, so mint them in the CI step that submits the job.'],
       ['A git clone or docker pull in the command fails', 'They run in your `--command`, with the credentials you pass as `--env` (see Using git / Using Docker). `git` or `docker` must be installed on that Client host; the Client itself needs neither. Try the command with `--dry-run` to see its environment.'],
       ['`--git-repo` / `--docker-image`: unknown option, or the job is refused', 'They were removed: clone the repository and run containers in `--command`, passing tokens with `--env`. An older Agent that still sends them is refused with that message; update it (`thub self-update`).'],
       ['Permission denied on `/dev/ttyUSB*` or `docker.sock`', 'Docker or the device groups were added after the Client was installed. Re-run `sudo npm i -g @andrian.yablonskyy/thub-client` so the unit gets the `dialout`/`plugdev`/`docker` groups, then restart.'],
       ['A configured device is shown as missing', 'Check the `devpath` with `udevadm info -a -n <dev>`, then `thub-client udev --print`; restart the Client to regenerate the udev rules.'],
+      ['`thub power` is refused', 'It says why: only the job\'s owner may switch power (`403`, even for an admin; `thub whoami` shows whose key you use); the job must be `PREPARING` or `RUNNING`; the Client must list ports in `hw-devices.usbPower.ports` and have `uhubctl` installed (restart it after installing, so it reports it). `--port` is a position in that list, not the hub\'s port number.'],
+      ['USB power: `No compatible devices detected!`', 'Either the hub can\'t switch per-port power (`sudo uhubctl` doesn\'t list it), or the Client\'s user may not switch it: install its udev rules with `sudo thub-client --config <path> udev` and check its service has the `plugdev` group (reinstall the Client if `plugdev` was created after it).'],
+      ['USB power is off, but the board stays on (or half-boots after a reset)', 'Something else powers it: a second USB cable (ST-Link and the board\'s own USB), a debugger or an external supply. Every cable that powers it needs a switched port in `usbPower.ports`. Some hubs also feed power back from upstream. A board that half-boots needs a longer delay: `--delay 3`.'],
+      ['A job can\'t reach its smart socket or PDU', 'The job\'s command runs on the Client host, so the device must be reachable from there: `curl -sI http://<device>/` as the Client\'s user. Tools such as `curl` or `snmpset` must be installed on that host. `BENCH_POWER is not set`: add the instance\'s systemd drop-in and restart it (`systemctl show thub-client@<instance> -p Environment`).'],
+      ['A board on a socket or PDU stays on after a canceled job', 'On cancel the Client sends `SIGTERM` to the job\'s shell only, then `SIGKILL` after 10 s. Start the wrapper with `exec`, run the tests in the background with `wait`, trap `TERM`, and finish within 10 s. Add a timer on the device (Shelly `toggle_after`, Tasmota `PulseTime`) as a backstop.'],
       ['Job `LOST`', 'The Client missed 3 heartbeats (network, reboot, crash). With `requeueOnLost` the job is retried once on another Client.'],
       ['A job can\'t write to `~` (read-only file system)', 'Under systemd the Client can only write to its state directory. Write to `$THUB_WORK_DIR`, and set `DOCKER_CONFIG` there for `docker login`.'],
       ['Live logs don\'t stream through the proxy', 'Turn off response buffering for the Coordinator (nginx: `proxy_buffering off;`, Caddy: `flush_interval -1`).'],