@andrian.yablonskyy/thub-coordinator 1.1.16 → 1.1.18

package/views/help/_ci.pug

@@ -1,8 +1,34 @@
-+section('ci', 'CI/CD integration (GitHub Actions)', 'github')
++section('ci', 'CI/CD integration (GitHub, GitLab, Bitbucket, Jenkins)', 'diagram-3')
   p.
-    The test job only needs the Agent and HTTPS access to the Coordinator, so it runs on an ordinary hosted runner.
-    Create a #[strong CI token] on #[a(href="/admin/agents") CI tokens] (admins and maintainers) and store it as the repository secret
-    #[code THUB_CI_TOKEN].
+    Any CI system can run TestHub tests. A pipeline step installs the Agent and runs #[code thub run … --wait]. The step's
+    exit code is the verdict, so the pipeline fails exactly when the tests do.
+
+  h3.h6 What every pipeline needs
+  ul.small
+    li
+      | #[strong A CI token]: create one on #[a(href="/admin/agents") CI tokens] (admins and maintainers). Store it as a masked or
+      | secret variable #[code THUB_KEY], and set #[code THUB_URL] to #[code #{coordinatorUrl}]. Its jobs get #[code A-] ids and the token's group.
+    li
+      | #[strong Node.js 24+ and outbound HTTPS to the Coordinator] on the CI runner. It needs no access to the lab. Use
+      | #[code npx -y @andrian.yablonskyy/thub-agent …] (pin a version, e.g. #[code @1.1.5], for reproducible pipelines) and set
+      | #[code THUB_NO_SELF_UPDATE=1] on short-lived runners.
+    li
+      | #[strong Exit codes] (#[code --wait]): #[code 0] PASSED, #[code 1] FAILED, #[code 2] ERROR/TIMEOUT/LOST, #[code 3] CANCELED,
+      | #[code 4] usage, auth or connection error.
+    li
+      | #[strong Canceling the pipeline cancels the job.] With #[code --wait], #[code SIGINT] and #[code SIGTERM] (how GitHub,
+      | GitLab and Jenkins stop a step) make the Agent cancel the TestHub job before exiting. If the runner kills it outright
+      | (#[code SIGKILL]), the job runs until its #[code --timeout], so keep that tight, or use #[code thub cancel].
+    li #[strong Timeouts]: make the CI step's timeout longer than the queue wait plus #[code --timeout].
+    li #[strong Secrets] go to the job with #[code --env NAME] (the value comes from the step's environment). The CI token never reaches the Client.
+    li #[strong Traceability]: #[code --meta] attaches the pipeline URL and commit to the job.
+    li
+      | #[strong Test reports]: the Coordinator keeps only the JUnit counts. For the CI system's own report view, have the command
+      | publish its JUnit XML to #[a(href="#storage") artifact storage], then download it in the CI step (examples below).
+
+  h3.h6 GitHub Actions
+  p.small.
+    Store the CI token as the repository secret #[code THUB_CI_TOKEN]. The test job runs on an ordinary hosted runner.
   +code('.github/workflows/firmware.yml').
     name: firmware
     on: [push, pull_request]
@@ -38,8 +64,171 @@
                            st-flash --reset write "$THUB_DOWNLOAD_1" 0x08000000 && ./ci/hw-tests.sh' \
                 --suite smoke --timeout 30m --wait \
                 --meta runUrl="$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID"
-  ul.small.mb-0
-    li The step's exit code is the verdict (#[a(href="#agent-cli") exit codes]), so the workflow fails exactly when the tests do.
-    li If the workflow is canceled, the Agent (in #[code --wait] mode) cancels the TestHub job, so abandoned runs don't hold hardware.
-    li Downloads carry no credentials. Publish the image to a URL the Client can read, or fetch it in #[code --command] with a token passed as #[code --env].
-    li Other CI systems (GitLab CI, Jenkins, Azure Pipelines) work the same way: install Node.js, set #[code THUB_URL]/#[code THUB_KEY] and run #[code thub run … --wait].
+  p.small.
+    If the workflow is canceled, the runner sends #[code SIGINT] and the Agent cancels the job.
+
+  h3.h6 GitLab CI/CD
+  p.small.
+    Add #[code THUB_KEY] (masked; protected if only protected branches should reach hardware) and #[code ART_TOKEN] under
+    #[strong Settings → CI/CD → Variables]. The tests are cloned with the job's own #[code CI_JOB_TOKEN]; allow it under
+    #[strong Settings → CI/CD → Job token permissions]. The build hands the image URL to the test job as a #[code dotenv] report,
+    and the JUnit XML comes back through storage for GitLab's test report.
+  +code('.gitlab-ci.yml').
+    # .gitlab-ci.yml
+    stages: [build, test]
+
+    variables:
+      THUB_URL: #{coordinatorUrl}
+      THUB_NO_SELF_UPDATE: "1"
+      ART_BASE: https://artifactory.example.com/fw-local/app
+
+    build:
+      stage: build
+      image: registry.example.com/toolchains/arm-gcc:13
+      script:
+        - cmake -B build -G Ninja && cmake --build build
+        - URL="$ART_BASE/$CI_COMMIT_SHORT_SHA-$CI_PIPELINE_IID/app.bin"
+        - 'curl -fsS -H "Authorization: Bearer $ART_TOKEN" -T build/app.bin "$URL"'
+        - echo "IMAGE_URL=$URL" > build.env
+      artifacts:
+        reports:
+          dotenv: build.env
+
+    test-hw:
+      stage: test
+      image: node:24
+      needs: [build]
+      timeout: 1h                 # > queue wait + --timeout
+      script:
+        - npm i -g @andrian.yablonskyy/thub-agent
+        - |
+          thub run --type hw --label board:nucleo-f401re \
+            --download-file "$IMAGE_URL" \
+            --env CI_JOB_TOKEN --env CI_SERVER_HOST --env CI_PROJECT_PATH --env CI_COMMIT_SHA --env CI_JOB_ID --env ART_TOKEN \
+            --command 'git init -q src && cd src &&
+                       git fetch -q --depth 1 "https://gitlab-ci-token:$CI_JOB_TOKEN@$CI_SERVER_HOST/$CI_PROJECT_PATH.git" "$CI_COMMIT_SHA" &&
+                       git checkout -q FETCH_HEAD &&
+                       st-flash --reset write "$THUB_DOWNLOAD_1" 0x08000000 && ./ci/hw-tests.sh; rc=$?
+                       for f in results/*.xml; do
+                         curl -fsS -H "Authorization: Bearer $ART_TOKEN" -T "$f" "https://artifactory.example.com/qa/gitlab/$CI_JOB_ID/$(basename "$f")"
+                       done
+                       exit $rc' \
+            --timeout 30m --wait \
+            --meta pipelineUrl="$CI_PIPELINE_URL" --meta commit="$CI_COMMIT_SHA" || rc=$?
+          mkdir -p test-results
+          curl -fsS -H "Authorization: Bearer $ART_TOKEN" -o test-results/junit.xml \
+            "https://artifactory.example.com/qa/gitlab/$CI_JOB_ID/junit.xml" || true
+          exit ${rc:-0}
+      artifacts:
+        when: always
+        reports:
+          junit: test-results/*.xml
+
+  h3.h6 Bitbucket Pipelines
+  p.small.
+    Add #[code THUB_URL], #[code THUB_KEY] (secured), #[code ART_TOKEN] (secured) and #[code TESTS_TOKEN] under
+    #[strong Repository settings → Repository variables]. #[code TESTS_TOKEN] is a repository access token with read
+    access, used as the password of the user #[code x-token-auth]. #[code >-] folds the command into one line, so separate its
+    script with #[code ;] and #[code &&]. #[code after-script] collects the JUnit XML into #[code test-results/], which
+    Bitbucket reads automatically.
+  +code('bitbucket-pipelines.yml').
+    # bitbucket-pipelines.yml
+    image: node:24
+
+    definitions:
+      steps:
+        - step: &build
+            name: Build
+            image: registry.example.com/toolchains/arm-gcc:13
+            script:
+              - cmake -B build -G Ninja && cmake --build build
+              - URL="https://artifactory.example.com/fw-local/app/${BITBUCKET_COMMIT:0:7}-$BITBUCKET_BUILD_NUMBER/app.bin"
+              - 'curl -fsS -H "Authorization: Bearer $ART_TOKEN" -T build/app.bin "$URL"'
+              - echo "$URL" > image_url.txt
+            artifacts: [image_url.txt]
+        - step: &test-hw
+            name: HW tests
+            max-time: 60            # minutes; > queue wait + --timeout
+            script:
+              - export THUB_NO_SELF_UPDATE=1 IMAGE_URL="$(cat image_url.txt)"
+              - npm i -g @andrian.yablonskyy/thub-agent
+              - >-
+                thub run --type hw --label board:nucleo-f401re
+                --download-file "$IMAGE_URL"
+                --env TESTS_TOKEN --env BITBUCKET_REPO_FULL_NAME --env BITBUCKET_COMMIT --env BITBUCKET_BUILD_NUMBER --env ART_TOKEN
+                --command 'git init -q src && cd src &&
+                git fetch -q --depth 1 "https://x-token-auth:$TESTS_TOKEN@bitbucket.org/$BITBUCKET_REPO_FULL_NAME.git" "$BITBUCKET_COMMIT" &&
+                git checkout -q FETCH_HEAD &&
+                st-flash --reset write "$THUB_DOWNLOAD_1" 0x08000000 && ./ci/hw-tests.sh; rc=$?;
+                curl -fsS -H "Authorization: Bearer $ART_TOKEN" -T results/junit.xml
+                "https://artifactory.example.com/qa/bitbucket/$BITBUCKET_BUILD_NUMBER/junit.xml"; exit $rc'
+                --timeout 30m --wait
+                --meta buildUrl="$BITBUCKET_GIT_HTTP_ORIGIN/pipelines/results/$BITBUCKET_BUILD_NUMBER"
+            after-script:
+              - mkdir -p test-results
+              - 'curl -fsS -H "Authorization: Bearer $ART_TOKEN" -o test-results/junit.xml "https://artifactory.example.com/qa/bitbucket/$BITBUCKET_BUILD_NUMBER/junit.xml" || true'
+
+    pipelines:
+      default:
+        - step: *build
+        - step: *test-hw
+
+  h3.h6 Jenkins
+  p.small.
+    Add the CI token as a #[strong Secret text] credential (#[code thub-ci-token]), plus #[code artifactory-token] and a
+    username/password for the tests repository (#[code tests-repo]). Run the stage in #[code node:24], or install Node.js with the
+    NodeJS plugin. #[code sh '''…'''] (single quotes) leaves every #[code $] to the shell, so no secret is interpolated by
+    Groovy. Aborting the build sends #[code SIGTERM], and the Agent cancels the job. In a freestyle job, put the same
+    #[code npx … run … --wait] line in an #[em Execute shell] step.
+  +code('Jenkinsfile').
+    // Jenkinsfile (declarative)
+    pipeline {
+      agent none
+      options { timeout(time: 1, unit: 'HOURS') }        // > queue wait + --timeout
+      environment {
+        THUB_URL            = '#{coordinatorUrl}'
+        THUB_NO_SELF_UPDATE = '1'
+      }
+      stages {
+        stage('Build') {
+          agent { label 'fw-build' }
+          environment { ART_TOKEN = credentials('artifactory-token') }
+          steps {
+            sh 'cmake -B build -G Ninja && cmake --build build'
+            script {
+              env.IMAGE_URL = "https://artifactory.example.com/fw-local/app/${env.GIT_COMMIT.take(7)}-${env.BUILD_NUMBER}/app.bin"
+            }
+            sh 'curl -fsS -H "Authorization: Bearer $ART_TOKEN" -T build/app.bin "$IMAGE_URL"'
+          }
+        }
+        stage('HW tests') {
+          agent { docker { image 'node:24' } }
+          environment {
+            THUB_KEY         = credentials('thub-ci-token')
+            ART_TOKEN        = credentials('artifactory-token')
+            NPM_CONFIG_CACHE = "${env.WORKSPACE}/.npm"
+          }
+          steps {
+            withCredentials([usernamePassword(credentialsId: 'tests-repo', usernameVariable: 'GIT_USER', passwordVariable: 'GIT_PASS')]) {
+              sh '''
+                npx -y @andrian.yablonskyy/thub-agent@1.1.5 run --type hw --label board:nucleo-f401re \
+                  --download-file "$IMAGE_URL" \
+                  --env GIT_USER --env GIT_PASS --env TESTS_COMMIT="$GIT_COMMIT" --env BUILD_TAG --env ART_TOKEN \
+                  --command 'git clone -q "https://$GIT_USER:$GIT_PASS@git.example.com/fw/firmware-tests.git" src && cd src &&
+                             git checkout -q "$TESTS_COMMIT" &&
+                             st-flash --reset write "$THUB_DOWNLOAD_1" 0x08000000 && ./ci/hw-tests.sh; rc=$?
+                             curl -fsS -H "Authorization: Bearer $ART_TOKEN" -T results/junit.xml "https://artifactory.example.com/qa/jenkins/$BUILD_TAG/junit.xml"
+                             exit $rc' \
+                  --timeout 30m --wait --meta buildUrl="$BUILD_URL"
+              '''
+            }
+          }
+          post {
+            always {
+              sh 'mkdir -p test-results && curl -fsS -H "Authorization: Bearer $ART_TOKEN" -o test-results/junit.xml "https://artifactory.example.com/qa/jenkins/$BUILD_TAG/junit.xml" || true'
+              junit allowEmptyResults: true, testResults: 'test-results/*.xml'
+            }
+          }
+        }
+      }
+    }

package/views/help/_client-machines.pug

@@ -16,6 +16,7 @@
             li A physical machine (a mini-PC, NUC or Raspberry-class ARM64 board) next to the bench.
             li USB for the DUT's debugger (ST-Link), USB-serial adapters (UART) and DUT USB. A #[strong powered USB hub] is recommended.
             li #[code stlink-tools] and/or #[code openocd], #[code usbutils] (for the dashboard's USB scan).
+            li To power-cycle boards: a hub with per-port power switching and #[code uhubctl] (#[a(href="#usb-power") USB port power]).
             li Docker only if jobs run containers in their #[code --command].
             li One Client instance per board. One machine can drive several boards (up to 8 ST-Links, UARTs and USB devices per instance).
     .col-md-6
@@ -85,3 +86,5 @@
     li Whatever tools your jobs' commands use (git, docker, a flasher) are installed. Their credentials come with each job as #[code --env] (#[a(href="#git") Using git], #[a(href="#docker") Using Docker]), not from the host.
     li Under systemd, the service can only write to its state directory. Jobs write to #[code $THUB_WORK_DIR], not to #[code ~].
     li Optional: a nightly reboot from the runner card's #[strong Reboot] tab (cron, host-local time). It waits for running jobs.
+    li Optional (HW): boards on switchable USB ports, so jobs and their owners can power-cycle them (#[a(href="#usb-power") USB port power]).
+    li Optional: boards on a smart socket or PDU outlet that jobs switch themselves. Install the tools they call (#[code curl], #[code snmp], …) and give each instance its device (#[a(href="#lab-devices") Smart sockets, PDUs, devices]).

package/views/help/_client-setup.pug

@@ -124,3 +124,5 @@
     thub-client restart
     thub-client --config ~/.config/thub/dut1.json status    # a specific instance
     thub-client udev --print                                # the udev rules this config generates
+    thub-client power status                                # USB port power (hw-devices.usbPower)
+    thub-client power reset --port 2 --delay 3              # power-cycle one board, 3 s off

package/views/help/_coordinator.pug

@@ -13,7 +13,7 @@
     #[code ~/.config/thub/coordinator.json] (with a random #[code sessionSecret]) and
     #[code ~/var/lib/thub] (database, avatars). It installs, enables and starts
     #[code thub-coordinator.service], plus the root helper behind the navbar's #[strong Update app] button
-    (off unless the Coordinator runs with #[code --self-update]; see Updates below).
+    (the unit starts the Coordinator with #[code --self-update]; see Updates below).
     The unit uses #[code ProtectSystem=strict] with #[code PrivateTmp=yes], so the data directory and a
     private #[code /tmp] (SQLite's temp files) are the only places the service can write.
 
@@ -176,11 +176,10 @@
   h3.h6 Updates
   p.small.mb-0.
     The Coordinator checks npm for new Coordinator, Agent and Client versions. Admins update Agents and Clients
-    from the #[a(href="/admin/agents") Agents] and #[a(href="/runners") Runners] pages; Clients install an update
-    only between jobs. Updating the #[strong Coordinator] from the dashboard (#[strong Update app] in the navbar)
-    is #[strong off by default]: start it with #[code --self-update] to turn it on. Under systemd:
-    #[code sudo sed -i 's|server.js$|server.js --self-update|' /etc/systemd/system/thub-coordinator.service],
-    then #[code sudo systemctl daemon-reload && sudo systemctl restart thub-coordinator] (a re-install keeps it).
-    Without it, a #[strong vX.Y.Z available] badge shows; update by hand — #[code thub-admin self-update],
-    #[code sudo npm i -g …], or a new container image. Manual alternatives for the others:
-    #[code thub self-update] and #[code thub-client self-update].
+    from the #[a(href="/admin/agents") CI tokens] and #[a(href="/runners") Runners] pages (always there); Clients install
+    an update only between jobs. The #[strong Coordinator]'s own update (the navbar's check for updates and
+    #[strong Update app]) is there only when it runs with #[code --self-update]. A systemd install has it on
+    (#[code sudo npm i -g] writes it into the unit); to turn it off, override #[code ExecStart] in a drop-in
+    (#[code sudo systemctl edit thub-coordinator]). Started any other way, e.g. in a container, it's off and the dashboard
+    shows nothing about Coordinator updates: update it by hand (#[code thub-admin self-update], #[code sudo npm i -g …])
+    or with a new image. Manual alternatives for the others: #[code thub self-update] and #[code thub-client self-update].

package/views/help/_docker.pug

@@ -41,6 +41,35 @@
         td: code --device /dev/thub/dut1-uart
         td HW: gives the container the DUT's UART (#[code /dev/thub/dut1-uart]). Use #[code --privileged] only if you must.
 
+  h3.h6 Registry logins, per registry
+  p.small.
+    Always #[code docker login <registry> -u <user> --password-stdin], with the secret piped in from #[code --env]
+    (never #[code -p]). Only the user and password differ. For cloud registries, mint the short-lived token in the CI step
+    and pass only that.
+  .table-responsive
+    table.table.table-sm.small.align-middle
+      thead
+        tr
+          th Registry
+          th User
+          th Password / token
+      tbody
+        each row in [['Your own (registry:2, Harbor)', 'a robot or service account', 'its password or token'], ['JFrog Artifactory', 'a service user', 'an access token (see Artifact storage)'], ['GitHub (ghcr.io)', 'any GitHub user name', 'a PAT with read:packages, or the workflow\'s GITHUB_TOKEN'], ['GitLab', 'gitlab-ci-token', 'the job\'s CI_JOB_TOKEN (or a deploy token)'], ['Docker Hub', 'your user', 'a personal access token'], ['AWS ECR', 'AWS', 'aws ecr get-login-password (12 h)'], ['Google Artifact Registry', 'oauth2accesstoken', 'gcloud auth print-access-token (1 h)'], ['Azure Container Registry', 'a service principal or token name', 'its secret or token password']]
+          tr
+            td= row[0]
+            td: code= row[1]
+            td= row[2]
+  +code('AWS ECR: token minted in the CI step, never AWS credentials in the job').
+    # CI step with AWS access (e.g. OIDC → an IAM role that can pull from ECR)
+    export ECR_PASSWORD=$(aws ecr get-login-password --region eu-central-1)
+    thub run --type sw --env ECR_PASSWORD --env REG=123456789012.dkr.ecr.eu-central-1.amazonaws.com \
+      --command 'echo "$ECR_PASSWORD" | docker login "$REG" -u AWS --password-stdin &&
+                 docker run --rm --user "$(id -u):$(id -g)" -v "$THUB_WORK_DIR/src:/work" -w /work "$REG/test-runner:1.4" ./run-tests.sh' \
+      --wait
+  p.small.
+    A registry with a private-CA certificate needs #[code /etc/docker/certs.d/<registry>[:port]/ca.crt] on the Client host
+    (plus #[code client.cert] / #[code client.key] there if it asks for a client certificate). Never use #[code insecure-registries].
+
   h3.h6 Clone and test inside a container, with a deploy key and a registry login from the job
   p.small.
     Everything comes from the Agent as #[code --env]: the registry and its credentials, and the private key the

package/views/help/_env.pug

@@ -39,6 +39,9 @@
       ['JOB_ARG / JOB_ARG_<n>', '--arg', 'all, space-separated / each (also "$@")'],
       ['JOB_TIMEOUT', '--timeout', 'seconds'],
       ['JOB_PRIORITY', '--priority', '0–100'],
+      ['JOB_POWER_ON_START', '--power-on-start', 'on, off or reset (HW, when given)'],
+      ['JOB_POWER_ON_END', '--power-on-end', 'on, off or reset (HW, when given)'],
+      ['JOB_POWER_RESET_DELAY', '--power-reset-delay', 'seconds (when given)'],
       ['JOB_META_<KEY>', '--meta key=value', 'the value'],
       ['<NAME>', '--env NAME=value', 'your own variables, under the names you chose']
     ]

package/views/help/_git.pug

@@ -18,19 +18,32 @@
     The token appears only inside the command's environment. To keep it out of #[code .git/config] as well, pass it as a header:
     #[code git -c http.extraHeader="Authorization: Bearer $GH_TOKEN" clone …].
 
-  h3.h6 SSH with a key from the job
-  +code('Private key as --env (base64), used for this job only').
-    export DEPLOY_KEY_B64="$(base64 < ~/.ssh/thub_deploy | tr -d '\n')"
-    thub run --type hw --label board:nucleo-f401re \
-      --env DEPLOY_KEY_B64 \
-      --command 'umask 077 && echo "$DEPLOY_KEY_B64" | base64 -d > "$THUB_WORK_DIR/key" &&
-                 GIT_SSH_COMMAND="ssh -i $THUB_WORK_DIR/key -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new" \
-                   git clone --depth 1 git@github.com:yourorg/firmware-tests.git src && cd src &&
+  h3.h6 SSH with a deploy key and a pinned host key
+  p.small.
+    Create a key pair for TestHub only, without a passphrase (#[code ssh-keygen -t ed25519 -N '' -f thub_deploy]), and add the
+    #[strong public] key as a read-only deploy key: GitHub, Settings → Deploy keys; GitLab, Settings → Repository → Deploy keys;
+    Bitbucket, Repository settings → Access keys. The private key goes with the job as #[code --env]. Pin the server's
+    host key too: fetch it once with #[code ssh-keyscan], check it against the fingerprints your git host publishes,
+    and store both in your CI's secret store.
+  +code('Deploy key and known_hosts from --env, used for this job only').
+    # once, then store both in your CI's secret store:
+    ssh-keyscan github.com > known_hosts            # GitLab: gitlab.com; Bitbucket: bitbucket.org; your server: -p <port> host
+    export GIT_KEY="$(cat thub_deploy)" GIT_KNOWN_HOSTS="$(cat known_hosts)"
+
+    thub run --type hw --label board:nucleo-f401re --env GIT_KEY --env GIT_KNOWN_HOSTS \
+      --command 'umask 077 &&
+                 printf "%s\n" "$GIT_KEY" > "$THUB_WORK_DIR/id" && printf "%s\n" "$GIT_KNOWN_HOSTS" > "$THUB_WORK_DIR/known_hosts" &&
+                 export GIT_SSH_COMMAND="ssh -i $THUB_WORK_DIR/id -o IdentitiesOnly=yes -o UserKnownHostsFile=$THUB_WORK_DIR/known_hosts -o StrictHostKeyChecking=yes" &&
+                 git clone --depth 1 --recurse-submodules --shallow-submodules git@github.com:yourorg/firmware-tests.git src && cd src &&
                  ./ci/test.sh' \
       --wait
-  p.small.
-    The job directory, key included, is deleted when the job ends. Add the public key as a read-only deploy key: GitHub,
-    Repository → Settings → Deploy keys; GitLab, Settings → Repository → Deploy keys; Bitbucket, Repository settings → Access keys.
+  ul.small
+    li #[code GIT_SSH_COMMAND] is exported, so #[code git fetch] and submodules on the same host use it as well.
+    li The key and #[code known_hosts] are written with #[code umask 077] into the job's directory and deleted with it.
+    li #[code StrictHostKeyChecking=yes] refuses an unknown host. #[code accept-new] protects nothing here: every job starts with an empty #[code known_hosts].
+    li Another port (Bitbucket Server, Gitea): #[code ssh://git@git.example.com:7999/proj/repo.git] and #[code ssh-keyscan -p 7999 git.example.com].
+    li To keep the key off the disk, load it into #[code ssh-agent]: #[code eval "$(ssh-agent -s)" && printf "%s\n" "$GIT_KEY" | ssh-add -].
+    li A key in the service user's #[code ~/.ssh] needs no #[code --env], but every job on that host can then use it.
 
   p.small.
     To clone #[em inside a container] with a key passed the same way (held by #[code ssh-agent], never written to disk), see

package/views/help/_kubernetes.pug

@@ -2,8 +2,8 @@
   p.
     The Coordinator runs as one container: the published package in a Node.js image, its SQLite database on a
     persistent volume, TLS at the ingress. Clients and Agents connect from outside at #[code publicUrl].
-    Self-update stays off: the image starts the Coordinator without #[code --self-update], and you update it by
-    deploying a new image tag.
+    Self-update stays off: the image starts the Coordinator without #[code --self-update], so the dashboard shows
+    nothing about Coordinator updates; you update it by deploying a new image tag.
 
   h3.h6 1. Build the image
   +code('Dockerfile').
@@ -160,7 +160,7 @@
   h3.h6 Things to know
   ul.small
     li #[strong One replica.] One SQLite file and an in-process scheduler: no scale-out, no HPA. #[code Recreate] means a few seconds of downtime per upgrade; Clients and Agents reconnect by themselves.
-    li #[strong Updating:] deploy a new image tag. Migrations run on the new pod's first start. #[strong Update app] stays off (no #[code --self-update]); admins see a #[strong vX.Y.Z available] badge. Agents and Clients still update from the dashboard.
+    li #[strong Updating:] deploy a new image tag. Migrations run on the new pod's first start. No #[code --self-update], so no Coordinator update check or #[strong Update app] on the dashboard. Agents and Clients still update from it.
     li #[strong Storage:] block storage (ReadWriteOnce), not NFS. A separate database volume: mount it and set #[code THUB_DB_PATH].
     li #[strong /tmp] is an #[code emptyDir]: SQLite needs a writable temp directory and the root filesystem is read-only.
     li #[strong Backups:] #[code kubectl -n thub exec deploy/thub-coordinator -- node -e "new (require('/app/node_modules/better-sqlite3'))('/var/lib/thub/thub.db').backup('/var/lib/thub/backup.db')"], then #[code kubectl cp].

package/views/help/_lab-devices.pug

@@ -0,0 +1,154 @@
++section('lab-devices', 'Smart sockets, PDUs and other lab devices', 'plug')
+  p.
+    TestHub has one power control built in: #[a(href="#usb-power") USB port power] with uhubctl. Anything else in the lab
+    is driven by #[strong the job itself]: a smart socket (Shelly, Tasmota, TP-Link Kasa, or any socket Home Assistant
+    controls), a PDU outlet (APC, Raritan, …), a USB relay board, a bench supply. The job's #[code --command], or a script
+    from its repository, calls the device with its own tools. The Client needs no configuration for it.
+
+  .table-responsive
+    table.table.table-sm.small.align-middle
+      thead
+        tr
+          th(style="width: 18%")
+          th USB port power (built in)
+          th Socket, PDU or other device
+      tbody
+        tr
+          th Configured in
+          td The Client's #[code hw-devices.usbPower]
+          td The job's command or script, plus a variable naming the bench's device
+        tr
+          th Start / end of job
+          td #[code --power-on-start] / #[code --power-on-end]. The end action runs even on cancel
+          td The script's own steps; on cancel, only as reliable as its trap (below)
+        tr
+          th While the job runs
+          td #[code thub power reset <jobId>] (owner)
+          td Whenever the script decides. It can't be triggered from outside
+        tr
+          th Credentials
+          td None
+          td #[code --env], or the Client host's environment
+
+  h3.h6 Before you start
+  ul.small
+    li #[strong The command runs on the Client host]: the device must be reachable from there (the lab network), never through the Coordinator.
+    li #[strong The tools must be on the Client host]: #[code curl], #[code snmpset] (#[code sudo apt install snmp]), #[code kasa] (#[code pipx install python-kasa]), …
+    li #[strong Credentials come with the job] as #[code --env], so they're masked and dropped when the job ends.
+    li
+      | #[strong Tell each Client which device is its bench's.] Jobs inherit the Client service's environment, so add a
+      | systemd drop-in per instance. Every job on that Client can read it, so put addresses there, not secrets.
+      | Alternatively, pin jobs with #[code --client] and map #[code $JOB_CLIENT] in your script.
+      +code('Client host').
+        sudo systemctl edit thub-client@dut1
+        #   [Service]
+        #   Environment=BENCH_POWER=shelly:10.0.20.11
+        sudo systemctl restart thub-client@dut1
+        systemctl show thub-client@dut1 -p Environment     # check
+
+  h3.h6 One command per device
+  p.small.
+    Each example switches the device off, waits, switches it on, then tests. API details vary between models and firmware,
+    so check your device's manual.
+  +code('Shelly Gen2/Plus/Pro (RPC; digest auth, user admin)').
+    thub run --type hw --label board:nucleo-f401re --env SHELLY_PASSWORD \
+      --command 'S="http://$BENCH_IP/rpc/Switch.Set?id=0"
+                 curl -fsS --digest -u "admin:$SHELLY_PASSWORD" "$S&on=false" && sleep 2 &&
+                 curl -fsS --digest -u "admin:$SHELLY_PASSWORD" "$S&on=true" && sleep 3 && ./ci/test.sh' --wait
+    # Gen1: http://$BENCH_IP/relay/0?turn=off|on (basic auth: curl -u user:password)
+  +code('Tasmota (Power2 … for multi-relay devices)').
+    thub run --type hw … --env TASMOTA_PASSWORD \
+      --command 'T="http://$BENCH_IP/cm?user=admin&password=$TASMOTA_PASSWORD&cmnd=Power%20"
+                 curl -fsS "${T}Off" && sleep 2 && curl -fsS "${T}On" && sleep 3 && ./ci/test.sh'
+  +code('Home Assistant (any socket it controls; a long-lived access token)').
+    thub run --type hw … --env HA_URL=http://ha.lab:8123 --env HA_TOKEN \
+      --command 'ha() { curl -fsS -X POST -H "Authorization: Bearer $HA_TOKEN" -H "Content-Type: application/json" \
+                             -d "{\"entity_id\":\"$BENCH_SWITCH\"}" "$HA_URL/api/services/switch/turn_$1"; }
+                 ha off && sleep 2 && ha on && sleep 3 && ./ci/test.sh'
+  +code('TP-Link Kasa / Tapo (python-kasa; newer firmware needs --username/--password)').
+    thub run --type hw … --command 'kasa --host "$BENCH_IP" off && sleep 2 && kasa --host "$BENCH_IP" on && sleep 3 && ./ci/test.sh'
+  +code('APC switched PDU (SNMP: 1 on, 2 off, 3 reboot; older AP79xx: sPDUOutletCtl …1.4.4.2.1.3.<outlet>)').
+    thub run --type hw … --env PDU_COMMUNITY \
+      --command 'snmpset -v1 -c "$PDU_COMMUNITY" "$PDU_HOST" 1.3.6.1.4.1.318.1.1.12.3.3.1.1.4.$PDU_OUTLET i 3 && sleep 15 && ./ci/test.sh'
+    # state: snmpget -v1 -c "$PDU_COMMUNITY" "$PDU_HOST" 1.3.6.1.4.1.318.1.1.12.3.5.1.1.4.$PDU_OUTLET   (1 on, 2 off)
+  p.small.
+    Anything else with a command line or a network interface works the same way: an SSH-managed PDU
+    (#[code ssh pdu1 "olOff 5"]), a USB relay board (#[code usbrelay]), a bench supply over SCPI
+    (#[code echo 'OUTP OFF' | nc psu1.lab 5025]).
+
+  h3.h6 A power script in the test repository
+  p.small.
+    Once several jobs need it, keep the device logic in the repository the job clones. A single script can cover every
+    kind of device in the lab, keyed by #[code $BENCH_POWER]:
+  +code('ci/power.sh').
+    #!/bin/sh
+    # ci/power.sh on|off|reset [off-seconds] — switch this bench's socket or PDU outlet.
+    # BENCH_POWER: shelly:10.0.20.11 | tasmota:10.0.20.12 | ha:switch.bench1 | apc:pdu1.lab:5
+    # Credentials (--env): POWER_PASSWORD (Shelly, Tasmota), HA_URL + HA_TOKEN, PDU_COMMUNITY.
+    set -eu
+    : "${BENCH_POWER:?BENCH_POWER is not set for this Client}"
+    kind=${BENCH_POWER%%:*} target=${BENCH_POWER#*:}
+
+    switch() {   # $1: on | off
+      case $kind in
+        shelly)
+          url="http://$target/rpc/Switch.Set?id=0&on=$([ "$1" = on ] && echo true || echo false)"
+          if [ -n "${POWER_PASSWORD:-}" ]; then curl -fsS --digest -u "admin:$POWER_PASSWORD" "$url"; else curl -fsS "$url"; fi ;;
+        tasmota)
+          curl -fsS "http://$target/cm?user=admin&password=${POWER_PASSWORD:-}&cmnd=Power%20$1" ;;
+        ha)
+          curl -fsS -X POST -H "Authorization: Bearer $HA_TOKEN" -H 'Content-Type: application/json' \
+            -d "{\"entity_id\":\"$target\"}" "$HA_URL/api/services/switch/turn_$1" ;;
+        apc)
+          snmpset -v1 -c "$PDU_COMMUNITY" "${target%:*}" "1.3.6.1.4.1.318.1.1.12.3.3.1.1.4.${target##*:}" \
+            i "$([ "$1" = on ] && echo 1 || echo 2)" ;;
+        *)
+          echo "power.sh: unknown device kind '$kind' in BENCH_POWER" >&2; exit 2 ;;
+      esac >/dev/null
+      echo "power: $BENCH_POWER $1"
+    }
+
+    case ${1:-} in
+      on|off) switch "$1" ;;
+      reset)  switch off; sleep "${2:-1}"; switch on ;;
+      *)      echo "usage: power.sh on|off|reset [off-seconds]" >&2; exit 2 ;;
+    esac
+  +code('Use it from the job').
+    thub run --type hw --label board:nucleo-f401re \
+      --download-file "$IMAGE_URL" --env GH_TOKEN --env POWER_PASSWORD \
+      --command 'git clone --depth 1 "https://x-access-token:$GH_TOKEN@github.com/yourorg/firmware-tests.git" src && cd src &&
+                 ci/power.sh reset 2 && sleep 3 &&
+                 st-flash --reset write "$THUB_DOWNLOAD_1" 0x08000000 && ./ci/test.sh' --wait
+  +code('Or download the script instead of cloning a repository').
+    thub run --type hw … --download-file https://artifactory.example.com/lab/tools/power.sh \
+      --command 'sh "$THUB_DOWNLOAD_1" reset 2 && ./run-tests.sh'
+
+  h3.h6 Switching it off again, also when the job is canceled
+  p.small.
+    When a job is canceled, times out or its Client is stopped, the Client sends #[code SIGTERM] to the job's
+    #[strong shell only], then #[code SIGKILL] 10 s later. To power the board off whatever happens:
+  ul.small
+    li Start the wrapper with #[code exec], so the wrapper is the shell that receives the signal.
+    li Run the tests in the background and #[code wait] for them. A shell runs a trap only after its current foreground command ends.
+    li Finish the cleanup within 10 s. A #[code SIGKILL] can't be trapped.
+  +code('ci/run.sh — cold-boot, flash, test, always power off').
+    #!/bin/sh
+    set -u
+    cd "$(dirname "$0")/.."
+    test_pid=
+    off() { ci/power.sh off || echo "power: could not switch $BENCH_POWER off" >&2; }
+    trap '[ -n "$test_pid" ] && kill "$test_pid" 2>/dev/null; off; exit 143' TERM INT
+
+    ci/power.sh reset 2 || exit 2
+    sleep 3
+    st-flash --reset write "$THUB_DOWNLOAD_1" 0x08000000 || { off; exit 2; }
+    ./ci/test.sh "$@" & test_pid=$!
+    wait "$test_pid"; rc=$?
+    off
+    exit "$rc"
+  +code('Started with exec').
+    thub run --type hw … --command 'git clone … src && cd src && exec ci/run.sh' --wait
+  +note.
+    For a guarantee independent of the job, also set a timer on the device as a dead-man switch: Shelly's
+    #[code toggle_after] (#[code Switch.Set?id=0&amp;on=true&amp;toggle_after=3600] switches it off again after an hour) or Tasmota's
+    #[code PulseTime]. A failing #[code power.sh] step fails the job, since the command's exit code is the verdict.