@andrian.yablonskyy/thub-coordinator 1.0.12 → 1.0.14

package/README.md

@@ -44,7 +44,7 @@ The Coordinator loads a plain **JSON** config file (no YAML support). Resolution
   "jobs": { "defaultTimeoutSec": 1800, "maxTimeoutSec": 14400 },
   "retention": { "logRetentionDays": 14, "artifactRetentionDays": 30 },
   "artifacts": { "maxUploadMb": 512, "linkTtlHours": 168 },
-  "updates": { "checkIntervalHours": 6, "registry": null }
+  "updates": { "checkIntervalMin": 15, "registry": null }
 }
 ```
 
@@ -132,7 +132,7 @@ All endpoints are under `/api/v1`, JSON, and require `Authorization: Bearer <tok
 | Method | Path | Description |
 |---|---|---|
 | `POST` | `/resources/register` | Self-register (every start/restart) using the shared `clientJoinKey`; upserts by `clientId`. `hostInfo.addresses` (`[{iface, address, family}]`) lists the host's non-loopback interface addresses; the request's source address is stored as the external one. |
-| `POST` | `/resources/:id/heartbeat` | Heartbeat and self-reported state, plus the current `addresses`; refreshes the external address; response may carry `commands[]` (`cancel-job`, `self-update`). |
+| `POST` | `/resources/:id/heartbeat` | Heartbeat and self-reported state, plus the current `addresses`, `hostUptimeSec` and `activity` (`{state, jobId, durationSec}`); refreshes the external address; response may carry `commands[]` (`cancel-job`, `self-update`, `unlock`, `cancel-update`). |
 | `POST` | `/resources/:id/status` | Explicit status change, e.g. a local lock. |
 | `GET` | `/resources/:id/jobs/next?wait=30` | Long-poll for an assigned job; `204` when nothing arrived within `wait` seconds. |
 | `POST` | `/jobs/:id/accept` | Acknowledge assignment. |
@@ -157,11 +157,11 @@ Server-rendered Pug templates styled with Bootstrap 5.3, with a little vanilla J
 | `/admin/agents` | Register/revoke CI and developer agents; token shown once. Shows each agent's last reported version. |
 | `/profile` | Every logged-in user's own account settings: display name, avatar, timezone (renders every dashboard timestamp), theme, idle session timeout, password change. |
 
-**Resource card.** Clicking a resource on `/` or `/resources` opens a card with its name, type, status, Client version, IP addresses, capabilities (reported at registration: HW udev devices with a **missing** flag for any whose device node doesn't exist, relay channels and power control; SW image, source and limits), labels, groups, busy source/reason, last heartbeat and (admins only) the maintenance/rotate-token actions. IP addresses are the Client host's interface addresses except loopback (reported at registration and on every heartbeat) plus the external address — the one the Coordinator last saw the Client connect from, taken from `X-Forwarded-For` when the request came through a proxy `trustProxy` trusts.
+**Resource card.** Clicking a resource on `/` or `/resources` opens a card with its name, type, status, host uptime and current task/status with its duration (both from the Client's heartbeats, anchored to the Coordinator's clock; as of the last heartbeat for an OUT_OF_SERVICE resource), Client version, IP addresses, capabilities (reported at registration: HW udev devices with a **missing** flag for any whose device node doesn't exist, relay channels and power control; SW image, source and limits), labels, groups, busy source/reason, last heartbeat and (admins only) the maintenance/rotate-token actions. IP addresses are the Client host's interface addresses except loopback (reported at registration and on every heartbeat) plus the external address — the one the Coordinator last saw the Client connect from, taken from `X-Forwarded-For` when the request came through a proxy `trustProxy` trusts. Admins get a **Cancel** button on it, enabled only while the Client is doing something: it cancels its current job, releases a manual local lock, or aborts a self-update hold on its host (the last two delivered as `unlock` / `cancel-update` heartbeat commands).
 
 The Coordinator's own version (`ver. X.Y.Z`) is shown under the TestHub logo, top left of every page.
 
-**Versions and self-update.** The Coordinator checks the npm registry for the latest Coordinator, Agent and Client every `updates.checkIntervalHours` (default 6, `0` disables; `updates.registry` for a private mirror) and on **Check for updates**. A newer Coordinator shows as an **Update app to X.Y.Z** button in the navbar for admins (a badge under the logo for viewers); Agents and Resources show each one's version with an **update available** or pending **→ vX.Y.Z** badge. Admins request self-updates per agent/resource (**Update**, which becomes **Cancel update**) or with **Update all agents** / **Update all clients**, always to the latest version:
+**Versions and self-update.** The Coordinator checks the npm registry for the latest Coordinator, Agent and Client every `updates.checkIntervalMin` (default 15 minutes, `0` disables; the older `checkIntervalHours` is ignored; `updates.registry` for a private mirror) and on **Check for updates**. A newer Coordinator shows as an **Update app to X.Y.Z** button in the navbar for admins (a badge under the logo for viewers); Agents and Resources show each one's version with an **update available** or pending **→ vX.Y.Z** badge. Admins request self-updates per agent/resource (**Update**, which becomes **Cancel update**) or with **Update all agents** / **Update all clients**, always to the latest version:
 - Clients get a `self-update` command on their next heartbeat and install it through their root `thub-client-update` helper, once none of the host's instances is busy.
 - Agents install it at the start of their next run (`GET /api/v1/agents/me/update`), then re-run the command on the new version.
 - The Coordinator itself: the **Update app** button writes `<dataDir>/update-request.json`; the root `thub-coordinator-update.path` unit (installed by `sudo npm i -g`) runs `npm i -g` for it, and the postinstall restarts the service (the dashboard and API are briefly down). Logs: `journalctl -u thub-coordinator-update`. Without that unit, use `thub-admin self-update` on the host.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@andrian.yablonskyy/thub-coordinator",
-  "version": "1.0.12",
+  "version": "1.0.14",
   "description": "TestHub Coordinator — job queue, resource registry, scheduler, heartbeat monitor, log/artifact store and web dashboard",
   "main": "src/server.js",
   "engines": {

package/src/api/resource.js

@@ -84,7 +84,7 @@ function createResourceRouter({ services, config }){
 
   router.post('/resources/:id/heartbeat', auth, requireOwnResource, (req, res, next) => {
     try {
-      const { state, activeJobId, localLock, metrics, addresses } = req.body;
+      const { state, activeJobId, localLock, metrics, addresses, hostUptimeSec, activity } = req.body;
       services.registry.heartbeat(req.params.id, {
         state,
         activeJobId,
@@ -92,7 +92,9 @@ function createResourceRouter({ services, config }){
         metrics,
         addresses,
         remoteAddr: req.ip,
-        clientVersion: appVersion(req, 'client')
+        clientVersion: appVersion(req, 'client'),
+        hostUptimeSec,
+        activity
       });
       const commands = services.commands.drain(req.params.id),
         // Repeated on every heartbeat until the Client reports the new

package/src/config.js

@@ -54,10 +54,13 @@ const DEFAULTS = {
     linkTtlHours: 168
   },
   // New-version check of the Coordinator/Agent/Client packages (README
-  // §10.2). 0 disables the periodic check; registry null = the public npm
-  // registry (or npm_config_registry).
+  // §10.2), every 15 minutes so a new Coordinator's "Update app" button
+  // shows up promptly. 0 disables the periodic check; registry null = the
+  // public npm registry (or npm_config_registry). The old
+  // `checkIntervalHours` (6) is no longer read — installed configs still
+  // carry it from their defaults.
   updates: {
-    checkIntervalHours: 6,
+    checkIntervalMin: 15,
     registry: null
   }
 };

package/src/db/migrations/012_add_uptime_activity.sql

@@ -0,0 +1,7 @@
+-- Reported on every heartbeat (README §10), converted to the
+-- Coordinator's clock: when the Client's host booted (host uptime), and
+-- what the Client is doing since when (JSON {state, jobId, since}; state
+-- idle | job | locked | update-hold) for the "Task / status" duration.
+-- NULL for Clients too old to report them.
+ALTER TABLE resources ADD COLUMN host_booted_at TEXT;
+ALTER TABLE resources ADD COLUMN activity TEXT;

package/src/services/jobs.js

@@ -202,6 +202,16 @@ function createJobsService(db, { bus, events, registry, artifacts, config }){
     return setState(id, JOB_STATES.CANCELED);
   }
 
+  // The job a resource is working on (ASSIGNED through RUNNING), if any —
+  // for the resource card's Cancel button.
+  function activeForResource(resourceId){
+    const placeholders = [...ACTIVE_JOB_STATES].map(() => '?').join(','),
+      row = db.prepare(
+        `SELECT id FROM jobs WHERE resource_id = ? AND state IN (${placeholders}) ORDER BY created_at DESC LIMIT 1`
+      ).get(resourceId, ...ACTIVE_JOB_STATES);
+    return row ? get(row.id) : null;
+  }
+
   // Resource: POST /jobs/:id/result — final verdict from the test runner.
   function applyResult(id, resourceId, { state, exitCode, summary }){
     const job = get(id);
@@ -341,6 +351,7 @@ function createJobsService(db, { bus, events, registry, artifacts, config }){
     create,
     setState,
     cancel,
+    activeForResource,
     applyResult,
     requeueUnacked,
     markLost,

package/src/services/registry.js

@@ -26,7 +26,8 @@ function rowToResource(row){
     labels: JSON.parse(row.labels || '[]'),
     group_ids: JSON.parse(row.group_ids || '[]'),
     host_info: row.host_info ? JSON.parse(row.host_info) : null,
-    capabilities: row.capabilities ? JSON.parse(row.capabilities) : null
+    capabilities: row.capabilities ? JSON.parse(row.capabilities) : null,
+    activity: row.activity ? JSON.parse(row.activity) : null
   };
 }
 
@@ -98,6 +99,26 @@ function sanitizeCapabilities(caps){
   return null;
 }
 
+// Heartbeat durations (README §10) arrive as seconds relative to "now" on
+// the Client and are anchored to the Coordinator's clock here. Anything
+// negative, non-numeric or over ~10 years is ignored.
+const ACTIVITY_STATES = new Set(['idle', 'job', 'locked', 'update-hold']),
+  MAX_DURATION_SEC = 10 * 365 * 86400;
+
+function secondsAgo(now, sec){
+  return Number.isFinite(sec) && sec >= 0 && sec < MAX_DURATION_SEC ? new Date(now - sec * 1000).toISOString() : null;
+}
+
+function sanitizeActivity(now, activity){
+  if (!activity || !ACTIVITY_STATES.has(activity.state)){
+    return null;
+  }
+  const since = secondsAgo(now, activity.durationSec);
+  return since
+    ? { state: activity.state, jobId: activity.state === 'job' ? str(activity.jobId, 64) : null, since }
+    : null;
+}
+
 function createRegistryService(db, { bus, events }){
   function get(id){
     return rowToResource(db.prepare('SELECT * FROM resources WHERE id = ?').get(id));
@@ -266,7 +287,10 @@ function createRegistryService(db, { bus, events }){
   }
 
   // Reconcile status from a heartbeat's self-reported state (§5.1).
-  function heartbeat(resourceId, { state, activeJobId, localLock, metrics, addresses, remoteAddr, clientVersion = null } = {}){
+  function heartbeat(
+    resourceId,
+    { state, activeJobId, localLock, metrics, addresses, remoteAddr, clientVersion = null, hostUptimeSec, activity } = {}
+  ){
     const resource = get(resourceId);
     if (!resource){
       throw Object.assign(new Error('Unknown resource'), { status: 404 });
@@ -298,23 +322,29 @@ function createRegistryService(db, { bus, events }){
       busyReason = null;
     }
 
-    // Older Clients don't send addresses — keep whatever registration stored.
-    const cleanAddresses = sanitizeAddresses(addresses),
+    const now = Date.now(),
+      cleanActivity = sanitizeActivity(now, activity),
+
+      // Older Clients don't send addresses — keep whatever registration stored.
+      cleanAddresses = sanitizeAddresses(addresses),
       hostInfo = cleanAddresses ? { ...(resource.host_info || {}), addresses: cleanAddresses } : resource.host_info;
 
     db.prepare(
       `UPDATE resources
        SET last_heartbeat_at = ?, status = ?, busy_source = ?, busy_reason = ?, host_info = ?,
-           remote_addr = COALESCE(?, remote_addr), client_version = COALESCE(?, client_version)
+           remote_addr = COALESCE(?, remote_addr), client_version = COALESCE(?, client_version),
+           host_booted_at = COALESCE(?, host_booted_at), activity = COALESCE(?, activity)
        WHERE id = ?`
     ).run(
-      new Date().toISOString(),
+      new Date(now).toISOString(),
       nextStatus,
       busySource,
       busyReason,
       hostInfo ? JSON.stringify(hostInfo) : null,
       normalizeRemoteAddr(remoteAddr),
       clientVersion,
+      secondsAgo(now, hostUptimeSec),
+      cleanActivity ? JSON.stringify(cleanActivity) : null,
       resourceId
     );
     completeUpdateIfDone(resourceId);

package/src/services/updates.js

@@ -22,7 +22,7 @@ const fs = require('node:fs'),
 // as root when the dashboard writes an update request (README §10.2).
 const UPDATE_PATH_UNIT = '/etc/systemd/system/thub-coordinator-update.path';
 
-// Latest published versions, refreshed every `updates.checkIntervalHours`
+// Latest published versions, refreshed every `updates.checkIntervalMin`
 // and on demand from the dashboard. Kept in memory only — a restart just
 // checks again.
 function createUpdatesService({ config }){
@@ -96,12 +96,12 @@ function createUpdatesService({ config }){
   }
 
   function start(){
-    const hours = Number(config.updates.checkIntervalHours);
-    if (!(hours > 0)){
+    const minutes = Number(config.updates.checkIntervalMin);
+    if (!(minutes > 0)){
       return;
     }
     checkNow().catch(() => {});
-    const timer = setInterval(() => checkNow().catch(() => {}), hours * 3600 * 1000);
+    const timer = setInterval(() => checkNow().catch(() => {}), minutes * 60 * 1000);
     timer.unref?.();
   }
 

package/src/web/routes.js

@@ -53,6 +53,11 @@ function createWebRouter({ services, config }){
     return typeof target === 'string' && /^\/(?![\/\\])/.test(target) ? target : fallback;
   }
 
+  // The job each resource is on (if any), for the resource card's Cancel.
+  function withActiveJobs(resources){
+    return resources.map((r) => ({ ...r, activeJob: services.jobs.activeForResource(r.id) }));
+  }
+
   router.use((req, res, next) => {
     const user = req.session?.user || null;
     res.locals.user = user;
@@ -69,6 +74,14 @@ function createWebRouter({ services, config }){
     // without this every user would see the *server's* local time — not
     // their own (§10.1). Falls back to UTC for the (rare) pre-login page.
     const tz = user?.timezone || 'UTC';
+    // "3d 4h", "5h 12m", "12m 5s", "8s" — resource card uptime/durations.
+    res.locals.fmtDuration = (ms) => {
+      const s = Math.max(0, Math.floor(ms / 1000)),
+        d = Math.floor(s / 86400),
+        h = Math.floor((s % 86400) / 3600),
+        m = Math.floor((s % 3600) / 60);
+      return d ? `${d}d ${h}h` : h ? `${h}h ${m}m` : m ? `${m}m ${s % 60}s` : `${s}s`;
+    };
     res.locals.fmtDate = (iso, fallback = '—') =>
       iso ? new Date(iso).toLocaleString('en-US', { timeZone: tz }) : fallback;
 
@@ -204,7 +217,7 @@ function createWebRouter({ services, config }){
   });
 
   router.get('/', (req, res) => {
-    const resources = services.registry.list(),
+    const resources = withActiveJobs(services.registry.list()),
       queueLength = services.jobs.list({ state: JOB_STATES.QUEUED, limit: 1000 }).length,
       dayAgo = new Date(Date.now() - 86400 * 1000).toISOString(),
       jobsLast24h = services.jobs
@@ -220,7 +233,7 @@ function createWebRouter({ services, config }){
     res.render('resources/list', {
       title: 'Resources',
       active: 'resources',
-      resources: services.registry.list(),
+      resources: withActiveJobs(services.registry.list()),
       groupsById
     });
   });
@@ -230,6 +243,45 @@ function createWebRouter({ services, config }){
     res.redirect(returnTo(req, '/resources'));
   });
 
+  // Cancel whatever the Client is doing (resource card): its job, a manual
+  // local lock, or a self-update hold (README §10). The Client applies the
+  // lock/update ones via heartbeat commands; a job is canceled right here.
+  router.post('/resources/:id/cancel-activity', requireAdminRole, (req, res) => {
+    const r = services.registry.get(req.params.id),
+      job = r && services.jobs.activeForResource(r.id),
+      activity = r?.activity?.state;
+    try {
+      if (!r){
+        throw Object.assign(new Error('Unknown resource'), { status: 404 });
+      }
+      if (job){
+        services.jobs.cancel(job.id, { isAdmin: true });
+        flash(req, 'warning', `Canceled job ${job.id} on ${r.name}.`);
+      }
+      else if (activity === 'locked'){
+        services.commands.push(r.id, { command: 'unlock' });
+        flash(req, 'warning', `Releasing the local lock on ${r.name} (on its next heartbeat).`);
+      }
+      else if (activity === 'update-hold'){
+        services.commands.push(r.id, { command: 'cancel-update' });
+        if (r.update_to){
+          services.registry.setUpdateTo(r.id, null);
+        }
+        flash(req, 'warning', `Canceling the self-update on ${r.name}'s host (on its next heartbeat).`);
+      }
+      else {
+        flash(req, 'info', `${r.name} isn't doing anything to cancel.`);
+      }
+      if (!job && ['locked', 'update-hold'].includes(activity)){
+        services.events.record('resource', r.id, 'resource.activity_canceled', { activity, by: req.session.user.username });
+      }
+    }
+    catch (err){
+      flash(req, 'danger', err.message);
+    }
+    res.redirect(returnTo(req, '/resources'));
+  });
+
   router.post('/resources/:id/rotate-token', requireAdminRole, (req, res) => {
     const { generateToken, hashToken } = require('../services/tokens');
     const token = generateToken('res');

package/views/mixins/resource.pug

@@ -89,6 +89,56 @@ mixin capabilitiesList(caps)
         if caps.hw.power.method === 'uhubctl' && caps.hw.power.hub
           span.text-body-secondary.small.ms-2= `hub ${caps.hw.power.hub} port ${caps.hw.power.port}`
 
+//- Host uptime and the Client's current task/status with its duration,
+//- both from the last heartbeat. Measured up to now — or, for a Client
+//- that stopped heartbeating, up to its last heartbeat.
+mixin uptimeRows(r)
+  - const stale = r.status === 'OUT_OF_SERVICE' && r.last_heartbeat_at
+  - const ref = stale ? new Date(r.last_heartbeat_at).getTime() : Date.now()
+  - const activityLabel = { idle: 'Idle', job: 'Running job', locked: 'Locked locally', 'update-hold': 'Holding for self-update' }
+  dt.col-sm-4 Host uptime
+  dd.col-sm-8
+    if r.host_booted_at
+      = fmtDuration(ref - new Date(r.host_booted_at).getTime())
+      span.text-body-secondary.small.ms-2= `booted ${fmtDate(r.host_booted_at)}`
+      if stale
+        span.text-body-secondary.small.ms-1 (at last heartbeat)
+    else
+      span.text-body-secondary —
+  dt.col-sm-4 Task / status
+  dd.col-sm-8
+    if r.activity
+      = activityLabel[r.activity.state] || r.activity.state
+      if r.activity.jobId
+        a.ms-1(href=`/jobs/${r.activity.jobId}`)= r.activity.jobId
+      = ` for ${fmtDuration(ref - new Date(r.activity.since).getTime())}`
+      span.text-body-secondary.small.ms-2= `since ${fmtDate(r.activity.since)}`
+      if stale
+        span.text-body-secondary.small.ms-1 (at last heartbeat)
+    else
+      span.text-body-secondary —
+
+//- Cancel the Client's current activity (its job, a local lock or a
+//- self-update hold); disabled while it's idle.
+mixin cancelActivityButton(r, returnTo)
+  - const state = r.activity && r.activity.state
+  - const what = r.activeJob ? `job ${r.activeJob.id}` : state === 'locked' ? 'the local lock' : state === 'update-hold' ? 'the self-update' : null
+  if what
+    form.d-inline.ms-1(
+      method="post"
+      action=`/resources/${r.id}/cancel-activity`
+      onsubmit=`return confirm('Cancel ${what} on ${r.name}?')`
+    )
+      input(type="hidden" name="returnTo" value=returnTo)
+      button.btn.btn-sm.btn-outline-danger(type="submit" data-bs-toggle="tooltip" data-bs-title=`Cancel ${what}`)
+        i.bi.bi-x-octagon.me-1
+        | Cancel
+  else
+    span.d-inline-block.ms-1(tabindex="0" data-bs-toggle="tooltip" data-bs-title="Nothing to cancel — the Client is idle")
+      button.btn.btn-sm.btn-outline-danger(type="button" disabled style="pointer-events: none;")
+        i.bi.bi-x-octagon.me-1
+        | Cancel
+
 mixin resourceCard(r, groupsById, returnTo)
   - const isLoopback = (a) => /^127\./.test(a) || a === '::1' || a === 'localhost'
   - const local = ((r.host_info && r.host_info.addresses) || []).filter((a) => !isLoopback(a.address))
@@ -107,6 +157,7 @@ mixin resourceCard(r, groupsById, returnTo)
             dd.col-sm-8.text-uppercase= r.type
             dt.col-sm-4 Status
             dd.col-sm-8: +resourceBadge(r.status)
+            +uptimeRows(r)
             dt.col-sm-4 Client version
             dd.col-sm-8: +versionCell(r.client_version, updates.latest.client, r.update_to)
             dt.col-sm-4 IP addresses
@@ -150,3 +201,4 @@ mixin resourceCard(r, groupsById, returnTo)
               dt.col-sm-4 Actions
               dd.col-sm-8.mb-0
                 +resourceActions(r, returnTo)
+                +cancelActivityButton(r, returnTo)