@vzn/vx-dte 0.0.649 → 0.0.651

package/README.md

@@ -88,7 +88,7 @@ before sealing, since no relay or socket compresses what is sealed. It
 is padded with trailing spaces to its size class, a power of two from
 256 bytes, then whole transport messages, so the relay learns no size
 within a class. Inside, each send is numbered and acknowledged
-end to end, and a quiet peer probed (Switchboard § Delivery, § Liveness).
+end to end, and a quiet worker reports (Switchboard § Delivery, § Liveness).
 
 ## Try it
 
@@ -204,7 +204,8 @@ every 30 s, asks with each answer's ETag (an unchanged answer is free),
 and asks nothing until a spent limit resets. A dispatch the limit refuses
 is sent again at the reset, if that comes within 5 minutes; past that it
 fails, naming when. A cancel the limit refuses is sent again at the
-reset, if that comes within a minute. A run that dies before it cancels
+reset, if that comes within a minute, unless a worker given up as late
+joined meanwhile: the run took it, so its workflow run goes on. A run that dies before it cancels
 (a cancelled or timed-out job) leaves it to the job's last step, which
 cancels each worker run the run listed in `RUNNER_TEMP`. Past both, a
 worker that starts reads its run's attempt (`parent`) first, and once
@@ -456,9 +457,11 @@ use your own instead.
 - A worker reports the cores and memory its machine has when it joins.
   One the run started is held to the size it was started as, so
   `local()` workers sharing a host split it rather than each take all of
-  it. Each frame it sends (a task begun, a command ended, a task saved,
-  its heartbeat) carries its machine's load: cores busy over the last
-  second and memory in use. A task goes to it only when it fits now: while
+  it. Each frame it sends (a task saved, its report, a fault) carries its
+  machine's load: cores busy over the last second and memory in use. A
+  heartbeat with nothing else to say, it sends its report: that load, the
+  tasks it runs and the last batch it read. A task of that batch or before
+  its report no longer names goes back to the queue at once. A task goes to it only when it fits now: while
   one of its cores is free and the task's memory fits. What a worker has
   in use is what its running tasks used on their last run (history's
   measured CPU and peak RAM), or its live load plus what was sent since
@@ -535,10 +538,10 @@ use your own instead.
   leaves no file. One waits there up to 10 s; past that, with that worker
   gone or of another platform, or with its uncached upstreams on two
   workers, the upstream runs again where the dependant can go, and the
-  dependant follows it. A task whose upload no remote took (too big, a 429) is kept the same way: its output comes to the orchestrator with
-  its result, its files stay on the worker, and a task on the
-  orchestrator's machine that needs them fails saying so. Before a task
-  runs, its
+  dependant follows it. A task whose upload no remote took after three
+  tries (too big, a 429, a reset connection) is its worker's fault: that
+  worker takes no more tasks and the task runs on another; a second fault
+  fails the run. Before a task runs, its
   worker removes the files under its outputs that an earlier task left,
   except committed files, nested projects' files, and its dependencies'
   outputs (an additive task builds on its upstream's tree). An upstream's
@@ -576,8 +579,8 @@ use your own instead.
   `timeouts`).
 - A worker that stops answering, stops saying it holds a task, or cannot
   run one (a full disk) gets no more work, and its tasks run again on
-  another. A task two such workers had begun fails, naming both; one an
-  worker had not begun yet does not count. With no worker of the run left
+  another. A task two such workers were sent fails, naming both; one whose
+  send failed does not count. With no worker of the run left
   to take it and none starting, it fails at once.
 - A relay that answers nothing for 10 s, every try counted, fails every task not done
   (`the relay is unreachable`); none runs here, which may be far smaller
@@ -587,8 +590,8 @@ use your own instead.
 - The relay stores nothing, so a restart forgets the run. The orchestrator
   joins again as its host and each worker joins again as itself, for two
   heartbeats (4 s on the default); tasks keep running meanwhile, and what
-  either side sent goes again once both are back, taken once. Leases
-  wait while either is away, and a worker sent a task it runs already
+  either side sent goes again once both are back, taken once. A worker
+  keeps what it runs, and one sent a task it runs already
   does not run it again, so no work repeats. One not back by then
   ends: a worker's tasks run elsewhere, an orchestrator's run fails.
   Joining has the same 10 s in all, its socket and request together: a
@@ -611,22 +614,32 @@ use your own instead.
   orchestrator's at its run's end is the final call that closes the run,
   and a worker's when it is done. Either whose socket drops without it
   dials again, 3 times within 10 s, and the relay keeps it, and its run,
-  for those 10 s; past them the run is over. A peer quiet on a socket that never
-  closed is lost within two heartbeats (4 s on the default): only a
-  peer heard from for none of a heartbeat and a half is probed (an
-  empty numbered send), which it acks at once; either side hearing
-  nothing from the other for 1.9 heartbeats loses it, a worker by
-  ending. A quiet pair trades one probe and its ack each heartbeat and a
-  half, the relay answers neither, and nobody pings. An orchestrator
+  for those 10 s; past them the run is over. Either side hearing nothing
+  from the other for 3 heartbeats (6 s on the default) loses it, and the
+  rest follows from that: a worker that said nothing for half of it (3 s)
+  sends its report; the orchestrator never speaks on a timer, only orders
+  and the acks of what it heard; a send unacknowledged for 3 s goes again.
+  A worker that loses its orchestrator ends, and says so. Nobody probes, the relay answers
+  neither, and nobody pings. An orchestrator
   with no worker yet asks the relay an empty frame instead; a frame the
   relay leaves unanswered a heartbeat closes the socket, which is dialed
   again.
+- The orchestrator's socket, its acks and redials run on a thread
+  of their own, so sync work on the run's loop never holds them (one CI
+  run went quiet 12 s that way and lost every worker). A loop held a
+  second or more says so, with how long; one held 5 heartbeats (10 s)
+  ends the run, as a dead orchestrator would.
 - Each send is numbered for its receiver and held until the receiver
   acknowledges it, so one a socket or the relay dropped goes again after
-  a heartbeat, or at once on a new socket; the receiver takes each once,
-  in order. An ack rides the next send back, and those with none to ride
-  go together in one frame per half heartbeat: no message of its own on
-  a busy run.
+  a heartbeat and a half, or at once on a new socket; the receiver takes
+  each once, in order. An ack rides the next send back; the orchestrator's,
+  with no order to ride, go together after half a heartbeat, except a
+  quiet worker's beat, acked at once.
+- A task sent runs at once: no message says it began. A command that
+  ended (`ran`, with when it began and ended by the worker's clock) rides
+  the worker's next frame, never alone; a task that saves first says it
+  in its `end` (begun, ran, ended). The run reads those times against the
+  clock of the frame that carried them.
 - A run where no task began or ended for a minute warns what it waits
   on, a line a task: which worker, and whether it runs or saves.
 - A message the relay would not take is a warning naming it, its
@@ -651,9 +664,9 @@ An idle worker the rest of the run cannot keep busy leaves (`left`): one
 beyond the peak of the cores the remaining tasks keep busy, so a lull
 between two wide parts keeps it. One a remaining task needs for its size
 stays, and so does one a later task needs when its size's `max` is
-spent, since none could start in its place. A lost worker's begun tasks run again on
+spent, since none could start in its place. A lost worker's tasks run again on
 another, and the run starts one in its place. A worker is lost when its
-socket closes, when it stops answering the orchestrator's probes, or at
+socket closes, when it says nothing for 3 heartbeats (6 s), or at
 once when its adapter sees its process or job end. Workers start at the first
 task the cache does not answer, so a warm run starts none. Once one has
 joined, its warm-up is what the next of its size takes at the least: work
@@ -670,10 +683,10 @@ run is over, one still starting is `not joined`), the tasks it
 has done, failed or had aborted by a stop, and the ones it runs now; while the run goes, those that left
 are one line (`3 workers  left  9 done`). A worker the
 run started is named by its size and the order it was started in; one
-that joined on its own is `worker#N`. A task sent to a worker but not yet
-begun there has no row; the tasks line counts it `scheduled`
+that joined on its own is `worker#N`. A task waiting for a worker has no
+row; the tasks line counts it `scheduled`
 (`30 success · 4 running · 12 scheduled · 62 total`). Its row and clock
-start when the worker begins it. A worker is `online` while the run
+start when it is sent: a worker runs what it is sent at once. A worker is `online` while the run
 goes; it ends `left` when the run let it go as not needed, `lost` when
 it crashed or went silent, and `stopped` when the run's end stopped it.
 Once the run is over, each worker adds its warm-up (from asking the

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@vzn/vx-dte",
-  "version": "0.0.649",
+  "version": "0.0.651",
   "license": "MIT",
   "description": "Distributed task execution for vx: one run spread over workers your own CI spawns, joined through a relay.",
   "keywords": [
@@ -29,10 +29,10 @@
     "bun": ">=1.4.4"
   },
   "dependencies": {
-    "@vzn/vx-dte-server": "0.0.649"
+    "@vzn/vx-dte-server": "0.0.651"
   },
   "peerDependencies": {
-    "@vzn/vx": "^0.0.649"
+    "@vzn/vx": "^0.0.651"
   },
   "repository": {
     "type": "git",

package/src/github.ts

@@ -138,9 +138,10 @@ export function github(options: GithubOptions = {}): DteAdapter {
    * else is said. One the spent API limit refused is asked again at its
    * reset, when that comes within `CANCEL_WAIT_MS`.
    */
-  const cancelOf = async (run: string): Promise<void> => {
+  const cancelOf = async (run: string, still: () => boolean = () => true): Promise<void> => {
     const end = Date.now() + CANCEL_WAIT_MS
     for (;;) {
+      if (!still()) return
       const res = await call(`${run}/cancel`, { method: 'POST' }).catch((err: unknown) => {
         failed(`cancelling ${runOf(run)}`, err instanceof Error ? err.message : String(err))
         return undefined
@@ -227,7 +228,9 @@ export function github(options: GithubOptions = {}): DteAdapter {
           return
         }
         for (const id of ids) decide(id, false)
-        return cancelOf(run)
+        // Asked again at an API limit's reset, it stops once the run took
+        // one of them after all (`kept`).
+        return cancelOf(run, () => ids.every((id) => cancelled.has(id)))
       })
       .finally(() => pending.delete(p))
     pending.add(p)
@@ -373,10 +376,13 @@ export function github(options: GithubOptions = {}): DteAdapter {
         throw new Error(`github(): dispatching ${workflow}: ${res.status} ${text}${fix}`)
       }
     }
-    const id =
+    // quiet: a body that is not JSON names no run, refused below
+    const said =
       res.status === 200
-        ? ((await res.json()) as { workflow_run_id?: number }).workflow_run_id
+        ? ((await res.json().catch(() => null)) as { workflow_run_id?: unknown } | null)
+            ?.workflow_run_id
         : undefined
+    const id = Number.isSafeInteger(said) && (said as number) > 0 ? (said as number) : undefined
     // An API naming no run (a 204) is refused, not trusted by repository
     // alone: any job of it with id-token: write would join as a worker.
     if (id === undefined) {
@@ -472,6 +478,11 @@ export function github(options: GithubOptions = {}): DteAdapter {
       for (const id of gone) decide(id, false)
       return Promise.all(left).then((l) => ids.filter((_, i) => l[i]))
     },
+    // One given up as late joined after all: its run is not cancelled.
+    kept(id) {
+      late.delete(id)
+      cancelled.delete(id)
+    },
     // A finished run answers 409; one still queued or running is the
     // run's no more: its results are in, or the run failed without them.
     async close() {
@@ -481,7 +492,7 @@ export function github(options: GithubOptions = {}): DteAdapter {
       await Promise.all(pending)
       for (const id of [...deciding.keys()]) decide(id, false)
       // A cancel that fails leaves the run to its timeout, and says so.
-      await Promise.all(runs.splice(0).map(cancelOf))
+      await Promise.all(runs.splice(0).map((run) => cancelOf(run)))
     },
   }
 }

package/src/messages.ts

@@ -2,15 +2,7 @@
 // (the relay never reads it): orders and status only. Every task's files
 // and output go through the workspace's cache, by the keys these name.
 
-import type { CacheFacts, CapturedChunk, ExecuteResult, SandboxConfig } from '@vzn/vx'
-
-/**
- * A worker says what it holds every this many heartbeats, and a task's
- * lease is three of them. Every heartbeat (2026-10-25: the fastest, not
- * a free plan's message count): a task its worker dropped while alive
- * goes back in 15 s, not 90.
- */
-export const HELD_EVERY = 1
+import type { CacheFacts, ExecuteResult, SandboxConfig } from '@vzn/vx'
 
 /** A worker's `join` info. */
 export interface WorkerInfo {
@@ -100,9 +92,13 @@ export interface ServerOrder {
 }
 
 export type ToWorker =
-  /** Tasks that fit what it has free, each begun at once; `root` is swapped for the worker's in `env`. */
+  /**
+   * Tasks that fit what it has free: taken, each runs at once. `n` counts
+   * its batches, so its report says which it had read.
+   */
   | {
       type: 'batch'
+      n: number
       root: string
       /** `root`'s real path, as sandbox paths spell it. */
       realRoot: string
@@ -146,26 +142,48 @@ export interface Load {
    * `cpu` (one that sent 3, each as a task started, read 0.6 of 2.0 used).
    */
   readonly avg?: number
+  /** When it was read, by the worker's clock: its task times are read against it. */
+  readonly at?: number
+}
+
+/** A task's times on its worker, epoch ms by the worker's clock: begun, command ended. */
+export interface Ran {
+  readonly startedAt: number
+  readonly ranAt: number
 }
 
 export type ToOrchestrator =
-  /** It began the task. */
-  | { type: 'start'; task: string; load?: Load }
-  /** Its command is done; its outputs save now, its cores are free. */
-  | { type: 'ran'; task: string; load?: Load }
-  /** Each heartbeat while it runs any: what it runs, which renews its lease on each. */
-  | { type: 'held'; running: string[]; load?: Load }
+  /**
+   * Its command is done; its outputs save now, its cores are free. Never
+   * sent alone: it rides the next message, or its `end` says it.
+   */
+  | ({ type: 'ran'; task: string; load?: Load } & Ran)
+  /**
+   * A heartbeat with nothing else said: what it runs, the last batch it
+   * read, its live load. A task of that batch or before it no longer
+   * names goes back.
+   */
+  | { type: 'report'; running: string[]; batch: number; load: Load }
+  /** Messages said at one moment, as one frame: waiting `ran`s and the event they ride. */
+  | { type: 'together'; messages: ToOrchestrator[] }
   /** It could not run a task it began, not a non-zero exit: a full disk, a failed unpack. */
-  | { type: 'fault'; task: string; reason: string }
+  | { type: 'fault'; task: string; reason: string; load?: Load }
   /** It ran the task and saved it under `key`; its output is in that entry. */
-  | { type: 'end'; task: string; key: string; result: WiredResult; load?: Load }
+  | ({
+      type: 'end'
+      task: string
+      key: string
+      result: WiredResult
+      endedAt: number
+      load?: Load
+    } & Ran)
   /**
    * Ahead of an `end` too long for one relay frame, some of its lists:
    * the `end` that follows adds its own to them. One `end` with a sandboxed
    * task's 9000 violations was over the relay's 1 MiB, and the run waited
    * for it (2026-10-25).
    */
-  | ({ type: 'part'; task: string } & Partial<Pick<WiredResult, PartList>>)
+  | ({ type: 'part'; task: string; load?: Load } & Partial<Pick<WiredResult, PartList>>)
 
 /**
  * A message's JSON in UTF-8 bytes, what a relay frame holds: counted in
@@ -175,7 +193,7 @@ export type ToOrchestrator =
 export const bytesOf = (body: unknown): number => Buffer.byteLength(JSON.stringify(body))
 
 /** The lists a long result sends ahead of its `end`, in parts. */
-export const PART_LISTS = ['violations', 'changed', 'removed', 'output'] as const
+export const PART_LISTS = ['violations', 'changed', 'removed'] as const
 export type PartList = (typeof PART_LISTS)[number]
 
 /** A result as JSON carries it: its output stays in the cache entry, and a violation's time is a string. */
@@ -187,13 +205,6 @@ export type WiredResult = Omit<ExecuteResult, 'violations' | 'outputs' | 'stdout
   removed?: string[]
   /** Dependency keys whose remote copy did not read back: this machine sends its own over them. */
   unread?: string[]
-  /**
-   * No remote took its entry's upload (too big, a 429): its files are on
-   * that worker alone, and its output comes here in `output`.
-   */
-  unshared?: true
-  /** Its output, both streams in the order written, when `unshared`. */
-  output?: CapturedChunk[]
 }
 
 /**