@vzn/vx-dte 0.0.650 → 0.0.652

package/README.md

@@ -88,7 +88,7 @@ before sealing, since no relay or socket compresses what is sealed. It
 is padded with trailing spaces to its size class, a power of two from
 256 bytes, then whole transport messages, so the relay learns no size
 within a class. Inside, each send is numbered and acknowledged
-end to end, and a quiet peer probed (Switchboard § Delivery, § Liveness).
+end to end, and a quiet worker reports (Switchboard § Delivery, § Liveness).
 
 ## Try it
 
@@ -204,7 +204,8 @@ every 30 s, asks with each answer's ETag (an unchanged answer is free),
 and asks nothing until a spent limit resets. A dispatch the limit refuses
 is sent again at the reset, if that comes within 5 minutes; past that it
 fails, naming when. A cancel the limit refuses is sent again at the
-reset, if that comes within a minute. A run that dies before it cancels
+reset, if that comes within a minute, unless a worker given up as late
+joined meanwhile: the run took it, so its workflow run goes on. A run that dies before it cancels
 (a cancelled or timed-out job) leaves it to the job's last step, which
 cancels each worker run the run listed in `RUNNER_TEMP`. Past both, a
 worker that starts reads its run's attempt (`parent`) first, and once
@@ -456,9 +457,11 @@ use your own instead.
 - A worker reports the cores and memory its machine has when it joins.
   One the run started is held to the size it was started as, so
   `local()` workers sharing a host split it rather than each take all of
-  it. Each frame it sends (a task begun, a command ended, a task saved,
-  its heartbeat) carries its machine's load: cores busy over the last
-  second and memory in use. A task goes to it only when it fits now: while
+  it. Each frame it sends (a task saved, its report, a fault) carries its
+  machine's load: cores busy over the last second and memory in use. A
+  heartbeat with nothing else to say, it sends its report: that load, the
+  tasks it runs and the last batch it read. A task of that batch or before
+  its report no longer names goes back to the queue at once. A task goes to it only when it fits now: while
   one of its cores is free and the task's memory fits. What a worker has
   in use is what its running tasks used on their last run (history's
   measured CPU and peak RAM), or its live load plus what was sent since
@@ -522,6 +525,8 @@ use your own instead.
   or of files a formatter changed) is sent in parts, none of it dropped.
   An order too long for one frame (a task with thousands of deps) sends
   its deps ahead of it the same way.
+- A worker gets each dependency in full once. Later orders name it, so a
+  deep graph's orders stay small.
 - A task runs with the orchestrator's environment under the worker's
   root. `HOME`, `TMPDIR`, `USER` and the like are the worker's own, unless
   the task's `env.define` sets them; `npm_execpath` is the workspace's
@@ -535,10 +540,10 @@ use your own instead.
   leaves no file. One waits there up to 10 s; past that, with that worker
   gone or of another platform, or with its uncached upstreams on two
   workers, the upstream runs again where the dependant can go, and the
-  dependant follows it. A task whose upload no remote took (too big, a 429) is kept the same way: its output comes to the orchestrator with
-  its result, its files stay on the worker, and a task on the
-  orchestrator's machine that needs them fails saying so. Before a task
-  runs, its
+  dependant follows it. A task whose upload no remote took after three
+  tries (too big, a 429, a reset connection) is its worker's fault: that
+  worker takes no more tasks and the task runs on another; a second fault
+  fails the run. Before a task runs, its
   worker removes the files under its outputs that an earlier task left,
   except committed files, nested projects' files, and its dependencies'
   outputs (an additive task builds on its upstream's tree). An upstream's
@@ -576,9 +581,14 @@ use your own instead.
   `timeouts`).
 - A worker that stops answering, stops saying it holds a task, or cannot
   run one (a full disk) gets no more work, and its tasks run again on
-  another. A task two such workers had begun fails, naming both; one an
-  worker had not begun yet does not count. With no worker of the run left
+  another. A task two such workers were sent fails, naming both; one whose
+  send failed does not count. With no worker of the run left
   to take it and none starting, it fails at once.
+- A task that failed and retries (its `retries`, or the one retry a
+  known-flaky task gets) runs on another worker than the attempt that
+  failed, as core asks (`ExecuteRequest.attempt`): the worker, or what
+  ran beside it, may be the flake. With no other worker of its platform
+  in the run, it runs there again rather than wait.
 - A relay that answers nothing for 10 s, every try counted, fails every task not done
   (`the relay is unreachable`); none runs here, which may be far smaller
   than the workers. What finished is saved, so a rerun resumes from it.
@@ -587,8 +597,8 @@ use your own instead.
 - The relay stores nothing, so a restart forgets the run. The orchestrator
   joins again as its host and each worker joins again as itself, for two
   heartbeats (4 s on the default); tasks keep running meanwhile, and what
-  either side sent goes again once both are back, taken once. Leases
-  wait while either is away, and a worker sent a task it runs already
+  either side sent goes again once both are back, taken once. A worker
+  keeps what it runs, and one sent a task it runs already
   does not run it again, so no work repeats. One not back by then
   ends: a worker's tasks run elsewhere, an orchestrator's run fails.
   Joining has the same 10 s in all, its socket and request together: a
@@ -611,22 +621,32 @@ use your own instead.
   orchestrator's at its run's end is the final call that closes the run,
   and a worker's when it is done. Either whose socket drops without it
   dials again, 3 times within 10 s, and the relay keeps it, and its run,
-  for those 10 s; past them the run is over. A peer quiet on a socket that never
-  closed is lost within two heartbeats (4 s on the default): only a
-  peer heard from for none of a heartbeat and a half is probed (an
-  empty numbered send), which it acks at once; either side hearing
-  nothing from the other for 1.9 heartbeats loses it, a worker by
-  ending. A quiet pair trades one probe and its ack each heartbeat and a
-  half, the relay answers neither, and nobody pings. An orchestrator
+  for those 10 s; past them the run is over. Either side hearing nothing
+  from the other for 3 heartbeats (6 s on the default) loses it, and the
+  rest follows from that: a worker that said nothing for half of it (3 s)
+  sends its report; the orchestrator never speaks on a timer, only orders
+  and the acks of what it heard; a send unacknowledged for 3 s goes again.
+  A worker that loses its orchestrator ends, and says so. Nobody probes, the relay answers
+  neither, and nobody pings. An orchestrator
   with no worker yet asks the relay an empty frame instead; a frame the
   relay leaves unanswered a heartbeat closes the socket, which is dialed
   again.
+- The orchestrator's socket, its acks and redials run on a thread
+  of their own, so sync work on the run's loop never holds them (one CI
+  run went quiet 12 s that way and lost every worker). A loop held a
+  second or more says so, with how long; one held 5 heartbeats (10 s)
+  ends the run, as a dead orchestrator would.
 - Each send is numbered for its receiver and held until the receiver
   acknowledges it, so one a socket or the relay dropped goes again after
-  a heartbeat, or at once on a new socket; the receiver takes each once,
-  in order. An ack rides the next send back, and those with none to ride
-  go together in one frame per half heartbeat: no message of its own on
-  a busy run.
+  a heartbeat and a half, or at once on a new socket; the receiver takes
+  each once, in order. An ack rides the next send back; the orchestrator's,
+  with no order to ride, go together after half a heartbeat, except a
+  quiet worker's beat, acked at once.
+- A task sent runs at once: no message says it began. A command that
+  ended (`ran`, with when it began and ended by the worker's clock) rides
+  the worker's next frame, never alone; a task that saves first says it
+  in its `end` (begun, ran, ended). The run reads those times against the
+  clock of the frame that carried them.
 - A run where no task began or ended for a minute warns what it waits
   on, a line a task: which worker, and whether it runs or saves.
 - A message the relay would not take is a warning naming it, its
@@ -651,9 +671,9 @@ An idle worker the rest of the run cannot keep busy leaves (`left`): one
 beyond the peak of the cores the remaining tasks keep busy, so a lull
 between two wide parts keeps it. One a remaining task needs for its size
 stays, and so does one a later task needs when its size's `max` is
-spent, since none could start in its place. A lost worker's begun tasks run again on
+spent, since none could start in its place. A lost worker's tasks run again on
 another, and the run starts one in its place. A worker is lost when its
-socket closes, when it stops answering the orchestrator's probes, or at
+socket closes, when it says nothing for 3 heartbeats (6 s), or at
 once when its adapter sees its process or job end. Workers start at the first
 task the cache does not answer, so a warm run starts none. Once one has
 joined, its warm-up is what the next of its size takes at the least: work
@@ -670,10 +690,10 @@ run is over, one still starting is `not joined`), the tasks it
 has done, failed or had aborted by a stop, and the ones it runs now; while the run goes, those that left
 are one line (`3 workers  left  9 done`). A worker the
 run started is named by its size and the order it was started in; one
-that joined on its own is `worker#N`. A task sent to a worker but not yet
-begun there has no row; the tasks line counts it `scheduled`
+that joined on its own is `worker#N`. A task waiting for a worker has no
+row; the tasks line counts it `scheduled`
 (`30 success · 4 running · 12 scheduled · 62 total`). Its row and clock
-start when the worker begins it. A worker is `online` while the run
+start when it is sent: a worker runs what it is sent at once. A worker is `online` while the run
 goes; it ends `left` when the run let it go as not needed, `lost` when
 it crashed or went silent, and `stopped` when the run's end stopped it.
 Once the run is over, each worker adds its warm-up (from asking the
@@ -733,6 +753,9 @@ machine together, and the share of their history the run's tasks used:
             probe    at 0.4s: 5 tasks left to run, 5 known misses
             asked    1 at 0.5s for 5 tasks (5 measured), plan 12.0s
             ideal    10.8s: each task as long as it took, placed perfectly
+            best     2 × small (4c/14G): 11.4s, the fewest of its sizes
+            more     3 × small (4c/14G): 10.8s, past its sizes' max
+            any      1 × (8c/28G): 10.9s, of any cores, the least core time
             total    cpu utilization 58%
 ```
 
@@ -763,6 +786,17 @@ real warm-up. That ideal end (`TaskExecutor.idealMs`) sits beside the
 real time in the run summary and the footer's `ideal` line, and
 `@vzn/vx-ci` prints it.
 
+The same replay says which workers to declare next time
+(`TaskExecutor.advice`), each from an empty run with the plan's own
+rules (the fewest that end it within its margin of the soonest): `best`
+is how many of each size; `more`, shown only when more than a size's
+`max` would end the run sooner past that margin, is how many it would
+take; `any` is one shape for each platform, of 2 to 64 cores (memory a
+core as the platform's sizes have most), the one ending as soon for the
+least core time. Raise a size's `max`, or change its `cpu` and
+`memory`, by them. They are in the footer and the run summary, and
+`@vzn/vx-ci` prints them under "Workers to declare".
+
 ## Environment
 
 What `@vzn/vx-dte` reads from the CI job it runs in:
@@ -804,16 +838,26 @@ process.env.CF_ACCESS_CLIENT_ID, … }`); an unset value is left out, and
     absent, planned as this machine's and not checked on join). With no adapter, the sizes' platforms are
     the ones joining workers may have. Only work counts:
     a task that missed the cache, or waits on a cached one that did, and
-    every uncached task, which runs every time; groups and hits need no worker. At each miss, finish and join the run plays
+    every uncached task, which runs every time; groups and hits need no worker. The run plays
     that work out on its workers and on N more (`src/planner.ts`), each new
     one joining after its size's warm-up: this run's first of that size,
     else an earlier run's, else any size's this run, else 20 s,
     and takes the fewest N that end within 5% (5 s at least) of the best.
+    It plays only two warm-ups and the longest task ahead; past that, the
+    rest ends no sooner than its longest chain, nor than its work spread
+    on the cores it has, and the next plan, a warm-up on, sees further.
     Sizes of different platforms come in turns, so of those N it drops
     each the plan ends as soon without, its platform's work no more than
     5 s later.
     That margin is spent once: later plans keep its deadline until new
     work is known. A task waiting on one that runs waits until it ends.
+    Then the run follows that plan: a start it wants later is asked when
+    due, and a worker idle with its planned work done goes. It plays again
+    only when reality leaves the plan: new work, work that leaves without
+    running, a worker that joins, leaves or warms up off plan, a task past
+    its time or ending sooner, one of no known time ending, a task starting
+    with its time, CPU or memory more than 5% off (1 s, 0.05 cores, 64 MiB
+    at least), or its window moving on.
     What is left, running tasks too, takes as many times its history as
     the run's finished tasks took theirs (once they had 20 s of it; never
     less): a task past its forecast would otherwise be planned to end now,
@@ -863,7 +907,7 @@ HEAD^`), and every task after one (`src/forecast.ts`): a lockfile or
   - `adapter` is the `DteAdapter` that starts workers, of the sizes in
     `workers`: at least one with `max` above 0, or `dte()` is refused.
     Absent, the run starts none and uses the workers that join `relay`.
-  - `relay` is the relay's URL, http(s); absent, the adapter's own.
+  - `relay` is the relay's base URL, http(s), without `/v1`, `?query` or `#fragment`; absent, the adapter's own.
   - `token` answers the relay's bearer token (default `VX_DTE_TOKEN`). It
     is asked again a minute before the token's `exp`, when it has one.
   - `unmeasured`, `{ cpu, memory }` (default 2 cores, `'4GB'`), is what a

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@vzn/vx-dte",
-  "version": "0.0.650",
+  "version": "0.0.652",
   "license": "MIT",
   "description": "Distributed task execution for vx: one run spread over workers your own CI spawns, joined through a relay.",
   "keywords": [
@@ -29,10 +29,10 @@
     "bun": ">=1.4.4"
   },
   "dependencies": {
-    "@vzn/vx-dte-server": "0.0.650"
+    "@vzn/vx-dte-server": "0.0.652"
   },
   "peerDependencies": {
-    "@vzn/vx": "^0.0.650"
+    "@vzn/vx": "^0.0.652"
   },
   "repository": {
     "type": "git",