@vzn/vx-dte 0.0.651 → 0.0.652
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +33 -2
- package/package.json +3 -3
- package/src/dispatch.ts +501 -26
- package/src/forecast.ts +1 -2
- package/src/messages.ts +14 -4
- package/src/orchestrator.ts +415 -202
- package/src/planner.ts +1322 -172
- package/src/plugin.ts +33 -3
- package/src/wire.ts +36 -8
- package/src/worker.ts +26 -4
package/README.md
CHANGED
|
@@ -525,6 +525,8 @@ use your own instead.
|
|
|
525
525
|
or of files a formatter changed) is sent in parts, none of it dropped.
|
|
526
526
|
An order too long for one frame (a task with thousands of deps) sends
|
|
527
527
|
its deps ahead of it the same way.
|
|
528
|
+
- A worker gets each dependency in full once. Later orders name it, so a
|
|
529
|
+
deep graph's orders stay small.
|
|
528
530
|
- A task runs with the orchestrator's environment under the worker's
|
|
529
531
|
root. `HOME`, `TMPDIR`, `USER` and the like are the worker's own, unless
|
|
530
532
|
the task's `env.define` sets them; `npm_execpath` is the workspace's
|
|
@@ -582,6 +584,11 @@ use your own instead.
|
|
|
582
584
|
another. A task two such workers were sent fails, naming both; one whose
|
|
583
585
|
send failed does not count. With no worker of the run left
|
|
584
586
|
to take it and none starting, it fails at once.
|
|
587
|
+
- A task that failed and retries (its `retries`, or the one retry a
|
|
588
|
+
known-flaky task gets) runs on another worker than the attempt that
|
|
589
|
+
failed, as core asks (`ExecuteRequest.attempt`): the worker, or what
|
|
590
|
+
ran beside it, may be the flake. With no other worker of its platform
|
|
591
|
+
in the run, it runs there again rather than wait.
|
|
585
592
|
- A relay that answers nothing for 10 s, every try counted, fails every task not done
|
|
586
593
|
(`the relay is unreachable`); none runs here, which may be far smaller
|
|
587
594
|
than the workers. What finished is saved, so a rerun resumes from it.
|
|
@@ -746,6 +753,9 @@ machine together, and the share of their history the run's tasks used:
|
|
|
746
753
|
probe at 0.4s: 5 tasks left to run, 5 known misses
|
|
747
754
|
asked 1 at 0.5s for 5 tasks (5 measured), plan 12.0s
|
|
748
755
|
ideal 10.8s: each task as long as it took, placed perfectly
|
|
756
|
+
best 2 × small (4c/14G): 11.4s, the fewest of its sizes
|
|
757
|
+
more 3 × small (4c/14G): 10.8s, past its sizes' max
|
|
758
|
+
any 1 × (8c/28G): 10.9s, of any cores, the least core time
|
|
749
759
|
total cpu utilization 58%
|
|
750
760
|
```
|
|
751
761
|
|
|
@@ -776,6 +786,17 @@ real warm-up. That ideal end (`TaskExecutor.idealMs`) sits beside the
|
|
|
776
786
|
real time in the run summary and the footer's `ideal` line, and
|
|
777
787
|
`@vzn/vx-ci` prints it.
|
|
778
788
|
|
|
789
|
+
The same replay says which workers to declare next time
|
|
790
|
+
(`TaskExecutor.advice`), each from an empty run with the plan's own
|
|
791
|
+
rules (the fewest that end it within its margin of the soonest): `best`
|
|
792
|
+
is how many of each size; `more`, shown only when more than a size's
|
|
793
|
+
`max` would end the run sooner past that margin, is how many it would
|
|
794
|
+
take; `any` is one shape for each platform, of 2 to 64 cores (memory a
|
|
795
|
+
core as the platform's sizes have most), the one ending as soon for the
|
|
796
|
+
least core time. Raise a size's `max`, or change its `cpu` and
|
|
797
|
+
`memory`, by them. They are in the footer and the run summary, and
|
|
798
|
+
`@vzn/vx-ci` prints them under "Workers to declare".
|
|
799
|
+
|
|
779
800
|
## Environment
|
|
780
801
|
|
|
781
802
|
What `@vzn/vx-dte` reads from the CI job it runs in:
|
|
@@ -817,16 +838,26 @@ process.env.CF_ACCESS_CLIENT_ID, … }`); an unset value is left out, and
|
|
|
817
838
|
absent, planned as this machine's and not checked on join). With no adapter, the sizes' platforms are
|
|
818
839
|
the ones joining workers may have. Only work counts:
|
|
819
840
|
a task that missed the cache, or waits on a cached one that did, and
|
|
820
|
-
every uncached task, which runs every time; groups and hits need no worker.
|
|
841
|
+
every uncached task, which runs every time; groups and hits need no worker. The run plays
|
|
821
842
|
that work out on its workers and on N more (`src/planner.ts`), each new
|
|
822
843
|
one joining after its size's warm-up: this run's first of that size,
|
|
823
844
|
else an earlier run's, else any size's this run, else 20 s,
|
|
824
845
|
and takes the fewest N that end within 5% (5 s at least) of the best.
|
|
846
|
+
It plays only two warm-ups and the longest task ahead; past that, the
|
|
847
|
+
rest ends no sooner than its longest chain, nor than its work spread
|
|
848
|
+
on the cores it has, and the next plan, a warm-up on, sees further.
|
|
825
849
|
Sizes of different platforms come in turns, so of those N it drops
|
|
826
850
|
each the plan ends as soon without, its platform's work no more than
|
|
827
851
|
5 s later.
|
|
828
852
|
That margin is spent once: later plans keep its deadline until new
|
|
829
853
|
work is known. A task waiting on one that runs waits until it ends.
|
|
854
|
+
Then the run follows that plan: a start it wants later is asked when
|
|
855
|
+
due, and a worker idle with its planned work done goes. It plays again
|
|
856
|
+
only when reality leaves the plan: new work, work that leaves without
|
|
857
|
+
running, a worker that joins, leaves or warms up off plan, a task past
|
|
858
|
+
its time or ending sooner, one of no known time ending, a task starting
|
|
859
|
+
with its time, CPU or memory more than 5% off (1 s, 0.05 cores, 64 MiB
|
|
860
|
+
at least), or its window moving on.
|
|
830
861
|
What is left, running tasks too, takes as many times its history as
|
|
831
862
|
the run's finished tasks took theirs (once they had 20 s of it; never
|
|
832
863
|
less): a task past its forecast would otherwise be planned to end now,
|
|
@@ -876,7 +907,7 @@ HEAD^`), and every task after one (`src/forecast.ts`): a lockfile or
|
|
|
876
907
|
- `adapter` is the `DteAdapter` that starts workers, of the sizes in
|
|
877
908
|
`workers`: at least one with `max` above 0, or `dte()` is refused.
|
|
878
909
|
Absent, the run starts none and uses the workers that join `relay`.
|
|
879
|
-
- `relay` is the relay's URL, http(s)
|
|
910
|
+
- `relay` is the relay's base URL, http(s), without `/v1`, `?query` or `#fragment`; absent, the adapter's own.
|
|
880
911
|
- `token` answers the relay's bearer token (default `VX_DTE_TOKEN`). It
|
|
881
912
|
is asked again a minute before the token's `exp`, when it has one.
|
|
882
913
|
- `unmeasured`, `{ cpu, memory }` (default 2 cores, `'4GB'`), is what a
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@vzn/vx-dte",
|
|
3
|
-
"version": "0.0.
|
|
3
|
+
"version": "0.0.652",
|
|
4
4
|
"license": "MIT",
|
|
5
5
|
"description": "Distributed task execution for vx: one run spread over workers your own CI spawns, joined through a relay.",
|
|
6
6
|
"keywords": [
|
|
@@ -29,10 +29,10 @@
|
|
|
29
29
|
"bun": ">=1.4.4"
|
|
30
30
|
},
|
|
31
31
|
"dependencies": {
|
|
32
|
-
"@vzn/vx-dte-server": "0.0.
|
|
32
|
+
"@vzn/vx-dte-server": "0.0.652"
|
|
33
33
|
},
|
|
34
34
|
"peerDependencies": {
|
|
35
|
-
"@vzn/vx": "^0.0.
|
|
35
|
+
"@vzn/vx": "^0.0.652"
|
|
36
36
|
},
|
|
37
37
|
"repository": {
|
|
38
38
|
"type": "git",
|