@sjawhar/claude-envoy 0.0.0 → 1.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +6 -0
- package/.mcp.json +11 -0
- package/.omp-plugin/plugin.json +6 -0
- package/AGENTS.md +343 -0
- package/README.md +639 -0
- package/bin/dispatch +6 -0
- package/bin/envoy-channel.ts +9 -0
- package/bin/envoy-send.ts +33 -0
- package/dist/THIRD_PARTY_NOTICES +816 -0
- package/dist/dispatch.js +27707 -0
- package/dist/envoy-channel.js +57695 -0
- package/dist/session-hook.js +26328 -0
- package/dist/session-title-hook.js +116 -0
- package/hooks/hooks.json +60 -0
- package/hooks/session-hook.ts +512 -0
- package/hooks/session-title-hook.ts +46 -0
- package/legion/.claude-plugin/plugin.json +4 -0
- package/legion/agents/deep-worker.md +68 -0
- package/legion/agents/oracle.md +35 -0
- package/legion/agents/plan-gap-analyst.md +65 -0
- package/legion/agents/plan-reviewer.md +104 -0
- package/legion/agents/thermonuclear-code-quality.md +30 -0
- package/legion/agents/thermonuclear-deep-review.md +30 -0
- package/legion/hooks/hooks.json +89 -0
- package/legion/skills/ce-simplify-code/LICENSE +21 -0
- package/legion/skills/ce-simplify-code/SKILL.md +66 -0
- package/legion/skills/ce-simplify-code/references/personas/code-quality-reviewer.md +17 -0
- package/legion/skills/ce-simplify-code/references/personas/code-reuse-reviewer.md +7 -0
- package/legion/skills/ce-simplify-code/references/personas/efficiency-reviewer.md +11 -0
- package/legion/skills/leading-a-work-stream/SKILL.md +96 -0
- package/legion/skills/leading-a-work-stream/references/daily-plan.md +8 -0
- package/legion/skills/leading-a-work-stream/references/decision-rights.md +6 -0
- package/legion/skills/leading-a-work-stream/references/direction-and-coaching.md +13 -0
- package/legion/skills/leading-a-work-stream/references/dispatch-upkeep.md +8 -0
- package/legion/skills/leading-a-work-stream/references/sources.md +9 -0
- package/legion/skills/leading-a-work-stream/references/speed-tick.md +44 -0
- package/legion/skills/leading-a-work-stream/references/user-quotes.md +7 -0
- package/legion/skills/legion-architect/SKILL.md +86 -0
- package/legion/skills/legion-architect/references/acceptance.md +208 -0
- package/legion/skills/legion-architect/references/decomposition.md +129 -0
- package/legion/skills/legion-architect/references/design-gate.md +206 -0
- package/legion/skills/legion-architect/references/merge-notices.md +91 -0
- package/legion/skills/legion-architect/references/wake-routing.md +41 -0
- package/legion/skills/legion-controller/SKILL.md +151 -0
- package/legion/skills/legion-controller/references/admission.md +22 -0
- package/legion/skills/legion-controller/references/backlog-stewardship.md +101 -0
- package/legion/skills/legion-controller/references/daily-report.md +75 -0
- package/legion/skills/legion-controller/references/escalation.md +82 -0
- package/legion/skills/legion-controller/references/finding-asks.md +181 -0
- package/legion/skills/legion-controller/references/parked-decisions.md +198 -0
- package/legion/skills/legion-controller/references/session.md +100 -0
- package/legion/skills/legion-controller/references/slack-mentions.md +154 -0
- package/legion/skills/legion-controller/references/triage.md +96 -0
- package/legion/skills/legion-controller/references/wakes.md +132 -0
- package/legion/skills/legion-oracle/SKILL.md +48 -0
- package/legion/skills/legion-retro/SKILL.md +112 -0
- package/legion/skills/legion-retro/references/learnings.md +60 -0
- package/legion/skills/legion-retro/references/pr-body-content.md +56 -0
- package/legion/skills/legion-worker/SKILL.md +69 -0
- package/legion/skills/legion-worker/references/cleanup-deletion.md +22 -0
- package/legion/skills/legion-worker/references/conflicts-and-rewrites.md +164 -0
- package/legion/skills/legion-worker/references/default-branch-check.md +53 -0
- package/legion/skills/legion-worker/references/findings.md +199 -0
- package/legion/skills/legion-worker/references/merge-gate.md +353 -0
- package/legion/skills/legion-worker/references/pr-body.md +218 -0
- package/legion/skills/legion-worker/references/review-threads.md +77 -0
- package/legion/skills/legion-worker/references/systematic-rename.md +19 -0
- package/legion/skills/legion-worker/references/tools.md +708 -0
- package/legion/skills/thermonuclear-code-quality/LICENSE +21 -0
- package/legion/skills/thermonuclear-code-quality/SKILL.md +60 -0
- package/legion/skills/thermonuclear-deep-review/LICENSE +21 -0
- package/legion/skills/thermonuclear-deep-review/SKILL.md +80 -0
- package/package.json +49 -1
- package/scripts/build-inputs.txt +84 -0
- package/scripts/build.ts +415 -0
- package/scripts/check-packed-skills.sh +50 -0
- package/scripts/pack-skills.sh +146 -0
- package/scripts/smoke-channel.sh +178 -0
- package/scripts/smoke-clear-rebind.sh +164 -0
- package/scripts/smoke-session-title.sh +245 -0
- package/scripts/verify-build.sh +88 -0
- package/skills/agent-secrets/SKILL.md +180 -0
- package/skills/asd-ste100/LICENSE +21 -0
- package/skills/asd-ste100/SKILL.md +140 -0
- package/skills/asd-ste100/examples/before-after.md +92 -0
- package/skills/asd-ste100/references/writing-rules.md +59 -0
- package/skills/dispatch/SKILL.md +89 -0
- package/skills/dispatch-first/SKILL.md +46 -0
- package/skills/dispatch-writing-specs/SKILL.md +80 -0
- package/skills/envoy/SKILL.md +82 -0
- package/skills/envoy/references/check-settlement.md +62 -0
- package/skills/envoy/references/deliveries.md +67 -0
- package/skills/envoy/references/legion.md +14 -0
- package/skills/envoy/references/roles.md +12 -0
- package/skills/envoy/references/sources.md +41 -0
- package/skills/outside-message/SKILL.md +111 -0
- package/skills/plain-words/SKILL.md +88 -0
- package/src/channel-forwarder.ts +262 -0
- package/src/claude-channel.ts +705 -0
- package/src/claude-session.ts +38 -0
- package/src/envoy-channel-server.ts +2115 -0
- package/src/envoy-client.ts +26 -0
- package/src/envoy-off.ts +62 -0
- package/src/legion-claim.ts +211 -0
- package/src/legion-inbox.ts +110 -0
- package/src/legion-phase-hook.ts +554 -0
- package/src/legion-tool.ts +69 -0
- package/src/replay-state.ts +97 -0
- package/src/send-arguments.ts +13 -0
- package/src/session-identity.ts +165 -0
- package/src/session-state.ts +142 -0
- package/src/session-title.ts +251 -0
- package/src/subscription-topics.ts +16 -0
|
@@ -0,0 +1,708 @@
|
|
|
1
|
+
# Getting the tool your work needs
|
|
2
|
+
|
|
3
|
+
Part of `skill://legion-worker`. Read it before you read a shared database, a session store or
|
|
4
|
+
another role's transcript among them; before you run a build or a test suite in your pod; the
|
|
5
|
+
moment your phase needs a tool the worker image may not carry — a database client, a linter, a
|
|
6
|
+
one-off CLI — before you install, download, or vendor anything; before you measure Oh My Pi's
|
|
7
|
+
binary; and when a test fails in your pod. Every path it cites is in cybermonklabs/legion.
|
|
8
|
+
|
|
9
|
+
## What runs in your pod, and what runs on the pull request
|
|
10
|
+
|
|
11
|
+
An issue pod runs no heavy build and no whole test suite. The pod's six roles share one memory
|
|
12
|
+
limit, and one command that fills it ends every role's turn, not only yours.
|
|
13
|
+
|
|
14
|
+
In the pod you compile and test the packages your change touches, build the program a check of
|
|
15
|
+
your issue drives (`go build ./cmd/legion`), vet the module you changed (`go vet -tags e2e ./...`,
|
|
16
|
+
which compiles and runs no test), and, in a package whose tests need Postgres, run only the tests
|
|
17
|
+
your change touches by name (`go test -run`).
|
|
18
|
+
|
|
19
|
+
A whole module's or whole package's test run goes to CI on the pull request, where a job of that
|
|
20
|
+
pull request runs it: `go test ./...`, a whole Postgres-backed package such as
|
|
21
|
+
`packages/envoy/internal/dispatch/docs` or `packages/envoy/internal/dispatch/api`, a browser suite
|
|
22
|
+
run whole, and `make -j` or `lake build` in a repository your deployment sends you to. Every other
|
|
23
|
+
heavy run goes where your deployment's instructions say.
|
|
24
|
+
|
|
25
|
+
## The database your tests use
|
|
26
|
+
|
|
27
|
+
- Never give a test, a migration or anything that creates or drops a database the URL in the
|
|
28
|
+
file `OMP_SESSION_SQL_DSN_FILE`: that is the live sessions database every pod shares, and it
|
|
29
|
+
holds every role's transcript.
|
|
30
|
+
- Give every Postgres variable a test reads `$LEGION_TEST_PG_DSN`, your pod's own server:
|
|
31
|
+
`LEGION_TEST_PG_DSN`, `DISPATCH_TEST_DATABASE_URL` and `BROKER_TEST_DATABASE_URL`. Each makes
|
|
32
|
+
and drops a database of its own per test, so your pod's roles can share the server.
|
|
33
|
+
- Anything that migrates into the database its DSN names, rather than making one per test, needs
|
|
34
|
+
a database of your own, because a sibling role writing the same one collides with you: the
|
|
35
|
+
Dispatch browser suite's `DATABASE_URL`, which resets what it is given, and the stage scripts'
|
|
36
|
+
`LEGION_E2E_PG_DSN`, which a daemon's `postgres_dsn` becomes and migrates Legion's schema into.
|
|
37
|
+
Make one and point the variable at it:
|
|
38
|
+
`createdb -h 127.0.0.1 -p 5433 -U legion_test "e2e_$(date +%s)"`.
|
|
39
|
+
- The server is empty in every new pod and dies with it. Every database made from `template1`
|
|
40
|
+
has pgvector.
|
|
41
|
+
- `LEGION_TEST_PG_DSN` set and the connection refused: that is not yours to repair. Starting the
|
|
42
|
+
server is the roles container's launcher's job — it runs `legion-testpg ensure` before any role
|
|
43
|
+
starts and again every 15 seconds — so wait about that interval and try again. Never start a
|
|
44
|
+
server of your own beside it, and never reach for another DSN. Still refused after that wait:
|
|
45
|
+
report it, with the error you got and the port you used, as a fault of the pod rather than of
|
|
46
|
+
your change.
|
|
47
|
+
- `LEGION_TEST_PG_DSN` unset: your deployment has not turned the pod's server on, so no launcher
|
|
48
|
+
is starting one and nothing will appear while you wait. With `legion-testpg` on `PATH`, start
|
|
49
|
+
one yourself on the issue's volume: the directory it defaults to is a 1 MiB filesystem your
|
|
50
|
+
role's own mounts cover, so `export LEGION_TESTPG_DIR=$XDG_DATA_HOME/testpg` first, then
|
|
51
|
+
`export LEGION_TEST_PG_DSN=postgres://legion_test@127.0.0.1:5433/postgres?sslmode=disable`
|
|
52
|
+
and `legion-testpg ensure`, which makes the cluster, the role and `vector` in `template1`. That
|
|
53
|
+
server lives in your generation: it dies when your generation ends, which takes a sibling
|
|
54
|
+
role's connections and databases with it mid-test, and the next `ensure` of either role finds
|
|
55
|
+
the same cluster on disk and serves it again. With no
|
|
56
|
+
`legion-testpg` either, stand up one with the harness below (**The worked case**) and set the DSN.
|
|
57
|
+
|
|
58
|
+
## Reading a shared database
|
|
59
|
+
|
|
60
|
+
A shared database is one outside your pod: the deployment's sessions database, another team's
|
|
61
|
+
database, any server whose other clients your role cannot stop. Your pod's own test server
|
|
62
|
+
(`$LEGION_TEST_PG_DSN`, above) is not one of them, because it dies with your pod.
|
|
63
|
+
|
|
64
|
+
- Read it through the deployment's read-only login. Where the deployment gives one, its URL is in
|
|
65
|
+
the file `SESSION_READER_DSN_FILE` names.
|
|
66
|
+
- Never read it through the session URL the file `OMP_SESSION_SQL_DSN_FILE` names. That URL is the
|
|
67
|
+
one the role's own agent writes its transcript with, so a read through it puts a write-capable
|
|
68
|
+
connection on a shared server for a read.
|
|
69
|
+
- Keep the read-only setting inside one transaction, never on the connection for its whole life. A
|
|
70
|
+
read-only setting made for the whole session stays on the backend after the connection goes back
|
|
71
|
+
to the pool. The next client of that pool gets it, and that client's writes fail. On 2026-11-05
|
|
72
|
+
one such read left the setting on a pooled backend, and 1,432 session writes failed over 16
|
|
73
|
+
minutes.
|
|
74
|
+
|
|
75
|
+
These three forms do it. Each one holds the transaction read-only, so a write inside it is
|
|
76
|
+
refused, and each one drops that state when the transaction ends.
|
|
77
|
+
|
|
78
|
+
```sql
|
|
79
|
+
BEGIN READ ONLY;
|
|
80
|
+
SELECT path FROM omp_session_files ORDER BY path LIMIT 1;
|
|
81
|
+
COMMIT;
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
```sql
|
|
85
|
+
BEGIN;
|
|
86
|
+
SET LOCAL transaction_read_only = on;
|
|
87
|
+
SELECT path FROM omp_session_files ORDER BY path LIMIT 1;
|
|
88
|
+
COMMIT;
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
```sh
|
|
92
|
+
psql "$URL" --single-transaction -q -X -v ON_ERROR_STOP=1 -c 'SET TRANSACTION READ ONLY' -f <file>
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
`-q` keeps psql's `SET` status line off stdout. Without it psql prints that line, and it becomes
|
|
96
|
+
the first line of a file you redirect the output into. `-X` keeps your `~/.psqlrc` out of the
|
|
97
|
+
run, and `ON_ERROR_STOP` ends the run at the first error. The file holds no `COMMIT` or
|
|
98
|
+
`ROLLBACK` of its own: one would end the read-only transaction early, and every statement after
|
|
99
|
+
it would run outside it.
|
|
100
|
+
|
|
101
|
+
Your deployment may give no `SESSION_READER_DSN_FILE`. Then read nothing of a shared database
|
|
102
|
+
through the session URL. Read your deployment's instructions for the route they give, and when
|
|
103
|
+
they name none, report it as a fault of the deployment rather than reaching for the writer URL.
|
|
104
|
+
|
|
105
|
+
## The five clients the image already carries
|
|
106
|
+
|
|
107
|
+
Check this list first: the worker image carries these five from the change that added them onward,
|
|
108
|
+
and they are on a role's `PATH` once its deployment pins a digest at or after that change — a role
|
|
109
|
+
still on an older digest finds none of them. Run `command -v psql redis-cli sqlite3 nc dig` to see
|
|
110
|
+
which you have; those need no runner. `command -v legion-testpg` says whether your digest also
|
|
111
|
+
carries the pod's own PostgreSQL server (the section above).
|
|
112
|
+
|
|
113
|
+
- `psql` — the Dispatch browser suite (`bun run e2e`) resets its database for every spec through
|
|
114
|
+
`packages/dispatch/e2e/psql.ts`, which `seed.ts` calls, and the local stack (`bun run
|
|
115
|
+
dispatch:local`, `packages/dispatch/e2e/local-instance.ts`) refuses to start without `go`,
|
|
116
|
+
`psql`, `flock`, `openssl` and `python3` on `PATH`, and without `docker` too unless a
|
|
117
|
+
`DATABASE_URL` is given. The image carries `go`, `flock`, `openssl` and `python3`, a
|
|
118
|
+
`DATABASE_URL` stands in for `docker`,
|
|
119
|
+
and `psql` was the one a role had no way to supply — so a role whose change owes that suite can
|
|
120
|
+
run it.
|
|
121
|
+
- `redis-cli` — no repository of the project speaks Redis today; it is carried anyway, at about a
|
|
122
|
+
megabyte, because a second image roll costs more than a megabyte.
|
|
123
|
+
- `sqlite3` — a repository's own SQLite fixture or database has no other way to open from a shell,
|
|
124
|
+
and Oh My Pi's own profile databases (`~/.omp/profiles/legion/agent/agent.db`, `models.db`,
|
|
125
|
+
`history.db`) are SQLite too. Your session is not one of them: it is `results.jsonl`,
|
|
126
|
+
`state.json` and `title` under `XDG_STATE_HOME`.
|
|
127
|
+
- `nc` — the reachability check you would otherwise write as an inline Python loop. The name comes
|
|
128
|
+
from the alternatives system, so ask for `nc`, never `nc.openbsd`.
|
|
129
|
+
- `dig` — how a role whose connection to a service name fails tells "the name does not resolve"
|
|
130
|
+
from "the packets are dropped".
|
|
131
|
+
|
|
132
|
+
## Why a sixth is an image change, not an install
|
|
133
|
+
|
|
134
|
+
Your role runs as an unprivileged user with no writable system prefix: `/usr/local/bin` and `/opt`
|
|
135
|
+
are read-only, `/home` is read-only too, and `apt-get` cannot take dpkg's lock even to download.
|
|
136
|
+
`$HOME` — `/home/legion`, the one writable directory under that read-only `/home` — is yours to
|
|
137
|
+
write.
|
|
138
|
+
|
|
139
|
+
```text
|
|
140
|
+
$ apt-get install -d -y postgresql-client
|
|
141
|
+
E: Could not open lock file /var/lib/dpkg/lock-frontend - open (13: Permission denied)
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
So that writable `$HOME`, and the issue's disk, are where an install for yourself would go, and on
|
|
145
|
+
the mechanics the answer is not a flat no: `redis-cli`, `sqlite3` and `dig` did run from an
|
|
146
|
+
extraction prefix under `$HOME` with `LD_LIBRARY_PATH` set, and `psql` did not — Debian's
|
|
147
|
+
`/usr/bin/psql` is a Perl wrapper that loads `PgCommon.pm` through Perl's own search path, which no
|
|
148
|
+
extraction prefix is on. What rules such an attempt out is not the unpacking: it is the next
|
|
149
|
+
section's test ("The test: where it lands, and what it leaves resolving"), and an extraction under
|
|
150
|
+
`$HOME` fails that test's first condition whatever you scope it to, because `/home/legion` is the
|
|
151
|
+
pod's disk rather than the issue's volume — the placement `roleEnvironment`
|
|
152
|
+
(`packages/daemon/internal/runtime/sandbox/manifest.go`) exists to move writes off. The sanctioned
|
|
153
|
+
extraction is the harness's: onto the issue's volume, and exported around its own commands and no
|
|
154
|
+
further (the Dispatch e2e stack, below, is the worked case). For everything the five do not cover,
|
|
155
|
+
the one documented step is a pinned run through one of the three package runners the image already
|
|
156
|
+
carries — plus `uv run --with` when what you need is a library rather than a command — and a tool
|
|
157
|
+
no runner reaches is an image change ("When no runner reaches it", below).
|
|
158
|
+
|
|
159
|
+
## What the withheld capability is, and what it is not
|
|
160
|
+
|
|
161
|
+
The withheld row reads `installs or configures its own dependencies in the pod: Legion owns its
|
|
162
|
+
dependencies` (`packages/daemon/internal/capabilities/capabilities.go`, ruling
|
|
163
|
+
dispatch://LEGION-200). Read as a sentence on its own, that wording would also forbid `uvx`,
|
|
164
|
+
`bunx --package` and `go run`, each of which fetches a package and writes a cache. It does not, and
|
|
165
|
+
three artefacts in this repository are what draw the line — not a claim about what the ruling meant:
|
|
166
|
+
|
|
167
|
+
- **What the image ships for a role to use.** `packages/daemon/docker/worker.Dockerfile` installs
|
|
168
|
+
`uv` and `uvx`, `bun` with its `bunx` link, and the Go toolchain, and the `toolchain` capability
|
|
169
|
+
row (`packages/daemon/internal/capabilities/capabilities.go`) grants them by name. A runner put
|
|
170
|
+
in the image deliberately and granted by a capability row is not a route around a withheld one.
|
|
171
|
+
- **What the pod configures for those runners to write.** A role's environment relocates
|
|
172
|
+
`UV_CACHE_DIR` and `UV_PYTHON_INSTALL_DIR` (`podkind.go`) and `BUN_INSTALL_CACHE_DIR`, `GOCACHE`
|
|
173
|
+
and `GOMODCACHE` (`roleEnvironment`, `packages/daemon/internal/runtime/sandbox/manifest.go`)
|
|
174
|
+
onto the issue's volume, shared by the issue's roles. Legion provisions the caches a pinned run
|
|
175
|
+
fills; filling them is what they are for.
|
|
176
|
+
- **What Legion itself does at launch.** `installBlock`
|
|
177
|
+
(`packages/daemon/internal/runtime/sandbox/plugins.go`) renders a `bun add` under
|
|
178
|
+
`/usr/bin/flock` into that shared bun install cache on every installing launch. Fetching a pinned
|
|
179
|
+
package at run time is something Legion does, not something it withholds.
|
|
180
|
+
|
|
181
|
+
So the withheld row is about a dependency **the pod carries**: something that outlives the command
|
|
182
|
+
that created it, and that a later launch or another role's process then resolves *without asking
|
|
183
|
+
for it*.
|
|
184
|
+
|
|
185
|
+
### The test: where it lands, and what it leaves resolving
|
|
186
|
+
|
|
187
|
+
An install of your own — an interpreter, a browser, a second toolchain, an extraction prefix — is
|
|
188
|
+
on the sanctioned side when both of these hold: what it writes lands on the issue's own volume,
|
|
189
|
+
under `/legion`, which outlives a role's relaunch in its pod and is shared by the issue's roles —
|
|
190
|
+
whether through a variable the runtime sets for you there or under a path on that volume your
|
|
191
|
+
harness names itself — but outside the issue's workspace (`$LEGION_WORKSPACE`, under
|
|
192
|
+
`/legion/workspaces`) and the runtime's own directories (`/legion/repos`, `/legion/feed`); and
|
|
193
|
+
after the command exits nothing else resolves differently. The workspace is excluded because its jj
|
|
194
|
+
working copy tracks every new file under jj's `snapshot.max-new-file-size` (1 MiB for a role), so
|
|
195
|
+
a root written there is added to the change every role shares, and a larger new
|
|
196
|
+
file is left untracked, every jj command in the workspace then warning `Refused to snapshot some
|
|
197
|
+
files`. The one write into it that is sanctioned is a repository's own dependency install into
|
|
198
|
+
its ignored directories, below. The variables the runtime sets on the volume, where a tool's own
|
|
199
|
+
cache already goes, are
|
|
200
|
+
`XDG_CACHE_HOME` (`/legion/scratch/cache`), `XDG_DATA_HOME`, `XDG_STATE_HOME`, `TMPDIR`, `GOCACHE`,
|
|
201
|
+
`GOMODCACHE`, `BUN_INSTALL_CACHE_DIR`, `UV_CACHE_DIR` (`/legion/uv/cache`) and
|
|
202
|
+
`UV_PYTHON_INSTALL_DIR` (`/legion/uv/python/<issue>`). (`XDG_CONFIG_HOME` is set too and is not
|
|
203
|
+
one of them: it is your role's own config home, a copy of the pod's made for your role on the
|
|
204
|
+
pod's disk, which no other role reads.) `$HOME` and `/tmp` are not on the volume:
|
|
205
|
+
both are your role's own, on the pod's disk.
|
|
206
|
+
|
|
207
|
+
It is the withheld capability when either condition fails — anything that writes into a
|
|
208
|
+
system prefix, and anything that leaves a toolchain, shim or binary which a later launch or
|
|
209
|
+
another role's process will resolve without asking for it. Between two installs that both land in
|
|
210
|
+
the right place, the test is resolution, not disk, and it reads as scope: a second toolchain
|
|
211
|
+
exported for the commands of your own run — by absolute path, or on a `PATH` your harness's own
|
|
212
|
+
script sets around its own invocations — is sanctioned; the same toolchain left where a later
|
|
213
|
+
launch, another role or a subagent picks it up without naming it is operator setup, whoever typed
|
|
214
|
+
the command.
|
|
215
|
+
|
|
216
|
+
What each shape leaves resolving:
|
|
217
|
+
|
|
218
|
+
- A pinned runner run: nothing. `uvx --from httpie@3.2.4 http --version` leaves a cache entry
|
|
219
|
+
addressed by that coordinate and that version, which the next command reaches only by writing the
|
|
220
|
+
same pin; no `PATH`, no interpreter and no library search of any other process changes. That is
|
|
221
|
+
the sanctioned side, and it is the side the forms below are on.
|
|
222
|
+
- A repository's own dependency set: nothing either, which is why the caches are provisioned at
|
|
223
|
+
all. `uv sync` or `bun install` in your workspace materialises what that project already
|
|
224
|
+
declares, into that project's own directory, which a process reaches by asking for that project
|
|
225
|
+
and no other way — and the runtime puts `UV_PYTHON_INSTALL_DIR` and the workspace itself on the
|
|
226
|
+
issue's volume exactly so every role of the issue's pod reuses it.
|
|
227
|
+
- An extraction prefix: whatever you leave resolving through it, which is yours to decide. Unpack
|
|
228
|
+
Debian packages under `$HOME`, export `LD_LIBRARY_PATH` in the shell you go on working in, and
|
|
229
|
+
you change what every later command in that shell — and every subagent that inherits the
|
|
230
|
+
environment — resolves `psql` or its libraries to, asked for or not, on a prefix nothing records,
|
|
231
|
+
pins or rebuilds: that fails both conditions, and it is the withheld capability. The harness
|
|
232
|
+
below is the shape that passes. The opposite shape is in the tree too, and superseded: telling
|
|
233
|
+
every subagent to prepend a directory to `PATH` because a subagent's shell starts without it
|
|
234
|
+
hands over an environment to processes that never named it. That is the difference between
|
|
235
|
+
standing a harness up and getting yourself a CLI.
|
|
236
|
+
|
|
237
|
+
The test is about what a command leaves for other processes, never about getting the same bytes
|
|
238
|
+
twice — because a pinned line does not promise the same bytes. What a pin binds is the named
|
|
239
|
+
distribution at that version; its dependency closure is resolved when the command runs. With
|
|
240
|
+
`UV_CACHE_DIR` emptied, `uvx -v --from httpie@3.2.4 http --version` logs `Selecting:
|
|
241
|
+
requests==2.34.2 [compatible]` and resolves `urllib3` and `certifi` from ranges, where the warm run
|
|
242
|
+
prints `3.2.4` at once off the cache that first run filled. When your work needs the closure pinned
|
|
243
|
+
too — a comparison that has to hold across days or across pods — that is a project's own lockfile
|
|
244
|
+
with `uv run` or `bun install` in a workspace, not a one-off runner line.
|
|
245
|
+
|
|
246
|
+
A pinned run whose tool then installs something of its own — an interpreter, a browser, a second
|
|
247
|
+
toolchain — is where the first condition does the work. `uv run --with pyyaml==6.0.2 --no-project`
|
|
248
|
+
passes both: the wheel lands in `UV_CACHE_DIR`, under `wheels-v6/pypi/pyyaml/6.0.2-…`, and
|
|
249
|
+
`UV_PYTHON_INSTALL_DIR` is never created, because the image's own `python3` served the run. The
|
|
250
|
+
interpreter is the other run: a project whose `.python-version` names an interpreter uv must fetch
|
|
251
|
+
puts about 70 MB under `UV_PYTHON_INSTALL_DIR`, where every role of the issue's pod finds it, while
|
|
252
|
+
the cache holds no wheel. A browser download passes too, and for Playwright — the browser driver this
|
|
253
|
+
repository's own Dispatch suite installs (`playwright install`, `packages/dispatch`'s
|
|
254
|
+
`e2e:install`) — it passes as it stands, with nothing pointed anywhere: playwright-core resolves
|
|
255
|
+
its browser registry under `XDG_CACHE_HOME`, which the runtime has already put on the issue's
|
|
256
|
+
volume, so the issue's roles share one registry under the exclusive install lock that
|
|
257
|
+
`docs/kubernetes.md` writes down in "Anatomy of a Sandbox pod", the paragraph on sharing the cache
|
|
258
|
+
home — worth reading before an install, for the two things that follow from sharing. Pointing
|
|
259
|
+
`PLAYWRIGHT_BROWSERS_PATH` elsewhere is what costs you: that shared registry, and the lock that
|
|
260
|
+
makes a second role installing at the same time wait rather than race. The condition is for a tool
|
|
261
|
+
whose cache the runtime does not already place — Puppeteer is one, resolving its browser cache to
|
|
262
|
+
`join(homedir(), '.cache', 'puppeteer')` and so to your role's own home on the pod's disk whatever
|
|
263
|
+
`XDG_CACHE_HOME` says. For that one, point the tool's own cache variable (`PUPPETEER_CACHE_DIR`
|
|
264
|
+
here) at a directory under one of the paths above yourself, since an unset one leaves the download
|
|
265
|
+
wherever the tool defaults, which is nowhere this runtime chose.
|
|
266
|
+
|
|
267
|
+
Where it lands is what you pay. Under one of those variables it is on the issue's volume: it
|
|
268
|
+
outlives a role's relaunch in its pod, and every role of the issue is told the same path — under
|
|
269
|
+
every one of those variables but
|
|
270
|
+
`XDG_STATE_HOME`, which is `/legion/scratch/state/<role>`, a directory named for your role
|
|
271
|
+
(`roleEnvironment`) because Oh My Pi names its browser broker's lock from the state root and an
|
|
272
|
+
abstract socket name is one name across every role of the pod, which share its network. It is not
|
|
273
|
+
private: every role of the issue can write it, as it can every path above — though it does not
|
|
274
|
+
outlive the pod: the issue's disk is the pod's own and is deleted with it, so a later pod of the
|
|
275
|
+
issue starts without it. Under
|
|
276
|
+
`$HOME` — `/home/legion`, which no volume is mounted on — it is your role's own home, an overlay
|
|
277
|
+
over the image's whose writes land on the pod's disk (an `emptyDir` every role's home and `/tmp`
|
|
278
|
+
share): one copy per role, per pod, kept across the role's relaunches in that pod and gone when
|
|
279
|
+
the pod is replaced, and charged to the pod's ephemeral-storage limit, which every role of the pod
|
|
280
|
+
shares and which is smaller than the volume. Moving exactly these writes off that
|
|
281
|
+
filesystem is what `roleEnvironment` (`packages/daemon/internal/runtime/sandbox/manifest.go`)
|
|
282
|
+
exists for, after GKE Autopilot evicted issue pods that charged their caches to it — so an install
|
|
283
|
+
under `$HOME` fails the first condition and is outside the sanctioned form, however little else it
|
|
284
|
+
changes. The three artefacts above are the whole of that line's ground. Disk is not: every cache is
|
|
285
|
+
already relocated onto the issue's volume, which has the room for them (`docs/kubernetes.md`, "The
|
|
286
|
+
tools a role's own work needs", carries the figure), and a sanctioned install is paid once per
|
|
287
|
+
issue pod, on the issue's own disk, by the issue's roles together.
|
|
288
|
+
|
|
289
|
+
**The worked case, and the one a role actually meets: the Dispatch e2e stack.** A change to
|
|
290
|
+
`packages/dispatch` is proved in the pod by driving the specs it touches, one spec a command
|
|
291
|
+
(`bun run e2e -- e2e/<one spec file>`); the whole browser suite runs in CI's `dispatch` job on the
|
|
292
|
+
pull request. One spec needs the same stack as all of them: a Postgres with pgvector. On a digest
|
|
293
|
+
carrying `legion-testpg`, use the pod's own server ("The database your tests use", above) and
|
|
294
|
+
unpack nothing. On a digest without it, the way
|
|
295
|
+
that stack is stood up here is Debian packages unpacked with `dpkg-deb -x` into one root on the
|
|
296
|
+
issue's volume — the five packages and the start sequence are in
|
|
297
|
+
`docs/solutions/testing/the-real-dispatch-stack-runs-inside-a-legion-pod-with-no-postgres-no-docker-and-no-root-INFRA-42.md`,
|
|
298
|
+
and
|
|
299
|
+
`docs/solutions/testing/a-browser-harness-in-an-issue-pod-lives-on-the-issue-volume-INFRA-43.md`
|
|
300
|
+
puts it on the volume, moves the caches with it, and writes it as one idempotent script the next
|
|
301
|
+
role re-runs. That passes both conditions, and the second is the one to get right: export the
|
|
302
|
+
root's `PATH` and `LD_LIBRARY_PATH` around the harness's own commands — `initdb`, `pg_ctl`, the
|
|
303
|
+
server, the Playwright run — rather than into the shell you go on working in. Handing the harness
|
|
304
|
+
to the next role is the same test passed again, and
|
|
305
|
+
`docs/solutions/legion/a-port-held-in-an-issue-pod-is-a-sibling-roles-harness-move-your-run-with-the-five-port-variables-and-your-own-database-and-never-stop-the-holder-LEGION-695.md`
|
|
306
|
+
is the illustration: it leaves the server running and records the `LD_LIBRARY_PATH`, `PATH` and
|
|
307
|
+
`DATABASE_URL` in its handoff's `trickyParts`, so what crosses to the next role is a note that
|
|
308
|
+
role sets deliberately rather than an environment it inherits. The suite shells out to `psql`,
|
|
309
|
+
and `psql` being in the image is what makes that harness work now without your extracting a
|
|
310
|
+
client for yourself at all.
|
|
311
|
+
|
|
312
|
+
## The three pinned runner forms
|
|
313
|
+
|
|
314
|
+
- `uvx --from <package>@<version> <command>`, for a Python tool: the distribution you pin and the
|
|
315
|
+
command you run are named separately, because they are usually different names.
|
|
316
|
+
`uvx --from httpie@3.2.4 http --version` prints `3.2.4`. Where the project's own name *is* its
|
|
317
|
+
command's — checked once, never assumed — the short `uvx <name>@<version>` runs it:
|
|
318
|
+
`uvx ruff@0.14.0 --version` prints `ruff 0.14.0`, in 1.7 s with the uv cache empty and 0.15 s
|
|
319
|
+
warm. A version that does not exist fails closed, exit 1, naming it: `uvx ruff@99.99.99
|
|
320
|
+
--version` prints `error: No solution found when resolving tool dependencies`, with `cause:
|
|
321
|
+
Because there is no version of ruff==99.99.99 …`.
|
|
322
|
+
- `bunx --package <name>@<version> <binary>`, for an npm tool, naming the package and the binary
|
|
323
|
+
separately because they differ. `bunx --package @biomejs/biome@2.6.3 biome --version` prints
|
|
324
|
+
`Version: 2.6.3`, in 0.3 s with bun's cache warm; `bunx --package @biomejs/biome@99.99.99 biome
|
|
325
|
+
--version` fails closed, exit 1, with `error: No version matching "99.99.99" found for specifier
|
|
326
|
+
"@biomejs/biome" (but package exists)`.
|
|
327
|
+
- `go run <module>@<version>`, for a Go tool. `go run github.com/nats-io/natscli/nats@v0.1.5
|
|
328
|
+
--version` prints `v0.1.5`, in 7.2 s with the module cache warm and 103 s on a cold module cache
|
|
329
|
+
— budget for the cold one the first time an issue runs a Go tool. `go run
|
|
330
|
+
github.com/nats-io/natscli/nats@v99.99.99 --version` fails closed, exit 1, with
|
|
331
|
+
`go: github.com/nats-io/natscli/nats@v99.99.99: invalid version: unknown revision nats/v99.99.99`.
|
|
332
|
+
|
|
333
|
+
## When you need a library, not a command
|
|
334
|
+
|
|
335
|
+
The three forms above run somebody else's command. When what you need is a package available to
|
|
336
|
+
code of your own — PyYAML to read a file, not a CLI — the form is `uv run --with
|
|
337
|
+
<package>==<version> --no-project` ahead of the program you are running:
|
|
338
|
+
|
|
339
|
+
```text
|
|
340
|
+
$ uv run --with pyyaml==6.0.2 --no-project python -c "import yaml; print(yaml.__version__)"
|
|
341
|
+
Installed 1 package in 12ms
|
|
342
|
+
6.0.2
|
|
343
|
+
```
|
|
344
|
+
|
|
345
|
+
That whole run is 0.597 s wall, the install included. `--no-project` keeps it off whatever
|
|
346
|
+
`pyproject.toml` happens to sit above your working directory, so it behaves the same wherever you
|
|
347
|
+
type it, and `--with` takes a requirement specifier (`==<version>`), not a runner coordinate
|
|
348
|
+
(`@<version>`). It is on the sanctioned side for the same reason a `uvx` run is: it fills
|
|
349
|
+
`UV_CACHE_DIR` and leaves nothing another process resolves. Reaching for a command runner instead
|
|
350
|
+
is the mistake the next section's `uvx pyyaml@6.0.2 --version` shows — uv installs the project and
|
|
351
|
+
then finds no executable, because a library has none to find.
|
|
352
|
+
|
|
353
|
+
## Always write the exact coordinate and the exact version
|
|
354
|
+
|
|
355
|
+
A pin has two halves, and a runner resolves what you asked for only when both are right:
|
|
356
|
+
|
|
357
|
+
1. **The coordinate** — the registry name the project itself publishes, scope and all:
|
|
358
|
+
`@biomejs/biome`, not `biome`. The command's name is no evidence of the package's. Take the
|
|
359
|
+
coordinate from the project's own published installation instructions — its documentation site
|
|
360
|
+
or its repository's README, reached independently of the registry rather than through the
|
|
361
|
+
registry entry of a name you guessed. That is the step, and it is the only one that establishes
|
|
362
|
+
identity. Do it once, before the tool's first use.
|
|
363
|
+
2. **The version** on that coordinate.
|
|
364
|
+
|
|
365
|
+
Everything a registry can tell you afterwards is corroboration, and it is worth knowing which kind
|
|
366
|
+
each piece is.
|
|
367
|
+
|
|
368
|
+
**Fields the publisher writes.** `npm view <name> version description repository.url homepage`
|
|
369
|
+
reads what the publisher put in its own `package.json`. npm verifies none of it: a squatter closes
|
|
370
|
+
this check by copying two strings, and a legitimate package that sets no `homepage` reads like a
|
|
371
|
+
squat. Evidence, never proof.
|
|
372
|
+
|
|
373
|
+
**A check whose evidence the subject controls is not a check.** Carry that past npm: every line
|
|
374
|
+
below the next prompt was written by the party whose identity is in question.
|
|
375
|
+
|
|
376
|
+
```text
|
|
377
|
+
$ npm view biome version description repository.url homepage
|
|
378
|
+
version = '0.3.3'
|
|
379
|
+
description = 'A simple way to manage environment variables on a per-project basis'
|
|
380
|
+
$ npm view @biomejs/biome version description repository.url homepage
|
|
381
|
+
version = '2.6.3'
|
|
382
|
+
description = 'Biome is a toolchain for the web: formatter, linter and more'
|
|
383
|
+
repository.url = 'git+https://github.com/biomejs/biome.git'
|
|
384
|
+
homepage = 'https://biomejs.dev'
|
|
385
|
+
```
|
|
386
|
+
|
|
387
|
+
Those two answers are *consistent with* `@biomejs/biome` being the project and `biome` being
|
|
388
|
+
someone else's package. What settles it is biomejs.dev telling you to install `@biomejs/biome` —
|
|
389
|
+
the site you reached without npm's help.
|
|
390
|
+
|
|
391
|
+
**What a registry can attest.** npm verifies a provenance attestation when it is published, so it
|
|
392
|
+
is the one part of an entry a publisher cannot forge by writing a string. What it proves is *build
|
|
393
|
+
origin*: the attestation binds the artifact to the repository and workflow that built it. Presence
|
|
394
|
+
is not identity — a squat published from its own public Actions workflow carries a genuine
|
|
395
|
+
attestation of its own repository — so the check is to read whose build it names, and compare that
|
|
396
|
+
with the repository the project's own documentation gave you:
|
|
397
|
+
|
|
398
|
+
```text
|
|
399
|
+
$ npm view @biomejs/biome dist.attestations.url
|
|
400
|
+
https://registry.npmjs.org/-/npm/v1/attestations/@biomejs%2fbiome@2.6.3
|
|
401
|
+
$ curl -s "$(npm view @biomejs/biome dist.attestations.url)" \
|
|
402
|
+
| jq -r '.attestations[] | select(.predicateType=="https://slsa.dev/provenance/v1") | .bundle.dsseEnvelope.payload' \
|
|
403
|
+
| base64 -d | jq -r '.predicate.buildDefinition.externalParameters.workflow.repository'
|
|
404
|
+
https://github.com/biomejs/biome
|
|
405
|
+
```
|
|
406
|
+
|
|
407
|
+
That field is the SLSA v1 predicate's `buildDefinition.externalParameters.workflow`, which here
|
|
408
|
+
reads `{"ref": "refs/heads/main", "repository": "https://github.com/biomejs/biome", "path":
|
|
409
|
+
".github/workflows/release.yml"}` — the repository biomejs.dev sent you to, building on its own
|
|
410
|
+
main branch. The comparison is what makes the attestation worth anything: on its own it says only
|
|
411
|
+
that *some* repository built these bytes. (`jq` in your bash tool is the harness's builtin, and a
|
|
412
|
+
script gets `/usr/bin/jq`; both print this.)
|
|
413
|
+
|
|
414
|
+
Its limit runs the other way: most packages publish no provenance at all — `npm view biome
|
|
415
|
+
dist.attestations` prints nothing — so an absent attestation is no evidence of a squat.
|
|
416
|
+
|
|
417
|
+
PyPI publishes attestations only for projects that release through a trusted publisher, and widely
|
|
418
|
+
used tools do without: for `ruff 0.14.0`, `https://pypi.org/pypi/ruff/0.14.0/json` carries no
|
|
419
|
+
`provenance` on any of its files and `https://pypi.org/integrity/ruff/0.14.0/<file>/provenance`
|
|
420
|
+
answers 404. For a Python tool, the project's own documentation is the whole of the check.
|
|
421
|
+
|
|
422
|
+
A Go module path is a domain-scoped URL, so the coordinate *is* the repository path, and you check
|
|
423
|
+
it by going to that path. `sum.golang.org` pins a version's bytes once they are published, which
|
|
424
|
+
stops the content changing under you later and says nothing about whether you picked the right
|
|
425
|
+
module path in the first place.
|
|
426
|
+
|
|
427
|
+
Pinning the version alone does not save you: it pins the squatter at a version, and obeys "write
|
|
428
|
+
the version" to the letter while doing it.
|
|
429
|
+
|
|
430
|
+
```text
|
|
431
|
+
$ bunx biome --version
|
|
432
|
+
0.3.3
|
|
433
|
+
$ bunx biome@0.3.3 --version
|
|
434
|
+
0.3.3
|
|
435
|
+
$ bunx --package @biomejs/biome@2.6.3 biome --version
|
|
436
|
+
Version: 2.6.3
|
|
437
|
+
```
|
|
438
|
+
|
|
439
|
+
Only the third runs the formatter, and what separates it from the second is the *name*, not the
|
|
440
|
+
version. No runner exempts you from the coordinate half; each one's syntax just makes a different
|
|
441
|
+
way of getting it wrong easy to type.
|
|
442
|
+
|
|
443
|
+
- `bunx <binary>@<version>` resolves the token as an npm package name, which is why
|
|
444
|
+
`--package <name>@<version> <binary>` exists: a bare `bunx biome@0.3.3` is a coordinate nobody
|
|
445
|
+
checked.
|
|
446
|
+
- `uvx <command>@<version>` does the same on PyPI, and carries it one step further: uv resolves
|
|
447
|
+
the token as the **project**, installs that project, and only then looks for a command of that
|
|
448
|
+
name.
|
|
449
|
+
|
|
450
|
+
```text
|
|
451
|
+
$ uvx pyyaml@6.0.2 --version
|
|
452
|
+
Installed 1 package in 23ms
|
|
453
|
+
Package `pyyaml` does not provide any executables.
|
|
454
|
+
```
|
|
455
|
+
|
|
456
|
+
Nothing there is about the version — `pyyaml` is a real project at a real version, and it is not
|
|
457
|
+
what you meant by typing a command's name. `uvx --help` is where the distinction lives:
|
|
458
|
+
`--from <FROM> Use the given package to provide the command`. Hence
|
|
459
|
+
`uvx --from shellcheck-py@0.10.0.1 shellcheck --version`, which prints `ShellCheck - shell
|
|
460
|
+
script analysis tool` and `version: 0.10.0`: the distribution and the command share no name at
|
|
461
|
+
all.
|
|
462
|
+
- `go run <module>@<version>` takes the coordinate as that domain-scoped URL, so a path you
|
|
463
|
+
invented usually fails to resolve rather than running someone else's code — but a module path
|
|
464
|
+
guessed from a binary's name is still a guess, and the checksum database will pin the wrong
|
|
465
|
+
module's bytes as faithfully as the right one's.
|
|
466
|
+
|
|
467
|
+
Nothing refuses an unpinned command: no guard, no lint and no check stops you typing a bare `bunx
|
|
468
|
+
biome`. What you are given is a correct form and a tool that is already there, not a gate — so both
|
|
469
|
+
halves are yours to type.
|
|
470
|
+
|
|
471
|
+
## When no runner reaches it
|
|
472
|
+
|
|
473
|
+
A tool that is neither in the image nor reachable through one of the three runners is an image
|
|
474
|
+
change: a pull request against cybermonklabs/legion that carries the same name through every site that
|
|
475
|
+
spells it out, and those sites are not all held by a test.
|
|
476
|
+
|
|
477
|
+
Do not work from a list of those sites. This page has carried one twice and it was wrong both
|
|
478
|
+
times, so take them from the repository instead: grep for a client the image already carries,
|
|
479
|
+
under both of its names — the command and the Debian package that provides it are spelled
|
|
480
|
+
differently, and different sites spell each — and carry the new name through everything the two
|
|
481
|
+
find.
|
|
482
|
+
|
|
483
|
+
```text
|
|
484
|
+
$ grep -rn redis-cli
|
|
485
|
+
$ grep -rn redis-tools
|
|
486
|
+
```
|
|
487
|
+
|
|
488
|
+
One kind of hit wants more than a substitution: the toolchain smoke step in
|
|
489
|
+
`packages/daemon/docker/worker.Dockerfile` runs each tool once, so a new one needs its own
|
|
490
|
+
invocation there (`nc` and `dig` take no `--version`). The licence file the image carries at
|
|
491
|
+
`/usr/share/doc/legion/THIRD_PARTY_NOTICES` needs no edit: its list of Debian packages is written at
|
|
492
|
+
build time from the image's own dpkg database.
|
|
493
|
+
|
|
494
|
+
Four of the files the grep finds are in Go, and those four are the gated ones: the `toolchain`
|
|
495
|
+
name list in `packages/daemon/internal/capabilities/image.go` makes the probe require it; that
|
|
496
|
+
row's summary in `packages/daemon/internal/capabilities/capabilities.go` has to name it, a test
|
|
497
|
+
holding the summary to the list; the stub binaries in
|
|
498
|
+
`packages/daemon/internal/capabilities/capabilitiestest/capabilitiestest.go` have to include it, or
|
|
499
|
+
the capability and `probe-image` tests fail on an image that does not carry it; and
|
|
500
|
+
`packages/daemon/internal/capabilities/image_test.go` spells the expected detail string out again.
|
|
501
|
+
|
|
502
|
+
Miss one of those four and the suite is red, which is what makes them easy to find. Nothing holds
|
|
503
|
+
a Dockerfile comment or the smoke step to anything, so a change that stops at the gated sites opens
|
|
504
|
+
green with comments that no longer describe the image and a smoke step that never runs the new
|
|
505
|
+
tool.
|
|
506
|
+
|
|
507
|
+
So when your work needs such a tool, say so to the architect that owns your issue (`envoy_publish`
|
|
508
|
+
to its role topic), with the tool, what your phase needs it for, and the runner forms you already
|
|
509
|
+
ruled out. What is not the step is improvising a CLI for your own convenience — a vendored binary,
|
|
510
|
+
a hand-rolled substitute, or an extraction prefix dropped on your own `PATH`: the architect decides
|
|
511
|
+
whether the image change is work to schedule. An extraction a harness builds for its own runs is
|
|
512
|
+
not improvising and is not what this rules out (the Dispatch e2e stack, above): the two cases part
|
|
513
|
+
on "The test: where it lands, and what it leaves resolving", which the convenience case fails.
|
|
514
|
+
|
|
515
|
+
The operator's side of this — why the image carries those five clients rather than anything else,
|
|
516
|
+
the five priced options with their measurements, and what the clients cost the image — is in
|
|
517
|
+
`docs/kubernetes.md`, the subsection "The tools a role's own work needs".
|
|
518
|
+
|
|
519
|
+
## Before you measure Oh My Pi's binary
|
|
520
|
+
|
|
521
|
+
Read Oh My Pi's source first, from the fork (`sjawhar/oh-my-pi`) at the tag `.omp-pin` names, not
|
|
522
|
+
from upstream's version tag: the fork's release carries code upstream's does not. `.omp-pin` reads
|
|
523
|
+
`github:sjawhar/oh-my-pi@<version>`, and the git tag adds a `v` in front of that text, so find it
|
|
524
|
+
rather than type it:
|
|
525
|
+
|
|
526
|
+
```text
|
|
527
|
+
$ git ls-remote --tags https://github.com/sjawhar/oh-my-pi.git 'v<version>'
|
|
528
|
+
```
|
|
529
|
+
|
|
530
|
+
Search that source by the name of a construct or a setting, never by its value: one value is
|
|
531
|
+
spelled in many places, and the code that sets it may spell it in none. If the module the binary
|
|
532
|
+
names is absent at that tag, the binary is the ground truth: measure it, and say the source did not
|
|
533
|
+
carry it. In a pod the binary (`$LEGION_OMP_PATH`) is the build the worker image was made from,
|
|
534
|
+
and your workspace's `.omp-pin` can name another, so the tag you read may not be the build you
|
|
535
|
+
measure: compare `"$LEGION_OMP_PATH" --version` with the pin first, and say which build each
|
|
536
|
+
figure came from
|
|
537
|
+
(`docs/solutions/testing/a-worker-pods-oh-my-pi-is-the-fleets-pin-not-the-branchs-fetch-the-branchs-build-and-say-which-pin-each-figure-came-from-LEGION-247.md`).
|
|
538
|
+
|
|
539
|
+
## The tests your pod cannot run
|
|
540
|
+
|
|
541
|
+
Some tests fail in a Kubernetes issue pod whatever your change, because of the pod, not the branch.
|
|
542
|
+
A tmux pane runs on its host, and none of this holds there. Their evidence is CI's run at the head:
|
|
543
|
+
cite that run, and name in your handoff each package the pod could not run. The cases below are
|
|
544
|
+
the known ones, not every one: a failure they do not name is yours to explain. Several of them
|
|
545
|
+
reach any work a pod runs, a test or a tool of your own, not a test suite alone. One habit holds
|
|
546
|
+
them together: check the result, never the request. A call's exit code, a reported limit and a
|
|
547
|
+
job's phase each answer a different question than the one you asked, so where you set something,
|
|
548
|
+
read it back and refuse on a mismatch, and where you read something, confirm the tool ran. An
|
|
549
|
+
empty result is evidence only then: a missing binary, a refused permission, or a truncated
|
|
550
|
+
search gives you silence that reads as a clean answer, so check that the thing you counted had
|
|
551
|
+
something to count.
|
|
552
|
+
|
|
553
|
+
- **Read the summary your work wrote, never the Job's phase.** A Kubernetes Job that reports
|
|
554
|
+
`Succeeded` says its container exited 0, and nothing more: read the artefact your work was to
|
|
555
|
+
produce, or the log line that proves it ran, before you believe any result. A lane measured
|
|
556
|
+
this on 2026-11-01: two pods running a native toolchain died with `Error: Stack overflow.` after 23
|
|
557
|
+
seconds, and their Job still reported `Succeeded`, because the process they ran (`rocq repl`)
|
|
558
|
+
exits 0 when a command inside it fails. The toolchain is incidental; any interpreter, driver or
|
|
559
|
+
wrapper that swallows an inner failure gives the same clean phase over a silent no-result, and
|
|
560
|
+
a lane that trusts the phase records it as a run that worked.
|
|
561
|
+
|
|
562
|
+
Two habits make that phase true. Detect the failure in the text your tool prints and exit
|
|
563
|
+
non-zero yourself, rather than passing the wrapper's exit code on. And write a verdict that
|
|
564
|
+
matters to durable storage as it is produced, not at the end: a node can vanish within a minute
|
|
565
|
+
of a pod ending, so an unpersisted log is gone even when nothing killed anything, and a
|
|
566
|
+
terminating pod's grace period can be as short as 25 seconds, which is no time to flush.
|
|
567
|
+
|
|
568
|
+
**The phase lies in the other direction too, and that one is worse.** A lane measured a Job
|
|
569
|
+
reporting `Running` for 23 minutes while its pod count was **zero**: the quota was full, every
|
|
570
|
+
pod create was refused (`exceeded quota … requested: requests.cpu=16, used: 198, limited:
|
|
571
|
+
200`), and the controller kept retrying. A watcher reading the phase would have called that
|
|
572
|
+
build running all night. **The phase is a request; the pod list is the result.** If you watch
|
|
573
|
+
a Job, watch `kubectl get pods -l batch.kubernetes.io/job-name=<job>`, or watch the marker
|
|
574
|
+
your own work writes — never the phase alone.
|
|
575
|
+
- **gVisor.** Every pod runs under gVisor, which does not give a test every interface the Linux
|
|
576
|
+
kernel does. In `packages/envoy` the tests of `cmd/agent-secrets`, `cmd/agent-secrets-helper`
|
|
577
|
+
and `internal/broker/helper` fail: the helper identifies its peer with `SO_PEERPIDFD`, which
|
|
578
|
+
gVisor answers `protocol not available` (`PEER_UNIDENTIFIED` in the test output). The
|
|
579
|
+
terminal-prompt tests of `cmd/agent-secrets` also need terminal job-control ioctls
|
|
580
|
+
(`inappropriate ioctl for device`), and the `SigCgt` line of `/proc/<pid>/status`, which the
|
|
581
|
+
pods' gVisor lacked on 2026-10-28; upstream gVisor added that line on 2026-09-24, so a node on a
|
|
582
|
+
later release may have it. A timing test can fail too: `internal/dispatch/pmdoc`'s linearity
|
|
583
|
+
tests time a call by the process's CPU time, which gVisor counts in 10 ms ticks, so a call under
|
|
584
|
+
10 ms can measure 0 s and the test reports `+Inf times`.
|
|
585
|
+
- **`O_NOFOLLOW` guards a leaf and not a component under gVisor.** A lane measured this on
|
|
586
|
+
2026-11-01, in a pod under gVisor (`uname` 4.19.0-gvisor) against the same probe on an ordinary
|
|
587
|
+
kernel, on two filesystems. Almost everything matches: `O_RDONLY|O_NOFOLLOW` on a symlink to a
|
|
588
|
+
file refuses `ELOOP`, `O_CREAT|O_EXCL|O_NOFOLLOW` on one refuses `EEXIST`, `O_PATH|O_NOFOLLOW`
|
|
589
|
+
opens the link itself, and the controls (a regular file, a real directory, a free name) succeed
|
|
590
|
+
on both. One call differs: `openat(dirfd, <a link to a directory>,
|
|
591
|
+
O_DIRECTORY|O_NOFOLLOW|O_RDONLY|O_CLOEXEC)`, which the ordinary kernel refuses `ENOTDIR` and
|
|
592
|
+
gVisor follows with no error. So code that walks a path one component at a time and opens each
|
|
593
|
+
with `O_NOFOLLOW` is defeated there while every leaf check still passes, and its tests stay
|
|
594
|
+
green on a developer's kernel. A whole block of such tests failing in a pod is that one
|
|
595
|
+
property of the environment, not one defect per test. The fix shape holds on both kernels: the
|
|
596
|
+
walk cannot trust the flag, so `lstat` the component, `fstat` the descriptor it opened, compare
|
|
597
|
+
`st_dev` and `st_ino`, and refuse when they differ or when the `lstat` says `S_IFLNK`.
|
|
598
|
+
`openat2` is not the alternative there: a lane measured it returning `ENOSYS` under gVisor for
|
|
599
|
+
every flag combination it tried, so the `lstat` and `fstat` comparison above is the portable
|
|
600
|
+
check rather than a fallback. The question is not which tool to use but whether anything you
|
|
601
|
+
run calls `openat2`, which you can read off the binary (two builds of one tool reporting an
|
|
602
|
+
identical `--version` differed here, and only the package version and the checksum told them
|
|
603
|
+
apart):
|
|
604
|
+
|
|
605
|
+
```sh
|
|
606
|
+
b=$(readlink -f "$(command -v <tool>)")
|
|
607
|
+
n=$(nm -D --undefined-only "$b" | wc -l)
|
|
608
|
+
bad=$(nm -D --undefined-only "$b" | grep -c openat2 || true) # grep -c exits 1 on no match
|
|
609
|
+
[ "$n" -gt 0 ] || { echo "no dynamic symbols: static binary or nm failed, use a tracer"; exit 2; }
|
|
610
|
+
[ "$bad" -eq 0 ] || { echo "this binary calls openat2 and fails under gVisor"; exit 2; }
|
|
611
|
+
```
|
|
612
|
+
|
|
613
|
+
The total count is the control: a total of zero is a static binary or a failed `nm`, not a
|
|
614
|
+
clean result. **An undefined `openat2` symbol proves the binary calls it. Its absence proves
|
|
615
|
+
nothing**, so the check is a positive test only: a caller can reach the syscall through
|
|
616
|
+
`syscall(SYS_openat2, …)`, which leaves `syscall` in the table and never `openat2`, and a libc
|
|
617
|
+
that exports no `openat2` wrapper produces the symbol for no binary at all — measured on a
|
|
618
|
+
glibc 2.39 host, where no binary in `/usr/bin` carries it and `tar` shows 227 undefined
|
|
619
|
+
symbols, `openat2` zero and `syscall` one. To establish that a binary is safe rather than
|
|
620
|
+
suspect, trace it.
|
|
621
|
+
- **Set the stack with `ulimit -s`, then read both limits back.** The check, not the number,
|
|
622
|
+
is the durable part. 131072 is the number for an ordinary job; name a larger one when a tool
|
|
623
|
+
you run polices its own limit and demands it:
|
|
624
|
+
|
|
625
|
+
```sh
|
|
626
|
+
want=131072 # a tool with its own stack guard may demand more; use its minimum
|
|
627
|
+
ulimit -S -s "$want" 2>/dev/null || { ulimit -H -s "$want" 2>/dev/null || true; ulimit -S -s "$want"; }
|
|
628
|
+
{ [ "$(ulimit -Hs)" = unlimited ] && ulimit -H -s "$want" 2>/dev/null; } || true
|
|
629
|
+
soft=$(ulimit -Ss); hard=$(ulimit -Hs)
|
|
630
|
+
[ "$soft" = "$want" ] || { echo "stack soft is $soft, want $want; refusing"; exit 2; }
|
|
631
|
+
[ "$hard" != unlimited ] && [ "$hard" -ge "$want" ] || { echo "stack hard is $hard; refusing"; exit 2; }
|
|
632
|
+
```
|
|
633
|
+
|
|
634
|
+
The soft limit must equal what you asked for, and the hard limit must be finite and no lower.
|
|
635
|
+
`unlimited` is refused by name, because it reports `unlimited` and gives 8 MiB.
|
|
636
|
+
Three things in that setter each answer a measured failure, and a simpler form breaks one of
|
|
637
|
+
them. **The soft limit is set first**, because lowering the hard limit below a soft
|
|
638
|
+
`unlimited` is `EINVAL` — `ulimit -H -s 131072` from that state fails with `cannot modify
|
|
639
|
+
limit: Invalid argument`, and chained with `&&` the soft set never runs, so the process keeps
|
|
640
|
+
`unlimited` and gets 8 MiB, the failure the rule exists to stop. **The hard limit is set only
|
|
641
|
+
when it reads `unlimited`**, because setting it unconditionally lowers a higher finite hard
|
|
642
|
+
limit the pod gave you, which is the headroom a tool with its own guard may need. The
|
|
643
|
+
**`|| true` around it** is not decoration: on the ordinary path, where the hard limit is
|
|
644
|
+
already finite, that test is false, and a bare `&&` list whose test fails is a non-zero exit
|
|
645
|
+
status. `set -e` does not abort there, so the script runs to the end and does all of its work
|
|
646
|
+
— and if that line is the last one in the file, the script exits 1 with everything correct.
|
|
647
|
+
Measured: as a last line, `exit=1`; with `|| true`, `exit=0`; and with any command after it,
|
|
648
|
+
`exit=0` either way, which is why it is easy to test and find nothing. And the
|
|
649
|
+
**fallback** covers a want above the current hard limit, where the soft set fails first.
|
|
650
|
+
Measured from three starting states: `unlimited/unlimited` and `8192/unlimited` both end at
|
|
651
|
+
`131072/131072`, and `131072/1048576` ends at `131072/1048576` with the higher hard limit
|
|
652
|
+
kept. Do not reduce it to a single `ulimit -s <n>`: that set both limits in a fresh shell
|
|
653
|
+
here and left the hard limit `unlimited` in a long-lived one, same binary and same starting
|
|
654
|
+
pair, cause unknown — which is why the read-back is the rule and the setter is not.
|
|
655
|
+
The ceiling is 128 MiB whichever finite value you choose —
|
|
656
|
+
gVisor maps the whole stack at exec and caps it at `maxStackSize = 128 << 20` in `MapStack`
|
|
657
|
+
(`pkg/sentry/mm/syscalls.go`), with `RLIM_INFINITY` falling back to the 8 MiB default — and a
|
|
658
|
+
larger soft limit costs rather than buys, since glibc sizes a default thread stack from it and
|
|
659
|
+
a tool that creates a thread pool can fail to allocate one. A lane measured both on 2026-11-01
|
|
660
|
+
in pods under gVisor (4.19.0-gvisor), with a C probe and a second probe in another language:
|
|
661
|
+
under `unlimited` the fault came at 7 MiB and 524,150 frames; at 131072 (KiB) both probes
|
|
662
|
+
faulted at 127 MiB and 8,388,470 frames. The same probes on an ordinary kernel track the
|
|
663
|
+
setting as expected, so this is the sandbox and not the probes.
|
|
664
|
+
- **Compare durations within one pod, never across pods.** A gVisor sandbox runs syscall-heavy
|
|
665
|
+
work 2.2 to 3.8 times slower than the same work outside it, and a pod lands on whatever node
|
|
666
|
+
the autoscaler provisions, so two pods are two machines, two sets of neighbours and two
|
|
667
|
+
moments. Three parts, all measured: within one pod, comparable — run both arms there, as two
|
|
668
|
+
containers on one node started together, and a difference between them is a difference between
|
|
669
|
+
the arms; across pods, not comparable; across platforms, not until something measures the gap.
|
|
670
|
+
Label every figure taken there a gVisor figure. That factor is the cost of syscalls and no
|
|
671
|
+
budget for a whole job: a lane measured two real builds of the same work faster in a pod than
|
|
672
|
+
on a virtual machine (1,014 s against 1,546 s, and 620 s against 1,117 s), with system time
|
|
673
|
+
roughly doubled and user time lower, so compute-bound work can run as fast or faster there.
|
|
674
|
+
- **No Docker.** The image carries no container runtime, so a test that starts one through
|
|
675
|
+
testcontainers panics `rootless Docker not found`: in `packages/envoy`, the tests that use
|
|
676
|
+
`internal/tests3` (`internal/dispatch/files`, which `DISPATCH_TEST_NO_CONTAINERS=1` skips) and
|
|
677
|
+
`internal/testpg` (unless `DISPATCH_TEST_DATABASE_URL` names a Postgres).
|
|
678
|
+
|
|
679
|
+
The daemon's NATS-backed tests have a way round it: `packages/daemon/internal/testnats` serves
|
|
680
|
+
the server `LEGION_TEST_NATS_URL` and `LEGION_TEST_NATS_MONITOR_URL` name, and starts no
|
|
681
|
+
container when both are set, so run one yourself and point them at it:
|
|
682
|
+
|
|
683
|
+
```sh
|
|
684
|
+
go run github.com/nats-io/nats-server/v2@v2.15.1 -js -p 14222 -m 18222 -sd "$TMPDIR/nats" &
|
|
685
|
+
export LEGION_TEST_NATS_URL=nats://127.0.0.1:14222 LEGION_TEST_NATS_MONITOR_URL=http://127.0.0.1:18222
|
|
686
|
+
```
|
|
687
|
+
|
|
688
|
+
Both or neither: one set without the other fails the test rather than falling back. The
|
|
689
|
+
monitor port is read, not decoration — the per-test reset waits there for every client
|
|
690
|
+
connection to end. One server serves one test binary at a time, so name one package a
|
|
691
|
+
command (or pass `-p 1`): two packages' binaries on one server delete each other's streams.
|
|
692
|
+
Pick ports no sibling role holds, and keep the store directory off your home. In your own
|
|
693
|
+
work, run the NATS-backed tests your change touches by name (`go test -run`), as the first
|
|
694
|
+
section says. Measured here, as proof of the route and not as a run to repeat:
|
|
695
|
+
`go run …@v2.15.1 --version` prints `nats-server: v2.15.1`, and with the two variables set
|
|
696
|
+
`go test -count=1 -timeout 40m ./internal/workflow/` runs whole in the pod, with no docker
|
|
697
|
+
panic.
|
|
698
|
+
- **The rest, by package**, for a named run the pod does make. A Postgres unpacked from Debian
|
|
699
|
+
packages (the harness under **The worked case**) has only the `C`, `C.utf8` and `POSIX`
|
|
700
|
+
locales, so its default collation is byte order, and `internal/dispatch/api`'s
|
|
701
|
+
`TestIssueRanksStayOrderedPastLowercaseDigitsAndProductionShapedMoves` refuses such a database
|
|
702
|
+
by its own message. The pod's own `legion-testpg` server is made with ICU `en-US`, so that
|
|
703
|
+
test runs there, as it does in CI's `envoy-go` job, whose Postgres has `en_US.utf8`.
|
|
704
|
+
`internal/dispatch/pmdoc`'s browser-parity tests need `bun install` to have run for
|
|
705
|
+
`@legion/proof-editor`. In the daemon's module, never export `GOWORK=off` inside
|
|
706
|
+
`$LEGION_WORKSPACE`: `packages/daemon/go.mod` replaces `github.com/sjawhar/envoy => ../envoy`,
|
|
707
|
+
and the go.sum entries its packages need live in `packages/envoy/go.sum`, which only the
|
|
708
|
+
`go.work` workspace mode consults.
|