@vzn/vx 0.0.631 → 0.0.633
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +21 -18
- package/docs/caching.md +2247 -0
- package/docs/cli.md +3214 -0
- package/docs/execution.md +708 -0
- package/docs/features.md +172 -0
- package/docs/patterns.md +219 -0
- package/docs/schema.md +2137 -0
- package/docs/security.md +209 -0
- package/package.json +14 -5
- package/schemas/cache.json +24 -0
- package/schemas/docs.json +28 -0
- package/schemas/error.json +23 -0
- package/schemas/info.json +147 -0
- package/schemas/init.json +57 -0
- package/schemas/last.json +188 -0
- package/schemas/lock.json +21 -0
- package/schemas/plan.json +88 -0
- package/schemas/show.json +124 -0
- package/schemas/summary.json +133 -0
- package/schemas/why.json +168 -0
- package/skills/vx/SKILL.md +89 -0
- package/src/bin.ts +9 -0
- package/src/cache/archive.ts +153 -51
- package/src/cache/cache.ts +542 -137
- package/src/cache/chained-cache.ts +9 -0
- package/src/cache/inputs.ts +5 -0
- package/src/cache/layer.ts +10 -5
- package/src/cache/layered-cache.ts +4 -0
- package/src/cache/schema.ts +44 -17
- package/src/cache/tar-stream.ts +204 -66
- package/src/cache/zstd.ts +1 -1
- package/src/cli/cache.ts +14 -14
- package/src/cli/docs-corpus.ts +21 -0
- package/src/cli/docs-text.d.ts +5 -0
- package/src/cli/docs.ts +167 -0
- package/src/cli/foreign-flags.ts +1 -1
- package/src/cli/help.ts +42 -7
- package/src/cli/index.ts +25 -11
- package/src/cli/info.ts +38 -9
- package/src/cli/init.ts +131 -24
- package/src/cli/last.ts +54 -3
- package/src/cli/lock.ts +43 -15
- package/src/cli/plan-format.ts +23 -1
- package/src/cli/plugin-commands.ts +3 -0
- package/src/cli/plugin-templates.ts +1 -1
- package/src/cli/run.ts +40 -21
- package/src/cli/show.ts +10 -7
- package/src/cli/upgrade.ts +19 -7
- package/src/cli/watch.ts +7 -2
- package/src/cli/why.ts +14 -40
- package/src/exec/runner.ts +27 -10
- package/src/graph/scheduler.ts +36 -0
- package/src/graph/task-graph.ts +4 -4
- package/src/index.ts +22 -5
- package/src/orchestrator/admission.ts +29 -1
- package/src/orchestrator/affected-tasks.ts +87 -18
- package/src/orchestrator/doctor.ts +14 -1
- package/src/orchestrator/execute-task.ts +149 -64
- package/src/orchestrator/failure-mode.ts +5 -3
- package/src/orchestrator/framed-output.ts +17 -3
- package/src/orchestrator/index.ts +11 -0
- package/src/orchestrator/logger.ts +57 -6
- package/src/orchestrator/metrics.ts +76 -4
- package/src/orchestrator/miss-reason.ts +3 -5
- package/src/orchestrator/miss-save.ts +6 -5
- package/src/orchestrator/options.ts +12 -0
- package/src/orchestrator/output-log.ts +14 -1
- package/src/orchestrator/path-links.ts +79 -0
- package/src/orchestrator/plan.ts +20 -3
- package/src/orchestrator/plugin.ts +7 -0
- package/src/orchestrator/prepare.ts +14 -4
- package/src/orchestrator/projects.ts +2 -2
- package/src/orchestrator/run-artifacts.ts +18 -2
- package/src/orchestrator/run-failures.ts +192 -0
- package/src/orchestrator/run-records.ts +10 -8
- package/src/orchestrator/run.ts +87 -22
- package/src/orchestrator/task-hash.ts +4 -4
- package/src/orchestrator/terminal-signals.ts +16 -0
- package/src/orchestrator/upstream.ts +65 -10
- package/src/util/errors.ts +55 -1
- package/src/util/index.ts +3 -0
- package/src/util/secret-mask.ts +13 -1
- package/src/util/verbs.ts +1 -0
- package/src/workspace/config-schema.ts +10 -8
- package/src/workspace/migration.ts +30 -3
- package/src/workspace/workspace.ts +1 -0
package/docs/caching.md
ADDED
|
@@ -0,0 +1,2247 @@
|
|
|
1
|
+
# Caching
|
|
2
|
+
|
|
3
|
+
`@vzn/vx`'s cache is content-addressed, opt-in per task, and shaped to
|
|
4
|
+
cascade through the dependency graph the same way Turborepo's does.
|
|
5
|
+
This page explains _what's in the cache key_, _what triggers
|
|
6
|
+
invalidation_, _what's actually stored_, and _why_.
|
|
7
|
+
|
|
8
|
+
## Why caching is opt-in
|
|
9
|
+
|
|
10
|
+
A task is cached iff its `TaskConfig` provides a `cache` block, with
|
|
11
|
+
**both** `inputs.files` and `outputs.files`. Omit `cache` and the
|
|
12
|
+
task always runs; no read, no write.
|
|
13
|
+
|
|
14
|
+
The reasoning:
|
|
15
|
+
|
|
16
|
+
- **Defaulting caching ON with implicit globs leads to silently stale
|
|
17
|
+
builds.** The first time a user forgets to revisit their config to
|
|
18
|
+
add an input, the cache returns a hit for an out-of-date snapshot.
|
|
19
|
+
Stale hits are the worst failure mode of a task runner — they erode
|
|
20
|
+
trust in the cache itself.
|
|
21
|
+
- **Forcing declaration makes "what does this task read?" and "what
|
|
22
|
+
does it produce?" conscious choices.** The user pays a one-time
|
|
23
|
+
cost (write the globs) for a permanent gain (the cache key actually
|
|
24
|
+
reflects reality).
|
|
25
|
+
- **The cost of a forgotten cache miss is small.** A task re-runs.
|
|
26
|
+
The cost of a stale cache hit is large. Asymmetric risk justifies
|
|
27
|
+
asymmetric defaults.
|
|
28
|
+
|
|
29
|
+
Turbo defaults caching ON for `outputs: []` tasks; Nx requires you to
|
|
30
|
+
opt _out_ via `cache: false`. We chose the strictest of the three.
|
|
31
|
+
|
|
32
|
+
## Cache key derivation
|
|
33
|
+
|
|
34
|
+
The cache key for one task is a **16-hex xxHash3 digest**, seed-chained
|
|
35
|
+
over (in order):
|
|
36
|
+
|
|
37
|
+
1. **`CACHE_VERSION`** — the key-derivation sentinel
|
|
38
|
+
(currently `'vx-cache-v42'`, in `src/cache/key-fold.ts`). Bumped when
|
|
39
|
+
the key derivation or the artifact container changes, or stored bytes
|
|
40
|
+
are wrong under an unchanged key. See
|
|
41
|
+
[§ Bumping CACHE_VERSION](#bumping-cache_version).
|
|
42
|
+
2. **`taskId`** — `${projectName}#${taskName}`. Two tasks with
|
|
43
|
+
identical everything else still produce distinct keys — protects
|
|
44
|
+
against e.g. `pkg-a#build` accidentally cache-hitting on a
|
|
45
|
+
`pkg-b#build` entry.
|
|
46
|
+
3. **Workspace fingerprint** — xxh3 of every supported workspace
|
|
47
|
+
marker found at the root (see
|
|
48
|
+
[`modules/fingerprint.md`](./modules/fingerprint.md)):
|
|
49
|
+
`pnpm-lock.yaml`, `package-lock.json`, `npm-shrinkwrap.json`,
|
|
50
|
+
`yarn.lock`, `bun.lock`, `bun.lockb`, `pnpm-workspace.yaml`,
|
|
51
|
+
`.yarnrc.yml` (Yarn 4 catalogs, which `yarn.lock` does not record),
|
|
52
|
+
`.npmrc` and `bunfig.toml` (install settings no lockfile records),
|
|
53
|
+
and the content of each patch the Bun lockfile names under
|
|
54
|
+
patchedDependencies (it records a patch by path alone). Any
|
|
55
|
+
install-resolved change (a `bun install` that bumps `bun.lock`) or
|
|
56
|
+
any workspace-shape change invalidates _every_ cache entry. This is
|
|
57
|
+
the single global "the world changed" lever — and a plugin can take
|
|
58
|
+
one file off it: `VxPlugin.fingerprint` claims a lockfile, core
|
|
59
|
+
leaves it out of this digest, and the plugin's `key` hook folds what
|
|
60
|
+
the file means for each project instead (`@vzn/vx-lockfile`'s `pnpm()`
|
|
61
|
+
folds the project's own resolved dependency closure and the root
|
|
62
|
+
package's, so `pnpm update foo` re-keys only the projects that depend
|
|
63
|
+
on `foo` — every project when the root does, since the root's tools
|
|
64
|
+
run from the root `node_modules/.bin` on every task's PATH; its `bun()` is the same
|
|
65
|
+
for `bun.lock`, and this repo declares that one). The
|
|
66
|
+
config-evaluation cache still keys on every file: a config may import
|
|
67
|
+
a dependency the lockfile resolved.
|
|
68
|
+
|
|
69
|
+
The fingerprint is read once per run, before any task. A task in the
|
|
70
|
+
root project may rewrite one of these files — `pnpm install` without
|
|
71
|
+
`--frozen-lockfile` updates the lockfile and the installed tree — and
|
|
72
|
+
every key taken before it names the old one. So a task that may
|
|
73
|
+
(unsandboxed in the root project, or granted a write over one) tells
|
|
74
|
+
the run when its command ran, the run re-checks the files then (one
|
|
75
|
+
`stat` each, following a symlink as both reads do — an `lstat` missed
|
|
76
|
+
a lockfile rewritten through a link, item 760 — the bytes compared
|
|
77
|
+
for any written since the read), and
|
|
78
|
+
once one moved nothing keyed on the old digest is probed or saved for
|
|
79
|
+
the rest of the run, with one status line naming the file. Every
|
|
80
|
+
reader after such a task is keyed late, not up front. Without it the
|
|
81
|
+
run after a lockfile rewrite restored a build made against the old
|
|
82
|
+
install, and a save filed a build against the new install under the
|
|
83
|
+
old lockfile's key, replayed whenever the tree went back (item 750,
|
|
84
|
+
[`modules/fingerprint-watch.md`](./modules/fingerprint-watch.md)).
|
|
85
|
+
|
|
86
|
+
4. **Project `package.json` hash** — the git blob OID of the project's
|
|
87
|
+
`package.json`: the index OID when the file is tracked and clean,
|
|
88
|
+
else `cache.hashFile`'s blob OID of the worktree bytes (empty when
|
|
89
|
+
there is none). Folded in implicitly (Turbo / Nx parity).
|
|
90
|
+
Covers the case where `cache.inputs.files: ['src/**']` is narrow
|
|
91
|
+
and a `package.json` dep change would otherwise leak undetected.
|
|
92
|
+
(Added at v12; rationale in [§ History](#history).)
|
|
93
|
+
5. **Task config hash** — `xxh3(JSON.stringify(hashableConfig(node.config)))`
|
|
94
|
+
of the _evaluated_ task config. The projection is one field wide:
|
|
95
|
+
`exec.remote` is dropped, because it says WHERE a task runs and not
|
|
96
|
+
what it does (§ What's NOT in the key). Captures:
|
|
97
|
+
- `exec` block (command, env declarations, timeout, persistent).
|
|
98
|
+
- `dependsOn` and `cache.inputs.tasks` declarations.
|
|
99
|
+
- `cache.outputs.files`, `cache.inputs.files`, `cache.inputs.env`,
|
|
100
|
+
`cache.inputs.runtime` / `workspaceRuntime` declarations (the
|
|
101
|
+
strings themselves; their resolved file content / env values /
|
|
102
|
+
command output contribute separately).
|
|
103
|
+
- `description` (because it's part of the resolved object — even
|
|
104
|
+
though it has no behavioural effect; a description change isn't a
|
|
105
|
+
correctness change but the cost of a re-run is low).
|
|
106
|
+
- **Imported / computed values** — anything a preset or
|
|
107
|
+
`process.env`-read at config-load time injected. Bun's native
|
|
108
|
+
`await import()` evaluates the module and bakes those values into
|
|
109
|
+
the object before we serialize.
|
|
110
|
+
6. **`forwardArgs`** — CLI args passed after `--`. Folded into the
|
|
111
|
+
key so `vx run test -- --watch` doesn't cache-hit a previous
|
|
112
|
+
`vx run test`. Scoped to the user-requested tasks only — dependsOn-
|
|
113
|
+
pulled deps don't see them (their cache identity stays clean) — and
|
|
114
|
+
to a command: a requested default `build` runs none and folds none
|
|
115
|
+
(X-119).
|
|
116
|
+
7. **`cache.inputs.env` resolved values** — `[name, value]` pairs
|
|
117
|
+
read from host `process.env` at hash time (delimited `name\0value`
|
|
118
|
+
so boundaries are unambiguous). Listed names get their current
|
|
119
|
+
values; an unset name folds its bare name, with no `\0`, so unset
|
|
120
|
+
and set-to-empty are two keys (the child tells them apart:
|
|
121
|
+
`passThrough` leaves an unset name out). A name holding a NUL is
|
|
122
|
+
refused at load.
|
|
123
|
+
8. **`cache.inputs.runtime` resolved output** — `[command, output]`
|
|
124
|
+
pairs, where `output` is the trimmed stdout of each command run via
|
|
125
|
+
`sh -c` in the **project dir** at hash time; a command with stderr
|
|
126
|
+
(or a NUL in stdout) folds `\0<stdout length>\0<stdout><stderr>`,
|
|
127
|
+
so bytes moved between the streams move the key.
|
|
128
|
+
The runtime-output analog of step 7: the command _strings_ are in
|
|
129
|
+
the resolved config (step 5), their _output_ is resolved live every
|
|
130
|
+
run. Folded with the command count + each `command\0output` pair.
|
|
131
|
+
9. **`cache.inputs.workspaceRuntime` resolved output** — same as step
|
|
132
|
+
8 but commands run at the **workspace root**, and the pairs fold
|
|
133
|
+
into a **distinct namespace** (`ws-runtime-values:`) so an identical
|
|
134
|
+
`(command, output)` never aliases the project-cwd `runtime` values.
|
|
135
|
+
10. **Filtered upstream task cache hashes** — every upstream task's
|
|
136
|
+
own cache key, filtered by `cache.inputs.tasks` (default: all of
|
|
137
|
+
them). Sorted by hash before folding so the ordering of `dependsOn`
|
|
138
|
+
doesn't change the key. This is the cascade mechanism: if anything
|
|
139
|
+
beneath you changes, your hash changes too. The set is `dependsOn`'s,
|
|
140
|
+
never the run's selection: a dependency `--exclude-dependencies`
|
|
141
|
+
keeps from running is still folded, with the key a full run would
|
|
142
|
+
derive for it (nx#35234), so the flag moves no key. Its outputs are
|
|
143
|
+
whatever is on disk, which nothing proves current, so a task whose
|
|
144
|
+
key folds one, and everything built on it, may hit but does not
|
|
145
|
+
save (`orchestrator/excluded-keys.ts`); a task with
|
|
146
|
+
`cache.inputs.tasks: []` folds none and saves as usual.
|
|
147
|
+
11. **Plugin key material** — the `{ name: value }` pairs a plugin's
|
|
148
|
+
`key(task, ctx)` stage returned for this task, stored on the node as
|
|
149
|
+
sorted `plugin/name` pairs and folded after the upstream keys and BEFORE the input files,
|
|
150
|
+
ONLY when non-empty — so a workspace with no `key` plugin derives
|
|
151
|
+
byte-identical keys to one before the stage existed (no
|
|
152
|
+
`CACHE_VERSION` bump when it shipped). `vx why` names a changed pair
|
|
153
|
+
as `plugin <plugin>/<name>`. Two plugins of one package (a plugin's
|
|
154
|
+
name is its package's) that return one name are told apart as
|
|
155
|
+
`<name>`, `<name>#2`, … in value order, so declaration order still
|
|
156
|
+
keys nothing (item 1028).
|
|
157
|
+
|
|
158
|
+
12. **Input files' content hashes** — `cache.inputs.files` resolved to
|
|
159
|
+
a concrete list of project-relative paths (gitignore-aware,
|
|
160
|
+
declared-outputs-excluded, nested-projects-excluded), each file
|
|
161
|
+
contributing its **git blob OID** (v20). On a clean tree the OID
|
|
162
|
+
comes straight from the index — the run's up-front enumeration is
|
|
163
|
+
three concurrent spawns, `git ls-files -s -v -z` (every tracked
|
|
164
|
+
path, its OID and its cache-state flag),
|
|
165
|
+
`git status --porcelain -z -uall --ignored=matching --no-renames`
|
|
166
|
+
(dirty tracked paths, the untracked files and the ignored ones) and
|
|
167
|
+
`git var -l` (the clean-filter gate's config) — so deriving these hashes
|
|
168
|
+
costs zero file reads and zero per-file stats. Each trusted OID's
|
|
169
|
+
blob must be the size the index recorded for the file (A-60): `git
|
|
170
|
+
add` under a clean filter (`core.autocrlf=true`, a `text` rule)
|
|
171
|
+
stores the LF blob of a CRLF file, and once the filter is gone git
|
|
172
|
+
holds that stat-clean entry clean without re-reading it, so status
|
|
173
|
+
and the filter gate (today's config) both let the LF blob key the
|
|
174
|
+
CRLF bytes. Which paths an index distrusts is a function of its
|
|
175
|
+
entries, so the verdict is kept in `blob_verdicts` by a hash of
|
|
176
|
+
the index file and the pathspecs: a warm run reads the file and
|
|
177
|
+
one row (the `--debug` listing and a lookup per entry cost 550 ms
|
|
178
|
+
at 100,000 files). The key stands only for an index written before
|
|
179
|
+
the run began and still in place after its listing, so a `git add`
|
|
180
|
+
between the two cannot pair one index's verdict with the other's
|
|
181
|
+
entries; any other run checks every blob. A changed index spawns
|
|
182
|
+
`git ls-files -s -v -z --debug` for the recorded sizes, takes each
|
|
183
|
+
blob's size from `blob_sizes` (fixed for its OID), and asks one
|
|
184
|
+
`git cat-file --batch-check` for the ones not yet known (65 ms
|
|
185
|
+
over 3,000 loose objects, 10 ms packed). The check is by size,
|
|
186
|
+
so a filter that keeps the size (a `filter` driver such as
|
|
187
|
+
`tr a-z A-Z`), removed after an add, still leaves a blob that
|
|
188
|
+
stands for other bytes: probed, the run hit the build of the
|
|
189
|
+
filtered bytes. Only a read of every trusted file would catch it,
|
|
190
|
+
the cost the index OIDs exist to avoid. After changing a filter,
|
|
191
|
+
`git add --renormalize .` makes the index describe the worktree
|
|
192
|
+
again. A
|
|
193
|
+
re-listing mid-run, or a nested repository's project, spawns
|
|
194
|
+
`git ls-files -s --others --exclude-standard -z .` in the project
|
|
195
|
+
dir instead, and its OIDs are not trusted: those files hash by
|
|
196
|
+
content. `--no-renames`, because status pairs a deleted file with a
|
|
197
|
+
similar unmerged path (a conflict mid-resolve) as its rename source
|
|
198
|
+
and prints only `UU <path>`: the deletion went unsaid, the file kept
|
|
199
|
+
its index OID, and the run hit the output built with it (A-59).
|
|
200
|
+
|
|
201
|
+
Ref storage is not a key input: every ref vx reads comes from a git
|
|
202
|
+
command, so a repository in reftable storage (git 2.45) keys every
|
|
203
|
+
task as its files-backend twin (`tests/git-reftable.unsafe.test.ts`).
|
|
204
|
+
|
|
205
|
+
A **submodule or an embedded repository** is enumerated by its own
|
|
206
|
+
git: the workspace repository lists the nested one as a single entry
|
|
207
|
+
(a gitlink, or `dir/` when untracked) and none of its files, so vx
|
|
208
|
+
replaces that entry with the files `git ls-files` lists inside it
|
|
209
|
+
(one spawn per nested repository per run, nested ones recursively),
|
|
210
|
+
and those files hash by content rather than by index OID. That holds
|
|
211
|
+
for a nested repository inside a project (`vendor/lib` under a
|
|
212
|
+
`**` glob — until 2026-09-27 its files never reached the key, and an
|
|
213
|
+
edit there was a hit on the old output), for a project inside one,
|
|
214
|
+
and for a `workspaceFiles` glob. A gitlink whose directory has no
|
|
215
|
+
`.git` (a submodule never initialised, or one whose `.git` was
|
|
216
|
+
removed to vendor its files, the gitlink left in the index) has no
|
|
217
|
+
repository to ask and `git status` says nothing of it: its files
|
|
218
|
+
are listed by a walk and hash by content (A-61), so an empty one
|
|
219
|
+
folds nothing. `--affected` follows the
|
|
220
|
+
same shape — git reports the nested repository as one changed path,
|
|
221
|
+
and every project under it is selected.
|
|
222
|
+
|
|
223
|
+
A task with **no `cache` block** derives a key too — its dependents
|
|
224
|
+
fold it (step 10) — and, declaring nothing, folds **every file in
|
|
225
|
+
its project**: `**/*`, gitignore-aware, untracked files included,
|
|
226
|
+
and its own outputs not excluded, since it declared none. So its
|
|
227
|
+
key moves whenever it writes a file git does not ignore: a cached
|
|
228
|
+
dependent misses once more after the upstream's first run and hits
|
|
229
|
+
from the third. `vx why` on the dependent names the upstream as
|
|
230
|
+
the moved component; on the upstream it says the file set moved
|
|
231
|
+
and that no fingerprints say which. Ignore the outputs in git, or
|
|
232
|
+
declare the block (`tests/uncached-upstream-key.test.ts` pins both
|
|
233
|
+
arms).
|
|
234
|
+
|
|
235
|
+
Declaring nothing, such a task may also have **written** anywhere
|
|
236
|
+
in its project, so what the run learned about that project at its
|
|
237
|
+
start — the git listing, the index OIDs, the `package.json` digest
|
|
238
|
+
— is dropped once its command exits, pass or fail, and a later key
|
|
239
|
+
re-lists and hashes by content. For the same reason a task after it
|
|
240
|
+
in the same project (`dependsOn`, even with `tasks: []`) is never
|
|
241
|
+
keyed up front by the local short-circuit or the remote prefetch.
|
|
242
|
+
Without both, an uncached `gen` that copies a seed into a `build`'s
|
|
243
|
+
declared `config.json` replayed seed B's build under seed A
|
|
244
|
+
(turborepo#13788, item 743). A sandbox narrows the reach to its
|
|
245
|
+
write grants: none reaches nothing, and a grant elsewhere in the
|
|
246
|
+
workspace reaches every project. An unsandboxed write into another
|
|
247
|
+
project crosses a project boundary and is not tracked, as for a
|
|
248
|
+
cached task's undeclared write (`tests/undeclared-writes.test.ts`).
|
|
249
|
+
|
|
250
|
+
A task **with** a `cache` block may still rewrite its own inputs in
|
|
251
|
+
place — a formatter declaring `outputs: []` — and a same-project
|
|
252
|
+
reader after it whose key does not fold its key (`tasks: []`, or a
|
|
253
|
+
filter that leaves it out, on every path) is not keyed up front
|
|
254
|
+
either: its key would be taken over the bytes before the rewrite, and
|
|
255
|
+
seeds A,B,B,A replayed B on the fourth run (item 750). A reader that
|
|
256
|
+
folds the rewriter's key keeps its up-front probe — that key names
|
|
257
|
+
the rewriter's inputs, which are what it may rewrite. A cached task
|
|
258
|
+
writing a file that is neither a declared output nor its own input
|
|
259
|
+
is out of contract: a hit restores only what it declared.
|
|
260
|
+
|
|
261
|
+
A file whose name is **not valid UTF-8** (Linux allows any byte
|
|
262
|
+
but `/` and NUL) cannot be opened from a string, so it cannot be
|
|
263
|
+
hashed: a task whose globs select one is refused by name, and so
|
|
264
|
+
is a task whose declared outputs hold one, rather than the file
|
|
265
|
+
dropping out of the key or the artifact without a word
|
|
266
|
+
(turborepo#9345). Rename it, or exclude it with a negated glob.
|
|
267
|
+
|
|
268
|
+
Your globs are a **filter over the set git reports**, so a filter
|
|
269
|
+
can only ever remove — a gitignored file can never be filtered back
|
|
270
|
+
in, however explicitly you name it. Naming one by hand is therefore
|
|
271
|
+
a **hard error** rather than a silent nothing: it would leave the
|
|
272
|
+
task ignoring a file its own config claims as an input, reporting
|
|
273
|
+
`up-to-date` while that file changed. Turbo lets an explicit entry
|
|
274
|
+
override gitignore; vx cannot, because the key would then depend on
|
|
275
|
+
a change `git diff` cannot see and `--affected` would stop selecting
|
|
276
|
+
the task. If the file is generated, depend on the task that produces
|
|
277
|
+
it via `cache.inputs.tasks`. A **glob** matching nothing stays
|
|
278
|
+
silent — that is legitimate — and so does a literal naming a file
|
|
279
|
+
that does not exist. A literal naming a **directory** is judged the
|
|
280
|
+
same way: an ignored one, or an empty one, has no file git lists
|
|
281
|
+
under it, folds nothing, and is refused (item 576).
|
|
282
|
+
|
|
283
|
+
A **bracket is literal** in a task glob, and there are no character
|
|
284
|
+
classes (item 667): `app/[id]/**` folds the route directory
|
|
285
|
+
`app/[id]`, and `app/\[id\]/**` is the same path. Read as a class
|
|
286
|
+
— what `Bun.Glob`, Turbo and Nx do — it matched `app/i/…` and never
|
|
287
|
+
the route, so the route's files keyed nothing (an edit replayed the
|
|
288
|
+
old output, green) and an output declared under it cleaned an
|
|
289
|
+
unrelated `app/i/page.js` before every run while the artifact
|
|
290
|
+
saved nothing.
|
|
291
|
+
|
|
292
|
+
An index OID is only trusted where git stores the worktree bytes
|
|
293
|
+
**verbatim**, so the enumeration prunes it three ways:
|
|
294
|
+
`git status --porcelain` drops paths whose working tree diverges;
|
|
295
|
+
the `-v` flag on the same `ls-files` spawn drops `skip-worktree` /
|
|
296
|
+
`assume-unchanged` entries, whose OID says nothing about what is
|
|
297
|
+
(or isn't) on disk; and, once those spawns return, a clean-filter
|
|
298
|
+
gate (a `git check-attr` spawn only when the gate needs one) drops paths where `text` / `eol` / `ident`,
|
|
299
|
+
a `filter` driver, `working-tree-encoding`
|
|
300
|
+
or `core.autocrlf` can rewrite bytes between index and worktree
|
|
301
|
+
(the blob would be the LF-normalized form while the task reads the
|
|
302
|
+
CRLF file). The gate costs nothing in a repo with no attributes
|
|
303
|
+
and no `core.autocrlf` — see "Clean filters" below. And when the
|
|
304
|
+
repository's config weakens the stat `git status` judges by —
|
|
305
|
+
`core.trustctime=false`, or `core.checkStat=minimal` (whole-second
|
|
306
|
+
mtime and size) — no index OID is trusted at all: a same-size
|
|
307
|
+
rewrite that keeps its mtime (`cp -p`, `tar -x`) reads clean to
|
|
308
|
+
git, and until 2026-09-27 (A-6) kept the old bytes' key. Every file
|
|
309
|
+
is then hashed through the memo below, which keys on ctime and
|
|
310
|
+
inode.
|
|
311
|
+
|
|
312
|
+
Every pruned path, plus untracked files, falls back to an
|
|
313
|
+
in-process `HASH("blob " + len + "\0" + content)` over the
|
|
314
|
+
**worktree bytes** (sha1, or sha256 in `--object-format=sha256`
|
|
315
|
+
repos), memoized in `file_hashes` on `(mtime, size, ctime, ino)`.
|
|
316
|
+
A file changed within `FILE_HASH_RACY_MS` (50 ms) of its stat is
|
|
317
|
+
hashed but not memoised; a ctime with no sub-second part, as a
|
|
318
|
+
file system that keeps whole seconds writes it (ext3, HFS+,
|
|
319
|
+
FAT/exFAT, some NFS), widens that window by a second, and an even
|
|
320
|
+
second by two, since FAT32 keeps even seconds; a rewrite later in
|
|
321
|
+
the same stamp keeps it (`racyWindowMs`; until 2026-09-27, A-2 and
|
|
322
|
+
A-38, such a rewrite of the same size was a hit on the first bytes'
|
|
323
|
+
output). That is the same value the index holds whenever no filter applies,
|
|
324
|
+
so a file's contribution doesn't flip across dirty↔clean
|
|
325
|
+
transitions. Folded as `(relPath, identity)` pairs, sorted for
|
|
326
|
+
stability across OSes and walk orders. The identity is the OID
|
|
327
|
+
prefixed by the file's git mode unless that mode is a plain
|
|
328
|
+
file's: `100755:<oid>` for an executable, `120000:<oid>` for a
|
|
329
|
+
symlink, the bare OID for 100644. The index gives the mode for a
|
|
330
|
+
clean file, and the stat gives it for any other file (owner
|
|
331
|
+
execute bit). A blob OID holds no mode, so until item 887 a
|
|
332
|
+
`chmod +x`, or a symlink swapped for a file holding its target
|
|
333
|
+
string, kept the key that `git status` and `--affected` saw move.
|
|
334
|
+
Under `core.fileMode=false` (WSL's DrvFs) git reports no chmod, so
|
|
335
|
+
there a clean file's mode comes from an lstat too, and its OID
|
|
336
|
+
from the index (item 1076).
|
|
337
|
+
|
|
338
|
+
A **symlink** folds as git folds it: the blob of its target
|
|
339
|
+
_string_ (its mode-120000 index OID), whether it points at a file,
|
|
340
|
+
a directory, or nothing. Retargeting a link changes the key; the
|
|
341
|
+
bytes behind a link to a directory, or to a file outside the
|
|
342
|
+
project, do not — `git diff` and `--affected` cannot see them
|
|
343
|
+
either, so declare them with `workspaceFiles`. A link to a file
|
|
344
|
+
inside the project tracks that file's content only when the file
|
|
345
|
+
matches the task's input globs too: it is then an input of its own.
|
|
346
|
+
A link under `src/**` to `other/data.txt` folds the link, not
|
|
347
|
+
`other/data.txt`; name the target in the globs to track it.
|
|
348
|
+
(Before 2026-09-09 a file link folded the
|
|
349
|
+
bytes behind it and a directory or dangling link fell out of the
|
|
350
|
+
input set entirely, so retargeting one was a stale hit.)
|
|
351
|
+
|
|
352
|
+
The composition is seed-chained (`xxh3(part, prevDigest)`) with a
|
|
353
|
+
label prefix per field, so two different field layouts can't collide.
|
|
354
|
+
Bun's xxHash3 reads only the low 32 bits of a seed, so `xxh3` feeds the
|
|
355
|
+
seed forward (`xxHash3(part, seed) ^ seed`) and the chain carries all 64
|
|
356
|
+
bits of state from step to step (item 682; `tests/hash-chain.test.ts`).
|
|
357
|
+
xxHash3 is non-cryptographic by design — cache keys need uniqueness
|
|
358
|
+
across honest inputs, not adversarial collision resistance.
|
|
359
|
+
|
|
360
|
+
## Cache lookup → restore
|
|
361
|
+
|
|
362
|
+
On a hit:
|
|
363
|
+
|
|
364
|
+
1. `cache.get(hash)` is **pure SQL**: the `entries` row carries the
|
|
365
|
+
metadata and the captured stdout; the `output_files` rows carry the
|
|
366
|
+
output fingerprints. The tar artifact is not touched by the probe
|
|
367
|
+
(its existence is verified with one stat), so hit cost doesn't
|
|
368
|
+
scale with artifact size.
|
|
369
|
+
2. If the on-disk output tree already matches the cached snapshot
|
|
370
|
+
(size + mode + **millisecond**-mtime check against `output_files`),
|
|
371
|
+
extraction is skipped entirely — the outcome reports **up-to-date**
|
|
372
|
+
(`restored: false`). Mode and millisecond mtime ride the artifact's
|
|
373
|
+
own `.vx-meta.json` sidecar — recorded from a stat while packing —
|
|
374
|
+
so the index rows and the restored tree carry the identical values
|
|
375
|
+
whether the entry was saved locally or ingested from a remote, and
|
|
376
|
+
the comparison is exact in steady state. The check also requires the
|
|
377
|
+
file's inode and ctime to equal the ones this machine recorded after
|
|
378
|
+
the save or restore that last wrote it (`recordOutputStamps`). Size,
|
|
379
|
+
mode and mtime alone cannot tell two entries apart when their
|
|
380
|
+
outputs carry one fixed mtime (`tar -x`, `cp -p`, `SOURCE_DATE_EPOCH`)
|
|
381
|
+
and one size: a v1 → v2 → v1 round trip reported up-to-date with v2's
|
|
382
|
+
bytes on disk (item 886). No task can set a ctime, and a forged mtime
|
|
383
|
+
(`touch -r`) moves the ctime too. The inode proves less than it
|
|
384
|
+
seems: a restore unlinks the file and renames a new one in, and ext4
|
|
385
|
+
hands the new one the freed inode (measured, item 941), so ctime is
|
|
386
|
+
the guard. A row with no stamp (an ingest from a remote, or a file
|
|
387
|
+
that had changed when the stamp was taken) is never current: the hit
|
|
388
|
+
restores and stamps. The residual: ctime ticks on the kernel's coarse
|
|
389
|
+
clock (4 ms at HZ=250), so a same-size rewrite inside the tick of
|
|
390
|
+
vx's own write, in place or as a new file that took the same inode,
|
|
391
|
+
with its mtime forged back, still matches.
|
|
392
|
+
3. Otherwise the task's declared outputs are wiped from the project
|
|
393
|
+
dir (`cleanOutputs`) — see
|
|
394
|
+
[§ Strict output ownership](#strict-output-ownership) — and the
|
|
395
|
+
`outputs/<rel>` (+ `workspace-outputs/<rel>`) entries are extracted
|
|
396
|
+
from the artifact into place: its bytes inline in the index, read in
|
|
397
|
+
one read transaction with the entry's rows, or `<cacheDir>/<hash>.tar.zst`.
|
|
398
|
+
4. Captured stdout is replayed to the live terminal via the logger —
|
|
399
|
+
the framed block looks like a fresh run. (stderr is not cached —
|
|
400
|
+
only successful runs are cached and their stderr is near-always
|
|
401
|
+
empty; live runs still stream stderr normally.)
|
|
402
|
+
5. The task is marked `cache-hit` (or `cache-hit-remote` when the
|
|
403
|
+
`LayeredCache` hydrated from the remote layer this run);
|
|
404
|
+
`durationMs` is the wallclock for the restore op, not the original
|
|
405
|
+
exec time.
|
|
406
|
+
|
|
407
|
+
The cached `exitCode` is preserved. A cached non-zero exit is
|
|
408
|
+
impossible by construction — see [§ Cache write](#cache-write) — and
|
|
409
|
+
"by construction" is literal: neither `save` nor `ingest` accepts an
|
|
410
|
+
`exitCode` at all, so the stored value is pinned to `0` rather than
|
|
411
|
+
supplied. The restore path still checks it, because the column outlives
|
|
412
|
+
this process: a row from a hand-edited or foreign `cache.db` with a
|
|
413
|
+
non-zero exit classifies the hit `failed` instead of restoring a broken
|
|
414
|
+
build's outputs under a green run.
|
|
415
|
+
|
|
416
|
+
### Local restore tier (two-tier scheduler)
|
|
417
|
+
|
|
418
|
+
`dependsOn` is an ordering gate, so a dependent normally can't even
|
|
419
|
+
probe the cache until its upstream finishes running. But a
|
|
420
|
+
**stable-key** task's key is provably independent of any upstream's
|
|
421
|
+
_outputs_ — its hit/miss status is knowable up front, and restoring
|
|
422
|
+
it needs none of its deps' output. So on local-only runs, `run()`
|
|
423
|
+
performs an up-front CLASSIFY (`orchestrator/local-shortcircuit.ts`):
|
|
424
|
+
|
|
425
|
+
1. Derive every stable, cacheable, local-read task's key (reusing the
|
|
426
|
+
run's `hashCache` memo) and probe local `cache.get` **once**,
|
|
427
|
+
building a `preProbed` map covering stable hits AND stable misses.
|
|
428
|
+
2. Confirmed hits become the **restore tier**: the scheduler makes
|
|
429
|
+
them ready immediately (no dep gate, and no failed-dep→skip check —
|
|
430
|
+
their key is dep-success-independent) but at LOW priority, so cache
|
|
431
|
+
**misses own the worker pool and restores only backfill idle
|
|
432
|
+
capacity**.
|
|
433
|
+
3. `execute-task` consumes `preProbed`, so the up-front probes ARE the
|
|
434
|
+
probes it would have done, hoisted — no double work, and every task
|
|
435
|
+
still flows through `execute()` so output is unchanged.
|
|
436
|
+
|
|
437
|
+
Stability gate (shared with remote prefetch via
|
|
438
|
+
`stable-keys.ts:deriveStableKeys`): a task whose input globs could
|
|
439
|
+
match a same-project upstream's declared `outputs.files` (compared by
|
|
440
|
+
literal prefix, less what its own `!` inputs take back whole; one with
|
|
441
|
+
undeclared writes reaches every input), whose
|
|
442
|
+
own `outputs.files` meet another same-project task's (it would restore
|
|
443
|
+
into that tree beside the producer's restore), or whose
|
|
444
|
+
`inputs.workspaceFiles` could reach any upstream's outputs, has a
|
|
445
|
+
_preliminary_ key and stays exec-tier / dep-gated. **Any**
|
|
446
|
+
`outputs.workspaceFiles` producer upstream also makes a dependent's key
|
|
447
|
+
preliminary, whatever that dependent reads: a root-anchored output is
|
|
448
|
+
boundary-ignoring by design, so it can land inside the dependent's own
|
|
449
|
+
project dir, where an ordinary project-relative glob reads it. Such a
|
|
450
|
+
dependent is therefore in NEITHER tier — it is excluded from probe
|
|
451
|
+
reuse as well as from the restore tier, because `execute-task` reuses a
|
|
452
|
+
`preProbed` hash verbatim, so probe reuse is itself a stale-hit path
|
|
453
|
+
when the key is preliminary. On top of that, an `outputs.workspaceFiles`
|
|
454
|
+
declaration keeps every task whose project directory its static prefix
|
|
455
|
+
can reach out of the restore tier — edge or no edge, because a
|
|
456
|
+
root-anchored output can land in any project's directory — and every
|
|
457
|
+
transitive dependant of one with it (their up-front keys fold a
|
|
458
|
+
preliminary key); a glob with no literal prefix reaches every project.
|
|
459
|
+
Before item 584 one such declaration emptied the tier graph-wide.
|
|
460
|
+
`rules.upfrontKeys` (on by default, X-54) refuses at load a task whose
|
|
461
|
+
input globs can match another task's declared outputs, so the clauses
|
|
462
|
+
on declared outputs fire only with the rule off; undeclared writes, an
|
|
463
|
+
`outputs.workspaceFiles` producer upstream, a rewriter the key does not
|
|
464
|
+
fold, and runtime probes after a writer still leave a key preliminary.
|
|
465
|
+
The
|
|
466
|
+
short-circuit never runs under a `LayeredCache` (remote prefetch owns
|
|
467
|
+
those runs — an up-front `get` there would put remote GETs on the
|
|
468
|
+
critical path), never fires with local reads off, and never throws —
|
|
469
|
+
any error degrades to the normal lazy schedule. Measured: −6.6% on a
|
|
470
|
+
mixed slow-upstream/warm-downstream workload; parity on all-hit warm
|
|
471
|
+
runs.
|
|
472
|
+
|
|
473
|
+
### Remote prefetch (async, remote-only)
|
|
474
|
+
|
|
475
|
+
When a run is backed by a remote cache — a plugin's `cache` capability
|
|
476
|
+
or an injected `RunOptions.remoteCache` layer — the network latency of
|
|
477
|
+
every remote GET
|
|
478
|
+
would otherwise sit on the critical path of the task that needs it.
|
|
479
|
+
So before execution starts, `run()` kicks off **background prefetches**:
|
|
480
|
+
|
|
481
|
+
1. Every cacheable task's key is derived once, up front, in
|
|
482
|
+
topological order (reusing the run's `hashCache` memo, so
|
|
483
|
+
`execute-task`'s later `computeTaskHash` for the same task hits the
|
|
484
|
+
memo — no double hashing). This derivation touches **no cache layer**
|
|
485
|
+
— keys only.
|
|
486
|
+
2. Each **stable-key** task's remote GET is fired concurrently under a
|
|
487
|
+
bounded pool (the run's concurrency). The prefetch ingests a hit
|
|
488
|
+
into the _local_ cache; misses/errors degrade to `false`.
|
|
489
|
+
3. Execution starts immediately — the prefetches race alongside it, so
|
|
490
|
+
remote latency overlaps real work instead of blocking it.
|
|
491
|
+
4. When `execute-task` later calls `cache.get(hash)`, the `LayeredCache`
|
|
492
|
+
**awaits the already-in-flight (resolved-or-pending) prefetch** for
|
|
493
|
+
that key rather than starting a fresh round-trip: **at most ONE
|
|
494
|
+
remote GET per key**, whether it was served by the prefetch, the
|
|
495
|
+
lazy read-through, or both.
|
|
496
|
+
|
|
497
|
+
## A current tree: when a warm hit restores nothing
|
|
498
|
+
|
|
499
|
+
A hit whose outputs are already on disk should cost a few stats, not an
|
|
500
|
+
extraction. Two checks decide, both against rows the cache recorded when
|
|
501
|
+
the tree last matched the entry (`output_files`, and since 2026-09-03
|
|
502
|
+
`output_dirs`):
|
|
503
|
+
|
|
504
|
+
1. **The set.** The files under the declared output globs must be exactly
|
|
505
|
+
the entry's files — no strays, nothing missing — because a restore
|
|
506
|
+
wipes and rewrites the declared outputs, and a stray left in place
|
|
507
|
+
would be a stale file the hit silently kept. Until Wave 6 this was a
|
|
508
|
+
glob walk on every hit (0.36 ms each; 365 ms of CPU on a warm
|
|
509
|
+
1000-project run). Now, for globs of the shape `<dir>/**` or a bare
|
|
510
|
+
literal `<dir>` (a whole subtree — `wholeSubtreePrefixes`; a literal
|
|
511
|
+
that names a file is recorded as a file, `mtime_ms` -2, and holds while
|
|
512
|
+
a regular file stands there, its bytes being the per-file check's),
|
|
513
|
+
the cache records every directory
|
|
514
|
+
under `<dir>` with its mtime after each save and restore; on the next
|
|
515
|
+
hit, unchanged mtimes on all of them prove the set unchanged, since a
|
|
516
|
+
file added or removed anywhere the glob could see bumps its parent
|
|
517
|
+
directory, and a new directory bumps the recorded directory that
|
|
518
|
+
contains it. The rows are taken per task and written together — one
|
|
519
|
+
transaction at the first read, at prune, at stats or at close, since
|
|
520
|
+
item 622; a thousand commits at run end were the whole snapshot stage
|
|
521
|
+
before — and a reader in the same process flushes them first. A
|
|
522
|
+
save's or restore's snapshot is taken at run end, once the directories
|
|
523
|
+
are past the racy window, and by then a later task or another process
|
|
524
|
+
may have written into them: the walk collects the files it passes and
|
|
525
|
+
records nothing unless they are the entry's rows (item 1087). Any other glob shape (`**/*.js`, `dist/*`), a missing
|
|
526
|
+
row set (a remote ingest records none), a moved directory, or more
|
|
527
|
+
than 8,192 directories (`OUTPUT_DIRS_CAP`) keeps the walk — and a walk that proves the tree
|
|
528
|
+
current records the directories so the following hit can skip it.
|
|
529
|
+
2. **The files.** Every recorded file's `(size, mode, mtime-ms)` must
|
|
530
|
+
match. This never left: the directory rule only replaces the
|
|
531
|
+
enumeration, not the fingerprint.
|
|
532
|
+
|
|
533
|
+
The trade is the one every mtime-based skip accepts, already documented
|
|
534
|
+
for files: a deliberately forged directory mtime (`touch -r`) hides a
|
|
535
|
+
stray. Both directions are pinned in `tests/output-dirs.test.ts`.
|
|
536
|
+
|
|
537
|
+
Hard invariants of the remote prefetch:
|
|
538
|
+
|
|
539
|
+
- **Remote-only.** This entire path is gated on a `LayeredCache` being
|
|
540
|
+
configured. A local-only run never prefetches; its up-front keys and
|
|
541
|
+
local probes are the short-circuit's (§ Local restore tier). The
|
|
542
|
+
prefetch never adds an upfront _local_ `get` / `isOutputsCurrent` /
|
|
543
|
+
stat pass.
|
|
544
|
+
- **Stable keys only.** A task with a _preliminary_ key (§ Local
|
|
545
|
+
restore tier's stability gate: undeclared writes, an
|
|
546
|
+
`outputs.workspaceFiles` producer upstream, or, with
|
|
547
|
+
`rules.upfrontKeys: false`, input globs that match an upstream's
|
|
548
|
+
declared outputs) is not prefetched: its key would name the wrong
|
|
549
|
+
artifact, and it resolves via the lazy read-through in `execute-task`. Instability propagates: a task that folds an
|
|
550
|
+
unstable upstream is itself unstable. When in doubt, skip.
|
|
551
|
+
- **At most once.** The `LayeredCache` keeps an in-flight map keyed by
|
|
552
|
+
hash; `prefetch` and `get` share it, and a settled `false` (remote
|
|
553
|
+
miss) prevents a second lazy probe of the same dead key.
|
|
554
|
+
- **Provenance preserved.** A hash pulled from remote — even when a
|
|
555
|
+
later `get` finds it as a now-local hit — still reports
|
|
556
|
+
`source: 'remote'`, so the outcome is `cache-hit-remote`.
|
|
557
|
+
- **A remote-read-off policy** (`--no-cache`, `--force`, or
|
|
558
|
+
`--cache=remote:`) fires no prefetch.
|
|
559
|
+
- **Never fail.** Every remote path (get / put / ingest / prefetch)
|
|
560
|
+
catches all errors and degrades to a cache miss. A remote 500, a
|
|
561
|
+
network drop, a corrupt artifact, or a failed integrity check can
|
|
562
|
+
never fail a run.
|
|
563
|
+
- **Said once, and said whole.** The warning names the operation
|
|
564
|
+
(`probe`, `download`, `upload`), the artifact and the layer's
|
|
565
|
+
`endpoint`, with a URL's credentials, query and fragment dropped:
|
|
566
|
+
`upload <hash> to <endpoint> failed: HTTP 413`. One failure class
|
|
567
|
+
(the error's `code` when it has one, else its message with the hash
|
|
568
|
+
taken out) is said once per run, and the repeats are counted at
|
|
569
|
+
close: `2 more requests failed the same way: <cause>`.
|
|
570
|
+
|
|
571
|
+
### Remote uploads (background, drained at run end)
|
|
572
|
+
|
|
573
|
+
Writes go to the local cache synchronously; the **remote PUT is a
|
|
574
|
+
fire-and-forget background upload**. The task's outcome (and its
|
|
575
|
+
dependents) never wait on upload latency — the uploads race alongside
|
|
576
|
+
the rest of the run and are drained before `cache.close()`, so a
|
|
577
|
+
short run still ships every artifact before the process exits. Upload
|
|
578
|
+
failures log via `onRemoteError` and are otherwise ignored (the task
|
|
579
|
+
already succeeded; the only loss is the remote entry).
|
|
580
|
+
|
|
581
|
+
An upload is the outputs as the task wrote them; secret masking does
|
|
582
|
+
not reach file contents, so a task whose outputs embed a secret is one
|
|
583
|
+
to leave uncached ([security](./security.md)).
|
|
584
|
+
|
|
585
|
+
### Planning probes (`--dry` / `--graph`)
|
|
586
|
+
|
|
587
|
+
The planning paths (`vx run --dry`, `--graph`) predict hits without
|
|
588
|
+
side effects: against a remote cache they use a **lightweight
|
|
589
|
+
existence probe** — no artifact download, no local ingest. A predicted
|
|
590
|
+
`hit-remote` means the artifact exists remotely; the bytes move only
|
|
591
|
+
when a real run needs them. Locally the probe reads whether the entry's
|
|
592
|
+
row is there and stats the artifact, never the row itself: the whole
|
|
593
|
+
row, its stored stdout included, made a 200-task plan over 1 MB outputs
|
|
594
|
+
230 ms against 28 (min of 11, 2026-10-02).
|
|
595
|
+
|
|
596
|
+
## Cache policy (read/write axes)
|
|
597
|
+
|
|
598
|
+
Caching is controlled by a four-axis `CachePolicy` — **localRead**,
|
|
599
|
+
**localWrite**, **remoteRead**, **remoteWrite** — independent toggles,
|
|
600
|
+
each enforced inside the matching cache layer at construction time. The
|
|
601
|
+
local `Cache` gets a `{ read, write }` slice gating its task artifact
|
|
602
|
+
get/save, the config-evaluation cache's reads and writes, and the
|
|
603
|
+
file-hash memo's writes (never `recordRun` / `stats` / `prune` / ingest
|
|
604
|
+
/ key derivation); the `LayeredCache` additionally gates its own remote
|
|
605
|
+
read-through (`remoteRead`), upload (`remoteWrite`), and prefetch
|
|
606
|
+
(`remoteRead`). The orchestrator derives two booleans per task:
|
|
607
|
+
|
|
608
|
+
- `willRead = task has a cache block AND it is not remote-only AND (localRead || remoteRead)`
|
|
609
|
+
- `willWrite = task has a cache block AND it is not remote-only AND (localWrite || remoteWrite)`
|
|
610
|
+
|
|
611
|
+
A task reads the cache only when `willRead`, saves only when
|
|
612
|
+
`willWrite`, and cleans its declared outputs before exec only when
|
|
613
|
+
`willWrite`. Remote-only is an `exec.remote: 'only'` task placed on a
|
|
614
|
+
remote executor: it never touches this machine's disk (no probe, no
|
|
615
|
+
restore, no output clean, no local save), and its result lives in the
|
|
616
|
+
remote executor's own store.
|
|
617
|
+
|
|
618
|
+
The CLI maps three flags to a policy (precedence: start all-on → apply
|
|
619
|
+
`--cache` → `--no-cache` forces all off → `--force` forces both reads
|
|
620
|
+
off): `--no-cache` = everything off (no read, no write, no output
|
|
621
|
+
clean); `--force` = reads off / writes on (re-execute and refresh the
|
|
622
|
+
cache, outputs cleaned); `--cache=<spec>` = explicit per-layer control.
|
|
623
|
+
See `docs/cli.md` § Cache control.
|
|
624
|
+
|
|
625
|
+
One subtlety: when `localWrite` is off but `remoteWrite` is on
|
|
626
|
+
(`--cache=local:,remote:rw`), there's no on-disk artifact for the
|
|
627
|
+
`LayeredCache` to read before uploading — so it packs the tar.zst bytes
|
|
628
|
+
in memory (`Cache.packArtifactBytes`) and uploads those.
|
|
629
|
+
|
|
630
|
+
## Cache write
|
|
631
|
+
|
|
632
|
+
A miss runs the task. If the final exit code is `0` and the task's
|
|
633
|
+
`willWrite` is true (it has a `cache` block AND at least one write axis
|
|
634
|
+
is on):
|
|
635
|
+
|
|
636
|
+
1. `cache.outputs.files` (and `outputs.workspaceFiles`) are resolved
|
|
637
|
+
against the project dir / workspace root.
|
|
638
|
+
2. The per-component input fingerprint is the one captured before the
|
|
639
|
+
command ran: on a miss `describeTaskInputs` fills `captureInto` once
|
|
640
|
+
(the same call yields the key re-checked below), so no second
|
|
641
|
+
`computeTaskHash` runs.
|
|
642
|
+
3. The artifact — one `stdout` entry (bounded: the first and last 8 Mi
|
|
643
|
+
characters of the task's output, the dropped middle named where it was), the `outputs/<rel>` (+
|
|
644
|
+
`workspace-outputs/<rel>`) entries, the `.vx-meta.json` sidecar and
|
|
645
|
+
the `.vx-sum` checksum — is packed in-process (no staging dir, no subprocess) into a
|
|
646
|
+
single `<hash>.tar.zst` and validated. One of at most 32 KiB
|
|
647
|
+
compressed (`INLINE_MAX`, almost every artifact: a one-file `dist/`
|
|
648
|
+
is a few hundred bytes) stays in memory; a larger one is written to a
|
|
649
|
+
temp name.
|
|
650
|
+
4. One `BEGIN IMMEDIATE` SQLite transaction upserts a small artifact's
|
|
651
|
+
bytes into `artifacts` (v33), or renames a large one into place and
|
|
652
|
+
deletes the key's inline bytes, and upserts the `entries` row (taskId, command, exit code,
|
|
653
|
+
duration, size, stdout, timestamps), the `output_files` fingerprint
|
|
654
|
+
rows, and the entry's one `entry_inputs` row (`INSERT OR IGNORE`).
|
|
655
|
+
Concurrent readers see either no entry or a complete entry — never a
|
|
656
|
+
partial one — and bytes and rows go live together: the rename waits
|
|
657
|
+
for the write lock, so no other writer's rows land beside it, and a
|
|
658
|
+
commit that fails takes the artifact back out (the key then misses).
|
|
659
|
+
Until 2026-09-27 (A-3) the rename came first, and a commit refused
|
|
660
|
+
past the busy timeout left the new bytes beside the old rows: every
|
|
661
|
+
later hit on the key failed the task as a corrupt artifact.
|
|
662
|
+
A re-save (`--force`) first moves the previous artifact aside under
|
|
663
|
+
a temp name and unlinks it after the commit: ext4 flushes the
|
|
664
|
+
incoming file when a rename replaces one, 0.55 ms a save on the main
|
|
665
|
+
thread against 0.04 (a forced 1,000-task run 4.01 s → 3.46 s,
|
|
666
|
+
2026-10-02). A reader probing between the two renames misses.
|
|
667
|
+
An inline save moves aside whatever file its key has, the same way
|
|
668
|
+
(a rename that finds nothing on a first save, in place of a statement
|
|
669
|
+
asking the index). As of every commit a key's bytes are in one place,
|
|
670
|
+
and a reader takes the inline bytes first, so a file an older vx
|
|
671
|
+
wrote beside them is never read and the sweep takes it. An inline save writes no file at all: no temp, no rename,
|
|
672
|
+
and a crash leaves nothing behind (the bytes and the rows are one
|
|
673
|
+
transaction in one database file).
|
|
674
|
+
|
|
675
|
+
**The key is re-checked before the save** (item 743). It was taken
|
|
676
|
+
before the command ran — at the task's start, or up front by the local
|
|
677
|
+
short-circuit — and the save files the outputs under it, so the inputs
|
|
678
|
+
must still be what it describes. Two checks, in order. The key
|
|
679
|
+
re-derived just before the command (`describeTaskInputs`, which the
|
|
680
|
+
executor seam needs anyway) must equal it. Then each input file and the
|
|
681
|
+
project's `package.json` is `lstat`ed once: a file whose ctime falls
|
|
682
|
+
after, or within `FILE_HASH_RACY_MS` before, the moment its digest was
|
|
683
|
+
learned — the enumeration's start for an index OID, the describe for a
|
|
684
|
+
hashed file — is hashed again and compared, and a file that is gone has
|
|
685
|
+
moved. A file written at or after the describe (just before the command)
|
|
686
|
+
has moved whatever it holds (a whole-second stamp counts as any moment
|
|
687
|
+
of its second, an even one of two, in both checks): an input changed and changed BACK while the
|
|
688
|
+
command ran matched its digest again, and its output, built from the
|
|
689
|
+
edit, was filed under the key and restored over the original (item
|
|
690
|
+
1015). When one moved, the task's result stands but no entry is saved,
|
|
691
|
+
one status line names the file (``[vx] app#format: `packages/app/a.ts`
|
|
692
|
+
changed after its key was taken — …``), and the run forgets what it
|
|
693
|
+
knew about the project, as after an uncached task (§ Cache key
|
|
694
|
+
derivation, step 12). This covers a formatter rewriting its own input
|
|
695
|
+
(turborepo#10111) and a user's edit mid-run (turborepo#1146). A task
|
|
696
|
+
left remote (`--download=none`) is held to the same checks: a moved
|
|
697
|
+
key still lets a local consumer fetch its outputs, but the fetch saves
|
|
698
|
+
no entry; until X-123 it saved them under the old key. A task
|
|
699
|
+
that rewrites its own input to the SAME bytes (`sed -i` always writes)
|
|
700
|
+
is not saved either: its write cannot be told from an edit reverted
|
|
701
|
+
mid-run, so it pays a re-run each time rather than risk a stale entry;
|
|
702
|
+
declare what it writes as an output, which the status line says when it
|
|
703
|
+
names a file other than `package.json` (TanStack Router's committed
|
|
704
|
+
`routeTree.gen.ts`, rewritten by every build). A file ADDED under an
|
|
705
|
+
input glob since the listing the key filtered (`addedInput`) withholds
|
|
706
|
+
the save the same way, named on the same line: the listing's
|
|
707
|
+
directories a glob reaches are `lstat`ed, one whose ctime moved since
|
|
708
|
+
the listing is read, and a new name the declaration matches (or an
|
|
709
|
+
unlisted directory) is asked of git, which alone knows what is ignored;
|
|
710
|
+
a literal input absent then and present now is asked too. Not seen: a
|
|
711
|
+
file added inside a directory that held no listed file before (only
|
|
712
|
+
ignored ones, or none) and was not itself created mid-run. Cost: one
|
|
713
|
+
`lstat` per input and per reached directory on a miss that saves, a
|
|
714
|
+
`readdir` per directory that changed, and a `git ls-files` only when a
|
|
715
|
+
candidate turned up; a hit runs no command and checks nothing.
|
|
716
|
+
A file listed but gone before the key hashes it (an upstream that
|
|
717
|
+
deletes a file its dependant's globs match) is keyed as absent, not
|
|
718
|
+
read: the dependant failed as `internal error … ENOENT` on every run
|
|
719
|
+
until A-55. Absent stays unmoved while it stays gone.
|
|
720
|
+
Its dependants whose keys fold its key save nothing either,
|
|
721
|
+
transitively: that key does not name the bytes they built from (`[vx] app#use: ran
|
|
722
|
+
over app#gen's outputs, which its key no longer describes — …`). Until
|
|
723
|
+
2026-09-27 (A-12) they saved, and once the input was put back they hit
|
|
724
|
+
the edit's output.
|
|
725
|
+
A workspace fingerprint a task rewrote since the run read it (§ Cache
|
|
726
|
+
key derivation, step 3) withholds the save the same way, and no key
|
|
727
|
+
taken before it is probed or restored: a hit the up-front probe found
|
|
728
|
+
goes back to the scheduler and runs once its deps are done. Until X-124
|
|
729
|
+
that hit restored the old install's bytes.
|
|
730
|
+
|
|
731
|
+
A miss that ran here and saves **nothing** — it failed, the cache
|
|
732
|
+
policy writes nothing (`--cache=local:r,remote:r`), an upstream failed
|
|
733
|
+
under `--continue` — runs the same input re-check and, when an input
|
|
734
|
+
moved, forgets the project as above; otherwise its declared outputs are
|
|
735
|
+
resolved and marked in the git snapshot exactly as a save marks them.
|
|
736
|
+
Before item 750 only a save marked them, so a same-run reader keyed
|
|
737
|
+
from the snapshot's index OID for an output (or for an input the task
|
|
738
|
+
rewrote) restored the bytes from before the command.
|
|
739
|
+
|
|
740
|
+
A declared set that resolves to **nothing** is said on the run's
|
|
741
|
+
status line. `cache.inputs matched no files (lib/**)` — the key would
|
|
742
|
+
not change when the source does — is said on every miss of a cacheable
|
|
743
|
+
task, right after `describeTaskInputs` and before the command, whether
|
|
744
|
+
or not it saves. `cache.outputs matched no files (build/**)` — an empty
|
|
745
|
+
artifact was saved and a later hit restores nothing — is said on the
|
|
746
|
+
save path. Both are almost always a glob
|
|
747
|
+
against the wrong directory; the output line names one other cause when
|
|
748
|
+
it applies: a sandboxed task with no `exec.sandbox.allow.write`, whose
|
|
749
|
+
writes never reached disk, or a `workspaceFiles` directory that is a
|
|
750
|
+
symlink out of the workspace, whose files vx drops as outside (a project
|
|
751
|
+
output directory linked out of the project refuses the task instead, X-88). `outputs.files: []` is a deliberate cached
|
|
752
|
+
no-op and says nothing; a task with no `cache` block is never checked.
|
|
753
|
+
|
|
754
|
+
**The outputs are what exists when the task's command exits.** The run
|
|
755
|
+
waits for the command's own process, not for everything it started — a
|
|
756
|
+
backgrounded grandchild that still holds the output pipe does not hold
|
|
757
|
+
the run (`tests/runner.test.ts`) — and the save resolves the output
|
|
758
|
+
globs at once. A descendant the command detached (`setsid … &`, a
|
|
759
|
+
daemon) that writes after that is not in the artifact: the entry saves
|
|
760
|
+
what was there, often nothing (with the `cache.outputs matched no
|
|
761
|
+
files` line), and the next hit's clean-before-restore removes what the
|
|
762
|
+
descendant wrote and restores the saved entry in its place (upstream
|
|
763
|
+
survey, turborepo#12786). vx does not wait for it and cannot see it: a
|
|
764
|
+
process that left the task's session is outside the process group vx
|
|
765
|
+
tracks and signals ([kill-tree](./modules/kill-tree.md)). So a command
|
|
766
|
+
finishes writing its outputs before it exits — `wait` for what it
|
|
767
|
+
backgrounds, or do not detach it.
|
|
768
|
+
|
|
769
|
+
If the task exits non-zero, **nothing is cached.** This is deliberate:
|
|
770
|
+
|
|
771
|
+
- Caching a failure prevents retry flows. The next run gets the same
|
|
772
|
+
failure even after the user fixes the underlying cause (the inputs
|
|
773
|
+
haven't changed, so the cache key matches).
|
|
774
|
+
- Failures should be transient by default — flaky tests, network
|
|
775
|
+
blips, transient resource exhaustion shouldn't bake into the cache.
|
|
776
|
+
|
|
777
|
+
Failed-task stdout / stderr still reach the user via the live stream
|
|
778
|
+
and the framed failure block replayed at run end. The `runs` table
|
|
779
|
+
records the failure (status + exit code) for analytics.
|
|
780
|
+
|
|
781
|
+
## Strict output ownership
|
|
782
|
+
|
|
783
|
+
Declared `cache.outputs.files` are wiped in two distinct places:
|
|
784
|
+
|
|
785
|
+
- **Before exec on a cache miss.** A leftover `dist/old.js` from a
|
|
786
|
+
prior build can't survive into a fresh build that doesn't rewrite
|
|
787
|
+
it.
|
|
788
|
+
- **Before restore on a cache hit.** The post-restore tree is the
|
|
789
|
+
cached snapshot byte-for-byte. Hand-edits to output files don't
|
|
790
|
+
persist through a cache replay. When the hit's own check walked the
|
|
791
|
+
output globs to find the tree stale, the clean deletes from that walk's
|
|
792
|
+
list instead of walking again: nothing is awaited between them (X-161).
|
|
793
|
+
|
|
794
|
+
Both branches use the same `cleanOutputs` helper (`src/cache/inputs.ts`)
|
|
795
|
+
with the same boundary rules, and two directories stay off the wipe
|
|
796
|
+
whatever the glob: `.git` and `.vx` (`OUTPUT_NEVER`). `node_modules` is
|
|
797
|
+
off it too unless a glob names it (`node_modules/**`, an install task's
|
|
798
|
+
legitimate output); until 2026-09-27 (A-13) `**/*.js` cleaned every
|
|
799
|
+
installed `.js`, and a `workspaceFiles` output glob reached even `.git`.
|
|
800
|
+
A task that fails while a git-tracked file another task's clean removed
|
|
801
|
+
is still missing gets one line naming the file and that task: vx cleans
|
|
802
|
+
before a run where Turbo does not, so a reader with no edge to the
|
|
803
|
+
producer of a committed output fails naming only the file (A-48,
|
|
804
|
+
vueuse's `metadata/index.json`).
|
|
805
|
+
Workspace outputs take the same rules, and a path an output `!` entry
|
|
806
|
+
takes back (A-44) is off the wipe, the artifact and the restore alike.
|
|
807
|
+
Skipped when:
|
|
808
|
+
|
|
809
|
+
- `cache.outputs.files` is empty (nothing declared as output).
|
|
810
|
+
- The task's `willWrite` is false — no write axis is enabled (e.g.
|
|
811
|
+
`--no-cache`, or a read-only `--cache=local:r`). The user is debugging
|
|
812
|
+
and managing the tree, so vx leaves it alone. `--force` keeps writes
|
|
813
|
+
on, so it DOES clean (the saved snapshot must be clean).
|
|
814
|
+
|
|
815
|
+
A declared output the process cannot remove (a `dist/` another user
|
|
816
|
+
wrote, a read-only checkout) fails the task with `cannot remove declared
|
|
817
|
+
output <path>: EACCES — …`, and a restore that cannot write into such a
|
|
818
|
+
directory with `restore of <hash> into <dir> could not write its outputs
|
|
819
|
+
(EACCES: …)`: the environment's failure, reported plainly, never as an
|
|
820
|
+
internal error or a corrupt artifact.
|
|
821
|
+
|
|
822
|
+
Why so strict? Turbo and Nx restore additively — files from a prior
|
|
823
|
+
state can survive. We've seen this cause:
|
|
824
|
+
|
|
825
|
+
- Wrong test runs (a deleted-but-resurrected snapshot file from a
|
|
826
|
+
cache miss survives a hit and now your test passes against the
|
|
827
|
+
wrong baseline).
|
|
828
|
+
- Wrong shipped artifacts (a deleted source-mapped file from a prior
|
|
829
|
+
build sits in `dist/` alongside the new bundle).
|
|
830
|
+
|
|
831
|
+
The strict-ownership behavior makes the project dir post-run a pure
|
|
832
|
+
function of the cache key.
|
|
833
|
+
|
|
834
|
+
### Additive outputs: two tasks, one tree, an edge between them
|
|
835
|
+
|
|
836
|
+
Two cached tasks whose declared outputs overlap are refused at graph
|
|
837
|
+
build. By default that holds even when one depends on the other: the
|
|
838
|
+
workspace rule `rules.exclusiveOutputs` (on unless set to `false` in
|
|
839
|
+
`vx.workspace.ts`, X-53) keeps one path to one task, and its refusal says
|
|
840
|
+
how to turn it off. The shape below is correct, only slower: a stamp
|
|
841
|
+
before every run, a diff after, and a clean by rows. Give each task its
|
|
842
|
+
own output path where you can (`dist-individual` beside `dist`).
|
|
843
|
+
|
|
844
|
+
With the rule off, an edge fixes the order, and the dependant is
|
|
845
|
+
**additive** (item 588): twenty's
|
|
846
|
+
`build` fills `dist` and `build:individual` depends on it and writes
|
|
847
|
+
`dist/individual`; strapi's `build:types` runs `tsc` into the same
|
|
848
|
+
`dist` as `build`. For the dependant:
|
|
849
|
+
|
|
850
|
+
- its **own output set** is what its run added or changed under its
|
|
851
|
+
declared outputs — the outputs are stamped (size, mtime, inode, ctime) before the
|
|
852
|
+
run and diffed after, the same proof a hit's "already current" check
|
|
853
|
+
trusts — and only that set is saved; `outputs.workspaceFiles` are
|
|
854
|
+
stamped the same way (until A-43 its miss cleaned them by glob, deleting
|
|
855
|
+
a same-tree upstream's root-anchored files before it read them);
|
|
856
|
+
- a run that **removes** a file it found (a bundler deleting the
|
|
857
|
+
upstream's intermediate) saves nothing: an artifact holds what a run
|
|
858
|
+
wrote, never what it took away, and the upstream's restore puts the
|
|
859
|
+
file back, so the dependant runs again on every warm run (X-32);
|
|
860
|
+
- it **cleans by recorded rows**, never by glob, before a run (nothing:
|
|
861
|
+
stale files of its own are its command's to clean, as under Turbo) and
|
|
862
|
+
before a restore (its rows only, pruning emptied directories only
|
|
863
|
+
inside its declared trees, so a sibling's fresh directory stays);
|
|
864
|
+
- its "already current" check requires its rows present and current and
|
|
865
|
+
ignores everything else under the glob;
|
|
866
|
+
- it is **never restore-tier**: it restores or runs after its upstream,
|
|
867
|
+
by the edge.
|
|
868
|
+
|
|
869
|
+
The upstream keeps strict ownership of its glob — a miss or a restore
|
|
870
|
+
wipes the whole tree, the dependant's additions included, and the
|
|
871
|
+
dependant restores or runs after — and its "already current" check
|
|
872
|
+
ignores strays a dependant's glob could have added, so a warm run stays
|
|
873
|
+
a no-op for both. A file the dependant rewrites in place (refine's
|
|
874
|
+
`types` regenerating `build`'s `.d.ts`) counts as the dependant's own
|
|
875
|
+
(its ctime moved, even where a tool stamps the size and mtime back; X-33),
|
|
876
|
+
which is correct and costs the upstream a restore on
|
|
877
|
+
the next warm run; the design note keeps that shape out of scope.
|
|
878
|
+
|
|
879
|
+
## Invalidation paths
|
|
880
|
+
|
|
881
|
+
A task's cache becomes invalid when any of these change:
|
|
882
|
+
|
|
883
|
+
| Trigger | Mechanism |
|
|
884
|
+
| ---------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
885
|
+
| Edit a file in the task's `inputs.files` set | step 12 of key derivation |
|
|
886
|
+
| Edit a file in the task's `inputs.workspaceFiles` set (root-anchored; may live in ANY project's dir — the documented boundary exception) | step 12 — resolved workspace files join the same input-file list |
|
|
887
|
+
| Any package manager updates a lockfile (`pnpm`, `npm`, `yarn`, `bun`) | step 3 (workspace fingerprint) — or, for a lockfile a plugin claims, that plugin's `key` material (`@vzn/vx-lockfile`: only the projects whose dependency closure moved) |
|
|
888
|
+
| Edit `pnpm-workspace.yaml` | step 3 |
|
|
889
|
+
| Edit `package.json`'s `workspaces` field | not step 3 — membership: a project that joins or leaves changes which projects exist and which nested dirs its parent's globs exclude (step 12); the file itself is hashed per project (step 4) |
|
|
890
|
+
| Edit the project's `package.json` (dep / version / scripts change) | step 4 (project package.json hash) |
|
|
891
|
+
| Edit the task's `vx.config.ts` | step 5 (task config hash) |
|
|
892
|
+
| Edit a config file that the task config imports | step 5 (configHash sees the resolved object after Bun evaluates imports) |
|
|
893
|
+
| Change CLI `forwardArgs` (after `--`) | step 6 |
|
|
894
|
+
| Change a `cache.inputs.env` host value | step 7 |
|
|
895
|
+
| Change the combined stdout+stderr of a `cache.inputs.runtime` command (resolved at hash time) | step 8 |
|
|
896
|
+
| Change the combined stdout+stderr of a `cache.inputs.workspaceRuntime` command (resolved at hash time) | step 9 |
|
|
897
|
+
| Upstream task's cache key changes (because its inputs changed) | step 10 |
|
|
898
|
+
| Bump `CACHE_VERSION` | step 1 — orphans every entry |
|
|
899
|
+
| Change `exec.env.passThrough` _values_ alone | **NOT a trigger** by design — passThrough values are host-specific |
|
|
900
|
+
| Change a file not in `inputs.files` / `inputs.workspaceFiles` | **NOT a trigger** by design — declare it explicitly |
|
|
901
|
+
| Edit `vx-lock.json` | **NOT a trigger** — globally excluded from inputs (v24); it's vx's own metadata, never a task input |
|
|
902
|
+
| Change a file in a nested project's dir | **NOT a trigger** for the parent's `files` globs — project boundaries are hard (workspaceFiles is the explicit exception) |
|
|
903
|
+
|
|
904
|
+
The cascade in step 10 is what makes monorepo caching work: edit a
|
|
905
|
+
file in `lib/`, and every package that depends on `lib`'s `build` task
|
|
906
|
+
re-runs automatically.
|
|
907
|
+
|
|
908
|
+
## Cross-project boundaries
|
|
909
|
+
|
|
910
|
+
A project's `cache.inputs.files` globs **never** reach into another
|
|
911
|
+
project's directory, even if a `**/*` pattern would otherwise match.
|
|
912
|
+
|
|
913
|
+
`workspace/nested-dirs.ts` computes the set of nested project
|
|
914
|
+
directories (projects rooted inside this one) once per `vx run`. A
|
|
915
|
+
workspace package counts whether or not it has a vx config: it is the
|
|
916
|
+
project `--affected` gives its files to, and its default `build` keys
|
|
917
|
+
them (X-57). The same set fences the output clean, so a root
|
|
918
|
+
`outputs.files: ['**/*.js']` never removes a member's source. Every
|
|
919
|
+
glob pass drops a path under one of them (an ancestor lookup,
|
|
920
|
+
A-11). The only way
|
|
921
|
+
for project A to depend on project B's state via project-relative
|
|
922
|
+
globs is `dependsOn` + upstream-hash propagation (step 10).
|
|
923
|
+
|
|
924
|
+
For a task with `exec.sandbox` and `cache`, the sandbox enforces that
|
|
925
|
+
"only". A workspace package reached through a `node_modules` link is
|
|
926
|
+
granted only when the task's key folds a task of that package, walking
|
|
927
|
+
the same selection `cache.inputs.tasks` applies at hash time
|
|
928
|
+
(`orchestrator/keyed-projects.ts`, over `upstream.ts`'s
|
|
929
|
+
`selectFoldedDeps`); every other link target inside the root is
|
|
930
|
+
withheld, so an import of a sibling the key never sees fails and is
|
|
931
|
+
reported instead of caching a result an edit there would not re-run.
|
|
932
|
+
Any exec task counts, cached or not: a task with no `cache` declares no
|
|
933
|
+
`inputs.files`, so its key folds every file of its project. A persistent
|
|
934
|
+
task counts the same way: its key folds its whole project, so a cached
|
|
935
|
+
e2e behind a dev server re-runs when the server's sources change.
|
|
936
|
+
The coverage is per package, not per file: an edge to `ui#source`
|
|
937
|
+
(inputs `src/**`) also admits a read of `ui/README.md`, whose edit moves
|
|
938
|
+
no key — the same limit a grant wider than a task's own inputs already
|
|
939
|
+
has. A task with no `cache` keeps every link, having no key to be stale.
|
|
940
|
+
|
|
941
|
+
**Exception:** `cache.inputs.workspaceFiles` /
|
|
942
|
+
`cache.outputs.workspaceFiles` are workspace-root-anchored and apply
|
|
943
|
+
NO boundary rule — a deliberate escape hatch (owner call: "they don't
|
|
944
|
+
care about boundaries; it is bad practice but is there"). Prefer
|
|
945
|
+
project-relative declarations; reach for workspaceFiles only for
|
|
946
|
+
genuinely root-anchored files.
|
|
947
|
+
|
|
948
|
+
## Clean filters (`text` / `eol` / `ident`, `core.autocrlf`)
|
|
949
|
+
|
|
950
|
+
Git can store a file's blob in a different form than the bytes in your
|
|
951
|
+
working tree. Under a `text`, `eol` or `ident` attribute — or with
|
|
952
|
+
`core.autocrlf` set to `true`/`input` — the index holds the
|
|
953
|
+
LF-normalized (or ident-collapsed) blob while your editor and your
|
|
954
|
+
build see the CRLF (or expanded) file.
|
|
955
|
+
|
|
956
|
+
That matters because `git status` compares **after** applying the
|
|
957
|
+
filter, so such a file reports _clean_: git considers it unmodified
|
|
958
|
+
even though the blob and the worktree file are different byte
|
|
959
|
+
sequences. Trusting the index OID as the file's content hash would
|
|
960
|
+
then let two genuinely different worktree contents fold the **same**
|
|
961
|
+
cache key, and a real change would be invisible.
|
|
962
|
+
|
|
963
|
+
So vx does not trust an index OID where a filter can apply. The check
|
|
964
|
+
is gated in three steps, and the common case pays nothing:
|
|
965
|
+
|
|
966
|
+
1. `core.autocrlf` is `true`/`input` — conversion applies to every
|
|
967
|
+
auto-detected text file with no attribute needed, so no OID is
|
|
968
|
+
trusted.
|
|
969
|
+
2. Otherwise, if no attributes source exists anywhere (no in-tree
|
|
970
|
+
`.gitattributes`, an ignored one included, which `git status
|
|
971
|
+
--ignored=matching` names from the walk it already does, no `$GIT_DIR/info/attributes`, and none of the
|
|
972
|
+
files git reads outside the tree: the global one, which is
|
|
973
|
+
`core.attributesFile` or by default `$XDG_CONFIG_HOME/git/attributes`
|
|
974
|
+
or `~/.config/git/attributes`, and the system one), no rule can name
|
|
975
|
+
a filter and vx does **no** extra work. This is the default
|
|
976
|
+
`git init` repo. The gate reads `git var -l`, one spawn, which from
|
|
977
|
+
git 2.42 names the global and system files itself
|
|
978
|
+
(`GIT_ATTR_GLOBAL`, `GIT_ATTR_SYSTEM`); for an older git the global
|
|
979
|
+
lookup is mirrored and the system file is looked for at
|
|
980
|
+
`/etc/gitattributes` and `<prefix>/etc/gitattributes` beside the
|
|
981
|
+
binary, where a standard build puts it. Before 2026-09-24 the gate
|
|
982
|
+
missed the default global file, so `* text` there left a CRLF file
|
|
983
|
+
keyed on its LF blob: a CRLF→LF edit was a stale hit.
|
|
984
|
+
3. Otherwise `git check-attr` resolves `text`, `eol`, `ident`, `filter`
|
|
985
|
+
and `working-tree-encoding` from the index — without reading worktree
|
|
986
|
+
content — and only the paths actually carrying one lose their OID. A
|
|
987
|
+
`filter` driver was missing from the list until item 978: a clean
|
|
988
|
+
driver that drops comment lines (`sed '/^#/d'`, nbstripout's shape)
|
|
989
|
+
stores one blob for two worktree files, so an edit git calls clean
|
|
990
|
+
replayed the old output. Git LFS files are a `filter` too, and are
|
|
991
|
+
hashed from disk.
|
|
992
|
+
|
|
993
|
+
`-text` (explicitly unset) and unspecified paths keep their OIDs:
|
|
994
|
+
both leave the blob byte-identical to the worktree file.
|
|
995
|
+
|
|
996
|
+
Losing an OID is not over-invalidation. It routes that path to the
|
|
997
|
+
content hasher, which hashes the worktree bytes — the correct source
|
|
998
|
+
either way. The only cost is reading the file, which is exactly what
|
|
999
|
+
the gate exists to avoid paying needlessly.
|
|
1000
|
+
|
|
1001
|
+
## Runtime inputs and the lock (the env parallel)
|
|
1002
|
+
|
|
1003
|
+
`cache.inputs.runtime` / `cache.inputs.workspaceRuntime` are modeled
|
|
1004
|
+
exactly on `cache.inputs.env`, and they share its asymmetry with the
|
|
1005
|
+
lockfile:
|
|
1006
|
+
|
|
1007
|
+
- **The command _strings_ live in the resolved config**, so `vx lock`
|
|
1008
|
+
freezes them into `vx-lock.json` (just as it freezes the env _names_
|
|
1009
|
+
a task declares).
|
|
1010
|
+
- **The command _output_ is resolved live at hash time on every run**
|
|
1011
|
+
— inside `resolveInputs`, the same place env _values_ are read from
|
|
1012
|
+
the host `process.env`. The lock never stores it.
|
|
1013
|
+
|
|
1014
|
+
A runtime command executes **once per run, per project** (memoized by
|
|
1015
|
+
`projectDir + command`), at key-derivation time — which, with the
|
|
1016
|
+
up-front classify/prefetch pass, is before any task runs. It is a
|
|
1017
|
+
run-level reading of the ENVIRONMENT (toolchain versions, resolved
|
|
1018
|
+
config), not a per-task probe: it is not asked again before a save, as
|
|
1019
|
+
input files are, because that would be one spawn per command per miss
|
|
1020
|
+
where a file costs one `lstat`. A command that reads another task's
|
|
1021
|
+
OUTPUT reads whatever is on disk when it is asked. When the reader's key
|
|
1022
|
+
folds that task's key, the upstream key covers it: an answer taken before
|
|
1023
|
+
the upstream ran costs a spurious miss on the run after the output
|
|
1024
|
+
changes, never a stale hit, and the answer stays shared. When it does
|
|
1025
|
+
not (`tasks: []`, a filter that leaves it out, transitively), nothing
|
|
1026
|
+
covered it until X-34: the save filed bytes built from the new state
|
|
1027
|
+
under the old answer, and a later run that started from the old state
|
|
1028
|
+
hit them. Now such a task (`probesAfterWrites`, stable-keys.ts) takes no
|
|
1029
|
+
key up front and is never restore-tier, and its probes run for it alone,
|
|
1030
|
+
after its upstream finished — one spawn per such task per run, not one
|
|
1031
|
+
per project (`tests/in-run-writes.test.ts`). Declaring the producing
|
|
1032
|
+
task's output as an input (`dependsOn` + files) is still the cheaper
|
|
1033
|
+
spelling: files are re-checked, answers are not.
|
|
1034
|
+
|
|
1035
|
+
A probe runs in its own process group. One still running when vx exits
|
|
1036
|
+
— a Ctrl-C, or a refusal, while a hung `git` or `node -e …` answers — is
|
|
1037
|
+
SIGKILLed with its whole tree; until 2026-09-27 (A-9) it ran on under
|
|
1038
|
+
init. A `kill -9` of vx runs no exit hook, and leaves it.
|
|
1039
|
+
|
|
1040
|
+
So `vx run --frozen` loads the frozen command strings but still spawns
|
|
1041
|
+
them and folds their current output into the key. A `node -v` that goes
|
|
1042
|
+
from `v20` to `v22` after the lock was written busts the cache under
|
|
1043
|
+
`--frozen`, exactly as a changed `NODE_ENV` value would — the TypeScript
|
|
1044
|
+
escape hatch (`define: { TSC_VERSION: execSync(...) }`) goes stale here
|
|
1045
|
+
because its value was baked into the config object at lock time, whereas
|
|
1046
|
+
a runtime input re-resolves.
|
|
1047
|
+
|
|
1048
|
+
Consequently **`vx lock --check` does not — and need not — flag
|
|
1049
|
+
runtime-output drift.** `lock --check` audits that the frozen config
|
|
1050
|
+
object still matches a fresh evaluation; the command output was never
|
|
1051
|
+
part of that object (only the strings are), so a changed probe output
|
|
1052
|
+
is correct, expected, live behavior rather than lock drift. This is the
|
|
1053
|
+
same reason `lock --check` ignores `inputs.env` value changes.
|
|
1054
|
+
|
|
1055
|
+
## Concurrent runs
|
|
1056
|
+
|
|
1057
|
+
Two vx processes on one workspace (a `vx watch` beside a `vx run`, two
|
|
1058
|
+
CI jobs on one checkout) take turns: a run takes the workspace's run
|
|
1059
|
+
lock before it schedules and releases it with its cache handle — before
|
|
1060
|
+
a persistent task's wait, so a dev server never holds it — and the
|
|
1061
|
+
second run waits, saying after a second whom it waits for:
|
|
1062
|
+
|
|
1063
|
+
```
|
|
1064
|
+
[vx] waiting for another vx run (pid 4821) on this workspace to finish…
|
|
1065
|
+
```
|
|
1066
|
+
|
|
1067
|
+
The cache itself was always safe (SQLite waits on its lock, artifacts
|
|
1068
|
+
land by rename; a transaction that reads before it writes takes the
|
|
1069
|
+
lock at BEGIN, since SQLite answers its later write `database is locked`
|
|
1070
|
+
at once: a run's history was lost so in 13 of 48 runs on one shared
|
|
1071
|
+
cache dir, X-105); a task's OUTPUT TREE was not — both runs cleaned and
|
|
1072
|
+
restored the same `dist/`, and a clean landing while the other run's
|
|
1073
|
+
restore was staging took its files out from under it. The lock is an
|
|
1074
|
+
atomic directory under the temp directory, keyed by the workspace root
|
|
1075
|
+
(`--cache-dir` does not make two runs strangers) and holding the
|
|
1076
|
+
holder's pid, so a lock a killed run left behind is reclaimed once its
|
|
1077
|
+
pid is gone. The temp directory is the one each process sees, `TMPDIR`
|
|
1078
|
+
included: two shells that set different ones (a `nix develop` shell
|
|
1079
|
+
sets its own, as can direnv or `sudo`) hold two locks and do not take
|
|
1080
|
+
turns (item 970). Run them under one `TMPDIR`. A lock under the
|
|
1081
|
+
workspace would survive a reboot that `/tmp` does not, and where a
|
|
1082
|
+
holder's start time is not readable (off Linux) a pid reused since
|
|
1083
|
+
would read as a live holder forever. Where the directory cannot be made at all (another user's
|
|
1084
|
+
lock, a temp directory this user cannot write) the run says so once and
|
|
1085
|
+
proceeds unlocked — a courtesy between cooperating runs, never a
|
|
1086
|
+
refusal — and a restore that then loses its staged files to the other
|
|
1087
|
+
run's clean fails with `restore of <hash> into <dir> was interrupted: a
|
|
1088
|
+
file it had just written vanished (ENOENT: …). Another vx run is using
|
|
1089
|
+
this workspace — re-run once it is done.`; the artifact is intact
|
|
1090
|
+
either way (a restore renames into place only once the whole archive
|
|
1091
|
+
has staged). Machines sharing a workspace over a network file system
|
|
1092
|
+
do not share a temp directory, so they do not share the lock.
|
|
1093
|
+
|
|
1094
|
+
An eviction can still land between a hit's probe and its restore: a
|
|
1095
|
+
`vx cache prune` in another shell, or the `cacheRetention` of another
|
|
1096
|
+
workspace that shares the `--cache-dir` (a different workspace, so a
|
|
1097
|
+
different lock), which cannot see this run's pending `accessed_at`
|
|
1098
|
+
bumps and reads its fresh hits as old. The restore then finds no
|
|
1099
|
+
artifact, and that entry is a **miss**: the run says `[vx] <id>: its
|
|
1100
|
+
cache artifact <hash> vanished before the restore … — running it`,
|
|
1101
|
+
runs the task, and saves it again. It failed every such task as an
|
|
1102
|
+
internal error until 2026-09-24 (upstream survey: nx#36688, nx#34032).
|
|
1103
|
+
A hit the local short-circuit was restoring ahead of its dependencies
|
|
1104
|
+
goes back to the schedule and runs once they are done, never in the
|
|
1105
|
+
restore's slot: built before them, its bytes would be saved under the
|
|
1106
|
+
healthy key (`tests/vanished-artifact.test.ts`).
|
|
1107
|
+
|
|
1108
|
+
A local artifact whose bytes are wrong (a failed checksum, a torn
|
|
1109
|
+
write, one past the artifact ceiling, an entry missing a recorded
|
|
1110
|
+
output, a name that now reads unsafe, X-115) is a miss the same way: `[vx] <id>: cache: corrupt artifact
|
|
1111
|
+
for <hash>: …; dropped it — running it`. The entry is dropped, so the
|
|
1112
|
+
task's save stores the key again; it failed the task as an internal
|
|
1113
|
+
error on every run until `--force` before A-52. A cache the run may
|
|
1114
|
+
only read keeps the entry (`…; left it (this cache is read-only)`).
|
|
1115
|
+
|
|
1116
|
+
## Storage layout
|
|
1117
|
+
|
|
1118
|
+
The run must be able to write here — it records its history at the
|
|
1119
|
+
end of every run, hit or miss — so a directory this user cannot write
|
|
1120
|
+
into (another user's `.vx`, a read-only checkout) fails the run before
|
|
1121
|
+
any task: `cache directory <path> is not writable (EACCES: …)`, with
|
|
1122
|
+
`--cache-dir <path>` as the way out. `vx cache prune` is refused the
|
|
1123
|
+
same way (its `--dry-run` only reads, and reads). The readers (`vx show`, `why`,
|
|
1124
|
+
`last`, `info`) open such a directory read-only and go on: a config
|
|
1125
|
+
that misses the evaluation cache is evaluated live and not stored.
|
|
1126
|
+
Where the directory cannot be created at all (a read-only checkout with
|
|
1127
|
+
no cache yet) a run says `cannot create cache directory <path>
|
|
1128
|
+
(EACCES: …)`, naming the workspace `cacheDir` field and `--cache-dir`;
|
|
1129
|
+
the readers and `vx cache prune` find no cache and go on.
|
|
1130
|
+
A full disk is the same kind of failure and is reported the same way,
|
|
1131
|
+
never as a corrupt artifact and never as a stack: a save that runs out
|
|
1132
|
+
of room is `[vx] cache save failed: ENOSPC …` (the task's work ran, the
|
|
1133
|
+
next run misses), a restore that does is the task's failure line
|
|
1134
|
+
`could not write its outputs (ENOSPC: …). Free space on that disk and
|
|
1135
|
+
re-run.`, and a run record that does is `[vx] run history not recorded:
|
|
1136
|
+
… — the verdict above stands` (the run exits by its tasks; `vx last`
|
|
1137
|
+
will not know that one). A full index (`SQLITE_FULL`) gives way where the
|
|
1138
|
+
write is only a memo: the file-hash memo, the output stamps and the
|
|
1139
|
+
access times are skipped and the run goes on; a prune unlinks the
|
|
1140
|
+
artifacts first, so the rows it then deletes have room to go, and a
|
|
1141
|
+
save's temp file is removed when its write fails.
|
|
1142
|
+
A process out of file descriptors (`EMFILE`, `ENFILE`) is named the
|
|
1143
|
+
same way: a save is `[vx] cache save failed: save of <hash> could not
|
|
1144
|
+
open a file (EMFILE: …) — … raise the limit (ulimit -n 4096) and
|
|
1145
|
+
re-run`, and a restore fails the task with that hint, never as a
|
|
1146
|
+
corrupt artifact (A-39). Opening the index is named the same way: SQLite reports a
|
|
1147
|
+
descriptor it could not get as `unable to open database file`, and vx
|
|
1148
|
+
says `the cache index <path> could not be opened (…) — …raise the limit`
|
|
1149
|
+
(A-56).
|
|
1150
|
+
A restore that cannot READ its artifact (a cache directory another
|
|
1151
|
+
user owns) names the cache, not the outputs: `restore of <hash> could
|
|
1152
|
+
not read its artifact (EACCES: …). Make the cache directory readable
|
|
1153
|
+
by this user, or point cacheDir / --cache-dir at one that is.` (A-40).
|
|
1154
|
+
An input the key must hash that this user cannot read fails its task
|
|
1155
|
+
naming the read: `<path> is not readable by this user (EACCES), and vx
|
|
1156
|
+
reads it to derive a cache key. Make it readable, or, for a task input,
|
|
1157
|
+
take it out of cache.inputs.files.` (A-50).
|
|
1158
|
+
|
|
1159
|
+
By default the entries and their artifacts live in a **shared store**
|
|
1160
|
+
in `~/.vx/<id>/cache`, and each workspace keeps only its own index in
|
|
1161
|
+
its `.vx/cache`: every checkout of the repository hits what another
|
|
1162
|
+
saved (a second clone, a worktree), and none reads another's history.
|
|
1163
|
+
The id is Nx 23's: 16 hex of a sha256 of the remote (`origin`, then
|
|
1164
|
+
`upstream`, `base`, the first; `host/owner/repo` in lower case, so ssh
|
|
1165
|
+
and https agree) and the workspace's path in the repository; with no
|
|
1166
|
+
remote, the first commit. A repository with neither (no commit yet, a
|
|
1167
|
+
shallow clone with no remote) shares nothing. Each level of `~/.vx` is
|
|
1168
|
+
owner-only; vx closes one of yours that others can read (chmod 700), and one others can write, or owned by another user, is not used. Design: [`design/shared-store-2026-10.md`](./design/shared-store-2026-10.md).
|
|
1169
|
+
|
|
1170
|
+
```
|
|
1171
|
+
~/.vx/<id>/cache/ the shared store
|
|
1172
|
+
├── store.db entries, entry_stdout, output_files,
|
|
1173
|
+
│ entry_inputs, store_meta; artifacts
|
|
1174
|
+
│ of at most 32 KiB compressed (v33)
|
|
1175
|
+
└── <hash>.tar.zst the larger artifacts (below)
|
|
1176
|
+
|
|
1177
|
+
<workspaceRoot>/.vx/cache/ this workspace's own index
|
|
1178
|
+
└── cache.db run history, memos, output stamps;
|
|
1179
|
+
attaches store.db as `store`
|
|
1180
|
+
```
|
|
1181
|
+
|
|
1182
|
+
The store's directory carries no version: every key is seeded with
|
|
1183
|
+
`CACHE_VERSION`, which moves when hashing or the artifact layout does, so
|
|
1184
|
+
two vx versions never read each other's artifacts. `store.db` is the
|
|
1185
|
+
artifacts' inventory and records its schema (`store_meta.schema`): a vx
|
|
1186
|
+
of another `SCHEMA_VERSION` drops its tables, prints nothing (the
|
|
1187
|
+
cache is vx's to keep), and keeps every artifact, each indexed again
|
|
1188
|
+
when its task next hits. The check, drop, re-create and stamp are one
|
|
1189
|
+
write transaction, so another version's open waits rather than landing
|
|
1190
|
+
between them. A home this user cannot write keeps the store
|
|
1191
|
+
in `<workspaceRoot>/.vx/cache/` instead, silently. Name a
|
|
1192
|
+
cache directory (`cacheDir` in vx.workspace.ts, `--cache-dir`, or
|
|
1193
|
+
`VX_CACHE_DIR`, in that order of precedence, relative to the workspace
|
|
1194
|
+
root) and it holds everything, shared with no other workspace:
|
|
1195
|
+
|
|
1196
|
+
```
|
|
1197
|
+
<cacheDir>/ (default .vx/cache when one is named)
|
|
1198
|
+
├── .gitignore `*` — written when the dir is created, or into an
|
|
1199
|
+
│ existing dir that lacks one, so the cache is
|
|
1200
|
+
│ never committed and never enumerated as an
|
|
1201
|
+
│ input (a user's own file is left alone)
|
|
1202
|
+
├── cache.db SQLite metadata + run history, and the
|
|
1203
|
+
│ artifacts of at most 32 KiB compressed,
|
|
1204
|
+
│ the same bytes as a file below (v33)
|
|
1205
|
+
├── cache.db-wal write-ahead log
|
|
1206
|
+
├── cache.db-shm shared memory
|
|
1207
|
+
├── failures/<runId>.json a failed run's task output and the files it
|
|
1208
|
+
│ names (`vx last --format json`, getFailures);
|
|
1209
|
+
│ the newest 50 kept
|
|
1210
|
+
└── <hash>.tar.zst one artifact per cache entry, the larger ones:
|
|
1211
|
+
├── stdout captured stdout (always present, may be empty)
|
|
1212
|
+
├── outputs/<rel> declared output files, project-relative (when any)
|
|
1213
|
+
├── workspace-outputs/<rel> declared outputs.workspaceFiles,
|
|
1214
|
+
│ WORKSPACE-ROOT-relative (when any)
|
|
1215
|
+
├── .vx-meta.json per-output [mode, mtimeMs], the key it was
|
|
1216
|
+
│ packed under (v35), the miss's CPU and RSS
|
|
1217
|
+
└── .vx-sum CRC-32 of every entry above (v36)
|
|
1218
|
+
```
|
|
1219
|
+
|
|
1220
|
+
`failures/` sits outside `cache.db` so the index layout, and every
|
|
1221
|
+
shared store under it, stays as it was. Each file is
|
|
1222
|
+
`{ runId, tasks: [{ taskId, project, task, exitCode, timedOut?, output, locations }] }`,
|
|
1223
|
+
written at run end (temp file + rename) only when a task failed with
|
|
1224
|
+
output; `output` is plain text (ANSI stripped, secrets masked, the first
|
|
1225
|
+
8 KiB and last 56 KiB). A refused write (`EACCES`, `ENOSPC`) costs only
|
|
1226
|
+
that output, never the run's verdict.
|
|
1227
|
+
|
|
1228
|
+
`<hash>` is the 16-hex xxh3 key. The `workspace-outputs/` namespace is
|
|
1229
|
+
additive: tasks that don't declare `outputs.workspaceFiles` produce
|
|
1230
|
+
byte-identical artifacts to the plain v17 format (which is why the
|
|
1231
|
+
field needed no `CACHE_VERSION` bump). `output_files` rows mirror the
|
|
1232
|
+
two namespaces — project rows store the bare rel, workspace rows store
|
|
1233
|
+
the full `workspace-outputs/<rel>` name as the discriminator.
|
|
1234
|
+
|
|
1235
|
+
The container is written and read by vx's own streaming tar code
|
|
1236
|
+
(`src/cache/tar-stream.ts`): ustar with the name/prefix split and a pax
|
|
1237
|
+
`path` record past it, header checksums, and nothing else — there is no
|
|
1238
|
+
`tar` subprocess, no libarchive, and no staging copy. **Save streams
|
|
1239
|
+
it**: each output is stat'd once, then read from disk as it is written
|
|
1240
|
+
and compressed straight into the temp file (measured 2026-09-03 on a
|
|
1241
|
+
150 MiB output: peak +705 MiB → +241–269 MiB; the streamed compressor
|
|
1242
|
+
is ~2.4× the one-call cost per byte, so a small artifact is packed in
|
|
1243
|
+
memory and compressed in one call — tiny saves unchanged within
|
|
1244
|
+
noise). Ingest lists entries through the reader without materialising
|
|
1245
|
+
a byte, and **restore streams it**: the zstd
|
|
1246
|
+
frame is decoded and the tar read as it arrives — ustar name/prefix
|
|
1247
|
+
(the prefix under POSIX `ustar\0` magic only: old GNU's `ustar ` keeps
|
|
1248
|
+
atime there, A-29),
|
|
1249
|
+
pax `path`/`size`, GNU long names, header checksums, truncation; an
|
|
1250
|
+
extended header (read whole) past 1 MiB or a pax `size` that is not a
|
|
1251
|
+
whole number is refused as corrupt, since a 1 GiB pax header costs 2 GiB
|
|
1252
|
+
of memory from a ~32 KB artifact — and
|
|
1253
|
+
every regular entry is written beside its target as a short `.vx-tmp-*` sibling (never a suffix on the target's name, which pushed a legal 242–255-byte name past NAME_MAX) and
|
|
1254
|
+
renamed into place only after the whole archive has ended cleanly and
|
|
1255
|
+
the index's recorded outputs are all present. vx itself holds one
|
|
1256
|
+
chunk of the tar at a time (measured 2026-09-03, incompressible
|
|
1257
|
+
artifacts, fresh process: 150 MiB peaked at +644 MiB through
|
|
1258
|
+
`Bun.Archive` and +49 MiB streamed at the same wall time; a single
|
|
1259
|
+
400 MiB entry restores at +30 MiB, the same as a 150 MiB one), while
|
|
1260
|
+
the per-entry buffers of many mid-sized entries are garbage the
|
|
1261
|
+
collector reclaims at its own pace (200 × 2 MiB measured +90–180 MiB,
|
|
1262
|
+
never the artifact's size). An artifact up to 4 MiB compressed is decoded in one call
|
|
1263
|
+
first — the stream setup costs ~35 µs each, 4% of the headline
|
|
1264
|
+
restore row when every artifact is a one-file `dist/` — and then fed
|
|
1265
|
+
to the same reader and extractor, so there is one extraction path.
|
|
1266
|
+
The reader reads a header in place when it lies within one chunk and
|
|
1267
|
+
its numeric fields off the bytes when they are plain octal (anything
|
|
1268
|
+
else takes the full parse): a 4-entry artifact's read 50–57 µs → 22–23
|
|
1269
|
+
(min of 15), and a 300-artifact, 20-file restore run's reader 224 → 92
|
|
1270
|
+
ms of main thread (2026-10-02).
|
|
1271
|
+
The 2 GiB decompression ceiling applies to both: declared size and
|
|
1272
|
+
output length for the one-call decode, a running count for the stream.
|
|
1273
|
+
An ingest bounds the compressed body first: a remote body past the
|
|
1274
|
+
ceiling's zstd bound (a length header, a Blob's size, or a running count
|
|
1275
|
+
on a chunked stream) is refused before it can fill the disk.
|
|
1276
|
+
A frame that declares no content size (what a streaming compressor
|
|
1277
|
+
writes — vx's own saves above 4 MiB) is always decoded as a stream, so
|
|
1278
|
+
a sizeless bomb has nowhere to expand and vx's artifacts ingest
|
|
1279
|
+
anywhere. So is a body that is not exactly one frame: the declaration is
|
|
1280
|
+
the first frame's alone, while the one-call decoder decodes every frame,
|
|
1281
|
+
so until 2026-09-27 (A-5) a 100-byte frame with a large one appended
|
|
1282
|
+
expanded whole in memory (2 GiB from a 32 KB body) before its length was
|
|
1283
|
+
checked. The frame's blocks are walked by their own sizes to tell. A save meets the same ceiling first: the pack plans the tar
|
|
1284
|
+
from one stat per output, and a tar past 2 GiB is refused before a byte
|
|
1285
|
+
is compressed — the run stays green, one status line names the task
|
|
1286
|
+
(`[vx] cache save failed: <task> is not cached: its outputs pack to …,
|
|
1287
|
+
past the 2.0 GB artifact ceiling a restore enforces`), and nothing is
|
|
1288
|
+
stored for a later run to hit and fail to restore. Left to the scan, a
|
|
1289
|
+
2.2 GB output paid the whole compress and a decode (6 to 14 s) to learn
|
|
1290
|
+
the same thing and called the task's outputs a corrupt artifact.
|
|
1291
|
+
|
|
1292
|
+
The last entry, `.vx-sum`, is a CRC-32 over every entry before it (each
|
|
1293
|
+
its name, a NUL, then its body) as 8 hex digits. Tar sums its headers
|
|
1294
|
+
only and neither zstd writer asks for a frame checksum, so a byte flipped
|
|
1295
|
+
in a raw zstd block (incompressible output: an image, a wasm, a tarball)
|
|
1296
|
+
decoded clean and a hit replayed the wrong bytes: 1,141 of 1,141 flips in
|
|
1297
|
+
a random 8 KB body went unseen (L-19). Scan and restore hash every entry
|
|
1298
|
+
as they read it and refuse an artifact whose sum is absent, wrong, or
|
|
1299
|
+
followed by another entry, before anything it holds lands; the refusal
|
|
1300
|
+
is a corrupt artifact, so a restore degrades to a miss and an ingest
|
|
1301
|
+
stores nothing. It catches damage in transit or at rest, not a store that
|
|
1302
|
+
forges artifacts: nothing a key carries can tell those from honest ones.
|
|
1303
|
+
Its cost is the CRC's (~10 GB/s): a 32 MB restore 11.3 → 14.4 ms and
|
|
1304
|
+
1,000 small files 16.2 → 18.9 ms on tmpfs, min of 15.
|
|
1305
|
+
|
|
1306
|
+
Tar headers carry mode and second mtimes. vx needs both permission bits (a lost executable bit builds cold
|
|
1307
|
+
and breaks warm) and millisecond mtimes (the skip-restore probe compares
|
|
1308
|
+
them) exactly, so the pack stats each output once and writes
|
|
1309
|
+
`.vx-meta.json` — `{ version, key, files: { <entry>: [mode, mtimeMs] }, exec? }` —
|
|
1310
|
+
into the archive. Restore applies both, at their edges too: a mode of
|
|
1311
|
+
000, an mtime of 0 (`SOURCE_DATE_EPOCH=0`), one before 1970, whose
|
|
1312
|
+
tar header carries 0 since ustar's field holds no sign, and one past
|
|
1313
|
+
March 2242, whose header carries the field's 11-digit maximum (X-114:
|
|
1314
|
+
the save failed `value … does not fit a 12-byte field`). Until 2026-09-27
|
|
1315
|
+
(A-4) the first two were skipped, so the file came back 0644 and
|
|
1316
|
+
stamped now and every later hit restored it again, and the third wrote
|
|
1317
|
+
a header the reader refused, so its task never saved. `exec` (`{ cpuMs?, peakRssBytes? }`,
|
|
1318
|
+
2026-09-12) is what the PRODUCING execution used: it rides the artifact
|
|
1319
|
+
so a machine that never ran the task — a fresh runner on a remote hit —
|
|
1320
|
+
still learns what the task needs (`@vzn/vx-schedule-history` packs on
|
|
1321
|
+
it), every wire ships the bytes verbatim, and an artifact without it
|
|
1322
|
+
reads as before. The ingest side takes the numbers only as plain
|
|
1323
|
+
non-negative numbers; anything else in a foreign sidecar is dropped. Entries that are not regular
|
|
1324
|
+
files (symlinks, hardlinks, devices) are never materialised — the
|
|
1325
|
+
reader reports them only to be skipped — so a
|
|
1326
|
+
poisoned artifact cannot smuggle one onto disk. Nor can it name a file
|
|
1327
|
+
the task did not declare: a remote artifact is ingested only when every
|
|
1328
|
+
file it carries matches the task's `cache.outputs` (the lookup passes
|
|
1329
|
+
them, `CacheGetContext.outputs`), and never when it holds one path as a
|
|
1330
|
+
file and a directory at once. Either used to reach the tree, an input
|
|
1331
|
+
overwritten or a `.git/hooks` file planted under a green
|
|
1332
|
+
`cache-hit-remote`, or a restore that failed every later run from the
|
|
1333
|
+
local copy (item 942); now the read is a miss and the task runs. On the save side a
|
|
1334
|
+
**symlinked output** is captured as its target's bytes and comes back
|
|
1335
|
+
as a regular file (a hit that finds the task's own link still current
|
|
1336
|
+
leaves it); a link to a directory, or a dangling one, has no bytes to
|
|
1337
|
+
store, so the save refuses it by name rather than cache an entry that
|
|
1338
|
+
restores to nothing. So does a link whose target is outside the
|
|
1339
|
+
project: vx reads outputs outside the task's sandbox, and a planted
|
|
1340
|
+
link packed a file the task could not read (L-23). Each refusal names
|
|
1341
|
+
the path as the config spells it (`workspaceFiles output gen/latest`). The clean before exec and restore removes every
|
|
1342
|
+
file AND symlink the output globs cover (a link is unlinked, never
|
|
1343
|
+
followed). An output directory that is a symlink (`dist -> real-out`) is
|
|
1344
|
+
followed by the clean as by the save and restore, so its target is the
|
|
1345
|
+
output and an entry holds only what its run wrote (X-88); one that
|
|
1346
|
+
resolves outside the project refuses the task, naming the link, and
|
|
1347
|
+
nothing is deleted through it (X-5). The clean prunes the directories it emptied (before a miss it keeps
|
|
1348
|
+
the directory a wildcard glob is rooted at, `dist` for `dist/**`, as the
|
|
1349
|
+
task writes there), so a task whose
|
|
1350
|
+
output changed shape — `dist/out` a directory one run and a file the
|
|
1351
|
+
next — restores either entry over the other's tree; a stray the globs
|
|
1352
|
+
do not cover, or an empty directory where the entry holds a file, that
|
|
1353
|
+
stands in an entry's way fails the restore naming it, not as a corrupt
|
|
1354
|
+
artifact (item 1094). A directory on the way that is a
|
|
1355
|
+
symbolic link OUT of the project (a `dist` made a link after the entry
|
|
1356
|
+
was saved) is never written through and never replaced — the link is
|
|
1357
|
+
the user's, and replacing it is the bug nx#37061 reports — so the
|
|
1358
|
+
restore refuses, naming it: `<dir>/dist is a symbolic link to
|
|
1359
|
+
<target>, outside <dir> — a cache restore never writes through a link
|
|
1360
|
+
that leaves its directory. Remove the link and re-run (the restore
|
|
1361
|
+
puts a real directory there), or stop declaring outputs under it.` A
|
|
1362
|
+
link that stays inside the project is written through, and so is one
|
|
1363
|
+
whose target is gone: the restore creates the directory it names —
|
|
1364
|
+
following a chain of links to its end — and writes through it, the
|
|
1365
|
+
link kept. Whether a link stays inside is decided on where it resolves,
|
|
1366
|
+
so a dangling link to `../elsewhere` or to an absolute path out of the
|
|
1367
|
+
project is the same refusal, nothing created outside, and a cycle of
|
|
1368
|
+
links is refused by name. (A link the output globs themselves cover,
|
|
1369
|
+
`dist/sub` under `dist/**`, is an output: the clean unlinks it.) Entry NAMES are
|
|
1370
|
+
validated by vx before anything decides where to write, and a bad
|
|
1371
|
+
entry anywhere, even the last, rejects the WHOLE archive: the temps
|
|
1372
|
+
are unlinked and the empty directories the extraction created are
|
|
1373
|
+
pruned (`tests/archive-security.test.ts`). A backslash in a name is a
|
|
1374
|
+
name character (vx runs on Linux and macOS; Windows through WSL), so an
|
|
1375
|
+
output like `dist/back\slash` caches and restores as written.
|
|
1376
|
+
|
|
1377
|
+
**Key properties:** one entry is one artifact — a row of `artifacts`
|
|
1378
|
+
when small, else one file, so eviction is one delete in the rows'
|
|
1379
|
+
transaction or a single unlink; no per-entry manifest, no separate
|
|
1380
|
+
`logs/` tree; and local + remote layers transport the exact same
|
|
1381
|
+
tar.zst bytes end-to-end, wherever the local copy lives.
|
|
1382
|
+
The artifact is the record and the index its inventory (owner,
|
|
1383
|
+
2026-10-06): a lookup reads the row first, and a key with no row whose
|
|
1384
|
+
`<hash>.tar.zst` is on disk (a `SCHEMA_VERSION` drop, a deleted
|
|
1385
|
+
`cache.db` or `store.db`) has the artifact indexed again from its own
|
|
1386
|
+
bytes, checked as a remote's are (its recorded key, its names against
|
|
1387
|
+
the task's declared outputs), and hits; one that fails the check is a
|
|
1388
|
+
miss, and the save that follows replaces it. An inline artifact with no
|
|
1389
|
+
row is indexed the same way, from its bytes, in one transaction; a reset
|
|
1390
|
+
keeps the `artifacts` table, but deleting `store.db` or `cache.db` by
|
|
1391
|
+
hand takes the inline artifacts with it. A `.tmp-*` a crashed save
|
|
1392
|
+
left is never a hit. `vx cache prune` sweeps row-less files, and so does a run whose workspace declares `cacheRetention`, at
|
|
1393
|
+
most once an hour (the sweep's clock is `schema_meta.orphans_swept_at`;
|
|
1394
|
+
the policy sums index rows, so orphans alone never make it due). A
|
|
1395
|
+
row-less artifact may be in use: two vx versions share one store, and
|
|
1396
|
+
each open drops the other's rows. So the policy judges it as it judges a
|
|
1397
|
+
row, its file time standing for `accessed_at`: past `olderThan` it goes,
|
|
1398
|
+
and under `maxSize` it counts, oldest use first with the rows. A hit
|
|
1399
|
+
renews a file time over an hour old, so the last use is read as the file
|
|
1400
|
+
time plus an hour. An inline artifact with no row is judged the same
|
|
1401
|
+
way, its `at` standing for the file time, and `vx info` counts it with
|
|
1402
|
+
the files. A temp a crashed save left goes once it is an hour
|
|
1403
|
+
old, and so does a file an inline artifact of its key shadows; nothing
|
|
1404
|
+
younger than an hour is taken. Re-indexing touches the
|
|
1405
|
+
artifact's mtime before it links its temp, so the sweep sees that one
|
|
1406
|
+
fresh too.
|
|
1407
|
+
|
|
1408
|
+
A row's `accessed_at` is renewed the same way: a hit renews it only once
|
|
1409
|
+
it is over an hour old (owner, 2026-10-08). Renewing every hit rewrote
|
|
1410
|
+
2,180 rows at a warm 1,090-package run's close, ~30 ms. So the policy
|
|
1411
|
+
reads a row's last use as `accessed_at` plus an hour too: an entry used
|
|
1412
|
+
within `olderThan` is never pruned, and one may stay up to an hour past
|
|
1413
|
+
it (`tests/cache.test.ts` › "an entry used within the age limit is never
|
|
1414
|
+
pruned").
|
|
1415
|
+
Captured stdout is stored twice on purpose: in the artifact (so it
|
|
1416
|
+
survives the remote round-trip) and in the `entries` row (so a local
|
|
1417
|
+
hit replays it with pure SQL, never decompressing the artifact).
|
|
1418
|
+
|
|
1419
|
+
### SQLite tables
|
|
1420
|
+
|
|
1421
|
+
`schema_meta.version` is the gate: an index written by any other
|
|
1422
|
+
`SCHEMA_VERSION`, earlier or newer, is reset by the first run that opens
|
|
1423
|
+
it (pre-alpha: no migrations; the index is an inventory, owner
|
|
1424
|
+
2026-10-06): every table but `schema_meta` is dropped and recreated,
|
|
1425
|
+
so each comes back in its current shape (A-54: `config_closures` and
|
|
1426
|
+
`output_dirs` kept an earlier vx's columns). The check, drop, re-create
|
|
1427
|
+
and stamp are one write transaction, so another version's open waits
|
|
1428
|
+
rather than landing between them; an index already current is opened
|
|
1429
|
+
without the lock.
|
|
1430
|
+
A reading verb (`vx why`, `vx last`, `vx info`) leaves it untouched and
|
|
1431
|
+
says why (item 896; `vx cache prune --dry-run` previews the reset
|
|
1432
|
+
instead, item 1083).
|
|
1433
|
+
That open prints nothing (owner, 2026-10-06: the cache is vx's to
|
|
1434
|
+
keep). The artifacts stay: each is indexed again from its own bytes
|
|
1435
|
+
when its task next asks for its key (below).
|
|
1436
|
+
|
|
1437
|
+
```sql
|
|
1438
|
+
-- src/cache/schema.ts (SCHEMA_VERSION = 'v34', in cache.ts)
|
|
1439
|
+
-- With a shared store, entries, entry_stdout, output_files, entry_inputs
|
|
1440
|
+
-- and store_meta live in its store.db, attached as `store`; the rest is
|
|
1441
|
+
-- the workspace's cache.db. A named cache dir holds all of them.
|
|
1442
|
+
|
|
1443
|
+
CREATE TABLE schema_meta (
|
|
1444
|
+
key TEXT PRIMARY KEY, -- 'version', 'cache_version', 'orphans_swept_at', 'file_hashes_swept_at', 'config_evals_swept_at', 'store_dir'
|
|
1445
|
+
value TEXT NOT NULL
|
|
1446
|
+
);
|
|
1447
|
+
|
|
1448
|
+
-- The config-evaluation cache (§ Config evaluation cache): the validated,
|
|
1449
|
+
-- JSON-serialised result of a provably pure config, keyed by everything
|
|
1450
|
+
-- the evaluation could have observed. Machine-local. Rows not written in
|
|
1451
|
+
-- 30 days are swept at most once a day (`config_evals_swept_at`).
|
|
1452
|
+
CREATE TABLE config_evals (
|
|
1453
|
+
key TEXT PRIMARY KEY,
|
|
1454
|
+
json TEXT NOT NULL,
|
|
1455
|
+
created_at INTEGER NOT NULL
|
|
1456
|
+
);
|
|
1457
|
+
|
|
1458
|
+
CREATE TABLE entries (
|
|
1459
|
+
hash TEXT PRIMARY KEY, -- the 16-hex xxh3 cache key
|
|
1460
|
+
project TEXT NOT NULL,
|
|
1461
|
+
task TEXT NOT NULL,
|
|
1462
|
+
command TEXT NOT NULL,
|
|
1463
|
+
exit_code INTEGER NOT NULL,
|
|
1464
|
+
duration_ms INTEGER NOT NULL,
|
|
1465
|
+
size_bytes INTEGER NOT NULL, -- artifact size
|
|
1466
|
+
created_at INTEGER NOT NULL, -- ms-epoch
|
|
1467
|
+
accessed_at INTEGER NOT NULL, -- ms-epoch; a hit renews it once over an hour old (LRU)
|
|
1468
|
+
cpu_ms INTEGER, -- v26: the producing execution's usage, from
|
|
1469
|
+
peak_rss_bytes INTEGER -- the artifact's sidecar (save + ingest)
|
|
1470
|
+
);
|
|
1471
|
+
|
|
1472
|
+
-- v29: stdout apart from its entry. An UPDATE rewrites a whole record, so
|
|
1473
|
+
-- the accessed_at bump rewrote each hit's stdout (up to 16 MB): 200 hits
|
|
1474
|
+
-- of 1 MB cost the run's close 125-150 ms.
|
|
1475
|
+
CREATE TABLE entry_stdout (
|
|
1476
|
+
hash TEXT PRIMARY KEY, -- FK entries(hash) ON DELETE CASCADE
|
|
1477
|
+
stdout TEXT NOT NULL -- captured stdout (pure-SQL hit replay); no row = ''
|
|
1478
|
+
);
|
|
1479
|
+
|
|
1480
|
+
CREATE TABLE runs (
|
|
1481
|
+
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
1482
|
+
hash TEXT NOT NULL, -- '' when the outcome derived no key (see below)
|
|
1483
|
+
project TEXT NOT NULL,
|
|
1484
|
+
task TEXT NOT NULL,
|
|
1485
|
+
status TEXT NOT NULL, -- success | failed | cache-hit | cache-hit-remote | skipped
|
|
1486
|
+
exit_code INTEGER NOT NULL,
|
|
1487
|
+
duration_ms INTEGER NOT NULL,
|
|
1488
|
+
forward_args TEXT, -- salted xxh3 of the JSON-encoded `--` args; null when none
|
|
1489
|
+
started_at INTEGER NOT NULL, -- ms-epoch
|
|
1490
|
+
ended_at INTEGER NOT NULL,
|
|
1491
|
+
run_id TEXT, -- UUIDv7 shared across all tasks in one invocation
|
|
1492
|
+
cpu_ms INTEGER,
|
|
1493
|
+
peak_rss_bytes INTEGER,
|
|
1494
|
+
wallclock_start_ns INTEGER, -- bigint; serialized as SQLite INTEGER (signed 64-bit)
|
|
1495
|
+
wallclock_end_ns INTEGER,
|
|
1496
|
+
cache_hit INTEGER, -- 0/1; convenience for flamegraph color
|
|
1497
|
+
attempts INTEGER, -- v23: attempts a retried task took (>1)
|
|
1498
|
+
cached INTEGER, -- v25: 1 = declared a cache block; 0 = runs every time
|
|
1499
|
+
-- v27: why the task failed or was skipped, as the run's footer said it
|
|
1500
|
+
-- (items 267–270); NULL where the reason does not apply
|
|
1501
|
+
blocked_by TEXT, -- a skip's root blocker, or what a task ran behind (a task id)
|
|
1502
|
+
timed_out INTEGER, -- 1 when vx's own timeout killed it
|
|
1503
|
+
sandbox_violations INTEGER, -- the sandbox's violation count
|
|
1504
|
+
not_ready TEXT, -- 'timeout' | 'exited' | 'spawn' (persistent task)
|
|
1505
|
+
restored INTEGER -- v31, on a hit: 1 = outputs restored, 0 = up to date
|
|
1506
|
+
);
|
|
1507
|
+
|
|
1508
|
+
-- Two indexes, both append-only under a run's inserts: every row of a run
|
|
1509
|
+
-- carries the same run_id and a started_at newer than everything before it.
|
|
1510
|
+
-- There is deliberately NO (project, task) index — it scattered every run's
|
|
1511
|
+
-- rows over one B-tree leaf per pair (a 1,000-hit warm run's record stage
|
|
1512
|
+
-- went 57–79 ms → 14–19 ms without it) while its readers gained nothing;
|
|
1513
|
+
-- the history reader bounds its scan by rowid instead (see history.md).
|
|
1514
|
+
CREATE INDEX runs_started_at ON runs(started_at);
|
|
1515
|
+
CREATE INDEX runs_run_id ON runs(run_id);
|
|
1516
|
+
-- The one keyed index, PARTIAL over failed rows: a green run's inserts only
|
|
1517
|
+
-- evaluate its predicate, and the flakiness probe after a miss
|
|
1518
|
+
-- (failure-mode.ts, "did this key ever fail?") reads a few leaves.
|
|
1519
|
+
CREATE INDEX runs_failed ON runs(hash) WHERE status = 'failed';
|
|
1520
|
+
|
|
1521
|
+
-- Every non-group, non-aborted outcome of a run gets a row, so
|
|
1522
|
+
-- `invocations.task_count` always equals `COUNT(*)` here for that run_id
|
|
1523
|
+
-- and the terminal summary's "N total". Two of those outcomes never derive
|
|
1524
|
+
-- a cache key — a `skipped` task (its upstream failed, so it never probed)
|
|
1525
|
+
-- and a `persistent` one (a dev server is not cacheable) — and they store
|
|
1526
|
+
-- `hash = ''`. `''` is impossible for a real key (16 hex chars), so it reads
|
|
1527
|
+
-- unambiguously as "no key recorded"; the key-diff surfaces (`vx why`'s
|
|
1528
|
+
-- whyDidThisRerun, the cache-key diff) guard it rather than reporting
|
|
1529
|
+
-- "inputs unchanged" from two rows that never had inputs to compare.
|
|
1530
|
+
--
|
|
1531
|
+
-- A `skipped` row is a task of the run but NOT an execution, so the rate and
|
|
1532
|
+
-- average aggregates in metrics.ts exclude it: counting a zero-duration
|
|
1533
|
+
-- non-event would dilute success rate, hit rate and mean duration. The
|
|
1534
|
+
-- completeness reads (listRuns / getRun / the run-detail timeline) include it.
|
|
1535
|
+
|
|
1536
|
+
-- The file-hash memo: a content hash per input file that git could not
|
|
1537
|
+
-- answer (untracked or dirty), keyed by the stat identity that proves
|
|
1538
|
+
-- the bytes unchanged. Machine-local; § Cache key derivation step 12.
|
|
1539
|
+
-- A writing close drops rows unwritten for 30 days, at most once a day
|
|
1540
|
+
-- (a memo miss is the whole cost of a dropped row; item 1082).
|
|
1541
|
+
CREATE TABLE file_hashes (
|
|
1542
|
+
path TEXT PRIMARY KEY,
|
|
1543
|
+
mtime_ms INTEGER NOT NULL,
|
|
1544
|
+
size_bytes INTEGER NOT NULL,
|
|
1545
|
+
ctime_ms INTEGER NOT NULL,
|
|
1546
|
+
ino INTEGER NOT NULL,
|
|
1547
|
+
content_hash TEXT NOT NULL,
|
|
1548
|
+
seen_at INTEGER NOT NULL
|
|
1549
|
+
);
|
|
1550
|
+
|
|
1551
|
+
-- The size of each index blob the enumeration checked against the
|
|
1552
|
+
-- worktree size git recorded (§ Cache key derivation, A-60): fixed for
|
|
1553
|
+
-- its OID, so a warm run asks git for none. Swept with file_hashes by
|
|
1554
|
+
-- seen_at, the time the row was written.
|
|
1555
|
+
CREATE TABLE blob_sizes (
|
|
1556
|
+
oid TEXT PRIMARY KEY,
|
|
1557
|
+
size INTEGER NOT NULL,
|
|
1558
|
+
seen_at INTEGER NOT NULL
|
|
1559
|
+
);
|
|
1560
|
+
|
|
1561
|
+
-- v30: the paths an index distrusts, by a hash of the index file and the
|
|
1562
|
+
-- pathspecs (A-60): a warm run reads one row, not one per blob. Swept
|
|
1563
|
+
-- with file_hashes.
|
|
1564
|
+
CREATE TABLE blob_verdicts (
|
|
1565
|
+
digest TEXT PRIMARY KEY,
|
|
1566
|
+
paths TEXT NOT NULL,
|
|
1567
|
+
seen_at INTEGER NOT NULL
|
|
1568
|
+
);
|
|
1569
|
+
|
|
1570
|
+
-- v16: per-output-file fingerprints, scoped by the entry that produced
|
|
1571
|
+
-- them — what a hit stats to skip the restore when the tree is already
|
|
1572
|
+
-- current (§ A current tree). ON DELETE CASCADE follows a prune.
|
|
1573
|
+
CREATE TABLE output_files (
|
|
1574
|
+
entry_hash TEXT NOT NULL,
|
|
1575
|
+
path TEXT NOT NULL,
|
|
1576
|
+
size_bytes INTEGER NOT NULL,
|
|
1577
|
+
mode INTEGER NOT NULL,
|
|
1578
|
+
mtime_ms INTEGER NOT NULL,
|
|
1579
|
+
PRIMARY KEY (entry_hash, path),
|
|
1580
|
+
FOREIGN KEY (entry_hash) REFERENCES entries(hash) ON DELETE CASCADE
|
|
1581
|
+
);
|
|
1582
|
+
|
|
1583
|
+
-- v28, its own table since v32: inode + ctime THIS workspace saw after
|
|
1584
|
+
-- the save/restore that last wrote the file (item 886); no row is never
|
|
1585
|
+
-- current. Apart from the shared entry: two worktrees overwrote each
|
|
1586
|
+
-- other's stamps and every switch restored. No foreign key (the entry
|
|
1587
|
+
-- may be the store's); prune and re-save delete what they orphan.
|
|
1588
|
+
CREATE TABLE output_stamps (
|
|
1589
|
+
entry_hash TEXT NOT NULL,
|
|
1590
|
+
path TEXT NOT NULL,
|
|
1591
|
+
ino INTEGER NOT NULL,
|
|
1592
|
+
ctime_ms INTEGER NOT NULL,
|
|
1593
|
+
PRIMARY KEY (entry_hash, path)
|
|
1594
|
+
);
|
|
1595
|
+
|
|
1596
|
+
-- Each config's ORDERED import closure (the config first), so a warm
|
|
1597
|
+
-- load keys it by stat-hashing the list through file_hashes instead of
|
|
1598
|
+
-- reading every file. Machine-local; pruned with config_evals.
|
|
1599
|
+
CREATE TABLE config_closures (
|
|
1600
|
+
config_path TEXT PRIMARY KEY,
|
|
1601
|
+
files_json TEXT NOT NULL,
|
|
1602
|
+
created_at INTEGER NOT NULL
|
|
1603
|
+
);
|
|
1604
|
+
|
|
1605
|
+
-- Every directory under a whole-subtree output glob, with its mtime as
|
|
1606
|
+
-- of the last save or restore on THIS machine: unchanged mtimes prove
|
|
1607
|
+
-- the output SET unchanged without a glob walk. Machine-local — a remote
|
|
1608
|
+
-- ingest writes none, and the first hit after it walks and records.
|
|
1609
|
+
CREATE TABLE output_dirs (
|
|
1610
|
+
entry_hash TEXT NOT NULL,
|
|
1611
|
+
path TEXT NOT NULL,
|
|
1612
|
+
mtime_ms INTEGER NOT NULL,
|
|
1613
|
+
PRIMARY KEY (entry_hash, path) -- no foreign key since v32, as output_stamps
|
|
1614
|
+
);
|
|
1615
|
+
|
|
1616
|
+
-- v22 (Tier 3): one header row per `vx run` invocation. The `runs`
|
|
1617
|
+
-- table is per-task; this is the per-invocation record carrying the
|
|
1618
|
+
-- command, git/CI/host context, tags, and run-level counts. Recorded
|
|
1619
|
+
-- atomically alongside `runs` via recordRunBundle (one transaction).
|
|
1620
|
+
CREATE TABLE invocations (
|
|
1621
|
+
run_id TEXT PRIMARY KEY, -- UUIDv7, == runs.run_id
|
|
1622
|
+
command TEXT NOT NULL, -- full argv, e.g. "vx run build --all"
|
|
1623
|
+
requested_tasks TEXT NOT NULL, -- JSON string[] of options.tasks
|
|
1624
|
+
cache_policy TEXT NOT NULL, -- compact flags, e.g. "lR,lW,rR,rW"
|
|
1625
|
+
concurrency INTEGER NOT NULL,
|
|
1626
|
+
flow TEXT, -- 'focused' | 'broad' | NULL (programmatic)
|
|
1627
|
+
started_at INTEGER NOT NULL, -- ms-epoch
|
|
1628
|
+
ended_at INTEGER NOT NULL,
|
|
1629
|
+
total_duration_ms INTEGER NOT NULL, -- wall clock of the whole run
|
|
1630
|
+
task_count INTEGER NOT NULL, -- non-group, non-aborted tasks recorded
|
|
1631
|
+
failed_count INTEGER NOT NULL,
|
|
1632
|
+
hit_count INTEGER NOT NULL, -- cache-hit + cache-hit-remote
|
|
1633
|
+
hit_local_count INTEGER NOT NULL, -- cache-hit
|
|
1634
|
+
hit_remote_count INTEGER NOT NULL, -- cache-hit-remote
|
|
1635
|
+
up_to_date_count INTEGER NOT NULL DEFAULT 0, -- v31: hits that restored nothing
|
|
1636
|
+
restored_local_count INTEGER NOT NULL DEFAULT 0, -- v31: hits that restored, local layer
|
|
1637
|
+
restored_remote_count INTEGER NOT NULL DEFAULT 0, -- v31: hits that restored, remote layer
|
|
1638
|
+
exit_ok INTEGER NOT NULL, -- 1 if the run's `ok`
|
|
1639
|
+
commit_sha TEXT, -- nullable: not a git repo / probe failed
|
|
1640
|
+
branch TEXT,
|
|
1641
|
+
dirty INTEGER, -- 1 if worktree had uncommitted changes
|
|
1642
|
+
ci INTEGER NOT NULL, -- 1 if a CI env was detected
|
|
1643
|
+
ci_provider TEXT, -- 'github' | 'gitlab' | 'buildkite' | 'circleci' | 'generic'
|
|
1644
|
+
host TEXT, -- os.hostname()
|
|
1645
|
+
os TEXT, -- process.platform
|
|
1646
|
+
arch TEXT, -- process.arch
|
|
1647
|
+
vx_version TEXT NOT NULL,
|
|
1648
|
+
tags TEXT NOT NULL DEFAULT '{}' -- JSON object {k:v} from --tag
|
|
1649
|
+
);
|
|
1650
|
+
CREATE INDEX invocations_started ON invocations(started_at);
|
|
1651
|
+
CREATE INDEX invocations_branch ON invocations(branch);
|
|
1652
|
+
CREATE INDEX invocations_ci ON invocations(ci);
|
|
1653
|
+
|
|
1654
|
+
-- v22 (Tier 3): the input-fingerprint moat. One row per cache-key
|
|
1655
|
+
-- component, keyed by the cache-ENTRY hash it belongs to (NOT a run
|
|
1656
|
+
-- id). Written INSIDE the entry-save transaction — only on a cache
|
|
1657
|
+
-- MISS, never on a hit — via INSERT OR IGNORE. A warm all-cache-hit
|
|
1658
|
+
-- run writes nothing here. The "why did this re-run?" diff resolves a
|
|
1659
|
+
-- run to its task hash (runs.hash), then joins two entries' components
|
|
1660
|
+
-- over (kind, name), no recompute. v34: one row per entry, the
|
|
1661
|
+
-- components one JSON array (a row per component cost a b-tree insert
|
|
1662
|
+
-- each: ~0.5 ms a save at 50 input files). ON DELETE CASCADE sweeps the
|
|
1663
|
+
-- row when a prune drops the entry.
|
|
1664
|
+
CREATE TABLE entry_inputs (
|
|
1665
|
+
entry_hash TEXT PRIMARY KEY, -- == entries.hash / runs.hash
|
|
1666
|
+
components TEXT NOT NULL, -- JSON [[kind, name, hash], …] in (kind, name) code-unit order
|
|
1667
|
+
FOREIGN KEY (entry_hash) REFERENCES entries(hash) ON DELETE CASCADE
|
|
1668
|
+
);
|
|
1669
|
+
-- kind: file|env|runtime|ws-runtime|upstream|package|config|forward|workspace|plugin|format
|
|
1670
|
+
-- name: file: workspace-rel path; env: var name; upstream: task id; …
|
|
1671
|
+
-- hash: env|runtime|ws-runtime|forward|plugin: xxh3hex(salt + value); an unset env var: 'unset'
|
|
1672
|
+
|
|
1673
|
+
-- v32: what belongs to the entries, not to one workspace: 'value_salt',
|
|
1674
|
+
-- the salt entry_inputs digests are taken under, so they compare with
|
|
1675
|
+
-- another workspace's run.
|
|
1676
|
+
CREATE TABLE store_meta (
|
|
1677
|
+
key TEXT PRIMARY KEY,
|
|
1678
|
+
value TEXT NOT NULL
|
|
1679
|
+
);
|
|
1680
|
+
|
|
1681
|
+
-- v33: an artifact of at most 32 KiB compressed, its exact <hash>.tar.zst
|
|
1682
|
+
-- bytes, written in the transaction that writes its entry rows. In the
|
|
1683
|
+
-- store (or a named cache dir's cache.db), and outside every drop a
|
|
1684
|
+
-- SCHEMA_VERSION reset makes: the artifact is the record.
|
|
1685
|
+
CREATE TABLE artifacts (
|
|
1686
|
+
hash TEXT PRIMARY KEY,
|
|
1687
|
+
at INTEGER NOT NULL, -- last use (ms): a row-less sweep's file time
|
|
1688
|
+
bytes BLOB NOT NULL
|
|
1689
|
+
) WITHOUT ROWID;
|
|
1690
|
+
|
|
1691
|
+
-- 'layout' = 'a1': the artifacts table's own version; another value
|
|
1692
|
+
-- drops the table, silently. Nothing else drops it. Read only where
|
|
1693
|
+
-- the schema was not this vx's, so a layout change bumps the schema.
|
|
1694
|
+
CREATE TABLE artifacts_meta (
|
|
1695
|
+
key TEXT PRIMARY KEY,
|
|
1696
|
+
value TEXT NOT NULL
|
|
1697
|
+
);
|
|
1698
|
+
```
|
|
1699
|
+
|
|
1700
|
+
WAL mode is on; readers don't block writers. `PRAGMA busy_timeout =
|
|
1701
|
+
5000` makes concurrent `vx run` invocations queue instead of failing
|
|
1702
|
+
with `SQLITE_BUSY`.
|
|
1703
|
+
|
|
1704
|
+
> **Trust boundary (Tier 3):** `entry_inputs` stores a digest of each
|
|
1705
|
+
> value-bearing component — `env`, `runtime`, `ws-runtime`, `forward`
|
|
1706
|
+
> and `plugin` rows hold `xxh3hex(salt + value)`, an unset env var the
|
|
1707
|
+
> literal `'unset'` — never the value, so a secret read as a cache input
|
|
1708
|
+
> does not land in `cache.db` as plaintext. The salt is 128 random bits
|
|
1709
|
+
> the store draws once (`store_meta` key `value_salt`): `vx why` prints
|
|
1710
|
+
> these digests, and an unkeyed xxh3 in a public CI log let anyone
|
|
1711
|
+
> confirm or brute-force a short secret. The "why did this re-run?" diff
|
|
1712
|
+
> only needs to know a component changed, which the digest tells it.
|
|
1713
|
+
> `cache.db` still records commands and captured stdout; it is a local,
|
|
1714
|
+
> gitignored, single-user file.
|
|
1715
|
+
|
|
1716
|
+
The Tier-3 tables persist components that were **already fed to
|
|
1717
|
+
`Cache.key()`** — the cache key derivation is unchanged, so the
|
|
1718
|
+
`CACHE_VERSION` is NOT bumped (only `SCHEMA_VERSION` rolls to `v22`).
|
|
1719
|
+
Capture happens as a pure side-channel inside the existing `key()` fold
|
|
1720
|
+
(`CacheKeyInput.captureInto`), **only on a cache miss** (the warm path
|
|
1721
|
+
captures nothing), and the rows are persisted with the entry — so a
|
|
1722
|
+
warm all-cache-hit run does no extra hashing, I/O, or DB writes for the
|
|
1723
|
+
moat.
|
|
1724
|
+
|
|
1725
|
+
### Why SQLite + a single artifact per entry
|
|
1726
|
+
|
|
1727
|
+
- **Index queries are fast.** Stats (`SELECT COUNT(*) FROM entries`),
|
|
1728
|
+
TTL pruning (`WHERE accessed_at < ?`), per-task lookup
|
|
1729
|
+
(`WHERE hash = ?`) all hit a B-tree.
|
|
1730
|
+
- **A hit costs SQL, not decompression.** Metadata + stdout live in
|
|
1731
|
+
the row; the artifact is only opened when outputs actually restore.
|
|
1732
|
+
- **One artifact = one wire payload.** The same tar.zst bytes serve
|
|
1733
|
+
local storage and the remote round-trip — no repacking.
|
|
1734
|
+
- **One handle, one schema-meta sentinel.** An older schema drops
|
|
1735
|
+
and recreates every table but `schema_meta` (pre-alpha) — there's no migration code to maintain.
|
|
1736
|
+
|
|
1737
|
+
## Config evaluation cache
|
|
1738
|
+
|
|
1739
|
+
Evaluating configs is the largest fixed cost of a warm run on a big
|
|
1740
|
+
workspace (~80 ms for 1000 synthetic configs; reading them as data is
|
|
1741
|
+
~12 ms). A config that is **provably pure** — every import relative or
|
|
1742
|
+
`@vzn/vx`, and no mention of `process`, `Bun`, `Date`, `fetch`,
|
|
1743
|
+
`import.meta`, `require`, a dynamic `import()`, `await`, … outside string
|
|
1744
|
+
literals and comments — is served from `cache.db`'s `config_evals` table,
|
|
1745
|
+
keyed by the git blob id of the config and of every file in its
|
|
1746
|
+
relative import closure, the workspace fingerprint (lockfiles) and Bun's
|
|
1747
|
+
version. Editing a shared preset moves the key. On a warm run the
|
|
1748
|
+
closure is remembered per config, so the key comes from a stat-backed
|
|
1749
|
+
identity per file — no read, no scan — for every config whose relative
|
|
1750
|
+
imports name their files outright: an explicit extension, the file
|
|
1751
|
+
itself, no symlink on the way (an extensionless import, a `.js` Bun
|
|
1752
|
+
answers with a `.ts`, or a link could be re-resolved without touching a
|
|
1753
|
+
listed file, so such a config keeps the scan). Anything the static check cannot prove
|
|
1754
|
+
pure evaluates live, exactly as before, so the cache can be slower but
|
|
1755
|
+
not wrong for a config written in good faith (the check is syntactic; one
|
|
1756
|
+
built to defeat it can). Details and the deny-list:
|
|
1757
|
+
[`modules/config-cache.md`](./modules/config-cache.md).
|
|
1758
|
+
|
|
1759
|
+
## Performance characteristics
|
|
1760
|
+
|
|
1761
|
+
- **Hashing cost** on a clean tree is near-zero per file: git index
|
|
1762
|
+
OIDs come from the bulk enumeration spawn, so key derivation does no
|
|
1763
|
+
file reads. Dirty/untracked files — and any path whose OID is not
|
|
1764
|
+
trustworthy (see "Clean filters") — hash in-process (whole-file
|
|
1765
|
+
read, behind a `(mtime, size, ctime, ino)` memo). Narrow
|
|
1766
|
+
`inputs.files` still helps on heavily dirty trees.
|
|
1767
|
+
- **Cache read** is three indexed `SELECT`s (the `entries` row, which
|
|
1768
|
+
joins whether the artifact is inline, its `output_files` and its
|
|
1769
|
+
`output_dirs`), plus a `stat` of the artifact when it is a file.
|
|
1770
|
+
Restore is a tar.zst extract, skipped entirely when the on-disk
|
|
1771
|
+
tree already matches. `accessed_at` bumps (only rows over an hour
|
|
1772
|
+
old) are batched into one UPDATE at flush time.
|
|
1773
|
+
- **Cache write** is one in-process tar.zst pack + one SQLite
|
|
1774
|
+
transaction, which holds a small artifact's bytes itself; a large one
|
|
1775
|
+
adds a temp write and an atomic rename. Hashing dominates the run; storage itself
|
|
1776
|
+
is cheap. The remote upload (if any) is backgrounded.
|
|
1777
|
+
- **Workspace fingerprint** is computed once per `vx run` invocation
|
|
1778
|
+
and reused for every task in that run; after a task that may rewrite
|
|
1779
|
+
one of its files ran, one `stat` per file re-checks it (item 750).
|
|
1780
|
+
|
|
1781
|
+
## What's NOT in the key (and why)
|
|
1782
|
+
|
|
1783
|
+
- **`exec.env.passThrough` _values_.** Would force cache misses across
|
|
1784
|
+
machines with different CI flags, regions, or shell prompts. The
|
|
1785
|
+
_names_ are in the config hash (step 5) so adding/removing a
|
|
1786
|
+
passthrough still bumps the key for affected tasks.
|
|
1787
|
+
- **Files outside the project directory that aren't declared.**
|
|
1788
|
+
Workspace-root configs (`tsconfig.base.json`, etc.) are not
|
|
1789
|
+
auto-included — declare them via `cache.inputs.workspaceFiles`
|
|
1790
|
+
(root-anchored globs).
|
|
1791
|
+
- **Node / Bun / OS / build-tool versions — unless you declare them.**
|
|
1792
|
+
The canonical mechanism is `cache.inputs.runtime` /
|
|
1793
|
+
`workspaceRuntime` (e.g. `workspaceRuntime: ['node -v']`): the
|
|
1794
|
+
command output is resolved live at hash time, so it stays correct
|
|
1795
|
+
under `--frozen`. Avoid baking versions via
|
|
1796
|
+
`define: { X: execSync(...) }` — that value freezes into the config
|
|
1797
|
+
object at lock time and goes stale.
|
|
1798
|
+
- **`vx-lock.json`** — globally excluded (v24); also filtered out of
|
|
1799
|
+
`--affected` change sets.
|
|
1800
|
+
- **`exec.remote`.** Pure PLACEMENT: it decides _where_ a task runs,
|
|
1801
|
+
never what it produces, so pinning a task to this machine does not
|
|
1802
|
+
bust its cache. It must not — the
|
|
1803
|
+
whole contract of a remote executor is that the same command over the
|
|
1804
|
+
same inputs yields the same outputs, so a key that moved with
|
|
1805
|
+
placement would split your laptop from the worker pool over nothing.
|
|
1806
|
+
|
|
1807
|
+
**`exec.timeout` and `exec.retries` are the deliberate exception** —
|
|
1808
|
+
they stay in the key. That asymmetry is easy to misread as an
|
|
1809
|
+
oversight, so: a timeout or a retry budget can change _whether the
|
|
1810
|
+
task completes at all_, which is a different question from where it
|
|
1811
|
+
ran. They were folded in before the placement fields existed, and
|
|
1812
|
+
stripping them now would be a `CACHE_VERSION` bump for no correctness
|
|
1813
|
+
gain.
|
|
1814
|
+
|
|
1815
|
+
## Bumping `CACHE_VERSION`
|
|
1816
|
+
|
|
1817
|
+
The cache records the version it was written under
|
|
1818
|
+
(`schema_meta.cache_version`). After an upgrade every cached task
|
|
1819
|
+
misses once and re-saves, with nothing printed (owner, 2026-10-06: the
|
|
1820
|
+
cache is vx's to keep). The index survives, so the old entries stay
|
|
1821
|
+
until they age out under `vx cache prune --older-than` or
|
|
1822
|
+
`cacheRetention`; no key derives to them again.
|
|
1823
|
+
|
|
1824
|
+
Required when:
|
|
1825
|
+
|
|
1826
|
+
- A new field is added to the cache key derivation (step list above).
|
|
1827
|
+
- The order or framing of existing key fields changes.
|
|
1828
|
+
- The on-disk layout changes (artifact format, entry naming).
|
|
1829
|
+
- The `CacheEntry` shape changes in a way that affects restore.
|
|
1830
|
+
- The SQLite schema changes in a way that affects existing rows
|
|
1831
|
+
(`SCHEMA_VERSION` also bumps in that case).
|
|
1832
|
+
|
|
1833
|
+
Not required when:
|
|
1834
|
+
|
|
1835
|
+
- Behavioural changes that adjust _which_ values flow into existing
|
|
1836
|
+
key components — those naturally produce different keys for
|
|
1837
|
+
affected tasks.
|
|
1838
|
+
- Changes to WHEN reads/writes fire (policy, prefetch, restore tier,
|
|
1839
|
+
background uploads) — key derivation and artifact bytes untouched.
|
|
1840
|
+
- Doc-only updates.
|
|
1841
|
+
- Refactors that don't change the bytes fed into the hash.
|
|
1842
|
+
|
|
1843
|
+
`tests/contract-stored-format.test.ts` holds the layout rows: it
|
|
1844
|
+
records the index's DDL beside `SCHEMA_VERSION` and a fixture
|
|
1845
|
+
artifact's entries, sidecar and digest beside `CACHE_VERSION`
|
|
1846
|
+
(`tests/contract/stored-format.json`), and fails when either layout
|
|
1847
|
+
moves under its recorded version. Every bump of either version
|
|
1848
|
+
regenerates the record
|
|
1849
|
+
(`VX_UPDATE_CONTRACT=1 bun test tests/contract-stored-format.test.ts`),
|
|
1850
|
+
which refuses a layout that moved without one.
|
|
1851
|
+
|
|
1852
|
+
The bump procedure has a dedicated skill at
|
|
1853
|
+
`.claude/skills/bump-cache-version/` (used as `/bump-cache-version`).
|
|
1854
|
+
Files touched, in the skill's order: `src/cache/key-fold.ts` (the constant),
|
|
1855
|
+
this doc (history), `docs/modules/cache.md` (the quoted version, and the
|
|
1856
|
+
key/entry shape if it changed), `CLAUDE.md` § Live invariants (the quoted
|
|
1857
|
+
version — the decision log it once named was retired 2026-09-02),
|
|
1858
|
+
`docs/STATUS.md` (the entry that says why the bump was needed, or why it
|
|
1859
|
+
was not), the cache tests, `tests/contract/stored-format.json`
|
|
1860
|
+
(regenerated, above), and
|
|
1861
|
+
`packages/vx-docs/src/content/docs/guides/upgrading.md` (the bump's
|
|
1862
|
+
breaking footer).
|
|
1863
|
+
|
|
1864
|
+
### History
|
|
1865
|
+
|
|
1866
|
+
- **v41 → v42**: the stored output became both streams in the order
|
|
1867
|
+
the run printed them (`orchestrator/output-log.ts`), so a hit replays
|
|
1868
|
+
stderr too. A v41 entry holds stdout alone.
|
|
1869
|
+
- **v40 → v41**: stored bytes wrong under an unchanged key (X-88). An
|
|
1870
|
+
output directory linked inside its project was never cleaned, so an
|
|
1871
|
+
entry could hold files a run of another key left there. The fix
|
|
1872
|
+
cannot reach an entry already saved that way.
|
|
1873
|
+
- **v39 → v40**: stored bytes wrong under an unchanged key (X-32, X-33,
|
|
1874
|
+
X-34). An additive task's entry a hit replayed over a file the task
|
|
1875
|
+
had removed, one that missed a same-size rewrite, and a runtime probe
|
|
1876
|
+
answered before its upstream wrote.
|
|
1877
|
+
- **v38 → v39**: stored bytes wrong under an unchanged key (A-61). A
|
|
1878
|
+
gitlink whose directory had lost its `.git` but held files listed
|
|
1879
|
+
none of them, so an entry built from them sits under the key the
|
|
1880
|
+
same directory empty derives.
|
|
1881
|
+
- **v37 → v38**: stored bytes wrong under an unchanged key (A-60). A
|
|
1882
|
+
file added under a clean filter that was later removed kept its LF
|
|
1883
|
+
index blob, git held it clean by its stat, and the key folded that
|
|
1884
|
+
blob while the task read the CRLF bytes. The blob-size check cannot
|
|
1885
|
+
reach an entry already saved that way.
|
|
1886
|
+
- **v36 → v37**: stored bytes wrong under an unchanged key (A-59).
|
|
1887
|
+
`git status` paired a deleted file with a similar unmerged path as its
|
|
1888
|
+
rename source and printed only `UU <path>`, so the file kept its index
|
|
1889
|
+
OID and the key folded it while the task ran without it. The fix
|
|
1890
|
+
(`--no-renames`) cannot reach an entry already saved that way.
|
|
1891
|
+
- **v35 → v36**: the container changes (L-19). Every artifact ends in a
|
|
1892
|
+
`.vx-sum` entry, a CRC-32 over the entries before it, and scan and
|
|
1893
|
+
restore refuse one whose sum is absent or wrong: a byte flipped in a
|
|
1894
|
+
raw zstd block restored as a hit, undetected. A v35 artifact carries no
|
|
1895
|
+
sum, so the bump retires them rather than refusing each on read.
|
|
1896
|
+
- **v34 → v35**: the container changes (item 943). The sidecar records
|
|
1897
|
+
the cache key the artifact was packed under, and ingest refuses bytes
|
|
1898
|
+
whose key is not the one it asked for, or that record none. Nothing
|
|
1899
|
+
tied an artifact to its key before: a remote layer that answered one
|
|
1900
|
+
key with another's bytes (a truncated or colliding key mapping, two
|
|
1901
|
+
namespaces mixed) replayed the other task's outputs under a green
|
|
1902
|
+
`cache-hit-remote`, probed with project `b` restoring project `a`'s
|
|
1903
|
+
`out.txt`. An artifact from v34 records no key, so the bump retires
|
|
1904
|
+
them rather than refusing each one on read.
|
|
1905
|
+
- **v33 → v34**: stored bytes wrong under a key a CORRECT derivation now
|
|
1906
|
+
produces (item 750). A root task that rewrote the lockfile mid-run
|
|
1907
|
+
(`pnpm install` without `--frozen-lockfile`) let a reader after it save
|
|
1908
|
+
a build made against the new install under the key of the lockfile the
|
|
1909
|
+
run started with, and that key is exactly what the fixed code derives
|
|
1910
|
+
when the tree is back on that lockfile. The fix stops new entries, not
|
|
1911
|
+
old ones. Probed, not argued: seeds A then B through the previous
|
|
1912
|
+
commit, then A twice through the fixed code under v33 — the last run
|
|
1913
|
+
started on lockfile A, installed A, and restored the build made against
|
|
1914
|
+
B; under v34 it built A. The item's other two edges poisoned nothing:
|
|
1915
|
+
their stale restores happened in runs that saved nothing (a read-only
|
|
1916
|
+
policy, a tainted upstream) or saved under a key the 743 re-check
|
|
1917
|
+
already guarded.
|
|
1918
|
+
- **v32 → v33**: stored bytes wrong under a key a CORRECT derivation now
|
|
1919
|
+
produces (item 743). The three stale-hit fixes of that item stop new
|
|
1920
|
+
poisoned entries, but not the ones already saved: a formatter's entry
|
|
1921
|
+
sits under the key of its unformatted input, an uncached `gen`'s
|
|
1922
|
+
consumer's under the key of the seed before `gen` ran, an edit-mid-run's
|
|
1923
|
+
under the key of the file before the edit — and each of those keys is
|
|
1924
|
+
exactly what the fixed code derives when the tree is back in that
|
|
1925
|
+
state. Not self-healing, unlike a fix whose old key no correct run
|
|
1926
|
+
derives. Probed, not argued: an entry saved by the previous commit's
|
|
1927
|
+
formatter, the input checked out unformatted, and the fixed code under
|
|
1928
|
+
v32 reported `up-to-date` and left it unformatted; under v33 it runs.
|
|
1929
|
+
- **v31 → v32**: stored bytes wrong under a key the fix does not change
|
|
1930
|
+
(item 739). A declared output whose name is not UTF-8 (`x\xffy`) came
|
|
1931
|
+
back from `Bun.Glob` decoded lossily as `x�y`, which names no file, so
|
|
1932
|
+
the save packed the tree without it and every hit restored a tree
|
|
1933
|
+
missing that file under an unchanged key. vx now refuses such a name
|
|
1934
|
+
by name; an artifact saved before the fix is the wrong bytes, so v31
|
|
1935
|
+
entries are not trusted.
|
|
1936
|
+
- **v30 → v31**: stored bytes wrong under a key the fix does not change
|
|
1937
|
+
(item 726), v30's shape for a sibling. A sandboxed task that declared
|
|
1938
|
+
`cache` was granted every workspace package linked in its
|
|
1939
|
+
`node_modules`, so it could import a sibling its key never folded and
|
|
1940
|
+
save what it built. The fix withholds that link unless the key answers
|
|
1941
|
+
for the package (§ Cross-project boundaries), so the next miss fails
|
|
1942
|
+
on the read; but an entry saved before it still hits when only the
|
|
1943
|
+
sibling changes. Probed, not argued: an entry saved by the previous
|
|
1944
|
+
commit, `@x/ui` edited, then this commit's run under v30 reported
|
|
1945
|
+
`up-to-date` and left the old `ui` in `dist/`; under v31 the same run
|
|
1946
|
+
misses and fails on the read, with the hint.
|
|
1947
|
+
|
|
1948
|
+
- **v29 → v30**: stored bytes wrong under a key the fix does not change
|
|
1949
|
+
(item 720). npm and Yarn link every workspace package at the root, the
|
|
1950
|
+
task's own included, and the sandbox granted every link target whole,
|
|
1951
|
+
so a sandboxed task was handed its own project back whatever its
|
|
1952
|
+
`allow.read` said. An undeclared read of its own file ran unreported
|
|
1953
|
+
and the result was saved under a key that never saw that file. The fix
|
|
1954
|
+
withholds the self-link, so the next miss fails on the read; but an
|
|
1955
|
+
entry saved before it hits for as long as only that file changes.
|
|
1956
|
+
|
|
1957
|
+
- **v28 → v29**: every key moves, and the bump is for the notice, not
|
|
1958
|
+
for wrong bytes (item 682). The key is a seed-chained xxHash3 fold,
|
|
1959
|
+
one step per field and per input file, and Bun's xxHash3 reads only
|
|
1960
|
+
the low 32 bits of its seed: a bare chain carried 32 bits of state, so
|
|
1961
|
+
two input sets whose running digests shared their low halves merged at
|
|
1962
|
+
the next step, a stale hit at about 2^-32 per step rather than 2^-64.
|
|
1963
|
+
A birthday search over 2^17 values of one env variable found two that
|
|
1964
|
+
gave one `Cache.key` in under a second. `xxh3` now feeds the seed
|
|
1965
|
+
forward (`xxHash3(part, seed) ^ seed`): states that share their low
|
|
1966
|
+
half keep their high-half difference through every later step. The
|
|
1967
|
+
old keys were wrong but their stored bytes are not, so the fix alone
|
|
1968
|
+
is self-healing; without the bump every entry would miss silently.
|
|
1969
|
+
|
|
1970
|
+
- **v27 → v28**: stored bytes wrong under a key the fix does not change —
|
|
1971
|
+
the v25/v26 shape (item 667). A bracket in a task glob became a literal
|
|
1972
|
+
character; before, `Bun.Glob` read it as a character class. An INPUT
|
|
1973
|
+
glob over a route directory folded the wrong files, and that fix is
|
|
1974
|
+
self-healing: the file set, and so the key, moves. An OUTPUT glob is
|
|
1975
|
+
not: `outputs: ['app/[id]/page.js']` folds as its text, which the fix
|
|
1976
|
+
leaves unchanged, and the entries under it hold nothing (or the class's
|
|
1977
|
+
namesake `app/i/page.js`). Proven through a real run: an entry written
|
|
1978
|
+
by v27 code, read by the fixed code, was a `cache-hit` that cleaned
|
|
1979
|
+
`app/[id]/page.js` and restored nothing; under v28 it is a miss, and
|
|
1980
|
+
the next hit restores the route.
|
|
1981
|
+
|
|
1982
|
+
- **v26 → v27**: the artifact CONTAINER changed, so the stored bytes
|
|
1983
|
+
under an unchanged key are no longer readable the same way — the
|
|
1984
|
+
layout case, not the wrong-bytes case. Packing moved from a `tar`
|
|
1985
|
+
subprocess (with a staged copy of every output and a per-host
|
|
1986
|
+
`--format=gnu` / `--format=gnutar` probe) to `Bun.Archive`, and the
|
|
1987
|
+
per-entry mode + millisecond mtime that tar headers carried natively
|
|
1988
|
+
now ride a `.vx-meta.json` sidecar. Measured on the real `Cache.save`
|
|
1989
|
+
path (300 outputs / 12 MB, min-of-5, interleaved arms against a
|
|
1990
|
+
`git worktree` of the previous commit): **158 ms → 11 ms** to pack,
|
|
1991
|
+
the artifact grows ~7 compressed bytes per output for the sidecar.
|
|
1992
|
+
Restore is indistinguishable up to ~12 MB, but on a 150 MB
|
|
1993
|
+
incompressible artifact it costs +28% time and +19% peak RSS
|
|
1994
|
+
(74 → 95 ms, 575 → 683 MB peak, fresh process per arm): `files()`
|
|
1995
|
+
returns entries that OWN their bytes, so they coexist with the
|
|
1996
|
+
decompressed tar where the old reader returned views into it. Peak is
|
|
1997
|
+
~4.5× artifact size, up from ~3.8×. The bump is mandatory rather than
|
|
1998
|
+
self-healing: a v26 artifact has no sidecar, so a v27 reader would
|
|
1999
|
+
restore its outputs mode-0644 and mtime-now — silently wrong on disk
|
|
2000
|
+
rather than a miss. Since 2026-09-03 the same layout is written and
|
|
2001
|
+
read by vx's own streaming tar code (no bump: the bytes are readable
|
|
2002
|
+
either way), and the peak above is history — save, ingest and restore
|
|
2003
|
+
hold one chunk now; see § Storage layout.
|
|
2004
|
+
|
|
2005
|
+
- **v25 → v26**: the same shape as v25 — stored bytes that are wrong
|
|
2006
|
+
under a key nothing about the fix changes — reached by a different
|
|
2007
|
+
producer. A task whose child was killed by a shutdown signal reports
|
|
2008
|
+
`aborted`, but `aborted` did not propagate to dependents the way
|
|
2009
|
+
`failed` and `skipped` do. So a dependent ran against the aborted
|
|
2010
|
+
task's PARTIAL outputs, succeeded, and cached what it had built.
|
|
2011
|
+
Because a dependent's key folds its upstream's INPUT key — and a
|
|
2012
|
+
signal changes no input — that entry sits under exactly the key a
|
|
2013
|
+
healthy run derives, and the next run replays it as a green hit with
|
|
2014
|
+
exit 0. Reproduced end to end: run 1 killed mid-write leaves
|
|
2015
|
+
`PARTIAL`, run 2 is fully healthy and still serves `PARTIAL` from
|
|
2016
|
+
cache.
|
|
2017
|
+
|
|
2018
|
+
Making `aborted` propagate stops new poison but cannot reach entries
|
|
2019
|
+
already written, and a `LayeredCache` uploads them — so the reach is
|
|
2020
|
+
a whole team's shared cache, not one developer's disk. That is what
|
|
2021
|
+
makes the trade worth it: one cold rebuild against a class of
|
|
2022
|
+
silently-wrong output. The interactive Ctrl-C path was never the
|
|
2023
|
+
vector (vx's handler exits before a dependent can cache); the
|
|
2024
|
+
reachable ones are an external `kill`, a supervisor, `docker stop`, a
|
|
2025
|
+
self-terminating script, and every `handleSignals: false` embedder —
|
|
2026
|
+
which includes `vx watch` and the distributed agent loop.
|
|
2027
|
+
|
|
2028
|
+
- **v24 → v25**: the ARTIFACT BYTES in every existing entry are wrong
|
|
2029
|
+
while the key addressing them is unchanged — the one situation a
|
|
2030
|
+
version bump exists for, and the opposite of the recent self-healing
|
|
2031
|
+
no-bump cases. Two defects on the pack/restore path, both silent data
|
|
2032
|
+
loss on an ordinary cache hit with no attacker involved:
|
|
2033
|
+
- **The executable bit was stripped from every cached output.**
|
|
2034
|
+
`packArtifact` staged each output with `Bun.write`, which creates
|
|
2035
|
+
the destination under the process umask and does NOT carry the
|
|
2036
|
+
source's mode, so the tar header recorded 0644 and the restore
|
|
2037
|
+
faithfully reproduced 0644. Any build emitting a CLI shim, a
|
|
2038
|
+
compiled binary or a generated script worked cold and broke warm —
|
|
2039
|
+
including this repo's own `build.bun.*` release binaries. Fixed by
|
|
2040
|
+
chmod-ing each staged copy to the source's mode.
|
|
2041
|
+
- **Outputs whose archive entry name exceeded 100 bytes were dropped
|
|
2042
|
+
on every restore.** POSIX ustar splits such a name into
|
|
2043
|
+
`prefix` + `name`; the reader read only `name`, which no longer
|
|
2044
|
+
starts with `outputs/`, so the file was neither restored nor
|
|
2045
|
+
indexed. Not self-healing: with no `output_files` row, the
|
|
2046
|
+
skip-restore check compared a truncated expectation against a
|
|
2047
|
+
truncated tree and agreed forever. Threshold is a
|
|
2048
|
+
project-relative output path of ~93 chars — ordinary for a modern
|
|
2049
|
+
bundler. Fixed by reading `prefix` (gated on the POSIX magic, since
|
|
2050
|
+
GNU headers reuse those bytes for atime) AND by packing
|
|
2051
|
+
`--format=gnu`, which carries long names in an `L` record.
|
|
2052
|
+
|
|
2053
|
+
The format switch also fixes a working build being reported as
|
|
2054
|
+
FAILED: ustar cannot split a single path COMPONENT over 100 bytes and
|
|
2055
|
+
exits non-zero ("file name is too long (cannot be split)"), which
|
|
2056
|
+
`packArtifact` raised _after_ the task had already succeeded. 120-char
|
|
2057
|
+
filenames are legal everywhere and routine in snapshot/fixture trees.
|
|
2058
|
+
|
|
2059
|
+
Shipped with three defence-in-depth fixes that needed no bump of their
|
|
2060
|
+
own: `restoreOutputs` now throws instead of returning quietly when the
|
|
2061
|
+
artifact is gone or cannot produce an output the index recorded (the
|
|
2062
|
+
caller has already wiped the declared outputs by then, so a quiet
|
|
2063
|
+
return reported a green hit over an emptied tree — reachable via a
|
|
2064
|
+
concurrent `vx cache prune`); the tar reader rejects an entry whose
|
|
2065
|
+
declared size runs past the end of the archive (`subarray` clamps, so
|
|
2066
|
+
it used to install short, NUL-padded content as a cache hit instead of
|
|
2067
|
+
degrading to a miss); and directory entries now get the same
|
|
2068
|
+
containment + realpath checks as file entries (`mkdir` follows a
|
|
2069
|
+
pre-existing symlink, so directories could be created outside the
|
|
2070
|
+
destination). No `SCHEMA_VERSION` bump — no table changed.
|
|
2071
|
+
|
|
2072
|
+
- **SCHEMA v23 → v24 (no `CACHE_VERSION` bump)**: `file_hashes` gains
|
|
2073
|
+
`ctime_ms` + `ino`. The memo keyed on `(mtime, size)` alone, and its
|
|
2074
|
+
row persists across runs, so any producer that preserves mtime —
|
|
2075
|
+
`tar -x`, `unzip`, `cp -p`, `rsync --times`, a `SOURCE_DATE_EPOCH`
|
|
2076
|
+
generator — got the previous run's digest for genuinely different
|
|
2077
|
+
bytes: a stale cache hit. `utimes` cannot suppress ctime
|
|
2078
|
+
unprivileged and an atomic write-then-rename changes the inode, so
|
|
2079
|
+
the two together close it (git's index keys on ctime+ino+dev for the
|
|
2080
|
+
same reason); both come free from the stat already taken. The key
|
|
2081
|
+
DERIVATION is unchanged — the memo simply stops answering wrongly —
|
|
2082
|
+
so an affected task's key moves from a wrong value to the right one:
|
|
2083
|
+
it misses once, re-runs, re-caches. Self-healing, never a wrong hit.
|
|
2084
|
+
Landed alongside two other stale-hit fixes that needed no schema
|
|
2085
|
+
change: the cache-miss path now marks the outputs `cleanOutputs`
|
|
2086
|
+
wiped (it was the only one of four sibling call sites that dropped
|
|
2087
|
+
the return, so a deleted output kept a live index OID and stayed in
|
|
2088
|
+
a consumer's input set), and `skip-worktree` / `assume-unchanged`
|
|
2089
|
+
entries no longer keep a trusted OID (they sit at stage 0 and
|
|
2090
|
+
`git status` reports nothing for them, so a sparse-checkout path
|
|
2091
|
+
that was absent from disk still counted as an input). The schema
|
|
2092
|
+
gate drops + recreates on the version mismatch (pre-alpha, no
|
|
2093
|
+
migration), so this costs one cold rebuild.
|
|
2094
|
+
- **SCHEMA v21 → v22 (no `CACHE_VERSION` bump)**: Tier-3 dashboard
|
|
2095
|
+
tables — `invocations` (one header row per `vx run` with command,
|
|
2096
|
+
git/CI/host context, tags, and run-level counts, recorded with `runs`
|
|
2097
|
+
via `Cache.recordRunBundle`) and `entry_inputs` (one row per cache-key
|
|
2098
|
+
component, keyed by the cache-ENTRY hash — the input-fingerprint
|
|
2099
|
+
moat). `entry_inputs` is written **inside the entry-save transaction,
|
|
2100
|
+
only on a cache miss** (`INSERT OR IGNORE`), so a warm all-cache-hit
|
|
2101
|
+
run writes nothing for the moat — Tier 3 has **zero warm-run cost**.
|
|
2102
|
+
The cache KEY derivation is unchanged: these tables persist components
|
|
2103
|
+
already fed to `Cache.key()` (captured via a pure side-channel,
|
|
2104
|
+
`CacheKeyInput.captureInto`, at the same fold sites — only on a miss),
|
|
2105
|
+
so existing artifacts stay valid and `CACHE_VERSION` stays `v24`. The
|
|
2106
|
+
schema gate drops + recreates on the version mismatch (pre-alpha, no
|
|
2107
|
+
migration).
|
|
2108
|
+
- **v23 → v24**: exclude `vx-lock.json` from the input file set
|
|
2109
|
+
globally (`ALWAYS_IGNORE` in `cache/inputs.ts`). The lockfile is
|
|
2110
|
+
committed, so git enumerates it, but it's vx's own frozen-config
|
|
2111
|
+
metadata — never a task input. Without this, a `vx lock` re-write
|
|
2112
|
+
busts every key on a project that globs the root lockfile (a broad
|
|
2113
|
+
`**/*` on the root project). Tasks whose `cache.inputs.files` never
|
|
2114
|
+
matched it derive byte-identical keys. No SCHEMA bump — only the
|
|
2115
|
+
hashed file set changed, not the key layout or on-disk format.
|
|
2116
|
+
|
|
2117
|
+
- **v22 → v23**: fold `cache.inputs.runtime` / `workspaceRuntime`
|
|
2118
|
+
command output into the key (two namespaced sections after
|
|
2119
|
+
env-values). The command _strings_ stay in the config hash (step 5);
|
|
2120
|
+
their combined trimmed stdout+stderr is resolved live at hash time
|
|
2121
|
+
and folded as `runtime-values:` (project-cwd) and `ws-runtime-values:`
|
|
2122
|
+
(root-cwd) sections, each `command\0output`. No SCHEMA bump — only
|
|
2123
|
+
`Cache.key` derivation gained two sections; the on-disk format is
|
|
2124
|
+
unchanged. Tasks declaring neither field fold a `:0` count for both
|
|
2125
|
+
and derive byte-identical keys to before the bump.
|
|
2126
|
+
|
|
2127
|
+
- **v21 → v22: pure-input transitive** (+ SCHEMA v21): reverted the v21
|
|
2128
|
+
output-fold. Downstream keys fold the upstream's **input key** (its
|
|
2129
|
+
own task hash) — a pure function of the filesystem, like Turbo/Nx.
|
|
2130
|
+
No output content participates in any cache key. **Early cutoff is
|
|
2131
|
+
gone**: an upstream that re-executes (comment edit, env change) but
|
|
2132
|
+
reproduces byte-identical output now still re-runs its dependents.
|
|
2133
|
+
This was a deliberate simplification — cutoff is rare in practice
|
|
2134
|
+
and not worth the cascade complexity (it forced output content into
|
|
2135
|
+
keys, which blocks any upfront/batched probe). **Multi-state is
|
|
2136
|
+
preserved**: branch ping-pong A→B→A still re-hits, because the
|
|
2137
|
+
upstream's _input_ differs per state and folds transitively into
|
|
2138
|
+
every dependent key. SCHEMA v21 drops the now-unused `outputs_hash`
|
|
2139
|
+
column; `CacheLayer.save` returns `void`.
|
|
2140
|
+
|
|
2141
|
+
- **v20 → v21: early cutoff** (+ SCHEMA v19, **reverted in v22**):
|
|
2142
|
+
downstream keys folded the upstream's output content identity
|
|
2143
|
+
(`outputsHash`) instead of its task hash. Removed — see v22.
|
|
2144
|
+
|
|
2145
|
+
- **v7 → v8** (PR #2): folded `forwardArgs` into the key for CLI
|
|
2146
|
+
argument-forwarding alignment.
|
|
2147
|
+
- **v8 → v9** (PR #3): `TaskConfig` shape changed — `exec` collapsed
|
|
2148
|
+
from an array to a single command, `tasks` nested under `run`.
|
|
2149
|
+
- **v9 → v10** (PR #7): on-disk layout switched from per-entry
|
|
2150
|
+
`meta.json` + `outputs/` directory to a workspace-wide `cache.db`
|
|
2151
|
+
(SQLite) plus output files directly at `<hash>/` and log files at
|
|
2152
|
+
`logs/<hash>.{stdout,stderr}`. Adds run history for `vx stats`.
|
|
2153
|
+
Removes the per-entry manifest.
|
|
2154
|
+
- **v10 → v11** (PR #19): analytics columns added to the `runs`
|
|
2155
|
+
table: `run_id` (UUIDv7), `cpu_ms`, `peak_rss_bytes`,
|
|
2156
|
+
`wallclock_start_ns` / `wallclock_end_ns`, `cache_hit`. All
|
|
2157
|
+
nullable; directly queryable via `sqlite3 cache.db`. The on-disk
|
|
2158
|
+
`<hash>/` layout itself was unchanged.
|
|
2159
|
+
- **v11 → v12** (PR #42): project's `package.json` bytes folded into
|
|
2160
|
+
every task's cache key implicitly. Matches Turbo / Nx "implicit
|
|
2161
|
+
dependencies" behavior — a `package.json` dep change invalidates
|
|
2162
|
+
the project's tasks even when `cache.inputs.files` is narrow and
|
|
2163
|
+
doesn't cover the file.
|
|
2164
|
+
- **v12 → v13** (PR #65): per-entry on-disk layout unified. Outputs
|
|
2165
|
+
moved from `<hash>/<rel>` (mixed with metadata) to
|
|
2166
|
+
`<hash>/outputs/<rel>`; stdout / stderr moved from the sibling
|
|
2167
|
+
`logs/<hash>.{stdout,stderr}` into `<hash>/stdout` and
|
|
2168
|
+
`<hash>/stderr`. Eviction collapses to a single `rm -rf <hash>/`.
|
|
2169
|
+
Also dropped the runner's `logs/<run_id>/<project>__<task>.{stdout,stderr}`
|
|
2170
|
+
dump — output is already streamed live, surfaced on the outcome
|
|
2171
|
+
object, and the cache entry covers successful runs; CI captures
|
|
2172
|
+
parent stdout natively. The duplicate sibling dump was pure
|
|
2173
|
+
redundancy.
|
|
2174
|
+
- **v13 → v14**: file enumeration switched from a `Bun.Glob` walker
|
|
2175
|
+
with our own `ignore`-library filter to `git ls-files --cached
|
|
2176
|
+
--others --exclude-standard`. Matches what Turborepo and Nx both do
|
|
2177
|
+
at the bottom of their hash pipelines. Side-effects user-visible:
|
|
2178
|
+
(a) nested `.gitignore` patterns are anchored to the gitignore's own
|
|
2179
|
+
directory, fixing the v13 footgun where `pkg/.gitignore: src/skip.ts`
|
|
2180
|
+
was misinterpreted as `<workspaceRoot>/src/skip.ts`; (b)
|
|
2181
|
+
`.git/info/exclude` and global excludes participate; (c)
|
|
2182
|
+
untracked-but-not-ignored files enter inputs immediately (no
|
|
2183
|
+
`git add` required). (The non-git fallback walker was later removed
|
|
2184
|
+
entirely — vx hard-requires git; a non-repo workspace gets a clean
|
|
2185
|
+
`UserError` telling the user to `git init`.)
|
|
2186
|
+
- **v14 → v15**: cache-key hash swapped from SHA-256 (via
|
|
2187
|
+
`Bun.CryptoHasher`) to xxHash3 (via `Bun.hash.xxHash3`). Key strings
|
|
2188
|
+
shrink from 64 hex chars to 16, matching Turbo's xxh64 output width;
|
|
2189
|
+
derivation is ~5× faster, dominating the cache-warm path that
|
|
2190
|
+
hashes hundreds of input files. xxHash3 has no streaming Hasher
|
|
2191
|
+
API, so `Cache.key()` chains via the seed parameter (each
|
|
2192
|
+
`xxh3(part, prevDigest)` folds one field into the running digest)
|
|
2193
|
+
and `hashFileFromDisk` reads the whole file before hashing — fine
|
|
2194
|
+
for source files (typically < 1MB each); the throughput win
|
|
2195
|
+
outweighs the memory hit. `SCHEMA_VERSION` bumps to `v15` at the
|
|
2196
|
+
same time (PR #86 already took `v14` for the tar.zst artifact
|
|
2197
|
+
layout): the `file_hashes.sha256` column is renamed to
|
|
2198
|
+
`content_hash`, and the schema-mismatch path now `DROP`s the stale
|
|
2199
|
+
tables before `CREATE TABLE IF NOT EXISTS` runs so the rename
|
|
2200
|
+
actually takes effect on existing DBs. Non-cryptographic by design
|
|
2201
|
+
— cache keys never need collision resistance against an adversary,
|
|
2202
|
+
just uniqueness across honest inputs.
|
|
2203
|
+
- **v15 → v16** (PR #86 series): artifact storage moved to a single
|
|
2204
|
+
compressed `<hash>.tar.zst` per entry; the manifest.json entry was
|
|
2205
|
+
dropped (file fingerprints live in the `output_files` table).
|
|
2206
|
+
- **v16 → v17**: artifact narrowed to exactly `stdout` +
|
|
2207
|
+
`outputs/<rel>` — no `meta.json`, no stderr (only successful runs
|
|
2208
|
+
are cached and their stderr is near-always empty noise). Local and
|
|
2209
|
+
remote layers transport the same bytes end-to-end; entry metadata
|
|
2210
|
+
lives solely in SQLite.
|
|
2211
|
+
- **v17 → v18**: env-value folding in `Cache.key()` switched its
|
|
2212
|
+
name/value delimiter from `=` to `\0`. `${n}=${v}` was ambiguous —
|
|
2213
|
+
`("A", "B=C")` and `("A=B", "C")` folded identical bytes. Env names
|
|
2214
|
+
containing `=` are unreachable from a real POSIX environ, so this
|
|
2215
|
+
is contract hardening rather than a field bug, but the key
|
|
2216
|
+
derivation's stated invariant is unambiguous part boundaries —
|
|
2217
|
+
now it holds everywhere (file inputs already used `\0`).
|
|
2218
|
+
- **v18 → v19**: `'^task'` dependsOn expansion switched from
|
|
2219
|
+
transitive-deps to nearest-holder frontier semantics (Turbo/Nx
|
|
2220
|
+
direct-deps parity, plus vx's sparse bridging through deps that
|
|
2221
|
+
don't declare the task). Task graphs lose the redundant deep edges,
|
|
2222
|
+
so the filtered-upstream-hash set (step 10) shrinks for any task
|
|
2223
|
+
whose deps chain `'^task'` themselves — same inputs now derive a
|
|
2224
|
+
different key. Reachability/ordering is unchanged whenever holders
|
|
2225
|
+
chain `'^task'` (the universal pattern); a holder that doesn't is
|
|
2226
|
+
now the documented stopping point. No on-disk format change.
|
|
2227
|
+
- **v19 → v20**: input-file content hashes switched from xxh3 to
|
|
2228
|
+
**git blob OIDs** (Turbo's technique). The bulk enumeration spawn
|
|
2229
|
+
became `git ls-files -s --others --exclude-standard` — `-s` lines
|
|
2230
|
+
carry `<mode> <oid> <stage>\t<path>` for tracked files, so one
|
|
2231
|
+
spawn yields the file list AND the index OIDs; a second
|
|
2232
|
+
`git status --porcelain -z` spawn prunes OIDs for paths whose
|
|
2233
|
+
working tree diverges from the index (renames drop both sides;
|
|
2234
|
+
stage>0 conflict entries never get one; symlinks did not either
|
|
2235
|
+
until 2026-09-09, when the fallback started hashing a link the way
|
|
2236
|
+
the index does). A clean
|
|
2237
|
+
tree's key derivation does zero reads / stats / SQLite per file.
|
|
2238
|
+
Everything else falls back to `Cache.hashFile`, which now computes
|
|
2239
|
+
the identical blob OID in-process (object format auto-detected via
|
|
2240
|
+
`git rev-parse --show-object-format`, sha1 default) behind the
|
|
2241
|
+
existing mtime+size memo. `SCHEMA_VERSION` bumps to `v18` in the
|
|
2242
|
+
same change: pre-v20 `file_hashes.content_hash` rows store 16-hex
|
|
2243
|
+
xxh3 digests that must not leak into the OID domain through the
|
|
2244
|
+
memo. File-set visibility semantics are unchanged (verified: the
|
|
2245
|
+
`-s --others` path set is identical to `--cached --others`,
|
|
2246
|
+
including staged-but-deleted files and per-stage conflict
|
|
2247
|
+
duplicates).
|