@vzn/vx 0.0.14 → 0.0.16
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +107 -123
- package/package.json +5 -5
- package/src/cache/layered-cache.ts +41 -0
- package/src/orchestrator/remote-prefetch.ts +19 -4
package/README.md
CHANGED
|
@@ -15,9 +15,6 @@ architecture, caching, and the full CLI / config reference.
|
|
|
15
15
|
```sh
|
|
16
16
|
# From npm — ships the prebuilt standalone binary (no Bun required):
|
|
17
17
|
npm install -g @vzn/vx # or: pnpm add -g @vzn/vx · bun add -g @vzn/vx
|
|
18
|
-
|
|
19
|
-
# Or the zero-dependency install script (no Node/npm needed):
|
|
20
|
-
curl -fsSL https://raw.githubusercontent.com/vznjs/vx/main/install.sh | sh
|
|
21
18
|
```
|
|
22
19
|
|
|
23
20
|
```ts
|
|
@@ -52,58 +49,6 @@ vx watch lint # re-run on file changes
|
|
|
52
49
|
vx run build --dry # show the plan, don't execute
|
|
53
50
|
```
|
|
54
51
|
|
|
55
|
-
## Beyond a task runner — what shipped in the 2026-06 platform arc
|
|
56
|
-
|
|
57
|
-
vx now ships an **open platform**, not just a CLI. Every surface
|
|
58
|
-
below is built into the binary; no external services required.
|
|
59
|
-
|
|
60
|
-
```sh
|
|
61
|
-
vx mcp # Model Context Protocol server (stdio)
|
|
62
|
-
# — Claude Code / Cursor / Continue.dev talk to vx as a typed tool
|
|
63
|
-
|
|
64
|
-
npm i -g @vzn/vx-cloud # the service CLI — also a standalone binary, no Bun
|
|
65
|
-
vx-cloud serve --ui --open # team backend + embedded dashboard (token-authenticated)
|
|
66
|
-
vx-cloud agent --url https://ci.acme.dev # join a distributed-execution pool (Nx-Agents-style DTE)
|
|
67
|
-
vx-cloud connect https://ci.acme.dev # one-URL analytics + remote cache for every `vx run`
|
|
68
|
-
```
|
|
69
|
-
|
|
70
|
-
### Open platform highlights
|
|
71
|
-
|
|
72
|
-
- **Provider-neutral core.** `vx` runs tasks and nothing else — the
|
|
73
|
-
dashboard, remote cache, distribution, and telemetry all arrive via
|
|
74
|
-
PLUGINS declared in `vx.workspace.ts`. `@vzn/vx-cloud` is the
|
|
75
|
-
first-party one; anyone can write another against the same seams
|
|
76
|
-
(backend / cache / telemetry).
|
|
77
|
-
- **MCP for AI agents, twice.** `vx mcp` (stdio, reads your real
|
|
78
|
-
`cache.db`: `getCacheStats`, `getRunHistory`, `explainCacheKey`,
|
|
79
|
-
`whyDidThisRerun`) and `POST /mcp` on any `vx-cloud serve` — local
|
|
80
|
-
or hosted, behind the bearer token.
|
|
81
|
-
- **Provable cache correctness.** `vx run --verify` re-runs each
|
|
82
|
-
cacheable task and byte-compares outputs; `--verify=inputs` runs it
|
|
83
|
-
sandboxed against the declared inputs and names any undeclared read.
|
|
84
|
-
The only runner that PROVES a cache entry safe instead of hoping.
|
|
85
|
-
- **Distributed CI (DTE).** `vx-cloud agent` joins a session-keyed
|
|
86
|
-
pool; a `vx run` fans tasks out across same-commit agents, outputs
|
|
87
|
-
propagate through the shared artifact store, and a standing pool
|
|
88
|
-
multiplexes concurrent runs with fair scheduling.
|
|
89
|
-
- **Remote cache, one URL.** `vx-cloud connect <url>` wires analytics
|
|
90
|
-
ingest AND a Turborepo-wire-compatible artifact store (HMAC signing,
|
|
91
|
-
trusted/untrusted tiers derived from the token). Third-party Turbo
|
|
92
|
-
cache servers work via two env vars.
|
|
93
|
-
- **OTel CI/CD spans.** Declare `otel()` from `@vzn/vx-otel` in
|
|
94
|
-
`vx.workspace.ts` — OTLP/HTTP traces + metrics with ZERO
|
|
95
|
-
OpenTelemetry SDK dependency; runs flow to Grafana / Honeycomb /
|
|
96
|
-
Datadog, including `--verify` hermeticity verdicts as span status.
|
|
97
|
-
- **Dashboard embedded in the `vx-cloud` binary.** Runs, flamegraphs,
|
|
98
|
-
live run cockpit with the task DAG, per-task logs + artifacts,
|
|
99
|
-
cache-key diffs ("why did this re-run?"), flaky-task detection.
|
|
100
|
-
Nothing to build or install; Docker image on ghcr for hosting.
|
|
101
|
-
- **Predictive scheduling.** Opt in with `predictive: true` — the
|
|
102
|
-
scheduler reads run history and starts the longest expected critical
|
|
103
|
-
path first.
|
|
104
|
-
|
|
105
|
-
Each surface lives behind a design doc under `docs/design/`.
|
|
106
|
-
|
|
107
52
|
## A cache that actually understands your build
|
|
108
53
|
|
|
109
54
|
Every task runner caches. vx caches _correctly_ — and stops work
|
|
@@ -138,14 +83,10 @@ with the invariant that keeps it valid —
|
|
|
138
83
|
`--verify=inputs` sandboxes it against the declared inputs and fails
|
|
139
84
|
loud naming any undeclared read (completeness). No other runner can
|
|
140
85
|
prove a cache entry safe.
|
|
141
|
-
- **
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
bearer token — a fork-PR artifact can never feed a trusted build.
|
|
146
|
-
- **Corruption can't go live.** Artifacts are validated before they
|
|
147
|
-
enter the store (zstd-bomb and oversize downloads refused); bad
|
|
148
|
-
bytes degrade to a cache miss, never a crash.
|
|
86
|
+
- **Corruption can't go live.** A remote artifact is verified against
|
|
87
|
+
its content digest and validated before it enters the store
|
|
88
|
+
(zstd-bomb and oversize downloads refused); bad bytes degrade to a
|
|
89
|
+
cache miss, never a wrong hit and never a crash.
|
|
149
90
|
- **Clean exits.** SIGINT/SIGTERM reap every child process — no
|
|
150
91
|
orphaned dev servers in CI.
|
|
151
92
|
- **Readiness you can bound.** Persistent tasks gate downstream work
|
|
@@ -183,37 +124,47 @@ TypeScript config with real imports · task graph with `^task`
|
|
|
183
124
|
resolution that bridges packages without the task · multi-task runs
|
|
184
125
|
with one shared graph · pnpm-style filters and `--affected` ·
|
|
185
126
|
watch mode · `--dry` / `--graph` plans · persistent dev servers ·
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
`vx
|
|
190
|
-
|
|
191
|
-
|
|
127
|
+
retries, timeouts, `--continue` modes · per-layer cache control
|
|
128
|
+
(`--cache=local:r,remote:`) · `vx info`, `--summarize`, `--profile`
|
|
129
|
+
Chrome traces, `--report` · `vx cache prune` with TTL and size caps ·
|
|
130
|
+
`vx migrate` from turbo.json or an Nx graph.
|
|
131
|
+
|
|
132
|
+
## Extensible by design — the core is provider-neutral
|
|
133
|
+
|
|
134
|
+
`vx` runs tasks and nothing else. A dashboard, a remote cache,
|
|
135
|
+
distributed execution, and telemetry export all arrive through
|
|
136
|
+
**plugins** declared in `vx.workspace.ts`, filling three seams the
|
|
137
|
+
core exposes — **backend** (where a task runs), **cache** (a remote
|
|
138
|
+
layer behind the local one), and **telemetry** (a read-only stream of
|
|
139
|
+
the versioned `TelemetryRecord` / `RunSummaryRecord` contract). Core
|
|
140
|
+
depends on none of them; the arrow only ever points plugin → core.
|
|
141
|
+
First-party plugins ship the OpenTelemetry exporter
|
|
142
|
+
([`@vzn/vx-otel`](https://www.npmjs.com/package/@vzn/vx-otel), OTLP
|
|
143
|
+
traces + metrics with zero OTel-SDK deps) and **vx Cloud** (below) —
|
|
144
|
+
and the same seams are open to anyone.
|
|
192
145
|
|
|
193
146
|
## How it compares
|
|
194
147
|
|
|
195
|
-
| | vx | Turborepo | Nx
|
|
196
|
-
| ------------------------- | --------------------------------------------------------------------- | ------------------------------ |
|
|
197
|
-
| Fully cached, 100 pkgs¹ | **144 ms** | 279 ms | 583+ ms
|
|
198
|
-
| Config | TypeScript, evaluated into the cache key | JSON (static) | JSON (static)
|
|
199
|
-
| Output ownership | **Strict** — wiped before exec AND restore | Additive (stale files survive) | Additive
|
|
200
|
-
| Clean-tree hashing | **Zero reads** (git index OIDs) | git OIDs | re-hash / daemon
|
|
201
|
-
| Daemon required for speed | **No** | Optional | Yes
|
|
202
|
-
|
|
|
203
|
-
|
|
|
204
|
-
|
|
|
205
|
-
|
|
|
206
|
-
|
|
|
207
|
-
|
|
|
208
|
-
| Self-hosted
|
|
209
|
-
|
|
|
210
|
-
| Predictive scheduling | **Yes** (opt-in: `predictive: true`) | No | No |
|
|
211
|
-
| OTel CI/CD spans | **Yes** — `otel()` plugin, zero OTel-SDK deps | No | Paid |
|
|
212
|
-
| Install | **Single binary** — npm or 1 curl line, no Node/Bun needed | npm + Node | npm + Node |
|
|
148
|
+
| | vx | Turborepo | Nx |
|
|
149
|
+
| ------------------------- | --------------------------------------------------------------------- | ------------------------------ | ---------------- |
|
|
150
|
+
| Fully cached, 100 pkgs¹ | **144 ms** | 279 ms | 583+ ms |
|
|
151
|
+
| Config | TypeScript, evaluated into the cache key | JSON (static) | JSON (static) |
|
|
152
|
+
| Output ownership | **Strict** — wiped before exec AND restore | Additive (stale files survive) | Additive |
|
|
153
|
+
| Clean-tree hashing | **Zero reads** (git index OIDs) | git OIDs | re-hash / daemon |
|
|
154
|
+
| Daemon required for speed | **No** | Optional | Yes |
|
|
155
|
+
| Per-task sandbox | **Yes** — kernel-level, opt-in | No | No |
|
|
156
|
+
| Provable cache safety | **Yes** — `--verify` (determinism) + `--verify=inputs` (completeness) | No | No |
|
|
157
|
+
| MCP server for AI agents | **Yes** — `vx mcp` (stdio, reads your local cache) | No | No |
|
|
158
|
+
| Plugin API | **Yes** — backend / cache / telemetry seams | No | Yes (TS-tied) |
|
|
159
|
+
| Predictive scheduling | **Yes** (opt-in: `predictive: true`) | No | No |
|
|
160
|
+
| OTel CI/CD spans | **Yes** — `otel()` plugin, zero OTel-SDK deps | No | Paid |
|
|
161
|
+
| Self-hosted platform² | **Yes** — dashboard, distributed CI, remote cache; Docker compose | Vercel-only cache | Paid (Nx Cloud) |
|
|
162
|
+
| Install | **Single binary** — npm or 1 curl line, no Node/Bun needed | npm + Node | npm + Node |
|
|
213
163
|
|
|
214
164
|
¹ Wall-clock, direct binaries, same machine and workspace — full
|
|
215
165
|
methodology and more scenarios in
|
|
216
|
-
[`docs/benchmarks.md`](docs/benchmarks.md).
|
|
166
|
+
[`docs/benchmarks.md`](docs/benchmarks.md). ² Optional, self-hosted,
|
|
167
|
+
OSS — see [vx Cloud](#vx-cloud).
|
|
217
168
|
|
|
218
169
|
## Switching from another runner
|
|
219
170
|
|
|
@@ -259,25 +210,69 @@ Differences to know:
|
|
|
259
210
|
- vx requires `cache.inputs.files` when caching is enabled (no default `$TURBO_DEFAULT$`).
|
|
260
211
|
- vx defaults caching **off**; opt in per task by adding the `cache` block.
|
|
261
212
|
- Persistent tasks: `persistent: { readyWhen: 'regex' }` (Turbo uses just `persistent: true`).
|
|
262
|
-
- Remote
|
|
213
|
+
- Remote caching is a plugin, not a built-in — connect one and every `vx run` reads through it.
|
|
263
214
|
|
|
264
215
|
Side-by-side feature matrix + every known gap: [`docs/comparison.md`](./docs/comparison.md).
|
|
265
216
|
|
|
217
|
+
## vx Cloud
|
|
218
|
+
|
|
219
|
+
An **optional, self-hosted CI platform** — the first-party service you
|
|
220
|
+
run yourself. It is a fully independent app (accounts, roles,
|
|
221
|
+
organizations, teams, multiple workspaces), with **Postgres** as the
|
|
222
|
+
system of record and an **S3-compatible bucket** for artifacts; the
|
|
223
|
+
controller keeps zero artifact bytes at rest. Core `vx` never depends
|
|
224
|
+
on it — it connects to a deployment through the plugin seams above.
|
|
225
|
+
|
|
226
|
+
What a connected platform adds:
|
|
227
|
+
|
|
228
|
+
- **Dashboard.** Runs, flamegraphs, a live run cockpit with the task
|
|
229
|
+
DAG, per-task logs + artifacts, cache-key diffs ("why did this
|
|
230
|
+
re-run?"), and flaky-task detection with the concrete fix.
|
|
231
|
+
- **Remote cache.** The vx-native `/v1/cache` wire, tenant-partitioned
|
|
232
|
+
per org/workspace, with **trusted / untrusted tiers derived from the
|
|
233
|
+
token** — a fork-PR artifact can never feed a trusted build — and an
|
|
234
|
+
always-on integrity digest the client verifies.
|
|
235
|
+
- **Distributed CI (DTE).** Agents form a session-keyed pool; a
|
|
236
|
+
`vx run` fans tasks out across same-commit agents, outputs propagate
|
|
237
|
+
through the shared artifact store, and a standing pool multiplexes
|
|
238
|
+
concurrent runs with fair scheduling.
|
|
239
|
+
- **MCP over the platform.** `POST /mcp` (Postgres-backed, behind the
|
|
240
|
+
token) exposes the analytics as typed tools for AI agents — alongside
|
|
241
|
+
the core `vx mcp` stdio server that reads your local cache.
|
|
242
|
+
|
|
243
|
+
Deploy the stack (app + Postgres + object storage) with Docker
|
|
244
|
+
Compose, then connect any workspace with one URL and a minted token:
|
|
245
|
+
|
|
246
|
+
```sh
|
|
247
|
+
# Deploy — open the URL, register the first admin, mint a CI token under Admin → Tokens:
|
|
248
|
+
VX_CLOUD_SECRET=$(openssl rand -hex 32) \
|
|
249
|
+
docker compose -f packages/cloud/deploy/docker-compose.yml up
|
|
250
|
+
|
|
251
|
+
# Connect a workspace (remote cache + analytics + distribution):
|
|
252
|
+
vx-cloud connect https://ci.acme.dev --token vxc_…
|
|
253
|
+
# …or set VX_CLOUD_URL + VX_CLOUD_TOKEN in the environment.
|
|
254
|
+
```
|
|
255
|
+
|
|
256
|
+
Install the platform CLI with `npm i -g @vzn/vx-cloud` (a standalone
|
|
257
|
+
binary, or the `ghcr.io/vznjs/vx-cloud` image). Full setup, RBAC,
|
|
258
|
+
distributed CI, and the wire protocol live in the **vx Cloud** section
|
|
259
|
+
of the [documentation site](https://vznjs.github.io/vx/).
|
|
260
|
+
|
|
266
261
|
## Architecture (one paragraph)
|
|
267
262
|
|
|
268
263
|
`bin.ts → cli/index.ts` dispatches subcommands.
|
|
269
264
|
`orchestrator/run.ts:run()` calls `prepareRun()` which discovers the
|
|
270
265
|
workspace, loads configs, builds the package + task graph, opens the
|
|
271
|
-
cache (local SQLite + optional remote layer), and installs plugins
|
|
266
|
+
cache (local SQLite + an optional remote layer), and installs plugins
|
|
272
267
|
from `vx.workspace.ts`. The two-tier scheduler runs the graph in
|
|
273
268
|
topological order with bounded concurrency (confirmed cache hits
|
|
274
269
|
restore ahead of their deps); each task hits the cache (hash → get →
|
|
275
270
|
restore on hit; spawn → save on miss) or short-circuits as a group /
|
|
276
271
|
persistent. Every observation flows through one event bus — the
|
|
277
|
-
terminal renderer subscribes directly, and
|
|
272
|
+
terminal renderer subscribes directly, and plugins receive the
|
|
278
273
|
versioned telemetry contract (`TelemetryRecord` / `RunSummaryRecord`)
|
|
279
|
-
— that's how
|
|
280
|
-
Core never imports a plugin; the arrow only points plugin → core.
|
|
274
|
+
— that's how telemetry and cache plugins export without core knowing
|
|
275
|
+
them. Core never imports a plugin; the arrow only points plugin → core.
|
|
281
276
|
Every module has a docs page; every interface is a swappable seam.
|
|
282
277
|
|
|
283
278
|
Read [`docs/architecture.md`](./docs/architecture.md) for the module
|
|
@@ -285,9 +280,8 @@ map; the design record lives under [`docs/design/`](./docs/design/).
|
|
|
285
280
|
|
|
286
281
|
## Documentation
|
|
287
282
|
|
|
288
|
-
Full technical docs live under [`docs/`](./docs/)
|
|
289
|
-
|
|
290
|
-
**Core**
|
|
283
|
+
Full technical docs live under [`docs/`](./docs/) and on the
|
|
284
|
+
[documentation site](https://vznjs.github.io/vx/):
|
|
291
285
|
|
|
292
286
|
- [`docs/architecture.md`](./docs/architecture.md) — module map + data flow
|
|
293
287
|
- [`docs/schema.md`](./docs/schema.md) — every config field
|
|
@@ -296,42 +290,32 @@ Full technical docs live under [`docs/`](./docs/):
|
|
|
296
290
|
- [`docs/cli.md`](./docs/cli.md) — every flag
|
|
297
291
|
- [`docs/comparison.md`](./docs/comparison.md) — Turbo / Nx / vite-task feature matrix
|
|
298
292
|
- [`docs/modules/`](./docs/modules/) — one reference page per source module
|
|
293
|
+
- The **vx Cloud** section of the [docs site](https://vznjs.github.io/vx/) — deploy, RBAC, dashboard, distributed CI, wire protocol
|
|
299
294
|
|
|
300
|
-
|
|
301
|
-
|
|
302
|
-
- [`architecture-north-star-2026-06.md`](./docs/design/architecture-north-star-2026-06.md) — the unified vision
|
|
303
|
-
- [`architecture-review-2026-06.md`](./docs/design/architecture-review-2026-06.md) — review + applied checklist
|
|
304
|
-
- [`wire-protocol-2026-06.md`](./docs/design/wire-protocol-2026-06.md) — JSON-RPC 2.0 + OTel envelope (shipped)
|
|
305
|
-
- [`distributed-ci-2026-06.md`](./docs/design/distributed-ci-2026-06.md) — original coordinator/worker (superseded by `vx-cloud agent` DTE — `distributed-execution-2026-07.md`)
|
|
306
|
-
- [`vx-cloud-2026-06.md`](./docs/design/vx-cloud-2026-06.md) — original CF cloud (superseded by the standalone `vx-cloud` service)
|
|
307
|
-
- [`extension-protocol-2026-06.md`](./docs/design/extension-protocol-2026-06.md) — subscriber/inspector/driver/plugin (Phase 1 shipped)
|
|
308
|
-
- [`predictive-execution-2026-06.md`](./docs/design/predictive-execution-2026-06.md) — history-aware scheduling (Phase A-B shipped)
|
|
309
|
-
- [`docs/progress/implementation-log-2026-06.md`](./docs/progress/implementation-log-2026-06.md) — phase-by-phase narrative
|
|
295
|
+
The full design record (including the platform-pivot notes) lives
|
|
296
|
+
under [`docs/design/`](./docs/design/).
|
|
310
297
|
|
|
311
298
|
## Status
|
|
312
299
|
|
|
313
300
|
**Pre-alpha.** The schema is settling; we bump `CACHE_VERSION` rather
|
|
314
|
-
than maintain back-compat. **1,
|
|
301
|
+
than maintain back-compat. **1,600+ tests (core + packages); CI green
|
|
315
302
|
on every commit**; the project dogfoods itself (`vx run ci`).
|
|
316
303
|
Published on npm: [`@vzn/vx`](https://www.npmjs.com/package/@vzn/vx)
|
|
317
|
-
and `@vzn/vx-cloud
|
|
304
|
+
(a prebuilt standalone binary) and `@vzn/vx-cloud`.
|
|
318
305
|
|
|
319
306
|
Production readiness for the **core task runner**: the semantics are
|
|
320
|
-
solid. The main operational rough edge
|
|
321
|
-
|
|
322
|
-
|
|
323
|
-
|
|
324
|
-
|
|
|
325
|
-
|
|
|
326
|
-
|
|
|
327
|
-
| `vx
|
|
328
|
-
|
|
|
329
|
-
|
|
|
330
|
-
|
|
|
331
|
-
|
|
|
332
|
-
| Predictive scheduling | **shippable as opt-in** | gated on `predictive: true` + observed data |
|
|
333
|
-
| Dashboard SPA (embedded in `vx-cloud`) | **iterating** | runs/logs/DAG/flaky live; data-first entity redesign in progress |
|
|
334
|
-
| OTel export (`@vzn/vx-otel`) | **shippable** | declare `otel()` in `vx.workspace.ts`; OTLP traces + metrics |
|
|
307
|
+
solid; it is dogfooded continuously. The main operational rough edge
|
|
308
|
+
is Windows (unsupported).
|
|
309
|
+
|
|
310
|
+
| Surface | Maturity | Notes |
|
|
311
|
+
| ---------------------------------------------- | ------------------------------ | ----------------------------------------------------------------- |
|
|
312
|
+
| Core task runner + caching | **production-ready** | dogfooded continuously; 1,200+ core tests, all green |
|
|
313
|
+
| `vx run --verify` (provable cache correctness) | **shippable** | determinism + input-completeness proofs; CI-gate recipe in docs |
|
|
314
|
+
| `vx mcp` | **shippable** | live cache.db tools over stdio |
|
|
315
|
+
| Plugin API (backend / cache / telemetry) | **shippable** | crash-isolated; the cloud + OTel plugins are ordinary plugins |
|
|
316
|
+
| Predictive scheduling | **shippable as opt-in** | gated on `predictive: true` + observed data |
|
|
317
|
+
| OTel export (`@vzn/vx-otel`) | **shippable** | declare `otel()` in `vx.workspace.ts`; OTLP traces + metrics |
|
|
318
|
+
| vx Cloud platform (self-hosted) | **shippable for self-hosting** | accounts/RBAC/orgs on Postgres, S3 artifacts, dashboard, DTE, MCP |
|
|
335
319
|
|
|
336
320
|
## Development
|
|
337
321
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@vzn/vx",
|
|
3
|
-
"version": "0.0.
|
|
3
|
+
"version": "0.0.16",
|
|
4
4
|
"description": "An open, extensible monorepo task runner.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"exports": {
|
|
@@ -17,10 +17,10 @@
|
|
|
17
17
|
"node": ">=18"
|
|
18
18
|
},
|
|
19
19
|
"optionalDependencies": {
|
|
20
|
-
"@vzn/vx-linux-x64": "0.0.
|
|
21
|
-
"@vzn/vx-linux-arm64": "0.0.
|
|
22
|
-
"@vzn/vx-darwin-x64": "0.0.
|
|
23
|
-
"@vzn/vx-darwin-arm64": "0.0.
|
|
20
|
+
"@vzn/vx-linux-x64": "0.0.16",
|
|
21
|
+
"@vzn/vx-linux-arm64": "0.0.16",
|
|
22
|
+
"@vzn/vx-darwin-x64": "0.0.16",
|
|
23
|
+
"@vzn/vx-darwin-arm64": "0.0.16"
|
|
24
24
|
},
|
|
25
25
|
"dependencies": {
|
|
26
26
|
"@anthropic-ai/sandbox-runtime": "^0.0.56"
|
|
@@ -47,6 +47,15 @@ import { FULL_CACHE_POLICY } from './cache.js'
|
|
|
47
47
|
export interface RemoteCacheLayer {
|
|
48
48
|
/** Existence probe (drives the plan path's `--dry` remote prediction). */
|
|
49
49
|
has(hash: string): Promise<boolean>
|
|
50
|
+
/**
|
|
51
|
+
* Optional batch existence probe: given N hashes, return the subset stored
|
|
52
|
+
* remotely in ONE round-trip. Lets the prefetch pass collapse N per-hash
|
|
53
|
+
* network probes into one, then fetch only the hits. A remote that can't
|
|
54
|
+
* batch omits this method (or returns `null`) and the layer falls back to
|
|
55
|
+
* the per-hash path. Never throws for control flow — `null` means "no batch
|
|
56
|
+
* info; use per-hash".
|
|
57
|
+
*/
|
|
58
|
+
hasMany?(hashes: readonly string[]): Promise<Set<string> | null>
|
|
50
59
|
/** Fetch an artifact's bytes; `null` = miss. `durationMs` is the
|
|
51
60
|
* producing task's duration when the wire carries it. */
|
|
52
61
|
get(hash: string): Promise<{ body: ArrayBuffer; durationMs: number | undefined } | null>
|
|
@@ -126,6 +135,38 @@ export class LayeredCache implements CacheLayer {
|
|
|
126
135
|
return await this.pullFromRemote(hash, ctx)
|
|
127
136
|
}
|
|
128
137
|
|
|
138
|
+
/**
|
|
139
|
+
* Batch existence probe over the remote layer — the subset of `hashes`
|
|
140
|
+
* present remotely, in one round-trip, or `null` when the remote can't
|
|
141
|
+
* batch (no `hasMany`, reads disabled, or an error). The prefetch pass uses
|
|
142
|
+
* this to fetch only the hits and to pre-mark the misses (`markRemoteAbsent`)
|
|
143
|
+
* so their lazy `get` skips the network. Never throws — a batch failure
|
|
144
|
+
* degrades to "no batch info" and the caller falls back to per-hash.
|
|
145
|
+
*/
|
|
146
|
+
async remoteHasMany(hashes: readonly string[]): Promise<Set<string> | null> {
|
|
147
|
+
if (!this.policy.remoteRead || this.remote.hasMany === undefined) return null
|
|
148
|
+
try {
|
|
149
|
+
return await this.remote.hasMany(hashes)
|
|
150
|
+
} catch (err) {
|
|
151
|
+
this.reportRemoteError(err)
|
|
152
|
+
return null
|
|
153
|
+
}
|
|
154
|
+
}
|
|
155
|
+
|
|
156
|
+
/**
|
|
157
|
+
* Record that the remote layer has NO artifact for each of `hashes` (from a
|
|
158
|
+
* batch probe), so a later `get`/`prefetch` short-circuits to a miss WITHOUT
|
|
159
|
+
* a network round-trip. Byte-for-byte equivalent to a background prefetch GET
|
|
160
|
+
* having resolved `false` into `inflight` — same at-most-once semantics, same
|
|
161
|
+
* point-in-time staleness window — but without spending the GET. Never
|
|
162
|
+
* overwrites an entry already in flight.
|
|
163
|
+
*/
|
|
164
|
+
markRemoteAbsent(hashes: Iterable<string>): void {
|
|
165
|
+
for (const hash of hashes) {
|
|
166
|
+
if (!this.inflight.has(hash)) this.inflight.set(hash, Promise.resolve(false))
|
|
167
|
+
}
|
|
168
|
+
}
|
|
169
|
+
|
|
129
170
|
async get(hash: string, ctx?: CacheGetContext): Promise<CacheEntry | null> {
|
|
130
171
|
const localHit = await this.local.get(hash, ctx)
|
|
131
172
|
if (localHit) {
|
|
@@ -63,15 +63,30 @@ async function runPrefetch(args: PrefetchArgs): Promise<void> {
|
|
|
63
63
|
const stableKeys = await deriveStableKeys(args)
|
|
64
64
|
if (stableKeys.length === 0) return
|
|
65
65
|
|
|
66
|
-
//
|
|
66
|
+
// Batch existence probe FIRST (when the remote supports it): one round-trip
|
|
67
|
+
// tells us which of the N stable hashes exist remotely, so we GET only the
|
|
68
|
+
// hits and pre-mark the misses — their lazy `get` then short-circuits with
|
|
69
|
+
// no network. This collapses N probe waves into 1 and skips every GET that
|
|
70
|
+
// would 404. When the remote can't batch (`null` — an older serve, or reads
|
|
71
|
+
// disabled), fall back to prefetching every stable key, exactly as before.
|
|
72
|
+
let toPrefetch = stableKeys
|
|
73
|
+
const uniqueHashes = [...new Set(stableKeys.map((k) => k.hash))]
|
|
74
|
+
const present = await args.cache.remoteHasMany(uniqueHashes)
|
|
75
|
+
if (present !== null) {
|
|
76
|
+
args.cache.markRemoteAbsent(uniqueHashes.filter((h) => !present.has(h)))
|
|
77
|
+
toPrefetch = stableKeys.filter((k) => present.has(k.hash))
|
|
78
|
+
if (toPrefetch.length === 0) return
|
|
79
|
+
}
|
|
80
|
+
|
|
81
|
+
// Bounded worker pool over the keys to fetch. Each prefetch is
|
|
67
82
|
// self-contained (LayeredCache owns the in-flight map + ingest); we
|
|
68
83
|
// cap concurrency so a 1000-task run doesn't open 1000 sockets at
|
|
69
84
|
// once. The pumps race alongside execution.
|
|
70
85
|
let next = 0
|
|
71
|
-
const workers = Math.max(1, Math.min(args.concurrency,
|
|
86
|
+
const workers = Math.max(1, Math.min(args.concurrency, toPrefetch.length))
|
|
72
87
|
const pump = async (): Promise<void> => {
|
|
73
|
-
while (next <
|
|
74
|
-
const { hash, node } =
|
|
88
|
+
while (next < toPrefetch.length) {
|
|
89
|
+
const { hash, node } = toPrefetch[next++]!
|
|
75
90
|
await args.cache
|
|
76
91
|
.prefetch(hash, { taskId: node.id, command: node.config.exec?.command ?? '' })
|
|
77
92
|
.catch(() => false)
|