@vzn/vx 0.0.637 → 0.0.639

package/docs/caching.md

@@ -453,8 +453,17 @@ transitive dependant of one with it (their up-front keys fold a
 preliminary key); a glob with no literal prefix reaches every project.
 Before item 584 one such declaration emptied the tier graph-wide.
 `rules.upfrontKeys` (on by default, X-54) refuses at load a task whose
-input globs can match another task's declared outputs, so the clauses
-on declared outputs fire only with the rule off; undeclared writes, an
+input globs can match another task's declared outputs, unless git
+ignores those outputs whole, so the clauses on declared outputs fire
+only with the rule off. An output git ignores whole is no reach either:
+a key reads only what git lists, so a `**/*` reader beside a gitignored
+`**/dist/**` producer keys up front, the same key before and after the
+build. The run asks git once, beside its walk and only when a reader
+reaches a same-project output: one `git check-ignore` over each such
+output's directory (`dist/`, and a fresh `__vx_probe__/dist/` for a
+glob that starts with `**`), and the walk's listing must hold no file
+the glob matches (a tracked output, a directory a `!` rule takes back);
+anything git cannot answer keeps the key late. Undeclared writes, an
 `outputs.workspaceFiles` producer upstream, a rewriter the key does not
 fold, and runtime probes after a writer still leave a key preliminary.
 The
@@ -553,7 +562,24 @@ Hard invariants of the remote prefetch:
   later `get` finds it as a now-local hit — still reports
   `source: 'remote'`, so the outcome is `cache-hit-remote`.
 - **A remote-read-off policy** (`--no-cache`, `--force`, or
-  `--cache-policy=remote:`) fires no prefetch.
+  `--cache-policy=remote:`) fires no prefetch. With remote writes on
+  (`--force`), the same up-front keys go to one `hasMany` per remote
+  the run writes but does not read and that can batch, in the
+  background: no task waits for it, only each upload does. An upload
+  of a key that remote holds is skipped (an entry is immutable) and is
+  counted nowhere. A key the probe did not cover, or a remote with no
+  `hasMany`, asks `has` for that key inside its upload job; a probe
+  that fails uploads, as before.
+- **No `has` before a `put`.** An entry is immutable: a remote's `put`
+  of a key it holds keeps the stored bytes, and a wire that would
+  overwrite checks inside (`turboCache()` asks HEAD unless a 404 or a
+  batch answer this run said the key is absent; `reapi()` looks the
+  action up beside its upload and keeps a stored result; Nx servers
+  answer 409). `ctx.cache.put` and a run's own entries send the `put`
+  alone.
+- **`--dry` batches too.** The plan derives every key first, then asks
+  each remote with `hasMany` once; a remote without it is asked `has`
+  per key.
 - **Never fail.** Every remote path (get / put / ingest / prefetch)
   catches all errors and degrades to a cache miss. A remote 500, a
   network drop, a corrupt artifact, or a failed integrity check can
@@ -573,16 +599,16 @@ fire-and-forget background upload**. The task's outcome (and its
 dependents) never wait on upload latency — the uploads race alongside
 the rest of the run and are drained before `cache.close()`, so a
 short run still ships every artifact before the process exits. Upload
-failures log via `onRemoteError` and are otherwise ignored (the task
+failures are warned once per failure class and are otherwise ignored (the task
 already succeeded; the only loss is the remote entry).
 
 An upload is the outputs as the task wrote them; secret masking does
 not reach file contents, so a task whose outputs embed a secret is one
 to leave uncached ([security](./security.md)).
 
-### Planning probes (`--dry` / `--graph`)
+### Planning probes (`--dry`)
 
-The planning paths (`vx run --dry`, `--graph`) predict hits without
+The planning path (`vx run --dry`) predicts hits without
 side effects: against a remote cache they use a **lightweight
 existence probe** — no artifact download, no local ingest. A predicted
 `hit-remote` means the artifact exists remotely; the bytes move only
@@ -615,8 +641,8 @@ remote executor's own store.
 The CLI maps three flags to a policy (precedence: start all-on → apply
 `--cache-policy` → `--no-cache` forces all off → `--force` forces both reads
 off): `--no-cache` = everything off (no read, no write, no output
-clean); `--force` = reads off / writes on (re-execute and refresh the
-cache, outputs cleaned); `--cache-policy=<spec>` = explicit per-layer control.
+clean); `--force` = reads off / writes on (re-execute, outputs cleaned;
+a key already stored, locally or in a remote, is kept as it is); `--cache-policy=<spec>` = explicit per-layer control.
 Off CI, unless a spec names the remote, `remoteWrite` starts off: a key
 holds only declared inputs, so a laptop's undeclared tool or file could
 replay on CI. See `docs/cli.md` § Cache control.
@@ -665,7 +691,10 @@ is on):
    Until 2026-09-27 (A-3) the rename came first, and a commit refused
    past the busy timeout left the new bytes beside the old rows: every
    later hit on the key failed the task as a corrupt artifact.
-   A re-save (`--force`) first moves the previous artifact aside under
+   A `--force` save of a key the index already holds is skipped, so
+   `--force` rewrites nothing. A re-save (a key dropped as corrupt, or
+   two writers that both found it absent) first moves the previous
+   artifact aside under
    a temp name and unlinks it after the commit: ext4 flushes the
    incoming file when a rename replaces one, 0.55 ms a save on the main
    thread against 0.04 (a forced 1,000-task run 4.01 s → 3.46 s,
@@ -839,10 +868,10 @@ Skipped when:
   on, so it DOES clean (the saved snapshot must be clean).
 - `cache.outputs.clean` is `false` (on the task, else the workspace's
   `cache.outputs.clean`): the command sees last run's outputs (tsc's
-  `.tsbuildinfo`). The first attempt stamps what it finds instead;
-  if the run leaves any old file unchanged, nothing is cached, so no
-  entry holds a file this run did not write. A retry cleans as usual; a hit still
-  cleans before its restore.
+  `.tsbuildinfo`). You opt out of the clean-tree guarantee: the save
+  stores every file the outputs match after the run, so a leftover from
+  an older build is cached and restored everywhere. A retry cleans as
+  usual; a hit still cleans before its restore.
 
 A declared output the process cannot remove (a `dist/` another user
 wrote, a read-only checkout) fails the task with `cannot remove declared
@@ -1192,7 +1221,8 @@ in `~/.vx/<id>/cache`, and each workspace keeps only its own index in
 its `.vx/cache`: every checkout of the repository hits what another
 saved (a second clone, a worktree), and none reads another's history.
 Task timings are shared too: `task_durations`, what a duration forecast
-reads, so a new worktree has one on its first run.
+reads, so a new worktree has one on its first run. So are plugins' own
+entries (`plugin_blobs`, a plugin's `ctx.cache`).
 The id is Nx 23's: 16 hex of a sha256 of the remote (`origin`, then
 `upstream`, `base`, the first; `host/owner/repo` in lower case, so ssh
 and https agree) and the workspace's path in the repository; with no
@@ -1204,7 +1234,7 @@ owner-only; vx closes one of yours that others can read (chmod 700), and one oth
 ~/.vx/<id>/cache/                           the shared store
 ├── store.db                                entries, entry_stdout, output_files,
 │                                           entry_inputs, task_durations,
-│                                           store_meta; artifacts
+│                                           plugin_blobs, store_meta; artifacts
 │                                           of at most 32 KiB compressed (v33)
 └── <hash>.tar.zst                          the larger artifacts (below)
 
@@ -1469,9 +1499,9 @@ keep). The artifacts stay: each is indexed again from its own bytes
 when its task next asks for its key (below).
 
 ```sql
--- src/cache/schema.ts (SCHEMA_VERSION = 'v36', in cache.ts)
+-- src/cache/schema.ts (SCHEMA_VERSION = 'v40', in cache.ts)
 -- With a shared store, entries, entry_stdout, output_files, entry_inputs,
--- store_meta and task_durations live in its store.db, attached as `store`; the rest is
+-- store_meta, task_durations, task_outcomes, blob_sizes and plugin_blobs live in its store.db, attached as `store`; the rest is
 -- the workspace's cache.db. A named cache dir holds all of them.
 
 CREATE TABLE schema_meta (
@@ -1526,7 +1556,8 @@ CREATE TABLE runs (
   timed_out           INTEGER,          -- 1 when vx's own timeout killed it
   sandbox_violations  INTEGER,          -- the sandbox's violation count
   not_ready           TEXT,             -- 'timeout' | 'exited' | 'spawn' (persistent task)
-  restored            INTEGER           -- v31, on a hit: 1 = outputs restored, 0 = up to date
+  restored            INTEGER,          -- v31, on a hit: 1 = outputs restored, 0 = up to date
+  remote_error        TEXT              -- v40: each remote read or write of the key that failed
 );
 
 -- Two indexes, both append-only under a run's inserts: every row of a run
@@ -1537,10 +1568,6 @@ CREATE TABLE runs (
 -- the history reader bounds its scan by rowid instead (see history.md).
 CREATE INDEX runs_started_at ON runs(started_at);
 CREATE INDEX runs_run_id     ON runs(run_id);
--- The one keyed index, PARTIAL over failed rows: a green run's inserts only
--- evaluate its predicate, and the flakiness probe after a miss
--- (failure-mode.ts, "did this key ever fail?") reads a few leaves.
-CREATE INDEX runs_failed ON runs(hash) WHERE status = 'failed';
 
 -- Every non-group, non-aborted outcome of a run gets a row, so
 -- `invocations.task_count` always equals `COUNT(*)` here for that run_id
@@ -1575,7 +1602,8 @@ CREATE TABLE file_hashes (
 -- The size of each index blob the enumeration checked against the
 -- worktree size git recorded (§ Cache key derivation, A-60): fixed for
 -- its OID, so a warm run asks git for none. Swept with file_hashes by
--- seen_at, the time the row was written.
+-- seen_at, the time the row was written. v38: in the store, so a worktree
+-- reads what another learned.
 CREATE TABLE blob_sizes (
   oid     TEXT PRIMARY KEY,
   size    INTEGER NOT NULL,
@@ -1630,13 +1658,13 @@ CREATE TABLE output_dirs (
 
 -- v22 (Tier 3): one header row per `vx run` invocation. The `runs`
 -- table is per-task; this is the per-invocation record carrying the
--- command, git/CI/host context, tags, and run-level counts. Recorded
+-- command, git/CI/host context, and run-level counts. Recorded
 -- atomically alongside `runs` via recordRunBundle (one transaction).
 CREATE TABLE invocations (
   run_id            TEXT PRIMARY KEY,         -- UUIDv7, == runs.run_id
   command           TEXT NOT NULL,            -- full argv, e.g. "vx run build --all"
   requested_tasks   TEXT NOT NULL,            -- JSON string[] of options.tasks
-  cache_policy      TEXT NOT NULL,            -- compact flags, e.g. "lR,lW,rR,rW"
+  cache_policy      TEXT NOT NULL,            -- the --cache-policy spec, e.g. "local:rw,remote:r"
   concurrency       INTEGER NOT NULL,
   flow              TEXT,                     -- 'focused' | 'broad' | NULL (programmatic)
   started_at        INTEGER NOT NULL,         -- ms-epoch
@@ -1659,8 +1687,7 @@ CREATE TABLE invocations (
   host              TEXT,                     -- os.hostname()
   os                TEXT,                     -- process.platform
   arch              TEXT,                     -- process.arch
-  vx_version        TEXT NOT NULL,
-  tags              TEXT NOT NULL DEFAULT '{}' -- JSON object {k:v} from --tag
+  vx_version        TEXT NOT NULL
 );
 CREATE INDEX invocations_started ON invocations(started_at);
 CREATE INDEX invocations_branch  ON invocations(branch);
@@ -1696,6 +1723,8 @@ CREATE TABLE store_meta (
 -- v36: each executed success's duration, appended in the run's history
 -- transaction, in the store so every worktree forecasts from it.
 -- source: 'branch:<name>'; the forecast reads the newest five per task.
+-- v37: 'remote' rows are a default-branch commit's timings a plugin
+-- imported, one per (sha, task).
 CREATE TABLE task_durations (
   id          INTEGER PRIMARY KEY AUTOINCREMENT,
   project     TEXT NOT NULL,
@@ -1706,6 +1735,33 @@ CREATE TABLE task_durations (
   recorded_at INTEGER NOT NULL
 );
 CREATE INDEX task_durations_task ON task_durations(project, task, source, id);
+CREATE UNIQUE INDEX task_durations_remote
+  ON task_durations(sha, project, task) WHERE source = 'remote';
+
+-- v38: the flaky record (failure-mode.ts), in the store so a key that
+-- passed in one worktree and failed in another is flaky in both. Every
+-- executed success or failure that names its key (keyed, cached, not run
+-- behind a failure), and a key's first hit: a later hit replays a pass on
+-- record, so a warm run on passed keys writes no row (one index probe,
+-- ~1 ms per 1,000 hits). Pruned by age with the run history.
+CREATE TABLE task_outcomes (
+  id          INTEGER PRIMARY KEY AUTOINCREMENT,
+  project     TEXT NOT NULL,
+  task        TEXT NOT NULL,
+  hash        TEXT NOT NULL,
+  status      TEXT NOT NULL,
+  attempts    INTEGER,
+  recorded_at INTEGER NOT NULL
+);
+CREATE INDEX task_outcomes_hash ON task_outcomes(hash, status);
+
+-- v37: a plugin's own entries (`ctx.cache`), by core's key for its
+-- name; written once, pruned by age with the run history.
+CREATE TABLE plugin_blobs (
+  key   TEXT PRIMARY KEY,
+  bytes BLOB NOT NULL,
+  at    INTEGER NOT NULL
+);
 
 -- v33: an artifact of at most 32 KiB compressed, its exact <hash>.tar.zst
 -- bytes, written in the transaction that writes its entry rows. In the
@@ -2114,7 +2170,7 @@ breaking footer).
   migration), so this costs one cold rebuild.
 - **SCHEMA v21 → v22 (no `CACHE_VERSION` bump)**: Tier-3 dashboard
   tables — `invocations` (one header row per `vx run` with command,
-  git/CI/host context, tags, and run-level counts, recorded with `runs`
+  git/CI/host context, and run-level counts, recorded with `runs`
   via `Cache.recordRunBundle`) and `entry_inputs` (one row per cache-key
   component, keyed by the cache-ENTRY hash — the input-fingerprint
   moat). `entry_inputs` is written **inside the entry-save transaction,