@vzn/vx-migrate 0.0.638 → 0.0.639

package/README.md

@@ -66,7 +66,7 @@ Rules:
 - `env` / `passThroughEnv`: explicit names go to `cache.inputs.env` (env only) and `exec.env.passThrough` (both, plus both globals); a wildcard expands over the run's environment under `turbo()`, which maps where the tasks run (a new variable name such a wildcard matches maps afresh); the CLI, which writes for another environment, lists the names the task's package and its workspace dependencies spell (a root task: the whole repo; clerk's `E2E_*`), noted once per entry, and one nothing spells is a todo; `*` is the only wildcard, as in Turbo, so `\*`, a leading `\!`, `?` and `[` are literal: unkey's `NEXT_PUBLIC_\*` names one variable, which vx cannot key, so it is dropped and reported once for the workspace with the count of tasks naming it (openstatus's root `build` env `\*` had written one todo into each of 45 configs; `turbo()`: only when the run's environment sets it, as Turbo hashes nothing otherwise); and a `!` entry (openstatus' `!NEXT_PUBLIC_VERCEL_URL`) removes the names it matches from its list, with no todo, since vx matches no name it is not given. A name both a global list and the task's own list carry is listed once. A name that is no shell variable name (`p-q`) is keyed but left out of `exec.env.passThrough`, as `sh` drops it before the task runs.
 - Framework inference (Turbo hashes and passes Next's `NEXT_PUBLIC_*`, Vite's `VITE_*`, … for a package that depends on the framework): `turbo()` expands the prefix over the run's environment; the CLI writes the names the tracked source and `.env.example` of the package and its workspace dependencies spell (the bare prefix, as a `startsWith` test spells it, is none) (Next bundles theirs: cal.com's web, 23 alone, 58 with them), and a note asks for any only an installed dependency reads.
 
-- **Two tasks of one package on one output path** (strapi's `build`, `build:code` and `build:types`, all on `dist/**`): vx cleans a task's outputs before it runs and before a restore, so the loader refuses two cached tasks whose outputs provably overlap. Under vx's default `rules.exclusiveOutputs` an edge between the two changes nothing (twenty's `build:individual` writing into `build`'s `dist`). The mapping resolves it before the file is written — `build` keeps its cache, else the task with a `^` edge, else the first declared (typescript-eslint's `typecheck` has `^typecheck` and writes declarations into `build`'s `dist`: keeping it left `build` uncached); the rest run uncached with a todo naming the keeper and the fix, their own output path. A shared tsc incremental file (create-t3-turbo's `build` and `typecheck` both on `.cache/tsbuildinfo.json`) stays with the keeper alone, and the other task keeps its cache when it emits nothing else (a typecheck); one that emits more runs uncached, as `tsc --incremental` skips files the keeper's tsbuildinfo calls built. Same rule for Nx targets.
+- **Two tasks of one package on one output path** (strapi's `build`, `build:code` and `build:types`, all on `dist/**`): vx cleans a task's outputs before it runs and before a restore, so the loader refuses two cached tasks whose outputs provably overlap. Under vx's default `rules.exclusiveOutputs` an edge between the two changes nothing (twenty's `build:individual` writing into `build`'s `dist`). The mapping resolves it before the file is written — `build` keeps its cache, else the task with a `^` edge, else the first declared (typescript-eslint's `typecheck` has `^typecheck` and writes declarations into `build`'s `dist`: keeping it left `build` uncached); the rest run uncached with a todo naming the keeper and the fix, their own output path. A shared tsc incremental file (create-t3-turbo's `build` and `typecheck` both on `.cache/tsbuildinfo.json`) stays with the keeper alone, and the other task keeps its cache when it emits nothing else (a typecheck); one that emits more runs uncached, as `tsc --incremental` skips files the keeper's tsbuildinfo calls built. A keeper whose output is a directory at any depth (solid's build: `**/dist`) meets a sibling's tree (`coverage/**`) only at a `dist` inside it: the keeper takes that tree back with `!coverage/**` and both keep their cache, as under Turbo. Same rule for Nx targets.
 - **A task whose inputs read another task's outputs** (a `test` keyed on `**/*` beside a `build` writing `dist/**`): vx's default `rules.upfrontKeys` refuses it, since such a key cannot be known before the producer ran. Each overlapping output is taken back from the reader's inputs with `!` (`['**/*', '!dist/**']`), in package and workspace inputs alike, and a workspace output another package writes into this one is taken back from its package inputs too; Turbo and Nx hash what git tracks, and a build's outputs are not, so the key reads what the adopted tool read. A literal input another task writes cannot be taken back: that task runs uncached with a todo. A committed file under such an output is still a source: a reader the `!` would hide it from runs uncached with a todo. A keyed group (an `nx-input:<name>` twin, a transit node) runs nothing, so core lets it read what a task writes: it gets no `!` for a literal, the `!` that would hide a committed file goes, and it keeps its key (typescript-eslint's ast-spec commits the `*.shot` snapshots its `test` outputs, and its twin was written as `{}`, which core refuses). An output that names one committed file (eslint's `eslint-suppressions.json` in a `lint` task's outputs) is dropped, and so is the `!` each reader got for it: the file stays in every key, as Turbo and Nx hash it, and the readers stay cached. Same for Nx targets.
 - **Two packages' tasks on one workspace output** (cal.com's shared `post-install` writes `../../node_modules/@prisma/client/**` from every package with the script): core refuses two cached tasks on one path with no edge between them, so `turbo()` (and `nx()`) keeps the first (in package and task order) cached and runs the rest uncached, each with a todo naming the keeper. A package's own outputs count at their workspace path: typescript-eslint's root project caches `dist` and each package's typecheck `dist/packages/<name>`, one path twice. Edges across packages are not read, so an ordered pair loses its cache too.
 - `cache: false` or `persistent: true` → no `cache` block; a persistent task gets `exec.persistent: {}` and, when some task depends on it, the consumer's `persistentTodo`.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@vzn/vx-migrate",
-  "version": "0.0.638",
+  "version": "0.0.639",
   "license": "MIT",
   "description": "Adopt @vzn/vx from Turborepo, Nx or package.json scripts: run the repo unchanged, migrate it to vx.config.ts, and keep your Turbo or Nx remote cache",
   "keywords": [
@@ -40,7 +40,7 @@
     "access": "public"
   },
   "peerDependencies": {
-    "@vzn/vx": "^0.0.638"
+    "@vzn/vx": "^0.0.639"
   },
   "repository": {
     "type": "git",

package/src/shared-outputs.ts

@@ -171,6 +171,8 @@ interface Cached {
   name: string
   files: string[]
   hasUpstreamEdge: boolean
+  /** Globs of a sibling's this task's `!` entries take back (`separable`). */
+  back: Set<string>
 }
 
 function cachedOutputs(t: GeneratedTask, index: number): Cached | null {
@@ -186,9 +188,32 @@ function cachedOutputs(t: GeneratedTask, index: number): Cached | null {
   const deps = task['dependsOn']
   const hasUpstreamEdge =
     Array.isArray(deps) && deps.some((d) => typeof d === 'string' && d.startsWith('^'))
-  return { index, name: t.name, files: strings, hasUpstreamEdge }
+  return { index, name: t.name, files: strings, hasUpstreamEdge, back: new Set() }
 }
 
+/**
+ * Can the keeper's `ga` give `gb`'s tree back? Only when `ga` is a
+ * directory at any depth (`**\/dist`, `**\/dist/**`) and `gb`'s literal
+ * head names no such directory: the two meet only at a `dist` nested in
+ * `gb`'s tree, which is the sibling's. solid's build outputs `**\/dist`
+ * beside types' `types/**`; Turbo caches both, and uncaching types made
+ * vx the slower of the two.
+ */
+function separable(ga: string, gb: string): boolean {
+  const name = /^\*\*\/([^*?[\]{}()!/\\]+)(?:\/\*\*)?$/.exec(ga)?.[1]
+  if (name === undefined) return false
+  const head: string[] = []
+  for (const seg of gb.replace(/^(\.\/)+/, '').split('/')) {
+    if (!isLiteralPattern(seg) || seg === '') break
+    head.push(seg)
+  }
+  return head.length > 0 && !head.includes(name)
+}
+
+/** Does keeper `k`'s `ga` still reach `gb` once its `!` entries are read? */
+const hits = (k: Cached, ga: string, gb: string): boolean =>
+  !k.back.has(gb) && outputsOverlap(ga, gb)
+
 /**
  * Uncache every task whose declared outputs overlap a keeper's, in place,
  * and return the tasks. Deterministic: keepers are chosen in declaration
@@ -220,14 +245,28 @@ export function resolveSharedOutputs(tasks: GeneratedTask[]): GeneratedTask[] {
     const kept: Cached[] = [keeper]
     for (const c of all) {
       if (c === keeper) continue
-      const clash = kept.find((k) =>
-        c.files.some((gb) => k.files.some((ga) => outputsOverlap(ga, gb))),
-      )
+      const clash = kept.find((k) => c.files.some((gb) => k.files.some((ga) => hits(k, ga, gb))))
       if (clash === undefined) {
         kept.push(c)
         continue
       }
       const t = tasks[c.index]!
+      const pairs = kept.flatMap((k) =>
+        c.files.flatMap((gb) =>
+          k.files.filter((ga) => hits(k, ga, gb)).map((ga) => ({ k, ga, gb })),
+        ),
+      )
+      if (pairs.every((p) => separable(p.ga, p.gb))) {
+        for (const { k, gb } of pairs) {
+          if (k.back.has(gb)) continue
+          k.back.add(gb)
+          ;(tasks[k.index]!.task!['cache'] as { outputs: { files: string[] } }).outputs.files.push(
+            `!${gb}`,
+          )
+        }
+        kept.push(c)
+        continue
+      }
       // tsc's incremental state is no product: create-t3-turbo's build and
       // typecheck both output `.cache/tsbuildinfo.json`, and typecheck ran
       // uncached on every first run (Growth funnel F8). Left to the keeper,
@@ -236,7 +275,7 @@ export function resolveSharedOutputs(tasks: GeneratedTask[]): GeneratedTask[] {
       // the keeper's tsbuildinfo calls built, missing or not, so its save
       // could be a partial dist.
       const sharedFiles = c.files.filter((gb) =>
-        kept.some((k) => k.files.some((ga) => outputsOverlap(ga, gb))),
+        kept.some((k) => k.files.some((ga) => hits(k, ga, gb))),
       )
       if (
         sharedFiles.length === c.files.length &&
@@ -249,7 +288,7 @@ export function resolveSharedOutputs(tasks: GeneratedTask[]): GeneratedTask[] {
         continue
       }
       dropped.add(c.index)
-      const shared = c.files.find((gb) => clash.files.some((ga) => outputsOverlap(ga, gb)))!
+      const shared = c.files.find((gb) => clash.files.some((ga) => hits(clash, ga, gb)))!
       delete t.task!['cache']
       t.todos.push(
         `declares the output ${JSON.stringify(shared)} that ${JSON.stringify(clash.name)} also ` +

package/src/turbo-cache/index.ts

@@ -385,6 +385,12 @@ export class TurboRemoteCache implements RemoteCacheLayer {
    * the reads off too, and every task after the first upload missed.
    */
   private writesRefused = false
+  /**
+   * Keys this process saw the server lack (a 404, a batch answer): `put`
+   * asks no HEAD for them. Turbo's PUT replaces a stored artifact and an
+   * entry is immutable, so any other key is asked first.
+   */
+  private readonly absent = new Set<string>()
   private readonly key: Uint8Array | undefined
   readonly endpoint: string
   /** What `TURBO_REMOTE_CACHE_READ_ONLY` and friends leave this run (`turboRemoteAccess`). */
@@ -517,7 +523,10 @@ export class TurboRemoteCache implements RemoteCacheLayer {
     const res = await this.request('HEAD', `/${hash}`)
     if (res === undefined) return false
     if (res.status === 200) return true
-    if (res.status === 404) return false
+    if (res.status === 404) {
+      this.absent.add(hash)
+      return false
+    }
     throw new Error(`HTTP ${res.status}`)
   }
 
@@ -538,12 +547,14 @@ export class TurboRemoteCache implements RemoteCacheLayer {
     // Each hash answers ArtifactInfo, null, or `{ error: { message } }`; an
     // error entry is not a stored artifact, and it was counted as one (a GET
     // spent on a 404, item 929).
-    return new Set(
+    const held = new Set(
       hashes.filter((h) => {
         const v = info[h]
         return typeof v === 'object' && v !== null && !('error' in v)
       }),
     )
+    for (const h of hashes) if (!held.has(h)) this.absent.add(h)
+    return held
   }
 
   async get(hash: string): Promise<{ body: Response; durationMs: number | undefined } | null> {
@@ -556,7 +567,10 @@ export class TurboRemoteCache implements RemoteCacheLayer {
     if (res === undefined) return null
     if (res.status !== 200) {
       await res.body?.cancel()
-      if (res.status === 404) return null
+      if (res.status === 404) {
+        this.absent.add(hash)
+        return null
+      }
       throw new Error(`HTTP ${res.status}`)
     }
     const duration = Number(res.headers.get('x-artifact-duration'))
@@ -608,6 +622,16 @@ export class TurboRemoteCache implements RemoteCacheLayer {
 
   async put(hash: string, body: Blob, meta: { durationMs: number }): Promise<void> {
     if (this.disabled || this.writesRefused) return
+    if (!this.absent.has(hash)) {
+      // Part of the upload, so under its deadline. Any answer but 200 or 404
+      // uploads: a lost store costs more. A server the HEAD cannot reach
+      // fails the put, as the PUT would.
+      const head = await this.request('HEAD', `/${hash}`, {
+        timeoutMs: this.config.uploadTimeoutMs,
+      })
+      if (head === undefined || head.status === 200) return
+      if (head.status === 404) this.absent.add(hash)
+    }
     const headers: Record<string, string> = {
       'Content-Type': 'application/octet-stream',
       'Content-Length': String(body.size),
@@ -625,6 +649,7 @@ export class TurboRemoteCache implements RemoteCacheLayer {
     // Turbo's client takes any 2xx (`error_for_status`); a server's 201 or
     // 204 stored the artifact and was counted a failed upload.
     if (!res.ok) throw new Error(`HTTP ${res.status}`)
+    this.absent.delete(hash)
   }
 }