@vzn/vx-migrate 0.0.641 → 0.0.642
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +38 -12
- package/package.json +2 -2
- package/src/migrate-turbo.ts +3 -0
- package/src/migrate.ts +12 -7
- package/src/nx/nx-map.ts +41 -4
- package/src/nx-cache/index.ts +46 -7
- package/src/remote-cache.ts +1 -1
- package/src/remote-retry.ts +52 -12
- package/src/remote-token.ts +48 -0
- package/src/tsc-buildinfo.ts +168 -0
- package/src/turbo/index.ts +11 -2
- package/src/turbo/turbo-map.ts +20 -0
- package/src/turbo-cache/index.ts +141 -27
- package/src/turbo-cache/oidc.ts +171 -0
package/README.md
CHANGED
|
@@ -62,7 +62,7 @@ Rules:
|
|
|
62
62
|
- A transit node (Turbo's documented pattern, its with-vitest example: `transit: { dependsOn: ["^transit"] }` with no script anywhere, and `test: { dependsOn: ["transit"] }`) is a key-only task in each package that defines it: a keyed group (no command, nothing spawns) keyed on its inputs, with its `^transit` edge, so a dependant's `test` re-keys on its dependencies' sources as Turbo re-hashes it. A `^` task that some package has a script for, or that nothing depends on, is no transit node. A package without the script of a `^` task others run gets the same key-only task, as Turbo hashes a no-op node there: an edit to it re-keys its dependants. A `build` whose only edge is `^build` is not written: core gives every project with no `build` that node (solid's three script-less packages each got a `build` of their own). A dependency without the script whose own edges lack `^x` gets it too, so a `^x` stops there as Turbo's does (opencode's `build: { dependsOn: [] }`: core walked past to the builds below, which Turbo's `^build` never reaches).
|
|
63
63
|
- `with` (tasks Turbo runs alongside, `web#dev` with `api#dev`): an edge to each sidecar that is persistent, so vx starts the task once the sidecar has spawned and runs the sidecar only when the task runs; a sidecar that ends would be waited for, so it is a todo instead, and a pair that names each other keeps one edge (two would be a cycle). A task with no script whose `with` names persistent sidecars (Turbo's with-tailwind example: `ui` has no `dev` script, its `dev` starts `dev:styles` and `dev:components`) is a group task that depends on them.
|
|
64
64
|
- `inputs`: a structured entry (Turbo 2.11) is its `globs`, plus `**/*` with `withDefaults`, for `startup` and `jit` alike; `dependencyOutputs` adds none where the producer is cached (vx folds each dependency's key); from a `cache: false` producer (vercel/vercel's `//#generate:cache-keys`, which records the host) the named files, or its outputs, are a `inputs.workspaceRuntime` probe, and `inputs.tasks` leaves the producer out, so core answers the probe after it ran. Then: absent or `[]` → `**/*` (Turbo's default); `$TURBO_DEFAULT$` → `**/*`; exclusions alone narrow `**/*`; `$TURBO_ROOT$/<path>` → `inputs.workspaceFiles` (negation kept); any other `$TURBO_ROOT$` use is a todo. `globalDependencies` and Turbo 1's `globalDotEnv` land in `workspaceFiles` too, and a task's Turbo 1 `dotEnv` in its `files`. Turbo 1's `$NAME` entries, in `globalDependencies` or a task's `dependsOn`, are env vars: they join `inputs.env` and `exec.env.passThrough`. A package nested in the task's package (cal.com's apps inside `@calcom/app-store`) is hashed by Turbo with its parent, and core's globs stop at a project: each glob that reaches one is also listed in `inputs.workspaceFiles`. Turbo's glob grammar is translated as Nx's is (`*.[jt]s` is `*.{[jt],j,t}s`); an input with no safe form widens to `**/*` with a todo. A wildcard `X/**` input, `!` or not, also lists `X`, as outputs do: Turbo's `**` matches zero segments, so `*/**` keys on the package-root files too. A `.env`-shaped input (`.env*`, `.env.local`, a task's Turbo 1 `dotEnv`, create-turbo's `globalDependencies: ["**/.env.*local"]`) is gitignored as a rule, so it is keyed by a probe that hashes the `.env` files it can name (`inputs.runtime`) or, for a root entry, the files its root-relative globs name, hidden ones too, as Turbo's `*` matches them (`workspaceRuntime`; `**/<name>` is a `find -name` that skips `node_modules`, and another `**` walks the workspace), rather than as a file glob git never reports. A package whose `.env` globs all sit at its root gets a one-shell probe of that directory; one below the root walks the package.
|
|
65
|
-
- `outputs`: `$TURBO_ROOT$/<path>` → `cache.outputs.workspaceFiles`; a negated output rides beside its positives and takes its paths back from the clean, the save and the restore (Next's `.next/**` minus `!.next/cache/**` keeps the cache), and one with no positive beside it takes back nothing and is dropped. Past its first segment an output takes Turbo's grammar (`dist/**/*.[cm]js`); the first stays a literal (a route directory). A wildcard `X/**` also lists `X`: Turbo's `**` matches zero segments, so medusa's `*/**` caches the package-root `tsconfig.tsbuildinfo` too (its `package.json` is taken back, below); a literal `dist/**` needs no `dist`. An output whose first segment is a wildcard (`**/*.d.ts`, medusa's `*/**`) stays cached, as in `nx()` (`turbo()` 2026-10-14): Turbo never cleans an output, vx cleans it before every run and every restore, and such a glob can reach the sources; core never cleans a committed file, and every untracked file it reaches is deleted, which `vx lock`'s warnings name (`output "*/**": … every untracked file this glob reaches is deleted`). Negations beside it (`!src/**`) keep what they name. Without git the files cannot be told apart, and the task runs uncached with a todo. A glob that reaches no tracked kind gets no warning: one segment with a literal extension the package tracks no file of (`*.xml`), `**/<dir>/**` when the package tracks nothing under a directory of that name (vercel/ai's `**/dist/**`), and a first segment no tracked top-level entry matches (tldraw's `dist-*/**`). An output that covers the package's own `package.json` (trpc's client build lists it: the build rewrites `exports`) takes the manifest back with `!package.json` and keeps the cache, in `turbo()` as in `nx()` (owner, 2026-10-03): vx would otherwise delete it before every run. The build rewrites a file it may list as an input (`@vzn/vx-lockfile` lists `package.json`), so the Turbo mapping also sets `inputStability: { strategy: 'mutable' }` and an identical rewrite counts as unchanged. So does one that covers the package's vx.config; `turbo()` and `nx()` check the config the package has, and `vx-migrate` the one it writes, so sanity's `*.js` shims stay cached beside none. `nx()` keeps a wildcard-first output cached the same way (owner, 2026-10-03). A committed file under an output (typescript-eslint's `data/sponsors.json` in a cached `data`) needs nothing: core never cleans a file git tracks, so it survives and the task keeps its cache.
|
|
65
|
+
- `outputs`: `$TURBO_ROOT$/<path>` → `cache.outputs.workspaceFiles`; a negated output rides beside its positives and takes its paths back from the clean, the save and the restore (Next's `.next/**` minus `!.next/cache/**` keeps the cache), and one with no positive beside it takes back nothing and is dropped. Past its first segment an output takes Turbo's grammar (`dist/**/*.[cm]js`); the first stays a literal (a route directory). A wildcard `X/**` also lists `X`: Turbo's `**` matches zero segments, so medusa's `*/**` caches the package-root `tsconfig.tsbuildinfo` too (its `package.json` is taken back, below); a literal `dist/**` needs no `dist`. An output whose first segment is a wildcard (`**/*.d.ts`, medusa's `*/**`) stays cached, as in `nx()` (`turbo()` 2026-10-14): Turbo never cleans an output, vx cleans it before every run and every restore, and such a glob can reach the sources; core never cleans a committed file, and every untracked file it reaches is deleted, which `vx lock`'s warnings name (`output "*/**": … every untracked file this glob reaches is deleted`). Negations beside it (`!src/**`) keep what they name. Without git the files cannot be told apart, and the task runs uncached with a todo. A glob that reaches no tracked kind gets no warning: one segment with a literal extension the package tracks no file of (`*.xml`), `**/<dir>/**` when the package tracks nothing under a directory of that name (vercel/ai's `**/dist/**`), and a first segment no tracked top-level entry matches (tldraw's `dist-*/**`). An output that covers the package's own `package.json` (trpc's client build lists it: the build rewrites `exports`) takes the manifest back with `!package.json` and keeps the cache, in `turbo()` as in `nx()` (owner, 2026-10-03): vx would otherwise delete it before every run. The build rewrites a file it may list as an input (`@vzn/vx-lockfile` lists `package.json`), so the Turbo mapping also sets `inputStability: { strategy: 'mutable' }` and an identical rewrite counts as unchanged. So does one that covers the package's vx.config; `turbo()` and `nx()` check the config the package has, and `vx-migrate` the one it writes, so sanity's `*.js` shims stay cached beside none. `nx()` keeps a wildcard-first output cached the same way (owner, 2026-10-03). A committed file under an output (typescript-eslint's `data/sponsors.json` in a cached `data`) needs nothing: core never cleans a file git tracks, so it survives and the task keeps its cache. A task whose command runs `tsc` that writes a `.tsbuildinfo` (`-b`, `composite` or `incremental`, read through the tsconfig's `extends` and, for `-b`, its `references`) outside its outputs also outputs that file: tsc re-emits only what its buildinfo calls changed, so with `rootDir: "src"` and `outDir: "dist"` its `tsconfig.tsbuildinfo` beside the tsconfig survived the clean before a miss and the miss cached a `dist/` holding only the edited file. Nx infers the file as an output; Turbo never cleans `dist/`.
|
|
66
66
|
- `env` / `passThroughEnv`: explicit names go to `inputs.env` (env only) and `exec.env.passThrough` (both, plus both globals); a wildcard expands over the run's environment under `turbo()`, which maps where the tasks run (a new variable name such a wildcard matches maps afresh); the CLI, which writes for another environment, lists the names the task's package and its workspace dependencies spell (a root task: the whole repo; clerk's `E2E_*`), noted once per entry, and one nothing spells is a todo; `*` is the only wildcard, as in Turbo, so `\*`, a leading `\!`, `?` and `[` are literal: unkey's `NEXT_PUBLIC_\*` names one variable, which vx cannot key, so it is dropped and reported once for the workspace with the count of tasks naming it (openstatus's root `build` env `\*` had written one todo into each of 45 configs; `turbo()`: only when the run's environment sets it, as Turbo hashes nothing otherwise); and a `!` entry (openstatus' `!NEXT_PUBLIC_VERCEL_URL`) removes the names it matches from its list, with no todo, since vx matches no name it is not given. A name both a global list and the task's own list carry is listed once. A name that is no shell variable name (`p-q`) is keyed but left out of `exec.env.passThrough`, as `sh` drops it before the task runs.
|
|
67
67
|
- Framework inference (Turbo hashes and passes Next's `NEXT_PUBLIC_*`, Vite's `VITE_*`, … for a package that depends on the framework): `turbo()` expands the prefix over the run's environment; the CLI writes the names the tracked source and `.env.example` of the package and its workspace dependencies spell (the bare prefix, as a `startsWith` test spells it, is none) (Next bundles theirs: cal.com's web, 23 alone, 58 with them), and a note asks for any only an installed dependency reads.
|
|
68
68
|
|
|
@@ -550,18 +550,19 @@ export default defineWorkspace({
|
|
|
550
550
|
|
|
551
551
|
Off CI a run reads the remote and does not write it unless its policy names the remote. Give the write token only to the default branch's CI and a read-only one to laptops and pull request jobs: the server's token is the boundary (`docs/security.md` § Cache poisoning). The same holds for `nxCache()`.
|
|
552
552
|
|
|
553
|
-
Every option falls back to the tool's own environment variable, so a self-hosted setup carries over unchanged. A token with no `apiUrl` means Vercel's hosted Remote Cache (`https://vercel.com/api`), exactly as it does for `turbo` — so `npx turbo login && npx turbo link`, then `turboCache()` with `TURBO_TOKEN` / `TURBO_TEAM` set, is the whole hosted setup. Below the environment, in Turbo's order: a Vercel build's `VERCEL_ARTIFACTS_TOKEN` and `VERCEL_ARTIFACTS_OWNER` (the team id, where `TURBO_TOKEN` with a team is not set), then the repo's `.turbo/config.json` (what `turbo link` writes: `apiUrl`, `teamId`, `teamSlug`, `token`), then the root `turbo.json`'s `remoteCache` (`apiUrl`, `teamId`, `teamSlug`), whose `enabled: false` declines unless the options name a cache. With no token the plugin **declines** and the run stays local.
|
|
553
|
+
Every option falls back to the tool's own environment variable, so a self-hosted setup carries over unchanged. A token with no `apiUrl` means Vercel's hosted Remote Cache (`https://vercel.com/api`), exactly as it does for `turbo` — so `npx turbo login && npx turbo link`, then `turboCache()` with `TURBO_TOKEN` / `TURBO_TEAM` set, is the whole hosted setup. Below the environment, in Turbo's order: a Vercel build's `VERCEL_ARTIFACTS_TOKEN` and `VERCEL_ARTIFACTS_OWNER` (the team id, where `TURBO_TOKEN` with a team is not set), then the repo's `.turbo/config.json` (what `turbo link` writes: `apiUrl`, `teamId`, `teamSlug`, `token`), then the root `turbo.json`'s `remoteCache` (`apiUrl`, `teamId`, `teamSlug`), whose `enabled: false` declines unless the options name a cache. A team with no `apiUrl` also means Vercel's. With a URL or a team but no token, the token comes from the OS keychain ([Tokens](#tokens-and-the-os-keychain)); with none of the three the plugin **declines** and the run stays local.
|
|
554
554
|
|
|
555
555
|
| Option | Environment variable | Meaning |
|
|
556
556
|
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
557
557
|
| `apiUrl` | `TURBO_API` | base URL of the cache server; default with a token: `https://vercel.com/api`; a `user:pass@` in it is refused |
|
|
558
|
-
| `token` | `TURBO_TOKEN` | Bearer token on every request
|
|
558
|
+
| `token` | `TURBO_TOKEN` | Bearer token on every request; then OIDC, `.turbo/config.json`, the OS keychain |
|
|
559
|
+
| `oidc` | `ACTIONS_ID_TOKEN_REQUEST_URL`, `ACTIONS_ID_TOKEN_REQUEST_TOKEN` (GitHub sets them) | `true` or `{ tokenUrl, revokeUrl, audience, params }` (`TurboOidcOptions`): exchange the GitHub Actions job's OIDC token for a short-lived one ([OIDC](#oidc-on-github-actions)) |
|
|
559
560
|
| `teamId` | `TURBO_TEAMID` | `teamId` query parameter; required with `signatureKey` |
|
|
560
561
|
| `teamSlug` | `TURBO_TEAM` | `slug` query parameter |
|
|
561
562
|
| `signatureKey` | `TURBO_REMOTE_CACHE_SIGNATURE_KEY`, read only under turbo.json's `remoteCache.signature: true` (or `TURBO_SIGNATURE`), as Turbo reads it | HMAC-SHA256 key (≥ 32 bytes, used raw); a download whose tag does not verify is a miss |
|
|
562
563
|
| `timeoutMs` | `TURBO_REMOTE_CACHE_TIMEOUT`, then `remoteCache.timeout` (seconds) | HEAD/GET/POST deadline (default 30 s; 0 none) |
|
|
563
564
|
| `uploadTimeoutMs` | `TURBO_REMOTE_CACHE_UPLOAD_TIMEOUT`, then `remoteCache.uploadTimeout` (seconds) | PUT deadline (default 60 s; 0 none) |
|
|
564
|
-
| `retries` | — | resends of a request answered
|
|
565
|
+
| `retries` | — | resends of a request answered 5xx (not 501) or never connected (default 1, Turbo's); a 429 waits out the limit; 0 turns both off |
|
|
565
566
|
| `preflight` | `TURBO_PREFLIGHT` (1 or 0), then `remoteCache.preflight` | send Turbo's `OPTIONS` preflight before each artifact request and go where its `Location` points (relative to `apiUrl`); the token goes along only when the API's `Access-Control-Allow-Headers` admits `Authorization`; a redirected preflight is not followed (default off) |
|
|
566
567
|
|
|
567
568
|
The signature is Turbo's current scheme (`artifact-signature:v2`: prefix, hash, team id and body, each length-prefixed, under HMAC-SHA256, base64 in `x-artifact-tag`). A signed body is written to a temp file before its tag can be checked, so one past core's artifact ceiling (2 GiB, at zstd's bound) is refused as it passes it, and is a miss.
|
|
@@ -581,22 +582,47 @@ export default defineWorkspace({
|
|
|
581
582
|
})
|
|
582
583
|
```
|
|
583
584
|
|
|
584
|
-
Every option falls back to the tool's own environment variable; with
|
|
585
|
+
Every option falls back to the tool's own environment variable, and a missing token to the OS keychain ([Tokens](#tokens-and-the-os-keychain)); with no server the plugin **declines** and the run stays local.
|
|
585
586
|
|
|
586
|
-
| Option | Environment variable | Meaning
|
|
587
|
-
| ------------- | ------------------------------------------ |
|
|
588
|
-
| `server` | `NX_SELF_HOSTED_REMOTE_CACHE_SERVER` | base URL of the cache server; a `user:pass@` in it is refused
|
|
589
|
-
| `accessToken` | `NX_SELF_HOSTED_REMOTE_CACHE_ACCESS_TOKEN` | Bearer token;
|
|
590
|
-
| `timeoutMs` | — | per-request deadline (default 30 s; a positive number of ms)
|
|
591
|
-
| `retries` | — | resends, as `turboCache()`'s (default 1)
|
|
587
|
+
| Option | Environment variable | Meaning |
|
|
588
|
+
| ------------- | ------------------------------------------ | ----------------------------------------------------------------------------- |
|
|
589
|
+
| `server` | `NX_SELF_HOSTED_REMOTE_CACHE_SERVER` | base URL of the cache server; a `user:pass@` in it is refused |
|
|
590
|
+
| `accessToken` | `NX_SELF_HOSTED_REMOTE_CACHE_ACCESS_TOKEN` | Bearer token, else the OS keychain; `''` (option or variable): an open server |
|
|
591
|
+
| `timeoutMs` | — | per-request deadline (default 30 s; a positive number of ms) |
|
|
592
|
+
| `retries` | — | resends, as `turboCache()`'s (default 1) |
|
|
592
593
|
|
|
593
594
|
The Nx spec has no existence probe, so `has` (the `--dry` prediction; the prefetch pass calls `get`) is a `GET` whose body is cancelled before it answers. The wire carries no producing-task duration, so a remote hit reports none.
|
|
594
595
|
|
|
596
|
+
## OIDC on GitHub Actions
|
|
597
|
+
|
|
598
|
+
`turboCache({ oidc: true })` needs no `TURBO_TOKEN` secret on GitHub Actions: before the first task, when the run's policy uses the remote, it asks GitHub for the job's OIDC token (`ACTIONS_ID_TOKEN_REQUEST_URL`, with `audience` when given), exchanges it at the token endpoint (RFC 8693) for a short-lived cache token, and revokes that (RFC 7009) when the run ends. With Vercel's hosted cache the endpoints, Vercel's OAuth client id and `team_id_or_slug` (the team id, else slug; one is required) are the defaults, as Vercel's own action sends them; `params` adds fields such as `policy_id`. Any other server names its `tokenUrl` (and `revokeUrl`, or none); `params` carries its extras. `revokeUrl: false` skips the revoke.
|
|
599
|
+
|
|
600
|
+
```yaml
|
|
601
|
+
permissions:
|
|
602
|
+
id-token: write
|
|
603
|
+
```
|
|
604
|
+
|
|
605
|
+
- Order: the `token` option, `TURBO_TOKEN`, then OIDC, then `.turbo/config.json`, then the OS keychain.
|
|
606
|
+
- A request is tried three times on a `429`, a `5xx` or no answer, 30 s each. A failed exchange is one warning and a local-only run; a failed revoke is a warning (the token expires on its own). No message quotes a token.
|
|
607
|
+
- A job without `id-token: write` gets no OIDC variables: one warning, and the sources below OIDC answer. A pull request from a fork, and a Dependabot one, never gets an OIDC token.
|
|
608
|
+
- `nxCache()` has no OIDC: Nx's cache spec names no exchange.
|
|
609
|
+
|
|
610
|
+
## Tokens and the OS keychain
|
|
611
|
+
|
|
612
|
+
With no token from an option, a variable or a config file, both plugins read the OS keychain through core's `ctx.secret` (`Bun.secrets`: macOS Keychain, libsecret, Windows Credential Manager), service `@vzn/vx-migrate`, entry named by the endpoint and team: `https://remote.cachely.dev/v8/artifacts#cachely` (`turboCache()`, team id else slug; no `#…` without a team) or `https://cache.example.com/v1/cache` (`nxCache()`). So separate caches never share a token.
|
|
613
|
+
|
|
614
|
+
- Read once per run, only by a run that uses the remote: before its first task, when its policy reads or writes the remote. `--no-cache`, `--cache-policy=remote:`, `--dry`, `vx show` and `vx info` never touch the keychain. Off CI, remote writes are already off by default ([`cli.md` § Cache control](../vx/docs/cli.md#cache-control---cache-policy---no-cache---force)).
|
|
615
|
+
- Missing, on a terminal (stdin and stderr TTYs, `CI` unset): asked once, input hidden — `Token for remote.cachely.dev (cachely), saved to the OS keychain: ` — and saved. Later runs read it without asking.
|
|
616
|
+
- Missing anywhere else (CI, piped), or an empty answer (Ctrl-C stops the run): one warning naming the remote and the fixes (set `TURBO_TOKEN` / `NX_SELF_HOSTED_REMOTE_CACHE_ACCESS_TOKEN`, run in a terminal, or `--cache-policy=remote:` to skip the remote), and the run goes on with the local cache.
|
|
617
|
+
- A keychain token the server refuses (`401` / `403`) is deleted, with the warning, so the next run asks again. A token from an option or a variable is never deleted.
|
|
618
|
+
- No keychain (headless Linux without libsecret) reads as empty; it never breaks a run.
|
|
619
|
+
- The token never reaches a task: the task environment does not pass `TURBO_TOKEN`, `VERCEL_ARTIFACTS_TOKEN` or Nx's variable.
|
|
620
|
+
|
|
595
621
|
## Remote-cache behaviour (both)
|
|
596
622
|
|
|
597
623
|
- A remote error degrades to a **miss** and a warning that names the request, the artifact and the server — `[vx] plugin '@vzn/vx-migrate': upload 32248a2a7c89e241 to https://cache.example.com/v8/artifacts failed: no answer within 60000 ms` — never the token. The run never fails because of the cache.
|
|
598
624
|
- The same failure is said once per run: an unreachable server fails the probe, the download and the upload alike, and the run prints the first and, at its end, `[vx] plugin '@vzn/vx-migrate': 2 more requests failed the same way: Unable to connect. Is the computer able to access the url?` (core does the counting, for every cache plugin).
|
|
599
|
-
- A request answered `
|
|
625
|
+
- A request answered `5xx` (not `501`), or one that never connected (a refused port, an unresolved host), is sent again after 2 s, `retries` times. A `429` pauses every request to that remote for its `Retry-After` (capped at 10 s a wait, 2 s without one), and the request is sent again when the pause ends, for up to 20 s from its first `429`, as Turbo's client does. A spent deadline is not resent: it has already cost its wait.
|
|
600
626
|
- Three outages in a row (no connection, no answer within the deadline, or a `5xx` once the resends are spent) open Turbo's outage breaker: no request is sent for 30 s, each lookup is a miss at once, then one request probes and its answer closes or reopens it. A hit is judged when its body ends, so a server that sends headers and then stalls the body counts the same way. A hung server costs three deadlines, not one per task.
|
|
601
627
|
- A refused token (`401`/`403`) warns **once** and turns the layer off for the rest of the process (a `403` on an upload is a read-only token — Nx's spec, or turborepo-remote-cache's `READ_ONLY` and write-less JWTs: it turns off uploads alone, and reads go on) — including the requests already in flight when the refusal lands, which degrade in silence rather than repeating it (a six-project run printed five identical lines before 2026-09-20).
|
|
602
628
|
- Policy (`--cache-policy=remote:r`, `VX_CACHE_POLICY`, `cache.policy`) is enforced by core, which layers each plugin's remote over the local cache — a read-only token pairs naturally with `remote:r`.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@vzn/vx-migrate",
|
|
3
|
-
"version": "0.0.
|
|
3
|
+
"version": "0.0.642",
|
|
4
4
|
"license": "MIT",
|
|
5
5
|
"description": "Adopt @vzn/vx from Turborepo, Nx or package.json scripts: run the repo unchanged, migrate it to vx.config.ts, and keep your Turbo or Nx remote cache",
|
|
6
6
|
"keywords": [
|
|
@@ -40,7 +40,7 @@
|
|
|
40
40
|
"access": "public"
|
|
41
41
|
},
|
|
42
42
|
"peerDependencies": {
|
|
43
|
-
"@vzn/vx": "^0.0.
|
|
43
|
+
"@vzn/vx": "^0.0.642"
|
|
44
44
|
},
|
|
45
45
|
"repository": {
|
|
46
46
|
"type": "git",
|
package/src/migrate-turbo.ts
CHANGED
|
@@ -24,6 +24,7 @@ import { DOTENV_GLOBS_HEAD, DOTENV_PROBE, DOTENV_PROBE_TOP } from './dotenv-prob
|
|
|
24
24
|
import { adoptedToolNotes } from './workspace-notes.js'
|
|
25
25
|
import { spelledNames } from './spelled-env.js'
|
|
26
26
|
import { sealsConfig } from './sealed-tsconfig.js'
|
|
27
|
+
import { onceReader, tscBuildInfo } from './tsc-buildinfo.js'
|
|
27
28
|
|
|
28
29
|
/** What a task's `npm_package_*` read: the manifest, so a bump reaches them. */
|
|
29
30
|
const MANIFEST_IMPORT = "import pkg from './package.json' with { type: 'json' }"
|
|
@@ -46,6 +47,7 @@ export async function migrateTurbo(
|
|
|
46
47
|
format: MigrationFormat = 'ts',
|
|
47
48
|
): Promise<MigrationPlan> {
|
|
48
49
|
const tracked = await trackedFiles(root)
|
|
50
|
+
const tsc = onceReader()
|
|
49
51
|
const mapping = await mapTurboWorkspace(root, await withRootProject(root, metas), {
|
|
50
52
|
splice: (kind) => [{ raw: `...${PRESET_NAMES[kind]}` }],
|
|
51
53
|
persistentTodo: PERSISTENT_TODO,
|
|
@@ -55,6 +57,7 @@ export async function migrateTurbo(
|
|
|
55
57
|
ownConfig: () => `vx.config.${format}`,
|
|
56
58
|
sourceNames: (dirs) => spelledNames(root, dirs, tracked),
|
|
57
59
|
ignored: (rels) => gitIgnored(root, rels),
|
|
60
|
+
buildInfo: (rel, command) => tscBuildInfo(tsc.read, path.join(root, rel), command),
|
|
58
61
|
})
|
|
59
62
|
// A reader whose `!` hides a committed file under another task's output
|
|
60
63
|
// runs uncached, as under turbo().
|
package/src/migrate.ts
CHANGED
|
@@ -674,18 +674,23 @@ export function adoptionNext(
|
|
|
674
674
|
return `${installArgv(root, pm, pinned).join(' ')} && ${vx} lock && ${run}`
|
|
675
675
|
}
|
|
676
676
|
|
|
677
|
-
/**
|
|
677
|
+
/**
|
|
678
|
+
* `build` when the runner's config names it, else the first task name it
|
|
679
|
+
* names. A key with `:` is an Nx executor (`@angular/build:application`)
|
|
680
|
+
* and one with `#` a Turbo package task: neither is a task to run `--all`.
|
|
681
|
+
*/
|
|
678
682
|
function firstTask(file: string, runner: 'turbo' | 'nx'): string {
|
|
679
|
-
let
|
|
683
|
+
let json: Record<string, unknown> | null
|
|
680
684
|
try {
|
|
681
|
-
|
|
685
|
+
json = Bun.JSONC.parse(readFileSync(file, 'utf8')) as Record<string, unknown> | null
|
|
682
686
|
} catch {
|
|
683
687
|
return 'build'
|
|
684
688
|
}
|
|
685
|
-
|
|
686
|
-
|
|
687
|
-
const
|
|
688
|
-
|
|
689
|
+
const field =
|
|
690
|
+
runner === 'turbo' ? (json?.['tasks'] ?? json?.['pipeline']) : json?.['targetDefaults']
|
|
691
|
+
const names = Object.keys(typeof field === 'object' && field !== null ? field : {})
|
|
692
|
+
if (names.includes('build')) return 'build'
|
|
693
|
+
return names.find((n) => !n.includes(':') && !n.includes('#')) ?? 'build'
|
|
689
694
|
}
|
|
690
695
|
|
|
691
696
|
/**
|
package/src/nx/nx-map.ts
CHANGED
|
@@ -563,6 +563,11 @@ export async function mapNxWorkspace(
|
|
|
563
563
|
followNxGraph(t.task, nodeName, meta.name, nxEdges, metaByNode, emittedIds, reached)
|
|
564
564
|
}
|
|
565
565
|
dropUnheldDeps(t, emitted, emittedIds)
|
|
566
|
+
const reader = dependentReaders.get(t)
|
|
567
|
+
if (reader !== undefined && !reachesProducer(t, meta.name, emittedIds, reached)) {
|
|
568
|
+
delete t.task!['cache']
|
|
569
|
+
t.todos.push(orphanReaderTodo(reader.globs, reader.dependsOn))
|
|
570
|
+
}
|
|
566
571
|
}
|
|
567
572
|
const tags = nxTags(nodeName === undefined ? undefined : nodeMap[nodeName])
|
|
568
573
|
projects.push({
|
|
@@ -895,9 +900,7 @@ function buildTask(
|
|
|
895
900
|
const orphanReader =
|
|
896
901
|
inputs.dependentOutputs.length > 0 && !deps.some((d) => !d.includes('#nx-input:'))
|
|
897
902
|
if (orphanReader && cacheWanted && !persistent) {
|
|
898
|
-
todos.push(
|
|
899
|
-
`reads ${inputs.dependentOutputs.map((g) => `\`${g}\``).join(', ')} from the outputs of the tasks it depends on, and no task it depends on is migrated (dependsOn ${JSON.stringify(target.dependsOn ?? [])}) — uncached here, so it never replays output generated from an older input; migrate the producer and depend on it to cache this task`,
|
|
900
|
-
)
|
|
903
|
+
todos.push(orphanReaderTodo(inputs.dependentOutputs, target.dependsOn))
|
|
901
904
|
}
|
|
902
905
|
const cacheEnabled = !persistent && cacheWanted && wild === undefined && !orphanReader
|
|
903
906
|
if (persistent && cacheWanted) {
|
|
@@ -1007,7 +1010,41 @@ function buildTask(
|
|
|
1007
1010
|
task.cache = { outputs }
|
|
1008
1011
|
}
|
|
1009
1012
|
|
|
1010
|
-
|
|
1013
|
+
const generated: GeneratedTask = { name: variant.name, todos, task }
|
|
1014
|
+
if (cacheEnabled && inputs.dependentOutputs.length > 0)
|
|
1015
|
+
dependentReaders.set(generated, { globs: inputs.dependentOutputs, dependsOn: target.dependsOn })
|
|
1016
|
+
return generated
|
|
1017
|
+
}
|
|
1018
|
+
|
|
1019
|
+
/**
|
|
1020
|
+
* The cached readers of `dependentTasksOutputFiles`, judged again once the
|
|
1021
|
+
* graph has its final edges: a `^build` kept because another project builds
|
|
1022
|
+
* reached no producer of this one (nx-examples' codegen, its .NET
|
|
1023
|
+
* ProductsApi no node once @nx/dotnet failed to load), and the key saw
|
|
1024
|
+
* nothing the task reads (Growth, nx-codegen-dropped-producer.sh).
|
|
1025
|
+
*/
|
|
1026
|
+
const dependentReaders = new WeakMap<GeneratedTask, { globs: string[]; dependsOn: unknown }>()
|
|
1027
|
+
|
|
1028
|
+
function orphanReaderTodo(globs: readonly string[], dependsOn: unknown): string {
|
|
1029
|
+
return `reads ${globs.map((g) => `\`${g}\``).join(', ')} from the outputs of the tasks it depends on, and no task it depends on is migrated (dependsOn ${JSON.stringify(dependsOn ?? [])}) — uncached here, so it never replays output generated from an older input; migrate the producer and depend on it to cache this task`
|
|
1030
|
+
}
|
|
1031
|
+
|
|
1032
|
+
/** Whether one of `t`'s final edges reaches a task vx runs. */
|
|
1033
|
+
function reachesProducer(
|
|
1034
|
+
t: GeneratedTask,
|
|
1035
|
+
project: string,
|
|
1036
|
+
emittedIds: ReadonlySet<string>,
|
|
1037
|
+
reached: (name: string) => ReadonlySet<string>,
|
|
1038
|
+
): boolean {
|
|
1039
|
+
const deps = t.task?.['dependsOn']
|
|
1040
|
+
if (!Array.isArray(deps)) return false
|
|
1041
|
+
return deps.some((d) => {
|
|
1042
|
+
if (typeof d !== 'string' || d.includes('*')) return true
|
|
1043
|
+
if (d.includes('#nx-input:')) return false
|
|
1044
|
+
if (d.startsWith('^'))
|
|
1045
|
+
return [...reached(project)].some((p) => emittedIds.has(`${p}#${d.slice(1)}`))
|
|
1046
|
+
return d.includes('#') || emittedIds.has(`${project}#${d}`)
|
|
1047
|
+
})
|
|
1011
1048
|
}
|
|
1012
1049
|
|
|
1013
1050
|
/**
|
package/src/nx-cache/index.ts
CHANGED
|
@@ -14,19 +14,29 @@
|
|
|
14
14
|
import {
|
|
15
15
|
definePlugin,
|
|
16
16
|
type RemoteCacheLayer,
|
|
17
|
+
type RemoteOpenContext,
|
|
17
18
|
type VxPlugin,
|
|
18
19
|
refuseUnknownOptions,
|
|
19
20
|
type PluginOptionKinds,
|
|
20
21
|
} from '@vzn/vx'
|
|
21
22
|
import { deadline } from '../remote-deadline.js'
|
|
22
23
|
import { OutageBreaker } from '../remote-breaker.js'
|
|
23
|
-
import { withRetry } from '../remote-retry.js'
|
|
24
|
-
import {
|
|
24
|
+
import { Cooldown, withRetry } from '../remote-retry.js'
|
|
25
|
+
import {
|
|
26
|
+
headerValueFault,
|
|
27
|
+
type KeychainRemote,
|
|
28
|
+
keychainToken,
|
|
29
|
+
remoteLabel,
|
|
30
|
+
} from '../remote-token.js'
|
|
25
31
|
|
|
26
32
|
export interface NxCacheOptions {
|
|
27
33
|
/** Base URL of the cache server, or `NX_SELF_HOSTED_REMOTE_CACHE_SERVER`. */
|
|
28
34
|
server?: string
|
|
29
|
-
/**
|
|
35
|
+
/**
|
|
36
|
+
* Bearer token, or `NX_SELF_HOSTED_REMOTE_CACHE_ACCESS_TOKEN`, then the OS
|
|
37
|
+
* keychain. `''` (option or variable) is a server that runs open: no
|
|
38
|
+
* token, nothing asked.
|
|
39
|
+
*/
|
|
30
40
|
accessToken?: string
|
|
31
41
|
/** Per-request deadline (default 30 s). */
|
|
32
42
|
timeoutMs?: number
|
|
@@ -36,6 +46,7 @@ export interface NxCacheOptions {
|
|
|
36
46
|
|
|
37
47
|
export interface NxCacheConfig {
|
|
38
48
|
server: string
|
|
49
|
+
/** Undefined: the layer's `open` asks the OS keychain; `''`: an open server. */
|
|
39
50
|
accessToken?: string
|
|
40
51
|
timeoutMs: number
|
|
41
52
|
retries: number
|
|
@@ -58,7 +69,7 @@ export function resolveNxCacheConfig(
|
|
|
58
69
|
)
|
|
59
70
|
}
|
|
60
71
|
const accessToken = options.accessToken ?? env['NX_SELF_HOSTED_REMOTE_CACHE_ACCESS_TOKEN']
|
|
61
|
-
const fault = accessToken ? headerValueFault(accessToken)
|
|
72
|
+
const fault = accessToken === undefined ? null : headerValueFault(accessToken)
|
|
62
73
|
if (fault !== null)
|
|
63
74
|
throw new Error(
|
|
64
75
|
`vx/nx-cache: the access token holds ${fault}, which no HTTP header can carry — check the secret (it is not printed)`,
|
|
@@ -75,7 +86,7 @@ export function resolveNxCacheConfig(
|
|
|
75
86
|
)
|
|
76
87
|
return {
|
|
77
88
|
server,
|
|
78
|
-
...(accessToken ? { accessToken } : {}),
|
|
89
|
+
...(accessToken !== undefined ? { accessToken } : {}),
|
|
79
90
|
timeoutMs: options.timeoutMs ?? 30_000,
|
|
80
91
|
retries,
|
|
81
92
|
}
|
|
@@ -106,12 +117,31 @@ export class NxRemoteCache implements RemoteCacheLayer {
|
|
|
106
117
|
*/
|
|
107
118
|
private writesRefused = false
|
|
108
119
|
readonly endpoint: string
|
|
120
|
+
/** The remote's rate limit, which every request of this client waits out. */
|
|
121
|
+
private readonly cooldown = new Cooldown()
|
|
122
|
+
/** Undefined until `open` reads the keychain; a layer never opened (a plan) asks the server nothing. */
|
|
123
|
+
private token: string | undefined
|
|
124
|
+
/** Set by `open` when the token came from the keychain: a refusal deletes it. */
|
|
125
|
+
private forget: (() => Promise<void>) | undefined
|
|
126
|
+
private readonly keychainRemote: KeychainRemote
|
|
109
127
|
constructor(
|
|
110
128
|
private readonly config: NxCacheConfig,
|
|
111
129
|
private readonly fetchImpl: typeof fetch = fetch,
|
|
112
130
|
private readonly wait: (ms: number) => Promise<void> = Bun.sleep,
|
|
113
131
|
) {
|
|
114
132
|
this.endpoint = `${config.server}/v1/cache`
|
|
133
|
+
this.token = config.accessToken
|
|
134
|
+
this.keychainRemote = {
|
|
135
|
+
key: this.endpoint,
|
|
136
|
+
label: remoteLabel(config.server, undefined),
|
|
137
|
+
env: 'NX_SELF_HOSTED_REMOTE_CACHE_ACCESS_TOKEN',
|
|
138
|
+
}
|
|
139
|
+
}
|
|
140
|
+
|
|
141
|
+
async open(ctx: RemoteOpenContext): Promise<void> {
|
|
142
|
+
if (this.token !== undefined) return
|
|
143
|
+
this.token = await keychainToken(ctx, this.keychainRemote)
|
|
144
|
+
this.forget = () => ctx.forgetSecret({ key: this.keychainRemote.key })
|
|
115
145
|
}
|
|
116
146
|
|
|
117
147
|
/**
|
|
@@ -125,8 +155,9 @@ export class NxRemoteCache implements RemoteCacheLayer {
|
|
|
125
155
|
hash: string,
|
|
126
156
|
body?: Blob,
|
|
127
157
|
): Promise<Response | undefined> {
|
|
158
|
+
if (this.token === undefined) return undefined
|
|
128
159
|
const headers: Record<string, string> = {}
|
|
129
|
-
if (this.
|
|
160
|
+
if (this.token !== '') headers['Authorization'] = `Bearer ${this.token}`
|
|
130
161
|
if (body !== undefined) {
|
|
131
162
|
headers['Content-Type'] = 'application/octet-stream'
|
|
132
163
|
headers['Content-Length'] = String(body.size)
|
|
@@ -148,6 +179,7 @@ export class NxRemoteCache implements RemoteCacheLayer {
|
|
|
148
179
|
}),
|
|
149
180
|
this.config.retries,
|
|
150
181
|
this.wait,
|
|
182
|
+
this.cooldown,
|
|
151
183
|
),
|
|
152
184
|
method === 'GET',
|
|
153
185
|
)
|
|
@@ -163,8 +195,15 @@ export class NxRemoteCache implements RemoteCacheLayer {
|
|
|
163
195
|
const first = !this.disabled
|
|
164
196
|
this.disabled = true
|
|
165
197
|
if (!first) return undefined
|
|
198
|
+
const why =
|
|
199
|
+
res.status === 401
|
|
200
|
+
? 'missing or invalid token'
|
|
201
|
+
: 'access forbidden (the token may not read)'
|
|
202
|
+
if (this.forget === undefined)
|
|
203
|
+
throw new Error(`HTTP ${res.status}: ${why}; remote cache off for this run`)
|
|
204
|
+
await this.forget()
|
|
166
205
|
throw new Error(
|
|
167
|
-
`HTTP ${res.status}: ${
|
|
206
|
+
`HTTP ${res.status}: ${why}; removed it from the OS keychain, so the next run asks again; remote cache off for this run`,
|
|
168
207
|
)
|
|
169
208
|
}
|
|
170
209
|
return res
|
package/src/remote-cache.ts
CHANGED
|
@@ -109,7 +109,7 @@ export function remoteCacheNote(
|
|
|
109
109
|
const cache = REMOTE_CACHE[runner]
|
|
110
110
|
if (signal !== undefined) {
|
|
111
111
|
return declared
|
|
112
|
-
? `${cache}(): ${signal}, so vx shares that remote cache (
|
|
112
|
+
? `${cache}(): ${signal}, so vx shares that remote cache (token: the variable, else the OS keychain).`
|
|
113
113
|
: `${signal}: add ${cache}() from @vzn/vx-migrate to its plugins and vx shares that remote cache.`
|
|
114
114
|
}
|
|
115
115
|
return runner === 'nx' && usesNxCloud(root)
|
package/src/remote-retry.ts
CHANGED
|
@@ -1,40 +1,80 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* When a remote's rate limit ends, shared by every request to that remote:
|
|
3
|
+
* Turbo's `RateLimit` (`turborepo-api-client/src/retry.rs`; tests
|
|
4
|
+
* `waits_for_known_cooldown_before_initial_send`,
|
|
5
|
+
* `scopes_artifact_cooldowns_to_client_team_and_token`). A request that
|
|
6
|
+
* starts inside it waits it out before its first send instead of spending
|
|
7
|
+
* a try on a 429. One per wire instance: a client, its server and token.
|
|
8
|
+
*/
|
|
9
|
+
export class Cooldown {
|
|
10
|
+
until = 0
|
|
11
|
+
constructor(private readonly now: () => number = Date.now) {}
|
|
12
|
+
|
|
13
|
+
/** Ms left of the limit, 0 when there is none. */
|
|
14
|
+
left(): number {
|
|
15
|
+
return Math.max(0, this.until - this.now())
|
|
16
|
+
}
|
|
17
|
+
|
|
18
|
+
extend(ms: number): void {
|
|
19
|
+
this.until = Math.max(this.until, this.now() + ms)
|
|
20
|
+
}
|
|
21
|
+
}
|
|
22
|
+
|
|
1
23
|
/**
|
|
2
24
|
* One request with Turbo's retry rule (`turborepo-api-client/src/retry.rs`):
|
|
3
|
-
* a
|
|
4
|
-
*
|
|
5
|
-
*
|
|
6
|
-
*
|
|
7
|
-
*
|
|
8
|
-
*
|
|
9
|
-
*
|
|
10
|
-
*
|
|
25
|
+
* a 5xx other than 501, or a request that never connected, is sent again
|
|
26
|
+
* after 2 s, `retries` times. A 429 opens the remote's `cooldown` for its
|
|
27
|
+
* `Retry-After` (capped at 10 s a wait, 2 s without one), and the request
|
|
28
|
+
* is resent once it ends, as often as fits in 20 s from its first 429: a
|
|
29
|
+
* limit longer than one capped wait lost every hit before (Growth's
|
|
30
|
+
* parity run, 2026-10-16). A spent deadline is not retried, unlike
|
|
31
|
+
* Turbo's reads: it has already cost its wait, and a hung server would
|
|
32
|
+
* cost it twice. Every request both wires make is idempotent
|
|
33
|
+
* (content-addressed, and Nx's second write is a 409 the caller takes as
|
|
34
|
+
* done), so a resend applies nothing twice. Shared by `turboCache()` and
|
|
35
|
+
* `nxCache()`.
|
|
11
36
|
*/
|
|
12
37
|
export async function withRetry(
|
|
13
38
|
send: () => Promise<Response>,
|
|
14
39
|
retries: number,
|
|
15
40
|
wait: (ms: number) => Promise<void> = Bun.sleep,
|
|
41
|
+
cooldown: Cooldown = new Cooldown(),
|
|
42
|
+
now: () => number = Date.now,
|
|
16
43
|
): Promise<Response> {
|
|
17
|
-
|
|
44
|
+
let firstLimit: number | undefined
|
|
45
|
+
for (let attempt = 0; ;) {
|
|
46
|
+
const held = cooldown.left()
|
|
47
|
+
if (held > 0) await wait(held)
|
|
18
48
|
let res: Response
|
|
19
49
|
try {
|
|
20
50
|
res = await send()
|
|
21
51
|
} catch (err) {
|
|
22
52
|
if (attempt >= retries || !neverConnected(err)) throw err
|
|
53
|
+
attempt++
|
|
23
54
|
await wait(BACKOFF_MS)
|
|
24
55
|
continue
|
|
25
56
|
}
|
|
57
|
+
if (res.status === 429) {
|
|
58
|
+
firstLimit ??= now()
|
|
59
|
+
const delay = retryAfterMs(res) ?? BACKOFF_MS
|
|
60
|
+
if (retries === 0 || now() + delay - firstLimit > RATE_LIMIT_BUDGET_MS) return res
|
|
61
|
+
cooldown.extend(delay)
|
|
62
|
+
await res.body?.cancel()
|
|
63
|
+
continue
|
|
64
|
+
}
|
|
26
65
|
if (attempt >= retries || !retryable(res.status)) return res
|
|
27
|
-
|
|
66
|
+
attempt++
|
|
28
67
|
await res.body?.cancel()
|
|
29
|
-
await wait(
|
|
68
|
+
await wait(BACKOFF_MS)
|
|
30
69
|
}
|
|
31
70
|
}
|
|
32
71
|
|
|
33
72
|
const BACKOFF_MS = 2_000
|
|
34
73
|
const MAX_RETRY_AFTER_MS = 10_000
|
|
74
|
+
const RATE_LIMIT_BUDGET_MS = 20_000
|
|
35
75
|
|
|
36
76
|
function retryable(status: number): boolean {
|
|
37
|
-
return status
|
|
77
|
+
return status >= 500 && status !== 501
|
|
38
78
|
}
|
|
39
79
|
|
|
40
80
|
/** A refused port or an unresolved host, as Bun's `fetch` codes them (measured on 1.4.2). */
|
package/src/remote-token.ts
CHANGED
|
@@ -1,3 +1,5 @@
|
|
|
1
|
+
import { isSecretUnavailable, type RemoteOpenContext } from '@vzn/vx'
|
|
2
|
+
|
|
1
3
|
/**
|
|
2
4
|
* Why a string cannot be an HTTP header value, or null when it can. Bun's
|
|
3
5
|
* `fetch` refuses a value holding a line break or NUL, or a character past
|
|
@@ -16,3 +18,49 @@ export function headerValueFault(value: string): string | null {
|
|
|
16
18
|
}
|
|
17
19
|
return past ? 'a character past Latin-1' : null
|
|
18
20
|
}
|
|
21
|
+
|
|
22
|
+
/** One remote's keychain entry: `key` names it, `label` and `env` are what a person reads. */
|
|
23
|
+
export interface KeychainRemote {
|
|
24
|
+
readonly key: string
|
|
25
|
+
readonly label: string
|
|
26
|
+
/** The variable that sets the token instead, named in the warning. */
|
|
27
|
+
readonly env: string
|
|
28
|
+
}
|
|
29
|
+
|
|
30
|
+
/**
|
|
31
|
+
* The token from the plugin's OS keychain, asked for on a terminal when
|
|
32
|
+
* missing (core's `ctx.secret`): the last source, after the option, the
|
|
33
|
+
* variable and the config files. A throw is the one warning that leaves the
|
|
34
|
+
* run on the local cache.
|
|
35
|
+
*/
|
|
36
|
+
export async function keychainToken(
|
|
37
|
+
ctx: RemoteOpenContext,
|
|
38
|
+
remote: KeychainRemote,
|
|
39
|
+
): Promise<string> {
|
|
40
|
+
let token: string
|
|
41
|
+
try {
|
|
42
|
+
token = await ctx.secret({
|
|
43
|
+
key: remote.key,
|
|
44
|
+
prompt: `Token for ${remote.label}, saved to the OS keychain: `,
|
|
45
|
+
})
|
|
46
|
+
} catch (err) {
|
|
47
|
+
if (!isSecretUnavailable(err)) throw err
|
|
48
|
+
throw new Error(
|
|
49
|
+
`no token for ${remote.label}: set ${remote.env}, run vx in a terminal to be asked for one (kept in the OS keychain), or pass --cache-policy=remote: to skip the remote; local cache only this run`,
|
|
50
|
+
)
|
|
51
|
+
}
|
|
52
|
+
const fault = headerValueFault(token)
|
|
53
|
+
if (fault !== null) {
|
|
54
|
+
await ctx.forgetSecret({ key: remote.key })
|
|
55
|
+
throw new Error(
|
|
56
|
+
`the OS keychain's token for ${remote.label} holds ${fault}, which no HTTP header can carry (it is not printed); removed it, so the next run asks again`,
|
|
57
|
+
)
|
|
58
|
+
}
|
|
59
|
+
return token.trim()
|
|
60
|
+
}
|
|
61
|
+
|
|
62
|
+
/** How a remote reads in a prompt or a warning: its host, then its team. */
|
|
63
|
+
export function remoteLabel(url: string, team: string | undefined): string {
|
|
64
|
+
const host = URL.canParse(url) ? new URL(url).host : url
|
|
65
|
+
return team === undefined ? host : `${host} (${team})`
|
|
66
|
+
}
|
|
@@ -0,0 +1,168 @@
|
|
|
1
|
+
// Where `tsc` keeps its incremental state. tsc reads its `.tsbuildinfo`
|
|
2
|
+
// and re-emits only the files it calls changed. vx empties a task's
|
|
3
|
+
// outputs before a miss, so a `tsc -b` whose buildinfo sits outside them
|
|
4
|
+
// (rootDir `src`, outDir `dist`: `tsconfig.tsbuildinfo` beside the
|
|
5
|
+
// tsconfig) re-emitted the edited file into an empty `dist/`, and vx cached
|
|
6
|
+
// that partial `dist/`. Turbo never cleans, so its `dist/` stayed whole; Nx
|
|
7
|
+
// infers the buildinfo as an output. A mapped task declares it too.
|
|
8
|
+
|
|
9
|
+
import { readFileSync } from 'node:fs'
|
|
10
|
+
import path from 'node:path'
|
|
11
|
+
|
|
12
|
+
/** A file's text, or null when it cannot be read. */
|
|
13
|
+
export type ReadText = (file: string) => string | null
|
|
14
|
+
|
|
15
|
+
interface Options {
|
|
16
|
+
composite?: unknown
|
|
17
|
+
incremental?: unknown
|
|
18
|
+
tsBuildInfoFile?: unknown
|
|
19
|
+
outDir?: unknown
|
|
20
|
+
rootDir?: unknown
|
|
21
|
+
outFile?: unknown
|
|
22
|
+
}
|
|
23
|
+
|
|
24
|
+
interface TsConfig {
|
|
25
|
+
extends?: unknown
|
|
26
|
+
references?: unknown
|
|
27
|
+
compilerOptions?: Options
|
|
28
|
+
}
|
|
29
|
+
|
|
30
|
+
function parse(read: ReadText, file: string): TsConfig | null {
|
|
31
|
+
const text = read(file)
|
|
32
|
+
if (text === null) return null
|
|
33
|
+
try {
|
|
34
|
+
const v = Bun.JSONC.parse(text)
|
|
35
|
+
return v !== null && typeof v === 'object' && !Array.isArray(v) ? (v as TsConfig) : null
|
|
36
|
+
} catch {
|
|
37
|
+
return null
|
|
38
|
+
}
|
|
39
|
+
}
|
|
40
|
+
|
|
41
|
+
/** The file an `extends` entry names, as tsc resolves it (a package through `node_modules`). */
|
|
42
|
+
function extended(read: ReadText, from: string, ext: string): string | null {
|
|
43
|
+
const withJson = (p: string) => (p.endsWith('.json') ? p : `${p}.json`)
|
|
44
|
+
if (ext.startsWith('.') || path.isAbsolute(ext)) return withJson(path.resolve(from, ext))
|
|
45
|
+
for (let dir = from; ; dir = path.dirname(dir)) {
|
|
46
|
+
const base = path.join(dir, 'node_modules', ext)
|
|
47
|
+
for (const f of [withJson(base), path.join(base, 'tsconfig.json')])
|
|
48
|
+
if (read(f) !== null) return f
|
|
49
|
+
if (path.dirname(dir) === dir) return null
|
|
50
|
+
}
|
|
51
|
+
}
|
|
52
|
+
|
|
53
|
+
/** The buildinfo path tsc writes for the project `file` (tsc's `getTsBuildInfoEmitOutputFilePath`). */
|
|
54
|
+
function buildInfoOf(read: ReadText, file: string, build: boolean): string | null {
|
|
55
|
+
const configDir = path.dirname(file)
|
|
56
|
+
const opts: { [K in keyof Options]?: string | boolean } = {}
|
|
57
|
+
const seen = new Set<string>()
|
|
58
|
+
// The nearest file that states a field decides it; a path is relative to that file.
|
|
59
|
+
const visit = (at: string, depth: number): void => {
|
|
60
|
+
if (depth > 16 || seen.has(at)) return
|
|
61
|
+
seen.add(at)
|
|
62
|
+
const cfg = parse(read, at)
|
|
63
|
+
if (cfg === null) return
|
|
64
|
+
const o = cfg.compilerOptions ?? {}
|
|
65
|
+
for (const k of ['composite', 'incremental'] as const)
|
|
66
|
+
if (opts[k] === undefined && typeof o[k] === 'boolean') opts[k] = o[k]
|
|
67
|
+
for (const k of ['tsBuildInfoFile', 'outDir', 'rootDir', 'outFile'] as const) {
|
|
68
|
+
const v = o[k]
|
|
69
|
+
if (opts[k] !== undefined || typeof v !== 'string') continue
|
|
70
|
+
opts[k] = v.startsWith('${configDir}')
|
|
71
|
+
? path.join(configDir, v.slice('${configDir}'.length))
|
|
72
|
+
: path.resolve(path.dirname(at), v)
|
|
73
|
+
}
|
|
74
|
+
// A later entry of an `extends` array wins over an earlier one.
|
|
75
|
+
const exts = typeof cfg.extends === 'string' ? [cfg.extends] : cfg.extends
|
|
76
|
+
if (!Array.isArray(exts)) return
|
|
77
|
+
for (const e of [...exts].reverse()) {
|
|
78
|
+
const next = typeof e === 'string' ? extended(read, path.dirname(at), e) : null
|
|
79
|
+
if (next !== null) visit(next, depth + 1)
|
|
80
|
+
}
|
|
81
|
+
}
|
|
82
|
+
visit(file, 0)
|
|
83
|
+
// No project there: tsc fails, and writes nothing.
|
|
84
|
+
if (parse(read, file) === null) return null
|
|
85
|
+
// tsc 5.6+ writes a buildinfo for every project `-b` builds.
|
|
86
|
+
if (!build && opts.incremental !== true && opts.composite !== true) return null
|
|
87
|
+
if (typeof opts.tsBuildInfoFile === 'string') return opts.tsBuildInfoFile
|
|
88
|
+
const bare = (p: string) => p.slice(0, p.length - path.extname(p).length)
|
|
89
|
+
if (typeof opts.outFile === 'string') return `${bare(opts.outFile)}.tsbuildinfo`
|
|
90
|
+
const config = bare(file)
|
|
91
|
+
const outDir = opts.outDir as string | undefined
|
|
92
|
+
const rootDir = opts.rootDir as string | undefined
|
|
93
|
+
const at =
|
|
94
|
+
outDir === undefined
|
|
95
|
+
? config
|
|
96
|
+
: rootDir === undefined
|
|
97
|
+
? path.join(outDir, path.basename(config))
|
|
98
|
+
: path.resolve(outDir, path.relative(rootDir, config))
|
|
99
|
+
return `${at}.tsbuildinfo`
|
|
100
|
+
}
|
|
101
|
+
|
|
102
|
+
/** A project argument's tsconfig: a directory names its `tsconfig.json`. */
|
|
103
|
+
const projectFile = (dir: string, arg: string): string =>
|
|
104
|
+
arg.endsWith('.json') ? path.resolve(dir, arg) : path.resolve(dir, arg, 'tsconfig.json')
|
|
105
|
+
|
|
106
|
+
/** Words that may run `tsc` before it: a runner, or an env assignment. */
|
|
107
|
+
const RUNNER = /^(?:npx|bunx|pnpx|pnpm|yarn|npm|bun|exec|run|-y|--yes|--|[A-Za-z_][\w]*=\S*)$/
|
|
108
|
+
|
|
109
|
+
/**
|
|
110
|
+
* The buildinfo files `command`'s `tsc` runs write, relative to the
|
|
111
|
+
* package `dir` (POSIX); one outside `dir` is another package's.
|
|
112
|
+
*/
|
|
113
|
+
export function tscBuildInfo(read: ReadText, dir: string, command: string): string[] {
|
|
114
|
+
const out = new Set<string>()
|
|
115
|
+
for (const segment of command.split(/&&|\|\||[;|&()\n]/)) {
|
|
116
|
+
const words = segment.trim().split(/\s+/)
|
|
117
|
+
const at = words.indexOf('tsc')
|
|
118
|
+
if (at < 0 || !words.slice(0, at).every((w) => RUNNER.test(w))) continue
|
|
119
|
+
const args = words.slice(at + 1)
|
|
120
|
+
if (args.some((a) => /^(?:-w|--watch|--clean)$/.test(a))) continue
|
|
121
|
+
const build = args[0] === '-b' || args[0] === '--build'
|
|
122
|
+
const files: string[] = []
|
|
123
|
+
if (build) {
|
|
124
|
+
const named = args.slice(1).filter((a) => !a.startsWith('-'))
|
|
125
|
+
files.push(...(named.length > 0 ? named : ['.']).map((a) => projectFile(dir, a)))
|
|
126
|
+
} else {
|
|
127
|
+
const p = args.findIndex((a) => a === '-p' || a === '--project')
|
|
128
|
+
files.push(projectFile(dir, p >= 0 && args[p + 1] !== undefined ? args[p + 1]! : '.'))
|
|
129
|
+
}
|
|
130
|
+
// `-b` builds each referenced project first, each with its own buildinfo.
|
|
131
|
+
const queue = [...files]
|
|
132
|
+
const visited = new Set<string>()
|
|
133
|
+
while (queue.length > 0) {
|
|
134
|
+
const file = queue.shift()!
|
|
135
|
+
if (visited.has(file)) continue
|
|
136
|
+
visited.add(file)
|
|
137
|
+
const info = buildInfoOf(read, file, build)
|
|
138
|
+
if (info !== null) {
|
|
139
|
+
const rel = path.relative(dir, info).split(path.sep).join('/')
|
|
140
|
+
if (!rel.startsWith('../') && !path.isAbsolute(rel)) out.add(rel)
|
|
141
|
+
}
|
|
142
|
+
if (!build) continue
|
|
143
|
+
const refs = parse(read, file)?.references
|
|
144
|
+
if (!Array.isArray(refs)) continue
|
|
145
|
+
for (const r of refs) {
|
|
146
|
+
const p = (r as { path?: unknown } | null)?.path
|
|
147
|
+
if (typeof p === 'string') queue.push(projectFile(path.dirname(file), p))
|
|
148
|
+
}
|
|
149
|
+
}
|
|
150
|
+
}
|
|
151
|
+
return [...out]
|
|
152
|
+
}
|
|
153
|
+
|
|
154
|
+
/** A reader that reads each file once, and the texts it read (a mapping's cache key). */
|
|
155
|
+
export function onceReader(): { read: ReadText; texts: Map<string, string | null> } {
|
|
156
|
+
const texts = new Map<string, string | null>()
|
|
157
|
+
const read: ReadText = (file) => {
|
|
158
|
+
if (!texts.has(file)) {
|
|
159
|
+
let text: string | null = null
|
|
160
|
+
try {
|
|
161
|
+
text = readFileSync(file, 'utf8')
|
|
162
|
+
} catch {}
|
|
163
|
+
texts.set(file, text)
|
|
164
|
+
}
|
|
165
|
+
return texts.get(file)!
|
|
166
|
+
}
|
|
167
|
+
return { read, texts }
|
|
168
|
+
}
|
package/src/turbo/index.ts
CHANGED
|
@@ -19,7 +19,8 @@ import { type AdoptionRun, adoptionPlugin } from '../adoption-plugin.js'
|
|
|
19
19
|
import type { AdoptionMapping } from '../mapping-cache.js'
|
|
20
20
|
import { collectGaps } from '../plugin-gaps.js'
|
|
21
21
|
import { gitIgnored, trackedKinds } from '../tracked-outputs.js'
|
|
22
|
-
import { relPosix } from '../paths.js'
|
|
22
|
+
import { packageScripts, relPosix } from '../paths.js'
|
|
23
|
+
import { onceReader, type ReadText, tscBuildInfo } from '../tsc-buildinfo.js'
|
|
23
24
|
import {
|
|
24
25
|
envNamesThatMap,
|
|
25
26
|
mapTurboWorkspace,
|
|
@@ -247,6 +248,11 @@ async function run(root: string, projects: readonly ProjectMeta[]): Promise<Adop
|
|
|
247
248
|
dirs.flatMap((d) => ['turbo.json', 'turbo.jsonc'].map((f) => textOf(path.join(d, f)))),
|
|
248
249
|
)
|
|
249
250
|
const envNames = envNamesThatMap(configs, envNamesNow())
|
|
251
|
+
// The tsconfigs a `tsc` script reads are the mapping's inputs too.
|
|
252
|
+
const tsc = onceReader()
|
|
253
|
+
for (const m of metas)
|
|
254
|
+
for (const s of Object.values(packageScripts(m)))
|
|
255
|
+
if (typeof s === 'string' && /\btsc\b/.test(s)) tscBuildInfo(tsc.read, m.dir, s)
|
|
250
256
|
return {
|
|
251
257
|
name: 'turbo',
|
|
252
258
|
spareTracked: true,
|
|
@@ -263,8 +269,9 @@ async function run(root: string, projects: readonly ProjectMeta[]): Promise<Adop
|
|
|
263
269
|
JSON.stringify(microfrontendsConfigs(root, dirs)),
|
|
264
270
|
// A config file added beside mapped tasks changes what an output may cover.
|
|
265
271
|
JSON.stringify(metas.map((m) => [m.name, m.dir, m.packageJson, m.configPath])),
|
|
272
|
+
JSON.stringify([...tsc.texts]),
|
|
266
273
|
],
|
|
267
|
-
map: (tracked) => mapAll(root, metas, envNames, tracked),
|
|
274
|
+
map: (tracked) => mapAll(root, metas, envNames, tracked, tsc.read),
|
|
268
275
|
}
|
|
269
276
|
}
|
|
270
277
|
|
|
@@ -279,6 +286,7 @@ async function mapAll(
|
|
|
279
286
|
metas: readonly PackageMeta[],
|
|
280
287
|
envNames: readonly string[],
|
|
281
288
|
trackedFiles: () => Promise<readonly string[] | null>,
|
|
289
|
+
read: ReadText,
|
|
282
290
|
): Promise<AdoptionMapping> {
|
|
283
291
|
const tracked = await trackedFiles()
|
|
284
292
|
const configs = new Map(
|
|
@@ -296,6 +304,7 @@ async function mapAll(
|
|
|
296
304
|
...(tracked === null ? {} : { tracked: trackedKinds(tracked) }),
|
|
297
305
|
ownConfig: (rel) => configs.get(rel === '.' ? '' : rel) ?? null,
|
|
298
306
|
ignored: (rels) => gitIgnored(root, rels),
|
|
307
|
+
buildInfo: (rel, command) => tscBuildInfo(read, path.join(root, rel), command),
|
|
299
308
|
})
|
|
300
309
|
const byName = new Map<string, TurboMappedProject>()
|
|
301
310
|
for (const project of mapped.projects) byName.set(project.name, project)
|
package/src/turbo/turbo-map.ts
CHANGED
|
@@ -349,6 +349,11 @@ export interface MapTurboOptions {
|
|
|
349
349
|
* path among them is keyed by a probe. Absent, none is.
|
|
350
350
|
*/
|
|
351
351
|
ignored?: (rels: readonly string[]) => Promise<ReadonlySet<string>>
|
|
352
|
+
/**
|
|
353
|
+
* The `.tsbuildinfo` files a root-relative package's command writes,
|
|
354
|
+
* relative to the package (`tscBuildInfo`). Absent, none.
|
|
355
|
+
*/
|
|
356
|
+
buildInfo?: (rel: string, command: string) => readonly string[]
|
|
352
357
|
}
|
|
353
358
|
|
|
354
359
|
/**
|
|
@@ -384,6 +389,15 @@ function withZeroSegments(glob: string): string[] {
|
|
|
384
389
|
return head === null ? [glob] : [glob, neg + head]
|
|
385
390
|
}
|
|
386
391
|
|
|
392
|
+
/** Does a positive glob of `globs` take `rel`, and no `!` give it back? */
|
|
393
|
+
function covers(globs: readonly string[], rel: string): boolean {
|
|
394
|
+
const hit = (g: string) => new Bun.Glob(g).match(rel)
|
|
395
|
+
return (
|
|
396
|
+
globs.some((g) => !g.startsWith('!') && hit(g)) &&
|
|
397
|
+
!globs.some((g) => g.startsWith('!') && hit(g.slice(1)))
|
|
398
|
+
)
|
|
399
|
+
}
|
|
400
|
+
|
|
387
401
|
/** No glob syntax: one path, which git may ignore. */
|
|
388
402
|
const isLiteral = (glob: string): boolean => !/[*?[\]{}()!]/.test(glob)
|
|
389
403
|
|
|
@@ -2141,6 +2155,12 @@ function buildTask(
|
|
|
2141
2155
|
return { name, todos, task, uses }
|
|
2142
2156
|
}
|
|
2143
2157
|
if (wild !== undefined) todos.push(wildcardCleanTodo(wild))
|
|
2158
|
+
// tsc re-emits only what its buildinfo calls changed: outside the
|
|
2159
|
+
// outputs it survives the clean before a miss, and the miss caches a
|
|
2160
|
+
// partial `dist/` (`tsc-buildinfo.ts`).
|
|
2161
|
+
if (outFiles.some((g) => !g.startsWith('!')))
|
|
2162
|
+
for (const rel of opts.buildInfo?.(pkgDir, command) ?? [])
|
|
2163
|
+
if (!covers(outFiles, rel)) outFiles.push(literalGlob(rel))
|
|
2144
2164
|
// An output over the project's own package.json (trpc's builds list
|
|
2145
2165
|
// it: tsdown rewrites `exports`) is taken back and keeps the cache, as
|
|
2146
2166
|
// nx() does (owner, 2026-10-03). The build rewrites an input the key
|
package/src/turbo-cache/index.ts
CHANGED
|
@@ -7,8 +7,8 @@
|
|
|
7
7
|
// Nothing is on by default: declare `turboCache()` in `.vx/workspace.ts`
|
|
8
8
|
// (the local store is the floor beneath it) and give it a URL and a token (options, or
|
|
9
9
|
// Turbo's own environment variables and turbo.json's `remoteCache` so a
|
|
10
|
-
// self-hosted setup carries over).
|
|
11
|
-
//
|
|
10
|
+
// self-hosted setup carries over; the OS keychain last). With no URL, team
|
|
11
|
+
// or token the plugin DECLINES and the run stays local.
|
|
12
12
|
//
|
|
13
13
|
// Imports core only through the public `@vzn/vx` specifier.
|
|
14
14
|
import { readFileSync, unlinkSync } from 'node:fs'
|
|
@@ -18,20 +18,40 @@ import path from 'node:path'
|
|
|
18
18
|
import {
|
|
19
19
|
definePlugin,
|
|
20
20
|
type RemoteCacheLayer,
|
|
21
|
+
type RemoteOpenContext,
|
|
21
22
|
type VxPlugin,
|
|
22
23
|
refuseUnknownOptions,
|
|
23
24
|
type PluginOptionKinds,
|
|
24
25
|
} from '@vzn/vx'
|
|
25
26
|
import { deadline } from '../remote-deadline.js'
|
|
26
27
|
import { OutageBreaker } from '../remote-breaker.js'
|
|
27
|
-
import { withRetry } from '../remote-retry.js'
|
|
28
|
-
import {
|
|
28
|
+
import { Cooldown, withRetry } from '../remote-retry.js'
|
|
29
|
+
import {
|
|
30
|
+
headerValueFault,
|
|
31
|
+
type KeychainRemote,
|
|
32
|
+
keychainToken,
|
|
33
|
+
remoteLabel,
|
|
34
|
+
} from '../remote-token.js'
|
|
35
|
+
import {
|
|
36
|
+
exchangeOidc,
|
|
37
|
+
type OidcConfig,
|
|
38
|
+
oidcSettings,
|
|
39
|
+
revokeOidc,
|
|
40
|
+
type TurboOidcOptions,
|
|
41
|
+
} from './oidc.js'
|
|
29
42
|
|
|
30
43
|
export interface TurboCacheOptions {
|
|
31
44
|
/** Base URL of the cache server (`https://cache.example.com`), or `TURBO_API`; with a token and neither, Vercel's hosted cache, as for `turbo`. */
|
|
32
45
|
apiUrl?: string
|
|
33
|
-
/** Bearer token every request carries, or `TURBO_TOKEN`. */
|
|
46
|
+
/** Bearer token every request carries, or `TURBO_TOKEN`, OIDC, `.turbo/config.json`, then the OS keychain. */
|
|
34
47
|
token?: string
|
|
48
|
+
/**
|
|
49
|
+
* On GitHub Actions with no token or `TURBO_TOKEN`, exchange the job's
|
|
50
|
+
* OIDC token for a short-lived one, revoked when the run ends. `true`:
|
|
51
|
+
* Vercel's hosted cache with Vercel's endpoints; any other server names
|
|
52
|
+
* its `tokenUrl`. The job needs `permissions: id-token: write`.
|
|
53
|
+
*/
|
|
54
|
+
oidc?: true | TurboOidcOptions
|
|
35
55
|
/** `teamId` query parameter, or `TURBO_TEAMID`. Required with `signatureKey`. */
|
|
36
56
|
teamId?: string
|
|
37
57
|
/** `slug` query parameter (a team slug), or `TURBO_TEAM`. */
|
|
@@ -84,7 +104,12 @@ export interface TurboLocalConfig {
|
|
|
84
104
|
|
|
85
105
|
export interface TurboCacheConfig {
|
|
86
106
|
apiUrl: string
|
|
87
|
-
|
|
107
|
+
/** Undefined: the layer's `open` exchanges OIDC or asks the OS keychain. */
|
|
108
|
+
token?: string
|
|
109
|
+
/** The job's OIDC exchange, when `oidc` is set and GitHub hands the job a token. */
|
|
110
|
+
oidc?: OidcConfig
|
|
111
|
+
/** `oidc` is set on GitHub Actions, but the job has no ID-token permission. */
|
|
112
|
+
oidcMissing?: true
|
|
88
113
|
teamId?: string
|
|
89
114
|
teamSlug?: string
|
|
90
115
|
signatureKey?: string
|
|
@@ -223,18 +248,41 @@ export function resolveTurboCacheConfig(
|
|
|
223
248
|
!turboPair && env['VERCEL_ARTIFACTS_TOKEN'] && env['VERCEL_ARTIFACTS_OWNER']
|
|
224
249
|
? { token: env['VERCEL_ARTIFACTS_TOKEN'], teamId: env['VERCEL_ARTIFACTS_OWNER'] }
|
|
225
250
|
: { token: undefined, teamId: turboPair ? undefined : env['VERCEL_ARTIFACTS_OWNER'] }
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
//
|
|
229
|
-
|
|
251
|
+
// Empty is unset, as Turbo reads it: a fork's pull request gets `''` for a secret.
|
|
252
|
+
const given = (options.token ?? env['TURBO_TOKEN'] ?? vercel.token) || undefined
|
|
253
|
+
// OIDC sits under the variables and over the config files.
|
|
254
|
+
const idTokenUrl = env['ACTIONS_ID_TOKEN_REQUEST_URL'] || undefined
|
|
255
|
+
const idTokenBearer = env['ACTIONS_ID_TOKEN_REQUEST_TOKEN'] || undefined
|
|
256
|
+
const exchanges =
|
|
257
|
+
options.oidc !== undefined &&
|
|
258
|
+
given === undefined &&
|
|
259
|
+
idTokenUrl !== undefined &&
|
|
260
|
+
idTokenBearer !== undefined
|
|
261
|
+
const token = given ?? (exchanges ? undefined : fromFile(local.token) || undefined)
|
|
262
|
+
const teamId =
|
|
263
|
+
options.teamId ??
|
|
264
|
+
env['TURBO_TEAMID'] ??
|
|
265
|
+
vercel.teamId ??
|
|
266
|
+
fromFile(local.teamId) ??
|
|
267
|
+
fromFile(file.teamId)
|
|
268
|
+
const teamSlug =
|
|
269
|
+
options.teamSlug ?? env['TURBO_TEAM'] ?? fromFile(local.teamSlug) ?? fromFile(file.teamSlug)
|
|
270
|
+
// Turbo's own default when a token or a team is set and no `apiUrl` is:
|
|
271
|
+
// Vercel's hosted Remote Cache. A token alone is a configured cache, as it
|
|
272
|
+
// is for `turbo`; a URL or a team without one asks the OS keychain at
|
|
273
|
+
// `open`; none of them is the declined, local run.
|
|
230
274
|
const apiUrl = (
|
|
231
275
|
options.apiUrl ??
|
|
232
276
|
env['TURBO_API'] ??
|
|
233
277
|
fromFile(local.apiUrl) ??
|
|
234
278
|
fromFile(file.apiUrl) ??
|
|
235
|
-
(token ? VERCEL_API : undefined)
|
|
279
|
+
(token || teamId || teamSlug ? VERCEL_API : undefined)
|
|
236
280
|
)?.replace(/\/+$/, '')
|
|
237
|
-
if (!apiUrl
|
|
281
|
+
if (!apiUrl) return undefined
|
|
282
|
+
const oidc =
|
|
283
|
+
options.oidc === undefined
|
|
284
|
+
? undefined
|
|
285
|
+
: oidcSettings(options.oidc, apiUrl === VERCEL_API, teamId ?? teamSlug)
|
|
238
286
|
// The URL is printed in every refusal line, so a `user:pass@` in it would
|
|
239
287
|
// leak to the log. Credentials go in the token.
|
|
240
288
|
if (URL.canParse(apiUrl)) {
|
|
@@ -244,19 +292,11 @@ export function resolveTurboCacheConfig(
|
|
|
244
292
|
'vx/turbo-cache: apiUrl carries credentials (user:pass@); pass them as the token instead',
|
|
245
293
|
)
|
|
246
294
|
}
|
|
247
|
-
const fault = headerValueFault(token)
|
|
295
|
+
const fault = token === undefined ? null : headerValueFault(token)
|
|
248
296
|
if (fault !== null)
|
|
249
297
|
throw new Error(
|
|
250
298
|
`vx/turbo-cache: the token holds ${fault}, which no HTTP header can carry — check the secret (it is not printed)`,
|
|
251
299
|
)
|
|
252
|
-
const teamId =
|
|
253
|
-
options.teamId ??
|
|
254
|
-
env['TURBO_TEAMID'] ??
|
|
255
|
-
vercel.teamId ??
|
|
256
|
-
fromFile(local.teamId) ??
|
|
257
|
-
fromFile(file.teamId)
|
|
258
|
-
const teamSlug =
|
|
259
|
-
options.teamSlug ?? env['TURBO_TEAM'] ?? fromFile(local.teamSlug) ?? fromFile(file.teamSlug)
|
|
260
300
|
const preflight =
|
|
261
301
|
options.preflight ??
|
|
262
302
|
truthy(env['TURBO_PREFLIGHT'], 'TURBO_PREFLIGHT') ??
|
|
@@ -291,7 +331,13 @@ export function resolveTurboCacheConfig(
|
|
|
291
331
|
}
|
|
292
332
|
return {
|
|
293
333
|
apiUrl,
|
|
294
|
-
token,
|
|
334
|
+
...(token !== undefined ? { token } : {}),
|
|
335
|
+
...(oidc !== undefined && exchanges
|
|
336
|
+
? { oidc: { ...oidc, idTokenUrl: idTokenUrl!, idTokenBearer: idTokenBearer! } }
|
|
337
|
+
: {}),
|
|
338
|
+
...(oidc !== undefined && !exchanges && given === undefined && env['GITHUB_ACTIONS'] === 'true'
|
|
339
|
+
? { oidcMissing: true as const }
|
|
340
|
+
: {}),
|
|
295
341
|
...(teamId ? { teamId } : {}),
|
|
296
342
|
...(teamSlug ? { teamSlug } : {}),
|
|
297
343
|
...(signatureKey !== undefined ? { signatureKey } : {}),
|
|
@@ -393,6 +439,15 @@ export class TurboRemoteCache implements RemoteCacheLayer {
|
|
|
393
439
|
private readonly absent = new Set<string>()
|
|
394
440
|
private readonly key: Uint8Array | undefined
|
|
395
441
|
readonly endpoint: string
|
|
442
|
+
/** Undefined until `open` reads the keychain; a layer never opened (a plan) asks the server nothing. */
|
|
443
|
+
private token: string | undefined
|
|
444
|
+
/** Set by `open` when the token came from the keychain: a refusal deletes it. */
|
|
445
|
+
private forget: (() => Promise<void>) | undefined
|
|
446
|
+
/** The token an OIDC exchange gave, revoked when the run ends. */
|
|
447
|
+
private exchanged: string | undefined
|
|
448
|
+
private readonly keychainRemote: KeychainRemote
|
|
449
|
+
/** The remote's rate limit, which every request of this client waits out. */
|
|
450
|
+
private readonly cooldown = new Cooldown()
|
|
396
451
|
constructor(
|
|
397
452
|
private readonly config: TurboCacheConfig,
|
|
398
453
|
private readonly fetchImpl: typeof fetch = fetch,
|
|
@@ -403,6 +458,34 @@ export class TurboRemoteCache implements RemoteCacheLayer {
|
|
|
403
458
|
) {
|
|
404
459
|
this.key = config.signatureKey === undefined ? undefined : Buffer.from(config.signatureKey)
|
|
405
460
|
this.endpoint = `${config.apiUrl}/v8/artifacts`
|
|
461
|
+
this.token = config.token
|
|
462
|
+
const team = config.teamId ?? config.teamSlug
|
|
463
|
+
this.keychainRemote = {
|
|
464
|
+
key: team === undefined ? this.endpoint : `${this.endpoint}#${team}`,
|
|
465
|
+
label: remoteLabel(config.apiUrl, team),
|
|
466
|
+
env: 'TURBO_TOKEN',
|
|
467
|
+
}
|
|
468
|
+
}
|
|
469
|
+
|
|
470
|
+
async open(ctx: RemoteOpenContext): Promise<void> {
|
|
471
|
+
if (this.config.oidcMissing)
|
|
472
|
+
ctx.warn(
|
|
473
|
+
'oidc: GitHub gave this job no OIDC token: it needs `permissions: id-token: write` (a fork or Dependabot pull request never gets one)',
|
|
474
|
+
)
|
|
475
|
+
if (this.token !== undefined) return
|
|
476
|
+
if (this.config.oidc !== undefined) {
|
|
477
|
+
try {
|
|
478
|
+
this.token = await exchangeOidc(this.config.oidc, this.fetchImpl, this.wait)
|
|
479
|
+
} catch (err) {
|
|
480
|
+
throw new Error(
|
|
481
|
+
`oidc: ${err instanceof Error ? err.message : String(err)}; local cache only this run`,
|
|
482
|
+
)
|
|
483
|
+
}
|
|
484
|
+
this.exchanged = this.token
|
|
485
|
+
return
|
|
486
|
+
}
|
|
487
|
+
this.token = await keychainToken(ctx, this.keychainRemote)
|
|
488
|
+
this.forget = () => ctx.forgetSecret({ key: this.keychainRemote.key })
|
|
406
489
|
}
|
|
407
490
|
|
|
408
491
|
private url(pathname: string): string {
|
|
@@ -417,7 +500,7 @@ export class TurboRemoteCache implements RemoteCacheLayer {
|
|
|
417
500
|
|
|
418
501
|
private headers(extra: Record<string, string> = {}, auth = true): Record<string, string> {
|
|
419
502
|
return {
|
|
420
|
-
...(auth ? { Authorization: `Bearer ${this.
|
|
503
|
+
...(auth ? { Authorization: `Bearer ${this.token}` } : {}),
|
|
421
504
|
'x-artifact-client-interactive': process.stdout.isTTY ? '1' : '0',
|
|
422
505
|
...extra,
|
|
423
506
|
}
|
|
@@ -444,7 +527,7 @@ export class TurboRemoteCache implements RemoteCacheLayer {
|
|
|
444
527
|
const res = await this.fetchImpl(url, {
|
|
445
528
|
method: 'OPTIONS',
|
|
446
529
|
headers: {
|
|
447
|
-
Authorization: `Bearer ${this.
|
|
530
|
+
Authorization: `Bearer ${this.token}`,
|
|
448
531
|
'Access-Control-Request-Method': method === 'HEAD' ? 'GET' : method,
|
|
449
532
|
'Access-Control-Request-Headers': ['Authorization', ...Object.keys(headers)].join(', '),
|
|
450
533
|
},
|
|
@@ -473,6 +556,7 @@ export class TurboRemoteCache implements RemoteCacheLayer {
|
|
|
473
556
|
pathname: string,
|
|
474
557
|
init: { body?: Blob | string; headers?: Record<string, string>; timeoutMs?: number } = {},
|
|
475
558
|
): Promise<Response | undefined> {
|
|
559
|
+
if (this.token === undefined) return undefined
|
|
476
560
|
const timeoutMs = init.timeoutMs ?? this.config.timeoutMs
|
|
477
561
|
// 0 is Turbo's "no deadline".
|
|
478
562
|
const signal = () => (timeoutMs === 0 ? undefined : deadline(timeoutMs))
|
|
@@ -496,6 +580,7 @@ export class TurboRemoteCache implements RemoteCacheLayer {
|
|
|
496
580
|
},
|
|
497
581
|
this.config.retries,
|
|
498
582
|
this.wait,
|
|
583
|
+
this.cooldown,
|
|
499
584
|
),
|
|
500
585
|
method === 'GET',
|
|
501
586
|
)
|
|
@@ -511,11 +596,24 @@ export class TurboRemoteCache implements RemoteCacheLayer {
|
|
|
511
596
|
const first = !this.disabled
|
|
512
597
|
this.disabled = true
|
|
513
598
|
if (!first) return undefined
|
|
514
|
-
|
|
599
|
+
if (this.forget === undefined)
|
|
600
|
+
throw new Error(`HTTP ${res.status}: the token was refused; remote cache off for this run`)
|
|
601
|
+
await this.forget()
|
|
602
|
+
throw new Error(
|
|
603
|
+
`HTTP ${res.status}: the token was refused; removed it from the OS keychain, so the next run asks again; remote cache off for this run`,
|
|
604
|
+
)
|
|
515
605
|
}
|
|
516
606
|
return res
|
|
517
607
|
}
|
|
518
608
|
|
|
609
|
+
/** Revoke the token `open` exchanged, once; a throw is the caller's warning. */
|
|
610
|
+
async revoke(): Promise<void> {
|
|
611
|
+
const token = this.exchanged
|
|
612
|
+
if (token === undefined) return
|
|
613
|
+
this.exchanged = undefined
|
|
614
|
+
await revokeOidc(this.config.oidc!, token, this.fetchImpl, this.wait)
|
|
615
|
+
}
|
|
616
|
+
|
|
519
617
|
async has(hash: string): Promise<boolean> {
|
|
520
618
|
if (this.disabled) return false
|
|
521
619
|
const res = await this.request('HEAD', `/${hash}`)
|
|
@@ -702,6 +800,7 @@ const TURBO_CACHE_KEYS: PluginOptionKinds<TurboCacheOptions> = {
|
|
|
702
800
|
uploadTimeoutMs: 'number',
|
|
703
801
|
retries: 'number',
|
|
704
802
|
preflight: 'boolean',
|
|
803
|
+
oidc: 'any',
|
|
705
804
|
}
|
|
706
805
|
|
|
707
806
|
/**
|
|
@@ -711,10 +810,12 @@ const TURBO_CACHE_KEYS: PluginOptionKinds<TurboCacheOptions> = {
|
|
|
711
810
|
* plugins: [turboCache({ apiUrl: 'https://cache.example.com', token: process.env.CACHE_TOKEN })]
|
|
712
811
|
* ```
|
|
713
812
|
*
|
|
714
|
-
* Declines
|
|
813
|
+
* Declines with no URL, team or token, so it is safe to leave declared.
|
|
715
814
|
*/
|
|
716
815
|
export function turboCache(options: TurboCacheOptions = {}): VxPlugin {
|
|
717
816
|
refuseUnknownOptions('turboCache()', options, TURBO_CACHE_KEYS)
|
|
817
|
+
// The run's layer and warn, for `teardown` to revoke an exchanged token.
|
|
818
|
+
let opened: { layer: TurboRemoteCache; warn: (m: string) => void } | undefined
|
|
718
819
|
return definePlugin(import.meta, {
|
|
719
820
|
cache(ctx): RemoteCacheLayer | undefined {
|
|
720
821
|
const config = resolveTurboCacheConfig(
|
|
@@ -727,7 +828,20 @@ export function turboCache(options: TurboCacheOptions = {}): VxPlugin {
|
|
|
727
828
|
// A signed download waits in a temp until its tag checks: in vx's
|
|
728
829
|
// cache directory, which the sandbox walls, not the shared temp dir a
|
|
729
830
|
// sandboxed task may read (L-45).
|
|
730
|
-
|
|
831
|
+
const layer = new TurboRemoteCache(config, fetch, ctx.cacheDir)
|
|
832
|
+
if (config.oidc !== undefined) opened = { layer, warn: ctx.warn }
|
|
833
|
+
return layer
|
|
834
|
+
},
|
|
835
|
+
async teardown() {
|
|
836
|
+
const run = opened
|
|
837
|
+
opened = undefined
|
|
838
|
+
await run?.layer
|
|
839
|
+
.revoke()
|
|
840
|
+
.catch((err: unknown) =>
|
|
841
|
+
run.warn(
|
|
842
|
+
`[vx] plugin '@vzn/vx-migrate': oidc: ${err instanceof Error ? err.message : String(err)}; the token expires on its own`,
|
|
843
|
+
),
|
|
844
|
+
)
|
|
731
845
|
},
|
|
732
846
|
})
|
|
733
847
|
}
|
|
@@ -0,0 +1,171 @@
|
|
|
1
|
+
// A GitHub Actions job's OIDC token, exchanged for a short-lived remote
|
|
2
|
+
// cache token (RFC 8693), and revoked when the run ends (RFC 7009): no
|
|
3
|
+
// long-lived TURBO_TOKEN secret in the repo. Vercel's flow as its own
|
|
4
|
+
// action runs it (vercel/setup-turborepo-remote-cache-action, read
|
|
5
|
+
// 2026-10-17); any other server names its token URL.
|
|
6
|
+
|
|
7
|
+
import { deadline } from '../remote-deadline.js'
|
|
8
|
+
|
|
9
|
+
/** `turboCache({ oidc })`: `true` is Vercel's hosted cache with its defaults. */
|
|
10
|
+
export interface TurboOidcOptions {
|
|
11
|
+
/** The RFC 8693 token endpoint; Vercel's by default with Vercel's cache, required otherwise. */
|
|
12
|
+
tokenUrl?: string
|
|
13
|
+
/** The RFC 7009 revocation endpoint, asked when the run ends; `false`: none. Vercel's by default. */
|
|
14
|
+
revokeUrl?: string | false
|
|
15
|
+
/** The `audience` GitHub mints the OIDC token for. */
|
|
16
|
+
audience?: string
|
|
17
|
+
/** Fields the exchange also sends (Vercel's `policy_id`); over the defaults. */
|
|
18
|
+
params?: Record<string, string>
|
|
19
|
+
}
|
|
20
|
+
|
|
21
|
+
export interface OidcConfig {
|
|
22
|
+
/** GitHub's ID-token endpoint and its bearer: the job's `ACTIONS_ID_TOKEN_REQUEST_*`. */
|
|
23
|
+
idTokenUrl: string
|
|
24
|
+
idTokenBearer: string
|
|
25
|
+
tokenUrl: string
|
|
26
|
+
revokeUrl?: string
|
|
27
|
+
audience?: string
|
|
28
|
+
params: Record<string, string>
|
|
29
|
+
}
|
|
30
|
+
|
|
31
|
+
const VERCEL_TOKEN_URL = 'https://api.vercel.com/login/oauth/token'
|
|
32
|
+
const VERCEL_REVOKE_URL = 'https://api.vercel.com/login/oauth/token/revoke'
|
|
33
|
+
/** Vercel's public OAuth client for the Remote Cache exchange, as its action sends it. */
|
|
34
|
+
const VERCEL_CLIENT_ID = 'cl_kyUx2zVvA4MGptBohkmtYHJly2XltXzD'
|
|
35
|
+
|
|
36
|
+
const ATTEMPTS = 3
|
|
37
|
+
const TIMEOUT_MS = 30_000
|
|
38
|
+
|
|
39
|
+
/**
|
|
40
|
+
* The exchange `oidc` asks for, or undefined when the option is off. A
|
|
41
|
+
* misconfigured one (a server that is not Vercel's with no `tokenUrl`, a
|
|
42
|
+
* Vercel exchange with no team) is refused at config load.
|
|
43
|
+
*/
|
|
44
|
+
export function oidcSettings(
|
|
45
|
+
oidc: true | TurboOidcOptions,
|
|
46
|
+
vercel: boolean,
|
|
47
|
+
team: string | undefined,
|
|
48
|
+
): Omit<OidcConfig, 'idTokenUrl' | 'idTokenBearer'> {
|
|
49
|
+
const o = oidc === true ? {} : oidc
|
|
50
|
+
if (typeof o !== 'object' || o === null)
|
|
51
|
+
throw new Error(
|
|
52
|
+
'vx/turbo-cache: oidc must be true or { tokenUrl, revokeUrl, audience, params }',
|
|
53
|
+
)
|
|
54
|
+
if (!vercel && o.tokenUrl === undefined)
|
|
55
|
+
throw new Error("vx/turbo-cache: oidc needs tokenUrl for a cache that is not Vercel's")
|
|
56
|
+
if (vercel && team === undefined)
|
|
57
|
+
throw new Error('vx/turbo-cache: oidc on Vercel needs teamId or teamSlug (TURBO_TEAM)')
|
|
58
|
+
const revokeUrl = o.revokeUrl ?? (vercel ? VERCEL_REVOKE_URL : undefined)
|
|
59
|
+
return {
|
|
60
|
+
tokenUrl: o.tokenUrl ?? VERCEL_TOKEN_URL,
|
|
61
|
+
...(revokeUrl === false || revokeUrl === undefined ? {} : { revokeUrl }),
|
|
62
|
+
...(o.audience !== undefined ? { audience: o.audience } : {}),
|
|
63
|
+
params: {
|
|
64
|
+
...(vercel ? { client_id: VERCEL_CLIENT_ID, team_id_or_slug: team! } : {}),
|
|
65
|
+
...o.params,
|
|
66
|
+
},
|
|
67
|
+
}
|
|
68
|
+
}
|
|
69
|
+
|
|
70
|
+
/** A request, sent again on a 429, a 5xx or no answer: up to three tries, 30 s each. */
|
|
71
|
+
async function send(
|
|
72
|
+
fetchImpl: typeof fetch,
|
|
73
|
+
url: string,
|
|
74
|
+
init: RequestInit,
|
|
75
|
+
wait: (ms: number) => Promise<void>,
|
|
76
|
+
): Promise<Response> {
|
|
77
|
+
for (let attempt = 1; ; attempt++) {
|
|
78
|
+
let res: Response | undefined
|
|
79
|
+
let failure: unknown
|
|
80
|
+
try {
|
|
81
|
+
res = await fetchImpl(url, { ...init, signal: deadline(TIMEOUT_MS) })
|
|
82
|
+
} catch (err) {
|
|
83
|
+
failure = err
|
|
84
|
+
}
|
|
85
|
+
const again = res === undefined || res.status === 429 || res.status >= 500
|
|
86
|
+
if (!again || attempt === ATTEMPTS) {
|
|
87
|
+
if (res === undefined) throw failure
|
|
88
|
+
return res
|
|
89
|
+
}
|
|
90
|
+
await res?.body?.cancel()
|
|
91
|
+
await wait(1000 * attempt)
|
|
92
|
+
}
|
|
93
|
+
}
|
|
94
|
+
|
|
95
|
+
const form = (fields: Record<string, string>) => ({
|
|
96
|
+
method: 'POST',
|
|
97
|
+
headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
|
|
98
|
+
body: new URLSearchParams(fields).toString(),
|
|
99
|
+
})
|
|
100
|
+
|
|
101
|
+
/** Where a URL points, never its query: GitHub's carries a runtime id. */
|
|
102
|
+
const where = (url: string) => (URL.canParse(url) ? new URL(url).host : 'the endpoint')
|
|
103
|
+
|
|
104
|
+
async function field(res: Response, name: string, what: string): Promise<string> {
|
|
105
|
+
if (!res.ok) {
|
|
106
|
+
await res.body?.cancel()
|
|
107
|
+
throw new Error(`${what} answered HTTP ${res.status}`)
|
|
108
|
+
}
|
|
109
|
+
const value = ((await res.json().catch(() => null)) as Record<string, unknown> | null)?.[name]
|
|
110
|
+
if (typeof value !== 'string' || value === '') throw new Error(`${what} sent no ${name}`)
|
|
111
|
+
return value
|
|
112
|
+
}
|
|
113
|
+
|
|
114
|
+
/** The job's OIDC token, exchanged for the cache's access token. Errors never quote a token. */
|
|
115
|
+
export async function exchangeOidc(
|
|
116
|
+
cfg: OidcConfig,
|
|
117
|
+
fetchImpl: typeof fetch = fetch,
|
|
118
|
+
wait: (ms: number) => Promise<void> = Bun.sleep,
|
|
119
|
+
): Promise<string> {
|
|
120
|
+
const idUrl = new URL(cfg.idTokenUrl)
|
|
121
|
+
if (cfg.audience !== undefined) idUrl.searchParams.set('audience', cfg.audience)
|
|
122
|
+
const idToken = await field(
|
|
123
|
+
await send(
|
|
124
|
+
fetchImpl,
|
|
125
|
+
idUrl.toString(),
|
|
126
|
+
{ headers: { Authorization: `bearer ${cfg.idTokenBearer}` } },
|
|
127
|
+
wait,
|
|
128
|
+
),
|
|
129
|
+
'value',
|
|
130
|
+
`GitHub's OIDC endpoint (${where(cfg.idTokenUrl)})`,
|
|
131
|
+
)
|
|
132
|
+
return field(
|
|
133
|
+
await send(
|
|
134
|
+
fetchImpl,
|
|
135
|
+
cfg.tokenUrl,
|
|
136
|
+
form({
|
|
137
|
+
grant_type: 'urn:ietf:params:oauth:grant-type:token-exchange',
|
|
138
|
+
subject_token: idToken,
|
|
139
|
+
subject_token_type: 'urn:ietf:params:oauth:token-type:id_token',
|
|
140
|
+
requested_token_type: 'urn:ietf:params:oauth:token-type:access_token',
|
|
141
|
+
...cfg.params,
|
|
142
|
+
}),
|
|
143
|
+
wait,
|
|
144
|
+
),
|
|
145
|
+
'access_token',
|
|
146
|
+
`the token exchange (${where(cfg.tokenUrl)})`,
|
|
147
|
+
)
|
|
148
|
+
}
|
|
149
|
+
|
|
150
|
+
/** Revoke the exchanged token; a throw is the caller's warning. */
|
|
151
|
+
export async function revokeOidc(
|
|
152
|
+
cfg: OidcConfig,
|
|
153
|
+
token: string,
|
|
154
|
+
fetchImpl: typeof fetch = fetch,
|
|
155
|
+
wait: (ms: number) => Promise<void> = Bun.sleep,
|
|
156
|
+
): Promise<void> {
|
|
157
|
+
if (cfg.revokeUrl === undefined) return
|
|
158
|
+
const clientId = cfg.params['client_id']
|
|
159
|
+
const res = await send(
|
|
160
|
+
fetchImpl,
|
|
161
|
+
cfg.revokeUrl,
|
|
162
|
+
form({
|
|
163
|
+
...(clientId !== undefined ? { client_id: clientId } : {}),
|
|
164
|
+
token,
|
|
165
|
+
token_type_hint: 'access_token',
|
|
166
|
+
}),
|
|
167
|
+
wait,
|
|
168
|
+
)
|
|
169
|
+
await res.body?.cancel()
|
|
170
|
+
if (!res.ok) throw new Error(`revoking at ${where(cfg.revokeUrl)} answered HTTP ${res.status}`)
|
|
171
|
+
}
|