@vzn/vx-cache-server 0.0.651 → 0.0.653

package/PROTOCOL.md

@@ -45,7 +45,7 @@ request with no token `401`.
 
 A request's `Authorization: Bearer <token>` is a CI job's OIDC token
 (GitHub Actions, GitLab CI): there is no other. Its signature,
-`RS256`, `exp` and `aud` (the server's own) are checked, and the token
+`RS256`, `exp` and `aud` (the server's own origin) are checked, and the token
 alone decides what it reads and where it writes, by the server's
 `ALLOW` (`issuer:owner/repo:workflow` entries):
 
@@ -83,20 +83,23 @@ An artifact lives in one scope, and a key names both: `{ scope, hash }`.
 does (`refs/heads/main` is `refs%2Fheads%2Fmain`), at most 512
 characters. Any other spelling (a plain ref, another encoding of it, a
 name not under `refs/`) is `400`, so one ref is one scope.
-`GET` and `HEAD` name one scope in `vx-scope`; a check names
-its scopes in its body.
-`hash` is 1 to 128 hex digits. A read is exact: the server answers the scope named and no
-other, so a client that wants its branch's work and then main's asks
-for both. Every read names a scope.
+`GET` and `HEAD` name their scopes in `vx-scope`, in the order to read
+them (`<scope>, <scope>`), up to `maxCheckHashes`; a check names its
+scopes in its body.
+`hash` is 1 to 128 hex digits. A read is exact: the server answers the scopes named and no
+other, the first that holds the hash, so a client that wants its
+branch's work and then main's names both. Every read names a scope.
 
 An artifact is immutable. A server may expire one at any time: a
 download after a check that held it may miss, which a client reads as a
 miss.
 
 `GET /v1/artifacts/<hash>` (as Turbo's `GET /v8/artifacts/<hash>`) with
-`vx-scope: <scope>`: `200` with the bytes (`application/octet-stream`,
+`vx-scope: <scope>, …`: `200` with the bytes of the first scope that
+holds them (`application/octet-stream`; from the first scope named,
 `Cache-Control: private, max-age=31536000, immutable`: the bytes under a
-key never change), or `404`. `HEAD` is the same with no bytes, and names
+key never change; from a later one, `private, no-cache`: an earlier
+scope may come to hold the hash), or `404`. `HEAD` is the same with no bytes, and names
 the size as `Content-Length`. A download may come chunked with no
 `Content-Length` (Bun.serve sends any streamed answer so), so a body cut
 short can end as cleanly as a whole one: a client checks the bytes
@@ -138,7 +141,12 @@ An operator sets each; `GET /v1/capabilities` names them:
 { "maxCheckHashes": 500, "maxUploadBytes": 100000000 }
 ```
 
-A client keeps to the limits the server names.
+A client keeps to the limits the server names. It need not ask them
+apart: every `/v1/artifacts` answer names them, so a client may send its
+first check whole and, refused for its size (`400` past
+`maxCheckHashes`, `413` past the bytes they take), send it again in
+chunks of the limits its refusal names. vx does: a run the local cache
+answers asks the server nothing.
 
 ## Errors
 

package/README.md

@@ -46,7 +46,8 @@ scope, which only runs of that branch read:
 
 `AUDIENCE` is the `aud` a token must name: the origin clients reach the
 server at (`https://cache.example.com`), which is what `vxCache()` asks
-GitHub for. On Workers, unset, it is that origin: Cloudflare routes only
+GitHub for. A URL is read as its origin (`https://cache.example.com/v1`
+is `https://cache.example.com`); one that is no URL stops the server. On Workers, unset, it is that origin: Cloudflare routes only
 the Worker's own hostnames to it. On Bun and Docker it is required, since
 there the origin is the request's `Host` header, which a client picks.
 `READ_ACCESS=public` lets a request with no token read a repository
@@ -113,16 +114,22 @@ artifact (a check, a `HEAD`, an upload's skip) is asked of R2, never
 of that copy: it can outlive R2's, and a writer told "held" there would
 skip an upload another data center then cannot read. An upload drops
 the copy in the data center it reaches: R2 had let those bytes go.
+A read R2 answers "try again" (10001, 10058) is asked up to three
+times before the request fails `502`. The deploy clears any cron: the
+Worker has no `scheduled()`.
 
 Set `EXPIRY_DAYS` to the bucket's lifecycle rule (below) and R2's
 "held" is kept in the data center's cache too: an upload, R2's
 answer to a lookup, or a download R2 serves marks the artifact held,
 in the background, until the rule may drop it, counted from its upload. A repeat check or `HEAD` in that data center then
 skips R2. Only "held" is kept: a miss always asks R2, so a later
-upload is never hidden. Unset, every lookup asks R2. A larger value
+upload is never hidden. An upload's skip always asks R2, never a
+mark: a client uploads only what its check missed, which no mark
+holds, so this costs no head a mark saved. Unset, every lookup asks R2. A larger value
 than the rule is wrong: a mark would outlive the artifact. After
-deleting artifacts by hand, a mark may still say "held" for up to
-`EXPIRY_DAYS`; vx then misses the download and runs the task.
+deleting artifacts by hand, a mark may still answer a check "held";
+vx then misses the download and runs the task, the miss drops that
+mark, and the task's upload is stored again.
 
 ## Expiry
 
@@ -149,8 +156,9 @@ over a bucket of your own (`src/index.ts`):
 - `createCacheServer(options: ServerOptions)` returns
   `(req: Request) => Promise<Response>`. `ServerOptions` is
   `{ blobs, authorize, limits? }`: `blobs` is a `Blobs` bucket
-  (`{ put(name, body, size), get(name), size(name) }`, `get` answering a
-  `Blob`, `{ body, size }`), `authorize` an `Authorize`, which maps a
+  (`{ put(name, body, size), get(name), size(name, bucket?) }`, `get` answering a
+  `Blob`, `{ body, size }`, `size` with `bucket` the bucket's own answer,
+  never one the backend kept: an upload's skip; a backend that keeps none ignores it), `authorize` an `Authorize`, which maps a
   request to its `Access` (`{ issuer, repository, write? }`: the token's issuer and
   repository name, which it reads, and the scope it may write) or throws, and `limits` any of `Limits`
   (`{ maxCheckHashes, maxUploadBytes }`) over

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@vzn/vx-cache-server",
-  "version": "0.0.651",
+  "version": "0.0.653",
   "license": "MIT",
   "description": "A remote cache server for vx that you host: on Bun or Docker with an S3-compatible bucket, or on Cloudflare Workers with R2.",
   "keywords": [
@@ -45,7 +45,7 @@
     "miniflare: tests/worker.test.ts runs src/worker.ts in workerd with R2, as Workers deploy it."
   ],
   "dependencies": {
-    "@vzn/vx-devtools": "0.0.651"
+    "@vzn/vx-devtools": "0.0.653"
   },
   "repository": {
     "type": "git",

package/src/bun.ts

@@ -11,7 +11,7 @@ import type { Blobs } from './store.js'
 const USAGE = `usage: vx-cache-server [--port 8080] [--host 0.0.0.0]
   ALLOW           who may take part, space separated, each issuer:owner/repo:workflow
                   (README.md § CI tokens); only their jobs' OIDC tokens are taken
-  AUDIENCE        the aud a CI token names: the URL clients reach this server at
+  AUDIENCE        the aud a CI token names: the origin clients reach this server at
   READ_ACCESS     public: a request with no token reads a repository ALLOW names
   S3_BUCKET       the bucket; S3_ENDPOINT, S3_REGION, S3_ACCESS_KEY_ID and
                   S3_SECRET_ACCESS_KEY as Bun's S3 client reads them
@@ -140,7 +140,9 @@ function main(argv: string[]): void {
   // header, which the client picks. Workers know their own hostnames.
   const audience = env.AUDIENCE?.trim() ?? ''
   if (audience === '')
-    fail(`set AUDIENCE: the aud a CI token names, the URL clients reach this server at\n${USAGE}`)
+    fail(
+      `set AUDIENCE: the aud a CI token names, the origin clients reach this server at\n${USAGE}`,
+    )
   const handle = createCacheServer({
     blobs: s3Blobs(client),
     authorize: oidcAuth({

package/src/server.ts

@@ -57,6 +57,13 @@ export function limitsFrom(env: object): Partial<Limits> {
 /** A held artifact's bytes never change (2026-10-25). */
 const IMMUTABLE = 'private, max-age=31536000, immutable'
 
+/**
+ * An answer from the first scope named is that key's, which never
+ * changes; private: its token chose the repository. One from a later
+ * scope changes once an earlier one holds the hash.
+ */
+const cacheOf = (at: number): string => (at === 0 ? IMMUTABLE : 'private, no-cache')
+
 /**
  * A scope on the wire: a ref (`refs/…`), percent-encoded as
  * `encodeURIComponent` does, its folder's name (2026-11-03). One spelling
@@ -91,6 +98,17 @@ export function createCacheServer(opts: ServerOptions): (req: Request) => Promis
     }
     return scope
   }
+  /**
+   * The scopes a download names in `vx-scope`, in the order to read them
+   * (`<scope>, <scope>`): up to `maxCheckHashes`, its lookups as a check's.
+   */
+  const scopesOf = (req: Request): string[] => {
+    const named = (req.headers.get(SCOPE_HEADER) ?? '').split(',').map((s) => s.trim())
+    if (named.length > limits.maxCheckHashes) {
+      throw new Refusal(400, `${SCOPE_HEADER}: up to ${limits.maxCheckHashes} scopes`)
+    }
+    return named.map(scopeOf)
+  }
   const nameOf = (access: Access, scope: string, hash: string): string =>
     blobName(access.issuer, access.repository, scope, hash)
   /** A read of the bucket: one it fails is its fault, `502`, never the server's `500`. */
@@ -137,32 +155,43 @@ export function createCacheServer(opts: ServerOptions): (req: Request) => Promis
     return { hashes: hashes as string[], scopes: scopes.map(scopeOf) }
   }
 
+  /**
+   * An artifact under the first scope named that holds it, each asked in
+   * turn: a run on a branch reads its own scope, then main's, and a client
+   * asking each by its own request paid a round-trip for every scope
+   * before the one that held it (2026-11-06).
+   */
   async function artifact(req: Request, hash: string): Promise<Response> {
     if (!isHash(hash)) throw new Refusal(400, 'a hash is 1 to 128 hex digits')
     const access = await authorize(req)
-    const scope = scopeOf(req.headers.get(SCOPE_HEADER))
-    const blob = await read(blobs.get(nameOf(access, scope, hash)))
-    if (blob === null) return errorJson(404, 'no artifact under this key')
-    return new Response(blob.body, {
-      headers: {
-        'content-length': String(blob.size),
-        'content-type': 'application/octet-stream',
-        // An artifact never changes under its key; private: its token chose the repository.
-        'cache-control': IMMUTABLE,
-      },
-    })
+    const scopes = scopesOf(req)
+    for (const [at, scope] of scopes.entries()) {
+      const blob = await read(blobs.get(nameOf(access, scope, hash)))
+      if (blob === null) continue
+      return new Response(blob.body, {
+        headers: {
+          'content-length': String(blob.size),
+          'content-type': 'application/octet-stream',
+          'cache-control': cacheOf(at),
+        },
+      })
+    }
+    return errorJson(404, 'no artifact under this key')
   }
 
-  /** Turbo's `HEAD /v8/artifacts/<hash>`: whether the scope named holds it, and its size, with no bytes. */
+  /** Turbo's `HEAD /v8/artifacts/<hash>`: whether a scope named holds it, and its size, with no bytes. */
   async function exists(req: Request, hash: string): Promise<Response> {
     if (!isHash(hash)) throw new Refusal(400, 'a hash is 1 to 128 hex digits')
     const access = await authorize(req)
-    const scope = scopeOf(req.headers.get(SCOPE_HEADER))
-    const size = await read(blobs.size(nameOf(access, scope, hash)))
-    if (size === null) return new Response(null, { status: 404 })
-    return new Response(null, {
-      headers: { 'content-length': String(size), 'cache-control': IMMUTABLE },
-    })
+    const scopes = scopesOf(req)
+    for (const [at, scope] of scopes.entries()) {
+      const size = await read(blobs.size(nameOf(access, scope, hash)))
+      if (size === null) continue
+      return new Response(null, {
+        headers: { 'content-length': String(size), 'cache-control': cacheOf(at) },
+      })
+    }
+    return new Response(null, { status: 404 })
   }
 
   // Every lookup at once, each exact: a hash answers the first scope
@@ -199,7 +228,7 @@ export function createCacheServer(opts: ServerOptions): (req: Request) => Promis
     if (length > limits.maxUploadBytes) {
       throw new Refusal(413, `an upload is at most ${limits.maxUploadBytes} bytes`)
     }
-    if ((await read(blobs.size(name))) !== null) {
+    if ((await read(blobs.size(name, true))) !== null) {
       await req.body?.cancel()
       return new Response(null, { status: 204 })
     }

package/src/store.ts

@@ -15,8 +15,12 @@ export interface Blobs {
   put(name: string, body: ReadableStream<Uint8Array>, size: number): Promise<void>
   /** The blob, or null when the bucket has none by that name. */
   get(name: string): Promise<Blob | null>
-  /** Its size, read off its metadata, or null when there is none. */
-  size(name: string): Promise<number | null>
+  /**
+   * Its size, read off its metadata, or null when there is none. `bucket`:
+   * the bucket's own answer, never one a backend kept (an upload's skip,
+   * which a stale "held" would drop).
+   */
+  size(name: string, bucket?: boolean): Promise<number | null>
 }
 
 /**

package/src/worker.ts

@@ -41,7 +41,7 @@ export interface Env {
   readonly BLOBS?: R2Bucket
   /** Who may take part (`parseAllow`): `issuer:owner/repo:workflow` entries, space separated. */
   readonly ALLOW?: string
-  /** The `aud` a CI token names; unset, the origin the Worker is reached at. */
+  /** The `aud` a CI token names, read as its origin; unset, the origin the Worker is reached at. */
   readonly AUDIENCE?: string
   /** `public`: a request with no token reads a repository ALLOW names. */
   readonly READ_ACCESS?: string
@@ -52,6 +52,28 @@ export interface Env {
   readonly EXPIRY_DAYS?: string | number
 }
 
+/**
+ * R2's answers that ask to be sent again: 10001 "internal error, please
+ * try again" and 10058 "reduce your rate of simultaneous reads on the
+ * same object". Each failed a check, a download or an upload's skip 502
+ * (2026-11-01 to 11-06: 57 in the dashboard's Issues), so the client lost
+ * a hit or sent its bytes again.
+ */
+const AGAIN = /\((10001|10058)\)$/
+const R2_TRIES = 3
+
+/** A read of R2, asked up to three times while R2 says to ask again, 50 then 100 ms apart. */
+async function again<T>(ask: () => Promise<T>): Promise<T> {
+  for (let attempt = 1; ; attempt++) {
+    try {
+      return await ask()
+    } catch (err) {
+      if (attempt === R2_TRIES || !AGAIN.test(String((err as Error)?.message))) throw err
+      await new Promise((r) => setTimeout(r, 50 * attempt))
+    }
+  }
+}
+
 /** An object's key in the data center's cache. */
 const keyOf = (name: string): Request => new Request(`https://vx-cache.blobs/${name}`)
 /** The key of the mark that R2 holds an object, and its size. */
@@ -67,8 +89,9 @@ const SIZE = 'vx-size'
  * With `expiryDays`, the bucket's lifecycle rule, whether R2 holds an
  * object is kept there too, once R2 takes, heads or serves it, until the
  * rule may drop it: counted from its upload, so a mark never outlives
- * R2's copy. Only "held" is kept: a
- * kept "missing" would hide an upload that lands later.
+ * R2's copy unless it is deleted by hand, which a download R2 misses
+ * clears. Only "held" is kept: a kept "missing" would hide an upload
+ * that lands later.
  */
 export function r2Blobs(
   bucket: R2Bucket,
@@ -111,22 +134,31 @@ export function r2Blobs(
       const hit = await (await cache).match(key)
       // No mark from this copy: it can outlive R2's.
       if (hit?.body) return { body: hit.body, size: Number(hit.headers.get('content-length')) }
-      const o = await bucket.get(name)
-      if (o === null) return null
+      const o = await again(() => bucket.get(name))
+      if (o === null) {
+        // R2 let it go before its expiry (a deletion by hand): the next
+        // check asks R2 again, and its miss sends the upload.
+        if (expiryDays !== undefined) ctx.waitUntil(cache.then((c) => c.delete(heldOf(name))))
+        return null
+      }
       mark(name, o.size, o.uploaded.getTime())
       // The copy is R2 read again, never a tee of the client's body: a tee
       // on Workers reads at its slower branch's pace, and the cache write
       // held a download for 19 minutes (main, 2026-10-28).
       ctx.waitUntil(
         (async () => {
-          const again = await bucket.get(name)
-          if (again === null) return
+          const copy = await again(() => bucket.get(name))
+          if (copy === null) return
           const headers = {
-            'content-length': String(again.size),
+            'content-length': String(copy.size),
             'cache-control': 'public, max-age=31536000, immutable',
           }
-          await (await cache).put(key, new Response(again.body, { headers }))
-        })(),
+          await (await cache).put(key, new Response(copy.body, { headers }))
+        })().catch((err: unknown) =>
+          // The client has its bytes; only the copy failed. Thrown, it was
+          // an uncaught exception in the dashboard's Issues (2026-11-01).
+          console.warn(`vx-cache-server: the data center's copy of ${name}: ${String(err)}`),
+        ),
       )
       return { body: o.body, size: o.size }
     },
@@ -135,12 +167,16 @@ export function r2Blobs(
     // which outlives R2's (expiry, a wipe), so a writer told "held" there
     // skipped its upload and a reader in another data center found nothing
     // (2026-10-27: a DTE worker's save, then the run's read, a 404).
-    size: async (name) => {
-      if (expiryDays !== undefined) {
+    // A PUT's skip asks R2 alone: a mark outlives a deletion by hand, and
+    // the upload it answered was dropped until the mark's expiry. It costs
+    // no head a mark saved: a client checks first and uploads only what
+    // the check missed, which no mark holds.
+    size: async (name, fromBucket) => {
+      if (expiryDays !== undefined && fromBucket !== true) {
         const held = await (await cache).match(heldOf(name))
         if (held !== undefined) return Number(held.headers.get(SIZE))
       }
-      const o = await bucket.head(name)
+      const o = await again(() => bucket.head(name))
       if (o === null) return null
       mark(name, o.size, o.uploaded.getTime())
       return o.size

package/wrangler.jsonc

@@ -13,6 +13,10 @@
   // EXPIRY_DAYS: the bucket's lifecycle rule in days, so a repeat check
   // skips R2 (README.md § On Cloudflare Workers); never above the rule.
   "vars": { "ALLOW": "" },
+  // No cron: the Worker exports no scheduled(). Unset, wrangler leaves a
+  // dashboard's or an old deploy's cron in place, which failed daily
+  // (2026-10-26 to 11-06); an empty list clears it on every deploy.
+  "triggers": { "crons": [] },
   // Expiry is the bucket's lifecycle rules (README.md § Expiry).
   "r2_buckets": [{ "binding": "BLOBS", "bucket_name": "vx-cache" }],
 }