zuplo 7.0.0 → 7.0.2

package/docs/caching/partial-response-caching.mdx

@@ -0,0 +1,641 @@
+---
+title: Cache part of a response
+description:
+  Cache the shared portion of a response body in the programmable cache and
+  fetch only the small caller-specific slice from your backend on every request.
+  A worked example that turns a thousand backend calls a minute into six.
+---
+
+A cinema-ticketing API serves `GET /v1/showtimes/board`. Every response contains
+two very different kinds of data:
+
+- **Citywide state.** Every showtime for the next seven days across 300 screens,
+  plus the cinema and movie metadata each one points at. Roughly 180 KB, and it
+  takes the backend about 340 ms to assemble. Every caller receives exactly the
+  same bytes.
+- **Caller-specific state.** The requesting account's seat holds and its loyalty
+  balance. Roughly 4 KB, and it takes the backend about 35 ms.
+
+The shared part is 90% of the backend's work and 98% of the bytes, and it's
+identical for every one of the thousand callers who hit the endpoint each
+minute. The backend builds it a thousand times a minute anyway, because it's
+glued to a 4 KB slice that differs per caller.
+
+Caching the shared fragment for 10 seconds fixes that. A 10-second freshness
+window means the snapshot is fetched at most six times a minute, no matter
+whether a thousand or a hundred thousand requests arrive. Backend calls for the
+shared fragment stop scaling with traffic and start scaling with the clock.
+
+| Per minute, at 1,000 requests    | Before  | After   |
+| -------------------------------- | ------- | ------- |
+| Backend calls for the snapshot   | 1,000   | 6       |
+| Backend calls for account data   | 1,000   | 1,000   |
+| Bytes read from the backend      | ~184 MB | ~5.1 MB |
+| Backend time on the request path | ~375 ms | ~35 ms  |
+
+At that rate the backend sends 265 GB a day before the change and 7.3 GB a day
+after it: a 97% cut in egress, and a 99.4% cut in the expensive call. The
+gateway still makes a thousand backend calls a minute, but they're the cheap
+ones.
+
+## Why the coarser cache layers can't help
+
+A [CDN](./cdn-caching.mdx) can't cache this response. Caching at the edge
+requires that two callers holding the same cache key receive identical bytes,
+and no two callers ever do, because one account's seat holds are in the body.
+Adding the API key to the cache key technically works, and it's the wrong shape:
+it stores a thousand near-duplicate 184 KB objects instead of one shared 180 KB
+object, each expires on its own schedule, and each miss pays the full backend
+cost. For the long tail of callers who show up once every few minutes, the hit
+rate rounds to zero.
+
+The [gateway response cache](./gateway-caching.mdx) has the same problem, plus a
+sharper edge. If the cache key omits the caller's identity, one account receives
+another account's seat holds. That's a correctness bug, not a tuning problem.
+
+Both layers cache **responses**. The unit actually cacheable here is a
+**fragment** of a response, and nothing outside the request can see fragments.
+Splitting a payload into a shared part and a personal part requires code that
+understands what the payload means, which means code running inside the request,
+with a cache it can address directly.
+
+## The two request paths
+
+Before, every request pulls the entire payload from the backend:
+
+<Diagram height="h-48">
+  <DiagramNode id="clients">1,000 requests/min</DiagramNode>
+  <DiagramNode id="gateway" variant="zuplo">
+    Zuplo gateway
+  </DiagramNode>
+  <DiagramNode id="backend" variant="orange">
+    Backend
+  </DiagramNode>
+  <DiagramEdge from="clients" to="gateway" />
+  <DiagramEdge
+    from="gateway"
+    to="backend"
+    label="1,000 full payloads/min"
+    variant="orange"
+  />
+</Diagram>
+
+After, the shared fragment comes from the cache and only the small per-caller
+call reaches the backend:
+
+<Diagram height="h-72">
+  <DiagramNode id="clients">1,000 requests/min</DiagramNode>
+  <DiagramNode id="handler" variant="zuplo">
+    Custom handler
+  </DiagramNode>
+  <DiagramNode id="cache" variant="green">
+    ZoneCache
+  </DiagramNode>
+  <DiagramNode id="backend" variant="orange">
+    Backend
+  </DiagramNode>
+  <DiagramEdge from="clients" to="handler" />
+  <DiagramEdge
+    from="handler"
+    to="cache"
+    label="snapshot: 1,000 hits/min"
+    variant="green"
+  />
+  <DiagramEdge
+    from="cache"
+    to="backend"
+    label="refresh: 6/min"
+    lineStyle="dashed"
+    variant="orange"
+  />
+  <DiagramEdge
+    from="handler"
+    to="backend"
+    label="account: 1,000/min"
+    variant="orange"
+    fromSide="bottom"
+    toSide="bottom"
+  />
+</Diagram>
+
+## Split the backend call in two
+
+Start from the handler that doesn't split anything. It forwards one request and
+returns one response, and there is nothing in it a cache can grab hold of.
+
+```ts title="modules/showtime-board.ts"
+import { environment, ZuploContext, ZuploRequest } from "@zuplo/runtime";
+
+export default async function handler(
+  request: ZuploRequest,
+  context: ZuploContext,
+) {
+  const accountId = request.user?.data.accountId;
+
+  return fetch(
+    `${environment.TICKETING_API_URL}/v1/showtimes/board/${accountId}`,
+    {
+      headers: { authorization: `Bearer ${environment.TICKETING_API_KEY}` },
+    },
+  );
+}
+```
+
+The backend needs to expose the two halves separately before anything can be
+cached: one endpoint for the citywide snapshot, one for a single account. Most
+APIs shaped like this already have both, because the composite endpoint was
+built on top of them.
+
+```ts title="modules/showtime-board.ts"
+import { environment } from "@zuplo/runtime";
+
+interface ShowtimeSnapshot {
+  asOf: string;
+  cinemas: { id: string; name: string; screens: number }[];
+  showtimes: {
+    id: string;
+    cinemaId: string;
+    movieId: string;
+    startsAt: string;
+    seatsAvailable: number;
+  }[];
+}
+
+interface AccountState {
+  holds: { showtimeId: string; seats: string[]; expiresAt: string }[];
+  loyalty: { points: number; tier: string };
+}
+
+async function fetchJson<T>(path: string): Promise<T> {
+  const response = await fetch(`${environment.TICKETING_API_URL}${path}`, {
+    headers: { authorization: `Bearer ${environment.TICKETING_API_KEY}` },
+  });
+  if (!response.ok) {
+    throw new Error(`Backend request to ${path} failed: ${response.status}`);
+  }
+  return (await response.json()) as T;
+}
+
+const fetchShowtimeSnapshot = () =>
+  fetchJson<ShowtimeSnapshot>("/v1/showtimes/snapshot");
+
+const fetchAccountState = (accountId: string) =>
+  fetchJson<AccountState>(`/v1/accounts/${accountId}`);
+```
+
+Two calls where there was one looks like a step backwards, and on a cache miss
+it is: the handler pays both round trips. The point is that one of the two is
+now addressable on its own.
+
+`fetchAccountState` never gets cached. It runs on every request, for every
+caller, and it is the reason the composed response carries
+`Cache-Control: no-store` at the end.
+
+## Cache the shared fragment
+
+[ZoneCache](../programmable-api/zone-cache.mdx) stores JSON-serializable values
+under a string key with a per-item TTL, shared across the isolates in one zone.
+The snapshot is JSON and every caller wants the same copy, so it fits exactly.
+
+```ts title="modules/showtime-board.ts"
+import { ZoneCache, ZuploContext } from "@zuplo/runtime";
+
+const SNAPSHOT_KEY = "showtime-snapshot:v1";
+const FRESH_FOR_SECONDS = 10;
+
+async function getShowtimeSnapshot(
+  context: ZuploContext,
+): Promise<ShowtimeSnapshot> {
+  const cache = new ZoneCache<ShowtimeSnapshot>("showtimes", context);
+
+  const cached = await cache.get(SNAPSHOT_KEY);
+  if (cached) {
+    return cached;
+  }
+
+  const snapshot = await fetchShowtimeSnapshot();
+  await cache.put(SNAPSHOT_KEY, snapshot, FRESH_FOR_SECONDS);
+  return snapshot;
+}
+```
+
+Ten seconds is a freshness budget, not a round number. Work it out from the
+product, in this order:
+
+1. **How fast does the source change?** The seat-inventory feed publishes every
+   two seconds. Caching for less than that buys nothing, because the backend
+   returns the same numbers.
+2. **What has the API promised?** If the documented contract says seat counts
+   are no more than 15 seconds old, that is the hard ceiling.
+3. **Leave room for the refresh itself.** A refresh that takes 340 ms against a
+   15-second ceiling wants a TTL comfortably under it. Ten seconds leaves five
+   seconds of margin.
+
+The `:v1` suffix on the key is worth the four characters. When
+`ShowtimeSnapshot` gains or renames a field, bump it to `:v2` and the deploy
+reads a cold key instead of deserializing old-shaped objects into new-shaped
+types.
+
+## Compose the response
+
+The handler fetches both halves in parallel and merges them into the body the
+client already expects. The response shape does not change, so no consumer has
+to be told about any of this.
+
+```ts title="modules/showtime-board.ts"
+import { HttpProblems, ZuploContext, ZuploRequest } from "@zuplo/runtime";
+
+export default async function handler(
+  request: ZuploRequest,
+  context: ZuploContext,
+) {
+  const accountId = request.user?.data.accountId;
+  if (!accountId) {
+    return HttpProblems.unauthorized(request, context);
+  }
+
+  const [snapshot, account] = await Promise.all([
+    getShowtimeSnapshot(context),
+    fetchAccountState(accountId),
+  ]);
+
+  return new Response(
+    JSON.stringify({
+      asOf: snapshot.asOf,
+      cinemas: snapshot.cinemas,
+      showtimes: snapshot.showtimes,
+      account,
+    }),
+    {
+      status: 200,
+      headers: {
+        "content-type": "application/json",
+        "cache-control": "no-store",
+      },
+    },
+  );
+}
+```
+
+`Promise.all` matters here. On a cache miss the handler pays
+`max(340 ms, 35 ms)` instead of `340 ms + 35 ms`, and on a hit the account call
+is the only thing left on the critical path.
+
+:::caution{title="The composed response is not cacheable"}
+
+`Cache-Control: no-store` is deliberate. The body now contains one account's
+seat holds, so it must not land in a CDN, a browser, or the gateway response
+cache. Caching happens strictly inside the handler, at the fragment level.
+
+:::
+
+## Collapse concurrent misses
+
+The snapshot expires ten seconds after it is written. At a thousand requests a
+minute (call it 17 a second), every request in flight at that instant sees an
+empty cache and calls the backend. The backend gets a burst of simultaneous
+snapshot requests, each one taking 340 ms, and during those 340 ms more requests
+arrive and pile onto the same burst. That is a cache stampede, and it lands
+hardest on exactly the backend the cache was supposed to protect.
+
+A module-scoped promise map fixes it. The first miss starts the fetch and stores
+the promise; every other miss for the same key awaits that promise instead of
+starting its own.
+
+```ts title="modules/showtime-board.ts"
+const inFlight = new Map<string, Promise<ShowtimeSnapshot>>();
+
+function loadOnce(
+  key: string,
+  load: () => Promise<ShowtimeSnapshot>,
+): Promise<ShowtimeSnapshot> {
+  const existing = inFlight.get(key);
+  if (existing) {
+    return existing;
+  }
+
+  const pending = load().finally(() => inFlight.delete(key));
+  inFlight.set(key, pending);
+  return pending;
+}
+```
+
+Module scope in a Zuplo project means isolate scope. The map lives as long as
+the isolate that loaded the module, and every request served by that isolate
+shares it.
+
+:::note
+
+This guard collapses concurrent misses **within one isolate**, not globally. Two
+isolates that miss at the same moment still make two backend calls, and a zone
+running twenty isolates can make twenty. That's the honest ceiling, and it's a
+fixed, small number set by your isolate count, rather than a number that grows
+with request volume.
+
+:::
+
+## Serve stale on backend failure
+
+Once a shared fragment is cached, a failed refresh does not have to become a
+failed request. Keep the cached copy alive well past its freshness window and
+fall back to it when the backend is down.
+
+Store the snapshot with a timestamp under a long TTL, and treat freshness as
+something the code decides rather than something the cache enforces. The cache
+now holds an entry rather than a bare snapshot, and it holds that entry far
+longer than the snapshot stays fresh:
+
+```ts title="modules/showtime-board.ts"
+interface SnapshotEntry {
+  snapshot: ShowtimeSnapshot;
+  storedAt: number;
+}
+
+const FRESH_FOR_SECONDS = 10;
+const SERVE_STALE_FOR_SECONDS = 300;
+```
+
+`getShowtimeSnapshot` now opens the cache as `ZoneCache<SnapshotEntry>`,
+compares `storedAt` against `FRESH_FOR_SECONDS`, returns the age alongside the
+snapshot, and logs a warning when it falls back so that stale serves show up in
+your logs. The full function is in [The complete handler](#the-complete-handler)
+below; these are the lines that carry the behavior:
+
+```ts title="modules/showtime-board.ts"
+const entry = await cache.get(SNAPSHOT_KEY);
+const ageSeconds = entry ? (Date.now() - entry.storedAt) / 1000 : Infinity;
+
+if (entry && ageSeconds < FRESH_FOR_SECONDS) {
+  return { snapshot: entry.snapshot, ageSeconds };
+}
+
+try {
+  const snapshot = await loadOnce(SNAPSHOT_KEY, async () => {
+    const fresh = await fetchShowtimeSnapshot();
+    await cache.put(
+      SNAPSHOT_KEY,
+      { snapshot: fresh, storedAt: Date.now() },
+      SERVE_STALE_FOR_SECONDS,
+    );
+    return fresh;
+  });
+  return { snapshot, ageSeconds: 0 };
+} catch (error) {
+  if (entry) {
+    return { snapshot: entry.snapshot, ageSeconds };
+  }
+  throw error;
+}
+```
+
+The cache TTL is now 300 seconds and the freshness window is 10. Between those
+two numbers sits a copy nobody is served under normal conditions and everybody
+is served during an outage. A five-minute backend failure degrades the endpoint
+from "seat counts at most 10 seconds old" to "seat counts up to 5 minutes old"
+instead of returning `502` to every caller.
+
+For a lot of APIs that availability win is worth more than the cost saving that
+motivated the change. Surface the age so clients can decide for themselves:
+`x-snapshot-age` in the complete handler below is an integer count of seconds.
+
+## The complete handler
+
+Everything above, in one file you can drop into `modules/` and adapt.
+
+```ts title="modules/showtime-board.ts"
+import {
+  environment,
+  HttpProblems,
+  ZoneCache,
+  ZuploContext,
+  ZuploRequest,
+} from "@zuplo/runtime";
+
+interface ShowtimeSnapshot {
+  asOf: string;
+  cinemas: { id: string; name: string; screens: number }[];
+  showtimes: {
+    id: string;
+    cinemaId: string;
+    movieId: string;
+    startsAt: string;
+    seatsAvailable: number;
+  }[];
+}
+
+interface AccountState {
+  holds: { showtimeId: string; seats: string[]; expiresAt: string }[];
+  loyalty: { points: number; tier: string };
+}
+
+interface SnapshotEntry {
+  snapshot: ShowtimeSnapshot;
+  storedAt: number;
+}
+
+const SNAPSHOT_KEY = "showtime-snapshot:v1";
+const FRESH_FOR_SECONDS = 10;
+const SERVE_STALE_FOR_SECONDS = 300;
+
+const inFlight = new Map<string, Promise<ShowtimeSnapshot>>();
+
+function loadOnce(
+  key: string,
+  load: () => Promise<ShowtimeSnapshot>,
+): Promise<ShowtimeSnapshot> {
+  const existing = inFlight.get(key);
+  if (existing) {
+    return existing;
+  }
+
+  const pending = load().finally(() => inFlight.delete(key));
+  inFlight.set(key, pending);
+  return pending;
+}
+
+async function fetchJson<T>(path: string): Promise<T> {
+  const response = await fetch(`${environment.TICKETING_API_URL}${path}`, {
+    headers: { authorization: `Bearer ${environment.TICKETING_API_KEY}` },
+  });
+  if (!response.ok) {
+    throw new Error(`Backend request to ${path} failed: ${response.status}`);
+  }
+  return (await response.json()) as T;
+}
+
+const fetchShowtimeSnapshot = () =>
+  fetchJson<ShowtimeSnapshot>("/v1/showtimes/snapshot");
+
+const fetchAccountState = (accountId: string) =>
+  fetchJson<AccountState>(`/v1/accounts/${accountId}`);
+
+async function getShowtimeSnapshot(
+  context: ZuploContext,
+): Promise<{ snapshot: ShowtimeSnapshot; ageSeconds: number }> {
+  const cache = new ZoneCache<SnapshotEntry>("showtimes", context);
+  const entry = await cache.get(SNAPSHOT_KEY);
+  const ageSeconds = entry ? (Date.now() - entry.storedAt) / 1000 : Infinity;
+
+  if (entry && ageSeconds < FRESH_FOR_SECONDS) {
+    return { snapshot: entry.snapshot, ageSeconds };
+  }
+
+  try {
+    const snapshot = await loadOnce(SNAPSHOT_KEY, async () => {
+      const fresh = await fetchShowtimeSnapshot();
+      await cache.put(
+        SNAPSHOT_KEY,
+        { snapshot: fresh, storedAt: Date.now() },
+        SERVE_STALE_FOR_SECONDS,
+      );
+      return fresh;
+    });
+    return { snapshot, ageSeconds: 0 };
+  } catch (error) {
+    if (entry) {
+      context.log.warn("Snapshot refresh failed, serving stale copy", {
+        ageSeconds,
+        error: String(error),
+      });
+      return { snapshot: entry.snapshot, ageSeconds };
+    }
+    throw error;
+  }
+}
+
+export default async function handler(
+  request: ZuploRequest,
+  context: ZuploContext,
+) {
+  const accountId = request.user?.data.accountId;
+  if (!accountId) {
+    return HttpProblems.unauthorized(request, context);
+  }
+
+  try {
+    const [shared, account] = await Promise.all([
+      getShowtimeSnapshot(context),
+      fetchAccountState(accountId),
+    ]);
+
+    return new Response(
+      JSON.stringify({
+        asOf: shared.snapshot.asOf,
+        cinemas: shared.snapshot.cinemas,
+        showtimes: shared.snapshot.showtimes,
+        account,
+      }),
+      {
+        status: 200,
+        headers: {
+          "content-type": "application/json",
+          "cache-control": "no-store",
+          "x-snapshot-age": String(Math.round(shared.ageSeconds)),
+        },
+      },
+    );
+  } catch (error) {
+    context.log.error("Showtime board composition failed", {
+      error: String(error),
+    });
+    return HttpProblems.badGateway(request, context, {
+      detail: "The showtimes service is unavailable.",
+    });
+  }
+}
+```
+
+Point a route at it. The handler reads `request.user`, so keep an authentication
+policy ahead of it in the inbound pipeline.
+
+```json title="config/routes.oas.json"
+{
+  "paths": {
+    "/v1/showtimes/board": {
+      "get": {
+        "summary": "Showtime board",
+        "x-zuplo-route": {
+          "corsPolicy": "none",
+          "handler": {
+            "export": "default",
+            "module": "$import(./modules/showtime-board)"
+          },
+          "policies": {
+            "inbound": ["api-key-inbound"]
+          }
+        }
+      }
+    }
+  }
+}
+```
+
+## Choose the cache primitive
+
+Three primitives can hold a fragment. They differ in scope, in what they can
+store, and in how long a value survives.
+
+| Primitive                                                                            | Scope                                                            | Stores                                            | Reach for it when                                                                                                               |
+| ------------------------------------------------------------------------------------ | ---------------------------------------------------------------- | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
+| [ZoneCache](../programmable-api/zone-cache.mdx)                                      | Shared across the isolates in a zone; survives isolate recycling | JSON-serializable values, up to 512 MB per object | The shared fragment. This is the default for the pattern on this page.                                                          |
+| [MemoryZoneReadThroughCache](../programmable-api/memory-zone-read-through-cache.mdx) | One isolate; dies with it                                        | Any in-memory value                               | A very hot, very small value where a zone round trip is measurable, such as a feature flag map or a decoded configuration blob. |
+| [Cache API](../programmable-api/cache.mdx) (`caches.open`)                           | Long-lived memory, keyed by `Request`                            | Full `Request`/`Response` pairs                   | The fragment is literally an upstream HTTP response you want to keep verbatim, with its `Cache-Control` and `ETag` respected.   |
+
+When the fragment is a parsed object that code composes into a larger body,
+ZoneCache wins: a recycled isolate warms from the zone rather than from the
+backend. Reach for the Cache API instead when the fragment is a response you
+want kept verbatim (an upstream document, an image, a signed payload) and
+re-parsing it on every request would be wasted work.
+
+## Measure the result
+
+Measure before and after, over the same window and the same routes. Expect tail
+latency to move more than the median, because a hit removes the slower of the
+two backend calls.
+
+1. **Fragment hit rate.** The cache itself does not report this, so count it:
+   log a structured field on hit and on miss, or increment a counter. A healthy
+   hit rate for a 10-second TTL at a thousand requests a minute is above 99%. A
+   number far below that means traffic is spread thinner across zones than
+   expected, or the TTL is shorter than the gap between requests.
+2. **Backend call volume.** Watch the request count against the snapshot
+   endpoint in the **Origins** section of Analytics, which breaks out volume,
+   error rate, and latency per upstream host. See
+   [Origins](../analytics/tabs/origins.md).
+3. **p95 on the route.** Compare against the same window a week earlier, not
+   against the ten minutes before the deploy. See
+   [Metrics glossary](../analytics/reference/metrics-glossary.md) for how p95 is
+   computed.
+
+## Limitations you own
+
+- **Hit rate depends on traffic distribution.** ZoneCache is zone-local, and a
+  write in one data center does nothing for any other. Spread the same thousand
+  requests a minute across six zones and the backend sees up to 36 snapshot
+  refreshes a minute rather than six. That is still a 96% reduction, but not the
+  99.4% a single-zone calculation predicts. Low-volume routes with globally
+  scattered traffic may miss more often than they hit.
+- **A stale fragment is stale for everybody at once.** With a per-caller cache,
+  one unlucky client sees old data. With a shared fragment, every caller sees
+  the same old data simultaneously, and they can compare notes. The TTL is a
+  product decision about what the API promises, not a knob to tune for hit rate.
+- **You own invalidation.** ZoneCache has no global purge and no tag-based
+  eviction. An entry disappears when its TTL expires or when code calls `delete`
+  in the zone it lives in. If a fragment must drop within a bounded time of an
+  upstream change, the TTL is the only mechanism that guarantees it. Choose it
+  accordingly, and version the key so a schema change does not depend on
+  invalidation at all.
+- **Two calls cost more on a miss.** A cold cache pays for both round trips.
+  `Promise.all` keeps that concurrent, but the first request after a deploy into
+  a fresh zone is slower than the unsplit version was. Weigh that against how
+  often it happens.
+
+## Related
+
+- [Caching in Zuplo](./overview.mdx) — which layer to reach for, and why.
+- [Cache at the gateway](./gateway-caching.mdx) — whole-response caching when
+  responses really are identical across callers.
+- [Build a custom caching policy](./custom-caching-policy.mdx) — the same
+  primitives packaged as a reusable policy instead of a handler.
+- [Function Handler](../handlers/custom-handler.mdx) — the handler contract used
+  throughout this page.

package/docs/cli/overview.mdx

@@ -11,10 +11,9 @@ directly, if you want to create your own tooling.
 
 ## Installing
 
-The Zuplo CLI is built using Node.js. It requires a minimum version of Node.js
-20.0.0 (Node.js 22 is recommended).
+The Zuplo CLI is built using Node.js. It requires Node.js 24.0.0 or greater.
 
-1. Install Node.js 20.0.0 or later. You can download it from
+1. Install Node.js 24.0.0 or later. You can download it from
    [nodejs.org](https://nodejs.org/en/download/).
 1. Install the Zuplo CLI globally by running the following command: