zuplo 7.0.0 → 7.0.2

package/docs/caching/custom-caching-policy.mdx

@@ -0,0 +1,426 @@
+---
+title: Build a custom caching policy
+description:
+  Write your own inbound and outbound policy pair with the Cache API when no
+  built-in policy fits your caching rules, from tenant-scoped keys and size and
+  status filters to upstream-driven TTLs and independent invalidation.
+---
+
+Custom caching code is the most expensive caching you can ship. You own it, you
+debug it, and the failure mode is never a slow API. It's one customer reading
+another customer's data. Reach for it only after a built-in policy has failed to
+express what you need.
+
+## Check the built-in policies first
+
+Most caching requirements are already a configuration option somewhere. Find
+your row before writing anything.
+
+| You want                                                    | Use this instead                                                                                                  |
+| ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
+| Cache responses at the gateway so repeats skip your backend | [Cache at the gateway](./gateway-caching.mdx)                                                                     |
+| A key that varies by `Accept` or `Accept-Language`          | The Caching policy's `headers` option                                                                             |
+| A key that varies by the raw `Authorization` header         | The Caching policy includes it by default                                                                         |
+| Flush every cached response at once                         | The [Caching policy's `cacheId`](./gateway-caching.mdx#invalidate-with-cacheid), bound to an environment variable |
+| Cache at the CDN so requests never reach the gateway at all | [Cache at the CDN](./cdn-caching.mdx)                                                                             |
+| A different TTL per route, or per response                  | The [`cacheConfig` function](./dynamic-cache-rules.mdx)                                                           |
+| Purge tags for targeted CDN invalidation                    | The CDN policy's `tags` option                                                                                    |
+| Cache GraphQL query results by query and variables          | [`graphql-cache-inbound`](../policies/graphql-cache-inbound.mdx)                                                  |
+| Cache LLM completions by prompt similarity                  | [`semantic-cache-inbound`](../policies/semantic-cache-inbound.mdx)                                                |
+| Store an arbitrary JSON value with a TTL from your own code | [`ZoneCache`](../programmable-api/zone-cache.mdx)                                                                 |
+
+## When custom code is the right answer
+
+What survives that table is a short list. These are the cases where no
+combination of built-in options gets you there:
+
+- **The key depends on a decoded value, not a raw header.** Two JWTs for the
+  same tenant are different strings, so keying on `Authorization` gives a
+  near-zero hit rate. The `org_id` claim gives one entry per tenant.
+- **You cache only some status codes, or only small responses.** A 5 MB export
+  isn't worth a cache slot; a 4 KB catalog page is.
+- **The TTL comes from the upstream response.** The backend knows how volatile
+  each record is and says so in a header.
+- **Each tenant needs its own namespace with independent invalidation.** One
+  tenant's write should drop that tenant's entries and leave every other
+  tenant's untouched.
+- **You want soft purge or stale-while-revalidate behavior.** Serve the stale
+  copy now and refresh it in the background.
+
+Everything else, which is most of it, belongs in a built-in policy.
+
+## An inbound and outbound policy pair
+
+A gateway cache is two policies on one route. The inbound policy looks for a
+stored copy and returns a `Response` to short-circuit the pipeline on a hit. The
+outbound policy stores the response on the way back out.
+
+<Diagram height="h-56">
+  <DiagramNode id="client">Client</DiagramNode>
+  <DiagramNode id="inbound" variant="zuplo">
+    Inbound: cache.match
+  </DiagramNode>
+  <DiagramNode id="backend">Backend</DiagramNode>
+  <DiagramNode id="outbound" variant="zuplo">
+    Outbound: cache.put
+  </DiagramNode>
+  <DiagramEdge from="client" to="inbound" />
+  <DiagramEdge from="inbound" to="backend" label="Miss" />
+  <DiagramEdge from="backend" to="outbound" />
+  <DiagramEdge from="inbound" to="client" label="Hit" lineStyle="dashed" />
+</Diagram>
+
+Both exports live in one module so they share the key function and the options
+type. Register the read side:
+
+```json title="config/policies.json"
+{
+  "policies": [
+    {
+      "name": "tenant-cache-read",
+      "policyType": "custom-code-inbound",
+      "handler": {
+        "export": "tenantCacheInbound",
+        "module": "$import(./modules/tenant-cache)",
+        "options": {
+          "namespace": "catalog",
+          "defaultTtlSeconds": 300,
+          "maxTtlSeconds": 3600,
+          "maxBytes": 262144,
+          "cacheableStatusCodes": [200, 203],
+          "varyByQuery": ["page", "pageSize", "sort"]
+        }
+      }
+    }
+  ]
+}
+```
+
+Add a second entry named `tenant-cache-write` with `policyType` set to
+`custom-code-outbound`, `export` set to `tenantCacheOutbound`, and the same
+`options` object. The two policies have to agree on the namespace and on every
+input to the key, or the write lands somewhere the read never looks.
+
+Order matters on the route: authentication runs before the cache read, because
+the key depends on the authenticated identity.
+
+```json title="config/routes.oas.json"
+{
+  "policies": {
+    "inbound": ["jwt-auth", "tenant-cache-read"],
+    "outbound": ["tenant-cache-write"]
+  }
+}
+```
+
+## Build the cache key
+
+The Cache API keys entries by `Request`, so a custom key is a synthetic URL you
+build yourself. Keep the function pure: same request in, same string out.
+
+```ts title="modules/tenant-cache.ts"
+import type { ZuploRequest } from "@zuplo/runtime";
+
+export interface CacheOptions {
+  namespace: string;
+  defaultTtlSeconds: number;
+  maxTtlSeconds: number;
+  maxBytes: number;
+  cacheableStatusCodes: number[];
+  varyByQuery: string[];
+}
+
+export type TenantRequest = ZuploRequest<{ UserData: { org_id?: string } }>;
+
+/** The tenant this request reads for, or undefined when unauthenticated. */
+export function getTenantId(request: TenantRequest): string | undefined {
+  return request.user?.data?.org_id;
+}
+
+/** Stable, collision-free cache key. Pure: no clocks, no randomness. */
+export function cacheKey(
+  request: TenantRequest,
+  tenantId: string,
+  options: CacheOptions,
+): string {
+  const url = new URL(request.url);
+
+  // Allowlist parameters, lowercase the names, and sort, so ?sort=name&page=2
+  // and ?page=2&sort=name produce one entry instead of two.
+  const params = [...url.searchParams]
+    .map(([name, value]) => [name.toLowerCase(), value] as [string, string])
+    .filter(([name]) => options.varyByQuery.includes(name))
+    .sort((a, b) => a[0].localeCompare(b[0]) || a[1].localeCompare(b[1]));
+
+  const search = new URLSearchParams(params).toString();
+  const query = search ? `?${search}` : "";
+  const tenant = encodeURIComponent(tenantId);
+  const path = url.pathname.toLowerCase();
+
+  return `https://cache.invalid/${options.namespace}/${tenant}${path}${query}`;
+}
+```
+
+Three rules do the real work here.
+
+**Determinism.** Sorting and lowercasing collapse the same logical request onto
+one entry. An unsorted key turns one cacheable resource into a pile of
+near-duplicates that each get read once.
+
+**Bounded cardinality.** `varyByQuery` is an allowlist, not a blocklist. A
+free-text `q=` parameter or a client-supplied timestamp has unbounded values, so
+every request writes an entry nobody reads again — a 0% hit rate that still
+costs a write per request.
+
+**Namespacing.** The tenant id sits in a fixed segment, URL-encoded so a value
+containing `/` cannot escape it. Dropping one tenant's entries becomes a prefix
+operation, and no two tenants can produce the same string.
+
+:::danger
+
+A cache key that omits the caller identity will serve one tenant's data to
+another. This is not a theoretical risk — it is the default outcome. If
+`getTenantId` returns `undefined`, the inbound policy must skip the cache
+entirely rather than fall back to a shared key. Never derive the tenant from a
+client-supplied header such as `X-Tenant-Id`; derive it from the authenticated
+`request.user` an authentication policy populated. Before shipping, add a test
+that sends one URL with two tenants' tokens and asserts the bodies differ.
+
+:::
+
+## Read the cache on the way in
+
+The inbound policy returns a `Response` to short-circuit, or the original
+`ZuploRequest` to continue to the backend.
+
+```ts title="modules/tenant-cache.ts"
+import { ContextData, environment, type ZuploContext } from "@zuplo/runtime";
+
+const CACHE_KEY = "tenant-cache-key";
+
+const cacheEnabled = () => environment.CACHE_DISABLED !== "true";
+
+export async function tenantCacheInbound(
+  request: TenantRequest,
+  context: ZuploContext,
+  options: CacheOptions,
+): Promise<ZuploRequest | Response> {
+  const tenantId = getTenantId(request);
+
+  // No identity, no shared cache. Never fall back to a tenant-less key.
+  if (!cacheEnabled() || request.method !== "GET" || !tenantId) {
+    return request;
+  }
+
+  const key = cacheKey(request, tenantId, options);
+  ContextData.set(context, CACHE_KEY, key);
+
+  const cache = await caches.open(options.namespace);
+  const stored = await cache.match(new Request(key));
+
+  if (!stored) {
+    context.log.debug("cache miss", { key: await keyDigest(key) });
+    return request;
+  }
+
+  // Copy into a fresh Response so the headers are mutable, then label the hit.
+  const storedAt = Number(stored.headers.get("x-cache-stored-at") ?? 0);
+  const age = Math.max(0, Math.floor((Date.now() - storedAt) / 1000));
+
+  const response = new Response(stored.body, {
+    status: stored.status,
+    headers: stored.headers,
+  });
+  response.headers.set("cache-status", "zuplo; hit");
+  response.headers.set("age", String(age));
+  response.headers.delete("x-cache-stored-at");
+
+  // The stored Cache-Control is the Cache API's expiry clock, not advice for
+  // the caller. A tenant-scoped body must not be shareable downstream.
+  response.headers.set("cache-control", "private, no-store");
+
+  context.log.debug("cache hit", { key: await keyDigest(key), age });
+  return response;
+}
+```
+
+Returning the stored response ends the pipeline: the backend is never called and
+the outbound policies never run. That is the point, and it is also why the hit
+path has to set its own headers — nothing downstream will do it for you.
+
+## Write the cache on the way out
+
+The outbound policy decides whether the response is worth storing, then stores a
+clone.
+
+```ts title="modules/tenant-cache.ts"
+/** Clamp an upstream TTL so one bad header cannot pin a copy for a year. */
+function resolveTtl(response: Response, options: CacheOptions): number {
+  const raw = response.headers.get("x-cache-ttl");
+  const upstream = raw === null ? Number.NaN : Number.parseInt(raw, 10);
+
+  return Number.isFinite(upstream) && upstream > 0
+    ? Math.min(upstream, options.maxTtlSeconds)
+    : options.defaultTtlSeconds;
+}
+
+/** Re-wrap a response so its headers are mutable, then label it. */
+function label(response: Response, status: string): Response {
+  const copy = new Response(response.body, {
+    status: response.status,
+    headers: response.headers,
+  });
+  copy.headers.set("cache-status", status);
+  return copy;
+}
+
+export async function tenantCacheOutbound(
+  response: Response,
+  request: TenantRequest,
+  context: ZuploContext,
+  options: CacheOptions,
+): Promise<Response> {
+  const key = ContextData.get<string>(context, CACHE_KEY);
+  const size = Number(response.headers.get("content-length") ?? Number.NaN);
+
+  const storable =
+    cacheEnabled() &&
+    options.cacheableStatusCodes.includes(response.status) &&
+    Number.isFinite(size) &&
+    size <= options.maxBytes;
+
+  if (key === undefined || !storable) {
+    return label(response, "zuplo; fwd=miss");
+  }
+
+  // Clone before storing. The original body still has to reach the client.
+  const copy = response.clone();
+  const headers = new Headers(copy.headers);
+  headers.set("cache-control", `max-age=${resolveTtl(response, options)}`);
+  headers.set("x-cache-stored-at", Date.now().toString());
+
+  // Reusing upstream headers means inheriting upstream cache headers. Drop the
+  // ones that would compete with the Cache-Control just set.
+  headers.delete("expires");
+  headers.delete("etag");
+  headers.delete("last-modified");
+
+  const toStore = new Response(copy.body, { status: copy.status, headers });
+  const cache = await caches.open(options.namespace);
+  void cache
+    .put(new Request(key), toStore)
+    .catch((err) => context.log.error("cache write failed", { err }));
+
+  return label(response, "zuplo; fwd=miss; stored");
+}
+```
+
+`Cache-Control` on the stored copy carries the TTL — the Cache API reads it back
+on `match`, so a stored response without it has no expiry you control. It is
+internal machinery, not advice for the caller, which is why the hit path
+overwrites it with `private, no-store` before returning: a tenant-scoped body
+must never carry a header that invites a browser or a shared proxy to hold onto
+it. A response with no `Content-Length`, such as a streamed or chunked body,
+fails the size check and is skipped, which is correct: you cannot bound what you
+cannot measure.
+
+:::warning
+
+Outbound policies run on every response, not only successful ones, so a `502`
+from a failing backend reaches this code. The `cacheableStatusCodes` allowlist
+is the only thing standing between an error and a cached error, so keep it
+explicit and keep it narrow. It also stops a `204` or `206` from being stored by
+accident.
+
+:::
+
+## Pass state between the two policies
+
+The outbound policy must not recompute the key. The request may have been
+rewritten by then, and any drift between the two computations means you write to
+a key nobody reads. Compute it once inbound and hand it forward with
+[`ContextData`](../programmable-api/context-data.mdx), which is scoped to one
+request and garbage collected when that request completes.
+
+Setting the key inbound doubles as the signal for the outbound policy. When
+`ContextData.get` returns `undefined`, the inbound policy skipped the cache —
+unauthenticated request, non-`GET` method, or kill switch on — so the outbound
+policy skips it too, with no second copy of that logic to keep in sync.
+
+## Do not block the response on the cache write
+
+The `void cache.put(...)` above is deliberately not awaited. A client waiting on
+a successful response should not also wait on a cache write; the write is an
+optimization for the _next_ request.
+
+The `.catch` on it is mandatory, not stylistic. A floating promise that rejects
+becomes an unhandled rejection, and an unhandled rejection can fail the request
+that was already on its way out the door — a cache write error turning into a
+`500` for a response that was otherwise fine. Catching and logging keeps the
+failure in your logs and invisible to the caller.
+
+## Correctness checklist
+
+Walk this before the pull request goes up.
+
+- **Never store a response whose body was already consumed.** `response.json()`
+  or `response.text()` drains the stream, and everything downstream gets a
+  `body-used` error. See
+  [Safely clone a request or response](../programmable-api/safely-clone-a-request-or-response.mdx).
+- **Clone before storing.** `response.clone()` gives you a second readable body:
+  store the clone, return the original.
+- **Never cache an error.** Check the status against an explicit allowlist. A
+  cached `502` outlives the outage that caused it.
+- **Never let one tenant's key collide with another's.** URL-encode the tenant
+  id and keep it in a fixed position in the key.
+- **Bound the key space.** Allowlist the query parameters that vary the response
+  and ignore everything else.
+- **Always have a kill switch.** `CACHE_DISABLED=true` turns both policies into
+  pass-throughs. Flipping an
+  [environment variable](../articles/environment-variables.mdx) and redeploying
+  takes minutes and needs no code edit or review while the cache is serving
+  something wrong.
+
+## Measure the hit rate
+
+The `Cache-Status` header set above
+([RFC 9211](https://www.rfc-editor.org/rfc/rfc9211.html)) makes every response
+self-describing, so a developer chasing stale data reads the answer off their
+network tab instead of asking you.
+
+For aggregate hit rate, log a short digest of the key rather than the key itself
+— the key holds the tenant id and the query string, and neither belongs in a log
+line:
+
+```ts title="modules/tenant-cache.ts"
+/** Short, stable, non-reversible label for a cache key. */
+async function keyDigest(key: string): Promise<string> {
+  const bytes = new TextEncoder().encode(key);
+  const digest = await crypto.subtle.digest("SHA-256", bytes);
+  return [...new Uint8Array(digest)]
+    .slice(0, 8)
+    .map((byte) => byte.toString(16).padStart(2, "0"))
+    .join("");
+}
+```
+
+Hits divided by hits plus misses gives the hit rate, and grouping by digest
+surfaces the keys that never repeat — the ones to stop caching. The runtime
+supports the [Web Crypto API](../programmable-api/web-crypto-apis.mdx) directly,
+so this needs no dependency.
+
+## Related
+
+If the reason you are here is that only part of each response is cacheable, read
+[Cache part of a response](./partial-response-caching.mdx) before building any
+of this. Caching a shared fragment and fetching only the caller-specific
+remainder is the same machinery one level down, and it usually beats caching the
+whole payload.
+
+- [Caching in Zuplo](./overview.mdx) — which layer to reach for, and why.
+- [Cache at the gateway](./gateway-caching.mdx) — the built-in policy this page
+  replaces, and the options to exhaust before writing any code.
+- [Cache API](../programmable-api/cache.mdx) — the `caches.open`, `match`, and
+  `put` reference behind the code above.
+- [ContextData](../programmable-api/context-data.mdx) — how the cache key
+  travels from the inbound policy to the outbound one.