@boy-offi9-inc/reqkit 0.7.0 → 0.8.0

package/README.md

@@ -9,7 +9,6 @@
   <a href="https://github.com/boy-offi9-inc/reqkit/actions/workflows/ci.yml"><img src="https://github.com/boy-offi9-inc/reqkit/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
   <a href="./LICENSE"><img src="https://img.shields.io/badge/License-MIT-yellow.svg" alt="License: MIT"></a>
   <img src="https://img.shields.io/badge/node-%3E%3D18-339933?logo=node.js&logoColor=white" alt="Node >=18">
-  <img src="https://img.shields.io/badge/npm-%40boy--offi9--inc%2Freqkit-CB3837?logo=npm&logoColor=white" alt="npm package">
   <img src="https://img.shields.io/badge/dependencies-zero-brightgreen" alt="Zero dependencies">
   <img src="https://img.shields.io/badge/module-CJS%20%2B%20ESM-blue" alt="CJS + ESM">
   <img src="https://img.shields.io/badge/types-included-3178C6?logo=typescript&logoColor=white" alt="TypeScript types included">
@@ -19,7 +18,7 @@
 
 **Tiny, zero-dependency HTTP helpers for `fetch`, axios, or anything else — retries that respect `Retry-After`, timeouts that really cancel, and errors that make sense.**
 
-Not a client, not an axios replacement. Pull in the one function you need, or let [`createFetch`](#createfetchdefaults) wire them together for you — you always get a plain `Response` back.
+Not a client, not an axios replacement. Use one function on its own, or let [`createFetch`](#createfetchdefaults) wire them together. You always get a plain `Response` back.
 
 ```js
 import { createFetch } from "@boy-offi9-inc/reqkit";
@@ -49,32 +48,18 @@ const pages = await limit.map(urls, (url) =>
 );
 ```
 
----
-
-## Playground
-
-Try everything in your browser — retries obeying `Retry-After`, rate limiting, the circuit breaker, caching, SSE streaming, an SSE stream that reconnects itself, and a resumable download that survives a dropped connection. It runs against a fake in-memory server, so there's nothing to install and no network involved.
-
-Run it locally: `npm run build && python3 -m http.server -d docs` → open `http://localhost:8000`. (ES modules don't load from `file://`, so it needs a server.)
-
-## Works with what you already use
-
-Recipes for **axios, ky, got, undici/node-fetch**, testing, and LLM streaming live in [RECIPES.md](./RECIPES.md): keep your client, bolt on just the piece you're missing (`Retry-After`-aware retries, a rate limiter, a circuit breaker).
-
-## Runtime support
+## Install
 
-| Runtime | Status |
-|---|---|
-| **Node 18, 20, 22, 24** | Primary target. Full test suite runs in CI on all four. |
-| **Deno, Bun, browsers, Cloudflare Workers** | Use the dependency-free ESM bundle (`@boy-offi9-inc/reqkit/browser`, or `import … from "npm:@boy-offi9-inc/reqkit"` in Deno). Only web-standard APIs are used. CI smoke-tests the bundle on Deno, Bun and headless Chromium — **experimental** until those jobs have a track record; please open an issue if one misbehaves. |
+```bash
+npm install @boy-offi9-inc/reqkit
+```
 
-```html
-<script type="module">
-  import { createFetch } from "https://cdn.jsdelivr.net/npm/@boy-offi9-inc/reqkit@0.5.0/dist/reqkit.mjs";
-</script>
+```js
+import { createFetch, withRetry } from "@boy-offi9-inc/reqkit";      // ESM
+const { createFetch, withRetry } = require("@boy-offi9-inc/reqkit");  // CommonJS
 ```
 
----
+Requires Node 18+. TypeScript types are included.
 
 ## Pick what you need
 
@@ -90,7 +75,8 @@ Recipes for **axios, ky, got, undici/node-fetch**, testing, and LLM streaming li
 | Run N things at a time | [`createLimiter`](#createlimiterconcurrency) |
 | Slow down *before* the server says 429 (obey `X-RateLimit-*`) | [`createRateLimiter`](#createratelimiteropts) |
 | Stop hammering a service that keeps failing | [`createCircuitBreaker`](#createcircuitbreakeropts) |
-| Cache GETs (TTL, stale-while-revalidate, ETag/304) | [`createFetch` → `cache`](#caching) |
+| Cache GETs (TTL, stale-while-revalidate, stale-if-error, ETag/304) | [`createFetch` → `cache`](#caching) |
+| Log or measure every request, retry and cache hit | [`createFetch` → `onEvent`](#events-and-logging) |
 | Attach a token and refresh it on 401 — once, even with many requests in flight | [`createFetch` → `auth`](#auth-token-refresh) |
 | Walk a paginated API (cursor, offset, `Link` headers) | [`paginate` / `paginateLinks`](#pagination-paginate-paginatepages-paginatelinks) |
 | Resume a download after the connection drops | [`downloadResumable`](#downloadresumableurl-opts) |
@@ -99,26 +85,48 @@ Recipes for **axios, ky, got, undici/node-fetch**, testing, and LLM streaming li
 | Test retry/timeout code without a network | [`createMockFetch`](#createmockfetch) |
 | All of the above, wired up | [`createFetch`](#createfetchdefaults) |
 
----
+## Why reqkit
 
-## Why this exists
+- **No client to adopt.** Standalone functions that compose with whatever you already use — fetch, axios, ky, got, undici.
+- **It reads the server.** Retries wait for `Retry-After` / `X-RateLimit-Reset` instead of backing off blindly, and a rate limiter can slow down *before* the server says 429.
+- **Safe by default.** Retries only touch idempotent methods, credentials never leave your origin, and parsers bound their memory and time on hostile input.
+- **Errors that make sense.** Node's `TypeError: fetch failed`, the browser's `Failed to fetch`, axios and undici errors all normalize to one shape, and a deliberate abort is never reported as a timeout.
+- **Zero dependencies**, CommonJS and ESM, hand-written TypeScript types.
 
-Most HTTP retry/timeout wrappers bundle everything into a client you have to fully adopt. This is the opposite: standalone functions that compose with whatever you're already using. Zero dependencies, CJS + ESM, TypeScript types included.
+## Runtime support
 
-**Honest note:** retries, timeouts and error normalization are a well-covered problem — there are other solid packages doing similar things (`fetchy`, `fetchpilot`, `p-retry`, to name a few). What reqkit leans on:
+| Runtime | |
+|---|---|
+| **Node 18, 20, 22, 24** | Supported. |
+| **Deno, Bun, Cloudflare Workers, browsers** | Use the dependency-free ESM bundle, `@boy-offi9-inc/reqkit/browser`. Only web-standard APIs are used. |
 
-- **It reads the server.** Retries wait for `Retry-After` / `X-RateLimit-Reset` instead of blind backoff on 429s — most retry libraries treat rate-limit responses like any other failure.
-- **It's safe by default.** `createFetch` only retries idempotent methods, so a timed-out `POST` can't double-charge anyone.
-- **It understands real-world errors.** Node's `TypeError: fetch failed` and the browser's `Failed to fetch` are classified as network errors, and a deliberate abort is never reported as a timeout.
-- **It's not tied to a client.** `withProgress`, `withDedupe` and `createLimiter` work with any streamed body or async function.
+```js
+// Deno
+import { createFetch } from "npm:@boy-offi9-inc/reqkit";
+```
 
----
+```html
+<!-- Browser, no build step (pin a version in production) -->
+<script type="module">
+  import { createFetch } from "https://cdn.jsdelivr.net/npm/@boy-offi9-inc/reqkit/dist/reqkit.mjs";
+</script>
+```
 
-## Install
+## Playground and recipes
 
-```bash
-npm install @boy-offi9-inc/reqkit
-```
+- **[Interactive playground](https://boy-offi9-inc.github.io/reqkit/)** — every feature running in your browser against a fake server; nothing to install.
+- **[Recipes](./docs/recipes.md)** — using reqkit alongside axios, ky, got and undici; testing; LLM streaming.
+
+## Security
+
+reqkit handles data a server controls (headers, streams, redirects), so the defaults are conservative:
+
+- **Credentials stay on your origin.** With a `baseUrl`, the default `auth` token and the default `Authorization`, `Proxy-Authorization`, `Cookie`, `X-API-Key` and `X-Auth-Token` headers are only sent to that origin (plus any `trustedOrigins`). An absolute URL to another host doesn't receive them. Headers you pass for a specific call are always sent. Without a `baseUrl` or `trustedOrigins` there is no origin to compare against, so set at least one.
+- **Pagination stays on origin.** `paginateLinks` refuses a `Link: rel="next"` that points to another origin (`code: "PAGINATE_CROSS_ORIGIN"`); allow more with `origins: [...]`.
+- **Bounded input.** `readLines`, `readNDJSON` and `readSSE` run in linear time and throw `LINE_TOO_LONG` past `maxLineLength` (default 16 Mi characters); `parseLinkHeader` ignores headers over 64 KB; `withProgress` and `downloadResumable` accept `maxBytes`.
+- **Safe retries.** `createFetch` retries only idempotent methods (or requests carrying an `Idempotency-Key`). Cross-origin redirects are left to `fetch`, which strips `Authorization`.
+
+reqkit does not block requests to private or internal addresses. If you fetch user-supplied URLs on a server, validate them first. To report a vulnerability, see [SECURITY.md](./SECURITY.md).
 
 ---
 
@@ -134,7 +142,7 @@ const data = await withRetry(
   { retries: 3, baseDelayMs: 300, onRetry: (err, attempt, delayMs) => console.log(`retry ${attempt} in ${delayMs}ms`) }
 );
 
-// fetch doesn't throw on 4xx/5xx — retry on statuses by returning the Response:
+// fetch doesn't throw on 4xx/5xx, so list the statuses to retry:
 const res = await withRetry(() => fetch(url), { retryStatusCodes: [429, 500, 502, 503, 504] });
 ```
 
@@ -158,16 +166,15 @@ const res = await withRetry(() => fetch(url), { retryStatusCodes: [429, 500, 502
 When retries run out on a *result* (e.g. a 503), the last result is returned rather than thrown — you decide what a bad status means. Bodies of retried responses are cancelled so connections are freed.
 
 ```js
-// Cancel a long retry/backoff sequence externally — e.g. the user navigated away
 const controller = new AbortController();
 const promise = withRetry(() => fetch(url), { retries: 5, signal: controller.signal });
 controller.abort(); // rejects with RetryAbortedError, even mid-backoff
 ```
 
-**Stop early with `StopRetry`** — throw it from inside `fn` when an error is not worth retrying (bad credentials, a 400...):
+**`StopRetry`** — throw it from inside `fn` to stop retrying immediately:
 
 ```js
-const { withRetry, StopRetry } = require("@boy-offi9-inc/reqkit");
+import { withRetry, StopRetry } from "@boy-offi9-inc/reqkit";
 
 await withRetry(async () => {
   const res = await fetch(url);
@@ -278,7 +285,6 @@ Coalesces concurrent calls with the same arguments into a single in-flight call
 ```js
 const getUser = withDedupe((id) => fetch(`/api/users/${id}`).then(r => r.json()));
 
-// Only one network request goes out — both callers share it.
 const [a, b] = await Promise.all([getUser(1), getUser(1)]);
 ```
 
@@ -291,7 +297,6 @@ Plain arguments are compared by value (object key order doesn't matter); anythin
 This is deduplication of *concurrent* calls, not a cache — once a call settles (success or failure), the next call with the same key starts fresh.
 
 ```js
-// Custom key when the default arg-based key isn't quite right
 const fn = withDedupe((req) => doFetch(req), { keyFn: (req) => req.url });
 ```
 
@@ -362,6 +367,8 @@ for await (const row of readNDJSON(response)) console.log(row); // newline-delim
 for await (const line of readLines(response)) console.log(line);
 ```
 
+All three take `{ signal, maxLineLength }` — a line (or one SSE event's data) longer than `maxLineLength` characters (default 16 Mi) throws `code: "LINE_TOO_LONG"` rather than buffering forever.
+
 `readSSE` follows the spec: multi-line `data:` is joined with `\n`, comments are ignored, `event` defaults to `"message"`, `id` carries over, `retry` is parsed, and an event cut off by the end of the stream is dropped. `readNDJSON` skips blank lines and names the bad line on a parse error (or pass `skipInvalid: true`). Line splitting handles `\n`, `\r\n`, `\r`, a BOM, and multi-byte characters split across chunks.
 
 ### Pagination: `paginate`, `paginatePages`, `paginateLinks`
@@ -466,10 +473,10 @@ String matchers match the full URL, `pathname + search`, or the bare `pathname`;
 
 ### `createFetch(defaults)`
 
-Opt-in assembly of the pieces above into a fetch-like function. It's still plain `fetch` underneath: you get a standard `Response`, and you can pass your own `fetch`.
+Assembles the pieces above into one fetch-like function. It's still plain `fetch` underneath: you get a standard `Response` back, and you can supply your own `fetch`.
 
 ```js
-const { createFetch } = require("@boy-offi9-inc/reqkit");
+import { createFetch, HTTPError } from "@boy-offi9-inc/reqkit";
 
 const api = createFetch({
   baseUrl: "https://api.example.com/v1",
@@ -502,6 +509,7 @@ const admin = api.extend({ headers: { "x-role": "admin" } }); // merges over the
 | `retry` | see below | `false` disables; otherwise `{ retries, statusCodes, methods, baseDelayMs, maxDelayMs, factor, jitter, maxRetryTime, shouldRetry, calculateDelay, onRetry }` |
 | `rateLimit` | — | `true` just learns from `X-RateLimit-*` / `Retry-After`; `{ requests, perMs }` also enforces a budget; or pass a shared [`createRateLimiter`](#createratelimiteropts) |
 | `circuitBreaker` | — | `true` / options → one breaker **per origin**; or pass a shared [`createCircuitBreaker`](#createcircuitbreakeropts), or `(url) => breaker`. An open circuit stops retries immediately. |
+| `trustedOrigins` | — | extra origins that may receive the default credentials — see [Security](#security) |
 | `auth` | — | attach a token and refresh it on 401 — see [Auth token refresh](#auth-token-refresh) |
 | `cache` | — | in-memory GET cache — see [Caching](#caching). `true` = 60s TTL. Per request: `cache: false` bypasses, `"reload"` refreshes |
 | `dedupe` | `false` | merge identical concurrent `GET`/`HEAD` calls; every caller gets its own readable clone. Requests with their own `signal` are never merged. |
@@ -509,6 +517,7 @@ const admin = api.extend({ headers: { "x-role": "admin" } }); // merges over the
 | `fetch` | `globalThis.fetch` | swap in undici, node-fetch, a test double... |
 | `onRequest(ctx)` | — | once per call; may mutate `ctx.url` / `ctx.init` (e.g. add a header) |
 | `onResponse(ctx)` | — | once per call, with the final response, before `HTTPError` is thrown |
+| `onEvent(event)` | — | one callback for everything that happens — see [Events and logging](#events-and-logging) |
 
 #### Auth token refresh
 
@@ -532,6 +541,35 @@ await Promise.all(urls.map((u) => api.json(u))); // 20 requests, all 401 → one
 - **Options:** `header` (default `"authorization"`), `scheme` (default `"Bearer"`, `""` for a bare token such as an `X-API-Key`), `shouldRefresh(response)` (default: status 401). `refresh({ token, response })` may return the new token or just update your store. If `refresh` throws, every waiting request rejects with that error — and the next request tries again.
 - `extend()` children that share the same `auth` object share one refresh; the cache keys entries per token.
 
+#### Events and logging
+
+One callback receives everything that happens, so you can plug reqkit into your logger or metrics without wrapping anything.
+
+```js
+const api = createFetch({
+  baseUrl: "https://api.example.com",
+  onEvent: (e) => {
+    if (e.type === "response") metrics.timing("http.request", e.durationMs, { status: e.status, cache: e.cache });
+    if (e.type === "retry") log.warn(`retry #${e.attempt} in ${e.delayMs}ms (${e.reason})`);
+    if (e.type === "error") log.error(e.error, { url: e.url });
+  },
+});
+```
+
+| `type` | Emitted when | Fields |
+|---|---|---|
+| `request` | a call starts | `url`, `method` |
+| `attempt` | a network attempt starts | `url`, `method`, `attempt` (1-based) |
+| `retry` | an attempt failed and another will follow | `url`, `method`, `attempt`, `delayMs`, `reason` (`"error"` \| `"status"`), `status?`, `error?` |
+| `response` | the final response arrives (including non-2xx, before `HTTPError` is thrown) | `url`, `method`, `status`, `ok`, `durationMs`, `attempts`, `cache?` (`"hit"` \| `"stale"` \| `"revalidated"` \| `"miss"` \| `"stale-if-error"`) |
+| `error` | the call fails without a response — network error, timeout, abort, open circuit | `url?`, `method?`, `error`, `durationMs`, `attempts` |
+| `cache` | cache activity | `url`, `method`, `event` (`hit`, `stale`, `revalidated`, `miss`, `store`, `stale-if-error`) |
+| `ratelimit` | a call waited on the rate limiter | `url`, `method`, `waitedMs` |
+| `circuit` | a circuit breaker changed state (breakers created from options — one per origin) | `origin`, `state`, `previous` |
+| `auth` | a 401 triggered token handling | `url`, `method`, `outcome` (`"refreshed"` \| `"reused"` \| `"failed"`), `error?` |
+
+Handlers can be sync or async, and anything they throw is ignored — logging never breaks a request. `extend()` children inherit the handler or replace it. A cache hit makes no network attempt, so its `response` event has `attempts: 0`.
+
 #### Caching
 
 ```js
@@ -550,10 +588,13 @@ await api.json("/config"); // fresh → served from memory, no request
 |---|---|---|
 | `ttl` | `60000` | serve without asking the server for this long |
 | `staleWhileRevalidate` | `0` | after the TTL, keep serving the stale copy this long while refreshing in the background |
+| `staleIfError` | `0` | after the TTL, serve the stale copy for this long **when the origin is failing** (network error, timeout, open circuit, 5xx) instead of surfacing the error |
 | `maxEntries` | `100` | size of the built-in LRU store |
 | `maxBodyBytes` | `1 MiB` | bigger responses aren't cached |
 | `store` | built-in LRU | bring your own (`get`/`set`/`delete`, sync or async) — Redis, IndexedDB, a `Map`… |
-| `onEvent` | — | `({ type: "hit" \| "stale" \| "revalidated" \| "miss" \| "store", url })` for logging/metrics |
+| `onEvent` | — | `({ type: "hit" \| "stale" \| "revalidated" \| "miss" \| "store" \| "stale-if-error", url })` — cache-only; see also [`onEvent`](#events-and-logging) on `createFetch` |
+
+With `staleIfError: 10 * 60_000`, a cached copy keeps being served for up to 10 minutes past its TTL while the origin is down, instead of your callers seeing the failure. 4xx responses and deliberate aborts are never masked, and `cache: "reload"` always goes to the origin.
 
 Only `GET` `200` responses are cached; `Cache-Control: no-store` and `Vary: *` are respected. **Every request header is part of the cache key**, so a response fetched with one user's `Authorization` can never be served to another. Each hit is a fresh `Response`, so bodies can be read as many times as you like. `extend()` children share the parent's cache, limiter and breakers.
 
@@ -561,15 +602,14 @@ Only `GET` `200` responses are cached; `Cache-Control: no-store` and `Vary: *` a
 
 ---
 
-## Design notes
+## Prior art
+
+reqkit's building blocks are small independent implementations of ideas from [`p-retry`](https://github.com/sindresorhus/p-retry), [`ky`](https://github.com/sindresorhus/ky), [`ofetch`](https://github.com/unjs/ofetch), [`got`](https://github.com/sindresorhus/got), [`axios-retry`](https://github.com/softonic/axios-retry), [`p-limit`](https://github.com/sindresorhus/p-limit), [`opossum`](https://github.com/nodeshift/opossum), [`bottleneck`](https://github.com/SGrondin/bottleneck), [`p-throttle`](https://github.com/sindresorhus/p-throttle), [`eventsource-parser`](https://github.com/rexxars/eventsource-parser) and [`nock`](https://github.com/nock/nock), plus the HTTP caching RFCs and the AWS "full jitter" backoff.
 
-- **Zero dependencies.** Nothing to audit, nothing to break underneath you.
-- **CJS + ESM.** `require()` and `import` both work.
-- **Composable, not a client.** Every function takes and returns plain values (`Response`, plain objects, `Uint8Array`) — nothing proprietary to learn.
-- **TypeScript types included**, hand-written (no build step to go wrong), with separate entries for CJS and ESM.
+## Contributing
 
----
+Issues and pull requests are welcome — see [CONTRIBUTING.md](./CONTRIBUTING.md) and the [architecture overview](./docs/architecture.md).
 
-## Prior art
+## License
 
-reqkit's building blocks are small, from-scratch implementations of ideas that proved themselves in popular libraries — go look at them, they're great: [`p-retry`](https://github.com/sindresorhus/p-retry) (`StopRetry`, time budgets), [`ky`](https://github.com/sindresorhus/ky) and [`ofetch`](https://github.com/unjs/ofetch) (`HTTPError`, `json`/`query`/`baseUrl`, `extend`, retry statuses), [`got`](https://github.com/sindresorhus/got) and [`axios-retry`](https://github.com/softonic/axios-retry) (retry only idempotent methods, `calculateDelay`), [`p-limit`](https://github.com/sindresorhus/p-limit) (concurrency limiter), [`opossum`](https://github.com/nodeshift/opossum) (circuit breaker), [`bottleneck`](https://github.com/SGrondin/bottleneck) / [`p-throttle`](https://github.com/sindresorhus/p-throttle) (rate limiting), [`eventsource-parser`](https://github.com/rexxars/eventsource-parser) (SSE), [`nock`](https://github.com/nock/nock) / undici's `MockAgent` (mocking), the HTTP caching RFCs (`stale-while-revalidate`, conditional requests), and the AWS Architecture Blog's "full jitter" backoff.
+[MIT](./LICENSE) © Boy Offi9