@boy-offi9-inc/reqkit 0.6.0 → 0.7.0

package/README.md

@@ -56,6 +56,7 @@ const pages = await limit.map(urls, (url) =>
 Try everything in your browser — retries obeying `Retry-After`, rate limiting, the circuit breaker, caching, SSE streaming, an SSE stream that reconnects itself, and a resumable download that survives a dropped connection. It runs against a fake in-memory server, so there's nothing to install and no network involved.
 
 Run it locally: `npm run build && python3 -m http.server -d docs` → open `http://localhost:8000`. (ES modules don't load from `file://`, so it needs a server.)
+
 ## Works with what you already use
 
 Recipes for **axios, ky, got, undici/node-fetch**, testing, and LLM streaming live in [RECIPES.md](./RECIPES.md): keep your client, bolt on just the piece you're missing (`Retry-After`-aware retries, a rate limiter, a circuit breaker).
@@ -90,6 +91,8 @@ Recipes for **axios, ky, got, undici/node-fetch**, testing, and LLM streaming li
 | Slow down *before* the server says 429 (obey `X-RateLimit-*`) | [`createRateLimiter`](#createratelimiteropts) |
 | Stop hammering a service that keeps failing | [`createCircuitBreaker`](#createcircuitbreakeropts) |
 | Cache GETs (TTL, stale-while-revalidate, ETag/304) | [`createFetch` → `cache`](#caching) |
+| Attach a token and refresh it on 401 — once, even with many requests in flight | [`createFetch` → `auth`](#auth-token-refresh) |
+| Walk a paginated API (cursor, offset, `Link` headers) | [`paginate` / `paginateLinks`](#pagination-paginate-paginatepages-paginatelinks) |
 | Resume a download after the connection drops | [`downloadResumable`](#downloadresumableurl-opts) |
 | Stay connected to an SSE endpoint (auto-reconnect, `Last-Event-ID`) | [`streamSSE`](#auto-reconnecting-sse-streamsse) |
 | Read SSE / NDJSON / lines from a streamed body (LLM tokens!) | [`readSSE` / `readNDJSON` / `readLines`](#streaming-readsse-readndjson-readlines) |
@@ -361,6 +364,42 @@ for await (const line of readLines(response)) console.log(line);
 
 `readSSE` follows the spec: multi-line `data:` is joined with `\n`, comments are ignored, `event` defaults to `"message"`, `id` carries over, `retry` is parsed, and an event cut off by the end of the stream is dropped. `readNDJSON` skips blank lines and names the bad line on a parse error (or pass `skipInvalid: true`). Line splitting handles `\n`, `\r\n`, `\r`, a BOM, and multi-byte characters split across chunks.
 
+### Pagination: `paginate`, `paginatePages`, `paginateLinks`
+
+Walk any paginated API with a plain `for await` loop. Pages are fetched **lazily** — `break` out early (or set `maxItems`) and the pages you didn't need are never requested.
+
+```js
+import { paginate, paginateLinks } from "@boy-offi9-inc/reqkit";
+
+// Cursor-based: you say how to fetch a page and what the next cursor is
+for await (const user of paginate(async (cursor) => {
+  const r = await api.json("/users", { query: { cursor, limit: 100 } });
+  return { items: r.data, next: r.next_cursor };   // undefined/null on the last page
+}, { maxItems: 500 })) {
+  console.log(user.name);
+}
+
+// Offset-based is the same idea: the cursor is just a number
+for await (const row of paginate(async (offset = 0) => {
+  const rows = await api.json("/rows", { query: { offset, limit: 50 } });
+  return { items: rows, next: rows.length === 50 ? offset + 50 : undefined };
+})) { /* ... */ }
+
+// GitHub-style `Link: <...>; rel="next"` headers — nothing to write
+for await (const repo of paginateLinks(api, "/orgs/acme/repos?per_page=100")) {
+  console.log(repo.full_name);
+}
+```
+
+| API | What it does |
+|---|---|
+| `paginate(fetchPage, opts?)` | yields **items**; `opts`: `start`, `maxPages`, `maxItems`, `signal` |
+| `paginatePages(fetchPage, opts?)` | yields **pages** `{ items, index, cursor }` |
+| `paginateLinks(request, url, opts?)` | follows `Link` `rel="next"`; `request` is any fetch-like (a `createFetch` instance or plain `fetch` — a non-2xx page throws `HTTPError`); `getItems(data)` defaults to the body if it's an array, else `items` / `data` / `results` / `records` |
+| `parseLinkHeader(header, base?)` | RFC 8288 `Link` parser → `{ next, prev, first, last, ... }`; handles commas in URLs and quoted params |
+
+A `next` that equals the cursor just used (or a `Link` that points back at the same page) throws `code: "PAGINATE_STUCK"` instead of looping forever.
+
 ### Auto-reconnecting SSE: `streamSSE`
 
 `readSSE` parses a stream you already have; **`streamSSE` opens it for you and keeps it open.** When the connection drops (or the server closes it) it reconnects by itself and sends `Last-Event-ID`, so the server can resume where you left off — what the browser's `EventSource` does, but built on `fetch`: custom headers, POST bodies, `AbortSignal`, your own client.
@@ -463,6 +502,7 @@ const admin = api.extend({ headers: { "x-role": "admin" } }); // merges over the
 | `retry` | see below | `false` disables; otherwise `{ retries, statusCodes, methods, baseDelayMs, maxDelayMs, factor, jitter, maxRetryTime, shouldRetry, calculateDelay, onRetry }` |
 | `rateLimit` | — | `true` just learns from `X-RateLimit-*` / `Retry-After`; `{ requests, perMs }` also enforces a budget; or pass a shared [`createRateLimiter`](#createratelimiteropts) |
 | `circuitBreaker` | — | `true` / options → one breaker **per origin**; or pass a shared [`createCircuitBreaker`](#createcircuitbreakeropts), or `(url) => breaker`. An open circuit stops retries immediately. |
+| `auth` | — | attach a token and refresh it on 401 — see [Auth token refresh](#auth-token-refresh) |
 | `cache` | — | in-memory GET cache — see [Caching](#caching). `true` = 60s TTL. Per request: `cache: false` bypasses, `"reload"` refreshes |
 | `dedupe` | `false` | merge identical concurrent `GET`/`HEAD` calls; every caller gets its own readable clone. Requests with their own `signal` are never merged. |
 | `throwHttpErrors` | `true` | throw `HTTPError` for non-2xx (`err.status`, `err.response`, `err.url`, `err.method`) |
@@ -470,6 +510,28 @@ const admin = api.extend({ headers: { "x-role": "admin" } }); // merges over the
 | `onRequest(ctx)` | — | once per call; may mutate `ctx.url` / `ctx.init` (e.g. add a header) |
 | `onResponse(ctx)` | — | once per call, with the final response, before `HTTPError` is thrown |
 
+#### Auth token refresh
+
+```js
+let token = await loadToken();
+
+const api = createFetch({
+  baseUrl: "https://api.example.com",
+  auth: {
+    getToken: () => token,                                // attached as `Authorization: Bearer <token>`
+    refresh: async () => { token = await loginAgain(); }, // called on 401 — ONCE, however many requests are waiting
+  },
+});
+
+await Promise.all(urls.map((u) => api.json(u))); // 20 requests, all 401 → one refresh, all 20 replayed
+```
+
+- **Single-flight:** concurrent 401s share one `refresh()` call. A request rejected *after* someone else already refreshed notices the token changed and just replays with the new one — no second refresh.
+- **Replay once:** if the replay is rejected too, you get the `401` (as `HTTPError`) — it never loops. Request bodies (JSON, strings…) are re-sent; a stream body can't be replayed, so its 401 is surfaced as is.
+- **Your header wins:** if a request sets its own `Authorization`, reqkit leaves it alone and won't refresh it.
+- **Options:** `header` (default `"authorization"`), `scheme` (default `"Bearer"`, `""` for a bare token such as an `X-API-Key`), `shouldRefresh(response)` (default: status 401). `refresh({ token, response })` may return the new token or just update your store. If `refresh` throws, every waiting request rejects with that error — and the next request tries again.
+- `extend()` children that share the same `auth` object share one refresh; the cache keys entries per token.
+
 #### Caching
 
 ```js

package/dist/reqkit.mjs

@@ -1,4 +1,4 @@
-/*! @boy-offi9-inc/reqkit v0.6.0 — ESM bundle (generated by scripts/build-esm.js, do not edit) */
+/*! @boy-offi9-inc/reqkit v0.7.0 — ESM bundle (generated by scripts/build-esm.js, do not edit) */
 const __defs = {
 "./HTTPError": function (module, exports, require) {
 class HTTPError extends Error {
@@ -454,6 +454,18 @@ function originOf(url) {
   }
 }
 
+// One in-flight token refresh per `auth` config object, so concurrent 401s —
+// even across extend() children — trigger exactly one refresh.
+const refreshStates = new WeakMap();
+function refreshStateFor(auth) {
+  let state = refreshStates.get(auth);
+  if (!state) {
+    state = { inflight: null };
+    refreshStates.set(auth, state);
+  }
+  return state;
+}
+
 function mergeDefaults(base, extra) {
   const merged = { ...base, ...extra };
 
@@ -487,6 +499,8 @@ function mergeDefaults(base, extra) {
  *  - `cache`: in-memory GET cache with TTL, stale-while-revalidate and
  *    ETag / Last-Modified revalidation (304s)
  *  - concurrent GET de-duplication (withDedupe)
+ *  - `auth`: attach a token, and on a 401 refresh it ONCE (shared by all
+ *    concurrent requests) and replay the request
  *  - `HTTPError` on non-2xx (turn off with `throwHttpErrors: false`)
  *  - `onRequest` / `onResponse` hooks
  *
@@ -504,6 +518,11 @@ function mergeDefaults(base, extra) {
  *   a per-origin breaker, a createCircuitBreaker() instance, or `(url) => breaker`
  * @param {boolean | object} [defaults.cache] - `{ ttl, staleWhileRevalidate,
  *   maxEntries, maxBodyBytes, store, onEvent }`; `true` = 60s TTL
+ * @param {object} [defaults.auth] - `{ getToken, refresh, header?, scheme?, shouldRefresh? }`.
+ *   `getToken()` supplies the current token (sync or async); on a 401 (or
+ *   whatever `shouldRefresh(response)` says) `refresh({ token, response })` is
+ *   called once for all concurrent callers and the request is replayed with the
+ *   new token. `header` defaults to "authorization", `scheme` to "Bearer" ("" for none).
  * @param {boolean} [defaults.dedupe=false] - merge identical concurrent GET/HEAD calls
  * @param {boolean} [defaults.throwHttpErrors=true]
  * @param {typeof fetch} [defaults.fetch=globalThis.fetch]
@@ -520,6 +539,7 @@ function createFetch(defaults = {}) {
     rateLimit,
     circuitBreaker,
     cache,
+    auth,
     dedupe = false,
     throwHttpErrors: defaultThrow = true,
     fetch: customFetch,
@@ -590,6 +610,27 @@ function createFetch(defaults = {}) {
     pendingStores.set(key, p);
   }
 
+  // Returns a token newer than `usedToken`, refreshing at most once no matter
+  // how many requests were rejected at the same time. If someone else already
+  // refreshed (the current token isn't the one we used), just use that.
+  async function obtainFreshToken(authCfg, tokenUsed, response) {
+    const current = await authCfg.getToken();
+    if (current && current !== tokenUsed) return current;
+
+    const state = refreshStateFor(authCfg);
+    if (!state.inflight) {
+      state.inflight = (async () => {
+        try {
+          const out = await authCfg.refresh({ token: tokenUsed, response });
+          return typeof out === "string" && out ? out : await authCfg.getToken();
+        } finally {
+          state.inflight = null;
+        }
+      })();
+    }
+    return state.inflight;
+  }
+
   async function request(input, init = {}) {
     const {
       query,
@@ -617,6 +658,15 @@ function createFetch(defaults = {}) {
     const finalHeaders = ctx.init.headers instanceof Headers ? ctx.init.headers : new Headers(ctx.init.headers);
     const finalMethod = String(ctx.init.method || method).toUpperCase();
 
+    // --- auth: attach the current token (unless the caller set the header) ----
+    const authHeader = auth ? String(auth.header || "authorization").toLowerCase() : null;
+    const formatToken = (token) => (auth.scheme === "" ? String(token) : `${auth.scheme === undefined ? "Bearer" : auth.scheme} ${token}`);
+    let usedToken = null;
+    if (auth && !finalHeaders.has(authHeader)) {
+      usedToken = (await auth.getToken()) || null;
+      if (usedToken) finalHeaders.set(authHeader, formatToken(usedToken));
+    }
+
     // --- retry policy ------------------------------------------------------
     const retryCfg = retryOpt === false || defaultRetry === false ? false : { ...defaultRetry, ...retryOpt };
     const methods = (retryCfg && retryCfg.methods) || DEFAULT_RETRY_METHODS;
@@ -692,7 +742,7 @@ function createFetch(defaults = {}) {
       !ctx.init.body &&
       cacheMode !== false &&
       cacheMode !== "no-store";
-    const key = useCache ? cacheKey(finalMethod, ctx.url, finalHeaders) : null;
+    let key = useCache ? cacheKey(finalMethod, ctx.url, finalHeaders) : null;
     let entry = null;
 
     const finish = async (response) => {
@@ -753,6 +803,24 @@ function createFetch(defaults = {}) {
       throw err;
     }
 
+    // --- auth: a rejected token is refreshed once, then the request replays ----
+    const needsRefresh = auth && (auth.shouldRefresh ? auth.shouldRefresh(response) : response.status === 401);
+    if (usedToken && needsRefresh && !isStreamBody(ctx.init.body)) {
+      const fresh = await obtainFreshToken(auth, usedToken, response);
+      if (fresh) {
+        await discardBody(response);
+        finalHeaders.set(authHeader, formatToken(fresh));
+        entry = null; // belongs to the old token's cache key
+        if (useCache) key = cacheKey(finalMethod, ctx.url, finalHeaders);
+        try {
+          response = await network(finalHeaders, signal);
+        } catch (err) {
+          if (err instanceof RetryAbortedError && signal && signal.aborted) throw signal.reason ?? err;
+          throw err;
+        }
+      }
+    }
+
     if (useCache) {
       if (response.status === 304 && entry) {
         entry.storedAt = Date.now();
@@ -1414,6 +1482,7 @@ const { createMockFetch } = require("./createMockFetch");
 const { downloadResumable } = require("./downloadResumable");
 const { readLines, readNDJSON, readSSE } = require("./streams");
 const { streamSSE } = require("./streamSSE");
+const { paginate, paginatePages, paginateLinks, parseLinkHeader } = require("./paginate");
 
 module.exports = {
   withRetry,
@@ -1443,6 +1512,10 @@ module.exports = {
   readNDJSON,
   readSSE,
   streamSSE,
+  paginate,
+  paginatePages,
+  paginateLinks,
+  parseLinkHeader,
 };
 
 },
@@ -1572,6 +1645,175 @@ function normalizeError(err) {
 
 module.exports = { normalizeError };
 
+},
+"./paginate": function (module, exports, require) {
+const { HTTPError } = require("./HTTPError");
+
+/**
+ * Parses an RFC 8288 `Link` header (GitHub, GitLab, WordPress... use it for
+ * pagination) into `{ rel: url }`. Pass `base` to resolve relative URLs.
+ *
+ * ```js
+ * parseLinkHeader('<https://api.x/items?page=2>; rel="next", <https://api.x/items?page=9>; rel="last"');
+ * // { next: "https://api.x/items?page=2", last: "https://api.x/items?page=9" }
+ * ```
+ *
+ * A link with several rels (`rel="next last"`) is listed under each; the first
+ * link wins when a rel appears twice. Commas inside `<...>` are handled.
+ *
+ * @param {string | null | undefined} header
+ * @param {string} [base]
+ * @returns {Record<string, string>}
+ */
+function parseLinkHeader(header, base) {
+  const links = {};
+  if (!header) return links;
+
+  // <url> followed by any number of `; name` / `; name=token` / `; name="quoted, value"` params
+  const linkRe = /<([^>]*)>((?:\s*;\s*[^=;,<\s]+(?:\s*=\s*(?:"[^"]*"|[^;,<\s]*))?)*)/g;
+  let match;
+  while ((match = linkRe.exec(header)) !== null) {
+    const rel = /;\s*rel\s*=\s*(?:"([^"]*)"|([^\s;,]+))/i.exec(match[2]);
+    if (!rel) continue;
+
+    let url = match[1].trim();
+    if (base) {
+      try {
+        url = new URL(url, base).href;
+      } catch {
+        // leave it as written
+      }
+    }
+    for (const name of (rel[1] ?? rel[2]).toLowerCase().split(/\s+/).filter(Boolean)) {
+      if (!(name in links)) links[name] = url;
+    }
+  }
+  return links;
+}
+
+function stuckError(message) {
+  return Object.assign(new Error(message), { code: "PAGINATE_STUCK" });
+}
+
+/**
+ * Walks a paginated API page by page. You write `fetchPage(cursor, ctx)` —
+ * it returns `{ items, next }`, where `next` is whatever identifies the next
+ * page (a cursor token, an offset, a URL...) or `undefined`/`null` on the
+ * last page — and you get an async iterator of pages.
+ *
+ * ```js
+ * for await (const { items, index } of paginatePages(async (cursor) => {
+ *   const r = await api.json("/users", { query: { cursor, limit: 100 } });
+ *   return { items: r.data, next: r.next_cursor };
+ * })) { ... }
+ * ```
+ *
+ * A `next` that equals the cursor just used throws `code: "PAGINATE_STUCK"`
+ * instead of looping forever.
+ *
+ * @template T
+ * @param {(cursor: any, ctx: { index: number, signal?: AbortSignal }) => Promise<{ items: T[], next?: any }>} fetchPage
+ * @param {object} [opts]
+ * @param {any} [opts.start] - the cursor for the first page (default undefined)
+ * @param {number} [opts.maxPages=Infinity]
+ * @param {AbortSignal} [opts.signal]
+ * @returns {AsyncGenerator<{ items: T[], index: number, cursor: any }>}
+ */
+async function* paginatePages(fetchPage, opts = {}) {
+  const { start, maxPages = Infinity, signal } = opts;
+  let cursor = start;
+  let index = 0;
+
+  for (;;) {
+    if (index >= maxPages) return;
+    signal?.throwIfAborted?.();
+
+    const result = await fetchPage(cursor, { index, signal });
+    if (!result || !Array.isArray(result.items)) {
+      throw new TypeError("paginate: fetchPage must return { items: array, next?: cursor }");
+    }
+    yield { items: result.items, index, cursor };
+
+    const next = result.next;
+    if (next === undefined || next === null) return;
+    if (next === cursor) {
+      throw stuckError("paginate: `next` equals the cursor just used, so the page didn't advance — stopping to avoid an infinite loop");
+    }
+    cursor = next;
+    index++;
+  }
+}
+
+/**
+ * Like `paginatePages`, but yields the items one by one. `maxItems` stops
+ * early (mid-page) — and since nothing is fetched until you ask for it,
+ * `break`-ing out of the loop never fetches a page you didn't need.
+ *
+ * ```js
+ * for await (const user of paginate(fetchUsersPage, { maxItems: 250 })) { ... }
+ * ```
+ *
+ * @template T
+ * @param {(cursor: any, ctx: { index: number, signal?: AbortSignal }) => Promise<{ items: T[], next?: any }>} fetchPage
+ * @param {object} [opts] - same as paginatePages, plus `maxItems`
+ * @param {number} [opts.maxItems=Infinity]
+ * @returns {AsyncGenerator<T>}
+ */
+async function* paginate(fetchPage, opts = {}) {
+  const { maxItems = Infinity } = opts;
+  if (maxItems <= 0) return;
+
+  let count = 0;
+  for await (const page of paginatePages(fetchPage, opts)) {
+    for (const item of page.items) {
+      yield item;
+      if (++count >= maxItems) return;
+    }
+  }
+}
+
+function defaultGetItems(data) {
+  if (Array.isArray(data)) return data;
+  for (const key of ["items", "data", "results", "records"]) {
+    if (data && Array.isArray(data[key])) return data[key];
+  }
+  throw new TypeError("paginateLinks: couldn't find the items array in the response — pass `getItems(data)`");
+}
+
+/**
+ * Follows `Link: <...>; rel="next"` headers (GitHub-style) and yields the
+ * items of every page. `request` is anything fetch-like — a `createFetch`
+ * instance or plain `fetch` (a non-2xx response throws `HTTPError`).
+ *
+ * ```js
+ * for await (const repo of paginateLinks(api, "/orgs/acme/repos?per_page=100")) { ... }
+ * ```
+ *
+ * @param {(url: string, init?: object) => Promise<Response>} request
+ * @param {string} url - the first page
+ * @param {object} [opts]
+ * @param {(data: any, response: Response) => any[]} [opts.getItems] - default: the body if it's an array, else its `items` / `data` / `results` / `records`
+ * @param {number} [opts.maxPages=Infinity]
+ * @param {number} [opts.maxItems=Infinity]
+ * @param {AbortSignal} [opts.signal]
+ * @param {object} [opts.init] - extra options passed to every `request`
+ */
+async function* paginateLinks(request, url, opts = {}) {
+  const { getItems = defaultGetItems, init = {}, signal } = opts;
+
+  async function fetchPage(pageUrl) {
+    const res = await request(pageUrl, { ...init, signal });
+    if (!res.ok) throw new HTTPError(res, { method: init.method || "GET", url: pageUrl });
+    const items = getItems(await res.json(), res);
+    const next = parseLinkHeader(res.headers.get("link"), pageUrl).next;
+    return { items, next };
+  }
+
+  yield* paginate(fetchPage, { ...opts, start: url });
+}
+
+module.exports = { paginate, paginatePages, paginateLinks, parseLinkHeader };
+
 },
 "./parseJsonSafe": function (module, exports, require) {
 /**
@@ -2492,7 +2734,11 @@ export const {
   downloadResumable,
   getWaitMsFromHeaders,
   normalizeError,
+  paginate,
+  paginateLinks,
+  paginatePages,
   parseJsonSafe,
+  parseLinkHeader,
   parseRetryAfter,
   readLines,
   readNDJSON,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@boy-offi9-inc/reqkit",
-  "version": "0.6.0",
+  "version": "0.7.0",
   "description": "Small, composable HTTP helpers — retry with backoff and Retry-After, timeouts, normalized errors, rate limiter, circuit breaker, response cache, resumable downloads, SSE/NDJSON streams, mock fetch, and an opt-in createFetch. Not a client; works alongside fetch/axios/anything.",
   "main": "./src/index.js",
   "types": "./types/index.d.ts",
@@ -57,7 +57,11 @@
     "deno",
     "bun",
     "cloudflare-workers",
-    "esm"
+    "esm",
+    "pagination",
+    "auth",
+    "token-refresh",
+    "link-header"
   ],
   "author": "Boy Offi9",
   "license": "MIT",

package/src/createFetch.js

@@ -54,6 +54,18 @@ function originOf(url) {
   }
 }
 
+// One in-flight token refresh per `auth` config object, so concurrent 401s —
+// even across extend() children — trigger exactly one refresh.
+const refreshStates = new WeakMap();
+function refreshStateFor(auth) {
+  let state = refreshStates.get(auth);
+  if (!state) {
+    state = { inflight: null };
+    refreshStates.set(auth, state);
+  }
+  return state;
+}
+
 function mergeDefaults(base, extra) {
   const merged = { ...base, ...extra };
 
@@ -87,6 +99,8 @@ function mergeDefaults(base, extra) {
  *  - `cache`: in-memory GET cache with TTL, stale-while-revalidate and
  *    ETag / Last-Modified revalidation (304s)
  *  - concurrent GET de-duplication (withDedupe)
+ *  - `auth`: attach a token, and on a 401 refresh it ONCE (shared by all
+ *    concurrent requests) and replay the request
  *  - `HTTPError` on non-2xx (turn off with `throwHttpErrors: false`)
  *  - `onRequest` / `onResponse` hooks
  *
@@ -104,6 +118,11 @@ function mergeDefaults(base, extra) {
  *   a per-origin breaker, a createCircuitBreaker() instance, or `(url) => breaker`
  * @param {boolean | object} [defaults.cache] - `{ ttl, staleWhileRevalidate,
  *   maxEntries, maxBodyBytes, store, onEvent }`; `true` = 60s TTL
+ * @param {object} [defaults.auth] - `{ getToken, refresh, header?, scheme?, shouldRefresh? }`.
+ *   `getToken()` supplies the current token (sync or async); on a 401 (or
+ *   whatever `shouldRefresh(response)` says) `refresh({ token, response })` is
+ *   called once for all concurrent callers and the request is replayed with the
+ *   new token. `header` defaults to "authorization", `scheme` to "Bearer" ("" for none).
  * @param {boolean} [defaults.dedupe=false] - merge identical concurrent GET/HEAD calls
  * @param {boolean} [defaults.throwHttpErrors=true]
  * @param {typeof fetch} [defaults.fetch=globalThis.fetch]
@@ -120,6 +139,7 @@ function createFetch(defaults = {}) {
     rateLimit,
     circuitBreaker,
     cache,
+    auth,
     dedupe = false,
     throwHttpErrors: defaultThrow = true,
     fetch: customFetch,
@@ -190,6 +210,27 @@ function createFetch(defaults = {}) {
     pendingStores.set(key, p);
   }
 
+  // Returns a token newer than `usedToken`, refreshing at most once no matter
+  // how many requests were rejected at the same time. If someone else already
+  // refreshed (the current token isn't the one we used), just use that.
+  async function obtainFreshToken(authCfg, tokenUsed, response) {
+    const current = await authCfg.getToken();
+    if (current && current !== tokenUsed) return current;
+
+    const state = refreshStateFor(authCfg);
+    if (!state.inflight) {
+      state.inflight = (async () => {
+        try {
+          const out = await authCfg.refresh({ token: tokenUsed, response });
+          return typeof out === "string" && out ? out : await authCfg.getToken();
+        } finally {
+          state.inflight = null;
+        }
+      })();
+    }
+    return state.inflight;
+  }
+
   async function request(input, init = {}) {
     const {
       query,
@@ -217,6 +258,15 @@ function createFetch(defaults = {}) {
     const finalHeaders = ctx.init.headers instanceof Headers ? ctx.init.headers : new Headers(ctx.init.headers);
     const finalMethod = String(ctx.init.method || method).toUpperCase();
 
+    // --- auth: attach the current token (unless the caller set the header) ----
+    const authHeader = auth ? String(auth.header || "authorization").toLowerCase() : null;
+    const formatToken = (token) => (auth.scheme === "" ? String(token) : `${auth.scheme === undefined ? "Bearer" : auth.scheme} ${token}`);
+    let usedToken = null;
+    if (auth && !finalHeaders.has(authHeader)) {
+      usedToken = (await auth.getToken()) || null;
+      if (usedToken) finalHeaders.set(authHeader, formatToken(usedToken));
+    }
+
     // --- retry policy ------------------------------------------------------
     const retryCfg = retryOpt === false || defaultRetry === false ? false : { ...defaultRetry, ...retryOpt };
     const methods = (retryCfg && retryCfg.methods) || DEFAULT_RETRY_METHODS;
@@ -292,7 +342,7 @@ function createFetch(defaults = {}) {
       !ctx.init.body &&
       cacheMode !== false &&
       cacheMode !== "no-store";
-    const key = useCache ? cacheKey(finalMethod, ctx.url, finalHeaders) : null;
+    let key = useCache ? cacheKey(finalMethod, ctx.url, finalHeaders) : null;
     let entry = null;
 
     const finish = async (response) => {
@@ -353,6 +403,24 @@ function createFetch(defaults = {}) {
       throw err;
     }
 
+    // --- auth: a rejected token is refreshed once, then the request replays ----
+    const needsRefresh = auth && (auth.shouldRefresh ? auth.shouldRefresh(response) : response.status === 401);
+    if (usedToken && needsRefresh && !isStreamBody(ctx.init.body)) {
+      const fresh = await obtainFreshToken(auth, usedToken, response);
+      if (fresh) {
+        await discardBody(response);
+        finalHeaders.set(authHeader, formatToken(fresh));
+        entry = null; // belongs to the old token's cache key
+        if (useCache) key = cacheKey(finalMethod, ctx.url, finalHeaders);
+        try {
+          response = await network(finalHeaders, signal);
+        } catch (err) {
+          if (err instanceof RetryAbortedError && signal && signal.aborted) throw signal.reason ?? err;
+          throw err;
+        }
+      }
+    }
+
     if (useCache) {
       if (response.status === 304 && entry) {
         entry.storedAt = Date.now();

package/src/index.js

@@ -15,6 +15,7 @@ const { createMockFetch } = require("./createMockFetch");
 const { downloadResumable } = require("./downloadResumable");
 const { readLines, readNDJSON, readSSE } = require("./streams");
 const { streamSSE } = require("./streamSSE");
+const { paginate, paginatePages, paginateLinks, parseLinkHeader } = require("./paginate");
 
 module.exports = {
   withRetry,
@@ -44,4 +45,8 @@ module.exports = {
   readNDJSON,
   readSSE,
   streamSSE,
+  paginate,
+  paginatePages,
+  paginateLinks,
+  parseLinkHeader,
 };

package/src/index.mjs

@@ -28,6 +28,10 @@ export const {
   readNDJSON,
   readSSE,
   streamSSE,
+  paginate,
+  paginatePages,
+  paginateLinks,
+  parseLinkHeader,
 } = reqkit;
 
 export default reqkit;

package/src/paginate.js

@@ -0,0 +1,166 @@
+const { HTTPError } = require("./HTTPError");
+
+/**
+ * Parses an RFC 8288 `Link` header (GitHub, GitLab, WordPress... use it for
+ * pagination) into `{ rel: url }`. Pass `base` to resolve relative URLs.
+ *
+ * ```js
+ * parseLinkHeader('<https://api.x/items?page=2>; rel="next", <https://api.x/items?page=9>; rel="last"');
+ * // { next: "https://api.x/items?page=2", last: "https://api.x/items?page=9" }
+ * ```
+ *
+ * A link with several rels (`rel="next last"`) is listed under each; the first
+ * link wins when a rel appears twice. Commas inside `<...>` are handled.
+ *
+ * @param {string | null | undefined} header
+ * @param {string} [base]
+ * @returns {Record<string, string>}
+ */
+function parseLinkHeader(header, base) {
+  const links = {};
+  if (!header) return links;
+
+  // <url> followed by any number of `; name` / `; name=token` / `; name="quoted, value"` params
+  const linkRe = /<([^>]*)>((?:\s*;\s*[^=;,<\s]+(?:\s*=\s*(?:"[^"]*"|[^;,<\s]*))?)*)/g;
+  let match;
+  while ((match = linkRe.exec(header)) !== null) {
+    const rel = /;\s*rel\s*=\s*(?:"([^"]*)"|([^\s;,]+))/i.exec(match[2]);
+    if (!rel) continue;
+
+    let url = match[1].trim();
+    if (base) {
+      try {
+        url = new URL(url, base).href;
+      } catch {
+        // leave it as written
+      }
+    }
+    for (const name of (rel[1] ?? rel[2]).toLowerCase().split(/\s+/).filter(Boolean)) {
+      if (!(name in links)) links[name] = url;
+    }
+  }
+  return links;
+}
+
+function stuckError(message) {
+  return Object.assign(new Error(message), { code: "PAGINATE_STUCK" });
+}
+
+/**
+ * Walks a paginated API page by page. You write `fetchPage(cursor, ctx)` —
+ * it returns `{ items, next }`, where `next` is whatever identifies the next
+ * page (a cursor token, an offset, a URL...) or `undefined`/`null` on the
+ * last page — and you get an async iterator of pages.
+ *
+ * ```js
+ * for await (const { items, index } of paginatePages(async (cursor) => {
+ *   const r = await api.json("/users", { query: { cursor, limit: 100 } });
+ *   return { items: r.data, next: r.next_cursor };
+ * })) { ... }
+ * ```
+ *
+ * A `next` that equals the cursor just used throws `code: "PAGINATE_STUCK"`
+ * instead of looping forever.
+ *
+ * @template T
+ * @param {(cursor: any, ctx: { index: number, signal?: AbortSignal }) => Promise<{ items: T[], next?: any }>} fetchPage
+ * @param {object} [opts]
+ * @param {any} [opts.start] - the cursor for the first page (default undefined)
+ * @param {number} [opts.maxPages=Infinity]
+ * @param {AbortSignal} [opts.signal]
+ * @returns {AsyncGenerator<{ items: T[], index: number, cursor: any }>}
+ */
+async function* paginatePages(fetchPage, opts = {}) {
+  const { start, maxPages = Infinity, signal } = opts;
+  let cursor = start;
+  let index = 0;
+
+  for (;;) {
+    if (index >= maxPages) return;
+    signal?.throwIfAborted?.();
+
+    const result = await fetchPage(cursor, { index, signal });
+    if (!result || !Array.isArray(result.items)) {
+      throw new TypeError("paginate: fetchPage must return { items: array, next?: cursor }");
+    }
+    yield { items: result.items, index, cursor };
+
+    const next = result.next;
+    if (next === undefined || next === null) return;
+    if (next === cursor) {
+      throw stuckError("paginate: `next` equals the cursor just used, so the page didn't advance — stopping to avoid an infinite loop");
+    }
+    cursor = next;
+    index++;
+  }
+}
+
+/**
+ * Like `paginatePages`, but yields the items one by one. `maxItems` stops
+ * early (mid-page) — and since nothing is fetched until you ask for it,
+ * `break`-ing out of the loop never fetches a page you didn't need.
+ *
+ * ```js
+ * for await (const user of paginate(fetchUsersPage, { maxItems: 250 })) { ... }
+ * ```
+ *
+ * @template T
+ * @param {(cursor: any, ctx: { index: number, signal?: AbortSignal }) => Promise<{ items: T[], next?: any }>} fetchPage
+ * @param {object} [opts] - same as paginatePages, plus `maxItems`
+ * @param {number} [opts.maxItems=Infinity]
+ * @returns {AsyncGenerator<T>}
+ */
+async function* paginate(fetchPage, opts = {}) {
+  const { maxItems = Infinity } = opts;
+  if (maxItems <= 0) return;
+
+  let count = 0;
+  for await (const page of paginatePages(fetchPage, opts)) {
+    for (const item of page.items) {
+      yield item;
+      if (++count >= maxItems) return;
+    }
+  }
+}
+
+function defaultGetItems(data) {
+  if (Array.isArray(data)) return data;
+  for (const key of ["items", "data", "results", "records"]) {
+    if (data && Array.isArray(data[key])) return data[key];
+  }
+  throw new TypeError("paginateLinks: couldn't find the items array in the response — pass `getItems(data)`");
+}
+
+/**
+ * Follows `Link: <...>; rel="next"` headers (GitHub-style) and yields the
+ * items of every page. `request` is anything fetch-like — a `createFetch`
+ * instance or plain `fetch` (a non-2xx response throws `HTTPError`).
+ *
+ * ```js
+ * for await (const repo of paginateLinks(api, "/orgs/acme/repos?per_page=100")) { ... }
+ * ```
+ *
+ * @param {(url: string, init?: object) => Promise<Response>} request
+ * @param {string} url - the first page
+ * @param {object} [opts]
+ * @param {(data: any, response: Response) => any[]} [opts.getItems] - default: the body if it's an array, else its `items` / `data` / `results` / `records`
+ * @param {number} [opts.maxPages=Infinity]
+ * @param {number} [opts.maxItems=Infinity]
+ * @param {AbortSignal} [opts.signal]
+ * @param {object} [opts.init] - extra options passed to every `request`
+ */
+async function* paginateLinks(request, url, opts = {}) {
+  const { getItems = defaultGetItems, init = {}, signal } = opts;
+
+  async function fetchPage(pageUrl) {
+    const res = await request(pageUrl, { ...init, signal });
+    if (!res.ok) throw new HTTPError(res, { method: init.method || "GET", url: pageUrl });
+    const items = getItems(await res.json(), res);
+    const next = parseLinkHeader(res.headers.get("link"), pageUrl).next;
+    return { items, next };
+  }
+
+  yield* paginate(fetchPage, { ...opts, start: url });
+}
+
+module.exports = { paginate, paginatePages, paginateLinks, parseLinkHeader };

package/types/index.d.ts

@@ -210,6 +210,8 @@ export interface CreateFetchDefaults {
   rateLimit?: boolean | RateLimiterOptions | RateLimiter;
   /** Options for a per-origin breaker, a shared breaker instance, or `(url) => breaker`. */
   circuitBreaker?: boolean | CircuitBreakerOptions | CircuitBreaker | ((url: string) => CircuitBreaker);
+  /** Attach a token and, on a 401, refresh it once (shared by all concurrent requests) and replay. */
+  auth?: AuthOptions;
   /** In-memory GET cache with TTL, stale-while-revalidate and ETag/Last-Modified revalidation. `true` = 60s TTL. */
   cache?: boolean | CacheOptions;
   /** Merge identical concurrent GET/HEAD calls. Default false. */
@@ -441,3 +443,76 @@ export interface StreamSSEOptions {
 
 /** Auto-reconnecting SSE: resumes with Last-Event-ID, honors `retry:` and Retry-After. */
 export function streamSSE(url: string, opts?: StreamSSEOptions): AsyncGenerator<SSEEvent, void, undefined>;
+
+export interface AuthOptions {
+  /** The current token (sync or async). Falsy = send no auth header. */
+  getToken(): string | null | undefined | Promise<string | null | undefined>;
+  /**
+   * Called ONCE when requests are rejected (401 by default), no matter how many
+   * are waiting. Return the new token, or update your store and return nothing
+   * (reqkit then calls getToken() again). Throwing rejects every waiting request.
+   */
+  refresh(ctx: { token: string; response: Response }): string | void | undefined | Promise<string | void | undefined>;
+  /** Default "authorization". */
+  header?: string;
+  /** Default "Bearer". Use "" to send the bare token (e.g. for an X-API-Key header). */
+  scheme?: string;
+  /** Default: `res.status === 401`. */
+  shouldRefresh?: (response: Response) => boolean;
+}
+
+export interface PageContext {
+  index: number;
+  signal?: AbortSignal;
+}
+
+export interface PageResult<T, C = any> {
+  items: T[];
+  /** Whatever identifies the next page (token, offset, URL...). undefined/null = last page. */
+  next?: C | null;
+}
+
+export interface Page<T, C = any> {
+  items: T[];
+  index: number;
+  /** The cursor this page was fetched with (undefined for the first, unless `start` was given). */
+  cursor: C | undefined;
+}
+
+export interface PaginateOptions<C = any> {
+  /** The cursor for the first page. */
+  start?: C;
+  maxPages?: number;
+  maxItems?: number;
+  signal?: AbortSignal;
+}
+
+export type FetchPage<T, C = any> = (cursor: C | undefined, ctx: PageContext) => Promise<PageResult<T, C>>;
+
+/** Yields every item, fetching pages lazily. */
+export function paginate<T, C = any>(fetchPage: FetchPage<T, C>, opts?: PaginateOptions<C>): AsyncGenerator<T, void, undefined>;
+/** Yields whole pages. */
+export function paginatePages<T, C = any>(
+  fetchPage: FetchPage<T, C>,
+  opts?: Omit<PaginateOptions<C>, "maxItems">
+): AsyncGenerator<Page<T, C>, void, undefined>;
+
+export interface PaginateLinksOptions<T = any> {
+  /** Default: the body if it's an array, else its `items` / `data` / `results` / `records`. */
+  getItems?: (data: any, response: Response) => T[];
+  maxPages?: number;
+  maxItems?: number;
+  signal?: AbortSignal;
+  /** Extra options passed to every request. */
+  init?: Record<string, any>;
+}
+
+/** Follows `Link: <...>; rel="next"` headers and yields the items of every page. */
+export function paginateLinks<T = any>(
+  request: (url: string, init?: any) => Promise<Response>,
+  url: string,
+  opts?: PaginateLinksOptions<T>
+): AsyncGenerator<T, void, undefined>;
+
+/** Parses an RFC 8288 Link header into `{ rel: url }`. Pass `base` to resolve relative URLs. */
+export function parseLinkHeader(header: string | null | undefined, base?: string): Record<string, string>;