varfetch 0.1.0 → 0.3.0

package/CHANGELOG.md

@@ -0,0 +1,23 @@
+# varfetch
+
+## 0.3.0
+
+### Security
+- **DNS pinning.** `fireRequest` now connects only to the addresses the SSRF guard validated (new `pinnedFetch`, built on `node:http(s)` with a pinned `lookup`). A resolver that answers differently on the second lookup (DNS rebinding) can no longer redirect the connection to an internal address. Host header, SNI and certificate verification still use the original host name. A custom `fetch` option replaces the pinned transport and is only checked, not pinned.
+- **Allow rules apply per address.** Before, one matching address was enough to allow a host, so a name resolving to `10.1.1.1` and `169.254.169.254` passed an `allow: ['10.0.0.0/8']` rule. Now every resolved address must be public or covered by an allow rule. Hostname patterns still cover all addresses of that host.
+- Deny rules match any resolved address and ignore a trailing dot in the host name (`x.example.` no longer slips past `*.example`).
+- More blocked ranges: documentation ranges, 6to4 relay, and the IPv6 forms that embed IPv4 (NAT64 `64:ff9b::/96`, 6to4 `2002::/16`, Teredo `2001::/32`), discard `100::/64`, documentation `2001:db8::/32`.
+- URLs with embedded credentials (`http://user:pw@host`) are refused.
+- The DNS lookup is bound to the request timeout; a hung resolver can no longer stall a request.
+- New `encodePathVariables` option encodes variable values inserted into the path.
+
+### Changed
+- `checkUrlAllowed` also returns the validated `addresses` when allowed.
+- The default transport no longer uses the global `fetch`; it sends `Accept-Encoding: identity` and decodes gzip, deflate and br itself (size limits count decompressed bytes). Environment proxies are not used (they were not by Node's fetch either).
+- Redirect response bodies are cancelled instead of left open.
+
+## 0.2.0
+- `responseType: "binary"` / `"image"` returns raw bytes and the content type.
+
+## 0.1.0
+- First release, extracted from the LCYT connectors plugin.

package/README.md

@@ -1,47 +1,102 @@
 # varfetch
 
-Fire a configured HTTP request, with `{{variable}}` values in its path, query, headers and body, and map the JSON response onto named variables. No dependencies, ESM, Node 18+.
+Fire a configured HTTP request with `{{variable}}` values in its path, query, headers and body, and map the JSON response onto named variables. Zero dependencies, ESM, Node 18+.
 
-It is the small, database-free core of a "connector" feature: the application stores connectors and requests however it likes (a database, a config file) and passes them to `fireRequest`.
+varfetch is the small, database-free core of a "connector" feature. Your application stores connectors and requests however it likes (a database, a config file, a UI) and hands them to `fireRequest`. varfetch does the part that is easy to get wrong: building the request, refusing to be turned into an SSRF proxy, and extracting values from the answer.
 
 ```js
 import { fireRequest } from 'varfetch';
 
-const connector = {
-  baseUrl: 'https://api.example.org',
-  auth: { type: 'bearer', token: '...' },          // none | bearer | api_key | basic | custom
-  headers: [{ key: 'Accept-Language', value: 'fi' }],
-};
-
-const request = {
-  method: 'GET',
-  path: '/api/v1/date/{{date}}',
-  query: [{ key: 'cycles', value: 'false' }],
-  mappings: [
-    { jsonPath: '$.holyDay.name', variable: 'holyday' },
-    { jsonPath: '$.holyDay.texts.gospel.reference', variable: 'gospel' },
-  ],
-};
-
-const result = await fireRequest({ connector, request, variables: { date: '2026-10-11' } });
-// { ok: true, status: 200, values: { holyday: '...', gospel: '...' }, body: {...} }
+const result = await fireRequest({
+  connector: {
+    baseUrl: 'https://api.example.org',
+    auth: { type: 'bearer', token: process.env.API_TOKEN },
+    headers: [{ key: 'Accept-Language', value: 'fi' }],
+  },
+  request: {
+    method: 'GET',
+    path: '/api/v1/date/{{date}}',
+    query: [{ key: 'cycles', value: 'false' }],
+    mappings: [
+      { jsonPath: '$.holyDay.name', variable: 'holyday' },
+      { jsonPath: '$.holyDay.texts.gospel.reference', variable: 'gospel' },
+    ],
+  },
+  variables: { date: '2026-10-11' },
+});
+
+if (result.ok) console.log(result.values); // { holyday: '...', gospel: '...' }
+else console.error(result.error);
 ```
 
-## What it does
+`fireRequest` never throws: failures come back as `{ ok: false, error }`.
+
+## Features
+
+| | |
+|---|---|
+| **Interpolation** | `{{name}}` in path, query values, header values, auth values and body. Names may contain letters (incl. ä/ö/å), digits, `_` and `-`. Missing variables render as an empty string. |
+| **Auth** | `none`, `bearer`, `api_key` (any header), `basic`, `custom` headers. Values are interpolated too. |
+| **JSON mapping** | `$`, `$.a.b`, `$.items[0].name`, `$['key']`. Own properties only (`$.constructor` resolves to nothing). Objects and arrays are stored as JSON text. |
+| **Response types** | `auto` (JSON when the content type says so), `json`, `text`, `binary` / `image` (raw `Buffer` plus content type). |
+| **SSRF guard** | Blocks loopback, private, link-local (cloud metadata), CGNAT, reserved, multicast and IPv6 forms that embed IPv4. Allow and deny rules by host, wildcard, IP or CIDR. Applied to every redirect hop. |
+| **DNS pinning** | The connection goes only to the addresses the guard validated, so DNS rebinding between check and connect is impossible. See [docs/security.md](docs/security.md). |
+| **Limits** | 10 s timeout, 5 MiB response (measured after decompression), at most 5 redirects. Credentials and connector headers are never sent to another origin after a redirect. |
+
+## Install
+
+```sh
+npm install varfetch
+```
 
-- **Interpolation:** `{{name}}` in path, query values, header values, auth values and body. Missing variables become an empty string.
-- **JSON path:** `$`, `$.a.b`, `$.items[0].name`, `$['key']`. Own properties only.
-- **Mapping:** each mapping writes a string (or JSON text for objects and arrays) to a variable; `skipIfNull` (default true) leaves unresolved paths out.
-- **SSRF guard:** every URL, redirects included, is resolved and checked. Loopback, private, link-local, CGNAT, reserved and multicast addresses are blocked unless you allow them: `network: { allow: ['10.1.0.0/16', 'anno.internal:3000'], deny: ['*.blocked.example'] }`. `deny` wins over `allow`. The check happens before the connection and does not defend against DNS rebinding.
-- **Limits:** 10 s timeout and 5 MiB response by default (`timeoutMs`, `maxBytes`), at most 5 redirects, credentials and other connector headers are not forwarded to another origin.
+Requires Node 18 or newer. There are no runtime dependencies. TypeScript types ship in the package.
+
+## Reaching private networks
+
+By default every private and internal address is refused. To let a server call an API on its own network, allow that target explicitly:
+
+```js
+await fireRequest({
+  connector, request, variables,
+  network: {
+    allow: ['anno.internal:3000', '10.1.0.0/16'],
+    deny: ['*.blocked.example'],   // deny always wins
+  },
+});
+```
 
-Variable values are put into the path as given. A value containing `/`, `?` or `#` changes the URL, so run untrusted values through `encodeURIComponent`, or use them in a query parameter, which is encoded for you.
+Keep these rules in server configuration (an environment variable, for example), not in anything a user of your application can edit. See the [integration guide](docs/integration-guide.md).
+
+## Path variables
+
+Variable values are put into the path as given. A value containing `/`, `?` or `#` changes the URL. If any value comes from a user, either pass `encodePathVariables: true` (each value goes through `encodeURIComponent`) or put it in a query parameter, which is always encoded.
+
+## API summary
+
+- `fireRequest({ connector, request, variables, network, timeoutMs, maxBytes, encodePathVariables, fetch })` → `{ ok, status?, values, body?, contentType?, error? }`
+- Building blocks: `buildRequest`, `buildAuthHeaders`, `mapResponse`, `interpolate`, `interpolatePairs`, `extractVariableNames`, `evaluateJsonPath`, `checkUrlAllowed`, `parsePattern`, `pinnedFetch`.
+
+Full reference: [docs/install-and-use.md](docs/install-and-use.md#api-reference). Types: [`src/index.d.ts`](src/index.d.ts).
+
+## Documentation
+
+- [Install and use guide](docs/install-and-use.md): data shapes, every option, recipes, troubleshooting
+- [Integration guide](docs/integration-guide.md): wiring varfetch into an application, with the Saarnavideo and LCYT integrations as worked examples
+- [Architecture](docs/architecture.md): modules, request lifecycle, design decisions
+- [Security model](docs/security.md): threat model, SSRF guard, DNS pinning, what is not covered
+- [Changelog](CHANGELOG.md)
+
+## Development
+
+```sh
+npm test    # node --test, no build step
+```
 
-## API
+CI runs the tests on Node 18, 20 and 22.
 
-`fireRequest({ connector, request, variables, network, fetch, timeoutMs, maxBytes })` returns `{ ok, status?, values, body?, error? }` and never throws.
+### Releases
 
-Building blocks: `buildRequest(connector, request, variables)`, `buildAuthHeaders`, `mapResponse(mappings, body)`, `interpolate`, `interpolatePairs`, `extractVariableNames`, `evaluateJsonPath`, `checkUrlAllowed(url, { allow, deny })`, `parsePattern`. Types are in `src/index.d.ts`.
+Versioning and publishing use [Changesets](https://github.com/changesets/changesets). For a user-facing change run `npm run changeset` and commit the file it creates. On merge to `main` the Release workflow opens a "Version Packages" PR; merging it publishes to npm through [Trusted Publishing](https://docs.npmjs.com/trusted-publishers) (OIDC, no npm token stored). Do not edit `version` or run `npm publish` by hand. One-time setup: on npmjs.com, package `varfetch` → Settings → Trusted Publisher → GitHub Actions, repository `jsilvanus/varfetch`, workflow `release.yml`.
 
 ## License
 

package/docs/architecture.md

@@ -0,0 +1,61 @@
+# Architecture
+
+varfetch is five small modules and a transport. It has no state, no I/O except the HTTP call itself, and no dependencies. All persistence, UI and scheduling belong to the host application.
+
+```
+                      fireRequest(args)                      src/fire.js
+                            │
+        ┌───────────────────┼───────────────────────┐
+        ▼                   ▼                       ▼
+  buildRequest         redirect loop           mapResponse
+  src/request.js    (≤ 5 hops, per hop:)       src/request.js
+   │  interpolate        │                         │
+   │  src/interpolate.js │                         └─ evaluateJsonPath
+   │                     ├─ checkUrlAllowed             src/json-path.js
+   │                     │   src/network-guard.js
+   │                     │   resolve → rules → validated addresses
+   │                     │
+   │                     └─ pinnedFetch(url, …, { addresses })
+   │                         src/transport.js  (node:http / node:https)
+   ▼
+ { url, method, headers, body }
+```
+
+## Modules
+
+| Module | Responsibility | Pure? |
+|---|---|---|
+| `interpolate.js` | `{{name}}` substitution, listing the names a string uses | yes |
+| `json-path.js` | Minimal JSONPath subset (`$`, `.key`, `[n]`, `['key']`), own properties only | yes |
+| `request.js` | `buildRequest` (URL, headers, auth, body from definitions + variables), `buildAuthHeaders`, `mapResponse` | yes |
+| `network-guard.js` | Pattern parsing, default restricted ranges, DNS resolution, allow/deny decision, returns validated addresses | DNS only |
+| `transport.js` | `pinnedFetch`: HTTP(S) request pinned to given addresses, returns a standard `Response` | network |
+| `fire.js` | Orchestration: build, guard, send, follow redirects, enforce limits, parse, map; never throws | network |
+
+`index.js` re-exports everything; `index.d.ts` holds the hand-written types. Both are the public API.
+
+## Request lifecycle
+
+1. **Build.** `buildRequest` interpolates the path, query values, connector headers and auth values, and the body. A bad base URL throws here; `fireRequest` turns it into `{ ok: false, error: 'Invalid request: …' }`.
+2. **Start the clock.** One `AbortSignal.timeout` covers the whole call: DNS, connect, all redirect hops and reading the body.
+3. **Guard (per hop).** `checkUrlAllowed` resolves the host and applies the rules (see [security.md](security.md)). A refusal ends the call with the reason as `error`.
+4. **Send (per hop).** `pinnedFetch` connects to the validated addresses only. A custom `fetch` replaces it.
+5. **Redirects.** `fetch`'s own redirect handling is never used. For a 3xx with `Location`, varfetch resolves the URL, drops all connector headers except `Content-Type`/`Accept` when the origin changes, turns 303 (and 301/302 after POST) into a bodyless GET, cancels the old body and loops back to step 3. More than 5 hops is an error.
+6. **Read.** The body is read with a hard byte limit (decompressed bytes). A non-2xx status is `{ ok: false, status, error: 'HTTP 404' }`.
+7. **Parse.** `binary`/`image` returns the `Buffer` and content type. Otherwise JSON is parsed when requested, or when `auto` and the content type contains `json`. A parse failure is an error.
+8. **Map.** `mapResponse` evaluates each `jsonPath` and writes strings (objects as JSON text) into `values`.
+
+## Design decisions
+
+- **Definitions are plain data.** A connector is `{ baseUrl, auth, headers }` and a request is `{ method, path, query, body, mappings }`. varfetch has no storage layer so it fits a database row, a YAML file, or an object literal. The host converts its rows to these shapes (Saarnavideo: `toVarfetch`).
+- **Never throws.** `fireRequest` always resolves to a result object, so callers can map failures straight onto an HTTP 502 or a UI message without try/catch.
+- **The guard sits where the URL is final.** Checking at the point of connection (and per redirect hop) means the guard cannot be bypassed by a request definition, a variable value or a redirect.
+- **Pinning through `lookup`, not through rewriting the URL.** Replacing the host with an IP would break TLS certificate checks and virtual hosting. A pinned `lookup` keeps the URL, Host and SNI intact and changes only where the socket goes.
+- **Own transport instead of `fetch`.** Node's `fetch` (undici) offers no dependency-free hook for a per-request resolver. `node:http(s)` does, and the result is wrapped in a standard `Response` so the rest of the code (and any custom `fetch`) shares one shape.
+- **Per-address allow rules.** An allow rule is a statement about addresses; evaluating it per address is the only reading that is safe when a name has several.
+- **Small JSONPath.** Only what connector mappings need. No wildcards, filters or recursion: they cost parsing complexity and make mappings unpredictable.
+- **Rules come from the host.** `allow`/`deny` are arguments, not environment reads, so each host decides where policy lives (env, database, per organisation). LCYT merges organisation and site rules from its database; Saarnavideo reads `CONNECTOR_ALLOW`/`CONNECTOR_DENY`.
+
+## Testing
+
+`node --test test/*.test.js`. Network tests start a local `http` server on `127.0.0.1` and allow it explicitly; DNS behaviour is tested with an injected `lookup`, so no real DNS is needed. The DNS pinning tests use an unresolvable name (`rebind.invalid`) and succeed only if the connection uses the validated address.

package/docs/install-and-use.md

@@ -0,0 +1,199 @@
+# Install and use
+
+## Install
+
+```sh
+npm install varfetch        # or pnpm add / yarn add
+```
+
+- Node 18 or newer (CI runs 18, 20 and 22).
+- ESM only: `import { fireRequest } from 'varfetch'`. From CommonJS use `const { fireRequest } = await import('varfetch')`.
+- No runtime dependencies. Types are included.
+
+## First request
+
+```js
+import { fireRequest } from 'varfetch';
+
+const result = await fireRequest({
+  connector: { baseUrl: 'https://api.example.org' },
+  request: {
+    path: '/api/v1/date/{{date}}',
+    mappings: [{ jsonPath: '$.holyDay.name', variable: 'holyday' }],
+  },
+  variables: { date: '2026-10-11' },
+});
+// { ok: true, status: 200, values: { holyday: '…' }, body: { …whole JSON… } }
+```
+
+Two objects describe what to call, one describes the values:
+
+- **connector**: where and as whom (`baseUrl`, `auth`, shared `headers`). Reused by many requests.
+- **request**: what to call (`method`, `path`, `query`, `body`) and what to take from the answer (`mappings`).
+- **variables**: a plain object whose values fill the `{{name}}` placeholders.
+
+## Data shapes
+
+### Connector
+
+```ts
+{
+  baseUrl: string;                       // https://host[:port][/prefix]
+  auth?: Auth;
+  headers?: { key: string; value: string }[];   // sent with every request, values interpolated
+}
+```
+
+| `auth.type` | Fields | Sends |
+|---|---|---|
+| `none` (or no `auth`) | | nothing |
+| `bearer` | `token` | `Authorization: Bearer <token>` |
+| `api_key` | `headerName`, `value` | `<headerName>: <value>` |
+| `basic` | `username`, `password?` | `Authorization: Basic base64(user:pass)` |
+| `custom` | `headers: { name: value }` | those headers |
+
+Auth values may contain `{{name}}`, so a token can come from `variables`. Auth headers win over connector headers with the same name.
+
+### Request
+
+```ts
+{
+  method?: string;                       // default GET
+  path?: string;                         // appended to baseUrl, {{name}} allowed
+  query?: { key: string; value: string }[];     // values interpolated and URL-encoded
+  bodyType?: 'none' | 'json' | 'text';   // body is sent only for non-GET/HEAD
+  body?: string;                         // interpolated; Content-Type set from bodyType unless you set one
+  responseType?: 'auto' | 'json' | 'text' | 'binary' | 'image';
+  mappings?: { jsonPath: string; variable: string; skipIfNull?: boolean }[];
+  timeoutMs?: number;                    // overridden by fireRequest's timeoutMs
+}
+```
+
+`responseType` `auto` (default) parses JSON when the response content type contains `json`, otherwise the body is text. With `text`, `$` maps the whole text. `binary` and `image` return a `Buffer` as `body` and the `contentType`, and ignore `mappings`.
+
+### Result
+
+```ts
+{ ok: boolean; status?: number; values: Record<string, string | null>; body?: unknown; contentType?: string; error?: string }
+```
+
+`values` holds one entry per mapping: strings as they are, objects and arrays as JSON text, unresolved paths left out (or `null` when `skipIfNull: false`). `error` is a short human-readable text (`HTTP 404`, `Timed out after 10000 ms`, `Blocked: target resolves to a private/internal/reserved address`, …). It never contains a stack trace or your secrets.
+
+## Variables
+
+- `{{name}}`: letters (including ä/ö/å), digits, `_` and `-`; whitespace inside the braces is fine (`{{ name }}`).
+- Missing or `null` variables become an empty string. Use `extractVariableNames(text)` to list what a string needs, for example to show input fields.
+- **Path values are inserted raw.** `{{d}}` = `../x?y` changes the URL. When the value is not under your control, set `encodePathVariables: true` or use a query parameter.
+
+```js
+await fireRequest({ connector, request, variables: { user: name }, encodePathVariables: true });
+```
+
+## JSON paths
+
+| Path | Meaning |
+|---|---|
+| `$` | the whole body |
+| `$.a.b` | nested keys |
+| `$.items[0].name` | array index |
+| `$['weird key']`, `$["k"]` | quoted key |
+
+No wildcards, filters or `..`. Only own properties resolve, so `$.constructor` and `$.__proto__` give nothing.
+
+## Options of `fireRequest`
+
+| Option | Default | |
+|---|---|---|
+| `network` | `{}` | `{ allow: string[], deny: string[] }`, see below |
+| `timeoutMs` | request's `timeoutMs`, else 10000 | covers DNS, connect, redirects and body |
+| `maxBytes` | 5 MiB | response limit, after decompression |
+| `encodePathVariables` | `false` | `encodeURIComponent` on values inserted into the path |
+| `fetch` | pinned transport | custom `fetch`; disables DNS pinning (see [security](security.md)) |
+
+Fixed: at most 5 redirects; connector headers are not forwarded to another origin.
+
+## Network rules
+
+```js
+network: {
+  allow: ['10.1.0.0/16', 'anno.internal:3000', '[fd00::]/8', '*.corp.example'],
+  deny:  ['10.1.9.9', '*.blocked.example'],
+}
+```
+
+- Without rules, only public addresses are reachable.
+- `allow` opens specific private targets. Each resolved address must be public or covered by an allow rule; a host name rule covers all addresses of that host.
+- `deny` wins over everything and is also useful to block public hosts.
+- A pattern without a port matches any port. Host names are case-insensitive; `*.example.com` also matches `example.com`.
+
+To see why a URL is refused:
+
+```js
+import { checkUrlAllowed } from 'varfetch';
+await checkUrlAllowed(new URL('http://10.0.0.5:3000/'), { allow: [] });
+// { allowed: false, reason: 'Blocked: target resolves to a private/internal/reserved address' }
+```
+
+## Recipes
+
+**POST with a JSON body**
+
+```js
+request: {
+  method: 'POST', path: '/v1/notes', bodyType: 'json',
+  body: '{"title": "{{title}}"}',      // values are inserted as text: escape JSON yourself if they may contain quotes
+  mappings: [{ jsonPath: '$.id', variable: 'noteId' }],
+}
+```
+
+If a value can contain quotes or newlines, build the body with `JSON.stringify` in your code and pass it as a variable: `body: '{{payload}}'` with `variables: { payload: JSON.stringify({ title }) }`.
+
+**Download an image**
+
+```js
+const r = await fireRequest({ connector, request: { path: '/logo.png', responseType: 'image' }, network });
+if (r.ok) fs.writeFileSync('logo.png', r.body); // r.contentType is e.g. 'image/png'
+```
+
+**Preview before applying.** `buildRequest(connector, request, variables)` returns `{ url, method, headers, body }` without sending anything; show it to the user, or log it with the headers masked.
+
+**Test your own code.** Pass `fetch` to avoid the network:
+
+```js
+const fakeFetch = async () => new Response(JSON.stringify({ a: 1 }), { headers: { 'content-type': 'application/json' } });
+await fireRequest({ connector: { baseUrl: 'https://api.example.org' }, request, fetch: fakeFetch, network: { allow: [] } });
+```
+
+(The guard still resolves the host. For a fully offline test use a literal public IP in `baseUrl`, or a local server allowed through `network.allow`.)
+
+## API reference
+
+| Export | Signature |
+|---|---|
+| `fireRequest(args)` | `Promise<FireResult>`, never rejects |
+| `buildRequest(connector, request, variables?, { encodePathVariables }?)` | `{ url: URL, method, headers, body? }`; throws on an invalid base URL |
+| `buildAuthHeaders(auth, variables?)` | `Record<string, string>` |
+| `mapResponse(mappings, body)` | `Record<string, string \| null>` |
+| `interpolate(text, variables?, encode?)` | `string` |
+| `interpolatePairs(pairs, variables?)` | `{ key, value }[]` |
+| `extractVariableNames(text)` | `string[]` |
+| `evaluateJsonPath(data, path)` | value or `undefined` |
+| `checkUrlAllowed(url, { allow, deny, lookup?, signal? })` | `{ allowed, reason?, addresses? }` |
+| `parsePattern(pattern)` | `{ kind: 'host' \| 'ip' \| 'cidr', value, port }` |
+| `pinnedFetch(url, init, { addresses })` | `Promise<Response>` connecting only to `addresses` |
+
+## Troubleshooting
+
+| `error` | Cause and fix |
+|---|---|
+| `Blocked: target resolves to a private/internal/reserved address` | The target is on a private network. Add an `allow` rule if it is intended. |
+| `Blocked by network policy` | A `deny` rule matched. |
+| `Could not resolve host: …` | DNS failure, or the name does not exist. |
+| `Invalid request: Invalid URL` | `baseUrl` or the interpolated path does not form a URL. |
+| `Timed out after N ms` | Raise `timeoutMs`, or the remote is slow. |
+| `Response larger than N bytes` | Raise `maxBytes` or fetch a smaller resource. |
+| `Response is not valid JSON` | `responseType` is `json` (or auto and the server says JSON) but the body is not. |
+| `HTTP 4xx/5xx` | The remote refused; `status` has the code. |
+| `Too many redirects` | More than five hops or a loop. |
+| `Credentials in the URL are not supported` | Remove `user:pass@` from `baseUrl`; use `auth`. |
+| `values` is empty although `ok` is true | The JSON paths did not resolve. Inspect `result.body`. |

package/docs/integration-guide.md

@@ -0,0 +1,135 @@
+# Integration guide
+
+How to wire varfetch into an application. The pattern is the same everywhere: store connectors and requests, convert a stored request to varfetch's plain shapes, fire it with server-owned network rules, and store the resulting values as variables.
+
+```
+ UI / API ──► stored connector + request ──► toVarfetch() ──► fireRequest() ──► values ──► variables
+                 (your database)                                  ▲
+                                                  network rules from server config
+```
+
+## 1. Decide where things live
+
+| Concern | Put it | Why |
+|---|---|---|
+| Connector and request definitions | Your database or config, editable by authorised users | They are data |
+| Secrets (tokens, passwords) | Same store, encrypted; return only `hasSecret` to clients | Never send them back to a browser |
+| **Network `allow`/`deny` rules** | **Server configuration** (env, or admin-only settings) | A user who can edit connectors must not be able to widen what the server may reach |
+| Variables | Wherever your app keeps project/session values | varfetch only produces the values |
+
+## 2. Model the data
+
+Anything that converts to these shapes works:
+
+```ts
+interface Connector { baseUrl: string; auth?: Auth; headers?: { key: string; value: string }[] }
+interface RequestDef { method?; path?; query?; bodyType?; body?; responseType?; mappings?: { jsonPath; variable; skipIfNull? }[]; timeoutMs? }
+```
+
+Validate on the way in (a Zod schema, JSON Schema): `baseUrl` must be `http(s)`, variable names must be valid (`[\p{L}_][\p{L}\p{N}_-]*`), header names must be tokens. Validate again when converting a row: JSON columns can hold anything.
+
+## 3. A minimal Express integration
+
+```js
+import express from 'express';
+import { fireRequest } from 'varfetch';
+
+const network = {
+  allow: (process.env.CONNECTOR_ALLOW ?? '').split(',').map((s) => s.trim()).filter(Boolean),
+  deny:  (process.env.CONNECTOR_DENY  ?? '').split(',').map((s) => s.trim()).filter(Boolean),
+};
+
+const app = express();
+app.use(express.json());
+
+app.post('/connectors/:id/requests/:rid/fire', async (req, res) => {
+  const { connector, request } = await loadFromDatabase(req.params.id, req.params.rid); // your code
+  const result = await fireRequest({
+    connector,
+    request,
+    variables: req.body.variables ?? {},
+    network,
+    encodePathVariables: true,          // values come from the client
+  });
+  if (!result.ok) return res.status(502).json({ error: result.error, status: result.status });
+  res.json({ values: result.values });
+});
+```
+
+Notes:
+
+- Answer upstream failures with **502** (and the remote `status` when present); keep your own 4xx for bad input.
+- Do not return `result.body` to the client unless you want the whole remote answer exposed; `values` is the designed output.
+- Never log `connector.auth`. If you log requests, use `buildRequest` and mask `Authorization` and any api-key header.
+
+## 4. Turning values into variables
+
+`values` is `Record<string, string | null>`. Typical flow ("fetch variables" button):
+
+1. Fire the request with the current variables as input (the request path may use them, e.g. `{{date}}`).
+2. Show the user old versus new values.
+3. Save only what they accept.
+
+Do not write `values` straight into state that triggers other work (renders, publications) without a confirmation step unless the connector is fully trusted. A remote server controls those strings.
+
+## 5. Worked example: Saarnavideo
+
+Saarnavideo uses varfetch for user-defined API variables, with the church-year service anno-api as the main use case. Nothing about that service is built in.
+
+- **Storage.** Prisma models `ApiConnector` and `ApiRequest` (global, both schemas). The connector secret is stored but only `hasSecret` is returned.
+- **Conversion.** `toVarfetch(connectorRow, requestRow)` (`src/domain/connectors.ts`) validates JSON columns with Zod (`asPairs`, `asAuth`, `mappingSchema`) and returns `{ connector, request }`.
+- **Network rules.** `networkRulesFromEnv()` reads `CONNECTOR_ALLOW` and `CONNECTOR_DENY` (comma-separated patterns). The app has no login, so rules are environment-only. To use an anno-api on a private network, run Saarnavideo with, for example, `CONNECTOR_ALLOW=anno.internal:3000` or `CONNECTOR_ALLOW=10.1.0.0/16`.
+- **Firing.** `runStoredRequest` (`src/app/api/_lib/connectors.ts`) loads the request, calls `fireRequest({ connector, request, variables, network })`, and answers 502 with `fireErrorMessage(result)` on failure.
+- **Variables.** `POST /api/projects/[id]/fetch-variables` fires a request with the project's variables (service date default: next Sunday as `paiva`) and the Lähde step's `FetchVariables` shows old versus new values before `saveVariables` stores the accepted ones.
+- **Preset.** "Lisää kirkkovuosipohja" adds a request preset for an anno-api style day endpoint (`/api/v1/date/{{paiva}}`) with mappings to ASCII variable names (`pyhapaiva`, `teema`, `evankeliumi`, `evankeliumiteksti`, `vari`, `jakso`, `aika`) to a connector whose address the user typed.
+
+Hardening worth doing in a login-less app: `paiva` goes into the path, so set `encodePathVariables: true` in `runStoredRequest`, and add a login before exposing the app beyond a trusted network.
+
+## 6. Worked example: LCYT
+
+`lcyt-connectors` uses varfetch for the same job inside a multi-tenant server and layers its own policy on top:
+
+- **Policy from the database.** Site-wide (admin) and per-organisation rules are loaded and merged into one `{ allow, deny }`: org deny and site deny become `deny`, org and site allow become `allow`. `fireRequest({ network })` then enforces them at every hop.
+- **Layered error messages.** Before firing, LCYT calls `checkUrlAllowed` once per deny layer with a memoised `lookup`, so the error can say whether the organisation or the site policy blocked the request. The `lookup` option accepts a function returning one result or an array.
+- **Binary responses.** `responseType: 'image'` returns the bytes and `contentType`; LCYT stores them as an asset instead of mapping text.
+- **Events.** Values written to variables emit an event so the UI updates live.
+
+## 7. Policy recipes
+
+| Goal | Rules |
+|---|---|
+| Public APIs only | no rules |
+| One internal service | `allow: ['anno.internal:3000']` |
+| Internal subnet, but not the database host | `allow: ['10.1.0.0/16']`, `deny: ['10.1.0.5']` |
+| Public APIs except one vendor | `deny: ['*.vendor.example']` |
+| Only specific public hosts (allow-list mode) | varfetch has no "deny all" rule: enforce it yourself by rejecting connectors whose host is not on your list before calling `fireRequest` |
+| Local development against `localhost` | `allow: ['127.0.0.1:3000']` (name `localhost` resolves to a loopback address, so `allow: ['localhost:3000']` works too) |
+
+## 8. Operations checklist
+
+- [ ] Network rules come from server config, not from user-editable data
+- [ ] Secrets encrypted at rest, never returned or logged
+- [ ] Upstream failures answered as 502; no stack traces to clients
+- [ ] `encodePathVariables: true` whenever a variable value comes from a user
+- [ ] Authorisation on who can create and edit connectors and fire requests
+- [ ] Rate limiting on the fire endpoint
+- [ ] Mapped values escaped when rendered
+- [ ] No custom `fetch` in production unless you enforce the network policy elsewhere (it disables DNS pinning)
+- [ ] Behind a corporate proxy: see [security.md](security.md#when-pinning-does-not-apply)
+
+## 9. Testing an integration
+
+Run a local `http` server, allow it by address, and fire real requests:
+
+```js
+const server = http.createServer((req, res) => { res.setHeader('content-type', 'application/json'); res.end('{"name":"Sunday"}'); });
+await new Promise((r) => server.listen(0, '127.0.0.1', r));
+const port = server.address().port;
+const result = await fireRequest({
+  connector: { baseUrl: `http://127.0.0.1:${port}` },
+  request: { path: '/', mappings: [{ jsonPath: '$.name', variable: 'n' }] },
+  network: { allow: [`127.0.0.1:${port}`] },
+});
+```
+
+To assert that your app refuses private targets, fire at `http://127.0.0.1:<port>` with no `allow` and expect `ok: false`.

package/docs/security.md

@@ -0,0 +1,73 @@
+# Security model
+
+varfetch exists so that an application can let someone configure an outbound HTTP request without handing that person a way to attack the server's own network. This page states what it defends against, how, and what it does not cover.
+
+## Threat model
+
+The **operator** runs the application and decides which networks it may reach. The **configurer** is whoever edits connectors and requests (an admin, an organisation owner, or, in an app without login, anyone who can open the page). The **remote server** is whatever the URL points at, and it may be hostile or compromised.
+
+| Threat | Defence |
+|---|---|
+| Configurer points a connector at an internal service or the cloud metadata endpoint (`169.254.169.254`) | Default-deny of private, loopback, link-local and reserved addresses |
+| A public-looking name that resolves to a private address | The check runs on the resolved addresses, not the name |
+| DNS rebinding: the name answers with a public address for the check and a private one for the connection | **DNS pinning**: the connection uses the validated addresses only |
+| Remote server redirects to an internal address | Redirects are followed by varfetch and every hop is checked and pinned |
+| Remote server redirects to another origin to harvest credentials | Connector headers (including auth) are dropped on a cross-origin redirect |
+| Huge or endless response | 5 MiB limit on decompressed bytes, 10 s timeout (both configurable) |
+| Compression bomb | The limit counts bytes after decoding |
+| Variable value alters the request path (`../`, `?`, `#`) | `encodePathVariables`, or put the value in a query parameter |
+| Prototype access through the JSON path (`$.__proto__`) | Own properties only |
+| Non-HTTP schemes (`file:`, `ftp:`) | Rejected |
+| `http://user:pw@host` URLs | Rejected, use the `auth` setting |
+| Hung DNS resolver | The lookup is bound to the request timeout |
+
+## The SSRF guard
+
+`checkUrlAllowed(url, { allow, deny })` decides per URL. Order of evaluation:
+
+1. Scheme other than `http`/`https` → blocked.
+2. Credentials in the URL → blocked.
+3. The host is resolved (`dns.lookup`, all addresses).
+4. A `deny` pattern matches **any** resolved address → blocked.
+5. For **each** resolved address: it must be covered by an `allow` pattern or not be in a restricted range; otherwise the whole URL is blocked.
+6. Otherwise allowed, and the validated `addresses` are returned.
+
+Step 5 is per address on purpose. A host that resolves to one allowed and one restricted address is refused, because the connection could use either one. A *hostname* allow pattern (`anno.internal`) covers all addresses of that host, because you named the host as trusted.
+
+Blocked by default:
+
+- IPv4: `0.0.0.0/8`, `10/8`, `100.64/10`, `127/8`, `169.254/16`, `172.16/12`, `192.0.0/24`, `192.0.2/24`, `192.88.99/24`, `192.168/16`, `198.18/15`, `198.51.100/24`, `203.0.113/24`, `224/4`, `240/4`
+- IPv6: `::1`, `::`, `fc00::/7`, `fe80::/10`, `ff00::/8`, `100::/64`, `2001::/32` (Teredo), `2001:db8::/32`, `2002::/16` (6to4), `64:ff9b::/96` (NAT64)
+- IPv4-mapped IPv6 (`::ffff:a.b.c.d`) is checked as the IPv4 address
+
+Pattern syntax: exact host (`api.example.com`), wildcard (`*.example.com`, also matches `example.com`), IP (`10.1.2.3`), CIDR (`10.0.0.0/8`, `[fc00::]/7`), each with an optional port (`10.1.2.3:3000`, `[::1]:11434`). No port means any port. Host names are compared in lower case without a trailing dot.
+
+## DNS pinning
+
+A plain "resolve, check, then `fetch`" has a gap: `fetch` resolves the name again. An attacker who controls the DNS zone returns a public address with TTL 0 for the check and `127.0.0.1` for the connection.
+
+varfetch closes the gap in `pinnedFetch` (`src/transport.js`). It calls `node:http` or `node:https` with a `lookup` function that returns the addresses `checkUrlAllowed` validated, so the operating system's resolver is never consulted for the connection. Everything else stays as if the name had been used: the URL, the `Host` header, the TLS server name (SNI) and certificate verification (the certificate must be valid for the host name, not for the IP).
+
+Details:
+
+- One DNS lookup per hop. If the resolver returns several addresses, Node's normal connection fallback applies, but only among validated addresses.
+- No connection pooling (`agent: false`), so a reused socket cannot carry an old validation to a new request.
+- The same pinning applies to every redirect hop.
+- IP literals in the URL need no lookup and are checked directly.
+
+### When pinning does not apply
+
+If you pass your own `fetch` to `fireRequest`, that function resolves the host itself. The guard still checks the URL (and every redirect), but the rebinding window is back. Use a custom `fetch` for tests, or when a corporate proxy is required (the proxy then does the resolving, so enforce the policy there). Do not use it to "add logging" in production without keeping this in mind.
+
+## What varfetch does not protect
+
+- **The remote server's content.** Response values are strings from a server you chose to call. If you render them in HTML, escape them; if you put them into another request, treat them as untrusted.
+- **Secrets at rest.** Auth tokens in a connector object are plain strings. Store them encrypted, never return them to a browser, and scrub them from error messages. (Saarnavideo stores the secret but only returns `hasSecret`.)
+- **Who may configure connectors.** If anyone can create a connector, anyone can make your server call the allowed targets. Decide your own authorisation. In particular, keep `allow` rules in server configuration, so the people who configure connectors cannot widen what the server may reach.
+- **Request volume.** There is no rate limiting. A connector can be used to send many requests to an allowed target.
+- **TOCTOU on allowed internal targets.** An allowed internal host is trusted by definition; pinning prevents the address from changing, not the service behind it from misbehaving.
+- **Proxies.** `HTTP_PROXY` and friends are ignored by the default transport. In a proxy-only network, use a custom `fetch` and enforce policy at the proxy.
+
+## Reporting a vulnerability
+
+Open a private security advisory on the GitHub repository (Security tab), or email the maintainer listed in `package.json`.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "varfetch",
-  "version": "0.1.0",
+  "version": "0.3.0",
   "description": "Fire configured HTTP requests with {{variable}} interpolation, an SSRF guard and JSON path mapping of the response onto named variables.",
   "type": "module",
   "main": "src/index.js",
@@ -11,11 +11,25 @@
       "default": "./src/index.js"
     }
   },
-  "files": ["src", "LICENSE", "README.md"],
+  "files": [
+    "src",
+    "docs",
+    "LICENSE",
+    "README.md",
+    "CHANGELOG.md"
+  ],
   "scripts": {
-    "test": "node --test test/*.test.js"
+    "test": "node --test test/*.test.js",
+    "changeset": "changeset"
   },
-  "keywords": ["http", "variables", "api", "connector", "ssrf", "jsonpath"],
+  "keywords": [
+    "http",
+    "variables",
+    "api",
+    "connector",
+    "ssrf",
+    "jsonpath"
+  ],
   "author": "Juha Itäleino <jsilvanus@gmail.com>",
   "license": "EUPL-1.2",
   "engines": {
@@ -28,5 +42,13 @@
   "bugs": {
     "url": "https://github.com/jsilvanus/varfetch/issues"
   },
-  "homepage": "https://github.com/jsilvanus/varfetch#readme"
+  "homepage": "https://github.com/jsilvanus/varfetch#readme",
+  "devDependencies": {
+    "@changesets/cli": "^2.31.1",
+    "@types/node": "^22.20.5"
+  },
+  "publishConfig": {
+    "access": "public",
+    "provenance": true
+  }
 }

package/src/fire.js

@@ -1,5 +1,6 @@
 import { buildRequest, mapResponse } from './request.js';
 import { checkUrlAllowed } from './network-guard.js';
+import { pinnedFetch } from './transport.js';
 
 const MAX_REDIRECTS = 5;
 const DEFAULT_TIMEOUT_MS = 10_000;
@@ -9,22 +10,26 @@ const DEFAULT_MAX_BYTES = 5 * 1024 * 1024;
  * Send a request and map the response onto variables.
  *
  * Every URL is checked with the SSRF guard, redirects included (they are
- * followed here, up to 5, instead of by fetch).
+ * followed here, up to 5, instead of by fetch). Unless a custom `fetch` is
+ * given, the connection goes only to the addresses the guard validated
+ * (`pinnedFetch`), so DNS rebinding between check and connect is not possible.
+ * A custom `fetch` resolves the host itself and is therefore checked, not pinned.
  *
  * @param {object} args
  * @param {import('./index.js').Connector} args.connector
  * @param {import('./index.js').RequestDef} args.request
  * @param {Record<string, unknown>} [args.variables] values for {{name}}
  * @param {{ allow?: string[], deny?: string[] }} [args.network] extra guard patterns, see network-guard.js
- * @param {typeof fetch} [args.fetch]
+ * @param {typeof fetch} [args.fetch] replaces the pinned transport (tests, proxies); DNS pinning does not apply to it
+ * @param {boolean} [args.encodePathVariables] run variable values in the path through encodeURIComponent
  * @param {number} [args.timeoutMs] default 10000
  * @param {number} [args.maxBytes] response size limit, default 5 MiB
  * @returns {Promise<import('./index.js').FireResult>}
  */
-export async function fireRequest({ connector, request, variables = {}, network = {}, fetch: fetchImpl = fetch, timeoutMs, maxBytes = DEFAULT_MAX_BYTES }) {
+export async function fireRequest({ connector, request, variables = {}, network = {}, fetch: fetchImpl, timeoutMs, maxBytes = DEFAULT_MAX_BYTES, encodePathVariables = false }) {
   let built;
   try {
-    built = buildRequest(connector, request, variables);
+    built = buildRequest(connector, request, variables, { encodePathVariables });
   } catch (err) {
     return { ok: false, values: {}, error: `Invalid request: ${err.message}` };
   }
@@ -40,12 +45,15 @@ export async function fireRequest({ connector, request, variables = {}, network
     let response;
 
     for (let hop = 0; ; hop++) {
-      const guard = await checkUrlAllowed(url, network);
+      const guard = await checkUrlAllowed(url, { ...network, signal });
       if (!guard.allowed) return { ok: false, values: {}, error: guard.reason };
 
-      response = await fetchImpl(url, { method, headers, body, redirect: 'manual', signal });
+      response = fetchImpl
+        ? await fetchImpl(url, { method, headers, body, redirect: 'manual', signal })
+        : await pinnedFetch(url, { method, headers, body, signal }, { addresses: guard.addresses });
       const location = response.status >= 300 && response.status < 400 ? response.headers.get('location') : null;
       if (!location) break;
+      await discardBody(response);
       if (hop >= MAX_REDIRECTS) return { ok: false, values: {}, status: response.status, error: 'Too many redirects' };
 
       const next = new URL(location, url);
@@ -60,10 +68,14 @@ export async function fireRequest({ connector, request, variables = {}, network
       url = next;
     }
 
-    const text = await readLimited(response, maxBytes);
+    const bytes = await readLimited(response, maxBytes);
     if (!response.ok) return { ok: false, status: response.status, values: {}, error: `HTTP ${response.status}` };
 
     const contentType = response.headers.get('content-type') || '';
+    if (request.responseType === 'binary' || request.responseType === 'image') {
+      return { ok: true, status: response.status, values: {}, body: bytes, contentType };
+    }
+    const text = bytes.toString('utf8');
     const wantsJson = request.responseType === 'json' || ((request.responseType ?? 'auto') === 'auto' && /json/i.test(contentType));
     let parsed = text;
     if (wantsJson) {
@@ -75,7 +87,7 @@ export async function fireRequest({ connector, request, variables = {}, network
     }
     return { ok: true, status: response.status, values: mapResponse(request.mappings, parsed), body: parsed };
   } catch (err) {
-    const message = err?.name === 'TimeoutError' ? `Timed out after ${timeout} ms` : err?.message ?? String(err);
+    const message = err?.name === 'TimeoutError' || (err?.name === 'AbortError' && signal.aborted) ? `Timed out after ${timeout} ms` : err?.message ?? String(err);
     return { ok: false, values: {}, error: message };
   }
 }
@@ -83,7 +95,7 @@ export async function fireRequest({ connector, request, variables = {}, network
 async function readLimited(response, maxBytes) {
   const declared = Number(response.headers.get('content-length'));
   if (declared > maxBytes) throw new Error(`Response larger than ${maxBytes} bytes`);
-  if (!response.body) return '';
+  if (!response.body) return Buffer.alloc(0);
   const reader = response.body.getReader();
   const chunks = [];
   let size = 0;
@@ -97,5 +109,14 @@ async function readLimited(response, maxBytes) {
     }
     chunks.push(value);
   }
-  return Buffer.concat(chunks).toString('utf8');
+  return Buffer.concat(chunks);
+}
+
+/** Release a redirect response's connection. Older Node versions throw on cancelling an already closed stream. */
+async function discardBody(response) {
+  try {
+    await response.body?.cancel();
+  } catch {
+    // The body is already finished: nothing to release.
+  }
 }

package/src/index.d.ts

@@ -31,7 +31,8 @@ export interface RequestDef {
   query?: Pair[];
   bodyType?: 'none' | 'json' | 'text';
   body?: string;
-  responseType?: 'auto' | 'json' | 'text';
+  /** `binary` and `image` return the raw bytes as `body` (a Buffer) and skip `mappings`. */
+  responseType?: 'auto' | 'json' | 'text' | 'binary' | 'image';
   mappings?: Mapping[];
   timeoutMs?: number;
 }
@@ -43,25 +44,33 @@ export interface FireResult {
   status?: number;
   /** Variable name to value; non-string values are JSON text. */
   values: Record<string, string | null>;
-  /** Parsed JSON, or the text for a non-JSON response. */
+  /** Parsed JSON, the text for a non-JSON response, or a Buffer for a `binary`/`image` request. */
   body?: unknown;
+  /** Response content-type, set for a `binary`/`image` request. */
+  contentType?: string;
   error?: string;
 }
 
-export function interpolate(text: string, variables?: Variables): string;
+export function interpolate(text: string, variables?: Variables, encode?: (value: string) => string): string;
 export function interpolatePairs(pairs: Pair[] | undefined, variables?: Variables): Pair[];
 export function extractVariableNames(text: string): string[];
 export function evaluateJsonPath(data: unknown, path: string): unknown;
 export function parsePattern(pattern: string): { kind: 'host' | 'ip' | 'cidr'; value: string; port: number | null };
 export function checkUrlAllowed(
   url: URL,
-  options?: NetworkRules & { lookup?: (host: string, opts: { all: true; verbatim: true }) => Promise<Array<{ address: string; family: number }>> },
-): Promise<{ allowed: boolean; reason?: string }>;
+  options?: NetworkRules & { signal?: AbortSignal; lookup?: (host: string, opts: { all: true; verbatim: true }) => Promise<Array<{ address: string; family: number }>> },
+): Promise<{ allowed: boolean; reason?: string; addresses?: Array<{ address: string; family: number }> }>;
+export function pinnedFetch(
+  url: URL,
+  init: { method?: string; headers?: Record<string, string>; body?: string; signal?: AbortSignal },
+  pin: { addresses: Array<{ address: string; family: number }> },
+): Promise<Response>;
 export function buildAuthHeaders(auth: Auth | undefined, variables?: Variables): Record<string, string>;
 export function buildRequest(
   connector: Connector,
   request: RequestDef,
   variables?: Variables,
+  options?: { encodePathVariables?: boolean },
 ): { url: URL; method: string; headers: Record<string, string>; body?: string };
 export function mapResponse(mappings: Mapping[] | undefined, body: unknown): Record<string, string | null>;
 export function fireRequest(args: {
@@ -72,4 +81,5 @@ export function fireRequest(args: {
   fetch?: typeof fetch;
   timeoutMs?: number;
   maxBytes?: number;
+  encodePathVariables?: boolean;
 }): Promise<FireResult>;

package/src/index.js

@@ -3,3 +3,4 @@ export { evaluateJsonPath } from './json-path.js';
 export { checkUrlAllowed, parsePattern } from './network-guard.js';
 export { buildRequest, buildAuthHeaders, mapResponse } from './request.js';
 export { fireRequest } from './fire.js';
+export { pinnedFetch } from './transport.js';

package/src/interpolate.js

@@ -9,12 +9,14 @@ const VAR_RE = /\{\{\s*([\p{L}_][\p{L}\p{N}_-]*)\s*\}\}/gu;
  * A missing variable renders as an empty string.
  * @param {string} text
  * @param {Record<string, unknown>} [variables]
+ * @param {(value: string) => string} [encode] applied to each inserted value
  */
-export function interpolate(text, variables) {
+export function interpolate(text, variables, encode) {
   if (typeof text !== 'string' || !text.includes('{{')) return text;
   return text.replace(VAR_RE, (_match, name) => {
     const value = variables?.[name];
-    return value === undefined || value === null ? '' : String(value);
+    if (value === undefined || value === null) return '';
+    return encode ? encode(String(value)) : String(value);
   });
 }
 

package/src/network-guard.js

@@ -24,8 +24,15 @@
  *   - bracketed IPv6 + port "[::1]:11434"
  * A pattern without a port matches that host or IP on any port.
  *
- * The addresses are resolved once, before the fetch. This does not defend
- * against DNS rebinding between the check and the connection.
+ * Allow rules are applied per resolved address: an `allow` CIDR or IP only
+ * exempts the addresses it matches, so a name that resolves to one allowed and
+ * one restricted address is blocked. A hostname pattern exempts every address
+ * of that host.
+ *
+ * `checkUrlAllowed` returns the addresses it validated (`addresses`). To close
+ * the DNS rebinding gap (a second lookup at connect time returning a different
+ * address) the caller must connect to exactly those addresses: `fireRequest`
+ * does this through `pinnedFetch` (src/transport.js).
  */
 import { BlockList, isIP } from 'node:net';
 import { lookup as dnsLookup } from 'node:dns/promises';
@@ -38,8 +45,12 @@ const DEFAULT_BLOCKED_V4 = [
   ['169.254.0.0', 16],  // link-local — includes cloud metadata (169.254.169.254)
   ['172.16.0.0', 12],   // private
   ['192.0.0.0', 24],    // IETF protocol assignments
+  ['192.0.2.0', 24],    // documentation
+  ['192.88.99.0', 24],  // 6to4 relay anycast
   ['192.168.0.0', 16],  // private
   ['198.18.0.0', 15],   // benchmarking
+  ['198.51.100.0', 24], // documentation
+  ['203.0.113.0', 24],  // documentation
   ['224.0.0.0', 4],     // multicast
   ['240.0.0.0', 4],     // reserved
 ];
@@ -49,6 +60,11 @@ const DEFAULT_BLOCKED_V6 = [
   ['fc00::', 7],  // unique local (ULA)
   ['fe80::', 10], // link-local
   ['ff00::', 8],  // multicast
+  ['100::', 64],  // discard-only
+  ['2001::', 32], // Teredo (embeds an IPv4 address)
+  ['2001:db8::', 32], // documentation
+  ['2002::', 16], // 6to4 (embeds an IPv4 address)
+  ['64:ff9b::', 96], // NAT64 (embeds an IPv4 address)
   // No explicit IPv4-mapped (::ffff:x.x.x.x) rule needed: net.BlockList
   // already matches a mapped address checked as 'ipv6' against the
   // corresponding plain-IPv4 rule above checked as 'ipv4' — adding an
@@ -105,12 +121,13 @@ function hostMatches(pattern, hostname) {
   return pattern === hostname;
 }
 
-function ruleMatches(pattern, { hostname, addresses, port }) {
+/** Does `pattern` cover this one resolved address of the host? */
+function ruleMatchesAddress(pattern, { hostname, port }, address) {
   const parsed = parsePattern(pattern);
   if (parsed.port != null && parsed.port !== port) return false;
 
   if (parsed.kind === 'host') return hostMatches(parsed.value, hostname);
-  if (parsed.kind === 'ip') return addresses.some((a) => a.address === parsed.value);
+  if (parsed.kind === 'ip') return sameAddress(address.address, parsed.value);
 
   // CIDR
   try {
@@ -119,7 +136,19 @@ function ruleMatches(pattern, { hostname, addresses, port }) {
     const family = isIP(net) === 6 ? 'ipv6' : 'ipv4';
     const bl = new BlockList();
     bl.addSubnet(net, prefix, family);
-    return addresses.some((a) => bl.check(a.address, a.family === 6 ? 'ipv6' : 'ipv4'));
+    return bl.check(address.address, address.family === 6 ? 'ipv6' : 'ipv4');
+  } catch {
+    return false;
+  }
+}
+
+function sameAddress(a, b) {
+  if (a === b) return true;
+  // Compare through BlockList so "::1" and "0:0:0:0:0:0:0:1" are the same address.
+  try {
+    const bl = new BlockList();
+    bl.addAddress(b, isIP(b) === 6 ? 'ipv6' : 'ipv4');
+    return bl.check(a, isIP(a) === 6 ? 'ipv6' : 'ipv4');
   } catch {
     return false;
   }
@@ -127,37 +156,56 @@ function ruleMatches(pattern, { hostname, addresses, port }) {
 
 /**
  * @param {URL} url
- * @param {{ allow?: string[], deny?: string[], lookup?: typeof dnsLookup }} [options]
- * @returns {Promise<{ allowed: boolean, reason?: string }>}
+ * @param {{ allow?: string[], deny?: string[], lookup?: typeof dnsLookup, signal?: AbortSignal }} [options]
+ * @returns {Promise<{ allowed: boolean, reason?: string, addresses?: Array<{ address: string, family: number }> }>}
+ *   `addresses` (when allowed) are the validated addresses to connect to.
  */
-export async function checkUrlAllowed(url, { allow = [], deny = [], lookup = dnsLookup } = {}) {
+export async function checkUrlAllowed(url, { allow = [], deny = [], lookup = dnsLookup, signal } = {}) {
   if (url.protocol !== 'http:' && url.protocol !== 'https:') {
     return { allowed: false, reason: `Unsupported protocol: ${url.protocol}` };
   }
+  if (url.username || url.password) {
+    return { allowed: false, reason: 'Credentials in the URL are not supported' };
+  }
 
   // WHATWG URL keeps brackets around an IPv6 literal in .hostname ("[::1]").
-  const hostname = url.hostname.replace(/^\[|\]$/g, '').toLowerCase();
+  // A trailing dot ("localhost.") names the same host, so it is dropped for matching.
+  const hostname = url.hostname.replace(/^\[|\]$/g, '').replace(/\.$/, '').toLowerCase();
   const port = Number(url.port) || (url.protocol === 'https:' ? 443 : 80);
 
   let addresses;
   try {
-    addresses = await lookup(hostname, { all: true, verbatim: true });
-  } catch {
+    addresses = await raceAbort(lookup(hostname, { all: true, verbatim: true }), signal);
+  } catch (err) {
+    if (signal?.aborted) throw err;
     addresses = [];
   }
+  if (!Array.isArray(addresses)) addresses = [addresses];
   if (addresses.length === 0) {
     return { allowed: false, reason: `Could not resolve host: ${hostname}` };
   }
 
-  const match = { hostname, addresses, port };
-  if (deny.some((pattern) => ruleMatches(pattern, match))) {
+  const match = { hostname, port };
+  if (deny.some((pattern) => addresses.some((a) => ruleMatchesAddress(pattern, match, a)))) {
     return { allowed: false, reason: 'Blocked by network policy' };
   }
-  if (allow.some((pattern) => ruleMatches(pattern, match))) {
-    return { allowed: true };
-  }
-  if (addresses.some((a) => isDefaultRestricted(a.address, a.family))) {
-    return { allowed: false, reason: 'Blocked: target resolves to a private/internal/reserved address' };
+  // Every address must be allowed by a rule or be public: pinning connects to any of them.
+  for (const a of addresses) {
+    if (allow.some((pattern) => ruleMatchesAddress(pattern, match, a))) continue;
+    if (isDefaultRestricted(a.address, a.family)) {
+      return { allowed: false, reason: 'Blocked: target resolves to a private/internal/reserved address' };
+    }
   }
-  return { allowed: true };
+  return { allowed: true, addresses };
+}
+
+/** Reject as soon as `signal` aborts, so a hung DNS lookup cannot outlive the request timeout. */
+function raceAbort(promise, signal) {
+  if (!signal) return promise;
+  if (signal.aborted) return Promise.reject(signal.reason);
+  return new Promise((resolve, reject) => {
+    const onAbort = () => reject(signal.reason);
+    signal.addEventListener('abort', onAbort, { once: true });
+    promise.then(resolve, reject).finally(() => signal.removeEventListener('abort', onAbort));
+  });
 }

package/src/request.js

@@ -33,16 +33,18 @@ export function buildAuthHeaders(auth, variables) {
  * Path, query, headers and body are interpolated with `variables`.
  *
  * Variable values are put into the path as given, so a value containing `/`,
- * `?` or `#` changes the URL. Pass untrusted values through `encodeURIComponent`
- * yourself, or write `{{name}}` into a query parameter, which is encoded for you.
+ * `?` or `#` changes the URL. For untrusted values set `encodePathVariables`
+ * (each value goes through `encodeURIComponent`), or write `{{name}}` into a
+ * query parameter, which is always encoded.
  *
  * @param {import('./index.js').Connector} connector
  * @param {import('./index.js').RequestDef} request
  * @param {Record<string, unknown>} [variables]
+ * @param {{ encodePathVariables?: boolean }} [options]
  */
-export function buildRequest(connector, request, variables = {}) {
+export function buildRequest(connector, request, variables = {}, { encodePathVariables = false } = {}) {
   const base = connector.baseUrl.replace(/\/+$/, '');
-  const path = interpolate(request.path || '', variables);
+  const path = interpolate(request.path || '', variables, encodePathVariables ? encodeURIComponent : undefined);
   const url = new URL(base + (path.startsWith('/') ? path : `/${path}`));
   for (const { key, value } of interpolatePairs(request.query, variables)) {
     if (key) url.searchParams.append(key, value ?? '');

package/src/transport.js

@@ -0,0 +1,153 @@
+/**
+ * HTTP transport that connects only to addresses the SSRF guard has validated.
+ *
+ * `fetch` resolves the host name itself, so a check done before the call can be
+ * defeated by DNS rebinding: the name answers with a public address for the
+ * check and a private one for the connection. `pinnedFetch` closes that gap by
+ * giving node:http(s) a `lookup` that returns the already validated addresses.
+ * The URL, the `Host` header, the TLS server name (SNI) and certificate
+ * verification still use the original host name, so nothing else changes.
+ *
+ * It returns a standard `Response`. Proxies from the environment are not used
+ * (neither does Node's built-in fetch).
+ */
+import http from 'node:http';
+import https from 'node:https';
+import zlib from 'node:zlib';
+
+const NULL_BODY_STATUS = new Set([101, 204, 205, 304]);
+
+/**
+ * @param {URL} url
+ * @param {{ method?: string, headers?: Record<string, string>, body?: string, signal?: AbortSignal }} init
+ * @param {{ addresses: Array<{ address: string, family: number }> }} pin validated addresses to connect to
+ * @returns {Promise<Response>}
+ */
+export function pinnedFetch(url, { method = 'GET', headers = {}, body, signal } = {}, { addresses }) {
+  if (!addresses?.length) return Promise.reject(new Error('No validated address to connect to'));
+  const lib = url.protocol === 'https:' ? https : http;
+
+  const lookup = (_hostname, options, callback) => {
+    if (typeof options === 'function') callback = options;
+    const wanted = typeof options === 'object' && options?.family ? Number(options.family) : 0;
+    const usable = addresses.filter((a) => !wanted || a.family === wanted);
+    if (usable.length === 0) {
+      callback(Object.assign(new Error('No validated address for the requested IP family'), { code: 'ENOTFOUND' }));
+    } else if (typeof options === 'object' && options?.all) {
+      callback(null, usable.map(({ address, family }) => ({ address, family })));
+    } else {
+      callback(null, usable[0].address, usable[0].family);
+    }
+  };
+
+  const requestHeaders = { 'accept-encoding': 'identity', ...lowerCaseKeys(headers) };
+  if (body !== undefined) requestHeaders['content-length'] = String(Buffer.byteLength(body));
+
+  return new Promise((resolve, reject) => {
+    const hostname = url.hostname.replace(/^\[|\]$/g, '');
+    const req = lib.request(
+      {
+        protocol: url.protocol,
+        hostname,
+        port: url.port || undefined,
+        path: `${url.pathname}${url.search}`,
+        method,
+        headers: requestHeaders,
+        lookup,
+        agent: false, // no pooling: a reused socket may be pinned to another validation
+        signal,
+      },
+      (res) => {
+        try {
+          resolve(toResponse(res));
+        } catch (err) {
+          res.destroy();
+          reject(err);
+        }
+      },
+    );
+    req.on('error', reject);
+    req.end(body);
+  });
+}
+
+function lowerCaseKeys(headers) {
+  return Object.fromEntries(Object.entries(headers).map(([name, value]) => [name.toLowerCase(), value]));
+}
+
+function toResponse(res) {
+  const responseHeaders = new Headers();
+  for (let i = 0; i < res.rawHeaders.length; i += 2) {
+    try {
+      responseHeaders.append(res.rawHeaders[i], res.rawHeaders[i + 1]);
+    } catch {
+      // A header the Headers class refuses is not needed by callers.
+    }
+  }
+
+  let stream = res;
+  const encoding = (res.headers['content-encoding'] || '').toLowerCase().trim();
+  if (encoding && encoding !== 'identity') {
+    const decoder = decoderFor(encoding);
+    if (!decoder) throw new Error(`Unsupported content-encoding: ${encoding}`);
+    stream = res.pipe(decoder);
+    res.on('error', (err) => decoder.destroy(err));
+    responseHeaders.delete('content-encoding');
+    responseHeaders.delete('content-length'); // the compressed length no longer applies
+  }
+
+  const status = res.statusCode;
+  if (NULL_BODY_STATUS.has(status)) {
+    stream.resume();
+    return new Response(null, { status, headers: responseHeaders });
+  }
+  return new Response(toWebStream(stream), { status, headers: responseHeaders });
+}
+
+function decoderFor(encoding) {
+  switch (encoding) {
+    case 'gzip':
+    case 'x-gzip':
+      return zlib.createGunzip();
+    case 'deflate':
+      return zlib.createInflate();
+    case 'br':
+      return zlib.createBrotliDecompress();
+    default:
+      return null;
+  }
+}
+
+/**
+ * Node stream -> web ReadableStream with backpressure. Readable.toWeb is avoided on purpose:
+ * on Node 18 it closes the controller twice for a response that ends before it is read.
+ */
+function toWebStream(stream) {
+  let finished = false;
+  return new ReadableStream({
+    start(controller) {
+      stream.on('data', (chunk) => {
+        if (finished) return;
+        controller.enqueue(new Uint8Array(chunk.buffer, chunk.byteOffset, chunk.byteLength));
+        if (controller.desiredSize <= 0) stream.pause();
+      });
+      stream.on('end', () => {
+        if (finished) return;
+        finished = true;
+        controller.close();
+      });
+      stream.on('error', (err) => {
+        if (finished) return;
+        finished = true;
+        controller.error(err);
+      });
+    },
+    pull() {
+      stream.resume();
+    },
+    cancel() {
+      finished = true;
+      stream.destroy();
+    },
+  });
+}