@allus-fyi/company-data 0.0.3

package/docs/model.md

@@ -0,0 +1,141 @@
+# Output model reference
+
+The conclusions — the only objects you work with. Importable from
+`@allus/company-data`. Each carries `.raw` (the underlying hardened API object;
+never contains the person's source field).
+
+## `RequestField`
+
+Your request-field **definition** — your config, never the person's fields.
+Returned by `client.requestFields()`.
+
+```ts
+class RequestField {
+  slug: string;        // the stable, company-set key — the contract for value access
+  label: string;       // the human label (rename freely; the slug stays)
+  type: string;        // email|phone|url|text|address|bank|creditcard|date|date_of_birth|photo|document|legal_document
+  oneTime: boolean;    // a one-time snapshot vs a live (auto-updating) answer
+  mandatory: boolean;  // mandatory-to-provide OR mandatory-to-stay-connected (the API's two flags, folded)
+  raw: Record<string, unknown>;
+}
+```
+
+## `Connection`
+
+A connected person — identity + the slug-keyed value map. No source field
+anywhere; `values` is keyed by **your** request slug.
+
+```ts
+class Connection {
+  id: string;
+  personId: string;
+  displayName: string | null;      // null on connection(id) (the list endpoint carries it)
+  connectedAt: Date | null;        // likewise null on connection(id)
+  values: Record<string, Value>;   // {<your_slug>: Value}
+  raw: Record<string, unknown>;
+}
+```
+
+```ts
+conn.values['work_email'].value     // "alice@acme.com"
+conn.values['mobile']                // undefined if the person didn't answer that slot
+```
+
+## `Value`
+
+One answer for one of your request slots.
+
+```ts
+class Value {
+  value: unknown;            // typed plaintext (see below)
+  live: boolean;             // true = "keep connected" (auto-updates); false = one-time snapshot
+  updatedAt: Date | null;    // when this answer last changed
+  raw: Record<string, unknown>;
+}
+```
+
+### `value` types (resolved from the field's `type`)
+
+| Field type | JS `value` | Notes |
+|------------|------------|-------|
+| `email`, `phone`, `url`, `text` | `string` | The decrypted plaintext. |
+| `address`, `bank`, `creditcard` | `object` | The decrypted plaintext is a JSON object → parsed. A non-JSON structured value throws `DecryptError`. |
+| `date`, `date_of_birth` | `Date` | Parsed from ISO `YYYY-MM-DD` (UTC midnight, the leading 10 chars); falls back to the raw string if unparseable. |
+| `photo`, `document`, `legal_document` | `BinaryHandle` | Lazy — nothing fetched/decrypted until `.bytes()`/`.save()`. |
+| unanswered / no value | `null` | The slot has no answer. |
+
+## `BinaryHandle`
+
+A lazy handle for a binary value. No network or decryption happens at construction.
+
+```ts
+class BinaryHandle {
+  get valueUrl(): string | null;        // the opaque slot-keyed file URL (read-only)
+  bytes(): Promise<Buffer>;             // fetch (if needed) → decrypt → decoded primary file bytes
+  save(path: string): Promise<number>;  // write bytes() to path; resolves to bytes written
+  static parseEnvelopeBytes(envelopeJson: string): Buffer;  // envelope string → file bytes
+}
+```
+
+On first `.bytes()`/`.save()`:
+
+1. GET the slot-keyed file endpoint → the API serves `{"encrypted": true, "value": <wrapper>}`.
+2. Decrypt the inner `{"_enc":1,…}` wrapper with the service key → a JSON file-envelope string (`{"full": "data:…", "thumb": …}` for photos, `{"file": "data:…", …}` for documents).
+3. Base64-decode the primary data URI (`full` for photos, `file` for documents) → a `Buffer`. Cached on the handle (repeated calls don't re-fetch).
+
+`.save()` is crash-safe (temp file → fsync → atomic rename — never a truncated
+output). An unanswered binary slot yields an empty handle; calling `.bytes()` on it
+throws `DecryptError`.
+
+## `Change`
+
+A change-feed / webhook event. Returned by the pump (`processChanges`, `drainBatch`)
+and the webhook helpers.
+
+```ts
+class Change {
+  id: string;                 // the stable server change-row id — YOUR dedup key
+  event: string;              // see the event table
+  personId: string | null;
+  shareCode: string | null;   // the person's profile share code (every event; may be null)
+  slug: string | null;        // field_updated/field_deleted/consent_* only
+  value: unknown;             // field_updated only; typed exactly like Value.value
+  live: boolean | null;       // field_updated only
+  at: Date | null;            // the change time (no separate updatedAt on a change)
+  raw: Record<string, unknown>;
+}
+```
+
+### Events
+
+| `event` | Carries |
+|---------|---------|
+| `connection_created` | identity only (no slot/value) |
+| `connection_deleted` | identity only (no slot/value) |
+| `field_updated` | `slug` + decrypted `value` (+ `live`); binary → a lazy `BinaryHandle` |
+| `field_deleted` | `slug`, no value |
+| `consent_accepted` / `consent_declined` | `slug` |
+
+`Change.id` is captured before the server's drain-delete, so it survives a crash +
+replay unchanged — dedup on it.
+
+## `LogEntry`
+
+A service activity-log entry — ops events only (email / purge / webhook), never
+person field data.
+
+```ts
+class LogEntry {
+  type: string;
+  message: string | null;
+  metadata: unknown;
+  at: Date | null;
+  raw: Record<string, unknown>;
+}
+```
+
+## `.raw`
+
+Every model has a `.raw` property: the underlying (hardened) API object, for
+debugging or an edge case the SDK didn't model. It never contains the person's
+source field — the hardened API doesn't return it.

package/docs/pump.md

@@ -0,0 +1,130 @@
+# The changes pump
+
+The changes feed is a server-side **drain-on-fetch queue**:
+`GET /api/company-data/changes?limit=N` returns up to N events (default 100, max
+500) **and deletes exactly those rows in the same transaction**. There is no
+offset/cursor/page, and the API keeps no copy after a fetch. So a consumer must:
+
+* not lose a drained batch if it crashes mid-batch (the API already deleted it), and
+* not materialize a huge backlog in memory.
+
+`client.processChanges(handler)` (delegating to the `Pump`) does both.
+
+## `processChanges(handler, options?)`
+
+```ts
+processChanges(
+  handler: (change: Change) => void | Promise<void>,
+  options?: {
+    batchSize?: number;                       // clamped to [1, 500]; default 100
+    maxRetries?: number;                      // default 3
+    onError?: 'deadletter' | 'halt';          // default 'deadletter'
+    backoff?: (attempt: number) => number;    // attempt(1-based) -> seconds
+  },
+): Promise<void>
+```
+
+Drains the feed through `handler` one `Change` at a time, **until the feed is empty,
+then resolves**. No follow/daemon mode — schedule re-runs yourself. The handler may
+be sync or async.
+
+## The cycle
+
+1. **Replay first** — deliver any un-acked events already in the local buffer (a previous crashed run), oldest-first.
+2. **Drain** — when the buffer is empty, fetch one batch (≤ `batchSize`, ≤ 500) and **persist it to the durable buffer (fsync) BEFORE handing anything out**.
+3. **Deliver one-by-one** — for each buffered event, oldest-first: decrypt its value *at delivery* (never on disk), build the typed `Change`, call `handler(change)`.
+4. **Ack / retry / dead-letter** — on handler success, remove the event from the buffer (ack). On a handler error, retry with `backoff` up to `maxRetries`; then:
+   * `onError="deadletter"` (default) → move it to the dead-letter store, log it, and continue (one poison event never wedges the stream);
+   * `onError="halt"` → re-throw the handler's error (the event stays un-acked in the buffer for the next run).
+   A **`DecryptError`** (corrupt/truncated ciphertext, rotated key) is special: the decrypt runs *inside* the delivery attempt, and an undecryptable event is **dead-lettered immediately** — re-decrypting can't fix it, so it does **not** burn `maxRetries`. Under `onError="halt"` it re-throws like a handler error. Either way it never propagates out of `processChanges` and wedges step-1 replay.
+5. Repeat until a drain returns empty **and** the buffer is drained → resolve.
+
+## Crash safety · at-least-once · idempotency
+
+A batch is durably buffered *before* any delivery, and acked per-item only *after*
+the handler succeeds. A crash between a handler's success and its ack re-delivers
+that event on the next run. Delivery is therefore **at-least-once**:
+
+> **Your handler must be idempotent. Dedup on `Change.id`** (the stable server
+> change-row id, captured before the server delete).
+
+## The durable buffer (on disk)
+
+Under `cacheDir`:
+
+```
+<cacheDir>/pending/<seq>_<change_id>.json      # un-acked events, oldest-first
+<cacheDir>/deadletter/<seq>_<change_id>.json   # events that exhausted retries
+```
+
+* Stored events keep their **ciphertext** `value`/`value_url` — **no plaintext PII is ever written to disk**. Decryption happens only at delivery.
+* `<seq>` is a zero-padded, monotonically increasing sequence, so lexicographic filename order == oldest-first (stable even if `at` timestamps are equal/missing).
+* Writes are crash-safe: temp file → `fsyncSync` → atomic rename → dir fsync. A crash never leaves a half-written file.
+* Re-instantiating the buffer on the same `cacheDir` recovers whatever is on disk — that recovery **is** the replay-on-restart.
+
+## Options
+
+| Option | Default | Meaning |
+|--------|---------|---------|
+| `batchSize` | 100 | Events per drain; clamped to `[1, 500]`. |
+| `maxRetries` | 3 | Handler retries before dead-letter/halt. |
+| `onError` | `"deadletter"` | `"deadletter"` (continue) or `"halt"` (re-throw). Any other value throws `TypeError`. |
+| `backoff` | exponential, capped 30s | `attempt -> seconds` between retries. |
+
+> `logger` is **not** a `processChanges` option — pass it to the `Client`
+> constructor (`Client.fromConfig('allus.json', { logger: myLogger })`). Every
+> drain, deliver, ack, retry, dead-letter, and replay is logged (any
+> console-compatible sink with `debug`/`info`/`warn`/`error`).
+
+## Durability guarantees
+
+Held by the pump across all six SDKs, validated by the test suite:
+
+1. **Decrypt inside the delivery attempt** — a poison/undecryptable event is dead-lettered immediately, never wedges replay, and does not burn retries.
+2. **A re-failing dead-letter is updated in place** within `deadletter/` (atomic temp+fsync+rename) — never routed back through `pending/` (which has a crash window that could resurrect a dead-letter as a live event).
+3. **Stored attempt count is monotonic** across separate retry runs (`max(existing, new)`).
+4. **At-least-once dead-lettering** — the new dead-letter copy is written *before* the pending copy is unlinked, so a crash between leaves the event in both dirs (harmless re-delivery, absorbed by the id-dedup handler). This is intentional — do not "fix" it by deleting-first.
+
+## No follow mode — schedule re-runs
+
+```ts
+const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
+for (;;) {
+  await client.processChanges(handle);   // resolves when the feed empties
+  await sleep(5000);                      // the feed is cheap to poll (see rate limits)
+}
+```
+
+A cron job, a worker loop, or any scheduler works equally well.
+
+## Dead-letter inspect / re-drive
+
+```ts
+client.deadLetters(): DeadLetterRecord[]
+client.retryDeadLetters(handler, options?): Promise<number>
+```
+
+* `deadLetters()` — each record is the stored (ciphertext) event with a flattened `error` and `attempts`, plus its `id`.
+* `retryDeadLetters(handler)` — re-drives every dead-lettered event through `handler`. On success the record is removed. On repeated failure (or a `DecryptError`) the dead-letter record is **updated in place** with the new error + attempt count and stays in `deadletter/` (`"deadletter"`), or the error re-throws (`"halt"`). Resolves to the count successfully re-driven.
+
+A re-failing dead-letter never re-enters `pending/` — it is rewritten in place
+within `deadletter/`, so a crash mid-re-drive can't resurrect it as a live event on
+the next run. Dead letters are **never silently dropped** and **never re-fetched
+from the API** (it already deleted them) — the local store is their only home,
+which is exactly why it's durable.
+
+```ts
+for (const dl of client.deadLetters()) console.log(dl.id, dl.error, dl.attempts);
+const fixed = await client.retryDeadLetters(handle);   // after fixing the handler bug
+```
+
+## Advanced: `drainBatch(max?)`
+
+```ts
+client.drainBatch(max?: number): Promise<Change[]>
+```
+
+A raw, **UNBUFFERED** drain: fetches one batch (clamped ≤ 500) and returns the
+decrypted `Change`s directly — it does **not** persist anything to the buffer, so
+**you own durability** if you use it (a crash loses what the API already deleted).
+Prefer `processChanges` for safe consumption.

package/docs/webhooks.md

@@ -0,0 +1,140 @@
+# Webhook receiver helpers
+
+The lower-latency push alternative to polling the changes feed. The platform POSTs
+each change event to your configured webhook URL with:
+
+* `X-Allus-Webhook-Id` — which webhook this is (selects the HMAC secret from config).
+* `X-Allus-Signature` — `HMAC-SHA256(rawBody, secret)` as lowercase hex.
+* the body — the same slug-keyed `Change` shape as the pull feed (JSON or XML). If `encrypt_payload` is on, the body is replaced by a `{"_enc":1,…}` envelope encrypted to the company **account** key (and the HMAC is over that envelope).
+
+**All secrets/keys come from config — these helpers take NO key or secret
+arguments.** Always pass the **raw request body bytes** (a `Buffer`); don't
+re-serialize a parsed body — the HMAC is over the exact bytes sent.
+
+## Client methods (the usual form)
+
+```ts
+client.verifyWebhook(rawBody: Buffer | Uint8Array | string, headers): boolean
+client.parseWebhook(rawBody, headers):  Change
+client.handleWebhook(rawBody, headers): Change   // verify + parse
+```
+
+| Method | Returns | Errors |
+|--------|---------|--------|
+| `verifyWebhook` | `boolean` — recomputes `HMAC-SHA256(rawBody, secret)` and constant-time-compares to `X-Allus-Signature`. `false` on missing signature / unknown id / mismatch. | **Never throws** for a bad signature. |
+| `parseWebhook` | a typed `Change`. Does **not** verify. Handles JSON, XML, and the `encrypt_payload` account-key envelope. | `WebhookError` on a malformed/unparseable body or envelope. |
+| `handleWebhook` | a typed `Change` — verify **then** parse. | `WebhookError` on a bad/unknown signature, or any `parseWebhook` error. |
+
+> The client webhook methods are **synchronous** but need the request-fields catalog
+> (to type the value). Call `await client.requestFields()` once at startup so the
+> catalog is cached — the methods then make no network calls. `headers` may be a
+> plain object or a Node `IncomingHttpHeaders` (case-insensitive lookup; array
+> header values use the first element).
+
+## Standalone functions
+
+The same three are importable as module functions. They take the `config` and the
+decrypt/type closures explicitly — used by `Client` internally; you'll normally use
+the client methods inside an app.
+
+```ts
+import { verifyWebhook, parseWebhook, handleWebhook } from '@allus/company-data';
+
+verifyWebhook(rawBody, headers, config): boolean
+parseWebhook(rawBody, headers, config, { typeForSlug, decryptValue, binaryFetch?, accountKey? }): Change
+handleWebhook(rawBody, headers, config, { typeForSlug, decryptValue, binaryFetch?, accountKey? }): Change
+```
+
+## In a web route
+
+### Express
+
+```ts
+import express from 'express';
+import { Client, WebhookError } from '@allus/company-data';
+
+const app = express();
+const client = Client.fromConfig('allus.json');
+await client.requestFields();   // warm the catalog (the webhook methods are sync)
+
+// Capture the RAW body bytes — do NOT let a JSON parser replace them.
+app.post('/allus/webhook', express.raw({ type: '*/*' }), (req, res) => {
+  let change;
+  try {
+    change = client.handleWebhook(req.body, req.headers);
+  } catch (e) {
+    if (e instanceof WebhookError) return res.sendStatus(401);
+    throw e;
+  }
+  if (!seen(change.id)) {          // idempotency — same rule as the pump
+    applyChange(change);
+    recordSeen(change.id);
+  }
+  res.sendStatus(204);
+});
+```
+
+### Fastify
+
+```ts
+import Fastify from 'fastify';
+import { Client, WebhookError } from '@allus/company-data';
+
+const app = Fastify();
+const client = Client.fromConfig('allus.json');
+await client.requestFields();
+
+// Keep the raw body: a contentTypeParser that returns the Buffer untouched.
+app.addContentTypeParser('*', { parseAs: 'buffer' }, (_req, body, done) => done(null, body));
+
+app.post('/allus/webhook', (req, reply) => {
+  let change;
+  try {
+    change = client.handleWebhook(req.body as Buffer, req.headers);
+  } catch (e) {
+    if (e instanceof WebhookError) return reply.code(401).send();
+    throw e;
+  }
+  if (!seen(change.id)) { applyChange(change); recordSeen(change.id); }
+  return reply.code(204).send();
+});
+```
+
+Split the steps if you prefer:
+
+```ts
+if (!client.verifyWebhook(rawBody, headers)) return res.sendStatus(401);
+const change = client.parseWebhook(rawBody, headers);
+```
+
+## Config-driven secrets
+
+Per-webhook HMAC secrets live in the config `webhooks` map, keyed by webhook id;
+the SDK reads `X-Allus-Webhook-Id` and looks up the matching secret. A
+single-webhook service can use the flat `"webhook_secret": "…"` shortcut (or
+`ALLUS_WEBHOOK_SECRET`). An unknown/unconfigured id ⇒ `verifyWebhook` returns
+`false` (and `handleWebhook` throws `WebhookError`).
+
+## The `encrypt_payload` account-key envelope
+
+If a webhook has `encrypt_payload` enabled, the whole body is a `{"_enc":1,…}`
+envelope encrypted to your company **account** key, and the HMAC is over that
+envelope. `parseWebhook`/`handleWebhook`:
+
+1. Unwrap the envelope with the configured `account_private_key` + `account_passphrase` (loaded once at `Client` construction — no per-webhook PBKDF2).
+2. Parse the inner payload (JSON or XML per `format`).
+3. Decrypt the inner field `value` (a service-key wrapper) with the service key.
+
+So an `encrypt_payload` `Change` is identical to a plain one. Receiving such a
+webhook without an `account_private_key` configured throws `WebhookError`.
+
+> The envelope uses RSA-OAEP-**SHA1** (OpenSSL's default), distinct from the
+> OAEP-SHA256 used for person field values. The SDK has two OAEP code paths and
+> handles this difference internally — you only supply the account key in config.
+
+## XXE safety
+
+XML webhook bodies (and the inner payload after an envelope unwrap) are parsed by
+the same **XXE-safe** parser the HTTP layer uses: no DOCTYPE/DTD processing, no
+custom or external entities. The HMAC is always computed over the **raw bytes**,
+never the parsed tree.

package/package.json

@@ -0,0 +1,54 @@
+{
+  "name": "@allus-fyi/company-data",
+  "version": "0.0.3",
+  "description": "TypeScript/Node SDK for the allus company-data API: typed, plaintext, slug-keyed conclusions with transparent decryption.",
+  "license": "MIT",
+  "author": "allme.fyi",
+  "keywords": [
+    "allus",
+    "allme",
+    "company-data",
+    "sdk"
+  ],
+  "homepage": "https://github.com/allus-fyi/company-data-typescript#readme",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/allus-fyi/company-data-typescript.git"
+  },
+  "bugs": {
+    "url": "https://github.com/allus-fyi/company-data-typescript/issues"
+  },
+  "type": "module",
+  "engines": {
+    "node": ">=18"
+  },
+  "exports": {
+    ".": {
+      "types": "./dist/types/index.d.ts",
+      "import": "./dist/esm/index.js",
+      "require": "./dist/cjs/index.js"
+    }
+  },
+  "main": "./dist/cjs/index.js",
+  "module": "./dist/esm/index.js",
+  "types": "./dist/types/index.d.ts",
+  "files": [
+    "dist",
+    "README.md",
+    "docs"
+  ],
+  "scripts": {
+    "clean": "rimraf dist",
+    "build": "npm run clean && npm run build:esm && npm run build:cjs && npm run build:types && node scripts/fixup-cjs.mjs",
+    "build:esm": "tsc -p tsconfig.esm.json",
+    "build:cjs": "tsc -p tsconfig.cjs.json",
+    "build:types": "tsc -p tsconfig.types.json",
+    "test": "node --test --import tsx test/*.test.ts"
+  },
+  "devDependencies": {
+    "@types/node": "^22.0.0",
+    "rimraf": "^5.0.0",
+    "tsx": "^4.19.0",
+    "typescript": "^5.6.0"
+  }
+}