@marktiderman/genesis-forms 0.6.0 → 0.8.0

package/README.md

@@ -83,7 +83,7 @@ const version = eventFeedback.hash();         // content fingerprint for publish
 | `isFormEvent(value)` | Guard for one well-formed event-log entry. |
 | `isSensitive(block)` | `sensitive: true` on a field, or the `props.sensitive` alias. Hidden inputs take `sensitive: true` too (`f.hidden.string({ source: "server", sensitive: true })`); devtools masks both. |
 | `/server` → `validateSubmission`, `publicDefinition` | Re-check a submission against the published definition: drops unknown and logic-hidden answers, ignores client values for server inputs, checks drafts by format only. Never throws on a malformed body. No IO. |
-| `/destinations` → `emailDestination` + `resendProvider`, `sheetsDestination`, `webhookDestination`, `responseRow` | Send a checked submission to an inbox, a Google Sheet or a signed webhook. fetch and WebCrypto only. See [Destinations](#destinations). |
+| `/destinations` → `emailDestination` + `resendProvider`, `sheetsDestination`, `webhookDestination`, `slackDestination`, `responseRow`, `createDeliveryRunner` | Send a checked submission to an inbox, a Google Sheet, Slack or a signed webhook, and run an app's outbox of deliveries. fetch and WebCrypto only. See [Destinations](#destinations). |
 | `explain(engine, blockId)` | "Why is this hidden?": the path, the jump that skipped a block, each show/hide rule with its result in plain English, and the field's validation. |
 | `/testing` → `simulate`, `coverage`, `runScenario`, `recordScenario`, `compactEvents` | Walk a form by key, enumerate its paths, and run JSON scenarios. See [Testing](#testing). |
 
@@ -143,7 +143,7 @@ A `photo` field asks for up to a few photos, such as a picture of broken equipme
 returned, never the bytes: `PhotoRef[]`, each `{ id, name, type, size }`. The renderer uploads
 through the host (`onUpload` on genesis-forms-web's `Form`), so the engine and `validateSubmission`
 stay storage-agnostic; the host's server should check that every reference is a file it stored
-for this respondent and form before saving.
+for this respondent and form before saving (`checkUploads`, see [Uploads](#uploads)).
 
 | Prop | Default | Rule |
 | --- | --- | --- |
@@ -268,9 +268,9 @@ An option can carry more than its value and label, from the question's own `opti
 app's list:
 
 ```ts
-{ value: "cff", label: "CrossFit Fairfield",
+{ value: "riverside", label: "Riverside Gym",
   description: "12 Main St",                    // a muted line under the label
-  image: "https://cdn.example.com/cff.png",      // a small picture; https only
+  image: "https://cdn.example.com/riverside.png",      // a small picture; https only
   attributes: { region: "north", coaches: ["ada", "sam"] } } // for filters and recall
 ```
 
@@ -288,7 +288,7 @@ f.singleChoice("coach", {
   label: "Who coached you?",
   optionsFrom: { source: "host", key: "coaches", filter: { attribute: "location", field: "location" } },
 });
-// a coach: { value: "ada", label: "Ada", attributes: { location: ["cff", "cornerstone"] } }
+// a coach: { value: "ada", label: "Ada", attributes: { location: ["riverside", "hilltop"] } }
 ```
 
 `field` is the earlier question's key (`parseForm` checks it exists and comes first). An option
@@ -307,7 +307,7 @@ options: `props.searchable` when it is true or false, else more than `SEARCHABLE
 query in the label, value or description, ignoring case and accents.
 
 **Preselect from a link.** `props.prefill: { param }` on a choice question takes a link's query
-parameter, such as `?location=cornerstone`:
+parameter, such as `?location=hilltop`:
 
 ```ts
 f.dropdown("location", { label: "Where?", optionsFrom: { source: "host", key: "locations" },
@@ -335,7 +335,7 @@ import { answerSnapshots, answersFromSnapshots, withHostOptions } from "@marktid
 
 const shown = withHostOptions(definition, lists);
 const snapshots = answerSnapshots(shown, result.submission.answers);
-// { location: { value: "cff", label: "CrossFit Fairfield" }, helpers: [{ value: "ada", label: "Ada" }] }
+// { location: { value: "riverside", label: "Riverside Gym" }, helpers: [{ value: "ada", label: "Ada" }] }
 
 // An edit link: find each pick again in today's lists.
 const { answers, missing } = answersFromSnapshots(withHostOptions(definition, listsNow), stored);
@@ -430,6 +430,64 @@ const key = await rateLimitKey({ kind: "submit", scope: practiceId, ip: clientIp
 if (!(await allow(store, { key, ...RATE_LIMIT_DEFAULTS.submit }, { log }))) return tooMany();
 ```
 
+### Uploads
+
+A `photo` answer holds references, never bytes, so the server must check each one is a file it
+stored for this respondent. Two pieces do it, with the app's database and storage behind an
+`UploadStore` (`beginUpload`, `signUpload`, `uploadRefs`):
+
+- `createUploadHandler({ resolveForm, store, edit?, rateLimit?, ... })` is the endpoint the browser
+  asks for one signed upload per file (`(request, params) => Response`). It checks the form is open,
+  the field takes files and the file's type and size, the visit's attempt id or the edit link
+  (`edit.response` loads the response, the handler compares its nonce), and the rate limit
+  (`kind: "upload"`, 120 per 10 minutes by default), then records the upload and signs it.
+- `checkUploads(definition, answers, { formId, owner, store })`, on submit after
+  `validateSubmission`: every reference must be an upload this attempt (`{ attemptHash }`) or this
+  edit (`{ responseId }`: the response's own files, or ones uploaded for the edit) made, for that
+  field, finished, of a type and size the field takes, and no more than it takes. The stored name,
+  type and size replace what the browser sent; a refusal is keyed by field id.
+
+```ts
+import { attemptHash, checkUploads, createUploadHandler } from "@marktiderman/genesis-forms/server";
+
+export const POST = createUploadHandler<{ slug: string }>({
+  resolveForm: async (request, { slug }) => {
+    const found = await loadForm(request, slug); // null: 404
+    return found && { id: found.id, definition: found.definition, open: found.open, scope: found.practiceId, allowsEdits: found.allowEdit };
+  },
+  store: uploadStore, // beginUpload, signUpload, uploadRefs
+  edit: { secret: editSecret, response: (id, form) => loadSentResponse(id, form) }, // { id, editNonce } or null
+  rateLimit: { store: counter, secret: rateLimitSecret },
+});
+
+// On submit:
+const checked = await checkUploads(definition, result.submission.answers, {
+  formId,
+  owner: editing ? { responseId } : { attemptHash: await attemptHash(body.attemptId) },
+  store: uploadStore,
+}); // throws when the store can't read: answer 503
+if (!checked.ok) return Response.json({ errors: checked.errors }, { status: 422 });
+```
+
+The JSON (`UploadRequest`, `UploadGrant`, `UploadRefusal` in the main entry, for the browser; every
+response is `cache-control: no-store`):
+
+| Status | Body | When |
+| --- | --- | --- |
+| 200 | `{ ref: { id, name, type, size }, upload: { path, token, bucket?, url? } }` | Recorded and signed; `upload` is what `signUpload` returned. |
+| 400 | `{ error }` | Not a JSON object; no `attemptId` (a UUID) without `edit`; the field isn't an upload field. |
+| 403 | `{ error, code: "bad_link" }` | An edit link that doesn't verify, isn't the response's newest, or a form without edits. |
+| 404 | `{ error }` | `resolveForm` found no form. |
+| 409 | `{ error }`, or `code: "closed"` for an edit | The form isn't open. |
+| 413 | `{ error }` | The request is over 4,000 bytes, or the file is empty, not whole bytes or over the field's size. |
+| 415 | `{ error }` | Not `application/json`, or a file type the field doesn't take. |
+| 429 | `{ error }`, or `code: "too_many_photos"` | The rate limit, or the store's `{ tooMany: true }`. |
+| 500 / 503 | `{ error }` | Recording or signing failed / the form or the edit's response couldn't be loaded; the cause is logged, never sent. |
+
+The request is `{ fieldId, name, type, size }` plus `attemptId` or `edit`. `attemptHash(attemptId)`
+is the hex SHA-256 the store keeps for the attempt. `messages` overrides any of the words
+(`UPLOAD_MESSAGES`). genesis-forms-web's `signedUploadHost` is the browser half.
+
 ## Destinations
 
 `@marktiderman/genesis-forms/destinations` sends a checked submission somewhere. No dependencies:
@@ -484,7 +542,10 @@ store and log: never a secret, an answer, contact details or the webhook URL. Co
 has a timeout (`timeoutMs`, default 10 s) and honours `options.signal`. `deliverBatch(items,
 config)` (the sheet) sends many in one request; all succeed or fail together. Write your own by
 implementing `Destination` and throwing `DeliveryError` inside; `deliveryFailure(error)` turns any
-error into a safe result.
+error into a safe result. A destination that deliberately sends nothing (no address, a cap reached)
+resolves `{ ok: true, notSent: reason }`. `options.update` marks the delivery of an edit, and
+`options.destinationId` is handed to option functions (a secret, an account, a URL, a token) as
+their second argument, `{ destinationId }`, so they can look up what is stored for it.
 
 **Rows.** `responseRow(submission, definition)` flattens a submission: `id`, `submitted_at`, the
 app's `meta` columns, then one column per field key in form order. Numbers stay numbers, choices
@@ -495,8 +556,94 @@ submission. On an anonymous form identifying fields get no column. `sensitive` f
 | Destination | Config | Notes |
 | --- | --- | --- |
 | `emailDestination({ provider, from, replyTo?, render?, sensitive? })` | `{ to: string \| string[], subject? }` | HTML and plain text from the definition (`renderEmail`), every value HTML-escaped. Subject, From, To and Reply-To are stripped of line breaks or refused, so no answer can add a header. At most 50 recipients. Sensitive fields masked by default. The idempotency key is the delivery id plus a hash of the recipients. `resendProvider({ apiKey })` is built in; any `EmailProvider` works. `render` and `emailBody(lines)` let an app write its own content and keep the escaping. |
-| `sheetsDestination({ account, escapeFormulas?, sensitive? })` | `{ spreadsheetId, sheetName? }` | Service-account JWT signed with WebCrypto (RS256), token cached per process. Creates the tab, writes and freezes the header, adds a column for a new key without moving others, appends with `valueInputOption=RAW`, so Sheets stores text exactly as typed and never runs it as a formula (`@team thanks` stays `@team thanks`). No `'` is added by default, since RAW would store it literally; `escapeFormulas: true` adds one (headers included) for a sheet that is exported and opened elsewhere. `escapeFormula(cell)` is the same opt-in guard for CSV you write yourself: text starting with `=`, `@`, tab or CR, or with `+`/`-` followed by anything but number punctuation, gets a leading `'`. Keeps the tab, header and last row in `state`, so a normal delivery is one request. On `redelivery`, skips ids already in the last 500 rows. A 403/404 says to share the sheet with the service account. |
-| `webhookDestination({ secret, allowInsecureUrls?, allowPrivateHosts?, sensitive? })` | `{ url }` | [Standard Webhooks](https://www.standardwebhooks.com): `webhook-id`, `webhook-timestamp`, `webhook-signature: v1,<HMAC-SHA256>`. The secret is base64 (at least 16 bytes decoded), usually `whsec_`-prefixed; as in every Standard Webhooks library the prefix is optional and the rest is always base64-decoded, so receivers verify with the same string. A secret that is not padded standard base64 (a raw passphrase, say) fails permanently rather than signing deliveries no receiver could verify. Generate one with `"whsec_" + btoa(String.fromCharCode(...crypto.getRandomValues(new Uint8Array(32))))`. Body `{ type: "form.submitted", timestamp, data: { form, submission, row } }`. https only, no URL credentials, no localhost or private, loopback or link-local address literals, redirects not followed. A hostname that resolves to a private address is not caught: if untrusted users pick URLs, send from a network that cannot reach internal services. Receivers check with `verifyWebhook({ secret, headers, body })` (constant-time, five-minute tolerance). |
+| `sheetsDestination({ account \| accessToken, escapeFormulas?, sensitive? })` | `{ spreadsheetId, sheetName? }` | Signs in with a service account (`account`) or with an access token the app supplies (`accessToken(config, { destinationId })`, for a sheet a person connected with Google sign-in: see `googleOAuthAccessToken`). Service-account JWT signed with WebCrypto (RS256), token cached per process. Creates the tab, writes and freezes the header, adds a column for a new key without moving others, appends with `valueInputOption=RAW`, so Sheets stores text exactly as typed and never runs it as a formula (`@team thanks` stays `@team thanks`). No `'` is added by default, since RAW would store it literally; `escapeFormulas: true` adds one (headers included) for a sheet that is exported and opened elsewhere. `escapeFormula(cell)` is the same opt-in guard for CSV you write yourself: text starting with `=`, `@`, tab or CR, or with `+`/`-` followed by anything but number punctuation, gets a leading `'`. Keeps the tab, header and last row in `state`, so a normal delivery is one request. On `redelivery`, skips ids already in the last 500 rows. On `update` (an edit), reads the id column and overwrites the response's row where it is, appending only when the row is gone. A 403/404 says to share the sheet with the service account (or, with `accessToken`, that it is no longer connected). |
+| `webhookDestination({ secret, url?, resolveHost?, allowInsecureUrls?, allowPrivateHosts?, sensitive? })` | `{ url }` | [Standard Webhooks](https://www.standardwebhooks.com): `webhook-id`, `webhook-timestamp`, `webhook-signature: v1,<HMAC-SHA256>`. The secret is base64 (at least 16 bytes decoded), usually `whsec_`-prefixed; as in every Standard Webhooks library the prefix is optional and the rest is always base64-decoded, so receivers verify with the same string. A secret that is not padded standard base64 (a raw passphrase, say) fails permanently rather than signing deliveries no receiver could verify. Generate one with `newWebhookSecret()`. Body `{ type: "form.submitted", timestamp, data: { form, submission, row } }` (`form.updated` on `update`). `url(config, { destinationId })` looks the address up when it is kept out of config. https only, no URL credentials, no localhost or private, loopback or link-local address literals, redirects not followed. Pass `resolveHost` to also refuse a hostname that resolves to a non-public address (checked before every request; see below). Without it, such a hostname is not caught; either way, if untrusted users pick URLs, send from a network that cannot reach internal services. Receivers check with `verifyWebhook({ secret, headers, body })` (constant-time, five-minute tolerance). |
+| `slackDestination({ url?, resolveHost?, message? })` | `{ url }` | One message per submission to a Slack incoming webhook (`https://hooks.slack.com/services/…` only; anything else fails permanently). The heading (`New response: <title>`, `Updated response` on `update`), each answer as `label: value` in form order, then a link back or the response id. Every value is escaped (`escapeSlack`), so typed text can't mention `@channel` or make a link; answers over 1,500 characters are cut. Sensitive fields masked by default. `message(input)` sets the heading, extra lines, a link and fields to `omit` per delivery; `slackMessage(submission, definition, options)` builds the same text on its own. The URL is the credential, so `url(config, { destinationId })` can look it up. |
+
+### Sending by answer
+
+A destination's `config.when` is a form `Condition` (the same shape logic rules use). The runner
+sends a submission only when it matches, reading answers by field id and hidden inputs by name
+(trusted `context` wins over a hidden input of the same name). A submission that doesn't match is
+recorded as `filtered`. A `when` that is not a valid condition fails the delivery permanently
+rather than guessing. `passesCondition(submission, definition, config)`, `conditionOf(config)` and
+`isCondition(value)` are exported for apps that check it themselves.
+
+### Hook URLs and SSRF
+
+`parseHookUrl(raw, "webhook" | "slack")` checks an address when someone adds it: https on port 443,
+no credentials, at most 2,048 characters, and a public host (no localhost, single-label or
+`.local`/`.internal`/`.lan`-style names, no private or reserved IP literal); a Slack address must
+be a Slack incoming webhook. It throws a `HookUrlError` with a message safe to show. Before every
+request, `checkHookHost(url, resolveHost)` (what `resolveHost` turns on in the webhook and Slack
+destinations) resolves the name and refuses it unless every address is public by
+`isPublicAddress`, which knows the private, loopback, link-local, CGNAT, documentation,
+benchmarking, multicast, IPv4-mapped, NAT64, 6to4 and Teredo ranges. A lookup that fails is
+retryable; a private address is not. In Node, `resolveHost` can be
+`(host) => dns.promises.lookup(host, { all: true }).then((a) => a.map((x) => x.address))`. The
+name is checked and then fetched, so a name re-pointed in between (DNS rebinding) is not caught:
+where untrusted people choose URLs, also send from a network that cannot reach internal services.
+
+### Google sign-in for sheets
+
+`googleOAuthAccessToken({ clientId, clientSecret }, refreshToken)` trades a refresh token from the
+app's own consent flow (`DRIVE_FILE_SCOPE` reaches only files the person picked or created through
+the app) for an access token, cached per grant until a minute before it expires.
+`refreshGoogleAccessToken` is the uncached call. `invalid_grant` fails permanently with "Connect
+the sheet again"; 429 and 5xx are retryable. The app keeps the refresh token; this only trades it.
+
+```ts
+sheetsDestination({
+  accessToken: async (config, { destinationId }) =>
+    googleOAuthAccessToken(oauthClient, await refreshTokenFor(destinationId)),
+});
+```
+
+### Secrets at rest
+
+Webhook and Slack URLs, signing secrets and refresh tokens should not sit in config. The helpers
+seal them for whatever store the app uses: `deriveSecretKey(appKey, purpose)` derives an AES-256-GCM
+key from a 32-byte app secret (base64 or base64url, as `openssl rand -base64 32` prints) with
+HKDF-SHA-256, one key per purpose; `sealSecret(key, plaintext, aad)` gives `v1.<iv>.<ciphertext>`
+(base64url), bound to `aad` (the row it belongs to), and `openSecret(key, sealed, aad)` opens it,
+failing permanently for another row, another purpose or a rotated key.
+
+### The outbox runner
+
+`createDeliveryRunner` claims a batch of due deliveries from the app's store, sends each to its
+destination, and records the outcome. The store maps the three calls to its own tables:
+
+```ts
+import { createDeliveryRunner } from "@marktiderman/genesis-forms/destinations";
+
+const runner = createDeliveryRunner<{ responseId?: string }>({
+  // Lease up to `limit` due rows (a claim function in SQL, say) with their submission and definition.
+  claim: ({ limit, filter }) => store.claimDeliveries(limit, filter),
+  // Write the outcome; resolve false if the lease had run out.
+  record: (delivery, outcome) => store.finishDelivery(delivery.id, outcome),
+  saveState: (destinationId, state) => store.saveDestinationState(destinationId, state),
+  destinations: { email, sheet, slack, webhook },
+  log: (line) => console.warn(line), // ids, kinds and safe reasons only
+});
+
+await runner.run({ filter: { responseId } }); // right after a save, or from a job
+```
+
+Each `ClaimedDelivery` carries its `id` (stable across retries), `destinationId`, `kind`, `config`,
+`state`, `submission`, `definition`, its `attempt` (from 1), how many times it was `deferrals`
+(deferred), and `redelivery` and `update` flags. A response sent and edited in one sheet batch is
+written once, as its latest version.
+Deliveries to one destination go in one `deliverBatch` call where the destination has one (a sheet
+appends them in one request), else one by one with the state passed along; new state is saved
+once. Outcomes: `sent` (`alreadyThere` on a redelivery that found its row), `filtered` (didn't
+match `config.when`), `skipped` (the destination returned `notSent`, a deliberate no-send such as a
+cap; never retried), `retry` with `retryAt`, and `failed`. A retryable failure with the service's
+Retry-After is deferred without counting an attempt (`counted: false`), up to `maxDeferrals`
+(default 24, counted by the store in `ClaimedDelivery.deferrals`); any other retryable
+failure backs off by attempt (`defaultBackoff`: 1 minute doubling to 6 hours) until `maxAttempts`
+(default 8), then fails. Permanent failures and unknown kinds fail at once. A destination that
+throws is reduced to a safe message. `run` returns a `DeliveryReport` of counts, including `stale`
+(record resolved false) and `unrecorded` (record threw; the lease runs out and the row comes back).
 
 ## Testing