@kurtbruns/kestrel 1.8.0 → 1.9.0

package/docs/README.md

@@ -28,3 +28,4 @@ The detail behind the steps, for when you need it.
 
 - [Configuration](reference/01-configuration.md)
 - [Sending-domain DNS](reference/02-sending-domain-dns.md)
+- [Import format](reference/03-import-format.md)

package/docs/get-started/04-ses.md

@@ -177,7 +177,7 @@ Until now the app has run with no email provider connected, refusing to send. Se
     npx wrangler secret put SNS_TOPIC_ARN --env production
     ```
 
-1. Store the configuration set's name, `kestrel-mail`. It's optional but recommended: the app names it on every email, so SES reports on each one even without the default:
+1. Store the configuration set's name, `kestrel-mail`. It's optional but recommended: the app names it on every email, so SES reports on each one even without the default. Type it exactly as the set is named in this region: if SES can't find the set, it refuses every email, and a list send is held until the name is fixed or the set created:
 
     ```bash
     npx wrangler secret put SES_CONFIGURATION_SET --env production

package/docs/get-started/06-go-live.md

@@ -35,14 +35,17 @@ Readers subscribe on a page the app hosts. Every signup confirms by email before
 
 ## As your list grows
 
-A new instance runs on Cloudflare's free Workers plan. As your list grows, two limits come into play.
+A new instance runs on Cloudflare's free Workers plan. As your list grows, three limits come into play.
 
 - **Your SES quota.** AWS caps how many emails your account sends a day. A send that hits the cap waits, and the app emails you. It resumes on its own once the quota frees up, or after AWS raises it.
 - **How fast a send goes out.** A send goes out a slice at a time, one slice a minute. SES takes one request a second, of up to 50 recipients, and no more recipients a second than your account's maximum send rate. So each request carries one second of that rate, up to 50.
+- **Your database's daily writes.** On the free plan, D1, the database the app runs on, allows 100,000 rows written a day. Past that, it refuses every query until midnight UTC, and the whole app stops: sends, signups, the public pages, and the editor.
+
+It's recommended to move to the $5 Workers Paid plan from about 5,000 subscribers. A send writes about 14 database rows a recipient, delivery receipts included. So one send uses up the free plan's day at about 7,000 recipients, and sooner when an import or anything else writes that day. Workers Paid counts [D1's writes](https://developers.cloudflare.com/d1/platform/pricing/) by the month instead, with 50 million included, and never stops the database for them. On the free plan, the app warns you before a send to more than 5,000 recipients, in the Schedule dialog and on the scheduled post. It never stops you, since your account may already be on Workers Paid.
 
 The free Workers plan sends four requests a minute. At 14 messages a second, a common starting rate, that's 56 recipients a minute, and 1,000 subscribers take about 18 minutes. At 50 a second or more, it's 200 a minute.
 
-To send faster, first ask AWS to raise your maximum send rate to 50, which costs nothing, then set [`SES_MAX_SEND_RATE`](../reference/01-configuration.md#ses_max_send_rate) to match, and deploy again. A rate above 50 adds nothing on the free plan, where a rate of 50 sends a list of about 12,000 in an hour. If a send still takes longer than you want, move to the $5 Workers Paid plan, and set [`SUBREQUEST_BUDGET`](../reference/01-configuration.md#subrequest_budget) to `10000`. It sends about 50 requests a minute: about 700 recipients at 14 a second, and about 2,500 at 50 or more.
+To send faster, first ask AWS to raise your maximum send rate to 50, which costs nothing, then set [`SES_MAX_SEND_RATE`](../reference/01-configuration.md#ses_max_send_rate) to match, and deploy again. A rate above 50 adds nothing on the free plan, where a rate of 50 sends 5,000 subscribers in about 25 minutes. A larger list belongs on Workers Paid for its database writes anyway. On Workers Paid, set [`SUBREQUEST_BUDGET`](../reference/01-configuration.md#subrequest_budget) to `10000`. It sends about 50 requests a minute: about 700 recipients at 14 a second, and about 2,500 at 50 or more.
 
 ## Check it
 

package/docs/reference/01-configuration.md

@@ -62,6 +62,12 @@ This setting is the subrequest cap. The app keeps database queries under theirs
 - **On Workers Paid,** which allows 10,000, set `"SUBREQUEST_BUDGET": "10000"`. A minute then sends about 50 SES requests, one a second: about 700 recipients at a rate of 14, and about 2,500 at 50 or more.
 - **If you raised the Worker's own limit** with `limits.subrequests`, you may set this to match.
 
+It's also how the app tells Workers Free from Workers Paid. At the default, the app runs on Free's limits:
+
+- One import takes up to 1,000 rows, since Free's 10 ms of CPU a request can't plan and commit more. Above it, up to 10,000 ([Import format](03-import-format.md#limits)).
+- An export reads 1,000 rows a page. Above it, 10,000.
+- The app warns before a send to more than 5,000 recipients, and after an import plan that leaves more than 3,000 confirmed subscribers. Either may use up D1's daily allowance of 100,000 rows written, which stops the app until midnight UTC ([As your list grows](../get-started/06-go-live.md#as-your-list-grows)). Neither warning stops you. Above the default, neither appears.
+
 Never set it above your plan's limit. A minute that hits the cap is cut off mid-batch, and the send stalls until a later minute picks it up. A value below 30 is raised to 30, the least a minute needs to send anything and still send a notification.
 
 ### `MIN_LEAD_SECONDS`
@@ -91,7 +97,7 @@ Used only with the `NOTIFY` binding. It must be an address on a domain set up fo
 | `ACCESS_ALLOWED_EMAILS` | a comma-separated list of the people allowed to sign in | optional |
 | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` | the IAM user's access key | SES |
 | `SNS_TOPIC_ARN` | the SNS topic the SES webhook accepts; it refuses every other topic | SES |
-| `SES_CONFIGURATION_SET` | the configuration set named on every email, so SES reports bounces and complaints | SES, recommended |
+| `SES_CONFIGURATION_SET` | the configuration set named on every email, so SES reports bounces and complaints; a name SES can't find in the region holds every list send until it's fixed | SES, recommended |
 
 `ACCESS_ALLOWED_EMAILS` applies to people only. A service token, such as the one Claude uses, carries no email, so the Access policy alone decides whether it gets in.
 

package/docs/reference/03-import-format.md

@@ -0,0 +1,103 @@
+# Import format
+
+Kestrel imports a subscriber list in one documented format, `kestrel-subscribers/1`, and exports the whole list in the same format. This page describes the format, what an import does to each address, and the limits.
+
+Kestrel never guesses what another service's export means, because a wrong guess about who agreed to be emailed can't be taken back. An export from Sendy, Mailchimp, or anywhere else is converted to this format first, by Claude or by hand. The API serves everything on this page at `GET /api/imports/format`, generated from the same definition the importer reads with, along with a CSV template at `GET /api/imports/template.csv` and a plain-list template at `GET /api/imports/template.txt`.
+
+## Two kinds of file
+
+- **The full CSV (preferred).** It carries unsubscribes, bounces, and how each person joined. It's what the export writes, and you can open it in a spreadsheet to check it before importing.
+- **A plain list (`.txt`).** One address per line, for a list that is only addresses. You say what they all are: subscribed (the default), unsubscribed, or bounced. It loses how each person joined, any times, the evidence of consent, and mixed states.
+
+The file's name decides which it is: `.csv` or `.txt`. Anything else is refused.
+
+## The CSV
+
+A CSV file (RFC 4180, UTF-8) with a header row of the column names below, in any order, then one row per address. Column names are written exactly, in lowercase. A row records what the source claims, never what Kestrel will do: the import decides that, under the rules further down.
+
+| Column | Required | Values | Meaning |
+| --- | --- | --- | --- |
+| `email` | yes | an email address | The address, trimmed and lowercased before it is matched. An invalid one is still a row, counted as invalid and reported, never dropped. |
+| `status` | yes | `subscribed`, `pending`, `unsubscribed`, `none` | The consent claim. `none` is a known address with no consent claim, present only to carry a suppression. |
+| `opt_in` | yes | `double`, `single`, `unknown` | How consent was collected. `unknown` unless the source states it. |
+| `origin` | yes | a short lowercase name | Where consent was first collected, as the file claims it: `sendy`, `mailchimp`, `kestrel`, or `unknown`. |
+| `suppression` | no | `bounce`, `complaint`, `manual`, `erased` | A suppression on top of `status`: the address bounced, complained, was suppressed by hand, or was erased. |
+| `subscribed_at` | no | an instant with its zone | When the person subscribed, if the source states it. |
+| `confirmed_at` | no | an instant with its zone | When the person confirmed, if the source states it. |
+| `unsubscribed_at` | no | an instant with its zone | When the person left, if the source states it. A reader who confirmed in Kestrel after it stays confirmed. |
+| `suppressed_at` | no | an instant with its zone | When the address was suppressed, if the source states it. |
+| `evidence` | no | a JSON object, at most 500 bytes | The source's consent-relevant columns, verbatim, such as a signup IP or a form name. Nothing reads it to decide anything: it is the proof of consent and the audit trail of a conversion. |
+| `format` | no | `kestrel-subscribers/1` | The version the file is written in. A file without it is read as version 1. |
+
+An instant is a UTC ISO-8601 time with its zone, such as `2024-03-01T09:30:00Z` or `2024-03-01T10:30:00+01:00`. A time without a zone is refused, never interpreted. An optional column may be left out, or left empty on a row.
+
+Any other column is refused, naming it. A source's other columns belong in `evidence`, as JSON. A file whose header names none of these columns isn't in the format at all, and is refused as such.
+
+Here is a small file:
+
+```
+email,status,opt_in,origin,suppression,unsubscribed_at,evidence
+ada@example.com,subscribed,double,sendy,,,"{""signup_ip"":""203.0.113.7""}"
+grace@example.com,subscribed,single,mailchimp,,,
+edsger@example.com,unsubscribed,unknown,unknown,,2025-01-20T08:00:00Z,
+bounced@example.com,none,unknown,unknown,bounce,,
+```
+
+**Versions.** Adding an optional column keeps the version. Changing what a column means makes a new version. Kestrel reads every version it has published, and refuses one it doesn't know, by name.
+
+## The plain list
+
+- One address per line. Each line is trimmed, blank lines are skipped, and CRLF line endings and a UTF-8 byte-order mark are accepted.
+- A line that isn't a single valid address, such as `Name <ada@example.com>` or a typo, is counted as invalid and listed among the rows left out. One bad line doesn't sink a list.
+- The whole file is refused when it isn't a plain list at all: not UTF-8 text, a line with a comma, tab, or semicolon (it's a table, so a `.csv`), or not one valid address in it.
+- Its rows arrive with `origin: unknown` and `opt_in: unknown`.
+
+## Plan, then commit
+
+An import happens in two steps. Planning shows what the import would do to every row, against your list as it stands, and changes nothing, so you can plan a file as often as you need. Committing applies the plan, all of it or none of it, and sends no email.
+
+The commit needs two things from you when they apply:
+
+- **Your attestation,** when anyone would be confirmed: "Everyone I'm importing as subscribed opted in to receive [your publication]. This isn't a purchased, rented, or scraped list." It is kept with the import, with who gave it and when. An import that only unsubscribes or suppresses needs none.
+- **Each send that would grow,** acknowledged by a person. A post goes to the list as it stands when it fires, so an import can add to the audience of a scheduled send, or of one that has just fired and not yet fixed its audience. Claude can attest for you, but can't acknowledge a growing send: you finish that import in the dashboard.
+
+If another import is committed after you plan, plan again. If your list changes between the plan and the commit in any other way, such as a reader unsubscribing, the commit still goes ahead. Every write checks the list as it stands, and the answer reports any row that turned out differently.
+
+## What an import does to each address
+
+Each row gets exactly one outcome. A row that both changes its status and adds a suppression reports its status outcome, and says it added a suppression too.
+
+| Outcome | When |
+| --- | --- |
+| `create_confirmed` | `subscribed`, with no subscriber and no suppression in Kestrel: added as confirmed. |
+| `confirm_pending` | `subscribed`, and pending in Kestrel (never confirmed): confirmed, and its confirmation link no longer works. |
+| `unchanged` | Already confirmed in Kestrel, or already in the state the row claims. A subscriber who joined through Kestrel keeps their own record. |
+| `mark_unsubscribed` | `unsubscribed`, and confirmed or pending in Kestrel, with no confirmation in Kestrel newer than the row's `unsubscribed_at`: unsubscribed. |
+| `create_unsubscribed` | `unsubscribed`, with no subscriber in Kestrel: recorded as unsubscribed, so the list remembers they left. |
+| `kept_confirmed_newer` | `unsubscribed`, but the reader confirmed in Kestrel after the source's `unsubscribed_at`: their newer action stands. |
+| `kept_unsubscribed` | `subscribed`, but unsubscribed in Kestrel, or pending again after leaving until they confirm in Kestrel themselves. An unsubscribe is never reversed. |
+| `blocked_by_suppression` | `subscribed`, but the address carries a suppression in Kestrel, a do-not-contact marker included: never added or confirmed. |
+| `suppression_added` | The row carries a suppression and the address has none in Kestrel, or only a do-not-contact marker, and nothing else changes. Applied whatever the row's `status`. |
+| `left_out` | `pending` or `none` with nothing to apply: counted, never confirmed. An import sends no email, so it can't start a confirmation either. |
+| `invalid` | The address isn't valid. |
+| `merged` | A repeat of an address earlier in the file, folded into it. |
+
+A suppression already in Kestrel is never replaced, with one exception: a bounce, a complaint, or a manual suppression replaces a do-not-contact marker left by an erasure, so a known bad address still gets no confirmation. An imported bounce or complaint is kept apart from one Kestrel saw itself, so the list says where each came from.
+
+Anyone can type an address into your subscribe form, which puts a reader who left back to pending until they confirm. An import treats such a reader as unsubscribed, so it never confirms someone who left in Kestrel.
+
+**Repeats of an address** are folded into one row first. Its status is the first of unsubscribed, subscribed, pending, and none that any repeat claims, and its suppression the first of erased, complaint, bounce, and manual. Its opt-in is the weakest claim. It keeps the earliest consent times and the latest unsubscribe and suppression times, and the origin and evidence of the row whose status won.
+
+## How imported subscribers read
+
+A subscriber an import confirmed is **confirmed**, like anyone who confirmed in Kestrel, and gets your posts. How they joined shows beside their state: imported, with the origin and opt-in method the file claimed, and the evidence. That holds even when the file says they confirmed through Kestrel itself. Only a confirmation Kestrel recorded reads as Kestrel's double opt-in, and a subscriber who later confirms in Kestrel becomes Kestrel's own.
+
+## Limits
+
+- **Rows:** up to 10,000 in one import. On the Workers Free plan's limits (`SUBREQUEST_BUDGET` unset), up to 1,000, since Free's 10 ms of CPU a request can't plan and commit more. Set [`SUBREQUEST_BUDGET`](01-configuration.md#subrequest_budget) on Workers Paid to take 10,000. Split a larger list into several files.
+- **Size:** a request of at most 10 MB, and `evidence` of at most 500 bytes a row.
+- **Database writes on Workers Free:** D1's free plan allows 100,000 rows written a day, and refuses every query past that until midnight UTC. Each new subscriber an import brings in writes about 7, and a send writes about 14 a recipient. So when an import leaves more than about 3,000 confirmed subscribers on Workers Free, import on one day and send on the next, by UTC. The plan says so when it applies, and never refuses. One import of 1,000 rows uses about 7% of a day's writes.
+
+## Exporting
+
+The export reads your whole list out in this format, a page at a time, at `GET /api/subscribers/export`. Each page holds up to 1,000 rows on the Workers Free plan's limits, or 10,000 otherwise, and its `next` value asks for the page after it. Following `next` until it's `null` gives every address exactly once, even if the list changes while you read. An address added or removed partway through may or may not appear. To make one CSV in this format, write the pages' rows under a single header. In the CSV, a value that starts with `=`, `+`, `-`, or `@` is written with an apostrophe before it, so a spreadsheet shows it as text rather than running it as a formula; an import takes the apostrophe off again. It includes every subscriber with how they joined, their evidence, and their suppression, and every address suppressed with no subscriber behind it, written as `status: none`. A confirmed subscriber is written as `subscribed`. Importing an export into another Kestrel carries the list over: the same states and suppressions, with every confirmation arriving as imported, carrying its original evidence. Pending subscribers are left out on the way in, since an import sends no confirmation.

package/migrations/0005_imports.sql

@@ -0,0 +1,74 @@
+-- 0005_imports: bringing an existing list in (SPEC §7, §8).
+--
+-- An import confirms a subscriber without their confirming here, on the strength of the
+-- previous service's evidence and the publisher's attestation, so the record has to say,
+-- per subscriber, how they joined, and keep each import's own record beside the list.
+--
+-- `imports` is that record: who committed it (a person, with their login, or a service
+-- token, which has none), when, the attestation's version when one was given, the file's
+-- name and SHA-256, and how many rows each outcome applied to. It never holds a
+-- subscriber's address, so an erasure has nothing to reach here: what an import brought in
+-- about a person lives on their subscriber row and leaves with it. `plan_hash` is UNIQUE
+-- because it is the commit's idempotency key: a commit replayed after a lost answer finds
+-- the import already made and returns it instead of applying it twice.
+
+CREATE TABLE imports (
+  id                  TEXT PRIMARY KEY,
+  created_at          INTEGER NOT NULL,
+  principal_kind      TEXT NOT NULL CHECK (principal_kind IN ('human', 'service')),
+  principal_email     TEXT CHECK (principal_email = lower(principal_email)),
+  attestation_version INTEGER,                  -- null when no row was confirmed
+  file_name           TEXT NOT NULL,
+  file_sha256         TEXT NOT NULL,
+  plan_hash           TEXT NOT NULL UNIQUE,
+  counts              TEXT NOT NULL CHECK (json_valid(counts)),  -- JSON: rows by outcome
+  suppressions_added  INTEGER NOT NULL DEFAULT 0,
+  CHECK ((principal_kind = 'human') = (principal_email IS NOT NULL))
+) STRICT;
+
+-- How each subscriber joined, beside their consent state and never a fourth state of it.
+-- `origin` and `opt_in` are where and how consent was collected; `consent_evidence` is the
+-- proof (a source's consent columns verbatim, or the IP a confirmation here came from),
+-- which nothing reads to decide anything; `import_id` names the import that confirmed or
+-- added the row, and is what makes a row read as imported whatever its file claimed. A
+-- confirmation recorded here clears it, since the consent is then this app's own.
+--
+-- The defaults are the backfill: every subscriber before this migration joined through
+-- this app's double opt-in (or is on the way to it), so it reads as `kestrel`, `double`.
+ALTER TABLE subscribers ADD COLUMN origin TEXT NOT NULL DEFAULT 'kestrel'
+  CHECK (origin = lower(origin) AND length(origin) BETWEEN 1 AND 40);
+ALTER TABLE subscribers ADD COLUMN opt_in TEXT NOT NULL DEFAULT 'double'
+  CHECK (opt_in IN ('double', 'single', 'unknown'));
+ALTER TABLE subscribers ADD COLUMN consent_evidence TEXT
+  CHECK (consent_evidence IS NULL OR json_valid(consent_evidence));
+ALTER TABLE subscribers ADD COLUMN import_id TEXT REFERENCES imports (id);
+
+-- The publisher is told of an import Claude commits (SPEC §8) through the notifications
+-- they already get, rather than a second table. `notifications.kind`'s CHECK doesn't admit
+-- it and a STRICT table's CHECK can't be altered, so the table is rebuilt with `import`
+-- added and an `import_id` for it: an import notice has an import and no send, a test has
+-- neither, and every other kind has a send. Each import is told at most once.
+CREATE TABLE notifications_next (
+  send_id    TEXT REFERENCES sends (id),       -- null for a test and an import notice
+  import_id  TEXT REFERENCES imports (id),     -- set only for an import notice
+  kind       TEXT NOT NULL
+               CHECK (kind IN ('finished', 'refused', 'stuck', 'wedged', 'missed', 'test', 'import')),
+  episode    INTEGER NOT NULL DEFAULT 0,
+  status     TEXT NOT NULL DEFAULT 'pending'
+               CHECK (status IN ('pending', 'sent', 'failed', 'unaddressed', 'cleared')),
+  attempts   INTEGER NOT NULL DEFAULT 0,
+  error      TEXT,
+  created_at INTEGER NOT NULL,
+  updated_at INTEGER NOT NULL,
+  UNIQUE (send_id, kind, episode),
+  UNIQUE (import_id),
+  CHECK ((send_id IS NULL) = (kind IN ('test', 'import'))),
+  CHECK ((import_id IS NULL) = (kind <> 'import'))
+) STRICT;
+
+INSERT INTO notifications_next (send_id, kind, episode, status, attempts, error, created_at, updated_at)
+  SELECT send_id, kind, episode, status, attempts, error, created_at, updated_at FROM notifications;
+
+DROP TABLE notifications;
+ALTER TABLE notifications_next RENAME TO notifications;
+CREATE INDEX idx_notifications_status ON notifications (status, created_at);

package/migrations/0006_deliveries_fewer_indexes.sql

@@ -0,0 +1,19 @@
+-- 0006_deliveries_fewer_indexes: drop two indexes on deliveries that no query uses, so each
+-- recipient of a send writes fewer index entries.
+--
+-- Every write to a delivery row also writes each index entry it moves, and on the Workers
+-- Free plan D1 refuses every query once a day's written rows are spent, so an index no query
+-- reads costs a send's reach for nothing.
+--
+-- idx_deliveries_in_flight indexed `updated_at` over the rows in flight, for a sweep that
+-- looked for in-flight rows gone stale. No query reads in-flight rows by time now: how long
+-- a send has run is read from the send, and a stuck send from its counters.
+--
+-- idx_deliveries_dispatch_key indexed the dispatch key alone. Every query that reads by key
+-- also names the send and a status, and reads through (send_id, status); none looks a row
+-- up by key alone.
+--
+-- Dropping an index writes no rows, so this costs nothing to apply at any size.
+
+DROP INDEX IF EXISTS idx_deliveries_in_flight;
+DROP INDEX IF EXISTS idx_deliveries_dispatch_key;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@kurtbruns/kestrel",
-  "version": "1.8.0",
+  "version": "1.9.0",
   "type": "module",
   "description": "An open-source, self-hosted newsletter app where you own the list and the archive. Work in the editor, or just ask Claude. Runs on Cloudflare Workers.",
   "license": "MIT",