@bumail/queue 0.1.0 → 0.3.0

package/docs/guide.md

@@ -13,6 +13,8 @@ workers share it.
 - [Events](#events)
 - [Admin](#admin)
 - [The stores](#the-stores)
+- [PostgreSQL](#postgresql)
+- [Redis](#redis)
 - [Testing](#testing)
 - [Writing a store](#writing-a-store)
 - [Options](#options)
@@ -234,7 +236,9 @@ is written as `utf-8; j\x{F6}rg@…`, RFC 6533), and every field is folded.
 
 ## Several workers, leases and stopping
 
-Any number of queues may share one store, in one process or several:
+Any number of queues may share one store, in one process or several —
+or, with [PostgreSQL](#postgresql) or [Redis](#redis), on several
+machines:
 
 ```ts
 const store = SqliteQueueStore.open({ directory: '/var/lib/bumail/queue' });
@@ -252,9 +256,17 @@ queue.start();
   stops them.
 - **A crashed worker** loses its items when their leases expire: another
   worker claims them then. A worker whose lease was taken meanwhile
-  records nothing, and says so on the `error` event (`LEASE_LOST`). An
-  item cancelled while it was delivered still has its outcomes told by
-  the events, with no DSN and no error.
+  records nothing, and says so on the `error` event (`LEASE_LOST`) — also
+  when that worker has since finished the item and dropped it, the
+  message then sent twice. An item cancelled while it was delivered, its
+  lease still holding, has its outcomes told by the events, with no DSN
+  and no error. Once the lease's expiry has passed, a worker cannot tell
+  a cancel from an item another worker finished: it reports `LEASE_LOST`,
+  lost or cancelled, and tells no outcome. One gap remains when the
+  instances' clocks are out of step: an instance whose clock runs ahead
+  can take the item while this worker's clock says the lease holds, and
+  if it finishes the item between two renewals, the attempt reads as a
+  cancel. Keep the machines on NTP.
 - **`stop()`** claims nothing more, lets every session under way end and
   records its outcome, and gives back what was claimed but not started —
   due at once, no attempt counted — so another worker can take it.
@@ -352,7 +364,7 @@ store.close(); // closing twice is fine
   each item is claimed once; a writer waits up to `busyTimeout`
   milliseconds for another.
   Not on a network file system: SQLite's locks do not hold there. For
-  several machines, a server database is on the roadmap.
+  several machines, use [PostgreSQL](#postgresql) or [Redis](#redis).
 - **Messages in a table of their own**, `messages`, dropped with their
   item in the same transaction. A queue holds a message for days at most
   and never shares one between items, so content addressing (as
@@ -361,6 +373,293 @@ store.close(); // closing twice is fine
 - **Migrations** run on open, in one transaction; a database written by a
   newer version is refused.
 
+### `PostgresQueueStore`
+
+On PostgreSQL, for instances on several machines: see
+[PostgreSQL](#postgresql).
+
+### `RedisQueueStore`
+
+On Redis, for instances on several machines: see [Redis](#redis).
+
+## PostgreSQL
+
+```ts
+import { createQueue } from '@bumail/queue';
+import { PostgresQueueStore } from '@bumail/queue/postgres';
+
+const sql = new Bun.SQL({ url: Bun.env['DATABASE_URL'], max: 10 });
+const store = PostgresQueueStore.open({ sql });
+await store.migrate();
+const queue = createQueue({ store, hostname, resolver, owner: `${host}-${process.pid}` });
+queue.start();
+
+process.on('SIGTERM', async () => {
+	await queue.stop();
+	await sql.close(); // yours: the store never closes a client it was given
+});
+```
+
+`@bumail/queue/postgres` runs on Bun's own `Bun.sql`: there is no
+driver to install and no peer to add. Every instance of your server opens
+a store on the same database, and they share one queue.
+
+### The client
+
+- **`sql`** is a `Bun.SQL` client of yours — you size its pool and close
+  it — or a `postgres://` (or `postgresql://`) URL, for which the store
+  opens a client with Bun's defaults and closes it in `close()`. A
+  `Bun.SQL` client for SQLite or MySQL is refused.
+- It is typed by its shape, `PostgresClient` (`unsafe`, `begin` and
+  `close`), so the declarations need nothing of `@types/bun`; a `Bun.SQL`
+  client fits it.
+- **Nothing connects at `open`**: a wrong option is refused there, and a
+  database out of reach on the first call.
+
+### The schema
+
+Three tables, each named by `tablePrefix` (`bumail_queue_` by default),
+in the connection's default schema (its `search_path`):
+
+| table | what it holds |
+| --- | --- |
+| `<prefix>items` | an item a row: `seq` (an identity, the order among items equally due), `id` (`text`, unique), `sender`, `recipients` (`jsonb`, each recipient's state), `size`, `created_at`, `next_attempt_at`, `attempts`, `delay_notified`, `lease_owner` and `lease_expires_at`. Times are milliseconds since the epoch, as `double precision`, as the queue gives them. `<prefix>items_due`, on `(next_attempt_at, seq)`, serves the claim |
+| `<prefix>messages` | the message as enqueued, whole, as `bytea`, keyed by its item's `id` and dropped with it (`ON DELETE CASCADE`) |
+| `<prefix>schema` | the version: how many migrations ran |
+
+They are the `bun:sqlite` store's tables, column for column. The prefix
+is lowercase letters, digits and underscores, starting with a letter or
+an underscore, 40 characters at most: it is written into the statements,
+never bound. Give each queue its own prefix to keep several in one
+database. A `bytea` holds 1 GB at most, far above `limits.maxMessageSize`
+(25 MiB by default).
+
+### Migrations
+
+`migrate()` makes the tables, or brings them to the last migration, in
+one transaction; without it, the first call that needs them does. It
+runs once per store. Instances starting together wait on one advisory
+lock (`pg_advisory_xact_lock`), so each migration runs once. A
+migration that fails is `INVALID`, `The PostgreSQL queue cannot be set
+up: …`, and the next call tries again; tables written by a newer version
+of the package are refused.
+
+The role needs `CREATE` on the schema the first time, then only `SELECT`,
+`INSERT`, `UPDATE` and `DELETE` on the three tables: a store whose tables
+are current only reads their version, with no lock and no DDL. So run
+`migrate()` once from a deploy step with an owner's role, and give the
+workers a narrower one:
+
+```sql
+GRANT SELECT, INSERT, UPDATE, DELETE
+	ON bumail_queue_items, bumail_queue_messages, bumail_queue_schema
+	TO bumail_worker;
+```
+
+After an upgrade that adds a migration, run that step again before the
+workers start: theirs cannot make the change.
+
+### Several instances and leases
+
+```sql
+UPDATE <prefix>items SET lease_owner = $1, lease_expires_at = $3
+WHERE seq = (SELECT seq FROM <prefix>items
+	WHERE next_attempt_at <= $2 AND (lease_owner IS NULL OR lease_expires_at <= $2)
+	ORDER BY next_attempt_at, seq LIMIT 1
+	FOR UPDATE SKIP LOCKED)
+RETURNING …
+```
+
+- **A claim** is that one statement: the earliest due item that no other
+  transaction is taking, locked as it is picked, then leased. Two
+  instances never take the same item, and none waits on an item another
+  is claiming: it takes the next.
+- **The lease** works as on `bun:sqlite`: renewed every third of
+  `leaseMs`, let go of with the outcome. `complete` locks the item's row
+  and records the outcome only while its owner holds the lease. An
+  instance that crashes loses its items when their leases expire, and
+  another takes them then; one that stalled past its lease records
+  nothing once another instance has claimed the item (`LEASE_LOST`):
+  `complete` checks who owns the lease, not when it expires.
+- **The clocks** of the instances must agree: each gives the time of its
+  own claims and leases, as the store keeps no clock. An instance whose
+  clock runs ahead sees leases expire early by as much. Keep the machines
+  on NTP; the 10-minute `leaseMs` leaves room for seconds of skew.
+- **`limits.maxItems`**: the adds that count wait on one advisory lock,
+  so two instances never both take the last place.
+
+### Pool size
+
+Each call holds a connection for one statement, `complete` and an add
+with `limits.maxItems` for one short transaction. A worker has at most
+`concurrency` items (20) in flight, each mostly waiting on a remote SMTP
+server, not the database: `Bun.SQL`'s default pool of 10 connections is
+enough for one, and a call waits for a free connection rather than fail.
+Raise `max` only when a busy worker waits on the pool, and keep every
+instance's pool, and your application's, within the server's
+`max_connections` (100 by default): four instances of `max: 10` take 40.
+
+## Redis
+
+```ts
+import { createQueue } from '@bumail/queue';
+import { RedisQueueStore } from '@bumail/queue/redis';
+
+const client = new Bun.RedisClient(Bun.env['REDIS_URL']);
+const store = RedisQueueStore.open({ client, keyPrefix: 'mail:queue:' });
+const queue = createQueue({ store, hostname, resolver, owner: `${host}-${process.pid}` });
+queue.start();
+
+process.on('SIGTERM', async () => {
+	await queue.stop();
+	client.close(); // yours: the store never closes a client it was given
+});
+```
+
+`@bumail/queue/redis` runs on Bun's own `Bun.RedisClient` (`Bun.redis`
+is its default instance): there is no driver to install and no peer to
+add. Every instance of your server opens a store on the same Redis, and
+they share one queue. It is held to the same contract specs as the other
+stores, and to the same specs of two instances delivering every item
+exactly once, against `redis:7`.
+
+### The client
+
+- **`client`** is a `Bun.RedisClient` of yours — you set its options and
+  close it — or **`url`** a `redis://` (`rediss://` for TLS, or the other
+  schemes `Bun.RedisClient` takes), for which the store opens a client
+  with Bun's defaults and closes it in `close()`. One or the other, never
+  both.
+- It is typed by its shape, `RedisQueueClient` (`send`, `getBuffer` and
+  `close`), so the declarations need nothing of `@types/bun`; a
+  `Bun.RedisClient` fits it.
+- **Nothing connects at `open`**: a wrong option is refused there, and a
+  server out of reach on the first call. With Bun's defaults a client
+  reconnects on its own, up to 20 times, and holds the commands sent
+  while it is disconnected until Redis is back (a `PING` sent during a
+  5-second stop answered once Redis started again). A command already
+  sent when the connection drops, or sent before the client noticed,
+  fails instead (`Connection closed`): a renewal tries again on its own,
+  and a claim or an outcome is told on `error` (see Troubleshooting).
+- One client is one connection, pipelined: every call of one store goes
+  through it, so there is no pool to size.
+
+### The keys
+
+Every key starts with `keyPrefix` (`bumail:queue:` by default):
+
+| key | type | what it holds |
+| --- | --- | --- |
+| `<prefix>item:<id>` | hash | an item: `id`, `from`, `recipients` (JSON, each recipient's state), `size`, `created`, `next`, `attempts`, `delay`, while leased `owner` and `expires`, its `member` in the sorted sets and `rev`, how many outcomes were recorded |
+| `<prefix>message:<id>` | string | the message as enqueued, whole, byte for byte |
+| `<prefix>items` | sorted set | every item, scored by its next attempt: `list` and `count` |
+| `<prefix>ready` | sorted set | the items no lease holds, scored by their next attempt |
+| `<prefix>leases` | sorted set | the leased items, scored by when their leases expire |
+| `<prefix>seq` | string | the counter that orders items equally due |
+| `<prefix>schema` | string | the layout version |
+
+A member of the sorted sets is the item's sequence number, sixteen
+digits, then `:` and its id, so items equally due come oldest first.
+Times are kept as the strings JavaScript writes for them, and read back
+exactly. The prefix is lowercase letters, digits, `_`, `:`, `.` and
+`-`, starting with a letter, 40 characters at most: no `{`, so no
+Cluster hash tag, and no glob character, so `SCAN 0 MATCH <prefix>*`
+finds the queue's keys and nothing else. A `<prefix>schema` that holds
+anything but a layout version is refused at the first call: the prefix
+is another application's. Give each queue its own
+prefix to keep several in one Redis database. An id the store did not
+make (`crypto.randomUUID()`'s form) is one no item has: no id a caller
+gives reaches another key.
+
+### Several instances and leases
+
+Every operation that writes is one Lua script, run by `EVALSHA` (by
+`EVAL` when Redis no longer has it, after a restart, a failover or a
+`SCRIPT FLUSH`): Redis runs a script whole, with no command of another
+instance between its steps.
+
+- **A claim** takes the first item of `ready` due by `now`, and every
+  lease of `leases` expired by `now`, keeps the earliest due of them,
+  then the oldest, and moves it from `ready` to `leases` with its new
+  expiry, in one script. Two instances never take the same item. An
+  expired lease is found at once, not by a sweep; there are only as many
+  as a crashed or stalled instance left.
+- **`complete`** reads the item, applies the outcome in JavaScript, and
+  records it with a script that checks the owner still holds the lease
+  and that no other outcome was recorded since (`rev`); otherwise it
+  reads again, at most 8 times, then records nothing (`LEASE_LOST`), as
+  it does when the lease was lost.
+- **`renew`**, **`reschedule`** and **`cancel`** are a script each, and
+  an `add` with `limits.maxItems` counts and adds in one, so two
+  instances never both take the last place.
+- **The clocks** of the instances must agree, as on PostgreSQL: the store
+  keeps no clock, and Redis's is never read.
+
+### Deploying Redis
+
+- **One Redis, or a primary with replicas — not Redis Cluster.** A script
+  reaches each item's keys, which are not all named up front, so the
+  queue's keys must live on one node. Point the store at the primary: a
+  replica refuses writes (`READONLY`). After a failover, the URL must
+  reach the new primary — a provider's endpoint that follows it, say.
+- **Durability is Redis's, and weaker than PostgreSQL's.** Redis
+  acknowledges a write once it is in memory:
+  - with `appendonly yes` and `appendfsync always`, a write is on disk
+    before it is acknowledged, as on `bun:sqlite` or PostgreSQL;
+  - with `appendfsync everysec` (Redis's default once AOF is on), a
+    crash of the server loses up to about a second of acknowledged
+    writes;
+  - with snapshots only (`save`, AOF off), it loses everything since the
+    last snapshot, minutes of it;
+  - **replication is asynchronous**: a primary that fails before its
+    replica has a write loses that write when the replica is promoted,
+    whatever the fsync setting.
+
+  A lost write is an enqueued message gone (the client was told it was
+  queued), an outcome forgotten (a recipient delivered and sent the
+  message again), or a lease given twice. Use `appendfsync always`
+  where a lost message matters, and PostgreSQL where it must survive a
+  failover.
+- **`maxmemory-policy noeviction`.** Every message lives in memory, whole,
+  until its item is done. Under any other policy a full Redis evicts
+  queue keys — an item, or its message — to make room; with
+  `noeviction` it refuses the write (`OOM command not allowed`), and
+  `enqueue` rejects with that `RedisError`. An item whose hash is gone
+  all the same (evicted, or deleted by hand) is dropped by the next
+  claim that meets it, its message with it, and never leased. Bound the queue with `limits.maxItems` and
+  `limits.maxMessageSize`, and size `maxmemory` for both.
+- **An ACL user** needs the commands the store sends and the ones its
+  scripts call, on its prefix's keys (`~<keyPrefix>*`; here the
+  example's `mail:queue:`), and nothing else:
+
+  ```
+  ACL SETUSER bumail-worker on >secret resetkeys ~mail:queue:* -@all +evalsha +eval +get +set +del +incr +hgetall +hget +hset +hdel +hincrby +zadd +zrem +zrange +zrangebyscore +zcard
+  ```
+
+### What `Bun.redis` does with bytes and text
+
+Measured on Bun 1.4.2 against Redis 7:
+
+- An argument may be a `Uint8Array`: `send` writes it as it is, every
+  byte value, to `EVAL` and `EVALSHA` too. That is how a message is
+  written, through the script that adds its item.
+- A bulk string in a reply is decoded as UTF-8 into a JavaScript string,
+  so bytes that are not UTF-8 do not come back from `send`. A message is
+  read with `getBuffer`, which keeps the bytes.
+- A string argument is encoded as UTF-8, and a lone surrogate becomes
+  U+FFFD without a word: the contract's checks refuse a lone surrogate
+  (and a NUL, which Redis could keep) before anything is sent, so what is
+  read back is always what was given.
+- `HGETALL` sent directly comes back as an object; the same from a script
+  as a flat list of names and values. A script's `false` is `null`.
+- An error reply rejects with a `RedisError` whose message is Redis's
+  (`NOSCRIPT No matching script…`); none of them repeats the URL.
+- A server out of reach: with Bun's defaults a call waits for 20
+  reconnections, about half a minute, then rejects with `Max
+  reconnection attempts reached`; a URL that is not one is not refused by
+  `new Bun.RedisClient`, only by its first call, so the store checks the
+  scheme itself.
+
 ## Testing
 
 Inject the clock and the sender, and call `deliverDue()` by hand: no
@@ -425,8 +724,16 @@ codes, and keeps its promises:
   dropped with its message.
 - Times are given by the caller; the store keeps no clock.
 - Nothing it returns is shared with what it keeps.
-
-The contract's specs, `describeQueueStore`, which both stores here run,
+- `reschedule` with an owner refuses an empty owner as `INVALID`, even
+  for an id no item has. An id a store cannot keep (one holding a NUL or
+  a lone surrogate) may instead be answered `false`, unknown, before the
+  owner is checked.
+- Text it keeps holds no NUL and no lone surrogate: the contract's
+  checks refuse them (`INVALID`) in what is added, claimed and recorded,
+  since PostgreSQL cannot keep them, and an id holding one is unknown.
+  The queue never gives a store such text: it cleans every reply.
+
+The contract's specs, `describeQueueStore`, which every store here runs,
 are internal for now: a store written outside this package cannot run
 them yet.
 

package/docs/roadmap.md

@@ -6,26 +6,24 @@ only number on it.
 
 ## Now
 
-- **The first slice, in review** — `createQueue`: every recipient's own
-  state, delivery by domain through `@bumail/smtp/client` (direct MX, a
-  smarthost, or a route per domain), a global and a per-domain
-  concurrency limit, retries with exponential back-off and jitter on RFC
-  5321 §4.5.4.1's schedule, a 5xx failed at once, DSNs (RFC 3464) for a
-  failure and a delay, never about a message from `<>`; the `QueueStore`
-  contract with an atomic claim and leases for several workers, its
-  memory store and its `bun:sqlite` store, held to one spec suite;
-  events, `list`, `retryNow` and `cancel`; an injectable clock and sender.
+Nothing in progress: the MongoDB store, first under Next, is the next
+one taken. The Redis store is merged (see Shipped).
 
 ## Next
 
-- **A PostgreSQL store**, on `Bun.sql` — the same contract for workers on
-  several machines, the claim a `SELECT … FOR UPDATE SKIP LOCKED`. It
-  runs the same contract specs.
-- **A Redis store**, on `Bun.redis` — the due items in a sorted set, the
-  claim and the lease in one script.
-- **A MongoDB store** — the contract answered structurally, by a
+More answers to the `QueueStore` contract, built for several instances of
+a server sharing one queue — each claim atomic across them, each lease
+taken back from an instance that crashed — and each held to the same
+contract specs as the memory, `bun:sqlite`, PostgreSQL and Redis
+stores. None adds a dependency.
+
+- **A MongoDB store** — the contract answered structurally, against a
   collection of the shape a MongoDB driver gives, so the package peers on
-  no driver; the claim a `findOneAndUpdate`.
+  no driver: the application passes in its own collection; the claim a
+  `findOneAndUpdate`.
+
+Then:
+
 - **MTA-STS (RFC 8461) and TLS-RPT (RFC 8460)** — a recipient domain's
   policy fetched, cached and enforced, `mxTls: 'required'` with the
   certificate checked against the policy's MX names, and the reports.
@@ -54,8 +52,9 @@ only number on it.
 ## Not planned
 
 - **A runtime dependency.** The memory store needs none, the `bun:sqlite`
-  store uses Bun's own SQLite, and the stores to come use Bun's own
-  `Bun.sql` and `Bun.redis`.
+  store uses Bun's own SQLite, the PostgreSQL store Bun's own `Bun.sql`,
+  the Redis store Bun's own `Bun.redis`, and the MongoDB store will take
+  the application's own collection, typed by its shape.
 - **Relaying on its own.** The queue sends what the app enqueues; who may
   send is decided before, by the SMTP server's AUTH or the app. No option
   will make the queue accept mail from the network.
@@ -64,4 +63,59 @@ only number on it.
 
 ## Shipped
 
-Nothing yet: the first slice is in review (see Now).
+### Unreleased — merged, not yet published
+
+- **A Redis store**, `@bumail/queue/redis`, on Bun's own `Bun.redis`,
+  for instances of a server on several machines sharing one queue:
+  `RedisQueueStore.open({ client })` takes a `Bun.RedisClient`, or
+  `{ url }` a `redis://` URL, and needs no driver. Every operation that
+  writes is one Lua script, which Redis runs whole: two instances never
+  take the same item, a crashed instance's leases expire and its items
+  are claimed again, and `maxItems` holds across instances. The due
+  items are in sorted sets, each item a hash, its message written byte
+  for byte, every key under a `keyPrefix`. Held to the same contract
+  specs as the other stores, and to specs of two instances delivering
+  every item exactly once. One Redis, or a primary with replicas, not
+  Cluster; its durability is Redis's, which the guide spells out.
+- **A lease lost to an instance that finished the item is reported** —
+  an instance that stalled past its lease, while another claimed the
+  item, delivered it and dropped it, no longer takes the item's absence
+  for a cancel: it reports `LEASE_LOST` on the `error` event, tells no
+  outcome and sends no DSN, so the operator learns the message may have
+  gone out twice. With no record of who dropped an item, a lease that
+  expired, or a renewal refused while the item was still there, is
+  reported as lost or cancelled; a cancel under a lease that
+  still held is told as before, and no longer draws a `LEASE_LOST` from
+  a renewal that finds the item gone.
+
+### 0.2.0
+
+- **A PostgreSQL store**, `@bumail/queue/postgres`, on Bun's own
+  `Bun.sql`, for instances of a server on several machines sharing one
+  queue: `PostgresQueueStore.open({ sql })` takes a `Bun.SQL` client or a
+  `postgres://` URL, and needs no driver. A claim is one `UPDATE …
+  RETURNING` whose item a `SELECT … FOR UPDATE SKIP LOCKED` picks, so two
+  instances never take the same item; a crashed instance's leases expire
+  and its items are claimed again. The tables — the `bun:sqlite` store's,
+  the message as `bytea` — are made by `migrate()` or the first call,
+  once however many instances start together, under a `tablePrefix`.
+  Held to the same contract specs as the other stores, and to specs of
+  two instances delivering every item exactly once. Once the tables are
+  current, a worker's role needs no `CREATE`.
+- **A reply cut at its limit never splits a character** — the text kept
+  for a recipient ends before a surrogate pair rather than inside it, and
+  every store refuses a NUL or a lone surrogate in what it keeps, so an
+  outcome is always recorded.
+
+### 0.1.0
+
+- **The first slice** — `createQueue`: every recipient's own state,
+  delivery by domain through `@bumail/smtp/client` (direct MX, a
+  smarthost, or a route per domain), a global and a per-domain
+  concurrency limit, retries with exponential back-off and jitter on RFC
+  5321 §4.5.4.1's schedule, a 5xx failed at once, DSNs (RFC 3464) for a
+  failure and a delay, never about a message from `<>`; the `QueueStore`
+  contract with an atomic claim and leases for several workers, its
+  memory store and its `bun:sqlite` store, held to one spec suite;
+  events, `list`, `retryNow` and `cancel`; an injectable clock and sender.
+  The package's first release.