treadle 0.0.0-stage → 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Timileyin Pelumi
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,109 @@
-# Temporary Holding Version
+# Treadle
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+Background jobs and step workflows for Bun and Postgres. Jobs live in one table and commit inside your own transaction, so you can run a queue on the database you already have.
+
+    bun add treadle
+
+Needs Bun 1.3 or later and Postgres 12 or later. Uses Bun's built in SQL client; there are no runtime dependencies.
+
+## Example
+
+```ts
+import { SQL } from "bun";
+import { dashboard, migrate, Treadle, Worker } from "treadle";
+
+const sql = new SQL(process.env.DATABASE_URL!);
+await migrate(sql);
+const treadle = new Treadle(sql);
+
+// Enqueue inside your own transaction. The job commits with your data or not at all.
+await sql.begin(async (tx) => {
+  await tx`insert into withdrawals (id, amount) values (${id}, ${amount})`;
+  await treadle.enqueue(tx, "notify", { id }, { idempotencyKey: `notify:${id}` });
+  await treadle.startWorkflow(tx, "withdraw", { id });
+});
+
+const worker = new Worker(sql, { queues: ["default"], concurrency: 10 });
+worker.register("notify", async ({ id }) => { /* send the email */ });
+worker.registerWorkflow("withdraw", [
+  { name: "reserve", run: async ({ id }) => ({ holdId: await reserve(id) }) },
+  { name: "send",    run: async ({ id }, results) => ({ ref: await send(id, results.reserve.holdId) }) },
+  { name: "settle",  run: async ({ id }, results) => { await settle(id, results.send.ref); } },
+]);
+await worker.start();
+
+Bun.serve({ port: 3000, fetch: dashboard(sql, { basePath: "/admin/jobs" }) });
+```
+
+## Documentation
+
+- [Getting started](docs/getting-started.md)
+- [Concepts](docs/concepts.md): the guarantees and what they ask of your handlers
+- [Use cases](docs/use-cases/README.md): signup emails, webhooks, withdrawals, order fulfilment, periodic jobs, media, bulk import, rate-limited APIs, multi-tenant work, each with a runnable example
+- [Operations](docs/operations.md): running workers in production
+- [API reference](docs/api.md)
+- [When to use it, and when not to](docs/comparison.md)
+- [Changelog](CHANGELOG.md)
+
+## What it guarantees
+
+- A job enqueued inside a transaction runs at least once if that transaction commits, and never if it rolls back. The jobs table is the transactional outbox.
+- A workflow that crashes resumes at the first step without a persisted result. Completed steps are not run again.
+- Two enqueues with the same idempotency key produce one job, under any concurrency. A thousand simultaneous enqueues of one key yield one row.
+- A worker that dies mid-job loses at most one lease duration before another worker picks the job up.
+
+Execution is at least once. A handler can run twice if the process dies between the handler finishing and the completion write. Make side effects idempotent, or key them on `ctx.jobId` and `ctx.attempt`.
+
+## How it works
+
+Workers claim jobs with `select ... for update skip locked` in one statement that also sets a lease and a worker id, so any number of workers pull from one table without blocking or double-claiming. A heartbeat extends the lease while a handler runs; a rescuer returns expired leases to the queue. Every write after the claim is guarded by the worker id, so a worker that lost its lease cannot overwrite what a rescuer or another worker did. Failed jobs back off exponentially with jitter and are discarded after `maxAttempts`. Finishing a job, including re-enqueueing a periodic job or the next workflow step, is a single statement, so it commits atomically without a transaction round trip. The claim index is partial on claimable states, so completed rows do not slow claims down.
+
+## Numbers
+
+Measured on a laptop with Postgres on the same machine, one process per worker, empty handlers. Relative figures transfer; absolute ones depend on hardware.
+
+| Measurement | Result |
+|---|---|
+| 1 worker, concurrency 10 | 863 jobs/s |
+| 8 workers, concurrency 10 | 4750 jobs/s |
+| Claim latency, 10k rows | p50 2.6 ms, p99 6.4 ms |
+| Claim latency, 1M rows, 999k completed | p50 2.1 ms, p99 8.2 ms |
+| Claim latency, 1M rows all waiting | p50 2.4 ms, p99 7.3 ms |
+| Recovery after a worker is killed, 5 s lease | 5.1 s |
+| 1000 concurrent enqueues of one idempotency key | 1 row, 167 ms |
+
+One known slow case: a large backlog of jobs a worker has no handler for, in the same queue it claims from, makes each claim walk past them. Use a queue per worker type.
+
+## API
+
+- `migrate(sql)` creates or updates the `treadle` schema. Safe to call from every process at startup.
+- `new Treadle(sql)` with `enqueue(tx, name, args, options)`, `startWorkflow(tx, name, input, options)`, `cancel(jobId)`, `retry(jobId)`, `cancelWorkflow(runId)`.
+- Enqueue options: `queue`, `priority`, `runAt`, `maxAttempts`, `idempotencyKey`, `every` (milliseconds, for periodic jobs).
+- `new Worker(sql, options)` with `register(name, handler)`, `registerWorkflow(name, steps)`, `start()`, `stop()`. Options: `queues`, `concurrency`, `pollIntervalMs`, `leaseMs`, `heartbeatMs`, `rescueIntervalMs`, `stopTimeoutMs`, `backoff`, `onError`, `onEvent`.
+- Handlers receive `(args, ctx)` with `ctx.jobId`, `ctx.attempt`, `ctx.signal` (aborted on cancel or lost lease), and `ctx.heartbeat()`.
+- Steps receive `(input, results, ctx)` where `results` holds earlier steps' return values keyed by step name.
+- `dashboard(sql, { basePath })` returns a fetch handler. Mount it behind your own auth. `#run=<id>` in the URL opens that run.
+
+Job ids and run ids are strings, because Postgres bigint does not fit in a JavaScript number.
+
+## Benchmarks
+
+    WORKERS=4 JOBS=10000 bun run load
+    ROWS=1000000 SAMPLES=1000 bun run claim-latency
+    LEASE_MS=5000 bun run recovery
+    N=1000 bun run storm
+
+All use the `treadle_load` database on the same Postgres and drop the treadle schema there first.
+
+## Develop
+
+Needs a Postgres reachable at `DATABASE_URL`, default `postgres://postgres:postgres@127.0.0.1:5432/treadle_test`. `docker compose up -d` provides one.
+
+    bun install
+    bun test
+
+Tests run against the real database. Five of them kill a child worker process with SIGKILL at chosen points and check what recovery does.
+
+## Licence
+
+MIT

package/package.json

@@ -1,6 +1,47 @@
 {
   "name": "treadle",
-  "version": "0.0.0-stage",
-  "stub": true,
-  "description": "Temporary package placeholder for staged publishing"
-}
+  "version": "0.1.0",
+  "description": "Background jobs and step workflows for Bun and Postgres",
+  "type": "module",
+  "exports": {
+    ".": "./src/index.ts"
+  },
+  "types": "./src/index.ts",
+  "files": [
+    "src",
+    "README.md",
+    "LICENSE"
+  ],
+  "license": "MIT",
+  "author": "timileyinpelumi",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/timileyinpelumi/treadle.git"
+  },
+  "homepage": "https://github.com/timileyinpelumi/treadle",
+  "keywords": [
+    "bun",
+    "postgres",
+    "job-queue",
+    "background-jobs",
+    "workflow",
+    "outbox",
+    "skip-locked"
+  ],
+  "engines": {
+    "bun": ">=1.3.0"
+  },
+  "scripts": {
+    "test": "bun test --timeout 20000",
+    "typecheck": "bunx tsc --noEmit",
+    "load": "bun scripts/load.ts",
+    "claim-latency": "bun scripts/claim-latency.ts",
+    "recovery": "bun scripts/recovery.ts",
+    "storm": "bun scripts/storm.ts",
+    "prepublishOnly": "bun test && bun run typecheck"
+  },
+  "devDependencies": {
+    "@types/bun": "^1.3.14",
+    "typescript": "^5.6.0"
+  }
+}

package/src/backoff.ts

@@ -0,0 +1,6 @@
+const MAX_MS = 3_600_000;
+
+export function defaultBackoff(attempt: number): number {
+  const base = 2 ** attempt * 1000;
+  return Math.min(base + Math.random() * 1000, MAX_MS);
+}

package/src/dashboard-page.ts

@@ -0,0 +1,129 @@
+export const PAGE = `<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<title>Treadle</title>
+<style>
+  :root { --ink: #16181d; --mute: #6b7078; --rule: #d9dce2; --ground: #f6f7f9; --bad: #9b2c2c; --good: #2f6b4f; }
+  * { box-sizing: border-box; }
+  body { margin: 0; background: var(--ground); color: var(--ink); font: 14px/1.5 system-ui, -apple-system, "Segoe UI", Roboto, sans-serif; }
+  main { max-width: 72rem; margin: 0 auto; padding: 1.5rem; }
+  header { display: flex; justify-content: space-between; align-items: baseline; margin-bottom: 1.5rem; }
+  h1 { margin: 0; font-size: 1.25rem; font-weight: 600; }
+  header span { color: var(--mute); font-size: 0.85rem; }
+  h2 { margin: 2rem 0 0.5rem; font-size: 0.95rem; font-weight: 600; }
+  .states { display: grid; grid-template-columns: repeat(6, 1fr); border-top: 1px solid var(--ink); }
+  .states div { padding: 0.75rem 0; border-bottom: 1px solid var(--rule); }
+  .states b { display: block; font-size: 1.5rem; font-weight: 600; line-height: 1.1; }
+  .states span { color: var(--mute); font-size: 0.85rem; }
+  table { width: 100%; border-collapse: collapse; border-top: 1px solid var(--ink); }
+  th, td { text-align: left; padding: 0.5rem 0.5rem 0.5rem 0; border-bottom: 1px solid var(--rule); vertical-align: top; }
+  th { font-weight: 500; color: var(--mute); font-size: 0.85rem; }
+  td.num, th.num { text-align: right; padding-right: 1rem; }
+  .err { color: var(--bad); font-family: ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 0.8rem; white-space: pre-wrap; max-width: 40rem; }
+  button { font: inherit; font-size: 0.85rem; color: var(--ink); background: none; border: 1px solid var(--rule); border-radius: 3px; padding: 0.15rem 0.5rem; cursor: pointer; margin-right: 0.25rem; }
+  button:hover { border-color: var(--ink); }
+  .minutes { display: flex; align-items: flex-end; gap: 2px; height: 3rem; border-bottom: 1px solid var(--rule); padding-bottom: 2px; }
+  .minutes div { flex: 1; display: flex; flex-direction: column-reverse; height: 100%; }
+  .minutes i { display: block; background: var(--good); }
+  .minutes i.bad { background: var(--bad); }
+  .steps { margin: 0.25rem 0 0.5rem 1rem; }
+  .steps div { padding: 0.25rem 0; border-bottom: 1px dashed var(--rule); }
+  .state-completed { color: var(--good); } .state-discarded, .state-failed { color: var(--bad); } .state-cancelled { color: var(--mute); }
+  .empty { color: var(--mute); padding: 0.75rem 0; }
+  a.run { cursor: pointer; text-decoration: underline; }
+  @media (max-width: 640px) { .states { grid-template-columns: repeat(3, 1fr); } }
+</style>
+</head>
+<body>
+<main>
+  <header><h1>Treadle</h1><span id="updated">Loading</span></header>
+  <div class="states" id="states"></div>
+  <div id="queues"></div>
+  <h2>Finished per minute, last hour</h2>
+  <div class="minutes" id="minutes"></div>
+  <h2>Recent failures</h2>
+  <div id="failures"></div>
+  <h2>Workflow runs</h2>
+  <div id="runs"></div>
+</main>
+<script>
+  const STATES = ["available", "running", "retryable", "completed", "discarded", "cancelled"];
+  const open = new Set();
+  if (location.hash.startsWith("#run=")) open.add(location.hash.slice(5));
+  const esc = (s) => String(s ?? "").replace(/[&<>"]/g, (c) => ({ "&": "&amp;", "<": "&lt;", ">": "&gt;", '"': "&quot;" }[c]));
+  const el = (id) => document.getElementById(id);
+
+  async function act(path) {
+    const r = await fetch("api/" + path, { method: "POST" });
+    if (!r.ok) el("updated").textContent = "Action did not apply";
+    load();
+  }
+
+  function renderStates(counts) {
+    const total = {};
+    for (const c of counts) total[c.state] = (total[c.state] || 0) + c.n;
+    el("states").innerHTML = STATES.map((s) => "<div><b>" + (total[s] || 0) + "</b><span>" + s + "</span></div>").join("");
+    const queues = [...new Set(counts.map((c) => c.queue))];
+    if (queues.length < 2) { el("queues").innerHTML = ""; return; }
+    el("queues").innerHTML = "<table><tr><th>Queue</th>" + STATES.map((s) => '<th class="num">' + s + "</th>").join("") + "</tr>" +
+      queues.map((q) => "<tr><td>" + esc(q) + "</td>" + STATES.map((s) => '<td class="num">' + (counts.find((c) => c.queue === q && c.state === s)?.n || 0) + "</td>").join("") + "</tr>").join("") + "</table>";
+  }
+
+  function renderMinutes(rows) {
+    const now = Date.now();
+    const bins = [];
+    for (let i = 59; i >= 0; i--) {
+      const t = new Date(Math.floor(now / 60000) * 60000 - i * 60000).toISOString().slice(0, 17) + "00Z";
+      bins.push({ t, completed: 0, discarded: 0 });
+    }
+    for (const r of rows) { const b = bins.find((x) => x.t === r.minute); if (b) b[r.state] = r.n; }
+    const max = Math.max(1, ...bins.map((b) => b.completed + b.discarded));
+    el("minutes").innerHTML = bins.map((b) =>
+      '<div title="' + b.t.slice(11, 16) + ": " + b.completed + " completed, " + b.discarded + ' discarded">' +
+      '<i style="height:' + (b.completed / max * 100) + '%"></i><i class="bad" style="height:' + (b.discarded / max * 100) + '%"></i></div>').join("");
+  }
+
+  function renderFailures(rows) {
+    if (!rows.length) { el("failures").innerHTML = '<div class="empty">No failures on record.</div>'; return; }
+    el("failures").innerHTML = "<table><tr><th>Job</th><th>Queue</th><th>State</th><th class=\\"num\\">Attempt</th><th>Error</th><th></th></tr>" +
+      rows.map((r) => "<tr><td>" + esc(r.name) + " <span style=\\"color:var(--mute)\\">#" + r.id + "</span></td><td>" + esc(r.queue) + "</td>" +
+        '<td class="state-' + r.state + '">' + r.state + '</td><td class="num">' + r.attempt + " / " + r.max_attempts + "</td>" +
+        '<td class="err">' + esc(r.last_error.split("\\n")[0]) + "</td>" +
+        '<td><button onclick="act(\\'jobs/' + r.id + '/retry\\')">Retry now</button><button onclick="act(\\'jobs/' + r.id + '/cancel\\')">Cancel</button></td></tr>').join("") + "</table>";
+  }
+
+  async function renderRuns(rows) {
+    if (!rows.length) { el("runs").innerHTML = '<div class="empty">No workflow runs yet.</div>'; return; }
+    let html = "<table><tr><th>Run</th><th>State</th><th class=\\"num\\">Step</th><th>Started</th><th></th></tr>";
+    for (const r of rows) {
+      html += '<tr><td><a class="run" onclick="toggle(\\'' + r.id + '\\')">' + esc(r.name) + " #" + r.id + "</a></td>" +
+        '<td class="state-' + r.state + '">' + r.state + '</td><td class="num">' + r.current_step + "</td><td>" + esc(r.created_at.slice(0, 19)) + "</td>" +
+        "<td>" + (r.state === "running" ? '<button onclick="act(\\'workflows/' + r.id + '/cancel\\')">Cancel</button>' : "") + "</td></tr>";
+      if (open.has(r.id)) {
+        const d = await (await fetch("api/workflows/" + r.id)).json();
+        html += '<tr><td colspan="5"><div class="steps">' + d.steps.map((s) =>
+          "<div>Step " + s.step_index + ' <span class="state-' + s.state + '">' + s.state + "</span>, attempt " + s.attempt +
+          (s.has_result ? " <span style=\\"color:var(--mute)\\">result " + esc(JSON.stringify(s.result)) + "</span>" : "") +
+          (s.last_error ? '<div class="err">' + esc(s.last_error.split("\\n")[0]) + "</div>" : "") +
+          (s.state === "discarded" ? ' <button onclick="act(\\'jobs/' + s.job_id + '/retry\\')">Retry step</button>' : "") + "</div>").join("") + "</div></td></tr>";
+      }
+    }
+    el("runs").innerHTML = html + "</table>";
+  }
+
+  function toggle(id) { open.has(id) ? open.delete(id) : open.add(id); load(); }
+
+  async function load() {
+    try {
+      const d = await (await fetch("api/overview")).json();
+      renderStates(d.counts); renderMinutes(d.perMinute); renderFailures(d.failures); await renderRuns(d.runs);
+      el("updated").textContent = "Updated " + new Date().toLocaleTimeString();
+    } catch (e) { el("updated").textContent = "Could not reach the API"; }
+  }
+  load();
+  setInterval(() => { if (!document.hidden) load(); }, 3000);
+</script>
+</body>
+</html>`;

package/src/dashboard-queries.ts

@@ -0,0 +1,62 @@
+import type { Sql } from "./sql";
+
+export interface CountRow { queue: string; state: string; n: number }
+export interface MinuteRow { minute: string; state: string; n: number }
+export interface FailureRow {
+  id: string; queue: string; name: string; state: string; attempt: number;
+  max_attempts: number; last_error: string; at: string;
+}
+export interface RunRow {
+  id: string; name: string; state: string; current_step: number; created_at: string; finished_at: string | null;
+}
+export interface StepRow {
+  step_index: number; job_id: string; state: string; attempt: number; last_error: string | null;
+  result: unknown; has_result: boolean;
+}
+
+export async function overview(sql: Sql): Promise<{ counts: CountRow[]; perMinute: MinuteRow[] }> {
+  const counts = await sql`
+    select queue, state, count(*)::int as n from treadle.jobs
+    group by queue, state order by queue, state`;
+  const perMinute = await sql`
+    select to_char(date_trunc('minute', finished_at at time zone 'UTC'), 'YYYY-MM-DD"T"HH24:MI:00"Z"') as minute, state, count(*)::int as n
+    from treadle.jobs
+    where finished_at > now() - interval '1 hour' and state in ('completed', 'discarded')
+    group by 1, 2 order by 1, 2`;
+  return { counts: counts as CountRow[], perMinute: perMinute as MinuteRow[] };
+}
+
+export async function recentFailures(sql: Sql, limit = 50): Promise<FailureRow[]> {
+  return (await sql`
+    select id::text as id, queue, name, state, attempt, max_attempts, last_error,
+           coalesce(finished_at, run_at)::text as at
+    from treadle.jobs
+    where last_error is not null and state in ('retryable', 'discarded')
+    order by coalesce(finished_at, run_at) desc, id desc
+    limit ${limit}`) as FailureRow[];
+}
+
+export async function workflowRuns(sql: Sql, limit = 50): Promise<RunRow[]> {
+  return (await sql`
+    select id::text as id, name, state, current_step, created_at::text as created_at, finished_at::text as finished_at
+    from treadle.workflow_runs order by id desc limit ${limit}`) as RunRow[];
+}
+
+export async function workflowRun(
+  sql: Sql,
+  id: string,
+): Promise<{ run: RunRow & { input: unknown }; steps: StepRow[] } | null> {
+  if (!/^\d+$/.test(id)) return null;
+  const [run] = await sql`
+    select id::text as id, name, state, current_step, input, created_at::text as created_at, finished_at::text as finished_at
+    from treadle.workflow_runs where id = ${id}::bigint`;
+  if (!run) return null;
+  const steps = await sql`
+    select j.step_index, j.id::text as job_id, j.state, j.attempt, j.last_error,
+           r.result, (r.workflow_run_id is not null) as has_result
+    from treadle.jobs j
+    left join treadle.step_results r on r.workflow_run_id = j.workflow_run_id and r.step_index = j.step_index
+    where j.workflow_run_id = ${id}::bigint
+    order by j.step_index`;
+  return { run: run as RunRow & { input: unknown }, steps: steps as StepRow[] };
+}

package/src/dashboard.ts

@@ -0,0 +1,51 @@
+import { PAGE } from "./dashboard-page";
+import { overview, recentFailures, workflowRun, workflowRuns } from "./dashboard-queries";
+import type { Sql } from "./sql";
+import { Treadle } from "./treadle";
+
+export interface DashboardOptions {
+  basePath?: string;
+}
+
+const json = (body: unknown, status = 200): Response =>
+  new Response(JSON.stringify(body), { status, headers: { "content-type": "application/json" } });
+
+export function dashboard(sql: Sql, options: DashboardOptions = {}) {
+  const base = (options.basePath ?? "").replace(/\/+$/, "");
+  const treadle = new Treadle(sql);
+
+  return async (req: Request): Promise<Response> => {
+    const url = new URL(req.url);
+    if (base && !url.pathname.startsWith(base)) return new Response("Not found", { status: 404 });
+    const path = url.pathname.slice(base.length);
+
+    if (path === "") return Response.redirect(`${base}/`, 308);
+    if (path === "/") {
+      return new Response(PAGE, { headers: { "content-type": "text/html; charset=utf-8" } });
+    }
+
+    if (path === "/api/overview") {
+      if (req.method !== "GET") return new Response("Method not allowed", { status: 405 });
+      const [o, failures, runs] = await Promise.all([overview(sql), recentFailures(sql), workflowRuns(sql)]);
+      return json({ ...o, failures, runs });
+    }
+
+    let m: RegExpMatchArray | null;
+    if ((m = path.match(/^\/api\/workflows\/([^/]+)$/))) {
+      if (req.method !== "GET") return new Response("Method not allowed", { status: 405 });
+      const detail = await workflowRun(sql, m[1]!);
+      return detail ? json(detail) : json({ error: "not found" }, 404);
+    }
+    if ((m = path.match(/^\/api\/(jobs|workflows)\/(\d+)\/(retry|cancel)$/))) {
+      if (req.method !== "POST") return new Response("Method not allowed", { status: 405 });
+      const [, kind, id, action] = m;
+      let ok = false;
+      if (kind === "jobs" && action === "retry") ok = await treadle.retry(id!);
+      else if (kind === "jobs" && action === "cancel") ok = await treadle.cancel(id!);
+      else if (kind === "workflows" && action === "cancel") ok = await treadle.cancelWorkflow(id!);
+      else return json({ error: "not found" }, 404);
+      return ok ? json({ ok: true }) : json({ ok: false }, 404);
+    }
+    return new Response("Not found", { status: 404 });
+  };
+}

package/src/index.ts

@@ -0,0 +1,9 @@
+export type { Sql } from "./sql";
+export type { EnqueueOptions, Handler, Job, JobContext, JobState, StartWorkflowOptions, Step, WorkerEvent, WorkerOptions } from "./types";
+export { migrate } from "./migrate";
+export { Treadle } from "./treadle";
+export { Worker } from "./worker";
+export { defaultBackoff } from "./backoff";
+export { workflowJobName } from "./workflows";
+export { dashboard } from "./dashboard";
+export type { DashboardOptions } from "./dashboard";

package/src/migrate.ts

@@ -0,0 +1,24 @@
+import type { Sql } from "./sql";
+import { migrations } from "./schema";
+
+export async function migrate(sql: Sql): Promise<void> {
+  await sql`create schema if not exists treadle`;
+  await sql`
+    create table if not exists treadle.migrations (
+      version     integer primary key,
+      applied_at  timestamptz not null default now()
+    )`;
+
+  await sql.begin(async (tx) => {
+    // Serialise concurrent migrate calls from several workers starting at once.
+    await tx`select pg_advisory_xact_lock(7231001)`;
+    const applied = new Set(
+      (await tx`select version from treadle.migrations`).map((r: { version: number }) => r.version),
+    );
+    for (const m of migrations) {
+      if (applied.has(m.version)) continue;
+      await tx.unsafe(m.sql);
+      await tx`insert into treadle.migrations (version) values (${m.version})`;
+    }
+  });
+}

package/src/queries.ts

@@ -0,0 +1,166 @@
+import type { Sql } from "./sql";
+import type { Job } from "./types";
+
+const columns = `
+  j.id::text as id, j.queue, j.name, j.args, j.state, j.priority, j.run_at,
+  j.attempt, j.max_attempts, j.lease_until, j.worker_id, j.idempotency_key,
+  j.every_ms, j.workflow_run_id::text as workflow_run_id, j.step_index,
+  j.last_error, j.created_at, j.started_at, j.finished_at`;
+
+// One statement per queue. With queue = any(array) Postgres cannot use the claim
+// index for ordering and falls back to a seq scan plus sort over the whole backlog.
+export async function claimJobs(
+  sql: Sql,
+  args: { queues: string[]; names: string[]; limit: number; leaseMs: number; workerId: string },
+): Promise<Job[]> {
+  if (args.limit < 1 || args.queues.length === 0 || args.names.length === 0) return [];
+  const claimed: Job[] = [];
+  for (const queue of args.queues) {
+    const remaining = args.limit - claimed.length;
+    if (remaining < 1) break;
+    const rows: Job[] = await sql.unsafe(
+      `with next as (
+         select id from treadle.jobs
+         where state in ('available', 'retryable')
+           and queue = $1
+           and name = any($2::text[])
+           and run_at <= now()
+         order by priority, run_at
+         limit $3
+         for update skip locked
+       )
+       update treadle.jobs j
+       set state = 'running',
+           attempt = j.attempt + 1,
+           lease_until = now() + ($4 * interval '1 millisecond'),
+           worker_id = $5,
+           started_at = coalesce(j.started_at, now())
+       from next
+       where j.id = next.id
+       returning ${columns}`,
+      [queue, toPgArray(args.names), remaining, args.leaseMs, args.workerId],
+    );
+    claimed.push(...rows);
+  }
+  return claimed;
+}
+
+// Bun sends JS arrays as a bare "a,b" literal, which Postgres rejects. Build the {} form ourselves.
+function toPgArray(items: string[]): string {
+  return "{" + items.map((s) => '"' + s.replace(/(["\\])/g, "\\$1") + '"').join(",") + "}";
+}
+
+export type FinalState = "completed" | "retryable" | "discarded" | "cancelled";
+
+// Finishing is one statement: the update and, for periodic jobs, the insert of the
+// next occurrence run as data-modifying CTEs and commit together. Wrapping these in
+// sql.begin instead made Bun's pool hand one caller another caller's rows under load.
+export async function completeJob(sql: Sql, jobId: string, workerId: string): Promise<FinalState | null> {
+  const rows = await sql`
+    with done as (
+      update treadle.jobs
+      set state = case when cancel_requested then 'cancelled' else 'completed' end,
+          finished_at = now(), lease_until = null, worker_id = null
+      where id = ${jobId} and state = 'running' and worker_id = ${workerId}
+      returning state, every_ms, queue, name, args, priority, max_attempts, workflow_run_id
+    ), next as (
+      insert into treadle.jobs (queue, name, args, priority, max_attempts, every_ms, run_at)
+      select queue, name, args, priority, max_attempts, every_ms,
+             now() + (every_ms * interval '1 millisecond')
+      from done where every_ms is not null and state = 'completed'
+    ), run as (
+      update treadle.workflow_runs w
+      set state = 'cancelled', finished_at = now()
+      from done
+      where w.id = done.workflow_run_id and done.state = 'cancelled' and w.state = 'running'
+    )
+    select state from done`;
+  return (rows[0]?.state as FinalState | undefined) ?? null;
+}
+
+export async function failJob(
+  sql: Sql,
+  jobId: string,
+  workerId: string,
+  error: string,
+  backoffMs: number,
+): Promise<FinalState | null> {
+  const rows = await sql`
+    with done as (
+      update treadle.jobs
+      set state = case
+            when cancel_requested then 'cancelled'
+            when attempt >= max_attempts then 'discarded'
+            else 'retryable' end,
+          last_error = ${error},
+          run_at = case
+            when cancel_requested or attempt >= max_attempts then run_at
+            else now() + (${backoffMs} * interval '1 millisecond') end,
+          finished_at = case
+            when cancel_requested or attempt >= max_attempts then now()
+            else null end,
+          lease_until = null, worker_id = null
+      where id = ${jobId} and state = 'running' and worker_id = ${workerId}
+      returning state, every_ms, queue, name, args, priority, max_attempts, workflow_run_id
+    ), next as (
+      insert into treadle.jobs (queue, name, args, priority, max_attempts, every_ms, run_at)
+      select queue, name, args, priority, max_attempts, every_ms,
+             now() + (every_ms * interval '1 millisecond')
+      from done where every_ms is not null and state = 'discarded'
+    ), run as (
+      update treadle.workflow_runs w
+      set state = case done.state when 'discarded' then 'failed' else 'cancelled' end,
+          finished_at = now()
+      from done
+      where w.id = done.workflow_run_id and done.state in ('discarded', 'cancelled') and w.state = 'running'
+    )
+    select state from done`;
+  return (rows[0]?.state as FinalState | undefined) ?? null;
+}
+
+export async function extendLease(
+  sql: Sql,
+  jobId: string,
+  workerId: string,
+  leaseMs: number,
+): Promise<{ held: boolean; cancelRequested: boolean }> {
+  const rows = await sql`
+    update treadle.jobs
+    set lease_until = now() + (${leaseMs} * interval '1 millisecond')
+    where id = ${jobId} and state = 'running' and worker_id = ${workerId}
+    returning cancel_requested`;
+  const row = rows[0] as { cancel_requested: boolean } | undefined;
+  return { held: row !== undefined, cancelRequested: row?.cancel_requested ?? false };
+}
+
+export async function rescueExpired(sql: Sql): Promise<number> {
+  const rows = await sql`
+    with done as (
+      update treadle.jobs
+      set state = case
+            when cancel_requested then 'cancelled'
+            when attempt >= max_attempts then 'discarded'
+            else 'retryable' end,
+          finished_at = case
+            when cancel_requested or attempt >= max_attempts then now()
+            else finished_at end,
+          -- the worker died, the job did not fail, so no backoff
+          run_at = now(),
+          lease_until = null, worker_id = null, last_error = 'lease expired'
+      where state = 'running' and lease_until < now()
+      returning state, every_ms, queue, name, args, priority, max_attempts, workflow_run_id
+    ), next as (
+      insert into treadle.jobs (queue, name, args, priority, max_attempts, every_ms, run_at)
+      select queue, name, args, priority, max_attempts, every_ms,
+             now() + (every_ms * interval '1 millisecond')
+      from done where every_ms is not null and state = 'discarded'
+    ), run as (
+      update treadle.workflow_runs w
+      set state = case done.state when 'discarded' then 'failed' else 'cancelled' end,
+          finished_at = now()
+      from done
+      where w.id = done.workflow_run_id and done.state in ('discarded', 'cancelled') and w.state = 'running'
+    )
+    select count(*)::int as n from done`;
+  return (rows[0]?.n as number) ?? 0;
+}

package/src/schema.ts

@@ -0,0 +1,58 @@
+export const migrations: ReadonlyArray<{ version: number; sql: string }> = [
+  {
+    version: 1,
+    sql: `
+      create table treadle.jobs (
+        id               bigint generated always as identity primary key,
+        queue            text not null default 'default',
+        name             text not null,
+        args             jsonb not null default '{}'::jsonb,
+        state            text not null default 'available'
+                         check (state in ('available','running','completed','retryable','discarded','cancelled')),
+        priority         smallint not null default 0,
+        run_at           timestamptz not null default now(),
+        attempt          smallint not null default 0,
+        max_attempts     smallint not null default 25,
+        lease_until      timestamptz,
+        worker_id        text,
+        idempotency_key  text,
+        every_ms         integer,
+        workflow_run_id  bigint,
+        step_index       smallint,
+        last_error       text,
+        cancel_requested boolean not null default false,
+        created_at       timestamptz not null default now(),
+        started_at       timestamptz,
+        finished_at      timestamptz
+      );
+
+      create index jobs_claim_idx on treadle.jobs (queue, priority, run_at)
+        where state in ('available', 'retryable');
+
+      create index jobs_rescue_idx on treadle.jobs (lease_until)
+        where state = 'running';
+
+      create unique index jobs_idempotency_key_idx on treadle.jobs (idempotency_key)
+        where idempotency_key is not null;
+
+      create table treadle.workflow_runs (
+        id            bigint generated always as identity primary key,
+        name          text not null,
+        input         jsonb not null default '{}'::jsonb,
+        state         text not null default 'running'
+                      check (state in ('running','completed','failed','cancelled')),
+        current_step  smallint not null default 0,
+        created_at    timestamptz not null default now(),
+        finished_at   timestamptz
+      );
+
+      create table treadle.step_results (
+        workflow_run_id  bigint not null references treadle.workflow_runs (id) on delete cascade,
+        step_index       smallint not null,
+        result           jsonb,
+        finished_at      timestamptz not null default now(),
+        primary key (workflow_run_id, step_index)
+      );
+    `,
+  },
+];

package/src/sql.ts

@@ -0,0 +1,3 @@
+import type { SQL } from "bun";
+
+export type Sql = SQL;

package/src/treadle.ts

@@ -0,0 +1,95 @@
+import type { Sql } from "./sql";
+import type { EnqueueOptions, StartWorkflowOptions } from "./types";
+import { cancelWorkflowRun, startWorkflowRun } from "./workflows";
+
+export class Treadle {
+  constructor(readonly sql: Sql) {}
+
+  async enqueue(
+    tx: Sql,
+    name: string,
+    args: unknown = {},
+    options: EnqueueOptions = {},
+  ): Promise<string> {
+    if (!name) throw new Error("enqueue: name is required");
+    const maxAttempts = options.maxAttempts ?? 25;
+    if (!Number.isInteger(maxAttempts) || maxAttempts < 1) {
+      throw new Error("enqueue: maxAttempts must be a positive integer");
+    }
+    if (args !== null && typeof args !== "object") {
+      throw new Error("enqueue: args must be an object, an array, or null");
+    }
+    if (options.every !== undefined && (!Number.isInteger(options.every) || options.every < 1)) {
+      throw new Error("enqueue: every must be a positive integer of milliseconds");
+    }
+
+    const rows = await tx`
+      insert into treadle.jobs
+        (queue, name, args, priority, run_at, max_attempts, idempotency_key, every_ms)
+      values (
+        ${options.queue ?? "default"},
+        ${name},
+        ${args ?? {}}::jsonb,
+        ${options.priority ?? 0},
+        ${options.runAt ?? new Date()},
+        ${maxAttempts},
+        ${options.idempotencyKey ?? null},
+        ${options.every ?? null}
+      )
+      on conflict (idempotency_key) where idempotency_key is not null do nothing
+      returning id::text as id`;
+
+    if (rows.length > 0) return rows[0]!.id as string;
+
+    const existing = await tx`
+      select id::text as id from treadle.jobs
+      where idempotency_key = ${options.idempotencyKey!}`;
+    return existing[0]!.id as string;
+  }
+
+  async cancel(jobId: string): Promise<boolean> {
+    const rows = await this.sql`
+      update treadle.jobs
+      set state = case when state = 'running' then state else 'cancelled' end,
+          cancel_requested = case when state = 'running' then true else cancel_requested end,
+          finished_at = case when state = 'running' then finished_at else now() end
+      where id = ${jobId} and state in ('available', 'retryable', 'running')
+      returning id`;
+    return rows.length === 1;
+  }
+
+  async retry(jobId: string): Promise<boolean> {
+    const rows = await this.sql`
+      with j as (
+        update treadle.jobs
+        set state = 'available', attempt = 0, run_at = now(), cancel_requested = false,
+            finished_at = null, lease_until = null, worker_id = null
+        where id = ${jobId} and state in ('discarded', 'cancelled', 'retryable')
+        returning id, workflow_run_id
+      ), run as (
+        update treadle.workflow_runs w
+        set state = 'running', finished_at = null
+        from j
+        where w.id = j.workflow_run_id and w.state in ('failed', 'cancelled')
+      )
+      select id from j`;
+    return rows.length === 1;
+  }
+
+  async startWorkflow(
+    tx: Sql,
+    name: string,
+    input: unknown = {},
+    options: StartWorkflowOptions = {},
+  ): Promise<string> {
+    if (!name) throw new Error("startWorkflow: name is required");
+    if (input !== null && typeof input !== "object") {
+      throw new Error("startWorkflow: input must be an object, an array, or null");
+    }
+    return startWorkflowRun(tx, name, input, options);
+  }
+
+  async cancelWorkflow(runId: string): Promise<boolean> {
+    return cancelWorkflowRun(this.sql, runId);
+  }
+}

package/src/types.ts

@@ -0,0 +1,77 @@
+export type JobState =
+  | "available"
+  | "running"
+  | "completed"
+  | "retryable"
+  | "discarded"
+  | "cancelled";
+
+export interface EnqueueOptions {
+  queue?: string;
+  priority?: number;
+  runAt?: Date;
+  maxAttempts?: number;
+  idempotencyKey?: string;
+  every?: number;
+}
+
+export interface Job {
+  id: string;
+  queue: string;
+  name: string;
+  args: unknown;
+  state: JobState;
+  priority: number;
+  run_at: Date;
+  attempt: number;
+  max_attempts: number;
+  lease_until: Date | null;
+  worker_id: string | null;
+  idempotency_key: string | null;
+  every_ms: number | null;
+  workflow_run_id: string | null;
+  step_index: number | null;
+  last_error: string | null;
+  created_at: Date;
+  started_at: Date | null;
+  finished_at: Date | null;
+}
+
+export interface JobContext {
+  jobId: string;
+  name: string;
+  queue: string;
+  attempt: number;
+  signal: AbortSignal;
+  heartbeat(): Promise<void>;
+}
+
+export type Handler<A = any> = (args: A, ctx: JobContext) => Promise<unknown> | unknown;
+
+export interface WorkerOptions {
+  queues?: string[];
+  concurrency?: number;
+  pollIntervalMs?: number;
+  leaseMs?: number;
+  heartbeatMs?: number;
+  rescueIntervalMs?: number;
+  stopTimeoutMs?: number;
+  workerId?: string;
+  backoff?: (attempt: number) => number;
+  onError?: (error: unknown, job?: Job) => void;
+  onEvent?: (event: WorkerEvent, job: Job) => void;
+}
+
+export interface StartWorkflowOptions {
+  queue?: string;
+  priority?: number;
+  maxAttempts?: number;
+  idempotencyKey?: string;
+}
+
+export interface Step<I = any> {
+  name: string;
+  run: (input: I, results: Record<string, any>, ctx: JobContext) => Promise<unknown> | unknown;
+}
+
+export type WorkerEvent = "claimed" | "finishing" | "completed" | "failed" | "stepFinished" | "leaseLost";

package/src/worker.ts

@@ -0,0 +1,205 @@
+import { hostname } from "node:os";
+import { defaultBackoff } from "./backoff";
+import { claimJobs, completeJob, extendLease, failJob, rescueExpired } from "./queries";
+import type { Sql } from "./sql";
+import type { Handler, Job, JobContext, Step, WorkerEvent, WorkerOptions } from "./types";
+import { loadRun, stepDone, workflowJobName } from "./workflows";
+
+type InternalHandler = (job: Job, ctx: JobContext) => Promise<unknown>;
+
+export class Worker {
+  readonly id: string;
+  readonly options: Required<Omit<WorkerOptions, "workerId" | "onError" | "backoff" | "onEvent">> & {
+    backoff: (attempt: number) => number;
+    onError: (error: unknown, job?: Job) => void;
+    onEvent: (event: WorkerEvent, job: Job) => void;
+  };
+
+  private handlers = new Map<string, InternalHandler>();
+  private inflight = new Map<string, Promise<void>>();
+  private running = false;
+  private loop: Promise<void> | null = null;
+  private rescueTimer: Timer | null = null;
+  private wake: (() => void) | null = null;
+
+  constructor(readonly sql: Sql, options: WorkerOptions = {}) {
+    this.id = options.workerId ?? `${hostname()}:${process.pid}:${Math.random().toString(36).slice(2, 8)}`;
+    this.options = {
+      queues: options.queues ?? ["default"],
+      concurrency: options.concurrency ?? 10,
+      pollIntervalMs: options.pollIntervalMs ?? 1000,
+      leaseMs: options.leaseMs ?? 30_000,
+      heartbeatMs: options.heartbeatMs ?? 10_000,
+      rescueIntervalMs: options.rescueIntervalMs ?? 15_000,
+      stopTimeoutMs: options.stopTimeoutMs ?? 30_000,
+      backoff: options.backoff ?? defaultBackoff,
+      onEvent: options.onEvent ?? (() => {}),
+      onError: options.onError ?? ((error, job) => console.error("treadle worker error", { jobId: job?.id, error })),
+    };
+  }
+
+  register(name: string, handler: Handler): this {
+    this.assertNotRunning();
+    this.handlers.set(name, async (job, ctx) => handler(job.args, ctx));
+    return this;
+  }
+
+  registerWorkflow(name: string, steps: Step[]): this {
+    this.assertNotRunning();
+    if (steps.length === 0) throw new Error("registerWorkflow: a workflow needs at least one step");
+    if (new Set(steps.map((s) => s.name)).size !== steps.length) {
+      throw new Error("registerWorkflow: step names must be unique");
+    }
+    this.handlers.set(workflowJobName(name), (job, ctx) => this.runStep(steps, job, ctx));
+    return this;
+  }
+
+  private assertNotRunning(): void {
+    if (this.running) throw new Error("cannot register handlers after start");
+  }
+
+  private async runStep(steps: Step[], job: Job, ctx: JobContext): Promise<void> {
+    const runId = job.workflow_run_id!;
+    const index = job.step_index!;
+    const step = steps[index];
+    if (!step) throw new Error(`workflow step ${index} is not defined`);
+    const run = await loadRun(this.sql, runId);
+    if (!run) throw new Error(`workflow run ${runId} not found`);
+    if (run.state !== "running") return;
+
+    let result: unknown;
+    if (run.results.has(index)) {
+      // The step finished before a crash but its job was never marked done. Do not run it again.
+      result = run.results.get(index);
+    } else {
+      const results: Record<string, unknown> = {};
+      for (let i = 0; i < index; i++) results[steps[i]!.name] = run.results.get(i);
+      result = await step.run(run.input, results, ctx);
+    }
+    await stepDone(this.sql, {
+      runId,
+      jobId: job.id,
+      stepIndex: index,
+      result: result === undefined ? null : result,
+      isLast: index === steps.length - 1,
+    });
+    this.emit("stepFinished", job);
+  }
+
+  private emit(event: WorkerEvent, job: Job): void {
+    try {
+      this.options.onEvent(event, job);
+    } catch (e) {
+      this.options.onError(e, job);
+    }
+  }
+
+  async start(): Promise<void> {
+    if (this.running) return;
+    this.running = true;
+    this.rescueTimer = setInterval(() => {
+      rescueExpired(this.sql).catch((e) => this.options.onError(e));
+    }, this.options.rescueIntervalMs);
+    this.rescueTimer.unref();
+    this.loop = this.runLoop();
+  }
+
+  async stop(): Promise<void> {
+    if (!this.running) return;
+    this.running = false;
+    if (this.rescueTimer) clearInterval(this.rescueTimer);
+    this.wakeUp();
+    await this.loop;
+    const drained = Promise.allSettled([...this.inflight.values()]);
+    await Promise.race([drained, Bun.sleep(this.options.stopTimeoutMs)]);
+  }
+
+  private async runLoop(): Promise<void> {
+    while (this.running) {
+      const free = this.options.concurrency - this.inflight.size;
+      let claimed = 0;
+      if (free > 0) {
+        try {
+          const jobs = await claimJobs(this.sql, {
+            queues: this.options.queues,
+            names: [...this.handlers.keys()],
+            limit: free,
+            leaseMs: this.options.leaseMs,
+            workerId: this.id,
+          });
+          claimed = jobs.length;
+          for (const job of jobs) {
+            this.emit("claimed", job);
+            this.track(job);
+          }
+        } catch (e) {
+          this.options.onError(e);
+        }
+      }
+      // A full batch means more may be waiting; poll again at once. Otherwise
+      // sleep until the interval passes, a slot frees up, or stop() is called.
+      if (claimed < free || free === 0) await this.sleep(this.options.pollIntervalMs);
+    }
+  }
+
+  private track(job: Job): void {
+    const p = this.run(job).finally(() => {
+      this.inflight.delete(job.id);
+      this.wakeUp();
+    });
+    this.inflight.set(job.id, p);
+  }
+
+  private async run(job: Job): Promise<void> {
+    const handler = this.handlers.get(job.name)!;
+    const abort = new AbortController();
+    const ctx: JobContext = {
+      jobId: job.id,
+      name: job.name,
+      queue: job.queue,
+      attempt: job.attempt,
+      signal: abort.signal,
+      heartbeat: () => this.heartbeat(job, abort),
+    };
+    const beat = setInterval(() => {
+      ctx.heartbeat().catch((e) => this.options.onError(e, job));
+    }, this.options.heartbeatMs);
+    try {
+      await handler(job, ctx);
+      this.emit("finishing", job);
+      const state = await completeJob(this.sql, job.id, this.id);
+      if (state === null) {
+        this.emit("leaseLost", job);
+        this.options.onError(new Error("lease lost before completion"), job);
+      } else {
+        this.emit("completed", job);
+      }
+    } catch (error) {
+      this.options.onError(error, job);
+      const message = error instanceof Error ? (error.stack ?? error.message) : String(error);
+      await failJob(this.sql, job.id, this.id, message, this.options.backoff(job.attempt))
+        .then(() => this.emit("failed", job))
+        .catch((e) => this.options.onError(e, job));
+    } finally {
+      clearInterval(beat);
+    }
+  }
+
+  private async heartbeat(job: Job, abort: AbortController): Promise<void> {
+    const { held, cancelRequested } = await extendLease(this.sql, job.id, this.id, this.options.leaseMs);
+    if (abort.signal.aborted) return;
+    if (!held) abort.abort(new Error("lease lost"));
+    else if (cancelRequested) abort.abort(new Error("cancelled"));
+  }
+
+  private sleep(ms: number): Promise<void> {
+    return new Promise((resolve) => {
+      const timer = setTimeout(() => { this.wake = null; resolve(); }, ms);
+      this.wake = () => { clearTimeout(timer); this.wake = null; resolve(); };
+    });
+  }
+
+  private wakeUp(): void {
+    this.wake?.();
+  }
+}

package/src/workflows.ts

@@ -0,0 +1,104 @@
+import type { Sql } from "./sql";
+import type { StartWorkflowOptions } from "./types";
+
+export function workflowJobName(name: string): string {
+  return `workflow:${name}`;
+}
+
+export function stepKey(runId: string, stepIndex: number): string {
+  return `wf:${runId}:${stepIndex}`;
+}
+
+export async function startWorkflowRun(
+  tx: Sql,
+  name: string,
+  input: unknown,
+  options: StartWorkflowOptions,
+): Promise<string> {
+  if (options.idempotencyKey) {
+    const existing = await tx`
+      select workflow_run_id::text as run from treadle.jobs
+      where idempotency_key = ${options.idempotencyKey}`;
+    if (existing[0]) return existing[0].run as string;
+  }
+  const [run] = await tx`
+    insert into treadle.workflow_runs (name, input)
+    values (${name}, ${input ?? {}}::jsonb)
+    returning id::text as id`;
+  const runId = run!.id as string;
+  const rows = await tx`
+    insert into treadle.jobs
+      (queue, name, args, priority, max_attempts, workflow_run_id, step_index, idempotency_key)
+    values (${options.queue ?? "default"}, ${workflowJobName(name)}, '{}'::jsonb,
+            ${options.priority ?? 0}, ${options.maxAttempts ?? 25}, ${runId}::bigint, 0,
+            ${options.idempotencyKey ?? stepKey(runId, 0)})
+    on conflict (idempotency_key) where idempotency_key is not null do nothing
+    returning id`;
+  if (rows.length === 0) {
+    // Lost a race on the idempotency key between the select above and this insert.
+    await tx`delete from treadle.workflow_runs where id = ${runId}::bigint`;
+    const [winner] = await tx`
+      select workflow_run_id::text as run from treadle.jobs
+      where idempotency_key = ${options.idempotencyKey!}`;
+    return winner!.run as string;
+  }
+  return runId;
+}
+
+export async function loadRun(
+  sql: Sql,
+  runId: string,
+): Promise<{ input: unknown; state: string; results: Map<number, unknown> } | null> {
+  const [run] = await sql`select input, state from treadle.workflow_runs where id = ${runId}::bigint`;
+  if (!run) return null;
+  const rows = await sql`
+    select step_index, result from treadle.step_results
+    where workflow_run_id = ${runId}::bigint order by step_index`;
+  const results = new Map<number, unknown>();
+  for (const r of rows as { step_index: number; result: unknown }[]) results.set(r.step_index, r.result);
+  return { input: run.input, state: run.state as string, results };
+}
+
+export async function stepDone(
+  sql: Sql,
+  args: { runId: string; jobId: string; stepIndex: number; result: unknown; isLast: boolean },
+): Promise<void> {
+  const next = args.stepIndex + 1;
+  await sql`
+    with saved as (
+      insert into treadle.step_results (workflow_run_id, step_index, result)
+      values (${args.runId}::bigint, ${args.stepIndex}::int, ${JSON.stringify(args.result ?? null)}::text::jsonb)
+      on conflict do nothing
+    ), enqueued as (
+      insert into treadle.jobs
+        (queue, name, args, priority, max_attempts, workflow_run_id, step_index, idempotency_key)
+      select j.queue, j.name, '{}'::jsonb, j.priority, j.max_attempts, j.workflow_run_id,
+             ${next}::int, ${stepKey(args.runId, next)}
+      from treadle.jobs j
+      where j.id = ${args.jobId}::bigint and ${args.isLast}::boolean = false
+      on conflict (idempotency_key) where idempotency_key is not null do nothing
+    )
+    update treadle.workflow_runs
+    set current_step = case when ${args.isLast}::boolean then current_step else ${next}::int end,
+        state = case when ${args.isLast}::boolean and state = 'running' then 'completed' else state end,
+        finished_at = case when ${args.isLast}::boolean and state = 'running' then now() else finished_at end
+    where id = ${args.runId}::bigint`;
+}
+
+export async function cancelWorkflowRun(sql: Sql, runId: string): Promise<boolean> {
+  const rows = await sql`
+    with j as (
+      update treadle.jobs
+      set state = case when state = 'running' then state else 'cancelled' end,
+          cancel_requested = case when state = 'running' then true else cancel_requested end,
+          finished_at = case when state = 'running' then finished_at else now() end
+      where workflow_run_id = ${runId}::bigint and state in ('available', 'retryable', 'running')
+      returning state
+    )
+    update treadle.workflow_runs w
+    set state = case when exists (select 1 from j where j.state = 'running') then w.state else 'cancelled' end,
+        finished_at = case when exists (select 1 from j where j.state = 'running') then w.finished_at else now() end
+    where w.id = ${runId}::bigint and w.state = 'running'
+    returning id`;
+  return rows.length === 1;
+}