groundwork-method 0.0.1 → 0.11.0

package/src/docs/principles/stack/postgres.md

@@ -0,0 +1,100 @@
+---
+title: Postgres
+description: Schema design, primary keys, JSONB, expand-contract migrations, indexing, connection pooling, queues, and pgvector as a production vector store.
+status: active
+last_reviewed: 2026-06-19
+---
+# Postgres
+
+## TL;DR
+
+Postgres is the canonical data store for every service that needs persistence. We design schemas explicitly, choose primary keys deliberately, migrate with the expand-contract pattern, index from evidence, pool connections, and use `pgvector` as our vector store. When the question is "which database?", the answer is Postgres unless we have a specific, written reason it cannot be.
+
+## Why this matters
+
+Every additional datastore in a system is a multiplier on operational complexity: another backup story, another failure mode, another skill profile to hire for, another surface to monitor. Postgres is a remarkable outlier — it does relational, JSONB document storage, full-text search, queueing, and vector similarity well enough that most workloads never need another engine. Committing to it as a default keeps the operational surface small and the engineers productive.
+
+## Our principles
+
+### 1. Schema design is a design document
+
+Every new table begins with a schema design: what does it represent, what identifies it, what are the invariants, what queries does it need to support, what retention does it live under. This is not a formality — schema shape is the contract that outlives any service that reads or writes the table ([Data Engineering](../system-design/data-engineering.md)). Push invariants into the schema, not just the application: `NOT NULL`, `CHECK`, `FOREIGN KEY`, and `UNIQUE` constraints are enforced by the one component every writer shares. An invariant that lives only in application code is an invariant that some other writer will violate.
+
+### 2. Prefer columns to JSONB for stable shape
+
+JSONB is powerful but it is not a replacement for column design. When a field is present on every row, queried often, or stable in meaning, it belongs in a column — columns get typed constraints, foreign keys, cheap statistics, and B-tree indexes the planner reasons about well. JSONB is the right call when the shape genuinely varies per row, is rarely filtered on, or is a bag of external metadata you store but do not own. When you do query inside JSONB, index it with GIN (or an expression index on the specific path you filter), and remember that you have traded away the constraint enforcement a column would have given you. The default is columns.
+
+### 3. Schema changes follow expand-contract; recovery is roll-forward
+
+Backwards-incompatible change is the source of migration outages, so we never do it in one step. Every change uses expand-contract (parallel change): **expand** the schema with the new, compatible shape; deploy code that writes both old and new; **backfill** existing rows in batched background jobs, never in the migration transaction; cut reads over to the new shape; then **contract** by dropping the old shape once nothing references it. "Migrations are additive" is the easy half of this — the discipline is sequencing the destructive contract step so it lands after every reader and writer has moved.
+
+Two rules make this safe in production, and both are non-obvious:
+
+- **Set `lock_timeout` (and a `statement_timeout`) on the migration connection.** A bare `ALTER TABLE` queues behind any in-flight query holding a conflicting lock, and every request arriving after it then queues behind the `ALTER` — one slow query becomes a full-table stall. A short `lock_timeout` (a few seconds) makes the migration fail fast and retry instead of cascading into an outage.
+- **Build indexes and validate constraints `CONCURRENTLY` / `NOT VALID` then `VALIDATE`.** These avoid the long-held `ACCESS EXCLUSIVE` lock that the naive form takes.
+
+The contested zone is rollback. "Every migration has a pre-written down migration" sounds rigorous but is mostly theater: in production, a down migration that reverses a data-bearing change either cannot run without losing data or has never been exercised under load. Our decision rule: **recovery in production is roll-forward** — you ship a new migration that corrects the problem, because expand-contract has kept the previous shape live and compatible the whole time. Keep a tested down path for the local and CI loop, and only for changes that are provably reversible without data loss. For tables too large for an in-place `ALTER`, reach for a tool built for the job (`pgroll`, `pg_osc`) rather than hand-rolling shadow tables.
+
+### 4. Indexes are evidence-based
+
+Most indexes are justified by a query pattern backed by real production data — `pg_stat_user_indexes` and `pg_stat_statements` tell us which queries are hot and which indexes are paying their cost. Unused indexes cost write throughput and disk; we remove them. Speculative indexes "in case we need them later" are the opposite of the principle.
+
+The honest exception: some indexes are required at table creation, before any production traffic exists. Unique constraints are indexes you cannot defer. Foreign keys are not auto-indexed by Postgres, and an unindexed FK turns every parent delete or update into a full scan of the child — index the referencing side up front. Beyond that, reach for the specific index the query needs, not the generic one: partial indexes for queries that always carry the same filter, covering indexes (`INCLUDE`) to serve index-only scans, expression indexes for computed predicates. Always build them `CONCURRENTLY` on a live table.
+
+### 5. `pgvector` is our vector store — to a threshold we name
+
+Semantic search, embedding similarity, RAG retrieval — all of this runs on `pgvector` in the same Postgres cluster as relational data. The payoff is real and specific: vectors live in the same transaction as the rows they describe, so you filter, join, and keep them consistent without a second system to sync, back up, and reconcile.
+
+This is a default, not a law, and the dishonest version of it ignores scale. Vanilla `pgvector` with an HNSW index serves low-latency, high-recall queries comfortably into the low millions of vectors, while the index fits in RAM; performance degrades as the dataset outgrows memory. The decision rule by scale:
+
+- **Up to a few million vectors:** `pgvector` + HNSW. No argument.
+- **Tens of millions:** stay in Postgres but switch to `pgvectorscale` (StreamingDiskANN), which keeps the index on disk and holds high QPS at high recall well past where in-memory HNSW falls over.
+- **Hundreds of millions and beyond, or hard requirements `pgvector` does not serve** (extreme-scale sharding, specialized hybrid-filtering engines): that is the written reason to run a dedicated vector store. The data and the requirement will make the case; until they do, the second system is unbought complexity.
+
+### 6. Connection management is explicit, and the pooler is the real answer
+
+A Postgres backend is a full OS process with a meaningful memory footprint, so the server tops out at low thousands of connections regardless of how big the box is. The standard architecture is a transaction-mode pooler (PgBouncer or Supavisor) in front of the database: hundreds or thousands of client connections multiplexed onto a small set of server connections, each held only for the duration of a transaction. Size the server-side pool to the database's capacity (a small multiple of CPU cores), not to the number of application instances.
+
+Transaction mode is the right default but it forbids session-scoped state — session-level `SET`, advisory locks, and `LISTEN`/`NOTIFY` break across pooled transactions; isolate those on a session-mode connection. Every service still sets explicit per-connection limits, idle timeouts, and a `statement_timeout`. "Just use the defaults" is how Postgres gets hammered into `too many connections` under load. Postgres is a shared resource; treat it like one.
+
+### 7. Query patterns are reviewed
+
+Every new query is reviewed for plan shape, not just correctness. `EXPLAIN (ANALYZE, BUFFERS)` on representative data is part of the PR for any non-trivial query. N+1 queries, full-table scans, and unbounded `IN` lists are caught in review, not in production.
+
+### 8. Backups, retention, and disaster recovery are not afterthoughts
+
+Automated backups run with RPO and RTO targets that the business has signed off on. We test restores — a backup we have never restored is not a backup. Retention policies are set per table at creation time and aligned with the privacy policy ([Privacy](../quality/privacy.md)).
+
+### 9. Primary keys are a deliberate choice, never UUIDv4
+
+The default is a `bigint GENERATED ALWAYS AS IDENTITY` key: compact, sequential, cache-friendly, and ideal for internal tables that never leave the cluster. Choose a UUID instead when the key is generated by the client or across distributed nodes, exposed in URLs or public APIs (where a guessable sequential id leaks volume and ordering), or merged across systems that must not collide.
+
+When you do reach for a UUID, use **UUIDv7**, never UUIDv4. A v4 key is random, so inserts scatter across the B-tree, fragmenting the index and inflating write amplification and WAL. UUIDv7 is time-ordered: it keeps the global-uniqueness and distributed-generation benefits while restoring the sequential insert locality that makes `bigint` fast — close enough to identity-key performance that the gap stops being a reason to avoid it. Postgres 18+ ships a native `uuidv7()`; on earlier versions generate it in the application or via an extension. The remaining honest cost is size — 16 bytes versus 8 — which multiplies across every secondary index that carries the key, so do not pay it without one of the reasons above.
+
+## How we apply this
+
+- [Data Engineering](../system-design/data-engineering.md) — the broader treatment of data contracts.
+- [Privacy](../quality/privacy.md) — the rules that shape retention and residency.
+
+## Anti-patterns we reject
+
+- **JSONB-everything.** Not a schema; a confession of avoided design.
+- **UUIDv4 primary keys.** Random keys fragment the index and tax every write. Use `bigint` by default, UUIDv7 when you need a UUID.
+- **Indexes "just in case."** Every index is a write tax; justify it from a query or remove it — with the narrow exception of unique constraints and foreign-key indexes, which are required up front.
+- **Migrations that lock a hot table.** `ALTER TABLE ... ADD COLUMN ... NOT NULL DEFAULT` on a 10M-row table with no `lock_timeout`. Add the column nullable, backfill in batches, then tighten — and fail fast on a lock you cannot get.
+- **Blind down migrations as a production safety net.** They are rarely exercised and often lossy. Expand-contract plus roll-forward is the real recovery story.
+- **Raw string interpolation into queries.** Parameterised queries, always. This is a security rule ([Security](../quality/security.md)) and a clarity rule.
+- **A second database "just because."** Adding Redis, DynamoDB, or a dedicated vector store without a specific, documented need Postgres cannot meet. Most of the time, Postgres can.
+
+### On using Postgres as a queue
+
+The reflexive "never use the database as a queue" is dated. `SELECT ... FOR UPDATE SKIP LOCKED` gives Postgres a correct, contention-free work queue, and for low-to-moderate throughput (roughly to the low thousands of jobs per second) a Postgres queue — raw `SKIP LOCKED`, or a mature layer like `pgmq` or Oban — is often the *right* call precisely because it honours the "no second datastore" principle: jobs are enqueued in the same transaction that creates the work, so you get exactly-once-with-the-write semantics for free, with one backup and one failure mode instead of two.
+
+The decision rule: **reach for a dedicated broker when the workload outgrows what a table does well** — sustained high throughput, fan-out to many consumers, streaming and replay, or strict ordered partitions (Kafka territory). And respect the one operational tax that is real: a `SKIP LOCKED` queue churns dead tuples, so it lives or dies by autovacuum — tune aggressive autovacuum on the queue table, or partition it, before load finds the bloat for you.
+
+## Further reading
+
+- *PostgreSQL: Up and Running*, Obe & Hsu — a practical, current reference.
+- *The Art of PostgreSQL*, Dimitri Fontaine — advanced patterns with a teaching bent.
+- *Designing Data-Intensive Applications*, Martin Kleppmann — the systems-level argument for relational-as-default.
+- *pgvector documentation* ([github.com/pgvector/pgvector](https://github.com/pgvector/pgvector)) — the canonical source for vector index strategies.

package/src/docs/principles/system-design/api-design.md

@@ -0,0 +1,86 @@
+---
+title: API Design
+description: Contract-first design, versioning, evolution, pagination, and AI-agent readiness.
+status: active
+last_reviewed: 2026-06-19
+---
+# API Design
+
+## TL;DR
+
+Every API starts as a contract — OpenAPI for HTTP, AsyncAPI for events — and the code is generated from that contract. APIs are versioned deliberately, evolved additively, and shaped so that both human developers and AI agents can consume them without surprise.
+
+## Why this matters
+
+An API is the most durable commitment a service makes. Once it is in production and a client depends on it, changing it is expensive; breaking it is catastrophic. The discipline of API design is not about getting the first version "right" — it is about making the next ten versions safe to ship. In 2026, the stakes are higher still: agents read our APIs programmatically, generate clients against them, and compose them into workflows we did not design. A poorly shaped API is no longer just a developer-experience problem; it is an agent-productivity problem.
+
+## Our principles
+
+### 1. Contract-first, code-generated
+
+A verified spec is the source of truth. Specs live in `/specs`; server handlers, typed clients, and reference docs are generated from them.
+
+The contested part is the *authoring direction*, and serious teams genuinely split. Design-first — write the spec, then implement — makes the contract a negotiation artifact teams agree on before code exists; it is the right default for public APIs and anything crossing a team boundary. Code-first — annotate handlers, emit the spec — keeps the contract welded to the implementation and removes the design-then-diverge gap; it is legitimate for internal services where one team owns both ends. Picking a side is a distraction. What is non-negotiable is that a single spec is authoritative and that CI proves the running service matches it (principle 9). Choose the direction that fits the boundary, and never let the spec decay into documentation that trails the code.
+
+Generated clients beat hand-rolled ones because the generator, not a human, tracks every schema change. But "generated" is not "good": stock OpenAPI Generator output is often bloated and does not validate responses at runtime, which is exactly why the modern SDK generators (Stainless, Fern, Speakeasy) exist. Generate clients — and treat the generator and its configuration as code you own and review, not a black box you run once.
+
+### 2. Explicit versioning, additive evolution
+
+The best version bump is the one you never ship. Additive, expand-then-contract evolution — new optional fields, new endpoints, new response codes, never a removed or repurposed field — keeps clients on v1 for years. Existing clients must never break because we extended the schema.
+
+When a break is truly unavoidable, it requires a new major version and a documented deprecation window for the prior one. URL path versioning (`/v2`) is the default: visible, cache-friendly, trivial to route, and obvious to a developer reading a log line. Date-pinned header versioning — Stripe's `Stripe-Version: 2026-05-20`, where each client is pinned to the API as it behaved on a given day — is more precise but demands a version-transformation layer that rewrites old request and response shapes from current internals. That machinery pays for itself only when you have many external customers you cannot coordinate; do not adopt it to serve three internal consumers you can just upgrade.
+
+### 3. Resources, not RPCs
+
+Edge HTTP endpoints model resources (`POST /items`, `GET /items/{id}`), not verbs (`POST /createItem`). The resource shape forces us to think about identity, lifecycle, and composition up front. When a true verb is unavoidable (`POST /items/{id}/publish`), we name it carefully and document why a resource shape does not fit. This is a discipline for public, REST-shaped surfaces — not a ban on RPC. Internal service-to-service calls over gRPC are verb-oriented by design, and that is correct; the resource rule applies where humans and agents browse the API, not where two services exchange procedure calls behind the edge.
+
+### 4. Idempotency by design
+
+HTTP already makes GET, PUT, and DELETE idempotent by definition — replaying them is safe with no extra machinery. The methods that need help are the non-idempotent ones: POST, and PATCH when it is not a full replace. Those endpoints accept an `Idempotency-Key` header. Clients that retry on failure — which includes every agent we run — depend on it.
+
+The server owns correctness, not the client. It stores the key with the first response's status and body, returns that same response on replay, and rejects a key reused with a *different* request payload as the client bug it is (`422`) rather than silently executing twice. The header is on the IETF Standards Track (`draft-ietf-httpapi-idempotency-key-header`) and not yet a finalized RFC, but the `Idempotency-Key` spelling is already the de facto convention — use it rather than inventing your own.
+
+### 5. Pagination and filtering are uniform
+
+Every collection endpoint paginates with the same shape, filters with the same query-string grammar, and returns the same `next`/`prev` link structure. Reading one collection teaches you every collection. Inconsistent pagination between endpoints is a design smell that never scales.
+
+Cursor pagination is the default: it stays correct when rows are inserted or deleted mid-scan, where offset/limit silently skips and duplicates rows under the reader. Offset paging is acceptable only for small, bounded, slow-changing sets where jump-to-page-N is a genuine user need. Pick one per collection-class and apply it without exception — an agent that learns your pagination once should never be surprised by the next endpoint.
+
+### 6. Errors are structured and machine-readable
+
+Every error response carries a stable code, a human message, and a `details` object. Clients — and especially agents — branch on the code, not on the prose. Error codes are catalogued and never renumbered.
+
+### 7. AI-agent readiness is a first-class concern
+
+A rich OpenAPI spec is the substrate: descriptions on every field, enumerations for every finite domain, explicit examples on every endpoint. An agent reading the spec should be able to use the API correctly without reading the handler — the difference between a spec that compiles and a spec that teaches.
+
+But agent-readiness in 2026 is more than good docs. The Model Context Protocol (MCP) has become the common way agents discover and invoke tools at runtime, and the live question is whether to ship REST, an MCP server, or both. REST stays the system-of-record interface for deterministic system-to-system calls; an MCP server is a thin, intent-shaped projection over that surface for agents selecting tools from natural language. It is a façade, not a second source of truth — it wraps the same handlers, enforces the same auth, and is generated and versioned alongside the spec. Building one before you have agent consumers is speculative generality; declaring a raw 200-endpoint REST surface "agent-ready" because it has an OpenAPI file is wishful.
+
+Design for the agent's failure modes, not just its happy path. Keep response payloads lean — an agent pays tokens for every field it reads, so a 40-field object is a tax on every call. Make each operation do one nameable thing, because tool selection degrades when operations overlap. Label side effects explicitly so a planner knows which calls are safe to retry or run speculatively.
+
+### 8. Async events are contracts too
+
+We treat WebSocket and message broker events with the same rigour as HTTP — an AsyncAPI spec, generated client and server models, additive evolution. Events that are "informal" today are the integration bugs of next quarter.
+
+### 9. The contract is enforced, not just authored
+
+A spec no one verifies drifts the moment a provider ships. We lint specs in CI (Spectral), run consumer-driven or bi-directional contract tests behind a `can-i-deploy` gate so a provider cannot break a consumer, bind errors to **RFC 9457 Problem Details**, and choose the protocol deliberately — REST at the edge, gRPC internal, federated GraphQL for composition, tRPC only inside a TypeScript monorepo. Contract-first authoring without contract testing is half the loop.
+
+## Anti-patterns we reject
+
+- **Breaking changes without a version bump.** "It is a small breaking change, no one uses that field" — the assumption is always wrong in an agent-consuming world.
+- **Hand-written clients.** Clients drift, and drift causes outages. Generate.
+- **Kitchen-sink endpoints.** `POST /doThing` that accepts a 40-field payload and does everything. Split it.
+- **Error payloads as strings.** A 400 response body of `"invalid input"` is unusable by any automated caller. Structured errors, always.
+- **Endpoint-scoped pagination conventions.** Cursor in the body here, page-number in a query string there, offset-limit somewhere else. Pick one and apply it universally.
+- **"Agent-ready" by assertion.** Terse field names, no examples, and a 200-endpoint surface — declared agent-ready because an OpenAPI file exists. An agent needs the same teaching a new engineer does.
+
+## Further reading
+
+- *Designing Web APIs*, Jin, Sahni, Shevat — the working bible of HTTP API design.
+- *Web API Design: The Missing Link*, Apigee — the short handbook that gets the REST vocabulary right.
+- *RFC 9457: Problem Details for HTTP APIs* ([rfc-editor.org](https://www.rfc-editor.org/rfc/rfc9457)) — the standard error-body format.
+- *Designing robust and predictable APIs with idempotency*, Stripe ([stripe.com/blog/idempotency](https://stripe.com/blog/idempotency)) — the reference treatment of idempotency keys.
+- *Model Context Protocol* ([modelcontextprotocol.io](https://modelcontextprotocol.io)) — the emerging standard for agent tool discovery and invocation.
+- *AsyncAPI Specification* ([asyncapi.com](https://www.asyncapi.com)) — the canonical format for async contracts.
+- *OpenAPI Specification* ([openapis.org](https://www.openapis.org)) — the canonical format for HTTP contracts.

package/src/docs/principles/system-design/architecture-decisions.md

@@ -0,0 +1,81 @@
+---
+title: Architecture Decisions
+description: Architecture Decision Records as governed, re-evaluable commitments — lean, assumption-explicit, immutable as records, and readable by humans and agents alike.
+status: active
+last_reviewed: 2026-06-19
+---
+# Architecture Decisions
+
+## TL;DR
+
+We capture every significant architectural decision as a short record — the context that forced it, what we chose, the assumptions it rests on, and the trade-offs we accepted. The record is immutable; the decision is not. When the world changes we supersede the record rather than overwrite it, so the trail of *why* is preserved and re-evaluation is a check against named assumptions rather than a debate from scratch. A decision record is governance, not paperwork — and in an agent-led codebase it is the memory that keeps autonomous and human decisions consistent.
+
+## Why this matters
+
+The reasoning behind a decision is the first thing lost and the most expensive to reconstruct. Code shows *what* was built; it cannot tell you that a boundary sits where it does because of a constraint that no longer exists. A team without decision records relitigates the same choices every year, cannot tell "we considered this and rejected it" from "nobody thought of it," and discovers its load-bearing assumptions only when one of them breaks in production.
+
+The discipline is cheap and the payoff compounds — but only if records stay *lean enough to write* and *governed enough to trust*. Twenty-field templates die unmaintained; bare records nobody can re-evaluate rot into folklore. The craft is the narrow middle: the few fields that make a decision both fast to capture and honest to revisit.
+
+## Our principles
+
+### 1. The reasoning is the record
+
+An ADR captures *Context* (what forced the decision, what was on the table), *Decision* (what we chose), and *Trade-offs* (what we gave up, what risk we accepted). The why is the deliverable. A record of the outcome without the reasoning is half a record — it cannot stop the next person from quietly undoing the decision because its rationale was invisible.
+
+### 2. Name the assumptions
+
+Every decision rests on conditions that were true when it was made — a load profile, a team size, a vendor price, a regulatory boundary. We write them down. Assumptions are the linchpin of governance: a decision is valid exactly as long as the assumptions under it hold, and naming them turns "should we revisit this?" from a vibe into a check you can run.
+
+### 3. Set a review trigger
+
+Decisions expire against reality, not the calendar. Each significant record names the condition that should bring it back for review — "when we add a second region," "if write throughput passes 50k/s," or, when nothing better exists, a date. The trigger is what makes re-evaluation *proactive*: the decision resurfaces when its world shifts, instead of when someone happens to remember it.
+
+A trigger is not free. An unmaintained trigger is the twenty-field template in another costume — a standing promise nobody keeps — and a stale log is worse than no log, because it reads with false confidence. So a named trigger is earned, not default: reserve it for decisions whose validity hinges on a fragile, measurable assumption (a throughput ceiling, a vendor term, a single-region bet). For everything else the assumption list and the supersession chain *are* the trigger — they let any later reader re-check on demand without anyone holding a vigil.
+
+### 4. The record is immutable; the decision is not
+
+Once accepted, a record's conclusions are not edited (fixing a typo or a dead link is fine). When a decision changes, we write a **new** record that supersedes the old one, and we link the two in both directions — the old marked `superseded by NNNN`, the new naming what it replaces. The original Context and Trade-offs stay true *for the world they were written in*; that is precisely why they are worth keeping. An immutable trail is what makes the set trustworthy enough to re-evaluate against. Some teams run ADRs as living documents instead, appending dated notes to the original; that is fine for anything that does not move the conclusion — a clarification, a fresh data point, a status change from proposed to accepted. The line is the conclusion itself: amend in place when the meaning holds, supersede when the meaning changes. Editing a *conclusion* in place is the single move that breaks the trail, because it rewrites history that other records and live code now depend on.
+
+### 5. Re-evaluation is the goal, not the exception
+
+Records exist to make changing our minds *productive*. When circumstances shift, the record lets us argue about what actually changed — does this assumption still hold? — rather than rebuilding the whole decision from memory. Re-deciding is healthy engineering; re-deciding *without recording it* is how a team loses its memory.
+
+### 6. Keep it lean
+
+Three to five sections plus a thin governance header is the whole of it. The fields earn their place because each is load-bearing — Context and Trade-offs carry the reasoning; Assumptions and the review trigger carry the governance; status, owner, and supersession links carry the history. We resist every field beyond these. A template that feels like a chore is a template that goes unwritten, and an unwritten decision is the most expensive kind.
+
+### 7. Owned, not orphaned
+
+Every standing decision has a current owner — the person or team accountable for it today, who is not necessarily whoever wrote it. An orphaned decision is one nobody will revisit when its assumptions break, which is the same as having no governance at all.
+
+### 8. The decision log is the agent's memory
+
+In an agent-led codebase, the record set is not just human documentation — it is the decision-context layer an agent reads before proposing or revisiting a choice. Relevance is governed by assumption-overlap and supersession status, not recency — a foundational record from the first month can bind a choice more tightly than last week's, and an agent that simply weights the newest records will miss it. So we keep the set lean, linked, assumption-explicit, and *current*, and that is what lets a human or an agent make a *consistent* next decision instead of contradicting one made last quarter. A stale log misleads an agent faster than it misleads a human: the agent reads it at full confidence and propagates a constraint that no longer holds. Records are written to be machine-consumable for the same reason every other interface is ([Agent-Native Systems](../ai-native/agent-native-systems.md)).
+
+### 9. Govern by advice, and pair the record with a check
+
+A record is not a gate. "Governed" means an **advice process** — the decider seeks advice from everyone meaningfully affected *and* everyone with relevant expertise, then keeps the decision — not a central review board teams route around. The advice is mandatory; consensus is not. That trade buys speed with trust and accountability: it degrades the moment deciders skip the advice or escape the consequences of a bad call, and it presumes systems loosely coupled enough that the affected set stays small. A standing review gate is still the right tool for the rare decision that is irreversible *and* cross-cutting *and* externally constrained — a regulated boundary, a one-way data-retention choice — so reserve gates for those rather than making them the default. And where a decision is mechanically checkable, we pair it with a **fitness function** that fails the build on violation: the record documents the choice, the fitness function assures it still holds ([Evolutionary Architecture](evolutionary-architecture.md)).
+
+## How we apply this
+
+Records live in `docs/architecture/decisions/NNNN-<slug>.md`, numbered sequentially. We record at the moment of the decision, not as after-the-fact paperwork — the cheapest time to capture the context is while it is still in the room. The significance test: would a new engineer or agent otherwise have to reconstruct, or relitigate, the reasoning from scratch? A reversible, low-cost, local choice does not earn a record; a durable, cross-cutting, expensive-to-reverse one does.
+
+- [How We Structure Code](code-structure.md) — the boundary decisions most often worth recording.
+- [Agent-Native Systems](../ai-native/agent-native-systems.md) — why records are written to be read by agents.
+
+## Anti-patterns we reject
+
+- **Frozen decisions.** "That was decided, we don't revisit it." The record exists to make revisiting *tractable*, not to forbid it.
+- **Decisions without assumptions.** A record that omits what it depended on cannot tell you when it has gone stale. It will go stale anyway — silently.
+- **Overwriting a record in place.** Editing the conclusion erases the trail that made the current state trustworthy. Supersede; never rewrite.
+- **The twenty-field template.** A form so heavy nobody fills it produces exactly zero records. Lean beats complete.
+- **Orphaned decisions.** No owner means no one will notice when the trigger fires.
+- **Paperwork after the fact.** A record written a month later, from memory, has lost the context that was the point.
+- **Relitigating from scratch.** If the conversation rebuilds the whole decision instead of testing what changed, the record is not being used.
+
+## Further reading
+
+- *Architecture Decision Record*, Michael Nygard (the original 2011 post) — the lightweight format everything here descends from.
+- *MADR* ([adr.github.io](https://adr.github.io/)) — the modern Markdown ADR template and status lifecycle, built on Nygard's five-part structure (title, status, context, decision, consequences).
+- *Facilitating Software Architecture*, Andrew Harmel-Law (O'Reilly, 2024) — the architecture advice process: decentralized, advice-bound, accountable decision-making.
+- *Context Matters: Evaluating Context Strategies for Automated ADR Generation Using LLMs* (2026) — evidence that recent decision records are a high-leverage context layer for agents.

package/src/docs/principles/system-design/code-structure.md

@@ -0,0 +1,104 @@
+---
+title: How We Structure Code
+description: A pure core, swappable edges, no leaked implementation details, and one obvious place for everything — the structural discipline that makes a codebase legible to a human and navigable to an agent, expressed in each language's own idiom.
+status: active
+last_reviewed: 2026-06-19
+---
+# How We Structure Code
+
+## TL;DR
+
+Every service is a **pure core** that makes decisions, wrapped in a **thin shell** that does I/O. The core depends only on abstractions it owns; the concrete implementations — the database, the LLM, the queue — plug in at the edges and are swappable. No implementation detail leaks inward: the core never knows it is talking to Postgres or Anthropic. And the structure is **opinionated and predictable**, so a developer or an agent never has to guess where a thing lives or where a new thing goes. The discipline is the same in every language; its expression is native to each.
+
+This is the single highest-leverage structural choice we make, and it is deliberately non-negotiable for new code.
+
+## Why this matters
+
+There is one idea here, and the industry has named it five times — Dependency Inversion, Hexagonal / Ports & Adapters, Onion, Clean Architecture, Functional Core / Imperative Shell. They are restatements of a single constraint. As Robert C. Martin put it of the variants, *"they all have the same objective, which is the separation of concerns… by dividing the software into layers"* — and the rule under all of them is the **Dependency Inversion Principle** (Martin, 1996): high-level policy depends on abstractions, not on low-level detail. We state the rules directly and in plain language; the named frameworks live in Further Reading, for the lineage. What we hold is the discipline, not a label — and we keep it shallow, never the "onion with ten rings."
+
+Holding the structure pays off twice:
+
+- **Changeability.** Because dependencies plug in behind abstractions the core owns, swapping a database, an LLM provider, or a message broker is a configuration change at the edge, not a rewrite through the middle.
+- **Testability.** A pure core has many branches and no dependencies, so it is tested exhaustively with plain input→output assertions and no mocks; the thin shell has few branches and many dependencies, so it is tested for real against the things it wraps (Gary Bernhardt's *Boundaries*). The architecture tells you what to stub and what to stand up.
+
+And there is a payoff specific to how we work in 2026: **an opinionated structure collapses an agent's decision space.** When "where does this code live?" is already answered by the convention, the agent inherits the layout instead of inventing one, and produces code that matches the existing shape. Two cautions keep this honest, both from recent evidence: conventions help an agent only when they are **legible** — explicit and discoverable in the code — not when they are hidden framework magic the agent must already know (the *Constraint Decay* finding, that agents do worse in convention-heavy environments whose rules are implicit). And an obvious structure beats verbose prose guidance: piling requirements into instruction files measurably *reduces* success. So we make the structure speak for itself, and we keep the guidance minimal.
+
+## Our principles
+
+### 1. A functional core, an imperative shell
+
+Decisions are pure functions over plain values — no I/O, no hidden mutation, no clock or network reached for in the middle. All side effects live in a thin shell at the edges that gathers inputs, calls the core, and acts on what it returns (Mark Seemann's *impure→pure→impure* sandwich). The core is deterministic and does not know the shell exists.
+
+This is a discipline, not a religion, and it bends where it must: when a decision genuinely needs more data part-way through, or a filter must run database-side for performance, the shell reaches back for it. That is expected. The goal is to keep *as much branching as possible* in the dependency-free core — not to achieve a zero-effect purity the language fights. In Go, that means functions over values and errors-as-values, not a framework of structs-with-methods; in Python, injecting a clock and keeping logic in pure functions; in TypeScript, the same. Keep it native.
+
+### 2. Depend on abstractions you own, and make them point inward
+
+The core declares the interface it needs, **in its own language** — `Store`, `Embedder`, `Notifier` — and the implementation conforms to it. The dependency points inward: the edge depends on the core's abstraction; the core never depends on the edge. This is not aesthetic. It is **enforceable in CI** — `depguard` in Go, `import-linter` in Python, ESLint import rules in TypeScript — and a violation fails the build. Enforcement is what turns the structure from a style we hope for into a guarantee we have.
+
+### 3. Never leak an implementation detail
+
+That storage is Postgres, that the model is Anthropic, that the queue is Kafka — none of it may shape the core's types or its method signatures. No ORM entities standing in as domain models, no `IQueryable`- or SQL-shaped methods on a port, no vendor SDK type in a port's signature. An abstraction expressed in the vendor's terms is a leak wearing a costume — when the implementation changes, the "abstraction" changes with it, which is the proof it was never one. The port speaks the domain's goals: *"methods that make sense to the goals of a user in a domain, not to a database"* (Brett Schuchert).
+
+Hold this as a direction, not a fantasy. *All non-trivial abstractions leak to some degree* (Joel Spolsky's law) — a query that is fast on an index and slow without one leaks through any storage port. We minimise leakage rigorously; we do not pretend it reaches zero.
+
+### 4. Abstract at the narrow seam you actually use — never speculatively
+
+A port exposes only what the application actually calls, named in the application's language: `Orders.GetById` and `Orders.Save` over an aggregate, an `Embed(text) → vector`. Narrow, domain-named ports are cheap to carry and easy to change — effort YAGNI explicitly exempts as *making the software easier to modify*. The speculative version is the trap: an interface invented "for mocking" with exactly one implementation, a wide wrapper mirroring a vendor SDK for a swap that never comes, or a generic `Repository<T>` / `IRepository<TEntity>`.
+
+The generic repository deserves a careful word, because it is seductive for a real reason: it *forces a clean break* between the domain and the store — a hard, uniform boundary — and that break is genuinely valuable. The lie is the word "generic." We have almost always seen exactly **one** implementation, so the generality is unproven — Brian Foote's *Speculative Generality*, "spotted when the only users are test cases." And a uniform CRUD surface (`GetAll`, `Find(query)`, an exposed `IQueryable`) ends up advertising the *store's* capabilities, not the *domain's* needs — the leak from principle 3, wearing the costume of an abstraction.
+
+So take the clean break from where it actually comes — an owned port, a persistence-ignorant core, and adapter tests against the real database (principle 5) — not from a generic CRUD contract. The decision rule:
+
+- **Simple CRUD or read-heavy work: no bespoke repository.** A mature ORM or query layer already *is* a unit of work and a collection-like store; wrapping it again is indirection that hides the details you need. Use it directly, or a query object.
+- **A rich aggregate with invariants: one repository per aggregate root** — domain-named methods, the interface owned by the core, the implementation in the adapter. Speak the ubiquitous language, never the schema; when query variety grows, reach for a *specification*, not more methods or a leaked query type.
+- **A generic base is legitimate only once the generality is proven** — three real aggregates exist (the rule of three), the base is *constrained* to enforce the aggregate-root rule, and it is reached *only* through specific domain-named ports. A generic base with one implementation is the lie; delete it.
+
+The same logic governs every external dependency: **wrap thin clients, not rich tools** — a driver, a raw HTTP SDK, an LLM API earn a narrow owned port; a tool that already implements the pattern does not (Jimmy Bogard's heuristic; Go's *"the bigger the interface, the weaker the abstraction"*). This is the principle that most needs **judgment**, which is why it is carried in the engineer skills (principle 8) and applied consistently rather than reinvented per file. The wrong abstraction costs more than the duplication it removes (Sandi Metz) — when in doubt, wait for the third case.
+
+### 5. Swappable by design; integration-tested for real
+
+Because implementations plug in behind owned abstractions, a dependency is replaceable — and we prove it the right way. Stub the port to test the core's logic exhaustively and fast; run the **real** adapter against the **real** dependency in a container, never a mock of it. We test the system, not a mock of the system. A boundary you cannot test cheaply is usually a boundary drawn in the wrong place. (See [Testing](../foundations/testing.md).)
+
+### 6. One obvious place for everything
+
+The structure is opinionated on purpose: for any change there is a single predictable place it belongs, and for any feature a single place to find all of its parts. The top-level layout should **announce the domain, not the framework** — it should read as *Payments* and *Catalogue*, not as the web framework underneath (Robert C. Martin's *Screaming Architecture*). Prefer colocating what changes together — a feature owns its handler, its logic, its types — over scattering a feature across technical layers; *"minimise coupling between slices, maximise coupling within a slice"* (Jimmy Bogard). Decide the thousand trivial layout choices **once**, in the scaffold, so no one re-litigates them per service — *"a place for everything, and everything in its place; constraints liberate"* (the Rails doctrine). Python's Zen says the same: *one obvious way*, and *in the face of ambiguity, refuse the temptation to guess.* The test of the structure is that nobody — human or agent — has to guess.
+
+### 7. Keep it shallow
+
+Three zones are enough: the **core**; the **abstractions plus the orchestration** that uses them; the **edge implementations**. Resist the extra rings — the five-hop DTO translation, the layer that exists only to forward a call. Indirection is not abstraction: a layer that hides no complexity is just a tax on every reader and every change, and it should be deleted. Shallow keeps cognitive load low, which is the real limit on how fast anyone moves through the code.
+
+### 8. The discipline is constant; the expression is native
+
+The rules above hold identically in Go, Python, and TypeScript — but each is written the way its own community writes it, and we never impose a uniform cross-language layout (a uniform layout would itself read as a framework fingerprint, and we avoid it). Go defines narrow interfaces at the point of use and returns concrete structs; Python expresses ports as `Protocol`s — structural typing, not inheritance ceremony; TypeScript leans on structural types, where a port costs almost nothing to declare. The **engineer skill for each stack is the carrier of its native expression** — it is where these principles become a specific, idiomatic shape the agent can scaffold, extend, and recognise.
+
+## How we apply this
+
+- **New services ship this shape from the scaffold** — the directory layout, the dependency-rule lint config wired into CI, and a stub core with one real adapter that demonstrates the flow. The convention is generated, not asked for.
+- **The per-language engineer skill is the delivery vehicle.** `groundwork-go-engineer`, `groundwork-python-engineer`, and the surface engineers teach the agent the native expression of these rules: where to look, where new code goes, and what idiomatic looks like for that stack. An agent equipped with the skill navigates and extends any codebase built to the convention without guessing — which is the whole point of holding a convention.
+- **Brownfield is guided toward the same shape.** The scan → extract → adopt path reads an existing codebase, names where it already follows these principles and where it diverges, and moves it toward the convention incrementally rather than demanding a rewrite.
+- **For frontends, apply the spirit:** isolate network I/O behind a data layer and keep rendering logic free of fetching concerns; the core/edge split survives even where the file conventions differ.
+
+## Right-size the boundaries
+
+Structure within a service is one decision; how many services exist is another, and the default is conservative. Start as a **modular monolith** — one deployable with strong internal module boundaries, one bounded context per module — and treat extracting a microservice as an *earned* move, justified only when multiple signals converge (the language and mental model shift, the scaling profile is incompatible, the deploy cadence is fundamentally different). The failure mode to name is the **distributed monolith**: services that deploy in lock-step, share a schema, or call each other synchronously three deep — the cost of microservices with none of the independence. Make the consolidation signal as first-class as the split: two services that always change together should be merged back. (See [Surface Architecture](surface-architecture.md) and [Evolutionary Architecture](evolutionary-architecture.md).)
+
+## Anti-patterns we reject
+
+- **Framework-coupled core.** `gin.Context`, `fastapi.Request`, or an ORM entity living in the core. The moment the core imports the framework, it is no longer the core.
+- **Leaked persistence.** The core knows it is Postgres; a "port" that returns `IQueryable`, a `DbSet`, or raw rows. Map at the edge, in the adapter — not by letting the storage shape reach inward.
+- **Vendor-shaped ports.** A port that mirrors an SDK's methods. Swapping the vendor then means rewriting the port and everything that calls it — the opposite of what the port was for.
+- **Speculative abstraction.** A generic `Repository<T>`, an interface with a single implementation added "for mocking," a wide wrapper over a mature tool to enable a swap no one will make. Indirection sold as foresight.
+- **Anaemic core, god service.** Data classes with no behaviour and one application service that holds every rule. Rules belong on the entities they govern.
+- **Over-layering.** Five DTO translations between the HTTP edge and the core. Adapters are thin: read the request, call the core, write the response.
+- **Guessing-required structure.** No single obvious home for a change; the parts of one feature scattered so widely you have to grep to assemble it. If placement is ambiguous, the structure has failed its one job.
+- **Mock-the-world testing.** Proving the system against mocks of its own dependencies. Stub the port to test the core; use the real thing to test the adapter.
+
+## Further reading
+
+These are the sources behind the idea. You may know the discipline under any of their names — read one treatment in full; they converge.
+
+- *The Dependency Inversion Principle*, Robert C. Martin (1996) — the root rule beneath every variant. Start here.
+- *The Clean Architecture* (Martin, 2012), *Hexagonal / Ports & Adapters* (Alistair Cockburn, 2005), and *Onion Architecture* (Jeffrey Palermo, 2008) — three names for the inward-dependency rule. Any one of them is the same message.
+- *Boundaries* / Functional Core, Imperative Shell, Gary Bernhardt (2012), and Mark Seemann's *Composition Root* and *impureim sandwich* essays on [blog.ploeh.dk](https://blog.ploeh.dk) — the pure-core framing and where it stops.
+- *Screaming Architecture* (Martin, 2011) and *Vertical Slice Architecture* (Jimmy Bogard, 2018) — making placement obvious.
+- The honest counter-camp, read so you abstract with judgment rather than reflex: Sandi Metz, *The Wrong Abstraction* (2016); Joel Spolsky, *The Law of Leaky Abstractions* (2002); and the repository-pattern critiques (Derek Comartin, Vladimir Khorikov) on when the abstraction is worth its cost and when it is ceremony.

package/src/docs/principles/system-design/data-engineering.md

@@ -0,0 +1,87 @@
+---
+title: Data Engineering
+description: Events, streams, CQRS, event sourcing, and the data contracts that outlive any service.
+status: active
+last_reviewed: 2026-06-19
+---
+# Data Engineering
+
+## TL;DR
+
+Data outlives services. We treat every event we emit and every table we own as a long-term contract, shaped so downstream consumers — today and in three years — can work with it without archaeology. Events are append-only, schemas are versioned, and the log of what happened is preserved even when the current-state projection is rebuilt.
+
+## Why this matters
+
+Services are replaced; data lives on. The data contracts we set today — table shapes, event payloads, field semantics — are the single most durable thing we will produce. Getting the contract right once is cheap; changing it retroactively after the data has multiplied is brutal.
+
+## Our principles
+
+### 1. Events are append-only and immutable
+
+Once an event is emitted, it is never rewritten. Correction happens through *compensating* events, not through mutation of the original. This is the discipline that lets downstream consumers trust the event log as a truthful history of the system.
+
+### 2. Schemas are versioned and evolvable
+
+Event payloads have explicit versions. New fields are additive; removed fields are deprecated with a deadline, not removed silently. Consumers can detect an old schema and handle it or refuse it — they are never surprised. This is the AsyncAPI discipline ([API Design](api-design.md)) applied to every stream.
+
+### 3. Partition keys are chosen deliberately
+
+Partitioning forces a tradeoff with no universal answer: ordering pulls toward concentrating related records on one partition, even distribution pulls toward spreading them. The key that guarantees per-entity order — the entity's ID — also creates a hot partition the moment one entity is far busier than the rest, and the broker will not rebalance skew away for you. You size for the hottest partition and waste the rest.
+
+Decision rule:
+
+- Per-entity ordering needed and entities are roughly uniform → partition by entity ID.
+- Ordering needed but some entities are hot → use a composite key (`tenantId|entityId`) to widen routing cardinality while preserving order for the inner entity, or salt the hot key into K buckets and accept that order is lost *across* buckets.
+- No ordering requirement → leave the key null and let the producer round-robin for even load.
+
+Choosing a partition key casually is one of the most expensive mistakes in a data system. Repartitioning a live topic is a migration, not a config change, so the key gets reviewed at design time.
+
+### 4. CQRS where it pays
+
+For read-heavy surfaces with complex projections, we maintain a read model separate from the write model. The write model owns truth; the read model owns query performance. We do not apply CQRS universally; we apply it where the read load and the write load have genuinely different shapes. The tax is eventual consistency — the read model lags the write, so any surface built on it must not assume read-after-write. If a single well-indexed table serves both paths, you do not have a CQRS problem; splitting the model early buys two things to keep in sync and nothing else.
+
+### 5. Event sourcing is a tool, not a religion
+
+For domains where the history of change is itself the product — audit logs, participation timelines — we store the event log as the primary artefact and derive current state from it. For domains where current state is what matters, we store current state and publish events as derivatives. Event sourcing every table "because it is purer" is overengineering.
+
+### 6. Data contracts are documented, versioned, and owned
+
+Every significant table and every published event has an owner, a documented schema, a migration history, and a compatibility policy. Unowned tables and undocumented events are a ticking integration-debt clock.
+
+### 7. Retention is a design decision
+
+Every dataset we store has a retention policy — deletion after N days, archival after M days, live forever. Retention is decided when the dataset is created, reviewed when the regulatory surface changes ([Privacy](../quality/privacy.md)), and enforced by automation. "We will figure it out later" is the decision that becomes a compliance incident three years later.
+
+### 8. Backfills are a planned operation
+
+Changing the shape of historical data — renaming a field, re-computing a derived column — is a project with a plan, a rollback, and a measurement. We do not backfill by running a script and hoping. Backfills are rehearsed in staging and measured in production.
+
+### 9. Change capture, enforced schemas, and the AI-era layer
+
+**Change capture vs. outbox.** The transactional outbox is the events we *mean* to publish, written in the same database transaction as the state change so there is no dual write to lose. CDC streams the raw row changes of a table. They are complementary, not rivals: the cleanest outbox relay *is* CDC — Debezium tailing the write-ahead log — rather than a polling loop. We reach for raw-table CDC as an integration backbone only when the source cannot give us a real contract; a CDC feed of someone else's schema is a contract we never negotiated, and they can break it without telling us.
+
+**Enforced, not just versioned.** Versioning (principle 2) is the policy; enforcement is the mechanism. A schema registry checks compatibility at registration and fails the producer's build on a breaking change, shifting the contract left into CI instead of into a consumer's pager at 3am. The contract is owned by the producer; an unowned contract is stale documentation.
+
+**The AI-era data layer is architecture, not a bolt-on.** Vector/embedding stores are the retrieval core of RAG, and chunking, hybrid (lexical + semantic) search, re-ranking, and metadata filtering are design decisions, not library defaults. Default to `pgvector` in the database you already run — under roughly ten million vectors it matches dedicated stores on latency and recall while saving you a whole system to operate and keep consistent. Move to a dedicated vector database when scale, recall under heavy concurrency, or index build time make Postgres the bottleneck. Embeddings are derived data: a model change invalidates them, so re-embedding is a planned backfill (principle 8) against a versioned embedding model, never an ad-hoc rerun.
+
+**Storage: layer, don't pick a camp.** Data mesh and the lakehouse answer different questions — mesh is an org model (domain-owned data products, federated governance), the lakehouse is a technical substrate. The working pattern is layered: a platform team owns storage, catalog, and CI templates; domain teams own the products on top. For the table format there is no clean winner — Delta Lake if you are Spark/Databricks-centric, Apache Iceberg if you want engine-agnostic optionality and the broader catalog ecosystem. Pick for the engines you actually run and the vendor coupling you can tolerate, not for the benchmark of the week.
+
+## How we apply this
+
+- [Postgres](../stack/postgres.md) — how we apply these principles inside our chosen database.
+
+## Anti-patterns we reject
+
+- **Silent schema changes.** Renaming a column in a hot table without coordinating consumers. This is how outages start.
+- **Mutable event logs.** Going back and "fixing" a past event. The event is what happened; the correction is a new event.
+- **Kitchen-sink "events" table.** One table that accepts a JSON blob for every kind of event. The type system is the best friend of a data contract; do not throw it away.
+- **Backfills in production without rehearsal.** See above.
+- **Retention by accident.** Tables that grow forever because no one considered retention at creation time.
+
+## Further reading
+
+- *Designing Data-Intensive Applications*, Martin Kleppmann — the single best survey of the territory, including the chapters on derived data, stream processing, and batch processing.
+- *Data Mesh*, Zhamak Dehghani — the argument for treating data as a first-class product with owners.
+- *Streaming Systems*, Akidau, Chernyak, Lax — the deep treatment of time, watermarks, and windowing in stream processing.
+- *Event Sourcing and CQRS*, Vaughn Vernon (the relevant chapters of *Implementing DDD*) — a grounded, implementation-focused view.
+- *Data Contracts*, Chad Sanderson, Mark Freeman & B. E. Schmidt (O'Reilly, 2025) — producer-owned, shift-left enforcement of the contracts in principles 6 and 9.

package/src/docs/principles/system-design/durable-execution.md

@@ -0,0 +1,89 @@
+---
+title: Durable Execution
+description: Workflow-as-code as a first-class primitive — moving reliability for multi-step, long-running, and must-complete processes out of application code and into the infrastructure.
+status: active
+last_reviewed: 2026-06-19
+---
+# Durable Execution
+
+## TL;DR
+
+For a process that has many steps, runs for a long time, or must either complete or compensate, we reach for **durable execution** — workflow-as-code on an engine that checkpoints progress so the process resumes where it left off after a crash, deploy, or hours-long wait. It moves the reliability guarantee out of hand-written saga, outbox, and retry glue and into the infrastructure. It is the same primitive that makes long-running agents and human-in-the-loop approvals safe. It is not free: it imposes a determinism contract on workflow code and a versioning discipline on every deploy, and below a complexity threshold an outbox or a queue is the better tool.
+
+## Why this matters
+
+The classic way to make a multi-step process reliable is to hand-assemble it: an outbox here, an idempotency key there, a retry table, a state column, a cron to sweep the stuck ones. Each piece is correct and the whole is a fragile machine that every engineer reasons about differently and that fails in the gaps between the pieces. Durable execution collapses that machine into ordinary code whose execution state is automatically persisted — "a function that survives a crash and resumes." The category predates the current AI wave (it grew out of Uber's Cadence and the workflow engines before it), but agent workloads — long-running, tool-calling, occasionally-paused processes that cannot afford to restart from zero — are exactly its best fit, and they drove it into the mainstream.
+
+## Our principles
+
+### 1. Workflow-as-code is a first-class primitive
+
+For multi-step, long-running, or compensating processes, we prefer a durable execution engine over hand-rolled saga + outbox + retry orchestration. The workflow is written as ordinary, mostly-linear code; the engine persists its progress. This sits beside choreography and orchestration as a named architectural option, not buried inside "saga."
+
+It is not the default for every async task, and the cost is real: a determinism contract, a versioning discipline, and often a new piece of infrastructure to operate. **Decision rule:** reach for durable execution when a process has three or more steps that must survive a crash *together*, holds state across minutes-to-days, or needs compensation and progress visibility. Below that — fire-and-forget work, a single idempotent event handler, "publish this event reliably" — an outbox or a queue is correct and a workflow engine is overkill. The honest failure mode runs the other way too: a queue plus cron plus a retry table that has accreted dedup windows, scattered timers, and compensations is a workflow engine you built by accident, missing the correctness guarantees and observability you would have gotten for free.
+
+### 2. Durability lives in the infrastructure, not the code
+
+The reliability guarantee — resume-where-you-left-off, durable timers, automatic retry — is provided by the engine, not re-implemented per process. Application code expresses the business steps; the platform owns crash recovery.
+
+What the engine guarantees is **effectively-once**, not exactly-once. Activities and steps run *at-least-once*: an engine can crash after a side effect completes but before its result is recorded, and on recovery it runs the step again. The engine makes the *workflow* converge to a single logical outcome; it does not make a non-idempotent `POST` safe. Exactly-once side effects are a property the step author provides with idempotency keys — the platform turns at-least-once delivery into effectively-once *outcomes* only when the steps cooperate. Treating "exactly-once" as a free platform guarantee is the most common way teams get double-charges in a system they believed was safe.
+
+### 3. Choose the engine by operational shape
+
+The category spans distinct operational shapes, and the choice is dominated by the burden you can carry, not by popularity:
+
+- **Self-hosted replay engines** — **Temporal** (a dedicated cluster, the most mature, proven at scale) and **Restate** (the same journal/replay model with a lighter footprint and HTTP-native, serverless-friendly ergonomics). Maximum power and flexibility; you operate (or pay for) the orchestrator.
+- **Managed state-machine orchestrators** — **AWS Step Functions** and the equivalents on other clouds. Workflows are declared as JSON/ASL state machines rather than code, with deep native integration to cloud services and nothing to run. The trade is expressiveness: complex branching and loops are awkward in a declarative state machine, and you are bound to one cloud. The default when the work is gluing managed services together on a single cloud.
+- **Embedded / library engines** — **DBOS** and the "just Postgres" approach (a queue via `SELECT … FOR UPDATE SKIP LOCKED` plus a checkpoint table). Durable execution as a library inside your existing process, reusing Postgres as the durability layer — no separate orchestrator, minimal code change. The low-footprint option for teams already on Postgres, with a real ceiling: every step is at least one write, so a hot workflow fanning out to thousands of children breaks first on Postgres contention — lock pressure on the status table, WAL pressure, autovacuum falling behind.
+
+**Decision rule:** already deep in one cloud and mostly orchestrating its services → managed state machine. Already on Postgres, modest scale, want to avoid new infrastructure → embedded/library. Complex long-lived workflows, high scale, or cross-environment (hybrid/on-prem) → a self-hosted replay engine.
+
+### 4. It complements the event log, it does not replace it
+
+A durable workflow and an event backbone (Kafka and friends) are complementary: the log decouples producers from consumers and is the source of truth for events; durable execution orchestrates stateful, multi-step work on top. Building long workflows on the raw log alone is significant custom machinery; reaching for a workflow engine where a single event handler suffices is overkill.
+
+### 5. Orchestration vs choreography is a deliberate decision
+
+Choreography (services react to each other's events, no central coordinator) keeps services loosely coupled and scales naturally; its cost is that no single place knows the state of the whole process, which makes debugging and compensation hard. Orchestration (a durable workflow drives the steps) buys you central visibility, explicit compensation, and one place to reason about the flow, at the cost of a coordinator the flow depends on.
+
+**Decision rule:** choose by coupling and the need for visibility, not by step count. Choreography when the steps are genuinely independent reactions and no one needs to ask "where is this process right now"; orchestration the moment you need compensation, branching, a process-level SLA, or an operator who can see and intervene. Step count is a symptom, not the criterion — a two-step flow that must compensate wants orchestration; a long chain of fire-and-forget reactions does not.
+
+### 6. Determinism is a replay constraint, and it lives in the control flow
+
+Replay-based engines (Temporal, Restate) recover by re-executing the workflow function and replaying recorded results for steps already done. That imposes a hard contract: the workflow's **control flow** must be deterministic, so the same history always drives the same branches. The side effects themselves need not be deterministic — a query or API call can and should return fresh data each run; they require *idempotency or duplication tolerance*, not determinism. The bug this prevents is concrete: branch on a raw `now()` or an unguarded random value and a replay can take a different path than the original, double-charging or skipping a step. Engines supply deterministic primitives — recorded timestamps, UUIDs, random — precisely so unavoidable non-determinism becomes a durable, replayable step.
+
+The constraint is engine-specific, not a universal law of the category. Checkpoint-based engines (DBOS, the Postgres approach) re-run the function and reuse recorded step outputs, so control-flow stability still matters but the rules are looser and differently shaped than full deterministic replay. **Decision rule:** know which model your engine uses before you write the workflow — the determinism contract is the single thing most likely to bite, and it differs between replay and checkpoint engines.
+
+### 7. Versioning running workflows is a first-class cost, not an afterthought
+
+A durable workflow can be in flight for days or months, so a deploy can land *while executions are mid-flight* against the old code. For a replay engine, changing the workflow's shape — reordering steps, adding a branch before an existing one — breaks replay of in-flight executions with a non-determinism error. This is the operational tax of long-lived workflows, and it must be designed for from the first deploy, not discovered in an incident.
+
+The two mechanisms are **patching** (guard the changed code path behind a version marker so old executions take the old branch and new ones take the new) and **worker versioning** (pin in-flight executions to the worker version they started on, and let new executions adopt new code). **Decision rule:** patch small, in-place changes to a workflow's logic; pin-and-drain whole workflow versions for larger reshapes. Either way, treat "can this change replay against everything currently running?" as a release gate. Embedded/checkpoint engines soften but do not remove this concern — a function that no longer matches the steps recorded for an in-flight run is still a hazard.
+
+### 8. It is the substrate for long-running agents
+
+A long-running agent — plan, act, observe, pause for a human, resume — is a durable workflow: its loop is checkpointed so it survives failure and resumes rather than repeating expensive tool calls, and a human approval is modelled as a **durable interrupt** that can wait for hours or days without holding a thread. The agent loop is also where the determinism contract is easiest to violate (the model output is non-deterministic by nature), so the LLM call belongs in a recorded step, never inline in the control flow. This is the reliability backbone of [Agentic Systems](../ai-native/agentic-systems.md).
+
+## How we apply this
+
+The architect names durable execution as the pattern when a process is long-running, multi-step, or must-complete, decides the orchestration/choreography split, and confirms the cost is warranted over an outbox or queue; the engineer skills implement the workflow, choose the concrete engine by operational shape, and own the determinism and versioning discipline. Durable execution is shared infrastructure the system rides, not per-process scaffolding.
+
+- [Integration Patterns](integration-patterns.md) — the outbox/saga/idempotency patterns durable execution subsumes.
+- [Agentic Systems](../ai-native/agentic-systems.md) — long-running agents are durable workflows.
+
+## Anti-patterns we reject
+
+- **Hand-rolled durability.** A retry table, a state column, and a sweeper cron re-implementing what an engine provides — fragile and bespoke per team.
+- **Workflow-as-code for everything.** A heavyweight engine where a single idempotent event handler or an outbox would do.
+- **Assuming exactly-once side effects.** Trusting the platform to make a non-idempotent `POST` safe; the engine gives at-least-once execution, and only idempotent steps make the outcome effectively-once.
+- **Non-deterministic workflows.** Branching on wall-clock time, unguarded randomness, or an inline LLM/API call, so replay diverges.
+- **Deploying without a versioning plan.** Reshaping a workflow with executions in flight and breaking their replay — versioning is a release gate, not a cleanup task.
+- **The event-sourced-monolith reflex.** Reaching for full orchestration when a simple two-step choreography is correct.
+- **Holding a thread for a human.** Blocking on an approval instead of a durable interrupt that resumes on decision.
+
+## Further reading
+
+- *Temporal*, *Restate*, *DBOS*, *AWS Step Functions* — durable execution engines spanning self-hosted replay, embedded-library, and managed state-machine shapes.
+- *Life Beyond Distributed Transactions*, Pat Helland — the reasoning the outbox and durable workflows descend from.
+- *Demystifying Determinism in Durable Execution*, Jack Vanlightly (2025) — why the constraint lives in control flow, not side effects, and how it differs across engines.
+- *The Rise of the Durable Execution Engine* — durable execution alongside an event backbone.