module-spec 0.0.0-stage → 0.2.0

package/spec/tests.md

@@ -0,0 +1,146 @@
+# Tests
+
+A module's shape decides its tests. Tests prove behaviour, never structure,
+so an agent can rebuild a module's inside and every test still holds (Ian
+Cooper, "TDD, Where Did It All Go Wrong", 2017).
+
+```mermaid
+---
+config:
+  look: handDrawn
+---
+flowchart LR
+  e2e["🌐 end-to-end<br/>a few, in the app"]:::test
+  behave["🧪 behaviour tests<br/>through the door,<br/>fakes passed in"]:::test
+  contract["📜 contract test<br/>one per port"]:::test
+  app["🚀 app"]:::part
+  subgraph mod["📦 a module"]
+    door["🚪 door"]:::part
+    shell["🔁 shell"]:::part
+    core["🧠 core"]:::part
+    adapter["🔧 real adapter"]:::part
+    fake["🧸 fake"]:::part
+  end
+  e2e --> app --> door
+  behave --> door
+  door --> shell --> core
+  shell --> adapter
+  contract --> adapter
+  contract --> fake
+  classDef test fill:#1b4332,stroke:#52b788,color:#d8f3dc
+  classDef part fill:#2b2b2b,stroke:#9ca3af,color:#e5e7eb
+```
+
+Green: the three kinds of test. Core and shell have no tests of their own:
+behaviour tests reach them through the door. Each adapter is covered by its
+port's contract test. Tests that span features live in an app, because
+[R1](roles.md#r1-a-feature-imports-only-shared-modules) holds for test files
+too. A frontend module's behaviour tests draw its page through the door, see
+[frontend.md](frontend.md#tests).
+
+## Good design deletes tests
+
+Move each fact as high up this ladder as it goes. Each step up deletes the
+tests below it.
+
+| Level | Example | Tests needed |
+|---|---|---|
+| Impossible, by a type | an `InvoiceId` cannot be a raw string | none |
+| Blocked, by a check | a core cannot import the database | none |
+| Tested | the fee for 3 hours is 300 | one |
+| Reviewed | a name reads well | none, a person decides |
+
+The top rung: make illegal states unrepresentable (Yaron Minsky, "Effective
+ML", 2011), parse, don't validate ([B7](backend.md#b7-parse-inputs-at-the-edge)),
+and design errors out of existence (John Ousterhout, *A Philosophy of Software
+Design*). Code that is hard to test is a design signal, not a testing problem
+(Steve Freeman and Nat Pryce, *Growing Object-Oriented Software, Guided by
+Tests*, 2009).
+
+## T1. Tests reach a module only through its door
+
+A module's tests import its door, never the files behind it. The one
+exception is a port's contract test ([T3](#t3-every-port-has-one-contract-test)),
+which imports the port and its adapters.
+
+**Why:** a test that imports an inside file breaks when the inside is
+rebuilt, even when the behaviour is unchanged. That false alarm teaches agents
+to edit tests until they pass.
+**Check:** the import checker. In Go, also the package clause: a test in
+`package billing` sees the inside, so tests use `package billing_test`.
+
+## T2. Tests pass fakes in, never patch
+
+A test hands the module its fakes through `createX(deps)`. It never patches:
+no `vi.mock`, `jest.mock`, `spyOn`, `mock.method`, `unittest.mock.patch`,
+`monkeypatch`, `mocker`, or a patching library such as Go's `gomonkey`.
+
+**Why:** a patch swaps what a module imports behind its back, so the test
+knows the module's inside. Each patch also marks a seam that is missing: a
+dependency the module should have taken as a port.
+**Check:** a scan of every test file for the patching calls of its language.
+
+## T3. Every port has one contract test
+
+A contract test is one suite of tests, written once against the port, that
+runs against every adapter of that port: the in-memory fake and the real one
+(J. B. Rainsberger, "Integrated Tests Are a Scam", 2009).
+
+```ts
+// billing/__tests__/InvoiceStore.contract.test.ts
+export function invoiceStoreContract(make: () => Promise<InvoiceStore>): void {
+  test("finds the invoice it saved", async () => {
+    const store = await make()
+    await store.save({ id: "inv-1", cents: 300 })
+    deepStrictEqual(await store.find("inv-1"), { id: "inv-1", cents: 300 })
+  })
+  test("finds nothing for an unknown id", async () => {
+    const store = await make()
+    deepStrictEqual(await store.find("inv-404"), null)
+  })
+}
+
+describe("in-memory fake", () => invoiceStoreContract(async () => inMemoryInvoiceStore()))
+describe("Postgres", () => invoiceStoreContract(async () => postgresInvoiceStore(await emptyTestDatabase())))
+```
+
+**Why:** every behaviour test runs against the fake. If the fake drifts from
+the real database, those tests pass against a lie. The contract test keeps the
+two the same, so behaviour tests can stay fast. The real run needs the real
+service, so it runs where one exists, such as CI with a database container.
+**Check:** every file in `ports/` has a contract test named after it, such as
+`InvoiceStore.contract.test.ts`, `test_invoice_store_contract.py` or
+`invoice_store_contract_test.go`. An adapter implements a port by importing
+it, and the contract test must import every adapter that does, so the real one
+cannot be left out. `new-port` writes the suite with the fake already in it.
+
+## Mutation testing
+
+A tool changes a module's code in one small way at a time, such as `>` to
+`>=`, and runs the module's tests against each change, a mutant (Richard
+DeMillo, Richard Lipton and Frederick Sayward, "Hints on Test Data
+Selection", 1978). Some test must fail for each one. A mutant every test
+passes with lives.
+
+**Why:** T1 and T2 cannot see a test that pins nothing, such as
+`expect(feeFor(3)).toBe(feeFor(3))`. It runs the code and passes, whatever the
+code does. Mutation testing catches it, because every mutant lives.
+
+- **What is mutated:** `core/`, `shell/` and the door. Not `ports/`, which
+  hold only types, not `adapters/`, whose contract tests need the real
+  service, and not `view/`, which holds no rules
+  ([F3](frontend.md#f3-a-view-holds-no-rules)).
+- **Which tests count:** only the module's own, and not its adapters' tests
+  or contract tests. Those prove an adapter against the outside and are slow.
+  The core they reach is proven through the door. A mutant that only another
+  module's test, or an adapter's test, catches still lives.
+- **Zero live mutants:** `module-spec mutate` fails on any live mutant, and
+  names it, even on a module's first run.
+- **When no test can tell:** some mutants behave exactly like the code, such
+  as a change to a log message. `module-spec mutate --accept` lists them in the
+  module's `mutation-report.txt`, so the diff shows a reviewer each one. A
+  module with no accepted mutant has no report.
+- **Cost:** a run takes seconds to minutes for one module, so it runs only on
+  the modules a change touched ([W1](workflow.md#w1-one-module-per-chat)).
+  Their dependents run their tests, but not mutation testing: their code and
+  tests did not change, so neither did their live mutants.

package/spec/workflow.md

@@ -0,0 +1,50 @@
+# Working on modules in parallel
+
+How agents work on several modules at once without breaking each other.
+
+## W1. One module per chat
+
+Each chat works in its own git worktree, on one module. Worktrees stop two
+chats from editing the same lines. They do not stop two chats from changing
+the same decision; the rules in [common.md](common.md) do that.
+
+**Check:** `check` fails when a branch changes more than one module besides
+apps. An app only wires, so a feature and the app that wires it may change
+together. Work that spans modules lands one module at a time
+([W2](#w2-a-door-change-lands-first-alone)).
+
+A chat tests what its change can break, not the whole project.
+`module-spec affected` lists the module it changed and every module that
+depends on it, directly or through others. The chat runs those modules' tests.
+Then `module-spec mutate` tests the tests of the modules it changed
+([tests.md](tests.md#mutation-testing)). A
+change outside every module, such as a new package version, is listed apart,
+because only the full suite covers it. [W3](#w3-test-the-merged-result) still
+runs everything on the merged result.
+
+## W2. A door change lands first, alone
+
+When modules need a new or changed door, that change merges as its own small
+pull request before the work that uses it. Each chat then builds against a
+door that is already on `main` (parallel change, Danilo Sato, "ParallelChange",
+martinfowler.com).
+
+A change that breaks callers lands in three steps, each one module and so
+within [W1](#w1-one-module-per-chat):
+
+1. **Expand:** the door exports the new shape beside the old one.
+2. **Migrate:** each calling module moves to the new shape on its own branch.
+3. **Contract:** once no caller uses the old shape, the door drops it, and
+   `module-spec api` shows the line go.
+
+Before a project has any `module.yaml`, the branch rules do not run, so doors
+can move with their callers in one branch.
+
+## W3. Test the merged result
+
+Two branches can each pass CI, merge with no conflict, and still break `main`,
+because both changed one decision (a semantic conflict). Two habits catch most
+of these:
+
+- CI runs on each branch rebased onto the current `main` before it merges.
+- Each user journey has one end-to-end test.