@nklisch/pi-agile-workflow 0.15.3

package/skills/e2e-test-design/references/anti-tautology.md

@@ -0,0 +1,261 @@
+# Anti-Tautology
+
+Patterns that make a test useless, and grep-able heuristics the `--audit` mode
+uses to find them. Referenced from SKILL.md Phase 5c (design-time
+guardrails) and Phase A2 (audit dispatch brief).
+
+## The core failure mode
+
+A tautological test verifies that the code does what the code does — it
+restates the implementation as an assertion. When the implementation changes,
+the test changes with it; when the implementation has a bug, the test has
+the same bug. It passes when the system is broken in the same way the test
+is broken. It catches nothing.
+
+The fix: tests must assert on **what the system promises a user**, not on
+**how the system computes**. The promise is the invariant. If you can't state
+the invariant in one English sentence without referring to function names or
+internal modules, you're describing implementation, not contract.
+
+## The five tautology patterns
+
+### 1. Mock-and-assert-the-mock
+
+The test sets up a mock, calls the system, then asserts the mock was called
+in a specific way. There is no real I/O, no real outcome, no real user. The
+test verifies that the wiring of mocks matches what the test author wrote
+into the mocks.
+
+```python
+# Tautological
+mock_db.save.return_value = {"id": 1}
+result = service.create_user("alice")
+assert mock_db.save.called_with(name="alice")  # asserting on the mock
+```
+
+```python
+# Non-tautological — real DB, real outcome
+result = service.create_user("alice")  # writes to a real Postgres container
+row = db.execute("SELECT name FROM users WHERE id = ?", result.id).fetchone()
+assert row["name"] == "alice"  # asserting on user-visible state
+```
+
+**Grep signal:** `assert.*called_with`, `expect.*toHaveBeenCalled`,
+`verify(mock`, `mock.assert_called`. Every match is a candidate for audit.
+
+### 2. Mirror-the-implementation
+
+The test reads as a step-by-step re-implementation of the function being
+tested. The assertions check that each internal step produced an expected
+intermediate value.
+
+```python
+# Tautological — testing the algorithm by re-running the algorithm
+def test_calculate_total():
+    items = [{"price": 10, "qty": 2}, {"price": 5, "qty": 3}]
+    expected = items[0]["price"] * items[0]["qty"] + items[1]["price"] * items[1]["qty"]
+    assert calculate_total(items) == expected
+```
+
+The `expected` value is computed by re-implementing `calculate_total`. If
+the function is wrong, the test is wrong the same way.
+
+```python
+# Non-tautological — concrete expected value
+def test_calculate_total():
+    items = [{"price": 10, "qty": 2}, {"price": 5, "qty": 3}]
+    assert calculate_total(items) == 35  # the contract: this input → this output
+```
+
+**Grep signal:** Tests where `expected` is a computed expression involving
+the same operations as the function under test. Harder to grep mechanically;
+audit sub-agent reads test bodies looking for this shape.
+
+### 3. Assertion-free tests
+
+The test runs the system but never asserts anything substantive. It "passes"
+as long as nothing throws. Most common in golden-path tests written under
+deadline pressure.
+
+```javascript
+// Tautological — runs the code, asserts nothing
+test('checkout works', async () => {
+    await page.goto('/checkout');
+    await page.click('#submit');
+});
+```
+
+The test fails only if `goto` or `click` themselves throw. It passes when
+the checkout silently submits a corrupt order.
+
+```javascript
+// Non-tautological
+test('checkout creates an order and redirects to confirmation', async () => {
+    await page.goto('/checkout');
+    await page.click('#submit');
+    await expect(page).toHaveURL(/\/confirmation\?order=\d+/);
+    const orderId = (await page.url()).match(/order=(\d+)/)[1];
+    const order = await db.query('SELECT * FROM orders WHERE id = ?', orderId);
+    expect(order.status).toBe('placed');
+});
+```
+
+**Grep signal:** test bodies with no `assert` / `expect` / `should` / `must`
+calls. Also tests whose only assertion is a `toBeTruthy()` / `toBeDefined()`
+on the call's return value with no value check.
+
+### 4. No-op golden path
+
+The test sets up a happy path but the system under test does nothing
+meaningful before the assertion fires. Common when test data accidentally
+matches the empty / default response.
+
+```python
+# Tautological — empty list assertion when list was never populated
+def test_list_users_returns_users():
+    response = client.get("/users")
+    assert isinstance(response.json(), list)  # passes for []
+```
+
+If the database was never seeded, `[]` satisfies this test. The system could
+be returning the empty list because it crashed, because it has a bug, or
+because everything works — the test can't distinguish.
+
+```python
+# Non-tautological — seeded data with specific assertion
+def test_list_users_returns_seeded_users():
+    seed_user(name="alice")
+    seed_user(name="bob")
+    response = client.get("/users")
+    names = [u["name"] for u in response.json()]
+    assert set(names) == {"alice", "bob"}
+```
+
+**Grep signal:** Hard to detect mechanically. Audit sub-agent reads tests
+where the assertion target could be satisfied by an empty/default response.
+
+### 5. Snapshot-only
+
+The test captures a snapshot and asserts the system matches it. When the
+implementation changes, the snapshot gets regenerated. No semantic invariant
+is ever stated.
+
+```javascript
+// Tautological — the snapshot becomes whatever the code produces
+test('user serialization', () => {
+    const user = serialize(buildUser());
+    expect(user).toMatchSnapshot();
+});
+```
+
+When the serialization changes to drop a critical field, the snapshot updates
+and the test still passes.
+
+```javascript
+// Non-tautological — semantic assertions
+test('user serialization includes id, name, email and never includes password', () => {
+    const user = serialize(buildUser({ password: 'secret' }));
+    expect(user).toHaveProperty('id');
+    expect(user).toHaveProperty('name');
+    expect(user).toHaveProperty('email');
+    expect(user).not.toHaveProperty('password');
+});
+```
+
+**Grep signal:** `toMatchSnapshot`, `assertSnapshot`, `.snap` files. Audit
+flags every snapshot test for human review.
+
+## Mock-boundary violation patterns
+
+Distinct from tautology but in the same audit pass. These flag tests that
+mock at the wrong boundary.
+
+### In-process mocks where a service-level mock exists
+
+```python
+# Violation — mocking the DB driver
+@patch("psycopg2.connect")
+def test_create_user(mock_connect):
+    ...
+
+# Fix — Testcontainers, real Postgres
+def test_create_user(postgres_container):
+    ...
+```
+
+**Grep signal:** `@patch.*psycopg`, `@patch.*boto3`, `@patch.*requests`,
+`jest.mock("aws-sdk")`, `sinon.stub.*http`. Every match is a candidate for
+promotion to service-level.
+
+### In-process mocks of code the system-under-test owns
+
+The system mocks its own internal modules, then asserts the integration. The
+"integration" never integrates.
+
+```python
+# Violation — mocking your own service
+@patch("app.services.payment.charge")
+def test_checkout(mock_charge):
+    mock_charge.return_value = {"status": "ok"}
+    ...
+```
+
+**Grep signal:** `@patch("app.` / `@patch("src.` / `jest.mock("./` /
+`jest.mock("@/`. Mocking internal modules in e2e tests is almost always
+wrong — the test should let the internal modules run.
+
+### Time-based mocks where libfaketime would work
+
+In-process time mocking (freezegun, sinon.useFakeTimers) is OK for unit tests
+but disqualifies an e2e test. Use libfaketime via `LD_PRELOAD` to skew the
+clock for the entire container.
+
+**Grep signal:** `freeze_time`, `freezegun`, `useFakeTimers`, `sinon.clock`.
+
+## Audit-mode heuristics
+
+The `--audit` sub-agent runs these ripgrep passes over the test directory
+and reports findings:
+
+```bash
+# 1. Mock-and-assert-the-mock
+rg -n 'assert.*called_with|expect.*toHaveBeenCalled|verify\(.*mock' tests/
+
+# 2. Internal-module mocking
+rg -n '@patch\("(app|src)\.|jest\.mock\("(\.|@/)' tests/
+
+# 3. AWS / DB driver mocks (should be service-level)
+rg -n '@patch.*\b(boto3|psycopg|pymongo|redis)|jest\.mock\("(aws-sdk|pg|mongodb|ioredis)"' tests/
+
+# 4. Time mocking (should be libfaketime in e2e)
+rg -n 'freeze_time|freezegun|useFakeTimers|sinon\.useFakeTimers' tests/
+
+# 5. Snapshot tests
+rg -n 'toMatchSnapshot|assertSnapshot' tests/ ; find tests/ -name '*.snap'
+
+# 6. Assertion-free tests (heuristic — fewer assertions than expected)
+# Sub-agent inspects per-test body; no clean grep available
+
+# 7. HTTP request mocks (should be WireMock / service-level)
+rg -n 'requests_mock|nock\(|httpretty|jest\.mock\("axios"' tests/
+```
+
+Each match is a **candidate**, not a confirmed finding. The sub-agent reads
+the matched test to confirm before filing an item.
+
+## Writing the invariant
+
+Every test in a design must have a one-line invariant stated in user-visible
+terms. Examples:
+
+- "After registration → login → POST /projects, the new project ID appears
+  in GET /dashboard"
+- "When the DB is unreachable, POST /projects returns 503 with Retry-After
+  set"
+- "parse(serialize(req)) returns a request semantically equal to req for all
+  valid requests"
+- "Checkout succeeds within 3 retries when the payment gateway has 500ms
+  latency; exactly one order row is created"
+
+If you can't write the invariant, the test is tautological. Stop and rethink
+what user-visible promise the test is supposed to verify.

package/skills/e2e-test-design/references/service-mocks.md

@@ -0,0 +1,234 @@
+# Service-Level Mocks Catalog
+
+Reference for `e2e-test-design` SKILL.md Phase 5a. Picks come from this
+catalog. If a dep type isn't here, the design must either find a service-level
+substitute, build a **custom mock container** (per the policy at the bottom),
+or carry a strong written justification for an in-process mock.
+
+## Mock-ladder principle
+
+1. **Off-the-shelf service mock** (preferred) — listed below by dep type
+2. **Custom mock container** (fallback) — language-agnostic patterns at the bottom
+3. **In-process mock** (last resort) — requires strong justification documented in the feature body
+
+Audit mode actively promotes in-process mocks to custom containers wherever
+feasible. Don't accept an in-process mock just because it's already there.
+
+## Catalog by dependency type
+
+### Cloud (AWS)
+
+**LocalStack** (`localstack/localstack`) — emulates ~30 AWS services
+(S3, DynamoDB, SQS, SNS, Lambda, IAM, Cognito, Secrets Manager, KMS, STS,
+API Gateway, EventBridge, Step Functions, CloudWatch, Kinesis, SES, SSM, etc.).
+
+```yaml
+services:
+  localstack:
+    image: localstack/localstack:latest
+    environment:
+      SERVICES: s3,sqs,dynamodb
+      AWS_DEFAULT_REGION: us-east-1
+    ports: ["4566:4566"]
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:4566/_localstack/health"]
+```
+
+Use the `awslocal` CLI or point AWS SDK clients at the endpoint URL.
+
+### Cloud (GCP)
+
+No single-image equivalent. Per-service options:
+- **Cloud Storage** → fake-gcs-server (`fsouza/fake-gcs-server`)
+- **Pub/Sub** → official emulator image (`gcr.io/google.com/cloudsdktool/cloud-sdk`)
+- **Firestore / Datastore** → official emulator
+- **BigQuery** → BigQuery emulator (community)
+
+### Cloud (Azure)
+
+- **Storage (blob/queue/table)** → Azurite (`mcr.microsoft.com/azure-storage/azurite`)
+- **Cosmos DB** → Cosmos DB Linux emulator (Microsoft official)
+- **Event Hubs / Service Bus** → no first-party emulator; consider a custom mock container
+
+### Relational databases
+
+**Testcontainers** (`testcontainers.com`) — most languages have bindings;
+spins up real database containers per test session.
+
+| Database | Image |
+|---|---|
+| Postgres | `postgres:<version>` |
+| MySQL | `mysql:<version>` |
+| MariaDB | `mariadb:<version>` |
+| MS SQL Server | `mcr.microsoft.com/mssql/server` |
+| Oracle Free | `gvenzl/oracle-free` |
+| CockroachDB | `cockroachdb/cockroach` |
+
+Use real schema migrations; seed with deterministic fixtures.
+
+### NoSQL / KV / cache
+
+| Service | Image |
+|---|---|
+| MongoDB | `mongo:<version>` |
+| Redis | `redis:<version>` |
+| DynamoDB local | `amazon/dynamodb-local` (or via LocalStack) |
+| Cassandra | `cassandra:<version>` |
+| ScyllaDB | `scylladb/scylla` |
+| Elasticsearch | `elasticsearch:<version>` |
+| OpenSearch | `opensearchproject/opensearch` |
+
+### HTTP APIs (third-party services)
+
+| Tool | When to use |
+|---|---|
+| **WireMock** (`wiremock/wiremock`) | OpenAPI-driven mocks; record/replay; configurable response delays and failure injection |
+| **Mockoon** (`mockoon/cli`) | GUI-built mock spec; export to docker |
+| **Prism** (`stoplight/prism`) | Mocks directly from an OpenAPI/Swagger document |
+| **MockServer** (`mockserver/mockserver`) | Programmatic mock expectations via REST or Java client |
+
+WireMock is the default workhorse — it handles record-and-replay, stateful
+scenarios, and is fully configurable as a docker service.
+
+### Email (SMTP)
+
+| Service | Image | Notes |
+|---|---|---|
+| **MailHog** | `mailhog/mailhog` | SMTP sink + HTTP UI for inspecting sent mail |
+| **smtp4dev** | `rnwood/smtp4dev` | Similar, with .NET-friendly features |
+| **MailCatcher** | `schickling/mailcatcher` | Lightweight |
+
+### Object storage (S3-compatible)
+
+| Service | Image |
+|---|---|
+| **MinIO** | `minio/minio` |
+| LocalStack S3 | `localstack/localstack` (S3 service enabled) |
+
+### Message queues / streaming
+
+| Service | Image |
+|---|---|
+| Kafka (single-node, fast) | `redpandadata/redpanda` (Kafka-compatible, faster startup) |
+| Kafka (official) | `apache/kafka` |
+| RabbitMQ | `rabbitmq:<version>-management` |
+| NATS | `nats:<version>` |
+| MQTT | `eclipse-mosquitto` |
+
+### Auth / identity
+
+| Service | Image | When |
+|---|---|---|
+| **Keycloak** | `quay.io/keycloak/keycloak` | OAuth2/OIDC/SAML — full IdP |
+| **dex** | `ghcr.io/dexidp/dex` | Lighter OIDC provider |
+| **mock-oauth2-server** | `ghcr.io/navikt/mock-oauth2-server` | Minimal OIDC for tests |
+
+### Network failure injection
+
+| Tool | Image / Binary | Capabilities |
+|---|---|---|
+| **Toxiproxy** | `ghcr.io/shopify/toxiproxy` | latency, bandwidth, jitter, slow-close, down, limit-data, slicer toxics |
+| **Pumba** | `gaiaadm/pumba` | container kill, pause, stop, rm; netem-based netem injection |
+| **tc / netem** | host kernel | direct Linux traffic-control rules |
+
+### Time / clock
+
+| Tool | Approach |
+|---|---|
+| **libfaketime** | `LD_PRELOAD` shim; overrides syscalls; works inside containers |
+| Frozen-time test helpers | language-level (freezegun in Python, sinon in JS) — these are in-process mocks; prefer libfaketime for true e2e |
+
+### Browsers / E2E UI
+
+Not "service mocks" exactly, but commonly part of the same stack:
+
+| Tool | Notes |
+|---|---|
+| **Playwright** | Multi-browser; can run headless in CI; built-in trace viewer |
+| **Selenium Grid** | Multi-node; mature; heavier |
+| Cypress (note: in-process) | Useful but its same-origin and in-process design means it's NOT an e2e tool by the strict definition here |
+
+### Search
+
+| Service | Image |
+|---|---|
+| Elasticsearch | `elasticsearch:<version>` |
+| OpenSearch | `opensearchproject/opensearch` |
+| Meilisearch | `getmeili/meilisearch` |
+| Typesense | `typesense/typesense` |
+
+### Background job systems
+
+Most are layered on a queue/DB and don't need a separate mock — bring up the
+underlying Redis/Postgres/RabbitMQ and the job system runs against it
+naturally.
+
+## Building a custom mock container
+
+When no off-the-shelf service-level mock exists for a dependency, design a
+purpose-built container. **Do not default to a single framework or language**
+— that's how mock containers become alien to the project they're embedded in.
+
+### Picking the framework
+
+In order:
+
+1. **Match the upstream service's actual tech.** If you're mocking an internal
+   Rails API, a Rails skeleton is the most faithful substitute — the same
+   serialization quirks, the same routing semantics. If the upstream is a Go
+   gRPC service, mock it in Go with the same `.proto` files.
+
+2. **Match the project's primary stack.** If the project is a TypeScript
+   monorepo, a TypeScript mock is one less language to maintain. If the
+   project is Python, the mock is Python.
+
+3. **Ask the user.** When neither rule suggests an obvious choice (e.g., a
+   greenfield project mocking a third-party SaaS), surface the decision via
+   `structured question tool`.
+
+### Container shape
+
+Regardless of language, a custom mock container should:
+
+- Be **stateless** between test runs (state in memory or a wiped volume)
+- Be **deterministic** — same input → same output, no randomness without a
+  seed knob
+- Expose a **healthcheck endpoint** for docker-compose `depends_on:
+  condition: service_healthy`
+- Expose a **reset endpoint** (`POST /__reset`) the test fixture calls between
+  tests to wipe state
+- Expose **inspection endpoints** (`GET /__received`) so tests can assert what
+  the system-under-test sent
+- Read **scenario config** from a file or env var so different tests can
+  configure different failure modes (slow, error, partial response, etc.)
+
+### Patterns
+
+**Record-and-replay** — when you have access to the real upstream, capture
+real traffic (WireMock's `record` mode, or a small proxy) and replay it in
+tests. Best for stable, read-heavy upstreams.
+
+**Configurable WireMock** — for HTTP-only deps, a WireMock container with
+mounted mappings JSON often replaces a hand-written custom server. Less code
+to maintain.
+
+**Mini-server with deterministic logic** — when the upstream has nontrivial
+behavior (state machines, calculations, side effects), a small server in the
+matched language is right. Keep the surface tight: only the endpoints the
+system-under-test actually calls.
+
+**Hybrid (real service + injected behavior)** — sometimes the upstream IS
+something you can run in docker (e.g., your own service) but you need to
+inject failure. Toxiproxy in front of the real service is the right tool;
+don't build a custom mock.
+
+### Anti-patterns
+
+- Building a Python mock for a Java project just because Python is "easier" —
+  the project ends up maintaining a Python toolchain solely for tests
+- Reimplementing the upstream's full surface — only mock what the
+  system-under-test actually calls
+- Mock containers that drift from the real upstream's behavior without anyone
+  noticing — pin to a real-upstream contract test where possible
+- Mock containers that share state between tests — every test should start
+  clean via `__reset`