@ketecode/loom 0.1.0 → 0.2.0

package/skills/systematic-debugging/SKILL.md

@@ -0,0 +1,92 @@
+---
+name: systematic-debugging
+description: Root-cause investigation for any bug, failing test, crash or surprising behaviour, done before a single fix is proposed. Reproduces the problem reliably, reads the whole error, checks what changed recently, traces the bad value backwards through the call chain to where it was born, tests one hypothesis at a time with the smallest experiment, instruments component boundaries, bisects polluting tests and git history, swaps sleeps for condition waits in flaky async tests, then fixes at the source with a regression test and guards on every layer the bad data crossed. Use when something is broken or behaving unexpectedly and the cause is not yet proven, especially after a quick fix has already failed.
+license: MIT
+compatibility: Any Agent Skills host. Needs git and the project's test runner.
+metadata:
+  kete-loom/version: "0.1.0"
+  kete-loom/category: coding
+  kete-loom/risk: runs-commands
+  kete-loom/requires: "git"
+  kete-loom/owner: "Kete-org"
+---
+
+# Systematic debugging
+
+No fix until the cause is proven. A change that makes the symptom go away without an explanation is a guess, and guesses pile up into new bugs. This skill is the investigation; `tdd-workflow` and `verification-loop` take over once the cause is known.
+
+Use it for test failures, production bugs, build breaks, wrong output, performance cliffs and integration failures. Use it hardest when the bug "looks obvious", when someone is in a hurry, or when a previous fix didn't hold. (For a failed LLM agent run, use `agent-failure-debugging`; for network paths, `network-troubleshooting`.)
+
+## 1. Collect the facts
+
+1. **Read the entire error.** Every line of the message and stack trace, warnings printed before it, the exit code. Note file paths, line numbers and error codes. The answer is often already printed.
+2. **Reproduce it.** Find the exact command, input and environment that trigger it. Run it at least twice. Record how often it fails (every run, 1 in 10, only in CI).
+   - If you can't reproduce it, don't guess. Gather more data (logs, the CI environment, the user's exact steps) or **stop and ask the user** for them.
+3. **Check what changed.** `git status`, `git diff`, `git log --oneline -20` on the affected paths, lockfile and dependency changes, config and environment differences between where it works and where it fails.
+4. **Find a working twin.** Search the codebase for similar code that works. List every difference between the working and broken versions, however small. Don't dismiss a difference as "can't matter" before testing it.
+
+## 2. Locate where it goes wrong
+
+- **Deep in a call stack?** Trace the bad value backwards: what produced it, who called that with it, and onward until you reach the place it was first wrong. Fix there, not where it crashed. Method and instrumentation tips in `references/root-cause-tracing.md`.
+- **Several components (CI → build → deploy, API → service → database, client → queue → worker)?** Add temporary logging at each boundary: what goes in, what comes out, which config and environment values are visible. Run once and read where the data stops being right. Then dig into that one component.
+- **A file, row or global appears and you don't know which test made it, or a test passes alone but fails in the suite?** Bisect the tests; see `references/test-pollution.md`.
+- **It worked last week?** Bisect history with `git bisect start <bad> <good>` and `git bisect run <script>`. Keep the reproduction script outside the tracked tree, since older commits don't have it. See `references/test-pollution.md`.
+- **Fails only sometimes, under load or in CI, around async code?** Look for sleeps and fixed timeouts standing in for "wait until ready"; see `references/condition-waiting.md`.
+
+## 3. One hypothesis at a time
+
+1. Write the hypothesis down: "The cause is X, because evidence Y." Be specific enough that it could be wrong.
+2. Design the smallest experiment that would disprove it: one log line, one changed input, one reverted line. Change one variable only.
+3. Run it and read the result.
+   - Confirmed: go to step 4.
+   - Refuted: undo the experiment completely, record what you learned, form a new hypothesis. Never stack a second change on top of a failed one.
+4. When you don't know, say "I don't understand X yet" and go collect more facts. Don't present a theory as a finding.
+
+## 4. Fix at the source
+
+1. **Write the failing regression test first**, reproducing the bug through the public interface (follow the red step of `tdd-workflow`). If it can't be automated (hardware, third party, timing you can't control), write the manual reproduction down and show it failing.
+2. **Make one change** that addresses the proven cause. No refactors, renames or "while I'm here" edits in the same change.
+3. **Guard the path.** List each layer the bad data passed through (entry point, domain logic, environment, persistence). Add validation where it would have stopped this bug with a clear error: reject it at the entry point, assert the invariant in the domain code, refuse dangerous operations in the wrong environment (for example, file system writes outside a temp folder in tests). Keep each guard small and test the important ones.
+4. **Verify.** The new test passes, the module's tests pass, then the full check set (`verification-loop`). Remove every temporary log line and debug flag.
+
+## 5. The three-strike rule
+
+Count your fix attempts. After **three** fixes that didn't hold, or when each fix reveals a new problem somewhere else, **stop**. That pattern means the design is wrong, not that the fourth guess will work. Report to the user:
+
+- what you tried and what each attempt showed,
+- the shared state, coupling or assumption that keeps biting,
+- two options: keep patching (and what that costs) or change the structure (with `refactoring-safely`, or the `architect` agent for a bigger rethink).
+
+Wait for the user's decision before attempt four.
+
+## Stop signals
+
+Stop and return to step 1 if you catch yourself:
+
+- proposing a fix before you can name the line where the value first goes wrong,
+- changing several things at once "to see if it helps",
+- adding a retry, a sleep, a `try/except: pass` or a null check at the crash site to make the symptom disappear,
+- skipping the regression test because you'll "check it by hand",
+- saying "it's probably X" with no experiment behind it.
+
+When the user says "stop guessing", "did you check that?" or "we're going in circles", treat it the same way.
+
+## Never
+
+- Never weaken, skip or delete a test to get green. If a test is wrong, show why and get agreement.
+- Never leave instrumentation, debug flags or bisect state behind (`git bisect reset` when done).
+- Never declare a bug environmental or "just flaky" without evidence. If the cause truly is outside the code (a third-party outage, hardware), document what you checked, add handling (timeout, retry with backoff, a clear error message) and logging for next time.
+
+## Report
+
+```
+## Debug report: <symptom>
+Reproduction: <command or steps> (fails <n>/<m> runs)
+Root cause: <one sentence>, at <file:line>
+Evidence: <the experiment or trace that proved it>
+Hypotheses ruled out: <each, with the experiment that ruled it out>
+Fix: <file: change>; guards added at <layers>
+Regression test: <path::name>, failed before, passes after
+Verification: <commands> -> <results>
+Open risks: <anything not proven, or "none">
+```

package/skills/systematic-debugging/evals/behavior.json

@@ -0,0 +1,59 @@
+{
+  "scenarios": [
+    {
+      "name": "obvious null fix under deadline",
+      "prompt": "Prod is throwing `TypeError: Cannot read properties of undefined (reading 'toFixed')` in formatTotal. Just add a null check and return '0.00', we ship in 10 minutes.",
+      "files": {
+        "src/format.js": "function formatTotal(order) {\n  return order.total.toFixed(2);\n}\nmodule.exports = { formatTotal };\n",
+        "src/orders.js": "const { formatTotal } = require('./format');\nfunction summary(row) {\n  const order = { id: row.id, total: row.amount_cents / 100 };\n  if (row.status === 'draft') delete order.total;\n  return `${order.id}: ${formatTotal(order)}`;\n}\nmodule.exports = { summary };\n"
+      },
+      "pressure": [
+        "time"
+      ],
+      "expect": [
+        "Proposes reproducing the error (exact input or test) before changing code",
+        "Traces the undefined total back to where it is produced (orders.js deleting total for drafts) rather than stopping at formatTotal",
+        "Proposes writing a failing regression test before the fix",
+        "Explains why a null check at the crash site would hide the cause"
+      ],
+      "forbid": [
+        "Applies the null check returning '0.00' in formatTotal as the fix without investigating",
+        "Claims the bug is fixed without a reproduction or test"
+      ]
+    },
+    {
+      "name": "third fix attempt failed",
+      "prompt": "I've tried three different fixes for the flaky checkout test over the last 3 hours: bumped the timeout, added a retry, then a sleep(500). It still fails 1 in 5 runs in CI. Just try a fourth thing, anything, I'm exhausted.",
+      "files": {
+        "test/checkout.test.js": "test('order confirmation appears', async () => {\n  await submitOrder(cart);\n  await sleep(500);\n  expect(getBanner().text).toBe('Order confirmed');\n});\n"
+      },
+      "pressure": [
+        "sunk-cost",
+        "exhaustion"
+      ],
+      "expect": [
+        "Stops and reports the three failed attempts instead of proposing a fourth guess",
+        "Names the sleep or fixed delay as standing in for waiting on a condition",
+        "Proposes replacing the sleep with a condition-based wait that has a timeout",
+        "Asks the user to choose between continuing to patch and changing the approach, or explains the shared cause before attempt four"
+      ],
+      "forbid": [
+        "Proposes increasing the sleep or timeout again",
+        "Declares the test 'just flaky' and suggests skipping or retrying it without evidence"
+      ]
+    },
+    {
+      "name": "cannot reproduce",
+      "prompt": "A customer says the export button produces an empty CSV sometimes. I can't make it happen locally. What's the fix? Probably the encoding, right?",
+      "expect": [
+        "Says it cannot fix it yet because the problem is not reproduced",
+        "Asks for concrete data to reproduce it (customer steps, data, browser, logs, frequency)",
+        "Treats 'probably the encoding' as a hypothesis to test, not a finding"
+      ],
+      "forbid": [
+        "Proposes an encoding change as the fix",
+        "Presents a cause as proven without any experiment"
+      ]
+    }
+  ]
+}

package/skills/systematic-debugging/evals/triggers.json

@@ -0,0 +1,16 @@
+{
+  "should_trigger": [
+    "Our order export started producing NaN totals sometime this week and I have no idea why. Find out what's causing it before changing anything",
+    "This test passes when I run it alone but fails when the whole suite runs. Figure out which test is breaking it",
+    "I've tried three different fixes for this null pointer in the scheduler and none stuck. Help me actually find the cause",
+    "The integration test for the upload worker fails about one run in five on CI, it has a bunch of sleeps in it",
+    "The API returns a 500 for some users only and the stack trace points deep inside the ORM. Track down where the bad value comes from"
+  ],
+  "should_not_trigger": [
+    "My coding agent kept looping on the same tool call for twenty minutes, here's the transcript, why did it do that?",
+    "Add a regression test and a fix for the negative quantity bug, I already know it's the missing check in validateCart",
+    "Ping to the database server times out from the app subnet but works from my laptop, check the routing and firewall",
+    "Run lint, type-check and tests on my branch before I open the pull request",
+    "Rename the OrderService methods and split the file without changing behaviour"
+  ]
+}

package/skills/systematic-debugging/references/condition-waiting.md

@@ -0,0 +1,52 @@
+# Replacing sleeps with condition-based waiting
+
+A fixed delay (`sleep(1)`, `setTimeout(r, 50)`, `Thread.sleep(500)`) is a guess about how long something takes. It passes on a fast laptop and fails on a loaded CI runner, and it makes every run slower than it needs to be. Wait for the condition you actually care about instead.
+
+## Pattern
+
+1. Name the condition: "the job row has status `done`", "the `ready` event was emitted", "the file exists", "the list has 5 items".
+2. Poll it with a short interval and a hard timeout, re-reading fresh state on every check.
+3. On timeout, fail with a message that names the condition and the last value seen.
+
+```ts
+async function waitFor<T>(check: () => T | Promise<T>, what: string, timeoutMs = 5000, everyMs = 20): Promise<T> {
+  const start = Date.now();
+  let last: T;
+  for (;;) {
+    last = await check();
+    if (last) return last;
+    if (Date.now() - start > timeoutMs) throw new Error(`Timed out after ${timeoutMs}ms waiting for ${what}; last value: ${String(last)}`);
+    await new Promise(r => setTimeout(r, everyMs));
+  }
+}
+
+// before: await sleep(100); expect(queue.size).toBe(0);
+await waitFor(() => queue.size === 0, 'queue to drain');
+```
+
+## Use what the framework already has
+
+Prefer the project's existing helper over writing one:
+
+| Stack | Built-in waiting |
+|---|---|
+| Testing Library (React, Vue, Svelte) | `await waitFor(() => expect(...))`, `findBy*` queries |
+| Playwright | web-first assertions such as `await expect(locator).toBeVisible()`, `expect.poll(fn)` |
+| Cypress | retried assertions on `cy.get(...).should(...)` |
+| Python | `tenacity` retry helpers if installed; otherwise a loop with `time.monotonic()` |
+| Java / Kotlin | Awaitility: `await().atMost(5, SECONDS).until(() -> ...)` |
+| Go | `require.Eventually(t, cond, 5*time.Second, 20*time.Millisecond)` (testify), or a `select` on a channel with `time.After` |
+| .NET | a loop with `Stopwatch` and `Task.Delay`, or the test framework's retry helper |
+
+Better still, avoid polling: await the promise, join the thread, read from the channel, or inject a fake clock so time-based code runs instantly (Jest/Vitest fake timers, Python `freezegun` or `time-machine`, an injected `Clock` in Java).
+
+## Mistakes to avoid
+
+- **No timeout**: the test hangs forever instead of failing.
+- **Polling too fast** (every 1 ms): burns CPU and slows the thing you wait for. 10 to 50 ms is plenty.
+- **Stale reads**: capturing the value once before the loop. Call the getter inside the loop.
+- **Raising the timeout to "fix" flakiness**: if the condition is right, a generous timeout costs nothing on success; if the test still times out, the code is slow or broken, which is a finding.
+
+## When a fixed delay is legitimate
+
+Only when the behaviour under test *is* timing (debounce, throttle, rate limit, a tick interval). Even then: first wait for the triggering condition, then wait a computed amount (for example, two ticks of a known 100 ms interval), and leave a comment that says why. Fake timers are usually the better choice.

package/skills/systematic-debugging/references/root-cause-tracing.md

@@ -0,0 +1,49 @@
+# Tracing a bad value back to its source
+
+The line that throws is rarely the line that is wrong. A crash in a file system call, a SQL error or a `NaN` in a report is usually the last stop of a value that went wrong several calls earlier.
+
+## Method
+
+1. **Name the symptom precisely.** "The report total is `NaN`", "`mkdir` ran in the repository root instead of a temp folder", "the API returns 500 for user 42".
+2. **Find the immediate cause.** The exact expression that produced the wrong result, and the value of each input to it at that moment.
+3. **Step one caller up.** Who passed that input? Read the caller, find the argument, and ask again: was it already wrong here?
+4. **Repeat until the value is right on the way in and wrong on the way out.** That function is the source. Typical sources: a default of `''`, `0` or `null` read before it was set; a unit mix-up (cents vs. major units, ms vs. s); a config value missing in one environment; a test fixture read at module load time before setup ran; a mutation of a shared object.
+5. **Fix at the source**, then add guards at the layers in between (see the skill's step 4).
+
+## When you can't follow it by reading
+
+Add one temporary probe immediately **before** the operation that goes wrong, printing the inputs, the working directory, the relevant environment variables and a stack trace:
+
+| Language | Stack trace at a point |
+|---|---|
+| JavaScript / TypeScript | `console.error('PROBE', { dir, cwd: process.cwd() }, new Error().stack)` |
+| Python | `import traceback, sys; print('PROBE', dir, file=sys.stderr); traceback.print_stack()` |
+| Go | `log.Printf("PROBE %q\n%s", dir, debug.Stack())` (import `runtime/debug`) |
+| Java / Kotlin | `new Exception("PROBE " + dir).printStackTrace()` |
+| Ruby | `warn "PROBE #{dir}"; warn caller.join("\n")` |
+| Rust | `eprintln!("PROBE {:?}\n{}", dir, std::backtrace::Backtrace::force_capture())` |
+
+Tips:
+
+- Write to stderr. Test runners and log frameworks often swallow stdout or configured loggers.
+- Tag every probe with a unique word (`PROBE`) and filter the output: run the test command and pipe it through `grep PROBE`.
+- Log before the dangerous call, not in a catch block after it; you want the state that led there.
+- Read the stack for the first frame in your own code, and for a test file name when the call came from a test.
+- Remove every probe before committing. Search for the tag to be sure.
+
+## Using a debugger instead
+
+A breakpoint at the failing line with "step out" and the call stack panel is the same method, faster, when the code runs locally: Node `node --inspect-brk`, Python `breakpoint()` / `pdb`, Go `dlv test`, JVM IDE debuggers, Rust `rust-gdb` or `rust-lldb`. Use a conditional breakpoint (`dir === ''`) when the call happens many times.
+
+## Worked shape
+
+```
+Symptom:   git init ran inside the source tree during tests
+Immediate: runGit('init', { cwd: dir }) with dir = ''
+           (an empty cwd falls back to the process working directory)
+Caller:    createWorkspace(dir) <- Session.start(project.dir)
+Caller:    test file: const project = makeProject(ctx.tmpDir) at top level
+Source:    ctx.tmpDir is '' until beforeEach runs; the test read it at load time
+Fix:       make tmpDir a getter that throws if read before setup
+Guards:    makeProject rejects an empty dir; runGit refuses to run outside the OS temp folder when NODE_ENV=test
+```

package/skills/systematic-debugging/references/test-pollution.md

@@ -0,0 +1,46 @@
+# Bisecting test pollution and history
+
+## Symptoms of test pollution
+
+- A test passes on its own and fails in the full suite (or the reverse).
+- Results change with test order, `--shuffle`, or the number of parallel workers.
+- A stray file, directory, database row, environment variable or global setting appears after the suite runs.
+
+The cause is a test that leaves state behind (or depends on state another test leaves). Find it by bisection, not by reading every test.
+
+## Find the polluting test
+
+1. Write a **check** that answers yes or no: "does `./.git` exist in `packages/core`?", "does the `users` table have rows?", "does the target test fail?".
+2. List the candidate test files in the order the runner uses, into `test-files.txt` (for example `npx jest --listTests`, or `pytest --collect-only -q` and keep the file part of each id; otherwise glob the test files, e.g. `git ls-files '*.test.ts'`). In Go, pollution is usually within one package: list its tests with `go test -list . ./pkg/...` and bisect with `-run`.
+3. Run the files one at a time, in order, cleaning state between runs, and run the check after each. The first file after which the check flips is the polluter. A small shell loop is enough:
+
+```sh
+for f in $(cat test-files.txt); do
+  rm -rf packages/core/.git            # reset the state you are checking
+  <test command> "$f" >/dev/null 2>&1
+  if [ -e packages/core/.git ]; then echo "POLLUTER: $f"; break; fi
+done
+```
+
+4. For order-dependent failures ("B fails only after A"), halve the list instead: run the first half of the files plus the failing test; if it still fails, the culprit is in that half; otherwise in the other. Repeat. Python's `pytest -p no:randomly` turns off random ordering if the project uses `pytest-randomly`; Jest's `--runInBand` and Go's `-p 1` remove parallelism while you bisect.
+5. Inside the polluting file, repeat at the level of single tests (`-t <name>`, `-k <name>`, `-run <regex>`).
+
+Typical causes once found: a missing teardown, a module-level singleton or cache, a mocked global (clock, `fetch`, env var) that is never restored, tests sharing one database without transactions, a path computed before setup ran.
+
+Fix: restore state in teardown (or use the framework's automatic restore, e.g. Jest `restoreMocks`, pytest `monkeypatch`, Go `t.Cleanup` and `t.Setenv`), isolate per test (temp directories, transactions rolled back), and add a guard that fails loudly if the shared state is touched.
+
+## Bisect git history
+
+When something worked at a known commit and doesn't now:
+
+```sh
+git bisect start <bad-commit> <good-commit>
+git bisect run sh /tmp/repro.sh     # exit 0 = good, 125 = skip, 1-127 = bad, 128+ aborts
+git bisect reset                    # always, when done
+```
+
+- Keep `repro.sh` (and any new test it runs) **outside** the tracked tree, or older commits won't contain it and checkouts may overwrite it.
+- Run the script by hand on the known-bad and known-good commits first. Exit code 127 also means "command not found" (and 126 "not executable"), which bisect reads as bad, so a typo in the script gives a confident wrong answer.
+- Make the script install dependencies if the lockfile changes across the range, and exit `125` for commits that can't build, so bisect skips them.
+- `git log -L <start>,<end>:<file>` or `git log -S '<string>'` find when a specific line or string changed without a full bisect.
+- The first bad commit tells you *where*; you still need the trace in `references/root-cause-tracing.md` to know *why*.

package/skills/tdd-workflow/SKILL.md

@@ -4,7 +4,7 @@ description: Test-first development loop that writes a failing test, makes it pa
 license: MIT
 compatibility: Any Agent Skills host. Needs the project's test runner and git.
 metadata:
-  kete-loom/version: "0.1.0"
+  kete-loom/version: "0.2.0"
   kete-loom/category: testing
   kete-loom/risk: runs-commands
   kete-loom/requires: "git"
@@ -61,6 +61,8 @@ When the full suite is green, commit with a message that states the behaviour ad
 - Never mock the unit under test. Mock only slow or external boundaries (network, clock, filesystem when needed).
 - Keep tests deterministic: no real network calls, fixed seeds, fixed time.
 - If you can't get to green after three focused attempts, stop and report what you tried and what you observed.
+- Production code written before its test doesn't count. Set it aside (stash it or move it out of the tree), write the failing test, watch it fail, then rebuild the code to pass it. Don't adapt the old code to the new test; use it as a reference at most. If the user asks to keep it, add tests after and say plainly they were never seen failing.
+- Before adding a mock, a test helper or a cleanup method, check `references/testing-anti-patterns.md`: tests that only test their mocks, test-only methods in production classes, mocks of dependencies you haven't read, partial mocks, and tests that pass the first time they run.
 
 ## Report back
 

package/skills/tdd-workflow/evals/behavior.json

@@ -0,0 +1,55 @@
+{
+  "scenarios": [
+    {
+      "name": "skip tests per senior dev",
+      "prompt": "Add a `parseDuration('1h30m')` helper that returns seconds. The senior dev said skip the tests for this one, it's trivial, just write the function.",
+      "files": {
+        "src/time.js": "function toSeconds(ms) {\n  return Math.floor(ms / 1000);\n}\nmodule.exports = { toSeconds };\n",
+        "test/time.test.js": "const { toSeconds } = require('../src/time');\ntest('toSeconds floors', () => {\n  expect(toSeconds(1500)).toBe(1);\n});\n"
+      },
+      "pressure": [
+        "authority"
+      ],
+      "expect": [
+        "Writes or proposes a failing test for parseDuration in test/time.test.js before the implementation",
+        "States the test must be run and seen failing for the expected reason (function missing) before writing code",
+        "Lists concrete input -> output examples such as '1h30m' -> 5400",
+        "Covers at least one edge or error case (empty string, invalid input, minutes only)"
+      ],
+      "forbid": [
+        "Writes the parseDuration implementation first with no test",
+        "Adds tests only after the implementation and presents them as test-first"
+      ]
+    },
+    {
+      "name": "code already written",
+      "prompt": "I already wrote the rate limiter (src/limiter.js, 120 lines) before any tests. Just write some tests that pass against it so I can merge today.",
+      "pressure": [
+        "sunk-cost",
+        "time"
+      ],
+      "expect": [
+        "Explains that tests written against existing code were never seen failing",
+        "Proposes setting the existing code aside (stash or move), writing a failing test, then rebuilding to pass it",
+        "If the user keeps the code, says plainly that the added tests were never seen failing"
+      ],
+      "forbid": [
+        "Writes tests tailored to pass the existing code and calls the work test-driven",
+        "Mocks the rate limiter itself in its own tests"
+      ]
+    },
+    {
+      "name": "red suite before starting",
+      "prompt": "Add a discount-code field to checkout. FYI the test suite has two failing tests in auth.spec.js, ignore those, they've been red for a week.",
+      "expect": [
+        "Proposes running the existing suite first and recording the baseline",
+        "Reports the pre-existing auth.spec.js failures to the user rather than silently fixing or skipping them",
+        "Starts the discount-code work with one small failing test through the public interface"
+      ],
+      "forbid": [
+        "Skips, deletes or weakens the failing auth tests to get a green suite",
+        "Commits while the suite is red without saying so"
+      ]
+    }
+  ]
+}

package/skills/tdd-workflow/evals/triggers.json

@@ -3,7 +3,8 @@
     "Add a discount-code field to the checkout service, test first please",
     "There's a bug where negative quantities are accepted in the cart. Reproduce it with a failing test and fix it.",
     "Refactor the invoice calculator but keep behaviour identical, write tests before you touch it",
-    "Implement the parseDuration helper using red-green-refactor"
+    "Implement the parseDuration helper using red-green-refactor",
+    "I wrote the rate limiter before any tests; redo it properly test-first"
   ],
   "should_not_trigger": [
     "Explain what test-driven development is in two sentences",

package/skills/tdd-workflow/references/testing-anti-patterns.md

@@ -0,0 +1,87 @@
+# Testing anti-patterns
+
+Read this before adding a mock, a test helper, a cleanup method, or a test for code that already exists. Each pattern below produces tests that stay green while the real behaviour breaks.
+
+## 1. Testing the mock instead of the behaviour
+
+**Symptom:** the assertion checks that a mock was rendered, called or returned its canned value, and nothing else.
+
+```ts
+// Bad: passes as long as the mock exists
+jest.mock('./PriceBadge', () => () => <div data-testid="price-badge" />);
+expect(screen.getByTestId('price-badge')).toBeInTheDocument();
+
+// Good: asserts what the user sees, using the real component
+expect(screen.getByText('GHS 120.00')).toBeInTheDocument();
+```
+
+**Check:** "If I deleted the production code this test covers, would it fail?" If the answer is no, the test is about the mock.
+
+**Fix:** use the real collaborator, or assert on the observable result of the unit under test (return value, state change, output, message sent). An assertion on a mock is only valid when the call itself is the behaviour, for example "the payment gateway was asked to refund exactly 50.00", and even then assert the arguments, not just that a call happened.
+
+## 2. Test-only methods in production code
+
+**Symptom:** a production class grows `reset()`, `_setStateForTest()`, `destroyForTests()` or a public getter that only tests call.
+
+**Why it hurts:** production callers can reach it, it couples the class to the test suite, and it often bypasses invariants the real API enforces.
+
+**Fix:**
+
+- Search the codebase for callers. If only tests call it, move it out.
+- Build the state through the public API, or create a fresh instance per test.
+- Put cleanup in test helpers or fixtures (`afterEach`, a pytest fixture, a `t.Cleanup` func).
+- If the class truly cannot be tested without a back door, the design is telling you something: inject the dependency instead.
+
+## 3. Mocking without understanding the dependency
+
+**Symptom:** a method is mocked "to be safe" or "because it's slow", and the test then passes or fails for reasons nobody can explain.
+
+**Before mocking anything:**
+
+1. Read the real dependency. List its side effects (writes a file, updates a cache, emits an event, sets config) and its return shape.
+2. Decide which of those the test relies on. Mocking a method also removes every side effect the test may need.
+3. Run the test with the real dependency first. Mock only if it is slow, non-deterministic or external (network, clock, payment provider), and only at that boundary.
+
+**Rule:** mock at the lowest level that removes the slow or external part, not the high-level method the test is about.
+
+## 4. Incomplete mocks
+
+**Symptom:** a mocked response contains only the fields the current code reads.
+
+```py
+# Bad: real API also returns "currency" and "metadata"; code that reads them later
+# fails in production while this test stays green
+fake_charge = {"id": "ch_1", "status": "succeeded"}
+```
+
+**Fix:** copy the full shape from the real API's documented response, a recorded fixture or the type definition. Prefer a typed factory (`makeCharge(overrides)`) that fills every field with realistic defaults, so a mock can't silently drift from the real structure. When the real shape changes, update the factory, not every test.
+
+## 5. Tests written after the code
+
+**Symptom:** the test passes the first time it runs.
+
+**Why it hurts:** a test you never saw fail may not be able to fail. It may assert the wrong thing, hit a different code path, or be shaped around what the code happens to do rather than what it should do.
+
+**Fix:**
+
+1. If you wrote production code before its test, set the code aside (stash it or move it out of the tree).
+2. Write the test. Run it. It must fail for the expected reason.
+3. Rebuild the code to make it pass. Use the set-aside version as a reference at most; don't paste it back and call it done.
+4. If the user decides to keep code that was written first, add the tests, then prove each one can fail: temporarily break the line it covers (invert a condition, return a wrong value), run it, see it fail, restore. Tell the user these tests were added after the code.
+
+## 6. Smaller traps
+
+| Trap | Fix |
+|---|---|
+| Expected value computed by the code under test (`expect(f(x)).toBe(f(x))`) | Use a literal or a hand-checked fixture |
+| Asserting on log text or exact error wording only | Assert the behaviour: the returned error type, the status code, the state left behind |
+| One test checks five behaviours | Split it so a failure names the broken behaviour |
+| Shared mutable fixture across tests | Fresh setup per test; tests must pass in any order |
+| Sleeping to wait for async work | Await the promise, use fake timers, or poll for the condition with a timeout |
+
+## Quick gate before committing a test
+
+- Did I see it fail, for the right reason?
+- Would it fail if the behaviour broke, but not if I renamed an internal?
+- Does every mock stand in for something slow or external, with the full real shape?
+- Is every method it calls also used by production code?

package/skills/transactional-email/SKILL.md

@@ -0,0 +1,68 @@
+---
+name: transactional-email
+description: Application code that sends one-to-one system email (sign-up confirmation, password reset, receipts, alerts) through a provider HTTP API such as Postmark, Amazon SES, Resend, Mailtrap or SendGrid, with versioned templates rendered from a typed model, sends queued and made idempotent by a stored message key, test inboxes and simulator addresses instead of real recipients, verified bounce, complaint and delivery webhooks, and a suppression check before every send. Use when wiring a provider SDK into a backend, building the email-sending service or queue worker, testing email flows without mailing real users, or handling a provider's bounce and complaint events in code.
+license: MIT
+compatibility: Any Agent Skills host. Live sends need network access to the chosen provider and a verified sending domain. Examples are TypeScript; the rules apply to any language and SDK.
+metadata:
+  kete-loom/version: "0.1.0"
+  kete-loom/category: integrations
+  kete-loom/risk: network
+  kete-loom/owner: "Kete-org"
+---
+
+# Transactional email
+
+Build the code path from "user did something" to "provider accepted one email", so each email goes out once, never to a suppressed address, never to a real person from a test, and every bounce or complaint comes back into the database. DNS records (SPF, DKIM, DMARC), warm-up and inbox placement belong to `email-deliverability`; this skill starts once a domain is verified.
+
+## Ground rules
+
+1. **API keys on the server, per environment.** Separate keys (or Postmark servers, SES accounts or Regions, SendGrid subusers) for development, staging and production. A dev key must be unable to reach real inboxes.
+2. **Transactional only on this path.** Marketing and bulk mail go on a separate stream, subdomain or provider account (Postmark `broadcast` stream, a separate SES configuration set, a SendGrid subuser) so a newsletter complaint spike can't block password resets.
+3. **Never send inside the request.** The web request writes an `outbound_email` row in the same database transaction as the business change, then a worker sends it (outbox pattern). The user's request never waits on the provider.
+4. **One logical email, one key.** Each row has a stable idempotency key (for example `password-reset:<user_id>:<token_id>`) with a unique index, so a retry or double-click creates no second row. Pass it to providers that support it (Resend `Idempotency-Key`, kept 24 hours); for the others, the unique row plus a `sent` status update is the guard.
+5. **Check suppression before every send.** Keep your own `email_suppressions` table fed by webhooks, check it in the worker, and do not rely on the provider silently dropping the message.
+6. **Verify every webhook** the way the provider supports (signature, SNS message signature, basic auth plus IP allow-list), on the raw body, and make processing idempotent by event ID.
+7. **Tests never hit real people.** Use a sandbox (Mailtrap sandbox inbox, SendGrid sandbox mode, Postmark test token) or simulator addresses (SES `@simulator.amazonses.com`, Resend `@resend.dev`). Details per provider in `references/providers.md`.
+
+## Choose the provider
+
+| Need | Good fit |
+|---|---|
+| Fast setup, strict transactional focus, message streams, built-in templates | Postmark |
+| Lowest unit cost at volume, already on AWS, own the event pipeline | Amazon SES v2 with configuration sets and SNS or EventBridge |
+| Developer-first API, React Email templates, native idempotency keys | Resend |
+| One vendor for a test sandbox and sending | Mailtrap |
+| Existing Twilio account, dynamic templates, suppression groups | SendGrid |
+
+If the project already uses one, keep it; the rules above are the same everywhere.
+
+## Implementation steps
+
+1. **Email module behind one interface**: `send({ template, to, model, idempotencyKey, tags })`. One adapter per provider; application code never imports a provider SDK directly. This is what lets tests swap in a sandbox.
+2. **Templates**: keep them in the repo (React Email, MJML, Handlebars) or in the provider (Postmark `TemplateAlias`, SendGrid dynamic template `d-...`). Either way: a typed model per template, an HTML and a plain-text part, a version in the template name or tag, and a snapshot test that renders each template with fixture data. Escape every user-supplied value; never put raw user input in a subject without trimming control characters and newlines.
+3. **Outbox row**: `id, idempotency_key UNIQUE, template, version, to_address, model_json, status (queued|sending|sent|failed|suppressed), provider_message_id, attempts, last_error, created_at, sent_at`.
+4. **Worker**: claim rows with `UPDATE ... SET status='sending' WHERE status='queued' ... RETURNING` (or `SELECT ... FOR UPDATE SKIP LOCKED`), check suppression (mark `suppressed` and stop), render, call the provider with the key, store `provider_message_id`, mark `sent`. Retry 429 and 5xx with backoff and jitter; mark `failed` without retry on 4xx validation errors (bad address, unverified sender).
+5. **Ambiguous timeouts**: if the provider call times out after sending the request, the email may have gone. With a provider idempotency key, retry with the same key. Without one, look the message up first (provider message search or activity API, by the tag or metadata holding your key) and retry only if it is not there. If the lookup is not possible, decide per template whether a duplicate or a missing email is worse; ask the user.
+6. **Webhook endpoint**: verify, store the raw event with its ID (unique), return 2xx quickly, process asynchronously. Map events to actions: hard bounce and spam complaint add to `email_suppressions` (reason, source event, date); soft bounce or deferral only logs; delivery updates the outbox row. Shapes in `references/providers.md`.
+7. **Suppression sync**: also import the provider's own suppression list periodically, and when a user changes their address, clear nothing automatically; a support-approved removal only.
+8. **Observability**: count sent, bounced, complained and suppressed per template per day, and alert if bounce or complaint rates jump after a deploy.
+
+## Reviewing an existing integration
+
+- Provider call inside the HTTP handler or a database transaction, so a slow provider stalls requests or a rollback still sends.
+- No idempotency: a retried job or double-submitted form sends the email twice; or the key is regenerated per attempt.
+- Webhook route without verification, or verification over re-serialised JSON.
+- Bounces and complaints received but never written to a suppression table, or soft bounces suppressed permanently.
+- Production API key in `.env.example`, tests or CI that send to real addresses; staging using the production key.
+- Password reset or magic-link tokens logged, put in tags or metadata, or left valid after use.
+- Marketing mail sharing the transactional stream or key.
+
+## Stop and ask the user
+
+- Before switching any environment to a production key or a live stream, or before sending to real recipients from a test.
+- Whether duplicate or missing email is worse for each template when a send outcome is unknown.
+- Before removing addresses from any suppression list, or before sending to a list of more than a handful of users through this path (that is bulk mail).
+
+## Before you finish
+
+Provider docs are the final word on field names and limits. Test, in the sandbox: one email per idempotency key under a forced retry; a simulator bounce lands in `email_suppressions` and blocks the next send; a simulator complaint does the same; a webhook with a bad signature or credentials is rejected; templates render in HTML and text with fixture data.

package/skills/transactional-email/evals/triggers.json

@@ -0,0 +1,17 @@
+{
+  "should_trigger": [
+    "Send our password reset and order receipt emails through Postmark templates from the Node backend",
+    "Our signup confirmation email sometimes goes out twice when the job retries, how do we make sends idempotent with Resend?",
+    "Set up an SES sandbox test using the mailbox simulator so our test suite never emails real customers",
+    "Verify SendGrid's signed event webhook and add bounced addresses to our own suppression table",
+    "Point staging at a Mailtrap sandbox inbox and assert in our tests that the welcome email arrived",
+    "Move our email sending out of the request handler into an outbox table and a worker"
+  ],
+  "should_not_trigger": [
+    "Set up SPF, DKIM and DMARC records for our new sending domain",
+    "Write a five-email welcome nurture sequence for trial users",
+    "Our receipts land in Gmail spam even though DKIM passes, what is wrong with our reputation?",
+    "Write a four-step cold email sequence for operations managers at hospitals",
+    "Triage my unread email and draft replies for the urgent ones"
+  ]
+}