mixdog 1.4.0 → 1.4.2

package/src/defaults/skills/skill-creator/references/evaluation.md

@@ -1,111 +1,52 @@
 # Evaluating a skill
 
-Use this reference after a draft exists or when a current skill has a reported
-defect.
+Use after a draft exists or when a skill misbehaves. A passing validator
+proves structure only, never behaviour.
 
-## Separate the questions
+## Scenarios
 
-A complete evaluation asks:
+Two to five realistic requests, with the context a real user would give:
 
-| Area | Question |
-|---|---|
-| Selection | Does the description attract relevant requests and reject close alternatives? |
-| Execution | Does the loaded workflow produce the required outcome? |
-| Integration | Do resources, paths, tools, and feature gates resolve? |
-| Recovery | Do failures remain bounded and actionable? |
-| Efficiency | Does the skill remove repeated work without adding ritual? |
-
-Passing a frontmatter check proves none of the execution questions.
-
-## Scenario set
-
-Use two to five realistic requests:
-
-1. a normal request that names the intent;
+1. a request that names the intent;
 2. an implicit request that should still select the skill;
-3. an important edge or failure condition;
-4. a close negative request owned elsewhere;
-5. a subjective artifact case when human judgement matters.
+3. an edge or failure condition that matters;
+4. a near-miss negative: shares the skill's words but belongs to another
+   owner;
+5. a subjective artifact, when taste or domain judgement decides.
 
-Do not use “load this skill” as a positive test. Include concrete context, files,
-or constraints that a real user would provide.
-
-An optional record shape:
-
-```json
-{
-  "id": "short-purpose",
-  "prompt": "Realistic user request",
-  "expected_owner": "skill-name or none",
-  "observable_result": "What must become true",
-  "inputs": []
-}
-```
-
-Keep evaluation artifacts outside the distributable directory unless the
-packaging process explicitly excludes them.
+Never use "load this skill" as a positive case. Keep scenario files outside
+the skill's folder.
 
 ## Baseline
 
-Before improving a skill, preserve its exact prior behavior through existing
-version history or a read-only evaluation snapshot. Compare identical prompts,
-inputs, model class, tool availability, and environment.
-
-For a new skill, compare against no skill only when the runner can isolate the
-draft from the baseline. Otherwise report a smoke test rather than implying a
-controlled A/B result.
-
-The evaluation harness must remain uniform. Do not add task-specific routing,
-timeouts, prompt clues, retries, or scoring exceptions to make one version win.
+Before changing a skill, keep a copy of the prior version and run the same
+prompts, inputs, model and tools against both. A new skill with no runnable
+baseline gets a smoke test, not a claimed comparison. Keep the harness the
+same for both: no task-specific hints, retries, timeouts or scoring
+exceptions.
 
 ## Evidence
 
-Use deterministic checks for mechanical outcomes:
-
-- required files exist and open;
-- structured fields and values are valid;
-- references resolve;
-- malformed input fails with useful guidance;
-- resource and retry limits hold;
-- an irreversible action waits for approval;
-- prohibited side effects do not occur.
-
-Do not test exact prose, source strings, internal algorithms, or snapshots whose
-expected behavior has intentionally changed.
-
-Use user review for clarity, visual quality, writing taste, and domain
-judgement. Present the full-size persisted output rather than a description of
-it.
+- Mechanical outcomes get deterministic checks: files exist and open, fields
+  are valid, references resolve, bad input fails with useful guidance, an
+  irreversible step waits for approval, no forbidden side effect happens.
+- Never assert exact prose or internal steps.
+- Clarity, taste and domain judgement go to the user, shown the real output.
 
 ## Diagnose before revising
 
-Classify a failure:
-
-- **selection** — description signal or neighboring boundary;
-- **instruction** — missing decision, sequence, or completion evidence;
-- **resource** — broken script, reference, asset, or path;
-- **integration** — unavailable tool, feature, dependency, or platform;
-- **harness** — invalid comparison or test infrastructure;
-- **underlying capability** — the tool or product cannot deliver the outcome.
-
-Edit the component that owns the failure. More prompt text does not repair a
-broken tool, and body changes cannot fix a description that never selects the
-skill.
+| Class | Where the fix goes |
+|---|---|
+| Selection | Trigger or the neighbour's boundary |
+| Instruction | A missing decision, step order or completion result in the body |
+| Resource | The broken script, reference, asset or path |
+| Integration | The unavailable tool, feature or dependency |
+| Harness | The test setup, not the skill |
+| Capability | The tool or product; the skill cannot deliver it |
 
 ## Improve without sediment
 
-After each revision:
-
-1. rerun affected scenarios;
-2. validate structure and resource paths;
-3. retain safety, recovery, and bounded execution;
-4. remove rules superseded by the fix;
-5. check whether the change generalizes beyond the failing example.
-
-Look for tests that always pass, high variance, repeated helper work, excessive
-token or time cost, and rules that change no outcome. Those are signals to
-improve the evaluation or simplify the skill.
-
-Stop when required behavior passes, subjective review is accepted, and further
-edits no longer produce meaningful gains.
-
+After each revision, rerun the affected scenarios, validate, keep safety and
+recovery, delete the rules the fix superseded, and check that the fix is not
+tailored to one example. Stop when the required behaviour passes, the user
+accepts the subjective parts, and further edits stop paying off.

package/src/defaults/skills/test-suite/SKILL.md

@@ -0,0 +1,194 @@
+---
+name: test-suite
+description: Write, run, diagnose, prune and audit a project's tests without overloading the machine.
+when_to_use: 'Write, fix, run, diagnose, prune or audit tests, flaky failures, suite health; not live UI checks.'
+dependencies:
+  tools:
+    - type: tool
+      value: read
+    - type: tool
+      value: shell
+---
+
+# Test suite
+
+Owns how tests in any project are written, run during a change, diagnosed when
+they fail intermittently, pruned and audited. The project's own
+commands, lanes and conventions win over every default here; this skill
+supplies the judgement and procedure around them. Driving a running app or
+device by hand is outside it.
+
+## Pick the mode
+
+| Request | Mode |
+|---|---|
+| Add a test, fix a bug test-first | Write |
+| Check a change while editing | Run |
+| A test fails sometimes, only in a full run, only in CI, only on this machine | Diagnose |
+| Duplicate, dead, obsolete or worthless tests | Prune |
+| Review the suite's health or a whole area | Audit |
+
+## Prerequisites: find the project's own way
+
+Before the first run, look for the project's focused test command in this
+order, and stop at the first source that gives it plus any constraint that
+affects the change (lane, setup, needed service): `AGENTS.md` or `CLAUDE.md`,
+`docs/testing*`, `CONTRIBUTING*`, package scripts (`package.json`, `Makefile`,
+`pyproject.toml`, `Cargo.toml`), then CI workflows. Do not survey every
+manifest or CI lane. Result: one focused command, copied from the project,
+never invented; a broader command only when the change is broad. When nothing
+is documented, fall back to the
+runner's own selection: `node --test <file>`, `vitest run <file>`,
+`jest <file>`, `pytest <file>::<test>`, `go test -run <Name> ./<pkg>`,
+`cargo test <name>`.
+
+## Machine rules
+
+Tests run on the user's working machine. These hold in every mode.
+
+- The Verification section of your system rules owns the evidence budget,
+  reruns, fix attempts, heavy runs and stopping what you started; follow it.
+- Never loop a test to "reproduce" flakiness here, and never shard in
+  parallel locally; repetition belongs to CI history.
+- Confirm by PID or command line that detached processes you started are gone.
+
+## Write
+
+What a change gets:
+
+| Change | New tests |
+|---|---|
+| Bug fix | One regression test for the reported behaviour |
+| New behaviour | One per observable behaviour the requirement names (its success and its stated rejections) |
+| Refactor, rename, move | None; the existing tests must still pass unchanged |
+| Removal | None; delete the removed behaviour's tests, never assert the absence |
+| Docs, copy, style, config values | None |
+
+Only a test owning a lasting contract joins the suite and its CI lane.
+Reproductions, probes and one-off checks are scratch: they live in a temp
+directory outside the repository, are never committed, and are deleted when
+the task ends. A scratch reproduction that proves a bug becomes the
+regression test above or is deleted, never both kept.
+
+1. Name the behaviour and its owner: the one contract this test owns, beside
+   the code it exercises. If a test already owns it, extend that test
+   (`references/suite-design.md`). Result: one file and test name chosen.
+2. Name the production change that would make the test fail; if you cannot,
+   redesign it around an observable behaviour (`references/suite-design.md`).
+   Write the test through the user's interface (caller, CLI,
+   HTTP, rendered output) with the expected value from the requirement, not
+   from running the code. For a new regression test, show it failing first:
+   the failure names the missing behaviour. An existing failing test already
+   counts as that evidence.
+3. Apply `references/reliability.md` for any time, process, port, file,
+   environment, locale or platform dependency. Result: no fixed sleep, no
+   fixed port, no unrestored global, no host-locale assumption.
+4. Make it pass with the smallest change, then rerun the same file. Result:
+   a new regression test's fail-before and pass-after pair goes into the
+   report. Add no further
+   tests unless the user asks.
+5. Put it in the lane its duration and needs call for (`suite-design.md`).
+
+## Run
+
+1. Run the covering file(s) only. Result: the selected count matches the files
+   you meant.
+2. On failure read the full log before editing; do not rerun to see if it
+   passes.
+3. Run a broader lane once after the last edit only if the change touches a
+   shared contract or an integration point; otherwise stop at the covering
+   file. Reuse an earlier pass only when no later edit touched the code it
+   checks.
+4. After two evidence-backed fix attempts, report what was tried and the log.
+
+## Diagnose
+
+An intermittent failure is a defect in the test or the product, never noise
+(`references/flaky-policy.md`).
+
+1. Collect evidence without reruns: the failing assertion, the full log, what
+   else was running, the OS and locale. Result: the failing line and the
+   condition it waited on.
+2. Classify the cause with the table in `references/reliability.md`. Result:
+   one named cause with the line that assumes it.
+3. Propose the fix: in the test, or in the product when real users can hit
+   the same race. A longer timeout, a retry or a skip is not a fix.
+
+Diagnose ends at a report: cause, evidence, proposed fix. Only when the user
+asked for a fix, apply it and prove it with the focused file plus a
+deterministic trigger of the cause (a held promise, a fake clock, a localized
+fixture), not by looping. It never becomes an open-ended fix loop. Never
+skip or disable a failing test; when the fix cannot land now, report it and
+leave that decision to the user. Only a capability skip (a platform or tool
+the test needs is absent) is allowed, and its reason names that capability.
+
+## Prune
+
+First tell redundant coverage from removing a supported behaviour, and keep
+distinct observable success, rejection, error and compatibility cases.
+Removing a behaviour with its tests needs evidence independent of those tests
+(current requirements, real callers, a specification); a test existing only
+for a branch, or a fallback and test justifying each other, is no evidence,
+and unresolved intent blocks deletion.
+
+- Delete a test only when it is dominated (same branch and result as a
+  stronger test), tautological (cannot fail), a permutation crossing no new
+  branch, or verification theater (verifier and producer share one failure
+  domain). Name the test that still owns the contract.
+- Deleting a contract test needs the user's approval.
+- Same contract owned twice → merge the unique cases into the owner at the
+  user's interface.
+- Asserts that something was removed, restates a constant, copies production
+  logic or source text into the test, or pins internals no user observes →
+  delete it, or rewrite it at the interface when a real contract is behind it.
+- Passes only alone → Diagnose; never mark it serial to hide it.
+
+Prune as you go: a change that touches a test file applies these rules to that
+file only. A whole-suite sweep is an Audit the user asked for.
+
+Result: each removal names the owning test that still covers the contract, or
+states that the contract is gone.
+
+## Audit
+
+1. Run `node ${MIXDOG_SKILL_DIR}/scripts/inventory.mjs <project-root>`
+   (`--json` for data). Result: per-area counts, lanes, languages, risk
+   markers, and defect skips for JS/TS, Python, Go and Rust.
+2. Read `references/case-management.md` and rank findings by impact: defect
+   skips, risky markers in the fast lane, duplicate owners.
+3. Report one table (area, finding, count, proposed action) and wait for the
+   user's choice of batches. Audit ends at this report; fix only the batches
+   the user picks, one or two at a time, each verified by its covering files.
+
+## Pitfalls
+
+- Parsing localized tool output (a console in another language, a busy host
+  dropping rows) passed on the author's machine and failed elsewhere. Query a
+  structured source.
+- Fixed sleeps passed alone and failed in bulk when other work ran.
+- A load generator or server started by an agent was detached, outlived the
+  agent and froze the machine. Nothing detached survives a report.
+- Render tests passed while the real screen hid a row: assert the rendered
+  output, not the inputs.
+- Full lanes that take many minutes fail under load; selection, not more
+  retries, fixes that.
+
+## Verification
+
+- Write/fix: the focused file passes; a new regression test also failed
+  before, and the report quotes both.
+- Diagnose: the named cause and its evidence. Only when the user asked for a
+  fix: the deterministic trigger and its passing run.
+- Every mode: no process you started is still running.
+
+## Resources
+
+- `references/reliability.md` — read when a test touches time, processes,
+  ports, files, environment, locale or the platform, or when diagnosing.
+- `references/flaky-policy.md` — read when a failure is intermittent.
+- `references/suite-design.md` — read when writing a test, choosing its lane
+  or file, or reviewing how a suite is organized.
+- `references/case-management.md` — read for Audit, or when adding a new
+  area.
+- `scripts/inventory.mjs` — run for Audit; read-only, bounded, Markdown or
+  `--json`.

package/src/defaults/skills/test-suite/references/case-management.md

@@ -0,0 +1,51 @@
+# Test case management
+
+Every test has an owner, a single contract and a lane. None of this belongs in a hand-written registry: hand lists
+drift from the code. It is derived from where the file lives, its name and its
+text, and `scripts/inventory.mjs` reports it.
+
+## Where a test lives
+
+- Beside the code it exercises, in the same area directory; the inventory's
+  area is the path below the source root, two levels deep.
+- One domain per file (`suite-design.md`).
+- Cross-area contract suites live in their own directory and still run in CI.
+
+## What the inventory reads
+
+| Ecosystem | Test files found by | Skip records |
+|---|---|---|
+| JS/TS (`node:test`, Vitest, Jest) | `*.test.*`, `*.spec.*`, `*-test.mjs`, `__tests__/` | `skip:` option, `it.skip`, `xit`, `skipIf` |
+| Python (pytest, unittest) | `test_*.py`, `*_test.py` | `pytest.mark.skip/skipif/xfail`, `unittest.skip*`, `pytest.skip` |
+| Go | `*_test.go` | `t.Skip`, `t.Skipf`, `t.SkipNow` |
+| Rust | `tests/` directories, files with `#[test]` or `#[cfg(test)]` | `#[ignore]`, `#[ignore = "…"]`, `cfg_attr(…, ignore)` |
+
+Lane comes from a `.slow.`, `.live.`, `.e2e.`, `.integration.` or `.electron.`
+name tag (JS/TS); everything else counts as fast. Vendored, build and cache
+directories are skipped and links are not followed.
+
+Skip kinds: `capability` (the reason names a missing platform, tool or
+privilege; a conditional skip with no reason counts here), `tracked` (`#N`),
+`untracked` (a defect-looking skip with no issue) and `expired` (`until
+YYYY-MM-DD` in the past). Every kind but `capability` is a defect skip that
+needs a fix; an issue number does not make it acceptable.
+
+## Risk markers
+
+| Marker | Meaning | Expected action |
+|---|---|---|
+| `timer` | Sleep, timeout or ticker in a test | Replace with a signal or fake clock unless time is the subject |
+| `spawn` | Real child process | Keep only when the process is the subject; await exit |
+| `port` | Listener on a fixed port | Bind port 0 |
+| `clock` | Real clock read without a fake clock | Inject or fake the clock when the verdict depends on it |
+| `source-text` | Reads source and matches its text | Rewrite at a runnable interface, or justify in the test |
+
+A fake-clock library in the file clears `timer` and `clock`. Markers are
+leads, not verdicts: a timer in a test about timers is correct.
+
+## Audit report
+
+One table per project: area, files, fast/slow counts, marker counts, defect
+skips. Rank batches by: defect skips, fast-lane
+files with `timer`+`spawn`, then the rest by area. The audit ends at the
+report; fix only the batches the user picks.

package/src/defaults/skills/test-suite/references/flaky-policy.md

@@ -0,0 +1,39 @@
+# Flaky-test policy
+
+## Rules
+
+1. **Zero flakes.** A test that passes and fails on the same code is a defect.
+   CI never retries automatically, and a passing rerun clears nothing.
+2. **Fix, never skip.** Fix the cause, or move a correctly slow test to the
+   slow lane. A failure that cannot be fixed now is reported; skipping it is
+   the user's decision, never the agent's.
+3. **Capability skips only.** A skip states the capability the platform or
+   machine lacks (a tool, an OS feature), never a defect.
+4. **Evidence over memory.** Decide from the failing log and CI history, not
+   from "it is always flaky". Keep both the failed and the passing log.
+5. **No local stress.** Reproduction happens through a deterministic trigger
+   in the test, or in CI history; never by loading or looping on the user's
+   machine.
+
+## A flake report
+
+When a flake is handed off, one tracked issue carries: the test and file, the
+failing assertion and log, the platform and load conditions, when it first
+appeared, how often in CI history, and the suspected cause with its evidence.
+One issue per test, so ownership stays clear.
+
+## Priority
+
+Fix in this order:
+
+1. Tests that block a release or a push.
+2. Tests failing most often across recent CI runs.
+3. Tests whose cause can also hit users (product races, localized output).
+4. Everything else, batched by area.
+
+## What does not count as a fix
+
+- Raising a timeout to absorb a sleep, poll or production interval.
+- Adding retries, a rerun-failed CI step, or a serial flag to hide contention.
+- Skipping on a platform that has the capability.
+- Deleting the assertion that failed.

package/src/defaults/skills/test-suite/references/reliability.md

@@ -0,0 +1,49 @@
+# Reliable tests
+
+A test must give the same verdict alone, beside other suites, on a busy
+machine, on a localized OS and in CI. Process isolation does not isolate
+ports, paths, child processes, the clock or the host's language.
+
+## Cause table
+
+| Cause | Signal | Fix |
+|---|---|---|
+| Timing | Passes alone, fails when other work runs; a sleep or a fixed budget decides the verdict | Wait on the event, promise or state the code produces; fake the clock when time is the subject |
+| Shared resource | Fixed port, fixed temp path, shared data dir, a leftover process | Bind port 0 and read the assigned port; a fresh temp dir per test; per-test data dirs |
+| Unrestored global | Later tests change behaviour; order matters | Restore environment variables, working directory, timers, mocks and locale in a `finally`/fixture teardown right after the change |
+| Platform / locale | Fails on another OS, language or console code page | Query structured sources; never parse localized tool text; remember Windows environment names are case-insensitive |
+| Order | Depends on a previous test's state | Each test creates and removes its own prerequisites |
+| Teardown | Next test sees a live child, open handle or locked file | Await child exit, server close and handle release before the test ends |
+
+## Time
+
+- A timeout only bounds a wait on a real signal; it never makes the assertion
+  true. Keep the outer wait far larger than any deadline under test.
+- When the deadline itself is the subject, inject or fake the clock and
+  restore real time in teardown.
+- Do not assert scheduler ordering unless the order is the product contract;
+  place races with deferred promises at a chosen point instead.
+
+## Processes and ports
+
+- Count spawns: a process start costs tens of milliseconds on Windows and much
+  more under load. Spawn once per file when the behaviour allows it.
+- A test that starts a process owns its whole tree: kill it and await exit,
+  detached children included, before the test ends.
+- Prefer an in-process fake over a real server unless the real process is the
+  subject.
+
+## Files and environment
+
+- Use a private temp root per test and remove it afterwards; never write into
+  the user's home configuration or the repository tree.
+- Windows releases handles asynchronously: a rename or delete right after a
+  close may need a bounded retry sized to the observed contention.
+- Point every home, data or cache directory variable the code reads at the
+  test's own root.
+
+## Proof
+
+A reliability fix is proven by a deterministic trigger of the named cause
+(held promise, fake clock, localized fixture, pre-bound port) that fails before
+and passes after, not by repeating the test until it stays green.

package/src/defaults/skills/test-suite/references/suite-design.md

@@ -0,0 +1,78 @@
+# Suite design
+
+How a suite stays small, fast and honest. Apply when writing, placing or
+reviewing tests.
+
+## A good test
+
+- **Names the break.** Before writing, state the production change that should
+  make it fail, and whether that change is a bug or a decision. Cannot name
+  one → redesign around an observable behaviour.
+- **Independent expected value.** A literal or hand-checked fixture, never a
+  value recomputed with the code under test or its helpers.
+- **No change detectors.** If only intentional decisions can fail it (a
+  constant, exact message wording, private structure), test the behaviour that
+  depends on the decision instead.
+- **Your code, not the framework's.** Test the contract at your boundaries;
+  trivial getters and forwarding earn tests only when they validate,
+  normalize, derive or cause side effects.
+- **Through the interface.** Mock only at system boundaries (network, clock,
+  randomness, slow or external stores), never your own modules; pass
+  dependencies in so a boundary is easy to replace.
+- **One behaviour per test**, small enough that a failure points at one cause.
+  Write one test at a time, as needed; never bulk-generate cases.
+
+## One invariant, one owning test
+
+Two tests asserting the same contract drift apart instead of reinforcing each
+other: when the contract changes, one is updated and the other keeps asserting
+a retired behaviour. Name an owner per contract; every other test covers a
+distinct observable boundary. When an overlap appears, reduce both to what is
+distinct, or keep the owner and move the unique cases into it.
+
+## Text matching is a last resort
+
+Asserting against source text pins names, spacing and constants instead of
+behaviour. It breaks on harmless refactors and passes on wrong behaviour, so it
+proves neither direction. Reserve it for artifacts with no runnable surface
+(generated scripts, workflow YAML, prompt text) or for ordering invariants in
+code a test cannot import (a guard must precede a send), and say why in the
+test. Keep the list of such files short and known.
+
+## A contract suite must run in CI
+
+A suite that asserts a contract but runs nowhere automatic is documentation
+with an expiry date: nothing forces it to follow the code, and it fails the
+first time someone runs it. Make it discoverable like every other test and give
+it a CI job. Checks that need a built artifact run after the build step, not
+nowhere. Scripts that are not test files (smoke scripts, harness runners) get a
+scheduled sweep that enumerates them, so new ones join automatically and stale
+exclusions fail closed.
+
+## One domain per file
+
+A multi-domain test file makes every change pay a whole-file search. Split when
+one change forces a search across domains, not on line count alone; shared
+fixtures move into a helper beside the files. One stateful fixture chain over
+one domain stays one file whatever its length.
+
+## Slow work gets its own lane
+
+A test over roughly ten seconds alone goes to a slow lane marked by name,
+directory or tag, whichever the runner selects on. It still runs in CI, in a
+parallel job, so the gate takes the longer of the two instead of their sum.
+Tests that need a built artifact or a live system get a live lane. If a file
+is slow only in a full run because processes contend, the cause is contention,
+not a slow file: Diagnose, do not move it.
+
+## Selection beats retries
+
+During a change run the files that reach the changed code (the project's
+changed-files selection, a path or name filter, a single test id). The full
+lanes gate the push. A suite that takes minutes is a design problem to fix by
+selection and sharding in CI, not by waiting on the user's machine.
+
+## Published artifacts carry no tests
+
+Check that packaging excludes test sources, and that test-only changes do not
+invalidate caches keyed on the shipped files.