coding-os 0.3.2__py3-none-any.whl

core/skills/testing-strategy/SKILL.md

@@ -0,0 +1,266 @@
+---
+name: testing-strategy
+tier: quality
+domain: [universal]
+description: Choose the right test type for every change — unit, integration, contract, end-to-end, property-based, mutation, fuzz. Use when adding tests to a new feature, deciding what to test for a bug fix, designing a test pyramid for a service, evaluating coverage targets, or untangling a slow test suite. Stack-agnostic; concrete recipes target Python (pytest), TypeScript (Vitest/Jest/Playwright), and Go (table tests + testify). Pairs with clean-code (error-path tests) and observability (CI signal hygiene).
+last_reviewed: "2026-05-11"
+---
+
+# Testing Strategy — Pick the Right Layer
+
+A practical playbook for designing test suites that catch real bugs without the suite itself becoming the bug. Stack-agnostic core; concrete recipes target this project's languages.
+
+## When to Use This Skill
+
+- Adding the first test to a new module — the shape you ship now sets the suite's trajectory.
+- Reviewing a PR where the test ratio looks off (e.g. all unit, no contract; all e2e, no unit).
+- Investigating a flaky / slow suite — the cure is almost never "add retries".
+- Deciding coverage targets for a service (90% lines is the wrong target).
+- Adding tests to a bug fix — the fix without a regression test will return.
+- Migrating from manual QA → automated CI.
+
+Skip when: writing throwaway exploration code. Real code, real tests.
+
+## The Test Pyramid (still right, refined for 2026)
+
+```
+                ┌─────────┐
+                │   E2E   │  ~5%   slow,    flake-prone, golden-path
+            ┌───┴─────────┴───┐
+            │   Contract +    │  ~15%  pact, OpenAPI, GraphQL schema
+            │   Integration   │
+        ┌───┴─────────────────┴───┐
+        │                          │
+        │      Unit + Property     │  ~80%  fast (ms), deterministic
+        │                          │
+        └──────────────────────────┘
+```
+
+Targets are guidance, not law. A library has more unit + property than a deployment-heavy SaaS. A regulated app has more contract + integration than a hackathon prototype. **What stays constant: most tests are fast; few tests are slow; zero tests are flaky.**
+
+## The Six Layers — When Each Pays Off
+
+| Layer | Speed | Catches | Use For |
+|---|---|---|---|
+| **Unit** | <10ms | Logic bugs, edge cases | Pure functions, calculators, parsers, mappers |
+| **Property-based** | <100ms | Bugs you didn't think of | Invariants, parsers, encoders, ordering rules |
+| **Integration** | <1s | Wiring bugs | DB queries (real DB, not mock), service composition |
+| **Contract** | <100ms | Producer/consumer drift | API schemas, MCP envelope, event payloads |
+| **E2E** | 5–60s | Full-stack regressions | One golden path per critical user flow |
+| **Smoke / run-the-deliverable** | seconds | Import/entry crashes the unit suite hides (e.g. `ModuleNotFoundError` when run as a file) | Every new runnable entrypoint — `--help` / `--dry-run` / `python -c "import x"`, from the same entry the user or cron uses |
+| **Mutation / Fuzz** | minutes (offline) | Tests that don't actually test | Quarterly audit, security-critical code |
+
+### Unit tests — the workhorse
+
+A unit test executes one function in isolation. **No I/O, no clock, no network, no shared state.** Sub-10ms each so you can run 10K of them in CI.
+
+```python
+# Python — pytest
+def test_calculate_discounted_price_applies_percentage() -> None:
+    result = calculate_discounted_price(Decimal("100"), Decimal("20"))
+    assert result == Decimal("80")
+
+def test_calculate_discounted_price_rejects_negative_discount() -> None:
+    with pytest.raises(InvalidDiscountError, match="negative"):
+        calculate_discounted_price(Decimal("100"), Decimal("-5"))
+```
+
+```typescript
+// TypeScript — Vitest
+describe("calculateDiscountedPrice", () => {
+  it("applies percentage to base price", () => {
+    expect(calculateDiscountedPrice(100, 20)).toBe(80);
+  });
+
+  it("rejects negative discount", () => {
+    expect(() => calculateDiscountedPrice(100, -5))
+      .toThrow(/negative/);
+  });
+});
+```
+
+**Anti-pattern:** mocking five frameworks to test one calculation. If you need five mocks for a unit test, the function isn't a unit. Extract the pure logic.
+
+### Property-based tests — bugs you didn't think of
+
+Instead of asserting `f(2) == 4`, you assert an *invariant* (`f(x) >= 0 for any x >= 0`) and let the framework generate hundreds of inputs to try and break it.
+
+```python
+# Python — Hypothesis
+from hypothesis import given, strategies as st
+
+@given(price=st.decimals(min_value=0, max_value=10_000), pct=st.decimals(min_value=0, max_value=100))
+def test_discount_never_negative(price: Decimal, pct: Decimal) -> None:
+    result = calculate_discounted_price(price, pct)
+    assert result >= 0
+    assert result <= price
+```
+
+```typescript
+// TypeScript — fast-check
+import fc from "fast-check";
+
+test("discount never produces negative or > base", () => {
+  fc.assert(
+    fc.property(fc.float({min: 0, max: 10000}), fc.float({min: 0, max: 100}),
+      (price, pct) => {
+        const r = calculateDiscountedPrice(price, pct);
+        return r >= 0 && r <= price;
+      }
+    )
+  );
+});
+```
+
+Use for: parsers, encoders, comparison/ordering, anything with mathematical invariants, anywhere "off-by-one" is a real risk.
+
+### Integration tests — the wiring layer
+
+Real DB (testcontainers / pytest-postgresql / sqlite-in-memory for sqlite-targeted code), real HTTP, real serialization. Mock only third-party HTTP services (and use a recorded-cassette tool like `vcrpy` / `nock` so the mock is honest).
+
+**Rule:** the integration test goes through the same code path production does. If production hits PostgreSQL, the test hits PostgreSQL — not SQLite, not a mock. Mock/prod divergence is the #1 source of "tests pass, prod breaks."
+
+```python
+# Python — pytest with testcontainers
+@pytest.fixture
+def db():
+    with PostgresContainer("postgres:16") as pg:
+        engine = create_engine(pg.get_connection_url())
+        Base.metadata.create_all(engine)
+        yield engine
+
+def test_user_repository_finds_by_email(db) -> None:
+    with Session(db) as session:
+        session.add(User(email="a@b.com", name="A"))
+        session.commit()
+
+        repo = UserRepository(session)
+        user = repo.find_by_email("a@b.com")
+
+        assert user is not None
+        assert user.name == "A"
+```
+
+### Contract tests — the seam between services
+
+When team A produces and team B consumes, **what you both agree on is the contract**, not your respective unit tests. Contract tests verify the schema (OpenAPI / GraphQL / event-payload / MCP envelope) is what both sides expect.
+
+For the coding-os meta-repo specifically: every `cos_*` MCP tool MUST return the envelope from [docs/engineering/mcp-error-envelope.md](../../../docs/engineering/mcp-error-envelope.md) — `ok(data)` or `fail(category, message)`. The contract test asserts this shape against the live tool output.
+
+```python
+# Contract test pattern — meta-repo MCP
+def test_cos_search_returns_envelope() -> None:
+    result = invoke_tool("cos_search", {"query": "test"})
+    assert "ok" in result
+    if result["ok"]:
+        assert "data" in result
+        assert "meta" in result["data"]
+    else:
+        assert "error" in result
+        assert "category" in result["error"]
+```
+
+For HTTP services: use `schemathesis` (Python) or `dredd` to drive tests directly from the OpenAPI spec. The spec IS the test.
+
+### E2E tests — one per golden path
+
+End-to-end through a real browser (Playwright) / real mobile device (Detox / Maestro) / real API (curl-style integration). They are slow and flaky, so **keep them few and intentional**.
+
+Rule of thumb: one E2E per user-facing outcome that, if broken, would generate a support ticket within an hour. Typical SaaS: 5–15 E2E tests total.
+
+```typescript
+// Playwright — golden path
+test("user can sign up, verify email, and reach dashboard", async ({page}) => {
+  await page.goto("/signup");
+  await page.fill('[name="email"]', `test-${Date.now()}@example.com`);
+  await page.fill('[name="password"]', "ValidPassword123!");
+  await page.click('button[type="submit"]');
+
+  // Don't poll forever — fail loud at 10s
+  await expect(page).toHaveURL(/dashboard/, {timeout: 10_000});
+});
+```
+
+### Mutation testing — the test of tests
+
+Mutation testing changes one operator at a time in your production code (`+` → `-`, `>` → `>=`) and re-runs your tests. If your tests still pass, that mutation is "alive" — your tests don't actually exercise that code. Run quarterly, not in CI.
+
+- Python: `mutmut`, `cosmic-ray`
+- TypeScript / JS: `stryker`
+- Go: `go-mutesting`
+
+For security-critical code (auth, payment, crypto), aim for ≥85% mutation score. For other code, ≥60% is good signal.
+
+## Three Rules of Test Authorship
+
+1. **Test the behavior, not the implementation.** A test that calls a private method or mocks an internal class breaks every refactor. Test the public API surface — what callers see.
+2. **One assertion idea per test.** A test asserting six unrelated things fails opaquely. Split into six tests with clear names.
+3. **Name the failure, not the success.** `test_returns_404_when_user_not_found` beats `test_get_user`. The failing test message should already tell you what broke.
+
+## The Error-Path Contract
+
+Every `try/except` (Python) / `try/catch` (TypeScript) / `if err != nil` (Go) block needs a test that triggers the error branch. This is the [clean-code](../clean-code/SKILL.md) contract. Repeating it here because most suites get this wrong:
+
+```python
+def test_verify_purchase_raises_on_verification_failure(mocker) -> None:
+    mocker.patch("apps.purchases.services.PurchaseService.get_verified",
+                 side_effect=VerificationError("upstream timeout"))
+
+    with pytest.raises(ServiceUnavailableError, match="Unable to verify"):
+        verify_purchase("user-1", "product-1")
+```
+
+## Coverage — the right target
+
+**Line coverage ≥ 80%** is a useful floor, not a goal. Bad code with 100% line coverage and no edge-case tests is worse than good code with 75% line coverage and complete branch + error-path tests.
+
+What to actually measure:
+
+| Metric | Target | Why |
+|---|---|---|
+| Line coverage | ≥ 80% | Useful floor, easy to game |
+| Branch coverage | ≥ 70% | Catches missed `else` / `case` arms |
+| **Mutation score** | ≥ 60% (≥85% for crypto/auth/payments) | The honest number |
+| **Error-path coverage** | 100% of `except`/`catch`/`if err` | Non-negotiable for prod code |
+
+## Test Hygiene — the rules that prevent suite rot
+
+- **Test independence.** Every test sets up its own state and tears it down. Order-dependent tests are bugs.
+- **No shared mutable globals.** Use fixtures with explicit scope.
+- **Deterministic clock + IDs.** Inject `time.now` / use `freezegun`. UUID generators with seeded RNG.
+- **No `time.sleep`.** Use polling with timeout, not blind sleeps. Replace `sleep(0.1)` with `wait_for(condition, timeout=2)`.
+- **Flake budget = 0.** A "sometimes-failing" test is broken. Quarantine and fix or delete.
+- **Fast = under 100ms per unit, under 1s per integration, under 10s per E2E.** If slower, investigate before merging.
+- **Hermetic CI.** Test container's network egress disabled where possible. No reaching out to a real Stripe sandbox in CI.
+
+## Anti-patterns (reject in review)
+
+- **Mocking what you don't own** — mocking a third-party SDK at the call boundary is fine; mocking `requests.get` in 40 tests means you're testing the mock library.
+- **`assert True` / `assert not None`** — assert the actual value.
+- **`try: ... except: pass` in test code** — swallowing an exception in a test hides bugs in the test code itself.
+- **Sleep-based wait** — replace with explicit poll.
+- **Snapshot tests for everything** — snapshot tests on dynamic output (timestamps, UUIDs) generate noise. Use them for stable HTML/JSON only.
+- **Testing the framework** — don't write tests for `pydantic.BaseModel` accepting a string. Test your code.
+- **One giant `test_main` function** — split.
+
+## Verification (meta-repo specific)
+
+This skill's own examples should pass in coding-os. Verify:
+
+- `uv run --extra rag pytest src/core/thinking_os/tests/ -q` — matrix command for thinking_os.
+- `uv run pytest tests/test_skill_frontmatter.py -q` — this skill's frontmatter is valid.
+
+## Tooling
+
+Gate coverage in CI (coverage.py JSON or Cobertura XML):
+`python3 scripts/coverage_gate.py coverage.json --min 80`
+
+## See also
+
+- [references/test-types.md](references/test-types.md) — when each test type pays off + the pyramid.
+- [assets/test-review-checklist.md](assets/test-review-checklist.md) — the review gate.
+- [clean-code](../clean-code/SKILL.md) — error-path test contract.
+- [observability](../observability/SKILL.md) — CI signal hygiene (don't trust a green run with broken telemetry).
+- [search](../search/SKILL.md) — finding existing tests before adding new ones.
+- [api-design](../api-design/SKILL.md) — contract test seam.
+- [src/core/rules/test-discipline.md](../../rules/test-discipline.md) — matrix-targeted test command discipline for this repo.

core/skills/testing-strategy/assets/test-review-checklist.md

@@ -0,0 +1,25 @@
+<!-- domain:UNIVERSAL | layer:asset | ssot:false | updated:2026-06-04 -->
+# Test Review Checklist
+
+Run when adding tests or reviewing a PR's tests.
+
+## Right type, right level
+- [ ] Logic covered by unit tests; the dependency seam by integration; the journey by one end-to-end.
+- [ ] No end-to-end test where a unit/integration test would give the same confidence cheaper.
+- [ ] API/MCP boundaries have a contract test (shape can't silently drift).
+- [ ] Bug fix ships with a regression test that fails on the old code.
+
+## Quality (F.I.R.S.T)
+- [ ] Fast — no unnecessary sleeps, real network, or full-stack spin-up for unit tests.
+- [ ] Isolated — passes alone and in any order; no shared mutable state.
+- [ ] Repeatable — clock/random/network controlled (no flake).
+- [ ] Self-validating — explicit assertions, not manual inspection or print debugging.
+- [ ] Error paths tested, not just the happy path (pairs with clean-code).
+
+## Coverage
+- [ ] `python3 scripts/coverage_gate.py <report> --min <floor>` → `pass`.
+- [ ] Coverage is a floor against regressions, not the goal; branches + error paths covered.
+- [ ] (Periodic) mutation testing confirms tests catch injected bugs.
+
+## Repo discipline
+- [ ] Ran the matrix-targeted command for what changed, not the full sweep (test-discipline.md).

core/skills/testing-strategy/references/test-types.md

@@ -0,0 +1,52 @@
+<!-- domain:UNIVERSAL | layer:reference | ssot:true | updated:2026-06-04 -->
+# Test Types — When Each Pays Off
+
+> P: Pick the cheapest test that gives the confidence the change needs.
+> R: Deciding what to test for a feature or bug fix.
+> S: The repo's matrix-command discipline — [test-discipline.md](../../../rules/test-discipline.md).
+> N: [SKILL.md](../SKILL.md), [test-review-checklist.md](../assets/test-review-checklist.md)
+
+> Nav: [Skill](../SKILL.md)
+
+## The pyramid (cost ↑, count ↓ as you go up)
+
+| Type | Proves | Cost | Use a lot? |
+|---|---|---|---|
+| unit | one function/branch, pure logic | ms | yes — the base |
+| integration | modules + a real dependency (DB, queue) | 10s–100s ms | moderate |
+| contract | producer/consumer agree on a shape | fast | per API boundary |
+| end-to-end | a full user journey across the stack | seconds | few, critical only |
+| smoke / run-the-deliverable | the shipped entrypoint actually runs (not just imports under pytest) | seconds | every new script / CLI / cron entry |
+| property-based | invariants over generated inputs | medium | for logic with many cases |
+| mutation | the tests actually catch bugs | slow | periodic audit |
+| fuzz | crashes on malformed input | slow | parsers, untrusted input |
+
+Most confidence per dollar is at the unit + integration layer. End-to-end is the
+tip ([end-to-end-testing](../../end-to-end-testing/SKILL.md)) — reserve it for
+journeys that mean "product is down" if broken.
+
+## Choosing for a change
+
+- **Pure function / business rule** → unit (+ property-based if many input classes).
+- **Repository / query / migration** → integration against a real DB (testcontainers / a test schema).
+- **HTTP/MCP endpoint shape** → contract test (schema assertion) so consumers don't break.
+- **Critical journey (checkout, login)** → one end-to-end.
+- **Bug fix** → a unit/integration test that fails on the old code, passes on the fix (regression lock).
+
+## Coverage is a floor, not a goal
+
+```bash
+python3 scripts/coverage_gate.py coverage.json --min 80    # CI gate
+```
+
+100% line coverage with no assertions proves nothing; 80% with error-path tests
+proves a lot. Gate a **floor** to stop regressions, but chase *behavior* coverage
+(every branch + every error path) not a number. Mutation testing
+(`mutmut`/`stryker`) measures whether tests actually catch bugs — the real signal.
+
+## Test quality (F.I.R.S.T)
+
+Fast · Isolated (no shared state/order dependence) · Repeatable (no clock/network
+flake) · Self-validating (a clear pass/fail, no manual inspection) · Timely
+(written with the code). A test that needs another test to run first, or a fixed
+clock it doesn't control, is a liability — fix it or delete it.

core/skills/testing-strategy/scripts/coverage_gate.py

@@ -0,0 +1,79 @@
+"""Enforce a coverage threshold from a coverage report (gate for CI).
+
+PURPOSE:      Fail a build when line coverage drops below a floor — one command
+              instead of eyeballing a report. Reads coverage.py JSON or
+              Cobertura XML so it works for Python and most JS/Go toolchains.
+INPUT:        report path (coverage.json / coverage.xml) or --file. [--min N]
+              required percent (default 80). [--json]
+OUTPUT:       Percent + verdict on stderr; "pass"/"fail" on stdout. Exit 0 if
+              >= min, 1 if below, 2 usage/parse error.
+DEPENDENCIES: stdlib only (json, xml.etree).
+NOTES:        Pure percent_from_* parsers are unit-testable. Spec:
+              docs/playbooks/skill-authoring.md; craft: ../SKILL.md.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+import xml.etree.ElementTree as ET
+from pathlib import Path
+
+
+def percent_from_coveragepy(text: str) -> float:
+    data = json.loads(text)
+    return float(data["totals"]["percent_covered"])
+
+
+def percent_from_cobertura(text: str) -> float:
+    root = ET.fromstring(text)
+    # Cobertura root carries line-rate in [0,1]; convert to percent.
+    rate = root.get("line-rate")
+    if rate is None:
+        raise ValueError("no line-rate on cobertura root")
+    return float(rate) * 100.0
+
+
+def parse_percent(path: Path) -> float:
+    text = path.read_text(encoding="utf-8")
+    if path.suffix == ".json":
+        return percent_from_coveragepy(text)
+    if path.suffix == ".xml":
+        return percent_from_cobertura(text)
+    # sniff
+    stripped = text.lstrip()
+    return (
+        percent_from_coveragepy(text) if stripped.startswith("{") else percent_from_cobertura(text)
+    )
+
+
+def main(argv: list[str]) -> int:
+    parser = argparse.ArgumentParser(description=__doc__.split("\n")[0])
+    parser.add_argument("path", nargs="?", default="coverage.json")
+    parser.add_argument("--file", default=None)
+    parser.add_argument("--min", default=80.0, type=float)
+    parser.add_argument("--json", action="store_true", dest="as_json")
+    args = parser.parse_args(argv)
+
+    target = Path(args.file or args.path)
+    try:
+        pct = parse_percent(target)
+    except FileNotFoundError:
+        print(f"error: {target} not found", file=sys.stderr)
+        return 2
+    except (json.JSONDecodeError, ET.ParseError, KeyError, ValueError) as exc:
+        print(f"error: {target}: {exc}", file=sys.stderr)
+        return 2
+
+    ok = pct >= args.min
+    print(f"  coverage {pct:.1f}% (min {args.min:.1f}%)", file=sys.stderr)
+    if args.as_json:
+        print(json.dumps({"percent": round(pct, 2), "min": args.min, "pass": ok}))
+    else:
+        print("pass" if ok else "fail")
+    return 0 if ok else 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main(sys.argv[1:]))