@zerwiz/ymir 0.1.34 → 0.1.35

package/apps/smidja-factory/cookbooks/update_config.md

@@ -0,0 +1,103 @@
+# Update Config
+
+Add or retune agents in `smidja.config.yaml`.
+
+## Retune model or thinking
+
+Edit the agent's entry in place:
+
+```yaml
+  - name: builder
+    model: google/gemini-3.6-flash   # ALWAYS provider/model-id
+    thinking: high                   # was medium
+```
+
+Write the model as `provider/model-id`, never a bare id. The same model is usually carried by several providers, and an ambiguous pattern raises in `agents.validate()` — grounding every agent that inherits it. See `references/config.md`.
+
+Thinking levels are Pi's reasoning effort: `off | minimal | low | medium | high | xhigh | max`. It only bites when the model is registered with `reasoning: true` in `~/.pi/agent/models.json`.
+
+**A model change means a fresh session.** `agent_map.json` records the model each coding-agent session was created with. When a joined run (`--smidja-id`) finds the config's model no longer matches the recorded one, that agent starts a **new** session rather than resuming — the map is updated, never a bad resume. Thinking changes do not invalidate a session; model changes do. Expect the agent to lose its accumulated context window on the first run after the change.
+
+## Recolor an agent's lane
+
+```yaml
+  - name: builder
+    color: "#22d3ee"      # hex; the starter roster ships violet/cyan/amber/green
+```
+
+Purely cosmetic and safe to change mid-project: the color rides the `agent_start` event and the `agent_sessions` row, so the visualizer picks it up on the next run without touching past sessions. Omit the key to let the UI's fallback palette choose.
+
+## Retune tools
+
+Pi's seven builtins: `read`, `bash`, `edit`, `write`, `grep`, `find`, `ls`. The last three are **off in bare Pi**, so an agent that doesn't name them will shell out through `bash` to search and list.
+
+Set the roster-wide floor in `defaults`, then narrow per agent:
+
+```yaml
+defaults:
+  tools: [read, bash, edit, write, grep, find, ls]
+
+agents:
+  - name: reviewer
+    tools:                # explicit list wins over defaults
+      - read
+      - grep
+      - find
+      - ls
+      - bash
+      - write
+```
+
+**Resolution:** the agent's own list wins → else it inherits `defaults.tools` → else `None`, meaning all tools. An empty list is not "all tools"; it is a tool-less agent, and it will stall.
+
+Narrow by role, not by reflex:
+
+- Any agent that must produce a `context_handoff/` artifact needs **`write`** — without it, it falls back to a `bash` heredoc to create the file the gate checks for.
+- Withhold `edit`/`write` only where the restriction *is* the guarantee. The reviewer's contract is "change nothing", so withholding `edit` makes that structural instead of merely prompted.
+- Recon agents should get the full read surface (`read`, `grep`, `find`, `ls`) — cheaper and more legible in the trace than the equivalent `bash` calls.
+
+**Extension tools count against the allowlist.** `--tools` filters built-in, extension, and custom tools alike. Once an agent has a `tools` list — its own, or inherited from `defaults` — a tool registered by one of its `harness_engineering` extensions is dropped unless it is named there. Nothing errors: the extension loads, the run passes, the tool is just never offered. Any agent with a tool-registering extension must list that tool by name.
+
+## Add harness extensions
+
+```yaml
+    harness_engineering:
+      - .pi/extensions/json_guard.ts    # a pi extension FILE PATH
+```
+
+Entries are pi extension **file paths**, passed through as `pi -e <path>`, applied to that agent only. Reach for an output-tightening extension when an agent keeps wrapping its envelope in prose and burning correction retries. The starter roster ships with none — this is an escape hatch, not a default.
+
+**Adding a tool-registering extension is a two-part edit.** The extension path goes in `harness_engineering`, *and* the tool name it registers goes in that agent's `tools` list:
+
+```yaml
+  - name: reviewer
+    harness_engineering:
+      - .pi/extensions/ast_query.ts     # registers tool: ast_query
+    tools:
+      - read
+      - grep
+      - find
+      - ls
+      - bash
+      - ast_query                       # REQUIRED — or the extension loads and its tool is filtered out
+```
+
+Skip the second half and it fails silently: extension loaded, run green, tool never available to the model. Extensions that only shape output or register flags — no new tool — need no `tools` change.
+
+## Add a new agent
+
+Three steps, all required — skipping any one fails `agents.validate()` at smidja startup, before anything spawns:
+
+1. **Prompts.** Create `smidja/smidja_data/prompt_engineering/{name}/system.md` (Purpose + Instructions — the agent's static identity, nothing else) and `user.md` (an h3 per incoming datum: `{{prompt}}`, `{{previous_envelope}}`, `{{context_handoff_dir}}`, then the task, then a `## Report` section showing the exact output JSON). Copy an existing pair as the shape.
+2. **Config entry.** Name, purpose, prompt refs, plus anything that differs from `defaults`.
+3. **An output type.** Every agent call parses against a concrete Pydantic model in `smidja_modules/data_types.py`. If none of `PlanOutput`, `BuildOutput`, `ScoutOutput`, `ReviewOutput`, `DocumentOutput` fits the new agent's report, add one — see `update_modules.md`. The user prompt's `Report` section must show exactly that JSON shape.
+
+Then name the agent in an smidja's `REQUIRED_AGENTS` and call it.
+
+## Rules that do not bend
+
+- smidja scripts name **agents**, never models. Swapping a model is a config edit and touches no Python.
+- One agent, one prompt, one purpose. If an entry needs two purposes, it is two agents.
+- Output types never appear in config — they live at the call site, paired with the user prompt.
+
+Full spec: `references/config.md`.

package/apps/smidja-factory/cookbooks/update_modules.md

@@ -0,0 +1,100 @@
+# Update Modules
+
+Extend `smidja/smidja_modules/` with new low-level logic.
+
+## The rule
+
+**ALL low-level logic lives in `smidja_modules/`; smidja scripts stay thin.** An `smidja_*.py` file declares agents, sequences phases, and returns an exit code. Anything else — subprocess handling, parsing, retry mechanics, git plumbing, reusable predicates — goes in a module.
+
+## Where things go
+
+| Module | Owns |
+|---|---|
+| `data_types.py` | Every Pydantic model: `AgentCall`, `PhaseParams`, `Phase`, `EnvelopeBase` + one output type per agent call, the config models (`AgentConfig`, `smidjaConfig`), `EventRecord`, and `PiRequest`/`PiResult` |
+| `agents.py` | `load_config`, `validate`, resolving an entry → coding-agent interface + model + thinking + harness extensions |
+| `runner.py` | the `Run` object; `run.phase(PhaseParams)` context manager; `ph.call(AgentCall)` |
+| `agent_pi.py` | the Pi interface (v1) — non-interactive `pi -p --mode json`, JSONL stream tailed live, model resolved against `~/.pi/agent/models.json`; `--session-id` creates-or-continues, so running and continuing an agent are the same call |
+| `agent_cc.py` | the Claude Code interface — stubbed in v1, lands in v2 |
+| `gates.py` | validation gates over envelope claims |
+| `changes.py` | deterministic change capture: resolve the base ref, `git diff` into `context_handoff/changes.diff`, adapt the `ChangeSet` into an envelope an agent can be handed |
+| `prompts.py` | load system/user prompt refs from config, render placeholders |
+| `session.py` | mint or join `smidja_id`, maintain `agent_map.json`, create session dirs incl. `context_handoff/` |
+| `tracer.py` | append JSONL **and** insert every event into `smidja.db` as it happens |
+| `console.py` | the terminal narrative — every line printed also lands in the db as a `log` event, so the UI reads the same story; plain sequential lines, no spinners |
+| `console.py` | the rich stdout reporter — every line printed is ALSO traced as a `log` event (`{message, level}`) so the terminal and the swim-lane UI tell the same story |
+| `git_helper.py` | branch, status, diff, commit — the raw plumbing `changes.py` composes |
+| `utils.py` | safe subprocess env, logging, `resolve_prompt` |
+
+## Never `print()`
+
+Modules report through `run.console` — never a bare `print()`. Each console method prints a rich line **and** writes it to `smidja.db` as a `log` event with payload `{message, level}`, both from one `_emit` helper, so the terminal narrative and the swim-lane UI can't drift. New output means a new method on `Console`, not a print at the call site.
+
+## The four-param rule
+
+**Any function taking more than 4 parameters gets them converted into a concrete data type in `data_types.py`.** `AgentCall` and `PhaseParams` are the pattern — `run.phase()` and `ph.call()` each take exactly one object. This is skill-wide: every module the smidja generates obeys it.
+
+```python
+class ReviewParams(BaseModel):
+    """Everything review_changes() needs. Passed as one object, never loose params."""
+    base_ref: str
+    paths: list[str]
+    max_diff_lines: int = 2000
+    ignore_generated: bool = True
+    reviewer: str = "scout"
+```
+
+## Adding an output type
+
+Every agent call parses against a concrete type. Extend `EnvelopeBase` — `status`, `summary`, `artifacts`, `notes_for_next_agent` — with only the fields that call actually needs:
+
+```python
+class ReviewOutput(EnvelopeBase):
+    approved: bool
+    blocking: list[str] = []
+```
+
+**The output contract is a synced triad — one change means three edits, always together:**
+
+1. The type in `data_types.py` (the enforcer).
+2. The agent's `user.md` `## Report` section showing exactly that JSON (the ask).
+3. Every call site passing `output_type=` (the binding) — `grep -rn "ReviewOutput" smidja/` to find them all.
+
+If the type and the Report example drift, the agent produces what the prompt asked for, the parser rejects what the type expects, and every call burns correction round-trips before landing — a slow, silent tax. Renaming or removing a field is the same triad edit. Schema details: `references/handoff.md`.
+
+## Adding a gate
+
+A gate is a callable — `gate(envelope, run) -> GateReport`. You record **one check per item you look at**, and the harness derives the verdict: any failed check is a violation, and no failed checks means pass.
+
+```python
+from smidja_modules.data_types import GateReport
+
+def tests_declared_passed(envelope, run) -> GateReport:
+    """Verify the envelope's own test claims, after the fact."""
+    report = GateReport()
+    for f in envelope.failures:
+        report.check(f.test, False, f.error)
+    report.check("suite", envelope.passed,
+                 "all declared tests passed" if envelope.passed
+                 else f"{len(envelope.failures)} declared failure(s)")
+    return report
+```
+
+`report.check(item, ok, note)` appends and returns the report, so a single-item gate is one line: `return GateReport().check(command, ok, f"exit {code}")`.
+
+**Write a note on passing checks too, not just failures.** The note is the evidence, and it is what makes a green gate worth reading — `artifacts_exist ✓ 1 checked · plan.md — exists, 454B` tells you what was verified, where a bare ✓ tells you nothing. Notes on failed checks double as the reason and are what the agent is told, so phrase them as the problem: `"claimed changed file does not exist"`.
+
+Rules that keep gates honest:
+
+- **Verify claims, never predict.** File names and counts are unknowable before the agent finishes; gates check what the envelope declared.
+- **Quantity as properties, not counts.** "at least one artifact", "ALL declared paths exist" — never `len(artifacts) == 3`.
+- **Record checks, don't raise.** The harness feeds the derived violations back into the same session as a correction — context intact, bounded by the phase's `retries` — and traces every check, passed or failed, to `gate_results.checks_json` and the `gate_pass`/`gate_fail` event payload.
+- **Check every item, even after one fails.** Don't early-return on the first problem; the agent fixes more per correction round when it sees every failure at once, and the trace shows the full picture.
+- **Don't gate the ungateable.** Plan quality and code taste are a reviewer agent's job or a human's.
+
+A gate that returns a plain `list[str]` of violations still works — the harness adapts it — but it records no evidence for the items that passed, so prefer a `GateReport`.
+
+Reusable gates live in `gates.py`; genuine one-offs can be defined inline at the smidja call site and passed in `gates=[...]`.
+
+## Before you finish
+
+Run the smoke smidja — `uv run smidja/smidja_prompt.py "ping"` — since every module change rides the same path a real run does.

package/apps/smidja-factory/cookbooks/update_smidja.md

@@ -0,0 +1,90 @@
+# Update smidja
+
+Modify an existing smidja chain — add phases, add gates, add a bounded fix loop.
+
+## Add a phase
+
+Insert a `with run.phase(...)` block where it belongs in the sequence. Pick the right `kind`: `agent` for a `ph.call(...)`, `code` for a deterministic step, `engineer` for a human touchpoint. If the new phase names an agent not already in `REQUIRED_AGENTS`, add it there too — otherwise validation passes and the run dies mid-flight instead of at startup.
+
+```python
+    with run.phase(PhaseParams(name="scout", kind="agent", owner="scout",
+                               description="Locate the code the request touches")) as ph:
+        found = ph.call(AgentCall(output_type=ScoutOutput, prompt=prompt))
+```
+
+Phase `name` must be unique within the run — that is what the UI keys blocks on. In a loop, suffix it (`f"test_{i}"`).
+
+`description` is **required**, and `PhaseParams` rejects both a blank one and one that merely restates the name. It is the single line of intent the trace, the console, and the UI phase block show, so write what the phase does and why — `"Land the code only now: green suite, approved review"`, not `"Commit build"`.
+
+A code phase does its work in the block body and logs what it did. The commit phase that closes `smidja_plan_build.py` and `smidja_plan_build_test.py` is the pattern:
+
+```python
+    with run.phase(PhaseParams(name="commit", kind="code", owner="git",
+                               description="Land the builder's changes, using the message it wrote")) as ph:
+        message = build.commit_message or f"smidja({run.smidja_id}): {build.summary}"
+        ph.log(sha=git_helper.commit_all(message), message=message)
+```
+
+`commit_message` is a field on `PlanOutput`, `BuildOutput`, and `DocumentOutput` that the agent fills in **for its own work product**, so always pair it with a fallback — it defaults to empty. `commit_all` raises if the cwd is not a git repo or nothing changed, which fails the phase rather than committing nothing. A chain that commits more than once (`smidja_simple_sdlc.py`) commits each product with its own author's message.
+
+## Remove a phase
+
+Delete the block, drop any now-unused agent from `REQUIRED_AGENTS`, and re-thread the chain: whatever the removed phase produced was probably somebody's `previous=`. Point that call at the surviving upstream envelope.
+
+## Add gates
+
+Gates are callables over the finished envelope — `gate(envelope, run) -> GateReport`, recording one `check(item, ok, note)` per thing they looked at, with violations derived from the failed ones. Compose them per call:
+
+```python
+        build = ph.call(AgentCall(output_type=BuildOutput, prompt=prompt, previous=plan,
+                                  gates=[gates.artifacts_exist, gates.diff_matches_claims]))
+```
+
+On violations the harness does **not** restart the agent — it sends the violation list back into the **same session** as a correction (pi's `--session-id` creates-or-continues, so the context window is intact), bounded by that phase's `retries`. Every gate result is traced to the `gate_results` table. Exhausting the retries raises `GateFailure` and fails the phase.
+
+Gate claims, not guesses: declared artifacts exist and are non-empty, declared JSON parses, declared changes appear in the diff, declared test commands pass. Never hardcode counts — express quantity as a property of the declared list ("at least one artifact", "ALL declared paths valid"). Plan quality and code taste are not gateable; that is a reviewer agent or a human. New reusable gates go in `smidja_modules/gates.py` (`update_modules.md`).
+
+## Add a bounded fix loop
+
+The pattern from `smidja_build_test.py` — always bounded by a module-level constant. The runner is a **code** phase, because the command is known; only repairing it needs an agent:
+
+```python
+MAX_FIX_LOOPS = 3
+
+    test = None
+    for i in range(1, MAX_FIX_LOOPS + 1):
+        with run.phase(PhaseParams(name=f"test_{i}", kind="code", owner="quality",
+                                   description="Run the suite — a known command, so code runs it")) as ph:
+            test = quality.run_tests(run)          # QualityResult, not an envelope
+            ph.log(passed=test.passed, artifacts=", ".join(test.artifacts))
+
+        if test.passed:
+            break
+
+        with run.phase(PhaseParams(name=f"fix_{i}", kind="agent", owner="builder", retries=1,
+                                   description="Repair what the suite reported, from its verbatim output")) as ph:
+            previous = ph.call(AgentCall(output_type=BuildOutput, prompt=prompt,
+                                         previous=quality.as_envelope(test, "tests"),
+                                         gates=[gates.diff_matches_claims]))
+
+    return run.finish(accepted=test is not None and test.passed,
+                      reason=f"the suite still failed after {MAX_FIX_LOOPS} fix attempt(s)")
+```
+
+`run.finish()` ends every smidja, and it takes the acceptance criterion the phase
+statuses cannot express. A test phase that ran a red suite **succeeded** — the
+runner did its job — so phases alone would report a green run that never passed
+its tests, in the db and the UI as well as the terminal. Pass `accepted=` and
+the exit code, the session status, and the banner are decided together.
+
+`quality.as_envelope` is the adapter: a deterministic result shaped as an envelope, so the builder cannot tell it came from code. Wire the real command in `quality.py` first — the stamped blocks are `echo` placeholders that announce themselves.
+
+Three distinctions worth keeping straight:
+
+- **Gate retries vs. JSON retries.** `retries` buys extra *gate*-correction rounds. Malformed final JSON is handled separately and always — `JSON_FIX_ATTEMPTS` in `smidja_modules/agents.py` (2 by default) re-prompts the same session for a valid object even on a phase with `retries=0`. Raising the phase's `retries` does not buy more JSON attempts, and vice versa.
+- **Phase retries vs. fix loops.** `retries=N` on `PhaseParams` re-attempts one agent phase's gate corrections, re-sent into the same session with its context intact. (Code-phase re-execution is not implemented in v1.) A fix loop is a *chain* of phases repeated — different agents, new envelopes each pass.
+- **The test phase succeeds when it runs and reports correctly.** A failing suite does not fail that phase; it fails the run, checked at the end. The runner did its job; the code didn't.
+
+## Keep scripts thin
+
+An smidja is sequencing and acceptance — nothing else. The moment you are writing parsing, subprocess handling, retry mechanics, or a reusable predicate inside `smidja_*.py`, it belongs in `smidja_modules/`. See `update_modules.md`.