@zerwiz/ymir 0.1.34 → 0.1.35

package/apps/smidja-factory/docs/enhancement-plan.md

@@ -0,0 +1,420 @@
+# Smíðja Enhancement Plan
+
+**Date**: September 2026  
+**Scope**: Visualizer (frontend) + Smíðja Backend (Python)  
+**Purpose**: Comprehensive roadmap for extending the smithy observability and execution capabilities
+
+---
+
+## Executive Summary
+
+The Smíðja is a **Smíðja** — deterministic Python scripts own sequencing/retries/acceptance; coding agents (Pi/opencode) work inside bounded phases; typed JSON envelopes carry context; everything streams into SQLite for the polled visualizer.
+
+This plan identifies high-value enhancements across both surfaces, organized by impact and implementation complexity.
+
+---
+
+## Current Architecture Snapshot
+
+### Visualizer (`apps/visualizer/`)
+| Layer | Technology | Key Files |
+|-------|------------|-----------|
+| UI Framework | Vue 3 + TypeScript + Vite | `src/App.vue`, `src/components/*.vue` |
+| State | Pinia-style composables | `src/lib/chat-store.ts`, `src/lib/api.ts` |
+| Server | Bun HTTP + SQLite (readonly) | `server/index.ts`, `server/db.ts` |
+| Real-time | 500ms polling + cursor pagination | `SessionTrace.vue` tick loop |
+| Themes | CSS custom properties | `neutral` (default) + `classic` (deep-space) |
+| Desktop | Electron (preload + main) | `desktop/main.js`, `desktop/preload.js` |
+| Memory | Kaia engram bridge (Python) | `/api/memory/*` proxy to `:4602` |
+
+**Current Views**: Sessions → Session Trace (lanes + waterfall) → Phase Detail + Envelopes + Gates + Thinking + Tool calls; Memory; Decisions (self-improving surface); Stats (tokens/cost/cache/savings); Orchestrator Chat (Kaia + session launch); Settings
+
+### Smíðja Backend (`templates/smidja/`)
+| Layer | Technology | Key Files |
+|-------|------------|-----------|
+| Orchestration | Python 3.11+ (uv scripts) | `smidja_*.py` |
+| Data Contracts | Pydantic v2 | `smidja_modules/data_types.py` |
+| Agent Runtime | `agent_pi.py` (Pi), `agent_opencode.py` (opencode) | |
+| Tracing | SQLite WAL + JSONL events | `smidja_modules/tracer.py` |
+| Quality Gates | Deterministic subprocess runs | `smidja_modules/quality.py` |
+| Permissions | Path-based write enforcement | `smidja_modules/permissions.py` |
+| Context Handoff | File-based `context_handoff/` dir | `smidja_modules/agents.py` |
+| Sub-agents | Task-tool lane materialization (G2) | `smidja_modules/agents.py` |
+
+**Current Smithies**: `smidja_scout`, `smidja_simple_sdlc`, `smidja_plan_build`, `smidja_build_test`, `smidja_plan_build_test`, `smidja_plan_build_test_quality`, `smidja_build_review`, `smidja_document`, `smidja_orchestrate`, `smidja_recon_iv`, `smidja_prompt`, `smidja_quality`
+
+---
+
+## Enhancement Categories
+
+### A. Visualizer — Observability & UX
+
+#### A.1 Live Collaboration & Multi-User
+| ID | Enhancement | Description | Effort | Priority |
+|----|-------------|-------------|--------|----------|
+| A.1.1 | **Presence indicators** | Show other engineers viewing the same session (WebSocket presence) | Medium | High |
+| A.1.2 | **Shared steer annotations** | Steer messages attributed to author, visible to all viewers | Medium | High |
+| A.1.3 | **Session handoff** | "Take over" a paused run from another engineer's steer context | Medium | Medium |
+| A.1.4 | **Comment threads on phases** | Pin discussion to specific phase blocks (like GitHub PR comments) | Medium | Medium |
+
+#### A.2 Advanced Trace Analysis
+| ID | Enhancement | Description | Effort | Priority |
+|----|-------------|-------------|--------|----------|
+| A.2.1 | **Comparative trace diff** | Side-by-side waterfall of two runs (baseline vs current) | Large | High |
+| A.2.2 | **Token flow Sankey** | Visualize input→cache→output→reasoning token flow per agent/phase | Medium | High |
+| A.2.3 | **Failure pattern miner** | Auto-cluster similar failures across runs with suggested fixes | Medium | High |
+| A.2.4 | **Critical path highlighter** | Auto-detect and highlight the longest dependency chain in waterfall | Small | Medium |
+| A.2.5 | **Phase duration heatmap** | Calendar view of phase durations across runs (CI-style) | Small | Medium |
+
+#### A.3 Kaia Memory & Knowledge
+| ID | Enhancement | Description | Effort | Priority |
+|----|-------------|-------------|--------|----------|
+| A.3.1 | **Memory graph explorer** | Interactive entity-relationship graph from engram facts/episodes | Large | Medium |
+| A.3.2 | **Cross-project memory search** | Query Kaia memory across all smidja repos from one UI | Medium | Medium |
+| A.3.3 | **Memory decay visualization** | Show salience/confidence decay over time per fact/episode | Small | Low |
+| A.3.4 | **Admission timeline** | Visualize Kaia's admission pipeline: prompt → recall → admit → dispatch | Small | Medium |
+
+#### A.4 Orchestrator Chat Enhancements
+| ID | Enhancement | Description | Effort | Priority |
+|----|-------------|-------------|--------|----------|
+| A.4.1 | **Inline tool result preview** | Expand tool calls in chat to show args/result without leaving thread | Small | High |
+| A.4.2 | **Chat-to-smidja trace linking** | Click a launched session in chat → jump to its trace (exists) + back-link | Small | High |
+| A.4.3 | **Structured task templates** | Quick-insert templates for common tasks (bug fix, feature, refactor) | Small | Medium |
+| A.4.4 | **Multi-model chat** | Switch Kaia's model mid-conversation; show model badge per message | Small | Medium |
+| A.4.5 | **Chat export / session resume** | Export chat + launched sessions as portable bundle; resume later | Medium | Low |
+
+#### A.5 Settings & Configuration UI
+| ID | Enhancement | Description | Effort | Priority |
+|----|-------------|-------------|--------|----------|
+| A.5.1 | **Visual roster editor** | Drag-and-drop agent roster builder (replaces YAML editing) | Large | High |
+| A.5.2 | **Model tier picker** | Visual model catalog with tier badges, pricing, local/online toggle | Medium | High |
+| A.5.3 | **Smíðja chain builder** | Visual pipeline editor for creating custom smidja scripts | Large | Medium |
+| A.5.4 | **Theme builder** | Custom CSS variable editor with live preview + export/import | Medium | Low |
+
+#### A.6 Mobile & Accessibility
+| ID | Enhancement | Description | Effort | Priority |
+|----|-------------|-------------|--------|----------|
+| A.6.1 | **Responsive waterfall** | Horizontal scroll + collapsed lanes for mobile viewport | Medium | Medium |
+| A.6.2 | **Screen reader support** | ARIA labels, live regions for polling updates, keyboard nav | Medium | High |
+| A.6.3 | **High contrast theme** | WCAG AAA compliant theme variant | Small | Medium |
+| A.6.4 | **PWA installability** | Service worker + manifest for offline session browsing | Medium | Low |
+
+---
+
+### B. Visualizer — Data & API
+
+#### B.1 API Extensions
+| ID | Enhancement | Description | Effort | Priority |
+|----|-------------|-------------|--------|----------|
+| B.1.1 | **WebSocket event stream** | Replace polling with server-sent events for live updates | Medium | High |
+| B.1.2 | **GraphQL endpoint** | Flexible queries for custom dashboards / external tools | Large | Medium |
+| B.1.3 | **Batch session export** | `/api/sessions/export?ids=...` → NDJSON/CSV/Parquet | Small | Medium |
+| B.1.4 | **Run comparison API** | `/api/compare?a=<id>&b=<id>` → structured diff | Medium | High |
+
+#### B.2 Data Enrichment
+| ID | Enhancement | Description | Effort | Priority |
+|----|-------------|-------------|--------|----------|
+| B.2.1 | **GitHub PR linking** | Auto-link sessions to PRs via commit messages / branch names | Medium | High |
+| B.2.2 | **Jira/Linear ticket sync** | Bidirectional sync: session ↔ ticket (status, comments, links) | Large | Medium |
+| B.2.3 | **Cost allocation tags** | Tag runs by project/team/feature for cost center reporting | Small | Medium |
+| B.2.4 | **Custom metric ingestion** | POST `/api/metrics` for arbitrary KPI tracking (deployment freq, etc.) | Small | Low |
+
+---
+
+### C. Smíðja Backend — Execution & Orchestration
+
+#### C.1 New Smíðja Chains (Templates)
+| ID | Enhancement | Description | Effort | Priority |
+|----|-------------|-------------|--------|----------|
+| C.1.1 | **`smidja_security_audit`** | Scout → threat model → code review → fix → retest → document | Medium | High |
+| C.1.2 | **`smidja_perf_optimize`** | Benchmark → profile → optimize → benchmark → regression test | Medium | High |
+| C.1.3 | **`smidja_migration`** | Analyze → plan migration → execute → verify → rollback plan | Medium | Medium |
+| C.1.3 | **`smidja_dependency_update`** | Scan → plan updates → test → staged rollout → verify | Small | Medium |
+| C.1.4 | **`smidja_incident_response`** | Triage → diagnose → fix → verify → postmortem → Kaia memory | Medium | High |
+| C.1.5 | **`smidja_feature_flag_rollout`** | Gradual rollout with metric gates + automatic rollback | Medium | Medium |
+
+#### C.2 Agent Capability Extensions
+| ID | Enhancement | Description | Effort | Priority |
+|----|-------------|-------------|--------|----------|
+| C.2.1 | **Multi-modal agents** | Vision input (screenshots, diagrams) for UI builder / documenter | Large | High |
+| C.2.2 | **Agent skill marketplace** | Pluggable agent capabilities (npm-style packages with prompts + tools) | Large | Medium |
+| C.2.3 | **Agent sandbox profiles** | Per-agent filesystem/network caps (beyond `writes:`) via WASM/deno | Large | Medium |
+| C.2.4 | **Reasoning budget control** | Per-phase token/$$ ceiling; auto-escalate to stronger model if needed | Medium | High |
+| C.2.5 | **Agent spec compliance** | Validate agent output against OpenAPI/AsyncAPI/GraphQL schemas | Medium | Medium |
+
+#### C.3 Quality & Gates
+| ID | Enhancement | Description | Effort | Priority |
+|----|-------------|-------------|--------|----------|
+| C.3.1 | **Property-based test gate** | Generate + run quickcheck-style tests from type signatures | Medium | High |
+| C.3.2 | **Contract test gate** | Consumer-driven contracts (Pact) for API changes | Medium | Medium |
+| C.3.3 | **Security gate (SAST/DAST)** | Integrate Semgrep/CodeQL/Trivy as deterministic quality blocks | Small | High |
+| C.3.4 | **Performance regression gate** | Compare benchmarks against baseline; fail on >5% regression | Medium | High |
+| C.3.5 | **Accessibility gate** | axe-core / lighthouse CI as quality block | Small | Medium |
+| C.3.6 | **License/compliance gate** | FOSSA / SPDX license check on dependency changes | Small | Low |
+
+#### C.4 Context & Handoff
+| ID | Enhancement | Description | Effort | Priority |
+|----|-------------|-------------|--------|----------|
+| C.4.1 | **Semantic context compression** | LLM-based summarization of handoff files (not truncation) | Medium | High |
+| C.4.2 | **Cross-repo context sharing** | Shared `context_handoff/` across monorepo smithies | Medium | Medium |
+| C.4.3 | **Context versioning** | Git-like history for handoff files with diff/blame | Small | Medium |
+| C.4.4 | **Structured checkpoint format** | Replace markdown checkpoints with typed `Checkpoint` envelopes | Medium | Medium |
+
+#### C.5 Orchestration & Scheduling
+| ID | Enhancement | Description | Effort | Priority |
+|----|-------------|-------------|--------|----------|
+| C.5.1 | **Parallel phase execution** | DAG-based phase scheduler (independent phases run concurrently) | Large | High |
+| C.5.2 | **Cron / scheduled smithies** | `smidja schedule "0 2 * * *" smidja_simple_sdlc "nightly refactor"` | Medium | Medium |
+| C.5.3 | **Event-driven smithies** | `process-event-sources` → smidja trigger (GitHub webhook, cron, etc.) | Medium | High |
+| C.5.4 | **Smíðja composition** | `smidja_chain: [smidja_plan, smidja_build_test, smidja_document]` | Medium | Medium |
+| C.5.5 | **Distributed execution** | Offload agent phases to remote workers (Kubernetes, modal, fly.io) | Large | Low |
+
+---
+
+### D. Smíðja Backend — Developer Experience
+
+#### D.1 Configuration & Onboarding
+| ID | Enhancement | Description | Effort | Priority |
+|----|-------------|-------------|--------|----------|
+| D.1.1 | **`smidja init --interactive`** | Guided setup: language, test runner, agents, models, quality blocks | Small | High |
+| D.1.2 | **Config validation CLI** | `smidja doctor` — full preflight (models, tools, prompts, perms) | Small | High |
+| D.1.3 | **Config schema docs generator** | Auto-generate markdown from `smidjaConfig` Pydantic model | Small | Medium |
+| D.1.4 | **Roster inheritance** | `extends: base-roster` in YAML for shared agent definitions | Small | Medium |
+
+#### D.2 Debugging & Diagnostics
+| ID | Enhancement | Description | Effort | Priority |
+|----|-------------|-------------|--------|----------|
+| D.2.1 | **`smidja replay <session>`** | Re-run a session from any phase with same context (deterministic replay) | Medium | High |
+| D.2.2 | **`smidja diagnose <session>`** | Auto-analyze failure: root cause, suggested fix, similar past failures | Medium | High |
+| D.2.3 | **Live agent REPL** | `smidja shell <session> <agent>` — interactive prompt in agent's context | Large | Medium |
+| D.2.4 | **Phase time-travel** | `smidja phase <session> <phase> --at <timestamp>` — inspect state at point | Medium | Low |
+
+#### D.3 Testing & Validation
+| ID | Enhancement | Description | Effort | Priority |
+|----|-------------|-------------|--------|----------|
+| D.3.1 | **Smíðja contract tests** | Test smidja scripts against known-good fixtures (golden runs) | Medium | High |
+| D.3.2 | **Agent prompt regression suite** | Test agent prompts against model versions for drift detection | Medium | Medium |
+| D.3.3 | **Chaos testing harness** | Inject faults (network, model errors, OOM) into smidja runs | Large | Low |
+
+---
+
+### E. Infrastructure & Platform
+
+#### E.1 Deployment & Operations
+| ID | Enhancement | Description | Effort | Priority |
+|----|-------------|-------------|--------|----------|
+| E.1.1 | **Docker/OCI images** | Pre-built images for smidja runner + visualizer + Kaia bridge | Medium | High |
+| E.1.2 | **Kubernetes operator** | `SmidjaRun` CRD for cluster-native execution + visualizer ingress | Large | Medium |
+| E.1.3 | **GitHub App integration** | Installable app: PR checks, status checks, auto-smidja on labels | Large | High |
+| E.1.4 | **Self-hosted telemetry** | OpenTelemetry export (traces/metrics/logs) to Tempo/Prometheus/Loki | Medium | Medium |
+
+#### E.2 Multi-Tenancy & Teams
+| ID | Enhancement | Description | Effort | Priority |
+|----|-------------|-------------|--------|----------|
+| E.2.1 | **Team workspaces** | Isolated smidja configs, visualizer instances, Kaia memory per team | Large | Medium |
+| E.2.2 | **RBAC for visualizer** | Viewer/Operator/Admin roles per workspace | Medium | Medium |
+| E.2.3 | **Audit log** | Immutable log of all smidja runs, steers, stops, config changes | Medium | High |
+
+---
+
+## Implementation Roadmap
+
+### Phase 1: Quick Wins (Weeks 1-2)
+**Goal**: High-impact, low-effort improvements to daily workflow
+
+| Task | Owner | Deliverable |
+|------|-------|-------------|
+| A.2.4 Critical path highlighter | Visualizer | Waterfall shows longest dependency chain |
+| A.4.1 Inline tool result preview | Visualizer | Expandable tool cards in chat |
+| A.4.2 Chat↔trace back-links | Visualizer | Bidirectional navigation |
+| B.1.3 Batch session export | API | `/api/sessions/export` endpoint |
+| B.2.1 GitHub PR linking | Backend | Auto-link via commit message parsing |
+| C.3.3 Security gate (SAST) | Backend | Semgrep integration in `quality.py` |
+| C.3.5 Accessibility gate | Backend | axe-core in `quality.py` |
+| D.1.1 `smidja init --interactive` | Backend | Guided setup wizard |
+| D.1.2 `smidja doctor` | Backend | Full preflight validation |
+
+### Phase 2: Core Enhancements (Weeks 3-6)
+**Goal**: Substantial new capabilities for power users
+
+| Task | Owner | Deliverable |
+|------|-------|-------------|
+| A.1.1 Presence indicators | Visualizer | WebSocket presence in SessionTrace |
+| A.1.2 Shared steer annotations | Visualizer | Attributed steer messages |
+| A.2.1 Comparative trace diff | Visualizer | Side-by-side waterfall view |
+| A.2.2 Token flow Sankey | Visualizer | Interactive token flow diagram |
+| A.2.3 Failure pattern miner | Visualizer + Backend | Clustered failures with fix suggestions |
+| A.3.1 Memory graph explorer | Visualizer | Kaia engram entity graph |
+| A.5.1 Visual roster editor | Visualizer | Drag-drop agent config UI |
+| A.5.2 Model tier picker | Visualizer | Visual model catalog |
+| B.1.1 WebSocket event stream | Visualizer + API | Replace 500ms polling |
+| B.1.4 Run comparison API | API | Structured diff endpoint |
+| C.1.1 Security audit smidja | Backend | New `smidja_security_audit.py` |
+| C.1.2 Perf optimize smidja | Backend | New `smidja_perf_optimize.py` |
+| C.1.5 Incident response smidja | Backend | New `smidja_incident_response.py` |
+| C.2.4 Reasoning budget control | Backend | Per-phase token/$$ ceilings |
+| C.3.1 Property-based test gate | Backend | Hypothesis/quickcheck integration |
+| C.3.4 Perf regression gate | Backend | Benchmark comparison gate |
+| C.4.1 Semantic context compression | Backend | LLM summarization of handoffs |
+| C.5.3 Event-driven smithies | Backend | `process-event-sources` integration |
+| D.2.1 `smidja replay` | Backend | Deterministic session replay |
+| D.2.2 `smidja diagnose` | Backend | Auto failure analysis |
+| E.1.1 Docker images | Infra | Published OCI images |
+| E.1.3 GitHub App | Infra | Installable PR integration |
+
+### Phase 3: Platform Maturity (Weeks 7-12)
+**Goal**: Enterprise readiness, extensibility, scale
+
+| Task | Owner | Deliverable |
+|------|-------|-------------|
+| A.1.3 Session handoff | Visualizer | Take over paused runs |
+| A.1.4 Comment threads | Visualizer | Phase-pinned discussions |
+| A.2.5 Phase duration heatmap | Visualizer | Calendar view |
+| A.3.2 Cross-project memory | Visualizer | Multi-repo Kaia query |
+| A.5.3 Smíðja chain builder | Visualizer | Visual pipeline editor |
+| A.6.1-6.3 Mobile + a11y | Visualizer | Responsive, WCAG AAA, PWA |
+| B.1.2 GraphQL endpoint | API | Flexible query layer |
+| B.2.2 Jira/Linear sync | Backend | Bidirectional ticket sync |
+| C.1.3 Migration smidja | Backend | `smidja_migration.py` |
+| C.1.4 Dependency update smidja | Backend | `smidja_dependency_update.py` |
+| C.1.5 Feature flag rollout | Backend | `smidja_feature_flag_rollout.py` |
+| C.2.1 Multi-modal agents | Backend | Vision input support |
+| C.2.2 Agent skill marketplace | Backend | Pluggable capability packages |
+| C.4.2 Cross-repo context | Backend | Shared handoff in monorepo |
+| C.4.3 Context versioning | Backend | Git-like handoff history |
+| C.4.4 Typed checkpoints | Backend | `Checkpoint` envelope type |
+| C.5.1 Parallel phase execution | Backend | DAG scheduler |
+| C.5.2 Cron scheduled smithies | Backend | `smidja schedule` command |
+| C.5.4 Smíðja composition | Backend | Chain smidja scripts |
+| D.1.3 Config docs generator | Backend | Auto-generated schema docs |
+| D.1.4 Roster inheritance | Backend | YAML `extends:` support |
+| D.2.3 Live agent REPL | Backend | Interactive agent shell |
+| D.3.1 Smíðja contract tests | Backend | Golden run test suite |
+| D.3.2 Agent prompt regression | Backend | Model drift detection |
+| E.1.2 K8s operator | Infra | `SmidjaRun` CRD |
+| E.1.4 OTel export | Infra | Tempo/Prometheus/Loki |
+| E.2.1 Team workspaces | Infra | Multi-tenant isolation |
+| E.2.2 RBAC | Infra | Role-based access |
+| E.2.3 Audit log | Infra | Immutable operation log |
+
+---
+
+## Technical Considerations
+
+### Visualizer Tech Debt to Address
+1. **Polling → WebSocket**: Current 500ms polling is simple but scales poorly; SSE/WebSocket for live updates
+2. **State management**: Migrate from composables to Pinia for devtools + persistence
+3. **Bundle size**: Code-split views (Memory, Decisions, Stats, Chat) — lazy load
+4. **Type safety**: Strict TypeScript + `vue-tsc --noEmit` in CI (already configured)
+5. **Test coverage**: Add Vitest unit tests for components + Playwright e2e for critical flows
+
+### Backend Tech Debt to Address
+1. **Plugin architecture**: Smíðja modules should be discoverable plugins, not hardcoded imports
+2. **Async execution**: `agent_pi.run` is synchronous; move to `asyncio` for parallel phases
+3. **Config hot-reload**: Watch `smidja.config.yaml` for changes without restart
+4. **Structured logging**: Replace `console.py` Rich output with structured JSON logs + OTel
+5. **Migration system**: Versioned DB migrations for `smidja.db` schema changes
+
+### Data Model Extensions Needed
+```python
+# New types for enhancement support
+class Checkpoint(EnvelopeBase):        # C.4.4
+    phase: str
+    summary: str
+    artifacts: list[str]
+    context_summary: str
+
+class MetricPoint(BaseModel):          # B.2.4
+    name: str
+    value: float
+    timestamp: str
+    tags: dict[str, str]
+
+class TeamWorkspace(BaseModel):        # E.2.1
+    id: str
+    name: str
+    smidja_config: str
+    visualizer_url: str
+    kaia_bridge_url: str
+    members: list[str]
+    rbac: dict[str, list[str]]         # role → permissions
+```
+
+---
+
+## Risk Assessment
+
+| Risk | Likelihood | Impact | Mitigation |
+|------|------------|--------|------------|
+| WebSocket complexity breaks polling fallback | Medium | High | Keep polling as fallback; feature flag WebSocket |
+| Multi-modal agents need new harness support | High | Medium | Start with Pi vision extensions; opencode later |
+| Parallel phases break context handoff assumptions | Medium | High | Design DAG with explicit data dependencies |
+| Kaia memory bridge becomes bottleneck | Medium | Medium | Add caching layer; async proxy |
+| Docker images bloat with all model deps | Low | Medium | Multi-stage builds; optional model layers |
+| GitHub App permissions scope creep | Medium | High | Minimal permissions; user-granted per repo |
+
+---
+
+## Success Metrics
+
+| Metric | Baseline | Target (6mo) | Target (12mo) |
+|--------|----------|--------------|---------------|
+| Mean time to detect failure | ~5 min (polling) | <30 sec (WebSocket) | <10 sec |
+| Smíðja script authoring time | ~2 hours | <30 min (visual builder) | <15 min |
+| Cross-run failure correlation | Manual | Auto-clustered | Auto-fix suggested |
+| Agent context window utilization | Unknown | Tracked + visualized | Optimized via compression |
+| Smíðja adoption (repos stamped) | ~5 | 25 | 100+ |
+| Visualizer daily active users | ~3 | 15 | 50+ |
+
+---
+
+## Appendix: Quick Reference — Current Commands
+
+```bash
+# Visualizer
+just ui                    # Start visualizer (API :4600, UI :4601)
+bun run server/index.ts    # API only
+bun run dev                # Vite dev server (proxies /api)
+
+# Smíðja
+uv run smidja/smidja_simple_sdlc.py "add health endpoint"
+uv run smidja/smidja_orchestrate.py "refactor auth" --smidja-id a1b2c3d4
+smidja doctor             # Preflight check
+smidja team list          # Show rosters
+smidja mission T1         # Bench missions
+
+# Kaia Memory
+just kaia                  # CLI memory interface
+# UI: http://localhost:4601/#/memory
+```
+
+---
+
+## Appendix: Key Files to Modify per Enhancement
+
+| Enhancement Area | Primary Files |
+|------------------|---------------|
+| Visualizer views | `apps/visualizer/src/components/*.vue` |
+| Visualizer API | `apps/visualizer/server/index.ts`, `server/db.ts` |
+| Visualizer types | `apps/visualizer/shared/types.ts`, `src/lib/types.ts` |
+| Smíðja chains | `templates/smidja/smidja_*.py` |
+| Smíðja modules | `templates/smidja/smidja_modules/*.py` |
+| Data contracts | `templates/smidja/smidja_modules/data_types.py` |
+| Agent prompts | `templates/prompt_engineering/<agent>/system.md`, `user.md` |
+| Config schema | `templates/smidja.config.yaml`, `references/config.md` |
+| Observability spec | `references/observability.md` |
+| Handoff protocol | `references/handoff.md` |
+
+---
+
+## Next Steps
+
+1. **Allfather reviews and prioritizes** — Select Phase 1 items for immediate sprint
+2. **Create implementation tickets** — Use `ticket-create` with measurable goals
+3. **Assign workstreams** — Visualizer vs Backend vs Infra tracks
+4. **Weekly sync** — Track progress against roadmap
+5. **Retrospective at Phase 1 end** — Adjust priorities based on learnings
+
+---
+
+*This plan is a living document. Update as enhancements are completed, new needs emerge, or priorities shift.*

package/apps/smidja-factory/references/config.md

@@ -0,0 +1,202 @@
+# Config Reference
+
+The full `smidja.config.yaml` spec: every field, how defaults merge, and how model / thinking / tools / extensions map onto the coding agent.
+
+It lives at **`smidja/smidja_smidja_config/smidja.config.yaml`** — the default path every `smidja_*.py` and the justfile resolve, and where `install.py` / `make_config.py` stamp it. Pass `--config <path>` to any smidja (or set `SMIDJA_CONFIG` for the justfile) to run against a different roster.
+
+## Shape
+
+```yaml
+defaults:
+  coding_agent: pi
+  model: google/gemini-3.6-flash        # ALWAYS provider/model-id
+  thinking: medium
+  harness_engineering: []
+  tools: [read, bash, edit, write, grep, find, ls]
+  data_dir: smidja/smidja_data
+
+observability:
+  db: smidja/smidja_data/smidja.db
+  poll_ms: 500
+
+agents:
+  - name: planner
+    coding_agent: pi
+    model: google/gemini-3.6-flash        # ALWAYS provider/model-id
+    thinking: high
+    color: "#a78bfa"
+    purpose: Turn a request into a plan the builder can implement without asking questions.
+    prompt_engineering:
+      system: smidja/smidja_data/prompt_engineering/planner/system.md
+      user: smidja/smidja_data/prompt_engineering/planner/user.md
+    harness_engineering:
+      - json-enforcer
+    tools:
+      - read
+      - bash
+```
+
+## Fields
+
+### `defaults`
+
+| Field | Type | Meaning |
+|---|---|---|
+| `coding_agent` | `pi` \| `claude_code` | Which interface runs the agent. **v1 implements `pi` only**; `claude_code` is specced and stubbed in `agent_cc.py`, landing in v2. |
+| `model` | string | Model id. For Pi, any id registered in `~/.pi/agent/models.json`. Default `gemini-3.6-flash`. |
+| `thinking` | enum | Reasoning effort — see below. Default `medium`. |
+| `color` | hex string | Lane color for every agent that does not set its own. Default empty — the visualizer falls back to its own palette. |
+| `harness_engineering` | list[string] | Coding-agent extensions. Pi: extension names. Claude Code: reserved (MCP, hooks). |
+| `tools` | list[string] | Roster-wide tool allowlist. Every agent that omits its own `tools` inherits this. Unset = all tools usable. |
+| `protected_files` | list[string] | Paths **no** agent may modify unless it names them in its own `writes`. Default: `smidja/smidja_modules/`, `smidja/smidja_smidja_config/`, `smidja/smidja_*.py` — an agent must not be able to edit the machinery that decides whether its work passed. |
+| `data_dir` | path | Runtime home. Sessions land at `{data_dir}/sessions/{smidja_id}/{agent_name}/`. Default `smidja/smidja_data`. |
+
+### `observability`
+
+| Field | Type | Meaning |
+|---|---|---|
+| `db` | path | SQLite trace db. `tracer.py` writes it directly; the visualizer polls it. Default `smidja/smidja_data/smidja.db`. |
+| `poll_ms` | int | Visualizer live-poll cadence in ms. History uses the same queries, lazy-paged. Default `500`. |
+
+### `agents[]`
+
+| Field | Required | Meaning |
+|---|---|---|
+| `name` | yes | The identifier smidja scripts use. **smidja name agents, never models.** |
+| `purpose` | yes | One sentence: what this agent is for. Should match its `system.md` Purpose. |
+| `prompt_engineering.system` | yes | Path to the system prompt — who the agent is, its single purpose, its output contract. |
+| `prompt_engineering.user` | yes | Path to the default user prompt — the task template with `{{prompt}}`, `{{previous_envelope}}`, `{{context_handoff_dir}}`. |
+| `color` | no | Hex swatch (`"#a78bfa"`) for this agent's lane in the visualizer. Travels config → `agent_sessions.color` → `/api/sessions/:smidja_id`, and rides the `agent_start` event so a lane is colored while the agent is still running. Unset = the UI's fallback palette. |
+| `coding_agent`, `model`, `thinking`, `color`, `harness_engineering` | no | Override the corresponding `defaults` key. |
+| `tools` | no | Allowlist. **Omitting the key means all tools usable.** A capability list, not a boundary — see `writes`. |
+| `writes` | no | What this agent may modify **in the repo**, enforced after every call. Omitted = unrestricted (still barred from `protected_files`). `[]` = no repo writes at all. A list = only those paths: a trailing `/` is a directory prefix, `*` matches within one path segment, `**` crosses segments, anything else is an exact path. Naming a `protected_files` path here is what unlocks it. **The session runtime under `data_dir` is always writable** — `writes: []` means read-only with respect to the repo, not unable to write its own report. |
+
+Output types are deliberately absent: config defines who an agent *is*; the smidja call site defines how it's *used*. One agent serves many calls — same system prompt, different user prompt + output type per call.
+
+## Defaults merging
+
+`agents.py` merges each entry **over** `defaults`, key by key. An entry states only what differs; anything unset inherits. `agents.validate(cfg, REQUIRED_AGENTS)` then confirms every name an smidja declares exists, resolves to a usable coding agent + model, and has both prompt files present on disk. Any miss fails the run immediately — **no agent is ever spawned against a half-valid config.**
+
+## Thinking levels
+
+Pi's reasoning-effort ladder, lowest to highest:
+
+```
+off | minimal | low | medium | high | xhigh | max
+```
+
+Mapped to Pi's reasoning effort control and honored when the model is registered with `reasoning: true` in `~/.pi/agent/models.json`. On a non-reasoning model the setting is inert — no error, no effect. Rough guidance: `high`/`xhigh` for planners and reviewers, `medium` for builders, `low` for mechanical read-and-report agents. (For Claude Code in v2, the same field maps to the thinking budget.)
+
+## Model resolution
+
+**Always write `model` as `provider/model-id`.** `agents.py` hands the string to the Pi interface, which resolves it against pi's merged catalog — `~/.pi/agent/models.json` plus pi's built-in providers. The same model is usually carried by more than one provider (`gemini-3.6-flash` lives under `google` *and* under `openrouter` as `google/gemini-3.6-flash`), and a bare id that matches several **raises at resolution**:
+
+```
+agent 'scout': model pattern 'gemini-3.6-flash' is ambiguous:
+  [('google', 'gemini-3.6-flash'), ('openrouter', 'google/gemini-3.6-flash'), ...]
+```
+
+That is `agents.validate()` doing its job — it fails before anything spawns rather than silently billing the wrong provider — but it means every agent in the roster inheriting that default is grounded until the pattern is qualified. Qualifying is the whole fix: `google/gemini-3.6-flash`, `openai/gpt-5.6-terra`, `fireworks/accounts/fireworks/models/kimi-k3`. The leading segment is matched against the provider list first, so the rest of the string can contain slashes.
+
+Other consequences worth knowing:
+
+- A model must be in the catalog before any agent can name it. An unknown id fails at resolution, before spawn. `pi --list-models` is the catalog the resolver actually reads.
+- **Ambiguity can appear without you touching the config.** Registering a new provider that carries a model you already use turns a formerly-fine bare pattern ambiguous. If a roster stops validating and nobody edited it, that is why.
+- Provider credentials come from the environment, not the config — the key that matches the provider you named (`GEMINI_API_KEY` for `google/...`, `OPENROUTER_API_KEY` for `openrouter/...`).
+- The resolved model is recorded per session in `agent_map.json` and mirrored into the `agent_sessions` table. **Changing an agent's model invalidates its session**: a joined run starts that agent fresh instead of resuming a context window built by a different model.
+
+## Tools
+
+`tools` maps to `pi --tools`. Pi's seven builtin tool names:
+
+| Tool | Purpose | Pi's own default |
+|---|---|---|
+| `read` | read file contents | on |
+| `bash` | execute bash commands | on |
+| `edit` | find/replace edits | on |
+| `write` | create/overwrite files | on |
+| `grep` | search file contents | **off** |
+| `find` | find files by glob | **off** |
+| `ls` | list directory contents | **off** |
+
+`grep`, `find`, and `ls` are off in bare Pi, so an agent that does not name them will shell out through `bash` to do the same work. The starter roster therefore sets `defaults.tools` to all seven and lets each agent narrow from there.
+
+**Resolution order:** an agent's own `tools` list wins; an agent that omits the key inherits `defaults.tools`; if neither is set, `tools` stays `None` and all tools are usable. An empty list is not "all tools" — it is a tool-less agent, and it will stall.
+
+## Write permissions — `writes` and `protected_files`
+
+`tools` cannot express a safety boundary, because two of the tools are general
+purpose. `bash` runs anything, including `git checkout`, which discards an
+engineer's uncommitted work; `write` reaches any path, not only the one report
+file an agent was granted it for. So "this agent changes nothing" is a claim a
+tool list can state but never keep.
+
+`smidja_modules/permissions.py` keeps it, the same way every other claim in this
+system is kept — after the fact, against the repo. Before an agent's first
+prompt the working tree's change-set is fingerprinted; after its last send
+(including JSON retries and gate corrections) it is fingerprinted again. Any
+path that appeared, vanished, or changed is attributed to that agent.
+
+Comparing change-sets rather than watching writes is deliberate: a path that was
+modified before the agent ran and is clean afterwards has been **reverted**, and
+a reversion is a modification. That is what catches `git checkout`.
+
+A breach is not a gate violation. Gates are for work an agent can be asked to
+redo; a write has already happened, so re-prompting fixes nothing. Instead:
+
+1. every unauthorized change the agent **introduced** is rolled back — tracked
+   files with `git checkout --`, untracked files by deletion;
+2. a path that was **already dirty** before the agent ran is left untouched. The
+   operator had uncommitted work there, and discarding it to tidy up would be
+   the same harm this module exists to prevent;
+3. the phase fails and names every path with what happened to it.
+
+```yaml
+defaults:
+  protected_files: [smidja/smidja_modules/, smidja/smidja_smidja_config/, "smidja/smidja_*.py"]
+
+agents:
+  - name: builder      # no `writes` key -> unrestricted, minus protected_files
+  - name: scout
+    writes: []         # no repo writes; its findings still land in context_handoff/
+  - name: planner
+    writes: [specs/]
+  - name: documenter
+    writes: [app_docs/, docs/, "**/*.md", "*.md"]
+```
+
+**The session runtime under `data_dir` is always writable, for every agent.**
+`context_handoff/` is how agents hand work to each other, and each agent's
+prompts, `raw_output.jsonl`, and `envelope.json` sit beside it. That grant comes
+from `data_dir` rather than from `.gitignore`: the runtime is normally ignored,
+so it never even appears in a snapshot, but an agent's ability to record its own
+work must not depend on a gitignore line someone can delete.
+
+Narrow by role, not by reflex. Anything that must produce a `context_handoff/` artifact needs `write`, or it will resort to a `bash` heredoc. Withhold `edit`/`write` only where the restriction *is* the guarantee — a reviewer that cannot edit cannot quietly fix what it was asked to report.
+
+### Extension tools must be named explicitly
+
+`pi --tools` is an allowlist over **built-in, extension, and custom tools alike** — not just builtins. So the moment an agent has a `tools` list at all (its own, or one inherited from `defaults`), any tool registered by its `harness_engineering` extensions is **excluded unless it appears in that list by name**.
+
+This fails quietly. The extension still loads, the run still succeeds, and the tool the extension exists to provide is simply never offered to the model — you find out by noticing the agent never called it.
+
+```yaml
+  - name: reviewer
+    harness_engineering:
+      - .pi/extensions/ast_query.ts     # registers tool: ast_query
+    tools:
+      - read
+      - grep
+      - find
+      - ls
+      - bash
+      - ast_query                       # REQUIRED — the extension's tool, named or lost
+```
+
+Rule: **every entry in `harness_engineering` that registers a tool must have that tool name added to the agent's `tools` list.** Adding an extension is therefore a two-line change, never one. The alternative is dropping the `tools` key *and* leaving `defaults.tools` unset so the agent resolves to `None` (all tools) — but with a roster-wide `defaults.tools` in place, that escape hatch is closed; naming the tool is the only path.
+
+## Harness engineering
+
+`harness_engineering` entries are pi extension **file paths**, passed through as `pi -e <path>`, one flag per entry, scoped to that agent only. This is where per-agent harness changes live — e.g. an output-tightening extension for an agent that keeps wrapping its envelope in prose. The starter roster ships with none. On Claude Code the field is reserved for MCP config and hooks in v2.
+
+**If the extension registers a tool, name that tool in the agent's `tools` list too** — `--tools` filters extension tools exactly like builtins, so an unnamed extension tool is silently unavailable no matter that the extension loaded fine. See [Extension tools must be named explicitly](#extension-tools-must-be-named-explicitly) above. Extensions that only shape output or add flags (no tool registration) need no `tools` change.