@r1z4x/tezgah 1.0.0

package/docs/layers.md

@@ -0,0 +1,189 @@
+# Layers
+
+This page answers one question: which layer is tezgah, and where does each
+neighbouring layer end. It is written for anyone about to write a claim of the
+form "apply X to the harness", and for a reader who has met that word in three
+different senses here. [architecture](architecture.md) says what tezgah
+contains, [glossary](glossary.md) says what a word means; this page draws the
+border around the layer and names what is on the other side of it.
+
+The four layers are listed bottom-up. Each one gets what it owns, what it must
+not own, and where this repository touches it.
+
+## Model and MCP transport
+
+Owns the wire: what the model is asked, and how a tool call reaches a server and
+its result comes back. MCP is a transport in that sense - it carries a call and
+a result and holds no rule.
+
+Must not own policy, provenance or evidence. A transport cannot refuse a call,
+and it writes no ledger [row](glossary.md#row).
+
+Here tezgah only *configures* servers and classifies a call by its name: the
+app-analysis servers are rendered into each host's own config from one spec
+(`hooks/tezgah_apps.py:1-5`), Codex's rows are appended by `ensure_toml_mcp`
+(`bin/tezgah-setup::ensure_toml_mcp`), and a tool name starting `mcp__` is classified
+`mcp` rather than trusted (`hooks/tezgah_integrity.py::MCP_TOOL`).
+
+## Agent framework
+
+Owns the building blocks a runtime is composed from: the chat pipeline, context
+providers, middleware and tool approval, compaction, bounded looping, telemetry.
+Microsoft's Agent Framework describes a `HarnessAgent` as composed of exactly
+those and notes that it remains a normal framework agent (Microsoft Agent
+Framework, `concepts/harness`).
+
+Must not own the user's rules or the record of the turn. A framework supplies
+the extension points; what a session is held to is not its subject.
+
+Here tezgah plugs into those points and never supplies one: omp's extension is
+"only the bridge" (`hosts/omp/tezgah-hook.ts.in:5-7`), opencode's plugin adds
+what its generated `instructions` files cannot carry
+(`hosts/opencode/plugins/tezgah.js:3-12`), and Codex's hook maps the host's
+events and tool names onto the shared vocabulary (`hosts/codex/hook.py::EVENTS`, `hosts/codex/hook.py::GATE_TOOLS`).
+
+## Agent host runtime
+
+Owns the loop: plan, call a tool, read the result, edit a file, report back -
+one job per session, with its own transcript, tool set and context window. The
+UHP spec states it the same way - "a harness is a complete agent runtime ...
+Codex, Claude Code and Hermes are harnesses" - with a task or job as the unit of
+exchange and a harness-to-application interface shaped like OpenAI Responses
+(the UHP spec, `github.com/HarnessRouter/harnessrouter`, file
+`protocol/README.md`).
+
+Must not own the user's rules, the evidence of the turn or the refusal. A
+runtime runs what the model asks; nothing inside it stops the model, which is
+the whole reason tezgah exists ([architecture](architecture.md)).
+
+Here the six hosts are the runtimes, one config dir per host
+(`hooks/tezgah_paths.py::HOST_DIRS`) with one presence test (`host_installed`
+`hooks/tezgah_paths.py::host_installed`), and the adapter under `hosts/<name>/` holds the
+host's envelope and nothing else ([hosts](hosts.md)). The README's own intro
+calls dsh a harness (`README.md:27-28`) - the literature's sense, applied to a
+host.
+
+## tezgah's policy and evidence layer
+
+Owns the rule text (`hooks/tezgah_policy.py::CORE`), the per-turn reminder
+(`hooks/tezgah_policy.py::PROMPT_REMINDER`), the refusal before a call
+(`hooks/tezgah_gate.py::decision`), the record after one (`note()`
+`hooks/tezgah_integrity.py::note`), the end-of-turn verdict (`stop_reason()`
+`hooks/tezgah_integrity.py::stop_reason`), the pre-write snapshots (`capture()`
+`hooks/tezgah_snapshot.py::capture`), the status marks, and the per-repo plans,
+lessons and research lines.
+
+Must not own the loop, the model, the tools or the network. tezgah never runs
+the work and never rolls back on its own - rollback is the explicit
+`bin/tezgah-rollback` - and the subagents it defines are rendered into a host's
+own agent surface for that host to run (`hooks/tezgah_agents.py:2-22`).
+
+## The boundary, one term at a time
+
+The test in each row is the one arXiv 2606.10106, "What makes a harness a
+harness", uses: the paper states necessary and sufficient conditions for a
+system to be an agent harness and operates them as an inclusion/exclusion test,
+bounding the term against agent framework, SDK, IDE plugin, eval harness and
+orchestrator. It also records why the word is ambiguous: it is used polysemously
+for a whole product, for an eval scaffold, and for things that are really
+frameworks.
+
+| Term | The test it must pass to be this layer | Where this repository touches it |
+|---|---|---|
+| host | does it run the loop for one session, with its own transcript and tool set? Then it is the runtime the literature calls a harness | six adapters under `hosts/<name>/`, no loop ([hosts](hosts.md)) |
+| agent framework | is it composed *from* elsewhere, or does it compose? A framework is composed from building blocks and stays a framework agent | tezgah plugs into its extension points, supplies none |
+| SDK | does it leave the iteration to its caller? A client library gives a call and stops | tezgah ships no client; it reads a host's event JSON on stdin (`hosts/codex/hook.py:1-2`) |
+| IDE plugin | is it a guest inside another program's surface? It draws and forwards and owns no task runtime | `hosts/opencode/plugins/tezgah.js:1-12` is one, and is a plugin |
+| eval harness | does it produce a score over a task set? It measures and enforces nothing | an eval scaffold is vendored as the `lm-evaluation-harness` skill body (`skills/ai-research/11-evaluation/lm-evaluation-harness/SKILL.md:11`), and this project's measurements live apart from the layer (`docs/README.md:88-89`) |
+| orchestrator | does it split one request across many runs and merge them? A harness runs one job to a report | the `harness` skill's route table (`skills/harness/SKILL.md:2-8`) and the graph workflows (`hooks/tezgah_policy.py::WORKFLOWS`) |
+| MCP | does it only carry a call and its result? Then it is transport, not a layer with policy or evidence | the section below |
+
+## Why tezgah is not an agent harness
+
+Under that test tezgah is not one: it holds no loop. It subscribes to a host's
+events (`hooks/hooks.json:2-32`), is handed the injected text at the host's
+session-start and prompt points (`hooks/tezgah_context.py::context_for`),
+refuses a call before the host runs it (`hooks/projects-pretooluse.py::main`) and
+answers in the host's own output envelope. It has no model, no tool set and no
+context window of its own.
+
+The same boundary is drawn from the other end of the wire by the trainer's side
+of the field. In what Agent Lightning v1.0 calls *harnessed agentic RL*, "the
+harness, rather than the training engine, owns the environment interaction loop,
+while the trainer observes only sequences of LLM request-response pairs" (arXiv
+2608.17528), and the same work names the model API as the one interface
+guaranteed to exist outside a harness. tezgah stands on that guarantee at a
+different seam - the host's own event envelope, the one interface guaranteed to
+exist outside the host's loop - which is why the two are one boundary seen from
+two sides, and why a trainer and a policy layer can describe the same system
+without either owning it.
+
+What it is: the policy and evidence layer a host loads. The rules travel as text
+into the host's context, the gate stands in front of the host's tool calls, and
+the ledger records what the host ran - one text, one gate, one ledger, which is
+[architecture](architecture.md)'s whole idea. None of this is a claim about
+quality: a policy layer is judged by what it refuses and what it can prove, not
+by solving the task.
+
+## Three senses of one word
+
+| Sense | What it names | Where it is defined |
+|---|---|---|
+| tezgah's own | the wrapper around a host - the injected contract, the per-turn text still travelling in the `<harness-reminder>` envelope (`hooks/tezgah_policy.py::PROMPT_REMINDER`), the gate and the ledger | [glossary](glossary.md#harness) |
+| the literature's | the host itself: a "complete agent runtime", which the README's intro also calls a harness for dsh (`README.md:27-28`) | [hosts](hosts.md) |
+| the skill's | multi-agent harness selection: which graph workflow or subagent set runs a task too wide for one window (`skills/harness/SKILL.md:2-8`, [skills](skills.md)) | [glossary](glossary.md#harness-skill) |
+
+A claim that says "apply X to the harness" is therefore unreadable until it says
+which. Write **host** for the runtime, **the harness skill** for multi-agent
+selection, and **tezgah's layer** (or "the policy and evidence layer") for this
+one.
+
+What not saying it costs is measured, and not small. The same model moved 13
+points across three harness configurations - the second trimming verbose tools
+and adding context compression and retries, the third adding self-checks, drift
+checks and rollback - while swapping the model inside one fixed harness moved
+the score 2.5-5; and a model moved out of the harness it was trained in shows a
+*degraded resolve rate*, or a *catastrophic format failure* whose output the new
+harness cannot parse at all. Those are a third party's measurements, reported in
+the FineEnvs multi-harness RL write-up (which cites Zhang et al. 2026 for the
+first pair and the Orchard paper for the second); this repository has run
+neither. They are here for one reason: a score that does not name the host it
+came from is not a number a reader can compare, which is the rule above applied
+to a benchmark instead of to a sentence.
+
+## MCP: the gate sees the call, not its contents
+
+This is the checkable consequence of the transport layer here. A hook is handed
+the call, so that is all the classification can use (`untrusted_source`
+`hooks/tezgah_integrity.py::untrusted_source`): an MCP answer enters the ledger as an
+untrusted channel, the channel being `mcp` (`untrusted_source`
+`hooks/tezgah_integrity.py::untrusted_source`), and an answer that is no step of work
+of its own earns a row of kind `external` (`note_tool`
+`hooks/tezgah_integrity.py::note_tool`). The compensating control is the taint
+notice: the first effect in a turn that has read an untrusted channel carries
+it (`hooks/tezgah_untrusted.py:2-22`); nothing refuses that effect since the
+sink rule was removed ([gate](gate.md)).
+
+The structural limit is that the write tools and the shell tools are disjoint
+tuples (`hooks/tezgah_integrity.py::WRITE_TOOLS`, `BASH_TOOLS`
+`hooks/tezgah_integrity.py::BASH_TOOLS`), so a rule that guards one does not guard
+the other - the class named C26 in an internal research line (not
+published). No content
+of an MCP answer is inspected, and nothing beyond this is built for MCP.
+
+## What this page does not change
+
+Documentation only: no rule label, no `CORE` paragraph and no injected byte
+changes, so the cost table in `README.md` stays true. Definitions stay in the
+[glossary](glossary.md) - this page draws the boundary and adds no term of its
+own.
+
+## Source of truth
+
+- `hooks/tezgah_policy.py`, `hooks/tezgah_context.py`, `hooks/tezgah_gate.py`,
+  `hooks/tezgah_integrity.py`, `hooks/tezgah_snapshot.py`, `hooks/tezgah_paths.py`
+- `hooks/hooks.json`, `hooks/projects-pretooluse.py`, `hooks/tezgah_apps.py`,
+  `hooks/tezgah_agents.py`, `bin/tezgah-setup`
+- `hosts/<name>/`, `skills/harness/SKILL.md`
+- `docs/glossary.md`, `docs/hosts.md`, `README.md` (the host list, the cost
+  section)

package/docs/models.md

@@ -0,0 +1,160 @@
+# Models: which model runs which delegated task, and how the table stays current
+
+The model table and the tier router. Read this page before you spawn a worker, when
+a host runs an agent on a model you did not expect, when you change a model in the
+table, or when `tezgah-route --check` says the table is stale. It is a capability,
+not a rule: the ORCHESTRATE paragraph tells the router to use it, and nothing here
+refuses a call.
+
+## The shape
+
+The main thread keeps one model for the whole session. The prompt cache is per
+model, so a switch re-reads the whole cached prefix at write price; measured on this
+repository's own omp logs (2026-08-31..2026-09-30), a main-thread Opus 5.5 turn read
+a mean of about 435k cached tokens, so one Opus -> Sonnet -> Opus switch pair costs
+about $3.26 against about $0.02 saved per explore turn. Routing therefore happens
+where it is free: at the subagent a task is handed to.
+
+Every generated agent has a slot (`AGENT_SLOT`, `hooks/tezgah_models.py::AGENT_SLOT`)
+and every slot a model per family (`SLOTS`, `hooks/tezgah_models.py::SLOTS`). The
+three tier workers - `tezgah-cheap`, `tezgah-standard`, `tezgah-frontier` - are
+generated in every tezgah root because routing needs no capability (`ROLES`,
+`hooks/tezgah_agents.py::ROLES`; their brief, `_worker_body`,
+`hooks/tezgah_agents.py::_worker_body`). A cheaper worker that meets work above its tier
+answers `ESCALATE: <why>`, and the router restarts the task on `tezgah-frontier`
+with the original brief rather than handing the failed trajectory up: continuing a
+cheap trajectory on a frontier model was the most expensive option measured
+(arXiv 2608.24358, Table 1: 1.61 cost units against 0.72 for frontier-only).
+
+## The table (snapshot 2026-09-30)
+
+| slot | agents | anthropic | zai (omp's own provider) | openai | any (OpenRouter id) |
+|---|---|---|---|---|
+| cheap | tezgah-cheap | Opus 5.5 @low | `zai/glm-5.3-flash` | GPT-6.1 Sol @low | GLM-5.3 Flash |
+| explore | omp's bundled `scout` | Opus 5.5 @medium | `zai/glm-5.3-flash` | GPT-6.1 Sol @low | DeepSeek V4.1 Flash |
+| standard | tezgah-standard | Opus 5.5 @medium | `zai/glm-5.3` | GPT-6.1 Sol @high | GPT-6.1 Sol @high |
+| frontier | tezgah-frontier, tezgah-reviewer | Opus 5.5 @high | `zai/glm-5.3` | GPT-6 Astra @high | Opus 5.5 @high |
+
+Why these, briefly. Inside the Anthropic family the lever is effort, not model:
+Opus 5.5 and Sonnet 5.5 read cache at the same $0.20 per 1M tokens, and on
+Artificial Analysis's per-effort runs (v4.3, read 2026-09-30) Sonnet 5.5 is not
+cheaper per task at an equal Terminal-Bench 4.0 score (@high 43.9 at $1.08, @xhigh
+57.1 at $2.74, against Opus 5.5 @high 56.6 at $1.82). Opus 5 lists at $5/$25 with an
+index of 50.8 against Opus 5.5's $4/$20 and 57.6, so it is never a cheaper tier.
+Across families the cheap tier is where the saving is: DeepSeek V4.1 Flash reads
+cache at $0.0029 per 1M tokens and scores 84.0 on AA's long-context reasoning
+against Opus 5.5's 84.7. Estimated on this repository's window, routing delegated
+work across families costs about 25-27% less than all-Opus subagents, and tier
+routing inside the Anthropic family at most 2.3% less; moving Opus subagents from
+@high to the table's efforts about 11% less (estimates on measured token mixes, not
+invoices).
+
+## Which host reads which column
+
+| Host | Family | Where the model lands |
+|---|---|---|
+| Claude Code | anthropic | the agent file's alias and `effort:` (`_model_lines`): an alias follows the provider and keeps the main session's variant, where a full id breaks on Bedrock, Vertex and a gateway (code.claude.com/docs/en/sub-agents) |
+| Codex | openai | `model` and `model_reasoning_effort` in `.codex/agents/*.toml` |
+| opencode | any | a selector `--refresh` found in `opencode models` for the slot's model under any configured provider (`resolve_opencode`); none found, no model line, so the agent inherits. The slot's effort rides the JSON agent config as `reasoningEffort` (opencode.ai/docs/agents) |
+| omp | anthropic when the session default is an Anthropic model, any otherwise (`omp_mode`); a non-Anthropic default is `any`, never a silent fall back to the Anthropic column | `task.agentModelOverrides`, which omp resolves before the agent or bundled model (omp's own docs: `omp://task-agent-discovery.md`, *Model and structured-output precedence*); every slot is written, the frontier row included, so no agent in the row runs on the session default; each override is a **cross-family fallback chain** (`omp_overrides`): the mode's family first, then every other family this machine holds a credential for (`funded_families` reads omp's auth store and the provider env keys; a family it cannot see is left out, never guessed in) - omp splits the chain into per-spawn patterns (omp's own docs: `omp://settings.md (its `retry.fallbackChains` section)`), so one provider's 429 falls through instead of killing the spawn, measured before this: 24 of 26 post-routing 429s hit a subagent's first request, all on one Opus account every slot shared; `apply_omp` also writes `modelRoles.plan` and `.slow` on that frontier row (same record, same ownership) so plan mode designs on it, and their fallbacks go to `retry.fallbackChains.<role>`, the only chain keys tezgah owns (`default` and provider-keyed chains stay the user's); `apply_omp` writes through the installer's own binary and agent dir, from HOME so a repository's `.omp/config.yml` never leaks machine-wide, and touches only the entries tezgah wrote (recorded in `~/.config/tezgah/models.json` as `omp_written`; a machine written before that record is adopted by its value, `_ours_by_shape`) |
+| dsh | - | none: no model setting was found to write |
+
+The orchestrator agent keeps `model: inherit`: it is the main thread's own agent.
+
+| Cursor | - | reads `.claude/agents/` today; Cursor wants its own model ids with the effort **inside** the id (`claude-opus-5[effort=high]`, cursor.com/docs/subagents), and no id this table names was verified against Cursor's models page - so a `.cursor/agents/` rendering that pins a model is not generated yet |
+
+omp's bundled agents are routed through the same record (omp's own docs: `omp://task-agent-discovery.md`, *Bundled agents*): `scout` on the explore slot, `sonic` on the cheap slot, `task` on the standard slot, and `reviewer`/`security-reviewer` left to inherit, because a review is frontier work. A user's own value for any of them wins (see the ownership rule above). Those two are the gap this leaves: the record carries no entry for them, so a session on a cheap model reviews on that model - routing them is one more `BUNDLED` entry, not decided here.
+
+## What is not routed yet
+
+- **Claude Code's built-in subagents** (Explore, general-purpose, Plan) run on the session model. `CLAUDE_CODE_SUBAGENT_MODEL` and a settings `env` block exist but carry no effort, and whether a project agent with a built-in's name replaces it is not stated where it can be cited - so nothing is generated on a guess. What would close it: a documented per-agent effort for built-ins, or a confirmed name-collision rule.
+- **Codex's default subagent** (spawned with no agent file): only `.codex/agents/*.toml` carries a model today. What would close it: a documented `[agents]` default settable from a repo-scoped file tezgah already writes.
+- **opencode's built-in agents**: the injected JSON could carry `model`/`reasoningEffort` for them, but the merge semantics for a built-in entry are not documented where they can be cited, and a wrong merge would replace a built-in prompt instead of adding a model.
+- **dsh**: the model is a plugin config, not an agent field. The composed profile carries `agent-default-model` with `config: {provider: deepseek-official, model: deepseek-flash}` (`dsh --profile web --dump-config`, 2026-09-30) and tezgah's managed patch block already patches plugin config by id (its `llm-pi-ai` row), so a `- id: agent-default-model` row would set the profile's model. Not written, because that knob is one model for the whole session - no `subagent*` entry in the composed tree carries a model - and a single session-wide model is the main thread's setting, which this design deliberately leaves alone ([why the main thread keeps one model](#the-shape)). What would close it: a per-agent model in dsh's config surface.
+
+## The router
+
+`tezgah-route "<brief>"` prints the worker to spawn and why (`main`,
+`bin/tezgah-route::main`; `route`, `hooks/tezgah_models.py::route`). The order is
+fixed:
+
+1. A brief naming stored data, a persistence or schema change, a migration,
+   credentials, a token or key shape, a private key, certificate, bearer token or
+   access token, a destructive data or history operation (dropping or truncating
+   a table, deleting rows, a production database, a force-push or history
+   rewrite), a privilege change, the gate or security goes to frontier by rule
+   (`OVERRIDE`, `hooks/tezgah_models.py::OVERRIDE`) - the class the judge
+   under-routed in its measurement, and the classes a wrong call cannot be
+   undone on. A 12-brief fixture pins it: 8 high-stakes briefs route to
+   frontier, 4 near-miss controls do not (`tests/test_models.py`).
+2. Otherwise the brief - redacted with the ledger's own reader
+   (`redact`, `hooks/tezgah_integrity.py::redact`) - goes to Jev as one Choice over three tiers (`TIER_QUESTION`,
+   `hooks/tezgah_models.py::TIER_QUESTION`). Measured on 40 English briefs labelled by
+   the same session that wrote the rubric (2026-09-30, twice): under-route 0.025,
+   accuracy 0.925 and 0.900, 392 ms median, about 656 input tokens per call; a
+   keyword rule on the same set under-routed 0.100.
+3. With no judgement (`judge-off`, no key, a failed call) `--phase` picks the tier
+   from the static table (`PHASE_TIER`, `hooks/tezgah_models.py::PHASE_TIER`), and
+   with no phase the middle tier is used.
+
+The brief leaves the machine for the judge, like every judgement ([judge](judge.md)).
+Every routed call writes one `route` row to the session's ledger. The row names the
+tier and the agent. `via` says what set the tier: `rule`, `jev`, `phase` or
+`default`. `static` is the static table's tier when the caller passed `--phase`.
+`model` names the worker's model on this host. `override` reads `user` when omp holds
+an entry the user owns, and `mode` when omp holds none and the session model runs.
+`judge` names the `provider/model` that answered a `via=jev` route, and `-`
+otherwise. `via` keeps the word `jev` even when the chat fallback answered,
+because the report groups history by it. `judge` is where the two differ.
+
+A routed worker runs in its own session, so its checks land in its own ledger. On omp
+and opencode the child's ledger opens with a `spawned` row that names its parent
+session. `tezgah-route --report` matches a route to a child by agent type first. On
+opencode the child's own title carries the type. On omp the parent's task call
+records which type each named child runs. Order is the fallback, and only when one
+child sits between each route and the next with no two spawns in one second. The
+report then reads each child's first check: `pass`, `fail`, `ran` (no outcome seen)
+or `none`. A route it cannot prove counts as `unjoined`. That covers a tie, an
+unrouted spawn, a nested subagent, and every host with no parent signal: Claude
+Code, Codex, Cursor and dsh. The parent's own next check never counts for the worker.
+One line per `via` and tier gives the first-attempt pass rate. The `jev-below-static`
+line counts Jev routes under the static tier and how many of them failed. That is the
+measurement that decides whether Jev routing becomes the default or the static table
+replaces it.
+
+The second-opinion panel (`bin/consult`) skips a member that shares the session's
+model family, not only its exact model. A family comes from a stem in the model name,
+so `deepseek-chat` and `deepseek/deepseek-flash` are one family. Bedrock's
+`us.anthropic.claude-sonnet-4` is Anthropic, and `o3` is OpenAI. EleutherAI's
+`gpt-j` is not OpenAI. `cli:claude` and `cli:codex` carry their vendor's family.
+`--use` naming the member asks it anyway.
+
+## Keeping it current
+
+`tezgah-route --refresh` re-reads OpenRouter's public model list, writes prices to
+`~/.config/tezgah/models.json`, flags a model that left the list or whose price
+moved against the snapshot (`SNAPSHOT`, `hooks/tezgah_models.py::SNAPSHOT`;
+`refresh`, `hooks/tezgah_models.py::refresh`), resolves opencode selectors, and
+re-applies omp's overrides. Scores are not re-read - they need a key - so
+`tezgah-route --check` reports the snapshot's age and exits 1 past 60 days or with
+a flag (`check`, `hooks/tezgah_models.py::check`); that is the moment to re-read
+the leaderboards and edit `SLOTS`. `tezgah-route --mode anthropic|zai|any|auto|off` pins
+omp's family instead of following the session default; `off` removes every entry
+and role the table wrote and writes nothing, which is the way back when the
+provider those selectors need has no budget - a measured case, not a
+hypothetical one. `tezgah-setup --install`
+writes omp's overrides and `--uninstall` removes only the `tezgah-*` entries; the
+omp status report carries a row for them.
+
+## Source of truth
+
+- `hooks/tezgah_models.py` - the table, the overlay, omp's overrides, refresh, the router.
+- `frontier_model(family)` / `cheap_model(family)` in `hooks/tezgah_models.py` - the
+  table's accessors: a caller that needs a strong or a cheap model names none of its
+  own, but asks for that row's `(model, effort)`, or `None` when the family has no
+  such row.
+- `bin/tezgah-route` - the CLI.
+- `hooks/tezgah_agents.py` - the tier workers and the per-host model lines.
+- `hooks/tezgah_policy.py` - the ORCHESTRATE paragraph that tells the router to use it.
+- `bin/tezgah-setup` - omp's overrides on install and uninstall, and the status row.
+- `tests/test_models.py`, `tests/test_agents.py` - the behaviour pinned.