pdl-taskmaster 2.5.1__tar.gz → 2.6.0__tar.gz

pdl_taskmaster-2.6.0/.gitignore

@@ -0,0 +1,19 @@
+.pytest_cache/
+__pycache__/
+*.egg-info/
+catalogue-runs/
+model-compare/
+.venv/
+venv/
+mlruns/
+runs/
+mlflow.db
+*.log
+.env
+dist/
+build/
+
+# acceptance-gate runner output (experiments/runner.py)
+gate-runs/
+site/
+experiments/adjudication_cache/

pdl_taskmaster-2.6.0/AGENTS.md

@@ -0,0 +1,69 @@
+# Agent Rules & Guidelines
+
+## Continuity (read first, every session and after every context compaction)
+- **`docs/plans/LEDGER.md` is the canonical record** of cross-cutting decisions and open items: each with its source (main chat, side chat, the user), status and link.
+- **Read it before starting work.** Then read the current PR's work list it points to (for example `docs/plans/pr2-worklist.md`).
+- **Update it in the same change** that settles, changes or completes an item. Mark items `done` as they land, citing the commit.
+- **A side-chat decision is not settled until it has a ledger row.**
+- **When two recorded decisions disagree:**
+  - a **minor** conflict: the side chat's decision wins; record it, noting the conflict in the row;
+  - a **major** one: mark the row `conflict` and ask the user.
+  Never pick silently.
+- **Within one turn, a built-in task list is fine for progress.** The ledger and the work list are what survive compaction. An external tracker may mirror them, never replace them.
+
+## Diagnosis and Verification Rule (CRITICAL / MANDATORY)
+- **Static first.** Diagnose from the code: trace the path and cite `file:line`. Read the artifacts existing runs already recorded (`call-trace.jsonl`, `events.jsonl`, `compiled-projection.json`, `model-response.txt`). If the code determines the answer, do not reproduce it live.
+- **Offline before live.** Verify host logic with the offline suite and recorded replays (`--worker recorded`). Controller transitions, exit codes, routing plumbing, grading, telemetry, docs and refactors need no API call.
+- **Live only when the answer depends on what code cannot determine:**
+  - what a model returns for a request, when no recorded output answers it;
+  - provider behaviour: schema acceptance, routing, stalls, token reporting, latency;
+  - a System 1 decision on an input it has not seen;
+  - a change to what the model or provider receives, checked once before merge (trigger below).
+- **The trigger for "receives".** Either of these:
+  - a recorded replay misses, because the rendered prompt's hash changed (`RecordedWorker` keys on the operation plus the prompt's SHA-256 and raises `ReplayMissError`);
+  - the provider-layer request changed (`providers/api_worker.py`: guidance text, effort, response format, caps, provider pinning).
+
+  Replay alone does not cover the second: those parts are added after the prompt is hashed.
+- **A static conclusion about runtime behaviour is a hypothesis.** Code can be read confidently and still not be what runs: environment variables, provider defaults, a stale install. A static claim about model or provider behaviour stays a hypothesis until a recorded artifact confirms it. Only when none exists does it justify a live run.
+- **A live run answers a stated question.** Before running, write what you expect and what would falsify it. Use the smallest run that answers it, in dev mode (`--dev`), from the repository root. No generic smoke runs.
+- **Label the evidence.** Every claim says how it was established: static (`file:line`), offline test, recorded artifact, or live run. Never report a static or offline result as "verified live". If a required live check can't run because `OPENROUTER_API_KEY` is unavailable, say so.
+
+## PDL Standard Adherence
+- Prompt Pseudocode MUST strictly conform to `PDL-01` through `PDL-08`.
+- NO invented field schemas (`TASK:`, `OUTPUT:`, `INCLUDE:`).
+- One operation per line (`PDL-02`), capitalized action verbs (`PDL-04`).
+- No drafting meta-rules or deferrals in the prompt or plan bodies (`PROMPT-01`, `PDL-08`).
+
+## Architecture & Verification Governance (ADR-0018, ADR-0019, ADR-0020)
+- **Pydantic SSOT (ADR-0018)**: NEVER use heuristic regex for domain detection, witness extraction, or wire verification. All parsing and validation MUST go through strict, schema-first Pydantic models with alias coercion (`OutputVerifier`, `wire_payloads.WitnessPayload`). No problem-specific checker ships in the harness plane.
+- **Autonomous Host Execution & First-Class Reasoning (GUARD-03)**:
+  - When Python code or solver scripts are present, the host sandbox automatically executes them to capture stdout witnesses. NEVER emit `REQUEST_INPUT` asking the user to run code.
+  - Analytical derivations, symbolic mathematics, word problems, and logical deductions are first-class deliverables (`GUARD-03`). The harness MUST NOT coerce symbolic or reasoning tasks into executable Python scripts or force models to fabricate concrete values for symbolic variables.
+- **System 1 Boundary Refusal (ADR-0020, GUARD-02)**: Out-of-bounds, network-dependent, or mathematically impossible tasks MUST be intercepted and refused fail-closed by System 1 in <2s conditioned on runtime environment variables (`PDLT_SANDBOX_NETWORK`, `PDLT_POLICY_SCOPE`); the knowledge cutoff (`PDLT_KNOWLEDGE_CUTOFF`) is injected into System 1's state, never matched by a year regex. NEVER hardcode prompt-specific benchmark tokens or entity names into production regex traps.
+- **Headless Automation & Exit Codes (ADR-0019)**:
+  - `0`: `CLOSED_SUCCESS` (deliverable verified).
+  - `0` also covers a published boundary refusal (`closure=REFUSED`, ADR-0019 amendment).
+  - `1`: `CLOSED_CANCELLED` / fail-closed error.
+  - `2`: `UNCONFIRMED_GATE` (execution stalled at review gate).
+  - `3`: `WAITING_INPUT` (legitimate pause awaiting external input).
+  - `4`: harness or provider error (`EXIT_HARNESS_ERROR`); never a protocol result.
+  - `5`: `CLOSED_UNVERIFIED`: a published result the host could not establish (a negative claim, a deliverable declaring requirements open, or a positive result established by an argument; ADR-0019 amendment, L116, ADR-0030).
+  - `130`: interrupted by the user.
+- **Catalogue Verification**: Before merging a PR that changes behaviour (what the model receives, routing, stage logic, verification or grading), run `python run_catalogue.py --fail-fast` from the repository root to check for regressions across all 16 categories. It is not a per-pass check. Never loosen `is_prompt_pass` or `graders.py` to obtain a green result.
+- **Runs live outside the repository**: catalogue and experiment runs are written to `$PDLT_CATALOGUE_RUNS`, else `PDLt-Catalogue-Archive` beside the checkout (`catalogue_runs.py`). Never commit run output.
+
+## Post-Implementation Anti-Overfitting & Integrity Gate (MANDATORY / SSOT)
+- **The Referee Invariant (GUARD-01, GUARD-04)**: The harness is strictly an objective protocol referee and governor—NEVER an AI task solver. The harness MUST NEVER:
+  - Inject prescriptive algorithmic advice, search methods (e.g. MRV, backtracking, DLX), or solution hints into prompts, plans, or `CARRIED_APPROACH_SOURCES`.
+  - Mechanically require specific algorithmic keywords in Plan Soundness gates.
+  - Scrape deliverable text with regex to synthesize witnesses.
+- **Feedback channel**: retry and lint feedback travels only through operator correction; `CARRIED_APPROACH_SOURCES` stays user-originated.
+- **Mandatory Anti-Overfitting Suite**: After ANY implementation round touching prompts, gates, checkers, or session engine, you MUST execute:
+  ```bash
+  pytest tests/test_harness_anti_overfitting.py
+  ```
+  Every test must pass with zero warnings, failures, or skips.
+- **Guardrail Cross-Check**: Any changes MUST strictly conform to `docs/guardrails/ANTI_OVERFITTING_AND_BENCHMARK_INTEGRITY.md` (`GUARD-01` through `GUARD-05`).
+- **Integrity Over Checkmarks**: **A failed benchmark is acceptable and diagnostic of genuine model capability boundaries (e.g. `REG-011`, `REG-012`), but a gamed benchmark is a critical integrity breach.** Never introduce harness crutches to force a green test.
+
+

pdl_taskmaster-2.6.0/ARCHITECTURE.md

@@ -0,0 +1,276 @@
+# PDL Taskmaster: Current Architecture
+
+**Describes:** `pdl-taskmaster` 2.6.0, the code in this repository.
+**Scope:** what exists and runs today. Where the project is heading is in [`TARGET_ARCHITECTURE.md`](TARGET_ARCHITECTURE.md); nothing in this document describes planned behaviour.
+**Normative authorities:** ADR-0001 through ADR-0022 ([`docs/adr/`](docs/adr/README.md)) and the guardrails in [`docs/guardrails/ANTI_OVERFITTING_AND_BENCHMARK_INTEGRITY.md`](docs/guardrails/ANTI_OVERFITTING_AND_BENCHMARK_INTEGRITY.md) (`GUARD-01` to `GUARD-05`). Where an ADR and the guardrails disagree, the guardrails win.
+
+---
+
+## 1. What PDL Taskmaster is
+
+PDL Taskmaster is a protocol referee and runtime governor for model-driven tasks. It exists because *the component carrying out a task cannot be the only one deciding what the task means*. Before anything runs, the model restates the request as short Prompt Pseudocode and then as a Response Plan; each is shown for review, and execution starts only after both are confirmed.
+
+Three rules shape the whole design:
+
+1. **Route the sandbox conditions, not the model.** The harness never grades plan quality or reasoning style, uses no rubric and no model self-grading. System 1 routes physical and operational conditions (environment, budgets, review intent); the review gates align the task with the human.
+2. **The referee invariant (`GUARD-01`, `GUARD-04`).** The harness is never a solver. It does not inject algorithmic advice, require algorithm keywords at review gates, or fabricate witnesses. A failed task is diagnostic signal; a gamed pass is an integrity breach.
+3. **Session-scoped execution sandbox.** One `ExecutionSandbox` is built per session and reused for every program run in it.
+
+### 1.1 Two planes
+
+| Plane | Location | Knows the benchmark? | Responsibility |
+|---|---|---|---|
+| **Harness** | `src/pdl_taskmaster/` | **Never** | Protocol governance, input containment, sandboxed execution, schema verification |
+| **Evaluation** | `run_catalogue.py`, `graders.py`, `prompts/`, `viewer/` | Yes | Drives headless catalogue runs, maps exit codes to verdicts, grades deliverables against `prompts/solutions/`, browses results |
+
+The harness never reads `prompts/` and contains no prompt IDs, fixture names or problem-class vocabulary; `tests/test_harness_anti_overfitting.py` scans for them. Only the harness is shipped in the wheel.
+
+---
+
+## 2. Components
+
+```
+src/pdl_taskmaster/
+├── api/             the library layer every interface uses: Session, TurnResult, workers,
+│                    run settings, closed sessions, sandboxes (docs/api.md)
+├── extensions/      versioned profiles (task type, verifier, provider), registries,
+│                    conformance kit (docs/extensions.md)
+├── host/            pdlt CLI (cli.py), REPL presentation (repl.py), PDLtHost turn API (app.py),
+│                    MCP server (mcp_server.py)
+├── runtime/         SessionEngine orchestrator, workspace (in-memory VFS), context compiler,
+│                    operation bridge (wire parsing), Pydantic wire models, Result IR, quarantine,
+│                    normative store (contract resolution)
+├── controller/      MechanicalController: the deterministic stage machine
+├── providers/       System 2 workers (api, codex, recorded), System 1 client and recipes
+├── verification/    output verifier, plan-soundness lint, error registry, ExecutionSandbox,
+│                    confinement backends (Landlock, Seatbelt, AppContainer, container)
+├── observation/     per-turn JSONL observation records
+└── contracts/       bundled normative contracts and standards (hash-pinned)
+```
+
+| Layer | Owns | Does not own |
+|---|---|---|
+| `api/` (`Session`) | The surface the REPL, headless runs and the MCP server share: sessions, review gates, results, exit status | Stage transitions |
+| `host/repl.py` | Terminal I/O, slash commands, paste handling, headless exit codes | Protocol state |
+| `host/app.py` (`PDLtHost`) | Process and session lifetime; one call per user turn (`handle`) | Stage transitions |
+| `runtime/session_engine.py` | Orchestrating operations, verification and sandbox runs | Terminal I/O (it never reads stdin or prints) |
+| `controller/mechanical_controller.py` | Every stage transition | Model calls |
+| `providers/` | One model call per request (`WorkerAdapter.call`) | Wire validity (the operation bridge decides) |
+
+### 2.1 Model tiers
+
+- **System 1** (`providers/sys1/`): a fast decision model (default `typesafe/jev-1.13` through OpenRouter's decisions endpoint) that returns calibrated label distributions, never text. Every recipe shares one confidence gate (`gating.py`): confidence $P \ge 0.85$, top-2 margin $\Delta p \ge 0.40$, normalized entropy $H(p) \le 0.35$.
+- **System 2** (`providers/api_worker.py`): a generative model (default `openai/gpt-oss-120b` through OpenRouter) that drafts the Prompt and Plan and executes. With no `--api-providers`, the provider order is Baseten, then Crusoe, with fallbacks allowed (`OPENROUTER_PROVIDER` and `OPENROUTER_PROVIDER_ORDER` override it); measured provider behaviour is in [`PROVIDERS.md`](PROVIDERS.md). Reasoning effort defaults to high before execution and low at `EXECUTE` (ADR-0022). `--api-base-url` points it at another OpenAI-compatible endpoint; provider routing options (`--api-providers`) are OpenRouter's.
+- **Other workers:** `codex` drives the Codex CLI as a System 2 worker; `recorded` replays fixtures for offline tests.
+
+---
+
+## 3. Protocol lifecycle
+
+### 3.1 Stages
+
+`MechanicalController` owns these stages; no worker output can move between them on its own.
+
+| Stage | Meaning |
+|---|---|
+| `PROMPT_REQUIRED` | A request was accepted; Prompt Pseudocode must be drafted |
+| `PROMPT_REVIEW` | Prompt Pseudocode is waiting for review |
+| `PLAN_REQUIRED` | The prompt is confirmed; a Response Plan must be drafted |
+| `PLAN_REVIEW` | The Response Plan is waiting for review |
+| `EXECUTION_READY` | Both artifacts are confirmed; execution may start |
+| `WAITING_INPUT` | Execution asked for required external input |
+| `OUTCOME_UNCERTAIN` | A session was restored while an execution was in flight; its outcome is unknown |
+| `CLOSED_SUCCESS` | A verified deliverable was published |
+| `CLOSED_CANCELLED` | Cancelled, or verification failed after the repairs allowed |
+
+Gates: no plan is drafted without a confirmed prompt, and nothing executes without a confirmed plan bound to that prompt. Empty input at a review gate re-prompts and is never taken as acceptance.
+
+### 3.2 Decision flow
+
+```mermaid
+flowchart TD
+    Ingest["User request"] --> S1_Activate{"S1: activation route"}
+    S1_Activate -- BLOCKED_BY_HIGHER_PRIORITY --> Refusal["Publish boundary refusal<br/>exit 0 (closure=REFUSED)"]
+    S1_Activate -- BYPASS / PROTOCOL_DISCUSSION --> Direct["Direct answer (no protocol instance)"]
+    S1_Activate -- "APPLY_PROTOCOL, or no usable S1 decision" --> S1_Class{"S1: problem class<br/>and execution profile"}
+    S1_Class --> DraftPrompt["S2: DRAFT_PROMPT"]
+    DraftPrompt --> PromptLint{"Grammar lint"}
+    PromptLint -- "violation (one redraft)" --> DraftPrompt
+    PromptLint -- clean --> PromptGate["PROMPT_REVIEW"]
+    PromptGate -- "/confirm or agreed" --> DraftPlan["S2: DRAFT_PLAN"]
+    PromptGate -- "/revise" --> DraftPrompt
+    PromptGate -- "/stop, /cancel" --> Cancelled["CLOSED_CANCELLED, exit 1"]
+    DraftPlan --> PlanLint{"Grammar lint"}
+    PlanLint -- "violation (one redraft)" --> DraftPlan
+    PlanLint -- clean --> PlanAdvance{"S1: plan advancement<br/>(PLAN-02)"}
+    PlanAdvance -- "restates the prompt (one redraft)" --> DraftPlan
+    PlanAdvance -- "advances, uncertain, or still restating with a host note" --> PlanGate["PLAN_REVIEW"]
+    PlanGate -- "/confirm or agreed" --> Execute["S2: EXECUTE"]
+    PlanGate -- "/revise" --> DraftPlan
+    PlanGate -- "/stop, /cancel" --> Cancelled
+    Execute -- REQUEST_INPUT --> Waiting["WAITING_INPUT, exit 3"]
+    Execute --> HasCode{"Executable Python?"}
+    HasCode -- yes --> Sandbox["Run in session sandbox,<br/>capture witness"]
+    HasCode -- "no (symbolic or analytical)" --> Verify
+    Sandbox --> Verify{"Output verifier"}
+    Verify -- valid --> Success["CLOSED_SUCCESS, exit 0"]
+    Verify -- invalid --> Repair["Bounded repair<br/>(1, or 2 on HEAVY_COMPUTE)"]
+    Repair --> Execute
+    Repair -- "repairs exhausted" --> Cancelled
+```
+
+Review gates accept the fast-path commands `/confirm`, `/revise <feedback>`, `/stop` and `/cancel` without a model call. In fast mode (`--fast`, `/fast on`) the user confirms in advance: a review whose artifact carries no host finding is accepted without waiting (recorded as `STANDING_CONFIRMATION`); one with a finding stops as usual. Other review text goes to System 1 (`ConfirmationMatch`, then `ReviewFacets`) and, when System 1 is not confident, to System 2 interpretation; the harness never assumes an intent. A review command sent when no review is open gets a notice and is never treated as a new request.
+
+### 3.3 Phases
+
+| Phase | Mechanism | Outcomes |
+|---|---|---|
+| **0. Activation** | System 1 `activation_route` over environment **recipe state**: policy scope (`PDLT_POLICY_SCOPE`, default `technical`), network (`PDLT_SANDBOX_NETWORK`, default `false`), knowledge cutoff (`PDLT_KNOWLEDGE_CUTOFF`, default `2024-06`) and the sandbox's execution environment. No keyword, pattern or date matching. These settings never reach System 2. | `APPLY_PROTOCOL` → phase 1; gated `BLOCKED_BY_HIGHER_PRIORITY` → refusal, exit 0; `BYPASS` / `PROTOCOL_DISCUSSION` → direct answer |
+| **1. Prompt review** | Grammar lint, then fast-path commands, then review intent | confirm → phase 2; revise → redraft; cancel → exit 1 |
+| **2. Plan lint** | Deterministic lint (`plan_soundness.py`): PDL-05 no fielded prefixes, PDL-06 no code fences, PDL-08 no deferral or meta markers, PLAN-10 no placeholder steps. One redraft carrying the finding, through operator correction only (never `CARRIED_APPROACH_SOURCES`). No algorithm or execution keywords are required. | clean → plan advancement |
+| **2b. Plan advancement (PLAN-02)** | System 1 `PlanAdvancementRecipe` compares the plan with the confirmed prompt on three task-neutral checks: the plan adds a solution action or deduction with its content, says how it handles the conditions that keep the task from being solved directly, and shows how the result will be obtained. A fourth question asks whether the prompt already states the method (then a plan has nothing to add). One confident failure gets one redraft whose operator correction names only the failed checks, never the decision text or a method; a plan that still fails is published unchanged (AUTH-05) with a `[host] PLAN-02` note, so fast mode never confirms it in advance. Uncertain or unavailable System 1 does not flag the plan. Every decision is a `PLAN_ADVANCEMENT` event. Live accuracy is measured by `run_plan_gate.py`. | advances → plan gate; restates → one redraft, then plan gate with a note |
+| **3. Plan review** | Fast-path commands, then review intent | confirm → phase 4; revise approach → redraft plan; revise task → phase 1; cancel → exit 1 |
+| **4. Execution** | System 2 `EXECUTE` | result → phase 5; `REQUEST_INPUT` → exit 3 |
+| **5. Verification** | Pydantic output verifier with witness authority (§4). Result IR citation bookkeeping is recorded as `RESULT_IR_CITATION_FINDINGS` and never blocks. | pass → exit 0; contract failure → bounded repair → exit 1 |
+
+**When System 1 is unavailable.** System 1 absent, unconfigured, failing or below its confidence gate yields no System 1 evidence: no refusal is published, the problem class defaults to `STANDARD_EXECUTION`, and the execution profile defaults to `STANDARD`. Boundary refusal therefore depends on a reachable System 1; the sandbox's own limits (no network, confinement) still apply whatever System 1 decides.
+
+### 3.4 Headless exit codes (ADR-0019, as amended)
+
+| Code | Meaning |
+|---|---|
+| `0` | `CLOSED_SUCCESS` (verified deliverable), or a published boundary refusal (`closure=REFUSED`) |
+| `1` | `CLOSED_CANCELLED`: cancellation, verification failure after repair, or a fatal protocol error |
+| `2` | Halted at a non-terminal stage, such as an unconfirmed review gate |
+| `3` | `WAITING_INPUT`: paused for required external input |
+| `4` | Harness or provider failure (`EXIT_HARNESS_ERROR`), for example a missing API key or a provider outage |
+| `130` | Interrupted by the user (Ctrl+C) |
+
+A run is headless when stdin is not a terminal or `--non-interactive` is given. `--exit-on-close` ends the process at the first closure; without it, further piped requests start new tasks in the same session.
+
+---
+
+## 4. Verification and witness authority
+
+- **The sandbox-reproduced witness is authoritative.** When the deliverable contains executable code, the witness is what the sandbox prints: exactly one `WITNESS: <json>` line, or a whole stdout that parses as JSON. It replaces any model-asserted witness, and `WITNESS_OVERRIDDEN_BY_SANDBOX` is recorded when the two differ.
+- **Model-asserted witnesses are provisional.** A witness no sandbox run reproduced is labelled `provisional` and never presented as verified.
+- **Claims of computation must come from computation.** A negative witness reporting an exhausted search is accepted only when a host-run program printed it. Proof-based negative witnesses (an argument, no search telemetry) are first-class under `GUARD-03`.
+- **An incomplete result must rest on an attempt.** Declaring requirements open without a witness is accepted only when a program actually ran in that attempt.
+- **No scraping.** There is no label-regex scan of stdout or deliverable text.
+- **Typed dispatch, no domain checkers shipped.** Checkers are selected only by a typed `witness.domain` field. This release registers no domain checkers, so every domain goes to `FallbackChecker`, whose verdicts are provisional: correctness rests on sandbox reproduction of the witness, not on a domain verifier.
+- **Repairs carry facts, not hints.** Each repair carries the latest findings from a closed, task-neutral error registry (`verification/error_registry.py`) through operator correction.
+
+---
+
+## 5. Execution sandbox
+
+### 5.1 Lifecycle
+
+```
+SessionEngine.__init__ : construct ExecutionSandbox once (limits, --sandbox mode)
+probe()                : is the selected backend available here?
+first run_code         : build the session root, owner record and policy; sweep stale roots
+each run               : fresh empty work/run-NNNN-* directory, deleted afterwards
+close()                : delete the root, release the backend
+```
+
+The session root is `<tempdir>/pdlt-sandboxes/<sid>/`, outside every tree the graders, runner and viewer read. A host killed before `close()` leaves a root whose owner process is dead; the next session's sweep removes it.
+
+### 5.2 Containment
+
+| Control | Mechanism |
+|---|---|
+| **Secret isolation** | The child environment is rebuilt from an allowlist (`PATH`, `TEMP`, `TMP`, `TMPDIR`; on Windows also `SYSTEMROOT`, `WINDIR`, `SYSTEMDRIVE`, `COMSPEC`, `PATHEXT`). API keys never reach model-authored code. |
+| **OS-native confinement** (ADR-0021) | Write only the run directory; read it, the base interpreter's install and standard library, and loader files; execute only the base interpreter; no network; no new processes. Linux: Landlock. macOS: `sandbox-exec` with a deny-by-default profile. Windows: a per-session AppContainer inside a Job Object. `--sandbox container`: Docker or Podman, one container per session (no network, read-only root, no capabilities, process cap). |
+| **Fail closed** | When the selected backend cannot apply, nothing runs (`sandbox_unavailable:<reason>`) and the REPL warns. `--sandbox audit-only` (`PDLT_SANDBOX=audit-only`) is the only opt-out and is announced loudly. The default mode is `auto`, which is native confinement; it does not fall back to the container. |
+| **Audit hook** | Defense in depth: blocks native-code loading, file access outside the run directory, network calls, signals and process creation. |
+| **Resource limits** | Windows: Job Object. POSIX: `RLIMIT_AS` and an `RLIMIT_CPU` backstop; macOS does not enforce `RLIMIT_AS`. A deterministic step counter enforces the routed step budget. |
+
+This is not a VM boundary: side channels and kernel exploits are out of scope. The container mode is the stronger option. Details and self-checks: [`docs/SANDBOX.md`](docs/SANDBOX.md).
+
+### 5.3 What a program can do
+
+A program can compute and print. It cannot reach the network, start processes, or read or write anything outside its own run directory, and everything it writes is deleted when the run ends. The Result IR's `files` list names and grounds the deliverable's files, but **nothing is written into the user's project**: the deliverable is text the user applies.
+
+---
+
+## 6. System 1 recipes
+
+All recipes live in `providers/sys1/recipes/`, emit discrete labels only, and contain no pattern matching (`test_routing_recipes_have_no_pattern_matching`).
+
+| Recipe | Labels | Use |
+|---|---|---|
+| `ActivationRouteRecipe` | `APPLY_PROTOCOL`, `PROTOCOL_DISCUSSION`, `BYPASS`, `BLOCKED_BY_HIGHER_PRIORITY` | Phase 0 boundary routing over environment state |
+| `ProblemClassRecipe` | `VERIFIED_EXECUTION`, `STANDARD_EXECUTION` | Whether the Result IR must carry a witness |
+| `ConfirmationMatchRecipe`, `ReviewFacetsRecipe` | agrees / rejects / unclear; change dimensions | Review intent for free-text review messages |
+| `ExecutionProfileRecipe` | `WITHIN_100K_STEPS` … `BEYOND_100M_STEPS` | Step-complexity routing to an execution budget |
+| `FollowUpRecipe` | follow-up vs. new task | Routing messages after closure |
+
+### 6.1 Execution budgets
+
+| Prediction | Tier | Step budget | Memory | Wall clock | Repairs |
+|---|---|---|---|---|---|
+| `WITHIN_100K_STEPS` | `MINIMAL` | 100,000 | 256 MB | 30 s | 1 |
+| `WITHIN_10M_STEPS` | `STANDARD` | 10,000,000 | 256 MB | 30 s | 1 |
+| `WITHIN_100M_STEPS` | `HEAVY_COMPUTE` | 100,000,000 | 512 MB | 120 s | 2 |
+| `BEYOND_100M_STEPS` | `HEAVY_COMPUTE` | 100,000,000 | 512 MB | 120 s | 2 |
+
+The budget is the smallest one System 1 believes suffices with cumulative probability of at least 0.85; a diffuse prediction yields `STANDARD`. The step counter, not the prediction, decides. A `VERIFIED_EXECUTION` task with no registered domain verifier is refused before any System 2 call when System 1 puts more than 0.5 probability on `BEYOND_100M_STEPS` (`BUDGET_REFUSAL`). After plan confirmation the same recipe may raise the tier, never lower it. The budget the sandbox enforces is the one declared to System 2 in `AVAILABLE_EXECUTION_TOOLS`.
+
+### 6.2 Cost controls
+
+Every response is capped by `--max-output-tokens` (default 16,384, reasoning included); one model call has a 300 s deadline (`--api-call-deadline`); `--max-repairs N` overrides the tier's repairs. There is no session-level time or token budget. The catalogue runner kills a prompt's process tree at 600 s.
+
+---
+
+## 7. Input containment and context
+
+- **Semantic bootstrap containment.** Raw user content is read only by `BOOTSTRAP_ANALYSIS`. Later operations (`DRAFT_PROMPT`, `DRAFT_PLAN`, `EXECUTE`) receive compiled projections, never raw user text. A redaction pass (`runtime/quarantine.py`) replaces canary, tripwire and override-directive tokens with `[REDACTED_IOC]`. Task entities are typed (ADR-0027: surface, kind, and what the request says about it, unknowns included) and are forwarded only when their surface is an exact substring of the sanitized request or summary.
+- **Context compilation.** `context_compiler.py` builds one projection per operation from `EXECUTION_CONTRACT.json`, the applicable standard clauses and the stage inputs, and records SHA-256 digests of each clause and of the projection.
+- **Contract resolution** (`normative_store.py`, ADR-0008), first match wins: `PDLT_STANDARDS_PATH`; `<candidate repo>/contracts/` (the candidate repo defaults to the current directory); `~/.pdlt/versions/<version>/contracts/`; the bundled copy in the package. An override replaces the whole contract set and is checked for structure only.
+- **Workspace.** Sessions use an in-memory workspace (`MemoryWorkspaceRun`) persisted per turn under `turns/turn_NNN/`. The previous closed turn's request and deliverable are carried into the next turn's task inputs, and System 1 (`FollowUpRecipe`) routes whether a new message continues it; rejected drafts are never carried.
+
+---
+
+## 8. Observability
+
+- The REPL prints stage and per-call telemetry in dev mode (`--dev`), writes a transcript, and points to a `worker-progress.log` updated during model calls.
+- `--observation-dir` writes one JSONL record per turn: model calls with usage (input, output, reasoning and cached tokens), latency, controller state before and after, and the turn's new workspace events. Records are written when a turn completes; there is no live event stream.
+- **Call lifecycle** (`providers/call_trace.py`). Every model call, System 1 included, records each HTTP attempt's progress: `prepared → connecting → request_sent → acknowledged → response_started → response_complete`, with the HTTP status, response bytes, outcome (completed, interrupted, http_error, transport_error, timeout, unreadable), whether the interruption was local (Ctrl+C) or remote, and whether it was retried. `request_sent` and `connecting` come from Python's own `http.client` audit events, `acknowledged` (the API receipt) from the status line. Each call also records its number among the session's calls of that operation (a repair is `EXECUTE call 2`), its failure category, and what the API reported: the response id (OpenRouter's generation id, which carries TTFT and the serving provider), status, finish reason, output and reasoning tokens, and whether the output schema constrained decoding. Each finished or interrupted call is appended to the session's `call-trace.jsonl`; dev mode prints a line for any call that retried or did not complete.
+- **One output contract per operation** (ADR-0028, IMPL-0001).
+  - The schema the model reads (`output_schema` in the projection) and the grammar sent to the provider are both generated from the operation's Pydantic model (`runtime/output_contracts.py`), in the same form for each call. That form is the grammar mode (schema, JSON mode or none) plus strict or default.
+  - Descriptions and advisory keywords live in the models (`contract(...)`).
+  - Optional fields stay optional, except for providers whose strict mode needs every field required (Groq, Cerebras).
+  - `EXECUTE` is sent in JSON mode.
+- **Replies stalled in whitespace.** A reply cut off at the output cap is kept in the invocation's output directory (`model-response.truncated.txt`), for diagnosis only: it is never parsed, published or sent back to the model. When it ends in at least 1,000 whitespace characters under the output schema, decoding stalled in the schema's whitespace rather than the model writing too much (Nemotron 3 Super wrote its whole `EXECUTE` answer, then only whitespace until the cap). The call stays a counted failed attempt; later calls of that operation in the session are sent without the decoding constraint, and the host still validates every reply against the schema.
+- **Interruptions.** On Ctrl+C the REPL reports where the call was (not sent; sent but not acknowledged; acknowledged, no response; response partly received; between calls) and the engine records `TURN_INTERRUPTED` (the in-flight operation, whether its output was recorded, the in-memory and persisted stage). A turn that never bound a protocol instance is marked `INTERRUPTED`, which is not a closed status, so chaining and follow-ups never mistake it for the previous turn; a bound turn keeps its persisted stage and the next input re-drives it. A review artifact committed to the controller but not yet published is republished from the controller state. State files (controller state, workspace files, the session pointer) are written replace-on-write (`fileio.py`), so an interruption never leaves a partial file.
+- The evaluation-plane viewer (`python -m viewer`) is read-only and binds `127.0.0.1` only.
+
+---
+
+## 9. Guardrails
+
+| Guardrail | Rule in this codebase |
+|---|---|
+| `GUARD-01` | `CARRIED_APPROACH_SOURCES` comes only from user-originated sources; lint and retry feedback travel only by operator correction. |
+| `GUARD-02` | No harness file targets benchmark tokens, prompt phrases or problem-class vocabulary. |
+| `GUARD-03` | No regex scraping of deliverables; proofs and analytical deductions are first-class deliverables. |
+| `GUARD-04` | Gates never require algorithm or execution keywords, and worker guidance never mandates code. |
+| `GUARD-05` | Contract files match `CONTRACT_MANIFEST.json` hashes, and the harness is scanned against tokens derived from the catalogue manifest (`tests/test_harness_anti_overfitting.py`). |
+
+---
+
+## 10. What this release does not do
+
+These are known limits of 2.6.0. Each is captured as future work in [`TARGET_ARCHITECTURE.md`](TARGET_ARCHITECTURE.md) and ADR-0023 to ADR-0026; where part of one has since landed, its entry says what. The release audit of 2.6.0rc1, including smaller follow-ups, is [`docs/audits/RELEASE_AUDIT_2.6.0rc1.md`](docs/audits/RELEASE_AUDIT_2.6.0rc1.md).
+
+- **The client interface is v0, with no live events.** `pdl_taskmaster.api` returns structured results (`TurnResult`: stage, exit status, provenance) and drives the review gates (`confirm`, `revise`, `run(..., confirm=True)`), but it is unversioned and has no live event stream (ADR-0023; [`docs/api.md`](docs/api.md)).
+- **One cost setting.** Reasoning depth, output length, verification depth and model choice are set separately by flags but not by a profile, and there is no session-level time or token budget (ADR-0024).
+- **No effects on the user's project.** Programs cannot create or edit project files, and there is no tool broker (ADR-0025).
+- **Extensions are profiles, not packs.** Task types, verifiers and model providers register through versioned profiles and pass a conformance kit ([`docs/extensions.md`](docs/extensions.md), L124). Operation prompts are still part of the code, contract overrides replace the whole set without a compatibility check, and there are no workflow packs (ADR-0026).
+- **System 1 is remote.** The default System 1 is OpenRouter's decisions endpoint; without it, boundary refusal does not run (§3.3).
+- **Groq gets `EXECUTE` and `EMIT_RESULT_IR` in JSON mode.** It rejects their schemas, so it serves them without one, and the host validates the reply (ADR-0028 rule 5); it is no longer routed past ([`PROVIDERS.md`](PROVIDERS.md) §3). This is covered by offline tests only, not yet run live on Groq. The provider tables are fixed in the worker until the per-stage profiles (IMPL-0003).
+- **`PDLT_SANDBOX_NETWORK` is routing state only.** Setting it to `true` changes what System 1 is told; the sandbox never grants network access.
+- **Headless review gates are confirmed by piping `/confirm`.** The catalogue measures autonomous drafting under the lint gates, not human review (`gate_policy` in `RUN_META.json`).
+- **The category 10 multi-turn scripts are not run.** Category 10 prompts run single-turn; their scripts are kept in the manifest's `multi_turn_script` field.
+- **Live runs need credentials.** `OPENROUTER_API_KEY` is required for live sessions and catalogue runs; the offline test suite needs none.

pdl_taskmaster-2.6.0/GOAL.md

@@ -0,0 +1,142 @@
+# PDLt System 2 Prompt Catalogue — Test Execution Goal
+
+## MISSION
+
+Execute the complete 112-prompt PDLt System 2 Prompt Catalogue through the live pdlt REPL
+to produce a definitive scoreboard measuring protocol fidelity on `gpt-oss-120b` at the
+harness's default reasoning (high before execution, low at EXECUTE; ADR-0022). Results are written to `<runs>/<run-timestamp>/`, where `<runs>` is `$PDLT_CATALOGUE_RUNS`, else `PDLt-Catalogue-Archive` beside the checkout (`catalogue_runs.py`).
+
+---
+
+## NON-NEGOTIABLE RULES
+
+### 1. No Retries, No Do-Overs
+Each prompt gets **exactly one execution attempt**. If it fails, log the failure and move on.
+No re-running a prompt "to see if it works this time." No modifying prompts mid-run. The
+manifest is frozen at run start.
+
+### 2. No Cherry-Picking
+All 112 prompts must be executed in manifest order. You may not skip prompts, reorder them
+to run easy ones first, or filter to only run categories you expect to pass.
+
+### 3. No Prompt Modification
+Prompt files are read-only inputs. You may not edit, simplify, clarify, or annotate a prompt
+before feeding it to pdlt. The whole point is testing the model's ability to interpret raw
+user input under protocol governance.
+
+### 4. No Manual Intervention in the Protocol
+The runner uses `--non-interactive` and `--exit-on-close` and confirms every review gate by
+piping `/confirm` (recorded in RUN_META as `gate_policy: evaluator_confirms_via_stdin`). No human
+may confirm, revise, edit, or override any gate. The model must produce Prompt Pseudocode and
+Response Plan Pseudocode autonomously; the harness's grammar lint is the only gate on them.
+
+### 5. Full Transcript Capture
+Every session must produce a transcript file. Sessions without transcripts are invalid.
+
+### 6. Single Model, Single Configuration
+The entire run uses one model (`openai/gpt-oss-120b`) at one reasoning configuration: the
+harness default (high before execution, low at EXECUTE; ADR-0022), which is also what a live
+session runs at. `RUN_META.json` records the effective effort per operation (`reasoning_effective`).
+No switching models mid-run. No escalating reasoning effort for hard prompts.
+
+---
+
+## EXECUTION
+
+### Step 1: Dry Run (Verify Setup)
+```powershell
+python run_catalogue.py --dry-run
+```
+Verify: 112 prompts listed, 57 marked VERIFIED (ground truth), manifest parses cleanly.
+
+### Step 2: Single-Prompt Smoke Test
+```powershell
+python run_catalogue.py --prompt-id 06-04 --timeout 120
+```
+Run the easiest prompt first (race condition counter, difficulty: easy) to confirm the
+harness pipeline works end-to-end. Check the generated result.json and transcript.
+
+### Step 3: Full Run
+```powershell
+python run_catalogue.py --model openai/gpt-oss-120b --timeout 180
+```
+This will take approximately 30-60 minutes depending on API latency.
+
+### Step 4: Review Scoreboard
+Open `<runs>/<run-timestamp>/SCOREBOARD.md` and report:
+1. Overall pass rate
+2. Pass rate by category
+3. Pass rate by difficulty tier
+4. The result of each regression prompt (the `REG-*` ids in the manifest: REG-001, REG-003, REG-011 to REG-014)
+5. Which categories had 0% pass rate (ceiling not reached)
+6. Which categories had 100% pass rate (ceiling not tested)
+
+### Step 5: Spot-Check Verified Prompts
+For every prompt with `ground_truth_status: "verified"` that reported CLOSED_SUCCESS,
+open the transcript and compare the model's output against the solution oracle file in
+`prompts/solutions/`. Flag any case where the verdict is CLOSED_SUCCESS but the answer
+is substantively wrong (false positive).
+
+---
+
+## OUTPUT STRUCTURE
+
+```
+PDLt-Catalogue-Archive/              # or $PDLT_CATALOGUE_RUNS
+    run-YYYYMMDD-HHMMSS/
+        RUN_META.json               # Frozen run configuration
+        SCOREBOARD.json             # Machine-readable aggregate results
+        SCOREBOARD.md               # Human-readable scoreboard with tables
+        results/
+            01-01_schur_triples_n15/
+                result.json         # Verdict, exit code, timing, ground truth
+                transcript.txt      # Full pdlt session transcript
+                stdout.txt          # Raw stdout from pdlt
+                stderr.txt          # Raw stderr (errors, warnings)
+                session/            # pdlt session workspace directory
+            01-02_exact_cover_dlx/
+                ...
+            ...                     # 112 result directories total
+```
+
+---
+
+## VERDICTS
+
+| Verdict | Meaning |
+|---------|---------|
+| `CLOSED_SUCCESS` | Full protocol traversal: PROMPT_REVIEW -> PLAN_REVIEW -> EXECUTION -> closed green |
+| `UNCONFIRMED_GATE` | Session exited while sitting at an unconfirmed review gate (exit code 2) |
+| `TIMEOUT` | Wall-clock timeout exceeded (prompt took too long) |
+| `EXIT_N` | pdlt exited with unexpected exit code N |
+
+---
+
+## WHAT SUCCESS LOOKS LIKE
+
+A successful run produces a scoreboard with:
+- **Every prompt attempted** (112 results, 0 skipped)
+- **RUN_META.json confirms**: `retries_allowed: 0, do_overs_allowed: false`
+- **Regression prompts reported by id** (REG-001, REG-003, REG-011 to REG-014): a failure is reported, never tuned away
+- **Verified prompts spot-checked**: CLOSED_SUCCESS verdicts match ground truth
+- **0 false positives** in SCOREBOARD `false_positives` (graded prompts: 01-01..01-07, 13-01; the other 13 verified prompts are MANUAL)
+
+The pass rate itself is the empirical measurement. We expect it won't be 100% — the whole
+point is finding where the ceiling is. But every failure must be a real failure, not an
+infrastructure bug or a skipped prompt.
+
+---
+
+## WHAT FAILURE LOOKS LIKE
+
+Invalid run conditions (must abort and restart from scratch):
+- Runner crashes before completing all 105 prompts
+- Network/API outage causes mass timeouts (>50% TIMEOUT verdicts)
+- PYTHONPATH misconfigured (pdlt import errors in stderr)
+- Manifest file modified during run
+
+These are NOT invalid — they are legitimate test results:
+- Model produces wrong answers (that's what we're measuring)
+- Model fails review gates (protocol enforcement working as designed)
+- Model times out on hard problems (ceiling found)
+- Adversarial prompts cause unexpected behavior (that's why they're in the catalogue)