@markstanden/opencode-test-container 0.0.0-stage → 0.1.0

package/docs/parity-record.md

@@ -0,0 +1,81 @@
+# Recorded-evidence parity record
+
+Run 2026-11-15 during reconciliation. No model calls; both analysers read the
+same recorded evidence.
+
+## Committed mini corpus
+
+Fixture: `tests/fixtures/docs-parity` (4 trials, copied from the recorded
+confirmation campaign).
+
+Command (OTC):
+
+```sh
+node dist/cli/otc.js replay \
+  --manifest tests/fixtures/docs-parity/manifest.json \
+  --evidence-root tests/fixtures/docs-parity/evidence \
+  --out /tmp/opencode/mini-new
+```
+
+Original command (docs @ extraction base):
+
+```sh
+node tests/fixtures/placement-harness/analysis/analyse.mts \
+  --campaign tests/fixtures/analysis/manifests/mini.json \
+  --evidence-root /tmp/opencode/mini-evidence \
+  --out /tmp/opencode/mini-old
+```
+
+Result: `analysis.json`, `REPORT.md` and `SCREENING.json` deep-equal;
+`analysis-receipt.json` equal after removing `renderedAt`. This is enforced by
+`tests/contract/docs-parity.test.ts` in `npm test`.
+
+## Recorded confirmation corpus
+
+Evidence: `/home/mark/tmp/recording-confirm-20261110` (40 trials, 20 pairs).
+Result: all three artefacts byte-identical; receipt differs only in
+`renderedAt`. Decision `inconclusive`; totals 40 planned/completed; utility
+20; bounds delta -13..7.
+
+Rerun 2026-11-15 at OTC head `4de97a0` against the original analyser at docs
+`76ed331`: `analysis.json`, `REPORT.md`, `SCREENING.json` and the receipt
+(apart from `renderedAt`) were byte-identical.
+
+## Recorded screening corpus
+
+Evidence: `/home/mark/tmp/docs-desc-fresh-20261110` (48 trials, six arms).
+Result: all three artefacts byte-identical; decision `not-defined`; utility
+24; 6 screening findings.
+
+Rerun 2026-11-15 at OTC head `4de97a0` against the original analyser at docs
+`76ed331`: all three artefacts and the receipt (apart from `renderedAt`) were
+byte-identical.
+
+## Fresh lifecycle rehearsal
+
+`npm run rehearse:fresh-docs` (2026-11-15, real Podman image
+`sha256:58505eac…`, OpenCode 2.2.6, scripted loopback provider) ran four docs
+slots and a separate exhausted-budget root:
+
+| Slot            | Primary | Outcome | Selected attempt | Judge requests |
+| --------------- | ------- | ------- | ---------------- | -------------- |
+| useful-write    | pass    | pass    | attempt-001      | 1              |
+| gate-blocked    | pass    | pass    | attempt-001      | 1              |
+| primary-failure | fail    | fail    | attempt-001      | 1              |
+| recovery        | —       | pass    | attempt-002      | 2              |
+
+The recovery's two attempts recorded the same `packetSha256`; the exhausted
+root refused the second dispatch after one claim and wrote no selection.
+Closure, analysis and post-close refusals held. Scripted chat requests: 6
+trial + 5 judge + 1 exhausted; readiness probes: 0; paid provider calls: 0.
+
+## Limits
+
+- The two larger corpora are ~56 MB and are not committed; the committed mini
+  corpus enforces the same pipeline offline.
+- Retrieval measures require a manifest measurement declaration. The recorded
+  manifests declare none, so retrieval-specific parity is covered by the
+  ported measures contract tests, not by a recorded retrieval campaign.
+- The frozen published `ANALYSIS-confirmation.json` predates the current
+  analysis schema and is not a byte target. The comparison is original
+  analyser at the extraction base versus the OTC port on the same evidence.

package/docs/pr-description.md

@@ -0,0 +1,103 @@
+# PR — extract OTC: an isolated test sandbox for OpenCode plugins, skills and instructions
+
+## Summary
+
+Adds `@markstanden/opencode-test-container` (`otc`) as a new package:
+
+- versioned suite/campaign/subject/runtime contracts with discriminated
+  `skill-trigger` and `docs-task` cases;
+- immutable preparation, freeze and delivery checks (resolved image ID, CLI
+  version, subject, prompt and product-checker hashes);
+- explicit scheduled execution through Podman and the real OpenCode CLI;
+- product-supplied primary checks executed in their own no-network container,
+  never mounted into the trial;
+- an arm-blind docs judge using the established prompt/parse/validate/failure
+  classification, with one registered `json-syntax` recovery from identical
+  packet bytes and durable per-slot selection;
+- append-only lifecycle ledger, deterministic assessment, idempotent closure,
+  read-only status and derived reports;
+- the established docs analysis implementation (recording credit, utility and
+  opportunity counts, retrieval, preservation, guardrails, pairing/bounds,
+  screening, decision and reporting) with its ported contract suite;
+- deterministic skill-activation evaluator;
+- emitted-JavaScript `otc` bin proven through `npm pack` in a temporary
+  consumer;
+- offline real-adapter rehearsals for both the skill/docs baseline and the
+  complete fresh docs lifecycle;
+- explicit legacy readers for frozen prod014 schedule/manifest records.
+
+## Design
+
+`docs/architecture.md` records the source-to-destination ownership manifest and
+the subject/runtime/suite/campaign/assessment split. `docs/schema.md` records
+the glossary and migration table. Core imports only contracts and adapters; it
+never imports a docs checkout or assumes A/B, eight pairs, a judge or a
+provider. Docs semantics live only in `src/evaluators/docs`.
+
+A checker is trusted product data. OTC accepts, stages and executes it; the
+docs repository still owns the case data, truth and checker bytes. The trial
+cannot read the checker, and the judge cannot read the campaign or arm
+identities.
+
+## Evidence-guarantee repair pass
+
+Four independently reproduced blockers are closed:
+
+- one checked-selection reader is shared by judge, status, closure and report;
+  it re-derives attempt identity, packet identity, policy hash, judgement
+  SHA-256, outcome/utility and the judged mutation state. A readable `{}` is
+  not a judgement and cannot permit closure;
+- judge and recovery claims now share `<root>/evidence`; a recovery settles both
+  claims by dispatch outcome, and an interrupted dispatch leaves visible
+  `spawned` claims that block closure;
+- the judge's standing permission is read from the pristine prepared workspace,
+  never the trial's live `AGENTS.md`; A/B instruction canaries are absent from
+  both packets;
+- the checked reader hashes the actual slot and attempt packet bytes, compares
+  them with registered identities and with the packet reconstructed from
+  current evidence, and reports missing/changed packets across judge, status,
+  closure and report;
+- the executing tool identity includes an artifact hash derived from the
+  executing module's own `src`/`dist` location and is compared with the freeze
+  before `run`, `judge`, `delivery-check` or `status` trusts it; source
+  execution beside a build and installed JavaScript execution each fingerprint
+  the right tree. Read-only historical replay remains independent of the
+  freeze.
+
+## Proof
+
+- `npm ci`, `npm test` (552 tests, including packaged CLI), `npm run
+typecheck`, `defined comply` all pass.
+- Coverage populations, all measured on this head:
+    - migrated docs analysis keeps the inherited 95/90/95 floors
+      (95.6/90.2/95.9);
+    - all OTC runtime has the explicit 75/55/80 ratchet (91.6/80.2/93.0);
+    - the genuinely new lifecycle/adapters/evaluators population has an explicit
+      82/66/84 ratchet (87.8/71.3/90.0).
+- Recorded-evidence parity: committed mini corpus deep-equal; recorded 40-trial
+  confirmation and 48-trial screening corpora byte-identical apart from the
+  receipt's `renderedAt`.
+- `npm run rehearse:offline`: real Podman image, OpenCode 2.2.6, scripted
+  provider, zero paid calls.
+- `npm run rehearse:fresh-docs`: primary pass/fail, useful and gate-blocked
+  writes, malformed-then-valid judge recovery selecting `attempt-002` with
+  identical packet bytes, exhausted budget refusal, checked selection,
+  closure, analysis and post-close refusal. 6 trial + 5 judge + 1 exhausted
+  scripted chat requests, 0 readiness probes, 0 paid calls.
+- Installed tarball runs the complete offline docs lifecycle from an unrelated
+  directory with spaces, with no checkout or `tsx`/Bun dependency. The receipt
+  records the tarball filename, SHA-256 and npm integrity.
+- The migrated `harness/run.ts` CLI is removed, so no OTC source can spawn the
+  docs repository's `placement-harness/harness.mts`. Only product-supplied
+  suite and checker bytes remain external data.
+
+## Limits
+
+- The judge is a model call when dispatched; it is not deterministic evidence.
+  The deterministic assessment remains the criterion.
+- `network: "host"` is documented honestly as host networking, not
+  provider-only egress.
+- The two larger recorded parity corpora remain external and are not
+  committed; the committed mini corpus enforces the same pipeline.
+- Publishing to npm and creating the remote are external steps; the repository
+  is ready to publish.

package/docs/schema.md

@@ -0,0 +1,97 @@
+# Schema and migration table
+
+## Glossary
+
+- **suite** — reusable case definitions plus one assessment kind.
+- **case** — one prompt/context and its expected behaviour.
+- **campaign** — one frozen scheduled execution.
+- **arm** — one treatment configuration.
+- **slot** — one case × arm × repetition.
+- **trial** — execution of a slot.
+- **assessment** — deterministic or model-based evaluation.
+- **judge attempt** — one separately accounted model assessment dispatch.
+- **report** — derived output outside immutable evidence.
+
+Canonical fields: `caseId`, `campaignId`, `armId`, `slotId`, `repetitionId`,
+`attemptId`. `repetitionId` is an opaque string; the legacy numeric repetition
+is rendered as `r<N>` only at the adapter boundary.
+
+## Versioned documents
+
+| Document      | Version | Notes                                                                 |
+| ------------- | ------- | --------------------------------------------------------------------- |
+| suite         | 1       | discriminated `skill-trigger` / `docs-task`; no executable fields     |
+| campaign      | 1       | explicit slots, runtime, budgets, assessment                          |
+| manifest      | 1       | embedded suite and normalised campaign; no source absolute paths      |
+| freeze        | 1       | `kind: "otc-freeze"`, immutable IDs, tool identity and `freezeSha256` |
+| assessment    | 1       | skill or docs shape; deterministic `attemptId`                        |
+| primary check | 1       | product checker identity, fixed interpreter and outcome               |
+| judgement     | 1       | arm-blind judge verdict and its failure class                         |
+| selection     | 1       | per-slot chosen attempt and every attempt fact                        |
+| report        | 1       | derived, `reportSha256` over content                                  |
+| ledger event  | 1       | trial, primary-check, assessment, judge and close events              |
+
+`docs-task` cases may also carry `truth` (arm-blind judge context) and `primaryCheck: { id, directory, script, timeoutMs? }`. The checker directory
+is staged under `prepare/primary-checks/<caseId>`; the fixed contract is
+`node /checker/<script> /live /pristine`. It is never mounted into the trial.
+
+A campaign `assessment` may carry `judge: { model, timeoutMs? }` and
+`recovery` (the narrow registration in `analysis/recovery.ts`). Judge and
+recovery budgets count dispatches, not HTTP requests; `providerRequestLimit`
+stays `null` unless an adapter exposes a real request count.
+
+The prepared manifest and freeze receipt also carry
+`tool: { name, version, evaluatorVersion, artifactSha256 }`. The artifact hash
+covers the executing runtime tree and package manifest, derived from the
+executing module's own `src`/`dist` location, so source execution beside a
+build and installed JavaScript execution each fingerprint the right tree. `run`, `judge`, `delivery-check` and
+`status` compare the executing identity with the frozen identity before
+dispatch; read-only historical replay does not require it.
+
+Judge and recovery claims live only under
+`<root>/evidence/{judge,recovery}`. A recovery settles both of its claims
+according to the dispatch outcome; an interrupted dispatch keeps the claim
+`spawned` and closure refuses it. Status counts `judged`, `pending`, `failed`,
+`invalid` and `unsettled` from the same checked state.
+
+A selection is trusted only through the shared checked reader: attempt set,
+packet bytes, policy hash, selected judgement SHA-256, outcome/utility and the
+judged documentation state are re-derived. The slot and attempt `packet.md`
+bytes are hashed and compared with the registered identity, with each other,
+and with the packet reconstructed from current evidence. A readable object
+alone is never a judgement and never permits closure; missing, changed or
+mismatched packets and metadata are explicit integrity diagnostics in judge,
+status, closure and report. Metadata-reading problems from every attempt are
+retained even when other attempts remain readable, so an earlier terminal
+failure never masks an inconsistent later recovery.
+
+## New/legacy names
+
+| Legacy                  | New                              | Handling                                        |
+| ----------------------- | -------------------------------- | ----------------------------------------------- |
+| `scenario`              | `caseId`                         | mapped by `legacy/prod014.ts`; old id unchanged |
+| `stamp`                 | `slotId`                         | the frozen stamp becomes the slot id, unchanged |
+| numeric `repetition`    | `repetitionId`                   | `r<N>` at the adapter boundary                  |
+| `pairId`                | `pairId` (legacy metadata)       | retained on `LegacySlot`                        |
+| `arm` A/B               | `armId`                          | retained, no implicit pair policy               |
+| `PLACEMENT_*`           | `OTC_*` or typed manifest fields | new code only; no silent alias                  |
+| `AC_*`                  | typed manifest fields            | new code only                                   |
+| `docs-operations.jsonl` | unchanged                        | genuine plugin interface                        |
+| `OPENCODE_*`            | unchanged                        | genuine upstream interface                      |
+
+Conflicting legacy/new environment aliases are not accepted. Frozen evidence
+and historical `.mjs` files are never rewritten by a rename.
+
+## Outcome states
+
+- validity: `valid`, `invalid`, `unknown`, `unrun`.
+- outcome: `pass`, `fail`, `unknown`.
+- A missing capture or failed target load is `unknown`/`invalid`, never pass.
+- Docs refused writes retain zero recording credit; attempts are reported.
+- A primary checker that fails makes the criterion fail; a timeout or spawn
+  error is `unknown`; a declared checker with no result is `invalid`.
+- Judge failure classes: `json-syntax`, `schema`, `semantic`,
+  `empty-or-truncated`, `transport-or-process`. Only a complete `json-syntax`
+  failure is recoverable, once, from identical packet bytes.
+- A judge attempt is a durable dispatch, not an HTTP request; request counts
+  are reported separately when the adapter exposes them.

package/examples/skills/README.md

@@ -0,0 +1,13 @@
+# Skill-only example
+
+A minimal, self-contained skill bundle and suite. It needs no docs plugin and
+no judge. Run it against any image containing the scripted provider route:
+
+```sh
+otc validate --campaign examples/skills/campaign.json
+```
+
+The campaign names `localhost/otc-sandbox:2.2.6`; build it with the arguments
+in `container/README.md`, or replace the image with one you already trust.
+`SCRIPTED_API_KEY` is a placeholder for the offline provider, never a real
+credential.

package/examples/skills/campaign.json

@@ -0,0 +1,73 @@
+{
+    "schemaVersion": 1,
+    "campaignId": "skill-basic-example",
+    "suite": {
+        "path": "suite.json"
+    },
+    "subjects": [
+        {
+            "armId": "skill-only",
+            "skills": [
+                {
+                    "id": "otel-helper",
+                    "path": "skills/otel-helper"
+                }
+            ]
+        }
+    ],
+    "schedule": {
+        "kind": "explicit",
+        "slots": [
+            {
+                "slotId": "s1",
+                "caseId": "positive-instrument",
+                "armId": "skill-only",
+                "repetitionId": "r0"
+            },
+            {
+                "slotId": "s2",
+                "caseId": "negative-unrelated",
+                "armId": "skill-only",
+                "repetitionId": "r0"
+            },
+            {
+                "slotId": "s3",
+                "caseId": "negative-near-miss",
+                "armId": "skill-only",
+                "repetitionId": "r0"
+            }
+        ]
+    },
+    "runtime": {
+        "image": "localhost/otc-sandbox:2.2.6",
+        "opencodeVersion": "2.2.6",
+        "provider": {
+            "route": "scripted",
+            "model": "scripted/scripted",
+            "overlay": {
+                "providers": {
+                    "zai-coding-plan": {
+                        "settings": {
+                            "baseURL": "http://127.0.0.1:8799/v1"
+                        }
+                    },
+                    "scripted": {
+                        "settings": {
+                            "baseURL": "http://127.0.0.1:8799/v1"
+                        }
+                    }
+                }
+            }
+        },
+        "timeoutMs": 120000,
+        "network": "host",
+        "passthroughEnv": ["SCRIPTED_API_KEY"]
+    },
+    "budgets": {
+        "trials": 3,
+        "judgeAttempts": 0
+    },
+    "assessment": {
+        "kind": "skill-activation"
+    }
+}

package/examples/skills/skills/otel-helper/SKILL.md

@@ -0,0 +1,10 @@
+---
+name: otel-helper
+description: Use when a task asks for OpenTelemetry span, trace or metric instrumentation in a Node service.
+---
+
+# OpenTelemetry helper
+
+Use this skill when the task names OpenTelemetry and asks for instrumentation
+of spans, traces or metrics. Prefer official `@opentelemetry/*` packages, name
+spans after the operation and never invent an exporter endpoint.

package/examples/skills/suite.json

@@ -0,0 +1,29 @@
+{
+    "schemaVersion": 1,
+    "suiteId": "skill-basic",
+    "description": "Synthetic activation suite for exercising the skill evaluator offline.",
+    "assessment": "skill-activation",
+    "cases": [
+        {
+            "caseId": "positive-instrument",
+            "kind": "skill-trigger",
+            "targetSkillId": "otel-helper",
+            "expected": "activation",
+            "prompt": "Instrument the checkout service with OpenTelemetry spans."
+        },
+        {
+            "caseId": "negative-unrelated",
+            "kind": "skill-trigger",
+            "targetSkillId": "otel-helper",
+            "expected": "no-activation",
+            "prompt": "Rename the local variable total to subtotal."
+        },
+        {
+            "caseId": "negative-near-miss",
+            "kind": "skill-trigger",
+            "targetSkillId": "otel-helper",
+            "expected": "no-activation",
+            "prompt": "Add a console log around the payment call; no telemetry is needed."
+        }
+    ]
+}

package/package.json

@@ -1,6 +1,46 @@
 {
-  "name": "@markstanden/opencode-test-container",
-  "version": "0.0.0-stage",
-  "stub": true,
-  "description": "Temporary package placeholder for staged publishing"
-}
+    "name": "@markstanden/opencode-test-container",
+    "version": "0.1.0",
+    "type": "module",
+    "description": "An isolated test sandbox for OpenCode plugins, skills and agent instructions.",
+    "repository": {
+        "type": "git",
+        "url": "git+https://github.com/markstanden/opencode-test-container.git"
+    },
+    "exports": {
+        ".": {
+            "types": "./dist/index.d.ts",
+            "default": "./dist/index.js"
+        }
+    },
+    "bin": {
+        "otc": "./dist/cli/otc.js"
+    },
+    "files": [
+        "dist",
+        "container",
+        "examples",
+        "docs",
+        "README.md",
+        "LICENSE"
+    ],
+    "engines": {
+        "node": ">=26"
+    },
+    "scripts": {
+        "build": "rm -rf dist && tsc -p tsconfig.build.json",
+        "prepack": "npm run build",
+        "test": "npm run build && node --test 'tests/**/*.test.ts'",
+        "test:unit": "node --test 'tests/unit/**/*.test.ts'",
+        "test:coverage": "npm run build && node scripts/check-coverage.mjs",
+        "typecheck": "tsc --noEmit",
+        "rehearse:offline": "npm run build && node rehearsals/offline.mts",
+        "rehearse:fresh-docs": "npm run build && node rehearsals/fresh-docs.mts",
+        "rehearse:installed": "npm run build && node rehearsals/installed.mts"
+    },
+    "devDependencies": {
+        "@types/node": "^26.6.4",
+        "typescript": "^6.0.3"
+    },
+    "license": "MIT"
+}