skill-harness 0.15.0 → 0.17.0

package/docs/PRODUCT-GUIDE.md

@@ -0,0 +1,190 @@
+# Factory: work, review, learn
+
+**Release candidate: skill-harness 0.17.0**, paired with **pi-daddy 0.27.3**.
+Principal 3.2.0 is unchanged. The session retrospective feature is merged in PR82
+with independent Sol review and green CI. This release adds explicit local session
+import, timeline/usage coverage and evidence-linked notes. The producer adds a
+visible count and toggle for completed ledger history.
+
+Canonical packaging, installed verification and publication are recorded separately
+in the release PR. Use a fresh Pi session after updating the harness: an existing
+immutable dashboard bridge survives reload. Other running sessions need not stop.
+
+## Who does what?
+
+**pi-daddy** coordinates permitted work and owns execution, steering and the future-order
+registry. **Herdr** displays its sessions and side panel. **skill-harness** retains
+measurement and learning evidence. The panel is not an autonomous worker. Finished
+execution is not human acceptance.
+
+Use the producer's `/grants work` guided setup for multiple obligations, dependencies,
+priority and explicit agent/model/thinking selections, then its supported run/host controls.
+For existing single-task use, `pi-daddy work add --id report --outcome "Review the report"`
+remains supported. Only listed controls are available. Pausing new dispatch does not
+cancel running work; changing a next-order policy does not mutate active sessions.
+Principal's ordinary current/stale/superseded reports remain unchanged.
+
+## Open learning in Pi
+
+With both matching extensions loaded, use **`/grants learning`** from the current host.
+The producer binds the exact current snapshot, archive, population and local review author.
+A changed scope must report a mismatch, not silently reuse an old learning workspace.
+
+The harness also exposes **`/skill-harness learning --state "WORKSPACE"`** (use the
+workspace path displayed by the host), or `skill-harness learning --state "WORKSPACE"`
+in a terminal. These open the **same guided flow and durable records**, not another app.
+
+1. **Readiness** explains what's retained, what's missing, and why automatic questions
+   are silent. Opening a review is deliberate: it neither reserves attention nor
+   represents an earned automatic question. It starts no model or delegation.
+2. **Connect retained input** lists compatible case batches and qualified comparisons
+   already in the selected archive. Pick one and give it a short name and title.
+   No hand-written manifests or ledger JSON. No compatible evidence is an honest empty state.
+3. **Review cases** shows nominations and lets you open their actual retained evidence
+   sources by selection (CLI: `learning evidence BATCH --item N --evidence N`). Redacted
+   or unavailable sources are marked, never called complete evidence. Record confirmed defect, expected
+   behavior, exemplar, uncertain or skip, with a reason. Corrections are append-only.
+   Confirming a case is not promoting a test or establishing a cause.
+4. **Review comparisons** opens complete retained artifacts under A/B labels. Acknowledge
+   each complete artifact only after reviewing it. Then choose one, tie, none acceptable,
+   or insufficient evidence. **Reveal model/cost** is a separate action, enabled only after
+   the durable quality receipt. Artifact contents themselves may disclose identity.
+5. **Record adopt / reject / defer** records a separate scope-bound decision. Reject and defer
+   are normal outcomes; neither forces a preference or reveal. Missing evidence can be
+   durably deferred. Adopt is intent, not activation.
+6. **Propose hypothesis** records prediction, disproof, downside, rollback and a competing
+   explanation from an actual case. **Link original hypothesis** only accepts the comparison's
+   actual frozen origin, never a retrospective replacement.
+
+Previously retained excerpt feedback concerned
+**excerpts**. Keep that receipt unchanged and linked as prior feedback. It is not full-artifact
+acceptance, detector calibration or a fresh blind experiment. An old unscoped quality choice
+also does not automatically become a full-artifact acknowledgement.
+
+## Configure trust without writing manifests
+
+Open **Trust / independent labels** in the same guided flow:
+
+- Before freezing, explicitly add any real unflagged incident evidence and select its split.
+  Only selected evidence files are retained; do not include private sessions or secrets.
+- Select a retained case batch, detector and evaluation split. The product derives the
+  cohort and predictions from actual nominations and groups repeated target observations
+  into one incident. Detector version and population remain exact.
+- Automatic questions default to **silent**. To configure them, explicitly select minimum
+  resolved independent incidents, the confidence lower bound, attention budget and expiry.
+  The form previews the exact frozen policy. It creates **no independent labels**.
+- Review cases independently, then choose **Link current case label**. Unflagged sample
+  labels have their own explicit evidence/author step. Preference, model self-report,
+  panel agreement and exit zero are not independent correctness labels.
+
+Missing calibration, missing unflagged samples, held-out absence, conflicting labels,
+expired policy and exhausted attention stay visible. Reopening neither resets choices nor
+refills attention. Correcting a linked case withdraws its old label from confidence.
+The original producer closing path still requires presence, quiescence and its exposure gate.
+
+## Scoped adoption and later work
+
+Use **`/grants learning` in interactive Pi** for activation/rollback. The matching producer
+implements two separate profiles: unchanged **`fixed-policy-v1`**, and
+**`ordinary-work-policy-v1`** for model/effort-only changes to the same selected task IDs and
+agent definitions. The latter does not change instructions, skills, grants, capabilities,
+topology or assessment policy; do not disguise those changes as model/effort or fixed policy.
+The producer prepares policy bytes and owns independent authority and the original registry.
+Source/built integration has been checked with synthetic fixtures; final installed-release
+verification remains pending. Bare legacy fixed-policy views lacking applied activation
+proof defer; they must not be treated as activation receipts.
+
+Activation needs the exact selected candidate, original confirmed case/hypothesis/comparison,
+current independently verified eligible facts, explicit adoption authority, and registry
+revision consent. The original registry revalidates these again. Only its actual activation
+receipt advances the learning record to activated. New ordinary-work orders pin the policy
+under fresh eligibility confirmation; active or history-bearing runs never migrate.
+Rollback similarly needs its own authorized request and original applied registry receipt.
+A stored lifecycle link, human preference or prepared receipt cannot substitute for either.
+
+**Adoption / outcomes** can inspect linked later observations or record one using complete
+artifact, original/current requirement and independent evidence files. Unknown is valid.
+Acceptance requires both the exact accepted artifact and independent acceptance evidence.
+Changed requirements, changed artifact, out-of-scope, success, caught defect and escape remain
+distinct. No outcome triggers automatic rollback, and no single outcome proves improvement.
+
+## CLI and local archive use
+
+```sh
+skill-harness learning status --state "WORKSPACE"       # human readiness
+skill-harness learning status --state "WORKSPACE" --json # integration/details
+skill-harness learning review reports --state "WORKSPACE"
+skill-harness learning help                             # all explicit operations
+skill-harness learning guide                            # this guide, packaged offline
+skill-harness learning current                          # packaged requirement register
+```
+
+Without a producer host, `skill-harness learning` guides connection to an explicitly selected
+archive and selecting an actual retained case scope. Or use
+`learning init --archive DIR --author NAME --confirm`; multiple retained scopes ask you to
+select `--scope-item N`. Snapshot digest and population are derived from actual cases, not
+typed hashes. For a comparison-only archive with no cases, `--scope TEXT --population TEXT`
+can explicitly declare a local review scope; it cannot bind mismatching case snapshots.
+Use the producer flow to connect a host. Noninteractive writes require explicit `--confirm`.
+All learning commands are offline/no-model. Command completion is never a quality verdict.
+Local author attribution and content hashes are not authentication or hostile-owner attestation.
+
+Current requirements and remaining evidence: [STATUS.md](STATUS.md).
+
+## Whole-session retrospective (0.17.0)
+
+Use this after a real task to inspect the conversation, model/thinking settings,
+tool use, individual delegation attempts and provider-reported usage. The importer
+reads only explicitly selected files and never contacts or interrupts an agent.
+
+Start with the terminal CLI (or prefix these arguments with
+"/skill-harness learning" in Pi instead of "skill-harness learning"):
+
+    skill-harness learning session preview --session /absolute/parent.jsonl --ledger /absolute/grants.jsonl
+    skill-harness learning session import --session /absolute/parent.jsonl --ledger /absolute/grants.jsonl --name task --title "Task retrospective" --expected DIGEST_FROM_PREVIEW --confirm --state /private/learning/workspace --archive /private/learning/archive --author operator
+    skill-harness learning session show task --state /private/learning/workspace
+    skill-harness learning session source task --item 1 --state /private/learning/workspace
+
+The first import can create a workspace. For an existing workspace omit --archive
+and --author. Optional --feedback FILE and --artifact FILE retain selected external
+feedback and artifact/source evidence. Use --sources FILE for a JSON array of
+additional selections, for example:
+
+    [{"kind":"child","path":"/absolute/child.jsonl","executionId":"exec:10000000-0000-4000-8000-000000000001"}]
+
+Every source must be a regular, unlinked file: at most 16 files, 8 MiB each,
+32 MiB total; Pi/ledger JSONL is bounded to 4096 records with complete final lines.
+Pi session format v3 and the existing frozen pi-daddy v3 ledger contract are
+supported. Re-preview if any selected bytes change. The ledger association and
+child execution mapping are operator declarations, not authenticated parentage.
+
+**Privacy:** import retains exact original bytes locally, potentially including
+credentials, reasoning, tool arguments or private text present in the source.
+Preview prints paths, hashes and coverage, not a redacted copy. Inspect selected
+inputs before confirming; no scanning, automatic redaction or publication occurs.
+Full-source readback deliberately displays the selected source; terminal control
+characters are escaped for display without altering archived bytes.
+
+Open the existing learning wizard with the same --state path to choose **Import
+session** or **Review sessions**. Editors display evidence; edits are discarded.
+The ordinary producer "/grants learning" bridge also reaches those menu items
+when its current host workspace is already connected. Reading does not reserve
+attention, run a model or change a next-order policy.
+
+Record a finding, proposal, acceptance observation or context note against a
+specific retained source line:
+
+    skill-harness learning session note task --kind proposal --note "Ask for the capability map before implementation; compare this on a later task." --item 1 --line 2 --confirm --state /private/learning/workspace
+
+Notes identify the configured local author. An acceptance note is an operator
+annotation, not an authenticated user approval or qualified comparison.
+The timeline is chronological and may include abandoned branches; excerpts are
+labelled. It does not reconstruct the active branch or verify a test claimed in prose.
+Child usage and missing evidence are reported separately; totals are provider
+observations, not billing or a complete parent-plus-child cost.
+
+This slice makes sessions reviewable. It does **not** automatically generate or
+measure skill/tool/instruction improvements. A later experiment still needs a
+frozen task and candidate, appropriate independent assessment and actual retained
+measurements before claiming faster or equally good output. Existing adoption
+supports its documented model/effort policy scope only; annotations do not expand it.

package/docs/STATUS.md

@@ -0,0 +1,56 @@
+# Factory requirement register
+
+Current entry point: [PRODUCT-GUIDE.md](PRODUCT-GUIDE.md). The dated
+[CURRENT.md](CURRENT.md), build reports and transport decisions are retained history,
+not current instructions to supply manifests or judge engineering repairs.
+
+## 0.17.0 release candidate — session learning
+
+Adds explicit Pi-session import, exact private retention, chronological evidence,
+per-execution delegation history, parent/child usage coverage, and line-bound
+operator annotations through the existing CLI/Pi learning workspace. No model
+calls, automatic acceptance, calibrated improvement or expanded adoption authority.
+See the whole-session section of PRODUCT-GUIDE.md. Feature PR82 is merged with independent review and green CI. Release metadata
+pairs harness 0.17.0 with producer 0.27.3; canonical publication and installed user
+testing remain separate. Entire-session optimization and measured skill/tool variants remain
+follow-up work.
+
+## Historical 0.16.0 release preparation
+
+Release pair: **skill-harness 0.16.0 / pi-daddy 0.27.0**; Principal 3.2.0 is unchanged.
+The prior audited baseline was harness 0.15.0 (`a725579`) / producer 0.26.1 (`93a9b01`).
+Harness PR80 merged at `c6cd55d4db9ccf9a1ec790459c3cc5e6b1eac69f` with green CI and
+Sol approval. Actual source/built cross-package learning/registry checks passed with
+synthetic fixtures. This is **release preparation**, not publication, final installed
+proof, human approval, calibration or measured improvement. Loaded resources, source
+HEAD, canonical archives, published bytes and active sessions remain separate identities.
+Canonical packing, publication and final installed verification are pending.
+
+| Requirement | Current implementation / boundary | Remaining evidence |
+|---|---|---|
+| Understandable side panel | Producer owns selected-outcome-first status, observed agents, available controls and Details. | Matching producer's installed narrow-pane/host checks. No invented progress/ETA/acceptance. |
+| Ordinary multi-obligation work | Producer owns work setup, dependencies, bounded parallel/sequential execution, model/effort choice and revision consent. | Installed checks of the matching producer's supported profiles. Existing Principal workflows remain unchanged. |
+| Retained cases/comparisons | Harness learning workspace composes existing archive, case-review, blind-choice and CAS journal primitives. Named bindings and guided CLI/Pi entrypoints; missing/stale/mismatched inputs defer. | Final installed producer connection; source review/CI and synthetic source/built linkage passed. No new model evidence claimed. |
+| Complete quality review | Complete-output acknowledgement, durable choice before reveal, immutable replay, separate adopt/reject/defer. Prior excerpt feedback stays separate. | A real full-artifact product choice remains absent where not actually recorded. Excerpt preference is not acceptance. |
+| Scope-bound trust | Guided frozen cohort/policy, actual nomination predictions, incident grouping, explicit independent case/unflagged labels and pure exposure preview. Existing automatic gates unchanged. | Real independent labels, unflagged coverage and held-out evidence where absent. Implementation cannot manufacture them. |
+| Hypothesis and comparison linkage | Human-authored falsifiable hypothesis form, confirmed-case linkage, monotonic optional context and retained lifecycle navigation. | Ongoing weekly model-backed retro is not demonstrated or automatically scheduled. Existing bounded weekly/model primitives remain separate authorized paths. |
+| Adoption and rollback | Harness validates scope/choice/current independent authority and eligible facts. Producer source supports unchanged `fixed-policy-v1` plus separate `ordinary-work-policy-v1` (model/effort only, same task IDs/agent definitions and scope/assessment). Producer owns activation/rollback and fresh-eligibility next-order pins; profile-agnostic harness linkage retains original receipts. | Synthetic source/built original activation/rollback linkage passed. Configured production adoption/rollback and final installed integration remain unverified; bare legacy views without applied activation proof defer. No active/history-bearing run migration or instruction/skill/grant/capability/topology/assessment expansion. |
+| Later outcomes | Explicit file-based CLI/Pi observation forms plus producer API linkage; independent reference attribution; changed requirements/artifacts/out-of-scope/unknown remain distinct. | Genuine later work, exact acceptance and held-out efficacy where absent. No automatic rollback or universal quality score. |
+| Packaging / portability | Public adapter exports, additive frozen dashboard bridge, CLI export and bundled Pi entrypoint; guide/register shipped offline via `learning guide` / `learning current`. A pre-existing immutable bridge needs a fresh Pi session. | Release-metadata review/CI, exact merged canonical pack with retained archives, empty-prefix installed Pi/registry verification, then authorized publication. |
+
+## Historical claims remain bounded
+
+- The failure-derived abort-race case and the public-weather comparison are different chains.
+- Previously retained feedback covered excerpts only. It does not advance full comparison
+  acceptance or detector trust.
+- Earlier no-tool measured SDK observations do not establish universal ordinary-child Usage,
+  writable-effect containment or actual provider-internal reasoning effort.
+- No OS sandbox or malicious-local-owner protection has been added.
+- No historical session/evidence, auth or unrelated worktree was modified in this candidate.
+
+Feature source review/CI and source/built cross-package checks are completed evidence,
+not final released-install proof. Metadata preparation runs consistency/diff validation only:
+no builds, installs, suites or model calls. Canonical pack and actual installed two-extension
+verification remain release gates under `PUBLISHING.md`. Prior live route smokes remain
+historical and are not replayed merely for a version bump. Human acceptance, calibration and
+later efficacy still require their own real evidence.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "skill-harness",
-  "version": "0.15.0",
+  "version": "0.17.0",
   "description": "Test/optimize loop for agent skills — run spec'd scenarios on pi, LLM-judge, score, review, re-run",
   "type": "module",
   "keywords": [
@@ -28,11 +28,13 @@
     "dist/prompt-capture-extension.js",
     "assets/report.template.html",
     "assets/report.grade.js",
+    "docs/PRODUCT-GUIDE.md",
+    "docs/STATUS.md",
     "LICENSE",
     "README.md"
   ],
   "dependencies": {
-    "@skill-harness/cli": "0.15.0",
+    "@skill-harness/cli": "0.17.0",
     "typebox": "^1.1.38"
   },
   "repository": {