@supersuit/superskill 0.4.0 → 0.6.0

package/CHANGELOG.md

@@ -1,5 +1,73 @@
 # Changelog
 
+## 0.6.0 (2026-10-06)
+
+**A skill's embodiment is what it has absorbed, not one run of it.** 0.5.0 made a golden from a
+real run the top-level requirement. Its first real test reversed it: nine harvested candidates,
+each a real accepted run, and none was the standard the skill should be held to. The tests and
+fixes a skill has absorbed are. **Breaking for anyone at superskill today:** the top level now
+needs a real-run record and a 0.6.0 `--run` (one that records its `skill_sha` and the triggers).
+
+- **Superskill is now** every miss closed with a regression eval (`misses-closed`, which replaces
+  `no-stale-open-miss`: an open miss fails however recent it is), and `evals-pass`: the last
+  `--run` was against the current `SKILL.md`, the suite passed (default 90% of runs), every fixed
+  miss's regression eval passed every run, and the trigger evals were right (default 90%).
+  `run-evidence` and `run-fresh` stand.
+- **`real-runs`:** a real-run record where one model+harness pair, alone, has at least 5 real runs
+  at 80% one-shot (no correction). Every pair is reported; pairs are never pooled, and a pair that
+  says `unknown` never counts. The record is a harness-neutral file, `superskill-real-runs/1`, read
+  from `--real-runs <dir>` / `SUPERSKILL_REAL_RUNS`, then `evals/real-runs.json`, then Freedom's
+  skill ledger live. It replaces `real-use`.
+- **Configurable bar:** `--min-real-runs`, `--min-one-shot`, `--min-pass-rate`,
+  `--min-trigger-rate` (and `SUPERSKILL_MIN_*`). Never frontmatter: a skill cannot lower its own bar.
+- **Goldens are optional evidence.** `golden-approved` never fails. A golden is an eval graded on
+  `goldens/<id>/expectations.json`, a checklist of expected behavior (grade outcomes, not paths);
+  `output.md` is the reference that proves the task solvable. Provenance rules still say which
+  goldens are real runs.
+- **`doctor --run` runs the trigger evals** (did the harness load the skill exactly when it
+  should) and writes `skill_sha` and `triggers` into `latest.json`.
+- **`history-in-skill` (warn, level skill):** a dated incident story in `SKILL.md` ("Earned
+  2026-09-08", "(Gary, 2026-09-16: ...)", "on 2026-09-13 a session ...") belongs in `MISSES.md`,
+  which now holds a story's full entry: id, date, what happened, fix, eval, optional `Quote:`, with
+  indented continuation lines. Tuned on Freedom's 103 shipped skills. Still no line-count rule.
+- `superskill miss` takes `--quote` and `--date`.
+- Tests: 17 new; each guard broken on purpose and seen red (a stale-sha pass counting, pooled pairs
+  counting, an unknown pair counting, an open miss passing, a half-passing regression eval passing,
+  examples and provenance stamps flagged as history).
+
+## 0.5.0 (2026-10-06)
+
+**Superskill needs a golden from a real run.** Before, any golden a person approved counted, so a
+skill could reach the top level on examples its author invented. (Gary Sheng, on five invented
+goldens written to lift Freedom's flagship skills: *"I'm just concerned about hallucination that
+we're accepting just to get higher doctor ratings."*) **Breaking for anyone at superskill today:**
+an approved golden without the new `PROVENANCE.json` no longer counts, and the doctor says which
+golden and why.
+
+- `goldens/<id>/PROVENANCE.json`: `source` (`real-run`, `synthetic`, `synthetic-reconstruction`),
+  the `run` it came from (session, commit or ledger id), and who `accepted` the output when it ran,
+  and when. Only `real-run` with all of that counts for `golden-approved`. An anonymized twin of a
+  private golden counts with `derived_from` and the anonymizer's `ANONYMIZED.json` receipt.
+- **Private goldens.** `--private-goldens <dir>` (or `SUPERSKILL_PRIVATE_GOLDENS`) on `doctor`,
+  `approve` and `doctor --run` also reads `<dir>/<skill-name>/<id>/`, for real runs too personal
+  to ship with the skill. `approve` writes the approval beside the golden it found.
+- `init --from-session` writes a `real-run` provenance with `accepted` left empty, so the golden
+  counts only once someone records who accepted it.
+- **`evals-real` (tested):** a `superskill init` `REPLACE:` placeholder, or the same prompt or
+  trigger query twice, is not a case. Measured in Freedom on 2026-10-05: the init scaffold copied up
+  to 3 evals and 10 triggers scored `tested` while testing nothing.
+- **`doctor --run` is never recorded as a real use.** Every harness spawns its child with
+  `FREEDOM_SKILL_LEDGER=off` and `SUPERSKILL_SANDBOX=1`. Measured 2026-10-05: three sandbox runs
+  landed in Freedom's skill ledger as perfect one-shot runs of a skill nobody had used.
+- **`real-use` (info):** where the run ledger records what the person's next message made of each
+  run, the doctor prints how many of the last 30 days' judged runs they accepted, beside the level.
+- `miss import --freedom-ledger` skips sandbox records, matches `freedom:<skill>` records, and
+  points a correction's "Should have" at the person's next message (session and time, never text).
+  It reads `FREEDOM_SKILL_LEDGER_HOME` when Freedom's ledger was re-pointed.
+- Tests: 11 new; each guard broken on purpose and seen red (invented golden reaching superskill,
+  placeholders counting, duplicate prompts counting, a sandbox run becoming a miss, a twin with no
+  receipt counting, the ledger off switch missing from the run env, codex spawned without it).
+
 ## 0.4.0 (2026-09-29)
 
 **Approve from your phone.** `superskill approve` worked only at an interactive terminal, so an

package/README.md

@@ -4,8 +4,9 @@ Score any agent skill folder as **skill**, **tested**, or **superskill**, and ge
 to-do list for the next level. Works on skills for Claude Code, Codex, or any harness that
 reads the [Agent Skills](https://agentskills.io) format. Zero dependencies, Node 20 or later.
 
-A superskill runs on frontier intelligence, is checked against examples you approved, and is
-fixed every time it gets something wrong. [What that means](https://supersuit.wiki/concepts/superskill);
+A superskill is fixed every time it gets something wrong, with a test that keeps each fix fixed;
+passes its whole eval suite on a current model; and does the job for real people without a
+correction, often enough to count. [What that means](https://supersuit.wiki/concepts/superskill);
 [the standard](SPEC.md).
 
 ## 30 seconds
@@ -34,12 +35,17 @@ file error. `--json` prints one JSON document and nothing else.
 
 1. **skill**: a valid `SKILL.md` (name matches the folder, description of 1024 characters or
    fewer that says when to use it, hard rules above the compaction fold, headings in long bodies, nothing said twice), references one level deep, no
-   hard-coded machine paths, nothing that reads like a prompt injection.
+   hard-coded machine paths, nothing that reads like a prompt injection. It warns on dated
+   incident stories in `SKILL.md`, which belong in `MISSES.md`.
 2. **tested**: at least three task evals with checks a machine can verify, and a trigger set of
    at least ten requests, some that should load the skill and some near-misses that should not.
-3. **superskill**: a golden a person approved; no miss open longer than 14 days and every fixed
-   miss guarded by an eval; a recent `--run` on file where the skill beats the same task done
-   without it. "Recent" follows `metadata.cadence` (a weekly skill's proof lasts 30 days).
+3. **superskill**: no open miss, and every fixed miss guarded by a regression eval; a recent
+   `--run` against the `SKILL.md` that is there now, where the suite passes (90%), every
+   regression eval passes every run, the trigger set is 90% right, and the skill beats the same
+   task done without it; and a real-run record where one model+harness has at least 5 real runs
+   at 80% one-shot (no correction). Pairs are never pooled. "Recent" follows `metadata.cadence`
+   (a weekly skill's proof lasts 30 days). Goldens are optional evidence. Every threshold is a
+   flag (`--min-real-runs`, `--min-one-shot`, `--min-pass-rate`, `--min-trigger-rate`).
 
 Every rule and threshold is in [SPEC.md](SPEC.md).
 
@@ -52,7 +58,7 @@ Every rule and threshold is in [SPEC.md](SPEC.md).
 | `superskill doctor <skill> --run [--harness claude\|codex] [--repeat 3] [--yes]` | Run the evals for real, with and without the skill, and write `evals/results/latest.json`. **The only command that spends model calls**; it prints an estimate and asks first. |
 | `superskill init <skill>` | Add missing `evals/`, `goldens/`, `MISSES.md`. Never overwrites. |
 | `superskill init <skill> --from-session <transcript>` | Turn the session where you did the job by hand into the first eval and a golden candidate (Claude Code `.jsonl`, or any text file as the request). |
-| `superskill miss <skill> "<what happened>" [--expected "..."]` | Log a time the skill got it wrong. |
+| `superskill miss <skill> "<what happened>" [--expected "..."] [--quote "..."] [--date YYYY-MM-DD]` | Log a time the skill got it wrong, or the story behind a rule it learned. |
 | `superskill fix <skill> <miss-id> --eval <id> [--commit <sha>]` | Close a miss. Refuses without an eval that exists. |
 | `superskill approve <skill> <golden> [--basis judgment\|outcome] [--rationale ...] [--evidence ...]` | A person signs off on a golden, saying why and what it rests on: `judgment` (it reads right) or `outcome` (it produced a checkable result, with evidence). At a terminal it asks for your name; from your phone, an agent records your tap with `--approved-by` and `--via`. Approvals accumulate. |
 | `superskill collection <folder...> [--budget <chars>] [--overlap 0.5]` | Listing budget used, descriptions that get cut off, pairs of skills an agent could confuse (with near-miss triggers to add). |
@@ -70,9 +76,10 @@ my-skill/
   SKILL.md
   evals/evals.json          task evals (Anthropic skill-creator format)
   evals/triggers.json       should / should-not load (skill-creator format)
-  goldens/<id>/             input.md, output.md, APPROVAL.json
-  MISSES.md                 every miss, open or fixed with its eval
-  evals/results/latest.json the last --run
+  goldens/<id>/             optional: input.md, expectations.json, output.md, PROVENANCE.json, APPROVAL.json
+  MISSES.md                 every miss and its story, open or fixed with its eval
+  evals/results/latest.json the last --run (which SKILL.md it proved, suite and triggers)
+  evals/real-runs.json      real uses per model+harness, one-shot or not (or --real-runs <dir>)
 ```
 
 Harnesses ignore folders they do not know, so none of this changes how the skill loads.
@@ -84,13 +91,16 @@ Harnesses ignore folders they do not know, so none of this changes how the skill
 - Codex: the skill is linked at `.agents/skills/<name>`. Codex cannot switch skills off, so a
   copy installed in `~/.agents/skills` can leak into the baseline; move it aside while proving.
 - Machine checks (`contains:`, `regex:`, `file_exists:`) are free. Each plain-language expectation
-  and each golden costs one grader call per run.
+  costs one grader call per run.
+- Trigger evals run each query in `evals/triggers.json` with the skill installed and record
+  whether the harness loaded it (Claude Code: a `Skill` call or a read of its `SKILL.md`; Codex:
+  a read of its `SKILL.md`).
 
 ## Freedom
 
 Nothing here needs [Freedom](https://getfreedom.wiki). If a skill has Freedom's `HDSOP.md`, the
 doctor shows it as a bonus; `miss import --freedom-ledger` reads Freedom's run ledger as plain
-files. `superskill snippet` gives any agent the same habits with no Freedom installed.
+files, and with no `evals/real-runs.json` the doctor reads the real-run record from it too. `superskill snippet` gives any agent the same habits with no Freedom installed.
 
 ## Releasing
 

package/SPEC.md

@@ -1,13 +1,21 @@
 # The superskill standard
 
-**Version 0.4.0** (2026-09-29). The reference checker is `@supersuit/superskill`; where this
+**Version 0.6.0** (2026-10-06). The reference checker is `@supersuit/superskill`; where this
 document and the checker disagree, the checker has a bug.
 
-A **superskill** runs on frontier intelligence, is checked against examples a person approved,
-and is fixed every time it gets something wrong
+A **superskill** runs on frontier intelligence, is fixed every time it gets something wrong, and
+does the job for real people without a correction
 ([definition](https://supersuit.wiki/concepts/superskill)). This standard turns each clause into
 a file in the skill's own folder, so the evidence travels with the skill wherever it is copied.
 
+**What a skill has absorbed is its embodiment, not one run of it (0.6.0).** Until 0.5.0 the top
+level required a golden: one real run a person approved as the standard. A random accepted run
+is not the ultimate embodiment of a skill; the misses it was fixed for, and the evals that keep
+each fix fixed, are. This follows Anthropic's guidance on agent evals: build the suite from real
+failures, hold regression evals near 100%, grade outcomes rather than paths, and use a reference
+solution to prove a task is solvable, not as the answer to match. Goldens remain, as optional
+evidence.
+
 ## Contents
 
 - [The clauses and their evidence](#the-clauses-and-their-evidence)
@@ -23,9 +31,10 @@ a file in the skill's own folder, so the evidence travels with the skill whereve
 | The clause | What proves it | Where it lives |
 |---|---|---|
 | A skill at all | A valid `SKILL.md` under the Agent Skills spec, plus the hygiene rules below | `SKILL.md` |
-| Checked against examples you approved | At least one golden: a real input, the output a person said was right, and a record of who approved it and when | `goldens/<id>/` |
-| Fixed every time it gets something wrong | A miss log where every miss is open (recently) or fixed with a regression eval that would catch it again | `MISSES.md` + `evals/evals.json` |
-| Runs on frontier intelligence | Its evals last passed on a current model, recently, and beat the same task run without the skill | `evals/results/latest.json` |
+| Fixed every time it gets something wrong | A miss log where every miss is fixed, each with a regression eval that would catch it again, and the story of each fix | `MISSES.md` + `evals/evals.json` |
+| Runs on frontier intelligence | Its whole suite and trigger set last passed against the current `SKILL.md`, on a current model, recently, and beat the same task run without the skill | `evals/results/latest.json` |
+| Does the job for real | Enough real runs on one model+harness, enough of them one-shot (no correction) | `evals/real-runs.json` (or an operator's export, see below) |
+| Optional: an example a person approved | A golden: a real input, the checklist a right answer meets, a reference output, where it came from | `goldens/<id>/` (or a private folder) |
 
 ```
 my-skill/
@@ -33,11 +42,14 @@ my-skill/
   references/ scripts/ ...          (as the Agent Skills spec allows)
   evals/evals.json                  task evals        (level: tested)
   evals/triggers.json               trigger evals     (level: tested)
-  goldens/<id>/input.md             approved examples (level: superskill)
+  MISSES.md                         miss log + history (level: superskill)
+  evals/results/latest.json         last --run        (level: superskill)
+  evals/real-runs.json              real-run record   (level: superskill)
+  goldens/<id>/input.md             optional evidence
+  goldens/<id>/expectations.json
   goldens/<id>/output.md
+  goldens/<id>/PROVENANCE.json
   goldens/<id>/APPROVAL.json
-  MISSES.md                         miss log          (level: superskill)
-  evals/results/latest.json         last --run        (level: superskill)
 ```
 
 ## Compatibility
@@ -77,6 +89,7 @@ A line in a bundled file containing `superskill-ignore` is skipped by `no-absolu
 | `body-size` | info | reports lines and estimated tokens (characters / 4) once the body passes about 5000 tokens. Length alone is never a defect. |
 | `rules-above-the-fold` | warn | in a body past about 5000 tokens, every hard rule (a shouted NEVER, ALWAYS, MUST, DO NOT, REFUSE, or a bolded **Never ...** command, outside code fences) appears in the first 5000 tokens, either there or restated there. After compaction Claude Code keeps only that much of each invoked skill. |
 | `reference-says-when` | warn | every link from SKILL.md to a markdown file sits on a line that says when to read it (before, when, if, for, read ...). Step files (`steps/<step>.md`) are the recommended way to keep a long skill's detail out of the always-loaded body: they are read fresh when the step comes up, so compaction does not lose them. |
+| `history-in-skill` | warn | no dated incident story in the body outside code: "Earned 2026-09-08", "(Gary, 2026-09-16: ...)", "Wilson, 2026-09-05: \*"..."\*", "on 2026-09-13 a session ...", "until 2026-09-21 the flag ...", "measured 2026-09-20", "(2026-09-30, #324)", "- 2026-09-15 (#147): ...". Matched per paragraph, since a line wrap can split a name from its date. Not flagged: a line with "e.g." or "example", a date in a code span or fence, a heading, an HTML comment (a generator's provenance stamp), "as of <date>". The story goes in `MISSES.md` (below); `SKILL.md` keeps the rule and a one-line why. There is deliberately no line-count rule (retired 2026-09-28): length alone is never a defect. |
 | `navigable` | warn | a body over 300 lines has no run of more than 150 lines without a heading |
 | `no-repeated-paragraphs` | warn | no paragraph of 100+ characters appears twice |
 | `references-one-deep` | fail | a markdown file linked from `SKILL.md` links on to no further local file |
@@ -91,18 +104,27 @@ A line in a bundled file containing `superskill-ignore` is skipped by `no-absolu
 |---|---|---|
 | `evals-present` | fail | `evals/evals.json` parses and there are at least 3 cases (each golden with an input and an output counts as one) |
 | `evals-verifiable` | fail | every case in `evals.json` has a prompt and at least one expectation |
+| `evals-real` | fail | no eval case and no trigger still holds a `superskill init` `REPLACE:` placeholder, no two cases share a prompt, and no two triggers share a query |
 | `triggers-present` | fail | `evals/triggers.json` has at least 10 queries, at least 3 that should load the skill and at least 3 near-misses that should not |
 
 ### Level 3: superskill
 
 | Rule | Severity | Threshold |
 |---|---|---|
-| `golden-approved` | fail | at least one golden has an approval with non-empty `approved_by` and a valid `approved_at`; info when it was approved against an earlier `SKILL.md`; info naming the weight (judgment and outcome approvals per golden), and saying so plainly when no golden has an outcome yet |
 | `misses-log-present` | fail | `MISSES.md` exists (it may have no entries) |
-| `no-stale-open-miss` | fail | no miss has been open more than 14 days |
+| `misses-closed` | fail | no miss is open. A miss is a known way the skill fails; a skill that still fails a known way is not at the top level, however recent the miss |
 | `fixed-miss-has-eval` | fail | every fixed miss names an eval id present in `evals.json` or `goldens/` |
+| `evals-pass` | fail | the last `--run` was against the current `SKILL.md` (`skill_sha` equals the sha256 of `SKILL.md`); its suite passed at least the minimum share of runs with the skill (default 90%); every fixed miss's regression eval passed every run; and it ran the trigger evals and got at least the minimum share right (default 90%) |
 | `run-evidence` | fail | `evals/results/latest.json` exists and `with_skill.pass_rate` > `without_skill.pass_rate` |
 | `run-fresh` | fail / warn | the last run is younger than the cadence window: `daily` or `weekly` 30 days, `monthly` 60, `quarterly` 120, `yearly` 365, none declared 30 (info). A `yearly` skill always warns to `--run` before its next real use. An unknown cadence warns |
+| `real-runs` | fail | a real-run record (below) in which at least one model+harness pair, on its own, has at least the minimum runs (default 5) and one-shot share (default 80%). Each pair is reported as info; **pairs are never pooled**, since a clean record on one model in one harness proves nothing about another. A pair whose model or harness is `unknown` is reported and never counts |
+| `golden-approved` | info / warn | never fails. Reports each golden that is approved and from a real run (with its approval weight), names approved goldens that are not from a real run and why, names a golden with no `expectations.json`, and warns on an `APPROVAL.json` or `PROVENANCE.json` that does not parse |
+
+**The thresholds are configurable, never by the skill itself.** A skill cannot lower its own bar,
+so they are not frontmatter. `doctor` takes `--min-real-runs <n>`, `--min-one-shot <0..1>`,
+`--min-pass-rate <0..1>` and `--min-trigger-rate <0..1>`, or the environment variables
+`SUPERSKILL_MIN_REAL_RUNS`, `SUPERSKILL_MIN_ONE_SHOT`, `SUPERSKILL_MIN_PASS_RATE` and
+`SUPERSKILL_MIN_TRIGGER_RATE`. A report scored with non-default thresholds should say so.
 
 `metadata.cadence` in `SKILL.md` frontmatter declares how often the skill really runs:
 
@@ -152,11 +174,18 @@ A bare array of cases is accepted on read, as is `assertions` for `expectations`
 ]
 ```
 
-### `goldens/<id>/`
+### `goldens/<id>/` (optional evidence)
+
+**A golden is an eval whose grading is a checklist of expected behavior (0.6.0).** It is never
+required for any level. Grade the outcome, not the path: the checklist says what a right answer
+does, and the reference output proves the task is solvable and calibrates the grader. A golden
+that exists is still held to the provenance rules below, because "a person accepted this when
+it ran" is a claim that has to be true.
 
 - `input.md`: the real request.
-- `output.md` (or any other file that is not `input.*` or `APPROVAL.json`): the output a person
-  said was right.
+- `expectations.json`: an array of expectations (the same forms as `evals.json`), graded by
+  `--run`. Without it, `--run` grades by likeness to the reference output.
+- `output.md` (or any other file that is not `input.*` or a metadata file): the reference output.
 - `APPROVAL.json`, written only by `superskill approve`: at an interactive terminal, or relayed by an agent with `--approved-by` and `--via` after the person approved with a tap (the `via` field records where):
 
 ```json
@@ -181,6 +210,35 @@ approval whose `note` is its rationale.
 A golden is also an eval: `--run` judges the skill's output for `input.md` against the approved
 output.
 
+**`PROVENANCE.json` says where the example came from (0.5.0).** Only a golden from a real run is
+reported as real evidence. An invented input with an invented output puts "a person said this
+was right" on something no person's work produced. Invented cases belong in `evals/evals.json`.
+
+```json
+{
+  "source": "real-run",
+  "run": { "session": "88696e1b-...", "ledger_id": "inv_2026-09-28T02-28-29Z_ab12", "commit": null, "at": "2026-09-28T02:28:29Z" },
+  "accepted": { "by": "Ann Example", "at": "2026-09-28T03:22:51Z", "signal": "close", "evidence": "her next message after the run" }
+}
+```
+
+- `source`: `real-run`, `synthetic`, or `synthetic-reconstruction`. Only `real-run` counts.
+- `run`: at least one of `session`, `commit`, `ledger_id` names the run it came from.
+- `accepted`: who accepted the output **when it ran** (`by`), and when (`at`). This is not the
+  approval: accepting is what the person did at the time (moved on, closed, said go); approving
+  is saying afterwards that this example is the standard. Both are required.
+- An **anonymized twin** of a private golden says `anonymized: true` and `derived_from` (a hash of
+  the original, never its path or content) in place of the run, and carries `ANONYMIZED.json`, the
+  anonymizer's receipt (`checker`, `checked_at`, `counts` by kind, `fingerprint`, never the
+  mapping). Without the receipt the twin does not count.
+- `superskill init --from-session` writes `source: real-run` with `accepted` empty, so the golden
+  is not reported as real until someone records who accepted it.
+
+**Private goldens.** A real run's input and output are usually about real people and should not
+travel with a skill that is shared. `--private-goldens <dir>` (or `SUPERSKILL_PRIVATE_GOLDENS`)
+makes `doctor`, `approve` and `doctor --run` also read `<dir>/<skill-name>/<id>/`, laid out exactly
+like `goldens/<id>/`. A miss is closed only by an eval or golden that ships with the skill.
+
 ### `MISSES.md`
 
 ```markdown
@@ -191,11 +249,29 @@ output.
 - Should have: Merged duplicates into one line.
 - Fix: a1b2c3d
 - Eval: m1
+- Quote: "why is Atlas in here twice"     (optional)
 - Source: freedom-ledger inv_2026-09-20T10-00-00Z_ab12   (optional)
 ```
 
 Headings are `## <id> · <YYYY-MM-DD> · <open|fixed>`; `|` or `-` also separate. Ids are `m1`,
-`m2`, ... Other headings are ignored.
+`m2`, ... Other headings are ignored. A field runs on over indented lines that follow it.
+
+**`MISSES.md` is where a skill's history lives (0.6.0).** `SKILL.md` is instructions, loaded on
+every run; the story of the incident that earned a rule is history, and Anthropic's authoring
+guidance is to keep time-sensitive content out of instructions. So a story becomes a miss entry:
+
+| Field | Holds |
+|---|---|
+| id, date | `m<N>`, and the date the incident happened (not the date it was written down) |
+| What happened | the incident, in full: what the skill did, what it cost, what was noticed |
+| Should have | what the skill should have done |
+| Fix | the commit, or the rule now in `SKILL.md` that the incident earned |
+| Eval | the regression eval that would catch it again. Empty until one exists, and the doctor says so (`fixed-miss-has-eval`): a story with no eval is a fix nothing guards yet |
+| Quote | optional: what the person said, verbatim |
+| Source | optional: where it was recorded (a ledger id, an issue) |
+
+`SKILL.md` keeps the rule and a one-line why. `superskill miss <skill> "<what>" --date <d> --quote
+"<q>"` writes one.
 
 ### `evals/results/latest.json`
 
@@ -206,15 +282,53 @@ Written only by `superskill doctor --run`:
   "run_at": "2026-09-20T10:00:00.000Z",
   "harness": "claude",
   "model": "<model id the harness reported>",
+  "skill_sha": "<sha256 of the SKILL.md the run proved>",
   "cases": 4,
   "repeat": 3,
   "with_skill": { "pass_rate": 1.0, "mean_ms": 21000, "mean_tokens": 4100 },
   "without_skill": { "pass_rate": 0.33, "mean_ms": 18000, "mean_tokens": 3900 },
-  "per_case": [{ "id": "1", "with_skill": { "runs": 3, "passes": 3 }, "without_skill": { "runs": 3, "passes": 1 }, "failures": [] }]
+  "per_case": [{ "id": "1", "with_skill": { "runs": 3, "passes": 3 }, "without_skill": { "runs": 3, "passes": 1 }, "failures": [] }],
+  "triggers": { "cases": 12, "runs": 36, "passes": 35, "pass_rate": 0.972,
+    "per_query": [{ "query": "write my weekly status", "should_trigger": true, "runs": 3, "passes": 3 }] }
 }
 ```
 
 A run passes when every expectation of its case passes; `pass_rate` is passing runs over runs.
+A trigger run passes when the harness loaded the skill exactly when `should_trigger` says
+(Claude Code: a `Skill` tool call naming it or a read of its `SKILL.md`; Codex: a read of its
+`SKILL.md`). A golden with `expectations.json` is graded on that checklist; one without is graded
+by likeness to its reference output.
+
+### `evals/real-runs.json` (the real-run record)
+
+Harness-neutral: any harness, ledger or script may write it, and the doctor only reads it.
+
+```json
+{
+  "format": "superskill-real-runs/1",
+  "skill": "weekly-status",
+  "generated_at": "2026-10-06T12:00:00Z",
+  "source": "freedom-skill-ledger",
+  "pairs": [
+    { "model": "claude-opus-5-5", "harness": "claude-code", "runs": 12, "one_shot": 11,
+      "first_at": "2026-09-01T09:00:00Z", "last_at": "2026-10-05T18:00:00Z" }
+  ]
+}
+```
+
+- A **run** is one real use of the skill by a person. Never a sandbox run (`doctor --run`), never
+  a test.
+- It is **one-shot** when the person needed no correction, rescue or redirect, it did not fail or
+  get abandoned, and it was not corrected after it handed back. A taste note is the person's
+  preference, not the skill's defect, and does not break one-shot.
+- `model` is the model id the harness ran (as the transcript or environment reports it);
+  `harness` is `claude-code`, `codex`, or another harness's name. A writer that cannot tell
+  writes `unknown`, never a guess. Counts only: the file carries no input, output or names, so it
+  can ship with the skill.
+
+**Where the doctor reads it**, first found wins: `<dir>/<skill-name>.json` where `<dir>` is
+`--real-runs` or `SUPERSKILL_REAL_RUNS` (an operator's own export, kept out of the skill); then
+`evals/real-runs.json` in the skill; then Freedom's skill ledger, read live (below).
 
 ## Collections and plugins
 
@@ -246,7 +360,17 @@ exist they are read as plain files:
   `{id, skill, started, outcome, interventions: [{kind, note|what}], errors}`. A record with an
   intervention of kind `redirect`, `correction` or `rescue`, or with `outcome: "failed"`, becomes
   an open miss dated from `started`; `taste` interventions are skipped; the ledger id is kept on
-  a `Source:` line so a record is never imported twice.
+  a `Source:` line so a record is never imported twice. A record with `synthetic: true` (a
+  sandbox run) is skipped. A record corrected after it handed back (`corrected_after`, with
+  `next_turn_ref`) gets a "Should have" that points at the person's correction: the session and
+  the time, never their words.
+- **The real-run record.** Freedom's skill ledger records, on every run, the `model` id and the
+  `harness` (`claude-code`, `codex`, ...), read from the session transcript or the environment and
+  `unknown` when neither says. Freedom exports it in the `superskill-real-runs/1` format above,
+  one file per skill, for `--real-runs`. With no export and no `evals/real-runs.json`, the doctor
+  reads the same ledger files live and counts them the same way (sandbox runs skipped).
+- **`doctor --run` is never recorded as a use.** Every harness spawns its child with
+  `FREEDOM_SKILL_LEDGER=off` and `SUPERSKILL_SANDBOX=1`, in a folder named `superskill-run-*`.
 
 ## Versioning
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@supersuit/superskill",
-  "version": "0.4.0",
+  "version": "0.6.0",
   "description": "Score any agent skill folder as skill, tested, or superskill. An open standard and a zero-dependency CLI.",
   "type": "module",
   "bin": {

package/src/commands/approve.mjs

@@ -5,8 +5,9 @@ import { createInterface } from "node:readline/promises";
 import { parseArgs, clock, UsageError } from "../args.mjs";
 import { skillDir, refuse } from "./common.mjs";
 import { readGoldens, approvalEntry, withApproval, BASES } from "../goldens.mjs";
+import { parseSkillFile } from "../frontmatter.mjs";
 
-export const help = `superskill approve <skill> <golden-id> [--rationale "<why it is right>"] [--basis judgment|outcome] [--evidence "<what happened, where to check>"]
+export const help = `superskill approve <skill> <golden-id> [--rationale "<why it is right>"] [--basis judgment|outcome] [--evidence "<what happened, where to check>"] [--private-goldens <dir>]
 
 Record that a person checked goldens/<id>/ and signs off on its output. At a terminal it asks
 for your name. From an agent (a phone tap on a board or review page), pass --approved-by
@@ -18,6 +19,11 @@ Every approval says WHY (--rationale, or asked) and WHAT IT RESTS ON (--basis, o
   outcome   it produced a result someone can check; --evidence is required
 Approvals accumulate: two people approving, or a judgment approval later backed by an outcome,
 all stay on the record. Writes goldens/<id>/APPROVAL.json with the SKILL.md hash.
+
+--private-goldens <dir> (or SUPERSKILL_PRIVATE_GOLDENS) also looks for the golden in
+<dir>/<skill-name>/<id>/, where an operator keeps goldens made from real runs that are too
+personal to travel with the skill. Approving records the approval; it never changes where a
+golden came from (PROVENANCE.json), and only a golden from a real run counts for superskill.
 `;
 
 export async function run(argv) {
@@ -26,8 +32,12 @@ export async function run(argv) {
   const dir = skillDir(a._[0], "approve");
   const id = a._[1];
   if (!id) throw new UsageError("approve needs a golden id");
-  const g = readGoldens(dir).find((x) => x.id === id);
-  if (!g) return refuse(`no goldens/${id}/`);
+  const name = parseSkillFile(readFileSync(join(dir, "SKILL.md"), "utf8")).data.name || "";
+  const privateGoldens = a.flags["private-goldens"] || process.env.SUPERSKILL_PRIVATE_GOLDENS || null;
+  const found = readGoldens(dir, { privateGoldens, name }).filter((x) => x.id === id);
+  // The private copy wins when both exist: it is the one the operator was shown.
+  const g = found.find((x) => x.private) || found[0];
+  if (!g) return refuse(`no goldens/${id}/${privateGoldens ? ` (nor ${join(privateGoldens, name, id)})` : ""}`);
   if (g.input === null || g.output === null || !g.output.trim()) return refuse(`goldens/${id}/ needs an input file and a non-empty output file before it can be approved`);
   if (!(process.stdin.isTTY && process.stdout.isTTY)) {
     // Mobile first: a person approves with a tap (a board, a review page) and an agent records it.
@@ -39,7 +49,7 @@ export async function run(argv) {
     const sha = createHash("sha256").update(readFileSync(join(dir, "SKILL.md"))).digest("hex");
     const made = approvalEntry({ name, rationale: a.flags.rationale || a.flags.note, basis: a.flags.basis || "judgment", evidence: a.flags.evidence, at: clock(a.flags).toISOString(), sha });
     if (made.error) return refuse(made.error);
-    const p = join(dir, "goldens", id, "APPROVAL.json");
+    const p = join(g.dir, "APPROVAL.json");
     const existed = existsSync(p);
     writeFileSync(p, JSON.stringify(withApproval(existed ? g.approval : null, { ...made.entry, via }), null, 2) + "\n");
     process.stdout.write(`${existed ? "added an approval to" : "approved"} goldens/${id}/ by ${name} (${made.entry.basis}, via ${via})\n`);
@@ -57,7 +67,7 @@ export async function run(argv) {
     const sha = createHash("sha256").update(readFileSync(join(dir, "SKILL.md"))).digest("hex");
     const made = approvalEntry({ name, rationale, basis, evidence, at: clock(a.flags).toISOString(), sha });
     if (made.error) return refuse(made.error);
-    const p = join(dir, "goldens", id, "APPROVAL.json");
+    const p = join(g.dir, "APPROVAL.json");
     const existed = existsSync(p);
     const prior = existed ? g.approval : null;
     writeFileSync(p, JSON.stringify(withApproval(prior, made.entry), null, 2) + "\n");

package/src/commands/doctor.mjs

@@ -8,8 +8,10 @@ import { changedFiles, touchedSkills, levelDrops } from "../changed.mjs";
 export const help = `superskill doctor <path...> [options]
 
 Score one skill, a folder of skills, or a plugin (skills/*/SKILL.md).
-Levels: skill (spec-valid, hygienic) < tested (evals + triggers) < superskill
-(approved golden, misses fixed with evals, a fresh --run that beats no-skill).
+Levels: skill (spec-valid, hygienic) < tested (real evals + triggers) < superskill
+(every miss fixed with a regression eval; the suite and triggers pass against this SKILL.md
+in a fresh --run that beats no-skill; and a real-run record: at least 5 real runs, 80%
+one-shot, on one model+harness). Goldens are optional evidence.
 
 Options:
   --level <skill|tested|superskill>  target level for the exit code (default skill)
@@ -20,6 +22,15 @@ Options:
   --baseline-json <file>             a previous --json; exit 1 if any skill's level dropped
   --run                              run the evals with and without the skill (costs
                                      model calls; see "superskill doctor --run --help")
+  --private-goldens <dir>            also read goldens from <dir>/<skill-name>/<id>/ (or set
+                                     SUPERSKILL_PRIVATE_GOLDENS): real runs kept out of the skill
+  --real-runs <dir>                  read the real-run record from <dir>/<skill-name>.json (or
+                                     set SUPERSKILL_REAL_RUNS); else evals/real-runs.json, else
+                                     Freedom's skill ledger
+  --min-real-runs <n>                real runs one model+harness needs (default 5)
+  --min-one-shot <0..1>              one-shot share that pair needs (default 0.8)
+  --min-pass-rate <0..1>             suite pass rate the last --run needs (default 0.9)
+  --min-trigger-rate <0..1>          trigger evals right in the last --run (default 0.9)
   --now <iso date>                   evaluate dates as of this moment
   --help                             this text
 
@@ -34,7 +45,7 @@ export async function run(argv) {
   const level = a.flags.level || "skill";
   if (!LEVELS.includes(level)) throw new UsageError(`--level must be one of ${LEVELS.join(", ")}`);
   if (!a._.length) throw new UsageError("doctor needs a path");
-  const opts = { level, now: clock(a.flags) };
+  const opts = { level, now: clock(a.flags), privateGoldens: a.flags["private-goldens"] || process.env.SUPERSKILL_PRIVATE_GOLDENS || null, ...thresholds(a.flags) };
   if (a.flags.changed) {
     const all = a._.flatMap((p) => findSkills(p));
     const files = a._.flatMap((p) => changedFiles(p, a.flags.base));
@@ -61,3 +72,21 @@ export async function run(argv) {
   }
   return result.ok ? 0 : 1;
 }
+
+/** The superskill bar's thresholds, from flags then environment. Unset means the documented default. */
+export function thresholds(flags, env = process.env) {
+  const pick = (flag, envName, lo, hi) => {
+    const raw = flags[flag] ?? env[envName];
+    if (raw === undefined || raw === "") return undefined;
+    const n = Number(raw);
+    if (!Number.isFinite(n) || n < lo || (hi !== null && n > hi)) throw new UsageError(`--${flag} must be a number${hi === null ? ` of at least ${lo}` : ` from ${lo} to ${hi}`}`);
+    return n;
+  };
+  return {
+    realRuns: flags["real-runs"] || env.SUPERSKILL_REAL_RUNS || null,
+    minRealRuns: pick("min-real-runs", "SUPERSKILL_MIN_REAL_RUNS", 1, null),
+    minOneShot: pick("min-one-shot", "SUPERSKILL_MIN_ONE_SHOT", 0, 1),
+    minPassRate: pick("min-pass-rate", "SUPERSKILL_MIN_PASS_RATE", 0, 1),
+    minTriggerRate: pick("min-trigger-rate", "SUPERSKILL_MIN_TRIGGER_RATE", 0, 1),
+  };
+}

package/src/commands/init.mjs

@@ -1,5 +1,5 @@
 import { existsSync, mkdirSync, writeFileSync, readFileSync } from "node:fs";
-import { join } from "node:path";
+import { join, basename } from "node:path";
 import { parseArgs, UsageError } from "../args.mjs";
 import { skillDir } from "./common.mjs";
 import { MISSES_HEADER } from "../misses.mjs";
@@ -15,7 +15,9 @@ Never overwrites a file that exists.
 --from-session <file>  build the first eval and golden candidate from the session where
                        the job was done by hand. A Claude Code .jsonl gives the first
                        request and the final answer; any other file is the request.
-                       The golden waits for a person: run \`superskill approve\`.
+                       The golden waits for a person: fill in accepted.by and
+                       accepted.at in its PROVENANCE.json (who accepted the
+                       output when it ran), then \`superskill approve\`.
 `;
 
 const TRIGGER_EXAMPLE = [
@@ -65,6 +67,10 @@ export async function run(argv) {
     writeFileSync(evalsPath, JSON.stringify(doc, null, 2) + "\n");
     put(`goldens/${id}/input.md`, sessionCase.prompt + "\n");
     put(`goldens/${id}/output.md`, sessionCase.output ? sessionCase.output + "\n" : "");
+    // A real run, but nobody has said yet that its output was accepted. Until accepted.by and
+    // accepted.at are filled in (by the person, or by a tool that read their next message), this
+    // golden cannot count toward superskill however it is approved.
+    put(`goldens/${id}/PROVENANCE.json`, JSON.stringify({ source: "real-run", run: { session: basename(session), ledger_id: null, commit: null, at: null }, accepted: { by: null, at: null, signal: null, evidence: "" } }, null, 2) + "\n");
     process.stdout.write(`eval ${id} and golden candidate goldens/${id}/ written from ${session}\n`);
     process.stdout.write(`a person approves it with: superskill approve ${a._[0]} ${id}\n`);
   }

package/src/commands/miss.mjs

@@ -2,11 +2,12 @@ import { parseArgs, clock, UsageError } from "../args.mjs";
 import { skillDir, today } from "./common.mjs";
 import { readMisses, appendMisses, nextMissId } from "../misses.mjs";
 
-export const help = `superskill miss <skill> "<what happened>" [--expected "<what should have>"]
+export const help = `superskill miss <skill> "<what happened>" [--expected "<what should have>"] [--quote "<what they said>"] [--date YYYY-MM-DD]
 superskill miss import <skill> --freedom-ledger [--ledger <file>]
 
-Log a time the skill got something wrong. The entry opens today; an open miss older than
-14 days blocks the superskill level. Close it with \`superskill fix\`.
+Log a time the skill got something wrong. The entry opens today (or on --date, for a story
+from the past); an open miss blocks the superskill level. Close it with \`superskill fix\`.
+This is also where the story behind a rule goes: SKILL.md keeps the rule and a one-line why.
 `;
 
 export async function run(argv) {
@@ -18,7 +19,9 @@ export async function run(argv) {
   if (!what) throw new UsageError('miss needs a description: superskill miss <skill> "<what happened>"');
   const misses = readMisses(dir) || [];
   const id = nextMissId(misses);
-  appendMisses(dir, [{ id, date: today(clock(a.flags)), status: "open", what, expected: a.flags.expected || "" }]);
+  const date = a.flags.date || today(clock(a.flags));
+  if (!/^\d{4}-\d{2}-\d{2}$/.test(date)) throw new UsageError("--date must be YYYY-MM-DD");
+  appendMisses(dir, [{ id, date, status: "open", what, expected: a.flags.expected || "", quote: a.flags.quote || "" }]);
   process.stdout.write(`${id} logged (open). Close it with: superskill fix ${a._[0]} ${id} --eval <id>\n`);
   return 0;
 }

package/src/doctor.mjs

@@ -37,6 +37,7 @@ export function findSkills(path) {
 /** Score one loaded skill. */
 export function scoreSkill(dir, opts = {}) {
   const ctx = loadSkill(dir);
+  if (opts.privateGoldens) ctx.privateGoldens = opts.privateGoldens;
   const findings = runRules(ctx, opts.rules || allRules, opts);
   const level = computeLevel(findings);
   const up = nextLevel(level);