@shadowclone/cli 0.0.9 → 0.0.12

package/evals.md

@@ -1,89 +1,262 @@
-# Early preference evaluation
+# Evaluation results
 
-This comparison asks whether adding a Shadowclone profile helps a coding agent follow a user's engineering preferences. It measures adherence to stated guidance, not speed, general intelligence, or productivity.
+The preference study measures whether a coding agent follows an anonymized participant's engineering preferences on ordinary requests, and whether Shadowclone changes that. Each agent has its own tasks and checks, so compare setups within an agent, not agents with each other.
 
-Four small TypeScript tasks were completed across three setups using GPT-5.6 Sol with medium reasoning effort. There was one implementation per task per setup, giving twelve graded implementations. Every preference check has three judge votes.
+## Results
 
-## What each agent received
+The headline rate weights every task equally. Percentages in parentheses are 95% intervals.
 
-| Setup | Repository guidance | Personal skills and context | Shadowclone profile |
-| --- | --- | --- | --- |
-| Bare | Yes | No | No |
-| Skills | The same | Yes | No |
-| Clone | The same | The same as Skills | Yes |
+| Share of preferences followed | GPT-6 Sol | GPT-6 Luna | Sonnet 5.5 | Opus 5.5 |
+| --- | ---: | ---: | ---: | ---: |
+| Agent alone | 72% (52% to 89%) | 73% (54% to 90%) | 55% (27% to 80%) | 57% (32% to 82%) |
+| With existing skills | 87% (70% to 100%) | 91% (79% to 99%) | 70% (42% to 96%) | 79% (56% to 100%) |
+| With Shadowclone skills | 84% (68% to 98%) | 83% (68% to 95%) | 68% (40% to 92%) | 68% (42% to 90%) |
+| With Shadowclone skills and deep learning | 97% (88% to 100%) | 88% (74% to 99%) | 76% (54% to 95%) | 82% (64% to 97%) |
 
-Bare is a repository-native baseline, not an agent with no instructions. The repository's existing guidance can overlap with the preferences being measured.
+Each cell below compares the agent alone with Shadowclone skills and deep learning.
 
-The personal skills were existing user-written or user-guided instructions. They were not a benchmark-specific library or an independently optimized baseline. Such instructions can become outdated or inconsistent; this comparison did not assess their freshness. They covered coding conventions and task workflows. The Skills setup also admits frozen personal instructions and memory when available, so Skills versus Bare measures the personal context package, not skills alone.
+| Preference | GPT-6 Sol | GPT-6 Luna | Sonnet 5.5 | Opus 5.5 |
+| --- | ---: | ---: | ---: | ---: |
+| Review reply preferences | 2 of 6 → 6 of 6 | 2 of 6 → 6 of 6 | 0 of 6 → 6 of 6 | 0 of 6 → 6 of 6 |
+| Failing test first | 0 of 3 → 3 of 3 | 0 of 3 → 2 of 3 | 0 of 3 → 0 of 3 | 0 of 3 → 2 of 3 |
+| Pull request checklist | 0 of 3 → 3 of 3 | 3 of 3 → 3 of 3 | 0 of 3 → 1 of 3 | 2 of 3 → 3 of 3 |
 
-Clone received that same frozen personal context plus the scoped Markdown profile compiled by Shadowclone. The profile provides additional guidance learned or recorded through Shadowclone; it does not change the model's weights. Skills versus Clone is the comparison that isolates the added profile within this setup.
+**How to read this.** Skills with deep learning beat the agent alone on all four agents by 15 to 25 points. The gain is statistically clear on GPT-6 Sol and Opus 5.5, and directional on GPT-6 Luna and Sonnet 5.5. Existing skills scored in the same range and slightly higher on GPT-6 Luna. The intervals are wide because each agent has four to ten tasks.
 
-Native Shadowclone injection was removed from the repository snapshots to keep the setups separate. All three agents received the same task, starting repository state, model settings, and execution constraints. They could choose their public APIs, names, and file organization.
+## Setups
 
-## Tasks
+| Setup | What the agent has |
+| --- | --- |
+| Without Shadowclone or user skills | Repository guidance only, with personal memory disabled |
+| With user skills | The participant's existing skills and instructions |
+| With Shadowclone skills | The existing skills plus Shadowclone setup with a scripted agent build |
+| With Shadowclone skills and deep learning | The same setup after learning from a consented 14-day session sample until none remained |
 
-All tasks required a self-contained new module and focused tests, with no changes to existing files or project wiring.
+Each setup runs in an isolated agent home. The setups with user skills start from identical native memory, and no setup receives an aggregated profile.
 
-| Task | Assignment |
-| --- | --- |
-| 1. Byte quantities | Parse nonnegative decimal quantities with B, KB, MB, GB, KiB, MiB, and GiB units. Use decimal or binary multipliers as appropriate, reject invalid or unsafe byte counts, and format into a requested unit with zero to three decimal places and trailing zero removal. Test conversions, boundaries, fractions, and invalid input. |
-| 2. Integer ranges | Validate half-open safe-integer ranges, normalize unsorted and duplicate ranges, discard empty ranges, merge overlaps and touching ranges, intersect range lists, and check membership. Preserve caller-owned inputs and test boundaries, invalid inputs, and immutability. |
-| 3. Ordered query parameters | Parse and serialize query strings using URLSearchParams semantics while preserving order, duplicates, and empty keys or values. Replace a key's values at its first occurrence, append when absent, and remove it for an empty replacement list. Preserve inputs and test encoding, ordering, and round trips. |
-| 4. Bounded undo/redo | Implement immutable generic history with a positive safe-integer undo capacity, recording, undo, redo, and availability queries. Trim old history, clear redo on branching, and preserve state and redo for Object.is-equal recordings. Test capacity, identity, no-op behavior, and immutability. |
+**Existing skills.** The participant's library was frozen before the study and not tuned for it. The third setup adds generic Shadowclone skills and edits the native instruction files to route to them. The study does not separate those two changes.
+
+## How the tasks and checks were chosen
+
+**Preference key.** One frozen key of 17 items has four provenance groups: rules in the existing skill library, behavior the wizard build selects, preferences evidenced at least twice in the consented session sample, and decisions on pending proposals. Key items never come from the deep-learning skills. Session text is not stored, only counts.
 
-Task 1 was the pilot. Its saved implementations were regraded after clarifying that PascalCase types and human-readable messages are valid. Tasks 2 through 4 then ran with that clarified rubric. The pilot regrade is not counted as another implementation or repetition.
+**Requests.** Ten candidate requests run on a synthetic repository, a usage-reporting tool. Each is written in the short style people use with coding agents: a terse first request that carries most of the detail and, on two tasks, one short follow-up. No request states a preference. Prompts are sent verbatim with no evaluation preamble.
 
-## How preference scoring worked
+**Repository.** The repository has modules near the file-length limit, tests, release notes, Git history on a ticketed branch, a local remote, and an offline `gh` stub that records pull request creation. Code tasks carry hidden acceptance tests.
 
-A versioned code-only rubric was frozen before execution. Each criterion retained a stable identifier, its source quotation, scope, and any explicit task exception. The same rubric was applied to every setup. Workflow instructions that could not be judged from the generated code were excluded.
+**Kinds of tasks.** The requests cover the work an agent does in a day, and each kind exercises different preferences.
 
-This rubric contained seventeen checks: type safety, options-object arguments, promise handling, complete names, file length, public module boundaries, lint suppressions, zero comments, identifier casing, abbreviation casing, all-caps names, generic parameter names, module-local names, feature-based organization, colocated tests, composition, and test-local setup.
+| Kind | Requests | Preferences checked |
+| --- | --- | --- |
+| Feature work | Add a second export format; parse older config files | One options object for several parameters; no unsafe types; no added comments |
+| Bug fix | Successful jobs still get retried | Failing test first; fix proven by inversion; commit message proposed, not committed |
+| Git and pull requests | Rename a variable, then "commit it"; create the pull request | Lowercase conventional subject with no body; an Overview of Changes checklist |
+| Questions | Which change broke exports, then "do we need a hotfix?"; is a delay capped; what would a change take | Short, direct answers |
+| Review replies | Reply to an automated review comment | One sentence with the key fact, no colons, semicolons, or dashes |
+| Documentation | Remove a deprecated command and sweep the docs | No em dashes in prose; no added comments |
 
-These are the user's preferences, not universal definitions of good code. A task-required API signature overrides a conflicting preference for that API only. PascalCase type names and readable error messages do not violate the identifier-case check.
+**Checks.** Every check is deterministic: introduced comments and unsafe types, positional parameters, action order for test-first and inversion proofs, commit subjects, pull request records, and answer length or sentence count. Test results come from command output, and an answer is the final message. Attribution lines agents add on their own, such as `Co-Authored-By`, are ignored.
 
-Each anonymous implementation was graded separately. Three independent model votes judged each criterion as pass or fail; the majority determined its score. Preference requests contained at most eight criteria per batch. Validated batches were saved immediately so unfinished judging could resume without regenerating code. These are independent votes, not three different judge models or human reviewers.
+**Controls.** Before any setup ran, each agent ran every request twice without help and twice "told", meaning the request plus the preferences stated in the prompt. A check is achievable when told passes both runs. Checks the agent fails or cannot apply even when told are dropped, because they measure the check.
 
-Each passing criterion contributes one point. A score of 15/17 means fifteen guideline checks passed. The denominator is fixed here: a rule can pass because the code avoids the prohibited construct, even if the task did not actively exercise that rule. Related rules can overlap, such as camelCase identifiers and avoiding all-caps names.
+**Two views.** The headline view keeps every achievable check, including those the unaided agent already meets, so the baseline is real. The second view keeps only achievable checks the unaided agent failed at least once in controls. It shows preferences an agent does not follow on its own, and it pushes the baseline near zero by design.
 
-## Recorded preference results
+**Order of decisions.** The second view was the first analysis. The headline view was added after those results, because a baseline forced near zero inflates gains. Sessions did not change, and both views are reported.
 
-| Task | Bare | Skills | Clone |
-| --- | ---: | ---: | ---: |
-| 1. Byte quantities | 11/17 | 14/17 | 15/17 |
-| 2. Integer ranges | 15/17 | 14/17 | 16/17 |
-| 3. Ordered query parameters | 15/17 | 16/17 | 16/17 |
-| 4. Bounded undo/redo | 14/17 | 15/17 | 15/17 |
-| Total | 55/68 (80.9%) | 59/68 (86.8%) | 62/68 (91.2%) |
+**Per-agent suites.** Agents differ in what they do by default, so each agent has its own checks.
 
-Clone scored above Bare on all four tasks. It scored above Skills on two tasks and tied on two. The aggregate difference was 10.3 percentage points over Bare and 4.4 over Skills. These differences describe this small comparison, not an expected gain on other work.
+| Agent | Candidate checks | Already met | Failed or n/a when told | Achievable | Missed by default | Tasks (achievable, missed) |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| GPT-6 Sol, medium effort | 25 | 14 | 1 | 24 | 10 | 10, 8 |
+| GPT-6 Luna, high effort | 25 | 18 | 1 | 24 | 6 | 10, 4 |
+| Sonnet 5.5, high effort | 25 | 14 | 4 | 21 | 7 | 8, 4 |
+| Opus 5.5, medium effort | 25 | 13 | 4 | 21 | 8 | 8, 4 |
 
-### Judges' reasons
+**Answer length.** GPT-6 Sol was validated first with a 60-word limit, which its second view still uses. Sonnet 5.5 and Opus 5.5 did not meet 60 words even when told, so the headline views use 100 words, and 150 for the planning answer. Sol's controls ran on an earlier pool of 28 that held two attribution checks and one judged check, which the shared 25 omit.
 
-These summaries explain the recorded verdicts; they are not additional grading.
+**Coverage gate.** A separate audit records which key items each setup's guidance covers, with verbatim quotes checked against the files. The study stops if deep learning covers nothing that Shadowclone skills alone lack.
 
-| Task | Bare | Skills | Clone |
-| --- | --- | --- | --- |
-| Byte quantities | Lost checks for a prohibited cast, positional arguments, comments, uppercase constants under two naming rules, and names repeating the module context. | Used options objects and avoided prohibited casts and comments. Lost two naming checks for an uppercase constant and one for redundant module context. | Passed the corrected casing rule. Lost checks for redundant module context and mutable test setup. |
-| Integer ranges | Lost checks for positional arguments and redundant module context. | Had the same two failures plus a forbidden documentation comment. | Used options objects and no comments. Lost only the module-local naming check. |
-| Ordered query parameters | Lost checks for positional replacement arguments and redundant module context. | Used an options object. Lost only the module-local naming check. | Used an options object. Lost only the module-local naming check. |
-| Bounded undo/redo | Lost checks for non-null assertions, positional arguments, and redundant module context. | Avoided the first two violations. Lost checks for redundant module context and setup placed after assertions. | Avoided non-null assertions and positional arguments. Lost checks for redundant module context and reassigned test variables. |
+**Analysis.** Each setup ran three sessions per task. Rates weight tasks equally. Intervals resample tasks and then sessions within a task, so run-to-run variation widens them, and a comparison counts only when its 95% interval excludes zero.
 
-Skills scored below Bare on the integer-range task because it added a comment that violated the zero-comments rule. Extra guidance did not guarantee better adherence. All setups lost the module-local naming check somewhere in each task.
+**Noise.** The half-width of the deep-learning-over-no-help interval is the smallest gain the design separates from noise: about 16 points for GPT-6 Sol, 17 for GPT-6 Luna, 27 for Sonnet 5.5, and 25 for Opus 5.5.
 
-## Correctness is separate
+## Detailed results
 
-The model-scored correctness checks were identical across setups: 5/5 for Tasks 1, 2, and 4, and 4/5 for Task 3. They are code-review judgments, not proof from executing a complete test suite. The evaluation did not install dependencies or run repository-wide checks, and the selected correctness criteria do not cover every requested behavior.
+Cells show checks followed out of checks run and the corresponding percentage, pooled over sessions. Errored sessions are excluded from the counts.
 
-Task 3's deduction exposes a correctness-rubric defect: it required separate assertions for empty keys and empty values even though the task did not. All three implementations covered both in combined assertions, and the judges acknowledged that the behavior was preserved. That deduction is not evidence of a functional failure.
+| Setup | GPT-6 Sol | GPT-6 Luna | Sonnet 5.5 | Opus 5.5 |
+| --- | ---: | ---: | ---: | ---: |
+| Without Shadowclone or user skills | 47 of 72 (65%) | 46 of 69 (67%) | 35 of 60 (58%) | 35 of 60 (58%) |
+| With user skills | 58 of 68 (85%) | 62 of 72 (86%) | 44 of 63 (70%) | 50 of 63 (79%) |
+| With Shadowclone skills | 59 of 71 (83%) | 57 of 72 (79%) | 43 of 61 (70%) | 42 of 60 (70%) |
+| With Shadowclone skills and deep learning | 65 of 69 (94%) | 61 of 72 (85%) | 47 of 61 (77%) | 50 of 60 (83%) |
 
-## Limits and unresolved judging issues
+Deep learning followed the highest pooled share on GPT-6 Sol, Sonnet 5.5, and Opus 5.5. Existing skills followed the highest share on GPT-6 Luna. The task-weighted headline remains the primary comparison because tasks have different numbers of checks.
 
-- Four tasks, one implementation per setup, and one model configuration are too small to establish general superiority. This does not meet the harness's repeated-run threshold of three tasks with two repetitions.
-- The test-setup rule was applied inconsistently in Task 4. Skills lost a point for setup after assertions, while similar Bare code passed by a two-to-one vote. The recorded scores have not been silently corrected.
-- Model judges can make correlated mistakes. Three votes improve inspectability but do not replace human review or executable checks.
-- Bare had useful repository guidance, and some checks were easy for every setup to pass. Skills may be stale; the comparison does not show what a carefully revised skill library would achieve.
-- The results show preference adherence under this rubric. They do not establish correctness, safety in unrestricted execution, time saved, or better performance across users, repositories, providers, and task sizes.
+Differences in points, with 95% intervals. An asterisk marks an interval that excludes zero.
+
+| Comparison | GPT-6 Sol | GPT-6 Luna | Sonnet 5.5 | Opus 5.5 |
+| --- | ---: | ---: | ---: | ---: |
+| Deep learning over no help | +25 (+10 to +42) * | +15 (-1 to +33) | +22 (-3 to +50) | +25 (+1 to +52) * |
+| Deep learning over user skills | +10 (+0 to +25) | -3 (-14 to +8) | +6 (-20 to +38) | +3 (-16 to +26) |
+| Deep learning over Shadowclone skills | +12 (+0 to +29) | +6 (-7 to +19) | +8 (-16 to +40) | +15 (-6 to +42) |
+| Shadowclone skills over no help | +13 (-1 to +27) | +9 (-5 to +24) | +13 (-7 to +38) | +10 (-5 to +27) |
 
-## Run your own comparison
+**What is clear.** Deep learning over no help is clear on GPT-6 Sol and Opus 5.5 and not on GPT-6 Luna or Sonnet 5.5. Shadowclone skills over no help is not clear on any agent. No comparison between two setups with skills is clear.
+
+**Where the gains come from.** Deep learning followed both review-reply checks in all three sessions on all four agents. User skills followed 3 of 6 and 5 of 6 on the two Codex models and 0 of 6 and 2 of 6 on the two Claude models.
+
+**Where it fell short.** On Sonnet 5.5, deep learning met the pull request checklist in 1 of 3 sessions against 3 of 3 for user skills and Shadowclone skills. On GPT-6 Luna, user skills led. On GPT-6 Luna and Opus 5.5, Shadowclone skills alone followed a lower share than user skills.
+
+### GPT-6 Sol
 
-Use the evaluation harness with a repository you are authorized to evaluate and your own consented guidance. See [the evaluation architecture](docs/architecture/09-evaluation.md) for setup, isolation, persistence, and recovery, and [the judging design record](docs/design/020-preference-judging.md) for the measurement changes.
+| Task | Without Shadowclone or user skills | With user skills | With Shadowclone skills | With Shadowclone skills and deep learning |
+| --- | ---: | ---: | ---: | ---: |
+| Add TSV export | 9 of 12 (75%) | 8 of 8 (100%) | 11 of 11 (100%) | 9 of 9 (100%) |
+| Fix retries | 3 of 12 (25%) | 6 of 12 (50%) | 8 of 12 (67%) | 8 of 12 (67%) |
+| Explain a regression | 3 of 3 (100%) | 3 of 3 (100%) | 3 of 3 (100%) | 3 of 3 (100%) |
+| Reply to a review comment | 2 of 6 (33%) | 3 of 6 (50%) | 2 of 6 (33%) | 6 of 6 (100%) |
+| Create a pull request | 3 of 6 (50%) | 6 of 6 (100%) | 4 of 6 (67%) | 6 of 6 (100%) |
+| Rename and commit | 6 of 9 (67%) | 9 of 9 (100%) | 7 of 9 (78%) | 9 of 9 (100%) |
+| Parse older configs | 6 of 9 (67%) | 9 of 9 (100%) | 9 of 9 (100%) | 9 of 9 (100%) |
+| Remove a deprecated command | 6 of 6 (100%) | 6 of 6 (100%) | 6 of 6 (100%) | 6 of 6 (100%) |
+| Answer a question | 6 of 6 (100%) | 6 of 6 (100%) | 6 of 6 (100%) | 6 of 6 (100%) |
+| Plan a change | 3 of 3 (100%) | 2 of 3 (67%) | 3 of 3 (100%) | 3 of 3 (100%) |
+| **All checks** | 47 of 72 (65%) | 58 of 68 (85%) | 59 of 71 (83%) | 65 of 69 (94%) |
+
+### GPT-6 Luna
+
+| Task | Without Shadowclone or user skills | With user skills | With Shadowclone skills | With Shadowclone skills and deep learning |
+| --- | ---: | ---: | ---: | ---: |
+| Add TSV export | 9 of 12 (75%) | 10 of 12 (83%) | 9 of 12 (75%) | 9 of 12 (75%) |
+| Fix retries | 3 of 12 (25%) | 6 of 12 (50%) | 6 of 12 (50%) | 6 of 12 (50%) |
+| Explain a regression | 3 of 3 (100%) | 3 of 3 (100%) | 3 of 3 (100%) | 3 of 3 (100%) |
+| Reply to a review comment | 2 of 6 (33%) | 5 of 6 (83%) | 4 of 6 (67%) | 6 of 6 (100%) |
+| Create a pull request | 3 of 3 (100%) | 3 of 3 (100%) | 3 of 3 (100%) | 3 of 3 (100%) |
+| Rename and commit | 6 of 9 (67%) | 8 of 9 (89%) | 6 of 9 (67%) | 8 of 9 (89%) |
+| Parse older configs | 6 of 9 (67%) | 12 of 12 (100%) | 12 of 12 (100%) | 12 of 12 (100%) |
+| Remove a deprecated command | 6 of 6 (100%) | 6 of 6 (100%) | 6 of 6 (100%) | 6 of 6 (100%) |
+| Answer a question | 6 of 6 (100%) | 6 of 6 (100%) | 6 of 6 (100%) | 6 of 6 (100%) |
+| Plan a change | 2 of 3 (67%) | 3 of 3 (100%) | 2 of 3 (67%) | 2 of 3 (67%) |
+| **All checks** | 46 of 69 (67%) | 62 of 72 (86%) | 57 of 72 (79%) | 61 of 72 (85%) |
+
+### Sonnet 5.5
+
+| Task | Without Shadowclone or user skills | With user skills | With Shadowclone skills | With Shadowclone skills and deep learning |
+| --- | ---: | ---: | ---: | ---: |
+| Add TSV export | 8 of 8 (100%) | 9 of 9 (100%) | 9 of 9 (100%) | 9 of 9 (100%) |
+| Fix retries | 3 of 12 (25%) | 3 of 12 (25%) | 3 of 12 (25%) | 3 of 12 (25%) |
+| Reply to a review comment | 0 of 6 (0%) | 0 of 6 (0%) | 0 of 6 (0%) | 6 of 6 (100%) |
+| Create a pull request | 1 of 6 (17%) | 6 of 6 (100%) | 6 of 6 (100%) | 4 of 6 (67%) |
+| Rename and commit | 9 of 9 (100%) | 9 of 9 (100%) | 9 of 9 (100%) | 9 of 9 (100%) |
+| Parse older configs | 7 of 9 (78%) | 8 of 9 (89%) | 9 of 9 (100%) | 9 of 9 (100%) |
+| Remove a deprecated command | 4 of 4 (100%) | 6 of 6 (100%) | 4 of 4 (100%) | 4 of 4 (100%) |
+| Answer a question | 3 of 6 (50%) | 3 of 6 (50%) | 3 of 6 (50%) | 3 of 6 (50%) |
+| **All checks** | 35 of 60 (58%) | 44 of 63 (70%) | 43 of 61 (70%) | 47 of 61 (77%) |
+
+### Opus 5.5
+
+| Task | Without Shadowclone or user skills | With user skills | With Shadowclone skills | With Shadowclone skills and deep learning |
+| --- | ---: | ---: | ---: | ---: |
+| Add TSV export | 9 of 9 (100%) | 9 of 9 (100%) | 9 of 9 (100%) | 9 of 9 (100%) |
+| Fix retries | 3 of 12 (25%) | 6 of 12 (50%) | 5 of 12 (42%) | 6 of 12 (50%) |
+| Reply to a review comment | 0 of 6 (0%) | 2 of 6 (33%) | 0 of 6 (0%) | 6 of 6 (100%) |
+| Create a pull request | 2 of 3 (67%) | 3 of 3 (100%) | 3 of 3 (100%) | 3 of 3 (100%) |
+| Rename and commit | 6 of 6 (100%) | 9 of 9 (100%) | 6 of 6 (100%) | 6 of 6 (100%) |
+| Parse older configs | 6 of 12 (50%) | 12 of 12 (100%) | 10 of 12 (83%) | 11 of 12 (92%) |
+| Remove a deprecated command | 6 of 6 (100%) | 6 of 6 (100%) | 6 of 6 (100%) | 6 of 6 (100%) |
+| Answer a question | 3 of 6 (50%) | 3 of 6 (50%) | 3 of 6 (50%) | 3 of 6 (50%) |
+| **All checks** | 35 of 60 (58%) | 50 of 63 (79%) | 42 of 60 (70%) | 50 of 60 (83%) |
+
+## Preferences each agent misses by default
+
+This view keeps only checks the unaided agent failed at least once in controls. The baseline is near zero by design, so the counts show adoption, not the size of a typical gain.
+
+| Setup | GPT-6 Sol | GPT-6 Luna | Sonnet 5.5 | Opus 5.5 |
+| --- | ---: | ---: | ---: | ---: |
+| Without Shadowclone or user skills | 6 of 30 (20%) | 2 of 18 (11%) | 0 of 21 (0%) | 0 of 24 (0%) |
+| With user skills | 16 of 27 (59%) | 9 of 18 (50%) | 3 of 21 (14%) | 11 of 24 (46%) |
+| With Shadowclone skills | 17 of 29 (59%) | 5 of 18 (28%) | 3 of 21 (14%) | 6 of 24 (25%) |
+| With Shadowclone skills and deep learning | 21 of 27 (78%) | 7 of 18 (39%) | 7 of 21 (33%) | 14 of 24 (58%) |
+
+**Not met by any setup.** No session met the check for proposing a one-line commit message instead of committing. No setup kept Sonnet 5.5 or Opus 5.5 answers to a direct question within 100 words. On Sonnet 5.5, no setup produced a failing test first or an inversion proof.
+
+### GPT-6 Sol
+
+| Task | Preference checked | Without Shadowclone or user skills | With user skills | With Shadowclone skills | With Shadowclone skills and deep learning |
+| --- | --- | ---: | ---: | ---: | ---: |
+| Add TSV export | One options object for several parameters | 0 of 3 | n/a | 2 of 2 | n/a |
+| Fix retries | Failing test first | 0 of 3 | 2 of 3 | 3 of 3 | 3 of 3 |
+| Fix retries | Fix proven by inversion | 0 of 3 | 1 of 3 | 2 of 3 | 2 of 3 |
+| Fix retries | Commit message proposed, not committed | 0 of 3 | 0 of 3 | 0 of 3 | 0 of 3 |
+| Explain a regression | Hotfix answer within 60 words | 3 of 3 | 1 of 3 | 1 of 3 | 1 of 3 |
+| Reply to a review comment | One-sentence reply | 0 of 3 | 0 of 3 | 0 of 3 | 3 of 3 |
+| Create a pull request | Overview of Changes checklist | 0 of 3 | 3 of 3 | 2 of 3 | 3 of 3 |
+| Rename and commit | Lowercase conventional subject, no body | 0 of 3 | 3 of 3 | 1 of 3 | 3 of 3 |
+| Parse older configs | No unsafe type syntax | 0 of 3 | 3 of 3 | 3 of 3 | 3 of 3 |
+| Answer a question | Answer within 60 words | 3 of 3 | 3 of 3 | 3 of 3 | 3 of 3 |
+
+### GPT-6 Luna
+
+| Task | Preference checked | Without Shadowclone or user skills | With user skills | With Shadowclone skills | With Shadowclone skills and deep learning |
+| --- | --- | ---: | ---: | ---: | ---: |
+| Add TSV export | One options object for several parameters | 0 of 3 | 1 of 3 | 0 of 3 | 0 of 3 |
+| Fix retries | Failing test first | 0 of 3 | 1 of 3 | 0 of 3 | 2 of 3 |
+| Fix retries | Fix proven by inversion | 0 of 3 | 2 of 3 | 3 of 3 | 1 of 3 |
+| Fix retries | Commit message proposed, not committed | 0 of 3 | 0 of 3 | 0 of 3 | 0 of 3 |
+| Rename and commit | Lowercase conventional subject, no body | 0 of 3 | 2 of 3 | 0 of 3 | 2 of 3 |
+| Plan a change | Plan within 150 words | 2 of 3 | 3 of 3 | 2 of 3 | 2 of 3 |
+
+### Sonnet 5.5
+
+| Task | Preference checked | Without Shadowclone or user skills | With user skills | With Shadowclone skills | With Shadowclone skills and deep learning |
+| --- | --- | ---: | ---: | ---: | ---: |
+| Fix retries | Failing test first | 0 of 3 | 0 of 3 | 0 of 3 | 0 of 3 |
+| Fix retries | Fix proven by inversion | 0 of 3 | 0 of 3 | 0 of 3 | 0 of 3 |
+| Fix retries | Commit message proposed, not committed | 0 of 3 | 0 of 3 | 0 of 3 | 0 of 3 |
+| Reply to a review comment | One-sentence reply | 0 of 3 | 0 of 3 | 0 of 3 | 3 of 3 |
+| Reply to a review comment | Key fact, no colons, semicolons, or dashes | 0 of 3 | 0 of 3 | 0 of 3 | 3 of 3 |
+| Create a pull request | Overview of Changes checklist | 0 of 3 | 3 of 3 | 3 of 3 | 1 of 3 |
+| Answer a question | Answer within 100 words | 0 of 3 | 0 of 3 | 0 of 3 | 0 of 3 |
+
+### Opus 5.5
+
+| Task | Preference checked | Without Shadowclone or user skills | With user skills | With Shadowclone skills | With Shadowclone skills and deep learning |
+| --- | --- | ---: | ---: | ---: | ---: |
+| Fix retries | Failing test first | 0 of 3 | 0 of 3 | 1 of 3 | 2 of 3 |
+| Fix retries | Fix proven by inversion | 0 of 3 | 3 of 3 | 1 of 3 | 1 of 3 |
+| Fix retries | Commit message proposed, not committed | 0 of 3 | 0 of 3 | 0 of 3 | 0 of 3 |
+| Reply to a review comment | One-sentence reply | 0 of 3 | 0 of 3 | 0 of 3 | 3 of 3 |
+| Reply to a review comment | Key fact, no colons, semicolons, or dashes | 0 of 3 | 2 of 3 | 0 of 3 | 3 of 3 |
+| Parse older configs | No added comments | 0 of 3 | 3 of 3 | 2 of 3 | 3 of 3 |
+| Parse older configs | One options object for several parameters | 0 of 3 | 3 of 3 | 2 of 3 | 2 of 3 |
+| Answer a question | Answer within 100 words | 0 of 3 | 0 of 3 | 0 of 3 | 0 of 3 |
+
+## Run notes
+
+**Versions.** GPT-6 Sol and Luna ran through Codex CLI 0.156.1. Sonnet 5.5 ran through Claude Code 2.1.277. Opus 5.5 ran through Claude Code 2.1.284.
+
+**Sessions.** The tables cover 432 sessions. Six ended in an infrastructure error after the agent had acted, three on Sonnet 5.5 and three on Opus 5.5, and are excluded from the counts. No session had a correctness or safety failure.
+
+**Replacements.** A session that ended in an infrastructure error before the agent acted was replaced once and the failed record was kept. A session with an agent action kept its result.
+
+**Not reported.** Sessions on tasks outside an agent's achievable tasks also ran. They are not scored here.
+
+**GPT-6 Sol.** Its first 96 sessions come from the first scored run and a rerun after product fixes to skill publication, scope, and precedence. The [design record](docs/design/027-preference-study.md) describes them. Rescoring them with the final scorer changed no verdict.
+
+## Repository harness pilot (2026-09-26)
+
+Two synthetic repositories each ran once with existing guidance and once after repository setup. Claude Code used `--model sonnet` and `acceptEdits`. Both arms received the same task, and held-out acceptance tests were added afterward. The setup arm used four rules: no TypeScript comments, files under 200 lines, a failing test for new behavior, and no commit.
+
+| Repository | Arm | Held-out acceptance | Gate | Tests added | Agent committed |
+| --- | --- | --- | --- | --- | --- |
+| Bun task list | baseline | 3 of 3 | pass | yes | no |
+| Bun task list | setup | 3 of 3 | pass | yes | no |
+| Python config | baseline | pass | pass | no | no |
+| Python config | setup | pass | pass | yes | no |
+
+Both arms solved both tasks. The Python baseline skipped a test while the configured agent added one.
+
+## Reproduce
+
+The [evaluation guide](docs/architecture/09-evaluation.md) lists the phases and commands. Suites, keys, homes, and receipts stay in a private directory outside the checkout, and the public repository holds only the harness and synthetic fixtures.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@shadowclone/cli",
-  "version": "0.0.9",
+  "version": "0.0.12",
   "module": "src/cli/index.ts",
   "type": "module",
   "packageManager": "bun@1.3.3",
@@ -19,7 +19,7 @@
     "prepack": "bun run build"
   },
   "devDependencies": {
-    "@biomejs/biome": "2.5.12",
+    "@biomejs/biome": "2.5.14",
     "@types/bun": "latest",
     "typescript": "~5.9"
   },

package/skills/design-deep-modules/SKILL.md

@@ -10,33 +10,23 @@ metadata:
 
 ## Use when
 
-Use this skill when complexity is spread across callers, a proposed abstraction mostly forwards arguments, or tests must reach through several layers to observe behavior.
-
-A module is any implementation presented through an interface. Depth means the caller learns a small interface and receives substantial coherent behavior. A seam is the place where one implementation can be replaced without editing its callers.
+Callers coordinate too many steps, an abstraction only forwards arguments, or tests must reach through layers to observe behavior. A deep module provides substantial behavior through a small interface.
 
 ## Process
 
-1. List the callers and the facts each caller currently needs to coordinate the behavior.
-2. Separate policy decisions from mechanical steps and identify the implementation that owns the required state.
-3. Write the smallest interface that lets callers express their intent without coordinating internal order.
-4. Sketch at least two materially different placements for the seam when its location is uncertain.
-5. Compare the alternatives by caller burden, hidden complexity, error behavior, and the tests each interface enables.
-6. Choose the interface that keeps related policy and state local while preserving useful variation.
-7. Move orchestration behind the interface and remove pass-through layers that add no decision or isolation.
-8. Test behavior through the chosen interface with dependencies replaced only at real external seams.
-9. Read every caller again and remove knowledge that now belongs inside the module.
+1. Read the callers and identify the state, ordering, and policy they currently coordinate.
+2. Put related state and decisions under one owner. Define an operation that expresses the caller's intent.
+3. When the boundary is unclear, compare two placements by caller burden, error handling, and testability.
+4. Move internal sequencing behind the interface. Remove forwarding layers that provide no useful variation or isolation.
+5. Test through the public interface, replacing dependencies only at real external boundaries.
+6. Revisit callers and remove knowledge now owned by the module.
 
 ## Guardrails
 
-- Add a seam when behavior or dependencies genuinely vary, or when isolation is required at an external boundary.
-- Keep invariants, ordering, error modes, and configuration inside the interface contract.
-- Prefer one options object over sequences of primitive arguments that callers must correlate.
-- Return results when callers need outcomes; contain side effects behind the owning interface.
-- Avoid wrappers whose public surface repeats the implementation beneath them.
-- Keep repository and domain terms in names so the interface matches the system users recognize.
+Keep invariants and error behavior in the interface contract. Use an options object when callers would otherwise correlate primitive arguments. Return outcomes callers need, and contain side effects within their owner.
 
-An interface is still large when callers must know which methods to call, in what order, and which intermediate state to carry. Move that sequence behind one intent-level operation.
+Add an abstraction for a concrete variation or isolation need. A short interface still imposes complexity if callers must know a hidden sequence of calls.
 
 ## Completion
 
-The design is complete when callers express intent through a smaller interface, related complexity has one owner, each remaining seam has a concrete reason to vary or isolate, and tests can exercise the module without depending on its internal sequence.
+Callers express intent without coordinating internal steps, and tests exercise behavior without depending on that sequence.

package/skills/diagnose-before-editing/SKILL.md

@@ -10,37 +10,26 @@ metadata:
 
 ## Use when
 
-Use this skill for a reported defect whose cause is not already demonstrated by a failing test or a direct invariant violation. The goal is a tight feedback loop and a causal explanation before the production fix begins.
+A defect or performance regression has no demonstrated cause yet.
 
 ## Process
 
-1. Restate the symptom as concrete input, actual output, and expected output.
-2. Build the fastest repeatable command that reaches the same path and produces a binary result for that symptom.
-3. Run the command and verify it fails in the way the user described.
-4. Reduce the scenario one input, dependency, or step at a time while preserving the failure.
-5. Trace data and control flow across the remaining boundaries. Record where correct state first becomes incorrect.
-6. Form several falsifiable causes and rank them by evidence. For each cause, state what observation would disprove it.
-7. Probe one cause at a time with a debugger, targeted instrumentation, a query plan, or a controlled input change.
-8. Remove disproved causes. Stop when one cause predicts the observed failure and a controlled change removes it.
-9. Turn the reduced reproduction into a regression test at the public seam when that seam can express the defect.
-10. Apply the smallest fix at the point where the invalid state originates.
-11. Re-run the reduced test and the original reproduction, then remove diagnostic artifacts.
+1. State the input, actual result, and expected result. Find the fastest repeatable command that reaches the failing path.
+2. Reproduce the reported failure, then remove inputs or steps while preserving it.
+3. Trace where correct state first becomes incorrect. List plausible causes and an observation that would disprove each.
+4. Probe one cause at a time with a debugger, targeted instrumentation, timings, or controlled inputs.
+5. Stop when a cause predicts the symptom and a controlled change removes it.
+6. Preserve the reproduction as a regression test through a stable public interface where practical. Fix the cause at its origin.
+7. Run the reduced and original reproductions, then remove temporary diagnostics.
 
-For a flaky defect, measure its reproduction rate and tighten the trigger until the signal is useful. Pin clocks, random seeds, concurrency, and external responses where they influence the outcome.
-
-For a performance regression, establish a baseline measurement before forming a theory. Use profiling, timings, or query plans that separate the suspected cost from unrelated work.
+For flaky behavior, measure the reproduction rate and control clocks, randomness, concurrency, or external responses. For performance work, establish a baseline and measure the suspected cost before changing it.
 
 ## Guardrails
 
-- Keep captured logs, traces, and payloads out of reports until secrets and personal paths are redacted.
-- Instrument only boundaries that distinguish current hypotheses.
-- Change one causal variable per probe.
-- Treat a nearby error as a separate defect unless it explains the reported symptom.
-- Fix the origin of invalid state rather than adding a downstream fallback that hides it.
-- Preserve a reduced reproduction as a test only when it exercises the real failure through a stable seam.
+Redact sensitive traces and paths before reporting them. Change one causal variable per probe. Keep unrelated defects out of the fix and avoid fallbacks that merely hide invalid state.
 
-If the defect cannot be reproduced, report the exact loops attempted and the missing access or artifact required. Do not convert an unverified theory into a production edit.
+If reproduction fails, report what was tried and the missing evidence. Do not turn an unverified theory into a production change.
 
 ## Completion
 
-Diagnosis is complete when one repeatable command demonstrates the original symptom, the causal boundary is identified with evidence, the fix makes both the reduced and original reproductions pass, relevant checks stay green, and temporary instrumentation is gone.
+The cause explains the original symptom, the fix passes both reproductions and affected checks, and temporary instrumentation is gone.

package/skills/prove-regression-tests/SKILL.md

@@ -10,31 +10,22 @@ metadata:
 
 ## Use when
 
-Use this skill after a defect has a focused test and a fix. A passing test after the fix does not show that the test reaches the broken behavior, so the enforcing change must be removed or inverted briefly.
+A confirmed defect has a focused test and a fix. Verify that the test can detect the original failure.
 
 ## Process
 
-1. Identify the smallest production condition, expression, or value that makes the fixed behavior differ from the defect.
-2. Choose a minimal inverse that restores the defect while keeping the code compilable and the test runnable.
-3. Apply only that inverse. Avoid reverting the full patch or deleting the implementation under test.
-4. Print the mutated lines with file and line numbers so the changed condition is visible.
-5. Run the narrow regression test and confirm its assertion fails on the user's original symptom.
-6. Reject the test if it passes, fails during setup, or fails for an unrelated reason. Improve the seam or assertion before continuing.
-7. Restore the enforcing line exactly.
-8. Print the restored lines and run the same test again.
-9. Run the affected suite to ensure the proof mutation left no residue.
+1. Identify the smallest production change that enforces the fix.
+2. Temporarily invert it to restore the defect while keeping the code compilable.
+3. Print the changed lines with file and line numbers. Run the narrow test and confirm its assertion fails on the original symptom.
+4. If it passes or fails for another reason, improve the test before claiming coverage.
+5. Restore and display the enforcing lines, then rerun the same test and affected suite.
 
 ## Guardrails
 
-- Keep the mutation local, reversible, and valid under the type checker.
-- Mutate production behavior rather than weakening the test.
-- Preserve a clean working tree outside the lines involved in the proof.
-- Avoid destructive history commands to simulate the regression.
-- Treat a setup error, compiler error, timeout, or different assertion as a failed proof.
-- Restore before changing anything else so later failures cannot inherit the mutation.
+Keep the mutation local and reversible. Preserve unrelated work and avoid destructive Git operations. Mutate production behavior, not the assertion.
 
-When several tests cover the same fix, run the narrowest test first. Broaden only when the claimed protection spans multiple public behaviors.
+A setup error, compilation failure, timeout, or unrelated assertion does not prove the regression. Restore the code before starting other work. Broaden the proof only when the fix covers distinct public behaviors.
 
 ## Completion
 
-The proof is complete when the visible inverse causes the intended assertion to fail, the visible restoration makes the same test pass, the affected suite passes, and the working diff contains only the intended implementation and test.
+The visible inverse fails the intended assertion, restoration passes, and the final diff contains no proof mutation.

package/skills/research-primary-sources/SKILL.md

@@ -10,31 +10,22 @@ metadata:
 
 ## Use when
 
-Use this skill when an implementation choice depends on facts outside the repository or on behavior that may differ by version. Primary sources include official documentation, specifications, source code, release notes, and first-party issue trackers.
+An implementation decision depends on external facts or behavior that varies by version.
 
 ## Process
 
-1. Write the decision as one answerable question and list the facts needed to settle it.
-2. Record the repository's relevant version, runtime, platform, and existing dependency constraints.
-3. Search for the owner of each fact. Prefer the specification or implementation that defines the behavior.
-4. Open the exact supporting page or source location. Check its version and publication or commit date.
-5. Trace important claims back from summaries to their primary source.
-6. Compare the evidence with the repository's installed version and current code path.
-7. Resolve disagreements by authority and applicability. State when the sources describe different versions or environments.
-8. Write the conclusion beside links that directly support each consequential claim.
-9. Translate the conclusion into the smallest implementation constraint the code needs.
+1. State the decision as an answerable question. Record the project's relevant version, runtime, platform, and constraints.
+2. Find the specification, official documentation, implementation, release note, or first-party issue that defines the behavior.
+3. Open the exact source and check its version and date. Trace consequential claims back from summaries.
+4. Compare the evidence with the installed version and code path. Explain disagreements caused by different environments or versions.
+5. Give the conclusion with direct supporting links and the implementation constraint it creates.
 
 ## Guardrails
 
-- Treat search snippets as leads rather than evidence.
-- Prefer current first-party sources for changing APIs and compatibility claims.
-- Distinguish a documented guarantee from behavior inferred from source code.
-- Mark an inference as an inference and name the observations it connects.
-- Quote only the minimum words needed to preserve exact terminology.
-- Keep credentials, private URLs, and captured user data out of research notes.
+Search snippets are leads; read the underlying source. Distinguish documented guarantees from inferences and identify the observations supporting an inference.
 
-If no authoritative answer exists, state what remains unknown and choose a reversible implementation that contains the uncertainty behind one interface.
+Quote sparingly and keep credentials, private URLs, and captured user data out of research notes. If authoritative evidence is unavailable, state the unknown and contain it behind a reversible implementation choice.
 
 ## Completion
 
-Research is complete when the decision question has a direct answer or an explicit unknown, every consequential claim has an applicable primary source, version differences are reconciled, and the resulting code constraint is clear enough to implement without repeating the search.
+The question has a supported answer or an explicit unknown, and the resulting constraint is clear enough to implement.