@shadowclone/cli 0.0.11 → 0.0.13

package/evals.md

@@ -1,84 +1,266 @@
 # Evaluation results
 
-These small studies measure adherence to stated engineering preferences. They do not establish productivity gains or general superiority. Both used the earlier profile-based system; the current [skills environment protocol](docs/architecture/09-evaluation.md#skills-environment-protocol) evaluates maintained skills separately.
+The preference study measured whether a coding agent followed one participant's engineering preferences on ordinary requests. These results were recomputed from the private receipts on 2026-09-30 using scorer `session-bootstrap-2`, without new model calls. Unknown and not-applicable checks are excluded consistently, complete sessions are resampled within tasks, and observable actions from incomplete sessions retain their checks.
 
-## Early profile comparison
+## Results
 
-GPT-5.6 Sol, at medium reasoning effort, completed four TypeScript tasks in three setups. One implementation per task and setup produced twelve graded implementations, with three judge votes per preference check.
+The headline weights every task equally. Parentheses show the 95% percentile bootstrap interval. Deep learning over existing user skills is the relevant incremental comparison; its interval includes zero for every model.
 
-| Setup | Repository guidance | Personal skills and context | Shadowclone profile |
-| --- | --- | --- | --- |
-| Bare | Yes | No | No |
-| Skills | The same | Yes | No |
-| Clone | The same | The same as Skills | Yes |
+| Share of preferences followed | GPT-6 Sol | GPT-6 Luna | Sonnet 5.5 | Opus 5.5 |
+| --- | ---: | ---: | ---: | ---: |
+| Agent alone | 72% (54% to 88%) | 73% (55% to 90%) | 59% (31% to 85%) | 59% (34% to 82%) |
+| With existing skills | 87% (72% to 100%) | 91% (79% to 99%) | 70% (44% to 94%) | 79% (58% to 94%) |
+| With Shadowclone skills | 84% (68% to 98%) | 83% (68% to 95%) | 72% (44% to 94%) | 72% (46% to 94%) |
+| With Shadowclone skills and deep learning | 97% (89% to 100%) | 88% (75% to 99%) | 80% (59% to 100%) | 86% (69% to 100%) |
 
-Bare already had useful repository instructions. Skills added existing user-written or user-guided instructions, including frozen personal memory where available. Their freshness was not assessed. Clone added the scoped Markdown profile; it did not change model weights.
+Each cell below compares the agent alone with Shadowclone skills and deep learning.
 
-All setups used the same task, starting state, model settings, and execution constraints. Native Shadowclone injections were removed from snapshots. Each task required a new self-contained module and focused tests, without changing existing files or project wiring.
+| Preference | GPT-6 Sol | GPT-6 Luna | Sonnet 5.5 | Opus 5.5 |
+| --- | ---: | ---: | ---: | ---: |
+| Review reply preferences | 2 of 6 → 6 of 6 | 2 of 6 → 6 of 6 | 0 of 6 → 6 of 6 | 0 of 6 → 6 of 6 |
+| Failing test first | 0 of 3 → 3 of 3 | 0 of 3 → 2 of 3 | 0 of 3 → 0 of 3 | 0 of 3 → 2 of 3 |
+| Pull request checklist | 0 of 3 → 3 of 3 | 3 of 3 → 3 of 3 | 0 of 3 → 1 of 3 | 2 of 3 → 3 of 3 |
 
-| Task | Assignment |
+The first-time arm is below existing skills on three models and slightly above on Sonnet after rescoring. Deep learning recovered a specific review-reply preference, but these results establish no advantage over an existing skill library. The earlier calculation had first-time estimates below existing skills on all four models; that conclusion changes when incomplete sessions are scored consistently.
+
+## Setups
+
+| Setup | What the agent has |
 | --- | --- |
-| 1. Byte quantities | Parse nonnegative decimal quantities with B, KB, MB, GB, KiB, MiB, and GiB units. Use decimal or binary multipliers as appropriate, reject invalid or unsafe byte counts, and format into a requested unit with zero to three decimal places and trailing zero removal. Test conversions, boundaries, fractions, and invalid input. |
-| 2. Integer ranges | Validate half-open safe-integer ranges, normalize unsorted and duplicate ranges, discard empty ranges, merge overlaps and touching ranges, intersect range lists, and check membership. Preserve caller-owned inputs and test boundaries, invalid inputs, and immutability. |
-| 3. Ordered query parameters | Parse and serialize query strings using URLSearchParams semantics while preserving order, duplicates, and empty keys or values. Replace a key's values at its first occurrence, append when absent, and remove it for an empty replacement list. Preserve inputs and test encoding, ordering, and round trips. |
-| 4. Bounded undo/redo | Implement immutable generic history with a positive safe-integer undo capacity, recording, undo, redo, and availability queries. Trim old history, clear redo on branching, and preserve state and redo for Object.is-equal recordings. Test capacity, identity, no-op behavior, and immutability. |
+| Without Shadowclone or user skills | Repository guidance only, with personal memory disabled |
+| With user skills | The participant's existing skills and instructions |
+| With Shadowclone skills | The existing skills plus Shadowclone setup with a scripted agent build |
+| With Shadowclone skills and deep learning | The same setup after learning from a consented 14-day session sample until none remained |
 
-The byte-quantity pilot was regraded after clarifying that PascalCase types and readable error messages were valid. The other tasks used that clarified rubric. Regrading did not add another implementation or repetition.
+Each setup runs in an isolated agent home. The setups with user skills start from identical native memory, and no setup receives an aggregated profile.
 
-### Scoring
+**Existing skills.** The participant's library was frozen before the study and not tuned for it. The third setup adds generic Shadowclone skills and edits the native instruction files to route to them. The study does not separate those two changes.
 
-A frozen code-only rubric retained each criterion's source quotation, scope, and task exceptions. Its 17 checks covered type safety, options-object arguments, promises, full names, file length, public module boundaries, suppressions, comments, identifier casing, abbreviation casing, all-caps names, generic names, module-local names, feature organization, colocated tests, composition, and test-local setup.
+## How the tasks and checks were chosen
 
-These are one user's preferences. Workflow instructions that could not be judged from code were excluded. A task-required signature overrode a conflicting preference only for that public API.
+**Preference key.** One frozen key of 17 items has four provenance groups: rules in the existing skill library, behavior the wizard build selects, preferences evidenced at least twice in the consented session sample, and decisions on pending proposals. Key items never come from the deep-learning skills. Session text is not stored, only counts.
 
-Anonymous candidates received three model votes per criterion, with a majority result. These were separate calls, not three different models or human reviewers. Requests contained at most eight criteria per batch, and validated batches were saved for recovery. Each pass earned one point; the fixed denominator allowed a rule to pass when the task never exercised the prohibited construct. Some naming checks overlapped.
+**Requests.** Ten candidate requests run on a synthetic repository, a usage-reporting tool. Each is written in the short style people use with coding agents: a terse first request that carries most of the detail and, on two tasks, one short follow-up. No request states a preference. Prompts are sent verbatim with no evaluation preamble.
 
-### Recorded results
+**Repository.** The repository has modules near the file-length limit, tests, release notes, Git history on a ticketed branch, a local remote, and an offline `gh` stub that records pull request creation. Code tasks carry hidden acceptance tests.
 
-| Task | Bare | Skills | Clone |
-| --- | ---: | ---: | ---: |
-| Byte quantities | 11/17 | 14/17 | 15/17 |
-| Integer ranges | 15/17 | 14/17 | 16/17 |
-| Ordered query parameters | 15/17 | 16/17 | 16/17 |
-| Bounded undo/redo | 14/17 | 15/17 | 15/17 |
-| Total | 55/68 (80.9%) | 59/68 (86.8%) | 62/68 (91.2%) |
+**Kinds of tasks.** The requests cover the work an agent does in a day, and each kind exercises different preferences.
 
-Clone exceeded Skills on two tasks and tied on two. The aggregate differences were 10.3 percentage points over Bare and 4.4 over Skills. Those are observations from this sample, not expected gains on other work.
+| Kind | Requests | Preferences checked |
+| --- | --- | --- |
+| Feature work | Add a second export format; parse older config files | One options object for several parameters; no unsafe types; no added comments |
+| Bug fix | Successful jobs still get retried | Failing test first; fix proven by inversion; commit message proposed, not committed |
+| Git and pull requests | Rename a variable, then "commit it"; create the pull request | Lowercase conventional subject with no body; an Overview of Changes checklist |
+| Questions | Which change broke exports, then "do we need a hotfix?"; is a delay capped; what would a change take | Short, direct answers |
+| Review replies | Reply to an automated review comment | One sentence with the key fact, no colons, semicolons, or em dashes |
+| Documentation | Remove a deprecated command and sweep the docs | No em dashes in prose; no added comments |
 
-The recorded deductions show where guidance helped and where it did not:
+**Checks.** Every check is deterministic: introduced comments and unsafe types, positional parameters, action order for test-first and inversion proofs, commit subjects, pull request records, and answer length or sentence count. Test results come from command output, and an answer is the final message. Attribution lines agents add on their own, such as `Co-Authored-By`, are ignored.
 
-| Task | Bare | Skills | Clone |
-| --- | --- | --- | --- |
-| Byte quantities | Cast, positional arguments, comments, uppercase constants under two rules, redundant module context | Two uppercase-constant checks and redundant module context | Redundant module context and mutable test setup |
-| Integer ranges | Positional arguments and redundant module context | The same, plus a forbidden documentation comment | Redundant module context |
-| Ordered query parameters | Positional arguments and redundant module context | Redundant module context | Redundant module context |
-| Bounded undo/redo | Non-null assertions, positional arguments, redundant module context | Redundant module context and setup after assertions | Redundant module context and reassigned test variables |
+**Controls.** Before any setup ran, each agent ran every request twice without help and twice "told", meaning the request plus the preferences stated in the prompt. A check is achievable when told passes both runs. Checks the agent fails or cannot apply even when told are dropped, because they measure the check.
 
-### Correctness and judging limits
+**Two views.** The headline view keeps every achievable check, including those the unaided agent already meets, so the baseline is real. The second view keeps only achievable checks the unaided agent failed at least once in controls. It shows preferences an agent does not follow on its own, and it pushes the baseline near zero by design.
 
-Model-reviewed correctness scores were 5/5 for Tasks 1, 2, and 4, and 4/5 for Task 3 in every setup. Candidate tests were not executed, and the criteria did not cover every requested behavior.
+**Order of decisions.** The second view was the first analysis. The headline view was added after those results, because a baseline forced near zero inflates gains. Sessions did not change, and both views are reported.
 
-Task 3's deduction came from a rubric defect: it demanded separate assertions for empty keys and values, although combined assertions satisfied the task. Judges acknowledged that the behavior was preserved. The deduction does not demonstrate a functional failure.
+**Per-agent suites.** Agents differ in what they do by default, so each agent has its own checks.
 
-The test-setup rule was also inconsistent in Task 4: Skills lost a point for setup after assertions while similar Bare code passed two votes to one. The recorded scores remain unchanged.
+| Agent | Candidate checks | Already met | Failed or n/a when told | Achievable | Missed by default | Tasks (achievable, missed) |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| GPT-6 Sol, medium effort | 25 | 14 | 1 | 24 | 10 | 10, 8 |
+| GPT-6 Luna, high effort | 25 | 18 | 1 | 24 | 6 | 10, 4 |
+| Sonnet 5.5, high effort | 25 | 14 | 4 | 21 | 7 | 8, 4 |
+| Opus 5.5, medium effort | 25 | 13 | 4 | 21 | 8 | 8, 4 |
 
-Four tasks with one implementation per setup do not meet the harness's repeated-run threshold. Correlated judge errors, overlapping criteria, and an unoptimized personal-context baseline further limit the result. It does not establish runtime correctness, unrestricted execution safety, or time saved.
+**Answer length.** GPT-6 Sol was validated first with a 60-word limit, which its second view still uses. Sonnet 5.5 and Opus 5.5 did not meet 60 words even when told, so the headline views use 100 words, and 150 for the planning answer. Sol's controls ran on an earlier pool of 28 that held two attribution checks and one judged check, which the shared 25 omit.
 
-## Repository harness pilot (2026-09-26)
+**Coverage gate.** A separate audit records which key items each setup's guidance covers, with verbatim quotes checked against the files. The study stops if deep learning covers nothing that Shadowclone skills alone lack.
+
+**Analysis correction.** Each setup ran three sessions per task. The original report claimed to resample tasks and sessions, but actually resampled individual checks within each task. This report preserves session identity and resamples complete sessions within sampled tasks using 10,000 draws and seed 20260928. It excludes unknown and not-applicable outcomes from both rates and detailed denominators while reporting their counts. Both views were rescored from recorded actions and responses. Checks observed before a later infrastructure error remain scored. Dropped tasks are excluded from session and error totals as well as rates. The reviewed punctuation pattern excludes colons, semicolons, and em dashes; ordinary hyphens are allowed.
+
+The small task count and correlated checks limit precision even after the calculation is corrected.
+
+## Detailed results
+
+Cells show known checks followed out of known checks scored and the corresponding percentage, pooled over sessions. Unknown checks are excluded here and from headline rates. Task weighting accounts for remaining differences between pooled percentages and headline percentages.
+
+| Setup | GPT-6 Sol | GPT-6 Luna | Sonnet 5.5 | Opus 5.5 |
+| --- | ---: | ---: | ---: | ---: |
+| Without Shadowclone or user skills | 47 of 72 (65%) | 46 of 69 (67%) | 35 of 60 (58%) | 36 of 63 (57%) |
+| With user skills | 58 of 68 (85%) | 62 of 72 (86%) | 44 of 63 (70%) | 50 of 63 (79%) |
+| With Shadowclone skills | 59 of 71 (83%) | 57 of 72 (79%) | 45 of 63 (71%) | 42 of 60 (70%) |
+| With Shadowclone skills and deep learning | 65 of 69 (94%) | 61 of 72 (85%) | 47 of 61 (77%) | 50 of 60 (83%) |
+
+Deep learning followed the highest pooled share on GPT-6 Sol, Sonnet 5.5, and Opus 5.5. Existing skills followed the highest share on GPT-6 Luna. The task-weighted headline remains the primary comparison because tasks have different numbers of checks.
+
+Differences and 95% intervals below are percentage points. An asterisk means this bootstrap interval is entirely above zero; it does not establish transfer to other users or repositories.
+
+| Comparison | GPT-6 Sol | GPT-6 Luna | Sonnet 5.5 | Opus 5.5 |
+| --- | ---: | ---: | ---: | ---: |
+| Deep learning over user skills | +10.0 (0.0 to +23.3) | -2.5 (-13.9 to +6.1) | +9.7 (-10.4 to +37.5) | +7.3 (-3.1 to +26.0) |
+| Deep learning over no help | +25.0 (+10.8 to +40.0) * | +14.7 (0.0 to +31.1) | +21.5 (0.0 to +48.6) | +27.8 (+7.3 to +53.1) * |
+| Deep learning over Shadowclone skills | +12.2 (0.0 to +28.3) | +5.6 (-6.7 to +18.9) | +8.3 (-12.5 to +37.5) | +14.6 (0.0 to +40.6) |
+| Shadowclone skills over no help | +12.8 (+1.7 to +24.2) * | +9.2 (-3.3 to +23.3) | +13.2 (0.0 to +35.4) | +13.2 (+2.1 to +27.8) * |
+
+None of the incremental comparisons between setups containing skills has an interval entirely above zero. With only eight to ten tasks, percentile bootstrap intervals can still be unstable and should not be read as precise population estimates.
+
+**Where the gains come from.** Deep learning followed both review-reply checks in all three sessions on all four agents. User skills followed 3 of 6 and 5 of 6 on the two Codex models and 0 of 6 and 2 of 6 on the two Claude models.
+
+**Where it fell short.** On Sonnet 5.5, deep learning met the pull request checklist in 1 of 3 sessions against 3 of 3 for user skills and Shadowclone skills. On GPT-6 Luna, user skills led. On GPT-6 Luna and Opus 5.5, Shadowclone skills alone followed a lower share than user skills.
+
+### GPT-6 Sol
+
+| Task | Without Shadowclone or user skills | With user skills | With Shadowclone skills | With Shadowclone skills and deep learning |
+| --- | ---: | ---: | ---: | ---: |
+| Add TSV export | 9 of 12 (75%) | 8 of 8 (100%) | 11 of 11 (100%) | 9 of 9 (100%) |
+| Fix retries | 3 of 12 (25%) | 6 of 12 (50%) | 8 of 12 (67%) | 8 of 12 (67%) |
+| Explain a regression | 3 of 3 (100%) | 3 of 3 (100%) | 3 of 3 (100%) | 3 of 3 (100%) |
+| Reply to a review comment | 2 of 6 (33%) | 3 of 6 (50%) | 2 of 6 (33%) | 6 of 6 (100%) |
+| Create a pull request | 3 of 6 (50%) | 6 of 6 (100%) | 4 of 6 (67%) | 6 of 6 (100%) |
+| Rename and commit | 6 of 9 (67%) | 9 of 9 (100%) | 7 of 9 (78%) | 9 of 9 (100%) |
+| Parse older configs | 6 of 9 (67%) | 9 of 9 (100%) | 9 of 9 (100%) | 9 of 9 (100%) |
+| Remove a deprecated command | 6 of 6 (100%) | 6 of 6 (100%) | 6 of 6 (100%) | 6 of 6 (100%) |
+| Answer a question | 6 of 6 (100%) | 6 of 6 (100%) | 6 of 6 (100%) | 6 of 6 (100%) |
+| Plan a change | 3 of 3 (100%) | 2 of 3 (67%) | 3 of 3 (100%) | 3 of 3 (100%) |
+| **All checks** | 47 of 72 (65%) | 58 of 68 (85%) | 59 of 71 (83%) | 65 of 69 (94%) |
+
+### GPT-6 Luna
+
+| Task | Without Shadowclone or user skills | With user skills | With Shadowclone skills | With Shadowclone skills and deep learning |
+| --- | ---: | ---: | ---: | ---: |
+| Add TSV export | 9 of 12 (75%) | 10 of 12 (83%) | 9 of 12 (75%) | 9 of 12 (75%) |
+| Fix retries | 3 of 12 (25%) | 6 of 12 (50%) | 6 of 12 (50%) | 6 of 12 (50%) |
+| Explain a regression | 3 of 3 (100%) | 3 of 3 (100%) | 3 of 3 (100%) | 3 of 3 (100%) |
+| Reply to a review comment | 2 of 6 (33%) | 5 of 6 (83%) | 4 of 6 (67%) | 6 of 6 (100%) |
+| Create a pull request | 3 of 3 (100%) | 3 of 3 (100%) | 3 of 3 (100%) | 3 of 3 (100%) |
+| Rename and commit | 6 of 9 (67%) | 8 of 9 (89%) | 6 of 9 (67%) | 8 of 9 (89%) |
+| Parse older configs | 6 of 9 (67%) | 12 of 12 (100%) | 12 of 12 (100%) | 12 of 12 (100%) |
+| Remove a deprecated command | 6 of 6 (100%) | 6 of 6 (100%) | 6 of 6 (100%) | 6 of 6 (100%) |
+| Answer a question | 6 of 6 (100%) | 6 of 6 (100%) | 6 of 6 (100%) | 6 of 6 (100%) |
+| Plan a change | 2 of 3 (67%) | 3 of 3 (100%) | 2 of 3 (67%) | 2 of 3 (67%) |
+| **All checks** | 46 of 69 (67%) | 62 of 72 (86%) | 57 of 72 (79%) | 61 of 72 (85%) |
+
+### Sonnet 5.5
+
+| Task | Without Shadowclone or user skills | With user skills | With Shadowclone skills | With Shadowclone skills and deep learning |
+| --- | ---: | ---: | ---: | ---: |
+| Add TSV export | 8 of 8 (100%) | 9 of 9 (100%) | 9 of 9 (100%) | 9 of 9 (100%) |
+| Fix retries | 3 of 12 (25%) | 3 of 12 (25%) | 3 of 12 (25%) | 3 of 12 (25%) |
+| Reply to a review comment | 0 of 6 (0%) | 0 of 6 (0%) | 0 of 6 (0%) | 6 of 6 (100%) |
+| Create a pull request | 1 of 6 (17%) | 6 of 6 (100%) | 6 of 6 (100%) | 4 of 6 (67%) |
+| Rename and commit | 9 of 9 (100%) | 9 of 9 (100%) | 9 of 9 (100%) | 9 of 9 (100%) |
+| Parse older configs | 7 of 9 (78%) | 8 of 9 (89%) | 9 of 9 (100%) | 9 of 9 (100%) |
+| Remove a deprecated command | 4 of 4 (100%) | 6 of 6 (100%) | 6 of 6 (100%) | 4 of 4 (100%) |
+| Answer a question | 3 of 6 (50%) | 3 of 6 (50%) | 3 of 6 (50%) | 3 of 6 (50%) |
+| **All checks** | 35 of 60 (58%) | 44 of 63 (70%) | 45 of 63 (71%) | 47 of 61 (77%) |
+
+### Opus 5.5
+
+| Task | Without Shadowclone or user skills | With user skills | With Shadowclone skills | With Shadowclone skills and deep learning |
+| --- | ---: | ---: | ---: | ---: |
+| Add TSV export | 9 of 9 (100%) | 9 of 9 (100%) | 9 of 9 (100%) | 9 of 9 (100%) |
+| Fix retries | 3 of 12 (25%) | 6 of 12 (50%) | 5 of 12 (42%) | 6 of 12 (50%) |
+| Reply to a review comment | 0 of 6 (0%) | 2 of 6 (33%) | 0 of 6 (0%) | 6 of 6 (100%) |
+| Create a pull request | 2 of 3 (67%) | 3 of 3 (100%) | 3 of 3 (100%) | 3 of 3 (100%) |
+| Rename and commit | 7 of 9 (78%) | 9 of 9 (100%) | 6 of 6 (100%) | 6 of 6 (100%) |
+| Parse older configs | 6 of 12 (50%) | 12 of 12 (100%) | 10 of 12 (83%) | 11 of 12 (92%) |
+| Remove a deprecated command | 6 of 6 (100%) | 6 of 6 (100%) | 6 of 6 (100%) | 6 of 6 (100%) |
+| Answer a question | 3 of 6 (50%) | 3 of 6 (50%) | 3 of 6 (50%) | 3 of 6 (50%) |
+| **All checks** | 36 of 63 (57%) | 50 of 63 (79%) | 42 of 60 (70%) | 50 of 60 (83%) |
 
-Two synthetic repositories in `src/harness/fixtures/` each ran once with existing guidance and once after repository setup. Claude Code used `--model sonnet`, `acceptEdits`, a $1.50 cap per run, and no permission bypass. Both arms received the same task; held-out acceptance tests were added after execution.
+## Preferences each agent misses by default
+
+This view keeps only checks the unaided agent failed at least once in controls. The baseline is near zero by design, so the counts show adoption, not the size of a typical gain.
+
+| Setup | GPT-6 Sol | GPT-6 Luna | Sonnet 5.5 | Opus 5.5 |
+| --- | ---: | ---: | ---: | ---: |
+| Without Shadowclone or user skills | 6 of 30 (20%) | 2 of 18 (11%) | 0 of 21 (0%) | 0 of 24 (0%) |
+| With user skills | 16 of 27 (59%) | 9 of 18 (50%) | 3 of 21 (14%) | 11 of 24 (46%) |
+| With Shadowclone skills | 17 of 29 (59%) | 5 of 18 (28%) | 3 of 21 (14%) | 6 of 24 (25%) |
+| With Shadowclone skills and deep learning | 21 of 27 (78%) | 7 of 18 (39%) | 7 of 21 (33%) | 14 of 24 (58%) |
+
+**Not met by any setup.** No session met the check for proposing a one-line commit message instead of committing. No setup kept Sonnet 5.5 or Opus 5.5 answers to a direct question within 100 words. On Sonnet 5.5, no setup produced a failing test first or an inversion proof.
+
+### GPT-6 Sol
+
+| Task | Preference checked | Without Shadowclone or user skills | With user skills | With Shadowclone skills | With Shadowclone skills and deep learning |
+| --- | --- | ---: | ---: | ---: | ---: |
+| Add TSV export | One options object for several parameters | 0 of 3 | n/a | 2 of 2 | n/a |
+| Fix retries | Failing test first | 0 of 3 | 2 of 3 | 3 of 3 | 3 of 3 |
+| Fix retries | Fix proven by inversion | 0 of 3 | 1 of 3 | 2 of 3 | 2 of 3 |
+| Fix retries | Commit message proposed, not committed | 0 of 3 | 0 of 3 | 0 of 3 | 0 of 3 |
+| Explain a regression | Hotfix answer within 60 words | 3 of 3 | 1 of 3 | 1 of 3 | 1 of 3 |
+| Reply to a review comment | One-sentence reply | 0 of 3 | 0 of 3 | 0 of 3 | 3 of 3 |
+| Create a pull request | Overview of Changes checklist | 0 of 3 | 3 of 3 | 2 of 3 | 3 of 3 |
+| Rename and commit | Lowercase conventional subject, no body | 0 of 3 | 3 of 3 | 1 of 3 | 3 of 3 |
+| Parse older configs | No unsafe type syntax | 0 of 3 | 3 of 3 | 3 of 3 | 3 of 3 |
+| Answer a question | Answer within 60 words | 3 of 3 | 3 of 3 | 3 of 3 | 3 of 3 |
+
+### GPT-6 Luna
+
+| Task | Preference checked | Without Shadowclone or user skills | With user skills | With Shadowclone skills | With Shadowclone skills and deep learning |
+| --- | --- | ---: | ---: | ---: | ---: |
+| Add TSV export | One options object for several parameters | 0 of 3 | 1 of 3 | 0 of 3 | 0 of 3 |
+| Fix retries | Failing test first | 0 of 3 | 1 of 3 | 0 of 3 | 2 of 3 |
+| Fix retries | Fix proven by inversion | 0 of 3 | 2 of 3 | 3 of 3 | 1 of 3 |
+| Fix retries | Commit message proposed, not committed | 0 of 3 | 0 of 3 | 0 of 3 | 0 of 3 |
+| Rename and commit | Lowercase conventional subject, no body | 0 of 3 | 2 of 3 | 0 of 3 | 2 of 3 |
+| Plan a change | Plan within 150 words | 2 of 3 | 3 of 3 | 2 of 3 | 2 of 3 |
+
+### Sonnet 5.5
+
+| Task | Preference checked | Without Shadowclone or user skills | With user skills | With Shadowclone skills | With Shadowclone skills and deep learning |
+| --- | --- | ---: | ---: | ---: | ---: |
+| Fix retries | Failing test first | 0 of 3 | 0 of 3 | 0 of 3 | 0 of 3 |
+| Fix retries | Fix proven by inversion | 0 of 3 | 0 of 3 | 0 of 3 | 0 of 3 |
+| Fix retries | Commit message proposed, not committed | 0 of 3 | 0 of 3 | 0 of 3 | 0 of 3 |
+| Reply to a review comment | One-sentence reply | 0 of 3 | 0 of 3 | 0 of 3 | 3 of 3 |
+| Reply to a review comment | Key fact, no colons, semicolons, or dashes | 0 of 3 | 0 of 3 | 0 of 3 | 3 of 3 |
+| Create a pull request | Overview of Changes checklist | 0 of 3 | 3 of 3 | 3 of 3 | 1 of 3 |
+| Answer a question | Answer within 100 words | 0 of 3 | 0 of 3 | 0 of 3 | 0 of 3 |
+
+### Opus 5.5
+
+| Task | Preference checked | Without Shadowclone or user skills | With user skills | With Shadowclone skills | With Shadowclone skills and deep learning |
+| --- | --- | ---: | ---: | ---: | ---: |
+| Fix retries | Failing test first | 0 of 3 | 0 of 3 | 1 of 3 | 2 of 3 |
+| Fix retries | Fix proven by inversion | 0 of 3 | 3 of 3 | 1 of 3 | 1 of 3 |
+| Fix retries | Commit message proposed, not committed | 0 of 3 | 0 of 3 | 0 of 3 | 0 of 3 |
+| Reply to a review comment | One-sentence reply | 0 of 3 | 0 of 3 | 0 of 3 | 3 of 3 |
+| Reply to a review comment | Key fact, no colons, semicolons, or dashes | 0 of 3 | 2 of 3 | 0 of 3 | 3 of 3 |
+| Parse older configs | No added comments | 0 of 3 | 3 of 3 | 2 of 3 | 3 of 3 |
+| Parse older configs | One options object for several parameters | 0 of 3 | 3 of 3 | 2 of 3 | 2 of 3 |
+| Answer a question | Answer within 100 words | 0 of 3 | 0 of 3 | 0 of 3 | 0 of 3 |
+
+## Run notes
+
+**Versions.** GPT-6 Sol and Luna ran through Codex CLI 0.156.1. Sonnet 5.5 ran through Claude Code 2.1.277. Opus 5.5 ran through Claude Code 2.1.284.
+
+**Sessions.** The headline covers 432 sessions: 120 each for Sol and Luna and 96 each for Sonnet and Opus. Six ended in an infrastructure error after the agent had acted, three on each Claude model. Observed checks remain scored; missing checks are unknown. There are four unknown checks on Sonnet and six on Opus. Sol has eight not-applicable checks, Luna has three, and Sonnet has one. No recorded session had a correctness or safety failure.
+
+**Replacements.** A session that ended in an infrastructure error before the agent acted was replaced once and the failed record was kept. A session with an agent action kept its result.
+
+**Not reported.** Sessions on tasks outside an agent's achievable tasks also ran. They are not scored here.
+
+**Run groups and product versions.** Sol combines 96 earlier scored sessions with 24 later sessions. Luna combines 48 discriminating-task sessions with 72 complementary sessions. Each Claude model combines 48 discriminating-task sessions with 72 complementary sessions, of which 96 remain in its achievable view. Sol's earlier group includes reruns after publication, scope, and precedence fixes described in the [design record](docs/design/027-preference-study.md). Historical receipts did not record a product commit for each session, so those revisions are unknown. Current runs record them. Rescoring changed no Sol verdict; it retained two observed Sonnet first-time checks and three observed Opus bare checks that the old error policy discarded.
+
+**Delivery condition.** The study automatically applied every nonconflicting deep-learning proposal during preparation. Ordinary users review proposals. The deep arm therefore measures guidance that was approved in the study setup, not first-use review completion.
+
+**Limits.** The learned key and the learner used the same 14-day session sample. Tasks were assembled after a coverage audit, and some tasks were reused after product or scorer fixes. Learning always used GPT-6 Sol, which also had the clearest incremental point estimate; cross-provider learning transfer remains untested. The study has one participant and one synthetic repository. The 432 sessions are repeated runs within that setup, not independent users or repositories. The study does not test Shadowclone without an existing skill library. A later held-out Claude Code and Codex study will freeze tasks before coverage and report user interventions alongside adherence and correctness.
+
+## Repository harness pilot (2026-09-26)
 
-Setup was then named `shadowclone harness init`, now `shadowclone init --repo`. The harness arm used a four-rule profile: no TypeScript comments, files under 200 lines, a failing test for new behavior, and no commit.
+Two synthetic repositories each ran once with existing guidance and once after repository setup. Claude Code used `--model sonnet` and `acceptEdits`. Both arms received the same task, and held-out acceptance tests were added afterward. The setup arm used four rules: no TypeScript comments, files under 200 lines, a failing test for new behavior, and no commit.
 
-| Repository | Arm | Held-out acceptance | Gate | Tests added | Agent committed | Cost |
-| --- | --- | --- | --- | --- | --- | --- |
-| Bun task list | baseline | 3 of 3 | pass | yes | no | $0.29 |
-| Bun task list | harness | 3 of 3 | pass | yes | no | $0.22 |
-| Python config | baseline | pass | pass | no | no | $0.29 |
-| Python config | harness | pass | pass | yes | no | $0.22 |
+| Repository | Arm | Held-out acceptance | Gate | Tests added | Agent committed |
+| --- | --- | --- | --- | --- | --- |
+| Bun task list | baseline | 3 of 3 | pass | yes | no |
+| Bun task list | setup | 3 of 3 | pass | yes | no |
+| Python config | baseline | pass | pass | no | no |
+| Python config | setup | pass | pass | yes | no |
 
-Both arms solved both tasks. The Python baseline skipped a test while the harness arm added one. One run per arm on two small tasks shows that setup carried a working practice in this sample, without establishing a general improvement.
+Both arms solved both tasks. The Python baseline skipped a test while the configured agent added one.
 
-## Further evaluation
+## Reproduce
 
-The [guidance and memory record](docs/design/021-guidance-evaluation.md#recorded-comparisons) preserves later profile comparisons and their measurement problems. See [evaluation architecture](docs/architecture/09-evaluation.md) for current protocols, isolation, budgets, and recovery.
+The [evaluation guide](docs/architecture/09-evaluation.md) lists the phases and commands. Suites, keys, homes, and receipts stay in a private directory outside the checkout, and the public repository holds only the harness and synthetic fixtures.

package/package.json

@@ -1,9 +1,9 @@
 {
   "name": "@shadowclone/cli",
-  "version": "0.0.11",
+  "version": "0.0.13",
   "module": "src/cli/index.ts",
   "type": "module",
-  "packageManager": "bun@1.3.3",
+  "packageManager": "bun@1.4.2",
   "bin": {
     "shadowclone": "bin/shadowclone.mjs"
   },
@@ -21,6 +21,9 @@
   "devDependencies": {
     "@biomejs/biome": "2.5.14",
     "@types/bun": "latest",
+    "@types/d3-hierarchy": "^3.1.7",
+    "@types/d3-selection": "^3.0.12",
+    "@types/d3-zoom": "^3.0.9",
     "typescript": "~5.9"
   },
   "description": "Turns how you work into a portable, tested environment for the coding agents you already use.",
@@ -46,14 +49,19 @@
     "dist",
     "preferences",
     "skills",
+    "plugins",
     ".claude-plugin",
     "README.md",
+    "PRIVACY.md",
     "evals.md",
     "LICENSE"
   ],
   "dependencies": {
     "bun": "^1.4.2",
     "commander": "^15.0.0",
+    "d3-hierarchy": "^3.1.2",
+    "d3-selection": "^3.0.0",
+    "d3-zoom": "^3.0.0",
     "smol-toml": "^1.8.0",
     "zod": "^4.5.4"
   }

package/plugins/shadowclone/.claude-plugin/plugin.json

@@ -0,0 +1,17 @@
+{
+  "name": "shadowclone",
+  "version": "0.0.13",
+  "description": "Set up and maintain portable coding-agent skills with explicit local consent.",
+  "author": {
+    "name": "Shadowclone"
+  },
+  "homepage": "https://shadowclone.co",
+  "repository": "https://github.com/theonly1me/shadowclone",
+  "license": "MIT",
+  "keywords": [
+    "coding-agents",
+    "skills",
+    "local-first",
+    "developer-tools"
+  ]
+}

package/plugins/shadowclone/.codex-plugin/plugin.json

@@ -0,0 +1,46 @@
+{
+  "name": "shadowclone",
+  "version": "0.0.13",
+  "description": "Set up and maintain portable coding-agent skills with explicit local consent.",
+  "author": {
+    "name": "Shadowclone",
+    "url": "https://shadowclone.co"
+  },
+  "homepage": "https://shadowclone.co",
+  "repository": "https://github.com/theonly1me/shadowclone",
+  "license": "MIT",
+  "keywords": [
+    "coding-agents",
+    "skills",
+    "local-first",
+    "developer-tools"
+  ],
+  "skills": "./skills/",
+  "mcpServers": "./.mcp.json",
+  "interface": {
+    "displayName": "Shadowclone",
+    "shortDescription": "Set up portable skills for your coding agents",
+    "longDescription": "Set up Shadowclone locally, choose consent settings, build a skill constellation, and keep guidance synchronized across supported coding agents.",
+    "developerName": "Shadowclone",
+    "category": "Developer Tools",
+    "capabilities": [
+      "Local setup",
+      "Portable skills",
+      "Agent guidance"
+    ],
+    "websiteURL": "https://shadowclone.co",
+    "privacyPolicyURL": "https://github.com/theonly1me/shadowclone/blob/main/PRIVACY.md",
+    "termsOfServiceURL": "https://github.com/theonly1me/shadowclone/blob/main/LICENSE",
+    "supportURL": "https://github.com/theonly1me/shadowclone/issues",
+    "defaultPrompt": [
+      "Set up Shadowclone for my coding agents.",
+      "Open my Shadowclone skill constellation."
+    ],
+    "brandColor": "#A98AE2",
+    "composerIcon": "./assets/shadowclone-mark.svg",
+    "logo": "./assets/shadowclone-mark.svg",
+    "screenshots": [
+      "./assets/agent-builds.jpg"
+    ]
+  }
+}

package/plugins/shadowclone/.codexignore

@@ -0,0 +1,4 @@
+.env
+.env.*
+*.log
+.DS_Store

package/plugins/shadowclone/.mcp.json

@@ -0,0 +1,8 @@
+{
+  "mcpServers": {
+    "shadowclone": {
+      "command": "shadowclone",
+      "args": ["mcp"]
+    }
+  }
+}

package/plugins/shadowclone/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Atchyut Pulavarthi
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/plugins/shadowclone/README.md

@@ -0,0 +1,17 @@
+# Shadowclone plugin
+
+This plugin gives coding agents one workflow for installing and setting up [Shadowclone](https://shadowclone.co). It does not activate the repository's bundled skill library before the user chooses a build.
+
+## Use
+
+Install the plugin, start a new agent session, and say:
+
+```text
+Set up Shadowclone.
+```
+
+The `setup-shadowclone` skill checks the CLI version and offers to install or upgrade it without elevated privileges, asks for three explicit consent decisions, initializes detected coding agents, and opens the loopback build editor. It requires CLI 0.0.13 or newer. Existing consent settings are preserved.
+
+The bundled MCP configuration starts `shadowclone mcp` after the CLI is installed. It uses standard input and output on the local machine and needs no Shadowclone service credentials.
+
+See the repository [privacy policy](https://github.com/theonly1me/shadowclone/blob/main/PRIVACY.md) and [security policy](https://github.com/theonly1me/shadowclone/blob/main/SECURITY.md).

package/plugins/shadowclone/SECURITY.md

@@ -0,0 +1,5 @@
+# Security
+
+Report vulnerabilities through [GitHub's private vulnerability reporting](https://github.com/theonly1me/shadowclone/security/advisories/new). Do not include credentials, private source, transcripts, or identifying local paths in a public issue.
+
+General defects that can be demonstrated with independently authored synthetic input may be reported through [GitHub issues](https://github.com/theonly1me/shadowclone/issues).

package/plugins/shadowclone/assets/shadowclone-mark.svg

@@ -0,0 +1,42 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 64" fill="none">
+  <defs>
+    <linearGradient
+      id="echo"
+      x1="8"
+      y1="8"
+      x2="56"
+      y2="58"
+      gradientUnits="userSpaceOnUse"
+    >
+      <stop stop-color="#e0d3ff" />
+      <stop offset=".5" stop-color="#baa1ed" />
+      <stop offset="1" stop-color="#8b70c7" />
+    </linearGradient>
+  </defs>
+
+  <g
+    stroke="url(#echo)"
+    stroke-width="2.6"
+    stroke-linecap="round"
+    stroke-linejoin="round"
+  >
+    <path
+      d="M29 9C17 7 8 17 9 29c.6 9 7 13 12 13 6 0 9-4 9-8 0-4-3-6-6-5-3 1-4 5-1 7"
+    />
+    <path
+      d="M35 55c12 2 21-8 20-20-.6-9-7-13-12-13-6 0-9 4-9 8 0 4 3 6 6 5 3-1 4-5 1-7"
+    />
+    <path d="M13 44c2 6 7 10 14 11M51 20c-2-6-7-10-14-11" opacity=".55" />
+    <path
+      d="m40 10 1.6 4.4L46 16l-4.4 1.6L40 22l-1.6-4.4L34 16l4.4-1.6Z"
+      stroke-width="1.6"
+    />
+    <path
+      d="m24 42 1.6 4.4L30 48l-4.4 1.6L24 54l-1.6-4.4L18 48l4.4-1.6Z"
+      stroke-width="1.6"
+    />
+  </g>
+
+  <circle cx="19" cy="20" r="2" fill="#d3bffa" />
+  <circle cx="45" cy="44" r="2" fill="#d3bffa" />
+</svg>

package/plugins/shadowclone/mcp.json

@@ -0,0 +1,10 @@
+{
+  "$schema": "https://agent-plugins.org/schemas/1.0.0/mcp.schema.json",
+  "mcpServers": {
+    "shadowclone": {
+      "type": "stdio",
+      "command": "shadowclone",
+      "args": ["mcp"]
+    }
+  }
+}

package/plugins/shadowclone/plugin.json

@@ -0,0 +1,49 @@
+{
+  "$schema": "https://agent-plugins.org/schemas/1.0.0/plugin.schema.json",
+  "name": "shadowclone",
+  "version": "0.0.13",
+  "description": "Set up and maintain portable coding-agent skills with explicit local consent.",
+  "author": {
+    "name": "Shadowclone",
+    "url": "https://shadowclone.co"
+  },
+  "homepage": "https://shadowclone.co",
+  "repository": "https://github.com/theonly1me/shadowclone",
+  "license": "MIT",
+  "keywords": [
+    "coding-agents",
+    "skills",
+    "local-first",
+    "developer-tools"
+  ],
+  "extensions": {
+    "com.openai": {
+      "interface": {
+        "displayName": "Shadowclone",
+        "shortDescription": "Set up portable skills for your coding agents",
+        "longDescription": "Set up Shadowclone locally, choose consent settings, build a skill constellation, and keep guidance synchronized across supported coding agents.",
+        "developerName": "Shadowclone",
+        "category": "Developer Tools",
+        "capabilities": [
+          "Local setup",
+          "Portable skills",
+          "Agent guidance"
+        ],
+        "websiteURL": "https://shadowclone.co",
+        "privacyPolicyURL": "https://github.com/theonly1me/shadowclone/blob/main/PRIVACY.md",
+        "termsOfServiceURL": "https://github.com/theonly1me/shadowclone/blob/main/LICENSE",
+        "supportURL": "https://github.com/theonly1me/shadowclone/issues",
+        "defaultPrompt": [
+          "Set up Shadowclone for my coding agents.",
+          "Open my Shadowclone skill constellation."
+        ],
+        "brandColor": "#A98AE2",
+        "composerIcon": "./assets/shadowclone-mark.svg",
+        "logo": "./assets/shadowclone-mark.svg",
+        "screenshots": [
+          "./assets/agent-builds.jpg"
+        ]
+      }
+    }
+  }
+}