@shadowclone/cli 0.0.14 → 0.0.15
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/evals.md
CHANGED
|
@@ -1,5 +1,35 @@
|
|
|
1
1
|
# Evaluation results
|
|
2
2
|
|
|
3
|
+
## Fixed four-setup results, 2026-10-01
|
|
4
|
+
|
|
5
|
+
The [fixed preference benchmark](docs/guides/fixed-evals.md) compares bare, existing synthetic skills, skills plus Shadowclone routing, and routing plus actual learning from synthetic corrections. Each model ran seven tasks with three repetitions per setup. This completed-session analysis excludes one timed-out Luna attempt after the runs; all completed sessions, including preference failures, remain. Scores measure preference adherence within each task with equal task weights. Correctness, safety, and whole-task success are separate outcomes.
|
|
6
|
+
|
|
7
|
+
| Share of preferences followed | GPT 6.1 Sol, medium | GPT 6 Luna, high | Sonnet 5.5, high | Opus 5.5, medium |
|
|
8
|
+
| --- | ---: | ---: | ---: | ---: |
|
|
9
|
+
| Bare | 96.2% | 97.1% | 59.4% | 52.1% |
|
|
10
|
+
| Existing user skills | 100% | 100% | 65.0% | 63.8% |
|
|
11
|
+
| Skills plus Shadowclone routing | 100% | 100% | 65.0% | 63.8% |
|
|
12
|
+
| Skills, routing, and deep learning | 100% | 100% | 81.7% | 96.4% |
|
|
13
|
+
|
|
14
|
+
| Deep learning over existing skills | Gain in percentage points | 95% bootstrap interval |
|
|
15
|
+
| --- | ---: | --- |
|
|
16
|
+
| GPT 6.1 Sol | 0.0 | 0.0 to 0.0 |
|
|
17
|
+
| GPT 6 Luna | 0.0 | 0.0 to 0.0 |
|
|
18
|
+
| Sonnet 5.5 | +16.7 | +6.0 to +31.0 |
|
|
19
|
+
| Opus 5.5 | +32.6 | +16.4 to +57.1 |
|
|
20
|
+
|
|
21
|
+
Routing alone tied existing skills in the completed-session analysis. Both Codex models already met every preference with existing skills. The Claude learning gains came primarily from shorter final answers; all thirteen remaining preference misses in their deep setups were answers exceeding eighty words. Every completed session passed correctness and safety checks: 335 passed, one unknown.
|
|
22
|
+
|
|
23
|
+
Luna's existing-skills setup timed out on the first turn of the second parser-extension repetition after 240 seconds. Its four preference checks, correctness, safety, and model confirmation remain unknown in the unchanged full-matrix report, which emits no final Luna score. The failed attempt was retained without replacement. The separately requested completed-session view uses twenty existing-skills sessions, including two parser-extension repetitions, and twenty-one sessions in each other setup. Its exclusion is a post-run analysis choice, not a successful or complete execution result.
|
|
24
|
+
|
|
25
|
+
The handwritten existing skill contains comments, type-safety, and options-object preferences. One shared production learning run processed two independent synthetic corrections about Git authorization and answer length, published two rules in six Codex Sol medium invocations, and preserved the manual skill. Learning output was frozen once for both hosts. The study used 383 scored agent turns and six learning turns. [The contributor guide](docs/guides/fixed-evals.md#authenticated-run-record) records versions, aggregate counts, and reproducibility details.
|
|
26
|
+
|
|
27
|
+
These seven public tasks had already been used for development, and the three-existing plus two-learned provenance split was authored after the earlier three-setup run. They are not held-out evidence. Bootstrap intervals describe variation in this small task set, with no multiple-comparison correction. The results establish a specific synthetic learning benefit on Claude, not an advantage over a complete handwritten profile, automatic hillclimbing, or useful engineering throughput.
|
|
28
|
+
|
|
29
|
+
## Historical preference study
|
|
30
|
+
|
|
31
|
+
Earlier three-setup delivery runs and the historical learning study below use different guidance and remain separate experiments.
|
|
32
|
+
|
|
3
33
|
The preference study measured whether a coding agent followed one participant's engineering preferences on ordinary requests. These results were recomputed from the private receipts on 2026-09-30 using scorer `session-bootstrap-2`, without new model calls. Unknown and not-applicable checks are excluded consistently, complete sessions are resampled within tasks, and observable actions from incomplete sessions retain their checks.
|
|
4
34
|
|
|
5
35
|
## Results
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"$schema": "https://agent-plugins.org/schemas/1.0.0/plugin.schema.json",
|
|
3
3
|
"name": "shadowclone",
|
|
4
|
-
"version": "0.0.
|
|
4
|
+
"version": "0.0.15",
|
|
5
5
|
"description": "Set up and maintain portable coding-agent skills with explicit local consent.",
|
|
6
6
|
"author": {
|
|
7
7
|
"name": "Shadowclone",
|