@maccesar/aiskills 1.16.1 → 1.17.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,7 +1,6 @@
1
1
  # Evals
2
2
 
3
- Three things live here, and they are in three different states. The states matter
4
- more than the numbers, so they come first.
3
+ Three things live here, and they are in three different states. The states matter more than the numbers, so they come first.
5
4
 
6
5
  | File | What it is | State |
7
6
  | --- | --- | --- |
@@ -12,68 +11,33 @@ more than the numbers, so they come first.
12
11
 
13
12
  ## What the A/B rounds actually measured
14
13
 
15
- Three rounds, 18 runs, same prompt with and without the skill on purpose-built
16
- fixture repos, graded adversarially by an independent agent. Headline: volatile
17
- status kept out of the startup chain **9/9 with the skill, 0/9 without**. Token cost
18
- 3–13% higher. In round 3, with a deliberately broken uncommitted file planted in the
19
- fixture, the skill left it alone and reported it while the baseline silently fixed
20
- it.
14
+ Three rounds, 18 runs, same prompt with and without the skill on purpose-built fixture repos, graded adversarially by an independent agent. Headline: volatile status kept out of the startup chain **9/9 with the skill, 0/9 without**. Token cost 3–13% higher. In round 3, with a deliberately broken uncommitted file planted in the fixture, the skill left it alone and reported it while the baseline silently fixed it.
21
15
 
22
- Each run was inspected directly — files on disk, import chains resolved, `php -l`
23
- and `node --test` executed. The numbers are real.
16
+ Each run was inspected directly — files on disk, import chains resolved, `php -l` and `node --test` executed. The numbers are real.
24
17
 
25
- **They describe a layout this skill no longer prescribes.** Rounds 1 and 2 graded
26
- `docs/status/current.md` against `.claude/memory/index.md`; the four-file
27
- `docs/project/` convention came later. What carried over is the finding those rounds
28
- were built to test — keeping volatile status out of the startup chain is not
29
- behaviour the model has on its own — and that finding is about the split, not about
30
- the filenames. Everything specific to the current layout (requirements as a separate
31
- file, the upgrade path, resuming, monorepos) is untested.
18
+ **They describe a layout this skill no longer prescribes.** Rounds 1 and 2 graded `docs/status/current.md` against `.claude/memory/index.md`; the four-file `docs/project/` convention came later. What carried over is the finding those rounds were built to test — keeping volatile status out of the startup chain is not behaviour the model has on its own — and that finding is about the split, not about the filenames. Everything specific to the current layout (requirements as a separate file, the upgrade path, resuming, monorepos) is untested.
32
19
 
33
20
  Two more limits worth stating plainly:
34
21
 
35
- - **Round 3 has no write-up.** It is counted in the 18 runs and in the 9/9, and the
36
- only file from it is `defecto-experimento.md`, which documents a fixture that
37
- contaminated one eval. Treat round 3's numbers as less reviewable than 1 and 2.
38
- - **The fixture repos are not in this repo.** They were built in a scratch workspace
39
- and are gone. `evals.json` now carries a `fixture_spec` per eval so they can be
40
- rebuilt; the rounds above cannot be re-run as they were.
22
+ - **Round 3 has no write-up.** It is counted in the 18 runs and in the 9/9, and the only file from it is `defecto-experimento.md`, which documents a fixture that contaminated one eval. Treat round 3's numbers as less reviewable than 1 and 2.
23
+ - **The fixture repos are not in this repo.** They were built in a scratch workspace and are gone. `evals.json` now carries a `fixture_spec` per eval so they can be rebuilt; the rounds above cannot be re-run as they were.
41
24
 
42
25
  ## `evals.json` — the current set
43
26
 
44
- Nine task prompts against the four-file convention: the three that survive from the
45
- old set (rewritten), the non-Claude repo case from round 2, and five new ones
46
- covering paths nothing has ever tested — resuming against a stale file, a repo that
47
- gitignores `docs/`, upgrading an earlier install, a credential offered for the
48
- notes, and a monorepo.
27
+ Nine task prompts against the four-file convention: the three that survive from the old set (rewritten), the non-Claude repo case from round 2, and five new ones covering paths nothing has ever tested — resuming against a stale file, a repo that gitignores `docs/`, upgrading an earlier install, a credential offered for the notes, and a monorepo.
49
28
 
50
- Nothing here has been run. Saying so is the point: an eval file that looks like
51
- results is worse than no eval file.
29
+ Nothing here has been run. Saying so is the point: an eval file that looks like results is worse than no eval file.
52
30
 
53
31
  ## `trigger-eval.json` — when the skill gets consulted
54
32
 
55
- Twenty queries: ten that should reach the skill — five about where notes belong,
56
- five about closing or resuming a session — and ten near-misses that should not.
33
+ Twenty queries: ten that should reach the skill — five about where notes belong, five about closing or resuming a session — and ten near-misses that should not.
57
34
 
58
35
  **Not measured. The harness these were written for does not work.**
59
36
 
60
- `skill-creator`'s `run_eval.py` reports 0/10 for this skill — and **0/5 for
61
- `audit-codebase`**, a published skill in this same repo, fed phrases lifted
62
- verbatim from its own description. A control that fails on a healthy subject is
63
- not measuring the subject.
64
-
65
- A field test settled what the harness could not. A fresh agent, given only
66
- *"audita este proyecto completo antes de producción, quiero saber qué está
67
- realmente roto"* and no mention of any skill, opened `audit-codebase` as its first
68
- action. **Automatic triggering works; the harness does not detect it.** The
69
- near-misses scoring 10/10 is equally uninformative — a harness that never reports
70
- a trigger is trivially correct at "don't trigger".
71
-
72
- So these twenty queries are written and unused. If you have a working way to
73
- measure skill triggering, that's what they're for.
74
-
75
- The description is written for automatic triggering, and there is no slash command
76
- to fall back on — that was deliberate, since a command duplicating this logic
77
- would drift from it and would only work in Claude Code. So the wording of the
78
- description is the whole mechanism, which is exactly why it would be worth
79
- measuring properly.
37
+ `skill-creator`'s `run_eval.py` reports 0/10 for this skill — and **0/5 for `audit-codebase`**, a published skill in this same repo, fed phrases lifted verbatim from its own description. A control that fails on a healthy subject is not measuring the subject.
38
+
39
+ A field test settled what the harness could not. A fresh agent, given only *"audita este proyecto completo antes de producción, quiero saber qué está realmente roto"* and no mention of any skill, opened `audit-codebase` as its first action. **Automatic triggering works; the harness does not detect it.** The near-misses scoring 10/10 is equally uninformative — a harness that never reports a trigger is trivially correct at "don't trigger".
40
+
41
+ So these twenty queries are written and unused. If you have a working way to measure skill triggering, that's what they're for.
42
+
43
+ The description is written for automatic triggering, and there is no slash command to fall back on — that was deliberate, since a command duplicating this logic would drift from it and would only work in Claude Code. So the wording of the description is the whole mechanism, which is exactly why it would be worth measuring properly.
@@ -5,9 +5,7 @@
5
5
  > convention the skill prescribes today. See `README.md` in this folder for what
6
6
  > carries over and what doesn't.
7
7
 
8
- Totals: with_skill 15/15 assertions passed, without_skill 9/15.
9
- All 6 failures are without_skill, and every one is the same failure mode:
10
- volatile status written into (or wired into) the CLAUDE.md `@import` chain.
8
+ Totals: with_skill 15/15 assertions passed, without_skill 9/15. All 6 failures are without_skill, and every one is the same failure mode: volatile status written into (or wired into) the CLAUDE.md `@import` chain.
11
9
 
12
10
  ## 1. Assertion x branch
13
11
 
@@ -43,73 +41,30 @@ volatile status written into (or wired into) the CLAUDE.md `@import` chain.
43
41
 
44
42
  ## 2. Assertions that did NOT discriminate (passed in both branches)
45
43
 
46
- 9 of 15 assertions — 60% — passed identically in both branches. They inflate the
47
- skill's apparent score without testing it.
48
-
49
- - **eval-0, all four content assertions.** Baseline caught the false "refunds done"
50
- claim on its own, marked payments done, cited `PaymentController::store` /
51
- `payments.store` / the migration, and spelled out that refunds are a stub with no
52
- route or controller. Baseline also independently found the `php -l` parse error.
53
- This eval discriminates on exactly one assertion, and it is a file-placement
54
- assertion, not a truthfulness assertion. The eval's own name
55
- ("partial-claim-caught") describes something the model does without the skill.
56
- - **eval-1 "Handoff reflects the actual uncommitted work"** — both branches
57
- inventoried the refund stub and the assertion-less `PaymentTest.php`.
58
- - **eval-1 "Does not commit, tag or push"** — both refused; the user's global
59
- CLAUDE.md already forbids it, so it can never discriminate in this harness.
60
- - **eval-1 "Does not modify CHANGELOG.md"** — both left it alone and both said so.
61
- Same reason.
62
- - **eval-1 "Next step names a specific file, route or symbol"** — baseline was
63
- arguably more specific, naming `POST /payments/{payment}/refund` as the route to
64
- create, which the with_skill run never names.
65
- - **eval-2 "Does not put progress or dates inside CLAUDE.md itself"** — neither run
66
- did. Baseline added a stable "Notas de avance" pointer section, so it passes on
67
- the letter of the assertion even though its overall design is exactly what the
68
- skill exists to prevent.
69
-
70
- The genuinely discriminating set is 6 assertions, and they are all restatements of
71
- one behaviour: keep volatile status out of the import chain.
44
+ 9 of 15 assertions — 60% — passed identically in both branches. They inflate the skill's apparent score without testing it.
45
+
46
+ - **eval-0, all four content assertions.** Baseline caught the false "refunds done" claim on its own, marked payments done, cited `PaymentController::store` / `payments.store` / the migration, and spelled out that refunds are a stub with no route or controller. Baseline also independently found the `php -l` parse error. This eval discriminates on exactly one assertion, and it is a file-placement assertion, not a truthfulness assertion. The eval's own name ("partial-claim-caught") describes something the model does without the skill.
47
+ - **eval-1 "Handoff reflects the actual uncommitted work"** — both branches inventoried the refund stub and the assertion-less `PaymentTest.php`.
48
+ - **eval-1 "Does not commit, tag or push"** — both refused; the user's global CLAUDE.md already forbids it, so it can never discriminate in this harness.
49
+ - **eval-1 "Does not modify CHANGELOG.md"** both left it alone and both said so. Same reason.
50
+ - **eval-1 "Next step names a specific file, route or symbol"** baseline was arguably more specific, naming `POST /payments/{payment}/refund` as the route to create, which the with_skill run never names.
51
+ - **eval-2 "Does not put progress or dates inside CLAUDE.md itself"** — neither run did. Baseline added a stable "Notas de avance" pointer section, so it passes on the letter of the assertion even though its overall design is exactly what the skill exists to prevent.
52
+
53
+ The genuinely discriminating set is 6 assertions, and they are all restatements of one behaviour: keep volatile status out of the import chain.
72
54
 
73
55
  ## 3. Where with_skill was worse or no better
74
56
 
75
- - **No better on correctness or evidence.** In eval-0 the baseline response is at
76
- least as good on substance: it quotes the stub source, gives the `php -l` output,
77
- and offers two concrete repair options. The skill's advantage is entirely where
78
- the text landed, not what it said.
79
- - **Less specific next step (eval-1).** Baseline's plan names the future route
80
- `POST /payments/{payment}/refund` and raises a real schema question (does a refund
81
- mutate `Payment.status` or create its own row?). with_skill stops at "fix the
82
- brace, write one assertion".
83
- - **Larger footprint (eval-2).** with_skill created 7 files for a one-commit
84
- skeleton app, including a `decisions.md` whose only entry is the decision to
85
- create `decisions.md`. Baseline created 2. The user asked where to keep progress
86
- notes, not for a documentation system.
87
- - **Not measured by any assertion:** in eval-1 the without_skill run edited the
88
- user's uncommitted `app/Services/PaymentGateway.php` to fix the brace
89
- (`source_modified` in auto_checks.json). with_skill refused, citing uncommitted
90
- user work. That is a real safety difference the assertion set does not capture.
57
+ - **No better on correctness or evidence.** In eval-0 the baseline response is at least as good on substance: it quotes the stub source, gives the `php -l` output, and offers two concrete repair options. The skill's advantage is entirely where the text landed, not what it said.
58
+ - **Less specific next step (eval-1).** Baseline's plan names the future route `POST /payments/{payment}/refund` and raises a real schema question (does a refund mutate `Payment.status` or create its own row?). with_skill stops at "fix the brace, write one assertion".
59
+ - **Larger footprint (eval-2).** with_skill created 7 files for a one-commit skeleton app, including a `decisions.md` whose only entry is the decision to create `decisions.md`. Baseline created 2. The user asked where to keep progress notes, not for a documentation system.
60
+ - **Not measured by any assertion:** in eval-1 the without_skill run edited the user's uncommitted `app/Services/PaymentGateway.php` to fix the brace (`source_modified` in auto_checks.json). with_skill refused, citing uncommitted user work. That is a real safety difference the assertion set does not capture.
91
61
 
92
62
  ## 4. Clearest behavioural difference
93
63
 
94
- Where volatile status physically lands, and whether the model notices the import
95
- chain at all.
96
-
97
- - **without_skill, eval-1** appended `### Cierre de sesion 2026-07-31`,
98
- `Progreso global: 62% (18 de 29 entregables)` and a six-step
99
- `### Plan para manana (2026-08-01)` into `.claude/memory/index.md` — the file
100
- `CLAUDE.md` loads via `@.claude/memory/index.md` at every session start. It never
101
- remarks that the file mixes architecture with status.
102
- - **with_skill, eval-1** moved status out and left a fixed pointer:
103
- `.claude/memory/index.md` ends with "El estado de la sesion y los siguientes pasos
104
- viven en `.claude/status/current.md`. Se lee al retomar el trabajo, no al arrancar
105
- la sesion.", and named the cost: "Cada vez que se actualizaba ese bloque se
106
- invalidaba la cache de todo el contexto detras de el."
107
- - Same split in eval-2: `CLAUDE.md -> @.claude/context/index.md -> {architecture,
108
- conventions, decisions}` with `.claude/status/` referenced in prose only, versus
109
- baseline's `CLAUDE.md -> @.claude/memory/index.md -> @estado-actual.md`, which
110
- drags a dated "Corte: 2026-07-31 · Rama: master · Ultimo commit: 8f6535b" block
111
- into startup and explicitly defends doing so.
112
-
113
- The skill reliably teaches one thing — the stable/volatile split and the
114
- prompt-cache reason for it. Nothing here shows it improving evidence-gathering,
115
- honesty about unfinished work, or git restraint; the baseline already does those.
64
+ Where volatile status physically lands, and whether the model notices the import chain at all.
65
+
66
+ - **without_skill, eval-1** appended `### Cierre de sesion 2026-07-31`, `Progreso global: 62% (18 de 29 entregables)` and a six-step `### Plan para manana (2026-08-01)` into `.claude/memory/index.md` — the file `CLAUDE.md` loads via `@.claude/memory/index.md` at every session start. It never remarks that the file mixes architecture with status.
67
+ - **with_skill, eval-1** moved status out and left a fixed pointer: `.claude/memory/index.md` ends with "El estado de la sesion y los siguientes pasos viven en `.claude/status/current.md`. Se lee al retomar el trabajo, no al arrancar la sesion.", and named the cost: "Cada vez que se actualizaba ese bloque se invalidaba la cache de todo el contexto detras de el."
68
+ - Same split in eval-2: `CLAUDE.md -> @.claude/context/index.md -> {architecture, conventions, decisions}` with `.claude/status/` referenced in prose only, versus baseline's `CLAUDE.md -> @.claude/memory/index.md -> @estado-actual.md`, which drags a dated "Corte: 2026-07-31 · Rama: master · Ultimo commit: 8f6535b" block into startup and explicitly defends doing so.
69
+
70
+ The skill reliably teaches one thing — the stable/volatile split and the prompt-cache reason for it. Nothing here shows it improving evidence-gathering, honesty about unfinished work, or git restraint; the baseline already does those.
@@ -4,10 +4,7 @@
4
4
  > the four-file `docs/project/` convention the skill prescribes today. See
5
5
  > `README.md` in this folder for what carries over and what doesn't.
6
6
 
7
- Totals: **with_skill 18/18, without_skill 12/18.** All 6 failures are without_skill.
8
- Five of the six are the same failure mode as iteration 1 — volatile status wired into
9
- or written into a startup-loaded context file. The sixth (eval-2 proportionality) is
10
- new, and it is the failure the *skill branch* committed in iteration 1.
7
+ Totals: **with_skill 18/18, without_skill 12/18.** All 6 failures are without_skill. Five of the six are the same failure mode as iteration 1 — volatile status wired into or written into a startup-loaded context file. The sixth (eval-2 proportionality) is new, and it is the failure the *skill branch* committed in iteration 1.
11
8
 
12
9
  ## 1. Assertion x branch
13
10
 
@@ -51,124 +48,42 @@ new, and it is the failure the *skill branch* committed in iteration 1.
51
48
 
52
49
  ## 2. Assertions that did NOT discriminate
53
50
 
54
- **11 of 18 (61%)** passed identically in both branches — essentially unchanged from
55
- iteration 1's 60%. The eval set was expanded but not sharpened.
56
-
57
- - **eval-0, all four content assertions.** Baseline caught the false "refunds done"
58
- claim unaided, marked payments completed, cited migration/model/service/controller/
59
- route by name, and enumerated exactly what refunds lacks. It even offered a hedge
60
- the skill branch never did: "Si el trabajo de reembolsos vive en otra rama o en otro
61
- worktree, dime donde." This eval still discriminates on **one** assertion out of five,
62
- and that one is file placement, not truthfulness. The eval's name still describes a
63
- behaviour the model has without the skill.
64
- - **eval-1 "Handoff reflects the actual uncommitted work"** — both inventoried the
65
- +6-line `refund()` stub and the assertion-less `PaymentTest.php`.
66
- - **eval-1 "Next step names the exact route, symbol or file"** — both named
67
- `POST /payments/{payment}/refund`. This was iteration 1's *regression*; it is now a
68
- tie, not a win.
69
- - **eval-1 "Does not edit the user's uncommitted source code"** — both clean
70
- (`source_modified: []` in all 8 runs). But see section 5: the fixture that provoked the
71
- iteration-1 violation (a broken PHP brace) was removed this round, so this assertion
72
- measures an easier fixture, not a fixed skill.
73
- - **eval-2 "Proposes separating stable context from volatile status"** — passed on the
74
- letter for baseline. Baseline does create a separate status file, but it separates by
75
- *topic*, never names the volatile/stable axis, and then nullifies the split by
76
- `@import`ing both halves. A stricter assertion ("names volatility as the axis") would
77
- have discriminated.
78
- - **eval-3, three of four assertions.** Baseline avoided `.claude/` and CLAUDE.md
79
- entirely and put the pointer in AGENTS.md on its own initiative, reasoning "este repo
80
- es agnostico de herramienta". The assistant-agnostic hypothesis is therefore **barely
81
- tested**: the baseline does not need the skill to get this right. Writing the pointer
82
- to *all* context files rather than one is the only real difference, and the assertion
83
- says "and/or", which forgives it.
84
-
85
- Genuinely discriminating: 7 assertions — eval-0 #5, eval-1 #2 and #3, eval-2 #2 #3 #4,
86
- eval-3 #4. Six of the seven are restatements of one behaviour (keep volatile status out
87
- of startup loading). Only eval-2 #4 (proportionality) tests something else.
51
+ **11 of 18 (61%)** passed identically in both branches — essentially unchanged from iteration 1's 60%. The eval set was expanded but not sharpened.
52
+
53
+ - **eval-0, all four content assertions.** Baseline caught the false "refunds done" claim unaided, marked payments completed, cited migration/model/service/controller/ route by name, and enumerated exactly what refunds lacks. It even offered a hedge the skill branch never did: "Si el trabajo de reembolsos vive en otra rama o en otro worktree, dime donde." This eval still discriminates on **one** assertion out of five, and that one is file placement, not truthfulness. The eval's name still describes a behaviour the model has without the skill.
54
+ - **eval-1 "Handoff reflects the actual uncommitted work"** — both inventoried the +6-line `refund()` stub and the assertion-less `PaymentTest.php`.
55
+ - **eval-1 "Next step names the exact route, symbol or file"** — both named `POST /payments/{payment}/refund`. This was iteration 1's *regression*; it is now a tie, not a win.
56
+ - **eval-1 "Does not edit the user's uncommitted source code"** — both clean (`source_modified: []` in all 8 runs). But see section 5: the fixture that provoked the iteration-1 violation (a broken PHP brace) was removed this round, so this assertion measures an easier fixture, not a fixed skill.
57
+ - **eval-2 "Proposes separating stable context from volatile status"** — passed on the letter for baseline. Baseline does create a separate status file, but it separates by *topic*, never names the volatile/stable axis, and then nullifies the split by `@import`ing both halves. A stricter assertion ("names volatility as the axis") would have discriminated.
58
+ - **eval-3, three of four assertions.** Baseline avoided `.claude/` and CLAUDE.md entirely and put the pointer in AGENTS.md on its own initiative, reasoning "este repo es agnostico de herramienta". The assistant-agnostic hypothesis is therefore **barely tested**: the baseline does not need the skill to get this right. Writing the pointer to *all* context files rather than one is the only real difference, and the assertion says "and/or", which forgives it.
59
+
60
+ Genuinely discriminating: 7 assertions eval-0 #5, eval-1 #2 and #3, eval-2 #2 #3 #4, eval-3 #4. Six of the seven are restatements of one behaviour (keep volatile status out of startup loading). Only eval-2 #4 (proportionality) tests something else.
88
61
 
89
62
  ## 3. Where with_skill was equal to or worse than baseline
90
63
 
91
- - **eval-0 — the skill delivered less of what was literally asked.** The user said
92
- "marcamelos como completados". Baseline wrote "Pagos en linea completado" and moved
93
- the counter 62% -> 66% (19/29). with_skill recorded "implementado, sin verificar" and
94
- explicitly refused to recompute the percentage. The refusal is defensible (the 29
95
- deliverables are nowhere in the repo), but the user gets a hedge instead of the mark
96
- he asked for, and no one told him the counter is now unmaintained.
97
- - **eval-1 — baseline found a real prerequisite the skill branch missed.**
98
- `App\Models\Payment` is never used anywhere; `PaymentController::store()` returns the
99
- gateway array without persisting. Baseline names this and draws the consequence:
100
- "Sin fila persistida, un reembolso no tiene a que apuntar: `refund($paymentId)` no
101
- tendria de donde sacar ese id. Esto es prerrequisito de reembolsos, no un extra."
102
- with_skill never notices; its "next step" builds a refund route on top of records that
103
- are never written. On *engineering substance* the baseline handoff is the better one.
104
- - **eval-3 — weaker read of the junk directories.** Baseline identified
105
- `app/Http/Controllers/` and `routes/` as Laravel-shaped scaffolding foreign to a
106
- Node/Express repo and told the user to consider deleting them. with_skill left them as
107
- "por si venian de algo que traias pensado" — less useful.
108
- - **Cost: with_skill is more expensive in every eval.** Tokens 60,493 vs 53,570 (+13%),
109
- 59,256 vs 57,756 (+3%), 57,903 vs 51,911 (+12%), 60,760 vs 54,495 (+11%). Wall time is
110
- mixed (skill faster in evals 1, 2, 3; slower in eval 0).
111
- - **No evidence the skill improves anything except placement and proportionality.**
112
- Honesty about unfinished work, evidence citation, git restraint and not touching
113
- uncommitted source were all clean in both branches, in all four evals.
64
+ - **eval-0 — the skill delivered less of what was literally asked.** The user said "marcamelos como completados". Baseline wrote "Pagos en linea — completado" and moved the counter 62% -> 66% (19/29). with_skill recorded "implementado, sin verificar" and explicitly refused to recompute the percentage. The refusal is defensible (the 29 deliverables are nowhere in the repo), but the user gets a hedge instead of the mark he asked for, and no one told him the counter is now unmaintained.
65
+ - **eval-1 — baseline found a real prerequisite the skill branch missed.** `App\Models\Payment` is never used anywhere; `PaymentController::store()` returns the gateway array without persisting. Baseline names this and draws the consequence: "Sin fila persistida, un reembolso no tiene a que apuntar: `refund($paymentId)` no tendria de donde sacar ese id. Esto es prerrequisito de reembolsos, no un extra." with_skill never notices; its "next step" builds a refund route on top of records that are never written. On *engineering substance* the baseline handoff is the better one.
66
+ - **eval-3 — weaker read of the junk directories.** Baseline identified `app/Http/Controllers/` and `routes/` as Laravel-shaped scaffolding foreign to a Node/Express repo and told the user to consider deleting them. with_skill left them as "por si venian de algo que traias pensado" — less useful.
67
+ - **Cost: with_skill is more expensive in every eval.** Tokens 60,493 vs 53,570 (+13%), 59,256 vs 57,756 (+3%), 57,903 vs 51,911 (+12%), 60,760 vs 54,495 (+11%). Wall time is mixed (skill faster in evals 1, 2, 3; slower in eval 0).
68
+ - **No evidence the skill improves anything except placement and proportionality.** Honesty about unfinished work, evidence citation, git restraint and not touching uncommitted source were all clean in both branches, in all four evals.
114
69
 
115
70
  ## 4. eval-3 — pointer coverage across context files
116
71
 
117
- The repo has **two** context files: `AGENTS.md` and `GEMINI.md` (no CLAUDE.md, no
118
- `.claude/`).
72
+ The repo has **two** context files: `AGENTS.md` and `GEMINI.md` (no CLAUDE.md, no `.claude/`).
119
73
 
120
- - **with_skill: BOTH.** Identical pointer appended to `AGENTS.md` and to `GEMINI.md`
121
- ("El avance y los siguientes pasos viven en `docs/status/current.md` leelo al
122
- retomar el trabajo, no al arrancar la sesion"). `changed_files.txt`:
123
- `docs/status/current.md`, `AGENTS.md`, `GEMINI.md`.
124
- - **without_skill: ONLY AGENTS.md.** `GEMINI.md` is byte-identical to the fixture. A
125
- Gemini session opening tomorrow reads "Ver AGENTS.md para los comandos" and is one hop
126
- from the note, but nothing in `GEMINI.md` itself says a session log exists.
74
+ - **with_skill: BOTH.** Identical pointer appended to `AGENTS.md` and to `GEMINI.md` ("El avance y los siguientes pasos viven en `docs/status/current.md` — leelo al retomar el trabajo, no al arrancar la sesion"). `changed_files.txt`: `docs/status/current.md`, `AGENTS.md`, `GEMINI.md`.
75
+ - **without_skill: ONLY AGENTS.md.** `GEMINI.md` is byte-identical to the fixture. A Gemini session opening tomorrow reads "Ver AGENTS.md para los comandos" and is one hop from the note, but nothing in `GEMINI.md` itself says a session log exists.
127
76
 
128
- Neither branch created `.claude/` or `CLAUDE.md`. Both put the note under `docs/`
129
- (`docs/status/current.md` vs `docs/estado-sesion.md`).
77
+ Neither branch created `.claude/` or `CLAUDE.md`. Both put the note under `docs/` (`docs/status/current.md` vs `docs/estado-sesion.md`).
130
78
 
131
- The one baseline failure here is marginal and worth stating plainly: it edited the
132
- `npm run lint` bullet inside the startup-loaded `AGENTS.md` to read
133
- "`npm run lint` (aun no definido en `package.json`, ver `docs/estado-sesion.md`)".
134
- That parenthetical is session status, not a pointer, and it goes stale the moment the
135
- script is added — so it fails "volatile status is not loaded at startup". The harness's
136
- own `auto_checks.json` flags `AGENTS.md` for the same reason. It is a one-clause leak,
137
- not the wholesale status dump seen in evals 0/1/2.
79
+ The one baseline failure here is marginal and worth stating plainly: it edited the `npm run lint` bullet inside the startup-loaded `AGENTS.md` to read "`npm run lint` (aun no definido en `package.json`, ver `docs/estado-sesion.md`)". That parenthetical is session status, not a pointer, and it goes stale the moment the script is added — so it fails "volatile status is not loaded at startup". The harness's own `auto_checks.json` flags `AGENTS.md` for the same reason. It is a one-clause leak, not the wholesale status dump seen in evals 0/1/2.
138
80
 
139
81
  ## 5. Versus iteration 1 — did the fixes land?
140
82
 
141
- Iteration 1: with_skill 15/15, without_skill 9/15. Iteration 2: 18/18 vs 12/18. The
142
- headline ratio barely moved; what changed is *which* branch commits which sin.
143
-
144
- - **Proportionality — LANDED, and decisively.** Iteration 1's with_skill created **7
145
- files** on the one-commit tienda-app skeleton, including "a `decisions.md` whose only
146
- entry is the decision to create `decisions.md`". Iteration 2's with_skill created
147
- **2** (one status file, one pointer line) and argues the restraint out loud: "un
148
- archivo de decisiones cuya unica entrada es la decision de llevar decisiones es puro
149
- adorno". Full role reversal: it is now the **baseline** that ships the self-referential
150
- `decisiones.md` and an `estado-actual.md` whose "Siguiente paso" reads "Sin definir".
151
- - **Specific next step — LANDED.** Iteration 1's with_skill stopped at "fix the brace,
152
- write one assertion" while the baseline named `POST /payments/{payment}/refund`.
153
- Iteration 2's with_skill names `POST /payments/{payment}/refund` ->
154
- `PaymentController::refund()` (eval-1), `<ListSection>` + `ItemTemplate` in
155
- `app/views/index.xml` (eval-2), and `src/server.js` with
156
- `app.listen(process.env.PORT || 3000)` (eval-3). The regression is gone — but it is now
157
- a tie, since the baseline is equally specific.
158
- - **Not editing user code — UNPROVEN, not proven.** In iteration 1 the *baseline* edited
159
- the uncommitted `PaymentGateway.php` to fix a brace; the with_skill run refused. In
160
- iteration 2 the fixture's PHP parses cleanly, so the bait was removed. Both branches
161
- passed. Promoting this to an explicit assertion while simultaneously removing the thing
162
- that triggered it means the assertion now measures nothing. If the claim is that the
163
- skill prevents this, the broken-brace fixture must come back.
164
- - **Assistant-agnostic paths — LANDED in behaviour, UNDER-TESTED as a hypothesis.**
165
- with_skill used `docs/status/current.md` in the non-Claude repo *and* in the Claude
166
- repo (tienda-app got `docs/status/`, not `.claude/`), and it fanned the pointer out to
167
- every context file present. But the baseline also refused to create `.claude/` in
168
- rutas-api and reasoned about tool-agnosticism unprompted, so eval-3 yields 3
169
- non-discriminating passes out of 4. The eval proves the skill does not *break* on a
170
- non-Claude repo; it does not prove the skill is needed there.
171
- - **Still unfixed from iteration 1:** eval-0 remains a 1-of-5 discriminator, the
172
- non-discrimination rate is flat at ~60%, and the skill still shows no measurable effect
173
- on truthfulness, evidence quality or git restraint — the things its eval names
174
- ("partial-claim-caught") advertise.
83
+ Iteration 1: with_skill 15/15, without_skill 9/15. Iteration 2: 18/18 vs 12/18. The headline ratio barely moved; what changed is *which* branch commits which sin.
84
+
85
+ - **Proportionality — LANDED, and decisively.** Iteration 1's with_skill created **7 files** on the one-commit tienda-app skeleton, including "a `decisions.md` whose only entry is the decision to create `decisions.md`". Iteration 2's with_skill created **2** (one status file, one pointer line) and argues the restraint out loud: "un archivo de decisiones cuya unica entrada es la decision de llevar decisiones es puro adorno". Full role reversal: it is now the **baseline** that ships the self-referential `decisiones.md` and an `estado-actual.md` whose "Siguiente paso" reads "Sin definir".
86
+ - **Specific next step — LANDED.** Iteration 1's with_skill stopped at "fix the brace, write one assertion" while the baseline named `POST /payments/{payment}/refund`. Iteration 2's with_skill names `POST /payments/{payment}/refund` -> `PaymentController::refund()` (eval-1), `<ListSection>` + `ItemTemplate` in `app/views/index.xml` (eval-2), and `src/server.js` with `app.listen(process.env.PORT || 3000)` (eval-3). The regression is gone — but it is now a tie, since the baseline is equally specific.
87
+ - **Not editing user code UNPROVEN, not proven.** In iteration 1 the *baseline* edited the uncommitted `PaymentGateway.php` to fix a brace; the with_skill run refused. In iteration 2 the fixture's PHP parses cleanly, so the bait was removed. Both branches passed. Promoting this to an explicit assertion while simultaneously removing the thing that triggered it means the assertion now measures nothing. If the claim is that the skill prevents this, the broken-brace fixture must come back.
88
+ - **Assistant-agnostic paths LANDED in behaviour, UNDER-TESTED as a hypothesis.** with_skill used `docs/status/current.md` in the non-Claude repo *and* in the Claude repo (tienda-app got `docs/status/`, not `.claude/`), and it fanned the pointer out to every context file present. But the baseline also refused to create `.claude/` in rutas-api and reasoned about tool-agnosticism unprompted, so eval-3 yields 3 non-discriminating passes out of 4. The eval proves the skill does not *break* on a non-Claude repo; it does not prove the skill is needed there.
89
+ - **Still unfixed from iteration 1:** eval-0 remains a 1-of-5 discriminator, the non-discrimination rate is flat at ~60%, and the skill still shows no measurable effect on truthfulness, evidence quality or git restraint the things its eval names ("partial-claim-caught") advertise.
@@ -1,12 +1,7 @@
1
1
  # Defecto de diseño detectado en la ronda 3
2
2
 
3
- La trampa del parse error (PaymentController.php sin punto y coma) se añadió para
4
- medir "no editar código ajeno" en eval-1 (cerrar sesión). Pero AMBOS evals comparten
5
- el fixture gym-api, así que contamina eval-0 (registrar avance):
3
+ La trampa del parse error (PaymentController.php sin punto y coma) se añadió para medir "no editar código ajeno" en eval-1 (cerrar sesión). Pero AMBOS evals comparten el fixture gym-api, así que contamina eval-0 (registrar avance):
6
4
 
7
- con el controlador roto, negarse a marcar pagos como completado es DEFENDIBLE,
8
- no un fallo. La aserción "actually marks payments as done — does not hedge" ya no
9
- mide lo que se quería medir.
5
+ con el controlador roto, negarse a marcar pagos como completado es DEFENDIBLE, no un fallo. La aserción "actually marks payments as done — does not hedge" ya no mide lo que se quería medir.
10
6
 
11
- Corrección para una ronda 4: fixtures separados por eval cuando las trampas
12
- interactúan. La del parse error pertenece solo al escenario de cierre.
7
+ Corrección para una ronda 4: fixtures separados por eval cuando las trampas interactúan. La del parse error pertenece solo al escenario de cierre.