@ara-commons/ara-skills 0.3.0 → 0.3.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/package.json +1 -1
- package/skills/compiler/SKILL.md +37 -11
- package/skills/compiler/references/ara-schema.md +66 -22
- package/skills/compiler/references/validation-checklist.md +32 -8
- package/skills/research-manager/SKILL.md +33 -8
- package/skills/research-manager/references/event-taxonomy.md +1 -1
- package/skills/rigor-reviewer/SKILL.md +1 -0
- package/skills/rigor-reviewer/references/review-dimensions.md +1 -0
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@ara-commons/ara-skills",
|
|
3
|
-
"version": "0.3.
|
|
3
|
+
"version": "0.3.1",
|
|
4
4
|
"description": "Install Agent-Native Research Artifact (ARA) skills — compiler, research-manager, rigor-reviewer — into Claude Code, Cursor, OpenCode, Gemini CLI, Codex, and more.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
package/skills/compiler/SKILL.md
CHANGED
|
@@ -120,11 +120,26 @@ For non-trivial figures (dense plots, log axes, multi-panel, anything needing re
|
|
|
120
120
|
**Stage 2 — Cognitive Mapping**
|
|
121
121
|
Map the atoms into `/logic/`:
|
|
122
122
|
- **problem.md**: observations (with numbers) → gaps → key insight → assumptions
|
|
123
|
-
- **claims.md**: falsifiable claims with proof pointers to experiment IDs (E01, E02…).
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
123
|
+
- **claims.md**: falsifiable claims with proof pointers to experiment IDs (E01, E02…). A claim's job
|
|
124
|
+
is the **takeaway, not the record**. Before writing a `Statement`, distill: for each result,
|
|
125
|
+
ablation, or dead-end, ask what it *reveals* — the mechanism or relationship behind the number, the
|
|
126
|
+
WHY a reader would reuse — and make THAT the `Statement`. Look across results too, not one at a
|
|
127
|
+
time: where several experiments together reveal a relationship none shows alone — whether they
|
|
128
|
+
agree on it or differ in a way that reveals what bounds it — make THAT relationship the claim
|
|
129
|
+
(`Proof` spanning them, `Dependencies` the narrower claims it rests on), rather than settling for
|
|
130
|
+
one claim per experiment. The recipe name, run IDs, and numbers are
|
|
131
|
+
the evidence *for* the takeaway, not the takeaway itself: they live in `Evidence basis`/`Proof`,
|
|
132
|
+
referenced and never restated in the Statement. A `Statement`'s subject is a mechanism/relationship,
|
|
133
|
+
never a named recipe/config/run, and carries no run numbers, scores, step counts, or p-values. Bound
|
|
134
|
+
every Statement with a `Conditions` field (the regime + the untested boundary) and a substantive
|
|
135
|
+
`Falsification criteria` (about the system for a mechanism claim, about the benchmark's behavior for
|
|
136
|
+
a methodological one) — this accountability, not a narrowed sentence, is what keeps a generalized
|
|
137
|
+
claim honest. Don't upgrade a validation-metric result into a claim about training dynamics without
|
|
138
|
+
training-side evidence. Stating the mechanism a result reveals is the goal **even from a single
|
|
139
|
+
instance** — what you must NOT do is extrapolate it into a universal law beyond its regime, or
|
|
140
|
+
assert a distinction the design cannot disentangle; that limit goes in `Conditions` so the
|
|
141
|
+
`Statement` can still carry the mechanism rather than collapsing back to a recipe-and-number.
|
|
142
|
+
**Ground every load-bearing number in a claim like code** (the `# Grounding` discipline,
|
|
128
143
|
applied to numbers): before writing it, open its source and copy the matched line verbatim into a
|
|
129
144
|
`**Sources**` entry — `<value> ← <source ref> «matched line» [input]` for values that were set
|
|
130
145
|
(cite where they're defined), `[result]` for values a run produced (cite the log/output that
|
|
@@ -135,7 +150,10 @@ Map the atoms into `/logic/`:
|
|
|
135
150
|
- **concepts.md**: the paper's genuine technical terms, formally defined
|
|
136
151
|
- **experiments.md**: declarative verification/analysis plans (NO exact numbers — directional
|
|
137
152
|
only). "Experiment" generalizes to the field's way of testing a claim: an eval run, a statistical
|
|
138
|
-
test, a proof obligation, a user study.
|
|
153
|
+
test, a proof obligation, a user study. Link each experiment to where its results are filed
|
|
154
|
+
(`Evidence`) and to what produced it (`Run`, including failed/ablated runs). Claims and experiments
|
|
155
|
+
are many-to-many — a claim that generalises across runs lists every experiment in its `Proof`;
|
|
156
|
+
don't mirror one experiment per claim.
|
|
139
157
|
- **solution/**: the method layer — `constraints.md` (limitations/assumptions) is always present;
|
|
140
158
|
beyond it, create the files the paper's content actually calls for (architecture, algorithm,
|
|
141
159
|
method, study design, formalization, proofs, heuristics — whatever fits the work). You decide
|
|
@@ -152,8 +170,11 @@ whichever layer fits best, preserving the source's granularity. Never silently d
|
|
|
152
170
|
or released form, *distinct from the prose that describes it*. `src/environment.md` is always
|
|
153
171
|
required (reproducibility). Beyond it, one rule decides everything:
|
|
154
172
|
|
|
155
|
-
> **
|
|
156
|
-
>
|
|
173
|
+
> **Represent every concrete artifact losslessly. When it persists in a linkable external store (a
|
|
174
|
+
> run database, a released/versioned repo), point to it — a comprehensive `src/artifacts.md` index,
|
|
175
|
+
> one link per artifact (every run, config, log, script), nothing aggregated or copied. Capture it
|
|
176
|
+
> into `src/execution/` only when it would otherwise be lost — code that lives solely inside the
|
|
177
|
+
> paper, or a source not externally persisted. Never re-encode a prose-only description as code.**
|
|
157
178
|
|
|
158
179
|
A concrete artifact is real content the cognitive layer doesn't already hold — capture it (grounded
|
|
159
180
|
in the real repo/files when provided), in whatever directory fits. But a method conveyed only in
|
|
@@ -218,7 +239,7 @@ Run ARA Seal Level 1. Check:
|
|
|
218
239
|
- Mandatory-core dirs exist (`logic/`, `logic/solution/`, `src/`, `trace/`, `evidence/`) and all
|
|
219
240
|
mandatory-core files exist and are non-empty
|
|
220
241
|
- PAPER.md has valid frontmatter (title, authors, year) + a Layer Index
|
|
221
|
-
- claims.md has C01+ blocks with Statement, Status, Falsification criteria, Proof
|
|
242
|
+
- claims.md has C01+ blocks with Statement, Conditions, Status, Falsification criteria, Proof; Conditions non-trivial
|
|
222
243
|
- experiments.md has E01+ blocks with Verifies, Setup, Procedure, Expected outcome (no exact numbers)
|
|
223
244
|
- concepts.md, related_work.md, constraints.md non-trivial; any heuristics blocks have Rationale,
|
|
224
245
|
Sensitivity, Bounds
|
|
@@ -235,6 +256,11 @@ Run ARA Seal Level 1. Check:
|
|
|
235
256
|
- **Cited locations verified** (Rule 15): every repo path/`file:line` exists and is in range;
|
|
236
257
|
spot-check that trace `source_refs` and evidence `Source` actually contain the cited content; no
|
|
237
258
|
repo fact transcribed from the paper without checking the real file
|
|
259
|
+
- **Statement is a takeaway, not a record** — its own dedicated FAIL pass, symmetric to the
|
|
260
|
+
number-sources pass: scan EVERY claim's `Statement`. It FAILS if the Statement's subject is a named
|
|
261
|
+
recipe/config/run, or if the Statement contains a run number, n-count, score, step/bin count, or
|
|
262
|
+
p-value. Such a claim is a leaderboard coordinate, not knowledge — the mechanism it reveals must
|
|
263
|
+
become the Statement and the numbers move to `Evidence basis`/`Proof`. Exhaustive, not spot-checked
|
|
238
264
|
- **Number sources bound** (claims & heuristics) — run this as its own dedicated pass, one job: for
|
|
239
265
|
*each* `**Sources**` entry, re-open the cited `file:line` (or trace `node:field`) and confirm the
|
|
240
266
|
verbatim «quote» is actually there and the number in the `Statement`/`Rationale` matches the value
|
|
@@ -266,11 +292,11 @@ key stats (claims, experiments, concepts, tree nodes, evidence tables/figures).
|
|
|
266
292
|
7. **"Not specified"**: if information is genuinely unavailable, write "Not specified in paper" — never guess
|
|
267
293
|
8. **No fake source labels**: never call a derived subset `Table N`/`Figure N` unless it faithfully reproduces the original
|
|
268
294
|
9. **No synthetic trace history**: don't invent decisions, dead ends, or experiments not explicit in the inputs; mark inferred trajectories as inferred or omit them
|
|
269
|
-
10. **
|
|
295
|
+
10. **Distill the takeaway, then bound it**: a `Statement` is the mechanism or relationship a result reveals — the reusable WHY — with the named recipe and its numbers demoted to `Evidence basis`/`Proof`, never restated in the sentence and never its subject. Keep it accountable by an explicit `Conditions` regime, a substantive `Falsification criteria` (about the system, or about the benchmark's behavior for a methodological claim), and grounded `Proof` — not by narrowing the sentence to a measured value. A single instance still licenses a mechanism `Statement`: what is forbidden is extrapolating it into a universal law beyond its regime, or asserting a distinction the design cannot disentangle — those limits go in `Conditions`, they do not shrink the Statement back to a recipe-and-number. Still separate observation from interpretation: the numbers stay in the evidence layer, reached via `Proof`/`Evidence basis`
|
|
270
296
|
11. **Visual extraction is honest extraction**: read figures by looking; mark estimates `≈` with extraction method + confidence; never present a digitized estimate as exact, invent points for an unreadable figure, or turn a diagram into a fake data table
|
|
271
297
|
12. **Complete, ordered evidence**: file EVERY numbered table and figure, in order — a systematic sweep, not a lucky sample — each as a markdown transcription PLUS a saved screenshot (`.png`). No early stopping; account for any object you don't file
|
|
272
298
|
13. **Fit the file set to the paper, not the paper to a template**: only PAPER.md + the mandatory core are required. Beyond them, generate the files THIS work actually warrants and nothing it doesn't have. Never force inappropriate files (e.g. model-training configs onto an eval or theory paper)
|
|
273
|
-
14. **`src/` holds concrete artifacts, not re-encoded prose**: capture every concrete artifact the source actually contains, in its native form, grounded in real files. Three sides: (a) never fabricate a code stub from a prose-only method — it already lives in `logic/`, so a `.py` just duplicates it; (b) never drop a concrete artifact that does exist — a lone `environment.md` is wrong when the work has one; (c) when the
|
|
299
|
+
14. **`src/` holds concrete artifacts, not re-encoded prose**: capture every concrete artifact the source actually contains, in its native form, grounded in real files. Three sides: (a) never fabricate a code stub from a prose-only method — it already lives in `logic/`, so a `.py` just duplicates it; (b) never drop a concrete artifact that does exist — a lone `environment.md` is wrong when the work has one; (c) when the work's artifacts **persist in a linkable external store** (a run database, a released or versioned repo), represent them as a **comprehensive pointer index** in `src/artifacts.md` — one link per artifact (every run, config, log, script), nothing aggregated into a vague bucket, nothing copied; a lossy subset-copy is the failure. **Transcribe real source into `src/execution/` only when it would otherwise be lost** — code that lives solely inside the paper, or a source not externally persisted (then `# Grounding: transcribed`, cite path). No implementation in the input → neither applies.
|
|
274
300
|
15. **Source-bounded minimums**: any count or required field is a target, never a license to invent. If the source supports fewer, produce what is real and note the shortfall; for an unstated field write "Not specified in paper" rather than guessing
|
|
275
301
|
16. **Cite by verification, and ask on conflict**: a source reference (evidence `Source`, trace `source_refs`, claim `Proof`, a repo `file:line`/path) promises the cited location actually contains the claim — open it and confirm. Never transcribe a *description* of an artifact as a verified fact about it. **When the code repo and the paper disagree on a fact (line count, path, value, behavior), do NOT pick one silently — surface the conflict to the user and ask which source to follow.** If unverifiable and the user is unavailable, attribute it ("per §X") or omit. Carry a statistic's scope/denominator in its `Source`. **This extends to every load-bearing number in a claim/heuristic `Statement`/`Rationale`: it carries a `**Sources**` entry whose verbatim «quote» you opened and confirmed contains that value — a memory-filled value or a bare path is fabrication; use `[pending]` when you cannot open the source**
|
|
276
302
|
|
|
@@ -159,20 +159,51 @@ Rule: if a filename includes a source label such as `table3` or `figure4`, it sh
|
|
|
159
159
|
|
|
160
160
|
Each claim MUST have ALL fields:
|
|
161
161
|
```markdown
|
|
162
|
-
## C{NN}: {
|
|
163
|
-
- **Statement**: {
|
|
162
|
+
## C{NN}: {generalized title — the takeaway, not a recipe/result name}
|
|
163
|
+
- **Statement**: {the generalized, mechanistic conclusion the evidence supports; subject = a mechanism/relationship, never a named recipe; carries NO run numbers}
|
|
164
|
+
- **Conditions**: {under what conditions it holds; the regime; the known untested boundary}
|
|
164
165
|
- **Status**: {hypothesis|supported|refuted}
|
|
165
|
-
- **Falsification criteria**: {
|
|
166
|
+
- **Falsification criteria**: {a concrete observation that would disprove it — for a mechanism claim, about the system/world; for a methodological/regime claim, about the benchmark's behavior. Not a tautology or a re-run of the same gate}
|
|
166
167
|
- **Proof**: [{experiment IDs: E01, E02}]
|
|
167
|
-
- **Evidence basis**: {
|
|
168
|
-
- **
|
|
169
|
-
- **Dependencies**: {other claim IDs, if any}
|
|
168
|
+
- **Evidence basis**: {what the cited evidence shows — point to it; do NOT restate run numbers in the Statement}
|
|
169
|
+
- **Dependencies**: {claim IDs this one rests on — the narrower claims a more general claim draws on, or a claim it corrects/refines; not mere shared setup; omit if it rests only on its own evidence}
|
|
170
170
|
- **Tags**: {comma-separated keywords}
|
|
171
171
|
```
|
|
172
172
|
|
|
173
173
|
Proof MUST reference experiment IDs from experiments.md.
|
|
174
174
|
Each proofed experiment should in turn be backed by evidence files whose rows or measurements actually match the claim being asserted.
|
|
175
|
-
`Statement`
|
|
175
|
+
`Statement` is the **generalized conclusion the evidence supports** — a mechanism or relationship,
|
|
176
|
+
not a restatement of run numbers. The claim is kept falsifiable and honest by `Conditions` (the
|
|
177
|
+
regime it holds in + the untested boundary) and a `Falsification criteria`, not by narrowing the
|
|
178
|
+
sentence to a single measured value. Numbers (n, scores, step counts, run IDs) live in the evidence
|
|
179
|
+
layer and are reached via `Proof`/`Evidence basis`, never pasted into `Statement`. `Conditions` is
|
|
180
|
+
mandatory: a generalized Statement with no Conditions is an unbounded slogan.
|
|
181
|
+
|
|
182
|
+
**Distill the mechanism; bound the reach.** Before writing a `Statement`, ask what the result
|
|
183
|
+
*reveals* — the mechanism or relationship a reader would reuse — and state that; the recipe and its
|
|
184
|
+
numbers are the evidence for it, not the claim, and never its subject. A single instance still
|
|
185
|
+
licenses a mechanism `Statement`; what is forbidden is extrapolating it into a universal law beyond
|
|
186
|
+
its regime, or asserting a distinction the design cannot disentangle. Put that boundary in
|
|
187
|
+
`Conditions` — it bounds *where* the claim holds and is not a license for the verb to over-reach.
|
|
188
|
+
`Conditions` carries the limits so the `Statement` can carry the mechanism.
|
|
189
|
+
|
|
190
|
+
**A claim's evidence may be one result or several read together.** Most claims distill what a single
|
|
191
|
+
result reveals; but where several experiments together reveal a relationship none shows alone —
|
|
192
|
+
whether they agree on it, or differ in a way that itself reveals what bounds or explains the
|
|
193
|
+
difference — that relationship is the claim. Write it as an ordinary `## C` block whose `Proof` lists
|
|
194
|
+
every experiment it draws on and whose `Dependencies` names the narrower claims it rests on; the same
|
|
195
|
+
distill-the-mechanism, bound-the-reach discipline applies. State the most general relationship the
|
|
196
|
+
evidence supports — bounded by `Conditions`, never asserted past what those experiments jointly show —
|
|
197
|
+
rather than settling for one claim per experiment. A claim need not be about the object under study:
|
|
198
|
+
a reusable relationship the work itself exposes, including in how it was run, is worth a claim.
|
|
199
|
+
|
|
200
|
+
**The attribution trap (the most common miss).** An ablation / leave-one-out that shows *which*
|
|
201
|
+
components dominate is the *evidence*, not the claim. A Statement that merely names the load-bearing
|
|
202
|
+
vs decorative components passes the no-numbers gate but is still a league table of *this* system.
|
|
203
|
+
Apply the **name-deletion test**: strike your system's component names from the Statement — if
|
|
204
|
+
nothing a stranger working on a different stack could reuse survives, you wrote attribution. State
|
|
205
|
+
instead what the ranking reveals about the *class* of system; the named components and their deltas
|
|
206
|
+
live in `Evidence basis`, reached via `Proof`.
|
|
176
207
|
|
|
177
208
|
---
|
|
178
209
|
|
|
@@ -192,11 +223,15 @@ borrowed terms to reach 5 (Rule 14). One section per concept:
|
|
|
192
223
|
|
|
193
224
|
## logic/experiments.md
|
|
194
225
|
|
|
195
|
-
≥3 experiments. Declarative plans, NOT scripts. NO exact numerical results.
|
|
226
|
+
≥3 experiments. Declarative plans, NOT scripts. NO exact numerical results. Experiments and claims
|
|
227
|
+
are **many-to-many**: one experiment may verify several claims, and a claim that generalises across
|
|
228
|
+
runs lists every experiment it draws on in its `Proof` — do not force a 1:1 claim↔experiment ledger.
|
|
196
229
|
|
|
197
230
|
```markdown
|
|
198
231
|
## E{NN}: {Short title}
|
|
199
|
-
- **Verifies**: {claim IDs
|
|
232
|
+
- **Verifies**: {claim IDs this run bears on — may be several}
|
|
233
|
+
- **Evidence**: {evidence file(s) where this run's results are recorded — `evidence/…`; "pending" if not yet filed}
|
|
234
|
+
- **Run**: {what produced this result — a `src/execution/` file (or other `src/` artifact) when captured, else a link/ref into the source repo or run database; give it for EVERY experiment, including failed or ablated runs}
|
|
200
235
|
- **Setup**:
|
|
201
236
|
- Model: {model name and size}
|
|
202
237
|
- Hardware: {GPU type, count, memory}
|
|
@@ -286,12 +321,20 @@ field format:
|
|
|
286
321
|
|
|
287
322
|
## src/execution/{module}.py (when the work warrants it — grounded or absent)
|
|
288
323
|
|
|
289
|
-
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
|
|
324
|
+
Capture here is the **fallback, not the default**: transcribe code into `src/execution/` only when it
|
|
325
|
+
would otherwise be **lost** — it exists solely inside the paper, or its source is not externally
|
|
326
|
+
persisted. When the work's code/runs **persist in a linkable external store** (a repo, a run
|
|
327
|
+
database), do NOT copy them here — index them comprehensively in `src/artifacts.md` (see below). When
|
|
328
|
+
capture IS the call: actual repo code → capture real runnable files in native form (transcribed); only
|
|
329
|
+
pseudocode/equations the paper prints → a reconstructed stub of the **novel mechanism**. Either way it
|
|
293
330
|
must be grounded — never fabricated.
|
|
294
331
|
|
|
332
|
+
When the input is a run database / repo of many experiment runs, index it **comprehensively** in
|
|
333
|
+
`src/artifacts.md`: a link for **every** run and artifact (the per-run logs — e.g. a `runs.jsonl`
|
|
334
|
+
already indexes each — plus every config, candidate, log, and script), nothing aggregated into a vague
|
|
335
|
+
bucket and nothing copied. Each experiment's `Run` field points at the relevant entries. A lossy
|
|
336
|
+
subset — only the winning run, or runs collapsed into a single directory link — is the failure.
|
|
337
|
+
|
|
295
338
|
Every file declares its grounding on the first line:
|
|
296
339
|
```python
|
|
297
340
|
# Grounding: transcribed — adapted from repo code; cite file:line in docstrings
|
|
@@ -320,17 +363,18 @@ pseudo-code — that information already lives in `logic/solution/`, and re-enco
|
|
|
320
363
|
duplicates it.** A concrete artifact that IS raw "code" — e.g. a prompt or template — is different:
|
|
321
364
|
store it verbatim in `src/prompts/`, don't paraphrase it. A hollow invented API is a hallucination.
|
|
322
365
|
|
|
323
|
-
## src/artifacts.md (
|
|
366
|
+
## src/artifacts.md (the artifact index — comprehensive pointer file when the source persists externally)
|
|
324
367
|
|
|
325
|
-
`src/` must
|
|
326
|
-
|
|
327
|
-
|
|
368
|
+
`src/` must represent the implementation **losslessly**. When the work's artifacts **persist in a
|
|
369
|
+
linkable external store** (a repo, a run database, a released tool/dataset), `artifacts.md` is the
|
|
370
|
+
**comprehensive pointer index** — a link to **every** artifact (every run, config, log, script,
|
|
371
|
+
released binary, dataset), grounded in the real files, nothing aggregated into a vague bucket and
|
|
372
|
+
nothing copied. One block (or row) per artifact:
|
|
328
373
|
|
|
329
|
-
**
|
|
330
|
-
|
|
331
|
-
|
|
332
|
-
|
|
333
|
-
location. Naming a real `.py`/`.js`/… file here instead of capturing it is a coverage failure.
|
|
374
|
+
**Capture is the fallback, not the default.** Transcribe a file into `src/execution/` only when it
|
|
375
|
+
would otherwise be **lost** — code that lives solely inside the paper, or a source not externally
|
|
376
|
+
persisted. When the source persists and is linkable, point to it here; copying a lossy subset (only
|
|
377
|
+
the winner, or files collapsed into a single directory link) is the failure.
|
|
334
378
|
|
|
335
379
|
```markdown
|
|
336
380
|
## {Artifact name}
|
|
@@ -35,14 +35,15 @@ where present, they are non-trivial — there is no fixed list. Model-training f
|
|
|
35
35
|
|
|
36
36
|
### logic/claims.md
|
|
37
37
|
- Has `## C\d+` blocks (at least one claim)
|
|
38
|
-
- Contains `**Statement**`
|
|
39
|
-
- Contains `**
|
|
38
|
+
- Contains `**Statement**` (the mechanism/takeaway a result reveals — subject is a mechanism/relationship, never a named recipe; no run numbers)
|
|
39
|
+
- Contains `**Conditions**` (non-trivial: the regime + the untested boundary)
|
|
40
|
+
- Contains `**Sources**`; every load-bearing number in a claim has a `Sources` entry carrying
|
|
40
41
|
a verbatim «quote» plus an `[input]`/`[result]` tag — no bare-path entries, no memory-filled numbers
|
|
41
42
|
- Contains `**Status**`
|
|
42
|
-
- Contains `**Falsification criteria**`
|
|
43
|
+
- Contains `**Falsification criteria**` (a substantive observation — about the system, or about the benchmark's behavior for a methodological claim — not a tautology or a re-run of a metric gate)
|
|
44
|
+
- `Statement` is the mechanism/takeaway a result reveals, not a record: a single instance may state the mechanism it reveals, but must not be extrapolated into a universal law beyond its regime, nor assert a distinction the design cannot disentangle — those limits live in `Conditions`
|
|
43
45
|
- Contains `**Proof**`
|
|
44
46
|
- Contains `**Evidence basis**`
|
|
45
|
-
- Contains `**Interpretation**`
|
|
46
47
|
|
|
47
48
|
### logic/problem.md
|
|
48
49
|
- Has `### O\d+` blocks (observations)
|
|
@@ -52,6 +53,8 @@ where present, they are non-trivial — there is no fixed list. Model-training f
|
|
|
52
53
|
### logic/experiments.md
|
|
53
54
|
- Has `## E\d+` blocks (at least 3)
|
|
54
55
|
- Contains `**Verifies**`
|
|
56
|
+
- Contains `**Evidence**` (link to where the run's results are filed, or "pending")
|
|
57
|
+
- Contains `**Run**` (what produced the run — a `src/execution/` file or a link/ref into the source repo/DB; failed/ablated runs are linked too, not omitted)
|
|
55
58
|
- Contains `**Setup**`
|
|
56
59
|
- Contains `**Procedure**`
|
|
57
60
|
- Contains `**Expected outcome**` or `**Expected results**`
|
|
@@ -89,9 +92,10 @@ fewer passes with fewer; what fails is fabricated filler.
|
|
|
89
92
|
- `src/execution/`: ≥1 `.py` file only when the work has implementable content (repo code / paper pseudocode / named interface). NOT mandatory otherwise; omitting it (with a note in `environment.md`) beats fabricating one.
|
|
90
93
|
- `evidence/tables/`, `evidence/figures/`, or `evidence/proofs/`: contains the filed evidence (see §11)
|
|
91
94
|
|
|
92
|
-
### Implementation layer (`src/`) —
|
|
93
|
-
- Concrete artifacts
|
|
94
|
-
- **
|
|
95
|
+
### Implementation layer (`src/`) — indexed when external, captured when it'd be lost
|
|
96
|
+
- Concrete artifacts are represented losslessly: prompts/templates verbatim in `src/prompts/`, config values in `src/configs/`, and — when the work's code/runs **persist in a linkable external store** — a **comprehensive pointer index** in `src/artifacts.md` linking every artifact. A lone `environment.md` is wrong when such artifacts exist.
|
|
97
|
+
- **Comprehensiveness** (external repo/run-database input): `src/artifacts.md` links **every** run and source file (per-run logs included — a `runs.jsonl` counts), nothing aggregated into a bare directory link or a "~N others" summary. FAIL on a lossy subset (only the winning run; real artifacts collapsed into a vague bucket).
|
|
98
|
+
- **Capture only when it'd be lost**: transcribe source into `src/execution/` (native form, `# Grounding: transcribed`, cite path) only when it exists solely inside the paper or its source is not externally persisted. Pointer-only is correct when the source persists; it FAILS only when the pointer would dangle (no persisted source).
|
|
95
99
|
- Conversely, a prose-only method (no code, no prompt, no config values) is NOT re-encoded as a `.py` stub or pseudo-code — it lives in `logic/solution/`; a lone `environment.md` is correct here. FAIL on a `.py` stub manufactured from prose (it just duplicates the cognitive layer).
|
|
96
100
|
|
|
97
101
|
### Code grounding (each `src/execution/*.py`, when present)
|
|
@@ -150,11 +154,18 @@ For each file in `evidence/figures/*.md` specifically:
|
|
|
150
154
|
### Claim Proof → Experiment Resolution
|
|
151
155
|
- Every `E\d+` in a claim's `**Proof**: [...]` must exist in experiments.md
|
|
152
156
|
- Proof-linked experiments should have evidence files whose labels and row contents actually match the compared systems or measurements
|
|
153
|
-
- Claim
|
|
157
|
+
- Claim `Statement` is a generalized mechanism/relationship auditable against `Evidence basis`; its reach is bounded by `Conditions` and its run numbers live in the evidence layer (not pasted into the Statement)
|
|
154
158
|
|
|
155
159
|
### Experiment Verifies → Claim Resolution
|
|
156
160
|
- Every `C\d+` in an experiment's `**Verifies**` must exist in claims.md
|
|
157
161
|
|
|
162
|
+
### Experiment Evidence / Run → Resolution
|
|
163
|
+
- Every `evidence/…` path in an experiment's `**Evidence**` is a filed evidence file (or "pending")
|
|
164
|
+
- Every experiment carries a `**Run**` ref — an entry in the comprehensive `src/artifacts.md` index (or, in the capture-fallback case, a `src/execution/` file) that links the source location; failed/ablated runs are linked there, not dropped
|
|
165
|
+
|
|
166
|
+
### Claim Dependencies → Claim Resolution
|
|
167
|
+
- Every `C\d+` in a claim's `**Dependencies**` must exist in claims.md (an unresolved ID FAILS)
|
|
168
|
+
|
|
158
169
|
### Heuristic Code Ref → File Resolution (only when heuristics.md + src/execution/ are both present)
|
|
159
170
|
- Every `src/...` path in `**Code ref**: [...]` must be an existing file
|
|
160
171
|
|
|
@@ -174,6 +185,19 @@ For each file in `evidence/figures/*.md` specifically:
|
|
|
174
185
|
- No fact ABOUT a repo artifact (line count, path, internal structure) is transcribed from the paper without checking the real file — when paper and repo disagree, the discrepancy is flagged, not silently resolved to the paper's number
|
|
175
186
|
- Spot-check trace `source_refs` and evidence `**Source**` labels: the cited section/table/appendix actually contains the claimed content
|
|
176
187
|
- A statistic carries its scope/denominator (N, population) in its `Source` — subset figures (e.g. "5 papers / 3,050 reqs") are not juxtaposed with full-corpus figures as if same-denominator
|
|
188
|
+
- **Claim Statements are takeaways** (exhaustive, not spot-checked — symmetric to the number-sources
|
|
189
|
+
pass): each `## C\d+` Statement FAILS if its subject is a named recipe/config/run, or if it
|
|
190
|
+
contains a run number, n-count, score, step/bin count, or p-value. The mechanism a result reveals
|
|
191
|
+
must be the subject; the numbers must live in `Evidence basis`/`Proof`
|
|
192
|
+
- **Attribution is not insight** (exhaustive, applied to each Statement that already passed the
|
|
193
|
+
takeaways gate above): a Statement also FAILS if it only identifies *which* named components of
|
|
194
|
+
this one system rank highest/lowest (load-bearing / dominant / decorative / inert / "largest
|
|
195
|
+
contributor") without stating what that ranking *reveals* — a relationship or mechanism a reader
|
|
196
|
+
could carry to a different system. **Operational tell: delete this system's component names from
|
|
197
|
+
the Statement; if no transferable relationship survives, it is attribution, not a mechanism — it
|
|
198
|
+
FAILS.** The fix is to state the generalization the ranking licenses; the named components and
|
|
199
|
+
their deltas stay in `Evidence basis`. This is the most common way a numerically-clean Statement
|
|
200
|
+
still fails the insight bar.
|
|
177
201
|
- **Claim/heuristic number sources** (exhaustive, not spot-checked): each `**Sources**` entry's cited
|
|
178
202
|
`file:line` (or trace `node:field`) exists, the verbatim «quote» is actually present there, and the
|
|
179
203
|
number in the `Statement`/`Rationale` matches the value inside that quote; `[input]` entries cite
|
|
@@ -15,7 +15,7 @@ argument-hint: "[optional: hint about what happened this turn]"
|
|
|
15
15
|
allowed-tools: Read, Write, Edit, Glob, Grep
|
|
16
16
|
metadata:
|
|
17
17
|
author: ara-commons
|
|
18
|
-
version: "2.
|
|
18
|
+
version: "2.4.0"
|
|
19
19
|
tags: [research, process-recording, provenance, progressive-crystallization, knowledge-management]
|
|
20
20
|
---
|
|
21
21
|
|
|
@@ -184,10 +184,17 @@ entries — staged observations belong to Stage 3. (History lives in the trace;
|
|
|
184
184
|
1. **Status updates** — flip a claim's `Status` field when evidence warrants.
|
|
185
185
|
2. **Content revisions** — rewrite a `Statement`, `Rationale`, or definition when new
|
|
186
186
|
evidence narrows scope, terminology changed, or wording no longer matches what's
|
|
187
|
-
actually supported.
|
|
187
|
+
actually supported. Keep `Statement` a generalized mechanism/relationship and sharpen
|
|
188
|
+
`Conditions` as the regime becomes clearer; new run numbers update `Proof`/`evidence`,
|
|
189
|
+
never the Statement. A rewrite re-grounds every number it now contains (Number grounding);
|
|
188
190
|
any changed value gets its own fresh `Sources` «quote», never a carried-over one.
|
|
189
191
|
3. **Structural changes** — split a claim into two, merge duplicates, repair
|
|
190
|
-
dependencies, rename ids when concepts are renamed.
|
|
192
|
+
dependencies, rename ids when concepts are renamed. Also **generalize**: when several
|
|
193
|
+
crystallized claims are together evidence for a more general relationship none states
|
|
194
|
+
alone, author a new claim whose `Dependencies` are those narrower claims and whose
|
|
195
|
+
`Proof` spans their evidence — keep the narrower claims in place; the new claim sits
|
|
196
|
+
above them, not instead of them (only when a signal this turn makes the relationship
|
|
197
|
+
evident — never a routine sweep).
|
|
191
198
|
4. **Consistency pass** — scan for broken cross-references (claim cites C05 which no
|
|
192
199
|
longer exists), terminology mismatch with `concepts.md`, dependency loops.
|
|
193
200
|
|
|
@@ -257,6 +264,8 @@ When a signal fires for entry `E` (claim, heuristic, or concept):
|
|
|
257
264
|
new id for the spin-off, update all cross-references.
|
|
258
265
|
- **Merge**: keep the lower id, mark the higher id as `withdrawn` with
|
|
259
266
|
`Merged into: C{XX}`, redirect cross-references.
|
|
267
|
+
- **Generalize**: allocate a new id for the more general claim, set its `Dependencies`
|
|
268
|
+
to the narrower claims, and leave those claims in place (they remain its grounding).
|
|
260
269
|
6. **Record full before/after in today's session record** under `logic_revisions:`
|
|
261
270
|
(see schema below). This is the ONLY place the prior wording is preserved — the
|
|
262
271
|
logic file does not keep it.
|
|
@@ -361,18 +370,34 @@ tree:
|
|
|
361
370
|
### Claim (`logic/claims.md`) — crystallized only
|
|
362
371
|
|
|
363
372
|
```markdown
|
|
364
|
-
## C{XX}: {title}
|
|
365
|
-
- **Statement**: {
|
|
366
|
-
- **
|
|
373
|
+
## C{XX}: {generalized title — the takeaway, not a recipe name}
|
|
374
|
+
- **Statement**: {the generalized, mechanistic conclusion; subject = a mechanism/relationship, never a named recipe; carries NO run numbers}
|
|
375
|
+
- **Conditions**: {under what conditions it holds; the regime; the known untested boundary}
|
|
376
|
+
- **Sources**: [{one entry per load-bearing number in the claim (now in `Conditions`/`Proof`): `<value> ← <file:line | trace-node:field> «verbatim line copied from source» [input|result]`, or `<value> ← [pending: reason]`}] # see "Number grounding"; a bare path with no «quote» is invalid
|
|
367
377
|
- **Status**: hypothesis | untested | testing | supported | weakened | refuted | withdrawn
|
|
368
378
|
- **Provenance**: user | ai-suggested | user-revised
|
|
369
|
-
- **Falsification
|
|
370
|
-
- **Proof**: [{evidence refs or "pending"}]
|
|
379
|
+
- **Falsification**: {a concrete observation that would disprove it — for a mechanism claim, about the system/world; for a methodological/regime claim, about the benchmark's behavior. NOT a tautology or a re-run of the same gate ("if the recipe fails the gate")}
|
|
380
|
+
- **Proof**: [{evidence refs (→ evidence/) or "pending"; run numbers/IDs/scores live HERE, not in Statement}]
|
|
371
381
|
- **Dependencies**: [C{YY}, ...]
|
|
372
382
|
- **Tags**: {comma-separated}
|
|
373
383
|
- **Last revised**: YYYY-MM-DD (turn-id) # pointer back to the trace; absent until first revision
|
|
374
384
|
```
|
|
375
385
|
|
|
386
|
+
**The Statement is the generalized conclusion the evidence supports — a mechanism or relationship,
|
|
387
|
+
not a restatement of run numbers.** What keeps it falsifiable and honest is `Conditions` (the regime
|
|
388
|
+
it holds in + the untested boundary) plus a `Falsification`, not a narrowed sentence. Numbers (run
|
|
389
|
+
IDs, n, scores, step counts) belong in `Proof` → `evidence/` (grounded per Number grounding), never
|
|
390
|
+
in `Statement`. `Conditions` is mandatory: a generalized Statement with no Conditions is an unbounded
|
|
391
|
+
slogan.
|
|
392
|
+
|
|
393
|
+
**Calibrate the Statement to what the evidence actually separates.** Do not assert a distinction the
|
|
394
|
+
design cannot disentangle (confounded factors — e.g. matrix "shape" vs "role" when they co-vary), or
|
|
395
|
+
a law from a single instance. When that's the case, hedge in the Statement itself — name the
|
|
396
|
+
unseparated factors together, or say "shown once here" — rather than only burying it in `Conditions`.
|
|
397
|
+
`Conditions` bounds *where* the claim applies; it is not a license for the Statement's verb to
|
|
398
|
+
over-reach. The Statement/Conditions may be sharpened on a later turn (Stage 4 content revision) as
|
|
399
|
+
the mechanism becomes clearer — no new closure signal is needed.
|
|
400
|
+
|
|
376
401
|
Current-state snapshot only — no prior statements, no `From staging`/`Crystallized via`
|
|
377
402
|
notes. Crystallization and every edit are recorded in the trace (`trace/sessions/…` under
|
|
378
403
|
`logic_revisions:` with before/after; source observation stays in `staging/`; reasoning in
|
|
@@ -72,7 +72,7 @@ What KIND of event is this?
|
|
|
72
72
|
→ ai-action [session record only]
|
|
73
73
|
|
|
74
74
|
Interpretation (something asserted to be true / general)?
|
|
75
|
-
|
|
75
|
+
Generalizable falsifiable assertion (a mechanism/relationship, bounded by conditions)?
|
|
76
76
|
→ STAGE as potential_type: claim
|
|
77
77
|
Implementation rule with rationale?
|
|
78
78
|
→ STAGE as potential_type: heuristic
|
|
@@ -148,6 +148,7 @@ For each claim's Falsification criteria field:
|
|
|
148
148
|
|
|
149
149
|
- **Over-claiming**: Does any Statement use universal scope markers ("all models", "any dataset", "state-of-the-art across all") while cited experiments cover only specific, narrow conditions? The gap must be substantial.
|
|
150
150
|
- **Under-claiming**: Are there important experimental results present in evidence/ that are not captured by any claim? (Evidence without a corresponding claim.)
|
|
151
|
+
- **Attribution vs mechanism**: Does any Statement merely name *which* components of this one system rank highest/lowest (load-bearing, dominant, decorative, inert) without stating what that ranking *reveals*? Apply the name-deletion test — strike the system's component names; if no transferable relationship survives, the Statement is attribution, not insight. Flag as `major` (the claim is a league table of this system, not a reusable finding); suggest the generalization the ranking licenses.
|
|
151
152
|
- **Assumption explicitness**: Are key assumptions stated in problem.md (Assumptions section) or constraints.md? Are there unstated assumptions implied by the experimental design?
|
|
152
153
|
- **Generalization boundaries**: Does the artifact clearly state what the claims do NOT apply to? Check constraints.md and limitations in the exploration tree.
|
|
153
154
|
- **Qualifier consistency**: When claims use hedging ("tends to", "in most cases"), is this consistent with the evidence strength?
|
|
@@ -65,6 +65,7 @@ resolution, field presence, YAML parsing) is handled entirely by Level 1.
|
|
|
65
65
|
|-------|---------------|-----------------|
|
|
66
66
|
| Over-claiming | Statement uses universal scope while evidence covers narrow conditions | critical if extreme, major if moderate |
|
|
67
67
|
| Under-claiming | Evidence files or experiment results not captured by any claim | minor |
|
|
68
|
+
| Attribution vs mechanism | Statement names which components rank where (name-deletion test leaves nothing transferable) instead of what the ranking reveals | major |
|
|
68
69
|
| Assumption explicitness | Key assumptions stated in problem.md or constraints.md | major if unstated assumptions affect validity |
|
|
69
70
|
| Generalization boundaries | Artifact states what claims do NOT apply to | minor |
|
|
70
71
|
| Qualifier consistency | Hedging language matches evidence strength | minor |
|