calkit-python 0.50.0__py3-none-any.whl → 0.50.2__py3-none-any.whl

calkit/agent_skills/build-paper-pipeline/SKILL.md

@@ -95,7 +95,7 @@ building a pipeline from scratch.
    number in it that is not traceable to a pipeline output. A finding there
    is a hole you closed by typing rather than by building, so go back to
    step 3 for it. Show the user the rendered sentences
-   (`calkit list questions`) and the built document. Never commit a derived
+   (`calkit show question <n>`) and the built document. Never commit a derived
    output yourself, and never edit a results file by hand.
 
 ## Data a person has to collect

calkit/agent_skills/check-questions/SKILL.md

@@ -12,7 +12,21 @@ An answer in `calkit.yaml` is a claim about the evidence as it was when the
 answer was last edited. The pipeline keeps the evidence current; nothing
 keeps the prose current. Calkit's deterministic check catches evidence
 changing underneath an answer. This skill covers what a check cannot:
-whether the sentence still follows from the evidence.
+whether the sentence still follows from the evidence, and whether the
+question's `approach` describes what its pipeline stages actually do.
+
+People sign off on a question in two ways, and the two are easy to mix up:
+
+- **Verification**, under `verifications`: the stages behind the evidence
+  implement the `approach` correctly, i.e., the answer was worked out
+  right. It is about code against the approach sentence, not about whether
+  the answer is true.
+- **Validation**, under `validations`: the approach suits the question and
+  the evidence justifies the answer, i.e., the right thing was worked out.
+  It is about whether the claim is logical and defensible, not about the
+  code.
+
+A sign-off is theirs, not yours: you read and report, and they sign off.
 
 ## Division of labor
 
@@ -26,6 +40,15 @@ Deterministic, done by `calkit check questions` — never re-derive by hand:
 - every publication `label` still exists in the LaTeX source;
 - no evidence has changed (Git history for Git-tracked outputs, `dvc.lock`
   for DVC-tracked ones) since the commit that last edited the question;
+- no verification is stale, i.e., neither the approach nor any of the
+  question's stages, their definitions or what `dvc.lock` lists as their
+  dependencies, has changed since the commit that added it;
+- no validation is stale, i.e., neither the question, its hypothesis,
+  approach, or answer, nor its evidence has changed since the commit that
+  added it;
+- which pipeline stages are behind each question's evidence, upstream
+  first, listed by `calkit check questions --verbose` and
+  `calkit show question <n>`;
 - each `value` entry reads a file a pipeline stage produces, or one
   declared with `imported_from`; anything else is an error, since a number
   nothing computes is a magic number;
@@ -37,7 +60,9 @@ Judgment, done here — the check reads paths and hashes, and cannot read a
 sentence:
 
 - **does the evidence actually support the answer**, which is the reason
-  this skill exists;
+  this skill exists and what a validation vouches for;
+- **does the `approach` describe what the stages implement**, which is what
+  a verification vouches for;
 - does the answer still follow from the evidence, given what changed;
 - are numbers retyped into the prose that should be `{name}` placeholders;
 - does a claim resting on a threshold use a conditional answer, with a
@@ -45,7 +70,7 @@ sentence:
 - is the answer concise, and does it point at the publication section that
   carries the argument rather than repeating it.
 
-## Does the evidence support the answer?
+## Does the evidence support the answer? (validation)
 
 A question can pass every deterministic check and still be wrong: the
 paths resolve, the numbers render, nothing has changed since — and the
@@ -85,6 +110,29 @@ evidence does not support the sentence, propose the sentence it does
 support and let the user decide; never edit the answer to match the
 evidence without showing them both.
 
+## Does the approach match the implementation? (verification)
+
+`approach` is one sentence on how the question is answered. The stages it
+summarizes are derived from the evidence, so they can't drift, but the
+sentence can. For each question with an `approach`:
+
+1. Get its stages from `calkit check questions --verbose --json` (the
+   `stages` list) and read each one's definition in `calkit.yaml`, then the
+   script, notebook, or command it runs.
+2. Check that the method the sentence names is the one the code runs,
+   e.g., a "discrete-event simulation" that is really a closed-form
+   calculation, or "a regression on all sites" that filters half of them.
+3. Check that anything the sentence states as fixed, e.g., a sample size,
+   a model, a dataset, a parameter, matches the code and its inputs.
+4. Check that no stage in the chain does something the sentence would lead
+   a reader to rule out, e.g., an exclusion step or a different estimator.
+
+Report mismatches with the line of code and the words they contradict, and
+propose a corrected sentence. A question with an answer and no `approach`
+is worth one: draft it from the stages and let the user decide. Whether
+the approach is the right one for the question belongs to validation, not
+here.
+
 ## Procedure
 
 1. Run `calkit check questions --json` and read the report. Questions with
@@ -94,41 +142,49 @@ evidence without showing them both.
    review or the answer is one the paper leans on.
 2. For each **stale** question, the report names the evidence that changed
    and the commit the answer dates from. Read the rendered answer
-   (`calkit list questions`) against the current evidence, and if it helps,
+   (`calkit show question <n>`) against the current evidence, and if it helps,
    the old evidence (`git show <commit>:<path>`). Decide:
-   - **The claim still holds**: say so, and edit the question to record
-     that you read it again. Any edit counts: the check anchors on the
-     commit where the question last changed. Adding or refining `notes` is
-     usually the honest edit to make, since it is where you say what you
-     checked.
+   - **The claim still holds**: say so, with what you checked. Recording
+     that is the user's sign-off, not yours: suggest they run
+     `calkit update question <n> --validate "<what they checked>"`, or
+     `--verify` for the approach against the code, adding
+     `--with-ai "<your model>"` if your reading informed theirs. Never add
+     or refresh a sign-off yourself unless they ask you to, and then only
+     with their Git identity and that disclosure.
    - **The claim no longer holds**: draft a corrected answer, show the user
      the old and new text side by side with the values that changed, and
      only after they agree, edit `calkit.yaml`.
    - **The evidence changed because a stage is not reproducible** (same
      inputs, different output): that is a pipeline defect, not an answer
      defect. Report it as such and do not rewrite the answer to match noise.
-3. For each **unattributed** evidence entry, ask where the file came
+3. For each **stale verification or validation**, the report names what
+   changed since the person signed off: for a verification, the approach,
+   a stage, or a dependency it ran; for a validation, evidence, with old
+   and new values, or the question's own text. Redo the matching reading
+   above against the version they signed off on
+   (`git show <commit>:calkit.yaml`), and tell the user who needs to look
+   again and at what.
+4. For each **unattributed** evidence entry, ask where the file came
    from. If a stage should produce it, that is a pipeline gap worth
    reporting. If it was imported or made by hand, declare it under
    `figures`, `datasets`, or `publications` with `imported_from` or
    `created_by` so the project says so. A `value` entry with no stage is
    reported as an error rather than `unattributed`: give it a stage.
-4. For each **error**, fix the reference: a missing path means the pipeline
+5. For each **error**, fix the reference: a missing path means the pipeline
    has not been run or pulled; a bad key or placeholder means a results
    file was restructured; a missing label means the publication was
    reorganized. Do not delete evidence to make an error go away.
-5. For every question you touched, move any number in the prose that the
+6. For every question you touched, move any number in the prose that the
    evidence carries into a `{name:...}` placeholder on a `value` entry, so
    it is read from the results file rather than retyped.
 
-6. Run the evidence-supports-the-answer review above on every question in
-   scope, `ok` ones included. A question the CLI passed is where an
+7. Run the validation and verification readings above on every question
+   in scope, `ok` ones included. A question the CLI passed is where an
    unsupported claim survives, because nothing else looks at it.
 
-Never edit a stale question just to clear the check. It is the act of
-reading the answer against the evidence that the edit is meant to record,
-and an edit made without doing that is a false statement about the
-project, not a tidy-up.
+Never edit a question or add a sign-off just to clear the check. A
+sign-off records a person having done the reading, and one added without
+that is a false statement about the project, not a tidy-up.
 
 ## Writing answers
 
@@ -178,7 +234,7 @@ project, not a tidy-up.
 
 ## Reporting
 
-Relay a short table: question index, status, what changed, what you did or
-propose. Quote values, not adjectives. If the pipeline is stale
+Relay a short table: question index, status, verification and validation
+status, what changed, what you did or propose. Quote values, not adjectives. If the pipeline is stale
 (`calkit status`), say so first, since the evidence may be about to change
 again.

calkit/agent_skills/conventions/SKILL.md

@@ -197,23 +197,24 @@ DVC handles:
 
 ## Key CLI commands
 
-| Command                         | What it does                                         |
-| ------------------------------- | ---------------------------------------------------- |
-| `calkit run`                    | Run the pipeline (skips unchanged stages)            |
-| `calkit run --force`            | Force re-run all stages                              |
-| `calkit status`                 | Show which stages are stale                          |
-| `calkit xr <file>`              | Auto-detect stage type, env, I/O, add to pipeline    |
-| `calkit xenv -n <env> -- <cmd>` | Run a command in a named environment                 |
-| `calkit push`                   | Push Git commits and DVC-tracked files to remotes    |
-| `calkit pull`                   | Pull latest code and data                            |
-| `calkit save`                   | Auto-add, commit, and push (Git + DVC)               |
-| `calkit commit -m "msg"`        | Commit all tracked changes (Git + DVC)               |
-| `calkit add <file>`             | Add a file to version control                        |
-| `calkit check env --name <env>` | Verify an environment matches its spec               |
-| `calkit new`                    | Create new project objects (notebook, dataset, etc.) |
-| `calkit new question "<text>"`  | Record a research question                           |
-| `calkit list questions`         | Show questions with answers rendered from evidence   |
-| `calkit check questions`        | Check answers against current evidence               |
+| Command                         | What it does                                           |
+| ------------------------------- | ------------------------------------------------------ |
+| `calkit run`                    | Run the pipeline (skips unchanged stages)              |
+| `calkit run --force`            | Force re-run all stages                                |
+| `calkit status`                 | Show which stages are stale                            |
+| `calkit xr <file>`              | Auto-detect stage type, env, I/O, add to pipeline      |
+| `calkit xenv -n <env> -- <cmd>` | Run a command in a named environment                   |
+| `calkit push`                   | Push Git commits and DVC-tracked files to remotes      |
+| `calkit pull`                   | Pull latest code and data                              |
+| `calkit save`                   | Auto-add, commit, and push (Git + DVC)                 |
+| `calkit commit -m "msg"`        | Commit all tracked changes (Git + DVC)                 |
+| `calkit add <file>`             | Add a file to version control                          |
+| `calkit check env --name <env>` | Verify an environment matches its spec                 |
+| `calkit new`                    | Create new project objects (notebook, dataset, etc.)   |
+| `calkit new question "<text>"`  | Record a research question                             |
+| `calkit list questions`         | List questions with what each has and lacks            |
+| `calkit show question <n>`      | Show a question with its answer rendered from evidence |
+| `calkit check questions`        | Check answers against current evidence                 |
 
 ## `calkit xr`: The fastest path to a reproducible stage