calkit-python 0.50.1__py3-none-any.whl → 0.50.2__py3-none-any.whl

calkit/agent_skills/build-paper-pipeline/SKILL.md

@@ -95,7 +95,7 @@ building a pipeline from scratch.
    number in it that is not traceable to a pipeline output. A finding there
    is a hole you closed by typing rather than by building, so go back to
    step 3 for it. Show the user the rendered sentences
-   (`calkit list questions`) and the built document. Never commit a derived
+   (`calkit show question <n>`) and the built document. Never commit a derived
    output yourself, and never edit a results file by hand.
 
 ## Data a person has to collect

calkit/agent_skills/check-questions/SKILL.md

@@ -12,7 +12,21 @@ An answer in `calkit.yaml` is a claim about the evidence as it was when the
 answer was last edited. The pipeline keeps the evidence current; nothing
 keeps the prose current. Calkit's deterministic check catches evidence
 changing underneath an answer. This skill covers what a check cannot:
-whether the sentence still follows from the evidence.
+whether the sentence still follows from the evidence, and whether the
+question's `approach` describes what its pipeline stages actually do.
+
+People sign off on a question in two ways, and the two are easy to mix up:
+
+- **Verification**, under `verifications`: the stages behind the evidence
+  implement the `approach` correctly, i.e., the answer was worked out
+  right. It is about code against the approach sentence, not about whether
+  the answer is true.
+- **Validation**, under `validations`: the approach suits the question and
+  the evidence justifies the answer, i.e., the right thing was worked out.
+  It is about whether the claim is logical and defensible, not about the
+  code.
+
+A sign-off is theirs, not yours: you read and report, and they sign off.
 
 ## Division of labor
 
@@ -26,6 +40,15 @@ Deterministic, done by `calkit check questions` — never re-derive by hand:
 - every publication `label` still exists in the LaTeX source;
 - no evidence has changed (Git history for Git-tracked outputs, `dvc.lock`
   for DVC-tracked ones) since the commit that last edited the question;
+- no verification is stale, i.e., neither the approach nor any of the
+  question's stages, their definitions or what `dvc.lock` lists as their
+  dependencies, has changed since the commit that added it;
+- no validation is stale, i.e., neither the question, its hypothesis,
+  approach, or answer, nor its evidence has changed since the commit that
+  added it;
+- which pipeline stages are behind each question's evidence, upstream
+  first, listed by `calkit check questions --verbose` and
+  `calkit show question <n>`;
 - each `value` entry reads a file a pipeline stage produces, or one
   declared with `imported_from`; anything else is an error, since a number
   nothing computes is a magic number;
@@ -37,7 +60,9 @@ Judgment, done here — the check reads paths and hashes, and cannot read a
 sentence:
 
 - **does the evidence actually support the answer**, which is the reason
-  this skill exists;
+  this skill exists and what a validation vouches for;
+- **does the `approach` describe what the stages implement**, which is what
+  a verification vouches for;
 - does the answer still follow from the evidence, given what changed;
 - are numbers retyped into the prose that should be `{name}` placeholders;
 - does a claim resting on a threshold use a conditional answer, with a
@@ -45,7 +70,7 @@ sentence:
 - is the answer concise, and does it point at the publication section that
   carries the argument rather than repeating it.
 
-## Does the evidence support the answer?
+## Does the evidence support the answer? (validation)
 
 A question can pass every deterministic check and still be wrong: the
 paths resolve, the numbers render, nothing has changed since — and the
@@ -85,6 +110,29 @@ evidence does not support the sentence, propose the sentence it does
 support and let the user decide; never edit the answer to match the
 evidence without showing them both.
 
+## Does the approach match the implementation? (verification)
+
+`approach` is one sentence on how the question is answered. The stages it
+summarizes are derived from the evidence, so they can't drift, but the
+sentence can. For each question with an `approach`:
+
+1. Get its stages from `calkit check questions --verbose --json` (the
+   `stages` list) and read each one's definition in `calkit.yaml`, then the
+   script, notebook, or command it runs.
+2. Check that the method the sentence names is the one the code runs,
+   e.g., a "discrete-event simulation" that is really a closed-form
+   calculation, or "a regression on all sites" that filters half of them.
+3. Check that anything the sentence states as fixed, e.g., a sample size,
+   a model, a dataset, a parameter, matches the code and its inputs.
+4. Check that no stage in the chain does something the sentence would lead
+   a reader to rule out, e.g., an exclusion step or a different estimator.
+
+Report mismatches with the line of code and the words they contradict, and
+propose a corrected sentence. A question with an answer and no `approach`
+is worth one: draft it from the stages and let the user decide. Whether
+the approach is the right one for the question belongs to validation, not
+here.
+
 ## Procedure
 
 1. Run `calkit check questions --json` and read the report. Questions with
@@ -94,41 +142,49 @@ evidence without showing them both.
    review or the answer is one the paper leans on.
 2. For each **stale** question, the report names the evidence that changed
    and the commit the answer dates from. Read the rendered answer
-   (`calkit list questions`) against the current evidence, and if it helps,
+   (`calkit show question <n>`) against the current evidence, and if it helps,
    the old evidence (`git show <commit>:<path>`). Decide:
-   - **The claim still holds**: say so, and edit the question to record
-     that you read it again. Any edit counts: the check anchors on the
-     commit where the question last changed. Adding or refining `notes` is
-     usually the honest edit to make, since it is where you say what you
-     checked.
+   - **The claim still holds**: say so, with what you checked. Recording
+     that is the user's sign-off, not yours: suggest they run
+     `calkit update question <n> --validate "<what they checked>"`, or
+     `--verify` for the approach against the code, adding
+     `--with-ai "<your model>"` if your reading informed theirs. Never add
+     or refresh a sign-off yourself unless they ask you to, and then only
+     with their Git identity and that disclosure.
    - **The claim no longer holds**: draft a corrected answer, show the user
      the old and new text side by side with the values that changed, and
      only after they agree, edit `calkit.yaml`.
    - **The evidence changed because a stage is not reproducible** (same
      inputs, different output): that is a pipeline defect, not an answer
      defect. Report it as such and do not rewrite the answer to match noise.
-3. For each **unattributed** evidence entry, ask where the file came
+3. For each **stale verification or validation**, the report names what
+   changed since the person signed off: for a verification, the approach,
+   a stage, or a dependency it ran; for a validation, evidence, with old
+   and new values, or the question's own text. Redo the matching reading
+   above against the version they signed off on
+   (`git show <commit>:calkit.yaml`), and tell the user who needs to look
+   again and at what.
+4. For each **unattributed** evidence entry, ask where the file came
    from. If a stage should produce it, that is a pipeline gap worth
    reporting. If it was imported or made by hand, declare it under
    `figures`, `datasets`, or `publications` with `imported_from` or
    `created_by` so the project says so. A `value` entry with no stage is
    reported as an error rather than `unattributed`: give it a stage.
-4. For each **error**, fix the reference: a missing path means the pipeline
+5. For each **error**, fix the reference: a missing path means the pipeline
    has not been run or pulled; a bad key or placeholder means a results
    file was restructured; a missing label means the publication was
    reorganized. Do not delete evidence to make an error go away.
-5. For every question you touched, move any number in the prose that the
+6. For every question you touched, move any number in the prose that the
    evidence carries into a `{name:...}` placeholder on a `value` entry, so
    it is read from the results file rather than retyped.
 
-6. Run the evidence-supports-the-answer review above on every question in
-   scope, `ok` ones included. A question the CLI passed is where an
+7. Run the validation and verification readings above on every question
+   in scope, `ok` ones included. A question the CLI passed is where an
    unsupported claim survives, because nothing else looks at it.
 
-Never edit a stale question just to clear the check. It is the act of
-reading the answer against the evidence that the edit is meant to record,
-and an edit made without doing that is a false statement about the
-project, not a tidy-up.
+Never edit a question or add a sign-off just to clear the check. A
+sign-off records a person having done the reading, and one added without
+that is a false statement about the project, not a tidy-up.
 
 ## Writing answers
 
@@ -178,7 +234,7 @@ project, not a tidy-up.
 
 ## Reporting
 
-Relay a short table: question index, status, what changed, what you did or
-propose. Quote values, not adjectives. If the pipeline is stale
+Relay a short table: question index, status, verification and validation
+status, what changed, what you did or propose. Quote values, not adjectives. If the pipeline is stale
 (`calkit status`), say so first, since the evidence may be about to change
 again.

calkit/agent_skills/conventions/SKILL.md

@@ -197,23 +197,24 @@ DVC handles:
 
 ## Key CLI commands
 
-| Command                         | What it does                                         |
-| ------------------------------- | ---------------------------------------------------- |
-| `calkit run`                    | Run the pipeline (skips unchanged stages)            |
-| `calkit run --force`            | Force re-run all stages                              |
-| `calkit status`                 | Show which stages are stale                          |
-| `calkit xr <file>`              | Auto-detect stage type, env, I/O, add to pipeline    |
-| `calkit xenv -n <env> -- <cmd>` | Run a command in a named environment                 |
-| `calkit push`                   | Push Git commits and DVC-tracked files to remotes    |
-| `calkit pull`                   | Pull latest code and data                            |
-| `calkit save`                   | Auto-add, commit, and push (Git + DVC)               |
-| `calkit commit -m "msg"`        | Commit all tracked changes (Git + DVC)               |
-| `calkit add <file>`             | Add a file to version control                        |
-| `calkit check env --name <env>` | Verify an environment matches its spec               |
-| `calkit new`                    | Create new project objects (notebook, dataset, etc.) |
-| `calkit new question "<text>"`  | Record a research question                           |
-| `calkit list questions`         | Show questions with answers rendered from evidence   |
-| `calkit check questions`        | Check answers against current evidence               |
+| Command                         | What it does                                           |
+| ------------------------------- | ------------------------------------------------------ |
+| `calkit run`                    | Run the pipeline (skips unchanged stages)              |
+| `calkit run --force`            | Force re-run all stages                                |
+| `calkit status`                 | Show which stages are stale                            |
+| `calkit xr <file>`              | Auto-detect stage type, env, I/O, add to pipeline      |
+| `calkit xenv -n <env> -- <cmd>` | Run a command in a named environment                   |
+| `calkit push`                   | Push Git commits and DVC-tracked files to remotes      |
+| `calkit pull`                   | Pull latest code and data                              |
+| `calkit save`                   | Auto-add, commit, and push (Git + DVC)                 |
+| `calkit commit -m "msg"`        | Commit all tracked changes (Git + DVC)                 |
+| `calkit add <file>`             | Add a file to version control                          |
+| `calkit check env --name <env>` | Verify an environment matches its spec                 |
+| `calkit new`                    | Create new project objects (notebook, dataset, etc.)   |
+| `calkit new question "<text>"`  | Record a research question                             |
+| `calkit list questions`         | List questions with what each has and lacks            |
+| `calkit show question <n>`      | Show a question with its answer rendered from evidence |
+| `calkit check questions`        | Check answers against current evidence                 |
 
 ## `calkit xr`: The fastest path to a reproducible stage
 

calkit/cli/list.py

@@ -63,34 +63,37 @@ def _list_artifacts(
     kind: Literal["figures", "datasets", "results", "presentations"],
     json_output: bool,
     declared_only: bool,
+    status: bool = False,
 ):
     """List figures or datasets, optionally including auto-detected ones.
 
     By default, artifacts declared in ``calkit.yaml`` are merged with any
     auto-detected from the project's files; ``--declared-only`` returns just the
     declared ones. Each entry carries a ``detected`` flag so callers can tell
-    declared and auto-detected artifacts apart.
+    declared and auto-detected artifacts apart, and with ``status``, the
+    stage that writes it and whether that stage is up to date.
     """
-    ck_info = calkit.load_calkit_info()
-    declared = [o for o in ck_info.get(kind) or [] if isinstance(o, dict)]
-    declared_paths = {o.get("path") for o in declared}
-    detected: list[dict] = []
-    if not declared_only:
-        found = calkit.detect.detect_project_artifacts(ck_info=ck_info)
-        for path in found.get(kind, []):
-            if path not in declared_paths:
-                detected.append({"path": path})
+    import calkit.pipeline
+    import calkit.results
+
+    # Which stage writes each one is read from calkit.yaml and costs
+    # nothing; whether it's current needs DVC, so it's asked for
+    try:
+        result = calkit.pipeline.list_artifacts(
+            kind,
+            declared_only=declared_only,
+            status=status,
+            pending_extensions=(
+                calkit.results.EXTENSIONS if kind == "results" else None
+            ),
+        )
+    except RuntimeError as e:
+        raise_error(f"Failed to determine status: {e}")
     if json_output:
-        result = [
-            {**o, "detected": False} for o in declared if isinstance(o, dict)
-        ]
-        result += [{**o, "detected": True} for o in detected]
         echo_json(result)
         return
-    for obj in declared:
+    for obj in result:
         _echo_object(obj)
-    for obj in detected:
-        _echo_object({**obj, "detected": True})
 
 
 @list_app.command(name="notebooks|nb")
@@ -158,9 +161,21 @@ def list_results(
             ),
         ),
     ] = False,
+    status: Annotated[
+        bool,
+        typer.Option(
+            "--status",
+            "-s",
+            help="Also say whether each result's stage is up to date.",
+        ),
+    ] = False,
 ):
-    """List results in the project."""
-    _list_artifacts("results", json_output, declared_only)
+    """List results in the project.
+
+    Shown with the stage that writes each one, which is what
+    'calkit run' would use to bring it up to date.
+    """
+    _list_artifacts("results", json_output, declared_only, status)
 
 
 @list_app.command(name="presentations|pres")
@@ -183,136 +198,107 @@ def list_presentations(
     _list_artifacts("presentations", json_output, declared_only)
 
 
-def _echo_question(n: int, question: str | dict, project: str = "") -> None:
-    """Print a single question in the human-readable YAML-ish listing format.
+@list_app.command(name="tables|tbls")
+def list_tables(
+    columns: Annotated[
+        bool,
+        typer.Option(
+            "--columns",
+            help="Also read each table's column names from its header.",
+        ),
+    ] = False,
+    status: Annotated[
+        bool,
+        typer.Option(
+            "--status",
+            "-s",
+            help="Also say whether each table's stage is up to date.",
+        ),
+    ] = False,
+    json_output: Annotated[
+        bool, typer.Option("--json", help="Output result as JSON.")
+    ] = False,
+    declared_only: Annotated[
+        bool,
+        typer.Option(
+            "--declared-only",
+            help=(
+                "Only list tables declared in calkit.yaml; "
+                "skip auto-detection."
+            ),
+        ),
+    ] = False,
+) -> None:
+    """List tables in the project.
 
-    String questions are shown as a numbered line. Rich (dict) questions are
-    shown with their text on the first line followed by hypothesis, answer,
-    evidence, and any other fields in YAML-like indented form. A
-    subproject's are numbered within it and prefixed with its path, as
-    ``calkit check questions`` names them.
+    Tabular files are detected wherever they're kept, so a table that is
+    also a dataset or a result is listed here too. Each is shown with the
+    stage that writes it, if any, which is the one 'calkit run' would use
+    to bring it up to date.
     """
-    where = f"{project}: " if project else ""
-    if isinstance(question, str):
-        typer.echo(f"{where}{n}. {question}")
+    import calkit.tables
+
+    try:
+        tables = calkit.tables.list_tables(
+            declared_only=declared_only, status=status
+        )
+    except RuntimeError as e:
+        raise_error(f"Failed to determine table status: {e}")
+    if columns:
+        for table in tables:
+            try:
+                table["columns"] = calkit.tables.read_columns(table["path"])
+            except Exception as e:
+                # A table that can't be read is still a table; say why
+                # here rather than failing the listing
+                table["columns_error"] = str(e)
+    if json_output:
+        echo_json(tables)
         return
-    # Copy so popping 'question' doesn't mutate the caller's dict.
-    question = dict(question)
-    text = question.pop("question", "")
-    typer.echo(f"{where}{n}. question: {text}")
-    # Rendering fills every field, so leave out the ones that aren't set
-    for k, v in question.items():
-        if v is None:
-            continue
-        if isinstance(v, dict):
-            typer.echo(f"    {k}:")
-            for k1, v1 in v.items():
-                typer.echo(f"      {k1}: {v1}")
-        elif isinstance(v, list):
-            typer.echo(f"    {k}:")
-            for item in v:
-                if isinstance(item, dict):
-                    item = {
-                        k1: v1 for k1, v1 in item.items() if v1 is not None
-                    }
-                    for n1, (k1, v1) in enumerate(item.items()):
-                        prefix = "      - " if n1 == 0 else "        "
-                        if isinstance(v1, dict):
-                            typer.echo(f"{prefix}{k1}:")
-                            for k2, v2 in v1.items():
-                                typer.echo(f"          {k2}: {v2}")
-                        else:
-                            typer.echo(f"{prefix}{k1}: {v1}")
-                else:
-                    typer.echo(f"        - {item}")
-        else:
-            typer.echo(f"    {k}: {v}")
+    for table in tables:
+        _echo_object(table)
 
 
-@list_app.command(name="questions")
+@list_app.command(name="questions|q")
 def list_questions(
     json_output: Annotated[
         bool, typer.Option("--json", help="Output result as JSON.")
     ] = False,
-    raw: Annotated[
-        bool,
-        typer.Option(
-            "--raw",
-            help="Show the text as written, with its {name} placeholders, "
-            "instead of rendered from the evidence.",
-        ),
-    ] = False,
-):
+) -> None:
     """List the project's questions (1-indexed).
 
-    Placeholders in the text, such as ``{improvement:.1f}``, are filled from
-    the question's value evidence, so numbers shown are read from the
-    results files rather than retyped into ``calkit.yaml``. Subprojects'
-    questions are listed too, numbered within each, and in the JSON output
-    each has a ``project`` key with the subproject's path.
+    Each has a row of letters: H if it has a hypothesis, A if it's answered,
+    then V if it's verified and V if it's validated, colored as in the hub,
+    with a dash for each it doesn't have yet. Evidence isn't checked against
+    the pipeline here; 'calkit check questions' does that. Subprojects'
+    questions are listed too, numbered within each. Use 'calkit show
+    question' to see one in full.
     """
-    from pathlib import Path
-
-    from calkit.questions import (
-        TEMPLATED_FIELDS,
-        placeholders,
-        render_question,
-    )
+    from calkit.questions import check_questions, format_letters
 
-    def _texts(question: dict) -> list[str | dict]:
-        evidence = question.get("evidence") or []
-        return [question.get(f) or "" for f in TEMPLATED_FIELDS] + [
-            ev.get("explanation") or ""
-            for ev in evidence
-            if isinstance(ev, dict)
-        ]
-
-    # Each with its 1-based index in its own project and that project's
-    # path, empty for this one
-    listed: list[tuple[int, str | dict, str]] = []
-    for wdir, info in calkit.iter_projects():
-        project = "" if wdir == "." else Path(wdir).as_posix()
-        for n, q in enumerate(info.get("questions", []) or [], start=1):
-            if not raw:
-                q = render_question(q, info, wdir=wdir)
-            listed.append((n, q, project))
-    questions = [q for _, q, _ in listed]
-    if not raw:
-        # A placeholder left as written looks exactly like text somebody
-        # meant literally, so say when one could not be filled rather than
-        # let a fresh clone read as a project that types its braces. Any
-        # placeholder left standing counts, whether it names evidence that
-        # could not be read or names nothing at all: a brace meant to stay
-        # in the text is written '{{' and never reaches here. A conditional
-        # answer still in clauses is one whose conditions could not be read.
-        unfilled = any(
-            isinstance(text, dict) or placeholders(text)
-            for q in questions
-            if isinstance(q, dict)
-            for text in _texts(q)
-        )
-        if unfilled:
-            warn(
-                "Some placeholders or conditions could not be filled from "
-                "the evidence. "
-                "Run 'calkit check questions' to see why; 'calkit pull' if "
-                "the results files are not here yet.",
-                err=json_output,
-            )
+    # Sign-offs are checked for their letters, but evidence isn't compared
+    # with its history, which nothing here shows
+    status = check_questions(check_pipeline=False, check_changes=False)
     if json_output:
         echo_json(
             [
-                q
-                if not project
-                else {"project": project, "question": q}
-                if isinstance(q, str)
-                else {"project": project, **q}
-                for _, q, project in listed
+                {
+                    "project": q.project,
+                    "index": q.index,
+                    "question": q.question,
+                    "hypothesis": q.hypothesis,
+                    "answered": q.answered,
+                    "status": q.status,
+                    "verification_status": q.verification_status,
+                    "validation_status": q.validation_status,
+                }
+                for q in status.questions
             ]
         )
         return
-    for n, question, project in listed:
-        _echo_question(n, question, project)
+    for q in status.questions:
+        where = f"{q.project}: " if q.project else ""
+        typer.echo(f"{where}{q.index}. {format_letters(q)}  {q.question}")
 
 
 @list_app.command(name="comments")