calkit-python 0.47.5__tar.gz → 0.47.6__tar.gz

{calkit_python-0.47.5 → calkit_python-0.47.6}/AGENTS.md

@@ -8,6 +8,8 @@
 - The Chrome extension lives in `browser-ext`
 - The Calkit Assistant desktop app lives in `assistant`
 - GitHub Actions live in `actions`, e.g., `calkit/calkit/actions/run`
+- Docker images we publish live in `images`, e.g., `images/latex`, which
+  is released on its own `latex-image/vX.Y.Z` tag rather than with the CLI
 - Config Calkit installs into projects, e.g., the dev container, VS Code, and
   GitHub Actions configs, lives in `calkit/resources`; see the README there
   before editing, since some of those files are generated

{calkit_python-0.47.5 → calkit_python-0.47.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: calkit-python
-Version: 0.47.5
+Version: 0.47.6
 Summary: Reproducibility simplified.
 Project-URL: Homepage, https://calkit.org
 Project-URL: Issues, https://github.com/calkit/calkit/issues

{calkit_python-0.47.5 → calkit_python-0.47.6}/agent-plugin/skills/build-paper-pipeline/SKILL.md

@@ -84,8 +84,11 @@ building a pipeline from scratch.
    `calkit.yaml`: the `question` as the paper poses it, the `answer` as the
    paper states it with `{name}` placeholders where numbers go, and
    `evidence` naming the results file and key, the figure, and the
-   publication section that carries the argument. Run
-   `calkit check questions` and fix what it reports.
+   publication section that carries the argument. The results file must be
+   one a stage from step 3 writes, never one written by hand. Where the
+   claim hinges on a threshold, write a conditional answer; the
+   `check-questions` skill covers both. Run `calkit check questions` and
+   fix what it reports.
 
 6. **Run the pipeline, then check your own work.** Run `calkit run`, then
    `calkit check repro`, which reads the manuscript back and reports any

{calkit_python-0.47.5 → calkit_python-0.47.6}/agent-plugin/skills/check-questions/SKILL.md

@@ -21,12 +21,17 @@ Deterministic, done by `calkit check questions` — never re-derive by hand:
 - every evidence path exists;
 - every `value` key resolves in its file, and every `{name}` placeholder in
   the prose resolves and formats;
+- every clause of a conditional answer parses, names only evidence, and
+  renders, including the clauses the current values don't select;
 - every publication `label` still exists in the LaTeX source;
 - no evidence has changed (Git history for Git-tracked outputs, `dvc.lock`
   for DVC-tracked ones) since the commit that last edited the question;
-- each evidence path is produced by a pipeline stage, or declared with
-  `imported_from` or `created_by`. This one is advisory: it is reported as
-  `unattributed` and does not fail the check.
+- each `value` entry reads a file a pipeline stage produces, or one
+  declared with `imported_from`; anything else is an error, since a number
+  nothing computes is a magic number;
+- each other evidence path is produced by a pipeline stage, or declared
+  with `imported_from` or `created_by`. This one is advisory: it is
+  reported as `unattributed` and does not fail the check.
 
 Judgment, done here — the check reads paths and hashes, and cannot read a
 sentence:
@@ -35,6 +40,8 @@ sentence:
   this skill exists;
 - does the answer still follow from the evidence, given what changed;
 - are numbers retyped into the prose that should be `{name}` placeholders;
+- does a claim resting on a threshold use a conditional answer, with a
+  threshold chosen before the value was known;
 - is the answer concise, and does it point at the publication section that
   carries the argument rather than repeating it.
 
@@ -104,7 +111,8 @@ evidence without showing them both.
    from. If a stage should produce it, that is a pipeline gap worth
    reporting. If it was imported or made by hand, declare it under
    `figures`, `datasets`, or `publications` with `imported_from` or
-   `created_by` so the project says so.
+   `created_by` so the project says so. A `value` entry with no stage is
+   reported as an error rather than `unattributed`: give it a stage.
 4. For each **error**, fix the reference: a missing path means the pipeline
    has not been run or pulled; a bad key or placeholder means a results
    file was restructured; a missing label means the publication was
@@ -128,6 +136,38 @@ project, not a tidy-up.
   what it means; leave the reasoning to the publication.
 - Numbers come from `value` evidence via placeholders, formatted to the
   precision the claim needs: `{ratio:.1f}x`, `{error:.0%}`.
+- A `value` entry must read a file a pipeline stage writes. A placeholder
+  over a results file written by hand, including one you write, is a
+  retyped number with extra steps, and the check fails it. If no stage
+  produces the number, add one (`/calkit:add-pipeline-stage`) instead of
+  writing the file. Never declare a file `imported_from` or `created_by`
+  to clear that error unless it really came from there, and never edit a
+  results file to change what an answer says.
+- When the claim itself depends on a value, not just the number in it,
+  e.g., significant or not, which method wins, write a conditional answer
+  so the wording follows the evidence on a rerun:
+
+  ```yaml
+  answer:
+    if p < 0.05: "The closure cuts error by {improvement:.1f}x."
+    else: "The closure does not measurably reduce error."
+  evidence:
+    - kind: value
+      path: results/closure.json
+      key: p-value
+      name: p
+    - kind: value
+      path: results/closure.json
+      key: improvement
+  ```
+
+  Conditions name `value` evidence, so those names must be valid Python
+  identifiers; give the entry a `name` otherwise. Every branch must be a
+  claim the evidence would support if it held. The threshold is part of
+  the claim: take it from the field's convention or the user, never pick
+  it to make the current branch hold, and don't use branches to hedge a
+  claim that should simply be weakened.
+
 - Point at the publication with a `publication` evidence entry carrying
   `section` (for the reader) and `label` (for the check), instead of an
   `explanation` that restates the argument.

{calkit_python-0.47.5 → calkit_python-0.47.6}/calkit/cli/list.py

@@ -243,7 +243,7 @@ def list_questions(
         render_question,
     )
 
-    def _texts(question: dict) -> list[str]:
+    def _texts(question: dict) -> list[str | dict]:
         evidence = question.get("evidence") or []
         return [question.get(f) or "" for f in TEMPLATED_FIELDS] + [
             ev.get("explanation") or ""
@@ -260,16 +260,18 @@ def list_questions(
         # let a fresh clone read as a project that types its braces. Any
         # placeholder left standing counts, whether it names evidence that
         # could not be read or names nothing at all: a brace meant to stay
-        # in the text is written '{{' and never reaches here.
+        # in the text is written '{{' and never reaches here. A conditional
+        # answer still in clauses is one whose conditions could not be read.
         unfilled = any(
-            placeholders(text)
+            isinstance(text, dict) or placeholders(text)
             for q in rendered
             if isinstance(q, dict)
             for text in _texts(q)
         )
         if unfilled:
             warn(
-                "Some placeholders could not be filled from the evidence. "
+                "Some placeholders or conditions could not be filled from "
+                "the evidence. "
                 "Run 'calkit check questions' to see why; 'calkit pull' if "
                 "the results files are not here yet.",
                 err=json_output,

{calkit_python-0.47.5 → calkit_python-0.47.6}/calkit/models/core.py

@@ -1841,7 +1841,15 @@ class Question(BaseModel):
 
     question: str
     hypothesis: str | None = None
-    answer: str | None = None
+    answer: str | dict[str, str] | None = Field(
+        default=None,
+        description=(
+            "The claim the evidence supports. A mapping keyed by 'if "
+            "<condition>', 'elif <condition>' and 'else' picks its wording "
+            "from the evidence, so an answer resting on a threshold states "
+            "the other outcome instead of going stale when a value moves."
+        ),
+    )
     notes: str | None = Field(
         default=None,
         description=(

{calkit_python-0.47.5 → calkit_python-0.47.6}/calkit/questions.py

@@ -32,13 +32,15 @@ updates itself.
 
 from __future__ import annotations
 
+import ast
 import glob
 import io
 import json
+import operator
 import os
 import re
 import string
-from typing import Any, Literal
+from typing import Any, Literal, TypeGuard
 
 from pydantic import BaseModel, Field
 
@@ -245,14 +247,208 @@ def placeholders(text: str) -> list[str]:
     return [m.group(1) for m in _PLACEHOLDER.finditer(text or "")]
 
 
-def render(text: str | None, values: dict[str, Any]) -> str | None:
+def _placeholder_problems(text: str, values: dict[str, Any]) -> list[str]:
+    if "{" not in text:
+        return []
+    try:
+        _FORMATTER.vformat(text, (), values)
+    except KeyError as e:
+        return [
+            f"placeholder {{{e.args[0]}}} names no evidence; write "
+            "'{{' and '}}' for braces meant to stay in the text"
+        ]
+    except (ValueError, IndexError) as e:
+        return [f"cannot render {text[:40]!r}...: {e}"]
+    return []
+
+
+_IF_KEY = re.compile(r"^\s*(if|elif)\s+(.+?)\s*$")
+_ELSE_KEY = re.compile(r"^\s*else\s*$")
+_COMPARISONS = {
+    ast.Lt: operator.lt,
+    ast.LtE: operator.le,
+    ast.Gt: operator.gt,
+    ast.GtE: operator.ge,
+    ast.Eq: operator.eq,
+    ast.NotEq: operator.ne,
+}
+
+
+def is_conditional(value: Any) -> TypeGuard[dict]:
+    """Whether a value picks its wording with ``if``/``elif``/``else``."""
+    return isinstance(value, dict)
+
+
+def parse_conditional(clauses: dict) -> list[tuple[str | None, str]]:
+    """Read a conditional's keys into ordered ``(condition, wording)``.
+
+    The condition is ``None`` for ``else``. Keys are read in the order
+    they appear in the file, so the clauses are tried in the order they
+    were written.
+    """
+    parsed: list[tuple[str | None, str]] = []
+    for position, (key, wording) in enumerate(clauses.items()):
+        opened = _IF_KEY.match(str(key))
+        if opened:
+            keyword, condition = opened.groups()
+            if keyword == "if" and position:
+                raise ValueError("only the first clause may be 'if'")
+            if keyword == "elif" and not parsed:
+                raise ValueError("'elif' with no 'if' before it")
+            parsed.append((condition, str(wording)))
+            continue
+        if _ELSE_KEY.match(str(key)):
+            if not parsed:
+                raise ValueError("'else' with no 'if' before it")
+            parsed.append((None, str(wording)))
+            continue
+        raise ValueError(f"expected 'if', 'elif' or 'else', got {key!r}")
+    if not parsed:
+        raise ValueError("a conditional needs at least an 'if' clause")
+    for condition, _ in parsed[:-1]:
+        if condition is None:
+            raise ValueError("'else' must be the last clause")
+    return parsed
+
+
+def _operand(node: ast.AST, values: dict[str, Any]) -> Any:
+    """One side of a comparison, resolved against the evidence values.
+
+    Names and literals are read directly; anything else is arithmetic and
+    goes to the same evaluator the calculations use, so there is one
+    audited path for arithmetic rather than two.
+    """
+    if isinstance(node, ast.Constant):
+        return node.value
+    if isinstance(node, ast.Name):
+        if node.id not in values:
+            raise KeyError(node.id)
+        return values[node.id]
+    for inner in ast.walk(node):
+        if isinstance(inner, ast.Name) and inner.id not in values:
+            raise KeyError(inner.id)
+    import arithmetic_eval  # type: ignore[import-untyped]
+
+    return arithmetic_eval.evaluate(ast.unparse(node), values)
+
+
+def _truth(node: ast.AST, values: dict[str, Any]) -> bool:
+    if isinstance(node, ast.BoolOp):
+        # Not short-circuited, so a misspelled name is an error whatever
+        # the current values are
+        outcomes = [_truth(v, values) for v in node.values]
+        if isinstance(node.op, ast.And):
+            return all(outcomes)
+        return any(outcomes)
+    if isinstance(node, ast.UnaryOp) and isinstance(node.op, ast.Not):
+        return not _truth(node.operand, values)
+    if isinstance(node, ast.Compare):
+        left = _operand(node.left, values)
+        for op, comparator in zip(node.ops, node.comparators):
+            right = _operand(comparator, values)
+            compare = _COMPARISONS.get(type(op))
+            if compare is None:
+                raise ValueError(
+                    f"{type(op).__name__} is not a supported comparison"
+                )
+            if not compare(left, right):
+                return False
+            left = right
+        return True
+    raise ValueError("a condition must compare values, e.g., 'p < 0.05'")
+
+
+def evaluate_condition(expression: str, values: dict[str, Any]) -> bool:
+    """Evaluate one ``if``/``elif`` condition against the evidence values.
+
+    Only comparisons, ``and``/``or``/``not`` and arithmetic are allowed, so
+    a condition read from ``calkit.yaml`` cannot call anything.
+    """
+    try:
+        tree = ast.parse(expression, mode="eval")
+    except SyntaxError as e:
+        raise ValueError(f"cannot parse condition {expression!r}: {e}") from e
+    try:
+        return bool(_truth(tree.body, values))
+    except KeyError:
+        # A name like 'paired-gain.vawt-8' reads as arithmetic and
+        # attribute access, so the failure would otherwise name a
+        # fragment of it and look like missing evidence.
+        unusable = [
+            name
+            for name in values
+            if not name.isidentifier() and name in expression
+        ]
+        if unusable:
+            raise ValueError(
+                f"condition {expression!r} refers to {unusable[0]!r}, which "
+                "cannot be read as a variable; give that evidence a 'name' "
+                "that is a valid Python identifier"
+            ) from None
+        raise
+    except ValueError:
+        raise
+    except Exception as e:
+        # E.g., comparing a string to a number, or syntax arithmetic_eval
+        # refuses, so callers only have to handle one kind of bad condition
+        raise ValueError(
+            f"cannot evaluate condition {expression!r}: {e}"
+        ) from e
+
+
+def check_conditional(clauses: dict, values: dict[str, Any]) -> list[str]:
+    """Problems with every clause of a conditional, not just the one chosen.
+
+    The clauses the current values don't select are the ones a rerun will
+    reach, so a typo in them is an error now rather than after the rerun.
+    """
+    try:
+        parsed = parse_conditional(clauses)
+    except ValueError as e:
+        return [f"conditional answer: {e}"]
+    messages: list[str] = []
+    held = False
+    for condition, wording in parsed:
+        if condition is None:
+            held = True
+        else:
+            try:
+                held = evaluate_condition(condition, values) or held
+            except KeyError as e:
+                messages.append(
+                    f"condition {condition!r} names no evidence {e.args[0]!r}"
+                )
+            except ValueError as e:
+                messages.append(str(e))
+        messages += _placeholder_problems(wording, values)
+    if not messages and not held:
+        messages.append(
+            "no condition of the conditional answer holds and there is no "
+            "'else' clause"
+        )
+    return messages
+
+
+def select_branch(clauses: dict, values: dict[str, Any]) -> str:
+    """The wording whose condition holds, for a conditional answer."""
+    for condition, wording in parse_conditional(clauses):
+        if condition is None or evaluate_condition(condition, values):
+            return wording
+    raise ValueError("no condition held and there is no 'else' clause")
+
+
+def render(text: str | dict | None, values: dict[str, Any]) -> str | None:
     """Fill a question text's placeholders from its evidence values.
 
     Raises ``KeyError`` for a name with no evidence and ``ValueError`` for
     a format spec the value cannot satisfy, so a template that cannot be
     rendered is an error rather than a silently unfilled sentence.
     """
-    if text is None or "{" not in text:
+    if text is None:
+        return text
+    if isinstance(text, dict):
+        text = select_branch(text, values)
+    if "{" not in text:
         return text
     return _FORMATTER.vformat(text, (), values)
 
@@ -596,7 +792,11 @@ def _check_publication_label(
 
 
 def _is_attributed(
-    path: str, stage: str | None, ck_info: dict, wdir: str
+    path: str,
+    stage: str | None,
+    ck_info: dict,
+    wdir: str,
+    computed: bool = False,
 ) -> bool:
     """Whether the project says where an evidence path came from.
 
@@ -605,7 +805,9 @@ def _is_attributed(
     may be declared as an artifact that records an import or a person,
     which is what :func:`calkit.provenance.has_provenance` reads---an
     imported dataset or a hand-drawn schematic is accounted for even
-    though there is nothing upstream to point at.
+    though there is nothing upstream to point at. With ``computed``, a
+    person doesn't count, since a number someone typed into a file is
+    still a magic number.
     """
     from calkit.provenance import has_provenance
 
@@ -624,7 +826,12 @@ def _is_attributed(
             if (
                 isinstance(artifact, dict)
                 and artifact.get("path") == path
-                and has_provenance(artifact)
+                and (
+                    artifact.get("stage") is not None
+                    or artifact.get("imported_from") is not None
+                    if computed
+                    else has_provenance(artifact)
+                )
             ):
                 return True
     return False
@@ -808,7 +1015,19 @@ def check_evidence(
                     ],
                 )
             )
-    if out.status == "ok" and not _is_attributed(
+    if (
+        out.status in ("ok", "changed")
+        and is_value_evidence(ev)
+        and not _is_attributed(path, out.stage, ck_info, wdir, computed=True)
+    ):
+        out.status = "error"
+        out.message = (
+            "no pipeline stage computes this value, so nothing can show "
+            "where the number came from or keep it current; produce the file "
+            "with a stage, or declare it with 'imported_from' if another "
+            "project computed it"
+        )
+    elif out.status == "ok" and not _is_attributed(
         path, out.stage, ck_info, wdir
     ):
         out.status = "unattributed"
@@ -874,17 +1093,10 @@ def check_question(
         ev.get("explanation") for ev in evidence
     ]
     for t in texts:
-        if not t or "{" not in t:
-            continue
-        try:
-            render(t, values)
-        except KeyError as e:
-            messages.append(
-                f"placeholder {{{e.args[0]}}} names no evidence; write "
-                "'{{' and '}}' for braces meant to stay in the text"
-            )
-        except (ValueError, IndexError) as e:
-            messages.append(f"cannot render {t[:40]!r}...: {e}")
+        if is_conditional(t):
+            messages += check_conditional(t, values)
+        elif t:
+            messages += _placeholder_problems(t, values)
     # Worst first, matching what the hub shows against each question: an
     # answer resting on nothing anyone can find is worse off than one
     # resting on something merely out of date.

{calkit_python-0.47.5 → calkit_python-0.47.6}/calkit/tests/cli/test_check.py

@@ -1182,11 +1182,24 @@ def test_check_questions(tmp_dir):
         calkit.ryaml.dump(ck_info, f)
     subprocess.check_call(["git", "add", "-A"])
     subprocess.check_call(["git", "commit", "-q", "-m", "Answer"])
+    # No stage computes the value, so it is a magic number and fails
+    proc = subprocess.run(
+        ["calkit", "check", "questions"], capture_output=True, text=True
+    )
+    assert proc.returncode != 0
+    assert "no pipeline stage computes this value" in proc.stdout
+    # Another project computed it, which is traceable
+    ck_info["datasets"] = [
+        {
+            "path": "results/findings.json",
+            "imported_from": {"project": "someone/else"},
+        }
+    ]
+    with open("calkit.yaml", "w") as f:
+        calkit.ryaml.dump(ck_info, f)
+    subprocess.check_call(["git", "commit", "-q", "-am", "Import it"])
     out = subprocess.check_output(["calkit", "check", "questions"], text=True)
     assert "Answers backed by current evidence: 1/1" in out
-    # Nothing in the project says where the results file came from, which
-    # is advice rather than a failure: the answer may be perfectly good
-    assert "Evidence with nothing recorded behind it: 1" in out
     # Listing renders the placeholder from the results file
     out = subprocess.check_output(["calkit", "list", "questions"], text=True)
     assert "answer: 8 of eight do." in out
@@ -1252,19 +1265,6 @@ def test_check_questions(tmp_dir):
     subprocess.check_call(["git", "commit", "-q", "-am", "Review"])
     out = subprocess.check_output(["calkit", "check", "questions"], text=True)
     assert "changed after the answer was written" not in out
-    # Declaring where the results file came from clears the advice
-    ck_info = calkit.load_calkit_info()
-    ck_info["datasets"] = [
-        {
-            "path": "results/findings.json",
-            "created_by": {"name": "A person"},
-        }
-    ]
-    with open("calkit.yaml", "w") as f:
-        calkit.ryaml.dump(ck_info, f)
-    subprocess.check_call(["git", "commit", "-q", "-am", "Declare"])
-    out = subprocess.check_output(["calkit", "check", "questions"], text=True)
-    assert "nothing recorded behind it" not in out
     # A console that cannot encode a check mark gets a '?' rather than a
     # UnicodeEncodeError, which on Windows would kill the command with no
     # output at all
@@ -1289,8 +1289,25 @@ def test_check_questions(tmp_dir):
     with open("calkit.yaml", "w") as f:
         calkit.ryaml.dump(ck_info, f)
     out = subprocess.check_output(["calkit", "list", "questions"], text=True)
-    assert "placeholders could not be filled" in out
+    assert "could not be filled" in out
     assert "It is {missing} of them." in out
+    # A conditional answer is listed as the branch that holds, and one with
+    # no branch holding is listed as written, with the same warning
+    ck_info["questions"][0]["answer"] = {
+        "if n_top > 4": "Most do.",
+        "else": "{n_top} do.",
+    }
+    with open("calkit.yaml", "w") as f:
+        calkit.ryaml.dump(ck_info, f)
+    out = subprocess.check_output(["calkit", "list", "questions"], text=True)
+    assert "answer: 0 do." in out
+    assert "could not be filled" not in out
+    del ck_info["questions"][0]["answer"]["else"]
+    with open("calkit.yaml", "w") as f:
+        calkit.ryaml.dump(ck_info, f)
+    out = subprocess.check_output(["calkit", "list", "questions"], text=True)
+    assert "could not be filled" in out
+    assert "if n_top > 4: Most do." in out
     # A project with no questions says so rather than printing nothing
     os.remove("calkit.yaml")
     with open("calkit.yaml", "w") as f: