calkit-python 0.44.0__tar.gz → 0.44.1__tar.gz

{calkit_python-0.44.0 → calkit_python-0.44.1}/CONTRIBUTING.md

@@ -27,6 +27,15 @@ This guide will help you get started.
    - [Miniforge](https://conda-forge.org/download/)
    - [uv](https://docs.astral.sh/uv/getting-started/installation/)
    - [Pixi](https://pixi.sh/latest/)
+1. **Create the virtual environment:**
+   ```bash
+   uv sync --all-packages
+   ```
+   This repo is a uv workspace: the hub backend is a member, and both it
+   and the `calkit` package share a single `.venv` at the repo root.
+   Always pass `--all-packages`, since a bare `uv sync` installs only one
+   member and uninstalls the other's dependencies, leaving either the
+   `calkit` CLI or the hub backend unable to import its dependencies.
 1. **Run tests** to ensure everything is working:
    ```bash
    uv run pytest

{calkit_python-0.44.0 → calkit_python-0.44.1}/Makefile

@@ -6,7 +6,10 @@ help: ## Show this help.
 .PHONY: install
 install: ## Create the project's virtual environment.
 	@echo "🚀 Creating virtual environment"
-	@uv sync
+	# --all-packages, since a bare `uv sync` would install only this
+	# package and uninstall the hub backend's dependencies from the
+	# workspace's single shared .venv
+	@uv sync --all-packages
 
 .PHONY: dev
 dev: ## Start up the hub containers for development.
@@ -57,8 +60,8 @@ sync-docs: schema ## Sync documentation content from docs/*.md into README.md.
 	@echo "🚀 Syncing documentation"
 	@uv run python scripts/sync-docs.py
 
-.PHONY: changes
-changes: ## List each product's commits since its last release.
+.PHONY: unreleased
+unreleased: ## List each product's commits since its last release.
 	@uv run python scripts/list-changes.py
 
 .PHONY: sync-resources

{calkit_python-0.44.0 → calkit_python-0.44.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: calkit-python
-Version: 0.44.0
+Version: 0.44.1
 Summary: Reproducibility simplified.
 Project-URL: Homepage, https://calkit.org
 Project-URL: Issues, https://github.com/calkit/calkit/issues

{calkit_python-0.44.0 → calkit_python-0.44.1}/calkit/git.py

@@ -13,6 +13,12 @@ from git.exc import InvalidGitRepositoryError
 
 __all__ = ["InvalidGitRepositoryError", "get_repo"]
 
+# Marks the block of exemptions ensure_path_is_not_filtered maintains, so
+# it's clear in a file Calkit shares with whatever installed the filter.
+CALKIT_ATTRIBUTES_COMMENT = (
+    "# Added by Calkit: pipeline outputs must be committed unfiltered"
+)
+
 
 def get_repo(path: str | None = None) -> git.Repo:
     """Return a git.Repo for ``path`` (or cwd), searching parent dirs.
@@ -460,6 +466,106 @@ def ensure_dvc_pointer_is_not_ignored(repo, path: str) -> None:
         f.write(exception + "\n")
 
 
+def get_filter_driver(repo: git.Repo, path: str | PathLike) -> str | None:
+    """The name of the clean/smudge filter Git applies to ``path``, if any.
+
+    Asks Git rather than reading attributes files, since a filter can be
+    declared in any of several places with different precedence---most
+    relevantly ``$GIT_DIR/info/attributes``, where ``nbstripout --install``
+    puts it, which is per-clone and never committed.
+    """
+    path = Path(path).as_posix()
+    try:
+        # -z gives NUL-separated <path> <attr> <value> triples, so a path
+        # containing ": " can't be mistaken for a field separator.
+        out = repo.git.check_attr("-z", "filter", "--", path)
+    except git.GitCommandError as e:
+        warnings.warn(
+            f"Failed to check Git attributes for {path}: {e}", stacklevel=2
+        )
+        return None
+    fields = out.split("\0")
+    if len(fields) < 3:
+        return None
+    value = fields[2]
+    # Git reports these two when no driver applies; anything else names one.
+    if value in ("unspecified", "unset"):
+        return None
+    return value
+
+
+def ensure_path_is_not_filtered(
+    repo: git.Repo, path: str | PathLike
+) -> bool | None:
+    """Ensure Git stores ``path`` byte-for-byte, applying no clean filter.
+
+    A clean filter rewrites content on its way into Git while leaving the
+    working tree alone, and Git compares the filtered forms, so the committed
+    bytes can differ from the file on disk with nothing showing up as
+    modified. For a pipeline output that is fatal: DVC hashes the working
+    tree file, so anything reading the repository instead---the Calkit hub,
+    a fresh clone, a collaborator---sees a hash that disagrees with
+    ``dvc.lock`` and reports the stage stale forever. ``nbstripout`` does
+    exactly this to notebooks.
+
+    The exemption is written to ``$GIT_DIR/info/attributes`` because that
+    file outranks every other source of attributes, including the
+    ``.gitattributes`` and the ``$GIT_DIR/info/attributes`` line
+    ``nbstripout --install`` writes. Returns True if a rule was added, None
+    if the path was already unfiltered.
+    """
+    path = Path(path).as_posix()
+    if get_filter_driver(repo, path) is None:
+        return
+    attributes_path = os.path.join(repo.git_dir, "info", "attributes")
+    # Quoted only when it has to be: an unquoted pattern is what a reader
+    # expects, and Git only needs the quotes for whitespace.
+    pattern = f'"{path}"' if any(c.isspace() for c in path) else path
+    # ``-filter`` unsets the attribute rather than leaving it unspecified, so
+    # no lower-precedence rule can put a driver back.
+    rule = f"{pattern} -filter"
+    existing_lines = []
+    if os.path.isfile(attributes_path):
+        with open(attributes_path, "r", encoding="utf-8") as f:
+            existing_lines = f.read().splitlines()
+    # Within one attributes file the last matching line wins, so the rule goes
+    # at the end. If it's already in there, a filter still applying means
+    # something was appended after it -- another `nbstripout --install`, most
+    # likely -- and moving ours back to the end is what fixes that. Dropping
+    # the old copy first keeps the file from growing a duplicate each time.
+    lines = [line for line in existing_lines if line != rule]
+    if CALKIT_ATTRIBUTES_COMMENT not in lines:
+        lines.append(CALKIT_ATTRIBUTES_COMMENT)
+    lines.append(rule)
+    os.makedirs(os.path.dirname(attributes_path), exist_ok=True)
+    with open(attributes_path, "w", encoding="utf-8") as f:
+        f.write("\n".join(lines) + "\n")
+    # Nothing higher-precedence than this file should exist, but a filter that
+    # survives the rewrite would otherwise strip content silently on the next
+    # commit, so say so rather than reporting success.
+    if get_filter_driver(repo, path) is not None:
+        warnings.warn(
+            f"{path} is still filtered by Git despite an exemption in "
+            f"{attributes_path}",
+            stacklevel=2,
+        )
+        return
+    # The rule alone doesn't fix what's already committed: Git trusts the
+    # index's cached stat info, so an unchanged file is never re-read and the
+    # filtered blob stays. Renormalizing re-hashes it through the new
+    # attributes, staging the real content so the next commit repairs the
+    # repository rather than waiting for the stage to run again.
+    if ls_files(repo, "--", path):
+        try:
+            repo.git.add("--renormalize", "--", path)
+        except git.GitCommandError as e:
+            warnings.warn(
+                f"Failed to renormalize {path} after unfiltering it: {e}",
+                stacklevel=2,
+            )
+    return True
+
+
 def resolve_ref(repo: git.Repo, ref: str) -> str | None:
     """Return the commit a revision points at, fetching if it isn't here.
 

{calkit_python-0.44.0 → calkit_python-0.44.1}/calkit/pipeline.py

@@ -1582,6 +1582,16 @@ def to_dvc(
                     calkit.git.ensure_path_is_ignored(repo, path=out.path)
                 elif isinstance(out, PathOutput) and out.storage == "git":
                     calkit.git.ensure_path_is_not_ignored(repo, path=out.path)
+                    if out.path.endswith(".ipynb"):
+                        # A notebook stage's storage is declared here, in the
+                        # pipeline, so a clean filter installed in the clone
+                        # (nbstripout) must not overrule it by stripping the
+                        # outputs back out on the way into Git. That would
+                        # commit bytes that don't match what DVC hashed, with
+                        # nothing showing as modified locally.
+                        calkit.git.ensure_path_is_not_filtered(
+                            repo, path=out.path
+                        )
                 elif isinstance(out, PathOutput) and out.storage == "dvc-zip":
                     calkit.git.ensure_path_is_ignored(repo, path=out.path)
                     calkit.dvc.zip.add(out.path, is_stage_output=True)

{calkit_python-0.44.0 → calkit_python-0.44.1}/calkit/tests/test_git.py

@@ -515,3 +515,79 @@ def test_check_branch_is_current(tmp_dir):
     msg = calkit.git.check_branch_is_current(repo)
     assert msg is not None
     assert f"'origin/{default_branch}'" in msg
+
+
+def _install_stripping_filter(repo: git.Repo, pattern: str = "*.ipynb"):
+    """Configure a clean filter that mangles content, the way nbstripout does.
+
+    A stand-in for nbstripout so the test doesn't need it installed: what
+    matters is that Git rewrites content on its way into the object store,
+    not what the rewrite is.
+    """
+    # Spelled with no quotes, spaces, or backslashes in any token: Git runs
+    # filter commands through a shell---its bundled sh on Windows---which eats
+    # the backslashes in a Windows interpreter path, leaving a command that
+    # never runs. Marked required so that failure is a loud error instead of a
+    # silent pass-through that looks like the content was never filtered.
+    repo.git.config("filter.stripper.clean", "sed -e s/.*/stripped/")
+    repo.git.config("filter.stripper.required", "true")
+    attributes = Path(repo.git_dir) / "info" / "attributes"
+    attributes.parent.mkdir(parents=True, exist_ok=True)
+    attributes.write_text(f"{pattern} filter=stripper\n")
+
+
+def test_get_filter_driver(tmp_dir):
+    repo = git.Repo.init()
+    Path("nb.ipynb").write_text("real content")
+    assert calkit.git.get_filter_driver(repo, "nb.ipynb") is None
+    _install_stripping_filter(repo)
+    assert calkit.git.get_filter_driver(repo, "nb.ipynb") == "stripper"
+    # A path the pattern doesn't cover is untouched.
+    assert calkit.git.get_filter_driver(repo, "notes.txt") is None
+
+
+def test_ensure_path_is_not_filtered(tmp_dir):
+    repo = git.Repo.init()
+    repo.git.config("user.email", "test@example.com")
+    repo.git.config("user.name", "Test")
+    path = ".calkit/notebooks/executed/nb.ipynb"
+    Path(path).parent.mkdir(parents=True)
+    Path(path).write_text("real content")
+    Path("other.ipynb").write_text("real content")
+    attributes = Path(repo.git_dir) / "info" / "attributes"
+    # A repo with no filters is left completely alone.
+    assert calkit.git.ensure_path_is_not_filtered(repo, path=path) is None
+    assert not attributes.exists()
+    _install_stripping_filter(repo)
+    repo.git.add("-A")
+    repo.git.commit("-m", "Init")
+    # The committed bytes don't match the working tree, and nothing shows as
+    # modified, which is what makes this worth guarding against.
+    assert repo.git.show(f"HEAD:{path}") == "stripped", (
+        "the stand-in clean filter didn't run, so there's nothing here for "
+        "the exemption to fix; check that its command works on this platform"
+    )
+    assert not repo.git.status("--porcelain")
+    assert calkit.git.ensure_path_is_not_filtered(repo, path=path)
+    assert calkit.git.get_filter_driver(repo, path) is None
+    # The exemption is scoped: everything else stays filtered.
+    assert calkit.git.get_filter_driver(repo, "other.ipynb") == "stripper"
+    # Already-committed content is repaired rather than left for the next run.
+    repo.git.commit("-m", "Unfilter")
+    assert repo.git.show(f"HEAD:{path}") == "real content"
+    assert repo.git.show("HEAD:other.ipynb") == "stripped"
+    # Idempotent: a second call neither re-adds the rule nor errors.
+    before = attributes.read_text()
+    assert calkit.git.ensure_path_is_not_filtered(repo, path=path) is None
+    assert attributes.read_text() == before
+    # Re-installing the filter appends a pattern after our exemption, and the
+    # last matching line in the file wins, so the notebook is filtered again.
+    with open(attributes, "a", encoding="utf-8") as f:
+        f.write("*.ipynb filter=stripper\n")
+    assert calkit.git.get_filter_driver(repo, path) == "stripper"
+    # Moving the exemption back to the end restores it, without the file
+    # accumulating a copy of the rule per call.
+    assert calkit.git.ensure_path_is_not_filtered(repo, path=path)
+    assert calkit.git.get_filter_driver(repo, path) is None
+    rule = f"{path} -filter"
+    assert attributes.read_text().splitlines().count(rule) == 1

{calkit_python-0.44.0 → calkit_python-0.44.1}/calkit/tests/test_pipeline.py

@@ -2,12 +2,16 @@
 
 import os
 import subprocess
+import sys
 import warnings
+from pathlib import Path
 
 import git
 import pytest
 
 import calkit
+import calkit.git
+import calkit.notebooks
 import calkit.pipeline
 from calkit.environments import get_env_lock_fpath
 from calkit.models.pipeline import LatexStage
@@ -2203,7 +2207,6 @@ def test_get_concurrent_scheduler_stages():
 
 
 def test_get_matrix_item_targets(tmp_dir):
-    import sys
 
     subprocess.check_call(["git", "init"])
     subprocess.check_call([sys.executable, "-m", "dvc", "init"])
@@ -2471,3 +2474,49 @@ def test_ref_resolver_is_not_shared_between_projects(tmp_dir):
         finally:
             os.chdir(cwd)
     assert shas["one"] != shas["two"]
+
+
+def test_to_dvc_unfilters_notebook_outputs(tmp_dir):
+    # Notebook output storage is declared in the pipeline, so a filter
+    # installed in the clone (nbstripout) must not overrule it by stripping
+    # the outputs back out on the way into Git---that commits bytes that
+    # disagree with what DVC hashed, with nothing showing as modified locally.
+    repo = git.Repo.init()
+    # Spelled with no quotes, spaces, or backslashes in any token: Git runs
+    # filter commands through a shell---its bundled sh on Windows---which eats
+    # the backslashes in a Windows interpreter path, leaving a command that
+    # never runs. Marked required so that failure is a loud error instead of a
+    # silent pass-through that looks like the content was never filtered.
+    repo.git.config("filter.stripper.clean", "sed -e s/.*/stripped/")
+    repo.git.config("filter.stripper.required", "true")
+    attributes = Path(repo.git_dir) / "info" / "attributes"
+    attributes.parent.mkdir(parents=True, exist_ok=True)
+    attributes.write_text("*.ipynb filter=stripper\n")
+    nb_path = "notebooks/my-notebook.ipynb"
+    os.makedirs("notebooks")
+    with open(nb_path, "w") as f:
+        f.write("{}")
+    ck_info = {
+        "environments": {
+            "py": {"kind": "uv-venv", "path": "requirements.txt"},
+        },
+        "pipeline": {
+            "stages": {
+                "notebook-1": {
+                    "kind": "jupyter-notebook",
+                    "environment": "py",
+                    "notebook_path": nb_path,
+                    "html_storage": "git",
+                    "executed_ipynb_storage": "git",
+                },
+            }
+        },
+    }
+    calkit.pipeline.to_dvc(ck_info=ck_info, write=True)
+    executed_path = calkit.notebooks.get_executed_notebook_path(
+        notebook_path=nb_path, to="notebook", as_posix=True
+    )
+    assert calkit.git.get_filter_driver(repo, executed_path) is None
+    # Scoped to what the pipeline declares: the source notebook a user cleans
+    # on purpose keeps its filter.
+    assert calkit.git.get_filter_driver(repo, nb_path) == "stripper"

{calkit_python-0.44.0 → calkit_python-0.44.1}/docs/calkit-yaml.md

@@ -138,6 +138,18 @@ tables:
 
 Like the other artifacts, a table is identified by its path.
 
+On the Calkit Hub, tables show up on the project's Tables page, where each
+one can be searched, sorted, and linked to by cell.
+Declaring a table isn't required to appear there: CSV, TSV, and JSON Lines
+files in a `tables` or `results` directory are detected automatically, as are
+LaTeX tables written to their own file, i.e., a `.tex` file in those
+directories holding a bare `tabular` environment, or one wrapped in a
+`standalone` document.
+A paper that happens to contain a table is not detected as one, since a table
+pulled out of a larger document is a fragment of that document rather than an
+artifact of its own.
+Declaring one only adds a title and description of your own.
+
 Columns aren't described yet, and neither is a symbolic name for a table.
 Both are expected to arrive alongside symbol metadata, which is where
 per-column types and units belong.

{calkit_python-0.44.0 → calkit_python-0.44.1}/docs/notebooks.md

@@ -36,6 +36,10 @@ we can focus on three rules:
    Git commits.
    This can be done by installing `nbstripout` and running
    `nbstripout --install` in the project directory.
+   Note that the notebooks Calkit generates as pipeline outputs (those under
+   `.calkit/notebooks`) are exempted from stripping automatically when the
+   pipeline is compiled, since their outputs are the pipeline's results and
+   stripping them would commit content that doesn't match what DVC hashed.
 1. A notebook must run in one of the project's [environments](environments.md).
 1. Notebooks should be incorporated into the project's
    [pipeline](pipeline/index.md).

{calkit_python-0.44.0 → calkit_python-0.44.1}/hub/AGENTS.md

@@ -76,6 +76,86 @@ If instead you are running the built image (plain `docker compose up`, not
 `make dev`), the frontend is a static Nginx build and will keep serving old code
 until you rebuild it: `docker compose build frontend`.
 
+### A frontend request fails with a network error
+
+A network error (as opposed to an HTTP status) means nothing answered, so the
+backend is most likely not running. Check with `docker compose ps -a`: if
+`backend` shows `exited (1)`, read `docker compose logs backend` for the
+traceback.
+
+An `ImportError` or `ModuleNotFoundError` from `calkit` there — for example
+`cannot import name 'MARIMO_DETECT_N_BYTES' from 'calkit.notebooks'` — usually
+means the container's virtual environment is stale, not that the code is wrong.
+In dev, `docker-compose.override.yml` mounts the repo at `/app` and masks the
+image's environment with an **anonymous volume** at `/app/.venv`. That volume
+outlives `docker compose build` and `up --force-recreate`, so an environment
+created months ago keeps shadowing the working tree no matter how often the
+image is rebuilt. Renew it explicitly:
+
+```sh
+docker compose build backend
+docker compose up -d --renew-anon-volumes backend
+```
+
+`--renew-anon-volumes` only discards anonymous volumes; named volumes such as
+the database keep their data. Reach for this after merging a branch that changes
+dependencies or moves code between calkit-python and the backend.
+
+Once the volume is current, calkit-python is installed as an editable workspace
+member resolving to `/app/calkit`, so host edits to it are live and this should
+not recur until the environment itself needs to change.
+
+### Postgres errors that a column does not exist
+
+A log full of `ERROR: column project.<something> does not exist` means the
+schema is behind the models: migrations did not run. Check why in the prestart
+logs, not the backend's:
+
+```sh
+docker compose logs prestart
+```
+
+`Can't locate revision identified by '<rev>'` means `alembic_version` points at
+a revision that isn't in `backend/app/alembic/versions/`, so alembic refuses to
+run anything and the schema freezes wherever it was. This comes from having
+migrated the dev database while on a branch whose migration was later squashed,
+renamed, or abandoned — the revision can be gone from git entirely, so don't
+assume it is recoverable (`git log --all -S '<rev>'` to find out).
+
+The reliable fix on a dev database is to recreate it, since all data is seeded
+by `create-initial-data.py` anyway:
+
+```sh
+docker compose down db
+docker volume rm calkit_app-db-data
+docker compose up -d db backend
+```
+
+Repairing in place is possible but fiddly: the abandoned branch has usually
+applied *some* of its DDL, so the database is genuinely half-migrated rather
+than merely mis-stamped. It takes `alembic stamp <rev> --purge` (plain `stamp`
+also reads the current revision and hits the same error) to a revision known to
+predate the divergence, then hand-applying whatever the next migration would
+have done that isn't already there, then `alembic upgrade head`.
+
+Either way, verify the result against a from-scratch migration rather than
+trusting the stamp — build a scratch database, migrate it from empty, and diff:
+
+```sh
+docker compose exec db psql -U postgres -d postgres -c "CREATE DATABASE app_check"
+docker compose exec -e POSTGRES_DB=app_check backend alembic upgrade head
+docker compose exec db pg_dump -U postgres -d app --schema-only > /tmp/a.sql
+docker compose exec db pg_dump -U postgres -d app_check --schema-only > /tmp/b.sql
+diff /tmp/a.sql /tmp/b.sql
+docker compose exec db psql -U postgres -d postgres -c "DROP DATABASE app_check"
+```
+
+Note that `alembic check` is not a clean signal here: it reports drift
+(a missing `ck_account_name_lowercase` check constraint, some foreign-key
+churn) even against a perfectly migrated database, because the models don't
+declare those constraints. The from-scratch diff is what distinguishes real
+divergence from that baseline noise.
+
 ## Common Patterns
 
 ### Modifying API Contracts

{calkit_python-0.44.0 → calkit_python-0.44.1}/hub/backend/README.md

@@ -53,12 +53,20 @@ go there and install it.
 
 The backend is a member of the repo's uv workspace, which installs
 calkit-python from the working tree.
-From this directory you can install all the dependencies with:
+The workspace shares a single virtual environment at the repo root, so
+install with `--all-packages`:
 
 ```console
-$ uv sync
+$ uv sync --all-packages
 ```
 
+A bare `uv sync` installs only one workspace member and uninstalls the
+other's dependencies, so running it here breaks the `calkit` CLI, and
+running it at the repo root leaves the backend unable to start with
+`ModuleNotFoundError` on packages like `mixpanel`.
+Always pass `--all-packages`, in particular after merging a change that
+touches dependencies.
+
 Then you can start a shell session with the new environment (which lives at
 the repo root) with: