calkit-python 0.41.16__tar.gz → 0.41.17__tar.gz

{calkit_python-0.41.16 → calkit_python-0.41.17}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: calkit-python
-Version: 0.41.16
+Version: 0.41.17
 Summary: Reproducibility simplified.
 Project-URL: Homepage, https://calkit.org
 Project-URL: Issues, https://github.com/calkit/calkit/issues
@@ -78,35 +78,30 @@ Description-Content-Type: text/markdown
 
 <!-- INCLUDE: docs/index.md -->
 
-Calkit helps you manage and automate research projects like a software
-engineer.
-
-Define computational environments,
-steps that process your data, create figures,
-presentations, and publications, connect to external tools,
-then iterate quickly and painlessly until your research questions are
-answered, tracking changes to all files along the way.
-At the end, deliver your entire project as a self-contained, self-documenting,
-version-controlled, and
-[single button reproducible](https://doi.org/10.1190/1.1822162)
-"calculation kit" so you and others can easily verify
-and build upon the results.
-
-## Guiding principles
-
-- Quality comes from iteration. Automation reduces the time and effort
-  needed to iterate, thereby increasing iteration and quality.
-- Automating a step can and should take roughly
-  the same amount of time as doing it once manually,
-  therefore it's almost always worth it.
-- Working in a "quick and dirty" way can easily become _not quick_ when
-  the dirtiness results in mistakes and/or discourages working in small steps.
+It's six months since you submitted your paper,
+do you know exactly how your figures and results were generated?
+
+You will if they're part of a Calkit project.
+
+Calkit helps you integrate code, data, figures, results, publications,
+and more into a cohesive, traceable, and portable _knowledge creation system_,
+so every output can be traced back to its source and reproduced with a
+single command.
+
+With industry standard tools combined into a unified and simplified experience
+tailored for research,
+you can reap the rewards of reproducibility and automation
+without the cognitive overhead.
+
+<!-- https://docs.google.com/drawings/d/1XMGnbgYYNFAVUBDyUaCyLfRB7efvJdrnrKmFlNmT19o/edit -->
+
+![pipeline](docs/img/pipeline.png)
 
 ## Features
 
 - A simplified [version control](https://docs.calkit.org/version-control)
   interface that unifies Git and DVC (Data Version Control),
-  so all materials can be kept in the same project repository.
+  so everything can be kept in the same project repository.
   This way, code doesn't need to be siloed away from other
   important artifacts like datasets, models, figures, or article PDFs,
   allowing you to work on all parts of a project without hopping around to
@@ -129,9 +124,10 @@ and build upon the results.
 - [Overleaf integration](https://docs.calkit.org/overleaf/), so
   analysis, visualization, and writing can all stay in sync
   (no more manual uploads!)
-- Support for running on high performance computing (HPC) systems that use
-  [SLURM schedulers](https://docs.calkit.org/pipeline/slurm).
-- Support for running with [GitHub Actions](https://docs.calkit.org/tutorials/github-actions).
+- Support for running on [high performance computing (HPC)](https://docs.calkit.org/hpc) systems
+  that use PBS or SLURM schedulers.
+- Support for automated running with
+  [GitHub Actions](https://docs.calkit.org/tutorials/github-actions).
 - Extensions for doing all of the above graphically in
   [JupyterLab](https://docs.calkit.org/jupyterlab) and
   [VS Code](https://marketplace.visualstudio.com/items?itemName=Calkit.calkit-vscode).
@@ -373,9 +369,37 @@ To back up or save the project, call:
 calkit save -am "Run pipeline"
 ```
 
+### With an AI coding agent
+
+Simply tell the [AI agent](https://docs.calkit.org/ai-tools):
+
+> Turn this folder into a Calkit project
+
+or
+
+> Create me a new Calkit project for investigating...
+
 <!-- END INCLUDE -->
 
 ## Get involved
 
 We welcome all kinds of contributions!
 See [CONTRIBUTING.md](CONTRIBUTING.md) to learn how to get involved.
+
+## Acknowledgements
+
+<!-- INCLUDE: docs/acknowledgements.md +1 -->
+
+Calkit is supported by the
+[Caltech Schmidt Academy of Software Engineering](https://sase.caltech.edu).
+
+<p align="center">
+  <a href="https://sase.caltech.edu" target="_blank">
+    <img width="40%" src="docs/img/caltech.png" alt="Caltech SASE">
+  </a>
+<a href="https://schmidtsciences.org/" target="_blank">
+    <img width="40%" src="docs/img/schmidt-sciences.png" alt="Schmidt Sciences">
+  </a>
+</p>
+
+<!-- END INCLUDE -->

{calkit_python-0.41.16 → calkit_python-0.41.17}/README.md

@@ -19,35 +19,30 @@
 
 <!-- INCLUDE: docs/index.md -->
 
-Calkit helps you manage and automate research projects like a software
-engineer.
-
-Define computational environments,
-steps that process your data, create figures,
-presentations, and publications, connect to external tools,
-then iterate quickly and painlessly until your research questions are
-answered, tracking changes to all files along the way.
-At the end, deliver your entire project as a self-contained, self-documenting,
-version-controlled, and
-[single button reproducible](https://doi.org/10.1190/1.1822162)
-"calculation kit" so you and others can easily verify
-and build upon the results.
-
-## Guiding principles
-
-- Quality comes from iteration. Automation reduces the time and effort
-  needed to iterate, thereby increasing iteration and quality.
-- Automating a step can and should take roughly
-  the same amount of time as doing it once manually,
-  therefore it's almost always worth it.
-- Working in a "quick and dirty" way can easily become _not quick_ when
-  the dirtiness results in mistakes and/or discourages working in small steps.
+It's six months since you submitted your paper,
+do you know exactly how your figures and results were generated?
+
+You will if they're part of a Calkit project.
+
+Calkit helps you integrate code, data, figures, results, publications,
+and more into a cohesive, traceable, and portable _knowledge creation system_,
+so every output can be traced back to its source and reproduced with a
+single command.
+
+With industry standard tools combined into a unified and simplified experience
+tailored for research,
+you can reap the rewards of reproducibility and automation
+without the cognitive overhead.
+
+<!-- https://docs.google.com/drawings/d/1XMGnbgYYNFAVUBDyUaCyLfRB7efvJdrnrKmFlNmT19o/edit -->
+
+![pipeline](docs/img/pipeline.png)
 
 ## Features
 
 - A simplified [version control](https://docs.calkit.org/version-control)
   interface that unifies Git and DVC (Data Version Control),
-  so all materials can be kept in the same project repository.
+  so everything can be kept in the same project repository.
   This way, code doesn't need to be siloed away from other
   important artifacts like datasets, models, figures, or article PDFs,
   allowing you to work on all parts of a project without hopping around to
@@ -70,9 +65,10 @@ and build upon the results.
 - [Overleaf integration](https://docs.calkit.org/overleaf/), so
   analysis, visualization, and writing can all stay in sync
   (no more manual uploads!)
-- Support for running on high performance computing (HPC) systems that use
-  [SLURM schedulers](https://docs.calkit.org/pipeline/slurm).
-- Support for running with [GitHub Actions](https://docs.calkit.org/tutorials/github-actions).
+- Support for running on [high performance computing (HPC)](https://docs.calkit.org/hpc) systems
+  that use PBS or SLURM schedulers.
+- Support for automated running with
+  [GitHub Actions](https://docs.calkit.org/tutorials/github-actions).
 - Extensions for doing all of the above graphically in
   [JupyterLab](https://docs.calkit.org/jupyterlab) and
   [VS Code](https://marketplace.visualstudio.com/items?itemName=Calkit.calkit-vscode).
@@ -314,9 +310,37 @@ To back up or save the project, call:
 calkit save -am "Run pipeline"
 ```
 
+### With an AI coding agent
+
+Simply tell the [AI agent](https://docs.calkit.org/ai-tools):
+
+> Turn this folder into a Calkit project
+
+or
+
+> Create me a new Calkit project for investigating...
+
 <!-- END INCLUDE -->
 
 ## Get involved
 
 We welcome all kinds of contributions!
 See [CONTRIBUTING.md](CONTRIBUTING.md) to learn how to get involved.
+
+## Acknowledgements
+
+<!-- INCLUDE: docs/acknowledgements.md +1 -->
+
+Calkit is supported by the
+[Caltech Schmidt Academy of Software Engineering](https://sase.caltech.edu).
+
+<p align="center">
+  <a href="https://sase.caltech.edu" target="_blank">
+    <img width="40%" src="docs/img/caltech.png" alt="Caltech SASE">
+  </a>
+<a href="https://schmidtsciences.org/" target="_blank">
+    <img width="40%" src="docs/img/schmidt-sciences.png" alt="Schmidt Sciences">
+  </a>
+</p>
+
+<!-- END INCLUDE -->

{calkit_python-0.41.16 → calkit_python-0.41.17}/calkit/cli/notebooks.py

@@ -490,7 +490,17 @@ def execute_notebook(
         typer.echo(f"Using {language} as notebook language")
     # First, ensure the specified environment has a kernel we can use
     # We need to check the environment type and create the kernel if needed
-    if language.lower() in ["python", "julia", "r"]:
+    docker_env = env.get("kind") == "docker"
+    if docker_env:
+        # Docker environments run the kernel inside the container, so there is
+        # no host kernel to register. Use the container's own kernel (default
+        # 'python3', overridable with the env's 'jupyter_kernel' key) and
+        # execute with Papermill inside the container further below.
+        kernel_name = env.get("jupyter_kernel") or {"r": "ir"}.get(
+            language.lower(), "python3"
+        )
+        display_name = kernel_name
+    elif language.lower() in ["python", "julia", "r"]:
         kernel_name, display_name = check_env_kernel(
             env_name=env_name,
             no_check=no_check,
@@ -563,10 +573,70 @@ def execute_notebook(
         typer.echo(f"Using kernel: {kernel_name}")
         typer.echo(f"Running with cwd: {notebook_dir}")
         typer.echo(f"Output will be saved to: {fpath_out_exec}")
-    # If this is a Python, Julia, or R notebook, we can use Papermill
+    # If this is a Python, Julia, or R notebook, we can use Papermill.
     # If it's a MATLAB notebook, we need to use the MATLAB kernel inside the
-    # specified environment
-    if language.lower() in ["python", "julia", "r"]:
+    # specified environment.
+    # Exception: If the environment is a Docker environment, we run Papermill
+    # inside the container so the kernel (which lives in the image) and
+    # Papermill share a process namespace---this avoids registering a host
+    # kernel and cross-container networking entirely
+    if docker_env:
+        # run_in_env mounts the project at the working directory and applies
+        # the env's user/platform/args, so we only need to invoke Papermill
+        # ipykernel is in the image, but Papermill may not be, so install it
+        # on demand (warning the user) before executing. We install into a
+        # writable temp dir on PYTHONPATH rather than the image's site-packages:
+        # run_in_env maps the host user into the container, so writing to the
+        # system location is typically denied, but /tmp is world-writable
+        pm_target = "/tmp/calkit-papermill"
+        pythonpath = f"PYTHONPATH={pm_target}"
+        pm_cmd = [
+            pythonpath,
+            "python",
+            "-c",
+            "import papermill",
+            "2>/dev/null",
+            "||",
+            "(",
+            "echo",
+            "papermill not found in image; installing it with pip...",
+            "&&",
+            "python",
+            "-m",
+            "pip",
+            "install",
+            "-q",
+            "--target",
+            pm_target,
+            "papermill",
+            ")",
+            "&&",
+            pythonpath,
+            "python",
+            "-m",
+            "papermill",
+            path,
+            fpath_out_exec,
+            "-k",
+            kernel_name,
+            "--log-output",
+        ]
+        if notebook_dir:
+            pm_cmd += ["--cwd", notebook_dir]
+        if parsed_params:
+            # JSON is valid YAML, so pass parameters as base64-encoded JSON to
+            # avoid shell-quoting issues across the container boundary
+            params_b64 = base64.b64encode(
+                json.dumps(parsed_params).encode()
+            ).decode()
+            pm_cmd += ["-b", params_b64]
+        run_in_env(
+            pm_cmd,
+            env_name=env_name,
+            no_check=no_check,
+            verbose=verbose,
+        )
+    elif language.lower() in ["python", "julia", "r"]:
         papermill.execute_notebook(
             input_path=path,
             output_path=fpath_out_exec,

{calkit_python-0.41.16 → calkit_python-0.41.17}/calkit/models/core.py

@@ -96,8 +96,6 @@ class Environment(BaseModel):
     # without a spec file (e.g. Matlab/Slurm/PBS) don't carry an unused field
     # and a required ``path`` doesn't conflict with the base's optional one.
     description: str | None = None
-    stage: str | None = None
-    default: bool | None = None
 
 
 class CondaEnvironment(Environment):
@@ -112,6 +110,8 @@ class VenvEnvironment(Environment):
     # If unset, the prefix is resolved on the fly (defaults to .venv next to
     # the spec file, nesting under .calkit/envs/{name}/.venv on conflict)
     prefix: str | None = None
+    # Python version to use when creating the environment
+    python: str | None = None
 
 
 class UvEnvironment(Environment):
@@ -125,6 +125,8 @@ class UvVenvEnvironment(Environment):
     # If unset, the prefix is resolved on the fly (defaults to .venv next to
     # the spec file, nesting under .calkit/envs/{name}/.venv on conflict)
     prefix: str | None = None
+    # Python version to use when creating the environment
+    python: str | None = None
 
 
 class PixiEnvironment(Environment):
@@ -153,6 +155,24 @@ class DockerEnvironment(Environment):
     shell: Literal["bash", "sh"] = "sh"
     command_mode: Literal["shell", "entrypoint"] = "shell"
     platform: str | None = None
+    # Working directory inside the container; defaults to ``/work`` at run time
+    wdir: str | None = None
+    # User to run the container as; defaults to the host user at run time
+    user: str | None = None
+    # Files added to the container as dependencies
+    deps: list[str] | None = None
+    # Environment variables to set in the container
+    env_vars: dict[str, str] | None = None
+    # Ports to expose, e.g. ``"8080:80"``
+    ports: list[str] | None = None
+    # GPUs to make available, passed to ``docker run --gpus``
+    gpus: str | None = None
+    # Extra arguments passed to ``docker run``
+    args: list[str] | None = None
+    # Name of the Jupyter kernel inside the image, used when executing
+    # notebooks with ``calkit nb execute``; defaults to ``python3`` (or ``ir``
+    # for R images)
+    jupyter_kernel: str | None = None
 
 
 class REnvironment(Environment):
@@ -195,6 +215,7 @@ class SSHEnvironment(BaseModel):
     key: str | None = None
     send_paths: list[str] = ["./*"]
     get_paths: list[str] = ["*"]
+    description: str | None = None
 
 
 class Software(BaseModel):

{calkit_python-0.41.16 → calkit_python-0.41.17}/calkit/tests/cli/test_notebooks.py

@@ -10,6 +10,7 @@ import sys
 import pytest
 
 import calkit
+import calkit.notebooks
 
 
 def test_clean_notebook_outputs(tmp_dir):
@@ -264,3 +265,87 @@ def test_execute_notebook_auto_env(tmp_dir):
     # TODO: Check if the notebook is saved in the project and if it
     # has an environment specified, either in the notebooks section or a
     # pipeline stage
+
+
+@pytest.mark.skipif(
+    sys.platform == "win32",
+    reason="Docker Linux images are unavailable on windows-latest GHA runners",
+)
+def test_execute_notebook_docker(tmp_dir):
+    # A Docker environment runs the kernel inside the container, so we execute
+    # with Papermill inside the container rather than registering a host kernel
+    if shutil.which("docker") is None:
+        pytest.skip("Docker is not available")
+    image = "calkit-nb-docker-test:latest"
+    # Bake ipykernel and Papermill into the image so execution is deterministic
+    with open("Dockerfile", "w") as f:
+        f.write(
+            "FROM python:3.10-slim\n"
+            "RUN pip install --no-cache-dir ipykernel papermill\n"
+            "WORKDIR /work\n"
+        )
+    subprocess.check_call(["docker", "build", "-t", image, "."])
+    subprocess.check_call(["calkit", "init"])
+    ck_info = {
+        "environments": {
+            "docker-env": {"kind": "docker", "image": image, "wdir": "/work"}
+        }
+    }
+    with open("calkit.yaml", "w") as f:
+        calkit.ryaml.dump(ck_info, f)
+    # A notebook with a 'parameters' cell so we can verify injection
+    nb = {
+        "cells": [
+            {
+                "cell_type": "code",
+                "execution_count": None,
+                "id": "params",
+                "metadata": {"tags": ["parameters"]},
+                "outputs": [],
+                "source": ["a = 1\n"],
+            },
+            {
+                "cell_type": "code",
+                "execution_count": None,
+                "id": "compute",
+                "metadata": {},
+                "outputs": [],
+                "source": ["result = a * 10\n", "print('result =', result)\n"],
+            },
+        ],
+        "metadata": {},
+        "nbformat": 4,
+        "nbformat_minor": 5,
+    }
+    os.makedirs("notebooks")
+    with open(os.path.join("notebooks", "main.ipynb"), "w") as f:
+        json.dump(nb, f)
+    subprocess.check_call(
+        [
+            "calkit",
+            "nb",
+            "exec",
+            "-e",
+            "docker-env",
+            "--no-check",
+            "-p",
+            "a=4",
+            "notebooks/main.ipynb",
+        ]
+    )
+    # The injected parameter should have produced result = 40 in the executed
+    # notebook saved under .calkit/notebooks/executed
+    exec_path = calkit.notebooks.get_executed_notebook_path(
+        notebook_path="notebooks/main.ipynb",
+        to="notebook",
+        parameters={"a": 4},
+    )
+    with open(exec_path) as f:
+        executed = json.load(f)
+    text = "".join(
+        "".join(o.get("text", []))
+        for cell in executed["cells"]
+        for o in cell.get("outputs", [])
+        if o.get("name") == "stdout"
+    )
+    assert "result = 40" in text

calkit_python-0.41.17/docs/acknowledgements.md

@@ -0,0 +1,13 @@
+# Acknowledgements
+
+Calkit is supported by the
+[Caltech Schmidt Academy of Software Engineering](https://sase.caltech.edu).
+
+<p align="center">
+  <a href="https://sase.caltech.edu" target="_blank">
+    <img width="40%" src="/img/caltech.png" alt="Caltech SASE">
+  </a>
+<a href="https://schmidtsciences.org/" target="_blank">
+    <img width="40%" src="/img/schmidt-sciences.png" alt="Schmidt Sciences">
+  </a>
+</p>

{calkit_python-0.41.16 → calkit_python-0.41.17}/docs/environments.md

@@ -626,7 +626,7 @@ Model class: `VenvEnvironment`
 | ----------- | --------------- | -------- |
 | kind        | Literal['venv'] | yes      |
 | path        | str             | yes      |
-| prefix      | str             | yes      |
+| prefix      | str             | no       |
 | python      | str             | no       |
 | description | str             | no       |
 
@@ -638,7 +638,7 @@ Model class: `UvVenvEnvironment`
 | ----------- | ------------------ | -------- |
 | kind        | Literal['uv-venv'] | yes      |
 | path        | str                | yes      |
-| prefix      | str                | yes      |
+| prefix      | str                | no       |
 | python      | str                | no       |
 | description | str                | no       |
 
@@ -657,32 +657,24 @@ Model class: `PixiEnvironment`
 
 Model class: `DockerEnvironment`
 
-| Parameter    | Type                           | Required |
-| ------------ | ------------------------------ | -------- |
-| kind         | Literal['docker']              | yes      |
-| image        | str                            | yes      |
-| path         | str                            | no       |
-| platform     | str                            | no       |
-| command_mode | Literal['shell'\|'entrypoint'] | no       |
-| shell        | str                            | no       |
-| deps         | list[str]                      | no       |
-| env_vars     | dict[str, str]                 | no       |
-| ports        | list[str]                      | no       |
-| gpus         | str                            | no       |
-| user         | str                            | no       |
-| wdir         | str                            | no       |
-| args         | list[str]                      | no       |
-| description  | str                            | no       |
-
-#### `renv`
-
-Model class: `REnvironment`
-
-| Parameter   | Type            | Required |
-| ----------- | --------------- | -------- |
-| kind        | Literal['renv'] | yes      |
-| path        | str             | yes      |
-| description | str             | no       |
+| Parameter      | Type                           | Required |
+| -------------- | ------------------------------ | -------- |
+| kind           | Literal['docker']              | yes      |
+| path           | str                            | no       |
+| image          | str                            | yes      |
+| layers         | list[str]                      | no       |
+| shell          | Literal['bash'\|'sh']          | no       |
+| command_mode   | Literal['shell'\|'entrypoint'] | no       |
+| platform       | str                            | no       |
+| wdir           | str                            | no       |
+| user           | str                            | no       |
+| deps           | list[str]                      | no       |
+| env_vars       | dict[str, str]                 | no       |
+| ports          | list[str]                      | no       |
+| gpus           | str                            | no       |
+| args           | list[str]                      | no       |
+| jupyter_kernel | str                            | no       |
+| description    | str                            | no       |
 
 #### `julia`
 
@@ -702,6 +694,7 @@ Model class: `MatlabEnvironment`
 | Parameter   | Type              | Required |
 | ----------- | ----------------- | -------- |
 | kind        | Literal['matlab'] | yes      |
+| version     | str               | no       |
 | products    | list[str]         | no       |
 | description | str               | no       |
 
@@ -709,12 +702,12 @@ Model class: `MatlabEnvironment`
 
 Model class: `NixEnvironment`
 
-| Parameter   | Type                            | Required |
-| ----------- | ------------------------------- | -------- |
-| kind        | Literal['nix']                  | yes      |
-| path        | str (must end with 'flake.nix') | yes      |
-| shell       | str (name of devShell to enter) | no       |
-| description | str                             | no       |
+| Parameter   | Type           | Required |
+| ----------- | -------------- | -------- |
+| kind        | Literal['nix'] | yes      |
+| path        | str            | yes      |
+| shell       | str            | no       |
+| description | str            | no       |
 
 #### `slurm`
 
@@ -728,6 +721,17 @@ Model class: `SlurmEnvironment`
 | default_setup   | list[str]        | no       |
 | description     | str              | no       |
 
+#### `renv`
+
+Model class: `REnvironment`
+
+| Parameter   | Type            | Required |
+| ----------- | --------------- | -------- |
+| kind        | Literal['renv'] | yes      |
+| path        | str             | yes      |
+| prefix      | str             | yes      |
+| description | str             | no       |
+
 #### `ssh`
 
 Model class: `SSHEnvironment`