calkit-python 0.37.6__tar.gz → 0.38.1__tar.gz

calkit_python-0.38.1/.claude-plugin/marketplace.json

@@ -0,0 +1,15 @@
+{
+  "name": "calkit",
+  "owner": { "name": "Calkit" },
+  "plugins": [
+    {
+      "name": "calkit",
+      "description": "Skills for working with Calkit projects.",
+      "source": {
+        "source": "git-subdir",
+        "url": "https://github.com/calkit/calkit.git",
+        "path": "agent-plugin"
+      }
+    }
+  ]
+}

{calkit_python-0.37.6 → calkit_python-0.38.1}/.gitignore

@@ -19,3 +19,4 @@ calk9
 dist
 *.vsix
 vscode-ext/out
+.claude

calkit_python-0.38.1/AGENTS.md

@@ -0,0 +1,18 @@
+# Agent instructions for working on Calkit
+
+## Repo structure
+
+- The main Python package/CLI lives in `calkit`
+- The JupyterLab extension lives in `src`
+- The VS Code extension lives in `vscode-ext`
+
+## Working
+
+See `CONTRIBUTING.md` for tool usage, style guidelines, etc.
+
+To run tests, use `uv run pytest`.
+
+To sync the docs and format all the code, run `make format`.
+
+Wrap prose at natural breakpoints in phrases or punctuation to keep max
+line length below 80 characters.

{calkit_python-0.37.6 → calkit_python-0.38.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: calkit-python
-Version: 0.37.6
+Version: 0.38.1
 Summary: Reproducibility simplified.
 Project-URL: Homepage, https://calkit.org
 Project-URL: Issues, https://github.com/calkit/calkit/issues
@@ -174,7 +174,15 @@ You may also want to install [Docker](https://docker.com),
 since that is the default method by which LaTeX environments are created.
 If you want to use the [Calkit Cloud](https://calkit.io)
 for collaboration and backup as a DVC remote,
-you can [set up cloud integration](https://docs.calkit.org/cloud-integration).
+you can [set up cloud integration](https://docs.calkit.org/cloud-integration) with:
+
+```sh
+calkit cloud login
+```
+
+If you use AI agents like Claude, Copilot, or Codex,
+see [AI tools](https://docs.calkit.org/ai-tools)
+to learn how to install agent skills for working with Calkit.
 
 ### Use without installing
 

{calkit_python-0.37.6 → calkit_python-0.38.1}/README.md

@@ -115,7 +115,15 @@ You may also want to install [Docker](https://docker.com),
 since that is the default method by which LaTeX environments are created.
 If you want to use the [Calkit Cloud](https://calkit.io)
 for collaboration and backup as a DVC remote,
-you can [set up cloud integration](https://docs.calkit.org/cloud-integration).
+you can [set up cloud integration](https://docs.calkit.org/cloud-integration) with:
+
+```sh
+calkit cloud login
+```
+
+If you use AI agents like Claude, Copilot, or Codex,
+see [AI tools](https://docs.calkit.org/ai-tools)
+to learn how to install agent skills for working with Calkit.
 
 ### Use without installing
 

calkit_python-0.38.1/agent-plugin/.claude-plugin/plugin.json

@@ -0,0 +1,5 @@
+{
+  "name": "calkit",
+  "description": "Agent skills for working with Calkit projects.",
+  "version": "0.1.0"
+}

calkit_python-0.38.1/agent-plugin/skills/add-pipeline-stage/SKILL.md

@@ -0,0 +1,122 @@
+---
+name: add-pipeline-stage
+description: Add a single new stage to an existing Calkit pipeline. Use when
+  the user invokes `/calkit:add-pipeline-stage` or asks to add a script,
+  notebook, or command to the pipeline.
+---
+
+# Add a pipeline stage
+
+Add a single new stage to an existing Calkit pipeline—wiring up the right
+environment, inputs, outputs, and storage mode.
+
+## Before you start
+
+Read `calkit.yaml` to understand what environments are already defined (you
+will reuse one if it fits) and what stages already exist and what they output
+(your new stage may depend on them).
+
+## Option A: Add with `calkit xr`
+
+Use `xr` when the stage is a script or notebook file (`.py`, `.ipynb`, `.R`,
+`.jl`, `.m`, `.sh`, `.tex`) and you want auto-detection of I/O and
+environment.
+
+```bash
+calkit xr scripts/new-stage.py --dry-run   # preview first
+calkit xr scripts/new-stage.py             # run for real
+```
+
+Override detected I/O or environment if needed:
+
+```bash
+calkit xr scripts/new-stage.py \
+  --input data/processed.csv \
+  --input config/params.yaml \
+  --output results/output.csv \
+  --environment main
+```
+
+After running, inspect the new stage in `calkit.yaml` to confirm I/O is
+correct.
+
+## Option B: Write the stage manually
+
+Write YAML directly when the stage is a shell command, or when you need
+`iterate_over`, specific storage modes, or other fine-grained control.
+
+| File type        | Stage kind                   |
+| ---------------- | ---------------------------- |
+| `.py`            | `python-script`              |
+| `.ipynb`         | `jupyter-notebook`           |
+| `.sh`            | `shell-script`               |
+| Inline command   | `shell-command` or `command` |
+| `.tex`           | `latex`                      |
+| `.R`             | `r-script`                   |
+| `.jl`            | `julia-script`               |
+| `.m`             | `matlab-script`              |
+| `docker run ...` | `docker-command`             |
+
+```yaml
+pipeline:
+  stages:
+    # ... existing stages ...
+
+    new-stage-name:
+      kind: python-script
+      script_path: scripts/new-stage.py
+      environment: main # must match a key in environments:
+      inputs:
+        - data/processed.csv
+      outputs:
+        - results/output.csv
+```
+
+## Wiring dependencies
+
+If the new stage reads files produced by an existing stage, declare them as
+inputs. List files explicitly, or depend on all outputs of a prior stage:
+
+```yaml
+inputs:
+  - data/processed.csv # explicit file
+  - from_stage_outputs: process-data # all outputs of that stage
+```
+
+If an existing stage reads files your new stage now produces, update that
+stage's inputs too.
+
+## Output storage
+
+- `git`—small or text files collaborators should see without `calkit pull`
+  (PDFs, summary CSVs, HTML reports)
+- `dvc` (default)—large or binary files
+
+```yaml
+outputs:
+  - results/large-matrix.npy # DVC (default)
+  - path: results/summary.csv
+    storage: git
+  - path: results/report.html
+    storage: git
+    delete_before_run: false # keep old version if stage is skipped
+```
+
+## After adding the stage
+
+```bash
+calkit run                                    # run full pipeline
+calkit run new-stage-name                     # run only this stage
+calkit status                                 # check what's stale
+calkit commit -am "Add new-stage-name stage"
+```
+
+## Common mistakes
+
+- **Wrong environment name**: Must exactly match a key in `environments`. Run
+  `calkit check env --name <env>` to verify.
+- **Two stages claiming the same output**: Each file can only be the output of
+  one stage.
+- **Missing inputs**: If a stage reads a file but doesn't declare it, DVC
+  won't re-run the stage when that file changes.
+- **Editing `dvc.yaml`**: Don't—it's auto-generated from `calkit.yaml`.

calkit_python-0.38.1/agent-plugin/skills/conventions/SKILL.md

@@ -0,0 +1,306 @@
+---
+name: conventions
+description: Calkit conventions and foundational context. Load whenever working
+  in a Calkit project—defines calkit.yaml structure, environments, pipeline
+  stages, CLI commands, and version control conventions.
+---
+
+# Calkit conventions
+
+Calkit is a tool for research project management focused on automation and
+reproducibility—enabling continuous delivery for research.
+It provides a unified
+interface over Git (source control) and DVC (data versioning), and adds
+environment management and pipeline orchestration on top. The central artifact
+is `calkit.yaml`, the project's metadata database.
+
+## The `calkit.yaml` file
+
+`calkit.yaml` lives at the repo root and contains:
+
+- `environments`—computational environments (Python venvs, Conda, Docker,
+  R, Julia, MATLAB, etc.)
+- `pipeline.stages`—the reproducible pipeline
+- `notebooks`—registered Jupyter notebooks
+- `datasets`, `figures`, `publications`—versioned project outputs
+- `procedures`, `calculations`, `references`—supporting metadata
+- `showcase`—elements shown on the project's Calkit Cloud homepage
+
+A minimal example:
+
+```yaml
+environments:
+  main:
+    kind: uv-venv
+    path: requirements.txt
+    python: "3.13"
+
+pipeline:
+  stages:
+    process-data:
+      kind: python-script
+      script_path: scripts/process.py
+      environment: main
+      inputs:
+        - data/raw.csv
+      outputs:
+        - data/processed.csv
+```
+
+## Environments
+
+Every pipeline stage must reference a named environment defined in
+`environments`. Calkit enforces this to ensure reproducibility. Supported
+kinds:
+
+| Kind      | Spec file                               |
+| --------- | --------------------------------------- |
+| `uv-venv` | `requirements.txt` or `pyproject.toml`  |
+| `venv`    | `requirements.txt`                      |
+| `conda`   | `environment.yml`                       |
+| `pixi`    | `pixi.toml`                             |
+| `docker`  | (image name, no local spec file needed) |
+| `renv`    | `renv.lock`                             |
+| `julia`   | `Project.toml`                          |
+| `matlab`  | (no spec file)                          |
+| `ssh`     | (remote host config)                    |
+| `slurm`   | (HPC cluster config)                    |
+
+Example environment definitions:
+
+```yaml
+environments:
+  main:
+    kind: uv-venv
+    path: requirements.txt
+    python: "3.13"
+
+  texlive:
+    kind: docker
+    image: texlive/texlive:latest-full
+
+  r-env:
+    kind: renv
+    path: renv.lock
+```
+
+Calkit generates a lock file for each environment under `.calkit/env-locks/`
+and uses it as a DVC dependency. If the environment changes, affected stages
+are automatically flagged for re-run.
+
+## Pipeline stages
+
+Stages live under `pipeline.stages` in `calkit.yaml`. Every stage requires
+`kind` and `environment`. Most stages also declare `inputs` and `outputs`.
+
+### Common parameters (all stage kinds)
+
+| Parameter      | Type   | Notes                                           |
+| -------------- | ------ | ----------------------------------------------- |
+| `kind`         | string | Required. See stage kinds below.                |
+| `environment`  | string | Required. Must match a key in `environments`.   |
+| `inputs`       | list   | Files this stage reads. Changes trigger re-run. |
+| `outputs`      | list   | Files this stage writes. Stored in Git or DVC.  |
+| `wdir`         | string | Working directory (relative to repo root).      |
+| `always_run`   | bool   | Force re-run even if nothing changed.           |
+| `iterate_over` | list   | Parameterize the stage over a list of values.   |
+| `description`  | string | Human-readable description.                     |
+
+### Stage kinds and their required fields
+
+**`python-script`**: Run a Python script
+
+```yaml
+kind: python-script
+script_path: scripts/run.py
+args: ["--flag", "value"] # optional
+```
+
+**`jupyter-notebook`**: Execute a Jupyter notebook
+
+```yaml
+kind: jupyter-notebook
+notebook_path: notebooks/analysis.ipynb
+html_storage: git # optional, default: dvc
+executed_ipynb_storage: git # optional, default: dvc
+parameters: { key: value } # optional, papermill parameters
+```
+
+**`shell-command`**: Run an arbitrary shell command
+
+```yaml
+kind: shell-command
+command: "python -m mymodule --arg val"
+shell: bash # optional, default: bash
+```
+
+**`shell-script`**: Run a shell script file
+
+```yaml
+kind: shell-script
+script_path: scripts/run.sh
+```
+
+**`latex`**: Compile a LaTeX document to PDF
+
+```yaml
+kind: latex
+target_path: paper/paper.tex
+pdf_storage: git # optional, default: dvc
+```
+
+**`r-script`**: Run an R script
+
+```yaml
+kind: r-script
+script_path: scripts/analysis.R
+```
+
+**`julia-script`** / **`julia-command`**: Run Julia code
+
+```yaml
+kind: julia-script
+script_path: scripts/run.jl
+```
+
+**`matlab-script`** / **`matlab-command`**: Run MATLAB code
+
+```yaml
+kind: matlab-script
+script_path: scripts/run.m
+```
+
+**`docker-command`**: Run a command inside a Docker container
+
+```yaml
+kind: docker-command
+command: "docker run --rm myimage mycommand"
+```
+
+**`command`**: Generic command (for tools that don't fit other kinds)
+
+```yaml
+kind: command
+command: "mytool --input data/raw.csv --output data/out.csv"
+```
+
+### Outputs: Git vs. DVC storage
+
+By default, outputs are stored with DVC (large file storage). Use
+`storage: git` for small files that belong in version control:
+
+```yaml
+outputs:
+  - data/processed.csv # DVC (default)
+  - path: data/meta.json
+    storage: git # committed to Git
+  - path: results/summary.txt
+    storage: git
+    delete_before_run: false # don't delete before re-running
+```
+
+### Declaring dependencies between stages
+
+Use `from_stage_outputs` to declare that a stage depends on another stage's
+outputs rather than listing individual files:
+
+```yaml
+stages:
+  collect-data:
+    kind: python-script
+    script_path: scripts/collect.py
+    environment: main
+    outputs:
+      - data/raw.csv
+
+  process-data:
+    kind: python-script
+    script_path: scripts/process.py
+    environment: main
+    inputs:
+      - from_stage_outputs: collect-data
+    outputs:
+      - data/processed.csv
+```
+
+### Iterating over parameters
+
+```yaml
+stages:
+  train-model:
+    kind: python-script
+    script_path: scripts/train.py
+    environment: main
+    args:
+      - "--model={model}"
+    iterate_over:
+      - arg_name: model
+        values:
+          - linear-regression
+          - random-forest
+    inputs:
+      - data/processed.csv
+    outputs:
+      - models/{model}.pkl
+```
+
+## Relationship to DVC
+
+Calkit compiles `calkit.yaml` into `dvc.yaml` when `calkit run` is called.
+Do not edit `dvc.yaml` directly—it is a generated file. The authoritative
+pipeline definition is always `calkit.yaml`.
+
+DVC handles:
+
+- Caching: stages that haven't changed since last run are skipped
+- Remote storage: large files pushed/pulled with `calkit push` / `calkit pull`
+- Dependency graph: run order is computed from declared inputs/outputs
+
+## Key CLI commands
+
+| Command                         | What it does                                         |
+| ------------------------------- | ---------------------------------------------------- |
+| `calkit run`                    | Run the pipeline (skips unchanged stages)            |
+| `calkit run --force`            | Force re-run all stages                              |
+| `calkit status`                 | Show which stages are stale                          |
+| `calkit xr <file>`              | Auto-detect stage type, env, I/O, add to pipeline    |
+| `calkit xenv -n <env> -- <cmd>` | Run a command in a named environment                 |
+| `calkit push`                   | Push Git commits and DVC-tracked files to remotes    |
+| `calkit pull`                   | Pull latest code and data                            |
+| `calkit save`                   | Auto-add, commit, and push (Git + DVC)               |
+| `calkit commit -m "msg"`        | Commit all tracked changes (Git + DVC)               |
+| `calkit add <file>`             | Add a file to version control                        |
+| `calkit check env --name <env>` | Verify an environment matches its spec               |
+| `calkit new`                    | Create new project objects (notebook, dataset, etc.) |
+
+## `calkit xr`: The fastest path to a reproducible stage
+
+`xr` ("execute and record") is the recommended way to add scripts and
+notebooks to the pipeline for the first time. It:
+
+1. Detects the stage kind from the file extension (`.py`, `.ipynb`, `.R`,
+   `.jl`, `.m`, `.sh`, `.tex`)
+2. Detects or creates the right environment
+3. Heuristically detects inputs and outputs from the script's file I/O calls
+4. Adds the stage to `calkit.yaml` and `dvc.yaml`
+5. Runs the stage
+
+```bash
+calkit xr scripts/run.py              # Python script
+calkit xr notebooks/analysis.ipynb    # Jupyter notebook
+calkit xr paper/paper.tex             # LaTeX document
+calkit xr scripts/run.R               # R script
+
+calkit xr scripts/run.py --input data/raw.csv --output results/out.csv
+calkit xr scripts/run.py --environment main
+calkit xr scripts/run.py --dry-run    # see what would happen without running
+```
+
+## Version control conventions
+
+- Source code and small outputs: tracked with Git
+- Large binary files, datasets, model weights: tracked with DVC
+- Lock files (`.calkit/env-locks/`): committed to Git, act as DVC dependencies
+- `dvc.yaml`: generated by Calkit—don't edit manually
+- `.calkit/`: Calkit's internal directory—commit its contents unless they
+  are large generated files

calkit_python-0.38.1/agent-plugin/skills/create-pipeline/SKILL.md

@@ -0,0 +1,189 @@
+---
+name: create-pipeline
+description: Convert an existing repo with ad hoc scripts into a fully
+  reproducible Calkit pipeline. Use when the user invokes
+  `/calkit:create-pipeline` or asks to make a project reproducible end-to-end.
+---
+
+# Create a Calkit pipeline
+
+Convert an existing repo with ad hoc scripts and manual steps into a fully
+reproducible Calkit pipeline. When complete, `calkit run` should reproduce all
+important outputs from scratch.
+
+## Step 1: Understand the existing repo
+
+Before writing any YAML, map out what's already there.
+Start with the README. Users will typically write manual environment
+creation steps, and lists of script and commands to run in order.
+This is like a manual pipeline.
+Next:
+
+1. List all scripts and notebooks: look in `scripts/`, `notebooks/`, `src/`,
+   and the repo root for `.py`, `.R`, `.jl`, `.m`, `.ipynb` files.
+2. Read each script to understand what it reads and writes.
+3. Identify the dependency order: which outputs of script A become inputs to
+   script B?
+4. Note which environment each script needs (Python version, packages, R,
+   Julia, MATLAB, Docker, etc.).
+
+Ask the user if the order or dependencies are unclear. Do not guess at data
+flow.
+
+## Step 2: Initialize the project
+
+If there is no `calkit.yaml`, run:
+
+```bash
+calkit init
+```
+
+This sets up Git (if needed) and DVC. If `calkit.yaml` already exists but has
+no `pipeline` section, skip this—you will add one.
+
+## Step 3: Define environments
+
+Every stage must reference a named environment. Identify what's needed:
+
+- **Python**: `requirements.txt` or `pyproject.toml` → `uv-venv`;
+  `environment.yml` → `conda`
+- **R**: `renv.lock` → `renv`
+- **Julia**: `Project.toml` → `julia`
+- **LaTeX**: `docker` with `texlive/texlive:latest-full`
+- **MATLAB**: `matlab`
+
+Add environments to `calkit.yaml`:
+
+```yaml
+environments:
+  main:
+    kind: uv-venv
+    path: requirements.txt
+    python: "3.13"
+```
+
+Name a single Python environment `main`. With multiple environments, use
+descriptive names (e.g., `analysis`, `paper`).
+
+## Step 4: Try `calkit xr` first
+
+For each script or notebook, try `xr` before writing YAML by hand. It
+auto-detects stage kind, environment, and I/O:
+
+```bash
+calkit xr scripts/collect-data.py --dry-run   # preview first
+calkit xr scripts/collect-data.py             # run for real
+```
+
+Work through scripts in dependency order. After each call, verify the new
+stage in `calkit.yaml`. Override detected I/O if needed:
+
+```bash
+calkit xr scripts/train.py \
+  --input data/processed.csv \
+  --input config/params.yaml \
+  --output models/model.pkl
+```
+
+`xr` is not magic—sometimes it won't get everything right. But it handles
+most of the boilerplate, so you can focus on verifying correctness and filling
+in gaps. Scan through scripts for missing inputs, outputs, environment
+dependencies, etc., and add those manually in `calkit.yaml` as needed.
+
+## Step 5: Write stages manually when `xr` isn't suitable
+
+Write stages directly in `calkit.yaml` when:
+
+- You need fine control over `always_run`, `iterate_over`, storage modes, etc.
+- `xr` does not support the stage kind
+
+```yaml
+pipeline:
+  stages:
+    collect-data:
+      kind: python-script
+      script_path: scripts/collect-data.py
+      environment: main
+      outputs:
+        - data/raw.csv
+
+    process-data:
+      kind: jupyter-notebook
+      notebook_path: notebooks/process.ipynb
+      environment: main
+      inputs:
+        - data/raw.csv
+      outputs:
+        - data/processed.csv
+        - figures/fig1.png
+
+    build-paper:
+      kind: latex
+      target_path: paper/paper.tex
+      environment: texlive
+      inputs:
+        - figures/fig1.png
+        - references.bib
+      outputs: []
+```
+
+Rules:
+
+- Stage names should be short, lowercase, hyphenated.
+- Every file one stage writes and another reads must be declared as both an
+  output and an input—otherwise stages may run out of order or miss re-runs.
+- Use `from_stage_outputs: stage-name` when a stage consumes all outputs of a
+  prior stage.
+
+## Step 6: Choose output storage
+
+For each output:
+
+- **Git**: small, text-based files (summary CSVs, JSON metadata, PDFs, HTML
+  reports)
+- **DVC** (default): large or binary files (raw data, model weights, matrices)
+
+```yaml
+outputs:
+  - data/raw.csv # DVC (default)
+  - path: data/meta.json
+    storage: git
+  - path: paper/paper.pdf
+    storage: git
+```
+
+When in doubt, ask the user. Storage mode affects whether collaborators can
+see the file without `calkit pull`.
+
+## Step 7: Run and verify
+
+```bash
+calkit run
+```
+
+Common errors:
+
+- **Missing inputs**: a stage lists an input that no earlier stage produces
+- **Environment errors**: run `calkit check env --name <env>` to diagnose
+- **Script errors**: fix the script, then re-run
+
+Run a single stage: `calkit run <stage-name>`
+Force re-run everything: `calkit run --force`
+
+## Step 8: Commit
+
+```bash
+calkit commit -m "Add reproducible pipeline"
+```
+
+## Common mistakes
+
+- **Missing intermediate files**: if `process.py` reads `data/raw.csv` but
+  that file isn't declared as an output of `collect-data`, DVC won't track the
+  dependency.
+- **Absolute paths in scripts**: use paths relative to the repo root (or
+  `wdir` if set).
+- **Editing `dvc.yaml` directly**: always edit `calkit.yaml`—`dvc.yaml` is
+  regenerated by Calkit.
+- **Missing `environment`**: every stage needs one. Use `_system` for stages
+  that only use system tools.