calkit-python 0.28.3__tar.gz → 0.29.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {calkit_python-0.28.3 → calkit_python-0.29.0}/PKG-INFO +1 -1
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/__init__.py +1 -1
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/cli/config.py +14 -4
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/cli/main.py +65 -16
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/cli/new.py +31 -3
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/cli/notebooks.py +59 -14
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/dvc.py +23 -13
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/models/pipeline.py +38 -15
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/tests/models/test_pipeline.py +16 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/notebooks.md +1 -1
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/quickstart.md +1 -1
- calkit_python-0.29.0/docs/tutorials/github-actions.md +83 -0
- calkit_python-0.29.0/docs/tutorials/img/actions-repo-secrets.png +0 -0
- calkit_python-0.29.0/docs/tutorials/img/latex-codespaces/new-token.png +0 -0
- calkit_python-0.29.0/docs/tutorials/img/quick-actions.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/index.md +1 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/mkdocs.yml +1 -0
- calkit_python-0.28.3/docs/tutorials/img/latex-codespaces/new-token.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/.github/FUNDING.yml +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/.github/workflows/docs.yml +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/.github/workflows/format.yml +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/.github/workflows/publish-test.yml +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/.github/workflows/publish.yml +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/.github/workflows/test.yml +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/.gitignore +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/.pre-commit-config.yaml +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/.python-version +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/CITATION.cff +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/CONTRIBUTING.md +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/LICENSE +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/Makefile +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/README.md +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/__main__.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/calc.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/check.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/cli/__init__.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/cli/check.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/cli/cloud.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/cli/core.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/cli/describe.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/cli/import_.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/cli/list.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/cli/office.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/cli/overleaf.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/cli/update.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/cloud.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/conda.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/config.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/core.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/datasets.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/docker.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/environments.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/git.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/github.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/gui.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/jupyter.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/magics.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/matlab.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/models/__init__.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/models/core.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/models/io.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/models/iteration.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/notebooks.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/office.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/ops.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/pipeline.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/releases.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/server.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/templates/__init__.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/templates/core.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/templates/latex/__init__.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/templates/latex/article/paper.tex +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/templates/latex/core.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/templates/latex/jfm/jfm.bst +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/templates/latex/jfm/jfm.cls +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/templates/latex/jfm/lineno-FLM.sty +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/templates/latex/jfm/paper.tex +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/templates/latex/jfm/upmath.sty +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/tests/__init__.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/tests/cli/__init__.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/tests/cli/test_check.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/tests/cli/test_config.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/tests/cli/test_list.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/tests/cli/test_main.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/tests/cli/test_new.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/tests/cli/test_notebooks.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/tests/models/__init__.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/tests/models/test_iteration.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/tests/test_calc.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/tests/test_check.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/tests/test_conda.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/tests/test_core.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/tests/test_dvc.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/tests/test_jupyter.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/tests/test_magics.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/tests/test_notebooks.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/tests/test_pipeline.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/tests/test_templates.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/calkit/zenodo.py +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/CNAME +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/apps.md +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/calculations.md +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/calkit-yaml.md +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/cli-reference.md +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/cloud-integration.md +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/datasets.md +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/dependencies.md +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/environments.md +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/examples.md +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/help.md +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/img/c-to-the-k-white.svg +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/img/calkit-no-bg.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/img/connect-zenodo.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/img/jupyterlab-params.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/img/vscode-nb-params.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/index.md +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/installation.md +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/local-server.md +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/overleaf.md +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/pipeline/index.md +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/pipeline/manual-steps.md +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/pipeline/running-and-logging.md +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/references.md +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/releases.md +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/adding-latex-pub-docker.md +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/conda-envs.md +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/existing-project.md +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/first-project.md +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/latex-codespaces/building-codespace.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/latex-codespaces/codespaces-secrets-2.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/latex-codespaces/editor-split.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/latex-codespaces/go-to-linked-code.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/latex-codespaces/issue-from-selection.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/latex-codespaces/new-project.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/latex-codespaces/new-pub-2.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/latex-codespaces/paper.tex.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/latex-codespaces/project-home-3.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/latex-codespaces/push.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/latex-codespaces/stage.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/office/anakin-excel.jpg +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/office/chart-more-rows.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/office/create-project.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/office/elsevier-research-data-guidelines.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/office/excel-chart.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/office/excel-data.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/office/insert-link-to-file.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/office/needs-clone.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/office/new-stage.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/office/phd-comics-version-control.webp +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/office/pipeline-out-of-date.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/office/status-more-rows.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/office/uncommitted-changes.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/office/untracked-data.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/office/updated-publication.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/office/word-to-pdf-stage-2.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/office/workflow-page.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/openfoam/clone.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/openfoam/create-project.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/openfoam/datasets-page.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/openfoam/figure-on-website-updated.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/openfoam/figure-on-website.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/openfoam/new-token.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/openfoam/reclone.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/openfoam/status-after-import-dataset.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/img/run-proc.png +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/latex-codespaces.md +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/matlab.md +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/notebook-pipeline.md +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/office.md +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/openfoam.md +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/tutorials/procedures.md +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/docs/version-control.md +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/pyproject.toml +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/scripts/install.ps1 +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/scripts/install.sh +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/test/nb-params.ipynb +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/test/nb-subdir.ipynb +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/test/pipeline.ipynb +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/test/test-log.log +0 -0
- {calkit_python-0.28.3 → calkit_python-0.29.0}/uv.lock +0 -0
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
from __future__ import annotations
|
|
4
4
|
|
|
5
|
+
import os
|
|
5
6
|
import subprocess
|
|
6
7
|
|
|
7
8
|
import git
|
|
@@ -82,11 +83,17 @@ def setup_remote(
|
|
|
82
83
|
configure_remote()
|
|
83
84
|
set_remote_auth()
|
|
84
85
|
except subprocess.CalledProcessError:
|
|
85
|
-
|
|
86
|
+
if not os.path.isfile(".dvc/config"):
|
|
87
|
+
raise_error(
|
|
88
|
+
"DVC remote config failed; have you run `calkit init`?"
|
|
89
|
+
)
|
|
90
|
+
raise_error(
|
|
91
|
+
"Failed to configure DVC remote; check DVC config for errors"
|
|
92
|
+
)
|
|
86
93
|
except InvalidGitRepositoryError:
|
|
87
94
|
raise_error("Current directory is not a Git repository")
|
|
88
|
-
except ValueError as e:
|
|
89
|
-
raise_error(e)
|
|
95
|
+
except (ValueError, RuntimeError) as e:
|
|
96
|
+
raise_error(f"Failed to set up DVC remote: {e}")
|
|
90
97
|
if not no_commit:
|
|
91
98
|
repo = git.Repo()
|
|
92
99
|
repo.git.add(".dvc/config")
|
|
@@ -101,7 +108,10 @@ def setup_remote_auth():
|
|
|
101
108
|
"""Store a Calkit cloud token in the local DVC config for all Calkit
|
|
102
109
|
remotes.
|
|
103
110
|
"""
|
|
104
|
-
|
|
111
|
+
try:
|
|
112
|
+
remotes = get_remotes()
|
|
113
|
+
except Exception:
|
|
114
|
+
raise_error("Cannot list DVC remotes; check DVC config for errors")
|
|
105
115
|
for name, url in remotes.items():
|
|
106
116
|
if name == "calkit" or name.startswith("calkit:"):
|
|
107
117
|
typer.echo(f"Setting up authentication for DVC remote: {name}")
|
|
@@ -981,22 +981,23 @@ def run(
|
|
|
981
981
|
except Exception as e:
|
|
982
982
|
os.environ.pop("CALKIT_PIPELINE_RUNNING", None)
|
|
983
983
|
raise_error(f"Pipeline compilation failed: {e}")
|
|
984
|
-
|
|
985
|
-
|
|
986
|
-
|
|
987
|
-
|
|
988
|
-
|
|
989
|
-
|
|
990
|
-
|
|
991
|
-
|
|
992
|
-
|
|
993
|
-
|
|
994
|
-
|
|
995
|
-
|
|
996
|
-
|
|
997
|
-
|
|
998
|
-
|
|
999
|
-
|
|
984
|
+
if save_log:
|
|
985
|
+
# Get status of Git repo before running
|
|
986
|
+
repo = git.Repo()
|
|
987
|
+
git_rev = repo.head.commit.hexsha
|
|
988
|
+
try:
|
|
989
|
+
git_branch = repo.active_branch.name
|
|
990
|
+
except TypeError:
|
|
991
|
+
# If no branch is checked out, we are in a detached HEAD state
|
|
992
|
+
git_branch = None
|
|
993
|
+
git_changed_files_before = calkit.git.get_changed_files(repo=repo)
|
|
994
|
+
git_staged_files_before = calkit.git.get_staged_files(repo=repo)
|
|
995
|
+
git_untracked_files_before = calkit.git.get_untracked_files(repo=repo)
|
|
996
|
+
# Get status of DVC repo before running
|
|
997
|
+
dvc_repo = dvc.repo.Repo()
|
|
998
|
+
dvc_status_before = dvc_repo.status()
|
|
999
|
+
dvc_data_status_before = dvc_repo.data_status()
|
|
1000
|
+
dvc_data_status_before.pop("git", None) # Remove git status
|
|
1000
1001
|
if targets is None:
|
|
1001
1002
|
targets = []
|
|
1002
1003
|
args = deepcopy(targets)
|
|
@@ -1855,3 +1856,51 @@ def run_jupyter(
|
|
|
1855
1856
|
"""Run a command with the Jupyter CLI."""
|
|
1856
1857
|
process = subprocess.run([sys.executable, "-m", "jupyter"] + sys.argv[2:])
|
|
1857
1858
|
sys.exit(process.returncode)
|
|
1859
|
+
|
|
1860
|
+
|
|
1861
|
+
@app.command(name="latexmk")
|
|
1862
|
+
def run_latexmk(
|
|
1863
|
+
tex_file: Annotated[str, typer.Argument(help="The .tex file to compile.")],
|
|
1864
|
+
environment: Annotated[
|
|
1865
|
+
str | None,
|
|
1866
|
+
typer.Option(
|
|
1867
|
+
"--env",
|
|
1868
|
+
"-e",
|
|
1869
|
+
help=("Environment in which to run latexmk, if applicable."),
|
|
1870
|
+
),
|
|
1871
|
+
] = None,
|
|
1872
|
+
):
|
|
1873
|
+
"""Compile a LaTeX document with latexmk.
|
|
1874
|
+
|
|
1875
|
+
If a Calkit environment is not specified, latexmk will be run in the
|
|
1876
|
+
system environment if available. If not available, a TeX Live Docker
|
|
1877
|
+
container will be used.
|
|
1878
|
+
"""
|
|
1879
|
+
latexmk_cmd = [
|
|
1880
|
+
"latexmk",
|
|
1881
|
+
"-pdf",
|
|
1882
|
+
"-cd",
|
|
1883
|
+
"-silent",
|
|
1884
|
+
"-synctex=1",
|
|
1885
|
+
"-interaction=nonstopmode",
|
|
1886
|
+
tex_file,
|
|
1887
|
+
]
|
|
1888
|
+
if environment is not None:
|
|
1889
|
+
cmd = ["calkit", "xenv", "--name", environment] + latexmk_cmd
|
|
1890
|
+
elif calkit.check_dep_exists("latexmk"):
|
|
1891
|
+
cmd = latexmk_cmd
|
|
1892
|
+
else:
|
|
1893
|
+
cmd = [
|
|
1894
|
+
"docker",
|
|
1895
|
+
"run",
|
|
1896
|
+
"--rm",
|
|
1897
|
+
"-v",
|
|
1898
|
+
f"{os.getcwd()}:/work",
|
|
1899
|
+
"-w",
|
|
1900
|
+
"/work",
|
|
1901
|
+
"texlive/texlive:latest-full",
|
|
1902
|
+
] + latexmk_cmd
|
|
1903
|
+
try:
|
|
1904
|
+
subprocess.check_call(cmd)
|
|
1905
|
+
except subprocess.CalledProcessError:
|
|
1906
|
+
raise_error("latexmk failed")
|
|
@@ -157,10 +157,16 @@ def new_project(
|
|
|
157
157
|
repo.git.commit(["-m", "Initialize DVC"])
|
|
158
158
|
ck_info_fpath = os.path.join(abs_path, "calkit.yaml")
|
|
159
159
|
if os.path.isfile(ck_info_fpath) and not overwrite:
|
|
160
|
-
|
|
160
|
+
ck_info = calkit.load_calkit_info(wdir=abs_path)
|
|
161
|
+
name = ck_info.get("name", name)
|
|
162
|
+
title = ck_info.get("title", title)
|
|
163
|
+
description = ck_info.get("description", description)
|
|
164
|
+
typer.echo(
|
|
161
165
|
"Destination is already a Calkit project; "
|
|
162
|
-
"use
|
|
166
|
+
"will use existing project info where possible"
|
|
163
167
|
)
|
|
168
|
+
else:
|
|
169
|
+
ck_info = {}
|
|
164
170
|
if os.path.isdir(abs_path) and os.listdir(abs_path) and repo is None:
|
|
165
171
|
warn(f"{abs_path} is not empty")
|
|
166
172
|
if name is None:
|
|
@@ -224,7 +230,29 @@ def new_project(
|
|
|
224
230
|
)
|
|
225
231
|
repo.git.add("calkit.yaml")
|
|
226
232
|
if not no_commit:
|
|
227
|
-
repo.git.commit(
|
|
233
|
+
repo.git.commit(
|
|
234
|
+
["calkit.yaml", "-m", "Create calkit.yaml"]
|
|
235
|
+
)
|
|
236
|
+
else:
|
|
237
|
+
# Merge with existing project info in calkit.yaml
|
|
238
|
+
typer.echo("Updating existing calkit.yaml file")
|
|
239
|
+
ck_info = (
|
|
240
|
+
dict(
|
|
241
|
+
owner=resp["owner_account_name"],
|
|
242
|
+
name=resp["name"],
|
|
243
|
+
title=resp["title"],
|
|
244
|
+
description=resp["description"],
|
|
245
|
+
git_repo_url=resp["git_repo_url"],
|
|
246
|
+
)
|
|
247
|
+
| ck_info
|
|
248
|
+
)
|
|
249
|
+
with open(calkit_fpath, "w") as f:
|
|
250
|
+
ryaml.dump(ck_info, f)
|
|
251
|
+
repo.git.add("calkit.yaml")
|
|
252
|
+
if not no_commit and repo.git.diff("--staged"):
|
|
253
|
+
repo.git.commit(
|
|
254
|
+
["calkit.yaml", "-m", "Update calkit.yaml"]
|
|
255
|
+
)
|
|
228
256
|
try:
|
|
229
257
|
calkit.dvc.configure_remote(wdir=abs_path)
|
|
230
258
|
except Exception:
|
|
@@ -163,6 +163,17 @@ def execute_notebook(
|
|
|
163
163
|
),
|
|
164
164
|
),
|
|
165
165
|
] = None,
|
|
166
|
+
language: Annotated[
|
|
167
|
+
str,
|
|
168
|
+
typer.Option(
|
|
169
|
+
"--language",
|
|
170
|
+
"-l",
|
|
171
|
+
help=(
|
|
172
|
+
"Notebook language; if 'matlab', MATLAB kernel must be "
|
|
173
|
+
"available in environment."
|
|
174
|
+
),
|
|
175
|
+
),
|
|
176
|
+
] = "python",
|
|
166
177
|
verbose: Annotated[
|
|
167
178
|
bool, typer.Option("--verbose", "-v", help="Print verbose output.")
|
|
168
179
|
] = False,
|
|
@@ -176,10 +187,24 @@ def execute_notebook(
|
|
|
176
187
|
|
|
177
188
|
if os.path.isabs(path):
|
|
178
189
|
raise ValueError("Path must be relative")
|
|
190
|
+
if language.lower() not in ["python", "matlab", "julia"]:
|
|
191
|
+
raise ValueError(
|
|
192
|
+
"Language must be one of 'python', 'matlab', or 'julia'"
|
|
193
|
+
)
|
|
179
194
|
# First, ensure the specified environment has a kernel we can use
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
195
|
+
# We need to check the environment type and create the kernel if needed
|
|
196
|
+
if language.lower() == "python":
|
|
197
|
+
kernel_name = check_env_kernel(
|
|
198
|
+
env_name=env_name, no_check=no_check, verbose=verbose
|
|
199
|
+
)
|
|
200
|
+
elif language.lower() == "matlab":
|
|
201
|
+
kernel_name = "jupyter_matlab_kernel"
|
|
202
|
+
else:
|
|
203
|
+
raise_error(f"Language '{language}' not yet supported")
|
|
204
|
+
# We can't handle parameters unless language is Python
|
|
205
|
+
if language.lower() != "python":
|
|
206
|
+
if params or params_json is not None or params_base64 is not None:
|
|
207
|
+
raise_error("Parameters can only be passed to Python notebooks")
|
|
183
208
|
# Parse parameters
|
|
184
209
|
if params:
|
|
185
210
|
try:
|
|
@@ -214,14 +239,34 @@ def execute_notebook(
|
|
|
214
239
|
typer.echo(f"Using kernel: {kernel_name}")
|
|
215
240
|
typer.echo(f"Running with cwd: {notebook_dir}")
|
|
216
241
|
typer.echo(f"Output will be saved to: {fpath_out_exec}")
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
|
|
242
|
+
# If this is a Python or Julia notebook, we can use Papermill
|
|
243
|
+
# If it's a MATLAB notebook, we need to use the MATLAB kernel inside the
|
|
244
|
+
# specified environment
|
|
245
|
+
if language.lower() in ["python", "julia"]:
|
|
246
|
+
papermill.execute_notebook(
|
|
247
|
+
input_path=path,
|
|
248
|
+
output_path=fpath_out_exec,
|
|
249
|
+
kernel_name=kernel_name,
|
|
250
|
+
log_output=True,
|
|
251
|
+
parameters=parsed_params,
|
|
252
|
+
cwd=notebook_dir,
|
|
253
|
+
)
|
|
254
|
+
elif language.lower() == "matlab":
|
|
255
|
+
# Use nbconvert to execute the notebook with the MATLAB kernel
|
|
256
|
+
cmd = [
|
|
257
|
+
"python",
|
|
258
|
+
"-m",
|
|
259
|
+
"jupyter",
|
|
260
|
+
"nbconvert",
|
|
261
|
+
"--to",
|
|
262
|
+
"notebook",
|
|
263
|
+
"--execute",
|
|
264
|
+
f"--ExecutePreprocessor.kernel_name={kernel_name}",
|
|
265
|
+
"--output",
|
|
266
|
+
fpath_out_exec,
|
|
267
|
+
path,
|
|
268
|
+
]
|
|
269
|
+
run_in_env(cmd, env_name=env_name, no_check=no_check, verbose=verbose)
|
|
225
270
|
for to_fmt in to:
|
|
226
271
|
if to_fmt != "notebook":
|
|
227
272
|
try:
|
|
@@ -250,6 +295,6 @@ def execute_notebook(
|
|
|
250
295
|
fname_out,
|
|
251
296
|
]
|
|
252
297
|
typer.echo(f"Exporting {to_fmt}")
|
|
253
|
-
|
|
254
|
-
|
|
255
|
-
|
|
298
|
+
p = subprocess.run(cmd)
|
|
299
|
+
if p.returncode != 0:
|
|
300
|
+
raise_error(f"nbconvert failed for format '{to_fmt}'")
|
|
@@ -18,7 +18,7 @@ logger = logging.getLogger(__package__)
|
|
|
18
18
|
logger.setLevel(logging.INFO)
|
|
19
19
|
|
|
20
20
|
|
|
21
|
-
def configure_remote(wdir: str = None):
|
|
21
|
+
def configure_remote(wdir: str | None = None):
|
|
22
22
|
try:
|
|
23
23
|
project_name = calkit.detect_project_name(wdir=wdir)
|
|
24
24
|
except ValueError as e:
|
|
@@ -70,7 +70,9 @@ def configure_remote(wdir: str = None):
|
|
|
70
70
|
|
|
71
71
|
|
|
72
72
|
def set_remote_auth(
|
|
73
|
-
remote_name: str
|
|
73
|
+
remote_name: str | None = None,
|
|
74
|
+
always_auth: bool = False,
|
|
75
|
+
wdir: str | None = None,
|
|
74
76
|
):
|
|
75
77
|
"""Get a token and set it in the local DVC config so we can interact with
|
|
76
78
|
the cloud as an HTTP remote.
|
|
@@ -85,7 +87,7 @@ def set_remote_auth(
|
|
|
85
87
|
)["access_token"]
|
|
86
88
|
settings.dvc_token = token
|
|
87
89
|
settings.write()
|
|
88
|
-
subprocess.
|
|
90
|
+
p1 = subprocess.run(
|
|
89
91
|
[
|
|
90
92
|
sys.executable,
|
|
91
93
|
"-m",
|
|
@@ -99,7 +101,7 @@ def set_remote_auth(
|
|
|
99
101
|
],
|
|
100
102
|
cwd=wdir,
|
|
101
103
|
)
|
|
102
|
-
subprocess.
|
|
104
|
+
p2 = subprocess.run(
|
|
103
105
|
[
|
|
104
106
|
sys.executable,
|
|
105
107
|
"-m",
|
|
@@ -113,6 +115,10 @@ def set_remote_auth(
|
|
|
113
115
|
],
|
|
114
116
|
cwd=wdir,
|
|
115
117
|
)
|
|
118
|
+
if p1.returncode != 0 or p2.returncode != 0:
|
|
119
|
+
raise RuntimeError(
|
|
120
|
+
f"Failed to set DVC remote authentication for {remote_name}"
|
|
121
|
+
)
|
|
116
122
|
|
|
117
123
|
|
|
118
124
|
def add_external_remote(owner_name: str, project_name: str) -> dict:
|
|
@@ -159,13 +165,15 @@ def get_remotes(wdir: str | None = None) -> dict[str, str]:
|
|
|
159
165
|
"""Get a dictionary of DVC remotes, keyed by name, with URL as the
|
|
160
166
|
value.
|
|
161
167
|
"""
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
.strip()
|
|
168
|
+
p = subprocess.run(
|
|
169
|
+
[sys.executable, "-m", "dvc", "remote", "list"],
|
|
170
|
+
cwd=wdir,
|
|
171
|
+
capture_output=True,
|
|
172
|
+
text=True,
|
|
168
173
|
)
|
|
174
|
+
if p.returncode != 0:
|
|
175
|
+
raise RuntimeError(f"Error getting DVC remotes: {p.stderr.strip()}")
|
|
176
|
+
out = p.stdout.strip()
|
|
169
177
|
if not out:
|
|
170
178
|
return {}
|
|
171
179
|
resp = {}
|
|
@@ -182,12 +190,14 @@ def get_remotes(wdir: str | None = None) -> dict[str, str]:
|
|
|
182
190
|
return resp
|
|
183
191
|
|
|
184
192
|
|
|
185
|
-
def list_paths(wdir: str = None, recursive=False) -> list[str]:
|
|
193
|
+
def list_paths(wdir: str | None = None, recursive=False) -> list[str]:
|
|
186
194
|
"""List paths tracked with DVC."""
|
|
187
|
-
return [
|
|
195
|
+
return [
|
|
196
|
+
p.get("path", "") for p in list_files(wdir=wdir, recursive=recursive)
|
|
197
|
+
]
|
|
188
198
|
|
|
189
199
|
|
|
190
|
-
def list_files(wdir: str = None, recursive=True) -> list[dict]:
|
|
200
|
+
def list_files(wdir: str | None = None, recursive=True) -> list[dict]:
|
|
191
201
|
"""Return a list with all files in DVC, including their path and md5
|
|
192
202
|
checksum.
|
|
193
203
|
"""
|
|
@@ -113,6 +113,7 @@ class Stage(BaseModel):
|
|
|
113
113
|
"python-script",
|
|
114
114
|
"latex",
|
|
115
115
|
"matlab-script",
|
|
116
|
+
"matlab-command",
|
|
116
117
|
"docker-command",
|
|
117
118
|
"shell-command",
|
|
118
119
|
"shell-script",
|
|
@@ -164,7 +165,9 @@ class Stage(BaseModel):
|
|
|
164
165
|
|
|
165
166
|
@property
|
|
166
167
|
def xenv_cmd(self) -> str:
|
|
167
|
-
|
|
168
|
+
if self.environment == "_system":
|
|
169
|
+
return ""
|
|
170
|
+
return f"calkit xenv -n {self.environment} --no-check --"
|
|
168
171
|
|
|
169
172
|
def to_dvc(self) -> dict:
|
|
170
173
|
"""Convert to a DVC stage.
|
|
@@ -193,7 +196,7 @@ class PythonScriptStage(Stage):
|
|
|
193
196
|
|
|
194
197
|
@property
|
|
195
198
|
def dvc_cmd(self) -> str:
|
|
196
|
-
cmd = f"{self.xenv_cmd}
|
|
199
|
+
cmd = f"{self.xenv_cmd} python {self.script_path}"
|
|
197
200
|
for arg in self.args:
|
|
198
201
|
cmd += f" {arg}"
|
|
199
202
|
return cmd
|
|
@@ -212,7 +215,7 @@ class LatexStage(Stage):
|
|
|
212
215
|
|
|
213
216
|
@property
|
|
214
217
|
def dvc_cmd(self) -> str:
|
|
215
|
-
cmd = f"{self.xenv_cmd}
|
|
218
|
+
cmd = f"{self.xenv_cmd} latexmk -cd -interaction=nonstopmode"
|
|
216
219
|
if not self.verbose:
|
|
217
220
|
cmd += " -silent"
|
|
218
221
|
if self.force:
|
|
@@ -247,7 +250,26 @@ class MatlabScriptStage(Stage):
|
|
|
247
250
|
|
|
248
251
|
@property
|
|
249
252
|
def dvc_cmd(self) -> str:
|
|
250
|
-
|
|
253
|
+
cmd = self.xenv_cmd
|
|
254
|
+
if self.environment == "_system":
|
|
255
|
+
cmd += "matlab -batch"
|
|
256
|
+
cmd += f" \"run('{self.script_path}');\""
|
|
257
|
+
return cmd
|
|
258
|
+
|
|
259
|
+
|
|
260
|
+
class MatlabCommandStage(Stage):
|
|
261
|
+
kind: Literal["matlab-command"] = "matlab-command"
|
|
262
|
+
command: str
|
|
263
|
+
|
|
264
|
+
@property
|
|
265
|
+
def dvc_cmd(self) -> str:
|
|
266
|
+
# We need to escape quotes in the command
|
|
267
|
+
matlab_cmd = self.command.replace('"', '\\"')
|
|
268
|
+
cmd = self.xenv_cmd
|
|
269
|
+
if self.environment == "_system":
|
|
270
|
+
cmd += "matlab -batch"
|
|
271
|
+
cmd += f' "{matlab_cmd}"'
|
|
272
|
+
return cmd
|
|
251
273
|
|
|
252
274
|
|
|
253
275
|
class ShellCommandStage(Stage):
|
|
@@ -257,14 +279,12 @@ class ShellCommandStage(Stage):
|
|
|
257
279
|
|
|
258
280
|
@property
|
|
259
281
|
def dvc_cmd(self) -> str:
|
|
260
|
-
cmd =
|
|
261
|
-
if self.environment != "_system":
|
|
262
|
-
cmd = f"{self.xenv_cmd} -- "
|
|
282
|
+
cmd = self.xenv_cmd
|
|
263
283
|
if self.shell == "zsh":
|
|
264
284
|
norc_args = "-f"
|
|
265
285
|
else:
|
|
266
286
|
norc_args = "--noprofile --norc"
|
|
267
|
-
cmd += f'{self.shell} {norc_args} -c "{self.command}"'
|
|
287
|
+
cmd += f' {self.shell} {norc_args} -c "{self.command}"'
|
|
268
288
|
return cmd
|
|
269
289
|
|
|
270
290
|
|
|
@@ -280,14 +300,12 @@ class ShellScriptStage(Stage):
|
|
|
280
300
|
|
|
281
301
|
@property
|
|
282
302
|
def dvc_cmd(self) -> str:
|
|
283
|
-
cmd =
|
|
284
|
-
if self.environment != "_system":
|
|
285
|
-
cmd = f"{self.xenv_cmd} -- "
|
|
303
|
+
cmd = self.xenv_cmd
|
|
286
304
|
if self.shell == "zsh":
|
|
287
305
|
norc_args = "-f"
|
|
288
306
|
else:
|
|
289
307
|
norc_args = "--noprofile --norc"
|
|
290
|
-
cmd += f"{self.shell} {norc_args} {self.script_path}"
|
|
308
|
+
cmd += f" {self.shell} {norc_args} {self.script_path}"
|
|
291
309
|
for arg in self.args:
|
|
292
310
|
cmd += f" {arg}"
|
|
293
311
|
return cmd
|
|
@@ -327,7 +345,7 @@ class JuliaScriptStage(Stage):
|
|
|
327
345
|
|
|
328
346
|
@property
|
|
329
347
|
def dvc_cmd(self) -> str:
|
|
330
|
-
cmd = f'{self.xenv_cmd}
|
|
348
|
+
cmd = f'{self.xenv_cmd} "include(\\"{self.script_path}\\")"'
|
|
331
349
|
return cmd
|
|
332
350
|
|
|
333
351
|
@property
|
|
@@ -343,7 +361,7 @@ class JuliaCommandStage(Stage):
|
|
|
343
361
|
def dvc_cmd(self) -> str:
|
|
344
362
|
# We need to escape quotes in the command
|
|
345
363
|
julia_cmd = self.command.replace('"', '\\"')
|
|
346
|
-
cmd = f'{self.xenv_cmd}
|
|
364
|
+
cmd = f'{self.xenv_cmd} "{julia_cmd}"'
|
|
347
365
|
return cmd
|
|
348
366
|
|
|
349
367
|
|
|
@@ -373,6 +391,7 @@ class JupyterNotebookStage(Stage):
|
|
|
373
391
|
executed_ipynb_storage: Literal["git", "dvc"] | None = "dvc"
|
|
374
392
|
html_storage: Literal["git", "dvc"] | None = "dvc"
|
|
375
393
|
parameters: dict[str, Any] = {}
|
|
394
|
+
language: Literal["python", "matlab", "julia"] = "python"
|
|
376
395
|
|
|
377
396
|
def update_parameters(self, params: dict) -> None:
|
|
378
397
|
"""If we have any templated parameters, update those, e.g., from
|
|
@@ -430,7 +449,10 @@ class JupyterNotebookStage(Stage):
|
|
|
430
449
|
|
|
431
450
|
@property
|
|
432
451
|
def dvc_cmd(self) -> str:
|
|
433
|
-
cmd =
|
|
452
|
+
cmd = (
|
|
453
|
+
f"calkit nb execute --environment {self.environment} "
|
|
454
|
+
f"--no-check --language {self.language}"
|
|
455
|
+
)
|
|
434
456
|
if self.html_storage:
|
|
435
457
|
cmd += " --to html"
|
|
436
458
|
if self.parameters:
|
|
@@ -533,6 +555,7 @@ class Pipeline(BaseModel):
|
|
|
533
555
|
PythonScriptStage
|
|
534
556
|
| LatexStage
|
|
535
557
|
| MatlabScriptStage
|
|
558
|
+
| MatlabCommandStage
|
|
536
559
|
| ShellCommandStage
|
|
537
560
|
| ShellScriptStage
|
|
538
561
|
| DockerCommandStage
|
|
@@ -7,6 +7,7 @@ from calkit.models.pipeline import (
|
|
|
7
7
|
JuliaCommandStage,
|
|
8
8
|
JupyterNotebookStage,
|
|
9
9
|
LatexStage,
|
|
10
|
+
MatlabCommandStage,
|
|
10
11
|
PythonScriptStage,
|
|
11
12
|
StageIteration,
|
|
12
13
|
WordToPdfStage,
|
|
@@ -131,3 +132,18 @@ def test_juliacommandstage():
|
|
|
131
132
|
assert sd["cmd"] == (
|
|
132
133
|
'calkit xenv -n j1 --no-check -- "println(\\"sup\\")"'
|
|
133
134
|
)
|
|
135
|
+
|
|
136
|
+
|
|
137
|
+
def test_matlabcommandstage():
|
|
138
|
+
s = MatlabCommandStage(environment="m1", command='disp("Hello, MATLAB!");')
|
|
139
|
+
sd = s.to_dvc()
|
|
140
|
+
print(sd)
|
|
141
|
+
assert sd["cmd"] == (
|
|
142
|
+
'calkit xenv -n m1 --no-check -- "disp(\\"Hello, MATLAB!\\");"'
|
|
143
|
+
)
|
|
144
|
+
s = MatlabCommandStage(
|
|
145
|
+
environment="_system", command='disp("Hello, MATLAB!");'
|
|
146
|
+
)
|
|
147
|
+
sd = s.to_dvc()
|
|
148
|
+
print(sd)
|
|
149
|
+
assert sd["cmd"] == 'matlab -batch "disp(\\"Hello, MATLAB!\\");"'
|
|
@@ -25,7 +25,7 @@ If you're using Conda for environment management,
|
|
|
25
25
|
e.g., with an `environment.yml` file,
|
|
26
26
|
you can use the `calkit new conda-env` command.
|
|
27
27
|
|
|
28
|
-
Next, we can start building our [pipeline](pipeline.md).
|
|
28
|
+
Next, we can start building our [pipeline](pipeline/index.md).
|
|
29
29
|
Let's say we have a Jupyter notebook called `collect-data.ipynb`
|
|
30
30
|
that produces raw data at `data/raw.h5`.
|
|
31
31
|
We can add a pipeline stage to run this notebook in the `main` environment
|
|
@@ -0,0 +1,83 @@
|
|
|
1
|
+
# Running Calkit in GitHub Actions
|
|
2
|
+
|
|
3
|
+
A project can be set up to automatically run the pipeline every time a
|
|
4
|
+
change is pushed, either to the main branch or on a pull request,
|
|
5
|
+
using GitHub Actions.
|
|
6
|
+
The latter allows for inspection of outputs before merging into main.
|
|
7
|
+
|
|
8
|
+
To get started, generate a DVC token and set it in your GitHub Actions
|
|
9
|
+
secrets as `CALKIT_DVC_TOKEN`,
|
|
10
|
+
either at your account or project level.
|
|
11
|
+
On calkit.io, there are shortcuts on the project page for managing both
|
|
12
|
+
Calkit tokens and GitHub Actions secrets:
|
|
13
|
+
|
|
14
|
+

|
|
15
|
+
|
|
16
|
+

|
|
17
|
+
|
|
18
|
+

|
|
19
|
+
|
|
20
|
+
Next, add a workflow to the project in the `.github/workflows` folder.
|
|
21
|
+
For example, we can put the content below into `.github/workflows/run.yml`:
|
|
22
|
+
|
|
23
|
+
```yaml
|
|
24
|
+
name: Run pipeline
|
|
25
|
+
|
|
26
|
+
on:
|
|
27
|
+
push:
|
|
28
|
+
branches:
|
|
29
|
+
- main
|
|
30
|
+
pull_request:
|
|
31
|
+
|
|
32
|
+
permissions:
|
|
33
|
+
contents: write
|
|
34
|
+
|
|
35
|
+
# Make sure we only ever run one per branch so we don't have issues pushing
|
|
36
|
+
# after running the pipeline
|
|
37
|
+
concurrency:
|
|
38
|
+
group: calkit-run-${{ github.ref }}
|
|
39
|
+
cancel-in-progress: false
|
|
40
|
+
|
|
41
|
+
jobs:
|
|
42
|
+
main:
|
|
43
|
+
name: Run
|
|
44
|
+
runs-on: ubuntu-latest
|
|
45
|
+
steps:
|
|
46
|
+
- uses: actions/checkout@v4
|
|
47
|
+
with:
|
|
48
|
+
# For PRs, checkout the head ref to avoid detached HEAD
|
|
49
|
+
ref: ${{ github.head_ref || github.ref_name }}
|
|
50
|
+
token: ${{ secrets.GITHUB_TOKEN }}
|
|
51
|
+
- name: Configure Git credentials
|
|
52
|
+
run: |
|
|
53
|
+
git config user.name github-actions[bot]
|
|
54
|
+
git config user.email 41898282+github-actions[bot]@users.noreply.github.com
|
|
55
|
+
- name: Setup uv
|
|
56
|
+
uses: astral-sh/setup-uv@v5
|
|
57
|
+
- name: Install Calkit
|
|
58
|
+
run: uv tool install calkit-python
|
|
59
|
+
- name: Run Calkit
|
|
60
|
+
uses: calkit/run-action@v1
|
|
61
|
+
with:
|
|
62
|
+
dvc_token: ${{ secrets.CALKIT_DVC_TOKEN }}
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
This particular example installs Calkit with uv,
|
|
66
|
+
meaning `uv` and `uv-venv` environment types will work without any additional
|
|
67
|
+
configuration.
|
|
68
|
+
Docker is also installed by default on the `ubuntu-latest` machine.
|
|
69
|
+
If other environment types are used in the project,
|
|
70
|
+
setup steps may be necessary for those, e.g.,
|
|
71
|
+
[`setup-miniconda`](https://github.com/marketplace/actions/setup-miniconda)
|
|
72
|
+
or [`install-juliaup`](https://github.com/marketplace/actions/install-juliaup).
|
|
73
|
+
|
|
74
|
+
By default, the Calkit GitHub Action will run the pipeline and save results.
|
|
75
|
+
This is why the workflow needs write permissions and configures Git
|
|
76
|
+
credentials to act as the GitHub Actions bot.
|
|
77
|
+
The workflow also limits concurrency so multiple jobs don't attempt to push
|
|
78
|
+
to the same branch at the same time.
|
|
79
|
+
|
|
80
|
+
It's possible to configure the action to not save results, e.g., if you
|
|
81
|
+
just want to check that the pipeline can run without errors.
|
|
82
|
+
See the [documentation](https://github.com/marketplace/actions/run-calkit)
|
|
83
|
+
for all available options.
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
@@ -10,3 +10,4 @@
|
|
|
10
10
|
- [Adding a new LaTeX-based publication with its own Docker build environment](adding-latex-pub-docker.md)
|
|
11
11
|
- [A reproducible workflow using Microsoft Office (Word and Excel)](office.md)
|
|
12
12
|
- [Using Calkit with MATLAB](matlab.md)
|
|
13
|
+
- [Automation with GitHub Actions](github-actions.md)
|