calkit-python 0.5.0__tar.gz → 0.6.1__tar.gz

{calkit_python-0.5.0 → calkit_python-0.6.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: calkit-python
-Version: 0.5.0
+Version: 0.6.1
 Summary: Reproducibility simplified.
 Project-URL: Homepage, https://github.com/calkit/calkit
 Project-URL: Issues, https://github.com/calkit/calkit/issues
@@ -31,19 +31,39 @@ Description-Content-Type: text/markdown
 
 # Calkit
 
-[Calkit](https://calkit.io) helps simplify reproducibility,
-acting as a layer on top of
-[Git](https://git-scm.com/), [DVC](https://dvc.org/),
-[Docker](https://docker.com),
-and adds a domain-specific data model
+Calkit is a lightweight framework for doing reproducible research.
+It acts as a top-level layer to integrate and simplify the use of enabling
+technologies such as
+[Git](https://git-scm.com/),
+[DVC](https://dvc.org/),
+[Conda](https://docs.conda.io/en/latest/),
+and [Docker](https://docker.com).
+Calkit also adds a domain-specific data model
 such that all aspects of the research process can be fully described in a
 single repository and therefore easily consumed by others.
 
+Our goal is to make reproducibility easier so it becomes more common.
+To do this, we try to make it easy for users to follow two simple rules:
+
+1. **Keep everything in version control.** This includes large files like
+   datasets, enabled by DVC. The [Calkit cloud](https://calkit.io)
+   serves as a simple default DVC remote storage location for those who do not
+   want to manage their own infrastructure.
+2. **Generate all important artifacts with a single pipeline.** There should be
+   no special instructions required to reproduce a project's artifacts.
+   It should be as simple as calling `calkit run`.
+   The DVC pipeline (in a project's `dvc.yaml` file) is therefore the main
+   thing to "build" throughout a research project.
+   Calkit provides helper functionality to build pipeline stages that
+   keep computational environments up-to-date and label their outputs for
+   convenient reuse.
+
 ## Tutorials
 
+- [Keeping track of conda environments](docs/tutorials/conda-envs.md)
 - [Defining and executing manual procedures](docs/tutorials/procedures.md)
 - [Adding a new LaTeX-based publication with its own Docker build environment](docs/tutorials/adding-latex-pub-docker.md)
-- [A reproducibly workflow using Microsoft Office (Word and Excel)](https://petebachant.me/office-repro/)
+- [A reproducible workflow using Microsoft Office (Word and Excel)](https://petebachant.me/office-repro/)
 - [Reproducible OpenFOAM simulations](https://petebachant.me/reproducible-openfoam/)
 
 ## Why does reproducibility matter?

{calkit_python-0.5.0 → calkit_python-0.6.1}/README.md

@@ -1,18 +1,38 @@
 # Calkit
 
-[Calkit](https://calkit.io) helps simplify reproducibility,
-acting as a layer on top of
-[Git](https://git-scm.com/), [DVC](https://dvc.org/),
-[Docker](https://docker.com),
-and adds a domain-specific data model
+Calkit is a lightweight framework for doing reproducible research.
+It acts as a top-level layer to integrate and simplify the use of enabling
+technologies such as
+[Git](https://git-scm.com/),
+[DVC](https://dvc.org/),
+[Conda](https://docs.conda.io/en/latest/),
+and [Docker](https://docker.com).
+Calkit also adds a domain-specific data model
 such that all aspects of the research process can be fully described in a
 single repository and therefore easily consumed by others.
 
+Our goal is to make reproducibility easier so it becomes more common.
+To do this, we try to make it easy for users to follow two simple rules:
+
+1. **Keep everything in version control.** This includes large files like
+   datasets, enabled by DVC. The [Calkit cloud](https://calkit.io)
+   serves as a simple default DVC remote storage location for those who do not
+   want to manage their own infrastructure.
+2. **Generate all important artifacts with a single pipeline.** There should be
+   no special instructions required to reproduce a project's artifacts.
+   It should be as simple as calling `calkit run`.
+   The DVC pipeline (in a project's `dvc.yaml` file) is therefore the main
+   thing to "build" throughout a research project.
+   Calkit provides helper functionality to build pipeline stages that
+   keep computational environments up-to-date and label their outputs for
+   convenient reuse.
+
 ## Tutorials
 
+- [Keeping track of conda environments](docs/tutorials/conda-envs.md)
 - [Defining and executing manual procedures](docs/tutorials/procedures.md)
 - [Adding a new LaTeX-based publication with its own Docker build environment](docs/tutorials/adding-latex-pub-docker.md)
-- [A reproducibly workflow using Microsoft Office (Word and Excel)](https://petebachant.me/office-repro/)
+- [A reproducible workflow using Microsoft Office (Word and Excel)](https://petebachant.me/office-repro/)
 - [Reproducible OpenFOAM simulations](https://petebachant.me/reproducible-openfoam/)
 
 ## Why does reproducibility matter?

{calkit_python-0.5.0 → calkit_python-0.6.1}/calkit/__init__.py

@@ -1,4 +1,4 @@
-__version__ = "0.5.0"
+__version__ = "0.6.1"
 
 from .core import *
 from . import git
@@ -9,3 +9,4 @@ from . import config
 from . import models
 from . import office
 from . import templates
+from . import conda

{calkit_python-0.5.0 → calkit_python-0.6.1}/calkit/cli/main.py

@@ -553,7 +553,9 @@ def run_in_env(
             typer.echo(f"Running command: {docker_cmd}")
         subprocess.call(docker_cmd, cwd=wdir)
     elif env["kind"] == "conda":
-        cmd = ["conda", "run", "-n", env_name] + cmd
+        with open(env["path"]) as f:
+            conda_env = calkit.ryaml.load(f)
+        cmd = ["conda", "run", "-n", conda_env["name"]] + cmd
         if verbose:
             typer.echo(f"Running command: {cmd}")
         subprocess.call(cmd, cwd=wdir)
@@ -784,3 +786,32 @@ def run_procedure(
             )
         if step.wait_after_s:
             wait(step.wait_after_s)
+
+
+@app.command(
+    name="check-conda-env",
+    help="Check a conda environment and rebuild if necessary.",
+)
+def check_conda_env(
+    env_fpath: Annotated[
+        str,
+        typer.Option(
+            "--file", "-f", help="Path to conda environment YAML file."
+        ),
+    ] = "environment.yml",
+    output_fpath: Annotated[
+        str,
+        typer.Option(
+            "--output",
+            "-o",
+            help=(
+                "Path to which existing environment should be exported. "
+                "If not specified, will have the same filename with '-loc' "
+                "appended to it, keeping the same extension."
+            ),
+        ),
+    ] = None,
+):
+    calkit.conda.check_env(
+        env_fpath=env_fpath, output_fpath=output_fpath, log_func=typer.echo
+    )

calkit_python-0.6.1/calkit/conda.py

@@ -0,0 +1,162 @@
+"""Functionality for working with conda environments."""
+
+import json
+import os
+import subprocess
+
+import calkit
+from calkit import ryaml
+
+
+def check_env(
+    env_fpath: str = "environment.yml",
+    use_mamba=True,
+    log_func=None,
+    output_fpath: str = None,
+):
+    """Check that a conda environment matches its spec.
+
+    If it doesn't match, recreate it.
+
+    Note that this only works with exact or no version specification.
+    Using greater than and less than operators is not supported.
+    """
+    if log_func is None:
+        log_func = calkit.logger.info
+    log_func(f"Checking conda env defined in {env_fpath}")
+    envs = json.loads(
+        subprocess.check_output(["conda", "env", "list", "--json"]).decode()
+    )["envs"]
+    # Get existing env names, but skip the base environment
+    existing_env_names = [env.split("/")[-1] for env in envs[1:]]
+    with open(env_fpath) as f:
+        env_spec = ryaml.load(f)
+    env_name = env_spec["name"]
+    env_check_fpath = os.path.join(
+        os.path.expanduser("~"),
+        ".calkit",
+        "conda-env-checks",
+        env_name + ".yml",
+    )
+    env_check_dir = os.path.dirname(env_check_fpath)
+    os.makedirs(env_check_dir, exist_ok=True)
+    conda = "mamba" if use_mamba else "conda"
+    # Check if env even exists
+    if env_name not in existing_env_names:
+        log_func(f"Environment {env_name} doesn't exist; creating")
+        # Environment doesn't exist, so create it
+        subprocess.check_call([conda, "env", "create", "-f", env_fpath])
+        env_needs_rebuild = False
+        env_needs_export = True
+    else:
+        env_needs_export = False
+        # Environment does exist, so check it
+        if os.path.isfile(env_check_fpath):
+            # Open up the env check result file
+            with open(env_check_fpath) as f:
+                env_check = ryaml.load(f)
+            # Check the prefix mtime saved to that file against the actual
+            # prefix mtime
+            # If they match, the environment saved in env_check is still
+            # valid, so we don't need to re-export
+            existing_mtime = env_check["mtime"]
+            current_mtime = os.path.getmtime(env_check["prefix"])
+            env_needs_export = existing_mtime != current_mtime
+        else:
+            env_needs_export = True
+        if env_needs_export:
+            log_func(f"Exporting existing env to {env_check_fpath}")
+            env_check = json.loads(
+                subprocess.check_output(
+                    [
+                        "conda",
+                        "env",
+                        "export",
+                        "-n",
+                        env_name,
+                        "--no-builds",
+                        "--json",
+                    ]
+                ).decode()
+            )
+            env_check["mtime"] = os.path.getmtime(env_check["prefix"])
+            with open(env_check_fpath, "w") as f:
+                ryaml.dump(env_check, f)
+        # Determine if the env matches
+        env_needs_rebuild = False
+        existing_conda_deps = env_check["dependencies"][:-1]
+        existing_pip_deps = env_check["dependencies"][-1]["pip"]
+        required_conda_deps = env_spec["dependencies"][:-1]
+        required_pip_deps = env_spec["dependencies"][-1]["pip"]
+        log_func("Checking conda dependencies")
+        for dep in required_conda_deps:
+            dep_split = dep.split("=")
+            package = dep_split[0]
+            if len(dep_split) > 1:
+                version = dep_split[1]
+            else:
+                version = None
+            if version is not None and dep not in existing_conda_deps:
+                log_func(f"Found missing dependency: {dep}")
+                env_needs_rebuild = True
+                break
+            elif version is None:
+                if package not in [
+                    d.split("=")[0] for d in existing_conda_deps
+                ]:
+                    log_func(f"Found missing dependency: {dep}")
+                    env_needs_rebuild = True
+                    break
+        if not env_needs_rebuild:
+            log_func("Checking pip dependencies")
+            for dep in required_pip_deps:
+                dep_split = dep.split("==")
+                package = dep_split[0]
+                if len(dep_split) > 1:
+                    version = dep_split[1]
+                else:
+                    version = None
+                if version is not None and dep not in existing_pip_deps:
+                    env_needs_rebuild = True
+                    log_func(f"Found missing dependency: {dep}")
+                    break
+                elif version is None:
+                    if package not in [
+                        d.split("==")[0] for d in existing_pip_deps
+                    ]:
+                        log_func(f"Found missing dependency: {dep}")
+                        env_needs_rebuild = True
+                        break
+    if env_needs_rebuild:
+        log_func(f"Rebuilding {env_name} since it does not match spec")
+        subprocess.check_call([conda, "env", "create", "-y", "-f", env_fpath])
+        env_needs_export = True
+    else:
+        log_func(f"Environment {env_name} matches spec")
+    # If the env was rebuilt, export the env check
+    if env_needs_export:
+        log_func(f"Exporting existing env to {env_check_fpath}")
+        env_check = json.loads(
+            subprocess.check_output(
+                [
+                    "conda",
+                    "env",
+                    "export",
+                    "-n",
+                    env_name,
+                    "--no-builds",
+                    "--json",
+                ]
+            ).decode()
+        )
+        env_check["mtime"] = os.path.getmtime(env_check["prefix"])
+        with open(env_check_fpath, "w") as f:
+            ryaml.dump(env_check, f)
+    if output_fpath is None:
+        fname, ext = os.path.splitext(env_fpath)
+        output_fpath = fname + "-lock" + ext
+    log_func(f"Exporting lock file to {output_fpath}")
+    with open(output_fpath, "w") as f:
+        _ = env_check.pop("mtime")
+        _ = env_check.pop("prefix")
+        ryaml.dump(env_check, f)

calkit_python-0.6.1/docs/tutorials/conda-envs.md

@@ -0,0 +1,64 @@
+# Keeping track of conda environments
+
+It can be difficult to know if a conda environment present on your machine
+matches one in your project's `environment.yml` file.
+You may be collaborating with a team on a project and someone adds a
+dependency, then all of a sudden things won't run on your
+machine.
+Or maybe you use multiple machines to run the same project.
+
+Calkit has a feature to make working with conda environments more
+reproducible,
+without needing to rebuild the environment all the time.
+If you're working on a project with a conda `environment.yml` file,
+you can simply run:
+
+```sh
+calkit check-conda-env
+```
+
+and the environment on your local machine will be rebuilt if it doesn't
+match the spec,
+or it will be created if it doesn't exist.
+Note that this will delete the existing environment and rebuild from scratch,
+so make sure you don't have any unsaved changes in there.
+Also note that for some combinations of `pip` dependencies,
+it may not be possible to arrive at an environment that matches the spec,
+so it is recommended to only put the "top-level" dependencies in
+`environment.yml` rather than a full export.
+
+We can also add an environment check to our DVC pipeline
+so if we're running any stages with that environment, we make sure
+the environment is correct before doing so.
+For example, we could have the following in `dvc.yaml`:
+
+```yaml
+stages:
+  check-env:
+    cmd: calkit check-conda-env
+    deps:
+      - environment.yml
+    outs:
+      - environment-lock.yml:
+          cache: false
+    always_changed: true
+  run-my-script:
+    cmd: conda run -n my-env python my-script.py
+    deps:
+      - my-script.py
+      - environment-lock.yml
+```
+
+In the example above, we use the `always_changed` option so the conda env
+will be checked in every call to `calkit run` or `dvc repro`.
+If the output file `environment-lock.yml` changes,
+DVC will rerun the `run-my-script` stage.
+With the pipeline setup this way,
+our collaborators (or ourselves on other computers)
+can simply call `calkit run` without needing to think about
+getting our conda environment into the correct state beforehand.
+
+Note that this pattern can also be expanded to projects that use multiple
+conda environments.
+For example, if an environment spec is saved to `env-2.yml`,
+we can call `calkit check-conda-env -f env-2.yml`.