calkit-python 0.3.3__tar.gz → 0.5.0__tar.gz

{calkit_python-0.3.3 → calkit_python-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: calkit-python
-Version: 0.3.3
+Version: 0.5.0
 Summary: Reproducibility simplified.
 Project-URL: Homepage, https://github.com/calkit/calkit
 Project-URL: Issues, https://github.com/calkit/calkit/issues
@@ -31,21 +31,25 @@ Description-Content-Type: text/markdown
 
 # Calkit
 
-[Calkit](https://calkit.io) simplifies reproducibility,
+[Calkit](https://calkit.io) helps simplify reproducibility,
 acting as a layer on top of
 [Git](https://git-scm.com/), [DVC](https://dvc.org/),
-[Docker](https://docker.com), and more,
-such that all all aspects of the research process can be fully described in a
-single repository.
+[Docker](https://docker.com),
+and adds a domain-specific data model
+such that all aspects of the research process can be fully described in a
+single repository and therefore easily consumed by others.
 
 ## Tutorials
 
+- [Defining and executing manual procedures](docs/tutorials/procedures.md)
+- [Adding a new LaTeX-based publication with its own Docker build environment](docs/tutorials/adding-latex-pub-docker.md)
+- [A reproducibly workflow using Microsoft Office (Word and Excel)](https://petebachant.me/office-repro/)
 - [Reproducible OpenFOAM simulations](https://petebachant.me/reproducible-openfoam/)
 
 ## Why does reproducibility matter?
 
 If your work is reproducible, that means that someone else can "run" it and
-get the same results or outputs.
+calculate the same results or outputs.
 This is a major step towards addressing
 [the replication crisis](https://en.wikipedia.org/wiki/Replication_crisis)
 and has some major benefits for both you as an individual and the research
@@ -63,7 +67,8 @@ community:
 ## Why another tool/platform?
 
 Git, GitHub, DVC, Docker et al. are amazing tools/platforms, but their
-use involves multiple fairly difficult learning curves.
+use involves multiple fairly difficult learning curves,
+and tying them together might mean developing something new for each project.
 Our goal is to provide a single tool and platform to unify all of these so
 that there is a single, gentle learning curve.
 However, it is not our goal to hide or replace these underlying components.
@@ -152,7 +157,7 @@ other researchers can find and reuse your work to accelerate their own.
    own, like a figure, or for an intermediate result that is expensive to
    generate.
 1. There should be the smallest number of
-   frequently used commands as possible, and they should require at little
+   frequently used commands as possible, and they should require as little
    memorization as possible to know how to execute, e.g., a user should be
    able to keep running `calkit run` and that's all they really need to do
    to make sure the project is up-to-date.

{calkit_python-0.3.3 → calkit_python-0.5.0}/README.md

@@ -1,20 +1,24 @@
 # Calkit
 
-[Calkit](https://calkit.io) simplifies reproducibility,
+[Calkit](https://calkit.io) helps simplify reproducibility,
 acting as a layer on top of
 [Git](https://git-scm.com/), [DVC](https://dvc.org/),
-[Docker](https://docker.com), and more,
-such that all all aspects of the research process can be fully described in a
-single repository.
+[Docker](https://docker.com),
+and adds a domain-specific data model
+such that all aspects of the research process can be fully described in a
+single repository and therefore easily consumed by others.
 
 ## Tutorials
 
+- [Defining and executing manual procedures](docs/tutorials/procedures.md)
+- [Adding a new LaTeX-based publication with its own Docker build environment](docs/tutorials/adding-latex-pub-docker.md)
+- [A reproducibly workflow using Microsoft Office (Word and Excel)](https://petebachant.me/office-repro/)
 - [Reproducible OpenFOAM simulations](https://petebachant.me/reproducible-openfoam/)
 
 ## Why does reproducibility matter?
 
 If your work is reproducible, that means that someone else can "run" it and
-get the same results or outputs.
+calculate the same results or outputs.
 This is a major step towards addressing
 [the replication crisis](https://en.wikipedia.org/wiki/Replication_crisis)
 and has some major benefits for both you as an individual and the research
@@ -32,7 +36,8 @@ community:
 ## Why another tool/platform?
 
 Git, GitHub, DVC, Docker et al. are amazing tools/platforms, but their
-use involves multiple fairly difficult learning curves.
+use involves multiple fairly difficult learning curves,
+and tying them together might mean developing something new for each project.
 Our goal is to provide a single tool and platform to unify all of these so
 that there is a single, gentle learning curve.
 However, it is not our goal to hide or replace these underlying components.
@@ -121,7 +126,7 @@ other researchers can find and reuse your work to accelerate their own.
    own, like a figure, or for an intermediate result that is expensive to
    generate.
 1. There should be the smallest number of
-   frequently used commands as possible, and they should require at little
+   frequently used commands as possible, and they should require as little
    memorization as possible to know how to execute, e.g., a user should be
    able to keep running `calkit run` and that's all they really need to do
    to make sure the project is up-to-date.

{calkit_python-0.3.3 → calkit_python-0.5.0}/calkit/__init__.py

@@ -1,4 +1,4 @@
-__version__ = "0.3.3"
+__version__ = "0.5.0"
 
 from .core import *
 from . import git
@@ -8,3 +8,4 @@ from . import jupyter
 from . import config
 from . import models
 from . import office
+from . import templates

{calkit_python-0.3.3 → calkit_python-0.5.0}/calkit/cli/core.py

@@ -24,5 +24,5 @@ def run_cmd(cmd: list[str]):
 
 
 def raise_error(txt):
-    typer.echo(txt, err=txt)
+    typer.echo(typer.style(txt, fg="red"), err=txt)
     raise typer.Exit(1)

{calkit_python-0.3.3 → calkit_python-0.5.0}/calkit/cli/list.py

@@ -77,3 +77,17 @@ def list_environments():
         typer.echo(name + ":")
         for k, v in env.items():
             typer.echo(f"    {k}: {v}")
+
+
+@list_app.command(name="templates")
+def list_templates():
+    for kind, tpl_dict in calkit.templates.TEMPLATES.items():
+        for name in tpl_dict:
+            typer.echo(f"{kind}/{name}")
+
+
+@list_app.command(name="procedures")
+def list_procedures():
+    ck_info = calkit.load_calkit_info()
+    for p in ck_info.get("procedures", {}):
+        typer.echo(p)

{calkit_python-0.3.3 → calkit_python-0.5.0}/calkit/cli/main.py

@@ -2,11 +2,14 @@
 
 from __future__ import annotations
 
+import csv
 import hashlib
 import json
 import os
 import subprocess
 import sys
+import time
+from datetime import UTC, datetime, timedelta
 
 import git
 import typer
@@ -20,6 +23,7 @@ from calkit.cli.list import list_app
 from calkit.cli.new import new_app
 from calkit.cli.notebooks import notebooks_app
 from calkit.cli.office import office_app
+from calkit.models import Procedure
 
 app = typer.Typer(
     invoke_without_command=True,
@@ -481,11 +485,21 @@ def run_in_env(
             ),
         ),
     ] = None,
+    wdir: Annotated[
+        str,
+        typer.Option(
+            "--wdir",
+            help=(
+                "Working directory. "
+                "By default will run current working directory."
+            ),
+        ),
+    ] = None,
     verbose: Annotated[
         bool, typer.Option("--verbose", "-v", help="Print verbose output.")
     ] = False,
 ):
-    ck_info = calkit.load_calkit_info()
+    ck_info = calkit.load_calkit_info(process_includes="environments")
     envs = ck_info.get("environments", {})
     if not envs:
         raise_error("No environments defined in calkit.yaml")
@@ -507,33 +521,42 @@ def run_in_env(
     if env_name is None:
         raise_error("Environment must be specified if there are multiple")
     env = envs[env_name]
-    cwd = os.getcwd()
+    if wdir is not None:
+        cwd = os.path.abspath(wdir)
+    else:
+        cwd = os.getcwd()
     image_name = env.get("image", env_name)
-    wdir = env.get("wdir", "/work")
+    docker_wdir = env.get("wdir", "/work")
+    shell = env.get("shell", "sh")
+    platform = env.get("platform")
     if env["kind"] == "docker":
-        cmd = " ".join(cmd)
-        cmd = [
+        shell_cmd = " ".join(cmd)
+        docker_cmd = [
             "docker",
             "run",
+        ]
+        if platform:
+            docker_cmd += ["--platform", platform]
+        docker_cmd += [
             "-it" if sys.stdin.isatty() else "-i",
             "--rm",
             "-w",
-            wdir,
+            docker_wdir,
             "-v",
-            f"{cwd}:{wdir}",
+            f"{cwd}:{docker_wdir}",
             image_name,
-            "bash",
+            shell,
             "-c",
-            f"{cmd}",
+            f"{shell_cmd}",
         ]
         if verbose:
-            typer.echo(f"Running command: {cmd}")
-        subprocess.call(cmd)
+            typer.echo(f"Running command: {docker_cmd}")
+        subprocess.call(docker_cmd, cwd=wdir)
     elif env["kind"] == "conda":
         cmd = ["conda", "run", "-n", env_name] + cmd
         if verbose:
             typer.echo(f"Running command: {cmd}")
-        subprocess.call(cmd)
+        subprocess.call(cmd, cwd=wdir)
     else:
         raise_error("Environment kind not supported")
 
@@ -621,3 +644,143 @@ def build_docker(
     inspect[0]["DockerfileMD5"] = dockerfile_md5
     with open(lock_fpath, "w") as f:
         json.dump(inspect, f, indent=4)
+
+
+@app.command(name="runproc", help="Run or execute a procedure.")
+def run_procedure(
+    name: Annotated[str, typer.Argument(help="The name of the procedure.")],
+    no_commit: Annotated[
+        bool,
+        typer.Option("--no-commit", help="Do not commit after each action."),
+    ] = False,
+):
+    def wait(seconds):
+        typer.echo(f"Wait {seconds} seconds")
+        dt = 0.1
+        while seconds >= 0:
+            mins, secs = divmod(seconds, 60)
+            mins, secs = int(mins), int(secs)
+            out = f"Time left: {mins:02d}:{secs:02d}\r"
+            typer.echo(out, nl=False)
+            time.sleep(dt)
+            seconds -= dt
+        typer.echo()
+
+    def convert_value(value, dtype):
+        if dtype == "int":
+            return int(value)
+        elif dtype == "float":
+            return float(value)
+        elif dtype == "str":
+            return str(value)
+        elif dtype == "bool":
+            return bool(value)
+        return value
+
+    ck_info = calkit.load_calkit_info(process_includes="procedures")
+    procs = ck_info.get("procedures", {})
+    if name not in procs:
+        raise_error(f"'{name}' is not defined as a procedure")
+    try:
+        proc = Procedure.model_validate(procs[name])
+    except Exception as e:
+        raise_error(f"Procedure '{name}' is invalid: {e}")
+    git_repo = git.Repo()
+    # Check to make sure the working tree is clean, so we know we ran the
+    # committed version of the procedure
+    git_status = git_repo.git.status()
+    if not "working tree clean" in git_status:
+        raise_error(
+            f"Cannot execute procedures unless repo is clean:\n\n{git_status}"
+        )
+    t_start_overall = calkit.utcnow()
+    # Formulate headers for CSV file, which must contain all inputs from all
+    # steps
+    headers = [
+        "calkit_version",
+        "procedure_name",
+        "step",
+        "start",
+        "end",
+    ]
+    for step in proc.steps:
+        if step.inputs:
+            for iname in step.inputs:
+                if iname not in headers:
+                    headers.append(iname)
+    # TODO: Add ability to process periodic logic
+    # See if now falls between start and end, and if there is a run with a
+    # timestamp corresponding to the period in which now falls
+    # If so, exit
+    # If not, continue
+    # Create empty CSV if one doesn't exist
+    t_start_overall_str = t_start_overall.isoformat(timespec="seconds")
+    fpath = f".calkit/procedure-runs/{name}/{t_start_overall_str}.csv"
+    dirname = os.path.dirname(fpath)
+    if not os.path.isdir(dirname):
+        os.makedirs(dirname)
+    if not os.path.isfile(fpath):
+        with open(fpath, "w") as f:
+            csv.writer(f).writerow(headers)
+    for n, step in enumerate(proc.steps):
+        typer.echo(f"Starting step {n}")
+        t_start = calkit.utcnow()
+        if step.wait_before_s:
+            wait(step.wait_before_s)
+        # Execute the step
+        inputs = step.inputs
+        input_vals = {}
+        if not inputs:
+            input(f"{step.summary} and press enter when complete: ")
+        else:
+            typer.echo(step.summary)
+            for input_name, i in inputs.items():
+                msg = f"Enter {input_name}"
+                if i.units:
+                    msg += f" ({i.units})"
+                msg += " and press enter: "
+                success = False
+                while not success:
+                    val = input(msg)
+                    if i.dtype:
+                        try:
+                            val = convert_value(val, i.dtype)
+                            success = True
+                        except ValueError:
+                            typer.echo(
+                                typer.style(
+                                    f"Invalid {i.dtype} value", fg="red"
+                                )
+                            )
+                    else:
+                        success = True
+                input_vals[input_name] = val
+        t_end = calkit.utcnow()
+        # Log step completion
+        row = (
+            dict(
+                procedure_name=name,
+                step=n,
+                calkit_version=calkit.__version__,
+                start=t_start.isoformat(),
+                end=t_end.isoformat(),
+            )
+            | input_vals
+        )
+        row = {k: row.get(k, "") for k in headers}
+        # Log this row to CSV
+        with open(fpath, "a") as f:
+            csv.writer(f).writerow(row.values())
+        typer.echo(f"Logged step {n} to {fpath}")
+        if not no_commit:
+            typer.echo("Committing to Git repo")
+            git_repo.git.reset()
+            git_repo.git.add(fpath)
+            git_repo.git.commit(
+                [
+                    "-m",
+                    f"Execute procedure {name} step {n}",
+                ]
+            )
+        if step.wait_after_s:
+            wait(step.wait_after_s)

{calkit_python-0.3.3 → calkit_python-0.5.0}/calkit/cli/new.py

@@ -3,6 +3,7 @@
 from __future__ import annotations
 
 import os
+import shutil
 import subprocess
 
 import git
@@ -472,3 +473,189 @@ def new_dataset(
             repo.git.add("dvc.yaml")
         if repo.git.diff("--staged"):
             repo.git.commit(["-m", f"Add dataset {path}"])
+
+
+@new_app.command(name="publication", help="Create a new publication.")
+def new_publication(
+    path: Annotated[
+        str,
+        typer.Argument(
+            help=(
+                "Path for the publication. "
+                "If using a template, this could be a directory."
+            )
+        ),
+    ],
+    title: Annotated[
+        str, typer.Option("--title", help="The title of the publication.")
+    ],
+    description: Annotated[
+        str,
+        typer.Option(
+            "--description", help="A description of the publication."
+        ),
+    ],
+    kind: Annotated[
+        str,
+        typer.Option(
+            "--kind", help="Kind of the publication, e.g., 'journal-article'."
+        ),
+    ],
+    stage_name: Annotated[
+        str,
+        typer.Option(
+            "--stage",
+            help="Name of the pipeline stage to build the output file.",
+        ),
+    ] = None,
+    deps: Annotated[
+        list[str], typer.Option("--dep", help="Path to stage dependency.")
+    ] = [],
+    outs_from_stage: Annotated[
+        str,
+        typer.Option(
+            "--deps-from-stage-outs",
+            help="Stage name from which to add outputs as dependencies.",
+        ),
+    ] = None,
+    template: Annotated[
+        str,
+        typer.Option(
+            "--template",
+            help=(
+                "Template with which to create the source files. "
+                "Should be in the format {type}/{name}."
+            ),
+        ),
+    ] = None,
+    env_name: Annotated[
+        str,
+        typer.Option(
+            "--environment",
+            help="Name of the build environment to create, if desired.",
+        ),
+    ] = None,
+    no_commit: Annotated[
+        bool,
+        typer.Option(
+            "--no-commit", help="Do not commit resulting changes to the repo."
+        ),
+    ] = False,
+    overwrite: Annotated[
+        bool,
+        typer.Option(
+            "--overwrite",
+            "-f",
+            help="Overwrite existing objects if they already exist.",
+        ),
+    ] = False,
+):
+    ck_info = calkit.load_calkit_info(process_includes=False)
+    pubs = ck_info.get("publications", [])
+    envs = ck_info.get("environments", {})
+    pub_paths = [p.get("path") for p in pubs]
+    if template is not None:
+        template_type, _ = template.split("/")
+    else:
+        template_type = None
+    # Check all of our inputs
+    if template_type not in ["latex"]:
+        raise_error(f"Unknown template type '{template_type}'")
+    if env_name is not None and template_type != "latex":
+        raise_error("Environments can only be created for latex templates")
+    if env_name is not None and env_name in envs and not overwrite:
+        raise_error(f"Environment '{env_name}' already exists")
+    if template_type is not None:
+        try:
+            template_obj = calkit.templates.get_template(template)
+        except ValueError:
+            raise_error(f"Template '{template}' does not exist")
+    # Parse outs from stage if specified
+    if outs_from_stage:
+        pipeline = calkit.dvc.read_pipeline()
+        stages = pipeline.get("stages", {})
+        if outs_from_stage not in stages:
+            raise_error(f"Stage {outs_from_stage} does not exist")
+        stage = stages[outs_from_stage]
+        if "foreach" in stage:
+            for val in stage["foreach"]:
+                for out in stage.get("do", {}).get("outs", []):
+                    deps.append(out.replace("${item}", val))
+        else:
+            deps += stage.get("outs", [])
+    # Create publication object
+    if template_type == "latex":
+        pub_fpath = os.path.join(
+            path, template_obj.target.removesuffix(".tex") + ".pdf"
+        )
+    else:
+        pub_fpath = path
+    if not overwrite and pub_fpath in pub_paths:
+        raise_error(f"Publication with path {pub_fpath} already exists")
+    elif overwrite and pub_fpath in pub_paths:
+        pubs = [p for p in pubs if p.get("path") != pub_fpath]
+    pub = dict(
+        path=pub_fpath,
+        kind=kind,
+        title=title,
+        description=description,
+        stage=stage_name,
+    )
+    pubs.append(pub)
+    ck_info["publications"] = pubs
+    repo = git.Repo()
+    # Create environment if applicable
+    if env_name is not None and template_type == "latex":
+        env_path = f".calkit/environments/{env_name}.yaml"
+        os.makedirs(".calkit/environments", exist_ok=True)
+        env = {"_include": env_path}
+        envs[env_name] = env
+        env_remote = dict(
+            kind="docker",
+            image="kjarosh/latex:2024.4",
+            description="TeXlive full from kjarosh.",
+            platform="linux/amd64",
+        )
+        with open(env_path, "w") as f:
+            calkit.ryaml.dump(env_remote, f)
+        ck_info["environments"] = envs
+        repo.git.add(env_path)
+    with open("calkit.yaml", "w") as f:
+        calkit.ryaml.dump(ck_info, f)
+    repo.git.add("calkit.yaml")
+    # Copy in template files if applicable
+    if template_type == "latex":
+        if overwrite and os.path.exists(path):
+            shutil.rmtree(path)
+        calkit.templates.use_template(
+            name=template, dest_dir=path, title=title
+        )
+        repo.git.add(path)
+    # Create stage if applicable
+    if stage_name is not None and template_type == "latex":
+        cmd = f"cd {path} && latexmk -pdf {template_obj.target}"
+        if env_name is not None:
+            cmd = f'calkit runenv -n {env_name} "{cmd}"'
+        target_dep = os.path.join(path, template_obj.target)
+        dvc_cmd = [
+            "dvc",
+            "stage",
+            "add",
+            "-n",
+            stage_name,
+            "-o",
+            pub_fpath,
+            "-d",
+            target_dep,
+        ]
+        if env_name is not None:
+            dvc_cmd += ["-d", env_path]
+        for dep in deps:
+            dvc_cmd += ["-d", dep]
+        if overwrite:
+            dvc_cmd.append("-f")
+        dvc_cmd.append(cmd)
+        subprocess.check_call(dvc_cmd)
+        repo.git.add("dvc.yaml")
+    if not no_commit and repo.git.diff("--staged"):
+        repo.git.commit(["-m", f"Add new publication {pub_fpath}"])

calkit_python-0.5.0/calkit/core.py

@@ -0,0 +1,98 @@
+"""Core functionality."""
+
+from __future__ import annotations
+
+import glob
+import logging
+import os
+from datetime import UTC, datetime
+
+import ruamel.yaml
+from git import Repo
+from git.exc import InvalidGitRepositoryError
+
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__package__)
+
+ryaml = ruamel.yaml.YAML()
+ryaml.indent(mapping=2, sequence=4, offset=2)
+ryaml.preserve_quotes = True
+ryaml.width = 70
+
+
+def find_project_dirs(relative=False, max_depth=3) -> list[str]:
+    """Find all Calkit project directories."""
+    if relative:
+        start = ""
+    else:
+        start = os.path.expanduser("~")
+    res = []
+    for i in range(max_depth):
+        pattern = os.path.join(start, *["*"] * (i + 1), "calkit.yaml")
+        res += glob.glob(pattern)
+        # Check GitHub documents for users who use GitHub Desktop
+        pattern = os.path.join(
+            start, "*", "GitHub", *["*"] * (i + 1), "calkit.yaml"
+        )
+        res += glob.glob(pattern)
+    final_res = []
+    for ck_fpath in res:
+        path = os.path.dirname(ck_fpath)
+        # Make sure this path is a Git repo
+        try:
+            Repo(path)
+        except InvalidGitRepositoryError:
+            continue
+        final_res.append(path)
+    return final_res
+
+
+def load_calkit_info(
+    wdir=None, process_includes: bool | str | list[str] = False
+) -> dict:
+    """Load Calkit project information.
+
+    Parameters
+    ----------
+    wdir : str
+        Working directory. Defaults to current working directory.
+    process_includes: bool, string or list of strings
+        Whether or not to process any '_include' keys for a given kind of
+        object. If a string is passed, only process includes for that kind.
+        Similarly, if a list of strings is passed, only process those kinds.
+        If True, process all default kinds.
+    """
+    info = {}
+    fpath = "calkit.yaml"
+    if wdir is not None:
+        fpath = os.path.join(wdir, fpath)
+    if os.path.isfile(fpath):
+        with open(fpath) as f:
+            info = ryaml.load(f)
+    # Check for any includes, i.e., entities with an _include key, for which
+    # we should merge in another file
+    default_includes_enabled = ["environments", "procedures"]
+    if process_includes:
+        if isinstance(process_includes, bool):
+            includes_enabled = default_includes_enabled
+        elif isinstance(process_includes, str):
+            includes_enabled = [process_includes]
+        elif isinstance(process_includes, list):
+            includes_enabled = process_includes
+        for kind in includes_enabled:
+            if kind in info:
+                for obj_name, obj in info[kind].items():
+                    if "_include" in obj:
+                        include_fpath = obj.pop("_include")
+                        with open(include_fpath) as f:
+                            include_data = ryaml.load(f)
+                        info[kind][obj_name] |= include_data
+    return info
+
+
+def utcnow(remove_tz=True) -> datetime:
+    """Return now in UTC, optionally stripping timezone information."""
+    dt = datetime.now(UTC)
+    if remove_tz:
+        dt = dt.replace(tzinfo=None)
+    return dt