calkit-python 0.37.1__tar.gz → 0.37.3__tar.gz

{calkit_python-0.37.1 → calkit_python-0.37.3}/Makefile

@@ -37,15 +37,12 @@ test-docs: sync-docs ## Test if documentation can be built without warnings or e
 	@uv run mkdocs build -s
 
 .PHONY: sync-docs
-sync-docs: cli-reference ## Sync documentation content from docs/*.md into README.md.
+sync-docs: ## Sync documentation content from docs/*.md into README.md.
+	@echo "🚀 Generating docs references"
+	@uv run python scripts/generate-docs-references.py
 	@echo "🚀 Syncing documentation"
 	@uv run python scripts/sync-docs.py
 
-.PHONY: cli-reference
-cli-reference: ## Generate docs/cli-reference.md from CLI help output.
-	@echo "🚀 Generating CLI reference"
-	@uv run python scripts/generate-cli-reference.py
-
 .PHONY: docs
 docs: sync-docs ## Build and serve the documentation.
 	@uv run mkdocs serve --livereload

{calkit_python-0.37.1 → calkit_python-0.37.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: calkit-python
-Version: 0.37.1
+Version: 0.37.3
 Summary: Reproducibility simplified.
 Project-URL: Homepage, https://calkit.org
 Project-URL: Issues, https://github.com/calkit/calkit/issues
@@ -76,56 +76,67 @@ Description-Content-Type: text/markdown
   </a>
 </p>
 
-Calkit makes it easy to create
-["single button"](https://doi.org/10.1190/1.1822162)
-reproducible research projects.
-
-Instead of a loosely related collection of files
-and manual instructions,
-turn your project into a version-controlled,
-self-contained "calculation kit,"
-tying together all phases or stages of the project:
-data collection, analysis, visualization, and writing,
-each of which can make use of the latest and greatest computational
-tools and languages.
-In other words, you, your collaborators, and readers will be able to go
-from raw data to research article with a single command,
-improving efficiency via faster iteration cycle time,
-reducing the likelihood of mistakes,
-and allowing others to more effectively build upon your work.
-
-Calkit makes this level of automation possible without extensive software
-engineering expertise by providing a project framework and toolset that unifies
-and simplifies the use of powerful enabling technologies like Git,
-DVC, Conda, Docker, and more,
-while guiding users away from common reproducibility pitfalls.
+<!-- INCLUDE: docs/index.md -->
+
+Calkit helps you manage and automate research projects like a software
+engineer.
+
+Define computational environments,
+steps that process your data, create figures,
+presentations, and publications, connect to external tools,
+then iterate quickly and painlessly until your research questions are
+answered, tracking changes to all files along the way.
+At the end, deliver your entire project as a self-contained, self-documenting,
+version-controlled, and
+[single button reproducible](https://doi.org/10.1190/1.1822162)
+"calculation kit" so you and others can easily verify
+and build upon the results.
+
+## Guiding principles
+
+- Quality comes from iteration. Automation reduces the time and effort
+  needed to iterate, thereby increasing iteration and quality.
+- Automating a step can and should take roughly
+  the same amount of time as doing it once manually,
+  therefore it's almost always worth it.
+- Working in a "quick and dirty" way can easily become _not quick_ when
+  the dirtiness results in mistakes and/or discourages working in small steps.
 
 ## Features
 
-- A declarative pipeline that guides users to define an environment
-  for every stage, so long lists of instructions in a README and
-  "but it works on my machine" are things of the past.
-- A CLI to run the project's pipeline to verify it's reproducible,
-  regenerating outputs as needed and
-  ensuring all
-  computational environments
-  (e.g., [Conda](https://docs.conda.io/en/latest/),
-  [Docker](https://docker.com), uv, Julia)
-  match their specification.
-- A schema to store structured metadata describing the
-  project's important outputs (in its `calkit.yaml` file)
-  and how they are created
-  (its computational environments and pipeline).
-- A command line interface (CLI) to simplify keeping code, text, and larger
-  data files backed up in the same project repo using both
-  [Git](https://git-scm.com/) and [DVC](https://dvc.org/).
+- A simplified [version control](https://docs.calkit.org/version-control)
+  interface that unifies Git and DVC (Data Version Control),
+  so all materials can be kept in the same project repository.
+  This way, code doesn't need to be siloed away from other
+  important artifacts like datasets, models, figures, or article PDFs,
+  allowing you to work on all parts of a project without hopping around to
+  different tools.
+- [Computational environment management](https://docs.calkit.org/environments) with support for many
+  languages and environment managers: Conda, Docker, uv, Julia, Renv, and more.
+  No need to create and update environments on your own. Calkit will handle
+  them as needed.
+- An environment-aware build system or [pipeline](https://docs.calkit.org/pipeline) with
+  a simple declarative syntax and
+  output caching so you don't need to think about which steps or stages
+  need to be rerun after changing any part of the project.
+  Simply call `calkit run`.
+  Compose your pipeline from many different kinds of stages,
+  including simple scripts, commands, Jupyter Notebooks, LaTeX, and more.
 - A complementary self-hostable and GitHub-integrated
-  [cloud system](https://github.com/calkit/calkit-cloud)
+  [cloud platform](https://github.com/calkit/calkit-cloud)
   to facilitate backup, collaboration,
   and sharing throughout the entire research lifecycle.
-- [Overleaf integration](https://docs.calkit.org/overleaf/), so code,
-  data, and LaTeX documents can all live in the same repo and be part of a
-  single pipeline (no more manual uploads!)
+- [Overleaf integration](https://docs.calkit.org/overleaf/), so
+  analysis, visualization, and writing can all stay in sync
+  (no more manual uploads!)
+- Support for running on high performance computing (HPC) systems that use
+  [SLURM schedulers](https://docs.calkit.org/pipeline/slurm).
+- Support for running with [GitHub Actions](https://docs.calkit.org/tutorials/github-actions).
+- Extensions for doing all of the above graphically in
+  [JupyterLab](https://docs.calkit.org/jupyterlab) and
+  [VS Code](https://marketplace.visualstudio.com/items?itemName=Calkit.calkit-vscode).
+
+<!-- END INCLUDE -->
 
 ## Installation
 

{calkit_python-0.37.1 → calkit_python-0.37.3}/README.md

@@ -17,56 +17,67 @@
   </a>
 </p>
 
-Calkit makes it easy to create
-["single button"](https://doi.org/10.1190/1.1822162)
-reproducible research projects.
-
-Instead of a loosely related collection of files
-and manual instructions,
-turn your project into a version-controlled,
-self-contained "calculation kit,"
-tying together all phases or stages of the project:
-data collection, analysis, visualization, and writing,
-each of which can make use of the latest and greatest computational
-tools and languages.
-In other words, you, your collaborators, and readers will be able to go
-from raw data to research article with a single command,
-improving efficiency via faster iteration cycle time,
-reducing the likelihood of mistakes,
-and allowing others to more effectively build upon your work.
-
-Calkit makes this level of automation possible without extensive software
-engineering expertise by providing a project framework and toolset that unifies
-and simplifies the use of powerful enabling technologies like Git,
-DVC, Conda, Docker, and more,
-while guiding users away from common reproducibility pitfalls.
+<!-- INCLUDE: docs/index.md -->
+
+Calkit helps you manage and automate research projects like a software
+engineer.
+
+Define computational environments,
+steps that process your data, create figures,
+presentations, and publications, connect to external tools,
+then iterate quickly and painlessly until your research questions are
+answered, tracking changes to all files along the way.
+At the end, deliver your entire project as a self-contained, self-documenting,
+version-controlled, and
+[single button reproducible](https://doi.org/10.1190/1.1822162)
+"calculation kit" so you and others can easily verify
+and build upon the results.
+
+## Guiding principles
+
+- Quality comes from iteration. Automation reduces the time and effort
+  needed to iterate, thereby increasing iteration and quality.
+- Automating a step can and should take roughly
+  the same amount of time as doing it once manually,
+  therefore it's almost always worth it.
+- Working in a "quick and dirty" way can easily become _not quick_ when
+  the dirtiness results in mistakes and/or discourages working in small steps.
 
 ## Features
 
-- A declarative pipeline that guides users to define an environment
-  for every stage, so long lists of instructions in a README and
-  "but it works on my machine" are things of the past.
-- A CLI to run the project's pipeline to verify it's reproducible,
-  regenerating outputs as needed and
-  ensuring all
-  computational environments
-  (e.g., [Conda](https://docs.conda.io/en/latest/),
-  [Docker](https://docker.com), uv, Julia)
-  match their specification.
-- A schema to store structured metadata describing the
-  project's important outputs (in its `calkit.yaml` file)
-  and how they are created
-  (its computational environments and pipeline).
-- A command line interface (CLI) to simplify keeping code, text, and larger
-  data files backed up in the same project repo using both
-  [Git](https://git-scm.com/) and [DVC](https://dvc.org/).
+- A simplified [version control](https://docs.calkit.org/version-control)
+  interface that unifies Git and DVC (Data Version Control),
+  so all materials can be kept in the same project repository.
+  This way, code doesn't need to be siloed away from other
+  important artifacts like datasets, models, figures, or article PDFs,
+  allowing you to work on all parts of a project without hopping around to
+  different tools.
+- [Computational environment management](https://docs.calkit.org/environments) with support for many
+  languages and environment managers: Conda, Docker, uv, Julia, Renv, and more.
+  No need to create and update environments on your own. Calkit will handle
+  them as needed.
+- An environment-aware build system or [pipeline](https://docs.calkit.org/pipeline) with
+  a simple declarative syntax and
+  output caching so you don't need to think about which steps or stages
+  need to be rerun after changing any part of the project.
+  Simply call `calkit run`.
+  Compose your pipeline from many different kinds of stages,
+  including simple scripts, commands, Jupyter Notebooks, LaTeX, and more.
 - A complementary self-hostable and GitHub-integrated
-  [cloud system](https://github.com/calkit/calkit-cloud)
+  [cloud platform](https://github.com/calkit/calkit-cloud)
   to facilitate backup, collaboration,
   and sharing throughout the entire research lifecycle.
-- [Overleaf integration](https://docs.calkit.org/overleaf/), so code,
-  data, and LaTeX documents can all live in the same repo and be part of a
-  single pipeline (no more manual uploads!)
+- [Overleaf integration](https://docs.calkit.org/overleaf/), so
+  analysis, visualization, and writing can all stay in sync
+  (no more manual uploads!)
+- Support for running on high performance computing (HPC) systems that use
+  [SLURM schedulers](https://docs.calkit.org/pipeline/slurm).
+- Support for running with [GitHub Actions](https://docs.calkit.org/tutorials/github-actions).
+- Extensions for doing all of the above graphically in
+  [JupyterLab](https://docs.calkit.org/jupyterlab) and
+  [VS Code](https://marketplace.visualstudio.com/items?itemName=Calkit.calkit-vscode).
+
+<!-- END INCLUDE -->
 
 ## Installation
 

{calkit_python-0.37.1 → calkit_python-0.37.3}/calkit/cli/check.py

@@ -31,6 +31,217 @@ from calkit.environments import (
 check_app = typer.Typer(no_args_is_help=True)
 
 
+def _juliaup_version_installed(julia_version: str) -> bool:
+    """Return True if juliaup already has the given channel installed locally."""
+    try:
+        result = subprocess.run(
+            ["juliaup", "status"],
+            capture_output=True,
+            text=True,
+            check=False,
+            timeout=10,
+        )
+    except FileNotFoundError:
+        return False
+    except subprocess.TimeoutExpired:
+        warn("Timed out while running `juliaup status`.")
+        return False
+    if result.returncode != 0:
+        err_output = (result.stderr or result.stdout or "").strip()
+        if err_output:
+            warn(f"`juliaup status` failed: {err_output}")
+        else:
+            warn(
+                f"`juliaup status` failed with exit code {result.returncode}."
+            )
+        return False
+    for line in result.stdout.splitlines():
+        parts = line.split()
+        if not parts:
+            continue
+        # Columns: [Default?] Channel Version [Update?]
+        # Default column is either empty or "*", so channel is parts[0] or parts[1]
+        for col in parts[:2]:
+            if col == julia_version:
+                return True
+    return False
+
+
+def _check_julia_env(
+    env_path: str,
+    julia_version: str | None = None,
+    verbose: bool = False,
+    cache_key: str | None = None,
+) -> str:
+    """Check a Julia environment and instantiate only when needed."""
+    abs_env_path = os.path.abspath(env_path)
+    env_fname = os.path.basename(abs_env_path)
+    if env_fname != "Project.toml":
+        raise_error(
+            "Julia environments require a path pointing to Project.toml"
+        )
+    env_dir = os.path.dirname(abs_env_path) or "."
+    env_path_for_cache = os.path.basename(abs_env_path)
+    env = {
+        "kind": "julia",
+        "path": env_path_for_cache,
+        "julia": julia_version or "",
+    }
+    cache_env_name = cache_key or (
+        f"julia::{abs_env_path}::{julia_version or ''}"
+    )
+    if calkit.environments.check_cache(
+        env_name=cache_env_name,
+        env=env,
+        wdir=env_dir,
+        respect_ttl=False,
+    ):
+        if verbose:
+            typer.echo(
+                "Julia environment cache is valid; skipping Pkg.instantiate()"
+            )
+        lock_fpath = calkit.environments.get_env_lock_fpath(
+            env=env,
+            env_name=cache_env_name,
+            wdir=env_dir,
+            as_posix=False,
+        )
+        return lock_fpath or os.path.join(env_dir, "Manifest.toml")
+    if julia_version:
+        if shutil.which("juliaup") is not None:
+            if _juliaup_version_installed(julia_version):
+                if verbose:
+                    typer.echo(
+                        f"Julia {julia_version} is already installed; "
+                        "skipping juliaup add"
+                    )
+            else:
+                cmd = ["juliaup", "add", julia_version]
+                if verbose:
+                    typer.echo(f"Running command: {cmd}")
+                try:
+                    subprocess.run(cmd, check=True)
+                except subprocess.CalledProcessError:
+                    raise_error(
+                        f"Failed to install Julia version {julia_version}"
+                    )
+        else:
+            try:
+                compatible = calkit.julia.current_version_is_compatible(
+                    julia_version
+                )
+            except ValueError as e:
+                raise_error(str(e))
+            if not compatible:
+                raise_error(
+                    "Current Julia version is not compatible with required "
+                    f"version ({julia_version}), and juliaup is not "
+                    "available to install it"
+                )
+    deps_to_add: list[str] = []
+    try:
+        with open(abs_env_path, "r") as f:
+            content = f.read()
+        lines = [line.rstrip() for line in content.splitlines()]
+        deps_section = False
+        deps_found = False
+        for line in lines:
+            stripped = line.strip()
+            if stripped == "[deps]":
+                deps_section = True
+                continue
+            if deps_section:
+                if stripped.startswith("[") and stripped.endswith("]"):
+                    break
+                if stripped and not stripped.startswith("#"):
+                    if "=" in stripped:
+                        deps_found = True
+                        break
+        if not deps_found:
+            for idx, line in enumerate(lines):
+                marker = "# Dependencies (add with Julia's Pkg.add):"
+                if line.strip() == marker and idx + 1 < len(lines):
+                    dep_line = lines[idx + 1].strip()
+                    if dep_line.startswith("#"):
+                        dep_line = dep_line.lstrip("#").strip()
+                    deps_to_add = [
+                        dep.strip()
+                        for dep in dep_line.split(",")
+                        if dep.strip()
+                    ]
+                    break
+    except OSError:
+        deps_to_add = []
+    if deps_to_add:
+        pkg_list = ", ".join(f'"{dep}"' for dep in deps_to_add)
+        cmd = ["julia"]
+        if julia_version:
+            cmd.append(f"+{julia_version}")
+        cmd += [
+            f"--project={env_dir}",
+            "-e",
+            f"using Pkg; Pkg.add([{pkg_list}]);",
+        ]
+        if julia_version:
+            try:
+                cmd = calkit.julia.check_version_in_command(cmd)
+            except Exception as e:
+                raise_error(f"Failed to check Julia version: {e}")
+        cmd = calkit.julia.ensure_startup_file_disabled_in_command(cmd)
+        try:
+            subprocess.check_call(
+                cmd,
+                env=os.environ.copy() | {"JULIA_LOAD_PATH": "@:@stdlib"},
+            )
+        except subprocess.CalledProcessError:
+            calkit.environments.save_cache(
+                env_name=cache_env_name,
+                env=env,
+                wdir=env_dir,
+                success=False,
+            )
+            raise_error("Failed to add Julia dependencies")
+    cmd = ["julia"]
+    if julia_version:
+        cmd.append(f"+{julia_version}")
+    cmd += [
+        f"--project={env_dir}",
+        "-e",
+        "using Pkg; Pkg.instantiate();",
+    ]
+    if julia_version:
+        try:
+            cmd = calkit.julia.check_version_in_command(cmd)
+        except Exception as e:
+            raise_error(f"Failed to check Julia version: {e}")
+    cmd = calkit.julia.ensure_startup_file_disabled_in_command(cmd)
+    try:
+        subprocess.check_call(
+            cmd, env=os.environ.copy() | {"JULIA_LOAD_PATH": "@:@stdlib"}
+        )
+    except subprocess.CalledProcessError:
+        calkit.environments.save_cache(
+            env_name=cache_env_name,
+            env=env,
+            wdir=env_dir,
+            success=False,
+        )
+        raise_error("Failed to check julia environment")
+    calkit.environments.save_cache(
+        env_name=cache_env_name,
+        env=env,
+        wdir=env_dir,
+        success=True,
+    )
+    lock_fpath = calkit.environments.get_env_lock_fpath(
+        env=env,
+        env_name=cache_env_name,
+        wdir=env_dir,
+        as_posix=False,
+    )
+    return lock_fpath or os.path.join(env_dir, "Manifest.toml")
+
+
 @check_app.command(name="repro")
 def check_repro(
     wdir: Annotated[
@@ -190,116 +401,47 @@ def check_environment(
         julia_version = env.get("julia")
         if julia_version is None:
             raise_error("Julia environments require a Julia version")
-        env_fname = os.path.basename(env_path)
-        if not env_fname == "Project.toml":
-            raise_error(
-                "Julia environments require a path pointing to Project.toml"
-            )
-        # First ensure the Julia version exists
-        if shutil.which("juliaup") is not None:
-            cmd = ["juliaup", "add", julia_version]
-            if verbose:
-                typer.echo(f"Running command: {cmd}")
-            try:
-                subprocess.run(cmd, check=True)
-            except subprocess.CalledProcessError:
-                raise_error(f"Failed to install Julia version {julia_version}")
-        else:
-            try:
-                compatible = calkit.julia.current_version_is_compatible(
-                    julia_version
-                )
-            except ValueError as e:
-                raise_error(str(e))
-            if not compatible:
-                raise_error(
-                    f"Current Julia version is not compatible with required "
-                    f"version ({julia_version}), and juliaup is not available to "
-                    "install it"
-                )
-        env_dir = os.path.dirname(env_path)
-        if not env_dir:
-            env_dir = "."
-        # If auto-detection couldn't resolve UUIDs, Project.toml includes a
-        # commented dependency list
-        # In that case, add those packages with
-        # Pkg.add before instantiating so the env is usable at run time
-        deps_to_add: list[str] = []
-        try:
-            with open(env_path, "r") as f:
-                content = f.read()
-            lines = [line.rstrip() for line in content.splitlines()]
-            deps_section = False
-            deps_found = False
-            for line in lines:
-                stripped = line.strip()
-                if stripped == "[deps]":
-                    deps_section = True
-                    continue
-                if deps_section:
-                    if stripped.startswith("[") and stripped.endswith("]"):
-                        break
-                    if stripped and not stripped.startswith("#"):
-                        if "=" in stripped:
-                            deps_found = True
-                            break
-            if not deps_found:
-                for idx, line in enumerate(lines):
-                    marker = "# Dependencies (add with Julia's Pkg.add):"
-                    if line.strip() == marker and idx + 1 < len(lines):
-                        dep_line = lines[idx + 1].strip()
-                        if dep_line.startswith("#"):
-                            dep_line = dep_line.lstrip("#").strip()
-                        deps_to_add = [
-                            dep.strip()
-                            for dep in dep_line.split(",")
-                            if dep.strip()
-                        ]
-                        break
-        except OSError:
-            deps_to_add = []
-        if deps_to_add:
-            pkg_list = ", ".join(f'"{dep}"' for dep in deps_to_add)
-            cmd = [
-                "julia",
-                f"+{julia_version}",
-                f"--project={env_dir}",
-                "-e",
-                f"using Pkg; Pkg.add([{pkg_list}]);",
-            ]
-            try:
-                cmd = calkit.julia.check_version_in_command(cmd)
-            except Exception as e:
-                raise_error(f"Failed to check Julia version: {e}")
-            try:
-                subprocess.check_call(
-                    cmd,
-                    env=os.environ.copy() | {"JULIA_LOAD_PATH": "@:@stdlib"},
-                )
-            except subprocess.CalledProcessError:
-                raise_error("Failed to add Julia dependencies")
-        cmd = [
-            "julia",
-            f"+{julia_version}",
-            f"--project={env_dir}",
-            "-e",
-            "using Pkg; Pkg.instantiate();",
-        ]
-        try:
-            cmd = calkit.julia.check_version_in_command(cmd)
-        except Exception as e:
-            raise_error(f"Failed to check Julia version: {e}")
-        try:
-            subprocess.check_call(
-                cmd, env=os.environ.copy() | {"JULIA_LOAD_PATH": "@:@stdlib"}
-            )
-        except subprocess.CalledProcessError:
-            raise_error("Failed to check julia environment")
+        _check_julia_env(
+            env_path=env_path,
+            julia_version=julia_version,
+            verbose=verbose,
+            cache_key=env_name,
+        )
     else:
         raise_error(f"Environment kind '{env['kind']}' not supported")
     return get_env_lock_fpath(env=env, env_name=env_name, as_posix=False)
 
 
+@check_app.command(
+    name="julia-env",
+    help=(
+        "Check a Julia environment and instantiate only when project, "
+        "manifest, and package cache state have changed."
+    ),
+)
+def check_julia_env(
+    env_path: Annotated[
+        str,
+        typer.Argument(help="Path to Julia Project.toml file."),
+    ] = "Project.toml",
+    julia_version: Annotated[
+        str | None,
+        typer.Option(
+            "--julia",
+            help="Julia version to enforce (e.g., 1.11).",
+        ),
+    ] = None,
+    verbose: Annotated[
+        bool, typer.Option("--verbose", help="Print verbose output.")
+    ] = False,
+) -> str:
+    return _check_julia_env(
+        env_path=env_path,
+        julia_version=julia_version,
+        verbose=verbose,
+    )
+
+
 @check_app.command(
     name="envs",
     help="Check that all environments are up-to-date.",

{calkit_python-0.37.1 → calkit_python-0.37.3}/calkit/cli/main/core.py

@@ -1910,6 +1910,9 @@ def run_in_env(
             julia_cmd = calkit.julia.check_version_in_command(julia_cmd)
         except Exception as e:
             raise_error(f"Failed to check Julia version: {e}")
+        julia_cmd = calkit.julia.ensure_startup_file_disabled_in_command(
+            julia_cmd
+        )
         if verbose:
             typer.echo(f"Running command: {julia_cmd}")
         try: