calkit-python 0.0.2__tar.gz → 0.0.4__tar.gz

{calkit_python-0.0.2 → calkit_python-0.0.4}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.3
 Name: calkit-python
-Version: 0.0.2
-Summary: Reproducibility made easy.
+Version: 0.0.4
+Summary: Reproducibility simplified.
 Project-URL: Homepage, https://github.com/calkit/calkit
 Project-URL: Issues, https://github.com/calkit/calkit/issues
 Author-email: Pete Bachant <petebachant@gmail.com>
@@ -15,23 +15,56 @@ Requires-Dist: eval-type-backport; python_version < '3.10'
 Requires-Dist: fastapi
 Requires-Dist: gitpython
 Requires-Dist: keyring
-Requires-Dist: pandas
-Requires-Dist: polars
+Requires-Dist: nbconvert
 Requires-Dist: pydantic-settings
 Requires-Dist: pydantic[email]
 Requires-Dist: pyjwt
 Requires-Dist: requests
 Requires-Dist: typer
+Provides-Extra: data
+Requires-Dist: pandas; extra == 'data'
+Requires-Dist: polars; extra == 'data'
 Description-Content-Type: text/markdown
 
 # Calkit
 
-[Calkit](https://calkit.io) makes reproducible research easy,
+[Calkit](https://calkit.io) simplifies reproducibility,
 acting as a layer on top of
-[Git](https://git-scm.com/) and [DVC](https://dvc.org/), such that all
-all materials involved in the research process can be fully described in a
+[Git](https://git-scm.com/), [DVC](https://dvc.org/),
+[Zenodo](https://zenodo.org), and more,
+such that all all aspects of the research process can be fully described in a
 single repository.
 
+## Why does reproducibility matter?
+
+If your work is reproducible, that means that someone else can "run" it and
+get the same results or outputs.
+This is a major step towards addressing
+[the replication crisis](https://en.wikipedia.org/wiki/Replication_crisis)
+and has some major benefits for both you as an individual and the research
+community:
+
+1. You will avoid mistakes caused by, e.g., running an old version of a script
+   and including a figure that wasn't created after fixing a bug in the data
+   processing pipeline.
+2. Since your project is "runnable," it's more likely that someone else will be
+   able to reuse part of your work to run it in a different context, thereby
+   producing a bigger impact and accelerating the pace of discovery.
+   If someone can take what you've done and use it to calculate a
+   prediction, you have just produced truly useful knowledge.
+
+## Why another tool/platform?
+
+Git, GitHub, DVC, Zenodo et al. are amazing tools/platforms, but their
+use involves multiple fairly difficult learning curves.
+Our goal is to provide a single tool and platform to unify all of these so
+that there is a single, gentle learning curve.
+However, it is not our goal to hide or replace these underlying components.
+Advanced users can use them directly, but new users aren't forced to, which
+helps them get up and running with less effort and training.
+Calkit should help users understand what is going on under the hood without
+forcing them to work at that lower level of abstraction.
+
 ## Installation
 
 Simply run

calkit_python-0.0.4/README.md

@@ -0,0 +1,81 @@
+# Calkit
+
+[Calkit](https://calkit.io) simplifies reproducibility,
+acting as a layer on top of
+[Git](https://git-scm.com/), [DVC](https://dvc.org/),
+[Zenodo](https://zenodo.org), and more,
+such that all all aspects of the research process can be fully described in a
+single repository.
+
+## Why does reproducibility matter?
+
+If your work is reproducible, that means that someone else can "run" it and
+get the same results or outputs.
+This is a major step towards addressing
+[the replication crisis](https://en.wikipedia.org/wiki/Replication_crisis)
+and has some major benefits for both you as an individual and the research
+community:
+
+1. You will avoid mistakes caused by, e.g., running an old version of a script
+   and including a figure that wasn't created after fixing a bug in the data
+   processing pipeline.
+2. Since your project is "runnable," it's more likely that someone else will be
+   able to reuse part of your work to run it in a different context, thereby
+   producing a bigger impact and accelerating the pace of discovery.
+   If someone can take what you've done and use it to calculate a
+   prediction, you have just produced truly useful knowledge.
+
+## Why another tool/platform?
+
+Git, GitHub, DVC, Zenodo et al. are amazing tools/platforms, but their
+use involves multiple fairly difficult learning curves.
+Our goal is to provide a single tool and platform to unify all of these so
+that there is a single, gentle learning curve.
+However, it is not our goal to hide or replace these underlying components.
+Advanced users can use them directly, but new users aren't forced to, which
+helps them get up and running with less effort and training.
+Calkit should help users understand what is going on under the hood without
+forcing them to work at that lower level of abstraction.
+
+## Installation
+
+Simply run
+
+```sh
+pip install calkit-python
+```
+
+## Cloud integration
+
+The Calkit cloud platform (https://calkit.io) serves as a project
+management interface and a DVC remote for easily storing all versions of your
+data/code/figures/publications, interacting with your collaborators,
+reusing others' research artifacts, etc.
+
+After signing up, visit the [settings](https://calkit.io/settings) page
+and create a token.
+Then run
+
+```sh
+calkit config set token ${YOUR_TOKEN_HERE}
+```
+
+Then, inside a project repo you'd like to connect to the cloud, run
+
+```sh
+calkit config setup-remote
+```
+
+This will setup the Calkit DVC remote, such that commands like `dvc push` will
+allow you to push versions of your data or pipeline outputs to the cloud
+for safe storage and sharing with your collaborators.
+
+## How it works
+
+Calkit creates a simple human-readable "database" inside the `calkit.yaml`
+file, which serves as a way to store important information about the project,
+e.g., what question(s) it seeks to answer,
+what files should be considered datasets, figures, publications, etc.
+The Calkit cloud reads this database and registers the various entities
+as part of the entire ecosystem such that if a project is made public,
+other researchers can find and reuse your work to accelerate their own.

{calkit_python-0.0.2 → calkit_python-0.0.4}/calkit/__init__.py

@@ -1,4 +1,4 @@
-__version__ = "0.0.2"
+__version__ = "0.0.4"
 
 from .core import *
 from . import git

calkit_python-0.0.4/calkit/cli.py

@@ -0,0 +1,267 @@
+"""The command line interface."""
+
+from __future__ import annotations
+
+import os
+import pty
+import subprocess
+import sys
+
+import git
+import typer
+from typing_extensions import Annotated, Optional
+
+from calkit.core import ryaml
+
+from . import config
+from .dvc import configure_remote, set_remote_auth
+
+app = typer.Typer()
+config_app = typer.Typer()
+new_app = typer.Typer()
+app.add_typer(config_app, name="config", help="Configure Calkit.")
+app.add_typer(new_app, name="new", help="Add new Calkit object.")
+
+
+@config_app.command(name="set")
+def set_config_value(key: str, value: str):
+    try:
+        cfg = config.read()
+        cfg = config.Settings.model_validate(cfg.model_dump() | {key: value})
+        # Kind of a hack for setting the password computed field
+        # Types have been validated above, so this won't hurt to do again
+        setattr(cfg, key, value)
+    except FileNotFoundError:
+        # TODO: This fails if we try to set password before any config has
+        # been written
+        # Username is fine
+        cfg = config.Settings.model_validate({key: value})
+    cfg.write()
+
+
+@config_app.command(name="get")
+def get_config_value(key: str) -> None:
+    cfg = config.read()
+    val = getattr(cfg, key)
+    if val is not None:
+        print(val)
+    else:
+        print()
+
+
+@config_app.command(name="setup-remote")
+def setup_remote():
+    configure_remote()
+    set_remote_auth()
+
+
+@app.command(name="status")
+def get_status():
+    """Get a unified Git and DVC status."""
+
+    def print_sep(name: str):
+        width = 66
+        txt_width = len(name) + 2
+        buffer_width = (width - txt_width) // 2
+        buffer = "-" * buffer_width
+        typer.echo(f"{buffer} {name} {buffer}")
+
+    def run_cmd(cmd: list[str]):
+        if os.name == "nt":
+            subprocess.call(cmd)
+        else:
+            pty.spawn(cmd, lambda fd: os.read(fd, 1024))
+
+    print_sep("Code (Git)")
+    run_cmd(["git", "status"])
+    typer.echo()
+    print_sep("Data (DVC)")
+    run_cmd(["dvc", "data", "status"])
+    typer.echo()
+    print_sep("Pipeline (DVC)")
+    run_cmd(["dvc", "status"])
+
+
+@app.command(name="add")
+def add(paths: list[str]):
+    """Add paths to the repo.
+
+    Code will be added to Git and data will be added to DVC.
+    """
+    dvc_extensions = [".png", ".h5", ".parquet", ".pickle"]
+    dvc_size_thresh_bytes = 1_000_000
+    dvc_folders = ["data", "figures"]
+    if "." in paths:
+        print("ERROR: Cannot add '.' with calkit; use git or dvc")
+        sys.exit(1)
+    for path in paths:
+        if os.path.isdir(path):
+            print("ERROR: Cannot add directories with calkit; use git or dvc")
+            sys.exit(1)
+        # Detect if this file should be tracked with Git or DVC
+        # TODO: Add to whatever
+
+
+@app.command(name="commit")
+def commit(
+    all: Annotated[Optional[bool], typer.Option("--all", "-a")] = False,
+    message: Annotated[Optional[str], typer.Option("--message", "-m")] = None,
+):
+    """Commit a change to the repo."""
+    print(all, message)
+    raise NotImplementedError
+
+
+@app.command(name="pull", help="Pull with both Git and DVC.")
+def pull():
+    typer.echo("Git pulling")
+    subprocess.call(["git", "pull"])
+    typer.echo("DVC pulling")
+    subprocess.call(["dvc", "pull"])
+
+
+@app.command(name="push", help="Push with both Git and DVC.")
+def push():
+    typer.echo("Pushing to Git remote")
+    subprocess.call(["git", "push"])
+    typer.echo("Pushing to DVC remote")
+    subprocess.call(["dvc", "push"])
+
+
+@new_app.command(name="figure")
+def new_figure(
+    path: str,
+    title: Annotated[str, typer.Option("--title")],
+    description: Annotated[str, typer.Option("--desc")] = None,
+    commit: Annotated[bool, typer.Option("--commit")] = False,
+):
+    """Add a new figure to ``calkit.yaml``."""
+    if os.path.isfile("calkit.yaml"):
+        with open("calkit.yaml") as f:
+            ck_info = ryaml.load(f)
+    else:
+        ck_info = {}
+    figures = ck_info.get("figures", [])
+    paths = [f.get("path") for f in figures]
+    if path in paths:
+        raise ValueError(f"Figure at path {path} already exists")
+    obj = dict(path=path, title=title)
+    if description is not None:
+        obj["description"] = description
+    figures.append(obj)
+    ck_info["figures"] = figures
+    with open("calkit.yaml", "w") as f:
+        ryaml.dump(ck_info, f)
+    if commit:
+        repo = git.Repo()
+        repo.git.add("calkit.yaml")
+        repo.git.commit(["-m", f"Add figure {path}"])
+
+
+@new_app.command("question")
+def new_question(
+    question: str,
+    commit: Annotated[bool, typer.Option("--commit")] = False,
+):
+    if os.path.isfile("calkit.yaml"):
+        with open("calkit.yaml") as f:
+            ck_info = ryaml.load(f)
+    else:
+        ck_info = {}
+    questions = ck_info.get("questions", [])
+    if question in questions:
+        raise ValueError("Question already exists")
+    if not question.endswith("?"):
+        raise ValueError("Questions must end with a question mark")
+    questions.append(question)
+    ck_info["questions"] = questions
+    with open("calkit.yaml", "w") as f:
+        ryaml.dump(ck_info, f)
+    if commit:
+        repo = git.Repo()
+        repo.git.add("calkit.yaml")
+        repo.git.commit(["-m", "Add question"])
+
+
+@app.command(name="server", help="Run the local server.")
+def run_server():
+    import uvicorn
+
+    uvicorn.run(
+        "calkit.server:app",
+        port=8866,
+        host="localhost",
+        reload=True,
+        reload_dirs=[os.path.dirname(__file__)],
+    )
+
+
+@app.command(
+    name="run",
+    add_help_option=False,
+    help="Run DVC pipeline (a wrapper for `dvc repro`).",
+)
+def run_dvc_repro(
+    targets: Optional[list[str]] = typer.Argument(default=None),
+    help: Annotated[bool, typer.Option("-h", "--help")] = False,
+    quiet: Annotated[bool, typer.Option("-q", "--quiet")] = False,
+    verbose: Annotated[bool, typer.Option("-v", "--verbose")] = False,
+    force: Annotated[bool, typer.Option("-f", "--force")] = False,
+    interactive: Annotated[bool, typer.Option("-i", "--interactive")] = False,
+    single_item: Annotated[bool, typer.Option("-s", "--single-item")] = False,
+    pipeline: Annotated[
+        Optional[str], typer.Option("-p", "--pipeline")
+    ] = None,
+    all_pipelines: Annotated[
+        bool, typer.Option("-P", "--all-pipelines")
+    ] = False,
+    recursive: Annotated[bool, typer.Option("-R", "--recursive")] = False,
+    downstream: Annotated[
+        Optional[list[str]], typer.Option("--downstream")
+    ] = None,
+    force_downstream: Annotated[
+        bool, typer.Option("--force-downstream")
+    ] = False,
+    pull: Annotated[bool, typer.Option("--pull")] = False,
+    allow_missing: Annotated[bool, typer.Option("--allow-missing")] = False,
+    dry: Annotated[bool, typer.Option("--dry")] = False,
+    keep_going: Annotated[bool, typer.Option("--keep-going", "-k")] = False,
+    ignore_errors: Annotated[bool, typer.Option("--ignore-errors")] = False,
+    glob: Annotated[bool, typer.Option("--glob")] = False,
+    no_commit: Annotated[bool, typer.Option("--no-commit")] = False,
+    no_run_cache: Annotated[bool, typer.Option("--no-run-cache")] = False,
+):
+    """A simple wrapper for ``dvc repro``."""
+    if targets is None:
+        targets = []
+    args = targets
+    # Extract any boolean args
+    for name in [
+        "help",
+        "quiet",
+        "verbose",
+        "force",
+        "interactive",
+        "single-item",
+        "all-pipelines",
+        "recursive",
+        "pull",
+        "allow-missing",
+        "dry",
+        "keep-going",
+        "force-downstream",
+        "glob",
+        "no-commit",
+        "no-run-cache",
+    ]:
+        if locals()[name.replace("-", "_")]:
+            args.append("--" + name)
+    if pipeline is not None:
+        args += ["--pipeline", pipeline]
+    if downstream is not None:
+        args += downstream
+    subprocess.call(["dvc", "repro"] + args)
+
+
+def run() -> None:
+    app()

{calkit_python-0.0.2 → calkit_python-0.0.4}/calkit/core.py

@@ -3,11 +3,21 @@
 from __future__ import annotations
 
 import glob
+import logging
 import os
 
+import ruamel.yaml
 from git import Repo
 from git.exc import InvalidGitRepositoryError
 
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__package__)
+
+ryaml = ruamel.yaml.YAML()
+ryaml.indent(mapping=2, sequence=4, offset=2)
+ryaml.preserve_quotes = True
+ryaml.width = 70
+
 
 def find_project_dirs(relative=False, max_depth=3) -> list[str]:
     """Find all Calkit project directories."""

{calkit_python-0.0.2 → calkit_python-0.0.4}/calkit/data.py

@@ -1,4 +1,9 @@
-"""Functionality for working with datasets."""
+"""Functionality for working with datasets.
+
+Since the dependencies here are optional, we need to ensure this isn't imported
+by default, or otherwise ensure ``import calkit`` works when the data
+dependencies are not installed.
+"""
 
 from __future__ import annotations
 

calkit_python-0.0.4/calkit/models.py

@@ -0,0 +1,81 @@
+"""Data models."""
+
+from typing import Literal
+
+from pydantic import BaseModel
+
+
+class _ImportedFrom(BaseModel):
+    project: str
+    path: str = None
+
+
+class _CalkitObject(BaseModel):
+    path: str
+    title: str
+    description: str
+    stage: str | None = None
+
+
+class Dataset(_CalkitObject):
+    pass
+
+
+class Figure(_CalkitObject):
+    pass
+
+
+class Publication(_CalkitObject):
+    kind: Literal[
+        "journal-article",
+        "conference-paper",
+        "presentation",
+        "poster",
+        "report",
+        "blog",
+    ]
+    is_published: bool = False
+    doi: str = None
+
+
+class ReferenceFile(BaseModel):
+    path: str
+    key: str
+
+
+class ReferenceCollection(BaseModel):
+    path: str
+    files: list[ReferenceFile] = []
+
+
+class Environment(BaseModel):
+    name: str
+    kind: Literal["conda", "docker", "pip", "poetry", "npm", "yarn"]
+    path: str
+
+
+class Software(BaseModel):
+    title: str
+    path: str
+    description: str
+
+
+class Notebook(_CalkitObject):
+    """A Jupyter notebook."""
+
+    pass
+
+
+class ProjectInfo(BaseModel):
+    """All of the project's information or metadata, written to the
+    ``calkit.yaml`` file.
+    """
+
+    questions: list[str] = []
+    datasets: list[Dataset] = []
+    figures: list[Figure] = []
+    publications: list[Publication] = []
+    references: list[ReferenceCollection] = []
+    environments: list[Environment] = []
+    software: list[Software] = []
+    notebooks: list[Notebook] = []

{calkit_python-0.0.2 → calkit_python-0.0.4}/pyproject.toml

@@ -1,34 +1,39 @@
 [build-system]
-requires = ["hatchling"]
 build-backend = "hatchling.build"
+requires = ["hatchling"]
 
 [project]
-name = "calkit-python"
-dynamic = ["version"]
 authors = [
-  { name="Pete Bachant", email="petebachant@gmail.com" },
+  {name = "Pete Bachant", email = "petebachant@gmail.com"},
 ]
-description = "Reproducibility made easy."
-readme = "README.md"
-requires-python = ">=3.8"
 classifiers = [
-    "Programming Language :: Python :: 3",
-    "License :: OSI Approved :: MIT License",
-    "Operating System :: OS Independent",
+  "Programming Language :: Python :: 3",
+  "License :: OSI Approved :: MIT License",
+  "Operating System :: OS Independent",
 ]
 dependencies = [
-    "dvc",
-    "eval-type-backport; python_version < '3.10'",
-    "fastapi",
-    "gitpython",
-    "keyring",
-    "pandas",
-    "polars",
-    "pydantic[email]",
-    "pydantic-settings",
-    "pyjwt",
-    "requests",
-    "typer",
+  "dvc",
+  "eval-type-backport; python_version < '3.10'",
+  "fastapi",
+  "gitpython",
+  "keyring",
+  "nbconvert",
+  "pydantic[email]",
+  "pydantic-settings",
+  "pyjwt",
+  "requests",
+  "typer",
+]
+description = "Reproducibility simplified."
+dynamic = ["version"]
+name = "calkit-python"
+readme = "README.md"
+requires-python = ">=3.8"
+
+[project.optional-dependencies]
+data = [
+  "pandas",
+  "polars",
 ]
 
 [project.urls]

calkit_python-0.0.2/README.md

@@ -1,50 +0,0 @@
-# Calkit
-
-[Calkit](https://calkit.io) makes reproducible research easy,
-acting as a layer on top of
-[Git](https://git-scm.com/) and [DVC](https://dvc.org/), such that all
-all materials involved in the research process can be fully described in a
-single repository.
-
-## Installation
-
-Simply run
-
-```sh
-pip install calkit-python
-```
-
-## Cloud integration
-
-The Calkit cloud platform (https://calkit.io) serves as a project
-management interface and a DVC remote for easily storing all versions of your
-data/code/figures/publications, interacting with your collaborators,
-reusing others' research artifacts, etc.
-
-After signing up, visit the [settings](https://calkit.io/settings) page
-and create a token.
-Then run
-
-```sh
-calkit config set token ${YOUR_TOKEN_HERE}
-```
-
-Then, inside a project repo you'd like to connect to the cloud, run
-
-```sh
-calkit config setup-remote
-```
-
-This will setup the Calkit DVC remote, such that commands like `dvc push` will
-allow you to push versions of your data or pipeline outputs to the cloud
-for safe storage and sharing with your collaborators.
-
-## How it works
-
-Calkit creates a simple human-readable "database" inside the `calkit.yaml`
-file, which serves as a way to store important information about the project,
-e.g., what question(s) it seeks to answer,
-what files should be considered datasets, figures, publications, etc.
-The Calkit cloud reads this database and registers the various entities
-as part of the entire ecosystem such that if a project is made public,
-other researchers can find and reuse your work to accelerate their own.

calkit_python-0.0.2/calkit/cli.py

@@ -1,119 +0,0 @@
-"""The command line interface."""
-
-from __future__ import annotations
-
-import os
-import pty
-import subprocess
-import sys
-
-import typer
-from typing_extensions import Annotated, Optional
-
-import calkit
-
-from . import config
-from .dvc import configure_remote, set_remote_auth
-
-app = typer.Typer()
-config_app = typer.Typer()
-app.add_typer(config_app, name="config", help="Configure Calkit.")
-
-
-@config_app.command(name="set")
-def set_config_value(key: str, value: str):
-    try:
-        cfg = config.read()
-        cfg = config.Settings.model_validate(cfg.model_dump() | {key: value})
-        # Kind of a hack for setting the password computed field
-        # Types have been validated above, so this won't hurt to do again
-        setattr(cfg, key, value)
-    except FileNotFoundError:
-        # TODO: This fails if we try to set password before any config has
-        # been written
-        # Username is fine
-        cfg = config.Settings.model_validate({key: value})
-    cfg.write()
-
-
-@config_app.command(name="get")
-def get_config_value(key: str) -> None:
-    cfg = config.read()
-    val = getattr(cfg, key)
-    if val is not None:
-        print(val)
-    else:
-        print()
-
-
-@config_app.command(name="setup-remote")
-def setup_remote():
-    configure_remote()
-    set_remote_auth()
-
-
-@app.command(name="status")
-def get_status():
-    """Get a unified Git and DVC status."""
-
-    def print_sep(name: str):
-        print(f"------------ {name} ------------")
-
-    print_sep("Code")
-    if os.name == "nt":
-        subprocess.call(["git", "status"])
-        print()
-        print_sep("data")
-        subprocess.call(["dvc", "status"])
-    else:
-        pty.spawn(["git", "status"], lambda fd: os.read(fd, 1024))
-        print()
-        print_sep("Data")
-        pty.spawn(["dvc", "status"], lambda fd: os.read(fd, 1024))
-
-
-@app.command(name="add")
-def add(paths: list[str]):
-    """Add paths to the repo.
-
-    Code will be added to Git and data will be added to DVC.
-    """
-    dvc_extensions = [".png", ".h5", ".parquet", ".pickle"]
-    dvc_size_thresh_bytes = 1_000_000
-    dvc_folders = ["data", "figures"]
-    if "." in paths:
-        print("ERROR: Cannot add '.' with calkit; use git or dvc")
-        sys.exit(1)
-    for path in paths:
-        if os.path.isdir(path):
-            print("ERROR: Cannot add directories with calkit; use git or dvc")
-            sys.exit(1)
-        # Detect if this file should be tracked with Git or DVC
-        # TODO: Add to whatever
-
-
-@app.command(name="commit")
-def commit(
-    all: Annotated[Optional[bool], typer.Option("--all", "-a")] = False,
-    message: Annotated[Optional[str], typer.Option("--message", "-m")] = None,
-):
-    """Commit a change to the repo."""
-    print(all, message)
-    raise NotImplementedError
-
-
-@app.command(name="server")
-def run_server():
-    import uvicorn
-
-    uvicorn.run(
-        "calkit.server:app",
-        port=8866,
-        host="localhost",
-        reload=True,
-        reload_dirs=[os.path.dirname(__file__)],
-    )
-
-
-def run() -> None:
-    app()