pivot 0.0.2__tar.gz → 0.2.0__tar.gz

pivot-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Sami Jawhar
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pivot-0.2.0/PKG-INFO

@@ -0,0 +1,157 @@
+Metadata-Version: 2.4
+Name: pivot
+Version: 0.2.0
+Summary: High-performance Python pipeline tool with automatic code change detection
+Author: Sami Jawhar
+Author-email: Sami Jawhar <sami@metr.org>
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 3 - Alpha
+Classifier: Operating System :: POSIX
+Classifier: Programming Language :: Python :: 3.13
+Requires-Dist: aiohttp>=3.9
+Requires-Dist: anyio>=4.11
+Requires-Dist: click>=8.0
+Requires-Dist: cloudpickle>=3.0
+Requires-Dist: deepmerge>=2.0
+Requires-Dist: dulwich>=0.22.0
+Requires-Dist: filelock>=3.12.0
+Requires-Dist: flatten-dict>=0.4.0
+Requires-Dist: grandalf>=0.8
+Requires-Dist: lmdb>=1.4.0
+Requires-Dist: loky>=3.4
+Requires-Dist: networkx>=3.0
+Requires-Dist: pathspec>=0.12.0
+Requires-Dist: pandas>=2.0
+Requires-Dist: pydantic>=2.13
+Requires-Dist: pygtrie>=2.5.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich>=13.0
+Requires-Dist: ruamel-yaml>=0.18.0
+Requires-Dist: tabulate>=0.9.0
+Requires-Dist: tqdm>=4.66
+Requires-Dist: watchfiles>=1.0
+Requires-Dist: wcwidth>=0.2.0
+Requires-Dist: xxhash>=3.0
+Requires-Dist: basedpyright>=1.0 ; extra == 'dev'
+Requires-Dist: botocore-stubs>=1.35.0 ; extra == 'dev'
+Requires-Dist: hypothesis>=6.140 ; extra == 'dev'
+Requires-Dist: joblib>=1.4.0 ; extra == 'dev'
+Requires-Dist: joblib-stubs>=1.5.0 ; extra == 'dev'
+Requires-Dist: matplotlib-stubs>=0.3 ; extra == 'dev'
+Requires-Dist: mutmut>=2.5.1,<3 ; extra == 'dev'
+Requires-Dist: mypy>=1.19.1 ; extra == 'dev'
+Requires-Dist: pandas-stubs>=2.0 ; extra == 'dev'
+Requires-Dist: pivot[dvc] ; extra == 'dev'
+Requires-Dist: pivot[matplotlib] ; extra == 'dev'
+Requires-Dist: pivot[s3] ; extra == 'dev'
+Requires-Dist: pytest-aioboto3>=0.1.0 ; extra == 'dev'
+Requires-Dist: pytest-asyncio>=0.24.0 ; extra == 'dev'
+Requires-Dist: pytest-cov>=4.0 ; extra == 'dev'
+Requires-Dist: pytest-mock>=3.15.1 ; extra == 'dev'
+Requires-Dist: pytest-rerunfailures>=16.1 ; extra == 'dev'
+Requires-Dist: pytest-watcher>=0.6.2 ; extra == 'dev'
+Requires-Dist: pytest-xdist>=3.8.0 ; extra == 'dev'
+Requires-Dist: pytest>=7.0 ; extra == 'dev'
+Requires-Dist: ruff>=0.8.0 ; extra == 'dev'
+Requires-Dist: types-aioboto3[all]>=13.0.0 ; extra == 'dev'
+Requires-Dist: mkdocs>=1.6.0 ; extra == 'docs'
+Requires-Dist: mkdocs-material>=9.5.0 ; extra == 'docs'
+Requires-Dist: mkdocstrings[python]>=0.26.0 ; extra == 'docs'
+Requires-Dist: mkdocs-gen-files>=0.5.0 ; extra == 'docs'
+Requires-Dist: mkdocs-literate-nav>=0.6.0 ; extra == 'docs'
+Requires-Dist: mkdocs-section-index>=0.3.9 ; extra == 'docs'
+Requires-Dist: griffe>=1.3.0 ; extra == 'docs'
+Requires-Dist: dvc>=3.0 ; extra == 'dvc'
+Requires-Dist: matplotlib>=3.5 ; extra == 'matplotlib'
+Requires-Dist: aioboto3>=13.0.0 ; extra == 's3'
+Requires-Dist: pivot-tui ; extra == 'tui'
+Requires-Python: >=3.13, <3.14
+Project-URL: Documentation, https://sjawhar.github.io/pivot/
+Project-URL: Homepage, https://github.com/sjawhar/pivot
+Project-URL: Issues, https://github.com/sjawhar/pivot/issues
+Project-URL: Repository, https://github.com/sjawhar/pivot
+Provides-Extra: dev
+Provides-Extra: docs
+Provides-Extra: dvc
+Provides-Extra: matplotlib
+Provides-Extra: s3
+Provides-Extra: tui
+Description-Content-Type: text/markdown
+
+# Pivot: High-Performance Python Pipeline Tool
+
+**Change your code. Pivot knows what to run.**
+
+Pivot is a Python pipeline tool with automatic code change detection. Stages are plain
+Python functions; you wire them together by calling them inside a `Pipeline` block, and
+Pivot works out what needs to re-run.
+
+- **Automatic code change detection** using Python introspection
+- **No paths in pipeline code**: Pivot chooses where every output is stored
+- **Per-stage lock files** for fast parallel writes
+- **Warm worker pools** with preloaded imports
+- **Content-addressable caching** with S3 remote storage
+
+**Python:** 3.13+ | **Platform:** Unix (Linux/macOS)
+
+## Quick Start
+
+```bash
+pip install pivot
+mkdir my_project && cd my_project
+git init && pivot init
+```
+
+Put a raw input under `data/raw/`:
+
+```bash
+mkdir -p data/raw
+printf 'name,value\nalice,100\nbob,200\ncarol,\n' > data/raw/records.csv
+```
+
+Define stages and the pipeline in `pipeline.py`:
+
+```python
+# pipeline.py
+import pandas as pd
+
+from pivot import Pipeline, StageParams, stage
+
+
+class CleanParams(StageParams):
+    min_value: int = 0
+
+
+@stage
+def clean(records: pd.DataFrame, params: CleanParams) -> pd.DataFrame:
+    kept = records.dropna()
+    return kept[kept["value"] >= params.min_value]
+
+
+@stage
+def summarize(rows: pd.DataFrame) -> dict:
+    return {"count": len(rows), "total": int(rows["value"].sum())}
+
+
+with Pipeline("demo") as pipeline:
+    records = pipeline.input("records.csv")
+    cleaned = clean(records, params=CleanParams(min_value=50))
+    summarize(cleaned)
+```
+
+Then run it; Pivot resolves the DAG from the handles passed between stages:
+
+```bash
+pivot repro           # runs clean, then summarize
+pivot repro           # nothing changed: both stages are cached
+pivot repro --watch   # re-run automatically on file changes
+```
+
+Pivot writes outputs to `data/demo/clean.jsonl` and `data/demo/summarize.yaml`.
+
+## Learn More
+
+- [Documentation](https://sjawhar.github.io/pivot/)
+- [Source code](https://github.com/sjawhar/pivot)
+- Interactive TUI: `pip install pivot[tui]`, then `pivot repro --tui`

pivot-0.2.0/README.md

@@ -0,0 +1,76 @@
+# Pivot: High-Performance Python Pipeline Tool
+
+**Change your code. Pivot knows what to run.**
+
+Pivot is a Python pipeline tool with automatic code change detection. Stages are plain
+Python functions; you wire them together by calling them inside a `Pipeline` block, and
+Pivot works out what needs to re-run.
+
+- **Automatic code change detection** using Python introspection
+- **No paths in pipeline code**: Pivot chooses where every output is stored
+- **Per-stage lock files** for fast parallel writes
+- **Warm worker pools** with preloaded imports
+- **Content-addressable caching** with S3 remote storage
+
+**Python:** 3.13+ | **Platform:** Unix (Linux/macOS)
+
+## Quick Start
+
+```bash
+pip install pivot
+mkdir my_project && cd my_project
+git init && pivot init
+```
+
+Put a raw input under `data/raw/`:
+
+```bash
+mkdir -p data/raw
+printf 'name,value\nalice,100\nbob,200\ncarol,\n' > data/raw/records.csv
+```
+
+Define stages and the pipeline in `pipeline.py`:
+
+```python
+# pipeline.py
+import pandas as pd
+
+from pivot import Pipeline, StageParams, stage
+
+
+class CleanParams(StageParams):
+    min_value: int = 0
+
+
+@stage
+def clean(records: pd.DataFrame, params: CleanParams) -> pd.DataFrame:
+    kept = records.dropna()
+    return kept[kept["value"] >= params.min_value]
+
+
+@stage
+def summarize(rows: pd.DataFrame) -> dict:
+    return {"count": len(rows), "total": int(rows["value"].sum())}
+
+
+with Pipeline("demo") as pipeline:
+    records = pipeline.input("records.csv")
+    cleaned = clean(records, params=CleanParams(min_value=50))
+    summarize(cleaned)
+```
+
+Then run it; Pivot resolves the DAG from the handles passed between stages:
+
+```bash
+pivot repro           # runs clean, then summarize
+pivot repro           # nothing changed: both stages are cached
+pivot repro --watch   # re-run automatically on file changes
+```
+
+Pivot writes outputs to `data/demo/clean.jsonl` and `data/demo/summarize.yaml`.
+
+## Learn More
+
+- [Documentation](https://sjawhar.github.io/pivot/)
+- [Source code](https://github.com/sjawhar/pivot)
+- Interactive TUI: `pip install pivot[tui]`, then `pivot repro --tui`

pivot-0.2.0/pyproject.toml

@@ -0,0 +1,95 @@
+[project]
+name = "pivot"
+version = "0.2.0"
+description = "High-performance Python pipeline tool with automatic code change detection"
+requires-python = ">=3.13,<3.14"
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{ name = "Sami Jawhar", email = "sami@metr.org" }]
+classifiers = [
+  "Development Status :: 3 - Alpha",
+  "Operating System :: POSIX",
+  "Programming Language :: Python :: 3.13",
+]
+dependencies = [
+  "aiohttp>=3.9",
+  "anyio>=4.11",
+  "click>=8.0",
+  "cloudpickle>=3.0",
+  "deepmerge>=2.0",
+  "dulwich>=0.22.0",
+  "filelock>=3.12.0",
+  "flatten-dict>=0.4.0",
+  "grandalf>=0.8",
+  "lmdb>=1.4.0",
+  "loky>=3.4",
+  "networkx>=3.0",
+  "pathspec>=0.12.0",
+  "pandas>=2.0",
+  "pydantic>=2.13",
+  "pygtrie>=2.5.0",
+  "pyyaml>=6.0",
+  "rich>=13.0",
+  "ruamel.yaml>=0.18.0",
+  "tabulate>=0.9.0",
+  "tqdm>=4.66",
+  "watchfiles>=1.0",
+  "wcwidth>=0.2.0",
+  "xxhash>=3.0",
+]
+
+[project.optional-dependencies]
+dvc = ["dvc>=3.0"]
+matplotlib = ["matplotlib>=3.5"]
+s3 = ["aioboto3>=13.0.0"]
+tui = ["pivot-tui"]
+docs = [
+  "mkdocs>=1.6.0",
+  "mkdocs-material>=9.5.0",
+  "mkdocstrings[python]>=0.26.0",
+  "mkdocs-gen-files>=0.5.0",
+  "mkdocs-literate-nav>=0.6.0",
+  "mkdocs-section-index>=0.3.9",
+  "griffe>=1.3.0",
+]
+dev = [
+  "basedpyright>=1.0",
+  "botocore-stubs>=1.35.0",
+  "hypothesis>=6.140",
+  "joblib>=1.4.0",
+  "joblib-stubs>=1.5.0",
+  "matplotlib-stubs>=0.3",
+  "mutmut>=2.5.1,<3",
+  "mypy>=1.19.1",
+  "pandas-stubs>=2.0",
+  "pivot[dvc]",
+  "pivot[matplotlib]",
+  "pivot[s3]",
+  "pytest-aioboto3>=0.1.0",
+  "pytest-asyncio>=0.24.0",
+  "pytest-cov>=4.0",
+  "pytest-mock>=3.15.1",
+  "pytest-rerunfailures>=16.1",
+  "pytest-watcher>=0.6.2",
+  "pytest-xdist>=3.8.0",
+  "pytest>=7.0",
+  "ruff>=0.8.0",
+  "types-aioboto3[all]>=13.0.0",
+]
+
+[project.scripts]
+pivot = "pivot.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/sjawhar/pivot"
+Documentation = "https://sjawhar.github.io/pivot/"
+Repository = "https://github.com/sjawhar/pivot"
+Issues = "https://github.com/sjawhar/pivot/issues"
+
+[build-system]
+requires = ["uv_build>=0.9.18,<0.10.0"]
+build-backend = "uv_build"
+
+[tool.uv.sources]
+pivot-tui = { workspace = true }

pivot-0.2.0/src/pivot/__init__.py

@@ -0,0 +1,52 @@
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+__version__ = "0.2.0a1"
+
+# Public API - only exports that users need when writing pipelines.
+# Internal modules are accessible via their full paths.
+
+if TYPE_CHECKING:
+    from pivot import loaders as loaders
+    from pivot import stage_def as stage_def
+    from pivot.compose import Pipeline as Pipeline
+    from pivot.compose import incremental as incremental
+    from pivot.compose import metric as metric
+    from pivot.compose import plot as plot
+    from pivot.compose import stage as stage
+    from pivot.decorators import no_fingerprint as no_fingerprint
+    from pivot.stage_def import StageParams as StageParams
+
+# Lazy import mapping for runtime: (module_path, attr_name or None for module import)
+_LAZY_IMPORTS: dict[str, tuple[str, str | None]] = {
+    "no_fingerprint": ("pivot.decorators", "no_fingerprint"),
+    "loaders": ("pivot.loaders", None),
+    "stage_def": ("pivot.stage_def", None),
+    "Pipeline": ("pivot.compose", "Pipeline"),
+    "stage": ("pivot.compose", "stage"),
+    "metric": ("pivot.compose", "metric"),
+    "incremental": ("pivot.compose", "incremental"),
+    "plot": ("pivot.compose", "plot"),
+    "StageParams": ("pivot.stage_def", "StageParams"),
+}
+
+
+def __getattr__(name: str) -> object:
+    """Lazily import public API members on first access."""
+    if name in _LAZY_IMPORTS:
+        module_path, attr_name = _LAZY_IMPORTS[name]
+        import importlib
+
+        module = importlib.import_module(module_path)
+        value = module if attr_name is None else getattr(module, attr_name)
+        # Cache in module globals for subsequent access
+        globals()[name] = value
+        return value
+    msg = f"module {__name__!r} has no attribute {name!r}"
+    raise AttributeError(msg)
+
+
+def __dir__() -> list[str]:
+    """List available attributes including lazy imports."""
+    return list(globals().keys()) + list(_LAZY_IMPORTS.keys())

pivot-0.2.0/src/pivot/ast_utils.py

@@ -0,0 +1,98 @@
+import ast
+import contextlib
+import inspect
+import textwrap
+import weakref
+from collections.abc import Callable
+from typing import Any
+
+# Cache for extract_module_attr_usage results using weak references.
+# This avoids repeated AST parsing for the same function when processing module dependencies.
+# Note: WeakKeyDictionary is not thread-safe. Fingerprinting runs single-threaded
+# per process (multiprocessing uses separate memory spaces), so this is safe.
+_module_attr_cache: weakref.WeakKeyDictionary[Callable[..., Any], list[tuple[str, str]]] = (
+    weakref.WeakKeyDictionary()
+)
+
+
+def extract_module_attr_usage(func: Callable[..., Any]) -> list[tuple[str, str]]:
+    """Extract module.attr patterns (e.g., 'np.array') from function AST."""
+    # WeakKeyDictionary raises TypeError for non-weakly-referenceable functions (builtins)
+    with contextlib.suppress(TypeError):
+        cached = _module_attr_cache.get(func)
+        if cached is not None:
+            return cached
+
+    result = _extract_module_attr_usage_impl(func)
+
+    with contextlib.suppress(TypeError):
+        _module_attr_cache[func] = result
+
+    return result
+
+
+def _extract_module_attr_usage_impl(func: Callable[..., Any]) -> list[tuple[str, str]]:
+    """Internal implementation of extract_module_attr_usage."""
+    try:
+        source = inspect.getsource(func)
+    except (OSError, TypeError):
+        return []
+
+    try:
+        tree = ast.parse(textwrap.dedent(source))
+    except SyntaxError:
+        return []
+
+    attrs = list[tuple[str, str]]()
+
+    for node in ast.walk(tree):
+        if isinstance(node, ast.Attribute):
+            chain = [node.attr]
+            current = node.value
+
+            while isinstance(current, ast.Attribute):
+                chain.append(current.attr)
+                current = current.value
+
+            if isinstance(current, ast.Name):
+                module_name = current.id
+                # Reverse chain to get correct order (module.sub.attr)
+                attr_path = ".".join(reversed(chain))
+                attrs.append((module_name, attr_path))
+
+    return list(dict.fromkeys(attrs))
+
+
+def get_function_ast(func: Callable[..., Any]) -> ast.FunctionDef | ast.AsyncFunctionDef:
+    """Parse function to AST node."""
+    try:
+        source = inspect.getsource(func)
+    except (OSError, TypeError) as e:
+        raise ValueError(f"Cannot get source for {func}") from e
+
+    tree = ast.parse(textwrap.dedent(source))
+
+    for node in ast.walk(tree):
+        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
+            return node
+
+    raise ValueError(f"No FunctionDef found in source for {func}")
+
+
+def normalize_ast(node: ast.AST) -> ast.AST:
+    """Remove docstrings and metadata for stable AST comparison."""
+    if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)) and (
+        node.body
+        and isinstance(node.body[0], ast.Expr)
+        and isinstance(node.body[0].value, ast.Constant)
+        and isinstance(node.body[0].value.value, str)
+    ):
+        node.body = node.body[1:]
+        # Add Pass() if body becomes empty (functions must have non-empty body)
+        if not node.body:
+            node.body = [ast.Pass()]
+
+    for child in ast.iter_child_nodes(node):
+        normalize_ast(child)
+
+    return node

pivot-0.2.0/src/pivot/cli/AGENTS.md

@@ -0,0 +1,116 @@
+# Pivot CLI - Development Guidelines
+
+## The `pivot_command` Decorator (Critical)
+
+**All new CLI commands MUST use `@cli_decorators.pivot_command()` instead of `@click.command()`.**
+
+The `pivot_command` decorator wraps Click commands with two essential behaviors:
+
+1. **Auto-discovery** - Discovers `pipeline.py` (`discovery.discover_pipeline()`) and stores the Pipeline in the Click context before the command runs
+2. **Error handling** - Converts `PivotError` exceptions to user-friendly `ClickException` messages
+
+```python
+from pivot.cli import decorators as cli_decorators
+
+# Standard command - auto-discovers stages before running
+@cli_decorators.pivot_command()
+def list_cmd() -> None:
+    """List registered stages."""
+    # Pipeline is guaranteed to be in context here
+    stages = cli_helpers.list_stages()
+    ...
+
+# Command that doesn't need the registry
+@cli_decorators.pivot_command(auto_discover=False)
+def init() -> None:
+    """Initialize new project."""
+    # No discovery needed - creating new project
+    ...
+```
+
+### When to use `auto_discover=False`
+
+Set `auto_discover=False` only for commands that don't use the stage registry:
+
+| Command | auto_discover | Reason |
+|---------|---------------|--------|
+| run | True (default) | Need registry to find stages |
+| list, dag | True (default), allow_all=True | Need registry; --all for multi-pipeline |
+| checkout | True (default) | Need registry; handles None pipeline gracefully |
+| track | True (default) | Handles None pipeline gracefully (no overlap check) |
+| status | True (default), allow_all=True | Handles None pipeline gracefully (skips stages section) |
+| init, doctor, check-ignore | False | No stage data needed |
+| history, show | False | Read run history only |
+| get | False | Restores files and `.pvt` targets from git; discovers the pipeline itself for stage targets, and a broken `pipeline.py` only stops stage targets |
+| import, update | False | Work from git revisions, lock files and `.pvt` files |
+| push, pull, fetch | True (default) | Need registry for output cache filtering |
+
+**Principle:** The pipeline registry is the canonical source of stage metadata. Commands that inspect stage properties (outputs, tags, deps) must use auto-discovery. Only commands that truly don't touch stage data should use `auto_discover=False`.
+
+### Why this matters
+
+Without auto-discovery, commands fail if called before any other pivot command:
+
+```bash
+# In a fresh shell, this would fail without auto-discovery:
+$ pivot list
+No stages registered.  # Wrong! Should discover pipeline.py first
+```
+
+The decorator ensures consistent behavior regardless of command order.
+
+### For group subcommands
+
+Commands under a `@click.group()` (like `pivot metrics show`) can't use `pivot_command`. Use `@with_error_handling` and call `ensure_stages_registered()` explicitly:
+
+```python
+from pivot.cli._run_common import ensure_stages_registered
+
+@metrics.command("show")
+@cli_decorators.with_error_handling
+def metrics_show() -> None:
+    ensure_stages_registered()  # Manual discovery for group subcommands
+    ...
+```
+
+## Input Validation (Critical)
+
+**Validate inputs as early as possible** - use Click's built-in validation in option/argument decorators.
+
+### Numeric Options
+
+Always use `click.IntRange` or `click.FloatRange` for numeric inputs:
+
+```python
+# Good - validates at argument parsing time
+@click.option("--precision", type=click.IntRange(min=0), default=5)
+@click.option("--jobs", type=click.IntRange(min=1), default=20)
+@click.option("--debounce", type=click.IntRange(min=0), default=300)
+
+# Bad - allows invalid values through
+@click.option("--precision", type=int, default=5)  # Allows negative!
+```
+
+### Path Options
+
+Use `click.Path` with appropriate parameters:
+
+```python
+@click.option("--output", type=click.Path(path_type=pathlib.Path))
+@click.option("--config", type=click.Path(exists=True, dir_okay=False))
+```
+
+### Choice Options
+
+Use `click.Choice` for limited valid values:
+
+```python
+@click.option("--format", type=click.Choice(["json", "yaml", "csv"]))
+```
+
+### Why Early Validation Matters
+
+1. **Better error messages** - Click provides user-friendly error messages automatically
+2. **Fail fast** - Don't waste time processing before discovering invalid input
+3. **Type safety** - Validated inputs have correct types in the function body
+4. **Consistency** - Users get the same error format for all validation failures

pivot-0.2.0/src/pivot/cli/CLAUDE.md

@@ -0,0 +1 @@
+@AGENTS.md