ab-plumber 0.1.0__py3-none-any.whl

ab_plumber-0.1.0.dist-info/METADATA

@@ -0,0 +1,133 @@
+Metadata-Version: 2.5
+Name: ab-plumber
+Version: 0.1.0
+Summary: A geospatial pipeline framework: phase functions run by a swappable execution strategy.
+Project-URL: Homepage, https://github.com/austinbreunig/plumber
+Project-URL: Repository, https://github.com/austinbreunig/plumber
+Project-URL: Issues, https://github.com/austinbreunig/plumber/issues
+Author: Austin Breunig
+License: MIT License
+        
+        Copyright (c) 2026 plumber contributors
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+License-File: LICENSE
+Keywords: geopandas,geospatial,gis,pipeline
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: GIS
+Requires-Python: >=3.9
+Requires-Dist: geopandas>=0.13
+Requires-Dist: pyarrow>=10
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == 'dev'
+Requires-Dist: ruff>=0.4; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# plumber
+
+A geospatial data-engineering pipeline framework. A stable function-shaped phase
+interface (`run(gdf, **params) -> gdf`) separates three concerns that are usually
+tangled: *what an algorithm does*, *how phases are sequenced and scaled*, and
+*how a pipeline is specified*. That separation is meant to make both
+prototyping→production scaling and safe agentic pipeline generation tractable —
+new logic drops into a fixed slot instead of modifying a monolith.
+
+## Status
+
+Phase 3 (Tracer). `run`, `check`, `localmp`, checkpoints and partial runs work. See
+[issue #1](https://github.com/austinbreunig/plumber/issues/1) for the wayfinder map and
+`plan.md` for the design.
+
+## Quickstart
+
+```bash
+pipx install ab-plumber
+```
+
+In any repo, write your input function, one or more phases, and an output function.
+A phase is `run(gdf, **params) -> gdf`. Then add a `plumber.yaml`:
+
+```yaml
+input:
+  path: myio:load_roads
+  params: {path: data/roads.parquet}
+phases:
+  - path: phases.buffer:run
+    partitionable: true
+    params: {distance: 10}
+output:
+  path: myio:write_parquet
+  params: {path: out/roads.parquet}
+```
+
+Check it, then run it:
+
+```bash
+plumber check                  # PASS/WARN/FAIL per phase, also writes report.json
+plumber run                    # one process (local)
+plumber run --strategy localmp --worker-count 4   # worker processes, same output
+```
+
+Optional: `checkpoint: true` on a slow phase saves its output, and
+`plumber run --from <phase> --to <phase>` runs a slice.
+
+## Execution strategy
+
+Default is `local` (one process). Switch to `localmp` (worker processes) with no phase
+changes:
+
+```yaml
+execution:
+  strategy: localmp
+  partition:            # pick exactly one
+    by: [zone_id]       # one piece per unique value combination
+    # chunk_size: 10000 # fixed-size row slices, in order
+    # worker_count: 8   # N roughly equal slices
+```
+
+CLI flags override the config: `--strategy localmp`, `--by zone_id`, `--chunk-size 10000`,
+`--worker-count 8`. Any partition flag replaces the config's whole `partition:` block.
+A phase with `partitionable: false` gets the joined data once. The output keeps the input's
+row order, and the output function runs once on the joined data.
+
+## Setup
+
+Install with `pipx install ab-plumber`, then follow the [Quickstart](#quickstart).
+
+## Wrapping your own function
+
+See [`docs/adapter-shim.md`](docs/adapter-shim.md).
+
+## Development workflow
+
+Follows the `ab_spatial` playbook: Discovery → POC → Tracer → MVP → Refinement.
+See `../../playbook/phases.md`.
+
+## Issue tracker
+
+GitHub Issues in [`austinbreunig/plumber`](https://github.com/austinbreunig/plumber/issues),
+via the `gh` CLI. See `docs/agents/issue-tracker.md`.

ab_plumber-0.1.0.dist-info/RECORD

@@ -0,0 +1,18 @@
+plumber/__init__.py,sha256=njBxINA1zb50rLgbcBil5rN-cAg01sKNFlsBwkKEakc,96
+plumber/check.py,sha256=3sYvKEWD3xP7k4F3N4VdWNUe7yIbandZjbIFxzgQSnU,9757
+plumber/checkpoint.py,sha256=b5L_drzx5vhm88pZ2eCvszoQywHK2i68-YWgmAejbYQ,778
+plumber/cli.py,sha256=g0OpXv7VyuhTAKdRexyRotGNT8xRwnfzfwnxEvoz8VE,2866
+plumber/entries.py,sha256=MJV0wHPCzl140qYbWyBxetcJYNX54eJNBpifux3qSe8,537
+plumber/local.py,sha256=AraWiXem79HdpsGg7eU2YSp59_xInerGFlyjKJP0yQ4,678
+plumber/localmp.py,sha256=7PnQ-eQwXvS8_k70TnawBLk8C6o7EhOfOiAeAu8xiH8,3171
+plumber/probe.py,sha256=DuozBTGSUTQE7nl9AeAKGEPJzkrl_05SdofP1cak73Y,2218
+plumber/protocols.py,sha256=KtiHYeaRNMKXHzSEOCnKI-91FTViGltypKV4wPHIuWw,1198
+plumber/resolver.py,sha256=cuS2_g6DB_3FlriSKL3EHDcFnvMNYDSLPvziXX7B_1U,726
+plumber/run.py,sha256=gJhL3INafTrho2WZMmhZWSR6FBX3u6omD1TBDpk92SA,4970
+plumber/examples/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+plumber/examples/snap_shim.py,sha256=to9tadVKD_XTpi6qtyseR50mFXTqZwRyqNN0hI9XWSQ,988
+ab_plumber-0.1.0.dist-info/METADATA,sha256=d45nm2-1aIXRGIjfaMBmVcC7u1eXeLzL5agtNRos450,5094
+ab_plumber-0.1.0.dist-info/WHEEL,sha256=W3fkpkm7-wf9vBI5Z-7s0eWkeM-spu78I8Neb98DeEg,87
+ab_plumber-0.1.0.dist-info/entry_points.txt,sha256=TNvhstZNqb-vgbBfrNjiQ8yXEUWq9HpJ0xglyZAAS4k,45
+ab_plumber-0.1.0.dist-info/licenses/LICENSE,sha256=HmOGqubXHZ0hjm0jSSvImvmAf6jI3ysSSHdbxkoGRFQ,1077
+ab_plumber-0.1.0.dist-info/RECORD,,

ab_plumber-0.1.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.32.4
+Root-Is-Purelib: true
+Tag: py3-none-any

ab_plumber-0.1.0.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+plumber = plumber.cli:main

ab_plumber-0.1.0.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 plumber contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

plumber/__init__.py

@@ -0,0 +1,3 @@
+"""plumber: geospatial pipelines with swappable execution strategies."""
+
+__version__ = "0.1.0"

plumber/check.py

@@ -0,0 +1,250 @@
+"""`plumber check`: cheap checks that run no phase code, plus the preflight inside `run`.
+
+Layers, cheapest first:
+  structural  config only, nothing imported
+  static      import + inspect.signature, no phase code runs
+  runtime     probe.py runs the phases on `check.sample` (only from `plumber check`, not `run`)
+
+Every check is a plain function `fn(config, overrides) -> list[Finding]`. To add a check,
+write one and append it to STRUCTURAL_CHECKS or STATIC_CHECKS.
+"""
+
+import inspect
+import json
+
+from plumber.entries import FAIL, PASS, SKIP, WARN, Finding, name_of, params_of, phase_label
+from plumber.probe import probe
+from plumber.resolver import resolve
+
+# Which `execution.partition:` keys each strategy understands. A key a strategy does not list
+# is a FAIL (so `worker_count` only works with `localmp`).
+STRATEGY_PARTITION_KEYS = {
+    "local": set(),
+    "localmp": {"by", "chunk_size", "worker_count"},
+}
+DEFAULT_STRATEGY = "local"
+
+
+def phase_labels(config):
+    return [phase_label(number) for number in range(1, len(config.get("phases") or []) + 1)]
+
+
+# ---- structural layer -------------------------------------------------------------------
+
+
+def check_phase_fields(config, overrides):
+    findings = []
+    for label, entry in zip(phase_labels(config), config.get("phases") or []):
+        if "path" not in entry:
+            findings.append(Finding(label, FAIL, "missing `path`"))
+        if "partitionable" not in entry:
+            findings.append(Finding(label, FAIL, "missing `partitionable` (true or false)"))
+        elif not isinstance(entry["partitionable"], bool):
+            findings.append(Finding(label, FAIL, "`partitionable` must be true or false"))
+        if not isinstance(entry.get("checkpoint", False), bool):
+            findings.append(Finding(label, FAIL, "`checkpoint` must be true or false"))
+    return findings
+
+
+def check_checkpoints_block(config, overrides):
+    block = config.get("checkpoints")
+    if block is None:
+        return []
+    if not isinstance(block, dict):
+        return [Finding("checkpoints", FAIL, "`checkpoints` must be a mapping with a `dir`")]
+    if "dir" in block and not isinstance(block["dir"], str):
+        return [Finding("checkpoints", FAIL, "`checkpoints.dir` must be a path string")]
+    return []
+
+
+def partition_value_problem(key, value):
+    """Why one partition value is wrong, or None if it is fine."""
+    if key == "by":
+        if not (isinstance(value, list) and value and all(isinstance(v, str) for v in value)):
+            return "`by` must be a list of column names"
+    elif not (isinstance(value, int) and not isinstance(value, bool) and value > 0):
+        return f"`{key}` must be a whole number above 0"
+    return None
+
+
+def check_partition_suits_strategy(config, overrides):
+    """The partition spec must suit the strategy: known keys only, and `localmp` needs
+    exactly one of them, with a sensible value."""
+    execution = config.get("execution") or {}
+    if not isinstance(execution, dict):
+        return [Finding("execution", FAIL, "`execution` must be a mapping")]
+    strategy = execution.get("strategy", DEFAULT_STRATEGY)
+    if strategy not in STRATEGY_PARTITION_KEYS:
+        known = ", ".join(sorted(STRATEGY_PARTITION_KEYS))
+        return [Finding("strategy", FAIL, f"unknown strategy {strategy!r} (known: {known})")]
+    partition = execution.get("partition") or {}
+    if not isinstance(partition, dict):
+        return [Finding("partition", FAIL, "`partition` must be a mapping")]
+    allowed = STRATEGY_PARTITION_KEYS[strategy]
+    findings = [
+        Finding("partition", FAIL, f"key {key!r} is not used by strategy {strategy!r}")
+        for key in partition
+        if key not in allowed
+    ]
+    if allowed and len(partition) != 1:
+        keys = ", ".join(sorted(allowed))
+        findings.append(Finding("partition", FAIL, f"strategy {strategy!r} needs one of: {keys}"))
+    for key, value in partition.items():
+        problem = key in allowed and partition_value_problem(key, value)
+        if problem:
+            findings.append(Finding("partition", FAIL, problem))
+    return findings
+
+
+def check_input_present(config, overrides):
+    if not config.get("input"):
+        return [Finding("input", FAIL, "missing `input:` block")]
+    return []
+
+
+STRUCTURAL_CHECKS = [
+    check_input_present,
+    check_phase_fields,
+    check_checkpoints_block,
+    check_partition_suits_strategy,
+]
+
+
+# ---- static layer -----------------------------------------------------------------------
+
+
+def entries_to_check(config):
+    """(label, entry, takes_data) for input, every phase, and output (if there is one)."""
+    entries = []
+    if config.get("input"):
+        entries.append(("input", config["input"], False))
+    for label, entry in zip(phase_labels(config), config.get("phases") or []):
+        entries.append((label, entry, True))
+    if config.get("output"):
+        entries.append(("output", config["output"], True))
+    return entries
+
+
+def check_signature(label, fn, params, takes_data):
+    """Compare the configured params with the function's signature."""
+    parameters = list(inspect.signature(fn).parameters.values())
+    if takes_data:
+        parameters = parameters[1:]  # the first argument is the data
+    accepts_any = any(p.kind is inspect.Parameter.VAR_KEYWORD for p in parameters)
+    named = [p for p in parameters if p.kind in (p.POSITIONAL_OR_KEYWORD, p.KEYWORD_ONLY)]
+    names = [p.name for p in named]
+    shown = ", ".join(str(p) for p in parameters)
+
+    findings = []
+    for key in params:
+        if key not in names and not accepts_any:
+            findings.append(Finding(label, FAIL, f"unexpected param {key!r} (signature: {shown})"))
+    for p in named:
+        if p.default is p.empty and p.name not in params:
+            findings.append(Finding(label, FAIL, f"missing required param {p.name!r}"))
+    return findings
+
+
+def check_paths_and_signatures(config, overrides):
+    findings = []
+    for label, entry, takes_data in entries_to_check(config):
+        if "path" not in entry:
+            continue  # the structural layer already reported it
+        try:
+            fn = resolve(entry["path"])
+        except Exception as error:  # importing runs user code, which can raise anything
+            findings.append(Finding(label, FAIL, f"cannot resolve {entry['path']!r}: {error}"))
+            continue
+        if callable(fn):
+            findings += check_signature(label, fn, params_of(entry), takes_data)
+        else:
+            findings.append(Finding(label, FAIL, f"{entry['path']!r} is not callable"))
+    return findings
+
+
+def check_unknown_overrides(config, overrides):
+    """A --params key that no phase lists is probably a typo."""
+    listed = {key for entry in config.get("phases") or [] for key in params_of(entry)}
+    return [
+        Finding("--params", WARN, f"{key!r}: no phase lists this param, so it is ignored")
+        for key in overrides
+        if key not in listed
+    ]
+
+
+STATIC_CHECKS = [check_paths_and_signatures, check_unknown_overrides]
+
+
+# ---- report -----------------------------------------------------------------------------
+
+
+def row_for(label, findings, config):
+    entries = {lab: entry for lab, entry, _ in entries_to_check(config)}
+    mine = [f for f in findings if f.label == label]
+    entry = entries.get(label)
+    statuses = {f.status for f in mine}
+    status = next((s for s in (FAIL, WARN, SKIP) if s in statuses), PASS)
+    return {
+        "label": label,
+        "name": name_of(entry) if entry and "path" in entry else None,
+        "status": status,
+        "messages": [f.message for f in mine],
+    }
+
+
+def run_probe(config, overrides, findings):
+    """The runtime layer. Returns (findings, note for the report's `runtime_probe` line)."""
+    sample = (config.get("check") or {}).get("sample")
+    if not sample:
+        return [], "not run (no check.sample)"
+    if any(f.status == FAIL for f in findings):
+        return [], "not run (fix the FAIL rows first)"
+    return probe(sample, config["phases"], overrides), f"ran on {sample}"
+
+
+def check(config, overrides=None, runtime=True):
+    """Run the structural, static and (if `runtime` and `check.sample` is set) runtime layers.
+
+    Returns the report as a plain dict. `run`'s preflight passes runtime=False.
+    """
+    overrides = overrides or {}
+    findings = []
+    for check_fn in STRUCTURAL_CHECKS + STATIC_CHECKS:
+        findings += check_fn(config, overrides)
+    probe_note = "not run (no check.sample)"
+    if runtime:
+        probe_findings, probe_note = run_probe(config, overrides, findings)
+        findings += probe_findings
+
+    labels = [label for label, _, _ in entries_to_check(config)]
+    labels += [f.label for f in findings if f.label not in labels]  # config-level rows
+    rows = [row_for(label, findings, config) for label in labels]
+    return {
+        "ok": all(row["status"] != FAIL for row in rows),
+        "rows": rows,
+        "runtime_probe": probe_note,
+    }
+
+
+def format_report(report):
+    lines = []
+    for row in report["rows"]:
+        name = row["name"] or ""
+        line = f"{row['label']:<10} {name:<28} {row['status']}"
+        if row["messages"]:
+            line += "   " + "; ".join(row["messages"])
+        lines.append(line.rstrip())
+    lines.append(f"runtime probe: {report['runtime_probe']}")
+    return "\n".join(lines)
+
+
+def write_report(report, path="report.json"):
+    with open(path, "w") as file:
+        json.dump(report, file, indent=2)
+
+
+def preflight(config):
+    """Run before `run`. Any FAIL stops the run; a WARN never does."""
+    report = check(config, runtime=False)
+    if not report["ok"]:
+        raise SystemExit("plumber check failed, nothing was run:\n" + format_report(report))

plumber/checkpoint.py

@@ -0,0 +1,27 @@
+"""Checkpoints: a phase's output saved as GeoParquet. Plumber writes these itself."""
+
+from pathlib import Path
+
+import geopandas as gpd
+
+DEFAULT_DIR = "./.plumber/checkpoints"
+
+
+def checkpoint_dir(run_params):
+    """The folder checkpoints go in, from the strategy's `run_params`."""
+    return run_params.get("checkpoint_dir", DEFAULT_DIR)
+
+
+def checkpoint_path(directory, name):
+    return Path(directory) / f"{name}.parquet"
+
+
+def write_checkpoint(data, directory, name):
+    """Save `data` to `<directory>/<name>.parquet`, replacing any old file."""
+    path = checkpoint_path(directory, name)
+    path.parent.mkdir(parents=True, exist_ok=True)
+    data.to_parquet(path)
+
+
+def read_checkpoint(directory, name):
+    return gpd.read_parquet(checkpoint_path(directory, name))

plumber/cli.py

@@ -0,0 +1,71 @@
+"""The `plumber` command line."""
+
+import argparse
+
+import yaml
+
+from plumber.check import check, format_report, write_report
+from plumber.run import load_config, run
+
+
+def parse_params(pairs):
+    """Turn ["a=1", "b=x"] into {"a": 1, "b": "x"}. Values are read as YAML (1 is an int)."""
+    params = {}
+    for pair in pairs:
+        key, sep, value = pair.partition("=")
+        if not (key and sep):
+            raise SystemExit(f"Bad --params {pair!r}: expected key=value")
+        params[key] = yaml.safe_load(value)
+    return params
+
+
+def apply_execution_flags(config, args):
+    """Copy --strategy and the partition flags into `config["execution"]`.
+
+    Any partition flag replaces the config's whole `partition:` block, so a flag never
+    ends up beside a different key from the file.
+    """
+    execution = dict(config.get("execution") or {})
+    if args.strategy:
+        execution["strategy"] = args.strategy
+    flags = {"by": args.by, "chunk_size": args.chunk_size, "worker_count": args.worker_count}
+    given = {key: value for key, value in flags.items() if value is not None}
+    if given:
+        execution["partition"] = given
+    if execution:
+        config["execution"] = execution
+
+
+def main(argv=None):
+    parser = argparse.ArgumentParser(prog="plumber")
+    commands = parser.add_subparsers(dest="command", required=True)
+    run_parser = commands.add_parser("run", help="run the pipeline in a config file")
+    check_parser = commands.add_parser("check", help="check the config without running phases")
+    for sub in (run_parser, check_parser):
+        sub.add_argument("--config", default="plumber.yaml", help="default: ./plumber.yaml")
+        sub.add_argument(
+            "--params",
+            nargs="+",
+            default=[],
+            metavar="key=value",
+            help="override phase params (applies to every phase that has that param)",
+        )
+        sub.add_argument("--strategy", help="execution strategy: local or localmp")
+        sub.add_argument("--by", nargs="+", metavar="column", help="localmp: one piece per value")
+        sub.add_argument("--chunk-size", type=int, help="localmp: rows per piece")
+        sub.add_argument("--worker-count", type=int, help="localmp: split into N pieces")
+    run_parser.add_argument("--from", dest="start", metavar="phase", help="start at this phase")
+    run_parser.add_argument("--to", dest="stop", metavar="phase", help="stop after this phase")
+    args = parser.parse_args(argv)
+
+    config = load_config(args.config)
+    apply_execution_flags(config, args)
+    overrides = parse_params(args.params)
+    if args.command == "check":
+        report = check(config, overrides)
+        print(format_report(report))
+        write_report(report)
+        if not report["ok"]:
+            raise SystemExit(1)
+    else:
+        run(config, overrides, args.start, args.stop)

plumber/entries.py

@@ -0,0 +1,25 @@
+"""Small readers for one config entry, plus the `Finding` shape that check and probe share."""
+
+from typing import NamedTuple
+
+PASS, WARN, FAIL, SKIP = "PASS", "WARN", "FAIL", "SKIP"
+
+
+class Finding(NamedTuple):
+    """One problem. `label` is the report row it belongs to ("input", "P1", "partition", ...)."""
+
+    label: str
+    status: str
+    message: str
+
+
+def phase_label(number):
+    return f"P{number}"
+
+
+def params_of(entry):
+    return entry.get("params") or {}
+
+
+def name_of(entry):
+    return entry.get("name", entry["path"])

plumber/examples/snap_shim.py

@@ -0,0 +1,28 @@
+"""The canonical adapter shim. Copy this pattern; see docs/adapter-shim.md.
+
+`snap_to_network` stands in for "your existing function": it does not have the phase shape.
+`run` is the shim: phase-shaped, pure, and nothing else.
+"""
+
+import geopandas as gpd
+from shapely.ops import nearest_points
+
+
+def snap_to_network(points, network, tolerance):
+    """Existing function (wrong shape for a phase): move points onto the network line."""
+    line = network.geometry.union_all()
+    snapped = []
+    for point in points.geometry:
+        on_line = nearest_points(line, point)[0]
+        snapped.append(on_line if point.distance(on_line) <= tolerance else point)
+    return points.set_geometry(snapped)
+
+
+def read_network(path):
+    return gpd.read_parquet(path)
+
+
+def run(gdf, *, network_path, tolerance):
+    """The shim. Reads its own side input from a path, keeps no state, returns a new gdf."""
+    network = read_network(network_path)
+    return snap_to_network(gdf, network, tolerance)

plumber/local.py

@@ -0,0 +1,18 @@
+"""LocalSequential: run everything in this process, one phase after another."""
+
+from plumber.checkpoint import checkpoint_dir, write_checkpoint
+from plumber.protocols import Result
+
+
+class LocalSequential:
+    def execute(self, input, steps, output, run_params):
+        input_fn, input_params = input
+        data = input_fn(**input_params)
+        for step in steps:
+            data = step.fn(data, **step.params)
+            if step.checkpoint:
+                write_checkpoint(data, checkpoint_dir(run_params), step.name)
+        if output is not None:
+            output_fn, output_params = output
+            output_fn(data, **output_params)
+        return Result(data)

plumber/localmp.py

@@ -0,0 +1,73 @@
+"""LocalMultiprocess (`localmp`): run partitionable phases on pieces, in worker processes.
+
+Flow: input -> split -> phases on pieces -> join -> (non-partitionable phase) -> split again
+-> ... -> join -> output. `split` and `join` are the two seams. Anything that needs the
+whole dataset (a non-partitionable phase, the output function, later checkpoints) goes
+through `join`.
+"""
+
+from concurrent.futures import ProcessPoolExecutor
+
+import numpy as np
+import pandas as pd
+
+from plumber.checkpoint import checkpoint_dir, write_checkpoint
+from plumber.protocols import Result
+
+
+def split(data, partition):
+    """Cut `data` into pieces by one of: `by` (columns), `chunk_size`, `worker_count`."""
+    if "by" in partition:
+        return [piece for _, piece in data.groupby(partition["by"], sort=True, dropna=False)]
+    if "chunk_size" in partition:
+        size = partition["chunk_size"]
+        return [data.iloc[start : start + size] for start in range(0, len(data), size)]
+    positions = np.array_split(np.arange(len(data)), partition["worker_count"])
+    return [data.iloc[p] for p in positions if len(p)]
+
+
+def join(pieces, order):
+    """Join pieces into one table in `order`, the index the data had before it was split.
+
+    Rows sort by where their index label sat in `order`. A phase must keep the index
+    (and it must be unique) for this to restore the input order.
+    """
+    joined = pd.concat(pieces)
+    return joined.iloc[np.argsort(order.get_indexer(joined.index), kind="stable")]
+
+
+def whole(data, pieces, order):
+    """All the data in input order: `data` if not split, else the joined pieces."""
+    return data if pieces is None else join(pieces, order)
+
+
+class LocalMultiprocess:
+    def __init__(self, partition):
+        self.partition = partition
+
+    def execute(self, input, steps, output, run_params):
+        input_fn, input_params = input
+        data = input_fn(**input_params)
+        workers = self.partition.get("worker_count")
+        pieces = None  # set while the data is split; None means `data` holds everything
+        order = None  # the index of `data` when it was split
+        with ProcessPoolExecutor(max_workers=workers) as pool:
+            for step in steps:
+                if step.partitionable:
+                    if pieces is None:
+                        order = data.index
+                        pieces = split(data, self.partition)
+                    futures = [pool.submit(step.fn, piece, **step.params) for piece in pieces]
+                    pieces = [future.result() for future in futures]
+                else:
+                    data, pieces = whole(data, pieces, order), None
+                    data = step.fn(data, **step.params)
+                if step.checkpoint:
+                    # Gather point: whole data in input order, write one file, split again.
+                    data, pieces = whole(data, pieces, order), None
+                    write_checkpoint(data, checkpoint_dir(run_params), step.name)
+        data = whole(data, pieces, order)
+        if output is not None:
+            output_fn, output_params = output
+            output_fn(data, **output_params)
+        return Result(data)

plumber/probe.py

@@ -0,0 +1,57 @@
+"""The runtime probe: run the phases on a small sample file and report what went wrong.
+
+Only `plumber check` calls this. It runs user code on real data, so `run`'s preflight skips it.
+It returns `Finding`s, like the other layers.
+"""
+
+import geopandas as gpd
+
+from plumber.entries import FAIL, SKIP, WARN, Finding, params_of, phase_label
+from plumber.resolver import resolve
+
+
+def probe_phase(fn, params, partitionable, data):
+    """Call one phase. Returns (output, [(status, message), ...]). Output is None on a crash."""
+    try:
+        output = fn(data.copy(), **params)
+    except Exception as error:  # user code can raise anything
+        return None, [(FAIL, f"{type(error).__name__}: {error}")]
+
+    results = []
+    if partitionable and not output.equals(fn(data.copy(), **params)):
+        results.append((FAIL, "different output on identical input (shared state?)"))
+    if getattr(output, "crs", None) != getattr(data, "crs", None):
+        results.append((WARN, f"CRS changed from {data.crs} to {output.crs}"))
+    dropped = [column for column in data.columns if column not in output.columns]
+    if dropped:
+        results.append((WARN, f"columns dropped: {', '.join(map(str, dropped))}"))
+    return output, results
+
+
+def probe(sample, phases, overrides):
+    """Chain the sample through `phases` (config entries) in order.
+
+    After a crash every later phase is SKIP.
+    """
+    try:
+        data = gpd.read_parquet(sample)
+    except Exception as error:
+        return [Finding("check.sample", FAIL, f"cannot read {sample!r}: {error}")]
+
+    findings = []
+    crashed = False
+    for number, entry in enumerate(phases, start=1):
+        label = phase_label(number)
+        if crashed:
+            findings.append(Finding(label, SKIP, "upstream phase failed"))
+            continue
+        params = {key: overrides.get(key, value) for key, value in params_of(entry).items()}
+        output, results = probe_phase(
+            resolve(entry["path"]), params, entry["partitionable"], data
+        )
+        findings += [Finding(label, status, message) for status, message in results]
+        if output is None:
+            crashed = True
+        else:
+            data = output
+    return findings

plumber/protocols.py

@@ -0,0 +1,48 @@
+"""The shapes every part of plumber agrees on: phases, steps, results, strategies."""
+
+from dataclasses import dataclass
+from typing import Any, Callable, Generic, Optional, Protocol, TypeVar
+
+Payload = TypeVar("Payload")
+
+
+class PhaseModule(Protocol[Payload]):
+    """A resolved phase: a plain function `run(data, **params) -> data`."""
+
+    def __call__(self, data: Payload, **params: Any) -> Payload: ...
+
+
+@dataclass(frozen=True)
+class Step:
+    """One configured phase, ready to run."""
+
+    name: str
+    fn: Callable
+    params: dict
+    partitionable: bool
+    checkpoint: bool
+
+
+class Result(Generic[Payload]):
+    """What a strategy hands back. Only `.collect()` is public."""
+
+    def __init__(self, data: Payload):
+        self._data = data
+
+    def collect(self) -> Payload:
+        return self._data
+
+
+# A pre-bound (function, params) pair. The strategy calls `fn(**params)` (input)
+# or `fn(data, **params)` (output).
+Bound = tuple[Callable, dict[str, Any]]
+
+
+class ExecutionStrategy(Protocol[Payload]):
+    def execute(
+        self,
+        input: Bound,
+        steps: list[Step],
+        output: Optional[Bound],
+        run_params: dict[str, Any],
+    ) -> Result[Payload]: ...

plumber/resolver.py

@@ -0,0 +1,21 @@
+"""Turn a "pkg.mod:attr" string into the callable it names."""
+
+import importlib
+import os
+import sys
+from typing import Callable
+
+
+def resolve(path: str) -> Callable:
+    """Import `pkg.mod` and return its `attr`. The `:attr` part is required."""
+    module_name, sep, attr = path.partition(":")
+    if not (module_name and sep and attr):
+        raise ValueError(f"Bad path {path!r}: expected 'module:attr' (':attr' is required)")
+    cwd = os.getcwd()
+    if cwd not in sys.path:
+        sys.path.insert(0, cwd)
+    module = importlib.import_module(module_name)
+    try:
+        return getattr(module, attr)
+    except AttributeError:
+        raise ValueError(f"Module {module_name!r} has no attribute {attr!r}") from None

plumber/run.py

@@ -0,0 +1,133 @@
+"""Read ./plumber.yaml, resolve the user's functions, and hand them to a strategy."""
+
+import time
+import warnings
+from pathlib import Path
+
+import yaml
+
+from plumber.check import DEFAULT_STRATEGY, preflight
+from plumber.checkpoint import DEFAULT_DIR, checkpoint_path, read_checkpoint
+from plumber.entries import name_of, params_of
+from plumber.local import LocalSequential
+from plumber.localmp import LocalMultiprocess
+from plumber.protocols import Step
+from plumber.resolver import resolve
+
+
+def load_config(path="plumber.yaml"):
+    return yaml.safe_load(Path(path).read_text())
+
+
+def bind(entry):
+    """Turn an `input:`/`output:` block into a (function, params) pair."""
+    return resolve(entry["path"]), params_of(entry)
+
+
+def build_step(entry, overrides):
+    """Build one Step. CLI overrides replace config params, key by key (shallow merge).
+
+    An override only applies to phases that already list that key in their params.
+    """
+    params = params_of(entry)
+    params = {key: overrides.get(key, value) for key, value in params.items()}
+    return Step(
+        name=name_of(entry),
+        fn=resolve(entry["path"]),
+        params=params,
+        partitionable=entry["partitionable"],
+        checkpoint=entry.get("checkpoint", False),
+    )
+
+
+def build_steps(config, overrides):
+    """Build every Step in config order. Names must be unique.
+
+    Warns about `--params` keys that no phase lists (likely typos).
+    """
+    entries = config["phases"]
+    first_seen = {}
+    for number, entry in enumerate(entries, start=1):
+        name = name_of(entry)
+        if name in first_seen:
+            raise ValueError(
+                f"Duplicate phase name {name!r}: entries {first_seen[name]} and {number}. "
+                "Give each a distinct `name:`."
+            )
+        first_seen[name] = number
+
+    listed = {key for entry in entries for key in params_of(entry)}
+    for key in overrides:
+        if key not in listed:
+            warnings.warn(
+                f"--params {key}: no phase lists this param, so it was ignored", stacklevel=2
+            )
+
+    return [build_step(entry, overrides) for entry in entries]
+
+
+def build_strategy(config):
+    """Pick the execution strategy from `execution:` (default `local`); `check` vetted it."""
+    execution = config.get("execution") or {}
+    if execution.get("strategy", DEFAULT_STRATEGY) == "localmp":
+        return LocalMultiprocess(execution["partition"])
+    return LocalSequential()
+
+
+def phase_index(steps, name, flag):
+    names = [step.name for step in steps]
+    if name not in names:
+        raise SystemExit(f"{flag} {name}: no such phase. Phases are: {', '.join(names)}")
+    return names.index(name)
+
+
+def checkpoint_input(steps, first, directory):
+    """The input pair for `--from`: read the checkpoint of the phase before `first`."""
+    previous = steps[first - 1].name
+    path = checkpoint_path(directory, previous)
+    if not path.exists():
+        raise SystemExit(
+            f"--from {steps[first].name}: no checkpoint for {previous!r} at {path}. "
+            f"Set `checkpoint: true` on {previous} and run through it once."
+        )
+    saved = time.strftime("%Y-%m-%d %H:%M", time.localtime(path.stat().st_mtime))
+    warnings.warn(
+        f"Starting at {steps[first].name} from checkpoint {path.name} (saved {saved})",
+        stacklevel=3,
+    )
+    return read_checkpoint, {"directory": directory, "name": previous}
+
+
+def run(config, overrides=None, start=None, stop=None):
+    """Run the pipeline. This never reads data; the strategy calls the user's functions.
+
+    `start`/`stop` are phase names (`--from`/`--to`). This function alone rewrites the call:
+    a later `start` swaps the input for the previous checkpoint, and an earlier `stop`
+    drops the output. The strategy just gets an ordinary `execute`.
+    """
+    preflight(config)
+    input_pair = bind(config["input"])
+    all_steps = build_steps(config, overrides or {})
+    if config.get("output"):
+        output_pair = bind(config["output"])
+    else:
+        warnings.warn("No `output:` in config: result discarded", stacklevel=2)
+        output_pair = None
+    directory = (config.get("checkpoints") or {}).get("dir", DEFAULT_DIR)
+
+    first = phase_index(all_steps, start, "--from") if start else 0
+    last = phase_index(all_steps, stop, "--to") if stop else len(all_steps) - 1
+    if first > last:
+        raise SystemExit(f"--from {start} comes after --to {stop}")
+    if first > 0:
+        input_pair = checkpoint_input(all_steps, first, directory)
+    if last < len(all_steps) - 1:
+        output_pair = None
+        if not all_steps[last].checkpoint:
+            warnings.warn(
+                f"--to {stop}: {stop} has no checkpoint: result discarded; runs anyway",
+                stacklevel=2,
+            )
+    steps = all_steps[first : last + 1]
+    run_params = {"checkpoint_dir": directory}
+    return build_strategy(config).execute(input_pair, steps, output_pair, run_params)