bathos 0.3.0__tar.gz → 0.4.0__tar.gz

{bathos-0.3.0 → bathos-0.4.0}/CHANGELOG.md

@@ -5,6 +5,16 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+### Fixed
+
+- **`bth sync` indefinite hang** — `sync_catalog()` now passes SSH options (`ConnectTimeout=10`, `BatchMode=yes`) to rsync so unreachable hosts fail in ≤10 s instead of blocking forever; adds `subprocess.run(timeout=120)` as a safety net; raises `RuntimeError` with actionable message on timeout; drops silent `--progress` flag (output was captured, not displayed)
+
+- **Parquet timestamp timezone-aware fix** — Ensure timestamps loaded from Parquet are timezone-aware (UTC) to avoid comparisons between naive and aware datetimes in query functions; prevents TypeError and incorrect filtering.
+
+---
+
 ## [0.3.0] - 2026-05-20
 
 ### Added

{bathos-0.3.0 → bathos-0.4.0}/CLAUDE.md

@@ -10,15 +10,21 @@ bathos is a standalone experiment tracking CLI for a single researcher across 10
 
 ---
 
-## Current Status (as of 2026-05-15)
+## Current Status (as of 2026-05-21)
 
-**v0.1: complete and merged to main.** 44 tests passing.
+**v0.4.0: complete and merged to main.** 333 tests passing.
 
 - Full design spec: `.praxia/specs/bathos-design.md`
 - v0.1 implementation plan: `.praxia/specs/bathos-v01-plan.md`
-- Backlog: items #124–139 in praxia DB (see below); #124–127 done
+- Backlog: items #124–142 in praxia DB (see below); #124–141 done
 
-**Next sprint candidates (P2):** #128 (FastMCP), #130 (bth check) + #128 unblocked once 130 done, #138 (bth sync), #139 (bth compact / warm tier), #131 (SLURM integration), #132 (bth new-experiment).
+**Shipped in v0.2:** FastMCP server (#128), `@bth.experiment` decorator (#129), `bth check` (#130), SLURM `_bth_env.sh` (#131), `bth new-experiment` (#132), `uv tool` packaging (#133), `bth lint` (#134), `bth migrate` (#135), `bth sync` (#138), schema versioning v2 (#140), `bth archive` (#141), `bth remote` subcommands, `bth export`, `bth catalog-version`.
+
+**Shipped in v0.3:** Agentic integrity gate (`--agent-mode`, `--no-sidecar`, `--derived-from`), lineage tracking (`bth lineage`), campaigns (`bth campaign` subcommands + MCP tools), sprint audit (`bth sprint-audit`), Tier-2 lint checks, schema v3 with campaign/lineage/integrity fields, `--campaign` flag on `bth run`.
+
+**Shipped in v0.4:** Per-project sync filtering — `bth sync` now pushes/pulls only the current project's runs (no cross-project contamination). Cool-tier layout changed to `runs/<slug>/run_<uuid>.parquet`. New command `bth migrate-to-project-subdirs` migrates flat catalogs. `sync_filter` config knob; `--no-project-filter` opt-out planned.
+
+**Remaining backlog:** #136 (bth-migrate praxia workflow), #137 (global instruction portability), #142 (results management design).
 
 ---
 
@@ -30,7 +36,7 @@ bathos is a standalone experiment tracking CLI for a single researcher across 10
 SLURM job / bth run
       │
       ▼  atomic write-then-rename
- cool/   ~/.bth/catalog/runs/run_<uuid>.parquet   ← minimal schema, SLURM-safe
+ cool/   ~/.bth/catalog/runs/<slug>/run_<uuid>.parquet   ← minimal schema, SLURM-safe
       │
  bth compact  (lazy on query or explicit)
       │
@@ -52,7 +58,7 @@ Cool schema is intentionally minimal (13 provenance fields). Warm adds `metadata
 
 ### Pre-Registration Schemas (per script directory)
 
-Every script in `scripts/experiments/` and `scripts/benchmarks/` must have a companion sidecar **`<script-stem>.bth.toml`** declaring its hypothesis, expected outcomes, and result schema before `bth run` will execute it.
+Every script in `scripts/experiments/` and `scripts/benchmarks/` should have a companion sidecar **`<script-stem>.bth.toml`** declaring its hypothesis, expected outcomes, and result schema. In v0.2+, `bth run` enforces this by default; pass `--no-sidecar` to bypass.
 
 **Sidecar format (experiments):**
 ```toml
@@ -118,7 +124,7 @@ fix = "str"
 | Catalog storage | **DuckDB + Parquet** (not pyiceberg) | pyiceberg is for distributed data lakes; DuckDB+Parquet is simpler, no overhead |
 | FastMCP | Mirror CLI tool-for-tool | `cli.py` and `mcp.py` both thin layers over same core modules |
 
-### Source Layout (planned)
+### Source Layout
 
 ```
 src/bathos/
@@ -133,8 +139,20 @@ src/bathos/
   init.py         # bth init logic
   runner.py       # bth run subprocess wrapper
   query.py        # list_runs, get_run, find_runs, run_sql
-  checker.py      # bth check — git-drift validity (v0.2)
-  sidecar.py      # .bth.toml sidecar parse, content-hash, outcome eval (v0.2)
+  checker.py      # bth check — git-drift freshness gate
+  sidecar.py      # .bth.toml sidecar parse, content-hash, outcome eval
+  campaigns.py    # bth campaign subcommands + Campaign dataclass
+  sprint_audit.py # bth sprint-audit cross-project audit
+  linter.py       # bth lint — Tier-1 and Tier-2 checks
+  migrate.py      # bth migrate — schema version upgrades
+  new_experiment.py # bth new-experiment scaffold
+  export.py       # bth export — skill + MCP registration
+  validate.py     # sidecar structural validation
+  remote.py       # bth remote subcommands
+  sync.py         # bth sync rsync wrapper
+  archive.py      # bth archive warm→cold export
+  prereg.py       # agentic integrity pre-registration gate
+  decorators.py   # @bth.experiment provenance decorator
   templates/
     _bth_env.sh   # SLURM env helper template
     experiment.py # bth new-experiment script skeleton
@@ -150,25 +168,24 @@ src/bathos/
 | 125 | `bth init` — dirs, .bth.toml, catalog bootstrap, SLURM config | P1 | 124 |
 | 126 | `bth run` — CLI wrapper, provenance capture | P1 | 124 |
 | 127 | `bth ls` / `show` / `find` / `sql` — query interface | P1 | 126 |
-| 128 | FastMCP server — mirrors CLI tool-for-tool | P1 | 126, 127, 130 |
-| 129 | `@bth.experiment` — provenance decorator for Typer scripts | P2 | 124 |
-| 130 | `bth check` — freshness/validity vs git HEAD | P2 | 126 |
-| 131 | SLURM `_bth_env.sh` integration | P2 | 125, 126 |
-| 132 | `bth new-experiment` — Typer + sidecar scaffold | P2 | 125 |
-| 133 | `uv tool` packaging + release | P3 | 126–130 |
-| 134 | Script convention linter | P3 | — |
-| 135 | `bth migrate` — Phase 1 mechanical (existing projects) | P2 | 125 |
-| 136 | `bth-migrate` praxia workflow — agentic classification + git mv plan | P2 | 135 |
-| 137 | Global instruction portability (separate design session needed) | P2 | — |
-| 138 | `bth sync` — rsync cool-tier catalog to/from cluster remote | P2 | 125, v0.1 done |
-| 139 | `bth compact` + warm-tier DuckDB (`bth sql` catalog queries) | P2 | v0.1 done |
-| 140 | Schema versioning + extended provenance (hostname, slurm_job_id, metadata JSON, migrations) | P2 | 139 |
-| 141 | `bth archive` — warm→cold partitioned Parquet export | P3 | 139, 140 |
-| 142 | Results management — output convention, file-count utilities, direct management interface design | P2 | 139 |
+| 128 | FastMCP server — mirrors CLI tool-for-tool | P1 ✓ done (v0.2) | 126, 127, 130 |
+| 129 | `@bth.experiment` — provenance decorator for Typer scripts | P2 ✓ done (v0.2) | 124 |
+| 130 | `bth check` — freshness/validity vs git HEAD | P2 ✓ done (v0.1/v0.2) | 126 |
+| 131 | SLURM `_bth_env.sh` integration | P2 ✓ done (v0.1) | 125, 126 |
+| 132 | `bth new-experiment` — Typer + sidecar scaffold | P2 ✓ done (v0.2) | 125 |
+| 133 | `uv tool` packaging + release | P3 ✓ done (v0.2) | 126–130 |
+| 134 | Script convention linter | P3 ✓ done (v0.2) | — |
+| 135 | `bth migrate` — Phase 1 mechanical (existing projects) | P2 ✓ done (v0.2) | 125 |
+| 136 | `bth-migrate` praxia workflow — agentic classification + git mv plan | P2 (not started) | 135 |
+| 137 | Global instruction portability (separate design session needed) | P2 (not started) | — |
+| 138 | `bth sync` — rsync cool-tier catalog to/from cluster remote | P2 ✓ done (v0.2) | 125, v0.1 done |
+| 139 | `bth compact` + warm-tier DuckDB (`bth sql` catalog queries) | P2 ✓ done (v0.1) | v0.1 done |
+| 140 | Schema versioning + extended provenance (hostname, slurm_job_id, metadata JSON, migrations) | P2 ✓ done (v0.2) | 139 |
+| 141 | `bth archive` — warm→cold partitioned Parquet export | P3 ✓ done (v0.2) | 139, 140 |
+| 142 | Results management — output convention, file-count utilities, direct management interface design | P2 (not started) | 139 |
 
 **Not yet backlogged:**
-- Sidecar pre-registration enforcement in `bth run` (add as P2, depends on 126)
-- Outcome evaluation + `runs.outcome` column in warm DuckDB (add as P2, depends on 139)
+- *(none — sidecar enforcement and outcome evaluation shipped in v0.2/v0.3)*
 
 ---
 

{bathos-0.3.0 → bathos-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: bathos
-Version: 0.3.0
+Version: 0.4.0
 Summary: Local-first, zero-server experiment tracking for researchers
 Project-URL: Homepage, https://github.com/mariellerossi/bathos
 Project-URL: Documentation, https://bathos.readthedocs.io
@@ -66,13 +66,59 @@ bth run scripts/experiments/benchmark_tip3p.py -- --n-steps 1000 --out outputs/r
 bth ls --since 7d
 bth show <run-id>
 
-# Check if a result is still valid (compares recorded git hash to HEAD)
-bth check outputs/run1.parquet
+# Check catalog runs for git-drift freshness
+bth check [--status <status>] [--check-outputs]
 
 # Escape hatch: raw DuckDB SQL against the catalog
 bth sql "SELECT project_slug, count(*) FROM runs GROUP BY 1"
 ```
 
+### `bth run` flags
+
+**v0.3+ flags:**
+- `--agent-mode collaborative|autonomous` — declare whether this run is agent-driven
+- `--derived-from <run-id>` — record parent run for lineage tracking
+- `--campaign <campaign-id>` — associate this run with a campaign
+- `--no-sidecar` — bypass sidecar enforcement (use for ad-hoc runs)
+
+## CLI Reference
+
+**Core commands:**
+- `bth init` — initialize bathos in a project
+- `bth run` — execute and track an experiment
+- `bth ls` — list recent runs
+- `bth show <run-id>` — display run details
+- `bth find` — search runs by criteria
+- `bth sql` — raw DuckDB query against catalog
+- `bth check` — check catalog runs for git-drift freshness
+- `bth lineage <run-id>` — show ancestor chain following parent_run_id links
+- `bth sprint-audit [--hours N]` — audit recent runs across all registered projects
+- `bth lint [--project-root PATH]` — check scripts/ for naming conventions and missing sidecars
+- `bth new-experiment <name>` — scaffold a new experiment script and sidecar
+- `bth migrate` — upgrade cool-tier Parquet fragments to current schema version
+- `bth migrate-to-project-subdirs [--dry-run]` — move flat catalog runs into `runs/<slug>/` subdirs (v0.4+)
+- `bth export` — export the using-bathos skill and register MCP server
+
+**`bth sync`** — Sync cool-tier catalog to/from cluster (v0.4+: per-project filtered)
+- `bth sync [<remote>]` — push only this project's runs to the remote (filtered by `project_slug`)
+- `bth sync [<remote>] --pull` — pull only this project's runs from the remote
+- Output: `Pushed 47 runs (filtered 275 from other projects) to 'engaging' in 1.2s`
+- Config: `sync_filter = "none"` in `.bth.toml` disables filtering (pushes all projects)
+
+**`bth remote`** — Manage sync remotes
+- `bth remote add <name> <url>` — add an SSH remote for catalog sync
+- `bth remote list` — list configured remotes
+- `bth remote remove <name>` — remove a remote
+- `bth remote test <name>` — test SSH connectivity to a remote
+
+**`bth campaign`** — Manage experiment campaigns (v0.3+)
+- `bth campaign create <id> --hypothesis <text>` — create a new campaign
+- `bth campaign add <run-id> --campaign <id>` — associate a run with a campaign
+- `bth campaign ls` — list campaigns
+- `bth campaign show <id>` — show campaign details and runs
+- `bth campaign review <id>` — statistical summary and anomaly detection
+- `bth campaign conclude <id> --outcome <label>` — close campaign with outcome
+
 ## Script conventions
 
 `bth init` scaffolds the following structure in your project:

{bathos-0.3.0 → bathos-0.4.0}/README.md

@@ -30,13 +30,59 @@ bth run scripts/experiments/benchmark_tip3p.py -- --n-steps 1000 --out outputs/r
 bth ls --since 7d
 bth show <run-id>
 
-# Check if a result is still valid (compares recorded git hash to HEAD)
-bth check outputs/run1.parquet
+# Check catalog runs for git-drift freshness
+bth check [--status <status>] [--check-outputs]
 
 # Escape hatch: raw DuckDB SQL against the catalog
 bth sql "SELECT project_slug, count(*) FROM runs GROUP BY 1"
 ```
 
+### `bth run` flags
+
+**v0.3+ flags:**
+- `--agent-mode collaborative|autonomous` — declare whether this run is agent-driven
+- `--derived-from <run-id>` — record parent run for lineage tracking
+- `--campaign <campaign-id>` — associate this run with a campaign
+- `--no-sidecar` — bypass sidecar enforcement (use for ad-hoc runs)
+
+## CLI Reference
+
+**Core commands:**
+- `bth init` — initialize bathos in a project
+- `bth run` — execute and track an experiment
+- `bth ls` — list recent runs
+- `bth show <run-id>` — display run details
+- `bth find` — search runs by criteria
+- `bth sql` — raw DuckDB query against catalog
+- `bth check` — check catalog runs for git-drift freshness
+- `bth lineage <run-id>` — show ancestor chain following parent_run_id links
+- `bth sprint-audit [--hours N]` — audit recent runs across all registered projects
+- `bth lint [--project-root PATH]` — check scripts/ for naming conventions and missing sidecars
+- `bth new-experiment <name>` — scaffold a new experiment script and sidecar
+- `bth migrate` — upgrade cool-tier Parquet fragments to current schema version
+- `bth migrate-to-project-subdirs [--dry-run]` — move flat catalog runs into `runs/<slug>/` subdirs (v0.4+)
+- `bth export` — export the using-bathos skill and register MCP server
+
+**`bth sync`** — Sync cool-tier catalog to/from cluster (v0.4+: per-project filtered)
+- `bth sync [<remote>]` — push only this project's runs to the remote (filtered by `project_slug`)
+- `bth sync [<remote>] --pull` — pull only this project's runs from the remote
+- Output: `Pushed 47 runs (filtered 275 from other projects) to 'engaging' in 1.2s`
+- Config: `sync_filter = "none"` in `.bth.toml` disables filtering (pushes all projects)
+
+**`bth remote`** — Manage sync remotes
+- `bth remote add <name> <url>` — add an SSH remote for catalog sync
+- `bth remote list` — list configured remotes
+- `bth remote remove <name>` — remove a remote
+- `bth remote test <name>` — test SSH connectivity to a remote
+
+**`bth campaign`** — Manage experiment campaigns (v0.3+)
+- `bth campaign create <id> --hypothesis <text>` — create a new campaign
+- `bth campaign add <run-id> --campaign <id>` — associate a run with a campaign
+- `bth campaign ls` — list campaigns
+- `bth campaign show <id>` — show campaign details and runs
+- `bth campaign review <id>` — statistical summary and anomaly detection
+- `bth campaign conclude <id> --outcome <label>` — close campaign with outcome
+
 ## Script conventions
 
 `bth init` scaffolds the following structure in your project:

{bathos-0.3.0 → bathos-0.4.0}/agent_assets/docs/claude-code-integration.md

@@ -38,14 +38,18 @@ The skill is loaded automatically at session start in Claude Code. In a new sess
 
 | MCP Tool | CLI equivalent | Status |
 |----------|---------------|--------|
-| `run_script` | `bth run` | ✅ Available |
+| `run` | `bth run` | ✅ Available |
 | `list_runs` | `bth ls` | ✅ Available |
 | `get_run` | `bth show <id>` | ✅ Available |
 | `find_runs` | `bth find` | ✅ Available |
 | `run_sql` | `bth sql` | ✅ Available |
 | `compact_catalog` | `bth compact` | ✅ Available |
-| `check_runs` | `bth check` | 🔜 Coming in v0.2 |
-| `archive_runs` | `bth archive` | 🔜 Coming in v0.2 |
+| `check` | `bth check` | ✅ Available |
+| `archive` | `bth archive` | ✅ Available |
+| `campaign_create` | `bth campaign create` | ✅ Available |
+| `campaign_list` | `bth campaign ls` | ✅ Available |
+| `campaign_review` | `bth campaign review` | ✅ Available |
+| `campaign_conclude` | `bth campaign conclude` | ✅ Available |
 
 ## Update
 

{bathos-0.3.0 → bathos-0.4.0}/agent_assets/using_bathos/SKILL.md

@@ -13,7 +13,7 @@ description: "Use when working in any bathos-tracked research project — experi
 
 **Scope:** Single researcher, 10+ projects, SLURM cluster integration.
 
-**Status:** ✅ v0.2 shipped — 207 tests passing. Full CLI + MCP server available.
+**Status:** ✅ v0.4.0 shipped — 333 tests passing. Full CLI + MCP server available, including campaigns, agentic integrity gates, and per-project sync filtering.
 
 ---
 
@@ -34,7 +34,7 @@ description: "Use when working in any bathos-tracked research project — experi
 | Command | Arguments | Status | Notes |
 |---------|-----------|--------|-------|
 | bth init | --slug, --remote, --slurm-partition | ✅ | Project initialization |
-| bth run | <script> [-- args], --tag, --out, --campaign, --agent-mode, --no-sidecar | ✅ | Execute + provenance capture |
+| bth run | <script> [-- args], --tag, --out, --campaign, --agent-mode, --derived-from, --no-sidecar | ✅ | Execute + provenance capture; v0.3: `--agent-mode` (collaborative/autonomous), `--derived-from` (lineage), `--campaign` (campaign association), `--no-sidecar` (bypass enforcement) |
 | bth ls | --since, --status, --limit, --project | ✅ | List recent runs + OUTCOME column |
 | bth show | <run-id> | ✅ | Full run details + git state |
 | bth find | --project, --since, --status, --tag, --output-file | ✅ | Flexible filtered query |

{bathos-0.3.0 → bathos-0.4.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "bathos"
-version = "0.3.0"
+version = "0.4.0"
 description = "Local-first, zero-server experiment tracking for researchers"
 readme = "README.md"
 requires-python = ">=3.12"

{bathos-0.3.0 → bathos-0.4.0}/src/bathos/__init__.py

@@ -1,4 +1,4 @@
-__version__ = "0.3.0"
+__version__ = "0.4.0"
 
 from bathos.decorators import experiment
 

{bathos-0.3.0 → bathos-0.4.0}/src/bathos/catalog.py

@@ -15,7 +15,7 @@ def init_catalog(catalog_dir: Path) -> None:
 
 def write_run(run: Run, catalog_dir: Path) -> None:
     """Write (or overwrite) a run record atomically."""
-    runs_dir = catalog_dir / "runs"
+    runs_dir = catalog_dir / "runs" / run.project_slug
     runs_dir.mkdir(parents=True, exist_ok=True)
     target = runs_dir / f"run_{run.id}.parquet"
     tmp = runs_dir / f"run_{run.id}.tmp.parquet"
@@ -28,7 +28,7 @@ def read_runs(catalog_dir: Path) -> list[Run]:
     runs_dir = catalog_dir / "runs"
     if not runs_dir.exists():
         return []
-    parquet_files = list(runs_dir.glob("run_*.parquet"))
+    parquet_files = list(runs_dir.rglob("run_*.parquet"))
     if not parquet_files:
         return []
     tables = [pq.read_table(f) for f in parquet_files]

{bathos-0.3.0 → bathos-0.4.0}/src/bathos/cli.py

@@ -595,9 +595,10 @@ def sync(
 
     try:
         result = sync_catalog(remote, config, catalog_dir, pull=pull)
-        direction = "pulled" if pull else "pushed"
+        direction = "Pulled" if pull else "Pushed"
+        filter_msg = f" (filtered {result.filtered} from other projects)" if result.filtered > 0 else ""
         typer.echo(
-            f"{direction.capitalize()} {result.transferred} runs to/from '{result.remote}' in {result.duration_s:.1f}s"
+            f"{direction} {result.transferred} runs{filter_msg} to/from '{result.remote}' in {result.duration_s:.1f}s"
         )
     except ValueError as e:
         typer.echo(str(e), err=True)
@@ -624,6 +625,27 @@ def migrate(
         typer.echo("  Nothing to migrate.")
 
 
+@app.command("migrate-to-project-subdirs")
+def migrate_to_subdirs_cmd(
+    dry_run: bool = typer.Option(False, "--dry-run", help="Show what would be moved without writing"),
+):
+    """Move flat cool-tier run parquets into per-project subdirectories.
+
+    Reads each run's project_slug and moves it to runs/<slug>/run_<uuid>.parquet.
+    Run this on both local and remote before using per-project sync filtering.
+    """
+    from bathos.migrate import migrate_to_project_subdirs
+
+    result = migrate_to_project_subdirs(_catalog_dir(), dry_run=dry_run)
+    action = "Would move" if dry_run else "Moved"
+    typer.echo(f"{action} {result.moved} run(s) into per-project subdirectories.")
+    if result.skipped:
+        typer.echo(f"  {result.skipped} already in place (skipped).")
+    if result.by_slug:
+        for slug, count in sorted(result.by_slug.items()):
+            typer.echo(f"  {slug}: {count}")
+
+
 @app.command("sprint-audit")
 def sprint_audit_cmd(
     hours: int = typer.Option(24, "--hours", help="Lookback window in hours"),

{bathos-0.3.0 → bathos-0.4.0}/src/bathos/compact.py

@@ -180,7 +180,7 @@ def _fragment_count(catalog_dir: Path) -> int:
     runs_dir = catalog_dir / "runs"
     if not runs_dir.exists():
         return 0
-    return len(list(runs_dir.glob("run_*.parquet")))
+    return len(list(runs_dir.rglob("run_*.parquet")))
 
 
 def should_compact(catalog_dir: Path) -> bool:

{bathos-0.3.0 → bathos-0.4.0}/src/bathos/config.py

@@ -14,6 +14,7 @@ class ProjectConfig:
     catalog_dir: Path = field(default_factory=lambda: Path.home() / ".bth" / "catalog")
     remotes: dict[str, dict] = field(default_factory=dict)
     slurm: dict = field(default_factory=dict)
+    sync_filter: str = "project_slug"
 
 
 def default_catalog_dir() -> Path:
@@ -38,6 +39,7 @@ def load_project_config(path: Path) -> ProjectConfig:
         catalog_dir=Path(project["catalog_dir"]) if "catalog_dir" in project else default_catalog_dir(),
         remotes=data.get("remotes", {}),
         slurm=data.get("slurm", {}),
+        sync_filter=project.get("sync_filter", "project_slug"),
     )
 
 

{bathos-0.3.0 → bathos-0.4.0}/src/bathos/mcp.py

@@ -316,6 +316,7 @@ def sync_tool(
             "transferred": result.transferred,
             "duration_s": result.duration_s,
             "remote": result.remote,
+            "filtered": result.filtered,
         }
         return json.dumps(result_dict, indent=2)
     except Exception as e:

{bathos-0.3.0 → bathos-0.4.0}/src/bathos/migrate.py

@@ -24,7 +24,7 @@ def migrate_catalog(catalog_dir: Path, dry_run: bool = False) -> MigrateResult:
     if not runs_dir.exists():
         return MigrateResult(scanned=0, already_current=0, migrated=0, dry_run=dry_run)
 
-    fragments = list(runs_dir.glob("run_*.parquet"))
+    fragments = list(runs_dir.rglob("run_*.parquet"))
     current_names = {f.name for f in COOL_SCHEMA}
     migrated = 0
 
@@ -62,6 +62,61 @@ def migrate_catalog(catalog_dir: Path, dry_run: bool = False) -> MigrateResult:
     )
 
 
+@dataclass
+class MigrateToSubdirsResult:
+    moved: int
+    skipped: int
+    by_slug: dict
+    dry_run: bool
+
+
+def migrate_to_project_subdirs(catalog_dir: Path, dry_run: bool = False) -> MigrateToSubdirsResult:
+    """Move flat cool-tier run parquets into per-project subdirectories.
+
+    Reads project_slug from each parquet and moves it to runs/<slug>/run_<uuid>.parquet.
+    Files already in a subdir are skipped. Idempotent.
+    Falls back to 'default' for parquets without a readable project_slug.
+    """
+    from collections import Counter
+
+    runs_dir = catalog_dir / "runs"
+    if not runs_dir.exists():
+        return MigrateToSubdirsResult(moved=0, skipped=0, by_slug={}, dry_run=dry_run)
+
+    # Only consider directly-nested parquets (flat layout), not already-subdir'd ones
+    flat_files = [f for f in runs_dir.glob("run_*.parquet")]
+
+    moves = []
+    for parquet in flat_files:
+        slug = _read_project_slug(parquet) or "default"
+        dst = runs_dir / slug / parquet.name
+        moves.append((parquet, dst))
+
+    if not dry_run:
+        for src, dst in moves:
+            dst.parent.mkdir(parents=True, exist_ok=True)
+            src.rename(dst)
+
+    by_slug = dict(Counter(dst.parent.name for _, dst in moves))
+    return MigrateToSubdirsResult(
+        moved=len(moves),
+        skipped=len(flat_files) - len(moves),
+        by_slug=by_slug,
+        dry_run=dry_run,
+    )
+
+
+def _read_project_slug(parquet: Path) -> str | None:
+    """Read project_slug from the first row of a parquet file."""
+    try:
+        tbl = pq.read_table(parquet, columns=["project_slug"])
+        if tbl.num_rows > 0:
+            return tbl["project_slug"][0].as_py()
+    except Exception:
+        pass
+    return None
+
+
 def _default_array(arrow_type: pa.DataType, n: int) -> pa.Array:
     if pa.types.is_string(arrow_type) or pa.types.is_large_string(arrow_type):
         return pa.array([""] * n, type=arrow_type)

{bathos-0.3.0 → bathos-0.4.0}/src/bathos/schema.py

@@ -137,6 +137,9 @@ class Run:
         ts = pydict["timestamp"][i]
         if not isinstance(ts, datetime):
             ts = ts.as_py()
+        # Ensure timestamp is timezone-aware in UTC for internal comparisons
+        if ts.tzinfo is None:
+            ts = ts.replace(tzinfo=UTC)
         return cls(
             id=pydict["id"][i],
             project_slug=pydict["project_slug"][i],

{bathos-0.3.0 → bathos-0.4.0}/src/bathos/sync.py

@@ -7,6 +7,8 @@ from pathlib import Path
 
 from bathos.config import ProjectConfig
 
+_SYNC_TIMEOUT_S = 120  # rsync kill switch: fail if no completion within 2 minutes
+
 
 @dataclass
 class SyncResult:
@@ -15,6 +17,7 @@ class SyncResult:
     transferred: int
     duration_s: float
     remote: str
+    filtered: int = 0
 
 
 def sync_catalog(
@@ -43,8 +46,27 @@ def sync_catalog(
     host = remote_config["host"]
     remote_root = remote_config["remote_root"]
 
-    local_runs = catalog_dir / "runs"
-    remote_runs = f"{host}:{remote_root}/.bth/catalog/runs/"
+    project_slug = config.slug
+    sync_filter = getattr(config, "sync_filter", "project_slug")
+
+    if sync_filter == "project_slug":
+        local_runs = catalog_dir / "runs" / project_slug
+        remote_runs = f"{host}:{remote_root}/.bth/catalog/runs/{project_slug}/"
+        # Count filtered runs (total in catalog minus this project's runs)
+        runs_root = catalog_dir / "runs"
+        if not pull and runs_root.exists():
+            total = len(list(runs_root.rglob("run_*.parquet")))
+            project_count = len(list(local_runs.glob("run_*.parquet"))) if local_runs.exists() else 0
+            filtered = total - project_count
+        else:
+            filtered = 0
+    else:
+        local_runs = catalog_dir / "runs"
+        remote_runs = f"{host}:{remote_root}/.bth/catalog/runs/"
+        filtered = 0
+
+    if not pull:
+        local_runs.mkdir(parents=True, exist_ok=True)
 
     if pull:
         src = remote_runs
@@ -53,17 +75,30 @@ def sync_catalog(
         src = str(local_runs) + "/"
         dst = remote_runs
 
-    # Construct rsync command
+    # ConnectTimeout=10: fail in 10s if host is unreachable rather than hanging
+    # BatchMode=yes: never prompt for a password — fail immediately instead
+    ssh_opts = "ssh -o ConnectTimeout=10 -o BatchMode=yes"
+
     cmd = [
         "rsync",
-        "-azP",
+        "-az",
+        "--partial",
+        f"-e{ssh_opts}",
         "--ignore-existing",
         src,
         dst,
     ]
 
     start_time = time.time()
-    result = subprocess.run(cmd, capture_output=True, text=True)
+    try:
+        result = subprocess.run(
+            cmd, capture_output=True, text=True, timeout=_SYNC_TIMEOUT_S
+        )
+    except subprocess.TimeoutExpired:
+        raise RuntimeError(
+            f"bth sync timed out after {_SYNC_TIMEOUT_S}s — "
+            "check that the remote host is reachable and SSH keys are configured"
+        )
     duration_s = time.time() - start_time
 
     if result.returncode != 0:
@@ -73,7 +108,7 @@ def sync_catalog(
     # rsync outputs something like "Number of regular files transferred: 42"
     transferred = _parse_transferred_count(result.stdout)
 
-    return SyncResult(transferred=transferred, duration_s=duration_s, remote=remote_name)
+    return SyncResult(transferred=transferred, duration_s=duration_s, remote=remote_name, filtered=filtered)
 
 
 def _parse_transferred_count(rsync_output: str) -> int:

{bathos-0.3.0 → bathos-0.4.0}/tests/test_catalog.py

@@ -24,7 +24,7 @@ def test_write_and_read_single_run(tmp_catalog: Path, sample_run: Run):
 def test_write_creates_parquet_file(tmp_catalog: Path, sample_run: Run):
     init_catalog(tmp_catalog)
     write_run(sample_run, tmp_catalog)
-    files = list((tmp_catalog / "runs").glob("*.parquet"))
+    files = list((tmp_catalog / "runs").rglob("*.parquet"))
     assert len(files) == 1
     assert sample_run.id in files[0].name
 

{bathos-0.3.0 → bathos-0.4.0}/tests/test_cli.py

@@ -186,8 +186,8 @@ def test_sql_allows_arbitrary_queries(tmp_path: Path, monkeypatch):
     # Create a run to have Parquet files
     runner.invoke(app, ["run", sys.executable, "--", "-c", "pass"])
 
-    # Query using read_parquet should work without warm DB
-    glob = str(catalog / "runs" / "run_*.parquet")
+    # Query using read_parquet should work without warm DB (runs are in per-project subdir)
+    glob = str(catalog / "runs" / "testproj" / "run_*.parquet")
     result = runner.invoke(app, ["sql", f"SELECT COUNT(*) FROM read_parquet('{glob}')"])
     assert result.exit_code == 0
     assert "1" in result.output

{bathos-0.3.0 → bathos-0.4.0}/tests/test_cluster_workflow.py

@@ -119,7 +119,7 @@ remote_root = "~/projects/testproj"
     write_run(run2, catalog)
 
     # Verify we have 2 cool fragments
-    run_files = list((catalog / "runs").glob("run_*.parquet"))
+    run_files = list((catalog / "runs").rglob("run_*.parquet"))
     assert len(run_files) == 2, f"Expected 2 run files, got {len(run_files)}"
 
     # Step 3: Compact cool fragments → warm DuckDB
@@ -177,7 +177,7 @@ remote_root = "~/projects/testproj"
 
         # Verify rsync command structure
         assert cmd[0] == "rsync"
-        assert "-azP" in cmd
+        assert "-az" in cmd
         assert "--ignore-existing" in cmd
         # Should have source and destination with runs/ directories
         assert any("runs/" in str(arg) for arg in cmd)
@@ -238,7 +238,7 @@ def test_cluster_workflow_with_slurm_job_id(git_repo: Path, monkeypatch):
     write_run(run, catalog)
 
     # Verify SLURM_JOB_ID is in the written fragment
-    run_files = list((catalog / "runs").glob("run_*.parquet"))
+    run_files = list((catalog / "runs").rglob("run_*.parquet"))
     assert len(run_files) == 1
 
     # Compact and verify slurm_job_id is preserved in warm tier