harborforge 1.0.0__tar.gz

harborforge-1.0.0/.github/workflows/ci.yml

@@ -0,0 +1,25 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+        with:
+          python-version: "3.13"
+      - name: Install dependencies
+        run: uv sync --extra dev
+      - name: Format check
+        run: uv run ruff format --check .
+      - name: Lint
+        run: uv run ruff check .
+      - name: Type check
+        run: uv run mypy harborforge/
+      - name: Tests
+        run: uv run pytest tests/ -v

harborforge-1.0.0/.gitignore

@@ -0,0 +1,19 @@
+.DS_Store
+.*
+!.github
+!.github/**
+!.gitignore
+requirements.txt
+__pycache__
+*.pyc
+*.pyo
+*.pyd
+*.pyw
+*.pyz
+*.pywz
+*.pyzw
+*.pyzwz
+trials/
+jobs/
+registry.json
+.docs/

harborforge-1.0.0/CLAUDE.md

@@ -0,0 +1,136 @@
+# CLAUDE.md — g9
+
+## What this is
+
+**g9** maps evaluation benchmark datasets into [Harbor](https://github.com/laude-institute/harbor)-compatible task directories, enabling large-scale parallel agent evaluation.
+
+The core package (`harborforge/`) is benchmark-agnostic. Each benchmark gets its own adapter in `adapters/`. The first reference implementation is DSGym.
+
+## Repo layout
+
+```
+harborforge/          # pip-installable core — abstract contracts only
+  mapper.py           # DataMapper base class (setup, map, run)
+  run.py              # Generic Harbor trial runner (wraps harbor jobs start)
+  registry.py         # generate_registry() — produces registry.json
+  handlers/
+    base.py           # DatasetHandler base class + verifier templates
+
+adapters/             # concrete benchmark implementations
+  dsgym/
+    mapper.py         # DSGymMapper(DataMapper)
+    base_image.py     # BASE_IMAGE_NAME, BASE_IMAGE constants
+    eval.Dockerfile   # shared base Docker image — built by `just build-base`
+    run_trial.py      # thin wrapper: calls harborforge.run with DSGym config
+    handlers/         # one file per DSGym dataset type
+    download.py       # Kaggle dataset downloader
+
+tests/
+  harborforge/        # unit + integration tests for core
+  adapters/dsgym/     # tests for the DSGym adapter
+
+tools/
+  benchmark_builds.py # Docker build-time benchmark (base image vs no base)
+
+.data/                # local data (gitignored)
+  task/               # raw DSGym task files (.json / .jsonl)
+  data/               # raw data files (CSVs, .h5ad, etc.)
+  tasks/              # generated Harbor task directories
+```
+
+## Setup
+
+```sh
+pip install uv          # Python package manager
+brew install just       # task runner
+
+just setup              # create venv + install deps + download data
+just build-base         # build shared Docker base image (run once)
+just data               # download raw data + run mapper → .data/tasks/
+```
+
+Credentials go in `.env` (gitignored):
+```
+ANTHROPIC_API_KEY=sk-ant-...
+KAGGLE_TOKEN=KGAT_...      # for DSPredict tasks
+```
+
+## Running things
+
+```sh
+just trial daeval/0               # single task
+just trial daeval -l 20           # 20 tasks from a dataset
+just trial dspredict/titanic      # Kaggle competition task
+just trial daeval/0 -k 3          # 3 attempts
+just test                         # pytest
+just format                       # ruff + mypy
+just view                         # open Harbor trajectory viewer
+just benchmark                    # Docker build-time benchmark
+```
+
+## Adding a new benchmark adapter
+
+1. Create `adapters/<name>/` with:
+   - `handlers/` — one `DatasetHandler` subclass per dataset type
+   - `mapper.py` — `DataMapper` subclass implementing `iter_tasks()`
+   - `__main__.py` — calls `DataMapper().run(output_dir, registry_path)`
+   - `run_trial.py` — calls `harborforge.run.run()` with your registry
+
+2. Implement the handler contract (see table below).
+
+3. Add a `just data <name>` call — the existing recipe dispatches to `python -m adapters.<name>`.
+
+See `adapters/dsgym/` as the reference implementation.
+
+## Handler contract
+
+Every handler subclasses `harborforge.handlers.DatasetHandler`:
+
+| Method | Required | Purpose |
+|--------|----------|---------|
+| `instruction(task_data)` | ✅ | Content for `instruction.md` — no answer leakage |
+| `test_sh(task_data)` | ✅ | Content for `tests/test.sh` — must write float reward to `/logs/verifier/reward.txt` |
+| `dockerfile(task_data)` | ✅ | Content for `environment/Dockerfile` |
+| `setup()` | optional | Download/prepare data for this dataset |
+| `data_files(task_data)` | optional | Local files to COPY into the image build context |
+| `artifacts()` | optional | Container paths to capture after trial |
+| `verifier_env_keys()` | optional | Env var keys to forward to the SEPARATE verifier |
+| `verifier_dockerfile(task_data)` | optional | Non-None triggers SEPARATE verifier mode |
+
+## Harbor task contract
+
+Each generated task directory must contain:
+
+```
+<task>/
+├── instruction.md          # shown to the agent (no answer)
+├── task.toml               # Harbor config (timeouts, env)
+├── environment/
+│   └── Dockerfile          # agent container — data files go here
+└── tests/
+    ├── test.sh             # verifier script — writes reward to /logs/verifier/reward.txt
+    └── Dockerfile          # (SEPARATE mode only) verifier container
+```
+
+## Verifier modes
+
+**SHARED** (default): verifier runs inside the agent container after the agent finishes.
+
+**SEPARATE**: verifier runs in its own isolated container. Triggered when `handler.verifier_dockerfile()` returns non-None. Harbor re-materializes agent artifacts at their original source paths inside the verifier container — e.g. `/app/submission.csv` in the agent maps to `/app/submission.csv` in the verifier.
+
+## Code style
+
+- Simple and concise over clever. Three similar lines beat a premature abstraction.
+- No comments on obvious code. Only comment on non-obvious invariants or workarounds.
+- No dead code. No backwards-compat shims, no unused parameters.
+- Validate at system boundaries only. Trust internal code.
+- Type hints on all public APIs.
+- `harborforge/` must stay benchmark-agnostic. No benchmark-specific strings belong there.
+- Base image: always import `BASE_IMAGE` from `adapters/<name>/base_image.py`. Never hardcode `FROM python:...` in handlers.
+
+## Python conventions
+
+- Python 3.13+, managed with `uv`.
+- `pyproject.toml` is the source of truth for dependencies.
+- Tests live in `tests/`, mirror the package structure, use `pytest`.
+- Formatters: `ruff` for linting/formatting, `mypy` for types. Run `just format` before committing.

harborforge-1.0.0/PKG-INFO

@@ -0,0 +1,8 @@
+Metadata-Version: 2.4
+Name: harborforge
+Version: 1.0.0
+Requires-Python: ==3.13.*
+Provides-Extra: dev
+Requires-Dist: mypy>=1.10; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: ruff>=0.8; extra == 'dev'

harborforge-1.0.0/dist/.gitignore

@@ -0,0 +1 @@
+*

harborforge-1.0.0/harborforge/__init__.py

@@ -0,0 +1,6 @@
+"""harborforge — abstract toolkit for mapping benchmark datasets to Harbor task directories."""
+
+from .handlers.base import DatasetHandler
+from .mapper import DataMapper
+
+__all__ = ["DataMapper", "DatasetHandler"]

harborforge-1.0.0/harborforge/handlers/__init__.py

@@ -0,0 +1,19 @@
+from .base import (
+    DatasetHandler,
+    case_insensitive_verifier,
+    exact_match_verifier,
+    list_verifier,
+    no_verifier,
+    numeric_verifier,
+    script_verifier,
+)
+
+__all__ = [
+    "DatasetHandler",
+    "case_insensitive_verifier",
+    "exact_match_verifier",
+    "list_verifier",
+    "no_verifier",
+    "numeric_verifier",
+    "script_verifier",
+]

harborforge-1.0.0/harborforge/handlers/base.py

@@ -0,0 +1,231 @@
+"""
+DatasetHandler contract.
+
+Each dataset subclass must implement:
+  - instruction(task_data)  → str   content for instruction.md (shown to the agent, no answer leakage)
+  - test_sh(task_data)      → str   content for tests/test.sh (must write reward to /logs/verifier/reward.txt)
+  - dockerfile(task_data)   → str   content for environment/Dockerfile
+
+Optional overrides:
+  - data_files(task_data)   → list[tuple[Path, str]]  (local_path, dest_in_build_context) pairs
+  - task_toml(task_id)      → str   content for task.toml
+"""
+
+from abc import ABC, abstractmethod
+from pathlib import Path
+from typing import Any
+
+
+class DatasetHandler(ABC):
+    dataset_name: str
+
+    @abstractmethod
+    def instruction(self, task_data: dict[str, Any]) -> str:
+        """Returns instruction.md content — what the agent sees. Must not include the answer."""
+
+    @abstractmethod
+    def test_sh(self, task_data: dict[str, Any]) -> str:
+        """Returns tests/test.sh content. Must write a float reward to /logs/verifier/reward.txt."""
+
+    @abstractmethod
+    def dockerfile(self, task_data: dict[str, Any]) -> str:
+        """Returns environment/Dockerfile content."""
+
+    def verifier_dockerfile(self, task_data: dict[str, Any]) -> str | None:
+        """Return a Dockerfile for a SEPARATE verifier container, or None for SHARED (default).
+
+        When non-None, Harbor routes the trial through SEPARATE mode: the verifier runs in
+        its own isolated container and receives agent artifacts as input. Use this when the
+        verifier needs different dependencies or ground truth data isolated from the agent.
+        """
+        return None
+
+    def setup(self) -> None:
+        """Download or prepare local data required by this dataset. No-op by default."""
+
+    def artifacts(self) -> list[str]:
+        """Container paths to capture as artifacts after a trial. Empty by default."""
+        return []
+
+    def verifier_env_keys(self) -> list[str]:
+        """Environment variable keys to forward to the SEPARATE verifier container."""
+        return []
+
+    def data_files(self, task_data: dict[str, Any]) -> list[tuple[Path, str]]:
+        """
+        Returns (local_path, dest_in_build_context) pairs for files to COPY into the image.
+        dest_in_build_context is relative to environment/ (e.g. 'data/train.csv').
+        """
+        return []
+
+    def task_toml(self, task_id: str) -> str:
+        return f"""\
+version = "1.0"
+
+[task]
+name = "{task_id}"
+
+[metadata]
+
+[verifier]
+timeout_sec = 900.0
+
+[agent]
+timeout_sec = 900.0
+
+[environment]
+build_timeout_sec = 600.0
+"""
+
+
+# ---------------------------------------------------------------------------
+# Verifier shell script templates
+# ---------------------------------------------------------------------------
+
+
+def exact_match_verifier(answer: Any) -> str:
+    """Reward 1.0 if agent output matches expected answer (whitespace-stripped string comparison)."""
+    safe = str(answer).replace("'", "'\\''")
+    return f"""\
+#!/bin/bash
+set -e
+mkdir -p /logs/verifier
+
+EXPECTED='{safe}'
+SUBMISSION=/app/submission.txt
+
+if [ ! -f "$SUBMISSION" ]; then
+    echo "0" > /logs/verifier/reward.txt
+    exit 0
+fi
+
+ACTUAL=$(cat "$SUBMISSION" | tr -d '[:space:]')
+EXPECTED_CLEAN=$(echo "$EXPECTED" | tr -d '[:space:]')
+
+if [ "$ACTUAL" = "$EXPECTED_CLEAN" ]; then
+    echo "1" > /logs/verifier/reward.txt
+else
+    echo "0" > /logs/verifier/reward.txt
+fi
+"""
+
+
+def numeric_verifier(answer: Any, tolerance: float = 0.05) -> str:
+    """Reward 1.0 if agent output is within `tolerance` (relative) of expected numeric value."""
+    safe = str(answer).replace("'", "'\\''")
+    return f"""\
+#!/bin/bash
+set -e
+mkdir -p /logs/verifier
+
+EXPECTED='{safe}'
+
+python3 - "$EXPECTED" <<'PYEOF'
+import sys
+
+try:
+    expected = float(sys.argv[1])
+    with open("/app/submission.txt") as f:
+        actual = float(f.read().strip())
+    denom = abs(expected) if expected != 0 else 1.0
+    reward = 1.0 if abs(actual - expected) / denom <= {tolerance} else 0.0
+except Exception:
+    reward = 0.0
+
+with open("/logs/verifier/reward.txt", "w") as f:
+    f.write(str(reward))
+PYEOF
+"""
+
+
+def list_verifier(answer: Any) -> str:
+    """Reward 1.0 if agent output contains all expected items (order-insensitive, case-insensitive).
+    Expected answer should be a Python list repr or comma-separated string."""
+    safe = str(answer).replace("'", "'\\''")
+    return f"""\
+#!/bin/bash
+set -e
+mkdir -p /logs/verifier
+
+python3 <<'PYEOF'
+import ast, re
+
+EXPECTED_RAW = '{safe}'
+try:
+    with open("/app/submission.txt") as f:
+        actual_raw = f.read().strip()
+except FileNotFoundError:
+    open("/logs/verifier/reward.txt", "w").write("0")
+    raise SystemExit
+
+def parse_list(s):
+    try:
+        val = ast.literal_eval(s)
+        if isinstance(val, list):
+            return {{str(x).strip().lower() for x in val}}
+    except Exception:
+        pass
+    return {{x.strip().lower() for x in re.split(r"[,\\n]+", s) if x.strip()}}
+
+expected = parse_list(EXPECTED_RAW)
+actual = parse_list(actual_raw)
+reward = 1.0 if expected and expected == actual else 0.0
+open("/logs/verifier/reward.txt", "w").write(str(reward))
+PYEOF
+"""
+
+
+def case_insensitive_verifier(answer: Any) -> str:
+    """Reward 1.0 if agent output matches expected answer, ignoring case and whitespace."""
+    safe = str(answer).replace("'", "'\\''")
+    return f"""\
+#!/bin/bash
+set -e
+mkdir -p /logs/verifier
+
+EXPECTED='{safe}'
+SUBMISSION=/app/submission.txt
+
+if [ ! -f "$SUBMISSION" ]; then
+    echo "0" > /logs/verifier/reward.txt
+    exit 0
+fi
+
+ACTUAL=$(cat "$SUBMISSION" | tr '[:upper:]' '[:lower:]' | tr -d '[:space:]')
+EXPECTED_CLEAN=$(echo "$EXPECTED" | tr '[:upper:]' '[:lower:]' | tr -d '[:space:]')
+
+if [ "$ACTUAL" = "$EXPECTED_CLEAN" ]; then
+    echo "1" > /logs/verifier/reward.txt
+else
+    echo "0" > /logs/verifier/reward.txt
+fi
+"""
+
+
+def script_verifier(script_container_path: str) -> str:
+    """Runs a Python eval script already present in the container.
+    The script is fully responsible for writing a float reward to /logs/verifier/reward.txt."""
+    return f"""\
+#!/bin/bash
+set -e
+mkdir -p /logs/verifier
+
+SCRIPT="{script_container_path}"
+
+if [ ! -f "$SCRIPT" ]; then
+    echo "0" > /logs/verifier/reward.txt
+    exit 0
+fi
+
+python3 "$SCRIPT" || echo "0" > /logs/verifier/reward.txt
+"""
+
+
+def no_verifier(reason: str = "External evaluation required") -> str:
+    """Always writes reward 0. Used for tasks that require external scoring (e.g. leaderboards)."""
+    return f"""\
+#!/bin/bash
+# {reason}
+mkdir -p /logs/verifier
+echo "0" > /logs/verifier/reward.txt
+"""

harborforge-1.0.0/harborforge/mapper.py

@@ -0,0 +1,142 @@
+"""
+DataMapper — abstract base for benchmark → Harbor task directory pipelines.
+
+Subclass and implement `iter_tasks()`. Override `setup()` to handle data
+downloads or prep. Call `run()` as the standard entry point, or `map()`
+directly for more control.
+"""
+
+import shutil
+import stat
+from abc import ABC, abstractmethod
+from collections.abc import Iterator
+from pathlib import Path
+from typing import Any
+
+from .handlers.base import DatasetHandler
+
+
+class DataMapper(ABC):
+    """
+    Abstract mapper from a benchmark dataset to Harbor task directories.
+
+    Minimal adapter implementation:
+      1. Implement `iter_tasks()` to yield tasks.
+      2. Optionally override `setup()` to download/prepare raw data.
+      3. Call `run()` as the entry point — it handles setup, mapping, and registry.
+    """
+
+    # ---------------------------------------------------------------------------
+    # Public API
+    # ---------------------------------------------------------------------------
+
+    def setup(self) -> None:
+        """Download or prepare raw data before mapping. No-op by default."""
+
+    def map(self, output_dir: Path, registry_path: Path | None = None) -> int:
+        """
+        Write all tasks to output_dir as Harbor task directories.
+        Wipes and recreates output_dir on each call.
+        If registry_path is given, auto-generates a Harbor registry.json.
+        Returns the number of tasks written.
+        """
+        if output_dir.exists():
+            shutil.rmtree(output_dir)
+
+        total = 0
+        current_dataset = ""
+        for task_id, dir_name, handler, task_data in self.iter_tasks():
+            self._write_task(output_dir / dir_name, task_id, handler, task_data)
+            total += 1
+            dataset = dir_name.split("/")[0]
+            if dataset != current_dataset:
+                if current_dataset:
+                    print()
+                current_dataset = dataset
+                print(f"  ↳ {dataset}", end="", flush=True)
+            print(f"\r  ↳ {dataset} ({total} written)", end="", flush=True)
+        if total:
+            print()
+
+        if registry_path is not None:
+            from harborforge.registry import generate_registry
+
+            counts = generate_registry(output_dir, registry_path)
+            n_datasets = len(counts)
+            n_tasks = sum(counts.values())
+            print(f"✅ {registry_path} — {n_tasks} tasks across {n_datasets} datasets")
+
+        return total
+
+    def run(self, output_dir: Path, registry_path: Path | None = None) -> None:
+        """
+        Standard entry point: setup() → map() → optional registry.
+        Adapters call this from __main__.py.
+        """
+        self.setup()
+        total = self.map(output_dir, registry_path)
+        print(f"📊 Total tasks written: {total}")
+
+    # ---------------------------------------------------------------------------
+    # Abstract
+    # ---------------------------------------------------------------------------
+
+    @abstractmethod
+    def iter_tasks(self) -> Iterator[tuple[str, str, DatasetHandler, dict[str, Any]]]:
+        """
+        Yield (task_id, dir_name, handler, raw_task_data) for each task.
+
+        - task_id:   unique identifier used in task.toml name field
+        - dir_name:  relative path under output_dir (e.g. 'daeval/0')
+        - handler:   DatasetHandler instance for this dataset
+        - raw_task_data: raw dict from the source benchmark file
+        """
+
+    # ---------------------------------------------------------------------------
+    # Internal
+    # ---------------------------------------------------------------------------
+
+    def _write_task(
+        self,
+        task_dir: Path,
+        task_id: str,
+        handler: DatasetHandler,
+        task_data: dict[str, Any],
+    ) -> None:
+        task_dir.mkdir(parents=True, exist_ok=True)
+
+        (task_dir / "instruction.md").write_text(handler.instruction(task_data), encoding="utf-8")
+
+        env_dir = task_dir / "environment"
+        env_dir.mkdir(exist_ok=True)
+
+        for local_path, dest_name in handler.data_files(task_data):
+            dest = env_dir / dest_name
+            dest.parent.mkdir(parents=True, exist_ok=True)
+            shutil.copy2(local_path, dest)
+
+        (env_dir / "Dockerfile").write_text(handler.dockerfile(task_data), encoding="utf-8")
+
+        # Auto-detect verifier mode: SEPARATE if handler provides a verifier Dockerfile.
+        # Harbor uses tests/ as the verifier build context in SEPARATE mode.
+        task_toml = handler.task_toml(task_id)
+        verifier_df = handler.verifier_dockerfile(task_data)
+        if verifier_df is not None:
+            task_toml += "\n[verifier.environment]\nbuild_timeout_sec = 300.0\n"
+
+        (task_dir / "task.toml").write_text(task_toml, encoding="utf-8")
+
+        tests_dir = task_dir / "tests"
+        tests_dir.mkdir(exist_ok=True)
+        test_sh = tests_dir / "test.sh"
+        test_sh.write_text(handler.test_sh(task_data), encoding="utf-8")
+        test_sh.chmod(test_sh.stat().st_mode | stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH)
+
+        # SEPARATE mode: write verifier Dockerfile into tests/ (Harbor's verifier build context).
+        # COPY test.sh into the image so Harbor can execute /tests/test.sh inside the container.
+        if verifier_df is not None:
+            verifier_df_with_copy = (
+                verifier_df.rstrip()
+                + "\nCOPY test.sh /tests/test.sh\nRUN chmod +x /tests/test.sh\n"
+            )
+            (tests_dir / "Dockerfile").write_text(verifier_df_with_copy, encoding="utf-8")

harborforge-1.0.0/harborforge/registry.py

@@ -0,0 +1,61 @@
+"""
+Generate a local Harbor registry.json from a directory of Harbor task directories.
+
+The tasks directory is expected to have the structure:
+    tasks_dir/
+        dataset_a/
+            task_0/
+            task_1/
+        dataset_b/
+            task_0/
+
+Each top-level subdirectory becomes a named dataset in the registry.
+"""
+
+import json
+from pathlib import Path
+
+
+def generate_registry(
+    tasks_dir: Path,
+    output_path: Path,
+    version: str = "1.0",
+    description_template: str = "{name} benchmark tasks",
+) -> dict[str, int]:
+    """
+    Scan tasks_dir and write a Harbor registry.json to output_path.
+
+    Returns a dict of {dataset_name: task_count}.
+    """
+    if not tasks_dir.exists():
+        raise FileNotFoundError(f"Tasks directory not found: {tasks_dir}")
+
+    datasets = []
+    counts: dict[str, int] = {}
+
+    for dataset_dir in sorted(tasks_dir.iterdir()):
+        if not dataset_dir.is_dir():
+            continue
+
+        tasks = [
+            {"name": task.name, "path": str(task)}
+            for task in sorted(dataset_dir.iterdir())
+            if task.is_dir()
+        ]
+
+        if not tasks:
+            continue
+
+        name = dataset_dir.name.lower()
+        datasets.append(
+            {
+                "name": name,
+                "version": version,
+                "description": description_template.format(name=dataset_dir.name),
+                "tasks": tasks,
+            }
+        )
+        counts[name] = len(tasks)
+
+    output_path.write_text(json.dumps(datasets, indent=2))
+    return counts