outerloop-science 0.1.0.dev4__tar.gz → 0.1.2__tar.gz

{outerloop_science-0.1.0.dev4 → outerloop_science-0.1.2}/CHANGELOG.md

@@ -6,6 +6,66 @@ Versions follow [SemVer](https://semver.org).
 
 ## [Unreleased]
 
+## [0.1.2] - 2026-09-11
+
+### Fixed
+
+- A GPU benchmark on local compute aborted at its first gate measure with "no GPU lane is configured". The measure placement now follows the launch placement's rule: local compute has no lanes, the job runs on the machine's own GPUs.
+
+## [0.1.1] - 2026-09-11
+
+### Fixed
+
+- Wakes never recorded a launch as ended. A nested `import time` in the wake function shadowed the module import, so the ledger step failed on every wake, and the failure was only logged to a stderr the scheduler discards. The PR's experiments table stayed empty and `history` showed every launch as not back. The nested imports are gone, the best-effort wrapper now logs the redacted traceback, and a wake writes its kernel log to `runs/<run>/kernel.log`.
+
+### Changed
+
+- The PyPI project page and the Python-version badge now read correctly: the package metadata lists the supported Python version (3.12) and the project's audience and topic. The README's Python badge is a static `3.12+` so it renders regardless of the release's metadata.
+
+### Deprecated
+
+- The `AUTORESEARCH_*` environment names and `.autoresearch` paths are still accepted. The 0.1.0 notes said they would go in 0.1.1; a bug-fix release should not also break a deployment's environment, so their removal moves to 0.2.0.
+
+## [0.1.0] - 2026-09-10
+
+The first release of Outerloop — autoresearch agents that improve your benchmark
+on your own code, your keys, and your compute. Built and used daily by the
+Agentic Learning AI Lab at NYU.
+
+### Highlights
+
+- **The loop.** An agent proposes a change, runs the experiment on your cluster,
+  measures it against the base tree at the same seed, and opens a pull request
+  only when the benchmark actually improves. Every attempt gets a short report,
+  negative results included.
+- **Verification first.** Outerloop re-measures every claim against the
+  repository's contract — it never trusts a number the agent reports — and a
+  reviewer panel reads the change and the claim before a pull request stands.
+- **Your guardrails apply.** Agents cannot touch the benchmark, the budgets, or
+  your CI; your branch protection and required checks apply to them as to any
+  contributor. A pull request waits for a human by default; a repo can also let
+  clean ones merge themselves.
+- **Runs where you do.** Slurm or a single GPU machine, behind one compute seam.
+  Sessions run in a scrubbed sandbox, and your model key and bot credentials
+  never leave the job.
+- **Onboarding in a few commands.** `outerloop init` writes the config and sets
+  up auth (a per-adopter GitHub App, or a token), `outerloop start` runs the
+  loop, and `outerloop upgrade` moves a local install to the newest release. The
+  container image is pulled, not built.
+- **Contracts and the board.** A repo declares its benchmark, scope, and budgets
+  in `.outerloop.yaml`; the climb board publishes every attempt and its outcome,
+  and the ledger keeps the honest record — refusals and negatives included.
+- **Research lines and depth.** Each agent works its own line and reintegrates
+  the moving base; a run hibernates through long experiments and wakes to read
+  its results.
+- **Advisory review and a weekly digest.** The reviewer posts structured,
+  advisory findings on every pull request, and a weekly maintainer scan files a
+  codebase-health digest.
+
+The `AUTORESEARCH_*` environment names and `.autoresearch` paths from before the
+rename are still accepted in 0.1; 0.1.1 kept them (see its notes) and 0.2.0
+removes them. The full history since the first pre-release follows.
+
 ### Changed
 
 - Install guide: a lab member who is not an org owner now has documented steps

outerloop_science-0.1.2/CITATION.cff

@@ -0,0 +1,17 @@
+cff-version: 1.2.0
+message: "If you use Outerloop in your research, please cite it."
+title: Outerloop
+abstract: >-
+  Outerloop runs AI agents on your own research code: an agent proposes a
+  change, runs the experiment on your cluster, and opens a pull request only when
+  your benchmark improves. Every claim is re-measured against the repository's
+  contract, every attempt is recorded, and it runs on your keys, your compute,
+  and your repositories.
+type: software
+authors:
+  - name: "Agentic Learning AI Lab, New York University"
+repository-code: "https://github.com/outerloop-science/outerloop"
+url: "https://outerloop.science"
+license: Apache-2.0
+version: 0.1.2
+date-released: "2026-09-11"

{outerloop_science-0.1.0.dev4 → outerloop_science-0.1.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: outerloop-science
-Version: 0.1.0.dev4
+Version: 0.1.2
 Summary: Autonomous research agents that improve the benchmark you point them at, one verified pull request at a time
 Project-URL: Homepage, https://outerloop.science
 Project-URL: Repository, https://github.com/outerloop-science/outerloop
@@ -10,6 +10,11 @@ Author: Agentic Learning AI Lab, New York University
 License-Expression: Apache-2.0
 License-File: LICENSE
 License-File: NOTICE
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
 Requires-Python: >=3.12
 Requires-Dist: cryptography>=42
 Requires-Dist: pydantic>=2
@@ -26,7 +31,7 @@ Description-Content-Type: text/markdown
 
 [![ci](https://github.com/outerloop-science/outerloop/actions/workflows/ci.yml/badge.svg)](https://github.com/outerloop-science/outerloop/actions/workflows/ci.yml)
 [![PyPI](https://img.shields.io/pypi/v/outerloop-science.svg)](https://pypi.org/project/outerloop-science/)
-[![Python](https://img.shields.io/pypi/pyversions/outerloop-science.svg)](https://pypi.org/project/outerloop-science/)
+[![Python](https://img.shields.io/badge/python-3.12+-blue.svg)](https://pypi.org/project/outerloop-science/)
 [![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)
 
 **Autoresearch agents that improve your benchmark.**

{outerloop_science-0.1.0.dev4 → outerloop_science-0.1.2}/README.md

@@ -7,7 +7,7 @@
 
 [![ci](https://github.com/outerloop-science/outerloop/actions/workflows/ci.yml/badge.svg)](https://github.com/outerloop-science/outerloop/actions/workflows/ci.yml)
 [![PyPI](https://img.shields.io/pypi/v/outerloop-science.svg)](https://pypi.org/project/outerloop-science/)
-[![Python](https://img.shields.io/pypi/pyversions/outerloop-science.svg)](https://pypi.org/project/outerloop-science/)
+[![Python](https://img.shields.io/badge/python-3.12+-blue.svg)](https://pypi.org/project/outerloop-science/)
 [![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)
 
 **Autoresearch agents that improve your benchmark.**

{outerloop_science-0.1.0.dev4 → outerloop_science-0.1.2}/RELEASING.md

@@ -8,8 +8,8 @@
 
 ## Cutting a release
 
-1. PR: bump `__version__` and move the `[Unreleased]` entries under the new
-   version. A dev or rc pre-release still bumps the version (PyPI never
+1. PR: bump `__version__`, set `CITATION.cff`'s `version` and `date-released`,
+   and move the `[Unreleased]` entries under the new version. A dev or rc pre-release still bumps the version (PyPI never
    accepts a version twice, so the next one is `.dev1`, `rc2`, ...) but leaves
    `[Unreleased]` in place until the final release.
 2. `git tag vX.Y.Z && git push origin vX.Y.Z`. The `release` workflow builds

{outerloop_science-0.1.0.dev4 → outerloop_science-0.1.2}/pyproject.toml

@@ -11,6 +11,13 @@ license = "Apache-2.0"
 license-files = ["LICENSE", "NOTICE"]
 requires-python = ">=3.12"
 authors = [{ name = "Agentic Learning AI Lab, New York University" }]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Science/Research",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
 dependencies = [
     "cryptography>=42",  # GitHub App auth (RS256 JWTs), the recommended identity
     "pydantic>=2",

{outerloop_science-0.1.0.dev4 → outerloop_science-0.1.2}/src/outerloop/__init__.py

@@ -2,7 +2,7 @@
 
 import os
 
-__version__ = "0.1.0.dev4"
+__version__ = "0.1.2"
 
 
 def _bridge_legacy_env() -> None:

{outerloop_science-0.1.0.dev4 → outerloop_science-0.1.2}/src/outerloop/attempt.py

@@ -19,6 +19,7 @@ import os
 import re
 import shutil
 import time
+import traceback
 from collections.abc import Callable, Iterable
 from dataclasses import dataclass
 from dataclasses import replace as dc_replace
@@ -319,18 +320,35 @@ class AttemptOutcome:
     report_path: str = ""
 
 
+def _attach_run_log(directory: Path) -> None:
+    """Also write the kernel's log lines to `<run dir>/kernel.log`: the scheduler
+    discards a wake job's own stderr, so this is the only trace a swallowed
+    step leaves. Only for a run that exists — never a stray directory."""
+    if not directory.is_dir():
+        return
+    try:
+        handler = logging.FileHandler(directory / "kernel.log", encoding="utf-8")
+    except OSError as exc:
+        log.warning("run log unavailable: %s", exc)
+        return
+    handler.setFormatter(logging.Formatter("%(asctime)s %(message)s"))
+    logging.getLogger().addHandler(handler)
+
+
 def _best_effort(what: str, fn: Callable[[], object], secrets: tuple[str, ...] = ()) -> bool:
-    """One ending step; a failure is logged, never raised.
+    """One ending step; a failure is logged with its traceback, never raised.
 
     The terminal sequence (record, report, issue post) must degrade
     independently: a full disk must not block the GitHub post, and a network
-    failure must not block the record.
+    failure must not block the record. The traceback is the only trace a
+    swallowed step leaves, so it is logged whole (redacted).
     """
     try:
         fn()
         return True
     except Exception as exc:
-        log.warning("%s failed: %s", what, redact(f"{type(exc).__name__}: {exc}", secrets))
+        detail = "".join(traceback.format_exception(exc)).rstrip()
+        log.warning("%s failed: %s", what, redact(detail, secrets))
         return False
 
 
@@ -1025,8 +1043,6 @@ def _wake_author_sleep(
         # existing wake path decides). Keep the NEW park's snapshot ref; the OLD
         # sleep ref is superseded once the new park persists.
         kept_ref = next((s.ref for s in snapshots if s.commit == p.candidate_sha), "")
-        import time
-
         try:
             _park_run(
                 run_root,
@@ -3017,8 +3033,6 @@ def live_attempt(
                 # commit); record and keep that same ref, drop the rest. An
                 # author-sleep's snapshot is the tree the wake re-delivers.
                 kept_ref = next((s.ref for s in snapshots if s.commit == p.candidate_sha), "")
-            import time
-
             # anchor the deadline to the PARK (when the evals were submitted),
             # not the run's start `now` — a session lasting hours would otherwise
             # eat the queue budget and let the sweep cancel a still-queued eval.
@@ -3419,11 +3433,22 @@ def main() -> int:
             )
         return value
 
+    def _run_id(value: str) -> str:
+        # the id names the run directory under runs/: one path segment, so a
+        # value cannot reach outside the run root before the record is read
+        # ("" is the default: a fresh climb, not a resume)
+        if value and not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_.@-]{0,127}", value):
+            raise argparse.ArgumentTypeError(
+                f"run id {value!r} is not a run directory name (want [A-Za-z0-9][A-Za-z0-9_.@-]*)"
+            )
+        return value
+
     parser.add_argument("--agent-id", default="agent-01", type=_agent_id)
     parser.add_argument("--run-root", required=True, type=Path)
     parser.add_argument(
         "--resume",
         default="",
+        type=_run_id,
         metavar="RUN_ID",
         help="wake a parked dispatched run instead of starting a fresh climb",
     )
@@ -3530,6 +3555,8 @@ def main() -> int:
     )
     args = parser.parse_args()
     logging.basicConfig(level=logging.INFO, format="%(asctime)s %(message)s")
+    if args.resume:
+        _attach_run_log(run_dir_of(args.run_root, args.resume))
     if not args.image and not args.uncontained:
         parser.error("--image is required (or pass --uncontained explicitly, dev only)")
     # NOTE: the codex author is validated on the EFFECTIVE author per path — the

{outerloop_science-0.1.0.dev4 → outerloop_science-0.1.2}/src/outerloop/measure.py

@@ -24,7 +24,7 @@ from dataclasses import dataclass
 from pathlib import Path
 from typing import Any
 
-from outerloop.compute import GONE, Compute, JobSpec, is_terminal
+from outerloop.compute import GONE, Compute, JobSpec, is_terminal, local_mode
 from outerloop.dispatch import (
     eval_job_spec,
     read_eval_result,
@@ -275,7 +275,9 @@ class DispatchedMeasurer:
     seed_cache: Path | None = None
 
     def _placement(self, m: Measure) -> tuple[str, str]:
-        if m.gpus <= 0:
+        # local compute has no lanes: the job is a subprocess on whatever GPUs
+        # the machine has (same rule as DispatchSettings.placement)
+        if m.gpus <= 0 or local_mode():
             return self.account, self.partition
         if not self.gpu_partition:
             raise ValueError(
@@ -492,8 +494,6 @@ class DispatchSettings:
         GPU job has no lane — a queue that can never run is worse than a
         loud refusal. Local compute has no lanes: jobs are subprocesses on
         whatever GPUs the machine has, so placement is empty by design."""
-        from outerloop.compute import local_mode
-
         if gpus <= 0 or local_mode():
             return self.account, self.partition
         if not self.gpu_partition:

{outerloop_science-0.1.0.dev4 → outerloop_science-0.1.2}/tests/test_attempt.py

@@ -3304,6 +3304,81 @@ def test_author_sleep_wake_delivers_results_and_flows_to_a_candidate_park(
     assert str(record.stage["candidate_ref"]) in refs[0]
 
 
+def test_author_sleep_wake_records_each_launch_as_ended_in_the_ledger(
+    tmp_path, monkeypatch
+) -> None:
+    """The wake writes one `ended` record per launch job under the sleep that
+    submitted it, so `history` and the PR's experiments table see the job come
+    back. A nested `import time` once made this step raise inside its
+    best-effort wrapper on every wake: no ended record was ever written."""
+    from outerloop.launchlog import append_submitted, history, read_ledger
+    from outerloop.measure import MeasurementPending
+    from outerloop.roles import author_spec
+    from outerloop.syscall import Launch
+
+    state, run_id, _wsroot, _ = _write_parked_author_sleep(
+        tmp_path, monkeypatch, raise_exc=MeasurementPending(("701", "702"))
+    )
+    run_dir = state / "runs" / run_id
+    append_submitted(
+        run_dir,
+        sleep=1,
+        launches=(Launch(name="probe", command="uv run probe.py", minutes=45, why="tails"),),
+        job_ids=["501"],
+        at=1_000_000.0,
+    )
+    resume_run(
+        state,
+        run_id,
+        dispatch=_fake_dispatch(),
+        github=CommentingGitHub(),  # type: ignore[arg-type]
+        bot_auth=NoAuth(),
+        now=1_000_100.0,
+        harness=ScriptedHarness(edits={"src/pilot/solvers/tsp.py": "def solve(): return 'p'\n"}),
+        spec=author_spec(),
+    )
+    ended = [r for r in read_ledger(run_dir) if r["event"] == "ended"]
+    assert [(r["sleep"], r["name"], r["exit_code"], r["last_line"]) for r in ended] == [
+        (1, "probe", 0, "tail improvement: 0.7")
+    ]
+    (entry,) = history(run_dir)
+    assert entry["jobs"] and entry["jobs"][0]["exit_code"] == 0
+
+
+def test_best_effort_logs_the_traceback_of_a_swallowed_step(caplog) -> None:
+    import logging
+
+    from outerloop.attempt import _best_effort
+
+    def boom() -> None:
+        raise NameError("cannot access free variable 'time' (token hunter2)")
+
+    with caplog.at_level(logging.WARNING):
+        assert _best_effort("launch ledger", boom, secrets=("hunter2",)) is False
+    assert "launch ledger failed" in caplog.text
+    assert "Traceback" in caplog.text and "in boom" in caplog.text
+    assert "hunter2" not in caplog.text
+
+
+def test_attach_run_log_persists_kernel_lines_beside_the_run(tmp_path) -> None:
+    import logging
+
+    from outerloop.attempt import _attach_run_log, log
+
+    _attach_run_log(tmp_path / "runs" / "nope")  # no such run: nothing created
+    assert not (tmp_path / "runs").exists()
+    (tmp_path / "runs" / "r1").mkdir(parents=True)
+    _attach_run_log(tmp_path / "runs" / "r1")
+    try:
+        log.warning("launch ledger failed: NameError")
+    finally:
+        for h in list(logging.getLogger().handlers):
+            if isinstance(h, logging.FileHandler) and h.baseFilename.endswith("kernel.log"):
+                logging.getLogger().removeHandler(h)
+                h.close()
+    assert "launch ledger failed" in (tmp_path / "runs" / "r1" / "kernel.log").read_text()
+
+
 def test_author_sleep_wake_can_sleep_again(tmp_path, monkeypatch) -> None:
     # the woken author launches more work and sleeps again: a fresh author-sleep
     # park with the counts advanced.
@@ -4205,6 +4280,26 @@ def test_cli_rejects_ref_shaping_agent_ids(capsys, monkeypatch) -> None:
         assert "cannot shape a git ref" in capsys.readouterr().err
 
 
+def test_resume_rejects_a_run_id_that_is_not_one_path_segment(
+    tmp_path, monkeypatch, capsys
+) -> None:
+    """`--resume` names the run directory under runs/. A traversal value is
+    refused by the parser, before the lease or the kernel log could be created
+    outside the run root."""
+    from outerloop.attempt import main as climb_main
+
+    (tmp_path / "elsewhere").mkdir()
+    for bad in ("../elsewhere", "a/b", ".."):
+        monkeypatch.setattr(
+            "sys.argv",
+            ["climb", "--resume", bad, "--run-root", str(tmp_path / "state"), "--image", "x.sif"],
+        )
+        with pytest.raises(SystemExit):
+            climb_main()
+        assert "is not a run directory name" in capsys.readouterr().err
+    assert not (tmp_path / "elsewhere" / "kernel.log").exists()
+
+
 def test_wake_refreshes_origin_refs(tmp_path, monkeypatch) -> None:
     """The clone's refs freeze at run start; a wake fetches origin so the
     resumed session reads current base and sibling refs locally."""

{outerloop_science-0.1.0.dev4 → outerloop_science-0.1.2}/tests/test_measure.py

@@ -399,6 +399,25 @@ def test_dispatch_settings_place_gpu_jobs_on_the_gpu_lane(tmp_path):
     assert own_account.placement(2) == ("gpu-acct", "h200")
 
 
+def test_local_compute_places_a_gpu_measure_without_a_lane(tmp_path, monkeypatch):
+    """Local compute has no lanes: a GPU measure is a subprocess on the machine's
+    own GPUs, so the gate's placement must not demand OUTERLOOP_GPU_PARTITION.
+    The launch placement already followed that rule; the measure placement did
+    not, and a GPU benchmark on a local box aborted at its first baseline."""
+    from outerloop.compute import LocalCompute
+    from outerloop.measure import DispatchSettings, Measure
+
+    settings = DispatchSettings(compute=LocalCompute(), image="", account="", partition="")
+    m = settings.measurer(tmp_path, repo_root=tmp_path, eval_minutes=15, run_tag="r")
+    gpu = Measure(name="baseline", tree_sha="a" * 40, command="x", metric="loss", gpus=1)
+    monkeypatch.delenv("OUTERLOOP_COMPUTE", raising=False)
+    with pytest.raises(ValueError, match="OUTERLOOP_GPU_PARTITION"):
+        m._placement(gpu)
+    monkeypatch.setenv("OUTERLOOP_COMPUTE", "local")
+    assert m._placement(gpu) == ("", "")
+    assert settings.placement(1) == ("", "")  # the launch side, same rule
+
+
 def test_mixed_suite_places_each_measure_on_its_own_lane(tmp_path):
     """A suite gate measures siblings through the ONE measurer: a GPU
     sibling of a CPU benchmark (or vice versa) must land on its own lane with