omnirun 0.2.0__tar.gz → 0.2.1__tar.gz

{omnirun-0.2.0 → omnirun-0.2.1}/DESIGN.md

@@ -282,10 +282,13 @@ Worker access to private repos — **no git credentials ever leave the laptop**:
     bundle base64 inside the kernel's `run.py`** (next to the base64 bootstrap) — *not* a
     dataset. The old dataset approach 409'd every kernel that referenced a still-processing
     dataset (a systematic race) and needed a create/delete lifecycle; embedding removes
-    both. A size guard (`MAX_EMBED_B64`, 40 MiB, covering bundle + env together) rejects
-    oversized repos, so this suits code-sized repos (data is never shipped — jobs fetch
-    their own). The bootstrap clones/fetches from the bundle into the object store. In the
-    public-repo case no bundle is embedded, so the ceiling is a non-issue.
+    both. A size guard (`KAGGLE_MAX_SOURCE_BYTES`, ~1 MiB, measuring the whole `run.py`
+    kernel source — embedded bootstrap + bundle + env — against Kaggle's real push limit,
+    overridable per backend via `max_source_bytes`) rejects oversized pushes early with a
+    size-naming error instead of an opaque Kaggle-side failure, so this suits code-sized
+    repos (data is never shipped — jobs fetch their own). The bootstrap clones/fetches from
+    the bundle into the object store. In the public-repo case no bundle is embedded, so the
+    ceiling is a non-issue.
 
   This keeps the invariant intact: a public clone needs no credentials, and private repos
   still never touch origin from the worker. What changed is that public-repo workers now

{omnirun-0.2.0 → omnirun-0.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: omnirun
-Version: 0.2.0
+Version: 0.2.1
 Summary: Run jobs from your repo anywhere: Slurm over SSH, any SSH box, Kaggle, Colab, or marketplace GPUs
 Author-email: Dmitrii Mukhutdinov <flyingleafe@gmail.com>
 License: MIT

{omnirun-0.2.0 → omnirun-0.2.1}/TESTING.md

@@ -116,15 +116,27 @@ bootstrap script — there is **no per-job dataset** anymore. This replaced a
 dataset-based design that hit a systematic **409** (Kaggle rejects a kernel
 referencing a still-processing dataset). Consequence: kaggle no longer creates or
 deletes any datasets; nothing worker-side to reap. Only caveat is bundle size — a
-code-only repo bundle is well under the embed cap; a repo with large committed
-blobs will be rejected client-side before push.
-
-**Newly added (unit-tested, NOT yet live-verified):** for a **public** repo the
-kernel now clones the repo directly over its own internet connection and no bundle
-is embedded (`repo.remote_clone_plan` gates on public + reachable). Kaggle also now
-injects a gitignored `<repo>/.env` (base64-embedded as `ENV_B64`, decoded to a 0600
-file and sourced) — parity with Colab. Both paths pass unit tests; a live Kaggle
-(and Colab) job against a public repo has not been re-run yet.
+code-only repo bundle is well under the source cap; a repo with large committed
+blobs is rejected client-side before push (see the measured cap below).
+
+**Public-repo direct clone — LIVE-VERIFIED (this session).** A CPU kernel was
+submitted against `github.com/flyingleafe/omnirun` (public) at the pushed master
+tip with `env=none`: the worker ran `git clone --bare` over its own connection,
+checked out the exact sha (`Preparing worktree (detached HEAD 6fcedf0)` →
+`CLONE OK cwd=…/.trees/6fcedf04a860`, `has_pyproject=True`), no bundle embedded,
+and `proof.txt` was pulled back. So a **public repo of any size** runs on Kaggle,
+unconstrained by the source cap. Kaggle also injects a gitignored `<repo>/.env`
+(base64 `ENV_B64` → 0600 file → sourced) — unit-tested, parity with Colab.
+
+**Kernel-source cap — MEASURED (this session).** Pushing `run.py` payloads of
+increasing size to the kernels API: `<=1 MiB` (1,048,570 B) **accepted**,
+`>=1.1 MiB` **rejected with HTTP 400** — so Kaggle's limit is **1 MiB**. The
+pre-submit guard (`KAGGLE_MAX_SOURCE_BYTES = 1 MiB`, override via
+`max_source_bytes`) now measures the full `run.py` (bootstrap + any bundle + any
+`.env`) against this and fails early naming size, instead of the old 40 MiB
+`MAX_EMBED_B64` guard that let 1–40 MiB pushes fail opaquely on Kaggle's side.
+(Aside: the current kaggle client exposes `kernels_delete(ref, no_confirm=True)`,
+used here to clean up the probe kernels — a future `gc`/`cancel` could use it.)
 
 - Provide: API token at `~/.config/kaggle/kaggle.json` (kaggle.com → Settings →
   Create New Token) or `KAGGLE_USERNAME`/`KAGGLE_KEY` env vars. The account
@@ -202,7 +214,8 @@ module.
 
 **Kaggle** (`src/omnirun/backends/kaggle.py`) — CONFIRMED LIVE (P100 job ran; output pulled)
 - [x] Bundle delivery (private/unpushed repo): **embedded base64 in `run.py`**, no dataset — sidesteps the 409 the old dataset design hit. `dataset_create_new`/`dataset_delete` are no longer used.
-- [ ] Public-repo **direct clone** (worker `git clone` over its own connection, no bundle) — unit-tested via `repo.remote_clone_plan`; not yet run live on Kaggle.
+- [x] Public-repo **direct clone** (worker `git clone` over its own connection, no bundle) — **LIVE-VERIFIED**: CPU job cloned `flyingleafe/omnirun` at master tip, ran in the worktree, `proof.txt` pulled.
+- [x] Kernel-source push cap — **MEASURED**: 1 MiB (≤1 MiB accepted, ≥1.1 MiB → HTTP 400); `KAGGLE_MAX_SOURCE_BYTES` guard set to match, overridable via `max_source_bytes`.
 - [ ] `.env` injection (`ENV_B64` embedded → 0600 file → sourced) — unit-tested; not yet run live on Kaggle.
 - [x] `kernels_status` response shape + status strings (`queued/running/complete/error/cancelAcknowledged`).
 - [x] `kernels_output` kwarg + downloaded log format (`<slug>.log`).

{omnirun-0.2.0 → omnirun-0.2.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "omnirun"
-version = "0.2.0"
+version = "0.2.1"
 description = "Run jobs from your repo anywhere: Slurm over SSH, any SSH box, Kaggle, Colab, or marketplace GPUs"
 readme = "README.md"
 requires-python = ">=3.12"

{omnirun-0.2.0 → omnirun-0.2.1}/src/omnirun/__init__.py

@@ -1,3 +1,3 @@
 """omnirun — run jobs from your repo anywhere."""
 
-__version__ = "0.2.0"
+__version__ = "0.2.1"

{omnirun-0.2.0 → omnirun-0.2.1}/src/omnirun/backends/colab.py

@@ -32,7 +32,7 @@ from datetime import datetime, timedelta, timezone
 from pathlib import Path
 from typing import Any
 
-from omnirun.backends import jobdir
+from omnirun.backends import jobdir, tarsafe
 from omnirun.backends.base import Backend, BackendError, register
 from omnirun.bootstrap import (
     HEARTBEAT_STALE_S,
@@ -583,7 +583,7 @@ class ColabBackend(Backend):
             if not local_tar.exists():
                 raise BackendError("colab download produced no local file")
             with tarfile.open(local_tar) as tf:
-                tf.extractall(td, filter="data")
+                tarsafe.extract_all(tf, Path(td))
             outputs = Path(td) / "outputs"
             if outputs.is_dir():
                 shutil.copytree(outputs, dest, dirs_exist_ok=True)

{omnirun-0.2.0 → omnirun-0.2.1}/src/omnirun/backends/kaggle.py

@@ -34,7 +34,7 @@ from pathlib import Path
 from typing import Any
 
 from omnirun.backends.base import Backend, BackendError, register
-from omnirun.backends import jobdir
+from omnirun.backends import jobdir, tarsafe
 from omnirun.bootstrap import (
     BootstrapParams,
     CodeSource,
@@ -76,10 +76,15 @@ MAX_MEM_GB = 30
 MAX_DISK_GB = 55
 KAGGLE_ROOT = "/kaggle/tmp/omnirun"  # venv + tree scratch; /kaggle/working stays small
 RESULT_TAR = "omnirun-job.tar.gz"
-# Cap on the base64 git bundle embedded in the kernel source. Kaggle rejects
-# oversized kernel pushes; fail early with a clear message instead. Generous —
-# a code-only repo bundle is normally well under a megabyte.
-MAX_EMBED_B64 = 40 * 1024 * 1024
+# Cap on the kernel source (run.py) omnirun pushes: the embedded bootstrap +
+# any git bundle (private/unpushed repos) + any out-of-band .env all count
+# against it. Kaggle rejects an oversized push with an opaque HTTP 400, so we
+# guard at the real threshold and fail early naming size as the cause. Measured
+# live against the kernels API: <=1 MiB accepted, >=1.1 MiB rejected — i.e. the
+# limit is 1 MiB. A private/unpushed repo therefore only fits when its bundle is
+# small; a *public* repo is cloned by the worker directly (no bundle shipped) and
+# is unaffected by this cap. Override per backend with `max_source_bytes`.
+KAGGLE_MAX_SOURCE_BYTES = 1024 * 1024
 LOG_POLL_INTERVAL_S = 30.0  # poll etiquette: >=30s against the kaggle API
 
 
@@ -365,6 +370,29 @@ class KaggleBackend(Backend):
 
     # ---- submit ---------------------------------------------------------------
 
+    def _guard_source_size(self, harness: str, *, shipped_bundle: bool) -> None:
+        """Fail early, with a size-naming message, when the kernel source is over
+        Kaggle's real push limit — instead of letting the push fail opaquely."""
+        limit = int(self.config.extra("max_source_bytes", KAGGLE_MAX_SOURCE_BYTES))
+        size = len(harness.encode())
+        if size <= limit:
+            return
+        # the git bundle (full history + tree at the sha) usually dominates when
+        # a private/unpushed repo forces one; a public repo ships no bundle.
+        remedy = (
+            "push the repo to a public remote so the worker clones it directly "
+            "(no bundle shipped), slim the git history (shallow/squash) or exclude "
+            "heavy dirs from the tree, "
+            if shipped_bundle
+            else "slim the tree or "
+        )
+        raise BackendError(
+            f"kaggle kernel source is {size / (1024 * 1024):.1f} MB, over the "
+            f"~{limit / (1024 * 1024):.1f} MB Kaggle push limit (the push would "
+            f"fail opaquely) — {remedy}run this job on another backend, or raise "
+            "the backend's `max_source_bytes` if Kaggle accepts more"
+        )
+
     def submit(self, spec: JobSpec, offer: Offer) -> JobHandle:
         api = self._api()
         user = self._username(api)
@@ -401,13 +429,6 @@ class KaggleBackend(Backend):
             env_b64 = (
                 base64.b64encode(envf.read_text().encode()).decode() if envf else ""
             )
-            if len(bundle_b64) + len(env_b64) > MAX_EMBED_B64:
-                raise BackendError(
-                    f"embedded payload is too large for a kaggle kernel "
-                    f"(~{(len(bundle_b64) + len(env_b64)) // (1024 * 1024)}MB base64) "
-                    "— slim the git history (shallow/squash), push a public repo, or "
-                    "run this job on another backend"
-                )
 
             script = generate_bootstrap(
                 notebook_env_spec(spec),
@@ -421,11 +442,12 @@ class KaggleBackend(Backend):
             )
             boot_b64 = base64.b64encode(script.encode()).decode()
 
+            harness = _render_harness(job_id, boot_b64, bundle_b64, env_b64)
+            self._guard_source_size(harness, shipped_bundle=bool(bundle_b64))
+
             k_dir = stage / "kernel"
             k_dir.mkdir()
-            (k_dir / "run.py").write_text(
-                _render_harness(job_id, boot_b64, bundle_b64, env_b64)
-            )
+            (k_dir / "run.py").write_text(harness)
             meta: dict[str, Any] = {
                 "id": kernel_ref,
                 "title": slug,  # title must slugify to the id slug
@@ -610,7 +632,7 @@ class KaggleBackend(Backend):
             extract = Path(td) / "extract"
             extract.mkdir()
             with tarfile.open(tar) as tf:
-                tf.extractall(extract, filter="data")
+                tarsafe.extract_all(tf, extract)
             outputs = extract / "outputs"
             if outputs.is_dir():
                 shutil.copytree(outputs, dest, dirs_exist_ok=True)

omnirun-0.2.1/src/omnirun/backends/tarsafe.py

@@ -0,0 +1,38 @@
+"""Tolerant tar extraction for the notebook backends (kaggle, colab), which
+return a job's outputs as a tarball the client must unpack.
+
+Python >=3.12 defaults tar extraction to the PEP-706 ``data`` filter. That
+filter is right (it refuses path traversal, device nodes and links pointing
+outside the destination), but ``extractall`` aborts the *whole* archive the
+first time the filter rejects a member. The common offender is a symlink whose
+target is an absolute path — every Weights & Biases run leaves one at
+``wandb/*/logs/debug-core.log`` — so a single such link would strand every
+output of an otherwise-successful job.
+
+``extract_all`` applies the same ``data`` filter per member but skips (with a
+warning) any member the filter rejects, so the rest of the tree still lands.
+"""
+
+from __future__ import annotations
+
+import sys
+import tarfile
+from pathlib import Path
+
+
+def extract_all(tf: tarfile.TarFile, dest: Path) -> list[str]:
+    """Extract every member of ``tf`` under ``dest`` with the safe ``data``
+    filter, skipping (not aborting on) any member the filter rejects. Returns
+    the names of the skipped members (also warned to stderr)."""
+    skipped: list[str] = []
+    for member in tf.getmembers():
+        try:
+            tf.extract(member, dest, filter="data")
+        except tarfile.FilterError as e:
+            skipped.append(member.name)
+            print(
+                f"omnirun: warning: skipped unsafe archive entry "
+                f"{member.name!r} ({type(e).__name__})",
+                file=sys.stderr,
+            )
+    return skipped

{omnirun-0.2.0 → omnirun-0.2.1}/src/omnirun/bootstrap.py

@@ -188,6 +188,33 @@ def _indent(block: str, n: int) -> str:
     return "".join(pad + line + "\n" for line in block.splitlines())
 
 
+def _heredoc_delim(body: str) -> str:
+    """A heredoc terminator that provably does not occur as a line in `body`,
+    so embedding an arbitrary command in a heredoc can never terminate early."""
+    lines = body.splitlines()
+    base = "OMNIRUN_CMD_EOF"
+    delim = base
+    i = 0
+    while delim in lines:
+        i += 1
+        delim = f"{base}_{i}"
+    return delim
+
+
+def _command_block(command: str) -> str:
+    """Body of run_cmd(): the user command carried byte-exact via a single-quoted
+    heredoc and run with `eval`.
+
+    A single-quoted heredoc disables every expansion and preserves the body
+    literally — newlines, embedded heredocs, quotes, backslashes all survive — so
+    multi-line scripts reach the worker unmangled. `eval` runs it in the current
+    shell, so env activation, forwarded exports and pre-run lines all still apply.
+    Crucially the command is NOT indented: indenting would shift the user's own
+    heredoc terminators off column 0 and break them."""
+    delim = _heredoc_delim(command)
+    return f"eval \"$(cat <<'{delim}'\n{command}\n{delim}\n)\"\n"
+
+
 def _default_project_root(slug: str) -> str:
     return f"$OMNIRUN_ROOT/projects/{slug}"
 
@@ -307,8 +334,7 @@ status running
 STARTED_AT=$(now)
 set +e
 run_cmd() {{
-{_indent(spec.command, 2)}\
-}}
+{_command_block(spec.command)}}}
 # run in a subshell: a bare `exit N` in the user command must not kill
 # bootstrap.sh itself (result.json would never be written -> job shows LOST)
 ( run_cmd ) > >(tee "$JOB_DIR/logs/stdout.log") 2> >(tee "$JOB_DIR/logs/stderr.log" >&2)

{omnirun-0.2.0 → omnirun-0.2.1}/tests/test_bootstrap.py

@@ -133,6 +133,38 @@ def test_failing_command_propagates_exit_code(
     assert "about to fail" in (job_dir / "logs" / "stdout.log").read_text()
 
 
+def test_multiline_heredoc_command_runs_byte_exact(
+    sample_repo: Path, job_spec: JobSpec, tmp_path: Path
+) -> None:
+    """Regression (#3): a multi-line command with a heredoc must reach the worker
+    byte-identical. The old indent-per-line embedding shifted the heredoc
+    terminator off column 0 (breaking it) and prefixed every body line — this
+    proves the body is written verbatim, indentation and all."""
+    command = (
+        'cat <<EOF > "$OMNIRUN_OUTPUT/cfg.yaml"\n'
+        "key: value\n"
+        "  two-space-indent: kept\n"
+        "EOF\n"
+        'echo "MULTILINE OK"'
+    )
+    spec = job_spec.model_copy(
+        update={
+            "job_id": JobSpec.make_job_id("heredoc"),
+            "command": command,
+            "outputs": [],
+        }
+    )
+    root = tmp_path / "omnirun_root"
+    script = stage(sample_repo, spec, root)
+    proc = run_bootstrap(script)
+
+    job_dir = root / "jobs" / spec.job_id
+    assert proc.returncode == 0, (job_dir / "logs" / "bootstrap.log").read_text()
+    cfg = job_dir / "outputs" / "cfg.yaml"
+    assert cfg.read_text() == "key: value\n  two-space-indent: kept\n"
+    assert "MULTILINE OK" in (job_dir / "logs" / "stdout.log").read_text()
+
+
 def test_bare_exit_in_command_still_writes_result(
     sample_repo: Path, job_spec: JobSpec, tmp_path: Path
 ) -> None:

{omnirun-0.2.0 → omnirun-0.2.1}/tests/test_kaggle.py

@@ -496,6 +496,49 @@ def test_pull_outputs(fake_api, backend, tmp_path):
     assert files == [dest / "model.txt"]
 
 
+def test_pull_outputs_tolerates_absolute_symlink(fake_api, backend, tmp_path):
+    """Regression (#1): every W&B run leaves a symlink to an absolute path under
+    wandb/*/logs/. Python's data filter rejects it; pull must skip that one entry
+    and still recover the real outputs rather than aborting the whole archive."""
+    buf = io.BytesIO()
+    with tarfile.open(fileobj=buf, mode="w:gz") as tf:
+        data = b"metric,value\n1,2\n"
+        good = tarfile.TarInfo("outputs/results.csv")
+        good.size = len(data)
+        tf.addfile(good, io.BytesIO(data))
+        link = tarfile.TarInfo("outputs/wandb/run-x/logs/debug-core.log")
+        link.type = tarfile.SYMTYPE
+        link.linkname = "/kaggle/tmp/omnirun/abs/debug-core.log"  # absolute target
+        tf.addfile(link)
+    fake_api.output_files = {"omnirun-job.tar.gz": buf.getvalue()}
+    dest = tmp_path / "results"
+    files = backend.pull_outputs(make_handle(), dest)
+    assert (dest / "results.csv").read_bytes() == data
+    assert [p.name for p in files] == ["results.csv"]  # link skipped, csv kept
+
+
+def test_submit_guards_oversized_source(fake_api, monkeypatch):
+    """Regression (#2): an oversized kernel source must fail early with a
+    size-naming message, not sail past the guard and fail opaquely on Kaggle."""
+    monkeypatch.setattr(
+        kaggle_mod, "_create_bundle", lambda root, sha, dest: _big_bundle(dest)
+    )
+    backend = KaggleBackend(
+        "kaggle",
+        BackendConfig.model_validate({"type": "kaggle", "max_source_bytes": 4096}),
+    )
+    spec = make_spec(gpu_type="P100")
+    offer = backend.probe(spec.resources)[0]
+    with pytest.raises(BackendError, match=r"kaggle kernel source is .* over the"):
+        backend.submit(spec, offer)
+
+
+def _big_bundle(dest: Path) -> Path:
+    dest = Path(dest)
+    dest.write_bytes(b"x" * 8192)  # base64 ~11k, over the 4k test limit
+    return dest
+
+
 def test_cancel_without_api_support_points_at_website(fake_api, backend):
     with pytest.raises(BackendError, match="kaggle.com"):
         backend.cancel(make_handle())

omnirun-0.2.1/tests/test_tarsafe.py

@@ -0,0 +1,53 @@
+"""Tolerant tar extraction (shared by the kaggle + colab pull paths)."""
+
+from __future__ import annotations
+
+import io
+import os
+import tarfile
+from pathlib import Path
+
+from omnirun.backends import tarsafe
+
+
+def _tar_with_abs_symlink(dest_target: str) -> io.BytesIO:
+    buf = io.BytesIO()
+    with tarfile.open(fileobj=buf, mode="w:gz") as tf:
+        data = b"real output\n"
+        good = tarfile.TarInfo("outputs/results.txt")
+        good.size = len(data)
+        tf.addfile(good, io.BytesIO(data))
+        link = tarfile.TarInfo("outputs/wandb/run-x/logs/debug-core.log")
+        link.type = tarfile.SYMTYPE
+        link.linkname = dest_target
+        tf.addfile(link)
+    buf.seek(0)
+    return buf
+
+
+def test_extract_all_skips_absolute_symlink_and_keeps_rest(tmp_path: Path) -> None:
+    # every W&B run leaves such a link; the default data filter would abort here.
+    buf = _tar_with_abs_symlink("/etc/hostname")
+    with tarfile.open(fileobj=buf) as tf:
+        skipped = tarsafe.extract_all(tf, tmp_path)
+    assert skipped == ["outputs/wandb/run-x/logs/debug-core.log"]
+    assert (tmp_path / "outputs" / "results.txt").read_bytes() == b"real output\n"
+    assert not (tmp_path / "outputs" / "wandb" / "run-x" / "logs").exists()
+
+
+def test_extract_all_extracts_normal_tree_completely(tmp_path: Path) -> None:
+    src = tmp_path / "src"
+    (src / "sub").mkdir(parents=True)
+    (src / "a.txt").write_text("a")
+    (src / "sub" / "b.txt").write_text("b")
+    os.symlink("a.txt", src / "rel")  # a *relative* symlink is allowed by data filter
+    tar = tmp_path / "t.tar.gz"
+    with tarfile.open(tar, "w:gz") as tf:
+        tf.add(src, arcname="outputs")
+    out = tmp_path / "out"
+    with tarfile.open(tar) as tf:
+        skipped = tarsafe.extract_all(tf, out)
+    assert skipped == []
+    assert (out / "outputs" / "a.txt").read_text() == "a"
+    assert (out / "outputs" / "sub" / "b.txt").read_text() == "b"
+    assert (out / "outputs" / "rel").is_symlink()