mlsec-raid 0.4.2__tar.gz → 0.5.0__tar.gz

mlsec_raid-0.5.0/PKG-INFO

@@ -0,0 +1,45 @@
+Metadata-Version: 2.3
+Name: mlsec-raid
+Version: 0.5.0
+Summary: Client for the RAID competition system
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+
+# raid
+
+Python client for the RAID competition system.
+
+```bash
+pip install mlsec-raid                                    # or: uv tool install mlsec-raid
+export RAID_KEY=<your-api-key>                            # or pass --api-key <your-api-key>
+
+task-fetch <challenge>            # downloads the challenge into ./<challenge>/
+task-check <challenge>            # scores it locally with scoring/score.py on its data, in a container
+task-check method <challenge>     # only the method: runs method/run like our backend's sanity check
+task-check artifact <challenge>   # only the artifact: scores artifact/result/
+task-submit <challenge>           # submits ./<challenge>/ without its data, scoring/ and .raidignore patterns
+task-show                         # lists your jobs with their state (pending, running, success, failed) and
+                                  # score, and the submissions left today on challenges with a daily limit
+task-log <job>                    # prints the log of a finished job, including the errors of your solution
+```
+
+Every command also works as `task <command>`, e.g. `task submit <challenge>` or `task check method <challenge>`.
+
+Challenges like dawn separate the method from the artifact:
+
+```
+<challenge>/
+  artifact/data/     the data, never submitted
+  artifact/result/   the artifact you submit, e.g. dawn's re-implementations
+  method/run         your method, which the backend runs once as a sanity check (dawn: on the simple target)
+  scoring/score.py   method(), artifact() and score(), the backend's scoring
+  setup.sh           your installs
+```
+
+`task-check` runs `scoring/score.py` in the container `raid-<challenge>`, built like the backend's sandbox: with Docker an image on Debian trixie (slim) with python3 and the challenge's Debian packages (`scoring/packages.txt`), in which the challenge's `setup.sh` runs as root during the build; with only udocker (`pip install udocker`) a udocker container of `debian:trixie` set up the same way. It is built again only when `setup.sh` or the packages change (their hash is the image's `raid.setup` label, or `/etc/raid-setup.sha256` in the udocker container), or with `--rebuild`. Like on the backend, `score.py` sees only Debian's Python packages; unlike there, the method may use the internet. `task-check method` calls `score.py`'s `method()`, `task-check artifact` its `artifact()` and `task-check` its `score()`, which checks the method first and then the artifact. Challenges without that split (e.g. self-assessment) have only `task-check <challenge>`. `--no-container` checks on this machine instead.
+
+The same is available in Python: `Raid(api_key).fetch(...)`, `.check(...)`, `.submit(...)`, `.push(...)`, `.wait(id)`, `.status()`, `.log(id)`, `.howto()`.
+
+Challenge authors: `task-check <challenge> --hidden --reference` scores `reference/` on `hidden/` (the packages come from `challenge.yaml`). Admins upload a challenge directory with `task-push <directory>` and their own API key (created in the admin console): the directory's `challenge.yaml` sets the name, version, wave, baseline, badge and `submissions_per_day` (keys it leaves out keep their current values), and the upload becomes the active version.
+
+Development: `uv sync` sets up the environment, `uv build` builds the package. Set `RAID_URL` to use another server. Release: bump `version` in `pyproject.toml`, then `uv build && uv publish` (PyPI token in `UV_PUBLISH_TOKEN`).

mlsec_raid-0.5.0/README.md

@@ -0,0 +1,38 @@
+# raid
+
+Python client for the RAID competition system.
+
+```bash
+pip install mlsec-raid                                    # or: uv tool install mlsec-raid
+export RAID_KEY=<your-api-key>                            # or pass --api-key <your-api-key>
+
+task-fetch <challenge>            # downloads the challenge into ./<challenge>/
+task-check <challenge>            # scores it locally with scoring/score.py on its data, in a container
+task-check method <challenge>     # only the method: runs method/run like our backend's sanity check
+task-check artifact <challenge>   # only the artifact: scores artifact/result/
+task-submit <challenge>           # submits ./<challenge>/ without its data, scoring/ and .raidignore patterns
+task-show                         # lists your jobs with their state (pending, running, success, failed) and
+                                  # score, and the submissions left today on challenges with a daily limit
+task-log <job>                    # prints the log of a finished job, including the errors of your solution
+```
+
+Every command also works as `task <command>`, e.g. `task submit <challenge>` or `task check method <challenge>`.
+
+Challenges like dawn separate the method from the artifact:
+
+```
+<challenge>/
+  artifact/data/     the data, never submitted
+  artifact/result/   the artifact you submit, e.g. dawn's re-implementations
+  method/run         your method, which the backend runs once as a sanity check (dawn: on the simple target)
+  scoring/score.py   method(), artifact() and score(), the backend's scoring
+  setup.sh           your installs
+```
+
+`task-check` runs `scoring/score.py` in the container `raid-<challenge>`, built like the backend's sandbox: with Docker an image on Debian trixie (slim) with python3 and the challenge's Debian packages (`scoring/packages.txt`), in which the challenge's `setup.sh` runs as root during the build; with only udocker (`pip install udocker`) a udocker container of `debian:trixie` set up the same way. It is built again only when `setup.sh` or the packages change (their hash is the image's `raid.setup` label, or `/etc/raid-setup.sha256` in the udocker container), or with `--rebuild`. Like on the backend, `score.py` sees only Debian's Python packages; unlike there, the method may use the internet. `task-check method` calls `score.py`'s `method()`, `task-check artifact` its `artifact()` and `task-check` its `score()`, which checks the method first and then the artifact. Challenges without that split (e.g. self-assessment) have only `task-check <challenge>`. `--no-container` checks on this machine instead.
+
+The same is available in Python: `Raid(api_key).fetch(...)`, `.check(...)`, `.submit(...)`, `.push(...)`, `.wait(id)`, `.status()`, `.log(id)`, `.howto()`.
+
+Challenge authors: `task-check <challenge> --hidden --reference` scores `reference/` on `hidden/` (the packages come from `challenge.yaml`). Admins upload a challenge directory with `task-push <directory>` and their own API key (created in the admin console): the directory's `challenge.yaml` sets the name, version, wave, baseline, badge and `submissions_per_day` (keys it leaves out keep their current values), and the upload becomes the active version.
+
+Development: `uv sync` sets up the environment, `uv build` builds the package. Set `RAID_URL` to use another server. Release: bump `version` in `pyproject.toml`, then `uv build && uv publish` (PyPI token in `UV_PUBLISH_TOKEN`).

{mlsec_raid-0.4.2 → mlsec_raid-0.5.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "mlsec-raid"
-version = "0.4.2"
+version = "0.5.0"
 description = "Client for the RAID competition system"
 readme = "README.md"
 requires-python = ">=3.9"

{mlsec_raid-0.4.2 → mlsec_raid-0.5.0}/src/raid/__init__.py

@@ -1,4 +1,5 @@
 """Client for the RAID competition system: fetch challenges, check and submit solutions."""
+import ast
 import fnmatch
 import importlib.util
 import json
@@ -15,11 +16,14 @@ import urllib.request
 from pathlib import Path
 
 __all__ = ["Raid", "RaidError"]
-__version__ = "0.4.2"
+__version__ = "0.5.0"
 
 DEFAULT_URL = "https://raid.mlsec.tu-berlin.de"
 PARTS = ("source", "data", "scoring")  # source/ is unpacked into the challenge directory itself
-OWN = {"data", "scoring", "hidden", "reference", "challenge.yaml"}  # parts of the challenge directory that are never submitted
+# Parts of the challenge directory that are never submitted: these top-level entries and artifact/data/
+OWN = {"data", "scoring", "hidden", "reference", "challenge.yaml", "entrypoint.sh"}
+ARTIFACT_DATA = Path("artifact", "data")  # the data/ part of challenges split into artifact/ and method/
+CHECKS = {"method": "Method score", "artifact": "Artifact score", None: "Score"}  # the part checked: what it prints
 SKIP = {"__pycache__", ".git", ".ipynb_checkpoints", ".venv", "venv", ".DS_Store"}
 NAME_RE = re.compile(r"^[a-z0-9][a-z0-9-]{0,62}$")
 
@@ -41,7 +45,7 @@ class Raid:
         return self._json("/api/howto")["text"]
 
     def fetch(self, challenge, force=False):
-        """Download a challenge into ./<challenge>/: its files, data/ and scoring/.
+        """Download a challenge into ./<challenge>/: its files, data/ (artifact/data/) and scoring/.
 
         An existing directory is left untouched unless force=True, which overwrites the challenge files.
         """
@@ -65,35 +69,62 @@ class Raid:
         print(f"Fetched {challenge} (version {info['version']}) into {target}/. Start with {target / 'README.txt'}.")
         return target
 
-    def check(self, challenge, hidden=False, reference=False):
-        """Score ./<challenge>/ locally: scoring/score.py's score() on data/. Returns the score.
+    def check(self, challenge, part=None, hidden=False, reference=False, container=True, rebuild=False):
+        """Check ./<challenge>/ locally with scoring/score.py on its data (artifact/data/ or data/). Returns the score.
+
+        part="method" calls score.py's method(), part="artifact" its artifact() and part=None its score(), which
+        checks the method first and then the artifact. It runs in the container raid-<challenge>, built like the
+        backend's sandbox with the challenge's setup.sh (Docker, else udocker) and rebuilt when setup.sh changes or
+        rebuild=True; container=False runs it on this machine instead.
 
         For challenge authors: hidden=True scores on hidden/, reference=True scores the files
-        in reference/ in place of the student files.
+        in reference/ in place of the student files. Raises RaidError when scoring stops without a score.
         """
+        if part not in CHECKS:
+            raise RaidError(f"Invalid check {part!r}, use method or artifact.")
         target = (self.path / _check(challenge)).resolve()
         script = target / "scoring" / "score.py"
-        dataset = target / ("hidden" if hidden else "data")
+        dataset = target / "hidden" if hidden else _data(target)
         solution = target / "reference" if reference else target
         for path in (script, dataset, solution):
             if not path.exists():
                 raise RaidError(f"{path} not found." + (" Fetch the challenge first." if path == script else ""))
-        paths = [str(script.parent), str(solution)]  # score.py imports the solution, e.g. `from solution import md5`
-        sys.path[:0] = paths
-        os.environ["RAID_SOLUTION"] = str(solution)  # where score.py finds files that are not imported, e.g. solution.c
-        try:
-            spec = importlib.util.spec_from_file_location("score", script)
-            module = importlib.util.module_from_spec(spec)
-            spec.loader.exec_module(module)
-            value = module.score(dataset)
-        finally:
-            del os.environ["RAID_SOLUTION"]
-            for p in paths:
-                sys.path.remove(p)
-            for name, mod in list(sys.modules.items()):  # a later check sees the current solution
-                if target in Path(getattr(mod, "__file__", None) or "/").resolve().parents:
-                    del sys.modules[name]
-        print(f"Score: {value}")
+        function = part or "score"
+        defined = _functions(script)
+        if part and defined is not None and part not in defined:
+            raise RaidError(f"{challenge} has no separate {part} check (scoring/score.py has no {part}()). "
+                            f"Run task check {challenge}.")
+        files = "reference/" if reference else "your files"
+        what = f"{challenge} {part}" if part else challenge
+        here = str(Path.cwd().resolve()) + os.sep  # shown paths are relative, e.g. self-assessment/solution.c
+        shown = f"{dataset.relative_to(target).as_posix()}/"
+        if container:
+            from . import _container
+            kind = _container.engine()
+            if kind is None:
+                raise RaidError("task check needs Docker or udocker (pip install udocker) to build the container "
+                                f"raid-{challenge}. Pass --no-container to check on this machine instead.")
+            box = _container.Container(challenge, target, kind)
+            print(f"Checking {what}: {files} on {shown} in {box.name} ({kind})")
+            try:
+                box.ensure(rebuild, status=lambda text: print(_indent(text, here)))
+            except _container.ContainerError as e:
+                raise RaidError(str(e)) from None
+            inside = (str(target) + os.sep).replace(here, "")  # /job/ in the container is the challenge directory
+            with _Indented(here, {"/job/": inside}) as out:
+                result = box.call(function, dataset, solution, out.write)
+            result["traceback"] = result.get("traceback", "").replace("/job/", inside)
+        else:
+            print(f"Checking {what}: {files} on {shown}")
+            result = _call_here(function, script, dataset, solution, target, here)
+        if "exit" in result:
+            raise RaidError("No score, scoring stopped:\n" + _indent(str(result["exit"]).strip(), here))
+        if "error" in result:
+            raise RaidError(f"No score, scoring/score.py raised {result['error']}:\n" + _indent(result["traceback"], here)
+                            + ("\nInstall the packages the challenge needs, see its setup.sh."
+                               if result["error"] == "ModuleNotFoundError" and not container else ""))
+        value = result["value"]
+        print(f"\n{_bold(f'{CHECKS[part]}: {value:.4f}')}")
         return value
 
     def submit(self, challenge):
@@ -123,7 +154,9 @@ class Raid:
         Its challenge.yaml supplies the name, the version and the metadata (title, wave, baseline, badge_score,
         badge_points); keys it leaves out keep their current values. Returns the server's summary."""
         target = Path(directory)
-        missing = [p for p in ("challenge.yaml", "data", "scoring", "hidden") if not (target / p).exists()]
+        missing = [p for p in ("challenge.yaml", "scoring", "hidden") if not (target / p).exists()]
+        if not (target / "data").is_dir() and not (target / ARTIFACT_DATA).is_dir():
+            missing.append(f"{ARTIFACT_DATA.as_posix()} (or data)")
         if missing:
             raise RaidError(f"Missing in {target}/: " + ", ".join(missing))
         with tempfile.TemporaryDirectory() as tmp:
@@ -211,6 +244,60 @@ def _status(text):
         sys.stderr.flush()
 
 
+class _Indented:
+    """Indent everything printed to stdout and stderr inside the with block by two spaces; paths under here become
+    relative, and the keys of paths (e.g. /job/ in the container) become their values."""
+
+    class _Stream:
+        def __init__(self, stream, paths):
+            self.stream, self.paths, self.line_start = stream, paths, True
+
+        def write(self, text):
+            for line in text.splitlines(keepends=True):
+                if self.line_start and line.strip():
+                    self.stream.write("  ")
+                for old, new in self.paths.items():
+                    line = line.replace(old, new)
+                self.stream.write(line)
+                self.line_start = line.endswith(("\n", "\r"))
+            self.stream.flush()
+            return len(text)
+
+        def __getattr__(self, name):
+            return getattr(self.stream, name)
+
+    def __init__(self, here=None, paths=None):
+        self.paths = {here: ""} if here else {}
+        self.paths.update(paths or {})
+
+    def __enter__(self):
+        self.saved = sys.stdout, sys.stderr
+        sys.stdout, sys.stderr = self._Stream(sys.stdout, self.paths), self._Stream(sys.stderr, self.paths)
+        return sys.stdout
+
+    def __exit__(self, *exc):
+        for stream in (sys.stdout, sys.stderr):
+            stream.flush()
+        sys.stdout, sys.stderr = self.saved
+
+
+def _indent(text, here):
+    return "\n".join("  " + line.replace(here, "") if line.strip() else line for line in text.splitlines())
+
+
+def _traceback(error, target):
+    """The traceback of an error in score.py, only with the frames of the challenge directory (score.py, solution)."""
+    import traceback
+    frames = [f for f in traceback.extract_tb(error.__traceback__)
+              if target in Path(f.filename).resolve().parents]
+    lines = traceback.format_list(frames) + traceback.format_exception_only(error)
+    return "".join(lines).rstrip()
+
+
+def _bold(text):
+    return f"\033[1m{text}\033[0m" if sys.stdout.isatty() and not os.environ.get("NO_COLOR") else text
+
+
 class _Progress:
     """A one-line progress bar on stderr, only when it is a terminal. Without a known total it counts bytes."""
 
@@ -261,6 +348,47 @@ def _size(n):
     return f"{n / 1e6:.1f} MB" if n >= 1e6 else f"{n / 1e3:.0f} kB"
 
 
+def _data(target):
+    """The challenge's data: artifact/data/ in challenges split into artifact/ and method/, else data/."""
+    return target / ARTIFACT_DATA if (target / ARTIFACT_DATA).is_dir() else target / "data"
+
+
+def _functions(script):
+    """The names of the top-level functions of score.py, None if it cannot be parsed (running it shows why)."""
+    try:
+        tree = ast.parse(script.read_bytes())
+    except (SyntaxError, ValueError):
+        return None
+    return {n.name for n in tree.body if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef))}
+
+
+def _call_here(function, script, dataset, solution, target, here):
+    """Call function of score.py on dataset in this process, as the container's runner does. Returns its result dict."""
+    paths = [str(script.parent), str(solution)]  # score.py imports the solution, e.g. `from solution import md5`
+    sys.path[:0] = paths
+    os.environ["RAID_SOLUTION"] = str(solution)  # where score.py finds files that are not imported, e.g. solution.c
+    try:
+        with _Indented(here):  # score.py's own output, set off from the result
+            spec = importlib.util.spec_from_file_location("score", script)
+            module = importlib.util.module_from_spec(spec)
+            spec.loader.exec_module(module)
+            value = getattr(module, function)(dataset)
+        if isinstance(value, bool) or not isinstance(value, (int, float)):
+            return {"error": "TypeError", "traceback": f"{function}() must return a number, not {value!r}"}
+        return {"value": float(value)}
+    except SystemExit as e:  # score.py stops with sys.exit("why"), e.g. when the solution does not compile
+        return {"exit": e.code if isinstance(e.code, str) else f"exit code {e.code}"}
+    except Exception as e:
+        return {"error": type(e).__name__, "traceback": _traceback(e, target)}
+    finally:
+        del os.environ["RAID_SOLUTION"]
+        for p in paths:
+            sys.path.remove(p)
+        for name, mod in list(sys.modules.items()):  # a later check sees the current solution
+            if target in Path(getattr(mod, "__file__", None) or "/").resolve().parents:
+                del sys.modules[name]
+
+
 def _check(challenge):
     if not isinstance(challenge, str) or not NAME_RE.match(challenge):
         raise RaidError(f"Invalid challenge name {challenge!r}.")
@@ -268,12 +396,14 @@ def _check(challenge):
 
 
 def _extract(archive, target, part):
-    """Extract only regular files and directories below part/, never outside target. source/ maps to target."""
+    """Extract only regular files and directories below part/, never outside target. source/ maps to target, the
+    data archive of a challenge split into artifact/ and method/ holds artifact/data/."""
     root = target.resolve()
+    allowed = [part, ARTIFACT_DATA.as_posix()] if part == "data" else [part]
     with tarfile.open(archive, "r:gz") as tar:
         for member in tar:
             name = posixpath.normpath(member.name)
-            if name != part and not name.startswith(part + "/"):
+            if not any(name == a or name.startswith(a + "/") for a in allowed) and not (part == "data" and name == "artifact"):
                 raise RaidError(f"Unexpected entry {member.name!r} in the {part} archive.")
             if part == "source":
                 name = name[len("source/"):] or "."
@@ -307,14 +437,16 @@ def _ignored(target):
 
 def _pack(target, archive, own=OWN):
     """tar.gz of the challenge directory without the top-level entries in own (by default data/, scoring/,
-    hidden/, reference/ and challenge.yaml) and the .raidignore patterns. Regular files only; caches,
-    VCS data and symlinks are left out."""
+    hidden/, reference/, challenge.yaml and entrypoint.sh, plus artifact/data/) and the .raidignore patterns.
+    Regular files only; caches, VCS data and symlinks are left out."""
+    nested = {ARTIFACT_DATA} if own is OWN else set()
     ignored = _ignored(target)
     with tarfile.open(archive, "w:gz") as tar:
         for dirpath, dirnames, filenames in os.walk(target):
             rel = Path(dirpath).relative_to(target)
             top = rel == Path(".")
-            dirnames[:] = sorted(d for d in dirnames if d not in SKIP and not (top and d in own) and not ignored(rel / d))
+            dirnames[:] = sorted(d for d in dirnames if d not in SKIP and not (top and d in own) and rel / d not in nested
+                                 and not ignored(rel / d))
             files = sorted(f for f in filenames if f not in SKIP and not (top and f in own) and not ignored(rel / f))
             for name in dirnames + files:
                 path = Path(dirpath) / name

mlsec_raid-0.5.0/src/raid/_container.py

@@ -0,0 +1,177 @@
+"""The local container raid-<challenge> in which task-check runs scoring/score.py, built like the backend's sandbox.
+
+With Docker it is an image on the backend's base (Debian trixie slim with python3), the challenge's Debian packages
+and the challenge's setup.sh, run as root during the build. With only udocker it is a udocker container of
+debian:trixie in which the same is installed. Either is built again only when setup.sh or the packages change: the
+image's label raid.setup (Docker) or the container's /etc/raid-setup.sha256 (udocker) holds their hash.
+"""
+import hashlib
+import json
+import os
+import re
+import shutil
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+DOCKER_BASE = "debian:trixie-slim"  # the base of the backend's sandbox (raid-scoring-service runner/)
+UDOCKER_BASE = "debian:trixie"
+RECIPE = "1"  # part of the hash: a new recipe rebuilds every container
+# Trusted directories first, as on the backend: setup.sh's commands in /usr/local come last.
+PATH = "/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin"
+PASSED_ENV = re.compile(r"^(RAID_LLM_\w+|https?_proxy|no_proxy|HTTPS?_PROXY|NO_PROXY)$")  # handed into the container
+RUNNER = Path(__file__).with_name("_runner.py")
+
+
+class ContainerError(Exception):
+    pass
+
+
+def packages(target):
+    """The challenge's Debian packages: scoring/packages.txt (from the server) or, for challenge authors,
+    requirements in challenge.yaml."""
+    listed = target / "scoring" / "packages.txt"
+    if listed.is_file():
+        return listed.read_text().split()
+    meta = target / "challenge.yaml"
+    return _requirements(meta.read_text()) if meta.is_file() else []
+
+
+def _requirements(text):
+    """The requirements list of challenge.yaml, as a block (- gcc) or a flow sequence ([gcc, make])."""
+    lines = [line.split("#", 1)[0].rstrip() for line in text.splitlines()]
+    for i, line in enumerate(lines):
+        m = re.match(r"^requirements:\s*(.*)$", line)
+        if not m:
+            continue
+        if m.group(1).startswith("["):
+            flow = " ".join([m.group(1)] + lines[i + 1:])
+            return [p.strip().strip("'\"") for p in flow[1:flow.index("]")].split(",") if p.strip()]
+        found = []
+        for item in lines[i + 1:]:
+            if item and not item[0].isspace():
+                break
+            if item.strip().startswith("-"):
+                found.append(item.strip()[1:].strip().strip("'\""))
+        return found
+    return []
+
+
+def engine():
+    """docker when it is installed and its daemon answers, else udocker when it is installed, else None."""
+    if shutil.which("docker"):
+        try:
+            if subprocess.run(["docker", "info"], capture_output=True, timeout=30).returncode == 0:
+                return "docker"
+        except subprocess.TimeoutExpired:
+            pass
+    return "udocker" if shutil.which("udocker") else None
+
+
+class Container:
+    def __init__(self, challenge, target, kind):
+        self.name, self.target, self.kind = f"raid-{challenge}", target, kind
+        self.packages = packages(target)
+        setup = target / "setup.sh"
+        self.setup = setup.read_bytes() if setup.is_file() else b""
+        base = DOCKER_BASE if kind == "docker" else UDOCKER_BASE
+        self.hash = hashlib.sha256(f"{RECIPE}\0{base}\0{' '.join(self.packages)}\0".encode() + self.setup).hexdigest()
+
+    # ---------- building ----------
+
+    def ensure(self, rebuild=False, status=print):
+        """Build the container unless it exists with the current setup.sh and packages."""
+        if not rebuild and self._built_hash() == self.hash:
+            return
+        status(f"Building {self.name} with {'setup.sh' if self.setup.strip() else 'no setup.sh'}"
+               f"{' and ' + ' '.join(self.packages) if self.packages else ''} (once, and again when they change) ...")
+        with tempfile.TemporaryDirectory() as tmp:
+            Path(tmp, "setup.sh").write_bytes(self.setup)
+            if self.kind == "docker":
+                self._build_docker(Path(tmp))
+            else:
+                self._build_udocker(Path(tmp))
+
+    def _install(self):
+        return ("apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends "
+                f"python3 ca-certificates {' '.join(self.packages)}")
+
+    def _built_hash(self):
+        if self.kind == "docker":
+            found = subprocess.run(["docker", "image", "inspect", "-f", '{{index .Config.Labels "raid.setup"}}',
+                                    self.name], capture_output=True, text=True)
+            return found.stdout.strip() if found.returncode == 0 else None
+        found = subprocess.run(self._udocker("inspect", "-p", self.name), capture_output=True, text=True)
+        if found.returncode:
+            return None
+        stamp = Path(found.stdout.strip().splitlines()[-1]) / "etc" / "raid-setup.sha256"
+        return stamp.read_text().strip() if stamp.is_file() else ""
+
+    def _build_docker(self, context):
+        dockerfile = (f"FROM {DOCKER_BASE} AS trusted\n"
+                      f"RUN {self._install()} && rm -rf /var/lib/apt/lists/*\n")
+        if self.setup.strip():  # as on the backend: afterwards the trusted image is copied back over setup.sh's changes
+            dockerfile += ("FROM trusted\n"
+                           "COPY setup.sh /tmp/raid-setup/setup.sh\n"
+                           "RUN cd /tmp/raid-setup && DEBIAN_FRONTEND=noninteractive bash setup.sh\n"
+                           "COPY --from=trusted / /\n"
+                           "RUN rm -rf /tmp/raid-setup\n")
+        (context / "Dockerfile").write_text(dockerfile)
+        self._step(["docker", "build", "-t", self.name, "--label", f"raid.setup={self.hash}", str(context)], setup=True)
+
+    def _build_udocker(self, context):
+        if self._built_hash() is not None:
+            self._step(self._udocker("rm", self.name))
+        if subprocess.run(self._udocker("inspect", UDOCKER_BASE), capture_output=True).returncode:
+            self._step(self._udocker("pull", UDOCKER_BASE))
+        self._step(self._udocker("create", f"--name={self.name}", UDOCKER_BASE))
+        script = (f"set -e; {self._install()}; "
+                  "cd /raid-setup && DEBIAN_FRONTEND=noninteractive bash setup.sh; "
+                  f"echo {self.hash} > /etc/raid-setup.sha256")
+        try:
+            self._step(self._udocker("run", "--user=root", f"-v={context}:/raid-setup", self.name,
+                                     "/bin/bash", "-c", script), setup=True)
+        except ContainerError:
+            subprocess.run(self._udocker("rm", self.name), capture_output=True)  # never reused half-built
+            raise
+
+    def _step(self, cmd, setup=False):
+        """Run a build command; on failure raise ContainerError with the end of its output. setup: the command
+        installs the packages and runs setup.sh."""
+        proc = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.DEVNULL)
+        if proc.returncode:
+            tail = "\n".join(proc.stdout.decode(errors="replace").rstrip().splitlines()[-40:])
+            step = " ".join(c for c in cmd[:4] if c not in ("-q", "--allow-root"))
+            raise ContainerError(f"Building {self.name} failed in `{step} ...`:\n{tail}"
+                                 + ("\nCheck setup.sh: it runs as root in the image build, with internet access." if setup else ""))
+
+    def _udocker(self, *args):
+        return ["udocker", "-q", *(["--allow-root"] if hasattr(os, "geteuid") and os.geteuid() == 0 else []), *args]
+
+    # ---------- running ----------
+
+    def call(self, function, dataset, solution, write=None):
+        """Run function (score, method or artifact) of scoring/score.py on dataset in the container, with the
+        challenge directory at /job. Output goes to write line by line. Returns the runner's result dict."""
+        write = write or sys.stdout.write
+        inside = lambda p: "/job/" + p.relative_to(self.target).as_posix() if p != self.target else "/job"  # noqa: E731
+        env = {"HOME": "/tmp", "PATH": PATH, "RAID_SOLUTION": inside(solution), "PYTHONDONTWRITEBYTECODE": "1",
+               **{k: v for k, v in os.environ.items() if PASSED_ENV.match(k)}}
+        with tempfile.TemporaryDirectory() as tmp:
+            shutil.copy(RUNNER, Path(tmp, "runner.py"))
+            command = ["python3", "-IS", "/raid-check/runner.py", function, inside(dataset), "/raid-check/result.json"]
+            if self.kind == "docker":
+                user = ["--user", f"{os.getuid()}:{os.getgid()}"] if hasattr(os, "getuid") else []
+                cmd = ["docker", "run", "--rm", "--init", *user, "-v", f"{self.target}:/job", "-v", f"{tmp}:/raid-check",
+                       "-w", "/job", *[f"--env={k}={v}" for k, v in env.items()], self.name, *command]
+            else:
+                cmd = self._udocker("run", f"-v={self.target}:/job", f"-v={tmp}:/raid-check", "--workdir=/job",
+                                    *[f"--env={k}={v}" for k, v in env.items()], self.name, *command)
+            with subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.DEVNULL) as proc:
+                for line in iter(proc.stdout.readline, b""):
+                    write(line.decode(errors="replace"))
+            result = Path(tmp, "result.json")
+            if not result.is_file():
+                return {"exit": f"the container {self.name} exited with code {proc.returncode} before score.py finished"}
+            return json.loads(result.read_text())

mlsec_raid-0.5.0/src/raid/_runner.py

@@ -0,0 +1,46 @@
+"""Runs inside the raid-<challenge> container: python3 -IS _runner.py <function> <dataset> <result.json>
+
+Calls <function> (score, method or artifact) of scoring/score.py in the working directory (the challenge directory)
+with the dataset path and writes {"value": ...}, {"exit": ...} or {"error": ..., "traceback": ...} to result.json.
+Like the backend, score.py sees only Debian's Python packages.
+"""
+import importlib.util
+import json
+import os
+import sys
+import traceback
+from pathlib import Path
+
+DEBIAN_PACKAGES = "/usr/lib/python3/dist-packages"
+
+
+def main(function, dataset, out):
+    job = Path.cwd()
+    script = job / "scoring" / "score.py"
+    solution = os.environ.get("RAID_SOLUTION", str(job))
+    sys.dont_write_bytecode = True  # no __pycache__/ in the challenge directory
+    sys.path.append(DEBIAN_PACKAGES)
+    sys.path[:0] = [str(script.parent), solution]  # score.py imports the solution, e.g. `from solution import md5`
+    try:
+        spec = importlib.util.spec_from_file_location("score", script)
+        module = importlib.util.module_from_spec(spec)
+        spec.loader.exec_module(module)
+        value = getattr(module, function)(Path(dataset))
+        if isinstance(value, bool) or not isinstance(value, (int, float)):
+            result = {"error": "TypeError", "traceback": f"{function}() must return a number, not {value!r}"}
+        else:
+            result = {"value": float(value)}
+    except SystemExit as e:  # score.py stops with sys.exit("why"), e.g. when the solution does not compile
+        result = {"exit": e.code if isinstance(e.code, str) else f"exit code {e.code}"}
+    except BaseException as e:
+        frames = [f for f in traceback.extract_tb(e.__traceback__) if job in Path(f.filename).resolve().parents]
+        result = {"error": type(e).__name__,
+                  "traceback": "".join(traceback.format_list(frames) + traceback.format_exception_only(e)).rstrip()}
+    sys.stdout.flush()
+    sys.stderr.flush()
+    with open(out, "w") as f:
+        json.dump(result, f)
+
+
+if __name__ == "__main__":
+    main(*sys.argv[1:4])

{mlsec_raid-0.4.2 → mlsec_raid-0.5.0}/src/raid/cli.py

@@ -5,8 +5,9 @@ import sys
 from . import Raid, RaidError
 
 
-def _run(description, action, key=True, options=(), target=("challenge", "challenge name, e.g. self-assessment")):
-    parser = argparse.ArgumentParser(description=description)
+def _run(description, action, key=True, options=(), target=("challenge", "challenge name, e.g. self-assessment"),
+         parser=None):
+    parser = parser or argparse.ArgumentParser(description=description)
     if key:
         parser.add_argument("--api-key", help="your API key (default: $RAID_KEY)")
     for flag, text in options:
@@ -26,13 +27,32 @@ def fetch():
 
 
 def check():
-    _run("Score ./<challenge>/ locally on data/.", lambda raid, a: raid.check(a.challenge, a.hidden, a.reference),
-         key=False, options=[("--hidden", "challenge authors: score on hidden/ instead of data/"),
-                             ("--reference", "challenge authors: score reference/ instead of the student files")])
+    """task check [method|artifact] <challenge>: the method, the artifact or both (first the method)."""
+    parser = argparse.ArgumentParser(
+        description="Check ./<challenge>/ locally on its data in the container raid-<challenge>, built like our "
+                    "backend's with the challenge's setup.sh. `task check method <challenge>` runs your method, "
+                    "`task check artifact <challenge>` scores your artifact, `task check <challenge>` does both.",
+        usage="%(prog)s [-h] [options] [method|artifact] challenge")
+    parser.add_argument("names", nargs="+", metavar="[method|artifact] challenge", help=argparse.SUPPRESS)
+
+    def action(raid, a):
+        names = a.names
+        if len(names) == 2 and names[1] in ("method", "artifact") and names[0] not in ("method", "artifact"):
+            names = names[::-1]  # task check dawn method
+        if len(names) > 2 or len(names) == 2 and names[0] not in ("method", "artifact"):
+            parser.error("expected [method|artifact] <challenge>")
+        part, challenge = (None, names[0]) if len(names) == 1 else names
+        raid.check(challenge, part, a.hidden, a.reference, container=not a.no_container, rebuild=a.rebuild)
+
+    _run(None, action, key=False, target=None, parser=parser, options=[
+        ("--rebuild", "build the container raid-<challenge> again even if setup.sh did not change"),
+        ("--no-container", "check on this machine instead of in the container (needs the challenge's packages)"),
+        ("--hidden", "challenge authors: score on hidden/ instead of the data"),
+        ("--reference", "challenge authors: score reference/ instead of the student files")])
 
 
 def submit():
-    _run("Submit ./<challenge>/ for scoring (without data/, scoring/ and .raidignore patterns).",
+    _run("Submit ./<challenge>/ for scoring (without its data, scoring/ and .raidignore patterns).",
          lambda raid, a: raid.submit(a.challenge))
 
 

mlsec_raid-0.4.2/PKG-INFO

@@ -1,30 +0,0 @@
-Metadata-Version: 2.3
-Name: mlsec-raid
-Version: 0.4.2
-Summary: Client for the RAID competition system
-Requires-Python: >=3.9
-Description-Content-Type: text/markdown
-
-# raid
-
-Python client for the RAID competition system.
-
-```bash
-pip install mlsec-raid                                    # or: uv tool install mlsec-raid
-export RAID_KEY=<your-api-key>                            # or pass --api-key <your-api-key>
-
-task-fetch <challenge>    # downloads the challenge into ./<challenge>/
-task-check <challenge>    # scores it locally with scoring/score.py on data/
-task-submit <challenge>   # submits ./<challenge>/ without data/, scoring/ and .raidignore patterns
-task-show                 # lists your jobs with their state (pending, running, success, failed) and score,
-                          # and the submissions left today on challenges with a daily limit
-task-log <job>            # prints the log of a finished job, including the errors of your solution
-```
-
-Every command also works as `task <command>`, e.g. `task submit <challenge>`.
-
-The same is available in Python: `Raid(api_key).fetch(...)`, `.check(...)`, `.submit(...)`, `.push(...)`, `.wait(id)`, `.status()`, `.log(id)`, `.howto()`.
-
-Challenge authors: `task-check <challenge> --hidden --reference` scores `reference/` on `hidden/`. Admins upload a challenge directory with `task-push <directory>` and their own API key (created in the admin console): the directory's `challenge.yaml` sets the name, version, wave, baseline, badge and `submissions_per_day` (keys it leaves out keep their current values), and the upload becomes the active version.
-
-Development: `uv sync` sets up the environment, `uv build` builds the package. Set `RAID_URL` to use another server. Release: bump `version` in `pyproject.toml`, then `uv build && uv publish` (PyPI token in `UV_PUBLISH_TOKEN`).

mlsec_raid-0.4.2/README.md

@@ -1,23 +0,0 @@
-# raid
-
-Python client for the RAID competition system.
-
-```bash
-pip install mlsec-raid                                    # or: uv tool install mlsec-raid
-export RAID_KEY=<your-api-key>                            # or pass --api-key <your-api-key>
-
-task-fetch <challenge>    # downloads the challenge into ./<challenge>/
-task-check <challenge>    # scores it locally with scoring/score.py on data/
-task-submit <challenge>   # submits ./<challenge>/ without data/, scoring/ and .raidignore patterns
-task-show                 # lists your jobs with their state (pending, running, success, failed) and score,
-                          # and the submissions left today on challenges with a daily limit
-task-log <job>            # prints the log of a finished job, including the errors of your solution
-```
-
-Every command also works as `task <command>`, e.g. `task submit <challenge>`.
-
-The same is available in Python: `Raid(api_key).fetch(...)`, `.check(...)`, `.submit(...)`, `.push(...)`, `.wait(id)`, `.status()`, `.log(id)`, `.howto()`.
-
-Challenge authors: `task-check <challenge> --hidden --reference` scores `reference/` on `hidden/`. Admins upload a challenge directory with `task-push <directory>` and their own API key (created in the admin console): the directory's `challenge.yaml` sets the name, version, wave, baseline, badge and `submissions_per_day` (keys it leaves out keep their current values), and the upload becomes the active version.
-
-Development: `uv sync` sets up the environment, `uv build` builds the package. Set `RAID_URL` to use another server. Release: bump `version` in `pyproject.toml`, then `uv build && uv publish` (PyPI token in `UV_PUBLISH_TOKEN`).