mlsec-raid 0.4.0__tar.gz → 0.5.0__tar.gz

mlsec_raid-0.5.0/PKG-INFO

@@ -0,0 +1,45 @@
+Metadata-Version: 2.3
+Name: mlsec-raid
+Version: 0.5.0
+Summary: Client for the RAID competition system
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+
+# raid
+
+Python client for the RAID competition system.
+
+```bash
+pip install mlsec-raid                                    # or: uv tool install mlsec-raid
+export RAID_KEY=<your-api-key>                            # or pass --api-key <your-api-key>
+
+task-fetch <challenge>            # downloads the challenge into ./<challenge>/
+task-check <challenge>            # scores it locally with scoring/score.py on its data, in a container
+task-check method <challenge>     # only the method: runs method/run like our backend's sanity check
+task-check artifact <challenge>   # only the artifact: scores artifact/result/
+task-submit <challenge>           # submits ./<challenge>/ without its data, scoring/ and .raidignore patterns
+task-show                         # lists your jobs with their state (pending, running, success, failed) and
+                                  # score, and the submissions left today on challenges with a daily limit
+task-log <job>                    # prints the log of a finished job, including the errors of your solution
+```
+
+Every command also works as `task <command>`, e.g. `task submit <challenge>` or `task check method <challenge>`.
+
+Challenges like dawn separate the method from the artifact:
+
+```
+<challenge>/
+  artifact/data/     the data, never submitted
+  artifact/result/   the artifact you submit, e.g. dawn's re-implementations
+  method/run         your method, which the backend runs once as a sanity check (dawn: on the simple target)
+  scoring/score.py   method(), artifact() and score(), the backend's scoring
+  setup.sh           your installs
+```
+
+`task-check` runs `scoring/score.py` in the container `raid-<challenge>`, built like the backend's sandbox: with Docker an image on Debian trixie (slim) with python3 and the challenge's Debian packages (`scoring/packages.txt`), in which the challenge's `setup.sh` runs as root during the build; with only udocker (`pip install udocker`) a udocker container of `debian:trixie` set up the same way. It is built again only when `setup.sh` or the packages change (their hash is the image's `raid.setup` label, or `/etc/raid-setup.sha256` in the udocker container), or with `--rebuild`. Like on the backend, `score.py` sees only Debian's Python packages; unlike there, the method may use the internet. `task-check method` calls `score.py`'s `method()`, `task-check artifact` its `artifact()` and `task-check` its `score()`, which checks the method first and then the artifact. Challenges without that split (e.g. self-assessment) have only `task-check <challenge>`. `--no-container` checks on this machine instead.
+
+The same is available in Python: `Raid(api_key).fetch(...)`, `.check(...)`, `.submit(...)`, `.push(...)`, `.wait(id)`, `.status()`, `.log(id)`, `.howto()`.
+
+Challenge authors: `task-check <challenge> --hidden --reference` scores `reference/` on `hidden/` (the packages come from `challenge.yaml`). Admins upload a challenge directory with `task-push <directory>` and their own API key (created in the admin console): the directory's `challenge.yaml` sets the name, version, wave, baseline, badge and `submissions_per_day` (keys it leaves out keep their current values), and the upload becomes the active version.
+
+Development: `uv sync` sets up the environment, `uv build` builds the package. Set `RAID_URL` to use another server. Release: bump `version` in `pyproject.toml`, then `uv build && uv publish` (PyPI token in `UV_PUBLISH_TOKEN`).

mlsec_raid-0.5.0/README.md

@@ -0,0 +1,38 @@
+# raid
+
+Python client for the RAID competition system.
+
+```bash
+pip install mlsec-raid                                    # or: uv tool install mlsec-raid
+export RAID_KEY=<your-api-key>                            # or pass --api-key <your-api-key>
+
+task-fetch <challenge>            # downloads the challenge into ./<challenge>/
+task-check <challenge>            # scores it locally with scoring/score.py on its data, in a container
+task-check method <challenge>     # only the method: runs method/run like our backend's sanity check
+task-check artifact <challenge>   # only the artifact: scores artifact/result/
+task-submit <challenge>           # submits ./<challenge>/ without its data, scoring/ and .raidignore patterns
+task-show                         # lists your jobs with their state (pending, running, success, failed) and
+                                  # score, and the submissions left today on challenges with a daily limit
+task-log <job>                    # prints the log of a finished job, including the errors of your solution
+```
+
+Every command also works as `task <command>`, e.g. `task submit <challenge>` or `task check method <challenge>`.
+
+Challenges like dawn separate the method from the artifact:
+
+```
+<challenge>/
+  artifact/data/     the data, never submitted
+  artifact/result/   the artifact you submit, e.g. dawn's re-implementations
+  method/run         your method, which the backend runs once as a sanity check (dawn: on the simple target)
+  scoring/score.py   method(), artifact() and score(), the backend's scoring
+  setup.sh           your installs
+```
+
+`task-check` runs `scoring/score.py` in the container `raid-<challenge>`, built like the backend's sandbox: with Docker an image on Debian trixie (slim) with python3 and the challenge's Debian packages (`scoring/packages.txt`), in which the challenge's `setup.sh` runs as root during the build; with only udocker (`pip install udocker`) a udocker container of `debian:trixie` set up the same way. It is built again only when `setup.sh` or the packages change (their hash is the image's `raid.setup` label, or `/etc/raid-setup.sha256` in the udocker container), or with `--rebuild`. Like on the backend, `score.py` sees only Debian's Python packages; unlike there, the method may use the internet. `task-check method` calls `score.py`'s `method()`, `task-check artifact` its `artifact()` and `task-check` its `score()`, which checks the method first and then the artifact. Challenges without that split (e.g. self-assessment) have only `task-check <challenge>`. `--no-container` checks on this machine instead.
+
+The same is available in Python: `Raid(api_key).fetch(...)`, `.check(...)`, `.submit(...)`, `.push(...)`, `.wait(id)`, `.status()`, `.log(id)`, `.howto()`.
+
+Challenge authors: `task-check <challenge> --hidden --reference` scores `reference/` on `hidden/` (the packages come from `challenge.yaml`). Admins upload a challenge directory with `task-push <directory>` and their own API key (created in the admin console): the directory's `challenge.yaml` sets the name, version, wave, baseline, badge and `submissions_per_day` (keys it leaves out keep their current values), and the upload becomes the active version.
+
+Development: `uv sync` sets up the environment, `uv build` builds the package. Set `RAID_URL` to use another server. Release: bump `version` in `pyproject.toml`, then `uv build && uv publish` (PyPI token in `UV_PUBLISH_TOKEN`).

{mlsec_raid-0.4.0 → mlsec_raid-0.5.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "mlsec-raid"
-version = "0.4.0"
+version = "0.5.0"
 description = "Client for the RAID competition system"
 readme = "README.md"
 requires-python = ">=3.9"

mlsec_raid-0.5.0/src/raid/__init__.py

@@ -0,0 +1,457 @@
+"""Client for the RAID competition system: fetch challenges, check and submit solutions."""
+import ast
+import fnmatch
+import importlib.util
+import json
+import os
+import posixpath
+import re
+import shutil
+import sys
+import tarfile
+import tempfile
+import time
+import urllib.error
+import urllib.request
+from pathlib import Path
+
+__all__ = ["Raid", "RaidError"]
+__version__ = "0.5.0"
+
+DEFAULT_URL = "https://raid.mlsec.tu-berlin.de"
+PARTS = ("source", "data", "scoring")  # source/ is unpacked into the challenge directory itself
+# Parts of the challenge directory that are never submitted: these top-level entries and artifact/data/
+OWN = {"data", "scoring", "hidden", "reference", "challenge.yaml", "entrypoint.sh"}
+ARTIFACT_DATA = Path("artifact", "data")  # the data/ part of challenges split into artifact/ and method/
+CHECKS = {"method": "Method score", "artifact": "Artifact score", None: "Score"}  # the part checked: what it prints
+SKIP = {"__pycache__", ".git", ".ipynb_checkpoints", ".venv", "venv", ".DS_Store"}
+NAME_RE = re.compile(r"^[a-z0-9][a-z0-9-]{0,62}$")
+
+
+class RaidError(Exception):
+    pass
+
+
+class Raid:
+    """raid = Raid(api_key); print(raid.howto())"""
+
+    def __init__(self, api_key=None, url=None, path="."):
+        self.api_key = (api_key or os.environ.get("RAID_KEY") or "").strip()
+        self.url = (url or os.environ.get("RAID_URL") or DEFAULT_URL).rstrip("/")
+        self.path = Path(path)
+
+    def howto(self):
+        """Explain the workflow."""
+        return self._json("/api/howto")["text"]
+
+    def fetch(self, challenge, force=False):
+        """Download a challenge into ./<challenge>/: its files, data/ (artifact/data/) and scoring/.
+
+        An existing directory is left untouched unless force=True, which overwrites the challenge files.
+        """
+        target = self.path / _check(challenge)
+        if target.exists() and not force:
+            print(f"{target}/ already exists. Fetch with force=True (task-fetch --force) to download it again "
+                  "(this overwrites the challenge files).")
+            return target
+        info = self._json(f"/api/challenges/{challenge}")
+        with tempfile.TemporaryDirectory() as tmp:
+            for part in PARTS:
+                with self._request("GET", f"/api/challenges/{challenge}/{part}.tar.gz", timeout=300) as r, \
+                        open(Path(tmp) / f"{part}.tar.gz", "wb") as f, \
+                        _Progress(f"Downloading {part}", r.headers.get("Content-Length")) as bar:
+                    while chunk := r.read(1 << 16):
+                        f.write(chunk)
+                        bar.update(len(chunk))
+            target.mkdir(parents=True, exist_ok=True)
+            for part in PARTS:
+                _extract(Path(tmp) / f"{part}.tar.gz", target, part)
+        print(f"Fetched {challenge} (version {info['version']}) into {target}/. Start with {target / 'README.txt'}.")
+        return target
+
+    def check(self, challenge, part=None, hidden=False, reference=False, container=True, rebuild=False):
+        """Check ./<challenge>/ locally with scoring/score.py on its data (artifact/data/ or data/). Returns the score.
+
+        part="method" calls score.py's method(), part="artifact" its artifact() and part=None its score(), which
+        checks the method first and then the artifact. It runs in the container raid-<challenge>, built like the
+        backend's sandbox with the challenge's setup.sh (Docker, else udocker) and rebuilt when setup.sh changes or
+        rebuild=True; container=False runs it on this machine instead.
+
+        For challenge authors: hidden=True scores on hidden/, reference=True scores the files
+        in reference/ in place of the student files. Raises RaidError when scoring stops without a score.
+        """
+        if part not in CHECKS:
+            raise RaidError(f"Invalid check {part!r}, use method or artifact.")
+        target = (self.path / _check(challenge)).resolve()
+        script = target / "scoring" / "score.py"
+        dataset = target / "hidden" if hidden else _data(target)
+        solution = target / "reference" if reference else target
+        for path in (script, dataset, solution):
+            if not path.exists():
+                raise RaidError(f"{path} not found." + (" Fetch the challenge first." if path == script else ""))
+        function = part or "score"
+        defined = _functions(script)
+        if part and defined is not None and part not in defined:
+            raise RaidError(f"{challenge} has no separate {part} check (scoring/score.py has no {part}()). "
+                            f"Run task check {challenge}.")
+        files = "reference/" if reference else "your files"
+        what = f"{challenge} {part}" if part else challenge
+        here = str(Path.cwd().resolve()) + os.sep  # shown paths are relative, e.g. self-assessment/solution.c
+        shown = f"{dataset.relative_to(target).as_posix()}/"
+        if container:
+            from . import _container
+            kind = _container.engine()
+            if kind is None:
+                raise RaidError("task check needs Docker or udocker (pip install udocker) to build the container "
+                                f"raid-{challenge}. Pass --no-container to check on this machine instead.")
+            box = _container.Container(challenge, target, kind)
+            print(f"Checking {what}: {files} on {shown} in {box.name} ({kind})")
+            try:
+                box.ensure(rebuild, status=lambda text: print(_indent(text, here)))
+            except _container.ContainerError as e:
+                raise RaidError(str(e)) from None
+            inside = (str(target) + os.sep).replace(here, "")  # /job/ in the container is the challenge directory
+            with _Indented(here, {"/job/": inside}) as out:
+                result = box.call(function, dataset, solution, out.write)
+            result["traceback"] = result.get("traceback", "").replace("/job/", inside)
+        else:
+            print(f"Checking {what}: {files} on {shown}")
+            result = _call_here(function, script, dataset, solution, target, here)
+        if "exit" in result:
+            raise RaidError("No score, scoring stopped:\n" + _indent(str(result["exit"]).strip(), here))
+        if "error" in result:
+            raise RaidError(f"No score, scoring/score.py raised {result['error']}:\n" + _indent(result["traceback"], here)
+                            + ("\nInstall the packages the challenge needs, see its setup.sh."
+                               if result["error"] == "ModuleNotFoundError" and not container else ""))
+        value = result["value"]
+        print(f"\n{_bold(f'{CHECKS[part]}: {value:.4f}')}")
+        return value
+
+    def submit(self, challenge):
+        """Pack ./<challenge>/ (see _pack) and submit it for scoring. Returns the submission id."""
+        target = self.path / _check(challenge)
+        if not target.is_dir():
+            raise RaidError(f"{target}/ not found. Fetch the challenge first.")
+        required = self._json(f"/api/challenges/{challenge}")["required_files"]
+        missing = [f for f in required if not (target / f).is_file()]
+        if missing:
+            raise RaidError(f"Missing required files in {target}/: " + ", ".join(missing))
+        with tempfile.TemporaryDirectory() as tmp:
+            archive = Path(tmp) / "submission.tar.gz"
+            _status(f"Packing {target}/ ...")
+            _pack(target, archive)
+            size = archive.stat().st_size
+            with open(archive, "rb") as f, _Progress("Uploading", size) as bar, self._request(
+                    "POST", f"/api/submit/{challenge}", data=bar.reader(f), timeout=900,
+                    headers={"Content-Type": "application/gzip", "Content-Length": str(size)}) as r:
+                result = json.load(r)
+        print("Job submitted!\nRun task-show to see the status of your jobs.")
+        return result["id"]
+
+    def push(self, directory):
+        """Admins: upload a challenge directory (everything but reference/) as a new, active version.
+
+        Its challenge.yaml supplies the name, the version and the metadata (title, wave, baseline, badge_score,
+        badge_points); keys it leaves out keep their current values. Returns the server's summary."""
+        target = Path(directory)
+        missing = [p for p in ("challenge.yaml", "scoring", "hidden") if not (target / p).exists()]
+        if not (target / "data").is_dir() and not (target / ARTIFACT_DATA).is_dir():
+            missing.append(f"{ARTIFACT_DATA.as_posix()} (or data)")
+        if missing:
+            raise RaidError(f"Missing in {target}/: " + ", ".join(missing))
+        with tempfile.TemporaryDirectory() as tmp:
+            archive = Path(tmp) / "challenge.tar.gz"
+            _status(f"Packing {target}/ ...")
+            _pack(target, archive, own={"reference"})
+            size = archive.stat().st_size
+            with open(archive, "rb") as f, _Progress("Uploading", size) as bar, self._request(
+                    "POST", "/api/admin/push", data=bar.reader(f), timeout=3600,
+                    headers={"Content-Type": "application/gzip", "Content-Length": str(size)}) as r:
+                result = json.load(r)
+        wave = f"wave {result['wave']}" if result["wave"] else "the self-assessment"
+        print(f"Pushed {result['name']} version {result['version']} ({size / 1e6:.2f} MB) to {wave}"
+              + (", now active." if result["active"] else "."))
+        return result
+
+    def wait(self, submission_id, poll=5):
+        """Wait until a submission is scored, print its log and return its details."""
+        print("Waiting for the scoring (Ctrl-C stops waiting, the scoring continues) ...", flush=True)
+        while True:
+            sub = self._json(f"/api/submissions/{int(submission_id)}")
+            if sub["status"] in ("success", "failed"):
+                break
+            time.sleep(poll)
+        print(sub["log"] or "No log output.")
+        print(f"Submission {sub['id']} (version {sub['version']}): {sub['status']}"
+              + ("" if sub["score"] is None else f", score {sub['score']:.4f}"))
+        return sub
+
+    def status(self, challenge=None):
+        """Print your jobs (submissions, newest first) and return them as a list of dicts."""
+        query = f"?challenge={_check(challenge)}" if challenge else ""
+        result = self._json(f"/api/submissions{query}")
+        subs = result["submissions"]
+        print(f"{'JOB':>6}  {'CHALLENGE':<20} {'VERSION':<8} {'STATE':<8} {'SCORE':>10}  SUBMITTED")
+        for s in subs:
+            score = "" if s["score"] is None else f"{s['score']:.4f}"
+            print(f"{s['id']:>6}  {s['challenge']:<20} {s['version']:<8} {s['status']:<8} {score:>10}  {s['submitted_at'][:19]}")
+        if not subs:
+            print("No jobs yet.")
+        else:
+            print("Run task-log <job> to see the log of a finished job.")
+        limits = result.get("daily_limits") or []
+        if limits:
+            print("\nDaily submission limits (in competition mode per team, reset at 00:00 UTC):")
+            for d in limits:
+                left = max(d["submissions_per_day"] - d["submitted_today"], 0)
+                print(f"  {d['challenge']:<20} {left} of {d['submissions_per_day']} left today")
+        return subs
+
+    def log(self, submission_id):
+        """Return the scoring log of one of your finished submissions: the output and errors of the scoring run."""
+        if not str(submission_id).isdigit():
+            raise RaidError(f"Invalid job id {submission_id!r}. task-show lists your jobs.")
+        sub = self._json(f"/api/submissions/{submission_id}")
+        if sub["status"] in ("pending", "running"):
+            raise RaidError(f"Job {sub['id']} is {sub['status']}. Its log is available once it has finished.")
+        return sub["log"] or "No log output."
+
+    def _request(self, method, endpoint, data=None, headers=None, timeout=60):
+        if not self.api_key:
+            raise RaidError("No API key. Pass --api-key or set RAID_KEY.")
+        req = urllib.request.Request(self.url + endpoint, data=data, method=method, headers={
+            "Authorization": f"Bearer {self.api_key}", "User-Agent": f"raid/{__version__}", **(headers or {})})
+        try:
+            return urllib.request.urlopen(req, timeout=timeout)
+        except urllib.error.HTTPError as e:
+            try:
+                message = json.load(e).get("error")
+            except (ValueError, AttributeError):
+                message = None
+            raise RaidError(message or f"{e.code} {e.reason}") from None
+        except urllib.error.URLError as e:
+            raise RaidError(f"Cannot reach {self.url}: {e.reason}") from None
+
+    def _json(self, endpoint):
+        with self._request("GET", endpoint) as r:
+            return json.load(r)
+
+
+def _status(text):
+    """A transient line on stderr, only when it is a terminal."""
+    if sys.stderr.isatty():
+        sys.stderr.write(f"\r\033[K{text}")
+        sys.stderr.flush()
+
+
+class _Indented:
+    """Indent everything printed to stdout and stderr inside the with block by two spaces; paths under here become
+    relative, and the keys of paths (e.g. /job/ in the container) become their values."""
+
+    class _Stream:
+        def __init__(self, stream, paths):
+            self.stream, self.paths, self.line_start = stream, paths, True
+
+        def write(self, text):
+            for line in text.splitlines(keepends=True):
+                if self.line_start and line.strip():
+                    self.stream.write("  ")
+                for old, new in self.paths.items():
+                    line = line.replace(old, new)
+                self.stream.write(line)
+                self.line_start = line.endswith(("\n", "\r"))
+            self.stream.flush()
+            return len(text)
+
+        def __getattr__(self, name):
+            return getattr(self.stream, name)
+
+    def __init__(self, here=None, paths=None):
+        self.paths = {here: ""} if here else {}
+        self.paths.update(paths or {})
+
+    def __enter__(self):
+        self.saved = sys.stdout, sys.stderr
+        sys.stdout, sys.stderr = self._Stream(sys.stdout, self.paths), self._Stream(sys.stderr, self.paths)
+        return sys.stdout
+
+    def __exit__(self, *exc):
+        for stream in (sys.stdout, sys.stderr):
+            stream.flush()
+        sys.stdout, sys.stderr = self.saved
+
+
+def _indent(text, here):
+    return "\n".join("  " + line.replace(here, "") if line.strip() else line for line in text.splitlines())
+
+
+def _traceback(error, target):
+    """The traceback of an error in score.py, only with the frames of the challenge directory (score.py, solution)."""
+    import traceback
+    frames = [f for f in traceback.extract_tb(error.__traceback__)
+              if target in Path(f.filename).resolve().parents]
+    lines = traceback.format_list(frames) + traceback.format_exception_only(error)
+    return "".join(lines).rstrip()
+
+
+def _bold(text):
+    return f"\033[1m{text}\033[0m" if sys.stdout.isatty() and not os.environ.get("NO_COLOR") else text
+
+
+class _Progress:
+    """A one-line progress bar on stderr, only when it is a terminal. Without a known total it counts bytes."""
+
+    def __init__(self, label, total=None):
+        self.label, self.total, self.done, self.shown = label, int(total) if total else None, 0, 0.0
+        self.tty = sys.stderr.isatty()
+
+    def __enter__(self):
+        self._draw(force=True)
+        return self
+
+    def __exit__(self, *exc):
+        if self.tty:
+            self._draw(force=True)
+            sys.stderr.write("\n" if exc[0] is None else "\r\033[K")
+            sys.stderr.flush()
+
+    def update(self, n):
+        self.done += n
+        self._draw()
+
+    def reader(self, f):
+        """f with a read() that advances the bar, for uploads."""
+        bar = self
+
+        class Reader:
+            def read(self, n=-1):
+                chunk = f.read(n)
+                bar.update(len(chunk))
+                return chunk
+        return Reader()
+
+    def _draw(self, force=False):
+        now = time.monotonic()
+        if not self.tty or (not force and now - self.shown < 0.1):
+            return
+        self.shown = now
+        if self.total:
+            frac = min(self.done / self.total, 1)
+            line = f"{self.label} [{'#' * int(frac * 30):<30}] {frac:4.0%}  {_size(self.done)} of {_size(self.total)}"
+        else:
+            line = f"{self.label} {_size(self.done)}"
+        sys.stderr.write(f"\r\033[K{line}")
+        sys.stderr.flush()
+
+
+def _size(n):
+    return f"{n / 1e6:.1f} MB" if n >= 1e6 else f"{n / 1e3:.0f} kB"
+
+
+def _data(target):
+    """The challenge's data: artifact/data/ in challenges split into artifact/ and method/, else data/."""
+    return target / ARTIFACT_DATA if (target / ARTIFACT_DATA).is_dir() else target / "data"
+
+
+def _functions(script):
+    """The names of the top-level functions of score.py, None if it cannot be parsed (running it shows why)."""
+    try:
+        tree = ast.parse(script.read_bytes())
+    except (SyntaxError, ValueError):
+        return None
+    return {n.name for n in tree.body if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef))}
+
+
+def _call_here(function, script, dataset, solution, target, here):
+    """Call function of score.py on dataset in this process, as the container's runner does. Returns its result dict."""
+    paths = [str(script.parent), str(solution)]  # score.py imports the solution, e.g. `from solution import md5`
+    sys.path[:0] = paths
+    os.environ["RAID_SOLUTION"] = str(solution)  # where score.py finds files that are not imported, e.g. solution.c
+    try:
+        with _Indented(here):  # score.py's own output, set off from the result
+            spec = importlib.util.spec_from_file_location("score", script)
+            module = importlib.util.module_from_spec(spec)
+            spec.loader.exec_module(module)
+            value = getattr(module, function)(dataset)
+        if isinstance(value, bool) or not isinstance(value, (int, float)):
+            return {"error": "TypeError", "traceback": f"{function}() must return a number, not {value!r}"}
+        return {"value": float(value)}
+    except SystemExit as e:  # score.py stops with sys.exit("why"), e.g. when the solution does not compile
+        return {"exit": e.code if isinstance(e.code, str) else f"exit code {e.code}"}
+    except Exception as e:
+        return {"error": type(e).__name__, "traceback": _traceback(e, target)}
+    finally:
+        del os.environ["RAID_SOLUTION"]
+        for p in paths:
+            sys.path.remove(p)
+        for name, mod in list(sys.modules.items()):  # a later check sees the current solution
+            if target in Path(getattr(mod, "__file__", None) or "/").resolve().parents:
+                del sys.modules[name]
+
+
+def _check(challenge):
+    if not isinstance(challenge, str) or not NAME_RE.match(challenge):
+        raise RaidError(f"Invalid challenge name {challenge!r}.")
+    return challenge
+
+
+def _extract(archive, target, part):
+    """Extract only regular files and directories below part/, never outside target. source/ maps to target, the
+    data archive of a challenge split into artifact/ and method/ holds artifact/data/."""
+    root = target.resolve()
+    allowed = [part, ARTIFACT_DATA.as_posix()] if part == "data" else [part]
+    with tarfile.open(archive, "r:gz") as tar:
+        for member in tar:
+            name = posixpath.normpath(member.name)
+            if not any(name == a or name.startswith(a + "/") for a in allowed) and not (part == "data" and name == "artifact"):
+                raise RaidError(f"Unexpected entry {member.name!r} in the {part} archive.")
+            if part == "source":
+                name = name[len("source/"):] or "."
+            dest = (root / name).resolve()
+            if dest != root and root not in dest.parents:
+                raise RaidError(f"Unsafe entry {member.name!r} in the {part} archive.")
+            if member.isdir():
+                dest.mkdir(parents=True, exist_ok=True)
+            elif member.isfile():
+                dest.parent.mkdir(parents=True, exist_ok=True)
+                with tar.extractfile(member) as src, open(dest, "wb") as out:
+                    shutil.copyfileobj(src, out, 1 << 20)
+                if member.mode & 0o111:
+                    dest.chmod(0o755)
+
+
+def _normalize(info):
+    info.uid = info.gid = 0
+    info.uname = info.gname = ""
+    info.mode = 0o755 if info.isdir() or info.mode & 0o111 else 0o644
+    return info
+
+
+def _ignored(target):
+    """Patterns of <challenge>/.raidignore (tar exclude patterns, matched against any path suffix)."""
+    path = target / ".raidignore"
+    lines = path.read_text().splitlines() if path.is_file() else []
+    patterns = [line.strip().rstrip("/") for line in lines if line.strip() and not line.startswith("#")]
+    return lambda rel: any(fnmatch.fnmatchcase("/".join(rel.parts[i:]), p) for p in patterns for i in range(len(rel.parts)))
+
+
+def _pack(target, archive, own=OWN):
+    """tar.gz of the challenge directory without the top-level entries in own (by default data/, scoring/,
+    hidden/, reference/, challenge.yaml and entrypoint.sh, plus artifact/data/) and the .raidignore patterns.
+    Regular files only; caches, VCS data and symlinks are left out."""
+    nested = {ARTIFACT_DATA} if own is OWN else set()
+    ignored = _ignored(target)
+    with tarfile.open(archive, "w:gz") as tar:
+        for dirpath, dirnames, filenames in os.walk(target):
+            rel = Path(dirpath).relative_to(target)
+            top = rel == Path(".")
+            dirnames[:] = sorted(d for d in dirnames if d not in SKIP and not (top and d in own) and rel / d not in nested
+                                 and not ignored(rel / d))
+            files = sorted(f for f in filenames if f not in SKIP and not (top and f in own) and not ignored(rel / f))
+            for name in dirnames + files:
+                path = Path(dirpath) / name
+                if path.is_symlink():
+                    print(f"Skipping symlink {path}")
+                    continue
+                if path.is_dir() or path.is_file():
+                    tar.add(path, arcname=(rel / name).as_posix(), recursive=False, filter=_normalize)

mlsec_raid-0.5.0/src/raid/_container.py

@@ -0,0 +1,177 @@
+"""The local container raid-<challenge> in which task-check runs scoring/score.py, built like the backend's sandbox.
+
+With Docker it is an image on the backend's base (Debian trixie slim with python3), the challenge's Debian packages
+and the challenge's setup.sh, run as root during the build. With only udocker it is a udocker container of
+debian:trixie in which the same is installed. Either is built again only when setup.sh or the packages change: the
+image's label raid.setup (Docker) or the container's /etc/raid-setup.sha256 (udocker) holds their hash.
+"""
+import hashlib
+import json
+import os
+import re
+import shutil
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+DOCKER_BASE = "debian:trixie-slim"  # the base of the backend's sandbox (raid-scoring-service runner/)
+UDOCKER_BASE = "debian:trixie"
+RECIPE = "1"  # part of the hash: a new recipe rebuilds every container
+# Trusted directories first, as on the backend: setup.sh's commands in /usr/local come last.
+PATH = "/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin"
+PASSED_ENV = re.compile(r"^(RAID_LLM_\w+|https?_proxy|no_proxy|HTTPS?_PROXY|NO_PROXY)$")  # handed into the container
+RUNNER = Path(__file__).with_name("_runner.py")
+
+
+class ContainerError(Exception):
+    pass
+
+
+def packages(target):
+    """The challenge's Debian packages: scoring/packages.txt (from the server) or, for challenge authors,
+    requirements in challenge.yaml."""
+    listed = target / "scoring" / "packages.txt"
+    if listed.is_file():
+        return listed.read_text().split()
+    meta = target / "challenge.yaml"
+    return _requirements(meta.read_text()) if meta.is_file() else []
+
+
+def _requirements(text):
+    """The requirements list of challenge.yaml, as a block (- gcc) or a flow sequence ([gcc, make])."""
+    lines = [line.split("#", 1)[0].rstrip() for line in text.splitlines()]
+    for i, line in enumerate(lines):
+        m = re.match(r"^requirements:\s*(.*)$", line)
+        if not m:
+            continue
+        if m.group(1).startswith("["):
+            flow = " ".join([m.group(1)] + lines[i + 1:])
+            return [p.strip().strip("'\"") for p in flow[1:flow.index("]")].split(",") if p.strip()]
+        found = []
+        for item in lines[i + 1:]:
+            if item and not item[0].isspace():
+                break
+            if item.strip().startswith("-"):
+                found.append(item.strip()[1:].strip().strip("'\""))
+        return found
+    return []
+
+
+def engine():
+    """docker when it is installed and its daemon answers, else udocker when it is installed, else None."""
+    if shutil.which("docker"):
+        try:
+            if subprocess.run(["docker", "info"], capture_output=True, timeout=30).returncode == 0:
+                return "docker"
+        except subprocess.TimeoutExpired:
+            pass
+    return "udocker" if shutil.which("udocker") else None
+
+
+class Container:
+    def __init__(self, challenge, target, kind):
+        self.name, self.target, self.kind = f"raid-{challenge}", target, kind
+        self.packages = packages(target)
+        setup = target / "setup.sh"
+        self.setup = setup.read_bytes() if setup.is_file() else b""
+        base = DOCKER_BASE if kind == "docker" else UDOCKER_BASE
+        self.hash = hashlib.sha256(f"{RECIPE}\0{base}\0{' '.join(self.packages)}\0".encode() + self.setup).hexdigest()
+
+    # ---------- building ----------
+
+    def ensure(self, rebuild=False, status=print):
+        """Build the container unless it exists with the current setup.sh and packages."""
+        if not rebuild and self._built_hash() == self.hash:
+            return
+        status(f"Building {self.name} with {'setup.sh' if self.setup.strip() else 'no setup.sh'}"
+               f"{' and ' + ' '.join(self.packages) if self.packages else ''} (once, and again when they change) ...")
+        with tempfile.TemporaryDirectory() as tmp:
+            Path(tmp, "setup.sh").write_bytes(self.setup)
+            if self.kind == "docker":
+                self._build_docker(Path(tmp))
+            else:
+                self._build_udocker(Path(tmp))
+
+    def _install(self):
+        return ("apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends "
+                f"python3 ca-certificates {' '.join(self.packages)}")
+
+    def _built_hash(self):
+        if self.kind == "docker":
+            found = subprocess.run(["docker", "image", "inspect", "-f", '{{index .Config.Labels "raid.setup"}}',
+                                    self.name], capture_output=True, text=True)
+            return found.stdout.strip() if found.returncode == 0 else None
+        found = subprocess.run(self._udocker("inspect", "-p", self.name), capture_output=True, text=True)
+        if found.returncode:
+            return None
+        stamp = Path(found.stdout.strip().splitlines()[-1]) / "etc" / "raid-setup.sha256"
+        return stamp.read_text().strip() if stamp.is_file() else ""
+
+    def _build_docker(self, context):
+        dockerfile = (f"FROM {DOCKER_BASE} AS trusted\n"
+                      f"RUN {self._install()} && rm -rf /var/lib/apt/lists/*\n")
+        if self.setup.strip():  # as on the backend: afterwards the trusted image is copied back over setup.sh's changes
+            dockerfile += ("FROM trusted\n"
+                           "COPY setup.sh /tmp/raid-setup/setup.sh\n"
+                           "RUN cd /tmp/raid-setup && DEBIAN_FRONTEND=noninteractive bash setup.sh\n"
+                           "COPY --from=trusted / /\n"
+                           "RUN rm -rf /tmp/raid-setup\n")
+        (context / "Dockerfile").write_text(dockerfile)
+        self._step(["docker", "build", "-t", self.name, "--label", f"raid.setup={self.hash}", str(context)], setup=True)
+
+    def _build_udocker(self, context):
+        if self._built_hash() is not None:
+            self._step(self._udocker("rm", self.name))
+        if subprocess.run(self._udocker("inspect", UDOCKER_BASE), capture_output=True).returncode:
+            self._step(self._udocker("pull", UDOCKER_BASE))
+        self._step(self._udocker("create", f"--name={self.name}", UDOCKER_BASE))
+        script = (f"set -e; {self._install()}; "
+                  "cd /raid-setup && DEBIAN_FRONTEND=noninteractive bash setup.sh; "
+                  f"echo {self.hash} > /etc/raid-setup.sha256")
+        try:
+            self._step(self._udocker("run", "--user=root", f"-v={context}:/raid-setup", self.name,
+                                     "/bin/bash", "-c", script), setup=True)
+        except ContainerError:
+            subprocess.run(self._udocker("rm", self.name), capture_output=True)  # never reused half-built
+            raise
+
+    def _step(self, cmd, setup=False):
+        """Run a build command; on failure raise ContainerError with the end of its output. setup: the command
+        installs the packages and runs setup.sh."""
+        proc = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.DEVNULL)
+        if proc.returncode:
+            tail = "\n".join(proc.stdout.decode(errors="replace").rstrip().splitlines()[-40:])
+            step = " ".join(c for c in cmd[:4] if c not in ("-q", "--allow-root"))
+            raise ContainerError(f"Building {self.name} failed in `{step} ...`:\n{tail}"
+                                 + ("\nCheck setup.sh: it runs as root in the image build, with internet access." if setup else ""))
+
+    def _udocker(self, *args):
+        return ["udocker", "-q", *(["--allow-root"] if hasattr(os, "geteuid") and os.geteuid() == 0 else []), *args]
+
+    # ---------- running ----------
+
+    def call(self, function, dataset, solution, write=None):
+        """Run function (score, method or artifact) of scoring/score.py on dataset in the container, with the
+        challenge directory at /job. Output goes to write line by line. Returns the runner's result dict."""
+        write = write or sys.stdout.write
+        inside = lambda p: "/job/" + p.relative_to(self.target).as_posix() if p != self.target else "/job"  # noqa: E731
+        env = {"HOME": "/tmp", "PATH": PATH, "RAID_SOLUTION": inside(solution), "PYTHONDONTWRITEBYTECODE": "1",
+               **{k: v for k, v in os.environ.items() if PASSED_ENV.match(k)}}
+        with tempfile.TemporaryDirectory() as tmp:
+            shutil.copy(RUNNER, Path(tmp, "runner.py"))
+            command = ["python3", "-IS", "/raid-check/runner.py", function, inside(dataset), "/raid-check/result.json"]
+            if self.kind == "docker":
+                user = ["--user", f"{os.getuid()}:{os.getgid()}"] if hasattr(os, "getuid") else []
+                cmd = ["docker", "run", "--rm", "--init", *user, "-v", f"{self.target}:/job", "-v", f"{tmp}:/raid-check",
+                       "-w", "/job", *[f"--env={k}={v}" for k, v in env.items()], self.name, *command]
+            else:
+                cmd = self._udocker("run", f"-v={self.target}:/job", f"-v={tmp}:/raid-check", "--workdir=/job",
+                                    *[f"--env={k}={v}" for k, v in env.items()], self.name, *command)
+            with subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.DEVNULL) as proc:
+                for line in iter(proc.stdout.readline, b""):
+                    write(line.decode(errors="replace"))
+            result = Path(tmp, "result.json")
+            if not result.is_file():
+                return {"exit": f"the container {self.name} exited with code {proc.returncode} before score.py finished"}
+            return json.loads(result.read_text())

mlsec_raid-0.5.0/src/raid/_runner.py

@@ -0,0 +1,46 @@
+"""Runs inside the raid-<challenge> container: python3 -IS _runner.py <function> <dataset> <result.json>
+
+Calls <function> (score, method or artifact) of scoring/score.py in the working directory (the challenge directory)
+with the dataset path and writes {"value": ...}, {"exit": ...} or {"error": ..., "traceback": ...} to result.json.
+Like the backend, score.py sees only Debian's Python packages.
+"""
+import importlib.util
+import json
+import os
+import sys
+import traceback
+from pathlib import Path
+
+DEBIAN_PACKAGES = "/usr/lib/python3/dist-packages"
+
+
+def main(function, dataset, out):
+    job = Path.cwd()
+    script = job / "scoring" / "score.py"
+    solution = os.environ.get("RAID_SOLUTION", str(job))
+    sys.dont_write_bytecode = True  # no __pycache__/ in the challenge directory
+    sys.path.append(DEBIAN_PACKAGES)
+    sys.path[:0] = [str(script.parent), solution]  # score.py imports the solution, e.g. `from solution import md5`
+    try:
+        spec = importlib.util.spec_from_file_location("score", script)
+        module = importlib.util.module_from_spec(spec)
+        spec.loader.exec_module(module)
+        value = getattr(module, function)(Path(dataset))
+        if isinstance(value, bool) or not isinstance(value, (int, float)):
+            result = {"error": "TypeError", "traceback": f"{function}() must return a number, not {value!r}"}
+        else:
+            result = {"value": float(value)}
+    except SystemExit as e:  # score.py stops with sys.exit("why"), e.g. when the solution does not compile
+        result = {"exit": e.code if isinstance(e.code, str) else f"exit code {e.code}"}
+    except BaseException as e:
+        frames = [f for f in traceback.extract_tb(e.__traceback__) if job in Path(f.filename).resolve().parents]
+        result = {"error": type(e).__name__,
+                  "traceback": "".join(traceback.format_list(frames) + traceback.format_exception_only(e)).rstrip()}
+    sys.stdout.flush()
+    sys.stderr.flush()
+    with open(out, "w") as f:
+        json.dump(result, f)
+
+
+if __name__ == "__main__":
+    main(*sys.argv[1:4])

{mlsec_raid-0.4.0 → mlsec_raid-0.5.0}/src/raid/cli.py

@@ -5,8 +5,9 @@ import sys
 from . import Raid, RaidError
 
 
-def _run(description, action, key=True, options=(), target=("challenge", "challenge name, e.g. self-assessment")):
-    parser = argparse.ArgumentParser(description=description)
+def _run(description, action, key=True, options=(), target=("challenge", "challenge name, e.g. self-assessment"),
+         parser=None):
+    parser = parser or argparse.ArgumentParser(description=description)
     if key:
         parser.add_argument("--api-key", help="your API key (default: $RAID_KEY)")
     for flag, text in options:
@@ -26,13 +27,32 @@ def fetch():
 
 
 def check():
-    _run("Score ./<challenge>/ locally on data/.", lambda raid, a: raid.check(a.challenge, a.hidden, a.reference),
-         key=False, options=[("--hidden", "challenge authors: score on hidden/ instead of data/"),
-                             ("--reference", "challenge authors: score reference/ instead of the student files")])
+    """task check [method|artifact] <challenge>: the method, the artifact or both (first the method)."""
+    parser = argparse.ArgumentParser(
+        description="Check ./<challenge>/ locally on its data in the container raid-<challenge>, built like our "
+                    "backend's with the challenge's setup.sh. `task check method <challenge>` runs your method, "
+                    "`task check artifact <challenge>` scores your artifact, `task check <challenge>` does both.",
+        usage="%(prog)s [-h] [options] [method|artifact] challenge")
+    parser.add_argument("names", nargs="+", metavar="[method|artifact] challenge", help=argparse.SUPPRESS)
+
+    def action(raid, a):
+        names = a.names
+        if len(names) == 2 and names[1] in ("method", "artifact") and names[0] not in ("method", "artifact"):
+            names = names[::-1]  # task check dawn method
+        if len(names) > 2 or len(names) == 2 and names[0] not in ("method", "artifact"):
+            parser.error("expected [method|artifact] <challenge>")
+        part, challenge = (None, names[0]) if len(names) == 1 else names
+        raid.check(challenge, part, a.hidden, a.reference, container=not a.no_container, rebuild=a.rebuild)
+
+    _run(None, action, key=False, target=None, parser=parser, options=[
+        ("--rebuild", "build the container raid-<challenge> again even if setup.sh did not change"),
+        ("--no-container", "check on this machine instead of in the container (needs the challenge's packages)"),
+        ("--hidden", "challenge authors: score on hidden/ instead of the data"),
+        ("--reference", "challenge authors: score reference/ instead of the student files")])
 
 
 def submit():
-    _run("Submit ./<challenge>/ for scoring (without data/, scoring/ and .raidignore patterns).",
+    _run("Submit ./<challenge>/ for scoring (without its data, scoring/ and .raidignore patterns).",
          lambda raid, a: raid.submit(a.challenge))
 
 
@@ -47,7 +67,7 @@ def log():
 
 def push():
     _run("Admins: upload a challenge directory (without reference/) as its new active version. "
-         "challenge.yaml supplies the name, the version and the metadata of a new challenge.",
+         "challenge.yaml supplies the name, the version and the metadata.",
          lambda raid, a: raid.push(a.directory), target=("directory", "challenge directory, e.g. self-assessment"))
 
 

mlsec_raid-0.4.0/PKG-INFO

@@ -1,29 +0,0 @@
-Metadata-Version: 2.3
-Name: mlsec-raid
-Version: 0.4.0
-Summary: Client for the RAID competition system
-Requires-Python: >=3.9
-Description-Content-Type: text/markdown
-
-# raid
-
-Python client for the RAID competition system.
-
-```bash
-pip install mlsec-raid                                    # or: uv tool install mlsec-raid
-export RAID_KEY=<your-api-key>                            # or pass --api-key <your-api-key>
-
-task-fetch <challenge>    # downloads the challenge into ./<challenge>/
-task-check <challenge>    # scores it locally with scoring/score.py on data/
-task-submit <challenge>   # submits ./<challenge>/ without data/, scoring/ and .raidignore patterns
-task-show                 # lists your jobs with their state (pending, running, success, failed) and score
-task-log <job>            # prints the log of a finished job, including the errors of your solution
-```
-
-Every command also works as `task <command>`, e.g. `task submit <challenge>`.
-
-The same is available in Python: `Raid(api_key).fetch(...)`, `.check(...)`, `.submit(...)`, `.push(...)`, `.wait(id)`, `.status()`, `.log(id)`, `.howto()`.
-
-Challenge authors: `task-check <challenge> --hidden --reference` scores `reference/` on `hidden/`. Admins upload a challenge directory with `task-push <directory>` and their own API key (created in the admin console): the directory's `challenge.yaml` sets the name and version (and the wave, baseline and badge of a new challenge), and the upload becomes the active version.
-
-Development: `uv sync` sets up the environment, `uv build` builds the package. Set `RAID_URL` to use another server. Release: bump `version` in `pyproject.toml`, then `uv build && uv publish` (PyPI token in `UV_PUBLISH_TOKEN`).

mlsec_raid-0.4.0/README.md

@@ -1,22 +0,0 @@
-# raid
-
-Python client for the RAID competition system.
-
-```bash
-pip install mlsec-raid                                    # or: uv tool install mlsec-raid
-export RAID_KEY=<your-api-key>                            # or pass --api-key <your-api-key>
-
-task-fetch <challenge>    # downloads the challenge into ./<challenge>/
-task-check <challenge>    # scores it locally with scoring/score.py on data/
-task-submit <challenge>   # submits ./<challenge>/ without data/, scoring/ and .raidignore patterns
-task-show                 # lists your jobs with their state (pending, running, success, failed) and score
-task-log <job>            # prints the log of a finished job, including the errors of your solution
-```
-
-Every command also works as `task <command>`, e.g. `task submit <challenge>`.
-
-The same is available in Python: `Raid(api_key).fetch(...)`, `.check(...)`, `.submit(...)`, `.push(...)`, `.wait(id)`, `.status()`, `.log(id)`, `.howto()`.
-
-Challenge authors: `task-check <challenge> --hidden --reference` scores `reference/` on `hidden/`. Admins upload a challenge directory with `task-push <directory>` and their own API key (created in the admin console): the directory's `challenge.yaml` sets the name and version (and the wave, baseline and badge of a new challenge), and the upload becomes the active version.
-
-Development: `uv sync` sets up the environment, `uv build` builds the package. Set `RAID_URL` to use another server. Release: bump `version` in `pyproject.toml`, then `uv build && uv publish` (PyPI token in `UV_PUBLISH_TOKEN`).

mlsec_raid-0.4.0/src/raid/__init__.py

@@ -1,256 +0,0 @@
-"""Client for the RAID competition system: fetch challenges, check and submit solutions."""
-import fnmatch
-import importlib.util
-import json
-import os
-import posixpath
-import re
-import shutil
-import sys
-import tarfile
-import tempfile
-import time
-import urllib.error
-import urllib.request
-from pathlib import Path
-
-__all__ = ["Raid", "RaidError"]
-__version__ = "0.4.0"
-
-DEFAULT_URL = "https://raid.mlsec.tu-berlin.de"
-PARTS = ("source", "data", "scoring")  # source/ is unpacked into the challenge directory itself
-OWN = {"data", "scoring", "hidden", "reference", "challenge.yaml"}  # parts of the challenge directory that are never submitted
-SKIP = {"__pycache__", ".git", ".ipynb_checkpoints", ".venv", "venv", ".DS_Store"}
-NAME_RE = re.compile(r"^[a-z0-9][a-z0-9-]{0,62}$")
-
-
-class RaidError(Exception):
-    pass
-
-
-class Raid:
-    """raid = Raid(api_key); print(raid.howto())"""
-
-    def __init__(self, api_key=None, url=None, path="."):
-        self.api_key = (api_key or os.environ.get("RAID_KEY") or "").strip()
-        self.url = (url or os.environ.get("RAID_URL") or DEFAULT_URL).rstrip("/")
-        self.path = Path(path)
-
-    def howto(self):
-        """Explain the workflow."""
-        return self._json("/api/howto")["text"]
-
-    def fetch(self, challenge, force=False):
-        """Download a challenge into ./<challenge>/: its files, data/ and scoring/.
-
-        An existing directory is left untouched unless force=True, which overwrites the challenge files.
-        """
-        target = self.path / _check(challenge)
-        if target.exists() and not force:
-            print(f"{target}/ already exists. Fetch with force=True (task-fetch --force) to download it again "
-                  "(this overwrites the challenge files).")
-            return target
-        info = self._json(f"/api/challenges/{challenge}")
-        with tempfile.TemporaryDirectory() as tmp:
-            for part in PARTS:
-                with self._request("GET", f"/api/challenges/{challenge}/{part}.tar.gz", timeout=300) as r, \
-                        open(Path(tmp) / f"{part}.tar.gz", "wb") as f:
-                    shutil.copyfileobj(r, f, 1 << 20)
-            target.mkdir(parents=True, exist_ok=True)
-            for part in PARTS:
-                _extract(Path(tmp) / f"{part}.tar.gz", target, part)
-        print(f"Fetched {challenge} (version {info['version']}) into {target}/. Start with {target / 'README.txt'}.")
-        return target
-
-    def check(self, challenge, hidden=False, reference=False):
-        """Score ./<challenge>/ locally: scoring/score.py's score() on data/. Returns the score.
-
-        For challenge authors: hidden=True scores on hidden/, reference=True scores the files
-        in reference/ in place of the student files.
-        """
-        target = (self.path / _check(challenge)).resolve()
-        script = target / "scoring" / "score.py"
-        dataset = target / ("hidden" if hidden else "data")
-        solution = target / "reference" if reference else target
-        for path in (script, dataset, solution):
-            if not path.exists():
-                raise RaidError(f"{path} not found." + (" Fetch the challenge first." if path == script else ""))
-        paths = [str(script.parent), str(solution)]  # score.py imports the solution, e.g. `from solution import md5`
-        sys.path[:0] = paths
-        os.environ["RAID_SOLUTION"] = str(solution)  # where score.py finds files that are not imported, e.g. solution.c
-        try:
-            spec = importlib.util.spec_from_file_location("score", script)
-            module = importlib.util.module_from_spec(spec)
-            spec.loader.exec_module(module)
-            value = module.score(dataset)
-        finally:
-            del os.environ["RAID_SOLUTION"]
-            for p in paths:
-                sys.path.remove(p)
-            for name, mod in list(sys.modules.items()):  # a later check sees the current solution
-                if target in Path(getattr(mod, "__file__", None) or "/").resolve().parents:
-                    del sys.modules[name]
-        print(f"Score: {value}")
-        return value
-
-    def submit(self, challenge):
-        """Pack ./<challenge>/ (see _pack) and submit it for scoring. Returns the submission id."""
-        target = self.path / _check(challenge)
-        if not target.is_dir():
-            raise RaidError(f"{target}/ not found. Fetch the challenge first.")
-        required = self._json(f"/api/challenges/{challenge}")["required_files"]
-        missing = [f for f in required if not (target / f).is_file()]
-        if missing:
-            raise RaidError(f"Missing required files in {target}/: " + ", ".join(missing))
-        with tempfile.TemporaryDirectory() as tmp:
-            archive = Path(tmp) / "submission.tar.gz"
-            _pack(target, archive)
-            size = archive.stat().st_size
-            with open(archive, "rb") as f, self._request(
-                    "POST", f"/api/submit/{challenge}", data=f, timeout=900,
-                    headers={"Content-Type": "application/gzip", "Content-Length": str(size)}) as r:
-                result = json.load(r)
-        print("Job submitted!\nRun task-show to see the status of your jobs.")
-        return result["id"]
-
-    def push(self, directory):
-        """Admins: upload a challenge directory (everything but reference/) as a new, active version.
-
-        Its challenge.yaml supplies the name, the version and, for a new challenge, the metadata (title, wave,
-        baseline, badge_score, badge_points; later changes happen in the admin console). Returns the server's summary."""
-        target = Path(directory)
-        missing = [p for p in ("challenge.yaml", "data", "scoring", "hidden") if not (target / p).exists()]
-        if missing:
-            raise RaidError(f"Missing in {target}/: " + ", ".join(missing))
-        with tempfile.TemporaryDirectory() as tmp:
-            archive = Path(tmp) / "challenge.tar.gz"
-            _pack(target, archive, own={"reference"})
-            size = archive.stat().st_size
-            with open(archive, "rb") as f, self._request(
-                    "POST", "/api/admin/push", data=f, timeout=3600,
-                    headers={"Content-Type": "application/gzip", "Content-Length": str(size)}) as r:
-                result = json.load(r)
-        wave = f"wave {result['wave']}" if result["wave"] else "the self-assessment"
-        print(f"Pushed {result['name']} version {result['version']} ({size / 1e6:.2f} MB) to {wave}"
-              + (", now active." if result["active"] else "."))
-        return result
-
-    def wait(self, submission_id, poll=5):
-        """Wait until a submission is scored, print its log and return its details."""
-        print("Waiting for the scoring (Ctrl-C stops waiting, the scoring continues) ...", flush=True)
-        while True:
-            sub = self._json(f"/api/submissions/{int(submission_id)}")
-            if sub["status"] in ("success", "failed"):
-                break
-            time.sleep(poll)
-        print(sub["log"] or "No log output.")
-        print(f"Submission {sub['id']} (version {sub['version']}): {sub['status']}"
-              + ("" if sub["score"] is None else f", score {sub['score']:.4f}"))
-        return sub
-
-    def status(self, challenge=None):
-        """Print your jobs (submissions, newest first) and return them as a list of dicts."""
-        query = f"?challenge={_check(challenge)}" if challenge else ""
-        subs = self._json(f"/api/submissions{query}")["submissions"]
-        print(f"{'JOB':>6}  {'CHALLENGE':<20} {'VERSION':<8} {'STATE':<8} {'SCORE':>10}  SUBMITTED")
-        for s in subs:
-            score = "" if s["score"] is None else f"{s['score']:.4f}"
-            print(f"{s['id']:>6}  {s['challenge']:<20} {s['version']:<8} {s['status']:<8} {score:>10}  {s['submitted_at'][:19]}")
-        if not subs:
-            print("No jobs yet.")
-        else:
-            print("Run task-log <job> to see the log of a finished job.")
-        return subs
-
-    def log(self, submission_id):
-        """Return the scoring log of one of your finished submissions: the output and errors of the scoring run."""
-        if not str(submission_id).isdigit():
-            raise RaidError(f"Invalid job id {submission_id!r}. task-show lists your jobs.")
-        sub = self._json(f"/api/submissions/{submission_id}")
-        if sub["status"] in ("pending", "running"):
-            raise RaidError(f"Job {sub['id']} is {sub['status']}. Its log is available once it has finished.")
-        return sub["log"] or "No log output."
-
-    def _request(self, method, endpoint, data=None, headers=None, timeout=60):
-        if not self.api_key:
-            raise RaidError("No API key. Pass --api-key or set RAID_KEY.")
-        req = urllib.request.Request(self.url + endpoint, data=data, method=method, headers={
-            "Authorization": f"Bearer {self.api_key}", "User-Agent": f"raid/{__version__}", **(headers or {})})
-        try:
-            return urllib.request.urlopen(req, timeout=timeout)
-        except urllib.error.HTTPError as e:
-            try:
-                message = json.load(e).get("error")
-            except (ValueError, AttributeError):
-                message = None
-            raise RaidError(message or f"{e.code} {e.reason}") from None
-        except urllib.error.URLError as e:
-            raise RaidError(f"Cannot reach {self.url}: {e.reason}") from None
-
-    def _json(self, endpoint):
-        with self._request("GET", endpoint) as r:
-            return json.load(r)
-
-
-def _check(challenge):
-    if not isinstance(challenge, str) or not NAME_RE.match(challenge):
-        raise RaidError(f"Invalid challenge name {challenge!r}.")
-    return challenge
-
-
-def _extract(archive, target, part):
-    """Extract only regular files and directories below part/, never outside target. source/ maps to target."""
-    root = target.resolve()
-    with tarfile.open(archive, "r:gz") as tar:
-        for member in tar:
-            name = posixpath.normpath(member.name)
-            if name != part and not name.startswith(part + "/"):
-                raise RaidError(f"Unexpected entry {member.name!r} in the {part} archive.")
-            if part == "source":
-                name = name[len("source/"):] or "."
-            dest = (root / name).resolve()
-            if dest != root and root not in dest.parents:
-                raise RaidError(f"Unsafe entry {member.name!r} in the {part} archive.")
-            if member.isdir():
-                dest.mkdir(parents=True, exist_ok=True)
-            elif member.isfile():
-                dest.parent.mkdir(parents=True, exist_ok=True)
-                with tar.extractfile(member) as src, open(dest, "wb") as out:
-                    shutil.copyfileobj(src, out, 1 << 20)
-                if member.mode & 0o111:
-                    dest.chmod(0o755)
-
-
-def _normalize(info):
-    info.uid = info.gid = 0
-    info.uname = info.gname = ""
-    info.mode = 0o755 if info.isdir() or info.mode & 0o111 else 0o644
-    return info
-
-
-def _ignored(target):
-    """Patterns of <challenge>/.raidignore (tar exclude patterns, matched against any path suffix)."""
-    path = target / ".raidignore"
-    lines = path.read_text().splitlines() if path.is_file() else []
-    patterns = [line.strip().rstrip("/") for line in lines if line.strip() and not line.startswith("#")]
-    return lambda rel: any(fnmatch.fnmatchcase("/".join(rel.parts[i:]), p) for p in patterns for i in range(len(rel.parts)))
-
-
-def _pack(target, archive, own=OWN):
-    """tar.gz of the challenge directory without the top-level entries in own (by default data/, scoring/,
-    hidden/, reference/ and challenge.yaml) and the .raidignore patterns. Regular files only; caches,
-    VCS data and symlinks are left out."""
-    ignored = _ignored(target)
-    with tarfile.open(archive, "w:gz") as tar:
-        for dirpath, dirnames, filenames in os.walk(target):
-            rel = Path(dirpath).relative_to(target)
-            top = rel == Path(".")
-            dirnames[:] = sorted(d for d in dirnames if d not in SKIP and not (top and d in own) and not ignored(rel / d))
-            files = sorted(f for f in filenames if f not in SKIP and not (top and f in own) and not ignored(rel / f))
-            for name in dirnames + files:
-                path = Path(dirpath) / name
-                if path.is_symlink():
-                    print(f"Skipping symlink {path}")
-                    continue
-                if path.is_dir() or path.is_file():
-                    tar.add(path, arcname=(rel / name).as_posix(), recursive=False, filter=_normalize)