codabench-client 0.1.0__tar.gz

codabench_client-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Abderrahmane Moujar [Scarface]
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

codabench_client-0.1.0/PKG-INFO

@@ -0,0 +1,258 @@
+Metadata-Version: 2.4
+Name: codabench-client
+Version: 0.1.0
+Summary: A small, readable Python client and CLI for the Codabench API.
+License: MIT
+Project-URL: Homepage, https://github.com/amoujar/codabench-client
+Project-URL: Issues, https://github.com/amoujar/codabench-client/issues
+Keywords: codabench,codalab,benchmark,competition,machine-learning
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests>=2.25
+Dynamic: license-file
+
+# codabench-client
+
+A small, readable Python client and CLI for the [Codabench](https://www.codabench.org/) API.
+
+Codabench is a great benchmark platform, but a lot of what you can do in its web UI —
+grabbing a starting kit, submitting a zip, pulling the scoring logs of a failed run —
+is tedious to click through and awkward to automate. This package does all of it from
+the terminal or from Python.
+
+```bash
+codabench show 17363                        # everything about a competition
+codabench files 16161                       # download the starting kit & public data
+codabench submit 17525 -z run.zip --wait    # submit and watch the score come back
+codabench outputs 17525                     # fetch the prediction & scoring output
+```
+
+---
+
+## Install
+
+```bash
+git clone https://github.com/<you>/api-codabench.git
+cd api-codabench
+pip install -e .
+```
+
+Python 3.9+ and `requests` are the only requirements. Without installing, everything
+also works as `python -m codabench ...` and `python examples/01_show_competition.py`.
+
+## Credentials
+
+Copy the template and fill it in:
+
+```bash
+cp .env.example .env
+```
+
+```ini
+CODABENCH_USERNAME=your_username
+CODABENCH_PASSWORD=your_password
+# CODABENCH_URL=https://dev.codabench.org/   # optional; default is www.codabench.org
+```
+
+`.env` is gitignored. Environment variables win over the file, so CI can just set
+`CODABENCH_USERNAME` / `CODABENCH_PASSWORD`. Reading a **public** competition needs no
+credentials at all.
+
+> **Accounts are per-instance.** A `www.codabench.org` login does not work on
+> `dev.codabench.org`, and vice versa.
+
+## Commands
+
+
+| Command                                      | What it does                                             |
+| ---------------------------------------------- | ---------------------------------------------------------- |
+| `codabench competitions [--search TEXT]`     | List competitions                                        |
+| `codabench show <id|url>`                    | Pages, phases, tasks, files, leaderboard                 |
+| `codabench files <id|url>`                   | List / download the "Files" tab                          |
+| `codabench submit <id|url> -z FILE`          | Upload a submission, optionally wait for the score       |
+| `codabench outputs <id|url>`                 | Download a submission's zip, prediction & scoring output |
+| `codabench rerun --submission ID --task KEY` | Re-run a submission on another task (robot accounts)     |
+| `codabench create -b bundle.zip`             | Create a competition from a bundle                       |
+
+
+| Command                                        | What it does                                             |
+| ------------------------------------------------ | ---------------------------------------------------------- |
+| `codabench competitions [--search TEXT]`       | List competitions                                        |
+| `codabench show &lt;id&#124;url&gt;`           | Pages, phases, tasks, files, leaderboard                 |
+| `codabench files &lt;id&#124;url&gt;`          | List / download the "Files" tab                          |
+| `codabench submit &lt;id&#124;url&gt; -z FILE` | Upload a submission, optionally wait for the score       |
+| `codabench outputs &lt;id&#124;url&gt;`        | Download a submission's zip, prediction & scoring output |
+| `codabench rerun --submission ID --task KEY`   | Re-run a submission on another task (robot accounts)     |
+| `codabench create -b bundle.zip`               | Create a competition from a bundle                       |
+
+Every command takes `--url`, `--username`, `--password`, `--env`, and `--help`.
+
+### Inspect a competition
+
+```bash
+codabench show 17363                # or the full https://www.codabench.org/competitions/17363/ URL
+codabench show 17363 --pages        # full text of every page
+codabench show 17363 --json         # raw API payload
+```
+
+Save the whole thing — description, one markdown file per page, and every file you are
+allowed to download:
+
+```bash
+codabench show 17363 --save-dir workdir/17363
+```
+
+```
+workdir/17363/
+├── description.md
+├── pages/
+│   ├── 00-overview.md
+│   ├── 01-data.md
+│   └── ...
+└── input/
+    └── Track 1 Starting Kit V5/
+```
+
+Add `--no-download` for the text only.
+
+### Download competition files
+
+```bash
+codabench files 16161 --list                 # see what exists first
+codabench files 16161                        # download into files/16161/
+codabench files 16161 --only starting_kit    # filter by type or name
+codabench files 16161 --no-extract           # keep the zips
+```
+
+### Submit and get scored
+
+```bash
+codabench submit 17525 -z submission.zip --wait
+codabench submit 17525 -z submission.zip --wait \
+    --results-json results.json --download-dir outputs
+```
+
+The phase is resolved automatically (the only one, or the currently open one); pass
+`--phase <id>` or `--phase-index <n>` when a competition has several open at once.
+Submitting **consumes your daily quota** — the command checks eligibility first and
+tells you if you are out.
+
+### Get a submission's output
+
+```bash
+codabench outputs 17525 --list              # your submissions on the phase
+codabench outputs 17525                     # newest submission
+codabench outputs 17525 --last 2            # the one before it
+codabench outputs --submission 856524       # a specific id
+codabench outputs 17525 --only scoring      # just the scoring step
+```
+
+Everything lands in `outputs/<submission id>/` with the zips extracted, plus a
+`details.json` holding the scores, logs and exit statuses — which is usually where the
+answer is when a submission fails.
+
+## Python API
+
+```python
+from codabench import CodabenchClient, collect_files, find_phase
+
+client = CodabenchClient()                     # reads .env / environment
+
+competition = client.competition(17363)        # id or URL
+phase = find_phase(competition)                # the open phase
+
+for entry in collect_files(competition):
+    path = client.download_dataset(entry.key, "input/", entry.filename)
+    print(entry.name, "->", path or "not authorized")
+
+submission = client.submit(phase["id"], "run.zip")
+client.wait_for_submission(submission["id"])
+print(client.results(submission["id"])["metrics"])
+```
+
+The client raises `CodabenchError` (with `AuthError` and `ApiError` subclasses) and
+never calls `sys.exit`, so it is safe to use inside a notebook or a larger program.
+
+## What you can actually download
+
+This trips everyone up, so it is worth stating plainly. Codabench decides per file, and
+this tool cannot widen what your account is allowed to see:
+
+
+| File                            | Who can download it                                                   |
+| --------------------------------- | ----------------------------------------------------------------------- |
+| Starting kit, public data       | Any approved participant                                              |
+| Ingestion / scoring program     | Participants**only if** the organizer set `make_programs_available`   |
+| Input data                      | Participants**only if** the organizer set `make_input_data_available` |
+| Reference data (the answer key) | Organizers only, always                                               |
+| Competition bundle              | Organizers only                                                       |
+
+Files your account may not fetch are skipped, not treated as an error. `codabench show`
+prints the organizer's two flags so you can see why something is missing.
+
+## How authentication works
+
+Codabench uses **two** mechanisms and you need both — the reason `CodabenchClient` holds
+a token *and* a session:
+
+
+| Endpoint                    | Auth                                                  |
+| ----------------------------- | ------------------------------------------------------- |
+| `/api/...`                  | Token from`POST /api/api-token-auth/`                 |
+| `/datasets/download/<key>/` | Django**session cookie** from `POST /accounts/login/` |
+
+The download route behind the "Files" tab is a plain Django view that ignores the API
+token, which is why a token-only client gets 403s on files a participant can plainly
+download in the browser. Both schemes read the same credentials, so callers never have
+to think about it.
+
+(There is also `GET /api/competitions/{id}/get_files/`, but it is organizer-only and
+returns 403 for participants — this package does not rely on it.)
+
+## Repository layout
+
+```
+codabench/              the library
+├── auth.py             credentials, token auth, session login
+├── client.py           CodabenchClient — every API call
+├── competitions.py     pure helpers over a competition payload
+├── downloads.py        saving & extracting artifacts
+├── text.py             HTML→text, slugs, sizes
+├── errors.py           CodabenchError / AuthError / ApiError
+└── cli.py              the `codabench` command
+
+examples/               the same flows as standalone scripts
+├── 01_show_competition.py
+├── 02_download_competition_files.py
+├── 03_submit_and_wait.py
+├── 04_download_submission_outputs.py
+├── 05_create_competition.py
+└── 06_rerun_submission.py
+
+assets/                 a sample submission zip and competition bundle
+```
+
+Run any example directly — no install needed:
+
+```bash
+python examples/01_show_competition.py 17363
+```
+
+## Notes
+
+* **Organizers**: `codabench create -b bundle.zip` uploads a competition bundle and polls
+  until it is unpacked. Try it on `https://dev.codabench.org/` first.
+* **Robot accounts**: `codabench rerun` re-runs an existing submission against another
+  task, reusing the uploaded data. It needs an account flagged `is_bot`, or the API
+  answers 403.
+
+## License
+
+MIT.

codabench_client-0.1.0/README.md

@@ -0,0 +1,238 @@
+# codabench-client
+
+A small, readable Python client and CLI for the [Codabench](https://www.codabench.org/) API.
+
+Codabench is a great benchmark platform, but a lot of what you can do in its web UI —
+grabbing a starting kit, submitting a zip, pulling the scoring logs of a failed run —
+is tedious to click through and awkward to automate. This package does all of it from
+the terminal or from Python.
+
+```bash
+codabench show 17363                        # everything about a competition
+codabench files 16161                       # download the starting kit & public data
+codabench submit 17525 -z run.zip --wait    # submit and watch the score come back
+codabench outputs 17525                     # fetch the prediction & scoring output
+```
+
+---
+
+## Install
+
+```bash
+git clone https://github.com/<you>/api-codabench.git
+cd api-codabench
+pip install -e .
+```
+
+Python 3.9+ and `requests` are the only requirements. Without installing, everything
+also works as `python -m codabench ...` and `python examples/01_show_competition.py`.
+
+## Credentials
+
+Copy the template and fill it in:
+
+```bash
+cp .env.example .env
+```
+
+```ini
+CODABENCH_USERNAME=your_username
+CODABENCH_PASSWORD=your_password
+# CODABENCH_URL=https://dev.codabench.org/   # optional; default is www.codabench.org
+```
+
+`.env` is gitignored. Environment variables win over the file, so CI can just set
+`CODABENCH_USERNAME` / `CODABENCH_PASSWORD`. Reading a **public** competition needs no
+credentials at all.
+
+> **Accounts are per-instance.** A `www.codabench.org` login does not work on
+> `dev.codabench.org`, and vice versa.
+
+## Commands
+
+
+| Command                                      | What it does                                             |
+| ---------------------------------------------- | ---------------------------------------------------------- |
+| `codabench competitions [--search TEXT]`     | List competitions                                        |
+| `codabench show <id|url>`                    | Pages, phases, tasks, files, leaderboard                 |
+| `codabench files <id|url>`                   | List / download the "Files" tab                          |
+| `codabench submit <id|url> -z FILE`          | Upload a submission, optionally wait for the score       |
+| `codabench outputs <id|url>`                 | Download a submission's zip, prediction & scoring output |
+| `codabench rerun --submission ID --task KEY` | Re-run a submission on another task (robot accounts)     |
+| `codabench create -b bundle.zip`             | Create a competition from a bundle                       |
+
+
+| Command                                        | What it does                                             |
+| ------------------------------------------------ | ---------------------------------------------------------- |
+| `codabench competitions [--search TEXT]`       | List competitions                                        |
+| `codabench show &lt;id&#124;url&gt;`           | Pages, phases, tasks, files, leaderboard                 |
+| `codabench files &lt;id&#124;url&gt;`          | List / download the "Files" tab                          |
+| `codabench submit &lt;id&#124;url&gt; -z FILE` | Upload a submission, optionally wait for the score       |
+| `codabench outputs &lt;id&#124;url&gt;`        | Download a submission's zip, prediction & scoring output |
+| `codabench rerun --submission ID --task KEY`   | Re-run a submission on another task (robot accounts)     |
+| `codabench create -b bundle.zip`               | Create a competition from a bundle                       |
+
+Every command takes `--url`, `--username`, `--password`, `--env`, and `--help`.
+
+### Inspect a competition
+
+```bash
+codabench show 17363                # or the full https://www.codabench.org/competitions/17363/ URL
+codabench show 17363 --pages        # full text of every page
+codabench show 17363 --json         # raw API payload
+```
+
+Save the whole thing — description, one markdown file per page, and every file you are
+allowed to download:
+
+```bash
+codabench show 17363 --save-dir workdir/17363
+```
+
+```
+workdir/17363/
+├── description.md
+├── pages/
+│   ├── 00-overview.md
+│   ├── 01-data.md
+│   └── ...
+└── input/
+    └── Track 1 Starting Kit V5/
+```
+
+Add `--no-download` for the text only.
+
+### Download competition files
+
+```bash
+codabench files 16161 --list                 # see what exists first
+codabench files 16161                        # download into files/16161/
+codabench files 16161 --only starting_kit    # filter by type or name
+codabench files 16161 --no-extract           # keep the zips
+```
+
+### Submit and get scored
+
+```bash
+codabench submit 17525 -z submission.zip --wait
+codabench submit 17525 -z submission.zip --wait \
+    --results-json results.json --download-dir outputs
+```
+
+The phase is resolved automatically (the only one, or the currently open one); pass
+`--phase <id>` or `--phase-index <n>` when a competition has several open at once.
+Submitting **consumes your daily quota** — the command checks eligibility first and
+tells you if you are out.
+
+### Get a submission's output
+
+```bash
+codabench outputs 17525 --list              # your submissions on the phase
+codabench outputs 17525                     # newest submission
+codabench outputs 17525 --last 2            # the one before it
+codabench outputs --submission 856524       # a specific id
+codabench outputs 17525 --only scoring      # just the scoring step
+```
+
+Everything lands in `outputs/<submission id>/` with the zips extracted, plus a
+`details.json` holding the scores, logs and exit statuses — which is usually where the
+answer is when a submission fails.
+
+## Python API
+
+```python
+from codabench import CodabenchClient, collect_files, find_phase
+
+client = CodabenchClient()                     # reads .env / environment
+
+competition = client.competition(17363)        # id or URL
+phase = find_phase(competition)                # the open phase
+
+for entry in collect_files(competition):
+    path = client.download_dataset(entry.key, "input/", entry.filename)
+    print(entry.name, "->", path or "not authorized")
+
+submission = client.submit(phase["id"], "run.zip")
+client.wait_for_submission(submission["id"])
+print(client.results(submission["id"])["metrics"])
+```
+
+The client raises `CodabenchError` (with `AuthError` and `ApiError` subclasses) and
+never calls `sys.exit`, so it is safe to use inside a notebook or a larger program.
+
+## What you can actually download
+
+This trips everyone up, so it is worth stating plainly. Codabench decides per file, and
+this tool cannot widen what your account is allowed to see:
+
+
+| File                            | Who can download it                                                   |
+| --------------------------------- | ----------------------------------------------------------------------- |
+| Starting kit, public data       | Any approved participant                                              |
+| Ingestion / scoring program     | Participants**only if** the organizer set `make_programs_available`   |
+| Input data                      | Participants**only if** the organizer set `make_input_data_available` |
+| Reference data (the answer key) | Organizers only, always                                               |
+| Competition bundle              | Organizers only                                                       |
+
+Files your account may not fetch are skipped, not treated as an error. `codabench show`
+prints the organizer's two flags so you can see why something is missing.
+
+## How authentication works
+
+Codabench uses **two** mechanisms and you need both — the reason `CodabenchClient` holds
+a token *and* a session:
+
+
+| Endpoint                    | Auth                                                  |
+| ----------------------------- | ------------------------------------------------------- |
+| `/api/...`                  | Token from`POST /api/api-token-auth/`                 |
+| `/datasets/download/<key>/` | Django**session cookie** from `POST /accounts/login/` |
+
+The download route behind the "Files" tab is a plain Django view that ignores the API
+token, which is why a token-only client gets 403s on files a participant can plainly
+download in the browser. Both schemes read the same credentials, so callers never have
+to think about it.
+
+(There is also `GET /api/competitions/{id}/get_files/`, but it is organizer-only and
+returns 403 for participants — this package does not rely on it.)
+
+## Repository layout
+
+```
+codabench/              the library
+├── auth.py             credentials, token auth, session login
+├── client.py           CodabenchClient — every API call
+├── competitions.py     pure helpers over a competition payload
+├── downloads.py        saving & extracting artifacts
+├── text.py             HTML→text, slugs, sizes
+├── errors.py           CodabenchError / AuthError / ApiError
+└── cli.py              the `codabench` command
+
+examples/               the same flows as standalone scripts
+├── 01_show_competition.py
+├── 02_download_competition_files.py
+├── 03_submit_and_wait.py
+├── 04_download_submission_outputs.py
+├── 05_create_competition.py
+└── 06_rerun_submission.py
+
+assets/                 a sample submission zip and competition bundle
+```
+
+Run any example directly — no install needed:
+
+```bash
+python examples/01_show_competition.py 17363
+```
+
+## Notes
+
+* **Organizers**: `codabench create -b bundle.zip` uploads a competition bundle and polls
+  until it is unpacked. Try it on `https://dev.codabench.org/` first.
+* **Robot accounts**: `codabench rerun` re-runs an existing submission against another
+  task, reusing the uploaded data. It needs an account flagged `is_bot`, or the API
+  answers 403.
+
+## License
+
+MIT.

codabench_client-0.1.0/codabench/__init__.py

@@ -0,0 +1,56 @@
+"""A small, readable Python client for the `Codabench <https://www.codabench.org/>`_ API.
+
+Typical use::
+
+    from codabench import CodabenchClient
+
+    client = CodabenchClient()                     # reads .env / environment
+    competition = client.competition(17363)        # URL or id both work
+    for f in collect_files(competition):
+        client.download_dataset(f.key, "input/", f.filename)
+
+Every command in the ``codabench`` CLI is a thin wrapper over these methods —
+see ``examples/`` for the same flows written as standalone scripts.
+"""
+
+from __future__ import annotations
+
+from .auth import Credentials, DEFAULT_URL, load_dotenv
+from .client import ARTIFACTS, CodabenchClient, TERMINAL_STATES, default_env_path
+from .competitions import (
+    CompetitionFile,
+    collect_files,
+    description_markdown,
+    find_phase,
+    pages,
+    parse_competition_id,
+    phases,
+    primary_metric,
+    save_description,
+)
+from .errors import ApiError, AuthError, CodabenchError
+
+__version__ = "0.1.0"
+
+__all__ = [
+    "ARTIFACTS",
+    "ApiError",
+    "AuthError",
+    "CodabenchClient",
+    "CodabenchError",
+    "CompetitionFile",
+    "Credentials",
+    "DEFAULT_URL",
+    "TERMINAL_STATES",
+    "__version__",
+    "collect_files",
+    "default_env_path",
+    "description_markdown",
+    "find_phase",
+    "load_dotenv",
+    "pages",
+    "parse_competition_id",
+    "phases",
+    "primary_metric",
+    "save_description",
+]

codabench_client-0.1.0/codabench/__main__.py

@@ -0,0 +1,8 @@
+"""Allow ``python -m codabench ...`` next to the installed ``codabench`` script."""
+
+import sys
+
+from .cli import main
+
+if __name__ == "__main__":
+    sys.exit(main())