relbench 3.0.0__tar.gz → 3.0.2__tar.gz

{relbench-3.0.0 → relbench-3.0.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: relbench
-Version: 3.0.0
+Version: 3.0.2
 Summary: RelBench: Relational Deep Learning Benchmark
 Keywords: relational deep learning,graph neural networks,relational databases,benchmark,machine learning
 Author-email: RelBench Team <relbench@cs.stanford.edu>
@@ -60,16 +60,7 @@ Provides-Extra: schema
 
 ## News
 
-RelBench 3 loads every dataset family below from the Hugging Face Hub with one call —
-`relbench.load_dataset("<org>/<repo>/<name>")` — the core databases
-([`stanford-star/relbench-v1`](https://huggingface.co/datasets/stanford-star/relbench-v1),
-[`stanford-star/relbench-v2-extra`](https://huggingface.co/datasets/stanford-star/relbench-v2-extra)),
-CTU/ReDeLEx ([`stanford-star/redelex`](https://huggingface.co/datasets/stanford-star/redelex)),
-4DBInfer ([`stanford-star/dbinfer`](https://huggingface.co/datasets/stanford-star/dbinfer)) and
-TGB ([`stanford-star/tgb`](https://huggingface.co/datasets/stanford-star/tgb)); the `[ctu]` extra is
-gone. MIMIC-IV requires PhysioNet credentials and is not hosted. TGB and 4DBInfer tasks that
-use their own scoring protocols load as data only (see [`MIGRATION.md`](MIGRATION.md)).
-
+- **Oct 2026** — [Agentic leaderboard submissions](leaderboard/README.md#automatically-eg-from-an-ai-agent): point your AI agent at the [leaderboard README](leaderboard/README.md) and it can submit end to end. Entries can now carry an optional repro link for reproducibility.
 - **Aug 2026** — RelBench v3 released: datasets and tasks load straight from [Hugging Face](https://huggingface.co/stanford-star), a new [leaderboard](https://star-project.stanford.edu/relbench/leaderboard/) with automated submissions, and bug fixes ([migration guide](MIGRATION.md)).
 - **Jun 2026** — Datasets, models, and community hub migrated to [Hugging Face](https://huggingface.co/stanford-star).
 - **Mar 2026** — [RelBench v2 paper](https://arxiv.org/abs/2602.12606) accepted at the [ICLR 2026 DATA-FM workshop](https://data-fm-iclr2026.github.io/).
@@ -123,31 +114,15 @@ Open these directly in Google Colab — no setup required:
 ## Leaderboard
 
 The [**RelBench leaderboard**](https://star-project.stanford.edu/relbench/leaderboard/)
-ranks methods by their test-set performance, averaged over a fixed task set. There are
-three independent boards — **classification** (12 tasks), **regression** (9), and
-**recommendation** (10); the task lists are in `relbench.submit.LEADERBOARD_TASKS`.
-You can submit to any of them; each requires predictions for *all* of its tasks.
-
-To submit:
-
-1. **Write one prediction CSV per task**, named `<dataset>__<task>.csv`, into a directory:
-
-   ```python
-   relbench.submit.write_prediction_table(task, test_pred, "preds/rel-f1__driver-position.csv")
-   ```
-
-2. **Validate and package** the directory — this scores every CSV against the test tables,
-   prints a verdict per leaderboard, and writes clean submission zip file(s):
-
-   ```bash
-   python -m relbench.submit preds/
-   ```
-
-3. **[Open a submission issue](https://github.com/stanford-star/relbench/issues/new?template=submit.yml)**
-   on this repository: fill in the short form and upload the zip file(s) into it.
+ranks methods by their test-set performance, averaged over a fixed task set, with three
+independent boards: **classification**, **regression**, and **recommendation**.
+To submit, write one prediction CSV per task, validate and package them with
+`python -m relbench.submit`, and open a
+[submission issue](https://github.com/stanford-star/relbench/issues/new?template=submit.yml)
+— or let `python -m relbench.submit --submit` do the upload and open the issue for you.
 
-The submission is validated automatically and the report is posted on the issue; once a
-maintainer approves, your entry appears on the leaderboard.
+[**`leaderboard/README.md`**](leaderboard/README.md) has the full instructions, including
+the prediction CSV format for writing predictions with your own code.
 
 ## BYOD (Bring Your Own Data)
 
@@ -178,13 +153,12 @@ If you use RelBench, please cite the benchmark papers:
   year={2024}
 }
 
-@misc{relbenchv2,
+@inproceedings{relbenchv2,
   title={RelBench v2: A Large-Scale Benchmark and Repository for Relational Data},
   author={Gu, Justin and Ranjan, Rishabh and Kanatsoulis, Charilaos and Tang, Haiming and Jurkovic, Martin and Hudovernik, Valter and Znidar, Mark and Chaturvedi, Pranshu and Shroff, Parth and Li, Fengyu and Leskovec, Jure},
+  booktitle={3rd Workshop on Navigating and Addressing Data Problems for Foundation Models (DATA-FM) at ICLR 2026},
   year={2026},
-  eprint={2602.12606},
-  archivePrefix={arXiv},
-  primaryClass={cs.LG},
+  note={arXiv:2602.12606 [cs.LG]},
   url={https://arxiv.org/abs/2602.12606}
 }
 ```

{relbench-3.0.0 → relbench-3.0.2}/README.md

@@ -21,16 +21,7 @@
 
 ## News
 
-RelBench 3 loads every dataset family below from the Hugging Face Hub with one call —
-`relbench.load_dataset("<org>/<repo>/<name>")` — the core databases
-([`stanford-star/relbench-v1`](https://huggingface.co/datasets/stanford-star/relbench-v1),
-[`stanford-star/relbench-v2-extra`](https://huggingface.co/datasets/stanford-star/relbench-v2-extra)),
-CTU/ReDeLEx ([`stanford-star/redelex`](https://huggingface.co/datasets/stanford-star/redelex)),
-4DBInfer ([`stanford-star/dbinfer`](https://huggingface.co/datasets/stanford-star/dbinfer)) and
-TGB ([`stanford-star/tgb`](https://huggingface.co/datasets/stanford-star/tgb)); the `[ctu]` extra is
-gone. MIMIC-IV requires PhysioNet credentials and is not hosted. TGB and 4DBInfer tasks that
-use their own scoring protocols load as data only (see [`MIGRATION.md`](MIGRATION.md)).
-
+- **Oct 2026** — [Agentic leaderboard submissions](leaderboard/README.md#automatically-eg-from-an-ai-agent): point your AI agent at the [leaderboard README](leaderboard/README.md) and it can submit end to end. Entries can now carry an optional repro link for reproducibility.
 - **Aug 2026** — RelBench v3 released: datasets and tasks load straight from [Hugging Face](https://huggingface.co/stanford-star), a new [leaderboard](https://star-project.stanford.edu/relbench/leaderboard/) with automated submissions, and bug fixes ([migration guide](MIGRATION.md)).
 - **Jun 2026** — Datasets, models, and community hub migrated to [Hugging Face](https://huggingface.co/stanford-star).
 - **Mar 2026** — [RelBench v2 paper](https://arxiv.org/abs/2602.12606) accepted at the [ICLR 2026 DATA-FM workshop](https://data-fm-iclr2026.github.io/).
@@ -84,31 +75,15 @@ Open these directly in Google Colab — no setup required:
 ## Leaderboard
 
 The [**RelBench leaderboard**](https://star-project.stanford.edu/relbench/leaderboard/)
-ranks methods by their test-set performance, averaged over a fixed task set. There are
-three independent boards — **classification** (12 tasks), **regression** (9), and
-**recommendation** (10); the task lists are in `relbench.submit.LEADERBOARD_TASKS`.
-You can submit to any of them; each requires predictions for *all* of its tasks.
-
-To submit:
-
-1. **Write one prediction CSV per task**, named `<dataset>__<task>.csv`, into a directory:
-
-   ```python
-   relbench.submit.write_prediction_table(task, test_pred, "preds/rel-f1__driver-position.csv")
-   ```
-
-2. **Validate and package** the directory — this scores every CSV against the test tables,
-   prints a verdict per leaderboard, and writes clean submission zip file(s):
-
-   ```bash
-   python -m relbench.submit preds/
-   ```
-
-3. **[Open a submission issue](https://github.com/stanford-star/relbench/issues/new?template=submit.yml)**
-   on this repository: fill in the short form and upload the zip file(s) into it.
+ranks methods by their test-set performance, averaged over a fixed task set, with three
+independent boards: **classification**, **regression**, and **recommendation**.
+To submit, write one prediction CSV per task, validate and package them with
+`python -m relbench.submit`, and open a
+[submission issue](https://github.com/stanford-star/relbench/issues/new?template=submit.yml)
+— or let `python -m relbench.submit --submit` do the upload and open the issue for you.
 
-The submission is validated automatically and the report is posted on the issue; once a
-maintainer approves, your entry appears on the leaderboard.
+[**`leaderboard/README.md`**](leaderboard/README.md) has the full instructions, including
+the prediction CSV format for writing predictions with your own code.
 
 ## BYOD (Bring Your Own Data)
 
@@ -139,13 +114,12 @@ If you use RelBench, please cite the benchmark papers:
   year={2024}
 }
 
-@misc{relbenchv2,
+@inproceedings{relbenchv2,
   title={RelBench v2: A Large-Scale Benchmark and Repository for Relational Data},
   author={Gu, Justin and Ranjan, Rishabh and Kanatsoulis, Charilaos and Tang, Haiming and Jurkovic, Martin and Hudovernik, Valter and Znidar, Mark and Chaturvedi, Pranshu and Shroff, Parth and Li, Fengyu and Leskovec, Jure},
+  booktitle={3rd Workshop on Navigating and Addressing Data Problems for Foundation Models (DATA-FM) at ICLR 2026},
   year={2026},
-  eprint={2602.12606},
-  archivePrefix={arXiv},
-  primaryClass={cs.LG},
+  note={arXiv:2602.12606 [cs.LG]},
   url={https://arxiv.org/abs/2602.12606}
 }
 ```

{relbench-3.0.0 → relbench-3.0.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "flit_core.buildapi"
 
 [project]
 name = "relbench"
-version = "3.0.0"
+version = "3.0.2"
 description = "RelBench: Relational Deep Learning Benchmark"
 authors = [{name = "RelBench Team", email = "relbench@cs.stanford.edu"}]
 readme = "README.md"

{relbench-3.0.0 → relbench-3.0.2}/relbench/base/task_autocomplete.py

@@ -139,7 +139,7 @@ class AutoCompleteTask(EntityTask):
         db = full_db if split == "test" else full_db.upto(self.dataset.test_timestamp)
 
         if split == "train":
-            start = db.min_timestamp - pd.Timedelta(1, "ns")
+            start = db.min_timestamp - self.timedelta
             end = self.dataset.val_timestamp
         elif split == "val":
             start, end = self.dataset.val_timestamp, self.dataset.test_timestamp

{relbench-3.0.0 → relbench-3.0.2}/relbench/submit.py

@@ -5,8 +5,8 @@ library, ``pandas``/``numpy`` and the rest of RelBench's data layer (no torch /
 torch_geometric / pyg), so it imports cleanly in a minimal environment and can score a
 submission produced by any modelling stack.
 
-Concept -- the prediction table
-================================
+The prediction table
+====================
 The test split of a task is a *task table*. A *prediction table* is the same table with
 the target column replaced by predictions:
 
@@ -46,6 +46,7 @@ from __future__ import annotations
 import argparse
 import json
 import os
+import sys
 from concurrent.futures import ProcessPoolExecutor
 from pathlib import Path
 from typing import Any, Dict, List, Optional, Sequence, Tuple, Union
@@ -806,6 +807,15 @@ def _use_color() -> bool:
         return False
 
 
+def _glyph(unicode: str, ascii_fallback: str) -> str:
+    encoding = getattr(sys.stdout, "encoding", None) or "ascii"
+    try:
+        unicode.encode(encoding)
+    except (UnicodeEncodeError, LookupError):
+        return ascii_fallback
+    return unicode
+
+
 class _Style:
     r"""Tiny ANSI styler; every method degrades to the identity without a TTY."""
 
@@ -933,18 +943,18 @@ def _print_report(result: Dict[str, Any]) -> None:
         for task_name in group_tasks:
             entry = tasks[task_name]
             if entry["status"] == "ok":
-                mark = st.green("✓")
+                mark = st.green(_glyph("✓", "OK"))
                 value = _format_value(entry["metric_name"], entry["metric"])
                 print(f"  {mark} {task_name.ljust(name_w)}  {value.rjust(7)}")
             else:
-                mark = st.red("✗")
+                mark = st.red(_glyph("✗", "X"))
                 print(f"  {mark} {task_name.ljust(name_w)}  {st.red('failed')}")
                 for line in _wrap_text(entry["error"] or "unknown error", indent=6):
                     print(st.dim(line))
         if fam is not None and fam["missing"]:
             for task_name in fam["missing"]:
                 print(
-                    f"  {st.yellow('·')} {task_name.ljust(name_w)}  "
+                    f"  {st.yellow(_glyph('·', '-'))} {task_name.ljust(name_w)}  "
                     + st.dim("missing")
                 )
         if fam is not None and fam["aggregate"] is not None:
@@ -1045,9 +1055,11 @@ def _markdown_report(result: Dict[str, Any]) -> str:
 # Where validated submissions are turned into leaderboard entries: a submission is a GitHub
 # issue on the RelBench repository, created from the submission form with the prediction
 # tables attached. CI validates it and a maintainer approval publishes the entry.
+SUBMISSION_REPO = "stanford-star/relbench"
 SUBMISSION_ISSUE_URL = (
-    "https://github.com/stanford-star/relbench/issues/new?template=submit.yml"
+    f"https://github.com/{SUBMISSION_REPO}/issues/new?template=submit.yml"
 )
+SUBMISSION_HF_REPO = "relbench-submissions"
 
 
 def _zip_files(pred_dir: Path, filenames: Sequence[str]) -> bytes:
@@ -1068,7 +1080,8 @@ def _package(
     validated: Sequence[str],
     extra: Sequence[str],
     out: Path,
-) -> None:
+    show_next_step: bool = True,
+) -> List[Path]:
     r"""Write one submission zip per validated family and print how to submit them.
 
     A single all-in-one zip easily exceeds GitHub's issue-attachment size limit; the
@@ -1091,11 +1104,134 @@ def _package(
         fam_out.write_bytes(zip_bytes)
         outputs.append((fam_out, len(zip_bytes)))
     print("\r", end="", flush=True)
+    if not show_next_step:
+        return [p for p, _ in outputs]
     print(st.bold("Next step     "))
     print(f"  Upload these files at {st.underline_cyan(SUBMISSION_ISSUE_URL)}:")
     for fam_out, size in outputs:
         print(f"    {st.bold(str(fam_out))} " + st.italic(f"({size / 1e6:.1f} MB)"))
     print()
+    return [p for p, _ in outputs]
+
+
+def _github_token() -> str:
+    import subprocess
+
+    for var in ("GH_TOKEN", "GITHUB_TOKEN"):
+        if os.environ.get(var):
+            return os.environ[var]
+    try:
+        token = subprocess.run(
+            ["gh", "auth", "token"], capture_output=True, text=True, check=True
+        ).stdout.strip()
+    except (OSError, subprocess.CalledProcessError):
+        token = ""
+    if not token:
+        raise RuntimeError(
+            "no GitHub token: set GH_TOKEN or GITHUB_TOKEN, or run `gh auth login`"
+        )
+    return token
+
+
+def _upload_to_hf(zips: Sequence[Path]) -> List[str]:
+    import tempfile
+    from datetime import datetime, timezone
+
+    from huggingface_hub import CommitOperationAdd, HfApi
+
+    api = HfApi()
+    user = api.whoami()["name"]
+    repo_id = f"{user}/{SUBMISSION_HF_REPO}"
+    api.create_repo(repo_id, repo_type="dataset", private=False, exist_ok=True)
+    folder = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
+    ops = [
+        CommitOperationAdd(path_in_repo=f"{folder}/{z.name}", path_or_fileobj=str(z))
+        for z in zips
+    ]
+    commit = api.create_commit(
+        repo_id,
+        operations=ops,
+        repo_type="dataset",
+        commit_message=f"RelBench leaderboard submission {folder}",
+    )
+    return [
+        f"https://huggingface.co/datasets/{repo_id}/resolve/{commit.oid}/{op.path_in_repo}"
+        for op in ops
+    ]
+
+
+def _issue_body(
+    name: str,
+    url: str,
+    in_context: bool,
+    note: str,
+    links: Sequence[str],
+    repro_link: str = "",
+) -> str:
+    files = "\n".join(f"[{u.rsplit('/', 1)[-1]}]({u})" for u in links)
+    return (
+        f"### Name\n\n{name}\n\n"
+        f"### Note\n\n{note or '_No response_'}\n\n"
+        f"### In-context?\n\n{'Yes' if in_context else 'No'}\n\n"
+        f"### URL\n\n{url}\n\n"
+        f"### Repro link\n\n{repro_link or '_No response_'}\n\n"
+        f"### File(s)\n\n{files}\n"
+    )
+
+
+def _edit_issue_body(
+    entry: int, name: str = "", url: str = "", repro_link: str = "", note: str = ""
+) -> str:
+    return "".join(
+        f"### {h}\n\n{v or '_No response_'}\n\n"
+        for h, v in [
+            ("Entry", str(entry)),
+            ("Name", name),
+            ("URL", url),
+            ("Repro link", repro_link),
+            ("Note", note),
+        ]
+    )
+
+
+def _open_issue(
+    body: str, title: str = "Leaderboard submission", label: str = "submit"
+) -> str:
+    import urllib.request
+
+    req = urllib.request.Request(
+        f"https://api.github.com/repos/{SUBMISSION_REPO}/issues",
+        data=json.dumps({"title": title, "body": body, "labels": [label]}).encode(),
+        headers={
+            "Authorization": f"Bearer {_github_token()}",
+            "Accept": "application/vnd.github+json",
+            "User-Agent": "relbench-submit",
+        },
+        method="POST",
+    )
+    with urllib.request.urlopen(req, timeout=60) as r:
+        return json.load(r)["html_url"]
+
+
+def _submit(zips: Sequence[Path], args: argparse.Namespace) -> None:
+    st = _Style(_use_color())
+    print("Uploading to the Hugging Face Hub...", flush=True)
+    links = _upload_to_hf(zips)
+    for u in links:
+        print(f"  {u}")
+    body = _issue_body(
+        args.name,
+        args.url,
+        args.in_context == "yes",
+        args.note or "",
+        links,
+        args.repro_link or "",
+    )
+    issue = _open_issue(body)
+    print()
+    print(st.bold("Submitted     ") + st.underline_cyan(issue))
+    print("  Validation results will be posted on the issue.")
+    print()
 
 
 def main(argv: Optional[Sequence[str]] = None) -> int:
@@ -1113,7 +1249,9 @@ def main(argv: Optional[Sequence[str]] = None) -> int:
         ),
     )
     parser.add_argument(
-        "pred_dir", help="directory of <dataset>__<task>.csv prediction files"
+        "pred_dir",
+        nargs="?",
+        help="directory of <dataset>__<task>.csv prediction files",
     )
     parser.add_argument(
         "--num-workers",
@@ -1129,7 +1267,64 @@ def main(argv: Optional[Sequence[str]] = None) -> int:
             "<base>-<family>.zip (default base: <dir-name>.zip in the cwd)"
         ),
     )
+    parser.add_argument(
+        "--submit",
+        action="store_true",
+        help=(
+            "upload the zips to <hf-user>/relbench-submissions on the Hugging Face Hub "
+            "(needs a Hugging Face login) and open the submission issue on GitHub "
+            "(needs GH_TOKEN / GITHUB_TOKEN or `gh auth login`)"
+        ),
+    )
+    parser.add_argument("--name", help="with --submit: display name on the leaderboard")
+    parser.add_argument("--url", help="with --submit: project / code / paper link")
+    parser.add_argument(
+        "--in-context",
+        choices=["yes", "no"],
+        help="with --submit: yes if the method did NOT train on the target database",
+    )
+    parser.add_argument("--note", help="with --submit: optional note, shown on hover")
+    parser.add_argument(
+        "--repro-link",
+        help="with --submit: optional link to code / instructions to reproduce the results",
+    )
+    parser.add_argument(
+        "--edit",
+        type=int,
+        metavar="ISSUE",
+        help=(
+            "instead of submitting, open an issue that updates the --name, --url, "
+            "--repro-link, and/or --note of the leaderboard entry published from "
+            "submission issue ISSUE (needs a GitHub token, as for --submit)"
+        ),
+    )
     args = parser.parse_args(argv)
+    if args.edit is not None:
+        if not (args.name or args.url or args.repro_link or args.note):
+            parser.error("--edit requires --name, --url, --repro-link, or --note")
+        issue = _open_issue(
+            _edit_issue_body(
+                args.edit,
+                args.name or "",
+                args.url or "",
+                args.repro_link or "",
+                args.note or "",
+            ),
+            title="Leaderboard entry edit",
+            label="edit",
+        )
+        print(f"Opened {issue}")
+        return 0
+    if args.pred_dir is None:
+        parser.error("pred_dir is required unless --edit is given")
+    if args.submit:
+        missing = [
+            f"--{k.replace('_', '-')}"
+            for k in ("name", "url", "in_context")
+            if not getattr(args, k)
+        ]
+        if missing:
+            parser.error("--submit requires " + ", ".join(missing))
     if args.out is not None and not args.out.endswith(".zip"):
         args.out += ".zip"
 
@@ -1139,18 +1334,21 @@ def main(argv: Optional[Sequence[str]] = None) -> int:
         st = _Style(_use_color())
         print(
             st.bold_red("Cannot package")
-            + " — no leaderboard was validated; fix the prediction tables first."
+            + f" {_glyph('—', '-')} no leaderboard was validated; fix the prediction tables first."
         )
         print()
         return 1
     out = Path(args.out or f"{Path(args.pred_dir).resolve().name}.zip")
-    _package(
+    zips = _package(
         args.pred_dir,
         result["families"],
         result["validated"],
         result.get("extra_files") or [],
         out,
+        show_next_step=not args.submit,
     )
+    if args.submit:
+        _submit(zips, args)
     return 0