benchhub-client 0.1.0__tar.gz

benchhub_client-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Yakir Matari and BenchHub contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

benchhub_client-0.1.0/PKG-INFO

@@ -0,0 +1,128 @@
+Metadata-Version: 2.4
+Name: benchhub-client
+Version: 0.1.0
+Summary: Python client + typed contract for the BenchHub benchmarking platform
+Author: BenchHub
+License: MIT
+Project-URL: Homepage, https://runbenchhub.com
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy
+Requires-Dist: Pillow
+Requires-Dist: requests
+Dynamic: license-file
+
+# BenchHub
+
+BenchHub is an open-source benchmarking platform: pick a dataset, define
+metrics in Python, upload predictions, and see how your model ranks. Live
+at **https://runbenchhub.com**.
+
+Originally built as a private dTOF SPAD pipeline benchmarking tool, then
+generalized into a public, multi-tenant web app.
+
+## Features
+
+- **OAuth sign-in (GitHub)** — no passwords; one-click account creation.
+- **Datasets and leaderboards** are global — no project namespace.
+- **Per-row visibility** (`public` / `unlisted` / `private`) on datasets,
+  leaderboards, and metric/visualization library entries.
+- **HuggingFace import**: pull a structured HF dataset repo as a one-click
+  alternative to a ZIP upload (see `scripts/seed_nyu_v2_curated.py` for
+  an example workflow).
+- **User-defined metrics in Python** — bring your own scoring code; the
+  metric engine resolves dependencies and runs them per-sample or
+  aggregated. Sandbox-isolated when `BENCHHUB_SANDBOX_METRICS=1`.
+- **Asynchronous processing** with Celery (Redis broker).
+- **Per-user quotas**: 50 MB storage, 5 datasets, 50 submissions / 24h
+  by default. Free-tier safe to expose to the open internet.
+- **API tokens** for programmatic uploads (`/settings/api_tokens`).
+- **Account deletion** (GDPR right-to-be-forgotten) with cascading cleanup.
+- **Public landing page** at `/`, `/leaderboards` for browsing the catalog,
+  `/u/<id>` for public profile pages.
+
+## Prerequisites
+
+- Python 3.10+
+- Redis (broker + result backend, default port 6379)
+
+## Installation
+
+```bash
+git clone <repository-url>
+cd BenchHub
+python -m venv venv && source venv/bin/activate
+pip install -r requirements.txt
+```
+
+## Running
+
+Three terminals:
+
+```bash
+# 1. Redis
+redis-server
+
+# 2. Celery worker
+celery -A app.celery worker --loglevel=info
+
+# 3. Flask app
+python app.py
+```
+
+Then open `http://localhost:6060`.
+
+Data lives outside the repo at `~/.dtofbenchmarking/` (database + uploads).
+Override with `BENCHHUB_DATA_DIR=/some/path`.
+
+## Tests
+
+```bash
+pytest
+```
+
+429 tests, ~3-4 seconds. Coverage gate is configured in `pytest.ini`.
+
+## Dataset / submission ZIP convention
+
+Folders are auto-detected by prefix:
+
+| Prefix          | Type      | Files                                  |
+| --------------- | --------- | -------------------------------------- |
+| `metric_`       | metric    | `<sample>.txt` containing a float      |
+| `hist_` / `raw_histogram` / `hist` | histogram | `<sample>.npz` (`bins`, `counts`) |
+| `raw_`          | depth/map | `<sample>_<W>x<H>.npz`                 |
+| (anything else) | image / scalar / json / text | by file extension          |
+
+`git_info.json` (or `git.info`) at the ZIP root attaches commit metadata
+to the resulting dataset/submission row.
+
+## DLP-safe code uploads
+
+Some networks block `.py` uploads. The metric editor encodes user code
+as `BASE64:<...>` client-side; the server decodes. Standalone helpers:
+
+- `scripts/obfuscator.html` — portable browser tool
+- `scripts/obfuscator_gui.py` — Tkinter GUI
+
+## Deployment
+
+The production app is self-hosted on a home Ubuntu 24.04 box (RTX 5090,
+128 GB RAM, 8 TB) reachable at https://runbenchhub.com. gunicorn + celery
++ redis run directly under systemd; nginx + certbot terminate TLS; the
+domain is on Cloudflare in DNS-only mode (no proxy) with `ddclient`
+keeping the A record pointed at the home WAN IP.
+
+**Operational runbook: [`docs/SELFHOST_RUNBOOK.md`](docs/SELFHOST_RUNBOOK.md)**
+— code-push procedure, `.env` keys, log tailing, DDNS, TLS renewal,
+rollback, and the breakages we've already hit.
+
+Fly.io is deprecated: the app was destroyed after the cutover to the home
+box. The Fly artifacts (`fly.toml`, `Dockerfile`, `DEPLOY.md`, …) are
+archived under [`archive/fly/`](archive/fly/) for the case where a future
+Fly redeploy needs to be reconstructed.
+
+## License
+
+(Choose and add a license file — repository currently has no LICENSE.)

benchhub_client-0.1.0/README.md

@@ -0,0 +1,113 @@
+# BenchHub
+
+BenchHub is an open-source benchmarking platform: pick a dataset, define
+metrics in Python, upload predictions, and see how your model ranks. Live
+at **https://runbenchhub.com**.
+
+Originally built as a private dTOF SPAD pipeline benchmarking tool, then
+generalized into a public, multi-tenant web app.
+
+## Features
+
+- **OAuth sign-in (GitHub)** — no passwords; one-click account creation.
+- **Datasets and leaderboards** are global — no project namespace.
+- **Per-row visibility** (`public` / `unlisted` / `private`) on datasets,
+  leaderboards, and metric/visualization library entries.
+- **HuggingFace import**: pull a structured HF dataset repo as a one-click
+  alternative to a ZIP upload (see `scripts/seed_nyu_v2_curated.py` for
+  an example workflow).
+- **User-defined metrics in Python** — bring your own scoring code; the
+  metric engine resolves dependencies and runs them per-sample or
+  aggregated. Sandbox-isolated when `BENCHHUB_SANDBOX_METRICS=1`.
+- **Asynchronous processing** with Celery (Redis broker).
+- **Per-user quotas**: 50 MB storage, 5 datasets, 50 submissions / 24h
+  by default. Free-tier safe to expose to the open internet.
+- **API tokens** for programmatic uploads (`/settings/api_tokens`).
+- **Account deletion** (GDPR right-to-be-forgotten) with cascading cleanup.
+- **Public landing page** at `/`, `/leaderboards` for browsing the catalog,
+  `/u/<id>` for public profile pages.
+
+## Prerequisites
+
+- Python 3.10+
+- Redis (broker + result backend, default port 6379)
+
+## Installation
+
+```bash
+git clone <repository-url>
+cd BenchHub
+python -m venv venv && source venv/bin/activate
+pip install -r requirements.txt
+```
+
+## Running
+
+Three terminals:
+
+```bash
+# 1. Redis
+redis-server
+
+# 2. Celery worker
+celery -A app.celery worker --loglevel=info
+
+# 3. Flask app
+python app.py
+```
+
+Then open `http://localhost:6060`.
+
+Data lives outside the repo at `~/.dtofbenchmarking/` (database + uploads).
+Override with `BENCHHUB_DATA_DIR=/some/path`.
+
+## Tests
+
+```bash
+pytest
+```
+
+429 tests, ~3-4 seconds. Coverage gate is configured in `pytest.ini`.
+
+## Dataset / submission ZIP convention
+
+Folders are auto-detected by prefix:
+
+| Prefix          | Type      | Files                                  |
+| --------------- | --------- | -------------------------------------- |
+| `metric_`       | metric    | `<sample>.txt` containing a float      |
+| `hist_` / `raw_histogram` / `hist` | histogram | `<sample>.npz` (`bins`, `counts`) |
+| `raw_`          | depth/map | `<sample>_<W>x<H>.npz`                 |
+| (anything else) | image / scalar / json / text | by file extension          |
+
+`git_info.json` (or `git.info`) at the ZIP root attaches commit metadata
+to the resulting dataset/submission row.
+
+## DLP-safe code uploads
+
+Some networks block `.py` uploads. The metric editor encodes user code
+as `BASE64:<...>` client-side; the server decodes. Standalone helpers:
+
+- `scripts/obfuscator.html` — portable browser tool
+- `scripts/obfuscator_gui.py` — Tkinter GUI
+
+## Deployment
+
+The production app is self-hosted on a home Ubuntu 24.04 box (RTX 5090,
+128 GB RAM, 8 TB) reachable at https://runbenchhub.com. gunicorn + celery
++ redis run directly under systemd; nginx + certbot terminate TLS; the
+domain is on Cloudflare in DNS-only mode (no proxy) with `ddclient`
+keeping the A record pointed at the home WAN IP.
+
+**Operational runbook: [`docs/SELFHOST_RUNBOOK.md`](docs/SELFHOST_RUNBOOK.md)**
+— code-push procedure, `.env` keys, log tailing, DDNS, TLS renewal,
+rollback, and the breakages we've already hit.
+
+Fly.io is deprecated: the app was destroyed after the cutover to the home
+box. The Fly artifacts (`fly.toml`, `Dockerfile`, `DEPLOY.md`, …) are
+archived under [`archive/fly/`](archive/fly/) for the case where a future
+Fly redeploy needs to be reconstructed.
+
+## License
+
+(Choose and add a license file — repository currently has no LICENSE.)

benchhub_client-0.1.0/benchhub/__init__.py

@@ -0,0 +1,50 @@
+"""BenchHub — shared types + client API.
+
+This package is imported by the server (`app.py`, `metric_engine.py`) AND
+shipped to submitters as `benchhub-client`. Single source of truth for
+the strict-typed contract between predictions, GT, and metrics.
+"""
+
+from benchhub.types import (
+    DataType,
+    DTYPES,
+    Image,
+    Mask,
+    Depth,
+    Audio,
+    Text,
+    BBoxes,
+    Label,
+    LabelList,
+    Scalar,
+    Json,
+    get_type,
+)
+from benchhub.client import (
+    BenchHubAPIError,
+    BHDatasetCreator,
+    Client,
+    FlaskTestClientTransport,
+    SubmissionBuilder,
+)
+
+__all__ = [
+    "DataType",
+    "DTYPES",
+    "Image",
+    "Mask",
+    "Depth",
+    "Audio",
+    "Text",
+    "BBoxes",
+    "Label",
+    "LabelList",
+    "Scalar",
+    "Json",
+    "get_type",
+    "Client",
+    "SubmissionBuilder",
+    "BHDatasetCreator",
+    "FlaskTestClientTransport",
+    "BenchHubAPIError",
+]