benchloop-client 0.2.22__tar.gz → 0.2.23__tar.gz

benchloop_client-0.2.23/PKG-INFO

@@ -0,0 +1,294 @@
+Metadata-Version: 2.5
+Name: benchloop_client
+Version: 0.2.23
+Summary: Benchloop CLI — command line tool for benchmark analysis data (metrics, trajectory, reports, artifacts).
+Requires-Python: >=3.10
+Requires-Dist: httpx>=0.25.2
+Requires-Dist: tomli>=1.1.0; python_version < '3.13'
+Requires-Dist: typer>=0.9.4
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# benchloop-client (`bl`, `bmcli`)
+
+Command line tool for fetching and downloading benchmark analysis data
+(run metrics, trajectory, metadata, pre-analysis reports, task/job lists,
+and artifacts) by `group_id` / `job_id` / download `task_id`, and for
+submitting Behavior Reward Model (`brm`) analysis tasks.
+
+> PyPI package name is `benchloop_client`; the installed commands are `bl` and `bmcli`.
+
+## Install package
+
+```
+pip install benchloop_client
+
+bl --api-key $BENCH_LOOP_API_KEY overview get -j $JOB_ID
+bl --api-key $BENCH_LOOP_API_KEY metrics get -j $JOB_ID
+
+# Option 2: config ~/.benchloop/config.toml by cli, and run bl without --api-key option:
+bl config set --api-key $BENCHLOOP_API_KEY --host https://benchmark-analysis.alibaba-inc.com
+
+bl overview get -j $JOB_ID
+bl metrics get -j $JOB_ID
+
+```
+
+The CLI checks PyPI for a newer release in the background at most once every
+24 hours. Update notices include the current/latest versions and the upgrade
+command. Notices are written only to stderr, so JSON output remains pipeable.
+
+## Install and run from source
+
+```bash
+# with uv (recommended)
+uv sync
+uv run bl --help
+
+# or editable install with pip
+python -m venv .venv && source .venv/bin/activate
+pip install -e .
+bl --help
+```
+
+## Auth & config
+
+Authentication uses a personal api key sent as the `X-API-Key` header.
+
+Precedence: CLI flag > env var > `~/.benchloop/config.toml` > default.
+
+- Env: `BENCHLOOP_API_KEY` (or `BENCHMARK_API_KEY`), `BENCHLOOP_HOST`, `BENCHLOOP_OUTPUT_DIR`
+- Default host (prod): `https://benchmark-analysis.alibaba-inc.com`
+- Pre host: `https://pre-benchmark-analysis.alibaba-inc.com`
+
+```bash
+bl config set --api-key $BENCHLOOP_API_KEY --host https://benchmark-analysis.alibaba-inc.com
+```
+
+## Usage
+
+Two-segment `bl <resource> <action>` form. `get` prints a JSON array to
+stdout (pipeable to `jq`); `download` writes the standard file layout.
+Global options such as `--api-key`, `--host`, `--timeout`, `--retry`,
+`--concurrency`, `--quiet`, and `--verbose` must be placed before the resource
+name. Use `--concurrency 1` for a per-file byte progress bar, or a higher value
+for concurrent requests/downloads.
+
+```bash
+bl metrics get -j J123 | jq '.[0]'
+bl metrics download -g G1 --scope all -o ./out
+bl --concurrency 10 trajectory download -g G1,G2 -o ./out
+bl --concurrency 10 trajectory download --task-id TASK_ID -o ./out
+bl metadata download --task-id TASK_ID -o ./out
+bl report download -j J123 -o ./out
+bl report get -g G123
+bl report download -g G123 -o ./out
+bl jobs get -g G1 --all          # get supports --all (auto-paginate)
+bl jobs download -g G1           # download is always full
+bl artifacts download -g G1 -o ./out
+bl artifacts download --task-id TASK_ID -o ./out
+bl pull -g G1 --with metrics,trajectory,jobs,artifacts -o ./out
+bl brm run -g G1
+bl brm run -g G1 --steps 10,20 --sample-size 512 --scoring-limit 32
+bl brm get --task-id UID
+bl brm get --task-id UID --scope overall
+bl brm get --task-id UID --scope detail
+bl brm get --task-id UID --scope status
+bl brm status --task-id UID
+bl brm list -g G1
+```
+
+Group downloads always auto-paginate and include every Job in the Group. In
+particular, `trajectory`, detailed `metrics`, `report`, and `artifacts` enumerate
+the complete Job list instead of relying on the backend Group endpoints that
+return at most 1000 Jobs. Existing aggregate files keep their original paths:
+
+```text
+out/G1/trajectory.json
+out/G1/detail_metrics.json
+out/G1_report.md
+out/G1/J123_report.md
+out/G1/J123/<artifact-name>
+out/TASK_ID_trajectories.zip
+out/TASK_ID_metadata.zip
+out/TASK_ID/J123.tgz
+```
+
+`report download -g` writes both the Group summary report and each Job report.
+`pull -g` uses the same complete download behavior for every selected resource.
+Use `--skip-existing` to resume file-based downloads after an interruption.
+
+IDs accept repeated flags, comma-separated values, and `-` to read from stdin.
+`-g` and `-j` are mutually exclusive in a single command. For trajectory and
+artifact downloads, `--task-id` is a third mutually exclusive input mode.
+
+### Download trajectories by task ID
+
+The benchmark analysis Jobs page can turn the current filters into a trajectory
+download task. Copy the command shown by the page and run it locally:
+
+```bash
+bl trajectory download --task-id TASK_ID -o ./out
+```
+
+The CLI fetches the task's complete job ID list, downloads job trajectories with
+fixed concurrency, and prints count/ETA progress to stderr. It creates
+`./out/TASK_ID_trajectories.zip` with this layout:
+
+```text
+trajectories/<jobId>_trajectory.json
+metadata/<jobId>.json
+manifest.json
+```
+
+Metadata is fetched from `/openapi/jobs/{job_id}/meta` and archived alongside
+the trajectory. If trajectory download succeeds but metadata fails, the
+trajectory is still archived, the metadata error is recorded in
+`manifest.metadata_failures`, and the command exits with code `1`. If only some
+jobs fail, the successful files are still archived. If every job fails, no empty
+ZIP is created.
+
+### Download metadata by task ID
+
+Use `metadata download` when you only need per-job metadata for a download task:
+
+```bash
+bl metadata download --task-id TASK_ID -o ./out
+```
+
+The CLI creates `./out/TASK_ID_metadata.zip` with this layout:
+
+```text
+metadata/<jobId>.json
+manifest.json
+```
+
+Partial metadata failures are recorded in `manifest.failures`; successful
+metadata files are still archived and the command exits with code `1`.
+
+### Download artifacts
+
+Artifacts can be downloaded by Job, Group, or download task:
+
+```bash
+bl artifacts download -j J123 -o ./out
+bl artifacts download -g G1 -o ./out
+bl artifacts download --task-id TASK_ID -o ./out
+bl artifacts download -g G1 --include "*.tgz" --exclude "debug-*" -o ./out
+bl artifacts download -g G1 --list-only
+```
+
+For Job and Group mode, the CLI still lists artifacts first via
+`/openapi/jobs/{job_id}/artifacts`, but binary download is routed through the
+backend proxy endpoint `/openapi/jobs/{job_id}/artifacts/download` instead of
+directly using OSS presigned URLs. In task mode, every job from the task
+snapshot is downloaded as `./out/TASK_ID/<jobId>.tgz`.
+
+When an artifact is missing, the backend may return either HTTP 404 or an HTML
+error page. The CLI treats both as a missing artifact and writes a valid empty
+`.tgz` placeholder instead of saving the error page, so downstream batch
+extraction can continue.
+
+Artifact downloads show byte-level progress. On a TTY the progress is refreshed
+in place; in CI or redirected stderr it falls back to periodic `[progress]`
+lines. `--quiet` suppresses progress output.
+
+### Submit a Behavior Reward Model (`brm`) task
+
+`brm run` creates a Behavior analysis task for exactly one Group. The CLI first
+calls `GET /openapi/behavior/steps?group_id=...` to detect whether the Group
+has step sampling, then posts `POST /openapi/behavior/tasks`.
+
+```bash
+# No-step groups (all_jobs sampling): omit --steps
+bl brm run -g G1
+
+# Step groups: --steps is required (comma-separated or repeatable)
+bl brm run -g G1 --steps 10,20
+bl brm run -g G1 --steps 10 --steps 20 --sample-size 256 --scoring-limit 64
+
+# Pipe the task id
+bl brm run -g G1 --steps 10,20 | jq '.[0].uid'
+```
+
+Defaults: `--sample-size 512` (step mode only), `--scoring-limit 32` (always
+sent). In step mode, omitting `--steps` is a usage error and the available
+steps are listed on stderr. Passing `--steps` on a no-step Group is also a
+usage error.
+
+On success, stderr logs a short confirmation and stdout prints a JSON array
+(one object) that includes at least `uid`, plus `group_id` and
+`samplingMode` (`step` or `all_jobs`). Use `.[0].uid` (not necessarily
+`task_id`) for follow-up `brm status` / `brm get` calls.
+
+### Query a Behavior Reward Model (`brm`) task
+
+`brm get` queries task status and/or analysis by Behavior task uid (from
+`brm run`). Multiple `--task-id` values are supported (repeatable or
+comma-separated); stdout is always a JSON array, one element per id, matching
+`metrics get -j ...`.
+
+```bash
+# Default --scope all (status + overall + detail); samplingMode only here
+bl brm get --task-id UID
+bl brm get --task-id UID --scope all | jq '.[0] | {status, overall, detail, samplingMode}'
+
+# overall: analysis.summary + categoryScores
+bl brm get --task-id UID --scope overall | jq '.[0] | {summary, categoryScores}'
+
+# detail: analysis.behaviorPatterns
+bl brm get --task-id UID --scope detail | jq '.[0].behaviorPatterns'
+
+# status only; `brm status` is equivalent
+bl brm get --task-id UID --scope status
+bl brm status --task-id UID
+bl brm status --task-id UID1,UID2
+```
+
+`--scope` values:
+
+| scope | source | payload |
+| --- | --- | --- |
+| `overall` | `GET .../analysis` | `summary`, `categoryScores` |
+| `detail` | `GET .../analysis` | `behaviorPatterns` |
+| `status` | `GET .../tasks/{uid}` | task status / progress |
+| `all` (default) | both endpoints | nested `status`, `overall`, `detail`, plus top-level `samplingMode` |
+
+`brm status` always uses the status endpoint (same as `brm get --scope status`).
+
+### List Behavior Reward Model (`brm`) tasks by Group
+
+`brm list` calls `GET /openapi/behavior/tasks?group_id=...` and prints a JSON
+array (one element per `-g`), matching `jobs get -g ...`:
+
+```bash
+bl brm list -g G1
+bl brm list -g G1,G2
+bl brm list -g G1 | jq '.[0].tasks[].uid'
+```
+
+Each element looks like `{ "group_id", "total", "tasks": [ ... ] }`. Task
+objects keep the backend fields and also set `task_id` from `uid` when present.
+
+## Exit codes
+
+| code | meaning |
+| ---- | ------- |
+| 0 | all success |
+| 1 | partial failure (some ids failed) |
+| 2 | usage / parameter error |
+| 3 | auth failure (HTTP 401) |
+| 4 | network / timeout |
+
+stdout carries data only; logs and progress go to stderr. `--quiet` suppresses
+non-error logs, `--verbose` adds debug logs.
+
+
+## Release new package
+
+```
+uv build
+twine check dist/benchloop_client-*.whl
+twine upload dist/benchloop_client-*.whl --verbose
+```

benchloop_client-0.2.23/README.md

@@ -0,0 +1,282 @@
+# benchloop-client (`bl`, `bmcli`)
+
+Command line tool for fetching and downloading benchmark analysis data
+(run metrics, trajectory, metadata, pre-analysis reports, task/job lists,
+and artifacts) by `group_id` / `job_id` / download `task_id`, and for
+submitting Behavior Reward Model (`brm`) analysis tasks.
+
+> PyPI package name is `benchloop_client`; the installed commands are `bl` and `bmcli`.
+
+## Install package
+
+```
+pip install benchloop_client
+
+bl --api-key $BENCH_LOOP_API_KEY overview get -j $JOB_ID
+bl --api-key $BENCH_LOOP_API_KEY metrics get -j $JOB_ID
+
+# Option 2: config ~/.benchloop/config.toml by cli, and run bl without --api-key option:
+bl config set --api-key $BENCHLOOP_API_KEY --host https://benchmark-analysis.alibaba-inc.com
+
+bl overview get -j $JOB_ID
+bl metrics get -j $JOB_ID
+
+```
+
+The CLI checks PyPI for a newer release in the background at most once every
+24 hours. Update notices include the current/latest versions and the upgrade
+command. Notices are written only to stderr, so JSON output remains pipeable.
+
+## Install and run from source
+
+```bash
+# with uv (recommended)
+uv sync
+uv run bl --help
+
+# or editable install with pip
+python -m venv .venv && source .venv/bin/activate
+pip install -e .
+bl --help
+```
+
+## Auth & config
+
+Authentication uses a personal api key sent as the `X-API-Key` header.
+
+Precedence: CLI flag > env var > `~/.benchloop/config.toml` > default.
+
+- Env: `BENCHLOOP_API_KEY` (or `BENCHMARK_API_KEY`), `BENCHLOOP_HOST`, `BENCHLOOP_OUTPUT_DIR`
+- Default host (prod): `https://benchmark-analysis.alibaba-inc.com`
+- Pre host: `https://pre-benchmark-analysis.alibaba-inc.com`
+
+```bash
+bl config set --api-key $BENCHLOOP_API_KEY --host https://benchmark-analysis.alibaba-inc.com
+```
+
+## Usage
+
+Two-segment `bl <resource> <action>` form. `get` prints a JSON array to
+stdout (pipeable to `jq`); `download` writes the standard file layout.
+Global options such as `--api-key`, `--host`, `--timeout`, `--retry`,
+`--concurrency`, `--quiet`, and `--verbose` must be placed before the resource
+name. Use `--concurrency 1` for a per-file byte progress bar, or a higher value
+for concurrent requests/downloads.
+
+```bash
+bl metrics get -j J123 | jq '.[0]'
+bl metrics download -g G1 --scope all -o ./out
+bl --concurrency 10 trajectory download -g G1,G2 -o ./out
+bl --concurrency 10 trajectory download --task-id TASK_ID -o ./out
+bl metadata download --task-id TASK_ID -o ./out
+bl report download -j J123 -o ./out
+bl report get -g G123
+bl report download -g G123 -o ./out
+bl jobs get -g G1 --all          # get supports --all (auto-paginate)
+bl jobs download -g G1           # download is always full
+bl artifacts download -g G1 -o ./out
+bl artifacts download --task-id TASK_ID -o ./out
+bl pull -g G1 --with metrics,trajectory,jobs,artifacts -o ./out
+bl brm run -g G1
+bl brm run -g G1 --steps 10,20 --sample-size 512 --scoring-limit 32
+bl brm get --task-id UID
+bl brm get --task-id UID --scope overall
+bl brm get --task-id UID --scope detail
+bl brm get --task-id UID --scope status
+bl brm status --task-id UID
+bl brm list -g G1
+```
+
+Group downloads always auto-paginate and include every Job in the Group. In
+particular, `trajectory`, detailed `metrics`, `report`, and `artifacts` enumerate
+the complete Job list instead of relying on the backend Group endpoints that
+return at most 1000 Jobs. Existing aggregate files keep their original paths:
+
+```text
+out/G1/trajectory.json
+out/G1/detail_metrics.json
+out/G1_report.md
+out/G1/J123_report.md
+out/G1/J123/<artifact-name>
+out/TASK_ID_trajectories.zip
+out/TASK_ID_metadata.zip
+out/TASK_ID/J123.tgz
+```
+
+`report download -g` writes both the Group summary report and each Job report.
+`pull -g` uses the same complete download behavior for every selected resource.
+Use `--skip-existing` to resume file-based downloads after an interruption.
+
+IDs accept repeated flags, comma-separated values, and `-` to read from stdin.
+`-g` and `-j` are mutually exclusive in a single command. For trajectory and
+artifact downloads, `--task-id` is a third mutually exclusive input mode.
+
+### Download trajectories by task ID
+
+The benchmark analysis Jobs page can turn the current filters into a trajectory
+download task. Copy the command shown by the page and run it locally:
+
+```bash
+bl trajectory download --task-id TASK_ID -o ./out
+```
+
+The CLI fetches the task's complete job ID list, downloads job trajectories with
+fixed concurrency, and prints count/ETA progress to stderr. It creates
+`./out/TASK_ID_trajectories.zip` with this layout:
+
+```text
+trajectories/<jobId>_trajectory.json
+metadata/<jobId>.json
+manifest.json
+```
+
+Metadata is fetched from `/openapi/jobs/{job_id}/meta` and archived alongside
+the trajectory. If trajectory download succeeds but metadata fails, the
+trajectory is still archived, the metadata error is recorded in
+`manifest.metadata_failures`, and the command exits with code `1`. If only some
+jobs fail, the successful files are still archived. If every job fails, no empty
+ZIP is created.
+
+### Download metadata by task ID
+
+Use `metadata download` when you only need per-job metadata for a download task:
+
+```bash
+bl metadata download --task-id TASK_ID -o ./out
+```
+
+The CLI creates `./out/TASK_ID_metadata.zip` with this layout:
+
+```text
+metadata/<jobId>.json
+manifest.json
+```
+
+Partial metadata failures are recorded in `manifest.failures`; successful
+metadata files are still archived and the command exits with code `1`.
+
+### Download artifacts
+
+Artifacts can be downloaded by Job, Group, or download task:
+
+```bash
+bl artifacts download -j J123 -o ./out
+bl artifacts download -g G1 -o ./out
+bl artifacts download --task-id TASK_ID -o ./out
+bl artifacts download -g G1 --include "*.tgz" --exclude "debug-*" -o ./out
+bl artifacts download -g G1 --list-only
+```
+
+For Job and Group mode, the CLI still lists artifacts first via
+`/openapi/jobs/{job_id}/artifacts`, but binary download is routed through the
+backend proxy endpoint `/openapi/jobs/{job_id}/artifacts/download` instead of
+directly using OSS presigned URLs. In task mode, every job from the task
+snapshot is downloaded as `./out/TASK_ID/<jobId>.tgz`.
+
+When an artifact is missing, the backend may return either HTTP 404 or an HTML
+error page. The CLI treats both as a missing artifact and writes a valid empty
+`.tgz` placeholder instead of saving the error page, so downstream batch
+extraction can continue.
+
+Artifact downloads show byte-level progress. On a TTY the progress is refreshed
+in place; in CI or redirected stderr it falls back to periodic `[progress]`
+lines. `--quiet` suppresses progress output.
+
+### Submit a Behavior Reward Model (`brm`) task
+
+`brm run` creates a Behavior analysis task for exactly one Group. The CLI first
+calls `GET /openapi/behavior/steps?group_id=...` to detect whether the Group
+has step sampling, then posts `POST /openapi/behavior/tasks`.
+
+```bash
+# No-step groups (all_jobs sampling): omit --steps
+bl brm run -g G1
+
+# Step groups: --steps is required (comma-separated or repeatable)
+bl brm run -g G1 --steps 10,20
+bl brm run -g G1 --steps 10 --steps 20 --sample-size 256 --scoring-limit 64
+
+# Pipe the task id
+bl brm run -g G1 --steps 10,20 | jq '.[0].uid'
+```
+
+Defaults: `--sample-size 512` (step mode only), `--scoring-limit 32` (always
+sent). In step mode, omitting `--steps` is a usage error and the available
+steps are listed on stderr. Passing `--steps` on a no-step Group is also a
+usage error.
+
+On success, stderr logs a short confirmation and stdout prints a JSON array
+(one object) that includes at least `uid`, plus `group_id` and
+`samplingMode` (`step` or `all_jobs`). Use `.[0].uid` (not necessarily
+`task_id`) for follow-up `brm status` / `brm get` calls.
+
+### Query a Behavior Reward Model (`brm`) task
+
+`brm get` queries task status and/or analysis by Behavior task uid (from
+`brm run`). Multiple `--task-id` values are supported (repeatable or
+comma-separated); stdout is always a JSON array, one element per id, matching
+`metrics get -j ...`.
+
+```bash
+# Default --scope all (status + overall + detail); samplingMode only here
+bl brm get --task-id UID
+bl brm get --task-id UID --scope all | jq '.[0] | {status, overall, detail, samplingMode}'
+
+# overall: analysis.summary + categoryScores
+bl brm get --task-id UID --scope overall | jq '.[0] | {summary, categoryScores}'
+
+# detail: analysis.behaviorPatterns
+bl brm get --task-id UID --scope detail | jq '.[0].behaviorPatterns'
+
+# status only; `brm status` is equivalent
+bl brm get --task-id UID --scope status
+bl brm status --task-id UID
+bl brm status --task-id UID1,UID2
+```
+
+`--scope` values:
+
+| scope | source | payload |
+| --- | --- | --- |
+| `overall` | `GET .../analysis` | `summary`, `categoryScores` |
+| `detail` | `GET .../analysis` | `behaviorPatterns` |
+| `status` | `GET .../tasks/{uid}` | task status / progress |
+| `all` (default) | both endpoints | nested `status`, `overall`, `detail`, plus top-level `samplingMode` |
+
+`brm status` always uses the status endpoint (same as `brm get --scope status`).
+
+### List Behavior Reward Model (`brm`) tasks by Group
+
+`brm list` calls `GET /openapi/behavior/tasks?group_id=...` and prints a JSON
+array (one element per `-g`), matching `jobs get -g ...`:
+
+```bash
+bl brm list -g G1
+bl brm list -g G1,G2
+bl brm list -g G1 | jq '.[0].tasks[].uid'
+```
+
+Each element looks like `{ "group_id", "total", "tasks": [ ... ] }`. Task
+objects keep the backend fields and also set `task_id` from `uid` when present.
+
+## Exit codes
+
+| code | meaning |
+| ---- | ------- |
+| 0 | all success |
+| 1 | partial failure (some ids failed) |
+| 2 | usage / parameter error |
+| 3 | auth failure (HTTP 401) |
+| 4 | network / timeout |
+
+stdout carries data only; logs and progress go to stderr. `--quiet` suppresses
+non-error logs, `--verbose` adds debug logs.
+
+
+## Release new package
+
+```
+uv build
+twine check dist/benchloop_client-*.whl
+twine upload dist/benchloop_client-*.whl --verbose
+```

{benchloop_client-0.2.22 → benchloop_client-0.2.23}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "benchloop_client"
-version = "0.2.22"
+version = "0.2.23"
 description = "Benchloop CLI — command line tool for benchmark analysis data (metrics, trajectory, reports, artifacts)."
 readme = "README.md"
 requires-python = ">=3.10"

{benchloop_client-0.2.22 → benchloop_client-0.2.23}/src/aliyun/bmcli/cli.py

@@ -10,6 +10,7 @@ from aliyun.bmcli import __version__, output, version_check
 from aliyun.bmcli.commands import (
     ap_cmd as ap_cmd_mod,
     artifacts as artifacts_cmd,
+    brm as brm_cmd,
     jobs as jobs_cmd,
     metadata as metadata_cmd,
     metrics as metrics_cmd,
@@ -137,6 +138,7 @@ app.add_typer(tasks_cmd.app, name="tasks")
 app.add_typer(jobs_cmd.app, name="jobs")
 app.add_typer(artifacts_cmd.app, name="artifacts")
 app.add_typer(metadata_cmd.app, name="metadata")
+app.add_typer(brm_cmd.app, name="brm")
 app.command(
     "ap-cmd",
     context_settings={"allow_extra_args": True, "ignore_unknown_options": True},