inspect-brief 0.1.1__tar.gz

inspect_brief-0.1.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Hirundo-io
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

inspect_brief-0.1.1/PKG-INFO

@@ -0,0 +1,281 @@
+Metadata-Version: 2.4
+Name: inspect-brief
+Version: 0.1.1
+Summary: Generate standardized concise metric summaries for Inspect evaluations.
+Author-email: Hirundo <dev@hirundo.io>
+License: MIT License
+        
+        Copyright (c) 2026 Hirundo-io
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+Project-URL: Homepage, https://github.com/Hirundo-io/inspect-brief
+Keywords: inspect-ai,evaluation,metrics,summarization,report generation
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3
+Requires-Python: <3.14,>=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: inspect-ai>=0.3.112
+Requires-Dist: python-dotenv
+Requires-Dist: tqdm
+Requires-Dist: typer
+Dynamic: license-file
+
+# inspect-brief
+
+Generate standardized concise metric summaries from [Inspect AI](https://inspect.aisi.org.uk/) evaluation logs and append them to a CSV.
+
+> [!IMPORTANT]
+> Inspect Brief currently supports Inspect `.eval` logs only. JSON-formatted
+> Inspect evaluation logs are not supported as input.
+
+## Features
+
+- Load Inspect `.eval` logs recursively or from explicit file paths
+- Optionally filter by task name and select target metrics per task
+- Append results to a CSV (rewrites the header when columns change, preserving existing rows)
+- Skip runs already present in the CSV via `--skip-existing`
+- Optionally export each task automatically through an Inspect Hook
+
+## Installation
+
+With `uv`:
+
+```bash
+uv add inspect-brief
+```
+
+Or with `pip`:
+
+```bash
+pip install inspect-brief
+```
+
+Installing the package provides the `inspect-brief` command and registers the
+Inspect extension entry point.
+
+For one-off use, `uvx` can run either workflow without adding Inspect Brief to
+the current project. The workflow sections below show the corresponding
+commands.
+
+For local development from a repository checkout, see
+[`CONTRIBUTING.md`](CONTRIBUTING.md).
+
+## Choose a workflow
+
+Inspect Brief can run in two separate ways:
+
+- **Inspect Hook:** automatically export a summary whenever an Inspect task finishes.
+- **CLI:** manually process existing `.eval` logs after an evaluation has completed.
+
+The hook is the recommended flow for automatic export during normal Inspect runs.
+Use the CLI for existing logs, one-off exports, or regenerating a summary CSV.
+
+## Automatic export with the Inspect Hook
+
+The opt-in hook receives each completed task directly from Inspect and appends its
+summary rows to the configured CSV. You continue running `inspect eval` normally;
+there is no separate `inspect-brief` command in this flow.
+
+### 1. Configure the output CSV
+
+Set the required output path in the shell that will run Inspect:
+
+```bash
+export INSPECT_BRIEF_CSV_PATH=results/brief_results.csv
+```
+
+The hook is enabled when `INSPECT_BRIEF_CSV_PATH` is set. Set
+`INSPECT_BRIEF_ENABLED=0` to disable it explicitly without removing the output
+configuration.
+
+For persistent local configuration, put the variables in an untracked `.env`
+file in the project directory. Inspect Brief loads this file when Inspect imports
+the hook:
+
+```dotenv
+INSPECT_BRIEF_CSV_PATH=results/brief_results.csv
+# Optional examples:
+INSPECT_BRIEF_TASKS=inspect_evals/gpqa_diamond
+INSPECT_BRIEF_TARGET_METRICS=target_metrics.json
+INSPECT_BRIEF_SKIP_EXISTING=true
+```
+
+### 2. Run Inspect normally
+
+When Inspect Brief is installed in the project, run Inspect normally:
+
+```bash
+inspect eval inspect_evals/gpqa_diamond --model ollama/llama3.2
+```
+
+For a one-off run, start Inspect with Inspect Brief included in its isolated
+tool environment. Every package the run needs must be named explicitly, including
+the one that provides the task. The example inlines the output path so the
+command stands on its own:
+
+```bash
+INSPECT_BRIEF_CSV_PATH=results/brief_results.csv \
+  uvx --from inspect-ai --with inspect-brief --with inspect-evals \
+  inspect eval inspect_evals/gpqa_diamond --model ollama/llama3.2
+```
+
+After every task completes, the hook appends the selected metric rows to the CSV
+and logs the number of rows exported. At the end of the Inspect run, it logs a
+summary containing the handled task count, failed task count, exported row count,
+and output path.
+
+CSV export is synchronous within an Inspect process. Inspect Brief does not use
+cross-process file locking, so each output CSV must have a single process writing
+to it. Do not point concurrent Inspect runs or CLI processes at the same CSV.
+
+### Hook configuration
+
+| Variable | Required | Description |
+| --- | --- | --- |
+| `INSPECT_BRIEF_CSV_PATH` | Yes | Output CSV path. The hook is disabled when it is unset. |
+| `INSPECT_BRIEF_ENABLED` | No | Set to `0`, `false`, `no`, or `off` to disable the hook. When unset, the hook is enabled if `INSPECT_BRIEF_CSV_PATH` is set. |
+| `INSPECT_BRIEF_TASKS` | No | Tasks to include (comma-separated). |
+| `INSPECT_BRIEF_TARGET_METRICS` | No | Target-metrics JSON object or path to a JSON file. |
+| `INSPECT_BRIEF_SKIP_EXISTING` | No | Set to `1`, `true`, `yes`, or `on` to skip Run IDs already in the CSV. |
+
+## Manual export with the CLI
+
+Use this flow when the hook was not enabled during the Inspect run, or when you
+need to process existing logs again.
+
+### 1. Choose the input logs
+
+At least one of `--log-dir` or `--log-files` is required:
+
+- `--log-dir` recursively discovers `.eval` logs under a directory.
+- `--log-files` accepts one or more explicit `.eval` paths or filesystem URIs.
+  Repeat the option, separate sources with commas, or combine both forms.
+- Supplying both combines the discovered and explicit logs and removes duplicates.
+
+JSON-formatted Inspect logs are not supported by either CLI input option.
+
+### 2. Run the export
+
+After installation, invoke the console script:
+
+```bash
+inspect-brief [OPTIONS]
+```
+
+Or run the CLI directly without adding it to the current project:
+
+```bash
+uvx inspect-brief [OPTIONS]
+```
+
+### CLI options
+
+| Option | Description |
+| --- | --- |
+| `--log-dir` | Directory containing Inspect logs; recursively finds `*.eval` files and combines them with `--log-files` when both are supplied |
+| `--log-files` | One or more explicit `.eval` paths or filesystem URIs (repeatable or comma-separated); combines them with logs found by `--log-dir` when both are supplied |
+| `--tasks` | Tasks to include (repeatable or comma-separated); others are skipped |
+| `--target-metrics` | JSON object (or path to a JSON file) mapping task → list of `InspectScore` objects |
+| `--csv-path` | Output CSV path (default: `brief_results.csv` under `--log-dir`, or the current directory) |
+| `--skip-existing` | Skip task runs whose Run ID is already in the CSV |
+
+### CLI examples
+
+Summarize every `.eval` under a log tree:
+
+```bash
+inspect-brief --log-dir /path/to/inspect/logs
+```
+
+Summarize specific files and write to a chosen CSV:
+
+```bash
+inspect-brief \
+  --log-files /path/to/a.eval \
+  --log-files /path/to/b.eval \
+  --csv-path results.csv
+```
+
+Provider-backed logs can be supplied directly:
+
+```bash
+inspect-brief --log-files s3://bucket/path/run.eval
+```
+
+Filter tasks and skip runs already recorded:
+
+```bash
+inspect-brief \
+  --log-dir /path/to/inspect/logs \
+  --tasks inspect_evals/gpqa_diamond,inspect_harbor/gorilla_bfcl_parity \
+  --csv-path results.csv \
+  --skip-existing
+```
+
+### Target metrics
+
+When `--target-metrics` is omitted, every metric present in the log scores is exported (with a scorer prefix when a log has multiple scorers).
+
+When provided, pass a JSON object (inline or as a file path) mapping each task name to a list of `InspectScore` objects. Each object must have exactly these keys:
+
+- `name` — metric name as it appears in the Inspect log
+- `is_percentage` — whether to treat the value as a percentage (appends ` (%)` to the metric label)
+- `is_higher_better` — appends `⬆️` or `⬇️` to the metric label
+- `is_normalized` — if `is_percentage` is true and this is true, multiply the value by `100`
+
+Example (quote the JSON for the shell):
+
+```bash
+inspect-brief --log-dir /path/to/logs --target-metrics '{
+  "inspect_evals/gpqa_diamond": [
+    {
+      "name": "accuracy",
+      "is_percentage": true,
+      "is_higher_better": true,
+      "is_normalized": true
+    }
+  ]
+}'
+```
+
+Or point at a file:
+
+```bash
+inspect-brief --log-dir /path/to/logs --target-metrics ./target_metrics.json
+```
+
+## Output
+
+Results are appended to the CSV. Columns:
+
+| Created | Run ID | Benchmark | Metric | Score | Runtime (sec) |
+| --- | --- | --- | --- | --- | --- |
+| 2026-08-17T16:54:14+00:00 | L67rTm5rz3wkwVdTLMGDme | inspect_evals/gpqa_diamond | accuracy | 0.3699 | 28 |
+
+- **Created** comes from `log.eval.created`, falling back to `log.stats.started_at`
+- **Score** is formatted to 2 decimal places when `> 1.0`, otherwise 4 decimal places
+- Failed or incomplete runs record a status string in the Score column when applicable
+
+## Contributing
+
+See [`CONTRIBUTING.md`](CONTRIBUTING.md) for local setup, verification steps,
+and pull-request expectations.

inspect_brief-0.1.1/README.md

@@ -0,0 +1,240 @@
+# inspect-brief
+
+Generate standardized concise metric summaries from [Inspect AI](https://inspect.aisi.org.uk/) evaluation logs and append them to a CSV.
+
+> [!IMPORTANT]
+> Inspect Brief currently supports Inspect `.eval` logs only. JSON-formatted
+> Inspect evaluation logs are not supported as input.
+
+## Features
+
+- Load Inspect `.eval` logs recursively or from explicit file paths
+- Optionally filter by task name and select target metrics per task
+- Append results to a CSV (rewrites the header when columns change, preserving existing rows)
+- Skip runs already present in the CSV via `--skip-existing`
+- Optionally export each task automatically through an Inspect Hook
+
+## Installation
+
+With `uv`:
+
+```bash
+uv add inspect-brief
+```
+
+Or with `pip`:
+
+```bash
+pip install inspect-brief
+```
+
+Installing the package provides the `inspect-brief` command and registers the
+Inspect extension entry point.
+
+For one-off use, `uvx` can run either workflow without adding Inspect Brief to
+the current project. The workflow sections below show the corresponding
+commands.
+
+For local development from a repository checkout, see
+[`CONTRIBUTING.md`](CONTRIBUTING.md).
+
+## Choose a workflow
+
+Inspect Brief can run in two separate ways:
+
+- **Inspect Hook:** automatically export a summary whenever an Inspect task finishes.
+- **CLI:** manually process existing `.eval` logs after an evaluation has completed.
+
+The hook is the recommended flow for automatic export during normal Inspect runs.
+Use the CLI for existing logs, one-off exports, or regenerating a summary CSV.
+
+## Automatic export with the Inspect Hook
+
+The opt-in hook receives each completed task directly from Inspect and appends its
+summary rows to the configured CSV. You continue running `inspect eval` normally;
+there is no separate `inspect-brief` command in this flow.
+
+### 1. Configure the output CSV
+
+Set the required output path in the shell that will run Inspect:
+
+```bash
+export INSPECT_BRIEF_CSV_PATH=results/brief_results.csv
+```
+
+The hook is enabled when `INSPECT_BRIEF_CSV_PATH` is set. Set
+`INSPECT_BRIEF_ENABLED=0` to disable it explicitly without removing the output
+configuration.
+
+For persistent local configuration, put the variables in an untracked `.env`
+file in the project directory. Inspect Brief loads this file when Inspect imports
+the hook:
+
+```dotenv
+INSPECT_BRIEF_CSV_PATH=results/brief_results.csv
+# Optional examples:
+INSPECT_BRIEF_TASKS=inspect_evals/gpqa_diamond
+INSPECT_BRIEF_TARGET_METRICS=target_metrics.json
+INSPECT_BRIEF_SKIP_EXISTING=true
+```
+
+### 2. Run Inspect normally
+
+When Inspect Brief is installed in the project, run Inspect normally:
+
+```bash
+inspect eval inspect_evals/gpqa_diamond --model ollama/llama3.2
+```
+
+For a one-off run, start Inspect with Inspect Brief included in its isolated
+tool environment. Every package the run needs must be named explicitly, including
+the one that provides the task. The example inlines the output path so the
+command stands on its own:
+
+```bash
+INSPECT_BRIEF_CSV_PATH=results/brief_results.csv \
+  uvx --from inspect-ai --with inspect-brief --with inspect-evals \
+  inspect eval inspect_evals/gpqa_diamond --model ollama/llama3.2
+```
+
+After every task completes, the hook appends the selected metric rows to the CSV
+and logs the number of rows exported. At the end of the Inspect run, it logs a
+summary containing the handled task count, failed task count, exported row count,
+and output path.
+
+CSV export is synchronous within an Inspect process. Inspect Brief does not use
+cross-process file locking, so each output CSV must have a single process writing
+to it. Do not point concurrent Inspect runs or CLI processes at the same CSV.
+
+### Hook configuration
+
+| Variable | Required | Description |
+| --- | --- | --- |
+| `INSPECT_BRIEF_CSV_PATH` | Yes | Output CSV path. The hook is disabled when it is unset. |
+| `INSPECT_BRIEF_ENABLED` | No | Set to `0`, `false`, `no`, or `off` to disable the hook. When unset, the hook is enabled if `INSPECT_BRIEF_CSV_PATH` is set. |
+| `INSPECT_BRIEF_TASKS` | No | Tasks to include (comma-separated). |
+| `INSPECT_BRIEF_TARGET_METRICS` | No | Target-metrics JSON object or path to a JSON file. |
+| `INSPECT_BRIEF_SKIP_EXISTING` | No | Set to `1`, `true`, `yes`, or `on` to skip Run IDs already in the CSV. |
+
+## Manual export with the CLI
+
+Use this flow when the hook was not enabled during the Inspect run, or when you
+need to process existing logs again.
+
+### 1. Choose the input logs
+
+At least one of `--log-dir` or `--log-files` is required:
+
+- `--log-dir` recursively discovers `.eval` logs under a directory.
+- `--log-files` accepts one or more explicit `.eval` paths or filesystem URIs.
+  Repeat the option, separate sources with commas, or combine both forms.
+- Supplying both combines the discovered and explicit logs and removes duplicates.
+
+JSON-formatted Inspect logs are not supported by either CLI input option.
+
+### 2. Run the export
+
+After installation, invoke the console script:
+
+```bash
+inspect-brief [OPTIONS]
+```
+
+Or run the CLI directly without adding it to the current project:
+
+```bash
+uvx inspect-brief [OPTIONS]
+```
+
+### CLI options
+
+| Option | Description |
+| --- | --- |
+| `--log-dir` | Directory containing Inspect logs; recursively finds `*.eval` files and combines them with `--log-files` when both are supplied |
+| `--log-files` | One or more explicit `.eval` paths or filesystem URIs (repeatable or comma-separated); combines them with logs found by `--log-dir` when both are supplied |
+| `--tasks` | Tasks to include (repeatable or comma-separated); others are skipped |
+| `--target-metrics` | JSON object (or path to a JSON file) mapping task → list of `InspectScore` objects |
+| `--csv-path` | Output CSV path (default: `brief_results.csv` under `--log-dir`, or the current directory) |
+| `--skip-existing` | Skip task runs whose Run ID is already in the CSV |
+
+### CLI examples
+
+Summarize every `.eval` under a log tree:
+
+```bash
+inspect-brief --log-dir /path/to/inspect/logs
+```
+
+Summarize specific files and write to a chosen CSV:
+
+```bash
+inspect-brief \
+  --log-files /path/to/a.eval \
+  --log-files /path/to/b.eval \
+  --csv-path results.csv
+```
+
+Provider-backed logs can be supplied directly:
+
+```bash
+inspect-brief --log-files s3://bucket/path/run.eval
+```
+
+Filter tasks and skip runs already recorded:
+
+```bash
+inspect-brief \
+  --log-dir /path/to/inspect/logs \
+  --tasks inspect_evals/gpqa_diamond,inspect_harbor/gorilla_bfcl_parity \
+  --csv-path results.csv \
+  --skip-existing
+```
+
+### Target metrics
+
+When `--target-metrics` is omitted, every metric present in the log scores is exported (with a scorer prefix when a log has multiple scorers).
+
+When provided, pass a JSON object (inline or as a file path) mapping each task name to a list of `InspectScore` objects. Each object must have exactly these keys:
+
+- `name` — metric name as it appears in the Inspect log
+- `is_percentage` — whether to treat the value as a percentage (appends ` (%)` to the metric label)
+- `is_higher_better` — appends `⬆️` or `⬇️` to the metric label
+- `is_normalized` — if `is_percentage` is true and this is true, multiply the value by `100`
+
+Example (quote the JSON for the shell):
+
+```bash
+inspect-brief --log-dir /path/to/logs --target-metrics '{
+  "inspect_evals/gpqa_diamond": [
+    {
+      "name": "accuracy",
+      "is_percentage": true,
+      "is_higher_better": true,
+      "is_normalized": true
+    }
+  ]
+}'
+```
+
+Or point at a file:
+
+```bash
+inspect-brief --log-dir /path/to/logs --target-metrics ./target_metrics.json
+```
+
+## Output
+
+Results are appended to the CSV. Columns:
+
+| Created | Run ID | Benchmark | Metric | Score | Runtime (sec) |
+| --- | --- | --- | --- | --- | --- |
+| 2026-08-17T16:54:14+00:00 | L67rTm5rz3wkwVdTLMGDme | inspect_evals/gpqa_diamond | accuracy | 0.3699 | 28 |
+
+- **Created** comes from `log.eval.created`, falling back to `log.stats.started_at`
+- **Score** is formatted to 2 decimal places when `> 1.0`, otherwise 4 decimal places
+- Failed or incomplete runs record a status string in the Score column when applicable
+
+## Contributing
+
+See [`CONTRIBUTING.md`](CONTRIBUTING.md) for local setup, verification steps,
+and pull-request expectations.

inspect_brief-0.1.1/inspect_brief/__init__.py

@@ -0,0 +1,4 @@
+"""Generate concise metric summaries from Inspect evaluation logs."""
+
+# Public package version; kept in sync by bumpver.
+__version__ = "0.1.1"

inspect_brief-0.1.1/inspect_brief/__main__.py

@@ -0,0 +1,6 @@
+"""Run the Inspect Brief command-line application."""
+
+if __name__ == "__main__":
+    from .cli import app
+
+    app()

inspect_brief-0.1.1/inspect_brief/_registry.py

@@ -0,0 +1,5 @@
+"""Register inspect-brief extensions with Inspect."""
+
+from inspect_brief.hooks import InspectBriefHooks
+
+__all__ = ["InspectBriefHooks"]