inspect-eval-utils 0.4.0__tar.gz

inspect_eval_utils-0.4.0/.gitignore

@@ -0,0 +1,9 @@
+__pycache__/
+*.pyc
+.venv/
+.ruff_cache/
+.pytest_cache/
+.mypy_cache/
+dist/
+build/
+*.egg-info/

inspect_eval_utils-0.4.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 METR
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

inspect_eval_utils-0.4.0/PKG-INFO

@@ -0,0 +1,521 @@
+Metadata-Version: 2.4
+Name: inspect-eval-utils
+Version: 0.4.0
+Summary: Shared utilities for METR Inspect AI eval repos: task scaffolder + common runtime helpers.
+Project-URL: Repository, https://github.com/METR/inspect-eval-utils
+Project-URL: Issues, https://github.com/METR/inspect-eval-utils/issues
+Author-email: METR <rasmus.faber-espensen@metr.org>
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 4 - Beta
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.13
+Requires-Python: >=3.13
+Requires-Dist: boto3>=1.40
+Requires-Dist: inspect-ai>=0.3.200
+Requires-Dist: jinja2>=3.0
+Requires-Dist: libcst>=1.5
+Requires-Dist: tomlkit>=0.13
+Provides-Extra: report
+Requires-Dist: matplotlib>=3.8; extra == 'report'
+Requires-Dist: universal-pathlib>=0.2; extra == 'report'
+Description-Content-Type: text/markdown
+
+# inspect-eval-utils
+
+Shared utilities for METR Inspect AI eval repos -- used by both task authors
+and agent scaffolding:
+
+- `inspect_eval_utils.setting`: the `Setting` protocol that lets tasks declare
+  what they need from agent scaffolding (workspaces, tools, callbacks,
+  environment features). Imported by *both* tasks and the scaffolding that
+  consumes them.
+- `new_task` CLI: scaffold a new Inspect AI task into any compatible repo.
+- `inspect_eval_utils.common`: runtime helpers for tasks (`get_sandbox_files`,
+  `expand_template`, `load_text_file`, etc.).
+
+## Installation
+
+### Install (recommended)
+
+```bash
+uv tool install inspect-eval-utils
+new_task my_eval
+```
+
+### For one-off use without installing
+
+```bash
+uvx --from inspect-eval-utils new_task my_eval
+```
+
+
+## Shared task secrets
+
+Updated evals can read shared task secrets directly while still working in old
+Hawk and local workflows. Use `get_task_secret()` for values that might come
+from either an environment variable or AWS Secrets Manager:
+
+```python
+from inspect_eval_utils.common import get_task_secret
+
+hf_token = get_task_secret("HF_TOKEN")
+```
+
+Lookup order is:
+
+1. Return the environment variable named `HF_TOKEN` if it is set.
+2. Otherwise fetch AWS Secrets Manager secret
+   `${INSPECT_TASK_SECRETS_DEFAULT_ARN_PREFIX}HF_TOKEN`.
+
+The secret name suffix is verbatim, so `HF_TOKEN` maps to
+`inspect-tasks/HF_TOKEN`, not a lowercased variant. In normal Hawk runs and
+`hawk local`, the default prefix is provided through
+`INSPECT_TASK_SECRETS_DEFAULT_ARN_PREFIX`. The prefix must include the trailing
+slash, for example
+`arn:aws:secretsmanager:us-west-2:123456789012:secret:inspect-tasks/`. When
+running `inspect eval` directly, set that variable yourself or pass `arn=` to
+`get_task_secret()`.
+
+## Setting protocol
+
+`Setting` is the contract a task publishes to agent scaffolding. It answers
+one question: *what does the agent need to operate on this task?* The
+scaffolding reads a Setting and wires up the agent accordingly. Neither side
+needs to know the other's internals.
+
+```python
+from inspect_eval_utils.setting import Setting, Workspace, Features
+
+Setting(
+    workspaces=(Workspace(name="default", description="Your working environment"),),
+    tools=(check_flag(),),
+    features=Features(internet=True),
+)
+```
+
+### Concepts
+
+#### What a Workspace is (and isn't)
+
+A Workspace is like an SSH login handed to the agent. It names a sandbox to
+which the agent should have direct shell and file access. For each workspace,
+the scaffolding should create a new instance of each of its normal environment
+interaction tools (e.g. a `bash` and a `python` tool) that is bound to that
+sandbox.
+
+**Not every sandbox is a Workspace.** A CTF task might have three containers --
+an attacker box, a target web server, and a database. Only the attacker box is a
+Workspace. The target and database are infrastructure; the agent reaches them
+over the network or through task tools. By leaving them out of `workspaces`, the
+task hides them from the agent by design.
+
+> If a human would SSH into it, it's a Workspace. If the agent attacks it over
+> the network, it's not.
+
+#### Setting is exhaustive
+
+When a Setting is present, it is authoritative. **Empty `workspaces` means no
+bash/python tools.** A task that wants shell access *and* custom tools must
+declare both:
+
+```python
+# Wrong -- custom tool but no shell access
+Setting(tools=(my_tool(),))
+
+# Right -- explicit about both
+Setting(workspaces=(Workspace(),), tools=(my_tool(),))
+```
+
+A pure-API task (call an endpoint, evaluate the result) genuinely has no
+workspace. Scaffolding that silently adds shell tools would undermine that
+constraint.
+
+#### Three layers of tools
+
+The agent's tool surface has three distinct origins:
+
+| Layer | Source | Examples |
+|---|---|---|
+| Task tools | `Setting.tools` | `check_flag`, `submit_image` |
+| Workspace tools | Scaffolding, per workspace | `bash`, `python` |
+| Framework tools | Scaffolding's own concerns | `set_timeout`, `submit` |
+
+The task owns the first layer. The scaffolding owns the other two.
+
+#### Per-turn callbacks: `on_turn` and `monitor`
+
+Some tasks need to do work between agent turns: check progress, advance a
+simulated clock, deliver a queued message, log score evolution. `Setting`
+exposes two callbacks for this, which differ in *who decides when they fire*
+and *whether they can steer the agent*.
+
+**`on_turn`** runs at the start of every agent-loop iteration, before the model
+generates. The task author owns the cadence: it's guaranteed to fire once per
+turn. It can steer the loop by what it returns:
+
+- `False` -- stop the loop (task is over: solved, irretrievably failed, time up)
+- `str`   -- inject this string as a user message before the next generation
+            (for example: "you have a new email", "10 simulated minutes passed")
+- `None` / `True` -- proceed normally
+
+This is useful when the task model needs to *react to the turn happening*: a
+mailbox task that surfaces new messages, a clock-driven simulation that ticks
+on each step, an end-condition check that the task wants to evaluate before
+spending another model call.
+
+**`monitor`** is observational. It returns `None` and cannot steer the loop.
+The scaffolding decides when to call it -- typically at turn boundaries for
+LLM agents, or on a wall-clock schedule for human/Claude-Code style agents
+where there are no clear turns. Use it for things that should run regardless
+of agent type and where missing a tick (or getting an extra one) is fine:
+periodic score logging, transcript annotations, sandbox health checks.
+
+Rule of thumb: if the task needs to *control* what happens next, use
+`on_turn`. If it just needs to *watch*, use `monitor`.
+
+#### Features vs. tools
+
+`Features` are boolean flags about the *environment* -- `vision`, `internet`.
+They tell scaffolding "this task involves images" or "this environment has
+network access." The scaffolding responds by providing generic tools
+(`view_image`, web search) if the model supports them. If the scaffolding
+doesn't support a feature, the task still runs -- scores reflect the outcome.
+
+`Setting.tools` is the other side of the split: tools that belong to the
+*task*. Think of an agent on a task as a carpenter on a job site: hammer,
+saw, and screwdriver belong to the carpenter; walls, doors, and windows
+belong to the house. The carpenter operates on all six, but ownership is
+clean. Scaffolding tools (`bash`, `view_image`, web search) are the
+carpenter's kit, lit up by `Features` when the task says what kind of job
+this is. `Setting.tools` (`check_flag`, `make_move`, `submit_design`) ship
+with the task itself -- outside it, they're meaningless.
+
+The test:
+
+> Would this tool still make sense on a different task? If yes -- gate it on
+> a Feature. If no -- it belongs in `Setting.tools`.
+
+### For task authors
+
+#### Declaring a task environment
+
+Construct a `Setting` and pass it to `use_setting()` in your task's setup:
+
+```python
+from inspect_eval_utils.setting import Setting, Workspace, Features, use_setting
+
+Task(
+    setup=use_setting(Setting(
+        workspaces=(Workspace(name="default", user="agent"),),
+        tools=(check_flag(),),
+        on_turn=my_callback,
+        features=Features(vision=True),
+    )),
+    solver=my_agent(),
+)
+```
+
+`use_setting` also accepts a factory for per-sample Settings:
+
+```python
+use_setting(lambda sample: Setting(
+    workspaces=(Workspace(name="default", user=sample.metadata["user"]),),
+))
+```
+
+#### Examples
+
+**Simple coding task.** One workspace, no extras.
+```python
+Setting(workspaces=(Workspace(name="dev"),))
+```
+
+**CTF task.** Attacker workspace, a scoring tool, no internet. Target machine is
+NOT listed -- it's infrastructure.
+```python
+Setting(
+    workspaces=(Workspace(name="attacker", description="Your attack machine", user="hacker"),),
+    tools=(check_flag(),),
+)
+```
+
+**Creative task with vision.** Workspace for building, vision enabled so
+scaffolding provides image viewing.
+```python
+Setting(
+    workspaces=(Workspace(name="default", user="agent"),),
+    features=Features(vision=True),
+)
+```
+
+**Pure-API task.** No workspace, just a custom tool.
+```python
+Setting(tools=(call_api(),))
+```
+
+**Dynamic tools via ToolSource.** When the available tools depend on task state
+(e.g. a game where legal moves change each turn), use a `ToolSource`:
+```python
+class GameToolSource(ToolSource):
+    async def tools(self) -> list[Tool]:
+        return [move for move in legal_moves()]
+
+Setting(tools=(GameToolSource(),))
+```
+
+Scaffolding calls `tools()` before each generation, so the set stays current.
+
+When `inspect_eval_utils.tool_cli.setting_tool_cli_running()` exposes these tools
+inside a sandbox, `ToolSource` is resolved dynamically at CLI invocation time.
+Use:
+
+```bash
+tools list
+tools describe <tool-name>
+tools call <tool-name> [args...]
+tools <tool-name> [args...]
+```
+
+`tools <tool-name>` is shorthand for `tools call <tool-name>`. If a tool name
+conflicts with a built-in command such as `list`, `describe`, or `call`, use
+`tools call <tool-name>`.
+
+For schemas that are awkward to express as shell flags, pass a JSON object:
+
+```bash
+tools call <tool-name> --json-args '{"arg": "value"}'
+```
+
+The CLI keeps a short cache for list/help/completion metadata, but tool calls
+refresh the current `ToolSource` before execution.
+
+#### Common mistakes
+
+- **Listing infrastructure sandboxes as Workspaces.** Only list sandboxes the
+  agent needs direct shell/file access to. Targets, databases, and services
+  should be omitted.
+- **Assuming empty `workspaces` means "use defaults."** It means no workspaces.
+  The agent gets no bash/python.
+- **Putting generic capabilities in `Setting.tools`.** Tools like `view_image`
+  are scaffolding concerns gated on Features, not task tools.
+
+### For scaffolding developers
+
+#### Reading the Setting
+
+The Setting lives in a `ContextVar`, set per-sample by `use_setting()`. When
+`setting()` returns `None`, the task predates this protocol -- scaffolding must
+remain functional without it.
+
+```python
+from inspect_eval_utils.setting import setting
+
+s = setting()  # returns Setting | None
+if s is not None:
+    # Use Setting-aware tool creation
+    tools.append(s.tools)
+else:
+    # Fall back to existing behavior
+```
+
+#### Creating tools from workspaces
+
+Each Workspace declares a sandbox name and user. The scaffolding creates
+whatever tools it wants for each workspace:
+
+```python
+for ws in s.workspaces:
+    tools.append(bash(sandbox=ws.name, user=ws.user, timeout=timeout))
+    tools.append(python(sandbox=ws.name, user=ws.user, timeout=timeout))
+```
+
+#### Handling on_turn callbacks
+
+Call `handle_on_turn()` at the top of each agent loop iteration, before
+generating:
+
+```python
+from inspect_eval_utils.setting import handle_on_turn
+
+result = await handle_on_turn()  # returns OnTurnResult
+# result.action: "break" | "notify" | "proceed"
+# result.message: str | None (only for "notify")
+```
+
+- `"break"` -- stop the agent loop
+- `"notify"` -- inject `result.message` as a user message, then continue
+- `"proceed"` -- continue normally (also returned when no Setting or no on_turn)
+
+#### Reading Features
+
+```python
+if s.features.vision:
+    tools.append(my_view_image_tool())
+if s.features.internet:
+    tools.append(my_web_search_tool())
+```
+
+Features are advisory. If the scaffolding doesn't support a feature, skip it
+gracefully -- don't error.
+
+## Scaffolding a new task
+
+From inside a target repo (e.g. `inspect-eval-examples`):
+
+```bash
+new_task my_eval
+uv sync --group tasks
+uv run inspect eval my_eval --model mockllm/replay --limit 1
+```
+
+### What gets created
+
+After running `new_task my_eval`, you'll see a new package under `tasks/`:
+
+```
+tasks/my_eval/
+├── pyproject.toml
+├── README.md
+└── src/
+    └── metr_tasks/          # or harder_tasks/, etc., based on the target's namespace
+        └── my_eval/
+            ├── __init__.py
+            ├── _registry.py
+            ├── task.py
+            ├── version.py
+            ├── py.typed
+            ├── sandbox/
+            │   ├── compose.yaml
+            │   └── Dockerfile
+            └── assets/
+                └── instructions.md
+```
+
+The scaffolder also edits the target's root `pyproject.toml` to wire the new
+task into the workspace: it appends the package to `dependency-groups.tasks`
+and adds an entry under `tool.uv.sources` (`<package> = { workspace = true }`).
+It does NOT modify `[tool.uv.workspace].members` — that's typically a glob like
+`["tasks/*"]` which automatically picks up the new directory. This is the most
+common surprise — the scaffolder modifies a file outside `tasks/my_eval/`, so
+review the diff before committing.
+
+### How substitution works
+
+The scaffolder rewrites two things in the same pass:
+
+1. **Task name**: every reference to `template` in the source (file names,
+   function names, imports, project name, etc.) is renamed to your new task
+   name.
+2. **Namespace**: imports like `from metr_tasks.template.task import template`
+   are rewritten to use your repo's actual Python namespace (e.g.
+   `from harder_tasks.my_eval.task import my_eval`). This is what makes the
+   same canonical template work for any METR repo.
+
+### Template selection
+
+The scaffolder uses, in order:
+
+1. `--template <path>` if specified.
+2. `<target>/tasks/template/` if it exists.
+3. The bundled canonical template (a known-good `metr_tasks` template).
+
+### Per-repo target configuration
+
+The scaffolder needs to know your target repo's Python namespace and project
+prefix. It picks them up via the following decision tree:
+
+- **Auto-detected (no config needed)**: if the target repo already has at
+  least one task under `tasks/`, the scaffolder reads its namespace and
+  project prefix from there.
+- **Required config**: if the target repo has no existing tasks under `tasks/`
+  for the scaffolder to inspect, you must declare the namespace explicitly.
+  Add the following to the root `pyproject.toml` (use whatever namespace your
+  repo uses; it's `metr_tasks` for `inspect-eval-examples`, `harder_tasks` for
+  `harder-tasks`, etc.). Without this, the scaffolder errors out on a fresh
+  repo even if you'd be using `metr_tasks`:
+
+  ```toml
+  [tool.task-scaffolder]
+  namespace = "your_namespace"
+  # project-prefix optional, defaults to namespace.replace("_", "-") + "-"
+  ```
+
+- **CLI override**: `--namespace` and `--project-prefix` flags always win,
+  useful for one-offs.
+
+### Examples
+
+#### Example 1 — canonical `metr_tasks` repo (e.g. `inspect-eval-examples`)
+
+```bash
+cd ~/src/metr/inspect-eval-examples
+new_task my_eval
+uv sync
+uv run inspect eval my_eval --model mockllm/replay --limit 1
+```
+
+What you get: `tasks/my_eval/` with the `metr_tasks.my_eval` namespace.
+
+#### Example 2 — cross-namespace repo (e.g. `harder-tasks`)
+
+First, ensure the target's root `pyproject.toml` has:
+
+```toml
+[tool.task-scaffolder]
+namespace = "harder_tasks"
+```
+
+(Skip this if the repo already has tasks the scaffolder can detect from.)
+
+Then:
+
+```bash
+cd ~/src/metr/harder-tasks
+new_task my_eval
+uv sync
+uv run inspect eval my_eval --model mockllm/replay --limit 1
+```
+
+What you get: `tasks/my_eval/` with the `harder_tasks.my_eval` namespace,
+automatically rewritten from the canonical `metr_tasks` template.
+
+### Troubleshooting
+
+- **"target has no pyproject.toml"** — the resolved target directory doesn't
+  contain a `pyproject.toml`. You're either not in the repo root, or
+  `--target <path>` pointed somewhere wrong. `cd` to the repo root, or pass
+  the correct `--target`.
+- **"task name 'template' matches the template name; choose a different
+  name"** — pick something else. `template` is reserved.
+- **"<path> already exists (use --force to overwrite)"** — pass `--force` if
+  you want to overwrite the existing task directory.
+
+## Common helpers
+
+```python
+from inspect_eval_utils.common import (
+    get_sandbox_files,
+    expand_template,
+    load_text_file,
+)
+```
+
+These were ported from `harder-tasks` and are now shared across METR
+Inspect AI eval repos.
+
+## Development
+
+```bash
+uv sync
+uv run pytest                       # fast tests
+uv run pytest --runslow             # + slow end-to-end
+uv run ruff check .
+uv run basedpyright
+```
+
+## License
+
+MIT. See [LICENSE](LICENSE).