inspect-ai 0.3.73__tar.gz → 0.3.75__tar.gz

inspect_ai-0.3.75/.github/workflows/docker.yml

@@ -0,0 +1,73 @@
+name: Build and Push Image to Docker Hub
+
+on:
+  workflow_dispatch:
+    inputs:
+      image:
+        description: Image to build
+        required: true
+        type: choice
+        options:
+          - inspect-computer-tool
+          - inspect-web-browser-tool
+      tag:
+        description: Tag to assign to the image
+        required: true
+        type: string
+        default: latest
+      platforms:
+        description: Target platforms (comma-separated list)
+        required: true
+        type: string
+        default: linux/amd64,linux/arm64
+      push:
+        description: Push the image to Docker Hub
+        required: true
+        type: boolean
+        default: true
+      org:
+        description: Docker Hub organization
+        required: true
+        type: string
+        default: aisiuk
+
+jobs:
+  build-and-push:
+    runs-on: ubuntu-latest
+    environment: docker
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v3
+      - name: Set build context path and image name
+        run: |
+          if [[ "${{ github.event.inputs.image }}" == "inspect-computer-tool" ]]; then
+            echo "BUILD_CONTEXT=src/inspect_ai/tool/_tools/_computer/_resources" >> $GITHUB_ENV
+          elif [[ "${{ github.event.inputs.image }}" == "inspect-web-browser-tool" ]]; then
+            echo "BUILD_CONTEXT=src/inspect_ai/tool/_tools/_web_browser/_resources" >> $GITHUB_ENV
+          else
+            echo "Invalid image name '${{ github.event.inputs.image }}'"
+            exit 1
+          fi
+      - name: Login to DockerHub
+        uses: docker/login-action@v2
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v2
+        with:
+          platforms: arm64,amd64
+      - name: Set up Docker Buildx
+        id: buildx
+        uses: docker/setup-buildx-action@v2
+        with:
+          install: true
+      - name: Build and (optionally) push multi-arch image
+        uses: docker/build-push-action@v4
+        with:
+          context: ${{ env.BUILD_CONTEXT }}
+          push: ${{ github.event.inputs.push }}
+          platforms: ${{ github.event.inputs.platforms }}
+          tags: ${{ github.event.inputs.org }}/${{ github.event.inputs.image }}:${{ github.event.inputs.tag }}
+          cache-from: type=gha
+          cache-to: type=gha,mode=max

{inspect_ai-0.3.73 → inspect_ai-0.3.75}/CHANGELOG.md

@@ -1,12 +1,36 @@
+## v0.3.75 (18 March 2025)
+
+- Model API: Specifying a default model (e.g. `--model`) is no longer required (as some evals have no model or use `get_model()` for model access).
+- Tasks can now directly specify a `model`, and model is no longer a required axis for parallel tasks.
+- Eval Set: Improved parallelisation in scheduler (all pending tasks are now run together rather than in model groups).
+- Don't generate `id` for `ChatMessage` when deserialising (`id` is now `str | None` and is only populated when messages are directly created).
+- Log: Support for zip64 extensions required to read some log files that are larger than 4GB.
+- Anthropic: Provide `reasoning_tokens` for standard thinking blocks (redacted thinking not counted).
+- Google: Improve checking of `APIError` status codes for retry.
+- CLI: Added `--env` option for defining environment variables for the duration of the `inspect` process.
+- Inspect View: Fix issue generating diffs for nested arrays.
+- Inspect View: Fix layout issue with sample error display in sample detail summary.
+- Inspect View: Better support large eval files (in excess of 4GB).
+- Inspect View: Correctly display 'None' when passed in tool calls.
+- Inspect View: Fix 'Access Denied' error when using `inspect view` and viewing the log in a browser.
+- Bugfix: Properly handle nested Pydantic models when reading typed store (`store_as()`) from log.
+- Bugfix: Enable passing `solver` list to `eval()` (decorate `chain` function with `@solver`).
+- Bugfix: Support deserializing custom sandbox configuration objects when said sandbox plugin is not installed.
+- Bugfix: Fix error in sample filtering autocomplete (could cause autocomplete to fail and show an error in js console).
+
+## v0.3.74 (15 March 2025)
+
+- Bugfix: Exclude chat message `id` from cache key (fixes regression in model output caching).
+
 ## v0.3.73 (14 March 2025)
 
-- Constrain model output to a particular JSON schema using [Structured Output](https://inspect.ai-safety-institute.org.uk/structured.html) (supported for OpenAI, Google, and Mistral).
+- Constrain model output to a particular JSON schema using [Structured Output](https://inspect.aisi.org.uk/structured.html) (supported for OpenAI, Google, and Mistral).
 - New "HTTP Retries" display (replacing the "HTTP Rate Limits" display) which counts all retries and does so much more consistently and accurately across providers.
 - The `ModelAPI` class now has a `should_retry()` method that replaces the deprecated `is_rate_limit()` method.
 - The "Generate..." progress message in the Running Samples view now shows the number of retries for the active call to `generate()`.
 - New `inspect trace http` command which will show all HTTP requests for a run.
 - More consistent use of `max_retries` and `timeout` configuration options. These options now exclusively control Inspect's outer retry handler; model providers use their default behaviour for the inner request, which is typically 2-4 retries and a service-appropriate timeout.
-- Improved async implementation using AnyIO (can now optionally run Trio rather than asyncio as the [async backend](https://inspect.ai-safety-institute.org.uk/parallelism.html#async-backends)).
+- Improved async implementation using AnyIO (can now optionally run Trio rather than asyncio as the [async backend](https://inspect.aisi.org.uk/parallelism.html#async-backends)).
 - Agent Bridge: Correct handling for `tool_choice` option.
 - Model API: `ChatMessage` now includes an `id` field (defaults to auto-generated uuid).
 - OpenAI: More flexible parsing of content parts (some providers omit the "type" field); support for "reasoning" content parts.
@@ -23,7 +47,7 @@
 
 ## v0.3.71 (01 March 2025)
 
-- Anthropic: Support for [extended thinking](https://inspect.ai-safety-institute.org.uk/reasoning.html#claude-3.7-sonnet) features of Claude Sonnet 3.7 (minimum version of `anthropic` package bumped to 0.47.1).
+- Anthropic: Support for [extended thinking](https://inspect.aisi.org.uk/reasoning.html#claude-3.7-sonnet) features of Claude Sonnet 3.7 (minimum version of `anthropic` package bumped to 0.47.1).
 - Reasoning: `ContentReasoning` type for representing model reasoning blocks.
 - Reasoning: `reasoning_tokens` for setting maximum reasoning tokens (currently only supported by Claude Sonnet 3.7)
 - Reasoning: `reasoning_history` can now be specified as "none", "all", "last", or "auto" (which yields a provider specific recommended default).
@@ -51,7 +75,7 @@
 
 ## v0.3.70 (25 February 2025)
 
-- [working_limit](https://inspect.ai-safety-institute.org.uk/errors_and_limits.html#working-limit) option for specifying a maximum working time (e.g. model generation, tool calls, etc.) for samples.
+- [working_limit](https://inspect.aisi.org.uk/errors_and_limits.html#working-limit) option for specifying a maximum working time (e.g. model generation, tool calls, etc.) for samples.
 - Added `SandboxEvent` to transcript for recording sandbox execution and I/O.
 - Sandboxes: `as_type()` function for checked downcasting of `SandboxEnvironment`
 - Remove root logging handlers upon Inspect logger initialisation (as they result in lots of log spam if left installed).
@@ -112,9 +136,9 @@
 
 ## v0.3.64 (14 February 2025)
 
-- [Reference documentation](https://inspect.ai-safety-institute.org.uk/reference/) for Python API and CLI commands.
-- Add support for [clustered standard errors](https://inspect.ai-safety-institute.org.uk/scorers.html#clustered-standard-errors) via a new `cluster` parameter for the `stderr()` metric.
-- Improvements to [scoring workflow](https://inspect.ai-safety-institute.org.uk/scorers.html#sec-scorer-workflow) (`inspect score` command and `score()` function).
+- [Reference documentation](https://inspect.aisi.org.uk/reference/) for Python API and CLI commands.
+- Add support for [clustered standard errors](https://inspect.aisi.org.uk/scorers.html#clustered-standard-errors) via a new `cluster` parameter for the `stderr()` metric.
+- Improvements to [scoring workflow](https://inspect.aisi.org.uk/scorers.html#sec-scorer-workflow) (`inspect score` command and `score()` function).
 - Metrics now take `list[SampleScore]` rather than `list[Score]` (previous signature is deprecated but still works with a warning).
 - Use a sample adjustment for the `var()` metric.
 - Google: Speculative fix for completion candidates not being returned as a list.
@@ -131,7 +155,7 @@
 
 ## v0.3.63 (07 February 2025)
 
-- Add [OpenRouter](https://inspect.ai-safety-institute.org.uk/providers.html#openrouter) model provider.
+- Add [OpenRouter](https://inspect.aisi.org.uk/providers.html#openrouter) model provider.
 - Inspect View: Convert codebase from JS/Preact to Typescript/React
 - Add `shuffle_choices` to dataset and dataset loading funtions. Deprecate `shuffle` parameter to the `multiple_choice` solver.
 - Add `stop_words` param to the `f1` scorer. `stop_words` will be removed from the target and answer during normalization.
@@ -167,8 +191,8 @@
 
 ## v0.3.60 (29 January 2025)
 
-- [Agent Bridge](https://inspect.ai-safety-institute.org.uk/agent-bridge.html) for integrating external agent frameworks with Inspect.
-- [Goodfire](https://inspect.ai-safety-institute.org.uk/models.html#goodfire) model provider.
+- [Agent Bridge](https://inspect.aisi.org.uk/agent-bridge.html) for integrating external agent frameworks with Inspect.
+- [Goodfire](https://inspect.aisi.org.uk/models.html#goodfire) model provider.
 - Add `@wraps` to functions wrapped by Inspect decorators to preserve type information.
 - Hugging Face: Add support for stop sequences for HF models.
 - Docker: More robust parsing of version strings (handle development versions).
@@ -186,12 +210,12 @@
 
 ## v0.3.59 (24 January 2025)
 
-- Beta version of [computer()](https://inspect.ai-safety-institute.org.uk/tools.html#sec-computer) tool which models with a computer desktop environment.
+- Beta version of [computer()](https://inspect.aisi.org.uk/tools.html#sec-computer) tool which models with a computer desktop environment.
 - `user_message()` solver for appending parameterised user messages.
 - `prompt_template()`, `system_message()` and `user_message()` solver now also include the sample `store` in substitution parameters.
 - Limits: Enforce token and message limit at lower level (not longer required to check `state.completed` for limit enforcement).
-- Limits: Enforce [custom limits](https://inspect.ai-safety-institute.org.uk/errors-and-limits.html#custom-limit) for samples by raising `SampleLimitExceededError`.
-- Tasks: Optional ability for solvers to [yield scores](https://inspect.ai-safety-institute.org.uk/solvers.html#sec-scoring-in-solvers) for a task.
+- Limits: Enforce [custom limits](https://inspect.aisi.org.uk/errors-and-limits.html#custom-limit) for samples by raising `SampleLimitExceededError`.
+- Tasks: Optional ability for solvers to [yield scores](https://inspect.aisi.org.uk/solvers.html#sec-scoring-in-solvers) for a task.
 - Model API: Log model calls that result in bad request errors.
 - Tools: `model_input` option that determines how tool call result content is played back to the model.
 - Tools: Don't attempt to marshall arguments of dynamic `ToolDef` with `**kwargs: Any` (just pass them through).
@@ -204,7 +228,7 @@
 
 ## v0.3.58 (16 January 2025)
 
-- Support for [audio and video](https://inspect.ai-safety-institute.org.uk/multimodal.html) inputs for Open AI and Google Gemini models.
+- Support for [audio and video](https://inspect.aisi.org.uk/multimodal.html) inputs for Open AI and Google Gemini models.
 - Task display: Added Timeout Tool button for manually timing out a tool call.
 - Task display: Automatically switch to "plain" mode when running in a background thread
 - Sandboxes: Setup and initialisation errors are now handled at the sample level.
@@ -222,7 +246,7 @@
 
 ## v0.3.57 (09 January 2025)
 
-- [Tracing API](https://inspect.ai-safety-institute.org.uk/tracing.html#tracing-api) for custom trace logging.
+- [Tracing API](https://inspect.aisi.org.uk/tracing.html#tracing-api) for custom trace logging.
 - Inspect View: never truncate tool result images and display at default width of 800px.
 - Inspect View: display tool error messages in transcript when tool errors occur.
 - Inspect View: display any completed samples even if the task fails because of an error
@@ -241,9 +265,9 @@
 
 ## v0.3.56 (01 January 2025)
 
-- [Human Agent](https://inspect.ai-safety-institute.org.uk/human-agent.html) solver for human baselining of computing tasks.
-- [Typed interfaces](https://inspect.ai-safety-institute.org.uk/typing.html) to `Sample` store and metadata using Pydantic models.
-- [Approval policies](https://inspect.ai-safety-institute.org.uk/approval.html#task-approvers) can now be defined at the `Task` level (`eval` level approval policies take precedence).
+- [Human Agent](https://inspect.aisi.org.uk/human-agent.html) solver for human baselining of computing tasks.
+- [Typed interfaces](https://inspect.aisi.org.uk/typing.html) to `Sample` store and metadata using Pydantic models.
+- [Approval policies](https://inspect.aisi.org.uk/approval.html#task-approvers) can now be defined at the `Task` level (`eval` level approval policies take precedence).
 - Tools can now return `ContentText` and `ContentImage`.
 - Move tool result images into subsequent user messages for models that don't support tools returning images.
 - `SandboxConnection` that contains login information from sandboxes.
@@ -260,7 +284,7 @@
 
 ## v0.3.54 (26 December 2024)
 
-- [Tracing](https://inspect.ai-safety-institute.org.uk/tracing.html) for diagnosing runs with unterminated action (e.g. model calls, docker commands, etc.).
+- [Tracing](https://inspect.aisi.org.uk/tracing.html) for diagnosing runs with unterminated action (e.g. model calls, docker commands, etc.).
 - Provide default timeout/retry for docker compose commands to mitigate unreliability in some configurations.
 - Switch to sync S3 writes to overcome unreliability observed when using async interface.
 - Task display: Added `--no-score-display` option to disable realtime scoring metrics.
@@ -343,9 +367,9 @@
 ## v0.3.48 (01 December 2024)
 
 - [Realtime display](https://github.com/UKGovernmentBEIS/inspect_ai/pull/865) of sample transcripts (including ability to cancel running samples).
-- Scoring: When using a dictionary to map metrics to score value dictionaries, you may now use globs as keys. See our [scorer documentation](https://inspect.ai-safety-institute.org.uk/scorers.html#sec-multiple-scorers) for more information.
+- Scoring: When using a dictionary to map metrics to score value dictionaries, you may now use globs as keys. See our [scorer documentation](https://inspect.aisi.org.uk/scorers.html#sec-multiple-scorers) for more information.
 - `EvalLog` now includes a [location](https://github.com/UKGovernmentBEIS/inspect_ai/pull/872) property indicating where it was read from.
-- Use [tool views](https://inspect.ai-safety-institute.org.uk/approval.html#tool-views) when rendering tool calls in Inspect View.
+- Use [tool views](https://inspect.aisi.org.uk/approval.html#tool-views) when rendering tool calls in Inspect View.
 - Consistent behavior for `max_samples` across sandbox and non-sandbox evals (both now apply `max_samples` per task, formerly evals with sandboxes applied `max_samples` globally).
 - Log files now properly deal with scores that produce Nan. (fixes [#834](https://github.com/UKGovernmentBEIS/inspect_ai/issues/834))
 - Bash tool: add `--login` option so that e.g. .bashrc is read before executing the command.
@@ -372,12 +396,12 @@
 - Google: Combine consecutive tool messages into single content part; ensure no empty text content parts.
 - AzureAI: Create and close client with each call to generate (fixes issue w/ using azureai on multiple passes of eval).
 - Bedrock: Migrate to the [Converse API](https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference-supported-models-features.html), which supports many more features including tool calling and multimodal models.
-- Scoring: When using a dictionary to map metrics to score value dictionaries, you may now use globs as keys. See our [scorer documentation](https://inspect.ai-safety-institute.org.uk/scorers.html#sec-multiple-scorers) for more information.
+- Scoring: When using a dictionary to map metrics to score value dictionaries, you may now use globs as keys. See our [scorer documentation](https://inspect.aisi.org.uk/scorers.html#sec-multiple-scorers) for more information.
 - Sample limit events will now appear in the transcript if a limit (e.g. message, token, or time limit) halt a sample. The sample list and sample detail also display the limit, if applicable.
 
 ## v0.3.46 (12 November 2024)
 
-- [eval](https://inspect.ai-safety-institute.org.uk/eval-logs.html#sec-log-format) is now the default log format (use `--log-format=json` to use old format).
+- [eval](https://inspect.aisi.org.uk/eval-logs.html#sec-log-format) is now the default log format (use `--log-format=json` to use old format).
 - Base 64 images are now logged by default for all log formats (disable with `--no-log-images`).
 - The log viewer now properly displays sample errors in the sample list for `eval` format log files.
 - Improve path handling when using `inspect log convert` to convert a single log file.
@@ -387,8 +411,8 @@
 
 ## v0.3.45 (11 November 2024)
 
-- [time_limit](https://inspect.ai-safety-institute.org.uk/errors_and_limits.html#sample-limits) option for specifying a maximum execution time for samples.
-- [read_eval_log_samples()](https://inspect.ai-safety-institute.org.uk/eval-logs.html#streaming) function for streaming reads of `.eval` log files.
+- [time_limit](https://inspect.aisi.org.uk/errors_and_limits.html#sample-limits) option for specifying a maximum execution time for samples.
+- [read_eval_log_samples()](https://inspect.aisi.org.uk/eval-logs.html#streaming) function for streaming reads of `.eval` log files.
 - Mistral: Support for multi-modal models (requires v1.2 of mistralai package).
 - Groq: Support for multi-modal models (requires v0.11.0 of groq package).
 - AzureAI: Use Model Inference API (preview) for implementation of model client.
@@ -408,7 +432,7 @@
 
 ## v0.3.43 (04 November 2024)
 
-- New binary [log format](https://inspect.ai-safety-institute.org.uk/eval-logs.html#sec-log-format) which yields substantial size and speed improvements (JSON format log files are still fully supported and utilities for converting between the formats are provided).
+- New binary [log format](https://inspect.aisi.org.uk/eval-logs.html#sec-log-format) which yields substantial size and speed improvements (JSON format log files are still fully supported and utilities for converting between the formats are provided).
 - [Grok](https://docs.x.ai/) model provider.
 - [llama-cpp-python](https://llama-cpp-python.readthedocs.io/en/latest/) local model provider.
 - Extensions: correctly load extensions in packages where package name differs from dist name.
@@ -430,7 +454,7 @@
 
 ## v0.3.42 (23 October 2024)
 
-- [ToolDef](https://inspect.ai-safety-institute.org.uk/tools.html#sec-dynamic-tools) class for dynamically creating tool definitions.
+- [ToolDef](https://inspect.aisi.org.uk/tools.html#sec-dynamic-tools) class for dynamically creating tool definitions.
 - Added `--tags` option to eval for tagging evaluation runs.
 - Added APIs for accessing sample event transcripts and for creating and resolving attachments for larger content items.
 - Cleanup Docker Containers immediately for samples with errors.
@@ -460,10 +484,10 @@
 
 ## v0.3.41 (11 October 2024)
 
-- [Approval mode](https://inspect.ai-safety-institute.org.uk/approval.html) for extensible approvals of tool calls (human and auto-approvers built in,  arbitrary other approval schemes via extensions).
-- [Trace mode](https://inspect.ai-safety-institute.org.uk/interactivity.html#sec-trace-mode) for printing model interactions to the terminal.
+- [Approval mode](https://inspect.aisi.org.uk/approval.html) for extensible approvals of tool calls (human and auto-approvers built in,  arbitrary other approval schemes via extensions).
+- [Trace mode](https://inspect.aisi.org.uk/interactivity.html#sec-trace-mode) for printing model interactions to the terminal.
 - Add `as_dict()` utility method to `Score`
-- [Sample limits](https://inspect.ai-safety-institute.org.uk/errors_and_limits.html#sample-limits) (`token_limit` and `message_limit`) for capping the number of tokens or messages used per sample ( `message_limit` replaces deprecated `max_messages`).
+- [Sample limits](https://inspect.aisi.org.uk/errors_and_limits.html#sample-limits) (`token_limit` and `message_limit`) for capping the number of tokens or messages used per sample ( `message_limit` replaces deprecated `max_messages`).
 - Add `metadata` field to `Task` and record in log `EvalSpec`.
 - Include datetime and level in file logger.
 - Correct llama3 and o1 tool calling when empty arguments passed.
@@ -512,7 +536,7 @@
 
 ## v0.3.36 (2 October 2024)
 
-- [Web Browser](https://inspect.ai-safety-institute.org.uk/tools.html#sec-web-browser) tool which provides a headless Chromium browser that supports navigation, history, and mouse/keyboard interactions.
+- [Web Browser](https://inspect.aisi.org.uk/tools.html#sec-web-browser) tool which provides a headless Chromium browser that supports navigation, history, and mouse/keyboard interactions.
 - `auto_id` option for dataset readers to assign an auto-incrementing ID to records.
 - Task args: don't attempt to serialise registry objects that don't have captured parameters.
 
@@ -577,7 +601,7 @@
 
 ## v0.3.30 (18 September 2024)
 
-- Added [fork()](https://inspect.ai-safety-institute.org.uk/agents-api.html#sec-forking) function to fork a `TaskState` and evaluate it against multiple solvers in parallel.
+- Added [fork()](https://inspect.aisi.org.uk/agents-api.html#sec-forking) function to fork a `TaskState` and evaluate it against multiple solvers in parallel.
 - Ensure that Scores produced after being reduced still retain `answer`, `explanation`, and `metadata`.
 - Fix error when running `inspect info log-types`
 - Improve scorer names imported from modules by not including the the module names.
@@ -601,9 +625,9 @@
 
 ## v0.3.28 (14 September 2024)
 
-- [basic_agent()](https://inspect.ai-safety-institute.org.uk/agents.html#sec-basic-agent) that provides a ReAct tool loop with support for retries and encouraging the model to continue if its gives up or gets stuck.
-- [score()](https://inspect.ai-safety-institute.org.uk/solvers.html#sec-scoring-in-solvers) function for accessing scoring logic from within solvers.
-- Ability to [publish](https://inspect.ai-safety-institute.org.uk/log-viewer.html#sec-publishing) a static standalone Inspect View website for a log directory.
+- [basic_agent()](https://inspect.aisi.org.uk/agents.html#sec-basic-agent) that provides a ReAct tool loop with support for retries and encouraging the model to continue if its gives up or gets stuck.
+- [score()](https://inspect.aisi.org.uk/solvers.html#sec-scoring-in-solvers) function for accessing scoring logic from within solvers.
+- Ability to [publish](https://inspect.aisi.org.uk/log-viewer.html#sec-publishing) a static standalone Inspect View website for a log directory.
 - `system_message()` now supports custom parameters and interpolation of `metadata` values from `Sample`.
 - `generate()` solver now accepts arbitrary generation config params.
 - `use_tools()` now accepts a variadic list of `Tool` in addition to literal `list[Tool]`.
@@ -631,9 +655,9 @@
 
 ## v0.3.26 (6 September 2024)
 
-- [Eval Sets](https://inspect.ai-safety-institute.org.uk/eval-sets.html) for running groups of tasks with automatic retries.
-- [Per-sample](https://inspect.ai-safety-institute.org.uk/sandboxing.html#sec-per-sample-sandbox) Sandbox environments can now be specified (e.g. allowing for a distinct Dockerfile or Docker compose file for each sample).
-- [input_screen()](https://inspect.ai-safety-institute.org.uk/interactivity.html) context manager to temporarily clear task display for user input.
+- [Eval Sets](https://inspect.aisi.org.uk/eval-sets.html) for running groups of tasks with automatic retries.
+- [Per-sample](https://inspect.aisi.org.uk/sandboxing.html#sec-per-sample-sandbox) Sandbox environments can now be specified (e.g. allowing for a distinct Dockerfile or Docker compose file for each sample).
+- [input_screen()](https://inspect.aisi.org.uk/interactivity.html) context manager to temporarily clear task display for user input.
 - Introduce two new scorers, `f1()` (precision and recall in text matching) and `exact()` (whether normalized text matches exactly).
 - Task `metrics` now override built in scorer metrics (previously they were merged). This enables improved re-use of existing scorers where they only change required is a different set of metrics.
 - `write_log_dir_manifest()` to write a log header manifest for a log directory.
@@ -652,11 +676,11 @@
 
 ## v0.3.25 (25 August 2024)
 
-- [Store](https://inspect.ai-safety-institute.org.uk/agents-api.html#sharing-state) for manipulating arbitrary sample state from within solvers and tools.
-- [Transcript](https://inspect.ai-safety-institute.org.uk/agents-api.html#transcripts) for detailed sample level tracking of model and tool calls, state changes, logging, etc.
-- [Subtasks](https://inspect.ai-safety-institute.org.uk/agents-api.html#sec-subtasks) for delegating work to helper models, sub-agents, etc.
-- Integration with Anthropic [prompt caching](https://inspect.ai-safety-institute.org.uk/caching.html#sec-provider-caching).
-- [fail_on_error](https://inspect.ai-safety-institute.org.uk/errors-and-limits.html#failure-threshold) option to tolerate some threshold of sample failures without failing the evaluation.
+- [Store](https://inspect.aisi.org.uk/agents-api.html#sharing-state) for manipulating arbitrary sample state from within solvers and tools.
+- [Transcript](https://inspect.aisi.org.uk/agents-api.html#transcripts) for detailed sample level tracking of model and tool calls, state changes, logging, etc.
+- [Subtasks](https://inspect.aisi.org.uk/agents-api.html#sec-subtasks) for delegating work to helper models, sub-agents, etc.
+- Integration with Anthropic [prompt caching](https://inspect.aisi.org.uk/caching.html#sec-provider-caching).
+- [fail_on_error](https://inspect.aisi.org.uk/errors-and-limits.html#failure-threshold) option to tolerate some threshold of sample failures without failing the evaluation.
 - Specify `init` value in default Docker compose file so that exit signals are handled correctly (substantially improves container shutdown performance).
 - Add `function` field to `ChatMessageTool` to indicate the name of the function called.
 - Added [RACE](https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/race-h/) benchmark.
@@ -710,12 +734,12 @@
 
 ## v0.3.19 (02 August 2024)
 
-- [vLLM](https://inspect.ai-safety-institute.org.uk/models.html#sec-vllm) model provider.
+- [vLLM](https://inspect.aisi.org.uk/models.html#sec-vllm) model provider.
 - [Groq](https://groq.com/) model provider.
-- [Google Vertex](https://inspect.ai-safety-institute.org.uk/models.html#google-vertex) model provider.
-- [Reduce scores](https://inspect.ai-safety-institute.org.uk/scorers.html##sec-reducing-epoch) in multi-epoch tasks before computing metrics (defaults to averaging sample values).
-- Replace the use of the `bootstrap_std` metric with `stderr` for built in scorers (see [rationale](https://inspect.ai-safety-institute.org.uk/scorers.html#stderr-note) for details).
-- Option to write Python logger entries to an [external file](https://inspect.ai-safety-institute.org.uk/log-viewer.html#sec-external-file).
+- [Google Vertex](https://inspect.aisi.org.uk/models.html#google-vertex) model provider.
+- [Reduce scores](https://inspect.aisi.org.uk/scorers.html##sec-reducing-epoch) in multi-epoch tasks before computing metrics (defaults to averaging sample values).
+- Replace the use of the `bootstrap_std` metric with `stderr` for built in scorers (see [rationale](https://inspect.aisi.org.uk/scorers.html#stderr-note) for details).
+- Option to write Python logger entries to an [external file](https://inspect.aisi.org.uk/log-viewer.html#sec-external-file).
 - Rename `ToolEnvironment` to `SandboxEnvironment` and `tool_environment()` to `sandbox()` (moving the renamed types from `inspect_ai.tool` to `inspect_ai.util`). Existing symbols will continue to work but will print deprecation errors.
 - Moved the `bash()`, `python()`, and `web_search()` functions from `inspect_ai.solver` to `inspect_ai.tool`.  Existing symbols will continue to work but will print deprecation errors.
 - Enable parallel execution of tasks that share a working directory.
@@ -753,8 +777,8 @@
 
 ## v0.3.18 (14 July 2024)
 
-- [Multiple Scorers](https://inspect.ai-safety-institute.org.uk/scorers.html#sec-multiple-scorers) are now supported for evaluation tasks.
-- [Multiple Models](https://inspect.ai-safety-institute.org.uk/parallelism.html#sec-multiple-models) can now be evaluated in parallel by passing a list of models to `eval()`.
+- [Multiple Scorers](https://inspect.aisi.org.uk/scorers.html#sec-multiple-scorers) are now supported for evaluation tasks.
+- [Multiple Models](https://inspect.aisi.org.uk/parallelism.html#sec-multiple-models) can now be evaluated in parallel by passing a list of models to `eval()`.
 - Add `api_key` to `get_model()` for explicitly specifying an API key for a model.
 - Improved handling of very large (> 100MB) log files in Inspect View.
 - Use `network_mode: none` for disabling networking by default in Docker tool environments.
@@ -784,8 +808,8 @@
 
 ## v0.3.15 (15 June 2024)
 
--   [Sandbox Environments](https://inspect.ai-safety-institute.org.uk/sandboxing.html) for executing tool code in a sandbox.
--   [Caching](https://inspect.ai-safety-institute.org.uk/caching.html) to reduce the number of model API calls made.
+-   [Sandbox Environments](https://inspect.aisi.org.uk/sandboxing.html) for executing tool code in a sandbox.
+-   [Caching](https://inspect.aisi.org.uk/caching.html) to reduce the number of model API calls made.
 -   The `multiple_choice()` solver now has support for questions with multiple correct answers.
 -   More fine grained handling of Claude `BadRequestError` (400) errors (which were formerly all treated as content moderation errors).
 -   Filter out empty TextBlockParam when playing messages back to Claude.

{inspect_ai-0.3.73 → inspect_ai-0.3.75}/CITATION.cff

@@ -8,6 +8,6 @@ authors:
   - name: UK AI Security Institute
     website: "https://www.aisi.gov.uk/"
 repository-code: "https://github.com/UKGovernmentBEIS/inspect_ai"
-url: "https://inspect.ai-safety-institute.org.uk/"
+url: "https://inspect.aisi.org.uk/"
 license: MIT
 date-released: "2024-05-10"

{inspect_ai-0.3.73 → inspect_ai-0.3.75}/PKG-INFO

@@ -1,10 +1,10 @@
 Metadata-Version: 2.2
 Name: inspect_ai
-Version: 0.3.73
+Version: 0.3.75
 Summary: Framework for large language model evaluations
 Author: UK AI Security Institute
 License: MIT License
-Project-URL: Documentation, https://inspect.ai-safety-institute.org.uk/
+Project-URL: Documentation, https://inspect.aisi.org.uk/
 Project-URL: Source Code, https://github.com/UKGovernmentBEIS/inspect_ai
 Project-URL: Issue Tracker, https://github.com/UKGovernmentBEIS/inspect_ai/issues
 Classifier: Development Status :: 4 - Beta
@@ -97,13 +97,13 @@ Provides-Extra: dist
 Requires-Dist: twine; extra == "dist"
 Requires-Dist: build; extra == "dist"
 
-[<img width="295" src="https://inspect.ai-safety-institute.org.uk/images/aisi-logo.svg" />](https://aisi.gov.uk/)
+[<img width="295" src="https://inspect.aisi.org.uk/images/aisi-logo.svg" />](https://aisi.gov.uk/)
 
 Welcome to Inspect, a framework for large language model evaluations created by the [UK AI Security Institute](https://aisi.gov.uk/).
 
 Inspect provides many built-in components, including facilities for prompt engineering, tool usage, multi-turn dialog, and model graded evaluations. Extensions to Inspect (e.g. to support new elicitation and scoring techniques) can be provided by other Python packages.
 
-To get started with Inspect, please see the documentation at <https://inspect.ai-safety-institute.org.uk/>.
+To get started with Inspect, please see the documentation at <https://inspect.aisi.org.uk/>.
 
 ***
 

{inspect_ai-0.3.73 → inspect_ai-0.3.75}/README.md

@@ -1,10 +1,10 @@
-[<img width="295" src="https://inspect.ai-safety-institute.org.uk/images/aisi-logo.svg" />](https://aisi.gov.uk/)
+[<img width="295" src="https://inspect.aisi.org.uk/images/aisi-logo.svg" />](https://aisi.gov.uk/)
 
 Welcome to Inspect, a framework for large language model evaluations created by the [UK AI Security Institute](https://aisi.gov.uk/).
 
 Inspect provides many built-in components, including facilities for prompt engineering, tool usage, multi-turn dialog, and model graded evaluations. Extensions to Inspect (e.g. to support new elicitation and scoring techniques) can be provided by other Python packages.
 
-To get started with Inspect, please see the documentation at <https://inspect.ai-safety-institute.org.uk/>.
+To get started with Inspect, please see the documentation at <https://inspect.aisi.org.uk/>.
 
 ***
 

inspect_ai-0.3.75/docs/CNAME

@@ -0,0 +1 @@
+inspect.aisi.org.uk

{inspect_ai-0.3.73 → inspect_ai-0.3.75}/docs/_quarto.yml

@@ -20,7 +20,7 @@ website:
    bread-crumbs: true
    page-navigation: true
    repo-url: https://github.com/UKGovernmentBEIS/inspect_ai
-   site-url: https://inspect.ai-safety-institute.org.uk/
+   site-url: https://inspect.aisi.org.uk/
    repo-actions: [issue]
    twitter-card:
       title: "Inspect"

inspect_ai-0.3.75/docs/_variables.yml

@@ -0,0 +1 @@
+examples-url: https://inspect.aisi.org.uk/examples.html

{inspect_ai-0.3.73 → inspect_ai-0.3.75}/docs/extensions.qmd

@@ -55,7 +55,7 @@ The layer of indirection (creating a function that returns a ModelAPI class) is
 
 The `__init__()` method *must* call the `super().__init__()` method, and typically instantiates the model client library.
 
-The `__init__()` method receive a `**model_args` parameter that will carry any custom `model_args` (or `-M` and `--model-config` arguments from the CLI) specified by the user. You can then pass these on to the appropriate place in your model initialisation code (for example, here is what many of the built-in providers do with `model_args` passed to them: <https://inspect.ai-safety-institute.org.uk/models.html#model-args>).
+The `__init__()` method receive a `**model_args` parameter that will carry any custom `model_args` (or `-M` and `--model-config` arguments from the CLI) specified by the user. You can then pass these on to the appropriate place in your model initialisation code (for example, here is what many of the built-in providers do with `model_args` passed to them: <https://inspect.aisi.org.uk/models.html#model-args>).
 
 The `generate()` method handles interacting with the model, converting inspect messages, tools, and config into model native data structures. Note that the generate method may optionally return a `tuple[ModelOutput,ModelCall]` in order to record the raw request and response to the model within the sample transcript.
 

inspect_ai-0.3.75/docs/llms.txt

@@ -0,0 +1,63 @@
+# Inspect AI
+
+> Inspect AI is a Python framework for large language model evaluations created by the [UK AI Security Institute](https://aisi.gov.uk). Inspect provides many built-in components, including facilities for prompt engineering, tool usage, multi-turn dialog, and model graded evaluations. Extensions to Inspect (e.g. to support new elicitation and scoring techniques) can be provided by other Python packages.
+
+## Docs
+
+- [Tutorial](https://inspect.aisi.org.uk/tutorial.html.md): Step-by-step walkthroughs of several basic examples of Inspect evaluations.
+- [Options](https://inspect.aisi.org.uk/options.html.md): Covers the various options available for evaluations as well as how to manage model credentials.
+- [Log Viewer](https://inspect.aisi.org.uk/log-viewer.html.md): Goes into more depth on how to use Inspect View to develop and debug evaluations, including how to provide additional log metadata and how to integrate it with Python's standard logging module.
+- [VS Code](https://inspect.aisi.org.uk/vscode.html.md) Provides documentation on using the Inspect VS Code Extension to run, tune, debug, and visualise evaluations.
+- [Tasks](https://inspect.aisi.org.uk/tasks.html.md) bring together datasets, solvers, and scorers to define a evaluation. This section explores strategies for creating flexible and re-usable tasks.
+- [Datasets](https://inspect.aisi.org.uk/datasets.html.md): Datasets provide samples to evaluation tasks. This section illustrates how to adapt various data sources for use with Inspect, as well as how to include multi-modal data (images, etc.) in your datasets.
+- [Solvers](https://inspect.aisi.org.uk/solvers.html.md): Solvers are the heart of Inspect, and encompass prompt engineering and various other elicitation strategies. Here we cover using the built-in solvers and creating your own more sophisticated ones.
+- [Tools](https://inspect.aisi.org.uk/tools.html.md): Tools provide a means of extending the capabilities of models by registering Python functions for them to call. This section describes how to create custom tools and use them in evaluations.
+- [Scorers](https://inspect.aisi.org.uk/scorers.html.md): Scorers evaluate the work of solvers and aggregate scores into metrics. Sophisticated evals often require custom scorers that use models to evaluate output. This section covers how to create them.
+- [Models](https://inspect.aisi.org.uk/models.html.md): Models provide a uniform API for both evaluating a variety of large language models and using models within evaluations (e.g. for critique or grading).
+- [Providers](ttps://inspect.aisi.org.uk/providers.html.md) covers usage details and available options for the various supported providers.
+- [Caching](https://inspect.aisi.org.uk/caching.html.md): Caching enables you to cache model output to reduce the number of API calls made, saving both time and expense.
+- [Multimodal](https://inspect.aisi.org.uk/multimodal.html.md) This article describes how to use images, audio, and video in evaluations.
+- [Reasoning](https://inspect.aisi.org.uk/reasoning.html.md) documents the additional options and data available for reasoning models.
+- [Structured Output](https://inspect.aisi.org.uk/structured.html.md) explains how to constrain model output to a particular JSON schema.
+
+- [Agents](https://inspect.aisi.org.uk/agents.html.md): Agents combine planning, memory, and tool usage to pursue more complex, longer horizon tasks. This section describes how to build agent evaluations with Inspect.
+- [Sandboxing](https://inspect.aisi.org.uk/sandboxing.html.md): Enables you to isolate code generated by models as well as set up more complex computing environments for tasks. 
+- [Agent API](https://inspect.aisi.org.uk/agents-api.html.md): This article describes advanced Inspect APIs available for creating evaluations with agents.
+- [Agent Bridge](agent-bridge.qmd): Facility for integrating agents from 3rd party frameworks like AutoGen or LangChain.
+- [Human Agent](https://inspect.aisi.org.uk/human-agent.html.md): This article describes the `human_agent()` solver which enables human baselining for computing tasks.
+- [Approval](https://inspect.aisi.org.uk/approval.html.md): Approvals enable you to create fine-grained policies for approving tool calls made by models.
+- [Eval Logs](https://inspect.aisi.org.uk/eval-logs.html.md): Explores how to get the most out of evaluation logs for developing, debugging, and analyzing evaluations.
+- [Eval Sets](https://inspect.aisi.org.uk/eval-sets.html.md): Covers Inspect's features for describing, running, and analysing larger sets of evaluation tasks.
+- [Errors and Limits](https://inspect.aisi.org.uk/errors-and-limits.html.md): This article covers various techniques for dealing with unexpected errors and setting limits on evaluation tasks and samples, including retrying failed evaluations, establishing a threshold of samples to tolerate errors for before failing an evaluation, and setting a maximum number of messages, tokens, or elapsed seconds in a sample before forcing the solver to give up.
+- [Typing](https://inspect.aisi.org.uk/typing.html.md): Provides guideance on using static type checking with Inspect, including creating typed interfaces to untyped storage (i.e. sample metadata and store).
+- [Tracing](https://inspect.aisi.org.uk/tracing.html.md): Describes advanced execution tracing tools used to diagnose runtime issues.
+- [Parallelism](https://inspect.aisi.org.uk/parallelism.html.md): Delves into how to obtain maximum performance for evaluations. Inspect uses a highly parallel async architecture---here we cover how to tune this parallelism (e.g to stay under API rate limits or to not overburden local compute) for optimal throughput.
+- [Interactivity](https://inspect.aisi.org.uk/interactivity.html.md): Covers various ways to introduce user interaction into the implementation of tasks (for example, confirming consequential actions or prompting the model dynamically based on the trajectory of the evaluation).
+- [Extensions](https://inspect.aisi.org.uk/extensions.html.md) describes the various ways you can extend Inspect, including adding support for new Model APIs, tool execution environments, and storage platforms (for datasets, prompts, and logs).
+
+## Reference: Python API
+
+- [inspect_ai](https://inspect.aisi.org.uk/reference/inspect_ai.html.md) describes the core types used to create tasks and run evaluations.
+- [inspect_ai.solver](https://inspect.aisi.org.uk/reference/inspect_ai.solver.html.md) describes built in solvers as well as the types used to create custom solvers.
+- [inspect_ai.tool](https://inspect.aisi.org.uk/reference/inspect_ai.tool.html.md) describes built in tools as well as the types used to create custom tools.
+- [inspect_ai.scorer](https://inspect.aisi.org.uk/reference/inspect_ai.scorer.html.md) describes built in scorers as well as the types used to create custom scorers.
+- [inspect_ai.model](https://inspect.aisi.org.uk/reference/inspect_ai.model.html.md) covers using the Inspect model API for accessing various language models.
+- [inspect_ai.dataset](https://inspect.aisi.org.uk/reference/inspect_ai.dataset.html.md) describes the types used to read and manipulate datasets and samples.
+- [inspect_ai.approval](https://inspect.aisi.org.uk/reference/inspect_ai.approval.html.md) covers using built in approvers as well as the types used to create custom approvers and approval policies.
+- [inspect_ai.log](https://inspect.aisi.org.uk/reference/inspect_ai.log.html.md) describes the types used to list, read, write, and traverse the contents of eval log files.
+- [inspect_ai.util](https://inspect.aisi.org.uk/reference/inspect_ai.util.html.md) covers various utility functions for concurrency, sandboxes, the store, and more.
+
+## Reference: Command Line 
+
+- [inspect_eval](https://inspect.aisi.org.uk/reference/inspect_eval.html.md): Evaluate one or more tasks.
+- [inspect_eval-retry](https://inspect.aisi.org.uk/reference/inspect_eval-retry.html.md): Retry an evaluation task.
+- [inspect_eval-set](https://inspect.aisi.org.uk/reference/inspect_eval-set.html.md): Evaluate a set of tasks with retries.
+- [inspect_score](https://inspect.aisi.org.uk/reference/inspect_score.html.md): Score a previous evaluation run.
+- [inspect_view](https://inspect.aisi.org.uk/reference/inspect_view.html.md): Inspect log viewer.
+- [inspect_log](https://inspect.aisi.org.uk/reference/inspect_log.html.md): Query, read, write, and convert logs.
+- [inspect_trace](https://inspect.aisi.org.uk/reference/inspect_trace.html.md): List and read execution traces.
+- [inspect_sandbox](https://inspect.aisi.org.uk/reference/inspect_sandbox.html.md): Manage sandbox environments.
+- [inspect_cache](https://inspect.aisi.org.uk/reference/inspect_cache.html.md): Manage the Inspect model cache.
+- [inspect_list](https://inspect.aisi.org.uk/reference/inspect_list.html.md): List tasks on the filesystem.
+- [inspect_info](https://inspect.aisi.org.uk/reference/inspect_info.html.md): Read version and configuration.
+

{inspect_ai-0.3.73 → inspect_ai-0.3.75}/docs/models.qmd

@@ -32,6 +32,22 @@ Alternatively, you can set the `INSPECT_EVAL_MODEL` environment variable (either
 INSPECT_EVAL_MODEL=google/gemini-1.5-pro
 ```
 
+#### No Model
+
+Some evaluations will either not make use of models or call the lower-level `get_model()` function to explicitly access models for different roles (see the [Model API](#model-api) section below for details on this). 
+
+In these cases, you are not required to specify a `--model`. If you happen to have an `INSPECT_EVAL_MODEL` defined and you want to prevent your evaluation from using it, you can explcitly specify no model as follows:
+
+```bash
+inspect eval arc.py --model none
+```
+
+Or from Python:
+
+```python
+eval("arc.py", model=None)
+```
+
 ## Generation Config
 
 There are a variety of configuration options that affect the behaviour of model generation. There are options which affect the generated tokens (`temperature`, `top_p`, etc.) as well as the connection to model providers (`timeout`, `max_retries`, etc.)