inspect-ai 0.3.57__tar.gz → 0.3.58__tar.gz

{inspect_ai-0.3.57 → inspect_ai-0.3.58}/.pre-commit-config.yaml

@@ -5,7 +5,7 @@ default_language_version:
   python: python3.11
 repos:
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.9.0
+    rev: v0.9.1
     hooks:
       # Run the linter.
       - id: ruff

{inspect_ai-0.3.57 → inspect_ai-0.3.58}/CHANGELOG.md

@@ -1,5 +1,23 @@
 # Changelog
 
+## v0.3.58 (16 January 2025)
+
+- Support for [audio and video](https://github.com/UKGovernmentBEIS/inspect_ai/pull/1102) inputs for Open AI and Google Gemini models.
+- Task display: Added Timeout Tool button for manually timing out a tool call.
+- Task display: Automatically switch to "plain" mode when running in a background thread
+- Sandboxes: Setup and initialisation errors are now handled at the sample level.
+- Sandboxes: Increase setup script timeout to 5 minutes (from 30 seconds) and do not retry setup scripts (in case they aren't idempotent).
+- Sandboxes: Add `timeout_retry` option (defaulting to `True`) to `exec()` function.
+- Sandboxes: Add `type` and  optional `container` properties to `SandboxConnection`.
+- Docker: Services which exit with status 0 during setup no longer cause an error.
+- `task_with()` function for creating task variants.
+- Added `--filter` argument to trace CLI commands for filtering on trace log message content.
+- Print model conversations to terminal with `--display=conversation` (was formerly `--trace`, which is now deprecated).
+- HuggingFace: Support models that don't provide a chat template (e.g. gpt2)
+- Eval Set: Ensure that logs with status 'started' are retried.
+- Rename the built in `bootstrap_std` metric to `bootstrap_stderr` (deprecate `bootstrap_std`)
+- Bugfix: Fix duplication of summaries when eval log file is rewritten.
+
 ## v0.3.57 (09 January 2025)
 
 - [Tracing API](https://inspect.ai-safety-institute.org.uk/tracing.html#tracing-api) for custom trace logging.

{inspect_ai-0.3.57 → inspect_ai-0.3.58}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: inspect_ai
-Version: 0.3.57
+Version: 0.3.58
 Summary: Framework for large language model evaluations
 Author: UK AI Safety Institute
 License: MIT License
@@ -67,7 +67,7 @@ Requires-Dist: pytest-asyncio; extra == "dev"
 Requires-Dist: pytest-cov; extra == "dev"
 Requires-Dist: pytest-dotenv; extra == "dev"
 Requires-Dist: pytest-xdist; extra == "dev"
-Requires-Dist: ruff==0.9.0; extra == "dev"
+Requires-Dist: ruff==0.9.1; extra == "dev"
 Requires-Dist: textual-dev>=0.86.2; extra == "dev"
 Requires-Dist: types-PyYAML; extra == "dev"
 Requires-Dist: types-beautifulsoup4; extra == "dev"

{inspect_ai-0.3.57 → inspect_ai-0.3.58}/docs/_quarto.yml

@@ -64,6 +64,7 @@ website:
             - eval-sets.qmd
             - text: "Errors & Limits"
               href: errors-and-limits.qmd
+            - multimodal.qmd
             - typing.qmd
             - tracing.qmd
             - caching.qmd

{inspect_ai-0.3.57 → inspect_ai-0.3.58}/docs/_sandboxenv-interface.md

@@ -10,6 +10,7 @@ class SandboxEnvironment:
         env: dict[str, str] = {},
         user: str | None = None,
         timeout: int | None = None,
+        timeout_retry: bool = True
     ) -> ExecResult[str]:
         """
         Raises:
@@ -64,4 +65,6 @@ The `read_file()` method should should preserve newline constructs (e.g. crlf sh
 
 The `connection()` method is optional, and provides commands that can be used to login to the sandbox container from a terminal or IDE.
 
+Note that to deal with potential unreliability of container services, the `exec()` method includes a `timeout_retry` parameter that defaults to `True`. For sandbox implementations this parameter is _advisory_ (they should only use it if potential unreliablity exists in their runtime). No more than 2 retries should be attempted and both with timeouts less than 60 seconds. If you are executing commands that are not idempotent (i.e. the side effects of a failed first attempt may affect the results of subsequent attempts) then you can specify `timeout_retry=False` to override this behavior.
+
 For each method there is a documented set of errors that are raised: these are _expected_ errors and can either be caught by tools or allowed to propagate in which case they will be reported to the model for potential recovery. In addition, _unexpected_ errors may occur (e.g. a networking error connecting to a remote container): these errors are not reported to the model and fail the `Sample` with an error state. 

{inspect_ai-0.3.57 → inspect_ai-0.3.58}/docs/approval.qmd

@@ -19,10 +19,10 @@ Approvers can be specified at either the eval level or at the task level. The ex
 The simplest approval policy is interactive human approval of all tool calls. You can enable this policy by using the `--approval human` CLI option (or the `approval = "human"`) argument to `eval()`:
 
 ``` bash
-inspect eval browser.py --approval human --trace
+inspect eval browser.py --approval human
 ```
 
-Note that we also enable [trace mode](interactivity.qmd#sec-trace-mode) so that we can see all messages exchanged with the model for context. This example provides the model with the built-in [web browser](tools.qmd#sec-web-browser) tool and asks it to navigate to a web and perform a search.
+This example provides the model with the built-in [web browser](tools.qmd#sec-web-browser) tool and asks it to navigate to a web and perform a search.
 
 ## Auto Approver
 
@@ -44,7 +44,7 @@ Navigational web browser tool calls (e.g. `web_browser_go`) are approved automat
 To use this policy, pass the path to the policy YAML file as the approver. For example:
 
 ``` bash
-inspect eval browser.py --approval approval.yaml --trace
+inspect eval browser.py --approval approval.yaml
 ```
 
 ## Approvers in Code

{inspect_ai-0.3.57 → inspect_ai-0.3.58}/docs/datasets.qmd

@@ -66,6 +66,10 @@ Files are copied into the default sandbox environment unless their name contains
 "victim:/shared/flag.txt": "flag.txt"
 ```
 
+### Setup
+
+The `setup` field contains either a path to a bash setup script (resolved relative to the dataset path) or the contents of a script to execute. Setup scripts are executed with a 5 minute timeout. If you have setup scripts that may take longer than this you should move some of your setup code into the container build setup (e.g. Dockerfile).
+
 ## Field Mapping
 
 If your dataset contains inputs and targets that don't use `input` and `target` as field names, you can map them into a `Dataset` using a `FieldSpec`. This same mechanism also enables you to collect arbitrary additional fields into the `Sample` `metadata` bucket. For example:
@@ -188,7 +192,7 @@ The most important data structure within `Sample` is the `ChatMessage`. Note tha
 | Field | Type | Description |
 |-------------------|---------------------|--------------------------------|
 | `role` | `"system" | "user" | "assistant" | "tool"` | Role of this chat message. |
-| `content` | `str | list[ChatContent]` | The content of the message. Can be a simple string or a list of content parts intermixing text and images. |
+| `content` | `str | list[Content]` | The content of the message. Can be a simple string or a list of content parts intermixing text and images. |
 
 : {tbl-colwidths="\[10,35,55\]"}
 
@@ -205,40 +209,6 @@ An input with chat messages in your dataset might will look something like this:
 
 Note that for this example we wouldn't normally use a full chat message object (rather we'd just provide a simple string). Chat message objects are more useful when you want to include a system prompt or prime the conversation with "assistant" responses.
 
-## Image Input
-
-::: {.callout-note appearance="simple"}
-Image input is currently only supported for OpenAI vision models (e.g. [gpt-4-vision-preview](https://platform.openai.com/docs/guides/vision)), Google Gemini vision models (e.g. [gemini-pro-vision](https://console.cloud.google.com/vertex-ai/publishers/google/model-garden/gemini-pro-vision)), and Anthropic Claude 3 models.
-:::
-
-To include an image, your dataset input might look like this:
-
-``` javascript
-"input": [
-  {
-    "role": "user",
-    "content": [
-        { "type": "text", "text": "What is this a picture of?"},
-        { "type": "image", "image": "picture.png"}
-    ]
-  }
-]
-```
-
-Where `"picture.png"` is resolved relative to the directory containing the dataset file. The image can be specified either as a URL (accessible to the model), a local file path, or a base64 encoded [Data URL](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URLs).
-
-If you are constructing chat messages programmatically, then the equivalent to the above would be:
-
-``` python
-ChatMessageUser(content = [
-    ContentText(text="What is this a picture of?"),
-    ContentImage(image="picture.png")
-])
-```
-
-If you are using paths or URLs to images and want the full base64 encoded content of images included in log files, use the `--log-images` CLI flag (or `log_images` argument to `eval`). Note however that you should generally not do this if you have either large images or a large quantity of images, as this can substantially increase the size of the log file, making it difficult to load into Inspect View with reasonable performance.
-
-
 
 ## Custom Reader {#sec-custom-reader}
 
@@ -266,4 +236,4 @@ def security_guide():
     )
 ```
 
-So if the built in dataset functions don't meet your needs, you can create a custom function that yields a `MemoryDataset`and pass those directly to your `Task`.
+So if the built in dataset functions don't meet your needs, you can create a custom function that yields a `MemoryDataset`and pass those directly to your `Task`.

{inspect_ai-0.3.57 → inspect_ai-0.3.58}/docs/index.qmd

@@ -288,6 +288,8 @@ These sections discuss more advanced features and workflow. You don't need to re
 
 -   [Errors and Limits](errors-and-limits.qmd) covers various techniques for dealing with unexpected errors and setting limits on evaluation tasks and samples.
 
+-   [Multimodal](multimodal.qmd) documents the APIs available for creating multimodal evaluations (including images, audio, and video).
+
 -   [Typing](typing.qmd): provides guideance on using static type checking with Inspect, including creating typed interfaces to untyped storage (i.e. sample metadata and store).
 
 -   [Tracing](tracing.qmd) Describes advanced execution tracing tools used to diagnose runtime issues.

{inspect_ai-0.3.57 → inspect_ai-0.3.58}/docs/interactivity.qmd

@@ -60,11 +60,18 @@ with input_screen() as console:
 
 The `Prompt` class is designed to be subclassed for more specialized inputs. The `IntPrompt` and `FloatPrompt` classes are built-in, but you can also create your own more customised prompts (the `Confirm` class is another example of this). See the [prompt.py](https://github.com/Textualize/rich/blob/master/rich/prompt.py) source code for additional details.
 
-## Trace Mode {#sec-trace-mode}
+## Conversation Display {#sec-conversation-display}
 
-When introducing interactions it's often useful to see a trace of message activity for additional context. You can do this via the `--trace` CLI option (or `trace` parameter of the `eval()` function). For example:
+When introducing interactions it's often useful to see the full chat conversation printed for additional context. You can do this via the `--display=conversation` CLI option, for example:
+
+``` bash
+$ inspect eval theory.py --display conversation
+```
+
+In conversation display mode, all messages exchanged with the model are printed to the terminal (tool output is truncated at 100 lines).
+
+Note that enabling conversation display automatically sets `max_tasks` and `max_samples` to 1, as otherwise messages from concurrently running samples would be interleaved together in an incoherent jumble.
 
-{{< include _trace.md >}}
 
 
 ## Progress

{inspect_ai-0.3.57 → inspect_ai-0.3.58}/docs/llms.txt

@@ -21,6 +21,7 @@
 - [Eval Logs](https://inspect.ai-safety-institute.org.uk/eval-logs.html.md): Explores how to get the most out of evaluation logs for developing, debugging, and analyzing evaluations.
 - [Eval Sets](https://inspect.ai-safety-institute.org.uk/eval-sets.html.md): Covers Inspect's features for describing, running, and analysing larger sets of evaluation tasks.
 - [Errors and Limits](https://inspect.ai-safety-institute.org.uk/errors-and-limits.html.md): This article covers various techniques for dealing with unexpected errors and setting limits on evaluation tasks and samples, including retrying failed evaluations, establishing a threshold of samples to tolerate errors for before failing an evaluation, and setting a maximum number of messages, tokens, or elapsed seconds in a sample before forcing the solver to give up.
+- [Multimodal](https://inspect.ai-safety-institute.org.uk/multimodal.html.md) This article describes how to use images, audio, and video in evaluations.
 - [Typing](https://inspect.ai-safety-institute.org.uk/typing.html.md): Provides guideance on using static type checking with Inspect, including creating typed interfaces to untyped storage (i.e. sample metadata and store).
 - [Tracing](https://inspect.ai-safety-institute.org.uk/tracing.html.md): Describes advanced execution tracing tools used to diagnose runtime issues.
 - [Caching](https://inspect.ai-safety-institute.org.uk/caching.html.md): Caching enables you to cache model output to reduce the number of API calls made, saving both time and expense.

inspect_ai-0.3.58/docs/multimodal.qmd

@@ -0,0 +1,176 @@
+---
+title: Multimodal
+---
+
+## Overview
+
+Many models now support multimodal inputs, including images, audio, and video. This article describes how to how to create evaluations that include these data types.
+
+The following providers currently have support for multimodal inputs:
+
+| Provider  | Images | Audio | Video |
+|-----------|:------:|:-----:|:-----:|
+| OpenAI    |   •    |   •   |       |
+| Anthropic |   •    |       |       |
+| Google    |   •    |   •   |   •   |
+| Vertex    |   •    |   •   |       |
+| Mistral   |   •    |       |       |
+| Grok      |   •    |       |       |
+| Bedrock   |   •    |       |       |
+| AzureAI   |   •    |       |       |
+| Groq      |   •    |       |       |
+
+: {tbl-colwidths=\[40,20,20,20\]}
+
+Note that model providers only support multimodal inputs for a subset of their models. In the sections below on images, audio, and video we'll enumerate which models can handle these input types. It's also always a good idea to check the provider documentation for the most up to date compatiblity matrix.
+
+## Images {#provider-notes}
+
+The following models currently support image inputs:
+
+-   OpenAI: GPT-4o series and the full o1 model,
+-   Anthropic: Claude 3.5 Sonnet and all of the Claude 3 series models.
+-   Google/Vertex: Gemini 1.5 and Gemini 2.0 models.
+-   Mistral: Pixstral models (e.g. `pixtral-12b-2409`)
+-   Grok: Vision models (e.g. `grok-vision-beta`)
+
+For Bedrock, AzureAI, and Groq, please consult model provider documentation for details on which models support image inputs.
+
+To include an image in a [dataset](datasets.qmd) you should use JSON input format (either standard JSON or JSON Lines). For example, here we include an image alongside some text content:
+
+``` javascript
+"input": [
+  {
+    "role": "user",
+    "content": [
+        { "type": "image", "image": "picture.png"},
+        { "type": "text", "text": "What is this a picture of?"}
+    ]
+  }
+]
+```
+
+The `"picture.png"` path is resolved relative to the directory containing the dataset file. The image can be specified either as a file path or a base64 encoded [Data URL](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URLs).
+
+If you are constructing chat messages programmatically, then the equivalent to the above would be:
+
+``` python
+input = [
+    ChatMessageUser(content = [
+        ContentImage(image="picture.png"),
+        ContentText(text="What is this a picture of?")
+    ])
+]
+```
+
+### Detail
+
+Some providers support a `detail` option that control over how the model processes the image and generates its textual understanding. Valid options are `auto` (the default), `low`, and `high`. See the [Open AI documentation](https://platform.openai.com/docs/guides/vision#low-or-high-fidelity-image-understanding) for more information on using this option. The Mistral, AzureAI, and Groq APIs also support the `detail` parameter. For example, here we explicitly specify image detail:
+
+``` python
+ContentImage(image="picture.png", detail="low")
+```
+
+## Audio
+
+The following models currently support audio inputs:
+
+-   Open AI: `gpt-4o-audio-preview`
+-   Google/Vertex: Gemini 1.5 and 2.0 models
+
+To include audio in a [dataset](datasets.qmd) you should use JSON input format (either standard JSON or JSON Lines). For example, here we include audio alongside some text content:
+
+``` javascript
+"input": [
+  {
+    "role": "user",
+    "content": [
+        { "type": "audio", "audio": "sample.mp3", "format": "mp3" },
+        { "type": "text", "text": "What words are spoken in this audio sample?"}
+    ]
+  }
+]
+```
+
+The "sample.mp3" path is resolved relative to the directory containing the dataset file. The audio file can be specified either as a file path or a base64 encoded [Data URL](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URLs).
+
+If you are constructing chat messages programmatically, then the equivalent to the above would be:
+
+``` python
+input = [
+    ChatMessageUser(content = [
+        ContentAudio(audio="sample.mp3", format="mp3"),
+        ContentText(text="What words are spoken in this audio sample?")
+    ])
+]
+```
+
+### Formats
+
+You can provide audio files in one of two formats:
+
+-   MP3
+-   WAV
+
+As demonstrated above, you should specify the format explicitly when including audio input.
+
+## Video
+
+The following models currently support video inputs:
+
+-   Google: Gemini 1.5 and 2.0 models
+
+To include video in a [dataset](datasets.qmd) you should use JSON input format (either standard JSON or JSON Lines). For example, here we include video alongside some text content:
+
+``` javascript
+"input": [
+  {
+    "role": "user",
+    "content": [
+        { "type": "video", "video": "video.mp4", "format": "mp4" },
+        { "type": "text", "text": "Can you please describe the attached video?"}
+    ]
+  }
+]
+```
+
+The "video.mp4" path is resolved relative to the directory containing the dataset file. The video file can be specified either as a file path or a base64 encoded [Data URL](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URLs).
+
+If you are constructing chat messages programmatically, then the equivalent to the above would be:
+
+``` python
+input = [
+    ChatMessageUser(content = [
+        ContentVideo(video="video.mp4", format="mp4"),
+        ContentText(text="Can you please describe the attached video?")
+    ])
+]
+```
+
+### Formats
+
+You can provide video files in one of three formats:
+
+-   MP4
+-   MPEG
+-   MOV
+
+As demonstrated above, you should specify the format explicitly when including video input.
+
+## Uploads
+
+When using audio and video with the Google Gemini API, media is first uploaded using the [File API](https://ai.google.dev/gemini-api/docs/audio?lang=python#upload-audio) and then the URL to the uploaded file is referenced in the chat message. This results in much faster performance for subsequent uses of the media file.
+
+The File API lets you store up to 20GB of files per project, with a per-file maximum size of 2GB. Files are stored for 48 hours. They can be accessed in that period with your API key, but cannot be downloaded from the API. The File API is available at no cost in all regions where the Gemini API is available.
+
+## Logging
+
+By default, full base64 encoded copies of media files are included in the log file. Media file logging will not create performance problems when using `.eval` logs, however if you are using `.json` logs then large numbers of media files could become unwieldy (i.e. if your `.json` log file grows to 100MB or larger as a result).
+
+You can disable all media logging using the `--no-log-images` flag. For example, here we enable the `.json` log format and disable media logging:
+
+``` bash
+inspect eval images.py --log-format=json --no-log-images
+```
+
+You can also use the `INSPECT_EVAL_LOG_IMAGES` environment variable to set a global default in your `.env` configuration file.

{inspect_ai-0.3.57 → inspect_ai-0.3.58}/docs/scorers.qmd

@@ -72,7 +72,7 @@ Task(
 ```
 
 ::: {#stderr-note .callout-note}
-The current development version of Inspect replaces the use of the `bootstrap_std` metric with `stderr` for the built in scorers enumerated above.
+The current development version of Inspect replaces the use of the `bootstrap_stderr` metric with `stderr` for the built in scorers enumerated above.
 
 Since eval scores are means of numbers having finite variance, we can compute standard errors using the Central Limit Theorem rather than bootstrapping. Bootstrapping is generally useful in contexts with more complex structure or non-mean summary statistics (e.g. quantiles). You will notice that the bootstrap numbers will come in quite close to the analytic numbers, since they are estimating the same thing.
 
@@ -217,7 +217,7 @@ def includes(ignore_case: bool = True):
 1.  The function applies the `@scorer` decorator and registers two metrics for use with the scorer.
 2.  The `score()` function is declared as `async`. This is so that it can participate in Inspect's optimised scheduling for expensive model generation calls (this scorer doesn't call a model but others will).
 3.  We make use of the `text` property on the `Target`. This is a convenience property to get a simple text value out of the `Target` (as targets can technically be a list of strings).
-4.  We use the special constants `CORRECT` and `INCORRECT` for the score value (as the `accuracy()`, `stderr()`, and `bootstrap_std()` metrics know how to convert these special constants to float values (1.0 and 0.0 respectively).
+4.  We use the special constants `CORRECT` and `INCORRECT` for the score value (as the `accuracy()`, `stderr()`, and `bootstrap_stderr()` metrics know how to convert these special constants to float values (1.0 and 0.0 respectively).
 5.  We provide the full model completion as the answer for the score (`answer` is optional, but highly recommended as it is often useful to refer to during evaluation development).
 
 ### Example: Model Grading
@@ -514,7 +514,7 @@ Inspect includes some simple built in metrics for calculating accuracy, mean, et
 
     Standard error of the mean.
 
--   `bootstrap_std()`
+-   `bootstrap_stderr()`
 
     Standard deviation of a bootstrapped estimate of the mean. 1000 samples are taken by default (modify this using the `num_samples` option).
 

{inspect_ai-0.3.57 → inspect_ai-0.3.58}/docs/scripts/post-render.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-files=("index" "tutorial" "workflow" "log-viewer" "vscode" "solvers" "tools" "scorers" "datasets" "models" "agents" "sandboxing" "agents-api" "human-agent" "approval" "eval-logs" "eval-sets" "typing" "errors-and-limits" "tracing" "caching" "parallelism" "interactivity" "extensions")
+files=("index" "tutorial" "workflow" "log-viewer" "vscode" "solvers" "tools" "scorers" "datasets" "models" "agents" "sandboxing" "agents-api" "human-agent" "approval" "eval-logs" "eval-sets"  "errors-and-limits" "multimodal" "typing" "tracing" "caching" "parallelism" "interactivity" "extensions")
 
 
 if [ "$QUARTO_PROJECT_RENDER_ALL" = "1" ]; then

{inspect_ai-0.3.57 → inspect_ai-0.3.58}/docs/tracing.qmd

@@ -31,6 +31,12 @@ Trace logs are written using [JSON Lines](https://jsonlines.org/) format and are
 inspect trace dump trace-86396.log.gz
 ```
 
+You can also apply a filter to the trace file using the `--filter` argument (which will match log message text case insensitively). For example:
+
+``` bash
+inspect trace dump trace-86396.log.gz --filter model
+```
+
 ## Anomalies {#anomalies}
 
 If an evaluation is running and is not terminating, you can execute the following command to list instances of actions (e.g. model API generates, docker compose commands, tool calls, etc.) that are still running:
@@ -57,6 +63,12 @@ inspect trace anomalies --all
 
 Note that errors and timeouts are not by themselves evidence of problems, since both occur in the normal course of running evaluations (e.g. model generate calls can return errors that are retried and Docker or S3 can also return retryable errors or timeout when they are under heavy load).
 
+As with the `inspect trace dump` command, you can apply a filter when listing anomolies. For example:
+
+```bash
+inspect trace anomalies --filter model
+```
+
 ## Tracing API {#tracing-api}
 
 In addition to the standard set of actions which are trace logged, you can do your own custom trace logging using the `trace_action()` and `trace_message()` APIs. Trace logging is a great way to make sure that logging context is *always captured* (since the last 10 trace logs are always available) without cluttering up the console or eval transcripts.

{inspect_ai-0.3.57 → inspect_ai-0.3.58}/docs/workflow.qmd

@@ -208,11 +208,6 @@ Note that `.env` files are searched for in parent directories, so if you run an
 `.env` files should *never* be checked into version control, as they nearly always contain either secret API keys or machine specific paths. A best practice is often to check in an `.env.example` file to version control which provides an outline (e.g. keys only not values) of variables that are required by the current project.
 :::
 
-## Trace Mode
-
-In some cases during development of an evaluation you'll want to see message activity in realtime. You can do this via the `--trace` CLI option (or `trace` parameter of the `eval()` function). For example:
-
-{{< include _trace.md >}}
 
 ## Exploratory
 

{inspect_ai-0.3.57 → inspect_ai-0.3.58}/examples/approval/README.md

@@ -4,17 +4,11 @@
 
 This is a demonstrate of Inspect's [Approval Mode](https://inspect.ai-safety-institute.org.uk/approval.html) whereby tool calls are approved by a combination of approvers and approval policies.
 
-To run this demonstration, specify the approval configuration file and the `--trace` option to see model interactions printed to the terminal:
+To run this demonstration, specify the approval configuration file as follows:
 
 ``` bash
 cd examples/approval
-inspect eval approval.py --approval approval.yaml --trace
-```
-
-You can also run the approval script directly and it will run the task in the appropriate approval and trace modes:
-
-``` bash
-python3 examples/approval/approval.py
+inspect eval approval.py --approval approval.yaml
 ```
 
 ## Task Setup
@@ -49,7 +43,7 @@ def approval_demo() -> Task:
 We'll evaluate this task using the approval policy defined in `approval.yaml`:
 
 ```bash
-inspect eval approval.py --approval approval.yaml --trace
+inspect eval approval.py --approval approval.yaml
 ```
 
 Here is the approval configuration:

{inspect_ai-0.3.57 → inspect_ai-0.3.58}/examples/approval/approval.py

@@ -1,9 +1,8 @@
 import ast
 import shlex
-from pathlib import Path
 from typing import Set
 
-from inspect_ai import Task, eval, task
+from inspect_ai import Task, task
 from inspect_ai.approval import Approval, Approver, approver
 from inspect_ai.dataset import Sample
 from inspect_ai.solver import TaskState, generate, system_message, use_tools
@@ -220,8 +219,3 @@ def python_allowlist(
         return Approval(decision="approve", explanation="Python code is approved.")
 
     return approve
-
-
-if __name__ == "__main__":
-    approval = (Path(__file__).parent / "approval.yaml").as_posix()
-    eval(approval_demo(), approval=approval)

{inspect_ai-0.3.57 → inspect_ai-0.3.58}/examples/intervention/README.md

@@ -4,17 +4,12 @@
 
 This is a prototype of an Inspect agent with human intervention. It utilises Inspect's [Interactivity features](https://inspect.ai-safety-institute.org.uk/interactivity.html). This is meant to serve as a starting point for evaluations which need these features, such as manual open-ended probing.
 
-This gives an overview of this task and how to customise it. Note that this task is intended to be run within Inspect's approval mode (so you can confirm tool calls) and trace mode (so you can see the conversation with the model). For example:
+This gives an overview of this task and how to customise it. Note that this task is intended to be run within Inspect's approval mode (so you can confirm tool calls) and conversation display mode (so you can see the conversation with the model). For example:
 
 ``` bash
-inspect eval intervention.py --approval human --trace
+inspect eval exmaples/intervention.py --approval human --display conversation
 ```
 
-You can also run the intervention script directly and it will run the task in the appropriate approval and trace modes:
-
-``` bash
-python3 intervention.py
-```
 
 ## Inspect Code
 

{inspect_ai-0.3.57 → inspect_ai-0.3.58}/examples/intervention/intervention.py

@@ -2,7 +2,7 @@ from textwrap import dedent
 
 from rich.prompt import Prompt
 
-from inspect_ai import Task, eval, task
+from inspect_ai import Task, task
 from inspect_ai.model import ChatMessageUser
 from inspect_ai.solver import (
     Generate,
@@ -108,7 +108,3 @@ def ask_for_next_action() -> str:
             default="",
             console=console,
         )
-
-
-if __name__ == "__main__":
-    eval(intervention(), approval="human", trace=True)

{inspect_ai-0.3.57 → inspect_ai-0.3.58}/pyproject.toml

@@ -129,7 +129,7 @@ dev = [
     "pytest-cov",
     "pytest-dotenv",
     "pytest-xdist",
-    "ruff==0.9.0", # match version specified in .pre-commit-config.yaml
+    "ruff==0.9.1", # match version specified in .pre-commit-config.yaml
     "textual-dev>=0.86.2",
     "types-PyYAML",
     "types-beautifulsoup4",

{inspect_ai-0.3.57 → inspect_ai-0.3.58}/src/inspect_ai/__init__.py

@@ -7,7 +7,7 @@ from inspect_ai._eval.evalset import eval_set
 from inspect_ai._eval.list import list_tasks
 from inspect_ai._eval.registry import task
 from inspect_ai._eval.score import score, score_async
-from inspect_ai._eval.task import Epochs, Task, TaskInfo, Tasks
+from inspect_ai._eval.task import Epochs, Task, TaskInfo, Tasks, task_with
 from inspect_ai._util.constants import PKG_NAME
 from inspect_ai.solver._human_agent.agent import human_agent
 
@@ -29,4 +29,5 @@ __all__ = [
     "TaskInfo",
     "Tasks",
     "task",
+    "task_with",
 ]

{inspect_ai-0.3.57 → inspect_ai-0.3.58}/src/inspect_ai/_cli/common.py

@@ -17,7 +17,7 @@ class CommonOptions(TypedDict):
     log_level: str
     log_level_transcript: str
     log_dir: str
-    display: Literal["full", "rich", "plain", "none"]
+    display: Literal["full", "conversation", "rich", "plain", "none"]
     no_ansi: bool | None
     debug: bool
     debug_port: int
@@ -64,7 +64,9 @@ def common_options(func: Callable[..., Any]) -> Callable[..., click.Context]:
     )
     @click.option(
         "--display",
-        type=click.Choice(["full", "rich", "plain", "none"], case_sensitive=False),
+        type=click.Choice(
+            ["full", "conversation", "rich", "plain", "none"], case_sensitive=False
+        ),
         default=DEFAULT_DISPLAY,
         envvar="INSPECT_DISPLAY",
         help="Set the display type (defaults to 'full')",

{inspect_ai-0.3.57 → inspect_ai-0.3.58}/src/inspect_ai/_cli/eval.py

@@ -118,6 +118,7 @@ def eval_options(func: Callable[..., Any]) -> Callable[..., click.Context]:
         "--trace",
         type=bool,
         is_flag=True,
+        hidden=True,
         envvar="INSPECT_EVAL_TRACE",
         help="Trace message interactions with evaluated model to terminal.",
     )
@@ -886,6 +887,7 @@ def parse_comma_separated(value: str | None) -> list[str] | None:
     "--trace",
     type=bool,
     is_flag=True,
+    hidden=True,
     help="Trace message interactions with evaluated model to terminal.",
     envvar="INSPECT_EVAL_TRACE",
 )