inspect-ai 0.3.58__tar.gz → 0.3.59__tar.gz

{inspect_ai-0.3.58 → inspect_ai-0.3.59}/.pre-commit-config.yaml

@@ -5,7 +5,7 @@ default_language_version:
   python: python3.11
 repos:
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.9.1
+    rev: v0.9.2
     hooks:
       # Run the linter.
       - id: ruff

{inspect_ai-0.3.58 → inspect_ai-0.3.59}/CHANGELOG.md

@@ -1,8 +1,26 @@
 # Changelog
 
+## v0.3.59 (24 January 2025)
+
+- Beta version of [computer()](https://inspect.ai-safety-institute.org.uk/tools.html#sec-computer) tool which models with a computer desktop environment.
+- `user_message()` solver for appending parameterised user messages.
+- `prompt_template()`, `system_message()` and `user_message()` solver now also include the sample `store` in substitution parameters.
+- Limits: Enforce token and message limit at lower level (not longer required to check `state.completed` for limit enforcement).
+- Limits: Enforce [custom limits](https://inspect.ai-safety-institute.org.uk/errors-and-limits.html#custom-limit) for samples by raising `SampleLimitExceededError`.
+- Tasks: Optional ability for solvers to [yield scores](https://inspect.ai-safety-institute.org.uk/solvers.html#sec-scoring-in-solvers) for a task.
+- Model API: Log model calls that result in bad request errors.
+- Tools: `model_input` option that determines how tool call result content is played back to the model.
+- Tools: Don't attempt to marshall arguments of dynamic `ToolDef` with `**kwargs: Any` (just pass them through).
+- Log warning when a non-fatal sample error occurs (i.e. errors permitted by the `fail_on_error` option) 
+- Inspect View: allow filtering samples by compound expressions including multiple scorers. (thanks @andrei-apollo)
+- Inspect View: improve rendering performance and stability for the viewer when viewing very large eval logs or samples with a large number of steps.
+- Task display: Improved `plain` mode with periodic updates on progress, metrics, etc.
+- Google: Update to v0.8.4 of google-generativeai (py.typed support and removal of logprobs generation options)
+- Google: Support for string enums (e.g. `Literal["a", "b", "c"])`) in tool function declarations.
+
 ## v0.3.58 (16 January 2025)
 
-- Support for [audio and video](https://github.com/UKGovernmentBEIS/inspect_ai/pull/1102) inputs for Open AI and Google Gemini models.
+- Support for [audio and video](https://inspect.ai-safety-institute.org.uk/multimodal.html) inputs for Open AI and Google Gemini models.
 - Task display: Added Timeout Tool button for manually timing out a tool call.
 - Task display: Automatically switch to "plain" mode when running in a background thread
 - Sandboxes: Setup and initialisation errors are now handled at the sample level.

{inspect_ai-0.3.58 → inspect_ai-0.3.59}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: inspect_ai
-Version: 0.3.58
+Version: 0.3.59
 Summary: Framework for large language model evaluations
 Author: UK AI Safety Institute
 License: MIT License
@@ -67,7 +67,7 @@ Requires-Dist: pytest-asyncio; extra == "dev"
 Requires-Dist: pytest-cov; extra == "dev"
 Requires-Dist: pytest-dotenv; extra == "dev"
 Requires-Dist: pytest-xdist; extra == "dev"
-Requires-Dist: ruff==0.9.1; extra == "dev"
+Requires-Dist: ruff==0.9.2; extra == "dev"
 Requires-Dist: textual-dev>=0.86.2; extra == "dev"
 Requires-Dist: types-PyYAML; extra == "dev"
 Requires-Dist: types-beautifulsoup4; extra == "dev"

{inspect_ai-0.3.58 → inspect_ai-0.3.59}/docs/_tools-scaffold.md

@@ -52,6 +52,17 @@ if output.stop_reason == "model_length":
     # do something to recover from context window overflow
 ```
 
+If you have written a scaffold loop that continues calling the model even after it stops calling tools, there may be values of `stop_reason` that indicate that the the loop should terminate anyway (because the error will just keep repeating on subsequent calls to the model). For example, the [basic agent](agents.qmd#sec-basic-agent) checks for `stop_reason` and exits under the following conditions:
+
+```python
+# check for stop reasons that indicate we should terminate
+if state.output.stop_reason == "model_length":
+    transcript().info(
+        f"Agent terminated (reason: {state.output.stop_reason})"
+    )
+    break
+```
+
 Here are the possible values for `StopReason` :
 
 | Stop Reason | Description |

{inspect_ai-0.3.58 → inspect_ai-0.3.59}/docs/approval.qmd

@@ -33,13 +33,14 @@ You can chain to together the `human` and `auto` approvers in an *approval polic
 ``` yaml
 approvers:
   - name: human
-    tools: ["web_browser_click", "web_browser_type*"]
+    tools: ["web_browser_click", "web_browser_type"]
 
   - name: auto
     tools: "*"
 ```
 
-Navigational web browser tool calls (e.g. `web_browser_go`) are approved automatically via the catch-all `auto` approver at the end of the chain. Note that when listing an approver in a policy you indicate which tools it should handle using a glob or list of globs.
+
+Navigational web browser tool calls (e.g. `web_browser_go`) are approved automatically via the catch-all `auto` approver at the end of the chain. Note that when listing an approver in a policy you indicate which tools it should handle using a glob or list of globs. These globs are prefix matched so the `web_browser_type` glob matches both `web_browser_type` and `web_browser_type_submit`.
 
 To use this policy, pass the path to the policy YAML file as the approver. For example:
 
@@ -47,6 +48,24 @@ To use this policy, pass the path to the policy YAML file as the approver. For e
 inspect eval browser.py --approval approval.yaml
 ```
 
+You can also match on tool arguments (for tools that dispatch many action types). For example, here is an approval policy for the [Computer Tool](tools.qmd#sec-computer) which allows typing and mouse movement but requires approval for key combos (e.g. Enter or a shortcut) and typing:
+
+
+```{.yaml filename="approval.yaml"}
+approvers:
+  - name: human
+    tools:
+      - computer(action='key'
+      - computer(action='left_click'
+      - computer(action='middle_click'
+      - computer(action='double_click'
+
+  - name: auto
+    tools: "*"
+```
+
+Note that since this is a prefix match and there could be other arguments, we don't end the tool match pattern with a parentheses.
+
 ## Approvers in Code
 
 We've demonstrated configuring approvers via a YAML approval policy file—you can also provide a policy directly in code (useful if it needs to be more dynamic). Here's a pure Python version of the example from the previous section:
@@ -152,7 +171,7 @@ Assuming we have properly [registered our approver](extensions.qmd#sec-extension
 ``` yaml
 approvers:
   - name: evaltools/bash_allowlist
-    tools: "*bash*"
+    tools: "bash"
     allowed_commands: ["ls", "echo", "cat"]
 
   - name: human

{inspect_ai-0.3.58 → inspect_ai-0.3.59}/docs/errors-and-limits.qmd

@@ -126,6 +126,8 @@ def agent_loop(message_limit: int = 50):
         ...
 ```
 
+Message limits are checked whenever you call `generate()` on the main model being evaluated. The `message_limit` is comparted to the number of messages passed in `input` parameter to `generate()`.
+
 ### Token Limit
 
 Here we set a `token_limit` of 500K for each sample within a task:
@@ -163,16 +165,17 @@ def agent_loop(token_limit: int = (1024 * 500)) -> Solver:
 It's important to note that the `token_limit` is for all tokens used within the execution of a sample. If you want to limit the number of tokens that can be yielded from a single call to the model you should use the `max_tokens` generation option.
 :::
 
-### Limit Checking
+### Custom Limit
 
-How and when are sample limits checked? Time limits are handled automatically by the code that runs the sample. Message and token limits are checked automatically when you access the `completed` property of `TaskState`. For example, most agents will use `state.completed` as their main loop condition:
+When limits are exceeded, a `SampleLimitExceededError` is raised and caught by the main Inspect sample execution logic. If you want to create custom limit types, you can enforce them by raising a `SampleLimitExceededError` as follows:
 
-``` python
-while not state.completed:
-    # call model
-    output = await model.generate(state.messages, state.tools)
-    
-    ...
-```
+```python
+from inspect_ai.util import SampleLimitExceededError
 
-If you are writing an agent loop you should check `state.completed` so that message and token limits can be enforced. Library code that calls a series of solvers in succession should also check `state.completed` (note that this is done automatically by the `chain()` function that is used to compose together lists of solvers).
+raise SampleLimitExceededError(
+    "custom", 
+    value=value,
+    limit=limit,
+    message=f"A custom limit was exceeded: {value}"
+)
+```

{inspect_ai-0.3.58 → inspect_ai-0.3.59}/docs/models.qmd

@@ -4,7 +4,7 @@ title: Models
 
 ## Overview
 
-Inspect has built in support for a variety of language model API providers and can be extended to support arbitrary additions ones. Built-in model API providers, their dependencies, and environment variables required to use them are as follows:
+Inspect has built in support for a variety of language model API providers and can be extended to support arbitrary additional ones. Built-in model API providers, their dependencies, and environment variables required to use them are as follows:
 
 | Model API | Dependencies | Environment Variables |
 |---------------|--------------------|--------------------------------------|

{inspect_ai-0.3.58 → inspect_ai-0.3.59}/docs/solvers.qmd

@@ -14,11 +14,11 @@ Solvers are the heart of Inspect evaluations and can serve a wide variety of pur
 5.  Multi-turn dialog
 6.  Running an agent scaffold
 
-Tasks have a single top-level solver that defines an execution plan. This solver could be implemented with arbitrary Python code (calling the model as required) or could consist of a set of other solvers composed together. Solvers can therefore play two differnet roles: 
+Tasks have a single top-level solver that defines an execution plan. This solver could be implemented with arbitrary Python code (calling the model as required) or could consist of a set of other solvers composed together. Solvers can therefore play two differnet roles:
 
-1. _Composite_ specifications for task execution; and
+1.  *Composite* specifications for task execution; and
 
-2. _Components_ that can be chained together.
+2.  *Components* that can be chained together.
 
 ### Example
 
@@ -99,7 +99,7 @@ The `generate` function passed to solvers is a convenience function that takes a
 
 Here are what some of the built-in solvers do with the `TaskState`:
 
-1.  The `system_message()` solver inserts a system message into the chat history.
+1.  The `system_message()` and `user_message()` solvers insert messages into the chat history.
 
 2.  The `chain_of_thought()` solver takes the original user prompt and re-writes it to ask the model to use chain of thought reasoning to come up with its answer.
 
@@ -117,14 +117,19 @@ You can also imagine solvers that call other models to help come up with a bette
 ## Built-In Solvers
 
 Inspect has a number of built-in solvers, each of which can be customised in some fashion. Built in solvers can be imported from the `inspect_ai.solver` module. Below is a summary of these solvers. There is not (yet) reference documentation on these functions so the best way to learn about how they can be customised, etc. is to use the **Go to Definition** command in your source editor.
+-   `prompt_template()`
+
+    Modify the user prompt by substituting the current prompt into the `{prompt}` placeholder within the specified template. Also automatically substitutes any variables defined in sample `metadata` as well as any other custom named paramters passed in `params`.
+
 
 -   `system_message()`
 
-    Prepend role="system" `message` to the list of messages (will follow any other system messages it finds in the message stream). Also automatically substitutes any variables defined in sample `metadata` as well as any other custom named paramters passed in `params`.
+    Prepend role="system" `message` to the list of messages (will follow any other system messages it finds in the message stream). Also automatically substitutes any variables defined in sample `metadata` and `store`, as well as any other custom named paramters passed in `params`.
 
--   `prompt_template()`
+-   `user_message()`
+
+    Append role="user" `message` to the list of messages. Also automatically substitutes any variables defined in sample `metadata` and `store`, as well as any other custom named paramters passed in `params`.
 
-    Modify the user prompt by substituting the current prompt into the `{prompt}` placeholder within the specified template. Also automatically substitutes any variables defined in sample `metadata` as well as any other custom named paramters passed in `params`.
 
 -   `chain_of_thought()`
 
@@ -173,10 +178,10 @@ We'll present an example and then discuss the various options below (in most cas
 
 Below is a full example of reading a dataset for use with `multiple choice()` and using it in an evaluation task. The underlying data in `mmlu.csv` has the following form:
 
-| Question | A | B | C | D | Answer |
+| Question                                                                            | A   | B   | C   | D   | Answer |
 |------------|------------|------------|------------|------------|:----------:|
-| Find the degree for the given field extension Q(sqrt(2), sqrt(3), sqrt(18)) over Q. | 0 | 4 | 2 | 6 | B |
-| Let p = (1, 2, 5, 4)(2, 3) in S_5 . Find the index of \<p\> in S_5. | 8 | 2 | 24 | 120 | C |
+| Find the degree for the given field extension Q(sqrt(2), sqrt(3), sqrt(18)) over Q. | 0   | 4   | 2   | 6   |   B    |
+| Let p = (1, 2, 5, 4)(2, 3) in S_5 . Find the index of \<p\> in S_5.                 | 8   | 2   | 24  | 120 |   C    |
 
 : {tbl-colwidths=\[50,10,10,10,10,10\]}
 
@@ -217,13 +222,12 @@ We use the `record_to_sample()` function to read the `choices` along with the `t
 
 The following options are available for further customisation of the multiple choice solver:
 
-| Option | Description |
+| Option             | Description                                                                                                                                                                                                                                                                                                                                                                                               |
 |------------------------------------|------------------------------------|
-| `template` | Use `template` to provide an alternate prompt template (note that if you do this your template should handle prompting for `multiple_correct` directly if required). You can access the built in templates using the `MultipleChoiceTemplate` enum. |
-| `cot` | Whether the solver should perform chain-of-thought reasoning before answering (defaults to `False`). NOTE: this has no effect if you provide a custom template. |
-| `multiple_correct` | By default, multiple choice questions have a single correct answer. Set `multiple_correct=True` if your target has defined multiple correct answers (for example, a `target` of `["B", "C"]`). In this case the model is prompted to provide one or more answers, and the sample is scored correct only if each of these answers are provided.  NOTE: this has no effect if you provide a custom template. |
-| `shuffle` | If you specify `shuffle=True`, then the order of the answers presented to the model will be randomised (this may or may not affect results, depending on the nature of the questions and the model being evaluated). |
-
+| `template`         | Use `template` to provide an alternate prompt template (note that if you do this your template should handle prompting for `multiple_correct` directly if required). You can access the built in templates using the `MultipleChoiceTemplate` enum.                                                                                                                                                       |
+| `cot`              | Whether the solver should perform chain-of-thought reasoning before answering (defaults to `False`). NOTE: this has no effect if you provide a custom template.                                                                                                                                                                                                                                           |
+| `multiple_correct` | By default, multiple choice questions have a single correct answer. Set `multiple_correct=True` if your target has defined multiple correct answers (for example, a `target` of `["B", "C"]`). In this case the model is prompted to provide one or more answers, and the sample is scored correct only if each of these answers are provided. NOTE: this has no effect if you provide a custom template. |
+| `shuffle`          | If you specify `shuffle=True`, then the order of the answers presented to the model will be randomised (this may or may not affect results, depending on the nature of the questions and the model being evaluated).                                                                                                                                                                                      |
 
 : {tbl-colwidths=\[35,65\]}
 
@@ -260,40 +264,34 @@ Typically solvers can be customised with parameters (e.g. `template` for prompt
 
 Before presenting the examples we'll take a more in-depth look at the `TaskState` class. Task states consist of both lower level data members (e.g. `messages`, `output`) as well as a number of convenience properties. The core members of `TaskState` that are *modified* by solvers are `messages` / `user_prompt` and `output`:
 
-| Member | Type | Description |
+| Member        | Type                | Description                                                                                                                                                                               |
 |-------------------|-------------------|----------------------------------|
-| `messages` | list\[ChatMessage\] | Chat conversation history for sample. It is automatically appended to by the `generate()` solver, and is often manipulated by other solvers (e.g. for prompt engineering or elicitation). |
-| `user_prompt` | ChatMessageUser | Convenience property for accessing the first user message in the message history (commonly used for prompt engineering). |
-| `output` | ModelOutput | The 'final' model output once we've completed all solving. This field is automatically updated with the last "assistant" message by the `generate()` solver. |
+| `messages`    | list\[ChatMessage\] | Chat conversation history for sample. It is automatically appended to by the `generate()` solver, and is often manipulated by other solvers (e.g. for prompt engineering or elicitation). |
+| `user_prompt` | ChatMessageUser     | Convenience property for accessing the first user message in the message history (commonly used for prompt engineering).                                                                  |
+| `output`      | ModelOutput         | The 'final' model output once we've completed all solving. This field is automatically updated with the last "assistant" message by the `generate()` solver.                              |
 
 ::: {.callout-note appearance="simple"}
 Note that the `generate()` solver automatically updates both the `messages` and `output` fields. For very simple evaluations modifying the `user_prompt` and then calling `generate()` encompasses all of the required interaction with `TaskState`.
 :::
 
-There are two additional fields that solvers might modify (but they are typically for more advanced use cases):
-
-| Member | Type | Description |
-|-------------------|-------------------|----------------------------------|
-| `metadata` | dict | Original metadata from `Sample`, as well as any other custom metadata that solvers choose to write (typically used to coordinate between solvers and/or for custom logging). |
-| `completed` | bool | Solvers can set `completed = True` to cause the task to exit the sample immediately. |
-
-Sometimes its import to have access to the *original* prompt input for the task (as other solvers may have re-written or even removed it entirely). This is available using the `input` and `input_text` properties:
+Sometimes its important to have access to the *original* prompt input for the task (as other solvers may have re-written or even removed it entirely). This is available using the `input` and `input_text` properties:
 
-| Member | Type | Description |
+| Member       | Type                       | Description                                                                         |
 |-------------------|-------------------|----------------------------------|
-| `input` | str \| list\[ChatMessage\] | Original `Sample` input. |
-| `input_text` | str | Convenience function for accessing the initial input from the `Sample` as a string. |
+| `input`      | str \| list\[ChatMessage\] | Original `Sample` input.                                                            |
+| `input_text` | str                        | Convenience function for accessing the initial input from the `Sample` as a string. |
 
 There are several other fields used to provide contextual data from either the task sample or evaluation:
 
-| Member | Type | Description |
+| Member      | Type                | Description                                               |
 |-------------------|-------------------|----------------------------------|
-| `sample_id` | int \| str | Unique ID for sample. |
-| `epoch` | int | Epoch for sample. |
-| `choices` | list\[str\] \| None | Choices from sample (used only in multiple-choice evals). |
-| `model` | ModelName | Name of model currently being evaluated. |
+| `sample_id` | int \| str          | Unique ID for sample.                                     |
+| `epoch`     | int                 | Epoch for sample.                                         |
+| `metadata`  | dict                | Original metadata from `Sample`                           |
+| `choices`   | list\[str\] \| None | Choices from sample (used only in multiple-choice evals). |
+| `model`     | ModelName           | Name of model currently being evaluated.                  |
 
-Finally, task states also include available tools as well as guidance for the model on which tools to use (if you haven't yet encountered the concept of tool use in language models, don't worry about understanding these fields, the [Tools](tools.qmd) article provides a more in-depth treatment):
+Task states also include available tools as well as guidance for the model on which tools to use (if you haven't yet encountered the concept of tool use in language models, don't worry about understanding these fields, the [Tools](tools.qmd) article provides a more in-depth treatment):
 
 | Member        | Type         | Description                  |
 |---------------|--------------|------------------------------|
@@ -422,7 +420,31 @@ Note that calls to `generate()` (for both the critique model and the model being
 
 ### Scoring in Solvers {#sec-scoring-in-solvers}
 
-In some cases it is useful for a solver to score a task directly to assist in deciding whether or how to continue. You can do this using the `score()` function:
+Typically, solvers don't score samples but rather leave that to externally specified [scorers](scorers.qmd). However, in some cases it is more convenient to have solvers also do scoring (e.g. when there is high coupling between the solver and scoring). The following two task state fields can be used for scoring:
+
+| Member   | Type               | Description                  |
+|----------|--------------------|------------------------------|
+| `target` | Target             | Scoring target from `Sample` |
+| `scores` | dict\[str, Score\] | Optional scores.             |
+
+
+Here is a trivial example of the code that might be used to yield scores from a solver:
+
+``` python
+async def solve(state: TaskState, generate: Generate):
+    # ...perform solver work
+    
+    # score
+    correct = state.output.completion == state.target.text
+    state.scores = { "correct": Score(value=correct) }
+    return state
+```
+
+Note that scores yielded by a `Solver` are combined with scores from the normal scoring provided by the scorer(s) defined for a `Task`.
+
+### Intermediate Scoring
+
+In some cases it is useful for a solver to score a task directly to generate an intermediate score or assist in deciding whether or how to continue. You can do this using the `score()` function:
 
 ``` python
 from inspect_ai.scorer import score
@@ -463,4 +485,4 @@ Early termination might also occur if you specify the `message_limit` option and
 ``` python
 # could terminate early
 eval(my_task, message_limit = 10)
-```
+```

{inspect_ai-0.3.58 → inspect_ai-0.3.59}/docs/tools.qmd

@@ -6,7 +6,7 @@ title: Tools
 
 Many models now have the ability to interact with client-side Python functions in order to expand their capabilities. This enables you to equip models with your own set of custom tools so they can perform a wider variety of tasks.
 
-Inspect natively supports registering Python functions as tools and providing these tools to models that support them (currently OpenAI, Claude 3, Google Gemini, and Mistral). Inspect also includes several built-in tools ([bash](#sec-bash-and-python), [python](#sec-bash-and-python), and [web_search](#sec-web-search)).
+Inspect natively supports registering Python functions as tools and providing these tools to models that support them (currently OpenAI, Claude 3, Google Gemini, and Mistral). Inspect also includes several built-in tools ([bash](#sec-bash-and-python), [python](#sec-bash-and-python), [computer](#sec-computer), [web browser](#sec-web-browser), and [web_search](#sec-web-search)).
 
 ::: callout-note
 ### Tools and Agents
@@ -22,6 +22,8 @@ Inspect has several built-in tools, including:
 
 -   [Web Browser](#sec-web-browser), which provides the model with a headless Chromium web browser that supports navigation, history, and mouse/keyboard interactions.
 
+-   [Computer](#sec-computer), which provides the model with a desktop computer (viewed through screenshots) that supports mouse and keyboard interaction.
+
 -   [Web Search](#sec-web-search), which uses the Google Search API to execute and summarise web searches.
 
 If you are only interested in using the built-in tools, check out their respective documentation links above. To learn more about creating your own tools read on immediately below.
@@ -371,16 +373,16 @@ Note that unlike some other tool functions like `bash()`, the `web_browser()` fu
 
 If you review the transcripts of a sample with access to the web browser tool, you'll notice that there are several distinct tools made available for control of the web browser. These tools include:
 
-| Tool | Description |
+| Tool                                        | Description                                                                           |
 |------------------------------------|------------------------------------|
-| `web_browser_go(url)` | Navigate the web browser to a URL. |
-| `web_browser_click(element_id)` | Click an element on the page currently displayed by the web browser. |
-| `web_browser_type(element_id)` | Type text into an input on a web browser page. |
+| `web_browser_go(url)`                       | Navigate the web browser to a URL.                                                    |
+| `web_browser_click(element_id)`             | Click an element on the page currently displayed by the web browser.                  |
+| `web_browser_type(element_id)`              | Type text into an input on a web browser page.                                        |
 | `web_browser_type_submit(element_id, text)` | Type text into a form input on a web browser page and press ENTER to submit the form. |
-| `web_browser_scroll(direction)` | Scroll the web browser up or down by one page. |
-| `web_browser_forward()` | Navigate the web browser forward in the browser history. |
-| `web_browser_back()` | Navigate the web browser back in the browser history. |
-| `web_browser_refresh()` | Refresh the current page of the web browser. |
+| `web_browser_scroll(direction)`             | Scroll the web browser up or down by one page.                                        |
+| `web_browser_forward()`                     | Navigate the web browser forward in the browser history.                              |
+| `web_browser_back()`                        | Navigate the web browser back in the browser history.                                 |
+| `web_browser_refresh()`                     | Refresh the current page of the web browser.                                          |
 
 : {tbl-colwidths=\[35,65\]}
 
@@ -420,6 +422,171 @@ CMD ["python3", "/app/web_browser/web_server.py"]
 
 Note that all of the Python files in the [\_resources](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/tool/_tools/_web_browser/_resources/) directory alongside the `Dockerfile` need to be available for copying when building the container.
 
+## Computer (Beta) {#sec-computer}
+
+
+The `computer()` tool provides models with a computer desktop environment along with the ability to view the screen and perform mouse and keyboard gestures. The computer tool is based on the Anthropic [Computer Use Beta](https://docs.anthropic.com/en/docs/build-with-claude/computer-use) reference implementation and works with any model that supports image input.
+
+The current release of the computer tool is a beta version (exported from the `inspect_ai.tool.beta` module). We expect to finalise the interface and move it into the main `inspect_ai.tool` module over the next several weeks.
+
+### Configuration
+
+The `computer()` tool runs within a Docker container. To use it with a task you need to reference the `aisiuk/inspect-computer-tool:latest-beta` image in your Docker compose file. For example:
+
+``` {.yaml filename="compose.yaml"}
+services:
+  default:
+    image: aisiuk/inspect-computer-tool:latest-beta
+```
+
+You can configure the container to not have Internet access as follows:
+
+``` {.yaml filename="compose.yaml"}
+services:
+  default:
+    image: aisiuk/inspect-computer-tool:latest-beta
+    network_mode: none
+```
+
+Note that if you'd like to be able to view the model's interactions with the computer desktop in realtime, you will need to also do some port mapping to enable a VNC connection with the container. See the [VNC Client](#vnc-client) section below for details on how to do this.
+
+The `aisiuk/inspect-computer-tool:latest-beta` image is based on the [ubuntu:22.04](https://hub.docker.com/layers/library/ubuntu/22.04/images/sha256-965fbcae990b0467ed5657caceaec165018ef44a4d2d46c7cdea80a9dff0d1ea?context=explore) image and includes the following additional applications pre-installed:
+
+-   Firefox
+-   VS Code
+-   Xpdf
+-   Xpaint
+-   galculator
+
+### Task Setup
+
+A task configured to use the computer tool might look like this:
+
+``` python
+from inspect_ai import Task, task
+from inspect_ai.scorer import match
+from inspect_ai.solver import generate, use_tools
+from inspect_ai.tool.beta import computer
+
+@task
+def computer_task():
+    return Task(
+        dataset=read_dataset(),
+        solver=[
+            use_tools([computer()]),
+            generate(),
+        ],
+        scorer=match(),
+        sandbox=("docker", "compose.yaml"),
+    )
+```
+
+#### Options
+
+The computer tool supports the following options:
+
+| Option            | Description                                                                                                           |
+|-------------------|-----------------------------------------------------------------------------------------------------------------------|
+| `max_screenshots` | The maximum number of screenshots to play back to the model as input. Defaults to 1 (set to `None` to have no limit). |
+| `timeout`         | Timeout in seconds for computer tool actions. Defaults to 180 (set to `None` for no timeout).                         |
+
+For example:
+
+``` python
+solver=[
+    use_tools([computer(max_screenshots=2, timeout=300)]),
+    generate()
+]
+```
+
+#### Examples
+
+Two of the Inspect examples demonstrate basic computer use:
+
+-   [computer](https://github.com/UKGovernmentBEIS/inspect_ai/tree/main/examples/computer/computer.py) — Three simple computing tasks as a minimal demonstration of computer use.
+
+    ``` bash
+    inspect eval examples/computer
+    ```
+
+-   [intervention](https://github.com/UKGovernmentBEIS/inspect_ai/tree/main/examples/intervention/intervention.py) — Computer task driven interactively by a human operator.
+
+    ``` bash
+    inspect eval examples/intervention -T mode=computer --display conversation
+    ```
+
+### VNC Client {#vnc-client}
+
+You can use a [VNC](https://en.wikipedia.org/wiki/VNC) connection to the container to watch computer use in real-time. This requires some additional port-mapping in the Docker compose file. You can define dynamic port ranges for VNC (5900) and a browser based noVNC client (6080) with the following `ports` entries:
+
+``` {.yaml filename="compose.yaml"}
+services:
+  default:
+    image: aisiuk/inspect-computer-tool:latest-beta
+    ports:
+      - "5900"
+      - "6080"
+```
+
+To connect to the container for a given sample, locate the sample in the **Running Samples** UI and expand the sample info panel at the top:
+
+![](images/vnc-port-info.png){.lightbox width="958"}
+
+Click on the link for the noVNC browser client, or use a native VNC client to connect to the VNC port. Note that the VNC server will take a few seconds to start up so you should give it some time and attempt to reconnect as required if the first connection fails.
+
+The browser based client provides a view-only interface. If you use a native VNC client you should also set it to "view only" so as to not interfere with the model's use of the computer. For example, for Real VNC Viewer:
+
+![](images/vnc-view-only.png){width="549"}
+
+### Approval
+
+If the container you are using is connected to the Internet, you may want to configure human approval for a subset of computer tool actions. Here are the possible actions (specified using the `action` parameter to the `computer` tool):
+
+-   `key`: Press a key or key-combination on the keyboard.
+-   `type`: Type a string of text on the keyboard.
+-   `cursor_position`: Get the current (x, y) pixel coordinate of the cursor on the screen.
+-   `mouse_move`: Move the cursor to a specified (x, y) pixel coordinate on the screen.
+-   Example: execute(action="mouse_move", coordinate=(100, 200))
+-   `left_click`: Click the left mouse button.
+-   `left_click_drag`: Click and drag the cursor to a specified (x, y) pixel coordinate on the screen.
+-   `right_click`: Click the right mouse button.
+-   `middle_click`: Click the middle mouse button.
+-   `double_click`: Double-click the left mouse button.
+-   `screenshot`: Take a screenshot.
+
+Here is an approval policy that requires approval for key combos (e.g. `Enter` or a shortcut) and mouse clicks:
+
+``` {.yaml filename="approval.yaml"}
+approvers:
+  - name: human
+    tools:
+      - computer(action='key'
+      - computer(action='left_click'
+      - computer(action='middle_click'
+      - computer(action='double_click'
+
+  - name: auto
+    tools: "*"
+```
+
+Note that since this is a prefix match and there could be other arguments, we don't end the tool match pattern with a parentheses.
+
+You can apply this policy using the `--approval` commmand line option:
+
+``` bash
+inspect eval computer.py --approval approval.yaml
+```
+
+### Tool Binding
+
+The computer tool's schema is based on the standard Anthropoic [computer tool-type](https://docs.anthropic.com/en/docs/build-with-claude/computer-use#computer-tool). When using Claude 3.5 the coputer tool will automatically bind to the native Claude computer tool definition. This presumably provides improved performance due to fine tuning on the use of the tool but we have not verified this.
+
+If you want to experiement with bypassing the native Claude computer tool type and just register the computer tool as a normal function based tool then specify the `--no-internal-tools` generation option as follows:
+
+``` bash
+inspect eval computer.py --no-internal-tools
+```
+
 ## Web Search {#sec-web-search}
 
 The `web_search()` tool provides models the ability to enhance their context window by performing a search. By default web searches retrieve 10 results from a provider, uses a model to determine if the contents is relevant then returns the top 3 relevant search results to the main model. Here is the definition of the `web_search()` function:
@@ -464,4 +631,4 @@ The `web_search()` tool uses [Google Programmable Search Engine](https://program
 
 -   `GOOGLE_CSE_ID` — Google Custom Search Engine ID
 
--   `GOOGLE_CSE_API_KEY` — Google API key used to enable the Search API
+-   `GOOGLE_CSE_API_KEY` — Google API key used to enable the Search API

inspect_ai-0.3.59/examples/computer/compose.yaml

@@ -0,0 +1,17 @@
+services:
+  default:
+    image: aisiuk/inspect-computer-tool:latest-beta
+    init: true
+
+    # You can vnc into each container by using the following port mapping which will 
+    # dynamically bind to host ports. The specific bindings can be found by using 
+    # `docker inspect <container_id_or_name>`. This info is included in Running Samples
+    # tab. The output will look something like:
+    #
+    #  service   container   host     url
+    #  VNC       5900        61029    vnc://localhost:61029
+    #  noVNC     6080        61030    http://localhost:61030?view_only=true&autoconnect=true
+
+    ports:
+      - "5900"
+      - "6080"