inspect-ai 0.3.75__tar.gz → 0.3.76__tar.gz

{inspect_ai-0.3.75 → inspect_ai-0.3.76}/.github/workflows/docker.yml

@@ -10,6 +10,7 @@ on:
         options:
           - inspect-computer-tool
           - inspect-web-browser-tool
+          - inspect-tool-support
       tag:
         description: Tag to assign to the image
         required: true
@@ -44,6 +45,8 @@ jobs:
             echo "BUILD_CONTEXT=src/inspect_ai/tool/_tools/_computer/_resources" >> $GITHUB_ENV
           elif [[ "${{ github.event.inputs.image }}" == "inspect-web-browser-tool" ]]; then
             echo "BUILD_CONTEXT=src/inspect_ai/tool/_tools/_web_browser/_resources" >> $GITHUB_ENV
+          elif [[ "${{ github.event.inputs.image }}" == "inspect-tool-support" ]]; then
+            echo "BUILD_CONTEXT=src/inspect_tool_support" >> $GITHUB_ENV
           else
             echo "Invalid image name '${{ github.event.inputs.image }}'"
             exit 1

inspect_ai-0.3.76/.vscode/launch.json

@@ -0,0 +1,24 @@
+{
+  // Use IntelliSense to learn about possible attributes.
+  // Hover to view descriptions of existing attributes.
+  // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
+  "version": "0.2.0",
+  "configurations": [
+    {
+      "name": "Python Debugger: Module",
+      "type": "debugpy",
+      "request": "launch",
+      "module": "inspect_ai",
+      "args": "${input:argsPrompt}",
+      "cwd": "${workspaceFolder}/src"
+    }
+  ],
+  "inputs": [
+    {
+      "id": "argsPrompt",
+      "type": "promptString",
+      "description": "Enter arguments for the module",
+      "default": ""
+    }
+  ]
+}

{inspect_ai-0.3.75 → inspect_ai-0.3.76}/CHANGELOG.md

@@ -1,3 +1,19 @@
+## v0.3.76 (23 March 2025)
+
+- [bash_session()](https://inspect.ai-safety-institute.org.uk/tools-standard.html#sec-bash-session) tool for creating a stateful bash shell that retains its state across calls from the model.
+- [text_editor()](https://inspect.ai-safety-institute.org.uk/tools-standard.html#sec-text-editor) tool which enables viewing, creating and editing text files.
+- Structured Output: Properly handle Pydantic BaseModel that contains other BaseModel definitions in its schema.
+- OpenAI: Support for .wav files in audio inputs for gpt-4o-audio-preview.
+- OpenAI: Strip 'azure' prefix from model_name so that model type checks all work correctly.
+- OpenAI: Don't send `reasoning_effort` parameter to o1-preview (as it is not supported).
+- Inspect View: Fix error sorting numeric or categorical score results.
+- Inspect View: Properly wrap model API call text in the transcript.
+- Bugfix: Only initialise display in eval_set if it wasn't initialised from the CLI
+- Bugfix: Set the global log level based on the specified Inspect log level.
+- Bugfix: Resolve issue when deserialising a SubtaskEvent from a log file which does not have a completed time.
+- Bugfix: Fix unnecessary warnings about task arguments.
+- Bugfix: When a task does not take a kwargs argument, only warn if the provided argument is not valid.
+
 ## v0.3.75 (18 March 2025)
 
 - Model API: Specifying a default model (e.g. `--model`) is no longer required (as some evals have no model or use `get_model()` for model access).
@@ -210,7 +226,7 @@
 
 ## v0.3.59 (24 January 2025)
 
-- Beta version of [computer()](https://inspect.aisi.org.uk/tools.html#sec-computer) tool which models with a computer desktop environment.
+- Beta version of [computer()](https://inspect.aisi.org.uk/tools-standard.html#sec-computer) tool which models with a computer desktop environment.
 - `user_message()` solver for appending parameterised user messages.
 - `prompt_template()`, `system_message()` and `user_message()` solver now also include the sample `store` in substitution parameters.
 - Limits: Enforce token and message limit at lower level (not longer required to check `state.completed` for limit enforcement).
@@ -454,7 +470,7 @@
 
 ## v0.3.42 (23 October 2024)
 
-- [ToolDef](https://inspect.aisi.org.uk/tools.html#sec-dynamic-tools) class for dynamically creating tool definitions.
+- [ToolDef](https://inspect.aisi.org.uk/tools-custom.html#sec-dynamic-tools) class for dynamically creating tool definitions.
 - Added `--tags` option to eval for tagging evaluation runs.
 - Added APIs for accessing sample event transcripts and for creating and resolving attachments for larger content items.
 - Cleanup Docker Containers immediately for samples with errors.
@@ -536,7 +552,7 @@
 
 ## v0.3.36 (2 October 2024)
 
-- [Web Browser](https://inspect.aisi.org.uk/tools.html#sec-web-browser) tool which provides a headless Chromium browser that supports navigation, history, and mouse/keyboard interactions.
+- [Web Browser](https://inspect.aisi.org.uk/tools-standard.html#sec-web-browser) tool which provides a headless Chromium browser that supports navigation, history, and mouse/keyboard interactions.
 - `auto_id` option for dataset readers to assign an auto-incrementing ID to records.
 - Task args: don't attempt to serialise registry objects that don't have captured parameters.
 

{inspect_ai-0.3.75 → inspect_ai-0.3.76}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.2
+Metadata-Version: 2.4
 Name: inspect_ai
-Version: 0.3.75
+Version: 0.3.76
 Summary: Framework for large language model evaluations
 Author: UK AI Security Institute
 License: MIT License
@@ -60,11 +60,13 @@ Requires-Dist: goodfire; extra == "dev"
 Requires-Dist: griffe; extra == "dev"
 Requires-Dist: groq; extra == "dev"
 Requires-Dist: ipython; extra == "dev"
+Requires-Dist: markdown; extra == "dev"
 Requires-Dist: mistralai; extra == "dev"
 Requires-Dist: moto[server]; extra == "dev"
 Requires-Dist: mypy; extra == "dev"
 Requires-Dist: nbformat; extra == "dev"
 Requires-Dist: openai; extra == "dev"
+Requires-Dist: panflute; extra == "dev"
 Requires-Dist: pip; extra == "dev"
 Requires-Dist: pre-commit; extra == "dev"
 Requires-Dist: pylint; extra == "dev"
@@ -96,6 +98,7 @@ Requires-Dist: griffe; extra == "doc"
 Provides-Extra: dist
 Requires-Dist: twine; extra == "dist"
 Requires-Dist: build; extra == "dist"
+Dynamic: license-file
 
 [<img width="295" src="https://inspect.aisi.org.uk/images/aisi-logo.svg" />](https://aisi.gov.uk/)
 

{inspect_ai-0.3.75 → inspect_ai-0.3.76}/docs/_quarto.yml

@@ -70,6 +70,12 @@ website:
                - tools.qmd
                - scorers.qmd
 
+         - section: "Tools"
+           contents:
+               - tools.qmd
+               - tools-custom.qmd
+               - tools-standard.qmd
+
          - section: "Models"
            contents:
                - models.qmd

inspect_ai-0.3.76/docs/_sandbox-dockerfile.md

@@ -0,0 +1,9 @@
+You should add the following to your sandbox `Dockerfile` in order to use this tool:
+
+``` dockerfile
+RUN python -m venv /opt/inspect_tool_support
+ENV PATH="/opt/inspect_tool_support/bin:$PATH"
+
+RUN pip install inspect-tool-support
+RUN inspect-tool-support post-install
+```

inspect_ai-0.3.76/docs/_sandbox-image.md

@@ -0,0 +1,10 @@
+
+If you don't have a custom Dockerfile, you can alternatively use the pre-built `aisiuk/inspect-tool-support` image:
+
+``` {.yaml filename="compose.yaml"}
+services:
+  default:
+    image: aisiuk/inspect-tool-support:latest
+    init: true
+```
+

inspect_ai-0.3.76/docs/_tools-basics.md

@@ -0,0 +1,57 @@
+
+Here's a simple tool that adds two numbers. The `@tool` decorator is used to register it with the system:
+
+``` python
+from inspect_ai.tool import tool
+
+@tool
+def add():
+    async def execute(x: int, y: int):
+        """
+        Add two numbers.
+
+        Args:
+            x: First number to add.
+            y: Second number to add.
+
+        Returns:
+            The sum of the two numbers.
+        """
+        return x + y
+
+    return execute
+```
+
+### Annotations
+
+{{< include _tools-annotations-required.md >}}
+
+Note that you while you are required to provide default descriptions for tools and their parameters within doc comments, you can also make these dynamically customisable by users of your tool (see the section on [Tool Descriptions](tools-custom.qmd#sec-tool-descriptions) for details on how to do this).
+
+## Using Tools
+
+We can use the `addition()` tool in an evaluation by passing it to the `use_tools()` Solver:
+
+``` python
+from inspect_ai import Task, task
+from inspect_ai.dataset ipmort Sample
+from inspect_ai.solver import generate, use_tools
+from inspect_ai.scorer import match
+
+@task
+def addition_problem():
+    return Task(
+        dataset=[Sample(input="What is 1 + 1?", target=["2"])],
+        solver=[
+            use_tools(add()), 
+            generate()
+        ],
+        scorer=match(numeric=True),
+    )
+```
+
+Note that this tool doesn't make network requests or do heavy computation, so is fine to run as inline Python code. If your tool does do more elaborate things, you'll want to make sure it plays well with Inspect's concurrency scheme. For network requests, this amounts to using `async` HTTP calls with `httpx`. For heavier computation, tools should use subprocesses as described in the next section.
+
+::: {.callout-note appearance="simple"}
+Note that when using tools with models, the models do not call the Python function directly. Rather, the model generates a structured request which includes function parameters, and then Inspect calls the function and returns the result to the model.
+:::

inspect_ai-0.3.76/docs/_tools-configuration.md

@@ -0,0 +1,17 @@
+
+
+::: {.callout-note appearance="simple" collapse="true"}
+### Sandbox Dependencies
+
+{{< include _sandbox-dockerfile.md >}}
+
+Note that Playwright (used for the `web_browser()` tool) does not support some versions of Linux (e.g. Kali Linux). If this is the case for your Linux distribution, you should add the `--no-web-browser` option to the `post-install`:
+
+```Dockerfile
+RUN inspect-tool-support post-install --no-web-browser
+```
+
+{{< include _sandbox-image.md >}}
+
+:::
+

inspect_ai-0.3.76/docs/_tools-standard.md

@@ -0,0 +1,13 @@
+Inspect has several standard tools built-in, including:
+
+-   [Bash and Python](tools-standard.qmd#sec-bash-and-python) for executing arbitrary shell and Python code.
+
+-   [Bash Session](tools-standard.qmd#sec-bash-session) for creating a stateful bash shell that retains its state across calls from the model.
+
+-   [Text Editor](tools-standard.qmd#sec-text-editor) which enables viewing, creating and editing text files.
+
+-   [Web Browser](tools-standard.qmd#sec-web-browser), which provides the model with a headless Chromium web browser that supports navigation, history, and mouse/keyboard interactions.
+
+-   [Computer](tools-standard.qmd#sec-computer), which provides the model with a desktop computer (viewed through screenshots) that supports mouse and keyboard interaction.
+
+-   [Web Search](tools-standard.qmd#sec-web-search), which uses the Google Search API to execute and summarise web searches.

{inspect_ai-0.3.75 → inspect_ai-0.3.76}/docs/approval.qmd

@@ -22,7 +22,7 @@ The simplest approval policy is interactive human approval of all tool calls. Yo
 inspect eval browser.py --approval human
 ```
 
-This example provides the model with the built-in [web browser](tools.qmd#sec-web-browser) tool and asks it to navigate to a web and perform a search.
+This example provides the model with the built-in [web browser](tools-standard.qmd#sec-web-browser) tool and asks it to navigate to a web and perform a search.
 
 ## Auto Approver
 
@@ -48,7 +48,7 @@ To use this policy, pass the path to the policy YAML file as the approver. For e
 inspect eval browser.py --approval approval.yaml
 ```
 
-You can also match on tool arguments (for tools that dispatch many action types). For example, here is an approval policy for the [Computer Tool](tools.qmd#sec-computer) which allows typing and mouse movement but requires approval for key combos (e.g. Enter or a shortcut) and typing:
+You can also match on tool arguments (for tools that dispatch many action types). For example, here is an approval policy for the [Computer Tool](tools-standard.qmd#sec-computer) which allows typing and mouse movement but requires approval for key combos (e.g. Enter or a shortcut) and typing:
 
 
 ```{.yaml filename="approval.yaml"}

{inspect_ai-0.3.75 → inspect_ai-0.3.76}/docs/index.qmd

@@ -195,10 +195,17 @@ These sections provide a more in depth treatment of the various components used
 
 -   [Solvers](solvers.qmd) are the heart of Inspect, and encompass prompt engineering and various other elicitation strategies (the `plan` in the example above). Here we cover using the built-in solvers and creating your own more sophisticated ones.
 
--   [Tools](tools.qmd) provide a means of extending the capabilities of models by registering Python functions for them to call. This section describes how to create custom tools and use them in evaluations.
-
 -   [Scorers](scorers.qmd) evaluate the work of solvers and aggregate scores into metrics. Sophisticated evals often require custom scorers that use models to evaluate output. This section covers how to create them.
 
+These sections cover defining custom tools as well as Inspect's standard built-in tools:
+
+- [Tool Basics](tools.qmd): Tools provide a means of extending the capabilities of models by registering Python functions for them to call. This section describes how to create custom tools and use them in evaluations.
+
+- [Custom Tools](tools-custom.qmd) provides details on more advanced custom tool features including sandboxing, error handling, and dynamic tool definitions. 
+
+- [Standard Tools](tools-standard.qmd) describes Inspect's built-in tools for code execution, text editing, computer use, web search, and web browsing.
+
+
 These sections cover how to use varoius language models with Inspect:
 
 -   [Models](models.qmd) describe various ways to specify and provide options to models in Inspect evaluations.

{inspect_ai-0.3.75 → inspect_ai-0.3.76}/docs/llms.txt

@@ -8,13 +8,21 @@
 - [Options](https://inspect.aisi.org.uk/options.html.md): Covers the various options available for evaluations as well as how to manage model credentials.
 - [Log Viewer](https://inspect.aisi.org.uk/log-viewer.html.md): Goes into more depth on how to use Inspect View to develop and debug evaluations, including how to provide additional log metadata and how to integrate it with Python's standard logging module.
 - [VS Code](https://inspect.aisi.org.uk/vscode.html.md) Provides documentation on using the Inspect VS Code Extension to run, tune, debug, and visualise evaluations.
+
 - [Tasks](https://inspect.aisi.org.uk/tasks.html.md) bring together datasets, solvers, and scorers to define a evaluation. This section explores strategies for creating flexible and re-usable tasks.
 - [Datasets](https://inspect.aisi.org.uk/datasets.html.md): Datasets provide samples to evaluation tasks. This section illustrates how to adapt various data sources for use with Inspect, as well as how to include multi-modal data (images, etc.) in your datasets.
 - [Solvers](https://inspect.aisi.org.uk/solvers.html.md): Solvers are the heart of Inspect, and encompass prompt engineering and various other elicitation strategies. Here we cover using the built-in solvers and creating your own more sophisticated ones.
-- [Tools](https://inspect.aisi.org.uk/tools.html.md): Tools provide a means of extending the capabilities of models by registering Python functions for them to call. This section describes how to create custom tools and use them in evaluations.
 - [Scorers](https://inspect.aisi.org.uk/scorers.html.md): Scorers evaluate the work of solvers and aggregate scores into metrics. Sophisticated evals often require custom scorers that use models to evaluate output. This section covers how to create them.
+
+- [Tool Basics](https://inspect.aisi.org.uk/tools.html.md): Tools provide a means of extending the capabilities of models by registering Python functions for them to call. This section describes how to create custom tools and use them in evaluations.
+
+- [Custom Tools](https://inspect.aisi.org.uk/tools-custom.html.md) provides details on more advanced custom tool features including sandboxing, error handling, and dynamic tool definitions. 
+
+- [Standard Tools](https://inspect.aisi.org.uk/tools-standard.html.md) describes Inspect's built-in tools for code execution, text editing, computer use, web search, and web browsing.
+
+
 - [Models](https://inspect.aisi.org.uk/models.html.md): Models provide a uniform API for both evaluating a variety of large language models and using models within evaluations (e.g. for critique or grading).
-- [Providers](ttps://inspect.aisi.org.uk/providers.html.md) covers usage details and available options for the various supported providers.
+- [Providers](ttps://inspect.ai-safety-institute.org.uk/providers.html.md) covers usage details and available options for the various supported providers.
 - [Caching](https://inspect.aisi.org.uk/caching.html.md): Caching enables you to cache model output to reduce the number of API calls made, saving both time and expense.
 - [Multimodal](https://inspect.aisi.org.uk/multimodal.html.md) This article describes how to use images, audio, and video in evaluations.
 - [Reasoning](https://inspect.aisi.org.uk/reasoning.html.md) documents the additional options and data available for reasoning models.
@@ -26,6 +34,7 @@
 - [Agent Bridge](agent-bridge.qmd): Facility for integrating agents from 3rd party frameworks like AutoGen or LangChain.
 - [Human Agent](https://inspect.aisi.org.uk/human-agent.html.md): This article describes the `human_agent()` solver which enables human baselining for computing tasks.
 - [Approval](https://inspect.aisi.org.uk/approval.html.md): Approvals enable you to create fine-grained policies for approving tool calls made by models.
+
 - [Eval Logs](https://inspect.aisi.org.uk/eval-logs.html.md): Explores how to get the most out of evaluation logs for developing, debugging, and analyzing evaluations.
 - [Eval Sets](https://inspect.aisi.org.uk/eval-sets.html.md): Covers Inspect's features for describing, running, and analysing larger sets of evaluation tasks.
 - [Errors and Limits](https://inspect.aisi.org.uk/errors-and-limits.html.md): This article covers various techniques for dealing with unexpected errors and setting limits on evaluation tasks and samples, including retrying failed evaluations, establishing a threshold of samples to tolerate errors for before failing an evaluation, and setting a maximum number of messages, tokens, or elapsed seconds in a sample before forcing the solver to give up.

{inspect_ai-0.3.75 → inspect_ai-0.3.76}/docs/reference/_sidebar.yml

@@ -79,12 +79,16 @@ website:
           href: reference/inspect_ai.tool.qmd#bash
         - text: python
           href: reference/inspect_ai.tool.qmd#python
-        - text: web_search
-          href: reference/inspect_ai.tool.qmd#web_search
+        - text: bash_session
+          href: reference/inspect_ai.tool.qmd#bash_session
+        - text: text_editor
+          href: reference/inspect_ai.tool.qmd#text_editor
         - text: web_browser
           href: reference/inspect_ai.tool.qmd#web_browser
         - text: computer
           href: reference/inspect_ai.tool.qmd#computer
+        - text: web_search
+          href: reference/inspect_ai.tool.qmd#web_search
         - text: tool_with
           href: reference/inspect_ai.tool.qmd#tool_with
         - text: ToolDef

{inspect_ai-0.3.75 → inspect_ai-0.3.76}/docs/reference/inspect_ai.tool.qmd

@@ -6,9 +6,11 @@ title: "inspect_ai.tool"
 
 ### bash
 ### python
-### web_search
+### bash_session
+### text_editor
 ### web_browser
 ### computer
+### web_search
 
 ## Dynamic 
 

inspect_ai-0.3.76/docs/scripts/post-render.sh

@@ -0,0 +1,23 @@
+#!/bin/bash
+
+files=("index" "tutorial" "options" "log-viewer" "vscode" "tasks" "datasets" "solvers" "scorers" "tools" "tools-custom" "tools-standard" "models" "providers" "caching" "multimodal" "reasoning" "structured" "agents" "sandboxing" "agents-api" "agent-bridge" "human-agent" "approval" "eval-logs" "eval-sets"  "errors-and-limits"  "typing" "tracing" "parallelism" "interactivity" "extensions" "reference/inspect_ai" "reference/inspect_ai.solver" "reference/inspect_ai.tool" "reference/inspect_ai.scorer" "reference/inspect_ai.model" "reference/inspect_ai.dataset" "reference/inspect_ai.approval" "reference/inspect_ai.log" "reference/inspect_ai.util" "reference/inspect_eval" "reference/inspect_eval-set" "reference/inspect_eval-retry" "reference/inspect_score" "reference/inspect_view" "reference/inspect_log" "reference/inspect_trace" "reference/inspect_sandbox" "reference/inspect_cache" "reference/inspect_list" "reference/inspect_info")
+
+
+if [ "$QUARTO_PROJECT_RENDER_ALL" = "1" ]; then
+    llms_full="_site/llms-full.txt"
+    rm -f "${llms_full}"
+    mv _quarto.yml _quarto.yml.bak
+    for file in "${files[@]}"; do
+        echo "llms: ${file}.qmd"
+        quarto render "${file}.qmd" --to gfm-raw_html --quiet --no-execute
+        output_file="${file}.md"
+        cat "${output_file}" >> "${llms_full}"
+        echo "" >> "${llms_full}"
+        mv $output_file "_site/${file}.html.md"
+    done
+    mv _quarto.yml.bak _quarto.yml
+fi
+
+
+
+

inspect_ai-0.3.76/docs/tools-custom.qmd

@@ -0,0 +1,198 @@
+---
+title: Custom Tools
+---
+
+## Overview
+
+Inspect natively supports registering Python functions as tools and providing these tools to models that support them. Inspect also supports secure sandboxes for running abritrary code produced by models, flexible error handling, as well as dynamic tool definitions. 
+
+We'll cover all of these features below, but we'll start with a very simple example to cover the basic mechanics of tool use.
+
+## Defining Tools
+
+{{< include _tools-basics.md >}}
+
+## Tool Errors
+
+Various errors can occur during tool execution, especially when interacting with the file system or network or when using [Sandbox Environments](sandboxing.qmd) to execute code in a container sandbox. As a tool writer you need to decide how you'd like to handle error conditions. A number of approaches are possible:
+
+1.  Notify the model that an error occurred to see whether it can recover.
+
+2.  Catch and handle the error internally (trying another code path, etc.).
+
+3.  Allow the error to propagate, resulting in the current `Sample` failing with an error state.
+
+There are no universally correct approaches as tool usage and semantics can vary widely—some rough guidelines are provided below.
+
+### Default Handling {#default-handling}
+
+If you do not explicitly handle errors, then Inspect provides some default error handling behaviour. Specifically, if any of the following errors are raised they will be handled and reported to the model:
+
+-   `TimeoutError` — Occurs when a call to `subprocess()` or `sandbox().exec()` times out.
+
+-   `PermissionError` — Occurs when there are inadequate permissions to read or write a file.
+
+-   `UnicodeDecodeError` — Occurs when the output from executing a process or reading a file is binary rather than text.
+
+-   `OutputLimitExceededError` - Occurs when one or both of the output streams from `sandbox().exec()` exceed 10 MiB or when attempting to read a file over 100 MiB in size.
+
+-   `ToolError` — Special error thrown by tools to indicate they'd like to report an error to the model.
+
+These are all errors that are *expected* (in fact the `SandboxEnvironment` interface documents them as such) and possibly recoverable by the model (try a different command, read a different file, etc.). Unexpected errors (e.g. a network error communicating with a remote service or container runtime) on the other hand are not automatically handled and result in the `Sample` failing with an error.
+
+Many tools can simply rely on the default handling to provide reasonable behaviour around both expected and unexpected errors.
+
+::: {.callout-note appearance="simple"}
+When we say that the errors are reported directly to the model, this refers to the behaviour when using the default `generate()`. If on the other hand, you are have created custom scaffolding for an agent, you can intercept tool errors and apply additional filtering and logic.
+:::
+
+### Explicit Handling
+
+In some cases a tool can implement a recovery strategy for error conditions. For example, an HTTP request might fail due to transient network issues, and retrying the request (perhaps after a delay) may resolve the problem. Explicit error handling strategies are generally applied when there are *expected* errors that are not already handled by Inspect's [Default Handling](#default-handling).
+
+Another type of explicit handling is re-raising an error to bypass Inspect's default handling. For example, here we catch at re-raise `TimeoutError` so that it fails the `Sample`:
+
+``` python
+try:
+  result = await sandbox().exec(
+    cmd=["decode", file], 
+    timeout=timeout
+  )
+except TimeoutError:
+  raise RuntimeError("Decode operation timed out.")
+  
+```
+
+## Sandboxing
+
+Tools may have a need to interact with a sandboxed environment (e.g. to provide models with the ability to execute arbitrary bash or python commands). The active sandbox environment can be obtained via the `sandbox()` function. For example:
+
+``` python
+from inspect_ai.tool import ToolError, tool
+from inspect_ai.util import sandbox
+
+@tool
+def list_files():
+    async def execute(dir: str):
+        """List the files in a directory.
+
+        Args:
+            dir (str): Directory
+
+        Returns:
+            File listing of the directory
+        """
+        result = await sandbox().exec(["ls", dir])
+        if result.success:
+            return result.stdout
+        else:
+            raise ToolError(result.stderr)
+
+    return execute
+```
+
+The following instance methods are available to tools that need to interact with a `SandboxEnvironment`:
+
+{{< include _sandboxenv-interface.md >}}
+
+See the documentation on [Sandbox Environments](sandboxing.qmd) for additional details.
+
+## Tool Choice
+
+By default models will use a tool if they think it's appropriate for the given task. You can override this behaviour using the `tool_choice` parameter of the `use_tools()` Solver. For example:
+
+``` python
+# let the model decide whether to use the tool
+use_tools(addition(), tool_choice="auto")
+
+# force the use of a tool
+use_tools(addition(), tool_choice=ToolFunction(name="addition"))
+
+# prevent use of tools
+use_tools(addition(), tool_choice="none")
+```
+
+The last form (`tool_choice="none"`) would typically be used to turn off tool usage after an initial generation where the tool used. For example:
+
+``` python
+solver = [
+  use_tools(addition(), tool_choice=ToolFunction(name="addition")),
+  generate(),
+  follow_up_prompt(),
+  use_tools(tool_choice="none"),
+  generate()
+]
+```
+
+## Tool Descriptions {#sec-tool-descriptions}
+
+Well crafted tools should include descriptions that provide models with the context required to use them correctly and productively. If you will be developing custom tools it's worth taking some time to learn how to provide good tool definitions. Here are some resources you may find helpful:
+
+-   [Best Practices for Tool Definitions](https://docs.anthropic.com/claude/docs/tool-use#best-practices-for-tool-definitions)
+-   [Function Calling with LLMs](https://www.promptingguide.ai/applications/function_calling)
+
+In some cases you may want to change the default descriptions created by a tool author—for example you might want to provide better disambiguation between multiple similar tools that are used together. You also might have need to do this during development of tools (to explore what descriptions are most useful to models).
+
+The `tool_with()` function enables you to take any tool and adapt its name and/or descriptions. For example:
+
+``` python
+from inspect_ai.tool import tool_with
+
+my_add = tool_with(
+  tool=add(), 
+  name="my_add",
+  description="a tool to add numbers", 
+  parameters={
+    "x": "the x argument",
+    "y": "the y argument"
+  })
+```
+
+You need not provide all of the parameters shown above, for example here are some examples where we modify just the main tool description or only a single parameter:
+
+``` python
+my_add = tool_with(add(), description="a tool to add numbers")
+my_add = tool_with(add(), parameters={"x": "the x argument"})
+```
+
+Note that the `tool_with()` function returns a copy of the passed tool with modified descriptions (the passed tool retains its original descriptions).
+
+## Dynamic Tools {#sec-dynamic-tools}
+
+As described above, normally tools are defined using `@tool` decorators and documentation comments. It's also possible to create a tool dynamically from any function by creating a `ToolDef`. For example:
+
+``` python
+from inspect_ai.solver import use_tools
+from inspect_ai.tool import ToolDef
+
+async def addition(x: int, y: int):
+    return x + y
+
+add = ToolDef(
+    tool=addition,
+    name="add",
+    description="A tool to add numbers", 
+    parameters={
+        "x": "the x argument",
+        "y": "the y argument"
+    })
+)
+
+use_tools([add])
+```
+
+This is effectively what happens under the hood when you use the `@tool` decorator. There is one critical requirement for functions that are bound to tools using `ToolDef`: type annotations must be provided in the function signature (e.g. `x: int, y: int`).
+
+For Inspect APIs, `ToolDef` can generally be used anywhere that `Tool` can be used (`use_tools()`, setting `state.tools`, etc.). If you are using a 3rd party API that does not take `Tool` in its interface, use the `ToolDef.as_tool()` method to adapt it. For example:
+
+``` python
+from inspect_agents import my_agent
+agent = my_agent(tools=[add.as_tool()])
+```
+
+If on the other hand you want to get the `ToolDef` for an existing tool (e.g. to discover its name, description, and parameters) you can just pass the `Tool` to the `ToolDef` constructor (including whatever overrides for `name`, etc. you want):
+
+``` python
+from inspect_ai.tool import ToolDef, bash
+bash_def = ToolDef(bash())
+```