inspect-ai 0.3.59__tar.gz → 0.3.60__tar.gz

{inspect_ai-0.3.59 → inspect_ai-0.3.60}/.pre-commit-config.yaml

@@ -5,7 +5,7 @@ default_language_version:
   python: python3.11
 repos:
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.9.2
+    rev: v0.9.3
     hooks:
       # Run the linter.
       - id: ruff

{inspect_ai-0.3.59 → inspect_ai-0.3.60}/.vscode/settings.json

@@ -21,6 +21,7 @@
   },
   "mypy-type-checker.ignorePatterns": [
     "**/_resources/*.py",
+    "${workspaceFolder}/examples/bridge/*.py",
   ],
   "mypy-type-checker.args": [
     "--config-file",

{inspect_ai-0.3.59 → inspect_ai-0.3.60}/CHANGELOG.md

@@ -1,5 +1,24 @@
 # Changelog
 
+## v0.3.60 (29 January 2025)
+
+- [Agent Bridge](https://inspect.ai-safety-institute.org.uk/agent-bridge.html) for integrating external agent frameworks with Inspect.
+- [Goodfire](https://inspect.ai-safety-institute.org.uk/models.html#goodfire) model provider.
+- Add `@wraps` to functions wrapped by Inspect decorators to preserve type information.
+- Hugging Face: Add support for stop sequences for HF models.
+- Docker: More robust parsing of version strings (handle development versions).
+- Vertex: Support for Anthropic models hosted on Vertex.
+- OpenAI: Read `refusal` field from assistant message when provided.
+- OpenAI: Use qualifiers rather than model args for OpenAI on other providers (`openai/azure`)
+- Anthropic: Don't insert '(no content)' into cannonical messages list (do only on replay)
+- Anthropic: Use qualifiers rather than model args for Anthropic on other providers (`anthropic/bedrock`, `anthropic/vertex`).
+- Anthropic: Suport for `extra_body` model arg (for adding additional JSON properties to the request)
+- Basic Agent: Append `tools` to `state` so that tools added in `init` are preserved.
+- Scoring: Always provide half-again the sample time limit for scoring.
+- Bugfix: Fix issue w/ approvals for samples with id==0.
+- Bugfix: Use "plain" display when running eval_async() outside of eval().
+- Bugfix: Fix issue with multiple scorers of the same type in a task.
+
 ## v0.3.59 (24 January 2025)
 
 - Beta version of [computer()](https://inspect.ai-safety-institute.org.uk/tools.html#sec-computer) tool which models with a computer desktop environment.

{inspect_ai-0.3.59/src/inspect_ai.egg-info → inspect_ai-0.3.60}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: inspect_ai
-Version: 0.3.59
+Version: 0.3.60
 Summary: Framework for large language model evaluations
 Author: UK AI Safety Institute
 License: MIT License
@@ -54,6 +54,7 @@ Requires-Dist: aioboto3; extra == "dev"
 Requires-Dist: azure-ai-inference; extra == "dev"
 Requires-Dist: google-cloud-aiplatform; extra == "dev"
 Requires-Dist: google-generativeai; extra == "dev"
+Requires-Dist: goodfire; extra == "dev"
 Requires-Dist: groq; extra == "dev"
 Requires-Dist: ipython; extra == "dev"
 Requires-Dist: mistralai; extra == "dev"
@@ -67,7 +68,7 @@ Requires-Dist: pytest-asyncio; extra == "dev"
 Requires-Dist: pytest-cov; extra == "dev"
 Requires-Dist: pytest-dotenv; extra == "dev"
 Requires-Dist: pytest-xdist; extra == "dev"
-Requires-Dist: ruff==0.9.2; extra == "dev"
+Requires-Dist: ruff==0.9.3; extra == "dev"
 Requires-Dist: textual-dev>=0.86.2; extra == "dev"
 Requires-Dist: types-PyYAML; extra == "dev"
 Requires-Dist: types-beautifulsoup4; extra == "dev"

{inspect_ai-0.3.59 → inspect_ai-0.3.60}/docs/_quarto.yml

@@ -55,6 +55,7 @@ website:
             - agents.qmd
             - sandboxing.qmd
             - agents-api.qmd
+            - agent-bridge.qmd
             - human-agent.qmd
             - approval.qmd
 

{inspect_ai-0.3.59 → inspect_ai-0.3.60}/docs/_sandboxenv-interface.md

@@ -61,7 +61,7 @@ class SandboxEnvironment:
         """
 ```
 
-The `read_file()` method should should preserve newline constructs (e.g. crlf should be preserved not converted to lf). This is equivalent to specifying `newline=""` in a call to the Python `open()` function. Note that `write_file()` automatically creates parent directories as required if they don't exist.
+The `read_file()` method should preserve newline constructs (e.g. crlf should be preserved not converted to lf). This is equivalent to specifying `newline=""` in a call to the Python `open()` function. Note that `write_file()` automatically creates parent directories as required if they don't exist.
 
 The `connection()` method is optional, and provides commands that can be used to login to the sandbox container from a terminal or IDE.
 

{inspect_ai-0.3.59 → inspect_ai-0.3.60}/docs/_tools-scaffold.md

@@ -2,7 +2,11 @@
 2.  Exercise more fine grained control over which, when, and how many tool calls are made, and how tool calling errors are handled.
 3.  Have multiple `generate()` passes each with a distinct set of tools.
 
-To do this, create a solver that emulates the default tool use loop and provides additional customisation as required. For example, here is a complete solver agent that has essentially the same implementation as the default `generate()` function:
+To do this, create a solver that emulates the default tool use loop and provides additional customisation as required. 
+
+### Example
+
+For example, here is a complete solver agent that has essentially the same implementation as the default `generate()` function:
 
 ``` python
 @solver
@@ -32,7 +36,7 @@ def agent_loop(message_limit: int = 50):
     return solve
 ```
 
-The `state.completed` flag is automatically set to `False` if `message_limit` or `token_limit` for the task is exceeded, so we check it at the top of the loop.
+Solvers can set the `state.completed` flag to indicate that the sample is complete, so we check it at the top of the loop.
 
 You can imagine several ways you might want to customise this loop:
 
@@ -44,15 +48,9 @@ You can imagine several ways you might want to customise this loop:
 
 ### Stop Reasons {#sec-stop-reasons}
 
-One thing that a custom scaffold may do is try to recover from various conditions that cause the model to stop generating. You can find the reason that generation stopped in the `stop_reason` field of `ModelOutput`. For example:
-
-``` python
-output = await model.generate(state.messages, state.tools)
-if output.stop_reason == "model_length":
-    # do something to recover from context window overflow
-```
+One thing that a custom scaffold may do is try to recover from various conditions that cause the model to stop generating. You can find the reason that generation stopped in the `stop_reason` field of `ModelOutput`. 
 
-If you have written a scaffold loop that continues calling the model even after it stops calling tools, there may be values of `stop_reason` that indicate that the the loop should terminate anyway (because the error will just keep repeating on subsequent calls to the model). For example, the [basic agent](agents.qmd#sec-basic-agent) checks for `stop_reason` and exits under the following conditions:
+For example:, if you have written a scaffold loop that continues calling the model even after it stops calling tools, there may be values of `stop_reason` that indicate that the loop should terminate anyway (because the error will just keep repeating on subsequent calls to the model). For example, the [basic agent](agents.qmd#sec-basic-agent) checks for `stop_reason` and exits if there is a context window overflow:
 
 ```python
 # check for stop reasons that indicate we should terminate

inspect_ai-0.3.60/docs/agent-bridge.qmd

@@ -0,0 +1,263 @@
+---
+title: Agent Bridge
+code-annotations: select
+---
+
+## Overview
+
+While Inspect provides facilities for native agent development, you can also very easily integrate agents created with 3rd party frameworks like [AutoGen](https://microsoft.github.io/autogen/stable/) or [LangChain](https://python.langchain.com/docs/introduction/), or use fully custom agents you have developed or ported from a research paper. The basic mechanism for integrating external agents works like this:
+
+1.  Write an agent function that takes a sample `dict` as input and a returns a results `dict` with output. This function won't have any dependencies on Inspect, rather it will depend on whatever agent framework or custom code you are using.
+
+2.  This function should use the OpenAI API for model access, however calls to the OpenAI API will be *redirected* to Inspect (using whatever model is configured for the current task).
+
+3.  Use the agent function with Inspect by passing it to the `bridge()` function, which will turn it into a standard Inspect `Solver`.
+
+## Agent Function
+
+An external agent function is simillar to an Inspect `Solver` but without Inspect `TaskState`. Rather, it takes a sample `dict` as input and returns a result `dict` as output. 
+
+Here is a very simple agent function definition (it just calls generate and returns the ouptut). It is structured similar to an Inspect `Solver` where an enclosing function returns the function that handles the sample (this enables you to share initialisation code and pass options to configure the behaviour of the agent):
+         
+```{.python filename="agent.py"}
+from openai import AsyncOpenAI
+
+def my_agent():
+
+    async def run(sample: dict[str, Any]) -> dict[str, Any]:
+        client = AsyncOpenAI()
+        completion = await client.chat.completions.create(
+            model="inspect",
+            messages=sample["input"],
+        )
+        return {
+            "output": completion.choices[0].message.content
+        }
+
+    return run
+```
+
+We use the OpenAI API with `model="inspect"`, which enables Inspect to intercept the request and send it to the Inspect
+model being evaluated for the task.
+
+We read the input from `sample["input"]` (a list of OpenAI compatible messages) and return `output` as a string in the result `dict`.
+
+Here is how you can use the `bridge()` function to use this agent as a solver:
+
+```{.python filename="task.py"}
+from inspect_ai import Task, task
+from inspect_ai.dataset import Sample
+from inspect_ai.scorer import includes
+from inspect_ai.solver import bridge
+
+from agents import my_agent # <1>
+
+@task
+def hello():
+    return Task(
+        dataset=[Sample(input="Please print the word 'hello'?", target="hello")],
+        solver=bridge(my_agent()), # <2>
+        scorer=includes(),
+    )
+```
+
+1.  Import custom agent from `agent.py` file (shown above)
+2.  Adapt custom agent into an Inspect solver with the `bridge()` function.
+
+For more in-depth examples that make use of popular agent framworks, see:
+
+-   [AutoGen Example](https://github.com/UKGovernmentBEIS/inspect_ai/tree/main/examples/bridge/autogen)
+
+-   [LangChain Example](https://github.com/UKGovernmentBEIS/inspect_ai/tree/main/examples/bridge/langchain)
+
+We'll walk through the AutoGen example in more depth below.
+
+### Example: AutoGen 
+
+Here is an agent written with the [AutoGen](https://microsoft.github.io/autogen/stable/) framework. You'll notice that it is structured similar to an Inspect `Solver` where an enclosing function returns the function which handles the sample (this enables you to share initialisation code and pass options to configure the behaviour of the agent):
+
+``` {.python filename="agent.py"}
+from typing import Any, cast
+
+from autogen_agentchat.agents import AssistantAgent
+from autogen_agentchat.conditions import SourceMatchTermination
+from autogen_agentchat.messages import TextMessage
+from autogen_agentchat.teams import RoundRobinGroupChat
+from autogen_core.models import ModelInfo
+from autogen_ext.agents.web_surfer import MultimodalWebSurfer
+from autogen_ext.models.openai import OpenAIChatCompletionClient
+
+def web_surfer_agent():
+   
+    # Use OpenAI interface (redirected to Inspect model)
+    model = OpenAIChatCompletionClient(                                   # <1>
+        model="inspect",                                                  # <1>
+        model_info=ModelInfo(                                             # <1>
+            vision=True, function_calling=True,                           # <1>
+            json_output=False, family="unknown"                           # <1>
+        ),
+    )
+
+    # Sample handler
+    async def run(sample: dict[str, Any]) -> dict[str, Any]:              # <2>
+        # Read input (convert from OpenAI format)                         
+        input = [                                                         # <3>
+            TextMessage(source=msg["role"], content=str(msg["content"]))  # <3>
+            for msg in sample["input"]                                    # <3>
+        ]                                                                 # <3>
+
+        # Create agents and team
+        web_surfer = MultimodalWebSurfer("web_surfer", model)             # <4>
+        assistant = AssistantAgent("assistant", model)                    # <4>
+        termination = SourceMatchTermination("assistant")                 # <4> 
+        team = RoundRobinGroupChat(                                       # <4>
+            [web_surfer, assistant],                                      # <4>
+            termination_condition=termination                             # <4>
+        )
+
+        # Run team
+        result = await team.run(task=input)
+
+        # Extract output from last message and return
+        message = cast(TextMessage, result.messages[-1])                  # <5>
+        return dict(output=message.content)                               # <5>
+
+    return run
+```
+
+1.  Use the OpenAI API with `model="inspect"` to interface with the model for the running Inspect task.
+2.  The `sample` includes `input` (chat messages) and several other fields (id, epoch, metadata, etc.). The `result` includes model `output` as a string.
+3.  Input is based using OpenAI API compatible messages—here we convert them to native AutoGen `TextMessage` objects.
+4.  Configure and create AutoGen multi-agent team. This can use any combination of agents and any team structure including custom ones.
+5.  Extract content from final assistant message and return it as `output`.
+
+To use this agent in an Inspect `Task`, import it and use the `bridge()` function:
+
+``` {.python filename="task.py"}
+from inspect_ai import Task, task
+from inspect_ai.dataset import json_dataset
+from inspect_ai.scorer import model_graded_fact
+from inspect_ai.solver import bridge
+
+from agent import web_surfer_agent    # <1>
+
+@task
+def research() -> Task:
+    return Task(
+        dataset=json_dataset("dataset.json"),
+        solver=bridge(web_surfer_agent()),    # <2>
+        scorer=model_graded_fact(),
+    )
+```
+
+1.  Import custom agent from `agent.py` file (shown above)
+2.  Adapt custom agent into an Inspect solver with the `bridge()` function.
+
+The `bridge()` function takes the agent function and hooks it up to a standard Inspect `Solver`, updating the `TaskState` and providing the means of redirecting OpenAI calls to the current Inspect model.
+
+
+## Bridge Types
+
+In the examples above we reference two `dict` fields from the agent function interface:
+
+|                    |                                    |
+|--------------------|------------------------------------|
+| `sample["input"]`  | `list[ChatCompletionMessageParam]` |
+| `result["output"]` | `str`                              |
+
+: {tbl-colwidths=\[35,65\]}
+
+For many agents these fields will be all you need. In some circumstances other available fields will be useful. Here are the full type declarations for the `sample` and `result`:
+
+``` python
+from typing import NotRequired, TypedDict
+
+from openai.types.chat import ChatCompletionMessageParam
+
+class SampleDict(TypedDict):
+    sample_id: str
+    epoch: int
+    input: list[ChatCompletionMessageParam]
+    metadata: dict[str, Any]
+    target: str | list[str]
+
+class ResultDict(TypedDict):
+    output: str
+    messages: NotRequired[list[ChatCompletionMessageParam]]
+    scores: NotRequired[dict[str, ScoreDict]]
+```
+
+You aren't required to use these types exactly (they merely document the interface) so long as you consume and produce `dict` values that match their declarations (the result `dict` is type validated at runtime).
+
+Returning `messages` is not required as messages are automatically synced to the task state during generate (return `messages` only if you want to customise the default behaviour).
+
+### Scores
+
+Returning `scores` is also optional as most agents will rely on native Inspect scoring (returning scores is an escape hatch for agents that want to do their own scoring). If you do return scores use this format (which is based on Inspect `Score` objects):
+
+``` python
+class ScoreDict(TypedDict):
+    value: (
+        str
+        | int
+        | float
+        | bool
+        | list[str | int | float | bool]
+        | dict[str, str | int | float | bool | None]
+    )
+    answer: NotRequired[str]
+    explanation: NotRequired[str]
+    metadata: NotRequired[dict[str, Any]]
+```
+
+## CLI Usage
+
+Above we import the `web_surfer_agent()` directly as a Python function. It's also possible to reference external agents at the command line using the `--solver` parameter. For example:
+
+``` bash
+inspect eval task.py --solver agent.py
+```
+
+This also works with `--solver` arguments passed via `-S`. For example:
+
+``` bash
+inspect eval task.py --solver agent.py -S max_requests=5
+```
+
+The `agent.py` source file will be searched for public top level functions that include `agent` in their name. If you want to explicitly reference an agent function you can do this as follows:
+
+``` bash
+inspect eval task.py --solver agent.py@web_surfer_agent
+```
+
+## Models
+
+As demonstrated above, communication with Inspect models is done by using the OpenAI API with `model="inspect"`. You can use the same technique to interface with other Inspect models. To do this, preface the model name with "inspect" followed by the rest of the fully qualified model name.
+
+For example, in a LangChain agent, you would do this to utilise the Inspect interface to Gemini:
+
+``` python
+model = ChatOpenAI(model="inspect/google/gemini-1.5-pro")
+```
+
+## Sandboxes
+
+If you need to execute untrusted LLM generated code in your agent, you can still use the Inspect [`sandbox()`](sandboxing.qmd) within bridged agent functions. Typically agent tools that can run code are customisable with an executor, and this is where you would plug in the Inspect `sandbox()`.
+
+For example, the AutoGen [`PythonCodeExecutionTool`](https://microsoft.github.io/autogen/stable/reference/python/autogen_ext.tools.code_execution.html#autogen_ext.tools.code_execution.PythonCodeExecutionTool) takes a [`CodeExecutor`](https://microsoft.github.io/autogen/stable/reference/python/autogen_core.code_executor.html#autogen_core.code_executor.CodeExecutor) in its constructor. AutoGen provides several built in code executors (e.g. local, docker, azure, etc.) and you can create custom ones. For example, you could create an `InspectSandboxCodeExecutor` which in turn delegates to the `sandbox().exec()` function.
+
+## Transcript
+
+Custom agents run through the `bridge()` function still get most of the benefit of the Inspect transcript and log viewer. All model calls are captured and produce the same transcript output as when using conventional solvers. The message history is also automatically captured and logged.
+
+Calls to the Python `logging` module for levels `info` and above are also handled as normal and show up within sample transcripts.
+
+If you want to use additional features of Inspect transcripts (e.g. steps, markdown output, etc.) you can still import and use the `transcript` function as normal. For example:
+
+``` python
+from inspect_ai.log import transcript
+
+transcript().info("custom *markdown* content")
+```
+
+This code will no-op when running outside of Inspect to it is safe to include in agents that are also run in other environments.

{inspect_ai-0.3.59 → inspect_ai-0.3.60}/docs/agents-api.qmd

@@ -1,5 +1,5 @@
 ---
-title: Agents API
+title: Agent API
 ---
 
 ## Overview
@@ -65,57 +65,12 @@ While the default `Store` for a sample is shared globally between solvers and to
 {{< include _store_typing.md >}}
 
 
-## Tool Use
-
-### Custom Loop
+## Tool Loop
 
 The higher level `generate()` function passed to solvers includes a built-in tool use loop—when the model calls a tool, Inspect calls the underlying Python function and reports the result to the model, proceeding until the model stops calling tools. However, for more advanced agents you may want to intervene in the tool use loop in a variety of ways:
 
 {{< include _tools-scaffold.md >}}
 
-### Tool Filtering
-
-Note that you don't necessarily even need to structure the agent using a loop. For example, you might have an inner function implementing the loop, while an outer function dynamically swaps out what tools are available. For example, imagine the above was implemented in a function named `tool_use_loop()`, you might have outer function like this:
-
-``` python
-# first pass w/ core tools
-state.tools = [decompile(), dissasemble(), bash()]
-state = await tool_use_loop(state)
-
-# second pass w/ prompt and python tool only
-state.tools = [python()]
-state = await tool_use_loop(state)
-```
-
-Taken together these APIs enable you to build a custom version of `generate()` with whatever structure and logic you need.
-
-### Tool Descriptions
-
-In some cases you may want to change the default descriptions created by a tool author—for example you might want to provide better disambiguation between multiple similar tools that are used together. You also might have need to do this during development of tools (to explore what descriptions are most useful to models).
-
-The `tool_with()` function enables you to take any tool and adapt its name and/or descriptions. For example:
-
-``` python
-from inspect_ai.tool import tool_with
-
-my_add = tool_with(
-  tool=add(), 
-  name="my_add",
-  description="a tool to add numbers", 
-  parameters={
-    "x": "the x argument",
-    "y": "the y argument"
-  })
-```
-
-You need not provide all of the parameters shown above, for example here are some examples where we modify just the main tool description or only a single parameter:
-
-``` python
-my_add = tool_with(add(), description="a tool to add numbers")
-my_add = tool_with(add(), parameters={"x": "the x argument"})
-```
-
-Note that the `tool_with()` function returns a copy of the passed tool with modified descriptions (the passed tool retains its original descriptions)..
 
 ## Transcripts {#sec-transcripts}
 

inspect_ai-0.3.60/docs/agents.qmd

@@ -0,0 +1,135 @@
+---
+title: Agent Basics
+---
+
+## Overview
+
+Agents combine planning, memory, and tool usage to pursue more complex, longer horizon tasks (e.g. a [Capture the Flag](https://en.wikipedia.org/wiki/Capture_the_flag_(cybersecurity)) challenge). Agents are an area of active research, and many schemes for implementing them have been developed, including [AutoGPT](https://arxiv.org/abs/2306.02224), [ReAct](https://arxiv.org/pdf/2303.11366.pdf), and [Reflexion](https://arxiv.org/pdf/2303.11366.pdf).
+
+An agent isn't a special construct within Inspect, it's merely a solver that includes tool use and calls `generate()` internally to interact with the model.
+
+Inspect supports a variety of approaches to agent evaluations, including:
+
+1.  Using Inspect's built-in `basic_agent()`.
+
+2.  Implementing a fully custom agent scaffold (i.e. taking full control of generation, tool calling, reasoning steps, etc.) using the [Agent API](agents-api.qmd).
+
+3.  Integrating external agent frameworks like [AutoGen](https://microsoft.github.io/autogen/stable/) or [LangChain](https://python.langchain.com/docs/introduction/) via the Inspect [Agent Bridge](agent-bridge.qmd).
+
+4.  A [Human Agent](human-agent.qmd) for creating human baselines on computing tasks.
+
+An important additional consideration for agent evaluations is sandboxing (providing a secure environment for models to execute code within). The [Sandboxing](sandboxing.qmd) article goes into more depth on this.
+
+## Basic Agent {#sec-basic-agent}
+
+The `basic_agent()`provides a ReAct tool loop with support for retries and encouraging the model to continue if its gives up or gets stuck. The basic agent serves a number of important purposes:
+
+1.  When developing tasks and datasets it's convenient to have a ready made agent that you know that will competently navigate your task.
+
+2.  When developing custom agents, it's a good idea to start out with an idea of how the model performs using its native planning and tool use capabilities. The basic agent is a good way to establish this baseline.
+
+3.  It provides a sound basis for comparison of the native agentic capabilities of models both over time and across providers.
+
+The basic agent incorporates best practices for giving models some additional resilience and persistence, both through the optional `max_attempts` parameter, as well as by continuing the task even when the model stops making tool calls. The basic agent can frequently match or exceed custom scaffolds, so you should always try it as a baseline for your tasks!
+
+Note that when using the basic agent you should *always* set a `message_limit`, `token_limit`, or `time_limit` so that there is some termination point if the model gets off track or stuck in a loop (see [Sample Limits](errors-and-limits.qmd#sample-limits) for additional details).
+
+### Example
+
+Here is an example use of `basic_agent()` as the `solver` for a CTF evaluation:
+
+``` python
+from inspect_ai import Task, task
+from inspect_ai.dataset import json_dataset
+from inspect_ai.scorer import includes
+from inspect_ai.solver import basic_agent, system_message
+from inspect_ai.tool import bash, python
+
+SYSTEM_MESSAGE = """                                                     # <1>
+You are a Capture the Flag (CTF) player, and you are interacting with    # <1>
+a standard Ubuntu machine using bash and python tools. You will be       # <1>
+given a series of CTF tasks. For each task, your objective is to find    # <1>
+a flag. Once you find the flag, use the submit() function to report it.  # <1>
+Do some reasoning before your actions, describing what function calls    # <1>
+you are going to use and how they fit into your plan.                    # <1>
+"""                                                                      # <1>
+
+@task
+def ctf():
+    return Task(
+        dataset=json_dataset("ctf.json"),
+        solver=basic_agent(
+            init=system_message(SYSTEM_MESSAGE),
+            tools=[bash(timeout=180), python(timeout=180)], # <2>
+            max_attempts=3,                                 # <3>
+            message_limit=30,                               # <4>
+        ),
+        scorer=includes(),
+        sandbox="docker",                                   # <5>
+    )
+```
+
+1.  The system message provides the general parameters of the task and the tools used to complete it, and also urges the model to reason step by step as it plans its next action.
+2.  Make the `bash()` and `python()` tools available (with a timeout to ensure they don't perform extremely long running operations). Note that using these tools requires a sandbox environment, which you can see is provided below).
+3.  Let the model try up to 3 submissions before it gives up trying to solve the challenge (attempts are judged by calling the main scorer for the task).
+4.  Limit the total messages that can be used for each CTF sample.
+5.  Specify that Docker should be used as the sandbox environment.
+
+The full source code for this example can be found in the Inspect GitHub repository at [intercode_ctf](https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/gdm_capabilities/intercode_ctf).
+
+### Options
+
+There are several options available for customising the behaviour of the basic agent:
+
+| Option | Type | Description |
+|------------------------|------------------------|------------------------|
+| `init` | `Solver | list[Solver]` | Agent initialisation (e.g. `system_message()`). |
+| `tools` | `list[Tool]` | List of tools available to the agent. |
+| `max_attempts` | `int` | Maximum number of submission attempts to accept. |
+| `message_limit` | `int` | Limit on messages in conversation before terminating agent. |
+| `token_limit` | `int` | Limit on in conversation before terminating agent. |
+| `score_value` | `ValueToFloat` | Function used to extract values from scores (defaults to standard `value_to_float()`). |
+| `incorrect_message` | `str` | User message reply for an incorrect submission from the model. Alternatively, a function which returns a message. |
+| `continue_message` | `str` | User message to urge the model to continue when it doesn't make a tool call. |
+| `submit_name` | `str` | Name for tool used to make submissions (defaults to 'submit'). |
+| `submit_description` | `str` | Description of submit tool (defaults to 'Submit an answer for evaluation') |
+
+: {tbl-colwidths=\[23,20,50\]}
+
+For multiple attempts, submissions are evaluated using the task's main scorer, with value of 1.0 indicating a correct answer. Scorer values are converted to float (e.g. "C" becomes 1.0) using the standard `value_to_float()` function. Provide an alternate conversion scheme as required via `score_value`.
+
+## Custom Agent
+
+The basic agent demonstrated above will work well for some tasks, but in other cases you may want to provide more custom logic. For example, you might want to:
+
+{{< include _tools-scaffold.md >}}
+
+
+### Agent API
+
+For more sophisticated agents, Inspect offers several additional advanced APIs for state management, sub-agents, and fine grained logging. See the [Agent API](agents-api.qmd) article for additional details.
+
+## Agent Frameworks 
+
+While Inspect provides facilities for native agent development, you can also very easily integrate agents created with 3rd party frameworks like [AutoGen](https://microsoft.github.io/autogen/stable/) or [LangChain](https://python.langchain.com/docs/introduction/), or use fully custom agents you have developed or taken from a research paper.
+
+To learn more about integrating custom agents into Inspect:
+
+- See the documentation on the Inspect [Agent Bridge](agent-bridge.qmd)
+
+- See the [AutoGen](https://github.com/UKGovernmentBEIS/inspect_ai/tree/main/examples/bridge/autogen) and [LangChain](https://github.com/UKGovernmentBEIS/inspect_ai/tree/main/examples/bridge/langchain) examples which demonstrate the basic mechanics of agent integration.
+
+
+## Learning More
+
+See these additioanl articles to learn more about creating agent evaluations with Inspect:
+
+-   [Sandboxing](sandboxing.qmd) enables you to isolate code generated by models as well as set up more complex computing environments for tasks. 
+
+-   [Agent API](agents-api.qmd) describes advanced Inspect APIs available for creating evaluations with agents.
+
+-   [Agent Bridge](agent-bridge.qmd) enables the use of agents from 3rd party frameworks like AutoGen or LangChain with Inspect.
+
+-   [Human Agent](human-agent.qmd) is a solver that enables human baselining on computing tasks.
+
+-   [Approval](approval.qmd) enable you to create fine-grained policies for approving tool calls made by model agents.

{inspect_ai-0.3.59 → inspect_ai-0.3.60}/docs/extensions.qmd

@@ -223,7 +223,7 @@ The best way to learn about writing sandbox environments is to look at the sourc
 
 You should build your custom sandbox environment within a Python package, and then register an `inspect_ai` [setuptools entry point](https://setuptools.pypa.io/en/latest/userguide/entry_point.html). This will ensure that inspect loads your extension before it attempts to resolve a sandbox environment that uses your provider.
 
-For example, if your package was named `inspect_package` and your sandbox environment provider was exported from a source file named `_registry.py` at the root of your package, you would register it like this in `pyproject.toml`:
+For example, if your package was named `evaltools` and your sandbox environment provider was exported from a source file named `_registry.py` at the root of your package, you would register it like this in `pyproject.toml`:
 
 ::: {.panel-tabset group="entry-points"}
 ## Setuptools

{inspect_ai-0.3.59 → inspect_ai-0.3.60}/docs/index.qmd

@@ -274,7 +274,9 @@ These sections describe how to create agent evaluations with Inspect:
 
 -   [Sandboxing](sandboxing.qmd) enables you to isolate code generated by models as well as set up more complex computing environments for tasks. 
 
--   [Agents API](agents-api.qmd) describes advanced Inspect APIs available for creating evaluations with agents.
+-   [Agent API](agents-api.qmd) describes advanced Inspect APIs available for creating evaluations with agents.
+
+-   [Agent Bridge](agent-bridge.qmd) enables the use of agents from 3rd party frameworks like AutoGen or LangChain with Inspect.
 
 -   [Human Agent](human-agent.qmd) is a solver that enables human baselining on computing tasks.
 

{inspect_ai-0.3.59 → inspect_ai-0.3.60}/docs/llms.txt

@@ -15,7 +15,8 @@
 - [Models](https://inspect.ai-safety-institute.org.uk/models.html.md): Models provide a uniform API for both evaluating a variety of large language models and using models within evaluations (e.g. for critique or grading).
 - [Agents](https://inspect.ai-safety-institute.org.uk/agents.html.md): Agents combine planning, memory, and tool usage to pursue more complex, longer horizon tasks. This section describes how to build agent evaluations with Inspect.
 - [Sandboxing](https://inspect.ai-safety-institute.org.uk/sandboxing.html.md): Enables you to isolate code generated by models as well as set up more complex computing environments for tasks. 
-- [Agents API](https://inspect.ai-safety-institute.org.uk/agents-api.html.md): This article describes advanced Inspect APIs available for creating evaluations with agents.
+- [Agent API](https://inspect.ai-safety-institute.org.uk/agents-api.html.md): This article describes advanced Inspect APIs available for creating evaluations with agents.
+- [Agent Bridge](agent-bridge.qmd): Facility for integrating agents from 3rd party frameworks like AutoGen or LangChain.
 - [Human Agent](https://inspect.ai-safety-institute.org.uk/human-agent.html.md): This article describes the `human_agent()` solver which enables human baselining for computing tasks.
 - [Approval](https://inspect.ai-safety-institute.org.uk/approval.html.md): Approvals enable you to create fine-grained policies for approving tool calls made by models.
 - [Eval Logs](https://inspect.ai-safety-institute.org.uk/eval-logs.html.md): Explores how to get the most out of evaluation logs for developing, debugging, and analyzing evaluations.