inspect-ai 0.3.87__tar.gz → 0.3.89__tar.gz

{inspect_ai-0.3.87 → inspect_ai-0.3.89}/CHANGELOG.md

@@ -1,3 +1,34 @@
+## v0.3.89 (17 April 2025)
+
+- [Model Roles](https://inspect.aisi.org.uk/models.html#model-roles) for creating aliases to models used in a task (e.g. "grader", "red_team", "blue_team", etc.)
+- New [openai-api](https://inspect.aisi.org.uk/providers.html#openai-api) model provider for interfacing with arbitrary services that have Open AI API compatible endpoints.
+- ReAct Agent: [truncation](https://inspect.aisi.org.uk/agents.html#truncation) option to trim conversation messages when the model context window is exceeded.
+- ReAct Agent: Improve default `on_continue` message, including using a dynamic name for the submit tool.
+- Agent Bridge: Add `metadata` field to bridge input for backward compatibility with solver-based bridge.
+- Added `default` argument to `get_model()` to explicitly specify a fallback model if the specified model isn't found.
+- Approval: Approvers now take `history` argument (rather than `TaskState`) to better handle agent conversation state.
+- Anthropic: Update string matching to correctly handle BadRequestErrors related to prompt + max_tokens being too long.
+- Google: Return "(no content)" when a generate call results in no completion choices.
+- CloudFlare: Use OpenAI compatible REST endpoint for interface to models.
+- Azure AI: Use `2025-03-01-preview` as default API version if none explicitly specified.
+- Model API: `trim_messages()` function for pruning messages to fit within model context windows.
+- Model API: Improved detection of context window overflow for Grok, Groq, and CloudFlare.
+- Task Display: Show both provider and model name when concurrency context is not shared across all models for a given provider.
+- Registry: Exported `registry_create()` function for dynamic creation of registry objects (e.g. `@task`, `@solver`, etc.).
+- Remove `chdir` option from `@task` (tasks can no longer change their working directory during execution).
+- `INSPECT_EVAL_LOG_FILE_PATTERN` environment variable for setting the eval log file pattern.
+- Bugfix: Eval retry now works correctly for models with a service prefix (e.g. `openai/azure/model-name`).
+- Bugfix: Correctly resolve approvers in the same source file as tasks. 
+- Bugfix: Ensure agent decorator resolves string annotations from `__future__` as needed.
+- Bugfix: Correctly handle string `dict` keys that are numeric in store diffs.
+
+## v0.3.88 (11 April 2025)
+
+- Tools: Restore formerly required (but now deprecated) `type` field to `ToolCall`.
+- Approval: Raise operator limit exceeded error for tool approval termination action.
+- Anthropic: Don't include side count of `reasoning_tokens` in `total_tokens` (they are already included).
+- Anthropic: Update string matching to correctly handle BadRequestErrors related to prompts being too long.
+
 ## v0.3.87 (10 April 2025)
 
 - Eval: Fix an error when attempting to display realtime metrics for an evaluation.

{inspect_ai-0.3.87 → inspect_ai-0.3.89}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: inspect_ai
-Version: 0.3.87
+Version: 0.3.89
 Summary: Framework for large language model evaluations
 Author: UK AI Security Institute
 License: MIT License

{inspect_ai-0.3.87 → inspect_ai-0.3.89}/docs/_model-providers.md

@@ -10,8 +10,9 @@
 
 If the provider you are using is not listed above, you may still be able to use it if:
 
-1. It is available via OpenRouter (see the docs on using [OpenRouter](providers.qmd#openrouter) with Inspect).
+1. It provides an OpenAI compatible API endpoint. In this scenario, use the Inspect [OpenAI Compatible API](providers.qmd#openai-api) interface.
+
+2. It is available via OpenRouter (see the docs on using [OpenRouter](providers.qmd#openrouter) with Inspect).
 
-2. It provides an OpenAI compatible API endpoint. In this scenario, use the Inspect [OpenAI](providers.qmd#openai) interface and set the `OPENAI_BASE_URL` environment variable to the apprpriate value for your provider.
 
 You can also create [Model API Extensions](extensions.qmd#model-apis) to add model providers using their native interface.

{inspect_ai-0.3.87 → inspect_ai-0.3.89}/docs/agent-custom.qmd

@@ -94,16 +94,22 @@ def my_agent(tools: list[Tool]):
 You might then further tailor the behaviour of this loop. For example, you might want to:
 
 1.  Add another termination condition for the output satisfying some criteria.
-
 2.  Add a critique / reflection step between tool calling and generate.
-
 3.  Urge the model to keep going after it decides to stop calling tools.
-
-4.  Handle context window overflow (`stop_reason=="model_length"`) by truncating or summarizing the `messages`.
-
+4.  Handle context window overflow (`stop_reason=="model_length"`) by truncating or summarising the `messages`.
 5.  Examine and possibly filter the tool calls before invoking `execute_tools()`
 
+For example, you might implement automatic context window truncation in response to context window overflow:
+
+``` python
+# check for context window overflow
+if state.output.stop_reason == "model_length":
+    if overflow is not None:
+        state.messages = trim_messages(state.messages)
+        continue
+```
 
+Note that the standard `react()` agent provides some of these agent loop enhancements (urging the model to continue and handling context window overflow).
 
 ## Sample Store {#agent-store}
 
@@ -347,4 +353,4 @@ searches = [
 results = await asyncio.gather(*searches)
 ```
 
-Note that we don't `await` the subtasks when building up our list of `searches`. Rather, we let `asyncio.gather()` await all of them, returning only when all of the results are available.
+Note that we don't `await` the subtasks when building up our list of `searches`. Rather, we let `asyncio.gather()` await all of them, returning only when all of the results are available.

{inspect_ai-0.3.87 → inspect_ai-0.3.89}/docs/agents.qmd

@@ -12,7 +12,7 @@ Agents combine planning, memory, and tool usage to pursue more complex, longer h
 4.  Using the [Human Agent](human-agent.qmd) for human baselining of computing tasks.
 5.  Composing any of the above agents into [Multi Agent](multi-agent.qmd) architectures.
 
-Below, we'll cover the basic role and function of agents in Inspect. Then, we'll describe how to use the built-in `react()` agent. Subsequent articles describe more advanced topics like multi-agent systems and creating custom agents from scratch.
+Below, we'll cover the basic role and function of agents in Inspect. Then, we'll describe how to use the built-in [ReAct agent](#react-agent). Subsequent articles describe more advanced topics like multi-agent systems and creating custom agents from scratch.
 
 ## Agent Basics
 
@@ -23,29 +23,65 @@ The Inspect `Agent` protocol enables the creation of agent components that can b
 3.  Delegated to in a multi-agent architecture.
 4.  Provided as a standard `Tool` to a model
 
-
 The agents module includes a flexible, general-purpose [react agent](#react-agent), which can be used standalone or to orchestrate a [multi agent](#multi-agent) system.
 
 ### Example
 
-The following is a simple `web_surfer()` agent that uses the `web_browser()` tool to do open-ended web research. We build this agent on top of the standard `react()` agent (described in more depth below):
+The following is a simple `web_surfer()` agent that uses the `web_browser()` tool to do open-ended web research. 
 
 ``` python
-from inspect_ai.agent react
+from inspect_ai.agent import Agent, AgentState, agent
+from inspect_ai.model import ChatMessageSystem, get_model
+from inspect_ai.tool import web_browser
+
+@agent
+def web_surfer() -> Agent:
+    async def execute(state: AgentState) -> AgentState:
+        """Web research assistant."""
+      
+        # some general guidance for the agent
+        state.messages.append(
+            ChatMessageSystem(
+                content="You are an expert at using a " + 
+                "web browser to answer questions."
+            )
+        )
+
+        # run a tool loop w/ the web_browser 
+        messages, output = await get_model().generate_loop(
+            state.messages, tools=web_browser()
+        )
+
+        # update and return state
+        state.output = output
+        state.messages.extend(messages)
+        return state
+
+    return execute
+```
+
+The agent calls the `generate_loop()` function which runs the model in a loop until it stops calling tools. In this case the model may make several calls to the [web_browser()](https://inspect.aisi.org.uk/reference/inspect_ai.tool.html#web_browser) tool to fulfil the request.
+
+While this example illustrates the basic mechanic of agents, you generally wouldn't write a custom agent that does only this (a system prompt with a tool use loop) as the `react()` agent provides a more sophisticated and flexible version of this pattern. Here is the equivalent `react()` agent: 
+
+``` python
+from inspect_ai.agent import react
 from inspect_ai.tool import web_browser
 
 web_surfer = react(
     name="web_surfer",
     description="Web research assistant",
-    prompt="You are a tenacious web researcher that is expert "
-           + "at using a web browser to answer questions.",
+    prompt="You are an expert at using a " + 
+           "web browser to answer questions.",
     tools=web_browser()   
 )
 ```
 
-This agent can be used in the following ways:
+### Using Agents
 
-1.  It can be passed as a `Solver` to any Inspect interface that takes a solver:
+Agents can be used in the following ways:
+
+1.  Agents can be passed as a `Solver` to any Inspect interface that takes a solver:
 
     ``` python
     from inspect_ai import eval
@@ -55,7 +91,7 @@ This agent can be used in the following ways:
 
     For other interfaces that aren't aware of agents, you can use the `as_solver()` function to convert an agent to a solver.
 
-2.  It can be executed directly using the `run()` function (you might do this in a multi-step agent workflow):
+2.  Agents can be executed directly using the `run()` function (you might do this in a multi-step agent workflow):
 
     ``` python
     from inspect_ai.agent import run
@@ -66,12 +102,11 @@ This agent can be used in the following ways:
     print(f"The most popular movies were: {state.output.completion}")
     ```
 
-
-3.  It can participate in a multi-agent system where the conversation history is shared across agents. Use the `handoff()` function to create a tool that enables handing off the conversation from one agent to another:
+3.  Agents can participate in multi-agent systems where the conversation history is shared across agents. Use the `handoff()` function to create a tool that enables handing off the conversation from one agent to another:
 
     ``` python
     from inspect_ai.agent import handoff
-    from inspect_ai.solver use_tools, generate
+    from inspect_ai.solver import use_tools, generate
     from math_tools import addition
 
     eval(
@@ -83,11 +118,11 @@ This agent can be used in the following ways:
     )
     ```
 
-4.  It can be used as a standard tool using the `as_tool()` function:
+4.  Agents can be used as a standard tool using the `as_tool()` function:
 
     ``` python
     from inspect_ai.agent import as_tool
-    from inspect_ai.solver use_tools, generate
+    from inspect_ai.solver import use_tools, generate
 
     eval(
         task="research_bench", 
@@ -101,7 +136,6 @@ This agent can be used in the following ways:
 
     The difference between `handoff()` and `as_tool()` is that `handoff()` forwards the entire conversation history to the agent (and enables the agent to add entries to it) whereas `as_tool()` provides a simple string in, string out interface to the agent.
 
-
 ## ReAct Agent {#react-agent}
 
 The `react()` agent is a general purpose agent based on the paper [ReAct: Synergizing Reasoning and Acting in Language Models](https://arxiv.org/abs/2210.03629). ReAct is the most common architecture used in agent frameworks and is the baseline against which you should measure more complex agents (it can be surprisingly difficult to hand-tune agents that perform better than a ReAct agent against a diverse set of tasks!).
@@ -158,12 +192,11 @@ task = Task(
 eval(task, model="openai/gpt-4o")
 ```
 
-
 ### Prompt
 
 In the examples above we provide a `prompt` to the agent. This prompt is layered with other default prompt(s) to compose the final system prompt. This includes an `asssistant` prompt and a `handoff` prompt (used only when a multi-agent system with `handoff()` is running). Here is the default `assistant` prompt:
 
-```python
+``` python
 DEFAULT_ASSISTANT_PROMPT = """
 You are a helpful assistant attempting to submit the best possible answer.
 You have several tools available to help with finding the answer. You will
@@ -179,7 +212,7 @@ tool to report it.
 
 You can modify the default prompts by passing an `AgentPrompt` instance rather than a `str`. For example:
 
-```python
+``` python
 react(
     description="Expert at completing cybersecurity challenges.",
     prompt=AgentPrompt(
@@ -191,12 +224,11 @@ react(
 )
 ```
 
-
 ### Attempts
 
 By default the `react()` agent is allowed a single attempt at calling the `submit()` function. If you want to give it multiple attempts, pass another value to `attempts`:
 
-```python
+``` python
 react(
     ...
     attempts=3,
@@ -207,22 +239,37 @@ Submissions are evaluated using the task's main scorer, with value of 1.0 indica
 
 ### Continuation
 
-In some cases models in a tool use loop will simply fail to call a tool (or just talk about calling the `submit()` tool but not actually call it!). This is typically an oversight, and models simply need to be encouraged to call `submit()` or alternatively continue if they haven't yet completed the task. 
+In some cases models in a tool use loop will simply fail to call a tool (or just talk about calling the `submit()` tool but not actually call it!). This is typically an oversight, and models simply need to be encouraged to call `submit()` or alternatively continue if they haven't yet completed the task.
 
 This behavior is controlled by the `on_continue` parameter, which by default yields the following user message to the model:
 
-```default
-If you believe you have completed the task, please call the 
-`submit()` tool with your answer.
+``` default
+Please proceed to the next step using your best judgement. If you believe you
+have completed the task, please call the `submit()` tool.
 ```
 
 You can pass a different continuation message, or alternative pass an `AgentContinue` function that can dynamically determine both whether to continue and what the message is.
 
+### Truncation
+
+If your agent runs for long enough, it may end up filling the entire model context window. By default, this will cause the agent to terminate (with a log message indicating the reason). Alternatively, you can specify that the conversation should be truncated and the agent loop continue.
+
+This behavior is controlled by the `truncation` parameter (which is `"disabled"` by default, doing no truncation). To perform truncation, specify either `"auto"` (which reduces conversation size by roughly 30%) or pass a custom `MessageFilter` function. For example:
+
+``` python
+react(... truncation="auto")
+react(..., truncation=custom_truncation)
+```
+
+The default `"auto"` truncation scheme calls the `trim_messages()` function with a `preserve` ratio of 0.7.
+
+Note that if you enable truncation then a [message limit](errors-and-limits.qmd#message-limit) may not work as expected because truncation will remove old messages, potentially keeping the conversation length below your message limit. In this case you can also consider applying a [time limit](errors-and-limits.qmd#time-limit) and/or [token limit](errors-and-limits.qmd#token-limit).
+
 ### Model
 
 The `model` parameter to `react()` agent lets you specify an alternate model to use for the agent loop (if not specified then the default model for the evaluation is used). In some cases you might want to do something fancier than just call a model (e.g. do a "best of n" sampling an pick the best response). Pass a `Agent` as the `model` parameter to implement this type of custom scheme. For example:
 
-```python
+``` python
 @agent
 def best_of_n(n: int, discriminator: str | Model):
 
@@ -240,7 +287,6 @@ def best_of_n(n: int, discriminator: str | Model):
 
 Note that when you pass an `Agent` as the `model` it must include a `tools` parameter so that the ReAct agent can forward its tools.
 
-
 ## Learning More
 
 See these additioanl articles to learn more about creating agent evaluations with Inspect:
@@ -253,4 +299,4 @@ See these additioanl articles to learn more about creating agent evaluations wit
 
 -   [Human Agent](human-agent.qmd) is a solver that enables human baselining on computing tasks.
 
--   [Sandboxing](sandboxing.qmd) enables you to isolate code generated by models as well as set up more complex computing environments for tasks. 
+-   [Sandboxing](sandboxing.qmd) enables you to isolate code generated by models as well as set up more complex computing environments for tasks.

{inspect_ai-0.3.87 → inspect_ai-0.3.89}/docs/approval.qmd

@@ -123,7 +123,7 @@ def auto_approver(decision: ApprovalDecision = "approve") -> Approver:
         message: str,
         call: ToolCall,
         view: ToolCallView,
-        state: TaskState | None = None,
+        history: list[ChatMessage],
     ) -> Approval:
         return Approval(decision=decision, explanation="Automatic decision.")
 
@@ -156,7 +156,7 @@ def bash_allowlist(
         message: str,
         call: ToolCall,
         view: ToolCallView,
-        state: TaskState | None = None,
+        history: list[ChatMessage],
     ) -> Approval:
 
         # Make approval decision

{inspect_ai-0.3.87 → inspect_ai-0.3.89}/docs/datasets.qmd

@@ -54,13 +54,13 @@ If your samples include `choices`, then the `target` should be a capital letter
 
 ### Files
 
-The `files` field maps container target file paths to file contents (where contents can be either a filesystem path, a URL, or a string with inline content). For example, to copy a local file named `flag.txt` into the container path `/shared/flag.txt` you would use this:
+The `files` field maps sandbox target file paths to file contents (where contents can be either a filesystem path, a URL, or a string with inline content). For example, to copy a local file named `flag.txt` into the sandbox path `/shared/flag.txt` you would use this:
 
 ```python
 "/shared/flag.txt": "flag.txt"
 ```
 
-Files are copied into the default sandbox environment unless their name contains a prefix mapping them into another environment. For example, to copy into the `victim` container:
+Files are copied into the default sandbox environment unless their name contains a prefix mapping them into another environment. For example, to copy into the `victim` sandbox:
 
 ```python
 "victim:/shared/flag.txt": "flag.txt"

{inspect_ai-0.3.87 → inspect_ai-0.3.89}/docs/eval-logs.qmd

@@ -100,6 +100,7 @@ inspect eval images.py --log-format=json --no-log-images
 
 You can also use the `INSPECT_EVAL_LOG_IMAGES` environment variable to set a global default in your `.env` configuration file.
 
+
 ## Log File API {#sec-log-file-api}
 
 ### EvalLog
@@ -262,6 +263,26 @@ AWS_DEFAULT_REGION=eu-west-2
 
 One thing to keep in mind if you are storing logs on S3 is that they will no longer be easily viewable using a local text editor. You will likely want to configure a [FUSE filesystem](https://github.com/s3fs-fuse/s3fs-fuse) so you can easily browse the S3 logs locally.
 
+
+## Log File Name
+
+By default, log files are named using the following convention:
+
+```
+{timestamp}_{task}_{id}
+```
+
+Where `timestamp` is the time the log was created; `task` is the name of the task the log corresponds to; and `id` is a unique task id. 
+
+The `{timestamp}` part of the log file name is required to ensure that log files appear in sequential order in the filesystem. However, the rest of the filename can be customized using the `INSPECT_EVAL_LOG_FILE_PATTERN` environment variable, which can include any combination of `task`, `model`, and `id` fields. For example, to include the `model` in log file names:
+
+```bash
+export INSPECT_EVAL_LOG_FILE_PATTERN={task}_{model}_{id}
+inspect eval ctf.py 
+```
+
+As with other log file oriented environment variables, you may find it convenient to define this in a `.env` file from the location where you run your evals.
+
 ## Log Commands {#sec-log-commands}
 
 We've shown a number of Python functions that let you work with eval logs from code. However, you may be writing an orchestration or visualisation tool in another language (e.g. TypeScript) where its not particularly convenient to call the Python API. The Inspect CLI has a few commands intended to make it easier to work with Inspect logs from other languages:

{inspect_ai-0.3.87 → inspect_ai-0.3.89}/docs/extensions.qmd

@@ -296,7 +296,7 @@ def auto_approver(decision: ApprovalDecision = "approve") -> Approver:
         message: str,
         call: ToolCall,
         view: ToolCallView,
-        state: TaskState | None = None,
+        history: list[ChatMessage],
     ) -> Approval:
         return Approval(
             decision=decision, 

{inspect_ai-0.3.87 → inspect_ai-0.3.89}/docs/models.qmd

@@ -34,17 +34,17 @@ INSPECT_EVAL_MODEL=google/gemini-1.5-pro
 
 #### No Model
 
-Some evaluations will either not make use of models or call the lower-level `get_model()` function to explicitly access models for different roles (see the [Model API](#model-api) section below for details on this). 
+Some evaluations will either not make use of models or call the lower-level `get_model()` function to explicitly access models for different roles (see the [Model API](#model-api) section below for details on this).
 
 In these cases, you are not required to specify a `--model`. If you happen to have an `INSPECT_EVAL_MODEL` defined and you want to prevent your evaluation from using it, you can explcitly specify no model as follows:
 
-```bash
+``` bash
 inspect eval arc.py --model none
 ```
 
 Or from Python:
 
-```python
+``` python
 eval("arc.py", model=None)
 ```
 
@@ -84,11 +84,11 @@ Inspect uses an asynchronous architecture to run task samples in parallel. If yo
 
 By default, Inspect uses a `max_connections` value of 10. You can increase this consistent with your account limits. If you are experiencing rate-limit errors you will need to experiment with the `max_connections` option to find the optimal value that keeps you under the rate limit (the section on [Parallelism](parallelism.qmd) includes additional documentation on how to do this).
 
-## Model API
+## Model API {#model-api}
 
 The `--model` which is set for an evaluation is automatically used by the `generate()` solver, as well as for other solvers and scorers built to use the currently evaluated model. If you are implementing a `Solver` or `Scorer` and want to use the currently evaluated model, call `get_model()` with no arguments:
 
-```python
+``` python
 from inspect_ai.model import get_model
 
 model = get_model()
@@ -97,7 +97,7 @@ response = await model.generate("Say hello")
 
 If you want to use other models in your solvers and scorers, call `get_model()` with an alternate model name, along with optional generation config. For example:
 
-```python
+``` python
 model = get_model("openai/gpt-4o")
 
 model = get_model(
@@ -108,7 +108,7 @@ model = get_model(
 
 You can also pass provider specific parameters as additional arguments to `get_model()`. For example:
 
-```python
+``` python
 model = get_model("hf/openai-community/gpt2", device="cuda:0")
 ```
 
@@ -116,36 +116,104 @@ model = get_model("hf/openai-community/gpt2", device="cuda:0")
 
 By default, calls to `get_model()` are memoized, meaning that calls with identical parameters resolve to a cached version of the model. You can disable this by passing `memoize=False`:
 
-```python
+``` python
 model = get_model("openai/gpt-4o", memoize=False)
 ```
 
 Finally, if you prefer to create and fully close model clients at their place of use, you can use the async context manager built in to the `Model` class. For example:
 
-```python
+``` python
 async with get_model("openai/gpt-4o") as model:
     eval(mytask(), model=model)
 ```
 
 If you are not in an async context there is also a sync context manager available:
 
-```python
+``` python
 with get_model("hf/Qwen/Qwen2.5-72B") as model:
     eval(mytask(), model=model)
 ```
 
 Note though that this *won't work* with model providers that require an async close operation (OpenAI, Anthropic, Grok, Together, Groq, Ollama, llama-cpp-python, and CloudFlare).
 
+## Model Roles
+
+Model roles enable you to create aliases for the various models used in your tasks, and then dynamically vary those roles when running an evaluation. For example, you might have a "critic" or "monitor" role, or perhaps "red_team" and "blue_team" roles. Roles are included in the log and displayed in model events within the transcript.
+
+Here is a scorer that utilises a "grader" role when binding to a model:
+
+``` python
+@scorer(metrics=[accuracy(), stderr()])
+def model_grader() -> Scorer:
+    async def score(state: TaskState, target: Target):
+        model = get_model(role="grader")
+        ...
+```
+
+By default if there is no "grader" role specified, the default model for the evaluation will be returned. Model roles can be specified when using `inspect eval` or calling the `eval()` function:
+
+``` bash
+inspect eval math.py --model-role grader=google/gemini-2.0-flash
+```
+
+Or with `eval()`:
+
+``` python
+eval("math.py", model_roles = { "grader": "google/gemini-2.0-flash" })
+```
+
+### Role Defaults
+
+By default if there is a no role explicitly defined then `get_model(role="...")` will return the default model for the evaluation. You can specify an alternate default model as follows:
+
+``` python
+model = get_model(role="grader", default="openai/gpt-4o")
+```
+
+This means that you can use model roles as a means of external configurability even if you aren't yet explicitly taking advantage of them.
+
+### Roles for Tasks
+
+In some cases it may not be convenient to specify `model_roles` in the top level call to `eval()`. For example, you might be running an [Eval Set](eval-sets.qmd) to explore the behaviour of different models for a given role. In this case, do not specify `model_roles` at the eval level, rather, specify them at the task level.
+
+For example, imagine we have a task named `blues_clues` that we want to vary the red and blue teams for in an eval set:
+
+``` python
+from inspect_ai import eval_set, task_with
+from ctf_tasks import blues_clues 
+
+tasks = [
+    task_with(blues_clues(), model_roles = {
+        "red_team": "openai/gpt-4o",
+        "blue_team": "google/gemini-2.0-flash"
+    }),()
+    task_with(blues_clues, model_roles = {
+        "red_team": "google/gemini-2.0-flash",
+        "blue_team": "openai/gpt-4o"
+    })
+]
+
+eval_set(tasks, log_dir="...")
+```
+
+Note that we also don't specify a `model` for this eval (it doesn't have a main model but rather just the red and blue team roles).
+
+As illustrated above, you can define as many named roles as you need. When using `eval()` or `Task` roles are specified using a dictionary. When using `inspect eval` you can include multiple `--model-role` options on the command line:
+
+``` bash
+inspect eval math.py \
+   --model-role red_team=google/gemini-2.0-flash \
+   --model-role blue_team=openai/gpt-4o-mini
+```
 
 ## Learning More
 
-- [Providers](providers.qmd) covers usage details and available options for the various supported providers.
+-   [Providers](providers.qmd) covers usage details and available options for the various supported providers.
 
-- [Caching](caching.qmd) explains how to cache model output to reduce the number of API calls made.
+-   [Caching](caching.qmd) explains how to cache model output to reduce the number of API calls made.
 
-- [Multimodal](multimodal.qmd) describes the APIs available for creating multimodal evaluations (including images, audio, and video).
+-   [Multimodal](multimodal.qmd) describes the APIs available for creating multimodal evaluations (including images, audio, and video).
 
-- [Reasoning](reasoning.qmd) documents the additional options and data available for reasoning models.
- 
-- [Structured Output](structured.qmd) explains how to constrain model output to a particular JSON schema.
+-   [Reasoning](reasoning.qmd) documents the additional options and data available for reasoning models.
 
+-   [Structured Output](structured.qmd) explains how to constrain model output to a particular JSON schema.

{inspect_ai-0.3.87 → inspect_ai-0.3.89}/docs/providers.qmd

@@ -54,11 +54,11 @@ You can then use the normal `openai` provider with the `azure` qualifier and the
 ``` bash
 export AZUREAI_OPENAI_API_KEY=your-api-key
 export AZUREAI_OPENAI_BASE_URL=https://your-url-at.azure.com
-export AZUREAI_OPENAI_API_VERSION=2025-02-01-preview
+export AZUREAI_OPENAI_API_VERSION=2025-03-01-preview
 inspect eval math.py --model openai/azure/gpt-4o-mini
 ```
 
-Note that if the `AZUREAI_OPENAI_API_VERSION` is not specified, Inspect will generally default to the latest deployed version, which as of this writing is `2025-02-01-preview`.
+Note that if the `AZUREAI_OPENAI_API_VERSION` is not specified, Inspect will generally default to the latest deployed version, which as of this writing is `2025-03-01-preview`.
 
 ## Anthropic {#anthropic}
 
@@ -228,14 +228,16 @@ inspect eval math.py --model mistral/azure/Mistral-Large-2411
 
 ## DeepSeek {#deepseek}
 
-To use [DeepSeek](https://www.deepseek.com/) models, install the `openai` package (which DeepSeek provides a compatible backend for) and set your credentials using the `OPENAI_API_KEY` environment variable. Then, when running inspect, use the  `--model-base-url` option to point to DeepSeek's base URL, and the `--model` option to specify a model:
+[DeepSeek](https://www.deepseek.com/) provides an OpenAI compatible API endpoint which you can use with Inspect via the `openai-api` provider. To do this, define the `DEEPSEEK_API_KEY` and `DEEPSEEK_BASE_URL` environment variables then refer to models with `openai-api/deepseek/<model-name>`. For example:
 
 ```bash
 pip install openai
-export OPENAI_API_KEY=your-deepseek-api-key
-inspect eval arc.py --model-base-url https://api.deepseek.com --model openai/deepseek-reasoner 
+export DEEPSEEK_API_KEY=your-deepseek-api-key
+export DEEPSEEK_BASE_URL=https://api.deepseek.com
+inspect eval arc.py --model openai-api/deepseek/deepseek-reasoner 
 ```
 
+
 ## Grok {#grok}
 
 To use the [Grok](https://x.ai/) provider, install the `openai` package (which the Grok service provides a compatible backend for), set your credentials, and specify a model using the `--model` option:
@@ -482,12 +484,13 @@ $ inspect eval arc.py --model vllm/local -M model_path=./my-model
 
 ### vLLM Server
 
-vLLM provides an HTTP server that implements OpenAI’s Chat API. To use this with Inspect, use the `openai` provider rather than the `vllm` provider, setting the model base URL to point to the vLLM server rather than OpenAI. For example:
+vLLM provides an HTTP server that implements OpenAI’s Chat API. To use this with Inspect, use the `openai-api` provider rather than the `vllm` provider, setting the model base URL to point to the vLLM server. For example:
 
 ``` bash
-$ export OPENAI_BASE_URL=http://localhost:8080/v1
-$ export OPENAI_API_KEY=<your-server-api-key>
-$ inspect eval arc.py --model openai/meta-llama/Meta-Llama-3-8B-Instruct
+$ export VLLM_BASE_URL=http://localhost:8080/v1
+$ export VLLM_API_KEY=<your-server-api-key>
+$ inspect eval arc.py \
+    --model openai-api/vllm/meta-llama/Meta-Llama-3-8B-Instruct
 ```
 
 See the vLLM documentation on [Server Mode](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html) for additional details.
@@ -526,6 +529,32 @@ The following environment variables are supported by the llama-cpp-python provid
 |----------------------------|--------------------------------------------|
 | `LLAMA_CPP_PYTHON_BASE_URL` | Base URL for requests (optional, defaults to `http://localhost:8000/v1`) |
 
+## OpenAI Compatible {#openai-api}
+
+
+If your model provider makes an OpenAI API compatible endpoint available, you can use it with Inspect via the `openai-api` provider, which uses the following model naming convention:
+
+```
+openai-api/<provider-name>/<model-name>
+```
+
+Inspect will read environment variables corresponding to the api key and base url of your provider using the following convention (note that the provider name is capitalized):
+
+```
+<PROVIDER-NAME>_API_KEY
+<PROVIDER-NAME>_BASE_URL
+```
+
+### Example
+
+Here is how you would access DeepSeek using the `openai-api` provider:
+
+```bash
+export DEEPSEEK_API_KEY=your-deepseek-api-key
+export DEEPSEEK_BASE_URL=https://api.deepseek.com
+inspect eval arc.py --model openai-api/deepseek/deepseek-reasoner 
+```
+
 ## OpenRouter
 
 To use the [OpenRouter](https://openrouter.ai/) provider, install the `openai` package (which the OpenRouter service provides a compatible backend for), set your credentials, and specify a model using the `--model` option: