inspect-ai 0.3.104__tar.gz → 0.3.105__tar.gz

{inspect_ai-0.3.104 → inspect_ai-0.3.105}/CHANGELOG.md

@@ -1,3 +1,21 @@
+## v0.3.105 (17 June 2025)
+
+- [background()](https://inspect.aisi.org.uk/agent-custom.html#background) function for executing work in the background of the current sample.
+- [sandbox_service()](https://inspect.aisi.org.uk/agent-custom.html#sandbox-service) function for making available methods to a sandbox for calling back into the main Inspect process.
+- [sample_limits()](https://inspect.aisi.org.uk/errors-and-limits.html#query-usage) function for determining the current status of sample limits.
+- React agent: Only do substitution on parts of the prompt that may contain a {submit} reference.
+- Agent handoff: Ensure that handoff tool call responses immediately follow the call.
+- Agent handoff: Only print handoff agent prefix if there is assistant message content.
+- Subprocess: Ensure that streams are drained when a cancellation occurs (prevent hanging on calls with large output payloads).
+- Eval log: Capture only limits that terminated the sample as `sample.limit` (as opposed to ones bound to context managers or agents).
+- Inspect View: Display metadata for Chat Messages.
+- Inspect View: Increase transcript outline font size.
+- Inspect View: Add support for filtering by sample id, sample metadata.
+- Bugfix: Eval set now correctly handles retries for tasks with defaulted args (regressed in v0.3.104).
+- Bugfix: Use correct bindings for Claude v4 native `text_editor` tool; don't use native tool definition for Haiku 3.5 or Opus 3.0.  
+- Bugfix: Restore preservation of `ContentReasoning` blocks for Gemini (regressed in v0.3.104). 
+- Bugfix: Dataset shuffling now works correctly with `seed` of 0.
+
 ## v0.3.104 (12 June 2025)
 
 - Web Search: Added provider for Anthropic's internal web search tool.

{inspect_ai-0.3.104 → inspect_ai-0.3.105}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: inspect_ai
-Version: 0.3.104
+Version: 0.3.105
 Summary: Framework for large language model evaluations
 Author: UK AI Security Institute
 License: MIT License

{inspect_ai-0.3.104 → inspect_ai-0.3.105}/docs/CHANGELOG.md

@@ -1,3 +1,21 @@
+## v0.3.105 (17 June 2025)
+
+- [background()](https://inspect.aisi.org.uk/agent-custom.html#background) function for executing work in the background of the current sample.
+- [sandbox_service()](https://inspect.aisi.org.uk/agent-custom.html#sandbox-service) function for making available methods to a sandbox for calling back into the main Inspect process.
+- [sample_limits()](https://inspect.aisi.org.uk/errors-and-limits.html#query-usage) function for determining the current status of sample limits.
+- React agent: Only do substitution on parts of the prompt that may contain a {submit} reference.
+- Agent handoff: Ensure that handoff tool call responses immediately follow the call.
+- Agent handoff: Only print handoff agent prefix if there is assistant message content.
+- Subprocess: Ensure that streams are drained when a cancellation occurs (prevent hanging on calls with large output payloads).
+- Eval log: Capture only limits that terminated the sample as `sample.limit` (as opposed to ones bound to context managers or agents).
+- Inspect View: Display metadata for Chat Messages.
+- Inspect View: Increase transcript outline font size.
+- Inspect View: Add support for filtering by sample id, sample metadata.
+- Bugfix: Eval set now correctly handles retries for tasks with defaulted args (regressed in v0.3.104).
+- Bugfix: Use correct bindings for Claude v4 native `text_editor` tool; don't use native tool definition for Haiku 3.5 or Opus 3.0.  
+- Bugfix: Restore preservation of `ContentReasoning` blocks for Gemini (regressed in v0.3.104). 
+- Bugfix: Dataset shuffling now works correctly with `seed` of 0.
+
 ## v0.3.104 (12 June 2025)
 
 - Web Search: Added provider for Anthropic's internal web search tool.

{inspect_ai-0.3.104 → inspect_ai-0.3.105}/docs/agent-custom.qmd

@@ -205,7 +205,6 @@ Limits are often applied at the sample level or using a context manager. It is a
 
 {{< include _agent_limits.md >}}
 
-
 ## Parameters
 
 The `web_surfer` agent used an example above doesn't take any parameters, however, like tools, agents can accept arbitrary parameters.
@@ -324,7 +323,7 @@ There are two reasons that you might want to create spans:
 
 Spans are automatically created for sample initialisation, solvers, scorers, subtasks, tool calls, and agent execution.
 
-## Parallel Execution
+## Parallelism
 
 You can execute subtasks in parallel using the `collect()` function. For example, to run 3 `web_search()` coroutines in parallel:
 
@@ -342,4 +341,79 @@ Note that `collect()` is similar to [`asyncio.gather()`](https://docs.python.org
 
 The Inspect `collect()` function also automatically includes each task in a `span()`, which ensures that its events are grouped together in the transcript.
 
-Using `collect()` in preference to `asyncio.gather()` is highly recommended for both Trio compatibility and more legible transcript output.
+Using `collect()` in preference to `asyncio.gather()` is highly recommended for both Trio compatibility and more legible transcript output.
+
+## Background Work
+
+The `background()` function enables you to execute an async task in the background of the current sample. The task terminates when the sample terminates. For example:
+
+``` python
+import anyio
+from inspect_ai.util import background
+
+async def worker():
+    try:
+        while True:
+            # background work
+            anyio.sleep(1.0)
+    finally:
+        # cleanup
+
+background(worker)
+```
+
+The above code demonstrates a couple of important characteristics of a sample background worker:
+
+1.  Background workers typically operate in a loop, often polling a a sandbox or other endpoint for activity. In a loop like this it's important to sleep at regular intervals so your background work doesn't monopolise CPU resources.
+
+2.  When the sample ends, background workers are cancelled (which results in a cancelled error being raised in the worker). Therefore, if you need to do cleanup in your worker it should occur in a `finally` block.
+
+
+## Sandbox Service
+
+Sandbox services make available a set of methods to a sandbox for calling back into the main Inspect process. For example, the [Human Agent](human-agent.qmd) uses a sandbox service to enable the human agent to start, stop, score, and submit tasks.
+
+Sandbox service are often run using the `background()` function to make them available for the lifetime of a sample.
+
+For example, here's a simple calculator service that provides add and subtract methods to Python code within a sandbox:
+
+```python
+from inspect_ai.util import background, sandbox_service
+
+async def calculator_service():
+    async def add(x: int, y: int) -> int:
+        return x + y
+
+    async def subtract(x: int, y: int) -> int:
+        return x - y
+
+    await sandbox_service(
+        name="calculator",
+        methods=[add, subtract],
+        until=lambda: True,
+        sandbox=sandbox()
+    )
+
+background(calculator_service)
+```
+
+To use the service from within a sandbox, either add it to the sys path or use importlib. For example, if the service is named 'calculator':
+
+```python
+import sys
+sys.path.append("/var/tmp/sandbox-services/calculator")
+import calculator
+```
+
+Or:
+
+```python
+import importlib.util
+spec = importlib.util.spec_from_file_location(
+    "calculator", 
+    "/var/tmp/sandbox-services/calculator/calculator.py"
+)
+calculator = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(calculator)
+```
+

{inspect_ai-0.3.104 → inspect_ai-0.3.105}/docs/errors-and-limits.qmd

@@ -205,7 +205,19 @@ raise LimitExceededError(
 )
 ```
 
-## Scoped Limits
+### Query Usage
+
+We can determine how much of a sample limit has been used, what the limit is, and how much of the resource is remaining:
+
+``` python
+sample_time_limit = sample_limits().time
+print(f"{sample_time_limit.remaining:.0f} seconds remaining")
+```
+
+Note that `sample_limits()` only retrieves the sample-level limits, not [scoped limits](#scoped-limits) or [agent limits](#agent-limits).
+
+
+## Scoped Limits {#scoped-limits}
 
 You can also apply limits at arbitrary scopes, independent of the sample or agent-scoped limits. For instance, applied to a specific block of code. For example:
 
@@ -341,6 +353,6 @@ def myagent() -> Solver:
 It's important to note that `token_limit()` is for all tokens used *while the context manager is open*. If you want to limit the number of tokens that can be yielded from a single call to the model you should use the `max_tokens` generation option.
 :::
 
-## Agent Limits
+## Agent Limits {#agent-limits}
 
 {{< include _agent_limits.md >}}

{inspect_ai-0.3.104 → inspect_ai-0.3.105}/docs/reference/_sidebar.yml

@@ -555,6 +555,10 @@ website:
           href: reference/inspect_ai.util.qmd#working_limit
         - text: apply_limits
           href: reference/inspect_ai.util.qmd#apply_limits
+        - text: sample_limits
+          href: reference/inspect_ai.util.qmd#sample_limits
+        - text: SampleLimits
+          href: reference/inspect_ai.util.qmd#samplelimits
         - text: Limit
           href: reference/inspect_ai.util.qmd#limit
         - text: LimitExceededError
@@ -581,6 +585,8 @@ website:
           href: reference/inspect_ai.util.qmd#resource
         - text: throttle
           href: reference/inspect_ai.util.qmd#throttle
+        - text: background
+          href: reference/inspect_ai.util.qmd#background
         - text: trace_action
           href: reference/inspect_ai.util.qmd#trace_action
         - text: trace_message
@@ -597,6 +603,8 @@ website:
           href: reference/inspect_ai.util.qmd#sandboxconnection
         - text: sandboxenv
           href: reference/inspect_ai.util.qmd#sandboxenv
+        - text: sandbox_service
+          href: reference/inspect_ai.util.qmd#sandbox_service
         - text: registry_create
           href: reference/inspect_ai.util.qmd#registry_create
         - text: RegistryType

{inspect_ai-0.3.104 → inspect_ai-0.3.105}/docs/reference/inspect_ai.util.qmd

@@ -16,6 +16,8 @@ title: "inspect_ai.util"
 ### time_limit
 ### working_limit
 ### apply_limits
+### sample_limits
+### SampleLimits
 ### Limit
 ### LimitExceededError
 
@@ -38,6 +40,7 @@ title: "inspect_ai.util"
 ### collect
 ### resource
 ### throttle
+### background
 ### trace_action
 ### trace_message
 
@@ -50,6 +53,7 @@ title: "inspect_ai.util"
 ### SandboxEnvironment
 ### SandboxConnection
 ### sandboxenv
+### sandbox_service
 
 ## Registry
 

{inspect_ai-0.3.104 → inspect_ai-0.3.105}/docs/tools-standard.qmd

@@ -3,10 +3,130 @@ title: Standard Tools
 tbl-colwidths: [40,60]
 ---
 
-## Overview
+## Overview 
 
 {{< include _tools-standard.md >}}
 
+
+## Web Search {#sec-web-search}
+
+The `web_search()` tool provides models the ability to enhance their context window by performing a search. Web searches are executed using a provider. Providers are split into two categories:
+
+-   Internal providers: `"openai"`, `"anthropic"`, and `"gemini"` - these use the model's built-in search capability and do not require separate API keys. These work only for their respective model provider (e.g. the "openai" search provider works only for `openai/*` models).
+
+-   External providers: `"tavily"`, `"exa"`, and `"google"`. These are external services that work with any model and require separate accounts and API keys. Note that "google" is different from "gemini" - "google" refers to Google's Programmable Search Engine service, while "gemini" refers to Google's built-in search capability for Gemini models.
+
+Internal providers will be prioritized if running on the corresponding model (e.g., "openai" provider will be used when running on `openai` models). If an internal provider is specified but the evaluation is run with a different model, a fallback external provider must also be specified.
+
+You can configure the `web_search()` tool in various ways:
+
+``` python
+from inspect_ai.tool import web_search
+
+# single provider
+web_search("tavily")
+
+# internal provider and fallback
+web_search(["openai", "tavily"])
+
+# multiple internal providers and fallback
+web_search(["openai", "anthropic", "gemini", "tavily"])
+
+# provider with specific options
+web_search({"tavily": {"max_results": 5}})
+
+# multiple providers with options
+web_search({
+    "openai": True, 
+    "google": {"num_results": 5}, 
+    "tavily": {"max_results": 5}
+})
+```
+
+### OpenAI Options
+
+The `web_search()` tool can use OpenAI's built-in search capability when running on a limited number of OpenAI models (currently "gpt-4o", "gpt-4o-mini", and "gpt-4.1"). This provider does not require any API keys beyond what's needed for the model itself.
+
+For more details on OpenAI's web search parameters, see [OpenAI Web Search Documentation](https://platform.openai.com/docs/guides/tools-web-search?api-mode=responses).
+
+Note that when using the "openai" provider, you should also specify a fallback external provider (like "tavily", "exa", or "google") if you are also running the evaluation with non-OpenAI model.
+
+### Anthropic Options
+
+The `web_search()` tool can use Anthropic's built-in search capability when running on a limited number of Anthropic models (currently "claude-opus-4-20250514", "claude-sonnet-4-20250514", "claude-3-7-sonnet-20250219", "claude-3-5-sonnet-latest", "claude-3-5-haiku-latest"). This provider does not require any API keys beyond what's needed for the model itself.
+
+For more details on Anthropic's web search parameters, see [Anthropic Web Search Documentation](https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/web-search-tool).
+
+Note that when using the "anthropic" provider, you should also specify a fallback external provider (like "tavily", "exa", or "google") if you are also running the evaluation with non-Anthropic model.
+
+### Gemini Options
+
+The `web_search()` tool can use Google's built-in search capability (called grounding) when running on Gemini 2.0 models and later. This provider does not require any API keys beyond what's needed for the model itself.
+
+This is distinct from the "google" provider (described below), which uses Google's external Programmable Search Engine service and requires separate API keys.
+
+For more details, see [Grounding with Google Search](https://ai.google.dev/gemini-api/docs/grounding).
+
+Note that when using the "gemini" provider, you should also specify a fallback external provider (like "tavily", "exa", or "google") if you are also running the evaluation with non-Gemini models.
+
+::: callout-warning
+Google's search grounding does not currently support use with other tools. Attempting to use `web_search("gemini")` alongside other tools will result in an error.
+:::
+
+### Tavily Options
+
+The `web_search()` tool can use [Tavily](https://tavily.com/)'s Research API. To use it you will need to set up your own Tavily account. Then, ensure that the following environment variable is defined:
+
+-   `TAVILY_API_KEY` — Tavily Research API key
+
+Tavily supports the following options:
+
+| Option | Description |
+|------------------------------------|------------------------------------|
+| `max_results` | Number of results to return |
+| `search_depth` | Can be "basic" or "advanced" |
+| `topic` | Can be "general" or "news" |
+| `include_domains` / `exclude_domains` | Lists of domains to include or exclude |
+| `time_range` | Time range for search results (e.g., "day", "week", "month") |
+| `max_connections` | Maximum number of concurrent connections |
+
+For more options, see the [Tavily API Documentation](https://docs.tavily.com/documentation/api-reference/endpoint/search).
+
+### Exa Options
+
+The `web_search()` tool can use [Exa](https://exa.ai/)'s Answer API. To use it you will need to set up your own Exa account. Then, ensure that the following environment variable is defined:
+
+-   `EXA_API_KEY` — Exa API key
+
+Exa supports the following options:
+
+| Option | Description |
+|------------------------------------|------------------------------------|
+| `text` | Whether to include text content in citations (defaults to true) |
+| `model` | LLM model to use for generating the answer ("exa" or "exa-pro") |
+| `max_connections` | Maximum number of concurrent connections |
+
+For more details, see the [Exa API Documentation](https://docs.exa.ai/reference/answer).
+
+### Google Options
+
+The `web_search()` tool can use [Google Programmable Search Engine](https://programmablesearchengine.google.com/about/) as an external provider. This is different from the "gemini" provider (described above), which uses Google's built-in search capability for Gemini models.
+
+To use the "google" provider you will need to set up your own Google Programmable Search Engine and also enable the [Programmable Search Element Paid API](https://developers.google.com/custom-search/docs/paid_element). Then, ensure that the following environment variables are defined:
+
+-   `GOOGLE_CSE_ID` — Google Custom Search Engine ID
+-   `GOOGLE_CSE_API_KEY` — Google API key used to enable the Search API
+
+
+Google supports the following options: 
+
+| Option | Description |
+|------------------------------------|------------------------------------|
+| `num_results` | The number of relevant webpages whose contents are returned  |
+| `max_provider_calls` | Number of times to retrieve more links in case previous ones were irrelevant (defaults to 3)  |
+| `max_connections` | Maximum number of concurrent connections (defaults to 10) |
+| `model` | Model to use to determine if search results are relevant (defaults to the model being evaluated) |
+
 ## Bash and Python {#sec-bash-and-python}
 
 The `bash()` and `python()` tools enable execution of arbitrary shell commands and Python code, respectively. These tools require the use of a [Sandbox Environment](sandboxing.qmd) for the execution of untrusted code. For example, here is how you might use them in an evaluation where the model is asked to write code in order to solve capture the flag (CTF) challenges:
@@ -364,126 +484,6 @@ If you want to experiment with bypassing the native computer tool types and just
 inspect eval computer.py --no-internal-tools
 ```
 
-## Web Search {#sec-web-search}
-
-The `web_search()` tool provides models the ability to enhance their context window by performing a search. Web searches are executed using a provider. Providers are split into two categories:
-
--   Internal providers: "openai", "anthropic", and "gemini" - these use the model's built-in search capability and do not require separate API keys. These work only for their respective model provider (e.g. the "openai" search provider works only for `openai/*` models).
-
--   External providers: "tavily", "exa", and "google". These are external services that work with any model and require separate accounts and API keys. Note that "google" is different from "gemini" - "google" refers to Google's Programmable Search Engine service, while "gemini" refers to Google's built-in search capability for Gemini models.
-
-Internal providers will be prioritized if running on the corresponding model (e.g., "openai" provider will be used when running on `openai` models). If an internal provider is specified but the evaluation is run with a different model, a fallback external provider must also be specified.
-
-You can configure the `web_search()` tool in various ways:
-
-``` python
-from inspect_ai.tool import web_search
-
-# single provider
-web_search("tavily")
-
-# with internal provider and fallback
-web_search(["openai", "tavily"])
-
-# with gemini provider and fallback
-web_search(["gemini", "tavily"])
-
-# provider with specific options
-web_search({"tavily": {"max_results": 5}})
-
-# using exa provider
-web_search("exa")
-
-# exa with specific options
-web_search({"exa": {"model": "exa-pro"}})
-
-# multiple providers with options
-web_search([{"openai": True}, {"google": {"num_results": 5}}])
-```
-
-### OpenAI Options
-
-The `web_search()` tool can use OpenAI's built-in search capability when running on a limited number of OpenAI models (currently "gpt-4o", "gpt-4o-mini", and "gpt-4.1"). This provider does not require any API keys beyond what's needed for the model itself.
-
-For more details on OpenAI's web search parameters, see [OpenAI Web Search Documentation](https://platform.openai.com/docs/guides/tools-web-search?api-mode=responses).
-
-Note that when using the "openai" provider, you should also specify a fallback external provider (like "tavily", "exa", or "google") if you are also running the evaluation with non-OpenAI model.
-
-### Anthropic Options
-
-The `web_search()` tool can use Anthropic's built-in search capability when running on a limited number of Anthropic models (currently "claude-opus-4-20250514", "claude-sonnet-4-20250514", "claude-3-7-sonnet-20250219", "claude-3-5-sonnet-latest", "claude-3-5-haiku-latest"). This provider does not require any API keys beyond what's needed for the model itself.
-
-For more details on Anthropic's web search parameters, see [Anthropic Web Search Documentation](https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/web-search-tool).
-
-Note that when using the "anthropic" provider, you should also specify a fallback external provider (like "tavily", "exa", or "google") if you are also running the evaluation with non-Anthropic model.
-
-### Gemini Options
-
-The `web_search()` tool can use Google's built-in search capability (called grounding) when running on Gemini 2.0 models and later. This provider does not require any API keys beyond what's needed for the model itself.
-
-This is distinct from the "google" provider (described below), which uses Google's external Programmable Search Engine service and requires separate API keys.
-
-For more details, see [Grounding with Google Search](https://ai.google.dev/gemini-api/docs/grounding).
-
-Note that when using the "gemini" provider, you should also specify a fallback external provider (like "tavily", "exa", or "google") if you are also running the evaluation with non-Gemini models.
-
-::: callout-warning
-Google's search grounding does not currently support use with other tools. Attempting to use `web_search("gemini")` alongside other tools will result in an error.
-:::
-
-### Tavily Options
-
-The `web_search()` tool can use [Tavily](https://tavily.com/)'s Research API. To use it you will need to set up your own Tavily account. Then, ensure that the following environment variable is defined:
-
--   `TAVILY_API_KEY` — Tavily Research API key
-
-Tavily supports the following options:
-
-| Option | Description |
-|------------------------------------|------------------------------------|
-| `max_results` | Number of results to return |
-| `search_depth` | Can be "basic" or "advanced" |
-| `topic` | Can be "general" or "news" |
-| `include_domains` / `exclude_domains` | Lists of domains to include or exclude |
-| `time_range` | Time range for search results (e.g., "day", "week", "month") |
-| `max_connections` | Maximum number of concurrent connections |
-
-For more options, see the [Tavily API Documentation](https://docs.tavily.com/documentation/api-reference/endpoint/search).
-
-### Exa Options
-
-The `web_search()` tool can use [Exa](https://exa.ai/)'s Answer API. To use it you will need to set up your own Exa account. Then, ensure that the following environment variable is defined:
-
--   `EXA_API_KEY` — Exa API key
-
-Exa supports the following options:
-
-| Option | Description |
-|------------------------------------|------------------------------------|
-| `text` | Whether to include text content in citations (defaults to true) |
-| `model` | LLM model to use for generating the answer ("exa" or "exa-pro") |
-| `max_connections` | Maximum number of concurrent connections |
-
-For more details, see the [Exa API Documentation](https://docs.exa.ai/reference/answer).
-
-#### Google Options
-
-The `web_search()` tool can use [Google Programmable Search Engine](https://programmablesearchengine.google.com/about/) as an external provider. This is different from the "gemini" provider (described above), which uses Google's built-in search capability for Gemini models.
-
-To use the "google" provider you will need to set up your own Google Programmable Search Engine and also enable the [Programmable Search Element Paid API](https://developers.google.com/custom-search/docs/paid_element). Then, ensure that the following environment variables are defined:
-
--   `GOOGLE_CSE_ID` — Google Custom Search Engine ID
--   `GOOGLE_CSE_API_KEY` — Google API key used to enable the Search API
-
-
-Google supports the following options: 
-
-| Option | Description |
-|------------------------------------|------------------------------------|
-| `num_results` | The number of relevant webpages whose contents are returned  |
-| `max_provider_calls` | Number of times to retrieve more links in case previous ones were irrelevant (defaults to 3)  |
-| `max_connections` | Maximum number of concurrent connections (defaults to 10) |
-| `model` | Model to use to determine if search results are relevant (defaults to the model being evaluated) |
 
 ## Think {#sec-think}
 
@@ -614,4 +614,4 @@ def swe_bench():
     )
 ```
 
-Note that the effectivess of using the system prompt will vary considerably across tasks, tools, and models, so should definitely be the subject of experimentation.
+Note that the effectivess of using the system prompt will vary considerably across tasks, tools, and models, so should definitely be the subject of experimentation.

{inspect_ai-0.3.104 → inspect_ai-0.3.105}/src/inspect_ai/_eval/evalset.py

@@ -578,7 +578,7 @@ def task_identifier(task: ResolvedTask | EvalLog) -> str:
     else:
         task_file = task.eval.task_file or ""
         task_name = task.eval.task
-        task_args = task.eval.task_args
+        task_args = task.eval.task_args_passed
         model = str(task.eval.model)
         model_roles = task.eval.model_roles or {}