inspect-ai 0.3.105__tar.gz → 0.3.107__tar.gz

{inspect_ai-0.3.105 → inspect_ai-0.3.107}/CHANGELOG.md

@@ -1,3 +1,16 @@
+## 0.3.107 (24 June 2025)
+
+- Bugfix: Shield critical shutdown code from cancel scope.
+
+## v0.3.106 (21 June 2025)
+
+- OpenAI: Use prefix matching when detecting compatible models for `web_search()`.
+- Groq: Capture `executed_tools` field as model output metadata.
+- ReAct agent: Always send `str` returned from `on_continue` to the model (formerly this was only done if there were no tool calls).
+- Web Search: Added provider for Perplexity's internal web search tool.
+- Eval: Wrap eval execution in TaskGroup.
+- Bugfix: Remove correlated reasoning content items when removing submit tool calls from ChatMessageAssistant instances in multi-agent scenarios.
+
 ## v0.3.105 (17 June 2025)
 
 - [background()](https://inspect.aisi.org.uk/agent-custom.html#background) function for executing work in the background of the current sample.

{inspect_ai-0.3.105 → inspect_ai-0.3.107}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: inspect_ai
-Version: 0.3.105
+Version: 0.3.107
 Summary: Framework for large language model evaluations
 Author: UK AI Security Institute
 License: MIT License

{inspect_ai-0.3.105 → inspect_ai-0.3.107}/docs/CHANGELOG.md

@@ -1,3 +1,16 @@
+## 0.3.107 (24 June 2025)
+
+- Bugfix: Shield critical shutdown code from cancel scope.
+
+## v0.3.106 (21 June 2025)
+
+- OpenAI: Use prefix matching when detecting compatible models for `web_search()`.
+- Groq: Capture `executed_tools` field as model output metadata.
+- ReAct agent: Always send `str` returned from `on_continue` to the model (formerly this was only done if there were no tool calls).
+- Web Search: Added provider for Perplexity's internal web search tool.
+- Eval: Wrap eval execution in TaskGroup.
+- Bugfix: Remove correlated reasoning content items when removing submit tool calls from ChatMessageAssistant instances in multi-agent scenarios.
+
 ## v0.3.105 (17 June 2025)
 
 - [background()](https://inspect.aisi.org.uk/agent-custom.html#background) function for executing work in the background of the current sample.

{inspect_ai-0.3.105 → inspect_ai-0.3.107}/docs/agent-bridge.qmd

@@ -237,4 +237,4 @@ from inspect_ai.log import transcript
 transcript().info("custom *markdown* content")
 ```
 
-This code will no-op when running outside of Inspect to it is safe to include in agents that are also run in other environments.
+This code will no-op when running outside of Inspect, so it is safe to include in agents that are also run in other environments.

{inspect_ai-0.3.105 → inspect_ai-0.3.107}/docs/react-agent.qmd

@@ -110,11 +110,22 @@ In some cases models in a tool use loop will simply fail to call a tool (or just
 This behaviour is controlled by the `on_continue` parameter, which by default yields the following user message to the model:
 
 ``` default
-Please proceed to the next step using your best judgement. If you believe you
-have completed the task, please call the `submit()` tool with your final answer
+Please proceed to the next step using your best judgement. 
+If you believe you have completed the task, please call the 
+`submit()` tool with your final answer,
 ```
 
-You can pass a different continuation message, or alternative pass an `AgentContinue` function that can dynamically determine both whether to continue and what the message is.
+You can pass a different continuation message, or alternatively pass an `AgentContinue` function that can dynamically determine both whether to continue and what the message is. Here is how `on_continue` affects the agent loop for various inputs:
+
+- `None`: A default user message will be appended only when there are no tool calls made by the model.
+
+- `str`: The returned user message will be appended only when there are no tool calls made by the model.
+
+- `Callable`: the function passed can return one of:
+  - `True`: Agent loop continues with no messages appended.
+  - `False`: Agent loop is exited early.
+  - `str`: Agent loop continues and the returned user message will be appended regardless of whether a tool call was made in the previous assistant message. If your custom function only wants to append a message when there are no tool calls made then you should check `state.output.message.tool_calls` explicitly (returning `True` rather than `str` when you want no message appended).
+
 
 ### Submit Tool
 

{inspect_ai-0.3.105 → inspect_ai-0.3.107}/docs/tools-standard.qmd

@@ -12,7 +12,7 @@ tbl-colwidths: [40,60]
 
 The `web_search()` tool provides models the ability to enhance their context window by performing a search. Web searches are executed using a provider. Providers are split into two categories:
 
--   Internal providers: `"openai"`, `"anthropic"`, and `"gemini"` - these use the model's built-in search capability and do not require separate API keys. These work only for their respective model provider (e.g. the "openai" search provider works only for `openai/*` models).
+-   Internal providers: `"openai"`, `"anthropic"`, `"gemini"`, and `"perplexity"` - these use the model's built-in search capability and do not require separate API keys. These work only for their respective model provider (e.g. the "openai" search provider works only for `openai/*` models).
 
 -   External providers: `"tavily"`, `"exa"`, and `"google"`. These are external services that work with any model and require separate accounts and API keys. Note that "google" is different from "gemini" - "google" refers to Google's Programmable Search Engine service, while "gemini" refers to Google's built-in search capability for Gemini models.
 
@@ -30,7 +30,7 @@ web_search("tavily")
 web_search(["openai", "tavily"])
 
 # multiple internal providers and fallback
-web_search(["openai", "anthropic", "gemini", "tavily"])
+web_search(["openai", "anthropic", "gemini", "perplexity", "tavily"])
 
 # provider with specific options
 web_search({"tavily": {"max_results": 5}})
@@ -73,6 +73,14 @@ Note that when using the "gemini" provider, you should also specify a fallback e
 Google's search grounding does not currently support use with other tools. Attempting to use `web_search("gemini")` alongside other tools will result in an error.
 :::
 
+### Perplexity Options
+
+The `web_search()` tool can use Perplexity's built-in search capability when running on Perplexity models. This provider does not require any API keys beyond what's needed for the model itself. Search parameters can be passed using the `perplexity` provider options and will be forwarded to the model API.
+
+For more details, see [Perplexity API Documentation](https://docs.perplexity.ai/api-reference/chat-completions-post).
+
+Note that when using the "perplexity" provider, you should also specify a fallback external provider (like "tavily", "exa", or "google") if you are also running the evaluation with non-Perplexity models.
+
 ### Tavily Options
 
 The `web_search()` tool can use [Tavily](https://tavily.com/)'s Research API. To use it you will need to set up your own Tavily account. Then, ensure that the following environment variable is defined:

{inspect_ai-0.3.105 → inspect_ai-0.3.107}/src/inspect_ai/_eval/context.py

@@ -1,4 +1,7 @@
+from anyio.abc import TaskGroup
+
 from inspect_ai._util.dotenv import init_dotenv
+from inspect_ai._util.eval_task_group import init_eval_task_group
 from inspect_ai._util.hooks import init_hooks
 from inspect_ai._util.logger import init_logger
 from inspect_ai.approval._apply import have_tool_approval, init_tool_approval
@@ -19,6 +22,7 @@ def init_eval_context(
     log_level: str | None,
     log_level_transcript: str | None,
     max_subprocesses: int | None = None,
+    task_group: TaskGroup | None = None,
 ) -> None:
     init_dotenv()
     init_logger(log_level, log_level_transcript)
@@ -27,6 +31,7 @@ def init_eval_context(
     init_hooks()
     init_active_samples()
     init_human_approval_manager()
+    init_eval_task_group(task_group)
 
 
 def init_task_context(

{inspect_ai-0.3.105 → inspect_ai-0.3.107}/src/inspect_ai/_eval/eval.py

@@ -4,11 +4,15 @@ import sys
 from pathlib import Path
 from typing import Any, Literal, cast
 
+import anyio
+from anyio.abc import TaskGroup
+
 from inspect_ai._eval.task.task import resolve_model_roles
 from inspect_ai._util.notgiven import NOT_GIVEN, NotGiven
 from inspect_ai.agent._agent import Agent, is_agent
 from inspect_ai.agent._as_solver import as_solver
 from inspect_ai.log._model import model_roles_config_to_model_roles
+from inspect_ai.util._anyio import inner_exception
 
 if sys.version_info < (3, 11):
     from exceptiongroup import ExceptionGroup
@@ -359,6 +363,112 @@ async def eval_async(
     Returns:
         List of EvalLog (one for each task)
     """
+    result: list[EvalLog] | None = None
+
+    async def run(tg: TaskGroup) -> None:
+        try:
+            nonlocal result
+            result = await _eval_async_inner(
+                tg=tg,
+                tasks=tasks,
+                model=model,
+                model_base_url=model_base_url,
+                model_args=model_args,
+                model_roles=model_roles,
+                task_args=task_args,
+                sandbox=sandbox,
+                sandbox_cleanup=sandbox_cleanup,
+                solver=solver,
+                tags=tags,
+                metadata=metadata,
+                approval=approval,
+                log_level=log_level,
+                log_level_transcript=log_level_transcript,
+                log_dir=log_dir,
+                log_format=log_format,
+                limit=limit,
+                sample_id=sample_id,
+                epochs=epochs,
+                fail_on_error=fail_on_error,
+                retry_on_error=retry_on_error,
+                debug_errors=debug_errors,
+                message_limit=message_limit,
+                token_limit=token_limit,
+                time_limit=time_limit,
+                working_limit=working_limit,
+                max_samples=max_samples,
+                max_tasks=max_tasks,
+                max_subprocesses=max_subprocesses,
+                max_sandboxes=max_sandboxes,
+                log_samples=log_samples,
+                log_realtime=log_realtime,
+                log_images=log_images,
+                log_buffer=log_buffer,
+                log_shared=log_shared,
+                log_header_only=log_header_only,
+                score=score,
+                score_display=score_display,
+                **kwargs,
+            )
+        finally:
+            tg.cancel_scope.cancel()
+
+    try:
+        async with anyio.create_task_group() as tg:
+            tg.start_soon(run, tg)
+    except Exception as ex:
+        raise inner_exception(ex)
+    except anyio.get_cancelled_exc_class():
+        # Cancelled exceptions are expected and handled by _eval_async_inner
+        pass
+
+    assert result is not None, "Eval async did not return a result."
+
+    return result
+
+
+async def _eval_async_inner(
+    tg: TaskGroup,
+    tasks: Tasks,
+    model: str | Model | list[str] | list[Model] | None | NotGiven = NOT_GIVEN,
+    model_base_url: str | None = None,
+    model_args: dict[str, Any] | str = dict(),
+    model_roles: dict[str, str | Model] | None = None,
+    task_args: dict[str, Any] | str = dict(),
+    sandbox: SandboxEnvironmentType | None = None,
+    sandbox_cleanup: bool | None = None,
+    solver: Solver | SolverSpec | Agent | list[Solver] | None = None,
+    tags: list[str] | None = None,
+    metadata: dict[str, Any] | None = None,
+    approval: str | list[ApprovalPolicy] | ApprovalPolicyConfig | None = None,
+    log_level: str | None = None,
+    log_level_transcript: str | None = None,
+    log_dir: str | None = None,
+    log_format: Literal["eval", "json"] | None = None,
+    limit: int | tuple[int, int] | None = None,
+    sample_id: str | int | list[str] | list[int] | list[str | int] | None = None,
+    epochs: int | Epochs | None = None,
+    fail_on_error: bool | float | None = None,
+    retry_on_error: int | None = None,
+    debug_errors: bool | None = None,
+    message_limit: int | None = None,
+    token_limit: int | None = None,
+    time_limit: int | None = None,
+    working_limit: int | None = None,
+    max_samples: int | None = None,
+    max_tasks: int | None = None,
+    max_subprocesses: int | None = None,
+    max_sandboxes: int | None = None,
+    log_samples: bool | None = None,
+    log_realtime: bool | None = None,
+    log_images: bool | None = None,
+    log_buffer: int | None = None,
+    log_shared: bool | int | None = None,
+    log_header_only: bool | None = None,
+    score: bool = True,
+    score_display: bool | None = None,
+    **kwargs: Unpack[GenerateConfigArgs],
+) -> list[EvalLog]:
     # only a single call to eval_async can be active at a time, this used
     # to be due to running tasks switching to the task's directory, however
     # that feature no longer exists so we may be able to revisit this
@@ -387,6 +497,7 @@ async def eval_async(
             max_subprocesses=max_subprocesses,
             log_level=log_level,
             log_level_transcript=log_level_transcript,
+            task_group=tg,
             **kwargs,
         )
 
@@ -934,10 +1045,11 @@ def eval_init(
     max_subprocesses: int | None = None,
     log_level: str | None = None,
     log_level_transcript: str | None = None,
+    task_group: TaskGroup | None = None,
     **kwargs: Unpack[GenerateConfigArgs],
 ) -> list[Model]:
     # init eval context
-    init_eval_context(log_level, log_level_transcript, max_subprocesses)
+    init_eval_context(log_level, log_level_transcript, max_subprocesses, task_group)
 
     # resolve model and task args
     model_args = resolve_args(model_args)

{inspect_ai-0.3.105 → inspect_ai-0.3.107}/src/inspect_ai/_eval/run.py

@@ -522,15 +522,16 @@ async def startup_sandbox_environments(
 
     # return shutdown method
     async def shutdown() -> None:
-        for cleanup_jobs in cleanups:
-            try:
-                cleanup_fn, config, task_run_dir = cleanup_jobs
-                with chdir(task_run_dir):
-                    await cleanup_fn("shutdown", config, cleanup)
-            except BaseException as ex:
-                log.warning(
-                    f"Error occurred shutting down sandbox environments: {exception_message(ex)}"
-                )
+        with anyio.CancelScope(shield=True):
+            for cleanup_jobs in cleanups:
+                try:
+                    cleanup_fn, config, task_run_dir = cleanup_jobs
+                    with chdir(task_run_dir):
+                        await cleanup_fn("shutdown", config, cleanup)
+                except BaseException as ex:
+                    log.warning(
+                        f"Error occurred shutting down sandbox environments: {exception_message(ex)}"
+                    )
 
     return shutdown
 

inspect_ai-0.3.107/src/inspect_ai/_util/eval_task_group.py

@@ -0,0 +1,15 @@
+from anyio.abc import TaskGroup
+
+_eval_task_group: TaskGroup | None = None
+
+
+def init_eval_task_group(tg: TaskGroup | None) -> None:
+    global _eval_task_group
+    _eval_task_group = tg
+
+
+def eval_task_group() -> TaskGroup:
+    global _eval_task_group
+    if _eval_task_group is None:
+        raise RuntimeError("Task group has not been initialized")
+    return _eval_task_group

{inspect_ai-0.3.105 → inspect_ai-0.3.107}/src/inspect_ai/agent/_react.py

@@ -82,9 +82,8 @@ def react(
           the submit tool within the message. Alternatively, an async function
           to call to determine whether the loop should continue and what message
           to play back. Note that this function is called on _every_ iteration of
-          the loop. If you return a `str` with a custom message it will be
-          sent to the model as a user message only in the case that no tool
-          calls were made.
+          the loop so if you only want to send a message back when the model fails
+          to call tools you need to code that behavior explicitly.
        truncation: Truncate the conversation history in the event of a context
           window overflow. Defaults to "disabled" which does no truncation. Pass
           "auto" to use `trim_messages()` to reduce the context size. Pass a
@@ -246,13 +245,12 @@ def react(
                                 )
                             )
                     elif isinstance(do_continue, str):
-                        # if there were no tool calls we need to send back the user message
-                        if not state.output.message.tool_calls:
-                            state.messages.append(
-                                ChatMessageUser(
-                                    content=do_continue.format(submit=submit_tool.name)
-                                )
+                        # send back the user message
+                        state.messages.append(
+                            ChatMessageUser(
+                                content=do_continue.format(submit=submit_tool.name)
                             )
+                        )
                     else:  # do_continue is False
                         break
 
@@ -328,11 +326,14 @@ def react_no_submit(
                 if on_continue:
                     do_continue = await _call_on_continue(on_continue, state)
                     if do_continue is True:
-                        do_continue = DEFAULT_CONTINUE_PROMOT_NO_SUBMIT
-                    if do_continue:
-                        # send back user message if there are no tool calls
                         if not state.output.message.tool_calls:
-                            state.messages.append(ChatMessageUser(content=do_continue))
+                            state.messages.append(
+                                ChatMessageUser(
+                                    content=DEFAULT_CONTINUE_PROMOT_NO_SUBMIT
+                                )
+                            )
+                    elif isinstance(do_continue, str):
+                        state.messages.append(ChatMessageUser(content=do_continue))
                     else:
                         break
                 elif not state.output.message.tool_calls:
@@ -471,12 +472,34 @@ def _remove_submit_tool(
 
         # remove submit tool from assistant messages
         if isinstance(message, ChatMessageAssistant) and message.tool_calls:
-            tools_calls = [
+            new_tools_calls = [
                 tool_call
                 for tool_call in message.tool_calls
                 if tool_call.function != submit_name
             ]
-            message = message.model_copy(update=dict(tool_calls=tools_calls))
+
+            # If a submit tool call was removed, we need to update the message
+            if len(new_tools_calls) < len(message.tool_calls):
+                message = message.model_copy(
+                    update=dict(
+                        tool_calls=new_tools_calls,
+                        # Some models (OpenAI) don't like to see the reasoning
+                        # content item that led to the submit tool call, so we
+                        # have to remove it too.
+                        content=(
+                            [
+                                content
+                                for content in message.content
+                                if (
+                                    isinstance(content, str)
+                                    or content.type != "reasoning"
+                                )
+                            ]
+                            if isinstance(message.content, list)
+                            else message.content
+                        ),
+                    )
+                )
 
         # always append message
         filtered.append(message)

{inspect_ai-0.3.105 → inspect_ai-0.3.107}/src/inspect_ai/model/_providers/_openai_web_search.py

@@ -14,7 +14,7 @@ def maybe_web_search_tool(model_name: str, tool: ToolInfo) -> WebSearchToolParam
             tool.name == "web_search"
             and tool.options
             and "openai" in tool.options
-            and model_name in COMPATIBLE_MODELS
+            and any(model_name.startswith(model) for model in COMPATIBLE_MODELS)
         )
         else None
     )

{inspect_ai-0.3.105 → inspect_ai-0.3.107}/src/inspect_ai/model/_providers/groq.py

@@ -156,6 +156,11 @@ class GroqAPI(ModelAPI):
                     "completion_time": completion.usage.completion_time,
                     "total_time": completion.usage.total_time,
                 }
+            if completion.choices[0].message.executed_tools:
+                metadata["executed_tools"] = [
+                    tool.model_dump()
+                    for tool in completion.choices[0].message.executed_tools
+                ]
 
             # extract output
             choices = self._chat_choices_from_response(completion, tools)

{inspect_ai-0.3.105 → inspect_ai-0.3.107}/src/inspect_ai/model/_providers/perplexity.py

@@ -49,7 +49,33 @@ class PerplexityAPI(OpenAICompatibleAPI):
         tool_choice: "ToolChoice",
         config: GenerateConfig,
     ) -> tuple[ModelOutput | Exception, "ModelCall"]:
-        result = await super().generate(input, tools, tool_choice, config)
+        search_options: dict[str, Any] | None = None
+        for tool in tools:
+            if (
+                tool.name == "web_search"
+                and tool.options
+                and "perplexity" in tool.options
+            ):
+                maybe_opts = tool.options["perplexity"]
+                if maybe_opts is not None:
+                    if maybe_opts is True:
+                        search_options = {}
+                    elif isinstance(maybe_opts, dict):
+                        search_options = maybe_opts
+                    else:
+                        raise TypeError(
+                            f"Expected a dictionary or True for perplexity_options, got {type(maybe_opts)}"
+                        )
+            else:
+                raise ValueError(
+                    "Perplexity does not support tools other than web_search with perplexity options"
+                )
+
+        if search_options:
+            extra_body = {**(config.extra_body or {}), **search_options}
+            config = config.merge(GenerateConfig(extra_body=extra_body))
+
+        result = await super().generate(input, [], tool_choice, config)
         output, call = cast(tuple[ModelOutput, "ModelCall"], result)
 
         if self._response:

{inspect_ai-0.3.105 → inspect_ai-0.3.107}/src/inspect_ai/model/_providers/providers.py

@@ -16,7 +16,7 @@ from .._registry import modelapi
 def groq() -> type[ModelAPI]:
     FEATURE = "Groq API"
     PACKAGE = "groq"
-    MIN_VERSION = "0.16.0"
+    MIN_VERSION = "0.28.0"
 
     # verify we have the package
     try:

{inspect_ai-0.3.105 → inspect_ai-0.3.107}/src/inspect_ai/tool/_tools/_web_search/_web_search.py

@@ -18,7 +18,7 @@ from ._tavily import TavilyOptions, tavily_search_provider
 from ._web_search_provider import SearchProvider
 
 Provider: TypeAlias = Literal[
-    "gemini", "openai", "anthropic", "tavily", "google", "exa"
+    "gemini", "openai", "anthropic", "perplexity", "tavily", "google", "exa"
 ]
 valid_providers = set(get_args(Provider))
 
@@ -35,6 +35,7 @@ class Providers(TypedDict, total=False):
     openai: dict[str, Any] | Literal[True]
     anthropic: dict[str, Any] | Literal[True]
     gemini: dict[str, Any] | Literal[True]
+    perplexity: dict[str, Any] | Literal[True]
     tavily: dict[str, Any] | Literal[True]
     google: dict[str, Any] | Literal[True]
     exa: dict[str, Any] | Literal[True]
@@ -44,6 +45,7 @@ class _NormalizedProviders(TypedDict, total=False):
     openai: dict[str, Any]
     anthropic: dict[str, Any]
     gemini: dict[str, Any]
+    perplexity: dict[str, Any]
     tavily: dict[str, Any]
     google: dict[str, Any]
     exa: dict[str, Any]
@@ -67,7 +69,7 @@ def web_search(
     Web searches are executed using a provider. Providers are split
     into two categories:
 
-    - Internal providers: "openai", "anthropic" - these use the model's built-in
+    - Internal providers: "openai", "anthropic", "gemini", "perplexity" - these use the model's built-in
       search capability and do not require separate API keys. These work only for
       their respective model provider (e.g. the "openai" search provider
       works only for `openai/*` models).
@@ -84,7 +86,7 @@ def web_search(
 
     Args:
       providers: Configuration for the search providers to use. Currently supported
-        providers are "openai", "anthropic", "tavily", "google", and "exa". The
+        providers are "openai", "anthropic", "perplexity", "tavily", "google", and "exa". The
         `providers` parameter supports several formats based on either a `str`
         specifying a provider or a `dict` whose keys are the provider names and
         whose values are the provider-specific options. A single value or a list
@@ -121,6 +123,9 @@ def web_search(
         - anthropic: Supports Anthropic's web search parameters.
           See https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/web-search-tool#tool-definition
 
+        - perplexity: Supports Perplexity's web search parameters.
+          See https://docs.perplexity.ai/api-reference/chat-completions-post
+
         - tavily: Supports options like `max_results`, `search_depth`, etc.
           See https://docs.tavily.com/documentation/api-reference/endpoint/search
 

{inspect_ai-0.3.105 → inspect_ai-0.3.107}/src/inspect_ai.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: inspect_ai
-Version: 0.3.105
+Version: 0.3.107
 Summary: Framework for large language model evaluations
 Author: UK AI Security Institute
 License: MIT License

{inspect_ai-0.3.105 → inspect_ai-0.3.107}/src/inspect_ai.egg-info/SOURCES.txt

@@ -310,6 +310,7 @@ src/inspect_ai/_util/dotenv.py
 src/inspect_ai/_util/entrypoints.py
 src/inspect_ai/_util/environ.py
 src/inspect_ai/_util/error.py
+src/inspect_ai/_util/eval_task_group.py
 src/inspect_ai/_util/exception.py
 src/inspect_ai/_util/file.py
 src/inspect_ai/_util/format.py