inspect-ai 0.3.93__tar.gz → 0.3.94__tar.gz

{inspect_ai-0.3.93 → inspect_ai-0.3.94}/.pylintrc

@@ -5,5 +5,5 @@
 ; E - Error
 enable=E
 ; We are starting conservatively disabling non-errors and a handful of currently noisy E's
-disable=R,W,C,E0401,E1101,E0611,E1128,E0606
+disable=R,W,C,E0401,E1101,E0611,E1128,E0606,E1135
 score=no

{inspect_ai-0.3.93 → inspect_ai-0.3.94}/CHANGELOG.md

@@ -1,3 +1,15 @@
+## v0.3.94 (06 May 2025)
+
+- [span()](https://inspect.aisi.org.uk/agent-custom.html#grouping-with-spans) function for grouping transcript events.
+- [collect()](https://inspect.aisi.org.uk/agent-custom.html#grouping-with-spans) function for enclosing parallel tasks in spans.
+- [Event tree](https://inspect.aisi.org.uk/reference/inspect_ai.log.html#event-tree) functions for organising transcript events into a tree of spans.
+- `inspect convert` now always fully re-writes log files even of the same format (so that e.g. sample summaries always exist in the converted logs).
+- React agent: `answer_only` and `answer_delimiter` to control how submitted answers are reflected in the assistant message content. 
+- Python tool: Execute using a bash login shell for consistency of Python versions across `bash()` and `python()` tools.
+- Task display: Realtime display of events that occur within tool calls and subtasks.
+- Multiple choice: Support for more than 26 choices.
+- Bugfix: Ensure that each MCP server gets its own cached tool list.
+
 ## v0.3.93 (01 May 2025)
 
 - [Scoped Limits](https://inspect.aisi.org.uk/errors-and-limits.html#scoped-limits) for enforcing token and message limits using a context manager.

{inspect_ai-0.3.93 → inspect_ai-0.3.94}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: inspect_ai
-Version: 0.3.93
+Version: 0.3.94
 Summary: Framework for large language model evaluations
 Author: UK AI Security Institute
 License: MIT License

{inspect_ai-0.3.93 → inspect_ai-0.3.94}/docs/agent-custom.qmd

@@ -297,20 +297,45 @@ transcript().info("here is some custom info")
 
 Strings passed to `info()` will be rendered as markdown. In addition to strings you can also pass arbitrary JSON serialisable objects to `info()`.
 
-### Grouping with Steps
+### Grouping with Spans
 
-You can create arbitrary groupings of transcript activity using the Transcript `step()` context manager. For example:
+You can create arbitrary groupings of transcript activity using the  `span()` context manager. For example:
 
 ``` python
-with transcript().step("reasoning"):
+from inspect_ai.util import span
+
+async with span("planning"):
     ...
-    state.store.set("next-action", next_action)
 ```
 
-There are two reasons that you might want to create steps:
+There are two reasons that you might want to create spans:
+
+1.  Any changes to the store which occur during a span will be collected into a `StoreEvent` that records the changes (in [JSON Patch](https://jsonpatch.com/) format) that occurred.
+2.  The Inspect log viewer will create a visual delineation for the span, which will make it easier to see the flow of activity within the transcript.
+
+Spans are automatically created for sample initialisation, solvers, scorers, subtasks, tool calls, and agent execution.
+
+
+## Parallel Execution
+
+You can execute subtasks in parallel using the `collect()` function. For example, to run 3 `web_search()` coroutines in parallel:
+
+``` python
+from inspect_ai.util import collect
+
+results = collect(
+  web_search(keywords="solar power"),
+  web_search(keywords="wind power"),
+  web_search(keywords="hydro power"),
+)
+```
+
+Note that `collect()` is similar to [`asyncio.gather()`](https://docs.python.org/3/library/asyncio-task.html#asyncio.gather),
+but also works when [Trio](https://trio.readthedocs.io/en/stable/) is the Inspect async backend.
+
+The Inspect `collect()` function also automatically includes each task in a `span()`, which ensures that its events are grouped together in the transcript.
 
-1.  Any changes to the store which occur during a step will be collected into a `StoreEvent` that records the changes (in [JSON Patch](https://jsonpatch.com/) format) that occurred.
-2.  The Inspect log viewer will create a visual delineation for the step, which will make it easier to see the flow of activity within the transcript.
+Using `collect()` in preference to `asyncio.gather()` is highly recommended for both Trio compatibility and more legible transcript output.
 
 ## Subtasks {#sec-subtasks}
 
@@ -318,7 +343,6 @@ Subtasks provide a mechanism for creating isolated, re-usable units of execution
 
 1.  They run in their own async coroutine.
 2.  They have their own isolated `Store` (no access to the sample `Store`).
-3.  They have their own isolated `Transcript`
 
 To create a subtask, declare an async function with the `@subtask` decorator. The function can take any arguments and return a value of any type. For example:
 
@@ -347,26 +371,8 @@ summary = await web_search(keywords="solar power")
 
 A few things will occur automatically when you run a subtask:
 
--   New isolated `Store` and `Transcript` objects will be created for the subtask (accessible via the `store()` and `transcript()` functions). Changes to the `Store` that occur during execution will be recorded in a `StoreEvent`.
+-   A new isolated `Store` object will be created for the subtask (accessible via the `store()` function). Changes to the `Store` that occur during execution will be recorded in a `StoreEvent`.
 
 -   A `SubtaskEvent` will be added to the current transcript. The event will include the name of the subtask, its input and results, and a transcript of all events that occur within the subtask.
 
-You can also include one or more steps within a subtask.
-
-### Parallel Execution
-
-You can execute subtasks in parallel using `asyncio.gather()`. For example, to run 3 `web_search()` subtasks in parallel:
-
-``` python
-import asyncio
-
-searches = [
-  web_search(keywords="solar power"),
-  web_search(keywords="wind power"),
-  web_search(keywords="hydro power"),
-]
-
-results = await asyncio.gather(*searches)
-```
-
-Note that we don't `await` the subtasks when building up our list of `searches`. Rather, we let `asyncio.gather()` await all of them, returning only when all of the results are available.
+You can also include one or more spans within a subtask.

{inspect_ai-0.3.93 → inspect_ai-0.3.94}/docs/reference/_sidebar.yml

@@ -415,6 +415,16 @@ website:
           href: reference/inspect_ai.log.qmd#transcript
         - text: Event
           href: reference/inspect_ai.log.qmd#event
+        - text: event_tree
+          href: reference/inspect_ai.log.qmd#event_tree
+        - text: event_sequence
+          href: reference/inspect_ai.log.qmd#event_sequence
+        - text: EventTree
+          href: reference/inspect_ai.log.qmd#eventtree
+        - text: EventNode
+          href: reference/inspect_ai.log.qmd#eventnode
+        - text: SpanNode
+          href: reference/inspect_ai.log.qmd#spannode
         - text: SampleInitEvent
           href: reference/inspect_ai.log.qmd#sampleinitevent
         - text: SampleLimitEvent
@@ -433,8 +443,6 @@ website:
           href: reference/inspect_ai.log.qmd#approvalevent
         - text: InputEvent
           href: reference/inspect_ai.log.qmd#inputevent
-        - text: StoreEvent
-          href: reference/inspect_ai.log.qmd#storeevent
         - text: ErrorEvent
           href: reference/inspect_ai.log.qmd#errorevent
         - text: LoggerEvent
@@ -445,8 +453,10 @@ website:
           href: reference/inspect_ai.log.qmd#loggingmessage
         - text: InfoEvent
           href: reference/inspect_ai.log.qmd#infoevent
-        - text: StepEvent
-          href: reference/inspect_ai.log.qmd#stepevent
+        - text: SpanBeginEvent
+          href: reference/inspect_ai.log.qmd#spanbeginevent
+        - text: SpanEndEvent
+          href: reference/inspect_ai.log.qmd#spanendevent
         - text: SubtaskEvent
           href: reference/inspect_ai.log.qmd#subtaskevent
       - section: inspect_ai.util
@@ -488,6 +498,10 @@ website:
           href: reference/inspect_ai.util.qmd#subtask
         - text: Subtask
           href: reference/inspect_ai.util.qmd#subtask
+        - text: span
+          href: reference/inspect_ai.util.qmd#span
+        - text: collect
+          href: reference/inspect_ai.util.qmd#collect
         - text: resource
           href: reference/inspect_ai.util.qmd#resource
         - text: throttle

{inspect_ai-0.3.93 → inspect_ai-0.3.94}/docs/reference/inspect_ai.log.qmd

@@ -45,6 +45,11 @@ title: "inspect_ai.log"
 ### transcript
 ### Transcript
 ### Event
+### event_tree
+### event_sequence
+### EventTree
+### EventNode
+### SpanNode
 ### SampleInitEvent
 ### SampleLimitEvent
 ### StateEvent
@@ -54,13 +59,13 @@ title: "inspect_ai.log"
 ### SandboxEvent
 ### ApprovalEvent
 ### InputEvent
-### StoreEvent
 ### ErrorEvent
 ### LoggerEvent
 ### LoggingLevel
 ### LoggingMessage
 ### InfoEvent
-### StepEvent
+### SpanBeginEvent
+### SpanEndEvent
 ### SubtaskEvent
 
 
@@ -71,3 +76,4 @@ title: "inspect_ai.log"
 
 
 
+

{inspect_ai-0.3.93 → inspect_ai-0.3.94}/docs/reference/inspect_ai.util.qmd

@@ -2,7 +2,6 @@
 title: "inspect_ai.util"
 ---
 
-
 ## Store
 
 ### Store
@@ -38,6 +37,8 @@ title: "inspect_ai.util"
 
 ## Utilities
 
+### span
+### collect
 ### resource
 ### throttle
 ### trace_action

{inspect_ai-0.3.93 → inspect_ai-0.3.94}/docs/tasks.qmd

@@ -19,7 +19,7 @@ Tasks provide a recipe for an evaluation consisting minimally of a dataset, a so
 
 ``` python
 from inspect_ai import Task, task
-from inspect_ai.dataset import json_datasets
+from inspect_ai.dataset import json_dataset
 from inspect_ai.scorer import model_graded_fact
 from inspect_ai.solver import chain_of_thought, generate
 
@@ -409,4 +409,4 @@ plot_results(logs)
 
 Note that we also pass a list of `model` to try out the task on multiple models. This eval set will produce in total 16 tasks accounting for the parameter and model variation.
 
-See the article on [Eval Sets](eval-sets.qmd) to learn more about using eval sets. See the article on [Eval Logs](eval-logs.qmd) for additional details on working with evaluation logs.
+See the article on [Eval Sets](eval-sets.qmd) to learn more about using eval sets. See the article on [Eval Logs](eval-logs.qmd) for additional details on working with evaluation logs.

{inspect_ai-0.3.93 → inspect_ai-0.3.94}/docs/tools-mcp.qmd

@@ -271,10 +271,11 @@ You are not required to inherit from the `aisiuk/inspect-tool-support` base imag
 If you are inheriting from a different base image then you can equivalently satisfy the `inspect-tool-support` dependency by adding this to your Dockerfile:
 
 ``` dockerfile
-ENV PATH="$PATH:/opt/inspect_tool_support/bin"
-RUN python -m venv /opt/inspect_tool_support && \
-    /opt/inspect_tool_support/bin/pip install inspect-tool-support && \
-    /opt/inspect_tool_support/bin/inspect-tool-support post-install
+RUN apt-get update && apt-get install -y pipx && \
+    apt-get clean && rm -rf /var/lib/apt/lists/* && \
+    pipx ensurepath
+ENV PATH="$PATH:/root/.local/bin"
+RUN pipx install inspect-tool-support && inspect-tool-support post-install
 ```
 :::
 

{inspect_ai-0.3.93 → inspect_ai-0.3.94}/src/inspect_ai/_display/textual/widgets/samples.py

@@ -591,10 +591,10 @@ class SampleToolbar(Horizontal):
                 )
                 if isinstance(last_event, ModelEvent):
                     # see if there are retries in play
-                    if sample.retry_count > 0:
-                        suffix = "retry" if sample.retry_count == 1 else "retries"
+                    if last_event.retries:
+                        suffix = "retry" if last_event.retries == 1 else "retries"
                         pending_caption_text = (
-                            f"Generating ({sample.retry_count:,} {suffix})..."
+                            f"Generating ({last_event.retries:,} {suffix})..."
                         )
                     else:
                         pending_caption_text = "Generating..."

{inspect_ai-0.3.93 → inspect_ai-0.3.94}/src/inspect_ai/_display/textual/widgets/transcript.py

@@ -30,7 +30,7 @@ from inspect_ai.log._transcript import (
     SampleInitEvent,
     SampleLimitEvent,
     ScoreEvent,
-    StepEvent,
+    SpanBeginEvent,
     SubtaskEvent,
     ToolEvent,
 )
@@ -211,10 +211,6 @@ def render_tool_event(event: ToolEvent) -> list[EventDisplay]:
     # render the call
     content = transcript_tool_call(event)
 
-    # render sub-events
-    if event.events:
-        content.extend(render_sub_events(event.events))
-
     # render the output
     if isinstance(event.result, list):
         result: ToolResult = "\n".join(
@@ -235,23 +231,6 @@ def render_tool_event(event: ToolEvent) -> list[EventDisplay]:
     return [EventDisplay("tool call", Group(*content))]
 
 
-def render_step_event(event: StepEvent) -> EventDisplay:
-    if event.type == "solver":
-        return render_solver_event(event)
-    if event.type == "scorer":
-        return render_scorer_event(event)
-    else:
-        return EventDisplay(step_title(event))
-
-
-def render_solver_event(event: StepEvent) -> EventDisplay:
-    return EventDisplay(step_title(event))
-
-
-def render_scorer_event(event: StepEvent) -> EventDisplay:
-    return EventDisplay(step_title(event))
-
-
 def render_score_event(event: ScoreEvent) -> EventDisplay:
     table = Table(box=None, show_header=False)
     table.add_column("", min_width=10, justify="left")
@@ -272,10 +251,6 @@ def render_subtask_event(event: SubtaskEvent) -> list[EventDisplay]:
     # render header
     content: list[RenderableType] = [transcript_function(event.name, event.input)]
 
-    # render sub-events
-    if event.events:
-        content.extend(render_sub_events(event.events))
-
     if event.result:
         content.append(Text())
         if isinstance(event.result, str | int | float | bool | None):
@@ -345,8 +320,8 @@ def render_message(message: ChatMessage) -> list[RenderableType]:
     return content
 
 
-def step_title(event: StepEvent) -> str:
-    return f"{event.type or 'step'}: {event.name}"
+def span_title(event: SpanBeginEvent) -> str:
+    return f"{event.type or 'span'}: {event.name}"
 
 
 EventRenderer = Callable[[Any], EventDisplay | list[EventDisplay] | None]
@@ -354,7 +329,6 @@ EventRenderer = Callable[[Any], EventDisplay | list[EventDisplay] | None]
 _renderers: list[tuple[Type[Event], EventRenderer]] = [
     (SampleInitEvent, render_sample_init_event),
     (SampleLimitEvent, render_sample_limit_event),
-    (StepEvent, render_step_event),
     (ModelEvent, render_model_event),
     (ToolEvent, render_tool_event),
     (SubtaskEvent, render_subtask_event),

{inspect_ai-0.3.93 → inspect_ai-0.3.94}/src/inspect_ai/_eval/task/run.py

@@ -24,7 +24,6 @@ from inspect_ai._util._async import tg_collect
 from inspect_ai._util.constants import (
     DEFAULT_EPOCHS,
     DEFAULT_MAX_CONNECTIONS,
-    SAMPLE_SUBTASK,
 )
 from inspect_ai._util.datetime import iso_now
 from inspect_ai._util.error import exception_message
@@ -65,8 +64,8 @@ from inspect_ai.log._transcript import (
     SampleInitEvent,
     SampleLimitEvent,
     ScoreEvent,
-    StepEvent,
     Transcript,
+    init_transcript,
     transcript,
 )
 from inspect_ai.model import (
@@ -91,7 +90,8 @@ from inspect_ai.solver._task_state import sample_state, set_sample_state, state_
 from inspect_ai.util._limit import LimitExceededError
 from inspect_ai.util._sandbox.context import sandbox_connections
 from inspect_ai.util._sandbox.environment import SandboxEnvironmentSpec
-from inspect_ai.util._subtask import init_subtask
+from inspect_ai.util._span import span
+from inspect_ai.util._store import init_subtask_store
 
 from ..context import init_task_context
 from ..task import Task
@@ -558,7 +558,9 @@ async def task_run_sample(
     # initialise subtask and scoring context
     init_sample_model_usage()
     set_sample_state(state)
-    sample_transcript: Transcript = init_subtask(SAMPLE_SUBTASK, state.store)
+    sample_transcript = Transcript()
+    init_transcript(sample_transcript)
+    init_subtask_store(state.store)
     if logger:
         sample_transcript._subscribe(
             lambda event: logger.log_sample_event(sample_id, state.epoch, event)
@@ -617,7 +619,8 @@ async def task_run_sample(
         results: dict[str, SampleScore] = {}
         try:
             # begin init
-            transcript()._event(StepEvent(action="begin", name="init"))
+            init_span = span("init", type="init")
+            await init_span.__aenter__()
 
             # sample init event (remove file bodies as they have content or absolute paths)
             event_sample = sample.model_copy(
@@ -639,7 +642,7 @@ async def task_run_sample(
                     active.sandboxes = await sandbox_connections()
 
                     # end init
-                    transcript()._event(StepEvent(action="end", name="init"))
+                    await init_span.__aexit__(None, None, None)
 
                     # initialise timeout context manager
                     timeout_cm = (
@@ -742,7 +745,7 @@ async def task_run_sample(
                                 scorer_name = unique_scorer_name(
                                     scorer, list(results.keys())
                                 )
-                                with transcript().step(name=scorer_name, type="scorer"):
+                                async with span(name=scorer_name, type="scorer"):
                                     score_result = (
                                         await scorer(state, Target(sample.target))
                                         if scorer

inspect_ai-0.3.94/src/inspect_ai/_util/answer.py

@@ -0,0 +1,26 @@
+def answer_character(index: int) -> str:
+    r"""
+    Helper to go from array index to char, for example:
+
+        0 -> 'A', 1 -> 'B', etc
+    """
+    if index < 26:
+        return chr(ord("A") + index)
+    else:
+        return str(index - 25)
+
+
+def answer_index(char: str) -> int:
+    r"""
+    Helper to go from char to array index, for example:
+
+        'A' -> 0, 'B' -> 1, etc
+    """
+    if char.isalpha() or char == "," or char == " ":
+        return ord(char.upper()) - ord("A")
+    elif char.isnumeric():
+        return 25 + int(char)
+    else:
+        raise ValueError(
+            f"Unepxected multiple choice answer: {char} (must be a letter or number)"
+        )

{inspect_ai-0.3.93 → inspect_ai-0.3.94}/src/inspect_ai/_util/constants.py

@@ -34,7 +34,6 @@ EVAL_LOG_FORMAT = "eval"
 DEFAULT_DISPLAY = "full"
 LOG_SCHEMA_VERSION = 2
 SCORED_SUFFIX = "-scored"
-SAMPLE_SUBTASK = "sample"
 CONSOLE_DISPLAY_WIDTH = 120
 BASE_64_DATA_REMOVED = "<base64-data-removed>"
 SANDBOX_SETUP_TIMEOUT = 300

{inspect_ai-0.3.93 → inspect_ai-0.3.94}/src/inspect_ai/_util/local_server.py

@@ -62,16 +62,24 @@ def release_port(lock_socket: socket.socket) -> None:
         logger.error(f"Error closing socket: {e}")
 
 
-def execute_shell_command(command: list[str]) -> subprocess.Popen[str]:
+def execute_shell_command(
+    command: list[str], env: Optional[dict[str, str]] = None
+) -> subprocess.Popen[str]:
     """
     Execute a command and return its process handle.
 
     Args:
         command: List of command arguments
+        env: Optional environment variables to pass to the subprocess
 
     Returns:
         A subprocess.Popen object representing the running process
     """
+    # Create a process environment by copying current environment and updating with new values
+    process_env = os.environ.copy()
+    if env:
+        process_env.update(env)
+
     # Create a process that redirects output to pipes so we can capture it
     process = subprocess.Popen(
         command,
@@ -79,6 +87,7 @@ def execute_shell_command(command: list[str]) -> subprocess.Popen[str]:
         stdout=subprocess.PIPE,
         stderr=subprocess.PIPE,
         bufsize=1,  # Line buffered
+        env=process_env,  # Pass the environment variables
     )
 
     # Set up background thread to read and log stdout
@@ -134,7 +143,10 @@ def kill_process_tree(pid: int) -> None:
 
 
 def launch_server_cmd(
-    command: list[str], host: str = "0.0.0.0", port: Optional[int] = None
+    command: list[str],
+    host: str = "0.0.0.0",
+    port: Optional[int] = None,
+    env: Optional[dict[str, str]] = None,
 ) -> Tuple[subprocess.Popen[str], int, list[str]]:
     """
     Launch a server process with the given base command and return the process, port, and full command.
@@ -143,6 +155,7 @@ def launch_server_cmd(
         command: Base command to execute
         host: Host to bind to
         port: Port to bind to. If None, a free port is reserved.
+        env: Optional environment variables to pass to the subprocess
 
     Returns:
         Tuple of (process, port, full_command)
@@ -155,7 +168,7 @@ def launch_server_cmd(
     full_command = command + ["--port", str(port)]
     logger.info(f"Launching server on port {port}")
 
-    process = execute_shell_command(full_command)
+    process = execute_shell_command(full_command, env=env)
 
     if lock_socket is not None:
         process_socket_map[process] = lock_socket
@@ -181,6 +194,7 @@ def wait_for_server(
     base_url: str,
     process: subprocess.Popen[str],
     full_command: Optional[list[str]] = None,
+    env: Optional[dict[str, str]] = None,
     timeout: Optional[int] = None,
     api_key: Optional[str] = None,
 ) -> None:
@@ -191,6 +205,7 @@ def wait_for_server(
         base_url: The base URL of the server
         process: The subprocess running the server
         full_command: The full command used to launch the server
+        env: The environment variables to use for the request
         timeout: Maximum time to wait in seconds. None means wait forever.
         api_key: The API key to use for the request
     """
@@ -198,7 +213,10 @@ def wait_for_server(
     start_time = time.time()
     debug_advice = "Try rerunning with '--log-level debug' to see the full traceback."
     if full_command:
-        debug_advice += f" Alternatively, you can run the following launch command manually to see the full traceback:\n\n{' '.join(full_command)}\n\n"
+        debug_advice += " Alternatively, you can run the following launch command manually to see the full traceback:\n\n"
+        if env:
+            debug_advice += " ".join([f"{k}={v}" for k, v in env.items()]) + " "
+        debug_advice += " ".join(full_command) + "\n\n"
 
     while True:
         # Check for timeout first
@@ -245,6 +263,7 @@ def start_local_server(
     server_type: str = "server",
     timeout: Optional[int] = DEFAULT_TIMEOUT,
     server_args: Optional[dict[str, Any]] = None,
+    env: Optional[dict[str, str]] = None,
 ) -> Tuple[str, subprocess.Popen[str], int]:
     """
     Start a server with the given command and handle potential errors.
@@ -257,6 +276,7 @@ def start_local_server(
         server_type: Type of server being started (for error messages)
         timeout: Maximum time to wait for server to become ready
         server_args: Additional server arguments to pass to the command
+        env: Optional environment variables to pass to the subprocess
     Returns:
         Tuple of (base_url, process, port)
 
@@ -266,15 +286,22 @@ def start_local_server(
     full_command = base_cmd
     server_process = None
 
+    # Initialize environment variables if not provided
+    process_env = {} if env is None else env.copy()
+
     if server_args:
         for key, value in server_args.items():
             # Convert Python style args (underscore) to CLI style (dash)
             cli_key = key.replace("_", "-")
-            full_command.extend([f"--{cli_key}", str(value)])
+            if value == "":
+                # If the value is empty, just add the flag
+                full_command.extend([f"--{cli_key}"])
+            else:
+                full_command.extend([f"--{cli_key}", str(value)])
 
     try:
         server_process, found_port, full_command = launch_server_cmd(
-            full_command, host=host, port=port
+            full_command, host=host, port=port, env=process_env
         )
         base_url = f"http://localhost:{found_port}/v1"
         wait_for_server(
@@ -283,6 +310,7 @@ def start_local_server(
             api_key=api_key,
             timeout=timeout,
             full_command=full_command,
+            env=process_env,
         )
         return base_url, server_process, found_port
     except Exception as e:
@@ -330,17 +358,18 @@ def merge_env_server_args(
 
 def configure_devices(
     server_args: dict[str, Any], parallel_size_param: str = "tensor_parallel_size"
-) -> dict[str, Any]:
-    """Configure device settings and return updated server args.
+) -> tuple[dict[str, Any], dict[str, str]]:
+    """Configure device settings and return updated server args and environment variables.
 
     Args:
         server_args: Dictionary of server arguments
         parallel_size_param: Name of parameter to set with device count if not specified
 
     Returns:
-        Updated server arguments dict
+        Tuple of (updated server arguments dict, environment variables dict)
     """
     result = server_args.copy()
+    env_vars = {}
 
     devices = None
     if "device" in result and "devices" in result:
@@ -350,19 +379,20 @@ def configure_devices(
     elif "device" in result:
         devices = result.pop("device")
 
-    # Convert device list to comma-separated string if needed
-    if isinstance(devices, list):
-        device_str = ",".join(map(str, devices))
-    else:
-        device_str = str(devices)
+    if devices is not None:
+        # Convert device list to comma-separated string if needed
+        if isinstance(devices, list):
+            device_str = ",".join(map(str, devices))
+        else:
+            device_str = str(devices)
 
-    # Set CUDA_VISIBLE_DEVICES environment variable
-    os.environ["CUDA_VISIBLE_DEVICES"] = device_str
+        # Add to env_vars instead of setting os.environ directly
+        env_vars["CUDA_VISIBLE_DEVICES"] = device_str
 
-    device_count = len(device_str.split(","))
+        device_count = len(device_str.split(","))
 
-    # Set parallel size parameter if not explicitly provided
-    if parallel_size_param not in result:
-        result[parallel_size_param] = device_count
+        # Set parallel size parameter if not explicitly provided
+        if parallel_size_param not in result:
+            result[parallel_size_param] = device_count
 
-    return result
+    return result, env_vars