inspect-ai 0.3.69__tar.gz → 0.3.70__tar.gz

{inspect_ai-0.3.69 → inspect_ai-0.3.70}/.github/dependabot.yml

@@ -16,3 +16,5 @@ updates:
         versions: [">=0.9.5"]
       - dependency-name: "textual"
         versions: [">=2.0.0"]
+      - dependency-name: "fsspec"
+        versions: [">=2024.12.0"]

{inspect_ai-0.3.69 → inspect_ai-0.3.70}/CHANGELOG.md

@@ -1,3 +1,20 @@
+## v0.3.70 (25 February 2025)
+
+- [working_limit](https://inspect.ai-safety-institute.org.uk/errors_and_limits.html#working-limit) option for specifying a maximum working time (e.g. model generation, tool calls, etc.) for samples.
+- Added `SandboxEvent` to transcript for recording sandbox execution and I/O.
+- Sandboxes: `as_type()` function for checked downcasting of `SandboxEnvironment`
+- Remove root logging handlers upon Inspect logger initialisation (as they result in lots of log spam if left installed).
+- Only explicitly set `state.completed=True` when entering scoring (`basic_agent()` no longer sets `completed` so can be used in longer compositions of solvers).
+- Add `uuid` property to `TaskState` and `EvalSample` (globally unique identifer for sample run).
+- Add `cleanup` to tasks for executing a function at the end of each sample run.
+- Agent `bridge()` is now compatible with the use of a custom `OPENAI_BASE_URL`.
+- Mistral: Bump required version of `mistralai` package to 1.5 (required for `working_limit`).
+- Truncate tracebacks included in evaluation log to a maximum of 1MB.
+- Compatiblity with textual version 2.0 (remove upper bound).
+- Align with HF datasets `fsspec` version contraints to avoid pip errors when installing alongside `datasets`.
+- Bugfix: Fix issue with tools that had an ordinary `dict` as a parameter.
+- Bugfix: Print the correct container `sample_id` for `--no-sandbox-cleanup`.
+
 ## v0.3.69 (20 February 2025)
 
 - Google provider updated to use the [Google Gen AI SDK](https://googleapis.github.io/python-genai/), which is now the recommended API for Gemini 2.0 models.

{inspect_ai-0.3.69 → inspect_ai-0.3.70}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: inspect_ai
-Version: 0.3.69
+Version: 0.3.70
 Summary: Framework for large language model evaluations
 Author: UK AI Security Institute
 License: MIT License
@@ -26,7 +26,7 @@ Requires-Dist: beautifulsoup4
 Requires-Dist: click>=8.1.3
 Requires-Dist: debugpy
 Requires-Dist: docstring-parser>=0.16
-Requires-Dist: fsspec>=2021.09.0
+Requires-Dist: fsspec<=2024.12.0,>=2023.1.0
 Requires-Dist: httpx
 Requires-Dist: ijson>=3.2.0
 Requires-Dist: jsonlines>=3.0.0
@@ -45,7 +45,7 @@ Requires-Dist: s3fs>=2023
 Requires-Dist: semver>=3.0.0
 Requires-Dist: shortuuid
 Requires-Dist: tenacity
-Requires-Dist: textual<=1.0.0,>=0.86.2
+Requires-Dist: textual>=0.86.2
 Requires-Dist: typing_extensions>=4.9.0
 Requires-Dist: zipp>=3.19.1
 Provides-Extra: dev

{inspect_ai-0.3.69 → inspect_ai-0.3.70}/docs/agents.qmd

@@ -32,7 +32,7 @@ The `basic_agent()`provides a ReAct tool loop with support for retries and encou
 
 The basic agent incorporates best practices for giving models some additional resilience and persistence, both through the optional `max_attempts` parameter, as well as by continuing the task even when the model stops making tool calls. The basic agent can frequently match or exceed custom scaffolds, so you should always try it as a baseline for your tasks!
 
-Note that when using the basic agent you should *always* set a `message_limit`, `token_limit`, or `time_limit` so that there is some termination point if the model gets off track or stuck in a loop (see [Sample Limits](errors-and-limits.qmd#sample-limits) for additional details).
+Note that when using the basic agent you should *always* set a `message_limit`, `token_limit`, `time_limit`, or `working_limit` so that there is some termination point if the model gets off track or stuck in a loop (see [Sample Limits](errors-and-limits.qmd#sample-limits) for additional details).
 
 ### Example
 

{inspect_ai-0.3.69 → inspect_ai-0.3.70}/docs/errors-and-limits.qmd

@@ -10,7 +10,9 @@ This article covers various techniques for dealing with unexpected errors and se
 
 1.  Retrying failed evaluations (while preserving the samples completed during the initial failed run).
 2.  Establishing a threshold (count or percentage) of samples to tolerate errors for before failing an evaluation.
-3.  Setting a maximum number of messages in a sample before forcing the model to give up.
+3.  Setting time limits for samples (either running time or more narrowly execution time).
+4.  Setting a maximum number of messages or tokens in a sample before forcing the model to give up.
+
 
 {{< include _errors_and_retries.md >}}
 
@@ -91,6 +93,33 @@ inspect eval ctf.py --time-limit 900
 
 Appropriate timeouts will vary depending on the nature of your task so please view the above as examples only rather than recommend values.
 
+### Working Limit
+
+The `working_limit` differs from the `time_limit` in that it measures only the time spent working (as opposed to retrying in response to rate limits or waiting on other shared resources). Here we set an `working_limit` of 10 minutes (10 x 60 seconds) for each sample within a task:
+
+``` python
+@task
+def intercode_ctf():
+    return Task(
+        dataset=read_dataset(),
+        solver=[
+            system_message("system.txt"),
+            use_tools([bash(timeout=3 * 60)]),
+            generate(),
+        ],
+        working_limit=10 * 60,
+        scorer=includes(),
+        sandbox="docker",
+    )
+```
+
+Working time is computed based on total clock time minus time spent on (a) unsuccessful model generations (e.g. rate limited requests); and (b) waiting on shared resources (e.g. Docker containers or subprocess execution).
+
+::: {.callout-note appearance="simple"}
+In order to distinguish successful generate requests from rate limited and retried requests, Inspect installs hooks into the HTTP client of various model packages. This is not possible for some models (`google`, `vertex`, `azureai`, and `goodfire`), and in these cases the `working_time` will include any internal retries that the model client performs. 
+:::
+
+
 ### Message Limit
 
 Here we set a `message_limit` of 30 for each sample within a task:

{inspect_ai-0.3.69 → inspect_ai-0.3.70}/docs/options.qmd

@@ -124,7 +124,8 @@ Below are sections for the various categories of options supported by `inspect e
 | `--no-fail-on-error` | Do not fail the eval if errors occur within samples (instead, continue running other samples)                                                                         |
 | `--message-limit`    | Limit on total messages used for each sample.                                                                                                                         |
 | `--token-limit`      | Limit on total tokens used for each sample.                                                                                                                           |
-| `--time-limit`       | Limit on total execution time for each sample.                                                                                                                        |
+| `--time-limit`       | Limit on total running time for each sample.                                                                                                                        |
+| `--working-limit`       | Limit on total working time (model generation, tool calls, etc.) for each sample.                                                                                                                        |
 
 ## Eval Logs
 

{inspect_ai-0.3.69 → inspect_ai-0.3.70}/docs/reference/_sidebar.yml

@@ -352,6 +352,8 @@ website:
           href: reference/inspect_ai.log.qmd#modelevent
         - text: ToolEvent
           href: reference/inspect_ai.log.qmd#toolevent
+        - text: SandboxEvent
+          href: reference/inspect_ai.log.qmd#sandboxevent
         - text: ApprovalEvent
           href: reference/inspect_ai.log.qmd#approvalevent
         - text: InputEvent

{inspect_ai-0.3.69 → inspect_ai-0.3.70}/docs/reference/inspect_ai.log.qmd

@@ -46,6 +46,7 @@ title: "inspect_ai.log"
 ### StoreEvent
 ### ModelEvent
 ### ToolEvent
+### SandboxEvent
 ### ApprovalEvent
 ### InputEvent
 ### StoreEvent

{inspect_ai-0.3.69 → inspect_ai-0.3.70}/docs/reference/inspect_ai.util.qmd

@@ -34,6 +34,7 @@ title: "inspect_ai.util"
 ### trace_action
 ### trace_message
 
+
 ## Sandbox
 
 ### sandbox

{inspect_ai-0.3.69 → inspect_ai-0.3.70}/docs/tasks.qmd

@@ -41,31 +41,32 @@ For convenience, tasks always define a default solver. That said, it is often de
 
 While many tasks can be defined with only a dataset, solver, and scorer, there are lots of other useful `Task` options. We won't describe these options in depth here, but rather provide a list along with links to other sections of the documentation that cover their usage:
 
-+------------------+------------------------------------------------------+-----------------------------------------------------------+
-| Option           | Description                                          | Docs                                                      |
-+==================+======================================================+===========================================================+
-| `epochs`         | Epochs to run for each dataset sample.               | [Epochs](scorers.qmd#reducing-epochs)                     |
-+------------------+------------------------------------------------------+-----------------------------------------------------------+
-| `config`         | Config for model generation.                         | [Generate Config](options.qmd#model-generation)           |
-+------------------+------------------------------------------------------+-----------------------------------------------------------+
-| `setup`          | Setup solver(s) to run prior to the main solver.     | [Sample Setup](#setup-parameter)                          |
-+------------------+------------------------------------------------------+-----------------------------------------------------------+
-| `sandbox`        | Sandbox configuration for un-trusted code execution. | [Sandboxing](sandboxing.qmd)                              |
-+------------------+------------------------------------------------------+-----------------------------------------------------------+
-| `approval`       | Approval policy for tool calls.                      | [Tool Approval](approval.qmd)                             |
-+------------------+------------------------------------------------------+-----------------------------------------------------------+
-| `metrics`        | Metrics to use in place of scorer metrics.           | [Scoring Metrics](scorers.qmd#scoring-metrics)            |
-+------------------+------------------------------------------------------+-----------------------------------------------------------+
-| `fail_on_error`  | Failure tolerance for samples.                       | [Sample Failure](errors-and-limits.qmd#failure-threshold) |
-+------------------+------------------------------------------------------+-----------------------------------------------------------+
-| `message_limit`\ | Limits to apply to sample execution.                 | [Sample Limits](errors-and-limits.qmd#sample-limits)      |
-| `token_limit`\   |                                                      |                                                           |
-| `time_limit`     |                                                      |                                                           |
-+------------------+------------------------------------------------------+-----------------------------------------------------------+
-| `name`\          | Eval log attributes for task.                        | [Eval Logs](eval-logs.qmd)                                |
-| `version`\       |                                                      |                                                           |
-| `metadata`       |                                                      |                                                           |
-+------------------+------------------------------------------------------+-----------------------------------------------------------+
++-------------------+------------------------------------------------------+-----------------------------------------------------------+
+| Option            | Description                                          | Docs                                                      |
++===================+======================================================+===========================================================+
+| `epochs`          | Epochs to run for each dataset sample.               | [Epochs](scorers.qmd#reducing-epochs)                     |
++-------------------+------------------------------------------------------+-----------------------------------------------------------+
+| `config`          | Config for model generation.                         | [Generate Config](options.qmd#model-generation)           |
++-------------------+------------------------------------------------------+-----------------------------------------------------------+
+| `setup`           | Setup solver(s) to run prior to the main solver.     | [Sample Setup](#setup-parameter)                          |
++-------------------+------------------------------------------------------+-----------------------------------------------------------+
+| `sandbox`         | Sandbox configuration for un-trusted code execution. | [Sandboxing](sandboxing.qmd)                              |
++-------------------+------------------------------------------------------+-----------------------------------------------------------+
+| `approval`        | Approval policy for tool calls.                      | [Tool Approval](approval.qmd)                             |
++-------------------+------------------------------------------------------+-----------------------------------------------------------+
+| `metrics`         | Metrics to use in place of scorer metrics.           | [Scoring Metrics](scorers.qmd#scoring-metrics)            |
++-------------------+------------------------------------------------------+-----------------------------------------------------------+
+| `fail_on_error`   | Failure tolerance for samples.                       | [Sample Failure](errors-and-limits.qmd#failure-threshold) |
++-------------------+------------------------------------------------------+-----------------------------------------------------------+
+| `message_limit`\  | Limits to apply to sample execution.                 | [Sample Limits](errors-and-limits.qmd#sample-limits)      |
+| `token_limit`\    |                                                      |                                                           |
+| `time_limit`\     |                                                      |                                                           |
+| `working_limit`   |                                                      |                                                           |
++-------------------+------------------------------------------------------+-----------------------------------------------------------+
+| `name`\           | Eval log attributes for task.                        | [Eval Logs](eval-logs.qmd)                                |
+| `version`\        |                                                      |                                                           |
+| `metadata`        |                                                      |                                                           |
++-------------------+------------------------------------------------------+-----------------------------------------------------------+
 
 : {tbl-colwidths=\[25,50,25\]}
 
@@ -212,7 +213,7 @@ inspect eval ctf.py --solver=ctf_agent -S max_attempts=5
 
 Note the use of the `-S` CLI option to pass an alternate value for `max_attempts` to the `ctf_agent()` solver.
 
-### Setup Parameter
+### Setup Parameter {#setup-parameter}
 
 In some cases, there will be important steps in the setup of a task that *should not be substituted* when another solver is used with the task. For example, you might have a step that does dynamic prompt engineering based on values in the sample `metadata` or you might have a step that initialises resources in a sample's sandbox.
 
@@ -242,6 +243,26 @@ def ctf(solver: Solver | None = None):
     )
 ```
 
+## Task Cleanup
+
+You can use the `cleanup` parameter for executing code at the end of each sample run. The `cleanup` function is passed the `TaskState` and is called for both successful runs and runs where are exception is thrown. Extending the example from above:
+
+```python
+async def ctf_cleanup(state: TaskState):
+    ## perform cleanup
+    ...
+
+Task(
+    dataset=read_dataset(),
+    setup=ctf_prompt(),
+    solver=solver,
+    cleanup=ctf_cleanup,
+    scorer=includes()
+)
+```
+
+Note that like solvers, cleanup functions should be `async`.
+
 ## Task Reuse {#task-reuse}
 
 The basic mechanism for task re-use is to create flexible and adaptable base `@task` functions (which often have many parameters) and then derive new higher-level tasks from them by creating additional `@task` functions that call the base function.

{inspect_ai-0.3.69 → inspect_ai-0.3.70}/requirements.txt

@@ -4,7 +4,7 @@ beautifulsoup4
 click>=8.1.3
 debugpy
 docstring-parser>=0.16
-fsspec>=2021.09.0
+fsspec>=2023.1.0,<=2024.12.0 # align with hf datasets to prevent pip errors
 httpx
 ijson>=3.2.0
 jsonlines>=3.0.0
@@ -23,6 +23,6 @@ s3fs>=2023
 semver>=3.0.0
 shortuuid
 tenacity
-textual>=0.86.2,<=1.0.0
+textual>=0.86.2
 typing_extensions>=4.9.0
 zipp>=3.19.1 # not directly required, pinned by Snyk to avoid a vulnerability

{inspect_ai-0.3.69 → inspect_ai-0.3.70}/src/inspect_ai/_cli/eval.py

@@ -218,9 +218,15 @@ def eval_options(func: Callable[..., Any]) -> Callable[..., click.Context]:
     @click.option(
         "--time-limit",
         type=int,
-        help="Limit on total execution time for each sample.",
+        help="Limit on total running time for each sample.",
         envvar="INSPECT_EVAL_TIME_LIMIT",
     )
+    @click.option(
+        "--working-limit",
+        type=int,
+        help="Limit on total working time (e.g. model generation, tool calls, etc.) for each sample.",
+        envvar="INSPECT_EVAL_WORKING_LIMIT",
+    )
     @click.option(
         "--fail-on-error",
         type=float,
@@ -468,6 +474,7 @@ def eval_command(
     message_limit: int | None,
     token_limit: int | None,
     time_limit: int | None,
+    working_limit: int | None,
     max_samples: int | None,
     max_tasks: int | None,
     max_subprocesses: int | None,
@@ -518,6 +525,7 @@ def eval_command(
         message_limit=message_limit,
         token_limit=token_limit,
         time_limit=time_limit,
+        working_limit=working_limit,
         max_samples=max_samples,
         max_tasks=max_tasks,
         max_subprocesses=max_subprocesses,
@@ -629,6 +637,7 @@ def eval_set_command(
     message_limit: int | None,
     token_limit: int | None,
     time_limit: int | None,
+    working_limit: int | None,
     max_samples: int | None,
     max_tasks: int | None,
     max_subprocesses: int | None,
@@ -684,6 +693,7 @@ def eval_set_command(
         message_limit=message_limit,
         token_limit=token_limit,
         time_limit=time_limit,
+        working_limit=working_limit,
         max_samples=max_samples,
         max_tasks=max_tasks,
         max_subprocesses=max_subprocesses,
@@ -737,6 +747,7 @@ def eval_exec(
     message_limit: int | None,
     token_limit: int | None,
     time_limit: int | None,
+    working_limit: int | None,
     max_samples: int | None,
     max_tasks: int | None,
     max_subprocesses: int | None,
@@ -817,6 +828,7 @@ def eval_exec(
             message_limit=message_limit,
             token_limit=token_limit,
             time_limit=time_limit,
+            working_limit=working_limit,
             max_samples=max_samples,
             max_tasks=max_tasks,
             max_subprocesses=max_subprocesses,

{inspect_ai-0.3.69 → inspect_ai-0.3.70}/src/inspect_ai/_display/textual/app.py

@@ -185,7 +185,8 @@ class TaskScreenApp(App[TR]):
         # force repaint
         self.refresh(repaint=True)
 
-        # enable mouse support (this broke in textual 2.0 when running in VS Code)
+        # enable mouse support (this broke in textual 2.0 when running in VS Code
+        # however is fixed in textual 2.1)
         assert self.app._driver
         textual_enable_mouse_support(self.app._driver)
 
@@ -315,7 +316,7 @@ class TaskScreenApp(App[TR]):
 
         def set_unread(unread: int | None) -> None:
             if unread is not None:
-                console_tab.label = f"Console ({unread}"  # type: ignore[assignment]
+                console_tab.label = f"Console ({unread})"  # type: ignore[assignment]
             else:
                 console_tab.label = "Console"  # type: ignore[assignment]
 

{inspect_ai-0.3.69 → inspect_ai-0.3.70}/src/inspect_ai/_display/textual/widgets/samples.py

@@ -39,7 +39,7 @@ class SamplesView(Widget):
         padding: 0 1 0 1;
         layout: grid;
         grid-size: 2 3;
-        grid-rows: auto 1fr auto;
+        grid-rows: auto 1fr 3;
         grid-columns: 32 1fr;
         grid-gutter: 1;
     }
@@ -141,8 +141,8 @@ class SamplesList(OptionList):
         if highlighted_sample and (highlighted_sample not in self.samples):
             self.samples.append(highlighted_sample)
 
-        # sort the samples by execution time
-        self.samples.sort(key=lambda sample: sample.execution_time, reverse=True)
+        # sort the samples by running time
+        self.samples.sort(key=lambda sample: sample.running_time, reverse=True)
 
         # rebuild the list
         self.clear_options()
@@ -154,9 +154,7 @@ class SamplesList(OptionList):
             table.add_column(width=1)
             task_name = Text.from_markup(f"{registry_unqualified_name(sample.task)}")
             task_name.truncate(18, overflow="ellipsis", pad=True)
-            task_time = Text.from_markup(
-                f"{format_progress_time(sample.execution_time)}"
-            )
+            task_time = Text.from_markup(f"{format_progress_time(sample.running_time)}")
             table.add_row(task_name, task_time, " ")
             sample_id = Text.from_markup(f"id: {sample.sample.id}")
             sample_id.truncate(18, overflow="ellipsis", pad=True)
@@ -423,10 +421,6 @@ class SampleToolbar(Horizontal):
     CANCEL_DISABLED = "Cancelling sample..."
 
     DEFAULT_CSS = f"""
-    SampleToolbar {{
-        grid-size: 5 1;
-        grid-columns: auto auto 1fr auto auto;
-    }}
     SampleToolbar #{STATUS_GROUP} {{
         width: 22;
     }}

{inspect_ai-0.3.69 → inspect_ai-0.3.70}/src/inspect_ai/_display/textual/widgets/transcript.py

@@ -193,16 +193,29 @@ def render_model_event(event: ModelEvent) -> EventDisplay:
     return EventDisplay(f"model: {event.model}", Group(*content))
 
 
-def render_tool_event(event: ToolEvent) -> list[EventDisplay]:
-    # render sub-events
-    display: list[EventDisplay] = []
-    if event.events:
-        for e in event.events:
-            display.extend(render_event(e) or [])
+def render_sub_events(events: list[Event]) -> list[RenderableType]:
+    content: list[RenderableType] = []
+    for e in events:
+        event_displays = render_event(e) or []
+        for d in event_displays:
+            if d.content:
+                content.append(Text("  "))
+                content.append(transcript_separator(d.title, "black", "··"))
+                if isinstance(d.content, Markdown):
+                    set_transcript_markdown_options(d.content)
+                content.append(d.content)
+
+    return content
+
 
+def render_tool_event(event: ToolEvent) -> list[EventDisplay]:
     # render the call
     content = transcript_tool_call(event)
 
+    # render sub-events
+    if event.events:
+        content.extend(render_sub_events(event.events))
+
     # render the output
     if isinstance(event.result, list):
         result: ToolResult = "\n".join(
@@ -220,7 +233,7 @@ def render_tool_event(event: ToolEvent) -> list[EventDisplay]:
         result = str(result).strip()
         content.extend(lines_display(result, 50))
 
-    return display + [EventDisplay("tool call", Group(*content))]
+    return [EventDisplay("tool call", Group(*content))]
 
 
 def render_step_event(event: StepEvent) -> EventDisplay:
@@ -257,13 +270,13 @@ def render_score_event(event: ScoreEvent) -> EventDisplay:
 
 
 def render_subtask_event(event: SubtaskEvent) -> list[EventDisplay]:
+    # render header
+    content: list[RenderableType] = [transcript_function(event.name, event.input)]
+
     # render sub-events
-    display: list[EventDisplay] = []
     if event.events:
-        for e in event.events:
-            display.extend(render_event(e) or [])
+        content.extend(render_sub_events(event.events))
 
-    content: list[RenderableType] = [transcript_function(event.name, event.input)]
     if event.result:
         content.append(Text())
         if isinstance(event.result, str | int | float | bool | None):
@@ -271,7 +284,7 @@ def render_subtask_event(event: SubtaskEvent) -> list[EventDisplay]:
         else:
             content.append(render_as_json(event.result))
 
-    return display + [EventDisplay(f"subtask: {event.name}", Group(*content))]
+    return [EventDisplay(f"subtask: {event.name}", Group(*content))]
 
 
 def render_input_event(event: InputEvent) -> EventDisplay:

{inspect_ai-0.3.69 → inspect_ai-0.3.70}/src/inspect_ai/_eval/eval.py

@@ -75,6 +75,7 @@ def eval(
     message_limit: int | None = None,
     token_limit: int | None = None,
     time_limit: int | None = None,
+    working_limit: int | None = None,
     max_samples: int | None = None,
     max_tasks: int | None = None,
     max_subprocesses: int | None = None,
@@ -132,7 +133,10 @@ def eval(
             so they can be debugged (defaults to False).
         message_limit: Limit on total messages used for each sample.
         token_limit: Limit on total tokens used for each sample.
-        time_limit: Limit on time (in seconds) for execution of each sample.
+        time_limit: Limit on clock time (in seconds) for samples.
+        working_limit: Limit on working time (in seconds) for sample. Working
+            time includes model generation, tool calls, etc. but does not include
+            time spent waiting on retries or shared resources.
         max_samples: Maximum number of samples to run in parallel
             (default is max_connections)
         max_tasks: Maximum number of tasks to run in parallel
@@ -186,6 +190,7 @@ def eval(
             message_limit=message_limit,
             token_limit=token_limit,
             time_limit=time_limit,
+            working_limit=working_limit,
             max_samples=max_samples,
             max_tasks=max_tasks,
             max_subprocesses=max_subprocesses,
@@ -227,6 +232,7 @@ async def eval_async(
     message_limit: int | None = None,
     token_limit: int | None = None,
     time_limit: int | None = None,
+    working_limit: int | None = None,
     max_samples: int | None = None,
     max_tasks: int | None = None,
     max_subprocesses: int | None = None,
@@ -281,7 +287,10 @@ async def eval_async(
            so they can be debugged (defaults to False).
         message_limit (int | None): Limit on total messages used for each sample.
         token_limit (int | None): Limit on total tokens used for each sample.
-        time_limit (int | None): Limit on time (in seconds) for execution of each sample.
+        time_limit: Limit on clock time (in seconds) for samples.
+        working_limit: Limit on working time (in seconds) for sample. Working
+            time includes model generation, tool calls, etc. but does not include
+            time spent waiting on retries or shared resources.
         max_samples (int | None): Maximum number of samples to run in parallel
            (default is max_connections)
         max_tasks (int | None): Maximum number of tasks to run in parallel
@@ -395,6 +404,7 @@ async def eval_async(
             message_limit=message_limit,
             token_limit=token_limit,
             time_limit=time_limit,
+            working_limit=working_limit,
             max_samples=max_samples,
             max_tasks=max_tasks,
             max_subprocesses=max_subprocesses,
@@ -702,6 +712,7 @@ async def eval_retry_async(
         message_limit = eval_log.eval.config.message_limit
         token_limit = eval_log.eval.config.token_limit
         time_limit = eval_log.eval.config.time_limit
+        working_limit = eval_log.eval.config.working_limit
         max_samples = max_samples or eval_log.eval.config.max_samples
         max_tasks = max_tasks or eval_log.eval.config.max_tasks
         max_subprocesses = max_subprocesses or eval_log.eval.config.max_subprocesses
@@ -763,6 +774,7 @@ async def eval_retry_async(
                 message_limit=message_limit,
                 token_limit=token_limit,
                 time_limit=time_limit,
+                working_limit=working_limit,
                 max_samples=max_samples,
                 max_tasks=max_tasks,
                 max_subprocesses=max_subprocesses,

{inspect_ai-0.3.69 → inspect_ai-0.3.70}/src/inspect_ai/_eval/evalset.py

@@ -79,6 +79,7 @@ def eval_set(
     message_limit: int | None = None,
     token_limit: int | None = None,
     time_limit: int | None = None,
+    working_limit: int | None = None,
     max_samples: int | None = None,
     max_tasks: int | None = None,
     max_subprocesses: int | None = None,
@@ -146,7 +147,10 @@ def eval_set(
             so they can be debugged (defaults to False).
         message_limit: Limit on total messages used for each sample.
         token_limit: Limit on total tokens used for each sample.
-        time_limit: Limit on time (in seconds) for execution of each sample.
+        time_limit: Limit on clock time (in seconds) for samples.
+        working_limit: Limit on working time (in seconds) for sample. Working
+            time includes model generation, tool calls, etc. but does not include
+            time spent waiting on retries or shared resources.
         max_samples: Maximum number of samples to run in parallel
             (default is max_connections)
         max_tasks: Maximum number of tasks to run in parallel
@@ -202,6 +206,7 @@ def eval_set(
             message_limit=message_limit,
             token_limit=token_limit,
             time_limit=time_limit,
+            working_limit=working_limit,
             max_samples=max_samples,
             max_tasks=max_tasks,
             max_subprocesses=max_subprocesses,

{inspect_ai-0.3.69 → inspect_ai-0.3.70}/src/inspect_ai/_eval/run.py

@@ -163,6 +163,12 @@ async def eval_run(
                 else:
                     task.time_limit = task_eval_config.time_limit
 
+                # sample execution limit
+                if task_eval_config.working_limit is None:
+                    task_eval_config.working_limit = task.working_limit
+                else:
+                    task.working_limit = task_eval_config.working_limit
+
                 # fail_on_error
                 if task_eval_config.fail_on_error is None:
                     task_eval_config.fail_on_error = task.fail_on_error