inspect-ai 0.3.68__tar.gz → 0.3.70__tar.gz

{inspect_ai-0.3.68 → inspect_ai-0.3.70}/.github/dependabot.yml

@@ -14,3 +14,7 @@ updates:
         versions: [">=1.6.0"]
       - dependency-name: "ruff"
         versions: [">=0.9.5"]
+      - dependency-name: "textual"
+        versions: [">=2.0.0"]
+      - dependency-name: "fsspec"
+        versions: [">=2024.12.0"]

{inspect_ai-0.3.68 → inspect_ai-0.3.70}/.github/workflows/build.yml

@@ -51,6 +51,18 @@ jobs:
         run: |
           mypy .
 
+  pre-commit:
+    runs-on: ubuntu-latest
+    env:
+      SKIP: "ruff"
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+          cache: pip
+      - uses: pre-commit/action@v3.0.1
+
   test:
     runs-on: ubuntu-latest
     strategy:

{inspect_ai-0.3.68 → inspect_ai-0.3.70}/.pre-commit-config.yaml

@@ -2,7 +2,7 @@
 # See https://pre-commit.com/hooks.html for more hooks
 repos:
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.9.5
+    rev: v0.9.6
     hooks:
       # Run the linter.
       - id: ruff
@@ -14,8 +14,9 @@ repos:
     hooks:
       - id: check-added-large-files
       - id: check-json
+        # The following are JSONC files.
+        exclude: '(^\.vscode/|^tools/vscode/|jsconfig\.json$|tsconfig\.json$)'
       - id: check-yaml
       - id: debug-statements
       - id: detect-private-key
-      - id: end-of-file-fixer
       - id: requirements-txt-fixer

{inspect_ai-0.3.68 → inspect_ai-0.3.70}/CHANGELOG.md

@@ -1,3 +1,32 @@
+## v0.3.70 (25 February 2025)
+
+- [working_limit](https://inspect.ai-safety-institute.org.uk/errors_and_limits.html#working-limit) option for specifying a maximum working time (e.g. model generation, tool calls, etc.) for samples.
+- Added `SandboxEvent` to transcript for recording sandbox execution and I/O.
+- Sandboxes: `as_type()` function for checked downcasting of `SandboxEnvironment`
+- Remove root logging handlers upon Inspect logger initialisation (as they result in lots of log spam if left installed).
+- Only explicitly set `state.completed=True` when entering scoring (`basic_agent()` no longer sets `completed` so can be used in longer compositions of solvers).
+- Add `uuid` property to `TaskState` and `EvalSample` (globally unique identifer for sample run).
+- Add `cleanup` to tasks for executing a function at the end of each sample run.
+- Agent `bridge()` is now compatible with the use of a custom `OPENAI_BASE_URL`.
+- Mistral: Bump required version of `mistralai` package to 1.5 (required for `working_limit`).
+- Truncate tracebacks included in evaluation log to a maximum of 1MB.
+- Compatiblity with textual version 2.0 (remove upper bound).
+- Align with HF datasets `fsspec` version contraints to avoid pip errors when installing alongside `datasets`.
+- Bugfix: Fix issue with tools that had an ordinary `dict` as a parameter.
+- Bugfix: Print the correct container `sample_id` for `--no-sandbox-cleanup`.
+
+## v0.3.69 (20 February 2025)
+
+- Google provider updated to use the [Google Gen AI SDK](https://googleapis.github.io/python-genai/), which is now the recommended API for Gemini 2.0 models.
+- Task display: Use cooperative cancellation for cancel buttons in task display.
+- Task display: Print task progress every 5 seconds for 'plain' display mode.
+- Task display: Handle click on running samples tab when there is no transcript.
+- Docker: Print stderr from `compose up` when no services startup successfully. 
+- Docker: Print sample id and epoch for each container when using `--no-sandbox-cleanup`
+- Mistral: Create and destroy client within generate.
+- Inspect View: Fix display of score dictionaries containing boolean values
+- Bugfix: Catch standard `TimeoutError` for subprocess timeouts (ensure kill/cleanup of timed out process).
+
 ## v0.3.68 (19 February 2025)
 
 - Task display: Improve spacing/layout of final task display.

{inspect_ai-0.3.68 → inspect_ai-0.3.70}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: inspect_ai
-Version: 0.3.68
+Version: 0.3.70
 Summary: Framework for large language model evaluations
 Author: UK AI Security Institute
 License: MIT License
@@ -26,7 +26,7 @@ Requires-Dist: beautifulsoup4
 Requires-Dist: click>=8.1.3
 Requires-Dist: debugpy
 Requires-Dist: docstring-parser>=0.16
-Requires-Dist: fsspec>=2021.09.0
+Requires-Dist: fsspec<=2024.12.0,>=2023.1.0
 Requires-Dist: httpx
 Requires-Dist: ijson>=3.2.0
 Requires-Dist: jsonlines>=3.0.0
@@ -45,7 +45,7 @@ Requires-Dist: s3fs>=2023
 Requires-Dist: semver>=3.0.0
 Requires-Dist: shortuuid
 Requires-Dist: tenacity
-Requires-Dist: textual<=1.0.0,>=0.86.2
+Requires-Dist: textual>=0.86.2
 Requires-Dist: typing_extensions>=4.9.0
 Requires-Dist: zipp>=3.19.1
 Provides-Extra: dev
@@ -53,7 +53,7 @@ Requires-Dist: anthropic; extra == "dev"
 Requires-Dist: aioboto3; extra == "dev"
 Requires-Dist: azure-ai-inference; extra == "dev"
 Requires-Dist: google-cloud-aiplatform; extra == "dev"
-Requires-Dist: google-generativeai; extra == "dev"
+Requires-Dist: google-genai; extra == "dev"
 Requires-Dist: goodfire; extra == "dev"
 Requires-Dist: griffe; extra == "dev"
 Requires-Dist: groq; extra == "dev"
@@ -71,7 +71,7 @@ Requires-Dist: pytest-asyncio; extra == "dev"
 Requires-Dist: pytest-cov; extra == "dev"
 Requires-Dist: pytest-dotenv; extra == "dev"
 Requires-Dist: pytest-xdist; extra == "dev"
-Requires-Dist: ruff==0.9.5; extra == "dev"
+Requires-Dist: ruff==0.9.6; extra == "dev"
 Requires-Dist: textual-dev>=0.86.2; extra == "dev"
 Requires-Dist: types-Markdown; extra == "dev"
 Requires-Dist: types-PyYAML; extra == "dev"

{inspect_ai-0.3.68 → inspect_ai-0.3.70}/docs/agents.qmd

@@ -32,7 +32,7 @@ The `basic_agent()`provides a ReAct tool loop with support for retries and encou
 
 The basic agent incorporates best practices for giving models some additional resilience and persistence, both through the optional `max_attempts` parameter, as well as by continuing the task even when the model stops making tool calls. The basic agent can frequently match or exceed custom scaffolds, so you should always try it as a baseline for your tasks!
 
-Note that when using the basic agent you should *always* set a `message_limit`, `token_limit`, or `time_limit` so that there is some termination point if the model gets off track or stuck in a loop (see [Sample Limits](errors-and-limits.qmd#sample-limits) for additional details).
+Note that when using the basic agent you should *always* set a `message_limit`, `token_limit`, `time_limit`, or `working_limit` so that there is some termination point if the model gets off track or stuck in a loop (see [Sample Limits](errors-and-limits.qmd#sample-limits) for additional details).
 
 ### Example
 

{inspect_ai-0.3.68 → inspect_ai-0.3.70}/docs/errors-and-limits.qmd

@@ -10,7 +10,9 @@ This article covers various techniques for dealing with unexpected errors and se
 
 1.  Retrying failed evaluations (while preserving the samples completed during the initial failed run).
 2.  Establishing a threshold (count or percentage) of samples to tolerate errors for before failing an evaluation.
-3.  Setting a maximum number of messages in a sample before forcing the model to give up.
+3.  Setting time limits for samples (either running time or more narrowly execution time).
+4.  Setting a maximum number of messages or tokens in a sample before forcing the model to give up.
+
 
 {{< include _errors_and_retries.md >}}
 
@@ -91,6 +93,33 @@ inspect eval ctf.py --time-limit 900
 
 Appropriate timeouts will vary depending on the nature of your task so please view the above as examples only rather than recommend values.
 
+### Working Limit
+
+The `working_limit` differs from the `time_limit` in that it measures only the time spent working (as opposed to retrying in response to rate limits or waiting on other shared resources). Here we set an `working_limit` of 10 minutes (10 x 60 seconds) for each sample within a task:
+
+``` python
+@task
+def intercode_ctf():
+    return Task(
+        dataset=read_dataset(),
+        solver=[
+            system_message("system.txt"),
+            use_tools([bash(timeout=3 * 60)]),
+            generate(),
+        ],
+        working_limit=10 * 60,
+        scorer=includes(),
+        sandbox="docker",
+    )
+```
+
+Working time is computed based on total clock time minus time spent on (a) unsuccessful model generations (e.g. rate limited requests); and (b) waiting on shared resources (e.g. Docker containers or subprocess execution).
+
+::: {.callout-note appearance="simple"}
+In order to distinguish successful generate requests from rate limited and retried requests, Inspect installs hooks into the HTTP client of various model packages. This is not possible for some models (`google`, `vertex`, `azureai`, and `goodfire`), and in these cases the `working_time` will include any internal retries that the model client performs. 
+:::
+
+
 ### Message Limit
 
 Here we set a `message_limit` of 30 for each sample within a task:

{inspect_ai-0.3.68 → inspect_ai-0.3.70}/docs/index.qmd

@@ -56,7 +56,7 @@ inspect eval arc.py --model anthropic/claude-3-5-sonnet-latest
 #### Google
 
 ``` bash
-pip install google-generativeai
+pip install google-genai
 export GOOGLE_API_KEY=your-google-api-key
 inspect eval arc.py --model google/gemini-1.5-pro
 ```

{inspect_ai-0.3.68 → inspect_ai-0.3.70}/docs/models.qmd

@@ -40,24 +40,24 @@ You can specify generation options either on the command line or in direct calls
 
 ``` bash
 inspect eval arc.py --model openai/gpt-4 --temperature 0.9
-inspect eval arc.py --model google/gemini-1.0-pro --max-connections 20
+inspect eval arc.py --model google/gemini-1.5-pro --max-connections 20
 ```
 
 Or:
 
 ``` python
 eval("arc.py", model="openai/gpt-4", temperature=0.9)
-eval("arc.py", model="google/gemini-1.0-pro", max_connections=20)
+eval("arc.py", model="google/gemini-1.5-pro", max_connections=20)
 ```
 
 Use `inspect eval --help` to learn about all of the available generation config options.
 
 ## Model Args
 
-If there is an additional aspect of a model you want to tweak that isn't covered by the `GenerateConfig`, you can use model args to pass additional arguments to model clients. For example, here we specify the `transport` option for a Google Gemini model:
+If there is an additional aspect of a model you want to tweak that isn't covered by the `GenerateConfig`, you can use model args to pass additional arguments to model clients. For example, here we specify the `location` option for a Google Gemini model:
 
 ``` bash
-inspect eval arc.py --model google/gemini-1.0-pro -M transport:grpc
+inspect eval arc.py --model google/gemini-1.5-pro -M location=us-east5
 ```
 
 See the documentation for the requisite model provider for information on how model args are passed through to model clients.

{inspect_ai-0.3.68 → inspect_ai-0.3.70}/docs/options.qmd

@@ -124,7 +124,8 @@ Below are sections for the various categories of options supported by `inspect e
 | `--no-fail-on-error` | Do not fail the eval if errors occur within samples (instead, continue running other samples)                                                                         |
 | `--message-limit`    | Limit on total messages used for each sample.                                                                                                                         |
 | `--token-limit`      | Limit on total tokens used for each sample.                                                                                                                           |
-| `--time-limit`       | Limit on total execution time for each sample.                                                                                                                        |
+| `--time-limit`       | Limit on total running time for each sample.                                                                                                                        |
+| `--working-limit`       | Limit on total working time (model generation, tool calls, etc.) for each sample.                                                                                                                        |
 
 ## Eval Logs
 

{inspect_ai-0.3.68 → inspect_ai-0.3.70}/docs/providers.qmd

@@ -8,7 +8,6 @@ Inspect has support for a wide variety of language model APIs and can be extende
 
 {{< include _model-providers.md >}}
 
-
 ## OpenAI {#openai}
 
 To use the [OpenAI](https://platform.openai.com/) provider, install the `openai` package, set your credentials, and specify a model using the `--model` option:
@@ -88,15 +87,15 @@ Authentication is doing using the standard Google Cloud CLI (i.e. if you have au
 
 ## Google {#google-gemini}
 
-To use the [Google](https://ai.google.dev/) provider, install the `google-generativeai` package, set your credentials, and specify a model using the `--model` option:
+To use the [Google](https://ai.google.dev/) provider, install the `google-genai` package, set your credentials, and specify a model using the `--model` option:
 
 ``` bash
-pip install google-generativeai
+pip install google-genai
 export GOOGLE_API_KEY=your-google-api-key
 inspect eval arc.py --model google/gemini-1.5-pro
 ```
 
-For the `google` provider, custom model args (`-M`) are forwarded to the `genai.configure` function.
+For the `google` provider, custom model args (`-M`) are forwarded to the `genai.Client` function.
 
 The following environment variables are supported by the Google provider
 
@@ -105,12 +104,38 @@ The following environment variables are supported by the Google provider
 | `GOOGLE_API_KEY`  | API key credentials (required).  |
 | `GOOGLE_BASE_URL` | Base URL for requests (optional) |
 
+### Gemini on Vertex AI
+
+To use Google Gemini models on Vertex, you can use the standard `google` model provider with the `vertex` qualifier (e.g. `google/vertex/gemini-2.0-flash`). You should also set two environment variables indicating your project ID and region. Here is a complete example:
+
+``` bash
+export GOOGLE_CLOUD_PROJECT=project-12345
+export GOOGLE_CLOUD_LOCATION=us-east5
+inspect eval ctf.py --model google/vertex/gemini-2.0-flash
+```
+
+You can alternatively pass the project and location as custom model args (`-M`). For example:
+
+``` bash
+inspect eval ctf.py --model google/vertex/gemini-2.0-flash \
+   -M project=project-12345 -M location=us-east5
+```
+
+Authentication is done using the standard Google Cloud CLI. For example:
+
+```bash
+gcloud auth application-default login
+```
+
+If you have authorised the CLI then no additional auth is needed for the model API.
+
 ### Safety Settings {#safety-settings}
 
-Google models make available [safety settings](https://ai.google.dev/gemini-api/docs/safety-settings) that you can adjust to determine what sorts of requests will be handled (or refused) by the model. The four categories of safety settings are as follows:
+Google models make available [safety settings](https://ai.google.dev/gemini-api/docs/safety-settings) that you can adjust to determine what sorts of requests will be handled (or refused) by the model. The five categories of safety settings are as follows:
 
 | Category | Description |
-|-----------------------|------------------------------------------------|
+|------------------------|-----------------------------------------------|
+| `civic_integrity`   |  Election-related queries.          |
 | `sexually_explicit` | Contains references to sexual acts or other lewd content. |
 | `hate_speech` | Content that is rude, disrespectful, or profane. |
 | `harassment` | Negative or harmful comments targeting identity and/or protected attributes. |
@@ -121,7 +146,7 @@ Google models make available [safety settings](https://ai.google.dev/gemini-api/
 For each category, the following block thresholds are available:
 
 | Block Threshold | Description |
-|------------------------|------------------------------------------------|
+|-------------------------|-----------------------------------------------|
 | `none` | Always show regardless of probability of unsafe content |
 | `only_high` | Block when high probability of unsafe content |
 | `medium_and_above` | Block when medium or high probability of unsafe content |
@@ -250,20 +275,24 @@ You can also use this option to disable tool emulation for Llama models with `em
 ## Vertex AI {#vertex-ai}
 
 ::: {.callout-note appearance="simple"}
-Vertex AI is a distinct service from Google AI, see a comparison matrix [here](https://cloud.google.com/vertex-ai/generative-ai/docs/migrate/migrate-google-ai#google-ai). Make sure you are using the appropriate model provider.
+If you are using Gemini or Anthropic models on Vertex AI, we recommend you use the Google and Anthropic providers (respectively) which both support models hosted on Vertex:
+
+- [Anthropic on Vertex AI](#anthropic-on-vertex-ai)
+- [Gemini on Vertex AI](#gemini-on-vertex-ai)
+
+If you are using other models hosted on Vertex (e.g. Mistral, Llama, Gemma, etc.) then you should instead use the `vertex` provider as described below.
 :::
 
 To use the [Vertex AI](https://cloud.google.com/vertex-ai) provider, install the `google-cloud-aiplatform` package, [configure your environment](https://cloud.google.com/vertex-ai/generative-ai/docs/start/quickstarts/quickstart-multimodal#expandable-1) for Vertex API access, and specify a model using the `--model` option:
 
 ``` bash
-inspect eval eval.py --model vertex/gemini-1.5-flash
+inspect eval eval.py --model vertex/mistral-large-2411
 ```
 
 The core libraries for Vertex AI interact directly with Google Cloud Platform so this provider doesn't use the standard `BASE_URL`/`API_KEY` approach that others do. Consequently you don't need to set these environment variables.
 
 Vertex AI also provides the same `safety_settings` outlined in the [Google](#safety-settings) provider.
 
-If you are using Anthropic on Vertex AI, you can alternatively use the [Anthropic provider](#anthropic-on-vertex-ai) as your means of access.
 
 ## Together AI {#together-ai}
 

{inspect_ai-0.3.68 → inspect_ai-0.3.70}/docs/reasoning.qmd

@@ -32,9 +32,7 @@ Reasoning traces are currently captured in two ways:
 
 2.  From `<think></think>` tags included in the main assistant `content` (this is how Together, Groq, and Ollama currently present reasoning traces).
 
-Gemini 2.0 Flash Thinking currently includes its reasoning inline with response content (there is currently no structured way to extract it, although this seems likely to change in the future).
-
-We have confirmed that `reasoning` traces can be extracted from models using the `together`, `groq`, and `ollama` providers.
+3.  Using the `Final Answer: ` delimiter currently used by Gemini 2.0 Flash Thinking (the API has a separate field for `thinking` but it is not currently used in responses).
 
 We would like to confirm this for other providers (e.g. `bedrock` and `azureai`) but as of yet have not been able to access reasoning models on those services (we'd very much welcome others contributing here, either to confirm that things work or to provide PRs which make the required changes).
 

{inspect_ai-0.3.68 → inspect_ai-0.3.70}/docs/reference/_sidebar.yml

@@ -352,6 +352,8 @@ website:
           href: reference/inspect_ai.log.qmd#modelevent
         - text: ToolEvent
           href: reference/inspect_ai.log.qmd#toolevent
+        - text: SandboxEvent
+          href: reference/inspect_ai.log.qmd#sandboxevent
         - text: ApprovalEvent
           href: reference/inspect_ai.log.qmd#approvalevent
         - text: InputEvent

{inspect_ai-0.3.68 → inspect_ai-0.3.70}/docs/reference/inspect_ai.log.qmd

@@ -46,6 +46,7 @@ title: "inspect_ai.log"
 ### StoreEvent
 ### ModelEvent
 ### ToolEvent
+### SandboxEvent
 ### ApprovalEvent
 ### InputEvent
 ### StoreEvent

{inspect_ai-0.3.68 → inspect_ai-0.3.70}/docs/reference/inspect_ai.util.qmd

@@ -34,6 +34,7 @@ title: "inspect_ai.util"
 ### trace_action
 ### trace_message
 
+
 ## Sandbox
 
 ### sandbox

{inspect_ai-0.3.68 → inspect_ai-0.3.70}/docs/scorers.qmd

@@ -117,7 +117,7 @@ The built-in model graded scorers also support using multiple grader models (whe
 ``` python
 model_graded_qa(
     model = [
-        "google/gemini-1.0-pro",
+        "google/gemini-1.5-pro",
         "anthropic/claude-3-opus-20240229" 
         "together/meta-llama/Llama-3-70b-chat-hf",
     ]
@@ -740,7 +740,7 @@ log = eval(popularity, model="openai/gpt-4")[0]
 grader_models = [
     "openai/gpt-4",
     "anthropic/claude-3-opus-20240229",
-    "google/gemini-1.0-pro",
+    "google/gemini-1.5-pro",
     "mistral/mistral-large-latest"
 ]
 
@@ -760,7 +760,7 @@ log = read_eval_log(input_log_path)
 grader_models = [
     "openai/gpt-4",
     "anthropic/claude-3-opus-20240229",
-    "google/gemini-1.0-pro",
+    "google/gemini-1.5-pro",
     "mistral/mistral-large-latest"
 ]
 

{inspect_ai-0.3.68 → inspect_ai-0.3.70}/docs/tasks.qmd

@@ -41,31 +41,32 @@ For convenience, tasks always define a default solver. That said, it is often de
 
 While many tasks can be defined with only a dataset, solver, and scorer, there are lots of other useful `Task` options. We won't describe these options in depth here, but rather provide a list along with links to other sections of the documentation that cover their usage:
 
-+------------------+------------------------------------------------------+-----------------------------------------------------------+
-| Option           | Description                                          | Docs                                                      |
-+==================+======================================================+===========================================================+
-| `epochs`         | Epochs to run for each dataset sample.               | [Epochs](scorers.qmd#reducing-epochs)                     |
-+------------------+------------------------------------------------------+-----------------------------------------------------------+
-| `config`         | Config for model generation.                         | [Generate Config](options.qmd#model-generation)           |
-+------------------+------------------------------------------------------+-----------------------------------------------------------+
-| `setup`          | Setup solver(s) to run prior to the main solver.     | [Sample Setup](#setup-parameter)                          |
-+------------------+------------------------------------------------------+-----------------------------------------------------------+
-| `sandbox`        | Sandbox configuration for un-trusted code execution. | [Sandboxing](sandboxing.qmd)                              |
-+------------------+------------------------------------------------------+-----------------------------------------------------------+
-| `approval`       | Approval policy for tool calls.                      | [Tool Approval](approval.qmd)                             |
-+------------------+------------------------------------------------------+-----------------------------------------------------------+
-| `metrics`        | Metrics to use in place of scorer metrics.           | [Scoring Metrics](scorers.qmd#scoring-metrics)            |
-+------------------+------------------------------------------------------+-----------------------------------------------------------+
-| `fail_on_error`  | Failure tolerance for samples.                       | [Sample Failure](errors-and-limits.qmd#failure-threshold) |
-+------------------+------------------------------------------------------+-----------------------------------------------------------+
-| `message_limit`\ | Limits to apply to sample execution.                 | [Sample Limits](errors-and-limits.qmd#sample-limits)      |
-| `token_limit`\   |                                                      |                                                           |
-| `time_limit`     |                                                      |                                                           |
-+------------------+------------------------------------------------------+-----------------------------------------------------------+
-| `name`\          | Eval log attributes for task.                        | [Eval Logs](eval-logs.qmd)                                |
-| `version`\       |                                                      |                                                           |
-| `metadata`       |                                                      |                                                           |
-+------------------+------------------------------------------------------+-----------------------------------------------------------+
++-------------------+------------------------------------------------------+-----------------------------------------------------------+
+| Option            | Description                                          | Docs                                                      |
++===================+======================================================+===========================================================+
+| `epochs`          | Epochs to run for each dataset sample.               | [Epochs](scorers.qmd#reducing-epochs)                     |
++-------------------+------------------------------------------------------+-----------------------------------------------------------+
+| `config`          | Config for model generation.                         | [Generate Config](options.qmd#model-generation)           |
++-------------------+------------------------------------------------------+-----------------------------------------------------------+
+| `setup`           | Setup solver(s) to run prior to the main solver.     | [Sample Setup](#setup-parameter)                          |
++-------------------+------------------------------------------------------+-----------------------------------------------------------+
+| `sandbox`         | Sandbox configuration for un-trusted code execution. | [Sandboxing](sandboxing.qmd)                              |
++-------------------+------------------------------------------------------+-----------------------------------------------------------+
+| `approval`        | Approval policy for tool calls.                      | [Tool Approval](approval.qmd)                             |
++-------------------+------------------------------------------------------+-----------------------------------------------------------+
+| `metrics`         | Metrics to use in place of scorer metrics.           | [Scoring Metrics](scorers.qmd#scoring-metrics)            |
++-------------------+------------------------------------------------------+-----------------------------------------------------------+
+| `fail_on_error`   | Failure tolerance for samples.                       | [Sample Failure](errors-and-limits.qmd#failure-threshold) |
++-------------------+------------------------------------------------------+-----------------------------------------------------------+
+| `message_limit`\  | Limits to apply to sample execution.                 | [Sample Limits](errors-and-limits.qmd#sample-limits)      |
+| `token_limit`\    |                                                      |                                                           |
+| `time_limit`\     |                                                      |                                                           |
+| `working_limit`   |                                                      |                                                           |
++-------------------+------------------------------------------------------+-----------------------------------------------------------+
+| `name`\           | Eval log attributes for task.                        | [Eval Logs](eval-logs.qmd)                                |
+| `version`\        |                                                      |                                                           |
+| `metadata`        |                                                      |                                                           |
++-------------------+------------------------------------------------------+-----------------------------------------------------------+
 
 : {tbl-colwidths=\[25,50,25\]}
 
@@ -212,7 +213,7 @@ inspect eval ctf.py --solver=ctf_agent -S max_attempts=5
 
 Note the use of the `-S` CLI option to pass an alternate value for `max_attempts` to the `ctf_agent()` solver.
 
-### Setup Parameter
+### Setup Parameter {#setup-parameter}
 
 In some cases, there will be important steps in the setup of a task that *should not be substituted* when another solver is used with the task. For example, you might have a step that does dynamic prompt engineering based on values in the sample `metadata` or you might have a step that initialises resources in a sample's sandbox.
 
@@ -242,6 +243,26 @@ def ctf(solver: Solver | None = None):
     )
 ```
 
+## Task Cleanup
+
+You can use the `cleanup` parameter for executing code at the end of each sample run. The `cleanup` function is passed the `TaskState` and is called for both successful runs and runs where are exception is thrown. Extending the example from above:
+
+```python
+async def ctf_cleanup(state: TaskState):
+    ## perform cleanup
+    ...
+
+Task(
+    dataset=read_dataset(),
+    setup=ctf_prompt(),
+    solver=solver,
+    cleanup=ctf_cleanup,
+    scorer=includes()
+)
+```
+
+Note that like solvers, cleanup functions should be `async`.
+
 ## Task Reuse {#task-reuse}
 
 The basic mechanism for task re-use is to create flexible and adaptable base `@task` functions (which often have many parameters) and then derive new higher-level tasks from them by creating additional `@task` functions that call the base function.

{inspect_ai-0.3.68 → inspect_ai-0.3.70}/pyproject.toml

@@ -115,7 +115,7 @@ dev = [
     "aioboto3", 
     "azure-ai-inference",
     "google-cloud-aiplatform",
-    "google-generativeai", 
+    "google-genai",
     "goodfire",
     "griffe",
     "groq", 
@@ -133,7 +133,7 @@ dev = [
     "pytest-cov",
     "pytest-dotenv",
     "pytest-xdist",
-    "ruff==0.9.5", # match version specified in .pre-commit-config.yaml
+    "ruff==0.9.6", # match version specified in .pre-commit-config.yaml
     "textual-dev>=0.86.2",
     "types-Markdown",
     "types-PyYAML",

{inspect_ai-0.3.68 → inspect_ai-0.3.70}/requirements.txt

@@ -4,7 +4,7 @@ beautifulsoup4
 click>=8.1.3
 debugpy
 docstring-parser>=0.16
-fsspec>=2021.09.0
+fsspec>=2023.1.0,<=2024.12.0 # align with hf datasets to prevent pip errors
 httpx
 ijson>=3.2.0
 jsonlines>=3.0.0
@@ -23,6 +23,6 @@ s3fs>=2023
 semver>=3.0.0
 shortuuid
 tenacity
-textual>=0.86.2,<=1.0.0
+textual>=0.86.2
 typing_extensions>=4.9.0
 zipp>=3.19.1 # not directly required, pinned by Snyk to avoid a vulnerability

{inspect_ai-0.3.68 → inspect_ai-0.3.70}/src/inspect_ai/_cli/eval.py

@@ -218,9 +218,15 @@ def eval_options(func: Callable[..., Any]) -> Callable[..., click.Context]:
     @click.option(
         "--time-limit",
         type=int,
-        help="Limit on total execution time for each sample.",
+        help="Limit on total running time for each sample.",
         envvar="INSPECT_EVAL_TIME_LIMIT",
     )
+    @click.option(
+        "--working-limit",
+        type=int,
+        help="Limit on total working time (e.g. model generation, tool calls, etc.) for each sample.",
+        envvar="INSPECT_EVAL_WORKING_LIMIT",
+    )
     @click.option(
         "--fail-on-error",
         type=float,
@@ -468,6 +474,7 @@ def eval_command(
     message_limit: int | None,
     token_limit: int | None,
     time_limit: int | None,
+    working_limit: int | None,
     max_samples: int | None,
     max_tasks: int | None,
     max_subprocesses: int | None,
@@ -518,6 +525,7 @@ def eval_command(
         message_limit=message_limit,
         token_limit=token_limit,
         time_limit=time_limit,
+        working_limit=working_limit,
         max_samples=max_samples,
         max_tasks=max_tasks,
         max_subprocesses=max_subprocesses,
@@ -629,6 +637,7 @@ def eval_set_command(
     message_limit: int | None,
     token_limit: int | None,
     time_limit: int | None,
+    working_limit: int | None,
     max_samples: int | None,
     max_tasks: int | None,
     max_subprocesses: int | None,
@@ -684,6 +693,7 @@ def eval_set_command(
         message_limit=message_limit,
         token_limit=token_limit,
         time_limit=time_limit,
+        working_limit=working_limit,
         max_samples=max_samples,
         max_tasks=max_tasks,
         max_subprocesses=max_subprocesses,
@@ -737,6 +747,7 @@ def eval_exec(
     message_limit: int | None,
     token_limit: int | None,
     time_limit: int | None,
+    working_limit: int | None,
     max_samples: int | None,
     max_tasks: int | None,
     max_subprocesses: int | None,
@@ -817,6 +828,7 @@ def eval_exec(
             message_limit=message_limit,
             token_limit=token_limit,
             time_limit=time_limit,
+            working_limit=working_limit,
             max_samples=max_samples,
             max_tasks=max_tasks,
             max_subprocesses=max_subprocesses,