PyPI - inspect-ai - Versions diffs - 0.3.52__tar.gz → 0.3.54__tar.gz - Mend

inspect-ai 0.3.52tar.gz → 0.3.54tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (818) hide show

{inspect_ai-0.3.52 → inspect_ai-0.3.54}/CHANGELOG.md RENAMED Viewed

@@ -1,5 +1,35 @@
 # Changelog
+## v0.3.54 (26 December 2024)
+- [Action tracing](https://github.com/UKGovernmentBEIS/inspect_ai/pull/1038) for diagnosing runs with unterminated action (e.g. model calls, docker commands, etc.).
+- Provide default timeout/retry for docker compose commands to mitigate unreliability in some configurations.
+- Switch to sync S3 writes to overcome unreliability observed when using async interface.
+- Task display: Added `--no-score-display` option to disable realtime scoring metrics.
+- Bugfix: Fix failure to fully clone samples that have message lists as input.
+- llama-cpp-python: Support for `logprobs`.
+## v0.3.53 (20 December 2024)
+- OpenAI: Support for o1 including native tool calling and `reasoning_effort` generation option.
+- Task API: Introduce `setup` step that always runs even if `solver` is replaced.
+- Bedrock: Support for tool calling on Nova models.
+- Bedrock: Support for custom `model_args` passed through to `session.Client`.
+- Bedrock: Support for `jpeg` images.
+- Bedrock: Correct max_tokens for llama3-8b, llama3-70b models on Bedrock.
+- Inspect View: Various improvements to appearance of tool calls in transcript.
+- Task display: Ensure that widths of progress elements are kept consistent across tasks.
+- Sandboxes: New `max_sandboxes` option for (per-provider) maximum number of running sandboxes.
+- Sandboxes: Remove use of aiofiles to mitigate potential for threading deadlocks.
+- Concurrency: Do not use `max_tasks` as a lower bound for `max_samples`.
+- Log recorder: Always re-open log buffer for `eval` format logs.
+- Bugfix: Proper handling of text find for eval raw JSON display
+- Bugfix: Correct handling for `--sample-id` integer comparisons.
+- Bugfix: Proper removal of model_args with falsey values (explicit check for `None`)
+- Bugfix: Properly handle custom metrics that return dictionaries or lists
+- Bugfix: Proper sample count display when retrying an evaluation
+- Bugfix: Fix inability to define and run tasks in a notebook.
 ## v0.3.52 (13 December 2024)
 - Eval: `--sample-id` option for evaluating specific sample id(s).
@@ -26,10 +56,10 @@
 - Task display: Resolve issue where task display would lose mouse input after VS Code reload.
 - Datasets: Validate that all IDs in datasets are unique (as several downstream problems occur w/ duplicate IDs).
 - Inspect View: Fix issue with incorrectly displayed custom tool views.
-- Human approval: Use fullcreen display (makes approval UI async and enables rapid processing of approvals via the `Enter` key).
+- Human approval: Use fullscreen display (makes approval UI async and enables rapid processing of approvals via the `Enter` key).
 - Added `input_panel()` API for adding custom panels to the fullscreen task display.
 - Log recorder: Methods are now async which will improve performance for fsspec filesystems with async implementations (e.g. S3)
-- Log recorder: Improve `.eval` log reading performance for remote filesystem (eaglery fetch log to local buffer).
+- Log recorder: Improve `.eval` log reading performance for remote filesystem (eagerly fetch log to local buffer).
 - Add `token_usage` property to `TaskState` which has current total tokens used across all calls to `generate()` (same value that is used for enforcing token limits).
 - Add `time` field to `ModelOutput` that records total time spent within call to ModelAPI `generate()`.
 - Web browser: Remove base64 images from web page contents (prevent filling up model context with large images).
@@ -42,7 +72,7 @@
 - Bugfix: Correct rate limit error display when running in fullscreen mode.
 - Bugfix: `hf_dataset` now explicitly requires the `split` argument (previously, it would crash when not specified).
 - Bugfix: Prevent cascading textual error when an error occurs during task initialisation.
-- Bugfix: Correctly restore sample summaries from log file after abend.
+- Bugfix: Correctly restore sample summaries from log file after amend.
 - Bugfix: Report errors that occur during task finalisation.
 ## v0.3.49 (03 December 2024)
@@ -50,9 +80,9 @@
 - Logging: Only call CreateBucket on Amazon S3 when the bucket does not already exist.
 - Improve cancellation feedback and prevent multiple cancellations when using fullscreen display.
 - Inspect View: Prevent circular reference error when rendering complex tool input.
-- Inspect View: Resolve display issue with sorting by sample then epoch.
+- Inspect View: Resolve display issue with sorting by sample then epoch.
-## v0.3.48 (01 December 2024)
+## v0.3.48 (01 December 2024)
 - [Realtime display](https://github.com/UKGovernmentBEIS/inspect_ai/pull/865) of sample transcripts (including ability to cancel running samples).
 - Scoring: When using a dictionary to map metrics to score value dictionaries, you may now use globs as keys. See our [scorer documentation](https://inspect.ai-safety-institute.org.uk/scorers.html#sec-multiple-scorers) for more information.
@@ -64,9 +94,9 @@
 - Google: Support for tools/functions that have no parameters.
 - Google/Vertex: Support for `logprobs` and other new 1.5 (002 series) options.
 - AzureAI: Change default max_tokens for Llama models to 2048 (4096 currently yields an error w/ Llama 3.1).
-- Mistral: Various compatiblity changes for their client and tool calling implementation.
+- Mistral: Various compatibility changes for their client and tool calling implementation.
 - Handle exponents in numeric normalisation for match, include, and answer scorers.
-- hf_dataset: Added `cached` argument to control whether to use a previously cached version of the dataset if available (defaults to `True`).
+- hf_dataset: Added `cached` argument to control whether to use a previously cached version of the dataset if available (defaults to `True`).
 - hf_dataset: Added `revision` option to load a specific branch or commit SHA (when using `revision` datasets are always revalidated on Hugging Face, i.e. `cached` is ignored).
 - Log viewer: Display sample ids rather than indexes.
 - Log viewer: Add timestamps to transcript events.
@@ -74,9 +104,9 @@
 - Log viewer: Show custom tool call views in messages display.
 - Bugfix: Correctly read and forward image detail property.
 - Bugfix: Correct resolution of global eval override of task or sample sandboxes.
-- Bugfix: Don't do eval log listing on background threads (s3fs can deadlock when run from mutliple threads).
+- Bugfix: Don't do eval log listing on background threads (s3fs can deadlock when run from multiple threads).
-## v0.3.47 (18 November 2024)
+## v0.3.47 (18 November 2024)
 - Basic agent: Ensure that the scorer is only run once when max_attempts = 1.
 - Basic agent: Support custom function for incorrect_message reply to model.
@@ -87,7 +117,7 @@
 - Scoring: When using a dictionary to map metrics to score value dictionaries, you may now use globs as keys. See our [scorer documentation](https://inspect.ai-safety-institute.org.uk/scorers.html#sec-multiple-scorers) for more information.
 - Sample limit events will now appear in the transcript if a limit (e.g. message, token, or time limit) halt a sample. The sample list and sample detail also display the limit, if applicable.
-## v0.3.46 (12 November 2024)
+## v0.3.46 (12 November 2024)
 - [eval](https://inspect.ai-safety-institute.org.uk/eval-logs.html#sec-log-format) is now the default log format (use `--log-format=json` to use old format).
 - Base 64 images are now logged by default for all log formats (disable with `--no-log-images`).
@@ -112,7 +142,7 @@
 - Log model calls when model providers return bad request errors
 - Better lay out large numbers of configuration and parameters when displaying log files.
 - The log viewer now properly displays sample scores for running tasks.
-- Add `metadata` field to `ModelOutput` and provide varioius fields for the Groq provider.
+- Add `metadata` field to `ModelOutput` and provide various fields for the Groq provider.
 ## v0.3.44 (04 November 2024)
@@ -128,7 +158,7 @@
 - View: properly render complex score objects in transcript.
 - Write custom tool call views into transcript for use by Inspect View.
 - Use `casefold()` for case-insensitive compare in `includes()`, `match()`, `exact()`, and `f1()` scorers.
-- OpenAI: eliminate use of `strict` tool calling (sporadically supported across models and we already interally validate).
+- OpenAI: eliminate use of `strict` tool calling (sporadically supported across models and we already internally validate).
 - Mistral: fix bug where base_url was not respected when passing both an api_key and base_url.
 - Don't include package scope for task name part of log files.
 - Improve performance of write_file for Docker sandboxes.
@@ -181,7 +211,7 @@
 - Correct llama3 and o1 tool calling when empty arguments passed.
 - Allow resolution of any sandbox name when there is only a single environment.
 - Introduce `--log-level-transcript` option for separate control of log entries recorded in the eval log file
-- Improve mime type detection for image content encoding (fixes issues w/ webp images).
+- Improve mime type detection for image content encoding (fixes issues w/ webp images).
 - Fix memory leak in Inspect View worker-based JSON parsing.
 - Add `fail_on_error` option for `eval_retry()` and `inspect eval-retry`.
 - Defer resolving helper models in `self_critique()` and `model_graded_qa()`.
@@ -204,7 +234,7 @@
 ## v0.3.39 (3 October 2024)
 - The sample transcript will now display the target for scoring in the Score Event (for newly run evaluations).
-- Provide setter for `max_messages` on `TaskState`.
+- Provide setter for `max_messages` on `TaskState`.
 - Provide `max_messages` option for `basic_agent()` (defaulting to 50) and use it rather than any task `max_messages` defined.
 - Improved implementation of disabling parallel tool calling (also fixes a transcript issue introduced by the original implementation).
 - Improve quality of error messages when a model API key environment variable is missing.
@@ -215,7 +245,7 @@
 - Rename `web_browser_tools()` to `web_browser()`, and don't export individual web browsing tools.
 - Add `parallel` option to `@tool` decorator and specify `parallel=False` for web browsing tools.
 - Improve prompting for web browser tools using more explicit examples.
-- Improve prompting for `</tool_call>` end sequence for Llama models.
+- Improve prompting for `</tool_call>` end sequence for Llama models.
 - Fix issue with failure to execute sample setup scripts.
 ## v0.3.37 (2 October 2024)
@@ -224,7 +254,7 @@
 ## v0.3.36 (2 October 2024)
-- [Web Browser](https://inspect.ai-safety-institute.org.uk/tools.html#sec-web-browser) tool which provides a headless Chromimum browser that supports navigation, history, and mouse/keyboard interactions.
+- [Web Browser](https://inspect.ai-safety-institute.org.uk/tools.html#sec-web-browser) tool which provides a headless Chromium browser that supports navigation, history, and mouse/keyboard interactions.
 - `auto_id` option for dataset readers to assign an auto-incrementing ID to records.
 - Task args: don't attempt to serialise registry objects that don't have captured parameters.
@@ -242,7 +272,7 @@
 - Support for `max_tokens` on OpenAI o1 models (map to `max_completion_tokens`).
 - Fix regression of log and debug options on `inspect view`
-- Improved focus management for Insepct View
+- Improved focus management for Inspect View
 - Raise error if `epochs` is less than 1
 - Improve code parsing for HumanEval (compatibility with Llama model output)
@@ -253,9 +283,9 @@
 - Option to disable ANSI terminal output with `--no-ansi` or `INSPECT_NO_ANSI`
 - Add chain of thought option to `multiple_choice()` and export `MultipleChoiceTemplate` enumeration
 - Allow Docker sandboxes configured with `x-default` to be referred to by their declared service name.
-- Improved error messages for Docier sandbox initialisation.
+- Improved error messages for Docker sandbox initialisation.
 - Improve legibility of Docker sandbox log entries (join rather than displaying as array)
-- Display user message immediately proceding assistant message in model call transcripts.
+- Display user message immediately proceeding assistant message in model call transcripts.
 - Display images created by tool calls in the Viewer.
 - Fix duplicated tool call output display in Viewer for Gemini and Llama models.
 - Require a `max_messages` for use of `basic_agent()` (as without it, the agent could end up in an infinite loop).
@@ -270,8 +300,8 @@
 - Fix issue w/ subtasks not getting a fresh store() (regression from introduction of `fork()` in v0.3.30)
 - Fix issue w/ subtasks that return None invalidating the log file.
-- Make subtasks collapsable in Inspect View.
-- Improved error reporting for missing `web_search()` provider environment variables.
+- Make subtasks collapsible in Inspect View.
+- Improved error reporting for missing `web_search()` provider environment variables.
 ## v0.3.31 (24 September 2024)
@@ -309,7 +339,7 @@
 - Don't log base64 images by default (re-enable logging with `--log-images`).
 - Provide unique tool id when parsing tool calls for models that don't support native tool usage.
 - Fix bug that prevented `epoch_reducer` from being used in eval-retry.
-- Fix bug that prevented eval() level `epoch` from overriding task level `epoch`.
+- Fix bug that prevented eval() level `epoch` from overriding task level `epoch`.
 ## v0.3.28 (14 September 2024)
@@ -353,7 +383,7 @@
 - Add optional user parameter to SandboxEnvironment.exec for specifying the user. Currently only DockerSandboxEnvironment is supported.
 - Fix issue with resolving Docker configuration files when not running from the task directory.
 - Only populate Docker compose config metadata values when they are used in the file.
-- Treat Sandbox exec `cwd` that are relative paths as relative to sample working directry.
+- Treat Sandbox exec `cwd` that are relative paths as relative to sample working directory.
 - Filter base64 encoded images out of model API call logs.
 - Raise error when a Solver does not return a TaskState.
 - Only run tests that use model APIs when the `--runapi` flag is passed to `pytest` (prevents unintended token usage)
@@ -362,7 +392,6 @@
 - Only enable `strict` mode for OpenAI tool calls when all function parameters are required.
 - Added [MMMU](https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/mmmu), [CommonsenseQA](https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/commonsense_qa), [MMLU-Pro](https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/mmlu_pro), and [XSTest](https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/xstest) benchmarks.
 ## v0.3.25 (25 August 2024)
 - [Store](https://inspect.ai-safety-institute.org.uk/agents-api.html#sharing-state) for manipulating arbitrary sample state from within solvers and tools.
@@ -383,7 +412,7 @@
 ## v0.3.23 (16 August 2024)
 - Support for tool calling for Llama 3.1 models on Azure AI and CloudFlare.
-- Incrase default `max_tokens` from 1024 to 2048.
+- Increase default `max_tokens` from 1024 to 2048.
 - Record individual sample reductions along with results for multi-epoch evals.
 - Change default to not log base64 encoded versions of images, as this often resulted in extremely large log files (use `--log-images` to opt back in).
 - Update to new Mistral API (v1.0.1 of `mistralai` is now required).
@@ -405,13 +434,13 @@
 - Set Claude default `max_tokens` to 4096
 - Combine user and assistant messages for Vertex models.
 - Warn when using the `name` parameter with task created from `@task` decorated function.
-- Make sample `metadata` available in prompt, grading, and self-criqique templates.
+- Make sample `metadata` available in prompt, grading, and self-critique templates.
 - Retry on several additional OpenAI errors (APIConnectionError | APITimeoutError | InternalServerError)
 - Fix a regression which would cause the 'answer' to be improperly recorded when scoring a sample.
 ## v0.3.20 (03 August 2024)
-- `Epochs` data type for specifying epochs and reducers together (deprecated `epochs_reducer` argument).
+- `Epochs` data type for specifying epochs and reducers together (deprecated `epochs_reducer` argument).
 - Enable customisation of model generation cache dir via `INSPECT_CACHE_DIR` environment variable.
 - Use doc comment description rather than `prompt` attribute of `@tool` for descriptions.
 - Include examples section from doc comments in tool descriptions.
@@ -464,8 +493,6 @@
 - Set maximum rate limit backoff time to 30 minutes
 - Retry with exponential backoff for web_search Google provider.
 ## v0.3.18 (14 July 2024)
 - [Multiple Scorers](https://inspect.ai-safety-institute.org.uk/scorers.html#sec-multiple-scorers) are now supported for evaluation tasks.
@@ -474,7 +501,7 @@
 - Improved handling of very large (> 100MB) log files in Inspect View.
 - Use `network_mode: none` for disabling networking by default in Docker tool environments.
 - Shorten the default shutdown grace period for Docker container cleanup to 1 second.
-- Allow sandbox environent providers to specify a default `max_samples` (set to 25 for the Docker provider).
+- Allow sandbox environment providers to specify a default `max_samples` (set to 25 for the Docker provider).
 - Prevent concurrent calls to `eval_async()` (unsafe because of need to change directories for tasks). Parallel task evaluation will instead be implemented as a top-level feature of `eval()` and `eval_async()`.
 - Match scorers now return answers consistently even when there is no match.
 - Relocate tool related types into a new top-level `inspect_ai.tool` module (previous imports still work fow now, but result in a runtime deprecation warning).
@@ -485,7 +512,7 @@
 - Optional increased control over the tool use loop via the `call_tools()` function and new `tool_calls` parameter for `generate()`.
 - New `per_epoch` option for `CachePolicy` to allow caching to ignore epochs.
-- Correctly handle `choices` and `files` when converting `Sample` images to base64.
+- Correctly handle `choices` and `files` when converting `Sample` images to base64.
 ## v0.3.16 (24 June 2024)
@@ -564,7 +591,7 @@
 -   Add `multi_scorer()` and `majority_vote()` functions for combining multiple scorers into a single score.
 -   Add support for multiple model graders in `model_graded_qa()`.
 -   Raise `TypeError` for solvers and scorers not declared as `async`.
--   Fallback to standard parase if `NaN` or `Inf` is encountered while reading log file header.
+-   Fallback to standard parse if `NaN` or `Inf` is encountered while reading log file header.
 -   Remove deprecated support for matching partial model names (e.g. "gpt" or "claude").
 ## v0.3.8 (07 May 2024)
@@ -596,7 +623,7 @@
 -   `write_eval_log()` now ignores unserializable objects in metadata fields.
 -   `read_eval_log()` now takes a `str` or `FileInfo` (for compatibility w/ list returned from `list_eval_logs()`).
 -   Registry name looks are now case sensitive (fixes issue w/ loading tasks w/ mixed case names).
--   Resiliancy to Python syntax errors that occur when enumerating tasks in a directory.
+-   Resiliency to Python syntax errors that occur when enumerating tasks in a directory.
 -   Do not throw error if unable to parse or load `.ipynb` file due to lack of dependencies (e.g. `nbformat`).
 -   Various additions to log viewer display (log file name, dataset/scorer in listing, filter by complex score types).
 -   Improvements to markdown rendering in log viewer (don't render intraword underscores, escape html tags).

{inspect_ai-0.3.52/src/inspect_ai.egg-info → inspect_ai-0.3.54}/PKG-INFO RENAMED Viewed

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: inspect_ai
-Version: 0.3.52
+Version: 0.3.54
 Summary: Framework for large language model evaluations
 Author: UK AI Safety Institute
 License: MIT License
@@ -20,7 +20,6 @@ Classifier: Operating System :: OS Independent
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE
-Requires-Dist: aiofiles
 Requires-Dist: aiohttp>=3.9.0
 Requires-Dist: anyio>=4.4.0
 Requires-Dist: beautifulsoup4
@@ -71,7 +70,6 @@ Requires-Dist: pytest-xdist; extra == "dev"
 Requires-Dist: ruff==0.8.3; extra == "dev"
 Requires-Dist: textual-dev>=0.86.2; extra == "dev"
 Requires-Dist: types-PyYAML; extra == "dev"
-Requires-Dist: types-aiofiles; extra == "dev"
 Requires-Dist: types-beautifulsoup4; extra == "dev"
 Requires-Dist: types-aioboto3; extra == "dev"
 Requires-Dist: types-boto3; extra == "dev"
@@ -98,22 +96,22 @@ To get started with Inspect, please see the documentation at <https://inspect.ai
 ***
 To work on development of Inspect, clone the repository and install with the `-e` flag and `[dev]` optional dependencies:
 ```bash
-$ git clone https://github.com/UKGovernmentBEIS/inspect_ai.git
-$ cd inspect_ai
-$ pip install -e ".[dev]"
+git clone https://github.com/UKGovernmentBEIS/inspect_ai.git
+cd inspect_ai
+pip install -e ".[dev]"
 ```
 Optionally install pre-commit hooks via
 ```bash
 make hooks
 ```
 Run linting, formatting, and tests via
 ```bash
 make check
 make test

{inspect_ai-0.3.52 → inspect_ai-0.3.54}/README.md RENAMED Viewed

@@ -8,22 +8,22 @@ To get started with Inspect, please see the documentation at <https://inspect.ai
 ***
 To work on development of Inspect, clone the repository and install with the `-e` flag and `[dev]` optional dependencies:
 ```bash
-$ git clone https://github.com/UKGovernmentBEIS/inspect_ai.git
-$ cd inspect_ai
-$ pip install -e ".[dev]"
+git clone https://github.com/UKGovernmentBEIS/inspect_ai.git
+cd inspect_ai
+pip install -e ".[dev]"
 ```
 Optionally install pre-commit hooks via
 ```bash
 make hooks
 ```
 Run linting, formatting, and tests via
 ```bash
 make check
 make test

{inspect_ai-0.3.52 → inspect_ai-0.3.54}/docs/_quarto.yml RENAMED Viewed

@@ -98,9 +98,6 @@ format:
      toc-depth: 3
      number-sections: false
      code-annotations: select
-   gfm:
-      output-ext: html.lmd
 execute:
   enabled: false

{inspect_ai-0.3.52 → inspect_ai-0.3.54}/docs/_sandboxenv-interface.md RENAMED Viewed

@@ -19,7 +19,7 @@ class SandboxEnvironment:
           PermissionError: If the user does not have
             permission to execute the command.
           OutputLimitExceededError: If an output stream
-            exceeds the 1 MiB limit.
+            exceeds the 10 MiB limit.
         """
         ...

{inspect_ai-0.3.52 → inspect_ai-0.3.54}/docs/eval-logs.qmd RENAMED Viewed

@@ -90,7 +90,7 @@ If you do need to interact with the underlying JSON (e.g., when reading logs fro
 ## Image Logging
-By default, full base64 encoded copies of images are included in the log file. Image logging will not create performance problems when using `.eval` logs, however if you are using `.json` logs then large numbers of images could become unweildy (i.e. if your `.json` log file grows to 100mb or larger as a result).
+By default, full base64 encoded copies of images are included in the log file. Image logging will not create performance problems when using `.eval` logs, however if you are using `.json` logs then large numbers of images could become unwieldy (i.e. if your `.json` log file grows to 100mb or larger as a result).
 You can disable this using the `--no-log-images` flag. For example, here we enable the `.json` log format and disable image logging:

{inspect_ai-0.3.52 → inspect_ai-0.3.54}/docs/extensions.qmd RENAMED Viewed

@@ -125,6 +125,10 @@ class PodmanSandboxEnvironment(SandboxEnvironment):
     def config_files(cls) -> list[str]:
         ...
+    @classmethod
+    def default_concurrency(cls) -> int | None:
+        ...
     @classmethod
     async def task_init(
         cls, task_name: str, config: SandboxEnvironmentConfigType | None
@@ -180,6 +184,7 @@ The class methods take care of various stages of initialisation, setup, and tear
 | Method | Lifecycle | Purpose |
 |-------------------|-------------------|----------------------------------|
 | `config_files()` | Called once to determine the names of 'default' config files for this provider (e.g. 'compose.yaml'). |  |
+| `default_concurrency()` | Called once to determine the default maximum number of sandboxes to run in parallel. Return `None` for no limit (the default behavior). |  |
 | `task_init()` | Called once for each unique sandbox environment config before executing the tasks in an `eval()` run. | Expensive initialisation operations (e.g. pulling or building images) |
 | `sample_init()` | Called at the beginning of each `Sample`. | Create `SandboxEnvironment` instances for the sample. |
 | `sample_cleanup()` | Called at the end of each `Sample` | Cleanup `SandboxEnvironment` instances for the sample. |
@@ -319,7 +324,7 @@ The `_registry.py` file serves a place to import things that you wan't registere
 from .approvers import auto_approver
 ```
-You can then register your `auto_approver` Inspect extension (and anyting else imported into `_registry.py`) like this in `pyproject.toml`:
+You can then register your `auto_approver` Inspect extension (and anything else imported into `_registry.py`) like this in `pyproject.toml`:
 ::: {.panel-tabset group="entry-points"}
 ## Setuptools

inspect_ai-0.3.54/docs/images/task-max-sandboxes.png ADDED Viewed

Binary file

{inspect_ai-0.3.52 → inspect_ai-0.3.54}/docs/parallelism.qmd RENAMED Viewed

@@ -122,13 +122,15 @@ This code will evaluate a total of 12 tasks (6 temperature variations against 2
 [Sandbox Environments](agents.qmd#sec-sandbox-environments) (e.g. Docker containers) often allocate resources on a per-sample basis, and also make use of the Inspect `subprocess()` function for executing commands within the environment.
-### Max Samples
+### Max Sandboxes
-The `max_samples` option determines how many samples are executed in parallel (and in the case of Docker containers how many containers are run in parallel). By default, `max_samples` is set to `max_connections` so that the connection to the Model API can be fully utilised.
+The `max_sandboxes` option determines how many sandboxes can be executed in parallel. Individual sandbox providers can establish their own default limits (for example, the Docker provider has a default of `2 * os.cpu_count()`). You can modify this option as required, but be aware that container runtimes have resource limits, and pushing up against and beyond them can lead to instability and failed evaluations.
-Since sandbox environments include additional expensive operations beyond calling models, you may want to increase `max_samples` to fully saturate both the Model API and container subprocesses used for tool execution. When running an evaluation you'll see an indicator of how many connections and how many subprocesses are currently active. If neither is at capacity then you will likely benefit from increasing `max_samples`.
+When a `max_sandboxes` is applied, an indicator at the bottom of the task status screen will be shown:
-Note that setting `max_samples` to an arbitrarily high number does have some disadvantages: you will consume more memory (especially if using sandbox environments) as well as wait longer for completed samples to be logged (so could be subject to losing more work if your eval task fails).
+![](images/task-max-sandboxes.png)
+Note that when `max_sandboxes` is applied this effectively creates a global `max_samples` limit that is equal to the `max_sandboxes`.
 ### Max Subprocesses

inspect_ai-0.3.54/docs/scripts/post-render.sh ADDED Viewed

@@ -0,0 +1,23 @@
+#!/bin/bash
+files=("index" "tutorial" "workflow" "log-viewer" "vscode" "solvers" "tools" "agents" "agents-api" "scorers" "datasets" "models" "eval-sets" "errors-and-limits" "caching" "parallelism" "interactivity" "approval" "eval-logs" "extensions")
+if [ "$QUARTO_PROJECT_RENDER_ALL" = "1" ]; then
+    llms_full="_site/llms-full.txt"
+    rm -f "${llms_full}"
+    mv _quarto.yml _quarto.yml.bak
+    for file in "${files[@]}"; do
+        echo "llms: ${file}.qmd"
+        quarto render "${file}.qmd" --to gfm --quiet --no-execute
+        output_file="${file}.md"
+        cat "${output_file}" >> "${llms_full}"
+        echo "" >> "${llms_full}"
+        mv $output_file "_site/${file}.html.md"
+    done
+    mv _quarto.yml.bak _quarto.yml
+fi

{inspect_ai-0.3.52 → inspect_ai-0.3.54}/docs/tools.qmd RENAMED Viewed

@@ -20,7 +20,7 @@ Inspect has several built-in tools, including:
 -   [Bash and Python](#sec-bash-and-python) for executing arbitrary shell and Python code.
--   [Web Browser](#sec-web-browser), which provides the model with a headless Chromimum web browser that supports navigation, history, and mouse/keyboard interactions.
+-   [Web Browser](#sec-web-browser), which provides the model with a headless Chromium web browser that supports navigation, history, and mouse/keyboard interactions.
 -   [Web Search](#sec-web-search), which uses the Google Search API to execute and summarise web searches.
@@ -100,11 +100,11 @@ If you do not explicitly handle errors, then Inspect provides some default error
 -   `UnicodeDecodeError` — Occurs when the output from executing a process or reading a file is binary rather than text.
--   `OutputLimitExceededError` - Occurs when one or both of the output streams from `sandbox().exec()` exceed 1 MiB or when attempting to read a file over 100 MiB in size.
+-   `OutputLimitExceededError` - Occurs when one or both of the output streams from `sandbox().exec()` exceed 10 MiB or when attempting to read a file over 100 MiB in size.
 -   `ToolError` — Special error thrown by tools to indicate they'd like to report an error to the model.
-These are all errors that are *expected* (in fact the `SandboxEnvironemnt` interface documents them as such) and possibly recoverable by the model (try a different command, read a different file, etc.). Unexpected errors (e.g. a network error communicating with a remote service or container runtime) on the other hand are not automatically handled and result in the `Sample` failing with an error.
+These are all errors that are *expected* (in fact the `SandboxEnvironment` interface documents them as such) and possibly recoverable by the model (try a different command, read a different file, etc.). Unexpected errors (e.g. a network error communicating with a remote service or container runtime) on the other hand are not automatically handled and result in the `Sample` failing with an error.
 Many tools can simply rely on the default handling to provide reasonable behaviour around both expected and unexpected errors.
@@ -120,7 +120,7 @@ Another type of explicit handling is re-raising an error to bypass Inspect's def
 ``` python
 try:
-  result = await sandobox().exec(
+  result = await sandbox().exec(
     cmd=["decode", file],
     timeout=timeout
   )
@@ -314,7 +314,7 @@ See the [Agents](#sec-agents) section for more details on how to build evaluatio
 ## Web Browser {#sec-web-browser}
-The web browser tools provids models with the ability to browse the web using a headless Chromium browser. Navigation, history, and mouse/keyboard interactions are all supported.
+The web browser tools provides models with the ability to browse the web using a headless Chromium browser. Navigation, history, and mouse/keyboard interactions are all supported.
 ### Configuration
@@ -408,7 +408,7 @@ RUN pip install playwright
 RUN playwright install
 RUN playwright install-deps
-# Install other dependancies
+# Install other dependencies
 RUN pip install dm-env-rpc pillow bs4 lxml
 # Copy Python files alongside the Dockerfile

{inspect_ai-0.3.52 → inspect_ai-0.3.54}/docs/tutorial.qmd RENAMED Viewed

@@ -78,7 +78,7 @@ inspect eval security_guide.py
 [HellaSwag](https://rowanzellers.com/hellaswag/) is a dataset designed to test commonsense natural language inference (NLI) about physical situations. It includes samples that are adversarially constructed to violate common sense about the physical world, so can be a challenge for some language models.
-For example, here is one of the questions in the dataset along with its set of possible answer (the correct answer is C):
+For example, here is one of the questions in the dataset along with its set of possible answers (the correct answer is C):
 > In home pet groomers demonstrate how to groom a pet. the person
 >
@@ -570,4 +570,4 @@ def ctf_agent(max_attempts=3, message_limit=30):
 The `basic_agent()` provides a ReAct tool loop with support for retries and encouraging the model to continue if its gives up or gets stuck. The `bash()` and `python()` tools are provided to the model with a 3-minute timeout to prevent long running commands from getting the evaluation stuck.
-See the [full source code](https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/gdm_capabilities/intercode_ctf) of the Intercode CTF example to explore the dataset and evaluation code in more depth.
+See the [full source code](https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/gdm_capabilities/intercode_ctf) of the Intercode CTF example to explore the dataset and evaluation code in more depth.

{inspect_ai-0.3.52 → inspect_ai-0.3.54}/pyproject.toml RENAMED Viewed

@@ -132,7 +132,6 @@ dev = [
     "ruff==0.8.3", # match version specified in .pre-commit-config.yaml
     "textual-dev>=0.86.2",
     "types-PyYAML",
-    "types-aiofiles",
     "types-beautifulsoup4",
     "types-aioboto3",
     "types-boto3",

{inspect_ai-0.3.52 → inspect_ai-0.3.54}/requirements.txt RENAMED Viewed

@@ -1,4 +1,3 @@
-aiofiles
 aiohttp>=3.9.0
 anyio>=4.4.0 # not directly required, pinned by Snyk to avoid a vulnerability
 beautifulsoup4

inspect-ai 0.3.52__tar.gz → 0.3.54__tar.gz

inspect-ai 0.3.52tar.gz → 0.3.54tar.gz