inspect-ai 0.3.55__tar.gz → 0.3.56__tar.gz

{inspect_ai-0.3.55 → inspect_ai-0.3.56}/CHANGELOG.md

@@ -1,5 +1,18 @@
 # Changelog
 
+## v0.3.56 (01 January 2025)
+
+- [Human Agent](https://inspect.ai-safety-institute.org.uk/human-agent.html) solver for human baselining of computing tasks.
+- [Typed interfaces](https://inspect.ai-safety-institute.org.uk/typing.html) to `Sample` store and metadata using Pydantic models.
+- [Approval policies](https://inspect.ai-safety-institute.org.uk/approval.html#task-approvers) can now be defined at the `Task` level (`eval` level approval policies take precedence).
+- Tools can now return `ContentText` and `ContentImage`.
+- Move tool result images into subsequent user messages for models that don't support tools returning images.
+- `SandboxConnection` that contains login information from sandboxes.
+- `display_type()` function for detecting the current display type (e.g. "full", "rich", etc.)
+- Trace: improved handling of `eval()` running in multiple processes at once (trace file per-process)
+- Docker: don't apply timeouts to `docker build` and `docker pull` commands.
+- Bugfix: fix issue w/ `store.get()` not auto-inserting `default` value.
+
 ## v0.3.55 (29 December 2024)
 
 - Bedrock: redact authentication model args from eval logs.

{inspect_ai-0.3.55/src/inspect_ai.egg-info → inspect_ai-0.3.56}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: inspect_ai
-Version: 0.3.55
+Version: 0.3.56
 Summary: Framework for large language model evaluations
 Author: UK AI Safety Institute
 License: MIT License

inspect_ai-0.3.56/docs/_metadata_typing.md

@@ -0,0 +1,34 @@
+
+
+If you want a more strongly typed interface to sample metadata, you can define a [Pydantic model](https://docs.pydantic.dev/latest/concepts/models/) and use it to both validate and read metadata.
+
+For validation, pass a `BaseModel` derived class in the `FieldSpec`. The interface to metadata is read-only so you must also specify `frozen=True`. For example:
+
+```python
+from pydantic import BaseModel
+
+class PopularityMetadata(BaseModel, frozen=True):
+    category: str
+    label_confidence: float
+
+dataset = json_dataset(
+    "popularity.jsonl",
+    FieldSpec(
+        input="question",
+        target="answer_matching_behavior",
+        id="question_id",
+        metadata=PopularityMetadata,
+    ),
+)
+```
+
+To read metadata in a typesafe fashion, us the `metadata_as()` method on `Sample` or `TaskState`:
+
+```python
+metadata = state.metadata_as(PopularityMetadata)
+```
+
+Note again that the intended semantics of `metadata` are read-only, so attempting to write into the returned metadata will raise a Pydantic `FrozenInstanceError`. 
+
+If you need per-sample mutable data, use the [sample store](agents-api.qmd#sample-store), which also supports [typing](agents-api.qmd#store-typing) using Pydantic models. 
+

{inspect_ai-0.3.55 → inspect_ai-0.3.56}/docs/_quarto.yml

@@ -55,6 +55,7 @@ website:
             - agents.qmd
             - sandboxing.qmd
             - agents-api.qmd
+            - human-agent.qmd
             - approval.qmd
 
         - section: "Advanced"
@@ -63,6 +64,7 @@ website:
             - eval-sets.qmd
             - text: "Errors & Limits"
               href: errors-and-limits.qmd
+            - typing.qmd
             - tracing.qmd
             - caching.qmd
             - parallelism.qmd

{inspect_ai-0.3.55 → inspect_ai-0.3.56}/docs/_sandboxenv-interface.md

@@ -51,8 +51,17 @@ class SandboxEnvironment:
             exceeds the 100 MiB limit.
         """
         ...
+
+    async def connection(self) -> SandboxConnection:
+        """
+        Raises:
+           NotImplementedError: For sandboxes that don't provide connections
+           ConnectionError: If sandbox is not currently running.
+        """
 ```
 
-Note that `write_file()` automatically creates parent directories as required if they don't exist.
+The `read_file()` method should should preserve newline constructs (e.g. crlf should be preserved not converted to lf). This is equivalent to specifying `newline=""` in a call to the Python `open()` function. Note that `write_file()` automatically creates parent directories as required if they don't exist.
+
+The `connection()` method is optional, and provides commands that can be used to login to the sandbox container from a terminal or IDE.
 
 For each method there is a documented set of errors that are raised: these are _expected_ errors and can either be caught by tools or allowed to propagate in which case they will be reported to the model for potential recovery. In addition, _unexpected_ errors may occur (e.g. a networking error connecting to a remote container): these errors are not reported to the model and fail the `Sample` with an error state. 

inspect_ai-0.3.56/docs/_store_typing.md

@@ -0,0 +1,52 @@
+
+If you prefer a typesafe interface to the sample store, you can define a [Pydantic model](https://docs.pydantic.dev/latest/concepts/models/) which reads and writes values into the store. There are several benefits to using Pydantic models for store access:
+
+1. You can provide type annotations and validation rules for all fields.
+2. Default values for all fields are declared using standard Pydantic syntax.
+3. Store names are automatically namespaced (to prevent conflicts between multiple store accessors).
+
+#### Definition
+
+First, derive a class from `StoreModel` (which in turn derives from Pydantic `BaseModel`):
+
+```python
+from pydantic import Field
+from inspect_ai.util import StoreModel
+
+class Activity(StoreModel):
+    active: bool = Field(default=False)
+    tries: int = Field(default=0)
+    actions: list[str] = Field(default_factory=list)
+```
+
+Note that we define defaults for all fields. This is generally required so that you can initialise your Pydantic model from an empty store. For collections (`list` and `dict`) you should use `default_factory` so that each instance gets its own default.
+
+#### Usage
+
+Use the `store_as()` function to get a typesafe interface to the store based on your model:
+
+```python
+# typed interface to store from state
+activity = state.store_as(Activity)
+activity.active = True
+activity.tries += 1
+
+# global store_as() function (e.g. for use from tools)
+from inspect_ai.util import store_as
+activity = store_as(Activity)
+```
+
+Note that all instances of `Activity` created within a running sample share the same sample `Store` so can see each other's changes. For example, you can call `state.store_as()` in multiple solvers and/or scorers and it will resolve to the same sample-scoped instance. 
+
+The names used in the underlying `Store` are namespaced to prevent collisions with other `Store` accessors. For example, the `active` field in the `Activity` class is written to the store with the name `Activity:active`.
+
+#### Explicit Store
+
+The `store_as()` function automatically binds to the current sample `Store`. You can alternatively create an explicit `Store` and pass it directly to the model (e.g. for testing purposes):
+
+```python
+from inspect_ai.util import Store
+store = Store()
+activity = Activity(store=store)
+```
+

{inspect_ai-0.3.55 → inspect_ai-0.3.56}/docs/agents-api.qmd

@@ -6,7 +6,7 @@ title: Agents API
 
 This article describes advanced Inspect APIs available for creating evaluations with agents. You can also build agents evals using Inspect's built in [Basic Agent](agents.qmd#sec-basic-agent) or by bridging to an external agent library (see the main [Agents](agents.qmd) article for further details). Topics covered in this article include:
 
-1.  Sharing state across solvers and tools
+1.  Sharing per-sample state across solvers and tools
 2.  Creating a custom tool use loop
 3.  Dynamically customising tool descriptions
 4.  Observability with sample transcripts.
@@ -15,19 +15,10 @@ This article describes advanced Inspect APIs available for creating evaluations
 
 We'll assume that you have already covered the basics of [Solvers](solvers.qmd), [Tools](tools.qmd), and [Agents](agents.qmd) (please review those articles as required before proceeding).
 
-## Use of `metadata`
 
-Before proceeding, it's important to point that some of the features described below were previously approximated by using the `metadata` field of `TaskState`, specifically `metadata` was often used as a catch-all storage location for:
+## Sample Store
 
--   Sharing state between solvers.
--   Providing a place to log additional structured data.
--   Recording calls to "helper" models used for elicitation or scoring.
-
-The `metadata` field no longer need be used for these scenarios (and in fact should now be treated as a read-only part of the `TaskState`). Below we'll describe how the `Store` can be used for state, how structured data can be logged to the sample `Transcript`, and how all model calls are now automatically recorded and included in the transcript.
-
-## Sharing State
-
-Sequences of solvers often need to store and manipulate shared state. Further, tools may often want their own persistent state (or groups of tools may want to share state). This can be accomplished in Inspect using the `Store`, which provides a scoped scratchpad for arbitrary values.
+Sequences of solvers executing against a sample often need to store and manipulate shared state. Further, tools may often want their own persistent state (or groups of tools may want to share state). This can be accomplished in Inspect using the `Store`, which provides a sample-scoped scratchpad for arbitrary values.
 
 The core of the `Store` interface is:
 
@@ -40,6 +31,8 @@ class Store:
     def delete(self, key: str) -> None
 ```
 
+Note that the core `Store` interface is a property bag without strong typing. See the section below on [typed store access](#store-typing) for details on how to interact with the store in a typesafe fashion.
+
 Basic views on the store's collection (e.g. `items()`, `keys()`, `values()`) are also provided. Note that the `get()` method will automatically add the `default` to the store if it doesn't exist.
 
 The `Store` can be accessed via `TaskState` as follows:
@@ -67,6 +60,11 @@ You should generally try to use JSON serialisable Python types in the `Store` (e
 
 While the default `Store` for a sample is shared globally between solvers and tools, a more narrowly scoped `Store` is created automatically for [Subtasks](#sec-subtasks).
 
+### Store Typing {#store-typing}
+
+{{< include _store_typing.md >}}
+
+
 ## Tool Use
 
 ### Custom Loop

{inspect_ai-0.3.55 → inspect_ai-0.3.56}/docs/agents.qmd

@@ -16,6 +16,8 @@ Inspect supports a variety of approaches to agent evaluations, including:
 
 3.  Adapting an agent provided by a research paper or open source library (for example, using a 3rd party agent library like [LangChain](https://python.langchain.com/docs/modules/agents/) or [Langroid](https://langroid.github.io/langroid/)).
 
+4.  A [Human Agent](human-agent.qmd) for creating human baselines on computing tasks.
+
 An important additional consideration for agent evaluations is sandboxing (providing a secure environment for models to execute code within). The [Sandboxing](sandboxing.qmd) article goes into more depth on this.
 
 ## Basic Agent {#sec-basic-agent}
@@ -266,3 +268,15 @@ def wikipedia() -> Task:
 ```
 
 The full source code for this example can be found in the Inspect GitHub repo at [examples/langchain](https://github.com/UKGovernmentBEIS/inspect_ai/tree/main/examples/langchain).
+
+## Learning More
+
+See these additioanl articles to learn more about creating agent evaluations with Inspect:
+
+-   [Sandboxing](sandboxing.qmd) enables you to isolate code generated by models as well as set up more complex computing environments for tasks. 
+
+-   [Agents API](agents-api.qmd) describes advanced Inspect APIs available for creating evaluations with agents.
+
+-   [Human Agent](human-agent.qmd) is a solver that enables human baselining on computing tasks.
+
+-   [Approval](approval.qmd) enable you to create fine-grained policies for approving tool calls made by model agents.

{inspect_ai-0.3.55 → inspect_ai-0.3.56}/docs/approval.qmd

@@ -12,6 +12,8 @@ Inspect's approval mode enables you to create fine-grained policies for approvin
 
 Custom approvers are very flexible, and can implement a wide variety of decision schemes including informal heuristics and assessments by models. They could also support human approval with a custom user interface on a remote system (whereby approvals are sent and received via message queues).
 
+Approvers can be specified at either the eval level or at the task level. The examples below will demonstrate eval-level approvers, see the [Task Approvers](#task-approvers) section for details on task-level approvers.
+
 ## Human Approver
 
 The simplest approval policy is interactive human approval of all tool calls. You can enable this policy by using the `--approval human` CLI option (or the `approval = "human"`) argument to `eval()`:
@@ -61,6 +63,33 @@ approval = [
 eval("browser.py", approval=approval, trace=True)
 ```
 
+## Task Approvers {#task-approvers}
+
+You can specify approval policies at the task level using the `approval` parameter when creating a `Task`. For example:
+
+```python
+from inspect_ai import Task, task
+from inspect_ai.scorer import match
+from inspect_ai.solver import generate, use_tools
+from inspect_ai.tool import bash, python
+from inspect_ai.approval import human_approver
+
+@task
+def linux_task():
+    return Task(
+        dataset=read_dataset(),
+        solver=[
+            use_tools([bash(), python()]),
+            generate(),
+        ],
+        scorer=match(),
+        sandbox=("docker", "compose.yaml"),
+        approval=human_approver()
+    )
+```
+
+Note that as with all of the other `Task` options, an `approval` policy defined at the eval-level will override a task-level approval policy.
+
 ## Custom Approvers
 
 Inspect includes two built-an approvers: `human` for interactive approval at the terminal and `auto` for automatically approving or rejecting specific tools. You can also create your own approvers that implement just about any scheme you can imagine.

{inspect_ai-0.3.55 → inspect_ai-0.3.56}/docs/datasets.qmd

@@ -102,6 +102,10 @@ def record_to_sample(record):
 dataset = json_dataset("popularity.jsonl", record_to_sample)
 ```
 
+### Typed Metadata
+
+{{< include _metadata_typing.md >}}
+
 ## Filter and Shuffle
 
 The `Dataset` class includes `filter()` and `shuffle()` methods, as well as support for the slice operator.

inspect_ai-0.3.56/docs/human-agent.qmd

@@ -0,0 +1,166 @@
+---
+title: Human Agent
+---
+
+## Overview
+
+The Inspect human agent enables human baselining of agentic tasks that run in a Linux environment. Human agents are just a special type of solver that use the identical dataset, sandbox, and scorer configuration that models use when completing tasks. However, rather than entering an agent loop, the `human_agent` solver provides the human baseliner with:
+
+1.  A description of the task to be completed (input/prompt from the sample).
+
+2.  Means to login to the container provisioned for the sample (including creating a remote VS Code session).
+
+3.  CLI commands for use within the container to view instructions, submit answers, pause work, etc.
+
+Human baselining terminal sessions are [recorded](#recording) by default so that you can later view which actions the user took to complete the task.
+
+## Example
+
+Here, we run a human baseline on an [Intercode CTF](https://ukgovernmentbeis.github.io/inspect_evals/evals/cybersecurity/intercode_ctf/) sample by using the `--solver` option to use `human_agent` rather than the task's default solver:
+
+``` bash
+inspect eval inspect_evals/gdm_intercode_ctf \
+    --sample-id 44 --solver human_agent
+```
+
+The evaluation runs as normal, and a **Human Agent** panel appears in the task UI to orient the human baseliner to the task and provide instructions for accessing the container. The user clicks the **VS Code Terminal** link and a terminal interface to the container is provided within VS Code:
+
+![](images/inspect-human-agent.png){.lightbox}
+
+Note that while this example makes use of VS Code, it is in no way required. Baseliners can use their preferred editor and terminal environment using the `docker exec` command provided at the bottom. Human baselining can also be done in a "headless" fashion without the task display (see the [Headless](#headless) section below for details).
+
+Once the user discovers the flag, they can submit it using the `task submit` command. For example:
+
+``` bash
+task submit picoCTF{73bfc85c1ba7}
+```
+
+## Usage
+
+Using the `human_agent` solver is as straightforward as specifying the `--solver` option for any existing task. Repeating the example above:
+
+``` bash
+inspect eval inspect_evals/gdm_intercode_ctf \
+    --sample-id 44 --solver human_agent
+```
+
+Or alternatively from within Python:
+
+``` python
+from inspect_ai import eval
+from inspect_ai.solver import human_agent
+from inspect_evals import gdm_intercode_ctf
+
+eval(gdm_intercode_ctf(), sample_id=44, solver=human_agent())
+```
+
+There are however some requirements that should be met by your task before using it with the human agent:
+
+1.  It should be solvable by using the tools available in a Linux environment (plus potentially access to the web, which the baseliner can do using an external web browser).
+
+2.  The dataset `input` must fully specify the instructions for the task. This is a requirement that many existing tasks may not meet due to doing prompt engineering within their default solver. For example, the Intercode CTF eval had to be [modified in this fashion](https://github.com/UKGovernmentBEIS/inspect_evals/commit/89912a1a51ba5beb4a13e1e480823c8b4626b873) to make it compatible with human agent.
+
+### Container Access
+
+The human agent works on the task within the default sandbox container for the task. Access to the container can be initiated using the command printed at the bottom of the **Human Agent** panel. For example:
+
+``` bash
+docker exec -it inspect-gdm_intercod-itmzq4e-default-1 bash -l
+```
+
+Alternatively, if the human agent is working within VS Code then two links are provided to access the container within VS Code:
+
+-   **VS Code Window** opens a new VS Code window logged in to the container. The human agent can than create terminals, browse the file system, etc. using the VS Code interface.
+
+-   **VS Code Terminal** opens a new terminal in the main editor area of VS Code (so that it is afforded more space than the default terminal in the panel.
+
+### Task Commands
+
+The Human agent solver installs agent task tools in the default sandbox and presents the user with both task instructions and documentation for the various tools (e.g. `task submit`, `task start`, `task stop`, `task instructions`, etc.). By default, the following command are available:
+
+| Command             | Description                               |
+|---------------------|-------------------------------------------|
+| `task submit`       | Submit your final answer for the task.    |
+| `task note`         | Record a note in the task transcript.     |
+| `task status`       | Print task status (clock, scoring , etc.) |
+| `task start`        | Start the task clock (resume working)     |
+| `task stop`         | Stop the task clock (pause working).      |
+| `task instructions` | Display task command and instructions.    |
+
+: {tbl-colwidths=\[40,60\]}
+
+Note that the instructions are also copied to an `instructions.txt` file in the container user's working directory.
+
+### Answer Submission
+
+When the human agent has completed the task, they submit their answer using the `task submit`command. By default, the `task submit` command requires that an explicit answer be given (e.g. `task submit picoCTF{73bfc85c1ba7}`).
+
+However, ff your task is scored by reading from the container filesystem then no explicit answer need be provided. Indicate this by passing `answer=False` to the `human_agent()`:
+
+``` python
+solver=human_agent(answer=False)
+```
+
+Or from the CLI, use the `-S` option:
+
+``` bash
+--solver human_agent -S answer=false
+```
+
+You can also specify a regex to match the answer against for validation, for example:
+
+``` python
+solver=human_agent(answer=r"picoCTF{\w+}")
+```
+
+### Intermediate Scoring
+
+You can optionally make intermediate scoring available to human baseliners so that they can check potential answers as they work. Use the `intermediate_scoring` option (which defaults to `False`) to do this:
+
+``` python
+solver=human_agent(intermediate_scoring=True)
+```
+
+Or from the CLI, use the `-S` option:
+
+``` bash
+--solver human_agent -S intermediate_scoring=true
+```
+
+With this option enabled, the human agent can check their potential score on the task for a given answer using the `task score` command. For example:
+
+``` bash
+task score picoCTF{73bfc85c1ba7}
+```
+
+## Recording {#recording}
+
+By default, human agent terminal sessions are recorded using the LInux [script command](https://www.redhat.com/en/blog/linux-script-command). Recorded sessions are saved in the sample store and available for playback within the Inspect View:
+
+![](images/inspect-terminal-transcript.png){.lightbox}
+
+
+You can disable session recording with the `record_session` option:
+
+``` bash
+--solver human_agent -S record_session=false
+```
+
+## Headless {#headless}
+
+The examples above demonstrate human baselining being initiated from the CLI and standard task display. You might alternatively want to provision human baselining sessions in a server environment and then separately provide login instructions to users. To suppress the standard task display in favour of just printing out the container login information, pass the `--display=plain` CLI option:
+
+``` bash
+inspect eval inspect_evals/gdm_intercode_ctf \
+    --sample-id 44 --solver human_agent --display plain
+```
+
+Which will print the following to the terminal:
+
+``` {.bash .code-overflow-wrap}
+Running task gdm_intercode_ctf (1 sample): anthropic/claude-3-5-sonnet-20240620 (config: sample_id: 44)...
+
+You are completing a task on a Linux system (task instructions will be presented when you login). Login to the system with the following command:
+
+docker exec -it inspect-gdm_intercod-iebwzkg-default-1 bash -l
+```

{inspect_ai-0.3.55 → inspect_ai-0.3.56}/docs/index.qmd

@@ -276,9 +276,9 @@ These sections describe how to create agent evaluations with Inspect:
 
 -   [Agents API](agents-api.qmd) describes advanced Inspect APIs available for creating evaluations with agents.
 
--   [Approval](approval.qmd) enable you to create fine-grained policies for approving tool calls made by model agents.
+-   [Human Agent](human-agent.qmd) is a solver that enables human baselining on computing tasks.
 
--   [Interactivity](interactivity.qmd) covers various ways to introduce user interaction into the implementation of tasks (for example, prompting the model dynamically based on the trajectory of the evaluation).
+-   [Approval](approval.qmd) enable you to create fine-grained policies for approving tool calls made by model agents.
 
 These sections discuss more advanced features and workflow. You don't need to review them at the outset, but be sure to revisit them as you get more comfortable with the basics.
 
@@ -288,10 +288,14 @@ These sections discuss more advanced features and workflow. You don't need to re
 
 -   [Errors and Limits](errors-and-limits.qmd) covers various techniques for dealing with unexpected errors and setting limits on evaluation tasks and samples.
 
+-   [Typing](typing.qmd): provides guideance on using static type checking with Inspect, including creating typed interfaces to untyped storage (i.e. sample metadata and store).
+
 -   [Tracing](tracing.qmd) Describes advanced execution tracing tools used to diagnose runtime issues.
 
 -   [Caching](caching.qmd) enables you to cache model output to reduce the number of API calls made, saving both time and expense.
 
 -   [Parallelism](parallelism.qmd) delves into how to obtain maximum performance for evaluations. Inspect uses a highly parallel async architecture---here we cover how to tune this parallelism (e.g to stay under API rate limits or to not overburden local compute) for optimal throughput.
 
+-   [Interactivity](interactivity.qmd) covers various ways to introduce user interaction into the implementation of tasks (for example, prompting the model dynamically based on the trajectory of the evaluation).
+
 -   [Extensions](extensions.qmd) describes the various ways you can extend Inspect, including adding support for new Model APIs, tool execution environments, and storage platforms (for datasets, prompts, and logs).

{inspect_ai-0.3.55 → inspect_ai-0.3.56}/docs/llms.txt

@@ -16,10 +16,12 @@
 - [Agents](https://inspect.ai-safety-institute.org.uk/agents.html.md): Agents combine planning, memory, and tool usage to pursue more complex, longer horizon tasks. This section describes how to build agent evaluations with Inspect.
 - [Sandboxing](https://inspect.ai-safety-institute.org.uk/sandboxing.html.md): Enables you to isolate code generated by models as well as set up more complex computing environments for tasks. 
 - [Agents API](https://inspect.ai-safety-institute.org.uk/agents-api.html.md): This article describes advanced Inspect APIs available for creating evaluations with agents.
+- [Human Agent](https://inspect.ai-safety-institute.org.uk/human-agent.html.md): This article describes the `human_agent()` solver which enables human baselining for computing tasks.
 - [Approval](https://inspect.ai-safety-institute.org.uk/approval.html.md): Approvals enable you to create fine-grained policies for approving tool calls made by models.
 - [Eval Logs](https://inspect.ai-safety-institute.org.uk/eval-logs.html.md): Explores how to get the most out of evaluation logs for developing, debugging, and analyzing evaluations.
 - [Eval Sets](https://inspect.ai-safety-institute.org.uk/eval-sets.html.md): Covers Inspect's features for describing, running, and analysing larger sets of evaluation tasks.
 - [Errors and Limits](https://inspect.ai-safety-institute.org.uk/errors-and-limits.html.md): This article covers various techniques for dealing with unexpected errors and setting limits on evaluation tasks and samples, including retrying failed evaluations, establishing a threshold of samples to tolerate errors for before failing an evaluation, and setting a maximum number of messages, tokens, or elapsed seconds in a sample before forcing the solver to give up.
+- [Typing](https://inspect.ai-safety-institute.org.uk/typing.html.md): Provides guideance on using static type checking with Inspect, including creating typed interfaces to untyped storage (i.e. sample metadata and store).
 - [Tracing](https://inspect.ai-safety-institute.org.uk/tracing.html.md): Describes advanced execution tracing tools used to diagnose runtime issues.
 - [Caching](https://inspect.ai-safety-institute.org.uk/caching.html.md): Caching enables you to cache model output to reduce the number of API calls made, saving both time and expense.
 - [Parallelism](https://inspect.ai-safety-institute.org.uk/parallelism.html.md): Delves into how to obtain maximum performance for evaluations. Inspect uses a highly parallel async architecture---here we cover how to tune this parallelism (e.g to stay under API rate limits or to not overburden local compute) for optimal throughput.

{inspect_ai-0.3.55 → inspect_ai-0.3.56}/docs/scripts/post-render.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-files=("index" "tutorial" "workflow" "log-viewer" "vscode" "solvers" "tools" "agents" "agents-api" "scorers" "datasets" "models" "eval-sets" "errors-and-limits" "caching" "parallelism" "interactivity" "approval" "eval-logs" "extensions")
+files=("index" "tutorial" "workflow" "log-viewer" "vscode" "solvers" "tools" "scorers" "datasets" "models" "agents" "sandboxing" "agents-api" "human-agent" "approval" "eval-logs" "eval-sets" "typing" "errors-and-limits" "tracing" "caching" "parallelism" "interactivity" "extensions")
 
 
 if [ "$QUARTO_PROJECT_RENDER_ALL" = "1" ]; then

{inspect_ai-0.3.55 → inspect_ai-0.3.56}/docs/tools.qmd

@@ -277,7 +277,7 @@ def web_browser_go() -> Tool:
 
 Specifying `parallel=False` results in two behaviours:
 
-1.  Models that support turing off parallel tool calling (currently OpenAI and Grok) will have it disabled when tools with `parallel=False` are passed to `generate()`.
+1.  Models that support turning off parallel tool calling (currently OpenAI and Grok) will have it disabled when tools with `parallel=False` are passed to `generate()`.
 
 2.  Inspect will execute tool calls serially (so that even for models that don't let you disable parallel tool calling, you can still be assured they will not ever run in parallel).
 
@@ -464,4 +464,4 @@ The `web_search()` tool uses [Google Programmable Search Engine](https://program
 
 -   `GOOGLE_CSE_ID` — Google Custom Search Engine ID
 
--   `GOOGLE_CSE_API_KEY` — Google API key used to enable the Search API
+-   `GOOGLE_CSE_API_KEY` — Google API key used to enable the Search API

{inspect_ai-0.3.55 → inspect_ai-0.3.56}/docs/tracing.qmd

@@ -25,10 +25,10 @@ Trace logging does not need to be explicitly enabled—logs for the last 10 top
 inspect trace list # --json for JSON output
 ```
 
-Trace logs are written using [JSON Lines](https://jsonlines.org/) format and all but the most recent one are gzip compressed, so reading them requires some special handing. The `inspect trace dump` command encapsulates this and gives you a normal JSON array with the contents of the trace log:
+Trace logs are written using [JSON Lines](https://jsonlines.org/) format and are gzip compressed, so reading them requires some special handing. The `inspect trace dump` command encapsulates this and gives you a normal JSON array with the contents of the trace log (note that trace log filenames include the ID of the process that created them):
 
 ``` bash
-inspect trace dump trace.log
+inspect trace dump trace-86396.log.gz
 ```
 
 ## Anomalies
@@ -44,7 +44,7 @@ You will first see currently running actions (useful mostly for a "live" evaluat
 Passing no arguments shows the most recent trace log, pass a log file name to view another log:
 
 ``` bash
-inspect trace anomalies trace.log.1.gz
+inspect trace anomalies trace-86396.log.gz
 ```
 
 ### Errors and Timeouts

inspect_ai-0.3.56/docs/typing.qmd

@@ -0,0 +1,18 @@
+---
+title: Typing
+---
+
+## Overview
+
+The Inspect codebase is written using strict [MyPy](https://mypy-lang.org/) type-checking---if you enable the same for your project along with installing the [MyPy VS Code Extension](https://marketplace.visualstudio.com/items?itemName=ms-python.mypy-type-checker) you'll benefit from all of these type definitions.
+
+The sample store and sample metadata interfaces are weakly typed to accommodate arbitrary user data structures. Below, we describe how to implement a [typed store](#typed-store) and [typed metadata](#typed-metadata) using Pydantic models.
+
+## Typed Store
+
+{{< include _store_typing.md >}}
+
+## Typed Metadata
+
+{{< include _metadata_typing.md >}}
+

{inspect_ai-0.3.55 → inspect_ai-0.3.56}/src/inspect_ai/__init__.py

@@ -9,6 +9,7 @@ from inspect_ai._eval.registry import task
 from inspect_ai._eval.score import score, score_async
 from inspect_ai._eval.task import Epochs, Task, TaskInfo, Tasks
 from inspect_ai._util.constants import PKG_NAME
+from inspect_ai.solver._human_agent.agent import human_agent
 
 __version__ = importlib_version(PKG_NAME)
 

{inspect_ai-0.3.55 → inspect_ai-0.3.56}/src/inspect_ai/_cli/common.py

@@ -10,7 +10,7 @@ from inspect_ai._util.constants import (
     DEFAULT_LOG_LEVEL,
     DEFAULT_LOG_LEVEL_TRANSCRIPT,
 )
-from inspect_ai._util.display import init_display_type
+from inspect_ai.util._display import init_display_type
 
 
 class CommonOptions(TypedDict):