inspect-ai 0.3.97__tar.gz → 0.3.99__tar.gz

{inspect_ai-0.3.97 → inspect_ai-0.3.99}/.github/workflows/log_viewer.yml

@@ -77,6 +77,47 @@ jobs:
       - name: Run tests
         run: yarn test
 
+  check-schema-and-types:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python 3.11
+        uses: actions/setup-python@v4
+        with:
+          python-version: 3.11
+      - name: Cache pip dependencies
+        uses: actions/cache@v3
+        with:
+          path: ~/.cache/pip
+          key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt', '**/pyproject.toml') }}-${{ matrix.python-version }}
+          restore-keys: |
+            ${{ runner.os }}-pip-
+      - name: Install pip dependencies
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install .[dev]
+
+      - name: Set up Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: "22.x"
+      - name: Install Node.js dependencies
+        run: yarn install --frozen-lockfile
+
+      - name: Update schema and types
+        run: python src/inspect_ai/_view/schema.py
+
+      - name: Ensure schema and types changes are checked in
+        run: |
+          if [[ $(git status --porcelain) != "" ]]
+          then
+            echo "Log format JSON Schema and TypeScript types are out of date. Please run python src/inspect_ai/_view/schema.py and check in the changes."
+            git status
+            git diff
+            exit 1
+          fi
+
   # TODO: This is failing even with a freshly generated build.js file
   # Need to debug or better understand the cause
   # build:

{inspect_ai-0.3.97 → inspect_ai-0.3.99}/CHANGELOG.md

@@ -1,3 +1,19 @@
+## v0.3.99 (22 May 2025)
+
+- Exported `view()` function for running Inspect View from Python.
+- Always return tasks in the same order they were passed to `eval()` or `eval_set()`.
+- Google: Updated required version of `google-genai` to 1.16.1 (which includes support for reasoning summaries and is now compatible with the trio async backend).
+- Anthropic: More flexible detection of "overloaded_error" for retires.
+- Inspect View: Improve text zooming and wrapping when rendering sample errors.
+- Inspect View: Preserve log mtime-ordering in the bundle output directory
+
+## v0.3.98 (18 May 2025)
+
+- Google: Disable reasoning when `reasoning_tokens` is set to 0.
+- Temporarily pin to textual < 3.0.0 to work around event loop breakage.
+- CLI display: improve performance of sample rendering by only rendering the 10 most recent events.
+- Inspect View: Improve sample score column layout, markdown render explanation.
+
 ## v0.3.97 (16 May 2025)
 
 - React agent: Use of `submit()` tool is now [optional](https://inspect.aisi.org.uk/agent.html#submit-tool).

{inspect_ai-0.3.97 → inspect_ai-0.3.99}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: inspect_ai
-Version: 0.3.97
+Version: 0.3.99
 Summary: Framework for large language model evaluations
 Author: UK AI Security Institute
 License: MIT License
@@ -49,7 +49,7 @@ Requires-Dist: semver>=3.0.0
 Requires-Dist: shortuuid
 Requires-Dist: sniffio
 Requires-Dist: tenacity
-Requires-Dist: textual>=0.86.2
+Requires-Dist: textual<v3.0.0,>=0.86.2
 Requires-Dist: typing_extensions>=4.9.0
 Requires-Dist: zipp>=3.19.1
 Provides-Extra: dev

{inspect_ai-0.3.97 → inspect_ai-0.3.99}/docs/errors-and-limits.qmd

@@ -210,13 +210,27 @@ raise LimitExceededError(
 
 ## Scoped Limits
 
-You can also apply limits at arbitrary scopes, independent of the sample or agent-scoped limits. For instance, applied to a specific agent or within a specific block of code. For example:
+You can also apply limits at arbitrary scopes, independent of the sample or agent-scoped limits. For instance, applied to a specific block of code. For example:
 
 ``` python
 with token_limit(1024*500):
     ...
 ```
 
+A `LimitExceededError` will be raised if the limit is exceeded. The `source` field on `LimitExceededError` will be set to the `Limit` instance that was exceeded.
+
+The `apply_limits()` function accepts a list of `Limit` instances. If any of the limits passed in are exceeded, the `limit_error` property on the `LimitScope` yielded when opening the context manager will be set to the exception. By default, all `LimitExceededError` exceptions are propagated. However, if `catch_errors` is true, errors which are as a direct result of exceeding one of the limits passed to it will be caught. It will always allow `LimitExceededError` exceptions triggered by other limits (e.g. Sample scoped limits) to propagate up the call stack.
+
+``` python
+with apply_limits(
+    [token_limit(1000), message_limit(10)], catch_errors=True
+) as limit_scope:
+    ...
+if limit_scope.limit_error:
+    print(f"One of our limits was hit: {limit_scope.limit_error}")
+
+```
+
 ### Message Limit
 
 {{< include _message_limits.md >}}
@@ -310,20 +324,16 @@ Note that when limits are exceeded during an agent's execution, the way this is
 
 -   For agents used via `as_solver()`, if a limit is exceeded then the sample will terminate (this is exactly how sample-level limits work).
 
--   For agents that are `run()` directly, by default an exception is thrown and the sample terminates. You may on the other hand catch the exception and take another action:
+-   For agents that are `run()` directly with limits, their limit exceptions will be caught and returned in the tuple. Limits other than the ones passed to `run()` will propagate up the stack.
 
     ``` python
     from inspect_ai.agent import run
 
-    try:
-        state = await run(
-            agent=web_surfer(), 
-            input="What were the 3 most popular movies of 2020?",
-            limits=[token_limit(1024*500)])
-        )
-    except LimitExceededError as ex:
-        if ex.type == "token":
-            ...
-        else:
-            raise
-    ```
+    state, limit_error = await run(
+        agent=web_surfer(), 
+        input="What were the 3 most popular movies of 2020?",
+        limits=[token_limit(1024*500)])
+    )
+    if limit_error:
+        ...
+    ```

{inspect_ai-0.3.97 → inspect_ai-0.3.99}/docs/parallelism.qmd

@@ -307,8 +307,7 @@ Note that there are some features of Inspect that do not yet work when using Tri
 
 2. Interaction with AWS S3 (e.g. for log storage) uses the [s3fs](https://s3fs.readthedocs.io/en/latest/) package, which currently works only with asyncio.
 
-3. The [Google](providers.qmd#google) and [Bedrock](providers.qmd#aws-bedrock) providers both depend on asyncio so cannot be used with the Trio backend.
-
+3. The [Bedrock](providers.qmd#aws-bedrock) provider depends on asyncio so cannot be used with the Trio backend.
 
 ### Portable Async
 

{inspect_ai-0.3.97 → inspect_ai-0.3.99}/docs/reasoning.qmd

@@ -116,9 +116,9 @@ When using tools, you should read Anthropic's documentation on [extended thinkin
 
 Google currently makes available several Gemini reasoning models, the most recent of which are:
 
-- [Gemini 2.5 Flash](https://cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-5-flash): `google/gemini-2.5-flash-preview-04-17`
+- [Gemini 2.5 Flash](https://cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-5-flash): `google/gemini-2.5-flash-preview-05-20`
 
-- [Gemini 2.5 Pro](https://cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-5-pro):  `google/gemini-2.5-pro-preview-03-25`
+- [Gemini 2.5 Pro](https://cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-5-pro):  `google/gemini-2.5-pro-preview-05-06`
 
 You can use the `--reasoning-tokens` option to control the amount of reasoning used by these models. For example:
 
@@ -128,7 +128,7 @@ inspect eval math.py \
   --reasoning-tokens 4096
 ```
 
-The Gemini API includes support for including reasoning in model output, however this feature is not yet enabled for any of their deployed reasoning models.
+The most recent Gemini models also include support for including a reasoning summary in model output.
 
 ## Grok
 

{inspect_ai-0.3.97 → inspect_ai-0.3.99}/docs/reference/inspect_ai.qmd

@@ -2,9 +2,6 @@
 title: inspect_ai
 ---
 
-<!-- TODO: Main reference page? -->
-<!-- TODO: CLI reference -->
-
 ## Evaluation
 
 ### eval
@@ -20,6 +17,10 @@ title: inspect_ai
 ### TaskInfo
 ### Tasks
 
+## View
+
+### view
+
 ## Decorators
 
 ### task

{inspect_ai-0.3.97 → inspect_ai-0.3.99}/requirements.txt

@@ -27,6 +27,6 @@ semver>=3.0.0
 shortuuid
 sniffio
 tenacity
-textual>=0.86.2
+textual>=0.86.2,<v3.0.0 # https://github.com/UKGovernmentBEIS/inspect_ai/issues/1891
 typing_extensions>=4.9.0
 zipp>=3.19.1 # not directly required, pinned by Snyk to avoid a vulnerability

{inspect_ai-0.3.97 → inspect_ai-0.3.99}/src/inspect_ai/__init__.py

@@ -10,6 +10,7 @@ from inspect_ai._eval.score import score, score_async
 from inspect_ai._eval.task import Epochs, Task, TaskInfo, task_with
 from inspect_ai._eval.task.tasks import Tasks
 from inspect_ai._util.constants import PKG_NAME
+from inspect_ai._view.view import view
 from inspect_ai.agent._human.agent import human_cli
 from inspect_ai.solver._human_agent import human_agent
 
@@ -32,4 +33,5 @@ __all__ = [
     "TaskInfo",
     "task",
     "task_with",
+    "view",
 ]

{inspect_ai-0.3.97 → inspect_ai-0.3.99}/src/inspect_ai/_cli/log.py

@@ -199,6 +199,6 @@ def view_resource(file: str) -> str:
 
 
 def view_type_resource(file: str) -> str:
-    resource = PKG_PATH / "_view" / "www" / "src" / "types" / file
+    resource = PKG_PATH / "_view" / "www" / "src" / "@types" / file
     with open(resource, "r", encoding="utf-8") as f:
         return f.read()

{inspect_ai-0.3.97 → inspect_ai-0.3.99}/src/inspect_ai/_display/textual/widgets/transcript.py

@@ -84,6 +84,7 @@ class TranscriptView(ScrollableContainer):
                 scroll_to_end = (
                     new_sample or abs(self.scroll_y - self.max_scroll_y) <= 20
                 )
+
                 async with self.batch():
                     await self.remove_children()
                     await self.mount_all(
@@ -100,9 +101,13 @@ class TranscriptView(ScrollableContainer):
         else:
             self._pending_sample = sample
 
-    def _widgets_for_events(self, events: Sequence[Event]) -> list[Widget]:
+    def _widgets_for_events(
+        self, events: Sequence[Event], limit: int = 10
+    ) -> list[Widget]:
         widgets: list[Widget] = []
-        for event in events:
+        widget_count = 0
+        # reverse the events so that the N most recents events are displayed
+        for event in events[::-1]:
             display = render_event(event)
             if display:
                 for d in display:
@@ -118,7 +123,14 @@ class TranscriptView(ScrollableContainer):
                             set_transcript_markdown_options(d.content)
                         widgets.append(Static(d.content, markup=False))
                         widgets.append(Static(Text(" ")))
-        return widgets
+                        widget_count += 1
+
+            # only render the N most recent events
+            if widget_count >= limit:
+                break
+
+        # reverse the list since we added the events in reverse order
+        return widgets[::-1]
 
 
 class EventDisplay(NamedTuple):

{inspect_ai-0.3.97 → inspect_ai-0.3.99}/src/inspect_ai/_eval/run.py

@@ -298,10 +298,13 @@ async def run_multiple(tasks: list[TaskRunOptions], parallel: int) -> list[EvalL
 
     # setup pending tasks, queue, and results
     pending_tasks = tasks.copy()
-    results: list[EvalLog] = []
+    results: list[tuple[int, EvalLog]] = []
     tasks_completed = 0
     total_tasks = len(tasks)
 
+    # Create a mapping from task to its original index
+    task_to_original_index = {id(task): i for i, task in enumerate(tasks)}
+
     # produce/consume tasks
     send_channel, receive_channel = anyio.create_memory_object_stream[TaskRunOptions](
         parallel * 2
@@ -322,7 +325,7 @@ async def run_multiple(tasks: list[TaskRunOptions], parallel: int) -> list[EvalL
             # among those models, pick one with the least usage
             model = min(models_with_pending, key=lambda m: model_counts[m])
 
-            # now we know there’s at least one pending task for this model so it’s safe to pick it
+            # now we know there's at least one pending task for this model so it's safe to pick it
             next_task = next(t for t in pending_tasks if str(t.model) == model)
             pending_tasks.remove(next_task)
             model_counts[str(next_task.model)] += 1
@@ -339,6 +342,8 @@ async def run_multiple(tasks: list[TaskRunOptions], parallel: int) -> list[EvalL
             nonlocal tasks_completed
             async for task_options in receive_channel:
                 result: EvalLog | None = None
+                # Get the original index of this task
+                original_index = task_to_original_index[id(task_options)]
 
                 # run the task
                 try:
@@ -354,11 +359,13 @@ async def run_multiple(tasks: list[TaskRunOptions], parallel: int) -> list[EvalL
                             # see: https://docs.python.org/3/faq/programming.html#why-do-lambdas-defined-in-a-loop-with-different-values-all-return-the-same-result
                             def create_task_runner(
                                 options: TaskRunOptions = task_options,
+                                idx: int = original_index,
                             ) -> Callable[[], Awaitable[None]]:
                                 async def run_task() -> None:
                                     nonlocal result
                                     result = await task_run(options)
-                                    results.append(result)
+                                    # Store result with its original index
+                                    results.append((idx, result))
 
                                 return run_task
 
@@ -426,7 +433,8 @@ async def run_multiple(tasks: list[TaskRunOptions], parallel: int) -> list[EvalL
 
             clear_task_screen()
 
-        return results
+        # Sort results by original index and return just the values
+        return [r for _, r in sorted(results)]
 
 
 def resolve_task_sample_ids(
@@ -475,7 +483,12 @@ async def startup_sandbox_environments(
     sandboxenvs: Set[TaskSandboxEnvironment] = set()
     for task in tasks:
         # resolve each sample and add to sandboxenvs
-        dataset = slice_dataset(task.task.dataset, config.limit, config.sample_id)
+        resolved_task_sample_ids = resolve_task_sample_ids(
+            task.task.name, config.sample_id
+        )
+        dataset = slice_dataset(
+            task.task.dataset, config.limit, resolved_task_sample_ids
+        )
         for sample in dataset:
             sandbox = await resolve_sandbox_for_task_and_sample(
                 eval_sandbox, task.task, sample

{inspect_ai-0.3.97 → inspect_ai-0.3.99}/src/inspect_ai/_eval/task/log.py

@@ -56,7 +56,7 @@ class TaskLogger:
     def __init__(
         self,
         task_name: str,
-        task_version: int,
+        task_version: int | str,
         task_file: str | None,
         task_registry_name: str | None,
         task_id: str | None,

{inspect_ai-0.3.97 → inspect_ai-0.3.99}/src/inspect_ai/_eval/task/task.py

@@ -64,7 +64,7 @@ class Task:
         time_limit: int | None = None,
         working_limit: int | None = None,
         name: str | None = None,
-        version: int = 0,
+        version: int | str = 0,
         metadata: dict[str, Any] | None = None,
         **kwargs: Unpack[TaskDeprecatedArgs],
     ) -> None:

{inspect_ai-0.3.97 → inspect_ai-0.3.99}/src/inspect_ai/_util/_async.py

@@ -136,7 +136,7 @@ def current_async_backend() -> Literal["asyncio", "trio"] | None:
 
 
 def configured_async_backend() -> Literal["asyncio", "trio"]:
-    backend = os.environ.get("INSPECT_ASYNC_BACKEND", "asyncio").lower()
+    backend = os.environ.get("INSPECT_ASYNC_BACKEND", "asyncio").lower() or "asyncio"
     return _validate_backend(backend)
 
 

{inspect_ai-0.3.97 → inspect_ai-0.3.99}/src/inspect_ai/_view/schema.py

@@ -30,6 +30,7 @@ def sync_view_schema() -> None:
         for key in defs.keys():
             defs[key] = schema_to_strict(defs[key])
         f.write(json.dumps(schema, indent=2))
+        f.write("\n")
 
         # generate types w/ json-schema-to-typescript
         subprocess.run(

{inspect_ai-0.3.97 → inspect_ai-0.3.99}/src/inspect_ai/_view/view.py

@@ -30,6 +30,20 @@ def view(
     log_level: str | None = None,
     fs_options: dict[str, Any] = {},
 ) -> None:
+    """Run the Inspect View server.
+
+    Args:
+        log_dir: Directory to view logs from.
+        recursive: Recursively list files in `log_dir`.
+        host: Tcp/ip host (defaults to "127.0.0.1").
+        port: Tcp/ip port (defaults to 7575).
+        authorization: Validate requests by checking for this authorization header.
+        log_level: Level for logging to the console: "debug", "http", "sandbox",
+            "info", "warning", "error", or "critical" (defaults to "warning").
+        fs_options: Additional arguments to pass through to the filesystem provider
+            (e.g. `S3FileSystem`). Use `{"anon": True }` if you are accessing a
+            public S3 bucket with no credentials.
+    """
     init_dotenv()
     init_logger(log_level)
 

{inspect_ai-0.3.97 → inspect_ai-0.3.99}/src/inspect_ai/_view/www/dist/assets/index.css

@@ -17166,41 +17166,41 @@ thead th {
   flex-direction: column;
   padding-top: 0.1em;
 }
-._container_8i3m0_1 {
+._container_181fj_1 {
   display: grid;
   grid-template-columns:
-    minmax(auto, 1fr) minmax(auto, 1fr) minmax(auto, 1fr)
-    2fr;
+    minmax(0, max-content) minmax(0, max-content) minmax(0, max-content)
+    5fr;
   column-gap: 0.75em;
 }
 
-._container_8i3m0_1 ._cell_8i3m0_9 {
+._container_181fj_1 ._cell_181fj_9 {
   margin-bottom: 0.5em;
 }
 
-._fullWidth_8i3m0_13 {
+._fullWidth_181fj_13 {
   grid-column: 1 / -1;
 }
 
-._heading_8i3m0_17 {
+._heading_181fj_17 {
   font-weight: 600;
 }
 
-._padded_8i3m0_21 {
+._padded_181fj_21 {
   padding-bottom: 3em;
 }
 
-._separator_8i3m0_25 {
+._separator_181fj_25 {
   height: 1px;
   background-color: var(--bs-light-border-subtle);
 }
 
-._separatorPadded_8i3m0_30 {
+._separatorPadded_181fj_30 {
   margin-top: 0.5em;
   margin-bottom: 0.5em;
 }
 
-._headerSep_8i3m0_35 {
+._headerSep_181fj_35 {
   margin-top: 0.1em;
   margin-bottom: 0.2em;
 }

{inspect_ai-0.3.97 → inspect_ai-0.3.99}/src/inspect_ai/_view/www/dist/assets/index.js

@@ -39446,7 +39446,7 @@ Please change the parent <Route path="${parentPath}"> to <Route path="${parentPa
           const rendered = entry2.value.trim();
           if (options2.renderString === "markdown") {
             return {
-              rendered
+              rendered: /* @__PURE__ */ jsxRuntimeExports.jsx(MarkdownDiv, { markdown: rendered })
             };
           } else {
             return {
@@ -51898,12 +51898,12 @@ self.onmessage = function (e) {
       );
       return scorerDescriptor == null ? void 0 : scorerDescriptor.render(scoreData.value);
     };
-    const container$6 = "_container_8i3m0_1";
-    const cell$1 = "_cell_8i3m0_9";
-    const fullWidth = "_fullWidth_8i3m0_13";
-    const separator$2 = "_separator_8i3m0_25";
-    const separatorPadded = "_separatorPadded_8i3m0_30";
-    const headerSep = "_headerSep_8i3m0_35";
+    const container$6 = "_container_181fj_1";
+    const cell$1 = "_cell_181fj_9";
+    const fullWidth = "_fullWidth_181fj_13";
+    const separator$2 = "_separator_181fj_25";
+    const separatorPadded = "_separatorPadded_181fj_30";
+    const headerSep = "_headerSep_181fj_35";
     const styles$x = {
       container: container$6,
       cell: cell$1,
@@ -52473,7 +52473,7 @@ self.onmessage = function (e) {
             {
               output: event.error.traceback_ansi,
               style: {
-                fontSize: "clamp(0.5rem, calc(0.25em + 1vw), 0.8rem)",
+                fontSize: "clamp(0.3rem, 1.1vw, 0.8rem)",
                 margin: "0.5em 0"
               }
             }
@@ -61749,7 +61749,7 @@ ${events}
                           output: sample2.error.traceback_ansi,
                           className: clsx("text-size-small", styles$A.ansi),
                           style: {
-                            fontSize: "clamp(0.4rem, calc(0.15em + 1vw), 0.8rem)",
+                            fontSize: "clamp(0.3rem, 1.1vw, 0.8rem)",
                             margin: "0.5em 0"
                           }
                         }
@@ -61764,7 +61764,7 @@ ${events}
                             output: retry.traceback_ansi,
                             className: clsx("text-size-small", styles$A.ansi),
                             style: {
-                              fontSize: "clamp(0.4rem, calc(0.15em + 1vw), 0.8rem)",
+                              fontSize: "clamp(0.3rem, 1.1vw, 0.8rem)",
                               margin: "0.5em 0"
                             }
                           }

{inspect_ai-0.3.97 → inspect_ai-0.3.99}/src/inspect_ai/_view/www/log-schema.json

@@ -1136,6 +1136,18 @@
           "default": null,
           "title": "Log Samples"
         },
+        "log_realtime": {
+          "anyOf": [
+            {
+              "type": "boolean"
+            },
+            {
+              "type": "null"
+            }
+          ],
+          "default": null,
+          "title": "Log Realtime"
+        },
         "log_images": {
           "anyOf": [
             {
@@ -1205,6 +1217,7 @@
         "max_sandboxes",
         "sandbox_cleanup",
         "log_samples",
+        "log_realtime",
         "log_images",
         "log_buffer",
         "log_shared",
@@ -1502,7 +1515,8 @@
             "reasoning_tokens": null,
             "reasoning_summary": null,
             "reasoning_history": null,
-            "response_schema": null
+            "response_schema": null,
+            "extra_body": null
           }
         }
       },
@@ -1944,7 +1958,7 @@
       "additionalProperties": false
     },
     "EvalSampleLimit": {
-      "description": "Limit encontered by sample.",
+      "description": "Limit encountered by sample.",
       "properties": {
         "type": {
           "enum": [
@@ -2277,6 +2291,10 @@
     "EvalSpec": {
       "description": "Eval target and configuration.",
       "properties": {
+        "eval_id": {
+          "title": "Eval Id",
+          "type": "string"
+        },
         "run_id": {
           "title": "Run Id",
           "type": "string"
@@ -2294,9 +2312,16 @@
           "type": "string"
         },
         "task_version": {
+          "anyOf": [
+            {
+              "type": "integer"
+            },
+            {
+              "type": "string"
+            }
+          ],
           "default": 0,
-          "title": "Task Version",
-          "type": "integer"
+          "title": "Task Version"
         },
         "task_file": {
           "anyOf": [
@@ -2500,6 +2525,7 @@
         }
       },
       "required": [
+        "eval_id",
         "run_id",
         "created",
         "task",
@@ -2897,6 +2923,19 @@
             }
           ],
           "default": null
+        },
+        "extra_body": {
+          "anyOf": [
+            {
+              "additionalProperties": true,
+              "type": "object"
+            },
+            {
+              "type": "null"
+            }
+          ],
+          "default": null,
+          "title": "Extra Body"
         }
       },
       "title": "GenerateConfig",
@@ -2927,7 +2966,8 @@
         "reasoning_tokens",
         "reasoning_summary",
         "reasoning_history",
-        "response_schema"
+        "response_schema",
+        "extra_body"
       ],
       "additionalProperties": false
     },