inspect-ai 0.3.95__tar.gz → 0.3.96__tar.gz

{inspect_ai-0.3.95 → inspect_ai-0.3.96}/.github/dependabot.yml

@@ -20,3 +20,5 @@ updates:
         versions: [">=14.0.0"]
       - dependency-name: "fsspec"
         versions: [">=2024.12.0"]
+      - dependency-name: "click"
+        versions: [">=8.2.0"]

{inspect_ai-0.3.95 → inspect_ai-0.3.96}/CHANGELOG.md

@@ -1,6 +1,13 @@
+## v0.3.96 (13 May 2025)
+
+- Dataframes: `events_df()` function, improved message reading, log filtering, don't re-sort passed logs
+- Model Context Protocol: Upgrade sandbox client to typing changes made in v1.8.0 of `mcp` package.
+- vLLM/SGLang: Fix dynamic port binding for local server on Mac OS X.
+- React Agent: Improve continue prompt to remind the model to include the answer in their call to `submit()`.
+
 ## v0.3.95 (10 May 2025)
 
-- [DataFrame](https://inspect.aisi.org.uk/dataframe.html) functions for reading dataframes from log files.
+- [Dataframe](https://inspect.aisi.org.uk/dataframe.html) functions for reading dataframes from log files.
 - Web Search: Added provider for [Tavily](https://inspect.aisi.org.uk/tools-standard.html#tavily-provider) Research API.
 - Multiple Choice: `max_tokens` option to control tokens used for `generate()`.
 - Don't enforce sample `working_limit` after solvers have completed executing (matching behavior of other sample limits).

{inspect_ai-0.3.95 → inspect_ai-0.3.96}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: inspect_ai
-Version: 0.3.95
+Version: 0.3.96
 Summary: Framework for large language model evaluations
 Author: UK AI Security Institute
 License: MIT License
@@ -23,7 +23,7 @@ License-File: LICENSE
 Requires-Dist: aiohttp>=3.9.0
 Requires-Dist: anyio>=4.8.0
 Requires-Dist: beautifulsoup4
-Requires-Dist: click>=8.1.3
+Requires-Dist: click<8.2.0,>=8.1.3
 Requires-Dist: debugpy
 Requires-Dist: docstring-parser>=0.16
 Requires-Dist: exceptiongroup>=1.0.2; python_version < "3.11"

{inspect_ai-0.3.95 → inspect_ai-0.3.96}/docs/dataframe.qmd

@@ -3,7 +3,7 @@ title: Log Dataframes
 ---
 
 ::: callout-note
-Log dataframe functions are currently in beta and are exported from the **inspect_ai.analysis.beta** module. The beta module will be preserved after final release so that code written against it now will continue to work after the beta.
+Dataframe functions are currently in beta and are exported from the **inspect_ai.analysis.beta** module. The beta module will be preserved after final release so that code written against it now will continue to work after the beta.
 :::
 
 ## Overview {#overview}
@@ -28,21 +28,22 @@ Inspect eval logs have a hierarchical structure which is well suited to flexibly
 | [evals_df()](#evals) | Evaluation level data (e.g. task, model, scores, etc.). One row per log file. |
 | [samples_df()](#samples) | Sample level data (e.g. input, metadata, scores, errors, etc.) One row per sample, where each log file contains many samples. |
 | [messages_df()](#messages) | Message level data (e.g. role, content, etc.). One row per message, where each sample contains many messages. |
+| [events_df()](#events) | Event level data (type, timing, content, etc.). One row per event, where each sample contains many events. |
 
 Each function extracts a default set of columns, however you can tailor column reading to work in whatever way you need for your analysis. Extracted dataframes can either be denormalized (e.g. if you want to immediately summarise or plot them) or normalised (e.g. if you are importing them into a SQL database).
 
 Below we'll walk through a few examples, then after that provide more in-depth documentation on customising how dataframes are read for various scenarios.
 
-## Examples
+## Basics
 
-### Import Basics
+### Reading Data
 
 Use the `evals_df()` function to read a dataframe containing a row for each log file (note that we import from `inspect_ai.analysis.beta` since the dataframe functions are currently in beta):
 
 ``` python
+# read logs from a given log directory
 from inspect_ai.analysis.beta import evals_df
-
-evals_df("logs")
+evals_df("logs")   
 ```
 
 ``` default
@@ -51,7 +52,7 @@ RangeIndex: 9 entries, 0 to 8
 Columns: 51 entries, eval_id to score_model_graded_qa_stderr
 ```
 
-The default configuration for `evals_df()` reads 51 columns. While this is the default, column reading can be customized in variety of ways (covered below in [Columns](#columns)).
+The default configuration for `evals_df()` reads a predefined set of columns. You can customise column reading in a variety of ways (covered below in [Columns](#columns)).
 
 Use the `samples_df()` function to read a dataframe with a record for each sample across a set of log files. For example, here we read all of the samples in the "logs" directory:
 
@@ -113,6 +114,37 @@ Columns: 27 entries, sample_id to retries
 
 This dataframe has 27 columns rather than than the 13 we saw for the default `samples_df()` behavior, reflecting the additional eval level columns. You can create your own column groups and definitions to further customise reading (see [Columns](#columns) for details).
 
+### Filtering Logs
+
+The above examples read all of the logs within a given directory. You can also use the `list_eval_logs()` function to filter the list of logs based on arbitrary criteria as well control whether log listings are recursive.
+
+For example, here we read only log files with a `status` of "success":
+
+``` python
+# read only successful logs from a given log directory
+logs = list_eval_logs("logs", filter=lambda log: log.status == "success")
+evals_df(logs)
+```
+
+Here we read only logs with the task name "popularity":
+
+``` python
+# read only logs with task name 'popularity'
+def task_filter(log: EvalLog) -> bool:
+    return log.eval.task == "popularity"
+    
+logs = list_eval_logs("logs", filter=task_filter)
+evals_df(logs)
+```
+
+We can also choose to read a directory non-recursively:
+
+``` python
+# read only the logs at the top level of 'logs'
+logs = list_eval_logs("logs", recursive=False)
+evals_df(logs)
+```
+
 ### Databases
 
 You can also read multiple dataframes and combine them into a relational database. Imported dataframes automatically include fields that can be used to join them (e.g. `eval_id` is in both the evals and samples tables).
@@ -221,19 +253,42 @@ if len(errors) > 0:
 
 | Type | Description |
 |------------------------------------|------------------------------------|
-| [EvalInfo](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/analysis/beta/_dataframe/evals/columns.py#L51) | Descriptive information (e.g. created, tags, metadata, git commit, etc.) |
-| [EvalTask](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/analysis/beta/_dataframe/evals/columns.py#L64) | Task configuration (name, file, args, solver, etc.) |
-| [EvalModel](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/analysis/beta/_dataframe/evals/columns.py#L77) | Model name, args, generation config, etc. |
-| [EvalDataset](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/analysis/beta/_dataframe/evals/columns.py#L86) | Dataset name, location, sample ids, etc. |
-| [EvalConfig](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/analysis/beta/_dataframe/evals/columns.py#L95) | Epochs, approval, sample limits, etc. |
-| [EvalResults](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/analysis/beta/_dataframe/evals/columns.py#L106) | Status, errors, samples completed, headline metric. |
-| [EvalScores](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/analysis/beta/_dataframe/evals/columns.py#L118) | All scores and metrics broken into separate columns. |
+| `EvalInfo` | Descriptive information (e.g. created, tags, metadata, git commit, etc.) |
+| `EvalTask` | Task configuration (name, file, args, solver, etc.) |
+| `EvalModel` | Model name, args, generation config, etc. |
+| `EvalDataset` | Dataset name, location, sample ids, etc. |
+| `EvalConfig` | Epochs, approval, sample limits, etc. |
+| `EvalResults` | Status, errors, samples completed, headline metric. |
+| `EvalScores` | All scores and metrics broken into separate columns. |
+
+#### Multi-Columns
+
+The `task_args` dictionary and eval scores data structure are both expanded into multiple columns by default:
+
+``` python
+EvalColumn("task_arg_*", path="eval.task_args")
+EvalColumn("score_*_*", path=eval_log_scores_dict)
+```
+
+Note that scores are a two-level dictionary of `score_<scorer>_<metric>` and are extracted using a custom function. If you want to handle scores a different way you can build your own set of eval columns with a custom scores handler. For example, here we take a subset of eval columns along with our own custom handler (`custom_scores_fn`) for scores:
+
+``` python
+evals_df(
+    logs="logs", 
+    columns=(
+        EvalInfo
+        + EvalModel
+        + EvalResults
+        + ([EvalColumn("score_*_*", path=custom_scores_fn)])
+    )
+)
+```
 
 #### Custom Extraction
 
-The `EvalColumn` class provides an additional facility for reading values based on a callback function that takes an `EvalLog`. This function is specified as the `path` in the column definition.
+The example above demonstrates the use of custom extraction functions, which take an `EvalLog` and return a `JsonValue`.
 
-For example, here is the `path` function used to extract a simple dictionary of scores/metrics from the more complex `list[EvalScore]` type provided as `log.scores`:
+For example, here is the default extraction function for the the dictionary of scores/metrics:
 
 ``` python
 def scores_dict(log: EvalLog) -> JsonValue:
@@ -269,7 +324,7 @@ By default, the `SampleSummary` column group is used, which reads only from summ
 SampleSummary: list[Column] = [
     SampleColumn("id", path="id", required=True, type=str),
     SampleColumn("epoch", path="epoch", required=True),
-    SampleColumn("input", path="input", required=True, value=input_as_str),
+    SampleColumn("input", path=sample_input_as_str, required=True),
     SampleColumn("target", path="target", required=True, value=list_as_str),
     SampleColumn("metadata_*", path="metadata"),
     SampleColumn("score_*", path="scores", value=score_values),
@@ -297,6 +352,33 @@ samples_df(
 
 Note that reading `SampleMessages` requires reading full sample content, so will take considerably longer than reading only summaries.
 
+When you create a samples data frame the `eval_id` of its parent evaluation is automatically included. You can additionally include other fields from the evals table, for example:
+
+``` python
+samples_df(
+    logs="logs", 
+    columns = EvalModel + SampleSummary + SampleMessages
+)
+```
+
+#### Multi-Columns
+
+Note that the `metadata` and `score` columns are both dictionaries that are expanded into multiple columns:
+
+``` python
+SampleColumn("metadata_*", path="metadata")
+SampleColumn("score_*", path="scores", value=score_values)
+```
+
+This might or might not be what you want for your data frame. To preserve them as JSON, remove the `_*`:
+
+``` python
+SampleColumn("metadata", path="metadata")
+SampleColumn("score", path="scores")
+```
+
+You could also write a custom [extraction](#custom-extraction-1) handler to read them in some other way.
+
 #### Full Samples
 
 `SampleColumn` will automatically determine whether it is referencing a field that requires a full sample read (for example, `messages` or `store`). There are five fields in sample summaries that have reduced footprint in the summary (`input`, `metadata`, and `scores`, `error`, and `limit`). For these, fields specify `full=True` to force reading from the full sample record. For example:
@@ -306,7 +388,7 @@ SampleColumn("limit_type", path="limit.type", full=True)
 SampleColumn("limit_value", path="limit.limit", full=True)
 ```
 
-#### Custom Extraction
+#### Custom Extraction {#custom-extraction-1}
 
 As with `EvalColumn`, you can also extract data from a sample using a callback function passed as the `path`:
 
@@ -327,7 +409,7 @@ $ inspect log convert ./logs --to eval --output-dir ./logs-amended
 
 ### Messages {#messages}
 
-The `messages_df()` function enables reading message level data from a set of eval logs. Each row corresponds to a message, and includes a `sample_id` and `event_id` for linking back to its parents.
+The `messages_df()` function enables reading message level data from a set of eval logs. Each row corresponds to a message, and includes a `sample_id` and `eval_id` for linking back to its parents.
 
 The `messages_df()` function takes a `filter` parameter which can either be a list of `role` designations or a function that performs filtering. For example:
 
@@ -356,6 +438,15 @@ MessageToolCalls: list[Column] = [
 MessageColumns: list[Column] = MessageContent + MessageToolCalls
 ```
 
+When you create a messages data frame the parent `sample_id` and `eval_id` are automatically included in each record. You can additionally include other fields from these tables, for example:
+
+``` python
+messages = messages_df(
+    logs="logs",
+    columns=EvalModel + MessageColumns             
+)
+```
+
 #### Custom Extraction
 
 Two of the fields above are resolved using custom extraction functions (`content` and `tool_calls`). Here is the source code for those functions:
@@ -364,7 +455,6 @@ Two of the fields above are resolved using custom extraction functions (`content
 def message_text(message: ChatMessage) -> str:
     return message.text
 
-
 def message_tool_calls(message: ChatMessage) -> str | None:
     if isinstance(message, ChatMessageAssistant) and message.tool_calls is not None:
         tool_calls = "\n".join(
@@ -380,6 +470,41 @@ def message_tool_calls(message: ChatMessage) -> str | None:
         return None
 ```
 
+### Events {#events}
+
+The `events_df()` function enables reading event level data from a set of eval logs. Each row corresponds to an event, and includes a `sample_id` and `eval_id` for linking back to its parents.
+
+Because events are so heterogeneous, there is no default `columns` specification for calls to `events_df()`. Rather, you can compose columns from the following pre-built groups:
+
+| Type | Description |
+|------------------------------------|------------------------------------|
+| `EventInfo` | Event type and span id. |
+| `EventTiming` | Start and end times (both clock time and working time) |
+| `ModelEventColumns` | Read data from model events. |
+| `ToolEventColumns` | Read data from tool events. |
+
+The `events_df()` function also takes a `filter` parameter which can either be a list of event types or a function that performs filtering. For example, to read all model events:
+
+``` python
+model_events = events_df(
+    logs="logs", 
+    columns=EventTiming + ModelEventColumns,
+    filter=["model"]
+)
+```
+
+To read all tool events:
+
+``` python
+model_events = events_df(
+    logs="logs", 
+    columns=EvalModel + EventTiming + ToolEventColumns,
+    filter=["tool"]
+)
+```
+
+Note that for tool events we also include the `EvalModel` column group as model information is not directly embedded in tool events (whereas it is within model events).
+
 ### Custom
 
 You can create custom column types that extract data based on additional parameters. For example, imagine you want to write a set of extraction functions that are passed a `ReportConfig` and an `EvalLog` (the report configuration might specify scores to extract, normalisation constraints, etc.)

{inspect_ai-0.3.95 → inspect_ai-0.3.96}/docs/reference/_sidebar.yml

@@ -500,6 +500,18 @@ website:
           href: reference/inspect_ai.analysis.qmd#messagetoolcalls
         - text: MessageColumns
           href: reference/inspect_ai.analysis.qmd#messagecolumns
+        - text: events_df
+          href: reference/inspect_ai.analysis.qmd#events_df
+        - text: EventColumn
+          href: reference/inspect_ai.analysis.qmd#eventcolumn
+        - text: EventInfo
+          href: reference/inspect_ai.analysis.qmd#eventinfo
+        - text: EventTiming
+          href: reference/inspect_ai.analysis.qmd#eventtiming
+        - text: ModelEventColumns
+          href: reference/inspect_ai.analysis.qmd#modeleventcolumns
+        - text: ToolEventColumns
+          href: reference/inspect_ai.analysis.qmd#tooleventcolumns
         - text: Column
           href: reference/inspect_ai.analysis.qmd#column
         - text: ColumnType

{inspect_ai-0.3.95 → inspect_ai-0.3.96}/docs/reference/inspect_ai.analysis.qmd

@@ -35,6 +35,15 @@ Analysis functions are currently in beta and are exported from the **inspect_ai.
 ### MessageToolCalls
 ### MessageColumns
 
+## Events
+
+### events_df
+### EventColumn
+### EventInfo
+### EventTiming
+### ModelEventColumns
+### ToolEventColumns
+
 ## Columns
 
 ### Column

{inspect_ai-0.3.95 → inspect_ai-0.3.96}/examples/popularity.py

@@ -1,6 +1,6 @@
 from inspect_ai import Task, task
 from inspect_ai.dataset import FieldSpec, example_dataset
-from inspect_ai.scorer import includes, match
+from inspect_ai.scorer import match
 from inspect_ai.solver import generate, system_message
 
 SYSTEM_MESSAGE = r"""
@@ -22,5 +22,5 @@ def popularity():
     return Task(
         dataset=dataset,
         solver=[system_message(SYSTEM_MESSAGE), generate()],
-        scorer=[match(), includes()],
+        scorer=[match()],
     )

{inspect_ai-0.3.95 → inspect_ai-0.3.96}/requirements.txt

@@ -1,7 +1,7 @@
 aiohttp>=3.9.0
 anyio>=4.8.0
 beautifulsoup4
-click>=8.1.3
+click>=8.1.3,<8.2.0
 debugpy
 docstring-parser>=0.16
 exceptiongroup>=1.0.2; python_version < '3.11'

{inspect_ai-0.3.95 → inspect_ai-0.3.96}/src/inspect_ai/_util/local_server.py

@@ -1,6 +1,7 @@
 import json
 import logging
 import os
+import platform
 import random
 import socket
 import subprocess
@@ -33,6 +34,21 @@ def reserve_port(
     Returns:
         A tuple (port, lock_socket) where `lock_socket` is kept open to hold the lock.
     """
+    is_macos = platform.system() == "Darwin"
+
+    if is_macos:
+        logger.info(
+            "MacOS system detected. A free binding port will be identified, but not reserved until the server binds to it."
+        )
+        # On macOS, let the OS pick a free port but not open it
+        # It leads to a small racode condition window until the port
+        # is actually opened by the llm server
+        with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
+            s.bind((host, 0))  # Bind to any free port
+            port = s.getsockname()[1]
+        return port, s
+
+    # Non-macOS behavior: try ports in range
     candidates = list(range(start, end))
     random.shuffle(candidates)
 

{inspect_ai-0.3.95 → inspect_ai-0.3.96}/src/inspect_ai/agent/_types.py

@@ -43,7 +43,7 @@ class AgentPrompt(NamedTuple):
 
 DEFAULT_CONTINUE_PROMPT = """
 Please proceed to the next step using your best judgement. If you believe you
-have completed the task, please call the `{submit}()` tool.
+have completed the task, please call the `{submit}()` tool with your final answer.
 """
 
 

{inspect_ai-0.3.95 → inspect_ai-0.3.96}/src/inspect_ai/analysis/beta/__init__.py

@@ -15,7 +15,13 @@ from ._dataframe.evals.columns import (
     EvalTask,
 )
 from ._dataframe.evals.table import evals_df
-from ._dataframe.events.columns import EventColumn
+from ._dataframe.events.columns import (
+    EventColumn,
+    EventInfo,
+    EventTiming,
+    ModelEventColumns,
+    ToolEventColumns,
+)
 from ._dataframe.events.table import events_df
 from ._dataframe.messages.columns import (
     MessageColumn,
@@ -50,6 +56,10 @@ __all__ = [
     "MessageFilter",
     "events_df",
     "EventColumn",
+    "EventInfo",
+    "EventTiming",
+    "ModelEventColumns",
+    "ToolEventColumns",
     "Column",
     "ColumnType",
     "ColumnError",

{inspect_ai-0.3.95 → inspect_ai-0.3.96}/src/inspect_ai/analysis/beta/_dataframe/evals/table.py

@@ -1,10 +1,11 @@
 from __future__ import annotations
 
-from typing import TYPE_CHECKING, Literal, overload
+from typing import TYPE_CHECKING, Callable, Literal, overload
 
-from inspect_ai._display import display
 from inspect_ai._util.path import pretty_path
+from inspect_ai.analysis.beta._dataframe.progress import import_progress
 from inspect_ai.log._file import (
+    list_eval_logs,
     read_eval_log,
 )
 
@@ -29,39 +30,32 @@ EVAL_SUFFIX = "_eval"
 
 @overload
 def evals_df(
-    logs: LogPaths,
+    logs: LogPaths = list_eval_logs(),
     columns: list[Column] = EvalColumns,
-    recursive: bool = True,
-    reverse: bool = False,
     strict: Literal[True] = True,
 ) -> "pd.DataFrame": ...
 
 
 @overload
 def evals_df(
-    logs: LogPaths,
+    logs: LogPaths = list_eval_logs(),
     columns: list[Column] = EvalColumns,
-    recursive: bool = True,
-    reverse: bool = False,
     strict: Literal[False] = False,
 ) -> tuple["pd.DataFrame", ColumnErrors]: ...
 
 
 def evals_df(
-    logs: LogPaths,
+    logs: LogPaths = list_eval_logs(),
     columns: list[Column] = EvalColumns,
-    recursive: bool = True,
-    reverse: bool = False,
     strict: bool = True,
 ) -> "pd.DataFrame" | tuple["pd.DataFrame", ColumnErrors]:
     """Read a dataframe containing evals.
 
     Args:
        logs: One or more paths to log files or log directories.
+          Defaults to the contents of the currently active log directory
+          (e.g. ./logs or INSPECT_LOG_DIR).
        columns: Specification for what columns to read from log files.
-       recursive: Include recursive contents of directories (defaults to `True`)
-       reverse: Reverse the order of the dataframe (by default, items
-          are ordered from oldest to newest).
        strict: Raise import errors immediately. Defaults to `True`.
           If `False` then a tuple of `DataFrame` and errors is returned.
 
@@ -73,7 +67,46 @@ def evals_df(
     verify_prerequisites()
 
     # resolve logs
-    log_paths = resolve_logs(logs, recursive=recursive, reverse=reverse)
+    log_paths = resolve_logs(logs)
+
+    with import_progress("reading logs", total=len(log_paths)) as (p, task_id):
+        if strict:
+            evals_table, _ = _read_evals_df(
+                log_paths, columns, True, lambda: p.update(task_id, advance=1)
+            )
+            return evals_table
+        else:
+            evals_table, all_errors, _ = _read_evals_df(
+                log_paths, columns, False, lambda: p.update(task_id, advance=1)
+            )
+            return evals_table, all_errors
+
+
+@overload
+def _read_evals_df(
+    log_paths: list[str],
+    columns: list[Column],
+    strict: Literal[True],
+    progress: Callable[[], None],
+) -> tuple["pd.DataFrame", int]: ...
+
+
+@overload
+def _read_evals_df(
+    log_paths: list[str],
+    columns: list[Column],
+    strict: Literal[False],
+    progress: Callable[[], None],
+) -> tuple["pd.DataFrame", ColumnErrors, int]: ...
+
+
+def _read_evals_df(
+    log_paths: list[str],
+    columns: list[Column],
+    strict: bool,
+    progress: Callable[[], None],
+) -> tuple["pd.DataFrame", int] | tuple["pd.DataFrame", ColumnErrors, int]:
+    verify_prerequisites()
 
     # resolve duplicate columns
     columns = resolve_duplicate_columns(columns)
@@ -85,27 +118,31 @@ def evals_df(
     ensure_eval_id(columns)
 
     # read logs
+    total_samples = 0
     records: list[dict[str, ColumnType]] = []
-    with display().progress(total=len(log_paths)) as p:
-        for log_path in log_paths:
-            log = read_eval_log(log_path, header_only=True)
-            if strict:
-                record = import_record(log, columns, strict=True)
-            else:
-                record, errors = import_record(log, columns, strict=False)
-                all_errors[pretty_path(log_path)] = errors
-            records.append(record)
-
-            p.update()
+    for log_path in log_paths:
+        log = read_eval_log(log_path, header_only=True)
+        if strict:
+            record = import_record(log, columns, strict=True)
+        else:
+            record, errors = import_record(log, columns, strict=False)
+            all_errors[pretty_path(log_path)] = errors
+        records.append(record)
+        total_samples += (
+            len(log.eval.dataset.sample_ids)
+            if log.eval.dataset.sample_ids is not None
+            else (log.eval.dataset.samples or 100)
+        )
+        progress()
 
     # return table (+errors if strict=False)
     evals_table = records_to_pandas(records)
     evals_table = reorder_evals_df_columns(evals_table, columns)
 
     if strict:
-        return evals_table
+        return evals_table, total_samples
     else:
-        return evals_table, all_errors
+        return evals_table, all_errors, total_samples
 
 
 def ensure_eval_id(columns: list[Column]) -> None:

inspect_ai-0.3.96/src/inspect_ai/analysis/beta/_dataframe/events/columns.py

@@ -0,0 +1,87 @@
+from datetime import datetime
+from typing import Any, Callable, Mapping, Type
+
+from jsonpath_ng import JSONPath  # type: ignore
+from pydantic import JsonValue
+from typing_extensions import override
+
+from inspect_ai.log._transcript import Event
+
+from ..columns import Column, ColumnType
+from .extract import (
+    completion_as_str,
+    model_event_input_as_str,
+    tool_choice_as_str,
+    tool_view_as_str,
+)
+
+
+class EventColumn(Column):
+    """Column which maps to `Event`."""
+
+    def __init__(
+        self,
+        name: str,
+        *,
+        path: str | JSONPath | Callable[[Event], JsonValue],
+        required: bool = False,
+        default: JsonValue | None = None,
+        type: Type[ColumnType] | None = None,
+        value: Callable[[JsonValue], JsonValue] | None = None,
+    ) -> None:
+        super().__init__(
+            name=name,
+            path=path if not callable(path) else None,
+            required=required,
+            default=default,
+            type=type,
+            value=value,
+        )
+        self._extract_event = path if callable(path) else None
+
+    @override
+    def path_schema(self) -> Mapping[str, Any] | None:
+        return None
+
+
+EventInfo: list[Column] = [
+    EventColumn("event", path="event"),
+    EventColumn("span_id", path="span_id"),
+]
+"""Event basic information columns."""
+
+EventTiming: list[Column] = [
+    EventColumn("timestamp", path="timestamp", type=datetime),
+    EventColumn("completed", path="completed", type=datetime),
+    EventColumn("working_start", path="working_start"),
+    EventColumn("working_time", path="working_time"),
+]
+"""Event timing columns."""
+
+ModelEventColumns: list[Column] = [
+    EventColumn("model_event_model", path="model"),
+    EventColumn("model_event_role", path="role"),
+    EventColumn("model_event_input", path=model_event_input_as_str),
+    EventColumn("model_event_tools", path="tools"),
+    EventColumn("model_event_tool_choice", path=tool_choice_as_str),
+    EventColumn("model_event_config", path="config"),
+    EventColumn("model_event_usage", path="output.usage"),
+    EventColumn("model_event_time", path="output.time"),
+    EventColumn("model_event_completion", path=completion_as_str),
+    EventColumn("model_event_retries", path="retries"),
+    EventColumn("model_event_error", path="error"),
+    EventColumn("model_event_cache", path="cache"),
+    EventColumn("model_event_call", path="call"),
+]
+"""Model event columns."""
+
+ToolEventColumns: list[Column] = [
+    EventColumn("tool_event_function", path="function"),
+    EventColumn("tool_event_arguments", path="arguments"),
+    EventColumn("tool_event_view", path=tool_view_as_str),
+    EventColumn("tool_event_result", path="result"),
+    EventColumn("tool_event_truncated", path="truncated"),
+    EventColumn("tool_event_error_type", path="error.type"),
+    EventColumn("tool_event_error_message", path="error.message"),
+]
+"""Tool event columns."""