inspect-ai 0.3.94__tar.gz → 0.3.95__tar.gz

{inspect_ai-0.3.94 → inspect_ai-0.3.95}/CHANGELOG.md

@@ -1,9 +1,30 @@
+## v0.3.95 (10 May 2025)
+
+- [DataFrame](https://inspect.aisi.org.uk/dataframe.html) functions for reading dataframes from log files.
+- Web Search: Added provider for [Tavily](https://inspect.aisi.org.uk/tools-standard.html#tavily-provider) Research API.
+- Multiple Choice: `max_tokens` option to control tokens used for `generate()`.
+- Don't enforce sample `working_limit` after solvers have completed executing (matching behavior of other sample limits).
+- Only pass `user` parameter on to sandboxes if is not `None` (eases compatibility with older sandbox providers).
+- Anthropic: Retry when `type` in the error message body is "overloaded_error". 
+- Agent Bridge: Compatibility with `request()` method in v1.78.0 of `openai` package (now the minimum required version).
+- Model Context Protocol: Update to typing changes made in v1.8.0 of `mcp` package (now the minimum required version).
+- TaskState: `input_text` and `user_prompt` properties now read the last rather than first user message.
+- Inspect View: Properly display 'more' options when content is collapsed.
+- Inspect View: Fix issue that prevented filtering of sample list when viewing a running evaluation.
+- Inspect View: Fix selection of specific metrics within scorers when a scorer produces more than one metric.
+- Ignore OSError that occurs while rotating trace files.
+- Restore logging `metadata` from `TaskState` rather than from `Sample`.
+- Bugfix: Restore ability of operator to terminate the current sample in tool call approval.
+- Bugfix: Ensure that "init" span is exited in the same async context when sandbox connection errors occur.
+- Bugfix: Protect against no `thought` argument being passed to `think()` tool.
+- Bugfix: Correct handling of `text_editor()` tool for Claude Sonnet 3.5.
+
 ## v0.3.94 (06 May 2025)
 
 - [span()](https://inspect.aisi.org.uk/agent-custom.html#grouping-with-spans) function for grouping transcript events.
 - [collect()](https://inspect.aisi.org.uk/agent-custom.html#grouping-with-spans) function for enclosing parallel tasks in spans.
 - [Event tree](https://inspect.aisi.org.uk/reference/inspect_ai.log.html#event-tree) functions for organising transcript events into a tree of spans.
-- `inspect convert` now always fully re-writes log files even of the same format (so that e.g. sample summaries always exist in the converted logs).
+- `inspect log convert` now always fully re-writes log files even of the same format (so that e.g. sample summaries always exist in the converted logs).
 - React agent: `answer_only` and `answer_delimiter` to control how submitted answers are reflected in the assistant message content. 
 - Python tool: Execute using a bash login shell for consistency of Python versions across `bash()` and `python()` tools.
 - Task display: Realtime display of events that occur within tool calls and subtasks.

{inspect_ai-0.3.94 → inspect_ai-0.3.95}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: inspect_ai
-Version: 0.3.94
+Version: 0.3.95
 Summary: Framework for large language model evaluations
 Author: UK AI Security Institute
 License: MIT License
@@ -32,6 +32,8 @@ Requires-Dist: httpx
 Requires-Dist: ijson>=3.2.0
 Requires-Dist: jsonlines>=3.0.0
 Requires-Dist: jsonpatch>=1.32
+Requires-Dist: jsonpath-ng>=1.7.0
+Requires-Dist: jsonref>=1.1.0
 Requires-Dist: jsonschema>3.1.1
 Requires-Dist: mmh3>3.1.0
 Requires-Dist: nest_asyncio
@@ -59,6 +61,7 @@ Requires-Dist: google-genai; extra == "dev"
 Requires-Dist: griffe; extra == "dev"
 Requires-Dist: groq; extra == "dev"
 Requires-Dist: ipython; extra == "dev"
+Requires-Dist: jsonpath-ng; extra == "dev"
 Requires-Dist: markdown; extra == "dev"
 Requires-Dist: mcp; extra == "dev"
 Requires-Dist: mistralai; extra == "dev"
@@ -66,9 +69,11 @@ Requires-Dist: moto[server]; extra == "dev"
 Requires-Dist: mypy; extra == "dev"
 Requires-Dist: nbformat; extra == "dev"
 Requires-Dist: openai; extra == "dev"
+Requires-Dist: pandas>=2.0.0; extra == "dev"
 Requires-Dist: panflute; extra == "dev"
 Requires-Dist: pip; extra == "dev"
 Requires-Dist: pre-commit; extra == "dev"
+Requires-Dist: pyarrow>=10.0.1; extra == "dev"
 Requires-Dist: pylint; extra == "dev"
 Requires-Dist: pytest; extra == "dev"
 Requires-Dist: pytest-asyncio; extra == "dev"
@@ -78,6 +83,8 @@ Requires-Dist: pytest-xdist; extra == "dev"
 Requires-Dist: ruff==0.9.6; extra == "dev"
 Requires-Dist: textual-dev>=0.86.2; extra == "dev"
 Requires-Dist: trio; extra == "dev"
+Requires-Dist: pandas-stubs; extra == "dev"
+Requires-Dist: pyarrow-stubs; extra == "dev"
 Requires-Dist: types-Markdown; extra == "dev"
 Requires-Dist: types-PyYAML; extra == "dev"
 Requires-Dist: types-beautifulsoup4; extra == "dev"

{inspect_ai-0.3.94 → inspect_ai-0.3.95}/docs/_quarto.yml

@@ -97,9 +97,13 @@ website:
                - agent-bridge.qmd
                - human-agent.qmd
 
-         - section: "Advanced"
+         - section: "Analysis"
            contents:
                - eval-logs.qmd
+               - dataframe.qmd
+             
+         - section: "Advanced"
+           contents:
                - eval-sets.qmd
                - text: "Errors & Limits"
                  href: errors-and-limits.qmd

inspect_ai-0.3.95/docs/dataframe.qmd

@@ -0,0 +1,431 @@
+---
+title: Log Dataframes
+---
+
+::: callout-note
+Log dataframe functions are currently in beta and are exported from the **inspect_ai.analysis.beta** module. The beta module will be preserved after final release so that code written against it now will continue to work after the beta.
+:::
+
+## Overview {#overview}
+
+```{=html}
+<style type="text/css">
+table a {
+    white-space: nowrap;
+}
+#overview table a {
+    text-decoration: none;
+    font-family: monospace;
+    font-size: 0.95rem;
+}
+</style>
+```
+
+Inspect eval logs have a hierarchical structure which is well suited to flexibly capturing all the elements of an evaluation. However, when analysing or visualising log data you will often want to transform logs into a [dataframe](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html). The **inspect_ai.analysis** module includes a variety of functions for extracting [Pandas](https://pandas.pydata.org/) dataframes from logs, including:
+
+| Function | Description |
+|---------------------------|---------------------------------------------|
+| [evals_df()](#evals) | Evaluation level data (e.g. task, model, scores, etc.). One row per log file. |
+| [samples_df()](#samples) | Sample level data (e.g. input, metadata, scores, errors, etc.) One row per sample, where each log file contains many samples. |
+| [messages_df()](#messages) | Message level data (e.g. role, content, etc.). One row per message, where each sample contains many messages. |
+
+Each function extracts a default set of columns, however you can tailor column reading to work in whatever way you need for your analysis. Extracted dataframes can either be denormalized (e.g. if you want to immediately summarise or plot them) or normalised (e.g. if you are importing them into a SQL database).
+
+Below we'll walk through a few examples, then after that provide more in-depth documentation on customising how dataframes are read for various scenarios.
+
+## Examples
+
+### Import Basics
+
+Use the `evals_df()` function to read a dataframe containing a row for each log file (note that we import from `inspect_ai.analysis.beta` since the dataframe functions are currently in beta):
+
+``` python
+from inspect_ai.analysis.beta import evals_df
+
+evals_df("logs")
+```
+
+``` default
+<class 'pandas.core.frame.DataFrame'>
+RangeIndex: 9 entries, 0 to 8
+Columns: 51 entries, eval_id to score_model_graded_qa_stderr
+```
+
+The default configuration for `evals_df()` reads 51 columns. While this is the default, column reading can be customized in variety of ways (covered below in [Columns](#columns)).
+
+Use the `samples_df()` function to read a dataframe with a record for each sample across a set of log files. For example, here we read all of the samples in the "logs" directory:
+
+``` python
+from inspect_ai.analysis.beta import samples_df
+
+samples_df("logs")
+```
+
+``` default
+<class 'pandas.core.frame.DataFrame'>
+RangeIndex: 408 entries, 0 to 407
+Columns: 13 entries, sample_id to retries
+```
+
+By default, `sample_df()` reads all of the columns in the `EvalSampleSummary` data structure (12 columns), along with the `eval_id` for linking back to the parent eval log file.
+
+### Column Groups
+
+When reading dataframes, there are a number of pre-built column groups you can use to read various subsets of columns. For example:
+
+``` python
+from inspect_ai.analysis.beta import (
+    EvalInfo, EvalModel, EvalResults, evals_df
+)
+
+evals_df(
+    logs="logs", 
+    columns=EvalInfo + EvalModel + EvalResults
+)
+```
+
+``` default
+<class 'pandas.core.frame.DataFrame'>
+RangeIndex: 9 entries, 0 to 8
+Columns: 23 entries, eval_id to score_headline_value
+```
+
+This dataframe has 23 columns rather than the 51 we saw when using the default `evals_df()` congiruation, reflecting the explicit columns groups specified.
+
+You can also use column groups to join columns for doing analysis or plotting. For example, here we include eval level data along with each sample:
+
+``` python
+from inspect_ai.analysis.beta import (
+    EvalInfo, EvalModel, SampleSummary, samples_df
+)
+
+samples_df(
+    logs="logs", 
+    columns=EvalInfo + EvalModel + SampleSummary
+)
+```
+
+``` default
+<class 'pandas.core.frame.DataFrame'>
+RangeIndex: 408 entries, 0 to 407
+Columns: 27 entries, sample_id to retries
+```
+
+This dataframe has 27 columns rather than than the 13 we saw for the default `samples_df()` behavior, reflecting the additional eval level columns. You can create your own column groups and definitions to further customise reading (see [Columns](#columns) for details).
+
+### Databases
+
+You can also read multiple dataframes and combine them into a relational database. Imported dataframes automatically include fields that can be used to join them (e.g. `eval_id` is in both the evals and samples tables).
+
+For example, here we read eval and sample level data from a log directory and import both tables into a DuckDb database:
+
+``` python
+import duckdb
+from inspect_ai.analysis.beta import evals_df, samples_df
+
+con = duckdb.connect()
+con.register('evals', evals_df("logs"))
+con.register('samples', samples_df("logs"))
+```
+
+We can now execute a query to find all samples generated using the `google` provider:
+
+``` python
+result = con.execute("""
+    SELECT * 
+    FROM evals e
+    JOIN samples s ON e.eval_id = s.eval_id
+    WHERE e.model LIKE 'google/%'
+""").fetchdf()
+```
+
+## Columns {#columns}
+
+The examples above all use built-in column specifications (e.g. `EvalModel`, `EvalResults`, `SampleSummary`, etc.). These specifications exist as a convenient starting point but can be replaced fully or partially by your own custom definitions.
+
+Column definitions specify how JSON data is mapped into dataframe columns, and are specified using subclasses of the `Column` class (e.g. `EvalColumn`, `SampleColumn`). For example, here is the definition of the built-in `EvalTask` column group:
+
+``` python
+EvalTask: list[Column] = [
+    EvalColumn("task_name", path="eval.task", required=True),
+    EvalColumn("task_version", path="eval.task_version", required=True),
+    EvalColumn("task_file", path="eval.task_file"),
+    EvalColumn("task_attribs", path="eval.task_attribs"),
+    EvalColumn("task_arg_*", path="eval.task_args"),
+    EvalColumn("solver", path="eval.solver"),
+    EvalColumn("solver_args", path="eval.solver_args"),
+    EvalColumn("sandbox_type", path="eval.sandbox.type"),
+    EvalColumn("sandbox_config", path="eval.sandbox.config"),
+]
+```
+
+Columns are defined with a `name`, a `path` (location within JSON to read their value from), and other options (e.g. `required`, `type`, etc.) . Column paths use [JSON Path](https://github.com/h2non/jsonpath-ng) expressions to indicate how they should be read from JSON.
+
+Many fields within eval logs are optional, and path expressions will automatically resolve to `None` when they include a missing field (unless the `required=True` option is specified).
+
+Here are are all of the options available for `Column` definitions:
+
+#### Column Options
+
+| Parameter | Type | Description |
+|-------------------|-------------------|----------------------------------|
+| `name` | `str` | Column name for dataframe. Can include wildcard characters (e.g. `task_arg_*`) for mapping dictionaries into multiple columns. |
+| `path` | `str` \| `JSONPath` | Path into JSON to extract the column from (uses [JSON Path](https://github.com/h2non/jsonpath-ng) expressions). Subclasses also implement path handlers that take e.g. an `EvalLog` and return a value. |
+| `required` | `bool` | Is the field required (i.e. should an error occur if it not found). |
+| `default` | `JsonValue` | Default value to yield if the field or its parents are not found in JSON. |
+| `type` | `Type[ColumnType]` | Validation check and directive to attempt to coerce the data into the specified `type`. Coercion from `str` to other types is done after interpreting the string using YAML (e.g. `"true"` -\> `True`). |
+| `value` | `Callable[[JsonValue], JsonValue]` | Function used to transform the value read from JSON into a value for the dataframe (e.g. converting a `list` to a comma-separated `str`). |
+
+Here are some examples that demonstrate the use of various options:
+
+``` python
+# required field
+EvalColumn("run_id", path="eval.run_id", required=True)
+
+# coerce field from int to str
+SampleColumn("id", path="id", required=True, type=str)
+
+# split metadata dict into multiple columns
+SampleColumn("metadata_*", path="metadata")
+
+# transform list[str] to str
+SampleColumn("target", path="target", value=list_as_str),
+```
+
+#### Column Merging
+
+If a column is name is repeated within a list of columns then the column definition encountered last is utilised. This makes it straightforward to override default column definitions. For example, here we override the behaviour of the default sample `metadata` columns (keeping it as JSON rather than splitting it into multiple columns):
+
+``` python
+ samples_df(
+     logs="logs",
+     columns=SampleSummary + [SampleColumn("metadata", path="metadata")]
+ )
+```
+
+#### Strict Mode
+
+By default, dataframes are read in `strict` mode, which means that if fields are missing or paths are invalid an error is raised and the import is aborted. You can optionally set `strict=False`, in which case importing will proceed and a tuple containing `pd.DataFrame` and a list of any errors encountered is returned. For example:
+
+``` python
+from inspect_ai.analysis.beta import evals_df
+
+evals, errors = evals_df("logs", strict=False)
+if len(errors) > 0:
+    print(errors)
+```
+
+### Evals {#evals}
+
+`EvalColumns` defines a default set of roughly 50 columns to read from the top level of an eval log. `EvalColumns` is in turn composed of several sets of column definitions that you can be used independently, these include:
+
+| Type | Description |
+|------------------------------------|------------------------------------|
+| [EvalInfo](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/analysis/beta/_dataframe/evals/columns.py#L51) | Descriptive information (e.g. created, tags, metadata, git commit, etc.) |
+| [EvalTask](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/analysis/beta/_dataframe/evals/columns.py#L64) | Task configuration (name, file, args, solver, etc.) |
+| [EvalModel](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/analysis/beta/_dataframe/evals/columns.py#L77) | Model name, args, generation config, etc. |
+| [EvalDataset](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/analysis/beta/_dataframe/evals/columns.py#L86) | Dataset name, location, sample ids, etc. |
+| [EvalConfig](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/analysis/beta/_dataframe/evals/columns.py#L95) | Epochs, approval, sample limits, etc. |
+| [EvalResults](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/analysis/beta/_dataframe/evals/columns.py#L106) | Status, errors, samples completed, headline metric. |
+| [EvalScores](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/analysis/beta/_dataframe/evals/columns.py#L118) | All scores and metrics broken into separate columns. |
+
+#### Custom Extraction
+
+The `EvalColumn` class provides an additional facility for reading values based on a callback function that takes an `EvalLog`. This function is specified as the `path` in the column definition.
+
+For example, here is the `path` function used to extract a simple dictionary of scores/metrics from the more complex `list[EvalScore]` type provided as `log.scores`:
+
+``` python
+def scores_dict(log: EvalLog) -> JsonValue:
+    if log.results is None:
+        return None
+    
+    metrics: JsonValue = [
+        {
+            score.name: {
+                metric.name: metric.value for metric in score.metrics.values()
+            }
+        }
+        for score in log.results.scores
+    ]
+    return metrics
+```
+
+Which is then used in the definition of the `EvalScores` column group as follows:
+
+``` python
+EvalScores: list[Column] = [
+    EvalColumn("score_*_*", path=scores_dict),
+]
+```
+
+### Samples {#samples}
+
+The `samples_df()` function can read from either sample summaries (`EvalSampleSummary`) or full sample records (`EvalSample`).
+
+By default, the `SampleSummary` column group is used, which reads only from summaries, resulting in considerably higher performance than reading full samples.
+
+``` python
+SampleSummary: list[Column] = [
+    SampleColumn("id", path="id", required=True, type=str),
+    SampleColumn("epoch", path="epoch", required=True),
+    SampleColumn("input", path="input", required=True, value=input_as_str),
+    SampleColumn("target", path="target", required=True, value=list_as_str),
+    SampleColumn("metadata_*", path="metadata"),
+    SampleColumn("score_*", path="scores", value=score_values),
+    SampleColumn("model_usage", path="model_usage"),
+    SampleColumn("total_time", path="total_time"),
+    SampleColumn("working_time", path="total_time"),
+    SampleColumn("error", path="error"),
+    SampleColumn("limit", path="limit"),
+    SampleColumn("retries", path="retries"),
+]
+```
+
+If you want to read all of the messages contained in a sample into a string column, use the `SampleMessages` column group. For example, here we read the summary field and the messages:
+
+``` python
+from inspect_ai.analysis.beta import (
+    SampleMessages, SampleSummary, samples_df
+)
+
+samples_df(
+    logs="logs", 
+    columns = SampleSummary + SampleMessages
+)
+```
+
+Note that reading `SampleMessages` requires reading full sample content, so will take considerably longer than reading only summaries.
+
+#### Full Samples
+
+`SampleColumn` will automatically determine whether it is referencing a field that requires a full sample read (for example, `messages` or `store`). There are five fields in sample summaries that have reduced footprint in the summary (`input`, `metadata`, and `scores`, `error`, and `limit`). For these, fields specify `full=True` to force reading from the full sample record. For example:
+
+``` python
+SampleColumn("limit_type", path="limit.type", full=True)
+SampleColumn("limit_value", path="limit.limit", full=True)
+```
+
+#### Custom Extraction
+
+As with `EvalColumn`, you can also extract data from a sample using a callback function passed as the `path`:
+
+``` python
+def model_reasoning_tokens(summary: EvalSampleSummary) -> JsonValue:
+    ## extract reasoning tokens from summary.model_usage
+
+SampleColumn("model_reasoning_tokens", path=model_reasoning_tokens)
+```
+
+::: {.callout-note appearance="simple"}
+Sample summaries were enhanced in version 0.3.93 (May 1, 2025) to include the `metadata`, `model_usage`, `total_time`, `working_time`, and `retries` fields. If you need to read any of these values you can update older logs with the new fields by round-tripping them through `inspect log convert`. For example:
+
+``` bash
+$ inspect log convert ./logs --to eval --output-dir ./logs-amended
+```
+:::
+
+### Messages {#messages}
+
+The `messages_df()` function enables reading message level data from a set of eval logs. Each row corresponds to a message, and includes a `sample_id` and `event_id` for linking back to its parents.
+
+The `messages_df()` function takes a `filter` parameter which can either be a list of `role` designations or a function that performs filtering. For example:
+
+``` python
+assistant_messages = messages_df("logs", filter=["assistant"])
+```
+
+#### Default Columns
+
+The default `MessageColumns` includes `MessageContent` and `MessageToolCalls`:
+
+``` python
+MessageContent: list[Column] = [
+    MessageColumn("role", path="role", required=True),
+    MessageColumn("content", path=message_text),
+    MessageColumn("source", path="source"),
+]
+
+MessageToolCalls: list[Column] = [
+    MessageColumn("tool_calls", path=message_tool_calls),
+    MessageColumn("tool_call_id", path="tool_call_id"),
+    MessageColumn("tool_call_function", path="function"),
+    MessageColumn("tool_call_error", path="error.message"),
+]
+
+MessageColumns: list[Column] = MessageContent + MessageToolCalls
+```
+
+#### Custom Extraction
+
+Two of the fields above are resolved using custom extraction functions (`content` and `tool_calls`). Here is the source code for those functions:
+
+``` python
+def message_text(message: ChatMessage) -> str:
+    return message.text
+
+
+def message_tool_calls(message: ChatMessage) -> str | None:
+    if isinstance(message, ChatMessageAssistant) and message.tool_calls is not None:
+        tool_calls = "\n".join(
+            [
+                format_function_call(
+                    tool_call.function, tool_call.arguments, width=1000
+                )
+                for tool_call in message.tool_calls
+            ]
+        )
+        return tool_calls
+    else:
+        return None
+```
+
+### Custom
+
+You can create custom column types that extract data based on additional parameters. For example, imagine you want to write a set of extraction functions that are passed a `ReportConfig` and an `EvalLog` (the report configuration might specify scores to extract, normalisation constraints, etc.)
+
+Here we define a new `ReportColumn` class that derives from `EvalColumn`:
+
+``` python
+import functools
+from typing import Callable
+from pydantic import BaseModel, JsonValue
+
+from inspect_ai.log import EvalLog
+from inspect_ai.analysis.beta import EvalColumn
+
+class ReportConfig(BaseModel):
+    # config fields
+    ...
+
+class ReportColumn(EvalColumn):
+    def __init__(
+        self,
+        name: str,
+        config: ReportConfig,
+        extract: Callable[[ReportConfig, EvalLog], JsonValue],
+        *,
+        required: bool = False,
+    ) -> None:
+        super().__init__(
+            name=name,
+            path=functools.partial(extract, config),
+            required=required,
+        )
+```
+
+The key here is using [functools.partial](https://www.geeksforgeeks.org/partial-functions-python/) to adapt the function that takes `config` and `log` into a function that takes `log` (which is what the `EvalColumn` class works with).
+
+We can now create extraction functions that take a `ReportConfig` and an `EvalLog` and pass them to `ReportColumn`:
+
+``` python
+# read dict scores from log according to config
+def read_scores(config: ReportConfig, log: EvalLog) -> JsonValue:
+    ...
+
+# config for a given report
+config = ReportConfig(...)
+
+# column that reads scores from log based on config
+ReportColumn("score_*", config, read_scores)
+```

{inspect_ai-0.3.94 → inspect_ai-0.3.95}/docs/eval-logs.qmd

@@ -1,5 +1,5 @@
 ---
-title: Eval Logs
+title: Log Files
 ---
 
 ## Overview

{inspect_ai-0.3.94 → inspect_ai-0.3.95}/docs/index.qmd

@@ -239,9 +239,14 @@ These sections describe how to create agent evaluations with Inspect:
 
 -   [Human Agent](human-agent.qmd) is a solver that enables human baselining on computing tasks.
 
+These sections outline how to analyze data generated from evaluations:
+
+-   [Eval Logs](eval-logs.qmd) explores log viewing, log file formats, and the Python API for reading log files.
+
+-   [Data Frames](dataframe.qmd) documents the APIs available for extracting dataframes of evals, samples, messages, and events from log files.
+
 These sections discuss more advanced features and workflows. You don't need to review them at the outset, but be sure to revisit them as you get more comfortable with the basics.
 
--   [Eval Logs](eval-logs.qmd) explores how to get the most out of evaluation logs for developing, debugging, and analyzing evaluations.
 
 -   [Eval Sets](eval-sets.qmd) covers Inspect's features for describing, running, and analysing larger sets of evaluation tasks.
 

{inspect_ai-0.3.94 → inspect_ai-0.3.95}/docs/llms.txt

@@ -35,6 +35,8 @@
 - [Human Agent](https://inspect.aisi.org.uk/human-agent.html.md): This article describes the `human_cli()` agent which enables human baselining for computing tasks.
 
 - [Eval Logs](https://inspect.aisi.org.uk/eval-logs.html.md): Explores how to get the most out of evaluation logs for developing, debugging, and analyzing evaluations.
+- [Data Frames](https://inspect.aisi.org.uk/dataframe.html.md) documents the APIs available for extracting dataframes of evals, samples, messages, and events from log files.
+
 - [Eval Sets](https://inspect.aisi.org.uk/eval-sets.html.md): Covers Inspect's features for describing, running, and analysing larger sets of evaluation tasks.
 - [Errors and Limits](https://inspect.aisi.org.uk/errors-and-limits.html.md): This article covers various techniques for dealing with unexpected errors and setting limits on evaluation tasks and samples, including retrying failed evaluations, establishing a threshold of samples to tolerate errors for before failing an evaluation, and setting a maximum number of messages, tokens, or elapsed seconds in a sample before forcing the solver to give up.
 - [Typing](https://inspect.aisi.org.uk/typing.html.md): Provides guidance on using static type checking with Inspect, including creating typed interfaces to untyped storage (i.e. sample metadata and store).
@@ -54,6 +56,7 @@
 - [inspect_ai.dataset](https://inspect.aisi.org.uk/reference/inspect_ai.dataset.html.md) describes the types used to read and manipulate datasets and samples.
 - [inspect_ai.approval](https://inspect.aisi.org.uk/reference/inspect_ai.approval.html.md) covers using built in approvers as well as the types used to create custom approvers and approval policies.
 - [inspect_ai.log](https://inspect.aisi.org.uk/reference/inspect_ai.log.html.md) describes the types used to list, read, write, and traverse the contents of eval log files.
+- [inspect_ai.analysis](https://inspect.aisi.org.uk/reference/inspect_ai.analysis.html.md) covers the Python API for reading logs into dataframes for analysis.
 - [inspect_ai.util](https://inspect.aisi.org.uk/reference/inspect_ai.util.html.md) covers various utility functions for concurrency, sandboxes, the store, and more.
 
 ## Reference: Command Line 

{inspect_ai-0.3.94 → inspect_ai-0.3.95}/docs/log-viewer.qmd

@@ -10,7 +10,7 @@ Inspect View provides a convenient way to visualize evaluation logs, including d
 
 Below we'll describe how to get the most out of using Inspect View.
 
-Note that this section covers *interactively* exploring log files. You can also use the `EvalLog` API to compute on log files (e.g. to compare across runs or to more systematically traverse results). See the section on [Eval Logs](#sec-eval-logs) to learn more about how to process log files with code.
+Note that this section covers *interactively* exploring log files. You can also use the `EvalLog` API to compute on log files (e.g. to compare across runs or to more systematically traverse results). See the sections on [Eval Logs](#sec-eval-logs) and [Data Frames](dataframe.qmd) to learn more about how to process log files with code.
 
 ## VS Code Extension
 

{inspect_ai-0.3.94 → inspect_ai-0.3.95}/docs/reference/_sidebar.yml

@@ -459,6 +459,55 @@ website:
           href: reference/inspect_ai.log.qmd#spanendevent
         - text: SubtaskEvent
           href: reference/inspect_ai.log.qmd#subtaskevent
+      - section: inspect_ai.analysis
+        href: reference/inspect_ai.analysis.qmd
+        contents:
+        - text: evals_df
+          href: reference/inspect_ai.analysis.qmd#evals_df
+        - text: EvalColumn
+          href: reference/inspect_ai.analysis.qmd#evalcolumn
+        - text: EvalColumns
+          href: reference/inspect_ai.analysis.qmd#evalcolumns
+        - text: EvalInfo
+          href: reference/inspect_ai.analysis.qmd#evalinfo
+        - text: EvalTask
+          href: reference/inspect_ai.analysis.qmd#evaltask
+        - text: EvalModel
+          href: reference/inspect_ai.analysis.qmd#evalmodel
+        - text: EvalConfig
+          href: reference/inspect_ai.analysis.qmd#evalconfig
+        - text: EvalResults
+          href: reference/inspect_ai.analysis.qmd#evalresults
+        - text: EvalScores
+          href: reference/inspect_ai.analysis.qmd#evalscores
+        - text: samples_df
+          href: reference/inspect_ai.analysis.qmd#samples_df
+        - text: SampleColumn
+          href: reference/inspect_ai.analysis.qmd#samplecolumn
+        - text: SampleSummary
+          href: reference/inspect_ai.analysis.qmd#samplesummary
+        - text: SampleMessages
+          href: reference/inspect_ai.analysis.qmd#samplemessages
+        - text: messages_df
+          href: reference/inspect_ai.analysis.qmd#messages_df
+        - text: MessageFilter
+          href: reference/inspect_ai.analysis.qmd#messagefilter
+        - text: MessageColumn
+          href: reference/inspect_ai.analysis.qmd#messagecolumn
+        - text: MessageContent
+          href: reference/inspect_ai.analysis.qmd#messagecontent
+        - text: MessageToolCalls
+          href: reference/inspect_ai.analysis.qmd#messagetoolcalls
+        - text: MessageColumns
+          href: reference/inspect_ai.analysis.qmd#messagecolumns
+        - text: Column
+          href: reference/inspect_ai.analysis.qmd#column
+        - text: ColumnType
+          href: reference/inspect_ai.analysis.qmd#columntype
+        - text: ColumnError
+          href: reference/inspect_ai.analysis.qmd#columnerror
+        - text: ColumnErrors
+          href: reference/inspect_ai.analysis.qmd#columnerrors
       - section: inspect_ai.util
         href: reference/inspect_ai.util.qmd
         contents:

{inspect_ai-0.3.94 → inspect_ai-0.3.95}/docs/reference/filter/sidebar.py

@@ -20,6 +20,7 @@ reference_docs = ["reference/inspect_ai.qmd"] + [
         "dataset.qmd",
         "approval.qmd",
         "log.qmd",
+        "analysis.qmd",
         "util.qmd",
     ]
 ]

{inspect_ai-0.3.94 → inspect_ai-0.3.95}/docs/reference/index.qmd

@@ -18,6 +18,7 @@ toc-depth: 4
 | [inspect_ai.dataset](inspect_ai.dataset.qmd) | Reading samples from datasets. |
 | [inspect_ai.approval](inspect_ai.approval.qmd) | Approvers and approval policies. |
 | [inspect_ai.log](inspect_ai.log.qmd) | List, read, write, and analyse logs. |
+| [inspect_ai.analysis](inspect_ai.analysis.qmd) | Data frames for analysis. |
 | [inspect_ai.util](inspect_ai.util.qmd) | Miscellaneous utility functions. |
 
 : {.borderless}