inspect-ai 0.3.65__tar.gz → 0.3.67__tar.gz

{inspect_ai-0.3.65 → inspect_ai-0.3.67}/.pre-commit-config.yaml

@@ -1,8 +1,5 @@
 # See https://pre-commit.com for more information
 # See https://pre-commit.com/hooks.html for more hooks
-# This should be the _latest_ version of python supported by us
-default_language_version:
-  python: python3.11
 repos:
   - repo: https://github.com/astral-sh/ruff-pre-commit
     rev: v0.9.5

{inspect_ai-0.3.65 → inspect_ai-0.3.67}/CHANGELOG.md

@@ -1,3 +1,22 @@
+## v0.3.67 (18 February 2025)
+
+- Memoize calls to `get_model()` so that model instances with the same parameters are cached and re-used (pass `memoize=False` to disable).
+- Async context manager for `Model` class for optional scoped usage of model clients.
+- New `assistant_message()` solver.
+- Prompt templates: Ignore template placeholders that don't map to passed parameters in `prompt_template()`, and system/user/assistnat solvers.
+- Google: Handle system messages with content lists and input with system but no user messages.
+- Google: Ensure that a completion choice is provided even when none are returned by the service.
+- Inspect View: Improve the display of subtasks with no inputs or events.
+- Inspect View: Fix transcript display of phantom subtask or other phantom events.
+- Inspect View: Fix formatting issues in sample error display
+- Bugfix: Raise error for empty dataset (rather than providing a dummy sample).
+- Bugfix: Specify markup=False for textual static controls (stricter parser in textual 2.0 leading to exceptions).
+- Bugfix: Temporarily pin to textual==1.0.0 while they chase all of their regressions in 2.0
+
+## v0.3.66 (17 February 2025)
+
+- Docker: Correct compose file generation for Dockerfiles w/ custom stem or extension.
+- Escape brackets when rendering task config (another textual 2.0 fix)
 
 ## v0.3.65 (16 February 2025)
 

{inspect_ai-0.3.65 → inspect_ai-0.3.67}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: inspect_ai
-Version: 0.3.65
+Version: 0.3.67
 Summary: Framework for large language model evaluations
 Author: UK AI Security Institute
 License: MIT License
@@ -45,7 +45,7 @@ Requires-Dist: s3fs>=2023
 Requires-Dist: semver>=3.0.0
 Requires-Dist: shortuuid
 Requires-Dist: tenacity
-Requires-Dist: textual>=0.86.2
+Requires-Dist: textual==1.0.0
 Requires-Dist: typing_extensions>=4.9.0
 Requires-Dist: zipp>=3.19.1
 Provides-Extra: dev
@@ -63,6 +63,7 @@ Requires-Dist: moto[server]; extra == "dev"
 Requires-Dist: mypy; extra == "dev"
 Requires-Dist: nbformat; extra == "dev"
 Requires-Dist: openai; extra == "dev"
+Requires-Dist: pip; extra == "dev"
 Requires-Dist: pre-commit; extra == "dev"
 Requires-Dist: pylint; extra == "dev"
 Requires-Dist: pytest; extra == "dev"

{inspect_ai-0.3.65 → inspect_ai-0.3.67}/docs/agents.qmd

@@ -4,7 +4,7 @@ title: Agent Basics
 
 ## Overview
 
-Agents combine planning, memory, and tool usage to pursue more complex, longer horizon tasks (e.g. a [Capture the Flag](https://en.wikipedia.org/wiki/Capture_the_flag_(cybersecurity)) challenge). Agents are an area of active research, and many schemes for implementing them have been developed, including [AutoGPT](https://arxiv.org/abs/2306.02224), [ReAct](https://arxiv.org/pdf/2303.11366.pdf), and [Reflexion](https://arxiv.org/pdf/2303.11366.pdf).
+Agents combine planning, memory, and tool usage to pursue more complex, longer horizon tasks (e.g. a [Capture the Flag](https://en.wikipedia.org/wiki/Capture_the_flag_(cybersecurity)) challenge). Agents are an area of active research, and many schemes for implementing them have been developed, including [AutoGPT](https://arxiv.org/abs/2306.02224), [ReAct](https://arxiv.org/abs/2210.03629), and [Reflexion](https://arxiv.org/pdf/2303.11366.pdf).
 
 An agent isn't a special construct within Inspect, it's merely a solver that includes tool use and calls `generate()` internally to interact with the model.
 

{inspect_ai-0.3.65 → inspect_ai-0.3.67}/docs/models.qmd

@@ -68,6 +68,50 @@ Inspect uses an asynchronous architecture to run task samples in parallel. If yo
 
 By default, Inspect uses a `max_connections` value of 10. You can increase this consistent with your account limits. If you are experiencing rate-limit errors you will need to experiment with the `max_connections` option to find the optimal value that keeps you under the rate limit (the section on [Parallelism](parallelism.qmd) includes additional documentation on how to do this).
 
+## Model API
+
+The `--model` which is set for an evaluation is automatically used by the `generate()` solver, as well as for other solvers and scorers built to use the currently evaluated model. If you are implementing a `Solver` or `Scorer` and want to use the currently evaluated model, call `get_model()` with no arguments:
+
+```python
+from inspect_ai.model import get_model
+
+model = get_model()
+response = await model.generate("Say hello")
+```
+
+If you want to use other models in your solvers and scorers, call `get_model()` with an alternate model name, along with optional generation config. For example:
+
+```python
+model = get_model("openai/gpt-4o")
+
+model = get_model(
+    "openai/gpt-4o",
+    config=GenerateConfig(temperature=0.9)
+)
+```
+
+You can also pass provider specific parameters as additional arguments to `get_model()`. For example:
+
+```python
+model = get_model("hf/openai-community/gpt2", device="cuda:0")
+```
+
+### Model Caching
+
+By default, calls to `get_model()` are memoized, meaning that calls with identical parameters resolve to a cached version of the model. You can disable this by passing `memoize=False`:
+
+```python
+model = get_model("openai/gpt-4o", memoize=False)
+```
+
+Finally, if you prefer to create and fully close model clients at their place of use, you can use the async context manager built in to the `Model` class. For example:
+
+```python
+async with get_model("openai/gpt-4o") as model:
+    response = await model.generate("Say hello")
+```
+
+
 ## Learning More
 
 - [Providers](providers.qmd) covers usage details and available options for the various supported providers.

{inspect_ai-0.3.65 → inspect_ai-0.3.67}/docs/reference/inspect_ai.solver.qmd

@@ -12,6 +12,7 @@ title: "inspect_ai.solver"
 ### prompt_template
 ### system_message
 ### user_message
+### assistant_message
 ### chain_of_thought
 ### self_critique
 ### multiple_choice

{inspect_ai-0.3.65 → inspect_ai-0.3.67}/docs/scorers.qmd

@@ -637,14 +637,17 @@ You can also add your own reducer with `@score_reducer` decorated functions. Her
 ``` python
 import statistics
 
-from inspect_ai.scorer import Score, ScoreReducer, score_reducer
+from inspect_ai.scorer import (
+    Score, ScoreReducer, score_reducer, value_to_float
+)
 
 @score_reducer(name="mean")
 def mean_score() -> ScoreReducer:
-    def reduce(scores: list[SampleScore]) -> Score:
-        """Compute a mean value of all scores."""
+    to_float = value_to_float()
 
-        values = [float(score.score.value) for score in scores]
+    def reduce(scores: list[Score]) -> Score:
+        """Compute a mean value of all scores."""
+        values = [to_float(score.value) for score in scores]
         mean_value = statistics.mean(values)
 
         return Score(value=mean_value)

{inspect_ai-0.3.65 → inspect_ai-0.3.67}/pyproject.toml

@@ -125,6 +125,7 @@ dev = [
     "mypy",
     "nbformat",
     "openai",
+    "pip",
     "pre-commit",
     "pylint",
     "pytest",

{inspect_ai-0.3.65 → inspect_ai-0.3.67}/requirements.txt

@@ -23,6 +23,6 @@ s3fs>=2023
 semver>=3.0.0
 shortuuid
 tenacity
-textual>=0.86.2
+textual==1.0.0
 typing_extensions>=4.9.0
 zipp>=3.19.1 # not directly required, pinned by Snyk to avoid a vulnerability

{inspect_ai-0.3.65 → inspect_ai-0.3.67}/src/inspect_ai/_display/core/config.py

@@ -35,6 +35,10 @@ def task_config(
             value = [str(v) for v in value]
             config_print.append(f"{name}: {','.join(value)}")
         elif name not in ["limit", "model"]:
+            if isinstance(value, list):
+                value = ",".join([str(v) for v in value])
+            if isinstance(value, str):
+                value = value.replace("[", "\\[")
             config_print.append(f"{name}: {value}")
     values = ", ".join(config_print)
     if values:

{inspect_ai-0.3.65 → inspect_ai-0.3.67}/src/inspect_ai/_display/textual/app.py

@@ -1,13 +1,21 @@
 import asyncio
 import contextlib
 from asyncio import CancelledError
-from typing import Any, AsyncIterator, ClassVar, Coroutine, Generic, Iterator, cast
+from typing import (
+    Any,
+    AsyncIterator,
+    ClassVar,
+    Coroutine,
+    Generic,
+    Iterator,
+    cast,
+)
 
 import rich
 from rich.console import Console
+from rich.text import Text
 from textual.app import App, ComposeResult
 from textual.binding import Binding, BindingType
-from textual.content import Content
 from textual.css.query import NoMatches
 from textual.events import Print
 from textual.widget import Widget
@@ -308,9 +316,9 @@ class TaskScreenApp(App[TR]):
 
         def set_unread(unread: int | None) -> None:
             if unread is not None:
-                console_tab.label = Content.from_rich_text(f"Console ({unread}")
+                console_tab.label = Text(f"Console ({unread}")
             else:
-                console_tab.label = Content.from_rich_text("Console")
+                console_tab.label = Text("Console")
 
         self.watch(console_view, "unread", set_unread)
 
@@ -377,7 +385,7 @@ class TaskScreenApp(App[TR]):
         def set_title(self, title: str) -> None:
             tabs = self.app.query_one(TabbedContent)
             tab = tabs.get_tab(self.tab_id)
-            tab.label = Content.from_rich_text(title)
+            tab.label = Text(title)
 
         def activate(self) -> None:
             # show the tab

{inspect_ai-0.3.65 → inspect_ai-0.3.67}/src/inspect_ai/_display/textual/widgets/footer.py

@@ -26,8 +26,8 @@ class AppFooter(Widget):
     right: reactive[RenderableType] = reactive("")
 
     def compose(self) -> ComposeResult:
-        yield Static(id="footer-left")
-        yield Static(id="footer-right")
+        yield Static(id="footer-left", markup=False)
+        yield Static(id="footer-right", markup=False)
 
     def watch_left(self, new_left: RenderableType) -> None:
         footer_left = cast(Static, self.query_one("#footer-left"))

{inspect_ai-0.3.65 → inspect_ai-0.3.67}/src/inspect_ai/_display/textual/widgets/sandbox.py

@@ -38,6 +38,6 @@ class SandboxView(Vertical):
         with Horizontal():
             yield Static("", classes="indent" if self.sandbox_name else "no_indent")
             with Vertical():
-                yield Static(self.connection.command)
+                yield Static(self.connection.command, markup=False)
                 if self.connection.ports:
                     yield PortMappingsView(self.connection.ports)

{inspect_ai-0.3.65 → inspect_ai-0.3.67}/src/inspect_ai/_display/textual/widgets/task_detail.py

@@ -233,20 +233,22 @@ class TaskMetrics(Widget):
         for metric in self.metrics:
             # Add the value static but keep it around
             # for future updates
-            self.value_widgets[metric.name] = Static(self._metric_value(metric.value))
+            self.value_widgets[metric.name] = Static(
+                self._metric_value(metric.value), markup=False
+            )
 
-            grid.mount(Static(metric.name))
+            grid.mount(Static(metric.name, markup=False))
             grid.mount(self.value_widgets[metric.name])
 
     def _title(self) -> Widget:
         if self.scorer is None:
             return Static("")
         elif self.reducer is None:
-            return Static(self.scorer)
+            return Static(self.scorer, markup=False)
         else:
             return Horizontal(
-                Static(self.scorer, classes="scorer"),
-                Static(f"({self.reducer})", classes="reducer"),
+                Static(self.scorer, classes="scorer", markup=False),
+                Static(f"({self.reducer})", classes="reducer", markup=False),
             )
 
     def _metric_value(self, val: float) -> str:

{inspect_ai-0.3.65 → inspect_ai-0.3.67}/src/inspect_ai/_display/textual/widgets/tasks.py

@@ -128,8 +128,8 @@ class TasksView(Container):
             progress_view.update_count_width(self.sample_count_width)
 
     def compose(self) -> ComposeResult:
-        yield Static(id="tasks-config")
-        yield Static(id="tasks-targets")
+        yield Static(id="tasks-config", markup=False)
+        yield Static(id="tasks-targets", markup=False)
         yield ScrollableContainer(id="tasks-progress")
 
     def watch_config(self, new_config: RenderableType) -> None:
@@ -191,8 +191,8 @@ class TaskProgressView(Widget):
         self.model_name_width = model_name_width
 
         self.progress_bar = ProgressBar(total=task.profile.steps, show_eta=False)
-        self.count_display = Static()
-        self.metrics_display = Static(id="task-metrics")
+        self.count_display = Static(markup=False)
+        self.metrics_display = Static(id="task-metrics", markup=False)
         self.task_progress = TaskProgress(self.progress_bar)
 
         self.toggle = Toggle()
@@ -211,10 +211,12 @@ class TaskProgressView(Widget):
         yield (self.toggle if self.display_metrics else Static())
         yield TaskStatusIcon()
         yield Static(
-            progress_description(self.t.profile, self.description_width, pad=True)
+            progress_description(self.t.profile, self.description_width, pad=True),
+            markup=False,
         )
         yield Static(
-            progress_model_name(self.t.profile.model, self.model_name_width, pad=True)
+            progress_model_name(self.t.profile.model, self.model_name_width, pad=True),
+            markup=False,
         )
         yield self.progress_bar
         yield self.count_display

{inspect_ai-0.3.65 → inspect_ai-0.3.67}/src/inspect_ai/_display/textual/widgets/transcript.py

@@ -117,7 +117,7 @@ class TranscriptView(ScrollableContainer):
                         )
                         if isinstance(d.content, Markdown):
                             set_transcript_markdown_options(d.content)
-                        widgets.append(Static(d.content))
+                        widgets.append(Static(d.content, markup=False))
                         widgets.append(Static(Text(" ")))
         return widgets
 

{inspect_ai-0.3.65 → inspect_ai-0.3.67}/src/inspect_ai/_eval/task/run.py

@@ -551,9 +551,11 @@ async def task_run_sample(
     # helper to handle exceptions (will throw if we've exceeded the limit)
     def handle_error(ex: BaseException) -> tuple[EvalError, BaseException | None]:
         err = sample_error(ex)
-        py_logger.warning(
-            f"Sample error (id: {sample.id}, epoch: {state.epoch}): {exception_message(ex)})"
-        )
+        # if we aren't raising the error then print a warning
+        if err[1] is None:
+            py_logger.warning(
+                f"Sample error (id: {sample.id}, epoch: {state.epoch}): {exception_message(ex)})"
+            )
         transcript()._event(ErrorEvent(error=err[0]))
         return err
 

{inspect_ai-0.3.65 → inspect_ai-0.3.67}/src/inspect_ai/_eval/task/task.py

@@ -331,7 +331,15 @@ def resolve_epochs(epochs: int | Epochs | None) -> Epochs | None:
 
 
 def resolve_dataset(dataset: Dataset | Sequence[Sample] | None) -> Dataset:
-    dataset = dataset or [Sample(input="prompt")]
+    # this is a convenience for tests that don't want to define a dummy sample
+    if dataset is None:
+        dataset = [Sample(input="prompt")]
+
+    # raise error if the dataset is empty
+    if len(dataset) == 0:
+        raise ValueError("The specified dataset is empty (has no samples)")
+
+    # resolve sequence to dataset if necessary
     return dataset if isinstance(dataset, Dataset) else MemoryDataset(list(dataset))
 
 

inspect_ai-0.3.67/src/inspect_ai/_util/format.py

@@ -0,0 +1,93 @@
+import pprint
+from string import Formatter
+from textwrap import indent
+from typing import Any
+
+
+def format_function_call(
+    func_name: str, args_dict: dict[str, Any], indent_spaces: int = 4, width: int = 80
+) -> str:
+    formatted_args = []
+    for key, value in args_dict.items():
+        formatted_value = format_value(value, width)
+        formatted_args.append(f"{key}={formatted_value}")
+
+    args_str = ", ".join(formatted_args)
+
+    if len(args_str) <= width - 1 - len(func_name) - 2:  # 2 for parentheses
+        return f"{func_name}({args_str})"
+    else:
+        indented_args = indent(",\n".join(formatted_args), " " * indent_spaces)
+        return f"{func_name}(\n{indented_args}\n)"
+
+
+def format_value(value: object, width: int) -> str:
+    if isinstance(value, str):
+        return f"'{value}'"
+    elif isinstance(value, list | tuple | dict):
+        return pprint.pformat(value, width=width)
+    return str(value)
+
+
+def format_progress_time(time: float, pad_hours: bool = True) -> str:
+    minutes, seconds = divmod(time, 60)
+    hours, minutes = divmod(minutes, 60)
+    hours_fmt = f"{hours:2.0f}" if pad_hours else f"{hours:.0f}"
+    return f"{hours_fmt}:{minutes:02.0f}:{seconds:02.0f}"
+
+
+def format_template(
+    template: str,
+    params: dict[str, Any],
+    skip_unknown: bool = True,
+) -> str:
+    """Format a template string, optionally preserving unknown placeholders.
+
+    Args:
+        template: A string containing {placeholders} to be formatted
+        params: Dictionary of parameters to substitute into the template
+        skip_unknown: If True, preserve unknown placeholders; if False, raise KeyError
+
+    Returns:
+        The formatted string with parameters substituted
+
+    Examples:
+        >>> format_template("Hello {name}!", {"name": "World"})
+        'Hello World!'
+        >>> format_template("Hello {name}!", {}, skip_unknown=True)
+        'Hello {name}!'
+    """
+
+    class SafeFormatter(Formatter):
+        def get_field(self, field_name: str, args: Any, kwargs: Any) -> Any:
+            try:
+                # Handle array indexing and nested attributes
+                first, rest = (
+                    field_name.split(".", 1)
+                    if "." in field_name
+                    else (field_name, None)
+                )
+                first = first.split("[")[0]  # Remove any array indexing for the check
+
+                if first not in params and skip_unknown:
+                    return "{" + field_name + "}", field_name
+
+                obj = params.get(first)
+                if obj is None and skip_unknown:
+                    return "{" + field_name + "}", field_name
+
+                return super().get_field(field_name, args, kwargs)
+            except (AttributeError, KeyError, IndexError) as e:
+                if skip_unknown:
+                    return "{" + field_name + "}", field_name
+                raise KeyError(f"Failed to format field '{field_name}'") from e
+
+        def format_field(self, value: Any, format_spec: str) -> Any:
+            try:
+                return super().format_field(value, format_spec)
+            except (ValueError, TypeError):
+                if skip_unknown:
+                    return "{" + str(value) + ":" + format_spec + "}"
+                raise
+
+    return SafeFormatter().format(template, **params)

{inspect_ai-0.3.65 → inspect_ai-0.3.67}/src/inspect_ai/_view/www/dist/assets/index.css

@@ -16346,22 +16346,24 @@ ul.jsondiffpatch-textdiff {
 ._noTop_14odp_27 {
   margin-top: 0;
 }
-._body_5y0hl_1 {
+._flatBody_gk2ju_1 {
   color: var(--bs-danger);
   display: grid;
-  grid-template-columns: 1fr;
-  align-content: align;
-  justify-items: center;
+  grid-template-columns: max-content max-content;
+  column-gap: 0.2em;
+  margin-top: 0.4rem;
 }
 
-._iconSmall_5y0hl_9 {
+._iconSmall_gk2ju_9 {
   font-size: var(--inspect-font-size-small);
   line-height: var(--inspect-font-size-small);
   height: var(--inspect-font-size-small);
 }
 
-._message_5y0hl_15 {
-  width: 300px;
+._lineBase_gk2ju_15 {
+  font-size: var(--inspect-font-size-base);
+  line-height: var(--inspect-font-size-base);
+  height: var(--inspect-font-size-base);
 }
 ._target_yamz4_1 {
   padding-left: 0;
@@ -19222,6 +19224,23 @@ span.ap-marker-container:hover span.ap-marker {
 .message-band-btn.warning {
   color: var(--bs-warning-text-emphasis);
 }
+._body_5y0hl_1 {
+  color: var(--bs-danger);
+  display: grid;
+  grid-template-columns: 1fr;
+  align-content: align;
+  justify-items: center;
+}
+
+._iconSmall_5y0hl_9 {
+  font-size: var(--inspect-font-size-small);
+  line-height: var(--inspect-font-size-small);
+  height: var(--inspect-font-size-small);
+}
+
+._message_5y0hl_15 {
+  width: 300px;
+}
 ._grid_1kcta_1 {
   display: grid;
   padding-top: 1em;
@@ -19585,16 +19604,17 @@ span.ap-marker-container:hover span.ap-marker {
   padding: 0 0.2em;
   justify-content: center;
 }
-._statusPanel_1fzh4_1 {
+._statusPanel_66f9o_1 {
   padding: 1em;
   margin-top: 0.5em;
   text-transform: uppercase;
   font-size: var(--inspect-font-size-smaller);
   display: grid;
   grid-template-columns: auto auto;
+  justify-content: end;
 }
 
-._statusIcon_1fzh4_10 {
+._statusIcon_66f9o_11 {
   font-size: var(--inspect-font-size-large);
   margin-right: 0.3em;
   margin-top: -0.1em;