inspect-ai 0.3.49__tar.gz → 0.3.51__tar.gz

{inspect_ai-0.3.49 → inspect_ai-0.3.51}/.pre-commit-config.yaml

@@ -5,7 +5,7 @@ default_language_version:
   python: python3.11
 repos:
 - repo: https://github.com/astral-sh/ruff-pre-commit
-  rev: v0.8.1
+  rev: v0.8.2
   hooks:
     # Run the linter.
     - id: ruff

{inspect_ai-0.3.49 → inspect_ai-0.3.51}/CHANGELOG.md

@@ -1,5 +1,40 @@
 # Changelog
 
+## v0.3.51 (13 December 2024)
+
+- Bugfix: Task display fails to load when no scorers are defined for a task.
+
+## v0.3.50 (12 December 2024)
+
+- Tools: Improved typing/schema support (unions, optional params, enums).
+- Tools: Added `append` argument to `use_tools()` for adding (rather than replacing) the currently available tools.
+- Docker sandbox: Streamed reads of stderr/stdout (enabling us to enforce output limits for read_file and exec at the source).
+- Sandbox API: Enable passing `BaseModel` types for sandbox `config` (formerly only a file path could be passed).
+- Task display: Show all task scores in realtime (expand task progress to see scores).
+- Task display: Show completed samples and align progress more closely to completed samples (as opposed to steps).
+- Task display: Show sample messages/tokens used (plus limits if specified).
+- Task display: Resolve issue where task display would lose mouse input after VS Code reload.
+- Datasets: Validate that all IDs in datasets are unique (as several downstream problems occur w/ duplicate IDs).
+- Inspect View: Fix issue with incorrectly displayed custom tool views.
+- Human approval: Use fullcreen display (makes approval UI async and enables rapid processing of approvals via the `Enter` key).
+- Added `input_panel()` API for adding custom panels to the fullscreen task display.
+- Log recorder: Methods are now async which will improve performance for fsspec filesystems with async implementations (e.g. S3)
+- Log recorder: Improve `.eval` log reading performance for remote filesystem (eaglery fetch log to local buffer).
+- Add `token_usage` property to `TaskState` which has current total tokens used across all calls to `generate()` (same value that is used for enforcing token limits).
+- Add `time` field to `ModelOutput` that records total time spent within call to ModelAPI `generate()`.
+- Web browser: Remove base64 images from web page contents (prevent filling up model context with large images).
+- Match scorer: If the target of a match isn’t numeric, ignore the numeric flag and instead use text matching (improved handling for percentages).
+- Hugging Face: Support for native HF tool calling for Llama, Mistral, Qwen, and others if they conform to various standard schemas.
+- Hugging Face: `tokenizer_call_args` dict to specify custom args during tokenization, such as `max_length` and `truncation`.
+- Azure AI: Fix schema validation error that occurred when model API returns `None` for `content`.
+- Display: Throttle updating of sample list based on number of samples.
+- Display: Add explicit 'ctrl+c' keybinding (as textual now disables this by default).
+- Bugfix: Correct rate limit error display when running in fullscreen mode.
+- Bugfix: `hf_dataset` now explicitly requires the `split` argument (previously, it would crash when not specified).
+- Bugfix: Prevent cascading textual error when an error occurs during task initialisation.
+- Bugfix: Correctly restore sample summaries from log file after abend.
+- Bugfix: Report errors that occur during task finalisation.
+  
 ## v0.3.49 (03 December 2024)
 
 - Logging: Only call CreateBucket on Amazon S3 when the bucket does not already exist.
@@ -95,7 +130,6 @@
 - Add optional `tool_call_id` param to `ModelOutput.for_tool_call()`.
 - Support all JSON and CSV dataset arguments in `file_dataset()` function.
 
-
 ## v0.3.42 (23 October 2024)
 
 - [ToolDef](https://inspect.ai-safety-institute.org.uk/tools.html#sec-dynamic-tools) class for dynamically creating tool definitions.

{inspect_ai-0.3.49/src/inspect_ai.egg-info → inspect_ai-0.3.51}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: inspect_ai
-Version: 0.3.49
+Version: 0.3.51
 Summary: Framework for large language model evaluations
 Author: UK AI Safety Institute
 License: MIT License
@@ -68,7 +68,7 @@ Requires-Dist: pytest-asyncio; extra == "dev"
 Requires-Dist: pytest-cov; extra == "dev"
 Requires-Dist: pytest-dotenv; extra == "dev"
 Requires-Dist: pytest-xdist; extra == "dev"
-Requires-Dist: ruff==0.8.1; extra == "dev"
+Requires-Dist: ruff==0.8.2; extra == "dev"
 Requires-Dist: textual-dev>=0.86.2; extra == "dev"
 Requires-Dist: types-PyYAML; extra == "dev"
 Requires-Dist: types-aiofiles; extra == "dev"

{inspect_ai-0.3.49 → inspect_ai-0.3.51}/docs/_quarto.yml

@@ -2,6 +2,8 @@ project:
    type: website
    resources: 
       - CNAME
+      - llms.txt
+   post-render: scripts/post-render.sh
   
 website:
    title: "Inspect"
@@ -96,6 +98,8 @@ format:
      toc-depth: 3
      number-sections: false
      code-annotations: select
+   gfm: 
+      output-ext: html.lmd
 
 
 execute: 

{inspect_ai-0.3.49 → inspect_ai-0.3.51}/docs/extensions.qmd

@@ -127,7 +127,7 @@ class PodmanSandboxEnvironment(SandboxEnvironment):
 
     @classmethod
     async def task_init(
-        cls, task_name: str, config: str | None
+        cls, task_name: str, config: SandboxEnvironmentConfigType | None
     ) -> None:
         ...
 
@@ -135,7 +135,7 @@ class PodmanSandboxEnvironment(SandboxEnvironment):
     async def sample_init(
         cls, 
         task_name: str, 
-        config: str | None, 
+        config: SandboxEnvironmentConfigType | None, 
         metadata: dict[str, str]
     ) -> dict[str, SandboxEnvironment]:
         ...
@@ -144,7 +144,7 @@ class PodmanSandboxEnvironment(SandboxEnvironment):
     async def sample_cleanup(
         cls,
         task_name: str,
-        config: str | None,
+        config: SandboxEnvironmentConfigType | None,
         environments: dict[str, SandboxEnvironment],
         interrupted: bool,
     ) -> None:
@@ -152,7 +152,10 @@ class PodmanSandboxEnvironment(SandboxEnvironment):
 
     @classmethod
     async def task_cleanup(
-        cls, task_name: str, config: str | None, cleanup: bool
+        cls,
+        task_name: str,
+        config: SandboxEnvironmentConfigType | None,
+        cleanup: bool,
     ) -> None:
        ...
 
@@ -253,6 +256,22 @@ Task(
 )
 ```
 
+Specialised configuration types which derive from Pydantic's `BaseModel` can also be passed as the `config` argument to `SandboxEnvironmentSpec`. Note: they must be hashable (i.e. `frozen=True`).
+
+``` python
+class PodmanSandboxEnvironmentConfig(BaseModel, frozen=True):
+    socket: str
+    runtime: str
+
+Task(
+    ...,
+    sandbox=SandboxEnvironmentSpec(
+        "podman",
+        PodmanSandboxEnvironmentConfig(socket="/podman-socket", runtime="crun"),
+    )
+)
+```
+
 ## Approvers {#sec-extensions-approvers}
 
 [Approvers](approval.qmd) enable you to create fine-grained policies for approving tool calls made by models. For example, the following are all supported:
@@ -399,4 +418,4 @@ myfs = "evaltools:MyFs"
 ```
 :::
 
-Once this package is installed, you'll be able to use `myfs://` with Inspect without any further registration.
+Once this package is installed, you'll be able to use `myfs://` with Inspect without any further registration.

inspect_ai-0.3.51/docs/llms.txt

@@ -0,0 +1,26 @@
+# Inspect AI
+
+> Inspect AI is a Python framework for large language model evaluations created by the [UK AI Safety Institute](https://aisi.gov.uk). Inspect provides many built-in components, including facilities for prompt engineering, tool usage, multi-turn dialog, and model graded evaluations. Extensions to Inspect (e.g. to support new elicitation and scoring techniques) can be provided by other Python packages.
+
+## Docs
+
+- [Tutorial](https://inspect.ai-safety-institute.org.uk/tutorial.html.md): Step-by-step walkthroughs of several basic examples of Inspect evaluations.
+- [Workflow](https://inspect.ai-safety-institute.org.uk/workflow.html.md): Covers the mechanics of running evaluations, including how to create evals in both scripts and notebooks, specifying configuration and options, how to parameterise tasks for different scenarios, and how to work with eval log files.
+- [Log Viewer](https://inspect.ai-safety-institute.org.uk/log-viewer.html.md): Goes into more depth on how to use Inspect View to develop and debug evaluations, including how to provide additional log metadata and how to integrate it with Python's standard logging module.
+- [VS Code](https://inspect.ai-safety-institute.org.uk/vscode.html.md) Provides documentation on using the Inspect VS Code Extension to run, tune, debug, and visualise evaluations.
+- [Solvers](https://inspect.ai-safety-institute.org.uk/solvers.html.md): Solvers are the heart of Inspect, and encompass prompt engineering and various other elicitation strategies. Here we cover using the built-in solvers and creating your own more sophisticated ones.
+- [Tools](https://inspect.ai-safety-institute.org.uk/tools.html.md): Tools provide a means of extending the capabilities of models by registering Python functions for them to call. This section describes how to create custom tools and use them in evaluations.
+- [Agents](https://inspect.ai-safety-institute.org.uk/agents.html.md): Agents combine planning, memory, and tool usage to pursue more complex, longer horizon tasks. This section describes how to build agent evaluations with Inspect.
+- [Agents API](https://inspect.ai-safety-institute.org.uk/agents-api.html.md): This article describes advanced Inspect APIs available for creating evaluations with agents.
+- [Scorers](https://inspect.ai-safety-institute.org.uk/scorers.html.md): Scorers evaluate the work of solvers and aggregate scores into metrics. Sophisticated evals often require custom scorers that use models to evaluate output. This section covers how to create them.
+- [Datasets](https://inspect.ai-safety-institute.org.uk/datasets.html.md): Datasets provide samples to evaluation tasks. This section illustrates how to adapt various data sources for use with Inspect, as well as how to include multi-modal data (images, etc.) in your datasets.
+- [Models](https://inspect.ai-safety-institute.org.uk/models.html.md): Models provide a uniform API for both evaluating a variety of large language models and using models within evaluations (e.g. for critique or grading).
+- [Eval Sets](https://inspect.ai-safety-institute.org.uk/eval-sets.html.md): Covers Inspect's features for describing, running, and analysing larger sets of evaluation tasks.
+- [Errors and Limits](https://inspect.ai-safety-institute.org.uk/errors-and-limits.html.md): This article covers various techniques for dealing with unexpected errors and setting limits on evaluation tasks and samples, including retrying failed evaluations, establishing a threshold of samples to tolerate errors for before failing an evaluation, and setting a maximum number of messages, tokens, or elapsed seconds in a sample before forcing the solver to give up.
+- [Caching](https://inspect.ai-safety-institute.org.uk/caching.html.md): Caching enables you to cache model output to reduce the number of API calls made, saving both time and expense.
+- [Parallelism](https://inspect.ai-safety-institute.org.uk/parallelism.html.md): Delves into how to obtain maximum performance for evaluations. Inspect uses a highly parallel async architecture---here we cover how to tune this parallelism (e.g to stay under API rate limits or to not overburden local compute) for optimal throughput.
+- [Interactivity](https://inspect.ai-safety-institute.org.uk/interactivity.html.md): Covers various ways to introduce user interaction into the implementation of tasks (for example, confirming consequential actions or prompting the model dynamically based on the trajectory of the evaluation).
+- [Approval](https://inspect.ai-safety-institute.org.uk/approval.html.md): Approvals enable you to create fine-grained policies for approving tool calls made by models.
+- [Eval Logs](https://inspect.ai-safety-institute.org.uk/eval-logs.html.md): Explores how to get the most out of evaluation logs for developing, debugging, and analyzing evaluations.
+- [Extensions](https://inspect.ai-safety-institute.org.uk/extensions.html.md) describes the various ways you can extend Inspect, including adding support for new Model APIs, tool execution environments, and storage platforms (for datasets, prompts, and logs).
+

{inspect_ai-0.3.49 → inspect_ai-0.3.51}/docs/models.qmd

@@ -46,7 +46,7 @@ To select a model for use in an evaluation task you specify it using a *model na
 | Ollama | `ollama/llama3` | [Ollama Models](https://ollama.com/library) |
 | llama-cpp-python | `llama-cpp-python/llama3` | [llama-cpp-python Models](https://llama-cpp-python.readthedocs.io/en/latest/#openai-compatible-web-server) |
 | TogetherAI | `together/google/gemma-7b-it` | [TogetherAI Models](https://docs.together.ai/docs/inference-models#chat-models) |
-| AWS Bedrock | `bedrock/meta.llama2-70b-chat-v1` | [AWS Bedrock Models](https://aws.amazon.com/bedrock/) |
+| AWS Bedrock | `bedrock/meta.llama2-70b-chat-v1` | [AWS Bedrock Models](https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.html) |
 | Azure AI | `azureai/azure-deployment-name` | [Azure AI Models](https://ai.azure.com/explore/models) |
 | Vertex | `vertex/gemini-1.5-flash` | [Google Models](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#supported_models) |
 | Groq | `groq/mixtral-8x7b-32768` | [Groq Models](https://console.groq.com/docs/models) |

inspect_ai-0.3.51/docs/scripts/post-render.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+
+files=("index" "tutorial" "workflow" "log-viewer" "vscode" "solvers" "tools" "agents" "agents-api" "scorers" "datasets" "models" "eval-sets" "errors-and-limits" "caching" "parallelism" "interactivity" "approval" "eval-logs" "extensions")
+
+llms_full="_site/llms-full.txt"
+rm -f "${llms_full}"
+for file in "${files[@]}"; do
+    path="_site/${file}.html"
+    mv "${path}.lmd" "${path}.md"
+    cat "${path}.md" >> "${llms_full}"
+    echo "" >> "${llms_full}"
+done
+

{inspect_ai-0.3.49 → inspect_ai-0.3.51}/docs/solvers.qmd

@@ -14,7 +14,7 @@ Solvers are the heart of Inspect evaluations and can serve a wide variety of pur
 5.  Multi-turn dialog
 6.  Running an agent scaffold
 
-Tasks have a single top-level solver that defines an execution plan. This solver could be implemented with arbitrary Python code (calling the model as required) or could consist of a set of other sovlers composed together. Solvers can therefore play two differnet roles: 
+Tasks have a single top-level solver that defines an execution plan. This solver could be implemented with arbitrary Python code (calling the model as required) or could consist of a set of other solvers composed together. Solvers can therefore play two differnet roles: 
 
 1. _Composite_ specifications for task execution; and
 
@@ -65,7 +65,7 @@ def theory_of_mind():
 
 Composite solvers by no means need to be implemented using chains. While chains are frequently used in more straightforward knowledge and reasoning evaluations, fully custom solver functions are often used for multi-turn dialog and agent evaluations.
 
-This section covers mostly solvers as components (both built in and creating your own). The [Agents](agents.qmd) section describes fully custom sovlers in more depth.
+This section covers mostly solvers as components (both built in and creating your own). The [Agents](agents.qmd) section describes fully custom solvers in more depth.
 
 ## Task States
 
@@ -463,4 +463,4 @@ Early termination might also occur if you specify the `message_limit` option and
 ``` python
 # could terminate early
 eval(my_task, message_limit = 10)
-```
+```

{inspect_ai-0.3.49 → inspect_ai-0.3.51}/docs/workflow.qmd

@@ -22,7 +22,7 @@ from inspect_ai.solver import (
 
 DEFAULT_PROMPT="{prompt}"
 
-from tree_of_thought imoprt TREE_PROMPT, generate_tree
+from tree_of_thought import TREE_PROMPT, generate_tree
 
 @solver 
 def critique():
@@ -395,4 +395,4 @@ $ inspect eval security/jeopardy
 $ inspect eval security/attack_defense 
 ```
 
-Inspect has lots of features aimed at running evaluation suites, including filtering tasks based on tags/metadata, recovering from partially completed suites (due to failed evals), and more. See the documentation on [Eval Sets](eval-sets.qmd) to learn more.
+Inspect has lots of features aimed at running evaluation suites, including filtering tasks based on tags/metadata, recovering from partially completed suites (due to failed evals), and more. See the documentation on [Eval Sets](eval-sets.qmd) to learn more.

{inspect_ai-0.3.49 → inspect_ai-0.3.51}/examples/approval/approval.py

@@ -224,4 +224,4 @@ def python_allowlist(
 
 if __name__ == "__main__":
     approval = (Path(__file__).parent / "approval.yaml").as_posix()
-    eval(approval_demo(), approval=approval, trace=True)
+    eval(approval_demo(), approval=approval)

{inspect_ai-0.3.49 → inspect_ai-0.3.51}/pyproject.toml

@@ -129,7 +129,7 @@ dev = [
     "pytest-cov",
     "pytest-dotenv",
     "pytest-xdist",
-    "ruff==0.8.1", # match version specified in .pre-commit-config.yaml
+    "ruff==0.8.2", # match version specified in .pre-commit-config.yaml
     "textual-dev>=0.86.2",
     "types-PyYAML",
     "types-aiofiles",

{inspect_ai-0.3.49 → inspect_ai-0.3.51}/src/inspect_ai/_cli/info.py

@@ -5,7 +5,7 @@ import click
 from inspect_ai import __version__
 from inspect_ai._util.constants import PKG_PATH
 from inspect_ai._view.server import resolve_header_only
-from inspect_ai.log._file import eval_log_json, read_eval_log
+from inspect_ai.log._file import eval_log_json_str, read_eval_log
 
 from .log import headers, schema, types
 
@@ -46,7 +46,7 @@ def log(path: str, header_only: int) -> None:
     header_only = resolve_header_only(path, header_only)
 
     log = read_eval_log(path, header_only=header_only)
-    print(eval_log_json(log))
+    print(eval_log_json_str(log))
 
 
 @info_command.command("log-file-headers", hidden=True)

{inspect_ai-0.3.49 → inspect_ai-0.3.51}/src/inspect_ai/_cli/log.py

@@ -14,7 +14,7 @@ from inspect_ai._util.constants import PKG_PATH
 from inspect_ai.log import list_eval_logs
 from inspect_ai.log._convert import convert_eval_logs
 from inspect_ai.log._file import (
-    eval_log_json,
+    eval_log_json_str,
     read_eval_log,
     read_eval_log_headers,
 )
@@ -127,7 +127,7 @@ def list_command(
 def dump_command(path: str, header_only: bool) -> None:
     """Print log file contents as JSON."""
     log = read_eval_log(path, header_only=header_only)
-    print(eval_log_json(log))
+    print(eval_log_json_str(log))
 
 
 @log_command.command("convert")

{inspect_ai-0.3.49 → inspect_ai-0.3.51}/src/inspect_ai/_cli/score.py

@@ -61,7 +61,7 @@ async def score(
 
     # read the eval log
     recorder = create_recorder_for_location(log_file, log_dir)
-    eval_log = recorder.read_log(log_file)
+    eval_log = await recorder.read_log(log_file)
 
     # check that there are samples therein
     if eval_log.samples is None or len(eval_log.samples) == 0:
@@ -88,7 +88,7 @@ async def score(
     scored = f"{SCORED_SUFFIX}{ext}"
     if not overwrite and not log_file.endswith(scored):
         log_file = log_file.removesuffix(ext) + scored
-    recorder.write_log(log_file, eval_log)
+    await recorder.write_log(log_file, eval_log)
 
     # print results
     display().print(f"\n{eval_log.eval.task}")

{inspect_ai-0.3.49 → inspect_ai-0.3.51}/src/inspect_ai/_display/core/display.py

@@ -19,6 +19,8 @@ from rich.console import Console
 from inspect_ai.log import EvalConfig, EvalResults, EvalStats
 from inspect_ai.model import GenerateConfig, ModelName
 
+from ...util._panel import InputPanel
+
 
 @runtime_checkable
 class Progress(Protocol):
@@ -81,6 +83,8 @@ class TaskWithResult:
 
 TR = TypeVar("TR")
 
+TP = TypeVar("TP", bound=InputPanel)
+
 
 class TaskScreen(contextlib.AbstractContextManager["TaskScreen"]):
     def __exit__(self, *excinfo: Any) -> None:
@@ -95,12 +99,27 @@ class TaskScreen(contextlib.AbstractContextManager["TaskScreen"]):
     ) -> Iterator[Console]:
         yield rich.get_console()
 
+    async def input_panel(self, title: str, panel: type[TP]) -> TP:
+        raise NotImplementedError("input_panel not implemented by current display")
+
+
+@dataclass
+class TaskDisplayMetric:
+    scorer: str
+    name: str
+    value: float | int
+    reducer: str | None
+
 
 @runtime_checkable
 class TaskDisplay(Protocol):
     @contextlib.contextmanager
     def progress(self) -> Iterator[Progress]: ...
 
+    def sample_complete(self, complete: int, total: int) -> None: ...
+
+    def update_metrics(self, scores: list[TaskDisplayMetric]) -> None: ...
+
     def complete(self, result: TaskResult) -> None: ...
 
 

{inspect_ai-0.3.49 → inspect_ai-0.3.51}/src/inspect_ai/_display/core/panel.py

@@ -1,3 +1,5 @@
+from typing import Tuple
+
 import rich
 from rich.console import RenderableType
 from rich.panel import Panel
@@ -16,6 +18,10 @@ def task_panel(
     profile: TaskProfile,
     show_model: bool,
     body: RenderableType,
+    subtitle: RenderableType
+    | str
+    | Tuple[RenderableType | str, RenderableType | str]
+    | None,
     footer: RenderableType | tuple[RenderableType, RenderableType] | None,
     log_location: str | None,
 ) -> Panel:
@@ -25,22 +31,39 @@ def task_panel(
     width = CONSOLE_DISPLAY_WIDTH if is_vscode_notebook(console) else None
     jupyter = console.is_jupyter
 
-    # setup table
+    # root table
     table = Table.grid(expand=True)
     table.add_column()
-    table.add_column(justify="right")
+
+    # setup table
+    if subtitle is not None:
+        subtitle_table = Table.grid(expand=True)
+        subtitle_table.add_column()
+        if isinstance(subtitle, tuple):
+            subtitle_table.add_column(justify="right")
+            subtitle_table.add_row(
+                to_renderable(subtitle[0]), to_renderable(subtitle[1], style=theme.meta)
+            )
+        else:
+            subtitle_table.add_row(to_renderable(subtitle))
+
+        table.add_row(subtitle_table)
 
     # main progress and task info
-    targets = Text.from_markup(task_targets(profile), style=theme.meta)
-    table.add_row(body, targets)
+    table.add_row()
+    table.add_row(body)
+    table.add_row()
 
     # footer if specified
     if footer:
-        table.add_row()
+        footer_table = Table.grid(expand=True)
+        footer_table.add_column()
         if isinstance(footer, tuple):
-            table.add_row(footer[0], footer[1])
+            footer_table.add_column(justify="right")
+            footer_table.add_row(footer[0], footer[1])
         else:
-            table.add_row(footer)
+            footer_table.add_row(footer)
+        table.add_row(footer_table)
 
     # enclose in outer table for log link footer
     root = table
@@ -75,6 +98,13 @@ def task_panel(
     return panel
 
 
+def to_renderable(item: RenderableType | str, style: str = "") -> RenderableType:
+    if isinstance(item, str):
+        return Text.from_markup(item, style=style)
+    else:
+        return item
+
+
 def tasks_title(completed: int, total: int) -> str:
     return f"{completed}/{total} tasks complete"
 

{inspect_ai-0.3.49 → inspect_ai-0.3.51}/src/inspect_ai/_display/core/progress.py

@@ -32,13 +32,20 @@ class RichProgress(Progress):
         model: str = "",
         status: Callable[[], str] | None = None,
         on_update: Callable[[], None] | None = None,
+        count: str = "",
+        score: str = "",
     ) -> None:
         self.total = total
         self.progress = progress
         self.status = status if status else lambda: ""
         self.on_update = on_update
         self.task_id = progress.add_task(
-            description, total=PROGRESS_TOTAL, model=model, status=self.status()
+            description,
+            total=PROGRESS_TOTAL,
+            model=model,
+            status=self.status(),
+            count=count,
+            score=score,
         )
 
     @override
@@ -56,6 +63,16 @@ class RichProgress(Progress):
             task_id=self.task_id, completed=PROGRESS_TOTAL, status=self.status()
         )
 
+    def update_count(self, complete: int, total: int) -> None:
+        self.progress.update(
+            task_id=self.task_id, count=progress_count(complete, total), refresh=True
+        )
+        if self.on_update:
+            self.on_update()
+
+    def update_score(self, score: str) -> None:
+        self.progress.update(task_id=self.task_id, score=score)
+
 
 def rich_progress() -> RProgress:
     console = rich.get_console()
@@ -65,10 +82,12 @@ def rich_progress() -> RProgress:
         TextColumn("{task.fields[model]}"),
         BarColumn(bar_width=40 if is_vscode_notebook(console) else None),
         TaskProgressColumn(),
+        TextColumn("{task.fields[count]}"),
+        TextColumn("{task.fields[score]}"),
         TimeElapsedColumn(),
         transient=True,
         console=console,
-        expand=not is_vscode_notebook(console),
+        expand=True,
     )
 
 
@@ -109,3 +128,11 @@ def progress_time(time: float) -> str:
     minutes, seconds = divmod(time, 60)
     hours, minutes = divmod(minutes, 60)
     return f"{hours:2.0f}:{minutes:02.0f}:{seconds:02.0f}"
+
+
+def progress_count(complete: int, total: int) -> str:
+    # Pad the display to keep it stable
+    total_str = f"{total:,}"
+    complete_str = f"{complete:,}"
+    padding = max(0, len(total_str) - len(complete_str))
+    return " " * padding + f"[{complete_str}/{total_str}]"