inspect-ai 0.3.100__tar.gz → 0.3.101__tar.gz

{inspect_ai-0.3.100 → inspect_ai-0.3.101}/CHANGELOG.md

@@ -1,3 +1,16 @@
+## v0.3.101 (05 June 2025)
+
+- Eval set: Default `max_tasks` to the greater of 4 and the number of models being evaluated.
+- Eval set: Do not read full eval logs into memory at task completion.
+- pass_at_k: Treat threshold as the the minimum inclusive value for passing (rather than checking equality)
+- Web search: Include links specified by providers in the results.
+- Inspect View: Display sample id & epoch in sample dialog title bar.
+- Inspect View: Don't open sample dialog when simply navigating the sample list.
+- Inspect View: Fix error that could occur when determine transcript outline collapse state.
+- Inspect View: Show the correct sample when opening a sample from a sorted list.
+- Bugfix: Ensure that dataset shuffle_choices=True always uses a distinct random seed.
+- Bugfix: Don't attempt to use OpenAI's web search preview against models that are known to not support it.
+
 ## v0.3.100 (01 June 2025)
 
 - [time_limit()](https://inspect.aisi.org.uk/errors-and-limits.html#time-limit) and [working_limit()](https://inspect.aisi.org.uk/errors-and-limits.html#working-limit) context managers for scoped application of time limits.

{inspect_ai-0.3.100/src/inspect_ai.egg-info → inspect_ai-0.3.101}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: inspect_ai
-Version: 0.3.100
+Version: 0.3.101
 Summary: Framework for large language model evaluations
 Author: UK AI Security Institute
 License: MIT License

{inspect_ai-0.3.100 → inspect_ai-0.3.101}/docs/eval-sets.qmd

@@ -41,6 +41,21 @@ eval_set(
 
 Note that in both cases we specified a custom log directory—this is actually a requirement for eval sets, as it provides a scope where completed work can be tracked.
 
+### Concurrency
+
+By default, `eval_set()` will run multiple tasks in parallel, using the greater of 4 and the number of models being evaluated as the default `max_tasks`. The eval set scheduler will always attempt to balance active tasks across models so that contention for a single model provider is minimized.
+
+Use the `max_tasks` option to override the default behavior:
+
+```python
+eval_set(
+   tasks=["mmlu.py", "mathematics.py", "ctf.py", "science.py"],
+   model=["openai/gpt-4o", "anthropic/claude-3-5-sonnet-20240620"],
+   max_tasks=8,
+   log_dir="logs-run-42"      
+)
+```
+
 ### Dynamic Tasks
 
 In the above examples tasks are ready from the filesystem. It is also possible to dynamically create a set of tasks and pass them to the `eval_set()` function. For example:

{inspect_ai-0.3.100 → inspect_ai-0.3.101}/docs/parallelism.qmd

@@ -250,7 +250,7 @@ def list_files():
         """List the files in a directory.
 
         Args:
-            dir (str): Directory
+            dir: Directory
 
         Returns:
             File listing of the directory

{inspect_ai-0.3.100 → inspect_ai-0.3.101}/docs/sandboxing.qmd

@@ -28,7 +28,7 @@ def list_files():
         """List the files in a directory.
 
         Args:
-            dir (str): Directory
+            dir: Directory
 
         Returns:
             File listing of the directory

{inspect_ai-0.3.100 → inspect_ai-0.3.101}/docs/tools-custom.qmd

@@ -77,7 +77,7 @@ def list_files():
         """List the files in a directory.
 
         Args:
-            dir (str): Directory
+            dir: Directory
 
         Returns:
             File listing of the directory

{inspect_ai-0.3.100 → inspect_ai-0.3.101}/docs/tools-standard.qmd

@@ -394,7 +394,7 @@ web_search([{"openai": None}, {"google": {"num_results": 5}}])
 
 ### OpenAI Options
 
-The `web_search()` tool can use OpenAI's built-in search capability when running on OpenAI models. This provider does not require any API keys beyond what's needed for the model itself.
+The `web_search()` tool can use OpenAI's built-in search capability when running on a limited number of OpenAI models (currently "gpt-4o", "gpt-4o-mini", and "gpt-4.1"). This provider does not require any API keys beyond what's needed for the model itself.
 
 For more details on OpenAI's web search parameters, see [OpenAI Web Search Documentation](https://platform.openai.com/docs/guides/tools-web-search?api-mode=responses).
 

{inspect_ai-0.3.100 → inspect_ai-0.3.101}/docs/tutorial.qmd

@@ -484,8 +484,8 @@ def add():
         Add two numbers.
 
         Args:
-            x (int): First number to add.
-            y (int): Second number to add.
+            x: First number to add.
+            y: Second number to add.
 
         Returns:
             The sum of the two numbers.

{inspect_ai-0.3.100 → inspect_ai-0.3.101}/examples/tool_use.py

@@ -17,8 +17,8 @@ def add():
         Add two numbers.
 
         Args:
-            x (int): First number to add.
-            y (int): Second number to add.
+            x: First number to add.
+            y: Second number to add.
 
         Returns:
             The sum of the two numbers.
@@ -43,7 +43,7 @@ def list_files():
         """List the files in a directory.
 
         Args:
-            dir (str): Directory
+            dir: Directory
 
         Returns:
             File listing of the directory
@@ -92,7 +92,7 @@ def read_file():
         """Read the contents of a file.
 
         Args:
-            file (str): File to read
+            file: File to read
 
         Returns:
             File contents
@@ -118,8 +118,8 @@ def write_file():
         """Write content to a file.
 
         Args:
-            file (str): File to write
-            contents (str): Contents of file
+            file: File to write
+            contents: Contents of file
         """
         return await sandbox().write_file(file, contents)
 

{inspect_ai-0.3.100 → inspect_ai-0.3.101}/src/inspect_ai/_cli/eval.py

@@ -35,7 +35,7 @@ from .util import (
 )
 
 MAX_SAMPLES_HELP = "Maximum number of samples to run in parallel (default is running all samples in parallel)"
-MAX_TASKS_HELP = "Maximum number of tasks to run in parallel (default is 1)"
+MAX_TASKS_HELP = "Maximum number of tasks to run in parallel (default is 1 for eval and 4 for eval-set)"
 MAX_SUBPROCESSES_HELP = (
     "Maximum number of subprocesses to run in parallel (default is os.cpu_count())"
 )
@@ -949,6 +949,7 @@ def eval_exec(
             log_images=log_images,
             log_buffer=log_buffer,
             log_shared=log_shared,
+            log_header_only=True,  # cli invocation doesn't need full log
             score=score,
             score_display=score_display,
         )

{inspect_ai-0.3.100 → inspect_ai-0.3.101}/src/inspect_ai/_eval/eval.py

@@ -105,6 +105,7 @@ def eval(
     log_images: bool | None = None,
     log_buffer: int | None = None,
     log_shared: bool | int | None = None,
+    log_header_only: bool | None = None,
     score: bool = True,
     score_display: bool | None = None,
     **kwargs: Unpack[GenerateConfigArgs],
@@ -181,6 +182,8 @@ def eval(
         log_shared: Sync sample events to log directory so that users on other systems
             can see log updates in realtime (defaults to no syncing). Specify `True`
             to sync every 10 seconds, otherwise an integer to sync every `n` seconds.
+        log_header_only: If `True`, the function should return only log headers rather
+            than full logs with samples (defaults to `False`).
         score: Score output (defaults to True)
         score_display: Show scoring metrics in realtime (defaults to True)
         **kwargs: Model generation options.
@@ -234,6 +237,7 @@ def eval(
                 log_images=log_images,
                 log_buffer=log_buffer,
                 log_shared=log_shared,
+                log_header_only=log_header_only,
                 score=score,
                 score_display=score_display,
                 **kwargs,
@@ -288,6 +292,7 @@ async def eval_async(
     log_images: bool | None = None,
     log_buffer: int | None = None,
     log_shared: bool | int | None = None,
+    log_header_only: bool | None = None,
     score: bool = True,
     score_display: bool | None = None,
     **kwargs: Unpack[GenerateConfigArgs],
@@ -344,7 +349,9 @@ async def eval_async(
         log_buffer: Number of samples to buffer before writing log file.
            If not specified, an appropriate default for the format and filesystem is
            chosen (10 for most all cases, 100 for JSON logs on remote filesystems).
-        log_shared: Indicate that the log directory is shared, which results in additional syncing of realtime log data for Inspect View.
+        log_shared: Indicate that the log directory is shared, which results in additional
+        syncing of realtime log data for Inspect View.
+        log_header_only: If `True`, the function should return only log headers rather than full logs with samples (defaults to `False`).
         score: Score output (defaults to True)
         score_display: Show scoring metrics in realtime (defaults to True)
         **kwargs: Model generation options.
@@ -432,6 +439,9 @@ async def eval_async(
         # resolve log_shared
         log_shared = DEFAULT_LOG_SHARED if log_shared is True else log_shared
 
+        # resolve header only
+        log_header_only = log_header_only is True
+
         # validate that --log-shared can't use used with 'json' format
         if log_shared and log_format == JSON_LOG_FORMAT:
             raise PrerequisiteError(
@@ -507,6 +517,7 @@ async def eval_async(
                         eval_config=eval_config,
                         eval_sandbox=sandbox,
                         recorder=recorder,
+                        header_only=log_header_only,
                         epochs_reducer=epochs_reducer,
                         solver=solver,
                         tags=tags,
@@ -532,6 +543,7 @@ async def eval_async(
                 eval_config=eval_config,
                 eval_sandbox=sandbox,
                 recorder=recorder,
+                header_only=log_header_only,
                 epochs_reducer=epochs_reducer,
                 solver=solver,
                 tags=tags,

{inspect_ai-0.3.100 → inspect_ai-0.3.101}/src/inspect_ai/_eval/evalset.py

@@ -168,7 +168,7 @@ def eval_set(
         max_samples: Maximum number of samples to run in parallel
             (default is max_connections)
         max_tasks: Maximum number of tasks to run in parallel
-            (defaults to number of models being evaluated)
+            (defaults to the greater of 4 and the number of models being evaluated)
         max_subprocesses: Maximum number of subprocesses to
             run in parallel (default is os.cpu_count())
         max_sandboxes: Maximum number of sandboxes (per-provider)
@@ -235,6 +235,7 @@ def eval_set(
             log_images=log_images,
             log_buffer=log_buffer,
             log_shared=log_shared,
+            log_header_only=True,
             score=score,
             **kwargs,
         )
@@ -277,7 +278,7 @@ def eval_set(
     retry_connections = retry_connections or 0.5
     retry_cleanup = retry_cleanup is not False
     max_connections = starting_max_connections(models, GenerateConfig(**kwargs))
-    max_tasks = max_tasks if max_tasks is not None else len(models)
+    max_tasks = max_tasks if max_tasks is not None else max(len(models), 4)
 
     # prepare console/status
     console = rich.get_console()

{inspect_ai-0.3.100 → inspect_ai-0.3.101}/src/inspect_ai/_eval/run.py

@@ -63,6 +63,7 @@ async def eval_run(
     eval_config: EvalConfig,
     eval_sandbox: SandboxEnvironmentType | None,
     recorder: Recorder,
+    header_only: bool,
     epochs_reducer: list[ScoreReducer] | None = None,
     solver: Solver | SolverSpec | None = None,
     tags: list[str] | None = None,
@@ -212,6 +213,7 @@ async def eval_run(
                     eval_config=task_eval_config,
                     metadata=((metadata or {}) | (task.metadata or {})) or None,
                     recorder=recorder,
+                    header_only=header_only,
                 )
                 await logger.init()
 

{inspect_ai-0.3.100 → inspect_ai-0.3.101}/src/inspect_ai/_eval/task/log.py

@@ -75,6 +75,7 @@ class TaskLogger:
         eval_config: EvalConfig,
         metadata: dict[str, Any] | None,
         recorder: Recorder,
+        header_only: bool,
     ) -> None:
         # determine versions
         git = git_context()
@@ -153,6 +154,7 @@ class TaskLogger:
 
         # stack recorder and location
         self.recorder = recorder
+        self.header_only = header_only
 
         # number of samples logged
         self._samples_completed = 0
@@ -238,7 +240,7 @@ class TaskLogger:
     ) -> EvalLog:
         # finish and get log
         log = await self.recorder.log_finish(
-            self.eval, status, stats, results, reductions, error
+            self.eval, status, stats, results, reductions, error, self.header_only
         )
 
         # cleanup the events db