inspect-ai 0.3.61__tar.gz → 0.3.62__tar.gz

{inspect_ai-0.3.61 → inspect_ai-0.3.62}/CHANGELOG.md

@@ -1,5 +1,15 @@
 # Changelog
 
+## v0.3.62 (03 February 2025)
+
+- Various improvements for [reasoning models](https://github.com/UKGovernmentBEIS/inspect_ai/pull/1229) including extracting reasoning content from assistant messages.
+- OpenAI: Handle `reasoning_effort`, `max_tokens`, `temperature`, and `parallel_tool_calls` correctly for o3 models.
+- OpenAI: Map some additional 400 status codes to `content_filter` stop reason.
+- Anthropic: Handle 413 status code (Payload Too Large) and map to `model_length` StopReason.
+- Tasks: Log sample with error prior to raising task-ending exception.
+- Computer: Various improvements to image including desktop, python, and VS Code configuration.
+- Bugfix: Don't download full log from S3 for header_only reads.
+
 ## v0.3.61 (31 January 2025)
 
 - Computer: Enable viewing computer tool's remote mouse cursor via VNC.
@@ -7,6 +17,7 @@
 - Limits: Amend `SampleLimitExceededError` with current `state` so that messages, etc. are preserved when limits are hit.
 - Tools: Properly handle image dispatching when multiple tool calls are made by assistant.
 - Anthropic: Raise error on 400 status not identified as model_length or content_filter.
+- Basic Agent: `incorrect_message` can now optionally be an async function.
 - Bugfix: Remove `suffix` from `eval-set` CLI args.
 - Bugfix: Only catch `Exception` from sandboxenv_init (allow cancelled to propagate)
 

{inspect_ai-0.3.61 → inspect_ai-0.3.62}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: inspect_ai
-Version: 0.3.61
+Version: 0.3.62
 Summary: Framework for large language model evaluations
 Author: UK AI Safety Institute
 License: MIT License

{inspect_ai-0.3.61 → inspect_ai-0.3.62}/docs/_metadata_typing.md

@@ -22,7 +22,7 @@ dataset = json_dataset(
 )
 ```
 
-To read metadata in a typesafe fashion, us the `metadata_as()` method on `Sample` or `TaskState`:
+To read metadata in a typesafe fashion, use the `metadata_as()` method on `Sample` or `TaskState`:
 
 ```python
 metadata = state.metadata_as(PopularityMetadata)

{inspect_ai-0.3.61 → inspect_ai-0.3.62}/docs/datasets.qmd

@@ -50,7 +50,7 @@ dataset = csv_dataset("security_guide.csv")
 
 Note that samples from datasets without an `id` field will automatically be assigned ids based on an auto-incrementing integer starting with 1.
 
-If your samples include `choices`, then the `target` should be a numeric index into the available `choices` rather than a letter (this is an implicit assumption of the `multiple_choice()` solver).
+If your samples include `choices`, then the `target` should be a capital letter representing the correct answer in `choices`, see [`multiple_choice`](solvers.qmd#multiple-choice)
 
 ### Files
 

{inspect_ai-0.3.61 → inspect_ai-0.3.62}/docs/index.qmd

@@ -113,7 +113,7 @@ from inspect_ai import Task, task
 from inspect_ai.dataset import example_dataset
 from inspect_ai.scorer import model_graded_fact
 from inspect_ai.solver import (               
-  chain_of_thought(), generate, self_critique   
+  chain_of_thought, generate, self_critique   
 )                                             
 
 @task

{inspect_ai-0.3.61 → inspect_ai-0.3.62}/docs/models.qmd

@@ -203,7 +203,7 @@ The following environment variables are supported by the Google provider
 | Variable         | Description       |
 |------------------|-----------------------------|
 | `GOOGLE_API_KEY`           | API key credentials (required).         |
-| `ANTHROPIC_BASE_URL`           | Base URL for requests (optional)          |
+| `GOOGLE_BASE_URL`           | Base URL for requests (optional)          |
 
 ### Safety Settings
 

{inspect_ai-0.3.61 → inspect_ai-0.3.62}/docs/tutorial.qmd

@@ -8,6 +8,7 @@ Below we'll walk step-by-step through several basic examples of Inspect evaluati
 
 | Example | Demonstrates |
 |-----------------------------|-------------------------------------------|
+| [Hello World](#hello-world) | Simplest eval to test setup. |
 | [Security Guide](#sec-security-guide) | Custom system prompt; Model grading of output. |
 | [HellaSwag](#sec-hellaswag) | Mapping external data formats into Inspect; Multiple choice questions. |
 | [GSM8K](#sec-gsm8k) | Using fewshot examples; Scoring numeric output. |
@@ -19,12 +20,37 @@ Below we'll walk step-by-step through several basic examples of Inspect evaluati
 
 
 
-See also the complete list of [Examples](#sec-examples) for demonstrations of more advanced features.
+See also the complete list of [Examples](https://github.com/UKGovernmentBEIS/inspect_ai/tree/main/examples) for demonstrations of more advanced features.
 
 ::: {.callout-note appearance="simple"}
 Note that in these examples we won't show a `--model` command line argument when we call `inspect eval` (the presumption being that it has been already established via the `INSPECT_EVAL_MODEL` environment variable).
 :::
 
+## Hello World {#hello-world}
+
+This is the simplest possible Inspect evaluation task. The solver (`generate()`) just calls the model with the prompt and the scorer (`exact()`) checks whether the model produces exactly the `target` from the dataset:
+
+```{python}
+from inspect_ai import Task, task
+from inspect_ai.dataset import Sample
+from inspect_ai.scorer import exact
+from inspect_ai.solver import generate
+
+@task
+def hello_world():
+    return Task(
+        dataset=[
+            Sample(
+                input="Just reply with Hello World",
+                target="Hello World",
+            )
+        ],
+        solver=[generate()],
+        scorer=exact(),
+    )
+```
+
+
 ## Security Guide {#sec-security-guide}
 
 The security guide example contains 16 question-answer pairs taken from the security guide dataset published in the [OpenAI Evals](https://github.com/openai/evals) repository. This eval tests how well models are doing at providing cybersecurity guidance. Here are some examples from the dataset:
@@ -39,7 +65,7 @@ The security guide example contains 16 question-answer pairs taken from the secu
 We'll start by importing the functions we need from Inspect and defining a system message that orients the model to its role as a computer security expert.
 
 ```{python}
-from inspect_ai import Task, eval, task
+from inspect_ai import Task, task
 from inspect_ai.dataset import example_dataset
 from inspect_ai.scorer import model_graded_fact
 from inspect_ai.solver import generate, system_message
@@ -92,7 +118,7 @@ For example, here is one of the questions in the dataset along with its set of p
 We'll start by importing the functions we need from Inspect, defining a system message, and writing a function to convert dataset records to samples (we need to do this to convert the index-based label in the dataset to a letter).
 
 ```{python}
-from inspect_ai import Task, eval, task
+from inspect_ai import Task, task
 from inspect_ai.dataset import Sample, hf_dataset
 from inspect_ai.scorer import choice
 from inspect_ai.solver import multiple_choice, system_message
@@ -116,7 +142,7 @@ Note that even though we don't use it for the evaluation, we save the `source_id
 
 ### Eval {.unlisted}
 
-We'll load the dataset from [HuggingFace](https://huggingface.co/datasets/Rowan/hellaswag) using the `hf_dataset()` function. We'll draw data from the validation split, and use the `record_to_sample()` function to parse the records (we'll also pass `trust=True` to indicate that we are okay with Hugging Face executing the dataset loading code provided by hellaswag):
+We'll load the dataset from [HuggingFace](https://huggingface.co/datasets/Rowan/hellaswag) using the `hf_dataset()` function. We'll draw data from the validation split, and use the `record_to_sample()` function to parse the records (we'll also pass `trust=True` to indicate that we are okay with locally executing the dataset loading code provided by hellaswag):
 
 ```{python}
 @task
@@ -197,7 +223,7 @@ def sample_to_fewshot(sample):
     )
 ```
 
-Note that we save the "reasoning" part of the answer in `metadata`—we do this so that we can use it to compose the fewshot prompt (as illustrated in `sample_to_fewshot()`).
+Note that we save the "reasoning" part of the answer in `metadata` — we do this so that we can use it to compose the [fewshot prompt](https://www.promptingguide.ai/techniques/fewshot) (as illustrated in `sample_to_fewshot()`).
 
 Here's the prompt we'll used to elicit a chain of thought answer in the right format:
 
@@ -330,7 +356,7 @@ def math(shuffle=True):
                 input="problem", 
                 target="solution"
             ),
-            shuffle=True,
+            shuffle=shuffle,
             trust=True,
         ),
         solver=[
@@ -419,7 +445,9 @@ Now we run the evaluation, limiting it to 500 problems (as there are over 12,000
 $ inspect eval math.py --limit 500
 ```
 
-This will draw 500 random samples from the dataset (because we defined `shuffle=True` in our call to load the dataset). The task lets you override this with a task parameter (e.g. in case you wanted to evaluate a specific sample or range of samples):
+This will draw 500 random samples from the dataset (because the default is `shuffle=True` in our call to load the dataset).
+
+The task lets you override this with a task parameter (e.g. in case you wanted to evaluate a specific sample or range of samples):
 
 ``` bash
 $ inspect eval math.py --limit 100-200 -T shuffle=false
@@ -442,14 +470,13 @@ If you want to use tools in your evals it's worth taking some time to learn how
 We'll demonstrate with a simple tool that adds two numbers, using the `@tool` decorator to register it with the system:
 
 ```{python}
-from inspect_ai import Task, eval, task
+from inspect_ai import Task, task
 from inspect_ai.dataset import Sample
-from inspect_ai.scorer import includes, match
+from inspect_ai.scorer import match
 from inspect_ai.solver import (
-    generate, system_message, use_tools
+    generate, use_tools
 )
 from inspect_ai.tool import tool
-from inspect_ai.util import subprocess
 
 @tool
 def add():

{inspect_ai-0.3.61 → inspect_ai-0.3.62}/src/inspect_ai/_cli/eval.py

@@ -385,6 +385,14 @@ def eval_options(func: Callable[..., Any]) -> Callable[..., click.Context]:
         help="Constrains effort on reasoning for reasoning models. Open AI o1 models only.",
         envvar="INSPECT_EVAL_REASONING_EFFORT",
     )
+    @click.option(
+        "--reasoning-history/--no-reasoning-history",
+        type=bool,
+        is_flag=True,
+        default=True,
+        help="Include reasoning in chat message history sent to generate.",
+        envvar="INSPECT_EVAL_REASONING_HISTORY",
+    )
     @click.option(
         "--log-format",
         type=click.Choice(["eval", "json"], case_sensitive=False),
@@ -444,6 +452,7 @@ def eval_command(
     max_tool_output: int | None,
     cache_prompt: str | None,
     reasoning_effort: str | None,
+    reasoning_history: bool | None,
     message_limit: int | None,
     token_limit: int | None,
     time_limit: int | None,
@@ -603,6 +612,7 @@ def eval_set_command(
     max_tool_output: int | None,
     cache_prompt: str | None,
     reasoning_effort: str | None,
+    reasoning_history: bool | None,
     message_limit: int | None,
     token_limit: int | None,
     time_limit: int | None,
@@ -841,6 +851,9 @@ def config_from_locals(locals: dict[str, Any]) -> GenerateConfigArgs:
             if key == "internal_tools":
                 if value is not False:
                     value = None
+            if key == "reasoning_history":
+                if value is not False:
+                    value = None
             config[key] = value  # type: ignore
     return config
 

{inspect_ai-0.3.61 → inspect_ai-0.3.62}/src/inspect_ai/_cli/view.py

@@ -63,6 +63,10 @@ def start(
     INSPECT_VIEW_AUTHORIZATION_TOKEN = "INSPECT_VIEW_AUTHORIZATION_TOKEN"
     authorization = os.environ.get(INSPECT_VIEW_AUTHORIZATION_TOKEN, None)
     if authorization:
+        # this indicates we are in vscode -- we want to set the log level to HTTP
+        # in vscode, updated versions of the extension do this but we set it
+        # manually here as a temporary bridge for running against older versions
+        common["log_level"] = "HTTP"
         del os.environ[INSPECT_VIEW_AUTHORIZATION_TOKEN]
         os.unsetenv(INSPECT_VIEW_AUTHORIZATION_TOKEN)
 

{inspect_ai-0.3.61 → inspect_ai-0.3.62}/src/inspect_ai/_display/textual/widgets/transcript.py

@@ -15,6 +15,7 @@ from inspect_ai._util.transcript import (
     set_transcript_markdown_options,
     transcript_function,
     transcript_markdown,
+    transcript_reasoning,
     transcript_separator,
 )
 from inspect_ai.log._samples import ActiveSample
@@ -33,7 +34,11 @@ from inspect_ai.log._transcript import (
     SubtaskEvent,
     ToolEvent,
 )
-from inspect_ai.model._chat_message import ChatMessage, ChatMessageUser
+from inspect_ai.model._chat_message import (
+    ChatMessage,
+    ChatMessageAssistant,
+    ChatMessageUser,
+)
 from inspect_ai.model._render import messages_preceding_assistant
 from inspect_ai.tool._tool import ToolResult
 from inspect_ai.tool._tool_transcript import transcript_tool_call
@@ -171,8 +176,8 @@ def render_model_event(event: ModelEvent) -> EventDisplay:
     # content
     content: list[RenderableType] = []
 
-    def append_message(message: ChatMessage, text: str | None = None) -> None:
-        content.extend(render_message(message, text))
+    def append_message(message: ChatMessage) -> None:
+        content.extend(render_message(message))
 
     # render preceding messages
     preceding = messages_preceding_assistant(event.input)
@@ -309,16 +314,17 @@ def render_as_json(json: Any) -> RenderableType:
     )
 
 
-def render_message(
-    message: ChatMessage, text: str | None = None
-) -> list[RenderableType]:
+def render_message(message: ChatMessage) -> list[RenderableType]:
     content: list[RenderableType] = [
         Text(message.role.capitalize(), style="bold"),
         Text(),
     ]
-    text = text or message.text
-    if text:
-        content.extend([transcript_markdown(text.strip(), escape=True)])
+
+    if isinstance(message, ChatMessageAssistant) and message.reasoning:
+        content.extend(transcript_reasoning(message.reasoning))
+
+    if message.text:
+        content.extend([transcript_markdown(message.text.strip(), escape=True)])
     return content
 
 

{inspect_ai-0.3.61 → inspect_ai-0.3.62}/src/inspect_ai/_eval/task/error.py

@@ -8,28 +8,24 @@ class SampleErrorHandler:
         self.fail_on_error = True if fail_on_error is None else fail_on_error
         self.total_samples = float(total_samples)
 
-    def __call__(self, ex: BaseException) -> EvalError:
+    def __call__(self, ex: BaseException) -> tuple[EvalError, BaseException | None]:
         # increment error count
         self.error_count += 1
 
         # create error (we may return it)
-        def sample_error() -> EvalError:
-            return eval_error(ex, type(ex), ex, ex.__traceback__)
+        def sample_error(
+            *, raise_error: bool
+        ) -> tuple[EvalError, BaseException | None]:
+            return eval_error(
+                ex, type(ex), ex, ex.__traceback__
+            ), ex if raise_error else None
 
         # check against limits
         if isinstance(self.fail_on_error, bool):
-            if self.fail_on_error:
-                raise ex
-            else:
-                return sample_error()
+            return sample_error(raise_error=self.fail_on_error)
         else:
             if self.fail_on_error < 1:
                 max_errors = self.fail_on_error * self.total_samples
-                if self.error_count >= max_errors:
-                    raise ex
-                else:
-                    return sample_error()
-            elif self.error_count >= self.fail_on_error:
-                raise ex
+                return sample_error(raise_error=self.error_count >= max_errors)
             else:
-                return sample_error()
+                return sample_error(raise_error=self.error_count >= self.fail_on_error)

{inspect_ai-0.3.61 → inspect_ai-0.3.62}/src/inspect_ai/_eval/task/run.py

@@ -496,7 +496,7 @@ async def task_run_sample(
     logger: TaskLogger | None,
     log_images: bool,
     sample_source: EvalSampleSource | None,
-    sample_error: Callable[[BaseException], EvalError],
+    sample_error: SampleErrorHandler,
     sample_complete: Callable[[dict[str, SampleScore]], None],
     fails_on_error: bool,
     time_limit: int | None,
@@ -548,12 +548,12 @@ async def task_run_sample(
     )
 
     # helper to handle exceptions (will throw if we've exceeded the limit)
-    def handle_error(ex: BaseException) -> EvalError:
+    def handle_error(ex: BaseException) -> tuple[EvalError, BaseException | None]:
         err = sample_error(ex)
         py_logger.warning(
             f"Sample error (id: {sample.id}, epoch: {state.epoch}): {exception_message(ex)})"
         )
-        transcript()._event(ErrorEvent(error=err))
+        transcript()._event(ErrorEvent(error=err[0]))
         return err
 
     # solver loop
@@ -572,6 +572,7 @@ async def task_run_sample(
         ) as active,
     ):
         error: EvalError | None = None
+        raise_error: BaseException | None = None
         results: dict[str, SampleScore] = {}
         try:
             async with sandboxenv_cm:
@@ -640,7 +641,7 @@ async def task_run_sample(
                                 state = sample_state() or state
                             case "error":
                                 # default error handling
-                                error = handle_error(ex)
+                                error, raise_error = handle_error(ex)
 
                     else:
                         raise
@@ -660,7 +661,7 @@ async def task_run_sample(
                     state.completed = True
 
                 except BaseException as ex:
-                    error = handle_error(ex)
+                    error, raise_error = handle_error(ex)
 
                 # set timeout for scoring. if the original timeout was hit we still
                 # want to provide opportunity for scoring, but we don't necessarily
@@ -737,11 +738,10 @@ async def task_run_sample(
                         )
 
                     # handle error (this will throw if we've exceeded the limit)
-                    error = handle_error(ex)
+                    error, raise_error = handle_error(ex)
 
-        # handle sandboxenv init errors
         except Exception as ex:
-            error = handle_error(ex)
+            error, raise_error = handle_error(ex)
 
         # complete the sample
         progress(SAMPLE_TOTAL_PROGRESS_UNITS)
@@ -772,6 +772,8 @@ async def task_run_sample(
             if results is not None:
                 sample_complete(results)
             return results
+        elif raise_error:
+            raise raise_error
         else:
             return None
 

{inspect_ai-0.3.61 → inspect_ai-0.3.62}/src/inspect_ai/_util/transcript.py

@@ -111,6 +111,17 @@ def transcript_panel(
     )
 
 
+def transcript_reasoning(reasoning: str) -> list[RenderableType]:
+    content: list[RenderableType] = []
+    content.append(
+        transcript_markdown(
+            f"**<think>**  \n{reasoning}  \n**</think>**\n\n", escape=True
+        )
+    )
+    content.append(Text())
+    return content
+
+
 def transcript_separator(title: str, color: str) -> RenderableType:
     return Rule(title=title, style=f"{color} bold", align="center", end="\n\n")
 

{inspect_ai-0.3.61 → inspect_ai-0.3.62}/src/inspect_ai/_view/www/dist/assets/index.css

@@ -15735,6 +15735,7 @@ pre.ap-terminal.ap-cursor-on .ap-line .ap-cursor.ap-inverse {
 }
 pre.ap-terminal:not(.ap-blink) .ap-line .ap-blink {
   color: transparent;
+  border-color: transparent;
 }
 pre.ap-terminal .ap-bright {
   font-weight: bold;