inspect-ai 0.3.11__tar.gz → 0.13.3__tar.gz

{inspect_ai-0.3.11 → inspect_ai-0.13.3}/CHANGELOG.md

@@ -1,5 +1,14 @@
 # Changelog
 
+## v0.3.13 (31 May 2024)
+
+- Bugfix: Inspect view was not reliably updating when new evaluation logs were written.
+
+## v0.3.12 (31 May 2024)
+
+- Bugfix: `results` was not defined when no scorer was provided resulting in an error being thrown. Fixed by setting `results = EvalResults()` when no scorer is provided.
+- Bugfix: The viewer was not properly handling samples without scores.
+
 ## v0.3.11 (30 May 2024)
 
 - Update to non-beta version of Anthropic tool use (remove legacy xml tools implementation).

{inspect_ai-0.3.11/src/inspect_ai.egg-info → inspect_ai-0.13.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: inspect_ai
-Version: 0.3.11
+Version: 0.13.3
 Summary: Framework for large language model evaluations
 Author: UK AI Safety Institute
 License: MIT License

{inspect_ai-0.3.11 → inspect_ai-0.13.3}/docs/solvers.qmd

@@ -1,3 +1,7 @@
+---
+tbl-colwidths: [20,25,45]
+---
+
 # Solvers {#sec-solvers}
 
 ## Overview
@@ -147,7 +151,7 @@ You will likely want to experiment with using a distinct `model` for generating
 
 ## Custom Solvers
 
-Let's take a look at the source code for a couple of the built in solvers as a jumping off point for implementing your own solvers. A solver is an implementation of the `Solver` protocol (a function that transforms a `TaskState`):
+In this section we'll take a look at the source code for a couple of the built in solvers as a jumping off point for implementing your own solvers. A solver is an implementation of the `Solver` protocol (a function that transforms a `TaskState`):
 
 ``` python
 async def solve(state: TaskState, generate: Generate) -> TaskState:
@@ -158,10 +162,52 @@ async def solve(state: TaskState, generate: Generate) -> TaskState:
 
 Typically solvers can be customised with parameters (e.g. `template` for prompt engineering solvers). This means that a `Solver` is actually a function which returns the `solve()` function referenced above (this will become more clear in the examples below).
 
+### Task States
+
+Before presenting the examples we'll take a more in-depth look at the `TaskState` class. Task states consist of both lower level data members (e.g. `messages`, `output`) as well as a number of convenience properties. The core members of `TaskState` that are *modified* by solvers are `messages` / `user_prompt` and `output`:
+
+| Member        | Type                | Description                                                                                                                                                                               |
+|-----------|-----------|---------------------------------------------------|
+| `messages`    | list\[ChatMessage\] | Chat conversation history for sample. It is automatically appended to by the `generate()` solver, and is often manipulated by other solvers (e.g. for prompt engineering or elicitation). |
+| `user_prompt` | ChatMessageUser     | Convenience property for accessing the first user message in the message history (commonly used for prompt engineering).                                                                  |
+| `output`      | ModelOutput         | The 'final' model output once we've completed all solving. This field is automatically updated with the last "assistant" message by the `generate()` solver.                              |
+
 ::: {.callout-note appearance="simple"}
-When creating custom solvers, it's critical that you understand Inspect's concurrency model. More specifically, if your solver is doing non-trivial work (e.g. calling REST APIs, executing external processes, etc.) please review [Eval Tuning](#sec-eval-tuning) before proceeding.
+Note that the `generate()` solver automatically updates both the `messages` and `output` fields. For very simple evaluations modifying the `user_prompt` and then calling `generate()` encompasses all of the required interaction with `TaskState`.
 :::
 
+There are two additional fields that solvers might modify (but they are typically for more advanced use cases):
+
+| Member      | Type | Description                                                                                                                                                                  |
+|---------|---------|------------------------------------------------------|
+| `metadata`  | dict | Original metadata from `Sample`, as well as any other custom metadata that solvers choose to write (typically used to coordinate between solvers and/or for custom logging). |
+| `completed` | bool | Solvers can set `completed = True` to cause the task to exit the plan immediately.                                                                                           |
+
+Sometimes its import to have access to the *original* prompt input for the task (as other solvers may have re-written or even removed it entirely). This is available using the `input` and `input_text` properties:
+
+| Member       | Type                       | Description                                                                         |
+|---------------|---------------|-------------------------------------------|
+| `input`      | str \| list\[ChatMessage\] | Original `Sample` input.                                                            |
+| `input_text` | str                        | Convenience function for accessing the initial input from the `Sample` as a string. |
+
+There are several other fields used to provide contextual data from either the task sample or evaluation:
+
+| Member      | Type                | Description                                               |
+|---------------|---------------|------------------------------------------|
+| `sample_id` | int \| str          | Unique ID for sample.                                     |
+| `epoch`     | int                 | Epoch for sample.                                         |
+| `choices`   | list\[str\] \| None | Choices from sample (used only in multiple-choice evals). |
+| `model`     | ModelName           | Name of model currently being evaluated.                  |
+
+Finally, task states also include available tools as well as guidance for the model on which tools to use (if you haven't yet encountered the concept of tool use in language models, don't worry about understanding these fields, the [Tools](tools.qmd) article provides a more in-depth treatment):
+
+| Member        | Type         | Description                  |
+|---------------|--------------|------------------------------|
+| `tools`       | list\[Tool\] | Tools available to the model |
+| `tool_choice` | ToolChoice   | Tool choice directive.       |
+
+These fields are typically modified via the `use_tools()` solver, but they can also be modified directly for more advanced use cases.
+
 ### Example: Prompt Template
 
 Here's the code for the `prompt_template()` solver:
@@ -188,8 +234,11 @@ def prompt_template(template: str, **params: dict[str, Any]):
 A few things to note about this implementation:
 
 1.  The function applies the `@solver` decorator—this registers the `Solver` with Inspect, making it possible to capture its name and parameters for logging, as well as make it callable from a configuration file (e.g. a YAML specification of an eval).
+
 2.  The `solve()` function is declared as `async`. This is so that it can participate in Inspect's optimised scheduling for expensive model generation calls (this solver doesn't call `generate()` but others will).
+
 3.  The `resource()` function is used to read the specified `template`. This function accepts a string, file, or URL as its argument, and then returns a string with the contents of the resource.
+
 4.  We make use of the `user_prompt` property on the `TaskState`. This is a convenience property for locating the first `role="user"` message (otherwise you might need to skip over system messages, etc). Since this is a string templating solver, we use the `state.user_prompt.text` property (so we are dealing with prompt as a string, recall that it can also be a list of messages).
 
 ### Example: Self Critique
@@ -278,6 +327,10 @@ def self_critique(
 
 Note that calls to `generate()` (for both the critique model and the model being evaluated) are called with `await`—this is critical to ensure that the solver participates correctly in the scheduling of generation work.
 
+### Concurrency
+
+When creating custom solvers, it's critical that you understand Inspect's concurrency model. More specifically, if your solver is doing non-trivial work (e.g. calling REST APIs, executing external processes, etc.) please review [Eval Tuning](#sec-eval-tuning) for a more in depth discussion.
+
 ## Early Termination
 
 In some cases a solver has the context available to request an early termination of the plan (i.e. don't call the rest of the solvers). In this case, setting the `TaskState.completed` field will result in forgoing remaining solvers in the plan. For example, here's a simple solver that terminates the plan early:
@@ -318,7 +371,7 @@ In this example the `finish_up()` solver will always be called even if the plan
 
 If your solvers allocate resources (for example, run a Docker container or mount a drive), you will want to make sure that these resources are cleaned up even in the case of an error occurring during the evaluation. To arrange for this, use a `Plan` object with a `cleanup` function:
 
-```python
+``` python
 
 async def cleanup(state):
     # cleanup resources

{inspect_ai-0.3.11 → inspect_ai-0.13.3}/examples/agents/langchain/inspect_langchain.py

@@ -47,8 +47,7 @@ from inspect_ai.solver import Generate, Solver, TaskState
 class LangChainAgent(Protocol):
     async def __call__(
         self, llm: BaseChatModel, input: dict[str, Any]
-    ) -> str | list[str | dict[str, Any]]:
-        ...
+    ) -> str | list[str | dict[str, Any]]: ...
 
 
 def langchain_solver(agent: LangChainAgent) -> Solver:

{inspect_ai-0.3.11 → inspect_ai-0.13.3}/src/inspect_ai/__init__.py

@@ -6,7 +6,7 @@ from inspect_ai._eval.eval import eval, eval_async, eval_retry, eval_retry_async
 from inspect_ai._eval.list import list_tasks
 from inspect_ai._eval.registry import task
 from inspect_ai._eval.score import score, score_async
-from inspect_ai._eval.types import Task, TaskInfo, Tasks
+from inspect_ai._eval.task import Task, TaskInfo, Tasks
 from inspect_ai._util.constants import PKG_NAME
 
 __version__ = importlib_version(PKG_NAME)

{inspect_ai-0.3.11 → inspect_ai-0.13.3}/src/inspect_ai/_cli/list.py

@@ -11,7 +11,7 @@ from typing_extensions import Unpack
 from inspect_ai._cli.common import CommonOptions, common_options, resolve_common_options
 from inspect_ai._cli.util import parse_cli_args
 from inspect_ai._eval.list import list_tasks
-from inspect_ai._eval.types import TaskInfo
+from inspect_ai._eval.task import TaskInfo
 from inspect_ai.log import list_eval_logs
 
 

{inspect_ai-0.3.11 → inspect_ai-0.13.3}/src/inspect_ai/_eval/eval.py

@@ -26,10 +26,10 @@ from inspect_ai.solver import Solver
 from inspect_ai.util._context import init_async_context
 
 from .loader import resolve_tasks
+from .task import Tasks, TaskSpec
 from .task.log import TaskLogger
 from .task.run import task_run
 from .task.util import task_file, task_run_dir
-from .types import Tasks, TaskSpec
 
 log = logging.getLogger(__name__)
 

{inspect_ai-0.3.11 → inspect_ai-0.13.3}/src/inspect_ai/_eval/list.py

@@ -8,7 +8,7 @@ from typing import Any, Callable
 from inspect_ai._util.error import exception_message
 from inspect_ai._util.file import file
 
-from .types import TaskInfo
+from .task import TaskInfo
 
 logger = getLogger(__name__)
 

{inspect_ai-0.3.11 → inspect_ai-0.13.3}/src/inspect_ai/_eval/loader.py

@@ -18,8 +18,8 @@ from inspect_ai.model import Model, ModelName
 
 from .list import task_files
 from .registry import task_create
+from .task import Task, TaskInfo, Tasks
 from .task.constants import TASK_FILE_ATTR, TASK_RUN_DIR_ATTR
-from .types import Task, TaskInfo, Tasks
 
 
 def resolve_tasks(

{inspect_ai-0.3.11 → inspect_ai-0.13.3}/src/inspect_ai/_eval/registry.py

@@ -14,7 +14,7 @@ from inspect_ai._util.registry import (
 )
 from inspect_ai.model import ModelName
 
-from .types import Task
+from .task import Task
 
 MODEL_PARAM = "model"
 

{inspect_ai-0.3.11 → inspect_ai-0.13.3}/src/inspect_ai/_eval/score.py

@@ -17,9 +17,9 @@ from inspect_ai.model import ModelName
 from inspect_ai.scorer import Metric, Score, Scorer, Target
 from inspect_ai.solver import TaskState
 
+from .task import Task
 from .task.results import eval_results
 from .task.util import task_run_dir
-from .types import Task
 
 
 def score(log: EvalLog, scorer: Scorer) -> EvalLog:

inspect_ai-0.13.3/src/inspect_ai/_eval/task/__init__.py

@@ -0,0 +1,3 @@
+from .task import Task, TaskInfo, TaskSpec, Tasks  # noqa: I001, F401
+
+__all__ = ["Task", "TaskInfo", "TaskSpec", "Tasks"]

{inspect_ai-0.3.11 → inspect_ai-0.13.3}/src/inspect_ai/_eval/task/run.py

@@ -23,6 +23,7 @@ from inspect_ai.log import (
     EvalConfig,
     EvalError,
     EvalLog,
+    EvalResults,
     EvalStats,
 )
 from inspect_ai.log._log import eval_error
@@ -34,7 +35,7 @@ from inspect_ai.model import (
 from inspect_ai.scorer import Score, Scorer, Target
 from inspect_ai.solver import Generate, Plan, Solver, TaskState
 
-from ..types import Task
+from ..task import Task
 from .generate import task_generate
 from .images import samples_with_base64_images, states_with_base64_images
 from .log import TaskLogger, collect_eval_data, log_output, log_plan
@@ -101,7 +102,9 @@ async def task_run(
         plan = (
             plan
             if isinstance(plan, Plan)
-            else Plan(plan) if plan is not None else task.plan
+            else Plan(plan)
+            if plan is not None
+            else task.plan
         )
         score = score and task.scorer is not None
         scorer: Scorer | None = task.scorer if (score and task.scorer) else None
@@ -132,7 +135,6 @@ async def task_run(
                     len(plan.steps) + (1 if plan.finish else 0) + (1)  # scorer
                 )
                 with td.progress(total=total_steps) as p:
-
                     # forward progress
                     def progress() -> None:
                         p.update(1)
@@ -195,6 +197,8 @@ async def task_run(
                         metrics=task.metrics,
                     )
                     logger.log_results(results)
+                else:
+                    results = EvalResults()
 
                 # collect eval data
                 collect_eval_data(stats, logger)
@@ -295,7 +299,6 @@ async def resolve_dataset(
     epochs: int,
     log_images: bool,
 ) -> tuple[Dataset, list[Sample], list[TaskState]]:
-
     # apply limit to dataset
     dataset_limit = (
         slice(0, len(dataset))

{inspect_ai-0.3.11 → inspect_ai-0.13.3}/src/inspect_ai/_eval/task/util.py

@@ -7,7 +7,7 @@ from inspect_ai.dataset import Sample
 from inspect_ai.model import ChatMessage, ChatMessageUser
 from inspect_ai.solver import TaskState
 
-from ..types import Task
+from ..task import Task
 from .constants import TASK_FILE_ATTR, TASK_RUN_DIR_ATTR
 
 

{inspect_ai-0.3.11 → inspect_ai-0.13.3}/src/inspect_ai/_view/www/App.mjs

@@ -20,6 +20,7 @@ import { WorkSpace } from "./src/workspace/WorkSpace.mjs";
 
 export function App() {
   const [selected, setSelected] = useState(-1);
+  const [pendingLog, setPendingLog] = useState(undefined);
   const [logs, setLogs] = useState({ log_dir: "", files: [] });
   const [logHeaders, setLogHeaders] = useState({});
   const [offcanvas, setOffcanvas] = useState(false);
@@ -126,24 +127,32 @@ export function App() {
   // Ensure that we have a selected index when there is are 
   // new logs
   useEffect(() => {
-    setSelected(0);
-  }, [logs])
+    if (logs && pendingLog) {
+      const index = logs.files.findIndex((val) => {
+        return pendingLog.endsWith(val.name);
+      });
+      if (index > -1) {
+        setSelected(index);    
+      }
+      setPendingLog(undefined);
+    }
+  }, [logs, pendingLog])
   
   // listen for updateState messages from vscode
   useEffect(() => {
-    const onMessage = (e) => {
+    const onMessage = async (e) => {
       switch (e.data.type || e.data.message) {
         case "updateState": {
           if (e.data.url) {
-
             const index = logs.files.findIndex((val) => {
-              return val.name.endsWith(e.data.url);
+              return e.data.url.endsWith(val.name);
             });
             if (index > -1) {
               // Select the correct index
               setSelected(index);    
             } else {
-              // TODO: Error
+              await loadLogs();
+              setPendingLog(e.data.url);
             }
           }
         }
@@ -153,7 +162,7 @@ export function App() {
     return () => {
       window.removeEventListener("message", onMessage);
     };
-  }, [setCurrentLog]);
+  }, [logs, setCurrentLog, setPendingLog]);
 
   useEffect(async () => {
     // See whether a specific task_file has been passed.
@@ -176,6 +185,9 @@ export function App() {
     // initial fetch of logs
     await load();
 
+    // Select the first log
+    setSelected(0);
+
     // poll every 1s for events
     setInterval(() => {
       api.client_events().then((events) => {

{inspect_ai-0.3.11 → inspect_ai-0.13.3}/src/inspect_ai/_view/www/src/samples/SamplesDescriptor.mjs

@@ -24,6 +24,7 @@ export const samplesDescriptor = (samples, epochs, context) => {
   const uniqScoreValues = [
     ...new Set(
       samples
+        .filter(sample => !!sample.score)
         .map((sample) => sample.score.value)
         .filter((value) => {
           return value !== null;

{inspect_ai-0.3.11 → inspect_ai-0.13.3}/src/inspect_ai/_view/www/src/samples/SamplesTab.mjs

@@ -92,7 +92,7 @@ export const SamplesTab = (props) => {
   // Focus the sample list
   useEffect(() => {
     const listEl = sampleListRef.current;
-    if (listEl) {
+    if (listEl && listEl.base) {
       listEl.base.focus();
     }
   }, [items]);

{inspect_ai-0.3.11 → inspect_ai-0.13.3}/src/inspect_ai/solver/__init__.py

@@ -6,7 +6,8 @@ from ._prompt import (
     prompt_template,
     system_message,
 )
-from ._solver import Generate, Solver, TaskState, generate, solver
+from ._solver import Generate, Solver, generate, solver
+from ._task_state import TaskState
 from ._tool.tool import Tool, tool
 from ._tool.use_tools import use_tools
 from ._tool.web_search import web_search

{inspect_ai-0.3.11 → inspect_ai-0.13.3}/src/inspect_ai/solver/_critique.py

@@ -5,7 +5,8 @@ from inspect_ai.model import (
 )
 from inspect_ai.util import resource
 
-from ._solver import Generate, Solver, TaskState, solver
+from ._solver import Generate, Solver, solver
+from ._task_state import TaskState
 
 
 @solver

{inspect_ai-0.3.11 → inspect_ai-0.13.3}/src/inspect_ai/solver/_multiple_choice.py

@@ -4,7 +4,8 @@ from random import Random
 
 from inspect_ai.util import resource
 
-from ._solver import Generate, Solver, TaskState, solver
+from ._solver import Generate, Solver, solver
+from ._task_state import TaskState
 
 logger = logging.getLogger(__name__)
 

{inspect_ai-0.3.11 → inspect_ai-0.13.3}/src/inspect_ai/solver/_plan.py

@@ -11,7 +11,8 @@ from inspect_ai._util.registry import (
     registry_tag,
 )
 
-from ._solver import Solver, TaskState
+from ._solver import Solver
+from ._task_state import TaskState
 
 
 class Plan:

{inspect_ai-0.3.11 → inspect_ai-0.13.3}/src/inspect_ai/solver/_prompt.py

@@ -3,7 +3,8 @@ from typing import Any
 from inspect_ai.model import ChatMessageSystem
 from inspect_ai.util import resource
 
-from ._solver import Generate, Solver, TaskState, solver
+from ._solver import Generate, Solver, solver
+from ._task_state import TaskState
 from ._util import append_system_message
 
 

{inspect_ai-0.3.11 → inspect_ai-0.13.3}/src/inspect_ai/solver/_solver.py

@@ -18,110 +18,9 @@ from inspect_ai._util.registry import (
     registry_name,
     registry_tag,
 )
-from inspect_ai.model import (
-    ChatMessage,
-    ChatMessageUser,
-    GenerateConfigArgs,
-    ModelName,
-    ModelOutput,
-    ToolChoice,
-)
-
-from ._tool.tool import Tool
-
+from inspect_ai.model import GenerateConfigArgs
 
-class TaskState:
-    def __init__(
-        self,
-        model: ModelName,
-        sample_id: int | str,
-        epoch: int,
-        input: str | list[ChatMessage],
-        choices: list[str] | None,
-        messages: list[ChatMessage],
-        tools: list[Tool] = [],
-        tool_choice: ToolChoice | None = None,
-        output: ModelOutput | None = None,
-        completed: bool = False,
-        metadata: dict[str, Any] = {},
-    ) -> None:
-        self._model = model
-
-        self.sample_id = sample_id
-        """Unique id for sample."""
-
-        self.epoch = epoch
-        """Epoch number for sample."""
-
-        self._input = input
-
-        self.choices = choices
-        """Sample choices."""
-
-        self.messages = messages
-        """Chat conversation history for sample."""
-
-        self.tools = tools
-        """Tools available to the model."""
-
-        self.tool_choice = tool_choice
-        """Tool choice directive."""
-
-        self.output = output if output else ModelOutput(model=str(model), choices=[])
-        """Model output."""
-
-        self.completed = completed
-        """Flag to indicate that the solver loop should terminate."""
-
-        self.metadata = metadata
-        """Additional task state metadata."""
-
-    @property
-    def model(self) -> ModelName:
-        """Name of model being evaluated."""
-        return self._model
-
-    @property
-    def input(self) -> str | list[ChatMessage]:
-        """Sample input."""
-        return self._input
-
-    @property
-    def input_text(self) -> str:
-        """Sample input as text."""
-        if isinstance(self._input, str):
-            return self._input
-        else:
-            input = next(
-                (message.text for message in self._input if message.role == "user"),
-                None,
-            )
-            if input:
-                return input
-            else:
-                raise ValueError(
-                    "input_text requested from TaskState but none available"
-                )
-
-    @property
-    def user_prompt(self) -> ChatMessageUser:
-        """User prompt for this state.
-
-        Tasks are very general and can have may types of inputs.
-        However, in many cases solvers assume they can interact with
-        the state as a "chat" in a predictable fashion (e.g. prompt
-        engineering solvers). This property enables easy read and
-        write access to the user chat prompt. Raises an
-        exception if there is no user prompt
-
-        Returns:
-           First user `ChatMessage` in the task state.
-        """
-        prompt = next((m for m in self.messages if m.role == "user"), None)
-        if prompt:
-            return prompt
-        else:
-            raise ValueError("user_prompt requested from TaskState but none available")
+from ._task_state import TaskState
 
 
 @runtime_checkable