inspect-ai 0.3.60__tar.gz → 0.3.61__tar.gz

{inspect_ai-0.3.60 → inspect_ai-0.3.61}/.pre-commit-config.yaml

@@ -5,7 +5,7 @@ default_language_version:
   python: python3.11
 repos:
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.9.3
+    rev: v0.9.4
     hooks:
       # Run the linter.
       - id: ruff

{inspect_ai-0.3.60 → inspect_ai-0.3.61}/CHANGELOG.md

@@ -1,5 +1,15 @@
 # Changelog
 
+## v0.3.61 (31 January 2025)
+
+- Computer: Enable viewing computer tool's remote mouse cursor via VNC.
+- Computer: Disable lock screen on from computer tool reference image.
+- Limits: Amend `SampleLimitExceededError` with current `state` so that messages, etc. are preserved when limits are hit.
+- Tools: Properly handle image dispatching when multiple tool calls are made by assistant.
+- Anthropic: Raise error on 400 status not identified as model_length or content_filter.
+- Bugfix: Remove `suffix` from `eval-set` CLI args.
+- Bugfix: Only catch `Exception` from sandboxenv_init (allow cancelled to propagate)
+
 ## v0.3.60 (29 January 2025)
 
 - [Agent Bridge](https://inspect.ai-safety-institute.org.uk/agent-bridge.html) for integrating external agent frameworks with Inspect.
@@ -222,7 +232,7 @@
 
 ## v0.3.45 (11 November 2024)
 
-- [time_limit](https://inspect.ai-safety-institute.org.uk/errors_and_limits.html#sec-sample-limits) option for specifying a maximum execution time for samples.
+- [time_limit](https://inspect.ai-safety-institute.org.uk/errors_and_limits.html#sample-limits) option for specifying a maximum execution time for samples.
 - [read_eval_log_samples()](https://inspect.ai-safety-institute.org.uk/eval-logs.html#streaming) function for streaming reads of `.eval` log files.
 - Mistral: Support for multi-modal models (requires v1.2 of mistralai package).
 - Groq: Support for multi-modal models (requires v0.11.0 of groq package).
@@ -298,7 +308,7 @@
 - [Approval mode](https://inspect.ai-safety-institute.org.uk/approval.html) for extensible approvals of tool calls (human and auto-approvers built in,  arbitrary other approval schemes via extensions).
 - [Trace mode](https://inspect.ai-safety-institute.org.uk/interactivity.html#sec-trace-mode) for printing model interactions to the terminal.
 - Add `as_dict()` utility method to `Score`
-- [Sample limits](https://inspect.ai-safety-institute.org.uk/errors_and_limits.html#sec-sample-limits) (`token_limit` and `message_limit`) for capping the number of tokens or messages used per sample ( `message_limit` replaces deprecated `max_messages`).
+- [Sample limits](https://inspect.ai-safety-institute.org.uk/errors_and_limits.html#sample-limits) (`token_limit` and `message_limit`) for capping the number of tokens or messages used per sample ( `message_limit` replaces deprecated `max_messages`).
 - Add `metadata` field to `Task` and record in log `EvalSpec`.
 - Include datetime and level in file logger.
 - Correct llama3 and o1 tool calling when empty arguments passed.

{inspect_ai-0.3.60 → inspect_ai-0.3.61}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: inspect_ai
-Version: 0.3.60
+Version: 0.3.61
 Summary: Framework for large language model evaluations
 Author: UK AI Safety Institute
 License: MIT License
@@ -68,7 +68,7 @@ Requires-Dist: pytest-asyncio; extra == "dev"
 Requires-Dist: pytest-cov; extra == "dev"
 Requires-Dist: pytest-dotenv; extra == "dev"
 Requires-Dist: pytest-xdist; extra == "dev"
-Requires-Dist: ruff==0.9.3; extra == "dev"
+Requires-Dist: ruff==0.9.4; extra == "dev"
 Requires-Dist: textual-dev>=0.86.2; extra == "dev"
 Requires-Dist: types-PyYAML; extra == "dev"
 Requires-Dist: types-beautifulsoup4; extra == "dev"

{inspect_ai-0.3.60 → inspect_ai-0.3.61}/docs/_quarto.yml

@@ -36,18 +36,20 @@ website:
         - section: "Basics"
           contents:
             - tutorial.qmd
-            - workflow.qmd
-            - examples/index.qmd
+            - options.qmd
+            - text: "Evals"
+              href: evals/index.qmd
             - log-viewer.qmd
             - text: "VS Code"
               href: vscode.qmd
 
         - section: "Components"
           contents: 
+            - tasks.qmd
+            - datasets.qmd
             - solvers.qmd
             - tools.qmd
             - scorers.qmd
-            - datasets.qmd
             - models.qmd
 
         - section: "Agents"

{inspect_ai-0.3.60 → inspect_ai-0.3.61}/docs/_tools-scaffold.md

@@ -16,27 +16,30 @@ def agent_loop(message_limit: int = 50):
         # establish messages limit so we have a termination condition
         state.message_limit = message_limit
 
-        # call the model in a loop
-        while not state.completed:
-            # call model
-            output = await get_model().generate(state.messages, state.tools)
-
-            # update state
-            state.output = output
-            state.messages.append(output.message)
-
-            # make tool calls or terminate if there are none
-            if output.message.tool_calls:
-                state.messages.extend(call_tools(output.message, state.tools))
-            else:
-                break
+        try:
+            # call the model in a loop
+            while not state.completed:
+                # call model
+                output = await get_model().generate(state.messages, state.tools)
+
+                # update state
+                state.output = output
+                state.messages.append(output.message)
+
+                # make tool calls or terminate if there are none
+                if output.message.tool_calls:
+                    state.messages.extend(call_tools(output.message, state.tools))
+                else:
+                    break
+        except SampleLimitExceededError as ex:
+            raise ex.with_state(state)
 
         return state
 
     return solve
 ```
 
-Solvers can set the `state.completed` flag to indicate that the sample is complete, so we check it at the top of the loop.
+Solvers can set the `state.completed` flag to indicate that the sample is complete, so we check it at the top of the loop. When sample limits (e.g. tokens or messages) are exceeded an exception is thrown, so we re-raise it along with the current state of our agent loop.
 
 You can imagine several ways you might want to customise this loop:
 

{inspect_ai-0.3.60 → inspect_ai-0.3.61}/docs/errors-and-limits.qmd

@@ -55,7 +55,7 @@ eval("intercode_ctf.py", fail_on_error=False)
 
 You might choose to do this if you want to tolerate a certain proportion of errors during development but want to ensure there are never errors when running in production.
 
-## Sample Limits {#sec-sample-limits}
+## Sample Limits {#sample-limits}
 
 In open-ended model conversations (for example, an agent evaluation with tool usage) it's possible that a model will get "stuck" attempting to perform a task with no realistic prospect of completing it. Further, sometimes models will call commands in a sandbox that take an extremely long time (or worst case, hang indefinitely).
 
@@ -170,7 +170,7 @@ It's important to note that the `token_limit` is for all tokens used within the
 When limits are exceeded, a `SampleLimitExceededError` is raised and caught by the main Inspect sample execution logic. If you want to create custom limit types, you can enforce them by raising a `SampleLimitExceededError` as follows:
 
 ```python
-from inspect_ai.util import SampleLimitExceededError
+from inspect_ai.solver import SampleLimitExceededError
 
 raise SampleLimitExceededError(
     "custom", 

inspect_ai-0.3.61/docs/evals/evals.css

@@ -0,0 +1,74 @@
+#refs {
+    display: none;
+  }
+  
+  
+  .listing-actions-group {
+    display: block;
+  }
+  
+  .quarto-listing {
+    margin-top: 2em;
+  }
+  
+  .quarto-listing-filter {
+    width: 200px;
+    position: absolute;
+    top: -8px;
+    right: 0;
+  }
+  
+  .listing-actions-group {
+    position: relative;
+  }
+  
+  #quarto-margin-sidebar {
+    margin-top: 1.5em;
+  }
+  
+  h2 {
+    margin-top: 0.7em;
+    margin-bottom: 0.6em;
+  }
+  
+  .listing-actions-group {
+    margin-bottom: 1em;
+  }
+  
+  
+  h3 {
+    margin-top: 0;
+  }
+  
+  ul.no-bullets {
+    list-style-type: none;
+    padding: 0;
+    margin: 0;
+  }
+  
+  li.group {
+    padding-bottom: 10px;
+  }
+  
+  li.example {
+    margin-bottom: 0.6em;
+  }
+  
+  li.example a {
+    text-decoration: none;
+  }
+  
+  div.example-card {
+    display: flex;
+  }
+  
+  div.example-icon {
+    width: 30px;
+    flex-shrink: 0;
+  }
+  
+  div.example-info {
+    flex-grow: 1;
+  }
+
+  

inspect_ai-0.3.60/docs/examples/examples.ejs → inspect_ai-0.3.61/docs/evals/evals.ejs

@@ -8,6 +8,8 @@ function groupIcon(group) {
     case "Assistants":
       return "info-circle";
     case "Cybersecurity":
+      return "laptop";
+    case "Safeguards":
       return "shield-lock";
     case "Mathematics":
       return "calculator";