inspect-evals-utils 0.1.0__tar.gz

inspect_evals_utils-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Matt Fisher
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

inspect_evals_utils-0.1.0/PKG-INFO

@@ -0,0 +1,86 @@
+Metadata-Version: 2.4
+Name: inspect-evals-utils
+Version: 0.1.0
+Summary: Runtime helpers for Inspect AI evaluations: dataset loading with retries, sample ids, prompts and README generation
+Author: Matt Fisher
+Author-email: Matt Fisher <m@ttfisher.com>
+License-Expression: MIT
+License-File: LICENSE
+Requires-Dist: backoff>=2.2.0
+Requires-Dist: inspect-ai>=0.3.262
+Requires-Dist: platformdirs
+Requires-Dist: pyyaml>=5.1.0
+Requires-Dist: datasets>=4.8.5 ; extra == 'huggingface'
+Requires-Dist: hf-xet ; extra == 'huggingface'
+Requires-Dist: httpx>=0.27.0 ; extra == 'huggingface'
+Requires-Dist: huggingface-hub>=1.2.0 ; extra == 'huggingface'
+Requires-Dist: requests>=2.32.0 ; extra == 'huggingface'
+Requires-Python: >=3.11
+Project-URL: Homepage, https://github.com/Generality-Labs/inspect-evals-utils
+Project-URL: Repository, https://github.com/Generality-Labs/inspect-evals-utils
+Project-URL: Issues, https://github.com/Generality-Labs/inspect-evals-utils/issues
+Provides-Extra: huggingface
+Description-Content-Type: text/markdown
+
+# inspect-evals-utils
+
+Runtime helpers for [Inspect AI](https://inspect.aisi.org.uk/) evaluations, moved out of [inspect_evals](https://github.com/UKGovernmentBEIS/inspect_evals) so that a standalone eval repo can use them without depending on the whole registry. inspect_evals will depend on this package and re-export everything under its old `inspect_evals.utils` paths, so existing imports keep working.
+
+## Install
+
+```bash
+uv add inspect-evals-utils
+# or, for the Hugging Face wrappers:
+uv add "inspect-evals-utils[huggingface]"
+```
+
+## Modules
+
+- `inspect_evals_utils.ids`: `create_stable_id`, `filter_duplicate_ids`, `drop_known_broken`, `require_optional_dependency`, `DuplicateIdError`, `KnownBrokenSampleError`.
+- `inspect_evals_utils.datasets`: `load_json_dataset` and `load_csv_dataset`, which cache remote files and retry transient errors, plus `cache_dataset_loader`, `is_remote_url`, `get_cached_fname`, `is_transient_error`.
+- `inspect_evals_utils.huggingface` (needs the `huggingface` extra): `hf_dataset`, `load_dataset`, `get_dataset_split_names`, `snapshot_download`, `hf_hub_download`, `sentence_transformer`, `transformers_pipeline`, `auto_model_for_sequence_classification`, `auto_tokenizer`. Each retries on rate limits and transient errors and requires a `revision=` keyword, so datasets are pinned. `set_backoff_config` changes the retry policy for later calls (the default is `CI_BACKOFF_CONFIG` when the `CI` environment variable is set, otherwise `STANDARD_BACKOFF_CONFIG`). `set_hf_call_recorder` and `set_backoff_recorder` register callbacks that test sessions use to count calls and retries; both default to doing nothing.
+- `inspect_evals_utils.code`: `extract_code_blocks`, `extract_code_block`, `strip_json_fence`, `defines`.
+- `inspect_evals_utils.messages`: `message_content_text`, `reasoning_to_text`.
+- `inspect_evals_utils.prompts`: `Prompt`, a dataclass recording a prompt with its provenance.
+- `inspect_evals_utils.shuffle`: `shuffle_and_seed`.
+- `inspect_evals_utils.cache`: `cache_path()` and `INSPECT_EVALS_CACHE_PATH`.
+
+The most-used names are also importable from the top level, for example `from inspect_evals_utils import create_stable_id`. The Hugging Face module is not, so the base package imports without the extra.
+
+## Cache directory
+
+Downloads land in `INSPECT_EVALS_CACHE_PATH`, which defaults to the platform user cache directory for `inspect_evals`. Set `INSPECT_EVALS_CACHE_DIR` to move it or point it at a pre-populated directory. It must be absolute or start with `~`, and it is read once when the package is imported. inspect_evals reads the same variable and default, so one machine has one cache.
+
+## README generation
+
+The `inspect-evals-readmes` command rewrites the generated sections of each eval's README (Contributors, Usage, Options and Parameters) from its `eval.yaml` and `@task` signatures. It finds evals as `<src>/<eval>/eval.yaml` (default `src`), imports each package to read the task parameters, and writes between the `<!-- Usage: Automatically Generated -->` style tag pairs.
+
+```bash
+uv run inspect-evals-readmes generate                      # rewrite the sections
+uv run inspect-evals-readmes generate --check              # exit 1 if any README is out of date; writes nothing
+uv run inspect-evals-readmes generate --eval my_eval --create-missing
+uv run inspect-evals-readmes add-section --key Parameters --insert-after Options
+```
+
+`add-section` inserts a missing tag pair after the first listed section it finds, or at the end of the file.
+
+## Private inspect_ai APIs
+
+`inspect_evals_utils.datasets` uses `inspect_ai._util.file.file` to open local and remote files through one interface. There is no public equivalent yet, so a future inspect_ai release could break it.
+
+## Development
+
+```bash
+uv sync
+uv run pre-commit install   # optional: run the lint stack on every commit
+uv run pytest
+uv run basedpyright src
+```
+
+Each pull request adds a changelog fragment rather than editing `CHANGELOG.md`, so concurrent PRs don't conflict. Run `uv run scriv create`, uncomment the sections that apply in the new file under `changelog.d/`, and commit it with the change, or delete it if the change needs no entry. To release, run the **Prepare release** workflow from the Actions tab. It bumps `version` in `pyproject.toml`, collects the fragments into `CHANGELOG.md`, and opens a release pull request, whose CI starts once you click **Approve workflows to run** on it; merging it tags the release. It needs _Settings → Actions → General_ → **Allow GitHub Actions to create and approve pull requests**. By hand, the same is `uv version --bump minor` and `uv run scriv collect`.
+
+Linting (ruff, [zizmor](https://docs.zizmor.sh/), mdformat) runs via [pre-commit](https://pre-commit.com); CI runs the same stack plus basedpyright and pytest via the shared [`python-ci`](https://github.com/Generality-Labs/python-project-template) reusable workflow.
+
+## Releasing
+
+See [RELEASING.md](RELEASING.md).

inspect_evals_utils-0.1.0/README.md

@@ -0,0 +1,62 @@
+# inspect-evals-utils
+
+Runtime helpers for [Inspect AI](https://inspect.aisi.org.uk/) evaluations, moved out of [inspect_evals](https://github.com/UKGovernmentBEIS/inspect_evals) so that a standalone eval repo can use them without depending on the whole registry. inspect_evals will depend on this package and re-export everything under its old `inspect_evals.utils` paths, so existing imports keep working.
+
+## Install
+
+```bash
+uv add inspect-evals-utils
+# or, for the Hugging Face wrappers:
+uv add "inspect-evals-utils[huggingface]"
+```
+
+## Modules
+
+- `inspect_evals_utils.ids`: `create_stable_id`, `filter_duplicate_ids`, `drop_known_broken`, `require_optional_dependency`, `DuplicateIdError`, `KnownBrokenSampleError`.
+- `inspect_evals_utils.datasets`: `load_json_dataset` and `load_csv_dataset`, which cache remote files and retry transient errors, plus `cache_dataset_loader`, `is_remote_url`, `get_cached_fname`, `is_transient_error`.
+- `inspect_evals_utils.huggingface` (needs the `huggingface` extra): `hf_dataset`, `load_dataset`, `get_dataset_split_names`, `snapshot_download`, `hf_hub_download`, `sentence_transformer`, `transformers_pipeline`, `auto_model_for_sequence_classification`, `auto_tokenizer`. Each retries on rate limits and transient errors and requires a `revision=` keyword, so datasets are pinned. `set_backoff_config` changes the retry policy for later calls (the default is `CI_BACKOFF_CONFIG` when the `CI` environment variable is set, otherwise `STANDARD_BACKOFF_CONFIG`). `set_hf_call_recorder` and `set_backoff_recorder` register callbacks that test sessions use to count calls and retries; both default to doing nothing.
+- `inspect_evals_utils.code`: `extract_code_blocks`, `extract_code_block`, `strip_json_fence`, `defines`.
+- `inspect_evals_utils.messages`: `message_content_text`, `reasoning_to_text`.
+- `inspect_evals_utils.prompts`: `Prompt`, a dataclass recording a prompt with its provenance.
+- `inspect_evals_utils.shuffle`: `shuffle_and_seed`.
+- `inspect_evals_utils.cache`: `cache_path()` and `INSPECT_EVALS_CACHE_PATH`.
+
+The most-used names are also importable from the top level, for example `from inspect_evals_utils import create_stable_id`. The Hugging Face module is not, so the base package imports without the extra.
+
+## Cache directory
+
+Downloads land in `INSPECT_EVALS_CACHE_PATH`, which defaults to the platform user cache directory for `inspect_evals`. Set `INSPECT_EVALS_CACHE_DIR` to move it or point it at a pre-populated directory. It must be absolute or start with `~`, and it is read once when the package is imported. inspect_evals reads the same variable and default, so one machine has one cache.
+
+## README generation
+
+The `inspect-evals-readmes` command rewrites the generated sections of each eval's README (Contributors, Usage, Options and Parameters) from its `eval.yaml` and `@task` signatures. It finds evals as `<src>/<eval>/eval.yaml` (default `src`), imports each package to read the task parameters, and writes between the `<!-- Usage: Automatically Generated -->` style tag pairs.
+
+```bash
+uv run inspect-evals-readmes generate                      # rewrite the sections
+uv run inspect-evals-readmes generate --check              # exit 1 if any README is out of date; writes nothing
+uv run inspect-evals-readmes generate --eval my_eval --create-missing
+uv run inspect-evals-readmes add-section --key Parameters --insert-after Options
+```
+
+`add-section` inserts a missing tag pair after the first listed section it finds, or at the end of the file.
+
+## Private inspect_ai APIs
+
+`inspect_evals_utils.datasets` uses `inspect_ai._util.file.file` to open local and remote files through one interface. There is no public equivalent yet, so a future inspect_ai release could break it.
+
+## Development
+
+```bash
+uv sync
+uv run pre-commit install   # optional: run the lint stack on every commit
+uv run pytest
+uv run basedpyright src
+```
+
+Each pull request adds a changelog fragment rather than editing `CHANGELOG.md`, so concurrent PRs don't conflict. Run `uv run scriv create`, uncomment the sections that apply in the new file under `changelog.d/`, and commit it with the change, or delete it if the change needs no entry. To release, run the **Prepare release** workflow from the Actions tab. It bumps `version` in `pyproject.toml`, collects the fragments into `CHANGELOG.md`, and opens a release pull request, whose CI starts once you click **Approve workflows to run** on it; merging it tags the release. It needs _Settings → Actions → General_ → **Allow GitHub Actions to create and approve pull requests**. By hand, the same is `uv version --bump minor` and `uv run scriv collect`.
+
+Linting (ruff, [zizmor](https://docs.zizmor.sh/), mdformat) runs via [pre-commit](https://pre-commit.com); CI runs the same stack plus basedpyright and pytest via the shared [`python-ci`](https://github.com/Generality-Labs/python-project-template) reusable workflow.
+
+## Releasing
+
+See [RELEASING.md](RELEASING.md).

inspect_evals_utils-0.1.0/pyproject.toml

@@ -0,0 +1,128 @@
+[build-system]
+requires = ["uv_build>=0.12.22,<0.14.0"]
+build-backend = "uv_build"
+
+[project]
+name = "inspect-evals-utils"
+version = "0.1.0"
+description = "Runtime helpers for Inspect AI evaluations: dataset loading with retries, sample ids, prompts and README generation"
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+requires-python = ">=3.11"
+dependencies = [
+    "backoff>=2.2.0",
+    "inspect_ai>=0.3.262",
+    "platformdirs",
+    "pyyaml>=5.1.0",
+]
+
+[[project.authors]]
+name = "Matt Fisher"
+email = "m@ttfisher.com"
+
+[project.optional-dependencies]
+huggingface = [
+    "datasets>=4.8.5",
+    "hf_xet",
+    "httpx>=0.27.0",
+    "huggingface_hub>=1.2.0",
+    "requests>=2.32.0",
+]
+
+[project.scripts]
+inspect-evals-readmes = "inspect_evals_utils.readmes.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/Generality-Labs/inspect-evals-utils"
+Repository = "https://github.com/Generality-Labs/inspect-evals-utils"
+Issues = "https://github.com/Generality-Labs/inspect-evals-utils/issues"
+
+[dependency-groups]
+dev = [
+    "pytest>=8.4.0",
+    "pytest-cov>=7.0.0",
+    "basedpyright>=1.39",
+    "pre-commit>=4.0.0",
+    "ruff==0.17.3",
+    "scriv>=1.8",
+    "datasets>=4.8.5",
+    "httpx>=0.27.0",
+    "huggingface_hub>=1.2.0",
+    "pytest-asyncio>=1.0",
+    "pytest-inspect-evals[huggingface]>=0.3.0",
+    "requests>=2.32.0",
+    "types-PyYAML",
+    "types-requests",
+]
+
+[tool.scriv]
+format = "md"
+fragment_directory = "changelog.d"
+categories = [
+    "Added",
+    "Changed",
+    "Deprecated",
+    "Removed",
+    "Fixed",
+    "Security",
+]
+skip_fragments = "[A-Z]*"
+new_fragment_template = "file: TEMPLATE.md"
+entry_title_template = "[{{ version }}] - {{ date.strftime('%Y-%m-%d') }}"
+md_header_level = "2"
+md_html_anchors = false
+compact_fragments = true
+version = "literal: pyproject.toml: project.version"
+
+[tool.ruff]
+line-length = 100
+src = ["src"]
+target-version = "py311"
+
+[tool.ruff.lint]
+select = [
+    "E",
+    "W",
+    "F",
+    "I",
+    "UP",
+    "B",
+    "SIM",
+    "D",
+    "C4",
+    "PT",
+    "PIE",
+    "DTZ",
+    "ISC",
+    "ASYNC",
+    "N",
+    "FURB",
+    "RUF",
+    "PLE",
+    "PLW",
+]
+ignore = [
+    "E501",
+    "D10",
+    "D415",
+    "ISC001",
+]
+
+[tool.ruff.lint.pydocstyle]
+convention = "google"
+
+[tool.ruff.lint.per-file-ignores]
+"tests/**" = ["D"]
+
+[tool.basedpyright]
+pythonVersion = "3.11"
+typeCheckingMode = "standard"
+reportMissingTypeStubs = false
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = "-rA --durations=10 --color=yes --cov=src --cov-report=term-missing"
+asyncio_mode = "auto"
+asyncio_default_fixture_loop_scope = "function"
+filterwarnings = ['ignore:The configuration option "asyncio_default_fixture_loop_scope" is unset:pytest.PytestDeprecationWarning']

inspect_evals_utils-0.1.0/pyproject.toml.orig

@@ -0,0 +1,134 @@
+[build-system]
+# uv's own backend. It builds src/inspect_evals_utils/ with no further config.
+# Keep the upper bound at the next uv minor, as `uv init` writes it, and raise
+# both bounds when moving to a new uv minor.
+requires = ["uv_build>=0.12.22,<0.14.0"]
+build-backend = "uv_build"
+
+[project]
+name = "inspect-evals-utils"
+version = "0.1.0"
+description = "Runtime helpers for Inspect AI evaluations: dataset loading with retries, sample ids, prompts and README generation"
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"] # uv_build ships only the license files named here
+authors = [{ name = "Matt Fisher", email = "m@ttfisher.com" }]
+requires-python = ">=3.11"
+dependencies = [
+    "backoff>=2.2.0",
+    "inspect_ai>=0.3.262",
+    "platformdirs",
+    "pyyaml>=5.1.0",
+]
+
+[project.optional-dependencies]
+# Only inspect_evals_utils.huggingface needs these.
+huggingface = [
+    "datasets>=4.8.5",
+    "hf_xet",
+    "httpx>=0.27.0",
+    "huggingface_hub>=1.2.0",
+    "requests>=2.32.0",
+]
+
+[project.scripts]
+inspect-evals-readmes = "inspect_evals_utils.readmes.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/Generality-Labs/inspect-evals-utils"
+Repository = "https://github.com/Generality-Labs/inspect-evals-utils"
+Issues = "https://github.com/Generality-Labs/inspect-evals-utils/issues"
+[dependency-groups]
+dev = [
+    "pytest>=8.4.0",
+    "pytest-cov>=7.0.0",
+    "basedpyright>=1.39",
+    "pre-commit>=4.0.0",
+    "ruff==0.17.3",
+    "scriv>=1.8",
+    "datasets>=4.8.5",
+    "httpx>=0.27.0",
+    "huggingface_hub>=1.2.0",
+    "pytest-asyncio>=1.0",
+    "pytest-inspect-evals[huggingface]>=0.3.0",
+    "requests>=2.32.0",
+    "types-PyYAML",
+    "types-requests",
+]
+
+
+[tool.scriv]
+# One fragment per pull request in changelog.d/, collected into CHANGELOG.md
+# at release time, so concurrent PRs never edit the same lines of the changelog.
+format = "md"
+fragment_directory = "changelog.d"
+categories = ["Added", "Changed", "Deprecated", "Removed", "Fixed", "Security"]
+skip_fragments = "[A-Z]*" # TEMPLATE.md
+new_fragment_template = "file: TEMPLATE.md"
+entry_title_template = "[{{ version }}] - {{ date.strftime('%Y-%m-%d') }}"
+md_header_level = "2"
+md_html_anchors = false
+# Without this, two fragments in one category leave a blank line between them,
+# which makes the list loose and fails mdformat on the release commit.
+compact_fragments = true
+# The version's single source is [project] version above, which
+# `uv version --bump` (and the Prepare release workflow) rewrites.
+version = "literal: pyproject.toml: project.version"
+
+[tool.ruff]
+line-length = 100
+src = ["src"]
+target-version = "py311"
+
+[tool.ruff.lint]
+select = [
+    "E",     # pycodestyle errors
+    "W",     # pycodestyle warnings
+    "F",     # pyflakes
+    "I",     # isort
+    "UP",    # pyupgrade
+    "B",     # flake8-bugbear
+    "SIM",   # flake8-simplify
+    "D",     # pydocstyle
+    "C4",    # flake8-comprehensions
+    "PT",    # flake8-pytest-style
+    "PIE",   # flake8-pie
+    "DTZ",   # flake8-datetimez (timezone-aware datetimes)
+    "ISC",   # implicit-str-concat (catches missing commas)
+    "ASYNC", # flake8-async
+    "N",     # pep8-naming
+    "FURB",  # refurb (modernization)
+    "RUF",   # ruff-specific rules
+    "PLE",   # pylint errors
+    "PLW",   # pylint warnings
+]
+ignore = [
+    "E501",   # line-too-long — the formatter owns line length
+    "D10",    # missing docstrings — don't force them
+    "D415",   # first-line punctuation — the module docstring is the project
+    # description, whose punctuation varies
+    "ISC001", # single-line implicit concat — conflicts with the formatter
+]
+
+[tool.ruff.lint.pydocstyle]
+convention = "google"
+
+[tool.ruff.lint.per-file-ignores]
+"tests/**" = ["D"] # tests don't need docstring linting
+
+[tool.basedpyright]
+pythonVersion = "3.11"
+# "standard", not "strict": v0.1 moves code verbatim from inspect_evals, which
+# was checked with mypy. Tighten once the code is ours.
+typeCheckingMode = "standard"
+reportMissingTypeStubs = false # third-party libs are often untyped
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = "-rA --durations=10 --color=yes --cov=src --cov-report=term-missing"
+asyncio_mode = "auto"
+asyncio_default_fixture_loop_scope = "function"
+filterwarnings = [
+    # pytester-style inner runs and moved tests have no pytest-asyncio config of their own.
+    'ignore:The configuration option "asyncio_default_fixture_loop_scope" is unset:pytest.PytestDeprecationWarning',
+]

inspect_evals_utils-0.1.0/src/inspect_evals_utils/__init__.py

@@ -0,0 +1,43 @@
+"""Runtime helpers for Inspect AI evaluations: dataset loading with retries, sample ids, prompts and README generation."""
+
+from importlib.metadata import PackageNotFoundError, version
+
+from inspect_evals_utils.cache import INSPECT_EVALS_CACHE_PATH
+from inspect_evals_utils.datasets import load_csv_dataset, load_json_dataset
+from inspect_evals_utils.ids import (
+    DuplicateIdError,
+    KnownBrokenSampleError,
+    create_stable_id,
+    drop_known_broken,
+    filter_duplicate_ids,
+    require_optional_dependency,
+)
+from inspect_evals_utils.messages import message_content_text, reasoning_to_text
+from inspect_evals_utils.prompts import Prompt
+from inspect_evals_utils.shuffle import shuffle_and_seed
+
+# The version is set once, in pyproject.toml; this reads it back from the
+# installed package metadata.
+try:
+    __version__ = version("inspect-evals-utils")
+except PackageNotFoundError:  # a source tree that was never installed
+    __version__ = "unknown"
+
+# inspect_evals_utils.huggingface is not imported here: it needs the
+# huggingface extra, and the base package must import without it.
+__all__ = [
+    "INSPECT_EVALS_CACHE_PATH",
+    "DuplicateIdError",
+    "KnownBrokenSampleError",
+    "Prompt",
+    "__version__",
+    "create_stable_id",
+    "drop_known_broken",
+    "filter_duplicate_ids",
+    "load_csv_dataset",
+    "load_json_dataset",
+    "message_content_text",
+    "reasoning_to_text",
+    "require_optional_dependency",
+    "shuffle_and_seed",
+]

inspect_evals_utils-0.1.0/src/inspect_evals_utils/cache.py

@@ -0,0 +1,33 @@
+"""The cache directory shared by inspect_evals and standalone evals.
+
+Set INSPECT_EVALS_CACHE_DIR to move it or point it at a pre-populated directory.
+"""
+
+import os
+from pathlib import Path
+
+from platformdirs import user_cache_dir
+
+
+def cache_path() -> Path:
+    """Resolve the cache directory, rejecting a value that depends on the cwd."""
+    configured = os.environ.get("INSPECT_EVALS_CACHE_DIR")
+    if not configured:
+        return Path(user_cache_dir("inspect_evals"))
+
+    # A leading ~ survives a .env file and a quoted shell assignment, which is
+    # how this variable is usually set, so expand it rather than treating it as
+    # a directory name.
+    path = Path(configured).expanduser()
+    if not path.is_absolute():
+        raise ValueError(
+            f"INSPECT_EVALS_CACHE_DIR must be absolute or start with ~, but is "
+            f"{configured!r}. It is read once when inspect_evals_utils is imported, so "
+            f"a relative path would point somewhere different depending on where "
+            f"the eval was started from."
+        )
+    return path
+
+
+# Read once at import, because evals derive their own module-level paths from it.
+INSPECT_EVALS_CACHE_PATH = cache_path()

inspect_evals_utils-0.1.0/src/inspect_evals_utils/code.py

@@ -0,0 +1,123 @@
+"""Helpers for extracting generated code and JSON from Markdown responses."""
+
+import ast
+import re
+from collections.abc import Callable
+
+_LANGUAGE_TAGS = {
+    "python": ("python", "python3", "py"),
+    "cpp": ("cpp", "c++", "cc", "cxx", "c"),
+    "java": ("java",),
+    "javascript": ("javascript", "js"),
+    "typescript": ("typescript", "ts"),
+    "csharp": ("csharp", "cs", "c#"),
+    "php": ("php",),
+    "shell": ("bash", "sh", "shell"),
+}
+
+# A block ends at a bare fence line, just before any other fence line (which a model
+# may write without closing the block first), at a fence glued to the end of the last
+# code line, or at the end of the text when the fence is never closed. The last case is
+# its own group so that an unclosed unlabelled fence, usually a stray closing fence
+# after unfenced code, can be ignored.
+_CLOSE = r"(?:^```[ \t\r]*$|(?=^```)|(?<=\S)```[ \t\r]*$|(\Z))"
+
+_FENCE = re.compile(
+    r"^```[ \t]*([^\s`]*)[^\n]*\r?\n(.*?)" + _CLOSE,
+    re.DOTALL | re.MULTILINE,
+)
+
+
+_JSON_FENCE = re.compile(
+    r"\A\s*```[ \t]*(?:json)?[ \t]*\r?\n(.*?)\s*```\s*\Z", re.DOTALL | re.IGNORECASE
+)
+
+
+def _labelled_fence(tags: tuple[str, ...]) -> re.Pattern[str]:
+    # A labelled fence may follow prose on the same line, but only when nothing
+    # follows the label, so that a fence mentioned in a sentence is not an opener.
+    tag = "|".join(re.escape(t) for t in sorted(tags, key=len, reverse=True))
+    opener = rf"(?:^```[ \t]*(?:{tag})(?=[\s`])[^\n]*|```[ \t]*(?:{tag})[ \t]*)\r?\n"
+    return re.compile(opener + "(.*?)" + _CLOSE, re.DOTALL | re.MULTILINE | re.IGNORECASE)
+
+
+def extract_code_blocks(completion: str, language: str = "python") -> list[str]:
+    """Return the language-labelled code blocks, or else the unlabelled ones.
+
+    Blocks are in the order they appear. Unlabelled blocks are returned only
+    when no block carries one of the language's labels. Labels are matched
+    case-insensitively and both LF and CRLF line endings are accepted. Fences
+    labelled for another language are not treated as unlabelled code, and empty
+    blocks are skipped. A labelled fence that is never closed runs to the end of
+    the text.
+    """
+    tags = _LANGUAGE_TAGS.get(language.lower(), (language.lower(),))
+    labelled = [
+        match.group(1)
+        for match in _labelled_fence(tags).finditer(completion)
+        if match.group(1).strip()
+    ]
+    if labelled:
+        return labelled
+    return [
+        match.group(2)
+        for match in _FENCE.finditer(completion)
+        if match.group(1) == "" and match.group(2).strip() and match.group(3) is None
+    ]
+
+
+def extract_code_block(
+    completion: str,
+    language: str = "python",
+    accept: Callable[[str], bool] | None = None,
+) -> str | None:
+    """Return the first block from `extract_code_blocks`, or None if there is none.
+
+    Language-labelled blocks are preferred over unlabelled blocks even when an
+    unlabelled block appears earlier in the response.
+
+    When `accept` is given, the first block it accepts is returned instead. If
+    it accepts none, the first block is returned as without it. Pass
+    `defines(name)` to take the block that implements the function the task
+    asks for, rather than a usage example or a draft that does not parse.
+    """
+    blocks = extract_code_blocks(completion, language)
+
+    if accept is not None:
+        for block in blocks:
+            if accept(block):
+                return block
+
+    return blocks[0] if blocks else None
+
+
+def strip_json_fence(text: str) -> str:
+    """Return the JSON inside one code fence wrapping the whole text, else the text unchanged.
+
+    The fence may be labelled `json` or unlabelled. Use it before parsing a
+    reply that should be bare JSON, such as one requested with a
+    `response_schema`: some providers accept the schema without enforcing it and
+    return the JSON in a fence. Text with prose around the fence, or more than
+    one fence, is returned unchanged, so it still fails validation.
+    """
+    match = _JSON_FENCE.match(text)
+    if match is None or re.search(r"^[ \t]*```", match.group(1), re.MULTILINE):
+        return text
+    return match.group(1)
+
+
+def defines(name: str) -> Callable[[str], bool]:
+    """Accept Python code that parses and defines `name` at the top level."""
+
+    def check(code: str) -> bool:
+        try:
+            tree = ast.parse(code)
+        except (SyntaxError, ValueError):
+            return False
+        return any(
+            isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef))
+            and node.name == name
+            for node in tree.body
+        )
+
+    return check