pytest-skill-engineering 1.0.2__tar.gz → 1.0.4__tar.gz

pytest_skill_engineering-1.0.4/PKG-INFO

@@ -0,0 +1,175 @@
+Metadata-Version: 2.5
+Name: pytest-skill-engineering
+Version: 1.0.4
+Summary: Reliable Copilot execution and evidence for coding-agent-led testing of MCP servers, CLI tools, skills, and custom agents.
+Project-URL: Homepage, https://github.com/sbroenne/pytest-skill-engineering
+Project-URL: Repository, https://github.com/sbroenne/pytest-skill-engineering
+Author: Stefan Brunner
+License: MIT
+License-File: LICENSE
+Keywords: agents,ai,custom-agents,llm,mcp,pytest,skill-engineering,skills,testing
+Classifier: Development Status :: 4 - Beta
+Classifier: Framework :: Pytest
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.11
+Requires-Dist: github-copilot-sdk>=1.0.28
+Requires-Dist: httpx>=0.28.1
+Requires-Dist: mcp>=2.3.0
+Requires-Dist: pydantic>=2.14.1
+Requires-Dist: pytest-asyncio>=1.4.1
+Requires-Dist: pytest>=9.2.0
+Requires-Dist: python-frontmatter>=1.3.0
+Requires-Dist: tomlkit>=0.15.2
+Requires-Dist: trio>=0.34.0
+Requires-Dist: websockets>=17.3
+Provides-Extra: dev
+Requires-Dist: pre-commit>=4.7.0; extra == 'dev'
+Requires-Dist: pyright>=1.1.414; extra == 'dev'
+Requires-Dist: pytest-cov>=7.1.0; extra == 'dev'
+Requires-Dist: python-dotenv>=1.2.4; extra == 'dev'
+Requires-Dist: ruff>=0.17.3; extra == 'dev'
+Requires-Dist: typeguard>=4.6.0; extra == 'dev'
+Provides-Extra: docs
+Requires-Dist: cairosvg>=2.9.1; extra == 'docs'
+Requires-Dist: mkdocs-material<9.8,>=9.7.7; extra == 'docs'
+Requires-Dist: mkdocs>=1.6.1; extra == 'docs'
+Requires-Dist: mkdocstrings[python]>=1.0.8; extra == 'docs'
+Requires-Dist: pillow>=13.0.0; extra == 'docs'
+Provides-Extra: test
+Requires-Dist: syrupy<7,>=6.1.1; extra == 'test'
+Description-Content-Type: text/markdown
+
+# pytest-skill-engineering
+
+[![PyPI version](https://img.shields.io/pypi/v/pytest-skill-engineering)](https://pypi.org/project/pytest-skill-engineering/)
+[![CI](https://github.com/sbroenne/pytest-skill-engineering/actions/workflows/ci.yml/badge.svg)](https://github.com/sbroenne/pytest-skill-engineering/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+
+**Test whether GitHub Copilot can complete tasks with your tools and instructions.**
+
+pytest-skill-engineering runs real Copilot sessions from pytest. Give Copilot a
+task, check the actual output with ordinary assertions, and save the execution
+as JSON.
+
+Use it to test MCP servers, CLI tools, skills, system prompts, plugins, and custom
+agents. It answers a different question from your tool's unit tests: **can
+Copilot use the interface correctly?** A working tool can still have confusing
+descriptions or arguments that lead the agent to the wrong result.
+
+You define what correctness means. The framework manages the Copilot session,
+execution limits, usage, and evidence; it does not grade answers with another AI.
+
+## Quick start
+
+You need Python 3.11+, [uv](https://docs.astral.sh/uv/), and a GitHub account with
+Copilot access.
+
+Run these commands in an existing Python project with a `pyproject.toml`:
+
+```bash
+uv add pytest-skill-engineering
+gh auth login --hostname github.com
+uv run pytest-skill-engineering init
+uv run pytest-skill-engineering doctor
+uv run python -m pytest tests/test_copilot_eval.py -v
+```
+
+`init` creates a test using the bundled Todo MCP server, configures JSON output,
+and adds starter-model rates to `pricing.toml`. It reports conflicts rather than
+overwriting existing files or settings. `doctor` checks setup and Copilot access.
+Real task execution may consume Copilot premium requests.
+
+The test checks the tool's arguments and returned task state. pytest prints the
+result; `aitest-reports/results.json` preserves the execution. See the
+[setup guide](docs/getting-started/index.md) or the standalone
+[quickstart project](examples/quickstart/).
+
+## A test that checks the result
+
+```python
+from __future__ import annotations
+
+import json
+
+from pytest_skill_engineering.copilot import CopilotEval
+
+
+async def test_order_total(copilot_eval, tmp_path):
+    agent = CopilotEval(
+        name="order-total",
+        model="gpt-5.6-sol",
+        instructions="Save the requested output as JSON. Use integer cents.",
+        working_directory=str(tmp_path),
+    )
+    result = await copilot_eval(
+        agent,
+        "Save order.json for two items at 1250 cents each plus 20% tax. "
+        "Include subtotal_cents, tax_cents, and total_cents.",
+    )
+    assert result.success, result.error
+    output = tmp_path / "order.json"
+    assert output.is_file()
+    assert json.loads(output.read_text(encoding="utf-8")) == {
+        "subtotal_cents": 2500,
+        "tax_cents": 500,
+        "total_cents": 3000,
+    }
+```
+
+`CopilotEval` configures the session. The second argument to `copilot_eval` is
+the user task. `result.success` checks execution completion; the file assertions
+check that the task produced the right result.
+
+For your own tools, check the resulting application state, saved files, or
+operation output. A tool call or a model's claim is not proof of task success.
+Use approved isolation for live tests: a temporary directory is not a sandbox,
+and generated code or tool calls can act with your user's access.
+
+## What the framework provides
+
+| You need | Use |
+| --- | --- |
+| Real task execution | `CopilotEval` and the `copilot_eval` pytest fixture |
+| Controlled runs | Timeouts, tool budgets, request controls, and permission settings |
+| Debugging evidence | Configuration, calls, arguments, outputs, errors, and completion records |
+| Repeatable comparisons | `ab_run` and pytest repetitions |
+| Usage visibility | Captured tokens, Copilot premium requests, and explicit price estimates |
+| Saved results | Native JSON with execution evidence and ordinary pytest outcomes |
+
+Your tests supply application fixtures, guarded tool adapters, and independent
+output checks. The framework makes one execution attempt per call; it does not
+retry failures automatically. Missing usage, prices, and incomplete evidence
+remain visible.
+
+## Investigate and compare
+
+Give your coding agent a failed test, its JSON evidence, the requirements, and
+the relevant source. It can investigate the failure and help you change the
+interface. Keep the success criteria fixed when rerunning a comparison.
+
+For A/B tests, record verification separately for each side: both saved
+executions share one pytest outcome. That outcome alone cannot establish
+independent pass rates or a causal improvement. See
+[comparing configurations](docs/getting-started/comparing.md).
+
+## Documentation
+
+- [Getting started](https://sbroenne.github.io/pytest-skill-engineering/getting-started/)
+- [Task guides](docs/how-to/index.md) and [API reference](docs/reference/api.md)
+- [Examples](examples/README.md), including historical skill comparisons
+- [Upgrading from 0.6.x](docs/migration.md): 1.0 removed AI judging, renderers,
+  rankings, and skill refinement
+
+No framework companion skill is required or distributed. Testing your own
+domain skills is supported.
+
+## License
+
+MIT. Inspired by [agent-benchmark](https://github.com/mykhaliev/agent-benchmark).

pytest_skill_engineering-1.0.4/README.md

@@ -0,0 +1,127 @@
+# pytest-skill-engineering
+
+[![PyPI version](https://img.shields.io/pypi/v/pytest-skill-engineering)](https://pypi.org/project/pytest-skill-engineering/)
+[![CI](https://github.com/sbroenne/pytest-skill-engineering/actions/workflows/ci.yml/badge.svg)](https://github.com/sbroenne/pytest-skill-engineering/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+
+**Test whether GitHub Copilot can complete tasks with your tools and instructions.**
+
+pytest-skill-engineering runs real Copilot sessions from pytest. Give Copilot a
+task, check the actual output with ordinary assertions, and save the execution
+as JSON.
+
+Use it to test MCP servers, CLI tools, skills, system prompts, plugins, and custom
+agents. It answers a different question from your tool's unit tests: **can
+Copilot use the interface correctly?** A working tool can still have confusing
+descriptions or arguments that lead the agent to the wrong result.
+
+You define what correctness means. The framework manages the Copilot session,
+execution limits, usage, and evidence; it does not grade answers with another AI.
+
+## Quick start
+
+You need Python 3.11+, [uv](https://docs.astral.sh/uv/), and a GitHub account with
+Copilot access.
+
+Run these commands in an existing Python project with a `pyproject.toml`:
+
+```bash
+uv add pytest-skill-engineering
+gh auth login --hostname github.com
+uv run pytest-skill-engineering init
+uv run pytest-skill-engineering doctor
+uv run python -m pytest tests/test_copilot_eval.py -v
+```
+
+`init` creates a test using the bundled Todo MCP server, configures JSON output,
+and adds starter-model rates to `pricing.toml`. It reports conflicts rather than
+overwriting existing files or settings. `doctor` checks setup and Copilot access.
+Real task execution may consume Copilot premium requests.
+
+The test checks the tool's arguments and returned task state. pytest prints the
+result; `aitest-reports/results.json` preserves the execution. See the
+[setup guide](docs/getting-started/index.md) or the standalone
+[quickstart project](examples/quickstart/).
+
+## A test that checks the result
+
+```python
+from __future__ import annotations
+
+import json
+
+from pytest_skill_engineering.copilot import CopilotEval
+
+
+async def test_order_total(copilot_eval, tmp_path):
+    agent = CopilotEval(
+        name="order-total",
+        model="gpt-5.6-sol",
+        instructions="Save the requested output as JSON. Use integer cents.",
+        working_directory=str(tmp_path),
+    )
+    result = await copilot_eval(
+        agent,
+        "Save order.json for two items at 1250 cents each plus 20% tax. "
+        "Include subtotal_cents, tax_cents, and total_cents.",
+    )
+    assert result.success, result.error
+    output = tmp_path / "order.json"
+    assert output.is_file()
+    assert json.loads(output.read_text(encoding="utf-8")) == {
+        "subtotal_cents": 2500,
+        "tax_cents": 500,
+        "total_cents": 3000,
+    }
+```
+
+`CopilotEval` configures the session. The second argument to `copilot_eval` is
+the user task. `result.success` checks execution completion; the file assertions
+check that the task produced the right result.
+
+For your own tools, check the resulting application state, saved files, or
+operation output. A tool call or a model's claim is not proof of task success.
+Use approved isolation for live tests: a temporary directory is not a sandbox,
+and generated code or tool calls can act with your user's access.
+
+## What the framework provides
+
+| You need | Use |
+| --- | --- |
+| Real task execution | `CopilotEval` and the `copilot_eval` pytest fixture |
+| Controlled runs | Timeouts, tool budgets, request controls, and permission settings |
+| Debugging evidence | Configuration, calls, arguments, outputs, errors, and completion records |
+| Repeatable comparisons | `ab_run` and pytest repetitions |
+| Usage visibility | Captured tokens, Copilot premium requests, and explicit price estimates |
+| Saved results | Native JSON with execution evidence and ordinary pytest outcomes |
+
+Your tests supply application fixtures, guarded tool adapters, and independent
+output checks. The framework makes one execution attempt per call; it does not
+retry failures automatically. Missing usage, prices, and incomplete evidence
+remain visible.
+
+## Investigate and compare
+
+Give your coding agent a failed test, its JSON evidence, the requirements, and
+the relevant source. It can investigate the failure and help you change the
+interface. Keep the success criteria fixed when rerunning a comparison.
+
+For A/B tests, record verification separately for each side: both saved
+executions share one pytest outcome. That outcome alone cannot establish
+independent pass rates or a causal improvement. See
+[comparing configurations](docs/getting-started/comparing.md).
+
+## Documentation
+
+- [Getting started](https://sbroenne.github.io/pytest-skill-engineering/getting-started/)
+- [Task guides](docs/how-to/index.md) and [API reference](docs/reference/api.md)
+- [Examples](examples/README.md), including historical skill comparisons
+- [Upgrading from 0.6.x](docs/migration.md): 1.0 removed AI judging, renderers,
+  rankings, and skill refinement
+
+No framework companion skill is required or distributed. Testing your own
+domain skills is supported.
+
+## License
+
+MIT. Inspired by [agent-benchmark](https://github.com/mykhaliev/agent-benchmark).

{pytest_skill_engineering-1.0.2 → pytest_skill_engineering-1.0.4}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "pytest-skill-engineering"
-version = "1.0.2"
+version = "1.0.4"
 description = "Reliable Copilot execution and evidence for coding-agent-led testing of MCP servers, CLI tools, skills, and custom agents."
 readme = "README.md"
 license = { text = "MIT" }
@@ -26,25 +26,25 @@ classifiers = [
     "Topic :: Software Development :: Testing",
 ]
 dependencies = [
-    "pytest>=9.1.1",
-    "mcp>=2.2.0",
-    "pydantic>=2.13.5",
+    "pytest>=9.2.0",
+    "mcp>=2.3.0",
+    "pydantic>=2.14.1",
     "python-frontmatter>=1.3.0",
-    "github-copilot-sdk>=1.0.16",
+    "github-copilot-sdk>=1.0.28",
     "httpx>=0.28.1",
     "trio>=0.34.0",
-    "tomlkit>=0.15.1",
-    "pytest-asyncio>=1.4.0",
-    "websockets>=17.1",
+    "tomlkit>=0.15.2",
+    "pytest-asyncio>=1.4.1",
+    "websockets>=17.3",
 ]
 
 [project.optional-dependencies]
 dev = [
     "pytest-cov>=7.1.0",
-    "python-dotenv>=1.2.3",
-    "ruff>=0.16.9",
+    "python-dotenv>=1.2.4",
+    "ruff>=0.17.3",
     "pyright>=1.1.414",
-    "pre-commit>=4.6.2",
+    "pre-commit>=4.7.0",
     "typeguard>=4.6.0",
 ]
 test = [
@@ -53,8 +53,8 @@ test = [
 docs = [
     "mkdocs>=1.6.1",
     "mkdocs-material>=9.7.7,<9.8",
-    "mkdocstrings[python]>=1.0.6",
-    "pillow>=12.3.0",
+    "mkdocstrings[python]>=1.0.8",
+    "pillow>=13.0.0",
     "cairosvg>=2.9.1",
 ]
 
@@ -98,7 +98,7 @@ quote-style = "double"
 indent-style = "space"
 
 [tool.pyright]
-include = ["src", "tests", "examples/skill-dogfood"]
+include = ["src", "tests", "examples/skill-dogfood", "examples/excel-skill-value"]
 pythonVersion = "3.11"
 typeCheckingMode = "basic"
 

pytest_skill_engineering-1.0.4/src/pytest_skill_engineering/copilot/config.py

@@ -0,0 +1,133 @@
+"""Configuration file loaders for Copilot and Claude Code projects.
+
+Provides utilities for loading MCP server configurations from standard
+config files (``.mcp.json``, ``.vscode/mcp.json``).
+"""
+
+from __future__ import annotations
+
+from copy import deepcopy
+from pathlib import Path
+from typing import TYPE_CHECKING, Any
+
+from pytest_skill_engineering.core.plugin import _read_json
+from pytest_skill_engineering.core.validation import _validate_mcp_servers
+
+if TYPE_CHECKING:
+    from pytest_skill_engineering.copilot.contracts import CopilotEvalConfig
+
+
+def snapshot_session_configuration(
+    agent: CopilotEvalConfig, session_config: dict[str, Any]
+) -> dict[str, Any]:
+    """Capture prepared settings without handlers, credentials, or server environments."""
+    system = session_config.get("system_message") or {}
+    snapshot = {
+        "name": agent.name,
+        "model": session_config.get("model"),
+        "instructions": system.get("content"),
+        "system_message_mode": system.get("mode"),
+        "reasoning_effort": session_config.get("reasoning_effort"),
+        "client_mode": agent.client_mode,
+        "image_detail": agent.image_detail,
+        "audit_requests": agent.audit_requests,
+        "max_tool_calls": agent.max_tool_calls,
+        "max_turns": agent.max_turns,
+        "timeout_s": agent.timeout_s,
+        "auto_confirm": agent.auto_confirm,
+        "allowed_tools": session_config.get("available_tools"),
+        "excluded_tools": session_config.get("excluded_tools"),
+        "skill_directories": session_config.get("skill_directories", []),
+        "disabled_skills": session_config.get("disabled_skills", []),
+        "active_agent": session_config.get("agent"),
+        "working_directory": session_config.get("working_directory"),
+        "persona": type(agent.persona).__name__,
+        "tools": [
+            {
+                "name": tool.name,
+                "description": tool.description,
+                "parameters": tool.parameters,
+                "metadata": tool.metadata,
+            }
+            for tool in session_config.get("tools") or []
+        ],
+        "mcp_servers": {
+            name: {"type": config.get("type"), "tools": config.get("tools")}
+            for name, config in (session_config.get("mcp_servers") or {}).items()
+        },
+        "custom_agents": [
+            {
+                key: value
+                for key, value in custom.items()
+                if key
+                in {
+                    "name",
+                    "prompt",
+                    "description",
+                    "display_name",
+                    "model",
+                    "reasoning_effort",
+                    "tools",
+                    "infer",
+                    "skills",
+                }
+            }
+            for custom in session_config.get("custom_agents") or []
+        ],
+    }
+    for key in (
+        "enable_config_discovery",
+        "enable_file_hooks",
+        "enable_on_demand_instruction_discovery",
+        "skip_custom_instructions",
+        "enable_managed_settings",
+        "enable_skills",
+    ):
+        if key in session_config:
+            snapshot[key] = session_config[key]
+    return deepcopy(snapshot)
+
+
+def load_mcp_config(path: str | Path) -> dict[str, dict[str, Any]]:
+    """Load MCP server configs from a ``.mcp.json`` file.
+
+    Supports both Claude Code / root-level ``.mcp.json`` and VS Code
+    ``.vscode/mcp.json`` formats.  Handles two common top-level keys:
+
+    * ``mcpServers`` — Claude Code / standard MCP convention
+    * ``servers`` — VS Code ``mcp.json`` convention
+
+    Exactly one supported key must be present. An explicit empty mapping is valid;
+    missing keys, conflicting keys, and malformed server definitions raise errors.
+
+    Returns:
+        Dict of ``server_name → config`` compatible with
+        ``CopilotEval(mcp_servers=...)``.
+
+    Raises:
+        FileNotFoundError: If the file does not exist.
+        ValueError: If the file is not valid JSON or has unexpected structure.
+
+    Example::
+
+        from pytest_skill_engineering.copilot.config import load_mcp_config
+
+        servers = load_mcp_config(".mcp.json")
+        agent = CopilotEval(mcp_servers=servers)
+    """
+    path = Path(path)
+    if not path.is_file():
+        msg = f"MCP config file not found: {path}"
+        raise FileNotFoundError(msg)
+
+    raw = _read_json(path)
+
+    if not isinstance(raw, dict):
+        msg = f"{path}: MCP config must be a JSON object, got {type(raw).__name__}"
+        raise ValueError(msg)
+
+    fields = [key for key in ("mcpServers", "servers") if key in raw]
+    if len(fields) != 1:
+        raise ValueError(f"{path}: specify exactly one of 'mcpServers' and 'servers'")
+
+    return _validate_mcp_servers(raw[fields[0]], path, field=fields[0])

{pytest_skill_engineering-1.0.2 → pytest_skill_engineering-1.0.4}/src/pytest_skill_engineering/copilot/contracts.py

@@ -3,24 +3,16 @@
 from __future__ import annotations
 
 from collections.abc import Awaitable, Callable, Mapping
-from dataclasses import dataclass
 from typing import Any, Literal, Protocol, TypeAlias
 
 from copilot.session import CustomAgentConfig, MCPServerConfig, ReasoningEffort, SessionHooks
 
+from pytest_skill_engineering.copilot.result import CopilotResult, CopilotResultAgent
+
 CopilotCustomAgentConfig: TypeAlias = CustomAgentConfig
 CopilotMCPServerConfig: TypeAlias = MCPServerConfig
 CopilotReasoningEffort: TypeAlias = ReasoningEffort
 CopilotSessionHooks: TypeAlias = SessionHooks
-SubagentStatus: TypeAlias = Literal["selected", "started", "completed", "failed"]
-
-
-class CopilotResultAgent(Protocol):
-    """Minimal agent surface stored on :class:`CopilotResult`."""
-
-    @property
-    def working_directory(self) -> str | None:
-        raise NotImplementedError
 
 
 class CopilotRunResult(Protocol):
@@ -53,6 +45,11 @@ class CopilotEventMapper(Protocol):
     def record_subagent_failed(self, *, invocation_id: str, name: str) -> None:
         raise NotImplementedError
 
+    def record_subagent_result(
+        self, *, invocation_id: str, name: str, result: CopilotResult
+    ) -> None:
+        raise NotImplementedError
+
 
 class CopilotEvalConfig(CopilotResultAgent, Protocol):
     """Configuration surface consumed by the runner and personas."""
@@ -141,7 +138,7 @@ class CopilotEvalConfig(CopilotResultAgent, Protocol):
 
 CopilotNestedRunner: TypeAlias = Callable[
     [CopilotEvalConfig, str],
-    Awaitable[CopilotRunResult],
+    Awaitable["CopilotResult"],
 ]
 
 
@@ -158,16 +155,6 @@ class CopilotPersona(Protocol):
         raise NotImplementedError
 
 
-@dataclass(slots=True)
-class SubagentInvocation:
-    """A single custom-agent dispatch observed during a run."""
-
-    invocation_id: str
-    name: str
-    status: SubagentStatus
-    duration_ms: float | None = None
-
-
 def declared_agent_tools(
     agent_config: CopilotCustomAgentConfig,
 ) -> list[str] | None:

{pytest_skill_engineering-1.0.2 → pytest_skill_engineering-1.0.4}/src/pytest_skill_engineering/copilot/eval.py

@@ -18,7 +18,6 @@ from pytest_skill_engineering.copilot.personas import ClaudeCodePersona, VSCodeP
 from pytest_skill_engineering.core.evals import load_custom_agent, load_instruction_file
 from pytest_skill_engineering.core.plugin import (
     _discover_skills,
-    _validate_mcp_servers,
     _validate_skill_reference_names,
 )
 
@@ -34,30 +33,20 @@ def _parse_agent_file(path: Path) -> CopilotCustomAgentConfig:
     fields and uses the Markdown body as the custom agent's system prompt.
     Parsing and metadata errors retain the definition's file path.
     """
-    loaded = load_custom_agent(path)
-    frontmatter = loaded["metadata"]
-    agent: CopilotCustomAgentConfig = {
-        "name": frontmatter.get("name", loaded["name"]),
-        "prompt": loaded["prompt"],
-    }
-    for key in (
-        "description",
-        "display_name",
-        "infer",
-        "skills",
-        "model",
-        "reasoning_effort",
-        "tools",
-    ):
-        if key in frontmatter:
-            agent[key] = frontmatter[key]
-    if "mcp-servers" in frontmatter:
-        agent["mcp_servers"] = cast(
-            dict[str, CopilotMCPServerConfig],
-            _validate_mcp_servers(frontmatter["mcp-servers"], path, field="mcp-servers"),
-        )
+    return cast(CopilotCustomAgentConfig, load_custom_agent(path))
+
 
-    return agent
+def _parse_agent_files(paths: list[Path]) -> list[CopilotCustomAgentConfig]:
+    agents: list[CopilotCustomAgentConfig] = []
+    names: set[str] = set()
+    for path in paths:
+        agent = _parse_agent_file(path)
+        name = require_custom_agent_name(agent)
+        if name in names:
+            raise ValueError(f"{path}: Duplicate custom agent name '{name}'")
+        names.add(name)
+        agents.append(agent)
+    return agents
 
 
 def _default_persona() -> CopilotPersona:
@@ -361,8 +350,7 @@ class CopilotEval:
         if agents_dir.exists():
             if not agents_dir.is_dir():
                 raise ValueError(f"{agents_dir}: custom agents path must be a directory")
-            for agent_file in sorted(agents_dir.rglob("*.agent.md")):
-                agents.append(_parse_agent_file(agent_file))
+            agents = _parse_agent_files(sorted(agents_dir.rglob("*.agent.md")))
 
         config: dict[str, Any] = {
             "instructions": instructions,
@@ -527,8 +515,7 @@ class CopilotEval:
             if not agents_dir.is_dir():
                 raise ValueError(f"{agents_dir}: custom agents path must be a directory")
             # Claude Code uses plain .md files for agents
-            for agent_file in sorted(agents_dir.glob("*.md")):
-                agents.append(_parse_agent_file(agent_file))
+            agents = _parse_agent_files(sorted(agents_dir.glob("*.md")))
 
         # 3. Discover skill directories from .claude/skills/
         skills = _discover_skills(claude_dir)