k-eval 0.3.0__tar.gz

k_eval-0.3.0/MANIFEST.in

@@ -0,0 +1,5 @@
+include README.md
+include pyproject.toml
+recursive-exclude results *
+exclude out.json
+exclude uv.lock

k_eval-0.3.0/PKG-INFO

@@ -0,0 +1,188 @@
+Metadata-Version: 2.4
+Name: k-eval
+Version: 0.3.0
+Summary: Context-aware evaluation framework for AI agents using MCP.
+Author: jsell-rh
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/jsell-rh/k-eval
+Project-URL: Repository, https://github.com/jsell-rh/k-eval
+Project-URL: Issues, https://github.com/jsell-rh/k-eval/issues
+Keywords: evaluation,ai,agents,mcp,llm
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.13
+Description-Content-Type: text/markdown
+Requires-Dist: claude-agent-sdk>=0.1.39
+Requires-Dist: litellm
+Requires-Dist: pydantic>=2.12.5
+Requires-Dist: pyyaml>=6.0.3
+Requires-Dist: rich>=14.3.3
+Requires-Dist: structlog>=25.5.0
+Requires-Dist: typer>=0.24.1
+Provides-Extra: vertex-ai
+Requires-Dist: google-cloud-aiplatform>=1.138.0; extra == "vertex-ai"
+Provides-Extra: all
+Requires-Dist: k-eval[vertex_ai]; extra == "all"
+
+# k-eval
+
+Context-aware evaluation framework for AI agents using MCP.
+
+## Quick Start
+
+k-eval uses [uv](https://docs.astral.sh/uv/) for dependency management. Install it first if you don't have it:
+
+```bash
+curl -LsSf https://astral.sh/uv/install.sh | sh
+```
+
+### Install `k-eval`
+
+```bash
+git clone https://github.com/jsell-rh/k-eval.git
+cd k-eval/src/k-eval
+
+# Core dependencies
+uv sync
+
+# With Vertex AI provider support
+uv sync --extra vertex_ai
+
+# All provider dependencies
+uv sync --extra all
+```
+
+### Run `k-eval`
+
+`k-eval` runs are configured using `yaml` configuration files (see [Configuration](#Configuration)).
+
+Once an evaluation is defined in a `yaml` file, you can invoke
+`k-eval` like:
+
+```bash
+cd src/k-eval
+uv run python -m k_eval.cli.main /path/to/config.yaml
+```
+
+See [docs/run-configuration.md](docs/run-configuration.md) for authentication setup and all CLI options.
+
+#### CLI Options
+
+```bash
+src/k-eval$ uv run python -m k_eval.cli.main --help
+                                                                                                                                    
+ Usage: python -m cli.main [OPTIONS] CONFIG_PATH                                                                                    
+                                                                                                                                    
+ Run a k-eval evaluation from a YAML config file.                                                                                   
+                                                                                                                                    
+╭─ Arguments ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
+│ *    config_path      PATH  Path to evaluation config YAML [required]                                                            │
+╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
+╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
+│ --output-dir  -o      PATH  Directory for output files [default: results]                                                        │
+│ --log-format          TEXT  Log format: 'console' or 'json' [default: console]                                                   │
+│ --quiet       -q            Suppress debug and info logs; show only the progress bar plus warnings/errors.                       │
+│ --help                      Show this message and exit.                                                                          │
+╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
+```
+
+### Understanding the Output
+
+Each run produces two files in `./results/` (or wherever you point `--output-dir`):
+
+```
+results/
+  my-eval_20260225_a1b2c3d4.json           # aggregate scores per condition
+  my-eval_20260225_a1b2c3d4.detailed.jsonl # one line per (question, condition) pair
+```
+
+**`{name}_{date}_{run_id}.json`** — the summary. One entry per condition with
+mean and standard deviation for each of the three metrics across all questions
+and repetitions. Use this to compare conditions at a glance.
+
+This file is intended to be mostly compliant with the [Every Eval Ever](https://evalevalai.com/projects/every-eval-ever/) schema.
+Notably, `k-eval` does not aggregate the three metrics into a single score.
+Thus, the individual metrics are written to `score_details.details`, and
+`score_details.score` is left `null`. 
+
+**`{name}_{date}_{run_id}.detailed.jsonl`** — the full record. One JSON object per
+`(question, condition)` pair containing the agent's raw responses for every
+repetition, per-repetition judge scores and reasoning, unverified claims, and
+token usage. Use this if you want to dig into why a condition scored the way it did.
+
+The three metrics are scored 1-5 by the judge model:
+
+| Metric | What it measures |
+|---|---|
+| `factual_adherence` | Does the response stick to facts in the golden answer? |
+| `completeness` | Does it cover all the essential points? |
+| `helpfulness_and_clarity` | Is it well-structured and easy to act on? |
+
+See [evaluation-methodology](docs/evaluation-methodology.md) for more details.
+
+### Configuration
+
+A config file defines your dataset, agent, judge, MCP servers, and the conditions you want to compare:
+
+> [!Important]
+>
+> For MCP servers that require authentication,
+> please reference [docs/run-configuration.md](docs/run-configuration.md).
+
+```yaml
+name: "my-eval"
+version: "1"
+
+dataset:
+  # JSONL file with your questions and golden answers
+  path: "./questions.jsonl"
+  # The name of the key used to reference the question within the JSONL file.
+  question_key: "question"
+  # They key used to reference the golden "reference" or answer within the JSON file.
+  answer_key: "answer"
+
+agent:
+  type: "claude_code_sdk" # currently the only supported type
+  model: "claude-sonnet-4-5"
+
+judge:
+  model: "vertex_ai/claude-opus-4-5" # any LiteLLM-compatible model string (See: https://models.litellm.ai/)
+  temperature: 0.0
+
+mcp_servers:
+  graph:
+    type: "stdio"
+    command: "python"
+    args: ["-m", "my_mcp_server"]
+
+conditions:
+  baseline:
+    mcp_servers: []
+    system_prompt: |
+        Answer using your own knowledge.
+  with_graph:
+    mcp_servers: [graph]
+    system_prompt: |
+        Use the graph tool to answer the question.
+
+execution:
+  # How many times each (question, condition) pair is evaluated.
+  # This is useful for managing variance in agent responses. Standard
+  # deviation between scores will be reported if num_repetitions >= 3
+  num_repetitions: 3
+  # (question, condition, repetition) tuples can be evaluated concurrently
+  # to reduce total evaluation time. The upper bound of this number is determined
+  # only by the resources on your computer and by the rate limit configuration
+  # of the agent and model providers.
+  #
+  # In practice, numbers even as high as 50 seem to be well tolerated 
+  # when using Vertex AI.
+  max_concurrent: 5
+```
+
+See [docs/run-configuration.md](docs/run-configuration.md) for the full reference including authentication setup.
+

k_eval-0.3.0/README.md

@@ -0,0 +1,158 @@
+# k-eval
+
+Context-aware evaluation framework for AI agents using MCP.
+
+## Quick Start
+
+k-eval uses [uv](https://docs.astral.sh/uv/) for dependency management. Install it first if you don't have it:
+
+```bash
+curl -LsSf https://astral.sh/uv/install.sh | sh
+```
+
+### Install `k-eval`
+
+```bash
+git clone https://github.com/jsell-rh/k-eval.git
+cd k-eval/src/k-eval
+
+# Core dependencies
+uv sync
+
+# With Vertex AI provider support
+uv sync --extra vertex_ai
+
+# All provider dependencies
+uv sync --extra all
+```
+
+### Run `k-eval`
+
+`k-eval` runs are configured using `yaml` configuration files (see [Configuration](#Configuration)).
+
+Once an evaluation is defined in a `yaml` file, you can invoke
+`k-eval` like:
+
+```bash
+cd src/k-eval
+uv run python -m k_eval.cli.main /path/to/config.yaml
+```
+
+See [docs/run-configuration.md](docs/run-configuration.md) for authentication setup and all CLI options.
+
+#### CLI Options
+
+```bash
+src/k-eval$ uv run python -m k_eval.cli.main --help
+                                                                                                                                    
+ Usage: python -m cli.main [OPTIONS] CONFIG_PATH                                                                                    
+                                                                                                                                    
+ Run a k-eval evaluation from a YAML config file.                                                                                   
+                                                                                                                                    
+╭─ Arguments ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
+│ *    config_path      PATH  Path to evaluation config YAML [required]                                                            │
+╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
+╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
+│ --output-dir  -o      PATH  Directory for output files [default: results]                                                        │
+│ --log-format          TEXT  Log format: 'console' or 'json' [default: console]                                                   │
+│ --quiet       -q            Suppress debug and info logs; show only the progress bar plus warnings/errors.                       │
+│ --help                      Show this message and exit.                                                                          │
+╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
+```
+
+### Understanding the Output
+
+Each run produces two files in `./results/` (or wherever you point `--output-dir`):
+
+```
+results/
+  my-eval_20260225_a1b2c3d4.json           # aggregate scores per condition
+  my-eval_20260225_a1b2c3d4.detailed.jsonl # one line per (question, condition) pair
+```
+
+**`{name}_{date}_{run_id}.json`** — the summary. One entry per condition with
+mean and standard deviation for each of the three metrics across all questions
+and repetitions. Use this to compare conditions at a glance.
+
+This file is intended to be mostly compliant with the [Every Eval Ever](https://evalevalai.com/projects/every-eval-ever/) schema.
+Notably, `k-eval` does not aggregate the three metrics into a single score.
+Thus, the individual metrics are written to `score_details.details`, and
+`score_details.score` is left `null`. 
+
+**`{name}_{date}_{run_id}.detailed.jsonl`** — the full record. One JSON object per
+`(question, condition)` pair containing the agent's raw responses for every
+repetition, per-repetition judge scores and reasoning, unverified claims, and
+token usage. Use this if you want to dig into why a condition scored the way it did.
+
+The three metrics are scored 1-5 by the judge model:
+
+| Metric | What it measures |
+|---|---|
+| `factual_adherence` | Does the response stick to facts in the golden answer? |
+| `completeness` | Does it cover all the essential points? |
+| `helpfulness_and_clarity` | Is it well-structured and easy to act on? |
+
+See [evaluation-methodology](docs/evaluation-methodology.md) for more details.
+
+### Configuration
+
+A config file defines your dataset, agent, judge, MCP servers, and the conditions you want to compare:
+
+> [!Important]
+>
+> For MCP servers that require authentication,
+> please reference [docs/run-configuration.md](docs/run-configuration.md).
+
+```yaml
+name: "my-eval"
+version: "1"
+
+dataset:
+  # JSONL file with your questions and golden answers
+  path: "./questions.jsonl"
+  # The name of the key used to reference the question within the JSONL file.
+  question_key: "question"
+  # They key used to reference the golden "reference" or answer within the JSON file.
+  answer_key: "answer"
+
+agent:
+  type: "claude_code_sdk" # currently the only supported type
+  model: "claude-sonnet-4-5"
+
+judge:
+  model: "vertex_ai/claude-opus-4-5" # any LiteLLM-compatible model string (See: https://models.litellm.ai/)
+  temperature: 0.0
+
+mcp_servers:
+  graph:
+    type: "stdio"
+    command: "python"
+    args: ["-m", "my_mcp_server"]
+
+conditions:
+  baseline:
+    mcp_servers: []
+    system_prompt: |
+        Answer using your own knowledge.
+  with_graph:
+    mcp_servers: [graph]
+    system_prompt: |
+        Use the graph tool to answer the question.
+
+execution:
+  # How many times each (question, condition) pair is evaluated.
+  # This is useful for managing variance in agent responses. Standard
+  # deviation between scores will be reported if num_repetitions >= 3
+  num_repetitions: 3
+  # (question, condition, repetition) tuples can be evaluated concurrently
+  # to reduce total evaluation time. The upper bound of this number is determined
+  # only by the resources on your computer and by the rate limit configuration
+  # of the agent and model providers.
+  #
+  # In practice, numbers even as high as 50 seem to be well tolerated 
+  # when using Vertex AI.
+  max_concurrent: 5
+```
+
+See [docs/run-configuration.md](docs/run-configuration.md) for the full reference including authentication setup.
+

k_eval-0.3.0/k_eval/agent/domain/agent.py

@@ -0,0 +1,14 @@
+"""Agent Protocol — structural interface for all agent implementations."""
+
+from typing import Protocol
+
+from k_eval.agent.domain.result import AgentResult
+
+
+class Agent(Protocol):
+    """Structural interface satisfied by any agent implementation.
+
+    Each instance is constructed once per (condition, sample) evaluation.
+    """
+
+    async def ask(self, question: str) -> AgentResult: ...

k_eval-0.3.0/k_eval/agent/domain/factory.py

@@ -0,0 +1,18 @@
+"""AgentFactory Protocol — structural interface for constructing Agent instances."""
+
+from typing import Protocol
+
+from k_eval.agent.domain.agent import Agent
+from k_eval.config.domain.condition_mcp_server import ConditionMcpServer
+
+
+class AgentFactory(Protocol):
+    """Constructs a new Agent instance for a given (condition, sample) pair."""
+
+    def create(
+        self,
+        condition: str,
+        sample_idx: str,
+        system_prompt: str,
+        mcp_servers: list[ConditionMcpServer],
+    ) -> Agent: ...

k_eval-0.3.0/k_eval/agent/domain/observer.py

@@ -0,0 +1,27 @@
+"""AgentObserver port — domain events emitted during agent invocations."""
+
+from typing import Protocol
+
+
+class AgentObserver(Protocol):
+    """Observer port for agent domain events.
+
+    Implementations may log to structlog, record for tests, or emit metrics.
+    """
+
+    def agent_invocation_started(
+        self, condition: str, sample_idx: str, model: str
+    ) -> None: ...
+
+    def agent_invocation_completed(
+        self,
+        condition: str,
+        sample_idx: str,
+        duration_ms: int,
+        num_turns: int,
+        cost_usd: float | None,
+    ) -> None: ...
+
+    def agent_invocation_failed(
+        self, condition: str, sample_idx: str, reason: str
+    ) -> None: ...

k_eval-0.3.0/k_eval/agent/domain/result.py

@@ -0,0 +1,18 @@
+"""AgentResult value object — the outcome of a single agent invocation."""
+
+from pydantic import BaseModel, ConfigDict
+
+from k_eval.agent.domain.usage import UsageMetrics
+
+
+class AgentResult(BaseModel, frozen=True):
+    """Immutable value object capturing all outcome data from one agent invocation."""
+
+    model_config = ConfigDict(frozen=True)
+
+    response: str
+    cost_usd: float | None
+    duration_ms: int
+    duration_api_ms: int
+    num_turns: int
+    usage: UsageMetrics | None

k_eval-0.3.0/k_eval/agent/domain/usage.py

@@ -0,0 +1,12 @@
+"""UsageMetrics value object — token usage from an agent invocation."""
+
+from pydantic import BaseModel, ConfigDict
+
+
+class UsageMetrics(BaseModel, frozen=True):
+    """Immutable value object capturing token usage from a single agent invocation."""
+
+    model_config = ConfigDict(frozen=True)
+
+    input_tokens: int | None
+    output_tokens: int | None

k_eval-0.3.0/k_eval/agent/infrastructure/claude_sdk.py

@@ -0,0 +1,222 @@
+"""ClaudeAgentSDKAgent — agent implementation using the Claude Agent SDK."""
+
+from typing import Any
+
+from claude_agent_sdk import query
+from claude_agent_sdk._errors import ClaudeSDKError
+from claude_agent_sdk.types import (
+    ClaudeAgentOptions,
+    McpHttpServerConfig,
+    McpSdkServerConfig,
+    McpSSEServerConfig,
+    McpStdioServerConfig,
+    ResultMessage,
+)
+
+from k_eval.agent.domain.observer import AgentObserver
+from k_eval.agent.domain.result import AgentResult
+from k_eval.agent.domain.usage import UsageMetrics
+from k_eval.agent.infrastructure.errors import AgentInvocationError
+from k_eval.config.domain.agent import AgentConfig
+from k_eval.config.domain.condition_mcp_server import ConditionMcpServer
+from k_eval.config.domain.mcp_server import HttpMcpServer, SseMcpServer, StdioMcpServer
+
+type McpServerConfigMap = dict[
+    str,
+    McpStdioServerConfig
+    | McpSSEServerConfig
+    | McpHttpServerConfig
+    | McpSdkServerConfig,
+]
+
+
+class ClaudeAgentSDKAgent:
+    """Agent implementation that delegates to the Claude Agent SDK.
+
+    One instance is constructed per (condition, sample) evaluation run.
+    The condition and sample_idx are injected at construction time so that
+    observer events carry full context without polluting the ask() signature.
+    """
+
+    def __init__(
+        self,
+        config: AgentConfig,
+        condition: str,
+        sample_idx: str,
+        system_prompt: str,
+        mcp_servers: list[ConditionMcpServer],
+        observer: AgentObserver,
+    ) -> None:
+        self._config = config
+        self._condition = condition
+        self._sample_idx = sample_idx
+        self._system_prompt = system_prompt
+        self._mcp_servers = mcp_servers
+        self._observer = observer
+
+    async def ask(self, question: str) -> AgentResult:
+        """Invoke the agent with a question and return the structured result.
+
+        Opens a new SDK session per call — correct for independent eval samples.
+
+        Raises:
+            AgentInvocationError: if the SDK raises, the agent returns an error,
+                or no ResultMessage is present in the response stream.
+        """
+        self._observer.agent_invocation_started(
+            condition=self._condition,
+            sample_idx=self._sample_idx,
+            model=self._config.model,
+        )
+
+        try:
+            options = ClaudeAgentOptions(
+                model=self._config.model,
+                system_prompt=self._system_prompt,
+                mcp_servers=self._build_mcp_servers(),
+                disallowed_tools=self._build_disallowed_tools(),
+                permission_mode="bypassPermissions",
+                setting_sources=[],
+            )
+
+            result_message = await self._collect_result(
+                prompt=question, options=options
+            )
+        except AgentInvocationError as exc:
+            reason = str(exc).removeprefix("Failed to invoke agent: ")
+            self._observer.agent_invocation_failed(
+                condition=self._condition,
+                sample_idx=self._sample_idx,
+                reason=reason,
+            )
+            raise
+
+        self._observer.agent_invocation_completed(
+            condition=self._condition,
+            sample_idx=self._sample_idx,
+            duration_ms=result_message.duration_ms,
+            num_turns=result_message.num_turns,
+            cost_usd=result_message.total_cost_usd,
+        )
+
+        assert result_message.result is not None  # guaranteed by _collect_result
+        return AgentResult(
+            response=result_message.result,
+            cost_usd=result_message.total_cost_usd,
+            duration_ms=result_message.duration_ms,
+            duration_api_ms=result_message.duration_api_ms,
+            num_turns=result_message.num_turns,
+            usage=self._map_usage(raw=result_message.usage),
+        )
+
+    async def _collect_result(
+        self, prompt: str, options: ClaudeAgentOptions
+    ) -> ResultMessage:
+        """Run the SDK query and extract the single ResultMessage.
+
+        Raises:
+            AgentInvocationError: on SDK errors or missing/error ResultMessage.
+        """
+        result_message: ResultMessage | None = None
+
+        try:
+            async for message in query(prompt=prompt, options=options):
+                if isinstance(message, ResultMessage):
+                    result_message = message
+        except ClaudeSDKError as exc:
+            raise AgentInvocationError(reason=str(exc), retriable=True) from exc
+        except Exception as exc:
+            # The SDK internally raises a bare Exception (not ClaudeSDKError) when
+            # its message reader encounters a fatal error (e.g. subprocess exit).
+            raise AgentInvocationError(reason=str(exc), retriable=True) from exc
+
+        if result_message is None:
+            raise AgentInvocationError(reason="no ResultMessage in response stream")
+
+        if result_message.is_error:
+            raise AgentInvocationError(
+                reason=f"agent returned error response: {result_message.result}"
+            )
+
+        if result_message.result is None:
+            raise AgentInvocationError(reason="ResultMessage has no result text")
+
+        return result_message
+
+    def _build_mcp_servers(self) -> McpServerConfigMap:
+        """Convert ConditionMcpServer list to the SDK's TypedDict format."""
+        servers: McpServerConfigMap = {}
+
+        for server in self._mcp_servers:
+            config = server.config
+
+            if isinstance(config, StdioMcpServer):
+                servers[server.name] = self._build_stdio_server(config=config)
+            elif isinstance(config, SseMcpServer):
+                servers[server.name] = self._build_sse_server(config=config)
+            elif isinstance(config, HttpMcpServer):
+                servers[server.name] = self._build_http_server(config=config)
+            else:
+                raise AgentInvocationError(
+                    reason=f"unsupported MCP server type for server '{server.name}'"
+                )
+
+        return servers
+
+    def _build_stdio_server(self, config: StdioMcpServer) -> McpStdioServerConfig:
+        """Build a McpStdioServerConfig TypedDict from a StdioMcpServer model."""
+        server: McpStdioServerConfig = McpStdioServerConfig(command=config.command)
+        if config.args:
+            server["args"] = list(config.args)
+        if config.env:
+            server["env"] = dict(config.env)
+        return server
+
+    def _build_sse_server(self, config: SseMcpServer) -> McpSSEServerConfig:
+        """Build a McpSSEServerConfig TypedDict from a SseMcpServer model."""
+        server: McpSSEServerConfig = McpSSEServerConfig(type="sse", url=config.url)
+        if config.headers:
+            server["headers"] = dict(config.headers)
+        return server
+
+    def _build_http_server(self, config: HttpMcpServer) -> McpHttpServerConfig:
+        """Build a McpHttpServerConfig TypedDict from an HttpMcpServer model."""
+        server: McpHttpServerConfig = McpHttpServerConfig(type="http", url=config.url)
+        if config.headers:
+            server["headers"] = dict(config.headers)
+        return server
+
+    def _build_disallowed_tools(self) -> list[str]:
+        """Build the disallowed tools list — all Claude built-in tools.
+
+        allowed_tools alone does not remove built-in tools from the agent's
+        context; it only controls approval requirements. Explicitly disallowing
+        all built-in tools ensures the agent cannot use web search, file I/O,
+        or any other built-in capability regardless of permission_mode.
+        """
+        return [
+            "Bash",
+            "Edit",
+            "Glob",
+            "Grep",
+            "LS",
+            "MultiEdit",
+            "NotebookEdit",
+            "NotebookRead",
+            "Read",
+            "Task",
+            "TodoRead",
+            "TodoWrite",
+            "WebFetch",
+            "WebSearch",
+            "Write",
+        ]
+
+    def _map_usage(self, raw: dict[str, Any] | None) -> UsageMetrics | None:
+        """Map the SDK's raw usage dict to a typed UsageMetrics value object."""
+        if raw is None:
+            return None
+        return UsageMetrics(
+            input_tokens=raw.get("input_tokens"),
+            output_tokens=raw.get("output_tokens"),
+        )