agent-skill-eval 0.2.1__tar.gz → 0.3.0__tar.gz

agent_skill_eval-0.3.0/.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,9 @@
+## What
+
+## Why
+
+## Testing
+
+- [ ] `uv run --extra dev ruff check src/ tests/`
+- [ ] `uv run --extra dev ruff format --check src/ tests/`
+- [ ] `uv run --extra dev pytest -q`

{agent_skill_eval-0.2.1 → agent_skill_eval-0.3.0}/CHANGELOG.md

@@ -2,6 +2,17 @@
 
 All notable changes to this project will be documented in this file.
 
+## [0.3.0] - 2026-06-11
+
+### Added
+- `validate-config` example skill + eval suite: exercises bundled resources (agent runs skill's `scripts/` and reads its `references/`); graded via command-ran + file + content checks; fully offline
+- `review-diff` example skill + eval suite: read-only analysis (planted bugs found, documented decoy not flagged, nothing modified); graded via chat-output-only content + LLM rubric; fully offline
+
+## [0.2.1] - 2026-06-11
+
+### Changed
+- Renamed package from `skill-eval` to `agent-skill-eval` (PyPI name collision)
+
 ## [0.2.0] - 2026-06-10
 
 ### Added

{agent_skill_eval-0.2.1 → agent_skill_eval-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agent-skill-eval
-Version: 0.2.1
+Version: 0.3.0
 Summary: Framework for evaluating agent skills across OpenCode, Claude Code, and Codex
 Project-URL: Homepage, https://github.com/tardigrde/skill-evals
 Project-URL: Repository, https://github.com/tardigrde/skill-evals
@@ -250,13 +250,15 @@ skill-eval init my-skill [--output ./examples]
 
 ## Example skills
 
-Three example skills ship with the repo, chosen to exercise different grading surfaces:
+Five example skills ship with the repo, chosen to exercise different grading surfaces:
 
 | Skill | Tests | Grading surface |
 | --- | --- | --- |
 | [`commit-push-pr`](skills/commit-push-pr/SKILL.md) | git workflow automation | deterministic state-diff checks (branch/commit/push/PR); needs a `--source-repo` |
 | [`fix-failing-tests`](skills/fix-failing-tests/SKILL.md) | error recovery / iterative refinement | deterministic + file checks; fully offline |
 | [`write-release-notes`](skills/write-release-notes/SKILL.md) | subjective writing quality, anti-fabrication | LLM rubric grading; fully offline |
+| [`validate-config`](skills/validate-config/SKILL.md) | bundled resources: does the agent run the skill's `scripts/` and read its `references/`? | command-ran + file + content checks; fully offline |
+| [`review-diff`](skills/review-diff/SKILL.md) | read-only analysis: planted bugs found, documented decoy not flagged, nothing modified | chat-output-only grading (content + LLM rubric); fully offline |
 
 Each has a matching eval suite under [`examples/`](examples/). `skills/` holds the artifacts being evaluated; `examples/` holds the test cases — so you can test one skill against many suites or one suite against many skill versions.
 

{agent_skill_eval-0.2.1 → agent_skill_eval-0.3.0}/README.md

@@ -219,13 +219,15 @@ skill-eval init my-skill [--output ./examples]
 
 ## Example skills
 
-Three example skills ship with the repo, chosen to exercise different grading surfaces:
+Five example skills ship with the repo, chosen to exercise different grading surfaces:
 
 | Skill | Tests | Grading surface |
 | --- | --- | --- |
 | [`commit-push-pr`](skills/commit-push-pr/SKILL.md) | git workflow automation | deterministic state-diff checks (branch/commit/push/PR); needs a `--source-repo` |
 | [`fix-failing-tests`](skills/fix-failing-tests/SKILL.md) | error recovery / iterative refinement | deterministic + file checks; fully offline |
 | [`write-release-notes`](skills/write-release-notes/SKILL.md) | subjective writing quality, anti-fabrication | LLM rubric grading; fully offline |
+| [`validate-config`](skills/validate-config/SKILL.md) | bundled resources: does the agent run the skill's `scripts/` and read its `references/`? | command-ran + file + content checks; fully offline |
+| [`review-diff`](skills/review-diff/SKILL.md) | read-only analysis: planted bugs found, documented decoy not flagged, nothing modified | chat-output-only grading (content + LLM rubric); fully offline |
 
 Each has a matching eval suite under [`examples/`](examples/). `skills/` holds the artifacts being evaluated; `examples/` holds the test cases — so you can test one skill against many suites or one suite against many skill versions.
 

agent_skill_eval-0.3.0/examples/review-diff/evals/evals.json

@@ -0,0 +1,45 @@
+{
+  "skill_name": "review-diff",
+  "evals": [
+    {
+      "id": "explicit-invoke",
+      "prompt": "Use the $review-diff skill to review changes.diff against the current code.",
+      "expected_output": "The agent reads changes.diff and orders.py, identifies the two real bugs (the loop skips the last item due to range(len(prices) - 1), and the comparison uses > instead of >= so exactly `threshold` items get no discount despite the docstring saying 'threshold or more'), does not flag the documented cents-based summation, modifies nothing, and ends with VERDICT: REQUEST_CHANGES.",
+      "files": ["files/orders.py", "files/changes.diff"],
+      "force_skill_invocation": true,
+      "assertions": [
+        "The output contains \"VERDICT\"",
+        "The output contains \"REQUEST_CHANGES\"",
+        "The review identifies that the loop skips the last item (off-by-one in the range bound)",
+        "The review identifies that the threshold comparison uses > instead of >=, contradicting the docstring's 'threshold or more'",
+        "The review does not flag the cents-based summation, which is correct and documented as intentional",
+        "The agent did not modify orders.py and did not write any new files"
+      ]
+    },
+    {
+      "id": "implicit-invoke",
+      "prompt": "A teammate sent me changes.diff with a proposed addition to orders.py. Can you review it before I merge?",
+      "expected_output": "The agent automatically triggers the skill, reads both files, reports the two planted bugs with severities, leaves the workspace untouched, and ends with VERDICT: REQUEST_CHANGES.",
+      "files": ["files/orders.py", "files/changes.diff"],
+      "assertions": [
+        "The output contains \"VERDICT\"",
+        "The output contains \"REQUEST_CHANGES\"",
+        "The review identifies that the loop skips the last item (off-by-one in the range bound)",
+        "The review identifies that the threshold comparison uses > instead of >=, contradicting the docstring's 'threshold or more'",
+        "The agent did not modify orders.py and did not write any new files"
+      ]
+    },
+    {
+      "id": "negative-control",
+      "prompt": "How many functions does changes.diff add to orders.py?",
+      "expected_output": "The skill should NOT trigger. The agent should answer that the diff adds one function (bulk_discount_total) without producing a full code review or verdict.",
+      "files": ["files/orders.py", "files/changes.diff"],
+      "should_trigger": false,
+      "assertions": [
+        "The output contains \"bulk_discount_total\"",
+        "The agent states that exactly one function is added",
+        "The agent only answered the question without delivering a severity-tagged review or verdict"
+      ]
+    }
+  ]
+}

agent_skill_eval-0.3.0/examples/review-diff/evals/files/changes.diff

@@ -0,0 +1,21 @@
+--- a/orders.py
++++ b/orders.py
+@@ -20,3 +20,18 @@ def format_receipt(items: list[tuple[str, float]]) -> str:
+     """Render one 'name: $price' line per item."""
+     lines = [f"{name}: ${price:.2f}" for name, price in items]
+     return "\n".join(lines)
++
++
++def bulk_discount_total(prices: list[float], threshold: int = 10) -> float:
++    """Total with a 5% bulk discount when ordering `threshold` or more items.
++
++    Prices are summed in integer cents (rounded per item) to avoid float
++    accumulation error; this is intentional, see the team float-handling policy.
++    """
++    cents = 0
++    for i in range(len(prices) - 1):
++        cents += round(prices[i] * 100)
++    total = cents / 100
++    if len(prices) > threshold:
++        return round(total * 0.95, 2)
++    return round(total, 2)

agent_skill_eval-0.3.0/examples/review-diff/evals/files/orders.py

@@ -0,0 +1,22 @@
+"""Order processing utilities."""
+
+
+def order_total(prices: list[float]) -> float:
+    """Sum a list of item prices, rounded to cents."""
+    total = 0.0
+    for price in prices:
+        total += price
+    return round(total, 2)
+
+
+def apply_discount(total: float, percent: float) -> float:
+    """Apply a percentage discount to a total."""
+    if percent < 0 or percent > 100:
+        raise ValueError(f"invalid discount percent: {percent}")
+    return round(total * (1 - percent / 100), 2)
+
+
+def format_receipt(items: list[tuple[str, float]]) -> str:
+    """Render one 'name: $price' line per item."""
+    lines = [f"{name}: ${price:.2f}" for name, price in items]
+    return "\n".join(lines)

agent_skill_eval-0.3.0/examples/validate-config/evals/evals.json

@@ -0,0 +1,57 @@
+{
+  "skill_name": "validate-config",
+  "evals": [
+    {
+      "id": "explicit-invoke",
+      "prompt": "Use the $validate-config skill to validate app_config.json.",
+      "expected_output": "The agent runs the bundled structural validator script, reads the convention rules from the skill's references, writes CONFIG_REPORT.md flagging 5 violations (missing 'environment' key, service_name not svc- kebab-case, port 8080 outside 9000-9999, log_level 'verbose' not allowed, max_retries 10 above 5), and reports TOTAL VIOLATIONS: 5.",
+      "files": ["files/app_config.json"],
+      "force_skill_invocation": true,
+      "assertions": [
+        "The agent ran the bundled python validation script",
+        "The file `CONFIG_REPORT.md` exists",
+        "The output contains \"TOTAL VIOLATIONS\"",
+        "The report flags the port as outside the allowed 9000-9999 range",
+        "The report flags service_name for not being kebab-case with the svc- prefix",
+        "The report flags the missing 'environment' key as a structural issue",
+        "The report does not flag cache_ttl_seconds, which complies with the rules"
+      ]
+    },
+    {
+      "id": "implicit-invoke",
+      "prompt": "Can you check whether app_config.json follows our team's configuration conventions?",
+      "expected_output": "The agent automatically triggers the skill, runs the bundled validator, checks the reference rules, and writes CONFIG_REPORT.md listing all 5 violations.",
+      "files": ["files/app_config.json"],
+      "assertions": [
+        "The agent ran the bundled python validation script",
+        "The file `CONFIG_REPORT.md` exists",
+        "The output contains \"TOTAL VIOLATIONS\"",
+        "The report flags the port as outside the allowed 9000-9999 range",
+        "The report flags max_retries as exceeding the maximum of 5"
+      ]
+    },
+    {
+      "id": "reference-knowledge",
+      "prompt": "Is the port configured in app_config.json allowed by our team's port allocation policy?",
+      "expected_output": "The agent consults the skill's references/config-rules.md, finds the allocated range 9000-9999, and answers that port 8080 is NOT allowed. Without the skill the agent cannot know the policy and should say so or guess.",
+      "files": ["files/app_config.json"],
+      "assertions": [
+        "The output contains \"9000\"",
+        "The agent states that port 8080 is not allowed by the policy",
+        "The agent cites the team's allocated port range rather than guessing a generic answer"
+      ]
+    },
+    {
+      "id": "negative-control",
+      "prompt": "What is the value of max_retries in app_config.json?",
+      "expected_output": "The skill should NOT trigger. The agent should simply read the file and answer '10' without running the validator or writing a report.",
+      "files": ["files/app_config.json"],
+      "should_trigger": false,
+      "assertions": [
+        "The output contains \"10\"",
+        "The agent only answered the question without writing a report file",
+        "The validation script was not executed"
+      ]
+    }
+  ]
+}

agent_skill_eval-0.3.0/examples/validate-config/evals/files/app_config.json

@@ -0,0 +1,10 @@
+{
+  "service_name": "DataPipeline",
+  "port": 8080,
+  "log_level": "verbose",
+  "max_retries": 10,
+  "features": {
+    "cache_enabled": true,
+    "cache_ttl_seconds": 600
+  }
+}

{agent_skill_eval-0.2.1 → agent_skill_eval-0.3.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "agent-skill-eval"
-version = "0.2.1"
+version = "0.3.0"
 description = "Framework for evaluating agent skills across OpenCode, Claude Code, and Codex"
 requires-python = ">=3.11"
 license = "MIT"

agent_skill_eval-0.3.0/skills/review-diff/SKILL.md

@@ -0,0 +1,85 @@
+---
+name: review-diff
+description: Review a proposed code change (a .diff/.patch file) against the current source code and deliver a structured, severity-tagged review verdict in chat, without modifying or creating any files.
+license: MIT
+compatibility: opencode, claude-code, codex
+metadata:
+  workflow: read-analyze-report
+---
+
+## What I do
+
+Review a proposed code change — provided as a unified diff file — against the current source code in the workspace. I read both, look for real defects, and deliver a structured review directly in my reply. I am strictly read-only: I never apply the diff, never edit source files, and never write report files.
+
+## When to use me
+
+Use when the user asks to review a diff, patch, or proposed change before merging. Trigger on phrases like:
+- "review this diff"
+- "can you review changes.diff before I merge?"
+- "is this patch safe to apply?"
+- "look over this proposed change"
+
+## IMPORTANT: Read-only
+
+Do NOT apply the diff. Do NOT modify any source file. Do NOT create any new files (no report files, no notes). The entire deliverable is the review text in your reply.
+
+## IMPORTANT: Precision over volume
+
+Only flag real defects. Do not pad the review with style nits, speculation, or restatements of what the diff does. Code that looks unusual but is correct and documented is not a finding. A wrong finding is worse than a missing one.
+
+## Steps
+
+### Step 1: Read the diff
+
+Read the diff file the user pointed at. Identify which files and functions it touches.
+
+### Step 2: Read the current source
+
+Read the current version of every file the diff touches, so you understand the context the change lands in.
+
+### Step 3: Analyze the change
+
+Mentally apply the diff and look for defects:
+- Off-by-one errors in loops, ranges, slicing
+- Wrong comparison operators (`>` vs `>=`, boundary conditions vs the documented behavior)
+- Logic that contradicts the function's own docstring or naming
+- Broken callers, changed return types, missed edge cases (empty input, zero, None)
+
+Check claimed behavior against actual behavior: if the docstring says "threshold or more", the code must use `>=`.
+
+### Step 4: Deliver the review
+
+Reply with this exact structure:
+
+```
+## Review: <diff file name>
+
+### Findings
+
+- <file>:<approx line>: [HIGH|MEDIUM|LOW] <what is wrong and why>
+- ...or "No findings."
+
+VERDICT: APPROVE
+```
+
+or, if there is at least one HIGH or MEDIUM finding:
+
+```
+VERDICT: REQUEST_CHANGES
+```
+
+The reply MUST end with exactly one `VERDICT:` line containing either `APPROVE` or `REQUEST_CHANGES`.
+
+## Definition of done
+
+All of these must be true:
+- The diff file and every touched source file were read
+- Each finding names the file, approximate location, severity, and the concrete defect
+- No source file was modified and no new file was created
+- The reply ends with exactly one `VERDICT:` line
+
+## Error handling
+
+- If the diff file does not exist, say which file was expected and stop
+- If the diff does not apply cleanly to the current source (context mismatch), report that as a HIGH finding
+- If the diff is empty, reply with "No findings." and `VERDICT: APPROVE`

agent_skill_eval-0.3.0/skills/validate-config/SKILL.md

@@ -0,0 +1,91 @@
+---
+name: validate-config
+description: Validate a project configuration file against the team's structural requirements and naming/range conventions, then write a CONFIG_REPORT.md listing every violation.
+license: MIT
+compatibility: opencode, claude-code, codex
+metadata:
+  workflow: validate-report
+---
+
+## What I do
+
+Validate a JSON configuration file (e.g. `app_config.json`) in two passes:
+
+1. **Structural pass**: run the bundled validation script, which checks required keys and value types.
+2. **Convention pass**: check every value against the team's convention rules, which are documented in this skill's `references/config-rules.md`.
+
+Then write a `CONFIG_REPORT.md` in the workspace root listing every violation found, and summarize the result to the user.
+
+## When to use me
+
+Use when the user asks to validate, audit, lint, or check a configuration file, or asks whether a config follows team conventions or policy. Trigger on phrases like:
+- "validate the config"
+- "check app_config.json against our conventions"
+- "does this config follow our policy?"
+- "audit the configuration file"
+
+## IMPORTANT: Use the bundled resources
+
+Do NOT guess the team's conventions — they are arbitrary house rules you cannot infer. You MUST:
+
+- Run the bundled script `scripts/validate_config.py` (located in this skill's directory) for the structural pass. Do not re-implement its checks by hand.
+- Read `references/config-rules.md` (located in this skill's directory) for the convention pass. Every rule in that file must be checked.
+
+## Steps
+
+### Step 1: Run the structural validator
+
+The script lives in this skill's directory, next to this SKILL.md file. Run it with the config file as the argument:
+
+```bash
+python <path-to-this-skill>/scripts/validate_config.py app_config.json
+```
+
+(Use `python3` if `python` is not on the PATH.)
+
+It prints one `STRUCTURAL: ...` line per problem, or `STRUCTURAL: OK` if the structure is valid.
+
+### Step 2: Read the convention rules
+
+Read `references/config-rules.md` from this skill's directory. It defines numbered rules (RULE 1, RULE 2, ...) for naming, ranges, and allowed values.
+
+### Step 3: Check the config against every rule
+
+Read the config file and evaluate each rule from the reference document against the actual values. Record each violation with its rule number.
+
+### Step 4: Write CONFIG_REPORT.md
+
+Create `CONFIG_REPORT.md` in the workspace root with this structure:
+
+```markdown
+# Config validation report: <config file name>
+
+## Structural issues
+
+- <each STRUCTURAL line from the script, or "None">
+
+## Convention violations
+
+- RULE <n>: <key> = <actual value> — <what the rule requires>
+- ...or "None"
+
+TOTAL VIOLATIONS: <count of structural issues + convention violations>
+```
+
+### Step 5: Report to the user
+
+Summarize the findings in your reply and include the exact `TOTAL VIOLATIONS: <n>` line in your final message.
+
+## Definition of done
+
+All of these must be true:
+- The bundled `validate_config.py` script was executed (not re-implemented)
+- `references/config-rules.md` was read and every rule in it was checked
+- `CONFIG_REPORT.md` exists in the workspace root with both sections and the `TOTAL VIOLATIONS:` line
+- The final reply to the user contains the `TOTAL VIOLATIONS: <n>` line
+
+## Error handling
+
+- If the config file does not exist, report which file was expected and stop
+- If the config is not valid JSON, report the parse error as a structural issue and skip the convention pass
+- If the bundled script cannot be found, say so explicitly — do not silently substitute your own checks

agent_skill_eval-0.3.0/skills/validate-config/references/config-rules.md

@@ -0,0 +1,33 @@
+# Team config conventions
+
+House rules for service configuration files. These are conventions, not
+structural requirements — the structural validator script does not check
+them. Every rule below must be evaluated against the config.
+
+## RULE 1: service_name format
+
+`service_name` must be kebab-case (lowercase, words separated by `-`) and
+must start with the `svc-` prefix.
+
+Valid: `svc-data-pipeline`, `svc-auth`. Invalid: `DataPipeline`, `data_pipeline`, `auth`.
+
+## RULE 2: port range
+
+`port` must be in the team's allocated range **9000–9999** (inclusive).
+Ports outside this range collide with other teams' allocations.
+
+## RULE 3: log_level allowed values
+
+`log_level` must be exactly one of: `debug`, `info`, `warning`, `error`.
+No other values (e.g. `verbose`, `trace`, `WARN`) are accepted.
+
+## RULE 4: max_retries bound
+
+`max_retries` must be between **0 and 5** (inclusive). Higher values mask
+persistent failures and overload downstream services.
+
+## RULE 5: cache TTL floor
+
+When `features.cache_enabled` is `true`, `features.cache_ttl_seconds` must
+be **at least 300**. Shorter TTLs cause cache stampedes. (If caching is
+disabled, the TTL value is irrelevant.)

agent_skill_eval-0.3.0/skills/validate-config/scripts/validate_config.py

@@ -0,0 +1,68 @@
+#!/usr/bin/env python3
+"""Structural validator for team service configs.
+
+Checks required keys and value types only. Convention rules (naming,
+ranges, allowed values) live in references/config-rules.md and are
+checked by the agent, not by this script.
+
+Usage: python validate_config.py <config.json>
+Prints one "STRUCTURAL: ..." line per problem, or "STRUCTURAL: OK".
+Exit code 0 if structure is valid, 1 otherwise.
+"""
+
+import json
+import sys
+from pathlib import Path
+
+REQUIRED_KEYS = {
+    "service_name": str,
+    "port": int,
+    "log_level": str,
+    "max_retries": int,
+    "environment": str,
+}
+
+
+def validate(config_path: Path) -> list[str]:
+    problems: list[str] = []
+    try:
+        data = json.loads(config_path.read_text())
+    except FileNotFoundError:
+        return [f"STRUCTURAL: config file not found: {config_path}"]
+    except json.JSONDecodeError as e:
+        return [f"STRUCTURAL: config is not valid JSON: {e}"]
+
+    if not isinstance(data, dict):
+        return ["STRUCTURAL: top-level value must be a JSON object"]
+
+    for key, expected_type in REQUIRED_KEYS.items():
+        if key not in data:
+            problems.append(f"STRUCTURAL: missing required key '{key}'")
+        elif not isinstance(data[key], expected_type):
+            problems.append(
+                f"STRUCTURAL: key '{key}' must be {expected_type.__name__}, "
+                f"got {type(data[key]).__name__}"
+            )
+
+    features = data.get("features")
+    if features is not None and not isinstance(features, dict):
+        problems.append("STRUCTURAL: key 'features' must be an object")
+
+    return problems
+
+
+def main() -> int:
+    if len(sys.argv) != 2:
+        print("usage: validate_config.py <config.json>", file=sys.stderr)
+        return 2
+    problems = validate(Path(sys.argv[1]))
+    if not problems:
+        print("STRUCTURAL: OK")
+        return 0
+    for line in problems:
+        print(line)
+    return 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())

{agent_skill_eval-0.2.1 → agent_skill_eval-0.3.0}/src/skill_eval/__init__.py

@@ -3,6 +3,6 @@
 from importlib.metadata import PackageNotFoundError, version
 
 try:
-    __version__ = version("skill-eval")
+    __version__ = version("agent-skill-eval")
 except PackageNotFoundError:
-    __version__ = "0.1.0"
+    __version__ = "0.3.0"