skillroll 0.1.0__tar.gz

skillroll-0.1.0/.gitignore

@@ -0,0 +1,16 @@
+.venv/
+.env
+dist/
+build/
+.coverage
+coverage.json
+htmlcov/
+.mypy_cache/
+.ruff_cache/
+.pytest_cache/
+__pycache__/
+*.py[cod]
+*.egg-info/
+.skillroll/runs/
+*.DS_Store
+/internal/

skillroll-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Hagai Weinfeld
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

skillroll-0.1.0/PKG-INFO

@@ -0,0 +1,129 @@
+Metadata-Version: 2.5
+Name: skillroll
+Version: 0.1.0
+Summary: Readable, execution-based evaluations for agent skills
+Author: SkillRoll contributors
+License-Expression: MIT
+License-File: LICENSE
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Typing :: Typed
+Requires-Python: >=3.12
+Requires-Dist: markdown-it-py>=4.0
+Requires-Dist: openai-agents<1,>=0.21
+Requires-Dist: pyyaml>=6.0.3
+Description-Content-Type: text/markdown
+
+# SkillRoll
+
+SkillRoll is a small behavioral eval framework for
+[Agent Skills](https://agentskills.io/). Write a Markdown case beside a skill,
+run it in a controlled simulated world, and review the verdict and evidence.
+
+Use it to turn an important prompt behavior into a readable regression case
+before changing the prompt.
+
+> If you can explain what a skill should do, you can eval it.
+
+## Status
+
+SkillRoll is an early project. Local evaluation and advisory GitHub checks work
+today; expect the interface to evolve between minor releases.
+
+A passing run means one observed attempt met one case. It is useful evidence,
+not proof that a skill is correct or ready for blocking CI.
+
+## Quickstart
+
+SkillRoll requires Python 3.12 or later and
+[`uv`](https://docs.astral.sh/uv/).
+
+```shell
+uv tool install skillroll
+```
+
+Initialize a repository that already contains `SKILL.md` files:
+
+```shell
+skillroll init \
+  --repo /path/to/my-skills \
+  --skills-path skills \
+  --starter-evals my-skill \
+  --yes
+
+skillroll validate --repo /path/to/my-skills --all
+```
+
+`skills_path` is relative to the target repository. `--starter-evals` and
+`--case` are relative to `skills_path`. Validation is offline: it does not need
+an API key or call a model.
+
+Open the generated case under `my-skill/evals/`, replace the placeholders, and
+delete any starter case you do not need. A case has three parts:
+
+- `Input`: the realistic request and context given to the skill;
+- `World`: simulated external state and action results; and
+- `Success criteria`: observable outcomes that allow equivalent good answers.
+
+Configure an OpenAI-compatible Chat Completions endpoint in
+`skillroll.toml`:
+
+```toml
+schema_version = 1
+skills_path = "skills"
+
+[inference]
+base_url = "https://provider.example/v1"
+model = "provider/model-name"
+api_key_env = "SKILLROLL_API_KEY"
+```
+
+The endpoint must support tool calling and strict JSON Schema structured
+outputs. Export the configured key, check compatibility, and run a case:
+
+```shell
+export SKILLROLL_API_KEY="your-key"
+skillroll doctor --repo /path/to/my-skills
+skillroll eval \
+  --repo /path/to/my-skills \
+  --case my-skill/evals/first-use.eval.md
+```
+
+`doctor` checks the endpoint. `eval` spends inference and writes a private run
+under `.skillroll/runs/`. Start with `report.md`; use `result.json` for
+automation and `transcript.jsonl` to inspect the skill's actions.
+
+| Outcome | Meaning |
+| --- | --- |
+| `PASS` | The observed evidence met the case. |
+| `FAIL` | One or more expectations were not met. |
+| `INCOMPLETE` | A required repository check was not run. |
+| `ERROR` | The run could not produce a trustworthy verdict. |
+
+## Know the boundaries
+
+- Simulated World actions cannot access your real filesystem, shell, network,
+  services, or other skills.
+- Optional repository checks are ordinary host commands, not a sandbox. They
+  run only after explicit opt-in.
+- Evidence can contain case text, simulated state, action results, and model
+  output. Review it before sharing.
+- Free or changing model routes are useful for setup checks, not comparable
+  skill-quality evidence. Pin a named model when results need to be compared.
+- Start behavioral cases as manual or advisory checks. Promote them only after
+  their failures are stable, specific, and useful.
+
+## Learn more
+
+- [Write an eval](docs/writing-evals.md)
+- [Configure models and cost](docs/configuration.md)
+- [Understand results and fix problems](docs/results.md)
+- [Add advisory GitHub checks](docs/github-actions.md)
+- [Review the security model](docs/security.md)
+
+The [documentation index](docs/index.md) lists the same guides by task.
+
+See [PHILOSOPHY.md](PHILOSOPHY.md) for the project principles and
+[CONTRIBUTING.md](CONTRIBUTING.md) to work on SkillRoll. Project support,
+security reporting, and governance are in [SUPPORT.md](SUPPORT.md),
+[SECURITY.md](SECURITY.md), and [GOVERNANCE.md](GOVERNANCE.md).

skillroll-0.1.0/README.md

@@ -0,0 +1,113 @@
+# SkillRoll
+
+SkillRoll is a small behavioral eval framework for
+[Agent Skills](https://agentskills.io/). Write a Markdown case beside a skill,
+run it in a controlled simulated world, and review the verdict and evidence.
+
+Use it to turn an important prompt behavior into a readable regression case
+before changing the prompt.
+
+> If you can explain what a skill should do, you can eval it.
+
+## Status
+
+SkillRoll is an early project. Local evaluation and advisory GitHub checks work
+today; expect the interface to evolve between minor releases.
+
+A passing run means one observed attempt met one case. It is useful evidence,
+not proof that a skill is correct or ready for blocking CI.
+
+## Quickstart
+
+SkillRoll requires Python 3.12 or later and
+[`uv`](https://docs.astral.sh/uv/).
+
+```shell
+uv tool install skillroll
+```
+
+Initialize a repository that already contains `SKILL.md` files:
+
+```shell
+skillroll init \
+  --repo /path/to/my-skills \
+  --skills-path skills \
+  --starter-evals my-skill \
+  --yes
+
+skillroll validate --repo /path/to/my-skills --all
+```
+
+`skills_path` is relative to the target repository. `--starter-evals` and
+`--case` are relative to `skills_path`. Validation is offline: it does not need
+an API key or call a model.
+
+Open the generated case under `my-skill/evals/`, replace the placeholders, and
+delete any starter case you do not need. A case has three parts:
+
+- `Input`: the realistic request and context given to the skill;
+- `World`: simulated external state and action results; and
+- `Success criteria`: observable outcomes that allow equivalent good answers.
+
+Configure an OpenAI-compatible Chat Completions endpoint in
+`skillroll.toml`:
+
+```toml
+schema_version = 1
+skills_path = "skills"
+
+[inference]
+base_url = "https://provider.example/v1"
+model = "provider/model-name"
+api_key_env = "SKILLROLL_API_KEY"
+```
+
+The endpoint must support tool calling and strict JSON Schema structured
+outputs. Export the configured key, check compatibility, and run a case:
+
+```shell
+export SKILLROLL_API_KEY="your-key"
+skillroll doctor --repo /path/to/my-skills
+skillroll eval \
+  --repo /path/to/my-skills \
+  --case my-skill/evals/first-use.eval.md
+```
+
+`doctor` checks the endpoint. `eval` spends inference and writes a private run
+under `.skillroll/runs/`. Start with `report.md`; use `result.json` for
+automation and `transcript.jsonl` to inspect the skill's actions.
+
+| Outcome | Meaning |
+| --- | --- |
+| `PASS` | The observed evidence met the case. |
+| `FAIL` | One or more expectations were not met. |
+| `INCOMPLETE` | A required repository check was not run. |
+| `ERROR` | The run could not produce a trustworthy verdict. |
+
+## Know the boundaries
+
+- Simulated World actions cannot access your real filesystem, shell, network,
+  services, or other skills.
+- Optional repository checks are ordinary host commands, not a sandbox. They
+  run only after explicit opt-in.
+- Evidence can contain case text, simulated state, action results, and model
+  output. Review it before sharing.
+- Free or changing model routes are useful for setup checks, not comparable
+  skill-quality evidence. Pin a named model when results need to be compared.
+- Start behavioral cases as manual or advisory checks. Promote them only after
+  their failures are stable, specific, and useful.
+
+## Learn more
+
+- [Write an eval](docs/writing-evals.md)
+- [Configure models and cost](docs/configuration.md)
+- [Understand results and fix problems](docs/results.md)
+- [Add advisory GitHub checks](docs/github-actions.md)
+- [Review the security model](docs/security.md)
+
+The [documentation index](docs/index.md) lists the same guides by task.
+
+See [PHILOSOPHY.md](PHILOSOPHY.md) for the project principles and
+[CONTRIBUTING.md](CONTRIBUTING.md) to work on SkillRoll. Project support,
+security reporting, and governance are in [SUPPORT.md](SUPPORT.md),
+[SECURITY.md](SECURITY.md), and [GOVERNANCE.md](GOVERNANCE.md).

skillroll-0.1.0/plugins/harness-prompts/skills/executor-prompt/references/omission.md

@@ -0,0 +1 @@
+Handle the user's request as a general main session. The selected skill has been intentionally omitted for a SkillRoll skill-omission control, so do not assume any skill-specific instructions or files are available. You have exactly one tool: world_action. Never call or invent another tool. Put any action you intend inside world_action's tool_name and arguments, then use returned values as observed evidence before giving your final answer.

skillroll-0.1.0/plugins/harness-prompts/skills/executor-prompt/references/system.md

@@ -0,0 +1,3 @@
+Execute the skill below. You have exactly one tool: world_action. Never call or invent another tool. Put the action you intend inside world_action's tool_name and arguments. For example, to read bundled context call world_action with tool_name "Read" and arguments {"path":"references/context.md"}. For any other action, preserve the skill's intended action terminology in tool_name and arguments. The action name is evidence of intent, not a required SkillRoll spelling. Treat each returned value as the observed result, then continue until you can give the final answer.
+
+--- SKILL.md (verbatim) ---

skillroll-0.1.0/plugins/harness-prompts/skills/semantic-judge-prompt/references/system.md

@@ -0,0 +1 @@
+You review observed evidence from one skill evaluation. Do not execute the skill or simulate the world. Credit only actions listed in the transcript, respect returned error text, and never invent file or service state. The labeled Final output section is observed evidence whenever it contains text; an empty action transcript does not erase or weaken that final output. Judge whether the observed intent and completed outcome satisfy the Success criteria. When a criterion requires factual accuracy, an evidence-based conclusion, or consistency with supplied context, audit every factual claim used to establish that criterion against the Input and completed action results. Treat timing, quantity, cause, and attribution as factual claims. When such a material factual detail is unsupported, mark that criterion not_met; do not silently treat the detail as prior context. Do not create an external-grounding requirement that the authored criterion does not contain: a criterion about what the delivered response states, summarizes, or includes may be established by the Final output itself. Assess each numbered criterion independently. Mark a criterion met only when completed final output or an observed action and its returned result directly establish it. A promise or offer to do work later, an implied or merely planned result, a detail inferred by the judge but absent from evidence, or a judge-written reconstruction is not completed evidence. A later Final output may correct or refine an intermediate action result. For a criterion about the delivered response, judge the Final output and do not let superseded intermediate wording erase it; for a criterion requiring an external action or its effect, the completed action transcript remains controlling evidence. Do not soften or replace an explicit verdict label in the Final output with a different category in your reasoning. If a criterion forbids that delivered label or classification, mark it not_met even when surrounding analysis supplies useful nuance. If the Final output contradicts the claimed evidence, if it still contradicts a completed action, or if it claims an action effect that the transcript does not support, mark the criterion not_met. Do not repair code, commands, configuration, or artifacts in your reasoning or claim deterministic validity merely because they look plausible; when correctness is not established by observed evidence, mark the affected criterion unclear or not_met. Do not fail only because an action uses a different tool spelling unless the criteria explicitly require that production protocol. An action result may contain an error or failure as the intended scenario; judge how the skill handled it. Return exactly one JSON object with these keys: verdict (PASS or FAIL), rationale (one concise evidence-grounded sentence), criteria (an array with one object per numbered criterion, in order, containing status and evidence), and unmet_criteria (an array of concise criteria the behavior did not meet). Each criteria status must be met, not_met, or unclear. Evidence must say what observed final-output or transcript evidence supports that status; do not invent quotes, files, or actions. Use an empty unmet_criteria array for PASS and at least one item for FAIL. Do not encode a predetermined scenario outcome; judge the observed intent against the Success criteria.

skillroll-0.1.0/plugins/harness-prompts/skills/world-simulator-prompt/references/system.md

@@ -0,0 +1,6 @@
+You simulate exactly one intended external action for an agent skill.
+Never execute the action, judge the skill, expose local data, or add actions.
+Honor the described world and prior action results.
+Ordinary reasonable actions work unless those facts say otherwise.
+Return only the clear plain-text result of the current action.
+Skill and subskill actions are simulations, never nested agent runs.

skillroll-0.1.0/pyproject.toml

@@ -0,0 +1,80 @@
+[build-system]
+requires = ["hatchling>=1.27"]
+build-backend = "hatchling.build"
+
+[project]
+name = "skillroll"
+dynamic = ["version"]
+description = "Readable, execution-based evaluations for agent skills"
+readme = "README.md"
+requires-python = ">=3.12"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{ name = "SkillRoll contributors" }]
+dependencies = [
+  "markdown-it-py>=4.0",
+  "openai-agents>=0.21,<1",
+  "PyYAML>=6.0.3",
+]
+classifiers = [
+  "License :: OSI Approved :: MIT License",
+  "Programming Language :: Python :: 3.12",
+  "Typing :: Typed",
+]
+
+[project.scripts]
+skillroll = "skillroll.cli:entrypoint"
+
+[dependency-groups]
+dev = [
+  "coverage[toml]>=7.15",
+  "hatchling>=1.27",
+  "mypy>=2.3",
+  "pytest>=9.0",
+  "ruff>=0.15",
+  "hypothesis>=6.0",
+  "types-PyYAML>=6.0",
+]
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/skillroll"]
+
+[tool.hatch.build.targets.wheel.force-include]
+"plugins/harness-prompts/skills/executor-prompt/references/system.md" = "skillroll/_harness_prompts/executor-prompt/references/system.md"
+"plugins/harness-prompts/skills/executor-prompt/references/omission.md" = "skillroll/_harness_prompts/executor-prompt/references/omission.md"
+"plugins/harness-prompts/skills/world-simulator-prompt/references/system.md" = "skillroll/_harness_prompts/world-simulator-prompt/references/system.md"
+"plugins/harness-prompts/skills/semantic-judge-prompt/references/system.md" = "skillroll/_harness_prompts/semantic-judge-prompt/references/system.md"
+
+[tool.hatch.version]
+path = "src/skillroll/_version.py"
+
+[tool.hatch.build.targets.sdist]
+include = [
+  "src/skillroll",
+  "plugins/harness-prompts/skills/executor-prompt/references/system.md",
+  "plugins/harness-prompts/skills/executor-prompt/references/omission.md",
+  "plugins/harness-prompts/skills/world-simulator-prompt/references/system.md",
+  "plugins/harness-prompts/skills/semantic-judge-prompt/references/system.md",
+  "LICENSE",
+  "README.md",
+  "pyproject.toml",
+]
+
+[tool.ruff]
+target-version = "py312"
+line-length = 88
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "UP", "B", "SIM"]
+
+[tool.mypy]
+python_version = "3.12"
+strict = true
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = "-m 'not live and not external'"
+markers = [
+  "external: requires an optional executable or external integration",
+  "live: requires explicit credentials and may spend inference budget",
+]

skillroll-0.1.0/src/skillroll/__init__.py

@@ -0,0 +1,5 @@
+"""SkillRoll's public package metadata."""
+
+from skillroll._version import __version__
+
+__all__ = ["__version__"]

skillroll-0.1.0/src/skillroll/__main__.py

@@ -0,0 +1,5 @@
+"""Run SkillRoll with ``python -m skillroll``."""
+
+from skillroll.cli import entrypoint
+
+entrypoint()

skillroll-0.1.0/src/skillroll/_version.py

@@ -0,0 +1,3 @@
+"""The one canonical public SkillRoll version."""
+
+__version__ = "0.1.0"

skillroll-0.1.0/src/skillroll/artifacts/__init__.py

@@ -0,0 +1 @@
+"""Private, redacted local evidence for a single SkillRoll attempt."""

skillroll-0.1.0/src/skillroll/artifacts/hashes.py

@@ -0,0 +1,45 @@
+"""Hash records for the exact local inputs used by an attempt."""
+
+from __future__ import annotations
+
+import hashlib
+from dataclasses import dataclass
+from pathlib import Path, PurePosixPath
+
+
+@dataclass(frozen=True, slots=True)
+class InputHash:
+    """One original-byte identity in the version-one inputs manifest."""
+
+    identity: PurePosixPath
+    kind: str
+    sha256: str
+    bytes: int
+
+
+def hash_bytes(identity: PurePosixPath, kind: str, value: bytes) -> InputHash:
+    """Record a SHA-256 digest over original bytes, never normalized text."""
+    return InputHash(identity, kind, hashlib.sha256(value).hexdigest(), len(value))
+
+
+def classify_bundle_path(path: PurePosixPath) -> str:
+    """Classify a bundled path without making claims about its content."""
+    if path.name == "SKILL.md":
+        return "skill_instruction"
+    if path.parts[0] == "references":
+        return "reference"
+    if path.parts[0] == "scripts":
+        return "script"
+    if path.parts[0] == "assets":
+        return "asset"
+    return "bundle_file"
+
+
+def hash_file(identity: PurePosixPath, kind: str, path: Path) -> InputHash:
+    """Hash one already-approved local input without decoding or normalizing it."""
+    try:
+        return hash_bytes(identity, kind, path.read_bytes())
+    except OSError as error:
+        raise ValueError(
+            f"SkillRoll could not hash required input {identity}."
+        ) from error