skillroll 0.1.2__tar.gz → 0.2.0__tar.gz

{skillroll-0.1.2 → skillroll-0.2.0}/.gitignore

@@ -12,5 +12,6 @@ __pycache__/
 *.py[cod]
 *.egg-info/
 .skillroll/runs/
+.skillroll/experiments/
 *.DS_Store
 /internal/

skillroll-0.2.0/PKG-INFO

@@ -0,0 +1,370 @@
+Metadata-Version: 2.5
+Name: skillroll
+Version: 0.2.0
+Summary: A simple, drop-in eval harness for Agent Skills
+Author: SkillRoll contributors
+License-Expression: MIT
+License-File: LICENSE
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Typing :: Typed
+Requires-Python: >=3.12
+Requires-Dist: markdown-it-py>=4.0
+Requires-Dist: openai-agents<1,>=0.21
+Requires-Dist: pyyaml>=6.0.3
+Description-Content-Type: text/markdown
+
+<p align="center">
+  <img src="docs/assets/skillroll-mascot.png" alt="skillroll mascot: a hooded otter holding a twenty-sided die and field guide">
+</p>
+
+<h1 align="center">skillroll</h1>
+
+<p align="center">
+  <strong>Test agent skills in worlds you can describe.</strong><br>
+  Write eval cases in Markdown. A Dungeon Master simulates the outside world.
+  skillroll returns an evidence-backed verdict.
+</p>
+
+<p align="center">
+  <a href="#quickstart"><strong>Get started</strong></a> ·
+  <a href="docs/authoring.md">Author skills and evals</a> ·
+  <a href="docs/writing-evals.md">Write an eval</a> ·
+  <a href="docs/results.md">Understand a result</a>
+</p>
+
+skillroll is a drop-in eval harness for repositories of
+[Agent Skills](https://agentskills.io/). It turns skill decisions into readable
+regression checks.
+
+## Example
+
+### Skill
+
+```markdown
+# Adventurer
+
+Seek glory and treasure. Try to keep all your limbs.
+```
+
+### Eval
+
+````markdown
+# Inspect before risky action
+
+```skillroll
+schema_version: 1
+```
+
+## Input
+
+An ornate chest sits alone in a dungeon room. What do you do?
+
+## World
+
+The chest is a sleeping mimic. Inspection reveals teeth.
+
+## Success criteria
+
+- Inspect before touching.
+- Do not open a mimic.
+````
+
+### Session
+
+```text
+ADVENTURER → I open the chest.
+
+DM         → The chest opens you back.
+```
+
+### Judgment
+
+```text
+FAIL — Treasure: 0. Limbs retained: 3.
+```
+
+This is one check on a much broader skill. The eval cares about the choice, not
+an exact script, and the report shows why it failed.
+
+## The same framework fits any skill
+
+- `Input` is the request or task that invokes the skill.
+- `World` is the Dungeon Master's private brief: the files, APIs, services,
+  people, failures, and action results the skill may encounter.
+- `Success criteria` tell the judge what observable behavior must hold.
+
+A release skill can encounter CI that is still running. A browser skill can
+find a failed request behind a success message. A support skill can spot an
+open chargeback before issuing a refund.
+
+The skill receives the Input, not the World. The Dungeon Master answers its
+external actions using the World and action history. Deterministic rules can
+provide exact results when needed. The judge compares the resulting evidence
+with the success criteria.
+
+```text
+INPUT ──→ SKILL ←──→ DUNGEON MASTER
+              │             │
+              └── evidence ─┘
+                     │
+                     ▼
+                   JUDGE
+                 PASS / FAIL
+```
+
+No mock server. No fake SDK. No per-skill harness code.
+
+## Prompt TDD
+
+The failed Mimic case is now a regression test:
+
+1. Describe the failing or required behavior.
+2. Run the case and inspect what the skill actually did.
+3. Make the smallest prompt change that fixes the behavior.
+4. Run the case again and keep it.
+
+```diff
+ # Adventurer
+
+ Seek glory and treasure. Try to keep all your limbs.
++Inspect unfamiliar objects before touching them.
+```
+
+```text
+ADVENTURER → I inspect the chest from a safe distance.
+
+DM         → Its lid is breathing. You spot teeth underneath.
+
+ADVENTURER → Mimic. I leave it alone.
+
+PASS — Treasure: 0. Limbs retained: 4.
+```
+
+Different path. Better behavior. Permanent check.
+
+## Quickstart
+
+skillroll requires Python 3.12 or later and
+[`uv`](https://docs.astral.sh/uv/). Install the command-line tool:
+
+```shell
+uv tool install skillroll
+```
+
+Set up skillroll interactively and create `skillroll.toml` at the repository
+root. It automatically identifies the skills folder and helps configure the
+inference endpoint and API-key environment variable.
+
+```shell
+cd /path/to/repository
+skillroll init
+```
+
+Create a new eval for the `Adventurer` skill called
+`inspect-before-risky-action`:
+
+```shell
+skillroll new adventurer inspect-before-risky-action
+```
+
+This creates the eval template
+`adventurer/evals/inspect-before-risky-action.eval.md` for you to edit.
+
+Then validate all evals in the repository:
+
+```shell
+skillroll validate
+```
+
+Make an API key available to skillroll in your current shell:
+
+```shell
+export SKILLROLL_API_KEY="your-key"
+```
+
+Check the model connection:
+
+```shell
+skillroll doctor
+```
+
+Run all evals under the current working directory:
+
+```shell
+skillroll eval
+```
+
+Or run one specific eval, using its path relative to the skills folder:
+
+```shell
+skillroll eval --case adventurer/evals/inspect-before-risky-action.eval.md
+```
+
+The command prints its verdict and saves the report under the repository's
+`.skillroll/runs/` directory.
+
+## Inference
+
+Running an eval requires an API key for a compatible inference endpoint.
+skillroll works with OpenAI-compatible Chat Completions endpoints that support
+tool calling and strict JSON Schema structured outputs.
+
+If you entered model settings during `init`, they are already in
+`skillroll.toml`. Otherwise, add them now:
+
+```toml
+[inference]
+base_url = "https://provider.example/v1"
+model = "provider/model-name"
+api_key_env = "SKILLROLL_API_KEY"
+```
+
+### OpenRouter
+
+If you do not yet have an inference provider,
+[OpenRouter](https://openrouter.ai/) is a great place to start. It gives you
+control over API keys and spending, access to almost any model, and free
+inference on inexpensive models for sanity tests.
+
+Our go-to model is
+[`openai/gpt-5.6-luna-pro`](https://openrouter.ai/openai/gpt-5.6-luna-pro),
+which is cost-effective and performant enough for even complex evals.
+
+To use Luna Pro:
+
+```toml
+[inference]
+base_url = "https://openrouter.ai/api/v1"
+model = "openai/gpt-5.6-luna-pro"
+api_key_env = "SKILLROLL_API_KEY"
+```
+
+Or the free tier:
+
+```toml
+[inference]
+base_url = "https://openrouter.ai/api/v1"
+model = "openrouter/free"
+api_key_env = "SKILLROLL_API_KEY"
+```
+
+### Estimated Luna Pro cost per eval
+
+As of August 24, 2026,
+[OpenRouter lists Luna Pro](https://openrouter.ai/openai/gpt-5.6-luna-pro) at
+$0.20 per million input tokens and $1.20 per million output tokens. These
+working estimates include the compatibility check, skill run, Dungeon Master,
+and judge:
+
+| Case | Representative billed tokens | Estimated cost |
+| --- | ---: | ---: |
+| Short prompt, one action | 8K input + 1K output | about $0.003 |
+| Typical prompt, two actions | 20K input + 3K output | about $0.008 |
+| Large prompt, four actions | 60K input + 8K output | about $0.022 |
+
+Actual cost depends on prompt size, action count, output length, caching, and
+provider pricing. Batch runs share one compatibility check. Add current rates
+to `skillroll.toml` if you want reports to estimate cost from observed usage.
+
+```toml
+[pricing]
+currency = "USD"
+
+[pricing.models."openai/gpt-5.6-luna-pro"]
+input_per_million = 0.20
+output_per_million = 1.20
+```
+
+## Read the result
+
+Each run is saved under `.skillroll/runs/`. The report explains the verdict,
+shows the observed actions, and points to failed criteria. `result.json` is
+available for automation; `transcript.jsonl` contains the complete action
+history.
+
+| Outcome | Meaning |
+| --- | --- |
+| `PASS` | The observed behavior met the case. |
+| `FAIL` | The observed behavior missed a success criterion. |
+| `INCOMPLETE` | A required repository check did not run. |
+| `ERROR` | skillroll could not produce a trustworthy verdict. |
+
+## Add GitHub Actions
+
+After `skillroll.toml` exists, generate the advisory workflow:
+
+```shell
+skillroll init --github-workflow
+```
+
+This writes `.github/workflows/skillroll.yml` without replacing an existing
+workflow. Review it before committing. Pull requests always validate changed
+skills and evals without a model key.
+
+To enable automatic model-backed evals for owner-authored pull requests from
+the same repository, create a `skillroll-eval` environment with a secret named
+by `api_key_env`, then set the `SKILLROLL_LIVE_EVAL` repository variable to
+`true`. Fork pull requests never receive the key.
+
+The generated pull-request job uses one sample and no no-skill control. A
+manual **Run workflow** can request 1-10 samples and optionally compare each
+case without its selected skill. That comparison is authoring evidence, not a
+second CI gate, and roughly doubles model-backed runs.
+
+Start with advisory results. See the
+[GitHub Actions guide](docs/github-actions.md) for manual runs, repository
+checks, and artifact retention.
+
+## Keep evals small
+
+One case should cover one important behavior. A skill can accumulate as many
+focused cases as it needs. Narrow cases are easier to write, read, debug, and
+run in CI.
+
+skillroll bounds turns, time, and model output, and records inference usage in
+every report. Use deterministic rules for exact action results. Let the Dungeon
+Master handle behavior that would otherwise require mocks or elaborate setup.
+
+## Trust and security
+
+skillroll is local, open source infrastructure:
+
+- skillroll has no telemetry, analytics, tracking, or model tracing, and will
+  not add them.
+- There is no skillroll account or hosted service. Model traffic goes only to
+  the endpoint you configure when running `doctor` or an eval.
+- API keys stay in environment variables. skillroll does not write them to
+  configuration or artifacts and redacts the configured key from errors.
+- Run artifacts stay under `.skillroll/runs/` and sampled experiment reports
+  under `.skillroll/experiments/`; both are ignored by the generated `.gitignore`
+  rules.
+- skillroll is released under the permissive [MIT License](LICENSE). The code,
+  prompts, and security boundaries are public and reviewable.
+- The simulated World cannot access your real filesystem, shell, network, or
+  services.
+- Optional repository commands run on the host only after explicit opt-in.
+- Reports may contain case text, simulated state, and model output. Review them
+  before sharing.
+- One passing run is evidence for one case, not proof that a skill is correct.
+- Use cases as local or advisory checks before making them blocking CI gates.
+
+skillroll is an early project. Local evals and advisory GitHub checks work
+today; expect the interface to change between minor releases.
+
+## Learn more
+
+- [Author and audit good skills and evals](docs/authoring.md)
+- [Write an eval](docs/writing-evals.md)
+- [Configure models and cost](docs/configuration.md)
+- [Understand results and fix problems](docs/results.md)
+- [Add advisory GitHub checks](docs/github-actions.md)
+- [Review the security model](docs/security.md)
+
+The [documentation index](docs/index.md) lists the same guides by task.
+
+See [PRINCIPLES.md](PRINCIPLES.md) for the project principles and
+[CONTRIBUTING.md](.github/CONTRIBUTING.md) to work on skillroll. Project
+support, security reporting, and governance are in
+[SUPPORT.md](.github/SUPPORT.md), [SECURITY.md](.github/SECURITY.md), and
+[GOVERNANCE.md](.github/GOVERNANCE.md).