doesitfire 0.1.2__py3-none-any.whl

doesitfire-0.1.2.dist-info/METADATA

@@ -0,0 +1,306 @@
+Metadata-Version: 2.5
+Name: doesitfire
+Version: 0.1.2
+Summary: Does your SKILL.md actually fire? Test skill activation in Claude Code, Codex and OpenCode with one command.
+Project-URL: Homepage, https://github.com/diegoperezg7/doesitfire
+Project-URL: Repository, https://github.com/diegoperezg7/doesitfire
+Project-URL: Issues, https://github.com/diegoperezg7/doesitfire/issues
+Project-URL: Changelog, https://github.com/diegoperezg7/doesitfire/blob/main/CHANGELOG.md
+Author: Diego Pérez
+License-Expression: MIT
+License-File: LICENSE
+Keywords: agent-skills,claude-code,codex,evaluation,llm,opencode,skill-md,testing
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.11
+Requires-Dist: openai==3.42.1
+Requires-Dist: pyyaml==6.0.3
+Requires-Dist: rich==15.0.0
+Requires-Dist: typer==0.27.3
+Description-Content-Type: text/markdown
+
+<p align="center">
+  <img src="docs/assets/banner.svg" alt="doesitfire" width="560">
+</p>
+
+<p align="center">
+  <a href="https://github.com/diegoperezg7/doesitfire/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/diegoperezg7/doesitfire/actions/workflows/ci.yml/badge.svg"></a>
+  <a href="LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/license-MIT-a1319b"></a>
+  <a href="https://github.com/diegoperezg7/doesitfire/releases"><img alt="Release" src="https://img.shields.io/github/v/release/diegoperezg7/doesitfire?color=a1319b"></a>
+  <a href="https://github.com/diegoperezg7/doesitfire/stargazers"><img alt="Stars" src="https://img.shields.io/github/stars/diegoperezg7/doesitfire?style=flat&color=a1319b"></a>
+</p>
+
+<p align="center"><b>Does your SKILL.md actually fire? Test skill activation in Claude Code, Codex and OpenCode with one command.</b></p>
+
+<p align="center"><a href="README.es.md">Leer en español</a> · <a href="https://diegoperezg7.github.io/doesitfire/">Example report</a></p>
+
+<p align="center"><img src="docs/assets/demo.gif" alt="doesitfire demo in a terminal" width="820"></p>
+
+You wrote a skill. Its `description` is the only thing the agent sees when it decides whether to load it.
+Does Claude Code load it when a user asks for it? Does Codex? Does OpenCode? Does it also load when nobody asked?
+doesitfire runs the real agent CLIs on prompts that should and should not trigger each skill, a few times each,
+and gives you one activation matrix per agent.
+
+## Why
+
+Other tools test part of this:
+
+| Tool | What it measures | Agents |
+|---|---|---|
+| [microsoft/waza](https://github.com/microsoft/waza) (1,499★) | Has trigger tests, run through its GitHub Copilot executor | GitHub Copilot |
+| `claude plugin eval` (built into the Claude Code CLI) | Eval cases with graders against a plugin or skills dir | Claude Code only |
+| [darkrishabh/agent-skills-eval](https://github.com/darkrishabh/agent-skills-eval) (809★) | Output quality with vs without the skill injected | Not activation |
+| [NVIDIA/SkillEvaluator](https://github.com/NVIDIA/SkillEvaluator) (626★) | Output quality with vs without the skill injected | Not activation |
+| **doesitfire** | **Whether the real agent loads the skill on its own**, and how often it loads it when it should not | **Claude Code, Codex, OpenCode** |
+
+Star counts from the GitHub API on 3 Nov 2026.
+
+## Quickstart
+
+The demo needs no API key, no Docker and no model. It shows a **recorded** real run of the three
+example skills in [`examples/`](examples):
+
+```bash
+uvx --from git+https://github.com/diegoperezg7/doesitfire@v0.1.2 doesitfire demo
+```
+
+or from a clone:
+
+```bash
+git clone https://github.com/diegoperezg7/doesitfire && cd doesitfire && uv run doesitfire demo
+```
+
+`uvx doesitfire` is coming to PyPI.
+
+## Example output
+
+<!-- results:start -->
+Recorded on 2026-11-03 on an Apple M4, 16 GB with doesitfire 0.1.0. Each agent ran **its own CLI** with the model shown; read each column on its own, not as a ranking of agents.
+
+| Agent | CLI | Model | Skills | Prompts | Repeats | Runs | Errored (excluded) |
+|---|---|---|---|---|---|---|---|
+| claude-code | 2.1.317 | ollama/qwen3:4b-16k | 2 | 12 of 24 | 1 | 12 | 1 |
+| opencode | 1.19.5 | mistral/ministral-8b-latest | 3 | 24 | 3 | 72 | 11 |
+
+Errors: claude-code: 1 x `timeout after 900s`; opencode: 11 x `Bad Request`
+
+Fired runs / valid runs. *should* = prompts that should load the skill; *should not* = near misses.
+
+| Skill | claude-code · should / should not | opencode · should / should not |
+|---|---|---|
+| `commit-message` | 1/4 / 0/4 | 12/12 / 3/12 |
+| `regex-explainer` | 0/3 / 0/0 | 6/6 / 0/12 |
+| `dockerfile-review` | 0/0 / 0/0 | 12/12 / 0/7 |
+| **all** | 1/7 / 0/4 | 30/30 / 3/31 |
+
+Recall per agent: claude-code 14%, opencode 100%. False-fire rate: claude-code 0%, opencode 10%.
+
+The raw JSON events of every run are in [`examples/recorded/logs/`](examples/recorded/logs).
+
+<p align="center"><img src="docs/assets/report-light.png" alt="HTML report" width="820"></p>
+<!-- results:end -->
+
+**claude-code: 11 valid runs of 24 prompts (12 run, 1 timeout excluded as infrastructure), 1 repeat. opencode: 61 valid runs of 72 (24 prompts x 3 repeats, 11 Mistral HTTP 400 excluded). codex: not in the recorded demo.**
+
+The suite is [`examples/doesitfire.yaml`](examples/doesitfire.yaml) (all agents on Ollama) and
+[`examples/doesitfire.mistral.yaml`](examples/doesitfire.mistral.yaml) (OpenCode on Mistral). OpenCode's errored runs are
+HTTP 400s from Mistral: when the model calls a tool that doesitfire denies (for example `WebFetch`), OpenCode's next
+request ends with an assistant message, which Mistral rejects. They are excluded from the rates, not counted as "did not fire".
+
+## Test your own skills
+
+You need Docker and either Ollama (free, local) or an API key.
+
+```bash
+cd your-repo                      # with skills in .claude/skills, .agents/skills, skills/, ...
+uvx --from git+https://github.com/diegoperezg7/doesitfire@v0.1.2 doesitfire init
+# edit doesitfire.yaml: 4+ prompts that should load each skill, 4+ near misses that should not
+uvx --from git+https://github.com/diegoperezg7/doesitfire@v0.1.2 doesitfire run
+```
+
+`run` builds the agent image on first use (about 1.9 GB, a few minutes), then prints the matrix and writes
+`doesitfire-results/` with `results.json`, `report.html`, `junit.xml`, `badge.svg` and `badge.json`.
+
+### Local and free with Ollama
+
+The agent CLIs send 6k to 11k token prompts, and Ollama loads models with a 4k context by default, so create a
+16k variant once (same weights, no extra download):
+
+```bash
+ollama pull qwen3:4b
+ollama create qwen3:4b-16k -f ollama/Modelfile   # FROM qwen3:4b + PARAMETER num_ctx 16384
+```
+
+On Linux, start Ollama with `OLLAMA_HOST=0.0.0.0` so containers can reach it.
+
+### Paid models
+
+Keys are read from environment variables only (see [`.env.example`](.env.example)). Paid providers need a
+price per million tokens so `--max-cost` (default 1 USD) can stop the run before it overspends:
+
+```yaml
+agents:
+  claude-code: {provider: anthropic, model: claude-haiku-5-5, price: {input: 1.0, output: 5.0}}
+  codex:       {provider: openai, model: gpt-5.6-luna, price: {input: 0.20, output: 1.20}}
+  opencode:    {provider: openrouter, model: deepseek/deepseek-v4.1-flash, price: {input: 0.045, output: 0.50}}
+```
+
+| Agent | Providers | API it needs |
+|---|---|---|
+| Claude Code | `anthropic` (`ANTHROPIC_API_KEY` or `CLAUDE_CODE_OAUTH_TOKEN`), `openrouter`, `ollama` | Anthropic Messages |
+| Codex | `openai`, `openrouter`, `ollama` | OpenAI Responses |
+| OpenCode | `ollama`, `openrouter`, `openai`, `mistral`, `anthropic` | Chat Completions or Anthropic |
+
+## How it works
+
+```mermaid
+flowchart LR
+  Y[doesitfire.yaml<br/>skills + prompts] --> R[runner<br/>skill x prompt x agent x repeat]
+  R --> C1
+  subgraph net [internal Docker network, no route out]
+    C1[agent container<br/>Claude Code / Codex / OpenCode<br/>skills in its discovery dir]
+    P[egress proxy<br/>model API only]
+    C1 --> P
+  end
+  P --> M[(model API<br/>or Ollama)]
+  C1 -- JSON events --> D[activation detector]
+  D --> O[matrix · results.json · JUnit · HTML · badge]
+```
+
+For every skill, prompt, agent and repeat, doesitfire starts a fresh container, copies the skills into that
+agent's discovery directory (`.claude/skills`, `.agents/skills`, `.opencode/skills`) and runs the CLI headless.
+It reads the CLI's JSON event stream as it arrives:
+
+| Agent | Command | A run counts as fired when |
+|---|---|---|
+| Claude Code | `claude -p --output-format stream-json --verbose` | a `Skill` tool call names the skill |
+| OpenCode | `opencode run --format json` | its native `skill` tool is called with the skill |
+| Codex (experimental) | `codex exec --json` | a command reads `<skill>/SKILL.md`. Codex documents no activation event, so this is inferred |
+
+With a local model, a run stops as soon as the outcome is known (a tested skill fired, or the turn cap was hit).
+With a paid model, it runs to the end so the CLI reports token usage for the budget.
+
+**Metrics**, per agent and skill: **recall** (should-fire runs that loaded the skill), **false-fire rate**
+(should-not runs that loaded it) and **precision** (loads that were wanted). Runs that error out (timeout,
+API error) are left out of every rate and shown apart.
+
+**Isolation.** Each container is non-root, with a read-only root filesystem, all capabilities dropped,
+`no-new-privileges`, CPU, memory, pids and time limits. Its only network peer is an egress proxy that allows the
+model API host and port and returns 403 for everything else (blocked attempts are listed in the report).
+Bash, Write, Edit and web tools are denied in Claude Code, edit, bash and webfetch in OpenCode, and turns are
+capped at 2. Codex has no read-only file tool and its own sandbox cannot start inside an unprivileged container,
+so its shell runs inside the container limits. doesitfire refuses to run without Docker. Details in
+[SECURITY.md](SECURITY.md).
+
+## Configuration
+
+```yaml
+repeats: 3        # runs per prompt and agent; activation is not deterministic
+timeout: 300      # seconds per agent run
+max_turns: 2      # agent turns per run; activation happens in the first one
+cpus: 1           # per container
+memory: 1g        # per container
+
+agents:
+  claude-code: {provider: ollama, model: qwen3:4b-16k}
+  codex:       {provider: ollama, model: qwen3:4b-16k}
+  opencode:    false            # skip an agent
+
+skills:
+  - path: .claude/skills/commit-message     # a directory with a SKILL.md
+    should_fire:
+      - "Write a commit message for my change: I added a login form."
+    should_not_fire:
+      - "How do I undo my last git commit but keep the changes?"
+```
+
+| Command | What it does |
+|---|---|
+| `doesitfire init [DIR]` | Writes `doesitfire.yaml` with every skill found under `DIR` |
+| `doesitfire run` | Runs the matrix. `--agents`, `--skills`, `--repeats`, `--max-cost`, `--min-recall`, `--max-false-fire`, `--out` |
+| `doesitfire report results.json` | Re-renders the matrix, HTML, JUnit and badge |
+| `doesitfire demo` | Shows the recorded example run |
+
+`--min-recall 0.75 --max-false-fire 0.25` makes `run` exit 1 when any agent x skill cell misses them; that is
+what you want in CI. Budget stops exit 3 and keep the partial results.
+
+## GitHub Action
+
+Linux runners only (the agents run in Docker; hosted macOS runners have no Docker):
+
+```yaml
+- uses: diegoperezg7/doesitfire@v0.1.2
+  with:
+    config: doesitfire.yaml
+    max-cost: "1.0"
+    min-recall: "0.75"
+    max-false-fire: "0.25"
+  env:
+    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+```
+
+A complete workflow that also uploads the report and the JUnit results is in
+[`examples/workflows/doesitfire.yml`](examples/workflows/doesitfire.yml).
+
+**Badge.** `run` writes `badge.svg` and `badge.json` (a [shields.io endpoint](https://shields.io/badges/endpoint-badge)).
+Commit `badge.svg` and link it, or publish `badge.json` (for example to GitHub Pages) and use
+`https://img.shields.io/endpoint?url=<url-of-badge.json>`.
+
+## What was live-tested
+
+| Agent | Live-tested with | Notes |
+|---|---|---|
+| Claude Code 2.1.317 | Ollama `qwen3:4b` (16k context) through Ollama's Anthropic-compatible `/v1/messages`. The recorded demo has 12 of the 24 prompts (1 repeat): at 4-14 minutes per run on this laptop the rest did not finish in time | No Claude model: this release was built without an Anthropic key. `provider: anthropic` (API key or `CLAUDE_CODE_OAUTH_TOKEN` from `claude setup-token`) is unit-tested, not live-tested. |
+| Codex 0.166.0 | Supported; activation detection validated in the Docker integration test (real Codex CLI against the mock model), not in the recorded demo | Detection is **experimental** (reads of `SKILL.md`). A live Ollama run was started but did not finish in time for v0.1.0. |
+| OpenCode 1.19.5 | Mistral `ministral-8b-latest` (paid, OpenAI-compatible) | The recorded demo uses it. Also checked against Ollama during development. |
+
+Mistral is the only paid provider called live (OpenCode, 72 runs, cost measured from usage). Anthropic, OpenAI and
+OpenRouter are wired and unit-tested, including the `--max-cost` guard, but were not called. CI runs the three real CLIs in Docker against a scripted mock
+model ([`tests/mockmodel.py`](tests/mockmodel.py)) on every push, so a CLI output change breaks the build, not your results.
+
+Every agent runs with its CLI defaults, thinking included. Turning qwen3's thinking off in Ollama looks like a speed-up,
+but the model then writes its reasoning as plain text and stops before emitting the tool call, so skills look like
+they never fire. doesitfire does not do that.
+
+One Ollama quirk is handled for you: Ollama's Responses API drops input messages with role `developer`, which is where
+Codex lists the available skills. For Codex on Ollama only, the egress proxy rewrites them to `system` (plain HTTP
+to your local Ollama; HTTPS traffic to paid APIs is tunnelled untouched).
+
+## Use it from Claude Code
+
+Install the plugin and ask Claude in plain words; its `test-skill-activation` skill finds your skills, writes prompts that should and should not trigger each one, agrees the budget with you, runs doesitfire and rewrites the descriptions of skills that miss, then re-runs them to check.
+
+```text
+/plugin install doesitfire --marketplace diegoperezg7/doesitfire
+```
+
+The skill tells Claude to ask before any paid run, and `--max-cost` still caps the spend.
+
+## Roadmap
+
+- Cursor, Gemini CLI and GitHub Copilot CLI.
+- Suggesting prompts and better descriptions with an LLM.
+- Multi-turn conversations.
+- Conflicts between skills (two skills competing for the same prompt).
+- GitHub Marketplace listing.
+
+## Part of a series
+
+Open source tools that evaluate agents and LLMs on your own data:
+
+- [memarena](https://github.com/diegoperezg7/memarena): Benchmark memory layers on your own conversations.
+- [parsearena](https://github.com/diegoperezg7/parsearena): Compare PDF parsers side by side and their effect on retrieval.
+- [commitarena](https://github.com/diegoperezg7/commitarena): Replay real bug fixes from your git history against any model.
+- [toolbloat](https://github.com/diegoperezg7/toolbloat): Measure the token cost of your MCP stack and tool-choice errors.
+- [rulelab](https://github.com/diegoperezg7/rulelab): Find which AGENTS.md rules earn their tokens.
+
+## License
+
+[MIT](LICENSE). The three example skills in `examples/skills` are MIT too and were written for this project.

doesitfire-0.1.2.dist-info/RECORD

@@ -0,0 +1,21 @@
+doesitfire/__init__.py,sha256=7TXg0i0AfNQVIZeN5Z-t8LrwhdhFSIO7y-EdXjcYJis,119
+doesitfire/agents.py,sha256=dYDZCSnDIKSrIqwViySahL6iBiXBHhYc-m6Wz3QWAI8,12463
+doesitfire/cli.py,sha256=PI1Ngjqtx3Cd_viYmA6ChdoygslvCMU3Rypg2Rg0eFc,10250
+doesitfire/config.py,sha256=o0eFm0MJoC0Y4U-rQX___PwqZOqUUso3lc0knWbdnsU,9011
+doesitfire/errors.py,sha256=44W96KDMvC1Dboq_7REXA3vDc3byHW16oo51u-E1j5A,468
+doesitfire/html.py,sha256=7bJ4DEaCVxBY0l_jfeS9tXAn2orO1DiBfJrwe2hEO2o,842
+doesitfire/metrics.py,sha256=M_KgSbz4IaKUyl3VwEbTtIj4VaRkDshwRV16yGITeTY,4354
+doesitfire/providers.py,sha256=21XMriMZNKnzn13YH10ecXswen3G1X5a_lr5KLh1bEI,4470
+doesitfire/report.py,sha256=x7mUZiauwQgxcwxLdzDR6cRs5u4RTBCFDN_AKkj3D5c,7380
+doesitfire/runner.py,sha256=GFl8ivn-YTsHFa-M9xbuRO_1kHGIRqDBJFydMkKC8S8,11822
+doesitfire/sandbox.py,sha256=CTv4Fnob0BbBkH9Zo3mxAzVGrJceB69WK5YxuuknX70,9782
+doesitfire/data/demo-results.json,sha256=9Ll86ytunHdFFpyktDNlpW390LWdUcia3Ez7DP3P4tE,43917
+doesitfire/docker/Dockerfile,sha256=r607pO-q9RQtZUlLSqpbBtsynEmVjdRpjVV5SmAvn6E,877
+doesitfire/docker/egress-proxy.mjs,sha256=LBJywgAi1NWbP-UBHjsvWyvHbfvHRL3JE0i8N6mps00,4018
+doesitfire/docker/entrypoint.sh,sha256=9gzFyXGldF37nsO0Fiz5PYQosgF_V_WfFXzJ8SRn3rY,494
+doesitfire/templates/report.html,sha256=GJrYa0DjD3fTLbnAxeuG-31-2hQWi_dSrJcQxkuBwhg,25221
+doesitfire-0.1.2.dist-info/METADATA,sha256=qipVW5uHlm5o1Sy_iGenhU4FsbItCZARkvpigf7_7Dw,15748
+doesitfire-0.1.2.dist-info/WHEEL,sha256=2iZrJkDIkeDh0k9SikKMqFuRP-ua6mb4Ir1AsER4BEw,87
+doesitfire-0.1.2.dist-info/entry_points.txt,sha256=MqMbl0n5oMepTHAzcXY3PaaZ_9qr1i_cjXZnAZLm-qI,51
+doesitfire-0.1.2.dist-info/licenses/LICENSE,sha256=A7Jh3ub2oGhc2l6P9raz5AeuQKzRIinCPYAXENE2Lig,1069
+doesitfire-0.1.2.dist-info/RECORD,,

doesitfire-0.1.2.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.33.0
+Root-Is-Purelib: true
+Tag: py3-none-any

doesitfire-0.1.2.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+doesitfire = doesitfire.cli:main

doesitfire-0.1.2.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Diego Pérez
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.