agent-skill-evals 0.1.0 → 0.1.1

package/README.md

@@ -1,155 +1,24 @@
 # agent-skill-evals
 
-Agent Skill Evals helps you test agent skills with [Promptfoo](https://www.promptfoo.dev/).
+Promptfoo-native testing for reusable agent skills with real agents and observable evidence.
 
-## Install
+Before running an eval, install and authenticate [Codex CLI](https://help.openai.com/en/articles/11096431), [Claude Code](https://docs.anthropic.com/en/docs/claude-code/getting-started), or [Pi](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/docs/quickstart.md). The npm dependencies do not include an agent runtime. For non-interactive Claude evals, export `CLAUDE_CODE_OAUTH_TOKEN` from `claude setup-token` or use `ANTHROPIC_API_KEY`; the isolated runtime cannot reuse a macOS Keychain login.
 
-```bash
-pnpm add -D promptfoo agent-skill-evals
+```sh
+npm install --save-dev agent-skill-evals promptfoo
+npx agent-skill-evals init --skill ./skills/my-skill --adapter claude-code
+npx agent-skill-evals check ./skills/my-skill
+npx promptfoo eval
 ```
 
-Use it to:
+`init` creates the Promptfoo wiring and a starter Test Pack. `check` validates the skill and tests without invoking an agent. Promptfoo remains the runtime and reporting UI; Agent Skill Evals creates an isolated World, records evidence, and grades skill-specific assertions.
 
-1. Check a `SKILL.md` file and its tests before you run an agent.
-2. Copy a sample project, run an agent in the copy, save evidence, and check what changed.
+Public entry points:
 
-Agent Skill Evals is a Promptfoo plugin package. Promptfoo is the eval runner:
-it reads the YAML configs, runs providers, and calls assertions. Keep using
-`promptfoo eval`; Agent Skill Evals adds skill-focused providers and assertions
-that Promptfoo can load from `file://` paths.
+- `agent-skill-evals/agent` — Promptfoo provider
+- `agent-skill-evals/assertions` — evidence assertions and budgets
+- `agent-skill-evals/test-generator` — clean Test Pack loader
 
-## Add Agent Skill Evals To Promptfoo
+Promptfoo is a peer dependency and remains directly visible to users.
 
-Create a `agent-skill-evals/` folder in your project and add small loader files:
-
-```js
-// agent-skill-evals/agent.js
-export { default } from "agent-skill-evals/agent";
-```
-
-```js
-// agent-skill-evals/skill-checks.js
-export { default } from "agent-skill-evals/skill-checks";
-```
-
-```js
-// agent-skill-evals/assertions.js
-export { default } from "agent-skill-evals/assertions";
-export * from "agent-skill-evals/assertions";
-```
-
-Use these files from normal Promptfoo configs with `file://./agent-skill-evals/...`.
-
-## Entrypoints
-
-The package uses Promptfoo-facing subpaths:
-
-- `agent-skill-evals/agent`
-- `agent-skill-evals/skill-checks`
-- `agent-skill-evals/assertions`
-
-There is no root import from `agent-skill-evals`.
-
-## Metrics
-
-- `skill.checks` checks a skill file and its tests before an agent runs.
-- `skill.test` checks evidence after an agent run.
-- `skill.budget` checks real-agent token usage after an agent run.
-- `skill.activation`, `skill.budgets`, `skill.context`, `skill.instructions`, `skill.tests`, and `skill.verifiers` run focused Skill Checks.
-
-## Example: Check A Skill Before Runtime
-
-Use `skill.checks` to check that a `SKILL.md` file and its tests are ready to
-run:
-
-```yaml
-description: Skill checks
-
-prompts:
-  - "skill-check"
-
-providers:
-  - id: file://./agent-skill-evals/skill-checks.js
-
-tests:
-  - description: bugfix skill checks
-    vars:
-      skillPath: ./skills/bugfix-workflow
-      testsGlob: ./tests/bugfix-workflow.yaml
-    assert:
-      - type: javascript
-        metric: skill.checks
-        value: file://./agent-skill-evals/assertions.js
-        config:
-          metric: skill.checks
-```
-
-Run it with Promptfoo:
-
-```bash
-promptfoo eval -c promptfoo.skill-checks.yaml
-```
-
-## Example: Test A Skill On A Copied Project
-
-Use `skill.test` when you want the agent to work on a copied fixture and then
-check the evidence from that run:
-
-```yaml
-description: Runtime skill test
-
-prompts:
-  - "{{prompt}}"
-
-providers:
-  - id: file://./agent-skill-evals/agent.js
-    config:
-      command: codex
-      args:
-        - exec
-        - --json
-        - --full-auto
-
-tests:
-  - description: fixes login redirect locally
-    vars:
-      prompt: Fix the login redirect bug.
-      fixture: ./fixtures/login-bug
-      preconditions:
-        - verifier.fails:
-            run: ./verify_login_redirect.sh
-      should:
-        - verifier.succeeds:
-            run: ./verify_login_redirect.sh
-        - file.contains:
-            path: app.js
-            text: /dashboard
-      should_not:
-        - file.changes_outside_scope:
-            scope:
-              - app.js
-    assert:
-      - type: javascript
-        metric: skill.test
-        value: file://./agent-skill-evals/assertions.js
-        config:
-          metric: skill.test
-```
-
-Run it with Promptfoo:
-
-```bash
-promptfoo eval -c promptfoo.codex.yaml
-```
-
-Agent Skill Evals copies `fixture` before the agent runs. The original sample project
-stays clean, and `skill.test` checks the recorded evidence: changed files,
-command results, tool calls, final output, usage, and run details.
-
-## Docs
-
-- [Getting Started](https://github.com/akshay5995/agent-skill-evals/blob/main/docs/guide/getting-started.md)
-- [Promptfoo Setup](https://github.com/akshay5995/agent-skill-evals/blob/main/docs/guide/promptfoo-setup.md)
-- [Runtime Checks](https://github.com/akshay5995/agent-skill-evals/blob/main/docs/guide/runtime-checks.md)
-- [Brand Deck Example](https://github.com/akshay5995/agent-skill-evals/blob/main/docs/examples/brand-deck-skill.md)
-- [Metrics](https://github.com/akshay5995/agent-skill-evals/blob/main/docs/guide/metrics.md)
+See the [Getting Started guide](https://akshay5995.github.io/agent-skill-evals/guide/getting-started) for the first working eval and the [Reference](https://akshay5995.github.io/agent-skill-evals/guide/reference) for Test Packs, runtime checks, mocks, evidence, and budgets.

package/dist/agent/index.d.mts

@@ -1,3 +1,2 @@
-import { i as EvidenceSnapshot } from "../internal-services-DbsekQ_K.mjs";
-import { i as evidenceFromSnapshot, n as AgentSkillEvalsProviderMetadata, r as EvidenceCollector, t as AgentSkillEvalsProvider } from "../index-4l7TCFny.mjs";
-export { AgentSkillEvalsProvider, AgentSkillEvalsProvider as default, AgentSkillEvalsProviderMetadata, EvidenceCollector, EvidenceSnapshot, evidenceFromSnapshot };
+import { a as evidenceFromSnapshot, i as EvidenceCollector, n as PreparedRun, o as EvidenceSnapshot, r as AgentSkillEvalsProviderMetadata, t as AgentSkillEvalsProvider } from "../index-D9lGL0Qg.mjs";
+export { AgentSkillEvalsProvider, AgentSkillEvalsProvider as default, AgentSkillEvalsProviderMetadata, EvidenceCollector, EvidenceSnapshot, PreparedRun, evidenceFromSnapshot };

package/dist/agent/index.mjs

@@ -1,2 +1,2 @@
-import { i as evidenceFromSnapshot, r as EvidenceCollector, t as AgentSkillEvalsProvider } from "../agent-CM7fIL_C.mjs";
+import { a as EvidenceCollector, o as evidenceFromSnapshot, t as AgentSkillEvalsProvider } from "../agent-B15R7Xlr.mjs";
 export { AgentSkillEvalsProvider, AgentSkillEvalsProvider as default, EvidenceCollector, evidenceFromSnapshot };