skill-harness 0.1.0 → 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +46 -0
- package/package.json +6 -4
package/README.md
ADDED
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
# skill-harness
|
|
2
|
+
|
|
3
|
+
A portable **test / optimize loop for agent skills**, driven from [pi](https://parallel.ai).
|
|
4
|
+
Point it at a repo of skills, and for any skill with a `tests/specification.yaml`
|
|
5
|
+
spec it will run each scenario, LLM-judge the transcript, score it against a ship
|
|
6
|
+
bar, and open an interactive review UI so you can measure a `SKILL.md` edit.
|
|
7
|
+
|
|
8
|
+
## Install
|
|
9
|
+
|
|
10
|
+
```bash
|
|
11
|
+
npm i -g skill-harness
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
## The loop
|
|
15
|
+
|
|
16
|
+
1. **list** — discover which skills have a spec.
|
|
17
|
+
2. **lint** — validate specs/fixtures, free (no models, no keys) — the CI gate.
|
|
18
|
+
3. **run** — run every scenario on `pi` (skill active), grade with an LLM judge,
|
|
19
|
+
write `results.yaml`, print a scorecard.
|
|
20
|
+
4. **review** — open an interactive matrix UI: flip verdicts, add notes, they
|
|
21
|
+
persist back to `results.yaml`.
|
|
22
|
+
5. **add-test** + re-`run` — add a scenario, then re-run to measure a `SKILL.md` edit.
|
|
23
|
+
|
|
24
|
+
## Commands
|
|
25
|
+
|
|
26
|
+
| command | does |
|
|
27
|
+
|---|---|
|
|
28
|
+
| `skill-harness list --skills <root>` | discovered skills + spec status |
|
|
29
|
+
| `skill-harness lint <skill\|all> --skills <root>` | validate specs/fixtures; CI gate, exits non-zero on findings |
|
|
30
|
+
| `skill-harness run <skill\|all> --skills <root> [--model p:m ...] [--judge p:m]` | run scenarios, grade, score |
|
|
31
|
+
| `skill-harness grade <run-dir> [--judge p:m]` | re-grade saved transcripts — no model re-run |
|
|
32
|
+
| `skill-harness review <skill> --skills <root> [--port N]` | serve the interactive review UI |
|
|
33
|
+
| `skill-harness add-test <skill> --skills <root> --id ID --title T ...` | scaffold a new scenario |
|
|
34
|
+
|
|
35
|
+
**Judge ≠ subject.** The judge model must differ from the model under test —
|
|
36
|
+
same-family grading inflates scores; `skill-harness` warns when they resemble
|
|
37
|
+
each other.
|
|
38
|
+
|
|
39
|
+
**`lint` and `list` are free** (pure static checks, no `pi`, no API keys);
|
|
40
|
+
`run` (and re-grading) spend model tokens.
|
|
41
|
+
|
|
42
|
+
## More
|
|
43
|
+
|
|
44
|
+
- Repo + full docs: https://github.com/mojomanyana/skill-harness
|
|
45
|
+
- Step-by-step usage: [`docs/USAGE.md`](https://github.com/mojomanyana/skill-harness/blob/main/docs/USAGE.md)
|
|
46
|
+
- CI Action (free spec lint on every PR): `uses: mojomanyana/skill-harness@v1`
|
package/package.json
CHANGED
|
@@ -1,17 +1,19 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "skill-harness",
|
|
3
|
-
"version": "0.1.
|
|
4
|
-
"description": "Test/optimize loop for agent skills
|
|
3
|
+
"version": "0.1.1",
|
|
4
|
+
"description": "Test/optimize loop for agent skills — run spec'd scenarios on pi, LLM-judge, score, review, re-run",
|
|
5
5
|
"type": "module",
|
|
6
|
+
"keywords": ["agent-skills", "skill", "llm", "testing", "eval", "grading", "pi", "ci", "harness"],
|
|
6
7
|
"bin": {
|
|
7
8
|
"skill-harness": "bin.js"
|
|
8
9
|
},
|
|
9
10
|
"files": [
|
|
10
11
|
"bin.js",
|
|
11
|
-
"LICENSE"
|
|
12
|
+
"LICENSE",
|
|
13
|
+
"README.md"
|
|
12
14
|
],
|
|
13
15
|
"dependencies": {
|
|
14
|
-
"@skill-harness/cli": "0.1.
|
|
16
|
+
"@skill-harness/cli": "0.1.1"
|
|
15
17
|
},
|
|
16
18
|
"repository": {
|
|
17
19
|
"type": "git",
|