skilltest 0.4.0 → 0.6.0

package/CLAUDE.md

@@ -22,7 +22,7 @@ The CLI is published as `skilltest` and built for `npx skilltest` usage.
 - `src/core/check-runner.ts`: orchestrates lint + trigger + eval with threshold gates
 - `src/core/grader.ts`: structured grader prompt + JSON parse
 - `src/providers/`: LLM provider abstraction (`sendMessage`) and provider implementations
-- `src/reporters/`: terminal rendering and JSON output helper
+- `src/reporters/`: terminal, JSON, and HTML output helpers
 - `src/utils/`: filesystem and API key config helpers
 
 ## Build and Test Locally
@@ -73,6 +73,10 @@ ANTHROPIC_API_KEY=your-key node dist/index.js trigger test-fixtures/sample-skill
 - `check` wraps lint + trigger + eval and enforces minimum thresholds:
   - trigger F1
   - eval assertion pass rate
+- Trigger/eval work is concurrency-limited instead of fully unbounded:
+  - default concurrency is `5`
+  - `--concurrency 1` preserves the old sequential behavior
+  - trigger RNG-dependent fake-skill setup is precomputed before requests begin, preserving seed determinism
 - JSON mode is strict:
   - no spinners
   - no colored output
@@ -106,6 +110,4 @@ ANTHROPIC_API_KEY=your-key node dist/index.js trigger test-fixtures/sample-skill
 ## Future Work (Not Implemented Yet)
 
 - Config file support (`.skilltestrc`)
-- Parallel execution
-- HTML reporting
 - Plugin linter rules

package/README.md

@@ -67,6 +67,18 @@ Run full quality gate:
 skilltest check ./path/to/skill --provider anthropic --min-f1 0.8 --min-assert-pass-rate 0.9
 ```
 
+Write a self-contained HTML report:
+
+```bash
+skilltest check ./path/to/skill --html ./reports/check.html
+```
+
+Model-backed commands default to `--concurrency 5`. Use `--concurrency 1` to force
+the old sequential execution order. Seeded trigger runs stay deterministic regardless
+of concurrency.
+All four commands also support `--html <path>` for an offline HTML report, and
+`--json` can be used with `--html` in the same run.
+
 Example lint summary:
 
 ```text
@@ -75,6 +87,35 @@ target: ./test-fixtures/sample-skill
 summary: 29/29 checks passed, 0 warnings, 0 failures
 ```
 
+## Configuration
+
+`skilltest` resolves config in this order:
+
+1. `.skilltestrc` in the target skill root
+2. `.skilltestrc` in the current working directory
+3. the nearest `package.json` containing `skilltestrc`
+
+CLI flags override config values.
+
+Example `.skilltestrc`:
+
+```json
+{
+  "provider": "anthropic",
+  "model": "claude-sonnet-4-5-20250929",
+  "concurrency": 5,
+  "trigger": {
+    "numQueries": 20,
+    "threshold": 0.8,
+    "seed": 123
+  },
+  "eval": {
+    "numRuns": 5,
+    "threshold": 0.9
+  }
+}
+```
+
 ## Commands
 
 ### `skilltest lint <path-to-skill>`
@@ -115,6 +156,10 @@ What it checks:
   - warns on provider-specific conventions such as `allowed-tools`
   - emits a likely compatibility summary
 
+Flags:
+
+- `--html <path>` write a self-contained HTML report
+
 ### `skilltest trigger <path-to-skill>`
 
 Measures trigger behavior for your skill description with model simulation.
@@ -128,12 +173,21 @@ Flow:
    - realistic fake skills
 4. Computes TP, TN, FP, FN, precision, recall, F1.
 
+For reproducible fake-skill sampling, pass `--seed <number>`. When a seed is used,
+terminal and JSON output include it so the run can be repeated exactly. If you use
+`.skilltestrc`, `trigger.seed` sets the default and the CLI flag overrides it.
+The fake-skill setup is precomputed before requests begin, so the same seed produces
+the same trigger cases at any concurrency level.
+
 Flags:
 
 - `--model <model>` default: `claude-sonnet-4-5-20250929`
 - `--provider <anthropic|openai>` default: `anthropic`
 - `--queries <path>` use custom queries JSON
 - `--num-queries <n>` default: `20` (must be even)
+- `--seed <number>` RNG seed for reproducible fake-skill sampling
+- `--concurrency <n>` default: `5`
+- `--html <path>` write a self-contained HTML report
 - `--save-queries <path>` save generated query set
 - `--api-key <key>` explicit key override
 - `--verbose` show full model decision text
@@ -155,6 +209,8 @@ Flags:
 - `--model <model>` default: `claude-sonnet-4-5-20250929`
 - `--grader-model <model>` default: same as `--model`
 - `--provider <anthropic|openai>` default: `anthropic`
+- `--concurrency <n>` default: `5`
+- `--html <path>` write a self-contained HTML report
 - `--save-results <path>` write full JSON result
 - `--api-key <key>` explicit key override
 - `--verbose` show full model responses
@@ -168,7 +224,8 @@ Default behavior:
 1. Run lint.
 2. Stop before model calls if lint has failures.
 3. Run trigger and eval only when lint passes.
-4. Fail quality gate when either threshold is below target.
+4. When concurrency is greater than `1`, run trigger and eval in parallel.
+5. Fail quality gate when either threshold is below target.
 
 Flags:
 
@@ -178,7 +235,10 @@ Flags:
 - `--api-key <key>` explicit key override
 - `--queries <path>` custom trigger queries JSON
 - `--num-queries <n>` default: `20` (must be even)
+- `--seed <number>` RNG seed for reproducible trigger sampling
 - `--prompts <path>` custom eval prompts JSON
+- `--concurrency <n>` default: `5` (`1` keeps the old sequential `check` behavior)
+- `--html <path>` write a self-contained HTML report
 - `--min-f1 <n>` default: `0.8`
 - `--min-assert-pass-rate <n>` default: `0.9`
 - `--save-results <path>` save combined check result JSON
@@ -240,6 +300,21 @@ skilltest eval ./skill --json
 skilltest check ./skill --json
 ```
 
+HTML report examples:
+
+```bash
+skilltest lint ./skill --html ./reports/lint.html
+skilltest trigger ./skill --html ./reports/trigger.html
+skilltest eval ./skill --html ./reports/eval.html
+skilltest check ./skill --json --html ./reports/check.html
+```
+
+Seeded trigger example:
+
+```bash
+skilltest trigger ./skill --seed 123
+```
+
 ## API Keys
 
 Anthropic:
@@ -343,7 +418,9 @@ Smoke tests:
 
 ```bash
 node dist/index.js lint test-fixtures/sample-skill/
+node dist/index.js lint test-fixtures/sample-skill/ --html lint-report.html
 node dist/index.js trigger test-fixtures/sample-skill/ --num-queries 2
+node dist/index.js trigger test-fixtures/sample-skill/ --queries path/to/queries.json --seed 123
 node dist/index.js eval test-fixtures/sample-skill/ --prompts test-fixtures/eval-prompts.json
 node dist/index.js check test-fixtures/sample-skill/ --num-queries 2 --prompts test-fixtures/eval-prompts.json
 ```