@uinaf/skillcheck 0.5.3 → 0.5.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/cli.js +1 -0
- package/docs/releasing.md +7 -6
- package/docs/usage.md +5 -3
- package/package.json +1 -1
package/dist/cli.js
CHANGED
|
@@ -61,6 +61,7 @@ function stateDirs(root) {
|
|
|
61
61
|
function runOptions(flags) {
|
|
62
62
|
const harness = flags.get("--harness") ?? "claude";
|
|
63
63
|
if (harness !== "claude" && harness !== "codex" && harness !== "cursor") fail(`--harness must be claude, codex, or cursor, got ${harness}`);
|
|
64
|
+
if (flags.has("--max-turns") && harness !== "claude") fail("--max-turns is only supported with --harness claude");
|
|
64
65
|
const agent = flags.get("--agent");
|
|
65
66
|
const judgeModel = flags.get("--judge") ?? "claude-opus-5";
|
|
66
67
|
const judgeEffort = flags.get("--judge-effort");
|
package/docs/releasing.md
CHANGED
|
@@ -6,7 +6,7 @@ A push to `main` runs one workflow, `.github/workflows/release.yml`:
|
|
|
6
6
|
|
|
7
7
|
```text
|
|
8
8
|
verify ──┐
|
|
9
|
-
├──> release npm publish, OIDC + uinaf-
|
|
9
|
+
├──> release npm publish, OIDC + uinaf-ci (release environment)
|
|
10
10
|
scan ────┘
|
|
11
11
|
```
|
|
12
12
|
|
|
@@ -20,15 +20,15 @@ The file name `release.yml` is load-bearing. See below.
|
|
|
20
20
|
## npm
|
|
21
21
|
|
|
22
22
|
`@uinaf/skillcheck` publishes from `.github/workflows/release.yml` via npm
|
|
23
|
-
Trusted Publishing (OIDC) and the `uinaf-
|
|
23
|
+
Trusted Publishing (OIDC) and the `uinaf-ci` GitHub App. There is no npm
|
|
24
24
|
token in this repository, in its environments, or in the organization.
|
|
25
25
|
|
|
26
26
|
Required on the `release` GitHub Environment:
|
|
27
27
|
|
|
28
|
-
| Name
|
|
29
|
-
|
|
|
30
|
-
| `
|
|
31
|
-
| `
|
|
28
|
+
| Name | Kind | Purpose |
|
|
29
|
+
| -------------------------- | ------ | ------------------------------------------- |
|
|
30
|
+
| `UINAF_CI_APP_CLIENT_ID` | var | GitHub App client id for the releaser bot |
|
|
31
|
+
| `UINAF_CI_APP_PRIVATE_KEY` | secret | GitHub App private key for the releaser bot |
|
|
32
32
|
|
|
33
33
|
The trusted publisher on npmjs.com is registered by **file path**, so
|
|
34
34
|
`.github/workflows/release.yml` cannot be renamed or moved without editing that
|
|
@@ -40,6 +40,7 @@ Deleting the `release` environment deletes both rows above with it, and there is
|
|
|
40
40
|
no repo-level fallback: `create-github-app-token` then runs with empty inputs
|
|
41
41
|
and the job fails at that step. The private key cannot be read back from
|
|
42
42
|
GitHub; recreating it means generating a new one in the App settings.
|
|
43
|
+
The App private key lives on the `release` Environment; the shared release-npm job binds that Environment, so GitHub substitutes the Environment value for the secret named in `release.yml`.
|
|
43
44
|
|
|
44
45
|
## Version history
|
|
45
46
|
|
package/docs/usage.md
CHANGED
|
@@ -32,7 +32,8 @@ clean, 1 with findings.
|
|
|
32
32
|
|
|
33
33
|
```sh
|
|
34
34
|
skillcheck run skills/<skill>/evals/<scenario>
|
|
35
|
-
skillcheck run <scenario-dir> --agent MODEL --judge MODEL --harness codex
|
|
35
|
+
skillcheck run <scenario-dir> --agent MODEL --judge MODEL --harness codex
|
|
36
|
+
skillcheck run <scenario-dir> --harness claude --max-turns 80
|
|
36
37
|
```
|
|
37
38
|
|
|
38
39
|
Materializes the scenario into `<root>/.skillcheck/scratch/<name>/workdir`,
|
|
@@ -47,8 +48,9 @@ message, and writes no provenance sidecar. It is never reported as
|
|
|
47
48
|
`FAIL score=0.0000`; only a real judged verdict can fail a run.
|
|
48
49
|
|
|
49
50
|
Defaults: `--harness claude`, agent `claude-opus-5`, judge `claude-opus-5`,
|
|
50
|
-
|
|
51
|
-
|
|
51
|
+
and a Claude agent limit of 50 turns. `--max-turns` changes that limit only for
|
|
52
|
+
Claude; passing it with `codex` or `cursor` fails before the eval starts. On
|
|
53
|
+
those harnesses, omitting `--agent` leaves the model to that CLI's own default.
|
|
52
54
|
|
|
53
55
|
`--harness cursor` drives the scenario through the Cursor Agent CLI
|
|
54
56
|
(`cursor-agent` on PATH) with the skill installed under `.cursor/skills/`;
|