@cristobalme/skill-test 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +140 -0
- package/action/action.yml +87 -0
- package/dist/action/comment.cjs +178 -0
- package/dist/action/comment.cjs.map +1 -0
- package/dist/action/comment.js +151 -0
- package/dist/action/comment.js.map +1 -0
- package/dist/bin/skill-test.cjs +1313 -0
- package/dist/bin/skill-test.cjs.map +1 -0
- package/dist/bin/skill-test.js +1290 -0
- package/dist/bin/skill-test.js.map +1 -0
- package/dist/index.cjs +1217 -0
- package/dist/index.cjs.map +1 -0
- package/dist/index.d.cts +461 -0
- package/dist/index.d.ts +461 -0
- package/dist/index.js +1159 -0
- package/dist/index.js.map +1 -0
- package/examples/skill-test.yml +28 -0
- package/package.json +77 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Cristobal Medina Meza
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
ADDED
|
@@ -0,0 +1,140 @@
|
|
|
1
|
+
# skill-test
|
|
2
|
+
|
|
3
|
+
[](https://www.npmjs.com/package/@cristobalme/skill-test)
|
|
4
|
+
|
|
5
|
+
Test agent **Skills** (`SKILL.md` files) before you ship them. `skill-test`
|
|
6
|
+
validates skills across three layers:
|
|
7
|
+
|
|
8
|
+
1. **Static lint** — offline, deterministic checks against the live
|
|
9
|
+
[Agent Skills spec](https://agentskills.io/specification): frontmatter, naming
|
|
10
|
+
rules, description length, body size, broken file references, and risky
|
|
11
|
+
instruction patterns.
|
|
12
|
+
2. **Triggering** — does the agent actually load your skill for the prompts it
|
|
13
|
+
should (and skip the ones it shouldn't)? Measured as precision/recall over a
|
|
14
|
+
labeled corpus.
|
|
15
|
+
3. **Behavioral** — does the skill produce correct output on real tasks?
|
|
16
|
+
(graded, sandboxed)
|
|
17
|
+
|
|
18
|
+
No telemetry. No phone-home. The static layer needs no network and no API key.
|
|
19
|
+
|
|
20
|
+
## Quick start
|
|
21
|
+
|
|
22
|
+
```bash
|
|
23
|
+
npx @cristobalme/skill-test lint ./my-skill
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
The package is published under the `@cristobalme` scope; the binary it installs
|
|
27
|
+
is named `skill-test`.
|
|
28
|
+
|
|
29
|
+
## Commands
|
|
30
|
+
|
|
31
|
+
```
|
|
32
|
+
skill-test lint <path...> # Layer 1 — static, offline, deterministic
|
|
33
|
+
skill-test trigger <path...> # Layer 2 — activation precision/recall (needs API key + spec)
|
|
34
|
+
skill-test run <path...> # Layer 3 — behavioral task grading
|
|
35
|
+
skill-test check <path...> # Runs every layer available given config/keys
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
`<path>` accepts a single `SKILL.md`, a skill directory, or a directory of many
|
|
39
|
+
skills (walked recursively).
|
|
40
|
+
|
|
41
|
+
### Global flags
|
|
42
|
+
|
|
43
|
+
| Flag | Effect |
|
|
44
|
+
| ---------------- | --------------------------------------------------------------- |
|
|
45
|
+
| `--json` | Emit machine-readable JSON to stdout |
|
|
46
|
+
| `--junit <file>` | Write a JUnit XML report to `<file>` (renders in CI dashboards) |
|
|
47
|
+
| `--cheap` | Skip the behavioral (`run`) layer |
|
|
48
|
+
| `--quiet` | Only print failures |
|
|
49
|
+
| `--no-color` | Disable ANSI color (also auto-off when stdout isn't a TTY) |
|
|
50
|
+
| `--model <id>` | Override the classifier model for the trigger layer |
|
|
51
|
+
|
|
52
|
+
### Exit codes
|
|
53
|
+
|
|
54
|
+
| Code | Meaning |
|
|
55
|
+
| ---- | ---------------------------------------------------------------------- |
|
|
56
|
+
| `0` | All checks that ran passed |
|
|
57
|
+
| `1` | One or more failures (lint error, or trigger false pos/negatives) |
|
|
58
|
+
| `2` | Usage or configuration error (no skill found, `trigger` without a key) |
|
|
59
|
+
|
|
60
|
+
Layers degrade gracefully: `check` runs lint always, and runs the trigger layer
|
|
61
|
+
only when an `ANTHROPIC_API_KEY` and a `SKILL.test.yaml` are present. A missing
|
|
62
|
+
key or spec is _skipped_, not failed — so `check` is safe to drop into any CI.
|
|
63
|
+
|
|
64
|
+
## The `SKILL.test.yaml` spec
|
|
65
|
+
|
|
66
|
+
Co-locate a `SKILL.test.yaml` next to your `SKILL.md` to enable the trigger layer:
|
|
67
|
+
|
|
68
|
+
```yaml
|
|
69
|
+
skill: ./SKILL.md
|
|
70
|
+
triggering:
|
|
71
|
+
should_activate:
|
|
72
|
+
- "fill out this PDF form"
|
|
73
|
+
- "complete the application pdf"
|
|
74
|
+
should_not_activate:
|
|
75
|
+
- "write me a poem"
|
|
76
|
+
- "summarize this spreadsheet"
|
|
77
|
+
tasks: [] # behavioral tasks — Phase 5
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
The trigger layer asks the model to make the same load/skip decision a host
|
|
81
|
+
agent makes at startup, using **only** the skill's `name` + `description` (never
|
|
82
|
+
the body). It reports precision/recall/F1 over the labeled prompts. Results are
|
|
83
|
+
cached on disk (keyed by model + description + prompt), so reruns are free.
|
|
84
|
+
|
|
85
|
+
## GitHub Action
|
|
86
|
+
|
|
87
|
+
Test every skill on each PR and get a results comment. Drop this into
|
|
88
|
+
`.github/workflows/skill-test.yml` (full copy in
|
|
89
|
+
[`examples/skill-test.yml`](examples/skill-test.yml)):
|
|
90
|
+
|
|
91
|
+
```yaml
|
|
92
|
+
name: skill-test
|
|
93
|
+
on: [pull_request]
|
|
94
|
+
permissions:
|
|
95
|
+
contents: read
|
|
96
|
+
pull-requests: write
|
|
97
|
+
jobs:
|
|
98
|
+
skill-test:
|
|
99
|
+
runs-on: ubuntu-latest
|
|
100
|
+
steps:
|
|
101
|
+
- uses: actions/checkout@v4
|
|
102
|
+
with: { fetch-depth: 0 }
|
|
103
|
+
- uses: OWNER/skill-test/action@v1
|
|
104
|
+
with:
|
|
105
|
+
path: .
|
|
106
|
+
# optional — enables the trigger layer; lint runs without it
|
|
107
|
+
anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
Replace `OWNER` with the org/user the action is published under. Without an API
|
|
111
|
+
key the Action runs the offline lint layer and still posts a comment. With one,
|
|
112
|
+
it adds activation precision/recall. It posts (and updates in place) a single PR
|
|
113
|
+
comment:
|
|
114
|
+
|
|
115
|
+
| Skill | Lint | Trigger |
|
|
116
|
+
| -------------- | ----------- | ------------------ |
|
|
117
|
+
| `good-skill` | ✅ | ✅ P 100% · R 100% |
|
|
118
|
+
| `broken-skill` | ❌ 2 errors | ⏭️ skipped |
|
|
119
|
+
|
|
120
|
+
### Add the badge
|
|
121
|
+
|
|
122
|
+
```markdown
|
|
123
|
+
[](https://www.npmjs.com/package/@cristobalme/skill-test)
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
## Privacy
|
|
127
|
+
|
|
128
|
+
No telemetry, no analytics, no phone-home. The static `lint` layer runs fully
|
|
129
|
+
offline. Only `trigger` and `run` call the Anthropic API, and only with the
|
|
130
|
+
metadata/inputs needed for the check.
|
|
131
|
+
|
|
132
|
+
## Status
|
|
133
|
+
|
|
134
|
+
v0.1.0 ships the static lint layer, the activation trigger layer, the unified
|
|
135
|
+
`check` with JSON/JUnit output, and the GitHub Action + badge. The behavioral
|
|
136
|
+
`run` layer (sandboxed task grading) is the next release.
|
|
137
|
+
|
|
138
|
+
## License
|
|
139
|
+
|
|
140
|
+
MIT
|
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
name: "skill-test"
|
|
2
|
+
description: "Lint and test agent Skills (SKILL.md) and comment the results on pull requests."
|
|
3
|
+
branding:
|
|
4
|
+
icon: "check-circle"
|
|
5
|
+
color: "purple"
|
|
6
|
+
|
|
7
|
+
inputs:
|
|
8
|
+
path:
|
|
9
|
+
description: "A SKILL.md, a skill directory, or a directory of many skills."
|
|
10
|
+
required: false
|
|
11
|
+
default: "."
|
|
12
|
+
fail-on:
|
|
13
|
+
description: "Fail the check on 'error' (default) or also on 'warn'."
|
|
14
|
+
required: false
|
|
15
|
+
default: "error"
|
|
16
|
+
comment:
|
|
17
|
+
description: "Post/update a single PR comment with the results."
|
|
18
|
+
required: false
|
|
19
|
+
default: "true"
|
|
20
|
+
changed-only:
|
|
21
|
+
description: "On pull_request events, only check skills whose files changed."
|
|
22
|
+
required: false
|
|
23
|
+
default: "true"
|
|
24
|
+
version:
|
|
25
|
+
description: "skill-test version to run via npx (e.g. 'latest' or '0.1.0')."
|
|
26
|
+
required: false
|
|
27
|
+
default: "latest"
|
|
28
|
+
anthropic-api-key:
|
|
29
|
+
description: "Enables the trigger layer. Omit to run lint-only (still useful, no key needed)."
|
|
30
|
+
required: false
|
|
31
|
+
default: ""
|
|
32
|
+
|
|
33
|
+
runs:
|
|
34
|
+
using: "composite"
|
|
35
|
+
steps:
|
|
36
|
+
- name: Set up Node
|
|
37
|
+
uses: actions/setup-node@v4
|
|
38
|
+
with:
|
|
39
|
+
node-version: "20"
|
|
40
|
+
|
|
41
|
+
- name: Determine skills to check
|
|
42
|
+
id: scope
|
|
43
|
+
shell: bash
|
|
44
|
+
run: |
|
|
45
|
+
set -euo pipefail
|
|
46
|
+
targets='${{ inputs.path }}'
|
|
47
|
+
if [ "${{ inputs.changed-only }}" = "true" ] && [ "${{ github.event_name }}" = "pull_request" ]; then
|
|
48
|
+
base='${{ github.event.pull_request.base.sha }}'
|
|
49
|
+
head='${{ github.event.pull_request.head.sha }}'
|
|
50
|
+
# Containing directories of any SKILL.md that changed in this PR.
|
|
51
|
+
dirs="$(git diff --name-only "$base" "$head" 2>/dev/null | grep -E '(^|/)SKILL\.md$' | xargs -r -n1 dirname | sort -u | tr '\n' ' ' || true)"
|
|
52
|
+
if [ -n "${dirs// /}" ]; then
|
|
53
|
+
targets="$dirs"
|
|
54
|
+
else
|
|
55
|
+
echo "No changed SKILL.md files in this PR; nothing to check."
|
|
56
|
+
echo "skip=true" >> "$GITHUB_OUTPUT"
|
|
57
|
+
fi
|
|
58
|
+
fi
|
|
59
|
+
echo "targets=$targets" >> "$GITHUB_OUTPUT"
|
|
60
|
+
|
|
61
|
+
- name: Run skill-test
|
|
62
|
+
id: run
|
|
63
|
+
if: steps.scope.outputs.skip != 'true'
|
|
64
|
+
shell: bash
|
|
65
|
+
env:
|
|
66
|
+
ANTHROPIC_API_KEY: ${{ inputs.anthropic-api-key }}
|
|
67
|
+
run: |
|
|
68
|
+
set +e
|
|
69
|
+
npx --yes @cristobalme/skill-test@${{ inputs.version }} check ${{ steps.scope.outputs.targets }} \
|
|
70
|
+
--fail-on "${{ inputs.fail-on }}" --json --junit skill-test-junit.xml --no-color \
|
|
71
|
+
> skill-test-report.json
|
|
72
|
+
code=$?
|
|
73
|
+
echo "exit=$code" >> "$GITHUB_OUTPUT"
|
|
74
|
+
echo "----- skill-test report -----"
|
|
75
|
+
cat skill-test-report.json || true
|
|
76
|
+
|
|
77
|
+
- name: Comment on the pull request
|
|
78
|
+
if: steps.scope.outputs.skip != 'true' && inputs.comment == 'true' && github.event_name == 'pull_request'
|
|
79
|
+
shell: bash
|
|
80
|
+
env:
|
|
81
|
+
GITHUB_TOKEN: ${{ github.token }}
|
|
82
|
+
run: npx --yes --package @cristobalme/skill-test@${{ inputs.version }} skill-test-comment skill-test-report.json
|
|
83
|
+
|
|
84
|
+
- name: Enforce result
|
|
85
|
+
if: steps.scope.outputs.skip != 'true'
|
|
86
|
+
shell: bash
|
|
87
|
+
run: exit ${{ steps.run.outputs.exit }}
|
|
@@ -0,0 +1,178 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
"use strict";
|
|
3
|
+
var __defProp = Object.defineProperty;
|
|
4
|
+
var __getOwnPropDesc = Object.getOwnPropertyDescriptor;
|
|
5
|
+
var __getOwnPropNames = Object.getOwnPropertyNames;
|
|
6
|
+
var __hasOwnProp = Object.prototype.hasOwnProperty;
|
|
7
|
+
var __export = (target, all) => {
|
|
8
|
+
for (var name in all)
|
|
9
|
+
__defProp(target, name, { get: all[name], enumerable: true });
|
|
10
|
+
};
|
|
11
|
+
var __copyProps = (to, from, except, desc) => {
|
|
12
|
+
if (from && typeof from === "object" || typeof from === "function") {
|
|
13
|
+
for (let key of __getOwnPropNames(from))
|
|
14
|
+
if (!__hasOwnProp.call(to, key) && key !== except)
|
|
15
|
+
__defProp(to, key, { get: () => from[key], enumerable: !(desc = __getOwnPropDesc(from, key)) || desc.enumerable });
|
|
16
|
+
}
|
|
17
|
+
return to;
|
|
18
|
+
};
|
|
19
|
+
var __toCommonJS = (mod) => __copyProps(__defProp({}, "__esModule", { value: true }), mod);
|
|
20
|
+
|
|
21
|
+
// action/comment.ts
|
|
22
|
+
var comment_exports = {};
|
|
23
|
+
__export(comment_exports, {
|
|
24
|
+
COMMENT_MARKER: () => COMMENT_MARKER,
|
|
25
|
+
buildCommentBody: () => buildCommentBody,
|
|
26
|
+
chooseTarget: () => chooseTarget,
|
|
27
|
+
upsertComment: () => upsertComment
|
|
28
|
+
});
|
|
29
|
+
module.exports = __toCommonJS(comment_exports);
|
|
30
|
+
var import_node_fs = require("fs");
|
|
31
|
+
var import_node_path = require("path");
|
|
32
|
+
var COMMENT_MARKER = "<!-- skill-test-report -->";
|
|
33
|
+
function skillLabel(skillPath) {
|
|
34
|
+
return (0, import_node_path.basename)((0, import_node_path.dirname)(skillPath)) || skillPath;
|
|
35
|
+
}
|
|
36
|
+
function pct(n) {
|
|
37
|
+
return `${Math.round(n * 100)}%`;
|
|
38
|
+
}
|
|
39
|
+
function lintCell(skill) {
|
|
40
|
+
if (!skill.lint) return "\u2014";
|
|
41
|
+
const errors = skill.lint.findings.filter((f) => f.severity === "error").length;
|
|
42
|
+
const warns = skill.lint.findings.filter((f) => f.severity === "warn").length;
|
|
43
|
+
if (errors > 0) return `\u274C ${errors} error${errors === 1 ? "" : "s"}`;
|
|
44
|
+
if (warns > 0) return `\u26A0\uFE0F ${warns} warning${warns === 1 ? "" : "s"}`;
|
|
45
|
+
return "\u2705";
|
|
46
|
+
}
|
|
47
|
+
function triggerCell(skill) {
|
|
48
|
+
const t = skill.trigger;
|
|
49
|
+
if (!t) return "\u2014";
|
|
50
|
+
if ("skipped" in t) return `\u23ED\uFE0F skipped`;
|
|
51
|
+
const base = `P ${pct(t.score.precision)} \xB7 R ${pct(t.score.recall)}`;
|
|
52
|
+
return t.ok ? `\u2705 ${base}` : `\u274C ${base}`;
|
|
53
|
+
}
|
|
54
|
+
function buildCommentBody(report) {
|
|
55
|
+
const rows = report.skills.map(
|
|
56
|
+
(s) => `| \`${skillLabel(s.skill)}\` | ${lintCell(s)} | ${triggerCell(s)} |`
|
|
57
|
+
);
|
|
58
|
+
const failing = report.skills.filter((s) => !s.ok).length;
|
|
59
|
+
const header = report.ok ? "### \u{1F9EA} skill-test \u2014 all skills passing" : `### \u{1F9EA} skill-test \u2014 ${failing} skill${failing === 1 ? " needs" : "s need"} attention`;
|
|
60
|
+
const detail = [];
|
|
61
|
+
for (const s of report.skills) {
|
|
62
|
+
const errs = s.lint?.findings.filter((f) => f.severity === "error") ?? [];
|
|
63
|
+
const t = s.trigger;
|
|
64
|
+
const triggerMisses = t && !("skipped" in t) && !t.ok ? [
|
|
65
|
+
...t.falseNegatives.map((p) => `should activate but didn't: \`${p}\``),
|
|
66
|
+
...t.falsePositives.map((p) => `should NOT activate but did: \`${p}\``)
|
|
67
|
+
] : [];
|
|
68
|
+
if (errs.length === 0 && triggerMisses.length === 0) continue;
|
|
69
|
+
const items = [
|
|
70
|
+
...errs.map((f) => `${f.ruleId}: ${f.message}${f.line ? ` (line ${f.line})` : ""}`),
|
|
71
|
+
...triggerMisses
|
|
72
|
+
].map((line) => ` - ${line}`).join("\n");
|
|
73
|
+
detail.push(
|
|
74
|
+
`<details><summary><code>${skillLabel(s.skill)}</code></summary>
|
|
75
|
+
|
|
76
|
+
${items}
|
|
77
|
+
|
|
78
|
+
</details>`
|
|
79
|
+
);
|
|
80
|
+
}
|
|
81
|
+
return [
|
|
82
|
+
COMMENT_MARKER,
|
|
83
|
+
header,
|
|
84
|
+
"",
|
|
85
|
+
"| Skill | Lint | Trigger |",
|
|
86
|
+
"| --- | --- | --- |",
|
|
87
|
+
...rows,
|
|
88
|
+
"",
|
|
89
|
+
...detail,
|
|
90
|
+
detail.length ? "" : "",
|
|
91
|
+
`<sub>${report.skills.length} skill${report.skills.length === 1 ? "" : "s"} checked \xB7 generated by [skill-test](https://www.npmjs.com/package/@cristobalme/skill-test). No telemetry.</sub>`
|
|
92
|
+
].join("\n");
|
|
93
|
+
}
|
|
94
|
+
function chooseTarget(existing) {
|
|
95
|
+
const found = existing.find((c) => c.body.includes(COMMENT_MARKER));
|
|
96
|
+
return found ? { action: "update", id: found.id } : { action: "create" };
|
|
97
|
+
}
|
|
98
|
+
async function upsertComment(api, body) {
|
|
99
|
+
const target = chooseTarget(await api.list());
|
|
100
|
+
if (target.action === "update") await api.update(target.id, body);
|
|
101
|
+
else await api.create(body);
|
|
102
|
+
}
|
|
103
|
+
function githubApi(token, repo, issueNumber) {
|
|
104
|
+
const base = `https://api.github.com/repos/${repo}`;
|
|
105
|
+
const headers = {
|
|
106
|
+
authorization: `Bearer ${token}`,
|
|
107
|
+
accept: "application/vnd.github+json",
|
|
108
|
+
"content-type": "application/json",
|
|
109
|
+
"user-agent": "skill-test-action"
|
|
110
|
+
};
|
|
111
|
+
return {
|
|
112
|
+
async list() {
|
|
113
|
+
const res = await fetch(`${base}/issues/${issueNumber}/comments?per_page=100`, { headers });
|
|
114
|
+
if (!res.ok) throw new Error(`list comments failed: ${res.status}`);
|
|
115
|
+
return await res.json();
|
|
116
|
+
},
|
|
117
|
+
async create(body) {
|
|
118
|
+
const res = await fetch(`${base}/issues/${issueNumber}/comments`, {
|
|
119
|
+
method: "POST",
|
|
120
|
+
headers,
|
|
121
|
+
body: JSON.stringify({ body })
|
|
122
|
+
});
|
|
123
|
+
if (!res.ok) throw new Error(`create comment failed: ${res.status}`);
|
|
124
|
+
},
|
|
125
|
+
async update(id, body) {
|
|
126
|
+
const res = await fetch(`${base}/issues/comments/${id}`, {
|
|
127
|
+
method: "PATCH",
|
|
128
|
+
headers,
|
|
129
|
+
body: JSON.stringify({ body })
|
|
130
|
+
});
|
|
131
|
+
if (!res.ok) throw new Error(`update comment failed: ${res.status}`);
|
|
132
|
+
}
|
|
133
|
+
};
|
|
134
|
+
}
|
|
135
|
+
function prNumberFromEvent(eventPath) {
|
|
136
|
+
if (!eventPath) return void 0;
|
|
137
|
+
try {
|
|
138
|
+
const event = JSON.parse((0, import_node_fs.readFileSync)(eventPath, "utf8"));
|
|
139
|
+
return event.pull_request?.number ?? event.number;
|
|
140
|
+
} catch {
|
|
141
|
+
return void 0;
|
|
142
|
+
}
|
|
143
|
+
}
|
|
144
|
+
async function main() {
|
|
145
|
+
const reportPath = process.argv[2];
|
|
146
|
+
if (!reportPath) {
|
|
147
|
+
process.stderr.write("usage: comment.js <report.json>\n");
|
|
148
|
+
process.exit(2);
|
|
149
|
+
}
|
|
150
|
+
const report = JSON.parse((0, import_node_fs.readFileSync)(reportPath, "utf8"));
|
|
151
|
+
const body = buildCommentBody(report);
|
|
152
|
+
const token = process.env.GITHUB_TOKEN;
|
|
153
|
+
const repo = process.env.GITHUB_REPOSITORY;
|
|
154
|
+
const prNumber = prNumberFromEvent(process.env.GITHUB_EVENT_PATH);
|
|
155
|
+
if (!token || !repo || !prNumber) {
|
|
156
|
+
process.stdout.write(body + "\n");
|
|
157
|
+
return;
|
|
158
|
+
}
|
|
159
|
+
await upsertComment(githubApi(token, repo, prNumber), body);
|
|
160
|
+
process.stdout.write(`skill-test: posted PR comment to ${repo}#${prNumber}
|
|
161
|
+
`);
|
|
162
|
+
}
|
|
163
|
+
var invokedDirectly = process.argv[1] !== void 0 && /comment\.(js|cjs|mjs|ts)$/.test(process.argv[1]);
|
|
164
|
+
if (invokedDirectly) {
|
|
165
|
+
main().catch((e) => {
|
|
166
|
+
process.stderr.write(`skill-test comment: ${e.message}
|
|
167
|
+
`);
|
|
168
|
+
process.exit(1);
|
|
169
|
+
});
|
|
170
|
+
}
|
|
171
|
+
// Annotate the CommonJS export names for ESM import in node:
|
|
172
|
+
0 && (module.exports = {
|
|
173
|
+
COMMENT_MARKER,
|
|
174
|
+
buildCommentBody,
|
|
175
|
+
chooseTarget,
|
|
176
|
+
upsertComment
|
|
177
|
+
});
|
|
178
|
+
//# sourceMappingURL=comment.cjs.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"sources":["../../action/comment.ts"],"sourcesContent":["#!/usr/bin/env node\n/**\n * Builds and upserts the single PR comment posted by the skill-test GitHub\n * Action. The comment body is built purely (and unit-tested); posting goes\n * through an injectable client so the network path is testable too.\n *\n * Run as a script: `node comment.js <report.json>`, with GITHUB_TOKEN,\n * GITHUB_REPOSITORY, and GITHUB_EVENT_PATH in the environment.\n */\nimport { readFileSync } from \"node:fs\";\nimport { basename, dirname } from \"node:path\";\n\n/** Marker kept in the comment so we can find and update it instead of spamming. */\nexport const COMMENT_MARKER = \"<!-- skill-test-report -->\";\n\n// Mirror of the report JSON shape (report/json.ts reportToJson output).\ninterface JsonFinding {\n ruleId: string;\n severity: \"error\" | \"warn\" | \"info\";\n message: string;\n line?: number;\n}\ninterface JsonSkill {\n skill: string;\n ok: boolean;\n lint?: { ok: boolean; findings: JsonFinding[] };\n trigger?:\n | { skipped: true; reason: string }\n | {\n ok: boolean;\n score: { precision: number; recall: number; f1: number };\n falseNegatives: string[];\n falsePositives: string[];\n };\n}\nexport interface JsonReport {\n tool: string;\n ok: boolean;\n skills: JsonSkill[];\n}\n\nfunction skillLabel(skillPath: string): string {\n return basename(dirname(skillPath)) || skillPath;\n}\n\nfunction pct(n: number): string {\n return `${Math.round(n * 100)}%`;\n}\n\nfunction lintCell(skill: JsonSkill): string {\n if (!skill.lint) return \"—\";\n const errors = skill.lint.findings.filter((f) => f.severity === \"error\").length;\n const warns = skill.lint.findings.filter((f) => f.severity === \"warn\").length;\n if (errors > 0) return `❌ ${errors} error${errors === 1 ? \"\" : \"s\"}`;\n if (warns > 0) return `⚠️ ${warns} warning${warns === 1 ? \"\" : \"s\"}`;\n return \"✅\";\n}\n\nfunction triggerCell(skill: JsonSkill): string {\n const t = skill.trigger;\n if (!t) return \"—\";\n if (\"skipped\" in t) return `⏭️ skipped`;\n const base = `P ${pct(t.score.precision)} · R ${pct(t.score.recall)}`;\n return t.ok ? `✅ ${base}` : `❌ ${base}`;\n}\n\n/** Build the full markdown comment body (including the hidden marker). */\nexport function buildCommentBody(report: JsonReport): string {\n const rows = report.skills.map(\n (s) => `| \\`${skillLabel(s.skill)}\\` | ${lintCell(s)} | ${triggerCell(s)} |`,\n );\n const failing = report.skills.filter((s) => !s.ok).length;\n const header = report.ok\n ? \"### 🧪 skill-test — all skills passing\"\n : `### 🧪 skill-test — ${failing} skill${failing === 1 ? \" needs\" : \"s need\"} attention`;\n\n const detail: string[] = [];\n for (const s of report.skills) {\n const errs = s.lint?.findings.filter((f) => f.severity === \"error\") ?? [];\n const t = s.trigger;\n const triggerMisses =\n t && !(\"skipped\" in t) && !t.ok\n ? [\n ...t.falseNegatives.map((p) => `should activate but didn't: \\`${p}\\``),\n ...t.falsePositives.map((p) => `should NOT activate but did: \\`${p}\\``),\n ]\n : [];\n if (errs.length === 0 && triggerMisses.length === 0) continue;\n const items = [\n ...errs.map((f) => `${f.ruleId}: ${f.message}${f.line ? ` (line ${f.line})` : \"\"}`),\n ...triggerMisses,\n ]\n .map((line) => ` - ${line}`)\n .join(\"\\n\");\n detail.push(\n `<details><summary><code>${skillLabel(s.skill)}</code></summary>\\n\\n${items}\\n\\n</details>`,\n );\n }\n\n return [\n COMMENT_MARKER,\n header,\n \"\",\n \"| Skill | Lint | Trigger |\",\n \"| --- | --- | --- |\",\n ...rows,\n \"\",\n ...detail,\n detail.length ? \"\" : \"\",\n `<sub>${report.skills.length} skill${report.skills.length === 1 ? \"\" : \"s\"} checked · generated by [skill-test](https://www.npmjs.com/package/@cristobalme/skill-test). No telemetry.</sub>`,\n ].join(\"\\n\");\n}\n\n/** Minimal GitHub comments API used by the upsert (injectable for tests). */\nexport interface CommentsApi {\n list(): Promise<{ id: number; body: string }[]>;\n create(body: string): Promise<void>;\n update(id: number, body: string): Promise<void>;\n}\n\n/** Decide whether to update an existing skill-test comment or create a new one. */\nexport function chooseTarget(\n existing: { id: number; body: string }[],\n): { action: \"update\"; id: number } | { action: \"create\" } {\n const found = existing.find((c) => c.body.includes(COMMENT_MARKER));\n return found ? { action: \"update\", id: found.id } : { action: \"create\" };\n}\n\n/** Post or update the single skill-test comment. */\nexport async function upsertComment(api: CommentsApi, body: string): Promise<void> {\n const target = chooseTarget(await api.list());\n if (target.action === \"update\") await api.update(target.id, body);\n else await api.create(body);\n}\n\n/** A CommentsApi backed by the GitHub REST API via global fetch. */\nfunction githubApi(token: string, repo: string, issueNumber: number): CommentsApi {\n const base = `https://api.github.com/repos/${repo}`;\n const headers = {\n authorization: `Bearer ${token}`,\n accept: \"application/vnd.github+json\",\n \"content-type\": \"application/json\",\n \"user-agent\": \"skill-test-action\",\n };\n return {\n async list() {\n const res = await fetch(`${base}/issues/${issueNumber}/comments?per_page=100`, { headers });\n if (!res.ok) throw new Error(`list comments failed: ${res.status}`);\n return (await res.json()) as { id: number; body: string }[];\n },\n async create(body) {\n const res = await fetch(`${base}/issues/${issueNumber}/comments`, {\n method: \"POST\",\n headers,\n body: JSON.stringify({ body }),\n });\n if (!res.ok) throw new Error(`create comment failed: ${res.status}`);\n },\n async update(id, body) {\n const res = await fetch(`${base}/issues/comments/${id}`, {\n method: \"PATCH\",\n headers,\n body: JSON.stringify({ body }),\n });\n if (!res.ok) throw new Error(`update comment failed: ${res.status}`);\n },\n };\n}\n\n/** Resolve the PR/issue number from the GitHub Actions event payload. */\nfunction prNumberFromEvent(eventPath: string | undefined): number | undefined {\n if (!eventPath) return undefined;\n try {\n const event = JSON.parse(readFileSync(eventPath, \"utf8\")) as {\n pull_request?: { number?: number };\n number?: number;\n };\n return event.pull_request?.number ?? event.number;\n } catch {\n return undefined;\n }\n}\n\nasync function main(): Promise<void> {\n const reportPath = process.argv[2];\n if (!reportPath) {\n process.stderr.write(\"usage: comment.js <report.json>\\n\");\n process.exit(2);\n }\n const report = JSON.parse(readFileSync(reportPath, \"utf8\")) as JsonReport;\n const body = buildCommentBody(report);\n\n const token = process.env.GITHUB_TOKEN;\n const repo = process.env.GITHUB_REPOSITORY;\n const prNumber = prNumberFromEvent(process.env.GITHUB_EVENT_PATH);\n\n if (!token || !repo || !prNumber) {\n // Not in a PR context (or no token): print the body so it still shows in logs.\n process.stdout.write(body + \"\\n\");\n return;\n }\n await upsertComment(githubApi(token, repo, prNumber), body);\n process.stdout.write(`skill-test: posted PR comment to ${repo}#${prNumber}\\n`);\n}\n\n// Only run main when executed directly (not when imported by tests).\nconst invokedDirectly =\n process.argv[1] !== undefined && /comment\\.(js|cjs|mjs|ts)$/.test(process.argv[1]);\nif (invokedDirectly) {\n main().catch((e) => {\n process.stderr.write(`skill-test comment: ${(e as Error).message}\\n`);\n process.exit(1);\n });\n}\n"],"mappings":";;;;;;;;;;;;;;;;;;;;;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AASA,qBAA6B;AAC7B,uBAAkC;AAG3B,IAAM,iBAAiB;AA4B9B,SAAS,WAAW,WAA2B;AAC7C,aAAO,+BAAS,0BAAQ,SAAS,CAAC,KAAK;AACzC;AAEA,SAAS,IAAI,GAAmB;AAC9B,SAAO,GAAG,KAAK,MAAM,IAAI,GAAG,CAAC;AAC/B;AAEA,SAAS,SAAS,OAA0B;AAC1C,MAAI,CAAC,MAAM,KAAM,QAAO;AACxB,QAAM,SAAS,MAAM,KAAK,SAAS,OAAO,CAAC,MAAM,EAAE,aAAa,OAAO,EAAE;AACzE,QAAM,QAAQ,MAAM,KAAK,SAAS,OAAO,CAAC,MAAM,EAAE,aAAa,MAAM,EAAE;AACvE,MAAI,SAAS,EAAG,QAAO,UAAK,MAAM,SAAS,WAAW,IAAI,KAAK,GAAG;AAClE,MAAI,QAAQ,EAAG,QAAO,gBAAM,KAAK,WAAW,UAAU,IAAI,KAAK,GAAG;AAClE,SAAO;AACT;AAEA,SAAS,YAAY,OAA0B;AAC7C,QAAM,IAAI,MAAM;AAChB,MAAI,CAAC,EAAG,QAAO;AACf,MAAI,aAAa,EAAG,QAAO;AAC3B,QAAM,OAAO,KAAK,IAAI,EAAE,MAAM,SAAS,CAAC,WAAQ,IAAI,EAAE,MAAM,MAAM,CAAC;AACnE,SAAO,EAAE,KAAK,UAAK,IAAI,KAAK,UAAK,IAAI;AACvC;AAGO,SAAS,iBAAiB,QAA4B;AAC3D,QAAM,OAAO,OAAO,OAAO;AAAA,IACzB,CAAC,MAAM,OAAO,WAAW,EAAE,KAAK,CAAC,QAAQ,SAAS,CAAC,CAAC,MAAM,YAAY,CAAC,CAAC;AAAA,EAC1E;AACA,QAAM,UAAU,OAAO,OAAO,OAAO,CAAC,MAAM,CAAC,EAAE,EAAE,EAAE;AACnD,QAAM,SAAS,OAAO,KAClB,uDACA,mCAAuB,OAAO,SAAS,YAAY,IAAI,WAAW,QAAQ;AAE9E,QAAM,SAAmB,CAAC;AAC1B,aAAW,KAAK,OAAO,QAAQ;AAC7B,UAAM,OAAO,EAAE,MAAM,SAAS,OAAO,CAAC,MAAM,EAAE,aAAa,OAAO,KAAK,CAAC;AACxE,UAAM,IAAI,EAAE;AACZ,UAAM,gBACJ,KAAK,EAAE,aAAa,MAAM,CAAC,EAAE,KACzB;AAAA,MACE,GAAG,EAAE,eAAe,IAAI,CAAC,MAAM,iCAAiC,CAAC,IAAI;AAAA,MACrE,GAAG,EAAE,eAAe,IAAI,CAAC,MAAM,kCAAkC,CAAC,IAAI;AAAA,IACxE,IACA,CAAC;AACP,QAAI,KAAK,WAAW,KAAK,cAAc,WAAW,EAAG;AACrD,UAAM,QAAQ;AAAA,MACZ,GAAG,KAAK,IAAI,CAAC,MAAM,GAAG,EAAE,MAAM,KAAK,EAAE,OAAO,GAAG,EAAE,OAAO,UAAU,EAAE,IAAI,MAAM,EAAE,EAAE;AAAA,MAClF,GAAG;AAAA,IACL,EACG,IAAI,CAAC,SAAS,OAAO,IAAI,EAAE,EAC3B,KAAK,IAAI;AACZ,WAAO;AAAA,MACL,2BAA2B,WAAW,EAAE,KAAK,CAAC;AAAA;AAAA,EAAwB,KAAK;AAAA;AAAA;AAAA,IAC7E;AAAA,EACF;AAEA,SAAO;AAAA,IACL;AAAA,IACA;AAAA,IACA;AAAA,IACA;AAAA,IACA;AAAA,IACA,GAAG;AAAA,IACH;AAAA,IACA,GAAG;AAAA,IACH,OAAO,SAAS,KAAK;AAAA,IACrB,QAAQ,OAAO,OAAO,MAAM,SAAS,OAAO,OAAO,WAAW,IAAI,KAAK,GAAG;AAAA,EAC5E,EAAE,KAAK,IAAI;AACb;AAUO,SAAS,aACd,UACyD;AACzD,QAAM,QAAQ,SAAS,KAAK,CAAC,MAAM,EAAE,KAAK,SAAS,cAAc,CAAC;AAClE,SAAO,QAAQ,EAAE,QAAQ,UAAU,IAAI,MAAM,GAAG,IAAI,EAAE,QAAQ,SAAS;AACzE;AAGA,eAAsB,cAAc,KAAkB,MAA6B;AACjF,QAAM,SAAS,aAAa,MAAM,IAAI,KAAK,CAAC;AAC5C,MAAI,OAAO,WAAW,SAAU,OAAM,IAAI,OAAO,OAAO,IAAI,IAAI;AAAA,MAC3D,OAAM,IAAI,OAAO,IAAI;AAC5B;AAGA,SAAS,UAAU,OAAe,MAAc,aAAkC;AAChF,QAAM,OAAO,gCAAgC,IAAI;AACjD,QAAM,UAAU;AAAA,IACd,eAAe,UAAU,KAAK;AAAA,IAC9B,QAAQ;AAAA,IACR,gBAAgB;AAAA,IAChB,cAAc;AAAA,EAChB;AACA,SAAO;AAAA,IACL,MAAM,OAAO;AACX,YAAM,MAAM,MAAM,MAAM,GAAG,IAAI,WAAW,WAAW,0BAA0B,EAAE,QAAQ,CAAC;AAC1F,UAAI,CAAC,IAAI,GAAI,OAAM,IAAI,MAAM,yBAAyB,IAAI,MAAM,EAAE;AAClE,aAAQ,MAAM,IAAI,KAAK;AAAA,IACzB;AAAA,IACA,MAAM,OAAO,MAAM;AACjB,YAAM,MAAM,MAAM,MAAM,GAAG,IAAI,WAAW,WAAW,aAAa;AAAA,QAChE,QAAQ;AAAA,QACR;AAAA,QACA,MAAM,KAAK,UAAU,EAAE,KAAK,CAAC;AAAA,MAC/B,CAAC;AACD,UAAI,CAAC,IAAI,GAAI,OAAM,IAAI,MAAM,0BAA0B,IAAI,MAAM,EAAE;AAAA,IACrE;AAAA,IACA,MAAM,OAAO,IAAI,MAAM;AACrB,YAAM,MAAM,MAAM,MAAM,GAAG,IAAI,oBAAoB,EAAE,IAAI;AAAA,QACvD,QAAQ;AAAA,QACR;AAAA,QACA,MAAM,KAAK,UAAU,EAAE,KAAK,CAAC;AAAA,MAC/B,CAAC;AACD,UAAI,CAAC,IAAI,GAAI,OAAM,IAAI,MAAM,0BAA0B,IAAI,MAAM,EAAE;AAAA,IACrE;AAAA,EACF;AACF;AAGA,SAAS,kBAAkB,WAAmD;AAC5E,MAAI,CAAC,UAAW,QAAO;AACvB,MAAI;AACF,UAAM,QAAQ,KAAK,UAAM,6BAAa,WAAW,MAAM,CAAC;AAIxD,WAAO,MAAM,cAAc,UAAU,MAAM;AAAA,EAC7C,QAAQ;AACN,WAAO;AAAA,EACT;AACF;AAEA,eAAe,OAAsB;AACnC,QAAM,aAAa,QAAQ,KAAK,CAAC;AACjC,MAAI,CAAC,YAAY;AACf,YAAQ,OAAO,MAAM,mCAAmC;AACxD,YAAQ,KAAK,CAAC;AAAA,EAChB;AACA,QAAM,SAAS,KAAK,UAAM,6BAAa,YAAY,MAAM,CAAC;AAC1D,QAAM,OAAO,iBAAiB,MAAM;AAEpC,QAAM,QAAQ,QAAQ,IAAI;AAC1B,QAAM,OAAO,QAAQ,IAAI;AACzB,QAAM,WAAW,kBAAkB,QAAQ,IAAI,iBAAiB;AAEhE,MAAI,CAAC,SAAS,CAAC,QAAQ,CAAC,UAAU;AAEhC,YAAQ,OAAO,MAAM,OAAO,IAAI;AAChC;AAAA,EACF;AACA,QAAM,cAAc,UAAU,OAAO,MAAM,QAAQ,GAAG,IAAI;AAC1D,UAAQ,OAAO,MAAM,oCAAoC,IAAI,IAAI,QAAQ;AAAA,CAAI;AAC/E;AAGA,IAAM,kBACJ,QAAQ,KAAK,CAAC,MAAM,UAAa,4BAA4B,KAAK,QAAQ,KAAK,CAAC,CAAC;AACnF,IAAI,iBAAiB;AACnB,OAAK,EAAE,MAAM,CAAC,MAAM;AAClB,YAAQ,OAAO,MAAM,uBAAwB,EAAY,OAAO;AAAA,CAAI;AACpE,YAAQ,KAAK,CAAC;AAAA,EAChB,CAAC;AACH;","names":[]}
|
|
@@ -0,0 +1,151 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
|
|
3
|
+
// action/comment.ts
|
|
4
|
+
import { readFileSync } from "fs";
|
|
5
|
+
import { basename, dirname } from "path";
|
|
6
|
+
var COMMENT_MARKER = "<!-- skill-test-report -->";
|
|
7
|
+
function skillLabel(skillPath) {
|
|
8
|
+
return basename(dirname(skillPath)) || skillPath;
|
|
9
|
+
}
|
|
10
|
+
function pct(n) {
|
|
11
|
+
return `${Math.round(n * 100)}%`;
|
|
12
|
+
}
|
|
13
|
+
function lintCell(skill) {
|
|
14
|
+
if (!skill.lint) return "\u2014";
|
|
15
|
+
const errors = skill.lint.findings.filter((f) => f.severity === "error").length;
|
|
16
|
+
const warns = skill.lint.findings.filter((f) => f.severity === "warn").length;
|
|
17
|
+
if (errors > 0) return `\u274C ${errors} error${errors === 1 ? "" : "s"}`;
|
|
18
|
+
if (warns > 0) return `\u26A0\uFE0F ${warns} warning${warns === 1 ? "" : "s"}`;
|
|
19
|
+
return "\u2705";
|
|
20
|
+
}
|
|
21
|
+
function triggerCell(skill) {
|
|
22
|
+
const t = skill.trigger;
|
|
23
|
+
if (!t) return "\u2014";
|
|
24
|
+
if ("skipped" in t) return `\u23ED\uFE0F skipped`;
|
|
25
|
+
const base = `P ${pct(t.score.precision)} \xB7 R ${pct(t.score.recall)}`;
|
|
26
|
+
return t.ok ? `\u2705 ${base}` : `\u274C ${base}`;
|
|
27
|
+
}
|
|
28
|
+
function buildCommentBody(report) {
|
|
29
|
+
const rows = report.skills.map(
|
|
30
|
+
(s) => `| \`${skillLabel(s.skill)}\` | ${lintCell(s)} | ${triggerCell(s)} |`
|
|
31
|
+
);
|
|
32
|
+
const failing = report.skills.filter((s) => !s.ok).length;
|
|
33
|
+
const header = report.ok ? "### \u{1F9EA} skill-test \u2014 all skills passing" : `### \u{1F9EA} skill-test \u2014 ${failing} skill${failing === 1 ? " needs" : "s need"} attention`;
|
|
34
|
+
const detail = [];
|
|
35
|
+
for (const s of report.skills) {
|
|
36
|
+
const errs = s.lint?.findings.filter((f) => f.severity === "error") ?? [];
|
|
37
|
+
const t = s.trigger;
|
|
38
|
+
const triggerMisses = t && !("skipped" in t) && !t.ok ? [
|
|
39
|
+
...t.falseNegatives.map((p) => `should activate but didn't: \`${p}\``),
|
|
40
|
+
...t.falsePositives.map((p) => `should NOT activate but did: \`${p}\``)
|
|
41
|
+
] : [];
|
|
42
|
+
if (errs.length === 0 && triggerMisses.length === 0) continue;
|
|
43
|
+
const items = [
|
|
44
|
+
...errs.map((f) => `${f.ruleId}: ${f.message}${f.line ? ` (line ${f.line})` : ""}`),
|
|
45
|
+
...triggerMisses
|
|
46
|
+
].map((line) => ` - ${line}`).join("\n");
|
|
47
|
+
detail.push(
|
|
48
|
+
`<details><summary><code>${skillLabel(s.skill)}</code></summary>
|
|
49
|
+
|
|
50
|
+
${items}
|
|
51
|
+
|
|
52
|
+
</details>`
|
|
53
|
+
);
|
|
54
|
+
}
|
|
55
|
+
return [
|
|
56
|
+
COMMENT_MARKER,
|
|
57
|
+
header,
|
|
58
|
+
"",
|
|
59
|
+
"| Skill | Lint | Trigger |",
|
|
60
|
+
"| --- | --- | --- |",
|
|
61
|
+
...rows,
|
|
62
|
+
"",
|
|
63
|
+
...detail,
|
|
64
|
+
detail.length ? "" : "",
|
|
65
|
+
`<sub>${report.skills.length} skill${report.skills.length === 1 ? "" : "s"} checked \xB7 generated by [skill-test](https://www.npmjs.com/package/@cristobalme/skill-test). No telemetry.</sub>`
|
|
66
|
+
].join("\n");
|
|
67
|
+
}
|
|
68
|
+
function chooseTarget(existing) {
|
|
69
|
+
const found = existing.find((c) => c.body.includes(COMMENT_MARKER));
|
|
70
|
+
return found ? { action: "update", id: found.id } : { action: "create" };
|
|
71
|
+
}
|
|
72
|
+
async function upsertComment(api, body) {
|
|
73
|
+
const target = chooseTarget(await api.list());
|
|
74
|
+
if (target.action === "update") await api.update(target.id, body);
|
|
75
|
+
else await api.create(body);
|
|
76
|
+
}
|
|
77
|
+
function githubApi(token, repo, issueNumber) {
|
|
78
|
+
const base = `https://api.github.com/repos/${repo}`;
|
|
79
|
+
const headers = {
|
|
80
|
+
authorization: `Bearer ${token}`,
|
|
81
|
+
accept: "application/vnd.github+json",
|
|
82
|
+
"content-type": "application/json",
|
|
83
|
+
"user-agent": "skill-test-action"
|
|
84
|
+
};
|
|
85
|
+
return {
|
|
86
|
+
async list() {
|
|
87
|
+
const res = await fetch(`${base}/issues/${issueNumber}/comments?per_page=100`, { headers });
|
|
88
|
+
if (!res.ok) throw new Error(`list comments failed: ${res.status}`);
|
|
89
|
+
return await res.json();
|
|
90
|
+
},
|
|
91
|
+
async create(body) {
|
|
92
|
+
const res = await fetch(`${base}/issues/${issueNumber}/comments`, {
|
|
93
|
+
method: "POST",
|
|
94
|
+
headers,
|
|
95
|
+
body: JSON.stringify({ body })
|
|
96
|
+
});
|
|
97
|
+
if (!res.ok) throw new Error(`create comment failed: ${res.status}`);
|
|
98
|
+
},
|
|
99
|
+
async update(id, body) {
|
|
100
|
+
const res = await fetch(`${base}/issues/comments/${id}`, {
|
|
101
|
+
method: "PATCH",
|
|
102
|
+
headers,
|
|
103
|
+
body: JSON.stringify({ body })
|
|
104
|
+
});
|
|
105
|
+
if (!res.ok) throw new Error(`update comment failed: ${res.status}`);
|
|
106
|
+
}
|
|
107
|
+
};
|
|
108
|
+
}
|
|
109
|
+
function prNumberFromEvent(eventPath) {
|
|
110
|
+
if (!eventPath) return void 0;
|
|
111
|
+
try {
|
|
112
|
+
const event = JSON.parse(readFileSync(eventPath, "utf8"));
|
|
113
|
+
return event.pull_request?.number ?? event.number;
|
|
114
|
+
} catch {
|
|
115
|
+
return void 0;
|
|
116
|
+
}
|
|
117
|
+
}
|
|
118
|
+
async function main() {
|
|
119
|
+
const reportPath = process.argv[2];
|
|
120
|
+
if (!reportPath) {
|
|
121
|
+
process.stderr.write("usage: comment.js <report.json>\n");
|
|
122
|
+
process.exit(2);
|
|
123
|
+
}
|
|
124
|
+
const report = JSON.parse(readFileSync(reportPath, "utf8"));
|
|
125
|
+
const body = buildCommentBody(report);
|
|
126
|
+
const token = process.env.GITHUB_TOKEN;
|
|
127
|
+
const repo = process.env.GITHUB_REPOSITORY;
|
|
128
|
+
const prNumber = prNumberFromEvent(process.env.GITHUB_EVENT_PATH);
|
|
129
|
+
if (!token || !repo || !prNumber) {
|
|
130
|
+
process.stdout.write(body + "\n");
|
|
131
|
+
return;
|
|
132
|
+
}
|
|
133
|
+
await upsertComment(githubApi(token, repo, prNumber), body);
|
|
134
|
+
process.stdout.write(`skill-test: posted PR comment to ${repo}#${prNumber}
|
|
135
|
+
`);
|
|
136
|
+
}
|
|
137
|
+
var invokedDirectly = process.argv[1] !== void 0 && /comment\.(js|cjs|mjs|ts)$/.test(process.argv[1]);
|
|
138
|
+
if (invokedDirectly) {
|
|
139
|
+
main().catch((e) => {
|
|
140
|
+
process.stderr.write(`skill-test comment: ${e.message}
|
|
141
|
+
`);
|
|
142
|
+
process.exit(1);
|
|
143
|
+
});
|
|
144
|
+
}
|
|
145
|
+
export {
|
|
146
|
+
COMMENT_MARKER,
|
|
147
|
+
buildCommentBody,
|
|
148
|
+
chooseTarget,
|
|
149
|
+
upsertComment
|
|
150
|
+
};
|
|
151
|
+
//# sourceMappingURL=comment.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"sources":["../../action/comment.ts"],"sourcesContent":["#!/usr/bin/env node\n/**\n * Builds and upserts the single PR comment posted by the skill-test GitHub\n * Action. The comment body is built purely (and unit-tested); posting goes\n * through an injectable client so the network path is testable too.\n *\n * Run as a script: `node comment.js <report.json>`, with GITHUB_TOKEN,\n * GITHUB_REPOSITORY, and GITHUB_EVENT_PATH in the environment.\n */\nimport { readFileSync } from \"node:fs\";\nimport { basename, dirname } from \"node:path\";\n\n/** Marker kept in the comment so we can find and update it instead of spamming. */\nexport const COMMENT_MARKER = \"<!-- skill-test-report -->\";\n\n// Mirror of the report JSON shape (report/json.ts reportToJson output).\ninterface JsonFinding {\n ruleId: string;\n severity: \"error\" | \"warn\" | \"info\";\n message: string;\n line?: number;\n}\ninterface JsonSkill {\n skill: string;\n ok: boolean;\n lint?: { ok: boolean; findings: JsonFinding[] };\n trigger?:\n | { skipped: true; reason: string }\n | {\n ok: boolean;\n score: { precision: number; recall: number; f1: number };\n falseNegatives: string[];\n falsePositives: string[];\n };\n}\nexport interface JsonReport {\n tool: string;\n ok: boolean;\n skills: JsonSkill[];\n}\n\nfunction skillLabel(skillPath: string): string {\n return basename(dirname(skillPath)) || skillPath;\n}\n\nfunction pct(n: number): string {\n return `${Math.round(n * 100)}%`;\n}\n\nfunction lintCell(skill: JsonSkill): string {\n if (!skill.lint) return \"—\";\n const errors = skill.lint.findings.filter((f) => f.severity === \"error\").length;\n const warns = skill.lint.findings.filter((f) => f.severity === \"warn\").length;\n if (errors > 0) return `❌ ${errors} error${errors === 1 ? \"\" : \"s\"}`;\n if (warns > 0) return `⚠️ ${warns} warning${warns === 1 ? \"\" : \"s\"}`;\n return \"✅\";\n}\n\nfunction triggerCell(skill: JsonSkill): string {\n const t = skill.trigger;\n if (!t) return \"—\";\n if (\"skipped\" in t) return `⏭️ skipped`;\n const base = `P ${pct(t.score.precision)} · R ${pct(t.score.recall)}`;\n return t.ok ? `✅ ${base}` : `❌ ${base}`;\n}\n\n/** Build the full markdown comment body (including the hidden marker). */\nexport function buildCommentBody(report: JsonReport): string {\n const rows = report.skills.map(\n (s) => `| \\`${skillLabel(s.skill)}\\` | ${lintCell(s)} | ${triggerCell(s)} |`,\n );\n const failing = report.skills.filter((s) => !s.ok).length;\n const header = report.ok\n ? \"### 🧪 skill-test — all skills passing\"\n : `### 🧪 skill-test — ${failing} skill${failing === 1 ? \" needs\" : \"s need\"} attention`;\n\n const detail: string[] = [];\n for (const s of report.skills) {\n const errs = s.lint?.findings.filter((f) => f.severity === \"error\") ?? [];\n const t = s.trigger;\n const triggerMisses =\n t && !(\"skipped\" in t) && !t.ok\n ? [\n ...t.falseNegatives.map((p) => `should activate but didn't: \\`${p}\\``),\n ...t.falsePositives.map((p) => `should NOT activate but did: \\`${p}\\``),\n ]\n : [];\n if (errs.length === 0 && triggerMisses.length === 0) continue;\n const items = [\n ...errs.map((f) => `${f.ruleId}: ${f.message}${f.line ? ` (line ${f.line})` : \"\"}`),\n ...triggerMisses,\n ]\n .map((line) => ` - ${line}`)\n .join(\"\\n\");\n detail.push(\n `<details><summary><code>${skillLabel(s.skill)}</code></summary>\\n\\n${items}\\n\\n</details>`,\n );\n }\n\n return [\n COMMENT_MARKER,\n header,\n \"\",\n \"| Skill | Lint | Trigger |\",\n \"| --- | --- | --- |\",\n ...rows,\n \"\",\n ...detail,\n detail.length ? \"\" : \"\",\n `<sub>${report.skills.length} skill${report.skills.length === 1 ? \"\" : \"s\"} checked · generated by [skill-test](https://www.npmjs.com/package/@cristobalme/skill-test). No telemetry.</sub>`,\n ].join(\"\\n\");\n}\n\n/** Minimal GitHub comments API used by the upsert (injectable for tests). */\nexport interface CommentsApi {\n list(): Promise<{ id: number; body: string }[]>;\n create(body: string): Promise<void>;\n update(id: number, body: string): Promise<void>;\n}\n\n/** Decide whether to update an existing skill-test comment or create a new one. */\nexport function chooseTarget(\n existing: { id: number; body: string }[],\n): { action: \"update\"; id: number } | { action: \"create\" } {\n const found = existing.find((c) => c.body.includes(COMMENT_MARKER));\n return found ? { action: \"update\", id: found.id } : { action: \"create\" };\n}\n\n/** Post or update the single skill-test comment. */\nexport async function upsertComment(api: CommentsApi, body: string): Promise<void> {\n const target = chooseTarget(await api.list());\n if (target.action === \"update\") await api.update(target.id, body);\n else await api.create(body);\n}\n\n/** A CommentsApi backed by the GitHub REST API via global fetch. */\nfunction githubApi(token: string, repo: string, issueNumber: number): CommentsApi {\n const base = `https://api.github.com/repos/${repo}`;\n const headers = {\n authorization: `Bearer ${token}`,\n accept: \"application/vnd.github+json\",\n \"content-type\": \"application/json\",\n \"user-agent\": \"skill-test-action\",\n };\n return {\n async list() {\n const res = await fetch(`${base}/issues/${issueNumber}/comments?per_page=100`, { headers });\n if (!res.ok) throw new Error(`list comments failed: ${res.status}`);\n return (await res.json()) as { id: number; body: string }[];\n },\n async create(body) {\n const res = await fetch(`${base}/issues/${issueNumber}/comments`, {\n method: \"POST\",\n headers,\n body: JSON.stringify({ body }),\n });\n if (!res.ok) throw new Error(`create comment failed: ${res.status}`);\n },\n async update(id, body) {\n const res = await fetch(`${base}/issues/comments/${id}`, {\n method: \"PATCH\",\n headers,\n body: JSON.stringify({ body }),\n });\n if (!res.ok) throw new Error(`update comment failed: ${res.status}`);\n },\n };\n}\n\n/** Resolve the PR/issue number from the GitHub Actions event payload. */\nfunction prNumberFromEvent(eventPath: string | undefined): number | undefined {\n if (!eventPath) return undefined;\n try {\n const event = JSON.parse(readFileSync(eventPath, \"utf8\")) as {\n pull_request?: { number?: number };\n number?: number;\n };\n return event.pull_request?.number ?? event.number;\n } catch {\n return undefined;\n }\n}\n\nasync function main(): Promise<void> {\n const reportPath = process.argv[2];\n if (!reportPath) {\n process.stderr.write(\"usage: comment.js <report.json>\\n\");\n process.exit(2);\n }\n const report = JSON.parse(readFileSync(reportPath, \"utf8\")) as JsonReport;\n const body = buildCommentBody(report);\n\n const token = process.env.GITHUB_TOKEN;\n const repo = process.env.GITHUB_REPOSITORY;\n const prNumber = prNumberFromEvent(process.env.GITHUB_EVENT_PATH);\n\n if (!token || !repo || !prNumber) {\n // Not in a PR context (or no token): print the body so it still shows in logs.\n process.stdout.write(body + \"\\n\");\n return;\n }\n await upsertComment(githubApi(token, repo, prNumber), body);\n process.stdout.write(`skill-test: posted PR comment to ${repo}#${prNumber}\\n`);\n}\n\n// Only run main when executed directly (not when imported by tests).\nconst invokedDirectly =\n process.argv[1] !== undefined && /comment\\.(js|cjs|mjs|ts)$/.test(process.argv[1]);\nif (invokedDirectly) {\n main().catch((e) => {\n process.stderr.write(`skill-test comment: ${(e as Error).message}\\n`);\n process.exit(1);\n });\n}\n"],"mappings":";;;AASA,SAAS,oBAAoB;AAC7B,SAAS,UAAU,eAAe;AAG3B,IAAM,iBAAiB;AA4B9B,SAAS,WAAW,WAA2B;AAC7C,SAAO,SAAS,QAAQ,SAAS,CAAC,KAAK;AACzC;AAEA,SAAS,IAAI,GAAmB;AAC9B,SAAO,GAAG,KAAK,MAAM,IAAI,GAAG,CAAC;AAC/B;AAEA,SAAS,SAAS,OAA0B;AAC1C,MAAI,CAAC,MAAM,KAAM,QAAO;AACxB,QAAM,SAAS,MAAM,KAAK,SAAS,OAAO,CAAC,MAAM,EAAE,aAAa,OAAO,EAAE;AACzE,QAAM,QAAQ,MAAM,KAAK,SAAS,OAAO,CAAC,MAAM,EAAE,aAAa,MAAM,EAAE;AACvE,MAAI,SAAS,EAAG,QAAO,UAAK,MAAM,SAAS,WAAW,IAAI,KAAK,GAAG;AAClE,MAAI,QAAQ,EAAG,QAAO,gBAAM,KAAK,WAAW,UAAU,IAAI,KAAK,GAAG;AAClE,SAAO;AACT;AAEA,SAAS,YAAY,OAA0B;AAC7C,QAAM,IAAI,MAAM;AAChB,MAAI,CAAC,EAAG,QAAO;AACf,MAAI,aAAa,EAAG,QAAO;AAC3B,QAAM,OAAO,KAAK,IAAI,EAAE,MAAM,SAAS,CAAC,WAAQ,IAAI,EAAE,MAAM,MAAM,CAAC;AACnE,SAAO,EAAE,KAAK,UAAK,IAAI,KAAK,UAAK,IAAI;AACvC;AAGO,SAAS,iBAAiB,QAA4B;AAC3D,QAAM,OAAO,OAAO,OAAO;AAAA,IACzB,CAAC,MAAM,OAAO,WAAW,EAAE,KAAK,CAAC,QAAQ,SAAS,CAAC,CAAC,MAAM,YAAY,CAAC,CAAC;AAAA,EAC1E;AACA,QAAM,UAAU,OAAO,OAAO,OAAO,CAAC,MAAM,CAAC,EAAE,EAAE,EAAE;AACnD,QAAM,SAAS,OAAO,KAClB,uDACA,mCAAuB,OAAO,SAAS,YAAY,IAAI,WAAW,QAAQ;AAE9E,QAAM,SAAmB,CAAC;AAC1B,aAAW,KAAK,OAAO,QAAQ;AAC7B,UAAM,OAAO,EAAE,MAAM,SAAS,OAAO,CAAC,MAAM,EAAE,aAAa,OAAO,KAAK,CAAC;AACxE,UAAM,IAAI,EAAE;AACZ,UAAM,gBACJ,KAAK,EAAE,aAAa,MAAM,CAAC,EAAE,KACzB;AAAA,MACE,GAAG,EAAE,eAAe,IAAI,CAAC,MAAM,iCAAiC,CAAC,IAAI;AAAA,MACrE,GAAG,EAAE,eAAe,IAAI,CAAC,MAAM,kCAAkC,CAAC,IAAI;AAAA,IACxE,IACA,CAAC;AACP,QAAI,KAAK,WAAW,KAAK,cAAc,WAAW,EAAG;AACrD,UAAM,QAAQ;AAAA,MACZ,GAAG,KAAK,IAAI,CAAC,MAAM,GAAG,EAAE,MAAM,KAAK,EAAE,OAAO,GAAG,EAAE,OAAO,UAAU,EAAE,IAAI,MAAM,EAAE,EAAE;AAAA,MAClF,GAAG;AAAA,IACL,EACG,IAAI,CAAC,SAAS,OAAO,IAAI,EAAE,EAC3B,KAAK,IAAI;AACZ,WAAO;AAAA,MACL,2BAA2B,WAAW,EAAE,KAAK,CAAC;AAAA;AAAA,EAAwB,KAAK;AAAA;AAAA;AAAA,IAC7E;AAAA,EACF;AAEA,SAAO;AAAA,IACL;AAAA,IACA;AAAA,IACA;AAAA,IACA;AAAA,IACA;AAAA,IACA,GAAG;AAAA,IACH;AAAA,IACA,GAAG;AAAA,IACH,OAAO,SAAS,KAAK;AAAA,IACrB,QAAQ,OAAO,OAAO,MAAM,SAAS,OAAO,OAAO,WAAW,IAAI,KAAK,GAAG;AAAA,EAC5E,EAAE,KAAK,IAAI;AACb;AAUO,SAAS,aACd,UACyD;AACzD,QAAM,QAAQ,SAAS,KAAK,CAAC,MAAM,EAAE,KAAK,SAAS,cAAc,CAAC;AAClE,SAAO,QAAQ,EAAE,QAAQ,UAAU,IAAI,MAAM,GAAG,IAAI,EAAE,QAAQ,SAAS;AACzE;AAGA,eAAsB,cAAc,KAAkB,MAA6B;AACjF,QAAM,SAAS,aAAa,MAAM,IAAI,KAAK,CAAC;AAC5C,MAAI,OAAO,WAAW,SAAU,OAAM,IAAI,OAAO,OAAO,IAAI,IAAI;AAAA,MAC3D,OAAM,IAAI,OAAO,IAAI;AAC5B;AAGA,SAAS,UAAU,OAAe,MAAc,aAAkC;AAChF,QAAM,OAAO,gCAAgC,IAAI;AACjD,QAAM,UAAU;AAAA,IACd,eAAe,UAAU,KAAK;AAAA,IAC9B,QAAQ;AAAA,IACR,gBAAgB;AAAA,IAChB,cAAc;AAAA,EAChB;AACA,SAAO;AAAA,IACL,MAAM,OAAO;AACX,YAAM,MAAM,MAAM,MAAM,GAAG,IAAI,WAAW,WAAW,0BAA0B,EAAE,QAAQ,CAAC;AAC1F,UAAI,CAAC,IAAI,GAAI,OAAM,IAAI,MAAM,yBAAyB,IAAI,MAAM,EAAE;AAClE,aAAQ,MAAM,IAAI,KAAK;AAAA,IACzB;AAAA,IACA,MAAM,OAAO,MAAM;AACjB,YAAM,MAAM,MAAM,MAAM,GAAG,IAAI,WAAW,WAAW,aAAa;AAAA,QAChE,QAAQ;AAAA,QACR;AAAA,QACA,MAAM,KAAK,UAAU,EAAE,KAAK,CAAC;AAAA,MAC/B,CAAC;AACD,UAAI,CAAC,IAAI,GAAI,OAAM,IAAI,MAAM,0BAA0B,IAAI,MAAM,EAAE;AAAA,IACrE;AAAA,IACA,MAAM,OAAO,IAAI,MAAM;AACrB,YAAM,MAAM,MAAM,MAAM,GAAG,IAAI,oBAAoB,EAAE,IAAI;AAAA,QACvD,QAAQ;AAAA,QACR;AAAA,QACA,MAAM,KAAK,UAAU,EAAE,KAAK,CAAC;AAAA,MAC/B,CAAC;AACD,UAAI,CAAC,IAAI,GAAI,OAAM,IAAI,MAAM,0BAA0B,IAAI,MAAM,EAAE;AAAA,IACrE;AAAA,EACF;AACF;AAGA,SAAS,kBAAkB,WAAmD;AAC5E,MAAI,CAAC,UAAW,QAAO;AACvB,MAAI;AACF,UAAM,QAAQ,KAAK,MAAM,aAAa,WAAW,MAAM,CAAC;AAIxD,WAAO,MAAM,cAAc,UAAU,MAAM;AAAA,EAC7C,QAAQ;AACN,WAAO;AAAA,EACT;AACF;AAEA,eAAe,OAAsB;AACnC,QAAM,aAAa,QAAQ,KAAK,CAAC;AACjC,MAAI,CAAC,YAAY;AACf,YAAQ,OAAO,MAAM,mCAAmC;AACxD,YAAQ,KAAK,CAAC;AAAA,EAChB;AACA,QAAM,SAAS,KAAK,MAAM,aAAa,YAAY,MAAM,CAAC;AAC1D,QAAM,OAAO,iBAAiB,MAAM;AAEpC,QAAM,QAAQ,QAAQ,IAAI;AAC1B,QAAM,OAAO,QAAQ,IAAI;AACzB,QAAM,WAAW,kBAAkB,QAAQ,IAAI,iBAAiB;AAEhE,MAAI,CAAC,SAAS,CAAC,QAAQ,CAAC,UAAU;AAEhC,YAAQ,OAAO,MAAM,OAAO,IAAI;AAChC;AAAA,EACF;AACA,QAAM,cAAc,UAAU,OAAO,MAAM,QAAQ,GAAG,IAAI;AAC1D,UAAQ,OAAO,MAAM,oCAAoC,IAAI,IAAI,QAAQ;AAAA,CAAI;AAC/E;AAGA,IAAM,kBACJ,QAAQ,KAAK,CAAC,MAAM,UAAa,4BAA4B,KAAK,QAAQ,KAAK,CAAC,CAAC;AACnF,IAAI,iBAAiB;AACnB,OAAK,EAAE,MAAM,CAAC,MAAM;AAClB,YAAQ,OAAO,MAAM,uBAAwB,EAAY,OAAO;AAAA,CAAI;AACpE,YAAQ,KAAK,CAAC;AAAA,EAChB,CAAC;AACH;","names":[]}
|