omakit 0.4.1 → 0.4.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +29 -95
- package/package.json +1 -1
- package/skills/omarchy-plugin-check/SKILL.md +2 -2
- package/skills/omarchy-plugin-submit/SKILL.md +2 -2
- package/skills/omarchy-plugin-validation-watch/SKILL.md +4 -4
- package/tools/marketplace/README.md +5 -1
- package/tools/marketplace/cli.mjs +36 -13
- package/tools/marketplace/github.mjs +13 -2
- package/tools/marketplace/measure-review-cost.mjs +39 -0
- package/tools/marketplace/measure-staleness.mjs +76 -0
- package/tools/marketplace/report.mjs +8 -1
- package/tools/marketplace/review-cost.mjs +107 -0
- package/tools/marketplace/submit.mjs +18 -2
- package/tools/marketplace/watch.mjs +64 -17
- package/tools/weigh/audit.mjs +21 -3
package/README.md
CHANGED
|
@@ -1,126 +1,60 @@
|
|
|
1
1
|
<p align="center">
|
|
2
|
-
<img src="docs/media/banner.gif" alt="omakit" width="440">
|
|
2
|
+
<img src="https://raw.githubusercontent.com/mtolhuys/omakit/main/docs/media/banner.gif" alt="omakit" width="440">
|
|
3
3
|
</p>
|
|
4
4
|
|
|
5
|
-
The
|
|
5
|
+
The marketplace validates one exact commit of your plugin. Push a fix or comment "fixed", and nothing re-runs ([M6](docs/MEASUREMENTS.md#m6-the-validated-commit-falls-behind-silently-and-that-is-the-centre-of-this-tool)). omakit runs the marketplace's own checks locally, watches your submission and posts nothing.
|
|
6
6
|
|
|
7
7
|
[](https://github.com/tcballard/omarchy-badges) [](https://www.npmjs.com/package/omakit) [](https://github.com/mtolhuys/omakit/actions/workflows/ci.yml) [](https://socket.dev/npm/package/omakit)
|
|
8
8
|
|
|
9
|
-
`omakit` is a zero-dependency Node CLI that checks an Omarchy Quattro plugin submission on your machine.
|
|
10
|
-
It is for a coding agent or a person submitting a plugin.
|
|
11
|
-
It never posts to the marketplace or writes into a plugin tree.
|
|
12
|
-
|
|
13
9
|
## Install
|
|
14
10
|
|
|
15
11
|
```bash
|
|
16
|
-
npm
|
|
17
|
-
omakit
|
|
18
|
-
```
|
|
19
|
-
|
|
20
|
-
See [docs/INSTALL.md](docs/INSTALL.md) for the clone route, PATH, requirements and upgrading.
|
|
21
|
-
|
|
22
|
-
## Commands
|
|
23
|
-
|
|
24
|
-
| Command | What it does |
|
|
25
|
-
| --- | --- |
|
|
26
|
-
| [`omakit setup`](docs/COMMANDS.md) | The environment, the pin, tab completion, and what to try first. |
|
|
27
|
-
| [`omakit submit <plugin-repo>`](docs/SUBMIT.md) | Every check, the issue title and body; asks for a category and tags at a terminal. |
|
|
28
|
-
| [`omakit watch [<issue-url>]`](docs/VALIDATION_WATCH.md) | Pick your marketplace issues, list them, or check all with `--all`. |
|
|
29
|
-
| [`omakit verify <plugin-repo>`](docs/COMMANDS.md) | The official security baseline over the local transport; `--json` for the document. |
|
|
30
|
-
| [`omakit parity`](docs/COMMANDS.md) | The baseline over GitHub versus the local transport, on real listings; writes the evidence. |
|
|
31
|
-
| [`omakit audit [<plugin>]`](docs/AUDIT.md) | Installed third-party commits against the exact commits the marketplace validated. |
|
|
32
|
-
| [`omakit weigh <plugin>`](docs/WEIGH.md) | What a plugin weighs on the shell, measured by restarting it without and with the plugin; asks first. |
|
|
33
|
-
| [`omakit doctor`](docs/COMMANDS.md) | What is installed, what is pinned, and what has moved. |
|
|
34
|
-
| [`omakit pin`](docs/COMMANDS.md) | What setup does for the pin, on its own. |
|
|
35
|
-
| [`omakit upgrade`](docs/COMMANDS.md) | Updates omakit through its own installer: npm, or a fast-forward. |
|
|
36
|
-
| [`omakit help --agent`](docs/COMMANDS.md) | The operating instructions, for the agent running this. |
|
|
37
|
-
|
|
38
|
-
Normal terminal use also checks for a newer npm release at most once daily
|
|
39
|
-
and shows the upgrade command. It installs nothing automatically; scripts
|
|
40
|
-
and JSON stay quiet. `DISABLE_UPDATE_NOTIFIER=1` disables the notice, and
|
|
41
|
-
`omakit doctor` checks explicitly. [Update behaviour](docs/INSTALL.md#updating).
|
|
42
|
-
|
|
43
|
-
Terminal help and reports adapt to the available width, up to 120 columns
|
|
44
|
-
for readability. Narrow windows wrap sooner; pipes and text files keep the
|
|
45
|
-
stable eighty-column layout. JSON and exact issue bodies remain intact.
|
|
46
|
-
|
|
47
|
-
### `submit`
|
|
48
|
-
|
|
49
|
-
```bash
|
|
50
|
-
omakit submit <plugin-repo> --category Widgets --tags bar,quickshell
|
|
12
|
+
npm i -g omakit && omakit setup
|
|
13
|
+
npx skills add mtolhuys/omakit
|
|
51
14
|
```
|
|
52
15
|
|
|
53
|
-
|
|
16
|
+
Requirements: [Node >=22](package.json); Omarchy Quattro.
|
|
54
17
|
|
|
55
|
-
|
|
18
|
+
Licence: [MIT](LICENSE).
|
|
56
19
|
|
|
57
|
-
|
|
20
|
+
## `omakit submit <plugin-repo>`
|
|
58
21
|
|
|
59
|
-
|
|
22
|
+

|
|
60
23
|
|
|
61
|
-
|
|
62
|
-
omakit watch <submission-issue-url>
|
|
63
|
-
omakit watch --all
|
|
64
|
-
omakit watch --list
|
|
65
|
-
omakit watch # choose one or several issues at a terminal
|
|
66
|
-
```
|
|
24
|
+
Checks the exact commit with the marketplace's own baseline and, when ready, prints the exact issue title and body for you to paste. The GIF shows a refusal with three blocking checks and their fixes.
|
|
67
25
|
|
|
68
|
-
|
|
26
|
+
## `omakit watch --all`
|
|
69
27
|
|
|
70
|
-
|
|
28
|
+

|
|
71
29
|
|
|
72
|
-
|
|
30
|
+
Checks your submission commits and names the action that re-runs stale validation: edit the issue body. The GIF shows five CURRENT issues in the counts and the first two issues with a discussion; CURRENT means matching commits, not approval.
|
|
73
31
|
|
|
74
|
-
|
|
32
|
+
## `omakit audit`
|
|
75
33
|
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
```bash
|
|
79
|
-
omakit weigh <plugin-id-or-dir>
|
|
80
|
-
```
|
|
34
|
+

|
|
81
35
|
|
|
82
|
-
|
|
36
|
+
Compares your installed plugin commits with the marketplace's validated commits. The GIF shows drift rows first: on the author's desktop, 9 of 18 audited plugins ran commits the marketplace never validated.
|
|
83
37
|
|
|
84
|
-
|
|
85
|
-
Weighs no CPU above the floor (0.13%) and runs 2 child processes using 8.2 MB and 0.1% CPU, on Omarchy 4.0.0.alpha, measured with omakit weigh on 2026-09-14
|
|
86
|
-
```
|
|
38
|
+
## `omakit weigh <plugin>`
|
|
87
39
|
|
|
88
|
-
|
|
40
|
+

|
|
89
41
|
|
|
90
|
-
|
|
42
|
+
Measures the shell with and without your plugin, reading Pss and CPU. The GIF shows three completed runs on the author's desktop, with baseline and plugin samples and a 0.33% CPU floor ([method](docs/WEIGH.md)).
|
|
91
43
|
|
|
92
44
|
## Evidence, not claims
|
|
93
45
|
|
|
94
|
-
|
|
|
46
|
+
| Measurement | Evidence |
|
|
95
47
|
| --- | --- |
|
|
96
|
-
|
|
|
97
|
-
|
|
|
98
|
-
|
|
|
99
|
-
|
|
|
100
|
-
|
|
|
101
|
-
|
|
|
102
|
-
| The GIFs above are real output | captures and renderer in [docs/media/](docs/media/) |
|
|
103
|
-
|
|
104
|
-
Committed evidence records a digest of each side rather than the results themselves, because findings about a specific third-party plugin are not this project's to publish.
|
|
48
|
+
| Baseline parity | 30/30 identical results, [recorded corpus](docs/evidence/parity/2026-09-12-local-vs-github-2.json), 2026-09-12 |
|
|
49
|
+
| Stale validated commit | 326/519 readable comparisons stale (62.8%); 64/583 unknown, [2026-09-15 data](docs/evidence/staleness/2026-09-15.json) |
|
|
50
|
+
| Registry churn | 4,201/4,293 registry-only commits in 30 days, 2026-09-13, [M7](docs/MEASUREMENTS.md#m7-the-registry-moves-by-the-hour-the-code-and-the-rules-move-by-the-week) |
|
|
51
|
+
| GIFs are recorded output | 5 GIFs with [captures and scenes](docs/media/README.md) |
|
|
52
|
+
| Posts nothing | 0 marketplace writes, [M10](docs/MEASUREMENTS.md#m10-readme-evidence-and-command-captures) |
|
|
53
|
+
| Zero dependencies | 0 runtime and 0 development dependencies, counted in [package.json](package.json) |
|
|
105
54
|
|
|
106
55
|
## Documentation
|
|
107
56
|
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
| [docs/COMMANDS.md](docs/COMMANDS.md) | command details, authentication and network behaviour |
|
|
113
|
-
| [docs/AUDIT.md](docs/AUDIT.md) | installed plugin drift against marketplace-validated commits, with JSON origins |
|
|
114
|
-
| [docs/SUBMIT.md](docs/SUBMIT.md) | every check and what it decides |
|
|
115
|
-
| [docs/WEIGH.md](docs/WEIGH.md) | what `weigh` measures, the noise floor, the `shell.json` mutation and its restore, and the JSON contract |
|
|
116
|
-
| [docs/VALIDATION_WATCH.md](docs/VALIDATION_WATCH.md) | the validation watch: what the marketplace validated, and what moves it |
|
|
117
|
-
| [docs/MEASUREMENTS.md](docs/MEASUREMENTS.md) | every number, its method and its limits |
|
|
118
|
-
| [docs/UPSTREAM_CONTRACT.md](docs/UPSTREAM_CONTRACT.md) | the seam, the pin, the boundaries |
|
|
119
|
-
| [docs/MARKETPLACE.md](docs/MARKETPLACE.md) | who this actually helps |
|
|
120
|
-
| [docs/PALETTE.md](docs/PALETTE.md) | every installed Omarchy theme measured, and which palette index each role gets |
|
|
121
|
-
| [docs/TUI.md](docs/TUI.md) | what the terminal shows, and why it looks that way |
|
|
122
|
-
| [AGENTS.md](AGENTS.md) | changing this repository |
|
|
123
|
-
|
|
124
|
-
MIT. Derived work built on public data from
|
|
125
|
-
`omacom/omarchy-plugin-marketplace`; not affiliated with or endorsed by that
|
|
126
|
-
project.
|
|
57
|
+
- Using: [install](docs/INSTALL.md), [commands](docs/COMMANDS.md), [audience](docs/MARKETPLACE.md).
|
|
58
|
+
- Checks and measurements: [submit](docs/SUBMIT.md), [watch](docs/VALIDATION_WATCH.md), [audit](docs/AUDIT.md), [evidence](docs/MEASUREMENTS.md).
|
|
59
|
+
- Method docs: [how](docs/HOW.md), [weigh](docs/WEIGH.md), [upstream contract](docs/UPSTREAM_CONTRACT.md), [palette](docs/PALETTE.md), [terminal](docs/TUI.md).
|
|
60
|
+
- Contributing: [repository rules](AGENTS.md), [releasing](docs/RELEASING.md), [media](docs/media/README.md).
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "omakit",
|
|
3
|
-
"version": "0.4.
|
|
3
|
+
"version": "0.4.2",
|
|
4
4
|
"description": "The safe place to find out: everything knowable about an Omarchy Quattro plugin submission before you post it, on your own machine. Agent-first, read-only against the marketplace, posts nothing, zero dependencies.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"author": "Maarten Tolhuijs",
|
|
@@ -92,8 +92,8 @@ own words, and so should you.
|
|
|
92
92
|
|
|
93
93
|
The marketplace validates the pushed default-branch HEAD, not whatever is
|
|
94
94
|
checked out locally. Run the check on the commit that will be pushed, then
|
|
95
|
-
commit and push before the real submission.
|
|
96
|
-
|
|
95
|
+
commit and push before the real submission. On 2026-09-15, 326/519 readable author-fixes
|
|
96
|
+
comparisons were stale, with 64 of 583 issues unknown; that is the
|
|
97
97
|
round this loop is meant to prevent.
|
|
98
98
|
|
|
99
99
|
## When the plugin is ready
|
|
@@ -133,8 +133,8 @@ strip files from their tree on your own.
|
|
|
133
133
|
|
|
134
134
|
**`submission.validation-commit`.** The marketplace validates the commit it
|
|
135
135
|
resolves when the issue is opened or edited, which is the pushed default-branch
|
|
136
|
-
HEAD, not whatever is checked out locally. Push first, then submit.
|
|
137
|
-
|
|
136
|
+
HEAD, not whatever is checked out locally. Push first, then submit. On 2026-09-15, 326/519 readable author-fixes comparisons were stale
|
|
137
|
+
(62.8%), with 64 of 583 issues unknown.
|
|
138
138
|
|
|
139
139
|
## What the baseline result means
|
|
140
140
|
|
|
@@ -18,10 +18,10 @@ the issue body**.
|
|
|
18
18
|
submissions.
|
|
19
19
|
|
|
20
20
|
So pushing a fix does nothing, and commenting "fixed in `abc123`" does nothing.
|
|
21
|
-
Both feel like progress. Neither is.
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
21
|
+
Both feel like progress. Neither is. The full author-fixes queue was measured on 2026-09-15: 326/519 readable
|
|
22
|
+
comparisons were stale (62.8%), with 64 of 583 issues unknown. The older
|
|
23
|
+
2026-09-12 sample found 68/93 readable issues stale (73.1%); that rate was
|
|
24
|
+
not a measurement of all 464 issues in the queue.
|
|
25
25
|
|
|
26
26
|
## Check it
|
|
27
27
|
|
|
@@ -18,6 +18,8 @@ local commit through the transport seam the marketplace tests itself
|
|
|
18
18
|
| `tree.mjs` | The installable tree of a subject at one exact commit, from the Git object database. |
|
|
19
19
|
| `plugin.mjs` | The root files the submission contract needs, and the declared plugin identity. |
|
|
20
20
|
| `agent-control.mjs` | The recursive agent-control warning, and its remedy. |
|
|
21
|
+
| `review-cost.mjs` | The advisory review-cost verdict, shared account discovery, and path classification between dated validated snapshots. M4 and M9 carry its evidence. |
|
|
22
|
+
| `measure-review-cost.mjs` | Reproduces M9 across the open update population at live marketplace HEAD, with compare sources and explicit skipped reasons in JSON. |
|
|
21
23
|
| `issue.mjs` | Renders the issue the way the form would, then has the marketplace's own parser judge it. |
|
|
22
24
|
| `submit.mjs` | Assembles every check with its measured reason, and withholds the body when a blocking check fails. Three outcomes: `ready` (the body), `refused` (a blocking check failed) and `listed` (the plugin is already listed by its own repository: `identity.available` passes with the listing's record, the five body checks are omitted rather than drawn as waiting, no body exists on purpose, and `listing` carries the listed commit against the local one and the form to use for a newer commit). Decides the category and tags after the registry: a listed plugin, own or taken, is asked for neither; an unlisted one without them is asked through `ask.mjs` at a terminal, and is a usage error otherwise. Ends with `reproduce`, the command line that repeats the run without asking. Under `--offline` the validation-commit check is `skipped`, not passed: verdict `skipped`, listed under `skipped` and not `unknown`, never blocking, and the READY line says "1 check skipped (--offline)". |
|
|
23
25
|
| `ask.mjs` | The two questions `submit` asks a person at a terminal, and only there: category and tags, numbered from the pinned form, with the marketplace's own presentation for the manifest's kinds (read from the pinned catalog builder) as the default where it is on the list. Prompts on stderr, nothing persisted. |
|
|
@@ -98,7 +100,9 @@ sequences and nothing else.
|
|
|
98
100
|
built-in `fetch`, which does not read proxy environment variables by default.
|
|
99
101
|
Behind a proxy, run them with `NODE_USE_ENV_PROXY=1`. `submit` reads two things
|
|
100
102
|
online, the subject's default-branch HEAD and the marketplace's current
|
|
101
|
-
registry
|
|
103
|
+
registry. A manual-review baseline also reads the account's open issue
|
|
104
|
+
discovery and issue bodies for batching advice; `--offline` turns these
|
|
105
|
+
reads off. `verify` needs no network at all
|
|
102
106
|
beyond fetching a reviewer-mode subject, and `tests/parity/offline.mjs` proves
|
|
103
107
|
it.
|
|
104
108
|
|
|
@@ -68,6 +68,16 @@ const REMEDY = Object.freeze({
|
|
|
68
68
|
"interrupted": "shell.json was restored; run it again when the desktop is yours to restart.",
|
|
69
69
|
})
|
|
70
70
|
|
|
71
|
+
/*
|
|
72
|
+
* A command that has written its result leaves through `process.exitCode`,
|
|
73
|
+
* never `process.exit()`: stdout is an API, and on a pipe whose reader has
|
|
74
|
+
* not started reading yet the exit cuts the output. Measured on 0.4.1:
|
|
75
|
+
* `omakit submit <listed plugin> --json | (sleep 2; cat)` delivered 8,192 of
|
|
76
|
+
* 14,033 bytes, and a parser downstream saw invalid JSON. The failure
|
|
77
|
+
* states below still exit at once: they write one short block to stderr,
|
|
78
|
+
* and their callers use them the way a throw is used.
|
|
79
|
+
*/
|
|
80
|
+
|
|
71
81
|
/**
|
|
72
82
|
* Every failure, in one register, on stderr. `usage` errors carry the
|
|
73
83
|
* signature that was expected, so the remedy is the reference and not a
|
|
@@ -90,9 +100,21 @@ function failFrom(error) {
|
|
|
90
100
|
throw error
|
|
91
101
|
}
|
|
92
102
|
|
|
103
|
+
/**
|
|
104
|
+
* The value of a valued option, written either way the table accepts,
|
|
105
|
+
* `--name value` or `--name=value`, the last occurrence winning as it does
|
|
106
|
+
* in options.mjs. Measured on 0.4.1: the table accepted `--out=FILE` and the
|
|
107
|
+
* value was looked up as the token after `--out`, so `doctor --out=x` wrote
|
|
108
|
+
* nothing and exited 0, and `submit --category=Widgets` said the flag was
|
|
109
|
+
* missing.
|
|
110
|
+
*/
|
|
93
111
|
function option(args, name) {
|
|
94
|
-
|
|
95
|
-
|
|
112
|
+
let value
|
|
113
|
+
for (let index = 0; index < args.length; index += 1) {
|
|
114
|
+
if (args[index] === name) value = args[index + 1]
|
|
115
|
+
else if (args[index].startsWith(`${name}=`)) value = args[index].slice(name.length + 1)
|
|
116
|
+
}
|
|
117
|
+
return value
|
|
96
118
|
}
|
|
97
119
|
|
|
98
120
|
/** The bare arguments, with every valued option's value (options.mjs, one table) left out. */
|
|
@@ -169,7 +191,8 @@ async function cmdSubmit(args) {
|
|
|
169
191
|
const usage = error.usage
|
|
170
192
|
if (json) {
|
|
171
193
|
process.stdout.write(`${JSON.stringify({ usage }, null, 2)}\n`)
|
|
172
|
-
process.
|
|
194
|
+
process.exitCode = 2
|
|
195
|
+
return
|
|
173
196
|
}
|
|
174
197
|
const flags = usage.missing.join(" and ")
|
|
175
198
|
fail("usage", `submit needs ${flags}: ${usage.missing.length === 1 ? "it is" : "they are"} an editorial choice nobody else can make, from the pinned form's own lists.`, 2,
|
|
@@ -185,7 +208,7 @@ async function cmdSubmit(args) {
|
|
|
185
208
|
emit(args, args.includes("--json") ? `${JSON.stringify(result, null, 2)}\n` : reportText(args, renderSubmit, result))
|
|
186
209
|
// Three outcomes, two exit codes: `ready` and `listed` are both healthy
|
|
187
210
|
// states, and only a refusal is a 1.
|
|
188
|
-
process.
|
|
211
|
+
process.exitCode = result.outcome === "refused" ? 1 : 0
|
|
189
212
|
}
|
|
190
213
|
|
|
191
214
|
async function cmdWatch(args) {
|
|
@@ -224,7 +247,7 @@ async function cmdWatch(args) {
|
|
|
224
247
|
spinner.done()
|
|
225
248
|
const render = result.mode === "list" ? renderWatchList : result.mode === "all" ? renderWatchAll : renderWatch
|
|
226
249
|
emit(args, args.includes("--json") ? `${JSON.stringify(result, null, 2)}\n` : reportText(args, render, result))
|
|
227
|
-
process.
|
|
250
|
+
process.exitCode = result.verdict?.state === "unknown" || result.summary?.unknown > 0 ? 2 : 0
|
|
228
251
|
}
|
|
229
252
|
|
|
230
253
|
async function cmdFrontDoor() {
|
|
@@ -243,15 +266,16 @@ async function cmdSetup(args) {
|
|
|
243
266
|
if (args.includes("--completion")) {
|
|
244
267
|
const identity = requirePin(ROOT).identity
|
|
245
268
|
const result = await completionStep({ repoRoot: ROOT, pin: identity.commit, version: VERSION, askRc: false })
|
|
246
|
-
process.
|
|
269
|
+
process.exitCode = result.state === "ok" ? 0 : 1
|
|
270
|
+
return
|
|
247
271
|
}
|
|
248
272
|
const result = await setup({ repoRoot: ROOT, entryPoint: resolve(ROOT, "bin/omakit"), yes: args.includes("--yes") })
|
|
249
|
-
process.
|
|
273
|
+
process.exitCode = result.ok ? 0 : 1
|
|
250
274
|
}
|
|
251
275
|
|
|
252
276
|
async function cmdUpgrade(args) {
|
|
253
277
|
const result = await upgrade({ repoRoot: ROOT, dryRun: args.includes("--dry-run") })
|
|
254
|
-
process.
|
|
278
|
+
process.exitCode = result.ok ? 0 : 1
|
|
255
279
|
}
|
|
256
280
|
|
|
257
281
|
async function cmdDoctor(args) {
|
|
@@ -259,7 +283,7 @@ async function cmdDoctor(args) {
|
|
|
259
283
|
const result = await doctor({ repoRoot: ROOT, offline: args.includes("--offline"), onPhase: spinner.phase })
|
|
260
284
|
spinner.done()
|
|
261
285
|
emit(args, args.includes("--json") ? `${JSON.stringify(result, null, 2)}\n` : reportText(args, renderDoctor, result))
|
|
262
|
-
process.
|
|
286
|
+
process.exitCode = result.problems ? 1 : 0
|
|
263
287
|
}
|
|
264
288
|
|
|
265
289
|
async function cmdVerify(args) {
|
|
@@ -325,7 +349,7 @@ async function cmdParity(args) {
|
|
|
325
349
|
} catch (error) {
|
|
326
350
|
failFrom(error)
|
|
327
351
|
}
|
|
328
|
-
process.
|
|
352
|
+
process.exitCode = ok ? 0 : 1
|
|
329
353
|
}
|
|
330
354
|
|
|
331
355
|
function notAudited(message, remedy = null, exit = 1) {
|
|
@@ -359,7 +383,7 @@ async function cmdAudit(args) {
|
|
|
359
383
|
}
|
|
360
384
|
if (parsed.options.has("--json")) process.stdout.write(json)
|
|
361
385
|
else process.stdout.write(`${renderAudit(document)}\n`)
|
|
362
|
-
process.
|
|
386
|
+
process.exitCode = document.ok ? 0 : 1
|
|
363
387
|
}
|
|
364
388
|
|
|
365
389
|
/**
|
|
@@ -399,7 +423,7 @@ async function cmdWeigh(args) {
|
|
|
399
423
|
throw error
|
|
400
424
|
}
|
|
401
425
|
process.stdout.write(json ? `${JSON.stringify(list.rows, null, 2)}\n` : `${renderList(list)}\n`)
|
|
402
|
-
|
|
426
|
+
return
|
|
403
427
|
}
|
|
404
428
|
if (!target && !all) notWeighed("usage", "weigh needs a plugin: `omakit weigh <plugin-id-or-dir>`, or `omakit weigh --all` for every enabled third-party plugin.", "omakit weigh <plugin-id-or-dir>", 2)
|
|
405
429
|
if (target && all) notWeighed("usage", `--all weighs every enabled third-party plugin, so ${JSON.stringify(target)} is one argument more than it takes.`, "omakit weigh --all, or omakit weigh <plugin-id-or-dir>", 2)
|
|
@@ -466,7 +490,6 @@ async function cmdWeigh(args) {
|
|
|
466
490
|
} else {
|
|
467
491
|
process.stdout.write(`\n${renderWeigh(document)}\n`)
|
|
468
492
|
}
|
|
469
|
-
process.exit(0)
|
|
470
493
|
}
|
|
471
494
|
|
|
472
495
|
const VERSION = JSON.parse(readFileSync(resolve(ROOT, "package.json"), "utf8")).version
|
|
@@ -180,10 +180,10 @@ export async function authenticatedUser() {
|
|
|
180
180
|
}
|
|
181
181
|
|
|
182
182
|
/** Repository issues by their creator. PRs are excluded; pagination never silently truncates. */
|
|
183
|
-
export async function repositoryIssues(owner, repository, creator, { readJson = getJson, maxPages = 100 } = {}) {
|
|
183
|
+
export async function repositoryIssues(owner, repository, creator, { readJson = getJson, maxPages = 100, labels } = {}) {
|
|
184
184
|
const all = []
|
|
185
185
|
for (let page = 1; page <= maxPages; page += 1) {
|
|
186
|
-
const query = new URLSearchParams({ creator, state: "open", sort: "updated", direction: "desc", per_page: "100", page: String(page) })
|
|
186
|
+
const query = new URLSearchParams({ ...(creator ? { creator } : {}), ...(labels ? { labels } : {}), state: "open", sort: "updated", direction: "desc", per_page: "100", page: String(page) })
|
|
187
187
|
const batch = await readJson(`https://api.github.com/repos/${owner}/${repository}/issues?${query}`)
|
|
188
188
|
if (!Array.isArray(batch)) throw new GitHubError("github-unavailable", "GitHub did not return an issue list")
|
|
189
189
|
all.push(...batch.filter((item) => !item.pull_request))
|
|
@@ -205,6 +205,17 @@ export async function issueComments(owner, repository, number, maxPages = 10, re
|
|
|
205
205
|
throw new GitHubError("comments-incomplete", `Issue #${number} exceeded ${maxPages} comment pages; its latest baseline cannot be determined`)
|
|
206
206
|
}
|
|
207
207
|
|
|
208
|
+
/** Compare exact validated snapshots. The API caps its file list at 300: never classify a truncated diff. */
|
|
209
|
+
export async function compareCommits(repositoryUrl, previous, validated, { readJson = getJson } = {}) {
|
|
210
|
+
const match = String(repositoryUrl).match(/^https:\/\/github\.com\/([A-Za-z0-9_.-]+)\/([A-Za-z0-9_.-]+?)(?:\.git)?\/?$/)
|
|
211
|
+
if (!match || ![previous, validated].every((commit) => /^[a-f0-9]{40}$/i.test(commit))) throw new GitHubError("usage", "compare needs a repository and two full commit identifiers")
|
|
212
|
+
const url = `https://api.github.com/repos/${match[1]}/${match[2]}/compare/${previous}...${validated}?per_page=1`
|
|
213
|
+
const result = await readJson(url)
|
|
214
|
+
if (!["ahead", "identical"].includes(result?.status)) throw new GitHubError("compare-not-forward", "validated snapshots are not a forward comparison")
|
|
215
|
+
if (!Array.isArray(result.files) || result.files.length >= 300) throw new GitHubError("compare-incomplete", "compare file list unavailable or at the 300-file API limit")
|
|
216
|
+
return { url, files: result.files }
|
|
217
|
+
}
|
|
218
|
+
|
|
208
219
|
/**
|
|
209
220
|
* The current default-branch HEAD of a repository.
|
|
210
221
|
*
|
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
// Reproduce M9 with GET-only issue discovery and compare reads. No sampling.
|
|
2
|
+
import { resolve } from "node:path"
|
|
3
|
+
import { pathToFileURL } from "node:url"
|
|
4
|
+
import { MARKETPLACE_PIN } from "./pin.mjs"
|
|
5
|
+
import { repositoryIssues } from "./github.mjs"
|
|
6
|
+
import { liveRegistry } from "./registry.mjs"
|
|
7
|
+
import { reviewPolicy } from "./review-cost.mjs"
|
|
8
|
+
import { validationWatchAll } from "./watch.mjs"
|
|
9
|
+
|
|
10
|
+
export async function measureReviewCost(repoRoot) {
|
|
11
|
+
const policy = await reviewPolicy(repoRoot)
|
|
12
|
+
const registry = await liveRegistry({ repoRoot })
|
|
13
|
+
if (registry.source !== "head") throw new Error(`cannot measure at HEAD: ${registry.reason}`)
|
|
14
|
+
const [owner, repository] = new URL(MARKETPLACE_PIN.repository).pathname.slice(1).split("/")
|
|
15
|
+
const openedAt = new Date().toISOString()
|
|
16
|
+
const subjects = await repositoryIssues(owner, repository, undefined, { labels: policy.updateLabel })
|
|
17
|
+
const issues = subjects.filter((subject) => subject.state === "open" && !subject.pull_request).map((subject) => ({
|
|
18
|
+
number: subject.number, url: `${MARKETPLACE_PIN.repository}/issues/${subject.number}`, title: subject.title,
|
|
19
|
+
state: subject.state, labels: subject.labels.map((label) => typeof label === "string" ? label : label.name),
|
|
20
|
+
}))
|
|
21
|
+
// Only the manual-review subset needs comment reads and comparisons. No plugin HEAD reads.
|
|
22
|
+
const manual = issues.filter((subject) => subject.labels.includes(policy.reviewLabel))
|
|
23
|
+
const batch = await validationWatchAll({ repoRoot, discovery: { account: null, marketplace: MARKETPLACE_PIN.repository, issues: manual },
|
|
24
|
+
readRegistry: async () => registry, github: { defaultBranchHead: async () => null } })
|
|
25
|
+
const counts = batch.reviewCostSummary
|
|
26
|
+
return {
|
|
27
|
+
measurement: "M9", openedAt, completedAt: new Date().toISOString(), marketplace: MARKETPLACE_PIN.repository,
|
|
28
|
+
marketplaceHead: registry.commit, command: "node tools/marketplace/measure-review-cost.mjs", sample: false,
|
|
29
|
+
pluginUpdates: issues.length, manualQueue: manual.length, manualQueueShare: issues.length ? manual.length / issues.length : null,
|
|
30
|
+
docsOnly: counts.docsOnly, compared: counts.compared, skipped: counts.skipped.length,
|
|
31
|
+
docsOnlyShareOfCompared: counts.compared ? counts.docsOnly / counts.compared : null,
|
|
32
|
+
docsOnlyShareOfManualQueue: counts.skipped.length || !manual.length ? null : counts.docsOnly / manual.length,
|
|
33
|
+
rows: batch.issues.map((row) => ({ issue: row.issue.number, ...row.documentationDiff })),
|
|
34
|
+
}
|
|
35
|
+
}
|
|
36
|
+
|
|
37
|
+
if (process.argv[1] && import.meta.url === pathToFileURL(resolve(process.argv[1])).href) {
|
|
38
|
+
process.stdout.write(`${JSON.stringify(await measureReviewCost(resolve(process.cwd())), null, 2)}\n`)
|
|
39
|
+
}
|
|
@@ -0,0 +1,76 @@
|
|
|
1
|
+
// M6 reproduction: open author-fixes issues, bot marker against commits.atom.
|
|
2
|
+
// All remote reads use the existing GET-only client. Unknowns stay unknown.
|
|
3
|
+
import { resolve } from "node:path"
|
|
4
|
+
import { pathToFileURL } from "node:url"
|
|
5
|
+
import { MARKETPLACE_PIN } from "./pin.mjs"
|
|
6
|
+
import { repositoryIssues, getText } from "./github.mjs"
|
|
7
|
+
import { validationWatch } from "./watch.mjs"
|
|
8
|
+
|
|
9
|
+
export function atomHead(feed, source) {
|
|
10
|
+
const commit = feed.match(/<id>tag:github\.com,2008:Grit::Commit\/([0-9a-f]{40})<\/id>/i)?.[1]
|
|
11
|
+
|| feed.match(/\/commit\/([0-9a-f]{40})/i)?.[1]
|
|
12
|
+
if (!commit) throw new Error("No full HEAD commit in commits.atom")
|
|
13
|
+
return { source, commit: commit.toLowerCase(), branch: null,
|
|
14
|
+
committedAt: feed.match(/<updated>([^<]+)<\/updated>/)?.[1] || null }
|
|
15
|
+
}
|
|
16
|
+
|
|
17
|
+
export function stalenessCounts(rows) {
|
|
18
|
+
const stale = rows.filter((row) => row.verdict === "stale").length
|
|
19
|
+
const current = rows.filter((row) => row.verdict === "current").length
|
|
20
|
+
const compared = stale + current
|
|
21
|
+
return { total: rows.length, compared, stale, current, unknown: rows.length - compared,
|
|
22
|
+
staleShareOfCompared: compared ? stale / compared : null,
|
|
23
|
+
staleShareOfPopulation: compared === rows.length && rows.length ? stale / rows.length : null }
|
|
24
|
+
}
|
|
25
|
+
|
|
26
|
+
export async function measureStaleness(repoRoot, { discover = repositoryIssues, watch = validationWatch, readText = getText } = {}) {
|
|
27
|
+
const openedAt = new Date().toISOString()
|
|
28
|
+
const [owner, repository] = new URL(MARKETPLACE_PIN.repository).pathname.slice(1).split("/")
|
|
29
|
+
const batches = await Promise.all(["needs-fixes", "security-needs-fixes"].map((labels) => discover(owner, repository, undefined, { labels })))
|
|
30
|
+
const subjects = [...new Map(batches.flat().map((subject) => [subject.number, subject])).values()]
|
|
31
|
+
.filter((subject) => subject.state === "open" && !subject.pull_request)
|
|
32
|
+
.sort((a, b) => a.number - b.number)
|
|
33
|
+
const heads = new Map()
|
|
34
|
+
function readHead(url) {
|
|
35
|
+
if (!heads.has(url)) {
|
|
36
|
+
const source = `${url.replace(/\/$/, "")}/commits.atom`
|
|
37
|
+
heads.set(url, Promise.resolve().then(async () => atomHead(await readText(source, "application/atom+xml"), source)))
|
|
38
|
+
}
|
|
39
|
+
return heads.get(url)
|
|
40
|
+
}
|
|
41
|
+
const rows = new Array(subjects.length)
|
|
42
|
+
let next = 0
|
|
43
|
+
async function worker() {
|
|
44
|
+
while (next < subjects.length) {
|
|
45
|
+
const index = next++
|
|
46
|
+
const subject = subjects[index]
|
|
47
|
+
const issueUrl = `${MARKETPLACE_PIN.repository}/issues/${subject.number}`
|
|
48
|
+
const observedAt = new Date().toISOString()
|
|
49
|
+
try {
|
|
50
|
+
const report = await watch({ repoRoot, issueUrl, github: { issue: async () => subject, defaultBranchHead: readHead } })
|
|
51
|
+
rows[index] = { issue: subject.number, issueUrl, observedAt, completedAt: new Date().toISOString(),
|
|
52
|
+
repository: report.plugin.repository, validatedCommit: report.validated?.commit || null,
|
|
53
|
+
validationSource: report.validated?.source || null, validatedAt: report.validated?.checkedAt || null,
|
|
54
|
+
validationCommentsSource: `https://api.github.com/repos/${owner}/${repository}/issues/${subject.number}/comments`,
|
|
55
|
+
defaultBranchHead: report.head?.commit || null, headSource: report.head?.source || null,
|
|
56
|
+
verdict: report.verdict.state,
|
|
57
|
+
reason: report.verdict.state === "unknown" ? report.verdict.summary : null }
|
|
58
|
+
} catch (error) {
|
|
59
|
+
rows[index] = { issue: subject.number, issueUrl, observedAt, completedAt: new Date().toISOString(),
|
|
60
|
+
repository: null, validatedCommit: null, defaultBranchHead: null, verdict: "unknown",
|
|
61
|
+
reason: `${error.code || "read-unavailable"}: ${error.message}` }
|
|
62
|
+
}
|
|
63
|
+
}
|
|
64
|
+
}
|
|
65
|
+
await Promise.all(Array.from({ length: Math.min(4, subjects.length) }, worker))
|
|
66
|
+
return { measurement: "M6", date: openedAt.slice(0, 10), openedAt, completedAt: new Date().toISOString(),
|
|
67
|
+
marketplace: MARKETPLACE_PIN.repository, marketplacePin: MARKETPLACE_PIN.commit,
|
|
68
|
+
command: "node tools/marketplace/measure-staleness.mjs", sample: false,
|
|
69
|
+
population: "All open non-PR marketplace issues labelled needs-fixes or security-needs-fixes at discovery",
|
|
70
|
+
method: "Latest bot security-baseline full commit against default-branch commits.atom HEAD; missing full markers or feeds are unknown; unequal commits are stale, without a claim about ancestry",
|
|
71
|
+
...stalenessCounts(rows), rows }
|
|
72
|
+
}
|
|
73
|
+
|
|
74
|
+
if (process.argv[1] && import.meta.url === pathToFileURL(resolve(process.argv[1])).href) {
|
|
75
|
+
process.stdout.write(`${JSON.stringify(await measureStaleness(resolve(process.cwd())), null, 2)}\n`)
|
|
76
|
+
}
|
|
@@ -266,12 +266,19 @@ export function renderWatchList(result, { colour = colourEnabled() } = {}) {
|
|
|
266
266
|
/** Compact batch report; exact commits and full discussion remain in JSON. */
|
|
267
267
|
export function renderWatchAll(result, { colour = colourEnabled() } = {}) {
|
|
268
268
|
const c = styler(colour)
|
|
269
|
-
const out = [...field("account", result.account, c), ...field("issues", `${result.summary.total} checked; ${result.summary.current} current, ${result.summary.stale} stale, ${result.summary.unknown} unknown`, c)
|
|
269
|
+
const out = [...field("account", result.account, c), ...field("issues", `${result.summary.total} checked; ${result.summary.current} current, ${result.summary.stale} stale, ${result.summary.unknown} unknown`, c)]
|
|
270
|
+
if (result.reviewCostSummary) {
|
|
271
|
+
const cost = result.reviewCostSummary
|
|
272
|
+
const skipped = cost.skipped.length ? `; ${cost.skipped.length} diff(s) skipped (reasons on issue rows)` : ""
|
|
273
|
+
out.push(...field("review cost", `${cost.manualQueue} of ${cost.pluginUpdates} plugin-update issue(s) on security-review-required; ${cost.docsOnly} docs-only validated diff(s) of ${cost.compared} compared${skipped}`, c))
|
|
274
|
+
}
|
|
275
|
+
out.push("")
|
|
270
276
|
for (const row of result.issues) {
|
|
271
277
|
const state = row.report?.verdict.state || "unknown"
|
|
272
278
|
const style = { current: "pass", stale: "fail", unknown: "unknown" }[state]
|
|
273
279
|
out.push(...verdict(style, state.toUpperCase(), `#${row.issue.number} ${watchIssueTitle(row.report?.read.title || row.issue.title)}`, c))
|
|
274
280
|
out.push(...field("issue", row.issue.url, c, { wrapValue: false }))
|
|
281
|
+
if (row.documentationDiff?.docsOnly === null) out.push(...field("diff skipped", watchIssueTitle(row.documentationDiff.reason), c))
|
|
275
282
|
if (row.error) {
|
|
276
283
|
out.push(...field("read error", watchIssueTitle(`${row.error.code}: ${row.error.message}`), c))
|
|
277
284
|
} else {
|
|
@@ -0,0 +1,107 @@
|
|
|
1
|
+
// Review cost is advice, never a marketplace rule. Evidence: MEASUREMENTS.md M4 and M9.
|
|
2
|
+
import { readFileSync } from "node:fs"
|
|
3
|
+
import { join } from "node:path"
|
|
4
|
+
import { pathToFileURL } from "node:url"
|
|
5
|
+
import { requirePin } from "./pin.mjs"
|
|
6
|
+
import { token, issue, parseIssueUrl, compareCommits } from "./github.mjs"
|
|
7
|
+
import { discoverWatchIssues, repositoryFor } from "./watch.mjs"
|
|
8
|
+
import { repositorySlug } from "./registry.mjs"
|
|
9
|
+
|
|
10
|
+
/** Outcome and update label names come from the pin, rather than a second policy. */
|
|
11
|
+
export async function reviewPolicy(repoRoot) {
|
|
12
|
+
const { dir } = requirePin(repoRoot)
|
|
13
|
+
const policy = await import(pathToFileURL(join(dir, "scripts/security-baseline-policy.mjs")).href)
|
|
14
|
+
const approval = readFileSync(join(dir, "scripts/approve-plugin-update.mjs"), "utf8")
|
|
15
|
+
const updateLabel = approval.match(/for \(const required of \["([^"]+)"/)?.[1]
|
|
16
|
+
if (!updateLabel) throw new Error("cannot read the update issue label from the pin")
|
|
17
|
+
const manual = policy.currentSecurityBaselinePolicy.maintainerVerificationOutcome
|
|
18
|
+
return { automated: policy.securityBaselineOutcome([], []), manual, updateLabel, reviewLabel: `security-${manual}` }
|
|
19
|
+
}
|
|
20
|
+
|
|
21
|
+
/** The same discovery as watch --all; no credential or incomplete reads are not zero issues. */
|
|
22
|
+
export async function openIssuesForRepository({ repoRoot, repository, offline = false, github = {} }) {
|
|
23
|
+
if (offline) return { count: null, reason: "not checked (--offline)" }
|
|
24
|
+
if (!(github.token || token)()) return { count: null, reason: "not checked: no GitHub credential" }
|
|
25
|
+
try {
|
|
26
|
+
const discovery = await discoverWatchIssues({ github })
|
|
27
|
+
const { dir } = requirePin(repoRoot)
|
|
28
|
+
let count = 0
|
|
29
|
+
let next = 0
|
|
30
|
+
async function worker() {
|
|
31
|
+
while (next < discovery.issues.length) {
|
|
32
|
+
const target = parseIssueUrl(discovery.issues[next++].url)
|
|
33
|
+
const subject = await (github.issue || issue)(target.owner, target.repository, target.number)
|
|
34
|
+
const parsed = await repositoryFor(dir, subject)
|
|
35
|
+
if (!parsed.url) throw new Error(`repository unknown on issue #${target.number}: ${parsed.error}`)
|
|
36
|
+
if (repositorySlug(parsed.url) === repositorySlug(repository)) count += 1
|
|
37
|
+
}
|
|
38
|
+
}
|
|
39
|
+
await Promise.all(Array.from({ length: Math.min(4, discovery.issues.length) }, worker))
|
|
40
|
+
return { count, reason: `watch --all discovery: ${count} open issue(s) for this repository` }
|
|
41
|
+
} catch (error) {
|
|
42
|
+
return { count: null, reason: `not checked (${error.code || "issue-discovery-unavailable"}): ${error.message}` }
|
|
43
|
+
}
|
|
44
|
+
}
|
|
45
|
+
|
|
46
|
+
export function reviewCostVerdict({ baseline, policy, openIssues = { count: null, reason: "not checked" }, why }) {
|
|
47
|
+
const capabilities = [...(baseline?.capabilities || [])]
|
|
48
|
+
const manual = baseline?.outcome === policy.manual
|
|
49
|
+
const automated = baseline?.outcome === policy.automated
|
|
50
|
+
const outcome = manual ? openIssues.count > 0 ? "manual queue, again" : "manual queue" : automated ? "automated" : null
|
|
51
|
+
const detail = manual
|
|
52
|
+
? `${outcome}: capabilities ${capabilities.join(", ") || "none"}. Every update of this plugin, including a docs-only one, lands in the manual queue.${openIssues.count > 0 ? ` You already have ${openIssues.count} open issue(s) for this repository.` : ""}`
|
|
53
|
+
: automated ? "automated: this update will not need a human for the security baseline." : "not checked: baseline.preflight has no passed or review-required outcome"
|
|
54
|
+
return {
|
|
55
|
+
reviewCost: { outcome, capabilities, openIssuesForRepository: openIssues.count, reason: `${detail} ${openIssues.reason}` },
|
|
56
|
+
check: {
|
|
57
|
+
id: "review.cost", source: "omakit", severity: "advisory", verdict: manual ? "fail" : automated ? "pass" : "unknown",
|
|
58
|
+
why, detail, paths: [],
|
|
59
|
+
remedy: manual && openIssues.count > 0 ? "consider batching: close or fold the open one before opening another" : null,
|
|
60
|
+
},
|
|
61
|
+
}
|
|
62
|
+
}
|
|
63
|
+
|
|
64
|
+
/** A rename must be documentation at both ends; an empty diff is not a docs-only update. */
|
|
65
|
+
export function documentationPath(path) {
|
|
66
|
+
return typeof path === "string" && !path.split("/").some((part) => part === "..") &&
|
|
67
|
+
(/^docs\//i.test(path) || /\.md$/i.test(path) || /(?:^|\/)LICENSE$/i.test(path) || /\.(?:png|jpe?g|gif|webp|svg|ico|avif|bmp|tiff?)$/i.test(path))
|
|
68
|
+
}
|
|
69
|
+
|
|
70
|
+
export function docsOnlyFiles(files) {
|
|
71
|
+
return Array.isArray(files) && files.length > 0 && files.every((file) =>
|
|
72
|
+
documentationPath(file.filename) && (!file.previous_filename || documentationPath(file.previous_filename)))
|
|
73
|
+
}
|
|
74
|
+
|
|
75
|
+
/** Select a dated validation record, never a parent guessed from Git history. */
|
|
76
|
+
export function previousValidatedCommit(registry, report, catalog = null) {
|
|
77
|
+
if (!repositorySlug(report.plugin.repository)) return null
|
|
78
|
+
const sources = Array.isArray(registry?.sources) ? registry.sources : Object.values(registry?.sources || {})
|
|
79
|
+
const source = sources.find((entry) => repositorySlug(entry.repo) === repositorySlug(report.plugin.repository))
|
|
80
|
+
if (!report.validated?.commit) return null
|
|
81
|
+
const target = report.validated.commit
|
|
82
|
+
const checkedAt = Date.parse(report.validated.checkedAt)
|
|
83
|
+
const candidates = [...(Array.isArray(source?.listingValidationHistory) ? source.listingValidationHistory : []), {
|
|
84
|
+
commit: source?.listingValidatedCommit, validatedAt: source?.listingValidatedAt,
|
|
85
|
+
}, { commit: report.previousValidated?.commit, validatedAt: report.previousValidated?.checkedAt }]
|
|
86
|
+
// The catalog also records successful upstream validation before an update
|
|
87
|
+
// is promoted to the listing. This is validation evidence, not branch HEAD.
|
|
88
|
+
for (const plugin of Array.isArray(catalog?.plugins) ? catalog.plugins : []) {
|
|
89
|
+
if (repositorySlug(plugin.repo) === repositorySlug(report.plugin.repository)) {
|
|
90
|
+
candidates.push({ commit: plugin.upstreamValidatedCommit, validatedAt: plugin.upstreamValidatedAt })
|
|
91
|
+
}
|
|
92
|
+
}
|
|
93
|
+
return candidates.filter((entry) => /^[a-f0-9]{40}$/i.test(entry.commit || "") && entry.commit.toLowerCase() !== target.toLowerCase() &&
|
|
94
|
+
Number.isFinite(Date.parse(entry.validatedAt)) && Date.parse(entry.validatedAt) <= checkedAt)
|
|
95
|
+
.sort((a, b) => Date.parse(a.validatedAt) - Date.parse(b.validatedAt)).at(-1)?.commit.toLowerCase() || null
|
|
96
|
+
}
|
|
97
|
+
|
|
98
|
+
export async function validatedDocumentationDiff({ report, registry, catalog, registryReason = null, compare = compareCommits }) {
|
|
99
|
+
const previous = previousValidatedCommit(registry, report, catalog)
|
|
100
|
+
if (!previous) return { previousCommit: null, validatedCommit: report.validated?.commit || null, docsOnly: null, files: null, source: null, reason: registryReason || "previous validated commit unknown in the marketplace registry" }
|
|
101
|
+
try {
|
|
102
|
+
const diff = await compare(report.plugin.repository, previous, report.validated.commit)
|
|
103
|
+
return { previousCommit: previous, validatedCommit: report.validated.commit, docsOnly: docsOnlyFiles(diff.files), files: diff.files.length, source: diff.url, reason: null }
|
|
104
|
+
} catch (error) {
|
|
105
|
+
return { previousCommit: previous, validatedCommit: report.validated.commit, docsOnly: null, files: null, source: null, reason: `${error.code || "compare-unavailable"}: ${error.message}` }
|
|
106
|
+
}
|
|
107
|
+
}
|
|
@@ -23,6 +23,7 @@ import { renderIssue, verifyAgainstOfficialParser } from "./issue.mjs"
|
|
|
23
23
|
import { defaultBranchHead } from "./github.mjs"
|
|
24
24
|
import { REFRESH_ACTION } from "./watch.mjs"
|
|
25
25
|
import { omakitCacheDir } from "./paths.mjs"
|
|
26
|
+
import { openIssuesForRepository, reviewCostVerdict, reviewPolicy } from "./review-cost.mjs"
|
|
26
27
|
|
|
27
28
|
/**
|
|
28
29
|
* One arrow per cause, in this order, so a person fixes the thing that is
|
|
@@ -135,6 +136,7 @@ export function reproduceCommand({ target, category, tags, pluginName, notes, su
|
|
|
135
136
|
* @param {{ repoRoot: string, target: string, category?: string, tags?: string|string[],
|
|
136
137
|
* notes?: string, suggestedTag?: string, pluginName?: string,
|
|
137
138
|
* allowDirty?: boolean, offline?: boolean,
|
|
139
|
+
* github?: object,
|
|
138
140
|
* readRegistry?: typeof liveRegistry,
|
|
139
141
|
* chooser?: (question: { contract: object, defaults: object, missing: string[] }) => Promise<{ category?: string, tags?: string[] }> }} options
|
|
140
142
|
* `readRegistry` is injectable for tests; the default reads the marketplace's
|
|
@@ -404,7 +406,7 @@ export async function submitPreflight(options) {
|
|
|
404
406
|
if (!options.offline && subject.repository.url) {
|
|
405
407
|
phase("reading the repository's default-branch HEAD")
|
|
406
408
|
try {
|
|
407
|
-
head = await defaultBranchHead(subject.repository.url)
|
|
409
|
+
head = await (options.github?.defaultBranchHead || defaultBranchHead)(subject.repository.url)
|
|
408
410
|
} catch (error) {
|
|
409
411
|
headError = { code: error.code || "head-unreadable", message: error.message }
|
|
410
412
|
}
|
|
@@ -412,9 +414,14 @@ export async function submitPreflight(options) {
|
|
|
412
414
|
const validationMatches = head ? head.commit === subject.commit.toLowerCase() : null
|
|
413
415
|
checks.push(check("submission.validation-commit", {
|
|
414
416
|
source: "omakit",
|
|
415
|
-
why: "The marketplace validates the default-branch HEAD it resolves when the issue is opened or edited, not the commit checked here.
|
|
417
|
+
why: "The marketplace validates the default-branch HEAD it resolves when the issue is opened or edited, not the commit checked here. M6 on 2026-09-15 found 326/519 readable author-fixes comparisons stale (62.8%), with 64 of 583 issues unknown, so a preflight against a commit that is not the pushed HEAD describes a tree nobody will review. Not a marketplace rule; an Omakit refusal to report on the wrong tree.",
|
|
416
418
|
severity: options.offline ? "advisory" : "blocking",
|
|
417
419
|
skipped: options.offline === true,
|
|
420
|
+
// No origin, no URL to read a HEAD from: the check waits on the one that
|
|
421
|
+
// says so. Measured on 0.4.1: it failed as a second root cause with
|
|
422
|
+
// "could not read the default-branch HEAD (unknown): " for a read that
|
|
423
|
+
// was never attempted.
|
|
424
|
+
waitedOn: [!options.offline && !subject.repository.url && "submission.repository-url"],
|
|
418
425
|
verdict: validationMatches === true,
|
|
419
426
|
detail: options.offline
|
|
420
427
|
? `not checked (--offline). Local commit ${subject.commit}.`
|
|
@@ -461,6 +468,14 @@ export async function submitPreflight(options) {
|
|
|
461
468
|
: null,
|
|
462
469
|
}))
|
|
463
470
|
|
|
471
|
+
const policy = await reviewPolicy(repoRoot)
|
|
472
|
+
const review = reviewCostVerdict({ baseline: consequence, policy,
|
|
473
|
+
openIssues: consequence?.outcome === policy.manual ? await openIssuesForRepository({ repoRoot, repository: subject.repository.url,
|
|
474
|
+
offline: options.offline === true, github: options.github }) : { count: null, reason: consequence?.outcome === policy.automated ? "open issue count not checked: automated baseline" : preflight.skipReason || preflight.refusal?.message || "baseline outcome needs findings resolved" },
|
|
475
|
+
why: `MEASUREMENTS.md M4: ${figure(figures.outcomes[policy.manual] || 0)} of ${figure(figures.withBaseline)} recorded listing baselines required review at the pin. M9: on 2026-09-15, 140 of 307 open update issues carried the manual-review label; 4 of 139 compared validated diffs were docs-only, with 1 unavailable. The baseline scans the whole snapshot, not the update diff, so unchanged capabilities also require another review. Sources and exact marketplace HEAD are recorded in MEASUREMENTS.md.`,
|
|
476
|
+
})
|
|
477
|
+
checks.push(review.check)
|
|
478
|
+
|
|
464
479
|
const blocking = checks.filter((entry) => entry.severity === "blocking" && entry.verdict === "fail")
|
|
465
480
|
const advisory = checks.filter((entry) => entry.severity === "advisory" && entry.verdict === "fail")
|
|
466
481
|
const unknown = checks.filter((entry) => entry.verdict === "unknown")
|
|
@@ -511,6 +526,7 @@ export async function submitPreflight(options) {
|
|
|
511
526
|
offline: options.offline === true,
|
|
512
527
|
}),
|
|
513
528
|
checks,
|
|
529
|
+
reviewCost: review.reviewCost,
|
|
514
530
|
outcome,
|
|
515
531
|
ready,
|
|
516
532
|
listing,
|
|
@@ -9,19 +9,11 @@
|
|
|
9
9
|
// compares branch HEADs only for repositories that are already listed. So
|
|
10
10
|
// pushing a fix does nothing, and commenting "fixed in abc123" does nothing.
|
|
11
11
|
//
|
|
12
|
-
// Measured reason (docs/MEASUREMENTS.md M6): of
|
|
13
|
-
//
|
|
14
|
-
//
|
|
15
|
-
//
|
|
16
|
-
//
|
|
17
|
-
// 9 had passed validation and passed the automated security baseline with zero
|
|
18
|
-
// findings and were blocked solely because their validated commit had fallen
|
|
19
|
-
// behind while they waited. 46% of the maintainer's own requests for a fresh validation never
|
|
20
|
-
// produced one; in the parked group 77% never did. The instruction that would
|
|
21
|
-
// fix this appears 22 times in the failure path of
|
|
22
|
-
// `scripts/submission-feedback.mjs` and zero times in the success path of
|
|
23
|
-
// `scripts/validate-submission.mjs`, which is the path 97 of 100 parked
|
|
24
|
-
// submissions took.
|
|
12
|
+
// Measured reason (docs/MEASUREMENTS.md M6), 2026-09-15: 326 of 519 readable
|
|
13
|
+
// author-fixes comparisons were stale (62.8%), with 64 of 583 issues unknown.
|
|
14
|
+
// The original 2026-09-12 research sampled 100 of a 464-issue queue: 68 of
|
|
15
|
+
// 93 readable pairs were stale (73.1%). It did not measure every queue item.
|
|
16
|
+
// Editing the issue body is the action the pinned workflow observes.
|
|
25
17
|
//
|
|
26
18
|
// This command reads. It never edits the issue, never comments, never labels.
|
|
27
19
|
// The action it names is the author's to take.
|
|
@@ -30,6 +22,8 @@ import { join } from "node:path"
|
|
|
30
22
|
import { pathToFileURL } from "node:url"
|
|
31
23
|
import { MARKETPLACE_PIN, requirePin } from "./pin.mjs"
|
|
32
24
|
import { authenticatedUser, repositoryIssues, defaultBranchHead, issue, issueComments, parseIssueUrl, token, GitHubError } from "./github.mjs"
|
|
25
|
+
import { liveRegistry } from "./registry.mjs"
|
|
26
|
+
import { reviewPolicy, validatedDocumentationDiff } from "./review-cost.mjs"
|
|
33
27
|
|
|
34
28
|
export class WatchError extends Error {
|
|
35
29
|
constructor(code, message) {
|
|
@@ -77,8 +71,11 @@ export async function discoverWatchIssues({ user, github = {}, onPhase = () => {
|
|
|
77
71
|
}
|
|
78
72
|
|
|
79
73
|
/** A batch keeps independent read failures visible and shares repository HEAD reads. */
|
|
80
|
-
export async function validationWatchAll({ repoRoot, discovery, github = {}, onPhase = () => {} }) {
|
|
74
|
+
export async function validationWatchAll({ repoRoot, discovery, github = {}, readRegistry = liveRegistry, onPhase = () => {} }) {
|
|
81
75
|
if (discovery.issues.length) requirePin(repoRoot)
|
|
76
|
+
const policy = discovery.issues.length ? await reviewPolicy(repoRoot) : null
|
|
77
|
+
let registry = null
|
|
78
|
+
const reviewCostSummary = { pluginUpdates: 0, manualQueue: 0, docsOnly: 0, compared: 0, skipped: [] }
|
|
82
79
|
const heads = new Map()
|
|
83
80
|
const readHead = github.defaultBranchHead || defaultBranchHead
|
|
84
81
|
const shared = { ...github, defaultBranchHead: (url) => {
|
|
@@ -102,7 +99,38 @@ export async function validationWatchAll({ repoRoot, discovery, github = {}, onP
|
|
|
102
99
|
await Promise.all(Array.from({ length: Math.min(4, discovery.issues.length) }, worker))
|
|
103
100
|
const summary = { total: results.length, current: 0, stale: 0, unknown: 0 }
|
|
104
101
|
for (const result of results) summary[result.report?.verdict.state || "unknown"] += 1
|
|
105
|
-
|
|
102
|
+
// M9: count labels on the listed issues, then compare only updates in the manual queue.
|
|
103
|
+
const manual = results.filter((row) => {
|
|
104
|
+
const labels = row.report?.read.labels || row.issue.labels
|
|
105
|
+
if (!labels.includes(policy?.updateLabel)) return false
|
|
106
|
+
reviewCostSummary.pluginUpdates += 1
|
|
107
|
+
return labels.includes(policy.reviewLabel)
|
|
108
|
+
})
|
|
109
|
+
reviewCostSummary.manualQueue = manual.length
|
|
110
|
+
if (manual.length) {
|
|
111
|
+
onPhase("reading the previous validated marketplace snapshots")
|
|
112
|
+
try { registry = await readRegistry({ repoRoot }) } catch (error) { registry = { reason: error.message } }
|
|
113
|
+
}
|
|
114
|
+
let compareNext = 0
|
|
115
|
+
async function compareWorker() {
|
|
116
|
+
while (compareNext < manual.length) {
|
|
117
|
+
const row = manual[compareNext++]
|
|
118
|
+
const diff = row.report
|
|
119
|
+
? await validatedDocumentationDiff({ report: row.report, registry: registry?.source === "head" ? registry.registry : null,
|
|
120
|
+
catalog: registry?.source === "head" ? registry.catalog : null,
|
|
121
|
+
registryReason: registry?.source === "head" ? null : `previous validated commit unknown: live registry unavailable (${registry?.reason || "unknown"})`, compare: github.compareCommits })
|
|
122
|
+
: { docsOnly: null, reason: `issue unreadable: ${row.error?.message || "unknown"}` }
|
|
123
|
+
row.documentationDiff = diff
|
|
124
|
+
if (diff.docsOnly === null) reviewCostSummary.skipped.push({ issue: row.issue.number, reason: diff.reason })
|
|
125
|
+
else {
|
|
126
|
+
reviewCostSummary.compared += 1
|
|
127
|
+
if (diff.docsOnly) reviewCostSummary.docsOnly += 1
|
|
128
|
+
}
|
|
129
|
+
}
|
|
130
|
+
}
|
|
131
|
+
await Promise.all(Array.from({ length: Math.min(4, manual.length) }, compareWorker))
|
|
132
|
+
reviewCostSummary.skipped.sort((a, b) => a.issue - b.issue)
|
|
133
|
+
return { mode: "all", account: discovery.account, marketplace: discovery.marketplace, summary, reviewCostSummary, issues: results }
|
|
106
134
|
}
|
|
107
135
|
|
|
108
136
|
// The one action that re-runs validation, in the register the marketplace itself
|
|
@@ -133,7 +161,7 @@ async function loadVerification(pinDir) {
|
|
|
133
161
|
* happens to recognise. So each form is read by the parser the marketplace
|
|
134
162
|
* itself uses for it, and the issue says which one it is.
|
|
135
163
|
*/
|
|
136
|
-
async function repositoryFor(pinDir, subject) {
|
|
164
|
+
export async function repositoryFor(pinDir, subject) {
|
|
137
165
|
const submission = await loadSubmission(pinDir)
|
|
138
166
|
const verification = await loadVerification(pinDir)
|
|
139
167
|
const title = String(subject.title || "")
|
|
@@ -221,11 +249,29 @@ export async function validationWatch({ repoRoot, issueUrl, onPhase, github = {}
|
|
|
221
249
|
baselineError = { code: error.code || "baseline-unreadable", message: error.message }
|
|
222
250
|
}
|
|
223
251
|
const fallback = validated ? null : validationCommentCommit(comments)
|
|
252
|
+
let previousValidated = null
|
|
253
|
+
if (validated) {
|
|
254
|
+
// Revalidation on an existing issue may precede the latest marker. Ignore
|
|
255
|
+
// repeated attestations of the same commit, not a different validated tree.
|
|
256
|
+
for (const comment of [...comments].reverse()) {
|
|
257
|
+
try {
|
|
258
|
+
const marker = record.findLatestSecurityBaseline([comment])
|
|
259
|
+
if (marker && marker.commitSha !== validated.commit && Date.parse(marker.checkedAt) <= Date.parse(validated.checkedAt)) {
|
|
260
|
+
previousValidated = { commit: marker.commitSha, checkedAt: marker.checkedAt, source: "security-baseline-marker" }
|
|
261
|
+
break
|
|
262
|
+
}
|
|
263
|
+
} catch { /* An incomplete run is not a previously validated commit. */ }
|
|
264
|
+
}
|
|
265
|
+
}
|
|
224
266
|
|
|
225
267
|
const labels = (subject.labels || []).map((label) => (typeof label === "string" ? label : label?.name)).filter(Boolean)
|
|
226
268
|
const authorComments = comments.filter((comment) => comment?.user?.login === subject.user?.login)
|
|
269
|
+
// The discussion is a person's: the marketplace's own bot and any other
|
|
270
|
+
// automation (GitHub marks an app's account `type: "Bot"`, and names it
|
|
271
|
+
// `<app>[bot]`) is neither a reviewer nor the last human review.
|
|
272
|
+
const isBot = (user) => user?.type === "Bot" || /\[bot\]$/i.test(String(user?.login || ""))
|
|
227
273
|
const maintainerComments = comments.filter(
|
|
228
|
-
(comment) => comment?.user?.login && comment.user.login !== subject.user?.login && comment.user
|
|
274
|
+
(comment) => comment?.user?.login && comment.user.login !== subject.user?.login && !isBot(comment.user),
|
|
229
275
|
)
|
|
230
276
|
|
|
231
277
|
let head = null
|
|
@@ -267,6 +313,7 @@ export async function validationWatch({ repoRoot, issueUrl, onPhase, github = {}
|
|
|
267
313
|
},
|
|
268
314
|
plugin: { repository: repositoryUrl, repositoryError, form: issueKind },
|
|
269
315
|
validated,
|
|
316
|
+
previousValidated,
|
|
270
317
|
validationCommentFallback: fallback,
|
|
271
318
|
baselineError,
|
|
272
319
|
head,
|
package/tools/weigh/audit.mjs
CHANGED
|
@@ -331,7 +331,12 @@ async function sampleConfig({ label, runIndex, config, plan, env, procRoot, sign
|
|
|
331
331
|
onPhase(`run ${runIndex} of ${plan.runs}: ${label}, sampling for ${windowSeconds}s`)
|
|
332
332
|
const started = utc()
|
|
333
333
|
const t0 = Date.now()
|
|
334
|
-
|
|
334
|
+
// A pid with no stat is a process that is gone, and a gone shell has no
|
|
335
|
+
// sample: nothing is read as zero in its place. Measured on 0.4.1: a pid
|
|
336
|
+
// absent from /proc "completed" the window with 0 ticks and a Pss of
|
|
337
|
+
// null read as 0 MB, a delta of minus the whole baseline.
|
|
338
|
+
const cpu0 = cpuTicks(procRoot, pid)
|
|
339
|
+
if (cpu0 === null) return { label, run: runIndex, failed: `the shell (pid ${pid}) is not in ${procRoot}` }
|
|
335
340
|
const child0 = childTicks(procRoot, pid) ?? 0
|
|
336
341
|
const rows = []
|
|
337
342
|
const trace = []
|
|
@@ -355,7 +360,8 @@ async function sampleConfig({ label, runIndex, config, plan, env, procRoot, sign
|
|
|
355
360
|
} catch {
|
|
356
361
|
configRewritten = null
|
|
357
362
|
}
|
|
358
|
-
const cpu1 = cpuTicks(procRoot, pid)
|
|
363
|
+
const cpu1 = cpuTicks(procRoot, pid)
|
|
364
|
+
if (cpu1 === null) return { label, run: runIndex, failed: `the shell (pid ${pid}) disappeared from ${procRoot} during the window` }
|
|
359
365
|
const child1 = childTicks(procRoot, pid) ?? child0
|
|
360
366
|
const memory = { pssKb: pssKb(procRoot, pid), rssKb: rssKb(procRoot, pid), memoryAt: "window-end", ...settled, trace }
|
|
361
367
|
const seconds = (t1 - t0) / 1000
|
|
@@ -605,6 +611,13 @@ export async function measureWeigh(plan, { env = plan.env || process.env, procRo
|
|
|
605
611
|
let restore = null
|
|
606
612
|
let restoreProblem = null
|
|
607
613
|
let comeBack = null
|
|
614
|
+
// What stopped the measurement early, an interrupt or a thrown error, is
|
|
615
|
+
// kept so the restore's outcome can be judged first: a restore that failed
|
|
616
|
+
// or did not verify is the fact the person is left with, and it is what
|
|
617
|
+
// is reported, with the stop as its context. Measured on 0.4.1: an
|
|
618
|
+
// interrupt propagated past a restore whose md5 differed, and the entry
|
|
619
|
+
// point closed with "shell.json was restored".
|
|
620
|
+
let stopped = null
|
|
608
621
|
try {
|
|
609
622
|
for (let runIndex = 1; runIndex <= plan.runs; runIndex += 1) {
|
|
610
623
|
for (const { label, config } of configs) {
|
|
@@ -619,6 +632,8 @@ export async function measureWeigh(plan, { env = plan.env || process.env, procRo
|
|
|
619
632
|
samples.push(sample)
|
|
620
633
|
}
|
|
621
634
|
}
|
|
635
|
+
} catch (error) {
|
|
636
|
+
stopped = error
|
|
622
637
|
} finally {
|
|
623
638
|
onPhase("restoring shell.json and restarting the shell")
|
|
624
639
|
try {
|
|
@@ -642,7 +657,10 @@ export async function measureWeigh(plan, { env = plan.env || process.env, procRo
|
|
|
642
657
|
onLine({ state: "fail", text: `the restore failed: ${error.message}; the backup is ${backup.backupFile}` })
|
|
643
658
|
}
|
|
644
659
|
}
|
|
645
|
-
|
|
660
|
+
const because = stopped ? `. The measurement had stopped first: ${stopped.message}` : ""
|
|
661
|
+
if (restoreProblem) throw new WeighError("restore-failed", `shell.json could not be restored from ${backup.backupFile}: ${restoreProblem.message}${because}`, `Copy ${backup.backupFile} over ${plan.configFile} yourself, then run omarchy-restart-shell.`)
|
|
662
|
+
if (stopped && !restore.restored) throw new WeighError("restore-unverified", `shell.json differs from the backup after the restore; the backup ${backup.backupFile} is kept${because}`, `Compare ${backup.backupFile} with ${plan.configFile} and copy it over if the difference is not yours.`)
|
|
663
|
+
if (stopped) throw stopped
|
|
646
664
|
const ended = utc()
|
|
647
665
|
const config = {
|
|
648
666
|
path: plan.configFile,
|