dsh-report-card 1.0.0 → 1.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +63 -9
- package/docs/example-card.md +1 -1
- package/docs/publishing.md +14 -13
- package/harness/auto.mjs +220 -0
- package/index.js +154 -22
- package/package.json +3 -3
package/README.md
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
|
|
1
|
+
# DSH Report Card
|
|
2
2
|
|
|
3
3
|
An independent, **dynamic** benchmark for DeepSeek Harness plugins. It installs
|
|
4
4
|
one plugin into a throwaway profile, boots the real harness with it, runs one
|
|
@@ -87,20 +87,26 @@ test fixture.
|
|
|
87
87
|
|
|
88
88
|
## Install
|
|
89
89
|
|
|
90
|
-
|
|
91
|
-
and an icon.
|
|
90
|
+
Published on npm as **`dsh-report-card@1.0.0`**. It ships as a bundle with
|
|
91
|
+
metadata, `locale/en.json`, `locale/zh.json` and an icon. The plugin manager adds
|
|
92
92
|
the dependency *and* enables the bundle in one step; removing it undoes both:
|
|
93
93
|
|
|
94
94
|
```
|
|
95
|
-
plugin_manager install_bundle target: "
|
|
95
|
+
plugin_manager install_bundle target: "dsh-report-card@1.0.0"
|
|
96
96
|
plugin_manager remove_bundle target: "dsh-report-card"
|
|
97
97
|
```
|
|
98
98
|
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
99
|
+
or from a terminal:
|
|
100
|
+
|
|
101
|
+
```powershell
|
|
102
|
+
dsh plugin --profile web add dsh-report-card
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
For development from a checkout, pass a directory spec instead
|
|
106
|
+
(`file:C:/path/to/dsh-report-card`). `install_bundle` is the sanctioned path
|
|
107
|
+
because a bundle is composed through the profile's bundle list, not just
|
|
108
|
+
`node_modules`; the bundle inserts one entry, `dsh-report-card`, which
|
|
109
|
+
contributes one tool.
|
|
104
110
|
|
|
105
111
|
## Use
|
|
106
112
|
|
|
@@ -108,8 +114,12 @@ From a session (the tool is model-facing):
|
|
|
108
114
|
|
|
109
115
|
```
|
|
110
116
|
report_card(pluginSpec: "dsh-trust-check@0.2.6", samples: 3)
|
|
117
|
+
report_cards_recent(limit: 10) # the last cards on disk, including automatic ones
|
|
111
118
|
```
|
|
112
119
|
|
|
120
|
+
`report_card` measures on demand; `report_cards_recent` reads the card index so
|
|
121
|
+
you can see what the background pass produced.
|
|
122
|
+
|
|
113
123
|
From a terminal — the same code path, printable output:
|
|
114
124
|
|
|
115
125
|
```powershell
|
|
@@ -126,6 +136,48 @@ sample round (surface × 2, turn × 2) plus one discarded warm-up round. Pass
|
|
|
126
136
|
`samples: 3` for a quick look and `samples: 8..10` when you need to resolve a
|
|
127
137
|
few milliseconds.
|
|
128
138
|
|
|
139
|
+
## Automatic measurement
|
|
140
|
+
|
|
141
|
+
Since 1.1.0 the card is produced **after a plugin is installed**, without
|
|
142
|
+
anyone asking. The plugin listens to the plugin manager's
|
|
143
|
+
`plugin-manager/operation` event and watches the profile manifest, queues the
|
|
144
|
+
new package, and measures it in the background — the same throwaway profile,
|
|
145
|
+
three samples, one measurement at a time.
|
|
146
|
+
|
|
147
|
+
```
|
|
148
|
+
install a plugin
|
|
149
|
+
│ manager event (Plugins page, plugin_manager tool)
|
|
150
|
+
│ or profile package.json change (dsh plugin add)
|
|
151
|
+
▼
|
|
152
|
+
debounce 2.5 s → list installed third-party bundles → skip ones already carded
|
|
153
|
+
▼
|
|
154
|
+
queue → measure in a throwaway profile → save card under $DSH_HOME/report-cards
|
|
155
|
+
▼
|
|
156
|
+
report_cards_recent lists it
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
What it skips: shipped/optional/official bundles, bundles that failed to load,
|
|
160
|
+
its own package, and any `name@version` that already has a card. What it does
|
|
161
|
+
**not** do: measure plugins that were already installed before this version
|
|
162
|
+
started, unless `autoBackfill` is on.
|
|
163
|
+
|
|
164
|
+
Automatic cards take roughly 45 s each (three samples) and run serially; a
|
|
165
|
+
manual `report_card` call queues behind a running automatic card so the two
|
|
166
|
+
never compete for the machine.
|
|
167
|
+
|
|
168
|
+
Config, in the profile patch (or the plugin's config in a preset):
|
|
169
|
+
|
|
170
|
+
```yaml
|
|
171
|
+
- id: dsh-report-card
|
|
172
|
+
config:
|
|
173
|
+
autoMeasure: true # false turns the background behavior off
|
|
174
|
+
autoSamples: 3 # 3..10; steadier numbers cost more time
|
|
175
|
+
autoBackfill: false # true: card already-installed plugins at startup
|
|
176
|
+
autoReportsDir: null # default: <DSH_HOME>/report-cards
|
|
177
|
+
```
|
|
178
|
+
|
|
179
|
+
`DSH_REPORT_CARD_AUTO=0` disables it for one process without touching config.
|
|
180
|
+
|
|
129
181
|
## Reading a card
|
|
130
182
|
|
|
131
183
|
```
|
|
@@ -177,6 +229,7 @@ Tests:
|
|
|
177
229
|
|
|
178
230
|
```powershell
|
|
179
231
|
node test/unit.test.mjs # units, ~1 s
|
|
232
|
+
node test/auto.test.mjs # install trigger and queue, ~1 s
|
|
180
233
|
node test/harness.test.mjs # real boots against fixtures, ~80 s
|
|
181
234
|
```
|
|
182
235
|
|
|
@@ -227,6 +280,7 @@ check that was run against the packed artifact.
|
|
|
227
280
|
| `harness/risk.mjs` | Static disclosure scan |
|
|
228
281
|
| `harness/report.mjs` | Grades, digest, drift, card assembly |
|
|
229
282
|
| `harness/render.mjs` | Markdown card |
|
|
283
|
+
| `harness/auto.mjs` | Install trigger, eligibility, and the background measurement queue |
|
|
230
284
|
| `harness/repeat.mjs` | Repeatability check for the harness itself |
|
|
231
285
|
| `docs/` | An example card and the acceptance evidence |
|
|
232
286
|
| `locale/`, `icon.svg`, `cordis.patch.yml` | Bundle metadata and its one insert row |
|
package/docs/example-card.md
CHANGED
package/docs/publishing.md
CHANGED
|
@@ -11,23 +11,24 @@ There are three publication surfaces, and they depend on each other in order:
|
|
|
11
11
|
3. **Your own index (optional)** — a format-1 JSON document you host; users add
|
|
12
12
|
it as an index source in Plugins → Discover.
|
|
13
13
|
|
|
14
|
-
The rest of this file is the exact sequence for this package.
|
|
15
|
-
|
|
14
|
+
The rest of this file is the exact sequence for this package. Release state
|
|
15
|
+
(2026-11-10):
|
|
16
16
|
|
|
17
|
-
|
|
|
17
|
+
| Surface | State |
|
|
18
18
|
|---|---|
|
|
19
|
-
|
|
|
20
|
-
|
|
|
21
|
-
|
|
|
22
|
-
|
|
|
23
|
-
|
|
|
19
|
+
| npm | `dsh-report-card@1.0.0` published (publisher `init3`), 25 files, 139 kB unpacked |
|
|
20
|
+
| GitHub | <https://github.com/tespio/dsh-report-card>, tag `v1.0.0` |
|
|
21
|
+
| Registry acceptance | `node harness/cli.mjs dsh-report-card@1.0.0 --samples 3` installs from npm, registers `report_card`, entry `active` |
|
|
22
|
+
| Live profile | installed from the registry (`"dsh-report-card": "1.0.0"`), entry active |
|
|
23
|
+
| Catalog | **pending** — add `data/plugins/tespio__dsh-report-card.yml` from [`catalog-entry.yml`](catalog-entry.yml) |
|
|
24
24
|
|
|
25
25
|
## 0. Create the repository
|
|
26
26
|
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
the catalog entry and the
|
|
27
|
+
Already done for 1.0.0; kept as the initial-setup reference. The catalog entry
|
|
28
|
+
points at a `https://github.com/owner/repo` URL, and the catalog's npm probe only
|
|
29
|
+
maps a package whose own `repository` field points back at that repo. So the repo
|
|
30
|
+
is not optional bookkeeping — it is the link between the catalog entry and the
|
|
31
|
+
npm package.
|
|
31
32
|
|
|
32
33
|
```powershell
|
|
33
34
|
cd dsh-report-card
|
|
@@ -56,7 +57,7 @@ catalog; the bundle keeps using the vector `icon.svg` for its in-app tile.
|
|
|
56
57
|
```powershell
|
|
57
58
|
npm login # once per machine
|
|
58
59
|
npm whoami # must print your npm user name
|
|
59
|
-
npm view dsh-report-card #
|
|
60
|
+
npm view dsh-report-card # 404 before the very first publish; already published as 1.0.0
|
|
60
61
|
npm test # unit + end-to-end; prepublishOnly also runs the units
|
|
61
62
|
npm publish --access public
|
|
62
63
|
```
|
package/harness/auto.mjs
ADDED
|
@@ -0,0 +1,220 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Automatic measurement of newly installed plugins.
|
|
3
|
+
*
|
|
4
|
+
* The trigger is the plugin manager's own event stream plus the profile
|
|
5
|
+
* manifest, so the card follows whatever path installed the bundle: the
|
|
6
|
+
* Plugins page, the `plugin_manager` tool, or a terminal `dsh plugin add`.
|
|
7
|
+
* Measurements run one at a time in the background; each is the same
|
|
8
|
+
* throwaway-profile measurement `report_card` performs, so nothing about the
|
|
9
|
+
* live profile is touched.
|
|
10
|
+
*/
|
|
11
|
+
import fs from 'node:fs'
|
|
12
|
+
|
|
13
|
+
/** Bundles worth measuring: third-party, installed, readable, not our own. */
|
|
14
|
+
export function eligibleBundles(bundles, { self = 'dsh-report-card' } = {}) {
|
|
15
|
+
return (bundles ?? []).filter((bundle) => bundle !== null
|
|
16
|
+
&& typeof bundle === 'object'
|
|
17
|
+
&& bundle.installed === true
|
|
18
|
+
&& bundle.availability === 'profile'
|
|
19
|
+
&& bundle.official !== true
|
|
20
|
+
&& bundle.optional !== true
|
|
21
|
+
&& bundle.error === undefined
|
|
22
|
+
&& bundle.name !== self
|
|
23
|
+
&& typeof bundle.version === 'string'
|
|
24
|
+
&& bundle.version !== '')
|
|
25
|
+
}
|
|
26
|
+
|
|
27
|
+
/**
|
|
28
|
+
* The spec to measure for one installed bundle.
|
|
29
|
+
*
|
|
30
|
+
* Registry dependencies are pinned to the installed version, not to whatever
|
|
31
|
+
* range the profile saved. Local, link, and git sources are used verbatim,
|
|
32
|
+
* because that is the exact code the user installed.
|
|
33
|
+
*/
|
|
34
|
+
export function measureSpecOf(bundle) {
|
|
35
|
+
const source = typeof bundle?.source === 'string' ? bundle.source.trim() : ''
|
|
36
|
+
const local = /^(file:|link:|git\+|github:|https?:|\.{1,2}[\\/]|[A-Za-z]:[\\/])/.test(source)
|
|
37
|
+
if (source !== '' && local) return source
|
|
38
|
+
return `${bundle.name}@${bundle.version}`
|
|
39
|
+
}
|
|
40
|
+
|
|
41
|
+
/** Stable key for one measured package version. */
|
|
42
|
+
export function bundleKey(bundle) {
|
|
43
|
+
return `${bundle?.name}@${bundle?.version}`
|
|
44
|
+
}
|
|
45
|
+
|
|
46
|
+
/**
|
|
47
|
+
* A serial queue with deduplication.
|
|
48
|
+
*
|
|
49
|
+
* @param options `{ measure, log }` — `measure(candidate)` resolves to a
|
|
50
|
+
* generated report card; errors are contained and reported, never thrown
|
|
51
|
+
* into the harness that scheduled them.
|
|
52
|
+
*/
|
|
53
|
+
export function createAutoQueue({ measure, log = () => {}, concurrency = 1 }) {
|
|
54
|
+
const pending = []
|
|
55
|
+
const known = new Set()
|
|
56
|
+
const results = []
|
|
57
|
+
let running = 0
|
|
58
|
+
let stopped = false
|
|
59
|
+
|
|
60
|
+
const pump = () => {
|
|
61
|
+
if (stopped) return
|
|
62
|
+
while (running < concurrency && pending.length > 0) {
|
|
63
|
+
const candidate = pending.shift()
|
|
64
|
+
running += 1
|
|
65
|
+
Promise.resolve()
|
|
66
|
+
.then(() => measure(candidate))
|
|
67
|
+
.then((card) => {
|
|
68
|
+
const entry = {
|
|
69
|
+
name: candidate.name,
|
|
70
|
+
version: candidate.version,
|
|
71
|
+
grade: card?.report?.grades?.overall ?? null,
|
|
72
|
+
digest: card?.report?.digest ?? null,
|
|
73
|
+
json: card?.paths?.json ?? null,
|
|
74
|
+
markdown: card?.paths?.markdown ?? null,
|
|
75
|
+
}
|
|
76
|
+
results.push(entry)
|
|
77
|
+
log(`measured ${bundleKey(candidate)} (grade ${entry.grade ?? '?'})`)
|
|
78
|
+
}, (error) => {
|
|
79
|
+
results.push({ name: candidate.name, version: candidate.version, error: String(error?.message ?? error) })
|
|
80
|
+
log(`failed ${bundleKey(candidate)}: ${String(error?.message ?? error)}`)
|
|
81
|
+
})
|
|
82
|
+
.finally(() => {
|
|
83
|
+
running -= 1
|
|
84
|
+
pump()
|
|
85
|
+
})
|
|
86
|
+
}
|
|
87
|
+
}
|
|
88
|
+
|
|
89
|
+
return {
|
|
90
|
+
/** Queue candidates not already queued or measured. Returns their keys. */
|
|
91
|
+
enqueue(candidates) {
|
|
92
|
+
const added = []
|
|
93
|
+
for (const candidate of candidates ?? []) {
|
|
94
|
+
const key = bundleKey(candidate)
|
|
95
|
+
if (known.has(key)) continue
|
|
96
|
+
known.add(key)
|
|
97
|
+
pending.push(candidate)
|
|
98
|
+
added.push(key)
|
|
99
|
+
}
|
|
100
|
+
if (added.length > 0) {
|
|
101
|
+
log(`queued ${added.join(', ')}`)
|
|
102
|
+
pump()
|
|
103
|
+
}
|
|
104
|
+
return added
|
|
105
|
+
},
|
|
106
|
+
/** Mark candidates as handled without measuring (startup baseline). */
|
|
107
|
+
markKnown(candidates) {
|
|
108
|
+
for (const candidate of candidates ?? []) known.add(bundleKey(candidate))
|
|
109
|
+
},
|
|
110
|
+
/** Whether this package version was queued or marked already. */
|
|
111
|
+
knows(bundle) {
|
|
112
|
+
return known.has(bundleKey(bundle))
|
|
113
|
+
},
|
|
114
|
+
stop() {
|
|
115
|
+
stopped = true
|
|
116
|
+
pending.length = 0
|
|
117
|
+
},
|
|
118
|
+
status() {
|
|
119
|
+
return { pending: pending.length, running, known: [...known], results: [...results] }
|
|
120
|
+
},
|
|
121
|
+
}
|
|
122
|
+
}
|
|
123
|
+
|
|
124
|
+
/**
|
|
125
|
+
* Start the automatic measurer.
|
|
126
|
+
*
|
|
127
|
+
* @param options `{ manager, reportsDir, samples, backfill, generate, log, initialDelayMs, debounceMs, profileDir }`
|
|
128
|
+
* @returns a handle with `sweep()`, `status()`, and `stop()`
|
|
129
|
+
*/
|
|
130
|
+
export function startAutoMeasurer({
|
|
131
|
+
manager,
|
|
132
|
+
reportsDir,
|
|
133
|
+
samples = 3,
|
|
134
|
+
backfill = false,
|
|
135
|
+
generate,
|
|
136
|
+
log = () => {},
|
|
137
|
+
initialDelayMs = 10_000,
|
|
138
|
+
debounceMs = 2500,
|
|
139
|
+
profileDir,
|
|
140
|
+
watch = true,
|
|
141
|
+
}) {
|
|
142
|
+
const queue = createAutoQueue({
|
|
143
|
+
log,
|
|
144
|
+
measure: async (candidate) => {
|
|
145
|
+
const result = await generate({
|
|
146
|
+
spec: measureSpecOf(candidate),
|
|
147
|
+
samples,
|
|
148
|
+
save: true,
|
|
149
|
+
outDir: reportsDir,
|
|
150
|
+
log: () => {},
|
|
151
|
+
})
|
|
152
|
+
log(`card saved for ${bundleKey(candidate)}: ${result?.paths?.json ?? reportsDir}`)
|
|
153
|
+
return result
|
|
154
|
+
},
|
|
155
|
+
})
|
|
156
|
+
|
|
157
|
+
let initialised = false
|
|
158
|
+
let timer
|
|
159
|
+
let disposed = false
|
|
160
|
+
let watcher
|
|
161
|
+
|
|
162
|
+
const schedule = (delayMs = debounceMs) => {
|
|
163
|
+
if (disposed) return
|
|
164
|
+
if (timer !== undefined) clearTimeout(timer)
|
|
165
|
+
timer = setTimeout(() => {
|
|
166
|
+
timer = undefined
|
|
167
|
+
sweep().catch((error) => log(`sweep failed: ${String(error?.message ?? error)}`))
|
|
168
|
+
}, delayMs)
|
|
169
|
+
if (typeof timer?.unref === 'function') timer.unref()
|
|
170
|
+
}
|
|
171
|
+
|
|
172
|
+
const sweep = async () => {
|
|
173
|
+
if (disposed) return
|
|
174
|
+
const bundles = eligibleBundles(await manager.listBundles())
|
|
175
|
+
if (!initialised) {
|
|
176
|
+
initialised = true
|
|
177
|
+
if (!backfill) {
|
|
178
|
+
// The first scan is the baseline: plugins that were already installed
|
|
179
|
+
// are not re-measured just because the harness restarted.
|
|
180
|
+
queue.markKnown(bundles)
|
|
181
|
+
log(`baseline: ${bundles.length} installed third-party bundle(s) known`)
|
|
182
|
+
return
|
|
183
|
+
}
|
|
184
|
+
}
|
|
185
|
+
const fresh = bundles.filter((bundle) => !queue.knows(bundle))
|
|
186
|
+
if (fresh.length > 0) queue.enqueue(fresh)
|
|
187
|
+
}
|
|
188
|
+
|
|
189
|
+
// A profile manifest change is the one signal both the manager and
|
|
190
|
+
// `dsh plugin add` produce; the manager's own events make it immediate.
|
|
191
|
+
if (watch && typeof profileDir === 'string' && profileDir !== '') {
|
|
192
|
+
try {
|
|
193
|
+
watcher = fs.watch(profileDir, { persistent: false }, (_event, filename) => {
|
|
194
|
+
if (filename !== null && filename !== 'package.json' && filename !== 'cordis.patch.yml') return
|
|
195
|
+
schedule()
|
|
196
|
+
})
|
|
197
|
+
watcher.on?.('error', () => {})
|
|
198
|
+
} catch (error) {
|
|
199
|
+
log(`manifest watch unavailable: ${String(error?.message ?? error)}`)
|
|
200
|
+
}
|
|
201
|
+
}
|
|
202
|
+
|
|
203
|
+
const initial = setTimeout(() => {
|
|
204
|
+
sweep().catch((error) => log(`initial sweep failed: ${String(error?.message ?? error)}`))
|
|
205
|
+
}, initialDelayMs)
|
|
206
|
+
if (typeof initial?.unref === 'function') initial.unref()
|
|
207
|
+
|
|
208
|
+
return {
|
|
209
|
+
sweep,
|
|
210
|
+
schedule,
|
|
211
|
+
status: () => queue.status(),
|
|
212
|
+
stop() {
|
|
213
|
+
disposed = true
|
|
214
|
+
clearTimeout(initial)
|
|
215
|
+
if (timer !== undefined) clearTimeout(timer)
|
|
216
|
+
watcher?.close?.()
|
|
217
|
+
queue.stop()
|
|
218
|
+
},
|
|
219
|
+
}
|
|
220
|
+
}
|
package/index.js
CHANGED
|
@@ -1,21 +1,60 @@
|
|
|
1
1
|
/**
|
|
2
2
|
* DSH Report Card — an independent, dynamic quality benchmark for DSH plugins.
|
|
3
3
|
*
|
|
4
|
-
* The plugin contributes
|
|
5
|
-
* measurement happens in disposable child processes: a fresh `DSH_HOME`, a
|
|
6
|
-
* fresh profile copied from the shipped `headless` template, and a local
|
|
7
|
-
* fixture endpoint instead of a real model. The plugin under test never runs
|
|
8
|
-
* in this process, and never touches the profile that is serving you.
|
|
4
|
+
* The plugin contributes two tools and one background behavior:
|
|
9
5
|
*
|
|
10
|
-
*
|
|
11
|
-
*
|
|
6
|
+
* report_card measure a plugin in a throwaway profile, return a card
|
|
7
|
+
* report_cards_recent list the cards on disk, including automatic ones
|
|
8
|
+
* auto-measure after a plugin is installed, measure it in the
|
|
9
|
+
* background and save its card (configurable)
|
|
10
|
+
*
|
|
11
|
+
* The measurement always happens in disposable child processes: a fresh
|
|
12
|
+
* `DSH_HOME`, a fresh profile copied from the shipped `headless` template, and
|
|
13
|
+
* a local fixture endpoint instead of a real model. The plugin under test never
|
|
14
|
+
* runs in this process, and never touches the profile that is serving you.
|
|
12
15
|
*/
|
|
16
|
+
import os from 'node:os'
|
|
17
|
+
import path from 'node:path'
|
|
18
|
+
import { startAutoMeasurer } from './harness/auto.mjs'
|
|
13
19
|
import { generateReport } from './harness/generate.mjs'
|
|
20
|
+
import { readIndex, resolveReportsDir } from './harness/store.mjs'
|
|
14
21
|
|
|
15
22
|
export const name = 'dsh-report-card'
|
|
16
23
|
|
|
17
24
|
export const inject = ['tools']
|
|
18
25
|
|
|
26
|
+
/** Defaults for the automatic behavior; every field is overridable in config. */
|
|
27
|
+
const DEFAULTS = {
|
|
28
|
+
autoMeasure: true,
|
|
29
|
+
autoSamples: 3,
|
|
30
|
+
autoBackfill: false,
|
|
31
|
+
autoReportsDir: null,
|
|
32
|
+
}
|
|
33
|
+
|
|
34
|
+
/** The tool description is the model's only contract; keep the cost visible. */
|
|
35
|
+
const DESCRIPTION = [
|
|
36
|
+
'Benchmark one DSH plugin or bundle and return a report card: boot-time delta,',
|
|
37
|
+
'per-turn hook overhead, context tax (tokens the plugin adds to the assembled',
|
|
38
|
+
'prompt), capability surface, and static risk flags. It installs the plugin into',
|
|
39
|
+
'a throwaway profile with a fresh DSH_HOME and measures real harness processes,',
|
|
40
|
+
'so it takes roughly 60-90 seconds for the default five samples (pass samples: 3',
|
|
41
|
+
'for a quick look). Use it before installing an unknown plugin, or to compare a',
|
|
42
|
+
'plugin version against its previous card. The card is saved under ./report-cards',
|
|
43
|
+
'and is reproducible from the command it prints. Newly installed plugins are also',
|
|
44
|
+
'measured automatically in the background; report_cards_recent lists those cards.',
|
|
45
|
+
].join(' ')
|
|
46
|
+
|
|
47
|
+
/**
|
|
48
|
+
* Measurements are serialized: an automatic card and a requested card must not
|
|
49
|
+
* compete for the machine, or both numbers become noise.
|
|
50
|
+
*/
|
|
51
|
+
let measurementChain = Promise.resolve()
|
|
52
|
+
function serialize(task) {
|
|
53
|
+
const run = measurementChain.then(task, task)
|
|
54
|
+
measurementChain = run.then(() => {}, () => {})
|
|
55
|
+
return run
|
|
56
|
+
}
|
|
57
|
+
|
|
19
58
|
/**
|
|
20
59
|
* The directory relative specs resolve against.
|
|
21
60
|
*
|
|
@@ -40,19 +79,8 @@ async function sessionCwd(ctx, exec) {
|
|
|
40
79
|
return process.cwd()
|
|
41
80
|
}
|
|
42
81
|
|
|
43
|
-
/**
|
|
44
|
-
|
|
45
|
-
'Benchmark one DSH plugin or bundle and return a report card: boot-time delta,',
|
|
46
|
-
'per-turn hook overhead, context tax (tokens the plugin adds to the assembled',
|
|
47
|
-
'prompt), capability surface, and static risk flags. It installs the plugin into',
|
|
48
|
-
'a throwaway profile with a fresh DSH_HOME and measures real harness processes,',
|
|
49
|
-
'so it takes roughly 60-90 seconds for the default five samples (pass samples: 3',
|
|
50
|
-
'for a quick look). Use it before installing an unknown plugin, or to compare a',
|
|
51
|
-
'plugin version against its previous card. The card is saved under ./report-cards',
|
|
52
|
-
'and is reproducible from the command it prints.',
|
|
53
|
-
].join(' ')
|
|
54
|
-
|
|
55
|
-
export function apply(ctx) {
|
|
82
|
+
/** Register `report_card`. */
|
|
83
|
+
function registerReportCard(ctx) {
|
|
56
84
|
ctx.effect(() => ctx.tools.register({
|
|
57
85
|
name: 'report_card',
|
|
58
86
|
description: DESCRIPTION,
|
|
@@ -90,7 +118,7 @@ export function apply(ctx) {
|
|
|
90
118
|
// Clamp here too: a schema minimum is advisory on some tool pipelines,
|
|
91
119
|
// and fewer than three samples cannot estimate the machine's noise.
|
|
92
120
|
const samples = Math.min(10, Math.max(3, Math.trunc(args.samples ?? 5)))
|
|
93
|
-
const result = await generateReport({
|
|
121
|
+
const result = await serialize(() => generateReport({
|
|
94
122
|
spec: args.pluginSpec,
|
|
95
123
|
samples,
|
|
96
124
|
task: args.task,
|
|
@@ -98,10 +126,114 @@ export function apply(ctx) {
|
|
|
98
126
|
cwd,
|
|
99
127
|
signal: exec?.signal,
|
|
100
128
|
log: (line) => ctx.logger?.info?.(`report-card: ${line}`),
|
|
101
|
-
})
|
|
129
|
+
}))
|
|
102
130
|
// The card already carries its saved paths, reproducer, and digest.
|
|
103
131
|
return result.markdown
|
|
104
132
|
},
|
|
105
133
|
}))
|
|
106
134
|
}
|
|
107
135
|
|
|
136
|
+
/** Format the last cards for the model, newest first. */
|
|
137
|
+
export function formatRecent(index, limit) {
|
|
138
|
+
const rows = [...(index?.reports ?? [])].slice(-limit).reverse()
|
|
139
|
+
if (rows.length === 0) {
|
|
140
|
+
return 'No report cards yet. Cards are written to ./report-cards (or the configured directory) and appear here after a measurement.'
|
|
141
|
+
}
|
|
142
|
+
const lines = [`${rows.length} most recent report card${rows.length === 1 ? '' : 's'}:`]
|
|
143
|
+
for (const row of rows) {
|
|
144
|
+
const boot = Number.isFinite(row.bootDeltaMs) ? `${Math.round(row.bootDeltaMs)} ms boot CPU` : 'no boot reading'
|
|
145
|
+
const context = Number.isFinite(row.contextDelta) ? `${Math.round(row.contextDelta)} tok context` : 'no context reading'
|
|
146
|
+
lines.push(`- ${row.subject} — grade ${row.grade}, ${boot}, ${context}, ${row.generatedAt}${row.json ? ` (${row.json})` : ''}`)
|
|
147
|
+
}
|
|
148
|
+
return lines.join('\n')
|
|
149
|
+
}
|
|
150
|
+
|
|
151
|
+
/** Register `report_cards_recent`. */
|
|
152
|
+
function registerRecentTool(ctx, reportsDirFor) {
|
|
153
|
+
ctx.effect(() => ctx.tools.register({
|
|
154
|
+
name: 'report_cards_recent',
|
|
155
|
+
description: 'List the most recent DSH report cards on disk, including cards produced automatically after a plugin was installed. Returns one line per card: subject, grade, boot and context deltas, time, and the JSON path.',
|
|
156
|
+
parameters: {
|
|
157
|
+
type: 'object',
|
|
158
|
+
properties: {
|
|
159
|
+
limit: {
|
|
160
|
+
type: 'integer',
|
|
161
|
+
minimum: 1,
|
|
162
|
+
maximum: 50,
|
|
163
|
+
description: 'how many cards to list, newest first (default 10)',
|
|
164
|
+
},
|
|
165
|
+
},
|
|
166
|
+
additionalProperties: false,
|
|
167
|
+
},
|
|
168
|
+
output: {
|
|
169
|
+
schema: { type: 'string' },
|
|
170
|
+
render: (_args, value) => [{ type: 'text', text: value }],
|
|
171
|
+
},
|
|
172
|
+
execute(args) {
|
|
173
|
+
const limit = Math.min(50, Math.max(1, Math.trunc(args.limit ?? 10)))
|
|
174
|
+
return formatRecent(readIndex(reportsDirFor()), limit)
|
|
175
|
+
},
|
|
176
|
+
}))
|
|
177
|
+
}
|
|
178
|
+
|
|
179
|
+
/** Start automatic measurement of newly installed plugins. */
|
|
180
|
+
function startAuto(ctx, settings, configuredDir) {
|
|
181
|
+
const log = (line) => ctx.logger?.info?.(`report-card auto: ${line}`)
|
|
182
|
+
ctx.inject(['pluginManager'], (sub) => {
|
|
183
|
+
const samples = Math.min(10, Math.max(3, Math.trunc(Number(settings.autoSamples) || 3)))
|
|
184
|
+
const manager = sub.pluginManager
|
|
185
|
+
const profileDir = typeof manager?.profile?.dir === 'string' ? manager.profile.dir : process.env.DSH_PROFILE_DIR
|
|
186
|
+
const reportsDir = configuredDir ?? dirFromProfile(profileDir)
|
|
187
|
+
const instrument = startAutoMeasurer({
|
|
188
|
+
manager,
|
|
189
|
+
reportsDir,
|
|
190
|
+
samples,
|
|
191
|
+
backfill: settings.autoBackfill === true,
|
|
192
|
+
profileDir,
|
|
193
|
+
generate: (options) => serialize(() => generateReport(options)),
|
|
194
|
+
log,
|
|
195
|
+
})
|
|
196
|
+
sub.effect(() => () => instrument.stop())
|
|
197
|
+
sub.on('plugin-manager/operation', (operation) => {
|
|
198
|
+
if (operation?.reason !== 'install' || operation?.phase !== 'settled') return
|
|
199
|
+
if (operation.application === 'failed' || operation.application === 'cancelled') return
|
|
200
|
+
instrument.schedule()
|
|
201
|
+
})
|
|
202
|
+
sub.on('plugin-manager/changed', () => instrument.schedule())
|
|
203
|
+
})
|
|
204
|
+
}
|
|
205
|
+
|
|
206
|
+
/** Where automatic cards live: the harness home, never the profile directory. */
|
|
207
|
+
function dirFromProfile(profileDir) {
|
|
208
|
+
if (typeof process.env.DSH_HOME === 'string' && process.env.DSH_HOME !== '') {
|
|
209
|
+
return path.join(process.env.DSH_HOME, 'report-cards')
|
|
210
|
+
}
|
|
211
|
+
if (typeof profileDir === 'string' && profileDir !== '') {
|
|
212
|
+
// <home>/profiles/<name> -> <home>
|
|
213
|
+
return path.resolve(profileDir, '..', '..', 'report-cards')
|
|
214
|
+
}
|
|
215
|
+
return path.join(os.homedir(), 'report-cards')
|
|
216
|
+
}
|
|
217
|
+
|
|
218
|
+
/** Resolve the reports directory for the session tools, preferring the manager's profile. */
|
|
219
|
+
function reportsDirFor(ctx, configured) {
|
|
220
|
+
if (configured !== undefined && configured !== null) return configured
|
|
221
|
+
try {
|
|
222
|
+
const manager = typeof ctx.reflect?.get === 'function' ? ctx.reflect.get('pluginManager') : undefined
|
|
223
|
+
if (typeof manager?.profile?.dir === 'string') return dirFromProfile(manager.profile.dir)
|
|
224
|
+
} catch {
|
|
225
|
+
// fall through to the environment
|
|
226
|
+
}
|
|
227
|
+
return dirFromProfile(process.env.DSH_PROFILE_DIR)
|
|
228
|
+
}
|
|
229
|
+
|
|
230
|
+
export function apply(ctx, config = {}) {
|
|
231
|
+
const settings = { ...DEFAULTS, ...(config ?? {}) }
|
|
232
|
+
const configuredDir = settings.autoReportsDir ?? undefined
|
|
233
|
+
registerReportCard(ctx)
|
|
234
|
+
registerRecentTool(ctx, () => reportsDirFor(ctx, configuredDir))
|
|
235
|
+
const disabled = settings.autoMeasure === false || process.env.DSH_REPORT_CARD_AUTO === '0'
|
|
236
|
+
if (!disabled) startAuto(ctx, settings, configuredDir)
|
|
237
|
+
}
|
|
238
|
+
|
|
239
|
+
export { resolveReportsDir }
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "dsh-report-card",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.1.0",
|
|
4
4
|
"type": "module",
|
|
5
5
|
"description": "Independent dynamic benchmark for DSH plugins: boot delta, per-turn hook overhead, context tax, capability surface and risk flags, measured in a throwaway profile.",
|
|
6
6
|
"keywords": [
|
|
@@ -43,9 +43,9 @@
|
|
|
43
43
|
"dsh": ">=0.2.2-rc.1 <0.3.0"
|
|
44
44
|
},
|
|
45
45
|
"scripts": {
|
|
46
|
-
"test": "node test/unit.test.mjs && node test/harness.test.mjs",
|
|
46
|
+
"test": "node test/unit.test.mjs && node test/auto.test.mjs && node test/harness.test.mjs",
|
|
47
47
|
"card": "node harness/cli.mjs",
|
|
48
|
-
"prepublishOnly": "node test/unit.test.mjs"
|
|
48
|
+
"prepublishOnly": "node test/unit.test.mjs && node test/auto.test.mjs"
|
|
49
49
|
},
|
|
50
50
|
"dsh": {
|
|
51
51
|
"manifestVersion": 1,
|