dsh-report-card 1.0.0 → 1.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,4 +1,4 @@
1
- # DSH Report Card
1
+ # DSH Report Card
2
2
 
3
3
  An independent, **dynamic** benchmark for DeepSeek Harness plugins. It installs
4
4
  one plugin into a throwaway profile, boots the real harness with it, runs one
@@ -87,20 +87,26 @@ test fixture.
87
87
 
88
88
  ## Install
89
89
 
90
- The plugin ships as a bundle with metadata, `locale/en.json`, `locale/zh.json`
91
- and an icon. Install it from this directory with the plugin manager — it adds
90
+ Published on npm as **`dsh-report-card@1.0.0`**. It ships as a bundle with
91
+ metadata, `locale/en.json`, `locale/zh.json` and an icon. The plugin manager adds
92
92
  the dependency *and* enables the bundle in one step; removing it undoes both:
93
93
 
94
94
  ```
95
- plugin_manager install_bundle target: "file:C:/path/to/dsh-report-card"
95
+ plugin_manager install_bundle target: "dsh-report-card@1.0.0"
96
96
  plugin_manager remove_bundle target: "dsh-report-card"
97
97
  ```
98
98
 
99
- `install_bundle` is the sanctioned path because a bundle is composed through the
100
- profile's bundle list, not just `node_modules`. The bundle inserts one entry,
101
- `dsh-report-card`, which contributes one tool. It was installed this way on dsh
102
- 0.2.2-rc.1 and appears in the profile's bundle list beside the existing plugins
103
- without touching them.
99
+ or from a terminal:
100
+
101
+ ```powershell
102
+ dsh plugin --profile web add dsh-report-card
103
+ ```
104
+
105
+ For development from a checkout, pass a directory spec instead
106
+ (`file:C:/path/to/dsh-report-card`). `install_bundle` is the sanctioned path
107
+ because a bundle is composed through the profile's bundle list, not just
108
+ `node_modules`; the bundle inserts one entry, `dsh-report-card`, which
109
+ contributes one tool.
104
110
 
105
111
  ## Use
106
112
 
@@ -108,8 +114,12 @@ From a session (the tool is model-facing):
108
114
 
109
115
  ```
110
116
  report_card(pluginSpec: "dsh-trust-check@0.2.6", samples: 3)
117
+ report_cards_recent(limit: 10) # the last cards on disk, including automatic ones
111
118
  ```
112
119
 
120
+ `report_card` measures on demand; `report_cards_recent` reads the card index so
121
+ you can see what the background pass produced.
122
+
113
123
  From a terminal — the same code path, printable output:
114
124
 
115
125
  ```powershell
@@ -126,6 +136,48 @@ sample round (surface × 2, turn × 2) plus one discarded warm-up round. Pass
126
136
  `samples: 3` for a quick look and `samples: 8..10` when you need to resolve a
127
137
  few milliseconds.
128
138
 
139
+ ## Automatic measurement
140
+
141
+ Since 1.1.0 the card is produced **after a plugin is installed**, without
142
+ anyone asking. The plugin listens to the plugin manager's
143
+ `plugin-manager/operation` event and watches the profile manifest, queues the
144
+ new package, and measures it in the background — the same throwaway profile,
145
+ three samples, one measurement at a time.
146
+
147
+ ```
148
+ install a plugin
149
+ │ manager event (Plugins page, plugin_manager tool)
150
+ │ or profile package.json change (dsh plugin add)
151
+ ▼
152
+ debounce 2.5 s → list installed third-party bundles → skip ones already carded
153
+ ▼
154
+ queue → measure in a throwaway profile → save card under $DSH_HOME/report-cards
155
+ ▼
156
+ report_cards_recent lists it
157
+ ```
158
+
159
+ What it skips: shipped/optional/official bundles, bundles that failed to load,
160
+ its own package, and any `name@version` that already has a card. What it does
161
+ **not** do: measure plugins that were already installed before this version
162
+ started, unless `autoBackfill` is on.
163
+
164
+ Automatic cards take roughly 45 s each (three samples) and run serially; a
165
+ manual `report_card` call queues behind a running automatic card so the two
166
+ never compete for the machine.
167
+
168
+ Config, in the profile patch (or the plugin's config in a preset):
169
+
170
+ ```yaml
171
+ - id: dsh-report-card
172
+ config:
173
+ autoMeasure: true # false turns the background behavior off
174
+ autoSamples: 3 # 3..10; steadier numbers cost more time
175
+ autoBackfill: false # true: card already-installed plugins at startup
176
+ autoReportsDir: null # default: <DSH_HOME>/report-cards
177
+ ```
178
+
179
+ `DSH_REPORT_CARD_AUTO=0` disables it for one process without touching config.
180
+
129
181
  ## Reading a card
130
182
 
131
183
  ```
@@ -177,6 +229,7 @@ Tests:
177
229
 
178
230
  ```powershell
179
231
  node test/unit.test.mjs # units, ~1 s
232
+ node test/auto.test.mjs # install trigger and queue, ~1 s
180
233
  node test/harness.test.mjs # real boots against fixtures, ~80 s
181
234
  ```
182
235
 
@@ -227,6 +280,7 @@ check that was run against the packed artifact.
227
280
  | `harness/risk.mjs` | Static disclosure scan |
228
281
  | `harness/report.mjs` | Grades, digest, drift, card assembly |
229
282
  | `harness/render.mjs` | Markdown card |
283
+ | `harness/auto.mjs` | Install trigger, eligibility, and the background measurement queue |
230
284
  | `harness/repeat.mjs` | Repeatability check for the harness itself |
231
285
  | `docs/` | An example card and the acceptance evidence |
232
286
  | `locale/`, `icon.svg`, `cordis.patch.yml` | Bundle metadata and its one insert row |
@@ -1,4 +1,4 @@
1
- # Report card — @local/rc-fake-plugin@0.1.0
1
+ # Report card — @local/rc-fake-plugin@0.1.0
2
2
 
3
3
  **Grade D** · graded boot +6.3 ms CPU · graded turn 0 ms · context +76 tok · risk D
4
4
 
@@ -11,23 +11,24 @@ There are three publication surfaces, and they depend on each other in order:
11
11
  3. **Your own index (optional)** — a format-1 JSON document you host; users add
12
12
  it as an index source in Plugins → Discover.
13
13
 
14
- The rest of this file is the exact sequence for this package. Current state on
15
- this machine (checked 2026-11-10):
14
+ The rest of this file is the exact sequence for this package. Release state
15
+ (2026-11-10):
16
16
 
17
- | Check | Result |
17
+ | Surface | State |
18
18
  |---|---|
19
- | `npm view dsh-report-card` | 404 — the name is free |
20
- | `npm whoami` | 401 — no npm credentials configured here; run `npm login` |
21
- | `git remote -v` | `github.com/tespio/dsh-report-card` (created for this release) |
22
- | `npm pack` | 24 files, 40.9 kB packed, 128.5 kB unpacked, probe included |
23
- | Install of the packed tarball in a throwaway profile | works; `report_card` registered, entry active, context tax +344 tok |
19
+ | npm | `dsh-report-card@1.0.0` published (publisher `init3`), 25 files, 139 kB unpacked |
20
+ | GitHub | <https://github.com/tespio/dsh-report-card>, tag `v1.0.0` |
21
+ | Registry acceptance | `node harness/cli.mjs dsh-report-card@1.0.0 --samples 3` installs from npm, registers `report_card`, entry `active` |
22
+ | Live profile | installed from the registry (`"dsh-report-card": "1.0.0"`), entry active |
23
+ | Catalog | **pending** — add `data/plugins/tespio__dsh-report-card.yml` from [`catalog-entry.yml`](catalog-entry.yml) |
24
24
 
25
25
  ## 0. Create the repository
26
26
 
27
- The catalog entry points at a `https://github.com/owner/repo` URL, and the
28
- catalog's npm probe only maps a package whose own `repository` field points back
29
- at that repo. So the repo is not optional bookkeeping — it is the link between
30
- the catalog entry and the npm package.
27
+ Already done for 1.0.0; kept as the initial-setup reference. The catalog entry
28
+ points at a `https://github.com/owner/repo` URL, and the catalog's npm probe only
29
+ maps a package whose own `repository` field points back at that repo. So the repo
30
+ is not optional bookkeeping — it is the link between the catalog entry and the
31
+ npm package.
31
32
 
32
33
  ```powershell
33
34
  cd dsh-report-card
@@ -56,7 +57,7 @@ catalog; the bundle keeps using the vector `icon.svg` for its in-app tile.
56
57
  ```powershell
57
58
  npm login # once per machine
58
59
  npm whoami # must print your npm user name
59
- npm view dsh-report-card # must still be 404 before the first publish
60
+ npm view dsh-report-card # 404 before the very first publish; already published as 1.0.0
60
61
  npm test # unit + end-to-end; prepublishOnly also runs the units
61
62
  npm publish --access public
62
63
  ```
@@ -0,0 +1,220 @@
1
+ /**
2
+ * Automatic measurement of newly installed plugins.
3
+ *
4
+ * The trigger is the plugin manager's own event stream plus the profile
5
+ * manifest, so the card follows whatever path installed the bundle: the
6
+ * Plugins page, the `plugin_manager` tool, or a terminal `dsh plugin add`.
7
+ * Measurements run one at a time in the background; each is the same
8
+ * throwaway-profile measurement `report_card` performs, so nothing about the
9
+ * live profile is touched.
10
+ */
11
+ import fs from 'node:fs'
12
+
13
+ /** Bundles worth measuring: third-party, installed, readable, not our own. */
14
+ export function eligibleBundles(bundles, { self = 'dsh-report-card' } = {}) {
15
+ return (bundles ?? []).filter((bundle) => bundle !== null
16
+ && typeof bundle === 'object'
17
+ && bundle.installed === true
18
+ && bundle.availability === 'profile'
19
+ && bundle.official !== true
20
+ && bundle.optional !== true
21
+ && bundle.error === undefined
22
+ && bundle.name !== self
23
+ && typeof bundle.version === 'string'
24
+ && bundle.version !== '')
25
+ }
26
+
27
+ /**
28
+ * The spec to measure for one installed bundle.
29
+ *
30
+ * Registry dependencies are pinned to the installed version, not to whatever
31
+ * range the profile saved. Local, link, and git sources are used verbatim,
32
+ * because that is the exact code the user installed.
33
+ */
34
+ export function measureSpecOf(bundle) {
35
+ const source = typeof bundle?.source === 'string' ? bundle.source.trim() : ''
36
+ const local = /^(file:|link:|git\+|github:|https?:|\.{1,2}[\\/]|[A-Za-z]:[\\/])/.test(source)
37
+ if (source !== '' && local) return source
38
+ return `${bundle.name}@${bundle.version}`
39
+ }
40
+
41
+ /** Stable key for one measured package version. */
42
+ export function bundleKey(bundle) {
43
+ return `${bundle?.name}@${bundle?.version}`
44
+ }
45
+
46
+ /**
47
+ * A serial queue with deduplication.
48
+ *
49
+ * @param options `{ measure, log }` — `measure(candidate)` resolves to a
50
+ * generated report card; errors are contained and reported, never thrown
51
+ * into the harness that scheduled them.
52
+ */
53
+ export function createAutoQueue({ measure, log = () => {}, concurrency = 1 }) {
54
+ const pending = []
55
+ const known = new Set()
56
+ const results = []
57
+ let running = 0
58
+ let stopped = false
59
+
60
+ const pump = () => {
61
+ if (stopped) return
62
+ while (running < concurrency && pending.length > 0) {
63
+ const candidate = pending.shift()
64
+ running += 1
65
+ Promise.resolve()
66
+ .then(() => measure(candidate))
67
+ .then((card) => {
68
+ const entry = {
69
+ name: candidate.name,
70
+ version: candidate.version,
71
+ grade: card?.report?.grades?.overall ?? null,
72
+ digest: card?.report?.digest ?? null,
73
+ json: card?.paths?.json ?? null,
74
+ markdown: card?.paths?.markdown ?? null,
75
+ }
76
+ results.push(entry)
77
+ log(`measured ${bundleKey(candidate)} (grade ${entry.grade ?? '?'})`)
78
+ }, (error) => {
79
+ results.push({ name: candidate.name, version: candidate.version, error: String(error?.message ?? error) })
80
+ log(`failed ${bundleKey(candidate)}: ${String(error?.message ?? error)}`)
81
+ })
82
+ .finally(() => {
83
+ running -= 1
84
+ pump()
85
+ })
86
+ }
87
+ }
88
+
89
+ return {
90
+ /** Queue candidates not already queued or measured. Returns their keys. */
91
+ enqueue(candidates) {
92
+ const added = []
93
+ for (const candidate of candidates ?? []) {
94
+ const key = bundleKey(candidate)
95
+ if (known.has(key)) continue
96
+ known.add(key)
97
+ pending.push(candidate)
98
+ added.push(key)
99
+ }
100
+ if (added.length > 0) {
101
+ log(`queued ${added.join(', ')}`)
102
+ pump()
103
+ }
104
+ return added
105
+ },
106
+ /** Mark candidates as handled without measuring (startup baseline). */
107
+ markKnown(candidates) {
108
+ for (const candidate of candidates ?? []) known.add(bundleKey(candidate))
109
+ },
110
+ /** Whether this package version was queued or marked already. */
111
+ knows(bundle) {
112
+ return known.has(bundleKey(bundle))
113
+ },
114
+ stop() {
115
+ stopped = true
116
+ pending.length = 0
117
+ },
118
+ status() {
119
+ return { pending: pending.length, running, known: [...known], results: [...results] }
120
+ },
121
+ }
122
+ }
123
+
124
+ /**
125
+ * Start the automatic measurer.
126
+ *
127
+ * @param options `{ manager, reportsDir, samples, backfill, generate, log, initialDelayMs, debounceMs, profileDir }`
128
+ * @returns a handle with `sweep()`, `status()`, and `stop()`
129
+ */
130
+ export function startAutoMeasurer({
131
+ manager,
132
+ reportsDir,
133
+ samples = 3,
134
+ backfill = false,
135
+ generate,
136
+ log = () => {},
137
+ initialDelayMs = 10_000,
138
+ debounceMs = 2500,
139
+ profileDir,
140
+ watch = true,
141
+ }) {
142
+ const queue = createAutoQueue({
143
+ log,
144
+ measure: async (candidate) => {
145
+ const result = await generate({
146
+ spec: measureSpecOf(candidate),
147
+ samples,
148
+ save: true,
149
+ outDir: reportsDir,
150
+ log: () => {},
151
+ })
152
+ log(`card saved for ${bundleKey(candidate)}: ${result?.paths?.json ?? reportsDir}`)
153
+ return result
154
+ },
155
+ })
156
+
157
+ let initialised = false
158
+ let timer
159
+ let disposed = false
160
+ let watcher
161
+
162
+ const schedule = (delayMs = debounceMs) => {
163
+ if (disposed) return
164
+ if (timer !== undefined) clearTimeout(timer)
165
+ timer = setTimeout(() => {
166
+ timer = undefined
167
+ sweep().catch((error) => log(`sweep failed: ${String(error?.message ?? error)}`))
168
+ }, delayMs)
169
+ if (typeof timer?.unref === 'function') timer.unref()
170
+ }
171
+
172
+ const sweep = async () => {
173
+ if (disposed) return
174
+ const bundles = eligibleBundles(await manager.listBundles())
175
+ if (!initialised) {
176
+ initialised = true
177
+ if (!backfill) {
178
+ // The first scan is the baseline: plugins that were already installed
179
+ // are not re-measured just because the harness restarted.
180
+ queue.markKnown(bundles)
181
+ log(`baseline: ${bundles.length} installed third-party bundle(s) known`)
182
+ return
183
+ }
184
+ }
185
+ const fresh = bundles.filter((bundle) => !queue.knows(bundle))
186
+ if (fresh.length > 0) queue.enqueue(fresh)
187
+ }
188
+
189
+ // A profile manifest change is the one signal both the manager and
190
+ // `dsh plugin add` produce; the manager's own events make it immediate.
191
+ if (watch && typeof profileDir === 'string' && profileDir !== '') {
192
+ try {
193
+ watcher = fs.watch(profileDir, { persistent: false }, (_event, filename) => {
194
+ if (filename !== null && filename !== 'package.json' && filename !== 'cordis.patch.yml') return
195
+ schedule()
196
+ })
197
+ watcher.on?.('error', () => {})
198
+ } catch (error) {
199
+ log(`manifest watch unavailable: ${String(error?.message ?? error)}`)
200
+ }
201
+ }
202
+
203
+ const initial = setTimeout(() => {
204
+ sweep().catch((error) => log(`initial sweep failed: ${String(error?.message ?? error)}`))
205
+ }, initialDelayMs)
206
+ if (typeof initial?.unref === 'function') initial.unref()
207
+
208
+ return {
209
+ sweep,
210
+ schedule,
211
+ status: () => queue.status(),
212
+ stop() {
213
+ disposed = true
214
+ clearTimeout(initial)
215
+ if (timer !== undefined) clearTimeout(timer)
216
+ watcher?.close?.()
217
+ queue.stop()
218
+ },
219
+ }
220
+ }
package/index.js CHANGED
@@ -1,21 +1,60 @@
1
1
  /**
2
2
  * DSH Report Card — an independent, dynamic quality benchmark for DSH plugins.
3
3
  *
4
- * The plugin contributes exactly one model-facing tool, `report_card`. The
5
- * measurement happens in disposable child processes: a fresh `DSH_HOME`, a
6
- * fresh profile copied from the shipped `headless` template, and a local
7
- * fixture endpoint instead of a real model. The plugin under test never runs
8
- * in this process, and never touches the profile that is serving you.
4
+ * The plugin contributes two tools and one background behavior:
9
5
  *
10
- * Every number the tool reports is wall-clock measured, not estimated from
11
- * source code, and every card stores its samples so a re-run can show drift.
6
+ * report_card measure a plugin in a throwaway profile, return a card
7
+ * report_cards_recent list the cards on disk, including automatic ones
8
+ * auto-measure after a plugin is installed, measure it in the
9
+ * background and save its card (configurable)
10
+ *
11
+ * The measurement always happens in disposable child processes: a fresh
12
+ * `DSH_HOME`, a fresh profile copied from the shipped `headless` template, and
13
+ * a local fixture endpoint instead of a real model. The plugin under test never
14
+ * runs in this process, and never touches the profile that is serving you.
12
15
  */
16
+ import os from 'node:os'
17
+ import path from 'node:path'
18
+ import { startAutoMeasurer } from './harness/auto.mjs'
13
19
  import { generateReport } from './harness/generate.mjs'
20
+ import { readIndex, resolveReportsDir } from './harness/store.mjs'
14
21
 
15
22
  export const name = 'dsh-report-card'
16
23
 
17
24
  export const inject = ['tools']
18
25
 
26
+ /** Defaults for the automatic behavior; every field is overridable in config. */
27
+ const DEFAULTS = {
28
+ autoMeasure: true,
29
+ autoSamples: 3,
30
+ autoBackfill: false,
31
+ autoReportsDir: null,
32
+ }
33
+
34
+ /** The tool description is the model's only contract; keep the cost visible. */
35
+ const DESCRIPTION = [
36
+ 'Benchmark one DSH plugin or bundle and return a report card: boot-time delta,',
37
+ 'per-turn hook overhead, context tax (tokens the plugin adds to the assembled',
38
+ 'prompt), capability surface, and static risk flags. It installs the plugin into',
39
+ 'a throwaway profile with a fresh DSH_HOME and measures real harness processes,',
40
+ 'so it takes roughly 60-90 seconds for the default five samples (pass samples: 3',
41
+ 'for a quick look). Use it before installing an unknown plugin, or to compare a',
42
+ 'plugin version against its previous card. The card is saved under ./report-cards',
43
+ 'and is reproducible from the command it prints. Newly installed plugins are also',
44
+ 'measured automatically in the background; report_cards_recent lists those cards.',
45
+ ].join(' ')
46
+
47
+ /**
48
+ * Measurements are serialized: an automatic card and a requested card must not
49
+ * compete for the machine, or both numbers become noise.
50
+ */
51
+ let measurementChain = Promise.resolve()
52
+ function serialize(task) {
53
+ const run = measurementChain.then(task, task)
54
+ measurementChain = run.then(() => {}, () => {})
55
+ return run
56
+ }
57
+
19
58
  /**
20
59
  * The directory relative specs resolve against.
21
60
  *
@@ -40,19 +79,8 @@ async function sessionCwd(ctx, exec) {
40
79
  return process.cwd()
41
80
  }
42
81
 
43
- /** The tool description is the model's only contract; keep the cost visible. */
44
- const DESCRIPTION = [
45
- 'Benchmark one DSH plugin or bundle and return a report card: boot-time delta,',
46
- 'per-turn hook overhead, context tax (tokens the plugin adds to the assembled',
47
- 'prompt), capability surface, and static risk flags. It installs the plugin into',
48
- 'a throwaway profile with a fresh DSH_HOME and measures real harness processes,',
49
- 'so it takes roughly 60-90 seconds for the default five samples (pass samples: 3',
50
- 'for a quick look). Use it before installing an unknown plugin, or to compare a',
51
- 'plugin version against its previous card. The card is saved under ./report-cards',
52
- 'and is reproducible from the command it prints.',
53
- ].join(' ')
54
-
55
- export function apply(ctx) {
82
+ /** Register `report_card`. */
83
+ function registerReportCard(ctx) {
56
84
  ctx.effect(() => ctx.tools.register({
57
85
  name: 'report_card',
58
86
  description: DESCRIPTION,
@@ -90,7 +118,7 @@ export function apply(ctx) {
90
118
  // Clamp here too: a schema minimum is advisory on some tool pipelines,
91
119
  // and fewer than three samples cannot estimate the machine's noise.
92
120
  const samples = Math.min(10, Math.max(3, Math.trunc(args.samples ?? 5)))
93
- const result = await generateReport({
121
+ const result = await serialize(() => generateReport({
94
122
  spec: args.pluginSpec,
95
123
  samples,
96
124
  task: args.task,
@@ -98,10 +126,114 @@ export function apply(ctx) {
98
126
  cwd,
99
127
  signal: exec?.signal,
100
128
  log: (line) => ctx.logger?.info?.(`report-card: ${line}`),
101
- })
129
+ }))
102
130
  // The card already carries its saved paths, reproducer, and digest.
103
131
  return result.markdown
104
132
  },
105
133
  }))
106
134
  }
107
135
 
136
+ /** Format the last cards for the model, newest first. */
137
+ export function formatRecent(index, limit) {
138
+ const rows = [...(index?.reports ?? [])].slice(-limit).reverse()
139
+ if (rows.length === 0) {
140
+ return 'No report cards yet. Cards are written to ./report-cards (or the configured directory) and appear here after a measurement.'
141
+ }
142
+ const lines = [`${rows.length} most recent report card${rows.length === 1 ? '' : 's'}:`]
143
+ for (const row of rows) {
144
+ const boot = Number.isFinite(row.bootDeltaMs) ? `${Math.round(row.bootDeltaMs)} ms boot CPU` : 'no boot reading'
145
+ const context = Number.isFinite(row.contextDelta) ? `${Math.round(row.contextDelta)} tok context` : 'no context reading'
146
+ lines.push(`- ${row.subject} — grade ${row.grade}, ${boot}, ${context}, ${row.generatedAt}${row.json ? ` (${row.json})` : ''}`)
147
+ }
148
+ return lines.join('\n')
149
+ }
150
+
151
+ /** Register `report_cards_recent`. */
152
+ function registerRecentTool(ctx, reportsDirFor) {
153
+ ctx.effect(() => ctx.tools.register({
154
+ name: 'report_cards_recent',
155
+ description: 'List the most recent DSH report cards on disk, including cards produced automatically after a plugin was installed. Returns one line per card: subject, grade, boot and context deltas, time, and the JSON path.',
156
+ parameters: {
157
+ type: 'object',
158
+ properties: {
159
+ limit: {
160
+ type: 'integer',
161
+ minimum: 1,
162
+ maximum: 50,
163
+ description: 'how many cards to list, newest first (default 10)',
164
+ },
165
+ },
166
+ additionalProperties: false,
167
+ },
168
+ output: {
169
+ schema: { type: 'string' },
170
+ render: (_args, value) => [{ type: 'text', text: value }],
171
+ },
172
+ execute(args) {
173
+ const limit = Math.min(50, Math.max(1, Math.trunc(args.limit ?? 10)))
174
+ return formatRecent(readIndex(reportsDirFor()), limit)
175
+ },
176
+ }))
177
+ }
178
+
179
+ /** Start automatic measurement of newly installed plugins. */
180
+ function startAuto(ctx, settings, configuredDir) {
181
+ const log = (line) => ctx.logger?.info?.(`report-card auto: ${line}`)
182
+ ctx.inject(['pluginManager'], (sub) => {
183
+ const samples = Math.min(10, Math.max(3, Math.trunc(Number(settings.autoSamples) || 3)))
184
+ const manager = sub.pluginManager
185
+ const profileDir = typeof manager?.profile?.dir === 'string' ? manager.profile.dir : process.env.DSH_PROFILE_DIR
186
+ const reportsDir = configuredDir ?? dirFromProfile(profileDir)
187
+ const instrument = startAutoMeasurer({
188
+ manager,
189
+ reportsDir,
190
+ samples,
191
+ backfill: settings.autoBackfill === true,
192
+ profileDir,
193
+ generate: (options) => serialize(() => generateReport(options)),
194
+ log,
195
+ })
196
+ sub.effect(() => () => instrument.stop())
197
+ sub.on('plugin-manager/operation', (operation) => {
198
+ if (operation?.reason !== 'install' || operation?.phase !== 'settled') return
199
+ if (operation.application === 'failed' || operation.application === 'cancelled') return
200
+ instrument.schedule()
201
+ })
202
+ sub.on('plugin-manager/changed', () => instrument.schedule())
203
+ })
204
+ }
205
+
206
+ /** Where automatic cards live: the harness home, never the profile directory. */
207
+ function dirFromProfile(profileDir) {
208
+ if (typeof process.env.DSH_HOME === 'string' && process.env.DSH_HOME !== '') {
209
+ return path.join(process.env.DSH_HOME, 'report-cards')
210
+ }
211
+ if (typeof profileDir === 'string' && profileDir !== '') {
212
+ // <home>/profiles/<name> -> <home>
213
+ return path.resolve(profileDir, '..', '..', 'report-cards')
214
+ }
215
+ return path.join(os.homedir(), 'report-cards')
216
+ }
217
+
218
+ /** Resolve the reports directory for the session tools, preferring the manager's profile. */
219
+ function reportsDirFor(ctx, configured) {
220
+ if (configured !== undefined && configured !== null) return configured
221
+ try {
222
+ const manager = typeof ctx.reflect?.get === 'function' ? ctx.reflect.get('pluginManager') : undefined
223
+ if (typeof manager?.profile?.dir === 'string') return dirFromProfile(manager.profile.dir)
224
+ } catch {
225
+ // fall through to the environment
226
+ }
227
+ return dirFromProfile(process.env.DSH_PROFILE_DIR)
228
+ }
229
+
230
+ export function apply(ctx, config = {}) {
231
+ const settings = { ...DEFAULTS, ...(config ?? {}) }
232
+ const configuredDir = settings.autoReportsDir ?? undefined
233
+ registerReportCard(ctx)
234
+ registerRecentTool(ctx, () => reportsDirFor(ctx, configuredDir))
235
+ const disabled = settings.autoMeasure === false || process.env.DSH_REPORT_CARD_AUTO === '0'
236
+ if (!disabled) startAuto(ctx, settings, configuredDir)
237
+ }
238
+
239
+ export { resolveReportsDir }
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "dsh-report-card",
3
- "version": "1.0.0",
3
+ "version": "1.1.0",
4
4
  "type": "module",
5
5
  "description": "Independent dynamic benchmark for DSH plugins: boot delta, per-turn hook overhead, context tax, capability surface and risk flags, measured in a throwaway profile.",
6
6
  "keywords": [
@@ -43,9 +43,9 @@
43
43
  "dsh": ">=0.2.2-rc.1 <0.3.0"
44
44
  },
45
45
  "scripts": {
46
- "test": "node test/unit.test.mjs && node test/harness.test.mjs",
46
+ "test": "node test/unit.test.mjs && node test/auto.test.mjs && node test/harness.test.mjs",
47
47
  "card": "node harness/cli.mjs",
48
- "prepublishOnly": "node test/unit.test.mjs"
48
+ "prepublishOnly": "node test/unit.test.mjs && node test/auto.test.mjs"
49
49
  },
50
50
  "dsh": {
51
51
  "manifestVersion": 1,