dsh-report-card 0.0.0-stage → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +236 -2
- package/cordis.patch.yml +4 -0
- package/docs/acceptance.md +127 -0
- package/docs/example-card.md +106 -0
- package/docs/publishing.md +215 -0
- package/harness/cli.mjs +119 -0
- package/harness/environment.mjs +169 -0
- package/harness/generate.mjs +116 -0
- package/harness/measure.mjs +367 -0
- package/harness/mock-llm.mjs +120 -0
- package/harness/probe/index.js +189 -0
- package/harness/probe/package.json +11 -0
- package/harness/profile.mjs +217 -0
- package/harness/render.mjs +216 -0
- package/harness/repeat.mjs +84 -0
- package/harness/report.mjs +393 -0
- package/harness/risk.mjs +175 -0
- package/harness/store.mjs +72 -0
- package/harness/surface.mjs +106 -0
- package/icon.svg +7 -0
- package/index.js +107 -0
- package/locale/en.json +6 -0
- package/locale/zh.json +6 -0
- package/package.json +57 -4
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 DSH Report Card contributors
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
CHANGED
|
@@ -1,3 +1,237 @@
|
|
|
1
|
-
#
|
|
1
|
+
# DSH Report Card
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
An independent, **dynamic** benchmark for DeepSeek Harness plugins. It installs
|
|
4
|
+
one plugin into a throwaway profile, boots the real harness with it, runs one
|
|
5
|
+
scripted turn against a local fixture endpoint, and reports what the plugin
|
|
6
|
+
costs: boot time, per-turn hook overhead, the tokens it adds to the assembled
|
|
7
|
+
prompt, the capability surface it contributes, and the static risk flags its
|
|
8
|
+
package carries.
|
|
9
|
+
|
|
10
|
+
Every number is a reading from a real harness process — wall clock, process CPU,
|
|
11
|
+
or the exact bytes of the request it sent — none of it estimated from source
|
|
12
|
+
code. Every card stores its samples, its digest, and the command that reproduces
|
|
13
|
+
it.
|
|
14
|
+
|
|
15
|
+
## What already exists, and why this is not a duplicate
|
|
16
|
+
|
|
17
|
+
Searched 2026-11-10: the `dsh-plugin-catalog` 2026.1110.18259 (9,959 plugins)
|
|
18
|
+
and npm. The adjacent tools are all **static scores or in-profile meters**:
|
|
19
|
+
|
|
20
|
+
| Existing | What it actually does | Why it is a different thing |
|
|
21
|
+
|---|---|---|
|
|
22
|
+
| `dsh-plugin-scorecard` 0.3.2 | Static quality/security score, rankings, history across the whole catalog | Scores package *files*; never boots the plugin |
|
|
23
|
+
| `dsh-quality-score` 0.1.4 | Static 0-100 from manifest, peers, dist-tags, docs | Same: a metadata rubric, no runtime |
|
|
24
|
+
| `dsh-trust-check` 0.2.6 | Static capability disclosure with `file:line` evidence | Facts about source, not behaviour under load |
|
|
25
|
+
| `dsh-xray` 0.11.0 | Per-plugin context-tax attribution, per-request token ledger | Prices what is **already loaded in your profile**; cannot evaluate a candidate |
|
|
26
|
+
| `dsh-context-doctor` | Token costs of instruction chains, skill catalogs, tool schemas | Static accounting of the current context |
|
|
27
|
+
| `dsh-agent-bench` 0.1.1 | Replays fixed agent cases and diffs scorecards | Measures *model behaviour*; the plugin is not the subject |
|
|
28
|
+
| `dsh-workspace-switch-cost` | Cold-start cost of switching workspaces | Per-workspace, not per candidate plugin |
|
|
29
|
+
|
|
30
|
+
**One line:** existing scorecards read a plugin's files, and the closest dynamic
|
|
31
|
+
instruments price what is already in your profile — nothing installs a candidate
|
|
32
|
+
into a disposable profile and times its boot, a scripted turn, and the exact
|
|
33
|
+
request it produces; that is what this does, so it complements them.
|
|
34
|
+
|
|
35
|
+
## What a card contains
|
|
36
|
+
|
|
37
|
+
| Dimension | How it is measured |
|
|
38
|
+
|---|---|
|
|
39
|
+
| **Boot wall** | Process start (`performance.timeOrigin`) → the measurement probe reporting the plugin tree composed, with and without the plugin |
|
|
40
|
+
| **Boot CPU** | The child's own CPU time (user + system) at that same dump: what composing the tree cost, independent of what else the machine was doing. This is the graded boot figure |
|
|
41
|
+
| **Launch** | The same point measured from process spawn, including OS process startup |
|
|
42
|
+
| **Turn / step** | `turn_start → turn_end` and `step_start → step_end` on one fixed scripted turn |
|
|
43
|
+
| **Assembly** | `step/start` (from the child's own session log inside the captured request) to the request reaching the fixture endpoint: prompt assembly and request serialization — the phase plugin hooks run in. This is the graded per-turn figure |
|
|
44
|
+
| **Context tax** | Model-visible bytes of the request the harness actually sent (`system` + `tools` + `messages`), in tokens at the documented 4-chars/token heuristic; harness extension fields are counted separately |
|
|
45
|
+
| **Capability surface** | Tools and schemas the model is offered, prompt sections and their sizes, event listeners per event, loader entries, prompt variables — all as a diff against the same profile without the plugin |
|
|
46
|
+
| **Static risk flags** | Install scripts, shipped binaries, network hosts, subprocess use, dynamic code, filesystem writes, `process.exit` — with file and line evidence. Nothing is executed to produce them |
|
|
47
|
+
| **Health** | `ok`, `install-rejected`, `install-failed`, `boot-failed`, `turn-failed`, `entry-failed`, `entry-absent`. A plugin that fails to activate is a failed card, never a passing one |
|
|
48
|
+
|
|
49
|
+
See [`docs/example-card.md`](docs/example-card.md) for a complete card of the
|
|
50
|
+
test fixture.
|
|
51
|
+
|
|
52
|
+
## How it measures
|
|
53
|
+
|
|
54
|
+
```
|
|
55
|
+
┌──────────────── throwaway DSH_HOME ────────────────┐
|
|
56
|
+
subject package ► │ profile "rc-measured" ← shipped "headless" template │
|
|
57
|
+
│ subject plugin: an insert row, or the bundle list │
|
|
58
|
+
│ probe: inserted last, dumps the surface, exits │
|
|
59
|
+
└────────────┬───────────────────────────┬────────────┘
|
|
60
|
+
│ surface runs │ turn runs
|
|
61
|
+
▼ ▼
|
|
62
|
+
spawn → probe dump → exit spawn → session → turn → exit
|
|
63
|
+
boot wall/CPU + surface assembly/step + exact request
|
|
64
|
+
DEEPSEEK_BASE_URL → local fixture
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
* **Fresh state.** Each report gets a new `DSH_HOME`, a new profile copied from
|
|
68
|
+
the shipped `headless` template, and an empty working directory. Your real
|
|
69
|
+
profile is never opened, so a measured plugin cannot disturb it.
|
|
70
|
+
* **No model calls.** `DEEPSEEK_BASE_URL` points at a loopback endpoint that
|
|
71
|
+
speaks the Anthropic-style Messages SSE stream the shipped adapter parses. The
|
|
72
|
+
reply is a fixed one-line text block, so the turn is deterministic — and the
|
|
73
|
+
*request* is captured verbatim, which is the context-tax measurement.
|
|
74
|
+
* **No background noise.** The fixture overlay disables session-title generation
|
|
75
|
+
and telemetry, and a discarded warm-up pair primes the file cache before any
|
|
76
|
+
sample is recorded.
|
|
77
|
+
* **Round-paired deltas.** Each subject run is paired with the baseline run
|
|
78
|
+
immediately before it; the delta is the median of those paired differences,
|
|
79
|
+
which cancels slow machine drift. Grades charge only the part of a delta that
|
|
80
|
+
clears the machine's measured noise (one standard error of the paired
|
|
81
|
+
differences), so a loaded machine cannot turn a free plugin into a bad grade —
|
|
82
|
+
and cannot hide real overhead either, because the raw numbers stay on the
|
|
83
|
+
card next to their floor.
|
|
84
|
+
* **Integrity.** The card is hashed (`sha256` over canonical JSON, digest
|
|
85
|
+
excluded). Re-running the same subject compares against the previous card in
|
|
86
|
+
the same directory and prints drift.
|
|
87
|
+
|
|
88
|
+
## Install
|
|
89
|
+
|
|
90
|
+
The plugin ships as a bundle with metadata, `locale/en.json`, `locale/zh.json`
|
|
91
|
+
and an icon. Install it from this directory with the plugin manager — it adds
|
|
92
|
+
the dependency *and* enables the bundle in one step; removing it undoes both:
|
|
93
|
+
|
|
94
|
+
```
|
|
95
|
+
plugin_manager install_bundle target: "file:C:/path/to/dsh-report-card"
|
|
96
|
+
plugin_manager remove_bundle target: "dsh-report-card"
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
`install_bundle` is the sanctioned path because a bundle is composed through the
|
|
100
|
+
profile's bundle list, not just `node_modules`. The bundle inserts one entry,
|
|
101
|
+
`dsh-report-card`, which contributes one tool. It was installed this way on dsh
|
|
102
|
+
0.2.2-rc.1 and appears in the profile's bundle list beside the existing plugins
|
|
103
|
+
without touching them.
|
|
104
|
+
|
|
105
|
+
## Use
|
|
106
|
+
|
|
107
|
+
From a session (the tool is model-facing):
|
|
108
|
+
|
|
109
|
+
```
|
|
110
|
+
report_card(pluginSpec: "dsh-trust-check@0.2.6", samples: 3)
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
From a terminal — the same code path, printable output:
|
|
114
|
+
|
|
115
|
+
```powershell
|
|
116
|
+
node harness/cli.mjs ./test/fixtures/fake-plugin # local directory
|
|
117
|
+
node harness/cli.mjs dsh-trust-check@0.2.6 --samples 3 # npm package
|
|
118
|
+
node harness/cli.mjs dsh-quality-score --json # raw card
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
Cards are written to `./report-cards/<name>@<version>/<timestamp>.{json,md}` and
|
|
122
|
+
indexed in `./report-cards/index.json`; set `DSH_REPORT_CARD_DIR` to move that.
|
|
123
|
+
|
|
124
|
+
The tool takes **~60–90 s** for the default five samples: four harness boots per
|
|
125
|
+
sample round (surface × 2, turn × 2) plus one discarded warm-up round. Pass
|
|
126
|
+
`samples: 3` for a quick look and `samples: 8..10` when you need to resolve a
|
|
127
|
+
few milliseconds.
|
|
128
|
+
|
|
129
|
+
## Reading a card
|
|
130
|
+
|
|
131
|
+
```
|
|
132
|
+
**Grade D** · graded boot +6.3 ms CPU · graded turn 0 ms · context +76 tok · risk D
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
* The headline shows the two **graded** deltas plus the exact context delta and
|
|
136
|
+
the risk grade. Every raw number — wall boot, process CPU, turn, assembly,
|
|
137
|
+
step, launch — is in the tables below with its samples and noise floor.
|
|
138
|
+
* Wall time is presented but never graded: it is hostage to whatever else the
|
|
139
|
+
machine is doing. The boot grade uses noise-adjusted **process CPU** at the
|
|
140
|
+
moment the plugin tree is ready; the turn grade uses the noise-adjusted
|
|
141
|
+
**assembly** time, the phase plugin hooks run in.
|
|
142
|
+
* `context` is the median of the captured requests; it is exactly reproducible,
|
|
143
|
+
since the prompt is a function of the profile.
|
|
144
|
+
* `risk` is disclosure, not a verdict: `A` none, `B` informational only,
|
|
145
|
+
`C` medium flags, `D` high flags. A postinstall script is a high flag.
|
|
146
|
+
* The overall grade is the worst dimension. `F` means the card could not measure
|
|
147
|
+
a running plugin.
|
|
148
|
+
|
|
149
|
+
| Dimension | A | B | C | D | F |
|
|
150
|
+
|---|---|---|---|---|---|
|
|
151
|
+
| Boot CPU (noise-adjusted) | ≤ 10 ms | ≤ 30 ms | ≤ 80 ms | ≤ 250 ms | > 250 ms |
|
|
152
|
+
| Assembly (noise-adjusted) | ≤ 10 ms | ≤ 30 ms | ≤ 80 ms | ≤ 200 ms | > 200 ms |
|
|
153
|
+
| Context token delta | ≤ 100 | ≤ 500 | ≤ 2000 | ≤ 8000 | > 8000 |
|
|
154
|
+
| Risk | clean | info | medium | high | — |
|
|
155
|
+
|
|
156
|
+
Numbers are comparable **only on the same machine identity** (platform, CPU
|
|
157
|
+
count, dsh version) — the card prints it. A delta inside the printed noise floor
|
|
158
|
+
is noise; the card says so in its notes.
|
|
159
|
+
|
|
160
|
+
## Reproduce and verify
|
|
161
|
+
|
|
162
|
+
Every card prints its own reproduce command. The harness's own repeatability
|
|
163
|
+
check runs one subject several times and reports the spread:
|
|
164
|
+
|
|
165
|
+
```powershell
|
|
166
|
+
node harness/repeat.mjs ./test/fixtures/fake-plugin --runs 2 --samples 3
|
|
167
|
+
```
|
|
168
|
+
|
|
169
|
+
Measured while a harness session was actively running on the same machine (full
|
|
170
|
+
output in [`docs/acceptance.md`](docs/acceptance.md)): the **context delta
|
|
171
|
+
repeated at 0.00%**, the **graded boot CPU and assembly deltas at 0.00%**, and
|
|
172
|
+
the raw round-paired wall deltas within 1.9% (boot) and 2.4% (assembly). Earlier
|
|
173
|
+
in the same session, with a busier machine, raw wall deltas moved by up to 19%.
|
|
174
|
+
The card publishes its own jitter and noise floor for exactly this reason.
|
|
175
|
+
|
|
176
|
+
Tests:
|
|
177
|
+
|
|
178
|
+
```powershell
|
|
179
|
+
node test/unit.test.mjs # units, ~1 s
|
|
180
|
+
node test/harness.test.mjs # real boots against fixtures, ~80 s
|
|
181
|
+
```
|
|
182
|
+
|
|
183
|
+
`test/harness.test.mjs` covers the whole harness with fake plugins: a healthy
|
|
184
|
+
plugin (surface, context tax, risk flags, digest), a plugin that throws during
|
|
185
|
+
activation (must be a failed card), a plugin that calls `process.exit` (must be
|
|
186
|
+
a failed boot that still returns), and a second card that must show drift
|
|
187
|
+
against the first.
|
|
188
|
+
|
|
189
|
+
## Publishing
|
|
190
|
+
|
|
191
|
+
The package is publish-ready: `npm publish` and the catalog PR are documented in
|
|
192
|
+
[`docs/publishing.md`](docs/publishing.md), including the exact
|
|
193
|
+
`data/plugins/<owner>__<repo>.yml` entry the awesome-dsh-plugin list expects, the
|
|
194
|
+
`repository` field the catalog's npm mapping depends on, and the tarball-install
|
|
195
|
+
check that was run against the packed artifact.
|
|
196
|
+
## Safety and limits
|
|
197
|
+
|
|
198
|
+
* **The subject executes.** It runs in a disposable profile with a read-only
|
|
199
|
+
file policy and a fresh `DSH_HOME`, as your user — that is enough to keep it
|
|
200
|
+
away from your real profile, but it is **not** a security sandbox. Read the
|
|
201
|
+
risk flags before measuring something you do not trust.
|
|
202
|
+
* **No `app.asar` edits.** The harness only spawns the shipped CLI and writes
|
|
203
|
+
into a temp directory.
|
|
204
|
+
* **Fail closed.** A package the runtime refuses to install (for example a peer
|
|
205
|
+
version gate) is reported as `install-rejected` or `install-failed`; the harness never grants a
|
|
206
|
+
version exemption and never bypasses the gate. A plugin that crashes the host
|
|
207
|
+
or hangs is bounded by a per-child timeout.
|
|
208
|
+
* **The heuristic is not billing.** Context tokens use four characters per
|
|
209
|
+
token, the same heuristic the shipped token meter documents; CJK and JSON
|
|
210
|
+
schemas are underpriced by it. Extension fields (`dsh_session_log`,
|
|
211
|
+
`dsh_plugin_packages`) are measured separately and excluded.
|
|
212
|
+
* **One turn, one profile shape.** The card does not measure multi-turn
|
|
213
|
+
behaviour, compaction, or client-side (browser) cost. Client slot
|
|
214
|
+
contributions are not observable host-side; only host-side surface is
|
|
215
|
+
reported.
|
|
216
|
+
|
|
217
|
+
## Layout
|
|
218
|
+
|
|
219
|
+
| Path | Role |
|
|
220
|
+
|---|---|
|
|
221
|
+
| `index.js` | The bundle entry: registers the `report_card` tool |
|
|
222
|
+
| `harness/cli.mjs` | Command line entry |
|
|
223
|
+
| `harness/generate.mjs` | measure → diff → grade → render → store |
|
|
224
|
+
| `harness/measure.mjs` | The four-run loop, event parsing, request pricing |
|
|
225
|
+
| `harness/mock-llm.mjs` | The local Messages endpoint and its SSE fixture |
|
|
226
|
+
| `harness/probe/` | The fixture plugin that dumps a profile's surface |
|
|
227
|
+
| `harness/risk.mjs` | Static disclosure scan |
|
|
228
|
+
| `harness/report.mjs` | Grades, digest, drift, card assembly |
|
|
229
|
+
| `harness/render.mjs` | Markdown card |
|
|
230
|
+
| `harness/repeat.mjs` | Repeatability check for the harness itself |
|
|
231
|
+
| `docs/` | An example card and the acceptance evidence |
|
|
232
|
+
| `locale/`, `icon.svg`, `cordis.patch.yml` | Bundle metadata and its one insert row |
|
|
233
|
+
| `test/` | Unit tests, end-to-end tests, fake plugins |
|
|
234
|
+
|
|
235
|
+
## License
|
|
236
|
+
|
|
237
|
+
MIT.
|
package/cordis.patch.yml
ADDED
|
@@ -0,0 +1,127 @@
|
|
|
1
|
+
# Acceptance evidence
|
|
2
|
+
|
|
3
|
+
This file records what was actually measured when the v1 was built, on this
|
|
4
|
+
machine, so the claims in the README can be checked rather than trusted.
|
|
5
|
+
|
|
6
|
+
Machine: win32-x64, 8 logical CPUs, dsh 0.2.2-rc.1, Node v24.19.0 for the
|
|
7
|
+
harness and Electron 44.7.0's Node v24.21.0 for the measured children.
|
|
8
|
+
|
|
9
|
+
**The machine was not quiet.** A DeepSeek Harness Web session was running and
|
|
10
|
+
actively issuing commands throughout, and several other builds shared the
|
|
11
|
+
machine. That matters for the numbers below and is stated again where it
|
|
12
|
+
changes their meaning.
|
|
13
|
+
|
|
14
|
+
## 1. Same plugin, two runs
|
|
15
|
+
|
|
16
|
+
Command:
|
|
17
|
+
|
|
18
|
+
```powershell
|
|
19
|
+
node harness/repeat.mjs ./test/fixtures/fake-plugin --runs 2 --samples 3
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
Result (raw output, 2026-11-10, final code):
|
|
23
|
+
|
|
24
|
+
```
|
|
25
|
+
metric run 1 run 2 spread % of baseline
|
|
26
|
+
boot CPU graded delta 0 0 0.0 0.00%
|
|
27
|
+
boot CPU raw delta -62 -172 110.0 4.24%
|
|
28
|
+
boot wall graded delta 11.8 0 11.8 0.54%
|
|
29
|
+
boot paired median delta 50.6 31.8 18.8 0.86%
|
|
30
|
+
boot best-case delta 50.6 150.8 100.1 4.58%
|
|
31
|
+
assembly graded delta 0.6 0 0.6 1.90%
|
|
32
|
+
assembly paired median delta 5 -3 8.0 24.24%
|
|
33
|
+
assembly best-case delta 3 -5 8.0 24.24%
|
|
34
|
+
step best-case delta 7 -7 14.0 29.79%
|
|
35
|
+
context token delta 76 76 0.0 0.00%
|
|
36
|
+
baseline boot (median) 2186.1 2267.2 81.1 3.71%
|
|
37
|
+
baseline assembly (median) 33 41 8.0 24.24%
|
|
38
|
+
|
|
39
|
+
worst spread on a graded delta: 1.90% of its baseline (target: under 5%)
|
|
40
|
+
PASS
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
Reading:
|
|
44
|
+
|
|
45
|
+
* The **context delta is exact** (identical bytes, identical tokens) because the
|
|
46
|
+
prompt is a function of the profile, not of the machine.
|
|
47
|
+
* The **graded deltas repeat inside 1.9% of their baseline**: boot CPU 0.00%,
|
|
48
|
+
assembly 1.90%. Both grades are noise-adjusted, so the fixture plugin (which
|
|
49
|
+
contributes one tool, one prompt section and two listeners) grades A in both
|
|
50
|
+
runs.
|
|
51
|
+
* The **raw wall-clock deltas** also repeated well in this run — the round-paired
|
|
52
|
+
boot median moved 0.86%, the boot best case 4.58% — but that is luck, not a
|
|
53
|
+
guarantee: earlier in the same session, on a busier machine, baseline boot
|
|
54
|
+
medians moved by 19% between cards. The card prints its jitter and noise floor
|
|
55
|
+
so a reader can tell which situation they are in.
|
|
56
|
+
* `step` (the whole `step_start → step_end` window) is the noisiest figure at
|
|
57
|
+
30% of a ~45 ms baseline; that is why it is reported but not graded. `assembly`
|
|
58
|
+
— the phase plugin hooks actually run in — is five times steadier.
|
|
59
|
+
|
|
60
|
+
## 2. A measured plugin cannot touch the real profile
|
|
61
|
+
|
|
62
|
+
Every report runs in a fresh tree and points the child at it:
|
|
63
|
+
|
|
64
|
+
* `DSH_HOME` is a new `%TEMP%\dsh-report-card-*` directory, removed after the
|
|
65
|
+
report unless `--keep` is passed.
|
|
66
|
+
* The profile is a fresh copy of the shipped `headless` template under that
|
|
67
|
+
home; the subject is copied or installed into *that* profile's
|
|
68
|
+
`node_modules`.
|
|
69
|
+
* The working directory is an empty `workspace` directory under the same temp
|
|
70
|
+
tree.
|
|
71
|
+
* The probe records the state it actually saw. A real card contains:
|
|
72
|
+
|
|
73
|
+
```json
|
|
74
|
+
"measuredEnvironment": {
|
|
75
|
+
"home": "C:\\Users\\conta\\AppData\\Local\\Temp\\dsh-report-card-XXXXXXXX\\home",
|
|
76
|
+
"cwd": "C:\\Users\\conta\\AppData\\Local\\Temp\\dsh-report-card-XXXXXXXX\\workspace"
|
|
77
|
+
}
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
so the isolation is part of the evidence, not a promise. `~/.dsh` is never
|
|
81
|
+
passed to the child; the parent harness process never imports the subject.
|
|
82
|
+
|
|
83
|
+
The one thing this does **not** do is sandbox the subject from the operating
|
|
84
|
+
system: it runs as your user inside that disposable process. The README says so
|
|
85
|
+
and the card lists the static risk flags before you install anything.
|
|
86
|
+
|
|
87
|
+
## 3. Every number is reproducible from a command
|
|
88
|
+
|
|
89
|
+
* The card prints its own command in the **Reproduce** section, e.g.
|
|
90
|
+
`node harness/cli.mjs "./test/fixtures/fake-plugin" --samples 3`.
|
|
91
|
+
* `harness/cli.mjs --json` prints the same card as JSON, including every sample
|
|
92
|
+
and the digest.
|
|
93
|
+
* `harness/repeat.mjs` runs a card N times and prints the spread (section 1).
|
|
94
|
+
* A card's `digest` is the SHA-256 of the canonical JSON with the digest field
|
|
95
|
+
removed; `digestReport()` in `harness/report.mjs` recomputes it, and
|
|
96
|
+
`test/unit.test.mjs` proves the round-trip.
|
|
97
|
+
|
|
98
|
+
## 4. `node test/*.mjs` covers the harness with fake plugins
|
|
99
|
+
|
|
100
|
+
```
|
|
101
|
+
$ node test/unit.test.mjs
|
|
102
|
+
12 passed, 0 failed
|
|
103
|
+
|
|
104
|
+
$ node test/harness.test.mjs
|
|
105
|
+
ok a fake plugin gets a complete, measured card
|
|
106
|
+
ok a plugin that fails to start is a failed card, not a crash
|
|
107
|
+
ok a plugin that kills the host is a failed boot, bounded
|
|
108
|
+
ok the second card compares with the first one for drift
|
|
109
|
+
|
|
110
|
+
4 passed, 0 failed
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
The end-to-end test asserts the fixture's exact capability surface
|
|
114
|
+
(`fixture_echo`, `tool:fixture_echo`, the `tools/pre-execute` listener, the
|
|
115
|
+
loader entry and its `active` phase), the risk flags (install script, network
|
|
116
|
+
host), the digest, the saved files, the failed-card path for a plugin that
|
|
117
|
+
throws in `apply()`, the failed-boot path for a plugin that calls
|
|
118
|
+
`process.exit()`, and drift on the second card.
|
|
119
|
+
|
|
120
|
+
## 5. What v1 does not claim
|
|
121
|
+
|
|
122
|
+
* Cross-machine comparability. Cards record the machine identity and are only
|
|
123
|
+
comparable within it.
|
|
124
|
+
* Multi-turn, compaction, or client-side cost. One scripted turn, host side.
|
|
125
|
+
* Sandboxing the subject. See section 2.
|
|
126
|
+
* Billing-grade tokens. The estimate is the shipped token meter's documented
|
|
127
|
+
4-characters-per-token heuristic; CJK and JSON schemas are underpriced.
|
|
@@ -0,0 +1,106 @@
|
|
|
1
|
+
# Report card — @local/rc-fake-plugin@0.1.0
|
|
2
|
+
|
|
3
|
+
**Grade D** · graded boot +6.3 ms CPU · graded turn 0 ms · context +76 tok · risk D
|
|
4
|
+
|
|
5
|
+
> Fake plugin used by the report-card harness tests: one tool, one prompt section, one listener.
|
|
6
|
+
|
|
7
|
+
## At a glance
|
|
8
|
+
|
|
9
|
+
| Dimension | Baseline | Subject | Delta | Grade |
|
|
10
|
+
| --- | --- | --- | --- | --- |
|
|
11
|
+
| Boot wall (process → plugin tree) | 2397 / 2619 / 2734 ms | 2465 / 2625 / 2805 ms | +67.9 ms | — |
|
|
12
|
+
| Boot CPU (process at ready) | 2780 / 2952 / 2953 ms | 2827 / 2921 / 2983 ms | +30 ms | A |
|
|
13
|
+
| Turn (turn_start → turn_end) | 142 / 101 / 154 ms | 114 / 111 / 114 ms | -28 ms | — |
|
|
14
|
+
| Assembly (step/start → request) | 88 / 38 / 60 ms | 48 / 46 / 45 ms | -15 ms | A |
|
|
15
|
+
| Step (prompt → model answered) | 105 / 52 / 79 ms | 65 / 65 / 61 ms | -18 ms | — |
|
|
16
|
+
| Context (model-visible tokens) | 6237 | 6313 | +76 tok | A |
|
|
17
|
+
| Static risk flags | — | 3 | — | D |
|
|
18
|
+
|
|
19
|
+
Samples: 3 per variant, medians shown. Boot jitter +2.5%; step jitter +67.1%.
|
|
20
|
+
|
|
21
|
+
Grades charge only overhead above this machine's noise floor: boot CPU +6.3 ms graded from +30 ms raw (floor +23.7 ms), turn assembly 0 ms graded from -15 ms raw (floor +13.9 ms). Wall time is reported but never graded: it is hostage to whatever else the machine is doing.
|
|
22
|
+
|
|
23
|
+
## Boot
|
|
24
|
+
|
|
25
|
+
| Sample | Baseline | Subject |
|
|
26
|
+
| --- | --- | --- |
|
|
27
|
+
| 1 | 2397 ms | 2465 ms |
|
|
28
|
+
| 2 | 2619 ms | 2625 ms |
|
|
29
|
+
| 3 | 2734 ms | 2805 ms |
|
|
30
|
+
|
|
31
|
+
Measured as process start (performance.timeOrigin) to the probe writing its dump (plugin tree composed). Deltas are round-paired medians (each subject run is paired with the baseline run before it), which cancels slow machine drift; the unpaired median difference was +6.3 ms. A negative delta means the subject booted sooner.
|
|
32
|
+
|
|
33
|
+
Launch (spawn → the same dump, including process startup): baseline 2724 ms, subject 2734 ms, delta +36 ms.
|
|
34
|
+
|
|
35
|
+
Boot CPU (the graded figure, total process CPU when the plugin tree is ready): baseline 2952 ms, subject 2921 ms, delta +30 ms raw, +6.3 ms after the noise floor. CPU time moves only when the plugin actually does work, so it stays stable on a loaded machine.
|
|
36
|
+
|
|
37
|
+
## One scripted turn
|
|
38
|
+
|
|
39
|
+
| Metric | Baseline | Subject | Delta |
|
|
40
|
+
| --- | --- | --- | --- |
|
|
41
|
+
| Turn | 142 / 101 / 154 ms | 114 / 111 / 114 ms | -28 ms |
|
|
42
|
+
| Assembly (hook phase) | 88 / 38 / 60 ms | 48 / 46 / 45 ms | -15 ms |
|
|
43
|
+
| Step (whole) | 105 / 52 / 79 ms | 65 / 65 / 61 ms | -18 ms |
|
|
44
|
+
|
|
45
|
+
Best-case assembly delta +7 ms; graded assembly delta 0 ms — the per-turn figure that sets the turn grade.
|
|
46
|
+
|
|
47
|
+
Measured as step_start to the first step_end on that turn, against local fixture endpoint, no network call.
|
|
48
|
+
|
|
49
|
+
## Context tax
|
|
50
|
+
|
|
51
|
+
| Field | Baseline | Subject | Delta |
|
|
52
|
+
| --- | --- | --- | --- |
|
|
53
|
+
| Model-visible chars | 24945 | 25249 | +304 |
|
|
54
|
+
| · system prompt | 2758 | 2826 | +68 |
|
|
55
|
+
| · tool schemas | 20369 | 20605 | +236 |
|
|
56
|
+
| · conversation | 1818 | 1818 | 0 |
|
|
57
|
+
| Estimated tokens | 6237 | 6313 | +76 |
|
|
58
|
+
| Harness extension fields | 34142 | 34497 | +355 |
|
|
59
|
+
|
|
60
|
+
Measured as model-visible bytes of the request the harness sent (system + tools + messages), tokens at 4 chars/token. Extension fields are request metadata the harness sends beside the model-visible prompt; they are listed separately and excluded from the token estimate.
|
|
61
|
+
|
|
62
|
+
## Capability surface
|
|
63
|
+
|
|
64
|
+
| Added | Detail |
|
|
65
|
+
| --- | --- |
|
|
66
|
+
| Tools | `fixture_echo` (235 schema chars) |
|
|
67
|
+
| Prompt sections | `tool:fixture_echo` (66 chars) |
|
|
68
|
+
| Event listeners | `tools/pre-execute` +1, `session/start` +1 |
|
|
69
|
+
| Loader entries | `@local/rc-fake-plugin` |
|
|
70
|
+
| Subject entry phase | active |
|
|
71
|
+
| Prompt variables | none |
|
|
72
|
+
| Profile size | 101 → 102 entries |
|
|
73
|
+
|
|
74
|
+
## Static risk flags
|
|
75
|
+
|
|
76
|
+
| Level | Flag | Detail |
|
|
77
|
+
| --- | --- | --- |
|
|
78
|
+
| high | `install-scripts` | runs code during install: postinstall |
|
|
79
|
+
| medium | `file-write` | writes to the filesystem |
|
|
80
|
+
| info | `network-hosts` | 1 host referenced: collector.example.test |
|
|
81
|
+
|
|
82
|
+
Evidence:
|
|
83
|
+
- `file-write` — index.js: `const writeReport = (file) => fs.writeFileSync(file, uploadUrl)`
|
|
84
|
+
|
|
85
|
+
## Notes
|
|
86
|
+
|
|
87
|
+
- Step samples vary by 67.1%; treat deltas below that as noise.
|
|
88
|
+
|
|
89
|
+
## Method and limits
|
|
90
|
+
|
|
91
|
+
Measured on win32-x64, 8 CPUs, dsh 0.2.2-rc.1, Node v24.19.0.
|
|
92
|
+
|
|
93
|
+
- Profile: `headless` template, fresh profile copied from a shipped template, fresh DSH_HOME per report.
|
|
94
|
+
- Isolation: empty workspace directory, read-only file policy; the subject never touches your real profile.
|
|
95
|
+
- The subject still runs with your user privileges inside that disposable process. Read the risk flags before installing anything you did not write.
|
|
96
|
+
- Numbers are comparable only on the same machine identity; the digest changes whenever any number does.
|
|
97
|
+
|
|
98
|
+
## Reproduce
|
|
99
|
+
|
|
100
|
+
```
|
|
101
|
+
node harness/cli.mjs "./test/fixtures/fake-plugin" --samples 3
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
Digest: `sha256:55ffdbf5e230a5fbc258445d58a126290e97e0e2c47fe8cfbaae36bd122918c3`
|
|
105
|
+
|
|
106
|
+
Generated 2026-11-10T16:27:44.348Z by `dsh-report-card` (dsh-report-card/1).
|