dsh-plugin-profiler 0.1.0 → 0.1.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 liuhange789
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md CHANGED
@@ -12,13 +12,21 @@ This is not another benchmark. `dsh-benchup` and `dsh-eval` answer "is DSH good?
12
12
  | `dsh-eval` | Does it choose the right tool? | a task suite |
13
13
  | this plugin | What is my installed plugin set costing me right now? | the session you are already in |
14
14
 
15
+ ## Install
16
+
17
+ ```bash
18
+ dsh plugin add dsh-plugin-profiler
19
+ ```
20
+
21
+ The bundle declares its own patch (`cordis.patch.yml`), so the command is the only install step. `/diet` works immediately in the running session.
22
+
15
23
  ## What it reports
16
24
 
17
25
  | Section | Source | Meaning |
18
26
  | --- | --- | --- |
19
27
  | Context tax | `ctx.tools.schemas()` | Estimated tokens for every tool definition sent on each request, plus the heaviest tools |
20
- | Hot path | prepended `agent/pre-step`, `agent/request`, `tools/pre-execute` listeners | Wall time of the remaining chain, builtins included, provider and tool body excluded |
21
- | Tool calls | `tools/pre-execute` → `tools/result` | Wall time per tool call, keyed by `callId` |
28
+ | Hot path | prepended `agent/pre-step`, `agent/request`, `llm/stream` setup, `tools/pre-execute` listeners | Wall time of the remaining chain, builtins included, provider and tool body excluded |
29
+ | Tool calls | `tools/pre-execute` → `tools/result` | Wall time per call, keyed by `callId`; includes approval and post-tool work, not only the tool body |
22
30
  | Listener fan-out | live `ctx.events._hooks` snapshot | Which plugin registered how many listeners, and how many sit on hot events |
23
31
  | Event volume | `internal/dispatch` | How often each event actually fires |
24
32
 
@@ -57,7 +65,7 @@ Read this before installing. It is the reason the card says "fan-out is a suspic
57
65
 
58
66
  - **No per-plugin milliseconds.** rc.7 gives `internal/listener` only a veto over registration (`vendor/cordis/src/events.ts`, `on()` returns a non-null value as the disposer instead of registering), and it has no post-dispatch event. A plugin therefore cannot wrap another plugin's callback. Any tool claiming "plugin A's pre-execute took N ms" on rc.7 is not measuring what it says.
59
67
  - **No tool owner.** `tools.register()` stores the definition but not the registering fiber, so the context tax is priced per tool, not per plugin.
60
- - **Two scopes are intentionally excluded from the tax line.** `llm/stream` is timed only up to interception, not stream consumption, and the provider call dominates it. Tool body time is measured separately under "Tool calls".
68
+ - **Two scopes are intentionally excluded from the tax line.** `llm/stream` is timed only up to interception, not stream consumption, and the provider call dominates it. rc.7 dispatches that event synchronously and iterates the handler's return value, so the listener must stay synchronous; an `async` wrapper would return a Promise and break every model call. Tool body time is measured separately under "Tool calls", which also contains approval and post-tool work.
61
69
 
62
70
  If a host seam lands later (a post-registration or post-dispatch hook, plus an owner field on tool records), this plugin can add real per-plugin attribution without changing its report format.
63
71
 
@@ -69,7 +77,7 @@ Written so it can never be the reason your host breaks:
69
77
  - A dispatch with no `next` callback (a plain emit of a waterfall event name) passes through untouched instead of being called as a function.
70
78
  - No `commands` service, or a `/diet` name already taken by another copy, costs the command only — counters keep running, `/profiler` is still attempted, and the rest of the session is unaffected.
71
79
  - `tools.schemas()` throwing, circular parameter objects, or a proxy that refuses enumeration degrade that one section and leave the rest of the card intact.
72
- - Counters are bounded (200 in-flight calls, 200 distinct events), so a very long session cannot grow them without limit.
80
+ - Counters are bounded (200 in-flight calls with oldest-first eviction, 200 distinct events, 200 tools), so a very long session cannot grow them without limit. A call that started before `/diet reset` is discarded instead of being backfilled into the empty window.
73
81
  - Tables are padded by display width, so Chinese, Japanese or emoji plugin names line up like ASCII ones.
74
82
 
75
83
  ## Usage
@@ -99,7 +107,7 @@ DeepSeek Harness `0.1.0-rc.7` (`@deepseek-ai/cordis` 4.x event bus). Node `^22.1
99
107
  npm test
100
108
  ```
101
109
 
102
- Ten tests run against the real vendored cordis runtime: silent degradation, chain timing with two plugins, tool call timing, tool-definition pricing, reset, missing commands service, duplicate command name, dispatch without `next`, unserializable tool parameters, and a throwing tools service.
110
+ Thirteen tests run against a real cordis runtime, loaded from `CORDIS_PATH` when set, otherwise from the `@deepseek-ai/cordis` dev dependency, so a clean clone can verify itself after `npm install`. They cover silent degradation, chain timing with two plugins, tool call timing, tool-definition pricing, reset (including an in-flight call started before it), missing commands service, duplicate command name, dispatch without `next`, unserializable tool parameters, a throwing tools service, the synchronous `llm/stream` return value, and listener removal on unload.
103
111
 
104
112
  ## License
105
113
 
@@ -0,0 +1,3 @@
1
+ - insert:
2
+ - id: plugin-profiler
3
+ name: dsh-plugin-profiler
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "dsh-plugin-profiler",
3
- "version": "0.1.0",
3
+ "version": "0.1.2",
4
4
  "description": "Plugin load audit for DSH: hot event chains, tool execution cost, tool-definition context cost, and per-plugin listener inventory",
5
5
  "type": "module",
6
6
  "main": "src/index.js",
@@ -9,8 +9,14 @@
9
9
  },
10
10
  "files": [
11
11
  "src",
12
- "README.md"
12
+ "README.md",
13
+ "cordis.patch.yml"
13
14
  ],
15
+ "dsh": {
16
+ "bundle": {
17
+ "patch": "./cordis.patch.yml"
18
+ }
19
+ },
14
20
  "keywords": [
15
21
  "dsh",
16
22
  "deepseek-harness",
@@ -35,5 +41,8 @@
35
41
  "url": "https://github.com/liuhange789/dsh-plugin-profiler/issues"
36
42
  },
37
43
  "author": "liuhange789",
38
- "license": "MIT"
44
+ "license": "MIT",
45
+ "devDependencies": {
46
+ "@deepseek-ai/cordis": "^4.0.5"
47
+ }
39
48
  }
package/src/index.js CHANGED
@@ -27,11 +27,16 @@ import {
27
27
 
28
28
  export const name = 'plugin-profiler'
29
29
 
30
- /** Waterfall events whose remaining chain is timed around `next()`. */
30
+ /**
31
+ * Waterfall events whose remaining chain is awaited, so `async` is safe.
32
+ *
33
+ * `llm/stream` is deliberately absent: rc.7 dispatches it synchronously and
34
+ * requires the handler to return the AsyncIterable itself. An async wrapper
35
+ * would hand the caller a Promise and break every model call.
36
+ */
31
37
  const TIMED_CHAINS = [
32
38
  ['agent/pre-step', 'before each step'],
33
39
  ['agent/request', 'request assembly'],
34
- ['llm/stream', 'stream interception only, not consumption'],
35
40
  ]
36
41
 
37
42
  /** Bounds that keep counters small even after a very long session. */
@@ -45,6 +50,8 @@ const clock = typeof performance !== 'undefined' && typeof performance.now === '
45
50
  function newState() {
46
51
  return {
47
52
  since: Date.now(),
53
+ /** Bumped by `/diet reset`; calls started earlier must not land in the new window. */
54
+ generation: 0,
48
55
  chains: new Map(),
49
56
  dispatches: new Map(),
50
57
  tools: new Map(),
@@ -85,6 +92,8 @@ function recordChain(state, event, note, ms) {
85
92
  function recordTool(state, tool, ms) {
86
93
  let row = state.tools.get(tool)
87
94
  if (!row) {
95
+ // Dynamic or renamed tools could otherwise grow this table forever.
96
+ if (state.tools.size >= MAX_EVENTS) return
88
97
  row = { tool, calls: 0, totalMs: 0, maxMs: 0 }
89
98
  state.tools.set(tool, row)
90
99
  }
@@ -170,38 +179,65 @@ export function apply(ctx) {
170
179
  }, { global: true })
171
180
 
172
181
  // Outermost listener: time everything the rest of the chain does, `next()` included.
173
- // Rest arguments keep this signature-agnostic across the three events.
182
+ // Rest arguments keep this signature-agnostic across both events.
174
183
  for (const [event, note] of TIMED_CHAINS) {
175
184
  ctx.on(event, async (...args) => {
176
185
  const next = args[args.length - 1]
177
186
  // A non-waterfall dispatch (a plain emit of the same name) has no `next`;
178
187
  // pass through instead of trying to call something that is not a function.
179
188
  if (typeof next !== 'function') return next
189
+ const generation = state.generation
180
190
  const start = clock()
181
191
  try {
182
192
  return await next()
183
193
  } finally {
184
- safely(() => recordChain(state, event, note, clock() - start))
194
+ if (generation === state.generation) {
195
+ safely(() => recordChain(state, event, note, clock() - start))
196
+ }
185
197
  }
186
198
  }, { prepend: true, global: true })
187
199
  }
188
200
 
201
+ // `llm/stream` is dispatched synchronously and its return value is iterated
202
+ // with `for await`. Stay synchronous and hand back the iterable unchanged;
203
+ // only the setup (adapter resolution) is timed, never chunk consumption.
204
+ ctx.on('llm/stream', (...args) => {
205
+ const next = args[args.length - 1]
206
+ if (typeof next !== 'function') return next
207
+ const generation = state.generation
208
+ const start = clock()
209
+ try {
210
+ return next()
211
+ } finally {
212
+ if (generation === state.generation) {
213
+ safely(() => recordChain(state, 'llm/stream', 'stream setup only, not chunk consumption', clock() - start))
214
+ }
215
+ }
216
+ }, { prepend: true, global: true })
217
+
189
218
  // Pre-tool gates: time the gate chain and remember when the call started.
190
219
  ctx.on('tools/pre-execute', async (...args) => {
191
220
  const next = args[args.length - 1]
192
221
  if (typeof next !== 'function') return next
222
+ const generation = state.generation
193
223
  const start = clock()
194
224
  safely(() => {
195
225
  const exec = args[0]
196
226
  const callId = callIdOf(exec)
197
227
  if (callId === undefined) return
198
- if (state.inflight.size >= MAX_INFLIGHT) state.inflight.clear()
199
- state.inflight.set(callId, { name: toolNameOf(exec), start })
228
+ if (!state.inflight.has(callId) && state.inflight.size >= MAX_INFLIGHT) {
229
+ // Drop the oldest call instead of losing every other in-flight one.
230
+ const oldest = state.inflight.keys().next().value
231
+ state.inflight.delete(oldest)
232
+ }
233
+ state.inflight.set(callId, { name: toolNameOf(exec), start, generation })
200
234
  })
201
235
  try {
202
236
  return await next()
203
237
  } finally {
204
- safely(() => recordChain(state, 'tools/pre-execute', 'pre-tool gates', clock() - start))
238
+ if (generation === state.generation) {
239
+ safely(() => recordChain(state, 'tools/pre-execute', 'pre-tool gates', clock() - start))
240
+ }
205
241
  }
206
242
  }, { prepend: true, global: true })
207
243
 
@@ -212,6 +248,7 @@ export function apply(ctx) {
212
248
  const started = state.inflight.get(callId)
213
249
  if (!started) return
214
250
  state.inflight.delete(callId)
251
+ if (started.generation !== state.generation) return
215
252
  recordTool(state, started.name, clock() - started.start)
216
253
  })
217
254
  }, { global: true })
@@ -262,6 +299,7 @@ export function apply(ctx) {
262
299
  }
263
300
 
264
301
  function reset(state) {
302
+ state.generation += 1
265
303
  state.since = Date.now()
266
304
  state.chains.clear()
267
305
  state.dispatches.clear()
package/src/report.js CHANGED
@@ -31,6 +31,15 @@ function safeText(value) {
31
31
  }
32
32
  }
33
33
 
34
+ /** Read a tool's name without trusting a hostile getter on the schema object. */
35
+ function schemaName(schema) {
36
+ try {
37
+ return safeText(schema?.name) || '?'
38
+ } catch {
39
+ return '?'
40
+ }
41
+ }
42
+
34
43
  /** Estimate prompt tokens for one tool schema (~4 characters per token). */
35
44
  export function estimateTokens(schema) {
36
45
  try {
@@ -226,7 +235,7 @@ export function renderReport(state, snapshot, now) {
226
235
  }
227
236
 
228
237
  if (schemas.length > 0) {
229
- const priced = schemas.map(schema => ({ name: safeText(schema?.name) || '?', tokens: estimateTokens(schema) }))
238
+ const priced = schemas.map(schema => ({ name: schemaName(schema), tokens: estimateTokens(schema) }))
230
239
  priced.sort((a, b) => b.tokens - a.tokens)
231
240
  const total = priced.reduce((sum, row) => sum + row.tokens, 0)
232
241
  lines.push('')