@amaster.ai/dsh-langfuse 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +49 -0
- package/cordis.patch.yml +5 -0
- package/lib/client.js +287 -0
- package/lib/index.js +529 -0
- package/lib/types/client.d.ts +146 -0
- package/lib/types/index.d.ts +66 -0
- package/package.json +81 -0
- package/preview.png +0 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 dsh-plugins contributors
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
ADDED
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
# @amaster.ai/dsh-langfuse
|
|
2
|
+
|
|
3
|
+

|
|
4
|
+
|
|
5
|
+
Langfuse observability for [DeepSeek Harness (dsh)](https://github.com/deepseek-ai/deepseek-harness):
|
|
6
|
+
|
|
7
|
+
- one **generation** per LLM call (`llm/stream` waterfall), plus a nested `llm-request` span carrying the verbatim loop-built request
|
|
8
|
+
- one **span** per tool call (`tools/execute` waterfall)
|
|
9
|
+
- one **trace** per session turn (`session/event`) — in the v5 SDK the trace IS its root span, ended (and thereby exported) at `turn/end`
|
|
10
|
+
- subagent child sessions nested under the parent's tree (`session/created` header link + `subagent/start` / `subagent/end`)
|
|
11
|
+
|
|
12
|
+
## Install
|
|
13
|
+
|
|
14
|
+
```sh
|
|
15
|
+
dsh plugin --profile my-agent add @amaster.ai/dsh-langfuse \
|
|
16
|
+
@langfuse/tracing @langfuse/otel @opentelemetry/sdk-trace-node \
|
|
17
|
+
@opentelemetry/api @opentelemetry/exporter-trace-otlp-http
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
The Langfuse JS SDK v5 peers (`@langfuse/tracing`, `@langfuse/otel`, `@opentelemetry/sdk-trace-node`) are loaded via lazy dynamic `import()`, so an unused install costs nothing — but they must all be present at runtime (`dsh plugin add` does not auto-install peer trees, so the OTEL api/exporter packages `@langfuse/otel` itself peers on are listed explicitly).
|
|
21
|
+
|
|
22
|
+
## Configuration
|
|
23
|
+
|
|
24
|
+
Disabled by default. Configure via the profile's `cordis.patch.yml`:
|
|
25
|
+
|
|
26
|
+
```yaml
|
|
27
|
+
- insert:
|
|
28
|
+
- id: langfuse
|
|
29
|
+
name: '@amaster.ai/dsh-langfuse'
|
|
30
|
+
config:
|
|
31
|
+
enabled: true
|
|
32
|
+
publicKey: pk-lf-...
|
|
33
|
+
secretKey: sk-lf-... # secret role
|
|
34
|
+
baseUrl: https://cloud.langfuse.com
|
|
35
|
+
traceName: dsh-turn
|
|
36
|
+
captureContent: true # set false to record metadata only
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
Buffered spans drain on `session/flush(session)`; the exporter shuts down with the plugin fiber.
|
|
40
|
+
|
|
41
|
+
Observability is a no-throw seam: backend errors are logged with a `[dsh-langfuse]` prefix and never escape into the agent loop.
|
|
42
|
+
|
|
43
|
+
## Compatibility
|
|
44
|
+
|
|
45
|
+
Pinned dsh/cordis versions live in the [root compat matrix](../../README.md#compatibility). Event payloads ride pre-release dsh APIs — check the `TODO(verify)` markers in `src/` before upgrading dsh.
|
|
46
|
+
|
|
47
|
+
## License
|
|
48
|
+
|
|
49
|
+
MIT
|
package/cordis.patch.yml
ADDED
package/lib/client.js
ADDED
|
@@ -0,0 +1,287 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Langfuse reporter — thin synchronous wrapper over the Langfuse JS SDK v5's
|
|
3
|
+
* OpenTelemetry-based tracing API (`@langfuse/tracing` + `@langfuse/otel`),
|
|
4
|
+
* and the plugin's single no-throw seam: every method swallows SDK failures
|
|
5
|
+
* with a `[dsh-langfuse]` console.error so observability can never break the
|
|
6
|
+
* agent loop.
|
|
7
|
+
*
|
|
8
|
+
* v5's observations-first data model has no trace object: a trace is the root
|
|
9
|
+
* observation plus the correlating attributes every observation carries. So:
|
|
10
|
+
* - a "trace" here is a root `LangfuseSpan` created with {@link NO_PARENT} —
|
|
11
|
+
* without it the SDK parents to whatever span is active in the host's
|
|
12
|
+
* ambient OTEL context (e.g. an instrumented HTTP server around dsh),
|
|
13
|
+
* smearing turn traces into foreign traces;
|
|
14
|
+
* - trace-level input/output/metadata live on that root span (the v5
|
|
15
|
+
* replacement for the removed `trace.update()`), and the root must be
|
|
16
|
+
* ENDED — spans only export on end, so an un-ended root never leaves the
|
|
17
|
+
* process;
|
|
18
|
+
* - the correlating `session.id` attribute is stamped on every observation
|
|
19
|
+
* via a handle-keyed WeakMap — the explicit-tree equivalent of v5's
|
|
20
|
+
* context-scoped `propagateAttributes()`, which cannot wrap this plugin's
|
|
21
|
+
* event-driven lifecycle.
|
|
22
|
+
*
|
|
23
|
+
* The SDK stack is a heavy optional peer set, so it loads via dynamic
|
|
24
|
+
* `import()` kicked off in the constructor — a disabled plugin never pays for
|
|
25
|
+
* it, and a missing or incompatible peer degrades the reporter to a no-op
|
|
26
|
+
* instead of breaking the plugin load. The exporter runs on an ISOLATED
|
|
27
|
+
* tracer provider (`setLangfuseTracerProvider`) rather than
|
|
28
|
+
* `provider.register()`: the process-global OTEL provider stays untouched and
|
|
29
|
+
* the host keeps its own tracing pipeline. {@link LangfuseReporter.ready}
|
|
30
|
+
* settles (never rejects) once the import+wiring finished: the plugin returns
|
|
31
|
+
* it from `apply()` so fiber readiness covers the import window, and
|
|
32
|
+
* {@link flush}/{@link shutdown} chain behind it. Observation calls landing
|
|
33
|
+
* before readiness are still dropped — a window that cannot exist once the
|
|
34
|
+
* plugin fiber has been awaited.
|
|
35
|
+
*/
|
|
36
|
+
/**
|
|
37
|
+
* OTEL's canonical invalid span context as a literal (a static
|
|
38
|
+
* `INVALID_SPAN_CONTEXT` import would load @opentelemetry/api eagerly, even
|
|
39
|
+
* for a disabled plugin): the SDK starts a fresh traceId for an invalid
|
|
40
|
+
* parent instead of adopting the ambient context's active span.
|
|
41
|
+
*/
|
|
42
|
+
const NO_PARENT = { traceId: '0'.repeat(32), spanId: '0'.repeat(16), traceFlags: 0 };
|
|
43
|
+
/**
|
|
44
|
+
* Map dsh token accounting onto Langfuse's `usageDetails`, keeping every
|
|
45
|
+
* bucket mutually exclusive (Langfuse's flat-bucket rule: `input` excludes
|
|
46
|
+
* `input_*`, `output` excludes `output_*`, `total` is the bucket sum —
|
|
47
|
+
* overlapping buckets double-count usage and inferred cost). dsh reports
|
|
48
|
+
* uncached input, separate cache buckets, and provider-style output that
|
|
49
|
+
* INCLUDES reasoning (verified in dsh-llm-deepseek@0.1.0-rc.7:
|
|
50
|
+
* `outputTokens: usage.completion_tokens`, with reasoning split out of
|
|
51
|
+
* `completion_tokens_details`), so the `output` bucket subtracts
|
|
52
|
+
* `reasoningTokens` into `output_reasoning`. Only `usageDetails` is sent.
|
|
53
|
+
*/
|
|
54
|
+
export function usageOf(usage) {
|
|
55
|
+
// The dsh type marks the two primary fields required, but a non-conformant
|
|
56
|
+
// adapter emitting a partial usage chunk would otherwise turn every bucket
|
|
57
|
+
// NaN (serialized as null by the SDK — silently corrupting billed usage).
|
|
58
|
+
const input = (usage.inputTokens ?? 0) + (usage.cacheReadTokens ?? 0) + (usage.cacheWriteTokens ?? 0);
|
|
59
|
+
const output = usage.outputTokens ?? 0;
|
|
60
|
+
const total = input + output;
|
|
61
|
+
// Clamp reasoning at the provider's output count: reasoning is a subset of
|
|
62
|
+
// output, and only a broken adapter would report more — clamping keeps the
|
|
63
|
+
// buckets summing to total even then.
|
|
64
|
+
const reasoning = Math.min(usage.reasoningTokens ?? 0, output);
|
|
65
|
+
const usageDetails = {
|
|
66
|
+
input: usage.inputTokens ?? 0,
|
|
67
|
+
output: output - reasoning,
|
|
68
|
+
total,
|
|
69
|
+
};
|
|
70
|
+
if (usage.cacheReadTokens)
|
|
71
|
+
usageDetails.input_cache_read = usage.cacheReadTokens;
|
|
72
|
+
if (usage.cacheWriteTokens)
|
|
73
|
+
usageDetails.input_cache_creation = usage.cacheWriteTokens;
|
|
74
|
+
if (reasoning)
|
|
75
|
+
usageDetails.output_reasoning = reasoning;
|
|
76
|
+
return usageDetails;
|
|
77
|
+
}
|
|
78
|
+
export class LangfuseReporter {
|
|
79
|
+
config;
|
|
80
|
+
tracing = null;
|
|
81
|
+
provider = null;
|
|
82
|
+
/**
|
|
83
|
+
* Observation → its trace's correlating attributes; children inherit their
|
|
84
|
+
* parent's. Stamped on every observation (`session.id`,
|
|
85
|
+
* `langfuse.trace.name`): v5's observations-first model wants trace context
|
|
86
|
+
* on every span (the v4 migration doc: "copied to every span where the name
|
|
87
|
+
* must be queryable"), and older Langfuse servers derive the trace row from
|
|
88
|
+
* ANY span carrying these — a root-only stamp can lose to child-derived
|
|
89
|
+
* trace events depending on ingestion order. This is the explicit
|
|
90
|
+
* handle-tree equivalent of v5's context-scoped propagateAttributes, which
|
|
91
|
+
* cannot wrap this plugin's event-driven lifecycle.
|
|
92
|
+
*/
|
|
93
|
+
traceContext = new WeakMap();
|
|
94
|
+
/** Trace roots (created by openTrace) — the spans that may carry trace-level IO. */
|
|
95
|
+
roots = new WeakSet();
|
|
96
|
+
/** Settles (never rejects) once the lazy SDK import+wiring finished. */
|
|
97
|
+
ready;
|
|
98
|
+
constructor(config) {
|
|
99
|
+
this.config = config;
|
|
100
|
+
this.ready = this.init();
|
|
101
|
+
}
|
|
102
|
+
/** Dynamically import the SDK and wire the isolated export pipeline. */
|
|
103
|
+
async init() {
|
|
104
|
+
try {
|
|
105
|
+
const [tracing, otel, sdkNode] = await Promise.all([
|
|
106
|
+
import('@langfuse/tracing'),
|
|
107
|
+
import('@langfuse/otel'),
|
|
108
|
+
import('@opentelemetry/sdk-trace-node'),
|
|
109
|
+
]);
|
|
110
|
+
const processor = new otel.LangfuseSpanProcessor({
|
|
111
|
+
publicKey: this.config.publicKey,
|
|
112
|
+
secretKey: this.config.secretKey,
|
|
113
|
+
baseUrl: this.config.baseUrl,
|
|
114
|
+
});
|
|
115
|
+
this.provider = new sdkNode.NodeTracerProvider({ spanProcessors: [processor] });
|
|
116
|
+
// Isolated provider, not provider.register(): the process-global OTEL
|
|
117
|
+
// provider stays untouched. Set `tracing` last — observation methods
|
|
118
|
+
// gate on it, and it must never be visible before the provider is wired.
|
|
119
|
+
tracing.setLangfuseTracerProvider(this.provider);
|
|
120
|
+
this.tracing = tracing;
|
|
121
|
+
}
|
|
122
|
+
catch (error) {
|
|
123
|
+
console.error('[dsh-langfuse] client init failed:', error);
|
|
124
|
+
}
|
|
125
|
+
}
|
|
126
|
+
/**
|
|
127
|
+
* Stamp the trace-correlating attributes on an observation and record them
|
|
128
|
+
* for future children. `session.id` = `LangfuseOtelSpanAttributes.TRACE_SESSION_ID`
|
|
129
|
+
* (the OTEL semconv key); `langfuse.trace.name` = `TRACE_NAME`.
|
|
130
|
+
*/
|
|
131
|
+
stampTraceContext(observation, context) {
|
|
132
|
+
if (!context)
|
|
133
|
+
return;
|
|
134
|
+
observation.otelSpan.setAttribute('langfuse.trace.name', context.traceName);
|
|
135
|
+
if (context.sessionId)
|
|
136
|
+
observation.otelSpan.setAttribute('session.id', context.sessionId);
|
|
137
|
+
this.traceContext.set(observation, context);
|
|
138
|
+
}
|
|
139
|
+
/** Open a trace (one per agent turn, or one-off for session-less calls). */
|
|
140
|
+
openTrace(input) {
|
|
141
|
+
if (!this.tracing)
|
|
142
|
+
return null;
|
|
143
|
+
try {
|
|
144
|
+
const root = this.tracing.startObservation(input.name, { metadata: input.metadata }, { parentSpanContext: NO_PARENT });
|
|
145
|
+
this.stampTraceContext(root, { sessionId: input.sessionId, traceName: input.name });
|
|
146
|
+
this.roots.add(root);
|
|
147
|
+
return root;
|
|
148
|
+
}
|
|
149
|
+
catch (error) {
|
|
150
|
+
console.error('[dsh-langfuse] trace creation failed:', error);
|
|
151
|
+
return null;
|
|
152
|
+
}
|
|
153
|
+
}
|
|
154
|
+
/** Open a generation (one per LLM call) under any observation parent. */
|
|
155
|
+
startGeneration(parent, input) {
|
|
156
|
+
if (!parent)
|
|
157
|
+
return null;
|
|
158
|
+
try {
|
|
159
|
+
const generation = parent.startObservation(input.name, {
|
|
160
|
+
model: input.model,
|
|
161
|
+
input: input.input,
|
|
162
|
+
modelParameters: input.modelParameters,
|
|
163
|
+
metadata: input.metadata,
|
|
164
|
+
}, { asType: 'generation' });
|
|
165
|
+
this.stampTraceContext(generation, this.traceContext.get(parent));
|
|
166
|
+
return generation;
|
|
167
|
+
}
|
|
168
|
+
catch (error) {
|
|
169
|
+
console.error('[dsh-langfuse] generation creation failed:', error);
|
|
170
|
+
return null;
|
|
171
|
+
}
|
|
172
|
+
}
|
|
173
|
+
endGeneration(generation, update) {
|
|
174
|
+
if (!generation)
|
|
175
|
+
return;
|
|
176
|
+
try {
|
|
177
|
+
generation.update({
|
|
178
|
+
output: update.output,
|
|
179
|
+
usageDetails: update.usage ? usageOf(update.usage) : undefined,
|
|
180
|
+
completionStartTime: update.completionStartTime,
|
|
181
|
+
level: update.level,
|
|
182
|
+
statusMessage: update.statusMessage,
|
|
183
|
+
metadata: update.metadata,
|
|
184
|
+
});
|
|
185
|
+
// v5 observation attributes have no `name` — renames ride the OTEL span.
|
|
186
|
+
if (update.name)
|
|
187
|
+
generation.otelSpan.updateName(update.name);
|
|
188
|
+
generation.end();
|
|
189
|
+
}
|
|
190
|
+
catch (error) {
|
|
191
|
+
console.error('[dsh-langfuse] generation end failed:', error);
|
|
192
|
+
}
|
|
193
|
+
}
|
|
194
|
+
/** Open a span (one per tool dispatch, or a nested detail span) under any observation parent. */
|
|
195
|
+
startSpan(parent, input) {
|
|
196
|
+
if (!parent)
|
|
197
|
+
return null;
|
|
198
|
+
try {
|
|
199
|
+
const span = parent.startObservation(input.name, {
|
|
200
|
+
input: input.input,
|
|
201
|
+
metadata: input.metadata,
|
|
202
|
+
});
|
|
203
|
+
this.stampTraceContext(span, this.traceContext.get(parent));
|
|
204
|
+
return span;
|
|
205
|
+
}
|
|
206
|
+
catch (error) {
|
|
207
|
+
console.error('[dsh-langfuse] span creation failed:', error);
|
|
208
|
+
return null;
|
|
209
|
+
}
|
|
210
|
+
}
|
|
211
|
+
/**
|
|
212
|
+
* Merge fields into an open span — subagent enrichment (label, provider),
|
|
213
|
+
* and the trace root's input (a v5 trace IS its root span, so trace-level
|
|
214
|
+
* updates are span updates; OTEL attributes merge per key, so partial
|
|
215
|
+
* updates compose). On trace roots the input/output also rides the
|
|
216
|
+
* deprecated `langfuse.trace.*` attributes (setTraceIO): older Langfuse
|
|
217
|
+
* servers derive the trace row's IO from exactly those keys.
|
|
218
|
+
*/
|
|
219
|
+
updateSpan(span, update) {
|
|
220
|
+
if (!span)
|
|
221
|
+
return;
|
|
222
|
+
try {
|
|
223
|
+
span.update({ input: update.input, metadata: update.metadata });
|
|
224
|
+
if (update.name)
|
|
225
|
+
span.otelSpan.updateName(update.name);
|
|
226
|
+
if (update.input !== undefined && this.roots.has(span)) {
|
|
227
|
+
span.setTraceIO({ input: update.input });
|
|
228
|
+
}
|
|
229
|
+
}
|
|
230
|
+
catch (error) {
|
|
231
|
+
console.error('[dsh-langfuse] span update failed:', error);
|
|
232
|
+
}
|
|
233
|
+
}
|
|
234
|
+
/**
|
|
235
|
+
* Close any span observation, trace roots included: final fields, then end —
|
|
236
|
+
* v5 spans only export on end, so an un-ended root never reaches Langfuse.
|
|
237
|
+
* On trace roots the output also rides the deprecated `langfuse.trace.*`
|
|
238
|
+
* attributes (setTraceIO) for older servers — same rule as updateSpan.
|
|
239
|
+
*/
|
|
240
|
+
endSpan(span, update = {}) {
|
|
241
|
+
if (!span)
|
|
242
|
+
return;
|
|
243
|
+
try {
|
|
244
|
+
span.update(update);
|
|
245
|
+
if (update.output !== undefined && this.roots.has(span)) {
|
|
246
|
+
span.setTraceIO({ output: update.output });
|
|
247
|
+
}
|
|
248
|
+
span.end();
|
|
249
|
+
}
|
|
250
|
+
catch (error) {
|
|
251
|
+
console.error('[dsh-langfuse] span end failed:', error);
|
|
252
|
+
}
|
|
253
|
+
}
|
|
254
|
+
/** Drain buffered spans (the `session/flush` checkpoint). */
|
|
255
|
+
async flush() {
|
|
256
|
+
await this.ready;
|
|
257
|
+
if (!this.provider)
|
|
258
|
+
return;
|
|
259
|
+
try {
|
|
260
|
+
await this.provider.forceFlush();
|
|
261
|
+
}
|
|
262
|
+
catch (error) {
|
|
263
|
+
console.error('[dsh-langfuse] flush failed:', error);
|
|
264
|
+
}
|
|
265
|
+
}
|
|
266
|
+
/** Shut the exporter down at fiber unload (provider.shutdown flushes internally). */
|
|
267
|
+
async shutdown() {
|
|
268
|
+
await this.ready;
|
|
269
|
+
const provider = this.provider;
|
|
270
|
+
const tracing = this.tracing;
|
|
271
|
+
this.provider = null;
|
|
272
|
+
this.tracing = null;
|
|
273
|
+
if (!provider || !tracing)
|
|
274
|
+
return;
|
|
275
|
+
try {
|
|
276
|
+
await provider.shutdown();
|
|
277
|
+
}
|
|
278
|
+
catch (error) {
|
|
279
|
+
console.error('[dsh-langfuse] shutdown failed:', error);
|
|
280
|
+
}
|
|
281
|
+
finally {
|
|
282
|
+
// Release the module-global isolated-provider slot so a reloaded fiber
|
|
283
|
+
// starts clean — even when the shutdown flush failed.
|
|
284
|
+
tracing.setLangfuseTracerProvider(null);
|
|
285
|
+
}
|
|
286
|
+
}
|
|
287
|
+
}
|
package/lib/index.js
ADDED
|
@@ -0,0 +1,529 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* dsh-langfuse — Langfuse observability for DeepSeek Harness.
|
|
3
|
+
*
|
|
4
|
+
* Instruments the documented seams, with zero core patches:
|
|
5
|
+
*
|
|
6
|
+
* - `llm/stream` waterfall → one Langfuse **generation** per LLM call
|
|
7
|
+
* (input = full request; output, token usage and finish reason collected
|
|
8
|
+
* from the chunk stream), plus a nested `llm-request` **span** holding the
|
|
9
|
+
* verbatim loop-built request and the collected chunk stream (the rawest
|
|
10
|
+
* request/response a plugin can observe — the provider HTTP body is
|
|
11
|
+
* assembled inside the adapter).
|
|
12
|
+
* - `tools/execute` waterfall → one Langfuse **span** per tool dispatch
|
|
13
|
+
* (routed to the session's trace via `exec.agent.id` — the agent/session
|
|
14
|
+
* shared identity).
|
|
15
|
+
* - `session/event` emit → one Langfuse **trace** per agent turn
|
|
16
|
+
* (`turn/start` opens a root span — in the v5 SDK the root observation IS
|
|
17
|
+
* the trace — the turn's first `user/message` sets the input, `turn/end`
|
|
18
|
+
* stamps the end reason and ends the root, which is what exports it).
|
|
19
|
+
* - subagent child sessions (`session/created` with `header.parentSession`)
|
|
20
|
+
* ride the parent's trace: a `subagent` **span** opened under the enclosing
|
|
21
|
+
* delegation tool span, the child's generations/tool spans nested under it,
|
|
22
|
+
* closed at `subagent/end` (enriched by `subagent/start` and the child's
|
|
23
|
+
* `subagent/descriptor` event).
|
|
24
|
+
*
|
|
25
|
+
* Event/waterfall shapes verified against the installed
|
|
26
|
+
* @deepseek-ai/dsh-{llm,session,tools,agent,subagent}@0.1.0-rc.7 package sources
|
|
27
|
+
* (types for shapes; `lib/*.js` for the session/created ordering claim).
|
|
28
|
+
*
|
|
29
|
+
* @module dsh-langfuse
|
|
30
|
+
*/
|
|
31
|
+
import { isTokenDelta } from '@deepseek-ai/dsh-llm/message';
|
|
32
|
+
import Schema from '@deepseek-ai/schemastery';
|
|
33
|
+
import { LangfuseReporter } from './client.js';
|
|
34
|
+
export const name = 'dsh-langfuse';
|
|
35
|
+
/**
|
|
36
|
+
* Waterfall listeners only need the events to exist; no hard service
|
|
37
|
+
* dependency is declared so the plugin still loads in minimal compositions
|
|
38
|
+
* (it simply never observes anything). Add 'llm' / 'tools' to `inject` if you
|
|
39
|
+
* prefer fail-fast wiring.
|
|
40
|
+
*/
|
|
41
|
+
export const inject = [];
|
|
42
|
+
export const Config = Schema.object({
|
|
43
|
+
enabled: Schema.boolean().default(false).description('master switch'),
|
|
44
|
+
publicKey: Schema.string().default('').description('Langfuse public key'),
|
|
45
|
+
secretKey: Schema.string().role('secret').default('').description('Langfuse secret key'),
|
|
46
|
+
baseUrl: Schema.string().default('https://cloud.langfuse.com'),
|
|
47
|
+
traceName: Schema.string().default('dsh-turn'),
|
|
48
|
+
/** Include full message/tool payloads in Langfuse IO. Disable to log metadata only. */
|
|
49
|
+
captureContent: Schema.boolean().default(true),
|
|
50
|
+
});
|
|
51
|
+
const messageOf = (error) => error instanceof Error ? error.message : String(error);
|
|
52
|
+
/**
|
|
53
|
+
* Provider/tool error text can embed prompt fragments, paths, or credentials —
|
|
54
|
+
* it crosses to Langfuse only when `captureContent` allows content. Structural
|
|
55
|
+
* facts (levels, kinds, error codes) always do.
|
|
56
|
+
*/
|
|
57
|
+
const contentMessage = (config, text) => config.captureContent ? text : undefined;
|
|
58
|
+
/** Trace input/output text: the message's text blocks, joined. */
|
|
59
|
+
function messageText(message) {
|
|
60
|
+
const text = message.content.flatMap((block) => (block.type === 'text' ? [block.text] : []));
|
|
61
|
+
return text.length > 0 ? text.join('\n') : undefined;
|
|
62
|
+
}
|
|
63
|
+
/**
|
|
64
|
+
* Langfuse model parameters: only the request fields actually set, coerced to
|
|
65
|
+
* the v5 value type (`string | number` — the stop list serializes). Stop
|
|
66
|
+
* sequences are arbitrary request strings — content, not metadata — so under
|
|
67
|
+
* `captureContent: false` they collapse to a count.
|
|
68
|
+
*/
|
|
69
|
+
function modelParametersOf(options, captureContent) {
|
|
70
|
+
const params = Object.fromEntries(Object.entries({
|
|
71
|
+
temperature: options.temperature,
|
|
72
|
+
maxTokens: options.maxTokens,
|
|
73
|
+
stop: captureContent && options.stop ? JSON.stringify(options.stop) : undefined,
|
|
74
|
+
stopCount: captureContent ? undefined : options.stop?.length,
|
|
75
|
+
reasoningEffort: options.reasoningEffort,
|
|
76
|
+
}).filter(([, value]) => value !== undefined));
|
|
77
|
+
return Object.keys(params).length > 0 ? params : undefined;
|
|
78
|
+
}
|
|
79
|
+
/**
|
|
80
|
+
* The loop-built request as Langfuse input: verbatim (minus the AbortSignal,
|
|
81
|
+
* which is not JSON-safe) when content capture is on, counts-only otherwise.
|
|
82
|
+
*/
|
|
83
|
+
function requestBodyOf(options, captureContent) {
|
|
84
|
+
if (!captureContent) {
|
|
85
|
+
return {
|
|
86
|
+
messageCount: options.messages.length,
|
|
87
|
+
hasSystem: options.system !== undefined,
|
|
88
|
+
toolCount: options.tools?.length,
|
|
89
|
+
};
|
|
90
|
+
}
|
|
91
|
+
const { signal: _, ...body } = options;
|
|
92
|
+
return body;
|
|
93
|
+
}
|
|
94
|
+
/** Generation names carry the reply's first non-empty line (capped) for list readability. */
|
|
95
|
+
function enhanceWithFirstLine(base, text) {
|
|
96
|
+
const firstLine = text
|
|
97
|
+
.split('\n')
|
|
98
|
+
.map((line) => line.trim())
|
|
99
|
+
.find((line) => line.length > 0);
|
|
100
|
+
if (!firstLine)
|
|
101
|
+
return undefined;
|
|
102
|
+
return `${base} [${firstLine.length > 80 ? `${firstLine.slice(0, 79)}…` : firstLine}]`;
|
|
103
|
+
}
|
|
104
|
+
/** Tool argument keys worth surfacing in a span name, in priority order. */
|
|
105
|
+
const ARG_SUMMARY_KEYS = [
|
|
106
|
+
'path',
|
|
107
|
+
'file_path',
|
|
108
|
+
'filePath',
|
|
109
|
+
'command',
|
|
110
|
+
'description',
|
|
111
|
+
'query',
|
|
112
|
+
'name',
|
|
113
|
+
'url',
|
|
114
|
+
'instruction',
|
|
115
|
+
];
|
|
116
|
+
/** A short argument summary for span names: the first matching key's first line, trimmed and capped. */
|
|
117
|
+
function toolCallSummary(args) {
|
|
118
|
+
if (args === null || typeof args !== 'object' || Array.isArray(args))
|
|
119
|
+
return undefined;
|
|
120
|
+
for (const key of ARG_SUMMARY_KEYS) {
|
|
121
|
+
const value = args[key];
|
|
122
|
+
if (typeof value !== 'string')
|
|
123
|
+
continue;
|
|
124
|
+
const first = value.split('\n', 1)[0].trim();
|
|
125
|
+
if (first.length > 0)
|
|
126
|
+
return first.slice(0, 80);
|
|
127
|
+
}
|
|
128
|
+
return undefined;
|
|
129
|
+
}
|
|
130
|
+
export function apply(ctx, config) {
|
|
131
|
+
if (!config.enabled || !config.publicKey || !config.secretKey)
|
|
132
|
+
return;
|
|
133
|
+
const reporter = new LangfuseReporter(config);
|
|
134
|
+
const sessions = new Map();
|
|
135
|
+
const children = new Map();
|
|
136
|
+
/** Per-session stack of currently open tools/execute spans (for subagent parenting). */
|
|
137
|
+
const openToolSpans = new Map();
|
|
138
|
+
/**
|
|
139
|
+
* The trace an observation for `sessionId` belongs to: the session's open
|
|
140
|
+
* turn trace, else a per-session one-off trace shared by that session's
|
|
141
|
+
* out-of-turn calls (between-turn maintenance like compaction, sessions
|
|
142
|
+
* that started before the plugin loaded) so those observations stay in one
|
|
143
|
+
* trace instead of fragmenting per call. The next `turn/start` replaces it.
|
|
144
|
+
* Session-less hand-built calls get one trace each — there is no key to
|
|
145
|
+
* cache under. Subagent child sessions resolve through their parent chain
|
|
146
|
+
* (a tree by construction) to the root trace.
|
|
147
|
+
*/
|
|
148
|
+
function traceFor(sessionId) {
|
|
149
|
+
let id = sessionId;
|
|
150
|
+
while (id && children.has(id))
|
|
151
|
+
id = children.get(id)?.parentSessionId;
|
|
152
|
+
const existing = id ? sessions.get(id) : undefined;
|
|
153
|
+
if (existing)
|
|
154
|
+
return existing.trace;
|
|
155
|
+
const trace = reporter.openTrace({ name: config.traceName, sessionId: id });
|
|
156
|
+
// hasInput starts true: one-offs never take a turn's first-message input.
|
|
157
|
+
if (trace && id)
|
|
158
|
+
sessions.set(id, { trace, hasInput: true });
|
|
159
|
+
return trace;
|
|
160
|
+
}
|
|
161
|
+
/**
|
|
162
|
+
* The observation a generation/span for `sessionId` parents under: the
|
|
163
|
+
* child's own `subagent` span when there is one, else the turn trace.
|
|
164
|
+
*/
|
|
165
|
+
function parentFor(sessionId) {
|
|
166
|
+
const child = sessionId ? children.get(sessionId) : undefined;
|
|
167
|
+
return child?.span ?? traceFor(sessionId);
|
|
168
|
+
}
|
|
169
|
+
// ------------------------------------------------------------------
|
|
170
|
+
// Subagent child sessions: session/created precedes both subagent/start
|
|
171
|
+
// and the child's first generation, and the durable header already links
|
|
172
|
+
// child → parent (verified against dsh-subagent@0.1.0-rc.7).
|
|
173
|
+
// ------------------------------------------------------------------
|
|
174
|
+
ctx.on('session/created', (session) => {
|
|
175
|
+
const parentSessionId = session.header.parentSession;
|
|
176
|
+
if (!parentSessionId)
|
|
177
|
+
return;
|
|
178
|
+
const childId = session.id;
|
|
179
|
+
// Parent under the enclosing delegation tool span when one is open
|
|
180
|
+
// (foreground runs execute inside the parent's tools/execute), else the
|
|
181
|
+
// parent's own subagent span (nested delegation), else the root trace.
|
|
182
|
+
const parentObservation = openToolSpans.get(parentSessionId)?.at(-1) ??
|
|
183
|
+
children.get(parentSessionId)?.span ??
|
|
184
|
+
traceFor(parentSessionId);
|
|
185
|
+
const span = reporter.startSpan(parentObservation, {
|
|
186
|
+
name: 'subagent',
|
|
187
|
+
metadata: {
|
|
188
|
+
childSessionId: childId,
|
|
189
|
+
delegationDepth: session.header.delegationDepth,
|
|
190
|
+
},
|
|
191
|
+
});
|
|
192
|
+
children.set(childId, { parentSessionId, span, hasInput: false });
|
|
193
|
+
});
|
|
194
|
+
ctx.on('subagent/start', (info) => {
|
|
195
|
+
const child = children.get(info.id);
|
|
196
|
+
if (!child)
|
|
197
|
+
return; // remote (ACP) runs have no local session — nothing to nest
|
|
198
|
+
reporter.updateSpan(child.span, { metadata: { provider: info.provider, local: info.local } });
|
|
199
|
+
});
|
|
200
|
+
ctx.on('subagent/end', (info) => {
|
|
201
|
+
const child = children.get(info.id);
|
|
202
|
+
if (!child)
|
|
203
|
+
return;
|
|
204
|
+
children.delete(info.id);
|
|
205
|
+
reporter.endSpan(child.span, {
|
|
206
|
+
output: {
|
|
207
|
+
stopReason: info.stopReason,
|
|
208
|
+
...(config.captureContent ? { lastAssistantMessage: info.lastAssistantMessage } : {}),
|
|
209
|
+
},
|
|
210
|
+
level: info.stopReason === 'error'
|
|
211
|
+
? 'ERROR'
|
|
212
|
+
: info.stopReason === 'aborted'
|
|
213
|
+
? 'WARNING'
|
|
214
|
+
: 'DEFAULT',
|
|
215
|
+
});
|
|
216
|
+
});
|
|
217
|
+
// ------------------------------------------------------------------
|
|
218
|
+
// Trace lifecycle: one trace per turn (top-level sessions only — child
|
|
219
|
+
// sessions ride the parent's trace under their subagent span).
|
|
220
|
+
// ------------------------------------------------------------------
|
|
221
|
+
ctx.on('session/event', (session, event) => {
|
|
222
|
+
const sessionId = session.id;
|
|
223
|
+
const child = children.get(sessionId);
|
|
224
|
+
if (child) {
|
|
225
|
+
// The delegated prompt (the child's first user/message) is the
|
|
226
|
+
// subagent span's input; the descriptor enriches the span name.
|
|
227
|
+
if (event.type === 'user/message' && !child.hasInput && config.captureContent) {
|
|
228
|
+
reporter.updateSpan(child.span, { input: messageText(event.data) });
|
|
229
|
+
child.hasInput = true;
|
|
230
|
+
}
|
|
231
|
+
else if (event.type === 'subagent/descriptor') {
|
|
232
|
+
// The label is the delegation's model-authored description — content.
|
|
233
|
+
reporter.updateSpan(child.span, {
|
|
234
|
+
name: config.captureContent && event.data.label
|
|
235
|
+
? `subagent: ${event.data.label}`
|
|
236
|
+
: 'subagent',
|
|
237
|
+
metadata: { mode: event.data.mode, provider: event.data.provider },
|
|
238
|
+
});
|
|
239
|
+
}
|
|
240
|
+
return;
|
|
241
|
+
}
|
|
242
|
+
switch (event.type) {
|
|
243
|
+
case 'turn/start': {
|
|
244
|
+
// A leftover entry is either a between-turns one-off trace or a turn
|
|
245
|
+
// whose `turn/end` was never seen (crash-orphaned): end and replace
|
|
246
|
+
// it — v5 spans only export on end, so the stale root must close to
|
|
247
|
+
// keep what it already recorded.
|
|
248
|
+
const stale = sessions.get(sessionId);
|
|
249
|
+
if (stale)
|
|
250
|
+
reporter.endSpan(stale.trace);
|
|
251
|
+
const trace = reporter.openTrace({
|
|
252
|
+
name: config.traceName,
|
|
253
|
+
sessionId,
|
|
254
|
+
metadata: { turn: event.data.turn },
|
|
255
|
+
});
|
|
256
|
+
if (trace)
|
|
257
|
+
sessions.set(sessionId, { trace, hasInput: false });
|
|
258
|
+
break;
|
|
259
|
+
}
|
|
260
|
+
case 'user/message': {
|
|
261
|
+
const state = sessions.get(sessionId);
|
|
262
|
+
if (state && !state.hasInput && config.captureContent) {
|
|
263
|
+
reporter.updateSpan(state.trace, { input: messageText(event.data) });
|
|
264
|
+
state.hasInput = true;
|
|
265
|
+
}
|
|
266
|
+
break;
|
|
267
|
+
}
|
|
268
|
+
case 'assistant/message': {
|
|
269
|
+
const state = sessions.get(sessionId);
|
|
270
|
+
if (state && config.captureContent) {
|
|
271
|
+
state.lastAssistantText = messageText(event.data.message) ?? state.lastAssistantText;
|
|
272
|
+
}
|
|
273
|
+
break;
|
|
274
|
+
}
|
|
275
|
+
case 'turn/end': {
|
|
276
|
+
const state = sessions.get(sessionId);
|
|
277
|
+
if (state) {
|
|
278
|
+
sessions.delete(sessionId);
|
|
279
|
+
// The turn's final answer becomes the trace's output; ending the
|
|
280
|
+
// root span is what exports the trace (v5 exports spans on end).
|
|
281
|
+
reporter.endSpan(state.trace, {
|
|
282
|
+
output: state.lastAssistantText,
|
|
283
|
+
metadata: { turn: event.data.turn, endReason: event.data.reason.kind },
|
|
284
|
+
});
|
|
285
|
+
}
|
|
286
|
+
break;
|
|
287
|
+
}
|
|
288
|
+
}
|
|
289
|
+
});
|
|
290
|
+
ctx.on('session/disposed', (session) => {
|
|
291
|
+
const sessionId = session.id;
|
|
292
|
+
const state = sessions.get(sessionId);
|
|
293
|
+
sessions.delete(sessionId);
|
|
294
|
+
openToolSpans.delete(sessionId);
|
|
295
|
+
// End an abandoned root (crash-orphaned turn or between-turns one-off
|
|
296
|
+
// trace) — v5 spans only export on end, so dropping it open would lose
|
|
297
|
+
// everything it recorded.
|
|
298
|
+
if (state)
|
|
299
|
+
reporter.endSpan(state.trace);
|
|
300
|
+
// Children: keep the entry — background/continuable subagents dispose
|
|
301
|
+
// their session BEFORE subagent/end fires, and the span's close must ride
|
|
302
|
+
// subagent/end (disposing first would either leak the span or close it
|
|
303
|
+
// with a bogus WARNING, the latter seen on a healthy run in a real
|
|
304
|
+
// trace). A crashed run leaves the span open — it never exports.
|
|
305
|
+
});
|
|
306
|
+
// ------------------------------------------------------------------
|
|
307
|
+
// Generation per LLM call. The waterfall is SYNCHRONOUS (cordis returns
|
|
308
|
+
// the outermost listener's return value as the stream), so the listener
|
|
309
|
+
// must not be async — it returns an async-generator tee instead.
|
|
310
|
+
// ------------------------------------------------------------------
|
|
311
|
+
ctx.on('llm/stream', (options, next) => {
|
|
312
|
+
const parent = parentFor(options.sessionId);
|
|
313
|
+
// A session-less call's trace root has no session lifecycle to close it
|
|
314
|
+
// (it is never cached in `sessions`), so it rides this call's: un-ended
|
|
315
|
+
// spans never export in v5 — end it when the call settles.
|
|
316
|
+
const oneOffRoot = options.sessionId ? null : parent;
|
|
317
|
+
const generationName = options.purpose ? `llm-call [${options.purpose}]` : 'llm-call';
|
|
318
|
+
const generation = reporter.startGeneration(parent, {
|
|
319
|
+
name: generationName,
|
|
320
|
+
model: options.model,
|
|
321
|
+
input: config.captureContent
|
|
322
|
+
? { messages: options.messages, system: options.system, tools: options.tools }
|
|
323
|
+
: { messageCount: options.messages.length },
|
|
324
|
+
modelParameters: modelParametersOf(options, config.captureContent),
|
|
325
|
+
metadata: { purpose: options.purpose },
|
|
326
|
+
});
|
|
327
|
+
// The rawest request a plugin can observe: the loop-built GenerateOptions
|
|
328
|
+
// verbatim, pre-adapter (the provider HTTP body is assembled inside the
|
|
329
|
+
// adapter — dsh exposes no seam for it). Nested under the generation,
|
|
330
|
+
// closed alongside it with the same outcome; its output is the collected
|
|
331
|
+
// chunk stream (the raw response at this seam).
|
|
332
|
+
const requestSpan = reporter.startSpan(generation, {
|
|
333
|
+
name: 'llm-request',
|
|
334
|
+
input: requestBodyOf(options, config.captureContent),
|
|
335
|
+
metadata: { provider: options.provider },
|
|
336
|
+
});
|
|
337
|
+
const rawChunks = [];
|
|
338
|
+
const closeRequest = (level, statusMessage) => {
|
|
339
|
+
reporter.endSpan(requestSpan, {
|
|
340
|
+
output: config.captureContent ? rawChunks : undefined,
|
|
341
|
+
level,
|
|
342
|
+
statusMessage,
|
|
343
|
+
metadata: { chunkCount: rawChunks.length },
|
|
344
|
+
});
|
|
345
|
+
};
|
|
346
|
+
let stream;
|
|
347
|
+
try {
|
|
348
|
+
stream = next();
|
|
349
|
+
}
|
|
350
|
+
catch (error) {
|
|
351
|
+
const statusMessage = contentMessage(config, messageOf(error));
|
|
352
|
+
closeRequest('ERROR', statusMessage);
|
|
353
|
+
reporter.endGeneration(generation, { level: 'ERROR', statusMessage });
|
|
354
|
+
reporter.endSpan(oneOffRoot);
|
|
355
|
+
throw error;
|
|
356
|
+
}
|
|
357
|
+
// Tee the chunk stream: replay every chunk unchanged, then close the
|
|
358
|
+
// generation from the terminal `finish` chunk and accumulated content.
|
|
359
|
+
// Stream failures surface as a `finish` chunk with an error/aborted
|
|
360
|
+
// reason rather than throwing; the single close in `finally` also covers
|
|
361
|
+
// consumer abandonment and mid-stream throws.
|
|
362
|
+
return (async function* () {
|
|
363
|
+
let text = '';
|
|
364
|
+
const toolCalls = [];
|
|
365
|
+
let usage;
|
|
366
|
+
let finish;
|
|
367
|
+
let completionStartTime;
|
|
368
|
+
let thrownMessage;
|
|
369
|
+
const outputBody = () => config.captureContent
|
|
370
|
+
? toolCalls.length > 0
|
|
371
|
+
? { text, toolCalls }
|
|
372
|
+
: text || undefined
|
|
373
|
+
: undefined;
|
|
374
|
+
try {
|
|
375
|
+
for await (const chunk of stream) {
|
|
376
|
+
rawChunks.push(chunk);
|
|
377
|
+
// TTFT is the first real token — block boundaries, usage frames and
|
|
378
|
+
// empty heartbeat deltas are not completions (dsh's shared predicate).
|
|
379
|
+
if (completionStartTime === undefined && isTokenDelta(chunk)) {
|
|
380
|
+
completionStartTime = new Date();
|
|
381
|
+
}
|
|
382
|
+
switch (chunk.type) {
|
|
383
|
+
case 'text-delta':
|
|
384
|
+
text += chunk.text;
|
|
385
|
+
break;
|
|
386
|
+
case 'block-end':
|
|
387
|
+
if (chunk.block.type === 'tool-call') {
|
|
388
|
+
toolCalls.push({ name: chunk.block.name, arguments: chunk.block.arguments });
|
|
389
|
+
}
|
|
390
|
+
break;
|
|
391
|
+
case 'usage':
|
|
392
|
+
usage = chunk.usage;
|
|
393
|
+
break;
|
|
394
|
+
case 'finish':
|
|
395
|
+
finish = chunk.reason;
|
|
396
|
+
break;
|
|
397
|
+
}
|
|
398
|
+
yield chunk;
|
|
399
|
+
}
|
|
400
|
+
}
|
|
401
|
+
catch (error) {
|
|
402
|
+
// The raw message stays local — redaction gates what crosses to
|
|
403
|
+
// Langfuse, and `undefined` (not an empty string) is the no-throw
|
|
404
|
+
// discriminator.
|
|
405
|
+
thrownMessage = messageOf(error);
|
|
406
|
+
throw error;
|
|
407
|
+
}
|
|
408
|
+
finally {
|
|
409
|
+
const failure = finish && (finish.kind === 'error' || finish.kind === 'aborted') ? finish : undefined;
|
|
410
|
+
// No terminal finish chunk (the consumer abandoned the stream or
|
|
411
|
+
// the adapter cut it short): the call's outcome is unknown — mark
|
|
412
|
+
// it instead of posing as a normal completion.
|
|
413
|
+
let level = 'DEFAULT';
|
|
414
|
+
let statusMessage;
|
|
415
|
+
if (thrownMessage !== undefined) {
|
|
416
|
+
level = 'ERROR';
|
|
417
|
+
statusMessage = contentMessage(config, thrownMessage);
|
|
418
|
+
}
|
|
419
|
+
else if (failure) {
|
|
420
|
+
level = failure.kind === 'aborted' ? 'WARNING' : 'ERROR';
|
|
421
|
+
// `failure` is contractually required on error/aborted finishes,
|
|
422
|
+
// but a non-conformant adapter must never turn observability into
|
|
423
|
+
// a generator-level TypeError escaping into the agent loop.
|
|
424
|
+
statusMessage = contentMessage(config, failure.failure?.message ?? 'unknown failure');
|
|
425
|
+
}
|
|
426
|
+
else if (!finish) {
|
|
427
|
+
level = 'WARNING';
|
|
428
|
+
statusMessage = 'stream closed before the terminal finish chunk';
|
|
429
|
+
}
|
|
430
|
+
closeRequest(level, statusMessage);
|
|
431
|
+
// Partial progress counts on every path: whatever streamed is kept.
|
|
432
|
+
reporter.endGeneration(generation, {
|
|
433
|
+
name: enhanceWithFirstLine(generationName, text),
|
|
434
|
+
output: outputBody(),
|
|
435
|
+
usage,
|
|
436
|
+
completionStartTime,
|
|
437
|
+
level,
|
|
438
|
+
statusMessage,
|
|
439
|
+
metadata: {
|
|
440
|
+
finishReason: finish?.kind,
|
|
441
|
+
errorCode: failure?.failure?.code,
|
|
442
|
+
incomplete: finish ? undefined : true,
|
|
443
|
+
toolCallCount: toolCalls.length > 0 ? toolCalls.length : undefined,
|
|
444
|
+
},
|
|
445
|
+
});
|
|
446
|
+
reporter.endSpan(oneOffRoot);
|
|
447
|
+
}
|
|
448
|
+
})();
|
|
449
|
+
});
|
|
450
|
+
// ------------------------------------------------------------------
|
|
451
|
+
// Span per tool dispatch. This waterfall IS async: `next()` returns the
|
|
452
|
+
// normalized dispatch result promise. Open spans are tracked per session
|
|
453
|
+
// so a subagent span opened mid-dispatch (foreground delegation) parents
|
|
454
|
+
// under the enclosing tool span.
|
|
455
|
+
// ------------------------------------------------------------------
|
|
456
|
+
ctx.on('tools/execute', async (exec, next) => {
|
|
457
|
+
const sessionId = exec.agent?.id;
|
|
458
|
+
const parent = parentFor(sessionId);
|
|
459
|
+
// An agent-less dispatch's trace root has no session lifecycle to close
|
|
460
|
+
// it — it rides this dispatch's (same one-off rule as llm/stream).
|
|
461
|
+
const oneOffRoot = sessionId ? null : parent;
|
|
462
|
+
// Argument summaries are argument content — they ride captureContent.
|
|
463
|
+
const argSummary = config.captureContent ? toolCallSummary(exec.arguments) : undefined;
|
|
464
|
+
const span = reporter.startSpan(parent, {
|
|
465
|
+
name: `tool:${exec.name}${argSummary ? ` [${argSummary}]` : ''}`,
|
|
466
|
+
input: config.captureContent ? exec.arguments : undefined,
|
|
467
|
+
metadata: { callId: exec.callId, toolName: exec.name },
|
|
468
|
+
});
|
|
469
|
+
if (sessionId && span) {
|
|
470
|
+
openToolSpans.set(sessionId, [...(openToolSpans.get(sessionId) ?? []), span]);
|
|
471
|
+
}
|
|
472
|
+
try {
|
|
473
|
+
const result = await next();
|
|
474
|
+
reporter.endSpan(span, {
|
|
475
|
+
output: config.captureContent ? result.content : undefined,
|
|
476
|
+
level: result.isError ? 'ERROR' : 'DEFAULT',
|
|
477
|
+
statusMessage: result.isError ? contentMessage(config, result.error.message) : undefined,
|
|
478
|
+
metadata: { errorCode: result.isError ? result.error.info?.code : undefined },
|
|
479
|
+
});
|
|
480
|
+
return result;
|
|
481
|
+
}
|
|
482
|
+
catch (error) {
|
|
483
|
+
reporter.endSpan(span, {
|
|
484
|
+
level: 'ERROR',
|
|
485
|
+
statusMessage: contentMessage(config, messageOf(error)),
|
|
486
|
+
});
|
|
487
|
+
throw error;
|
|
488
|
+
}
|
|
489
|
+
finally {
|
|
490
|
+
reporter.endSpan(oneOffRoot);
|
|
491
|
+
if (sessionId && span) {
|
|
492
|
+
const remaining = (openToolSpans.get(sessionId) ?? []).filter((s) => s !== span);
|
|
493
|
+
if (remaining.length > 0)
|
|
494
|
+
openToolSpans.set(sessionId, remaining);
|
|
495
|
+
else
|
|
496
|
+
openToolSpans.delete(sessionId);
|
|
497
|
+
}
|
|
498
|
+
}
|
|
499
|
+
});
|
|
500
|
+
// Durability checkpoint: the store awaits every session/flush listener.
|
|
501
|
+
ctx.on('session/flush', () => reporter.flush());
|
|
502
|
+
// Lifecycle: this cordis fork has no ready/dispose events — cleanup rides
|
|
503
|
+
// the fiber unload via ctx.effect() (shutdown() chains behind the lazy
|
|
504
|
+
// import, so a still-importing SDK is never shut down mid-construction).
|
|
505
|
+
ctx.effect(() => {
|
|
506
|
+
return async () => {
|
|
507
|
+
// End every tracked observation still open before shutting the exporter
|
|
508
|
+
// down — v5 spans only export on end, so unloading mid-turn (profile
|
|
509
|
+
// reload) would otherwise silently drop them. (An in-flight generation
|
|
510
|
+
// isn't tracked; its own tee finally closes it if the stream settles.)
|
|
511
|
+
for (const state of sessions.values())
|
|
512
|
+
reporter.endSpan(state.trace);
|
|
513
|
+
for (const child of children.values())
|
|
514
|
+
reporter.endSpan(child.span);
|
|
515
|
+
for (const spans of openToolSpans.values()) {
|
|
516
|
+
for (const span of spans)
|
|
517
|
+
reporter.endSpan(span);
|
|
518
|
+
}
|
|
519
|
+
sessions.clear();
|
|
520
|
+
children.clear();
|
|
521
|
+
openToolSpans.clear();
|
|
522
|
+
await reporter.shutdown();
|
|
523
|
+
};
|
|
524
|
+
});
|
|
525
|
+
// Returning the init promise folds the lazy SDK import into this fiber's
|
|
526
|
+
// load transition: cordis awaits a plugin callback's thenable, so a boot
|
|
527
|
+
// that awaits plugin readiness can never observe the import window.
|
|
528
|
+
return reporter.ready;
|
|
529
|
+
}
|
|
@@ -0,0 +1,146 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Langfuse reporter — thin synchronous wrapper over the Langfuse JS SDK v5's
|
|
3
|
+
* OpenTelemetry-based tracing API (`@langfuse/tracing` + `@langfuse/otel`),
|
|
4
|
+
* and the plugin's single no-throw seam: every method swallows SDK failures
|
|
5
|
+
* with a `[dsh-langfuse]` console.error so observability can never break the
|
|
6
|
+
* agent loop.
|
|
7
|
+
*
|
|
8
|
+
* v5's observations-first data model has no trace object: a trace is the root
|
|
9
|
+
* observation plus the correlating attributes every observation carries. So:
|
|
10
|
+
* - a "trace" here is a root `LangfuseSpan` created with {@link NO_PARENT} —
|
|
11
|
+
* without it the SDK parents to whatever span is active in the host's
|
|
12
|
+
* ambient OTEL context (e.g. an instrumented HTTP server around dsh),
|
|
13
|
+
* smearing turn traces into foreign traces;
|
|
14
|
+
* - trace-level input/output/metadata live on that root span (the v5
|
|
15
|
+
* replacement for the removed `trace.update()`), and the root must be
|
|
16
|
+
* ENDED — spans only export on end, so an un-ended root never leaves the
|
|
17
|
+
* process;
|
|
18
|
+
* - the correlating `session.id` attribute is stamped on every observation
|
|
19
|
+
* via a handle-keyed WeakMap — the explicit-tree equivalent of v5's
|
|
20
|
+
* context-scoped `propagateAttributes()`, which cannot wrap this plugin's
|
|
21
|
+
* event-driven lifecycle.
|
|
22
|
+
*
|
|
23
|
+
* The SDK stack is a heavy optional peer set, so it loads via dynamic
|
|
24
|
+
* `import()` kicked off in the constructor — a disabled plugin never pays for
|
|
25
|
+
* it, and a missing or incompatible peer degrades the reporter to a no-op
|
|
26
|
+
* instead of breaking the plugin load. The exporter runs on an ISOLATED
|
|
27
|
+
* tracer provider (`setLangfuseTracerProvider`) rather than
|
|
28
|
+
* `provider.register()`: the process-global OTEL provider stays untouched and
|
|
29
|
+
* the host keeps its own tracing pipeline. {@link LangfuseReporter.ready}
|
|
30
|
+
* settles (never rejects) once the import+wiring finished: the plugin returns
|
|
31
|
+
* it from `apply()` so fiber readiness covers the import window, and
|
|
32
|
+
* {@link flush}/{@link shutdown} chain behind it. Observation calls landing
|
|
33
|
+
* before readiness are still dropped — a window that cannot exist once the
|
|
34
|
+
* plugin fiber has been awaited.
|
|
35
|
+
*/
|
|
36
|
+
import type { TokenUsage } from '@deepseek-ai/dsh-llm';
|
|
37
|
+
import type { LangfuseGeneration, LangfuseSpan } from '@langfuse/tracing';
|
|
38
|
+
export interface LangfuseConnectionConfig {
|
|
39
|
+
publicKey: string;
|
|
40
|
+
secretKey: string;
|
|
41
|
+
baseUrl: string;
|
|
42
|
+
}
|
|
43
|
+
export type ObservationLevel = 'DEFAULT' | 'WARNING' | 'ERROR';
|
|
44
|
+
/** Any observation that can parent a span (trace root span or nested span/generation). */
|
|
45
|
+
type Observation = LangfuseSpan | LangfuseGeneration;
|
|
46
|
+
/**
|
|
47
|
+
* Map dsh token accounting onto Langfuse's `usageDetails`, keeping every
|
|
48
|
+
* bucket mutually exclusive (Langfuse's flat-bucket rule: `input` excludes
|
|
49
|
+
* `input_*`, `output` excludes `output_*`, `total` is the bucket sum —
|
|
50
|
+
* overlapping buckets double-count usage and inferred cost). dsh reports
|
|
51
|
+
* uncached input, separate cache buckets, and provider-style output that
|
|
52
|
+
* INCLUDES reasoning (verified in dsh-llm-deepseek@0.1.0-rc.7:
|
|
53
|
+
* `outputTokens: usage.completion_tokens`, with reasoning split out of
|
|
54
|
+
* `completion_tokens_details`), so the `output` bucket subtracts
|
|
55
|
+
* `reasoningTokens` into `output_reasoning`. Only `usageDetails` is sent.
|
|
56
|
+
*/
|
|
57
|
+
export declare function usageOf(usage: TokenUsage): Record<string, number>;
|
|
58
|
+
export declare class LangfuseReporter {
|
|
59
|
+
private config;
|
|
60
|
+
private tracing;
|
|
61
|
+
private provider;
|
|
62
|
+
/**
|
|
63
|
+
* Observation → its trace's correlating attributes; children inherit their
|
|
64
|
+
* parent's. Stamped on every observation (`session.id`,
|
|
65
|
+
* `langfuse.trace.name`): v5's observations-first model wants trace context
|
|
66
|
+
* on every span (the v4 migration doc: "copied to every span where the name
|
|
67
|
+
* must be queryable"), and older Langfuse servers derive the trace row from
|
|
68
|
+
* ANY span carrying these — a root-only stamp can lose to child-derived
|
|
69
|
+
* trace events depending on ingestion order. This is the explicit
|
|
70
|
+
* handle-tree equivalent of v5's context-scoped propagateAttributes, which
|
|
71
|
+
* cannot wrap this plugin's event-driven lifecycle.
|
|
72
|
+
*/
|
|
73
|
+
private traceContext;
|
|
74
|
+
/** Trace roots (created by openTrace) — the spans that may carry trace-level IO. */
|
|
75
|
+
private roots;
|
|
76
|
+
/** Settles (never rejects) once the lazy SDK import+wiring finished. */
|
|
77
|
+
readonly ready: Promise<void>;
|
|
78
|
+
constructor(config: LangfuseConnectionConfig);
|
|
79
|
+
/** Dynamically import the SDK and wire the isolated export pipeline. */
|
|
80
|
+
private init;
|
|
81
|
+
/**
|
|
82
|
+
* Stamp the trace-correlating attributes on an observation and record them
|
|
83
|
+
* for future children. `session.id` = `LangfuseOtelSpanAttributes.TRACE_SESSION_ID`
|
|
84
|
+
* (the OTEL semconv key); `langfuse.trace.name` = `TRACE_NAME`.
|
|
85
|
+
*/
|
|
86
|
+
private stampTraceContext;
|
|
87
|
+
/** Open a trace (one per agent turn, or one-off for session-less calls). */
|
|
88
|
+
openTrace(input: {
|
|
89
|
+
name: string;
|
|
90
|
+
sessionId?: string;
|
|
91
|
+
metadata?: Record<string, unknown>;
|
|
92
|
+
}): LangfuseSpan | null;
|
|
93
|
+
/** Open a generation (one per LLM call) under any observation parent. */
|
|
94
|
+
startGeneration(parent: Observation | null, input: {
|
|
95
|
+
name: string;
|
|
96
|
+
model?: string;
|
|
97
|
+
input?: unknown;
|
|
98
|
+
modelParameters?: Record<string, string | number>;
|
|
99
|
+
metadata?: Record<string, unknown>;
|
|
100
|
+
}): LangfuseGeneration | null;
|
|
101
|
+
endGeneration(generation: LangfuseGeneration | null, update: {
|
|
102
|
+
name?: string;
|
|
103
|
+
output?: unknown;
|
|
104
|
+
usage?: TokenUsage;
|
|
105
|
+
completionStartTime?: Date;
|
|
106
|
+
level?: ObservationLevel;
|
|
107
|
+
statusMessage?: string;
|
|
108
|
+
metadata?: Record<string, unknown>;
|
|
109
|
+
}): void;
|
|
110
|
+
/** Open a span (one per tool dispatch, or a nested detail span) under any observation parent. */
|
|
111
|
+
startSpan(parent: Observation | null, input: {
|
|
112
|
+
name: string;
|
|
113
|
+
input?: unknown;
|
|
114
|
+
metadata?: Record<string, unknown>;
|
|
115
|
+
}): LangfuseSpan | null;
|
|
116
|
+
/**
|
|
117
|
+
* Merge fields into an open span — subagent enrichment (label, provider),
|
|
118
|
+
* and the trace root's input (a v5 trace IS its root span, so trace-level
|
|
119
|
+
* updates are span updates; OTEL attributes merge per key, so partial
|
|
120
|
+
* updates compose). On trace roots the input/output also rides the
|
|
121
|
+
* deprecated `langfuse.trace.*` attributes (setTraceIO): older Langfuse
|
|
122
|
+
* servers derive the trace row's IO from exactly those keys.
|
|
123
|
+
*/
|
|
124
|
+
updateSpan(span: LangfuseSpan | null, update: {
|
|
125
|
+
name?: string;
|
|
126
|
+
input?: unknown;
|
|
127
|
+
metadata?: Record<string, unknown>;
|
|
128
|
+
}): void;
|
|
129
|
+
/**
|
|
130
|
+
* Close any span observation, trace roots included: final fields, then end —
|
|
131
|
+
* v5 spans only export on end, so an un-ended root never reaches Langfuse.
|
|
132
|
+
* On trace roots the output also rides the deprecated `langfuse.trace.*`
|
|
133
|
+
* attributes (setTraceIO) for older servers — same rule as updateSpan.
|
|
134
|
+
*/
|
|
135
|
+
endSpan(span: LangfuseSpan | null, update?: {
|
|
136
|
+
output?: unknown;
|
|
137
|
+
level?: ObservationLevel;
|
|
138
|
+
statusMessage?: string;
|
|
139
|
+
metadata?: Record<string, unknown>;
|
|
140
|
+
}): void;
|
|
141
|
+
/** Drain buffered spans (the `session/flush` checkpoint). */
|
|
142
|
+
flush(): Promise<void>;
|
|
143
|
+
/** Shut the exporter down at fiber unload (provider.shutdown flushes internally). */
|
|
144
|
+
shutdown(): Promise<void>;
|
|
145
|
+
}
|
|
146
|
+
export {};
|
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* dsh-langfuse — Langfuse observability for DeepSeek Harness.
|
|
3
|
+
*
|
|
4
|
+
* Instruments the documented seams, with zero core patches:
|
|
5
|
+
*
|
|
6
|
+
* - `llm/stream` waterfall → one Langfuse **generation** per LLM call
|
|
7
|
+
* (input = full request; output, token usage and finish reason collected
|
|
8
|
+
* from the chunk stream), plus a nested `llm-request` **span** holding the
|
|
9
|
+
* verbatim loop-built request and the collected chunk stream (the rawest
|
|
10
|
+
* request/response a plugin can observe — the provider HTTP body is
|
|
11
|
+
* assembled inside the adapter).
|
|
12
|
+
* - `tools/execute` waterfall → one Langfuse **span** per tool dispatch
|
|
13
|
+
* (routed to the session's trace via `exec.agent.id` — the agent/session
|
|
14
|
+
* shared identity).
|
|
15
|
+
* - `session/event` emit → one Langfuse **trace** per agent turn
|
|
16
|
+
* (`turn/start` opens a root span — in the v5 SDK the root observation IS
|
|
17
|
+
* the trace — the turn's first `user/message` sets the input, `turn/end`
|
|
18
|
+
* stamps the end reason and ends the root, which is what exports it).
|
|
19
|
+
* - subagent child sessions (`session/created` with `header.parentSession`)
|
|
20
|
+
* ride the parent's trace: a `subagent` **span** opened under the enclosing
|
|
21
|
+
* delegation tool span, the child's generations/tool spans nested under it,
|
|
22
|
+
* closed at `subagent/end` (enriched by `subagent/start` and the child's
|
|
23
|
+
* `subagent/descriptor` event).
|
|
24
|
+
*
|
|
25
|
+
* Event/waterfall shapes verified against the installed
|
|
26
|
+
* @deepseek-ai/dsh-{llm,session,tools,agent,subagent}@0.1.0-rc.7 package sources
|
|
27
|
+
* (types for shapes; `lib/*.js` for the session/created ordering claim).
|
|
28
|
+
*
|
|
29
|
+
* @module dsh-langfuse
|
|
30
|
+
*/
|
|
31
|
+
import type { Context } from '@deepseek-ai/cordis';
|
|
32
|
+
import Schema from '@deepseek-ai/schemastery';
|
|
33
|
+
export declare const name = "dsh-langfuse";
|
|
34
|
+
/**
|
|
35
|
+
* Waterfall listeners only need the events to exist; no hard service
|
|
36
|
+
* dependency is declared so the plugin still loads in minimal compositions
|
|
37
|
+
* (it simply never observes anything). Add 'llm' / 'tools' to `inject` if you
|
|
38
|
+
* prefer fail-fast wiring.
|
|
39
|
+
*/
|
|
40
|
+
export declare const inject: string[];
|
|
41
|
+
export declare const Config: Schema<Schemastery.ObjectS<{
|
|
42
|
+
enabled: Schema<boolean, boolean>;
|
|
43
|
+
publicKey: Schema<string, string>;
|
|
44
|
+
secretKey: Schema<string, string>;
|
|
45
|
+
baseUrl: Schema<string, string>;
|
|
46
|
+
traceName: Schema<string, string>;
|
|
47
|
+
/** Include full message/tool payloads in Langfuse IO. Disable to log metadata only. */
|
|
48
|
+
captureContent: Schema<boolean, boolean>;
|
|
49
|
+
}>, Schemastery.ObjectT<{
|
|
50
|
+
enabled: Schema<boolean, boolean>;
|
|
51
|
+
publicKey: Schema<string, string>;
|
|
52
|
+
secretKey: Schema<string, string>;
|
|
53
|
+
baseUrl: Schema<string, string>;
|
|
54
|
+
traceName: Schema<string, string>;
|
|
55
|
+
/** Include full message/tool payloads in Langfuse IO. Disable to log metadata only. */
|
|
56
|
+
captureContent: Schema<boolean, boolean>;
|
|
57
|
+
}>>;
|
|
58
|
+
export interface LangfusePluginConfig {
|
|
59
|
+
enabled: boolean;
|
|
60
|
+
publicKey: string;
|
|
61
|
+
secretKey: string;
|
|
62
|
+
baseUrl: string;
|
|
63
|
+
traceName: string;
|
|
64
|
+
captureContent: boolean;
|
|
65
|
+
}
|
|
66
|
+
export declare function apply(ctx: Context, config: LangfusePluginConfig): Promise<void> | void;
|
package/package.json
ADDED
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "@amaster.ai/dsh-langfuse",
|
|
3
|
+
"version": "0.1.0",
|
|
4
|
+
"description": "Langfuse observability plugin for DeepSeek Harness: one generation per LLM call with a nested llm-request span, one span per tool call, one trace per session turn, subagent child sessions nested under the parent's tree.",
|
|
5
|
+
"type": "module",
|
|
6
|
+
"main": "lib/index.js",
|
|
7
|
+
"types": "lib/types/index.d.ts",
|
|
8
|
+
"exports": {
|
|
9
|
+
".": {
|
|
10
|
+
"types": "./lib/types/index.d.ts",
|
|
11
|
+
"default": "./lib/index.js"
|
|
12
|
+
},
|
|
13
|
+
"./package.json": "./package.json"
|
|
14
|
+
},
|
|
15
|
+
"files": [
|
|
16
|
+
"lib",
|
|
17
|
+
"cordis.patch.yml",
|
|
18
|
+
"preview.png"
|
|
19
|
+
],
|
|
20
|
+
"license": "MIT",
|
|
21
|
+
"publishConfig": {
|
|
22
|
+
"access": "public"
|
|
23
|
+
},
|
|
24
|
+
"repository": {
|
|
25
|
+
"type": "git",
|
|
26
|
+
"url": "https://github.com/TGYD-helige/dsh-plugins.git",
|
|
27
|
+
"directory": "packages/dsh-langfuse"
|
|
28
|
+
},
|
|
29
|
+
"dsh": {
|
|
30
|
+
"bundle": {
|
|
31
|
+
"patch": "./cordis.patch.yml"
|
|
32
|
+
}
|
|
33
|
+
},
|
|
34
|
+
"dependencies": {
|
|
35
|
+
"@deepseek-ai/schemastery": "^3.18.1"
|
|
36
|
+
},
|
|
37
|
+
"peerDependencies": {
|
|
38
|
+
"@deepseek-ai/cordis": "^4.0.1",
|
|
39
|
+
"@deepseek-ai/dsh-llm": "0.1.0-rc.7",
|
|
40
|
+
"@deepseek-ai/dsh-session": "0.1.0-rc.7",
|
|
41
|
+
"@deepseek-ai/dsh-subagent": "0.1.0-rc.7",
|
|
42
|
+
"@deepseek-ai/dsh-tools": "0.1.0-rc.7",
|
|
43
|
+
"@langfuse/otel": "^5.10.1",
|
|
44
|
+
"@langfuse/tracing": "^5.10.1",
|
|
45
|
+
"@opentelemetry/sdk-trace-node": "^2.10.0"
|
|
46
|
+
},
|
|
47
|
+
"peerDependenciesMeta": {
|
|
48
|
+
"@langfuse/otel": {
|
|
49
|
+
"optional": true
|
|
50
|
+
},
|
|
51
|
+
"@langfuse/tracing": {
|
|
52
|
+
"optional": true
|
|
53
|
+
},
|
|
54
|
+
"@opentelemetry/sdk-trace-node": {
|
|
55
|
+
"optional": true
|
|
56
|
+
}
|
|
57
|
+
},
|
|
58
|
+
"devDependencies": {
|
|
59
|
+
"@deepseek-ai/cordis": "^4.0.1",
|
|
60
|
+
"@deepseek-ai/dsh-agent": "0.1.0-rc.7",
|
|
61
|
+
"@deepseek-ai/dsh-llm": "0.1.0-rc.7",
|
|
62
|
+
"@deepseek-ai/dsh-session": "0.1.0-rc.7",
|
|
63
|
+
"@deepseek-ai/dsh-subagent": "0.1.0-rc.7",
|
|
64
|
+
"@deepseek-ai/dsh-tools": "0.1.0-rc.7",
|
|
65
|
+
"@langfuse/otel": "5.10.1",
|
|
66
|
+
"@langfuse/tracing": "5.10.1",
|
|
67
|
+
"@opentelemetry/api": "^1.9.1",
|
|
68
|
+
"@opentelemetry/sdk-trace-node": "^2.10.0",
|
|
69
|
+
"@types/node": "^22.0.0",
|
|
70
|
+
"typescript": "^5.6.3",
|
|
71
|
+
"vitest": "^4.1.11"
|
|
72
|
+
},
|
|
73
|
+
"engines": {
|
|
74
|
+
"node": "^20.19 || ^22.12 || >=24.0"
|
|
75
|
+
},
|
|
76
|
+
"scripts": {
|
|
77
|
+
"build": "tsc -p tsconfig.build.json",
|
|
78
|
+
"typecheck": "tsc -p tsconfig.json --noEmit",
|
|
79
|
+
"test": "vitest run"
|
|
80
|
+
}
|
|
81
|
+
}
|
package/preview.png
ADDED
|
Binary file
|