@amaster.ai/dsh-langfuse 0.1.23 → 0.1.25

package/README.md

@@ -5,7 +5,7 @@
 Langfuse observability for [DeepSeek Harness (dsh)](https://github.com/deepseek-ai/deepseek-harness):
 
 - one **generation** per LLM call (`llm/stream` waterfall), plus a nested `llm-request` span carrying the verbatim loop-built request
-- one **tool observation** per tool call (`tools/execute` waterfall)
+- one **tool observation** per tool call (`tools/pre-execute` through `tools/result`)
 - one **span** per `hook/invoked` / `hook/result` pair, with hook point, decision, exit code, and duration (stderr summary follows `captureContent`)
 - one **trace** per session turn (`session/event`) — in the v5 SDK the trace IS its root span, ended (and thereby exported) at `turn/end`
 - subagent child sessions nested under the parent's tree (`session/created` header link + `subagent/start` / `subagent/end`)
@@ -87,7 +87,7 @@ Generation `usageDetails` has disjoint `input`, `cache_read_input_tokens`, `cach
 
 Compaction and session-title calls passing through `llm/stream` already receive their own generation and `purpose` metadata, including out-of-turn calls. Tools using the same LLM service are observed there too; adding a second generation from tool-result usage would duplicate accounting.
 
-Hook spans appear when a dsh hook bridge emits the paired session records. The current `hook/result` record does not contain stdout; successful script output can appear in the LLM request when the bridge injects it as context, while the hook span reports decision, exit code, duration, and bounded stderr. `SessionStart` runs before the first turn and does not emit these records, so it has no hook span.
+Hook spans appear when a dsh hook bridge emits the paired session records. With content capture enabled, they show `hook:<point>:<name>`, optional input and parsed result, and available bounded process output alongside execution status. Producers must emit these fields; upgrading this plugin cannot recover omitted data. `SessionStart` only appears if a producer reports it and the installed runtime admits the records.
 
 ### Custom hook producers
 
@@ -115,6 +115,7 @@ appendHookEvent(session, 'hook/invoked', {
   ...identity,
   dialect: 'company-validator',
   name: 'Validate request',
+  input: { request: 'example' },
 });
 // After execution, including exceptions, timeout or cancellation:
 appendHookEvent(session, 'hook/result', {
@@ -123,22 +124,29 @@ appendHookEvent(session, 'hook/result', {
   outcome: 'success',
   exitCode: 0,
   durationMs: 37,
+  result: { reason: 'valid', additionalContext: 'checked', continue: true },
 });
 ```
 
-Every custom execution should have a unique `invocationId` and a result emitted in the producer's `finally` path. Both records must share `turn`, `point`, `handlerId`, and `invocationId`. `handlerId` identifies the configured handler; `invocationId` identifies one execution, including concurrent executions of that handler. Optional `name` labels the hook. Unmatched/skipped hooks must not emit an invocation: these records represent actual execution. The producer bounds `stderrSummary` and omits unavailable optional fields rather than writing `undefined` into the JSON session log.
+Every custom execution should have a unique `invocationId` and a result emitted in the producer's `finally` path. Both records must share `turn`, `point`, `handlerId`, and `invocationId`. `handlerId` identifies the configured handler; `invocationId` identifies one execution, including concurrent executions of that handler. Optional `name` labels the observation when content capture is enabled. `HookInvoked.input` carries the script payload; `HookResult.result` carries the parsed output object, including available reasons, additional context and continue/stop instructions. The consumer preserves these producer-defined fields without interpreting them as decisions. Unmatched/skipped hooks must not emit an invocation: these records represent actual execution. Optional `stdout`, `stderr` and legacy `stderrSummary` are masked, then limited to 4096 characters each in observations, with `stdoutTruncated`, `stderrTruncated` and `stderrSummaryTruncated` metadata. Producers should also bound persisted process output and omit unavailable optional fields rather than writing `undefined` into the JSON session log.
+
+Tool hooks should include the execution's `callId` in both records. The plugin matches it within the originating session and turn; it never chooses the last active tool. Tool observations begin at `tools/pre-execute` (a prepended observer) and end at `tools/result`, covering synchronous/awaited BeforeTool and AfterTool hooks, pre-dispatch denials, and final post-policy output. Bridges must run BeforeTool within the pre-execute waterfall and AfterTool within post-execute, before the final result notification. A bridge registered with prepend ahead of this observer must delegate before reporting its hook. Agent hooks and legacy records without `callId` remain under the turn/subagent. Unknown or already finished calls fall back to that parent; late invocations outside the originating turn use the detached-root behavior below.
 
 `hook/result.outcome` describes process execution independently of `decision`: `success` is normal completion, `failed` is execution failure, `timed-out` is deadline termination, and `cancelled` is cancellation. The producer maps its process semantics to these values; an exit code expressing a policy denial can still be a successful execution. Optional `exitCode` and `signal` preserve process details. Failed/timed-out executions receive ERROR observations, cancellation receives WARNING, and successful executions retain decision-based warnings. Records without an explicit outcome preserve the legacy exit-code interpretation (0/2 are non-error). The plugin records `durationMs` and never infers a timeout or cancellation from a missing exit code.
 
-For records admitted by the session, runs with `invocationId` retain correlation after `turn/end` and a subsequent `turn/start`; a late result ends the original hook span without attaching it to the new turn. Invocations arriving outside their originating turn receive a separate root with `detachedHook: true` and the original turn number; their result ends both hook and root so they export without waiting for another turn. These detached hook roots do not consume `langfuse/context` queued for the next turn. Detached producers must finish reporting before the session is disposed. Session disposal/plugin unload closes unfinished runs with `hook result not observed`; this means missing telemetry, not proof of execution failure. Legacy records without `invocationId` still close at turn boundaries.
+For records admitted by the session, runs with `invocationId` retain correlation after `turn/end` and a subsequent `turn/start`; a late result ends the original hook span without attaching it to the new turn. Delayed invocations, including `UserCancel`, reuse their originating turn's parent span even after the next turn starts. The plugin retains the latest 128 completed-turn associations across all sessions, clearing a session's associations on disposal and all associations on unload. Reporting never reopens the ended root or changes its end time or final status; each hook records its own duration. When the originating parent is unavailable or evicted, the invocation receives a separate root with `detachedHook: true` and the original turn number; its result ends both hook and fallback root. Neither path consumes `langfuse/context` queued for the next turn. Detached producers must finish reporting before the session is disposed. Session disposal/plugin unload closes unfinished runs with `hook result not observed`; this means missing telemetry, not proof of execution failure. Legacy records without `invocationId` still close at turn boundaries.
 
-Custom names and stderr follow `captureContent`; credentials and `redactFields` are masked by the existing reporter. New outcomes, correlation IDs, dialect, point, duration and process metadata remain available with content capture disabled. The event writer does not sanitize the session log itself: application producers remain responsible for what they persist. Applications must explicitly replace any private hook events with this standard producer path, reuse the same `invocationId`, and report the actual outcome; installing/upgrading the plugin does not adapt an application bridge or repair historical Trace overview IO (Issue #59).
+Custom names, hook input, parsed results and raw process output follow `captureContent`; credentials and `redactFields` are masked by the existing reporter. New outcomes, correlation IDs, dialect, point, duration and process metadata remain available with content capture disabled. The event writer does not sanitize the session log itself: application producers remain responsible for what they persist. Applications must explicitly replace any private hook events with this standard producer path, reuse the same `invocationId`, and report the actual outcome; installing/upgrading the plugin does not adapt an application bridge or repair historical Trace overview IO (Issue #59).
 
 Observability is a no-throw seam: backend errors are logged with a `[dsh-langfuse]` prefix and never escape into the agent loop.
 
 ## Compatibility
 
-Run `pnpm test:sdk` at the repository root for a secrets-free check using the real Langfuse SDK and the existing local OTLP receiver. It verifies export, native tool type, SDK label redaction, ambient-baggage isolation, external trace ownership, agent-less tool unload, and custom hook export across turn closure. It is separate from the mocked unit suite and the gateway/real-Langfuse E2E legs.
+Run `pnpm test:sdk` at the repository root for a secrets-free check using the real Langfuse SDK and the existing local OTLP receiver. It verifies export, native tool type, SDK label redaction, ambient-baggage isolation, external trace ownership, agent-less tool unload, the real tool runtime and hook processes, tool-hook parenting and enclosing timestamps, content masking, and custom hook export across turn closure. It is separate from the mocked unit suite and the gateway/real-Langfuse E2E legs.
+
+The independent `dsh-langfuse.tool-hooks` E2E leg boots the packed plugin in the real headless profile, asks the model to read a marker file, and executes BeforeTool/AfterTool scripts through an E2E-only producer. It requires both gateway and Langfuse credentials, then polls the real Observations API to verify named hooks, tool-call parenting, enclosing tool timestamps, script input and parsed output, credential masking, and stdout truncation. The existing `dsh-langfuse.hook` leg continues to check the upstream Codex UserPromptSubmit bridge; the new fixture verifies the shared event contract and does not substitute for the application's Gemini bridge integration.
+
+The `dsh-langfuse.cancel-hook` E2E leg cancels a real agent at a tool boundary, queues its next turn, and asynchronously runs a UserCancel process after that turn starts. Real Langfuse API read-back must show the hook under the cancelled turn's root and trace ID, with the root still aborted and ended before the hook's own duration. It requires the existing gateway and Langfuse secrets in `integration.yml`.
 
 ### Trace overview compatibility
 

package/lib/hooks.js

@@ -8,6 +8,7 @@ function baseFields(data) {
         value.point.length > 0 &&
         typeof value.handlerId === 'string' &&
         value.handlerId.length > 0 &&
+        (value.callId === undefined || (typeof value.callId === 'string' && value.callId.length > 0)) &&
         (value.invocationId === undefined ||
             (typeof value.invocationId === 'string' && value.invocationId.length > 0)));
 }
@@ -27,7 +28,11 @@ export function isHookResult(data) {
             ['success', 'failed', 'timed-out', 'cancelled'].includes(data.outcome)) &&
         (data.exitCode === undefined || Number.isSafeInteger(data.exitCode)) &&
         (data.signal === undefined || typeof data.signal === 'string') &&
-        (data.stderrSummary === undefined || typeof data.stderrSummary === 'string'));
+        (data.stderrSummary === undefined || typeof data.stderrSummary === 'string') &&
+        (data.stdout === undefined || typeof data.stdout === 'string') &&
+        (data.stderr === undefined || typeof data.stderr === 'string') &&
+        (data.result === undefined ||
+            (data.result !== null && typeof data.result === 'object' && !Array.isArray(data.result))));
 }
 /**
  * Append standard telemetry without changing hook decisions.

package/lib/index.js

@@ -8,7 +8,7 @@
  *   from the chunk stream), plus a nested `llm-request` **span** holding the
  *   same options and collected chunk stream (the rawest request/response a
  *   plugin can observe — the provider HTTP body is assembled inside the adapter).
- * - `tools/execute` waterfall → one Langfuse **span** per tool dispatch
+ * - `tools/pre-execute` → `tools/result` → one Langfuse **span** per tool pipeline
  *   (routed to the session's trace via `exec.agent.id` — the agent/session
  *   shared identity).
  * - `hook/invoked` / `hook/result` session records → one **span** per hook run.
@@ -204,8 +204,21 @@ export function apply(ctx, config) {
     const reporter = new LangfuseReporter(config);
     const sessions = new Map();
     const children = new Map();
-    /** Per-session stack of currently open tools/execute spans (for subagent parenting). */
+    const completedTurns = new Map();
+    function rememberTurn(sessionId, turn, parent, turnTrace) {
+        if (!parent)
+            return;
+        const key = JSON.stringify([sessionId, turn]);
+        completedTurns.delete(key);
+        completedTurns.set(key, { sessionId, parent, turnTrace });
+        // Bound retained SDK handles across all sessions; older hooks use the standalone fallback.
+        if (completedTurns.size > 128)
+            completedTurns.delete(completedTurns.keys().next().value);
+    }
+    /** Per-session stack of currently open tool pipeline spans (for subagent parenting). */
     const openToolSpans = new Map();
+    const toolCalls = new Map();
+    const toolResults = new WeakMap();
     const openHooks = new Map();
     const openOperations = new Set();
     const traceContexts = new Map();
@@ -395,7 +408,10 @@ export function apply(ctx, config) {
                 reporter.endSpan(previous.root, { level: 'WARNING' });
             const child = children.get(sessionId);
             const inTurn = child ? child.turn === data.turn : sessions.get(sessionId)?.turn === data.turn;
-            const root = data.invocationId && !inTurn
+            const completed = data.invocationId && !inTurn
+                ? completedTurns.get(JSON.stringify([sessionId, data.turn]))
+                : undefined;
+            const root = data.invocationId && !inTurn && !completed
                 ? // A late hook must not consume context queued for the next turn.
                     reporter.openTrace({
                         name: config.traceName,
@@ -403,8 +419,16 @@ export function apply(ctx, config) {
                         metadata: { turn: data.turn, detachedHook: true },
                     })
                 : undefined;
-            const span = reporter.startSpan(root === undefined ? parentFor(sessionId) : root, {
-                name: `hook:${data.point}`,
+            const tool = data.callId ? toolCalls.get(sessionId)?.get(data.callId) : undefined;
+            const hookName = config.captureContent ? reporter.mask({ name: data.name }).name : undefined;
+            const span = reporter.startSpan(completed?.parent ??
+                (root === undefined
+                    ? tool?.turn === data.turn
+                        ? tool.span
+                        : parentFor(sessionId)
+                    : root), {
+                name: `hook:${data.point}${hookName ? `:${hookName}` : ''}`,
+                input: config.captureContent ? data.input : undefined,
                 metadata: {
                     turn: data.turn,
                     point: data.point,
@@ -412,6 +436,7 @@ export function apply(ctx, config) {
                     matcher: data.matcher,
                     handlerId: data.handlerId,
                     ...(data.invocationId ? { invocationId: data.invocationId } : {}),
+                    ...(data.callId ? { callId: data.callId } : {}),
                     ...(config.captureContent && data.name ? { name: data.name } : {}),
                 },
             });
@@ -422,7 +447,12 @@ export function apply(ctx, config) {
                 point: data.point,
                 handlerId: data.handlerId,
                 persistent: !!data.invocationId,
-                turnTrace: root === undefined ? (child?.turnTrace ?? sessions.get(sessionId)?.trace) : undefined,
+                callId: data.callId,
+                turnTrace: completed
+                    ? completed.turnTrace
+                    : root === undefined
+                        ? (child?.turnTrace ?? sessions.get(sessionId)?.trace)
+                        : undefined,
             });
             openHooks.set(sessionId, pending);
             return;
@@ -437,7 +467,8 @@ export function apply(ctx, config) {
             if (!hook ||
                 hook.handlerId !== data.handlerId ||
                 hook.turn !== data.turn ||
-                hook.point !== data.point)
+                hook.point !== data.point ||
+                (data.callId !== undefined && hook.callId !== data.callId))
                 return;
             const span = hook.span;
             const failed = data.outcome
@@ -449,6 +480,21 @@ export function apply(ctx, config) {
                     ? 'WARNING'
                     : 'DEFAULT';
             if (span) {
+                const processOutput = {};
+                const truncation = {};
+                if (config.captureContent) {
+                    const masked = reporter.mask({
+                        stdout: data.stdout,
+                        stderr: data.stderr,
+                        stderrSummary: data.stderrSummary,
+                    });
+                    for (const [field, text] of Object.entries(masked)) {
+                        if (text === undefined)
+                            continue;
+                        processOutput[field] = text.slice(0, 4096);
+                        truncation[`${field}Truncated`] = text.length > 4096;
+                    }
+                }
                 if (failed && sessions.get(rootSessionId(sessionId) ?? '')?.trace === hook.turnTrace)
                     noteError(sessionId);
                 reporter.endSpan(span, {
@@ -457,10 +503,10 @@ export function apply(ctx, config) {
                         ...(data.outcome ? { outcome: data.outcome } : {}),
                         ...(data.signal ? { signal: data.signal } : {}),
                         exitCode: data.exitCode,
-                        stderrSummary: config.captureContent ? data.stderrSummary : undefined,
+                        ...(config.captureContent ? { result: data.result, ...processOutput } : {}),
                     },
                     level,
-                    metadata: { durationMs: data.durationMs },
+                    metadata: { durationMs: data.durationMs, ...truncation },
                 });
             }
             reporter.endSpan(hook.root ?? null, { level });
@@ -469,14 +515,18 @@ export function apply(ctx, config) {
                 openHooks.delete(sessionId);
             return;
         }
+        if (event.type === 'turn/end' || event.type === 'turn/start')
+            toolCalls.delete(sessionId);
         if (event.type === 'turn/end')
             closeHooks(sessionId, true);
         const child = children.get(sessionId);
         if (child) {
             if (event.type === 'turn/start')
                 child.turn = event.data.turn;
-            if (event.type === 'turn/end')
+            if (event.type === 'turn/end') {
+                rememberTurn(sessionId, event.data.turn, child.span, child.turnTrace ?? null);
                 child.turn = undefined;
+            }
             // The delegated prompt (the child's first user/message) is the
             // subagent span's input; the descriptor enriches the span name.
             if (event.type === 'user/message' && !child.hasInput && config.captureContent) {
@@ -528,6 +578,7 @@ export function apply(ctx, config) {
                 const state = sessions.get(sessionId);
                 if (state) {
                     sessions.delete(sessionId);
+                    rememberTurn(sessionId, event.data.turn, state.trace, state.trace);
                     // The turn's final answer becomes the trace's output; ending the
                     // root span is what exports the trace (v5 exports spans on end).
                     afterMedia(state.trace, (endedAt) => reporter.endSpan(state.trace, {
@@ -560,7 +611,12 @@ export function apply(ctx, config) {
         const state = sessions.get(sessionId);
         sessions.delete(sessionId);
         traceContexts.delete(sessionId);
+        for (const [key, completed] of completedTurns) {
+            if (completed.sessionId === sessionId)
+                completedTurns.delete(key);
+        }
         openToolSpans.delete(sessionId);
+        toolCalls.delete(sessionId);
         // End an abandoned root (crash-orphaned turn or between-turns one-off
         // trace) — v5 spans only export on end, so dropping it open would lose
         // everything it recorded.
@@ -756,12 +812,12 @@ export function apply(ctx, config) {
         })();
     });
     // ------------------------------------------------------------------
-    // Span per tool dispatch. This waterfall IS async: `next()` returns the
-    // normalized dispatch result promise. Open spans are tracked per session
+    // Span per tool pipeline, including pre/post hooks and denied calls.
+    // Open spans are tracked per session
     // so a subagent span opened mid-dispatch (foreground delegation) parents
     // under the enclosing tool span.
     // ------------------------------------------------------------------
-    ctx.on('tools/execute', async (exec, next) => {
+    ctx.on('tools/pre-execute', (exec, next) => {
         const sessionId = exec.agent?.id;
         const parent = parentFor(sessionId);
         // An agent-less dispatch's trace root has no session lifecycle to close
@@ -778,6 +834,12 @@ export function apply(ctx, config) {
         });
         if (sessionId && span) {
             openToolSpans.set(sessionId, [...(openToolSpans.get(sessionId) ?? []), span]);
+            const calls = toolCalls.get(sessionId) ?? new Map();
+            calls.set(exec.callId, {
+                span,
+                turn: children.get(sessionId)?.turn ?? sessions.get(sessionId)?.turn,
+            });
+            toolCalls.set(sessionId, calls);
         }
         let closed = false;
         const closeTool = () => finishTool({ level: 'WARNING', statusMessage: 'plugin unloaded during tool call' });
@@ -787,6 +849,13 @@ export function apply(ctx, config) {
             closed = true;
             const endedAt = new Date();
             openOperations.delete(closeTool);
+            toolResults.delete(exec);
+            if (sessionId && toolCalls.get(sessionId)?.get(exec.callId)?.span === span) {
+                const calls = toolCalls.get(sessionId);
+                calls.delete(exec.callId);
+                if (!calls.size)
+                    toolCalls.delete(sessionId);
+            }
             if (sessionId && span) {
                 const remaining = (openToolSpans.get(sessionId) ?? []).filter((s) => s !== span);
                 if (remaining.length)
@@ -807,10 +876,9 @@ export function apply(ctx, config) {
                 end();
         }
         openOperations.add(closeTool);
-        try {
-            const result = await next();
+        toolResults.set(exec, (result) => {
             if (closed)
-                return result;
+                return;
             if (result.isError)
                 noteError(sessionId);
             const update = {
@@ -819,21 +887,18 @@ export function apply(ctx, config) {
                 statusMessage: result.isError ? contentMessage(config, result.error.message) : undefined,
                 metadata: { errorCode: result.isError ? result.error.info?.code : undefined },
             };
-            if (span && config.captureContent && result.content.some((block) => block.type === 'image')) {
+            if (span &&
+                config.captureContent &&
+                result.content.some((block) => block.type === 'image')) {
                 finishTool(update, displayBlocks(result.content, ctx.get('attachments'), mediaOptions, exec.signal));
             }
             else
                 finishTool(update);
-            return result;
-        }
-        catch (error) {
-            noteError(sessionId);
-            finishTool({
-                level: 'ERROR',
-                statusMessage: contentMessage(config, messageOf(error)),
-            });
-            throw error;
-        }
+        });
+        return next();
+    }, { prepend: true });
+    ctx.on('tools/result', (exec, result) => {
+        toolResults.get(exec)?.(result);
     });
     // Durability checkpoint: the store awaits every session/flush listener.
     ctx.on('session/flush', async () => {
@@ -859,9 +924,11 @@ export function apply(ctx, config) {
             for (const child of children.values())
                 reporter.endSpan(child.span, {}, endedAt);
             sessions.clear();
+            completedTurns.clear();
             traceContexts.clear();
             children.clear();
             openToolSpans.clear();
+            toolCalls.clear();
             await reporter.shutdown();
         };
     });

package/lib/types/hooks.d.ts

@@ -9,12 +9,20 @@ export interface HookInvoked {
     /** Unique per execution, shared with its result; required for concurrent runs. */
     invocationId?: string;
     name?: string;
+    /** Tool execution identity; omit for agent hooks. */
+    callId?: string;
+    input?: unknown;
 }
 export interface HookResult {
     turn: number;
     point: string;
     handlerId: string;
     invocationId?: string;
+    callId?: string;
+    /** Parsed hook output; process status remains in the fields below. */
+    result?: Record<string, unknown>;
+    stdout?: string;
+    stderr?: string;
     decision: string;
     outcome?: 'success' | 'failed' | 'timed-out' | 'cancelled';
     exitCode?: number;

package/lib/types/index.d.ts

@@ -8,7 +8,7 @@
  *   from the chunk stream), plus a nested `llm-request` **span** holding the
  *   same options and collected chunk stream (the rawest request/response a
  *   plugin can observe — the provider HTTP body is assembled inside the adapter).
- * - `tools/execute` waterfall → one Langfuse **span** per tool dispatch
+ * - `tools/pre-execute` → `tools/result` → one Langfuse **span** per tool pipeline
  *   (routed to the session's trace via `exec.agent.id` — the agent/session
  *   shared identity).
  * - `hook/invoked` / `hook/result` session records → one **span** per hook run.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@amaster.ai/dsh-langfuse",
-  "version": "0.1.23",
+  "version": "0.1.25",
   "description": "Langfuse observability plugin for DeepSeek Harness: one generation per LLM call with a nested llm-request span, one span per tool call, one trace per session turn, subagent child sessions nested under the parent's tree.",
   "type": "module",
   "main": "lib/index.js",