@sebastianandreasson/pi-autonomous-agents 0.15.1 → 0.15.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +8 -0
- package/docs/PI_REQUEST_TELEMETRY_EXTENSION.md +19 -2
- package/package.json +1 -1
- package/pi-extensions/request-telemetry/README.md +16 -2
- package/pi-extensions/request-telemetry/index.mjs +77 -1
- package/src/index.mjs +2 -0
- package/src/pi-client.mjs +2 -0
- package/src/pi-config.mjs +2 -0
- package/src/pi-history.mjs +1 -0
- package/src/pi-request-telemetry.mjs +141 -9
- package/src/pi-sdk-turn.mjs +16 -0
- package/templates/pi.config.example.json +2 -0
package/README.md
CHANGED
|
@@ -82,6 +82,14 @@ Start from [templates/pi.config.example.json](./templates/pi.config.example.json
|
|
|
82
82
|
|
|
83
83
|
Request telemetry is enabled by default for SDK runs. `pi-harness` writes a managed Pi extension package under `.pi/extensions/pi-harness-request-telemetry/` in the consuming repo, with a `package.json` manifest and `index.mjs` shim that Pi auto-discovers on the next resource reload. Disable that with `PI_REQUEST_TELEMETRY_ENABLED=0` or `"piRequestTelemetryEnabled": false`.
|
|
84
84
|
|
|
85
|
+
By default the extension now stores compact request telemetry only:
|
|
86
|
+
- `requests.jsonl` with exact request totals and summarized tool/file attribution
|
|
87
|
+
- `spans.jsonl` with byte counts and attribution metadata, but not full prompt text
|
|
88
|
+
|
|
89
|
+
Verbose hook traces and raw span text are opt-in for debugging:
|
|
90
|
+
- `PI_REQUEST_TELEMETRY_STORE_HOOKS=1` or `"piRequestTelemetryStoreHooks": true`
|
|
91
|
+
- `PI_REQUEST_TELEMETRY_STORE_SPAN_TEXT=1` or `"piRequestTelemetryStoreSpanText": true`
|
|
92
|
+
|
|
85
93
|
## CLI
|
|
86
94
|
|
|
87
95
|
```bash
|
|
@@ -4,14 +4,28 @@ This document describes the repo-local Pi extension prototype under [pi-extensio
|
|
|
4
4
|
|
|
5
5
|
In normal `pi-harness` SDK runs, this extension is auto-enabled by installing a managed extension package under `.pi/extensions/pi-harness-request-telemetry/` in the consuming repo before Pi reloads resources. That package contains a `package.json` Pi manifest plus the generated `index.mjs` shim. Opt out with `PI_REQUEST_TELEMETRY_ENABLED=0` or `"piRequestTelemetryEnabled": false`.
|
|
6
6
|
|
|
7
|
+
The default storage mode is compact:
|
|
8
|
+
|
|
9
|
+
- `requests.jsonl` is always kept
|
|
10
|
+
- `spans.jsonl` keeps attribution metadata and byte counts, but not full prompt text
|
|
11
|
+
- `hooks.jsonl` is disabled by default
|
|
12
|
+
|
|
13
|
+
Enable deeper debug capture only when needed:
|
|
14
|
+
|
|
15
|
+
- `PI_REQUEST_TELEMETRY_STORE_HOOKS=1` or `"piRequestTelemetryStoreHooks": true`
|
|
16
|
+
- `PI_REQUEST_TELEMETRY_STORE_SPAN_TEXT=1` or `"piRequestTelemetryStoreSpanText": true`
|
|
17
|
+
|
|
7
18
|
The purpose of this extension is to gather request-level data directly from Pi extension hooks before we decide whether to patch `@mariozechner/pi-coding-agent` or `pi-ai`.
|
|
8
19
|
|
|
9
20
|
## Produced Artifacts
|
|
10
21
|
|
|
11
|
-
- `pi-output/request-telemetry/hooks.jsonl`
|
|
12
22
|
- `pi-output/request-telemetry/requests.jsonl`
|
|
13
23
|
- `pi-output/request-telemetry/spans.jsonl`
|
|
14
24
|
|
|
25
|
+
Optional when hook tracing is enabled:
|
|
26
|
+
|
|
27
|
+
- `pi-output/request-telemetry/hooks.jsonl`
|
|
28
|
+
|
|
15
29
|
These artifacts are intentionally separate from the existing `pi-output/token-usage/*` files.
|
|
16
30
|
|
|
17
31
|
`token-usage` remains the harness-level normalized output.
|
|
@@ -109,7 +123,7 @@ Interpretation:
|
|
|
109
123
|
- `message_end`
|
|
110
124
|
- `turn_end`
|
|
111
125
|
|
|
112
|
-
Use it to debug request association problems or confirm whether Pi emitted provider-boundary hooks for a specific run.
|
|
126
|
+
Use it to debug request association problems or confirm whether Pi emitted provider-boundary hooks for a specific run. This file is off by default because it grows quickly and is only useful for telemetry debugging.
|
|
113
127
|
|
|
114
128
|
## Span Artifact Schema
|
|
115
129
|
|
|
@@ -131,6 +145,9 @@ Each row in `spans.jsonl` contains one extracted prompt span:
|
|
|
131
145
|
- `primaryPath`
|
|
132
146
|
- `charCount`
|
|
133
147
|
- `byteCount`
|
|
148
|
+
|
|
149
|
+
Optional only when raw span text capture is enabled:
|
|
150
|
+
|
|
134
151
|
- `text`
|
|
135
152
|
- `preview`
|
|
136
153
|
|
package/package.json
CHANGED
|
@@ -4,10 +4,13 @@ This is a repo-local Pi extension prototype for capturing lower-level request te
|
|
|
4
4
|
|
|
5
5
|
It writes:
|
|
6
6
|
|
|
7
|
-
- `pi-output/request-telemetry/hooks.jsonl`
|
|
8
7
|
- `pi-output/request-telemetry/requests.jsonl`
|
|
9
8
|
- `pi-output/request-telemetry/spans.jsonl`
|
|
10
9
|
|
|
10
|
+
Optional when debug tracing is enabled:
|
|
11
|
+
|
|
12
|
+
- `pi-output/request-telemetry/hooks.jsonl`
|
|
13
|
+
|
|
11
14
|
The goal is to capture exact request boundaries and exact prompt composition before deciding whether we need to patch `pi-mono` itself.
|
|
12
15
|
|
|
13
16
|
## What It Captures
|
|
@@ -20,12 +23,16 @@ The goal is to capture exact request boundaries and exact prompt composition bef
|
|
|
20
23
|
- provider payload summary from `before_provider_request`
|
|
21
24
|
- response status and headers from `after_provider_response`
|
|
22
25
|
- final assistant-message usage if Pi exposes it on `message.usage`
|
|
23
|
-
- lifecycle hook traces in `hooks.jsonl` so you can debug request association
|
|
24
26
|
- `spanSource` on each request so consumers can distinguish:
|
|
25
27
|
- `provider_payload`
|
|
26
28
|
- `context`
|
|
27
29
|
- `session_history`
|
|
28
30
|
|
|
31
|
+
Default storage is compact:
|
|
32
|
+
|
|
33
|
+
- `spans.jsonl` keeps attribution metadata and byte counts, not full prompt text
|
|
34
|
+
- `hooks.jsonl` is off by default because it is mainly for telemetry debugging
|
|
35
|
+
|
|
29
36
|
## What It Does Not Claim Yet
|
|
30
37
|
|
|
31
38
|
- exact per-file token spend
|
|
@@ -50,6 +57,13 @@ Disable that path with:
|
|
|
50
57
|
- `PI_REQUEST_TELEMETRY_ENABLED=0`
|
|
51
58
|
- `"piRequestTelemetryEnabled": false` in `pi.config.json`
|
|
52
59
|
|
|
60
|
+
Enable deeper telemetry capture only when needed:
|
|
61
|
+
|
|
62
|
+
- `PI_REQUEST_TELEMETRY_STORE_HOOKS=1`
|
|
63
|
+
- `"piRequestTelemetryStoreHooks": true`
|
|
64
|
+
- `PI_REQUEST_TELEMETRY_STORE_SPAN_TEXT=1`
|
|
65
|
+
- `"piRequestTelemetryStoreSpanText": true`
|
|
66
|
+
|
|
53
67
|
## Running It From This Repo
|
|
54
68
|
|
|
55
69
|
Use the extension file directly:
|
|
@@ -5,11 +5,13 @@ import {
|
|
|
5
5
|
appendRequestTelemetryArtifacts,
|
|
6
6
|
collectMessageSpans,
|
|
7
7
|
collectProviderPayloadSpans,
|
|
8
|
+
collectToolHookSpans,
|
|
8
9
|
deriveToolPaths,
|
|
9
10
|
extractMessagesFromProviderPayload,
|
|
10
11
|
extractUsageFromMessage,
|
|
11
12
|
getRequestTelemetryPaths,
|
|
12
13
|
readRequestTelemetryContextFromEnv,
|
|
14
|
+
readRequestTelemetryStorageOptionsFromEnv,
|
|
13
15
|
summarizeProviderPayload,
|
|
14
16
|
summarizeRequestSpans,
|
|
15
17
|
} from '../../src/pi-request-telemetry.mjs'
|
|
@@ -49,6 +51,7 @@ function createTurnState({ turnIndex = 0, startedAt = '', model = '' } = {}) {
|
|
|
49
51
|
contextMessageCount: 0,
|
|
50
52
|
contextSpanCount: 0,
|
|
51
53
|
lastAssistantMessage: null,
|
|
54
|
+
recentToolEvents: [],
|
|
52
55
|
}
|
|
53
56
|
}
|
|
54
57
|
|
|
@@ -182,6 +185,7 @@ function applyAssistantMessage(requestState, message) {
|
|
|
182
185
|
|
|
183
186
|
export function createRequestTelemetryExtension({ cwd = process.cwd() } = {}) {
|
|
184
187
|
const artifacts = getRequestTelemetryPaths({ cwd })
|
|
188
|
+
const storage = readRequestTelemetryStorageOptionsFromEnv()
|
|
185
189
|
const state = {
|
|
186
190
|
sessionId: randomUUID(),
|
|
187
191
|
currentModel: '',
|
|
@@ -213,6 +217,10 @@ export function createRequestTelemetryExtension({ cwd = process.cwd() } = {}) {
|
|
|
213
217
|
}
|
|
214
218
|
|
|
215
219
|
function trace(type, detail = {}) {
|
|
220
|
+
if (!storage.storeHooks) {
|
|
221
|
+
return Promise.resolve()
|
|
222
|
+
}
|
|
223
|
+
|
|
216
224
|
state.hookSequence += 1
|
|
217
225
|
const activeRequest = getLatestPendingRequest()
|
|
218
226
|
const currentTurn = state.currentTurn
|
|
@@ -326,6 +334,32 @@ export function createRequestTelemetryExtension({ cwd = process.cwd() } = {}) {
|
|
|
326
334
|
return false
|
|
327
335
|
}
|
|
328
336
|
|
|
337
|
+
function applyToolHookSnapshot(requestState) {
|
|
338
|
+
const currentTurn = getCurrentTurn()
|
|
339
|
+
const toolEvents = Array.isArray(currentTurn.recentToolEvents) ? currentTurn.recentToolEvents : []
|
|
340
|
+
if (toolEvents.length === 0) {
|
|
341
|
+
return false
|
|
342
|
+
}
|
|
343
|
+
|
|
344
|
+
const spans = collectToolHookSpans({
|
|
345
|
+
requestId: requestState.requestId,
|
|
346
|
+
sessionId: requestState.sessionId,
|
|
347
|
+
turnIndex: requestState.turnIndex,
|
|
348
|
+
toolEvents,
|
|
349
|
+
})
|
|
350
|
+
if (spans.length === 0) {
|
|
351
|
+
return false
|
|
352
|
+
}
|
|
353
|
+
|
|
354
|
+
applySpanSnapshot(requestState, {
|
|
355
|
+
messages: [],
|
|
356
|
+
spans,
|
|
357
|
+
source: 'tool_hooks',
|
|
358
|
+
})
|
|
359
|
+
mergeSpanSummary(requestState)
|
|
360
|
+
return true
|
|
361
|
+
}
|
|
362
|
+
|
|
329
363
|
function createProviderRequest(overrides = {}) {
|
|
330
364
|
const currentTurn = getCurrentTurn()
|
|
331
365
|
return createRequestState({
|
|
@@ -411,6 +445,8 @@ export function createRequestTelemetryExtension({ cwd = process.cwd() } = {}) {
|
|
|
411
445
|
...requestState.usage,
|
|
412
446
|
},
|
|
413
447
|
spans: requestState.spans,
|
|
448
|
+
}, {
|
|
449
|
+
includeSpanText: storage.storeSpanText,
|
|
414
450
|
})
|
|
415
451
|
}
|
|
416
452
|
|
|
@@ -482,6 +518,7 @@ export function createRequestTelemetryExtension({ cwd = process.cwd() } = {}) {
|
|
|
482
518
|
})
|
|
483
519
|
|
|
484
520
|
pi.on('before_provider_request', (event) => {
|
|
521
|
+
const currentTurn = getCurrentTurn()
|
|
485
522
|
const requestState = createProviderRequest()
|
|
486
523
|
const providerApplied = applyProviderPayloadSnapshot(requestState, event?.payload)
|
|
487
524
|
|
|
@@ -513,10 +550,25 @@ export function createRequestTelemetryExtension({ cwd = process.cwd() } = {}) {
|
|
|
513
550
|
requestState.files = sessionHistoryCandidate.files
|
|
514
551
|
}
|
|
515
552
|
|
|
516
|
-
|
|
553
|
+
const toolHookCandidate = createComparableRequestState(requestState)
|
|
554
|
+
const toolHookApplied = applyToolHookSnapshot(toolHookCandidate)
|
|
555
|
+
if (toolHookApplied && shouldPreferSnapshot(requestState, toolHookCandidate)) {
|
|
556
|
+
requestState.contextMessages = toolHookCandidate.contextMessages
|
|
557
|
+
requestState.spans = toolHookCandidate.spans
|
|
558
|
+
requestState.spanSource = toolHookCandidate.spanSource
|
|
559
|
+
requestState.contextMessageCount = toolHookCandidate.contextMessageCount
|
|
560
|
+
requestState.spanCount = toolHookCandidate.spanCount
|
|
561
|
+
requestState.textChars = toolHookCandidate.textChars
|
|
562
|
+
requestState.textBytes = toolHookCandidate.textBytes
|
|
563
|
+
requestState.toolNames = toolHookCandidate.toolNames
|
|
564
|
+
requestState.files = toolHookCandidate.files
|
|
565
|
+
}
|
|
566
|
+
|
|
567
|
+
if (!providerApplied && requestState.spans.length === 0 && !contextApplied && !sessionHistoryApplied && !toolHookApplied) {
|
|
517
568
|
applySessionHistorySnapshot(requestState)
|
|
518
569
|
}
|
|
519
570
|
state.pendingRequests.push(requestState)
|
|
571
|
+
currentTurn.recentToolEvents = []
|
|
520
572
|
|
|
521
573
|
void trace('before_provider_request', {
|
|
522
574
|
requestId: requestState.requestId,
|
|
@@ -544,15 +596,28 @@ export function createRequestTelemetryExtension({ cwd = process.cwd() } = {}) {
|
|
|
544
596
|
})
|
|
545
597
|
|
|
546
598
|
pi.on('tool_execution_start', (event) => {
|
|
599
|
+
const currentTurn = getCurrentTurn()
|
|
547
600
|
const toolCallId = String(event?.toolCallId ?? '').trim()
|
|
548
601
|
const toolName = String(event?.toolName ?? '').trim()
|
|
549
602
|
const paths = deriveToolPaths(toolName, event?.args)
|
|
550
603
|
if (toolCallId !== '') {
|
|
551
604
|
state.toolCallIndex.set(toolCallId, { toolName, paths })
|
|
552
605
|
}
|
|
606
|
+
currentTurn.recentToolEvents = [
|
|
607
|
+
...(currentTurn.recentToolEvents ?? []).filter((item) => String(item?.toolCallId ?? '') !== toolCallId),
|
|
608
|
+
{
|
|
609
|
+
toolCallId,
|
|
610
|
+
toolName,
|
|
611
|
+
args: event?.args,
|
|
612
|
+
details: undefined,
|
|
613
|
+
content: [],
|
|
614
|
+
timestamp: new Date().toISOString(),
|
|
615
|
+
},
|
|
616
|
+
]
|
|
553
617
|
})
|
|
554
618
|
|
|
555
619
|
pi.on('tool_result', (event) => {
|
|
620
|
+
const currentTurn = getCurrentTurn()
|
|
556
621
|
const toolCallId = String(event?.toolCallId ?? '').trim()
|
|
557
622
|
if (toolCallId === '') {
|
|
558
623
|
return
|
|
@@ -571,6 +636,17 @@ export function createRequestTelemetryExtension({ cwd = process.cwd() } = {}) {
|
|
|
571
636
|
toolName: current.toolName,
|
|
572
637
|
paths: [...new Set([...current.paths, ...resultPaths])],
|
|
573
638
|
})
|
|
639
|
+
currentTurn.recentToolEvents = [
|
|
640
|
+
...(currentTurn.recentToolEvents ?? []).filter((item) => String(item?.toolCallId ?? '') !== toolCallId),
|
|
641
|
+
{
|
|
642
|
+
toolCallId,
|
|
643
|
+
toolName: current.toolName,
|
|
644
|
+
args: event?.input,
|
|
645
|
+
details: event?.details,
|
|
646
|
+
content: Array.isArray(event?.content) ? event.content : [],
|
|
647
|
+
timestamp: new Date().toISOString(),
|
|
648
|
+
},
|
|
649
|
+
]
|
|
574
650
|
})
|
|
575
651
|
|
|
576
652
|
pi.on('message_end', async (event) => {
|
package/src/index.mjs
CHANGED
|
@@ -31,12 +31,14 @@ export {
|
|
|
31
31
|
appendRequestTelemetryArtifacts,
|
|
32
32
|
collectMessageSpans,
|
|
33
33
|
collectProviderPayloadSpans,
|
|
34
|
+
compactRequestSpanRecord,
|
|
34
35
|
createEmptyRequestTelemetryBreakdown,
|
|
35
36
|
createEmptyRequestUsage,
|
|
36
37
|
deriveRequestTelemetryAnalytics,
|
|
37
38
|
deriveRequestTelemetryBreakdown,
|
|
38
39
|
deriveToolPaths,
|
|
39
40
|
readRequestTelemetryContextFromEnv,
|
|
41
|
+
readRequestTelemetryStorageOptionsFromEnv,
|
|
40
42
|
readRequestTelemetryRecords,
|
|
41
43
|
extractMessagesFromProviderPayload,
|
|
42
44
|
extractUsageFromMessage,
|
package/src/pi-client.mjs
CHANGED
|
@@ -234,6 +234,8 @@ async function runSdkTransportTurn({ config, model, sessionId, sessionFile, prom
|
|
|
234
234
|
thinking: config.piThinking,
|
|
235
235
|
noExtensions: config.piNoExtensions,
|
|
236
236
|
requestTelemetryEnabled: config.piRequestTelemetryEnabled,
|
|
237
|
+
requestTelemetryStoreHooks: config.piRequestTelemetryStoreHooks,
|
|
238
|
+
requestTelemetryStoreSpanText: config.piRequestTelemetryStoreSpanText,
|
|
237
239
|
noSkills: config.piNoSkills,
|
|
238
240
|
noPromptTemplates: config.piNoPromptTemplates,
|
|
239
241
|
noThemes: config.piNoThemes,
|
package/src/pi-config.mjs
CHANGED
|
@@ -266,6 +266,8 @@ export function loadConfig(mode = 'once') {
|
|
|
266
266
|
piThinking: readString('PI_THINKING', file.piThinking, ''),
|
|
267
267
|
piNoExtensions: readBool('PI_NO_EXTENSIONS', file.piNoExtensions, false),
|
|
268
268
|
piRequestTelemetryEnabled: readBool('PI_REQUEST_TELEMETRY_ENABLED', file.piRequestTelemetryEnabled, true),
|
|
269
|
+
piRequestTelemetryStoreHooks: readBool('PI_REQUEST_TELEMETRY_STORE_HOOKS', file.piRequestTelemetryStoreHooks, false),
|
|
270
|
+
piRequestTelemetryStoreSpanText: readBool('PI_REQUEST_TELEMETRY_STORE_SPAN_TEXT', file.piRequestTelemetryStoreSpanText, false),
|
|
269
271
|
piNoSkills: readBool('PI_NO_SKILLS', file.piNoSkills, false),
|
|
270
272
|
piNoPromptTemplates: readBool('PI_NO_PROMPT_TEMPLATES', file.piNoPromptTemplates, false),
|
|
271
273
|
piNoThemes: readBool('PI_NO_THEMES', file.piNoThemes, true),
|
package/src/pi-history.mjs
CHANGED
|
@@ -24,6 +24,7 @@ export function collectHistoryTargets(config) {
|
|
|
24
24
|
config.lastIterationSummaryFile,
|
|
25
25
|
config.tokenUsageEventsFile,
|
|
26
26
|
config.tokenUsageSummaryFile,
|
|
27
|
+
path.join(config.cwd, 'pi-output/request-telemetry'),
|
|
27
28
|
config.piRuntimeDir,
|
|
28
29
|
config.visualFeedbackFile,
|
|
29
30
|
config.testerFeedbackFile,
|
|
@@ -12,6 +12,10 @@ export const REQUEST_TELEMETRY_ENV_KEYS = Object.freeze({
|
|
|
12
12
|
kind: 'PI_REQUEST_KIND',
|
|
13
13
|
task: 'PI_REQUEST_TASK',
|
|
14
14
|
})
|
|
15
|
+
export const REQUEST_TELEMETRY_STORAGE_ENV_KEYS = Object.freeze({
|
|
16
|
+
storeHooks: 'PI_REQUEST_TELEMETRY_STORE_HOOKS',
|
|
17
|
+
storeSpanText: 'PI_REQUEST_TELEMETRY_STORE_SPAN_TEXT',
|
|
18
|
+
})
|
|
15
19
|
|
|
16
20
|
const scriptDir = path.dirname(fileURLToPath(import.meta.url))
|
|
17
21
|
const packageRoot = path.resolve(scriptDir, '..')
|
|
@@ -94,6 +98,24 @@ function asObject(value) {
|
|
|
94
98
|
return value && typeof value === 'object' && !Array.isArray(value) ? value : {}
|
|
95
99
|
}
|
|
96
100
|
|
|
101
|
+
function parseBooleanFlag(value, fallback = false) {
|
|
102
|
+
if (value === undefined || value === null || value === '') {
|
|
103
|
+
return fallback
|
|
104
|
+
}
|
|
105
|
+
if (typeof value === 'boolean') {
|
|
106
|
+
return value
|
|
107
|
+
}
|
|
108
|
+
|
|
109
|
+
const normalized = String(value).trim().toLowerCase()
|
|
110
|
+
if (normalized === '1' || normalized === 'true' || normalized === 'yes') {
|
|
111
|
+
return true
|
|
112
|
+
}
|
|
113
|
+
if (normalized === '0' || normalized === 'false' || normalized === 'no') {
|
|
114
|
+
return false
|
|
115
|
+
}
|
|
116
|
+
return fallback
|
|
117
|
+
}
|
|
118
|
+
|
|
97
119
|
function createBucketMap() {
|
|
98
120
|
return new Map()
|
|
99
121
|
}
|
|
@@ -279,6 +301,13 @@ export function readRequestTelemetryContextFromEnv(env = process.env) {
|
|
|
279
301
|
}
|
|
280
302
|
}
|
|
281
303
|
|
|
304
|
+
export function readRequestTelemetryStorageOptionsFromEnv(env = process.env) {
|
|
305
|
+
return {
|
|
306
|
+
storeHooks: parseBooleanFlag(env?.[REQUEST_TELEMETRY_STORAGE_ENV_KEYS.storeHooks], false),
|
|
307
|
+
storeSpanText: parseBooleanFlag(env?.[REQUEST_TELEMETRY_STORAGE_ENV_KEYS.storeSpanText], false),
|
|
308
|
+
}
|
|
309
|
+
}
|
|
310
|
+
|
|
282
311
|
export function normalizeRequestUsage(value) {
|
|
283
312
|
if (!value || typeof value !== 'object') {
|
|
284
313
|
return createEmptyRequestUsage()
|
|
@@ -643,6 +672,17 @@ export function normalizeRequestSpanRecord(record) {
|
|
|
643
672
|
}
|
|
644
673
|
}
|
|
645
674
|
|
|
675
|
+
export function compactRequestSpanRecord(record, { includeText = false, includePreview = false } = {}) {
|
|
676
|
+
const normalized = normalizeRequestSpanRecord(record)
|
|
677
|
+
if (!includeText) {
|
|
678
|
+
delete normalized.text
|
|
679
|
+
}
|
|
680
|
+
if (!includeText || !includePreview) {
|
|
681
|
+
delete normalized.preview
|
|
682
|
+
}
|
|
683
|
+
return normalized
|
|
684
|
+
}
|
|
685
|
+
|
|
646
686
|
export function createEmptyRequestTelemetryBreakdown() {
|
|
647
687
|
return createEmptyBreakdownShape('request_telemetry')
|
|
648
688
|
}
|
|
@@ -805,6 +845,84 @@ export function collectProviderPayloadSpans({
|
|
|
805
845
|
}
|
|
806
846
|
}
|
|
807
847
|
|
|
848
|
+
function joinToolContentText(content = []) {
|
|
849
|
+
if (!Array.isArray(content)) {
|
|
850
|
+
return ''
|
|
851
|
+
}
|
|
852
|
+
|
|
853
|
+
return content
|
|
854
|
+
.map((item) => {
|
|
855
|
+
const object = asObject(item)
|
|
856
|
+
if (normalizeString(object.type, '') === 'text') {
|
|
857
|
+
return String(object.text ?? '')
|
|
858
|
+
}
|
|
859
|
+
return ''
|
|
860
|
+
})
|
|
861
|
+
.filter(Boolean)
|
|
862
|
+
.join('\n')
|
|
863
|
+
}
|
|
864
|
+
|
|
865
|
+
export function collectToolHookSpans({
|
|
866
|
+
requestId = '',
|
|
867
|
+
sessionId = '',
|
|
868
|
+
turnIndex = 0,
|
|
869
|
+
toolEvents = [],
|
|
870
|
+
timestamp = now(),
|
|
871
|
+
} = {}) {
|
|
872
|
+
const spans = []
|
|
873
|
+
|
|
874
|
+
for (let index = 0; index < (Array.isArray(toolEvents) ? toolEvents.length : 0); index += 1) {
|
|
875
|
+
const event = asObject(toolEvents[index])
|
|
876
|
+
const toolCallId = normalizeString(event.toolCallId, '')
|
|
877
|
+
const toolName = normalizeString(event.toolName, '')
|
|
878
|
+
const args = parseJsonLikeString(event.args ?? event.input)
|
|
879
|
+
const details = parseJsonLikeString(event.details)
|
|
880
|
+
const contentText = joinToolContentText(event.content)
|
|
881
|
+
const detailText = safeJson(details)
|
|
882
|
+
const resultText = [contentText, detailText].filter(Boolean).join('\n')
|
|
883
|
+
const paths = normalizeStringList([
|
|
884
|
+
...deriveToolPaths(toolName, args),
|
|
885
|
+
...deriveToolPaths(toolName, details),
|
|
886
|
+
])
|
|
887
|
+
const base = createSpanBase({
|
|
888
|
+
requestId,
|
|
889
|
+
sessionId,
|
|
890
|
+
turnIndex,
|
|
891
|
+
timestamp: event.timestamp ?? timestamp,
|
|
892
|
+
role: 'toolResult',
|
|
893
|
+
messageIndex: index,
|
|
894
|
+
spanIndex: 0,
|
|
895
|
+
source: 'tool_hooks',
|
|
896
|
+
})
|
|
897
|
+
|
|
898
|
+
if (args !== undefined) {
|
|
899
|
+
spans.push(createTextSpan(base, {
|
|
900
|
+
spanIndex: 0,
|
|
901
|
+
spanKind: 'tool_call',
|
|
902
|
+
toolCallId,
|
|
903
|
+
toolName,
|
|
904
|
+
paths,
|
|
905
|
+
primaryPath: paths[0] ?? '',
|
|
906
|
+
text: safeJson(args),
|
|
907
|
+
}))
|
|
908
|
+
}
|
|
909
|
+
|
|
910
|
+
if (resultText !== '' || details !== undefined || Array.isArray(event.content)) {
|
|
911
|
+
spans.push(createTextSpan(base, {
|
|
912
|
+
spanIndex: args !== undefined ? 1 : 0,
|
|
913
|
+
spanKind: 'tool_result',
|
|
914
|
+
toolCallId,
|
|
915
|
+
toolName,
|
|
916
|
+
paths,
|
|
917
|
+
primaryPath: paths[0] ?? '',
|
|
918
|
+
text: resultText,
|
|
919
|
+
}))
|
|
920
|
+
}
|
|
921
|
+
}
|
|
922
|
+
|
|
923
|
+
return spans
|
|
924
|
+
}
|
|
925
|
+
|
|
808
926
|
export function summarizeRequestSpans(spans = []) {
|
|
809
927
|
const toolNames = new Set()
|
|
810
928
|
const files = new Set()
|
|
@@ -996,12 +1114,14 @@ async function readJsonlRecords(filePath, normalize) {
|
|
|
996
1114
|
}
|
|
997
1115
|
}
|
|
998
1116
|
|
|
999
|
-
export async function ensureRequestTelemetryFiles(paths) {
|
|
1000
|
-
const
|
|
1001
|
-
|
|
1002
|
-
|
|
1117
|
+
export async function ensureRequestTelemetryFiles(paths, options = {}) {
|
|
1118
|
+
const targets = [
|
|
1119
|
+
options?.includeHooks === true ? String(paths?.hooksFile ?? '').trim() : '',
|
|
1120
|
+
options?.includeRequests !== false ? String(paths?.requestsFile ?? '').trim() : '',
|
|
1121
|
+
options?.includeSpans !== false ? String(paths?.spansFile ?? '').trim() : '',
|
|
1122
|
+
]
|
|
1003
1123
|
|
|
1004
|
-
for (const filePath of
|
|
1124
|
+
for (const filePath of targets) {
|
|
1005
1125
|
if (filePath === '') {
|
|
1006
1126
|
continue
|
|
1007
1127
|
}
|
|
@@ -1033,16 +1153,28 @@ export async function appendRequestTelemetryHook(paths, event) {
|
|
|
1033
1153
|
detail: asObject(event?.detail),
|
|
1034
1154
|
}
|
|
1035
1155
|
|
|
1036
|
-
await ensureRequestTelemetryFiles(paths
|
|
1156
|
+
await ensureRequestTelemetryFiles(paths, {
|
|
1157
|
+
includeHooks: true,
|
|
1158
|
+
includeRequests: false,
|
|
1159
|
+
includeSpans: false,
|
|
1160
|
+
})
|
|
1037
1161
|
await fs.appendFile(hooksFile, `${JSON.stringify(normalizedEvent)}\n`, 'utf8')
|
|
1038
1162
|
return normalizedEvent
|
|
1039
1163
|
}
|
|
1040
1164
|
|
|
1041
|
-
export async function appendRequestTelemetryArtifacts(paths, { request, spans = [] } = {}) {
|
|
1165
|
+
export async function appendRequestTelemetryArtifacts(paths, { request, spans = [] } = {}, options = {}) {
|
|
1042
1166
|
const normalizedRequest = normalizeRequestTelemetryRecord(request)
|
|
1043
|
-
const normalizedSpans = (Array.isArray(spans) ? spans : [])
|
|
1167
|
+
const normalizedSpans = (Array.isArray(spans) ? spans : [])
|
|
1168
|
+
.map((span) => compactRequestSpanRecord(span, {
|
|
1169
|
+
includeText: options?.includeSpanText === true,
|
|
1170
|
+
includePreview: options?.includeSpanPreview === true,
|
|
1171
|
+
}))
|
|
1044
1172
|
|
|
1045
|
-
await ensureRequestTelemetryFiles(paths
|
|
1173
|
+
await ensureRequestTelemetryFiles(paths, {
|
|
1174
|
+
includeHooks: false,
|
|
1175
|
+
includeRequests: String(paths?.requestsFile ?? '').trim() !== '',
|
|
1176
|
+
includeSpans: String(paths?.spansFile ?? '').trim() !== '' && normalizedSpans.length > 0,
|
|
1177
|
+
})
|
|
1046
1178
|
|
|
1047
1179
|
if (String(paths?.requestsFile ?? '').trim() !== '') {
|
|
1048
1180
|
await fs.appendFile(paths.requestsFile, `${JSON.stringify(normalizedRequest)}\n`, 'utf8')
|
package/src/pi-sdk-turn.mjs
CHANGED
|
@@ -16,6 +16,7 @@ import {
|
|
|
16
16
|
} from './pi-token-analysis.mjs'
|
|
17
17
|
import {
|
|
18
18
|
REQUEST_TELEMETRY_ENV_KEYS,
|
|
19
|
+
REQUEST_TELEMETRY_STORAGE_ENV_KEYS,
|
|
19
20
|
ensureBundledRequestTelemetryExtension,
|
|
20
21
|
readRequestTelemetryContextFromEnv,
|
|
21
22
|
} from './pi-request-telemetry.mjs'
|
|
@@ -124,6 +125,10 @@ function addTokenUsage(total, value) {
|
|
|
124
125
|
|
|
125
126
|
function applyRequestTelemetryEnv(request) {
|
|
126
127
|
const previous = readRequestTelemetryContextFromEnv()
|
|
128
|
+
const previousStorage = {
|
|
129
|
+
storeHooks: String(process.env[REQUEST_TELEMETRY_STORAGE_ENV_KEYS.storeHooks] ?? '').trim(),
|
|
130
|
+
storeSpanText: String(process.env[REQUEST_TELEMETRY_STORAGE_ENV_KEYS.storeSpanText] ?? '').trim(),
|
|
131
|
+
}
|
|
127
132
|
const nextValues = {
|
|
128
133
|
[REQUEST_TELEMETRY_ENV_KEYS.runId]: String(process.env.PI_RUN_ID ?? '').trim(),
|
|
129
134
|
[REQUEST_TELEMETRY_ENV_KEYS.iteration]: Number.isFinite(Number(request?.metadata?.iteration))
|
|
@@ -133,6 +138,8 @@ function applyRequestTelemetryEnv(request) {
|
|
|
133
138
|
[REQUEST_TELEMETRY_ENV_KEYS.role]: String(request?.role ?? '').trim(),
|
|
134
139
|
[REQUEST_TELEMETRY_ENV_KEYS.kind]: String(request?.kind ?? '').trim(),
|
|
135
140
|
[REQUEST_TELEMETRY_ENV_KEYS.task]: String(request?.task ?? '').trim(),
|
|
141
|
+
[REQUEST_TELEMETRY_STORAGE_ENV_KEYS.storeHooks]: request?.requestTelemetryStoreHooks === true ? '1' : '0',
|
|
142
|
+
[REQUEST_TELEMETRY_STORAGE_ENV_KEYS.storeSpanText]: request?.requestTelemetryStoreSpanText === true ? '1' : '0',
|
|
136
143
|
}
|
|
137
144
|
|
|
138
145
|
for (const [key, value] of Object.entries(nextValues)) {
|
|
@@ -152,6 +159,15 @@ function applyRequestTelemetryEnv(request) {
|
|
|
152
159
|
}
|
|
153
160
|
process.env[key] = previousValue
|
|
154
161
|
}
|
|
162
|
+
|
|
163
|
+
for (const [field, key] of Object.entries(REQUEST_TELEMETRY_STORAGE_ENV_KEYS)) {
|
|
164
|
+
const previousValue = String(previousStorage?.[field] ?? '').trim()
|
|
165
|
+
if (previousValue === '') {
|
|
166
|
+
delete process.env[key]
|
|
167
|
+
continue
|
|
168
|
+
}
|
|
169
|
+
process.env[key] = previousValue
|
|
170
|
+
}
|
|
155
171
|
}
|
|
156
172
|
}
|
|
157
173
|
|
|
@@ -9,6 +9,8 @@
|
|
|
9
9
|
"largeSpecWarningLines": 300,
|
|
10
10
|
"piModel": "local/text-model",
|
|
11
11
|
"piRequestTelemetryEnabled": true,
|
|
12
|
+
"piRequestTelemetryStoreHooks": false,
|
|
13
|
+
"piRequestTelemetryStoreSpanText": false,
|
|
12
14
|
"models": {
|
|
13
15
|
"local/text-model": {
|
|
14
16
|
"baseUrl": "http://localhost:8000/v1",
|