@sebastianandreasson/pi-autonomous-agents 0.13.1 → 0.14.0

package/README.md

@@ -80,6 +80,8 @@ Typical scripts:
 
 Start from [templates/pi.config.example.json](./templates/pi.config.example.json), [templates/DEVELOPER.md](./templates/DEVELOPER.md), [templates/TESTER.md](./templates/TESTER.md), and [templates/gitignore.fragment](./templates/gitignore.fragment).
 
+Request telemetry is enabled by default for SDK runs. `pi-harness` writes a managed Pi extension shim to `.pi/extensions/pi-harness-request-telemetry/index.mjs` in the consuming repo, and Pi auto-discovers it on the next resource reload. Disable that with `PI_REQUEST_TELEMETRY_ENABLED=0` or `"piRequestTelemetryEnabled": false`.
+
 ## CLI
 
 ```bash
@@ -183,6 +185,7 @@ Common fields in `pi.config.json`:
 - `testerInstructionsFile`
 - `transport` (`sdk` or `mock`)
 - `piModel`
+- `piRequestTelemetryEnabled`
 - `models`
 - `roleModels`
 - `commitMode`
@@ -324,6 +327,8 @@ That clears configured harness runtime/history artifacts and verifies they are g
   More detailed flow, transport, telemetry, and runtime documentation.
 - [docs/TOKEN_USAGE_ARTIFACTS.md](./docs/TOKEN_USAGE_ARTIFACTS.md)
   Agent-facing contract and usage guidance for token-usage artifacts and downstream tooling.
+- [docs/PI_REQUEST_TELEMETRY_EXTENSION.md](./docs/PI_REQUEST_TELEMETRY_EXTENSION.md)
+  Repo-local Pi extension prototype for request-level telemetry via Pi hooks.
 - [templates/PROJECT_SETUP.md](./templates/PROJECT_SETUP.md)
   Minimal consuming-repo layout summary.
 

package/docs/PI_REQUEST_TELEMETRY_EXTENSION.md

@@ -0,0 +1,182 @@
+# Pi Request Telemetry Extension
+
+This document describes the repo-local Pi extension prototype under [pi-extensions/request-telemetry](../pi-extensions/request-telemetry/README.md).
+
+In normal `pi-harness` SDK runs, this extension is auto-enabled by installing a managed shim under `.pi/extensions/pi-harness-request-telemetry/` in the consuming repo before Pi reloads resources. Opt out with `PI_REQUEST_TELEMETRY_ENABLED=0` or `"piRequestTelemetryEnabled": false`.
+
+The purpose of this extension is to gather request-level data directly from Pi extension hooks before we decide whether to patch `@mariozechner/pi-coding-agent` or `pi-ai`.
+
+## Produced Artifacts
+
+- `pi-output/request-telemetry/hooks.jsonl`
+- `pi-output/request-telemetry/requests.jsonl`
+- `pi-output/request-telemetry/spans.jsonl`
+
+These artifacts are intentionally separate from the existing `pi-output/token-usage/*` files.
+
+`token-usage` remains the harness-level normalized output.
+`request-telemetry` is the lower-level Pi-extension probe for real request boundaries and prompt composition.
+
+## Hook Coverage
+
+The extension listens to:
+
+- `session_start`
+- `session_switch`
+- `model_select`
+- `turn_start`
+- `context`
+- `before_provider_request`
+- `after_provider_response`
+- `tool_execution_start`
+- `tool_result`
+- `message_end`
+- `turn_end`
+
+The documented hooks come from Pi’s extension API:
+
+- [Extension Hooks](https://pt-act-pi-mono.mintlify.app/api/coding-agent/hooks)
+- [Extension System](https://pt-act-pi-mono.mintlify.app/concepts/extensions)
+
+The current source also exposes undocumented provider-boundary hooks:
+
+- `before_provider_request`
+- `after_provider_response`
+
+Source:
+
+- [packages/coding-agent/src/core/extensions/types.ts](https://github.com/badlogic/pi-mono/blob/main/packages/coding-agent/src/core/extensions/types.ts)
+
+## Request Artifact Schema
+
+Each row in `requests.jsonl` contains one provider request finalized by the following assistant `message_end`:
+
+- `schemaVersion`
+- `timestamp`
+- `requestId`
+- `sessionId`
+- `turnIndex`
+- `startedAt`
+- `finishedAt`
+- `durationMs`
+- `statusCode`
+- `provider`
+- `model`
+- `api`
+- `stopReason`
+- `usageSource`
+- `source`
+- `spanSource`
+- `contextMessageCount`
+- `spanCount`
+- `textChars`
+- `textBytes`
+- `toolNames`
+- `files`
+- `providerPayloadSummary`
+- `responseHeaders`
+- `inputTokens`
+- `outputTokens`
+- `totalTokens`
+- `cacheReadTokens`
+- `cacheWriteTokens`
+
+`usageSource` is one of:
+
+- `message_usage`
+- `unavailable`
+
+This is intentionally conservative. The extension does not invent totals when the provider does not expose them.
+
+`spanSource` describes where the prompt-span reconstruction came from:
+
+- `provider_payload`
+- `context`
+- `session_history`
+
+Interpretation:
+
+- `provider_payload` is the strongest source because it comes from the final request payload.
+- `context` is also exact for the pre-request message set Pi exposed to the extension.
+- `session_history` is a fallback when Pi omits both `context` and provider payload hooks for a given assistant response.
+
+`hooks.jsonl` is a lifecycle trace of:
+
+- `turn_start`
+- `context`
+- `before_provider_request`
+- `after_provider_response`
+- `message_end`
+- `turn_end`
+
+Use it to debug request association problems or confirm whether Pi emitted provider-boundary hooks for a specific run.
+
+## Span Artifact Schema
+
+Each row in `spans.jsonl` contains one extracted prompt span:
+
+- `schemaVersion`
+- `timestamp`
+- `requestId`
+- `sessionId`
+- `turnIndex`
+- `source`
+- `role`
+- `messageIndex`
+- `spanIndex`
+- `spanKind`
+- `toolCallId`
+- `toolName`
+- `paths`
+- `primaryPath`
+- `charCount`
+- `byteCount`
+- `text`
+- `preview`
+
+Current `spanKind` values include:
+
+- `text`
+- `thinking`
+- `tool_call`
+- `tool_result`
+- `image`
+- `unknown`
+
+## Exact vs Inferred
+
+This extension is meant to move the data model closer to real accounting.
+
+Exact today:
+
+- provider request boundaries when `before_provider_request` fires
+- pre-request context messages
+- provider-payload-derived spans when Pi exposes the final payload
+- provider payload summary
+- tool/file context seen by extension hooks
+- final request totals when `message.usage` is present
+
+Not exact today:
+
+- token counts for each prompt span
+- per-file token billing
+- request totals when the backend omits usage entirely
+- `session_history` span reconstructions when neither `context` nor provider payload is exposed
+
+That means these artifacts are suitable for:
+
+- validating whether Pi/provider already exposes exact request usage
+- seeing the true prompt composition for each provider request
+- building tokenizer-backed attribution later from exact captured spans
+
+They are not yet a replacement for exact per-file cost accounting.
+
+## Recommended Next Step
+
+Use this extension first to answer two questions:
+
+1. Does your backend populate `message.usage` consistently?
+2. Does `before_provider_request` give enough final payload detail to tokenize exact prompt spans?
+
+If both answers are yes, the next step is tokenizer-backed per-span accounting.
+If usage is still missing, the right fix is likely upstream in `pi-ai` or `pi-coding-agent`, not more harness-side heuristics.

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@sebastianandreasson/pi-autonomous-agents",
   "private": false,
-  "version": "0.13.1",
+  "version": "0.14.0",
   "type": "module",
   "description": "Portable unattended PI harness for developer/tester/visual-review loops.",
   "license": "MIT",
@@ -19,8 +19,8 @@
     "@mariozechner/pi-coding-agent": "^0.66.1"
   },
   "scripts": {
-    "check": "node --check src/cli.mjs && node --check src/pi-clear-history.mjs && node --check src/pi-client.mjs && node --check src/pi-config.mjs && node --check src/pi-debug-live.mjs && node --check src/pi-flow.mjs && node --check src/pi-heartbeat.mjs && node --check src/pi-history.mjs && node --check src/pi-preflight.mjs && node --check src/pi-prompts.mjs && node --check src/pi-repo.mjs && node --check src/pi-report.mjs && node --check src/pi-sdk-turn.mjs && node --check src/pi-supervisor.mjs && node --check src/pi-telemetry.mjs && node --check src/pi-token-analysis.mjs && node --check src/pi-visual-once.mjs && node --check src/pi-visual-review.mjs && node --check src/pi-visualizer.mjs && node --check src/pi-visualizer-server.mjs && node --check src/pi-visualizer-shared.mjs && node --check src/index.mjs && node --check test/pi-heartbeat.test.mjs && node --check test/pi-lifecycle.test.mjs && node --check test/pi-role-models.test.mjs && node --check test/pi-flow.test.mjs && node --check test/pi-history.test.mjs && node --check test/pi-prompts.test.mjs && node --check test/pi-preflight.test.mjs && node --check test/pi-repo.test.mjs && node --check test/pi-sdk-supervisor.test.mjs && node --check test/pi-sdk-turn.test.mjs && node --check test/pi-telemetry.test.mjs && node --check test/pi-token-analysis.test.mjs && node --check test/pi-visualizer-shared.test.mjs && node --check test/fixtures/fake-pi.mjs && node --check test/fixtures/fake-pi-sdk.mjs && node --check test/fixtures/fake-live-pi-sdk.mjs",
-    "test": "node --test test/pi-heartbeat.test.mjs test/pi-lifecycle.test.mjs test/pi-role-models.test.mjs test/pi-flow.test.mjs test/pi-history.test.mjs test/pi-prompts.test.mjs test/pi-preflight.test.mjs test/pi-repo.test.mjs test/pi-sdk-supervisor.test.mjs test/pi-sdk-turn.test.mjs test/pi-telemetry.test.mjs test/pi-token-analysis.test.mjs test/pi-visualizer-shared.test.mjs",
+    "check": "node --check src/cli.mjs && node --check src/pi-clear-history.mjs && node --check src/pi-client.mjs && node --check src/pi-config.mjs && node --check src/pi-debug-live.mjs && node --check src/pi-flow.mjs && node --check src/pi-heartbeat.mjs && node --check src/pi-history.mjs && node --check src/pi-preflight.mjs && node --check src/pi-prompts.mjs && node --check src/pi-repo.mjs && node --check src/pi-report.mjs && node --check src/pi-request-telemetry.mjs && node --check src/pi-sdk-turn.mjs && node --check src/pi-supervisor.mjs && node --check src/pi-telemetry.mjs && node --check src/pi-token-analysis.mjs && node --check src/pi-visual-once.mjs && node --check src/pi-visual-review.mjs && node --check src/pi-visualizer.mjs && node --check src/pi-visualizer-server.mjs && node --check src/pi-visualizer-shared.mjs && node --check src/index.mjs && node --check pi-extensions/request-telemetry/index.mjs && node --check test/pi-heartbeat.test.mjs && node --check test/pi-lifecycle.test.mjs && node --check test/pi-request-telemetry.test.mjs && node --check test/pi-role-models.test.mjs && node --check test/pi-flow.test.mjs && node --check test/pi-history.test.mjs && node --check test/pi-prompts.test.mjs && node --check test/pi-preflight.test.mjs && node --check test/pi-repo.test.mjs && node --check test/pi-sdk-supervisor.test.mjs && node --check test/pi-sdk-turn.test.mjs && node --check test/pi-telemetry.test.mjs && node --check test/pi-token-analysis.test.mjs && node --check test/pi-visualizer-shared.test.mjs && node --check test/fixtures/fake-pi.mjs && node --check test/fixtures/fake-pi-sdk.mjs && node --check test/fixtures/fake-live-pi-sdk.mjs",
+    "test": "node --test test/pi-heartbeat.test.mjs test/pi-lifecycle.test.mjs test/pi-request-telemetry.test.mjs test/pi-role-models.test.mjs test/pi-flow.test.mjs test/pi-history.test.mjs test/pi-prompts.test.mjs test/pi-preflight.test.mjs test/pi-repo.test.mjs test/pi-sdk-supervisor.test.mjs test/pi-sdk-turn.test.mjs test/pi-telemetry.test.mjs test/pi-token-analysis.test.mjs test/pi-visualizer-shared.test.mjs",
     "debug:live-ui": "node src/cli.mjs debug-live --reset",
     "dev:visualizer:ui": "npm --prefix visualizer-ui run dev",
     "build:visualizer:ui": "npm --prefix visualizer-ui run build",
@@ -28,6 +28,7 @@
   },
   "files": [
     "src",
+    "pi-extensions",
     "templates",
     "docs",
     "visualizer-ui/dist",

package/pi-extensions/request-telemetry/README.md

@@ -0,0 +1,66 @@
+# Request Telemetry Pi Extension
+
+This is a repo-local Pi extension prototype for capturing lower-level request telemetry than `pi-harness` can see on its own.
+
+It writes:
+
+- `pi-output/request-telemetry/hooks.jsonl`
+- `pi-output/request-telemetry/requests.jsonl`
+- `pi-output/request-telemetry/spans.jsonl`
+
+The goal is to capture exact request boundaries and exact prompt composition before deciding whether we need to patch `pi-mono` itself.
+
+## What It Captures
+
+- one request record per provider request
+- exact `context` messages seen before the provider call
+- exact prompt spans extracted from the provider payload when available
+- context-derived fallback spans when Pi exposes `context` but not the final payload
+- tool names and file paths seen through tool hooks
+- provider payload summary from `before_provider_request`
+- response status and headers from `after_provider_response`
+- final assistant-message usage if Pi exposes it on `message.usage`
+- lifecycle hook traces in `hooks.jsonl` so you can debug request association
+- `spanSource` on each request so consumers can distinguish:
+  - `provider_payload`
+  - `context`
+  - `session_history`
+
+## What It Does Not Claim Yet
+
+- exact per-file token spend
+- exact per-span token counts
+- exact request usage when the provider does not return usage
+
+Those need either:
+
+- normalized request-usage exposure from Pi/provider adapters
+- or tokenizer-backed accounting from the exact serialized payload
+
+## Normal Package Behavior
+
+When `pi-harness` runs through the SDK transport, it now installs a managed shim at:
+
+- `.pi/extensions/pi-harness-request-telemetry/index.mjs`
+
+That makes request telemetry auto-load in consuming repos without needing `--extension`.
+
+Disable that path with:
+
+- `PI_REQUEST_TELEMETRY_ENABLED=0`
+- `"piRequestTelemetryEnabled": false` in `pi.config.json`
+
+## Running It From This Repo
+
+Use the extension file directly:
+
+```bash
+pi --extension ./pi-extensions/request-telemetry/index.mjs
+```
+
+Or copy or symlink it into a Pi extension directory:
+
+- project-local: `./.pi/extensions/`
+- global: `~/.pi/agent/extensions/`
+
+If you move only `index.mjs`, it will break because it currently imports helpers from this repo at `src/pi-request-telemetry.mjs`. Keep the repo checkout intact for this prototype.