pi-ui-extend 1.0.38 → 1.0.40
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +1 -1
- package/dist/app/extensions/extension-ui-controller.js +3 -0
- package/dist/app/rendering/extension-entry-renderer.js +2 -0
- package/dist/bundled-extensions/question/contract.d.ts +3 -0
- package/dist/bundled-extensions/question/contract.js +24 -0
- package/dist/bundled-extensions/question/desktop.d.ts +11 -0
- package/dist/bundled-extensions/question/desktop.js +143 -0
- package/dist/bundled-extensions/question/index.d.ts +1 -0
- package/dist/bundled-extensions/question/index.js +5 -1
- package/dist/bundled-extensions/question/render.js +6 -0
- package/dist/bundled-extensions/question/result.js +71 -9
- package/dist/bundled-extensions/question/tool-description.js +4 -3
- package/dist/bundled-extensions/question/tui.js +127 -12
- package/dist/bundled-extensions/question/types.d.ts +23 -2
- package/dist/tool-renderers/question.js +20 -1
- package/docs/desktop-markdown-media.md +77 -0
- package/docs/desktop-mvp.md +134 -0
- package/docs/desktop-task-manager.md +124 -0
- package/external/pi-tools-suite/README.md +207 -0
- package/external/pi-tools-suite/docs/evals.md +684 -0
- package/external/pi-tools-suite/package.json +7 -3
- package/external/pi-tools-suite/src/async-subagents/async-subagents.sample.jsonc +8 -0
- package/external/pi-tools-suite/src/async-subagents/core/agent-strategy.ts +41 -2
- package/external/pi-tools-suite/src/async-subagents/core/config.ts +12 -0
- package/external/pi-tools-suite/src/async-subagents/index.ts +6 -2
- package/external/pi-tools-suite/src/async-subagents/subagent-overlay.ts +1 -1
- package/external/pi-tools-suite/src/coding-discipline/index.ts +41 -142
- package/external/pi-tools-suite/src/config.ts +0 -21
- package/external/pi-tools-suite/src/dcp/auto-compress.ts +1 -1
- package/external/pi-tools-suite/src/dcp/compression-blocks.ts +1 -51
- package/external/pi-tools-suite/src/dcp/debug-log.ts +6 -0
- package/external/pi-tools-suite/src/dcp/index.ts +27 -125
- package/external/pi-tools-suite/src/dcp/prompts.ts +2 -2
- package/external/pi-tools-suite/src/dcp/provider-tool-results.ts +2 -1
- package/external/pi-tools-suite/src/dcp/pruner-candidates.ts +31 -10
- package/external/pi-tools-suite/src/dcp/pruner-compression-blocks.ts +6 -7
- package/external/pi-tools-suite/src/dcp/pruner-message-ids.ts +153 -111
- package/external/pi-tools-suite/src/dcp/pruner-metadata.ts +28 -17
- package/external/pi-tools-suite/src/dcp/pruner-nudge.ts +61 -27
- package/external/pi-tools-suite/src/dcp/pruner.ts +21 -13
- package/external/pi-tools-suite/src/dcp/state.ts +59 -0
- package/external/pi-tools-suite/src/default-pi-tools-suite-config.ts +27 -6
- package/external/pi-tools-suite/src/lib/rpc-session-state.ts +34 -0
- package/external/pi-tools-suite/src/todo/todo.ts +8 -3
- package/external/pi-tools-suite/src/tool-descriptions.ts +4 -3
- package/package.json +7 -4
|
@@ -0,0 +1,124 @@
|
|
|
1
|
+
# Spec: Desktop Project Task Manager
|
|
2
|
+
|
|
3
|
+
## Type
|
|
4
|
+
|
|
5
|
+
Change
|
|
6
|
+
|
|
7
|
+
## Goal
|
|
8
|
+
|
|
9
|
+
Add a project-scoped task list to Pix Desktop in a persistent left sidebar and
|
|
10
|
+
allow a saved task to start work immediately in a new session tab.
|
|
11
|
+
|
|
12
|
+
## Scope
|
|
13
|
+
|
|
14
|
+
- A collapsible, resizable left sidebar with `Tasks` and `Project` tabs.
|
|
15
|
+
- A compact task list with create, edit, delete, filtering, and manual status
|
|
16
|
+
changes.
|
|
17
|
+
- Task type (`bug`, `feature`, `improvement`), status (`backlog`, `todo`,
|
|
18
|
+
`in-progress`, `done`), and priority (`low`, `medium`, `high`, `urgent`).
|
|
19
|
+
- Project-local persistence in `.pi/tasks.json` with a versioned schema.
|
|
20
|
+
- Starting a task in a new ACP session, automatically sending a generated
|
|
21
|
+
prompt, marking the task in progress, and storing the new session id.
|
|
22
|
+
- Opening an already-linked session instead of creating a duplicate.
|
|
23
|
+
|
|
24
|
+
## Non-goals
|
|
25
|
+
|
|
26
|
+
- Kanban, drag and drop, subtasks, dependencies, assignees, due dates,
|
|
27
|
+
comments, or task history.
|
|
28
|
+
- Automatic transition to `done` when an agent turn finishes.
|
|
29
|
+
- Synchronizing the project task file with the agent's session-local todo list.
|
|
30
|
+
- Concurrent multi-window file merging.
|
|
31
|
+
|
|
32
|
+
## Behavior
|
|
33
|
+
|
|
34
|
+
1. The sidebar is visible below the title/session chrome, starts on `Tasks`,
|
|
35
|
+
remembers its width and collapsed state locally, and preserves the main
|
|
36
|
+
session workspace at narrow window sizes.
|
|
37
|
+
2. Task rows show a title, type, status, priority, and actions. Status and
|
|
38
|
+
priority are communicated by a Lucide icon, text, and color so meaning never
|
|
39
|
+
depends on color alone. `done` is green, `in-progress` is yellow, and idle
|
|
40
|
+
states are neutral; all colors support light and dark themes.
|
|
41
|
+
3. Creating or editing validates a non-empty title. A description is optional.
|
|
42
|
+
4. Filters affect only the visible list and do not modify persisted ordering.
|
|
43
|
+
5. A missing `.pi/tasks.json` is treated as an empty version-1 document.
|
|
44
|
+
6. Starting an unlinked task creates and selects a new ACP session, persists
|
|
45
|
+
its id and `in-progress` status, appends the generated user prompt to the
|
|
46
|
+
transcript, and sends it immediately.
|
|
47
|
+
7. Starting a linked task loads that session. A missing/stale linked session is
|
|
48
|
+
reported as a recoverable error and does not silently create another one.
|
|
49
|
+
8. Task completion remains manual.
|
|
50
|
+
|
|
51
|
+
## Contracts
|
|
52
|
+
|
|
53
|
+
- Project file: `.pi/tasks.json` containing `{ version: 1, tasks: [...] }`.
|
|
54
|
+
- Every task has a unique id, title, type, status, priority, `createdAt`, and
|
|
55
|
+
`updatedAt`; description and `sessionId` may be omitted.
|
|
56
|
+
- Tauri exposes narrow read/write commands for this one file. Workspace paths
|
|
57
|
+
are validated and `.pi` symlinks may not escape the workspace.
|
|
58
|
+
- The webview sends a complete validated document on each mutation. Writes use
|
|
59
|
+
a same-directory temporary file and replacement to avoid partial JSON.
|
|
60
|
+
|
|
61
|
+
## Invariants
|
|
62
|
+
|
|
63
|
+
- The task document version must be supported before it is displayed or saved.
|
|
64
|
+
- Duplicate ids, unknown enum values, empty titles, and malformed timestamps
|
|
65
|
+
are rejected.
|
|
66
|
+
- A task is linked to at most one session in this MVP.
|
|
67
|
+
- Starting a task is disabled while another prompt/session operation is active.
|
|
68
|
+
- A failed save does not leave the UI claiming a task state that was not
|
|
69
|
+
persisted.
|
|
70
|
+
|
|
71
|
+
## Edge cases
|
|
72
|
+
|
|
73
|
+
- Switching workspaces discards the previous workspace's in-memory task view
|
|
74
|
+
and loads the new project file.
|
|
75
|
+
- Empty and missing files have distinct behavior: missing means no tasks;
|
|
76
|
+
malformed or empty JSON is an error.
|
|
77
|
+
- Save, session creation, and prompt failures remain visible and retryable.
|
|
78
|
+
- Collapsing the sidebar keeps its tab rail available; choosing a tab expands
|
|
79
|
+
it.
|
|
80
|
+
|
|
81
|
+
## Related files
|
|
82
|
+
|
|
83
|
+
- `desktop/src/App.svelte`
|
|
84
|
+
- `desktop/src/components/`
|
|
85
|
+
- `desktop/src/lib/`
|
|
86
|
+
- `desktop/src/styles.css`
|
|
87
|
+
- `desktop/src-tauri/src/lib.rs`
|
|
88
|
+
|
|
89
|
+
## Verification
|
|
90
|
+
|
|
91
|
+
- Unit tests for task parsing, validation, filtering, prompt generation, and
|
|
92
|
+
sidebar preference parsing.
|
|
93
|
+
- Rust tests for missing/read/write/malformed files and workspace escape.
|
|
94
|
+
- `npm --prefix desktop test`
|
|
95
|
+
- `npm --prefix desktop run check`
|
|
96
|
+
- `npm --prefix desktop run build:web`
|
|
97
|
+
- `cargo test --manifest-path desktop/src-tauri/Cargo.toml`
|
|
98
|
+
|
|
99
|
+
## Risks / unknowns
|
|
100
|
+
|
|
101
|
+
- Whole-document writes assume one active desktop writer per project.
|
|
102
|
+
- A linked session may later be removed outside Pix Desktop; the first version
|
|
103
|
+
reports this rather than automatically unlinking it.
|
|
104
|
+
- Windows replacement semantics may require a fallback around replacing an
|
|
105
|
+
existing task file while retaining best-effort crash safety.
|
|
106
|
+
|
|
107
|
+
## Evidence
|
|
108
|
+
|
|
109
|
+
- Confirmed by code: desktop session creation/loading and prompt submission are
|
|
110
|
+
orchestrated in `desktop/src/App.svelte` through `AcpClient`.
|
|
111
|
+
- Confirmed by code: Tauri already validates workspace-contained project file
|
|
112
|
+
reads in `desktop/src-tauri/src/lib.rs`.
|
|
113
|
+
- Confirmed by tests: session tab ordering and ACP request behavior have focused
|
|
114
|
+
Vitest coverage under `desktop/src/lib/`.
|
|
115
|
+
- Confirmed by docs: `DESIGN.md` specifies a compact persistent sidebar,
|
|
116
|
+
semantic theme roles, Lucide icons, and non-color status cues.
|
|
117
|
+
- Implemented: `.pi/tasks.json` now has an explicit version-1 contract and
|
|
118
|
+
rejects unsupported versions rather than guessing a migration.
|
|
119
|
+
- Verified by tests: all 82 desktop Vitest tests and all 8 Rust tests pass;
|
|
120
|
+
Svelte/TypeScript checks report no errors or warnings.
|
|
121
|
+
- Verified visually: browser QA covered clean preference state, resize/collapse,
|
|
122
|
+
tabs, filters, and semantic colors in light/dark themes.
|
|
123
|
+
- Verified natively: actual Tauri UI/backend CRUD, Run, persisted linkage,
|
|
124
|
+
Open session without duplication, and delete cleanup all passed.
|
|
@@ -188,9 +188,28 @@ Install the language servers used by the bundled example config. The commands be
|
|
|
188
188
|
# TypeScript / JavaScript
|
|
189
189
|
npm install -g typescript typescript-language-server
|
|
190
190
|
|
|
191
|
+
# Svelte
|
|
192
|
+
npm install -g svelte-language-server
|
|
193
|
+
|
|
194
|
+
# Vue
|
|
195
|
+
npm install -g @vue/language-server
|
|
196
|
+
|
|
191
197
|
# Python
|
|
192
198
|
python3 -m pip install --user python-lsp-server
|
|
193
199
|
|
|
200
|
+
# Go
|
|
201
|
+
go install golang.org/x/tools/gopls@latest
|
|
202
|
+
|
|
203
|
+
# C / C++ (clangd)
|
|
204
|
+
brew install llvm
|
|
205
|
+
# or use your distro's clangd package: apt install clangd, dnf install clang-tools-extra, ...
|
|
206
|
+
|
|
207
|
+
# Lua
|
|
208
|
+
brew install lua-language-server
|
|
209
|
+
|
|
210
|
+
# Bash
|
|
211
|
+
npm install -g bash-language-server
|
|
212
|
+
|
|
194
213
|
# C# / Unity
|
|
195
214
|
dotnet tool install -g Microsoft.CodeAnalysis.LanguageServer
|
|
196
215
|
|
|
@@ -252,6 +271,160 @@ Minimal shared config shape:
|
|
|
252
271
|
|
|
253
272
|
Project-local overrides can be added in `.pi/pi-tools-suite.jsonc`; pi-tools-suite asks for trust before using project-local LSP binaries.
|
|
254
273
|
|
|
274
|
+
### Popular language-server examples
|
|
275
|
+
|
|
276
|
+
Copy the entries you need into `lsp.servers` of the shared config. Values mirror the commented templates shipped in the generated config file; where they disagree, the generated file is authoritative. Both diagnostics modes are on by default: servers that support pull diagnostics (`textDocument/diagnostic`) are queried directly, and push diagnostics (`publishDiagnostics`) are awaited for every server — `pullDiagnostics: false` / `waitForPublishDiagnostics: false` disable either side explicitly. Servers start lazily: one spawns only after a mutating tool (Edit/Write/ast-grep/apply_patch) touches a file matching `include`, and diagnostics land in that tool's result.
|
|
277
|
+
|
|
278
|
+
```jsonc
|
|
279
|
+
{
|
|
280
|
+
"lsp": {
|
|
281
|
+
"servers": [
|
|
282
|
+
// Svelte (verified with svelte-language-server): compiler + embedded TS/JS diagnostics
|
|
283
|
+
{
|
|
284
|
+
"id": "svelte",
|
|
285
|
+
"include": ["**/*.svelte"],
|
|
286
|
+
"exclude": ["**/node_modules/**"],
|
|
287
|
+
"rootMarkers": ["svelte.config.js", "package.json"],
|
|
288
|
+
"bin": "svelteserver",
|
|
289
|
+
"args": ["--stdio"],
|
|
290
|
+
"startupTimeoutMs": 30000,
|
|
291
|
+
"diagnosticsWaitMs": 8000,
|
|
292
|
+
"languageIdByExtension": { ".svelte": "svelte" }
|
|
293
|
+
},
|
|
294
|
+
// Vue (Volar)
|
|
295
|
+
{
|
|
296
|
+
"id": "vue",
|
|
297
|
+
"include": ["**/*.vue"],
|
|
298
|
+
"exclude": ["**/node_modules/**"],
|
|
299
|
+
"rootMarkers": ["package.json"],
|
|
300
|
+
"bin": "vue-language-server",
|
|
301
|
+
"args": ["--stdio"],
|
|
302
|
+
"startupTimeoutMs": 30000,
|
|
303
|
+
"diagnosticsWaitMs": 8000,
|
|
304
|
+
"languageIdByExtension": { ".vue": "vue" }
|
|
305
|
+
},
|
|
306
|
+
// Python (python-lsp-server)
|
|
307
|
+
{
|
|
308
|
+
"id": "python",
|
|
309
|
+
"include": ["**/*.py", "**/*.pyi"],
|
|
310
|
+
"exclude": ["**/.git/**", "**/node_modules/**", "**/__pycache__/**", "**/.venv/**", "**/venv/**", "**/.tox/**", "**/.mypy_cache/**", "**/.ruff_cache/**"],
|
|
311
|
+
"rootMarkers": ["pyproject.toml", "setup.py", "setup.cfg", "requirements.txt", "Pipfile", "poetry.lock", ".git"],
|
|
312
|
+
"bin": "pylsp",
|
|
313
|
+
"args": [],
|
|
314
|
+
"languageIdByExtension": { ".py": "python", ".pyi": "python" }
|
|
315
|
+
},
|
|
316
|
+
// Go (gopls, push diagnostics)
|
|
317
|
+
{
|
|
318
|
+
"id": "go",
|
|
319
|
+
"include": ["**/*.go"],
|
|
320
|
+
"exclude": ["**/.git/**", "**/vendor/**"],
|
|
321
|
+
"rootMarkers": ["go.mod", ".git"],
|
|
322
|
+
"bin": "gopls",
|
|
323
|
+
"args": [],
|
|
324
|
+
"startupTimeoutMs": 20000,
|
|
325
|
+
"diagnosticsWaitMs": 8000,
|
|
326
|
+
"languageIdByExtension": { ".go": "go" }
|
|
327
|
+
},
|
|
328
|
+
// Rust (rust-analyzer, push diagnostics)
|
|
329
|
+
{
|
|
330
|
+
"id": "rust",
|
|
331
|
+
"include": ["**/*.rs"],
|
|
332
|
+
"exclude": ["**/.git/**", "**/node_modules/**", "**/target/**"],
|
|
333
|
+
"rootMarkers": ["Cargo.toml", "rust-project.json", ".git"],
|
|
334
|
+
"bin": "rust-analyzer",
|
|
335
|
+
"args": [],
|
|
336
|
+
"startupTimeoutMs": 20000,
|
|
337
|
+
"diagnosticsWaitMs": 20000,
|
|
338
|
+
"pullDiagnostics": false,
|
|
339
|
+
"waitForPublishDiagnostics": true,
|
|
340
|
+
"languageIdByExtension": { ".rs": "rust" }
|
|
341
|
+
},
|
|
342
|
+
// C / C++ (clangd, push diagnostics)
|
|
343
|
+
{
|
|
344
|
+
"id": "clangd",
|
|
345
|
+
"include": ["**/*.c", "**/*.cc", "**/*.cpp", "**/*.cxx", "**/*.h", "**/*.hh", "**/*.hpp"],
|
|
346
|
+
"exclude": ["**/.git/**", "**/node_modules/**"],
|
|
347
|
+
"rootMarkers": ["compile_commands.json", "CMakeLists.txt", "Makefile", ".clang-format", ".git"],
|
|
348
|
+
"bin": "clangd",
|
|
349
|
+
"args": [],
|
|
350
|
+
"startupTimeoutMs": 20000,
|
|
351
|
+
"diagnosticsWaitMs": 8000,
|
|
352
|
+
"languageIdByExtension": {
|
|
353
|
+
".c": "c", ".cc": "cpp", ".cpp": "cpp", ".cxx": "cpp",
|
|
354
|
+
".h": "c", ".hh": "cpp", ".hpp": "cpp"
|
|
355
|
+
}
|
|
356
|
+
},
|
|
357
|
+
// C# / Unity (Roslyn)
|
|
358
|
+
{
|
|
359
|
+
"id": "csharp",
|
|
360
|
+
"include": ["**/*.cs", "**/*.csx"],
|
|
361
|
+
"exclude": ["**/.git/**", "**/node_modules/**", "**/bin/**", "**/obj/**", "**/.vs/**", "**/Library/**", "**/Temp/**", "**/Logs/**"],
|
|
362
|
+
"rootMarkers": ["*.sln", "*.csproj", "global.json", "Directory.Build.props", "Directory.Packages.props", "Packages/manifest.json", "ProjectSettings/ProjectVersion.txt", ".git"],
|
|
363
|
+
"bin": "~/.dotnet/tools/roslyn-language-server",
|
|
364
|
+
"args": ["--stdio", "--autoLoadProjects", "--logLevel", "Error"],
|
|
365
|
+
"startupTimeoutMs": 30000,
|
|
366
|
+
"diagnosticsWaitMs": 15000,
|
|
367
|
+
"languageIdByExtension": { ".cs": "csharp", ".csx": "csharp" }
|
|
368
|
+
},
|
|
369
|
+
// Ruby
|
|
370
|
+
{
|
|
371
|
+
"id": "ruby",
|
|
372
|
+
"include": ["**/*.rb", "**/*.rake", "**/Gemfile", "**/Rakefile", "**/*.gemspec"],
|
|
373
|
+
"exclude": ["**/.git/**", "**/node_modules/**", "**/vendor/bundle/**", "**/.bundle/**", "**/tmp/**", "**/log/**"],
|
|
374
|
+
"rootMarkers": ["Gemfile.lock", "*.gemspec", "Rakefile", ".ruby-version", ".git"],
|
|
375
|
+
"bin": "ruby-lsp",
|
|
376
|
+
"args": [],
|
|
377
|
+
"startupTimeoutMs": 60000,
|
|
378
|
+
"diagnosticsWaitMs": 10000,
|
|
379
|
+
"languageIdByExtension": { ".rb": "ruby", ".rake": "ruby", ".gemspec": "ruby" }
|
|
380
|
+
},
|
|
381
|
+
// Lua (lua-language-server)
|
|
382
|
+
{
|
|
383
|
+
"id": "lua",
|
|
384
|
+
"include": ["**/*.lua"],
|
|
385
|
+
"exclude": ["**/.git/**", "**/node_modules/**"],
|
|
386
|
+
"rootMarkers": [".luarc.json", ".git"],
|
|
387
|
+
"bin": "lua-language-server",
|
|
388
|
+
"args": [],
|
|
389
|
+
"startupTimeoutMs": 30000,
|
|
390
|
+
"diagnosticsWaitMs": 6000,
|
|
391
|
+
"languageIdByExtension": { ".lua": "lua" }
|
|
392
|
+
},
|
|
393
|
+
// Bash
|
|
394
|
+
{
|
|
395
|
+
"id": "bash",
|
|
396
|
+
"include": ["**/*.sh", "**/*.bash"],
|
|
397
|
+
"exclude": ["**/.git/**", "**/node_modules/**"],
|
|
398
|
+
"rootMarkers": [".git"],
|
|
399
|
+
"bin": "bash-language-server",
|
|
400
|
+
"args": ["start"],
|
|
401
|
+
"startupTimeoutMs": 15000,
|
|
402
|
+
"diagnosticsWaitMs": 5000,
|
|
403
|
+
"languageIdByExtension": { ".sh": "shellscript", ".bash": "shellscript" }
|
|
404
|
+
},
|
|
405
|
+
// Markdown (link validation settings ship in the generated config template)
|
|
406
|
+
{
|
|
407
|
+
"id": "markdown",
|
|
408
|
+
"include": ["**/*.md", "**/*.markdown", "**/*.mdown", "**/*.mkd", "**/*.mmd"],
|
|
409
|
+
"exclude": ["**/.git/**", "**/node_modules/**"],
|
|
410
|
+
"rootMarkers": [".git", "package.json", "README.md"],
|
|
411
|
+
"bin": "vscode-markdown-language-server",
|
|
412
|
+
"args": ["--stdio"],
|
|
413
|
+
"startupTimeoutMs": 15000,
|
|
414
|
+
"diagnosticsWaitMs": 5000,
|
|
415
|
+
"languageIdByExtension": { ".md": "markdown", ".markdown": "markdown", ".mdown": "markdown", ".mkd": "markdown", ".mmd": "markdown" }
|
|
416
|
+
}
|
|
417
|
+
]
|
|
418
|
+
}
|
|
419
|
+
}
|
|
420
|
+
```
|
|
421
|
+
|
|
422
|
+
Notes:
|
|
423
|
+
|
|
424
|
+
- Svelte resolves `svelte` and `typescript` from the workspace `node_modules`, so project-local versions win; `.svelte.js`/`.svelte.ts` runes modules are not covered because their extensions collide with the TypeScript server.
|
|
425
|
+
- Vue requires `@vue/language-server` v2+ (the `vue-language-server` binary) plus the workspace's `vue` package for template type-checking.
|
|
426
|
+
- The full commented templates (including GDScript via a headless Godot wrapper and the complete Markdown link-validation `settings`) are written to the shared config file on first run.
|
|
427
|
+
|
|
255
428
|
## Async sub-agents
|
|
256
429
|
|
|
257
430
|
Sub-agent model routing normally follows task overrides, subagent type config, then `ASYNC_SUBAGENTS_MODEL` / `PI_SUBAGENTS_MODEL` fallbacks. Set `ASYNC_SUBAGENTS_FORCE_CURRENT_MODEL=1` (or `PI_SUBAGENTS_FORCE_CURRENT_MODEL=1`) to ignore task/config/env model choices and launch every sub-agent with the current parent session model. When this flag is enabled, any `--model` entries in sub-agent extra args are stripped so they cannot override the current model.
|
|
@@ -520,6 +693,40 @@ npm run test:prompt-evals:dcp
|
|
|
520
693
|
|
|
521
694
|
The default live model is `zai/glm-5-turbo`. Override it for the whole suite with `PI_TOOLS_SUITE_E2E_MODEL=provider/model`, or use the existing component variables such as `TOOL_SELECTION_E2E_MODEL`, `ASYNC_SUBAGENTS_MODEL`, `ASYNC_SUBAGENTS_ROUTING_E2E_MODEL`, and `DCP_SUMMARY_E2E_MODEL`. The normal deterministic coverage remains `npm test`; run prompt evals after changing tool descriptions, routing/classifier prompts, DCP summary prompts, or the default evaluation model.
|
|
522
695
|
|
|
696
|
+
### Unified eval harness
|
|
697
|
+
|
|
698
|
+
`test/evals/` adds a shared deterministic + live-model eval layer. The coverage
|
|
699
|
+
gate requires every registered extension and model-facing tool to have a
|
|
700
|
+
deterministic contract. The initial live corpus contains 20 cases across tool
|
|
701
|
+
selection, coding quality, orchestration/escalation, and negative overuse
|
|
702
|
+
controls. Coding-quality fixtures use executable behavioral checks rather than
|
|
703
|
+
an LLM judge, while reports compare parent/worker tokens, provider-reported cost,
|
|
704
|
+
tool calls, changed files, and elapsed time.
|
|
705
|
+
|
|
706
|
+
```bash
|
|
707
|
+
# Deterministic coverage/contract gate only
|
|
708
|
+
npm run test:evals:contracts
|
|
709
|
+
|
|
710
|
+
# Live matrix as Bun tests. Models are comma/semicolon separated.
|
|
711
|
+
PI_TOOLS_SUITE_EVAL_MODELS='zai/glm-5.3,openai-codex/gpt-5.6-luna,openai-codex/gpt-5.6-terra,openai-codex/gpt-5.6-sol' \
|
|
712
|
+
npm run test:evals:live
|
|
713
|
+
|
|
714
|
+
# Produce JSON + Markdown comparison artifacts.
|
|
715
|
+
PI_TOOLS_SUITE_EVAL_MODELS='zai/glm-5.3,openai-codex/gpt-5.6-terra,openai-codex/gpt-5.6-sol' \
|
|
716
|
+
npm run evals:report
|
|
717
|
+
|
|
718
|
+
# Focus the report runner when iterating
|
|
719
|
+
PI_TOOLS_SUITE_EVAL_MODELS='zai/glm-5.3' \
|
|
720
|
+
PI_TOOLS_SUITE_EVAL_CATEGORIES='coding-quality,negative' \
|
|
721
|
+
npm run evals:report
|
|
722
|
+
```
|
|
723
|
+
|
|
724
|
+
Live evals are opt-in. The deterministic coverage registry is part of normal
|
|
725
|
+
tests, so adding an extension or tool without eval coverage fails the gate. See
|
|
726
|
+
[`docs/evals.md`](docs/evals.md) for the architecture, complete 20-case catalog,
|
|
727
|
+
fixtures, assertions, metrics, model matrix, report format, environment
|
|
728
|
+
variables, CI recommendations, and the procedure for adding new evals.
|
|
729
|
+
|
|
523
730
|
Supporting docs and historical standalone README content are kept in `docs/`; third-party license texts are kept in `licenses/`.
|
|
524
731
|
|
|
525
732
|
## SDK pin
|