mlx-bun 0.2.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +71 -588
- package/docs/README.md +91 -0
- package/docs/{design → archive/investigations}/audio-input-plan.md +4 -4
- package/docs/{design → archive/investigations}/diffusion-gemma-port.md +3 -3
- package/docs/{design → archive/investigations}/faithful-l1-consolidation.md +4 -4
- package/docs/{design → archive/investigations}/minicpm5-decode-megakernel.md +11 -11
- package/docs/{design → archive/investigations}/optimization_plan.md +1 -1
- package/docs/{design → archive/investigations}/pre-colibri-stabilization.md +5 -5
- package/docs/{design → archive/investigations}/the-dreaming-handoff.md +2 -2
- package/docs/design/batching.md +569 -0
- package/docs/design/decode-speed-program.md +3178 -169
- package/docs/design/dreaming-nightly-pipeline.md +675 -282
- package/docs/design/generic-model-support.md +579 -334
- package/docs/design/kv-cache.md +680 -0
- package/docs/design/orpo-training.md +673 -1239
- package/docs/design/speculative-decoding.md +1627 -0
- package/docs/design/turboquant.md +2102 -0
- package/docs/design/unified-engine-frontier-plan.md +1831 -727
- package/docs/design/web-chat-redesign.md +666 -1083
- package/docs/planning/PRODUCT_ROADMAP.md +34 -1
- package/docs/planning/release-notes-v0.3.0.md +35 -0
- package/docs/planning/video-series.md +2 -2
- package/docs/reference/benchmarks.md +1162 -0
- package/docs/reference/cli.md +394 -159
- package/docs/reference/distribution.md +321 -182
- package/docs/reference/environment.md +348 -0
- package/docs/reference/glossary.md +39 -0
- package/docs/reference/library-api.md +566 -168
- package/docs/reference/memory.md +227 -64
- package/docs/reference/models.md +104 -6
- package/docs/reference/server-api.md +886 -425
- package/docs/reference/server-config.md +638 -330
- package/docs/reference/training.md +393 -254
- package/docs/reference/troubleshooting.md +166 -0
- package/package.json +9 -4
- package/src/audio/decode.ts +2 -2
- package/src/audio/features.ts +2 -2
- package/src/backends/mlx/autoregressive.ts +108 -0
- package/src/backends/mlx/batch-group.ts +1646 -0
- package/src/backends/mlx/diffusion.ts +39 -0
- package/src/backends/mlx/draft-target.ts +47 -0
- package/src/backends/mlx/evaluation.ts +105 -0
- package/src/backends/mlx/fill-append.ts +45 -0
- package/src/backends/mlx/gateway-binding.ts +71 -0
- package/src/backends/mlx/graph.ts +38 -0
- package/src/backends/mlx/memory-client.ts +189 -0
- package/src/backends/mlx/memory-guard.ts +40 -0
- package/src/backends/mlx/methods.ts +90 -0
- package/src/backends/mlx/model-prompt.ts +164 -0
- package/src/backends/mlx/model-serving.ts +79 -0
- package/src/backends/mlx/prefill.ts +33 -0
- package/src/backends/mlx/request-policy.ts +20 -0
- package/src/backends/mlx/rollback.ts +51 -0
- package/src/backends/mlx/serial-executor.ts +273 -0
- package/src/backends/mlx/speculative.ts +114 -0
- package/src/backends/mlx/state-views.ts +26 -0
- package/src/backends/mlx/text-engine.ts +73 -0
- package/src/cli.ts +58 -25
- package/src/client.ts +66 -0
- package/src/config.ts +21 -4
- package/src/contracts/completion.ts +16 -0
- package/src/contracts/execution.ts +49 -0
- package/src/contracts/generation.ts +103 -0
- package/src/contracts/host.ts +7 -0
- package/src/contracts/jobs.ts +66 -0
- package/src/contracts/pi-provider.ts +30 -0
- package/src/contracts/pi.ts +259 -0
- package/src/contracts/resources.ts +18 -0
- package/src/contracts/scheduling.ts +27 -0
- package/src/contracts/tsconfig.json +14 -0
- package/src/diffusion/diffusion-generate.ts +218 -153
- package/src/embed.ts +1 -1
- package/src/engine/admission.ts +59 -0
- package/src/engine/cancellation.ts +38 -0
- package/src/engine/engine.ts +56 -0
- package/src/engine/execution-coordinator.ts +70 -0
- package/src/engine/execution-plan.ts +51 -0
- package/src/engine/resources.ts +42 -0
- package/src/engine/scheduler.ts +67 -0
- package/src/engine/session.ts +247 -0
- package/src/eval/README.md +44 -1
- package/src/eval/kl.ts +236 -3
- package/src/eval/reference-logits.ts +207 -0
- package/src/eval/runner.ts +8 -96
- package/src/fill/echo-index.ts +188 -0
- package/src/fill/fill-session.ts +452 -0
- package/src/fill/proposal.ts +68 -0
- package/src/fill/schema-rows.ts +506 -0
- package/src/fill/tool-boundary.ts +103 -0
- package/src/fit.ts +1 -1
- package/src/generate.ts +656 -268
- package/src/grammar.ts +12 -7
- package/src/harness-pi.ts +14 -33
- package/src/hub-rest.ts +4 -4
- package/src/index.ts +7 -0
- package/src/inference/denoising.ts +17 -0
- package/src/inference/graph.ts +33 -0
- package/src/inference/prefill.ts +39 -0
- package/src/inference/rollback.ts +7 -0
- package/src/jobs/db.ts +4 -0
- package/src/jobs/index.ts +2 -0
- package/src/jobs/job-entry.ts +12 -12
- package/src/jobs/runner.ts +98 -13
- package/src/jobs/task-client.ts +19 -0
- package/src/jobs/types.ts +2 -58
- package/src/kv-store.ts +109 -49
- package/src/lab/README.md +18 -0
- package/src/{assets → lab/curve}/curve-designer.html +2 -2
- package/src/{curve-sampler.ts → lab/curve/curve-sampler.ts} +2 -2
- package/src/{expert-trace.ts → lab/expert-trace/expert-trace.ts} +5 -5
- package/src/{model → lab/paged-kv}/paged-kv.ts +9 -6
- package/src/library.ts +40 -0
- package/src/lora.ts +18 -0
- package/src/media-fetch.ts +1 -1
- package/src/memory/article.ts +1 -1
- package/src/memory/chunk.ts +1 -1
- package/src/memory/model.ts +30 -169
- package/src/memory/prompts.ts +1 -1
- package/src/memory/rest.ts +1 -1
- package/src/memory/schedule.ts +1 -1
- package/src/memory/stages.ts +4 -1
- package/src/memory/vault.ts +1 -3
- package/src/mlx/array.ts +9 -10
- package/src/mlx/compile.ts +12 -2
- package/src/mlx/ffi.ts +58 -11
- package/src/mlx/materialize.ts +23 -0
- package/src/mlx/metal-kernel.ts +10 -11
- package/src/mlx/ops.ts +33 -13
- package/src/mlx/quantized-rows.ts +36 -0
- package/src/mlx/token-bitmask.ts +52 -0
- package/src/mlx/turboquant-kv-decode.ts +82 -0
- package/src/mlx/turboquant-kv-inverse.ts +122 -0
- package/src/mlx/turboquant-ops.ts +33 -3
- package/src/mlx/turboquant-tables.ts +2 -2
- package/src/model/batched-mask.ts +8 -2
- package/src/model/batched-quant.ts +1 -1
- package/src/model/batched-rotating-quant.ts +2 -2
- package/src/model/batched-rotating.ts +1 -1
- package/src/model/compiled-decode.ts +1 -1
- package/src/model/diffusion-gemma.ts +8 -8
- package/src/model/factory.ts +76 -33
- package/src/model/fingerprint.ts +1 -1
- package/src/model/flash-attention.ts +3 -3
- package/src/model/gemma4-base.ts +26 -9
- package/src/model/gemma4.ts +6 -6
- package/src/model/generated/gemma4-12b.ts +2 -2
- package/src/model/generated/gemma4-26b.ts +2 -2
- package/src/model/generated/gemma4-e4b.ts +2 -2
- package/src/model/glm52-dsa.ts +13 -9
- package/src/model/glm52-memory.ts +1 -1
- package/src/model/implementation.ts +69 -0
- package/src/model/minicpm5.ts +6 -8
- package/src/model/profile.ts +12 -2
- package/src/model/qwen-append.ts +12 -0
- package/src/model/qwen-conv.ts +52 -0
- package/src/model/qwen3_5-checkpoint.ts +164 -0
- package/src/model/qwen3_5.ts +140 -56
- package/src/model/support.ts +1 -1
- package/src/model/trellis-balanced-scatter.ts +51 -0
- package/src/model/trellis-linear.ts +672 -0
- package/src/model/trellis-shared-m.ts +151 -0
- package/src/model/trellis-shared-scatter.ts +144 -0
- package/src/model/trellis-splitk-prefill.ts +158 -0
- package/src/model/trellis-tiled-prefill.ts +136 -0
- package/src/model/trellis-vector-expand.ts +62 -0
- package/src/model/trellis-wide-prefill.ts +94 -0
- package/src/model/universal/dense.ts +1 -1
- package/src/model/universal/modules.ts +8 -1
- package/src/native-pack.ts +6 -5
- package/src/pi-provider.ts +6 -64
- package/src/pi-web.ts +8 -315
- package/src/prompt-cache.ts +96 -53
- package/src/quantize/allocator.ts +1 -1
- package/src/quantize/config-writer.ts +10 -3
- package/src/quantize/rotate.ts +4 -4
- package/src/quantize/trellis.ts +586 -0
- package/src/responses.ts +28 -1
- package/src/runtime-config.ts +18 -4
- package/src/sampler.ts +4 -4
- package/src/serve/batch-scheduler.ts +8 -1671
- package/src/serve/chat-request.ts +391 -0
- package/src/serve/chat-stage.ts +195 -0
- package/src/serve/checkpoint-identity.ts +64 -0
- package/src/serve/completion-executor.ts +15 -7
- package/src/serve/discovery-routes.ts +12 -13
- package/src/serve/generation-gateway.ts +154 -193
- package/src/serve/http.ts +159 -0
- package/src/serve/inference-request.ts +79 -0
- package/src/serve/isolate.ts +297 -105
- package/src/serve/lab-routes.ts +3 -0
- package/src/serve/lane-registry.ts +2 -1
- package/src/serve/model-admin-routes.ts +7 -5
- package/src/serve/model-binding.ts +43 -0
- package/src/serve/model-host.ts +512 -0
- package/src/serve/openai-wire.ts +103 -0
- package/src/serve/parent-application.ts +56 -0
- package/src/serve/pipeline.ts +33 -0
- package/src/serve/preparation.ts +33 -0
- package/src/serve/request-plan.ts +19 -15
- package/src/serve/request-prep.ts +346 -0
- package/src/serve/response-lease.ts +24 -0
- package/src/serve/responses-client.ts +69 -0
- package/src/serve/session-completion-engine.ts +110 -0
- package/src/serve/session-search.ts +1 -1
- package/src/serve/static-routes.ts +5 -5
- package/src/serve/text-completion-stage.ts +104 -0
- package/src/serve/token-streams.ts +519 -0
- package/src/server.ts +317 -2649
- package/src/spec/assistant-source.ts +10 -44
- package/src/spec/deepspec-source.ts +6 -6
- package/src/spec/dflash-source.ts +8 -8
- package/src/spec/dspark/ab-stats.ts +3 -3
- package/src/spec/dspark/calibration.ts +2 -2
- package/src/spec/dspark/deepspec-module.ts +2 -2
- package/src/spec/dspark/generate-dflash.ts +2 -2
- package/src/spec/dspark/generate.ts +2 -2
- package/src/spec/dspark/loader.ts +1 -1
- package/src/spec/dspark/module-dflash.ts +4 -3
- package/src/spec/dspark/module.ts +3 -3
- package/src/spec/dspark/quantize-drafter.ts +1 -1
- package/src/spec/generate.ts +1 -1
- package/src/spec/glm52-mtp-source.ts +1 -1
- package/src/spec/ngram-source.ts +1 -1
- package/src/spec/prefix-state.ts +45 -0
- package/src/spec/qwen-mtp-source.ts +166 -69
- package/src/spec/serve-loop.ts +154 -156
- package/src/spec/source.ts +42 -10
- package/src/spec/two-model.ts +1 -1
- package/src/ssd-cache.ts +80 -12
- package/src/ssd-durability.ts +25 -13
- package/src/tool-call.ts +51 -14
- package/src/train/flash-cce.ts +2 -2
- package/src/train/forward.ts +6 -0
- package/src/train/loss.ts +1 -1
- package/src/train/prefix-shared.ts +5 -1
- package/src/train/segmented.ts +5 -3
- package/src/train/trainer.ts +6 -6
- package/src/vision/prompt.ts +1 -1
- package/src/vision/qwen3vl-tower.ts +52 -10
- package/src/web/app.html +1 -1
- package/src/web/app.js +5 -3
- package/src/web/src/adapters-panel.ts +3 -3
- package/src/web/src/assistant.ts +2 -36
- package/src/web/src/chat.ts +4 -4
- package/src/web/src/composer.ts +4 -4
- package/src/web/src/hub.ts +1 -1
- package/src/web/src/markdown.ts +0 -0
- package/src/web/src/memory-panel.ts +1 -1
- package/src/web/src/model-picker.ts +1 -1
- package/src/web/src/palette.ts +1 -1
- package/src/web/src/protocol.ts +3 -39
- package/src/web/src/sessions.ts +1 -1
- package/src/web/src/ui-catalog.ts +4 -18
- package/src/web-assets.ts +47 -0
- package/src/weights.ts +55 -9
- package/docs/archive/investigations/curve-runs/bisector-route.json +0 -98
- package/docs/archive/investigations/curve-runs/bisector-trace.json +0 -1393
- package/docs/archive/investigations/curve-runs/curve-terrain.json +0 -1179
- package/docs/archive/investigations/curve-runs/curve-terrain.svg +0 -42
- package/docs/archive/investigations/hlg-ab.html +0 -34
- package/docs/archive/investigations/hlg-explorer.html +0 -238
- package/docs/archive/investigations/hlg-figs/cliff.svg +0 -1
- package/docs/archive/investigations/hlg-figs/cross.svg +0 -1
- package/docs/archive/investigations/hlg-figs/edges.svg +0 -1
- package/docs/archive/investigations/hlg-figs/k20-grids.svg +0 -1
- package/docs/archive/investigations/hlg-figs/k20-peaks.svg +0 -1
- package/docs/archive/investigations/hlg-figs/lw-curve.svg +0 -1
- package/docs/archive/investigations/hlg-figs/terrain.svg +0 -1
- package/docs/archive/investigations/hlg-report.html +0 -262
- package/docs/archive/investigations/hlg-runs/hlg-ab.json +0 -149
- package/docs/archive/investigations/hlg-runs/hlg-blind-prompts.md +0 -87
- package/docs/archive/investigations/hlg-runs/hlg-blind.json +0 -107
- package/docs/archive/investigations/hlg-runs/hlg-cliff.json +0 -197
- package/docs/archive/investigations/hlg-runs/hlg-cross.json +0 -505
- package/docs/archive/investigations/hlg-runs/hlg-edge.json +0 -388
- package/docs/archive/investigations/hlg-runs/hlg-eval1.json +0 -25
- package/docs/archive/investigations/hlg-runs/hlg-frontier.json +0 -86
- package/docs/archive/investigations/hlg-runs/hlg-grid.json +0 -242
- package/docs/archive/investigations/hlg-runs/hlg-grid2.json +0 -343
- package/docs/archive/investigations/hlg-runs/hlg-lw-coarse.json +0 -110
- package/docs/archive/investigations/hlg-runs/hlg-map.json +0 -383
- package/docs/archive/investigations/hlg-runs/run0-original.log +0 -159
- package/docs/archive/investigations/hlg-runs/runA-recommended.log +0 -131
- package/docs/archive/investigations/hlg-runs/runAA-cliff-cpm5.log +0 -26
- package/docs/archive/investigations/hlg-runs/runAB-grid2-e4b.log +0 -18
- package/docs/archive/investigations/hlg-runs/runAC-peak-e4b-partial.log +0 -25
- package/docs/archive/investigations/hlg-runs/runAD-ab-samples.log +0 -8
- package/docs/archive/investigations/hlg-runs/runAE-edge-e4b.log +0 -51
- package/docs/archive/investigations/hlg-runs/runAF-logit-calibration.log +0 -10
- package/docs/archive/investigations/hlg-runs/runAG-eval-config2.log +0 -16
- package/docs/archive/investigations/hlg-runs/runB-gain-variants.log +0 -192
- package/docs/archive/investigations/hlg-runs/runC-pivots.log +0 -162
- package/docs/archive/investigations/hlg-runs/runD-oetf.log +0 -132
- package/docs/archive/investigations/hlg-runs/runE-eotf.log +0 -162
- package/docs/archive/investigations/hlg-runs/runF-shaper-working.log +0 -86
- package/docs/archive/investigations/hlg-runs/runG-shaper-sweep.log +0 -101
- package/docs/archive/investigations/hlg-runs/runH-wsweep.log +0 -176
- package/docs/archive/investigations/hlg-runs/runI-asweep.log +0 -176
- package/docs/archive/investigations/hlg-runs/runJ-knobmap.log +0 -49
- package/docs/archive/investigations/hlg-runs/runK-frontier.log +0 -19
- package/docs/archive/investigations/hlg-runs/runL-widemap.log +0 -65
- package/docs/archive/investigations/hlg-runs/runM-frontier-emb.log +0 -15
- package/docs/archive/investigations/hlg-runs/runN-verify.log +0 -116
- package/docs/archive/investigations/hlg-runs/runO-frontier-seed5000.log +0 -15
- package/docs/archive/investigations/hlg-runs/runP-grid-AxSm.log +0 -15
- package/docs/archive/investigations/hlg-runs/runQ-grid-tgW.log +0 -15
- package/docs/archive/investigations/hlg-runs/runR-blind-transcript.log +0 -79
- package/docs/archive/investigations/hlg-runs/runS-cross-coherence.log +0 -33
- package/docs/archive/investigations/hlg-runs/runT-cross-cpm5.log +0 -33
- package/docs/archive/investigations/hlg-runs/runU-cross-12b.log +0 -33
- package/docs/archive/investigations/hlg-runs/runV-lw-e4b.log +0 -22
- package/docs/archive/investigations/hlg-runs/runW-lw-12b.log +0 -22
- package/docs/archive/investigations/hlg-runs/runX-lw-cpm5.log +0 -22
- package/docs/archive/investigations/hlg-runs/runY-cliff-e4b.log +0 -26
- package/docs/archive/investigations/hlg-runs/runZ-cliff-12b.log +0 -26
- package/docs/archive/mlx-bun-lab-report.html +0 -275
- package/docs/archive/wwdc-mlx-bun.html +0 -848
- package/docs/design/adapters-end-to-end.md +0 -205
- package/docs/design/batching-perf-path.md +0 -255
- package/docs/design/batching-v2-plan.md +0 -529
- package/docs/design/bucketing-stage.md +0 -197
- package/docs/design/compat-cli-surface-design.md +0 -230
- package/docs/design/docs-reorg-plan.md +0 -165
- package/docs/design/dspark-serving-program.md +0 -371
- package/docs/design/dspark-speculative-decoding.md +0 -318
- package/docs/design/grammar-spec-batching-integration.md +0 -301
- package/docs/design/memory-inference-path.md +0 -422
- package/docs/design/memory-synthesis.md +0 -287
- package/docs/design/memory-system.md +0 -237
- package/docs/design/mlx-lm-serving-execution-seam.md +0 -81
- package/docs/design/mlx-lm-tool-parity-plan.md +0 -545
- package/docs/design/omlx-adoption-map.md +0 -98
- package/docs/design/orpo-dynamic-lambda.md +0 -220
- package/docs/design/orpo-future-enhancements.md +0 -340
- package/docs/design/paged-kv-cache.md +0 -214
- package/docs/design/parallel-slots.md +0 -567
- package/docs/design/parity-tier-dag.md +0 -161
- package/docs/design/repo-cleanup-plan.md +0 -231
- package/docs/design/runtime-isolation.md +0 -263
- package/docs/design/segmented-backward-training.md +0 -543
- package/docs/design/spec-decode-larger-targets.md +0 -229
- package/docs/design/ssd-kv-cold-tier.md +0 -415
- package/docs/design/structured-output.md +0 -484
- package/docs/design/tauri-desktop-app.md +0 -127
- package/docs/design/the-dreaming-master-plan.md +0 -445
- package/docs/design/turboquant-kv.md +0 -235
- package/docs/design/turboquant-weights.md +0 -467
- package/docs/design/w4a16-compute-precision-spike.md +0 -232
- package/docs/design/web-chat-beat-matrix.md +0 -315
- package/docs/design/write-pipeline-entity.md +0 -468
- package/docs/planning/IDEAS.md +0 -69
- package/docs/reference/embedding.md +0 -197
- package/docs/reference/features-matrix.md +0 -101
- package/docs/reference/orpo-quickstart.md +0 -135
- package/src/flags.ts +0 -11
- /package/docs/{design → archive/investigations}/colibri-glm52-port.md +0 -0
- /package/docs/{planning → archive/planning}/chunk-task-roadmap.md +0 -0
- /package/docs/{planning → archive/planning}/curve-sampler-research-plan.md +0 -0
- /package/docs/{planning → archive/planning}/memory-docs-and-dag-plan.md +0 -0
- /package/docs/{planning → archive/planning}/pi-integration-review.md +0 -0
- /package/docs/{planning → archive/planning}/release-notes-v0.2.0.md +0 -0
- /package/src/{memory-plan.ts → admission-plan.ts} +0 -0
- /package/{docs/archive/investigations → src/serve/assets}/curve-terrain.html +0 -0
- /package/{docs/dag → src/serve/assets}/training-inference-map.html +0 -0
|
@@ -1,1112 +1,695 @@
|
|
|
1
|
-
|
|
2
|
-
|
|
3
|
-
|
|
1
|
+
---
|
|
2
|
+
status: active
|
|
3
|
+
axis: USING
|
|
4
|
+
canonical-for: web-chat
|
|
5
|
+
plan-anchor: none
|
|
6
|
+
last-verified: 2026-08-23
|
|
7
|
+
---
|
|
4
8
|
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
`
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
9
|
+
# Web chat — the built-in chat surface (canonical design)
|
|
10
|
+
|
|
11
|
+
Scope: `src/web/` (app.html + `src/web/src/*.ts` built into `app.js`),
|
|
12
|
+
`src/pi-web.ts` (the WebSocket bridge), the thin REST wrappers the UI calls
|
|
13
|
+
(`src/memory/rest.ts`, `src/hub-rest.ts`, `src/serve/session-search.ts`,
|
|
14
|
+
`src/serve/model-admin-routes.ts`), `src/tool-approvals.ts`.
|
|
15
|
+
|
|
16
|
+
This is the one canonical doc for the web chat. It absorbs the former
|
|
17
|
+
beat matrix (`docs/design/web-chat-redesign.md`, kept as Appendix A), the web half
|
|
18
|
+
of `docs/design/web-chat-redesign.md` (§3.6 here), and the desktop-shell idea (one
|
|
19
|
+
History line). Status/changelog prose belongs in PLAN.md; this file holds
|
|
20
|
+
the design, the as-built contract, and the open items.
|
|
21
|
+
|
|
22
|
+
**Standing decision (2026-06-14): the chat UI is a Pi embed, not a
|
|
23
|
+
hand-rolled chat.** The browser talks to an in-process
|
|
24
|
+
`@earendil-works/pi-coding-agent` `AgentSession` over `/ws/chat`
|
|
25
|
+
(`src/pi-web.ts`). Pi owns the agent loop, tool execution, session files
|
|
26
|
+
(new/list/open/fork/delete, `SessionManager.forkFrom`), the tree
|
|
27
|
+
navigation behind edit/regenerate (`AgentSession.navigateTree`),
|
|
28
|
+
compaction, thinking levels, and the steering/follow-up queue. Nothing in
|
|
29
|
+
this design rebuilds those; the web work is (a) frames that expose them,
|
|
30
|
+
(b) UI that makes them legible, (c) the few capabilities Pi has no
|
|
31
|
+
concept of (memory vault, LoRA adapters, serving lanes, hardware fit,
|
|
32
|
+
the app-aware assistant).
|
|
12
33
|
|
|
13
34
|
---
|
|
14
35
|
|
|
15
|
-
## 1.
|
|
16
|
-
|
|
17
|
-
**
|
|
18
|
-
|
|
19
|
-
better at knowing *you
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
(
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
**
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
every
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
always-on proof — tok/s, TTFT, which lane is active (serial / spec-decode /
|
|
57
|
-
batched), and exactly what a tool call touched on disk — because we control
|
|
58
|
-
the whole stack from Metal kernel to composer pixel and nobody else
|
|
59
|
-
surveyed does. The differentiators are the reason to *switch*; they are
|
|
60
|
-
never an excuse for a missing table stake.
|
|
61
|
-
|
|
62
|
-
**The pragmatic constraint:** almost none of this requires new backend
|
|
63
|
-
capability. Session fork, tool approval, adapter-scoped KV, the memory tool
|
|
64
|
-
suite, structured output, the sampler extension set — all of it exists
|
|
65
|
-
server-side today with zero or thin UI. The fastest, highest-leverage path
|
|
66
|
-
is wiring existing capability to new UI, not a rewrite. Where a rewrite *is*
|
|
67
|
-
warranted (the streaming-render architecture, message-tree branching,
|
|
68
|
-
virtualized long threads), the plan says so explicitly and scopes it as its
|
|
69
|
-
own phase.
|
|
36
|
+
## 1. Thesis and principles
|
|
37
|
+
|
|
38
|
+
**Thesis.** mlx-bun's web chat should not try to be a better ChatGPT. It
|
|
39
|
+
is the local chat surface where the assistant visibly, provably gets
|
|
40
|
+
better at knowing *you* (a git-tracked memory vault you can open and
|
|
41
|
+
diff), where switching to a domain-specific brain is a routing decision
|
|
42
|
+
(LoRA adapters, stackable), and where "local, private, fast" is shown on
|
|
43
|
+
screen (lane badge, tok/s, TTFT, provenance chips) instead of asserted.
|
|
44
|
+
|
|
45
|
+
**Doctrine (revised 2026-07-06): superset, not wedge.** Concede nothing a
|
|
46
|
+
daily LM Studio / Open WebUI / optiq-Lab user would miss — every
|
|
47
|
+
capability in Appendix A is MATCH, BEAT, or a written SKIP with a
|
|
48
|
+
substrate reason — and stack the differentiators on top. The
|
|
49
|
+
differentiators are the reason to switch, never an excuse for a missing
|
|
50
|
+
table stake.
|
|
51
|
+
|
|
52
|
+
**Principles.**
|
|
53
|
+
|
|
54
|
+
1. **Wire before you invent.** The server already had most of the
|
|
55
|
+
capability (sampler set, structured output, adapters, memory tools,
|
|
56
|
+
fork, approval gate); the highest-leverage work was wiring, and the
|
|
57
|
+
remaining open items are still mostly wiring.
|
|
58
|
+
2. **Show, don't assert.** Privacy, speed, memory each get a UI proof.
|
|
59
|
+
3. **Progressive disclosure, one flag.** Chat is the whole app for a
|
|
60
|
+
first-time user; Quantize / Fine-tune / Build Dataset / Status /
|
|
61
|
+
Curves / Routes sit behind one persisted Developer toggle.
|
|
62
|
+
4. **Quiet retrieval, legible provenance.** Memory tools fire silently;
|
|
63
|
+
every consulted article gets a small expandable chip.
|
|
64
|
+
5. **Speed is ambient.** tok/s, TTFT, context-fill, and the serving lane
|
|
65
|
+
live next to the composer, server-reported, never client-inferred.
|
|
66
|
+
6. **Local-only is an honest simplification.** Single user, own machine:
|
|
67
|
+
"runs on your machine, with your files" plus a per-call approval gate,
|
|
68
|
+
not multi-tenant sandbox theater.
|
|
69
|
+
7. **Keyboard-first, mouse-complete.** Every primary action has a key
|
|
70
|
+
path and a visible one.
|
|
71
|
+
8. **Never hijack.** No spotlight, panel, or approval ever traps focus or
|
|
72
|
+
blocks the screen; the user is never in a state they have to escape.
|
|
73
|
+
9. **The bar for done is visual polish** at 375 / 768 / 1024 px and
|
|
74
|
+
desktop, 200 % zoom, `prefers-reduced-motion` honored, no layout shift
|
|
75
|
+
while streaming. Visual QA at phase boundaries has the standing of
|
|
76
|
+
tests.
|
|
70
77
|
|
|
71
78
|
---
|
|
72
79
|
|
|
73
|
-
## 2.
|
|
74
|
-
|
|
75
|
-
### 2.1
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
error,
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
### 2.
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
- **Lane badge**: the serving lanes (serial / spec-decode / batched) exist
|
|
194
|
-
internally (`GenerationGateway` routing, `src/server.ts:1162-1168`) but
|
|
195
|
-
**no field on any usage payload or WS frame reports which lane served a
|
|
196
|
-
turn**. The perf strip's tok/s / TTFT / context-fill are client-computed
|
|
197
|
-
already; the lane badge needs a small per-turn `lane` field added
|
|
198
|
-
server-side (usage payload + the pi-web `turn_end`/usage frame) first.
|
|
199
|
-
- **Memory panel data**: `memory_list`/`memory_search`/`memory_links` are
|
|
200
|
-
**agent tools** (model-callable, `src/memory/tools.ts`) and `vaultStatus()`
|
|
201
|
-
is a library function — none have HTTP routes. The vault browser (§5.5)
|
|
202
|
-
needs thin REST wrappers (e.g. `GET /api/memory/status|list|search|article`
|
|
203
|
-
+ a git log/diff read) added in Phase 2.
|
|
80
|
+
## 2. Architecture as built
|
|
81
|
+
|
|
82
|
+
### 2.1 The bridge: `src/pi-web.ts`
|
|
83
|
+
|
|
84
|
+
One `PiWebSession` per WebSocket connection wraps one Pi `AgentSession`.
|
|
85
|
+
Pi events map to frames through the pure, unit-tested `mapEventToFrames`
|
|
86
|
+
(`tests/pi-web.test.ts`). The frame types are exported **type-only** and
|
|
87
|
+
imported by the frontend (`src/web/src/chat.ts`), so contract drift
|
|
88
|
+
between browser and server is a compile error, not a runtime bug.
|
|
89
|
+
|
|
90
|
+
Client → server (`ClientMessage`):
|
|
91
|
+
|
|
92
|
+
| Frame | Purpose |
|
|
93
|
+
|---|---|
|
|
94
|
+
| `prompt` (text, images?) | send a turn; images ride as attachments |
|
|
95
|
+
| `abort` | stop the turn (`session.abort()`; verified to abort in-flight tool subprocesses) |
|
|
96
|
+
| `approval` (callId, decision, editedArgs?, alwaysAllow?) | answer a gated tool call; edited args are substituted in place; `alwaysAllow` persists to `src/tool-approvals.ts` |
|
|
97
|
+
| `set_thinking` | reasoning channel on/off (Pi thinking level medium ↔ off) |
|
|
98
|
+
| `set_coding_tools` | opt-in "let it touch files" (default OFF); takes effect on the *next* session because Pi bakes the tool list at session creation |
|
|
99
|
+
| `set_adapter` (id \| null) | active LoRA adapter for subsequent turns (`"a+b"` stacks) |
|
|
100
|
+
| `set_sampling` (scope session \| next_turn, temperature … seed) | full sampler set; `next_turn` is a one-shot override composed over the session-level one |
|
|
101
|
+
| `set_system_prompt` | per-session user prompt layered over the built-in surface prompt (applied per turn via `before_agent_start`) |
|
|
102
|
+
| `new_session` / `list_sessions` / `open_session` / `fork_session` / `delete_session` | Pi session lifecycle |
|
|
103
|
+
| `regenerate` / `edit_resend` / `switch_sibling` | message actions; all are sibling branches in the same session file via `navigateTree` |
|
|
104
|
+
| `context` (AppUiContext) | app-aware assistant: route/view/step + `uiSnapshot`, pushed on route and wizard-step change |
|
|
105
|
+
|
|
106
|
+
Server → client (`ServerMessage`):
|
|
107
|
+
|
|
108
|
+
| Frame | Purpose |
|
|
109
|
+
|---|---|
|
|
110
|
+
| `ready` (model, vision, audio, thinking, genDefaults) | capability negotiation; drives image attach, thinking toggle, sampling defaults |
|
|
111
|
+
| `turn_start` / `text_delta` / `thinking_delta` / `turn_end` (lane?) | streaming; `lane` is `"serial" \| "serial+spec" \| "batched"` from `src/serve/lane-registry.ts`, keyed by the assistant message's `responseId`, absent when unknown — never guessed |
|
|
112
|
+
| `tool_start` / `tool_update` / `tool_end` / `tool_approval_request` | tool cards and the approval card |
|
|
113
|
+
| `queue_update` (steering[], followUp[]) | live steering vs queued follow-ups |
|
|
114
|
+
| `history` / `sessions` / `siblings` | transcript replay, sidebar, `< i/n >` toggle data |
|
|
115
|
+
| `context` (tokens, contextWindow, percent) | context-window meter (same name as the client frame, opposite direction) |
|
|
116
|
+
| `coding_tools` (active, pending) | honest current-vs-pending state of the opt-in |
|
|
117
|
+
| `tool_approvals` (alwaysAllow[]) | durable always-allow set, on `ready` and after change |
|
|
118
|
+
| `ui_navigate` / `ui_spotlight` | app-aware assistant side channel, validated server-side |
|
|
119
|
+
| `error` | surfaced in the thread |
|
|
120
|
+
|
|
121
|
+
Injection hooks (all Pi extension hooks, no custom provider plumbing):
|
|
122
|
+
`before_provider_request` injects `adapter` (`injectAdapter`) and sampling
|
|
123
|
+
(`injectSampling` / `composeSampling`); `before_agent_start` injects the
|
|
124
|
+
user system prompt and the ambient app-context line; `tool_call` /
|
|
125
|
+
`tool_result` carry the approval gate and loop hygiene.
|
|
126
|
+
|
|
127
|
+
### 2.2 The frontend: `src/web/`
|
|
128
|
+
|
|
129
|
+
Vanilla TypeScript, no framework. `scripts/build-web.ts` bundles
|
|
130
|
+
`src/web/src/main.ts` into `src/web/app.js`, which `src/server.ts` serves
|
|
131
|
+
at `/assets/app.js`; `app.html` holds the markup and CSS. Runtime stays
|
|
132
|
+
self-contained (no CDN); the only vendored asset is highlight.js
|
|
133
|
+
(`src/web/vendor/`). The bundler errors if a value import ever pulls
|
|
134
|
+
server code into the browser bundle (type-only imports from `pi-web.ts`
|
|
135
|
+
are the intended seam).
|
|
136
|
+
|
|
137
|
+
| Module | Owns |
|
|
138
|
+
|---|---|
|
|
139
|
+
| `shell.ts` | DOM helpers, toast, focus trap, theme (auto/dark/light), shortcut sheet (Cmd/Ctrl+/), mobile drawer, hash router, connection pill + model identity poll, Developer toggle (`DEV_TABS`), `/dag` probe for the Routes tab |
|
|
140
|
+
| `chat.ts` | the WebSocket, thread render, streaming, tool cards, approval card, queue bar, Sources panel, memory-chip dispatch, app-context push |
|
|
141
|
+
| `composer.ts` | attachments, `#` mention picker, sampling popover (full set + Advanced + scope), system prompt + presets v1 (localStorage), adapter quick-select, perf strip (tok/s, TTFT, context, lane) |
|
|
142
|
+
| `sessions.ts` | sidebar list/search/fork/delete/export, full-text "in messages" search, sibling toggle, per-message actions (copy / regenerate / edit) |
|
|
143
|
+
| `markdown.ts` | `mdToHtml`, block-memoized `renderBlocksIncremental`, highlight pass, `[n]` citation linkify, Canvas toggle for html/svg fences |
|
|
144
|
+
| `memory-panel.ts` | Memory panel, provenance chips, personalized hero chips + first-run consent card |
|
|
145
|
+
| `adapters-panel.ts` | adapter routing table (three states, RAM cost, stacking) |
|
|
146
|
+
| `model-picker.ts` | `#nav-model` popover over `/library` with per-row `/fit` verdict |
|
|
147
|
+
| `hub.ts` | Model Hub panel (`/api/hub/*`), download progress via `/downloads` |
|
|
148
|
+
| `rag.ts` | client-side BM25 over attached-file text |
|
|
149
|
+
| `palette.ts` | command palette (Cmd/Ctrl+K), built via createElement |
|
|
150
|
+
| `assistant.ts` + `ui-catalog.ts` | `captureUiSnapshot`, spotlight resolution/overlay, route + spotlight catalog |
|
|
151
|
+
| `quantize.ts` / `finetune.ts` / `dataset.ts` / `status.ts` | the Developer tabs (job SSE via `api.ts` `jobStream`) |
|
|
152
|
+
| `protocol.ts` / `api.ts` | frontend-only shapes (job stream) and the JSON/SSE helpers |
|
|
153
|
+
|
|
154
|
+
`sw.js` + `manifest.webmanifest`: shell-only cache-first service worker
|
|
155
|
+
for installability and instant shell — explicitly not offline chat.
|
|
156
|
+
|
|
157
|
+
### 2.3 Tool surface and trust model
|
|
158
|
+
|
|
159
|
+
`webChatToolAllowlist(memoryEnabled, codingTools)` in `pi-web.ts`:
|
|
160
|
+
|
|
161
|
+
- Always: `read`, `web_search` (`WELCOME_TOOLS`) and the three app-aware
|
|
162
|
+
tools (`get_current_app_context`, `navigate_app`, `spotlight_ui`).
|
|
163
|
+
- With a vault: the memory and reference read tools (`src/memory/tools.ts`).
|
|
164
|
+
- Opt-in only: `grep`, `find`, `ls`, and the gated `bash` / `edit` /
|
|
165
|
+
`write`.
|
|
166
|
+
|
|
167
|
+
Everything except the gated three is `READ_ONLY_TOOLS` (auto-allowed).
|
|
168
|
+
Gated calls open the approval card: argument preview, editable arguments,
|
|
169
|
+
allow / deny / always-allow, auto-deny after `APPROVAL_TIMEOUT_MS`.
|
|
170
|
+
Always-allow is durable and keyed by tool name (`src/tool-approvals.ts`).
|
|
171
|
+
Loop hygiene (`LOOP_HYGIENE`): consecutive-identical-call blocking (with
|
|
172
|
+
the prior result inlined, since Pi's `tool_call` hook can only block with
|
|
173
|
+
an error result), a failed-retry budget nudge, and a turn cap with
|
|
174
|
+
force-finish. Format-aware tool-call repair lives in `src/tool-call.ts`
|
|
175
|
+
(`RepairKind`), tagged on the parsed call.
|
|
176
|
+
|
|
177
|
+
### 2.4 REST the UI depends on
|
|
178
|
+
|
|
179
|
+
`GET /v1/adapters` (mounted; `ram_bytes`), `GET /v1/adapters/available`
|
|
180
|
+
(on disk; `compatible`, `mounted`), `POST` / `DELETE /v1/adapters`;
|
|
181
|
+
`GET /fit`, `/library`, `/downloads`, `/stats`, `/v1/models`;
|
|
182
|
+
`/api/memory/{status,list,search,article,links,history,diff}` +
|
|
183
|
+
`POST /api/memory/init` (`src/memory/rest.ts`); `/api/hub/{local,search}`,
|
|
184
|
+
`POST /api/hub/download`, `POST /api/hub/serve` (`src/hub-rest.ts`);
|
|
185
|
+
`/api/sessions/{search,export}`; `/api/gc/{plan,execute}`;
|
|
186
|
+
`/api/jobs/:id/stream`. User-facing contract: `docs/reference/server-api.md`.
|
|
187
|
+
|
|
188
|
+
### 2.5 Regression harness
|
|
189
|
+
|
|
190
|
+
`tests/web-app.test.ts` (happy-dom) covers the pure client functions:
|
|
191
|
+
streamed-then-finalized output must equal one-shot `mdToHtml` (the
|
|
192
|
+
renderer's own parity oracle), Canvas fence detection and view-state
|
|
193
|
+
survival, `renderQueue` length semantics, `api()` envelope unwrapping,
|
|
194
|
+
`esc()` discipline on every interpolation site, adapter/model/hub row
|
|
195
|
+
renderers, mention picker, BM25, Sources panel, snapshot capture and
|
|
196
|
+
spotlight resolution, catalog validation. `tests/pi-web.test.ts` covers
|
|
197
|
+
the frame mapping, sampling composition, loop hygiene, and app-aware
|
|
198
|
+
tools. Frontend modules type-check in the same `tsc` pass as the server.
|
|
204
199
|
|
|
205
200
|
---
|
|
206
201
|
|
|
207
|
-
## 3.
|
|
208
|
-
|
|
209
|
-
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
|
|
229
|
-
|
|
230
|
-
|
|
231
|
-
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
|
|
245
|
-
|
|
246
|
-
|
|
247
|
-
|
|
248
|
-
|
|
249
|
-
|
|
250
|
-
|
|
251
|
-
|
|
252
|
-
|
|
253
|
-
|
|
254
|
-
|
|
202
|
+
## 3. Surfaces
|
|
203
|
+
|
|
204
|
+
### 3.1 Onboarding / hero
|
|
205
|
+
|
|
206
|
+
Server-driven chips instead of static marketing copy: one drawn from real
|
|
207
|
+
vault content when `/api/memory/status` reports articles, one offering an
|
|
208
|
+
available adapter, and a first-run, skippable memory-consent card that
|
|
209
|
+
creates the vault via `POST /api/memory/init` — consent happens in the
|
|
210
|
+
same first minute as the first chat, not as a CLI step discovered later.
|
|
211
|
+
The system prompt (`buildWebChatSystemPrompt`) is deliberately short: a
|
|
212
|
+
long welcome prompt made the ~1B starter model greet instead of answer.
|
|
213
|
+
|
|
214
|
+
### 3.2 Composer and message actions
|
|
215
|
+
|
|
216
|
+
- Sampling popover: temperature / top_p / top_k plus an Advanced
|
|
217
|
+
disclosure with `min_p`, XTC, repetition / presence / frequency
|
|
218
|
+
penalties with context size, and seed; defaults from `ready.genDefaults`
|
|
219
|
+
(per-model, not a hardcoded shape); scope control session vs next turn.
|
|
220
|
+
- System prompt field + presets v1 (name + prompt + sampling; localStorage
|
|
221
|
+
only, no wire shape).
|
|
222
|
+
- Adapter quick-select (`#chat-adapter`) with a gear to the routing table
|
|
223
|
+
(§3.6). A swap note says so: a new adapter starts a fresh KV segment.
|
|
224
|
+
- `#` mention picker: attached files and vault articles as one retrieval
|
|
225
|
+
gesture.
|
|
226
|
+
- Perf strip: tok/s, TTFT, context-fill, lane badge (from `turn_end.lane`).
|
|
227
|
+
- Per assistant turn: copy, regenerate; per user turn: edit-and-resend
|
|
228
|
+
with a `< i/n >` sibling toggle. All three are branches of the same Pi
|
|
229
|
+
session (no new file), so history stays in one place. A full tree
|
|
230
|
+
*view* is deliberately not built (open, §5).
|
|
231
|
+
|
|
232
|
+
### 3.3 Streaming render
|
|
233
|
+
|
|
234
|
+
`renderBlocksIncremental` splits the message at block boundaries, leaves
|
|
235
|
+
settled blocks alone, and re-parses only the live tail, throttled to
|
|
236
|
+
animation frames; `finishStreaming` runs one full `mdToHtml` pass so the
|
|
237
|
+
final DOM is byte-identical to a one-shot render (tested). Syntax
|
|
238
|
+
highlighting (vendored highlight.js) applies to completed blocks only.
|
|
239
|
+
Steering and follow-up queues render from `queue_update` with length
|
|
240
|
+
checks (the array-truthiness bug is fixed and regression-tested).
|
|
241
|
+
|
|
242
|
+
### 3.4 Tool cards and approval
|
|
243
|
+
|
|
244
|
+
Generic tool calls keep the running / ok / fail card. Memory and reference
|
|
245
|
+
tool calls render as a **provenance chip** (citation style, expandable,
|
|
246
|
+
opens the article in the Memory panel) — the "assistant looked something
|
|
247
|
+
up about you" moment reads differently from "ran a shell command". The
|
|
248
|
+
approval card (§2.3) is inline and non-blocking. The settings copy says
|
|
249
|
+
that toggling coding tools applies to the next new chat (`coding_tools`
|
|
250
|
+
frame), never implying immediacy Pi can't deliver.
|
|
251
|
+
|
|
252
|
+
### 3.5 Memory panel
|
|
253
|
+
|
|
254
|
+
Reachable from the chat sidebar, not a nav tab (memory is chat-adjacent,
|
|
255
|
+
it is the product, not a developer tool). Two screens: list (status strip
|
|
256
|
+
from `vaultStatus()`, search, personal articles and a `Reference/`
|
|
257
|
+
sub-list) and article (rendered content, inbound/outbound wikilinks line,
|
|
258
|
+
a History toggle backed by `/api/memory/history` and `/api/memory/diff` —
|
|
259
|
+
the "watch it self-heal after a correction" moment made literal, which
|
|
260
|
+
cloud memory UX structurally cannot offer). The agent-tool surface stays
|
|
261
|
+
read-only; only nightly synthesis writes (`docs/design/dreaming-nightly-pipeline.md`,
|
|
262
|
+
`docs/design/dreaming-nightly-pipeline.md`). The plan's wikilink *graph* view is not built —
|
|
263
|
+
the panel shows a links line (open, §5).
|
|
264
|
+
|
|
265
|
+
### 3.6 Model and adapter switching
|
|
266
|
+
|
|
267
|
+
**Model picker.** `#nav-model` opens a popover over `/library`, each row
|
|
268
|
+
with the `/fit` verdict computed for this Mac (predicted decode tok/s,
|
|
269
|
+
safe context). There is **no in-process model swap**: the runtime is one
|
|
270
|
+
model per process (like mlx-lm), `POST /api/hub/serve` answers
|
|
271
|
+
`restart_required` with the exact `mlx-bun serve` command, and the picker
|
|
272
|
+
offers that command to copy. The live-swap mechanism that exists
|
|
273
|
+
(`ModelPool` spawn-overlap) lives only on the opt-in `--isolate` proxy,
|
|
274
|
+
which 501s `/ws/chat` — see `docs/reference/server-config.md`. Model Hub (`hub.ts`)
|
|
275
|
+
adds browse / HF search / download with progress; it never auto-downloads
|
|
276
|
+
or auto-serves.
|
|
277
|
+
|
|
278
|
+
**Adapters — the three-state model** (folded from
|
|
279
|
+
`docs/design/web-chat-redesign.md`; the design keeps the states separate and lets
|
|
280
|
+
the user move an adapter between them explicitly):
|
|
281
|
+
|
|
282
|
+
1. **Available** — on disk, compatible or not, costs nothing.
|
|
283
|
+
`GET /v1/adapters/available` scans `~/.cache/mlx-bun-finetunes/` and
|
|
284
|
+
`~/.cache/mlx-bun/adapters/` (`listAvailableAdapters`, `src/lora.ts`);
|
|
285
|
+
`id` = directory basename; `rank` / `scale` / `base_model` read from
|
|
286
|
+
`optiq_lora_config.json` or `adapter_config.json`, with the base repo id
|
|
287
|
+
recovered from the stored snapshot path.
|
|
288
|
+
2. **Loaded / mounted** — `POST /v1/adapters {id, path}` into the running
|
|
289
|
+
server; costs memory (`ram_bytes` on `GET /v1/adapters`);
|
|
290
|
+
`DELETE /v1/adapters/:id` unmounts.
|
|
291
|
+
3. **Selected for this request** — the `adapter` field: `"none"` (default)
|
|
292
|
+
| `"<id>"` | `"a+b"` (stacked, `resolveSpec`). Unknown id throws —
|
|
293
|
+
loud, never a silent base fallback.
|
|
294
|
+
|
|
295
|
+
The routing table (`adapters-panel.ts`, behind the composer gear) shows
|
|
296
|
+
every on-disk adapter with badges for all three states, base-model / rank
|
|
297
|
+
/ size / RAM meta, mount / unmount / select / clear actions, and "stack"
|
|
298
|
+
tick-boxes that compose two mounted adapters as `"a+b"`. Selecting an
|
|
299
|
+
unmounted adapter mounts it first, then sends `set_adapter`; the
|
|
300
|
+
`before_provider_request` hook injects the id. Invariants: default none;
|
|
301
|
+
an adapter must be mounted before it can be selected; the quick-select and
|
|
302
|
+
the table are two views of one selection.
|
|
303
|
+
|
|
304
|
+
Compatibility as built: `compatible = base_model == null ||
|
|
305
|
+
bareName(base_model) === bareName(servedModelId)` — a bare-name match on
|
|
306
|
+
the base model, not the repo-id match the original note described.
|
|
307
|
+
|
|
308
|
+
The **CLI twin** is the Pi extension
|
|
309
|
+
`scripts/packaging/pi-extensions/mlx-bun-adapter.ts` (`/adapter`,
|
|
310
|
+
`/adapter <id>`, `/adapter off`; server via `MLX_BUN_URL`; same
|
|
311
|
+
`before_provider_request` mechanism, in-memory selection). It belongs to
|
|
312
|
+
the CLI reference (`docs/reference/cli.md`, alongside `serve --adapter`),
|
|
313
|
+
not to this doc.
|
|
314
|
+
|
|
315
|
+
### 3.7 Sessions
|
|
316
|
+
|
|
317
|
+
Sidebar with title filter, a full-text "in messages" fallback
|
|
318
|
+
(`/api/sessions/search`, confined to the session dir), fork / delete /
|
|
319
|
+
export (Markdown / JSON via `/api/sessions/export`). Titles derive from
|
|
320
|
+
the session name or first user message (`toSessionListItems`). On narrow
|
|
321
|
+
viewports the sidebar becomes a focus-trapped slide-over drawer.
|
|
322
|
+
|
|
323
|
+
### 3.8 Information architecture and settings
|
|
324
|
+
|
|
325
|
+
Chat is the product; everything else is a developer tool. One persisted
|
|
326
|
+
Developer toggle (`shell.ts`) reveals Quantize / Fine-tune / Build Dataset
|
|
327
|
+
/ Status / Curves / Routes. Curves is a separate full page (`/curves`,
|
|
328
|
+
labelled as such); Routes is a hash route whose tab visibility is gated by
|
|
329
|
+
a `/dag` probe so the compiled binary never shows a broken iframe. Deep
|
|
330
|
+
links keep working regardless of the toggle. Status's RAM/KV "Memory"
|
|
331
|
+
card keeps meaning system memory; the vault is always "Memory panel".
|
|
332
|
+
Settings group as Chat (sampling, thinking, coding-tools opt-in,
|
|
333
|
+
always-allow list), Memory (status, consent), Developer (the toggle).
|
|
334
|
+
|
|
335
|
+
### 3.9 Chat-with-files (RAG v1)
|
|
336
|
+
|
|
337
|
+
Attached text files below an inline threshold go into the prompt verbatim;
|
|
338
|
+
above it, a pure client-side BM25 retriever (`rag.ts`, no embeddings, no
|
|
339
|
+
network) selects top chunks, injects them as a numbered context block, and
|
|
340
|
+
the reply's `[n]` markers linkify into a collapsed Sources panel.
|
|
341
|
+
Citations survive regenerate / edit; they are not persisted into history
|
|
342
|
+
replay (needs a `HistoryItem` field — open). A vector upgrade over
|
|
343
|
+
`src/embed.ts` is explicitly later.
|
|
344
|
+
|
|
345
|
+
### 3.10 Code and Canvas
|
|
346
|
+
|
|
347
|
+
Fenced code gets copy + highlighting. `html` / `svg` fences get a
|
|
348
|
+
Preview | Source toggle: a lazily created `sandbox="allow-scripts"`
|
|
349
|
+
`srcdoc` iframe, never `allow-same-origin`, height-clamped, view state
|
|
350
|
+
preserved across re-render and streaming finish. The persistent Artifacts
|
|
351
|
+
side panel is not built (open, §5).
|
|
352
|
+
|
|
353
|
+
### 3.11 The app-aware assistant
|
|
354
|
+
|
|
355
|
+
The agent can see and act on the app it lives in — no screenshots, no
|
|
356
|
+
vision model in the loop. Product theory: guided tours fail on timing
|
|
357
|
+
(help before intent), chatbot help fails on grounding (intent without
|
|
358
|
+
state); this is the only shape with both, and the acceptance test is a
|
|
359
|
+
feeling — someone is there with you.
|
|
360
|
+
|
|
361
|
+
- **See:** on route / wizard-step change the browser pushes a `context`
|
|
362
|
+
frame (route, view, step, `uiSnapshot` of visible interactive elements,
|
|
363
|
+
capped ~120, agent chrome excluded). `get_current_app_context` returns
|
|
364
|
+
it; a compact ambient line (route / view / step) is auto-prepended to
|
|
365
|
+
every turn so the model never answers blind.
|
|
366
|
+
- **Navigate:** `navigate_app` validated server-side against the route
|
|
367
|
+
catalog → `ui_navigate` → the hash router. Reversible, no approval.
|
|
368
|
+
- **Point:** `spotlight_ui` resolves ref → selector → label → catalog
|
|
369
|
+
target against the last snapshot and shows a hand-rolled overlay:
|
|
370
|
+
`pointer-events: none` (the control stays clickable through it), no
|
|
371
|
+
focus trap, auto-dismiss in seconds, any input dismisses, reduced-motion
|
|
372
|
+
honored. Model-invented selectors are a miss, never a throw.
|
|
373
|
+
|
|
374
|
+
Scope guard: these tools exist only on the pi-web bridge (never the
|
|
375
|
+
OpenAI / Anthropic API surface); the snapshot never includes cross-origin
|
|
376
|
+
content; approval-gated `ui_act` (fill this form, start this job) is open
|
|
377
|
+
work and must render through the §3.4 approval card.
|
|
378
|
+
|
|
379
|
+
### 3.12 Speed and trust as ambient proof
|
|
380
|
+
|
|
381
|
+
The perf strip's lane badge is server-reported per turn (risk: a stale or
|
|
382
|
+
wrong badge is a P0, because the differentiator rests on it being
|
|
383
|
+
trustworthy). Context-fill comes from Pi's `getContextUsage`. What is still
|
|
384
|
+
missing from the trust surface is listed in §5.
|
|
255
385
|
|
|
256
386
|
---
|
|
257
387
|
|
|
258
|
-
## 4.
|
|
259
|
-
|
|
260
|
-
|
|
261
|
-
|
|
262
|
-
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
|
|
266
|
-
|
|
267
|
-
|
|
268
|
-
|
|
269
|
-
|
|
270
|
-
|
|
271
|
-
|
|
272
|
-
|
|
273
|
-
|
|
274
|
-
|
|
275
|
-
|
|
276
|
-
|
|
277
|
-
|
|
278
|
-
|
|
279
|
-
4. **Quiet retrieval, legible provenance.** Memory tools fire silently by
|
|
280
|
-
design (never a context dump) but every consulted source gets a small,
|
|
281
|
-
dismissable, expandable citation — the AnythingLLM/ChatGPT memory-panel
|
|
282
|
-
legibility bar, applied to a strictly deeper (weight-level) mechanism.
|
|
283
|
-
5. **Speed is ambient, not a stat you have to look up.** tok/s, TTFT,
|
|
284
|
-
context-fill, and which serving lane is active (serial / spec-decode /
|
|
285
|
-
batched ×N) are always visible near the composer, the way LM Studio 0.4
|
|
286
|
-
made context-fullness and processing state ambient chrome.
|
|
287
|
-
6. **Wire before you invent.** Prefer surfacing an existing backend
|
|
288
|
-
capability over designing a new one. Section 2.3's free-wins table is the
|
|
289
|
-
backlog triage order, not a suggestion.
|
|
290
|
-
7. **Local-only is an honest simplification, not a hedge.** Single-user,
|
|
291
|
-
local-machine execution means tool access, file writes, and shell
|
|
292
|
-
commands can be presented plainly ("runs on your machine, with your
|
|
293
|
-
files") instead of building sandbox theater Open WebUI/Odysseus need for
|
|
294
|
-
multi-tenant deployments we don't have.
|
|
295
|
-
8. **Keyboard-first for the power user, mouse-complete for everyone else.**
|
|
296
|
-
Every primary action (new chat, send, stop, regenerate, switch
|
|
297
|
-
model/adapter, command palette) has a keyboard path, but nothing is
|
|
298
|
-
keyboard-only-discoverable — mirror ChatGPT's documented shortcut sheet
|
|
299
|
-
(`Cmd/Ctrl+/`) as the baseline, not a stretch goal.
|
|
300
|
-
9. **The bar for done is perfect visual polish.** No overlapping text at
|
|
301
|
-
any viewport width, no animation jitter, no layout shift on stream, no
|
|
302
|
-
flash of unstyled/mispositioned anything. A feature that works but
|
|
303
|
-
looks unfinished is not done — it ships when it feels like a polished,
|
|
304
|
-
professional, excellent experience. Concretely: every animation is
|
|
305
|
-
compositor-friendly (transform/opacity only), interruptible, and
|
|
306
|
-
respects `prefers-reduced-motion`; every overlay/popover is tested at
|
|
307
|
-
375px, 768px, 1024px (iPad — where the 2026-07-06 nav-clip /
|
|
308
|
-
placeholder-wrap / pill-overflow jank actually surfaced), and desktop
|
|
309
|
-
widths and at 200% zoom; streaming text never
|
|
310
|
-
causes scroll jank or reflow of completed content (§5.3's memoization
|
|
311
|
-
is a polish requirement, not just a perf fix). Visual QA at phase
|
|
312
|
-
boundaries is part of the definition of done, same standing as tests.
|
|
388
|
+
## 4. Tech-stack decisions
|
|
389
|
+
|
|
390
|
+
- **Vanilla, no SPA framework.** The framework-native ideas (block
|
|
391
|
+
memoization, windowing, focus traps) port as functions; a migration is a
|
|
392
|
+
multi-month distraction at this file's complexity. Recorded so a future
|
|
393
|
+
session doesn't relitigate it without new evidence.
|
|
394
|
+
- **Typed module split over a single inline script.** The split's payoff
|
|
395
|
+
is the typed WS/API contract, not navigability: the Phase 0 bug class
|
|
396
|
+
(field-name drift, array truthiness, response-shape mismatch) became
|
|
397
|
+
build errors.
|
|
398
|
+
- **Vendored, no-CDN assets only.** highlight.js is vendored; KaTeX was
|
|
399
|
+
planned and is not vendored (open).
|
|
400
|
+
- **PWA = installability + instant shell**, never offline chat.
|
|
401
|
+
- **Theming:** auto / dark / light via `data-theme`, follows
|
|
402
|
+
`prefers-color-scheme`; animations transform/opacity only and gated by
|
|
403
|
+
`prefers-reduced-motion`.
|
|
404
|
+
- **A11y:** `role="log" aria-live="polite"` on the thread, focus traps with
|
|
405
|
+
Escape-returns-focus on every overlay, a shortcut sheet with ChatGPT's
|
|
406
|
+
vocabulary where sensible.
|
|
407
|
+
- **Hard rule for every overlay:** the spotlight is not a modal and must
|
|
408
|
+
never be wired into the modal machinery (`closeTopOverlay` / `trapFocus`).
|
|
313
409
|
|
|
314
410
|
---
|
|
315
411
|
|
|
316
|
-
## 5.
|
|
317
|
-
|
|
318
|
-
|
|
319
|
-
|
|
320
|
-
|
|
321
|
-
|
|
322
|
-
|
|
323
|
-
|
|
324
|
-
|
|
325
|
-
|
|
326
|
-
|
|
327
|
-
|
|
328
|
-
|
|
329
|
-
|
|
330
|
-
|
|
331
|
-
|
|
332
|
-
|
|
333
|
-
|
|
334
|
-
|
|
335
|
-
|
|
336
|
-
|
|
337
|
-
|
|
338
|
-
-
|
|
339
|
-
|
|
340
|
-
|
|
341
|
-
|
|
342
|
-
|
|
343
|
-
|
|
344
|
-
|
|
345
|
-
|
|
346
|
-
|
|
347
|
-
|
|
348
|
-
|
|
349
|
-
-
|
|
350
|
-
|
|
351
|
-
`
|
|
352
|
-
|
|
353
|
-
|
|
354
|
-
|
|
355
|
-
|
|
356
|
-
|
|
357
|
-
|
|
358
|
-
|
|
359
|
-
|
|
360
|
-
-
|
|
361
|
-
|
|
362
|
-
|
|
363
|
-
-
|
|
364
|
-
|
|
365
|
-
|
|
366
|
-
|
|
367
|
-
|
|
368
|
-
|
|
369
|
-
|
|
370
|
-
|
|
371
|
-
|
|
372
|
-
|
|
373
|
-
|
|
374
|
-
|
|
375
|
-
|
|
376
|
-
|
|
377
|
-
|
|
378
|
-
|
|
379
|
-
|
|
380
|
-
plain text. This is not optional polish: on a fast local model (the whole
|
|
381
|
-
product pitch), the current renderer makes the *frontend* the visible
|
|
382
|
-
bottleneck.
|
|
383
|
-
|
|
384
|
-
Steering and the follow-up queue already work server-side
|
|
385
|
-
(`queue_update` carries distinct `steering`/`followUp` arrays,
|
|
386
|
-
`pi-web.ts:390-397`) but render broken today (truthy-tested instead of
|
|
387
|
-
length-tested arrays, `web-ui-pass-plan.md` #2) — fix as part of this pass
|
|
388
|
-
since it's touching the same render path.
|
|
389
|
-
|
|
390
|
-
### 5.4 Tool-call & approval display
|
|
391
|
-
|
|
392
|
-
Keep the existing running/ok/fail visual language
|
|
393
|
-
(`app.html:1545-1584`). Two concrete changes:
|
|
394
|
-
1. **Wire the web chat's tool allowlist to include gated tools** (`bash`/`edit`/`write`)
|
|
395
|
-
behind an explicit, off-by-default "let it touch files" toggle in
|
|
396
|
-
settings (`buildPiAgentSurface({ codingTools: true })` on that path) —
|
|
397
|
-
so the already-built approval-card UI (`app.html:1586-1610`) stops being
|
|
398
|
-
dead code. Extend the card to show editable arguments before
|
|
399
|
-
approve/deny (LM Studio's pattern, beyond mlx-bun's current binary
|
|
400
|
-
allow/deny) and persist "always allow this tool" to a real config file
|
|
401
|
-
keyed by tool identity (fixing the fragile-persistence failure mode
|
|
402
|
-
documented against Claude Desktop/Cowork in the research).
|
|
403
|
-
2. **Give memory tool calls a distinct card style** — not the wrench icon,
|
|
404
|
-
something closer to a citation chip — so "the assistant looked something
|
|
405
|
-
up about you" reads differently from "the assistant ran a shell
|
|
406
|
-
command." This is the single highest-leverage rendering change in the
|
|
407
|
-
whole plan given §2.2 finding #1.
|
|
408
|
-
|
|
409
|
-
### 5.5 Memory surface (view / edit / forget, LoRA indicator)
|
|
410
|
-
|
|
411
|
-
A real **Memory panel**, reachable from the chat sidebar (not a 7th peer tab
|
|
412
|
-
to Quantize/Fine-tune — memory is chat-adjacent, one click away, because
|
|
413
|
-
it's the product, not a dev tool). Contents:
|
|
414
|
-
- **Vault browser + search**, backed by `memory_list`/`memory_search`,
|
|
415
|
-
with a `Reference/` sub-view for mlx-bun's own docs (clearly separated
|
|
416
|
-
from personal articles) via `reference_list`/`reference_search`.
|
|
417
|
-
- **Per-article view with a git-log/diff toggle** — the "watch it self-heal"
|
|
418
|
-
moment made literal: after a correction, a user can see the exact article
|
|
419
|
-
diff from the next nightly synthesis. This is the sharpest wedge against
|
|
420
|
-
ChatGPT/Claude's memory UX, which shows *what* is remembered but never
|
|
421
|
-
*how* — because their synthesis runs on someone else's infrastructure.
|
|
422
|
-
Ours runs locally and the mechanism itself is inspectable.
|
|
423
|
-
- **Status strip**: `vaultStatus()` already returns article count, git
|
|
424
|
-
state, last-synthesis time — render it plainly.
|
|
425
|
-
- **Wikilink graph view** using `memory_links` (inbound/outbound per
|
|
426
|
-
article) — cheap given the tool already resolves this data; a genuine
|
|
427
|
-
"knowledge graph of what the model knows about you" view nobody else in
|
|
428
|
-
the survey has because nobody else has a git-tracked personal wiki.
|
|
429
|
-
- **Per-turn provenance chip** on any assistant message that consulted
|
|
430
|
-
memory tools (from §5.4) — links into this same panel.
|
|
431
|
-
|
|
432
|
-
### 5.6 Model & adapter switching
|
|
433
|
-
|
|
434
|
-
Make the nav model label (`#nav-model`, currently a dead `<span>` with no
|
|
435
|
-
click handler) clickable to a lightweight picker: what's loaded, hardware-fit
|
|
436
|
-
signal (green/yellow/red, LM Studio's pattern) computed against this
|
|
437
|
-
specific Mac's measured bandwidth via `GET /fit` — not a generic heuristic,
|
|
438
|
-
since we already have per-SKU fit prediction in `src/fit.ts`. Model swap
|
|
439
|
-
should preserve session history and re-prime cleanly (every competitor
|
|
440
|
-
surveyed is documented as bad at this — LM Studio has an open bug where
|
|
441
|
-
switching models clears the selector and forces manual reselection; a
|
|
442
|
-
genuinely smooth swap is a believable, demonstrable win).
|
|
443
|
-
|
|
444
|
-
Adapter selection: promote from flat dropdown (§5.2) to a routing table in
|
|
445
|
-
settings showing all three states (available on disk / loaded+mounted /
|
|
446
|
-
selected for this request) with RAM cost visible per loaded adapter (optiq's
|
|
447
|
-
own numbers: ~50MB/adapter, 10 resident ≈ 6.1GB — a concrete budget to design
|
|
448
|
-
the UI affordance against), and expose `"a+b"` stacking since `resolveSpec`
|
|
449
|
-
already supports it server-side.
|
|
450
|
-
|
|
451
|
-
### 5.7 Sessions / projects
|
|
452
|
-
|
|
453
|
-
Add a search box above the session list (trivial, local-only, full-text —
|
|
454
|
-
no reason to skip it given Claude's own title-only search is a widely-cited
|
|
455
|
-
annoyance and we have no server-cost constraint). On mobile, replace the
|
|
456
|
-
currently-vanishing sidebar (`app.html:345`) with a slide-over drawer
|
|
457
|
-
triggered by a hamburger — nothing fancier needed for v1.
|
|
458
|
-
|
|
459
|
-
A lightweight **persona/bundle** concept (system prompt + adapter selection
|
|
460
|
-
+ tool allowlist + memory scope, saved and nameable) is a Phase 5 item (see
|
|
461
|
-
§6) — not required for the core redesign, but the composer/settings data
|
|
462
|
-
model should be shaped so a "save this as a named bundle" action is a
|
|
463
|
-
thin layer on top, not a re-architecture.
|
|
464
|
-
|
|
465
|
-
### 5.8 Settings
|
|
466
|
-
|
|
467
|
-
Collapse into three groups: **Chat** (sampling, thinking toggle, tool
|
|
468
|
-
allowlist/gating), **Memory** (vault path, consent, synthesis schedule
|
|
469
|
-
status — read from `memory_status`), **Developer** (the progressive-disclosure
|
|
470
|
-
gate for Quantize/Fine-tune/Build-Dataset/Curves/Routes — see §8).
|
|
471
|
-
|
|
472
|
-
### 5.9 Structured output
|
|
473
|
-
|
|
474
|
-
Server already supports `response_format` json_schema/json_object and
|
|
475
|
-
`guided_grammar`/`guided_regex`/`guided_choice` via xgrammar
|
|
476
|
-
(`src/server.ts:458-471`) with zero UI anywhere. Ship a lightweight
|
|
477
|
-
schema/grammar picker in the Advanced sampling disclosure (§5.2) — even a
|
|
478
|
-
raw JSON-schema textarea with live validation is pure whitespace relative to
|
|
479
|
-
every competitor (mlx-lm has no `response_format` at all; most local UIs
|
|
480
|
-
don't expose it even when their server supports it).
|
|
481
|
-
|
|
482
|
-
### 5.10 Vision / multimodal
|
|
483
|
-
|
|
484
|
-
Vision is already wired end-to-end correctly (capability negotiation via
|
|
485
|
-
the `ready` frame's `vision` flag, `pi-provider.ts:93-100`). The only fix
|
|
486
|
-
needed here is the stale copy bug (`web-ui-pass-plan.md` #6: "switch to the
|
|
487
|
-
12B for vision" is wrong now that e2b/e4b/26B/31B are all vision-capable via
|
|
488
|
-
`visionCapable(m)`). No new UI work required for v1 beyond that correction
|
|
489
|
-
and folding image/doc attach into the unified `#` retrieval gesture (§5.2).
|
|
490
|
-
|
|
491
|
-
### 5.11 Artifacts / code
|
|
492
|
-
|
|
493
|
-
Fenced-code rendering gets a copy button (already exists) plus syntax
|
|
494
|
-
highlighting (§5.3). **Canvas v1 — fenced HTML rendered in a sandboxed
|
|
495
|
-
iframe with a view-source toggle — is a Phase 3 item** (optiq Lab ships
|
|
496
|
-
exactly this today; "I miss optiq's Canvas" is a disallowed outcome). The
|
|
497
|
-
full Artifacts/Canvas *side-panel* (persistent, versioned,
|
|
498
|
-
highlight-to-scope-edit) stays a Phase 5 stretch — both cloud incumbents'
|
|
499
|
-
own experience (OpenAI partially walked Canvas back in May 2026 for the
|
|
500
|
-
common case) argues for opt-in-by-heuristic (auto-open above ~10 lines of
|
|
501
|
-
code / high fence density), not a mandatory pane.
|
|
412
|
+
## 5. Open work
|
|
413
|
+
|
|
414
|
+
Nothing below has a PLAN.md phase; it is the backlog this doc owns.
|
|
415
|
+
Ordered by leverage.
|
|
416
|
+
|
|
417
|
+
**Residuals from the shipped phases (verified against src 2026-08-23):**
|
|
418
|
+
|
|
419
|
+
- KaTeX / math rendering — planned in the render pass, not vendored.
|
|
420
|
+
- Structured-output picker (JSON schema / grammar textarea with live
|
|
421
|
+
validation) — server supports `response_format` and
|
|
422
|
+
`guided_grammar|regex|choice`; no composer UI.
|
|
423
|
+
- Wikilink graph view — `/api/memory/links` exists; the panel renders a
|
|
424
|
+
links line only.
|
|
425
|
+
- Session-preserving live model swap — blocked on a reload seam at the
|
|
426
|
+
engine boundary (`docs/reference/server-config.md`); until then the Hub / picker
|
|
427
|
+
hand back the restart command.
|
|
428
|
+
- RAG citations persisted into history replay (`HistoryItem` field).
|
|
429
|
+
- Long-thread windowing (anchored-to-bottom DOM recycling) — not built;
|
|
430
|
+
months-long sessions are more likely here than in a stateless chatbot.
|
|
431
|
+
- Adapter metadata: the discovery rows are indistinguishable when several
|
|
432
|
+
adapters share base / rank / scale; the trainer should stamp task / data
|
|
433
|
+
/ date (and a stable repo id) into the adapter config at save time.
|
|
434
|
+
- Overnight training from the web UI ("start before bed, wake up to an
|
|
435
|
+
adapter"): the fine-tune wizard streams a job (`/api/jobs/:id/stream`),
|
|
436
|
+
but a reconnectable monitor with completion → "available" hand-off is
|
|
437
|
+
not designed. `--force-adapter` (serve pinned to one adapter for an
|
|
438
|
+
unattended pipeline) is a CLI item, tracked in the CLI reference.
|
|
439
|
+
|
|
440
|
+
**Trust and speed differentiators (former Phase 4):**
|
|
441
|
+
|
|
442
|
+
- "Temporary chat" pill — provably never written to disk.
|
|
443
|
+
- Per-session disk-touch tally (files read / written) as a trust log
|
|
444
|
+
distinct from tool cards; per-session "zero cloud calls" indicator.
|
|
445
|
+
- Prompt-cache-hit telemetry per turn ("reused N cached tokens").
|
|
446
|
+
- logprobs-derived confidence overlay (server already returns
|
|
447
|
+
`logprobs` / `top_logprobs`).
|
|
448
|
+
- HLG tone-curve controls in the composer (today only on the Curves page).
|
|
449
|
+
- App-aware assistant maturity: a persistent minimized affordance on every
|
|
450
|
+
view (Developer tabs included) that minimizes on engagement and
|
|
451
|
+
re-expands only on invocation; process-state enrichment of the snapshot
|
|
452
|
+
(job progress, field completion) beyond route / step.
|
|
453
|
+
|
|
454
|
+
**Stretch (former Phase 5; pick one or two at a time):**
|
|
455
|
+
|
|
456
|
+
- Full conversation tree view over the sibling groups Pi already stores.
|
|
457
|
+
- Persona bundles (system prompt + adapter + tool allowlist + memory scope)
|
|
458
|
+
— presets v1's data model is shaped so this is a thin layer.
|
|
459
|
+
- Compare / split view: one prompt against base vs adapter, or two quant
|
|
460
|
+
tiers, side by side with per-pane tok/s (optiq Arena's mechanics, plus
|
|
461
|
+
the comparisons Arena cannot express).
|
|
462
|
+
- Approval-gated `ui_act` tools.
|
|
463
|
+
- Artifacts side panel, opt-in by heuristic; Mermaid; vector RAG upgrade;
|
|
464
|
+
MCP client; Skills-style procedural memory (all Appendix A rows with a
|
|
465
|
+
written state).
|
|
466
|
+
- Memory that graduates into a trained memory LoRA — roadmap; the UI must
|
|
467
|
+
never claim it before it ships.
|
|
468
|
+
|
|
469
|
+
**Risks that stay live:** the superset doctrine ballooning scope (keep
|
|
470
|
+
items independently shippable; re-audit Appendix A at each boundary, not
|
|
471
|
+
mid-stream); the lane badge going stale in a mode transition (server-
|
|
472
|
+
driven only); always-allow persistence regressing to the fragility seen in
|
|
473
|
+
other desktop agents (versioned config keyed by tool identity, migration
|
|
474
|
+
test); the memory diff view unsettling a user (user-initiated inspection,
|
|
475
|
+
pause / reset one click away).
|
|
502
476
|
|
|
503
477
|
---
|
|
504
478
|
|
|
505
|
-
##
|
|
506
|
-
|
|
507
|
-
|
|
508
|
-
|
|
509
|
-
|
|
510
|
-
|
|
511
|
-
|
|
512
|
-
|
|
513
|
-
|
|
514
|
-
chat
|
|
515
|
-
|
|
516
|
-
|
|
517
|
-
|
|
518
|
-
|
|
519
|
-
|
|
520
|
-
|
|
521
|
-
|
|
522
|
-
|
|
523
|
-
|
|
524
|
-
|
|
525
|
-
|
|
526
|
-
|
|
527
|
-
optiq proved the resident-multi-adapter mechanism works (~50MB/adapter,
|
|
528
|
-
6.1GB for 10 resident vs. 56GB for 10 full model copies) but shipped it
|
|
529
|
-
Python-API-only. LM Studio's own bug tracker confirms adapter hot-swap is
|
|
530
|
-
unsupported. mlx-bun already has `/v1/adapters`, adapter-namespaced KV, and
|
|
531
|
-
per-request injection working end to end (`src/lora.ts`,
|
|
532
|
-
`docs/design/adapters-end-to-end.md`). The UI work is turning this into a
|
|
533
|
-
routing table: which adapter is active per-surface, why (matched from
|
|
534
|
-
surface / explicitly selected), its RAM cost, and — the genuine escalation —
|
|
535
|
-
support for stacking (`"a+b"`) directly in the picker, something the backend
|
|
536
|
-
already does and no UI anywhere exposes.
|
|
537
|
-
|
|
538
|
-
### 6.3 On-device privacy, proven not asserted
|
|
539
|
-
|
|
540
|
-
A one-click "Temporary chat" pill (matching ChatGPT/Claude's composer-header
|
|
541
|
-
convention) that — unlike cloud temporary chat, which OpenAI's own
|
|
542
|
-
disclosures say is still retained 30 days for "safety" — can be made
|
|
543
|
-
**provably** never written to disk at all, because we control the whole
|
|
544
|
-
stack. Pair this with a per-session "zero cloud calls" indicator and a
|
|
545
|
-
running tally of what the agent touched on disk (files read/written this
|
|
546
|
-
session) as a trust surface distinct from generic tool-call cards.
|
|
547
|
-
|
|
548
|
-
### 6.4 Speed surfaced as a felt experience, not a hidden stat
|
|
549
|
-
|
|
550
|
-
The perf strip (§5.2) — tok/s, TTFT, context-fill, lane badge — is the
|
|
551
|
-
direct answer to "serving speed is the user metric" (MEMORY.md). No
|
|
552
|
-
surveyed competitor (including the MLX-native peers macMLX/mlx-serve, which
|
|
553
|
-
do have a Benchmark tab) integrates lane-awareness (serial vs. spec-decode
|
|
554
|
-
vs. batched) into the *live chat surface* itself — they show a benchmark
|
|
555
|
-
number in a separate tab, we can show which lane is running the actual
|
|
556
|
-
conversation right now. This directly explains latency variance instead of
|
|
557
|
-
letting it read as jank, and ties concretely into the active `dspark`
|
|
558
|
-
spec-decode branch.
|
|
559
|
-
|
|
560
|
-
### 6.5 Local tool transparency as a trust surface, not a permission dialog
|
|
561
|
-
|
|
562
|
-
Every competitor's tool-approval UX is either a blocking modal (old Jan,
|
|
563
|
-
Claude Desktop) or a binary allow/deny with no argument preview (most).
|
|
564
|
-
mlx-bun already has the approval round-trip built (`pi-web.ts:748-774`); the
|
|
565
|
-
differentiator is finishing it properly: inline non-blocking cards (Jan
|
|
566
|
-
v0.8.0's pattern) with editable arguments before execution (LM Studio's
|
|
567
|
-
pattern) and durable, tool-identity-keyed "always allow" persistence (fixing
|
|
568
|
-
the failure mode publicly documented against Claude Desktop). Combined with
|
|
569
|
-
§6.3's disk-touch tally, this reframes "the agent can run bash" from a scary
|
|
570
|
-
permission dialog into a legible, reviewable log — the natural trust story
|
|
571
|
-
for a single-user local product that structurally doesn't need Open WebUI's
|
|
572
|
-
sandbox theater.
|
|
573
|
-
|
|
574
|
-
### 6.6 The app-aware assistant (added 2026-07-06)
|
|
575
|
-
|
|
576
|
-
The chat agent can see and act on the app it lives in. Verified against the
|
|
577
|
-
full beat matrix: **no product in the surveyed field has this** (matrix
|
|
578
|
-
Axis 12). Reference implementation proven in-house on the same pi SDK:
|
|
579
|
-
`~/Code/PortfolioManager`.
|
|
580
|
-
|
|
581
|
-
**Why this matters (the product theory, not the feature):** both existing
|
|
582
|
-
in-app help paradigms fail on the same axis from opposite ends. Guided
|
|
583
|
-
tours fail on *timing* — they front-load help before intent exists, so
|
|
584
|
-
users skip them and the knowledge is gone by the time they care. Chatbot
|
|
585
|
-
help fails on *grounding* — it arrives at the moment of intent but blind:
|
|
586
|
-
no idea what screen you're on, what step you're in, what you just tried; a
|
|
587
|
-
search bar with a face, and users feel it immediately. The app-aware
|
|
588
|
-
assistant is the only shape with both: help at the moment of intent,
|
|
589
|
-
grounded in current state. Instead of screenshotting and circling a thing
|
|
590
|
-
for an AI, you ask — and it highlights, navigates, takes you there. **The
|
|
591
|
-
acceptance test is a feeling: like someone is there with you.** Not a
|
|
592
|
-
tour, not a search bar — presence.
|
|
593
|
-
|
|
594
|
-
**Design consequences the feeling forces** (a naive port misses these):
|
|
595
|
-
|
|
596
|
-
- **Ambient across every view, not confined to the Chat tab.** A guide you
|
|
597
|
-
must leave the Quantize tab to talk to is blind again by construction.
|
|
598
|
-
Ship a lightweight persistent assistant affordance on every view —
|
|
599
|
-
Developer tabs included — with the Chat tab as the deep surface
|
|
600
|
-
(PortfolioManager's floating-panel pattern proves it on our SDK).
|
|
601
|
-
**And the panel itself is never intrusive:** the moment the user engages
|
|
602
|
-
the page (clicks Quantize, focuses a form field), the panel minimizes
|
|
603
|
-
with a smooth animation to a small persistent affordance — you know it's
|
|
604
|
-
there, it isn't blocking you. It re-expands on invocation, never on its
|
|
605
|
-
own initiative.
|
|
606
|
-
- **The snapshot carries *process state*, not just elements.** "Knows what
|
|
607
|
-
part of the process you're in" means the context push includes wizard
|
|
608
|
-
step (Quantize step 2 of 4), running job progress, which fields are
|
|
609
|
-
filled — beyond PortfolioManager's visible-element list.
|
|
610
|
-
- **Never answer blind.** Tool-pull alone (the agent *choosing* to call
|
|
611
|
-
`get_current_app_context`) sometimes won't fire, and one ungrounded
|
|
612
|
-
answer breaks the "with you" illusion. Auto-attach a compact ambient
|
|
613
|
-
context line (route + view + step) to every turn; the deep snapshot
|
|
614
|
-
stays on-demand.
|
|
615
|
-
- **Choreography:** the spotlight lands *while* the explanation streams
|
|
616
|
-
("it's right here —" and the highlight appears), not as a report
|
|
617
|
-
afterwards.
|
|
618
|
-
- **Never hijack (hard constraint).** The worst tour UX is blocking the
|
|
619
|
-
screen when the user wants to DO something. The spotlight is an overlay,
|
|
620
|
-
never a modal: it traps no focus, blocks no clicks (the highlighted
|
|
621
|
-
control stays clickable *through* it), auto-dismisses in seconds, and
|
|
622
|
-
any user input dismisses it instantly. Navigation only happens when the
|
|
623
|
-
user asked to be taken somewhere — never uninvited mid-task. The agent
|
|
624
|
-
is a hand pointing, not a hand on the wheel; there is never a state the
|
|
625
|
-
user has to escape from. (Same rule already governs approval cards —
|
|
626
|
-
inline and non-blocking, §6.5.)
|
|
627
|
-
|
|
628
|
-
Mechanism — three tools plus one client loop, no screenshots, no vision
|
|
629
|
-
model in the loop:
|
|
630
|
-
|
|
631
|
-
1. **See**: on route change the web app pushes a structured `uiSnapshot`
|
|
632
|
-
(visible interactive elements + labeled regions as
|
|
633
|
-
`{ref, label, role, selector}`, ~120-element cap, agent chrome
|
|
634
|
-
excluded) as a `context` WS frame; a `get_current_app_context` tool
|
|
635
|
-
returns it. The screenshot→upload→"what am I looking at" round-trip
|
|
636
|
-
disappears because the app narrates itself, locally.
|
|
637
|
-
2. **Navigate**: `navigate_app` validated against a curated route catalog
|
|
638
|
-
(`data-spotlight` anchors + route map, PortfolioManager's
|
|
639
|
-
`ui-catalog.json` pattern) → `ui_navigate` frame → the browser routes.
|
|
640
|
-
"Take me to Quantize" literally clicks the tab. Reversible, no approval
|
|
641
|
-
needed.
|
|
642
|
-
3. **Point**: `spotlight_ui` locates a control by ref/label/catalog-id,
|
|
643
|
-
optionally navigates first, and highlights it with a hint popover
|
|
644
|
-
(hand-rolled backdrop-cutout overlay in our tokens, ~80 lines —
|
|
645
|
-
no vendored tour library needed), auto-dismissing.
|
|
646
|
-
|
|
647
|
-
Why it's ours to win: (a) our web chat IS a pi agent over WS — the port is
|
|
648
|
-
mechanical (`pi-web.ts` tools + two frames + a snapshot builder in the
|
|
649
|
-
app); (b) it resolves the Developer-toggle discoverability tension §8
|
|
650
|
-
creates — hidden capability becomes conversationally discoverable ("can
|
|
651
|
-
this make a smaller model?" → reveal, navigate, spotlight); (c) the
|
|
652
|
-
escalation nobody else can ship: approval-gated `ui_act` tools (fill this
|
|
653
|
-
form, start this job) rendered through the §6.5 approval-card trust
|
|
654
|
-
surface — guided *actions* with visible allow/deny, where PortfolioManager
|
|
655
|
-
deliberately stops at navigate+point and generic computer-use agents act
|
|
656
|
-
with no in-app trust model at all; (d) with the memory vault, "take me to
|
|
657
|
-
that adapter I trained last week" actually resolves.
|
|
658
|
-
|
|
659
|
-
Scope guard: the snapshot never includes cross-origin content; the three
|
|
660
|
-
tools live on the pi-web bridge only (never the OpenAI/Anthropic API
|
|
661
|
-
surface); `ui_act` is Phase 5 and requires the Phase-2 approval surface
|
|
662
|
-
first.
|
|
479
|
+
## History
|
|
480
|
+
|
|
481
|
+
- 2026-06-12 — minimal `/chat` page (SSE against `/v1/chat/completions`).
|
|
482
|
+
- 2026-06-14 — chat replaced with the Pi embed over `/ws/chat`
|
|
483
|
+
(`src/pi-web.ts`), unified hash-routed SPA with the Developer tabs.
|
|
484
|
+
- 2026-06-16 — adapter discovery endpoint, web selector, and the Pi CLI
|
|
485
|
+
extension (`docs/design/web-chat-redesign.md` written; its web half now lives in
|
|
486
|
+
§3.6, the CLI half belongs to the CLI reference). The Tauri desktop-shell
|
|
487
|
+
idea (`docs/design/web-chat-redesign.md`: separate repo, sidecar `mlx-bun serve`,
|
|
488
|
+
WKWebView on `/#/chat`) was written in this period; it is tracked in
|
|
489
|
+
neither PLAN.md nor STATUS.md and is not a plan — the only desktop-specific
|
|
490
|
+
item kept here is that a shell must wrap the same page, no bespoke UI.
|
|
491
|
+
- 2026-07-06 — this plan merged three design visions and the market survey;
|
|
492
|
+
Phase 0 (UI-only bug pass) and Phase 1 (block-memoized render,
|
|
493
|
+
highlighting, full sampler popover, perf strip + lane, message actions,
|
|
494
|
+
drawer, theme, shortcuts, a11y) landed. Beat matrix written (Appendix A).
|
|
495
|
+
- 2026-07-07 — Phase 2 (typed module split, memory panel + REST, adapter
|
|
496
|
+
routing table, system prompt + presets, approval gate wired, Developer
|
|
497
|
+
IA) and Phase 3 (files RAG, Model Hub, Canvas, per-turn sampling scope,
|
|
498
|
+
self-healing tool calls + loop hygiene, app-aware assistant v1, command
|
|
499
|
+
palette, full-text search, export, PWA) landed.
|
|
500
|
+
- 2026-08-23 — consolidated into this canonical doc; verified against src.
|
|
663
501
|
|
|
664
502
|
---
|
|
665
503
|
|
|
666
|
-
##
|
|
667
|
-
|
|
668
|
-
**Recommendation: stay vanilla-single-file, but stop hand-rolling
|
|
669
|
-
everything.** Do not migrate to React/Svelte/a SPA framework. Reasoning:
|
|
670
|
-
|
|
671
|
-
- The research is explicit that every framework-based pattern found
|
|
672
|
-
(Streamdown's block memoization, react-virtuoso's windowing,
|
|
673
|
-
Radix/Base-UI's accessible primitives) assumes a component-diffing
|
|
674
|
-
runtime — but the *ideas* (memoize completed blocks, window a long list,
|
|
675
|
-
trap focus in a popover) port cleanly into vanilla JS without importing
|
|
676
|
-
the library. The current file already proves this pattern works (the
|
|
677
|
-
hand-rolled markdown renderer is competitive).
|
|
678
|
-
- A framework migration is a multi-month distraction that trades a
|
|
679
|
-
self-contained, no-build, no-CDN served single file (the current
|
|
680
|
-
`app.html` header comment's own stated constraint) for a build pipeline,
|
|
681
|
-
for a benefit (component reuse, cleaner state management) that matters
|
|
682
|
-
more at team scale than at this file's actual complexity once the
|
|
683
|
-
concrete gaps (memoized rendering, syntax highlighting, math, a11y) are
|
|
684
|
-
closed directly.
|
|
685
|
-
- The three visions converged independently on the same conclusion ("no
|
|
686
|
-
React needed, just a smarter diff in the existing vanilla renderer") —
|
|
687
|
-
treat that convergence as signal, not coincidence.
|
|
688
|
-
|
|
689
|
-
**What does change:**
|
|
690
|
-
- **Split `app.html` into modules that build into the single served file**,
|
|
691
|
-
not a runtime framework. At 2737 lines and growing (memory panel, adapter
|
|
692
|
-
routing table, perf strip, message actions are all net-new surface), a
|
|
693
|
-
single file becomes a real maintenance cost even without a framework.
|
|
694
|
-
Introduce a minimal build step (`bun build` with inline output, still
|
|
695
|
-
zero external CDN dependency at runtime) that concatenates
|
|
696
|
-
`src/web/{shell,chat,memory,adapters,quantize,finetune,dataset,status}.{html,css,ts}`
|
|
697
|
-
into the one file the server ships. This preserves every runtime
|
|
698
|
-
constraint (self-contained, no CSP-breaking externals) while making the
|
|
699
|
-
source tree navigable.
|
|
700
|
-
**The split's primary payoff is the typed contract, not navigability**
|
|
701
|
-
(amended 2026-07-06): once the inline script is TypeScript, it imports
|
|
702
|
-
`ServerMessage`/`ClientMessage` from `src/pi-web.ts` and the job-stream /
|
|
703
|
-
API response shapes from their source modules — and the single largest
|
|
704
|
-
empirical bug class disappears at compile time. Look at what Phase 0
|
|
705
|
-
actually fixed: `e.lr` vs `e.learning_rate`, array-truthiness on a WS
|
|
706
|
-
frame, a response-shape mismatch, a dead string-enum — every one is
|
|
707
|
-
contract drift between untyped inline JS and typed TS, and every one
|
|
708
|
-
becomes a build error after the split. **Scheduled: the split is Phase 2's
|
|
709
|
-
first work item** — before the memory panel and routing table add major
|
|
710
|
-
new surface to the monolith, not after.
|
|
711
|
-
- **Frontend regression harness** (amended 2026-07-06 — this repo
|
|
712
|
-
oracle-gates everything else; the UI can't be the one untested surface):
|
|
713
|
-
(a) formalize the inline-script syntax check (extract `<script>`,
|
|
714
|
-
`new Function()` it) as a test/hygiene-gate step instead of an ad-hoc
|
|
715
|
-
command; (b) a `tests/web-app.test.ts` DOM-level unit file (happy-dom or
|
|
716
|
-
equivalent) covering the pure client functions — `splitBlocks`/streaming
|
|
717
|
-
memoization (streamed-then-finalized output must equal one-shot
|
|
718
|
-
`mdToHtml` on a fixture corpus: the renderer's own parity oracle),
|
|
719
|
-
`renderQueue`, `api()` error unwrapping, esc() discipline on
|
|
720
|
-
interpolation sites; (c) post-split, type-checking the frontend modules
|
|
721
|
-
in the same `tsc --noEmit` pass as the server. Playwright-class
|
|
722
|
-
end-to-end stays out of scope until there's a model-free server stub to
|
|
723
|
-
drive it against.
|
|
724
|
-
- **Inline, self-hosted Shiki (or a lighter alternative) and KaTeX** as
|
|
725
|
-
vendored, no-CDN assets bundled at build time — a real dependency-weight
|
|
726
|
-
tradeoff against the "no build, no CDN" constraint, but justified because
|
|
727
|
-
monochrome code and raw-LaTeX-backslash output are concrete, embarrassing
|
|
728
|
-
gaps (§2.2 #9) with no vanilla-hand-rolled alternative that doesn't
|
|
729
|
-
reinvent a TextMate-grammar highlighter from scratch.
|
|
730
|
-
- **Hand-roll block-memoized streaming and message-list windowing** rather
|
|
731
|
-
than adopt react-virtuoso — port the *idea* (completed-block skip-reparse,
|
|
732
|
-
windowed DOM recycling anchored to bottom scroll) as vanilla functions.
|
|
733
|
-
|
|
734
|
-
**Tauri desktop compatibility:** the Tauri shell today wraps "the same page,
|
|
735
|
-
no bespoke UI" (`docs/design/tauri-desktop-app.md:50-51`). Keep that
|
|
736
|
-
contract — every feature in this plan ships to both surfaces for free by
|
|
737
|
-
construction, since nothing here depends on a browser-only API. The one
|
|
738
|
-
Tauri-specific opportunity (deferred to Phase 5, not required): a global
|
|
739
|
-
hotkey + floating companion window (mirroring ChatGPT macOS's Option+Space
|
|
740
|
-
and mlx-serve's Ctrl-Space quick-launcher) — a concrete way for the desktop
|
|
741
|
-
wrap to earn native-app status instead of being a browser tab in a frame,
|
|
742
|
-
made cheap by the vision pipeline already existing server-side.
|
|
743
|
-
|
|
744
|
-
**Theming:** add `prefers-color-scheme` handling and a light theme (dark-only
|
|
745
|
-
today, hardcoded `--bg:#000`). Respect `prefers-reduced-motion` on the
|
|
746
|
-
ambient bloom/shimmer animations.
|
|
747
|
-
|
|
748
|
-
**Responsive/mobile/PWA:** mobile drawer (§5.7); add a web app manifest +
|
|
749
|
-
minimal cache-first service worker for the app shell (installable
|
|
750
|
-
"add to home screen," zero-cost given the app is already a single served
|
|
751
|
-
page) — explicitly scoped as *shell-instant-load + installability*, not
|
|
752
|
-
meaningful offline chat (the app is useless without the local inference
|
|
753
|
-
server running, so deep offline-caching strategy is not worth over-investing
|
|
754
|
-
in). Test standalone-mode viewport height explicitly since browser chrome
|
|
755
|
-
disappearing changes available height in ways normal responsive testing
|
|
756
|
-
misses.
|
|
757
|
-
|
|
758
|
-
**A11y:** add `role="log" aria-live="polite" aria-atomic="false"` on the
|
|
759
|
-
message thread, throttled to paragraph/sentence boundaries (not per-token —
|
|
760
|
-
combining focus-move and live-region announcement in the same update is the
|
|
761
|
-
documented anti-pattern per WCAG ARIA23), verified against actual VoiceOver
|
|
762
|
-
testing rather than one blog post's guidance. Full keyboard tab order
|
|
763
|
-
through composer/sidebar/popovers, visible focus rings, Escape-to-close on
|
|
764
|
-
every popover with focus returned to trigger. Ship a `Cmd/Ctrl+/` shortcut
|
|
765
|
-
sheet matching ChatGPT's vocabulary where sensible (`Cmd/Ctrl+Shift+O` new
|
|
766
|
-
chat, `Shift+Esc` focus composer, `Cmd/Ctrl+Shift+C` copy last response) —
|
|
767
|
-
zero cost to originality, closes an embarrassing baseline gap, and "copy
|
|
768
|
-
last response" reuses the existing copy-button code.
|
|
769
|
-
|
|
770
|
-
**Long-thread performance:** windowed rendering (anchored-to-bottom DOM
|
|
771
|
-
recycling) once a session exceeds ~150-200 rendered turns — the
|
|
772
|
-
memory-flywheel/long-session north star means power users will accumulate
|
|
773
|
-
very long sessions over months, more likely here than in a typical
|
|
774
|
-
stateless chatbot, so this failure mode (documented against Open WebUI as
|
|
775
|
-
issue #23067) is a higher-probability event for us than for competitors.
|
|
504
|
+
## Appendix A — beat matrix (condensed)
|
|
776
505
|
|
|
777
|
-
|
|
506
|
+
Every capability a user judges a local chat app by, judged against the best
|
|
507
|
+
current implementation in the surveyed field (mlx-lm, mlx-optiq, Ollama,
|
|
508
|
+
Open WebUI, LM Studio, Odysseus, Jan, Msty, GPT4All, AnythingLLM,
|
|
509
|
+
Enchanted, macMLX/mlx-serve, ChatGPT, Claude.ai). **MATCH** = parity is
|
|
510
|
+
enough; **BEAT** = structurally better because of something only we have;
|
|
511
|
+
**SKIP** = the engine lacks the substrate, stated so silence is never an
|
|
512
|
+
oversight. State column verified against src 2026-08-23. (Caveat: the
|
|
513
|
+
Odysseus rows rest on a public-web identification of that product; no
|
|
514
|
+
verdict flips on them.)
|
|
778
515
|
|
|
779
|
-
|
|
780
|
-
|
|
781
|
-
**Chat is the product. Everything else is a developer tool.** Collapse the
|
|
782
|
-
current 7 hash-routed tabs (Chat / Quantize / Fine-tune / Build Dataset /
|
|
783
|
-
Status / Curves / Routes) behind a single top-level split:
|
|
784
|
-
|
|
785
|
-
- **Chat** (default, always visible) — the entire app for a first-time or
|
|
786
|
-
casual user. Contains, reachable via sidebar/settings without leaving the
|
|
787
|
-
view: Memory panel, adapter routing, sampling/structured-output advanced
|
|
788
|
-
settings, model switcher.
|
|
789
|
-
- **Developer** (one nav toggle, LM Studio's User/Power-User/Developer
|
|
790
|
-
pattern collapsed to a binary since we don't need three tiers yet) —
|
|
791
|
-
reveals Quantize, Fine-tune, Build Dataset, Status, Curves, Routes as a
|
|
792
|
-
sub-nav. Persisted as a local preference (don't re-ask every session).
|
|
793
|
-
|
|
794
|
-
This is not hiding capability — it's matching disclosure to audience. A
|
|
795
|
-
user who never intends to quantize a model should never see that tab; a
|
|
796
|
-
developer flips one switch once and gets everything, including the existing
|
|
797
|
-
tabs completely unchanged in their internal design. The app-aware assistant
|
|
798
|
-
(§6.6) is the escape hatch that makes the collapse safe: hidden capability
|
|
799
|
-
stays conversationally discoverable — "can this make a smaller model?" and
|
|
800
|
-
the agent reveals the Developer tabs, navigates, and spotlights the control,
|
|
801
|
-
so progressive disclosure never becomes buried capability (risk #4's
|
|
802
|
-
strongest mitigation). Status's RAM/KV "Memory"
|
|
803
|
-
card keeps its current meaning (system memory) — the personal-memory vault
|
|
804
|
-
gets its own distinctly-named panel (§5.5) precisely to avoid the homonym
|
|
805
|
-
confusion the audit flagged.
|
|
806
|
-
|
|
807
|
-
Routes (currently an iframe of a DAG, 404s in the compiled binary per
|
|
808
|
-
`web-ui-pass-plan.md` #17) and Curves (a separate full-page route, never
|
|
809
|
-
highlights in nav per #19) both move under Developer and get their standing
|
|
810
|
-
bugs fixed as part of the IA move (§9 Phase 1).
|
|
516
|
+
**Axis 1 — Chat fundamentals**
|
|
811
517
|
|
|
812
|
-
|
|
518
|
+
| Capability | Verdict | State |
|
|
519
|
+
|---|---|---|
|
|
520
|
+
| Streaming with a prominent Stop | MATCH | shipped |
|
|
521
|
+
| Stop aborts in-flight tool work, not just tokens | MATCH | shipped (verified via Pi abort) |
|
|
522
|
+
| Disk persistence, no account | MATCH | shipped (Pi sessions) |
|
|
523
|
+
| Regenerate | MATCH | shipped |
|
|
524
|
+
| Edit-and-resend as a sibling branch, `< i/n >` | MATCH | shipped |
|
|
525
|
+
| Full conversation tree view | BEAT | open |
|
|
526
|
+
| Live steering + follow-up queue while streaming | BEAT | shipped |
|
|
527
|
+
| Keyboard reach for stop / regenerate / new chat | MATCH | shipped |
|
|
528
|
+
| Full-message copy | MATCH | shipped |
|
|
529
|
+
| Delete a single message | MATCH | open |
|
|
530
|
+
| Compaction seam marked in the thread | MATCH | open |
|
|
531
|
+
|
|
532
|
+
**Axis 2 — Rendering**
|
|
533
|
+
|
|
534
|
+
| Capability | Verdict | State |
|
|
535
|
+
|---|---|---|
|
|
536
|
+
| GFM markdown (hand-rolled, competitive) | MATCH | shipped |
|
|
537
|
+
| Syntax highlighting, vendored | MATCH | shipped |
|
|
538
|
+
| LaTeX / KaTeX | MATCH | open |
|
|
539
|
+
| Mermaid | MATCH (low priority) | open |
|
|
540
|
+
| Block-memoized streaming (no O(n²) re-parse) | BEAT | shipped |
|
|
541
|
+
| Sandboxed HTML / SVG preview with view-source | MATCH | shipped (Canvas v1) |
|
|
542
|
+
| Persistent Artifacts side panel | MATCH (opt-in) | open |
|
|
543
|
+
| logprobs confidence overlay | BEAT | open (server side exists) |
|
|
544
|
+
| Structured-output / grammar picker | BEAT | open (server side exists) |
|
|
545
|
+
|
|
546
|
+
**Axis 3 — Model discovery and management**
|
|
547
|
+
|
|
548
|
+
| Capability | Verdict | State |
|
|
549
|
+
|---|---|---|
|
|
550
|
+
| Hardware-fit verdict per model, measured for this Mac | BEAT | shipped (picker + Hub) |
|
|
551
|
+
| Browse / search / download / load in one panel | BEAT | shipped, except live load (restart command) |
|
|
552
|
+
| Capability badges (vision, quant tier) | MATCH | shipped (Hub chips) |
|
|
553
|
+
| In-chat model switch without restart | MATCH | open (no in-process swap seam) |
|
|
554
|
+
| Swap preserves session history | BEAT | open (same blocker) |
|
|
555
|
+
| Resident multi-model pool | MATCH | open |
|
|
556
|
+
| Running-models resource table with adapter / KV tier / draft columns | BEAT | open |
|
|
557
|
+
| Per-layer quantization sensitivity visualization | BEAT | open |
|
|
558
|
+
| `mlx-bun launch <tool>` one-command coding-tool wiring | BEAT | open (CLI) |
|
|
559
|
+
|
|
560
|
+
**Axis 4 — Prompts, presets, sampling**
|
|
561
|
+
|
|
562
|
+
| Capability | Verdict | State |
|
|
563
|
+
|---|---|---|
|
|
564
|
+
| Full sampler set in the UI | MATCH | shipped (`logit_bias` not exposed) |
|
|
565
|
+
| Per-model correct defaults | MATCH | shipped (`ready.genDefaults`) |
|
|
566
|
+
| Per-message sampling scope | MATCH | shipped (`next_turn`) |
|
|
567
|
+
| Saved presets (prompt + sampling) | MATCH | shipped v1 (local only) |
|
|
568
|
+
| Slash-command prompt library with variables | MATCH | open |
|
|
569
|
+
| Model-string variant shorthand (`:no-think`) | MATCH | open |
|
|
570
|
+
| Persona bundles bound to a real adapter swap | BEAT | open |
|
|
571
|
+
| Explicit "warm this session" | MATCH | open |
|
|
572
|
+
|
|
573
|
+
**Axis 5 — Documents and RAG**
|
|
574
|
+
|
|
575
|
+
| Capability | Verdict | State |
|
|
576
|
+
|---|---|---|
|
|
577
|
+
| Zero-config attach-and-answer | MATCH | shipped |
|
|
578
|
+
| Transparent inline-vs-retrieval decision | MATCH | shipped (threshold in `rag.ts`) |
|
|
579
|
+
| `[n]` citations + Sources panel | MATCH | shipped |
|
|
580
|
+
| Unified `#` mention over files + vault | MATCH | shipped |
|
|
581
|
+
| Per-document retrieval-mode toggle | MATCH | open |
|
|
582
|
+
| Hybrid BM25 + vector + rerank | SKIP (for now) | open, deferred tier |
|
|
583
|
+
| Semantic search over the user's own chat history | BEAT | partial (full-text body search shipped; semantic open) |
|
|
813
584
|
|
|
814
|
-
|
|
815
|
-
|
|
816
|
-
Each phase is independently shippable and includes the relevant tactical
|
|
817
|
-
fixes from `docs/archive/planning/web-ui-pass-plan.md` by number. Ordered by
|
|
818
|
-
effort/reward, starting with free wins, ending with differentiator
|
|
819
|
-
features. The [beat matrix](web-chat-beat-matrix.md) is the coverage
|
|
820
|
-
checklist across all phases: at any release point, every matrix row is
|
|
821
|
-
either shipped, in a scheduled phase below, or a written SKIP — a MATCH row
|
|
822
|
-
with none of those three states is a planning bug.
|
|
823
|
-
|
|
824
|
-
### Phase 0 — Bug fixes, one UI-only PR (no server changes, ~2-3 days)
|
|
825
|
-
|
|
826
|
-
Ship first, blocks nothing, touches no backend:
|
|
827
|
-
|
|
828
|
-
- `web-ui-pass-plan.md` #1: fine-tune live stats field-name mismatch
|
|
829
|
-
(`e.lr`/`e.tps` → `e.learning_rate`/`e.tokens_per_sec`).
|
|
830
|
-
- #2: queue-bar array-truthiness bug (`.length` guards, `app.html:1683-1684`).
|
|
831
|
-
- #4: `[object Object]` adapter-mount error (unwrap `data.error.message` in
|
|
832
|
-
`api()`, `app.html:1080`).
|
|
833
|
-
- #5: quantize inspect result parsing (`support: boolean`, not string enum).
|
|
834
|
-
- #6: stale "switch to 12B for vision" copy → model-agnostic.
|
|
835
|
-
- #15 (partial): escape `a.id`/`a.path` interpolation in the adapter
|
|
836
|
-
dropdown (`app.html:1897-1898`) — this is a live unescaped-HTML injection
|
|
837
|
-
point, fix immediately regardless of the larger adapter-UI rework in
|
|
838
|
-
Phase 2.
|
|
839
|
-
- #16: delete dead `src/status-page.html` + `scripts/status-page-stub.ts`.
|
|
840
|
-
- #18: CLI banner copy fix (`src/cli.ts:64`).
|
|
841
|
-
|
|
842
|
-
### Phase 1 — Free wins: wire existing backend, fix the render bottleneck (1-2 weeks)
|
|
843
|
-
|
|
844
|
-
The highest-leverage phase — almost all wiring of existing capability +
|
|
845
|
-
one architectural render fix. Two deliberate, thin server additions ride
|
|
846
|
-
along (scoped in §2.3's caveat): the per-turn `lane` field for the perf
|
|
847
|
-
strip, and the gc plan/execute endpoints (#9):
|
|
848
|
-
|
|
849
|
-
- **Streaming render fix** (§5.3): block-memoized append, replacing
|
|
850
|
-
`appendDelta`'s full re-parse. This unblocks every other visual change in
|
|
851
|
-
this phase since they all touch the same render path.
|
|
852
|
-
- Syntax highlighting (vendored Shiki or equivalent) + KaTeX, applied to
|
|
853
|
-
completed blocks only.
|
|
854
|
-
- Sampling popover: full parameter set (`min_p`, XTC, penalty families +
|
|
855
|
-
context windows, `logit_bias`, seed) behind an Advanced disclosure;
|
|
856
|
-
per-model `generation_config.json`-sourced defaults (fixes
|
|
857
|
-
`web-ui-pass-plan.md` #14 and #8). Wire end-to-end: popover → `set_sampling`
|
|
858
|
-
WS frame → `SamplingOverrides` → `injectSampling` (all three currently
|
|
859
|
-
hardcode 3 fields, `pi-web.ts:171-176`).
|
|
860
|
-
- Perf strip: tok/s, TTFT, context-fill, lane badge (serial/spec-decode/batched)
|
|
861
|
-
under the composer. Server side: add the per-turn `lane` field to the
|
|
862
|
-
usage payload + pi-web frame (see §2.3 caveat) — the badge must be
|
|
863
|
-
server-driven, never client-inferred (risk #5).
|
|
864
|
-
- Message actions: copy full assistant message, regenerate, edit-message-as-branch
|
|
865
|
-
with `< 1/2 >` toggle (reuses `SessionManager.forkFrom`).
|
|
866
|
-
- Adapter dropdown → chip with specialty description + `compatible` graying
|
|
867
|
-
(still single-select in this phase; full routing table is Phase 2).
|
|
868
|
-
- `web-ui-pass-plan.md` #3 (library staleness), #7 (`sft_scope` control),
|
|
869
|
-
#9 (gc affordance — needs the new `/api/gc/plan`/`/api/gc/execute`
|
|
870
|
-
endpoints), #11-13 (the queued `listCanonical`/`visionCapable`/`supportTier`
|
|
871
|
-
wiring, land as its own small PR once server-side lands), #20 (library
|
|
872
|
-
"too big" row fix), #21 documented as a known limitation.
|
|
873
|
-
- Mobile sidebar → slide-over drawer.
|
|
874
|
-
- Session search box.
|
|
875
|
-
- `prefers-color-scheme` + light theme + `prefers-reduced-motion` guards.
|
|
876
|
-
- `Cmd/Ctrl+/` shortcut sheet + ChatGPT-vocabulary bindings.
|
|
877
|
-
- `role="log" aria-live="polite"` on the thread, focus-trap/Escape on
|
|
878
|
-
popovers.
|
|
879
|
-
|
|
880
|
-
### Phase 2 — Module split, memory surface + adapter routing table (2-3 weeks)
|
|
881
|
-
|
|
882
|
-
The two headline differentiators, now that rendering is fixed and settings
|
|
883
|
-
have room — preceded by the structural fix that keeps the growing surface
|
|
884
|
-
honest:
|
|
885
|
-
|
|
886
|
-
- **Module split FIRST** (§7, amended): split `app.html` into
|
|
887
|
-
build-concatenated `src/web/*.ts` modules (one served file unchanged at
|
|
888
|
-
runtime) so the frontend imports `ServerMessage`/`ClientMessage` from
|
|
889
|
-
`src/pi-web.ts` and job/API shapes from their source modules — the
|
|
890
|
-
Phase-0 bug class (contract drift) becomes a compile error before the
|
|
891
|
-
memory panel and routing table add major new surface. Ship the frontend
|
|
892
|
-
regression harness (§7) in the same pass: formalized script-syntax gate,
|
|
893
|
-
`tests/web-app.test.ts` over the pure client functions (streaming
|
|
894
|
-
memoization vs. one-shot `mdToHtml` parity fixtures, `renderQueue`,
|
|
895
|
-
`api()` unwrapping), frontend modules in the same `tsc --noEmit` pass.
|
|
896
|
-
- **Per-chat system prompt** (beat matrix Axis 4 — table stakes in every
|
|
897
|
-
competitor, currently impossible: the web chat's prompt is hardcoded in
|
|
898
|
-
`buildPiAgentSurface`/`buildWebChatSystemPrompt`): an editable
|
|
899
|
-
system-prompt field per session, layered over the built-in surface prompt,
|
|
900
|
-
carried on the WS session; **presets v1** (named system-prompt + sampling
|
|
901
|
-
bundle, saved locally) shaped so Phase 5's persona bundles (adapter +
|
|
902
|
-
tools + memory scope) are a thin extension.
|
|
903
|
-
- **Session auto-titling** (Axis 11): v1 derives from the first user
|
|
904
|
-
message; model-generated titles optional and never queued ahead of real
|
|
905
|
-
turns.
|
|
906
|
-
- **Vault-init endpoint** (§5.1 prerequisite): the first-run consent card
|
|
907
|
-
creates the vault from the web UI — needs a thin
|
|
908
|
-
`POST /api/memory/init` wrapping what `mlx-bun memory init` does in the
|
|
909
|
-
CLI (today it's CLI-only and invisible to non-terminal users).
|
|
910
|
-
- Memory REST wrappers (prerequisite, server-side — see §2.3 caveat):
|
|
911
|
-
`GET /api/memory/status|list|search|article` + a git log/diff read,
|
|
912
|
-
thin delegating wrappers over `src/memory/vault.ts` / `tools.ts`
|
|
913
|
-
internals. The agent-tool surface stays read-only and unchanged.
|
|
914
|
-
- Memory panel (§5.5): vault browser, search, per-article git-diff view,
|
|
915
|
-
status strip, `Reference/` sub-view, wikilink graph via `memory_links`.
|
|
916
|
-
- Memory provenance chips on assistant turns that consulted memory tools
|
|
917
|
-
(distinct card style from generic tool calls, §5.4).
|
|
918
|
-
- Personalized hero/empty-state chips driven by `vaultStatus()` +
|
|
919
|
-
`/v1/adapters/available` (§5.1); inline memory-consent card for first run.
|
|
920
|
-
- Adapter routing table: three-state visibility (available/loaded/selected),
|
|
921
|
-
RAM cost per loaded adapter, `"a+b"` stacking support in the picker.
|
|
922
|
-
- Model switcher: clickable `#nav-model`, hardware-fit badge via `GET /fit`,
|
|
923
|
-
session-preserving swap.
|
|
924
|
-
- Tool-approval gate properly wired: `codingTools: true` opt-in toggle in
|
|
925
|
-
settings, editable arguments before approve/deny, durable tool-identity-keyed
|
|
926
|
-
"always allow" persistence.
|
|
927
|
-
- Structured-output picker (JSON-schema textarea + live validation) in the
|
|
928
|
-
Advanced disclosure.
|
|
929
|
-
- Unified `#` retrieval mention (files + vault as one picker).
|
|
930
|
-
- Information architecture move: collapse Quantize/Fine-tune/Build
|
|
931
|
-
Dataset/Status/Curves/Routes behind the Developer toggle (§8); fix
|
|
932
|
-
`web-ui-pass-plan.md` #17 (Routes 404 in compiled binary) and #19 (Curves
|
|
933
|
-
nav highlight) as part of this move.
|
|
934
|
-
|
|
935
|
-
### Phase 3 — Concede nothing: the table-stakes sweep (2-3 weeks)
|
|
936
|
-
|
|
937
|
-
(Added 2026-07-06 under the superset doctrine — the beat-matrix MATCH rows
|
|
938
|
-
a daily LM Studio / Open WebUI / optiq-Lab user would immediately miss.
|
|
939
|
-
Each item names its matrix axis.)
|
|
940
|
-
|
|
941
|
-
- **Chat-with-files RAG v1** (Axis 5): attach-and-answer over attached
|
|
942
|
-
documents with `[n]` citation markers + a sources panel — optiq Lab's
|
|
943
|
-
BM25 dependency-free bar is the v1 target; we have `src/embed.ts` for a
|
|
944
|
-
vector upgrade later, but don't block v1 on it. Extend the unified `#`
|
|
945
|
-
mention (Phase 2) so files and vault articles are one retrieval gesture.
|
|
946
|
-
- **Model Hub** (Axis 3): in-app browse (published quants / HF search /
|
|
947
|
-
local) → download with progress → one-click load-to-server + new-chat,
|
|
948
|
-
session-preserving hot-swap — optiq's Hub flow, PLUS the `/fit`
|
|
949
|
-
hardware-verdict column per row that optiq lacks. "Install to chat under
|
|
950
|
-
a minute" stops dead-ending at the CLI.
|
|
951
|
-
- **Canvas v1** (Axis 2): fenced-HTML blocks rendered in a sandboxed
|
|
952
|
-
iframe with a view-source toggle (optiq's shipped pattern; the full
|
|
953
|
-
Artifacts side-panel stays Phase 5).
|
|
954
|
-
- **Per-message sampling scope** (Axis 4): apply-to-next-turn-only option
|
|
955
|
-
in the sampling popover (optiq's per-message temp/max-tokens/thinking
|
|
956
|
-
granularity), on top of the session-level overrides from Phase 1.
|
|
957
|
-
- **Self-healing tool-call loop** (Axis 7): investigate what the pi SDK
|
|
958
|
-
already retries; add format-aware malformed-call repair (we own the
|
|
959
|
-
per-model parsers in `src/tool-call.ts`), consecutive-duplicate dedup, a
|
|
960
|
-
retry budget, and a loop cap with force-finish — small local models
|
|
961
|
-
malform calls more than cloud frontiers, so this matters more here than
|
|
962
|
-
it does for optiq.
|
|
963
|
-
- **Stop kills tool subprocesses** (Axis 1): verify/wire abort through the
|
|
964
|
-
pi tool layer so Stop ends running bash/web-fetch work, not just the
|
|
965
|
-
token stream.
|
|
966
|
-
- **Chat export** (Axis 10): Markdown/plain-text per session; portable
|
|
967
|
-
JSON export/import (sessions are already local JSON on disk).
|
|
968
|
-
- **Command palette** (Axis 10, `Cmd/Ctrl+K`): new chat / session search /
|
|
969
|
-
model + adapter switch / thinking toggle / theme.
|
|
970
|
-
- **Full-text session search** (Axis 10 BEAT): body search, not
|
|
971
|
-
title-only — no cloud indexing-cost excuse applies locally.
|
|
972
|
-
- **PWA installability** (Axis 10): manifest + shell service worker
|
|
973
|
-
(installability + instant shell, explicitly not offline chat).
|
|
974
|
-
- **App-aware assistant v1** (§6.6, matrix Axis 12 — promoted from Phase 4
|
|
975
|
-
2026-07-06: best wow-per-effort in this phase): `uiSnapshot` context push
|
|
976
|
-
on route change + a compact ambient context line (route/view/step)
|
|
977
|
-
auto-attached to every turn, `get_current_app_context` + `navigate_app` +
|
|
978
|
-
`spotlight_ui` tools on the pi-web bridge, curated route/anchor catalog,
|
|
979
|
-
hand-rolled spotlight overlay choreographed with the streaming
|
|
980
|
-
explanation. Pattern proven in `~/Code/PortfolioManager` on the same pi
|
|
981
|
-
SDK. The ambient every-view floating panel and process-state snapshot
|
|
982
|
-
enrichment mature in Phase 4; approval-gated `ui_act` stays Phase 5.
|
|
983
|
-
|
|
984
|
-
### Phase 4 — Trust & speed differentiators (1-2 weeks)
|
|
585
|
+
**Axis 6 — Memory and personalization**
|
|
985
586
|
|
|
986
|
-
|
|
987
|
-
|
|
988
|
-
|
|
989
|
-
|
|
990
|
-
|
|
991
|
-
|
|
992
|
-
|
|
993
|
-
|
|
994
|
-
|
|
995
|
-
|
|
996
|
-
|
|
997
|
-
|
|
998
|
-
|
|
999
|
-
|
|
1000
|
-
|
|
1001
|
-
|
|
1002
|
-
|
|
1003
|
-
|
|
1004
|
-
|
|
1005
|
-
|
|
1006
|
-
|
|
1007
|
-
|
|
1008
|
-
|
|
1009
|
-
-
|
|
1010
|
-
|
|
1011
|
-
|
|
1012
|
-
-
|
|
1013
|
-
|
|
1014
|
-
|
|
1015
|
-
|
|
1016
|
-
|
|
1017
|
-
|
|
1018
|
-
|
|
1019
|
-
-
|
|
1020
|
-
|
|
1021
|
-
|
|
1022
|
-
|
|
1023
|
-
|
|
1024
|
-
|
|
1025
|
-
|
|
1026
|
-
|
|
1027
|
-
|
|
1028
|
-
|
|
1029
|
-
|
|
1030
|
-
|
|
1031
|
-
|
|
1032
|
-
which neither PortfolioManager (stops at navigate+point) nor generic
|
|
1033
|
-
computer-use agents (act with no in-app trust model) ship. Requires
|
|
1034
|
-
Phase 2's approval surface + Phase 4's v1.
|
|
587
|
+
| Capability | Verdict | State |
|
|
588
|
+
|---|---|---|
|
|
589
|
+
| Visible, searchable memory (vault browser) | BEAT | shipped |
|
|
590
|
+
| Per-turn provenance chip | BEAT | shipped |
|
|
591
|
+
| Git history / diff of what synthesis changed | BEAT | shipped |
|
|
592
|
+
| Wikilink graph view | BEAT | open (links line only) |
|
|
593
|
+
| Scope control (session / surface / global) | MATCH | open |
|
|
594
|
+
| Pause vs reset as distinct actions | MATCH | open |
|
|
595
|
+
| Memory graduating into trained weights, marked in UI | BEAT | roadmap — never claim early |
|
|
596
|
+
| First-run consent in the hero | BEAT | shipped |
|
|
597
|
+
| Synthesis schedule status visible | MATCH | shipped (status strip) |
|
|
598
|
+
| Self-evolving procedural "Skills" | SKIP (v1) | open, deferred |
|
|
599
|
+
|
|
600
|
+
**Axis 7 — Tools and agents**
|
|
601
|
+
|
|
602
|
+
| Capability | Verdict | State |
|
|
603
|
+
|---|---|---|
|
|
604
|
+
| Tool calls as distinct cards | MATCH | shipped |
|
|
605
|
+
| Inline non-blocking approval cards | MATCH | shipped |
|
|
606
|
+
| Editable arguments before approval | MATCH | shipped |
|
|
607
|
+
| Durable tool-identity-keyed always-allow | BEAT | shipped (`src/tool-approvals.ts`) |
|
|
608
|
+
| Gated tools reachable in web chat (opt-in) | MATCH | shipped |
|
|
609
|
+
| Sandboxed Python tool | MATCH | open |
|
|
610
|
+
| Self-healing malformed tool calls | MATCH | shipped (`src/tool-call.ts`) |
|
|
611
|
+
| Loop hygiene (dedup, retry budget, cap, force-finish) | MATCH | shipped |
|
|
612
|
+
| Argument / diff preview before approval | BEAT | shipped (args); file diff open |
|
|
613
|
+
| Per-session disk-touch tally | BEAT | open |
|
|
614
|
+
| Curated default tool pack on first run | MATCH | partial (read, web_search, memory, app-aware) |
|
|
615
|
+
| MCP client | MATCH | open |
|
|
616
|
+
| Web search + URL fetch without keys | MATCH | `web_search` shipped; `web_fetch` not in the web allowlist |
|
|
617
|
+
| Approval-fatigue-aware defaults (read-only auto-allowed) | MATCH | shipped |
|
|
618
|
+
| `tool_choice` forcing in the chat UI | SKIP | API only by design |
|
|
619
|
+
| Visual agent-flow builder | SKIP | not planned |
|
|
620
|
+
| Deep-research trace panel | MATCH | open |
|
|
621
|
+
|
|
622
|
+
**Axis 8 — Multimodal**
|
|
623
|
+
|
|
624
|
+
| Capability | Verdict | State |
|
|
625
|
+
|---|---|---|
|
|
626
|
+
| Image attach + inference | MATCH | shipped (gated by `ready.vision`) |
|
|
627
|
+
| No text-only tax from the vision path | MATCH | verify |
|
|
628
|
+
| Image generation | SKIP | no substrate |
|
|
629
|
+
| Speech-to-speech voice | SKIP | no substrate |
|
|
630
|
+
| Native OS dictation / TTS | SKIP | no substrate |
|
|
631
|
+
| Document attach (PDF / DOCX) with inline-vs-RAG | MATCH | text files shipped; binary formats open |
|
|
632
|
+
| Global hotkey + floating companion window with screenshot | BEAT | open (needs a desktop shell) |
|
|
1035
633
|
|
|
1036
|
-
|
|
634
|
+
**Axis 9 — Performance and transparency**
|
|
1037
635
|
|
|
1038
|
-
|
|
1039
|
-
|
|
1040
|
-
|
|
1041
|
-
|
|
1042
|
-
-
|
|
1043
|
-
|
|
1044
|
-
|
|
1045
|
-
|
|
1046
|
-
|
|
1047
|
-
|
|
1048
|
-
-
|
|
1049
|
-
|
|
1050
|
-
|
|
1051
|
-
|
|
1052
|
-
|
|
1053
|
-
|
|
1054
|
-
|
|
1055
|
-
|
|
1056
|
-
|
|
1057
|
-
|
|
1058
|
-
|
|
1059
|
-
|
|
1060
|
-
|
|
1061
|
-
|
|
1062
|
-
|
|
1063
|
-
|
|
1064
|
-
|
|
1065
|
-
|
|
1066
|
-
|
|
1067
|
-
|
|
1068
|
-
|
|
1069
|
-
|
|
1070
|
-
|
|
1071
|
-
|
|
1072
|
-
|
|
1073
|
-
|
|
1074
|
-
|
|
1075
|
-
|
|
1076
|
-
|
|
1077
|
-
|
|
1078
|
-
|
|
1079
|
-
|
|
1080
|
-
|
|
1081
|
-
|
|
1082
|
-
|
|
1083
|
-
|
|
1084
|
-
|
|
1085
|
-
|
|
1086
|
-
|
|
1087
|
-
|
|
1088
|
-
|
|
1089
|
-
|
|
1090
|
-
|
|
1091
|
-
|
|
1092
|
-
|
|
1093
|
-
|
|
1094
|
-
|
|
1095
|
-
|
|
1096
|
-
|
|
1097
|
-
|
|
1098
|
-
correction can fail by attempting everything at once. Mitigation: the
|
|
1099
|
-
phases stay independently shippable and strictly ordered
|
|
1100
|
-
(fundamentals → structure → table-stakes sweep → trust/speed →
|
|
1101
|
-
stretch); within Phase 3 the sweep is ordered by "what would a daily
|
|
1102
|
-
competitor user miss first"; the matrix's shipped/scheduled/SKIP states
|
|
1103
|
-
are re-audited at each phase boundary rather than renegotiated
|
|
1104
|
-
mid-phase.
|
|
1105
|
-
9. **(Added 2026-07-06) The untested, untyped frontend erodes as it
|
|
1106
|
-
grows.** Phase 0's bug class (contract drift between inline JS and
|
|
1107
|
-
typed TS) recurs with every new surface until the module split lands.
|
|
1108
|
-
Mitigation: the split + frontend regression harness are Phase 2's
|
|
1109
|
-
FIRST work item (§7), before the memory panel and routing table add
|
|
1110
|
-
the next thousand lines; until then, every new WS frame or API shape
|
|
1111
|
-
consumed by `app.html` gets a line in `tests/pi-web.test.ts` proving
|
|
1112
|
-
the producing side's contract.
|
|
636
|
+
| Capability | Verdict | State |
|
|
637
|
+
|---|---|---|
|
|
638
|
+
| Live tok/s + TTFT | MATCH | shipped |
|
|
639
|
+
| Context-window meter | MATCH | shipped |
|
|
640
|
+
| Serving-lane badge in the live chat | BEAT | shipped |
|
|
641
|
+
| Prompt-cache-hit telemetry per turn | BEAT | open |
|
|
642
|
+
| In-app benchmark tab | MATCH | open (harness exists: `scripts/bench-serve.ts`) |
|
|
643
|
+
| Compare / split view (base vs adapter, parity vs fast path) | BEAT | open |
|
|
644
|
+
| Concurrency / batch-slot indicator | MATCH | open |
|
|
645
|
+
| `/fit` surfaced beyond Status | BEAT | shipped (picker, Hub) |
|
|
646
|
+
| Auto-sized context to available memory | BEAT | open (UI); admission clamps server-side |
|
|
647
|
+
| Provable zero-cloud-calls indicator | BEAT | open |
|
|
648
|
+
|
|
649
|
+
**Axis 10 — Polish**
|
|
650
|
+
|
|
651
|
+
| Capability | Verdict | State |
|
|
652
|
+
|---|---|---|
|
|
653
|
+
| Dark + light, `prefers-color-scheme` | MATCH | shipped |
|
|
654
|
+
| `prefers-reduced-motion` | MATCH | shipped |
|
|
655
|
+
| Shortcut sheet (Cmd/Ctrl+/) | MATCH | shipped |
|
|
656
|
+
| Command palette (Cmd/Ctrl+K) | MATCH | shipped |
|
|
657
|
+
| Responsive to 375 px with a drawer | MATCH | shipped |
|
|
658
|
+
| PWA installability | MATCH | shipped |
|
|
659
|
+
| `aria-live` thread, focus traps, Escape | MATCH | shipped |
|
|
660
|
+
| Developer progressive disclosure | MATCH | shipped |
|
|
661
|
+
| Server-driven personalized onboarding chips | BEAT | shipped |
|
|
662
|
+
| Chat export (Markdown / JSON) | MATCH | shipped |
|
|
663
|
+
| Shareable links | SKIP | scope decision (single-user, local) |
|
|
664
|
+
| Full-text search across message bodies | BEAT | shipped |
|
|
665
|
+
| Session pins / folders | MATCH | open |
|
|
666
|
+
| Clean quit / no zombie process in a desktop shell | MATCH | n/a until a shell exists |
|
|
667
|
+
|
|
668
|
+
**Axis 11 — Sessions**
|
|
669
|
+
|
|
670
|
+
| Capability | Verdict | State |
|
|
671
|
+
|---|---|---|
|
|
672
|
+
| New / list / open / fork / delete | MATCH | shipped |
|
|
673
|
+
| Session search box | MATCH | shipped |
|
|
674
|
+
| Per-message branch-from-here | MATCH | shipped (edit / regenerate siblings) |
|
|
675
|
+
| Projects (scoped knowledge + memory + adapter) | BEAT | open (persona bundles) |
|
|
676
|
+
| Temporary / incognito chat, provably not on disk | BEAT | open |
|
|
677
|
+
| Multi-adapter compare | BEAT | open |
|
|
678
|
+
| Portable JSON export / import | MATCH | export shipped; import open |
|
|
679
|
+
|
|
680
|
+
**Axis 12 — App-aware assistant**
|
|
681
|
+
|
|
682
|
+
| Capability | Verdict | State |
|
|
683
|
+
|---|---|---|
|
|
684
|
+
| Agent sees the live UI (structured snapshot) | BEAT | shipped |
|
|
685
|
+
| Agent navigates the app | BEAT | shipped |
|
|
686
|
+
| Agent spotlights a control | BEAT | shipped |
|
|
687
|
+
| Approval-gated agent UI actions (`ui_act`) | BEAT (stretch) | open |
|
|
688
|
+
|
|
689
|
+
**The beats** (ranked by visible wow × structural defensibility): lane
|
|
690
|
+
badge in live chat · memory git-diff view · per-turn provenance chip ·
|
|
691
|
+
legible, stackable adapter routing · base-vs-adapter compare view · model
|
|
692
|
+
swap that preserves the session · prompt-cache-hit telemetry · provably
|
|
693
|
+
local temporary chat · diff preview before approval · full-text body
|
|
694
|
+
search · the app-aware assistant. The first four and the last two are
|
|
695
|
+
shipped; the rest are §5.
|