pi-lens 3.8.69 → 3.8.71

package/docs/lsp-capability-matrix.md

@@ -0,0 +1,157 @@
+# LSP capability matrix — affirmative-clean-signal strategy
+
+How pi-lens knows a just-edited file is **clean** (no diagnostics) vs the server
+simply **hasn't answered yet** (cold/crashed/silent). pi-lens waits *synchronously*
+for a verdict, so — unlike an editor, which renders asynchronously and never needs
+to decide — it must have a positive signal. There is no signal for "silence", so we
+classify each server and pick a per-server strategy. (Background: #240; mechanism
+confirmed against Neovim's LSP client, which sidesteps this entirely by being async.)
+
+Generate/refresh this matrix with `node scripts/characterize-lsp.mjs [--install]`
+(the `mode` column) and `node scripts/probe-clean-signal.mjs [--install]` (the
+`clean-behavior` column — 4-way: 2 / 2* / 3 / unknown). Both **merge** in place: a server the
+running host couldn't spawn keeps its prior row, so an ubuntu-poor run can't
+regress a richer one (#390). The nightly **tool-smoke** workflow runs both (plus
+`server-capabilities.mjs`) and opens/updates a single auto-PR
+(`bot/lsp-docs-refresh`, "docs(nightly): refresh LSP capability docs") with the
+regenerated docs — so this file self-populates from CI without manual copy-paste.
+
+`probe-clean-signal.mjs` also runs a **drift check** (#529): it compares each
+probed server's observed `clean-behavior` against the hand-set `silentOnClean`
+marker in `clients/lsp/server-strategies.ts` and writes any mismatch to the
+`## silentOnClean drift` section below. This is telemetry only — never a CI
+gate — because the probe is a timing-based negative observation; a mismatch
+just tells a human the marker may need updating. `unknown` observations are
+never compared in either direction (a slow/absent server isn't evidence of
+anything). The native TS7 launch variant (`typescript7`/`typescript7-clean`,
+#524/#526) is deliberately excluded from comparison against classic's marker —
+they share a server id but not a verified clean-signal behavior.
+
+## The strategies
+
+| Tier | Signal | Affirmative clean? | Example |
+|---|---|---|---|
+| **1 — pull** | `textDocument/diagnostic` returns an authoritative report (empty = clean) | YES, deterministic | rust-analyzer |
+| **2 — push, publishes-versioned** | `publishDiagnostics([])` **with version** on every scan, incl. clean→clean | YES, currency-proven via version | ast-grep |
+| **2\* — push, publishes-unversioned** | re-publishes on a clean scan but **version-less** — the wait still early-returns (the client accepts a version-less publish as fresh: it can't be proven stale), but currency is only *temporally correlated*, not proven | YES at runtime, with a staleness-risk caveat (not a latency cost) | opengrep |
+| **3 — push, silent on clean** | server publishes nothing when nothing changed | **NO** — budget-wait floor (safe; a timeout is *not* a false clean). **This tier is #458's learned-deadline target set.** | typescript-language-server |
+
+Detection is **cached** at `initialize` (`detectWorkspaceDiagnosticsSupport` →
+`state.workspaceDiagnosticsSupport.mode`, upgraded on `client/registerCapability`),
+so the tier is free at collection time — no per-edit probe.
+
+## Matrix (dev box + CI nightly; mode last refreshed 2026-06-17 from run 27713958681, clean-behavior probed on the dev box 2026-07-08 — #460)
+
+`mode` from cached capabilities; `clean-behavior` from the phase-aware publish-trace
+probe. The probe attributes publishes to two phases — the **dirty touch** (proves the
+server is live) and the **clean transitions** (the discriminator) — and classifies
+4-way: `publishes-versioned` (tier 2: publish WITH version on a clean transition —
+affirmative + currency-proven), `publishes-unversioned` (tier 2\*: version-less
+publish on a clean transition — the wait still early-returns at runtime, currency
+only temporally correlated), `silent` (tier 3: alive on dirty, silent on clean —
+budget-wait, the #458 target), `unknown` (no publish at all — slow/absent,
+conservatively not classified). `src` = where a row was measured: **ci** = the
+nightly steps, **dev** = the dev box (a row measured on both reads `dev+ci`).
+Merges never blank a prior good value, so a CI non-result leaves the dev
+classification standing.
+
+| lang | server | mode | clean-behavior | tier | src |
+|---|---|---|---|---|---|
+| json | vscode-json-language-server | pull | — | 1 | dev+ci |
+| css | vscode-css-language-server | pull | — | 1 | dev+ci |
+| html | vscode-html-language-server | pull | — | 1 | dev+ci |
+| rust | rust-analyzer | pull | — | 1 | dev |
+| svelte | svelte-language-server | pull | — | 1 | dev+ci |
+| deno | deno (alt of typescript) | pull | — | 1 | dev+ci |
+| ruby | ruby-lsp | pull | — | 1 | ci |
+| csharp | csharp-ls | pull | — | 1 | ci |
+| typescript | typescript-language-server | push-only | silent | 3 | dev+ci |
+| python | pyright | push-only | publishes-versioned | 2 | dev+ci |
+| jedi | jedi-language-server (alt of python) | push-only | publishes-versioned | 2 | ci |
+| yaml | yaml-language-server | push-only | publishes-unversioned | 2* | dev+ci |
+| shell | bash-language-server | push-only | publishes-versioned | 2 | dev+ci |
+| dockerfile | docker-langserver | push-only | publishes-unversioned | 2* | dev+ci |
+| toml | taplo | push-only | publishes-unversioned | 2* | dev+ci |
+| terraform | terraform-ls | push-only | TBD | 2/3? | dev+ci |
+| prisma | @prisma/language-server | push-only | publishes-unversioned | 2* | dev+ci |
+| php | intelephense | push-only | TBD | 2/3? | dev |
+| zig | zls | push-only | publishes-unversioned | 2* | dev+ci |
+| vue | @vue/language-server | push-only | TBD | 2/3? | dev+ci |
+| dart | dart language-server | push-only | publishes-unversioned | 2* | ci |
+| gleam | gleam lsp | push-only | publishes-unversioned | 2* | ci |
+| clojure | clojure-lsp | push-only | publishes-unversioned | 2* | ci |
+| opengrep | opengrep (aux) | push-only | publishes-unversioned | 2* | dev+ci |
+| ast-grep | ast-grep (aux) | push-only | publishes-versioned | 2 | dev+ci |
+
+**Unknown — fixture exists, mode not yet captured.** The toolchain-gated family
+(no auto-install today; tracked in #241) — `go` (gopls), `java` (jdtls),
+`kotlin`, `swift` (sourcekit-lsp), `lua`, `cpp` (clangd), `haskell`, `elixir`,
+`ocaml`, `nix` (nixd), `fsharp`. Their servers don't install in the nightly, so
+characterize reports `unknown` (a non-failure ⚠). Once #241 lands they'll fill in
+the same way clojure-lsp/gleam now do (both auto-install via the github strategy
+and were characterized `push-only` in the run above).
+
+## Key findings
+- **Mode ≠ tier, and the split needs BOTH axes.** Push-only further splits along
+  latency (does anything publish on a clean transition? — silence is the only
+  budget-wait case, because pi-lens's publish handler emits and early-returns the
+  wait on EVERY publish, versioned or not) and currency-proof (is the publish
+  versioned, i.e. provably about the live edit?). The 4-way
+  `probe-clean-signal.mjs` measurement drives this: ast-grep → 2, yaml/opengrep →
+  2\*, typescript (clean file) → 3.
+- **opengrep is 2\*, not 3 and not plain 2.** It *does* re-publish on a clean scan
+  (the wait early-returns at runtime — fast), but every push carries
+  `pubVersion=undefined`, so currency is only temporally correlated, not proven —
+  a staleness-risk note, not a latency cost. An earlier hand-note called it Tier 2
+  on the "re-publishes" observation alone; the phase-aware probe refines it to 2\*.
+- **typescript's clean behavior is diagnostic-set-dependent (major probe finding).**
+  On a DIRTY file it re-publishes (version-lessly) after every change — the dirty
+  fixture measures 2\*. On a genuinely CLEAN file (the `typescript-clean` fixture)
+  it publishes nothing on a clean→clean edit — silent, Tier 3. The clean-file
+  behavior is the production case (the observed budget-wait timeouts), so the
+  matrix row records the clean fixture's verdict; the probe prefers `clean: true`
+  fixtures for exactly this reason. Corollary: a 2\* measured only on a dirty
+  fixture may overstate a server whose publishes stop when its set goes empty —
+  langs without a clean fixture carry that caveat.
+- **#458's learned-deadline target set = the tier-3 rows only.** 2\* rows resolve
+  the wait at runtime and must NOT be given learned deadlines.
+- **Tier 3 is budget-bound by necessity**, not laziness: a silent server's silence is
+  ambiguous (clean-unchanged vs still-analyzing), so shortening the wait or reusing
+  `lastKnownDiagnostics` would risk a false clean. The wait *is* the safety mechanism.
+- **ast-grep (Phase 2 / #239) is Tier 2** — it self-signals clean on every scan, so it
+  is not the bottleneck. The cost on a clean with-auxiliary touch is the *silent
+  primary* (typescript), a pre-existing Tier-3 cost independent of ast-grep.
+
+## Completing the matrix
+The fixtures (`tests/fixtures/tool-smoke/<lang>/`) are durable and cover every
+registered server. `mode` is read from the server's advertised capabilities at
+`initialize`, so it is **content-independent** — for languages that already had a
+tool-layer fixture we point `characterize-lsp.mjs` at the existing (deliberately
+dirty) `bad.*` source rather than a colliding clean duplicate; new languages get a
+minimal clean source. Either way the mode reported is the same.
+
+The nightly **tool-smoke** workflow runs `characterize-lsp.mjs --install` **and**
+`probe-clean-signal.mjs --install` (after the LSP handshake layer) on `ubuntu-latest`,
+then opens/updates an auto-PR with the regenerated docs (#390) — so both the `mode`
+and `clean-behavior` columns self-populate in CI without manual copy-paste. They fill
+for servers that either auto-install (npm/pip/github — including clojure-lsp and gleam,
+both github-strategy as of f263cf3) or whose toolchain the workflow provisions
+(Ruby/Dart/Zig + .NET→csharp). The remaining `unknown` rows are the toolchain-gated
+family (#241): until `runtimeInstall` + canonical-bin discovery land, their servers
+don't install in CI and stay ⚠. The **merge guard** means a CI run that can't reach a
+server never blanks its dev-measured row.
+
+The `clean-behavior` split (4-way: 2 publishes-versioned / 2\* publishes-unversioned /
+3 silent / unknown) is now measured per-server by `probe-clean-signal.mjs`, no longer a
+manual one-off. Locally confirmed: ast-grep + ast-grep-baseline → 2, yaml + opengrep →
+2\*, typescript → 3 on its clean fixture (2\* on the dirty fixture — see Key findings);
+the rest fill in as the nightly reaches them.
+
+## silentOnClean drift (nightly-generated)
+
+Telemetry only — never a CI gate. Compares each probed server's observed
+`clean-behavior` against `clients/lsp/server-strategies.ts`'s `silentOnClean`
+marker; a mismatch means the marker may need a human update (#529). `unknown`
+observations are never compared (a slow/absent server is not evidence either way).
+
+_None observed as of the last probe run._

package/docs/lsp-latency-benchmark.md

@@ -0,0 +1,169 @@
+# LSP latency benchmark
+
+Cold/warm per-edit latency for pi-lens's registered language servers, measured to
+gate the ast-grep LSP consolidation (#239 Gate A: an auxiliary scanner is
+acceptable as long as its warm latency is in the range of the primary language
+servers pi-lens already tolerates).
+
+- **Reproduce:** `npm run build:dist && node scripts/bench-lsp.mjs [lang …] [--install]`
+- **Machine:** Windows 11, dev box, warm npm cache. Numbers are **indicative, not
+  absolute** — single machine, small fixtures, one run.
+- **cold** = first touch: spawn + `initialize` handshake + first scan.
+- **warm** = re-touch after a content edit (the per-edit latency the agent waits on),
+  averaged over 3 edits.
+- Source fixtures: `tests/fixtures/tool-smoke/*` (driven through the real
+  `LSPService.touchFile`, same entry the dispatch path uses).
+- **Isolation (since #242):** each fixture measures **one server alone**. The
+  service + workspace config are reset between fixtures (no lingering server from a
+  previous measurement), and every *other* server matching the file is disabled —
+  so an auxiliary number is the auxiliary by itself (primaries off) and a primary
+  number is the default by itself (alternates + auxiliaries off). This removes the
+  primary↔auxiliary cross-contamination that produced the old "20557ms" artifact
+  (see the resolved-history note below).
+
+> **Coverage:** this run measured **31 servers** (27 primary/alternate + 4
+> auxiliary) — those whose toolchain is installed on this box. **Newly measuring
+> since the previous run:** **clangd** (cpp) and **zizmor** (auxiliary) now
+> auto-install via the archive/github strategies (#241/#272), **PowerShell Editor
+> Services** via its pwsh-bootstrapped bundle (#278), and **marksman** (markdown,
+> #274). **11** fixtures remain toolchain-gated and reported `unavailable` here
+> (csharp-ls, jdtls [java + java-lombok], kotlin-language-server, sourcekit-lsp,
+> dart, lua-language-server, haskell-language-server, elixir-ls, ocamllsp, nixd) —
+> run with `--install` on a box with their runtimes to measure them (#241).
+>
+> **Most fixtures are intentionally broken** (they carry a known defect so the smoke
+> harness can assert a diagnostic). That means such a server always has a diagnostic
+> to early-return on — see the clean-file note below, which a broken fixture masks.
+> The `typescript-clean` fixture exists precisely to expose that cost.
+
+## Results (2026-06-21 run, isolated)
+
+Sorted by warm/edit. `role`: primary = the file's language server; alternate =
+second server for a language (reached when the default is disabled); auxiliary =
+cross-cutting, attaches alongside the primary in production but measured alone here.
+
+| lang | role | cold | warm/edit | server |
+|---|---|---:|---:|---|
+| ruby | primary | 5608ms | 159ms | ruby-lsp |
+| deno | alternate | 1521ms | 165ms | deno (alternate of typescript) |
+| **ast-grep-baseline** | auxiliary | 1906ms | **394ms** | ast-grep (no-sgconfig baseline) |
+| **ast-grep** | auxiliary | 1614ms | **399ms** | ast-grep |
+| dockerfile | primary | 1715ms | 470ms | docker-langserver |
+| toml | primary | 1622ms | 471ms | taplo |
+| go | primary | 1777ms | 473ms | gopls |
+| prisma | primary | 1935ms | 473ms | @prisma/language-server |
+| zig | primary | 1237ms | 496ms | zls |
+| yaml | primary | 1908ms | 506ms | yaml-language-server |
+| python | primary | 2025ms | 520ms | pyright |
+| rust | primary | 1015ms | 576ms | rust-analyzer |
+| gleam | primary | 1219ms | 585ms | gleam lsp |
+| typescript | primary | 1811ms | 605ms | typescript-language-server |
+| **cpp** | primary | 983ms | 607ms | clangd (#241) |
+| clojure | primary | 5496ms | 624ms | clojure-lsp |
+| **opengrep** | auxiliary | 2156ms | **651ms** | opengrep |
+| css | primary | 2204ms | 967ms | vscode-css-language-server |
+| json | primary | 2014ms | 972ms | vscode-json-language-server |
+| html | primary | 2234ms | 981ms | vscode-html-language-server |
+| shell | primary | 2327ms | 1012ms | bash-language-server |
+| svelte | primary | 3311ms | 1028ms | svelte-language-server |
+| typescript-clean | primary | 1827ms | 1043ms | typescript-language-server (clean file) |
+| php | primary | 1703ms | 1167ms | intelephense |
+| jedi | alternate | 2477ms | 1278ms | jedi (alternate of pyright) |
+| **powershell** | primary | 5053ms | 1401ms | PowerShell Editor Services (#278) |
+| markdown | primary | 2979ms | 1725ms | marksman (#274) |
+| terraform | primary | 2331ms | 1729ms | terraform-ls |
+| fsharp | primary | 2958ms | 1730ms | fsautocomplete (`dotnet tool`, #241) |
+| vue | primary | 5545ms | 1731ms | @vue/language-server |
+| **zizmor** | auxiliary | 962ms | **2158ms†** | zizmor (#272) |
+
+**Primary/alternate warm:** min 159ms · avg 870ms · max 1731ms (n=27).
+**Auxiliary warm:** ast-grep 399ms · ast-grep-baseline 394ms · opengrep 651ms —
+all at the **fast end** of the primary band. **zizmor 2158ms†** is a bench
+artifact, not its real per-edit cost — see the caveat below; its true warm/edit is
+**~680ms**, also fast-end.
+
+> **† zizmor caveat (the bench number is a no-signal artifact).** The harness's
+> warm edit appends `\n// bench edit N\n`, but `//` is **not** a YAML comment
+> (`#` is) — so each warm touch hands zizmor an *unparseable workflow*. zizmor
+> collects no auditable input and publishes nothing, so the touch has no diagnostic
+> to early-return on and runs to its `aggregateWaitMs` budget (the same class of
+> "no-signal, budget-bound" cost as a clean file — see below). Re-measured with a
+> *valid* edit (appending a `#` comment so the workflow stays parseable + still
+> flagged), zizmor early-returns on the version-matched republish:
+>
+> | edit kind | cold | warm/edit | diags |
+> |---|---:|---:|---:|
+> | bench `//` (unparseable) | 962ms | 2158ms | 1 cold, 0 warm |
+> | valid `#` (parseable, flagged) | 2376ms | **682ms** | 1 every touch |
+>
+> So zizmor's real per-edit latency sits with opengrep (~650ms), not at 2158ms.
+
+> **fsautocomplete caveat:** it returned **0 diagnostics** (the fixture isn't a
+> restored .NET project, so the server surfaces nothing), so its 1730ms is the
+> *clean-file* near-budget cost — see the clean-file note below — not a real
+> per-edit-with-diagnostic latency. It is a heavy server regardless; this number is
+> an upper bound, not a typical edit.
+
+### Auxiliary verdict (#239 Gate A, re-confirmed #272)
+**Pass, decisively.** Measured in isolation, every auxiliary's real warm latency is
+at or below the median primary (ast-grep 399ms, opengrep 651ms, zizmor ~682ms vs
+primary avg 870ms). None regresses the hot path.
+
+### Shared with-auxiliary floor on YAML (#272 re-measure)
+Editing a workflow YAML now attaches **three** auxiliaries alongside the yaml
+primary — opengrep, ast-grep, **and zizmor** (all include YAML in their kind set).
+The concern (#272) was that each new auxiliary joins the same `with-auxiliary`
+`Promise.all` and could extend the per-edit floor. It does **not**, because of the
+#242 fix: each attached server gets its **own** deadline bounded by the caller cap
+as a ceiling (`min(callerCap, aggregateWaitMs)`), and they run concurrently — so
+the touch waits for the *slowest single* aux, not their sum. opengrep
+(`min(2500, 3500) = 2500`) already sets that ceiling; zizmor
+(`min(2500, 2000) = 2000`) sits under it and never extends the floor. On a normal
+(parseable) edit all three early-return well inside the cap (~400–680ms). zizmor's
+`aggregateWaitMs` is intentionally left at 2000ms for online-mode (GitHub-API)
+headroom; a late online finding is cached and surfaces on the next edit.
+
+## Resolved: the with-auxiliary measurement artifact (#240 + #242)
+
+An earlier full-suite bench reported **ast-grep warm = 20557ms** (and ~2000–3700ms
+at lower caps). That was **never ast-grep being slow** — it was two compounding bugs,
+both now fixed:
+
+1. **The benchmark co-spawned the primary.** An auxiliary fixture was touched with
+   `clientScope: "with-auxiliary"`, which spawns the file's primary language server
+   *and* the auxiliary, then waits for **all** of them (`Promise.all`). The fixtures
+   are clean JS with no TypeScript errors, so the primary (typescript, push-silent on
+   a clean file) never published and never early-returned — the touch ran to the
+   deadline even though ast-grep published in ~0.5s. **Fix:** the bench now disables
+   the primary and measures each server alone (this doc, isolation note above).
+2. **The with-auxiliary deadline was a floor, not a ceiling.** The collection used
+   `timeoutMs = max(callerCap, maxStrategyWait)` for *every* attached server, so a
+   large cap *became* the deadline and a slow auxiliary could override the per-edit
+   cap. **Fix (#242):** each server now gets its own deadline bounded by the caller
+   cap as a ceiling — `min(callerCap, strategyWait)` — so a clean/push-silent primary
+   can no longer hold the touch and an auxiliary can't blow the per-edit budget.
+   ast-grep's `aggregateWaitMs` was also raised 1000 → 1800 (its scan is ~1.3s; 1000
+   was under-budgeted, masked before only by the global floor).
+
+Correctness of the early-return signal was hardened separately in **#240**: pull
+diagnostics now return a discriminated `found | clean | unavailable` outcome and a
+failed pull (dead/null/threw) is **never** read as clean — only an affirmative,
+version-matched publish (or authoritative empty pull) ends the wait early. A
+crashed/cold/stale/errored server is never painted "clean".
+
+### Clean-file cost is real and irreducible for push-silent servers
+A server that processes an edit and finds **no new diagnostics** gives the wait
+nothing to trip on, so a clean file costs more than a broken one on the *same* server:
+
+| fixture | warm/edit |
+|---|---:|
+| typescript (broken — persistent error, re-published each edit) | 605ms |
+| typescript-clean (no diagnostics) | 1043ms |
+
+The clean file is ~1.7× slower: with a persistent diagnostic the server re-publishes
+(version bump → early-return); with nothing to report, a Tier-3 push-silent server
+(typescript publishes nothing on clean) gives no signal, so the touch waits its
+budget. This is **budget-bound by necessity** — silence is irreducibly ambiguous, so
+the cap is the only safe backstop. Tier-2 servers that re-publish empty-with-version
+(ast-grep, opengrep) *do* early-return on clean files, which is why both measure fast
+above even though their fixtures could be clean on a given edit.