specrails-desktop 2.8.0 → 2.9.0

package/docs/internals/adding-a-provider.md

@@ -1,19 +1,30 @@
 # Adding a new AI provider to Specrails
 
+> Last verified against `main` (app 2.8.0), which ships **three** registered
+> providers: **Claude, Codex, and Gemini**. Gemini is the freshest worked
+> example — `server/providers/gemini-adapter.ts` is the cleanest single-file
+> template to copy.
+
 The app is provider-agnostic by design. Every manager that spawns an AI
 CLI consumes a `ProviderAdapter` rather than branching on a hardcoded
-`if (provider === 'claude')`. Adding a new provider is mostly one adapter
-file plus one entry in the registry — but the codebase still carries a
-handful of `'claude' | 'codex'` type unions and two hardcoded provider
-lists that you must widen by hand today. Those are tracked in the
-**Type-union widening** and **Manual wiring still required** sections
-below.
+`if (provider === 'claude')`. Adding a provider is, in the ideal case,
+**one adapter file plus one registry line** — and that ideal is now real:
+`ProviderId` is just `string` (no `'claude' | 'codex'` union to widen),
+and the provider-discovery routes are registry-driven, so a newly-registered
+adapter surfaces in the API and the Add Project UI with **zero** further edits.
+You only fill a few non-adapter seams that are inherently provider-specific
+(pricing rows, a spawn-env quirk, install hints, a branded UI chip) — tracked
+in **Optional polish** and the conditional steps 3–6 below.
 
 If you find yourself wanting to write `if (this._provider === 'X')` in
 a manager, **the design has drifted** — find the capability you're
 gating on, add a flag to `ProviderCapabilities`, and branch on the
 flag instead.
 
+The remaining genuine id-keyed sites (the legacy `normaliseResultEvent`
+branch and the Codex rail slash-command rewrite) are inventoried in
+**Don't break the principle** at the end.
+
 ## The recipe
 
 ### 1. Implement the adapter
@@ -52,24 +63,79 @@ export const exampleAdapter: ProviderAdapter = {
   extractResult: (events): NormalisedResult => { /* events → tokens/cost/session */ },
   baselineAgents: () => ['sr-architect', 'sr-developer', 'sr-reviewer'],
   detectInstalled: async (): Promise<DetectionResult> => { /* `which` + `--version` */ },
+  // Optional — see "(Optional) prepareHeadlessSpawn" below. Gemini uses it to
+  // pre-acknowledge project subagents so they load in headless `gemini -p` rails.
+  // prepareHeadlessSpawn: (projectPath: string): void => { /* best-effort prep */ },
 }
 ```
 
 The `ProviderAdapter` interface is documented in
 `server/providers/types.ts`. Read the existing
-`server/providers/{claude,codex}-adapter.ts` for the patterns —
+`server/providers/{claude,codex,gemini}-adapter.ts` for the patterns —
 `SpawnAction` shapes per provider, `text-delta` event normalisation
-across native JSONL formats, etc. Note the shipped baseline is the
+across native JSONL formats, etc. **`gemini-adapter.ts` is the newest and
+most representative end-to-end exemplar** to copy: it shows the optional
+`prepareHeadlessSpawn` hook, `systemPromptArg: false` system-prompt folding
+(via the `GEMINI_SYSTEM_MD` env), native OTEL with `nativeCostUsd: false`,
+and a per-action `buildArgs` switch that throws defensively on the
+`chat-stream` action it doesn't support. Note the shipped baseline is the
 three-agent trio `['sr-architect', 'sr-developer', 'sr-reviewer']`;
 `ProfileManager` validation requires exactly your `baselineAgents()` to
 be present in every profile chain, so don't add agents your scaffold
 won't actually create.
 
+The `SpawnAction` union also includes **`chat-stream`** — used by the
+Explore persistent-stdin fast-path and interactive jobs. Only providers
+that advertise `capabilities.persistentStdin` (claude today) ever receive
+it; if your CLI has no persistent-stdin transport, `throw` in that `case`
+(as Gemini does) so a misrouted spawn fails loudly instead of emitting a
+broken argv.
+
+#### (Optional) prepareHeadlessSpawn
+
+`prepareHeadlessSpawn?(projectPath)` is an optional, best-effort filesystem
+prep run **right before a headless rail spawn** (cwd = the project path).
+Gemini implements it to pre-acknowledge the project's custom subagents
+(writing `~/.gemini/acknowledgments/agents.json`) so they load in
+`gemini -p` mode instead of falling back to a generic agent. Claude and
+Codex omit it. Managers wrap the call so a thrown error never blocks the
+spawn — adding it is allowed and does **not** violate the no-id-branching
+principle (it's an adapter member, not a `provider === 'X'` check in a
+manager). See `server/providers/gemini-agent-ack.ts` for the worked use.
+
 `parseStreamLine` returns `null` for empty input lines **and** for lines
 that fail `JSON.parse` (see `server/providers/codex-adapter.ts`); unknown
 JSON event types resolve to `{ kind: 'other' }`. Write your tests
 accordingly — don't assume null-only-on-empty.
 
+#### Two capability flags that bite
+
+- **`systemPromptArg: false`** ⇒ the CLI has no `--system-prompt` flag, so the
+  adapter must fold the system prompt into the user prompt before spawning —
+  **except** Explore (`chat-turn`) turns, which must stay user-text-only (a
+  long system prompt drowns a short Explore message); those trust the
+  app-managed instructions file (`<instructionsFilename>`) in the explore cwd
+  instead. Folding isn't the only mechanism: Gemini *also* exports its system
+  prompt via the `GEMINI_SYSTEM_MD` env, so an implementer should pick whatever
+  the CLI actually honours rather than assuming string concatenation.
+- **`nativeCostUsd: false`** ⇒ `extractResult` leaves `total_cost_usd`
+  undefined and the framework estimates it from a rate card (step 3). If your
+  token usage reports `cached` as a **subset** of input (it usually does — see
+  Gemini's `stats.cached`), map it to `tokens_cache_read`; `pricing.ts` already
+  bills `tokens_in - tokens_cache_read` at the input rate and the cached portion
+  at the cache rate, so don't double-count.
+
+#### Mirror the model catalog into spec-models.ts
+
+The model catalog is **duplicated** by design: the adapter's `modelCatalog()`
+drives the spawn-time UI, but spec-model validation reads a *second* copy in
+`server/spec-models.ts` (`PROVIDER_MODELS` + `PROVIDER_DEFAULT_MODEL`, both
+`Record<string, …>` keyed by provider id). The file even comments that
+`GEMINI_MODELS` "Mirrors GEMINI_MODELS in server/providers/gemini-adapter.ts".
+If you add the catalog only to the adapter, your models won't validate in the
+Add Spec flow. **Add your provider to both.** (Unmatched providers fall back to
+the Claude catalog, so the failure is silent — easy to miss.)
+
 ### 2. Register it
 
 Append the import to `server/providers/index.ts`:
@@ -90,43 +156,55 @@ provider with zero edits):
 
 - `getAdapter` / `listAdapters` / `hasAdapter` (`server/providers/registry.ts`).
 - `detectAvailableCLIs` (`server/core-compat.ts`).
+- `GET /api/available-providers` (`server/desktop-router.ts`) — returns the
+  **full detected map** built by iterating the registry, not a hardcoded
+  `{ claude, codex }` literal. A new provider shows up here with no edit
+  (the only per-id touch on this route is an optional beta gate, below).
 - `POST /api/projects` provider validation (`server/desktop-router.ts`,
   via `hasAdapter` / `listAdapters`).
+- `AddProjectDialog` (`client/src/components/AddProjectDialog.tsx`) — it
+  fetches `/available-providers` generically (`Object.entries(data)`) and a
+  `providerRenderOrder()` helper appends **any** detected-but-unlisted
+  provider after the canonical-ordered known ones, with a neutral chip
+  fallback. A new provider appears in the Add Project UI automatically.
 - `setup-prerequisites` provider rows (`server/setup-prerequisites.ts`,
   iterates `listAdapters()`).
 - Analytics `byProvider` (`server/spending.ts`) and the
   `ProviderBreakdownCard` (`client/src/components/analytics/ProviderBreakdownCard.tsx`).
 
-**Manual wiring still required** (these hardcode `claude` / `codex`
-today and will NOT surface a 4th provider until edited):
+### `ProviderId` is `string` — no compile-time blocker
 
-- `GET /api/available-providers` (`server/desktop-router.ts`) returns a
-  literal `{ claude, codex }` shape — add your key.
-- `AddProjectDialog` (`client/src/components/AddProjectDialog.tsx`)
-  hardcodes `PROVIDER_ORDER = ['claude', 'codex']` and reads
-  `data.claude` / `data.codex` explicitly. Without editing it the
-  provider won't appear in the Add Project UI.
-- `providerInstallUrl` / `providerInstallHint`
-  (`server/setup-prerequisites.ts`) have generic `default:` fallbacks so
-  nothing crashes, but a good install hint needs a `case` for your id.
-
-### Type-union widening
+There is **no type-union to widen**. `ProviderId` is `export type ProviderId = string`
+(`server/providers/types.ts`), and every provider-typed seam already aliases it:
+`CliProvider` (`server/desktop-db.ts`), `SpecProvider` (`server/spec-models.ts`),
+`EnqueueOptions.provider` and `_jobProviderSelection` (`server/queue-manager.ts`),
+`ChatManager`, `AgentRefineManager`, and `ProjectRegistry`. Adding a provider is
+**not** a compile-time blocker — the old "widen ~8 unions" step is paid off (Gemini
+is the proof it works). Any remaining `'claude' | 'codex'` literals live in test
+fixtures and code comments, not the production type system.
 
-A 4th provider is a **compile-time blocker** until you widen the
-`'claude' | 'codex'` unions still scattered across the server. The build
-fails until they're widened (or migrated to a shared `ProviderId`):
+### Optional polish (auto-works, but nicer with an edit)
 
-- `CliProvider` (`server/desktop-db.ts`) — the canonical project-row provider type.
-- `SpecProvider` (`server/spec-models.ts`).
-- The inline `'claude' | 'codex'` unions in `server/queue-manager.ts`
-  (`EnqueueOptions.provider`, `_jobProviderSelection`),
-  `server/chat-manager.ts`, `server/agent-refine-manager.ts`, and
-  `server/project-registry.ts`.
+None of these block a new provider — it works without them — but each one
+makes it look first-class instead of falling back to a raw id:
 
-Expect to widen roughly eight unions. The long-term goal is to delete
-them in favour of a single `ProviderId` derived from the registry — if
-you add a new hardcoded site instead of widening the existing ones, file
-an OpenSpec change first (see the closing section).
+- **Canonical position + branded chip.** `PROVIDER_ORDER` / `PROVIDER_META`
+  in `AddProjectDialog` give the provider a fixed position in the picker and
+  a custom icon + label (otherwise it's appended last with a `•` neutral chip).
+- **Install hint.** `providerInstallUrl` / `providerInstallHint`
+  (`server/setup-prerequisites.ts`) have generic `default:` fallbacks so
+  nothing crashes, but a good install hint needs a `case` for your id.
+- **Beta gate (only if you ship behind a kill switch).** If your CLI's stream
+  schema isn't yet validated against a live binary, gate selection behind a
+  `SPECRAILS_<ID>_BETA` env var. Mirror Codex/Gemini: add an
+  `isXBetaDisabled()` helper in `server/desktop-router.ts` (return `true`
+  only when the env value is the **exact string `'0'`** — both providers are
+  default-**enabled**), force the provider `false` in the `/available-providers`
+  response when disabled, and reject it in `POST /projects` with the same 400
+  the others use. There is no `isGeminiBetaEnabled` — the helper is
+  `isGeminiBetaDisabled` (returns `true` to DISABLE). Codex additionally honours
+  the legacy `SPECRAILS_HUB_CODEX_BETA` name as a fallback; a brand-new provider
+  has no such legacy alias.
 
 ### 3. (If `nativeCostUsd === false`) add pricing entries
 
@@ -144,16 +222,22 @@ back to this table automatically and returns an `estimated` flag that
 `total_cost_usd_estimated = 1` on the `ai_invocations` row, which in
 turn lights up the `~` tilde + Hero footnote on the AnalyticsPage.
 
-### 4. (If `nativeOtelEnv === false`) confirm the OTEL bridge works
+### 4. OTEL telemetry
 
-The synthetic OTEL bridge at `server/codex-otel-bridge.ts` is
-provider-neutral despite its name — it consumes the canonical
-`AdapterEvent` stream. As long as your adapter's `parseStreamLine`
-emits `text-delta`, `tool-use`, `session-started`, and `result` events,
-the bridge will synthesise traces / metrics / logs for free. The
-exported factory is still named `createCodexOtelBridge`; if that bugs
-you, renaming it to `createSyntheticOtelBridge` (and updating callers)
-is safe — same logic.
+Two paths, picked by your `nativeOtelEnv` capability flag:
+
+- **`nativeOtelEnv: true`** (Claude, Gemini) ⇒ nothing to do. `QueueManager`
+  injects the **standard `OTEL_*` env vars** (`buildTelemetryEnv`) into every
+  rail spawn and the CLI exports OTLP/JSON to the app's receiver natively. There
+  is no `GEMINI_TELEMETRY_*` env var — Gemini honours the same `OTEL_*` vars as
+  Claude.
+- **`nativeOtelEnv: false`** (Codex) ⇒ the synthetic OTEL bridge at
+  `server/codex-otel-bridge.ts` fills the gap. It's provider-neutral despite its
+  name — it consumes the canonical `AdapterEvent` stream. As long as your
+  adapter's `parseStreamLine` emits `text-delta`, `tool-use`, `session-started`,
+  and `result` events, the bridge synthesises traces / metrics / logs for free.
+  The exported factory is still named `createCodexOtelBridge`; renaming it to
+  `createSyntheticOtelBridge` (and updating callers) is safe — same logic.
 
 ### 5. (If `mcpRegistration === 'cli-add'`) wire the plugin install path
 
@@ -166,7 +250,33 @@ app-level `PluginManager` already threads `providerId` through every
 relevant method.
 
 For `mcpRegistration === 'project-json'` providers, the existing
-`.mcp.json` surgical-merge path applies — nothing to add.
+`.mcp.json` surgical-merge path applies — nothing to add. (This is why
+Serena-style plugins resolve for Claude and Gemini rails but are skipped
+for Codex-only projects, which use `cli-add`.)
+
+### 6. (If the binary needs a spawn-env quirk) add it to cli-prompt.ts
+
+Per-binary spawn quirks live in `server/util/cli-prompt.ts` — the spawn layer
+already special-cases `spawnClaude` / `spawnCodex` / `spawnGemini`, dispatched
+by binary name. Gemini needed `GEMINI_CLI_TRUST_WORKSPACE=true` injected into
+every spawn (its "trusted folders" gate otherwise silently disables `--yolo`
+and blocks headless tool calls). If your CLI has an equivalent env that
+non-interactive spawns require, add a `spawn<Provider>` helper there. Most
+providers need nothing here.
+
+### CLI version floor vs. specrails-core package floor
+
+Two distinct version concepts — don't conflate them:
+
+- **`minCliVersion`** is the **binary** floor surfaced by `detectInstalled`
+  (Claude = `null` / none pinned, Codex = `0.128.0`, Gemini = `0.11.0`). It
+  guards stream-format / flag availability for that one CLI.
+- **`specrails-core@^4.8.0`** (`CORE_PACKAGE_SPEC`, `server/core-package.ts`) is
+  the single shared package floor the app installs/probes for **all** providers.
+  It matters when your provider's **rails** rely on core-side scaffolding: core
+  must emit the provider's command/agent/skill tree (e.g. Gemini needed core
+  `4.8.0` to ship the `.gemini/` commands + `sr-*` agents). The desktop adapter
+  alone covers spec / explore / quick; rails need the matching core target.
 
 ## Known gotchas
 
@@ -174,7 +284,7 @@ For `mcpRegistration === 'project-json'` providers, the existing
   (`server/result-event.ts`) is still live for any callsite not yet
   migrated to `finaliseInvocationResult`. It only special-cases
   `provider === 'claude'`; everything else falls into the non-claude
-  (codex-shaped) branch. A 4th provider hitting that path would be
+  (codex-shaped) branch. A new provider hitting that path would be
   silently parsed as codex — migrate the callsite or extend the branch.
 - **Rail slash-command translation is provider-specific.** In
   `server/queue-manager.ts` the rail prompt builder rewrites
@@ -199,9 +309,12 @@ Required coverage:
 
 - Identity: id / binary / projectDirName / instructionsFilename /
   mcpRegistration / capability flags / model catalog.
-- `buildArgs` for every `SpawnAction` the manager flow uses (today:
+- `buildArgs` for every `SpawnAction` the manager flow uses:
   `chat-turn`, `chat-resume`, `rail-job`, `spec-gen`, `agent-refine`,
-  `setup-enrich`, `setup-enrich-resume`, `auto-title`).
+  `setup-enrich`, `setup-enrich-resume`, `auto-title`, and `chat-stream`
+  (the persistent-stdin / interactive-job action — test either the argv
+  you emit or the defensive `throw`, like Gemini does, if your CLI has no
+  persistent-stdin transport).
 - `parseStreamLine` per event type, including an "unknown type maps
   to kind: 'other'" defensive test and a "returns null on empty input
   and on unparseable JSON" test.
@@ -216,9 +329,9 @@ npx vitest run server/providers server/pricing server/result-event server/plugin
 ```
 
 `npm run typecheck` runs `tsc --noEmit` for both the server and the
-client — important here, because the type-union widening above touches
-types both halves import, and a missed union surfaces only on the side
-you didn't check.
+client — run both halves, because a provider touches types each side
+imports (`ProviderId`, the model catalog) and a mistake can surface only
+on the side you didn't check.
 
 Then a manual smoke test: register a project via the UI with the new
 provider, run a chat turn, run a rail, confirm tokens + cost land on
@@ -226,13 +339,33 @@ the AnalyticsPage.
 
 ## Don't break the principle
 
-The drift inventory above — the two hardcoded provider lists, the ~8
-`'claude' | 'codex'` unions, the legacy `normaliseResultEvent` branch,
-and the provider-specific rail rewrite — is the current debt. The
-long-term goal is to delete every one of those sites in favour of a
-registry-derived `ProviderId` so that adding a fifth provider really is
-just the adapter file plus the registry entry. If you find a NEW
-hardcoded `if (provider === 'X')` site that this guide doesn't list,
-**the architecture has drifted further** — file an OpenSpec change at
-`openspec/changes/<your-change-name>/` and capture the drift before
+Most of the old debt is **paid off** — and Gemini's addition is what paid it:
+
+- ✅ The `'claude' | 'codex'` type unions are gone (`ProviderId = string`).
+- ✅ The two hardcoded provider lists are gone (`/available-providers` and
+  `AddProjectDialog` are registry-driven with fallbacks).
+
+The genuine id-keyed sites that **remain** are:
+
+- **`normaliseResultEvent` legacy branch** (`server/result-event.ts`) — the
+  pre-`finaliseInvocationResult` path that special-cases `provider === 'claude'`.
+- **Codex rail slash-command rewrite** (`server/queue-manager.ts`) —
+  `adapter.id === 'codex'`.
+
+Plus a few cosmetic per-id sites that are *expected* edits, not drift: the
+provider beta-gate env checks (`server/desktop-router.ts`), the install-hint
+`switch` (`server/setup-prerequisites.ts`), the `PROVIDER_ORDER` / `PROVIDER_META`
+chip, and the duplicated model catalog (`server/spec-models.ts`).
+
+The long-term goal is to delete the two remaining branches so adding a fourth
+provider really is just the adapter file plus the registry entry. If you find a
+**new** hardcoded `if (provider === 'X')` site in a manager that this guide
+doesn't list, **the architecture has drifted further** — file an OpenSpec
+change at `openspec/changes/<your-change-name>/` and capture the drift before
 papering over it with another manager-level branch.
+
+## See also
+
+- [`docs/codex.md`](../codex.md) and [`docs/gemini.md`](../gemini.md) — the
+  user-facing guides for the two non-Claude providers; `gemini.md` is the most
+  recent worked example end-to-end.