mlx-bun 0.0.4

package/docs/planning/PRODUCT_ROADMAP.md

@@ -0,0 +1,426 @@
+# PRODUCT ROADMAP
+
+Working notes for the product shape of mlx-bun. This is intentionally
+less settled than PLAN.md: PLAN is the engineering contract; this file is
+the evolving UI/UX/DX north star.
+
+## Product north star
+
+Make the answer to "what is the best way to use local AI on my Mac?"
+boringly simple:
+
+```sh
+mlx-bun
+```
+
+The user should not need to research Hugging Face variants, quantization
+formats, context windows, Python environments, server commands, or agent
+provider config before they can try local AI. They should install one
+binary, run one command, wait for one resumable model download, and get a
+working local chat UI plus a local AI server.
+
+The runtime can still expose every advanced knob. The default path should
+not require them.
+
+## The three modes (the organizing design law)
+
+mlx-bun has not published anything: the entire CLI, command, and flag
+surface is still ours to invent, and we are not married to any flag
+name. Where a de-facto standard name exists (mlx-lm's), we adopt it
+rather than reinvent. The surface is designed once, to serve three
+audiences at three depths — progressive disclosure over ONE set of
+verbs and one flag vocabulary, not three separate CLIs:
+
+1. **Automatic mode** — first-time local-AI users. `mlx-bun` (and bare
+   verbs) with zero flags. Smart, sane defaults from the device-profile
+   table (see Core idea). Typing to a chatbot in a web UI in under a
+   minute, with no knowledge of HF variants, quant formats, or context
+   windows required.
+2. **Compat mode** — people moving over from mlx-lm. The same verbs,
+   plus dotted aliases (`mlx-bun.server` *is* `mlx-bun serve`), using
+   mlx-lm's flag vocabulary, so existing muscle memory and scripts port
+   with a one-token rename. mlx-lm is canonical because it is the
+   default Apple references at WWDC and outweighs the alternative by
+   ~425× monthly installs (1.64M vs 3.85k; mlx-lm 5.8k stars / 765
+   forks; mlx-optiq has no public repo — measured 2026-06-12).
+3. **Stick-shift mode** — power users and researchers. The full knob
+   surface: every mlx-lm flag, every optiq feature, plus our own
+   envelope-pushing ones, all exposed and composable. Build the exact
+   system you want.
+
+### Laws that keep the three modes coherent
+
+- **One surface, mlx-lm vocabulary.** The dotted form is a pure alias
+  of the space form. Adopt mlx-lm's flag names wherever they are the
+  standard; only name a flag ourselves where mlx-lm has no concept for
+  it (and there we are free to improve on optiq's name, not bound to
+  it).
+- **Compatibility = superset, never intersection.** Every documented
+  mlx-lm flag/verb behaves faithfully so scripts drop in unchanged. We
+  never subtract or gate a capability we already have for free to
+  resemble an upstream — e.g. we serve OpenAI + Anthropic + Responses
+  from one server, and that stays on: it costs nothing and is invisible
+  to a beginner.
+- **Gate only on conflict.** A capability is always on UNLESS leaving
+  it on would degrade the automatic or compat experience (it changes
+  output determinism, trades quality, or surprises a script). Only then
+  does it move behind a stick-shift flag. This is rule 3 yielding to
+  rules 1 and 2.
+- **optiq is a feature set to absorb, not a flag dialect to honor.**
+  Its distinctive features (sensitivity mixed-precision convert,
+  per-layer KV sensitivity profiling, LoRA rank scaling, dual-protocol
+  serving, latency prediction, the Lab UI) graft onto the mlx-lm spine
+  as additive flags and a few new verbs.
+
+### Status (2026-06-12)
+
+The compat layer does not exist yet: today's commands (`serve`, `fit`,
+`benchmark`, `evals`, `get`/`scan`/`ls`, `pi`, `harness`) are ad-hoc,
+not a designed mlx-lm-matching surface — though their flags already
+overlap mlx-lm/optiq heavily, so this is mostly rename-and-confirm, not
+migration. The work: design the full surface against mlx-lm vocabulary,
+reconcile the existing commands into it, and fill the capability matrix
+behind it (convert/quantize, LoRA training, eval/perplexity, fuse,
+cache_prompt, upload/share). Decomposed into phases in PLAN.md.
+
+## Core idea
+
+The local-AI problem is not continuous for Mac users. People own a small
+set of real Apple Silicon SKUs: 16 GB, 24 GB, 32 GB, 36 GB, 48 GB,
+64 GB, and larger workstation tiers. For each profile, we can research
+and preselect the best known model artifact, context target, KV mode,
+memory budget, and runtime flags.
+
+That means first-run behavior can be a lookup table, not a wizard or a
+benchmark:
+
+```text
+detect Mac profile
+look up recommended runtime profile
+download/resume the exact model artifact
+start the tuned local server
+open the built-in UI
+show how to connect external tools and harnesses
+```
+
+The research work feeds the table. The table feeds the user.
+
+## Recommended profiles
+
+Initial profile work should stay narrow and excellent. Start with the
+Gemma 4 OptiQ family and expand only when the project has a better answer.
+
+Candidate shape:
+
+| Device profile | Default model | Intent |
+|---|---|---|
+| MacBook Air / 16 GB | `mlx-community/gemma-4-e4b-it-OptiQ-4bit` | Fast first local-AI experience. |
+| MacBook Pro / 24-32 GB | `mlx-community/gemma-4-12B-it-OptiQ-4bit` | Strong default chat/agent model. |
+| Mac Studio / 64 GB | `mlx-community/gemma-4-26B-A4B-it-OptiQ-4bit` | Larger local-agent profile. |
+
+Exact model choices, context lengths, and flags are not frozen here. They
+should be updated from benchmarks, evals, fit predictions, and real use.
+
+Each profile should eventually carry:
+
+```ts
+{
+  id: "macbook-pro-24gb",
+  model: "mlx-community/gemma-4-12B-it-OptiQ-4bit",
+  context: 65536,
+  kv: "mixed",
+  memoryBudgetBytes: "...",
+  compiledDecode: true,
+  perfKernel: false,
+  expectedDecodeTps: 30,
+  expectedTtftMs: 100,
+  label: "Recommended for this Mac",
+}
+```
+
+The important rule: an optimization matters for product defaults when it
+changes a device-tier decision, unlocks a larger useful context window,
+keeps decode above a user-visible threshold, improves quality at the same
+budget, or prevents failure. Saving memory for its own sake is secondary.
+
+## First-run UX
+
+The no-args command should be the friendly path:
+
+```sh
+mlx-bun
+```
+
+Likely behavior:
+
+1. Detect the hardware profile.
+2. Pick the recommended runtime profile.
+3. Show the selected model and why it was chosen.
+4. Start or resume the background download.
+5. Start the local server when ready.
+6. Open or serve the built-in chat UI.
+7. Show connection details for OpenAI/Anthropic-compatible clients.
+8. Offer harness setup actions for Pi, Hermes, OpenClaw, and others.
+
+This should feel like an appliance, not a framework tutorial.
+
+## Built-in UI
+
+Bun can serve the UI directly from the same executable. The default UI
+does not need to be elaborate at first, but it should work immediately.
+
+Expected surfaces:
+
+- Chat UI backed by the local model server.
+- Download progress, resume status, and model readiness.
+- Current device profile and selected model.
+- Server URL and API compatibility hints.
+- Simple controls for common settings: context, temperature, max tokens.
+- Advanced panel for explicit model/profile overrides.
+- Integration panel for harnesses and external clients.
+
+### Model hot-swap from the UI (candidate)
+
+Let the user switch the served model at runtime — e.g. Gemma e4b →
+MiniCPM5 — from a picker in the web UI, without restarting the server.
+Weight load is cheap (mmap is lazy, load runs on the CPU stream; ~0.5 s
+for a small model), so a swap can feel like changing a setting rather
+than a relaunch. `/library` already enumerates every model on disk with
+a per-machine fit assessment and flags the active one (`serving`), so the
+menu and the "does it fit" data already exist — the missing pieces are a
+swap action and a mutable served context.
+
+Sketch:
+- Today the server is single-model-per-process: `serve()` captures one
+  `ServerContext` as a const in the `Bun.serve` closure. Hot-swap needs a
+  mutable "current context" ref that all handlers and the generation
+  gateway read through.
+- A swap endpoint (e.g. `POST /library/activate`) drains in-flight
+  generations, frees the old model's GPU buffers + KV/prompt caches,
+  `loadContext()`s the target, rebuilds the gateway, then atomically
+  swaps the ref. `/v1/models`, `/status`, and the Pi probe report the new
+  id afterward.
+
+Constraints to honor:
+- On a memory-bound Mac you can't hold two models at once — unload before
+  load. There is an unavoidable brief "swapping…" gap, not an instant
+  switch. Validate fit() before unloading the current model.
+- Load time scales with model size; don't promise instant for large ones.
+- Decide adapter behavior on swap (dropped vs. model-scoped) and surface
+  it. Active LoRA adapters are tied to the base model.
+
+The UI should not hide that this is local. It should make "local" feel
+easy: predictable memory, transparent downloads, offline-capable serving,
+and clear integration instructions.
+
+### The UI must never lag while the AI is busy (candidate, 2026-06-16)
+
+A product guarantee: **the AI may crash, the UI never may** — and the UI
+must stay responsive no matter how hard the model is working. Waiting for
+a *chat reply* is fine and expected (local inference takes time). A laggy
+web UI — slow refreshes, stalled `/status`, a frozen page while an agentic
+pipeline hammers the GPU — is not.
+
+One `mlx-bun` process plays three roles on one event-loop thread: it
+serves the UI, serves the API, and runs the AI. The AI work is synchronous
+blocking GPU/FFI (each `mlx_eval` is atomic and uninterruptible), so it
+starves the loop — worst in the non-streaming path and during prefill,
+where nothing hands a macrotask back. Yielding between tokens helps the
+common case but cannot isolate the UI under sustained load.
+
+Direction (decided: **subprocess**, not worker thread — only address-space
+isolation survives the uncatchable GPU OOM): move the inference runtime
+into its own persistent child process behind the existing
+`GenerationGateway` seam. The main process becomes a pure reactor (UI +
+API + routing, zero MLX calls); one GPU lease serializes all GPU consumers
+(chat, API, quantize, finetune) — reusing the spawn/lease pattern jobs
+already prove. Converges with model hot-swap (model lives off the main
+thread either way). Full design + phasing: `docs/design/runtime-isolation.md`.
+
+## Harness UX
+
+Harness setup should be executable, not just documented.
+
+```sh
+mlx-bun harness
+mlx-bun harness pi
+mlx-bun harness hermes
+mlx-bun harness openclaw
+```
+
+`mlx-bun harness` should open an interactive picker. Direct subcommands
+should configure a specific harness.
+
+Each harness adapter can:
+
+- Detect whether the harness is installed.
+- Preserve existing user config.
+- Add or update the local mlx-bun provider entry.
+- Point it at the local server.
+- Select the right API protocol.
+- Validate the connection with a tiny request.
+- Show the command to launch the harness.
+
+Pi deserves a first-class path because it can be embedded through its SDK:
+
+```sh
+mlx-bun harness pi
+mlx-bun pi
+```
+
+`mlx-bun harness pi` configures the user's existing Pi install. `mlx-bun
+pi` should start the local mlx-bun runtime and launch Pi itself against
+that runtime. If the Pi TUI is the right surface, use it. If an embedded
+SDK session is the right surface, use that.
+
+Other harnesses should follow the same adapter pattern.
+
+Sketch:
+
+```ts
+interface HarnessAdapter {
+  id: "pi" | "hermes" | "openclaw";
+  label: string;
+  detect(): Promise<Detection>;
+  configure(profile: RuntimeProfile): Promise<ConfigureResult>;
+  validate(profile: RuntimeProfile): Promise<ValidationResult>;
+  launch?(profile: RuntimeProfile): Promise<void>;
+}
+```
+
+## CLI shape
+
+Possible command map:
+
+```sh
+mlx-bun                         # recommended model + server + UI
+mlx-bun serve                   # server only
+mlx-bun ui                      # UI only, connect to local/running server
+mlx-bun repl                    # Bun REPL with a local AI client preloaded
+mlx-bun pi                      # launch embedded Pi against mlx-bun
+mlx-bun harness                 # interactive harness setup
+mlx-bun harness pi              # configure existing Pi
+mlx-bun harness hermes          # configure Hermes
+mlx-bun harness openclaw        # configure OpenClaw
+mlx-bun models                  # installed and recommended models
+mlx-bun fit                     # device/model fit information
+mlx-bun doctor                  # diagnostics
+```
+
+Flags should compose with these modes. The default is opinionated, not
+restrictive.
+
+Examples:
+
+```sh
+mlx-bun --model gemma-4-12b --context 65536
+mlx-bun serve --no-ui --port 8090
+mlx-bun pi --model qwen-profile
+mlx-bun harness pi --base-url http://localhost:8090/v1
+```
+
+## Research loop behind the product
+
+The product table should be backed by the same research discipline as
+PLAN.md:
+
+1. Pick a small set of real target devices.
+2. Pick a small set of candidate model artifacts.
+3. Measure fit, context, decode speed, TTFT, memory, and quality.
+4. Specialize the runtime for winning artifacts.
+5. Promote only defaults that pass clean-machine measurements.
+6. Keep fallback and explicit override paths for research and power users.
+
+The first family is Gemma 4 OptiQ. Qwen is the next obvious family to
+evaluate. The default model should be whichever gives the best experience
+for a device profile and use case, not whichever family arrived first.
+
+## Artifact design
+
+Longer term, mlx-bun may not only choose among existing model artifacts.
+It may create artifacts for exact device/use-case targets.
+
+The useful question is not "can we save 1% RAM?" It is:
+
+```text
+Can this model fit on this device,
+with this context window,
+above this decode-speed floor,
+while preserving enough quality for this use case?
+```
+
+TurboQuant plus sensitivity analysis fits this direction. The quantizer
+can target concrete SKU boundaries and context tiers:
+
+- Make a specific model fit on 16 GB with useful context.
+- Make a 12B-class model hit 64k context on 24-32 GB.
+- Keep decode above 30 tok/s on the recommended profile.
+- Preserve tool/agent behavior under the chosen quantization.
+
+This turns quantization from generic compression into product-driven
+artifact design.
+
+## Runtime specialization
+
+When a user asks to run a specific model, there is no ambiguity in the hot
+path. The artifact, config, tensor shapes, KV config, quantization layout,
+chat template, and architectural quirks are known.
+
+The generic runtime is scaffolding for research, portability, and
+fallback. The served instance can be specialized:
+
+- Generated per-model execution paths.
+- Branches and loops removed where the model shape is fixed.
+- Per-layer quant bits/group sizes baked in.
+- Known cache layouts and attention patterns used directly.
+- Custom kernels generated or selected for exact dispatch sites.
+
+Compatibility paths remain the oracle. Specialized paths are promoted
+only when they are correct, measured, and useful.
+
+## Lucien / memory-system direction (candidate, 2026-06-12)
+
+A second product direction under debate alongside the UI/harness work:
+make mlx-bun the local-model engine for Lucien (the Dreaming memory
+system), in two stages.
+
+1. **Prove the models do real work.** Run Lucien's pipeline tasks
+   (session ingestion, synthesis into articles, recall-grounded chat)
+   against mlx-bun-served models. Measure quality and performance per
+   task — this is a real-workload eval, not a benchmark row, and it
+   feeds the profile table (which model tier can sustain synthesis
+   work? what context does recall actually need?).
+2. **Package it.** If the models hold up: mlx-bun as a Lucien backend
+   option — one executable that creates the memory store, gathers
+   session transcripts, synthesizes them into articles, and serves
+   conversations grounded on those articles. Same appliance principle
+   as the chat default: one binary, one command, a working memory
+   system.
+
+This stresses exactly the things the runtime is good at measuring:
+long-context prefill, sustained batch-ish generation, prompt-cache
+reuse across pipeline stages.
+
+## Non-goals for the default UX
+
+- Do not make the user choose from thousands of Hugging Face variants on
+  first run.
+- Do not benchmark their machine before giving a recommendation.
+- Do not require Python, a venv, or manual server setup.
+- Do not require hand-editing harness config files.
+- Do not expose every research flag before the user has a working local
+  AI experience.
+
+## Open questions
+
+- What exact profile table should ship first?
+- Should no-args open the UI automatically, print a local URL, or both?
+- Which UI stack should be embedded?
+- How much of Pi should be bundled, embedded, or configured externally?
+- What is the minimum first-run UI that feels good enough?
+- What is the right install path: Homebrew, curl installer, GitHub
+  release, or all of the above?
+- How should background downloads interact with server startup and UI
+  readiness?
+- How should profile recommendations be versioned as benchmarks improve?
+- When Qwen lands, what evals decide whether it replaces a Gemma default?

package/docs/planning/ResearchTopics.md

@@ -0,0 +1,119 @@
+# Research Topics
+
+Speculative, exploratory directions for mlx-bun — theses worth prototyping, not
+committed work. Each entry carries enough detail to pick up cold. PLAN.md is for
+work in flight; this is for ideas waiting their turn.
+
+---
+
+## 1. Tonal Grading for Token Distributions ("HDR Sampling")
+
+> Replace the global scalar sampling knobs (temperature, top-k, top-p) with a
+> region-aware **transfer function** applied to logits before softmax — the
+> color-grading move, ported to token distributions.
+
+**Framing — the Wes Anderson point.** A film's symmetrical pastel look isn't a
+colorist failing to neutralize; it's a *target*. Grade Marvel that way and it's
+"wrong" only against Marvel's acceptance function, not against any absolute.
+Different films, different targets — and you have a lot of films. The
+Lucien-synthesis agent and the test-writing agent want different *looks*, and
+"temperature + a system-prompt vibe word" is the crude contrast slider standing
+in for a real grade.
+
+**Core thesis.** Current LLM sampling is "SDR": the available controls
+(temperature, top-k, top-p) are crude *global* knobs — the equivalent of a single
+contrast slider. Color grading abandoned "just turn the contrast knob" long ago
+in favor of transfer functions with region-specific behavior. The same move is
+available for sampling: replace global scalar knobs with a region-aware transfer
+function applied to the logits before softmax.
+
+**The reframe that makes it work.** Signal vs. noise is defined by *intent*, not
+by magnitude — exactly as on a camera sensor, where the noise floor is "noise"
+because it doesn't represent what's wanted, not because it's small. The full
+logit vector is the model's signal, but it is not the signal *I* care about. My
+acceptance criterion defines which part is signal. The low-logit tail is
+therefore legitimately treatable as a noise floor to roll off.
+
+**The three regions** — one curve, three treatments. This is precisely what
+temperature *cannot* do, since it scales everywhere uniformly:
+
+- **Shoulder** (head / confident tokens): soft rolloff instead of hard
+  truncation. Top-k/top-p slam into the ceiling and clip; a shoulder eases the
+  top tokens toward co-equality while *preserving their ordering* — it tames
+  single-token dominance without destroying the relationships among the spared
+  tokens.
+- **Midtones** (the plausible-but-distinct band): steepen for local contrast.
+  Push apart the tokens that matter so detail comes out sharp. This has no name
+  in current sampling because sampling has no concept of distributional
+  midtones — and it's the most novel piece.
+- **Toe** (tail / unwanted tokens): roll into the floor. Defines them as noise
+  relative to intent.
+
+**Curve vocabulary, ported from grading:**
+
+- **Pivot/cut** placed *relative* to the distribution (by rank or percentile),
+  recomputed every step — never an absolute logit value, because logits are only
+  meaningful up to an additive constant (softmax shift-invariance). HLG's
+  relative / scene-referred design is the right philosophical match; LogC3 is
+  more absolute / exposure-locked.
+- **Slope-matched splices** between regions (C1 continuity — no kink in how
+  adjacent-ranked tokens get treated).
+- **Curve family:** signed / `log1p`-style forms to stay finite on negative
+  logits. An HLG-style architecture (gamma toe + log shoulder, slope-matched) is
+  the leading candidate because its two-regime structure happens to match the
+  two-regime structure of a real logit distribution — temperature-style shaping
+  on the uncertain tail, log-style compression on the overconfident head.
+
+**Why it's not overclaiming.** I am *not* claiming buried capability that a curve
+"unclips" — the logits aren't a compressed encoding of something richer. The
+honest claim: there is an acceptable manifold the model can't articulate and the
+default transform doesn't reliably land on; the curve encodes my
+*observer-relative acceptance function* onto a distribution that has no opinion
+of its own. This is why sampling knobs exist at all — "friendly / neutral /
+professional" are crude gestures at exactly this, and a grade is the
+high-resolution version.
+
+**The per-agent-target insight (the payoff).** "Correctly transformed to display"
+is not universally acceptable — it's unacceptable for an Oscar-buzz feature and
+fine for a sports broadcast, because they have different acceptance functions.
+Grading Marvel like Wes Anderson isn't incompetence; it's the wrong *target*. My
+agents are different films:
+
+- A **conversation→article synthesis** agent wants range, texture, and
+  surprising-but-apt word choices — "cinematic."
+- An **acceptance-criteria→tests** agent wants tight, predictable, low-variance
+  determinism — "broadcast neutral."
+
+They want different *grades*, not different *temperatures*. Per-task tonal
+curves, not per-task scalar knobs.
+
+**Open engineering question — build this first.** The colorist works in a tight
+perceptual loop against a calibrated reference monitor; a sampler's curve is
+chosen *upstream* of seeing output, with a slower, noisier judgment loop. So the
+gating question isn't "is the curve good" — it's **"what is the monitor?"** What
+readout tells me the midtones got crisper and the highlights rolled off well, per
+target? Candidates: held-out preference, diversity-at-fixed-quality, task-specific
+pass rates (for the test agent) vs. human preference (for the synthesis agent).
+Define the per-target reference monitor first; the curve design follows from it
+the way a grade follows from a trustworthy display.
+
+**Implementation.** A pure elementwise function on the logits tensor inside
+mlx-bun's per-step logits-processor hook: sort once for the rank-relative pivot,
+apply the slope-matched toe/mid/shoulder curve, renormalize, sample. The hook
+already exists — logits processors run at
+[`generate.ts:236`](../../src/generate.ts:236)
+(`for (const p of processors) logits = disposing(logits, p(history, logits))`),
+built by `makeLogitsProcessors` in [`src/sampler.ts`](../../src/sampler.ts) as pure
+`(history, logits) => logits` functions applied before `toLogprobs` and the
+sampler — so the curve drops in as one more processor in that loop.
+
+Given dispatch dominates decode wall time (~90–96%), the curve's arithmetic cost
+rides inside work already dispatched — wall-time-negligible. Parameterize: pivot
+rank, toe exponent, midtone slope, shoulder compression. A/B against plain
+temperature **at matched effective entropy**, judged on the per-target monitor —
+looking for outputs that land in the target region *unreachably* by
+temperature-plus-top-p at any setting, not merely "different."
+
+*Relation to current code:* the `--temperature` / `--top-p` / `--top-k` server
+defaults (and `generation_config.json`) are the SDR knobs this topic proposes to
+supersede — useful as the A/B baseline, not the destination.

package/docs/planning/chunk-task-roadmap.md

@@ -0,0 +1,77 @@
+# Chunk-task quality roadmap (2026-06-17)
+
+Where the MiniCPM5 chunking adapter stands and what raises real segmentation
+quality. Companion to [[chunk-eval-metric-vs-accuracy]] and
+[`docs/investigations/chunk-finetune-experiment.md`](../investigations/chunk-finetune-experiment.md).
+
+## Where we are
+
+- Adapter **solved the output contract**: 0/25 invalid JSON (base model: 22/25
+  invalid — unusable). That's the entire 95.10 well-formedness score.
+- Adapter is **mediocre at the actual task**: vs-gold boundary ~35.6, label ~52.0,
+  all 25 cases in disagreement (`reports/chunk-eval-cpm-2026-06-17.html`).
+- Dominant failures: `under_split` 17/25, TAIL GAP / `missed_content` 9,
+  hallucinated anchors 4 — explained by the training-data bias below.
+
+## Step 1 (the lever): fix the training data
+
+The data (`chunk-data-le4000`) is **86% single-chunk** (626/730 examples = 1 chunk,
+mean 1.16, max 4). The model learned a strong "one short chunk" prior → it
+under-splits and truncates coverage. A 1B model on this diet isn't at its capacity
+ceiling, it's mimicking its data.
+
+- Rebuild the dataset so the **chunk-count distribution matches real operational
+  segmentation**, not a single-topic majority.
+- Teach **full-transcript coverage** (spans that reach the end — kill the tail gaps).
+- This is the #1 lever. Do it before touching capacity or iters.
+
+## Eval methodology: do NOT judge against codex's gold
+
+**Codex over-segments — it splits articles ~2× what we actually do in operations.**
+So the current gold is inflated: judging the adapter as `under_split` is partly an
+artifact of a gold that over-splits. The under-split signal is real but smaller than
+the verdict count suggests.
+
+- Recalibrate the gold to **operational granularity** (use real-ops segmentations as
+  the reference, or down-weight codex's extra splits) before trusting `bs`/`ls`.
+- Until the gold matches ops, the report **understates** the adapter.
+- Going forward, optimize against operational gold, not codex, and not the saturated
+  well-formedness score.
+
+## rank / scale: tune for a single narrow task
+
+We are telling CPM to do exactly **one** thing: take a sequence of messages and find
+the topic seams. That narrowness should drive the LoRA capacity knobs deliberately —
+revisit the inherited rank 16 / scale 1 rather than carrying old defaults. Folds into
+the "configurable training params with mlx-lm defaults" task (rank / scale / num_layers
+/ LR / dropout selectable at run start). The question to answer empirically: how little
+rank does this one transformation need, and what scale/LR pairs with it.
+
+## Streaming / windowed segmentation mode
+
+Whole-transcript-at-once is one framing; it likely isn't the operational one. Explore:
+
+- A **sliding message window** — segment incrementally as messages arrive, over a
+  bounded window, rather than ingesting the entire conversation.
+- "Lots of different ways to segment these conversations" — try multiple task
+  formulations (whole-transcript vs windowed vs streaming/online) and measure which
+  matches operations and trains best on a small model.
+- Implications for training data shape (windowed targets) and for serving (a streaming
+  endpoint / incremental decode).
+
+## Evaluate other base models
+
+CPM5-1B is tiny and was a memory-driven pick. Run a **bake-off**: score other base
+models on the chunk task (same gold, same harness) to find the best base for the
+chunking adapter. Capacity is the second lever after data — and the e4b +
+segmented-backward work exists precisely to train bigger models at long context. See
+[[e4b-lora-training-seqlen-ceiling]], [[segmented-backward-landed]].
+
+## Ordered plan
+
+1. **Fix the training data** (multi-chunk + full coverage, operational granularity).
+2. **Recalibrate the eval gold** off codex toward operational segmentation.
+3. **Tune rank/scale** for the single narrow task (with configurable params).
+4. **Prototype streaming/windowed** segmentation; pick the framing that fits ops.
+5. **Model bake-off**; move up in capacity (e4b via segmented backward) if data alone
+   plateaus.