lobes-cli 0.90.3__tar.gz → 0.92.0__tar.gz

{lobes_cli-0.90.3 → lobes_cli-0.92.0}/CHANGELOG.md

@@ -4,6 +4,38 @@ All notable changes to this project are documented here. The format is based on
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project
 adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.92.0] - 2026-10-26
+
+### Added
+
+- Catalog entry `nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB` (74B-A9B MoE hybrid, text+image+video, 262144 native, modelopt mixed IQ2/Q8/NVFP4, DFlash LiLiCorr draft) with its doc, a `qwen3_coder` parser rule, and a Spark GB10 acceptance transcript (text, image, tool call and a 231,935-token needle pass).
+
+### Changed
+
+- `worker` is now Nemotron 3.5 Bolt (operator decision, 2026-10-26), hosted on the DGX Spark; `nvidia/Qwen3.6-35B-A3B-NVFP4` is demoted to a kept candidate. The worker contract is unchanged. Bolt needs the card's vLLM nightly plus `b12x` and is sm_12x-only; the packaged `vllm-worker` lane cannot express its flags yet, so the Spark serves it through a full command in its hand-kept override.
+
+## [0.91.0] - 2026-10-23
+
+### Added
+
+- Hebrew Whisper STT (audio-he overlay): decode-quality knobs for long recordings (issue #329) - STT_NO_REPEAT_NGRAM_SIZE (default 6), STT_REPETITION_PENALTY (default 1.0 = off), STT_COMPRESSION_RATIO_MAX (default 2.4), STT_TEMPERATURE_FALLBACK (default 0.2,0.4), STT_LOGPROB_STAT (default median; mean = legacy), STT_MAX_WORD_RUN (default 4; 0/off disables), STT_VAD (default auto) with STT_VAD_THRESHOLD / STT_VAD_MIN_SPEECH_MS; each has an off value that restores the previous decode exactly.
+- Word-run knobs (#329 follow-up): STT_MAX_WORD_RUN (default 4; 0/off disables) TRIMS a run of one word, or one two-word phrase (Unicode word tokens, punctuation ignored), to 3 repeats and keeps the rest of the chunk, so emphatic speech is never dropped; STT_WORD_RUN_FAIL (default 8; 0/off disables) fails a multi-window chunk with a run that long like a compression-ratio failure - retried through the temperature ladder, then dropped with reason=word_run. stt_chunk lines carry word_run=, max_word_run= and word_run_fail=, and a trim logs event=trimmed.
+- Structured stt_chunk (retry / dropped / trimmed / skipped) and per-request stt_request log lines in the stt container, so dropped and retried chunks are observable server-side; stt_request names the request policy (policy=single_window|multi_window stat=mean|median); the client-facing dropped[] field waits for verbose_json (#328).
+
+### Changed
+
+- STT_LOGPROB_STAT now defaults to median (threshold STT_MIN_AVG_LOGPROB=-0.35 unchanged) for multi-window uploads: on a live A/B on the DGX Spark the mean gate dropped 118 of 215 decoded chunks and the median gate 39 of 215.
+- Single-window requests (every realtime turn) keep the pre-#329 confidence gate exactly: the mean statistic, one greedy decode, no temperature fallback, a word run trimmed but never failed. A short silence hallucination (mean -0.58, median -0.15) would pass a median gate.
+- The stt handler runs the whole transcription (resample, Silero scan, every Whisper decode) off the event loop in a worker thread, one transcription at a time behind one lock, so /v1/health/ready stays responsive during a long upload; the Silero wrapper's lock now covers a whole chunk scan (reset plus every window).
+- DecodeSettings, DecodePolicy (both frozen), ChunkOutcome and RequestSummary are dataclasses; STT_MIN_AVG_LOGPROB accepts the same off tokens as the other knobs (false and no included).
+- Dockerfile.whisper-stt installs silero-vad==6.2.1 with --no-deps (its torchaudio dependency would replace the base image torch); the server loads the bundled TorchScript model by path and falls back to decoding every chunk if it cannot.
+
+### Fixed
+
+- Hebrew batch STT no longer passes a phrase looped to the token cap (n-gram block plus a per-character zlib compression-ratio check), and re-decodes a chunk failing either gate at sampling temperatures before dropping it (#329). The live A/B measured the median gate with a fail-at-4 word-run gate; the single-window policy and the trim-versus-fail word-run design have not run live.
+- The last chunk of a multi-window upload is never VAD-skipped (a short closing word was lost), and a chunk too short for one VAD window counts as speech.
+- A retry that comes back empty after an earlier attempt failed a gate is a drop (reason=empty_after_retry), logged and counted as one, never as a recovery.
+
 ## [0.90.3] - 2026-10-18
 
 ### Fixed

{lobes_cli-0.90.3 → lobes_cli-0.92.0}/CLAUDE.md

@@ -345,6 +345,19 @@ entry, and the `thor-muse` shape all **stay in-tree** (cite-don't-delete) —
 dormant, not deleted — and the tier vocabulary above still ranks `worker` <
 `muse`. See `docs/gemma-4-31b-nvfp4.md`.
 
+**`worker` re-checkpointed again (2026-10-26, operator decision): Nemotron 3.5
+Bolt 74B-A9B 24GB, hosted on the DGX Spark.** The catalog's `worker` role_hint
+moved from `nvidia/Qwen3.6-35B-A3B-NVFP4` (now a kept candidate) to
+`nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB`: a ~9B-active MoE hybrid with a
+RADIO ViT (image+video), 262144 native, DFlash from its LiLiCorr draft, the
+same multimodal-coder contract. Its IQ2 experts run only on sm_12x, so the
+Spark hosts it (cortex is dropped there and served by spark2) and the Thor
+stops hosting worker. The packaged `vllm-worker` lane cannot express its flags
+yet; the Spark serves it through a full command in its hand-kept override. See
+`docs/nemotron-3.5-bolt-74b-a9b-24gb.md` and
+`docs/evidence/2026-10-26-accept-nemotron-bolt-spark.txt`. The paragraphs
+below are the record of the Qwen and Lightning workers before it.
+
 **`worker` — the eighth role (opt-in hosting), the fast ground-work DOER —
 RE-CHECKPOINTED AGAIN, issue #244 (2026-09-10), off Lightning onto
 `nvidia/Qwen3.6-35B-A3B-NVFP4`.** Deviation d1 (2026-08-20) had put worker on
@@ -383,8 +396,8 @@ exactly (`WORKER_FEASIBLE` / `WORKER_PEER_ORIGIN` / `WORKER_PEER_PROXY` /
 like `muse`). Under pressure `worker` sheds (429) exactly like
 cortex/senses/muse.
 
-**Hosted on the Jetson AGX Thor again (#244); the DGX Spark reaches it by
-proxy (2026-09-11).** The Thor serves the lane locally through the
+**(Superseded 2026-10-26 — see the Bolt paragraph above.) Hosted on the Jetson
+AGX Thor again (#244); the DGX Spark reaches it by proxy (2026-09-11).** The Thor serves the lane locally through the
 `thor-worker` shape at `gpu_mem_util=0.45`, full `max_model_len=262144`,
 DFlash speculation, and `WORKER_MAX_NUM_SEQS=1`. One request at a time is
 **by design** and is still to be tested under a real multi-agent load. See

{lobes_cli-0.90.3 → lobes_cli-0.92.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: lobes-cli
-Version: 0.90.3
+Version: 0.92.0
 Summary: lobes — run, assess, and switch the local vLLM model.
 Project-URL: Homepage, https://github.com/agentculture/lobes-cli
 Project-URL: Issues, https://github.com/agentculture/lobes-cli/issues

lobes_cli-0.92.0/docs/evidence/2026-10-26-accept-nemotron-bolt-spark.txt

@@ -0,0 +1,117 @@
+ACCEPTANCE — nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB served standalone on the DGX Spark GB10
+Date: 2026-10-25 22:36Z - 22:47Z (UTC)
+Box: DGX Spark GB10 (sm_121, 128 GB unified memory)
+Scope: one standalone vLLM container via sparkrun, OUTSIDE the lobes fleet compose and gateway.
+       Proves the checkpoint boots and serves on this card with the recipe below. It does NOT
+       prove `lobes switch`, a fleet lane, a shape or a gateway route — none of those exist for it.
+
+== Checkpoint ==
+model:      nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB @ c6cc2e8ffa353403213d6b9b18aaf8eb684d705a
+draft:      nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr @ 449d24140160b8e51871bfc0f0a7338abf640aa1
+config.json: NemotronHPuzzleForCausalLM in a nemotron_h_omni wrapper; RADIO vision_config; no sound_config;
+             llm_config.max_position_embeddings 262144
+quantization_config: quant_method modelopt, quant_algo MIXED_PRECISION
+             IQ2_XXS 27,728 layers / Q8_0 113 / W4A16_NVFP4 80
+
+== Engine ==
+image:  vllm/vllm-openai:nightly-77154ca6116ae34cbfe314beef1016bde9b63989 + b12x 1.5.1 baked in
+        (the card installs b12x at container start; baked so the boot runs offline)
+vllm:   0.32.0rc1.dev100+g77154ca61   torch 2.13.0+cu130
+Resolved architecture: NemotronH_Omni_Reasoning_V3 (target), LiLiCorrDraftModel (draft)
+
+== Serve flags (the card's vLLM command, with the deviations marked) ==
+--revision c6cc2e8f... --trust-remote-code --load-format auto --tensor-parallel-size 1
+--max-model-len 262144 --max-num-seqs 1 --enable-chunked-prefill --kv-cache-dtype fp8
+--linear-backend b12x --moe-backend b12x --attention-backend FLASHINFER
+--mamba-backend flashinfer --mamba-ssm-cache-dtype bfloat16 --mamba-cache-dtype bfloat16 --mamba-cache-mode align
+--limit-mm-per-prompt '{"video":1,"image":1}' --video-pruning-rate 0.5
+--media-io-kwargs '{"video":{"fps":2,"num_frames":256}}'
+--enable-auto-tool-choice --tool-call-parser qwen3_coder
+--reasoning-parser-plugin <checkpoint>/ultra_v3_reasoning_parser.py --reasoning-parser ultra_v3
+--speculative-config '{"method":"dflash","model":"...-LiLiCorr","revision":"449d2414...","num_speculative_tokens":7,
+                       "rejection_sample_method":"standard","attention_backend":"FLASHINFER","draft_load_config":{"load_format":"auto"}}'
+--seed 42
+DEVIATION  --kv-cache-memory-bytes 4294967296 (4 GiB) added; --gpu-memory-utilization 0.40 then only gates the
+           startup free-memory check. Reason: on this unified-memory box a util-sized KV pool is not
+           deterministic (vLLM's profiling counts other processes' memory growth), measured the same night on a
+           different checkpoint at util 0.70 / 0.60 / 0.63 -> 22.99 / 5.87 / 2.94 GiB.
+DEVIATION  --allowed-local-media-path / dropped (it lets any caller read any file on the box via file:// URLs).
+DEVIATION  revisions pinned; bound to 127.0.0.1.
+
+== Boot ==
+22:39:14  Loading weights took 125.55 seconds (target); 22:39:34 5.63 seconds (draft)
+22:39:35  Model loading took 24.09 GiB memory and 150.801226 seconds
+22:40:18  Initial free memory 105.18 GiB, reserved 4.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes
+22:40:18  GPU KV cache size: 350,464 tokens, Maximum concurrency for 262,144 tokens per request: 1.34x
+22:40:30  Application startup complete; inference ready
+WARNING   max_num_scheduled_tokens is set to 2048 based on the speculative decoding settings (vLLM's own note)
+
+== Probes (temperature 1.0, top_p 0.95, thinking on by default) ==
+/v1/models: ['nemotron-3.5-bolt', 'nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB']
+known-answer  "What is 17*23?"                 -> '391', reasoning split off (149 chars)        0.8 s   PASS
+image         64x64 solid red PNG              -> 'Red'                                         2.0 s   PASS
+image         64x64 solid blue PNG             -> 'Blue'                                        3.1 s   PASS
+image         64x64 solid green PNG            -> 'Green'                                       2.2 s   PASS
+tool call     get_weather, "weather in Haifa?" -> finish_reason tool_calls,
+                                                  get_weather({"city": "Haifa"}), content None  1.2 s   PASS
+speed         palindrome function              -> 359 completion tokens in 5.2 s wall = 69.3 tok/s
+                                                  (one request; wall clock includes prefill and reasoning)
+long context  231,935-token prompt, needle mid-document
+                                               -> 'PELICAN-4471'                              200.3 s   PASS
+The image probes are synthetic solid colours: they prove the vision path is wired and the colour
+negative controls separate, not real-image quality.
+
+== Host after the probes ==
+host MemAvailable  70,146 MiB
+kernel NV_ERR_NO_MEMORY lines since the boot: 0
+hottest ACPI zone  86 C (just after the 200 s long-context prefill)
+
+== Not proven here ==
+- Any lobes path: `lobes switch` (the stock template's image lacks the nightly + b12x), a fleet lane, a gateway route.
+- Other cards: the IQ2 expert path is sm_12x-only in vLLM; the Orin (sm_87) refuses it, the Thor (sm_110) is untried.
+- Concurrency above 1, sustained decode speed under load, DFlash acceptance rates.
+
+======================================================================================================
+PART 2 — Bolt as THE `worker`, served by the Spark's lobes `vllm-worker` lane (2026-10-25 22:56Z - 23:20Z)
+======================================================================================================
+Operator decision: `worker` = Bolt, hosted on the Spark (sm_12x only); cortex dropped on the Spark and served
+by spark2 alone; the Thor stops hosting its nvidia/Qwen3.6-35B-A3B-NVFP4 worker and reaches `worker` by mesh.
+
+== Spark deployment (hand-kept files, each backed up first) ==
+.env:  WORKER_MODEL / WORKER_SERVED_NAME = nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB, WORKER_MAX_MODEL_LEN=262144,
+       WORKER_QUANTIZATION=modelopt_mixed (what vLLM resolves for this config), WORKER_GPU_MEM_UTIL=0.40,
+       WORKER_IMAGE=<the nightly + b12x image above>, WORKER_FEASIBLE=true, COMPOSE_PROFILES=worker,
+       PRIMARY_FEASIBLE=false
+docker-compose.override.yml: vllm-worker `command:` replaced with the PART 1 flag list (the packaged lane cannot
+       express --linear-backend, the mamba cache flags, --kv-cache-memory-bytes, the media limits or the
+       reasoning-parser plugin), `depends_on: !reset null` (the pooling lanes it waits on are not hosted here)
+docker-compose.shape.yml: vllm-primary parked in the inert shape-dropped profile
+`lobes up worker --apply`, then `lobes up gateway --apply`
+
+== Lane boot (cold compile cache) ==
+22:59:20  Model loading took 24.09 GiB memory and 155.386063 seconds
+23:01:24  GPU KV cache size: 350,464 tokens, Maximum concurrency for 262,144 tokens per request: 1.34x
+          Application startup complete; container healthy
+
+== Thor ==
+.env backed up; WORKER_FEASIBLE=false (explicit), worker dropped from COMPOSE_PROFILES, WORKER_MODEL /
+WORKER_SERVED_NAME set to the Bolt id (advert only). `lobes up worker --down --apply` (with worker still in
+COMPOSE_PROFILES: the verb refuses a role the .env does not activate), then `lobes up gateway --apply`.
+Host MemAvailable after the gateway was healthy: 62 GiB.
+
+== Through each member's gateway (API key on) ==
+                 /capabilities worker                                           model=worker   model=cortex
+spark (host)     feasible true, ready true, context 262144, quant modelopt      '391' local    '391' via spark2
+thor             feasible false, ready true, hosted_by spark, proxied true      '391' via spark '391' via spark2
+spark2           feasible false, ready true, hosted_by spark, proxied true      '391' via spark '391' local
+Every forward carried X-Lobes-Mesh-Member (spark for worker, spark2 for cortex) and X-Lobes-Proxied-By.
+
+Via the Thor's gateway (mesh forward to the Spark lane):
+image  64x64 solid red  -> 'red'   5.5 s   PASS
+image  64x64 solid blue -> 'Blue'  2.1 s   PASS
+tool   get_weather      -> finish_reason tool_calls, get_weather({"city": "Haifa"})  1.9 s   PASS
+
+== Not proven by PART 2 ==
+- The packaged template serving Bolt (it cannot yet; the Spark uses an override command).
+- Load: `--max-num-seqs 1`, so concurrent worker callers queue.
+- The Orin member's view (not probed).

lobes_cli-0.92.0/docs/hebrew-realtime.md

@@ -0,0 +1,325 @@
+# Hebrew realtime — a duplex Hebrew voice session with tool calling
+
+An **opt-in overlay** on top of the realtime audio overlay
+([`realtime-pipeline.md`](realtime-pipeline.md)): the same `GET /v1/realtime`
+WebSocket session, serving Hebrew. A person speaks Hebrew, the box answers in
+Hebrew, and the model can call tools that the **client** owns. The English
+overlay's files are byte-identical with or without it.
+
+If you are building a client, read the contract instead:
+[`contracts/realtime-tool-calling.md`](contracts/realtime-tool-calling.md).
+
+> **Status (the #108 rule).** Everything here was built and measured on ONE box
+> — the DGX Spark GB10, 2026-09-18 — from a hand-carried deployment of this
+> branch's wheel. The measurements are real and cited; the *packaged* path
+> (`lobes init --audio --audio-lang he` from a released wheel) is
+> **DECLARED / UNVALIDATED** until an acceptance transcript from that path
+> lands under `docs/evidence/`. No Jetson has run any of it (issue #276).
+
+## The stack
+
+| stage | engine | notes |
+|---|---|---|
+| VAD / turn-taking | Silero, `server_vad` | PulseVAD was tried and false-triggered on all four non-speech clips |
+| STT | `ivrit-ai/whisper-large-v3-turbo`, transformers fp16 (`Dockerfile.whisper-stt`, `listen_server_whisper.py`) | ~170–200 ms per turn |
+| generate | the gateway's own `/v1/chat/completions`, `OPENAI_MODEL=multimodal` | on the Spark: `nvidia/Gemma-4-26B-A4B-NVFP4`, `gemma4` tool + reasoning parsers, 32–44 tok/s; native 262144 window since 2026-09-19 (was 32768; `docs/evidence/2026-09-19-accept-multimodal-gemma26b-native-context-spark.txt`) |
+| TTS (shipped default) | Chatterbox **Multilingual**, `language_id="he"`, phonikud niqqud (`Dockerfile.chatterbox-ml`) | seconds per sentence; needs niqqud; samples non-deterministically, hence a runaway guard |
+| TTS (measured best) | **BlueTTS**, CPU ONNX (`Dockerfile.bluetts`, `lobes.realtime.bluetts_server`) | 93–305 ms per sentence, its own G2P, no niqqud wanted — see *BlueTTS* below for why it is not the default |
+
+Evidence: `docs/evidence/2026-09-hebrew-*.txt` (seven files) and
+`docs/evidence/2026-09-hebrew-realtime-streaming-bluetts-spark.txt`.
+
+### Three things that are easy to get wrong
+
+- **Whisper hallucinates on non-speech** — this fine-tune emits plain text such
+  as `תודה רבה` for a door slam, and its `<|nospeech|>` probability reads 0.000.
+  The mean token log-probability separates cleanly (speech ≥ −0.12,
+  hallucination ≤ −0.58), so the sidecar drops a transcript below
+  `STT_MIN_AVG_LOGPROB` (default −0.35; `off` disables). 4/4 non-speech clips
+  dropped, 22/22 speech clips kept.
+- **Chatterbox Multilingual without niqqud is unusable for Hebrew** (92 % → ~9 %
+  round-trip WER with phonikud), and a Latin-script word derails it.
+- **An NVFP4A16 export of Gemma 4 12B damaged Hebrew** where bf16 and Google's
+  QAT checkpoint did not. Check Hebrew output before promoting a quantization.
+
+### Batch transcription: formats and long clips
+
+`POST /v1/audio/transcriptions` takes any format the image's ffmpeg decodes —
+m4a/aac, mp3, ogg/opus, webm, flac, and WAV at any bit depth — not only 16-bit
+PCM WAV. A 16-bit RIFF/WAVE upload (the realtime bridge's own turn audio) is
+still read directly and never touches ffmpeg; anything else is transcoded to
+16 kHz mono PCM16 first, and an undecodable upload is a 400 `invalid_audio`.
+
+An upload longer than Whisper's 30 s window is **chunked**, not refused: each
+cut goes at the quietest 20 ms frame in the last 5 s of its window, every
+window is judged on its own (silence skip, gates and retries, below), and the
+non-empty pieces are joined with a space. `STT_MAX_AUDIO_SECONDS` (default 600) caps the
+total; past it the reply is a 413 `clip_too_long`. The upload byte cap
+(`STT_MAX_UPLOAD_BYTES`, 16 MiB) still applies first, so ten minutes of
+uncompressed 16 kHz WAV (~19 MB) needs it raised; a compressed voice note does
+not. MEASURED on the DGX Spark, 2026-10-09, with synthetic BlueTTS speech: an
+80 s m4a was transcribed in 3 windows in 3.9 s with every sentence intact at
+the seams; mp3, ogg/opus, webm, 24-bit WAV and m4a single sentences each came
+back verbatim. A real microphone recording is still UNVALIDATED (#108).
+
+### Decode quality: loops, retries and silence (issue #329)
+
+On a real 56-minute recording, the decode above failed in two ways. A two-word
+phrase looped until the token cap, and a loop is high-confidence, so it passed
+the gate. The gate also dropped 61 of 125 chunks whole with no second try, and
+some of them were real speech. Each chunk now goes through the steps below.
+Each step has a knob whose off value gives back the old decode exactly (for
+`STT_LOGPROB_STAT`, the legacy value `mean`).
+
+**Two request policies.** A request of **one** window (30 s or less), which
+covers every realtime turn, and a **multi-window** upload are judged
+differently. The `stt_request` log line names the policy a request used
+(`policy=single_window stat=mean` or `policy=multi_window stat=median`).
+
+| | single window (realtime) | multi-window upload |
+|---|---|---|
+| confidence statistic | `mean`, always (the pre-#329 gate) | `STT_LOGPROB_STAT` (default `median`) |
+| decodes per chunk | one greedy decode, never retried | greedy, then `STT_TEMPERATURE_FALLBACK` |
+| word run of 4–7 | trimmed to 3 | trimmed to 3 |
+| word run of 8 or more | trimmed to 3 | fails the chunk: retry, then drop |
+| Silero VAD | never consulted | every chunk except the last |
+
+The −0.35 threshold was measured on the **mean**. A short silence
+hallucination can fail the mean while passing a median gate: in live data,
+4 tokens with log-probabilities −1.9, −0.1, −0.1, −0.2 have a mean of −0.58
+and a median of −0.15. Among short (6 tokens or fewer) first attempts that
+failed the mean gate, 7 of 24 would have passed a median gate. A long upload
+gains more from the median than it loses, but a realtime turn is usually
+short, so it keeps the mean. It also keeps one greedy decode: re-sampling a
+silent turn's hallucination at a higher temperature until one sample passes
+would add up to two decodes of latency for nothing.
+
+1. **Silence skipping (`STT_VAD`).** On an upload longer than one window,
+   Silero VAD reads each chunk first. Three kinds of chunk are never skipped:
+   a single-window request (the bridge's own VAD already opened that turn, and
+   a one-word reply may hold very little speech), the **last** chunk of an
+   upload (`plan_chunks` leaves it 0–5 s long, so a closing word may hold less
+   than `STT_VAD_MIN_SPEECH_MS` of speech), and a chunk shorter than one 32 ms
+   VAD window (there is nothing to judge, so it counts as speech). If Silero
+   finds less than `STT_VAD_MIN_SPEECH_MS` of speech, Whisper never decodes
+   the chunk, so a silent chunk cannot turn into `תודה רבה`. The scan stops as
+   soon as it has found enough speech, so a speech chunk costs only a few VAD
+   calls. The image installs the `silero-vad` wheel `--no-deps`, because its
+   torchaudio dependency would replace the base image's torch. The server loads
+   the bundled TorchScript model by path. If Silero fails to load, or fails on
+   a call, the server logs a warning and decodes the chunk as before.
+2. **Loop prevention at decode time.** `STT_NO_REPEAT_NGRAM_SIZE` sets the
+   transformers n-gram block. `STT_REPETITION_PENALTY` is off by default,
+   because it penalises every recurring token, Hebrew prefix letters included.
+   Beam search was considered and left out. It would cost about 4× the GPU
+   time on a shared box, and the per-token log-probabilities the gate reads
+   would need different bookkeeping.
+3. **Gates per attempt.** The first is the confidence gate,
+   `STT_MIN_AVG_LOGPROB` (−0.35) on the request policy's statistic (above).
+   The second is OpenAI Whisper's zlib compression ratio,
+   `STT_COMPRESSION_RATIO_MAX`, default 2.4, on both policies. The ratio is
+   taken one byte per character, not over UTF-8. Every Hebrew letter shares
+   one UTF-8 lead byte, which alone pushes ordinary Hebrew prose to about 2.3.
+   Counted per character, the same prose scores about 1.7, like English. Those
+   figures come from synthetic text, not from a recording. The third is the
+   **word-run failure check**, `STT_WORD_RUN_FAIL`, default 8, on multi-window
+   uploads only (next step).
+4. **Word runs: trim, or fail only when far beyond speech.** A word run is one
+   word, or one two-word phrase, repeated back to back. Words are Unicode
+   word tokens compared case-folded and without niqqud, so punctuation between
+   the repeats does not break the run. The n-gram block misses such a run (the
+   punctuation tokens make every n-gram different), and so does the
+   compression ratio (one short run inside a full chunk of speech). Emphatic
+   speech repeats words too ("no no no no, I disagree", or a realtime "yes yes
+   yes yes"), and failing the whole chunk for it would drop 30 s of correct
+   text. So a run of `STT_MAX_WORD_RUN` (4) or more is **trimmed** to
+   `STT_MAX_WORD_RUN − 1` repeats, and the rest of the chunk is kept. The cut
+   goes from the last kept repeat to the end of the run, so the punctuation
+   and words after it follow on. Only a run of `STT_WORD_RUN_FAIL` (8) or
+   more, far beyond normal speech, fails a multi-window chunk with
+   `reason=word_run`. A single window only trims. The trade-off: the live A/B
+   below found hallucinated one-word runs of 11 and 5, both absent from a
+   reference transcript. With these defaults the 11 is re-decoded or dropped,
+   but the 5 is trimmed to 3, so three words of that hallucination stay in the
+   text.
+5. **Retry before dropping (`STT_TEMPERATURE_FALLBACK`, multi-window only).**
+   A chunk that fails any gate is decoded again at each sampling temperature
+   in turn. Sampling uses the full distribution, and the gate undoes the
+   temperature, so it reads the same quantity every time. The first result
+   that passes every gate is kept. If none passes, the chunk is dropped, as
+   before, with the failing gates as its `reason` (`low_logprob`,
+   `compression_ratio`, `word_run`). An empty or tags-only transcript on the
+   first attempt counts as silence and is never retried. An empty retry after
+   a failed attempt is a **drop** (`reason=empty_after_retry`), not a
+   recovery: the first attempt heard something, and it is gone.
+
+| knob | default | off | what it does |
+|---|---|---|---|
+| `STT_LOGPROB_STAT` | `median` | `mean` | Which statistic the confidence gate reads on a multi-window upload. `median` ignores a few rare tokens, such as a name, in an otherwise confident chunk. A single window always reads `mean`. The log lines carry both. |
+| `STT_NO_REPEAT_NGRAM_SIZE` | `6` | `0` | Blocks any repeated token n-gram within a chunk. |
+| `STT_REPETITION_PENALTY` | `1.0` | `1.0` | transformers' repetition penalty. `1.1` is the issue's suggestion. |
+| `STT_COMPRESSION_RATIO_MAX` | `2.4` | `off` | Treats a chunk whose text compresses better than this as a loop. |
+| `STT_MAX_WORD_RUN` | `4` | `0` / `off` | Trims a run of one word or two-word phrase this long to one fewer repeats. Values below 2 fall back to the default. |
+| `STT_WORD_RUN_FAIL` | `8` | `0` / `off` | Fails a multi-window chunk with a run this long (retry, then drop). Values below 2 fall back to the default. |
+| `STT_TEMPERATURE_FALLBACK` | `0.2,0.4` | `off` | Re-decode temperatures for a failing multi-window chunk. Each one is another Whisper decode. |
+| `STT_VAD` | `auto` | `off` | Skips non-speech chunks of a multi-window upload with Silero before decoding. |
+| `STT_VAD_THRESHOLD` | `0.5` | — | Silero speech probability per 32 ms window. |
+| `STT_VAD_MIN_SPEECH_MS` | `250` | — | How much speech makes a chunk worth decoding. |
+
+**Off the event loop.** The handler runs the whole transcription (resample,
+VAD scan and every Whisper decode) in a worker thread, as it already did the
+ffmpeg transcode, so `/v1/health/ready` answers while a long upload decodes.
+One transcription runs at a time, behind one lock, so concurrent requests
+never share the GPU or Silero's recurrent state. A realtime turn that arrives
+during a long upload still waits for that upload to finish.
+
+**Observability.** The stt container logs one line for each retried, dropped,
+trimmed or skipped chunk:
+
+```text
+stt_chunk event=retry chunk=3/18 attempt=0 temperature=0.0 reason=compression_ratio avg_logprob=-0.04 median_logprob=-0.01 compression_ratio=9.80 word_run=30 policy=multi_window stat=median ... tokens=440
+stt_chunk event=dropped chunk=7/18 attempt=2 temperature=0.4 reason=low_logprob ... text='...'
+stt_chunk event=dropped chunk=8/18 attempt=1 temperature=0.2 reason=empty_after_retry ... previous_reason=compression_ratio
+stt_chunk event=trimmed chunk=5/18 word_run=5 runs=1 max_word_run=4
+stt_chunk event=skipped chunk=9/18 reason=no_speech
+```
+
+It also logs one summary line per request:
+`stt_request policy= stat= chunks= decoded= vad_skipped= retried= recovered= dropped= trimmed= decodes= seconds=`.
+Clients cannot see any of this yet. A `dropped[]` field and per-segment
+`avg_logprob` / `compression_ratio` wait for `response_format=verbose_json`
+(#328).
+
+**Live A/B, DGX Spark, 2026-10-23 — why the default is now the median.** The
+first build of this section ran on the live Hebrew `stt` lane against two
+private recordings, cut into 16 requests of 480 s (269 chunks). Counts only:
+
+- **Loops are fixed.** The longest run of one phrase fell from 220 (pre-#329)
+  to 3 at the defaults of the time.
+- **The mean gate was the binding constraint.** It still dropped 118 of 215
+  decoded chunks (55 %); retries recovered 7. Every drop was `low_logprob`.
+- **The median gate dropped 39 of 215 (18 %)**, and every window then carried
+  at least as much text as a reference transcript of the same span (a rough
+  speech reference, not ground truth). That is why `median` became the
+  multi-window default.
+- **It let through a confident hallucination.** In two windows a short word
+  was repeated 11 and 5 times in a row, with punctuation between, and a
+  two-word run of 5 sat in the same span. The word was absent from the
+  reference transcript. That is what the word-run knobs were added for.
+- Silero loaded in the built image and skipped 54 of the 269 chunks.
+
+A second run measured the median default with a word-run gate that **failed**
+the chunk at 4 repeats (an earlier design of step 4): still 39 of 215 dropped
+(18 %), the longest run of one phrase 3, three chunks hitting the word-run
+gate. 15 of the 16 requests in both runs were multi-window. The 16th, the
+20 s tail of one recording, was a single window: it returned 6 characters
+in the second run, where the pre-#329 build returned none. That one request
+is the only single-window evidence, and it predates the single-window
+policy.
+
+**Status (#108).** The single-window/multi-window split, the trim-versus-fail
+word-run design, the never-skipped last chunk, `empty_after_retry` and the
+off-loop, serialized transcription are unit-tested with a fake decoder and a
+fake VAD only; none of them has run live. 18 % still misses the issue's 10 %
+drop target. To validate, re-run the recordings on the live lane, plus a set
+of short realtime turns, and read the `stt_request` and `stt_chunk` lines.
+Acceptance means no phrase repeated more than 3 times in a row, fewer than
+10 % of non-silent chunks dropped, and still no text from silence.
+
+## Latency: what was built, and what it bought
+
+Measured live, a human on a reSpeaker XVF3800, from the moment the server
+decides the turn ended to the first audio byte:
+
+| configuration | first audio |
+|---|---|
+| whole-reply generate + Chatterbox Multilingual | 2.6 s (one sentence), 7.3 s (longer) |
+| + sentence streaming (`GENERATE_STREAM`, default on) + BlueTTS | 0.67–1.0 s |
+| + hidden speculation (`VAD_EAGER_MS=250`) | **1–113 ms** |
+
+…plus, always, the silence the VAD waits through before it believes the turn
+ended (`VAD_SILENCE_MS`). Once speculation is on, that wait is essentially the
+whole perceived delay:
+
+```text
+perceived delay ≈ max( VAD_SILENCE_MS , VAD_EAGER_MS + pipeline )
+pipeline ≈ 600 ms  =  STT ~170  +  first clause ~300  +  TTS ~120
+```
+
+| knob | default | what it does |
+|---|---|---|
+| `GENERATE_STREAM` | `true` | Stream the generate call; speak sentence by sentence. `false` is the exact previous path. **On for every language, English included.** |
+| `REPLY_FIRST_CLAUSE_MIN_CHARS` | `24` | How long the opening clause must be before a comma may end the first spoken piece. `5` on the Spark: first audio ~1.1 s → ~0.6 s, at the price of a short first piece. |
+| `VAD_MIN_LEVEL_PCT` | `0` (off) | **Ignore quiet voices** (background talk, a TV, the next room). A chunk counts as speech only while the PEAK input level — held ~250 ms, like a level meter — is at least this % of full scale; below it, speech opens no turn, interrupts no reply and does not keep a turn open. Silero cannot do this: a far voice scores as high as a near one. On a reSpeaker direct speech peaks at 15–55 % and the room floor at 1–2 %, so 3–5 is a sensible start (`4` on the Spark). Peak, not RMS — speech RMS is only 5–9 %. Depends on microphone gain and AGC: tune per device. |
+| `VAD_EAGER_MS` | `0` (off) | **Hidden speculation.** At this much provisional silence the bridge runs STT → generate → TTS out of sight. Speech resumes → discarded without trace. Turn confirmed → adopted, but only if the real generate request is byte-identical to the speculative one. Costs a wasted STT + generate on every mid-sentence pause. |
+| `CONTINUATION_WINDOW_MS` | `0` (off) | **Continuation merge.** An onset this soon after a silence commit is the speaker carrying on: the reply stops (inside the barge-in guard window too), the half-turn leaves history, both halves are re-transcribed as one turn. One take-back per utterance. What makes a short `VAD_SILENCE_MS` affordable. |
+| `CONTINUATION_TOOL_HOLD_MS` | `500` | With the merge on, a finished tool call is held this long after the commit — a call the client already has cannot be taken back. Speech is never held. A fixed cost on every tool turn. |
+
+The Spark runs `VAD_SILENCE_MS=500`, `VAD_EAGER_MS=160`,
+`CONTINUATION_WINDOW_MS=1200`: plain replies 328–370 ms after the commit
+(~830–870 ms after the speaker stops), tool turns 1.16–1.64 s.
+
+**Honest limits.** The continuation merge has been proven on a recording only —
+no live session has yet produced one. A noise burst with no words in it still
+counts as a barge-in. A merged turn sends two `transcription.completed` events
+(the half, then the whole) and nothing on the wire marks the second as
+superseding the first. English words inside Hebrew speech are misheard and hard
+to follow when spoken (issue #277).
+
+## BlueTTS — built, measured, deliberately not the default
+
+`lobes.realtime.bluetts_server` is a drop-in for the `chatterbox` service (same
+port keys, same `POST /v1/audio/synthesize` → raw PCM16 mono 24 kHz). CPU-only,
+resamples its native 44.1 kHz, strips any niqqud (it runs its own RenikudPlus
+G2P — leave the bridge's `PHONIKUD_MODEL_PATH` unset), and reports ready only
+after a warm-up synthesis has loaded that G2P.
+
+The engine **code** (github.com/maxmelichov/BlueTTS) is MIT and pinned by
+commit; `Dockerfile.bluetts` installs its dependencies from the engine's own
+`uv.lock`, because an unpinned resolve picked a `renikud-plus` that 404s at
+warm-up. The **weights** repo declared **no licence** when this was written, so:
+
+- nothing in this repo downloads the weights or names their repo as a default;
+- the operator downloads them deliberately and mounts the directory at
+  `BLUETTS_ONNX_DIR`;
+- the Hebrew overlay's shipped voice stays Chatterbox Multilingual, and no
+  compose file wires BlueTTS in.
+
+To run it, add a service built from `Dockerfile.bluetts` with that mount and
+point the bridge's `TTS_URL` at it. `BLUETTS_VOICE` (`noa`), `BLUETTS_STEPS`
+(5; 2–3 buys ~20–25 %, quality not auditioned), `BLUETTS_SPEED`.
+
+## Audio hardware notes (both cost hours)
+
+- **Check the pipewire card profile first.** The reSpeaker XVF3800 had flipped
+  to its *digital* output at 34 % volume: almost nothing reached the speaker on
+  its 3.5 mm jack, its echo canceller reported `converged: false` all day, and
+  the session interrupted itself (6 times in 7 turns). `pactl set-card-profile
+  <card> output:analog-stereo+input:analog-stereo` plus full sink volume →
+  `converged: true`, zero self-interruptions. The profile can flip when the
+  device re-enumerates, and the source/sink numeric suffixes change with it.
+- **Wire the speaker to the device that owns the microphone.** An echo
+  canceller only cancels what it plays itself; audio sent to a monitor over
+  HDMI has no reference and makes echo worse. For the same reason you cannot
+  test a session by playing a recording through that speaker — the canceller
+  removes it.
+- **A client must drain its player before exiting.** The server delivers ahead
+  of the playhead; closing the player on `response.done` cut replies
+  mid-sentence. In a browser the same gap shows up as stutter unless playback
+  (re)starts a jitter buffer ahead of now.
+
+## Selecting it
+
+```bash
+lobes init --fleet --audio --audio-lang he        # dry-run; add --apply
+```
+
+materialises `docker-compose.audio-he.yml`, `env.audio-he.example` (merged
+append-only into `.env`), `Dockerfile.whisper-stt`, `Dockerfile.chatterbox-ml`,
+`Dockerfile.bluetts` and `listen_server_whisper.py`; `lobes fleet up` includes
+the overlay when it is present. `--audio` alone, or `--audio-lang en`, is
+exactly the English overlay. Language is an overlay choice — not a shape, not a
+variation.
+
+`GET /capabilities` names what is served: `stt`/`tts` carry the declared
+`model`, `runtime` and — only when declared — `language`.

lobes_cli-0.92.0/docs/nemotron-3.5-bolt-74b-a9b-24gb.md

@@ -0,0 +1,105 @@
+# NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB — the `worker` role
+
+> One entry in lobes's **supported catalog** (`lobes overview --list`). For the
+> catalog-vs-warm distinction — what you *can* load vs. what's loaded *now* —
+> see
+> [`gateway-fleet.md`](gateway-fleet.md#supported-catalog-vs-warm-backends).
+>
+> **Status: the `worker` role_hint (operator decision, 2026-10-26)**,
+> replacing `nvidia/Qwen3.6-35B-A3B-NVFP4` (now a kept candidate — see
+> [`nvidia-qwen3.6-35b-a3b-nvfp4.md`](nvidia-qwen3.6-35b-a3b-nvfp4.md)). The
+> worker contract does not move: Bolt is a fast multimodal (image+video)
+> coder/doer with tool calls, which is exactly what `roles.py` already
+> promises. It is hosted on the **DGX Spark** — the only card family its IQ2
+> experts run on — and every other mesh member reaches `worker` through the
+> mesh. Measured as a standalone container first
+> ([`evidence/2026-10-26-accept-nemotron-bolt-spark.txt`](evidence/2026-10-26-accept-nemotron-bolt-spark.txt)).
+>
+> **The packaged template cannot express Bolt's flags yet.** The fleet
+> `vllm-worker` lane has no knob for `--linear-backend`, the mamba cache
+> flags, `--kv-cache-memory-bytes`, the media limits or the reasoning-parser
+> plugin, and it always passes `--quantization`. The Spark serves it through a
+> full `command:` replacement in its hand-kept `docker-compose.override.yml`
+> (the [recipe](#serving-it-on-the-spark) below). `lobes switch` to this id
+> will not boot on the stock template either (see [Engine](#engine)).
+
+## What it is
+
+| | |
+|---|---|
+| id | `nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB` |
+| revision | `c6cc2e8ffa353403213d6b9b18aaf8eb684d705a` |
+| architecture | `NemotronHPuzzleForCausalLM` in a `nemotron_h_omni` wrapper: hybrid Mamba-2 / attention with sparse MoE, about 9B active of 74B total |
+| modalities | text, image and video (RADIO vision encoder); no audio |
+| context | 262144 native (`llm_config.max_position_embeddings`) |
+| quantization | `modelopt_mixed` (the checkpoint's `quant_method` is modelopt, `MIXED_PRECISION`): IQ2_XXS on the routed experts (~2.06 bits, ggml packing), Q8_0, and W4A16 NVFP4 on the mixer projections — 24.09 GiB loaded |
+| tool parser | `qwen3_coder` (the card's own serve command) |
+| reasoning parser | `ultra_v3`, shipped as a plugin file inside the checkpoint |
+| speculative decoding | DFlash, 7 draft tokens, from `nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr` @ `449d2414…` |
+| licence | OpenMDW 1.1 (the card) |
+
+All of the above was read from the checkpoint's own `config.json` and
+`hf_quant_config.json` at the pinned revision, and from its card's vLLM
+command.
+
+## Engine
+
+- **sm_12x only.** vLLM's IQ2 expert path is gated to Blackwell consumer/GB10
+  silicon. The Jetson AGX Orin (sm_87) refuses it on v0.32.0 and on main; the
+  Jetson AGX Thor (sm_110) has not been tried.
+- **Nightly plus `b12x`.** The card names
+  `vllm/vllm-openai:nightly-77154ca6116ae34cbfe314beef1016bde9b63989` and
+  installs `b12x` at container start (`--linear-backend b12x --moe-backend
+  b12x`). The measured run used that nightly with `b12x==1.5.1` baked into a
+  derived image so the boot runs offline (vLLM `0.32.0rc1.dev100+g77154ca61`).
+
+## Serving it on the Spark
+
+The measured recipe is the card's vLLM command with three deviations:
+
+1. **`--kv-cache-memory-bytes 4294967296` (4 GiB).** On the Spark's unified
+   memory a KV pool sized by `--gpu-memory-utilization` is not
+   deterministic: vLLM's profiling counts other processes' memory growth, so
+   the same flag gives very different pools from boot to boot. Pinning the
+   pool makes the footprint predictable. 4 GiB holds 350,464 fp8 tokens = 1.34
+   full 262144-token requests; `--max-num-seqs` is 1 as on the card.
+2. **`--allowed-local-media-path /` dropped.** That flag lets any caller read
+   any file on the box through a `file://` URL.
+3. **Revisions pinned**, and the server bound to `127.0.0.1`.
+
+The full flag list is in the evidence transcript.
+
+## Measured — DGX Spark GB10, 2026-10-25
+
+From
+[`evidence/2026-10-26-accept-nemotron-bolt-spark.txt`](evidence/2026-10-26-accept-nemotron-bolt-spark.txt):
+
+| | |
+|---|---|
+| weights | 24.09 GiB, 150.8 s to load |
+| KV pool | 4.0 GiB pinned = 350,464 tokens (1.34× at 262144) |
+| host memory available after the probes | 70,146 MiB |
+| known-answer | PASS (`391`, reasoning split off) |
+| image | PASS on red / blue / green solid-colour controls (synthetic) |
+| tool call | PASS — `get_weather({"city": "Haifa"})`, `finish_reason: tool_calls` |
+| speed | 69.3 tok/s wall on one short request (includes prefill and reasoning) |
+| long context | PASS — 231,935-token needle retrieved in 200.3 s |
+
+## Measured — as the `worker` lane, 2026-10-25
+
+PART 2 of the same transcript: the Spark's lobes `vllm-worker` lane (cold
+compile, the same 350,464-token pinned pool), reached as `model=worker` from
+the spark, thor and spark2 gateways — the Thor and spark2 by mesh forward,
+with `X-Lobes-Mesh-Member: spark`. Through the Thor's forward the red/blue
+image controls and a `get_weather` tool call passed.
+
+## Not proven
+
+- The packaged template serving it (the Spark uses an override command), a
+  deployment shape that hosts it, and `lobes switch` to it — switch prints
+  the gear's serving note and, on `--apply`, writes `.env` without
+  restarting.
+- Any card other than the Spark GB10.
+- Concurrency above 1, sustained decode under load, and DFlash acceptance
+  rates.
+- Real-image and video quality (the image probes are synthetic).

{lobes_cli-0.90.3 → lobes_cli-0.92.0}/docs/nvidia-qwen3.6-35b-a3b-nvfp4.md

@@ -1,5 +1,13 @@
 # `nvidia/Qwen3.6-35B-A3B-NVFP4` — the Thor `worker` lane
 
+> **SUPERSEDED as the `worker` role_hint (2026-10-26, operator decision).**
+> The catalog's `worker` moved to `nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB`,
+> hosted on the DGX Spark (see
+> [`nemotron-3.5-bolt-74b-a9b-24gb.md`](nemotron-3.5-bolt-74b-a9b-24gb.md)).
+> This checkpoint is a kept `candidate` (cite-don't-delete); the
+> `thor-worker` shape still renders it. Read "the `worker` lane" below as
+> history.
+
 **Status: VALIDATED live on the physical Jetson AGX Thor, 2026-09-10, on the
 nightly `0.26.1rc1.dev942`; re-accepted on vLLM 0.31.0 (2026-10-17) for the
 items listed in [Engine relabel](#engine-relabel-310-t15) only** — see