@miller-tech/uap 1.13.14 → 1.13.15

package/docs/INDEX.md

@@ -43,6 +43,7 @@
 
 - [Validation Plan](benchmarks/VALIDATION_PLAN.md) -- Benchmark methodology and test cases
 - [Validation Results](benchmarks/VALIDATION_RESULTS.md) -- Benchmark results
+- [Speculative Decoding Journey (2026-03)](benchmarks/SPECULATIVE_DECODING_JOURNEY_2026-03.md) -- End-to-end tuning, fixes, and soak outcomes
 - [Token Optimization](benchmarks/TOKEN_OPTIMIZATION.md) -- Per-feature token savings analysis
 - [Accuracy Analysis](benchmarks/ACCURACY_ANALYSIS.md) -- Internal vs Terminal-Bench comparison
 

package/docs/benchmarks/SPECULATIVE_DECODING_JOURNEY_2026-03.md

@@ -0,0 +1,221 @@
+# Speculative Decoding Journey (2026-03)
+
+This document records the end-to-end speculative decoding stabilization journey across `llama.cpp` runtime tuning and `uap-anthropic-proxy` guardrails, including fixes, benchmark results, and the production profile now in use.
+
+## Scope
+
+- Runtime: `llama.cpp` with Qwen3.5 models, CUDA, `ctx-size=262144`.
+- Gateway: Anthropic-compatible proxy (`tools/agents/scripts/anthropic_proxy.py`).
+- Client behavior: agentic coding loops with tool calls (Claude Code style).
+
+## Goals
+
+1. Preserve high speculative decoding throughput.
+2. Eliminate pathological loops and malformed visible output.
+3. Keep tool-call behavior reliable under long sessions.
+4. Keep production context window at `262144`.
+
+## Phase 1 - Llama.cpp Speculative Stability
+
+### Problems Observed
+
+- Rollback loops and instability under aggressive speculative settings.
+- `find_slot` and related server warnings during long agentic sessions.
+- Throughput regressions compared to known fast baseline.
+
+### Work Performed
+
+- Implemented and tested multiple rollback strategies in `llama.cpp` worktree branches.
+- Compared baseline fast commit vs newer speculative logic.
+- Restored proven fast runtime path for production service while preserving learned guardrails.
+
+### Key Runtime Decisions
+
+- Keep production on fast validated binary lineage (`029edcafc` baseline family).
+- Use strict balanced speculative profile for 35B operations:
+  - `speculative.n_max=12`
+  - `speculative.n_min=2`
+  - `speculative.p_min=0.80`
+
+### Representative Throughput Findings
+
+- Qwen3.5-27B, `ctx=262144`, q4 KV cache:
+  - No spec: ~43 tok/s coding, ~41 tok/s pattern.
+  - Spec (balanced): ~43 tok/s coding, ~102 tok/s pattern.
+  - Main uplift appears in pattern-heavy turns, not all coding turns.
+
+## Phase 2 - Proxy Reasoning Fallback Leak Fix
+
+### Problems Observed
+
+- Empty visible output (`output_tokens=0`) with large hidden reasoning payloads.
+- Proxy emitted malformed chain-of-thought text as fallback, causing user-visible garbage:
+  - repeated fragments like `</parameter>`, tool schema echoes, policy text loops.
+
+### Fixes Implemented
+
+- Added explicit streaming fallback policy:
+  - `PROXY_STREAM_REASONING_FALLBACK=off|sanitized|visible`
+  - `PROXY_STREAM_REASONING_MAX_CHARS`
+- Set production default to `off`.
+
+### Result
+
+- Malformed reasoning fallback leakage is suppressed by default.
+- Debugging remains possible with `sanitized`/`visible` modes when intentionally enabled.
+
+## Phase 3 - Token Floor and Prune Controls
+
+### Problems Observed
+
+- Hardcoded `max_tokens` floor (`16384`) forced very long failure turns.
+- Pruning threshold flag alone could trigger pruning path without meaningful message reduction.
+
+### Fixes Implemented
+
+- Added configurable max token floor:
+  - `PROXY_MAX_TOKENS_FLOOR` (`0` disables floor)
+- Added configurable prune target:
+  - `PROXY_CONTEXT_PRUNE_TARGET_FRACTION`
+
+### Live A/B Result (Production-Like)
+
+`PROXY_MAX_TOKENS_FLOOR=16384` vs `4096`:
+
+- Silent reasoning-heavy turn:
+  - `16384`: avg `78.749s`
+  - `4096`: avg `19.777s`
+  - Latency reduction: ~`74.9%`
+  - Predicted throughput unchanged (~`208 tok/s` class)
+- Normal tool turns remained stable and slightly faster with `4096`.
+
+## Phase 4 - Malformed Tool-Loop Hardening
+
+### Problem Pattern
+
+Under adversarial or degraded prompt states, the model can emit pseudo-tool text instead of valid tool calls, e.g.:
+
+- `</parameter>` fragments
+- echoed policy snippets (`you MUST call a tool...`)
+- long no-progress text with no `tool_calls`
+
+### Feature Set Added (Flag Controlled)
+
+1. **Malformed tool guardrail + retry**
+   - `PROXY_MALFORMED_TOOL_GUARDRAIL`
+   - `PROXY_MALFORMED_TOOL_RETRY_MAX`
+   - `PROXY_MALFORMED_TOOL_RETRY_MAX_TOKENS`
+   - `PROXY_MALFORMED_TOOL_RETRY_TEMPERATURE`
+
+2. **Strict stream guardrail path**
+   - `PROXY_MALFORMED_TOOL_STREAM_STRICT`
+   - For stream+tools requests, proxy runs guarded non-stream upstream call, then replays SSE.
+
+3. **Tool narrowing (optional)**
+   - `PROXY_TOOL_NARROWING`
+   - `PROXY_TOOL_NARROWING_KEEP`
+   - `PROXY_TOOL_NARROWING_MIN_TOOLS`
+
+4. **Disable thinking on tool turns (optional)**
+   - `PROXY_DISABLE_THINKING_ON_TOOL_TURNS`
+
+5. **Session contamination breaker (optional safety net)**
+   - `PROXY_SESSION_CONTAMINATION_BREAKER`
+   - `PROXY_SESSION_CONTAMINATION_THRESHOLD`
+   - `PROXY_SESSION_CONTAMINATION_KEEP_LAST`
+
+6. **Agentic supplement mode**
+   - `PROXY_AGENTIC_SUPPLEMENT_MODE=clean|legacy`
+
+### Test Coverage
+
+- Unit tests in `tools/agents/tests/test_anthropic_proxy_streaming.py`
+- Current targeted suite count in this workstream: `16` passing tests.
+
+## Benchmark Highlights (Per-Option Toggles)
+
+### Artifact Stress Benchmark (v3)
+
+Source: `/tmp/proxy_visibility_benchmark_v3.json`
+
+| Mode | Key Flags | Outcome Summary |
+| --- | --- | --- |
+| Baseline | none | no tool call, policy-echo text surfaced |
+| Option 1 | malformed guardrail + strict stream | malformed detected and retried; returned `tool_use` with empty visible text |
+| Option 2 | tool narrowing only | not sufficient alone in stress case |
+| Option 3 | disable thinking only | not sufficient alone in stress case |
+| Option 4 | contamination breaker only | not sufficient alone in this synthetic workload |
+| Option 5 | clean supplement only | not sufficient alone in stress case |
+
+### Practical Conclusion
+
+- Strongest primary mitigation: **Option 1** (malformed guardrail + strict stream + bounded retry).
+- Other options are secondary tuning aids and should not replace Option 1 for this failure class.
+
+## 10-Turn Live Stability Soak
+
+Source: `/tmp/proxy_10turn_soak_results.json`
+
+- 10 turns, alternating malformed-stress and normal tool-call turns, single live session id.
+- Results:
+  - Error rate: `0.0%`
+  - Malformed visible output rate (stress turns): `0.0%`
+  - Normal tool-call success rate: `100.0%`
+  - Duration p50/p95: `10.2s` / `21.366s`
+  - Stop reasons: `tool_use=6`, `max_tokens=3`, `end_turn=1`
+
+## Production Profile (Current)
+
+File: `/home/cogtek/.config/uap/anthropic-proxy.env`
+
+```bash
+PROXY_MAX_TOKENS_FLOOR=4096
+PROXY_STREAM_REASONING_FALLBACK=off
+
+PROXY_MALFORMED_TOOL_GUARDRAIL=on
+PROXY_MALFORMED_TOOL_STREAM_STRICT=on
+PROXY_MALFORMED_TOOL_RETRY_MAX=1
+PROXY_MALFORMED_TOOL_RETRY_MAX_TOKENS=512
+PROXY_MALFORMED_TOOL_RETRY_TEMPERATURE=0
+
+PROXY_TOOL_NARROWING=off
+PROXY_DISABLE_THINKING_ON_TOOL_TURNS=off
+PROXY_SESSION_CONTAMINATION_BREAKER=off
+PROXY_AGENTIC_SUPPLEMENT_MODE=legacy
+```
+
+Rationale:
+
+- Keep the strongest practical fix enabled (malformed guardrail + strict stream path).
+- Keep latency-optimized floor (`4096`).
+- Keep optional secondary heuristics off unless new evidence warrants enablement.
+
+## Reproduction Checklist
+
+1. Restart services:
+
+```bash
+systemctl --user restart uap-llama-server.service
+systemctl --user restart uap-anthropic-proxy.service
+```
+
+2. Run targeted unit tests:
+
+```bash
+python3 -m pytest tools/agents/tests/test_anthropic_proxy_streaming.py -q
+```
+
+3. Run soak script (or equivalent alternating malformed/normal stream sequence).
+
+4. Validate logs:
+
+- `MALFORMED TOOL PAYLOAD`
+- `MALFORMED RETRY ...`
+- `STRICT STREAM GUARDRAIL`
+- Absence of user-visible malformed fragments.
+
+## Open Follow-Ups
+
+- Add a dedicated persistent benchmark harness under `scripts/` for this exact soak profile.
+- Add branch/commit links from `llama.cpp` worktrees for cross-repo traceability.
+- Optionally evaluate enabling `PROXY_TOOL_NARROWING` in production only after longer mixed-workload soak data.

package/docs/deployment/QWEN35_LLAMA_CPP.md

@@ -221,11 +221,38 @@ All settings are via environment variables:
 | `PROXY_LOG_LEVEL`       | `INFO`                               | Logging level (DEBUG/INFO/WARNING/ERROR) |
 | `PROXY_READ_TIMEOUT`    | `600`                                | Read timeout (seconds) for LLM streaming |
 | `PROXY_MAX_CONNECTIONS` | `20`                                 | Max concurrent upstream connections      |
+| `PROXY_MAX_TOKENS_FLOOR` | `16384`                             | Minimum floor applied to incoming `max_tokens` (`0` disables floor) |
+| `PROXY_CONTEXT_PRUNE_TARGET_FRACTION` | `0.65`            | Target context utilization after pruning (`0.0 < value < 1.0`) |
 | `PROXY_STREAM_REASONING_FALLBACK` | `off`                      | Streaming behavior for reasoning-only empty turns (`off`, `sanitized`, `visible`) |
 | `PROXY_STREAM_REASONING_MAX_CHARS` | `240`                      | Max fallback length when `PROXY_STREAM_REASONING_FALLBACK=sanitized` |
+| `PROXY_TOOL_NARROWING` | `off`                                | Narrow large tool lists to top relevant tools per turn |
+| `PROXY_TOOL_NARROWING_KEEP` | `8`                             | Number of tools to keep when narrowing is enabled |
+| `PROXY_TOOL_NARROWING_MIN_TOOLS` | `12`                       | Minimum tool count before narrowing activates |
+| `PROXY_DISABLE_THINKING_ON_TOOL_TURNS` | `off`                 | Sends `enable_thinking=false` when tools are present |
+| `PROXY_MALFORMED_TOOL_GUARDRAIL` | `on`                      | Detects malformed pseudo tool payloads and retries with strict settings |
+| `PROXY_MALFORMED_TOOL_RETRY_MAX` | `1`                        | Number of malformed-tool retries |
+| `PROXY_MALFORMED_TOOL_RETRY_MAX_TOKENS` | `2048`             | Retry cap for `max_tokens` during malformed-tool recovery |
+| `PROXY_MALFORMED_TOOL_RETRY_TEMPERATURE` | `0`                | Retry temperature for malformed-tool recovery |
+| `PROXY_MALFORMED_TOOL_STREAM_STRICT` | `off`                 | For stream+tools requests, use guarded non-stream upstream path then replay SSE |
+| `PROXY_SESSION_CONTAMINATION_BREAKER` | `on`                 | Resets long-running malformed sessions to recent context |
+| `PROXY_SESSION_CONTAMINATION_THRESHOLD` | `3`                | Consecutive malformed turns before reset |
+| `PROXY_SESSION_CONTAMINATION_KEEP_LAST` | `8`                 | Number of latest messages to preserve during contamination reset |
+| `PROXY_AGENTIC_SUPPLEMENT_MODE` | `clean`                   | Agentic system supplement variant (`clean`, `legacy`) |
 
 For agentic coding workloads, keep `PROXY_STREAM_REASONING_FALLBACK=off` (default) to avoid leaking malformed internal reasoning as user-visible output. Use `sanitized` only for debugging.
 
+For Claude Code + Qwen malformed-tool loops, recommended starting profile:
+
+```bash
+PROXY_STREAM_REASONING_FALLBACK=off
+PROXY_MAX_TOKENS_FLOOR=4096
+PROXY_MALFORMED_TOOL_GUARDRAIL=on
+PROXY_TOOL_NARROWING=on
+PROXY_DISABLE_THINKING_ON_TOOL_TURNS=on
+PROXY_SESSION_CONTAMINATION_BREAKER=on
+PROXY_AGENTIC_SUPPLEMENT_MODE=clean
+```
+
 ### Example: Custom upstream
 
 ```bash

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@miller-tech/uap",
-  "version": "1.13.14",
+  "version": "1.13.15",
   "description": "Autonomous AI agent memory system with CLAUDE.md protocol enforcement",
   "type": "module",
   "main": "dist/index.js",