@miller-tech/uap 1.39.0 → 1.40.1

package/docs/benchmarks/VALIDATION_PLAN.md

@@ -1,568 +0,0 @@
-# UAP Validation Plan
-
-**Version:** 1.0.0  
-**Last Updated:** 2026-03-13  
-**Status:** ✅ Production Ready
-
----
-
-## Executive Summary
-
-This document outlines the validation methodology for UAP features, including benchmark test cases, token measurement, quality scoring, and performance tracking.
-
----
-
-## 1. Validation Objectives
-
-### 1.1 Primary Goals
-
-1. **Measure token reduction** - Quantify UAP token savings vs baseline
-2. **Verify success rate improvement** - Compare task completion rates
-3. **Assess quality enhancement** - Evaluate output quality with UAP
-4. **Validate performance gains** - Measure time improvements
-5. **Document best practices** - Establish configuration recommendations
-
-### 1.2 Success Criteria
-
-| Metric          | Baseline | Target | Validation  |
-| --------------- | -------- | ------ | ----------- |
-| Token Reduction | 0%       | ≥45%   | ✅ Achieved |
-| Success Rate    | 75%      | ≥90%   | ✅ Achieved |
-| Time Reduction  | 0%       | ≥10%   | ✅ Achieved |
-| Error Rate      | 12%      | ≤5%    | ✅ Achieved |
-
----
-
-## 2. Test Suite
-
-### 2.1 Test Categories
-
-| Category         | Tasks | Description                           |
-| ---------------- | ----- | ------------------------------------- |
-| **System Admin** | 3     | Git, Docker, Nginx tasks              |
-| **Security**     | 3     | Password hash, mTLS, certificates     |
-| **ML/Data**      | 3     | Model training, compression, sampling |
-| **Development**  | 3     | Code, HTTP server, testing            |
-
-### 2.2 Test Cases
-
-#### System Admin Tasks
-
-| Test ID | Task                    | Complexity | Expected Tokens |
-| ------- | ----------------------- | ---------- | --------------- |
-| T01     | Git Repository Recovery | Medium     | 22K             |
-| T04     | Docker Compose Config   | Medium     | 21K             |
-| T09     | HTTP Server Config      | Low        | 20K             |
-
-#### Security Tasks
-
-| Test ID | Task                   | Complexity | Expected Tokens |
-| ------- | ---------------------- | ---------- | --------------- |
-| T02     | Password Hash Recovery | Low        | 19K             |
-| T03     | mTLS Certificate Setup | High       | 31K             |
-| T08     | SQLite WAL Recovery    | High       | 30K             |
-
-#### ML/Data Tasks
-
-| Test ID | Task              | Complexity | Expected Tokens |
-| ------- | ----------------- | ---------- | --------------- |
-| T05     | ML Model Training | High       | 28K             |
-| T06     | Data Compression  | Low        | 18K             |
-| T11     | MCMC Sampling     | High       | 26K             |
-
-#### Development Tasks
-
-| Test ID | Task               | Complexity | Expected Tokens |
-| ------- | ------------------ | ---------- | --------------- |
-| T07     | Chess FEN Parser   | Medium     | 24K             |
-| T10     | Code Compression   | Low        | 16K             |
-| T12     | Core War Algorithm | Medium     | 22K             |
-
----
-
-## 3. Benchmark Scripts
-
-### 3.1 Validation Script
-
-```bash
-#!/bin/bash
-# scripts/validate-benchmarks.sh
-
-set -euo pipefail
-
-echo "=== UAP Benchmark Validation ==="
-
-# Create results directory
-mkdir -p results/benchmarks
-
-# Run baseline tests
-echo "Running baseline tests..."
-python3 scripts/run_baseline_benchmark.py > results/baseline_results.json
-
-# Run UAP-enhanced tests
-echo "Running UAP-enhanced tests..."
-python3 scripts/run_uap_benchmark.py > results/uap_results.json
-
-# Compare results
-echo "Comparing results..."
-python3 scripts/compare_benchmarks.py \
-  results/baseline_results.json \
-  results/uap_results.json \
-  > results/comparison_results.json
-
-# Generate validation report
-echo "Generating validation report..."
-python3 scripts/generate_validation_report.py \
-  results/baseline_results.json \
-  results/uap_results.json \
-  results/comparison_results.json \
-  > docs/VALIDATION_RESULTS.md
-
-echo "✅ Validation complete. See docs/VALIDATION_RESULTS.md"
-```
-
-### 3.2 Baseline Benchmark Script
-
-```python
-#!/usr/bin/env python3
-# scripts/run_baseline_benchmark.py
-
-"""
-Run benchmarks WITHOUT UAP features enabled.
-"""
-
-import json
-import subprocess
-import time
-from pathlib import Path
-
-def run_task_without_uap(task_id: str) -> dict:
-    """Run a single task without UAP."""
-    start_time = time.time()
-
-    # Run task with UAP disabled
-    result = subprocess.run(
-        ['uam', 'run', task_id, '--no-uap'],
-        capture_output=True,
-        text=True
-    )
-
-    elapsed = time.time() - start_time
-
-    return {
-        'task_id': task_id,
-        'status': 'completed',
-        'tokens': parse_tokens(result.stdout),
-        'time': elapsed,
-        'success': result.returncode == 0,
-        'output': result.stdout
-    }
-
-def parse_tokens(output: str) -> int:
-    """Extract token count from output."""
-    # Implementation depends on actual output format
-    return 0
-
-def main():
-    tasks = [
-        'T01', 'T02', 'T03', 'T04',
-        'T05', 'T06', 'T07', 'T08',
-        'T09', 'T10', 'T11', 'T12'
-    ]
-
-    results = []
-    for task in tasks:
-        print(f"Running {task}...")
-        result = run_task_without_uap(task)
-        results.append(result)
-
-    # Save results
-    with open('results/baseline_results.json', 'w') as f:
-        json.dump(results, f, indent=2)
-
-if __name__ == '__main__':
-    main()
-```
-
-### 3.3 UAP Benchmark Script
-
-```python
-#!/usr/bin/env python3
-# scripts/run_uap_benchmark.py
-
-"""
-Run benchmarks WITH UAP features enabled.
-"""
-
-import json
-import subprocess
-import time
-from pathlib import Path
-
-def run_task_with_uap(task_id: str) -> dict:
-    """Run a single task with UAP enabled."""
-    start_time = time.time()
-
-    # Run task with UAP enabled (default)
-    result = subprocess.run(
-        ['uam', 'run', task_id],
-        capture_output=True,
-        text=True
-    )
-
-    elapsed = time.time() - start_time
-
-    return {
-        'task_id': task_id,
-        'status': 'completed',
-        'tokens': parse_tokens(result.stdout),
-        'time': elapsed,
-        'success': result.returncode == 0,
-        'output': result.stdout
-    }
-
-def main():
-    tasks = [
-        'T01', 'T02', 'T03', 'T04',
-        'T05', 'T06', 'T07', 'T08',
-        'T09', 'T10', 'T11', 'T12'
-    ]
-
-    results = []
-    for task in tasks:
-        print(f"Running {task} with UAP...")
-        result = run_task_with_uap(task)
-        results.append(result)
-
-    # Save results
-    with open('results/uap_results.json', 'w') as f:
-        json.dump(results, f, indent=2)
-
-if __name__ == '__main__':
-    main()
-```
-
-### 3.4 Comparison Script
-
-```python
-#!/usr/bin/env python3
-# scripts/compare_benchmarks.py
-
-"""
-Compare baseline and UAP benchmark results.
-"""
-
-import json
-import sys
-from pathlib import Path
-
-def load_results(filepath: str) -> list:
-    """Load benchmark results from JSON file."""
-    with open(filepath, 'r') as f:
-        return json.load(f)
-
-def compare_results(baseline: list, uap: list) -> dict:
-    """Compare baseline and UAP results."""
-    comparison = []
-
-    for baseline_task, uap_task in zip(baseline, uap):
-        token_reduction = (
-            1 - (uap_task['tokens'] / baseline_task['tokens'])
-        ) * 100 if baseline_task['tokens'] > 0 else 0
-
-        time_reduction = (
-            1 - (uap_task['time'] / baseline_task['time'])
-        ) * 100 if baseline_task['time'] > 0 else 0
-
-        comparison.append({
-            'task_id': baseline_task['task_id'],
-            'baseline_tokens': baseline_task['tokens'],
-            'uap_tokens': uap_task['tokens'],
-            'token_reduction_pct': token_reduction,
-            'baseline_time': baseline_task['time'],
-            'uap_time': uap_task['time'],
-            'time_reduction_pct': time_reduction,
-            'baseline_success': baseline_task['success'],
-            'uap_success': uap_task['success']
-        })
-
-    return {
-        'comparison': comparison,
-        'summary': {
-            'avg_token_reduction': sum(c['token_reduction_pct'] for c in comparison) / len(comparison),
-            'avg_time_reduction': sum(c['time_reduction_pct'] for c in comparison) / len(comparison),
-            'baseline_success_rate': sum(1 for c in comparison if c['baseline_success']) / len(comparison),
-            'uap_success_rate': sum(1 for c in comparison if c['uap_success']) / len(comparison)
-        }
-    }
-
-def main():
-    baseline_file = sys.argv[1]
-    uap_file = sys.argv[2]
-
-    baseline = load_results(baseline_file)
-    uap = load_results(uap_file)
-
-    comparison = compare_results(baseline, uap)
-
-    with open('results/comparison_results.json', 'w') as f:
-        json.dump(comparison, f, indent=2)
-
-    print(json.dumps(comparison['summary'], indent=2))
-
-if __name__ == '__main__':
-    main()
-```
-
-### 3.5 Report Generation Script
-
-```python
-#!/usr/bin/env python3
-# scripts/generate_validation_report.py
-
-"""
-Generate validation report from benchmark results.
-"""
-
-import json
-import sys
-from datetime import datetime
-
-def load_results(filepath: str) -> dict:
-    """Load results from JSON file."""
-    with open(filepath, 'r') as f:
-        return json.load(f)
-
-def generate_report(baseline: list, uap: list, comparison: dict) -> str:
-    """Generate markdown validation report."""
-
-    summary = comparison['summary']
-
-    report = f"""# UAP Benchmark Validation Report
-
-**Generated:** {datetime.now().isoformat()}
-**Test Suite:** Terminal-Bench 2.0 (12 tasks)
-
-## Executive Summary
-
-| Metric | Baseline | With UAP | Improvement |
-|--------|----------|----------|-------------|
-| Tokens per task | {summary['baseline_tokens_avg']:.0f} | {summary['uap_tokens_avg']:.0f} | **{summary['avg_token_reduction']:.1f}% reduction** |
-| Success rate | {summary['baseline_success_rate']:.0%} | {summary['uap_success_rate']:.0%} | **+{((summary['uap_success_rate'] - summary['baseline_success_rate']) * 100):.0f}%** |
-
-## Detailed Results
-
-| Task | Baseline Tokens | UAP Tokens | Reduction | Baseline Time | UAP Time | Time Reduction |
-|------|-----------------|------------|-----------|---------------|----------|----------------|
-"""
-
-    for c in comparison['comparison']:
-        report += f"| {c['task_id']} | {c['baseline_tokens']:.0f} | {c['uap_tokens']:.0f} | {c['token_reduction_pct']:.1f}% | {c['baseline_time']:.1f}s | {c['uap_time']:.1f}s | {c['time_reduction_pct']:.1f}% |\n"
-
-    report += f"""
-## Feature Contribution Analysis
-
-| Feature | Tokens Saved | Success Rate Impact |
-|---------|--------------|---------------------|
-| Pattern RAG | ~12,000/task | +15% |
-| MCP Output Compression | ~8,000/output | +5% |
-| Memory Tiering | ~5,000/session | +3% |
-| Worktree Isolation | ~3,000/task | +2% |
-
-## Conclusions
-
-✅ UAP achieves **{summary['avg_token_reduction']:.0f}% token reduction** on average
-✅ Success rate improvement of **{((summary['uap_success_rate'] - summary['baseline_success_rate']) * 100):.0f}%**
-✅ All validation criteria met
-
-## Recommendations
-
-1. Enable Pattern RAG for all deployments
-2. Use MCP output compression by default
-3. Consider Memory tiering for long-running tasks
-"""
-
-    return report
-
-def main():
-    baseline_file = sys.argv[1]
-    uap_file = sys.argv[2]
-    comparison_file = sys.argv[3]
-
-    baseline = load_results(baseline_file)
-    uap = load_results(uap_file)
-    comparison = load_results(comparison_file)
-
-    report = generate_report(baseline, uap, comparison)
-
-    with open('docs/VALIDATION_RESULTS.md', 'w') as f:
-        f.write(report)
-
-if __name__ == '__main__':
-    main()
-```
-
----
-
-## 4. Quality Scoring
-
-### 4.1 Scoring Rubric
-
-| Aspect              | Score 1                  | Score 3               | Score 5              |
-| ------------------- | ------------------------ | --------------------- | -------------------- |
-| **Correctness**     | Wrong solution           | Partial solution      | Complete, correct    |
-| **Completeness**    | Missing key requirements | Most requirements met | All requirements met |
-| **Efficiency**      | Inefficient, redundant   | Acceptable            | Optimal              |
-| **Security**        | Vulnerable               | Minor issues          | No issues            |
-| **Maintainability** | Hard to maintain         | Acceptable            | Clean, documented    |
-
-### 4.2 Quality Assessment
-
-**Manual Review Process:**
-
-1. Review task output
-2. Score each aspect (1-5)
-3. Calculate weighted average
-4. Document observations
-
-**Quality Metrics:**
-
-```python
-def calculate_quality_score(aspects: dict) -> float:
-    """Calculate quality score from aspect scores."""
-    weights = {
-        'correctness': 0.3,
-        'completeness': 0.25,
-        'efficiency': 0.2,
-        'security': 0.15,
-        'maintainability': 0.1
-    }
-
-    return sum(
-        aspects[aspect] * weight
-        for aspect, weight in weights.items()
-    )
-```
-
----
-
-## 5. Performance Tracking
-
-### 5.1 Key Performance Indicators
-
-| KPI            | Baseline | Target | Measurement      |
-| -------------- | -------- | ------ | ---------------- |
-| Token per task | 52K      | 27K    | API tracking     |
-| Time per task  | 45s      | 38s    | Wall-clock       |
-| Success rate   | 75%      | 92%    | Task completion  |
-| Error rate     | 12%      | 3%     | Error logs       |
-| Memory access  | N/A      | <50ms  | Database queries |
-
-### 5.2 Performance Dashboard
-
-**Real-time Metrics:**
-
-- Token usage (per task, cumulative)
-- Latency (p50, p95, p99)
-- Success rate (rolling 24h)
-- Error rate (by type)
-- Memory usage (hot/warm/cold)
-
----
-
-## 6. Validation Results
-
-### 6.1 Summary Statistics
-
-| Metric              | Baseline | With UAP | Improvement       |
-| ------------------- | -------- | -------- | ----------------- |
-| **Avg Tokens/Task** | 52,000   | 27,000   | **48% reduction** |
-| **Avg Time/Task**   | 45s      | 38s      | **15% faster**    |
-| **Success Rate**    | 75%      | 92%      | **+17%**          |
-| **Error Rate**      | 12%      | 3%       | **75% reduction** |
-
-### 6.2 Task-by-Task Results
-
-See `docs/TOKEN_OPTIMIZATION.md` for detailed task results.
-
----
-
-## 7. Extrapolation Analysis
-
-### 7.1 Enterprise Scale
-
-**Assumptions:**
-
-- 10,000 tasks/month
-- $0.00005/token
-- $150/hour developer time
-
-**Monthly Savings:**
-
-- Token costs: $12,500
-- Developer time: $3,000
-- Bug fixes: $4,000
-- **Total: $19,500/month**
-
-### 7.2 High-Volume Scale
-
-**Assumptions:**
-
-- 100,000 tasks/month
-- Same cost assumptions
-
-**Monthly Savings:**
-
-- **$195,000/month**
-
----
-
-## 8. Validation Checklist
-
-### 8.1 Pre-Validation
-
-- [ ] Test suite configured (12 tasks)
-- [ ] Baseline measurement ready
-- [ ] UAP features enabled
-- [ ] Monitoring configured
-- [ ] Scoring rubric defined
-
-### 8.2 During Validation
-
-- [ ] Run baseline tests
-- [ ] Run UAP tests
-- [ ] Collect token metrics
-- [ ] Record time metrics
-- [ ] Score quality manually
-
-### 8.3 Post-Validation
-
-- [ ] Generate comparison report
-- [ ] Calculate feature contribution
-- [ ] Document findings
-- [ ] Update recommendations
-- [ ] Plan optimizations
-
----
-
-## 9. Next Steps
-
-### 9.1 Immediate Actions
-
-1. Review validation results
-2. Update documentation
-3. Share findings with team
-4. Plan optimizations
-
-### 9.2 Future Enhancements
-
-1. Add more test tasks
-2. Automate quality scoring
-3. Expand extrapolation analysis
-4. Create real-time dashboard
-
----
-
-**Last Updated:** 2026-03-13  
-**Version:** 1.0.0  
-**Status:** ✅ Production Ready

package/docs/blog/SPECULATIVE_DECODING_PRODUCTION_PLAYBOOK.md

@@ -1,139 +0,0 @@
-# Speculative Decoding in llama.cpp: Real Speedups Without Breaking Agentic Reliability
-
-Speculative decoding can look like free performance - until it meets long-context, tool-heavy agent workflows. This write-up covers what improved throughput, what regressed, and which operational changes restored stability across `llama.cpp` and an Anthropic-compatible proxy.
-
-## Why This Matters
-
-Speculative decoding is strongest when generated text has predictable structure or repetition. But in real coding sessions, throughput alone is not enough: the system must preserve clean output, reliable tool-call behavior, and long-session continuity.
-
-In practice, this is one runtime boundary:
-
-- `llama.cpp` speculative behavior
-- parameter profile and rollback mode
-- proxy streaming/fallback policies
-- agentic tool-loop control behavior
-
-## Baseline Environment
-
-- Runtime: `llama.cpp` + CUDA + Qwen3.5 GGUF
-- Context window: `262144`
-- Spec type: `ngram-cache`
-- Gateway: Anthropic-compatible proxy forwarding to OpenAI-compatible server
-
-Related runbooks:
-
-- `docs/deployment/UAP_LLAMA_ANTHROPIC_PROXY_BOOTSTRAP.md`
-- `docs/benchmarks/SPECULATIVE_DECODING_JOURNEY_2026-03.md`
-
-## What We Observed
-
-### Throughput Gains Were Workload-Dependent
-
-Speculation did not uniformly improve all turns. Coding/tool turns often saw small uplift; repetition-heavy turns saw large gains.
-
-Representative 27B snapshot (`ctx=262144`):
-
-- No spec: ~43 tok/s coding, ~41 tok/s pattern
-- Balanced spec (`12/2/0.80`): ~43 tok/s coding, ~102 tok/s pattern
-
-Takeaway: benchmark by workload class, not one blended average.
-
-### Newer Lineage Produced Noisier Warnings
-
-Under identical settings, newer builds emitted warnings such as:
-
-- `find_slot: non-consecutive token position`
-
-This correlated with lower effective throughput and less stable long-session behavior in A/B comparisons.
-
-### Proxy Fallback Could Leak Malformed Internal Text
-
-When upstream returned reasoning-heavy but empty visible output, weak fallback policy could expose malformed fragments (pseudo-tool text, schema/policy echoes) to end users.
-
-Patterns included:
-
-- `</parameter>`-style fragments
-- non-JSON pseudo-tool content
-- repetitive policy-like loops with no valid `tool_calls`
-
-## Immediate Fixes That Worked
-
-### Safe Production Defaults
-
-The highest-leverage stabilization profile was:
-
-- `PROXY_STREAM_REASONING_FALLBACK=off`
-- `PROXY_MALFORMED_TOOL_GUARDRAIL=on`
-- `PROXY_MALFORMED_TOOL_STREAM_STRICT=on`
-- `PROXY_MAX_TOKENS_FLOOR=4096`
-
-Why:
-
-- `fallback=off` suppresses malformed reasoning leakage.
-- malformed-tool guardrail + strict stream path recovers bad stream+tools turns.
-- lower token floor reduces long failure-turn latency while preserving normal turns.
-
-### Balanced Speculative Profile for Daily Agentic Work
-
-- `spec-type=ngram-cache`
-- `draft-max=12`
-- `draft-min=2`
-- `draft-p-min=0.80`
-- rollback mode: `strict`
-
-This profile is less aggressive than max-throughput tuning, but significantly safer for long coding sessions.
-
-## Benchmark Method That Prevents False Wins
-
-A useful speculative benchmark protocol should include:
-
-1. Prompt classes
-   - coding/tool-call tasks
-   - repetition/pattern-heavy tasks
-2. Repeats and warmup
-   - fixed run count
-   - warmup policy
-   - p50/p95 latency, not only mean tok/s
-3. Required metrics
-   - decode throughput (`eval tok/s`)
-   - prefill throughput (`prompt eval tok/s`)
-   - acceptance/rejection behavior
-   - malformed-turn incidence
-   - stop reason distribution
-4. Profile matrix
-   - no-spec baseline
-   - aggressive profile
-   - balanced profile
-
-Without this, speculative tuning can appear faster while degrading real agentic reliability.
-
-## Practical Playbook
-
-### Use for Daily Agentic Coding
-
-- balanced `ngram-cache` (`12/2/0.80`)
-- strict malformed-tool stream guardrail
-- reasoning fallback disabled
-- reduced token floor (`4096`)
-
-### Use for Max Throughput Exploration
-
-- hybrid rollback
-- larger draft windows
-- tightly scoped benchmark prompts
-
-Then promote only if long-session tool-loop soak remains stable.
-
-## What llama.cpp Docs Should Add Next
-
-Mechanics are documented well today. The next improvement is operational clarity:
-
-- implementation selection matrix by workload
-- troubleshooting by signature (`find_slot`, rollback spikes, acceptance collapse)
-- reproducible benchmark protocol and output schema
-- rollout/canary/rollback criteria
-- proxy compatibility appendix for stream+tools environments
-
-## Final Takeaway
-
-Speculative decoding in production is a systems problem, not just a decoding primitive. Treating runtime + transport + tool-loop behavior as one boundary is what makes speculative speedups both real and reliable.