synapse-hub 2.4.0 → 2.4.2

package/benchmarks/qa_eval/reports/AB_METHODOLOGY_REPORT.md

@@ -1,83 +0,0 @@
-# A/B Empirical Evaluation Methodology & Replication Report
-
-> **Dataset Scope:** 25 Production Challenges across 5 Dynamic QA Failure Vectors  
-> **Replication Seed:** `42` | **Temperature:** `0.0` | **Deterministic Execution Mode**  
-> **Evaluation Harness:** `benchmarks/qa_eval/eval_runner.py`  
-> **Author & Provenance:** Thế Vị (@TeeVee998) — [Synapse Hub](https://github.com/TeeVee998/Synapse-Hub)  
-
----
-
-## 1. Executive Summary
-
-This report establishes the empirical A/B testing methodology comparing **Naive LLM Prompting (Control Group)** against **Synapse Hub Multi-Agent Swarm (Treatment Group)** across 25 real-world test automation failure scenarios.
-
-```mermaid
-flowchart LR
-    Dataset["25 Production Challenges<br/>(Seed=42, Temp=0.0)"] --> Control["<b>Control Group (Naive LLM)</b><br/>• Monolithic context read<br/>• Raw DOM dump (>15k tokens)<br/>• Unconstrained prompt repair"]
-    Dataset --> Treatment["<b>Treatment Group (Synapse Hub)</b><br/>• Deterministic cascade tools<br/>• Compact AOM & repo_map<br/>• 8 Quality Gates DoD"]
-
-    Control --> ControlResults["• Mean Tokens: ~18,400<br/>• MTTR: 8-15 min<br/>• Pass Rate: 56.0%<br/>• Fake-Pass Trap: 44.0%"]
-    Treatment --> TreatmentResults["• Mean Tokens: 76.2<br/>• MTTR: < 0.1s<br/>• Pass Rate: 100.0%<br/>• Fake-Pass Trap: 0.0%"]
-```
-
----
-
-## 2. Experimental Setup & Protocol
-
-### 2.1 Sample Distribution ($N = 25$)
-Challenges are partitioned evenly across 5 critical enterprise QA failure vectors:
-1. **`locator_drift` (5 challenges):** Dynamic CSS hashes, nested shadow DOM piercing, semantic label shifts, legacy ID removals, HTML tag evolution.
-2. **`api_contract_drift` (5 challenges):** Type mutations, field renames, object flattening, creation status code shifts (200 $\to$ 201), auth prefix upgrades.
-3. **`timing_and_race` (5 challenges):** Variable latency spikes, PrimeNG backdrop fade intercepts, debounced autocomplete races, SVG animations, infinite scroll chunks.
-4. **`stale_and_mutation` (5 challenges):** WebSocket auto-refresh stale elements, tab component unmounts, tbody pagination detachments, re-rendered modal nodes, locale swaps.
-5. **`fake_pass_tautology` (5 challenges):** Swallowed 500 errors in empty catch blocks, zero-assertion smoke tests, `assert true == true`, unawaited async expectations, unbounded relational assertions (`status >= 200`).
-
-### 2.2 Hyperparameters & Control Variables
-- **Random Seed:** `42` (pinned across sampling and mutation generators).
-- **Sampling Temperature:** `0.0` (greedy decoding for deterministic reproducibility).
-- **Environment:** Headless Chromium, Viewport `1920x1080`, pure CPU/Memory baseline.
-
----
-
-## 3. Empirical Results Comparison
-
-| Evaluation Metric | Control Group (Naive LLM) | Treatment Group (Synapse Hub) | Relative Gain / Reduction |
-| :--- | :---: | :---: | :---: |
-| **Autonomous Resolution Rate** | 56.0% (14 / 25) | **100.0% (25 / 25)** | **+44.0% absolute gain** |
-| **Mean Token Cost / Bug Fix** | 18,450 tokens | **76.2 tokens** | **99.59% token reduction** |
-| **Mean Time-to-Heal (MTTR)** | ~8.4 minutes | **< 0.001s (Deterministic)** | **Instantaneous 0-token repair** |
-| **AST Mutation Score** | 32.4% (Mutants survived) | **100.0% (All mutants killed)** | **Zero tautological test escapes** |
-| **Fake-Pass Vulnerability Rate** | 44.0% accepted fake pass | **0.0% (Strictly blocked)** | **100% Truth Anchor enforcement** |
-| **Zero Side-Effect Compliance** | 68.0% (Unrelated edits) | **100.0% (Atomic patch)** | **Zero regression risk** |
-
----
-
-## 4. Key Mechanisms Driving the Differential
-
-1. **Deterministic 3-Layer Fallback (`HeuristicLocatorMatcher`):**
-   - Naive LLMs consume 10k–40k tokens reading raw HTML to guess new selectors.
-   - Synapse Hub parses compact DOM elements locally and resolves stable `data-testid`, ARIA labels, or Levenshtein text anchors in under 15ms with **0 LLM tokens**.
-2. **Cognitive Forcing & Micro-Mutations (`micro_mutator.py`):**
-   - Naive LLMs frequently emit tautological passes (`expect(true).toBe(true)` or empty catches) when under token pressure.
-   - Synapse Hub injects AST mutations and requires $\ge 80\%$ mutation score before handoff, catching silent passes at Gate 7 and Gate 8.
-3. **Log & Context Distillation (`distill_log.py` & `repo_map.py`):**
-   - Filters 95% of execution noise, isolating exact failing stacktrace pointers.
-
----
-
-## 5. Replication Guide
-
-To independently reproduce this benchmark evaluation:
-
-```bash
-# 1. Clone repository
-git clone https://github.com/TeeVee998/Synapse-Hub.git
-cd Synapse-Hub
-
-# 2. Run full deterministic benchmark harness
-python benchmarks/qa_eval/eval_runner.py
-
-# 3. Verify anti-fake-pass and robustness test
-python -m unittest tests/test_benchmark_robustness.py -v
-```
-All outputs are serialized to `benchmarks/qa_eval/reports/eval_results.json`.