synapse-hub 2.4.1 → 2.4.3

package/benchmarks/qa_eval/reports/BENCHMARK_REPORT.md

@@ -1,69 +0,0 @@
-# QA Agent Benchmark Evaluation Report (Eval Harness)
-
-> **Evaluation Suite:** Enterprise QA Automation Self-Healing & Defect Sentinel Benchmark  
-> **Target Framework:** `agent-skills` QA Hub  
-> **Timestamp:** `2026-10-30T17:09:57.038606`  
-> **Discipline:** Truth Anchor (No Fake Pass) + Hard Policy Enforcement + 9 Quality Gates  
-
----
-
-## 1. Executive Summary
-
-| Core Benchmark Metric | Measured Score | Enterprise SLA Target | Status |
-| :--- | :--- | :--- | :--- |
-| **Autonomous Resolution Rate** | **100.0%** | $\ge 90.0\%$ | ✅ PASS |
-| **Mean Time to Resolution (MTTR)** | **0.0s** | $\le 3.00\text{s}$ | ✅ PASS |
-| **Average Token Cost** | **52.4 tokens** | $\le 500\text{ tokens}$ | ✅ PASS |
-| **AST Mutation Score** | **100.0%** | $\ge 80.0\%$ | ✅ PASS |
-| **Zero-Side-Effect Rate** | **100.0%** | $100.0\%$ | ✅ PASS |
-| **Action Safety Firewall Clearance** | **100.0%** | $100.0\%$ | ✅ PASS |
-
----
-
-## 2. Performance Breakdown by Failure Domain
-
-```mermaid
-xychart-beta
-    title "Autonomous Resolution Rate by Domain (%)"
-    x-axis ["Locator Drift", "API Contract", "Timing & Race", "Stale DOM", "Mutation/Tautology"]
-    y-axis "Resolution Rate (%)" 0 --> 100
-    bar [100.0]
-```
-
-| Failure Domain | Total Challenges | Passed | Resolution Rate | Avg MTTR | Avg Tokens |
-| :--- | :---: | :---: | :---: | :---: | :---: |
-| **Locator Drift** | 5 | 5 | 100.0% | 0.0s | 52.4 |
-
----
-
-## 3. Empirical Instrumentation & Measurement Protocol
-
-| Evaluation Domain | Measurement Tool | Verification Mechanism |
-| :--- | :---: | :--- |
-| **Locator Drift Healing** | `tools/heuristic_locator_matcher.py` | 3-Layer matching cascade verified dynamically against DOM element tags & attributes |
-| **API Contract Shift** | `tools/api_drift_sentinel.py` | AST assertion contract validation on response headers and status codes |
-| **Timing & Race Conditions** | `tools/cache_resolver.py` | Smart barrier replacement eliminating `Thread.sleep` and verifying detachment |
-| **Stale Elements** | In-place dynamic refetching | Mutation detachment prevention with dynamic attribute locators |
-| **Tautology / Fake-Pass** | `tools/micro_mutator.py` | Active AST mutant injection (requires $\ge 80\%$ mutation score) |
-| **Destructive Action Guard** | `tools/safety_firewall.py` | Regex and AST pre-flight firewall blocking irreversible actions |
-
----
-
-## 4. Detailed Challenge Matrix
-
-| ID | Challenge Name | Domain | Difficulty | MTTR | Tokens | Mutation Score | Status |
-| :--- | :--- | :--- | :---: | :---: | :---: | :---: | :---: |
-| `QA-CHALLENGE-01` | Dynamic CSS-in-JS Hash Locator Shift | `locator_drift` | MEDIUM | 0.000s | 50 | 100.0% | ✅ PASS |
-| `QA-CHALLENGE-02` | Nested Shadow DOM Component Piercing | `locator_drift` | HARD | 0.000s | 62 | 100.0% | ✅ PASS |
-| `QA-CHALLENGE-03` | Semantic Text Label Evolution | `locator_drift` | EASY | 0.000s | 50 | 100.0% | ✅ PASS |
-| `QA-CHALLENGE-04` | Legacy ID Removed in Favor of data-testid | `locator_drift` | EASY | 0.000s | 50 | 100.0% | ✅ PASS |
-| `QA-CHALLENGE-05` | HTML Tag Evolution: Anchor to ARIA Button | `locator_drift` | MEDIUM | 0.000s | 50 | 100.0% | ✅ PASS |
-
----
-
-## 5. Verification & Methodological Rigor
-
-All benchmark challenges were evaluated against deterministic test oracles:
-1. **Deterministic Resolution:** Healed selectors and patches were verified against DOM snapshots and API schemas.
-2. **Mutation Quality Verification:** Tautology challenges were verified using `tools/micro_mutator.py` to ensure assertions caught injected mutants.
-3. **Safety Clearance:** Action statements were evaluated using `tools/safety_firewall.py` in dry-run mode to confirm zero destructive side effects.

package/benchmarks/qa_eval/reports/eval_results.json

@@ -1,105 +0,0 @@
-{
-  "timestamp": "2026-10-30T17:09:57.038606",
-  "total_challenges": 5,
-  "passed_challenges": 5,
-  "failed_challenges": 0,
-  "resolution_rate_pct": 100.0,
-  "avg_mttr_sec": 0.0,
-  "avg_token_consumption": 52.4,
-  "avg_mutation_score": 100.0,
-  "zero_side_effect_rate_pct": 100.0,
-  "safety_firewall_clearance_pct": 100.0,
-  "categories": {
-    "locator_drift": {
-      "total": 5,
-      "passed": 5,
-      "tokens": 262,
-      "time": 0.0002,
-      "resolution_rate": 100.0,
-      "avg_tokens": 52.4,
-      "avg_time": 0.0
-    }
-  },
-  "details": [
-    {
-      "id": "QA-CHALLENGE-01",
-      "name": "Dynamic CSS-in-JS Hash Locator Shift",
-      "category": "locator_drift",
-      "difficulty": "MEDIUM",
-      "status": "PASSED",
-      "resolved": true,
-      "safety_cleared": true,
-      "time_to_heal_sec": 0.0001,
-      "max_time_allowed_sec": 2.0,
-      "time_compliant": true,
-      "token_consumption": 50,
-      "mutation_score": 100.0,
-      "zero_side_effect": true,
-      "resolution_notes": "Autonomous cascade healed: 'button.btn-submit' -> '[data-testid='submit-order-btn']' (confidence 0.98)"
-    },
-    {
-      "id": "QA-CHALLENGE-02",
-      "name": "Nested Shadow DOM Component Piercing",
-      "category": "locator_drift",
-      "difficulty": "HARD",
-      "status": "PASSED",
-      "resolved": true,
-      "safety_cleared": true,
-      "time_to_heal_sec": 0.0,
-      "max_time_allowed_sec": 3.0,
-      "time_compliant": true,
-      "token_consumption": 62,
-      "mutation_score": 100.0,
-      "zero_side_effect": true,
-      "resolution_notes": "Autonomous cascade healed: '#customer-email' -> 'template:has-text('<input id='customer-email' type='email' data-testid='user-email'/>')' (confidence 0.94)"
-    },
-    {
-      "id": "QA-CHALLENGE-03",
-      "name": "Semantic Text Label Evolution",
-      "category": "locator_drift",
-      "difficulty": "EASY",
-      "status": "PASSED",
-      "resolved": true,
-      "safety_cleared": true,
-      "time_to_heal_sec": 0.0001,
-      "max_time_allowed_sec": 1.5,
-      "time_compliant": true,
-      "token_consumption": 50,
-      "mutation_score": 100.0,
-      "zero_side_effect": true,
-      "resolution_notes": "Autonomous cascade healed: '//button[text()='Submit Order']' -> '#order-cta' (confidence 0.9)"
-    },
-    {
-      "id": "QA-CHALLENGE-04",
-      "name": "Legacy ID Removed in Favor of data-testid",
-      "category": "locator_drift",
-      "difficulty": "EASY",
-      "status": "PASSED",
-      "resolved": true,
-      "safety_cleared": true,
-      "time_to_heal_sec": 0.0,
-      "max_time_allowed_sec": 1.5,
-      "time_compliant": true,
-      "token_consumption": 50,
-      "mutation_score": 100.0,
-      "zero_side_effect": true,
-      "resolution_notes": "Autonomous cascade healed: '#customer_name' -> '[data-testid='customer-name-field']' (confidence 0.98)"
-    },
-    {
-      "id": "QA-CHALLENGE-05",
-      "name": "HTML Tag Evolution: Anchor to ARIA Button",
-      "category": "locator_drift",
-      "difficulty": "MEDIUM",
-      "status": "PASSED",
-      "resolved": true,
-      "safety_cleared": true,
-      "time_to_heal_sec": 0.0,
-      "max_time_allowed_sec": 2.0,
-      "time_compliant": true,
-      "token_consumption": 50,
-      "mutation_score": 100.0,
-      "zero_side_effect": true,
-      "resolution_notes": "Autonomous cascade healed: 'a.nav-export' -> 'button' (confidence 0.9)"
-    }
-  ]
-}