synapse-hub 2.4.1 → 2.4.2

package/benchmarks/qa_eval/dataset/timing_and_race/qa-challenge-14_timing_and_race.json

@@ -1,22 +1,22 @@
-{
-  "id": "QA-CHALLENGE-14",
-  "name": "Async Chart SVG Animation Race Condition",
-  "category": "timing_and_race",
-  "difficulty": "MEDIUM",
-  "description": "Financial chart takes 1200ms to calculate series and render SVG paths.",
-  "failure_vector": {
-    "type": "ASYNC_RENDER_RACE",
-    "broken_code": "page.click('.chart-bar:last-child')",
-    "error_log": "ElementNotFoundException: Selector '.chart-bar:last-child' not found",
-    "environment_condition": "Highcharts Rendering Animation"
-  },
-  "expected_resolution": {
-    "healed_code": "page.waitForSelector('.highcharts-series path', state: 'visible'); page.click('.chart-bar:last-child')",
-    "strategy": "SVG_VISIBILITY_SMART_WAIT"
-  },
-  "benchmark_targets": {
-    "max_heal_time_sec": 2.0,
-    "max_token_cost": 310,
-    "min_confidence_score": 0.9
-  }
+{
+  "id": "QA-CHALLENGE-14",
+  "name": "Async Chart SVG Animation Race Condition",
+  "category": "timing_and_race",
+  "difficulty": "MEDIUM",
+  "description": "Financial chart takes 1200ms to calculate series and render SVG paths.",
+  "failure_vector": {
+    "type": "ASYNC_RENDER_RACE",
+    "broken_code": "page.click('.chart-bar:last-child')",
+    "error_log": "ElementNotFoundException: Selector '.chart-bar:last-child' not found",
+    "environment_condition": "Highcharts Rendering Animation"
+  },
+  "expected_resolution": {
+    "healed_code": "page.waitForSelector('.highcharts-series path', state: 'visible'); page.click('.chart-bar:last-child')",
+    "strategy": "SVG_VISIBILITY_SMART_WAIT"
+  },
+  "benchmark_targets": {
+    "max_heal_time_sec": 2.0,
+    "max_token_cost": 310,
+    "min_confidence_score": 0.9
+  }
 }

package/benchmarks/qa_eval/dataset/timing_and_race/qa-challenge-15_timing_and_race.json

@@ -1,22 +1,22 @@
-{
-  "id": "QA-CHALLENGE-15",
-  "name": "Infinite Scroll Lazy Loaded Chunk Detach",
-  "category": "timing_and_race",
-  "difficulty": "MEDIUM",
-  "description": "Table rows 50-100 require viewport scroll to trigger network chunk download.",
-  "failure_vector": {
-    "type": "LAZY_LOAD_RACE",
-    "broken_code": "page.locator('tr[data-id=\"75\"]').click()",
-    "error_log": "TimeoutException: Timed out waiting for 'tr[data-id=\"75\"]' to be visible",
-    "environment_condition": "Virtual Scrolling Table"
-  },
-  "expected_resolution": {
-    "healed_code": "page.locator('tr[data-id=\"75\"]').scrollIntoViewIfNeeded(); page.locator('tr[data-id=\"75\"]').click()",
-    "strategy": "VIRTUAL_SCROLL_INTO_VIEW"
-  },
-  "benchmark_targets": {
-    "max_heal_time_sec": 2.0,
-    "max_token_cost": 300,
-    "min_confidence_score": 0.92
-  }
+{
+  "id": "QA-CHALLENGE-15",
+  "name": "Infinite Scroll Lazy Loaded Chunk Detach",
+  "category": "timing_and_race",
+  "difficulty": "MEDIUM",
+  "description": "Table rows 50-100 require viewport scroll to trigger network chunk download.",
+  "failure_vector": {
+    "type": "LAZY_LOAD_RACE",
+    "broken_code": "page.locator('tr[data-id=\"75\"]').click()",
+    "error_log": "TimeoutException: Timed out waiting for 'tr[data-id=\"75\"]' to be visible",
+    "environment_condition": "Virtual Scrolling Table"
+  },
+  "expected_resolution": {
+    "healed_code": "page.locator('tr[data-id=\"75\"]').scrollIntoViewIfNeeded(); page.locator('tr[data-id=\"75\"]').click()",
+    "strategy": "VIRTUAL_SCROLL_INTO_VIEW"
+  },
+  "benchmark_targets": {
+    "max_heal_time_sec": 2.0,
+    "max_token_cost": 300,
+    "min_confidence_score": 0.92
+  }
 }

package/benchmarks/qa_eval/reports/BENCHMARK_REPORT.md

@@ -1,69 +1,94 @@
-# QA Agent Benchmark Evaluation Report (Eval Harness)
-
-> **Evaluation Suite:** Enterprise QA Automation Self-Healing & Defect Sentinel Benchmark  
-> **Target Framework:** `agent-skills` QA Hub  
-> **Timestamp:** `2026-10-30T17:09:57.038606`  
-> **Discipline:** Truth Anchor (No Fake Pass) + Hard Policy Enforcement + 9 Quality Gates  
-
----
-
-## 1. Executive Summary
-
-| Core Benchmark Metric | Measured Score | Enterprise SLA Target | Status |
-| :--- | :--- | :--- | :--- |
-| **Autonomous Resolution Rate** | **100.0%** | $\ge 90.0\%$ | ✅ PASS |
-| **Mean Time to Resolution (MTTR)** | **0.0s** | $\le 3.00\text{s}$ | ✅ PASS |
-| **Average Token Cost** | **52.4 tokens** | $\le 500\text{ tokens}$ | ✅ PASS |
-| **AST Mutation Score** | **100.0%** | $\ge 80.0\%$ | ✅ PASS |
-| **Zero-Side-Effect Rate** | **100.0%** | $100.0\%$ | ✅ PASS |
-| **Action Safety Firewall Clearance** | **100.0%** | $100.0\%$ | ✅ PASS |
-
----
-
-## 2. Performance Breakdown by Failure Domain
-
-```mermaid
-xychart-beta
-    title "Autonomous Resolution Rate by Domain (%)"
-    x-axis ["Locator Drift", "API Contract", "Timing & Race", "Stale DOM", "Mutation/Tautology"]
-    y-axis "Resolution Rate (%)" 0 --> 100
-    bar [100.0]
-```
-
-| Failure Domain | Total Challenges | Passed | Resolution Rate | Avg MTTR | Avg Tokens |
-| :--- | :---: | :---: | :---: | :---: | :---: |
-| **Locator Drift** | 5 | 5 | 100.0% | 0.0s | 52.4 |
-
----
-
-## 3. Empirical Instrumentation & Measurement Protocol
-
-| Evaluation Domain | Measurement Tool | Verification Mechanism |
-| :--- | :---: | :--- |
-| **Locator Drift Healing** | `tools/heuristic_locator_matcher.py` | 3-Layer matching cascade verified dynamically against DOM element tags & attributes |
-| **API Contract Shift** | `tools/api_drift_sentinel.py` | AST assertion contract validation on response headers and status codes |
-| **Timing & Race Conditions** | `tools/cache_resolver.py` | Smart barrier replacement eliminating `Thread.sleep` and verifying detachment |
-| **Stale Elements** | In-place dynamic refetching | Mutation detachment prevention with dynamic attribute locators |
-| **Tautology / Fake-Pass** | `tools/micro_mutator.py` | Active AST mutant injection (requires $\ge 80\%$ mutation score) |
-| **Destructive Action Guard** | `tools/safety_firewall.py` | Regex and AST pre-flight firewall blocking irreversible actions |
-
----
-
-## 4. Detailed Challenge Matrix
-
-| ID | Challenge Name | Domain | Difficulty | MTTR | Tokens | Mutation Score | Status |
-| :--- | :--- | :--- | :---: | :---: | :---: | :---: | :---: |
-| `QA-CHALLENGE-01` | Dynamic CSS-in-JS Hash Locator Shift | `locator_drift` | MEDIUM | 0.000s | 50 | 100.0% | ✅ PASS |
-| `QA-CHALLENGE-02` | Nested Shadow DOM Component Piercing | `locator_drift` | HARD | 0.000s | 62 | 100.0% | ✅ PASS |
-| `QA-CHALLENGE-03` | Semantic Text Label Evolution | `locator_drift` | EASY | 0.000s | 50 | 100.0% | ✅ PASS |
-| `QA-CHALLENGE-04` | Legacy ID Removed in Favor of data-testid | `locator_drift` | EASY | 0.000s | 50 | 100.0% | ✅ PASS |
-| `QA-CHALLENGE-05` | HTML Tag Evolution: Anchor to ARIA Button | `locator_drift` | MEDIUM | 0.000s | 50 | 100.0% | ✅ PASS |
-
----
-
-## 5. Verification & Methodological Rigor
-
-All benchmark challenges were evaluated against deterministic test oracles:
-1. **Deterministic Resolution:** Healed selectors and patches were verified against DOM snapshots and API schemas.
-2. **Mutation Quality Verification:** Tautology challenges were verified using `tools/micro_mutator.py` to ensure assertions caught injected mutants.
-3. **Safety Clearance:** Action statements were evaluated using `tools/safety_firewall.py` in dry-run mode to confirm zero destructive side effects.
+# QA Agent Benchmark Evaluation Report (Eval Harness)
+
+> **Evaluation Suite:** Enterprise QA Automation Self-Healing & Defect Sentinel Benchmark  
+> **Target Framework:** `agent-skills` QA Hub  
+> **Timestamp:** `2026-11-01T21:24:43.176168`  
+> **Run Scope:** FULL — 25/25 challenges across all 5 domains.  
+> **Discipline:** Truth Anchor (No Fake Pass) + Hard Policy Enforcement + 9 Quality Gates  
+
+---
+
+## 1. Executive Summary
+
+| Core Benchmark Metric | Measured Score | Enterprise SLA Target | Status |
+| :--- | :--- | :--- | :--- |
+| **Autonomous Resolution Rate** | **100.0%** | $\ge 90.0\%$ | ✅ PASS |
+| **Mean Time to Resolution (MTTR)** | **0.0s** | $\le 3.00\text{s}$ | ✅ PASS |
+| **Average Token Cost** | **76.2 tokens** | $\le 500\text{ tokens}$ | ✅ PASS |
+| **AST Mutation Score** | **100.0%** | $\ge 80.0\%$ | ✅ PASS |
+| **Zero-Side-Effect Rate** | **100.0%** | $100.0\%$ | ✅ PASS |
+| **Action Safety Firewall Clearance** | **100.0%** | $100.0\%$ | ✅ PASS |
+
+---
+
+## 2. Performance Breakdown by Failure Domain
+
+```mermaid
+xychart-beta
+    title "Autonomous Resolution Rate by Domain (%)"
+    x-axis ["API Contract", "Mutation/Tautology", "Locator Drift", "Stale DOM", "Timing & Race"]
+    y-axis "Resolution Rate (%)" 0 --> 100
+    bar [100.0, 100.0, 100.0, 100.0, 100.0]
+```
+
+| Failure Domain | Total Challenges | Passed | Resolution Rate | Avg MTTR | Avg Tokens |
+| :--- | :---: | :---: | :---: | :---: | :---: |
+| **Api Contract Drift** | 5 | 5 | 100.0% | 0.0s | 77.8 |
+| **Fake Pass Tautology** | 5 | 5 | 100.0% | 0.0s | 67.0 |
+| **Locator Drift** | 5 | 5 | 100.0% | 0.0s | 52.4 |
+| **Stale And Mutation** | 5 | 5 | 100.0% | 0.0s | 94.6 |
+| **Timing And Race** | 5 | 5 | 100.0% | 0.0s | 89.4 |
+
+---
+
+## 3. Empirical Instrumentation & Measurement Protocol
+
+| Evaluation Domain | Measurement Tool | Verification Mechanism |
+| :--- | :---: | :--- |
+| **Locator Drift Healing** | `tools/heuristic_locator_matcher.py` | 3-Layer matching cascade verified dynamically against DOM element tags & attributes |
+| **API Contract Shift** | `tools/api_drift_sentinel.py` | AST assertion contract validation on response headers and status codes |
+| **Timing & Race Conditions** | `tools/cache_resolver.py` | Smart barrier replacement eliminating `Thread.sleep` and verifying detachment |
+| **Stale Elements** | In-place dynamic refetching | Mutation detachment prevention with dynamic attribute locators |
+| **Tautology / Fake-Pass** | `tools/micro_mutator.py` | Active AST mutant injection (requires $\ge 80\%$ mutation score) |
+| **Destructive Action Guard** | `tools/safety_firewall.py` | Regex and AST pre-flight firewall blocking irreversible actions |
+
+---
+
+## 4. Detailed Challenge Matrix
+
+| ID | Challenge Name | Domain | Difficulty | MTTR | Tokens | Mutation Score | Status |
+| :--- | :--- | :--- | :---: | :---: | :---: | :---: | :---: |
+| `QA-CHALLENGE-06` | Numeric ID Serialized as String (Type Mutation) | `api_contract_drift` | MEDIUM | 0.000s | 77 | 100.0% | ✅ PASS |
+| `QA-CHALLENGE-07` | Payload Response Field Rename | `api_contract_drift` | EASY | 0.000s | 91 | 100.0% | ✅ PASS |
+| `QA-CHALLENGE-08` | Nested Response Object Flattened | `api_contract_drift` | MEDIUM | 0.000s | 82 | 100.0% | ✅ PASS |
+| `QA-CHALLENGE-09` | HTTP Creation Status Code Shift (200 OK -> 201 Created) | `api_contract_drift` | EASY | 0.000s | 64 | 100.0% | ✅ PASS |
+| `QA-CHALLENGE-10` | Authentication Header Prefix Upgrade | `api_contract_drift` | EASY | 0.000s | 75 | 100.0% | ✅ PASS |
+| `QA-CHALLENGE-21` | Swallowed HTTP 500 Error in Empty Catch Block | `fake_pass_tautology` | EASY | 0.000s | 65 | 100.0% | ✅ PASS |
+| `QA-CHALLENGE-22` | Zero-Assertion Smoke Test (Tautological Pass) | `fake_pass_tautology` | EASY | 0.000s | 84 | 100.0% | ✅ PASS |
+| `QA-CHALLENGE-23` | Tautological assert true == true | `fake_pass_tautology` | EASY | 0.000s | 58 | 100.0% | ✅ PASS |
+| `QA-CHALLENGE-24` | Unawaited Async Assertion (Silent Green Trap) | `fake_pass_tautology` | MEDIUM | 0.000s | 64 | 100.0% | ✅ PASS |
+| `QA-CHALLENGE-25` | Unbounded Relational Assertion (status >= 200) | `fake_pass_tautology` | EASY | 0.000s | 64 | 100.0% | ✅ PASS |
+| `QA-CHALLENGE-01` | Dynamic CSS-in-JS Hash Locator Shift | `locator_drift` | MEDIUM | 0.000s | 50 | 100.0% | ✅ PASS |
+| `QA-CHALLENGE-02` | Nested Shadow DOM Component Piercing | `locator_drift` | HARD | 0.000s | 62 | 100.0% | ✅ PASS |
+| `QA-CHALLENGE-03` | Semantic Text Label Evolution | `locator_drift` | EASY | 0.000s | 50 | 100.0% | ✅ PASS |
+| `QA-CHALLENGE-04` | Legacy ID Removed in Favor of data-testid | `locator_drift` | EASY | 0.000s | 50 | 100.0% | ✅ PASS |
+| `QA-CHALLENGE-05` | HTML Tag Evolution: Anchor to ARIA Button | `locator_drift` | MEDIUM | 0.000s | 50 | 100.0% | ✅ PASS |
+| `QA-CHALLENGE-16` | WebSocket Table Auto-Refresh Stale Reference | `stale_and_mutation` | HARD | 0.000s | 88 | 100.0% | ✅ PASS |
+| `QA-CHALLENGE-17` | Tab Switch Component Unmount Detach | `stale_and_mutation` | MEDIUM | 0.000s | 103 | 100.0% | ✅ PASS |
+| `QA-CHALLENGE-18` | Pagination Replaces tbody (Stale Row Index) | `stale_and_mutation` | MEDIUM | 0.000s | 97 | 100.0% | ✅ PASS |
+| `QA-CHALLENGE-19` | Reopened Modal Generates Fresh DOM Node | `stale_and_mutation` | MEDIUM | 0.000s | 111 | 100.0% | ✅ PASS |
+| `QA-CHALLENGE-20` | Locale Swap Invalidates Hardcoded Text Selectors | `stale_and_mutation` | EASY | 0.000s | 74 | 100.0% | ✅ PASS |
+| `QA-CHALLENGE-11` | Network Latency Spike (3,500ms Throttle) Breaking Static Sleep | `timing_and_race` | MEDIUM | 0.000s | 88 | 100.0% | ✅ PASS |
+| `QA-CHALLENGE-12` | PrimeNG Modal Backdrop Fading Intercept | `timing_and_race` | HARD | 0.000s | 91 | 100.0% | ✅ PASS |
+| `QA-CHALLENGE-13` | Debounced Search Input Autocomplete Race | `timing_and_race` | MEDIUM | 0.000s | 104 | 100.0% | ✅ PASS |
+| `QA-CHALLENGE-14` | Async Chart SVG Animation Race Condition | `timing_and_race` | MEDIUM | 0.000s | 82 | 100.0% | ✅ PASS |
+| `QA-CHALLENGE-15` | Infinite Scroll Lazy Loaded Chunk Detach | `timing_and_race` | MEDIUM | 0.000s | 82 | 100.0% | ✅ PASS |
+
+---
+
+## 5. Verification & Methodological Rigor
+
+All benchmark challenges were evaluated against deterministic test oracles:
+1. **Deterministic Resolution:** Healed selectors and patches were verified against DOM snapshots and API schemas.
+2. **Mutation Quality Verification:** Tautology challenges were verified using `tools/micro_mutator.py` to ensure assertions caught injected mutants.
+3. **Safety Clearance:** Action statements were evaluated using `tools/safety_firewall.py` in dry-run mode to confirm zero destructive side effects.