synapse-hub 2.4.1 → 2.4.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/rules/graphify.md +14 -14
- package/.agents/rules/mandatory-playwright-live.md +29 -29
- package/.agents/skills/bats-testing-patterns/resources/implementation-playbook.md +614 -614
- package/.agents/skills/clean-pom-crafter/examples/standard_test_case.groovy +28 -28
- package/.agents/skills/clean-pom-crafter/references/pom_patterns.groovy +138 -138
- package/.agents/skills/e2e-testing-patterns/resources/implementation-playbook.md +522 -522
- package/.agents/skills/failure-autopsy-and-healing/references/healing_matrix.md +64 -64
- package/.agents/skills/failure-autopsy-and-healing/references/qa_investigation_protocol.md +56 -56
- package/.agents/skills/failure-autopsy-and-healing/scripts/triage_failures.sh +98 -98
- package/.agents/skills/katalon-cli-runner/references/groovy-patterns.md +209 -209
- package/.agents/skills/katalon-cli-runner/references/katalon-api.md +155 -155
- package/.agents/skills/katalon-cli-runner/references/runner_commands.md +67 -67
- package/.agents/skills/playwright-live-interaction/references/advanced_superpowers.md +160 -160
- package/.agents/skills/playwright-live-interaction/references/compact_accessibility_snapshots.md +75 -75
- package/.agents/skills/playwright-live-interaction/references/recipes.md +388 -388
- package/.agents/skills/playwright-live-interaction/references/swarm_orchestration.md +134 -134
- package/.agents/skills/wcag-audit-patterns/resources/implementation-playbook.md +541 -541
- package/.agents/templates/completion_report.md +57 -57
- package/.agents/workflows/graphify.md +10 -10
- package/.cursor/rules/language.mdc +7 -7
- package/README.en.md +1 -1
- package/README.md +1 -1
- package/benchmarks/qa_eval/dataset/api_contract_drift/qa-challenge-06_api_contract_drift.json +21 -21
- package/benchmarks/qa_eval/dataset/api_contract_drift/qa-challenge-07_api_contract_drift.json +21 -21
- package/benchmarks/qa_eval/dataset/api_contract_drift/qa-challenge-08_api_contract_drift.json +21 -21
- package/benchmarks/qa_eval/dataset/api_contract_drift/qa-challenge-09_api_contract_drift.json +21 -21
- package/benchmarks/qa_eval/dataset/api_contract_drift/qa-challenge-10_api_contract_drift.json +21 -21
- package/benchmarks/qa_eval/dataset/fake_pass_tautology/qa-challenge-21_fake_pass_tautology.json +20 -20
- package/benchmarks/qa_eval/dataset/fake_pass_tautology/qa-challenge-22_fake_pass_tautology.json +20 -20
- package/benchmarks/qa_eval/dataset/fake_pass_tautology/qa-challenge-23_fake_pass_tautology.json +20 -20
- package/benchmarks/qa_eval/dataset/fake_pass_tautology/qa-challenge-24_fake_pass_tautology.json +20 -20
- package/benchmarks/qa_eval/dataset/fake_pass_tautology/qa-challenge-25_fake_pass_tautology.json +20 -20
- package/benchmarks/qa_eval/dataset/locator_drift/qa-challenge-01_locator_drift.json +22 -22
- package/benchmarks/qa_eval/dataset/locator_drift/qa-challenge-02_locator_drift.json +22 -22
- package/benchmarks/qa_eval/dataset/locator_drift/qa-challenge-03_locator_drift.json +22 -22
- package/benchmarks/qa_eval/dataset/locator_drift/qa-challenge-04_locator_drift.json +22 -22
- package/benchmarks/qa_eval/dataset/locator_drift/qa-challenge-05_locator_drift.json +22 -22
- package/benchmarks/qa_eval/dataset/stale_and_mutation/qa-challenge-16_stale_and_mutation.json +21 -21
- package/benchmarks/qa_eval/dataset/stale_and_mutation/qa-challenge-17_stale_and_mutation.json +21 -21
- package/benchmarks/qa_eval/dataset/stale_and_mutation/qa-challenge-18_stale_and_mutation.json +21 -21
- package/benchmarks/qa_eval/dataset/stale_and_mutation/qa-challenge-19_stale_and_mutation.json +21 -21
- package/benchmarks/qa_eval/dataset/stale_and_mutation/qa-challenge-20_stale_and_mutation.json +22 -22
- package/benchmarks/qa_eval/dataset/timing_and_race/qa-challenge-11_timing_and_race.json +21 -21
- package/benchmarks/qa_eval/dataset/timing_and_race/qa-challenge-12_timing_and_race.json +21 -21
- package/benchmarks/qa_eval/dataset/timing_and_race/qa-challenge-13_timing_and_race.json +21 -21
- package/benchmarks/qa_eval/dataset/timing_and_race/qa-challenge-14_timing_and_race.json +21 -21
- package/benchmarks/qa_eval/dataset/timing_and_race/qa-challenge-15_timing_and_race.json +21 -21
- package/bin/agent-qa.js +0 -0
- package/docs/ARCHITECTURE.md +12 -1
- package/package.json +1 -1
- package/scripts/install.ps1 +112 -112
- package/scripts/lint_skills.ps1 +57 -57
- package/skills_manifest.json +1 -1
- package/synapse.config.json +1 -1
- package/tools/__init__.py +1 -1
- package/tools/agent_canary.py +501 -501
- package/tools/api_fuzzer.py +11 -2
- package/tools/browser_helpers.js +2 -2
- package/tools/bump_version.py +1 -0
- package/tools/check_env.py +71 -13
- package/tools/cli.py +2 -2
- package/tools/dashboard_generator.py +2 -2
- package/tools/e2e_flow_prober.sh +0 -0
- package/tools/generate_allure_report.sh +0 -0
- package/tools/generate_test_reports.sh +0 -0
- package/tools/mcp_excel_server.py +3 -1
- package/tools/mcp_excel_server.sh +0 -0
- package/tools/mcp_server.py +1 -1
- package/tools/pinpoint_failure.py +88 -88
- package/tools/setup_project.py +8 -0
- package/tools/update_sentinel.py +10 -9
- package/tools/verify_all.sh +0 -0
- package/tools/visual_som_helper.py +14 -3
- package/tools/welcome_page.py +1 -1
- package/benchmarks/qa_eval/reports/AB_METHODOLOGY_REPORT.md +0 -83
- package/benchmarks/qa_eval/reports/BENCHMARK_REPORT.md +0 -69
- package/benchmarks/qa_eval/reports/eval_results.json +0 -105
package/tools/api_fuzzer.py
CHANGED
|
@@ -115,12 +115,21 @@ class ApiFuzzer:
|
|
|
115
115
|
parsed = urllib.parse.urlparse(self.base_url)
|
|
116
116
|
host = (parsed.hostname or "").lower().strip()
|
|
117
117
|
is_safe_env = self._is_safe_non_prod_host(host)
|
|
118
|
-
|
|
118
|
+
allowed_hosts = {
|
|
119
|
+
h.strip().lower() for h in os.environ.get("SYNAPSE_ALLOW_PROD_HOSTS", "").split(",") if h.strip()
|
|
120
|
+
}
|
|
121
|
+
is_host_specifically_allowed = host in allowed_hosts
|
|
122
|
+
global_override = os.environ.get("SYNAPSE_ALLOW_PROD_FUZZ") in ("1", "true", "TRUE")
|
|
123
|
+
|
|
124
|
+
if host and not is_safe_env and not (is_host_specifically_allowed or global_override):
|
|
119
125
|
raise PermissionError(
|
|
120
126
|
f"[SAFETY FIREWALL - REJECTED] Target host '{host}' is not in the allowed non-production environment allowlist "
|
|
121
127
|
"(localhost, private IP, .local/.internal, or verified dev/staging/qa/test subdomain). "
|
|
122
|
-
"
|
|
128
|
+
f"To authorize this host specifically, set SYNAPSE_ALLOW_PROD_HOSTS='{host}' or SYNAPSE_ALLOW_PROD_FUZZ=1 if authorized."
|
|
123
129
|
)
|
|
130
|
+
if host and not is_safe_env and global_override and not is_host_specifically_allowed:
|
|
131
|
+
print(f" ⚠️ [SECURITY WARNING] SYNAPSE_ALLOW_PROD_FUZZ=1 global override active for host '{host}'. "
|
|
132
|
+
f"Prefer setting SYNAPSE_ALLOW_PROD_HOSTS='{host}' for host-scoped permission.")
|
|
124
133
|
|
|
125
134
|
# Prepare SSL context (support self-signed staging certificates via explicit --insecure flag)
|
|
126
135
|
self.ssl_context = ssl.create_default_context()
|
package/tools/browser_helpers.js
CHANGED
|
@@ -13,7 +13,7 @@
|
|
|
13
13
|
(function () {
|
|
14
14
|
'use strict';
|
|
15
15
|
|
|
16
|
-
if (window.__qa && window.__qa.__version === '2.4.
|
|
16
|
+
if (window.__qa && window.__qa.__version === '2.4.3') {
|
|
17
17
|
return window.__qa;
|
|
18
18
|
}
|
|
19
19
|
|
|
@@ -47,7 +47,7 @@
|
|
|
47
47
|
}
|
|
48
48
|
|
|
49
49
|
const qa = {
|
|
50
|
-
__version: '2.4.
|
|
50
|
+
__version: '2.4.3',
|
|
51
51
|
|
|
52
52
|
/**
|
|
53
53
|
* 1. Smart Action Batching
|
package/tools/bump_version.py
CHANGED
|
@@ -196,6 +196,7 @@ def main():
|
|
|
196
196
|
update_file("tools/mcp_excel_server.ps1", r'(version\s*=\s*")[^"]+(")', rf"\g<1>{new_ver}\g<2>", args.dry_run)
|
|
197
197
|
|
|
198
198
|
# 7b. tools/mcp_excel_server.py
|
|
199
|
+
update_file("tools/mcp_excel_server.py", r'(SERVER_VERSION\s*=\s*")[^"]+(")', rf"\g<1>{new_ver}\g<2>", args.dry_run)
|
|
199
200
|
update_file("tools/mcp_excel_server.py", r'("serverInfo":\s*\{[^}]*"version":\s*")[^"]+(")', rf"\g<1>{new_ver}\g<2>", args.dry_run)
|
|
200
201
|
|
|
201
202
|
# 8. docs/ARCHITECTURE.md
|
package/tools/check_env.py
CHANGED
|
@@ -75,6 +75,7 @@ class DiagnosticRunner:
|
|
|
75
75
|
self.results = []
|
|
76
76
|
self.warnings = []
|
|
77
77
|
self.errors = []
|
|
78
|
+
self.is_standalone = False
|
|
78
79
|
|
|
79
80
|
def log_check(self, name, status, message, fix_hint=""):
|
|
80
81
|
badge = f"{GREEN}[PASS]{RESET}" if status == "PASS" else (f"{YELLOW}[WARN]{RESET}" if status == "WARN" else f"{RED}[FAIL]{RESET}")
|
|
@@ -206,8 +207,8 @@ class DiagnosticRunner:
|
|
|
206
207
|
self.log_check("Python Virtual Environment", "PASS", f"Python {py_ver} (Optional: create .venv if using auxiliary runners)")
|
|
207
208
|
|
|
208
209
|
def check_project_structure(self):
|
|
210
|
+
# 1. QA Hub Repository Check
|
|
209
211
|
is_hub = (self.target_dir / "skills_manifest.json").exists() or (self.target_dir / ".agents" / "skills").is_dir()
|
|
210
|
-
|
|
211
212
|
if is_hub:
|
|
212
213
|
required_dirs = [".agents/skills", ".agents/rules", "tools", "tests"]
|
|
213
214
|
missing = [rd for rd in required_dirs if not (self.target_dir / rd).exists()]
|
|
@@ -220,24 +221,78 @@ class DiagnosticRunner:
|
|
|
220
221
|
"Restore from official agent-skills Git repository.")
|
|
221
222
|
return
|
|
222
223
|
|
|
223
|
-
|
|
224
|
+
# 2. Standalone Host / User Home Detection
|
|
225
|
+
is_user_home = False
|
|
226
|
+
try:
|
|
227
|
+
is_user_home = self.target_dir.resolve() == Path.home().resolve()
|
|
228
|
+
except Exception:
|
|
229
|
+
pass
|
|
230
|
+
|
|
231
|
+
has_project_marker = any([
|
|
232
|
+
(self.target_dir / marker).exists() for marker in [
|
|
233
|
+
".git", "package.json", "requirements.txt", "pom.xml", "build.gradle",
|
|
234
|
+
"pytest.ini", "cypress.config.ts", "cypress.config.js",
|
|
235
|
+
"playwright.config.ts", "playwright.config.js", "Keywords", "Scripts"
|
|
236
|
+
]
|
|
237
|
+
]) or bool(list(self.target_dir.glob("*.prj")))
|
|
238
|
+
|
|
239
|
+
if is_user_home or not has_project_marker:
|
|
240
|
+
self.is_standalone = True
|
|
241
|
+
self.log_check("Project Structure (Standalone)", "PASS",
|
|
242
|
+
"Host machine environment verified. (To diagnose a specific test project: agent-qa doctor <path>)")
|
|
243
|
+
return
|
|
244
|
+
|
|
245
|
+
# 3. Playwright Check
|
|
246
|
+
pkg_json = self.target_dir / "package.json"
|
|
247
|
+
has_playwright_pkg = pkg_json.exists() and "@playwright/test" in pkg_json.read_text(encoding="utf-8", errors="ignore")
|
|
248
|
+
is_playwright = (self.target_dir / "playwright.config.ts").exists() or (self.target_dir / "playwright.config.js").exists() or has_playwright_pkg
|
|
224
249
|
if is_playwright:
|
|
225
|
-
|
|
250
|
+
has_tests = (self.target_dir / "tests").is_dir() or (self.target_dir / "e2e").is_dir() or (self.target_dir / "src").is_dir()
|
|
251
|
+
if has_tests:
|
|
252
|
+
self.log_check("Project Structure (Playwright)", "PASS", "Playwright test project structure verified")
|
|
253
|
+
else:
|
|
254
|
+
self.log_check("Project Structure (Playwright)", "WARN", "Missing 'tests/' or 'e2e/' directory for Playwright specs",
|
|
255
|
+
"Create a 'tests/' directory or run 'npx playwright test --init'")
|
|
256
|
+
return
|
|
257
|
+
|
|
258
|
+
# 4. Cypress Check
|
|
259
|
+
is_cypress = (self.target_dir / "cypress.config.ts").exists() or (self.target_dir / "cypress.config.js").exists() or (self.target_dir / "cypress").is_dir()
|
|
260
|
+
if is_cypress:
|
|
261
|
+
if (self.target_dir / "cypress").is_dir():
|
|
262
|
+
self.log_check("Project Structure (Cypress)", "PASS", "Cypress test project structure verified")
|
|
263
|
+
else:
|
|
264
|
+
self.log_check("Project Structure (Cypress)", "WARN", "Missing 'cypress/' directory",
|
|
265
|
+
"Create a 'cypress/' directory for E2E specs")
|
|
266
|
+
return
|
|
267
|
+
|
|
268
|
+
# 5. Pytest / Python Selenium Check
|
|
269
|
+
is_pytest = (self.target_dir / "pytest.ini").exists() or (self.target_dir / "conftest.py").exists()
|
|
270
|
+
has_py_tests = (self.target_dir / "tests").is_dir() or bool(list(self.target_dir.glob("test_*.py")))
|
|
271
|
+
if is_pytest or (has_py_tests and (self.target_dir / "requirements.txt").exists()):
|
|
272
|
+
self.log_check("Project Structure (Pytest)", "PASS", "Python/Pytest test project structure verified")
|
|
273
|
+
return
|
|
274
|
+
|
|
275
|
+
# 6. Katalon Studio (Enterprise Web Clean POM)
|
|
276
|
+
is_katalon = bool(list(self.target_dir.glob("*.prj"))) or (self.target_dir / "Keywords").is_dir() or (self.target_dir / "Scripts").is_dir()
|
|
277
|
+
if is_katalon:
|
|
278
|
+
required_dirs = ["Scripts", "Keywords", "Profiles"]
|
|
226
279
|
missing = [rd for rd in required_dirs if not (self.target_dir / rd).exists()]
|
|
227
280
|
if not missing:
|
|
228
|
-
self.log_check("Project Structure (
|
|
281
|
+
self.log_check("Project Structure (Clean POM)", "PASS", "Clean POM structure verified (Keywords, Profiles, Scripts)")
|
|
229
282
|
else:
|
|
230
|
-
self.log_check("Project Structure (
|
|
283
|
+
self.log_check("Project Structure (POM)", "WARN" if len(missing) < len(required_dirs) else "FAIL",
|
|
284
|
+
f"Missing directories: {', '.join(missing)}",
|
|
285
|
+
"Verify git clone command or select target test project directory.")
|
|
231
286
|
return
|
|
232
287
|
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
|
|
288
|
+
# 7. Generic Test Project (any project with tests/ or test/)
|
|
289
|
+
if (self.target_dir / "tests").is_dir() or (self.target_dir / "test").is_dir() or (self.target_dir / "e2e").is_dir():
|
|
290
|
+
self.log_check("Project Structure (Automated Tests)", "PASS", "Standard automated test directory detected")
|
|
291
|
+
return
|
|
292
|
+
|
|
293
|
+
# Fallback: General project structure missing test suite
|
|
294
|
+
self.log_check("Project Structure", "WARN", "No automated test directories detected (tests/, Scripts/, cypress/)",
|
|
295
|
+
"Run 'agent-qa setup .' to initialize Clean POM / Playwright configuration for this project.")
|
|
241
296
|
|
|
242
297
|
def check_network(self):
|
|
243
298
|
env_file = self.target_dir / ".env"
|
|
@@ -253,6 +308,9 @@ class DiagnosticRunner:
|
|
|
253
308
|
|
|
254
309
|
base = os.environ.get("BASE_URL", "").rstrip("/")
|
|
255
310
|
if not base:
|
|
311
|
+
if getattr(self, "is_standalone", False):
|
|
312
|
+
self.log_check("Network / Server Base URL", "PASS", "Host standalone check (configure BASE_URL in project .env when targeting an AUT)")
|
|
313
|
+
return
|
|
256
314
|
self.log_check("Network / Server Base URL", "WARN", "BASE_URL not configured in .env",
|
|
257
315
|
"Copy .env.example to .env and configure BASE_URL to verify connectivity.")
|
|
258
316
|
return
|
package/tools/cli.py
CHANGED
|
@@ -41,7 +41,7 @@ RESET = "\033[0m"
|
|
|
41
41
|
def cmd_version(args):
|
|
42
42
|
"""Displays the current hub version, skill catalog count, and license information."""
|
|
43
43
|
manifest_path = os.path.join(REPO_ROOT, "skills_manifest.json")
|
|
44
|
-
version = "2.4.
|
|
44
|
+
version = "2.4.3"
|
|
45
45
|
if os.path.exists(manifest_path):
|
|
46
46
|
try:
|
|
47
47
|
import json
|
|
@@ -516,7 +516,7 @@ def cmd_mcp_excel(args):
|
|
|
516
516
|
|
|
517
517
|
def _hub_version():
|
|
518
518
|
manifest_path = os.path.join(REPO_ROOT, "skills_manifest.json")
|
|
519
|
-
version = "2.4.
|
|
519
|
+
version = "2.4.3"
|
|
520
520
|
if os.path.exists(manifest_path):
|
|
521
521
|
try:
|
|
522
522
|
import json
|
|
@@ -1885,7 +1885,7 @@ def generate_hub_html(
|
|
|
1885
1885
|
|
|
1886
1886
|
translations_vi = {
|
|
1887
1887
|
"header_title": "Bảng Điều Hành Chất Lượng Doanh Nghiệp",
|
|
1888
|
-
"header_subtitle": "Synapse-Hub v2.4.
|
|
1888
|
+
"header_subtitle": "Synapse-Hub v2.4.2 • Hệ Sinh Thái Kiểm Thử AI Agent Tự Chủ",
|
|
1889
1889
|
"nav_welcome": "Giới Thiệu",
|
|
1890
1890
|
"meta_mode": "🏷️ Chế độ: <strong>HỆ SINH THÁI QA HUB</strong>",
|
|
1891
1891
|
"meta_gen": f"🕒 Khởi tạo: <strong>{now_str}</strong>",
|
|
@@ -1963,7 +1963,7 @@ def generate_hub_html(
|
|
|
1963
1963
|
|
|
1964
1964
|
translations_en = {
|
|
1965
1965
|
"header_title": "Enterprise QA Executive Dashboard",
|
|
1966
|
-
"header_subtitle": "Synapse-Hub v2.4.
|
|
1966
|
+
"header_subtitle": "Synapse-Hub v2.4.2 • Autonomous AI Agent Testing Ecosystem",
|
|
1967
1967
|
"nav_welcome": "Welcome",
|
|
1968
1968
|
"meta_mode": "🏷️ Mode: <strong>QA HUB ECOSYSTEM</strong>",
|
|
1969
1969
|
"meta_gen": f"🕒 Generated: <strong>{now_str}</strong>",
|
package/tools/e2e_flow_prober.sh
CHANGED
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
@@ -23,6 +23,8 @@ if hasattr(sys.stdout, 'reconfigure'):
|
|
|
23
23
|
except Exception:
|
|
24
24
|
pass
|
|
25
25
|
|
|
26
|
+
SERVER_VERSION = "2.4.3"
|
|
27
|
+
|
|
26
28
|
NS_MAIN = "{http://schemas.openxmlformats.org/spreadsheetml/2006/main}"
|
|
27
29
|
NS_REL = "{http://schemas.openxmlformats.org/officeDocument/2006/relationships}"
|
|
28
30
|
|
|
@@ -194,7 +196,7 @@ def process_message(msg: Dict[str, Any]) -> Dict[str, Any]:
|
|
|
194
196
|
"result": {
|
|
195
197
|
"protocolVersion": "2024-11-05",
|
|
196
198
|
"capabilities": {"tools": {}},
|
|
197
|
-
"serverInfo": {"name": "synapse-mcp-excel", "version":
|
|
199
|
+
"serverInfo": {"name": "synapse-mcp-excel", "version": SERVER_VERSION}
|
|
198
200
|
}
|
|
199
201
|
}
|
|
200
202
|
elif method == "tools/list":
|
|
File without changes
|
package/tools/mcp_server.py
CHANGED
|
@@ -1,88 +1,88 @@
|
|
|
1
|
-
#!/usr/bin/env python3
|
|
2
|
-
"""
|
|
3
|
-
Fast Zero-Token Failure Pinpointer for Katalon & Jenkins Reports.
|
|
4
|
-
Scans the latest Reports/**/JUnit_Report.xml, extracts failed Test Cases,
|
|
5
|
-
exact failed lines in Groovy/Script, and the root-cause exception.
|
|
6
|
-
Outputs < 100 tokens.
|
|
7
|
-
|
|
8
|
-
Usage:
|
|
9
|
-
python3 tools/pinpoint_failure.py
|
|
10
|
-
python3 tools/pinpoint_failure.py path/to/JUnit_Report.xml
|
|
11
|
-
"""
|
|
12
|
-
import os
|
|
13
|
-
import sys
|
|
14
|
-
import glob
|
|
15
|
-
import xml.etree.ElementTree as ET
|
|
16
|
-
import re
|
|
17
|
-
|
|
18
|
-
def find_latest_report():
|
|
19
|
-
reports = glob.glob("Reports/**/JUnit_Report.xml", recursive=True)
|
|
20
|
-
if not reports:
|
|
21
|
-
return None
|
|
22
|
-
reports.sort(key=lambda x: os.path.getmtime(x), reverse=True)
|
|
23
|
-
return reports[0]
|
|
24
|
-
|
|
25
|
-
def parse_failures(xml_path):
|
|
26
|
-
if not os.path.exists(xml_path):
|
|
27
|
-
print(f"Error: {xml_path} does not exist.")
|
|
28
|
-
return []
|
|
29
|
-
|
|
30
|
-
tree = ET.parse(xml_path)
|
|
31
|
-
root = tree.getroot()
|
|
32
|
-
failures = []
|
|
33
|
-
|
|
34
|
-
suites = [root] if root.tag == "testsuite" else root.findall("testsuite")
|
|
35
|
-
|
|
36
|
-
for suite in suites:
|
|
37
|
-
for tc in suite.findall("testcase"):
|
|
38
|
-
tc_name = tc.get("name", "Unknown")
|
|
39
|
-
tc_time = tc.get("time", "0")
|
|
40
|
-
fail = tc.find("failure")
|
|
41
|
-
err = tc.find("error")
|
|
42
|
-
target = fail if fail is not None else err
|
|
43
|
-
|
|
44
|
-
if target is not None:
|
|
45
|
-
msg = target.get("message", "")
|
|
46
|
-
text = target.text or ""
|
|
47
|
-
combined = (msg + "\n" + text).replace("
", "\n")
|
|
48
|
-
|
|
49
|
-
# Extract Root Cause Reason
|
|
50
|
-
reason_match = re.search(r"Reason:\s*(.+?)(?=\n\s*at|\Z)", combined, re.DOTALL)
|
|
51
|
-
if reason_match:
|
|
52
|
-
reason = reason_match.group(1).strip()
|
|
53
|
-
else:
|
|
54
|
-
lines = [l.strip() for l in combined.split("\n") if l.strip()]
|
|
55
|
-
reason = lines[0] if lines else "Unknown Error"
|
|
56
|
-
|
|
57
|
-
# Extract Failed Line in Groovy Page / Script
|
|
58
|
-
Enterprise_lines = re.findall(r"(?:Enterprise\.pages\.[\w]+|TC-[\w\s\u00C0-\u1EF9]+|Script\d+)\.groovy:\d+|(?:TC-[\w\s\u00C0-\u1EF9]+:\d+)", combined)
|
|
59
|
-
|
|
60
|
-
failures.append({
|
|
61
|
-
"tc": tc_name.split("/")[-1],
|
|
62
|
-
"time": f"{float(tc_time):.1f}s",
|
|
63
|
-
"reason": reason.split("\n")[0][:180],
|
|
64
|
-
"failed_at": Enterprise_lines[:2] if Enterprise_lines else ["See JUnit report"]
|
|
65
|
-
})
|
|
66
|
-
|
|
67
|
-
return failures
|
|
68
|
-
|
|
69
|
-
def main():
|
|
70
|
-
xml_path = sys.argv[1] if len(sys.argv) > 1 else find_latest_report()
|
|
71
|
-
if not xml_path:
|
|
72
|
-
print("No JUnit_Report.xml found in Reports/.")
|
|
73
|
-
sys.exit(0)
|
|
74
|
-
|
|
75
|
-
print(f"=== Report: {xml_path} ===")
|
|
76
|
-
failures = parse_failures(xml_path)
|
|
77
|
-
if not failures:
|
|
78
|
-
print("✅ 100% PASSED: Zero failures found in report.")
|
|
79
|
-
return
|
|
80
|
-
|
|
81
|
-
print(f"❌ Found {len(failures)} Failed Test Case(s):")
|
|
82
|
-
for i, f in enumerate(failures, 1):
|
|
83
|
-
print(f"[{i}] {f['tc']} ({f['time']})")
|
|
84
|
-
print(f" Cause: {f['reason']}")
|
|
85
|
-
print(f" Line: {', '.join(f['failed_at'])}")
|
|
86
|
-
|
|
87
|
-
if __name__ == "__main__":
|
|
88
|
-
main()
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
"""
|
|
3
|
+
Fast Zero-Token Failure Pinpointer for Katalon & Jenkins Reports.
|
|
4
|
+
Scans the latest Reports/**/JUnit_Report.xml, extracts failed Test Cases,
|
|
5
|
+
exact failed lines in Groovy/Script, and the root-cause exception.
|
|
6
|
+
Outputs < 100 tokens.
|
|
7
|
+
|
|
8
|
+
Usage:
|
|
9
|
+
python3 tools/pinpoint_failure.py
|
|
10
|
+
python3 tools/pinpoint_failure.py path/to/JUnit_Report.xml
|
|
11
|
+
"""
|
|
12
|
+
import os
|
|
13
|
+
import sys
|
|
14
|
+
import glob
|
|
15
|
+
import xml.etree.ElementTree as ET
|
|
16
|
+
import re
|
|
17
|
+
|
|
18
|
+
def find_latest_report():
|
|
19
|
+
reports = glob.glob("Reports/**/JUnit_Report.xml", recursive=True)
|
|
20
|
+
if not reports:
|
|
21
|
+
return None
|
|
22
|
+
reports.sort(key=lambda x: os.path.getmtime(x), reverse=True)
|
|
23
|
+
return reports[0]
|
|
24
|
+
|
|
25
|
+
def parse_failures(xml_path):
|
|
26
|
+
if not os.path.exists(xml_path):
|
|
27
|
+
print(f"Error: {xml_path} does not exist.")
|
|
28
|
+
return []
|
|
29
|
+
|
|
30
|
+
tree = ET.parse(xml_path)
|
|
31
|
+
root = tree.getroot()
|
|
32
|
+
failures = []
|
|
33
|
+
|
|
34
|
+
suites = [root] if root.tag == "testsuite" else root.findall("testsuite")
|
|
35
|
+
|
|
36
|
+
for suite in suites:
|
|
37
|
+
for tc in suite.findall("testcase"):
|
|
38
|
+
tc_name = tc.get("name", "Unknown")
|
|
39
|
+
tc_time = tc.get("time", "0")
|
|
40
|
+
fail = tc.find("failure")
|
|
41
|
+
err = tc.find("error")
|
|
42
|
+
target = fail if fail is not None else err
|
|
43
|
+
|
|
44
|
+
if target is not None:
|
|
45
|
+
msg = target.get("message", "")
|
|
46
|
+
text = target.text or ""
|
|
47
|
+
combined = (msg + "\n" + text).replace("
", "\n")
|
|
48
|
+
|
|
49
|
+
# Extract Root Cause Reason
|
|
50
|
+
reason_match = re.search(r"Reason:\s*(.+?)(?=\n\s*at|\Z)", combined, re.DOTALL)
|
|
51
|
+
if reason_match:
|
|
52
|
+
reason = reason_match.group(1).strip()
|
|
53
|
+
else:
|
|
54
|
+
lines = [l.strip() for l in combined.split("\n") if l.strip()]
|
|
55
|
+
reason = lines[0] if lines else "Unknown Error"
|
|
56
|
+
|
|
57
|
+
# Extract Failed Line in Groovy Page / Script
|
|
58
|
+
Enterprise_lines = re.findall(r"(?:Enterprise\.pages\.[\w]+|TC-[\w\s\u00C0-\u1EF9]+|Script\d+)\.groovy:\d+|(?:TC-[\w\s\u00C0-\u1EF9]+:\d+)", combined)
|
|
59
|
+
|
|
60
|
+
failures.append({
|
|
61
|
+
"tc": tc_name.split("/")[-1],
|
|
62
|
+
"time": f"{float(tc_time):.1f}s",
|
|
63
|
+
"reason": reason.split("\n")[0][:180],
|
|
64
|
+
"failed_at": Enterprise_lines[:2] if Enterprise_lines else ["See JUnit report"]
|
|
65
|
+
})
|
|
66
|
+
|
|
67
|
+
return failures
|
|
68
|
+
|
|
69
|
+
def main():
|
|
70
|
+
xml_path = sys.argv[1] if len(sys.argv) > 1 else find_latest_report()
|
|
71
|
+
if not xml_path:
|
|
72
|
+
print("No JUnit_Report.xml found in Reports/.")
|
|
73
|
+
sys.exit(0)
|
|
74
|
+
|
|
75
|
+
print(f"=== Report: {xml_path} ===")
|
|
76
|
+
failures = parse_failures(xml_path)
|
|
77
|
+
if not failures:
|
|
78
|
+
print("✅ 100% PASSED: Zero failures found in report.")
|
|
79
|
+
return
|
|
80
|
+
|
|
81
|
+
print(f"❌ Found {len(failures)} Failed Test Case(s):")
|
|
82
|
+
for i, f in enumerate(failures, 1):
|
|
83
|
+
print(f"[{i}] {f['tc']} ({f['time']})")
|
|
84
|
+
print(f" Cause: {f['reason']}")
|
|
85
|
+
print(f" Line: {', '.join(f['failed_at'])}")
|
|
86
|
+
|
|
87
|
+
if __name__ == "__main__":
|
|
88
|
+
main()
|
package/tools/setup_project.py
CHANGED
|
@@ -394,6 +394,14 @@ def inject_antigravity_global_plugin():
|
|
|
394
394
|
if not os.path.exists(dst_readme) or not os.path.samefile(src_readme, dst_readme):
|
|
395
395
|
shutil.copy2(src_readme, dst_readme)
|
|
396
396
|
|
|
397
|
+
# 3. LICENSE & NOTICE (MIT License Provenance & Author Attribution)
|
|
398
|
+
for meta_file in ("LICENSE", "NOTICE"):
|
|
399
|
+
src_meta = os.path.join(HUB_ROOT, meta_file)
|
|
400
|
+
dst_meta = os.path.join(target_plugin_dir, meta_file)
|
|
401
|
+
if os.path.exists(src_meta):
|
|
402
|
+
if not os.path.exists(dst_meta) or not os.path.samefile(src_meta, dst_meta):
|
|
403
|
+
shutil.copy2(src_meta, dst_meta)
|
|
404
|
+
|
|
397
405
|
# Helper to sync directory (junction or copy)
|
|
398
406
|
def sync_dir(src, dst):
|
|
399
407
|
if not os.path.exists(src):
|
package/tools/update_sentinel.py
CHANGED
|
@@ -65,7 +65,7 @@ def get_local_version():
|
|
|
65
65
|
try:
|
|
66
66
|
with open(pkg_file, "r", encoding="utf-8") as f:
|
|
67
67
|
data = json.load(f)
|
|
68
|
-
return data.get("version", "2.4.
|
|
68
|
+
return data.get("version", "2.4.3")
|
|
69
69
|
except (OSError, json.JSONDecodeError):
|
|
70
70
|
pass
|
|
71
71
|
|
|
@@ -74,11 +74,11 @@ def get_local_version():
|
|
|
74
74
|
try:
|
|
75
75
|
with open(manifest_file, "r", encoding="utf-8") as f:
|
|
76
76
|
data = json.load(f)
|
|
77
|
-
return data.get("version", "2.4.
|
|
77
|
+
return data.get("version", "2.4.3")
|
|
78
78
|
except (OSError, json.JSONDecodeError):
|
|
79
79
|
pass
|
|
80
80
|
|
|
81
|
-
return "2.4.
|
|
81
|
+
return "2.4.3"
|
|
82
82
|
|
|
83
83
|
|
|
84
84
|
def fetch_remote_version():
|
|
@@ -321,14 +321,15 @@ def perform_update():
|
|
|
321
321
|
except (subprocess.SubprocessError, OSError):
|
|
322
322
|
pass
|
|
323
323
|
else:
|
|
324
|
-
# Refresh
|
|
324
|
+
# Refresh metadata files in global plugin
|
|
325
325
|
try:
|
|
326
326
|
import shutil
|
|
327
|
-
|
|
328
|
-
|
|
329
|
-
|
|
330
|
-
|
|
331
|
-
|
|
327
|
+
for meta_f in ("plugin.json", "LICENSE", "NOTICE", "README.md"):
|
|
328
|
+
src_f = os.path.join(HUB_ROOT, meta_f)
|
|
329
|
+
dst_f = os.path.join(GLOBAL_PLUGIN_DIR, meta_f)
|
|
330
|
+
if os.path.exists(src_f):
|
|
331
|
+
shutil.copy2(src_f, dst_f)
|
|
332
|
+
print(f" {GREEN}[OK] Refreshed metadata & LICENSE in: {GLOBAL_PLUGIN_DIR}{RESET}")
|
|
332
333
|
except OSError:
|
|
333
334
|
pass
|
|
334
335
|
|
package/tools/verify_all.sh
CHANGED
|
File without changes
|
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
#!/usr/bin/env python3
|
|
2
|
+
from __future__ import annotations
|
|
2
3
|
"""
|
|
3
4
|
Enterprise Vision-Grounded Fallback & Set-of-Marks (SoM) Engine
|
|
4
5
|
--------------------------------------------------------------
|
|
@@ -36,6 +37,11 @@ except ImportError:
|
|
|
36
37
|
ImageFilter = None
|
|
37
38
|
HAS_PIL = False
|
|
38
39
|
|
|
40
|
+
if HAS_PIL and Image is not None:
|
|
41
|
+
ImageSourceType = Union[Image.Image, str]
|
|
42
|
+
else:
|
|
43
|
+
ImageSourceType = Any
|
|
44
|
+
|
|
39
45
|
# Enforce UTF-8 stdout encoding for Windows console compatibility
|
|
40
46
|
if hasattr(sys.stdout, 'reconfigure'):
|
|
41
47
|
try:
|
|
@@ -58,11 +64,13 @@ MARK_COLORS = [
|
|
|
58
64
|
]
|
|
59
65
|
|
|
60
66
|
|
|
61
|
-
def compute_visual_hash(image_source:
|
|
67
|
+
def compute_visual_hash(image_source: ImageSourceType, bbox: Optional[List[int]] = None, hash_size: int = 8) -> str:
|
|
62
68
|
"""
|
|
63
69
|
Computes a 64-bit difference hash (dHash) for an image or cropped component.
|
|
64
70
|
Fast, deterministic, and 100% offline.
|
|
65
71
|
"""
|
|
72
|
+
if not HAS_PIL or Image is None:
|
|
73
|
+
raise ImportError("Pillow is required for visual perceptual operations (pip install Pillow).")
|
|
66
74
|
if isinstance(image_source, str):
|
|
67
75
|
if not os.path.exists(image_source):
|
|
68
76
|
raise FileNotFoundError(f"Image not found: {image_source}")
|
|
@@ -135,7 +143,7 @@ class VisualSoMHelper:
|
|
|
135
143
|
def lookup_cached_coordinates(
|
|
136
144
|
self,
|
|
137
145
|
component_name: str,
|
|
138
|
-
image_source:
|
|
146
|
+
image_source: ImageSourceType,
|
|
139
147
|
bbox: Optional[List[int]] = None,
|
|
140
148
|
threshold: float = 0.98,
|
|
141
149
|
db_path: Optional[str] = None
|
|
@@ -198,7 +206,7 @@ class VisualSoMHelper:
|
|
|
198
206
|
def save_cached_coordinates(
|
|
199
207
|
self,
|
|
200
208
|
component_name: str,
|
|
201
|
-
image_source:
|
|
209
|
+
image_source: ImageSourceType,
|
|
202
210
|
center_x: int,
|
|
203
211
|
center_y: int,
|
|
204
212
|
bbox: Optional[List[int]] = None,
|
|
@@ -324,6 +332,9 @@ class VisualSoMHelper:
|
|
|
324
332
|
If elements is omitted, autonomously runs detect_canvas_interactive_regions.
|
|
325
333
|
Returns (marked_image_path, marks_map).
|
|
326
334
|
"""
|
|
335
|
+
if not HAS_PIL or Image is None:
|
|
336
|
+
raise ImportError("Pillow is required for Set-of-Marks overlay generation (pip install Pillow).")
|
|
337
|
+
|
|
327
338
|
if not os.path.exists(image_path):
|
|
328
339
|
img = Image.new("RGB", (self.viewport_width, self.viewport_height), color="#1E1E2E")
|
|
329
340
|
draw = ImageDraw.Draw(img)
|
package/tools/welcome_page.py
CHANGED
|
@@ -48,7 +48,7 @@ def get_hub_metadata() -> Dict[str, Any]:
|
|
|
48
48
|
"""Retrieves version and catalog count from skills_manifest.json."""
|
|
49
49
|
manifest_path = os.path.join(HUB_ROOT, "skills_manifest.json")
|
|
50
50
|
meta = {
|
|
51
|
-
"version": "2.4.
|
|
51
|
+
"version": "2.4.3",
|
|
52
52
|
"totalSkills": 38,
|
|
53
53
|
"author": "Thế Vị (@TeeVee998)",
|
|
54
54
|
"repo": "https://github.com/TeeVee998/Synapse-Hub"
|
|
@@ -1,83 +0,0 @@
|
|
|
1
|
-
# A/B Empirical Evaluation Methodology & Replication Report
|
|
2
|
-
|
|
3
|
-
> **Dataset Scope:** 25 Production Challenges across 5 Dynamic QA Failure Vectors
|
|
4
|
-
> **Replication Seed:** `42` | **Temperature:** `0.0` | **Deterministic Execution Mode**
|
|
5
|
-
> **Evaluation Harness:** `benchmarks/qa_eval/eval_runner.py`
|
|
6
|
-
> **Author & Provenance:** Thế Vị (@TeeVee998) — [Synapse Hub](https://github.com/TeeVee998/Synapse-Hub)
|
|
7
|
-
|
|
8
|
-
---
|
|
9
|
-
|
|
10
|
-
## 1. Executive Summary
|
|
11
|
-
|
|
12
|
-
This report establishes the empirical A/B testing methodology comparing **Naive LLM Prompting (Control Group)** against **Synapse Hub Multi-Agent Swarm (Treatment Group)** across 25 real-world test automation failure scenarios.
|
|
13
|
-
|
|
14
|
-
```mermaid
|
|
15
|
-
flowchart LR
|
|
16
|
-
Dataset["25 Production Challenges<br/>(Seed=42, Temp=0.0)"] --> Control["<b>Control Group (Naive LLM)</b><br/>• Monolithic context read<br/>• Raw DOM dump (>15k tokens)<br/>• Unconstrained prompt repair"]
|
|
17
|
-
Dataset --> Treatment["<b>Treatment Group (Synapse Hub)</b><br/>• Deterministic cascade tools<br/>• Compact AOM & repo_map<br/>• 8 Quality Gates DoD"]
|
|
18
|
-
|
|
19
|
-
Control --> ControlResults["• Mean Tokens: ~18,400<br/>• MTTR: 8-15 min<br/>• Pass Rate: 56.0%<br/>• Fake-Pass Trap: 44.0%"]
|
|
20
|
-
Treatment --> TreatmentResults["• Mean Tokens: 76.2<br/>• MTTR: < 0.1s<br/>• Pass Rate: 100.0%<br/>• Fake-Pass Trap: 0.0%"]
|
|
21
|
-
```
|
|
22
|
-
|
|
23
|
-
---
|
|
24
|
-
|
|
25
|
-
## 2. Experimental Setup & Protocol
|
|
26
|
-
|
|
27
|
-
### 2.1 Sample Distribution ($N = 25$)
|
|
28
|
-
Challenges are partitioned evenly across 5 critical enterprise QA failure vectors:
|
|
29
|
-
1. **`locator_drift` (5 challenges):** Dynamic CSS hashes, nested shadow DOM piercing, semantic label shifts, legacy ID removals, HTML tag evolution.
|
|
30
|
-
2. **`api_contract_drift` (5 challenges):** Type mutations, field renames, object flattening, creation status code shifts (200 $\to$ 201), auth prefix upgrades.
|
|
31
|
-
3. **`timing_and_race` (5 challenges):** Variable latency spikes, PrimeNG backdrop fade intercepts, debounced autocomplete races, SVG animations, infinite scroll chunks.
|
|
32
|
-
4. **`stale_and_mutation` (5 challenges):** WebSocket auto-refresh stale elements, tab component unmounts, tbody pagination detachments, re-rendered modal nodes, locale swaps.
|
|
33
|
-
5. **`fake_pass_tautology` (5 challenges):** Swallowed 500 errors in empty catch blocks, zero-assertion smoke tests, `assert true == true`, unawaited async expectations, unbounded relational assertions (`status >= 200`).
|
|
34
|
-
|
|
35
|
-
### 2.2 Hyperparameters & Control Variables
|
|
36
|
-
- **Random Seed:** `42` (pinned across sampling and mutation generators).
|
|
37
|
-
- **Sampling Temperature:** `0.0` (greedy decoding for deterministic reproducibility).
|
|
38
|
-
- **Environment:** Headless Chromium, Viewport `1920x1080`, pure CPU/Memory baseline.
|
|
39
|
-
|
|
40
|
-
---
|
|
41
|
-
|
|
42
|
-
## 3. Empirical Results Comparison
|
|
43
|
-
|
|
44
|
-
| Evaluation Metric | Control Group (Naive LLM) | Treatment Group (Synapse Hub) | Relative Gain / Reduction |
|
|
45
|
-
| :--- | :---: | :---: | :---: |
|
|
46
|
-
| **Autonomous Resolution Rate** | 56.0% (14 / 25) | **100.0% (25 / 25)** | **+44.0% absolute gain** |
|
|
47
|
-
| **Mean Token Cost / Bug Fix** | 18,450 tokens | **76.2 tokens** | **99.59% token reduction** |
|
|
48
|
-
| **Mean Time-to-Heal (MTTR)** | ~8.4 minutes | **< 0.001s (Deterministic)** | **Instantaneous 0-token repair** |
|
|
49
|
-
| **AST Mutation Score** | 32.4% (Mutants survived) | **100.0% (All mutants killed)** | **Zero tautological test escapes** |
|
|
50
|
-
| **Fake-Pass Vulnerability Rate** | 44.0% accepted fake pass | **0.0% (Strictly blocked)** | **100% Truth Anchor enforcement** |
|
|
51
|
-
| **Zero Side-Effect Compliance** | 68.0% (Unrelated edits) | **100.0% (Atomic patch)** | **Zero regression risk** |
|
|
52
|
-
|
|
53
|
-
---
|
|
54
|
-
|
|
55
|
-
## 4. Key Mechanisms Driving the Differential
|
|
56
|
-
|
|
57
|
-
1. **Deterministic 3-Layer Fallback (`HeuristicLocatorMatcher`):**
|
|
58
|
-
- Naive LLMs consume 10k–40k tokens reading raw HTML to guess new selectors.
|
|
59
|
-
- Synapse Hub parses compact DOM elements locally and resolves stable `data-testid`, ARIA labels, or Levenshtein text anchors in under 15ms with **0 LLM tokens**.
|
|
60
|
-
2. **Cognitive Forcing & Micro-Mutations (`micro_mutator.py`):**
|
|
61
|
-
- Naive LLMs frequently emit tautological passes (`expect(true).toBe(true)` or empty catches) when under token pressure.
|
|
62
|
-
- Synapse Hub injects AST mutations and requires $\ge 80\%$ mutation score before handoff, catching silent passes at Gate 7 and Gate 8.
|
|
63
|
-
3. **Log & Context Distillation (`distill_log.py` & `repo_map.py`):**
|
|
64
|
-
- Filters 95% of execution noise, isolating exact failing stacktrace pointers.
|
|
65
|
-
|
|
66
|
-
---
|
|
67
|
-
|
|
68
|
-
## 5. Replication Guide
|
|
69
|
-
|
|
70
|
-
To independently reproduce this benchmark evaluation:
|
|
71
|
-
|
|
72
|
-
```bash
|
|
73
|
-
# 1. Clone repository
|
|
74
|
-
git clone https://github.com/TeeVee998/Synapse-Hub.git
|
|
75
|
-
cd Synapse-Hub
|
|
76
|
-
|
|
77
|
-
# 2. Run full deterministic benchmark harness
|
|
78
|
-
python benchmarks/qa_eval/eval_runner.py
|
|
79
|
-
|
|
80
|
-
# 3. Verify anti-fake-pass and robustness test
|
|
81
|
-
python -m unittest tests/test_benchmark_robustness.py -v
|
|
82
|
-
```
|
|
83
|
-
All outputs are serialized to `benchmarks/qa_eval/reports/eval_results.json`.
|