@teevee98/synapse-hub 4.3.0 → 4.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (60) hide show
  1. package/.agents/agents/api-engineer.md +8 -0
  2. package/.agents/agents/perf-engineer.md +10 -2
  3. package/.agents/hooks/session_start.py +52 -0
  4. package/.agents/hooks.json +12 -0
  5. package/.agents/rules/performance_testing_rules.md +4 -3
  6. package/.agents/skills/01-web-automation/clean-pom-crafter/SKILL.md +6 -3
  7. package/.agents/skills/01-web-automation/e2e-testing-patterns/SKILL.md +3 -1
  8. package/.agents/skills/02-test-design/rbt-manual-testing/SKILL.md +11 -5
  9. package/.agents/skills/04-performance-and-load/backend-load-and-stress-patterns/SKILL.md +4 -1
  10. package/.agents/skills/04-performance-and-load/backend-load-and-stress-patterns/references/performance_test_plan_29119.md +106 -0
  11. package/.agents/skills/04-performance-and-load/web-vitals-and-performance-profiler/SKILL.md +75 -14
  12. package/.agents/skills/05-security-and-compliance/security-dast-sentinel/SKILL.md +13 -2
  13. package/.agents/skills/05-security-and-compliance/wcag-audit-patterns/SKILL.md +4 -1
  14. package/.agents/skills/07-data-and-state/file-and-report-validator/SKILL.md +8 -1
  15. package/.agents/skills/07-data-and-state/synthetic-test-data-generator/SKILL.md +4 -1
  16. package/.agents/skills/11-ai-and-prompting/mipro-prompt-compiler/SKILL.md +8 -0
  17. package/.agents/templates/performance-test-plan-template.md +106 -0
  18. package/README.en.md +2 -2
  19. package/README.md +2 -2
  20. package/agentskills.json +1 -1
  21. package/docs/ARCHITECTURE.en.md +95 -2
  22. package/docs/ARCHITECTURE.md +86 -2
  23. package/docs/TOOL_CATALOG.md +11 -2
  24. package/package.json +1 -1
  25. package/plugin.json +1 -1
  26. package/skills_manifest.json +1 -1
  27. package/synapse.config.json +1 -1
  28. package/tools/.infer_baseline.json +44 -3
  29. package/tools/TOOLS_RUNBOOK.md +237 -0
  30. package/tools/TOOL_CATALOG.json +138 -3
  31. package/tools/__init__.py +1 -1
  32. package/tools/ai_eval_sentinel.py +403 -0
  33. package/tools/asvs_auditor.py +410 -0
  34. package/tools/browser_helpers.js +2 -2
  35. package/tools/chaos_resilience_probe.py +362 -0
  36. package/tools/cldr_l10n_verifier.py +309 -0
  37. package/tools/cli.py +191 -3
  38. package/tools/cvss_calculator.py +412 -0
  39. package/tools/dashboard_generator.py +8 -8
  40. package/tools/dspy_qa_signatures.py +105 -1
  41. package/tools/etl_data_reconciler.py +460 -0
  42. package/tools/generate_jmx_plan.py +12 -4
  43. package/tools/jtl_analyzer.py +23 -1
  44. package/tools/mcp_docx_server.py +1 -1
  45. package/tools/mcp_excel_server.py +1 -1
  46. package/tools/mcp_server.py +1 -1
  47. package/tools/mobile_resilience_sentinel.py +273 -0
  48. package/tools/observability_tester.py +235 -0
  49. package/tools/payment_interop_simulator.py +337 -0
  50. package/tools/pre_push_guard.py +306 -0
  51. package/tools/regulatory_compliance.py +336 -0
  52. package/tools/rules_router.py +1 -1
  53. package/tools/task_memory_manager.py +12 -1
  54. package/tools/test_memory_os.py +48 -0
  55. package/tools/tmmi_auditor.py +172 -0
  56. package/tools/tot_exploratory_tester.py +17 -0
  57. package/tools/ui_server.py +1 -1
  58. package/tools/update_sentinel.py +109 -43
  59. package/tools/vpat_exporter.py +428 -0
  60. package/tools/welcome_page.py +1 -1
@@ -21,6 +21,14 @@ tools:
21
21
 
22
22
  # Enterprise API & Contract Testing Specialist (api-engineer)
23
23
 
24
+ > **International Standards Alignment:**
25
+ > - **Google SRE Four Golden Signals:** Latency (Success vs Error separation), Traffic (RPS/TPS), Errors (Explicit 4xx/5xx & Implicit false 200s), Saturation (CPU, DB Pool, Memory).
26
+ > - **Percentile Tail Latency:** Mandatory P50, P95, P99, P99.9 (strict ban on Average).
27
+ > - **Industry Tiered SLAs:** 4-tier budgets (Simple/Cache, Standard CRUD, Complex/Payment, Async 202).
28
+ > - **Technical Standards:** IETF RFC 9110/9112/9113, OpenAPI OAS 3.0/3.1, CNCF gRPC/Protobuf.
29
+ > - **Architectural Resilience:** RFC 6585 Rate Limiting (429 + Retry-After), Timeout/Circuit Breaker, HikariCP Connection Pool Saturation.
30
+ > - **Mandatory Rule:** Adheres strictly to [api_testing_rules.md](../rules/api_testing_rules.md).
31
+
24
32
  ## Core Responsibilities:
25
33
  - **OpenAPI & Swagger Test Matrix Synthesis:**
26
34
  - Ingest OpenAPI 3.x / Swagger specifications, Postman collections, and API contracts.
@@ -21,10 +21,18 @@ tools:
21
21
 
22
22
  # Enterprise Performance & Load Engineering Agent (perf-engineer)
23
23
 
24
+ > **International Standards Alignment:**
25
+ > - **ISO/IEC 25010:2023 (SQuaRE):** Performance Efficiency (Time Behaviour, Resource Utilization, Capacity).
26
+ > - **ISO/IEC/IEEE 29119-3 & 29119-4:** Non-Functional Test Plan (PTP), Test Design Specs, and Risk Reporting.
27
+ > - **ISTQB CT-PT:** 6 Test Archetypes (Smoke, Load, Stress, Spike, Soak, Scalability), Pareto 70/20/10 Operational Profile.
28
+ > - **TPC / SPEC:** Strict steady-state measurement interval, warm-up/cool-down exclusion.
29
+ > - **W3C Web Performance & Google SRE:** Navigation Timing L2, Core Web Vitals (LCP/CLS/INP/TTFB), Gate 10 SLA Sentinel, and OpenTelemetry Distributed Tracing (`traceparent`).
30
+
24
31
  ## Core Responsibilities:
25
- - **Workload Modeling & Archetype Planning:**
32
+ - **Workload Modeling & Archetype Planning (ISTQB CT-PT & ISO 25010):**
26
33
  - Ingest Non-Functional Requirements (NFRs), target SLAs (P90, P95, P99 latencies, max error rate < 1.0%), and peak concurrency goals.
27
- - Plan progressive test execution sequences: Smoke (1–5 VUs) -> Baseline Average Load -> Stress -> Spike -> Soak test.
34
+ - Plan progressive test execution sequences: Smoke (1–5 VUs) -> Baseline Average Load -> Stress -> Spike -> Soak -> Scalability test.
35
+ - Formulate ISO 29119-3 compliant Performance Test Plans (`.agents/templates/performance-test-plan-template.md`).
28
36
  - Model realistic user think time (1s–5s) and open arrival rate profiles for public traffic (`tools/generate_jmx_plan.py --workload`).
29
37
  - **Autonomous Script Synthesis (JMeter / k6 / Locust):**
30
38
  - Generate production-grade JMeter `.jmx` test plans and Locust `locustfile.py` directly from OpenAPI/Swagger specs, Postman collections, or HAR captures via `tools/generate_jmx_plan.py`.
@@ -0,0 +1,52 @@
1
+ #!/usr/bin/env python3
2
+ # SPDX-FileCopyrightText: Copyright (c) 2026 Thế Vị (@TeeVee98). All rights reserved.
3
+ # SPDX-License-Identifier: MIT
4
+ # Project: Synapse-Hub (https://github.com/TeeVee98/Synapse-Hub)
5
+
6
+ """
7
+ Antigravity SessionStart Hook — Automated Upstream Drift & Update Sentinel
8
+ -------------------------------------------------------------------------
9
+ Invoked automatically by Antigravity on session initialization.
10
+ Checks upstream NPM registry / GitHub for newer Synapse Hub releases
11
+ and non-destructively synchronizes the agent environment.
12
+ Outputs strict JSON matching the Antigravity SessionStart hook contract.
13
+ """
14
+
15
+ import os
16
+ import sys
17
+ import json
18
+
19
+ # Enforce UTF-8 stdout encoding for Windows console compatibility
20
+ if hasattr(sys.stdout, 'reconfigure'):
21
+ try:
22
+ sys.stdout.reconfigure(encoding='utf-8')
23
+ sys.stderr.reconfigure(encoding='utf-8')
24
+ except Exception:
25
+ pass
26
+
27
+ CUR_DIR = os.path.dirname(os.path.abspath(__file__))
28
+ # Check both repo layout (.agents/hooks/..) and plugin layout
29
+ CANDIDATE_ROOTS = [
30
+ os.path.abspath(os.path.join(CUR_DIR, "..", "..")), # repo: root/tools/
31
+ os.path.abspath(os.path.join(CUR_DIR, "..")), # root/.agents/hooks
32
+ os.path.abspath(CUR_DIR)
33
+ ]
34
+
35
+ sentinel_found = False
36
+ for root in CANDIDATE_ROOTS:
37
+ tools_dir = os.path.join(root, "tools")
38
+ if os.path.isdir(tools_dir) and os.path.exists(os.path.join(tools_dir, "update_sentinel.py")):
39
+ if tools_dir not in sys.path:
40
+ sys.path.insert(0, tools_dir)
41
+ try:
42
+ import update_sentinel
43
+ update_sentinel.run_auto_update_hook()
44
+ sentinel_found = True
45
+ break
46
+ except Exception as err:
47
+ sys.stderr.write(f"[session_start_hook] Error running update_sentinel: {err}\n")
48
+ break
49
+
50
+ if not sentinel_found:
51
+ # Failsafe fallback: output clean empty payload to never break Antigravity session
52
+ print(json.dumps({"injectSteps": []}))
@@ -0,0 +1,12 @@
1
+ {
2
+ "synapse-auto-updater": {
3
+ "enabled": true,
4
+ "SessionStart": [
5
+ {
6
+ "type": "command",
7
+ "command": "python -c \"import os, subprocess, sys; s = 'hooks/session_start.py' if os.path.exists('hooks/session_start.py') else '.agents/hooks/session_start.py'; subprocess.run([sys.executable, s])\"",
8
+ "timeout": 15
9
+ }
10
+ ]
11
+ }
12
+ }
@@ -6,15 +6,16 @@ profiles: [perf, full]
6
6
 
7
7
  # Performance & Load Testing Invariants (k6 · JMeter · Locust)
8
8
 
9
- > **Scope:** Applies whenever designing, executing, or analyzing performance, load, stress, spike, or soak test workloads.
9
+ > **Scope:** Applies whenever designing, executing, or analyzing performance, load, stress, spike, soak, or scalability workloads.
10
+ > **International Standards Alignment:** Codifies ISO/IEC 25010:2023 (Performance Efficiency), ISO/IEC/IEEE 29119-3/4, ISTQB CT-PT, TPC/SPEC Steady-State discipline, Google SRE SLA budgets, and W3C Trace Context.
10
11
  > Complement to `AGENTS.md` enforcing enterprise safety, workload realism, and zero-hallucination analysis.
11
12
 
12
13
  ---
13
14
 
14
15
  ## 1. Safety Gate & Pre-Flight Invariants
15
16
  1. **Dedicated Environments Only:** Never run load tests against production, shared staging, or third-party APIs without written approval.
16
- 2. **Pre-Run Declaration:** Declare target host, archetype (Smoke, Average Load, Stress, Spike, Soak), VU/arrival profile, duration, and kill-switch threshold (`abortOnFail`).
17
- 3. **Progressive Escalation:** Smoke (1–5 VUs) -> Baseline Average Load -> Stress / Spike / Soak. Never start high-VU runs autonomously.
17
+ 2. **Pre-Run Declaration:** Declare target host, archetype (Smoke, Average Load, Stress, Spike, Soak, Scalability), VU/arrival profile, duration, and kill-switch threshold (`abortOnFail`).
18
+ 3. **Progressive Escalation:** Smoke (1–5 VUs) -> Baseline Average Load -> Stress / Spike / Soak / Scalability. Never start high-VU runs autonomously.
18
19
  4. **Hard Caps & Kill-Switch:** Scripts must enforce finite durations, rate caps, and automated abort (`abortOnFail` when error rate > 10%). Unbounded loops are strictly forbidden.
19
20
  5. **Isolation & Cleanup:** Mock third-party providers (Payment, SMS, Captcha). Use synthetic test data (`PERF_AUTO_`); teardown after runs.
20
21
 
@@ -47,9 +47,12 @@ policies:
47
47
  - Naming convention: `TC-<Number>_<FeatureName>` (e.g. `TC-101_CreateUserAccount.groovy` or `tc-101-create-user.spec.ts`).
48
48
  2. **Unified Inheritance from `BasePage`:**
49
49
  - All Page Classes inherit from `BasePage`.
50
- 3. **Zero Raw Locators in Scripts:**
51
- - Scripts only orchestrate business workflows. All locators and UI interactions are encapsulated in Page Classes.
52
- 4. **Mandatory 100% UTF-8 No BOM:**
50
+ 3. **Zero Raw Locators in Scripts (ISO/IEC/IEEE 29119-5 Keyword-Driven Separation):**
51
+ - Scripts only orchestrate business workflows. All locators and UI interactions are encapsulated in Page Classes, strictly adhering to ISO/IEC/IEEE 29119-5 Keyword-Driven separation of business logic from technical implementation.
52
+ 4. **Clean POM Flat vs Screenplay Pattern Rationale:**
53
+ - Enforce Clean POM Flat Architecture (max 1-level shallow inheritance from `BasePage`).
54
+ - Screenplay Pattern (Actor/Task/Interaction/Ability micro-class fragmentation) is intentionally prohibited to prevent LLM Context Blowout and token waste.
55
+ 5. **Mandatory 100% UTF-8 No BOM:**
53
56
  - Zero BOM bytes (`0xEF, 0xBB, 0xBF`) across all test files.
54
57
 
55
58
  ### Phase 2: Master Locator Priority Map (6 Tiers)
@@ -38,7 +38,7 @@ Engineering standard for structuring, scaling, and maintaining large-scale End-t
38
38
 
39
39
  ---
40
40
 
41
- ### Test Pyramid & E2E Scope Boundary
41
+ ### Test Pyramid & E2E Scope Boundary (Mike Cohn Golden Ratio)
42
42
 
43
43
  ```mermaid
44
44
  flowchart TD
@@ -49,6 +49,8 @@ flowchart TD
49
49
  Unit --> Integ --> E2E
50
50
  ```
51
51
 
52
+ - **Mike Cohn's Test Pyramid Standard:** 60% Unit Tests — 30% Service/Integration/API Tests — 10% E2E UI Tests.
53
+ - **ISTQB CT-TAE Alignment:** Implements TAA (Test Automation Architecture) layered separation: Test Generation (`test-designer`), Test Definition (`pom-crafter`), Test Execution (`test-runner`), and Test Adaptation (Playwright CDP / W3C WebDriver BiDi).
52
54
  - **What belongs in E2E:** High-risk customer journeys (Sign-up, Onboarding, Checkout, File Export, Role transitions).
53
55
  - **What to avoid in E2E:** Exhaustive boundary value checks of simple form fields (delegate to Unit/API tests).
54
56
 
@@ -36,7 +36,7 @@ policies:
36
36
 
37
37
  ### Overview & Capabilities
38
38
 
39
- **RBT Manual Testing** is an enterprise-grade skill for designing manual test suites using **Risk-Based Testing (RBT)** and coverage matrix analysis. It provides two operational modes:
39
+ **RBT Manual Testing** is an enterprise-grade skill for designing manual test suites using **Risk-Based Testing (RBT)** and coverage matrix analysis (fully aligned with **ISO/IEC/IEEE 29119-2** test processes and **ISO/IEC/IEEE 29119-3** test documentation standards). It provides two operational modes:
40
40
 
41
41
  | Mode | When to Use | Scope & Characteristics | Execution Time |
42
42
  | :--- | :--- | :--- | :--- |
@@ -46,7 +46,7 @@ policies:
46
46
  ### Core Tenets:
47
47
  1. **Human Strategy:** Engineers define business boundaries, risk appetite, and resolve flagged ambiguities.
48
48
  2. **AI Execution:** Agent decomposes modules, builds coverage matrices, applies the 15-field validation checklist, and computes risk scores.
49
- 3. **Enterprise Deliverables:** Standardized 12-column Markdown tables with automation metadata, ready for export to Excel (`.xlsx`), Jira Xray, or TestRail.
49
+ 3. **Enterprise Deliverables (ISO/IEC/IEEE 29119-3 Aligned):** Standardized 12-column Markdown tables with automation metadata, ready for export to Excel (`.xlsx`), Jira Xray, or TestRail.
50
50
 
51
51
  ---
52
52
 
@@ -124,8 +124,8 @@ When `Concurrency Gate = ON`, generate targeted cases for applicable vectors:
124
124
  3. **Stale State / Optimistic Locking:** User A and User B open the same record; User A saves first, then User B saves outdated data -> verify system blocks silent overwrite (version/ETag check) or prompts merge/refresh.
125
125
  4. **Retry & Idempotency:** Simulate network timeout after server commits, then retry with the same `Idempotency-Key` -> verify server returns the original result without creating duplicate records.
126
126
 
127
- #### Conditional ISTQB Advanced Test Design Techniques
128
- In addition to Equivalence Partitioning (EP), Boundary Value Analysis (BVA), and Exploratory Testing, conditionally activate targeted formal techniques:
127
+ #### Conditional ISTQB & ISO/IEC/IEEE 29119 Advanced Test Design Techniques
128
+ In addition to Equivalence Partitioning (EP), Boundary Value Analysis (BVA), and Exploratory Testing, conditionally activate targeted formal techniques (CT-FL / CT-TAE / ISO 29119-4):
129
129
 
130
130
  | Technique | Auto-Activation Rule | Mandatory Generation Rule |
131
131
  | :--- | :--- | :--- |
@@ -198,11 +198,17 @@ Generated workbooks automatically include:
198
198
 
199
199
  ---
200
200
 
201
+ ### TMMi (Test Maturity Model integration) Maturity Assessment
202
+ - Assess repository and test engineering practices against TMMi Levels 1 through 5 (Level 2 Managed, Level 3 Defined, Level 4 Measured, Level 5 Optimization) via `tools/tmmi_auditor.py` (`agent-qa tmmi`).
203
+
204
+ ---
205
+
201
206
  ## 4. Verification Points & DoD
202
207
 
203
208
  1. Every test case carries explicit Risk Priority Number (RPN = Impact x Likelihood).
204
209
  2. Test steps include preconditions, concrete test data, and verifiable expected outcomes.
205
- 3. Exported test suite generates clean tabular data (Excel / CSV / Markdown).
210
+ 3. Test process conforms to ISO/IEC/IEEE 29119-2/3 and meets TMMi Level 3+ maturity requirements (`agent-qa tmmi --gate-level 3`).
211
+ 4. Exported test suite generates clean tabular data (Excel / CSV / Markdown).
206
212
 
207
213
  ---
208
214
 
@@ -39,7 +39,7 @@ policies:
39
39
 
40
40
  ---
41
41
 
42
- ### Test Archetypes & SLA Targets
42
+ ### Test Archetypes & SLA Targets (ISTQB CT-PT Standards)
43
43
 
44
44
  | Archetype | VUs / Rate | Duration | Objective | SLA Assertion |
45
45
  | :--- | :--- | :--- | :--- | :--- |
@@ -48,6 +48,9 @@ policies:
48
48
  | **Stress Test** | 200 -> 2,000 VUs | 20 - 45 min | Find degradation cliff | Graceful 429/503 (no JVM OOM) |
49
49
  | **Spike Test** | 10 -> 1,000 VUs | 5 - 10 min | Autoscaling & rate limiters | Gateway recovery < 30s |
50
50
  | **Soak Test** | 50 - 100 VUs | 2 - 12 hr | Memory leaks & connection leaks | Stable heap & steady latency |
51
+ | **Scalability Test** | 50 -> 100 -> 200 VUs | 30 - 60 min | Linear scale-up/out elasticity | Throughput scales $\ge 80\%$ linearly |
52
+
53
+ 👉 **Formal Test Plan Schema:** Refer to [**ISO/IEC/IEEE 29119-3 Performance Test Plan (`references/performance_test_plan_29119.md`)**](references/performance_test_plan_29119.md) for full environmental topology, NFR SLO targets, and risk declarations.
51
54
 
52
55
  ---
53
56
 
@@ -0,0 +1,106 @@
1
+ # Performance Test Plan (PTP) — ISO/IEC/IEEE 29119-3 Reference
2
+
3
+ > **Standard Compliance:** Conforms to ISO/IEC/IEEE 29119-3 (Non-Functional Test Plan), ISO/IEC 25010:2023 (Performance Efficiency), ISTQB CT-PT, and Google SRE Framework.
4
+ > **Target Execution Engines:** Apache JMeter 5.x, k6, Locust.
5
+
6
+ ---
7
+
8
+ ## 1. Document Overview & Context (ISO 29119-3 §5.2)
9
+
10
+ - **Test Plan ID:** `PTP-PERF-YYYYMMDD-[MODULE]`
11
+ - **Target System / AUT:** `[Application / Microservices under test]`
12
+ - **Revision History:** `v1.0 (Initial Draft)`
13
+ - **Author / Lead Performance QA:** `[AI Agent perf-engineer / Performance Architect]`
14
+ - **Scope of Testing:**
15
+ - *In-Scope Endpoints / Journeys:* `[List of APIs, gRPC methods, or user journeys]`
16
+ - *Out-of-Scope Dependencies:* `[Third-party payment gateways, SMS providers, external CDNs — to be mocked]`
17
+ - **Key Business Stakeholders:** `[Product Owner, Lead Architect, SRE Manager]`
18
+
19
+ ---
20
+
21
+ ## 2. Software Quality Criteria (ISO/IEC 25010:2023 SQuaRE)
22
+
23
+ | SQuaRE Sub-Characteristic | Target Metric / Evaluation Criteria | Verification Tool / Engine |
24
+ | :--- | :--- | :--- |
25
+ | **Time Behaviour** | P90, P95, P99 Tail Latencies, Peak Throughput (TPS/RPS), TTFB | `tools/jtl_analyzer.py` / `k6` |
26
+ | **Resource Utilization** | CPU (< 80%), RAM/JVM GC (< 85%), DB Connection Pool (< 80%), I/O | `tools/telemetry_correlator.py` |
27
+ | **Capacity** | Maximum Concurrency Envelope & Degradation Cliff without crash | `tools/generate_jmx_plan.py` |
28
+
29
+ ---
30
+
31
+ ## 3. SRE Non-Functional Requirements & SLA/SLO Budgets
32
+
33
+ - **Service Level Indicator (SLI):** Percentiles calculated on steady-state traffic (excluding warm-up/cool-down transients).
34
+ - **Service Level Objective (SLO):**
35
+ - **P95 Latency:** $\le 500\text{ms}$ (Warning at $400\text{ms}$)
36
+ - **P99 Latency:** $\le 1000\text{ms}$
37
+ - **Max Allowable Error Rate:** $< 1.0\%$ (HTTP 5xx, timeouts, connection drops)
38
+ - **Throughput Budget:** $\ge [Target]\text{ RPS}$ under normal peak load
39
+ - **Service Level Agreement (SLA Gate):** Automated pipeline halt (Exit Code 1) if P95 breaches $500\text{ms}$ or error rate $\ge 1.0\%$.
40
+
41
+ ---
42
+
43
+ ## 4. Operational Profile & Workload Modeling (ISTQB CT-PT)
44
+
45
+ - **Traffic Distribution (Pareto 70/20/10 Rule):**
46
+ - High-frequency browsing / querying (Read): `70%`
47
+ - Mutation / Transactional creation (Write): `20%`
48
+ - Complex aggregation / Reporting (Heavy): `10%`
49
+ - **User Behavior & Pacing:**
50
+ - **Realistic Think-Time:** Uniform Random Timers `1,000ms – 5,000ms` (Zero think-time is strictly forbidden).
51
+ - **Traffic Model:** Open Arrival-Rate (`ramping-arrival-rate` / throughput pacing) for public web APIs; Concurrency VU pool for private employee portals.
52
+ - **Session Isolation:** Pre-seeded user pools (`users.csv` or `userPool SharedArray`) with dedicated auth tokens per VU.
53
+
54
+ ---
55
+
56
+ ## 5. Workload Archetypes & Execution Schedule
57
+
58
+ | # | Archetype | Target Concurrency / Rate | Duration | Objective / Risk Addressed |
59
+ |---|:---|:---|:---|:---|
60
+ | **1** | **Smoke Test** | 1 – 5 VUs | 2 min | Pre-flight sanity check & endpoint reachability. |
61
+ | **2** | **Average Load Test** | 50 – 200 VUs | 15 – 30 min | Verify SLA compliance under expected peak volume. |
62
+ | **3** | **Stress Test (Step-Up)**| 200 $\rightarrow$ 2,000 VUs | 30 – 45 min | Locate degradation cliff, saturation knee & recovery. |
63
+ | **4** | **Spike Test** | 10 $\rightarrow$ 1,000 VUs | 5 – 10 min | Test autoscaling speed & circuit breaker recovery. |
64
+ | **5** | **Soak / Endurance** | 50 – 100 VUs | 2 – 12 hours | Uncover JVM memory leaks, GC pauses, connection leaks. |
65
+ | **6** | **Scalability Test** | 50 $\rightarrow$ 100 $\rightarrow$ 200 VUs | 30 – 60 min | Verify linear scale-up / scale-out elasticity ($\ge 80\%$). |
66
+
67
+ ---
68
+
69
+ ## 6. Test Environment Topology & SPEC/TPC Disclosure
70
+
71
+ - **Hardware & Host Specification:**
72
+ - Application Server: `[e.g., 4 vCPU, 16GB RAM, Linux Kernel 5.15]`
73
+ - Database Cluster: `[e.g., PostgreSQL 15, max_connections=200, pool_size=50]`
74
+ - Network Bandwidth: `[e.g., 10 Gbps VPC / local internal virtual bridge]`
75
+ - **Steady-State Measurement Interval (TPC Standard):**
76
+ - Warm-up Period: `[e.g., 3 minutes (excluded from SLA calculations)]`
77
+ - Steady-State Window: `[e.g., 10 minutes (official evaluation window)]`
78
+ - Cool-down Period: `[e.g., 2 minutes (excluded from SLA calculations)]`
79
+
80
+ ---
81
+
82
+ ## 7. Safety Gates, Pre-Flight Checks & Kill-Switch
83
+
84
+ - [ ] **Dedicated Isolated Environment:** Verified non-production target; staging approval granted.
85
+ - [ ] **Synthetic Test Data:** Parameterized with dynamic prefix `PERF_AUTO_${timestamp}_${thread}`.
86
+ - [ ] **Kill-Switch Threshold:** Configured `abortOnFail` if aggregate error rate $> 10\%$ over a 30s window.
87
+ - [ ] **Teardown & Cleanup Automation:** Reversible sandbox rollback (`reversible_sandbox.py`) scheduled post-test.
88
+
89
+ ---
90
+
91
+ ## 8. Telemetry, APM & Observability (OpenTelemetry)
92
+
93
+ - **Distributed Tracing Header:** Injected W3C Trace Context `traceparent` (`00-{trace_id}-{span_id}-01`).
94
+ - **Telemetry Streaming:** InfluxDB Backend Listener / Prometheus Node Exporter.
95
+ - **Root Cause Analysis (RCA):** Automated Pearson correlation matrix connecting tail latencies with CPU, Memory, and Database thread pools.
96
+
97
+ ---
98
+
99
+ ## 9. Definition of Done & Gate 10 Acceptance
100
+
101
+ - [ ] Test execution completed headlessly without manual browser GUI interference.
102
+ - [ ] Steady-state P95 latency is strictly below SLA budget.
103
+ - [ ] Error rate is strictly below 1.0% (0% on authentication & transaction routes).
104
+ - [ ] Zero database primary key collisions or lock contention from unparameterized test data.
105
+ - [ ] Artifacts generated and saved to `Reports/perf/`: `perf_results.jtl` and `k6_summary.json`.
106
+ - [ ] Gate 10 DoD status: **PASSED (`python tools/verify_all.py --mode hub`)**.
@@ -42,10 +42,11 @@ policies:
42
42
 
43
43
  1. **Performance as an Assertion:**
44
44
  - Functional tests must double as non-functional guardrails. If a critical workflow takes > 10 seconds to load, it should fail performance budgets.
45
- 2. **Google Core Web Vitals Metrics:**
46
- - **LCP (Largest Contentful Paint):** Target $\le 2.5\text{s}$ (Good).
47
- - **CLS (Cumulative Layout Shift):** Target $\le 0.1$ (Good).
48
- - **TTFB (Time to First Byte):** Target $\le 800\text{ms}$ (Good).
45
+ 2. **Google Core Web Vitals & W3C Metrics:**
46
+ - **LCP (Largest Contentful Paint):** Target $\le 2.5\text{s}$ (Good), $> 4.0\text{s}$ (Poor).
47
+ - **CLS (Cumulative Layout Shift):** Target $\le 0.1$ (Good), $> 0.25$ (Poor).
48
+ - **INP (Interaction to Next Paint):** Target $\le 200\text{ms}$ (Good), $> 500\text{ms}$ (Poor) — Official W3C/Google replacement for legacy FID.
49
+ - **TTFB (Time to First Byte):** Target $\le 800\text{ms}$ (Good) via W3C Navigation Timing Level 2.
49
50
  3. **Observation-Masked Telemetry:**
50
51
  - Never dump thousands of CDP trace lines into context. Only output calculated metric scores and flagged slow endpoints (> 1500ms).
51
52
 
@@ -53,28 +54,88 @@ policies:
53
54
 
54
55
  ### Standard Playwright Performance Measurement
55
56
 
56
- ### Pattern A: Extracting Core Web Vitals via JavaScript Evaluation
57
+ ### Pattern A: W3C Navigation Timing Level 2 (Modern Performance Audit)
57
58
  <!-- classification: illustrative -->
58
59
  ```typescript
59
60
  import { test, expect } from '@playwright/test';
60
61
 
61
- test('audit core web vitals on dashboard', async ({ page }) => {
62
+ test('audit navigation timing level 2 and ttfb on dashboard', async ({ page }) => {
62
63
  await page.goto('/dashboard');
63
64
  await page.waitForLoadState('networkidle');
64
65
 
65
- const performanceTiming = JSON.parse(
66
- await page.evaluate(() => JSON.stringify(window.performance.timing))
67
- );
66
+ // W3C Navigation Timing Level 2 Standard
67
+ const navTiming = await page.evaluate(() => {
68
+ const navEntries = performance.getEntriesByType('navigation') as PerformanceNavigationTiming[];
69
+ if (!navEntries || navEntries.length === 0) return null;
70
+ const nav = navEntries[0];
71
+ return {
72
+ ttfb: Math.round(nav.responseStart - nav.requestStart),
73
+ domContentLoaded: Math.round(nav.domContentLoadedEventEnd - nav.startTime),
74
+ loadComplete: Math.round(nav.loadEventEnd - nav.startTime),
75
+ transferSize: nav.transferSize,
76
+ };
77
+ });
78
+
79
+ expect(navTiming).not.toBeNull();
80
+ console.log(`[W3C NAV2] TTFB: ${navTiming?.ttfb}ms | DOM Loaded: ${navTiming?.domContentLoaded}ms`);
81
+ expect(navTiming!.ttfb).toBeLessThan(800); // W3C Good TTFB threshold
82
+ });
83
+ ```
84
+
85
+ ### Pattern B: Measuring Interaction to Next Paint (INP) via PerformanceObserver
86
+ <!-- classification: illustrative -->
87
+ ```typescript
88
+ test('audit interaction to next paint (INP) on button click', async ({ page }) => {
89
+ await page.goto('/checkout');
90
+ await page.waitForLoadState('networkidle');
91
+
92
+ // Inject W3C PerformanceObserver tracking event durations
93
+ await page.evaluate(() => {
94
+ (window as any).__maxEventDuration = 0;
95
+ const observer = new PerformanceObserver((list) => {
96
+ for (const entry of list.getEntries()) {
97
+ const duration = (entry as any).duration;
98
+ if (duration > (window as any).__maxEventDuration) {
99
+ (window as any).__maxEventDuration = duration;
100
+ }
101
+ }
102
+ });
103
+ observer.observe({ type: 'event', buffered: true, durationThreshold: 16 } as any);
104
+ });
68
105
 
69
- const ttfb = performanceTiming.responseStart - performanceTiming.navigationStart;
70
- const domLoadTime = performanceTiming.domContentLoadedEventEnd - performanceTiming.navigationStart;
106
+ // Perform realistic user interaction
107
+ await page.locator('#submit-order-button').click();
108
+ await page.waitForTimeout(500); // Allow interaction cycle to complete
71
109
 
72
- console.log(`[PERF] TTFB: ${ttfb}ms | DOM Load: ${domLoadTime}ms`);
73
- expect(ttfb).toBeLessThan(1500); // 1.5s threshold
110
+ const maxInp = await page.evaluate(() => (window as any).__maxEventDuration || 0);
111
+ console.log(`[CORE WEB VITALS] Measured INP: ${maxInp}ms`);
112
+ expect(maxInp).toBeLessThan(200); // Google/W3C INP budget <= 200ms
113
+ });
114
+ ```
115
+
116
+ ### Pattern C: CDP Network Throttling & Fast 3G Simulation
117
+ <!-- classification: illustrative -->
118
+ ```typescript
119
+ test('simulate fast 3g conditions and 4x cpu slowdown', async ({ page }) => {
120
+ const client = await page.context().newCDPSession(page);
121
+
122
+ // Emulate mobile network conditions (Fast 3G: 1.6 Mbps download, 750 kbps upload, 150ms latency)
123
+ await client.send('Network.emulateNetworkConditions', {
124
+ offline: false,
125
+ latency: 150,
126
+ downloadThroughput: (1.6 * 1024 * 1024) / 8,
127
+ uploadThroughput: (750 * 1024) / 8,
128
+ });
129
+
130
+ // Emulate 4x CPU slowdown (budget mobile CPU emulation)
131
+ await client.send('Emulation.setCPUThrottlingRate', { rate: 4 });
132
+
133
+ await page.goto('/catalog');
134
+ await page.waitForLoadState('networkidle');
74
135
  });
75
136
  ```
76
137
 
77
- ### Pattern B: Tracking Slow Network Endpoints
138
+ ### Pattern D: Tracking Slow Network Endpoints
78
139
  <!-- classification: illustrative -->
79
140
  ```typescript
80
141
  test('detect slow backend microservices', async ({ page }) => {
@@ -153,11 +153,22 @@ For systems with integrated LLMs, tool-calling agents, or MCP servers, inspect:
153
153
 
154
154
  ---
155
155
 
156
+ ### OWASP ASVS 4.0.3, CVSS v3.1/v4.0 & Regulatory Compliance
157
+ - Execute automated Level 1 ASVS probes and checklist generation via `tools/asvs_auditor.py` (`agent-qa asvs --probe <url> --level 1`).
158
+ - Compute quantitative CVSS Base/Temporal vectors and qualitative severity ratings via `tools/cvss_calculator.py` (`agent-qa cvss "<vector>"`).
159
+ - Audit regulatory privacy and payment requirements via `tools/regulatory_compliance.py` (`agent-qa compliance`):
160
+ - **GDPR / PDPA:** Right to be Forgotten (Article 17) and Consent logs (Article 7).
161
+ - **PCI-DSS v4.0:** Requirement 3.4 (PAN Masking with Luhn validation) and Requirement 3.2 (CVV exclusion).
162
+ - **ISO 15489:** Electronic Records Management retention period verification.
163
+
164
+ ---
165
+
156
166
  ## 4. Verification Points & DoD
157
167
 
158
168
  1. Target endpoints verified against authorization whitelist and consent grant.
159
- 2. Security scan covers OWASP Top 10 vulnerabilities (SQLi, XSS, SSRF, broken auth, security headers).
160
- 3. Findings categorized by CVSS severity with reproduction steps and remediation guidance.
169
+ 2. Security scan covers OWASP Top 10 vulnerabilities (SQLi, XSS, SSRF, broken auth, security headers) and ASVS Level 1 baseline.
170
+ 3. Zero unmasked PAN leaks or stored CVVs detected across test logs and responses (`tools/regulatory_compliance.py --gate`).
171
+ 4. Findings categorized by CVSS v3.1/v4.0 severity vector with reproduction steps and remediation guidance.
161
172
 
162
173
  ---
163
174
 
@@ -148,6 +148,9 @@ python tools/wcag_audit.py https://staging.example.com --url -o docs/wcag_audit_
148
148
 
149
149
  # CLI integration via agent-qa
150
150
  agent-qa wcag-audit path/to/page.html
151
+
152
+ # Export standardized Section 508 & EN 301 549 VPAT 2.5 Report
153
+ agent-qa vpat --url https://staging.example.com --out-md docs/VPAT_REPORT.md
151
154
  ```
152
155
 
153
156
  ---
@@ -156,7 +159,7 @@ agent-qa wcag-audit path/to/page.html
156
159
 
157
160
  1. Automated accessibility scans run via axe-core / Playwright with 0 Level A/AA violations.
158
161
  2. Manual check protocol verifies keyboard tab order, focus rings, and skip links.
159
- 3. Audit report maps violations to specific WCAG 2.2 criteria with remediation code.
162
+ 3. Audit report maps violations to specific WCAG 2.2 criteria with remediation code and exports Section 508 / EN 301 549 VPAT 2.5 reports (`tools/vpat_exporter.py`).
160
163
 
161
164
  ---
162
165
 
@@ -124,11 +124,18 @@ def assert_pdf_contains_text(pdf_path: str, expected_keywords: list):
124
124
 
125
125
  ---
126
126
 
127
+ ### ISO/IEC 25012 Data Quality & DAMA-DMBOK ETL Reconciliation
128
+ - Audit datasets against ISO 25012 characteristics (Completeness, Consistency, Accuracy, Currentness SLA) via `tools/etl_data_reconciler.py --quality-audit <file>`.
129
+ - Reconcile Source vs Target ETL migration files (row count, dropped key deltas, mutated values, aggregate sum balance) via `agent-qa etl --reconcile --source src.csv --target tgt.csv --key id --gate`.
130
+
131
+ ---
132
+
127
133
  ## 4. Verification Points & DoD
128
134
 
129
135
  1. Downloaded file format, MIME type, and byte size verified.
130
136
  2. Deep content inspection validates specific text strings, cell coordinates, and row counts.
131
- 3. Downloaded test files cleaned up from temporary sandbox upon test completion.
137
+ 3. ETL data migrations and report exports reconciled with zero dropped records (`tools/etl_data_reconciler.py --gate`).
138
+ 4. Downloaded test files cleaned up from temporary sandbox upon test completion.
132
139
 
133
140
  ---
134
141
 
@@ -52,8 +52,11 @@ policies:
52
52
  - Tax IDs, phone numbers, email RFC formats, and GIS coordinates (e.g., non-overlapping closed polygons).
53
53
  4. **PII Safety Invariant:**
54
54
  - Never copy or mirror real customer production data (PII) into test fixtures. All synthesized entities must use purely artificial data with identifiable prefixes (`AUTO_`, `PERF_AUTO_`, `AUTO_RACE_`) paired with automated teardown cleanup hooks.
55
- 5. **Vietnam Locale Standards:**
55
+ 5. **Vietnam Locale & Unicode CLDR Standards:**
56
56
  - Support national telecom prefixes (`03x`, `05x`, `07x`, `08x`, `09x`), 12-digit Citizen Identity Card (CCCD) format, full UTF-8 Vietnamese diacritics, and VND currency conventions (`100.000 ₫` / `100.000 VND`).
57
+ - Validate international formatting rules (dates, currencies, separators) across world locales (`en-US`, `vi-VN`, `de-DE`, `fr-FR`, `ja-JP`, `ar-SA`) using `tools/cldr_l10n_verifier.py` (`agent-qa cldr`).
58
+ 6. **Pseudo-Localization for UI Resilience:**
59
+ - Generate accented, length-expanded test strings (`[!!! Çĥèçķòùţ ~one two~ !!!]`) using `tools/cldr_l10n_verifier.py --pseudo` to automatically expose hardcoded text, character encoding flaws, and UI clipping.
57
60
 
58
61
  ---
59
62
 
@@ -78,10 +78,18 @@ python tools/mipro_assertion_tuner.py compile-module \
78
78
  # 2. Evaluate compiled prompt module against testset
79
79
  python tools/mipro_assertion_tuner.py evaluate \
80
80
  --module compiled/optimized_assertion_gen.json
81
+
82
+ # 3. Evaluate RAG & LLM Faithfulness & Hallucination via NIST AI RMF
83
+ agent-qa ai-eval --query "..." --context "..." --answer "..." --gate-faithfulness 0.8
81
84
  ```
82
85
 
83
86
  ---
84
87
 
88
+ ### NIST AI RMF & ISO 42001 Evaluation Sentinel
89
+ - Evaluate RAG triplets (Query-Context-Answer) and LLM outputs for groundedness (anti-hallucination), relevance, and prompt injection safety via `tools/ai_eval_sentinel.py` (`agent-qa ai-eval`).
90
+
91
+ ---
92
+
85
93
  ### Strict Operating Invariants
86
94
 
87
95
  - **Mutation Score Hard Floor:** Prompts must achieve a minimum mutant kill rate of **85%**.