@teevee98/synapse-hub 4.3.0 → 4.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/agents/api-engineer.md +8 -0
- package/.agents/agents/perf-engineer.md +10 -2
- package/.agents/hooks/session_start.py +52 -0
- package/.agents/hooks.json +12 -0
- package/.agents/rules/performance_testing_rules.md +4 -3
- package/.agents/skills/01-web-automation/clean-pom-crafter/SKILL.md +6 -3
- package/.agents/skills/01-web-automation/e2e-testing-patterns/SKILL.md +3 -1
- package/.agents/skills/02-test-design/rbt-manual-testing/SKILL.md +11 -5
- package/.agents/skills/04-performance-and-load/backend-load-and-stress-patterns/SKILL.md +4 -1
- package/.agents/skills/04-performance-and-load/backend-load-and-stress-patterns/references/performance_test_plan_29119.md +106 -0
- package/.agents/skills/04-performance-and-load/web-vitals-and-performance-profiler/SKILL.md +75 -14
- package/.agents/skills/05-security-and-compliance/security-dast-sentinel/SKILL.md +13 -2
- package/.agents/skills/05-security-and-compliance/wcag-audit-patterns/SKILL.md +4 -1
- package/.agents/skills/07-data-and-state/file-and-report-validator/SKILL.md +8 -1
- package/.agents/skills/07-data-and-state/synthetic-test-data-generator/SKILL.md +4 -1
- package/.agents/skills/11-ai-and-prompting/mipro-prompt-compiler/SKILL.md +8 -0
- package/.agents/templates/performance-test-plan-template.md +106 -0
- package/README.en.md +2 -2
- package/README.md +2 -2
- package/agentskills.json +1 -1
- package/docs/ARCHITECTURE.en.md +95 -2
- package/docs/ARCHITECTURE.md +86 -2
- package/docs/TOOL_CATALOG.md +11 -2
- package/package.json +1 -1
- package/plugin.json +1 -1
- package/skills_manifest.json +1 -1
- package/synapse.config.json +1 -1
- package/tools/.infer_baseline.json +44 -3
- package/tools/TOOLS_RUNBOOK.md +237 -0
- package/tools/TOOL_CATALOG.json +138 -3
- package/tools/__init__.py +1 -1
- package/tools/ai_eval_sentinel.py +403 -0
- package/tools/asvs_auditor.py +410 -0
- package/tools/browser_helpers.js +2 -2
- package/tools/chaos_resilience_probe.py +362 -0
- package/tools/cldr_l10n_verifier.py +309 -0
- package/tools/cli.py +191 -3
- package/tools/cvss_calculator.py +412 -0
- package/tools/dashboard_generator.py +8 -8
- package/tools/dspy_qa_signatures.py +105 -1
- package/tools/etl_data_reconciler.py +460 -0
- package/tools/generate_jmx_plan.py +12 -4
- package/tools/jtl_analyzer.py +23 -1
- package/tools/mcp_docx_server.py +1 -1
- package/tools/mcp_excel_server.py +1 -1
- package/tools/mcp_server.py +1 -1
- package/tools/mobile_resilience_sentinel.py +273 -0
- package/tools/observability_tester.py +235 -0
- package/tools/payment_interop_simulator.py +337 -0
- package/tools/pre_push_guard.py +306 -0
- package/tools/regulatory_compliance.py +336 -0
- package/tools/rules_router.py +1 -1
- package/tools/task_memory_manager.py +12 -1
- package/tools/test_memory_os.py +48 -0
- package/tools/tmmi_auditor.py +172 -0
- package/tools/tot_exploratory_tester.py +17 -0
- package/tools/ui_server.py +1 -1
- package/tools/update_sentinel.py +109 -43
- package/tools/vpat_exporter.py +428 -0
- package/tools/welcome_page.py +1 -1
|
@@ -21,6 +21,14 @@ tools:
|
|
|
21
21
|
|
|
22
22
|
# Enterprise API & Contract Testing Specialist (api-engineer)
|
|
23
23
|
|
|
24
|
+
> **International Standards Alignment:**
|
|
25
|
+
> - **Google SRE Four Golden Signals:** Latency (Success vs Error separation), Traffic (RPS/TPS), Errors (Explicit 4xx/5xx & Implicit false 200s), Saturation (CPU, DB Pool, Memory).
|
|
26
|
+
> - **Percentile Tail Latency:** Mandatory P50, P95, P99, P99.9 (strict ban on Average).
|
|
27
|
+
> - **Industry Tiered SLAs:** 4-tier budgets (Simple/Cache, Standard CRUD, Complex/Payment, Async 202).
|
|
28
|
+
> - **Technical Standards:** IETF RFC 9110/9112/9113, OpenAPI OAS 3.0/3.1, CNCF gRPC/Protobuf.
|
|
29
|
+
> - **Architectural Resilience:** RFC 6585 Rate Limiting (429 + Retry-After), Timeout/Circuit Breaker, HikariCP Connection Pool Saturation.
|
|
30
|
+
> - **Mandatory Rule:** Adheres strictly to [api_testing_rules.md](../rules/api_testing_rules.md).
|
|
31
|
+
|
|
24
32
|
## Core Responsibilities:
|
|
25
33
|
- **OpenAPI & Swagger Test Matrix Synthesis:**
|
|
26
34
|
- Ingest OpenAPI 3.x / Swagger specifications, Postman collections, and API contracts.
|
|
@@ -21,10 +21,18 @@ tools:
|
|
|
21
21
|
|
|
22
22
|
# Enterprise Performance & Load Engineering Agent (perf-engineer)
|
|
23
23
|
|
|
24
|
+
> **International Standards Alignment:**
|
|
25
|
+
> - **ISO/IEC 25010:2023 (SQuaRE):** Performance Efficiency (Time Behaviour, Resource Utilization, Capacity).
|
|
26
|
+
> - **ISO/IEC/IEEE 29119-3 & 29119-4:** Non-Functional Test Plan (PTP), Test Design Specs, and Risk Reporting.
|
|
27
|
+
> - **ISTQB CT-PT:** 6 Test Archetypes (Smoke, Load, Stress, Spike, Soak, Scalability), Pareto 70/20/10 Operational Profile.
|
|
28
|
+
> - **TPC / SPEC:** Strict steady-state measurement interval, warm-up/cool-down exclusion.
|
|
29
|
+
> - **W3C Web Performance & Google SRE:** Navigation Timing L2, Core Web Vitals (LCP/CLS/INP/TTFB), Gate 10 SLA Sentinel, and OpenTelemetry Distributed Tracing (`traceparent`).
|
|
30
|
+
|
|
24
31
|
## Core Responsibilities:
|
|
25
|
-
- **Workload Modeling & Archetype Planning:**
|
|
32
|
+
- **Workload Modeling & Archetype Planning (ISTQB CT-PT & ISO 25010):**
|
|
26
33
|
- Ingest Non-Functional Requirements (NFRs), target SLAs (P90, P95, P99 latencies, max error rate < 1.0%), and peak concurrency goals.
|
|
27
|
-
- Plan progressive test execution sequences: Smoke (1–5 VUs) -> Baseline Average Load -> Stress -> Spike -> Soak test.
|
|
34
|
+
- Plan progressive test execution sequences: Smoke (1–5 VUs) -> Baseline Average Load -> Stress -> Spike -> Soak -> Scalability test.
|
|
35
|
+
- Formulate ISO 29119-3 compliant Performance Test Plans (`.agents/templates/performance-test-plan-template.md`).
|
|
28
36
|
- Model realistic user think time (1s–5s) and open arrival rate profiles for public traffic (`tools/generate_jmx_plan.py --workload`).
|
|
29
37
|
- **Autonomous Script Synthesis (JMeter / k6 / Locust):**
|
|
30
38
|
- Generate production-grade JMeter `.jmx` test plans and Locust `locustfile.py` directly from OpenAPI/Swagger specs, Postman collections, or HAR captures via `tools/generate_jmx_plan.py`.
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
# SPDX-FileCopyrightText: Copyright (c) 2026 Thế Vị (@TeeVee98). All rights reserved.
|
|
3
|
+
# SPDX-License-Identifier: MIT
|
|
4
|
+
# Project: Synapse-Hub (https://github.com/TeeVee98/Synapse-Hub)
|
|
5
|
+
|
|
6
|
+
"""
|
|
7
|
+
Antigravity SessionStart Hook — Automated Upstream Drift & Update Sentinel
|
|
8
|
+
-------------------------------------------------------------------------
|
|
9
|
+
Invoked automatically by Antigravity on session initialization.
|
|
10
|
+
Checks upstream NPM registry / GitHub for newer Synapse Hub releases
|
|
11
|
+
and non-destructively synchronizes the agent environment.
|
|
12
|
+
Outputs strict JSON matching the Antigravity SessionStart hook contract.
|
|
13
|
+
"""
|
|
14
|
+
|
|
15
|
+
import os
|
|
16
|
+
import sys
|
|
17
|
+
import json
|
|
18
|
+
|
|
19
|
+
# Enforce UTF-8 stdout encoding for Windows console compatibility
|
|
20
|
+
if hasattr(sys.stdout, 'reconfigure'):
|
|
21
|
+
try:
|
|
22
|
+
sys.stdout.reconfigure(encoding='utf-8')
|
|
23
|
+
sys.stderr.reconfigure(encoding='utf-8')
|
|
24
|
+
except Exception:
|
|
25
|
+
pass
|
|
26
|
+
|
|
27
|
+
CUR_DIR = os.path.dirname(os.path.abspath(__file__))
|
|
28
|
+
# Check both repo layout (.agents/hooks/..) and plugin layout
|
|
29
|
+
CANDIDATE_ROOTS = [
|
|
30
|
+
os.path.abspath(os.path.join(CUR_DIR, "..", "..")), # repo: root/tools/
|
|
31
|
+
os.path.abspath(os.path.join(CUR_DIR, "..")), # root/.agents/hooks
|
|
32
|
+
os.path.abspath(CUR_DIR)
|
|
33
|
+
]
|
|
34
|
+
|
|
35
|
+
sentinel_found = False
|
|
36
|
+
for root in CANDIDATE_ROOTS:
|
|
37
|
+
tools_dir = os.path.join(root, "tools")
|
|
38
|
+
if os.path.isdir(tools_dir) and os.path.exists(os.path.join(tools_dir, "update_sentinel.py")):
|
|
39
|
+
if tools_dir not in sys.path:
|
|
40
|
+
sys.path.insert(0, tools_dir)
|
|
41
|
+
try:
|
|
42
|
+
import update_sentinel
|
|
43
|
+
update_sentinel.run_auto_update_hook()
|
|
44
|
+
sentinel_found = True
|
|
45
|
+
break
|
|
46
|
+
except Exception as err:
|
|
47
|
+
sys.stderr.write(f"[session_start_hook] Error running update_sentinel: {err}\n")
|
|
48
|
+
break
|
|
49
|
+
|
|
50
|
+
if not sentinel_found:
|
|
51
|
+
# Failsafe fallback: output clean empty payload to never break Antigravity session
|
|
52
|
+
print(json.dumps({"injectSteps": []}))
|
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
{
|
|
2
|
+
"synapse-auto-updater": {
|
|
3
|
+
"enabled": true,
|
|
4
|
+
"SessionStart": [
|
|
5
|
+
{
|
|
6
|
+
"type": "command",
|
|
7
|
+
"command": "python -c \"import os, subprocess, sys; s = 'hooks/session_start.py' if os.path.exists('hooks/session_start.py') else '.agents/hooks/session_start.py'; subprocess.run([sys.executable, s])\"",
|
|
8
|
+
"timeout": 15
|
|
9
|
+
}
|
|
10
|
+
]
|
|
11
|
+
}
|
|
12
|
+
}
|
|
@@ -6,15 +6,16 @@ profiles: [perf, full]
|
|
|
6
6
|
|
|
7
7
|
# Performance & Load Testing Invariants (k6 · JMeter · Locust)
|
|
8
8
|
|
|
9
|
-
> **Scope:** Applies whenever designing, executing, or analyzing performance, load, stress, spike, or
|
|
9
|
+
> **Scope:** Applies whenever designing, executing, or analyzing performance, load, stress, spike, soak, or scalability workloads.
|
|
10
|
+
> **International Standards Alignment:** Codifies ISO/IEC 25010:2023 (Performance Efficiency), ISO/IEC/IEEE 29119-3/4, ISTQB CT-PT, TPC/SPEC Steady-State discipline, Google SRE SLA budgets, and W3C Trace Context.
|
|
10
11
|
> Complement to `AGENTS.md` enforcing enterprise safety, workload realism, and zero-hallucination analysis.
|
|
11
12
|
|
|
12
13
|
---
|
|
13
14
|
|
|
14
15
|
## 1. Safety Gate & Pre-Flight Invariants
|
|
15
16
|
1. **Dedicated Environments Only:** Never run load tests against production, shared staging, or third-party APIs without written approval.
|
|
16
|
-
2. **Pre-Run Declaration:** Declare target host, archetype (Smoke, Average Load, Stress, Spike, Soak), VU/arrival profile, duration, and kill-switch threshold (`abortOnFail`).
|
|
17
|
-
3. **Progressive Escalation:** Smoke (1–5 VUs) -> Baseline Average Load -> Stress / Spike / Soak. Never start high-VU runs autonomously.
|
|
17
|
+
2. **Pre-Run Declaration:** Declare target host, archetype (Smoke, Average Load, Stress, Spike, Soak, Scalability), VU/arrival profile, duration, and kill-switch threshold (`abortOnFail`).
|
|
18
|
+
3. **Progressive Escalation:** Smoke (1–5 VUs) -> Baseline Average Load -> Stress / Spike / Soak / Scalability. Never start high-VU runs autonomously.
|
|
18
19
|
4. **Hard Caps & Kill-Switch:** Scripts must enforce finite durations, rate caps, and automated abort (`abortOnFail` when error rate > 10%). Unbounded loops are strictly forbidden.
|
|
19
20
|
5. **Isolation & Cleanup:** Mock third-party providers (Payment, SMS, Captcha). Use synthetic test data (`PERF_AUTO_`); teardown after runs.
|
|
20
21
|
|
|
@@ -47,9 +47,12 @@ policies:
|
|
|
47
47
|
- Naming convention: `TC-<Number>_<FeatureName>` (e.g. `TC-101_CreateUserAccount.groovy` or `tc-101-create-user.spec.ts`).
|
|
48
48
|
2. **Unified Inheritance from `BasePage`:**
|
|
49
49
|
- All Page Classes inherit from `BasePage`.
|
|
50
|
-
3. **Zero Raw Locators in Scripts:**
|
|
51
|
-
- Scripts only orchestrate business workflows. All locators and UI interactions are encapsulated in Page Classes.
|
|
52
|
-
4. **
|
|
50
|
+
3. **Zero Raw Locators in Scripts (ISO/IEC/IEEE 29119-5 Keyword-Driven Separation):**
|
|
51
|
+
- Scripts only orchestrate business workflows. All locators and UI interactions are encapsulated in Page Classes, strictly adhering to ISO/IEC/IEEE 29119-5 Keyword-Driven separation of business logic from technical implementation.
|
|
52
|
+
4. **Clean POM Flat vs Screenplay Pattern Rationale:**
|
|
53
|
+
- Enforce Clean POM Flat Architecture (max 1-level shallow inheritance from `BasePage`).
|
|
54
|
+
- Screenplay Pattern (Actor/Task/Interaction/Ability micro-class fragmentation) is intentionally prohibited to prevent LLM Context Blowout and token waste.
|
|
55
|
+
5. **Mandatory 100% UTF-8 No BOM:**
|
|
53
56
|
- Zero BOM bytes (`0xEF, 0xBB, 0xBF`) across all test files.
|
|
54
57
|
|
|
55
58
|
### Phase 2: Master Locator Priority Map (6 Tiers)
|
|
@@ -38,7 +38,7 @@ Engineering standard for structuring, scaling, and maintaining large-scale End-t
|
|
|
38
38
|
|
|
39
39
|
---
|
|
40
40
|
|
|
41
|
-
### Test Pyramid & E2E Scope Boundary
|
|
41
|
+
### Test Pyramid & E2E Scope Boundary (Mike Cohn Golden Ratio)
|
|
42
42
|
|
|
43
43
|
```mermaid
|
|
44
44
|
flowchart TD
|
|
@@ -49,6 +49,8 @@ flowchart TD
|
|
|
49
49
|
Unit --> Integ --> E2E
|
|
50
50
|
```
|
|
51
51
|
|
|
52
|
+
- **Mike Cohn's Test Pyramid Standard:** 60% Unit Tests — 30% Service/Integration/API Tests — 10% E2E UI Tests.
|
|
53
|
+
- **ISTQB CT-TAE Alignment:** Implements TAA (Test Automation Architecture) layered separation: Test Generation (`test-designer`), Test Definition (`pom-crafter`), Test Execution (`test-runner`), and Test Adaptation (Playwright CDP / W3C WebDriver BiDi).
|
|
52
54
|
- **What belongs in E2E:** High-risk customer journeys (Sign-up, Onboarding, Checkout, File Export, Role transitions).
|
|
53
55
|
- **What to avoid in E2E:** Exhaustive boundary value checks of simple form fields (delegate to Unit/API tests).
|
|
54
56
|
|
|
@@ -36,7 +36,7 @@ policies:
|
|
|
36
36
|
|
|
37
37
|
### Overview & Capabilities
|
|
38
38
|
|
|
39
|
-
**RBT Manual Testing** is an enterprise-grade skill for designing manual test suites using **Risk-Based Testing (RBT)** and coverage matrix analysis. It provides two operational modes:
|
|
39
|
+
**RBT Manual Testing** is an enterprise-grade skill for designing manual test suites using **Risk-Based Testing (RBT)** and coverage matrix analysis (fully aligned with **ISO/IEC/IEEE 29119-2** test processes and **ISO/IEC/IEEE 29119-3** test documentation standards). It provides two operational modes:
|
|
40
40
|
|
|
41
41
|
| Mode | When to Use | Scope & Characteristics | Execution Time |
|
|
42
42
|
| :--- | :--- | :--- | :--- |
|
|
@@ -46,7 +46,7 @@ policies:
|
|
|
46
46
|
### Core Tenets:
|
|
47
47
|
1. **Human Strategy:** Engineers define business boundaries, risk appetite, and resolve flagged ambiguities.
|
|
48
48
|
2. **AI Execution:** Agent decomposes modules, builds coverage matrices, applies the 15-field validation checklist, and computes risk scores.
|
|
49
|
-
3. **Enterprise Deliverables:** Standardized 12-column Markdown tables with automation metadata, ready for export to Excel (`.xlsx`), Jira Xray, or TestRail.
|
|
49
|
+
3. **Enterprise Deliverables (ISO/IEC/IEEE 29119-3 Aligned):** Standardized 12-column Markdown tables with automation metadata, ready for export to Excel (`.xlsx`), Jira Xray, or TestRail.
|
|
50
50
|
|
|
51
51
|
---
|
|
52
52
|
|
|
@@ -124,8 +124,8 @@ When `Concurrency Gate = ON`, generate targeted cases for applicable vectors:
|
|
|
124
124
|
3. **Stale State / Optimistic Locking:** User A and User B open the same record; User A saves first, then User B saves outdated data -> verify system blocks silent overwrite (version/ETag check) or prompts merge/refresh.
|
|
125
125
|
4. **Retry & Idempotency:** Simulate network timeout after server commits, then retry with the same `Idempotency-Key` -> verify server returns the original result without creating duplicate records.
|
|
126
126
|
|
|
127
|
-
#### Conditional ISTQB Advanced Test Design Techniques
|
|
128
|
-
In addition to Equivalence Partitioning (EP), Boundary Value Analysis (BVA), and Exploratory Testing, conditionally activate targeted formal techniques:
|
|
127
|
+
#### Conditional ISTQB & ISO/IEC/IEEE 29119 Advanced Test Design Techniques
|
|
128
|
+
In addition to Equivalence Partitioning (EP), Boundary Value Analysis (BVA), and Exploratory Testing, conditionally activate targeted formal techniques (CT-FL / CT-TAE / ISO 29119-4):
|
|
129
129
|
|
|
130
130
|
| Technique | Auto-Activation Rule | Mandatory Generation Rule |
|
|
131
131
|
| :--- | :--- | :--- |
|
|
@@ -198,11 +198,17 @@ Generated workbooks automatically include:
|
|
|
198
198
|
|
|
199
199
|
---
|
|
200
200
|
|
|
201
|
+
### TMMi (Test Maturity Model integration) Maturity Assessment
|
|
202
|
+
- Assess repository and test engineering practices against TMMi Levels 1 through 5 (Level 2 Managed, Level 3 Defined, Level 4 Measured, Level 5 Optimization) via `tools/tmmi_auditor.py` (`agent-qa tmmi`).
|
|
203
|
+
|
|
204
|
+
---
|
|
205
|
+
|
|
201
206
|
## 4. Verification Points & DoD
|
|
202
207
|
|
|
203
208
|
1. Every test case carries explicit Risk Priority Number (RPN = Impact x Likelihood).
|
|
204
209
|
2. Test steps include preconditions, concrete test data, and verifiable expected outcomes.
|
|
205
|
-
3.
|
|
210
|
+
3. Test process conforms to ISO/IEC/IEEE 29119-2/3 and meets TMMi Level 3+ maturity requirements (`agent-qa tmmi --gate-level 3`).
|
|
211
|
+
4. Exported test suite generates clean tabular data (Excel / CSV / Markdown).
|
|
206
212
|
|
|
207
213
|
---
|
|
208
214
|
|
|
@@ -39,7 +39,7 @@ policies:
|
|
|
39
39
|
|
|
40
40
|
---
|
|
41
41
|
|
|
42
|
-
### Test Archetypes & SLA Targets
|
|
42
|
+
### Test Archetypes & SLA Targets (ISTQB CT-PT Standards)
|
|
43
43
|
|
|
44
44
|
| Archetype | VUs / Rate | Duration | Objective | SLA Assertion |
|
|
45
45
|
| :--- | :--- | :--- | :--- | :--- |
|
|
@@ -48,6 +48,9 @@ policies:
|
|
|
48
48
|
| **Stress Test** | 200 -> 2,000 VUs | 20 - 45 min | Find degradation cliff | Graceful 429/503 (no JVM OOM) |
|
|
49
49
|
| **Spike Test** | 10 -> 1,000 VUs | 5 - 10 min | Autoscaling & rate limiters | Gateway recovery < 30s |
|
|
50
50
|
| **Soak Test** | 50 - 100 VUs | 2 - 12 hr | Memory leaks & connection leaks | Stable heap & steady latency |
|
|
51
|
+
| **Scalability Test** | 50 -> 100 -> 200 VUs | 30 - 60 min | Linear scale-up/out elasticity | Throughput scales $\ge 80\%$ linearly |
|
|
52
|
+
|
|
53
|
+
👉 **Formal Test Plan Schema:** Refer to [**ISO/IEC/IEEE 29119-3 Performance Test Plan (`references/performance_test_plan_29119.md`)**](references/performance_test_plan_29119.md) for full environmental topology, NFR SLO targets, and risk declarations.
|
|
51
54
|
|
|
52
55
|
---
|
|
53
56
|
|
|
@@ -0,0 +1,106 @@
|
|
|
1
|
+
# Performance Test Plan (PTP) — ISO/IEC/IEEE 29119-3 Reference
|
|
2
|
+
|
|
3
|
+
> **Standard Compliance:** Conforms to ISO/IEC/IEEE 29119-3 (Non-Functional Test Plan), ISO/IEC 25010:2023 (Performance Efficiency), ISTQB CT-PT, and Google SRE Framework.
|
|
4
|
+
> **Target Execution Engines:** Apache JMeter 5.x, k6, Locust.
|
|
5
|
+
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
## 1. Document Overview & Context (ISO 29119-3 §5.2)
|
|
9
|
+
|
|
10
|
+
- **Test Plan ID:** `PTP-PERF-YYYYMMDD-[MODULE]`
|
|
11
|
+
- **Target System / AUT:** `[Application / Microservices under test]`
|
|
12
|
+
- **Revision History:** `v1.0 (Initial Draft)`
|
|
13
|
+
- **Author / Lead Performance QA:** `[AI Agent perf-engineer / Performance Architect]`
|
|
14
|
+
- **Scope of Testing:**
|
|
15
|
+
- *In-Scope Endpoints / Journeys:* `[List of APIs, gRPC methods, or user journeys]`
|
|
16
|
+
- *Out-of-Scope Dependencies:* `[Third-party payment gateways, SMS providers, external CDNs — to be mocked]`
|
|
17
|
+
- **Key Business Stakeholders:** `[Product Owner, Lead Architect, SRE Manager]`
|
|
18
|
+
|
|
19
|
+
---
|
|
20
|
+
|
|
21
|
+
## 2. Software Quality Criteria (ISO/IEC 25010:2023 SQuaRE)
|
|
22
|
+
|
|
23
|
+
| SQuaRE Sub-Characteristic | Target Metric / Evaluation Criteria | Verification Tool / Engine |
|
|
24
|
+
| :--- | :--- | :--- |
|
|
25
|
+
| **Time Behaviour** | P90, P95, P99 Tail Latencies, Peak Throughput (TPS/RPS), TTFB | `tools/jtl_analyzer.py` / `k6` |
|
|
26
|
+
| **Resource Utilization** | CPU (< 80%), RAM/JVM GC (< 85%), DB Connection Pool (< 80%), I/O | `tools/telemetry_correlator.py` |
|
|
27
|
+
| **Capacity** | Maximum Concurrency Envelope & Degradation Cliff without crash | `tools/generate_jmx_plan.py` |
|
|
28
|
+
|
|
29
|
+
---
|
|
30
|
+
|
|
31
|
+
## 3. SRE Non-Functional Requirements & SLA/SLO Budgets
|
|
32
|
+
|
|
33
|
+
- **Service Level Indicator (SLI):** Percentiles calculated on steady-state traffic (excluding warm-up/cool-down transients).
|
|
34
|
+
- **Service Level Objective (SLO):**
|
|
35
|
+
- **P95 Latency:** $\le 500\text{ms}$ (Warning at $400\text{ms}$)
|
|
36
|
+
- **P99 Latency:** $\le 1000\text{ms}$
|
|
37
|
+
- **Max Allowable Error Rate:** $< 1.0\%$ (HTTP 5xx, timeouts, connection drops)
|
|
38
|
+
- **Throughput Budget:** $\ge [Target]\text{ RPS}$ under normal peak load
|
|
39
|
+
- **Service Level Agreement (SLA Gate):** Automated pipeline halt (Exit Code 1) if P95 breaches $500\text{ms}$ or error rate $\ge 1.0\%$.
|
|
40
|
+
|
|
41
|
+
---
|
|
42
|
+
|
|
43
|
+
## 4. Operational Profile & Workload Modeling (ISTQB CT-PT)
|
|
44
|
+
|
|
45
|
+
- **Traffic Distribution (Pareto 70/20/10 Rule):**
|
|
46
|
+
- High-frequency browsing / querying (Read): `70%`
|
|
47
|
+
- Mutation / Transactional creation (Write): `20%`
|
|
48
|
+
- Complex aggregation / Reporting (Heavy): `10%`
|
|
49
|
+
- **User Behavior & Pacing:**
|
|
50
|
+
- **Realistic Think-Time:** Uniform Random Timers `1,000ms – 5,000ms` (Zero think-time is strictly forbidden).
|
|
51
|
+
- **Traffic Model:** Open Arrival-Rate (`ramping-arrival-rate` / throughput pacing) for public web APIs; Concurrency VU pool for private employee portals.
|
|
52
|
+
- **Session Isolation:** Pre-seeded user pools (`users.csv` or `userPool SharedArray`) with dedicated auth tokens per VU.
|
|
53
|
+
|
|
54
|
+
---
|
|
55
|
+
|
|
56
|
+
## 5. Workload Archetypes & Execution Schedule
|
|
57
|
+
|
|
58
|
+
| # | Archetype | Target Concurrency / Rate | Duration | Objective / Risk Addressed |
|
|
59
|
+
|---|:---|:---|:---|:---|
|
|
60
|
+
| **1** | **Smoke Test** | 1 – 5 VUs | 2 min | Pre-flight sanity check & endpoint reachability. |
|
|
61
|
+
| **2** | **Average Load Test** | 50 – 200 VUs | 15 – 30 min | Verify SLA compliance under expected peak volume. |
|
|
62
|
+
| **3** | **Stress Test (Step-Up)**| 200 $\rightarrow$ 2,000 VUs | 30 – 45 min | Locate degradation cliff, saturation knee & recovery. |
|
|
63
|
+
| **4** | **Spike Test** | 10 $\rightarrow$ 1,000 VUs | 5 – 10 min | Test autoscaling speed & circuit breaker recovery. |
|
|
64
|
+
| **5** | **Soak / Endurance** | 50 – 100 VUs | 2 – 12 hours | Uncover JVM memory leaks, GC pauses, connection leaks. |
|
|
65
|
+
| **6** | **Scalability Test** | 50 $\rightarrow$ 100 $\rightarrow$ 200 VUs | 30 – 60 min | Verify linear scale-up / scale-out elasticity ($\ge 80\%$). |
|
|
66
|
+
|
|
67
|
+
---
|
|
68
|
+
|
|
69
|
+
## 6. Test Environment Topology & SPEC/TPC Disclosure
|
|
70
|
+
|
|
71
|
+
- **Hardware & Host Specification:**
|
|
72
|
+
- Application Server: `[e.g., 4 vCPU, 16GB RAM, Linux Kernel 5.15]`
|
|
73
|
+
- Database Cluster: `[e.g., PostgreSQL 15, max_connections=200, pool_size=50]`
|
|
74
|
+
- Network Bandwidth: `[e.g., 10 Gbps VPC / local internal virtual bridge]`
|
|
75
|
+
- **Steady-State Measurement Interval (TPC Standard):**
|
|
76
|
+
- Warm-up Period: `[e.g., 3 minutes (excluded from SLA calculations)]`
|
|
77
|
+
- Steady-State Window: `[e.g., 10 minutes (official evaluation window)]`
|
|
78
|
+
- Cool-down Period: `[e.g., 2 minutes (excluded from SLA calculations)]`
|
|
79
|
+
|
|
80
|
+
---
|
|
81
|
+
|
|
82
|
+
## 7. Safety Gates, Pre-Flight Checks & Kill-Switch
|
|
83
|
+
|
|
84
|
+
- [ ] **Dedicated Isolated Environment:** Verified non-production target; staging approval granted.
|
|
85
|
+
- [ ] **Synthetic Test Data:** Parameterized with dynamic prefix `PERF_AUTO_${timestamp}_${thread}`.
|
|
86
|
+
- [ ] **Kill-Switch Threshold:** Configured `abortOnFail` if aggregate error rate $> 10\%$ over a 30s window.
|
|
87
|
+
- [ ] **Teardown & Cleanup Automation:** Reversible sandbox rollback (`reversible_sandbox.py`) scheduled post-test.
|
|
88
|
+
|
|
89
|
+
---
|
|
90
|
+
|
|
91
|
+
## 8. Telemetry, APM & Observability (OpenTelemetry)
|
|
92
|
+
|
|
93
|
+
- **Distributed Tracing Header:** Injected W3C Trace Context `traceparent` (`00-{trace_id}-{span_id}-01`).
|
|
94
|
+
- **Telemetry Streaming:** InfluxDB Backend Listener / Prometheus Node Exporter.
|
|
95
|
+
- **Root Cause Analysis (RCA):** Automated Pearson correlation matrix connecting tail latencies with CPU, Memory, and Database thread pools.
|
|
96
|
+
|
|
97
|
+
---
|
|
98
|
+
|
|
99
|
+
## 9. Definition of Done & Gate 10 Acceptance
|
|
100
|
+
|
|
101
|
+
- [ ] Test execution completed headlessly without manual browser GUI interference.
|
|
102
|
+
- [ ] Steady-state P95 latency is strictly below SLA budget.
|
|
103
|
+
- [ ] Error rate is strictly below 1.0% (0% on authentication & transaction routes).
|
|
104
|
+
- [ ] Zero database primary key collisions or lock contention from unparameterized test data.
|
|
105
|
+
- [ ] Artifacts generated and saved to `Reports/perf/`: `perf_results.jtl` and `k6_summary.json`.
|
|
106
|
+
- [ ] Gate 10 DoD status: **PASSED (`python tools/verify_all.py --mode hub`)**.
|
|
@@ -42,10 +42,11 @@ policies:
|
|
|
42
42
|
|
|
43
43
|
1. **Performance as an Assertion:**
|
|
44
44
|
- Functional tests must double as non-functional guardrails. If a critical workflow takes > 10 seconds to load, it should fail performance budgets.
|
|
45
|
-
2. **Google Core Web Vitals Metrics:**
|
|
46
|
-
- **LCP (Largest Contentful Paint):** Target $\le 2.5\text{s}$ (Good).
|
|
47
|
-
- **CLS (Cumulative Layout Shift):** Target $\le 0.1$ (Good).
|
|
48
|
-
- **
|
|
45
|
+
2. **Google Core Web Vitals & W3C Metrics:**
|
|
46
|
+
- **LCP (Largest Contentful Paint):** Target $\le 2.5\text{s}$ (Good), $> 4.0\text{s}$ (Poor).
|
|
47
|
+
- **CLS (Cumulative Layout Shift):** Target $\le 0.1$ (Good), $> 0.25$ (Poor).
|
|
48
|
+
- **INP (Interaction to Next Paint):** Target $\le 200\text{ms}$ (Good), $> 500\text{ms}$ (Poor) — Official W3C/Google replacement for legacy FID.
|
|
49
|
+
- **TTFB (Time to First Byte):** Target $\le 800\text{ms}$ (Good) via W3C Navigation Timing Level 2.
|
|
49
50
|
3. **Observation-Masked Telemetry:**
|
|
50
51
|
- Never dump thousands of CDP trace lines into context. Only output calculated metric scores and flagged slow endpoints (> 1500ms).
|
|
51
52
|
|
|
@@ -53,28 +54,88 @@ policies:
|
|
|
53
54
|
|
|
54
55
|
### Standard Playwright Performance Measurement
|
|
55
56
|
|
|
56
|
-
### Pattern A:
|
|
57
|
+
### Pattern A: W3C Navigation Timing Level 2 (Modern Performance Audit)
|
|
57
58
|
<!-- classification: illustrative -->
|
|
58
59
|
```typescript
|
|
59
60
|
import { test, expect } from '@playwright/test';
|
|
60
61
|
|
|
61
|
-
test('audit
|
|
62
|
+
test('audit navigation timing level 2 and ttfb on dashboard', async ({ page }) => {
|
|
62
63
|
await page.goto('/dashboard');
|
|
63
64
|
await page.waitForLoadState('networkidle');
|
|
64
65
|
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
66
|
+
// W3C Navigation Timing Level 2 Standard
|
|
67
|
+
const navTiming = await page.evaluate(() => {
|
|
68
|
+
const navEntries = performance.getEntriesByType('navigation') as PerformanceNavigationTiming[];
|
|
69
|
+
if (!navEntries || navEntries.length === 0) return null;
|
|
70
|
+
const nav = navEntries[0];
|
|
71
|
+
return {
|
|
72
|
+
ttfb: Math.round(nav.responseStart - nav.requestStart),
|
|
73
|
+
domContentLoaded: Math.round(nav.domContentLoadedEventEnd - nav.startTime),
|
|
74
|
+
loadComplete: Math.round(nav.loadEventEnd - nav.startTime),
|
|
75
|
+
transferSize: nav.transferSize,
|
|
76
|
+
};
|
|
77
|
+
});
|
|
78
|
+
|
|
79
|
+
expect(navTiming).not.toBeNull();
|
|
80
|
+
console.log(`[W3C NAV2] TTFB: ${navTiming?.ttfb}ms | DOM Loaded: ${navTiming?.domContentLoaded}ms`);
|
|
81
|
+
expect(navTiming!.ttfb).toBeLessThan(800); // W3C Good TTFB threshold
|
|
82
|
+
});
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
### Pattern B: Measuring Interaction to Next Paint (INP) via PerformanceObserver
|
|
86
|
+
<!-- classification: illustrative -->
|
|
87
|
+
```typescript
|
|
88
|
+
test('audit interaction to next paint (INP) on button click', async ({ page }) => {
|
|
89
|
+
await page.goto('/checkout');
|
|
90
|
+
await page.waitForLoadState('networkidle');
|
|
91
|
+
|
|
92
|
+
// Inject W3C PerformanceObserver tracking event durations
|
|
93
|
+
await page.evaluate(() => {
|
|
94
|
+
(window as any).__maxEventDuration = 0;
|
|
95
|
+
const observer = new PerformanceObserver((list) => {
|
|
96
|
+
for (const entry of list.getEntries()) {
|
|
97
|
+
const duration = (entry as any).duration;
|
|
98
|
+
if (duration > (window as any).__maxEventDuration) {
|
|
99
|
+
(window as any).__maxEventDuration = duration;
|
|
100
|
+
}
|
|
101
|
+
}
|
|
102
|
+
});
|
|
103
|
+
observer.observe({ type: 'event', buffered: true, durationThreshold: 16 } as any);
|
|
104
|
+
});
|
|
68
105
|
|
|
69
|
-
|
|
70
|
-
|
|
106
|
+
// Perform realistic user interaction
|
|
107
|
+
await page.locator('#submit-order-button').click();
|
|
108
|
+
await page.waitForTimeout(500); // Allow interaction cycle to complete
|
|
71
109
|
|
|
72
|
-
|
|
73
|
-
|
|
110
|
+
const maxInp = await page.evaluate(() => (window as any).__maxEventDuration || 0);
|
|
111
|
+
console.log(`[CORE WEB VITALS] Measured INP: ${maxInp}ms`);
|
|
112
|
+
expect(maxInp).toBeLessThan(200); // Google/W3C INP budget <= 200ms
|
|
113
|
+
});
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
### Pattern C: CDP Network Throttling & Fast 3G Simulation
|
|
117
|
+
<!-- classification: illustrative -->
|
|
118
|
+
```typescript
|
|
119
|
+
test('simulate fast 3g conditions and 4x cpu slowdown', async ({ page }) => {
|
|
120
|
+
const client = await page.context().newCDPSession(page);
|
|
121
|
+
|
|
122
|
+
// Emulate mobile network conditions (Fast 3G: 1.6 Mbps download, 750 kbps upload, 150ms latency)
|
|
123
|
+
await client.send('Network.emulateNetworkConditions', {
|
|
124
|
+
offline: false,
|
|
125
|
+
latency: 150,
|
|
126
|
+
downloadThroughput: (1.6 * 1024 * 1024) / 8,
|
|
127
|
+
uploadThroughput: (750 * 1024) / 8,
|
|
128
|
+
});
|
|
129
|
+
|
|
130
|
+
// Emulate 4x CPU slowdown (budget mobile CPU emulation)
|
|
131
|
+
await client.send('Emulation.setCPUThrottlingRate', { rate: 4 });
|
|
132
|
+
|
|
133
|
+
await page.goto('/catalog');
|
|
134
|
+
await page.waitForLoadState('networkidle');
|
|
74
135
|
});
|
|
75
136
|
```
|
|
76
137
|
|
|
77
|
-
### Pattern
|
|
138
|
+
### Pattern D: Tracking Slow Network Endpoints
|
|
78
139
|
<!-- classification: illustrative -->
|
|
79
140
|
```typescript
|
|
80
141
|
test('detect slow backend microservices', async ({ page }) => {
|
|
@@ -153,11 +153,22 @@ For systems with integrated LLMs, tool-calling agents, or MCP servers, inspect:
|
|
|
153
153
|
|
|
154
154
|
---
|
|
155
155
|
|
|
156
|
+
### OWASP ASVS 4.0.3, CVSS v3.1/v4.0 & Regulatory Compliance
|
|
157
|
+
- Execute automated Level 1 ASVS probes and checklist generation via `tools/asvs_auditor.py` (`agent-qa asvs --probe <url> --level 1`).
|
|
158
|
+
- Compute quantitative CVSS Base/Temporal vectors and qualitative severity ratings via `tools/cvss_calculator.py` (`agent-qa cvss "<vector>"`).
|
|
159
|
+
- Audit regulatory privacy and payment requirements via `tools/regulatory_compliance.py` (`agent-qa compliance`):
|
|
160
|
+
- **GDPR / PDPA:** Right to be Forgotten (Article 17) and Consent logs (Article 7).
|
|
161
|
+
- **PCI-DSS v4.0:** Requirement 3.4 (PAN Masking with Luhn validation) and Requirement 3.2 (CVV exclusion).
|
|
162
|
+
- **ISO 15489:** Electronic Records Management retention period verification.
|
|
163
|
+
|
|
164
|
+
---
|
|
165
|
+
|
|
156
166
|
## 4. Verification Points & DoD
|
|
157
167
|
|
|
158
168
|
1. Target endpoints verified against authorization whitelist and consent grant.
|
|
159
|
-
2. Security scan covers OWASP Top 10 vulnerabilities (SQLi, XSS, SSRF, broken auth, security headers).
|
|
160
|
-
3.
|
|
169
|
+
2. Security scan covers OWASP Top 10 vulnerabilities (SQLi, XSS, SSRF, broken auth, security headers) and ASVS Level 1 baseline.
|
|
170
|
+
3. Zero unmasked PAN leaks or stored CVVs detected across test logs and responses (`tools/regulatory_compliance.py --gate`).
|
|
171
|
+
4. Findings categorized by CVSS v3.1/v4.0 severity vector with reproduction steps and remediation guidance.
|
|
161
172
|
|
|
162
173
|
---
|
|
163
174
|
|
|
@@ -148,6 +148,9 @@ python tools/wcag_audit.py https://staging.example.com --url -o docs/wcag_audit_
|
|
|
148
148
|
|
|
149
149
|
# CLI integration via agent-qa
|
|
150
150
|
agent-qa wcag-audit path/to/page.html
|
|
151
|
+
|
|
152
|
+
# Export standardized Section 508 & EN 301 549 VPAT 2.5 Report
|
|
153
|
+
agent-qa vpat --url https://staging.example.com --out-md docs/VPAT_REPORT.md
|
|
151
154
|
```
|
|
152
155
|
|
|
153
156
|
---
|
|
@@ -156,7 +159,7 @@ agent-qa wcag-audit path/to/page.html
|
|
|
156
159
|
|
|
157
160
|
1. Automated accessibility scans run via axe-core / Playwright with 0 Level A/AA violations.
|
|
158
161
|
2. Manual check protocol verifies keyboard tab order, focus rings, and skip links.
|
|
159
|
-
3. Audit report maps violations to specific WCAG 2.2 criteria with remediation code.
|
|
162
|
+
3. Audit report maps violations to specific WCAG 2.2 criteria with remediation code and exports Section 508 / EN 301 549 VPAT 2.5 reports (`tools/vpat_exporter.py`).
|
|
160
163
|
|
|
161
164
|
---
|
|
162
165
|
|
|
@@ -124,11 +124,18 @@ def assert_pdf_contains_text(pdf_path: str, expected_keywords: list):
|
|
|
124
124
|
|
|
125
125
|
---
|
|
126
126
|
|
|
127
|
+
### ISO/IEC 25012 Data Quality & DAMA-DMBOK ETL Reconciliation
|
|
128
|
+
- Audit datasets against ISO 25012 characteristics (Completeness, Consistency, Accuracy, Currentness SLA) via `tools/etl_data_reconciler.py --quality-audit <file>`.
|
|
129
|
+
- Reconcile Source vs Target ETL migration files (row count, dropped key deltas, mutated values, aggregate sum balance) via `agent-qa etl --reconcile --source src.csv --target tgt.csv --key id --gate`.
|
|
130
|
+
|
|
131
|
+
---
|
|
132
|
+
|
|
127
133
|
## 4. Verification Points & DoD
|
|
128
134
|
|
|
129
135
|
1. Downloaded file format, MIME type, and byte size verified.
|
|
130
136
|
2. Deep content inspection validates specific text strings, cell coordinates, and row counts.
|
|
131
|
-
3.
|
|
137
|
+
3. ETL data migrations and report exports reconciled with zero dropped records (`tools/etl_data_reconciler.py --gate`).
|
|
138
|
+
4. Downloaded test files cleaned up from temporary sandbox upon test completion.
|
|
132
139
|
|
|
133
140
|
---
|
|
134
141
|
|
|
@@ -52,8 +52,11 @@ policies:
|
|
|
52
52
|
- Tax IDs, phone numbers, email RFC formats, and GIS coordinates (e.g., non-overlapping closed polygons).
|
|
53
53
|
4. **PII Safety Invariant:**
|
|
54
54
|
- Never copy or mirror real customer production data (PII) into test fixtures. All synthesized entities must use purely artificial data with identifiable prefixes (`AUTO_`, `PERF_AUTO_`, `AUTO_RACE_`) paired with automated teardown cleanup hooks.
|
|
55
|
-
5. **Vietnam Locale Standards:**
|
|
55
|
+
5. **Vietnam Locale & Unicode CLDR Standards:**
|
|
56
56
|
- Support national telecom prefixes (`03x`, `05x`, `07x`, `08x`, `09x`), 12-digit Citizen Identity Card (CCCD) format, full UTF-8 Vietnamese diacritics, and VND currency conventions (`100.000 ₫` / `100.000 VND`).
|
|
57
|
+
- Validate international formatting rules (dates, currencies, separators) across world locales (`en-US`, `vi-VN`, `de-DE`, `fr-FR`, `ja-JP`, `ar-SA`) using `tools/cldr_l10n_verifier.py` (`agent-qa cldr`).
|
|
58
|
+
6. **Pseudo-Localization for UI Resilience:**
|
|
59
|
+
- Generate accented, length-expanded test strings (`[!!! Çĥèçķòùţ ~one two~ !!!]`) using `tools/cldr_l10n_verifier.py --pseudo` to automatically expose hardcoded text, character encoding flaws, and UI clipping.
|
|
57
60
|
|
|
58
61
|
---
|
|
59
62
|
|
|
@@ -78,10 +78,18 @@ python tools/mipro_assertion_tuner.py compile-module \
|
|
|
78
78
|
# 2. Evaluate compiled prompt module against testset
|
|
79
79
|
python tools/mipro_assertion_tuner.py evaluate \
|
|
80
80
|
--module compiled/optimized_assertion_gen.json
|
|
81
|
+
|
|
82
|
+
# 3. Evaluate RAG & LLM Faithfulness & Hallucination via NIST AI RMF
|
|
83
|
+
agent-qa ai-eval --query "..." --context "..." --answer "..." --gate-faithfulness 0.8
|
|
81
84
|
```
|
|
82
85
|
|
|
83
86
|
---
|
|
84
87
|
|
|
88
|
+
### NIST AI RMF & ISO 42001 Evaluation Sentinel
|
|
89
|
+
- Evaluate RAG triplets (Query-Context-Answer) and LLM outputs for groundedness (anti-hallucination), relevance, and prompt injection safety via `tools/ai_eval_sentinel.py` (`agent-qa ai-eval`).
|
|
90
|
+
|
|
91
|
+
---
|
|
92
|
+
|
|
85
93
|
### Strict Operating Invariants
|
|
86
94
|
|
|
87
95
|
- **Mutation Score Hard Floor:** Prompts must achieve a minimum mutant kill rate of **85%**.
|