ai-testing-swarm 0.1.15__tar.gz → 0.1.17__tar.gz

{ai_testing_swarm-0.1.15 → ai_testing_swarm-0.1.17}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ai-testing-swarm
-Version: 0.1.15
+Version: 0.1.17
 Summary: AI-powered testing swarm
 Author-email: Arif Shah <ashah7775@gmail.com>
 License: MIT
@@ -10,6 +10,8 @@ Requires-Dist: requests>=2.28
 Requires-Dist: PyYAML>=6.0
 Provides-Extra: openapi
 Requires-Dist: jsonschema>=4.0; extra == "openapi"
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
 
 # AI Testing Swarm
 
@@ -70,7 +72,9 @@ A report is written under:
 - `./ai_swarm_reports/<METHOD>_<endpoint>/<METHOD>_<endpoint>_<timestamp>.<json|md|html>`
 
 Reports include:
-- per-test results
+- per-test results (including deterministic `risk_score` 0..100)
+- endpoint-level risk gate (`PASS`/`WARN`/`BLOCK`)
+- trend vs previous run for the same endpoint (risk delta + regressions)
 - summary counts by status code / failure type
 - optional AI summary (if enabled)
 
@@ -115,6 +119,10 @@ Reports include:
 - Base URL is read from `spec.servers[0].url`.
 - When using OpenAPI input, the swarm will also *optionally* validate response status codes against `operation.responses`.
 - If `jsonschema` is installed (via `ai-testing-swarm[openapi]`) and the response is JSON, response bodies are validated against the OpenAPI `application/json` schema.
+- If an OpenAPI requestBody schema exists for the operation, the planner can also generate **schema-based request-body fuzz cases**:
+  - schema-valid edge cases (boundary values that should still validate)
+  - schema-invalid variants (missing required fields, wrong types, out-of-range, etc.)
+  - Controlled via CLI flags like `--no-openapi-fuzz` / `--openapi-fuzz-max-*`.
   - Override with `AI_SWARM_OPENAPI_BASE_URL` if your spec doesn’t include servers.
 
 ---
@@ -140,6 +148,28 @@ Then generates broad coverage across:
 
 ---
 
+## Auth matrix runner (multiple tokens/headers)
+
+To run the *same* request under multiple auth contexts (e.g., user/admin tokens), create `auth_matrix.yaml`:
+
+```yaml
+cases:
+  - name: user
+    headers:
+      Authorization: "Bearer USER_TOKEN"
+  - name: admin
+    headers:
+      Authorization: "Bearer ADMIN_TOKEN"
+```
+
+Run:
+
+```bash
+ai-test --input request.json --auth-matrix auth_matrix.yaml
+```
+
+Each auth case is written as a separate report using a `run_label` suffix (e.g. `__auth-user`).
+
 ## Safety mode (recommended for CI/demos)
 
 Mutation testing can be noisy and may accidentally stress a real environment.
@@ -191,6 +221,21 @@ Reports include:
 - `summary.counts_by_failure_type`
 - `summary.counts_by_status_code`
 - `summary.slow_tests` (based on SLA)
+- `meta.endpoint_risk_score` + `meta.gate_status`
+- `trend.*` (previous comparison if a prior report exists)
+
+A static dashboard index is generated at:
+- `./ai_swarm_reports/index.html` (latest JSON report per endpoint, sorted by regressions/risk)
+
+### Regression gate (CI)
+
+To fail CI when the run regresses vs the previous JSON report for the same endpoint:
+
+```bash
+ai-test --input request.json --fail-on-regression
+```
+
+This checks `report.trend.regression_count` and exits with a non-zero code if regressions are detected.
 
 SLA threshold:
 - `AI_SWARM_SLA_MS` (default: `2000`)

{ai_testing_swarm-0.1.15 → ai_testing_swarm-0.1.17}/README.md

@@ -57,7 +57,9 @@ A report is written under:
 - `./ai_swarm_reports/<METHOD>_<endpoint>/<METHOD>_<endpoint>_<timestamp>.<json|md|html>`
 
 Reports include:
-- per-test results
+- per-test results (including deterministic `risk_score` 0..100)
+- endpoint-level risk gate (`PASS`/`WARN`/`BLOCK`)
+- trend vs previous run for the same endpoint (risk delta + regressions)
 - summary counts by status code / failure type
 - optional AI summary (if enabled)
 
@@ -102,6 +104,10 @@ Reports include:
 - Base URL is read from `spec.servers[0].url`.
 - When using OpenAPI input, the swarm will also *optionally* validate response status codes against `operation.responses`.
 - If `jsonschema` is installed (via `ai-testing-swarm[openapi]`) and the response is JSON, response bodies are validated against the OpenAPI `application/json` schema.
+- If an OpenAPI requestBody schema exists for the operation, the planner can also generate **schema-based request-body fuzz cases**:
+  - schema-valid edge cases (boundary values that should still validate)
+  - schema-invalid variants (missing required fields, wrong types, out-of-range, etc.)
+  - Controlled via CLI flags like `--no-openapi-fuzz` / `--openapi-fuzz-max-*`.
   - Override with `AI_SWARM_OPENAPI_BASE_URL` if your spec doesn’t include servers.
 
 ---
@@ -127,6 +133,28 @@ Then generates broad coverage across:
 
 ---
 
+## Auth matrix runner (multiple tokens/headers)
+
+To run the *same* request under multiple auth contexts (e.g., user/admin tokens), create `auth_matrix.yaml`:
+
+```yaml
+cases:
+  - name: user
+    headers:
+      Authorization: "Bearer USER_TOKEN"
+  - name: admin
+    headers:
+      Authorization: "Bearer ADMIN_TOKEN"
+```
+
+Run:
+
+```bash
+ai-test --input request.json --auth-matrix auth_matrix.yaml
+```
+
+Each auth case is written as a separate report using a `run_label` suffix (e.g. `__auth-user`).
+
 ## Safety mode (recommended for CI/demos)
 
 Mutation testing can be noisy and may accidentally stress a real environment.
@@ -178,6 +206,21 @@ Reports include:
 - `summary.counts_by_failure_type`
 - `summary.counts_by_status_code`
 - `summary.slow_tests` (based on SLA)
+- `meta.endpoint_risk_score` + `meta.gate_status`
+- `trend.*` (previous comparison if a prior report exists)
+
+A static dashboard index is generated at:
+- `./ai_swarm_reports/index.html` (latest JSON report per endpoint, sorted by regressions/risk)
+
+### Regression gate (CI)
+
+To fail CI when the run regresses vs the previous JSON report for the same endpoint:
+
+```bash
+ai-test --input request.json --fail-on-regression
+```
+
+This checks `report.trend.regression_count` and exits with a non-zero code if regressions are detected.
 
 SLA threshold:
 - `AI_SWARM_SLA_MS` (default: `2000`)

{ai_testing_swarm-0.1.15 → ai_testing_swarm-0.1.17}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "ai-testing-swarm"
-version = "0.1.15"
+version = "0.1.17"
 description = "AI-powered testing swarm"
 readme = "README.md"
 requires-python = ">=3.9"
@@ -22,6 +22,9 @@ dependencies = [
 openapi = [
   "jsonschema>=4.0",
 ]
+dev = [
+  "pytest>=8.0",
+]
 
 [project.scripts]
 ai-test = "ai_testing_swarm.cli:main"

ai_testing_swarm-0.1.17/src/ai_testing_swarm/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.1.17"

{ai_testing_swarm-0.1.15 → ai_testing_swarm-0.1.17}/src/ai_testing_swarm/agents/test_planner_agent.py

@@ -363,7 +363,58 @@ class TestPlannerAgent:
             for key, value in body.items():
                 add_field_mutations("body", key, value)
 
-        # 6) Optional OpenAI augmentation (best-effort)
+        # 6) Optional OpenAPI schema-based request-body fuzzing (best-effort)
+        try:
+            from ai_testing_swarm.core.config import (
+                AI_SWARM_OPENAPI_FUZZ,
+                AI_SWARM_OPENAPI_FUZZ_MAX_DEPTH,
+                AI_SWARM_OPENAPI_FUZZ_MAX_INVALID,
+                AI_SWARM_OPENAPI_FUZZ_MAX_VALID,
+            )
+
+            if AI_SWARM_OPENAPI_FUZZ and isinstance(request.get("_openapi"), dict):
+                ctx = request.get("_openapi") or {}
+                spec = ctx.get("spec")
+                path0 = ctx.get("path")
+                method0 = ctx.get("method")
+
+                if isinstance(spec, dict) and isinstance(path0, str) and isinstance(method0, str):
+                    from ai_testing_swarm.core.openapi_fuzzer import generate_request_body_fuzz_cases
+
+                    fuzz_cases = generate_request_body_fuzz_cases(
+                        spec=spec,
+                        path=path0,
+                        method=method0,
+                        max_valid=AI_SWARM_OPENAPI_FUZZ_MAX_VALID,
+                        max_invalid=AI_SWARM_OPENAPI_FUZZ_MAX_INVALID,
+                        max_depth=AI_SWARM_OPENAPI_FUZZ_MAX_DEPTH,
+                    )
+
+                    existing = {t["name"] for t in tests}
+                    for fc in fuzz_cases:
+                        name = fc.name
+                        if name in existing:
+                            continue
+                        tests.append(
+                            {
+                                "name": name,
+                                "mutation": {
+                                    "target": "body_whole",
+                                    "path": [],
+                                    "operation": "REPLACE",
+                                    "original_value": body,
+                                    "new_value": fc.body,
+                                    "strategy": "openapi_fuzz_valid" if fc.is_valid else "openapi_fuzz_invalid",
+                                    "openapi": {"is_valid": fc.is_valid, "details": fc.details},
+                                },
+                            }
+                        )
+                        existing.add(name)
+        except Exception:
+            # Never fail planning due to fuzzing issues
+            pass
+
+        # 7) Optional OpenAI augmentation (best-effort)
         try:
             from ai_testing_swarm.core.config import (
                 AI_SWARM_MAX_AI_TESTS,

{ai_testing_swarm-0.1.15 → ai_testing_swarm-0.1.17}/src/ai_testing_swarm/cli.py

@@ -107,6 +107,54 @@ def main():
         help="Report format to write (default: json)",
     )
 
+    # OpenAPI schema-based request-body fuzzing
+    parser.add_argument(
+        "--openapi-fuzz",
+        dest="openapi_fuzz",
+        action="store_true",
+        default=True,
+        help="Enable OpenAPI schema-based request-body fuzzing when OpenAPI context is present (default: on)",
+    )
+    parser.add_argument(
+        "--no-openapi-fuzz",
+        dest="openapi_fuzz",
+        action="store_false",
+        help="Disable OpenAPI schema-based request-body fuzzing",
+    )
+    parser.add_argument(
+        "--openapi-fuzz-max-valid",
+        type=int,
+        default=8,
+        help="Max schema-valid OpenAPI fuzz cases to add (default: 8)",
+    )
+    parser.add_argument(
+        "--openapi-fuzz-max-invalid",
+        type=int,
+        default=8,
+        help="Max schema-invalid OpenAPI fuzz cases to add (default: 8)",
+    )
+    parser.add_argument(
+        "--openapi-fuzz-max-depth",
+        type=int,
+        default=6,
+        help="Max recursion depth for OpenAPI schema instance generation (default: 6)",
+    )
+
+    parser.add_argument(
+        "--fail-on-regression",
+        action="store_true",
+        help="Exit non-zero if the run regresses vs the previous JSON report for this endpoint",
+    )
+
+    parser.add_argument(
+        "--auth-matrix",
+        default="",
+        help=(
+            "Optional path to auth_matrix.yaml/json to run the same endpoint under multiple auth headers. "
+            "Each case is reported separately via a run label suffix."
+        ),
+    )
+
     # Batch1: risk gate thresholds (backward compatible defaults)
     parser.add_argument(
         "--gate-warn",
@@ -137,33 +185,89 @@ def main():
     # ------------------------------------------------------------
     # Run swarm
     # ------------------------------------------------------------
+    import os
+
     if args.public_only:
-        import os
         os.environ["AI_SWARM_PUBLIC_ONLY"] = "1"
 
-    decision, results = SwarmOrchestrator().run(
-        request,
-        report_format=args.report_format,
-        gate_warn=args.gate_warn,
-        gate_block=args.gate_block,
-    )
+    # CLI flags -> env toggles used by core.config
+    os.environ["AI_SWARM_OPENAPI_FUZZ"] = "1" if args.openapi_fuzz else "0"
+    os.environ["AI_SWARM_OPENAPI_FUZZ_MAX_VALID"] = str(int(args.openapi_fuzz_max_valid))
+    os.environ["AI_SWARM_OPENAPI_FUZZ_MAX_INVALID"] = str(int(args.openapi_fuzz_max_invalid))
+    os.environ["AI_SWARM_OPENAPI_FUZZ_MAX_DEPTH"] = str(int(args.openapi_fuzz_max_depth))
 
-    # ------------------------------------------------------------
-    # Console output
-    # ------------------------------------------------------------
-    print("\n=== RELEASE DECISION ===")
-    print(decision)
-
-    print("\n=== TEST RESULTS ===")
-    for r in results:
-        response = r.get("response", {})
-        status_code = response.get("status_code")
-
-        print(
-            f"{r.get('name'):25} "
-            f"{str(status_code):5} "
-            f"{r.get('reason')}"
+    orch = SwarmOrchestrator()
+
+    def _print_console(decision, results, *, label: str = ""):
+        if label:
+            print(f"\n=== AUTH CASE: {label} ===")
+        print("\n=== RELEASE DECISION ===")
+        print(decision)
+        print("\n=== TEST RESULTS ===")
+        for r in results:
+            response = r.get("response", {})
+            status_code = response.get("status_code")
+            print(f"{r.get('name'):25} {str(status_code):5} {r.get('reason')}")
+
+    def _maybe_fail_on_regression(report_path: str):
+        try:
+            with open(report_path, "r", encoding="utf-8") as f:
+                report = json.load(f)
+        except Exception:
+            return
+
+        trend = report.get("trend") or {}
+        regression_count = trend.get("regression_count")
+        try:
+            reg_i = int(regression_count or 0)
+        except Exception:
+            reg_i = 0
+
+        if reg_i > 0:
+            print(f"\n❌ Regression gate failed: regressions={reg_i} (see trend in report)")
+            raise SystemExit(2)
+
+    if args.auth_matrix:
+        from ai_testing_swarm.core.auth_matrix import load_auth_matrix, merge_auth_headers
+
+        cases = load_auth_matrix(args.auth_matrix)
+        for c in cases:
+            req2 = merge_auth_headers(request, c)
+            report_format = args.report_format
+            if args.fail_on_regression and report_format != "json":
+                # regression gate needs a machine-readable report
+                report_format = "json"
+
+            decision, results, report_path = orch.run(
+                req2,
+                report_format=report_format,
+                gate_warn=args.gate_warn,
+                gate_block=args.gate_block,
+                run_label=f"auth-{c.name}",
+                fail_on_regression=args.fail_on_regression,
+                return_report_path=True,
+            )
+            _print_console(decision, results, label=c.name)
+
+            if args.fail_on_regression:
+                _maybe_fail_on_regression(report_path)
+    else:
+        report_format = args.report_format
+        if args.fail_on_regression and report_format != "json":
+            report_format = "json"
+
+        decision, results, report_path = orch.run(
+            request,
+            report_format=report_format,
+            gate_warn=args.gate_warn,
+            gate_block=args.gate_block,
+            fail_on_regression=args.fail_on_regression,
+            return_report_path=True,
         )
+        _print_console(decision, results)
+
+        if args.fail_on_regression:
+            _maybe_fail_on_regression(report_path)
 
 
 if __name__ == "__main__":

ai_testing_swarm-0.1.17/src/ai_testing_swarm/core/auth_matrix.py

@@ -0,0 +1,93 @@
+from __future__ import annotations
+
+import json
+from dataclasses import dataclass
+from pathlib import Path
+
+import yaml
+
+
+@dataclass(frozen=True)
+class AuthCase:
+    name: str
+    headers: dict[str, str]
+
+
+def _sanitize_case_name(name: str) -> str:
+    name = str(name or "").strip()
+    if not name:
+        return "case"
+    # Keep it filesystem-friendly.
+    out = []
+    for ch in name:
+        if ch.isalnum() or ch in ("-", "_", "."):
+            out.append(ch)
+        else:
+            out.append("-")
+    return "".join(out).strip("-") or "case"
+
+
+def load_auth_matrix(path: str | Path) -> list[AuthCase]:
+    """Load an auth matrix config (yaml/json).
+
+    Schema:
+      {
+        "cases": [
+          {"name": "user", "headers": {"Authorization": "Bearer ..."}},
+          {"name": "admin", "headers": {"Authorization": "Bearer ..."}}
+        ]
+      }
+
+    Notes:
+      - This is intentionally minimal and explicit.
+      - Headers are merged into the base request headers (case wins).
+    """
+
+    p = Path(path)
+    raw = p.read_text(encoding="utf-8")
+    if p.suffix.lower() in {".yaml", ".yml"}:
+        data = yaml.safe_load(raw) or {}
+    else:
+        data = json.loads(raw)
+
+    cases = data.get("cases") if isinstance(data, dict) else None
+    if not isinstance(cases, list) or not cases:
+        raise ValueError("auth matrix must contain a non-empty 'cases' list")
+
+    out: list[AuthCase] = []
+    for i, c in enumerate(cases):
+        if not isinstance(c, dict):
+            raise ValueError(f"auth case #{i} must be an object")
+        name = _sanitize_case_name(c.get("name") or f"case{i+1}")
+        headers = c.get("headers") or {}
+        if not isinstance(headers, dict):
+            raise ValueError(f"auth case '{name}' headers must be an object")
+        # stringify values (avoid accidental ints)
+        headers2 = {str(k): str(v) for k, v in headers.items() if v is not None}
+        out.append(AuthCase(name=name, headers=headers2))
+
+    # Ensure unique names
+    seen: set[str] = set()
+    uniq: list[AuthCase] = []
+    for c in out:
+        nm = c.name
+        if nm not in seen:
+            uniq.append(c)
+            seen.add(nm)
+        else:
+            j = 2
+            while f"{nm}-{j}" in seen:
+                j += 1
+            new = f"{nm}-{j}"
+            uniq.append(AuthCase(name=new, headers=c.headers))
+            seen.add(new)
+
+    return uniq
+
+
+def merge_auth_headers(request: dict, auth_case: AuthCase) -> dict:
+    req = dict(request)
+    base_headers = dict(req.get("headers") or {})
+    base_headers.update(auth_case.headers or {})
+    req["headers"] = base_headers
+    return req

{ai_testing_swarm-0.1.15 → ai_testing_swarm-0.1.17}/src/ai_testing_swarm/core/config.py

@@ -20,3 +20,9 @@ AI_SWARM_USE_OPENAI = os.getenv("AI_SWARM_USE_OPENAI", "0").lower() in {"1", "tr
 AI_SWARM_OPENAI_MODEL = os.getenv("AI_SWARM_OPENAI_MODEL", "gpt-4.1-mini")
 AI_SWARM_MAX_AI_TESTS = env_int("AI_SWARM_MAX_AI_TESTS", 20)
 AI_SWARM_AI_SUMMARY = os.getenv("AI_SWARM_AI_SUMMARY", "0").lower() in {"1", "true", "yes"}
+
+# OpenAPI schema-based request-body fuzzing (on by default when OpenAPI context exists)
+AI_SWARM_OPENAPI_FUZZ = os.getenv("AI_SWARM_OPENAPI_FUZZ", "1").lower() in {"1", "true", "yes"}
+AI_SWARM_OPENAPI_FUZZ_MAX_VALID = env_int("AI_SWARM_OPENAPI_FUZZ_MAX_VALID", 8)
+AI_SWARM_OPENAPI_FUZZ_MAX_INVALID = env_int("AI_SWARM_OPENAPI_FUZZ_MAX_INVALID", 8)
+AI_SWARM_OPENAPI_FUZZ_MAX_DEPTH = env_int("AI_SWARM_OPENAPI_FUZZ_MAX_DEPTH", 6)