ai-testing-swarm 0.1.1__tar.gz → 0.1.11__tar.gz

ai_testing_swarm-0.1.11/PKG-INFO

@@ -0,0 +1,231 @@
+Metadata-Version: 2.4
+Name: ai-testing-swarm
+Version: 0.1.11
+Summary: AI-powered testing swarm
+Author-email: Arif Shah <ashah7775@gmail.com>
+License: MIT
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+
+# AI Testing Swarm
+
+AI Testing Swarm is a **super-advanced, mutation-driven API testing framework** (with optional OpenAPI + OpenAI augmentation) built on top of **pytest**.
+
+It generates a large set of deterministic negative/edge/security test cases for an API request, executes them (optionally in parallel, with retries/throttling), and produces a JSON report with summaries.
+
+> Notes:
+> - UI testing is not the focus of the current releases.
+> - OpenAI features are **optional** and disabled by default.
+
+---
+
+## Installation
+
+```bash
+pip install ai-testing-swarm
+```
+
+CLI entrypoint:
+
+```bash
+ai-test --help
+```
+
+---
+
+## Quick start (cURL input)
+
+Create `request.json`:
+
+```json
+{
+  "curl": "curl --location https://postman-echo.com/post --header \"Content-Type: application/json\" --data \"{\\\"hello\\\":\\\"world\\\",\\\"count\\\":1}\""
+}
+```
+
+Run:
+
+```bash
+ai-test --input request.json
+```
+
+A JSON report is written under:
+
+- `./ai_swarm_reports/<METHOD>_<endpoint>/<METHOD>_<endpoint>_<timestamp>.json`
+
+Reports include:
+- per-test results
+- summary counts by status code / failure type
+- optional AI summary (if enabled)
+
+---
+
+## Input formats
+
+### 1) Raw cURL
+
+```json
+{ "curl": "curl ..." }
+```
+
+### 2) Normalized request
+
+```json
+{
+  "method": "POST",
+  "url": "https://example.com/api/login",
+  "headers": {"content-type": "application/json"},
+  "params": {"a": "b"},
+  "body": {"username": "u", "password": "p"}
+}
+```
+
+### 3) OpenAPI-driven (optional)
+
+```json
+{
+  "openapi": "./openapi.json",
+  "path": "/pets",
+  "method": "get",
+  "headers": {"accept": "application/json"},
+  "path_params": {"petId": "123"},
+  "query_params": {"limit": 10},
+  "body": null
+}
+```
+
+- OpenAPI **JSON** works by default.
+- OpenAPI **YAML** requires `PyYAML` installed.
+- Base URL is read from `spec.servers[0].url`.
+  - Override with `AI_SWARM_OPENAPI_BASE_URL` if your spec doesn’t include servers.
+
+---
+
+## What test cases are generated?
+
+The swarm always includes:
+- **happy_path** (baseline)
+
+Then generates broad coverage across:
+- **Method misuse**: same path with wrong HTTP methods (GET/PUT/PATCH/DELETE etc.)
+- **Headers**: missing/invalid `Content-Type`, accept variations, and other header tampering
+- **Auth** (if `Authorization` header exists): missing/invalid token tests
+- **Body/query mutations** (per field):
+  - missing / null / empty / whitespace
+  - type probes (int/bool/float/array/object)
+  - boundary inputs (very long strings, huge ints, negative values)
+  - unicode + special character payloads
+- **Security payload probes** (per field): SQLi/XSS/path traversal/log4j patterns
+- **Whole-body mutations**: null body, empty object, extra unexpected field
+
+> Output is deterministic unless OpenAI augmentation is enabled.
+
+---
+
+## Safety mode (recommended for CI/demos)
+
+Mutation testing can be noisy and may accidentally stress a real environment.
+To force safe demo runs only against public test hosts:
+
+```bash
+ai-test --input request.json --public-only
+```
+
+Or via env:
+
+```bash
+export AI_SWARM_PUBLIC_ONLY=1
+```
+
+Allowed hosts in public-only mode:
+- `httpbin.org`
+- `postman-echo.com`
+- `reqres.in`
+
+---
+
+## Performance features
+
+### Parallel execution
+- Enabled by default via thread pool.
+- Control with:
+  - `AI_SWARM_WORKERS` (default: `5`)
+
+### Retry + backoff (flaky endpoints)
+- Retries on transient errors and status codes (408/429/5xx etc.)
+- Control with:
+  - `AI_SWARM_RETRY_COUNT` (default: `1`)
+  - `AI_SWARM_RETRY_BACKOFF_MS` (default: `250`)
+
+### Throttling (RPS)
+- Global throttle to avoid hammering a target:
+  - `AI_SWARM_RPS` (default: `0` = disabled)
+
+### Max test cap
+- Avoids accidental DoS / CI timeouts:
+  - `AI_SWARM_MAX_TESTS` (default: `80`)
+
+---
+
+## Reporting
+
+Reports include:
+- `summary.counts_by_failure_type`
+- `summary.counts_by_status_code`
+- `summary.slow_tests` (based on SLA)
+
+SLA threshold:
+- `AI_SWARM_SLA_MS` (default: `2000`)
+
+Security:
+- Sensitive headers are redacted in the report (Authorization/Cookie/api tokens etc.)
+
+---
+
+## Optional OpenAI augmentation (advanced)
+
+### A) Generate additional test cases (planner augmentation)
+Enable:
+
+```bash
+export AI_SWARM_USE_OPENAI=1
+export OPENAI_API_KEY=... 
+export AI_SWARM_MAX_AI_TESTS=30
+```
+
+### B) Human-readable AI summary in report
+Enable:
+
+```bash
+export AI_SWARM_USE_OPENAI=1
+export AI_SWARM_AI_SUMMARY=1
+export OPENAI_API_KEY=...
+```
+
+Model selection:
+- `AI_SWARM_OPENAI_MODEL` (default: `gpt-4.1-mini`)
+
+---
+
+## CLI help
+
+```bash
+ai-test --help
+```
+
+---
+
+## Release decisions
+
+The swarm produces a release decision:
+- `APPROVE_RELEASE`
+- `APPROVE_RELEASE_WITH_RISKS`
+- `REJECT_RELEASE`
+
+The decision is derived from deterministic rules (not an LLM).
+
+---
+
+## License
+
+MIT

ai_testing_swarm-0.1.11/README.md

@@ -0,0 +1,222 @@
+# AI Testing Swarm
+
+AI Testing Swarm is a **super-advanced, mutation-driven API testing framework** (with optional OpenAPI + OpenAI augmentation) built on top of **pytest**.
+
+It generates a large set of deterministic negative/edge/security test cases for an API request, executes them (optionally in parallel, with retries/throttling), and produces a JSON report with summaries.
+
+> Notes:
+> - UI testing is not the focus of the current releases.
+> - OpenAI features are **optional** and disabled by default.
+
+---
+
+## Installation
+
+```bash
+pip install ai-testing-swarm
+```
+
+CLI entrypoint:
+
+```bash
+ai-test --help
+```
+
+---
+
+## Quick start (cURL input)
+
+Create `request.json`:
+
+```json
+{
+  "curl": "curl --location https://postman-echo.com/post --header \"Content-Type: application/json\" --data \"{\\\"hello\\\":\\\"world\\\",\\\"count\\\":1}\""
+}
+```
+
+Run:
+
+```bash
+ai-test --input request.json
+```
+
+A JSON report is written under:
+
+- `./ai_swarm_reports/<METHOD>_<endpoint>/<METHOD>_<endpoint>_<timestamp>.json`
+
+Reports include:
+- per-test results
+- summary counts by status code / failure type
+- optional AI summary (if enabled)
+
+---
+
+## Input formats
+
+### 1) Raw cURL
+
+```json
+{ "curl": "curl ..." }
+```
+
+### 2) Normalized request
+
+```json
+{
+  "method": "POST",
+  "url": "https://example.com/api/login",
+  "headers": {"content-type": "application/json"},
+  "params": {"a": "b"},
+  "body": {"username": "u", "password": "p"}
+}
+```
+
+### 3) OpenAPI-driven (optional)
+
+```json
+{
+  "openapi": "./openapi.json",
+  "path": "/pets",
+  "method": "get",
+  "headers": {"accept": "application/json"},
+  "path_params": {"petId": "123"},
+  "query_params": {"limit": 10},
+  "body": null
+}
+```
+
+- OpenAPI **JSON** works by default.
+- OpenAPI **YAML** requires `PyYAML` installed.
+- Base URL is read from `spec.servers[0].url`.
+  - Override with `AI_SWARM_OPENAPI_BASE_URL` if your spec doesn’t include servers.
+
+---
+
+## What test cases are generated?
+
+The swarm always includes:
+- **happy_path** (baseline)
+
+Then generates broad coverage across:
+- **Method misuse**: same path with wrong HTTP methods (GET/PUT/PATCH/DELETE etc.)
+- **Headers**: missing/invalid `Content-Type`, accept variations, and other header tampering
+- **Auth** (if `Authorization` header exists): missing/invalid token tests
+- **Body/query mutations** (per field):
+  - missing / null / empty / whitespace
+  - type probes (int/bool/float/array/object)
+  - boundary inputs (very long strings, huge ints, negative values)
+  - unicode + special character payloads
+- **Security payload probes** (per field): SQLi/XSS/path traversal/log4j patterns
+- **Whole-body mutations**: null body, empty object, extra unexpected field
+
+> Output is deterministic unless OpenAI augmentation is enabled.
+
+---
+
+## Safety mode (recommended for CI/demos)
+
+Mutation testing can be noisy and may accidentally stress a real environment.
+To force safe demo runs only against public test hosts:
+
+```bash
+ai-test --input request.json --public-only
+```
+
+Or via env:
+
+```bash
+export AI_SWARM_PUBLIC_ONLY=1
+```
+
+Allowed hosts in public-only mode:
+- `httpbin.org`
+- `postman-echo.com`
+- `reqres.in`
+
+---
+
+## Performance features
+
+### Parallel execution
+- Enabled by default via thread pool.
+- Control with:
+  - `AI_SWARM_WORKERS` (default: `5`)
+
+### Retry + backoff (flaky endpoints)
+- Retries on transient errors and status codes (408/429/5xx etc.)
+- Control with:
+  - `AI_SWARM_RETRY_COUNT` (default: `1`)
+  - `AI_SWARM_RETRY_BACKOFF_MS` (default: `250`)
+
+### Throttling (RPS)
+- Global throttle to avoid hammering a target:
+  - `AI_SWARM_RPS` (default: `0` = disabled)
+
+### Max test cap
+- Avoids accidental DoS / CI timeouts:
+  - `AI_SWARM_MAX_TESTS` (default: `80`)
+
+---
+
+## Reporting
+
+Reports include:
+- `summary.counts_by_failure_type`
+- `summary.counts_by_status_code`
+- `summary.slow_tests` (based on SLA)
+
+SLA threshold:
+- `AI_SWARM_SLA_MS` (default: `2000`)
+
+Security:
+- Sensitive headers are redacted in the report (Authorization/Cookie/api tokens etc.)
+
+---
+
+## Optional OpenAI augmentation (advanced)
+
+### A) Generate additional test cases (planner augmentation)
+Enable:
+
+```bash
+export AI_SWARM_USE_OPENAI=1
+export OPENAI_API_KEY=... 
+export AI_SWARM_MAX_AI_TESTS=30
+```
+
+### B) Human-readable AI summary in report
+Enable:
+
+```bash
+export AI_SWARM_USE_OPENAI=1
+export AI_SWARM_AI_SUMMARY=1
+export OPENAI_API_KEY=...
+```
+
+Model selection:
+- `AI_SWARM_OPENAI_MODEL` (default: `gpt-4.1-mini`)
+
+---
+
+## CLI help
+
+```bash
+ai-test --help
+```
+
+---
+
+## Release decisions
+
+The swarm produces a release decision:
+- `APPROVE_RELEASE`
+- `APPROVE_RELEASE_WITH_RISKS`
+- `REJECT_RELEASE`
+
+The decision is derived from deterministic rules (not an LLM).
+
+---
+
+## License
+
+MIT

{ai_testing_swarm-0.1.1 → ai_testing_swarm-0.1.11}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "ai-testing-swarm"
-version = "0.1.1"
+version = "0.1.11"
 description = "AI-powered testing swarm"
 readme = "README.md"
 requires-python = ">=3.9"

ai_testing_swarm-0.1.11/src/ai_testing_swarm/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.1.11"

ai_testing_swarm-0.1.11/src/ai_testing_swarm/agents/execution_agent.py

@@ -0,0 +1,186 @@
+import random
+import time
+from copy import deepcopy
+
+import requests
+
+from ai_testing_swarm.core.config import (
+    AI_SWARM_RETRY_BACKOFF_MS,
+    AI_SWARM_RETRY_COUNT,
+    AI_SWARM_RPS,
+)
+
+
+class ExecutionAgent:
+    def _set_path(self, obj: dict, path: list[str], value, remove: bool = False):
+        """Set/remove nested key in a dict. Creates intermediate dicts if needed."""
+        if not path:
+            return
+        cur = obj
+        for key in path[:-1]:
+            if key not in cur or not isinstance(cur[key], dict):
+                cur[key] = {}
+            cur = cur[key]
+
+        last = path[-1]
+        if remove:
+            cur.pop(last, None)
+        else:
+            cur[last] = value
+
+    def apply_mutation(self, data: dict, mutation: dict):
+        if not mutation:
+            return deepcopy(data)
+
+        data = deepcopy(data)
+        path = mutation.get("path") or []
+        op = mutation.get("operation")
+
+        if op == "REMOVE":
+            self._set_path(data, path, None, remove=True)
+        elif op == "REPLACE":
+            self._set_path(data, path, mutation.get("new_value"), remove=False)
+
+        return data
+
+    def _throttle(self):
+        # Optional global throttling to avoid hammering target services.
+        if AI_SWARM_RPS and AI_SWARM_RPS > 0:
+            time.sleep(1.0 / float(AI_SWARM_RPS))
+
+    def _should_retry(self, status_code: int | None, error: str | None) -> bool:
+        if error:
+            return True
+        if status_code in (408, 425, 429, 500, 502, 503, 504):
+            return True
+        return False
+
+    def execute(self, request: dict, test: dict):
+        mutation = test.get("mutation")
+
+        # Method (allow method mutations)
+        method = request["method"].upper()
+        if mutation and mutation.get("target") == "method" and mutation.get("operation") == "REPLACE":
+            method = str(mutation.get("new_value") or method).upper()
+
+        params = (
+            self.apply_mutation(request.get("params", {}) or {}, mutation)
+            if mutation and mutation.get("target") == "query"
+            else deepcopy(request.get("params", {}) or {})
+        )
+
+        body = deepcopy(request.get("body", {}) or {})
+        if mutation and mutation.get("target") == "body":
+            body = self.apply_mutation(body, mutation)
+        elif mutation and mutation.get("target") == "body_whole" and mutation.get("operation") == "REPLACE":
+            body = mutation.get("new_value")
+
+        headers = deepcopy(request.get("headers", {}) or {})
+        if mutation and mutation.get("target") == "headers":
+            headers = self.apply_mutation(headers, mutation)
+
+        # Payload strategy (basic content-type aware)
+        send_json = method != "GET"
+        ct = (headers.get("content-type") or headers.get("Content-Type") or "").lower()
+
+        json_payload = None
+        data_payload = None
+        if method == "GET":
+            json_payload = None
+            data_payload = None
+        else:
+            # If body is not a dict, send raw data
+            if not isinstance(body, (dict, list)):
+                send_json = False
+                data_payload = body
+            elif "application/x-www-form-urlencoded" in ct:
+                send_json = False
+                data_payload = body
+            else:
+                json_payload = body
+
+        # Execute with light retry/backoff on transient failures.
+        attempt = 0
+        last_error = None
+        last_response = None
+
+        while True:
+            attempt += 1
+            self._throttle()
+
+            started = time.time()
+            try:
+                resp = requests.request(
+                    method=method,
+                    url=request["url"],
+                    headers=headers,
+                    params=params or None,
+                    json=json_payload if send_json else None,
+                    data=data_payload if not send_json else None,
+                    timeout=15,
+                )
+                elapsed_ms = int((time.time() - started) * 1000)
+
+                last_error = None
+                last_response = (resp, elapsed_ms)
+
+                if attempt <= (AI_SWARM_RETRY_COUNT + 1) and self._should_retry(resp.status_code, None):
+                    # jittered backoff
+                    backoff = (AI_SWARM_RETRY_BACKOFF_MS / 1000.0) * (2 ** (attempt - 1))
+                    backoff = backoff * (0.75 + random.random() * 0.5)
+                    time.sleep(min(backoff, 5.0))
+                    if attempt <= AI_SWARM_RETRY_COUNT:
+                        continue
+
+                try:
+                    body_snippet = resp.json()
+                except Exception:
+                    body_snippet = resp.text
+
+                return {
+                    "name": test["name"],
+                    "mutation": mutation,
+                    "request": {
+                        "method": method,
+                        "url": request["url"],
+                        "headers": headers,
+                        "params": params,
+                        "body": body if method != "GET" else None,
+                    },
+                    "response": {
+                        "status_code": resp.status_code,
+                        "elapsed_ms": elapsed_ms,
+                        "attempt": attempt,
+                        "body_snippet": body_snippet,
+                    },
+                }
+
+            except requests.RequestException as e:
+                elapsed_ms = int((time.time() - started) * 1000)
+                last_error = str(e)
+                last_response = None
+
+                if attempt <= AI_SWARM_RETRY_COUNT and self._should_retry(None, last_error):
+                    backoff = (AI_SWARM_RETRY_BACKOFF_MS / 1000.0) * (2 ** (attempt - 1))
+                    backoff = backoff * (0.75 + random.random() * 0.5)
+                    time.sleep(min(backoff, 5.0))
+                    continue
+
+                return {
+                    "name": test["name"],
+                    "mutation": mutation,
+                    "request": {
+                        "method": method,
+                        "url": request["url"],
+                        "headers": headers,
+                        "params": params,
+                        "body": body if method != "GET" else None,
+                    },
+                    "response": {
+                        "status_code": None,
+                        "elapsed_ms": elapsed_ms,
+                        "attempt": attempt,
+                        "error": last_error,
+                        "body_snippet": None,
+                    },
+                }

{ai_testing_swarm-0.1.1 → ai_testing_swarm-0.1.11}/src/ai_testing_swarm/agents/learning_agent.py

@@ -2,6 +2,7 @@ import json
 import os
 
 DB = "memory/failure_memory.json"
+MAX_ENTRIES = 200
 
 
 class LearningAgent:
@@ -20,10 +21,11 @@ class LearningAgent:
                 # Corrupted file → reset
                 data = []
 
-        data.append({
-            "test": test_name,
-            "reason": reasoning
-        })
+        data.append({"test": test_name, "reason": reasoning})
+
+        # Keep file bounded
+        if len(data) > MAX_ENTRIES:
+            data = data[-MAX_ENTRIES:]
 
         with open(DB, "w") as f:
             json.dump(data, f, indent=2)