aws-test-plugin 0.1.0__py3-none-any.whl

aws_test_plugin/.claude-plugin/plugin.json

@@ -0,0 +1,25 @@
+{
+  "name": "aws-test-plugin",
+  "description": "AI-powered test generation skills for AWS Python projects — Lambda, API Gateway, Step Functions, and Batch. Generates E2E, integration, contract, performance, and load tests by reading actual handler code.",
+  "version": "0.1.0",
+  "author": {
+    "name": "whitewhiteqq"
+  },
+  "homepage": "https://github.com/whitewhiteqq/aws-test-plugin",
+  "repository": "https://github.com/whitewhiteqq/aws-test-plugin",
+  "license": "Apache-2.0",
+  "keywords": [
+    "aws",
+    "testing",
+    "lambda",
+    "api-gateway",
+    "step-functions",
+    "batch",
+    "pytest",
+    "moto",
+    "locust",
+    "e2e",
+    "integration",
+    "contract"
+  ]
+}

aws_test_plugin/AGENTS.md

@@ -0,0 +1,66 @@
+# AWS Test Plugin — Agent Instructions
+
+This project uses `aws-test-plugin` to generate comprehensive tests
+for AWS Python projects (Lambda, API Gateway, Step Functions, Batch).
+
+## Available Test Types
+
+| Type | Tool | Local | Deployed |
+|------|------|-------|----------|
+| Unit | pytest + mocks | Yes | No |
+| Integration | moto + pytest | Yes | No |
+| Contract | jsonschema + pytest | Yes | No |
+| Performance | pytest-benchmark | Yes | No |
+| E2E | requests + pytest | No | Yes |
+| Load | Locust | No | Yes |
+
+## Testing Workflow
+
+1. **Discover** — scan for Lambda handlers (`main.py`, `handler.py`), Batch jobs
+   (`Dockerfile`), Step Functions (`*.asl.json`), API specs (`swagger.yml`)
+2. **Read code** — analyze handler source for branches, AWS calls, validation
+3. **Generate tests** — create test files covering every code path, branch, and boundary
+4. **Run** — `python scripts/run_tests.py all` (1-click for all offline tests)
+5. **Fix** — read failures, adjust test logic, re-run
+
+## Secrets Management
+
+- **Never** hardcode API keys, tokens, or passwords in test files
+- Two strategies available (see secrets-and-config.md for full details):
+  - **Strategy A**: `.env` files (gitignored) — secrets loaded via `os.getenv()` with `pytest.skip()` fallback
+  - **Strategy B**: AWS Secrets Manager + SSM Parameter Store — secrets retrieved via `boto3` with `pytest.skip()` fallback
+- **AWS credentials**: use the default AWS profile (`~/.aws/credentials`) — boto3 picks it up automatically
+- Unit/integration/contract tests work with fake `"testing"` credentials (moto) — no real AWS access needed
+- E2E/load tests load real credentials from `.env` or Secrets Manager depending on chosen strategy
+
+## Commands
+
+```bash
+# Scaffold test directories + .env.example
+aws-test-plugin scaffold .
+
+# 1-click: run ALL offline tests (unit + integration + contract + performance)
+python scripts/run_tests.py all
+
+# 1-click: run EVERYTHING (offline + E2E; tests self-skip if env vars missing)
+python scripts/run_tests.py full
+
+# Individual categories
+pytest tests/unit/ -m unit -v --cov=src/ --cov-branch
+pytest tests/integration/ -m integration -v
+pytest tests/contract/ -m contract -v
+pytest tests/performance/ -m performance --benchmark-enable
+pytest tests/e2e/ -m e2e -v  # reads API_BASE_URL from .env
+
+# Load tests (requires deployed API)
+locust -f tests/load/locustfile.py --host=$LOAD_TEST_HOST \
+  --users=50 --spawn-rate=5 --run-time=5m --headless
+```
+
+## Safety
+
+- Never run E2E or load tests against production without confirmation
+- All test data uses prefixes: `TEST-`, `E2E-TEST-`, `LOAD-TEST-`
+- Integration tests use moto (no real AWS calls)
+- Load tests must have `--run-time` set
+- No secrets in test files — use `.env` (gitignored) or AWS Secrets Manager / SSM Parameter Store

aws_test_plugin/__init__.py

@@ -0,0 +1,3 @@
+"""aws-test-plugin — AI-powered test generation for AWS Python projects."""
+
+__version__ = "0.1.0"

aws_test_plugin/agents/aws-test-engineer.md

@@ -0,0 +1,95 @@
+---
+name: aws-test-engineer
+purpose: >
+  Generate, run, and fix comprehensive tests for AWS Python projects.
+  Analyzes actual handler code to produce E2E, integration, contract,
+  performance, and load tests. Works with Lambda, API Gateway,
+  Step Functions, and Batch jobs.
+skills:
+  - aws-test-orchestrator
+  - aws-unit-testing
+  - aws-e2e-testing
+  - aws-integration-testing
+  - aws-contract-testing
+  - aws-perf-load-testing
+definition_of_done:
+  - All discovered AWS components have corresponding test files
+  - Unit tests cover every branch, boundary value, and edge case in business logic
+  - Every code path (happy path, error handling, edge cases) has a test
+  - Tests pass locally with moto/testcontainers (no real AWS calls)
+  - Test coverage for handler logic is >= 80% (branch coverage)
+  - Performance benchmarks have defined thresholds
+  - Load test locustfiles match API spec endpoints
+  - Test data uses prefixes for easy cleanup (TEST-, E2E-TEST-, LOAD-TEST-)
+  - conftest.py provides shared fixtures for all test types
+  - pytest.ini configures markers and test paths
+  - No secrets, API keys, or tokens hardcoded in test files
+  - .env.example documents required configuration variables
+  - All offline tests can run in 1 click via `python scripts/run_tests.py all`
+  - Installation guidance lists the required test dependency groups
+safety:
+  - Never run tests against production without explicit user confirmation
+  - Never make real AWS API calls — use moto or testcontainers
+  - Never store credentials, API keys, or tokens in test files
+  - Never delete production data
+  - Load tests must have --run-time set (no unbounded runs)
+  - All test data is prefixed and self-cleaning
+  - Use .env files (gitignored) for secrets, never commit .env
+---
+
+# AWS Test Engineer Agent
+
+I generate comprehensive tests for AWS Python projects by reading the actual
+source code, not from generic templates.
+
+## Workflow
+
+1. **Discover** — Scan the project for Lambda handlers, Batch jobs, Step
+   Function definitions, API Gateway specs, and existing tests.
+
+2. **Analyze Code** — Read each handler to understand:
+   - Input event shapes and validation rules
+   - AWS service interactions (boto3 calls)
+   - Branching logic and error handling paths
+   - Return value structures
+   - Environment variable dependencies
+
+3. **Generate Tests** — Create test files that cover every code path:
+   - Unit tests for pure business logic (all branches, boundaries, edge cases)
+   - Integration tests with moto for AWS service interactions
+   - E2E tests for deployed API endpoints
+   - Contract tests for OpenAPI/Swagger compliance
+   - Performance benchmarks for latency and memory
+   - Load tests for throughput and SLA validation
+
+4. **Scaffold** — Set up test infrastructure:
+   - `conftest.py` with shared fixtures
+   - `pytest.ini` with markers and configuration
+   - Test directory structure matching the project layout
+
+5. **Run & Fix** — Execute tests and iteratively fix failures by reading
+   error output and adjusting test logic.
+
+## When to Use Me
+
+| User Says | What I Do |
+|-----------|-----------|
+| "Write tests for my Lambda" | Discover handlers → read code → generate unit + integration tests |
+| "Test all branches" | Read handler code → map every if/else/try/except → generate unit tests |
+| "Test boundary values" | Identify numeric/string/collection fields → parametrized boundary tests |
+| "Add E2E tests for my API" | Find API spec → read handlers → generate endpoint tests |
+| "Test my Step Function" | Find ASL definition → read state handlers → generate flow tests |
+| "Benchmark my handler" | Read handler → mock deps → generate pytest-benchmark tests |
+| "Load test my API" | Read API spec → generate Locust users with weighted endpoints |
+| "Set up test infrastructure" | Scaffold conftest.py, pytest.ini, .env.example, directory layout |
+| "What's not tested?" | Compare handler code paths vs existing tests → report gaps |
+| "Run all tests" | 1-click: `python scripts/run_tests.py all` (or `full` for E2E too) |
+
+## Test Categories
+
+| Category | Tool | Runs Locally | Needs Deploy |
+|----------|------|-------------|--------------|| Unit | pytest + mocks | Yes | No || Integration | moto + pytest | Yes | No |
+| Contract | jsonschema + pytest | Yes | No |
+| Performance | pytest-benchmark | Yes | No |
+| E2E | requests + pytest | No | Yes |
+| Load | Locust | No | Yes |

aws_test_plugin/cli.py

@@ -0,0 +1,199 @@
+"""CLI entry point for aws-test-plugin."""
+
+import shutil
+import sys
+from importlib.resources import files
+from pathlib import Path
+
+
+def _get_pkg_dir() -> Path:
+    """Return the installed package directory (contains AGENTS.md, scripts/)."""
+    return Path(str(files("aws_test_plugin")))
+
+
+def _get_data_dir() -> Path:
+    """Return the directory containing skills/ and agents/.
+
+    In an installed wheel, force-include maps root skills/ and agents/ into
+    the package directory.  During editable/development installs the package
+    dir won't have them, so fall back to the repo root.
+    """
+    pkg = _get_pkg_dir()
+    if (pkg / "skills").is_dir():
+        return pkg
+    # Editable install: src/aws_test_plugin -> repo root is two levels up
+    repo_root = pkg.parent.parent
+    if (repo_root / "skills").is_dir():
+        return repo_root
+    return pkg
+
+
+def _copy_tree(src: Path, dst: Path, label: str) -> int:
+    """Copy a directory tree, reporting what's copied. Returns file count."""
+    count = 0
+    for item in sorted(src.rglob("*")):
+        if item.is_file():
+            rel = item.relative_to(src)
+            target = dst / rel
+            target.parent.mkdir(parents=True, exist_ok=True)
+            if target.exists():
+                print(f"  skip  {label}/{rel}  (already exists)")
+            else:
+                shutil.copy2(item, target)
+                print(f"  +     {label}/{rel}")
+                count += 1
+    return count
+
+
+def cmd_init(target: str, agents: list[str] | None = None) -> None:
+    """Install skills and agents into a target project."""
+    root = Path(target).resolve()
+    data = _get_data_dir()
+
+    agents_to_install = agents or ["claude", "copilot", "codex"]
+
+    total = 0
+    skills_src = data / "skills"
+    agents_src = data / "agents"
+
+    # Always install skills
+    if skills_src.is_dir():
+        for agent in agents_to_install:
+            if agent in ("claude",):
+                dst = root / ".claude" / "skills"
+                total += _copy_tree(skills_src, dst, ".claude/skills")
+            if agent in ("copilot",):
+                dst = root / ".github" / "skills"
+                total += _copy_tree(skills_src, dst, ".github/skills")
+
+    # Install agent definitions
+    if agents_src.is_dir():
+        for agent in agents_to_install:
+            if agent in ("claude",):
+                dst = root / ".claude" / "agents"
+                total += _copy_tree(agents_src, dst, ".claude/agents")
+            if agent in ("copilot",):
+                dst = root / ".github" / "agents"
+                total += _copy_tree(agents_src, dst, ".github/agents")
+
+    # For Codex: generate AGENTS.md if requested
+    if "codex" in agents_to_install:
+        agents_md = root / "AGENTS.md"
+        if not agents_md.exists():
+            # AGENTS.md lives in the package dir, not with skills/agents
+            agents_md_src = _get_pkg_dir() / "AGENTS.md"
+            if agents_md_src.exists():
+                shutil.copy2(agents_md_src, agents_md)
+                print("  +     AGENTS.md")
+                total += 1
+
+    print(f"\nInstalled {total} files into {root}")
+    print("Next: pip install 'aws-test-plugin[test]'  (or uv sync --extra test)")
+
+
+def cmd_list() -> None:
+    """List available skills."""
+    data = _get_data_dir()
+    skills_dir = data / "skills"
+    if not skills_dir.is_dir():
+        print("No skills found.")
+        return
+
+    print("Available skills:\n")
+    for skill in sorted(skills_dir.iterdir()):
+        if skill.is_dir() and (skill / "SKILL.md").exists():
+            # Read first line of description from frontmatter
+            desc = ""
+            in_frontmatter = False
+            in_desc = False
+            for line in (skill / "SKILL.md").read_text(encoding="utf-8").splitlines():
+                if line.strip() == "---" and not in_frontmatter:
+                    in_frontmatter = True
+                    continue
+                if line.strip() == "---" and in_frontmatter:
+                    break
+                if line.startswith("description:"):
+                    desc = line.split(":", 1)[1].strip().strip(">").strip()
+                    in_desc = True
+                    continue
+                if in_desc and line.startswith("  "):
+                    desc += " " + line.strip()
+                    continue
+                if in_desc:
+                    in_desc = False
+
+            refs = list((skill / "references").glob("*.md")) if (skill / "references").is_dir() else []
+            print(f"  {skill.name}")
+            if desc:
+                print(f"    {desc[:100]}")
+            if refs:
+                print(f"    references: {', '.join(r.stem for r in refs)}")
+            print()
+
+
+def cmd_scaffold(target: str) -> None:
+    """Run the scaffold script to discover and set up test directories."""
+    # Scripts live in the package dir, not with skills/agents
+    pkg = _get_pkg_dir()
+    scaffold_script = pkg / "scripts" / "scaffold.py"
+    if not scaffold_script.exists():
+        print("Error: scaffold.py not found in package data.")
+        sys.exit(1)
+
+    # Import and run scaffold.main() directly
+    import importlib.util
+
+    spec = importlib.util.spec_from_file_location("scaffold", scaffold_script)
+    if spec is None or spec.loader is None:
+        print("Error: unable to load scaffold.py from package data.")
+        sys.exit(1)
+
+    mod = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(mod)
+    sys.argv = ["scaffold", "--project-root", target]
+    mod.main()
+
+
+def main() -> None:
+    """CLI entry point."""
+    import argparse
+
+    parser = argparse.ArgumentParser(
+        prog="aws-test-plugin",
+        description="Install AWS test skills for AI coding agents (Claude Code, GitHub Copilot, Codex)",
+    )
+    sub = parser.add_subparsers(dest="command", required=True)
+
+    # init
+    p_init = sub.add_parser("init", help="Install skills and agents into a project")
+    p_init.add_argument("target", nargs="?", default=".", help="Project root (default: .)")
+    p_init.add_argument(
+        "--agent",
+        "-a",
+        action="append",
+        choices=["claude", "copilot", "codex", "all"],
+        help="Which agent(s) to install for (default: all). Repeatable.",
+    )
+
+    # list
+    sub.add_parser("list", help="List available skills")
+
+    # scaffold
+    p_scaffold = sub.add_parser("scaffold", help="Scaffold test directories for an AWS project")
+    p_scaffold.add_argument("target", nargs="?", default=".", help="Project root (default: .)")
+
+    args = parser.parse_args()
+
+    if args.command == "init":
+        agents = args.agent or ["all"]
+        if "all" in agents:
+            agents = ["claude", "copilot", "codex"]
+        cmd_init(args.target, agents)
+    elif args.command == "list":
+        cmd_list()
+    elif args.command == "scaffold":
+        cmd_scaffold(args.target)
+
+
+if __name__ == "__main__":
+    main()

aws_test_plugin/scripts/analyze_results.py

@@ -0,0 +1,208 @@
+#!/usr/bin/env python3
+"""
+Analyze test results from JUnit XML, Locust CSV, and pytest-benchmark JSON.
+
+Usage:
+    python analyze_results.py tests/reports/
+"""
+
+import csv
+import json
+import sys
+from pathlib import Path
+
+from defusedxml import ElementTree as ET
+
+
+def analyze_junit(xml_path: Path) -> dict:
+    """Parse JUnit XML and return summary."""
+    tree = ET.parse(xml_path)
+    root = tree.getroot()
+    if root is None:
+        raise ValueError(f"Invalid JUnit XML file: {xml_path}")
+
+    suites = root.findall(".//testsuite") if root.tag != "testsuite" else [root]
+    total = errors = failures = skipped = 0
+    failed_tests = []
+
+    for suite in suites:
+        total += int(suite.get("tests", 0))
+        errors += int(suite.get("errors", 0))
+        failures += int(suite.get("failures", 0))
+        skipped += int(suite.get("skipped", 0))
+
+        for tc in suite.findall("testcase"):
+            failure = tc.find("failure")
+            error = tc.find("error")
+            detail = failure if failure is not None else error
+            if detail is not None:
+                msg = detail.get("message", "")
+                failed_tests.append(
+                    {
+                        "name": f"{tc.get('classname')}.{tc.get('name')}",
+                        "message": msg[:200],
+                    }
+                )
+
+    return {
+        "file": xml_path.name,
+        "total": total,
+        "passed": total - errors - failures - skipped,
+        "failed": errors + failures,
+        "skipped": skipped,
+        "failed_tests": failed_tests,
+    }
+
+
+def analyze_locust(stats_path: Path) -> dict:
+    """Parse Locust stats CSV and return summary."""
+    results = []
+    aggregated = None
+
+    with open(stats_path) as f:
+        reader = csv.DictReader(f)
+        for row in reader:
+            entry = {
+                "name": row.get("Name", ""),
+                "requests": int(row.get("Request Count", 0)),
+                "failures": int(row.get("Failure Count", 0)),
+                "avg_ms": float(row.get("Average Response Time", 0)),
+                "p50_ms": float(row.get("50%", 0)),
+                "p95_ms": float(row.get("95%", 0)),
+                "p99_ms": float(row.get("99%", 0)),
+                "rps": float(row.get("Requests/s", 0)),
+            }
+            if entry["name"] == "Aggregated":
+                aggregated = entry
+            else:
+                results.append(entry)
+
+    return {
+        "file": stats_path.name,
+        "endpoints": results,
+        "aggregated": aggregated,
+    }
+
+
+def analyze_benchmark(json_path: Path) -> dict:
+    """Parse pytest-benchmark JSON and return summary."""
+    with open(json_path) as f:
+        data = json.load(f)
+
+    benchmarks = []
+    for bm in data.get("benchmarks", []):
+        benchmarks.append(
+            {
+                "name": bm.get("name", ""),
+                "min_ms": bm["stats"]["min"] * 1000,
+                "max_ms": bm["stats"]["max"] * 1000,
+                "mean_ms": bm["stats"]["mean"] * 1000,
+                "median_ms": bm["stats"]["median"] * 1000,
+                "stddev_ms": bm["stats"]["stddev"] * 1000,
+                "rounds": bm["stats"]["rounds"],
+            }
+        )
+
+    return {
+        "file": json_path.name,
+        "benchmarks": benchmarks,
+    }
+
+
+def print_junit_report(result: dict):
+    """Print JUnit analysis."""
+    print(f"\n--- {result['file']} ---")
+    status = "PASS" if result["failed"] == 0 else "FAIL"
+    print(f"  Status: {status}")
+    print(
+        f"  Total: {result['total']}  "
+        f"Passed: {result['passed']}  "
+        f"Failed: {result['failed']}  "
+        f"Skipped: {result['skipped']}"
+    )
+
+    if result["failed_tests"]:
+        print("  Failures:")
+        for ft in result["failed_tests"]:
+            print(f"    - {ft['name']}: {ft['message']}")
+
+
+def print_locust_report(result: dict):
+    """Print Locust analysis."""
+    print(f"\n--- {result['file']} ---")
+    agg = result.get("aggregated")
+    if agg:
+        error_pct = agg["failures"] / agg["requests"] * 100 if agg["requests"] > 0 else 0
+        print(f"  Total Requests: {agg['requests']}")
+        print(f"  Error Rate: {error_pct:.1f}%")
+        print(
+            f"  Avg: {agg['avg_ms']:.0f}ms  "
+            f"p50: {agg['p50_ms']:.0f}ms  "
+            f"p95: {agg['p95_ms']:.0f}ms  "
+            f"p99: {agg['p99_ms']:.0f}ms"
+        )
+        print(f"  Throughput: {agg['rps']:.1f} req/s")
+
+    if result["endpoints"]:
+        print("\n  Per-endpoint:")
+        for ep in result["endpoints"]:
+            print(
+                f"    {ep['name']}: avg={ep['avg_ms']:.0f}ms "
+                f"p95={ep['p95_ms']:.0f}ms "
+                f"({ep['requests']} reqs, {ep['failures']} fails)"
+            )
+
+
+def print_benchmark_report(result: dict):
+    """Print benchmark analysis."""
+    print(f"\n--- {result['file']} ---")
+    for bm in result["benchmarks"]:
+        print(f"  {bm['name']}:")
+        print(
+            f"    mean={bm['mean_ms']:.2f}ms  "
+            f"median={bm['median_ms']:.2f}ms  "
+            f"min={bm['min_ms']:.2f}ms  "
+            f"max={bm['max_ms']:.2f}ms  "
+            f"({bm['rounds']} rounds)"
+        )
+
+
+def main():
+    reports_dir = Path(sys.argv[1] if len(sys.argv) > 1 else "tests/reports")
+
+    if not reports_dir.exists():
+        print(f"Reports directory not found: {reports_dir}")
+        sys.exit(1)
+
+    print(f"Analyzing results in {reports_dir}/\n")
+    print("=" * 60)
+
+    has_failures = False
+
+    # JUnit XML files
+    for xml_file in sorted(reports_dir.glob("*_results.xml")):
+        result = analyze_junit(xml_file)
+        print_junit_report(result)
+        if result["failed"] > 0:
+            has_failures = True
+
+    # Locust CSV files
+    for csv_file in sorted(reports_dir.glob("*_stats.csv")):
+        result = analyze_locust(csv_file)
+        print_locust_report(result)
+
+    # Benchmark JSON files
+    for json_file in sorted(reports_dir.glob("*benchmark*.json")):
+        result = analyze_benchmark(json_file)
+        print_benchmark_report(result)
+
+    print(f"\n{'=' * 60}")
+    if has_failures:
+        print("RESULT: Some tests failed. See details above.")
+        sys.exit(1)
+    else:
+        print("RESULT: All analyzed results look good.")
+
+
+if __name__ == "__main__":
+    main()