pytest-flakehunter 1.0.1__tar.gz

pytest_flakehunter-1.0.1/PKG-INFO

@@ -0,0 +1,238 @@
+Metadata-Version: 2.4
+Name: pytest-flakehunter
+Version: 1.0.1
+Summary: Re-run tests N times, visualize failure heatmaps, and get AI root cause hypotheses
+Keywords: pytest,flaky,testing,qa,heatmap,ai
+Author: jkasser
+License-Expression: MIT
+Classifier: Framework :: Pytest
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Testing
+Classifier: Operating System :: OS Independent
+Requires-Dist: pytest>=7.0
+Requires-Dist: pytest-xdist ; extra == 'xdist'
+Requires-Python: >=3.9
+Project-URL: Homepage, https://github.com/jkasser/pytest-flakehunter
+Project-URL: Repository, https://github.com/jkasser/pytest-flakehunter
+Project-URL: Bug Tracker, https://github.com/jkasser/pytest-flakehunter/issues
+Provides-Extra: ai
+Provides-Extra: xdist
+Description-Content-Type: text/markdown
+
+# pytest-flakehunter 🎯
+
+**Re-run tests N times, visualize failure heatmaps, and get AI-powered root cause hypotheses.**
+
+Flaky tests are expensive. They erode trust in CI, waste engineer time, and mask real bugs.
+`pytest-flakehunter` helps you stop guessing and start diagnosing.
+
+---
+
+## What it does
+
+Run any test suite with `--fh` and get a self-contained HTML report with:
+
+- **Flake rate badges** per test — instantly see your worst offenders
+- **Duration scatter plots** — spot timeout-related flakes vs. race conditions at a glance
+- **Failure heatmaps** — shows *exactly* which line of code fails, and in which run attempts
+- **Stack trace clustering** — groups failures by fingerprint so you know if it's one bug or three
+- **AI root cause hypotheses** — Claude analyzes the clusters and tells you *why* it's probably flaking
+
+![FlakeHunter report screenshot](docs/screenshot.png)
+
+---
+
+## Installation
+
+```bash
+pip install pytest-flakehunter
+```
+
+## Usage
+
+If you use `uv`, pass your `.env` file explicitly:
+```bash
+uv run --env-file=.env pytest tests/ --fh --fh-ai
+```
+
+```bash
+# Basic: run each test 10 times, generate report
+pytest tests/ --fh
+
+# Custom run count
+pytest tests/ --fh --fh-runs=20
+
+# With AI analysis (requires ANTHROPIC_API_KEY env var)
+pytest tests/ --fh --fh-runs=15 --fh-ai
+
+# Custom report path
+pytest tests/ --fh --fh-report=reports/flake_$(date +%Y%m%d).html
+
+# Full fixture isolation between attempts (slower but catches session-state flakes)
+pytest tests/ --fh --fh-isolate
+```
+
+## CLI options
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `--fh` | off | Enable flake hunter mode |
+| `--fh-runs N` | 10 | Re-run count per test |
+| `--fh-report PATH` | `flakehunter_report.html` | Output report path |
+| `--fh-ai` | off | AI root cause analysis via Claude |
+| `--fh-history-dir PATH` | `.flakehunter/history` | Directory for persistent run history |
+| `--fh-no-history` | off | Skip writing to history this run |
+| `--fh-isolate` | off | Full fixture teardown between every attempt (see below) |
+
+---
+
+## Reading the report
+
+### Duration scatter
+Each dot is one run attempt. **Red = failed, green = passed.** 
+- Failures clustering at a consistent high duration → likely a **timeout**
+- Failures scattered at random durations → likely a **race condition or state issue**
+- Single outlier failure at low duration → likely **environment or setup flake**
+
+### Failure commonality heatmap
+A multi-dimensional view of what your failures have in common.
+
+**Current run** — columns are failed attempts, rows are:
+- **Duration** — was this attempt slow, medium, or fast relative to the run? Helps spot timeout-related flakes.
+- **Traceback** — which frames appeared in the stack trace, and in what % of failures.
+
+**Historical** (when history is enabled) — columns are past runs, rows are:
+- **Environment** — which host had failures, and how often.
+- **Branch** — which git branch failures occurred on.
+- **Date** — which week failures clustered in.
+
+Cell brightness = how common that value is across failures. Bright = strong pattern.
+
+### Stack trace clusters
+Failures are grouped by a fingerprint of their innermost stack frames.
+Multiple clusters = multiple distinct failure modes (e.g., two different bugs).
+One cluster = one root cause, just intermittently triggered.
+
+---
+
+## Historical tracking
+
+By default, flakehunter appends each run's results to per-test CSV files in `.flakehunter/history/`. These accumulate across runs, giving the report progressively richer data:
+
+- **Flake rate trend** — see if a test is getting more or less flaky over time
+- **Environment breakdown** — which CI host/branch has the most failures
+- **Argument correlation** — for parametrized tests, which parameter combos fail most
+
+To opt out for a single run:
+```bash
+pytest tests/ --fh --fh-no-history
+```
+
+To store history in a non-default location (e.g., shared network path):
+```bash
+pytest tests/ --fh --fh-history-dir=.cache/flakehunter
+```
+
+The CSV files are safe to commit — they're small, append-only, and give you a free audit trail.
+
+---
+
+## Fixture isolation
+
+By default, flakehunter preserves session- and module-scoped fixtures across re-runs. This means a Playwright browser instance, database connection pool, or asyncio event loop is only created once per test — making repeated runs fast.
+
+**When to use `--fh-isolate`:**
+
+```bash
+pytest tests/ --fh --fh-isolate
+```
+
+This passes a full teardown signal after every attempt, so every re-run starts with a completely clean fixture state. It is slower but will surface a class of flakiness that the default mode can miss — things like:
+
+- Connection pools that degrade after a certain number of operations
+- Session-scoped caches or queues that accumulate state
+- Event loop pollution from uncancelled async tasks
+
+If your tests are fast and stateless, the default is fine. If you're seeing flakiness that only appears after many runs in a long session, try `--fh-isolate`.
+
+---
+
+## AI analysis
+
+Enable with `--fh-ai`. Requires an `ANTHROPIC_API_KEY` — set it using whichever method fits your workflow:
+
+**Shell / CI environment variable** (recommended for CI):
+```bash
+# Linux / macOS
+export ANTHROPIC_API_KEY=sk-ant-...
+pytest tests/ --fh --fh-ai
+
+# Windows CMD
+set ANTHROPIC_API_KEY=sk-ant-...
+pytest tests/ --fh --fh-ai
+
+# Windows PowerShell
+$env:ANTHROPIC_API_KEY = "sk-ant-..."
+pytest tests/ --fh --fh-ai
+```
+
+**`.env` file** (recommended for local development — never commit this file):
+```
+# .env
+ANTHROPIC_API_KEY=sk-ant-...
+```
+If you use `uv`, pass the file explicitly:
+```bash
+uv run --env-file=.env pytest --fh --fh-ai
+```
+
+> **Note:** Do not put your API key in `pytest.ini` or `pyproject.toml` — those files are typically committed to version control.
+
+```
+⚡ AI Root Cause Hypothesis
+
+This test fails most often in setup_db_connection() after ~2.3s, which is
+consistent with a connection pool exhaustion pattern under parallel test
+execution. Consider increasing the test pool size or adding explicit
+teardown that returns connections to the pool between retry attempts.
+```
+
+The AI receives the clustered stack traces, failure rates, and timing data —
+not raw code — so it works on any codebase without needing source access.
+
+---
+
+## Architecture
+
+```
+pytest_flakehunter/
+├── plugin.py       # pytest hooks: --fh option, re-run protocol, xdist support
+├── reporter.py     # Self-contained HTML report with inline SVG charts
+├── history.py      # Append-only CSV history per test, cross-run trend analysis
+└── ai_analysis.py  # Anthropic API integration for root cause hypotheses
+```
+
+**No external dependencies** beyond pytest itself. The HTML report is fully
+self-contained (no CDN calls, no JS frameworks) — safe to commit to repos or
+attach to Jira tickets.
+
+---
+
+## Contributing
+
+```bash
+git clone https://github.com/jkasser/pytest-flakehunter
+cd pytest-flakehunter
+pip install -e ".[dev]"
+pytest demo/sample_tests.py --fh --fh-runs=15
+```
+
+---
+
+## License
+
+MIT

pytest_flakehunter-1.0.1/README.md

@@ -0,0 +1,213 @@
+# pytest-flakehunter 🎯
+
+**Re-run tests N times, visualize failure heatmaps, and get AI-powered root cause hypotheses.**
+
+Flaky tests are expensive. They erode trust in CI, waste engineer time, and mask real bugs.
+`pytest-flakehunter` helps you stop guessing and start diagnosing.
+
+---
+
+## What it does
+
+Run any test suite with `--fh` and get a self-contained HTML report with:
+
+- **Flake rate badges** per test — instantly see your worst offenders
+- **Duration scatter plots** — spot timeout-related flakes vs. race conditions at a glance
+- **Failure heatmaps** — shows *exactly* which line of code fails, and in which run attempts
+- **Stack trace clustering** — groups failures by fingerprint so you know if it's one bug or three
+- **AI root cause hypotheses** — Claude analyzes the clusters and tells you *why* it's probably flaking
+
+![FlakeHunter report screenshot](docs/screenshot.png)
+
+---
+
+## Installation
+
+```bash
+pip install pytest-flakehunter
+```
+
+## Usage
+
+If you use `uv`, pass your `.env` file explicitly:
+```bash
+uv run --env-file=.env pytest tests/ --fh --fh-ai
+```
+
+```bash
+# Basic: run each test 10 times, generate report
+pytest tests/ --fh
+
+# Custom run count
+pytest tests/ --fh --fh-runs=20
+
+# With AI analysis (requires ANTHROPIC_API_KEY env var)
+pytest tests/ --fh --fh-runs=15 --fh-ai
+
+# Custom report path
+pytest tests/ --fh --fh-report=reports/flake_$(date +%Y%m%d).html
+
+# Full fixture isolation between attempts (slower but catches session-state flakes)
+pytest tests/ --fh --fh-isolate
+```
+
+## CLI options
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `--fh` | off | Enable flake hunter mode |
+| `--fh-runs N` | 10 | Re-run count per test |
+| `--fh-report PATH` | `flakehunter_report.html` | Output report path |
+| `--fh-ai` | off | AI root cause analysis via Claude |
+| `--fh-history-dir PATH` | `.flakehunter/history` | Directory for persistent run history |
+| `--fh-no-history` | off | Skip writing to history this run |
+| `--fh-isolate` | off | Full fixture teardown between every attempt (see below) |
+
+---
+
+## Reading the report
+
+### Duration scatter
+Each dot is one run attempt. **Red = failed, green = passed.** 
+- Failures clustering at a consistent high duration → likely a **timeout**
+- Failures scattered at random durations → likely a **race condition or state issue**
+- Single outlier failure at low duration → likely **environment or setup flake**
+
+### Failure commonality heatmap
+A multi-dimensional view of what your failures have in common.
+
+**Current run** — columns are failed attempts, rows are:
+- **Duration** — was this attempt slow, medium, or fast relative to the run? Helps spot timeout-related flakes.
+- **Traceback** — which frames appeared in the stack trace, and in what % of failures.
+
+**Historical** (when history is enabled) — columns are past runs, rows are:
+- **Environment** — which host had failures, and how often.
+- **Branch** — which git branch failures occurred on.
+- **Date** — which week failures clustered in.
+
+Cell brightness = how common that value is across failures. Bright = strong pattern.
+
+### Stack trace clusters
+Failures are grouped by a fingerprint of their innermost stack frames.
+Multiple clusters = multiple distinct failure modes (e.g., two different bugs).
+One cluster = one root cause, just intermittently triggered.
+
+---
+
+## Historical tracking
+
+By default, flakehunter appends each run's results to per-test CSV files in `.flakehunter/history/`. These accumulate across runs, giving the report progressively richer data:
+
+- **Flake rate trend** — see if a test is getting more or less flaky over time
+- **Environment breakdown** — which CI host/branch has the most failures
+- **Argument correlation** — for parametrized tests, which parameter combos fail most
+
+To opt out for a single run:
+```bash
+pytest tests/ --fh --fh-no-history
+```
+
+To store history in a non-default location (e.g., shared network path):
+```bash
+pytest tests/ --fh --fh-history-dir=.cache/flakehunter
+```
+
+The CSV files are safe to commit — they're small, append-only, and give you a free audit trail.
+
+---
+
+## Fixture isolation
+
+By default, flakehunter preserves session- and module-scoped fixtures across re-runs. This means a Playwright browser instance, database connection pool, or asyncio event loop is only created once per test — making repeated runs fast.
+
+**When to use `--fh-isolate`:**
+
+```bash
+pytest tests/ --fh --fh-isolate
+```
+
+This passes a full teardown signal after every attempt, so every re-run starts with a completely clean fixture state. It is slower but will surface a class of flakiness that the default mode can miss — things like:
+
+- Connection pools that degrade after a certain number of operations
+- Session-scoped caches or queues that accumulate state
+- Event loop pollution from uncancelled async tasks
+
+If your tests are fast and stateless, the default is fine. If you're seeing flakiness that only appears after many runs in a long session, try `--fh-isolate`.
+
+---
+
+## AI analysis
+
+Enable with `--fh-ai`. Requires an `ANTHROPIC_API_KEY` — set it using whichever method fits your workflow:
+
+**Shell / CI environment variable** (recommended for CI):
+```bash
+# Linux / macOS
+export ANTHROPIC_API_KEY=sk-ant-...
+pytest tests/ --fh --fh-ai
+
+# Windows CMD
+set ANTHROPIC_API_KEY=sk-ant-...
+pytest tests/ --fh --fh-ai
+
+# Windows PowerShell
+$env:ANTHROPIC_API_KEY = "sk-ant-..."
+pytest tests/ --fh --fh-ai
+```
+
+**`.env` file** (recommended for local development — never commit this file):
+```
+# .env
+ANTHROPIC_API_KEY=sk-ant-...
+```
+If you use `uv`, pass the file explicitly:
+```bash
+uv run --env-file=.env pytest --fh --fh-ai
+```
+
+> **Note:** Do not put your API key in `pytest.ini` or `pyproject.toml` — those files are typically committed to version control.
+
+```
+⚡ AI Root Cause Hypothesis
+
+This test fails most often in setup_db_connection() after ~2.3s, which is
+consistent with a connection pool exhaustion pattern under parallel test
+execution. Consider increasing the test pool size or adding explicit
+teardown that returns connections to the pool between retry attempts.
+```
+
+The AI receives the clustered stack traces, failure rates, and timing data —
+not raw code — so it works on any codebase without needing source access.
+
+---
+
+## Architecture
+
+```
+pytest_flakehunter/
+├── plugin.py       # pytest hooks: --fh option, re-run protocol, xdist support
+├── reporter.py     # Self-contained HTML report with inline SVG charts
+├── history.py      # Append-only CSV history per test, cross-run trend analysis
+└── ai_analysis.py  # Anthropic API integration for root cause hypotheses
+```
+
+**No external dependencies** beyond pytest itself. The HTML report is fully
+self-contained (no CDN calls, no JS frameworks) — safe to commit to repos or
+attach to Jira tickets.
+
+---
+
+## Contributing
+
+```bash
+git clone https://github.com/jkasser/pytest-flakehunter
+cd pytest-flakehunter
+pip install -e ".[dev]"
+pytest demo/sample_tests.py --fh --fh-runs=15
+```
+
+---
+
+## License
+
+MIT

pytest_flakehunter-1.0.1/pyproject.toml

@@ -0,0 +1,36 @@
+[build-system]
+requires = ["uv_build >= 0.10.10, <0.11.0"]
+build-backend = "uv_build"
+
+[project]
+name = "pytest-flakehunter"
+version = "1.0.1"
+description = "Re-run tests N times, visualize failure heatmaps, and get AI root cause hypotheses"
+readme = "README.md"
+requires-python = ">=3.9"
+license = "MIT"
+authors = [{ name = "jkasser" }]
+keywords = ["pytest", "flaky", "testing", "qa", "heatmap", "ai"]
+classifiers = [
+    "Framework :: Pytest",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Software Development :: Testing",
+    "Operating System :: OS Independent",
+]
+dependencies = ["pytest>=7.0"]
+
+[project.urls]
+Homepage = "https://github.com/jkasser/pytest-flakehunter"
+Repository = "https://github.com/jkasser/pytest-flakehunter"
+"Bug Tracker" = "https://github.com/jkasser/pytest-flakehunter/issues"
+
+[project.optional-dependencies]
+xdist = ["pytest-xdist"]
+ai = []  # Anthropic API uses stdlib urllib; env vars loaded by uv from .env automatically
+
+[project.entry-points."pytest11"]
+flakehunter = "pytest_flakehunter.plugin"

pytest_flakehunter-1.0.1/src/pytest_flakehunter/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.1.1"

pytest_flakehunter-1.0.1/src/pytest_flakehunter/ai_analysis.py

@@ -0,0 +1,134 @@
+"""
+AI-powered root cause analysis for flaky tests.
+Calls the Anthropic API with clustered stack traces AND historical trend data
+to generate richer, more specific hypotheses.
+"""
+
+from __future__ import annotations
+import os
+import json
+from typing import TYPE_CHECKING, Optional
+
+if TYPE_CHECKING:
+    from pytest_flakehunter.plugin import TestFlakeRecord
+
+from pytest_flakehunter.plugin import extract_frames, extract_error, short_path
+
+
+def analyze_flaky_test(
+    record: "TestFlakeRecord",
+    history_summary: Optional[dict] = None,
+) -> str:
+    api_key = os.environ.get("ANTHROPIC_API_KEY", "")
+    if not api_key:
+        return ""
+
+    try:
+        import urllib.request
+        import urllib.error
+
+        clusters = record.failure_clusters()
+        cluster_summaries = []
+        for fp, attempts in clusters.items():
+            fr = attempts[0].failed_report
+            if not fr:
+                continue
+            error_type, error_msg = extract_error(fr.longrepr)
+            frames = extract_frames(fr.longrepr)
+            frames_text = "\n".join(
+                f"  {short_path(f[0])}:{f[1]} in {f[2]}() -- {f[3]}"
+                for f in frames[-5:]
+            )
+            durations = [a.total_duration for a in attempts]
+            cluster_summaries.append(
+                f"Cluster {fp} ({len(attempts)} occurrences, "
+                f"avg {sum(durations)/len(durations):.2f}s):\n"
+                f"Error: {error_type}: {error_msg}\n"
+                f"Stack (innermost last):\n{frames_text}"
+            )
+
+        if not cluster_summaries and not history_summary:
+            return ""
+
+        history_section = ""
+        if history_summary and history_summary.get("total_attempts", 0) > 0:
+            hs = history_summary
+            lines = [
+                f"\nHISTORICAL DATA ({hs.get('date_range', ('?','?'))[0]} -> {hs.get('date_range', ('?','?'))[1]}):",
+                f"  Total historical attempts: {hs.get('total_attempts')}",
+                f"  Overall historical flake rate: {hs.get('overall_flake_rate', 0):.1%}",
+                f"  Avg duration: {hs.get('avg_duration_s', 0):.3f}s  P95: {hs.get('p95_duration_s', 0):.3f}s",
+            ]
+            run_rates = hs.get("run_flake_rates", [])
+            if len(run_rates) >= 3:
+                recent_3 = [r["flake_rate"] for r in run_rates[-3:]]
+                early_3  = [r["flake_rate"] for r in run_rates[:3]]
+                trend = sum(recent_3)/3 - sum(early_3)/3
+                direction = "INCREASING" if trend > 0.05 else "DECREASING" if trend < -0.05 else "STABLE"
+                lines.append(f"  Flake rate trend: {direction} (delta {trend:+.1%} vs earliest runs)")
+            env = hs.get("env_breakdown", {})
+            if len(env) > 1:
+                lines.append("  Per-host flake rates:")
+                for host, v in env.items():
+                    rate = v["failed"] / v["total"] if v["total"] else 0
+                    lines.append(f"    {host}: {rate:.0%} ({v['failed']}/{v['total']})")
+            arg_corr = hs.get("arg_correlation", {})
+            if arg_corr:
+                high_fail = {k: v for k, v in arg_corr.items() if v["rate"] > 0.3 and v["total"] >= 3}
+                if high_fail:
+                    lines.append("  High-failure parameter combinations:")
+                    for combo, v in sorted(high_fail.items(), key=lambda x: -x[1]["rate"])[:3]:
+                        lines.append(f"    {combo}: {v['rate']:.0%} fail rate ({v['failed']}/{v['total']})")
+            branches = hs.get("git_branches_seen", [])
+            if branches:
+                lines.append(f"  Git branches with failures: {', '.join(branches[:5])}")
+            history_section = "\n".join(lines)
+
+        prompt = f"""You are an expert software engineer analyzing a flaky pytest test.
+
+CURRENT RUN:
+Test: {record.nodeid}
+Flake rate this run: {record.flake_rate:.0%} ({record.fail_count} failures in {len(record.attempts)} runs)
+Avg duration on failure: {_avg_fail_duration(record):.2f}s
+
+Failure clusters:
+{chr(10).join(cluster_summaries) if cluster_summaries else "(no failures this run -- see historical data)"}
+{history_section}
+
+In 3-4 sentences, give a concise hypothesis about WHY this test is flaky.
+Be specific: timing races, state pollution, resource exhaustion, external deps, parameter-specific bugs.
+If historical data shows a trend, host correlation, or parameter correlation, call it out explicitly.
+Do not repeat the stack trace. Be direct and actionable."""
+
+        payload = json.dumps({
+            "model": "claude-sonnet-4-6",
+            "max_tokens": 400,
+            "messages": [{"role": "user", "content": prompt}]
+        }).encode()
+
+        req = urllib.request.Request(
+            "https://api.anthropic.com/v1/messages",
+            data=payload,
+            headers={
+                "x-api-key": api_key,
+                "anthropic-version": "2023-06-01",
+                "content-type": "application/json",
+            }
+        )
+        try:
+            with urllib.request.urlopen(req, timeout=20) as resp:
+                data = json.loads(resp.read())
+                return data["content"][0]["text"].strip()
+        except urllib.error.HTTPError as exc:
+            body = exc.read().decode("utf-8", errors="replace")
+            print(f"\n[flakehunter] AI analysis failed for {record.nodeid}: {exc} — {body}")
+            return ""
+
+    except Exception as exc:
+        print(f"\n[flakehunter] AI analysis failed for {record.nodeid}: {exc}")
+        return ""
+
+
+def _avg_fail_duration(record: "TestFlakeRecord") -> float:
+    durations = [a.total_duration for a in record.attempts if a.outcome == "failed"]
+    return sum(durations) / len(durations) if durations else 0.0