flarebisect 0.3.1__tar.gz

flarebisect-0.3.1/.gitignore

@@ -0,0 +1,12 @@
+__pycache__/
+*.pyc
+.venv/
+venv/
+*.egg-info/
+dist/
+build/
+.pytest_cache/
+.ruff_cache/
+.env
+.DS_Store
+demo/flaky-counter/

flarebisect-0.3.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Tawin Tangsukson
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

flarebisect-0.3.1/PKG-INFO

@@ -0,0 +1,192 @@
+Metadata-Version: 2.4
+Name: flarebisect
+Version: 0.3.1
+Summary: git bisect that treats flakiness as a signal, not noise
+Project-URL: Homepage, https://github.com/kaorii-ako/FlareBisect
+Project-URL: Repository, https://github.com/kaorii-ako/FlareBisect
+Project-URL: Issues, https://github.com/kaorii-ako/FlareBisect/issues
+Author: Tawin Tangsukson
+License: MIT
+License-File: LICENSE
+Keywords: bisect,cli,debugging,flaky-tests,git,testing
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Debuggers
+Classifier: Topic :: Software Development :: Testing
+Classifier: Topic :: Software Development :: Version Control :: Git
+Requires-Python: >=3.10
+Requires-Dist: anthropic>=0.40
+Requires-Dist: openai>=1.40
+Requires-Dist: rich>=13.7
+Requires-Dist: typer>=0.12
+Provides-Extra: dev
+Requires-Dist: build>=1.2; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: twine>=5.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# FlareBisect 🔥
+
+[![PyPI](https://img.shields.io/pypi/v/flarebisect.svg)](https://pypi.org/project/flarebisect/)
+[![CI](https://github.com/kaorii-ako/FlareBisect/actions/workflows/ci.yml/badge.svg)](https://github.com/kaorii-ako/FlareBisect/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+
+FlareBisect finds the commit that made your tests flaky, not just the one
+that broke them: it bisects on flake-rate drift instead of pass/fail, then
+flares the culprit with an AI-written root cause.
+
+`git bisect` gives you confidently wrong answers when the test you're
+bisecting on is flaky — it treats every run as a clean pass/fail signal, so
+a test that was already unstable *before* you started bisecting can point
+straight at an innocent commit.
+
+`flarebisect` bisects on **flake rate**, not pass/fail. It runs each
+candidate commit's test N times in parallel, computes a failure rate, and
+finds the commit where that rate jumps — then asks an LLM to explain, in
+plain English, whether the commit broke the test cleanly or made it flakier.
+
+```
+12 commits in range · 20 runs per commit · parallel worktrees
+
+commit   flake rate                                  result  status
+a1b2c3                                                  5/5  good
+4f9e21                                                  5/5  stable
+7d3aa0   ███████                                        4/5  wobbling
+9c1f88   ████████████████████████████                   1/5  flare
+HEAD     ████████████████████████████                   1/5  bad
+
+╭────────────────────────────────────────────────────────────────╮
+│ ✦ culprit found · commit 9c1f88                                    │
+│                                                                        │
+│ flake rate jumped 20% → 80% at this commit                             │
+│ "add request counter for rate limiting"                                │
+│                                                                        │
+│   💡 likely cause — shared counter incremented without a lock.        │
+│   concurrent test workers race on the same variable.                   │
+╰────────────────────────────────────────────────────────────────╯
+⏱ 8.4s    ⚡ 5 checked    ⚙ 4 parallel workers
+```
+
+## Install
+
+```bash
+pip install flarebisect
+```
+
+For local development:
+
+```bash
+pip install -e ".[dev]"
+pytest -q
+```
+
+Works on Linux, macOS, and Windows — the only external dependency is `git`
+on your `PATH`. Every candidate commit runs in its own `git worktree` under a
+temp directory; your actual working tree and index are never touched.
+
+## AI provider setup
+
+The root-cause explanation step works with Claude, OpenAI, Gemini, or a
+local model — anything speaking an OpenAI-compatible chat API (Ollama, LM
+Studio, llama.cpp server, vLLM, and similar all qualify).
+
+```bash
+# cloud
+flarebisect config set-key anthropic sk-ant-...
+flarebisect config set-key openai sk-...
+flarebisect config set-key google AI...
+
+# local, no key needed - just have Ollama running
+flarebisect config use ollama
+flarebisect config set-model ollama llama3.1
+
+# or let flarebisect pick a model sized to your GPU and pull it for you
+flarebisect models detect   # shows detected GPU/VRAM + recommended model
+flarebisect models pull     # downloads it and sets it active (run with no args)
+
+# any other OpenAI-compatible endpoint
+flarebisect config use custom
+flarebisect config set-base-url custom http://localhost:8080/v1
+flarebisect config set-key custom sk-local
+```
+
+Run `flarebisect config` with no arguments for a guided setup wizard — for
+Ollama it detects your GPU/VRAM, prefills a right-sized model, and offers to
+pull it on the spot.
+
+`flarebisect config show` lists the active provider and stored settings
+(keys are masked). Config lives in a JSON file under the OS config dir
+(`~/.config/flarebisect/config.json` on Linux/macOS, `%APPDATA%` on Windows),
+written with owner-only permissions.
+
+Per-run overrides skip the config file entirely:
+
+```bash
+flarebisect run ... --provider openai --model gpt-4o-mini --api-key sk-...
+flarebisect run ... --provider ollama --base-url http://localhost:11434/v1
+```
+
+Env vars (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GOOGLE_API_KEY`) work too,
+as a fallback under whatever's in the config file.
+
+## Usage
+
+```bash
+flarebisect run \
+  --repo /path/to/repo \
+  --good <known-good-sha> \
+  --bad <known-bad-sha> \
+  --test "pytest -k my_flaky_test" \
+  --runs 20 \
+  --threshold 0.3
+```
+
+- `--runs` — test executions per commit (default 20), run in parallel
+  (capped at your core count so it doesn't oversubscribe). Lower counts are
+  faster but noisier — a low sample can misattribute the culprit near the
+  threshold boundary.
+- `--threshold` — flake-rate jump (0-1) vs. the `good` baseline required to
+  call a commit the culprit (default 0.3).
+- `--no-explain` — skip the LLM root-cause call entirely (fully offline).
+
+## Demo
+
+See [`demo/`](demo/) for a self-contained, seeded-flaky-bug repo you can
+bisect against with no network access required for the bisection itself.
+
+## How it works
+
+1. Measure the flake rate at `good` and `bad` as baselines.
+2. Binary-search the commit range between them.
+3. At each candidate, run the test `N` times in an isolated worktree and
+   compute `failed / N`.
+4. The culprit is the first commit whose flake rate jumps by `>= threshold`
+   over the `good` baseline.
+5. Verdict: **clean break** (rate goes ~0% → ~100%) vs. **flakiness
+   regression** (rate goes ~0% → somewhere in between).
+6. The configured LLM gets the culprit's diff + the flake-rate evidence and
+   returns a short root-cause read (race condition, shared mutable state,
+   timing assumption, etc.).
+
+Binary search assumes the flake rate is roughly monotonic across the range,
+same as ordinary `git bisect` assumes pass/fail is. At low `--runs` counts
+a noisy sample can occasionally violate that near the threshold boundary —
+bump `--runs` if a result looks off.
+
+## Releasing
+
+Version is single-sourced from `src/flarebisect/__init__.py`. Bump it, commit,
+tag (`vX.Y.Z`), and cut a GitHub Release — `.github/workflows/publish.yml`
+builds and publishes to PyPI via trusted publishing (OIDC, no stored token).
+
+## License
+
+MIT

flarebisect-0.3.1/README.md

@@ -0,0 +1,157 @@
+# FlareBisect 🔥
+
+[![PyPI](https://img.shields.io/pypi/v/flarebisect.svg)](https://pypi.org/project/flarebisect/)
+[![CI](https://github.com/kaorii-ako/FlareBisect/actions/workflows/ci.yml/badge.svg)](https://github.com/kaorii-ako/FlareBisect/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+
+FlareBisect finds the commit that made your tests flaky, not just the one
+that broke them: it bisects on flake-rate drift instead of pass/fail, then
+flares the culprit with an AI-written root cause.
+
+`git bisect` gives you confidently wrong answers when the test you're
+bisecting on is flaky — it treats every run as a clean pass/fail signal, so
+a test that was already unstable *before* you started bisecting can point
+straight at an innocent commit.
+
+`flarebisect` bisects on **flake rate**, not pass/fail. It runs each
+candidate commit's test N times in parallel, computes a failure rate, and
+finds the commit where that rate jumps — then asks an LLM to explain, in
+plain English, whether the commit broke the test cleanly or made it flakier.
+
+```
+12 commits in range · 20 runs per commit · parallel worktrees
+
+commit   flake rate                                  result  status
+a1b2c3                                                  5/5  good
+4f9e21                                                  5/5  stable
+7d3aa0   ███████                                        4/5  wobbling
+9c1f88   ████████████████████████████                   1/5  flare
+HEAD     ████████████████████████████                   1/5  bad
+
+╭────────────────────────────────────────────────────────────────╮
+│ ✦ culprit found · commit 9c1f88                                    │
+│                                                                        │
+│ flake rate jumped 20% → 80% at this commit                             │
+│ "add request counter for rate limiting"                                │
+│                                                                        │
+│   💡 likely cause — shared counter incremented without a lock.        │
+│   concurrent test workers race on the same variable.                   │
+╰────────────────────────────────────────────────────────────────╯
+⏱ 8.4s    ⚡ 5 checked    ⚙ 4 parallel workers
+```
+
+## Install
+
+```bash
+pip install flarebisect
+```
+
+For local development:
+
+```bash
+pip install -e ".[dev]"
+pytest -q
+```
+
+Works on Linux, macOS, and Windows — the only external dependency is `git`
+on your `PATH`. Every candidate commit runs in its own `git worktree` under a
+temp directory; your actual working tree and index are never touched.
+
+## AI provider setup
+
+The root-cause explanation step works with Claude, OpenAI, Gemini, or a
+local model — anything speaking an OpenAI-compatible chat API (Ollama, LM
+Studio, llama.cpp server, vLLM, and similar all qualify).
+
+```bash
+# cloud
+flarebisect config set-key anthropic sk-ant-...
+flarebisect config set-key openai sk-...
+flarebisect config set-key google AI...
+
+# local, no key needed - just have Ollama running
+flarebisect config use ollama
+flarebisect config set-model ollama llama3.1
+
+# or let flarebisect pick a model sized to your GPU and pull it for you
+flarebisect models detect   # shows detected GPU/VRAM + recommended model
+flarebisect models pull     # downloads it and sets it active (run with no args)
+
+# any other OpenAI-compatible endpoint
+flarebisect config use custom
+flarebisect config set-base-url custom http://localhost:8080/v1
+flarebisect config set-key custom sk-local
+```
+
+Run `flarebisect config` with no arguments for a guided setup wizard — for
+Ollama it detects your GPU/VRAM, prefills a right-sized model, and offers to
+pull it on the spot.
+
+`flarebisect config show` lists the active provider and stored settings
+(keys are masked). Config lives in a JSON file under the OS config dir
+(`~/.config/flarebisect/config.json` on Linux/macOS, `%APPDATA%` on Windows),
+written with owner-only permissions.
+
+Per-run overrides skip the config file entirely:
+
+```bash
+flarebisect run ... --provider openai --model gpt-4o-mini --api-key sk-...
+flarebisect run ... --provider ollama --base-url http://localhost:11434/v1
+```
+
+Env vars (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GOOGLE_API_KEY`) work too,
+as a fallback under whatever's in the config file.
+
+## Usage
+
+```bash
+flarebisect run \
+  --repo /path/to/repo \
+  --good <known-good-sha> \
+  --bad <known-bad-sha> \
+  --test "pytest -k my_flaky_test" \
+  --runs 20 \
+  --threshold 0.3
+```
+
+- `--runs` — test executions per commit (default 20), run in parallel
+  (capped at your core count so it doesn't oversubscribe). Lower counts are
+  faster but noisier — a low sample can misattribute the culprit near the
+  threshold boundary.
+- `--threshold` — flake-rate jump (0-1) vs. the `good` baseline required to
+  call a commit the culprit (default 0.3).
+- `--no-explain` — skip the LLM root-cause call entirely (fully offline).
+
+## Demo
+
+See [`demo/`](demo/) for a self-contained, seeded-flaky-bug repo you can
+bisect against with no network access required for the bisection itself.
+
+## How it works
+
+1. Measure the flake rate at `good` and `bad` as baselines.
+2. Binary-search the commit range between them.
+3. At each candidate, run the test `N` times in an isolated worktree and
+   compute `failed / N`.
+4. The culprit is the first commit whose flake rate jumps by `>= threshold`
+   over the `good` baseline.
+5. Verdict: **clean break** (rate goes ~0% → ~100%) vs. **flakiness
+   regression** (rate goes ~0% → somewhere in between).
+6. The configured LLM gets the culprit's diff + the flake-rate evidence and
+   returns a short root-cause read (race condition, shared mutable state,
+   timing assumption, etc.).
+
+Binary search assumes the flake rate is roughly monotonic across the range,
+same as ordinary `git bisect` assumes pass/fail is. At low `--runs` counts
+a noisy sample can occasionally violate that near the threshold boundary —
+bump `--runs` if a result looks off.
+
+## Releasing
+
+Version is single-sourced from `src/flarebisect/__init__.py`. Bump it, commit,
+tag (`vX.Y.Z`), and cut a GitHub Release — `.github/workflows/publish.yml`
+builds and publishes to PyPI via trusted publishing (OIDC, no stored token).
+
+## License
+
+MIT

flarebisect-0.3.1/pyproject.toml

@@ -0,0 +1,68 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "flarebisect"
+dynamic = ["version"]
+description = "git bisect that treats flakiness as a signal, not noise"
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "MIT" }
+authors = [
+    { name = "Tawin Tangsukson" },
+]
+keywords = ["git", "bisect", "testing", "flaky-tests", "cli", "debugging"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Environment :: Console",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Software Development :: Debuggers",
+    "Topic :: Software Development :: Testing",
+    "Topic :: Software Development :: Version Control :: Git",
+]
+dependencies = [
+    "typer>=0.12",
+    "rich>=13.7",
+    "anthropic>=0.40",
+    "openai>=1.40",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=8.0",
+    "build>=1.2",
+    "twine>=5.0",
+]
+
+[project.urls]
+Homepage = "https://github.com/kaorii-ako/FlareBisect"
+Repository = "https://github.com/kaorii-ako/FlareBisect"
+Issues = "https://github.com/kaorii-ako/FlareBisect/issues"
+
+[project.scripts]
+flarebisect = "flarebisect.cli:app"
+
+[tool.hatch.version]
+path = "src/flarebisect/__init__.py"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/flarebisect"]
+
+[tool.hatch.build.targets.sdist]
+include = [
+    "/src",
+    "/tests",
+    "/README.md",
+    "/LICENSE",
+]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

flarebisect-0.3.1/src/flarebisect/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.3.1"

flarebisect-0.3.1/src/flarebisect/ai_explain.py

@@ -0,0 +1,37 @@
+"""Ask the configured LLM for a short root-cause read on the culprit commit."""
+
+from __future__ import annotations
+
+from .bisect import BisectOutcome
+from .providers import ProviderConfig, complete
+
+PROMPT_TEMPLATE = """A commit changed a test's failure rate. Diagnose the likely cause.
+
+commit {sha}: "{subject}"
+
+flake rate before: {before_rate:.0%}
+flake rate after:  {culprit_rate:.0%}
+bisection threshold: {threshold:.0%}
+verdict: {verdict}
+
+diff:
+```diff
+{diff}
+```
+
+Answer in exactly 1-2 short sentences, no preamble, no restating the numbers above.
+Point at the specific code change responsible (shared state, missing lock, timing/ordering
+assumption, resource leak, off-by-one, whatever it actually is)."""
+
+
+def explain(outcome: BisectOutcome, diff: str, threshold: float, provider_cfg: ProviderConfig) -> str:
+    prompt = PROMPT_TEMPLATE.format(
+        sha=outcome.culprit.sha[:12],
+        subject=outcome.culprit.subject,
+        before_rate=outcome.before.result.flake_rate,
+        culprit_rate=outcome.culprit.result.flake_rate,
+        threshold=threshold,
+        verdict=outcome.verdict,
+        diff=diff[:6000],
+    )
+    return complete(provider_cfg, prompt)

flarebisect-0.3.1/src/flarebisect/bisect.py

@@ -0,0 +1,145 @@
+"""Flake-rate bisection: binary search over commits for the point where the
+flake rate jumps by >= threshold relative to the known-good baseline."""
+
+from __future__ import annotations
+
+import time
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Callable
+
+from . import git_ops
+from .runner import RunResult, measure_flake_rate, worker_count_for
+
+
+@dataclass
+class CommitMeasurement:
+    sha: str
+    subject: str
+    result: RunResult
+    position: int  # -1 = good baseline, 0..n-1 = index within the good..bad range
+    status: str = ""  # filled in once the culprit is known: good/stable/wobbling/flare/bad
+
+
+@dataclass
+class BisectOutcome:
+    culprit: CommitMeasurement
+    before: CommitMeasurement  # last known-good-ish measurement (parent side)
+    good_baseline: RunResult
+    bad_baseline: RunResult
+    verdict: str  # "clean break" | "flakiness regression"
+    measurements: list[CommitMeasurement]  # every distinct commit measured, in range order
+    total_commits: int
+    runs: int
+    workers: int
+    elapsed_seconds: float
+
+
+def _measure(repo: Path, sha: str, test_cmd: str, runs: int, position: int) -> CommitMeasurement:
+    wt = git_ops.add_worktree(repo, sha)
+    try:
+        result = measure_flake_rate(test_cmd, wt.path, runs)
+    finally:
+        git_ops.remove_worktree(repo, wt)
+    return CommitMeasurement(
+        sha=sha,
+        subject=git_ops.commit_subject(repo, sha),
+        result=result,
+        position=position,
+    )
+
+
+def classify(good_rate: float, culprit_rate: float) -> str:
+    if culprit_rate >= 0.9 and good_rate <= 0.1:
+        return "clean break"
+    return "flakiness regression"
+
+
+def status_label(rate: float, is_first: bool, is_last: bool, is_culprit: bool) -> str:
+    if is_culprit:
+        return "flare"
+    if is_last:
+        return "bad"
+    if rate <= 0.0:
+        return "good" if is_first else "stable"
+    return "wobbling"
+
+
+def run_bisect(
+    repo: Path,
+    good: str,
+    bad: str,
+    test_cmd: str,
+    runs: int = 5,
+    threshold: float = 0.3,
+    on_measure: Callable[[str, CommitMeasurement], None] | None = None,
+) -> BisectOutcome:
+    started = time.perf_counter()
+
+    good_sha = git_ops.resolve_sha(repo, good)
+    bad_sha = git_ops.resolve_sha(repo, bad)
+
+    good_m = _measure(repo, good_sha, test_cmd, runs, position=-1)
+    if on_measure:
+        on_measure("good", good_m)
+
+    commits = git_ops.commit_list(repo, good_sha, bad_sha)
+    if not commits:
+        raise ValueError("no commits between good and bad")
+
+    baseline_rate = good_m.result.flake_rate
+
+    cache: dict[str, CommitMeasurement] = {}
+
+    def measure_idx(idx: int) -> CommitMeasurement:
+        sha = commits[idx]
+        if sha not in cache:
+            m = _measure(repo, sha, test_cmd, runs, position=idx)
+            cache[sha] = m
+            if on_measure:
+                on_measure(f"candidate[{idx}]", m)
+        return cache[sha]
+
+    bad_m = measure_idx(len(commits) - 1)
+
+    lo, hi = 0, len(commits) - 1
+    culprit_idx = hi  # fall back to bad commit if search never narrows
+    culprit_measurement = bad_m
+
+    while lo <= hi:
+        mid = (lo + hi) // 2
+        m = measure_idx(mid)
+        jumped = (m.result.flake_rate - baseline_rate) >= threshold
+        if jumped:
+            culprit_idx = mid
+            culprit_measurement = m
+            hi = mid - 1
+        else:
+            lo = mid + 1
+
+    before_measurement = measure_idx(culprit_idx - 1) if culprit_idx > 0 else good_m
+
+    verdict = classify(before_measurement.result.flake_rate, culprit_measurement.result.flake_rate)
+
+    last_idx = len(commits) - 1
+    ordered = [good_m] + [cache[sha] for sha in commits if sha in cache]
+    for m in ordered:
+        m.status = status_label(
+            m.result.flake_rate,
+            is_first=(m.position == -1),
+            is_last=(m.position == last_idx),
+            is_culprit=(m.sha == culprit_measurement.sha),
+        )
+
+    return BisectOutcome(
+        culprit=culprit_measurement,
+        before=before_measurement,
+        good_baseline=good_m.result,
+        bad_baseline=bad_m.result,
+        verdict=verdict,
+        measurements=ordered,
+        total_commits=len(commits),
+        runs=runs,
+        workers=worker_count_for(runs),
+        elapsed_seconds=time.perf_counter() - started,
+    )