flakestat 0.1.0__tar.gz

flakestat-0.1.0/PKG-INFO

@@ -0,0 +1,95 @@
+Metadata-Version: 2.4
+Name: flakestat
+Version: 0.1.0
+Summary: Find flaky tests in any language - pytest, Jest, go test, JUnit. Local CLI, no SaaS, no account.
+Author: flakestat contributors
+License: MIT
+Project-URL: Homepage, https://github.com/rowhitswami/flakestat
+Project-URL: Issues, https://github.com/rowhitswami/flakestat/issues
+Project-URL: Source, https://github.com/rowhitswami/flakestat
+Keywords: flaky,flaky-tests,flaky-test-detection,test-flakiness,flake-detection,test-stability,pytest,testing,junit,ci,quarantine
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Software Development :: Testing
+Classifier: Topic :: Software Development :: Quality Assurance
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+
+# flakestat
+
+Find flaky tests in any language. No SaaS, no account, no data leaving your machine.
+
+A flaky test passes and fails randomly **on the same code**. flakestat tells you
+exactly which ones, ranked worst first.
+
+```bash
+flakestat hunt --runs 20 \
+  --junit 'reports/junit-{run}.xml' \
+  -- pytest -q --junitxml='reports/junit-{run}.xml'
+```
+
+```
+VERDICT               SCORE  RUNS  PASS/FAIL  TEST
+flaky                  0.62    20       14/6  tests.test_auth::test_refresh_token
+consistently-failing   0.00    20       0/20  tests.test_billing::test_discount
+
+1 flaky, 0 suspect, 1 consistently failing, 1 stable, 1 unscored (of 4 tests)
+```
+
+## Install
+
+```bash
+pip install flakestat
+```
+
+This package downloads the prebuilt `flakestat` binary for your platform
+(macOS, Linux, Windows on x86_64/arm64) from GitHub Releases on first use. It has
+no Python dependencies, so it cannot interfere with your project's resolution.
+
+## Why the verdicts matter
+
+**Flakiness is inconsistency, not failure.** A test that fails 100% of the time
+scores 1.0 on failure rate but isn't flaky — it's broken. flakestat scores
+*state transitions* (pass→fail→pass), so an always-failing test correctly scores
+zero and is reported separately as `consistently-failing` instead of being mixed
+in with real flakes.
+
+Skipped tests are excluded rather than treated as passes, so `@pytest.mark.skip`
+never manufactures a phantom transition.
+
+## How it compares
+
+`pytest-rerunfailures` and `pytest-flakefinder` retry tests so a run goes green.
+That hides the symptom. flakestat *measures* which tests are unreliable and by
+how much, so they can be fixed — and it works across every suite in a polyglot
+repo, not just the Python one.
+
+## Tracking over time
+
+A burst proves flakiness exists. History *measures* it, and catches
+environment-dependent flakes a local burst never will.
+
+```bash
+flakestat ingest 'reports/**/*.xml'   # in CI, after tests
+flakestat report --top 20
+flakestat report --fail-on-flaky      # exit 1 to gate a pipeline
+```
+
+History lives in `.flakestat/runs.ndjson` — one JSON object per line. Commit it
+for shared team history, or keep it as a CI artifact.
+
+## Environment variables
+
+| Variable | Effect |
+|---|---|
+| `FLAKESTAT_BINARY` | Use this binary instead of downloading |
+| `FLAKESTAT_SKIP_DOWNLOAD` | Fail instead of downloading |
+| `FLAKESTAT_CACHE_DIR` | Where the binary is cached |
+
+## Full documentation
+
+<https://github.com/rowhitswami/flakestat>
+
+MIT

flakestat-0.1.0/README.md

@@ -0,0 +1,76 @@
+# flakestat
+
+Find flaky tests in any language. No SaaS, no account, no data leaving your machine.
+
+A flaky test passes and fails randomly **on the same code**. flakestat tells you
+exactly which ones, ranked worst first.
+
+```bash
+flakestat hunt --runs 20 \
+  --junit 'reports/junit-{run}.xml' \
+  -- pytest -q --junitxml='reports/junit-{run}.xml'
+```
+
+```
+VERDICT               SCORE  RUNS  PASS/FAIL  TEST
+flaky                  0.62    20       14/6  tests.test_auth::test_refresh_token
+consistently-failing   0.00    20       0/20  tests.test_billing::test_discount
+
+1 flaky, 0 suspect, 1 consistently failing, 1 stable, 1 unscored (of 4 tests)
+```
+
+## Install
+
+```bash
+pip install flakestat
+```
+
+This package downloads the prebuilt `flakestat` binary for your platform
+(macOS, Linux, Windows on x86_64/arm64) from GitHub Releases on first use. It has
+no Python dependencies, so it cannot interfere with your project's resolution.
+
+## Why the verdicts matter
+
+**Flakiness is inconsistency, not failure.** A test that fails 100% of the time
+scores 1.0 on failure rate but isn't flaky — it's broken. flakestat scores
+*state transitions* (pass→fail→pass), so an always-failing test correctly scores
+zero and is reported separately as `consistently-failing` instead of being mixed
+in with real flakes.
+
+Skipped tests are excluded rather than treated as passes, so `@pytest.mark.skip`
+never manufactures a phantom transition.
+
+## How it compares
+
+`pytest-rerunfailures` and `pytest-flakefinder` retry tests so a run goes green.
+That hides the symptom. flakestat *measures* which tests are unreliable and by
+how much, so they can be fixed — and it works across every suite in a polyglot
+repo, not just the Python one.
+
+## Tracking over time
+
+A burst proves flakiness exists. History *measures* it, and catches
+environment-dependent flakes a local burst never will.
+
+```bash
+flakestat ingest 'reports/**/*.xml'   # in CI, after tests
+flakestat report --top 20
+flakestat report --fail-on-flaky      # exit 1 to gate a pipeline
+```
+
+History lives in `.flakestat/runs.ndjson` — one JSON object per line. Commit it
+for shared team history, or keep it as a CI artifact.
+
+## Environment variables
+
+| Variable | Effect |
+|---|---|
+| `FLAKESTAT_BINARY` | Use this binary instead of downloading |
+| `FLAKESTAT_SKIP_DOWNLOAD` | Fail instead of downloading |
+| `FLAKESTAT_CACHE_DIR` | Where the binary is cached |
+
+## Full documentation
+
+<https://github.com/rowhitswami/flakestat>
+
+MIT

flakestat-0.1.0/flakestat/__init__.py

@@ -0,0 +1,37 @@
+"""flakestat - find flaky tests in any language.
+
+This package is a thin wrapper that fetches and runs the flakestat binary.
+The tool itself is written in Go; see https://github.com/rowhitswami/flakestat.
+"""
+
+import subprocess
+import sys
+
+from ._binary import InstallError, ensure_binary
+
+# Kept in step with the released binary by CI at publish time.
+__version__ = "0.0.0"
+
+__all__ = ["main", "ensure_binary", "InstallError", "__version__"]
+
+
+def main() -> int:
+    """Console-script entry point: exec the binary, preserving the exit code."""
+    try:
+        binary = ensure_binary()
+    except InstallError as exc:
+        print(str(exc), file=sys.stderr)
+        return 1
+
+    try:
+        # CI gates depend on the exit code, so pass it straight through.
+        return subprocess.call([str(binary)] + sys.argv[1:])
+    except KeyboardInterrupt:
+        return 130
+    except OSError as exc:
+        print("flakestat: {}".format(exc), file=sys.stderr)
+        return 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())

flakestat-0.1.0/flakestat/__main__.py

@@ -0,0 +1,8 @@
+"""Support `python -m flakestat` alongside the console script."""
+
+import sys
+
+from . import main
+
+if __name__ == "__main__":
+    sys.exit(main())

flakestat-0.1.0/flakestat/_binary.py

@@ -0,0 +1,184 @@
+"""Resolve the flakestat binary, downloading it on first use.
+
+pip has no post-install hook, and building a Go binary at install time would
+require a Go toolchain on the user's machine. So the binary is fetched from
+GitHub Releases the first time it is needed and cached per version.
+
+Everything here is stdlib: the wrapper must never interfere with a user's
+dependency resolution.
+"""
+
+import hashlib
+import os
+import platform
+import shutil
+import stat
+import sys
+import tarfile
+import tempfile
+import urllib.error
+import urllib.request
+import zipfile
+from pathlib import Path
+
+OWNER = "rowhitswami"
+REPO = "flakestat"
+
+
+class InstallError(RuntimeError):
+    """Raised when the binary cannot be obtained."""
+
+
+def _version() -> str:
+    from . import __version__
+
+    return __version__
+
+
+def _target():
+    """Map this interpreter's platform onto the release archive naming."""
+    system = platform.system().lower()
+    goos = {"darwin": "darwin", "linux": "linux", "windows": "windows"}.get(system)
+
+    machine = platform.machine().lower()
+    goarch = {
+        "x86_64": "amd64",
+        "amd64": "amd64",
+        "aarch64": "arm64",
+        "arm64": "arm64",
+    }.get(machine)
+
+    if not goos or not goarch:
+        raise InstallError(
+            "flakestat: unsupported platform {}/{}.\n"
+            "Prebuilt binaries cover macOS, Linux and Windows on x86_64 and arm64.\n"
+            "Build from source instead:\n"
+            "  go install github.com/{}/{}/cmd/{}@latest".format(
+                system, machine, OWNER, REPO, REPO
+            )
+        )
+
+    return goos, goarch
+
+
+def _cache_dir() -> Path:
+    """Per-version cache, so upgrading the wrapper fetches a matching binary."""
+    if os.environ.get("FLAKESTAT_CACHE_DIR"):
+        base = Path(os.environ["FLAKESTAT_CACHE_DIR"])
+    elif sys.platform == "win32":
+        base = Path(os.environ.get("LOCALAPPDATA", Path.home() / "AppData" / "Local"))
+    elif sys.platform == "darwin":
+        base = Path.home() / "Library" / "Caches"
+    else:
+        base = Path(os.environ.get("XDG_CACHE_HOME", Path.home() / ".cache"))
+
+    return base / "flakestat" / _version()
+
+
+def _binary_name() -> str:
+    return "flakestat.exe" if sys.platform == "win32" else "flakestat"
+
+
+def _fetch(url: str) -> bytes:
+    try:
+        with urllib.request.urlopen(url) as resp:  # noqa: S310 - fixed https host
+            return resp.read()
+    except urllib.error.HTTPError as exc:
+        raise InstallError("flakestat: GET {} -> {}".format(url, exc.code)) from exc
+    except urllib.error.URLError as exc:
+        raise InstallError(
+            "flakestat: could not reach {} ({})".format(url, exc.reason)
+        ) from exc
+
+
+def _verify(archive_name: str, blob: bytes, base_url: str) -> None:
+    try:
+        sums = _fetch("{}/checksums.txt".format(base_url)).decode("utf-8")
+    except InstallError:
+        print("flakestat: checksums.txt unavailable, skipping verification", file=sys.stderr)
+        return
+
+    expected = None
+    for line in sums.splitlines():
+        if line.strip().endswith(archive_name):
+            expected = line.split()[0]
+            break
+
+    if expected is None:
+        return
+
+    actual = hashlib.sha256(blob).hexdigest()
+    if expected != actual:
+        raise InstallError(
+            "flakestat: checksum mismatch for {}\n  expected: {}\n  actual:   {}".format(
+                archive_name, expected, actual
+            )
+        )
+
+
+def _download(dest: Path) -> None:
+    goos, goarch = _target()
+    version = _version()
+    ext = "zip" if goos == "windows" else "tar.gz"
+
+    archive = "{}_{}_{}_{}.{}".format(REPO, version, goos, goarch, ext)
+    base_url = "https://github.com/{}/{}/releases/download/v{}".format(OWNER, REPO, version)
+
+    print(
+        "flakestat: downloading {} v{} ({}/{})".format(REPO, version, goos, goarch),
+        file=sys.stderr,
+    )
+
+    blob = _fetch("{}/{}".format(base_url, archive))
+    _verify(archive, blob, base_url)
+
+    name = _binary_name()
+
+    with tempfile.TemporaryDirectory() as tmp:
+        tmp_path = Path(tmp)
+        archive_path = tmp_path / archive
+        archive_path.write_bytes(blob)
+
+        if ext == "zip":
+            with zipfile.ZipFile(archive_path) as zf:
+                zf.extract(name, tmp_path)
+        else:
+            with tarfile.open(archive_path) as tf:
+                member = tf.getmember(name)
+                # Refuse absolute paths and traversal in archive members.
+                if member.name != name or Path(member.name).is_absolute():
+                    raise InstallError("flakestat: unexpected archive member {}".format(member.name))
+                tf.extract(member, tmp_path)
+
+        dest.parent.mkdir(parents=True, exist_ok=True)
+
+        # Move into place via a temporary name so concurrent installs cannot
+        # observe a half-written binary.
+        staged = dest.parent / (dest.name + ".tmp")
+        shutil.move(str(tmp_path / name), str(staged))
+        staged.chmod(staged.stat().st_mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH)
+        os.replace(str(staged), str(dest))
+
+    print("flakestat: installed to {}".format(dest), file=sys.stderr)
+
+
+def ensure_binary() -> Path:
+    """Return the path to the binary, downloading it if needed."""
+    override = os.environ.get("FLAKESTAT_BINARY")
+    if override:
+        path = Path(override)
+        if not path.exists():
+            raise InstallError("flakestat: FLAKESTAT_BINARY={} does not exist".format(override))
+        return path
+
+    dest = _cache_dir() / _binary_name()
+    if dest.exists():
+        return dest
+
+    if os.environ.get("FLAKESTAT_SKIP_DOWNLOAD"):
+        raise InstallError(
+            "flakestat: binary missing and FLAKESTAT_SKIP_DOWNLOAD is set"
+        )
+
+    _download(dest)
+    return dest

flakestat-0.1.0/flakestat.egg-info/PKG-INFO

@@ -0,0 +1,95 @@
+Metadata-Version: 2.4
+Name: flakestat
+Version: 0.1.0
+Summary: Find flaky tests in any language - pytest, Jest, go test, JUnit. Local CLI, no SaaS, no account.
+Author: flakestat contributors
+License: MIT
+Project-URL: Homepage, https://github.com/rowhitswami/flakestat
+Project-URL: Issues, https://github.com/rowhitswami/flakestat/issues
+Project-URL: Source, https://github.com/rowhitswami/flakestat
+Keywords: flaky,flaky-tests,flaky-test-detection,test-flakiness,flake-detection,test-stability,pytest,testing,junit,ci,quarantine
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Software Development :: Testing
+Classifier: Topic :: Software Development :: Quality Assurance
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+
+# flakestat
+
+Find flaky tests in any language. No SaaS, no account, no data leaving your machine.
+
+A flaky test passes and fails randomly **on the same code**. flakestat tells you
+exactly which ones, ranked worst first.
+
+```bash
+flakestat hunt --runs 20 \
+  --junit 'reports/junit-{run}.xml' \
+  -- pytest -q --junitxml='reports/junit-{run}.xml'
+```
+
+```
+VERDICT               SCORE  RUNS  PASS/FAIL  TEST
+flaky                  0.62    20       14/6  tests.test_auth::test_refresh_token
+consistently-failing   0.00    20       0/20  tests.test_billing::test_discount
+
+1 flaky, 0 suspect, 1 consistently failing, 1 stable, 1 unscored (of 4 tests)
+```
+
+## Install
+
+```bash
+pip install flakestat
+```
+
+This package downloads the prebuilt `flakestat` binary for your platform
+(macOS, Linux, Windows on x86_64/arm64) from GitHub Releases on first use. It has
+no Python dependencies, so it cannot interfere with your project's resolution.
+
+## Why the verdicts matter
+
+**Flakiness is inconsistency, not failure.** A test that fails 100% of the time
+scores 1.0 on failure rate but isn't flaky — it's broken. flakestat scores
+*state transitions* (pass→fail→pass), so an always-failing test correctly scores
+zero and is reported separately as `consistently-failing` instead of being mixed
+in with real flakes.
+
+Skipped tests are excluded rather than treated as passes, so `@pytest.mark.skip`
+never manufactures a phantom transition.
+
+## How it compares
+
+`pytest-rerunfailures` and `pytest-flakefinder` retry tests so a run goes green.
+That hides the symptom. flakestat *measures* which tests are unreliable and by
+how much, so they can be fixed — and it works across every suite in a polyglot
+repo, not just the Python one.
+
+## Tracking over time
+
+A burst proves flakiness exists. History *measures* it, and catches
+environment-dependent flakes a local burst never will.
+
+```bash
+flakestat ingest 'reports/**/*.xml'   # in CI, after tests
+flakestat report --top 20
+flakestat report --fail-on-flaky      # exit 1 to gate a pipeline
+```
+
+History lives in `.flakestat/runs.ndjson` — one JSON object per line. Commit it
+for shared team history, or keep it as a CI artifact.
+
+## Environment variables
+
+| Variable | Effect |
+|---|---|
+| `FLAKESTAT_BINARY` | Use this binary instead of downloading |
+| `FLAKESTAT_SKIP_DOWNLOAD` | Fail instead of downloading |
+| `FLAKESTAT_CACHE_DIR` | Where the binary is cached |
+
+## Full documentation
+
+<https://github.com/rowhitswami/flakestat>
+
+MIT

flakestat-0.1.0/flakestat.egg-info/SOURCES.txt

@@ -0,0 +1,10 @@
+README.md
+pyproject.toml
+flakestat/__init__.py
+flakestat/__main__.py
+flakestat/_binary.py
+flakestat.egg-info/PKG-INFO
+flakestat.egg-info/SOURCES.txt
+flakestat.egg-info/dependency_links.txt
+flakestat.egg-info/entry_points.txt
+flakestat.egg-info/top_level.txt

flakestat-0.1.0/flakestat.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

flakestat-0.1.0/flakestat.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+flakestat = flakestat:main

flakestat-0.1.0/flakestat.egg-info/top_level.txt

@@ -0,0 +1 @@
+flakestat

flakestat-0.1.0/pyproject.toml

@@ -0,0 +1,47 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "flakestat"
+version = "0.1.0"
+description = "Find flaky tests in any language - pytest, Jest, go test, JUnit. Local CLI, no SaaS, no account."
+readme = "README.md"
+requires-python = ">=3.8"
+license = { text = "MIT" }
+authors = [{ name = "flakestat contributors" }]
+keywords = [
+    "flaky",
+    "flaky-tests",
+    "flaky-test-detection",
+    "test-flakiness",
+    "flake-detection",
+    "test-stability",
+    "pytest",
+    "testing",
+    "junit",
+    "ci",
+    "quarantine",
+]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Topic :: Software Development :: Testing",
+    "Topic :: Software Development :: Quality Assurance",
+]
+# No runtime dependencies on purpose: everything used here is stdlib, so the
+# wrapper cannot break a user's dependency resolution.
+dependencies = []
+
+[project.urls]
+Homepage = "https://github.com/rowhitswami/flakestat"
+Issues = "https://github.com/rowhitswami/flakestat/issues"
+Source = "https://github.com/rowhitswami/flakestat"
+
+[project.scripts]
+flakestat = "flakestat:main"
+
+[tool.setuptools]
+packages = ["flakestat"]

flakestat-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+