flakeguard 0.1.0__tar.gz

flakeguard-0.1.0/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,28 @@
+---
+name: Bug report
+about: Something isn't working as expected
+title: ""
+labels: bug
+assignees: ""
+---
+
+**What did you run?**
+The exact command, please.
+
+```
+flakeguard scan ...
+```
+
+**What happened?**
+
+**What did you expect to happen?**
+
+**A snippet of the report file**
+If you can share a (redacted) piece of the report that caused it, that helps
+enormously.
+
+**Environment**
+
+- flakeguard version (`flakeguard --version`):
+- Python version:
+- OS:

flakeguard-0.1.0/.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,5 @@
+blank_issues_enabled: true
+contact_links:
+  - name: Question or idea
+    url: https://github.com/PrathameshKulkarni2k4/flakeguard/discussions
+    about: Ask questions and discuss ideas with the community.

flakeguard-0.1.0/.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,14 @@
+---
+name: Feature request
+about: Suggest an idea for flakeguard
+title: ""
+labels: enhancement
+assignees: ""
+---
+
+**What problem are you trying to solve?**
+Describe the pain, not just the feature. It helps us find the best fix.
+
+**What would you like to happen?**
+
+**Anything else?**

flakeguard-0.1.0/.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,14 @@
+## What does this change?
+
+<!-- A short description of what and why. -->
+
+## Related issue
+
+<!-- e.g. Closes #12 -->
+
+## Checklist
+
+- [ ] I added or updated tests for this change.
+- [ ] `pytest -q` passes locally.
+- [ ] I updated the docs / CHANGELOG if behaviour changed.
+- [ ] The change keeps the flakiness score explainable (no black box).

flakeguard-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,24 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.9", "3.11", "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install
+        run: pip install -e ".[dev]"
+      - name: Test
+        run: pytest -q
+      - name: Self-check (dogfood)
+        run: flakeguard scan examples/reports

flakeguard-0.1.0/.github/workflows/example.yml

@@ -0,0 +1,40 @@
+# Example: run Flakeguard on every pull request.
+#
+# Copy this into your own repository as .github/workflows/flakeguard.yml,
+# then adjust the "Generate reports" step to run *your* test suite a few
+# times and collect JUnit XML into a directory.
+
+name: Flaky test check
+
+on:
+  pull_request:
+
+jobs:
+  flakeguard:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      pull-requests: write # needed to post the report as a PR comment
+    steps:
+      - uses: actions/checkout@v4
+
+      # 1. Produce test reports. Replace this with your real test command.
+      #    Running the suite several times is what makes flakiness visible.
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Generate reports
+        run: |
+          pip install pytest
+          mkdir -p reports
+          for i in 1 2 3 4 5; do
+            pytest --junitxml="reports/run-$i.xml" || true
+          done
+
+      # 2. Analyse them and comment on the PR.
+      - name: Flakeguard
+        uses: ./
+        with:
+          reports: reports
+          min-runs: 3
+          max-flaky: 0 # fail the check if any flaky test is found

flakeguard-0.1.0/.github/workflows/release.yml

@@ -0,0 +1,47 @@
+# Publish to PyPI when a version tag is pushed.
+#
+# Setup (one time, done by the maintainer):
+#   1. Create the project on PyPI.
+#   2. Add a "Trusted Publisher" on PyPI pointing at this repository and the
+#      workflow file name "release.yml" (PyPI > project > Publishing).
+#   3. Push a tag:  git tag v0.1.0 && git push origin v0.1.0
+#
+# No API token is stored anywhere - PyPI trusts this workflow directly (OIDC).
+
+name: Release
+
+on:
+  push:
+    tags:
+      - "v*"
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Build
+        run: |
+          python -m pip install --upgrade build
+          python -m build
+      - uses: actions/upload-artifact@v4
+        with:
+          name: dist
+          path: dist/
+
+  publish:
+    needs: build
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write # required for trusted publishing
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          name: dist
+          path: dist/
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

flakeguard-0.1.0/.gitignore

@@ -0,0 +1,23 @@
+# Byte-compiled / cache
+__pycache__/
+*.py[cod]
+*.egg-info/
+.eggs/
+build/
+dist/
+
+# Virtual envs
+.venv/
+venv/
+
+# Test / tooling caches
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.coverage
+htmlcov/
+
+# Editors / OS
+.idea/
+.vscode/
+.DS_Store

flakeguard-0.1.0/CHANGELOG.md

@@ -0,0 +1,34 @@
+# Changelog
+
+All notable changes to this project are documented here. The format is based
+on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project
+aims to follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+### Fixed
+
+- Test identity when a framework repeats the name in both `classname` and
+  `name` (e.g. jest-junit), which could otherwise merge distinct tests.
+- Clearer CLI errors when report files are missing or not valid JUnit XML.
+
+### Changed
+
+- Hardened the parser against real reports from pytest and jest-junit, with
+  regression fixtures captured from those tools.
+
+## [0.1.0] - 2026-10-18
+
+### Added
+
+- JUnit XML parser, so one tool serves every language.
+- Explainable flakiness scoring:
+  `2 * min(passes, failures) / (passes + failures)`.
+- CLI: `flakeguard scan` with `text`, `json`, and `markdown` output.
+- `--min-runs`, `--min-score`, `--output`, and `--fail-on-flaky` options.
+- GitHub Action: analyses reports, comments on pull requests, and enforces a
+  flakiness budget.
+- Test suite, MIT license, and CI workflow.
+
+[Unreleased]: https://github.com/PrathameshKulkarni2k4/flakeguard/compare/v0.1.0...HEAD
+[0.1.0]: https://github.com/PrathameshKulkarni2k4/flakeguard/releases/tag/v0.1.0

flakeguard-0.1.0/CODE_OF_CONDUCT.md

@@ -0,0 +1,58 @@
+# Contributor Covenant Code of Conduct
+
+## Our pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics,
+gender identity and expression, level of experience, education,
+socio-economic status, nationality, personal appearance, race, religion, or
+sexual identity and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive, and healthy community.
+
+## Our standards
+
+Examples of behaviour that contributes to a positive environment:
+
+- Demonstrating empathy and kindness toward other people.
+- Being respectful of differing opinions, viewpoints, and experiences.
+- Giving and gracefully accepting constructive feedback.
+- Accepting responsibility and apologising to those affected by our mistakes,
+  and learning from the experience.
+- Focusing on what is best for the community.
+
+Examples of unacceptable behaviour:
+
+- The use of sexualised language or imagery, and sexual attention or advances
+  of any kind.
+- Trolling, insulting or derogatory comments, and personal or political
+  attacks.
+- Public or private harassment.
+- Publishing others' private information without their explicit permission.
+- Other conduct which could reasonably be considered inappropriate in a
+  professional setting.
+
+## Enforcement responsibilities
+
+Project maintainers are responsible for clarifying and enforcing these
+standards and will take appropriate and fair corrective action in response to
+any behaviour they deem inappropriate, threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies
+when an individual is officially representing the community in public spaces.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behaviour may be
+reported to the maintainer at prathameshkulkarni22092004@gmail.com. All
+complaints will be reviewed and investigated promptly and fairly.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor
+Covenant](https://www.contributor-covenant.org), version 2.1, available at
+https://www.contributor-covenant.org/version/2/1/code_of_conduct.html.

flakeguard-0.1.0/CONTRIBUTING.md

@@ -0,0 +1,80 @@
+# Contributing to flakeguard
+
+Thanks for wanting to help — genuinely. This project exists for the
+developer community, and it only gets better with more hands on it.
+
+## The quickest way to help
+
+If you are new to open source, the best first contribution is **adding
+support for a new test-report format**. The parsers are deliberately small
+and self-contained, and there is a clear pattern to follow (see below).
+
+Look for issues labelled **`good first issue`**.
+
+## Setting up a development environment
+
+```bash
+git clone https://github.com/PrathameshKulkarni2k4/flakeguard
+cd flakeguard
+python -m venv .venv
+source .venv/bin/activate          # Windows: .venv\Scripts\activate
+pip install -e ".[dev]"
+pytest -q
+```
+
+You should see all tests pass. If they don't, that's a bug — please open an
+issue.
+
+## Running the tool on the examples
+
+```bash
+flakeguard scan examples/reports
+flakeguard scan examples/reports --format json
+flakeguard scan examples/reports --fail-on-flaky
+```
+
+## Adding a new report format
+
+1. Create `src/flakeguard/parsers/<format>.py` with a function that takes a
+   file path and returns a list of `flakeguard.models.TestCase`.
+2. Register it in `src/flakeguard/parsers/__init__.py` under `by_suffix`.
+3. Add a fixture file under `examples/reports/` and a test under `tests/`.
+
+That's the whole job. A good parser is small and boring, which is exactly
+what we want.
+
+## Guidelines
+
+- **Keep the score explainable.** Never replace the flakiness formula with a
+  black box. Trust is the product.
+- **Never be confidently wrong.** If a parser is unsure, it should error
+  clearly rather than guess.
+- **Small pull requests.** One change per PR is easier to review and merge.
+- **Add a test** for anything you change. Tests are how we keep the promise
+  that the tool is trustworthy.
+- **Match the existing style.** No configs to fight — just read the file you
+  are editing.
+
+## Commit messages
+
+A short imperative summary is enough:
+
+```
+Add TAP report parser
+Fix crash on reports with no <testsuite>
+```
+
+## Reporting bugs
+
+Please include:
+
+- what you ran (the exact command),
+- what happened,
+- what you expected,
+- a snippet of the report file if you can share it (redact anything
+  sensitive).
+
+## Code of Conduct
+
+By taking part, you agree to follow our
+[Code of Conduct](CODE_OF_CONDUCT.md). Be kind — we were all beginners once.

flakeguard-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Prathamesh Kulkarni
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

flakeguard-0.1.0/PKG-INFO

@@ -0,0 +1,141 @@
+Metadata-Version: 2.5
+Name: flakeguard
+Version: 0.1.0
+Summary: Find, rank, and quarantine flaky tests from any language's test reports.
+Project-URL: Homepage, https://github.com/PrathameshKulkarni2k4/flakeguard
+Author: Prathamesh Kulkarni
+License: MIT
+License-File: LICENSE
+Keywords: ci,developer-tools,flaky-tests,junit,test-reports,testing
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.9
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# flakeguard
+
+### Your test suite is lying to you.
+
+![flakeguard demo](docs/demo.gif)
+
+Flakeguard finds the tests that **randomly pass and fail**, ranks them by how
+unreliable they are, and shows you why — so you can trust your tests again.
+
+**Free · private · works with any language.** No account, no sign-up, and
+nothing leaves your machine.
+
+[![CI](https://github.com/PrathameshKulkarni2k4/flakeguard/actions/workflows/ci.yml/badge.svg)](https://github.com/PrathameshKulkarni2k4/flakeguard/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+[![Python 3.9+](https://img.shields.io/badge/python-3.9%2B-blue.svg)](https://www.python.org/)
+
+---
+
+## The problem, in 30 seconds
+
+You have a test that fails once in a while. You re-run it. It passes. Was
+that a real bug? A flaky test? Nobody knows — so everyone ignores it, and
+that is exactly how real bugs hide.
+
+A flaky test costs a team hours, and slowly destroys trust in the entire
+test suite. Flakeguard fixes the hard half of the problem: it tells you
+**exactly which tests are unreliable**, and how unreliable.
+
+## Try it in 60 seconds
+
+```bash
+pipx install flakeguard
+flakeguard scan ./reports        # a folder of test reports
+```
+
+No reports yet? Run it against the examples bundled in this repo:
+
+```bash
+git clone https://github.com/PrathameshKulkarni2k4/flakeguard
+cd flakeguard
+flakeguard scan examples/reports
+```
+
+## What it does
+
+- Reads your test results from **many runs** (the more runs, the sharper the
+  picture).
+- Finds the tests that **disagree with themselves** across those runs.
+- Gives each one a score from **0 (solid) to 1 (a coin-flip)**.
+- Shows the **latest failure message**, so you can go fix it.
+- Runs in your **CI** and can leave a comment on the pull request — or block
+  it when flakiness spikes.
+
+## Works with any language
+
+Flakeguard reads **JUnit XML**, the report format almost every test
+framework can produce. One tool, every language:
+
+| Language / framework | How to produce the report |
+|---|---|
+| Python (pytest) | `pytest --junitxml=report.xml` |
+| Java (Maven / Gradle) | Surefire writes JUnit XML automatically |
+| Go | `gotestsum --junitfile report.xml` |
+| JavaScript (Jest) | the `jest-junit` reporter |
+| Ruby (RSpec) | the `rspec_junit_formatter` gem |
+| .NET, PHPUnit, … | built-in or plugin JUnit output |
+
+## Use in CI (GitHub Actions)
+
+```yaml
+- name: Flakeguard
+  uses: PrathameshKulkarni2k4/flakeguard@main
+  with:
+    reports: reports      # dir/glob of JUnit XML files
+    min-runs: 3           # ignore tests seen in fewer runs
+    max-flaky: 0          # fail if any flaky test is found (-1 = never fail)
+    comment: "true"       # post the report as a PR comment
+```
+
+See [`.github/workflows/example.yml`](.github/workflows/example.yml) for a
+complete workflow that runs a suite several times and feeds the reports in.
+
+To gate locally or in any other CI, use the exit code:
+
+```bash
+flakeguard scan ./reports --fail-on-flaky   # exits 1 if anything is flaky
+```
+
+## How the score works
+
+For each test we count passes vs. non-passes across all runs (skips are
+ignored). The flakiness score is:
+
+```
+score = 2 * min(passes, failures) / (passes + failures)
+```
+
+- always passes, or always fails → **0.0** (consistent, *not* flaky)
+- flips on every run (50/50) → **1.0** (maximally flaky)
+
+It is deliberately explainable — no black box. You can read the number and
+know exactly how it was derived.
+
+## Roadmap
+
+- [x] JUnit XML parsing, flakiness scoring, text/JSON/Markdown reports
+- [x] GitHub Action with PR comments and a flakiness budget
+- [ ] TAP and native JSON report formats
+- [ ] CI history so it learns across builds
+- [ ] Failure-signature clustering (group the *same* flake across tests)
+- [ ] Quarantine file so known flakes are set aside, not deleted
+
+## Contributing
+
+Contributions are very welcome — see [CONTRIBUTING.md](CONTRIBUTING.md).
+Adding support for a new report format is a great first contribution and is
+designed to be small. Please be kind; this project follows a
+[Code of Conduct](CODE_OF_CONDUCT.md).
+
+## License
+
+MIT — see [LICENSE](LICENSE).

flakeguard-0.1.0/README.md

@@ -0,0 +1,122 @@
+# flakeguard
+
+### Your test suite is lying to you.
+
+![flakeguard demo](docs/demo.gif)
+
+Flakeguard finds the tests that **randomly pass and fail**, ranks them by how
+unreliable they are, and shows you why — so you can trust your tests again.
+
+**Free · private · works with any language.** No account, no sign-up, and
+nothing leaves your machine.
+
+[![CI](https://github.com/PrathameshKulkarni2k4/flakeguard/actions/workflows/ci.yml/badge.svg)](https://github.com/PrathameshKulkarni2k4/flakeguard/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+[![Python 3.9+](https://img.shields.io/badge/python-3.9%2B-blue.svg)](https://www.python.org/)
+
+---
+
+## The problem, in 30 seconds
+
+You have a test that fails once in a while. You re-run it. It passes. Was
+that a real bug? A flaky test? Nobody knows — so everyone ignores it, and
+that is exactly how real bugs hide.
+
+A flaky test costs a team hours, and slowly destroys trust in the entire
+test suite. Flakeguard fixes the hard half of the problem: it tells you
+**exactly which tests are unreliable**, and how unreliable.
+
+## Try it in 60 seconds
+
+```bash
+pipx install flakeguard
+flakeguard scan ./reports        # a folder of test reports
+```
+
+No reports yet? Run it against the examples bundled in this repo:
+
+```bash
+git clone https://github.com/PrathameshKulkarni2k4/flakeguard
+cd flakeguard
+flakeguard scan examples/reports
+```
+
+## What it does
+
+- Reads your test results from **many runs** (the more runs, the sharper the
+  picture).
+- Finds the tests that **disagree with themselves** across those runs.
+- Gives each one a score from **0 (solid) to 1 (a coin-flip)**.
+- Shows the **latest failure message**, so you can go fix it.
+- Runs in your **CI** and can leave a comment on the pull request — or block
+  it when flakiness spikes.
+
+## Works with any language
+
+Flakeguard reads **JUnit XML**, the report format almost every test
+framework can produce. One tool, every language:
+
+| Language / framework | How to produce the report |
+|---|---|
+| Python (pytest) | `pytest --junitxml=report.xml` |
+| Java (Maven / Gradle) | Surefire writes JUnit XML automatically |
+| Go | `gotestsum --junitfile report.xml` |
+| JavaScript (Jest) | the `jest-junit` reporter |
+| Ruby (RSpec) | the `rspec_junit_formatter` gem |
+| .NET, PHPUnit, … | built-in or plugin JUnit output |
+
+## Use in CI (GitHub Actions)
+
+```yaml
+- name: Flakeguard
+  uses: PrathameshKulkarni2k4/flakeguard@main
+  with:
+    reports: reports      # dir/glob of JUnit XML files
+    min-runs: 3           # ignore tests seen in fewer runs
+    max-flaky: 0          # fail if any flaky test is found (-1 = never fail)
+    comment: "true"       # post the report as a PR comment
+```
+
+See [`.github/workflows/example.yml`](.github/workflows/example.yml) for a
+complete workflow that runs a suite several times and feeds the reports in.
+
+To gate locally or in any other CI, use the exit code:
+
+```bash
+flakeguard scan ./reports --fail-on-flaky   # exits 1 if anything is flaky
+```
+
+## How the score works
+
+For each test we count passes vs. non-passes across all runs (skips are
+ignored). The flakiness score is:
+
+```
+score = 2 * min(passes, failures) / (passes + failures)
+```
+
+- always passes, or always fails → **0.0** (consistent, *not* flaky)
+- flips on every run (50/50) → **1.0** (maximally flaky)
+
+It is deliberately explainable — no black box. You can read the number and
+know exactly how it was derived.
+
+## Roadmap
+
+- [x] JUnit XML parsing, flakiness scoring, text/JSON/Markdown reports
+- [x] GitHub Action with PR comments and a flakiness budget
+- [ ] TAP and native JSON report formats
+- [ ] CI history so it learns across builds
+- [ ] Failure-signature clustering (group the *same* flake across tests)
+- [ ] Quarantine file so known flakes are set aside, not deleted
+
+## Contributing
+
+Contributions are very welcome — see [CONTRIBUTING.md](CONTRIBUTING.md).
+Adding support for a new report format is a great first contribution and is
+designed to be small. Please be kind; this project follows a
+[Code of Conduct](CODE_OF_CONDUCT.md).
+
+## License
+
+MIT — see [LICENSE](LICENSE).