flakebox 0.4.4__tar.gz

flakebox-0.4.4/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Calas Automations Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

flakebox-0.4.4/MANIFEST.in

@@ -0,0 +1 @@
+include action.yml

flakebox-0.4.4/PKG-INFO

@@ -0,0 +1,365 @@
+Metadata-Version: 2.4
+Name: flakebox
+Version: 0.4.4
+Summary: Self-hosted flaky-test triage for small teams.
+Author: Calas Automations Inc.
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/calasdan-sketch/flakebox
+Project-URL: Issues, https://github.com/calasdan-sketch/flakebox/issues
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Classifier: Environment :: Console
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# Flakebox (`flakebox`)
+
+Self-hosted flaky-test triage for small teams. Flaky tests eat 10–25% of
+CI compute and erode trust in every red build. Trunk and Datadog will sell
+you the fix per-seat; flakebox is the free, self-hosted alternative:
+ingest JUnit XML, detect flakiness statistically, quarantine the flakes,
+and get told when they're stable enough to come back. SQLite, stdlib-only,
+no SaaS.
+
+## Install
+
+Zero dependencies — Python 3.10+ stdlib only.
+
+```bash
+pip install git+https://github.com/calasdan-sketch/flakebox
+flakebox --version
+```
+
+## Usage
+
+```bash
+# in CI, after your test step (pytest --junitxml=reports/junit.xml):
+flakebox record --junit reports/junit.xml --commit="$GITHUB_SHA"
+
+# what is flaky?
+flakebox report
+flakebox report --format json
+
+# quarantine the flaky set for pytest:
+flakebox quarantine --format pytest -o .
+python -m pytest -p flakebox_quarantine   # flakes are skipped, build stays green
+
+# ...or keep running the flakes without letting them fail the build, so
+# they keep recording results and can earn their way back out:
+flakebox quarantine --format pytest --mode xfail -o .
+
+# the full quarantine set as shell-quoted pytest --deselect args (best
+# effort, see below; eval parses the quoting) or as JSON for other runners
+eval "pytest $(flakebox quarantine --format deselect)"
+flakebox quarantine --format json
+
+# when a quarantined test stabilizes, the report tells you:
+flakebox unquarantine --test "pkg.Class::test_x"
+flakebox history --test "pkg.Class::test_x"   # per-test run history, same
+                                              # thresholds/status words as report
+
+# carry the history around (CI artifact -> laptop, job -> aggregator):
+flakebox export --out flakebox-export.json
+flakebox import flakebox-export.json           # merges; re-import adds nothing
+```
+
+`--db PATH` picks the database and goes before **or** after the
+subcommand: `flakebox --db ci.db report` and `flakebox report --db ci.db`
+are the same command. Default `~/.flakebox/history.db`
+(`%USERPROFILE%\.flakebox\history.db` on Windows). A `--db` that is not a
+SQLite database is a clean `error: cannot use database ...`, exit 2.
+So are the other arguments flakebox cannot use: each ends in an `error:`
+line and exit 2, never a traceback. That covers an `-o` that is a file or
+cannot be created, an integer option that reaches the database
+(`--retire-after`, `history --limit`) outside the 64-bit range SQLite
+stores (-9223372036854775808 to 9223372036854775807), and a `--commit`,
+`--branch` or `--test` that is not valid UTF-8 text (a stray byte in the
+command line).
+
+`flakebox record` hashes the report: recording the same file for the
+same commit twice says `already recorded as run N` and adds nothing
+(`--force` records it anyway), so a retried CI step cannot double-count.
+Nested `<testsuite>` elements (surefire aggregates) are read, `time="0,5"`
+is 500 ms, and a `time` that is not a finite, non-negative number
+(`fast`, `inf`, `nan`, `-5`, `1e300`) is recorded as 0 with a warning
+instead of rejecting the report. A report that declares a `DOCTYPE` or an
+`ENTITY` is refused (exit 2) as a possible XML bomb, before anything is
+expanded; text that merely mentions `<!DOCTYPE html>`, such as captured
+output in a CDATA section, is fine.
+
+A report with no test cases at all (`<testsuite tests="0"/>`, or a file
+written before the tests ran) is not recorded. It is nearly always a test
+step that broke, and a run without tests makes every test look absent:
+three of them in a row retire the whole suite, and `report` says "No flaky
+tests detected". `record` prints a warning on stderr and exits 0, so a
+build that really ran no tests (docs only) is not failed. `--allow-empty`
+records the report anyway; it then counts like any other run. Empty runs
+that an earlier flakebox recorded stay in the history (there is no
+command to delete a run) and stop mattering once `--retire-after` (3) real
+runs have followed, or at once with `--retire-after 0`.
+
+`flakebox quarantine` is idempotent and safe to run on every build: it
+adds the current flaky set (never re-stamping tests already in
+quarantine) and always emits the **whole** quarantine set, not just the
+newly added tests. The one-line summary (`quarantined 1 new test(s); 3
+in quarantine.`) goes to stderr for `--format deselect`, so stdout is
+only ever the `--deselect` arguments.
+
+Run the plugin with `python -m pytest -p flakebox_quarantine`, not as the
+bare `pytest` command: `-p` imports `flakebox_quarantine` at start-up, and
+only `python -m pytest` puts the current directory (where `-o .` wrote the
+plugin) on `sys.path`. The bare command fails with an `ImportError` (`No
+module named 'flakebox_quarantine'`). If `-o` wrote the plugin to another
+directory, put that directory on `PYTHONPATH` first
+(`PYTHONPATH=ci python -m pytest -p flakebox_quarantine`; `set
+PYTHONPATH=ci` in `cmd.exe`, `$env:PYTHONPATH = "ci"` in PowerShell); with
+it set, the bare `pytest` command works too.
+
+`--format deselect` is **best effort**: JUnit ids are not pytest node
+ids, so flakebox maps the common pytest shapes
+(`pkg.module.Class::test_x` → `pkg/module.py::Class::test_x`,
+`pkg.module::test_x` → `pkg/module.py::test_x`) and pytest ignores a
+`--deselect` that matches nothing. A lowercase class name, a
+`python_files` pattern other than `*.py`, or a non-pytest producer breaks
+the guess, and pytest's `--deselect` is a *prefix* match (`--deselect
+t.py::test_x` also drops `test_xy` and `test_x[1]`); the generated plugin
+(`--format pytest`) matches whole names and path components instead (see
+below) and is the reliable path. The line quotes each argument for
+the shell (POSIX quoting; `cmd.exe` quoting on Windows), so the shell
+has to parse it once more: `eval "pytest $(flakebox quarantine --format
+deselect)"` hands pytest every id as one argument, even
+`test_param[a b]` or one containing `$(...)`. A bare `pytest
+$(flakebox quarantine --format deselect)` does not: command
+substitution splits the output at whitespace and never removes the
+quotes in it, so an id with a space or a shell character arrives in
+pieces with the quotes still attached (ids without either are fine).
+PowerShell has no `$(...)` word splitting, so `Invoke-Expression "pytest
+$(flakebox quarantine --format deselect)"` is the counterpart of `eval`,
+but it is **untested**: the Windows line is quoted for `cmd.exe`, which
+leaves `;`, `&`, `,`, `$`, backticks and parentheses bare, and PowerShell
+may parse those itself (an id that contains one may be split, or even
+run). If an id can contain one, use the plugin.
+
+On Windows the `--format deselect` line is written with a plain LF ending,
+not CR LF: text-mode output would end the last id with a carriage return,
+which Git-Bash's `$(...)` keeps (the id then matches nothing). That is
+simulated in the tests with a text stream that writes CR LF; it has not
+been run on Windows.
+
+The plugin matches a test to a quarantined id by name and by whole path
+components, in either spelling (`tests/test_mod.py::TestA::test_x` and
+`tests.test_mod.TestA::test_x` are the same test): the names, with any
+`[parameters]`, must be equal, and so must the module path and class
+names, except that one side may lack leading directories, because CI and
+your machine can have different rootdirs. So `test_mod.py::test_x` is the
+quarantined `tests.test_mod::test_x`, but `pkg_a/test_util.py::test_x` is
+never `pkg_b/test_util.py::test_x`. When the *recorded* id is the shorter
+one, it cannot say which directory it came from and every test that ends
+the same way matches: record and test from the same rootdir. Parametrised
+ids are matched whole: quarantining `test_version[1.5]` skips exactly that
+case, never `test_unrelated[0.5]` or `test_version[2.0]`.
+
+Example report:
+
+```
+## Flaky (pass AND fail — quarantine candidates)
+| test | flake rate | fails/passes | runs |
+|---|---|---|---|
+| `tests.test_a.TestA::test_flaky` | 50% | 3/3 | 6 |
+
+## Consistently failing (broken, not flaky — fix, don't quarantine)
+- `tests.test_b::test_broken` (6 failures, 0 passes)
+
+## Stable — ready to un-quarantine
+- `tests.test_c::test_old_flake` (12 consecutive passes)
+
+## Retired (absent from the last 3 runs — deleted or renamed, not counted)
+- `tests.test_d::test_removed` (was failing; last seen run 41, commit 9f3c2a1)
+```
+
+## How detection works
+
+No ML, fully explainable. Only *executed* runs count — a skipped run is
+not evidence either way, so it never pushes a test over `--min-runs` and
+never breaks a pass streak. "Newest" always means the newest recorded
+time of a run (its row id only breaks a tie), so merging an older export
+into a DB never makes old runs look recent.
+
+- **flaky**: at least one pass *and* at least one failure (`failed` or
+  `error`) across ≥5 executed runs (`--min-runs`), and not currently on
+  a pass streak of ≥10 (`--stable-runs`).
+- **recovered**: would be flaky, but the newest ≥10 executed runs all
+  passed — it has recovered and is left alone, so quarantining it,
+  releasing it and re-quarantining it from the same stale failures can't
+  loop. `report` leaves it out of the flaky table and `flakebox history`
+  says `recovered (18 consecutive passes)`: one `classify()` decides the
+  word for every command, and `history` takes the same `--min-runs`,
+  `--stable-runs` and `--retire-after` flags as `report`.
+- **failing** (the "Consistently failing" section): ≥5 executed runs and
+  never a pass. Broken, not flaky — different problem, don't quarantine
+  it. `flakebox history` labels it `failing` and shows no flake rate.
+- **stable**: a quarantined test whose newest ≥10 executed runs all
+  passed — the report tells you to un-quarantine it.
+- **retired**: a test with history that appears in none of the newest 3
+  recorded runs (`--retire-after`, `0` disables) — deleted or renamed.
+  It is listed on its own with where it was last seen and counted
+  nowhere else, so a test you removed does not stay "consistently
+  failing" for the rest of its 50-run window. One or two partial runs
+  (someone recorded a single file) do not retire anything. A retired
+  test still in quarantine is flagged with the `unquarantine` command
+  to run.
+- Everything else (always passing, or too few executed runs) is not
+  reported.
+
+The flake rate is failures ÷ executed runs; the `runs` column is
+executed runs. History lives in SQLite (`--db` to override).
+
+### Quarantine modes
+
+The generated pytest plugin reads `flakebox_quarantine.json` (the sorted
+ids, the mode and a count; no timestamp, so regenerating an unchanged set
+rewrites identical files and `git` shows no diff):
+
+- `--mode skip` (default): quarantined tests are skipped. They record no
+  result, so they cannot become *stable* on their own — run them
+  somewhere non-blocking (a separate job) and `flakebox record` that, or
+  use xfail mode.
+- `--mode xfail`: quarantined tests still run as non-strict `xfail`,
+  so they can never fail the build, but their real pass/fail lands in
+  the JUnit report (`flakebox record` maps the plugin's xfail back to a
+  failure) and the report can tell you when they have stabilised.
+
+### Export / import
+
+`flakebox export --out FILE` writes the whole DB (runs with results, and
+the quarantine set) as one JSON document. `flakebox import FILE...`
+merges exports into a DB: a run already present (same commit and same
+report hash) is skipped, so importing the same file twice, or into the
+DB it came from, adds nothing; quarantine entries merge as a set. Runs
+recorded without a report hash (flakebox < 0.3, or through the Python
+API) carry a content **fingerprint** (`legacy:` + a digest of commit,
+timestamp and results) instead, stamped when the DB is opened, so they
+are recognised too. Imported runs keep their original timestamp, and
+the newest-runs windows (the pass streak, `--retire-after`, `history`)
+follow timestamps, not the order rows were added; a run exported
+without a timestamp is imported as the oldest. A malformed export is
+refused as a whole before anything is written: every field is checked
+first (types, finite numbers that fit a 64-bit integer, text UTF-8 can
+encode, `"flakebox_export": 1` exactly) and the error names the first bad
+one (`error: cannot import ...: runs[0] must be an object`, exit 2).
+A path that cannot be read (a directory, no permission) and JSON nested
+too deeply to parse are the same one-line error and exit 2, and so is an
+`export --out` that cannot be written.
+This is how CI's history reaches a laptop, and how per-job DBs of a
+matrix build become one (see the Action).
+
+## GitHub Action
+
+`action.yml` is a composite action: restores the DB from cache, records
+the JUnit report, posts the flakebox report to the job summary, and —
+optionally — quarantines the flaky set and exports the history as
+artifact-ready files.
+
+```yaml
+- uses: calasdan-sketch/flakebox@main
+  with:
+    junit: reports/junit.xml
+    min-runs: "5"          # -> flakebox report --min-runs 5
+    fail-on-flaky: "true"
+    quarantine-output: flakebox/quarantine.json
+    export-output: flakebox/history.json
+- uses: actions/upload-artifact@v4
+  with:
+    name: flakebox
+    path: flakebox/
+```
+
+`@main` is this repository's default branch. A `@v1` reference works only once a
+release is tagged `v1`; until then use the branch, or pin a commit SHA.
+
+| input               | default    | what it does                                  |
+|---------------------|------------|-----------------------------------------------|
+| `junit`             | (required) | path to the JUnit XML → `flakebox record --junit` (a report with no test cases is warned about, not recorded) |
+| `db-key`            | `flakebox` | cache key prefix for the history DB           |
+| `min-runs`          | `5`        | executed runs before "flaky" → `flakebox report --min-runs` |
+| `fail-on-flaky`     | `false`    | fail the step when flaky tests are found that are not quarantined |
+| `quarantine-output` | (off)      | run `flakebox quarantine --format json` after the report and write the whole set here |
+| `export-output`     | (off)      | `flakebox export --out` the history DB here   |
+
+**The acknowledgement loop.** The history DB lives in the CI cache, where
+nobody can run `flakebox quarantine` by hand — so on its own,
+`fail-on-flaky: true` fails *every* build from the first flake. With
+`quarantine-output` set, the action quarantines the flaky set in the
+history DB right after the report (before the fail step): a new flake
+fails exactly one build, is acknowledged from the next run on, and the
+JSON file names it. Then, on a laptop:
+
+```bash
+gh run download --name flakebox              # quarantine.json + history.json
+flakebox import flakebox/history.json         # CI's history, locally
+flakebox quarantine --format pytest -o .      # flakebox_quarantine.{py,json}
+git add flakebox_quarantine.py flakebox_quarantine.json && git commit
+```
+
+and run the suite with `python -m pytest -p flakebox_quarantine` (skip
+mode) or `--mode xfail` so the quarantined tests keep reporting and can
+earn their way back out. When the report says a test is stable,
+`flakebox unquarantine --test ...` locally, regenerate the plugin, commit.
+
+Input names use dashes everywhere (`min-runs`, never `min_runs`); the test
+suite reads `action.yml` and checks that every referenced input is
+declared, every declared input is used, and every CLI flag it passes
+exists. Matrix builds: see the cache-race note at the top of `action.yml`
+(per-job DBs, then an aggregator job that `flakebox import`s every job's
+export).
+
+**No expression ever reaches a shell.** GitHub pastes the value of a
+`${{ ... }}` expression into a `run:` script *before* bash parses it, so a
+branch named `x";touch${IFS}pwned;echo"y` (a legal git ref) would run a
+command in your job. The action passes every input and `github.*` value
+through the step's `env:` and reads it as a quoted variable
+(`"$FB_BRANCH"`), so the shell only ever sees a string. Each option takes
+its value as one word, `--branch="$FB_BRANCH"`, so a value that starts
+with `-` (a ref or a path can) is still that option's value and never the
+start of another option. The test suite scans `action.yml` for `${{`
+inside any `run:` and for an option and its value written as two words,
+and runs each script under bash with hostile values and with values that
+start with `-`. The two `actions/cache` steps are first-party
+and use the `v4` tag, not a commit SHA; pin them in a fork if your policy
+requires it.
+
+## Limitations
+
+- **Recency is wall-clock time.** "Newest" is when a run was recorded
+  (an imported run keeps the time it was exported with); runs recorded in
+  the same second tie and fall back to the order they were added, and
+  machines whose clocks disagree order their merged runs by those clocks.
+- **`--format deselect` is a guess** at pytest node ids (see above); the
+  generated plugin is the reliable path, `eval` is needed for ids with a
+  space or a shell character, and the PowerShell form is untested.
+- **Any DOCTYPE is refused**, harmless or not: JUnit reports never need
+  one, and refusing them all keeps the XML-bomb check simple.
+- **The Action's cache steps use the `v4` tag**, not a commit SHA (above).
+
+## Tests
+
+```bash
+python3 -m unittest discover -s tests   # no network, stdlib only
+python3 -m pytest -q                    # same suite
+```
+
+No PyYAML anywhere (the tests read `action.yml` with a tiny flat reader
+and prove `import flakebox` works with `yaml` blocked). Tests point
+`HOME`/`USERPROFILE`/`APPDATA` at a throwaway directory, so they never
+touch your real `~/.flakebox`. A few tests drive real `pytest` as a
+subprocess to prove the mapped `--deselect` ids and the generated plugin
+work on parametrised ids (skipped when pytest is not installed). Runs on Windows 11: stdout/stderr are
+reconfigured to replace unencodable characters (the report's emoji on a
+cp1252 console), and every text file is opened as UTF-8.
+
+## License
+
+MIT — see `LICENSE`.