pytest-flakemark 1.0.0__tar.gz

pytest_flakemark-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Khushdeep Sharma, Chitkara University
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pytest_flakemark-1.0.0/PKG-INFO

@@ -0,0 +1,263 @@
+Metadata-Version: 2.4
+Name: pytest-flakemark
+Version: 1.0.0
+Summary: FLAKEMARK — Differential execution tracer that finds the exact file, line, and root cause of any flaky test.
+Home-page: https://github.com/khushsharma509/flakemark
+Author: Khushdeep Sharma
+Author-email: itskhushsharma@gmail.com
+Project-URL: Bug Reports, https://github.com/khushsharma509/flakemark/issues
+Project-URL: Source, https://github.com/khushsharma509/flakemark
+Keywords: pytest,flaky-tests,flakemark,debugging,root-cause,differential-analysis,test-diagnosis,execution-trace
+Classifier: Development Status :: 4 - Beta
+Classifier: Framework :: Pytest
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Software Development :: Testing
+Classifier: Topic :: Software Development :: Debuggers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pytest>=7.0
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: keywords
+Dynamic: license-file
+Dynamic: project-url
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# FLAKEMARK
+
+**pytest-flakemark — Find the exact line where your flaky test breaks.**
+
+Not "your test is flaky." The actual file. The actual line. The actual fix.
+
+Built by Khushdeep Sharma — Chitkara University, 2026.
+
+---
+
+## The Problem
+
+```
+FAILED tests/test_login.py::test_user_session
+[Flaky — rerunning]
+PASSED tests/test_login.py::test_user_session
+```
+
+Every existing tool gives you this. It tells you nothing new.
+
+## What FLAKEMARK Gives You
+
+```
+--------------------------------------------------------
+FLAKEMARK - Flaky Test Root Cause Found
+--------------------------------------------------------
+File:       tests/test_login.py
+Line:       47
+Function:   test_user_session
+Type:       timing_delta
+Cause:      Race condition or timing dependency
+
+Detail:     Line 47: 1.2ms (pass) vs 148.3ms (fail) — 124x timing difference.
+
+Fix:        Replace time.sleep(N) with threading.Event().wait()
+            or asyncio.wait_for(). Never hardcode sleep durations in tests.
+
+Confidence: 85%  |  Total divergences: 1
+--------------------------------------------------------
+```
+
+---
+
+## How FLAKEMARK Works
+
+FLAKEMARK instruments your test at the AST level, runs it twice simultaneously,
+records every operation in both runs, then finds the exact line where the two
+executions diverged. That divergence is your bug.
+
+```
+Your test
+  ├── Run 1 (instrumented) → ExecutionTrace A  [op, op, op ...]
+  └── Run 2 (instrumented) → ExecutionTrace B  [op, op, op ...]
+                                                     ↓
+                                       DifferentialAnalyser
+                                       Two-pointer trace walk
+                                                     ↓
+                                   "Line 47: 124x timing difference"
+```
+
+---
+
+## Install
+
+```bash
+pip install pytest-flakemark
+```
+
+Or use directly (no install needed):
+
+```bash
+git clone https://github.com/khushsharma509/flakemark
+cd flakemark
+# Add to PYTHONPATH and use
+```
+
+---
+
+## Usage — 4 Ways
+
+### 1. Source string
+
+```python
+from flakemark import FlakeMark
+
+source = """
+import random
+def test_flaky():
+    result = random.randint(0, 1)
+    assert result == 1
+"""
+
+report = FlakeMark.diagnose_source(source, "test_flaky", runs=10)
+print(report)
+```
+
+### 2. Real test file
+
+```python
+from flakemark import FlakeMark
+
+report = FlakeMark.diagnose_file(
+    filepath       = "tests/test_api.py",
+    test_func_name = "test_user_session",
+    runs           = 6,
+    project_root   = "/path/to/your/project",
+)
+print(report)
+```
+
+### 3. Batch scan entire folder
+
+```python
+from flakemark import FlakeMark
+
+results = FlakeMark.diagnose_batch("tests/", runs=4)
+
+flaky = {k: v for k, v in results.items() if v.is_found()}
+print(f"FLAKEMARK found {len(flaky)} flaky tests:\n")
+for name, report in flaky.items():
+    print(f"  {name}")
+    print(f"  Line {report.primary.line} — {report.primary.divergence_type.value}")
+    print(f"  Fix: {report.primary.fix[:60]}")
+```
+
+### 4. pytest CLI (after pip install)
+
+```bash
+pytest --flakemark-diagnose tests/
+pytest --flakemark-diagnose --flakemark-runs=8 tests/test_api.py
+```
+
+---
+
+## What FLAKEMARK Detects
+
+| Type | What it means | Root cause |
+|---|---|---|
+| `value_mismatch` | Same line, different value | `random`, shared state |
+| `timing_delta` | Same op, 3x+ slower | `time.sleep()`, race condition |
+| `thread_race` | Same op, different thread | Missing `Lock()` |
+| `sequence_break` | Different execution path | Test order dependency |
+| `missing_event` | One run skipped an operation | Conditional on external state |
+| `early_termination` | One run ended much sooner | Timeout, unhandled exception |
+
+---
+
+## Parameters
+
+| Parameter | Default | Meaning |
+|---|---|---|
+| `runs` | `4` | Times to run. Use `10+` for low-frequency flakes |
+| `timeout` | `30` | Seconds before a run is killed |
+| `project_root` | `os.getcwd()` | Project root so imports work |
+
+---
+
+## Accuracy
+
+```bash
+python tests/test_accuracy.py
+```
+
+```
+Score: 15/15 = 100%
+  ✓  VALUE_MISMATCH detected   (78% confidence)
+  ✓  TIMING_DELTA detected     (85% confidence)
+  ✓  THREAD_RACE detected      (93% confidence)
+  ✓  EARLY_TERMINATION detected (55% confidence)
+  ✓  No false positive on stable test (0%)
+  ✓  Async test handled correctly
+```
+
+---
+
+## Comparison to Other Tools
+
+| | FLAKEMARK | FlakyGuard | pytest-randomly | CANNIER | Divergent |
+|---|---|---|---|---|---|
+| Finds exact root cause line | **Yes** | No | No | No | Yes (JS only) |
+| Python / pytest | **Yes** | No (Java) | Yes | Yes | No (JS) |
+| AST instrumentation | **Yes** | No | No | No | Partial |
+| Subprocess isolation | **Yes** | No | No | No | No |
+| Async test support | **Yes** | No | Yes | No | No |
+| Zero dependencies | **Yes** | No | No | No | No |
+
+---
+
+## Project Structure
+
+```
+flakemark/
+├── flakemark/
+│   ├── __init__.py              FlakeMark, DivergenceReport, DivergenceType
+│   ├── engine.py                FlakeMark class — diagnose_file/source/batch
+│   ├── pytest_plugin.py         pytest plugin — --flakemark-diagnose flag
+│   ├── tracer/
+│   │   └── instrument.py        TraceInserter + run_instrumented()   ← CORE IP
+│   └── differ/
+│       └── divergence.py        DifferentialAnalyser + DivergenceType ← CORE IP
+├── tests/
+│   └── test_accuracy.py         Accuracy test suite
+├── setup.py                     PyPI config
+└── README.md
+```
+
+---
+
+## What to Copyright
+
+| File | What is original |
+|---|---|
+| `tracer/instrument.py` — `TraceInserter` | AST NodeTransformer for execution recording |
+| `tracer/instrument.py` — `run_instrumented()` | Subprocess isolation runner with JSON protocol |
+| `differ/divergence.py` — `DifferentialAnalyser._walk()` | Two-pointer differential trace walk |
+| `differ/divergence.py` — `DivergenceType` | Six-category flakiness classification schema |
+| `differ/divergence.py` — `_score()` | Confidence scoring formula |
+| `engine.py` — `FlakeMark._collect_traces()` | Concurrent dual-run orchestration |
+
+---
+
+## License
+
+MIT — Copyright © 2026 Khushdeep Sharma, Chitkara University
+
+---
+
+*FLAKEMARK — Find the line. Fix the test.*

pytest_flakemark-1.0.0/README.md

@@ -0,0 +1,228 @@
+# FLAKEMARK
+
+**pytest-flakemark — Find the exact line where your flaky test breaks.**
+
+Not "your test is flaky." The actual file. The actual line. The actual fix.
+
+Built by Khushdeep Sharma — Chitkara University, 2026.
+
+---
+
+## The Problem
+
+```
+FAILED tests/test_login.py::test_user_session
+[Flaky — rerunning]
+PASSED tests/test_login.py::test_user_session
+```
+
+Every existing tool gives you this. It tells you nothing new.
+
+## What FLAKEMARK Gives You
+
+```
+--------------------------------------------------------
+FLAKEMARK - Flaky Test Root Cause Found
+--------------------------------------------------------
+File:       tests/test_login.py
+Line:       47
+Function:   test_user_session
+Type:       timing_delta
+Cause:      Race condition or timing dependency
+
+Detail:     Line 47: 1.2ms (pass) vs 148.3ms (fail) — 124x timing difference.
+
+Fix:        Replace time.sleep(N) with threading.Event().wait()
+            or asyncio.wait_for(). Never hardcode sleep durations in tests.
+
+Confidence: 85%  |  Total divergences: 1
+--------------------------------------------------------
+```
+
+---
+
+## How FLAKEMARK Works
+
+FLAKEMARK instruments your test at the AST level, runs it twice simultaneously,
+records every operation in both runs, then finds the exact line where the two
+executions diverged. That divergence is your bug.
+
+```
+Your test
+  ├── Run 1 (instrumented) → ExecutionTrace A  [op, op, op ...]
+  └── Run 2 (instrumented) → ExecutionTrace B  [op, op, op ...]
+                                                     ↓
+                                       DifferentialAnalyser
+                                       Two-pointer trace walk
+                                                     ↓
+                                   "Line 47: 124x timing difference"
+```
+
+---
+
+## Install
+
+```bash
+pip install pytest-flakemark
+```
+
+Or use directly (no install needed):
+
+```bash
+git clone https://github.com/khushsharma509/flakemark
+cd flakemark
+# Add to PYTHONPATH and use
+```
+
+---
+
+## Usage — 4 Ways
+
+### 1. Source string
+
+```python
+from flakemark import FlakeMark
+
+source = """
+import random
+def test_flaky():
+    result = random.randint(0, 1)
+    assert result == 1
+"""
+
+report = FlakeMark.diagnose_source(source, "test_flaky", runs=10)
+print(report)
+```
+
+### 2. Real test file
+
+```python
+from flakemark import FlakeMark
+
+report = FlakeMark.diagnose_file(
+    filepath       = "tests/test_api.py",
+    test_func_name = "test_user_session",
+    runs           = 6,
+    project_root   = "/path/to/your/project",
+)
+print(report)
+```
+
+### 3. Batch scan entire folder
+
+```python
+from flakemark import FlakeMark
+
+results = FlakeMark.diagnose_batch("tests/", runs=4)
+
+flaky = {k: v for k, v in results.items() if v.is_found()}
+print(f"FLAKEMARK found {len(flaky)} flaky tests:\n")
+for name, report in flaky.items():
+    print(f"  {name}")
+    print(f"  Line {report.primary.line} — {report.primary.divergence_type.value}")
+    print(f"  Fix: {report.primary.fix[:60]}")
+```
+
+### 4. pytest CLI (after pip install)
+
+```bash
+pytest --flakemark-diagnose tests/
+pytest --flakemark-diagnose --flakemark-runs=8 tests/test_api.py
+```
+
+---
+
+## What FLAKEMARK Detects
+
+| Type | What it means | Root cause |
+|---|---|---|
+| `value_mismatch` | Same line, different value | `random`, shared state |
+| `timing_delta` | Same op, 3x+ slower | `time.sleep()`, race condition |
+| `thread_race` | Same op, different thread | Missing `Lock()` |
+| `sequence_break` | Different execution path | Test order dependency |
+| `missing_event` | One run skipped an operation | Conditional on external state |
+| `early_termination` | One run ended much sooner | Timeout, unhandled exception |
+
+---
+
+## Parameters
+
+| Parameter | Default | Meaning |
+|---|---|---|
+| `runs` | `4` | Times to run. Use `10+` for low-frequency flakes |
+| `timeout` | `30` | Seconds before a run is killed |
+| `project_root` | `os.getcwd()` | Project root so imports work |
+
+---
+
+## Accuracy
+
+```bash
+python tests/test_accuracy.py
+```
+
+```
+Score: 15/15 = 100%
+  ✓  VALUE_MISMATCH detected   (78% confidence)
+  ✓  TIMING_DELTA detected     (85% confidence)
+  ✓  THREAD_RACE detected      (93% confidence)
+  ✓  EARLY_TERMINATION detected (55% confidence)
+  ✓  No false positive on stable test (0%)
+  ✓  Async test handled correctly
+```
+
+---
+
+## Comparison to Other Tools
+
+| | FLAKEMARK | FlakyGuard | pytest-randomly | CANNIER | Divergent |
+|---|---|---|---|---|---|
+| Finds exact root cause line | **Yes** | No | No | No | Yes (JS only) |
+| Python / pytest | **Yes** | No (Java) | Yes | Yes | No (JS) |
+| AST instrumentation | **Yes** | No | No | No | Partial |
+| Subprocess isolation | **Yes** | No | No | No | No |
+| Async test support | **Yes** | No | Yes | No | No |
+| Zero dependencies | **Yes** | No | No | No | No |
+
+---
+
+## Project Structure
+
+```
+flakemark/
+├── flakemark/
+│   ├── __init__.py              FlakeMark, DivergenceReport, DivergenceType
+│   ├── engine.py                FlakeMark class — diagnose_file/source/batch
+│   ├── pytest_plugin.py         pytest plugin — --flakemark-diagnose flag
+│   ├── tracer/
+│   │   └── instrument.py        TraceInserter + run_instrumented()   ← CORE IP
+│   └── differ/
+│       └── divergence.py        DifferentialAnalyser + DivergenceType ← CORE IP
+├── tests/
+│   └── test_accuracy.py         Accuracy test suite
+├── setup.py                     PyPI config
+└── README.md
+```
+
+---
+
+## What to Copyright
+
+| File | What is original |
+|---|---|
+| `tracer/instrument.py` — `TraceInserter` | AST NodeTransformer for execution recording |
+| `tracer/instrument.py` — `run_instrumented()` | Subprocess isolation runner with JSON protocol |
+| `differ/divergence.py` — `DifferentialAnalyser._walk()` | Two-pointer differential trace walk |
+| `differ/divergence.py` — `DivergenceType` | Six-category flakiness classification schema |
+| `differ/divergence.py` — `_score()` | Confidence scoring formula |
+| `engine.py` — `FlakeMark._collect_traces()` | Concurrent dual-run orchestration |
+
+---
+
+## License
+
+MIT — Copyright © 2026 Khushdeep Sharma, Chitkara University
+
+---
+
+*FLAKEMARK — Find the line. Fix the test.*

pytest_flakemark-1.0.0/flakemark/__init__.py

@@ -0,0 +1,11 @@
+# flakemark/__init__.py
+# FLAKEMARK — Differential Execution Tracer for Flaky Test Root Cause Diagnosis
+# Copyright: Khushdeep Sharma, Chitkara University, 2026
+
+from flakemark.engine import FlakeMark
+from flakemark.differ.divergence import DivergenceReport, DivergenceType
+
+__version__ = "1.0.0"
+__author__  = "Khushdeep Sharma — Chitkara University 2026"
+__email__   = "itskhushsharma@gmail.com"
+__all__     = ["FlakeMark", "DivergenceReport", "DivergenceType"]

pytest_flakemark-1.0.0/flakemark/differ/__init__.py

@@ -0,0 +1 @@
+# flakemark/differ/__init__.py