pytest-sort 1.2.1__tar.gz → 1.3.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- pytest-sort-1.3.0/PKG-INFO +107 -0
- pytest-sort-1.3.0/README.md +64 -0
- {pytest-sort-1.2.1 → pytest-sort-1.3.0}/pyproject.toml +40 -3
- {pytest-sort-1.2.1 → pytest-sort-1.3.0}/src/pytest_sort/config.py +11 -10
- {pytest-sort-1.2.1 → pytest-sort-1.3.0}/src/pytest_sort/core.py +14 -4
- {pytest-sort-1.2.1 → pytest-sort-1.3.0}/src/pytest_sort/diffcov.py +32 -5
- pytest-sort-1.3.0/src/pytest_sort.egg-info/PKG-INFO +107 -0
- {pytest-sort-1.2.1 → pytest-sort-1.3.0}/test/test_config.py +33 -30
- {pytest-sort-1.2.1 → pytest-sort-1.3.0}/test/test_core.py +112 -57
- {pytest-sort-1.2.1 → pytest-sort-1.3.0}/test/test_database.py +37 -40
- {pytest-sort-1.2.1 → pytest-sort-1.3.0}/test/test_diffcov.py +94 -26
- {pytest-sort-1.2.1 → pytest-sort-1.3.0}/test/test_plugin.py +1 -1
- pytest-sort-1.2.1/PKG-INFO +0 -346
- pytest-sort-1.2.1/README.md +0 -304
- pytest-sort-1.2.1/src/pytest_sort.egg-info/PKG-INFO +0 -346
- {pytest-sort-1.2.1 → pytest-sort-1.3.0}/LICENSE +0 -0
- {pytest-sort-1.2.1 → pytest-sort-1.3.0}/setup.cfg +0 -0
- {pytest-sort-1.2.1 → pytest-sort-1.3.0}/src/pytest_sort/__init__.py +0 -0
- {pytest-sort-1.2.1 → pytest-sort-1.3.0}/src/pytest_sort/database.py +0 -0
- {pytest-sort-1.2.1 → pytest-sort-1.3.0}/src/pytest_sort/plugin.py +0 -0
- {pytest-sort-1.2.1 → pytest-sort-1.3.0}/src/pytest_sort.egg-info/SOURCES.txt +0 -0
- {pytest-sort-1.2.1 → pytest-sort-1.3.0}/src/pytest_sort.egg-info/dependency_links.txt +0 -0
- {pytest-sort-1.2.1 → pytest-sort-1.3.0}/src/pytest_sort.egg-info/entry_points.txt +0 -0
- {pytest-sort-1.2.1 → pytest-sort-1.3.0}/src/pytest_sort.egg-info/requires.txt +0 -0
- {pytest-sort-1.2.1 → pytest-sort-1.3.0}/src/pytest_sort.egg-info/top_level.txt +0 -0
|
@@ -0,0 +1,107 @@
|
|
|
1
|
+
Metadata-Version: 2.1
|
|
2
|
+
Name: pytest-sort
|
|
3
|
+
Version: 1.3.0
|
|
4
|
+
Summary: Tools for sorting test cases
|
|
5
|
+
Author-email: WiredNerd <pbuschmail-pytestsort@yahoo.com>
|
|
6
|
+
Maintainer-email: WiredNerd <pbuschmail-pytestsort@yahoo.com>
|
|
7
|
+
License: MIT License
|
|
8
|
+
|
|
9
|
+
Copyright (c) 2023 Peter Busch
|
|
10
|
+
|
|
11
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
12
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
13
|
+
in the Software without restriction, including without limitation the rights
|
|
14
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
15
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
16
|
+
furnished to do so, subject to the following conditions:
|
|
17
|
+
|
|
18
|
+
The above copyright notice and this permission notice shall be included in all
|
|
19
|
+
copies or substantial portions of the Software.
|
|
20
|
+
|
|
21
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
22
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
23
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
24
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
25
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
26
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
27
|
+
SOFTWARE.
|
|
28
|
+
|
|
29
|
+
Project-URL: Documentation, https://pytest-sort.readthedocs.io/
|
|
30
|
+
Project-URL: Homepage, https://github.com/WiredNerd/pytest-sort
|
|
31
|
+
Project-URL: Issue Tracker, https://github.com/WiredNerd/pytest-sort/issues
|
|
32
|
+
Keywords: test,pytest,order,pytest-order,sort,random,pytest-random,pytest-random-order,randomly,pytest-randomly
|
|
33
|
+
Classifier: Programming Language :: Python :: 3
|
|
34
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
35
|
+
Classifier: Operating System :: OS Independent
|
|
36
|
+
Classifier: Framework :: Pytest
|
|
37
|
+
Requires-Python: >=3.8
|
|
38
|
+
Description-Content-Type: text/markdown
|
|
39
|
+
License-File: LICENSE
|
|
40
|
+
Requires-Dist: pytest>=7.4.0
|
|
41
|
+
Requires-Dist: whatthepatch>=1.0.5
|
|
42
|
+
Requires-Dist: coverage>=7.3.1
|
|
43
|
+
|
|
44
|
+
[](https://github.com/WiredNerd/pytest-sort)
|
|
45
|
+
[](https://pypi.org/project/pytest-sort)
|
|
46
|
+
[](https://pypi.org/project/pytest-sort)
|
|
47
|
+
[](https://pypistats.org/packages/pytest-sort)
|
|
48
|
+
[](https://github.com/WiredNerd/pytest-sort/blob/main/LICENSE)
|
|
49
|
+
|
|
50
|
+
[](https://pytest-cov.readthedocs.io)
|
|
51
|
+
[](https://poodle.readthedocs.io/)
|
|
52
|
+
[](https://pytest-sort.readthedocs.io/)
|
|
53
|
+
|
|
54
|
+
[](https://pycqa.github.io/isort/)
|
|
55
|
+
[](https://black.readthedocs.io)
|
|
56
|
+
[](https://beta.ruff.rs/docs/)
|
|
57
|
+
[](https://snyk.io/)
|
|
58
|
+
|
|
59
|
+
|
|
60
|
+
|
|
61
|
+
# Pytest-Sort
|
|
62
|
+
|
|
63
|
+
This pytest plugin is used to help identify application state leaks in your test suite. It does this by automatically changing the execution order of test cases.
|
|
64
|
+
|
|
65
|
+
How frustrating is it to have a test case that works fine when runs by itself, but fails when run with the rest of the suite? By running tests in a random order, the culprit of such problems can be identified much earlier in the development process. Alternately, running in ordered and reverse order can help find the cause of the problem in existing test suites.
|
|
66
|
+
|
|
67
|
+
## Features
|
|
68
|
+
|
|
69
|
+
* Multiple sort options including: Random, Fastest, Reverse, and more.
|
|
70
|
+
* Group test cases into buckets that are each sorted separately.
|
|
71
|
+
* Control sort order of the buckets.
|
|
72
|
+
* Use Pytest markers to always run specific test cases in order.
|
|
73
|
+
|
|
74
|
+
## Quick Start
|
|
75
|
+
|
|
76
|
+
Installation:
|
|
77
|
+
|
|
78
|
+
```
|
|
79
|
+
pip install pytest-sort --upgrade
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
Running with pytest-sort:
|
|
83
|
+
|
|
84
|
+
```
|
|
85
|
+
pytest --sort-mode=random
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
## Documentation:
|
|
89
|
+
|
|
90
|
+
- Usage, Examples, Configuration, and more: https://pytest-sort.readthedocs.io/
|
|
91
|
+
|
|
92
|
+
## Contribute
|
|
93
|
+
|
|
94
|
+
- Issue Tracker: https://github.com/WiredNerd/pytest-sort/issues
|
|
95
|
+
- Source Code: https://github.com/WiredNerd/pytest-sort
|
|
96
|
+
|
|
97
|
+
## Support
|
|
98
|
+
|
|
99
|
+
If you are having issues, please let us know.
|
|
100
|
+
|
|
101
|
+
I can be contacted at: pbuschmail-pytestsort@yahoo.com
|
|
102
|
+
|
|
103
|
+
Or by opening an issue: https://github.com/WiredNerd/pytest-sort/issues
|
|
104
|
+
|
|
105
|
+
## License
|
|
106
|
+
|
|
107
|
+
The project is licensed under the MIT license.
|
|
@@ -0,0 +1,64 @@
|
|
|
1
|
+
[](https://github.com/WiredNerd/pytest-sort)
|
|
2
|
+
[](https://pypi.org/project/pytest-sort)
|
|
3
|
+
[](https://pypi.org/project/pytest-sort)
|
|
4
|
+
[](https://pypistats.org/packages/pytest-sort)
|
|
5
|
+
[](https://github.com/WiredNerd/pytest-sort/blob/main/LICENSE)
|
|
6
|
+
|
|
7
|
+
[](https://pytest-cov.readthedocs.io)
|
|
8
|
+
[](https://poodle.readthedocs.io/)
|
|
9
|
+
[](https://pytest-sort.readthedocs.io/)
|
|
10
|
+
|
|
11
|
+
[](https://pycqa.github.io/isort/)
|
|
12
|
+
[](https://black.readthedocs.io)
|
|
13
|
+
[](https://beta.ruff.rs/docs/)
|
|
14
|
+
[](https://snyk.io/)
|
|
15
|
+
|
|
16
|
+
|
|
17
|
+
|
|
18
|
+
# Pytest-Sort
|
|
19
|
+
|
|
20
|
+
This pytest plugin is used to help identify application state leaks in your test suite. It does this by automatically changing the execution order of test cases.
|
|
21
|
+
|
|
22
|
+
How frustrating is it to have a test case that works fine when runs by itself, but fails when run with the rest of the suite? By running tests in a random order, the culprit of such problems can be identified much earlier in the development process. Alternately, running in ordered and reverse order can help find the cause of the problem in existing test suites.
|
|
23
|
+
|
|
24
|
+
## Features
|
|
25
|
+
|
|
26
|
+
* Multiple sort options including: Random, Fastest, Reverse, and more.
|
|
27
|
+
* Group test cases into buckets that are each sorted separately.
|
|
28
|
+
* Control sort order of the buckets.
|
|
29
|
+
* Use Pytest markers to always run specific test cases in order.
|
|
30
|
+
|
|
31
|
+
## Quick Start
|
|
32
|
+
|
|
33
|
+
Installation:
|
|
34
|
+
|
|
35
|
+
```
|
|
36
|
+
pip install pytest-sort --upgrade
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
Running with pytest-sort:
|
|
40
|
+
|
|
41
|
+
```
|
|
42
|
+
pytest --sort-mode=random
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
## Documentation:
|
|
46
|
+
|
|
47
|
+
- Usage, Examples, Configuration, and more: https://pytest-sort.readthedocs.io/
|
|
48
|
+
|
|
49
|
+
## Contribute
|
|
50
|
+
|
|
51
|
+
- Issue Tracker: https://github.com/WiredNerd/pytest-sort/issues
|
|
52
|
+
- Source Code: https://github.com/WiredNerd/pytest-sort
|
|
53
|
+
|
|
54
|
+
## Support
|
|
55
|
+
|
|
56
|
+
If you are having issues, please let us know.
|
|
57
|
+
|
|
58
|
+
I can be contacted at: pbuschmail-pytestsort@yahoo.com
|
|
59
|
+
|
|
60
|
+
Or by opening an issue: https://github.com/WiredNerd/pytest-sort/issues
|
|
61
|
+
|
|
62
|
+
## License
|
|
63
|
+
|
|
64
|
+
The project is licensed under the MIT license.
|
|
@@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta"
|
|
|
5
5
|
[project]
|
|
6
6
|
name = "pytest-sort"
|
|
7
7
|
description = "Tools for sorting test cases"
|
|
8
|
-
version = "1.
|
|
8
|
+
version = "1.3.0"
|
|
9
9
|
license = { file = "LICENSE" }
|
|
10
10
|
keywords = [
|
|
11
11
|
"test",
|
|
@@ -34,8 +34,9 @@ classifiers = [
|
|
|
34
34
|
dependencies = ["pytest>=7.4.0", "whatthepatch>=1.0.5", "coverage>=7.3.1"]
|
|
35
35
|
|
|
36
36
|
[project.urls]
|
|
37
|
+
"Documentation" = "https://pytest-sort.readthedocs.io/"
|
|
37
38
|
"Homepage" = "https://github.com/WiredNerd/pytest-sort"
|
|
38
|
-
"
|
|
39
|
+
"Issue Tracker" = "https://github.com/WiredNerd/pytest-sort/issues"
|
|
39
40
|
|
|
40
41
|
[project.entry-points."pytest11"]
|
|
41
42
|
sort = "pytest_sort.plugin"
|
|
@@ -43,7 +44,7 @@ sort = "pytest_sort.plugin"
|
|
|
43
44
|
[tool.pytest.ini_options]
|
|
44
45
|
pythonpath = ["src", "src_mut"]
|
|
45
46
|
testpaths = ["test"]
|
|
46
|
-
addopts = "-vv --cov-reset --cov-report=html:
|
|
47
|
+
addopts = "-vv --cov-reset --cov-report=html:cov-html --cov-report=term"
|
|
47
48
|
|
|
48
49
|
# https://pytest-cov.readthedocs.io/en/latest/config.html
|
|
49
50
|
# https://coverage.readthedocs.io/en/latest/config.html
|
|
@@ -55,6 +56,12 @@ branch = true
|
|
|
55
56
|
skip_empty = true
|
|
56
57
|
exclude_also = ["import .*", "from .* import .*", "if TYPE_CHECKING:"]
|
|
57
58
|
|
|
59
|
+
[tool.poodle]
|
|
60
|
+
reporters = ["summary", "not_found", "json"]
|
|
61
|
+
|
|
62
|
+
[tool.poodle.runner_opts]
|
|
63
|
+
command_line = "pytest -x --assert=plain -o pythonpath='{PYTHONPATH}' --sort-mode=mutcov"
|
|
64
|
+
|
|
58
65
|
[tool.black]
|
|
59
66
|
line-length = 120
|
|
60
67
|
target-version = ['py38', 'py39', 'py310', 'py311']
|
|
@@ -66,15 +73,19 @@ profile = 'black'
|
|
|
66
73
|
|
|
67
74
|
[tool.mypy]
|
|
68
75
|
files = "src"
|
|
76
|
+
mypy_path = "src"
|
|
69
77
|
|
|
70
78
|
[tool.ruff]
|
|
71
79
|
target-version = "py38"
|
|
72
80
|
show-source = true
|
|
73
81
|
select = ["ALL"]
|
|
74
82
|
ignore = [
|
|
83
|
+
"ANN101", # Missing type annotation for self in method
|
|
84
|
+
"COM812", # missing-trailing-comma
|
|
75
85
|
"D203", # 1 blank line required before class docstring
|
|
76
86
|
"D213", # multi-line-summary-second-line
|
|
77
87
|
"EM101", # Exception must not use a string literal, assign to variable first
|
|
88
|
+
"ISC001", # single-line-implicit-string-concatenation
|
|
78
89
|
"S311", # Standard pseudo-random generators are not suitable for cryptographic purposes
|
|
79
90
|
"TRY003", # Avoid specifying long messages outside the exception class
|
|
80
91
|
]
|
|
@@ -92,6 +103,10 @@ exclude = [
|
|
|
92
103
|
"dist",
|
|
93
104
|
"venv",
|
|
94
105
|
]
|
|
106
|
+
|
|
107
|
+
[tool.ruff.lint.isort]
|
|
108
|
+
known-first-party = ["pytest_sort"]
|
|
109
|
+
|
|
95
110
|
[tool.ruff.per-file-ignores]
|
|
96
111
|
"src/pytest_sort/core.py" = [
|
|
97
112
|
"ARG005", # Unused lambda argument
|
|
@@ -100,3 +115,25 @@ exclude = [
|
|
|
100
115
|
"src/pytest_sort/database.py" = [
|
|
101
116
|
"PLW0603", # global-statement
|
|
102
117
|
]
|
|
118
|
+
"test/*" = [
|
|
119
|
+
"ANN001", # Missing type annotation for function argument
|
|
120
|
+
"ANN201", # Missing return type annotation for public function
|
|
121
|
+
"ANN202", # Missing return type annotation for private function
|
|
122
|
+
"ANN205", # Missing return type annotation for staticmethod
|
|
123
|
+
"ANN206", # Missing return type annotation for classmethod
|
|
124
|
+
"D100", # Missing docstring in public module
|
|
125
|
+
"D101", # Missing docstring in public class
|
|
126
|
+
"D102", # Missing docstring in public method
|
|
127
|
+
"D103", # Missing docstring in public function
|
|
128
|
+
"D104", # Missing docstring in public package
|
|
129
|
+
"D106", # Missing docstring in public nested class
|
|
130
|
+
"D107", # Missing docstring in ...
|
|
131
|
+
"ERA001", # Found commented-out code
|
|
132
|
+
"FBT003", # Boolean positional value in function call
|
|
133
|
+
"N802", # Function name should be lowercase
|
|
134
|
+
"N803", # Argument name should be lowercase
|
|
135
|
+
"PLR0913", # Too many arguments in function definition
|
|
136
|
+
"PLR2004", # Magic value used in comparison
|
|
137
|
+
"S101", # Use of `assert` detected
|
|
138
|
+
"SLF001", # Private member accessed
|
|
139
|
+
]
|
|
@@ -11,7 +11,7 @@ from pytest_sort import database
|
|
|
11
11
|
if TYPE_CHECKING:
|
|
12
12
|
import pytest
|
|
13
13
|
|
|
14
|
-
modes = ["ordered", "reverse", "md5", "random", "fastest", "diffcov"]
|
|
14
|
+
modes = ["ordered", "reverse", "md5", "random", "fastest", "diffcov", "mutcov"]
|
|
15
15
|
bucket_types = ["session", "package", "module", "class", "function", "parent", "grandparent"]
|
|
16
16
|
|
|
17
17
|
legacy_modes = {
|
|
@@ -25,23 +25,24 @@ legacy_bucket_types = {
|
|
|
25
25
|
class SortConfig:
|
|
26
26
|
"""Statoc class for storing configuration of pytest_sort."""
|
|
27
27
|
|
|
28
|
-
mode: str = "ordered"
|
|
29
|
-
bucket: str = "parent"
|
|
30
|
-
bucket_mode: str = "sort_mode"
|
|
31
|
-
record: bool | None = None # pragma: no mutate
|
|
32
|
-
reset: bool = False
|
|
33
|
-
report: bool = False
|
|
28
|
+
mode: ClassVar[str] = "ordered"
|
|
29
|
+
bucket: ClassVar[str] = "parent"
|
|
30
|
+
bucket_mode: ClassVar[str] = "sort_mode"
|
|
31
|
+
record: ClassVar[bool | None] = None # pragma: no mutate
|
|
32
|
+
reset: ClassVar[bool] = False
|
|
33
|
+
report: ClassVar[bool] = False
|
|
34
34
|
|
|
35
|
-
seed = random.randint(0, 1_000_000)
|
|
35
|
+
seed: ClassVar[int] = random.randint(0, 1_000_000)
|
|
36
36
|
|
|
37
|
-
debug = False
|
|
37
|
+
debug: ClassVar[bool] = False
|
|
38
38
|
|
|
39
39
|
recorded_times: ClassVar[dict] = {}
|
|
40
40
|
item_totals: ClassVar[dict] = {}
|
|
41
41
|
item_sort_keys: ClassVar[dict] = {}
|
|
42
42
|
item_bucket_id: ClassVar[dict] = {}
|
|
43
43
|
bucket_sort_keys: ClassVar[dict] = {}
|
|
44
|
-
|
|
44
|
+
diff_cov_scores: ClassVar[dict] = {}
|
|
45
|
+
mut_cov_scores: ClassVar[dict] = {}
|
|
45
46
|
|
|
46
47
|
@staticmethod
|
|
47
48
|
def from_pytest(config: pytest.Config) -> None:
|
|
@@ -13,7 +13,7 @@ from _pytest import nodes as pytest_nodes
|
|
|
13
13
|
|
|
14
14
|
from pytest_sort.config import SortConfig
|
|
15
15
|
from pytest_sort.database import get_all_totals, get_stats
|
|
16
|
-
from pytest_sort.diffcov import
|
|
16
|
+
from pytest_sort.diffcov import get_diff_test_scores, get_mut_test_scores
|
|
17
17
|
|
|
18
18
|
md5: Callable = hashlib.md5
|
|
19
19
|
if sys.version_info >= (3, 9):
|
|
@@ -84,7 +84,8 @@ create_item_key = {
|
|
|
84
84
|
"md5": lambda item, idx, count: md5(item.nodeid.encode()).digest(),
|
|
85
85
|
"random": lambda item, idx, count: random.random(),
|
|
86
86
|
"fastest": lambda item, idx, count: SortConfig.item_totals.get(item.nodeid, 0),
|
|
87
|
-
"diffcov": lambda item, idx, count: SortConfig.
|
|
87
|
+
"diffcov": lambda item, idx, count: SortConfig.diff_cov_scores.get(item.nodeid, 0),
|
|
88
|
+
"mutcov": lambda item, idx, count: SortConfig.mut_cov_scores.get(item.nodeid, 0),
|
|
88
89
|
}
|
|
89
90
|
|
|
90
91
|
|
|
@@ -95,7 +96,12 @@ def get_bucket_total(bucket_id: str) -> int:
|
|
|
95
96
|
|
|
96
97
|
def get_bucket_score(bucket_id: str) -> int:
|
|
97
98
|
"""Get all scores from nodes matching this bucket and return min."""
|
|
98
|
-
return min([score for nodeid, score in SortConfig.
|
|
99
|
+
return min([score for nodeid, score in SortConfig.diff_cov_scores.items() if nodeid.startswith(bucket_id)] + [0])
|
|
100
|
+
|
|
101
|
+
|
|
102
|
+
def get_mut_bucket_score(bucket_id: str) -> int:
|
|
103
|
+
"""Get all scores from nodes matching this bucket and return min."""
|
|
104
|
+
return min([score for nodeid, score in SortConfig.mut_cov_scores.items() if nodeid.startswith(bucket_id)] + [0])
|
|
99
105
|
|
|
100
106
|
|
|
101
107
|
create_bucket_key = {
|
|
@@ -105,6 +111,7 @@ create_bucket_key = {
|
|
|
105
111
|
"random": lambda bucket_id, idx, count: random.random(),
|
|
106
112
|
"fastest": lambda bucket_id, idx, count: get_bucket_total(bucket_id),
|
|
107
113
|
"diffcov": lambda bucket_id, idx, count: get_bucket_score(bucket_id),
|
|
114
|
+
"mutcov": lambda bucket_id, idx, count: get_mut_bucket_score(bucket_id),
|
|
108
115
|
}
|
|
109
116
|
|
|
110
117
|
|
|
@@ -220,7 +227,10 @@ def sort_items(items: list[pytest.Item]) -> None:
|
|
|
220
227
|
random.seed(SortConfig.seed)
|
|
221
228
|
|
|
222
229
|
if SortConfig.mode == "diffcov" or SortConfig.bucket_mode == "diffcov":
|
|
223
|
-
SortConfig.
|
|
230
|
+
SortConfig.diff_cov_scores = get_diff_test_scores()
|
|
231
|
+
|
|
232
|
+
if SortConfig.mode == "mutcov" or SortConfig.bucket_mode == "mutcov":
|
|
233
|
+
SortConfig.mut_cov_scores = get_mut_test_scores()
|
|
224
234
|
|
|
225
235
|
if SortConfig.mode == "fastest" or SortConfig.bucket_mode == "fastest":
|
|
226
236
|
SortConfig.item_totals = get_all_totals()
|
|
@@ -1,7 +1,8 @@
|
|
|
1
|
-
"""Logic for prioritizing tests by coverage of
|
|
1
|
+
"""Logic for prioritizing tests by coverage of changes."""
|
|
2
2
|
|
|
3
3
|
from __future__ import annotations
|
|
4
4
|
|
|
5
|
+
import os
|
|
5
6
|
import re
|
|
6
7
|
import shlex
|
|
7
8
|
import subprocess
|
|
@@ -27,7 +28,7 @@ def get_git_diff_patch() -> str:
|
|
|
27
28
|
return output.stdout.decode()
|
|
28
29
|
|
|
29
30
|
|
|
30
|
-
def
|
|
31
|
+
def get_diff_changed_lines(git_folder: str, git_diff_patch: str) -> dict[Path, set[int]]:
|
|
31
32
|
"""Use whatthepatch to parse the patch string.
|
|
32
33
|
|
|
33
34
|
Return map of resolved path of file to set of inserted, updated, or deleted line numbers.
|
|
@@ -42,6 +43,23 @@ def get_changed_lines(git_folder: str, git_diff_patch: str) -> dict[Path, set[in
|
|
|
42
43
|
return changed_lines
|
|
43
44
|
|
|
44
45
|
|
|
46
|
+
def get_mut_changed_lines() -> dict[Path, set[int]]:
|
|
47
|
+
"""Use environment variables set by mutation testing tool to determine changed lines.
|
|
48
|
+
|
|
49
|
+
Return map of resolved path of file to set of inserted, updated, or deleted line numbers.
|
|
50
|
+
"""
|
|
51
|
+
changed_lines = {}
|
|
52
|
+
mut_source_file = os.environ.get("MUT_SOURCE_FILE", None)
|
|
53
|
+
mut_lineno = os.environ.get("MUT_LINENO", 0)
|
|
54
|
+
mut_end_lineno = os.environ.get("MUT_END_LINENO", mut_lineno)
|
|
55
|
+
|
|
56
|
+
if mut_source_file and mut_lineno:
|
|
57
|
+
rpath = Path(mut_source_file).resolve()
|
|
58
|
+
changed_lines[rpath] = set(range(int(mut_lineno), int(mut_end_lineno) + 1))
|
|
59
|
+
|
|
60
|
+
return changed_lines
|
|
61
|
+
|
|
62
|
+
|
|
45
63
|
def get_line_coverage() -> Generator[tuple[Path, str, str, int], Any, None]:
|
|
46
64
|
"""Retrieve coverage data from coverage.py.
|
|
47
65
|
|
|
@@ -59,10 +77,9 @@ def get_line_coverage() -> Generator[tuple[Path, str, str, int], Any, None]:
|
|
|
59
77
|
yield rpath, nodeid, when, line
|
|
60
78
|
|
|
61
79
|
|
|
62
|
-
def get_test_scores() -> dict[str, int]:
|
|
63
|
-
"""Genarate a 'score' for each test case that has some coverage of the changed
|
|
80
|
+
def get_test_scores(changed_lines: dict[Path, set[int]]) -> dict[str, int]:
|
|
81
|
+
"""Genarate a 'score' for each test case that has some coverage of the changed lines."""
|
|
64
82
|
test_scores = {}
|
|
65
|
-
changed_lines = get_changed_lines(get_git_toplevel_folder(), get_git_diff_patch())
|
|
66
83
|
for rpath, nodeid, when, line in get_line_coverage():
|
|
67
84
|
if rpath in changed_lines:
|
|
68
85
|
# If test coverage includes changed module
|
|
@@ -77,3 +94,13 @@ def get_test_scores() -> dict[str, int]:
|
|
|
77
94
|
else:
|
|
78
95
|
test_scores[nodeid] -= 1
|
|
79
96
|
return test_scores
|
|
97
|
+
|
|
98
|
+
|
|
99
|
+
def get_diff_test_scores() -> dict[str, int]:
|
|
100
|
+
"""Genarate a 'score' for each test case that has some coverage of the changed files."""
|
|
101
|
+
return get_test_scores(get_diff_changed_lines(get_git_toplevel_folder(), get_git_diff_patch()))
|
|
102
|
+
|
|
103
|
+
|
|
104
|
+
def get_mut_test_scores() -> dict[str, int]:
|
|
105
|
+
"""Genarate a 'score' for each test case that has some coverage of the mutated lines."""
|
|
106
|
+
return get_test_scores(get_mut_changed_lines())
|
|
@@ -0,0 +1,107 @@
|
|
|
1
|
+
Metadata-Version: 2.1
|
|
2
|
+
Name: pytest-sort
|
|
3
|
+
Version: 1.3.0
|
|
4
|
+
Summary: Tools for sorting test cases
|
|
5
|
+
Author-email: WiredNerd <pbuschmail-pytestsort@yahoo.com>
|
|
6
|
+
Maintainer-email: WiredNerd <pbuschmail-pytestsort@yahoo.com>
|
|
7
|
+
License: MIT License
|
|
8
|
+
|
|
9
|
+
Copyright (c) 2023 Peter Busch
|
|
10
|
+
|
|
11
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
12
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
13
|
+
in the Software without restriction, including without limitation the rights
|
|
14
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
15
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
16
|
+
furnished to do so, subject to the following conditions:
|
|
17
|
+
|
|
18
|
+
The above copyright notice and this permission notice shall be included in all
|
|
19
|
+
copies or substantial portions of the Software.
|
|
20
|
+
|
|
21
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
22
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
23
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
24
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
25
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
26
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
27
|
+
SOFTWARE.
|
|
28
|
+
|
|
29
|
+
Project-URL: Documentation, https://pytest-sort.readthedocs.io/
|
|
30
|
+
Project-URL: Homepage, https://github.com/WiredNerd/pytest-sort
|
|
31
|
+
Project-URL: Issue Tracker, https://github.com/WiredNerd/pytest-sort/issues
|
|
32
|
+
Keywords: test,pytest,order,pytest-order,sort,random,pytest-random,pytest-random-order,randomly,pytest-randomly
|
|
33
|
+
Classifier: Programming Language :: Python :: 3
|
|
34
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
35
|
+
Classifier: Operating System :: OS Independent
|
|
36
|
+
Classifier: Framework :: Pytest
|
|
37
|
+
Requires-Python: >=3.8
|
|
38
|
+
Description-Content-Type: text/markdown
|
|
39
|
+
License-File: LICENSE
|
|
40
|
+
Requires-Dist: pytest>=7.4.0
|
|
41
|
+
Requires-Dist: whatthepatch>=1.0.5
|
|
42
|
+
Requires-Dist: coverage>=7.3.1
|
|
43
|
+
|
|
44
|
+
[](https://github.com/WiredNerd/pytest-sort)
|
|
45
|
+
[](https://pypi.org/project/pytest-sort)
|
|
46
|
+
[](https://pypi.org/project/pytest-sort)
|
|
47
|
+
[](https://pypistats.org/packages/pytest-sort)
|
|
48
|
+
[](https://github.com/WiredNerd/pytest-sort/blob/main/LICENSE)
|
|
49
|
+
|
|
50
|
+
[](https://pytest-cov.readthedocs.io)
|
|
51
|
+
[](https://poodle.readthedocs.io/)
|
|
52
|
+
[](https://pytest-sort.readthedocs.io/)
|
|
53
|
+
|
|
54
|
+
[](https://pycqa.github.io/isort/)
|
|
55
|
+
[](https://black.readthedocs.io)
|
|
56
|
+
[](https://beta.ruff.rs/docs/)
|
|
57
|
+
[](https://snyk.io/)
|
|
58
|
+
|
|
59
|
+
|
|
60
|
+
|
|
61
|
+
# Pytest-Sort
|
|
62
|
+
|
|
63
|
+
This pytest plugin is used to help identify application state leaks in your test suite. It does this by automatically changing the execution order of test cases.
|
|
64
|
+
|
|
65
|
+
How frustrating is it to have a test case that works fine when runs by itself, but fails when run with the rest of the suite? By running tests in a random order, the culprit of such problems can be identified much earlier in the development process. Alternately, running in ordered and reverse order can help find the cause of the problem in existing test suites.
|
|
66
|
+
|
|
67
|
+
## Features
|
|
68
|
+
|
|
69
|
+
* Multiple sort options including: Random, Fastest, Reverse, and more.
|
|
70
|
+
* Group test cases into buckets that are each sorted separately.
|
|
71
|
+
* Control sort order of the buckets.
|
|
72
|
+
* Use Pytest markers to always run specific test cases in order.
|
|
73
|
+
|
|
74
|
+
## Quick Start
|
|
75
|
+
|
|
76
|
+
Installation:
|
|
77
|
+
|
|
78
|
+
```
|
|
79
|
+
pip install pytest-sort --upgrade
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
Running with pytest-sort:
|
|
83
|
+
|
|
84
|
+
```
|
|
85
|
+
pytest --sort-mode=random
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
## Documentation:
|
|
89
|
+
|
|
90
|
+
- Usage, Examples, Configuration, and more: https://pytest-sort.readthedocs.io/
|
|
91
|
+
|
|
92
|
+
## Contribute
|
|
93
|
+
|
|
94
|
+
- Issue Tracker: https://github.com/WiredNerd/pytest-sort/issues
|
|
95
|
+
- Source Code: https://github.com/WiredNerd/pytest-sort
|
|
96
|
+
|
|
97
|
+
## Support
|
|
98
|
+
|
|
99
|
+
If you are having issues, please let us know.
|
|
100
|
+
|
|
101
|
+
I can be contacted at: pbuschmail-pytestsort@yahoo.com
|
|
102
|
+
|
|
103
|
+
Or by opening an issue: https://github.com/WiredNerd/pytest-sort/issues
|
|
104
|
+
|
|
105
|
+
## License
|
|
106
|
+
|
|
107
|
+
The project is licensed under the MIT license.
|