pytest-receptor 1.0.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- pytest_receptor-1.0.0/CHANGELOG.md +100 -0
- pytest_receptor-1.0.0/LICENSE +21 -0
- pytest_receptor-1.0.0/MANIFEST.in +1 -0
- pytest_receptor-1.0.0/PKG-INFO +440 -0
- pytest_receptor-1.0.0/README.md +383 -0
- pytest_receptor-1.0.0/pyproject.toml +109 -0
- pytest_receptor-1.0.0/pytest_receptor/__init__.py +33 -0
- pytest_receptor-1.0.0/pytest_receptor/_version.py +1 -0
- pytest_receptor-1.0.0/pytest_receptor/artifact.py +295 -0
- pytest_receptor-1.0.0/pytest_receptor/model.py +182 -0
- pytest_receptor-1.0.0/pytest_receptor/plugin.py +1902 -0
- pytest_receptor-1.0.0/pytest_receptor.egg-info/PKG-INFO +440 -0
- pytest_receptor-1.0.0/pytest_receptor.egg-info/SOURCES.txt +22 -0
- pytest_receptor-1.0.0/pytest_receptor.egg-info/dependency_links.txt +1 -0
- pytest_receptor-1.0.0/pytest_receptor.egg-info/entry_points.txt +2 -0
- pytest_receptor-1.0.0/pytest_receptor.egg-info/requires.txt +38 -0
- pytest_receptor-1.0.0/pytest_receptor.egg-info/top_level.txt +1 -0
- pytest_receptor-1.0.0/setup.cfg +4 -0
- pytest_receptor-1.0.0/tests/test_artifact.py +324 -0
- pytest_receptor-1.0.0/tests/test_model.py +117 -0
- pytest_receptor-1.0.0/tests/test_output_contract.py +63 -0
- pytest_receptor-1.0.0/tests/test_packaging.py +121 -0
- pytest_receptor-1.0.0/tests/test_parity.py +378 -0
- pytest_receptor-1.0.0/tests/test_plugin.py +1619 -0
|
@@ -0,0 +1,100 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project are recorded here. The format follows
|
|
4
|
+
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and releases follow
|
|
5
|
+
[Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
6
|
+
|
|
7
|
+
## [Unreleased]
|
|
8
|
+
|
|
9
|
+
## [1.0.0] - 2026-08-12
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
|
|
13
|
+
- A normalized evidence model shared by terminal rendering and machine output.
|
|
14
|
+
- The opt-in `pytest-receptor.events@1` JSONL artifact and supported reader.
|
|
15
|
+
- Exact text-contract tests and differential pytest/JUnit parity tests.
|
|
16
|
+
- Tested coexistence with xdist, coverage, reruns, subtests, and JUnit.
|
|
17
|
+
- Auditable artifact size limits and reproducible wall-time/peak-RSS benchmarks.
|
|
18
|
+
- Stable root-cause fingerprints and recoverable, auditable terminal truncation.
|
|
19
|
+
|
|
20
|
+
### Fixed
|
|
21
|
+
|
|
22
|
+
- Mixed valid and nonexistent paths under xdist now retain an actionable invalid
|
|
23
|
+
selection diagnostic while preserving pytest's exit status.
|
|
24
|
+
|
|
25
|
+
### Changed
|
|
26
|
+
|
|
27
|
+
- CI, documentation, packaging metadata, and release gates now exercise the
|
|
28
|
+
declared Python and pytest support matrix.
|
|
29
|
+
- Rerun attempts and subtests retain distinct normalized identities without
|
|
30
|
+
inflating logical-test counts.
|
|
31
|
+
|
|
32
|
+
## [0.7.0] - 2026-08-02
|
|
33
|
+
|
|
34
|
+
### Changed
|
|
35
|
+
|
|
36
|
+
- Consolidated the post-0.6 development guide and proposal triage.
|
|
37
|
+
|
|
38
|
+
## [0.6.0] - 2026-07-28
|
|
39
|
+
|
|
40
|
+
### Changed
|
|
41
|
+
|
|
42
|
+
- Rebuilt compact reporting around pytest's public hooks and truth-preserving
|
|
43
|
+
exit semantics.
|
|
44
|
+
- Added owner-only full reports, credential-pattern redaction, deterministic
|
|
45
|
+
failure grouping, warning visibility, progress, and real-suite regressions.
|
|
46
|
+
|
|
47
|
+
## [0.5.0] - 2026-07-17
|
|
48
|
+
|
|
49
|
+
### Added
|
|
50
|
+
|
|
51
|
+
- Experimental integration and event-reader work, subsequently redesigned by
|
|
52
|
+
the 0.6 correctness audit.
|
|
53
|
+
|
|
54
|
+
## [0.4.0] - 2026-07-17
|
|
55
|
+
|
|
56
|
+
### Added
|
|
57
|
+
|
|
58
|
+
- Experimental CI watchdog behavior, subsequently removed in 0.6.
|
|
59
|
+
|
|
60
|
+
## [0.3.0] - 2026-07-17
|
|
61
|
+
|
|
62
|
+
### Added
|
|
63
|
+
|
|
64
|
+
- Initial event-model and semantic-budget experiments.
|
|
65
|
+
|
|
66
|
+
## [0.2.0] - 2026-07-17
|
|
67
|
+
|
|
68
|
+
### Added
|
|
69
|
+
|
|
70
|
+
- Initial correctness and outcome handling.
|
|
71
|
+
|
|
72
|
+
## [0.1.2] - 2026-07-17
|
|
73
|
+
|
|
74
|
+
### Added
|
|
75
|
+
|
|
76
|
+
- Conda build recipe.
|
|
77
|
+
|
|
78
|
+
## [0.1.1] - 2026-07-17
|
|
79
|
+
|
|
80
|
+
### Changed
|
|
81
|
+
|
|
82
|
+
- English documentation and explicit Python support metadata.
|
|
83
|
+
|
|
84
|
+
## [0.1.0] - 2026-07-17
|
|
85
|
+
|
|
86
|
+
### Added
|
|
87
|
+
|
|
88
|
+
- Human, LLM, and CI output profiles.
|
|
89
|
+
|
|
90
|
+
[Unreleased]: https://github.com/uibcdf/pytest-receptor/compare/1.0.0...HEAD
|
|
91
|
+
[1.0.0]: https://github.com/uibcdf/pytest-receptor/compare/0.7.0...1.0.0
|
|
92
|
+
[0.7.0]: https://github.com/uibcdf/pytest-receptor/compare/0.6.0...0.7.0
|
|
93
|
+
[0.6.0]: https://github.com/uibcdf/pytest-receptor/compare/0.5.0...0.6.0
|
|
94
|
+
[0.5.0]: https://github.com/uibcdf/pytest-receptor/compare/0.4.0...0.5.0
|
|
95
|
+
[0.4.0]: https://github.com/uibcdf/pytest-receptor/compare/0.3.0...0.4.0
|
|
96
|
+
[0.3.0]: https://github.com/uibcdf/pytest-receptor/compare/0.2.0...0.3.0
|
|
97
|
+
[0.2.0]: https://github.com/uibcdf/pytest-receptor/compare/0.1.2...0.2.0
|
|
98
|
+
[0.1.2]: https://github.com/uibcdf/pytest-receptor/compare/0.1.1...0.1.2
|
|
99
|
+
[0.1.1]: https://github.com/uibcdf/pytest-receptor/compare/0.1.0...0.1.1
|
|
100
|
+
[0.1.0]: https://github.com/uibcdf/pytest-receptor/releases/tag/0.1.0
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Unit of Research on Computational Biology and Drug Design. Mexico Children's Hospital Federico Gómez.
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
include CHANGELOG.md
|
|
@@ -0,0 +1,440 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: pytest-receptor
|
|
3
|
+
Version: 1.0.0
|
|
4
|
+
Summary: A pytest plugin to adapt test output for different receptors (humans, LLMs, CI).
|
|
5
|
+
Author: UIBCDF Development Team
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/uibcdf/pytest-receptor
|
|
8
|
+
Project-URL: Repository, https://github.com/uibcdf/pytest-receptor
|
|
9
|
+
Project-URL: Issues, https://github.com/uibcdf/pytest-receptor/issues
|
|
10
|
+
Project-URL: Changelog, https://github.com/uibcdf/pytest-receptor/blob/main/CHANGELOG.md
|
|
11
|
+
Keywords: pytest,plugin,testing,LLM,CI
|
|
12
|
+
Classifier: Development Status :: 5 - Production/Stable
|
|
13
|
+
Classifier: Framework :: Pytest
|
|
14
|
+
Classifier: Intended Audience :: Developers
|
|
15
|
+
Classifier: Operating System :: OS Independent
|
|
16
|
+
Classifier: Programming Language :: Python :: 3
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
20
|
+
Classifier: Topic :: Software Development :: Testing
|
|
21
|
+
Requires-Python: <3.14,>=3.11
|
|
22
|
+
Description-Content-Type: text/markdown
|
|
23
|
+
License-File: LICENSE
|
|
24
|
+
Requires-Dist: pytest>=8.0.0
|
|
25
|
+
Provides-Extra: test
|
|
26
|
+
Requires-Dist: packaging>=24; extra == "test"
|
|
27
|
+
Requires-Dist: pytest-cov<8,>=6; extra == "test"
|
|
28
|
+
Requires-Dist: pytest-rerunfailures<17,>=15; extra == "test"
|
|
29
|
+
Requires-Dist: pytest-subtests<0.16,>=0.14; extra == "test"
|
|
30
|
+
Requires-Dist: pytest-xdist<4,>=3.6; extra == "test"
|
|
31
|
+
Provides-Extra: lint
|
|
32
|
+
Requires-Dist: ruff==0.16.1; extra == "lint"
|
|
33
|
+
Provides-Extra: docs
|
|
34
|
+
Requires-Dist: myst-parser<5,>=4; extra == "docs"
|
|
35
|
+
Requires-Dist: sphinx<10,>=8; extra == "docs"
|
|
36
|
+
Requires-Dist: sphinx-rtd-theme<4,>=3; extra == "docs"
|
|
37
|
+
Provides-Extra: benchmark
|
|
38
|
+
Requires-Dist: tiktoken<1,>=0.9; extra == "benchmark"
|
|
39
|
+
Provides-Extra: release
|
|
40
|
+
Requires-Dist: build<2,>=1.2; extra == "release"
|
|
41
|
+
Requires-Dist: packaging>=24; extra == "release"
|
|
42
|
+
Requires-Dist: twine<7,>=6; extra == "release"
|
|
43
|
+
Provides-Extra: dev
|
|
44
|
+
Requires-Dist: build<2,>=1.2; extra == "dev"
|
|
45
|
+
Requires-Dist: myst-parser<5,>=4; extra == "dev"
|
|
46
|
+
Requires-Dist: packaging>=24; extra == "dev"
|
|
47
|
+
Requires-Dist: pytest-cov<8,>=6; extra == "dev"
|
|
48
|
+
Requires-Dist: pytest-rerunfailures<17,>=15; extra == "dev"
|
|
49
|
+
Requires-Dist: pytest-subtests<0.16,>=0.14; extra == "dev"
|
|
50
|
+
Requires-Dist: pytest-xdist<4,>=3.6; extra == "dev"
|
|
51
|
+
Requires-Dist: ruff==0.16.1; extra == "dev"
|
|
52
|
+
Requires-Dist: sphinx<10,>=8; extra == "dev"
|
|
53
|
+
Requires-Dist: sphinx-rtd-theme<4,>=3; extra == "dev"
|
|
54
|
+
Requires-Dist: tiktoken<1,>=0.9; extra == "dev"
|
|
55
|
+
Requires-Dist: twine<7,>=6; extra == "dev"
|
|
56
|
+
Dynamic: license-file
|
|
57
|
+
|
|
58
|
+
# pytest-receptor
|
|
59
|
+
|
|
60
|
+
[](https://github.com/uibcdf/pytest-receptor/actions/workflows/tests.yml)
|
|
61
|
+
[](https://opensource.org/licenses/MIT)
|
|
62
|
+
[](https://www.python.org/)
|
|
63
|
+
[](https://docs.pytest.org/)
|
|
64
|
+
|
|
65
|
+
A pytest reporter for coding agents.
|
|
66
|
+
|
|
67
|
+
When pytest is driven by an agent such as Claude Code, Codex, or an autonomous
|
|
68
|
+
TDD loop, its output is read by something that pays for every token and cannot
|
|
69
|
+
scroll back. `pytest-receptor` renders the same run for that consumer: it says
|
|
70
|
+
what happened, groups repeated failures by root cause, and tells the agent
|
|
71
|
+
exactly what to re-run.
|
|
72
|
+
|
|
73
|
+
> **1.0 release candidate.** The reliability, CLI, outcome, and
|
|
74
|
+
> `pytest-receptor.events@1` compatibility contracts are frozen. The receptor
|
|
75
|
+
> never reports an unsuccessful or incomplete run as a success, and a failure
|
|
76
|
+
> inside the receptor itself never costs you the run.
|
|
77
|
+
|
|
78
|
+
---
|
|
79
|
+
|
|
80
|
+
## The problem
|
|
81
|
+
|
|
82
|
+
Your agent was told to run the tests, so it runs `pytest`, and pytest answers
|
|
83
|
+
the way it has always answered: for a human sitting at a terminal. Banner,
|
|
84
|
+
`rootdir`, plugin list, progress bar, and the full source of every failing test.
|
|
85
|
+
The agent pays for all of it, on every iteration.
|
|
86
|
+
|
|
87
|
+
Tuning the flags does not fix it. `pytest -q --tb=line` is compact but strips
|
|
88
|
+
the assertion diff, so the agent guesses and loops. And nothing pytest offers
|
|
89
|
+
handles the case that hurts most: one broken fixture fails forty tests, and the
|
|
90
|
+
agent reads the same traceback forty times.
|
|
91
|
+
|
|
92
|
+
## What it does
|
|
93
|
+
|
|
94
|
+
```console
|
|
95
|
+
$ pytest --receptor=llm
|
|
96
|
+
|
|
97
|
+
FAIL exit=1 | 38 errors, 90 passed | 12.40s | 1 root cause
|
|
98
|
+
|
|
99
|
+
[1] TypeError | 38 tests | setup
|
|
100
|
+
conftest.py:31
|
|
101
|
+
TypeError: 'NoneType' object is not subscriptable
|
|
102
|
+
tests:
|
|
103
|
+
tests/test_merge.py::test_merge[0]
|
|
104
|
+
tests/test_merge.py::test_merge[1]
|
|
105
|
+
tests/test_merge.py::test_merge[2]
|
|
106
|
+
+35 more
|
|
107
|
+
rerun: pytest tests/test_merge.py -q
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
That is 109 tokens. Plain `pytest` spends 3,300 on the same run.
|
|
111
|
+
|
|
112
|
+
---
|
|
113
|
+
|
|
114
|
+
## Install
|
|
115
|
+
|
|
116
|
+
```bash
|
|
117
|
+
conda install -c uibcdf pytest-receptor
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
Version 1.0 is planned as the first PyPI release (`pip install
|
|
121
|
+
pytest-receptor`). Its official `pytest11` metadata makes pytest discover it
|
|
122
|
+
automatically after installation. Until then, use conda or install from source.
|
|
123
|
+
See the [installation guide](https://uibcdf.github.io/pytest-receptor/installation.html).
|
|
124
|
+
|
|
125
|
+
Requires Python 3.11-3.13 and pytest 8 or later. Every combination of
|
|
126
|
+
Python 3.11/3.12/3.13 with pytest 8 and 9 is exercised in CI, so the support
|
|
127
|
+
claim is evidence rather than intent.
|
|
128
|
+
|
|
129
|
+
## Use
|
|
130
|
+
|
|
131
|
+
```bash
|
|
132
|
+
pytest # unchanged pytest -- installing this changes nothing
|
|
133
|
+
pytest --receptor=llm # compact output for a coding agent
|
|
134
|
+
pytest --receptor=ci # compact output for a CI log
|
|
135
|
+
pytest --receptor=human # the default, stated explicitly
|
|
136
|
+
pytest --receptor=llm --receptor-full # expand every failure group
|
|
137
|
+
pytest --receptor=llm --receptor-stats # what did this actually save?
|
|
138
|
+
pytest --receptor=llm --receptor-events=events.jsonl # normalized evidence
|
|
139
|
+
pytest --receptor=llm --receptor-events=events.jsonl \
|
|
140
|
+
--receptor-events-max-bytes=104857600 # explicit artifact cap
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
The opt-in JSONL stream is versioned, owner-only, integrity-checked on
|
|
144
|
+
finalization, and readable with `pytest_receptor.read_artifact`. A missing final
|
|
145
|
+
session record is reported as incomplete. See the
|
|
146
|
+
[artifact guide](https://uibcdf.github.io/pytest-receptor/artifacts.html).
|
|
147
|
+
|
|
148
|
+
**Installing the plugin does not change anything until you ask it to.** The
|
|
149
|
+
default is `human`, and `human` is a true passthrough: the plugin registers
|
|
150
|
+
nothing at all, so `pytest` on its own produces output byte-identical to not
|
|
151
|
+
having it installed. There is a test asserting exactly that.
|
|
152
|
+
|
|
153
|
+
This matters in a shared environment. You can install it for yourself, or for
|
|
154
|
+
your agent, without altering what anyone else sees.
|
|
155
|
+
|
|
156
|
+
**`llm` vs `ci` — one question: will the reader be able to open the on-disk
|
|
157
|
+
report?** `llm` assumes yes (the agent shares the filesystem), so on a
|
|
158
|
+
pathological spread of failures — past ~10 distinct root causes — it shows the
|
|
159
|
+
first ten in full and points at the report for the rest, which the agent opens
|
|
160
|
+
only if it needs to. `ci` assumes no: a CI runner is destroyed at job end and the
|
|
161
|
+
log gets one shot, so nothing is held back and no report path is printed — every
|
|
162
|
+
root cause is expanded inline. Same renderer and same truth-preserving rules;
|
|
163
|
+
they differ only in what survives for the reader, so neither is simply "more"
|
|
164
|
+
than the other. Details in the
|
|
165
|
+
[usage guide](https://uibcdf.github.io/pytest-receptor/usage.html#choosing-between-llm-and-ci).
|
|
166
|
+
|
|
167
|
+
You do not need to combine these with pytest's own quieting flags —
|
|
168
|
+
`--receptor=llm` already sets the equivalent of `-qq --no-header --no-summary`,
|
|
169
|
+
so adding them changes nothing. One caveat: do **not** pass `--tb=line` or
|
|
170
|
+
`--tb=no`. Those control how pytest *builds* the traceback, not how it prints it,
|
|
171
|
+
so they save a handful of tokens and silently cost the receptor the call chain.
|
|
172
|
+
See the [usage guide](https://uibcdf.github.io/pytest-receptor/usage.html).
|
|
173
|
+
|
|
174
|
+
---
|
|
175
|
+
|
|
176
|
+
## Behavior
|
|
177
|
+
|
|
178
|
+
### It always tells you what actually happened
|
|
179
|
+
|
|
180
|
+
The numeric exit status comes from pytest, never from the absence of failure
|
|
181
|
+
reports. The label refines that status only when pytest merges distinct states:
|
|
182
|
+
collection errors and interrupts both use exit 2, and xdist can return exit 5
|
|
183
|
+
for an invocation that contains nonexistent filesystem targets.
|
|
184
|
+
|
|
185
|
+
```text
|
|
186
|
+
PASS exit=0 | 126 passed, 2 skipped | 4.21s | 3 warnings
|
|
187
|
+
FAIL exit=1 | 2 failed, 87 passed | 12.40s | 2 root causes
|
|
188
|
+
NO_TESTS exit=5
|
|
189
|
+
INTERRUPTED exit=2 | incomplete: 12 of 128 executed
|
|
190
|
+
COLLECTION_ERROR exit=2
|
|
191
|
+
USAGE_ERROR exit=5 | invalid selection
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
A run stopped by `-x`, `--maxfail`, or an interrupt is marked incomplete even
|
|
195
|
+
when nothing failed, so a partial run cannot be mistaken for a clean one.
|
|
196
|
+
|
|
197
|
+
### It groups by root cause, not by test
|
|
198
|
+
|
|
199
|
+
Failures are grouped by exception type, phase, crash location, and cause chain.
|
|
200
|
+
Forty tests broken by one fixture become one group that keeps all forty test
|
|
201
|
+
IDs, and a parametrized test failing on twenty inputs is one bug with twenty
|
|
202
|
+
messages rather than twenty bugs. Failures crashing in unrelated places stay
|
|
203
|
+
separate, and so do two failures wrapping different underlying errors.
|
|
204
|
+
|
|
205
|
+
`raise X from Y` reports both, because the wrapper is usually the less
|
|
206
|
+
informative half:
|
|
207
|
+
|
|
208
|
+
```text
|
|
209
|
+
ValueError: could not build topology
|
|
210
|
+
caused by: KeyError: 'atoms'
|
|
211
|
+
```
|
|
212
|
+
|
|
213
|
+
### It gives you everything on stdout
|
|
214
|
+
|
|
215
|
+
Every root cause is rendered in full. Grouping has already collapsed the volume,
|
|
216
|
+
so withholding on top of it saves almost nothing and costs double if you then
|
|
217
|
+
have to read the file back: measured at five distinct causes, holding back saved
|
|
218
|
+
forty tokens and cost two hundred. Only a pathological spread — more than ten
|
|
219
|
+
distinct causes — is summarized, and only when the on-disk report exists to hold
|
|
220
|
+
what was left out.
|
|
221
|
+
|
|
222
|
+
Occurrence lists are the exception, and for the opposite reason: a group of
|
|
223
|
+
thirty-eight failing tests names three and counts the rest, because the rerun
|
|
224
|
+
command already selects all of them.
|
|
225
|
+
|
|
226
|
+
### It says why tests were skipped
|
|
227
|
+
|
|
228
|
+
A suite built on optional dependencies skips in the hundreds, and `412 skipped`
|
|
229
|
+
does not say which capability is missing:
|
|
230
|
+
|
|
231
|
+
```text
|
|
232
|
+
skipped: 412 in 3 groups
|
|
233
|
+
x380 | openmm not installed
|
|
234
|
+
x30 | requires a GPU
|
|
235
|
+
x2 | (no reason declared)
|
|
236
|
+
```
|
|
237
|
+
|
|
238
|
+
The last group is deliberate: a skip nobody documented is worth knowing about.
|
|
239
|
+
Warnings and xfails are grouped the same way. Each section is bounded by the
|
|
240
|
+
variety of reasons rather than the number of tests.
|
|
241
|
+
|
|
242
|
+
### It degrades safely
|
|
243
|
+
|
|
244
|
+
If the receptor itself raises, you get `RECEPTOR_ERROR`, the underlying
|
|
245
|
+
exception, the raw pytest evidence, and pytest's original exit status. The worst
|
|
246
|
+
case of enabling this plugin is standard pytest plus one line of noise.
|
|
247
|
+
|
|
248
|
+
Test output is treated as untrusted input: ANSI escapes and control characters
|
|
249
|
+
are stripped, values that look like credentials are redacted before anything is
|
|
250
|
+
rendered or written, and no text produced by a test can forge a verdict line.
|
|
251
|
+
The redaction is a conservative net for obvious shapes — `api_key=`, `Bearer ...`
|
|
252
|
+
— not a security boundary.
|
|
253
|
+
|
|
254
|
+
### It shows the run is alive
|
|
255
|
+
|
|
256
|
+
pytest streams a progress character per test; suppressing those leaves a long
|
|
257
|
+
suite completely silent. Progress now goes to **stderr** — never stdout, so the
|
|
258
|
+
report stays as parseable as before — once per twenty percent of the suite:
|
|
259
|
+
|
|
260
|
+
```text
|
|
261
|
+
receptor: 20% 1867/9332 52s
|
|
262
|
+
receptor: 40% 3733/9332 108s
|
|
263
|
+
```
|
|
264
|
+
|
|
265
|
+
Reporting by percentage rather than by clock bounds this at five lines whether
|
|
266
|
+
the run takes five minutes or three hours, and the elapsed time exposes pace: a
|
|
267
|
+
step that suddenly takes four times longer is visible before the run ends.
|
|
268
|
+
|
|
269
|
+
It is a liveness signal, not a hang detector: the line is emitted when a test
|
|
270
|
+
finishes, so a stuck test produces no further output. What survives is how far
|
|
271
|
+
the run got, which is what you want when something kills it.
|
|
272
|
+
|
|
273
|
+
### It works under `pytest-xdist`
|
|
274
|
+
|
|
275
|
+
A distributed run produces **byte-identical output to a serial one**. Workers
|
|
276
|
+
finish in arbitrary order, so occurrences and groups are given a total order
|
|
277
|
+
before rendering; otherwise the same failure would render differently on every
|
|
278
|
+
run. Counts, grouping, and exit status are unaffected by `-n`.
|
|
279
|
+
|
|
280
|
+
Worker identity is not reported, and that is a decision rather than a gap. The
|
|
281
|
+
signal it would provide — a group of failures landing entirely on one worker —
|
|
282
|
+
is confounded by the distribution mode: under `--dist loadfile` or `loadscope`,
|
|
283
|
+
failures from one file land on one worker by construction. The bare identifier
|
|
284
|
+
without execution order also does not help reproduce anything, which is what
|
|
285
|
+
`-n0` is for.
|
|
286
|
+
|
|
287
|
+
---
|
|
288
|
+
|
|
289
|
+
## How it works
|
|
290
|
+
|
|
291
|
+
Worth knowing before you trust it with your suite.
|
|
292
|
+
|
|
293
|
+
**It does not replace pytest's reporter.** Earlier versions unregistered
|
|
294
|
+
pytest's `TerminalReporter` and substituted a subclass of it. This one leaves it
|
|
295
|
+
in place — so any plugin that looks it up still finds it — and quietens it
|
|
296
|
+
through its documented options: `verbose = -2` and `no_header`, an emptied
|
|
297
|
+
`reportchars`, and a wrapper around `pytest_report_teststatus` that drops the
|
|
298
|
+
progress characters while preserving pytest's own categorization.
|
|
299
|
+
|
|
300
|
+
`no_summary` is deliberately *not* used, although it looks like the obvious
|
|
301
|
+
switch. It gates the whole `pytest_terminal_summary` hook, which is where
|
|
302
|
+
third-party plugins write, and setting it swallowed pytest-cov's report
|
|
303
|
+
entirely. Reported upstream as
|
|
304
|
+
[pytest#14724](https://github.com/pytest-dev/pytest/issues/14724).
|
|
305
|
+
|
|
306
|
+
**It collects from public hooks.** `pytest_runtest_logreport` for phase results,
|
|
307
|
+
`pytest_collectreport` for collection failures, `pytest_warning_recorded` for
|
|
308
|
+
warnings, and `pytest_sessionfinish` to render. Rendering happens in
|
|
309
|
+
`sessionfinish` rather than `terminal_summary` because pytest does not call the
|
|
310
|
+
latter for internal errors, and an internal error is exactly when you most need
|
|
311
|
+
to be told the truth.
|
|
312
|
+
|
|
313
|
+
**Grouping is call-site aware.** The key is exception type, phase, crash
|
|
314
|
+
location, and cause chain. The message is deliberately excluded: keying on it
|
|
315
|
+
fragmented a parametrized test into one group per input, which defeats grouping
|
|
316
|
+
exactly where suites are most repetitive. Differing messages are kept as
|
|
317
|
+
variants inside the group and shown. Crash location rather than test line is
|
|
318
|
+
what makes this right — a bug in `merge.py:117` groups every caller.
|
|
319
|
+
|
|
320
|
+
**Tracebacks keep the decisive frame.** Every local frame is kept, because that
|
|
321
|
+
is the code you can change. External frames are pruned to the boundary you
|
|
322
|
+
entered the dependency at and the frame that actually broke, with elisions
|
|
323
|
+
marked:
|
|
324
|
+
|
|
325
|
+
```text
|
|
326
|
+
frames: tests/test_merge.py:12 -> molsysmt/merge.py:41 -> numpy/core/shape.py:88 (ext) -> ... -> numpy/core/_methods.py:52 (ext)
|
|
327
|
+
```
|
|
328
|
+
|
|
329
|
+
Dropping external frames entirely is cheaper, and wrong: when a failure
|
|
330
|
+
originates inside NumPy or a serializer, the external frame *is* the answer.
|
|
331
|
+
|
|
332
|
+
**Nothing is thrown away, and nothing is deferred.** Grouping is a presentation
|
|
333
|
+
decision; every occurrence keeps its node ID, phase, and location. The complete
|
|
334
|
+
report is written to `.pytest_cache/d/receptor/last-run.txt` while the run is
|
|
335
|
+
still going, owner-only and refusing symlinks. Detail is only ever held back
|
|
336
|
+
when that file exists to hold it, so a consumer can never be left with
|
|
337
|
+
information reachable solely by running the suite again.
|
|
338
|
+
|
|
339
|
+
**`--tb` is deliberately left alone.** It controls how pytest *builds*
|
|
340
|
+
`longrepr`, not how it prints it. Forcing `--tb=no` would look like a sensible
|
|
341
|
+
way to suppress tracebacks and would silently destroy every frame this plugin
|
|
342
|
+
exists to summarize.
|
|
343
|
+
|
|
344
|
+
---
|
|
345
|
+
|
|
346
|
+
## What your agent is doing right now
|
|
347
|
+
|
|
348
|
+
Your agent runs `pytest`. Plain, because that is the obvious command and nobody
|
|
349
|
+
told it otherwise. So every test run spends tokens on a platform banner, a
|
|
350
|
+
`rootdir` line, a plugin list, a progress bar, and the source code of every
|
|
351
|
+
failing test — none of which the agent needs, all of which you pay for, on every
|
|
352
|
+
iteration of every loop.
|
|
353
|
+
|
|
354
|
+
Measured with `tiktoken` (`cl100k_base`):
|
|
355
|
+
|
|
356
|
+
| Scenario | `pytest` | `--receptor=llm` | Change |
|
|
357
|
+
| :--- | ---: | ---: | ---: |
|
|
358
|
+
| Cascade (38 failures, one cause) | 3300 | **105** | **-96.8%** |
|
|
359
|
+
| Green suite (128 tests) | 118 | **15** | -87.3% |
|
|
360
|
+
| Green with warnings | 181 | **45** | -75.1% |
|
|
361
|
+
| Green with many distinct warnings | 1692 | **664** | -60.8% |
|
|
362
|
+
| Single assertion failure | 349 | **165** | -52.7% |
|
|
363
|
+
| Five distinct causes | 405 | **211** | -47.9% |
|
|
364
|
+
| Collection error | 286 | **213** | -25.5% |
|
|
365
|
+
| Mixed states (skip, xfail, xpass) | 124 | **77** | -37.9% |
|
|
366
|
+
|
|
367
|
+
Every scenario is cheaper, most of them by half or better. In a TDD loop that
|
|
368
|
+
runs the suite twenty times, the cascade row alone is sixty thousand tokens.
|
|
369
|
+
|
|
370
|
+
And the saving grows with the suite. Measured on eight thousand tests under
|
|
371
|
+
twelve xdist workers, against `pytest -q -n 12` — a pytest that has *already*
|
|
372
|
+
been quietened:
|
|
373
|
+
|
|
374
|
+
| Scenario | `-q -n 12` | `--receptor=llm -n 12` | Saving |
|
|
375
|
+
| :--- | ---: | ---: | ---: |
|
|
376
|
+
| Whole suite green | 812 | **17** | 97.9% |
|
|
377
|
+
| One fixture breaks 200 tests | 25,681 | **107** | 99.6% |
|
|
378
|
+
| Six unrelated bugs | 1,503 | **278** | 81.5% |
|
|
379
|
+
|
|
380
|
+
`-q` prints one progress character per test, so a *successful* eight-thousand
|
|
381
|
+
test run costs 812 tokens of dots before anything has gone wrong.
|
|
382
|
+
|
|
383
|
+
### And if you already tuned pytest
|
|
384
|
+
|
|
385
|
+
If you are the kind of person who already runs `pytest -q --no-header
|
|
386
|
+
--tb=short`, the picture is narrower and you deserve to see that too:
|
|
387
|
+
|
|
388
|
+
| Scenario | tuned pytest | `--receptor=llm` | Change |
|
|
389
|
+
| :--- | ---: | ---: | ---: |
|
|
390
|
+
| Cascade (38 failures, one cause) | 2863 | **105** | -96.3% |
|
|
391
|
+
| Green with many distinct warnings | 1598 | **664** | -58.4% |
|
|
392
|
+
| Green with warnings | 87 | **45** | -48.3% |
|
|
393
|
+
| Five distinct causes | 316 | **211** | -33.2% |
|
|
394
|
+
| Green suite (128 tests) | 23 | **15** | -34.8% |
|
|
395
|
+
| Single assertion failure | 197 | **165** | -16.2% |
|
|
396
|
+
| Collection error | 192 | **213** | +10.9% |
|
|
397
|
+
| Mixed states (skip, xfail, xpass) | 31 | **77** | +148.4% |
|
|
398
|
+
|
|
399
|
+
Both positive rows are scenarios of a handful of tests, where any fixed overhead
|
|
400
|
+
looks enormous as a percentage: +148.4% is forty-six tokens. They buy the
|
|
401
|
+
reason behind every skip and xfail and the name of the test that passed
|
|
402
|
+
unexpectedly, where `pytest -q` says `1 skipped, 1 xfailed, 1 xpassed` and
|
|
403
|
+
leaves you to re-run with `-rs` to find out which. Those sections are bounded by
|
|
404
|
+
the variety of reasons, not the number of tests, so four hundred skips across
|
|
405
|
+
three reasons still cost three lines.
|
|
406
|
+
|
|
407
|
+
The cascade row does not move: grouping forty failures into one root cause is
|
|
408
|
+
something no combination of pytest flags does.
|
|
409
|
+
|
|
410
|
+
### Measure it on your own suite
|
|
411
|
+
|
|
412
|
+
```bash
|
|
413
|
+
pytest --receptor=llm --receptor-stats
|
|
414
|
+
```
|
|
415
|
+
|
|
416
|
+
```text
|
|
417
|
+
receptor stats: 38 tokens vs 148 for pytest as you configured it | 110 fewer (-74.3%) | cl100k_base
|
|
418
|
+
```
|
|
419
|
+
|
|
420
|
+
The baseline here is *your* pytest configuration, not the strict one used in the
|
|
421
|
+
table above, because the question this answers is personal: against how you
|
|
422
|
+
actually run pytest, what does this save you? It is measured rather than
|
|
423
|
+
estimated — pytest genuinely renders into a temporary file during the same run,
|
|
424
|
+
which is then tokenized and deleted. No second invocation, no extra memory, and
|
|
425
|
+
your own output is unaffected.
|
|
426
|
+
|
|
427
|
+
Reproduce the table above with `python devtools/benchmarks/run_benchmarks.py`.
|
|
428
|
+
|
|
429
|
+
---
|
|
430
|
+
|
|
431
|
+
## Documentation
|
|
432
|
+
|
|
433
|
+
Full documentation: [uibcdf.github.io/pytest-receptor](https://uibcdf.github.io/pytest-receptor/)
|
|
434
|
+
|
|
435
|
+
Design notes, the audit that shaped the current scope, and the open work queue
|
|
436
|
+
live in [`devguide/`](devguide/README.md).
|
|
437
|
+
|
|
438
|
+
## License
|
|
439
|
+
|
|
440
|
+
MIT. See [LICENSE](LICENSE).
|