pytest-parallex 0.1.1__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (27) hide show
  1. pytest_parallex-0.1.1/.gitignore +28 -0
  2. pytest_parallex-0.1.1/CHANGELOG.md +45 -0
  3. pytest_parallex-0.1.1/LICENSE +21 -0
  4. pytest_parallex-0.1.1/PKG-INFO +237 -0
  5. pytest_parallex-0.1.1/README.md +210 -0
  6. pytest_parallex-0.1.1/docs/benchmarks.md +110 -0
  7. pytest_parallex-0.1.1/docs/fork-safety-references.md +431 -0
  8. pytest_parallex-0.1.1/pyproject.toml +90 -0
  9. pytest_parallex-0.1.1/src/pytest_parallex/__init__.py +17 -0
  10. pytest_parallex-0.1.1/src/pytest_parallex/_loop.py +66 -0
  11. pytest_parallex-0.1.1/src/pytest_parallex/_reports.py +89 -0
  12. pytest_parallex-0.1.1/src/pytest_parallex/_safety.py +247 -0
  13. pytest_parallex-0.1.1/src/pytest_parallex/hooks.py +32 -0
  14. pytest_parallex-0.1.1/src/pytest_parallex/plugin.py +214 -0
  15. pytest_parallex-0.1.1/src/pytest_parallex/runner_async.py +88 -0
  16. pytest_parallex-0.1.1/src/pytest_parallex/runner_fork.py +302 -0
  17. pytest_parallex-0.1.1/src/pytest_parallex/runner_process.py +199 -0
  18. pytest_parallex-0.1.1/src/pytest_parallex/runner_thread.py +85 -0
  19. pytest_parallex-0.1.1/src/pytest_parallex/scheduler.py +77 -0
  20. pytest_parallex-0.1.1/src/pytest_parallex/session_scope.py +208 -0
  21. pytest_parallex-0.1.1/testing/conftest.py +1 -0
  22. pytest_parallex-0.1.1/testing/test_async.py +52 -0
  23. pytest_parallex-0.1.1/testing/test_plugin.py +57 -0
  24. pytest_parallex-0.1.1/testing/test_process.py +37 -0
  25. pytest_parallex-0.1.1/testing/test_regressions.py +378 -0
  26. pytest_parallex-0.1.1/testing/test_scheduler.py +39 -0
  27. pytest_parallex-0.1.1/testing/test_thread.py +159 -0
@@ -0,0 +1,28 @@
1
+ # Python
2
+ __pycache__/
3
+ *.py[cod]
4
+ *.egg-info/
5
+ .eggs/
6
+
7
+ # Build artifacts
8
+ /dist/
9
+ /build/
10
+
11
+ # uv / venv
12
+ .venv/
13
+ uv.lock.bak
14
+
15
+ # Tool caches
16
+ .pytest_cache/
17
+ .ruff_cache/
18
+ .pyright/
19
+ .mypy_cache/
20
+ .moon/cache/
21
+
22
+ # Editor
23
+ .vscode/
24
+ .idea/
25
+ *.swp
26
+
27
+ # Local session logs
28
+ logs/
@@ -0,0 +1,45 @@
1
+ # Changelog
2
+
3
+ All notable changes to this project are documented here.
4
+
5
+ The version in `pyproject.toml` is the single source of truth, and merging a version bump
6
+ to `main` publishes it — see "Releasing" in the README.
7
+
8
+ ## 0.1.1 — first release
9
+
10
+ The first published version is 0.1.1, not 0.1.0: `version-guard` requires a code change to
11
+ carry a version bump, and 0.1.0 never left this repo. Nothing was released as 0.1.0.
12
+
13
+ ### Added
14
+
15
+ - `--parallel=fork`: collect once in the controller, then fork warm workers that inherit
16
+ the imported suite and the collected items via copy-on-write.
17
+ - **Controller-side session fixtures.** `scope="session"` runs once, in the controller,
18
+ before the fork; children inherit the cached result. Under pytest-xdist the same fixture
19
+ runs once *per worker*. `--parallex-no-session-scope` restores the per-worker behaviour.
20
+ - `--parallel=thread`, `--parallel=async`, `--parallel=process`.
21
+ - `--parallel-workers=N` (default `auto` = CPU count).
22
+ - Fork-safety refusal: the run stops with a message naming the offending thread or event
23
+ loop rather than forking a process that would deadlock in the child.
24
+ - Logging `QueueListener` threads are stopped and restarted around the fork, so a suite
25
+ that merely imports litestar (or anything using `QueueHandler`) can still fork.
26
+ - Hooks: `pytest_parallex_setup`, `pytest_parallex_teardown`,
27
+ `pytest_parallex_auto_num_workers`.
28
+ - Fixtures: `parallex_worker_id`, `parallex_mode`, `parallex_setup_data`.
29
+ - Dynamic work-claiming: workers race for module groups off a shared counter rather than
30
+ being dealt a static partition, so a slow module can't strand a worker.
31
+
32
+ ### Requires
33
+
34
+ - **pytest >= 8.1**, not 8.0: `FixtureManager.getfixturedefs` took a nodeid string until
35
+ 8.1 and a `Node` from 8.1 on, and the fork runner passes a `Node`.
36
+ - Tested against pytest 8.1, 8.4, 9.0 and 9.1 across all four modes. CI pins the floor.
37
+
38
+ ### Worker-scoped fixtures
39
+
40
+ - A `scope="session"` fixture that requests `worker_id` (directly or transitively) is
41
+ per-worker and is never hoisted to the controller — so `postgres_server` (one container)
42
+ and `database_url(worker_id)` (one database per worker) can coexist, which is what
43
+ xdist's filelock recipe exists to fake.
44
+ - `worker_id` is provided by parallex when `--parallel` is active, registered late enough
45
+ to win the name from pytest-xdist. A plain `pytest -n 4` run keeps xdist's.
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Jorge Cardona
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,237 @@
1
+ Metadata-Version: 2.4
2
+ Name: pytest-parallex
3
+ Version: 0.1.1
4
+ Summary: Parallel test execution for pytest where scope="session" means session — fork-based warm workers that inherit session fixtures, plus thread, async and process modes
5
+ Project-URL: Homepage, https://gitlab.com/jorgeecardona/pytest-parallex
6
+ Project-URL: Repository, https://gitlab.com/jorgeecardona/pytest-parallex
7
+ Project-URL: Changelog, https://gitlab.com/jorgeecardona/pytest-parallex/-/blob/main/CHANGELOG.md
8
+ Author-email: Jorge Cardona <jorgeecardona@gmail.com>
9
+ License-Expression: MIT
10
+ License-File: LICENSE
11
+ Keywords: concurrency,fork,parallel,pytest,session-fixtures,test-speed,testing,xdist
12
+ Classifier: Development Status :: 4 - Beta
13
+ Classifier: Framework :: Pytest
14
+ Classifier: Intended Audience :: Developers
15
+ Classifier: Operating System :: MacOS
16
+ Classifier: Operating System :: POSIX :: Linux
17
+ Classifier: Programming Language :: Python :: 3
18
+ Classifier: Programming Language :: Python :: 3.10
19
+ Classifier: Programming Language :: Python :: 3.11
20
+ Classifier: Programming Language :: Python :: 3.12
21
+ Classifier: Programming Language :: Python :: 3.13
22
+ Classifier: Topic :: Software Development :: Testing
23
+ Classifier: Typing :: Typed
24
+ Requires-Python: >=3.10
25
+ Requires-Dist: pytest>=8.1
26
+ Description-Content-Type: text/markdown
27
+
28
+ # pytest-parallex
29
+
30
+ Parallel test execution for pytest, with one idea the other plugins can't offer:
31
+ **`scope="session"` means session — once, not once per worker.**
32
+
33
+ ```bash
34
+ pip install pytest-parallex
35
+ pytest --parallel=fork
36
+ ```
37
+
38
+ ## The problem it solves
39
+
40
+ Under pytest-xdist, every worker is a fresh interpreter that runs its own full session, so
41
+ a `scope="session"` fixture executes **once per worker**. Their docs say so plainly, and
42
+ ship a `FileLock`-and-shared-tmpdir recipe as the workaround. It isn't a bug in your test
43
+ suite — their architecture has no moment at which a controller could build something once
44
+ and hand it to everybody.
45
+
46
+ Forking has that moment.
47
+
48
+ ```python
49
+ @pytest.fixture(scope="session")
50
+ def postgres():
51
+ with PostgresContainer("postgres:16") as pg: # boots ONCE
52
+ yield pg.get_connection_url() # every worker gets this URL
53
+ ```
54
+
55
+ `--parallel=fork` sets session fixtures up in the controller *before* forking. Children
56
+ inherit the `FixtureDef` — cached result and all — through copy-on-write, so they use it
57
+ and never re-run setup. One container, one compiled asset, one login, shared by every
58
+ worker, with no lockfile.
59
+
60
+ Same suite, same four workers, counting how many times the fixture body actually ran:
61
+
62
+ ```console
63
+ $ pytest -n 4 # pytest-xdist
64
+ 8 passed session fixture booted 4 times
65
+
66
+ $ pytest --parallel=fork --parallel-workers=4 # pytest-parallex
67
+ 8 passed session fixture booted 1 time
68
+ ```
69
+
70
+ That leaves the scope pytest never had a name for: **the worker**. What xdist calls
71
+ "session" is really per-worker, and conflating the two is what forces the recipe. Here
72
+ they're separate — `session` runs in the controller, and each forked child is a worker.
73
+
74
+ ## Modes
75
+
76
+ ```bash
77
+ pytest --parallel=fork --parallel-workers=8
78
+ ```
79
+
80
+ | mode | isolation | `scope="session"` runs | use when |
81
+ |---|---|---|---|
82
+ | `fork` | process per worker, warm | **once, in the controller** | Linux/macOS, and the process is quiet at fork time |
83
+ | `thread` | shared address space | once | tests are I/O-bound and tolerate sharing globals |
84
+ | `async` | shared address space | once | same as `thread`, dispatched via asyncio |
85
+ | `process` | fresh interpreter per worker | once **per worker** | tests mutate process-global state, or you're on Windows |
86
+
87
+ `--parallel-workers` defaults to `auto` (CPU count).
88
+
89
+ ## What fork actually costs and buys
90
+
91
+ Measured, and worth being precise about, because the intuitive pitch is wrong:
92
+
93
+ - **Removing xdist's worker boot is a flat ~1.5–2.3s.** It does *not* grow with suite size,
94
+ import weight, or worker count. On a 3s local run that's 1.84x and you feel it; on a 292s
95
+ CI suite it's under 1% and it's noise. Redundant imports parallelize across idle cores —
96
+ xdist's N-way collection is wasted *CPU*, not wasted *time*, and you only get paid for
97
+ removing it if the cores were already busy.
98
+ - **Session-fixture reuse is the part that scales.** A 10s container boot across 8 workers
99
+ is 80s under xdist and 10s here. That's the reason to reach for this.
100
+
101
+ Full method, numbers, and three traps that produced convincing wrong answers:
102
+ [docs/benchmarks.md](docs/benchmarks.md).
103
+
104
+ ## Fork safety
105
+
106
+ `fork()` duplicates only the calling thread. Anything another thread held — a lock, a
107
+ half-written buffer, a connection's state machine — is duplicated mid-flight and belongs
108
+ to nobody in the child. That's how forked children deadlock, and CPython 3.12 warns about
109
+ exactly it.
110
+
111
+ So parallex refuses rather than warns, and names the offender:
112
+
113
+ ```
114
+ --parallel=fork requires a quiet process at fork time, but 1 non-main thread(s)
115
+ are running (Thread-1 (_monitor)). Move the offending setup into a fixture so it
116
+ runs after the fork, or use --parallel=process.
117
+ ```
118
+
119
+ Logging `QueueListener` threads (which litestar, and anything using `QueueHandler`, starts
120
+ at import) are stopped and restarted around the fork automatically.
121
+
122
+ The constraint this imposes on a pre-fork session fixture: whatever it builds must survive
123
+ a fork. **An address survives — a URL, a path, a port. A live connection, a thread, or an
124
+ event loop does not.** The natural split is that the controller owns the *server* and each
125
+ worker opens its own *connection* to it:
126
+
127
+ ```python
128
+ @pytest.fixture(scope="session")
129
+ def db_url(): # controller: owns the server. Forkable.
130
+ with PostgresContainer("postgres:16") as pg:
131
+ yield pg.get_connection_url()
132
+
133
+ @pytest.fixture # worker: owns the connection. Not forkable, so per-test.
134
+ def db(db_url):
135
+ conn = psycopg.connect(db_url)
136
+ yield conn
137
+ conn.close()
138
+ ```
139
+
140
+ If a session fixture can't survive the fork, `--parallex-no-session-scope` defers them all
141
+ to the workers (restoring xdist's once-per-worker behaviour).
142
+
143
+ ## The fixture that has to differ per worker
144
+
145
+ Some `scope="session"` fixtures exist precisely *to* differ between workers — the archetype
146
+ being one that carves a database per worker so tests don't tread on each other. Hoisting
147
+ that one would be actively wrong: every worker gets the same database, and a per-test
148
+ `create_all`/`drop_all` then wipes its siblings' schemas mid-test.
149
+
150
+ **A fixture that requests `worker_id` — directly or through another fixture — is
151
+ per-worker, and is never hoisted.** Everything else that says `scope="session"` means it.
152
+ No new syntax, because suites already write it this way:
153
+
154
+ ```python
155
+ @pytest.fixture(scope="session")
156
+ def postgres_server(): # -> controller. One container.
157
+ with PostgresContainer("postgres:16") as pg:
158
+ yield pg.get_connection_url()
159
+
160
+ @pytest.fixture(scope="session")
161
+ def database_url(worker_id, postgres_server): # -> per worker. Requests worker_id.
162
+ url = f"{postgres_server}/test_{worker_id}"
163
+ create_database(url)
164
+ yield url
165
+ drop_database(url)
166
+ ```
167
+
168
+ One container for the run, one database per worker on it — which is the thing xdist's
169
+ filelock recipe exists to fake. This names the scope xdist conflates: what it calls
170
+ "session" is really per-worker, and here you can have both.
171
+
172
+ `worker_id` is also xdist's fixture name, deliberately — it's what makes existing suites
173
+ work unchanged. When both plugins are installed, `--parallel` takes the name so you get a
174
+ real per-worker id; a plain `pytest -n 4` run leaves xdist's alone. `parallex_worker_id`
175
+ is an alias if you'd rather be explicit, and marks a fixture per-worker just the same.
176
+
177
+ ## Fixtures
178
+
179
+ | fixture | gives you |
180
+ |---|---|
181
+ | `worker_id` | `'f0'`, `'w1'`, `'a2'`… or `'main'`. Requesting it marks a fixture per-worker |
182
+ | `parallex_worker_id` | the same value, under a name xdist can't shadow |
183
+ | `parallex_mode` | the active mode, or `None` |
184
+ | `parallex_setup_data` | whatever `pytest_parallex_setup` returned |
185
+
186
+ ## Hooks
187
+
188
+ ```python
189
+ # conftest.py
190
+ def pytest_parallex_setup(config):
191
+ """Run once in the controller, before any worker starts. Pre-fork: no I/O handles."""
192
+ return {"token": build_expensive_thing()}
193
+
194
+ def pytest_parallex_teardown(config, data):
195
+ """Run once in the controller, after every worker has finished."""
196
+
197
+ def pytest_parallex_auto_num_workers(config):
198
+ """Override --parallel-workers=auto. Default: os.cpu_count()."""
199
+ return 8
200
+ ```
201
+
202
+ ## Known limitations
203
+
204
+ - **`--maxfail` doesn't stop a fork run early.** Workers run their claimed group to
205
+ completion and the controller replays the reports afterwards, so the count is right but
206
+ the run isn't cut short.
207
+ - **Fork parallelizes by module.** A whole suite in one file uses one worker no matter what
208
+ `--parallel-workers` says.
209
+ - **`fork` is Linux/macOS only.** On Windows, use `--parallel=process`.
210
+ - **`thread`/`async` don't escape the GIL.** They're for suites that wait, not suites that
211
+ compute.
212
+
213
+ ## Developing
214
+
215
+ ```bash
216
+ uv sync # toolchain
217
+ make install-hooks # one-time, after cloning
218
+ make check # lint + typecheck + test, exactly what CI runs
219
+ ```
220
+
221
+ ## Releasing
222
+
223
+ The version in `pyproject.toml` is the single source of truth, and releases are automated:
224
+
225
+ 1. **`pre-commit` auto-bumps** the patch version whenever a commit touches `src/` (run
226
+ `make install-hooks` once after cloning). Doc/test/config-only commits don't bump. For
227
+ an intentional minor/major release, bump deliberately: `make bump TYPE=minor`.
228
+ 2. **`version-guard`** (part of `moon run :ci`) enforces the same rule non-bypassably in
229
+ CI: a push or MR that changes `src/` without a version bump fails the pipeline, catching
230
+ `--no-verify` and unhooked clones.
231
+ 3. **`:release`** runs on a green `main` pipeline: if `pyproject`'s version isn't on PyPI
232
+ yet, it publishes via OIDC Trusted Publishing. So merging a version bump to `main`
233
+ releases itself — no second pipeline, no tag required.
234
+
235
+ ## License
236
+
237
+ MIT
@@ -0,0 +1,210 @@
1
+ # pytest-parallex
2
+
3
+ Parallel test execution for pytest, with one idea the other plugins can't offer:
4
+ **`scope="session"` means session — once, not once per worker.**
5
+
6
+ ```bash
7
+ pip install pytest-parallex
8
+ pytest --parallel=fork
9
+ ```
10
+
11
+ ## The problem it solves
12
+
13
+ Under pytest-xdist, every worker is a fresh interpreter that runs its own full session, so
14
+ a `scope="session"` fixture executes **once per worker**. Their docs say so plainly, and
15
+ ship a `FileLock`-and-shared-tmpdir recipe as the workaround. It isn't a bug in your test
16
+ suite — their architecture has no moment at which a controller could build something once
17
+ and hand it to everybody.
18
+
19
+ Forking has that moment.
20
+
21
+ ```python
22
+ @pytest.fixture(scope="session")
23
+ def postgres():
24
+ with PostgresContainer("postgres:16") as pg: # boots ONCE
25
+ yield pg.get_connection_url() # every worker gets this URL
26
+ ```
27
+
28
+ `--parallel=fork` sets session fixtures up in the controller *before* forking. Children
29
+ inherit the `FixtureDef` — cached result and all — through copy-on-write, so they use it
30
+ and never re-run setup. One container, one compiled asset, one login, shared by every
31
+ worker, with no lockfile.
32
+
33
+ Same suite, same four workers, counting how many times the fixture body actually ran:
34
+
35
+ ```console
36
+ $ pytest -n 4 # pytest-xdist
37
+ 8 passed session fixture booted 4 times
38
+
39
+ $ pytest --parallel=fork --parallel-workers=4 # pytest-parallex
40
+ 8 passed session fixture booted 1 time
41
+ ```
42
+
43
+ That leaves the scope pytest never had a name for: **the worker**. What xdist calls
44
+ "session" is really per-worker, and conflating the two is what forces the recipe. Here
45
+ they're separate — `session` runs in the controller, and each forked child is a worker.
46
+
47
+ ## Modes
48
+
49
+ ```bash
50
+ pytest --parallel=fork --parallel-workers=8
51
+ ```
52
+
53
+ | mode | isolation | `scope="session"` runs | use when |
54
+ |---|---|---|---|
55
+ | `fork` | process per worker, warm | **once, in the controller** | Linux/macOS, and the process is quiet at fork time |
56
+ | `thread` | shared address space | once | tests are I/O-bound and tolerate sharing globals |
57
+ | `async` | shared address space | once | same as `thread`, dispatched via asyncio |
58
+ | `process` | fresh interpreter per worker | once **per worker** | tests mutate process-global state, or you're on Windows |
59
+
60
+ `--parallel-workers` defaults to `auto` (CPU count).
61
+
62
+ ## What fork actually costs and buys
63
+
64
+ Measured, and worth being precise about, because the intuitive pitch is wrong:
65
+
66
+ - **Removing xdist's worker boot is a flat ~1.5–2.3s.** It does *not* grow with suite size,
67
+ import weight, or worker count. On a 3s local run that's 1.84x and you feel it; on a 292s
68
+ CI suite it's under 1% and it's noise. Redundant imports parallelize across idle cores —
69
+ xdist's N-way collection is wasted *CPU*, not wasted *time*, and you only get paid for
70
+ removing it if the cores were already busy.
71
+ - **Session-fixture reuse is the part that scales.** A 10s container boot across 8 workers
72
+ is 80s under xdist and 10s here. That's the reason to reach for this.
73
+
74
+ Full method, numbers, and three traps that produced convincing wrong answers:
75
+ [docs/benchmarks.md](docs/benchmarks.md).
76
+
77
+ ## Fork safety
78
+
79
+ `fork()` duplicates only the calling thread. Anything another thread held — a lock, a
80
+ half-written buffer, a connection's state machine — is duplicated mid-flight and belongs
81
+ to nobody in the child. That's how forked children deadlock, and CPython 3.12 warns about
82
+ exactly it.
83
+
84
+ So parallex refuses rather than warns, and names the offender:
85
+
86
+ ```
87
+ --parallel=fork requires a quiet process at fork time, but 1 non-main thread(s)
88
+ are running (Thread-1 (_monitor)). Move the offending setup into a fixture so it
89
+ runs after the fork, or use --parallel=process.
90
+ ```
91
+
92
+ Logging `QueueListener` threads (which litestar, and anything using `QueueHandler`, starts
93
+ at import) are stopped and restarted around the fork automatically.
94
+
95
+ The constraint this imposes on a pre-fork session fixture: whatever it builds must survive
96
+ a fork. **An address survives — a URL, a path, a port. A live connection, a thread, or an
97
+ event loop does not.** The natural split is that the controller owns the *server* and each
98
+ worker opens its own *connection* to it:
99
+
100
+ ```python
101
+ @pytest.fixture(scope="session")
102
+ def db_url(): # controller: owns the server. Forkable.
103
+ with PostgresContainer("postgres:16") as pg:
104
+ yield pg.get_connection_url()
105
+
106
+ @pytest.fixture # worker: owns the connection. Not forkable, so per-test.
107
+ def db(db_url):
108
+ conn = psycopg.connect(db_url)
109
+ yield conn
110
+ conn.close()
111
+ ```
112
+
113
+ If a session fixture can't survive the fork, `--parallex-no-session-scope` defers them all
114
+ to the workers (restoring xdist's once-per-worker behaviour).
115
+
116
+ ## The fixture that has to differ per worker
117
+
118
+ Some `scope="session"` fixtures exist precisely *to* differ between workers — the archetype
119
+ being one that carves a database per worker so tests don't tread on each other. Hoisting
120
+ that one would be actively wrong: every worker gets the same database, and a per-test
121
+ `create_all`/`drop_all` then wipes its siblings' schemas mid-test.
122
+
123
+ **A fixture that requests `worker_id` — directly or through another fixture — is
124
+ per-worker, and is never hoisted.** Everything else that says `scope="session"` means it.
125
+ No new syntax, because suites already write it this way:
126
+
127
+ ```python
128
+ @pytest.fixture(scope="session")
129
+ def postgres_server(): # -> controller. One container.
130
+ with PostgresContainer("postgres:16") as pg:
131
+ yield pg.get_connection_url()
132
+
133
+ @pytest.fixture(scope="session")
134
+ def database_url(worker_id, postgres_server): # -> per worker. Requests worker_id.
135
+ url = f"{postgres_server}/test_{worker_id}"
136
+ create_database(url)
137
+ yield url
138
+ drop_database(url)
139
+ ```
140
+
141
+ One container for the run, one database per worker on it — which is the thing xdist's
142
+ filelock recipe exists to fake. This names the scope xdist conflates: what it calls
143
+ "session" is really per-worker, and here you can have both.
144
+
145
+ `worker_id` is also xdist's fixture name, deliberately — it's what makes existing suites
146
+ work unchanged. When both plugins are installed, `--parallel` takes the name so you get a
147
+ real per-worker id; a plain `pytest -n 4` run leaves xdist's alone. `parallex_worker_id`
148
+ is an alias if you'd rather be explicit, and marks a fixture per-worker just the same.
149
+
150
+ ## Fixtures
151
+
152
+ | fixture | gives you |
153
+ |---|---|
154
+ | `worker_id` | `'f0'`, `'w1'`, `'a2'`… or `'main'`. Requesting it marks a fixture per-worker |
155
+ | `parallex_worker_id` | the same value, under a name xdist can't shadow |
156
+ | `parallex_mode` | the active mode, or `None` |
157
+ | `parallex_setup_data` | whatever `pytest_parallex_setup` returned |
158
+
159
+ ## Hooks
160
+
161
+ ```python
162
+ # conftest.py
163
+ def pytest_parallex_setup(config):
164
+ """Run once in the controller, before any worker starts. Pre-fork: no I/O handles."""
165
+ return {"token": build_expensive_thing()}
166
+
167
+ def pytest_parallex_teardown(config, data):
168
+ """Run once in the controller, after every worker has finished."""
169
+
170
+ def pytest_parallex_auto_num_workers(config):
171
+ """Override --parallel-workers=auto. Default: os.cpu_count()."""
172
+ return 8
173
+ ```
174
+
175
+ ## Known limitations
176
+
177
+ - **`--maxfail` doesn't stop a fork run early.** Workers run their claimed group to
178
+ completion and the controller replays the reports afterwards, so the count is right but
179
+ the run isn't cut short.
180
+ - **Fork parallelizes by module.** A whole suite in one file uses one worker no matter what
181
+ `--parallel-workers` says.
182
+ - **`fork` is Linux/macOS only.** On Windows, use `--parallel=process`.
183
+ - **`thread`/`async` don't escape the GIL.** They're for suites that wait, not suites that
184
+ compute.
185
+
186
+ ## Developing
187
+
188
+ ```bash
189
+ uv sync # toolchain
190
+ make install-hooks # one-time, after cloning
191
+ make check # lint + typecheck + test, exactly what CI runs
192
+ ```
193
+
194
+ ## Releasing
195
+
196
+ The version in `pyproject.toml` is the single source of truth, and releases are automated:
197
+
198
+ 1. **`pre-commit` auto-bumps** the patch version whenever a commit touches `src/` (run
199
+ `make install-hooks` once after cloning). Doc/test/config-only commits don't bump. For
200
+ an intentional minor/major release, bump deliberately: `make bump TYPE=minor`.
201
+ 2. **`version-guard`** (part of `moon run :ci`) enforces the same rule non-bypassably in
202
+ CI: a push or MR that changes `src/` without a version bump fails the pipeline, catching
203
+ `--no-verify` and unhooked clones.
204
+ 3. **`:release`** runs on a green `main` pipeline: if `pyproject`'s version isn't on PyPI
205
+ yet, it publishes via OIDC Trusted Publishing. So merging a version bump to `main`
206
+ releases itself — no second pipeline, no tag required.
207
+
208
+ ## License
209
+
210
+ MIT
@@ -0,0 +1,110 @@
1
+ # What fork actually buys — measured, 2026-07-14
2
+
3
+ Written after a session that produced three wrong answers before a right one. The wrong turns are
4
+ kept here on purpose: each one is a trap that looks exactly like a result.
5
+
6
+ ## Bottom line
7
+
8
+ **`--parallel=fork` removes xdist's fixed worker-boot cost — a flat ~1.5–2.3s. That is the whole
9
+ win.** It does *not* scale with suite size, import weight, or worker count.
10
+
11
+ That makes fork mode a **local TDD-loop feature, not a CI feature**:
12
+
13
+ | suite | xdist | fork | verdict |
14
+ |---|---|---|---|
15
+ | 3s local run | 3.21s | 1.75s | **1.84x — you feel this** |
16
+ | paas CI `api:test` (292s) | — | −2s | **under 1%. noise.** |
17
+
18
+ ## The thesis that died
19
+
20
+ The pitch was: *xdist re-imports and re-collects in every worker (its own FAQ admits it, and
21
+ [#353](https://github.com/pytest-dev/pytest-xdist/issues/353) reports a 4x collection penalty on a
22
+ 16k-test suite), so collect-once-then-fork should scale better the heavier the import.*
23
+
24
+ Measured, with a CPU-bound import (not a `sleep` — see trap 3), 8 workers on 16 cores:
25
+
26
+ | import cost | xdist -n8 | fork x8 | ratio | absolute saving |
27
+ |---|---|---|---|---|
28
+ | 0.03s | 3.21s | 1.75s | 1.83x | 1.46s |
29
+ | 0.55s | 3.82s | 2.17s | 1.76x | 1.65s |
30
+ | 2.01s | 5.37s | 3.67s | 1.46x | 1.70s |
31
+ | 4.65s | 8.75s | 6.40s | 1.37x | 2.35s |
32
+
33
+ **The ratio shrinks as the import grows, and the absolute saving is flat.** The prediction was
34
+ backwards, and the reason is the thing to internalize:
35
+
36
+ > **Redundant imports parallelize across idle cores.** Eight workers each importing on eight free
37
+ > cores costs ~1x wall time, not 8x. xdist's N-way collection is wasted *CPU*, not wasted *time*.
38
+ > You only get paid for removing it if the cores were already busy.
39
+
40
+ So xdist's real, addressable overhead is just gateway boot: `subprocess.Popen` of a virgin
41
+ interpreter per worker, plus the execnet handshake. Constant, and small.
42
+
43
+ ## Three traps that produced convincing wrong numbers
44
+
45
+ **1. The runner silently never ran.** `pytest_runtestloop` is a `firstresult` hook — the first impl
46
+ to return non-None wins. Registering with `trylast=True` handed the loop to pytest's builtin, which
47
+ returned `True`. Every test still passed, serially, and the "fork" benchmarks were plain serial
48
+ pytest for hours. The tell was CPU utilization: **0.9 cores busy on a 16-worker run** (`user 5m44s`
49
+ / `real 7m14s`), against xdist's 8.9 (`user 18m30s` / `real 2m10s`). *A parallel runner that isn't
50
+ running looks like a slow parallel runner, not like a broken one.* Check `user`/`real` before
51
+ believing any parallel benchmark.
52
+
53
+ **2. Benchmarking a suite that can't fork.** With the hook fixed, paas's suite refuses outright:
54
+
55
+ ```
56
+ --parallel=fork requires a quiet process at fork time, but 1 non-main thread(s)
57
+ are running (Thread-1 (_monitor)).
58
+ ```
59
+
60
+ `litestar.logging.standard.LoggingQueueListener` starts a logging thread **at import**. This is the
61
+ predicted #1 hazard arriving immediately: you cannot stop threads you don't own. Android's zygote
62
+ only gets away with fork because `ZygoteHooks.preFork()` calls `Daemons.stop()` and restarts them
63
+ after — it owns the entire runtime. A pytest plugin owns none of it.
64
+
65
+ **3. `time.sleep()` is not an import.** The first synthetic modelled import cost as a sleep. Sleep
66
+ parallelizes perfectly, so it showed fork's advantage *growing* with import weight — the opposite of
67
+ the truth. Model CPU-bound work with CPU-bound work.
68
+
69
+ ## What was actually confirmed
70
+
71
+ - **xdist cannot fork, and it isn't close.** execnet's `get_execmodel()` closes over
72
+ `thread`/`main_thread_only`/`eventlet`/`gevent` with an exhaustive `else: raise`; workers are
73
+ spawned `subprocess.Popen([python, "-u", "-c", bootstrap])`. There is nothing to inherit. The
74
+ ground is genuinely unoccupied.
75
+ - **The stdlib `random` frozen-seed hazard does not apply to CPython.** `Lib/random.py:992` does
76
+ `_os.register_at_fork(after_in_child=_inst.seed)` — children reseed automatically. Verified by
77
+ forking after `random.seed(42)`: children diverge. The Spring
78
+ [#113](https://github.com/rails/spring/issues/113) precedent that froze a test seed for five years
79
+ is *Ruby*, whose `Random` has no such hook. Libraries that seed their own state (numpy, faker)
80
+ still need checking; stdlib does not.
81
+ - **The zygote effect is real but narrow.** Forking a parent that has already paid a 904ms
82
+ `create_app()` costs **1.56ms/child, flat out to 248 children**. Whether that is worth anything
83
+ depends entirely on whether the suite is CPU-bound — see below.
84
+ - **Dynamic scheduling beat static by only 6%** on the paas suite (411.5s → 386.5s), far under the
85
+ 1.43x predicted from per-module costs. Both numbers are void anyway (trap 1), but the
86
+ claim-counter queue is the right design regardless: a static deal must guess a group's cost from
87
+ its test *count*, and on a suite whose modules ranged 0.1s–167s that guess is bad.
88
+
89
+ ## Where this leaves the project
90
+
91
+ Fork mode is worth shipping for the **inner loop** — the same niche Discord's pytest daemon targets
92
+ (median 20s → 2s) without inheriting their hard problem (correct incremental reload of a long-lived
93
+ dirty process; a forked child is a throwaway copy and the parent is never mutated).
94
+
95
+ It is **not** a CI play, and the prior art says so too: DHH removed Spring from Rails defaults in
96
+ [rails/rails#42997](https://github.com/rails/rails/pull/42997) with the reason *"Faster computers
97
+ have meant that most apps won't see a big benefit... the pain of dealing with the occasional spring
98
+ issue is no longer warranted by default."* The cost of a preloader is constant; the benefit shrinks
99
+ with hardware. That is the same shape as the flat 1.5–2.3s measured above.
100
+
101
+ Open questions, in order:
102
+ 1. **Can a suite be made forkable?** paas can't fork because of one third-party logging thread. A
103
+ `pytest_parallex_pre_fork` / `post_fork` hook pair would let the suite author quiesce and restart
104
+ what the plugin can't know about (Android's `Daemons.stop()`, Passenger's
105
+ `:starting_worker_process`, Spring's `after_fork` — every prior system converged on this).
106
+ 2. **Is the inner-loop win defensible against `--lf`/`-k`?** A dev rerunning one module already
107
+ pays only that module's import. Measure fork against the workflows people actually use, not
108
+ against a full-suite run.
109
+ 3. **Does the zygote matter for anyone?** It only pays where per-test setup is expensive AND
110
+ CPU-bound. paas's setup is expensive but I/O-bound (Postgres), so it wouldn't have helped there.