pytest-xdist-memory 0.1.0__tar.gz

pytest_xdist_memory-0.1.0/.gitignore

@@ -0,0 +1,65 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+*.egg
+*.egg-info/
+.eggs/
+build/
+dist/
+.installed.cfg
+MANIFEST
+
+# Virtualenvs
+.venv/
+venv/
+ENV/
+env/
+
+# uv
+.uv/
+uv.lock
+
+# Tooling
+.mypy_cache/
+.ruff_cache/
+.pytest_cache/
+.coverage
+.coverage.*
+htmlcov/
+coverage.xml
+*.cover
+
+# IDE
+.idea/
+.vscode/
+*.swp
+*.swo
+
+# OS
+.DS_Store
+Thumbs.db
+
+# django-run-site / django-dev-helpers dotfiles
+.dev_helpers_token
+.dev_helpers_port
+.dev_helpers_pg_port
+.dev_helpers_redis_port
+.run-site-*
+
+# Django
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Test artifacts
+.tox/
+.cache/
+test-output/
+
+# Examples — generated at runtime
+examples/test_site/.venv/
+examples/test_site/db.sqlite3
+.pytest-xdist-memory.json

pytest_xdist_memory-0.1.0/CHANGELOG.md

@@ -0,0 +1,9 @@
+# Changelog
+
+## 0.1.0 (unreleased)
+
+- First release: per-worker memory measurement including child processes
+  (PSS on Linux), optional docker container measurement (by name or by
+  published port), per-machine profile per `-m` expression, `-n auto`
+  answered from available RAM, optional watchdog, ini equivalents for all
+  options.

pytest_xdist_memory-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Michał Pasternak
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pytest_xdist_memory-0.1.0/PKG-INFO

@@ -0,0 +1,166 @@
+Metadata-Version: 2.5
+Name: pytest-xdist-memory
+Version: 0.1.0
+Summary: Size pytest-xdist by available RAM: measure worker memory (with browsers and other child processes), keep a per-machine profile, answer -n auto from it, and optionally kill a run before it freezes the machine.
+Project-URL: Homepage, https://github.com/iplweb/pytest-xdist-memory
+Project-URL: Source, https://github.com/iplweb/pytest-xdist-memory
+Project-URL: Issues, https://github.com/iplweb/pytest-xdist-memory/issues
+Author-email: Michał Pasternak <michal.dtz@gmail.com>
+License: MIT License
+        
+        Copyright (c) 2026 Michał Pasternak
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+License-File: LICENSE
+Keywords: RAM,memory,oom,parallel,playwright,pytest,xdist
+Classifier: Development Status :: 3 - Alpha
+Classifier: Framework :: Pytest
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.10
+Requires-Dist: psutil>=5
+Requires-Dist: pytest>=7
+Provides-Extra: dev
+Requires-Dist: pytest-xdist>=3; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Requires-Dist: ruff<0.16,>=0.15; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# pytest-xdist-memory
+
+Size [pytest-xdist](https://github.com/pytest-dev/pytest-xdist) by **available
+RAM**, not by CPU count.
+
+On a developer laptop the bottleneck of a parallel test run is usually memory.
+`pytest -n auto` starts one worker per core; with heavy workers (Django and
+a database, a browser per worker for Playwright, a live server...) a 14-core,
+14 GB laptop goes into swap thrashing and freezes. This plugin measures how
+much a worker of *your* test suite needs on *this* machine and answers
+`-n auto` with as many workers as fit into the memory available right now.
+
+> Status: alpha.
+
+## What it does
+
+With `--xdist-memory` (or `xdist_memory = true` in the ini file):
+
+- **Measures** the peak memory of every worker **together with all its child
+  processes**: browsers, live servers, subprocesses. Linux uses PSS (a fair
+  share of pages shared between processes, so the several processes of one
+  Chromium are not counted several times over); other systems use RSS.
+- **Measures containers** too, if you ask it to: a test database started by
+  testcontainers is not a child of pytest. Containers can be matched by name
+  (`--xdist-memory-docker SUBSTR`) or, more precisely, by the host port they
+  publish (`--xdist-memory-docker-port-env VAR`, where `VAR` is the
+  environment variable your fixtures export with the port). Matching by port
+  is exact even when parallel runs from other checkouts have containers with
+  similar names. The peak is divided by the number of workers.
+- **Records** the peaks in a per-machine profile, `.pytest-xdist-memory.json`.
+  Each `-m` marker expression is a separate "suite": `-m playwright` workers
+  are much heavier than `-m "not playwright"` ones. The last 5 runs are kept,
+  and the estimate uses the maximum. Runs with fewer than 20 tests, or
+  interrupted runs, are not recorded.
+- **Answers `-n auto` / `-n logical`**:
+  `workers = (available - reserve - controller) / (worker_peak × safety + extra)`,
+  capped by the CPU count. With `--xdist-memory-performance-cores`, Apple
+  Silicon is capped at the number of performance cores.
+- **Optional watchdog** (`--xdist-memory-watchdog`): kills the run when the
+  whole process tree, plus the measured containers, exceeds the memory
+  available at start minus the reserve.
+
+Nothing changes until you enable it. `PYTEST_XDIST_AUTO_NUM_WORKERS` still
+wins over the estimate.
+
+## Usage
+
+```bash
+pip install pytest-xdist-memory
+
+# First run: no profile yet, so it assumes 1024 MB per worker; the run is measured.
+pytest -n auto --xdist-memory
+
+# Later runs use what was measured (per -m expression):
+pytest -n auto --xdist-memory -m "not playwright"
+pytest -n auto --xdist-memory -m playwright --xdist-memory-performance-cores
+```
+
+The terminal summary shows the measured peaks and the calculation:
+
+```
+================================ xdist-memory =================================
+-n auto [default]: 3 workers: available 7282 MB - reserve 2048 - controller 215 = 5019 MB; per worker 1357 MB (910 x safety + extra 220); cpu cap 14; from profile (4 runs)
+peak per worker 905 MB (all: [905, 871, 860]), controller 210 MB, containers 214 MB/worker
+recorded in .pytest-xdist-memory.json [default]
+```
+
+Add `.pytest-xdist-memory.json` to `.gitignore`: the profile describes one
+machine.
+
+## Options
+
+| Option | ini | Default | |
+|---|---|---|---|
+| `--xdist-memory` | `xdist_memory` | off | enable |
+| `--xdist-memory-profile PATH` | `xdist_memory_profile` | `.pytest-xdist-memory.json` | relative to rootdir |
+| `--xdist-memory-reserve MB` | `xdist_memory_reserve` | 2048 | left for the desktop/system |
+| `--xdist-memory-safety F` | `xdist_memory_safety` | 1.25 | multiplier on the worker peak |
+| `--xdist-memory-default-worker MB` | `xdist_memory_default_worker` | 1024 | used until a run is recorded |
+| `--xdist-memory-extra MB` | `xdist_memory_extra` | 0 | fixed extra per worker (e.g. its DB) |
+| `--xdist-memory-docker SUBSTR` | `xdist_memory_docker` | | measure containers by name |
+| `--xdist-memory-docker-port-env VAR` | `xdist_memory_docker_port_env` | | measure the container publishing port `$VAR` |
+| `--xdist-memory-performance-cores` | `xdist_memory_performance_cores` | off | Apple Silicon: P-cores only |
+| `--xdist-memory-min-tests N` | `xdist_memory_min_tests` | 20 | smaller runs are not recorded |
+| `--xdist-memory-watchdog` | `xdist_memory_watchdog` | off | kill the run over budget |
+| `--xdist-memory-interval S` | `xdist_memory_interval` | 2 | sampling interval |
+
+## Watchdog vs. a cgroup limit
+
+The watchdog is portable but **best effort**: it checks once per sampling
+interval, so a very fast allocation spike can get past it. When it fires, it
+kills the workers and exits with code 3, without pytest's teardown: containers
+are left to their own reaper (Ryuk for testcontainers).
+
+On Linux with systemd, a cgroup limit is the hard guarantee: the kernel kills
+the run instead of swapping the desktop to death. You can use both:
+
+```bash
+systemd-run --user --scope -q -p MemoryMax=6G -p MemorySwapMax=0 \
+    pytest -n auto --xdist-memory
+```
+
+**Neither is needed for measuring.** Measuring uses only `psutil` (and
+`docker stats`, if you ask for containers).
+
+## Requirements
+
+Python ≥ 3.10, pytest ≥ 7, psutil. pytest-xdist is optional: without it, the
+plugin only reports the peak of the whole process tree.
+
+## License
+
+MIT

pytest_xdist_memory-0.1.0/README.md

@@ -0,0 +1,113 @@
+# pytest-xdist-memory
+
+Size [pytest-xdist](https://github.com/pytest-dev/pytest-xdist) by **available
+RAM**, not by CPU count.
+
+On a developer laptop the bottleneck of a parallel test run is usually memory.
+`pytest -n auto` starts one worker per core; with heavy workers (Django and
+a database, a browser per worker for Playwright, a live server...) a 14-core,
+14 GB laptop goes into swap thrashing and freezes. This plugin measures how
+much a worker of *your* test suite needs on *this* machine and answers
+`-n auto` with as many workers as fit into the memory available right now.
+
+> Status: alpha.
+
+## What it does
+
+With `--xdist-memory` (or `xdist_memory = true` in the ini file):
+
+- **Measures** the peak memory of every worker **together with all its child
+  processes**: browsers, live servers, subprocesses. Linux uses PSS (a fair
+  share of pages shared between processes, so the several processes of one
+  Chromium are not counted several times over); other systems use RSS.
+- **Measures containers** too, if you ask it to: a test database started by
+  testcontainers is not a child of pytest. Containers can be matched by name
+  (`--xdist-memory-docker SUBSTR`) or, more precisely, by the host port they
+  publish (`--xdist-memory-docker-port-env VAR`, where `VAR` is the
+  environment variable your fixtures export with the port). Matching by port
+  is exact even when parallel runs from other checkouts have containers with
+  similar names. The peak is divided by the number of workers.
+- **Records** the peaks in a per-machine profile, `.pytest-xdist-memory.json`.
+  Each `-m` marker expression is a separate "suite": `-m playwright` workers
+  are much heavier than `-m "not playwright"` ones. The last 5 runs are kept,
+  and the estimate uses the maximum. Runs with fewer than 20 tests, or
+  interrupted runs, are not recorded.
+- **Answers `-n auto` / `-n logical`**:
+  `workers = (available - reserve - controller) / (worker_peak × safety + extra)`,
+  capped by the CPU count. With `--xdist-memory-performance-cores`, Apple
+  Silicon is capped at the number of performance cores.
+- **Optional watchdog** (`--xdist-memory-watchdog`): kills the run when the
+  whole process tree, plus the measured containers, exceeds the memory
+  available at start minus the reserve.
+
+Nothing changes until you enable it. `PYTEST_XDIST_AUTO_NUM_WORKERS` still
+wins over the estimate.
+
+## Usage
+
+```bash
+pip install pytest-xdist-memory
+
+# First run: no profile yet, so it assumes 1024 MB per worker; the run is measured.
+pytest -n auto --xdist-memory
+
+# Later runs use what was measured (per -m expression):
+pytest -n auto --xdist-memory -m "not playwright"
+pytest -n auto --xdist-memory -m playwright --xdist-memory-performance-cores
+```
+
+The terminal summary shows the measured peaks and the calculation:
+
+```
+================================ xdist-memory =================================
+-n auto [default]: 3 workers: available 7282 MB - reserve 2048 - controller 215 = 5019 MB; per worker 1357 MB (910 x safety + extra 220); cpu cap 14; from profile (4 runs)
+peak per worker 905 MB (all: [905, 871, 860]), controller 210 MB, containers 214 MB/worker
+recorded in .pytest-xdist-memory.json [default]
+```
+
+Add `.pytest-xdist-memory.json` to `.gitignore`: the profile describes one
+machine.
+
+## Options
+
+| Option | ini | Default | |
+|---|---|---|---|
+| `--xdist-memory` | `xdist_memory` | off | enable |
+| `--xdist-memory-profile PATH` | `xdist_memory_profile` | `.pytest-xdist-memory.json` | relative to rootdir |
+| `--xdist-memory-reserve MB` | `xdist_memory_reserve` | 2048 | left for the desktop/system |
+| `--xdist-memory-safety F` | `xdist_memory_safety` | 1.25 | multiplier on the worker peak |
+| `--xdist-memory-default-worker MB` | `xdist_memory_default_worker` | 1024 | used until a run is recorded |
+| `--xdist-memory-extra MB` | `xdist_memory_extra` | 0 | fixed extra per worker (e.g. its DB) |
+| `--xdist-memory-docker SUBSTR` | `xdist_memory_docker` | | measure containers by name |
+| `--xdist-memory-docker-port-env VAR` | `xdist_memory_docker_port_env` | | measure the container publishing port `$VAR` |
+| `--xdist-memory-performance-cores` | `xdist_memory_performance_cores` | off | Apple Silicon: P-cores only |
+| `--xdist-memory-min-tests N` | `xdist_memory_min_tests` | 20 | smaller runs are not recorded |
+| `--xdist-memory-watchdog` | `xdist_memory_watchdog` | off | kill the run over budget |
+| `--xdist-memory-interval S` | `xdist_memory_interval` | 2 | sampling interval |
+
+## Watchdog vs. a cgroup limit
+
+The watchdog is portable but **best effort**: it checks once per sampling
+interval, so a very fast allocation spike can get past it. When it fires, it
+kills the workers and exits with code 3, without pytest's teardown: containers
+are left to their own reaper (Ryuk for testcontainers).
+
+On Linux with systemd, a cgroup limit is the hard guarantee: the kernel kills
+the run instead of swapping the desktop to death. You can use both:
+
+```bash
+systemd-run --user --scope -q -p MemoryMax=6G -p MemorySwapMax=0 \
+    pytest -n auto --xdist-memory
+```
+
+**Neither is needed for measuring.** Measuring uses only `psutil` (and
+`docker stats`, if you ask for containers).
+
+## Requirements
+
+Python ≥ 3.10, pytest ≥ 7, psutil. pytest-xdist is optional: without it, the
+plugin only reports the peak of the whole process tree.
+
+## License
+
+MIT

pytest_xdist_memory-0.1.0/pyproject.toml

@@ -0,0 +1,63 @@
+[build-system]
+requires = ["hatchling>=1.18"]
+build-backend = "hatchling.build"
+
+[project]
+name = "pytest-xdist-memory"
+version = "0.1.0"
+description = "Size pytest-xdist by available RAM: measure worker memory (with browsers and other child processes), keep a per-machine profile, answer -n auto from it, and optionally kill a run before it freezes the machine."
+readme = "README.md"
+license = { file = "LICENSE" }
+requires-python = ">=3.10"
+authors = [
+    { name = "Michał Pasternak", email = "michal.dtz@gmail.com" },
+]
+keywords = ["pytest", "xdist", "memory", "RAM", "parallel", "playwright", "oom"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Framework :: Pytest",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
+    "Topic :: Software Development :: Testing",
+]
+dependencies = ["pytest>=7", "psutil>=5"]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=8",
+    "pytest-xdist>=3",
+    "ruff>=0.15,<0.16",
+]
+
+[project.entry-points.pytest11]
+xdist_memory = "pytest_xdist_memory.plugin"
+
+[project.urls]
+Homepage = "https://github.com/iplweb/pytest-xdist-memory"
+Source = "https://github.com/iplweb/pytest-xdist-memory"
+Issues = "https://github.com/iplweb/pytest-xdist-memory/issues"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/pytest_xdist_memory"]
+
+[tool.hatch.build.targets.sdist]
+include = ["src/", "tests/", "README.md", "CHANGELOG.md", "LICENSE"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = "-p pytester"
+
+[tool.ruff]
+line-length = 88
+target-version = "py310"
+
+[tool.ruff.lint]
+select = ["E", "F", "W", "I", "B", "UP"]

pytest_xdist_memory-0.1.0/src/pytest_xdist_memory/__init__.py

@@ -0,0 +1,3 @@
+"""Size pytest-xdist by available RAM, not by CPU count."""
+
+__version__ = "0.1.0"

pytest_xdist_memory-0.1.0/src/pytest_xdist_memory/memory.py

@@ -0,0 +1,122 @@
+"""Measuring memory: process subtrees, available RAM, docker containers."""
+
+from __future__ import annotations
+
+import shutil
+import subprocess
+import sys
+
+import psutil
+
+MB = 1024 * 1024
+
+_UNITS = (("GiB", 1024.0), ("MiB", 1.0), ("KiB", 1 / 1024), ("GB", 1000.0))
+
+
+def process_mb(proc: psutil.Process) -> int:
+    """Memory of one process in MB.
+
+    PSS on Linux: a fair share of pages shared with other processes. Browsers
+    (Chromium runs several processes sharing most of their pages) would be
+    counted several times over by summing RSS. Elsewhere RSS.
+    """
+    if sys.platform.startswith("linux"):
+        try:
+            return proc.memory_full_info().pss // MB
+        except psutil.AccessDenied:
+            pass  # no access to smaps -- fall back to RSS below
+    return proc.memory_info().rss // MB
+
+
+def subtree_mb(proc: psutil.Process) -> int:
+    """Memory of a process and ALL its descendants (browsers, servers...)."""
+    try:
+        procs = [proc, *proc.children(recursive=True)]
+    except psutil.NoSuchProcess:
+        return 0
+    total = 0
+    for p in procs:
+        try:
+            total += process_mb(p)
+        except (psutil.NoSuchProcess, psutil.AccessDenied):
+            continue  # exited between listing and measuring
+    return total
+
+
+def available_mb() -> int:
+    return psutil.virtual_memory().available // MB
+
+
+def _parse_usage(usage: str) -> int | None:
+    used = usage.split("/")[0].strip()
+    for unit, mult in _UNITS:
+        if used.endswith(unit):
+            try:
+                return int(float(used[: -len(unit)]) * mult)
+            except ValueError:
+                return None
+    return None
+
+
+def _docker(*args: str) -> str | None:
+    try:
+        return subprocess.run(
+            ["docker", *args], capture_output=True, text=True, timeout=20
+        ).stdout
+    except (subprocess.TimeoutExpired, OSError):
+        return None
+
+
+def docker_mb(match: str = "", port: str | None = None) -> int | None:
+    """Total memory of running containers whose name contains ``match``
+    and/or which publish host ``port``.
+
+    ``None`` when docker is unavailable or nothing matches. Containers are
+    not children of pytest (e.g. a database started by testcontainers), so
+    they cannot be found by walking the process tree. Matching by the
+    published port picks exactly THIS run's container even when parallel
+    runs (other checkouts) have containers with similar names.
+    """
+    if not shutil.which("docker") or not (match or port):
+        return None
+    names: list[str] = []
+    if port:
+        out = _docker("ps", "--filter", f"publish={port}", "--format", "{{.Names}}")
+        names = (out or "").split()
+        if not names:
+            return None
+    out = _docker("stats", "--no-stream", "--format", "{{.Name}} {{.MemUsage}}", *names)
+    if out is None:
+        return None
+    total = None
+    for line in out.splitlines():
+        name, _, usage = line.partition(" ")
+        if match not in name:
+            continue
+        mb = _parse_usage(usage)
+        if mb is not None:
+            total = (total or 0) + mb
+    return total
+
+
+def cpu_cap(performance_cores: bool) -> int:
+    """Upper bound on workers from the CPU.
+
+    With ``performance_cores`` on Apple Silicon: only the performance cores.
+    Efficiency cores are too slow to drive e.g. their own browser and only
+    add contention.
+    """
+    cpus = psutil.cpu_count(logical=True) or 1
+    if performance_cores and sys.platform == "darwin":
+        try:
+            out = subprocess.run(
+                ["sysctl", "-n", "hw.perflevel0.logicalcpu"],
+                capture_output=True,
+                text=True,
+                timeout=5,
+            ).stdout.strip()
+        except (subprocess.TimeoutExpired, OSError):
+            return cpus
+        if out.isdigit():
+            return int(out)
+    return cpus

pytest_xdist_memory-0.1.0/src/pytest_xdist_memory/plugin.py

@@ -0,0 +1,419 @@
+"""pytest plugin: size pytest-xdist by RAM, not by CPU count.
+
+Why this exists: on a developer machine (laptop with a desktop) the
+bottleneck of a parallel test run is usually memory, not CPU. ``-n auto``
+spawns one worker per core; with heavy workers (Django + a database,
+a browser per worker for Playwright) that is enough to push the machine
+into swap thrashing and freeze it.
+
+The plugin, once enabled with ``--xdist-memory``:
+
+* measures, in every worker, the peak memory of the worker AND all of its
+  descendants (browsers, live servers...) -- PSS on Linux, RSS elsewhere;
+* optionally measures docker containers by name or by published port (a test
+  database started by testcontainers is not a child of pytest);
+* records the peaks in a per-machine profile file, separately per ``-m``
+  marker expression ("suite");
+* answers ``-n auto`` / ``-n logical`` with as many workers as fit into the
+  currently available RAM (capped by CPU count);
+* optionally (``--xdist-memory-watchdog``) aborts the run when the whole
+  process tree exceeds the budget -- a portable, best-effort stand-in for a
+  cgroup limit (``systemd-run --scope -p MemoryMax=...``).
+
+Nothing changes until you enable it.
+"""
+
+from __future__ import annotations
+
+import os
+import sys
+import threading
+from pathlib import Path
+from typing import Any
+
+import psutil
+import pytest
+
+from . import memory, profile
+
+_KEY = "xdist_memory"
+
+
+def pytest_addoption(parser: pytest.Parser) -> None:
+    group = parser.getgroup("xdist-memory", "size pytest-xdist by available RAM")
+    group.addoption(
+        "--xdist-memory",
+        action="store_true",
+        default=None,
+        help="measure worker memory, record it in the profile and answer "
+        "-n auto from it (ini: xdist_memory).",
+    )
+    group.addoption(
+        "--xdist-memory-profile",
+        default=None,
+        metavar="PATH",
+        help="profile file, relative to rootdir; keep it out of version "
+        "control (ini: xdist_memory_profile; default .pytest-xdist-memory.json).",
+    )
+    group.addoption(
+        "--xdist-memory-reserve",
+        type=int,
+        default=None,
+        metavar="MB",
+        help="memory to leave for the desktop / system "
+        "(ini: xdist_memory_reserve; default 2048).",
+    )
+    group.addoption(
+        "--xdist-memory-safety",
+        type=float,
+        default=None,
+        metavar="FACTOR",
+        help="multiplier on the measured worker peak "
+        "(ini: xdist_memory_safety; default 1.25).",
+    )
+    group.addoption(
+        "--xdist-memory-default-worker",
+        type=int,
+        default=None,
+        metavar="MB",
+        help="worker peak assumed while the profile has no runs for this suite "
+        "(ini: xdist_memory_default_worker; default 1024).",
+    )
+    group.addoption(
+        "--xdist-memory-extra",
+        type=int,
+        default=None,
+        metavar="MB",
+        help="fixed extra memory per worker outside the process tree, e.g. "
+        "its database (ini: xdist_memory_extra; default 0).",
+    )
+    group.addoption(
+        "--xdist-memory-docker",
+        default=None,
+        metavar="SUBSTRING",
+        help="also measure running docker containers whose name contains "
+        "SUBSTRING and count their peak / workers as extra per worker "
+        "(ini: xdist_memory_docker).",
+    )
+    group.addoption(
+        "--xdist-memory-docker-port-env",
+        default=None,
+        metavar="VAR",
+        help="also measure the docker container publishing the host port in "
+        "environment variable VAR, e.g. the one testcontainers exports for "
+        "the test database; exact even with parallel runs "
+        "(ini: xdist_memory_docker_port_env).",
+    )
+    group.addoption(
+        "--xdist-memory-performance-cores",
+        action="store_true",
+        default=None,
+        help="on Apple Silicon cap workers at the number of performance cores "
+        "(ini: xdist_memory_performance_cores).",
+    )
+    group.addoption(
+        "--xdist-memory-min-tests",
+        type=int,
+        default=None,
+        metavar="N",
+        help="record a run in the profile only if at least N tests ran "
+        "(ini: xdist_memory_min_tests; default 20).",
+    )
+    group.addoption(
+        "--xdist-memory-watchdog",
+        action="store_true",
+        default=None,
+        help="kill the run when the process tree (+ measured containers) "
+        "exceeds the memory available at start minus the reserve "
+        "(ini: xdist_memory_watchdog).",
+    )
+    group.addoption(
+        "--xdist-memory-interval",
+        type=float,
+        default=None,
+        metavar="SECONDS",
+        help="sampling interval (ini: xdist_memory_interval; default 2).",
+    )
+    for name, kind, default in (
+        ("xdist_memory", "bool", False),
+        ("xdist_memory_profile", "string", ".pytest-xdist-memory.json"),
+        ("xdist_memory_reserve", "string", "2048"),
+        ("xdist_memory_safety", "string", "1.25"),
+        ("xdist_memory_default_worker", "string", "1024"),
+        ("xdist_memory_extra", "string", "0"),
+        ("xdist_memory_docker", "string", ""),
+        ("xdist_memory_docker_port_env", "string", ""),
+        ("xdist_memory_performance_cores", "bool", False),
+        ("xdist_memory_min_tests", "string", "20"),
+        ("xdist_memory_watchdog", "bool", False),
+        ("xdist_memory_interval", "string", "2"),
+    ):
+        help_text = f"see --{name.replace('_', '-')}"
+        parser.addini(name, help_text, type=kind, default=default)
+
+
+class Settings:
+    def __init__(self, config: pytest.Config) -> None:
+        def opt(name: str, cast):
+            value = config.getoption(name)
+            if value is None:
+                value = config.getini(name)
+            return cast(value)
+
+        self.enabled = opt("xdist_memory", bool)
+        self.profile = config.rootpath / opt("xdist_memory_profile", str)
+        self.reserve_mb = opt("xdist_memory_reserve", int)
+        self.safety = opt("xdist_memory_safety", float)
+        self.default_worker_mb = opt("xdist_memory_default_worker", int)
+        self.extra_mb = opt("xdist_memory_extra", int)
+        self.docker = opt("xdist_memory_docker", str)
+        self.docker_port_env = opt("xdist_memory_docker_port_env", str)
+        self.performance_cores = opt("xdist_memory_performance_cores", bool)
+        self.min_tests = opt("xdist_memory_min_tests", int)
+        self.watchdog = opt("xdist_memory_watchdog", bool)
+        self.interval = opt("xdist_memory_interval", float)
+        self.suite = profile.suite_key(config.getoption("markexpr", None))
+
+    def estimate(self) -> profile.Estimate:
+        return profile.estimate(
+            profile.load(self.profile),
+            self.suite,
+            available_mb=memory.available_mb(),
+            reserve_mb=self.reserve_mb,
+            safety=self.safety,
+            cpu_cap=memory.cpu_cap(self.performance_cores),
+            default_worker_mb=self.default_worker_mb,
+            # Unknown before the first measured run; the controller is a
+            # plain pytest process, rarely more than a few hundred MB.
+            default_controller_mb=250,
+            extra_per_worker_mb=self.extra_mb,
+        )
+
+
+@pytest.hookimpl(tryfirst=True, optionalhook=True)
+def pytest_xdist_auto_num_workers(config: pytest.Config) -> int | None:
+    if os.environ.get("PYTEST_XDIST_AUTO_NUM_WORKERS"):
+        return None  # explicit override wins; let xdist's own hook read it
+    settings = Settings(config)
+    if not settings.enabled:
+        return None
+    est = settings.estimate()
+    config.stash[_ESTIMATE] = est
+    return est.workers
+
+
+class Sampler:
+    """Background thread recording the peak memory of a process."""
+
+    def __init__(
+        self,
+        interval: float,
+        subtree: bool,
+        docker: str = "",
+        docker_port_env: str = "",
+    ) -> None:
+        self.proc = psutil.Process()
+        self.interval = interval
+        self.subtree = subtree
+        self.docker = docker
+        self.docker_port_env = docker_port_env
+        self.peak_mb = 0
+        self.docker_peak_mb = 0
+        self.on_sample = None
+        self._stop = threading.Event()
+        self._thread = threading.Thread(
+            target=self._run, name="xdist-memory", daemon=True
+        )
+
+    def start(self) -> None:
+        self.sample()
+        self._thread.start()
+
+    def stop(self) -> None:
+        self._stop.set()
+        if self._thread.is_alive():
+            self._thread.join(timeout=5)
+        self.sample()
+
+    def sample(self) -> int:
+        if self.subtree:
+            mb = memory.subtree_mb(self.proc)
+        else:
+            mb = memory.process_mb(self.proc)
+        self.peak_mb = max(self.peak_mb, mb)
+        return mb
+
+    def sample_docker(self) -> None:
+        # Read the port at sample time: plugins (testcontainers) may export
+        # it after we were configured.
+        port = os.environ.get(self.docker_port_env) if self.docker_port_env else None
+        if self.docker_port_env and not port:
+            return
+        mb = memory.docker_mb(self.docker, port) or 0
+        self.docker_peak_mb = max(self.docker_peak_mb, mb)
+
+    def _run(self) -> None:
+        # docker stats is slow (~1 s); sample containers less often.
+        docker_every = max(1, int(10 / max(self.interval, 0.1)))
+        tick = 0
+        while not self._stop.wait(self.interval):
+            mb = self.sample()
+            if (self.docker or self.docker_port_env) and tick % docker_every == 0:
+                self.sample_docker()
+            tick += 1
+            if self.on_sample is not None:
+                self.on_sample(mb)
+
+
+class Watchdog:
+    """Kill the run before it takes the machine down.
+
+    Best effort: it reacts within one sampling interval, so a very fast
+    allocation spike can still get through. A cgroup limit (``systemd-run
+    --user --scope -p MemoryMax=...``) is the hard guarantee on Linux.
+    """
+
+    def __init__(self, settings: Settings, sampler: Sampler) -> None:
+        self.sampler = sampler
+        own = memory.process_mb(sampler.proc)
+        self.limit_mb = max(memory.available_mb() - settings.reserve_mb + own, 512)
+
+    def __call__(self, _own_mb: int) -> None:
+        total = memory.subtree_mb(self.sampler.proc) + self.sampler.docker_peak_mb
+        if total <= self.limit_mb:
+            return
+        sys.stderr.write(
+            f"\npytest-xdist-memory: process tree uses {total} MB, over the "
+            f"limit of {self.limit_mb} MB -- killing the test run.\n"
+        )
+        sys.stderr.flush()
+        for child in self.sampler.proc.children(recursive=True):
+            try:
+                child.kill()
+            except psutil.NoSuchProcess:
+                continue  # already gone
+        os._exit(3)
+
+
+def _is_worker(config: pytest.Config) -> bool:
+    return hasattr(config, "workerinput")
+
+
+def _is_controller(config: pytest.Config) -> bool:
+    # Same test as xdist's own is_xdist_controller(); the "dsession" plugin is
+    # not registered yet when our pytest_configure runs.
+    return (
+        not _is_worker(config)
+        and getattr(config.option, "dist", "no") != "no"
+        and bool(getattr(config.option, "tx", None))
+    )
+
+
+class Runtime:
+    def __init__(self, config: pytest.Config, settings: Settings) -> None:
+        self.config = config
+        self.settings = settings
+        self.worker = _is_worker(config)
+        self.controller = _is_controller(config)
+        # A worker (or a plain non-xdist run) measures its whole subtree; the
+        # controller only itself, its children are the workers.
+        self.sampler = Sampler(
+            settings.interval,
+            subtree=not self.controller,
+            docker=settings.docker if not self.worker else "",
+            docker_port_env=settings.docker_port_env if not self.worker else "",
+        )
+        self.workers: dict[str, dict[str, Any]] = {}
+        self.tests = 0
+        self.watchdog = None
+        if settings.watchdog and not self.worker:
+            self.watchdog = Watchdog(settings, self.sampler)
+            self.sampler.on_sample = self.watchdog
+
+    def pytest_runtest_logreport(self, report: pytest.TestReport) -> None:
+        if report.when == "call" or (report.when == "setup" and report.skipped):
+            self.tests += 1
+
+    @pytest.hookimpl(optionalhook=True)
+    def pytest_testnodedown(self, node, error) -> None:
+        data = getattr(node, "workeroutput", {}).get(_KEY)
+        if data:
+            self.workers[node.gateway.id] = data
+
+    def finish(self, session: pytest.Session) -> None:
+        self.sampler.stop()
+        if self.worker:
+            self.config.workeroutput[_KEY] = {"peak_mb": self.sampler.peak_mb}
+            return
+        if not self.controller:
+            return  # plain run: nothing to size, just report in the summary
+        peaks = sorted((w["peak_mb"] for w in self.workers.values()), reverse=True)
+        if not peaks:
+            return
+        n = len(peaks)
+        self.run = {
+            "host": os.uname().nodename if hasattr(os, "uname") else "",
+            "metric": "pss" if sys.platform.startswith("linux") else "rss",
+            "workers": n,
+            "tests": self.tests,
+            "worker_peak_mb": peaks[0],
+            "worker_peaks_mb": peaks,
+            "controller_peak_mb": self.sampler.peak_mb,
+            "extra_per_worker_mb": (self.sampler.docker_peak_mb // n) or None,
+        }
+        interrupted = session.shouldstop or session.shouldfail
+        if self.tests >= self.settings.min_tests and not interrupted:
+            profile.record_run(self.settings.profile, self.settings.suite, self.run)
+            self.recorded = True
+
+
+_RUNTIME = pytest.StashKey[Runtime]()
+_ESTIMATE = pytest.StashKey[profile.Estimate]()
+
+
+@pytest.hookimpl(trylast=True)
+def pytest_configure(config: pytest.Config) -> None:
+    settings = Settings(config)
+    if not settings.enabled:
+        return
+    runtime = Runtime(config, settings)
+    config.stash[_RUNTIME] = runtime
+    config.pluginmanager.register(runtime, "xdist-memory-runtime")
+    runtime.sampler.start()
+
+
+def pytest_sessionfinish(session: pytest.Session) -> None:
+    runtime = session.config.stash.get(_RUNTIME, None)
+    if runtime is not None:
+        runtime.finish(session)
+
+
+def pytest_terminal_summary(terminalreporter, exitstatus, config: pytest.Config):
+    runtime = config.stash.get(_RUNTIME, None)
+    if runtime is None or runtime.worker:
+        return
+    tr = terminalreporter
+    tr.write_sep("=", "xdist-memory")
+    est = config.stash.get(_ESTIMATE, None)
+    if est is not None:
+        tr.write_line(f"-n auto [{runtime.settings.suite}]: {est.describe()}")
+    run = getattr(runtime, "run", None)
+    if run is None:
+        tr.write_line(f"peak (whole tree, no xdist): {runtime.sampler.peak_mb} MB")
+        return
+    extra = run["extra_per_worker_mb"]
+    tr.write_line(
+        f"peak per worker {run['worker_peak_mb']} MB (all: {run['worker_peaks_mb']}),"
+        f" controller {run['controller_peak_mb']} MB"
+        + (f", containers {extra} MB/worker" if extra else "")
+    )
+    path = Path(runtime.settings.profile)
+    if getattr(runtime, "recorded", False):
+        tr.write_line(f"recorded in {path.name} [{runtime.settings.suite}]")
+    else:
+        tr.write_line(
+            f"not recorded ({runtime.tests} tests < "
+            f"{runtime.settings.min_tests} or run interrupted)"
+        )
+    after = runtime.settings.estimate()
+    tr.write_line(f"next -n auto would give: {after.describe()}")

pytest_xdist_memory-0.1.0/src/pytest_xdist_memory/profile.py

@@ -0,0 +1,115 @@
+"""The memory profile file and the worker-count calculation.
+
+The profile is per machine (keep it out of version control). It holds, per
+"suite" (the ``-m`` marker expression, or ``default``), the last few runs.
+The estimate takes the MAXIMUM over them: one lucky light run should not let
+the next one spawn too many workers.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import time
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any
+
+VERSION = 1
+KEEP_RUNS = 5
+
+
+def suite_key(markexpr: str | None) -> str:
+    return (markexpr or "").strip() or "default"
+
+
+def load(path: Path) -> dict[str, Any]:
+    try:
+        data = json.loads(path.read_text())
+    except FileNotFoundError:
+        return {"version": VERSION, "suites": {}}
+    except (OSError, ValueError) as exc:
+        raise ProfileError(f"cannot read memory profile {path}: {exc}") from exc
+    if data.get("version") != VERSION:
+        return {"version": VERSION, "suites": {}}
+    return data
+
+
+class ProfileError(Exception):
+    pass
+
+
+def record_run(path: Path, key: str, run: dict[str, Any]) -> None:
+    data = load(path)
+    runs = data["suites"].setdefault(key, {"runs": []})["runs"]
+    runs.append({"measured_at": time.strftime("%Y-%m-%d %H:%M:%S"), **run})
+    del runs[:-KEEP_RUNS]
+    tmp = path.with_name(path.name + f".{os.getpid()}.tmp")
+    tmp.write_text(json.dumps(data, indent=2, ensure_ascii=False) + "\n")
+    tmp.replace(path)
+
+
+@dataclass
+class Estimate:
+    workers: int
+    from_profile: bool
+    worker_mb: int
+    controller_mb: int
+    extra_per_worker_mb: int
+    per_worker_mb: int
+    available_mb: int
+    reserve_mb: int
+    budget_mb: int
+    cpu_cap: int
+    runs: int = 0
+    notes: list[str] = field(default_factory=list)
+
+    def describe(self) -> str:
+        src = f"profile ({self.runs} runs)" if self.from_profile else "defaults"
+        return (
+            f"{self.workers} workers: available {self.available_mb} MB"
+            f" - reserve {self.reserve_mb} - controller {self.controller_mb}"
+            f" = {self.budget_mb} MB; per worker {self.per_worker_mb} MB"
+            f" ({self.worker_mb} x safety + extra {self.extra_per_worker_mb});"
+            f" cpu cap {self.cpu_cap}; from {src}"
+        )
+
+
+def _peak(runs: list[dict[str, Any]], name: str) -> int | None:
+    values = [r[name] for r in runs if r.get(name)]
+    return max(values) if values else None
+
+
+def estimate(
+    data: dict[str, Any],
+    key: str,
+    *,
+    available_mb: int,
+    reserve_mb: int,
+    safety: float,
+    cpu_cap: int,
+    default_worker_mb: int,
+    default_controller_mb: int,
+    extra_per_worker_mb: int,
+) -> Estimate:
+    runs = data.get("suites", {}).get(key, {}).get("runs", [])
+    worker = _peak(runs, "worker_peak_mb") or default_worker_mb
+    controller = _peak(runs, "controller_peak_mb") or default_controller_mb
+    measured_extra = _peak(runs, "extra_per_worker_mb")
+    extra = max(extra_per_worker_mb, measured_extra or 0)
+    per_worker = int(worker * safety) + extra
+    budget = available_mb - reserve_mb - controller
+    n = max(1, min(budget // max(per_worker, 1), cpu_cap))
+    return Estimate(
+        workers=n,
+        from_profile=bool(runs),
+        worker_mb=worker,
+        controller_mb=controller,
+        extra_per_worker_mb=extra,
+        per_worker_mb=per_worker,
+        available_mb=available_mb,
+        reserve_mb=reserve_mb,
+        budget_mb=budget,
+        cpu_cap=cpu_cap,
+        runs=len(runs),
+    )

pytest_xdist_memory-0.1.0/tests/test_plugin.py

@@ -0,0 +1,234 @@
+import json
+
+import pytest
+
+from pytest_xdist_memory import profile
+
+TESTS = """
+import pytest
+
+@pytest.mark.parametrize("i", range(6))
+def test_many(i):
+    pass
+
+@pytest.mark.heavy
+@pytest.mark.parametrize("i", range(3))
+def test_heavy(i):
+    pass
+"""
+
+INI = """
+[pytest]
+markers = heavy
+"""
+
+
+def _profile(pytester) -> dict:
+    return json.loads((pytester.path / ".pytest-xdist-memory.json").read_text())
+
+
+def test_inactive_by_default(pytester):
+    pytester.makepyfile(TESTS)
+    pytester.makeini(INI)
+    result = pytester.runpytest("-n", "2")
+    result.assert_outcomes(passed=9)
+    result.stdout.no_fnmatch_line("*= xdist-memory =*")
+    assert not (pytester.path / ".pytest-xdist-memory.json").exists()
+
+
+def test_records_worker_peaks_per_suite(pytester):
+    pytester.makepyfile(TESTS)
+    pytester.makeini(INI)
+    args = ("-n", "2", "--xdist-memory", "--xdist-memory-min-tests", "1")
+    pytester.runpytest(*args).assert_outcomes(passed=9)
+    # (xdist's controller does not report the deselected count)
+    pytester.runpytest(*args, "-m", "heavy").assert_outcomes(passed=3)
+
+    suites = _profile(pytester)["suites"]
+    assert set(suites) == {"default", "heavy"}
+    run = suites["default"]["runs"][0]
+    assert run["workers"] == 2
+    assert run["tests"] == 9
+    assert len(run["worker_peaks_mb"]) == 2
+    # A Python interpreter with pytest loaded: tens of MB at the very least.
+    assert run["worker_peak_mb"] > 10
+    assert run["controller_peak_mb"] > 10
+
+
+def test_too_few_tests_are_not_recorded(pytester):
+    pytester.makepyfile(TESTS)
+    pytester.makeini(INI)
+    result = pytester.runpytest("-n", "2", "--xdist-memory")  # min-tests 20
+    result.assert_outcomes(passed=9)
+    result.stdout.fnmatch_lines(["*= xdist-memory =*", "*not recorded (9 tests < 20*"])
+    assert not (pytester.path / ".pytest-xdist-memory.json").exists()
+
+
+def test_auto_uses_profile(pytester, monkeypatch):
+    monkeypatch.delenv("PYTEST_XDIST_AUTO_NUM_WORKERS", raising=False)
+    pytester.makepyfile(TESTS)
+    pytester.makeini(INI)
+    # A worker "needing" more than this machine has: -n auto must give 1.
+    profile.record_run(
+        pytester.path / ".pytest-xdist-memory.json",
+        "default",
+        {"worker_peak_mb": 10_000_000, "controller_peak_mb": 100},
+    )
+    result = pytester.runpytest("-n", "auto", "--xdist-memory")
+    result.assert_outcomes(passed=9)
+    result.stdout.fnmatch_lines(["*-n auto [[]default[]]: 1 workers*from profile*"])
+
+
+def test_env_override_wins(pytester, monkeypatch):
+    monkeypatch.setenv("PYTEST_XDIST_AUTO_NUM_WORKERS", "2")
+    pytester.makepyfile(TESTS)
+    pytester.makeini(INI)
+    result = pytester.runpytest("-n", "auto", "--xdist-memory")
+    result.assert_outcomes(passed=9)
+    result.stdout.fnmatch_lines(["*2 workers*"])
+    result.stdout.no_fnmatch_line("*-n auto [[]default[]]*")
+
+
+def test_child_processes_count_towards_the_worker(pytester):
+    # Each test keeps a ~200 MB child process alive for a while: the worker
+    # peak must include it (that is what Chromium / Daphne look like).
+    pytester.makepyfile(
+        """
+        import subprocess, sys, time
+
+        # Touch every page so it is really resident, then hold it.
+        CHILD = (
+            "b = bytearray(200 * 1024 * 1024)\\n"
+            "for i in range(0, len(b), 4096): b[i] = 1\\n"
+            "import time; time.sleep(3)"
+        )
+
+        def test_child():
+            p = subprocess.Popen([sys.executable, "-c", CHILD])
+            time.sleep(2.5)
+            p.wait()
+        """
+    )
+    result = pytester.runpytest(
+        "-n",
+        "1",
+        "--xdist-memory",
+        "--xdist-memory-min-tests",
+        "1",
+        "--xdist-memory-interval",
+        "0.5",
+    )
+    result.assert_outcomes(passed=1)
+    run = _profile(pytester)["suites"]["default"]["runs"][0]
+    assert run["worker_peak_mb"] >= 200
+
+
+def test_watchdog_kills_run_over_budget(pytester):
+    # Reserve > all RAM: the limit drops to its 512 MB floor, and a 700 MB
+    # child process must trip it.
+    pytester.makepyfile(
+        """
+        import subprocess, sys, time
+
+        CHILD = (
+            "b = bytearray(700 * 1024 * 1024)\\n"
+            "for i in range(0, len(b), 4096): b[i] = 1\\n"
+            "import time; time.sleep(30)"
+        )
+
+        def test_hog():
+            subprocess.Popen([sys.executable, "-c", CHILD]).wait()
+        """
+    )
+    result = pytester.runpytest_subprocess(
+        "-n",
+        "1",
+        "--xdist-memory",
+        "--xdist-memory-watchdog",
+        "--xdist-memory-reserve",
+        "100000000",
+        "--xdist-memory-interval",
+        "0.5",
+        timeout=60,
+    )
+    assert result.ret == 3
+    result.stderr.fnmatch_lines(["*over the limit of 512 MB*killing the test run*"])
+
+
+def test_estimate_takes_max_of_recent_runs():
+    data = {
+        "suites": {
+            "s": {
+                "runs": [
+                    {"worker_peak_mb": 500, "controller_peak_mb": 100},
+                    {"worker_peak_mb": 900, "controller_peak_mb": 150},
+                    {"worker_peak_mb": 600, "controller_peak_mb": 120},
+                ]
+            }
+        }
+    }
+    est = profile.estimate(
+        data,
+        "s",
+        available_mb=10_000,
+        reserve_mb=2_000,
+        safety=1.0,
+        cpu_cap=64,
+        default_worker_mb=1,
+        default_controller_mb=1,
+        extra_per_worker_mb=100,
+    )
+    assert est.worker_mb == 900
+    assert est.controller_mb == 150
+    assert est.per_worker_mb == 1000
+    assert est.workers == (10_000 - 2_000 - 150) // 1000
+
+
+@pytest.mark.parametrize(
+    "available, cap, expected", [(100, 8, 1), (100_000, 3, 3), (6_000, 64, 3)]
+)
+def test_estimate_bounds(available, cap, expected):
+    est = profile.estimate(
+        {},
+        "default",
+        available_mb=available,
+        reserve_mb=1_000,
+        safety=1.0,
+        cpu_cap=cap,
+        default_worker_mb=1_500,
+        default_controller_mb=0,
+        extra_per_worker_mb=0,
+    )
+    assert est.workers == expected
+    assert not est.from_profile
+
+
+def test_profile_keeps_last_runs(tmp_path):
+    path = tmp_path / "p.json"
+    for i in range(profile.KEEP_RUNS + 3):
+        profile.record_run(path, "default", {"worker_peak_mb": i})
+    runs = profile.load(path)["suites"]["default"]["runs"]
+    assert [r["worker_peak_mb"] for r in runs] == list(range(3, profile.KEEP_RUNS + 3))
+
+
+def test_docker_matching_by_port_and_name(monkeypatch):
+    from pytest_xdist_memory import memory
+
+    calls = []
+
+    def fake_docker(*args):
+        calls.append(args)
+        if args[0] == "ps":
+            return "mine-psql\n" if args[2] == "publish=5432" else ""
+        names = args[4:] or ("mine-psql", "other-psql", "redis")
+        usage = {"mine-psql": "300MiB", "other-psql": "1.5GiB", "redis": "20MiB"}
+        return "".join(f"{n} {usage[n]} / 14GiB\n" for n in names)
+
+    monkeypatch.setattr(memory.shutil, "which", lambda _: "/usr/bin/docker")
+    monkeypatch.setattr(memory, "_docker", fake_docker)
+
+    assert memory.docker_mb(port="5432") == 300
+    assert calls[-1][4:] == ("mine-psql",)  # stats only for that container
+    assert memory.docker_mb(match="psql") == 300 + 1536
+    assert memory.docker_mb(port="9999") is None
+    assert memory.docker_mb() is None