pytest-digline 0.1.0__tar.gz

pytest_digline-0.1.0/.gitignore

@@ -0,0 +1,45 @@
+# Python bytecode and build output
+__pycache__/
+*.py[cod]
+build/
+dist/
+*.egg-info/
+
+# Environment. Recreated by `uv sync`; it pins absolute paths, so it must
+# never be committed.
+.venv/
+.env
+
+# Tool caches. Each already drops its own `.gitignore`; listed here so a
+# fresh clone is clean before the tools have run once.
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+
+# IDE. Excluded because the project files carry machine-specific SDK paths
+# (`digline.iml` names the interpreter by absolute path). Drop these two lines
+# to version the shared part, and keep ignoring `.idea/workspace.xml`.
+.idea/
+*.iml
+
+# Local Claude Code settings. `.claude/settings.json`, if it appears, is shared
+# and stays versioned. `CLAUDE.local.md` is the personal working agreement —
+# how I want to be worked with — as against `CLAUDE.md`, which is the project.
+.claude/settings.local.json
+CLAUDE.local.md
+
+# macOS
+.DS_Store
+
+# Working material that stays local and is not part of the package.
+private/
+
+# NOT ignored: `.digline/`. Decision 2 — baselines are versioned, run
+# artifacts are not — and the split is enforced one level down, by the
+# `.gitignore` the store itself writes into `.digline/` (`*/runs/`).
+# Ignoring `.digline/` here would take the baselines out of git with it.
+to-publish/
+
+# The site checkout that ci.yml's `docs` job makes, and that RELEASING
+# tells you to make to reproduce it. Never committed here.
+_site/

pytest_digline-0.1.0/PKG-INFO

@@ -0,0 +1,124 @@
+Metadata-Version: 2.5
+Name: pytest-digline
+Version: 0.1.0
+Summary: Gate a pytest run on a digline comparison: one row per check, against the baseline committed in your repo.
+Project-URL: Homepage, https://digline.dev/
+Project-URL: Documentation, https://digline.dev/product/pytest/
+Project-URL: Changelog, https://digline.dev/product/changelog/
+Project-URL: Repository, https://github.com/digline/digline
+Project-URL: Issues, https://github.com/digline/digline/issues
+Author-email: Alessandro Prandini <alessandro.prandini@ict-group.it>
+License-Expression: Apache-2.0
+Classifier: Development Status :: 3 - Alpha
+Classifier: Framework :: Pytest
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Testing
+Classifier: Typing :: Typed
+Requires-Python: >=3.12
+Requires-Dist: digline>=0.9.0
+Requires-Dist: pytest>=8.0
+Description-Content-Type: text/markdown
+
+# pytest-digline
+
+The digline comparison, as rows in pytest's own report.
+
+```sh
+pip install pytest-digline
+pytest --digline-suite eval/suite.py
+```
+
+```
+digline: comparing 1 suite(s) against the committed baseline
+
+eval/suite.py::alpha::contains          .
+eval/suite.py::alpha::llm_rubric        F
+eval/suite.py::beta::contains           .
+eval/suite.py::gamma                    s
+
+=================================== FAILURES ===================================
+digline: alpha · llm_rubric
+  dropped from 0.910000 to 0.640000, below its threshold of 0.700000,
+  and beyond the 0.880000–0.950000 this check measured across 5 samples
+  reason: signed=True, concise=False
+
+==================================== digline ===================================
+1 check got worse; every case could be judged; 1 case is suspended; the rules
+are unchanged.
+```
+
+One row per **check** — one assertion on one case — because that is digline's
+unit of verdict. `-k` and `--last-failed` select them like any other test.
+
+## What it does, and what it costs
+
+By default it **compares and never runs**: it reads the latest stored run of
+each suite, holds it against the baseline committed in your repository, and
+makes no network call at all. Producing a run stays a separate act —
+`digline run`, the official image, the GitHub Action.
+
+`--digline-run` runs each suite first. That calls your provider and spends
+money, so it is a flag on the command line where the cost is visible, it prints
+the planned call count before the first call, and it **refuses under
+`--collect-only`**: a command whose job is to list test names must never be able
+to spend a hundred model calls.
+
+## The four states
+
+| digline | pytest |
+|---|---|
+| fine | passed |
+| got worse | **FAILED** |
+| could not be judged | **ERROR** — an error is neither green nor a regression |
+| suspended | **SKIPPED**, with the reason the suite declared |
+
+The suspension is the one digline state an exit code cannot express: it never
+fails, so it disappears into `0`. pytest has had a state for *a decision rather
+than an outcome* since it was written, and this is it.
+
+**What pytest cannot carry**: the process exit code. `digline compare` exits `1`
+for a regression and `2` for a run it could not judge; a pytest run exits `1`
+for either. The distinction survives in the report — `F` and `E` are counted
+separately, `-rE` lists the errored rows — and in `--junit-xml`. A job that
+needs `1` versus `2` runs `digline compare`, which is one command away.
+
+## Naming a suite
+
+Nothing is discovered by convention: a suite is a file that *executes*, and a
+file found by convention is a file that runs by accident. Name it, either way:
+
+```sh
+pytest --digline-suite eval/suite.py --digline-suite eval/billing.toml
+```
+
+```ini
+# pyproject.toml
+[tool.pytest.ini_options]
+digline_suites = ["eval/suite.py"]
+```
+
+The command line replaces the ini list rather than adding to it, so narrowing a
+run to one suite is expressible. `--digline-root` names the perimeter holding
+`.digline/` and defaults to pytest's rootdir.
+
+Select rows with `-k`; a node id passed as an argument is not supported.
+
+## What it will not do
+
+**It does not promote.** There is no flag, no fixture and no marker that makes
+a run the new baseline — not refused, *absent*, and a test in this package
+sweeps the sources to keep it that way. A baseline is an approved reference;
+the approval is a person's, and a green pytest run is the single most likely
+place for a promotion to happen by accident.
+
+**It has no thresholds of its own.** Every bar it reports against was declared
+in your suite and frozen in the baseline somebody promoted and committed.
+
+**It depends on digline and pytest, and nothing else.**
+
+Turn it off for one run with `-p no:digline`. With no suite named it collects
+nothing, prints nothing and adds no header.
+
+The reasoning is [ADR 0013](https://digline.dev/product/adr/0013-the-pytest-plugin/).

pytest_digline-0.1.0/README.md

@@ -0,0 +1,101 @@
+# pytest-digline
+
+The digline comparison, as rows in pytest's own report.
+
+```sh
+pip install pytest-digline
+pytest --digline-suite eval/suite.py
+```
+
+```
+digline: comparing 1 suite(s) against the committed baseline
+
+eval/suite.py::alpha::contains          .
+eval/suite.py::alpha::llm_rubric        F
+eval/suite.py::beta::contains           .
+eval/suite.py::gamma                    s
+
+=================================== FAILURES ===================================
+digline: alpha · llm_rubric
+  dropped from 0.910000 to 0.640000, below its threshold of 0.700000,
+  and beyond the 0.880000–0.950000 this check measured across 5 samples
+  reason: signed=True, concise=False
+
+==================================== digline ===================================
+1 check got worse; every case could be judged; 1 case is suspended; the rules
+are unchanged.
+```
+
+One row per **check** — one assertion on one case — because that is digline's
+unit of verdict. `-k` and `--last-failed` select them like any other test.
+
+## What it does, and what it costs
+
+By default it **compares and never runs**: it reads the latest stored run of
+each suite, holds it against the baseline committed in your repository, and
+makes no network call at all. Producing a run stays a separate act —
+`digline run`, the official image, the GitHub Action.
+
+`--digline-run` runs each suite first. That calls your provider and spends
+money, so it is a flag on the command line where the cost is visible, it prints
+the planned call count before the first call, and it **refuses under
+`--collect-only`**: a command whose job is to list test names must never be able
+to spend a hundred model calls.
+
+## The four states
+
+| digline | pytest |
+|---|---|
+| fine | passed |
+| got worse | **FAILED** |
+| could not be judged | **ERROR** — an error is neither green nor a regression |
+| suspended | **SKIPPED**, with the reason the suite declared |
+
+The suspension is the one digline state an exit code cannot express: it never
+fails, so it disappears into `0`. pytest has had a state for *a decision rather
+than an outcome* since it was written, and this is it.
+
+**What pytest cannot carry**: the process exit code. `digline compare` exits `1`
+for a regression and `2` for a run it could not judge; a pytest run exits `1`
+for either. The distinction survives in the report — `F` and `E` are counted
+separately, `-rE` lists the errored rows — and in `--junit-xml`. A job that
+needs `1` versus `2` runs `digline compare`, which is one command away.
+
+## Naming a suite
+
+Nothing is discovered by convention: a suite is a file that *executes*, and a
+file found by convention is a file that runs by accident. Name it, either way:
+
+```sh
+pytest --digline-suite eval/suite.py --digline-suite eval/billing.toml
+```
+
+```ini
+# pyproject.toml
+[tool.pytest.ini_options]
+digline_suites = ["eval/suite.py"]
+```
+
+The command line replaces the ini list rather than adding to it, so narrowing a
+run to one suite is expressible. `--digline-root` names the perimeter holding
+`.digline/` and defaults to pytest's rootdir.
+
+Select rows with `-k`; a node id passed as an argument is not supported.
+
+## What it will not do
+
+**It does not promote.** There is no flag, no fixture and no marker that makes
+a run the new baseline — not refused, *absent*, and a test in this package
+sweeps the sources to keep it that way. A baseline is an approved reference;
+the approval is a person's, and a green pytest run is the single most likely
+place for a promotion to happen by accident.
+
+**It has no thresholds of its own.** Every bar it reports against was declared
+in your suite and frozen in the baseline somebody promoted and committed.
+
+**It depends on digline and pytest, and nothing else.**
+
+Turn it off for one run with `-p no:digline`. With no suite named it collects
+nothing, prints nothing and adds no header.
+
+The reasoning is [ADR 0013](https://digline.dev/product/adr/0013-the-pytest-plugin/).

pytest_digline-0.1.0/pyproject.toml

@@ -0,0 +1,57 @@
+[project]
+name = "pytest-digline"
+version = "0.1.0"
+description = "Gate a pytest run on a digline comparison: one row per check, against the baseline committed in your repo."
+readme = "README.md"
+requires-python = ">=3.12"
+license = "Apache-2.0"
+authors = [
+    { name = "Alessandro Prandini", email = "alessandro.prandini@ict-group.it" },
+]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Framework :: Pytest",
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3 :: Only",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Software Development :: Testing",
+    "Typing :: Typed",
+]
+dependencies = [
+    # 0.9.0 is the release that first publishes `report.check_line`, the
+    # per-delta sentence this package prints on a failing row (ADR 0013 §6).
+    # Held there by `tests/test_plugin_floors.py`, which computes the floor
+    # from the newest name these sources import rather than taking it on trust.
+    "digline>=0.9.0",
+    # A floor and no cap, unlike `digline-mcp`'s `mcp<3`. That cap records a
+    # rename that actually happened; pytest has not moved the `pytest11` entry
+    # point in a decade, and a cap here would fight the one dependency the user
+    # certainly already has. 8.0 is where `@pytest.hookimpl(wrapper=True)` — the
+    # hook that keeps an ERROR row free of a traceback — became available.
+    "pytest>=8.0",
+]
+
+# The same storefront as `digline` itself: a plugin's PyPI page is where
+# somebody meets it first, so it points at the same site and the same repo
+# rather than being a page of no links at all.
+[project.urls]
+Homepage = "https://digline.dev/"
+Documentation = "https://digline.dev/product/pytest/"
+Changelog = "https://digline.dev/product/changelog/"
+Repository = "https://github.com/digline/digline"
+Issues = "https://github.com/digline/digline/issues"
+
+# Named `digline` and not `pytest_digline`: this is the name a user types to
+# switch it off — `pytest -p no:digline` — and it is what the header prints.
+[project.entry-points.pytest11]
+digline = "pytest_digline.plugin"
+
+[build-system]
+requires = ["hatchling>=1.27"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/pytest_digline"]
+
+[tool.uv.sources]
+digline = { workspace = true }

pytest_digline-0.1.0/src/pytest_digline/__init__.py

@@ -0,0 +1,15 @@
+"""`pytest-digline`: the comparison, as rows in pytest's own report.
+
+The plugin lives in `pytest_digline.plugin`, which is what the `pytest11` entry
+point names. Nothing is exported here: a pytest plugin is not a library, and a
+name importable from the package root is a surface somebody would write against.
+"""
+
+from importlib.metadata import version as _distribution_version
+
+# Read from the installed distribution rather than written here, for the reason
+# `digline.__version__` is: a hand-written copy is neither derived nor gated,
+# and `tests/test_versions.py` sweeps these sources for exactly that.
+__version__ = _distribution_version("pytest-digline")
+
+__all__ = ["__version__"]