pytest-regtest 2.1.1__tar.gz → 2.3.3__tar.gz

pytest_regtest-2.3.3/MANIFEST.in

@@ -0,0 +1,2 @@
+include LICENSE.txt
+include tests/*.py

pytest_regtest-2.3.3/PKG-INFO

@@ -0,0 +1,91 @@
+Metadata-Version: 2.1
+Name: pytest-regtest
+Version: 2.3.3
+Summary: pytest plugin for snapshot regression testing
+Author-email: Uwe Schmitt <uwe.schmitt@id.ethz.ch>
+License: MIT License
+Project-URL: Source, https://gitlab.com/uweschmitt/pytest-regtest
+Project-URL: Documentation, https://pytest-regtest.readthedocs.org
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: License :: OSI Approved :: MIT License
+Description-Content-Type: text/markdown
+License-File: LICENSE.txt
+Requires-Dist: pytest>7.2
+
+![](https://gitlab.com/uweschmitt/pytest-regtest/badges/main/pipeline.svg)
+![](https://gitlab.com/uweschmitt/pytest-regtest/badges/main/coverage.svg?job=coverage)
+
+
+The full documentation for this package are available at
+https://pytest-regtest.readthedocs.org
+
+# About
+
+## Introduction
+
+`pytest-regtest` is a plugin for [pytest](https://pytest.org) to implement
+**regression testing**.
+
+Unlike [functional testing](https://en.wikipedia.org/wiki/Functional_testing),
+[regression testing](https://en.wikipedia.org/wiki/Regression_testing)
+does not test whether the software produces the correct
+results, but whether it behaves as it did before changes were introduced.
+
+More specifically, `pytest-regtest` provides **snapshot testing**, which
+implements regression testing by recording data within a test function
+and comparing this recorded output to a previously recorded reference
+output.
+
+
+## Installation
+
+To install and activate this plugin execute:
+
+    $ pip install pytest-regtest
+
+
+!!! note
+
+     `pytest-regtest` provides some functionality specific to `NumPy`,
+     `pandas`, and `polars`. These dependencies are not installed when
+     you install `pytest-regtest`. For example, if you are using NumPy
+     snapshots, we assume that your production code (the code under
+     test) uses NumPy and therefore should be part of your project's
+     setup.
+
+
+## Use case 1: Changing code with no or little testing setup yet
+If you're working with code that has little or no unit testing, you
+can use regression testing to ensure that your changes don't break or
+alter previous results.
+
+**Example**:
+This can be useful when working with data analysis scripts, which often
+start as one long script and then are restructured into different
+functions as they evolve.
+
+
+## Use case 2:  Testing complex data
+If a unit tests contains many `assert` statements to check a complex
+data structure you can use regression tests instead.
+
+**Example**: To test code which ingests data into a database one can
+use regression tests on textual database dumps.
+
+## Use case 3: Testing NumPy arrays or pandas data frames
+
+If your code generates numerical results, such as `NumPy` arrays,
+`pandas` or `polars` data frames, you can use `pytest-regtest` to simply record such
+results and test them later, taking into account relative and absolute
+tolerances.
+
+
+**Example**:
+A function creates a 10 x 10 matrix. Either you have to write 100
+assert statements or you use summary statistics to test your result.
+In both cases, you may get little debugging information if a test
+fails.

pytest_regtest-2.3.3/README.md

@@ -0,0 +1,73 @@
+![](https://gitlab.com/uweschmitt/pytest-regtest/badges/main/pipeline.svg)
+![](https://gitlab.com/uweschmitt/pytest-regtest/badges/main/coverage.svg?job=coverage)
+
+
+The full documentation for this package are available at
+https://pytest-regtest.readthedocs.org
+
+# About
+
+## Introduction
+
+`pytest-regtest` is a plugin for [pytest](https://pytest.org) to implement
+**regression testing**.
+
+Unlike [functional testing](https://en.wikipedia.org/wiki/Functional_testing),
+[regression testing](https://en.wikipedia.org/wiki/Regression_testing)
+does not test whether the software produces the correct
+results, but whether it behaves as it did before changes were introduced.
+
+More specifically, `pytest-regtest` provides **snapshot testing**, which
+implements regression testing by recording data within a test function
+and comparing this recorded output to a previously recorded reference
+output.
+
+
+## Installation
+
+To install and activate this plugin execute:
+
+    $ pip install pytest-regtest
+
+
+!!! note
+
+     `pytest-regtest` provides some functionality specific to `NumPy`,
+     `pandas`, and `polars`. These dependencies are not installed when
+     you install `pytest-regtest`. For example, if you are using NumPy
+     snapshots, we assume that your production code (the code under
+     test) uses NumPy and therefore should be part of your project's
+     setup.
+
+
+## Use case 1: Changing code with no or little testing setup yet
+If you're working with code that has little or no unit testing, you
+can use regression testing to ensure that your changes don't break or
+alter previous results.
+
+**Example**:
+This can be useful when working with data analysis scripts, which often
+start as one long script and then are restructured into different
+functions as they evolve.
+
+
+## Use case 2:  Testing complex data
+If a unit tests contains many `assert` statements to check a complex
+data structure you can use regression tests instead.
+
+**Example**: To test code which ingests data into a database one can
+use regression tests on textual database dumps.
+
+## Use case 3: Testing NumPy arrays or pandas data frames
+
+If your code generates numerical results, such as `NumPy` arrays,
+`pandas` or `polars` data frames, you can use `pytest-regtest` to simply record such
+results and test them later, taking into account relative and absolute
+tolerances.
+
+
+**Example**:
+A function creates a 10 x 10 matrix. Either you have to write 100
+assert statements or you use summary statistics to test your result.
+In both cases, you may get little debugging information if a test
+fails.

pytest_regtest-2.3.3/pyproject.toml

@@ -0,0 +1,53 @@
+[build-system]
+requires = ["setuptools>=61"]
+build-backend = "setuptools.build_meta"
+
+
+
+[project]
+name = "pytest-regtest"
+version = "2.3.3"
+description = "pytest plugin for snapshot regression testing"
+readme = "README.md"
+authors = [
+  {name = "Uwe Schmitt", email = "uwe.schmitt@id.ethz.ch"}
+]
+
+license = {text = "MIT License"}
+
+classifiers = [
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "License :: OSI Approved :: MIT License"
+]
+
+dependencies = [
+  "pytest>7.2",
+]
+
+[project.urls]
+Source = "https://gitlab.com/uweschmitt/pytest-regtest"
+Documentation = "https://pytest-regtest.readthedocs.org"
+
+
+[project.entry-points.pytest11]
+regtest = "pytest_regtest"
+
+[tool.ruff]
+line-length = 88
+exclude = ["_regtest_output"]
+
+[tool.ruff.lint]
+ignore = ["E731", "E203"]
+
+[tool.uv]
+dev-dependencies = [
+    "twine", "build", "hatchling", "wheel", "pre-commit", "ruff", "black",
+    "pytest-cov", "numpy", "pandas", "mkdocs", "mkdocs-material", "mistletoe",
+    "mkdocs-awesome-pages-plugin", "jinja2-cli", "mkdocstrings[python]",
+    "numpy>=2", "pandas>=2", "polars>=1.9",
+    "md-transformer>=0.0.3"
+]

pytest_regtest-2.3.3/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

pytest_regtest-2.3.3/src/pytest_regtest/__init__.py

@@ -0,0 +1,108 @@
+from importlib.metadata import version as _version
+
+import pytest
+
+from .pytest_regtest import clear_converters  # noqa: F401
+from .pytest_regtest import patch_terminal_size  # noqa: F401
+from .pytest_regtest import register_converter_post  # noqa: F401
+from .pytest_regtest import register_converter_pre  # noqa: F401
+from .pytest_regtest import (
+    PytestRegtestCommonHooks,
+    PytestRegtestPlugin,
+    RegtestStream,
+    Snapshot,
+    SnapshotPlugin,
+)
+from .register_third_party_handlers import (
+    register_numpy_handler,
+    register_pandas_handler,
+    register_polars_handler,
+)
+
+from .snapshot_handler import register_python_object_handler
+
+__version__ = _version(__package__)
+
+
+def pytest_addoption(parser):
+    """Add options to control the timeout plugin"""
+    group = parser.getgroup("regtest", "regression test plugin")
+    group.addoption(
+        "--regtest-reset",
+        action="store_true",
+        help="do not run regtest but record current output",
+    )
+    group.addoption(
+        "--regtest-tee",
+        action="store_true",
+        default=False,
+        help="print recorded results to console too",
+    )
+    group.addoption(
+        "--regtest-consider-line-endings",
+        action="store_true",
+        default=False,
+        help="do not strip whitespaces at end of recorded lines",
+    )
+    group.addoption(
+        "--regtest-nodiff",
+        action="store_true",
+        default=False,
+        help="do not show diff output for failed regresson tests",
+    )
+    group.addoption(
+        "--regtest-disable-stdconv",
+        action="store_true",
+        default=False,
+        help=(
+            "do not apply standard output converters to clean up indeterministic output"
+        ),
+    )
+
+
+def pytest_configure(config):
+    common = PytestRegtestCommonHooks()
+    config.pluginmanager.register(common)
+    config.pluginmanager.register(PytestRegtestPlugin(common))
+    config.pluginmanager.register(SnapshotPlugin(common))
+
+
+@pytest.fixture
+def regtest(request):
+    yield RegtestStream(request)
+
+
+@pytest.fixture
+def snapshot(request):
+    yield Snapshot(request)
+
+
+@pytest.fixture
+def regtest_all(regtest):
+    yield regtest
+
+
+snapshot_all_output = regtest_all
+
+register_python_object_handler()
+
+try:
+    import pandas  # noqa: F401
+
+    register_pandas_handler()
+except ImportError:
+    pass
+
+try:
+    import numpy  # noqa: F401
+
+    register_numpy_handler()
+except ImportError:
+    pass
+
+try:
+    import polars  # noqa: F401
+
+    register_polars_handler()
+except ImportError:
+    pass

pytest_regtest-2.3.3/src/pytest_regtest/numpy_handler.py

@@ -0,0 +1,216 @@
+import difflib
+import io
+import os.path
+import warnings
+
+import numpy as np
+
+from .snapshot_handler import BaseSnapshotHandler
+from .utils import highlight_mismatches
+
+
+class NumpyHandler(BaseSnapshotHandler):
+    def __init__(self, handler_options, pytest_config, tw):
+        self.atol = handler_options.get("atol", 0.0)
+        self.rtol = handler_options.get("rtol", 0.0)
+        self.equal_nan = handler_options.get("equal_nan", True)
+        if handler_options.get("print_options"):
+            warnings.warn(
+                "please use the numpy.printoptions context manager instead of"
+                " the print_options argument.",
+                DeprecationWarning,
+            )
+
+        self.print_options = np.get_printoptions() | handler_options.get(
+            "print_options", {}
+        )
+
+    def _filename(self, folder):
+        return os.path.join(folder, "arrays.npy")
+
+    def save(self, folder, obj):
+        np.save(self._filename(folder), obj)
+
+    def load(self, folder):
+        return np.load(self._filename(folder))
+
+    def show(self, obj):
+        stream = io.StringIO()
+        with np.printoptions(**self.print_options):
+            print(obj, file=stream)
+        return stream.getvalue().splitlines()
+
+    def compare(self, current_obj, recorded_obj):
+        return (
+            isinstance(current_obj, np.ndarray)
+            and current_obj.shape == recorded_obj.shape
+            and current_obj.dtype == recorded_obj.dtype
+            and np.allclose(
+                recorded_obj,
+                current_obj,
+                atol=self.atol,
+                rtol=self.rtol,
+                equal_nan=self.equal_nan,
+            )
+        )
+
+    def show_differences(self, current_obj, recorded_obj, has_markup):
+        lines = []
+
+        if recorded_obj.dtype != current_obj.dtype:
+            lines.extend(
+                [
+                    f"dtype mismatch: current dtype: {current_obj.dtype}",
+                    f"               recorded dtype: {recorded_obj.dtype}",
+                ]
+            )
+
+        recorded_as_text = self.show(recorded_obj)
+        current_as_text = self.show(current_obj)
+
+        if recorded_obj.shape == current_obj.shape:
+            if np.allclose(current_obj, recorded_obj, rtol=self.rtol, atol=self.atol):
+                return lines or None
+
+            lines.extend(self.error_diagnostics(recorded_obj, current_obj))
+
+        else:
+            lines.extend(
+                [
+                    f"shape mismatch: current shape: {current_obj.shape}",
+                    f"               recorded shape: {recorded_obj.shape}",
+                ]
+            )
+
+        if recorded_obj.ndim > 2:
+            return lines
+
+        if recorded_obj.ndim == 1:
+            diff_lines = list(
+                difflib.unified_diff(
+                    current_as_text,
+                    recorded_as_text,
+                    "current",
+                    "expected",
+                    lineterm="",
+                )
+            )
+            lines.append("")
+            lines.extend(diff_lines)
+
+        else:
+            diff_lines = self.error_diagnostics_2d_linewise(
+                current_obj,
+                current_as_text,
+                recorded_obj,
+                recorded_as_text,
+                has_markup,
+            )
+            lines.extend(diff_lines)
+
+        if not diff_lines:
+            lines.append("diff is empty, you may want to change the print options")
+
+        return lines
+
+    def error_diagnostics(self, recorded_obj, current_obj):
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", RuntimeWarning)
+            rel_err = np.abs(current_obj - recorded_obj) / recorded_obj
+        rel_err[(recorded_obj == 0) * (current_obj == recorded_obj)] = 0.0
+        rel_err_max_1 = np.max(rel_err)
+        rel_err_max_2 = np.max(rel_err[recorded_obj != 0])
+
+        abs_err = np.abs(current_obj - recorded_obj)
+        abs_err_max = np.max(abs_err)
+
+        lines = []
+
+        if rel_err_max_1 == rel_err_max_2:
+            lines.append(f"max relative deviation: {rel_err_max_1:e}")
+        else:
+            lines.append(f"max relative deviation: {rel_err_max_1:e}")
+            lines.append(f"max relative deviation except inf: {rel_err_max_2:e}")
+
+        lines.append(f"max absolute deviation: {abs_err_max:e}")
+
+        n_diff = np.sum(
+            np.logical_not(
+                np.isclose(current_obj, recorded_obj, rtol=self.rtol, atol=self.atol)
+            )
+        )
+
+        lines.append(
+            f"both arrays differ in {n_diff} out of {np.prod(recorded_obj.shape)}"
+            " entries"
+        )
+        lines.append(
+            f"up to given precision settings rtol={self.rtol:e} and"
+            f" atol={self.atol:e}"
+        )
+
+        return lines
+
+    def error_diagnostics_2d_linewise(
+        self, current_obj, current_as_text, recorded_obj, recorded_as_text, has_markup
+    ):
+        sub_diff = []
+
+        for i, (l1, l2, r1, r2) in enumerate(
+            zip(current_as_text, recorded_as_text, current_obj, recorded_obj)
+        ):
+            if r1.shape == r2.shape and np.allclose(
+                r1, r2, rtol=self.rtol, atol=self.atol
+            ):
+                continue
+
+            if r1.shape == r2.shape:
+                # enforces more uniform formatting of both lines:
+                rows_together = np.vstack((r1, r2))
+                lines_together = self.show(rows_together)
+                line_diff = list(
+                    difflib.unified_diff(
+                        [lines_together[0][1:].strip()],
+                        [lines_together[1][:-1].strip()],
+                        "current",
+                        "expected",
+                        lineterm="",
+                    )
+                )
+            else:
+                row_1 = self.show(r1)
+                row_2 = self.show(r2)
+                line_diff = list(
+                    difflib.unified_diff(
+                        row_1,
+                        row_2,
+                        "current",
+                        "expected",
+                        lineterm="",
+                    )
+                )
+
+            if line_diff:
+                if not sub_diff:
+                    sub_diff = line_diff[:2]
+
+                l1, l2 = line_diff[-2], line_diff[-1]
+                if has_markup:
+                    l1, l2 = highlight_mismatches(l1, l2)
+
+                sub_diff.append(f"row {i:3d}: {l1}")
+                sub_diff.append(f"         {l2}")
+
+        missing = len(current_as_text) - len(recorded_as_text)
+        if missing > 0:
+            for i, row in enumerate(current_as_text[-missing:], len(recorded_as_text)):
+                # remove duplicate brackets
+                row = row.rstrip("]") + "]"
+                sub_diff.append(f"row {i:3d}: -{row.lstrip()}")
+        if missing < 0:
+            for i, row in enumerate(recorded_as_text[missing:], len(current_as_text)):
+                # remove duplicate brackets
+                row = row.rstrip("]") + "]"
+                sub_diff.append(f"row {i:3d}: +{row.lstrip()}")
+
+        return sub_diff

pytest_regtest-2.3.3/src/pytest_regtest/pandas_handler.py

@@ -0,0 +1,143 @@
+import difflib
+import io
+import os.path
+import warnings
+
+import numpy as np
+import pandas as pd
+
+from .snapshot_handler import BaseSnapshotHandler
+
+
+class DataFrameHandler(BaseSnapshotHandler):
+    def __init__(self, handler_options, pytest_config, tw):
+        if handler_options.get("display_options"):
+            warnings.warn(
+                "please use the 'pandas.option_context' context manager instead of"
+                " the display_options argument.",
+                DeprecationWarning,
+            )
+
+        # default contains a few nested dicts and we flatten those, e.g.
+        # { "html": {"border": 1} } -> { "html.border": 1 }
+        default = list(pd.options.display.d.items())
+        default_flattened = {}
+        for k, v in default:
+            if isinstance(v, dict):
+                for k0, v0 in v.items():
+                    default_flattened[f"{k}.{k0}"] = v0
+            else:
+                default_flattened[k] = v
+
+        # overwrite with user settings:
+        items = (default_flattened | handler_options.get("display_options", {})).items()
+
+        # flatten items as required by pandas.option_context:
+        self.display_options_flat = [
+            entry for item in items for entry in (f"display.{item[0]}", item[1])
+        ]
+        self.atol = handler_options.get("atol", 0.0)
+        self.rtol = handler_options.get("rtol", 0.0)
+
+    def _filename(self, folder):
+        return os.path.join(folder, "dataframe.pkl")
+
+    def save(self, folder, obj):
+        obj.to_pickle(self._filename(folder), compression="gzip")
+
+    def load(self, folder):
+        return pd.read_pickle(self._filename(folder), compression="gzip")
+
+    def show(self, obj):
+        stream = io.StringIO()
+        with pd.option_context(*self.display_options_flat):
+            print(obj, file=stream)
+        return stream.getvalue().splitlines()
+
+    def compare(self, current, recorded):
+        missing = set(
+            n
+            for (n, t) in set(zip(recorded.columns, recorded.dtypes))
+            ^ set(zip(current.columns, current.dtypes))
+        )
+
+        if missing:
+            return False
+
+        common = set(
+            n
+            for (n, t) in set(zip(recorded.columns, recorded.dtypes))
+            & set(zip(current.columns, current.dtypes))
+        )
+        current_reduced = current[[n for n in current.columns if n in common]]
+        recorded_reduced = recorded[[n for n in recorded.columns if n in common]]
+
+        def extract(df, selector):
+            return df[[n for (n, t) in zip(df.columns, df.dtypes) if selector(t)]]
+
+        current_reduced_floats = extract(
+            current_reduced, lambda t: t.type is np.float64
+        ).to_numpy()
+
+        current_reduced_other = extract(
+            current_reduced, lambda t: t.type is not np.float64
+        )
+
+        recorded_reduced_floats = extract(
+            recorded_reduced, lambda t: t.type is np.float64
+        ).to_numpy()
+
+        recorded_reduced_other = extract(
+            recorded_reduced, lambda t: t.type is not np.float64
+        )
+
+        return np.allclose(
+            current_reduced_floats,
+            recorded_reduced_floats,
+            atol=self.atol,
+            rtol=self.rtol,
+            equal_nan=True,
+        ) and (current_reduced_other == recorded_reduced_other).all(axis=None)
+
+    def show_differences(self, current, recorded, has_markup):
+        lines = []
+
+        stream = io.StringIO()
+        current.info(buf=stream, verbose=True, memory_usage=False)
+        current_info = stream.getvalue().splitlines()[2:][:-1]
+
+        stream = io.StringIO()
+        recorded.info(buf=stream, verbose=True, memory_usage=False)
+        recorded_info = stream.getvalue().splitlines()[2:][:-1]
+
+        info_diff = list(
+            difflib.unified_diff(
+                current_info,
+                recorded_info,
+                "current",
+                "expected",
+                lineterm="",
+            )
+        )
+        lines.extend(info_diff)
+
+        recorded_as_text = self.show(recorded)
+        current_as_text = self.show(current)
+
+        diffs = list(
+            difflib.unified_diff(
+                current_as_text,
+                recorded_as_text,
+                "current",
+                "expected",
+                lineterm="",
+            )
+        )
+
+        lines.append("")
+        if diffs:
+            lines.extend(diffs)
+        else:
+            lines.append("diff is empty, you may want to change the print options")
+
+        return lines