pytest-regtest 2.2.0a1__py2.py3-none-any.whl

pytest_regtest-2.2.0a1.dist-info/METADATA

@@ -0,0 +1,352 @@
+Metadata-Version: 2.3
+Name: pytest-regtest
+Version: 2.2.0a1
+Summary: pytest plugin for snapshot regression testing
+Project-URL: Source, https://gitlab.com/uweschmitt/pytest-regtest
+Project-URL: Documentation, https://pytest-regtest.readthedocs.org
+Author-email: Uwe Schmitt <uwe.schmitt@id.ethz.ch>
+License: MIT License
+License-File: LICENSE.txt
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Requires-Dist: pytest>7.2
+Provides-Extra: dev
+Requires-Dist: black; extra == 'dev'
+Requires-Dist: build; extra == 'dev'
+Requires-Dist: hatchling; extra == 'dev'
+Requires-Dist: numpy; extra == 'dev'
+Requires-Dist: pandas; extra == 'dev'
+Requires-Dist: pre-commit; extra == 'dev'
+Requires-Dist: pytest-cov; extra == 'dev'
+Requires-Dist: ruff; extra == 'dev'
+Requires-Dist: tox; extra == 'dev'
+Requires-Dist: twine; extra == 'dev'
+Requires-Dist: wheel; extra == 'dev'
+Description-Content-Type: text/markdown
+
+
+# Home
+
+## About
+
+`pytest-regtest` is a plugin for [pytest](https://pytest.org) to implement
+**regression testing**.
+
+Unlike [functional testing](https://en.wikipedia.org/wiki/Functional_testing),
+[regression testing](https://en.wikipedia.org/wiki/Regression_testing)
+testing does not test whether the software produces the correct
+results, but whether it behaves as it did before changes were introduced.
+
+More specifically, `pytest-regtest` provides **snapshot testing**, which
+implements regression testing by recording the textual output of a test
+function and comparing this recorded output to a reference output.
+
+**Regression testing** is a common technique to implement basic testing
+before refactoring legacy code that lacks a test suite.
+
+Snapshot testing can also be used to implement tests for complex outcomes, such
+as recording textual database dumps or the results of a scientific analysis
+routine.
+
+
+## Installation
+
+To install and activate this plugin execute:
+
+    $ pip install pytest-regtest
+
+## Basic Usage
+
+
+### Write a test
+
+The `pytest-regtest` plugin provides multiple fixtures.
+To record output, use the fixture `regtest` that works like a file handle:
+
+```py
+def test_squares(regtest):
+
+    result = [i*i for i in range(10)]
+
+    # one way to record output:
+    print(result, file=regtest)
+
+    # alternative method to record output:
+    regtest.write("done")
+```
+
+You can also use the `regtest_all` fixture. This enables all output to stdout to be
+recorded in a test function.
+
+
+### Run the test
+
+If you run this test script with *pytest* the first time there is no
+recorded output for this test function so far and thus the test will
+fail with a message including a diff:
+
+```
+$ pytest -v test_squares.py
+============================= test session starts ==============================
+platform darwin -- Python 3.11.4, pytest-8.0.0, pluggy-1.3.0 -- ...
+cachedir: .pytest_cache
+rootdir: ...
+plugins: cov-4.1.0, pytest_regtest-2.1.0
+collecting ... collected 1 item
+
+test_squares.py::test_squares FAILED                                     [100%]
+
+=================================== FAILURES ===================================
+_________________________________ test_squares _________________________________
+
+regression test output not recorded yet for test_squares.py::test_squares:
+
+[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
+done
+---------------------------- pytest-regtest report -----------------------------
+total number of failed regression tests: 1
+=========================== short test summary info ============================
+FAILED test_squares.py::test_squares
+============================== 1 failed in 0.02s ===============================
+```
+
+This is a diff of the current output `is` to a previously recorded output
+`tobe`. Since we did not record output yet, the diff contains no lines marked
+`+`.
+
+
+### Reset the test
+
+To record the current output, we run *pytest* with the *--regtest-reset*
+flag:
+
+```
+$ pytest -v --regtest-reset test_squares.py
+============================= test session starts ==============================
+platform darwin -- Python 3.11.4, pytest-8.0.0, pluggy-1.3.0 -- ...
+cachedir: .pytest_cache
+rootdir: ...
+plugins: cov-4.1.0, pytest_regtest-2.1.0
+collecting ... collected 1 item
+
+test_squares.py::test_squares RESET                                      [100%]
+
+---------------------------- pytest-regtest report -----------------------------
+total number of failed regression tests: 0
+the following output files have been reset:
+  _regtest_outputs/test_squares.test_squares.out
+============================== 1 passed in 0.00s ===============================
+```
+
+You can also see from the output that the recorded output is in the
+`_regtest_outputs` folder which in the same folder as the test script.
+Don't forget to commit this folder to your version control system!
+
+### Run the test again
+
+When we run the test again, it succeeds:
+
+```
+$ pytest -v test_squares.py
+============================= test session starts ==============================
+platform darwin -- Python 3.11.4, pytest-8.0.0, pluggy-1.3.0 -- ...
+cachedir: .pytest_cache
+rootdir: ...
+plugins: cov-4.1.0, pytest_regtest-2.1.0
+collecting ... collected 1 item
+
+test_squares.py::test_squares PASSED                                     [100%]
+
+---------------------------- pytest-regtest report -----------------------------
+total number of failed regression tests: 0
+============================== 1 passed in 0.00s ===============================
+```
+
+### Break the test
+
+Let us break the test by changing the test function to compute
+11 instead of 10 square numbers:
+
+```py
+def test_squares(regtest):
+
+    result = [i*i for i in range(11)]
+
+    # one way to record output:
+    print(result, file=regtest)
+
+    # alternative method to record output:
+    regtest.write("done")
+```
+
+The next run of pytest delivers a nice diff of the current and expected output
+from this test function:
+
+```
+$ pytest -v test_squares.py
+============================= test session starts ==============================
+platform darwin -- Python 3.11.4, pytest-8.0.0, pluggy-1.3.0 -- ...
+cachedir: .pytest_cache
+rootdir: ...
+plugins: cov-4.1.0, pytest_regtest-2.1.0
+collecting ... collected 1 item
+
+test_squares.py::test_squares FAILED                                     [100%]
+
+=================================== FAILURES ===================================
+_________________________________ test_squares _________________________________
+
+regression test output differences for test_squares.py::test_squares:
+    (recorded output from _regtest_outputs/test_squares.test_squares.out)
+
+    >   --- current
+    >   +++ expected
+    >   @@ -1,2 +1,2 @@
+    >   -[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
+    >   +[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
+    >    done
+
+---------------------------- pytest-regtest report -----------------------------
+total number of failed regression tests: 1
+=========================== short test summary info ============================
+FAILED test_squares.py::test_squares
+============================== 1 failed in 0.02s ===============================
+```
+
+
+## Other features
+
+### Using the `regtest` fixture as context manager
+
+The `regtest` fixture also works as a context manager  to capture
+all output from the wrapped code block:
+
+```py
+def test_squares(regtest):
+
+    result = [i*i for i in range(10)]
+
+    with regtest:
+        print(result)
+```
+
+### The `regtest_all` fixture
+
+The `regtest_all` fixture leads to recording of all output to `stdout` in a
+test function.
+
+```py
+def test_all(regtest_all):
+    print("this line will be recorded.")
+    print("and this line also.")
+```
+
+### Reset individual tests
+
+You can reset recorded output of files and functions individually as:
+
+```sh
+$ py.test --regtest-reset test_demo.py
+$ py.test --regtest-reset test_demo.py::test_squares
+```
+
+### Suppress diff for failed tests
+
+To hide the diff and just show the number of lines changed, use:
+
+```sh
+$ py.test --regtest-nodiff ...
+```
+
+
+### Show all recorded output
+
+
+For complex diffs it helps to see the full recorded output also.
+To enable this use:
+
+```sh
+$ py.test --regtest-tee...
+```
+
+
+### Line endings
+
+Per default `pytest-regtest` ignores different line endings in the output.
+In case you want to disable this feature, use the `-regtest-consider-line-endings`
+flag.
+
+
+## Clean indeterministic output before recording
+
+Output can contain data which is changing from test run to test
+run, e.g. paths created with the `tmpdir` fixture, hexadecimal object ids or
+timestamps.
+
+Per default the plugin helps to make output more deterministic by:
+
+- replacing all temporary folder in the output with `<tmpdir...>` or similar markers,
+  depending on the origin of the temporary folder (`tempfile` module, `tmpdir` fixture,
+   ...)
+- replacing hexadecimal numbers ` 0x...` of arbitary length by the fixed string `0x?????????`.
+
+You can also implement your own cleanup routines as described below.
+
+### Register own cleanup functions
+
+You can register own converters in `conftest.py`:
+
+```py
+import re
+import pytest_regtest
+
+@pytest_regtest.register_converter_pre
+def remove_password_lines(txt):
+    '''modify recorded output BEFORE the default fixes
+    like temp folders or hex object ids are applied'''
+
+    # remove lines with passwords:
+    lines = txt.splitlines(keepends=True)
+    lines = [l for l in lines if "password is" not in l]
+    return "".join(lines)
+
+@pytest_regtest.register_converter_post
+def fix_time_measurements(txt):
+    '''modify recorded output AFTER the default fixes
+    like temp folders or hex object ids are applied'''
+
+    # fix time measurements:
+    return re.sub(
+        "\d+(\.\d+)? seconds",
+        "<SECONDS> seconds",
+        txt
+    )
+```
+
+If you register multiple converters they will be applied in the order of
+registration.
+
+In case your routines replace, improve or conflict with the standard cleanup converters,
+you can use the flag `--regtest-disable-stdconv` to disable the default cleanup
+procedure.
+
+## Command line options summary
+
+These are all supported command line options:
+
+```
+$ pytest --help
+...
+
+regression test plugin:
+  --regtest-reset       do not run regtest but record current output
+  --regtest-tee         print recorded results to console too
+  --regtest-consider-line-endings
+                        do not strip whitespaces at end of recorded lines
+  --regtest-nodiff      do not show diff output for failed regresson tests
+  --regtest-disable-stdconv
+                        do not apply standard output converters to clean up
+                        indeterministic output
+...
+```

pytest_regtest-2.2.0a1.dist-info/RECORD

@@ -0,0 +1,12 @@
+pytest_regtest/__init__.py,sha256=Y9D7u9DiyRbGMGW2X1wsl583tp_BAHTBUKJnKps8kMo,1833
+pytest_regtest/numpy_handler.py,sha256=75xwG9UiFVGOp6Px0JXzr7pKxiI6cyezX3nx76h39Jo,6327
+pytest_regtest/pandas_handler.py,sha256=wlCVYLQU_CJ0x6r9Nok3ToF4CXm8eydBkoil-Nu2rEk,4249
+pytest_regtest/pytest_regtest.py,sha256=LmKSMVpzV7E_gh3Sm3asUqNrpmlCOsUVZ_9GoQTOgag,19105
+pytest_regtest/register_third_party_handlers.py,sha256=OU0BqDBBxVPV5OidbWCd_PyI-7P3YSPhuxZhQz_6wkI,926
+pytest_regtest/snapshot_handler.py,sha256=PBqpCuqIdUJrtLZ4xrLI5TstukCDxSBmQqtKxbgB204,2066
+pytest_regtest/utils.py,sha256=2jYTlV_qL5hH6FCeg7T1HJJvKual-Kux2scJ9_aB1lY,811
+pytest_regtest-2.2.0a1.dist-info/METADATA,sha256=0VQKTaWIgTMIePGqbIx-mjaVWfLrfJvvgpN_N56dM6w,10599
+pytest_regtest-2.2.0a1.dist-info/WHEEL,sha256=fl6v0VwpzfGBVsGtkAkhILUlJxROXbA3HvRL6Fe3140,105
+pytest_regtest-2.2.0a1.dist-info/entry_points.txt,sha256=4VuIhXeMGhDo0ATbaUfyjND0atofmZjV_P-o6_uEk2s,36
+pytest_regtest-2.2.0a1.dist-info/licenses/LICENSE.txt,sha256=Tue36uAzpW79-9WAqzkwPhsDDVd1X-VWUmdZ0MfGYvk,1068
+pytest_regtest-2.2.0a1.dist-info/RECORD,,

pytest_regtest-2.2.0a1.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.25.0
+Root-Is-Purelib: true
+Tag: py2-none-any
+Tag: py3-none-any

pytest_regtest-2.2.0a1.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[pytest11]
+regtest = pytest_regtest

pytest_regtest-2.2.0a1.dist-info/licenses/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2016 Uwe Schmitt
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.