robotframework-snapshot 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- robotframework_snapshot-0.1.0/.gitignore +13 -0
- robotframework_snapshot-0.1.0/CHANGELOG.md +19 -0
- robotframework_snapshot-0.1.0/CONTRIBUTING.md +132 -0
- robotframework_snapshot-0.1.0/LICENSE +21 -0
- robotframework_snapshot-0.1.0/PKG-INFO +432 -0
- robotframework_snapshot-0.1.0/README.md +407 -0
- robotframework_snapshot-0.1.0/atest/resources/OutputReader.py +53 -0
- robotframework_snapshot-0.1.0/atest/resources/atest_resource.robot +128 -0
- robotframework_snapshot-0.1.0/atest/robot/lifecycle.robot +120 -0
- robotframework_snapshot-0.1.0/atest/robot/parallel.robot +42 -0
- robotframework_snapshot-0.1.0/atest/robot/unused.robot +87 -0
- robotframework_snapshot-0.1.0/atest/robot/values.robot +100 -0
- robotframework_snapshot-0.1.0/atest/testdata/basics.robot +33 -0
- robotframework_snapshot-0.1.0/atest/testdata/files.robot +17 -0
- robotframework_snapshot-0.1.0/atest/testdata/no_warn.robot +6 -0
- robotframework_snapshot-0.1.0/atest/testdata/normalizers.robot +22 -0
- robotframework_snapshot-0.1.0/atest/testdata/save_actual.robot +6 -0
- robotframework_snapshot-0.1.0/atest/testdata/shared.robot +15 -0
- robotframework_snapshot-0.1.0/atest/testdata/suite_setup.robot +7 -0
- robotframework_snapshot-0.1.0/atest/testdata/xml.robot +22 -0
- robotframework_snapshot-0.1.0/pyproject.toml +44 -0
- robotframework_snapshot-0.1.0/src/SnapshotLibrary/__init__.py +6 -0
- robotframework_snapshot-0.1.0/src/SnapshotLibrary/__main__.py +77 -0
- robotframework_snapshot-0.1.0/src/SnapshotLibrary/core.py +149 -0
- robotframework_snapshot-0.1.0/src/SnapshotLibrary/jsonpath.py +96 -0
- robotframework_snapshot-0.1.0/src/SnapshotLibrary/library.py +519 -0
- robotframework_snapshot-0.1.0/src/SnapshotLibrary/normalizers.py +92 -0
- robotframework_snapshot-0.1.0/src/SnapshotLibrary/py.typed +0 -0
- robotframework_snapshot-0.1.0/src/SnapshotLibrary/serializers.py +148 -0
- robotframework_snapshot-0.1.0/src/SnapshotLibrary/store.py +60 -0
- robotframework_snapshot-0.1.0/src/SnapshotLibrary/unused.py +191 -0
- robotframework_snapshot-0.1.0/src/SnapshotLibrary/version.py +1 -0
- robotframework_snapshot-0.1.0/src/SnapshotLibrary/xmlpath.py +74 -0
- robotframework_snapshot-0.1.0/utest/test_core.py +151 -0
- robotframework_snapshot-0.1.0/utest/test_jsonpath.py +54 -0
- robotframework_snapshot-0.1.0/utest/test_normalizers.py +70 -0
- robotframework_snapshot-0.1.0/utest/test_serializers.py +120 -0
- robotframework_snapshot-0.1.0/utest/test_unused.py +99 -0
- robotframework_snapshot-0.1.0/utest/test_xmlpath.py +43 -0
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## 0.1.0 (unreleased)
|
|
4
|
+
|
|
5
|
+
- `Should Match Snapshot` for text and structured data, with default, update
|
|
6
|
+
(`REFERENCE_RUN`) and strict (`SNAPSHOT_STRICT`) modes.
|
|
7
|
+
- `Should Match File Snapshot`, `Add Snapshot Normalizer`, `Set Snapshot Directory`, `Get Snapshot`.
|
|
8
|
+
- Built-in normalizers: `timestamp`, `timezone`, `uuid`, `duration`, `path`.
|
|
9
|
+
- `ignore=` for masking values by JSONPath in JSON and by XPath in XML.
|
|
10
|
+
- `shared=True` to let several tests compare against one named snapshot.
|
|
11
|
+
- XML stored with sorted attributes, without comments and indented, with `format=xml`;
|
|
12
|
+
XML elements are detected automatically. `Should Match File Snapshot` takes `format=json|xml`.
|
|
13
|
+
- Lists of rows are stored one row per line, so a changed row is one changed line.
|
|
14
|
+
- Warning at the end of a suite for snapshot files no test used (`warn_unused`), and
|
|
15
|
+
`python -m SnapshotLibrary unused <output dir> [--delete]` for parallel runs and CI.
|
|
16
|
+
- Unified diff in the failure message, shortened at the end to fit `--maxerrorlines`. The log has
|
|
17
|
+
the full diff in colour, in a block that can be folded.
|
|
18
|
+
- With `save_actual` (or `SNAPSHOT_SAVE_ACTUAL`) the actual value is also saved under
|
|
19
|
+
`${OUTPUT_DIR}/snapshot_actual/`.
|
|
@@ -0,0 +1,132 @@
|
|
|
1
|
+
# Contributing
|
|
2
|
+
|
|
3
|
+
This document describes how to get the project up and running, how to run the tests and how a release is made.
|
|
4
|
+
|
|
5
|
+
## Getting Started
|
|
6
|
+
|
|
7
|
+
Clone the repository and create a virtual environment in the root of the project:
|
|
8
|
+
```sh
|
|
9
|
+
git clone https://github.com/timdegroot1996/robotframework-snapshot.git
|
|
10
|
+
cd robotframework-snapshot
|
|
11
|
+
python -m venv .venv
|
|
12
|
+
```
|
|
13
|
+
Activate it, on Windows:
|
|
14
|
+
```
|
|
15
|
+
.venv\Scripts\activate
|
|
16
|
+
```
|
|
17
|
+
or on Linux and macOS:
|
|
18
|
+
```sh
|
|
19
|
+
source .venv/bin/activate
|
|
20
|
+
```
|
|
21
|
+
Then install the library in editable mode, together with everything the tests need (pytest and pabot):
|
|
22
|
+
```
|
|
23
|
+
scripts\install.bat
|
|
24
|
+
```
|
|
25
|
+
or
|
|
26
|
+
```sh
|
|
27
|
+
bash scripts/install.sh
|
|
28
|
+
```
|
|
29
|
+
Editable mode means changes in `src/SnapshotLibrary/` are picked up right away, there is no need to reinstall after
|
|
30
|
+
every change.
|
|
31
|
+
|
|
32
|
+
> **Note:** keep the virtual environment activated when running the scripts below. pabot starts `robot` from your
|
|
33
|
+
> `PATH`, and without the activated environment it picks up another Python installation that does not have the
|
|
34
|
+
> library installed.
|
|
35
|
+
|
|
36
|
+
## Project Layout
|
|
37
|
+
|
|
38
|
+
| Path | What |
|
|
39
|
+
| --- | --- |
|
|
40
|
+
| `src/SnapshotLibrary/library.py` | The keywords, and the listener that tracks tests and suites |
|
|
41
|
+
| `src/SnapshotLibrary/core.py` | The compare, record and update decision, without any Robot Framework import |
|
|
42
|
+
| `src/SnapshotLibrary/serializers.py` | Turning values into text: sorted JSON, rows on one line, sorted XML |
|
|
43
|
+
| `src/SnapshotLibrary/normalizers.py` | Built-in and custom normalizers |
|
|
44
|
+
| `src/SnapshotLibrary/jsonpath.py`, `xmlpath.py` | `ignore=` for JSON and XML |
|
|
45
|
+
| `src/SnapshotLibrary/store.py` | Snapshot file names and reading and writing files |
|
|
46
|
+
| `src/SnapshotLibrary/unused.py`, `__main__.py` | Unused snapshot detection and `python -m SnapshotLibrary unused` |
|
|
47
|
+
| `utest/` | Python unit tests |
|
|
48
|
+
| `atest/` | Robot Framework acceptance tests |
|
|
49
|
+
| `scripts/` | Install, test and release scripts |
|
|
50
|
+
|
|
51
|
+
## Tests
|
|
52
|
+
|
|
53
|
+
There are two levels of tests in this project.
|
|
54
|
+
|
|
55
|
+
### Python Unit Tests
|
|
56
|
+
Located in `utest/` and run with pytest. They test the parts that do not need a Robot Framework run.
|
|
57
|
+
```
|
|
58
|
+
scripts\python-tests.bat
|
|
59
|
+
```
|
|
60
|
+
or
|
|
61
|
+
```sh
|
|
62
|
+
bash scripts/python-tests.sh
|
|
63
|
+
```
|
|
64
|
+
Extra arguments are passed on to pytest, for example `scripts\python-tests.bat -k xml`.
|
|
65
|
+
|
|
66
|
+
### Robot Framework Acceptance Tests
|
|
67
|
+
Located in `atest/` and set up the same way as the acceptance tests of Robot Framework itself:
|
|
68
|
+
|
|
69
|
+
| Path | What |
|
|
70
|
+
| --- | --- |
|
|
71
|
+
| `atest/testdata/` | Suites that use SnapshotLibrary, the way a user would |
|
|
72
|
+
| `atest/robot/` | The actual tests. Each one runs suites from `testdata` with `robot` or `pabot` and checks the result |
|
|
73
|
+
| `atest/resources/atest_resource.robot` | Keywords to run the test data and check the outcome: `Run Tests`, `Snapshot Should Be`, `Warnings Should Be`, ... |
|
|
74
|
+
| `atest/resources/OutputReader.py` | Reads the statuses, messages and warnings from the `output.xml` of such a run |
|
|
75
|
+
|
|
76
|
+
A typical test runs the test data twice and looks at what happened in between:
|
|
77
|
+
```robotframework
|
|
78
|
+
Reference Run Updates
|
|
79
|
+
Run Tests basics.robot
|
|
80
|
+
Run Tests basics.robot TEXT=new text REFERENCE_RUN=True
|
|
81
|
+
Run Should Have Passed
|
|
82
|
+
Snapshot Should Be basics/Text_Snapshot.txt new text\n
|
|
83
|
+
```
|
|
84
|
+
Arguments in the form `NAME=value` become `--variable NAME:value` for the inner run, other arguments are passed to
|
|
85
|
+
`robot` as they are. Every test works on its own copy of `atest/testdata` in `results/workspaces/<suite>-<test>/`,
|
|
86
|
+
so snapshots recorded by one test never affect another. The workspaces are kept after the run, so you can look at
|
|
87
|
+
the snapshot files and the inner `output.xml` of a failed test.
|
|
88
|
+
|
|
89
|
+
Run all acceptance tests in parallel with pabot:
|
|
90
|
+
```
|
|
91
|
+
scripts\robot-tests.bat
|
|
92
|
+
```
|
|
93
|
+
or
|
|
94
|
+
```sh
|
|
95
|
+
bash scripts/robot-tests.sh
|
|
96
|
+
```
|
|
97
|
+
The number of processes defaults to 4 and can be changed with the `ROBOT_PROCESSES` environment variable. Extra
|
|
98
|
+
arguments are passed on to pabot, for example `scripts\robot-tests.bat --test "Reference*"`. The results end up in
|
|
99
|
+
`results/` (`log.html`, `report.html`, `output.xml`).
|
|
100
|
+
|
|
101
|
+
A single suite also runs fine with plain Robot Framework:
|
|
102
|
+
```sh
|
|
103
|
+
robot --outputdir results atest/robot/lifecycle.robot
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
### In GitHub Actions
|
|
107
|
+
Both test levels run on every push to `main` and on every pull request, see `.github/workflows/tests.yml`. The
|
|
108
|
+
matrix covers Python 3.9 to 3.13, Robot Framework 6.1, 7.0 and the latest version, on Linux, and the latest versions
|
|
109
|
+
on Windows and macOS. When the acceptance tests fail, their `log.html` and `output.xml` are uploaded as an artifact of
|
|
110
|
+
the workflow run.
|
|
111
|
+
|
|
112
|
+
## Keyword Documentation
|
|
113
|
+
|
|
114
|
+
The keyword documentation is generated from the docstrings in `library.py` and published to
|
|
115
|
+
[GitHub Pages](https://timdegroot1996.github.io/robotframework-snapshot/) on every push to `main`, see
|
|
116
|
+
`.github/workflows/docs.yml`. To look at it before pushing:
|
|
117
|
+
```sh
|
|
118
|
+
python -m robot.libdoc SnapshotLibrary results/SnapshotLibrary.html
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
## Releasing
|
|
122
|
+
|
|
123
|
+
Releases are made by hand, so no PyPI credentials are stored anywhere.
|
|
124
|
+
|
|
125
|
+
1. Update the version in `src/SnapshotLibrary/version.py` and move the changes in `CHANGELOG.md` under that version.
|
|
126
|
+
2. Commit, tag the commit (`git tag v0.1.0`) and push both.
|
|
127
|
+
3. Build and upload to PyPI:
|
|
128
|
+
```
|
|
129
|
+
scripts\release.bat
|
|
130
|
+
```
|
|
131
|
+
twine asks for your PyPI API token.
|
|
132
|
+
4. Create a release on GitHub for the tag, with the changelog entries as description.
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Tim de Groot
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,432 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: robotframework-snapshot
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Snapshot testing for text and structured data in Robot Framework
|
|
5
|
+
Project-URL: Homepage, https://github.com/timdegroot1996/robotframework-snapshot
|
|
6
|
+
Project-URL: Issues, https://github.com/timdegroot1996/robotframework-snapshot/issues
|
|
7
|
+
Author: Tim de Groot
|
|
8
|
+
License-Expression: MIT
|
|
9
|
+
License-File: LICENSE
|
|
10
|
+
Keywords: approval testing,golden file,robotframework,snapshot,testing
|
|
11
|
+
Classifier: Development Status :: 3 - Alpha
|
|
12
|
+
Classifier: Framework :: Robot Framework
|
|
13
|
+
Classifier: Framework :: Robot Framework :: Library
|
|
14
|
+
Classifier: Intended Audience :: Developers
|
|
15
|
+
Classifier: Intended Audience :: Information Technology
|
|
16
|
+
Classifier: Operating System :: OS Independent
|
|
17
|
+
Classifier: Programming Language :: Python :: 3
|
|
18
|
+
Classifier: Topic :: Software Development :: Testing
|
|
19
|
+
Requires-Python: >=3.9
|
|
20
|
+
Requires-Dist: robotframework>=6.1
|
|
21
|
+
Provides-Extra: dev
|
|
22
|
+
Requires-Dist: pytest>=7; extra == 'dev'
|
|
23
|
+
Requires-Dist: robotframework-pabot>=4; extra == 'dev'
|
|
24
|
+
Description-Content-Type: text/markdown
|
|
25
|
+
|
|
26
|
+
# Robot Framework Snapshot
|
|
27
|
+
[](https://pypi.org/project/robotframework-snapshot)
|
|
28
|
+
[](LICENSE)
|
|
29
|
+
|
|
30
|
+
Looking for the keywords? Here is the [Keyword Documentation](https://timdegroot1996.github.io/robotframework-snapshot/).
|
|
31
|
+
|
|
32
|
+
Robot Framework Snapshot is a library for [Robot Framework](https://robotframework.org) that checks text and data
|
|
33
|
+
against a stored expected value, without you writing that expected value by hand. The first time a test runs, the
|
|
34
|
+
library saves the value to a file next to your suite. Every run after that compares against the file and fails with a
|
|
35
|
+
diff when something changed. It works well for command line output, API responses, database rows, XML and generated
|
|
36
|
+
files: anything where the expected value is long, tedious to type out and changes now and then on purpose.
|
|
37
|
+
|
|
38
|
+
## Contents
|
|
39
|
+
|
|
40
|
+
- [Installation](#installation)
|
|
41
|
+
- [Getting Started](#getting-started)
|
|
42
|
+
- [Modes](#modes)
|
|
43
|
+
- [What You Can Snapshot](#what-you-can-snapshot)
|
|
44
|
+
- [Should Match Snapshot or Should Match File Snapshot?](#should-match-snapshot-or-should-match-file-snapshot)
|
|
45
|
+
- [How Snapshot Files Are Found](#how-snapshot-files-are-found)
|
|
46
|
+
- [Values That Change Every Run: Normalizers and Ignore](#values-that-change-every-run-normalizers-and-ignore)
|
|
47
|
+
- [Normalizers: replace changing text with a placeholder](#normalizers-replace-changing-text-with-a-placeholder)
|
|
48
|
+
- [Ignore: mask fields in JSON and XML](#ignore-mask-fields-in-json-and-xml)
|
|
49
|
+
- [Unused Snapshots](#unused-snapshots)
|
|
50
|
+
- [Parallel Runs and CI](#parallel-runs-and-ci)
|
|
51
|
+
- [Keywords](#keywords)
|
|
52
|
+
- [Images, PDFs and Screenshots](#images-pdfs-and-screenshots)
|
|
53
|
+
- [Contributions](#contributions)
|
|
54
|
+
- [License](#license)
|
|
55
|
+
|
|
56
|
+
## Installation
|
|
57
|
+
|
|
58
|
+
Install Robot Framework 6.1 or higher (if not already installed):
|
|
59
|
+
```bash
|
|
60
|
+
pip install robotframework
|
|
61
|
+
```
|
|
62
|
+
Install Robot Framework Snapshot:
|
|
63
|
+
```bash
|
|
64
|
+
pip install robotframework-snapshot
|
|
65
|
+
```
|
|
66
|
+
Python 3.9 or higher is required.
|
|
67
|
+
|
|
68
|
+
## Getting Started
|
|
69
|
+
|
|
70
|
+
```robotframework
|
|
71
|
+
*** Settings ***
|
|
72
|
+
Library Process
|
|
73
|
+
Library SnapshotLibrary
|
|
74
|
+
|
|
75
|
+
*** Test Cases ***
|
|
76
|
+
Help Text Is Stable
|
|
77
|
+
${result}= Run Process mytool --help
|
|
78
|
+
Should Match Snapshot ${result.stdout}
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
**First run:** there is no snapshot yet, so the library writes the output of `mytool --help` to
|
|
82
|
+
`__snapshots__/<suite file name>/Help_Text_Is_Stable.txt` and the test passes with a warning:
|
|
83
|
+
```
|
|
84
|
+
[ WARN ] Snapshot 'tests/__snapshots__/cli/Help_Text_Is_Stable.txt' did not exist and was recorded. Review and commit it.
|
|
85
|
+
```
|
|
86
|
+
Open the file, check that the content is what you expect, and commit it together with your tests.
|
|
87
|
+
|
|
88
|
+
**Every run after that:** the output is compared with the file. When they differ the test fails, and the failure
|
|
89
|
+
message shows exactly what changed, in the console, `log.html` and `report.html`:
|
|
90
|
+
```
|
|
91
|
+
Snapshot 'tests/__snapshots__/cli/Help_Text_Is_Stable.txt' does not match.
|
|
92
|
+
--- snapshot
|
|
93
|
+
+++ actual
|
|
94
|
+
@@ -3,3 +3,3 @@
|
|
95
|
+
Options:
|
|
96
|
+
- -o, --output FILE write results to FILE
|
|
97
|
+
+ -o, --output PATH write results to PATH
|
|
98
|
+
-h, --help show this help
|
|
99
|
+
|
|
100
|
+
If the change is intended, update the snapshot with: --variable REFERENCE_RUN:True
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
**When the change is intended:** run once with `--variable REFERENCE_RUN:True`. Every snapshot that differs is
|
|
104
|
+
overwritten with the new value and the tests pass. Review the changed files like any other change and commit them.
|
|
105
|
+
|
|
106
|
+
## Modes
|
|
107
|
+
|
|
108
|
+
| Mode | How to switch it on | Snapshot file missing | Value differs from the file |
|
|
109
|
+
| --- | --- | --- | --- |
|
|
110
|
+
| Default | nothing | Recorded, test passes with a warning | Test fails with a diff |
|
|
111
|
+
| Update | `--variable REFERENCE_RUN:True` | Recorded | File is overwritten, test passes |
|
|
112
|
+
| Strict | `--variable SNAPSHOT_STRICT:True` or `Library SnapshotLibrary strict=True` | Test fails | Test fails with a diff |
|
|
113
|
+
|
|
114
|
+
I recommend strict mode in CI. In default mode a snapshot you forgot to commit is simply recorded on the build machine,
|
|
115
|
+
so the test passes without checking anything.
|
|
116
|
+
|
|
117
|
+
The failure message shows as much of the diff as fits within Robot Framework's `--maxerrorlines` (40 lines by
|
|
118
|
+
default), starting at the top, and says how many lines were left out. Run with `--maxerrorlines NONE` to get the whole
|
|
119
|
+
diff in the message. The log also has the full diff in colour, removed lines red and added lines green: folded away
|
|
120
|
+
when the message already shows everything, open when the message had to leave lines out.
|
|
121
|
+
|
|
122
|
+
By default the diff only ends up in the test message and the log. When you'd also like the full actual value as a file,
|
|
123
|
+
for example to open it in a diff tool, switch on `save_actual`. It is then written to
|
|
124
|
+
`${OUTPUT_DIR}/snapshot_actual/<suite file name>/` whenever a snapshot does not match:
|
|
125
|
+
```robotframework
|
|
126
|
+
Library SnapshotLibrary save_actual=True
|
|
127
|
+
```
|
|
128
|
+
or for a single run: `--variable SNAPSHOT_SAVE_ACTUAL:True`.
|
|
129
|
+
|
|
130
|
+
## What You Can Snapshot
|
|
131
|
+
|
|
132
|
+
| Type | Stored as | Example |
|
|
133
|
+
| --- | --- | --- |
|
|
134
|
+
| Text | `.txt` | `Should Match Snapshot ${result.stdout}` |
|
|
135
|
+
| JSON: dictionaries and lists | `.json`, keys sorted, two-space indent | `Should Match Snapshot ${response.json()}` |
|
|
136
|
+
| JSON string | `.json`, keys sorted, two-space indent | `Should Match Snapshot ${response.text} format=json` |
|
|
137
|
+
| Rows: database results, table contents | `.json`, one row per line | `Should Match Snapshot ${rows}` |
|
|
138
|
+
| XML string | `.xml`, attributes sorted, comments removed, two-space indent | `Should Match Snapshot ${body} format=xml` |
|
|
139
|
+
| XML element (from the XML library) | `.xml`, same as above | `Should Match Snapshot ${root}` |
|
|
140
|
+
| File | the file's own extension | `Should Match File Snapshot ${OUTPUT_DIR}/export.csv` |
|
|
141
|
+
|
|
142
|
+
Sorting keys and attributes and fixing the indentation means the snapshot only changes when the content changes, not
|
|
143
|
+
when the system happens to write the same data in another order or layout.
|
|
144
|
+
|
|
145
|
+
For example, this list of rows:
|
|
146
|
+
```robotframework
|
|
147
|
+
${rows}= Evaluate [("2026-03-14", "Checkout", 12, 0), ("2026-03-15", "Checkout", 11, 1)]
|
|
148
|
+
Should Match Snapshot ${rows}
|
|
149
|
+
```
|
|
150
|
+
is stored as:
|
|
151
|
+
```json
|
|
152
|
+
[
|
|
153
|
+
["2026-03-14", "Checkout", 12, 0],
|
|
154
|
+
["2026-03-15", "Checkout", 11, 1]
|
|
155
|
+
]
|
|
156
|
+
```
|
|
157
|
+
so a changed row shows up as one changed line in the diff.
|
|
158
|
+
|
|
159
|
+
The same goes for JSON and XML: because of the fixed layout, a changed value is one changed line. When the stored order
|
|
160
|
+
says `"status": "paid"` and the API now returns `refunded`:
|
|
161
|
+
```
|
|
162
|
+
Snapshot 'tests/__snapshots__/orders/Order_Has_Expected_Body.json' does not match.
|
|
163
|
+
--- snapshot
|
|
164
|
+
+++ actual
|
|
165
|
+
@@ -1,5 +1,5 @@
|
|
166
|
+
{
|
|
167
|
+
"id": 1042,
|
|
168
|
+
- "status": "paid",
|
|
169
|
+
+ "status": "refunded",
|
|
170
|
+
"total": 19.95
|
|
171
|
+
}
|
|
172
|
+
|
|
173
|
+
If the change is intended, update the snapshot with: --variable REFERENCE_RUN:True
|
|
174
|
+
```
|
|
175
|
+
and when an XML total changed from `19.95` to `24.95`:
|
|
176
|
+
```
|
|
177
|
+
Snapshot 'tests/__snapshots__/orders/Order_Xml.xml' does not match.
|
|
178
|
+
--- snapshot
|
|
179
|
+
+++ actual
|
|
180
|
+
@@ -1,3 +1,3 @@
|
|
181
|
+
<order id="1042">
|
|
182
|
+
- <total>19.95</total>
|
|
183
|
+
+ <total>24.95</total>
|
|
184
|
+
</order>
|
|
185
|
+
|
|
186
|
+
If the change is intended, update the snapshot with: --variable REFERENCE_RUN:True
|
|
187
|
+
```
|
|
188
|
+
|
|
189
|
+
### Should Match Snapshot or Should Match File Snapshot?
|
|
190
|
+
|
|
191
|
+
- `Should Match Snapshot` takes a **value**: the content of a variable, the return value of a keyword.
|
|
192
|
+
- `Should Match File Snapshot` takes a **path to a file** the system under test created, reads that file and compares
|
|
193
|
+
its content. The snapshot keeps the file's extension, so `export.csv` is stored as a `.csv` snapshot. Add
|
|
194
|
+
`format=json` or `format=xml` to have the file sorted and indented like the table above.
|
|
195
|
+
|
|
196
|
+
## How Snapshot Files Are Found
|
|
197
|
+
|
|
198
|
+
There is no variable or setting that points to a snapshot file. The location follows from where the keyword is called:
|
|
199
|
+
|
|
200
|
+
```
|
|
201
|
+
<folder of the suite file>/__snapshots__/<suite file name>/<test name>.<extension>
|
|
202
|
+
```
|
|
203
|
+
|
|
204
|
+
The suite file name is used without `.robot`, and characters that are not letters, digits, `.` or `-` in the test
|
|
205
|
+
name become `_`. The extension follows from the type of the value (see the table above).
|
|
206
|
+
|
|
207
|
+
When a test takes more than one snapshot, the first one gets the plain test name, the next ones a number in the order
|
|
208
|
+
they are taken. Give a snapshot a `name=` to get a readable file name that does not depend on the order:
|
|
209
|
+
|
|
210
|
+
```robotframework
|
|
211
|
+
*** Test Cases ***
|
|
212
|
+
Create Order
|
|
213
|
+
Should Match Snapshot ${response.text} # Create_Order.txt
|
|
214
|
+
Should Match Snapshot ${confirmation_email} # Create_Order__2.txt
|
|
215
|
+
Should Match Snapshot ${response.headers} name=headers # Create_Order__headers.json
|
|
216
|
+
```
|
|
217
|
+
|
|
218
|
+
```
|
|
219
|
+
tests/
|
|
220
|
+
├── orders.robot
|
|
221
|
+
└── __snapshots__/
|
|
222
|
+
└── orders/
|
|
223
|
+
├── Create_Order.txt
|
|
224
|
+
├── Create_Order__2.txt
|
|
225
|
+
└── Create_Order__headers.json
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
A snapshot taken in a suite setup or teardown is stored as `__suite__.<extension>`.
|
|
229
|
+
|
|
230
|
+
**Several tests, one snapshot.** When several tests should compare against the same file, for example the short and
|
|
231
|
+
the long form of a command line option, add `shared=True` together with a `name`. The file is then stored under that
|
|
232
|
+
name alone, `__snapshots__/<suite file name>/<name>.<extension>`:
|
|
233
|
+
|
|
234
|
+
```robotframework
|
|
235
|
+
*** Test Cases ***
|
|
236
|
+
Short Help Flag
|
|
237
|
+
${result}= Run Process mytool -h
|
|
238
|
+
Should Match Snapshot ${result.stdout} name=help shared=True # help.txt
|
|
239
|
+
|
|
240
|
+
Long Help Flag
|
|
241
|
+
${result}= Run Process mytool --help
|
|
242
|
+
Should Match Snapshot ${result.stdout} name=help shared=True # the same help.txt
|
|
243
|
+
```
|
|
244
|
+
|
|
245
|
+
**All snapshots in one folder.** Import the library with `snapshot_directory=snapshots` (relative to the directory
|
|
246
|
+
you run `robot` from) or call `Set Snapshot Directory`. The suite file name and test name part stays the same:
|
|
247
|
+
`snapshots/<suite file name>/<test name>.<extension>`.
|
|
248
|
+
|
|
249
|
+
## Values That Change Every Run: Normalizers and Ignore
|
|
250
|
+
|
|
251
|
+
Timestamps, generated IDs and durations differ on every run. There are two ways to keep them out of the comparison:
|
|
252
|
+
normalizers for any value, and `ignore=` for fields in JSON and XML.
|
|
253
|
+
|
|
254
|
+
### Normalizers: replace changing text with a placeholder
|
|
255
|
+
|
|
256
|
+
A normalizer is a regular expression that replaces matching text with a fixed placeholder, before the value is compared
|
|
257
|
+
and before it is recorded. They work on every type of value.
|
|
258
|
+
|
|
259
|
+
```robotframework
|
|
260
|
+
*** Test Cases ***
|
|
261
|
+
Order Confirmation Is Stable
|
|
262
|
+
# ${output} is: Order ORD-1042 created at 2026-03-14 09:26:53 by 3f2a9c1e-5b7d-4e8a-9c21-7d4e5f6a8b90
|
|
263
|
+
${output}= Get Confirmation Message
|
|
264
|
+
Add Snapshot Normalizer order_id pattern=ORD-\\d+ scope=test
|
|
265
|
+
Should Match Snapshot ${output} normalizers=timestamp,uuid
|
|
266
|
+
```
|
|
267
|
+
is stored, and from then on compared, as:
|
|
268
|
+
```
|
|
269
|
+
Order <ORDER_ID> created at <TIMESTAMP> by <UUID>
|
|
270
|
+
```
|
|
271
|
+
so the next run with `ORD-1043`, another time and another UUID still passes.
|
|
272
|
+
|
|
273
|
+
Built-in normalizers:
|
|
274
|
+
|
|
275
|
+
| Name | Replaces | Example | Becomes |
|
|
276
|
+
| --- | --- | --- | --- |
|
|
277
|
+
| `timestamp` | Date and time | `2026-03-14 09:26:53.123+01:00` | `<TIMESTAMP>` |
|
|
278
|
+
| `timezone` | Only the UTC offset after a date and time | `2026-03-14 09:26:53+01:00` | `2026-03-14 09:26:53<TZ>` |
|
|
279
|
+
| `uuid` | UUIDs | `3f2a9c1e-5b7d-4e8a-9c21-7d4e5f6a8b90` | `<UUID>` |
|
|
280
|
+
| `duration` | Durations | `1.2s`, `350 ms`, `3 seconds` | `<DURATION>` |
|
|
281
|
+
| `path` | The working, temp and home directory | `C:\Users\me\project\out` | `<CWD>\out` |
|
|
282
|
+
|
|
283
|
+
You can switch normalizers on for the whole run, a suite, a test or a single call:
|
|
284
|
+
|
|
285
|
+
```robotframework
|
|
286
|
+
*** Settings ***
|
|
287
|
+
Library SnapshotLibrary normalizers=uuid # every snapshot in the run
|
|
288
|
+
|
|
289
|
+
*** Test Cases ***
|
|
290
|
+
Import Job Summary
|
|
291
|
+
Add Snapshot Normalizer timestamp # every snapshot in this suite (scope=suite is the default)
|
|
292
|
+
Add Snapshot Normalizer job_id pattern=JOB-\\d+ scope=test # your own, this test only
|
|
293
|
+
# ${summary} is: JOB-77 by 3f2a9c1e-5b7d-4e8a-9c21-7d4e5f6a8b90 started 2026-03-14 09:26:53, took 1.2s
|
|
294
|
+
${summary}= Get Job Summary
|
|
295
|
+
Should Match Snapshot ${summary} normalizers=duration # this call only
|
|
296
|
+
```
|
|
297
|
+
is stored as:
|
|
298
|
+
```
|
|
299
|
+
<JOB_ID> by <UUID> started <TIMESTAMP>, took <DURATION>
|
|
300
|
+
```
|
|
301
|
+
`Add Snapshot Normalizer` also takes `scope=global`, which keeps the normalizer for the rest of the run.
|
|
302
|
+
|
|
303
|
+
A custom normalizer replaces its matches with `<NAME>` in capitals, or with your own `replacement=`, which can use
|
|
304
|
+
groups from the pattern: `pattern=(localhost):\\d+ replacement=\\1:<PORT>` turns `localhost:8080` into
|
|
305
|
+
`localhost:<PORT>`.
|
|
306
|
+
|
|
307
|
+
### Ignore: mask fields in JSON and XML
|
|
308
|
+
|
|
309
|
+
For JSON and XML you can point at the fields to leave out with `ignore=`. Their value is replaced by `<IGNORED>`:
|
|
310
|
+
|
|
311
|
+
```robotframework
|
|
312
|
+
# ${order} is: {"id": 1042, "status": "paid", "total": 19.95, "updated_at": "2026-03-14T09:26:53"}
|
|
313
|
+
Should Match Snapshot ${order} ignore=$.id;$.updated_at
|
|
314
|
+
```
|
|
315
|
+
```json
|
|
316
|
+
{
|
|
317
|
+
"id": "<IGNORED>",
|
|
318
|
+
"status": "paid",
|
|
319
|
+
"total": 19.95,
|
|
320
|
+
"updated_at": "<IGNORED>"
|
|
321
|
+
}
|
|
322
|
+
```
|
|
323
|
+
|
|
324
|
+
```robotframework
|
|
325
|
+
# ${body} is: <order status="paid" id="1042"><total>19.95</total><created>2026-03-14T09:26:53</created></order>
|
|
326
|
+
Should Match Snapshot ${body} format=xml ignore=.//created;@id
|
|
327
|
+
```
|
|
328
|
+
```xml
|
|
329
|
+
<order id="<IGNORED>" status="paid">
|
|
330
|
+
<total>19.95</total>
|
|
331
|
+
<created><IGNORED></created>
|
|
332
|
+
</order>
|
|
333
|
+
```
|
|
334
|
+
|
|
335
|
+
| Type | Path syntax | Examples |
|
|
336
|
+
| --- | --- | --- |
|
|
337
|
+
| JSON | JSONPath | `$.id`, `$.items[0]`, `$.items[*].price`, `$.*`, `$..updated_at` (any depth) |
|
|
338
|
+
| XML | XPath, as supported by Python's ElementTree, relative to the root element | `.//created`, `items/item`, `.//item[@type='gift']`, `@id`, `.//item/@id` |
|
|
339
|
+
|
|
340
|
+
Separate several paths with `;`. A path ending in `/@name` masks that attribute. Namespace prefixes declared in the
|
|
341
|
+
document can be used in the path, for example `.//soap:Body`. Plain text has no fields, so `ignore=` does not apply to
|
|
342
|
+
it; use a normalizer there.
|
|
343
|
+
|
|
344
|
+
## Unused Snapshots
|
|
345
|
+
|
|
346
|
+
When you rename or remove a test, its snapshot file stays behind. At the end of each suite the library warns about
|
|
347
|
+
files in that suite's snapshot folder that no test used:
|
|
348
|
+
```
|
|
349
|
+
[ WARN ] Unused snapshot: tests/__snapshots__/orders/Old_Test_Name.json. No test used them in this run. Delete them if they are no longer needed.
|
|
350
|
+
```
|
|
351
|
+
|
|
352
|
+
A suite is only checked when all of its tests ran and passed. Otherwise a test you filtered out with `--test`, or one
|
|
353
|
+
that failed before it reached its snapshot, would make its snapshot look unused. Turn the warning off with
|
|
354
|
+
`Library SnapshotLibrary warn_unused=False`.
|
|
355
|
+
|
|
356
|
+
### Parallel Runs and CI
|
|
357
|
+
|
|
358
|
+
With `pabot --testlevelsplit` every test runs in its own process, so no process sees a whole suite and the warning
|
|
359
|
+
never shows up. Check the finished run from the command line instead. This works for plain `robot` runs too:
|
|
360
|
+
|
|
361
|
+
```bash
|
|
362
|
+
python -m SnapshotLibrary unused results/ # list them, exit code 1 if there are any
|
|
363
|
+
python -m SnapshotLibrary unused results/ --delete # delete them
|
|
364
|
+
```
|
|
365
|
+
|
|
366
|
+
The command reads the usage records the library writes to `<output dir>/snapshot_usage/`. It also reports snapshot
|
|
367
|
+
folders whose suite file no longer exists, which is what renaming or deleting a suite file leaves behind.
|
|
368
|
+
|
|
369
|
+
Good to know:
|
|
370
|
+
|
|
371
|
+
- A snapshot that is only taken under a condition (inside an `IF`) is reported as unused in runs where the condition
|
|
372
|
+
is false.
|
|
373
|
+
- Folders of renamed or deleted suites are only found when you use the default `__snapshots__` folders. Those sit next
|
|
374
|
+
to the suite files, so the command can see that `__snapshots__/orders/` has no `orders.robot` beside it. With
|
|
375
|
+
`snapshot_directory` all suites share one folder, possibly from several test directories, so the command cannot tell
|
|
376
|
+
which suite file a folder belonged to. Unused files inside the folders of suites that did run are still found.
|
|
377
|
+
- Snapshots taken in the setup of an `__init__.robot` are only checked when every test below that folder passed.
|
|
378
|
+
|
|
379
|
+
## Keywords
|
|
380
|
+
|
|
381
|
+
| Keyword | What it does |
|
|
382
|
+
| --- | --- |
|
|
383
|
+
| `Should Match Snapshot` | Compares a value (text, dictionary, list, XML) with its snapshot file |
|
|
384
|
+
| `Should Match File Snapshot` | Reads a file from disk and compares its content with its snapshot file |
|
|
385
|
+
| `Add Snapshot Normalizer` | Switches on a built-in normalizer or adds your own, for a test, a suite or the whole run |
|
|
386
|
+
| `Set Snapshot Directory` | Stores snapshots in one folder of your choice instead of `__snapshots__` next to each suite |
|
|
387
|
+
| `Get Snapshot` | Returns the content of a snapshot file, for your own checks |
|
|
388
|
+
|
|
389
|
+
All arguments and more examples are in the [Keyword Documentation](https://timdegroot1996.github.io/robotframework-snapshot/).
|
|
390
|
+
|
|
391
|
+
## Images, PDFs and Screenshots
|
|
392
|
+
|
|
393
|
+
This library compares text and data only, and that will stay so. For images, PDFs, print jobs and screenshots use
|
|
394
|
+
[DocTestLibrary](https://github.com/manykarim/robotframework-doctestlibrary).
|
|
395
|
+
|
|
396
|
+
The two work well side by side. Both read the same `REFERENCE_RUN` variable, so one run updates text snapshots and
|
|
397
|
+
visual baselines together:
|
|
398
|
+
|
|
399
|
+
```robotframework
|
|
400
|
+
*** Settings ***
|
|
401
|
+
Library Browser
|
|
402
|
+
Library SnapshotLibrary
|
|
403
|
+
Library DocTest.WebVisualTest
|
|
404
|
+
|
|
405
|
+
*** Test Cases ***
|
|
406
|
+
Checkout Page
|
|
407
|
+
New Page https://shop.example.com/checkout
|
|
408
|
+
${items}= Evaluate JavaScript ${None} () => window.cart.items
|
|
409
|
+
Should Match Snapshot ${items} # the data is right
|
|
410
|
+
Compare Page To Baseline checkout # the page looks right
|
|
411
|
+
```
|
|
412
|
+
|
|
413
|
+
```bash
|
|
414
|
+
robot --variable REFERENCE_RUN:True tests/
|
|
415
|
+
```
|
|
416
|
+
|
|
417
|
+
`Compare Page To Baseline` is part of `DocTest.WebVisualTest`; check the DocTestLibrary documentation for the version
|
|
418
|
+
that includes it.
|
|
419
|
+
|
|
420
|
+
The way this library records on the first run and updates with `REFERENCE_RUN` follows DocTestLibrary by Many
|
|
421
|
+
Kasiriha, so that people using both get the same experience.
|
|
422
|
+
|
|
423
|
+
## Contributions
|
|
424
|
+
|
|
425
|
+
Contributions are welcome! If you run into an issue, have an idea for an improvement or would like to add something
|
|
426
|
+
yourself, feel free to open an issue or a pull request. How to set up the project and run the tests is described in
|
|
427
|
+
[Contributing](./CONTRIBUTING.md).
|
|
428
|
+
|
|
429
|
+
## License
|
|
430
|
+
This project is licensed under the MIT License.
|
|
431
|
+
|
|
432
|
+
> **Note:** This project is not officially affiliated with or endorsed by Robot Framework.
|