lekman-qualify 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- lekman_qualify-0.1.0/.gitignore +12 -0
- lekman_qualify-0.1.0/PKG-INFO +355 -0
- lekman_qualify-0.1.0/README.md +325 -0
- lekman_qualify-0.1.0/pyproject.toml +105 -0
- lekman_qualify-0.1.0/src/qualify/__init__.py +46 -0
- lekman_qualify-0.1.0/src/qualify/__main__.py +5 -0
- lekman_qualify-0.1.0/src/qualify/api.py +162 -0
- lekman_qualify-0.1.0/src/qualify/cli.py +364 -0
- lekman_qualify-0.1.0/src/qualify/context.py +511 -0
- lekman_qualify-0.1.0/src/qualify/declaration.py +190 -0
- lekman_qualify-0.1.0/src/qualify/envelope.py +173 -0
- lekman_qualify-0.1.0/src/qualify/evidence.py +87 -0
- lekman_qualify-0.1.0/src/qualify/jsontext.py +206 -0
- lekman_qualify-0.1.0/src/qualify/merge.py +200 -0
- lekman_qualify-0.1.0/src/qualify/outcome.py +48 -0
- lekman_qualify-0.1.0/src/qualify/plugin.py +314 -0
- lekman_qualify-0.1.0/src/qualify/py.typed +0 -0
- lekman_qualify-0.1.0/src/qualify/recorder.py +164 -0
- lekman_qualify-0.1.0/src/qualify/redaction.py +138 -0
- lekman_qualify-0.1.0/src/qualify/report/__init__.py +1 -0
- lekman_qualify-0.1.0/src/qualify/report/content.py +420 -0
- lekman_qualify-0.1.0/src/qualify/report/layout.py +544 -0
- lekman_qualify-0.1.0/src/qualify/report/pdf.py +129 -0
- lekman_qualify-0.1.0/src/qualify/runtime.py +125 -0
- lekman_qualify-0.1.0/src/qualify/schema.py +81 -0
- lekman_qualify-0.1.0/src/qualify/schemas/__init__.py +1 -0
- lekman_qualify-0.1.0/src/qualify/schemas/evidence.schema.json +488 -0
- lekman_qualify-0.1.0/src/qualify/schemas/report-theme.schema.json +152 -0
- lekman_qualify-0.1.0/src/qualify/schemas/trace.schema.json +166 -0
- lekman_qualify-0.1.0/src/qualify/settings.py +219 -0
- lekman_qualify-0.1.0/src/qualify/site.py +54 -0
- lekman_qualify-0.1.0/src/qualify/theme.py +163 -0
- lekman_qualify-0.1.0/src/qualify/themes/lekman/fonts/Geist-Regular.ttf +0 -0
- lekman_qualify-0.1.0/src/qualify/themes/lekman/fonts/Geist-SemiBold.ttf +0 -0
- lekman_qualify-0.1.0/src/qualify/themes/lekman/fonts/GeistMono-Regular.ttf +0 -0
- lekman_qualify-0.1.0/src/qualify/themes/lekman/fonts/OFL.txt +93 -0
- lekman_qualify-0.1.0/src/qualify/themes/lekman/logo.png +0 -0
- lekman_qualify-0.1.0/src/qualify/themes/lekman/theme.json +46 -0
- lekman_qualify-0.1.0/src/qualify/themes/plain/fonts/NotoSans-Bold.ttf +0 -0
- lekman_qualify-0.1.0/src/qualify/themes/plain/fonts/NotoSans-Regular.ttf +0 -0
- lekman_qualify-0.1.0/src/qualify/themes/plain/fonts/OFL.txt +93 -0
- lekman_qualify-0.1.0/src/qualify/themes/plain/theme.json +42 -0
- lekman_qualify-0.1.0/src/qualify/trace.py +237 -0
- lekman_qualify-0.1.0/src/qualify/values.py +216 -0
|
@@ -0,0 +1,355 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: lekman-qualify
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Qualification tests (IQ, OQ, PQ, intended use) for pytest that write redacted JSON evidence and a PDF report.
|
|
5
|
+
Project-URL: Homepage, https://github.com/lekman/qa-toolkit/tree/main/packages/qualify-python
|
|
6
|
+
Project-URL: Repository, https://github.com/lekman/qa-toolkit
|
|
7
|
+
Project-URL: Issues, https://github.com/lekman/qa-toolkit/issues
|
|
8
|
+
Project-URL: Changelog, https://github.com/lekman/qa-toolkit/blob/main/packages/qualify-python/CHANGELOG.md
|
|
9
|
+
Project-URL: Specification, https://github.com/lekman/qa-toolkit/blob/main/spec/qualify/README.md
|
|
10
|
+
Author: Tobias Lekman
|
|
11
|
+
License-Expression: MIT
|
|
12
|
+
License-File: src/qualify/themes/lekman/fonts/OFL.txt
|
|
13
|
+
License-File: src/qualify/themes/plain/fonts/OFL.txt
|
|
14
|
+
Keywords: evidence,pytest,qualification,traceability,validation
|
|
15
|
+
Classifier: Development Status :: 3 - Alpha
|
|
16
|
+
Classifier: Framework :: Pytest
|
|
17
|
+
Classifier: Intended Audience :: Developers
|
|
18
|
+
Classifier: Operating System :: OS Independent
|
|
19
|
+
Classifier: Programming Language :: Python :: 3
|
|
20
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
21
|
+
Classifier: Topic :: Software Development :: Quality Assurance
|
|
22
|
+
Classifier: Topic :: Software Development :: Testing
|
|
23
|
+
Classifier: Typing :: Typed
|
|
24
|
+
Requires-Python: >=3.10
|
|
25
|
+
Requires-Dist: fpdf2>=2.8.3
|
|
26
|
+
Requires-Dist: jsonschema>=4.18
|
|
27
|
+
Requires-Dist: pytest>=8.3
|
|
28
|
+
Requires-Dist: pyyaml>=6.0.1
|
|
29
|
+
Description-Content-Type: text/markdown
|
|
30
|
+
|
|
31
|
+
# lekman-qualify
|
|
32
|
+
|
|
33
|
+
> Qualification tests (IQ, OQ, PQ, intended use) for pytest that leave redacted
|
|
34
|
+
> JSON evidence and a PDF report a reviewer can read without the code.
|
|
35
|
+
|
|
36
|
+
This is the Python implementation of the
|
|
37
|
+
[qualify specification](../../spec/qualify/README.md), as a pytest plugin. It
|
|
38
|
+
writes the same evidence files as the TypeScript reference
|
|
39
|
+
[@lekman/qualify](../qualify-ts/README.md) and passes the same conformance
|
|
40
|
+
scenarios. It is not published yet.
|
|
41
|
+
|
|
42
|
+
## Why
|
|
43
|
+
|
|
44
|
+
A passing test run says that the tests passed. It does not say what each test
|
|
45
|
+
claimed, what it checked, which requirement it covers, or where it ran, and it
|
|
46
|
+
cannot be read six months later without the code. A qualification test here is
|
|
47
|
+
an ordinary pytest test with an ID, a narrative (given, when, then) and
|
|
48
|
+
recorded checks. When it runs, the process that ran it writes one JSON record,
|
|
49
|
+
with secrets already removed. `qualify merge` builds one run file from the
|
|
50
|
+
records, and `qualify report` renders it as a PDF to read, sign and file.
|
|
51
|
+
|
|
52
|
+
pytest still runs the tests. There is no wrapper command: select tests with
|
|
53
|
+
pytest's own `-k` and `-m`.
|
|
54
|
+
|
|
55
|
+
## Install
|
|
56
|
+
|
|
57
|
+
```bash
|
|
58
|
+
pip install lekman-qualify
|
|
59
|
+
uv add --dev lekman-qualify
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
Python 3.10 or later and pytest 8.3 or later. The plugin registers itself
|
|
63
|
+
through the `pytest11` entry point; there is nothing to add to `conftest.py`.
|
|
64
|
+
|
|
65
|
+
## A Test
|
|
66
|
+
|
|
67
|
+
```python
|
|
68
|
+
import qualify
|
|
69
|
+
|
|
70
|
+
|
|
71
|
+
@qualify.oq(
|
|
72
|
+
"OQ-EXPORT-003",
|
|
73
|
+
"The bank file never changes an amount",
|
|
74
|
+
given="a day of payments in three currencies",
|
|
75
|
+
when="the export writes the bank file",
|
|
76
|
+
then="every amount in the file equals the amount in the ledger",
|
|
77
|
+
requirements=["EXPORT-003"],
|
|
78
|
+
risk="high",
|
|
79
|
+
scope="read",
|
|
80
|
+
tags=["export"],
|
|
81
|
+
)
|
|
82
|
+
def test_bank_file_keeps_amounts(ctx: qualify.Context, ledger) -> None:
|
|
83
|
+
file = export_bank_file(ledger) # ledger is an ordinary fixture
|
|
84
|
+
ctx.check("bank file", "EUR keeps two decimals").that(file.amount("EUR")).equals("100.50")
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
`qualify.iq`, `qualify.oq`, `qualify.pq` and `qualify.iu` declare the level.
|
|
88
|
+
`scope` is required for OQ and PQ (`read` or `read-write`); IQ is always
|
|
89
|
+
`read` and IU always `read-write`. Further clauses after `then` go in `and_`,
|
|
90
|
+
because `and` is a Python keyword. The function must still be named `test_...`
|
|
91
|
+
for pytest to collect it; the plugin warns when a declared function is not.
|
|
92
|
+
|
|
93
|
+
```bash
|
|
94
|
+
pytest -k OQ-EXPORT-003 # one test, by ID
|
|
95
|
+
pytest -m oq # every OQ test
|
|
96
|
+
pytest -m "oq and not slow" # by level and tag
|
|
97
|
+
pytest --qualify-merge # also build the run file at the end
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
A declaration error, such as an OQ test with no `scope` or an ID that does
|
|
101
|
+
not match its level, fails collection with one error per test and writes no
|
|
102
|
+
evidence. Parametrised qualification tests are refused in the same way,
|
|
103
|
+
because one ID must map to one record; declare each case with its own ID.
|
|
104
|
+
|
|
105
|
+
## The Context
|
|
106
|
+
|
|
107
|
+
| Call | Does |
|
|
108
|
+
| --------------------------------------------------- | -------------------------------------------------------------------- |
|
|
109
|
+
| `ctx.check(context, description).that(v).equals(x)` | A hard check: records, then raises `CheckFailed` if it fails |
|
|
110
|
+
| `ctx.check(...).soft().that(v)...` | A soft check: records and continues; the test fails at the end |
|
|
111
|
+
| `ctx.delegate(context, description, fn)` | Runs `fn`, which has its own assertions, and records it as one check |
|
|
112
|
+
| `ctx.attach(filename, body, content_type)` | An inline attachment; `application/json` also takes a value |
|
|
113
|
+
| `ctx.secret(name)` | Reads `QUALIFY_SECRET_<NAME>`, registers it, returns a `Secret` |
|
|
114
|
+
| `ctx.redact(value)` | Registers a value for redaction |
|
|
115
|
+
| `ctx.skip(reason)` | Skips the test at runtime (`pytest.skip`), with the reason recorded |
|
|
116
|
+
| `ctx.run_context(extra)` | Attaches `run-context.json` with the run's source, actor and target |
|
|
117
|
+
| `ctx.id`, `ctx.level`, `ctx.environment` | Identity, and where the test runs |
|
|
118
|
+
|
|
119
|
+
Matchers: `equals`, `not_equals`, `contains`, `matches`, `is_true`,
|
|
120
|
+
`is_false`, `is_null`, `is_not_null`, `has_length`, `at_least`, `at_most`,
|
|
121
|
+
`one_of`. The evidence records the shared camel-case names (`notEquals`,
|
|
122
|
+
`isTrue`). Values are compared after the normalisation in section 7.1 of the
|
|
123
|
+
specification, so `NaN` equals `NaN`, `1` equals `1.0`, and key order does not
|
|
124
|
+
matter.
|
|
125
|
+
|
|
126
|
+
`qualify.Secret` is a `str` subclass whose `repr()` hides the value and which
|
|
127
|
+
registers the value for redaction when it is created. Use it for any secret
|
|
128
|
+
your own fixtures read:
|
|
129
|
+
|
|
130
|
+
```python
|
|
131
|
+
@pytest.fixture
|
|
132
|
+
def api_token() -> qualify.Secret:
|
|
133
|
+
return qualify.Secret(vault.read("api-token"))
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
### Outcomes
|
|
137
|
+
|
|
138
|
+
The outcome comes from a `pytest_runtest_makereport` hook wrapper that reads
|
|
139
|
+
the exception of the phase that ended the test.
|
|
140
|
+
|
|
141
|
+
| The test ends with | Outcome |
|
|
142
|
+
| ----------------------------------------------------------------- | ------------------------------------- |
|
|
143
|
+
| Nothing, and every check passed | `pass` |
|
|
144
|
+
| A failed check, an `AssertionError`, or `pytest.fail()` | `fail` |
|
|
145
|
+
| Any other exception, also from a fixture during setup | `error`, with the type and message |
|
|
146
|
+
| `ctx.skip()`, `pytest.skip()`, a skip marker, or `pytest.xfail()` | `skip`, with the reason |
|
|
147
|
+
| A write-protection or "not applicable" rule | `skip`, before any fixture runs |
|
|
148
|
+
| The process dies (a crash, a killed worker) | `error`, `DidNotFinish`, from `merge` |
|
|
149
|
+
|
|
150
|
+
A hard check that fails raises `CheckFailed`. If the body catches it, the test
|
|
151
|
+
still fails at the end, because a record says `pass` only when every check
|
|
152
|
+
passed. An `xfail` marker does not change the record: a test that fails is
|
|
153
|
+
recorded as failing, whatever pytest reports. A teardown error after the body
|
|
154
|
+
does not change the record either.
|
|
155
|
+
|
|
156
|
+
## Settings
|
|
157
|
+
|
|
158
|
+
| Variable | pytest ini option | Default | Meaning |
|
|
159
|
+
| -------------------------- | -------------------------- | ------------------------ | ------------------------------------------------------------ |
|
|
160
|
+
| `QUALIFY_ENVIRONMENT` | `qualify_environment` | `local` | The environment name |
|
|
161
|
+
| `QUALIFY_PRODUCTION` | `qualify_production` | `production,prod` | Environments where `read-write` tests are skipped |
|
|
162
|
+
| `QUALIFY_ALLOW_WRITE` | `qualify_allow_write` | none | `ID=reason;ID=reason`: tests allowed to write in production |
|
|
163
|
+
| `QUALIFY_NO_OVERRIDES` | `qualify_no_overrides` | off | `1` refuses every override and stops the run at start |
|
|
164
|
+
| `QUALIFY_EVIDENCE_DIR` | `qualify_evidence_dir` | `qualification/evidence` | Where records and the run file go |
|
|
165
|
+
| `QUALIFY_ATTACHMENT_LIMIT` | `qualify_attachment_limit` | `1048576` | Bytes per attachment; a larger one ends the test as an error |
|
|
166
|
+
| `QUALIFY_RUN_ID` | | created per run | Share one run across jobs, machines or languages |
|
|
167
|
+
| `QUALIFY_TARGET_<NAME>` | | none | Facts about the target, recorded as `run.target.<name>` |
|
|
168
|
+
|
|
169
|
+
The environment variable wins over the ini option. `QUALIFY_EVIDENCE_DIR` is
|
|
170
|
+
relative to the working directory; `qualify_evidence_dir` is relative to
|
|
171
|
+
pytest's rootdir. A setting that cannot be used stops the run before any test
|
|
172
|
+
starts, with a usage error.
|
|
173
|
+
|
|
174
|
+
Code can give the same settings, and three that only code can give, in
|
|
175
|
+
`conftest.py`. The variables and ini options still win.
|
|
176
|
+
|
|
177
|
+
```python
|
|
178
|
+
# conftest.py
|
|
179
|
+
import qualify
|
|
180
|
+
|
|
181
|
+
qualify.configure(
|
|
182
|
+
resolve_environment=lambda: deploy_target(), # if it raises: production
|
|
183
|
+
secret_source=lambda name: vault.read(name), # behind ctx.secret
|
|
184
|
+
not_applicable=[qualify.NotApplicable("no GPU in CI", tags=["gpu"], environments=["ci"])],
|
|
185
|
+
)
|
|
186
|
+
```
|
|
187
|
+
|
|
188
|
+
## pytest-xdist
|
|
189
|
+
|
|
190
|
+
Each worker writes its own records. The controller creates one run ID (or
|
|
191
|
+
takes `QUALIFY_RUN_ID`) and hands it to every worker, so all records of one
|
|
192
|
+
`pytest -n 4` run share it. With `--qualify-merge`, the controller merges once
|
|
193
|
+
the workers have finished. The conformance suite runs scenarios 1, 2 and 13 on
|
|
194
|
+
two workers, twice, and checks both run files are identical.
|
|
195
|
+
|
|
196
|
+
## The CLI
|
|
197
|
+
|
|
198
|
+
```text
|
|
199
|
+
qualify merge [--evidence-dir DIR] [--allow-runs ID,ID] [--out FILE]
|
|
200
|
+
qualify validate FILE...
|
|
201
|
+
qualify report RUNFILE --pdf FILE [--protocol protocol.yaml] [--map trace.yaml]
|
|
202
|
+
[--theme NAME|PATH] [--title TEXT] [--subtitle TEXT]
|
|
203
|
+
[--document-id TEXT] [--classification TEXT]
|
|
204
|
+
[--signatures ROLE;ROLE;...] [--no-signatures]
|
|
205
|
+
```
|
|
206
|
+
|
|
207
|
+
`python -m qualify` is the same command. Exit codes are those of section 8 of
|
|
208
|
+
the specification: 0 pass, 1 a test failed or errored, 2 invalid input (no
|
|
209
|
+
file written), 3 incomplete, 64 usage error. `merge` writes
|
|
210
|
+
`test-evidence.json` and its `.sha256` sidecar. `report` exits 0 whenever it
|
|
211
|
+
writes the PDF, because a failed run needs its report too.
|
|
212
|
+
|
|
213
|
+
`qualify trace` is in the TypeScript package only. The Python command prints
|
|
214
|
+
where to find it and exits 64:
|
|
215
|
+
|
|
216
|
+
```bash
|
|
217
|
+
npx @lekman/qualify trace qualification/evidence/test-evidence.json \
|
|
218
|
+
--protocol protocol.yaml --md trace.md --json trace.json
|
|
219
|
+
```
|
|
220
|
+
|
|
221
|
+
## The PDF Report
|
|
222
|
+
|
|
223
|
+
```bash
|
|
224
|
+
qualify report qualification/evidence/test-evidence.json --pdf report.pdf \
|
|
225
|
+
--protocol protocol.yaml --theme plain --document-id PROJ-123-OQ
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
`report` renders a run file as a PDF a reviewer can read, sign and file
|
|
229
|
+
without the code: a cover page with the verdict and the counts, the run
|
|
230
|
+
details with the run file's SHA-256, the summary and trace table, one section
|
|
231
|
+
per test with its narrative, checks and attachments, and blank signature
|
|
232
|
+
lines. The layout is section 9 of the specification; how it looks comes from a
|
|
233
|
+
theme. Two themes ship, byte for byte the ones in
|
|
234
|
+
[spec/qualify/report/](../../spec/qualify/report/README.md): `lekman`, the
|
|
235
|
+
default, and `plain`, with no logo and Noto Sans. `--theme` takes a name or the
|
|
236
|
+
path of your own `theme.json`, and `QUALIFY_REPORT_THEME` does the same; the
|
|
237
|
+
flag wins. Set `QUALIFY_REPORT_TIME` to an ISO 8601 UTC instant to reproduce a
|
|
238
|
+
report with a fixed generation time.
|
|
239
|
+
|
|
240
|
+
The first three pages of [a sample report](docs/sample-report-lekman.pdf) with
|
|
241
|
+
the default theme, rendered from the conformance run of scenarios 1, 2 and 6:
|
|
242
|
+
|
|
243
|
+

|
|
244
|
+
|
|
245
|
+

|
|
246
|
+
|
|
247
|
+

|
|
248
|
+
|
|
249
|
+
The same run with `--theme plain` ([PDF](docs/sample-report-plain.pdf)):
|
|
250
|
+
|
|
251
|
+

|
|
252
|
+
|
|
253
|
+

|
|
254
|
+
|
|
255
|
+

|
|
256
|
+
|
|
257
|
+
## What It Costs
|
|
258
|
+
|
|
259
|
+
- **Three runtime dependencies besides pytest, about 31 MB installed.** A
|
|
260
|
+
correct draft 2020-12 validator and a PDF writer with font subsetting are not
|
|
261
|
+
worth writing again.
|
|
262
|
+
|
|
263
|
+
| Package | Installed | For |
|
|
264
|
+
| -------------------- | --------- | ------------------------------------------------------------------------------------- |
|
|
265
|
+
| `fpdf2` 2.8 or later | 28 MB | Writing the PDF. Most of it is `fontTools` (14 MB) and `Pillow` (12 MB), both needed. |
|
|
266
|
+
| `jsonschema` 4.18 | 2 MB | Schema validation, with `attrs`, `referencing` and `rpds-py` (a compiled extension). |
|
|
267
|
+
| `PyYAML` 6 | 0.6 MB | Reading `protocol.yaml` and `trace.yaml`, as YAML 1.2 reads them. |
|
|
268
|
+
|
|
269
|
+
The plugin imports none of the report's dependencies; only `qualify report`
|
|
270
|
+
does. `fpdf2` is under the LGPL-3.0, used unmodified as a library.
|
|
271
|
+
|
|
272
|
+
- **Two bundled themes, 1.7 MB.** The wheel is about 900 KB, most of it the
|
|
273
|
+
fonts: Geist for `lekman` and Noto Sans for `plain`, under the SIL Open Font
|
|
274
|
+
License. A report embeds only the glyphs it uses, so a short one is about
|
|
275
|
+
20 KB (`plain`) to 65 KB (`lekman`, with its logo). A character the theme's
|
|
276
|
+
font cannot draw prints as `?`; the evidence keeps the exact text. The
|
|
277
|
+
`lekman` logo and name are the author's marks and not under the MIT licence;
|
|
278
|
+
use `plain`, or your own theme, for your reports.
|
|
279
|
+
- **The plugin loads in every pytest run** in an environment that has it
|
|
280
|
+
installed. Without a qualification test it writes nothing and runs no git
|
|
281
|
+
command. It still checks the `QUALIFY_*` settings when the session starts.
|
|
282
|
+
- **Sites come from the stack.** A check's site is the first frame outside
|
|
283
|
+
qualify's own code, so a helper that wraps `ctx.check` records the helper's
|
|
284
|
+
line.
|
|
285
|
+
- **Attachments are inline**, base64 in the JSON, 1 MiB each by default. A
|
|
286
|
+
larger attachment ends the test as an error; it is never truncated.
|
|
287
|
+
- **Redaction is process-wide.** A value registered in one test is redacted in
|
|
288
|
+
every later record of the same process, and in pytest's own failure output
|
|
289
|
+
for those tests. Values shorter than 6 characters are refused with a
|
|
290
|
+
`RedactionWarning`, because they would redact ordinary words.
|
|
291
|
+
- **`delegate` runs synchronous functions only.** For async code, await it in
|
|
292
|
+
the test and check its result.
|
|
293
|
+
- **Python types with no JSON counterpart.** `Decimal`, `UUID` and paths are
|
|
294
|
+
recorded as strings, an `Enum` as its value, a dataclass as a map of its
|
|
295
|
+
fields, a naive `datetime` as UTC, and anything else as
|
|
296
|
+
`[unserialisable: <type>]`.
|
|
297
|
+
- **`timeout_ms` needs pytest-timeout.** The declaration turns it into a
|
|
298
|
+
`pytest.mark.timeout` marker; without that plugin, the marker does nothing.
|
|
299
|
+
|
|
300
|
+
## Layout and Contributing
|
|
301
|
+
|
|
302
|
+
```text
|
|
303
|
+
src/qualify/
|
|
304
|
+
api.py the iq, oq, pq and iu decorators
|
|
305
|
+
plugin.py the pytest hooks and the ctx fixture
|
|
306
|
+
context.py ctx, checks, matchers, Secret
|
|
307
|
+
recorder.py the started marker and the redacted record of one test
|
|
308
|
+
declaration.py sections 2.1 and 4: declarations and IDs
|
|
309
|
+
settings.py section 5: settings, write protection, not-applicable rules
|
|
310
|
+
envelope.py section 6.4: the run envelope, git, ULIDs
|
|
311
|
+
evidence.py section 6: file names and the atomic writer
|
|
312
|
+
values.py section 7.1: normalisation, and numbers as JavaScript prints them
|
|
313
|
+
jsontext.py section 6.3: how every evidence file is written
|
|
314
|
+
redaction.py section 7.2
|
|
315
|
+
merge.py section 6.5
|
|
316
|
+
schema.py validation against schemas/, copies of spec/qualify/*.schema.json
|
|
317
|
+
cli.py section 8: merge, validate and report
|
|
318
|
+
trace.py the trace rows the report shows, and protocol.yaml and trace.yaml
|
|
319
|
+
theme.py section 9.3: loading and checking themes, and their templates
|
|
320
|
+
report/ section 9: content.py what it says, layout.py where, pdf.py drawn with fpdf2
|
|
321
|
+
themes/ the bundled themes, copies of spec/qualify/report/themes/
|
|
322
|
+
tests/ unit tests, and the plugin through pytester
|
|
323
|
+
conformance/ one test file per scenario, run.py (the harness), and pdf_text.py
|
|
324
|
+
scripts/ render_pages.py: PDF pages to PNG with pypdfium2, for the images above
|
|
325
|
+
```
|
|
326
|
+
|
|
327
|
+
```bash
|
|
328
|
+
uv sync
|
|
329
|
+
uv run ruff format --check . && uv run ruff check .
|
|
330
|
+
uv run mypy # strict
|
|
331
|
+
uv run pytest --cov # fails under 80 % line coverage
|
|
332
|
+
bun run build # at the repository root, for the trace scenarios
|
|
333
|
+
uv run python conformance/run.py # every scenario, diffed against the expected files
|
|
334
|
+
```
|
|
335
|
+
|
|
336
|
+
The harness runs each scenario's test file through pytest in a fresh evidence
|
|
337
|
+
directory, with every other `QUALIFY_*` and `PYTEST_*` variable removed, then
|
|
338
|
+
runs this package's `qualify merge`, masks the run file and compares it byte
|
|
339
|
+
for byte with `spec/qualify/conformance/expected/`. Scenarios 12, 19 and 20 run
|
|
340
|
+
`node packages/qualify-ts/dist/cli.js trace` on the Python run file. Scenarios 21
|
|
341
|
+
and 22 render reports, extract their text with `pypdf`, and find every line of
|
|
342
|
+
the outline files in order. The harness never writes an expected file; they
|
|
343
|
+
belong to the TypeScript reference.
|
|
344
|
+
|
|
345
|
+
Text extraction cannot see a glyph drawn wrong, so look at the pages too.
|
|
346
|
+
`uv run python conformance/run.py --only report --sample docs` keeps the two
|
|
347
|
+
sample PDFs, and `uv run python scripts/render_pages.py
|
|
348
|
+
docs/sample-report-lekman.pdf docs/report-lekman` writes the page images.
|
|
349
|
+
|
|
350
|
+
## Related
|
|
351
|
+
|
|
352
|
+
- [spec/qualify/README.md](../../spec/qualify/README.md): the contract every
|
|
353
|
+
implementation follows.
|
|
354
|
+
- [practices/qualification-tests/README.md](../../practices/qualification-tests/README.md):
|
|
355
|
+
the problem, the pattern, and when not to use it.
|