lekman-qualify 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (44) hide show
  1. lekman_qualify-0.1.0/.gitignore +12 -0
  2. lekman_qualify-0.1.0/PKG-INFO +355 -0
  3. lekman_qualify-0.1.0/README.md +325 -0
  4. lekman_qualify-0.1.0/pyproject.toml +105 -0
  5. lekman_qualify-0.1.0/src/qualify/__init__.py +46 -0
  6. lekman_qualify-0.1.0/src/qualify/__main__.py +5 -0
  7. lekman_qualify-0.1.0/src/qualify/api.py +162 -0
  8. lekman_qualify-0.1.0/src/qualify/cli.py +364 -0
  9. lekman_qualify-0.1.0/src/qualify/context.py +511 -0
  10. lekman_qualify-0.1.0/src/qualify/declaration.py +190 -0
  11. lekman_qualify-0.1.0/src/qualify/envelope.py +173 -0
  12. lekman_qualify-0.1.0/src/qualify/evidence.py +87 -0
  13. lekman_qualify-0.1.0/src/qualify/jsontext.py +206 -0
  14. lekman_qualify-0.1.0/src/qualify/merge.py +200 -0
  15. lekman_qualify-0.1.0/src/qualify/outcome.py +48 -0
  16. lekman_qualify-0.1.0/src/qualify/plugin.py +314 -0
  17. lekman_qualify-0.1.0/src/qualify/py.typed +0 -0
  18. lekman_qualify-0.1.0/src/qualify/recorder.py +164 -0
  19. lekman_qualify-0.1.0/src/qualify/redaction.py +138 -0
  20. lekman_qualify-0.1.0/src/qualify/report/__init__.py +1 -0
  21. lekman_qualify-0.1.0/src/qualify/report/content.py +420 -0
  22. lekman_qualify-0.1.0/src/qualify/report/layout.py +544 -0
  23. lekman_qualify-0.1.0/src/qualify/report/pdf.py +129 -0
  24. lekman_qualify-0.1.0/src/qualify/runtime.py +125 -0
  25. lekman_qualify-0.1.0/src/qualify/schema.py +81 -0
  26. lekman_qualify-0.1.0/src/qualify/schemas/__init__.py +1 -0
  27. lekman_qualify-0.1.0/src/qualify/schemas/evidence.schema.json +488 -0
  28. lekman_qualify-0.1.0/src/qualify/schemas/report-theme.schema.json +152 -0
  29. lekman_qualify-0.1.0/src/qualify/schemas/trace.schema.json +166 -0
  30. lekman_qualify-0.1.0/src/qualify/settings.py +219 -0
  31. lekman_qualify-0.1.0/src/qualify/site.py +54 -0
  32. lekman_qualify-0.1.0/src/qualify/theme.py +163 -0
  33. lekman_qualify-0.1.0/src/qualify/themes/lekman/fonts/Geist-Regular.ttf +0 -0
  34. lekman_qualify-0.1.0/src/qualify/themes/lekman/fonts/Geist-SemiBold.ttf +0 -0
  35. lekman_qualify-0.1.0/src/qualify/themes/lekman/fonts/GeistMono-Regular.ttf +0 -0
  36. lekman_qualify-0.1.0/src/qualify/themes/lekman/fonts/OFL.txt +93 -0
  37. lekman_qualify-0.1.0/src/qualify/themes/lekman/logo.png +0 -0
  38. lekman_qualify-0.1.0/src/qualify/themes/lekman/theme.json +46 -0
  39. lekman_qualify-0.1.0/src/qualify/themes/plain/fonts/NotoSans-Bold.ttf +0 -0
  40. lekman_qualify-0.1.0/src/qualify/themes/plain/fonts/NotoSans-Regular.ttf +0 -0
  41. lekman_qualify-0.1.0/src/qualify/themes/plain/fonts/OFL.txt +93 -0
  42. lekman_qualify-0.1.0/src/qualify/themes/plain/theme.json +42 -0
  43. lekman_qualify-0.1.0/src/qualify/trace.py +237 -0
  44. lekman_qualify-0.1.0/src/qualify/values.py +216 -0
@@ -0,0 +1,12 @@
1
+ # Python build and tool output.
2
+ .venv/
3
+ __pycache__/
4
+ *.egg-info/
5
+ build/
6
+ dist/
7
+ .coverage
8
+ coverage.xml
9
+ junit.xml
10
+ .pytest_cache/
11
+ .mypy_cache/
12
+ .ruff_cache/
@@ -0,0 +1,355 @@
1
+ Metadata-Version: 2.5
2
+ Name: lekman-qualify
3
+ Version: 0.1.0
4
+ Summary: Qualification tests (IQ, OQ, PQ, intended use) for pytest that write redacted JSON evidence and a PDF report.
5
+ Project-URL: Homepage, https://github.com/lekman/qa-toolkit/tree/main/packages/qualify-python
6
+ Project-URL: Repository, https://github.com/lekman/qa-toolkit
7
+ Project-URL: Issues, https://github.com/lekman/qa-toolkit/issues
8
+ Project-URL: Changelog, https://github.com/lekman/qa-toolkit/blob/main/packages/qualify-python/CHANGELOG.md
9
+ Project-URL: Specification, https://github.com/lekman/qa-toolkit/blob/main/spec/qualify/README.md
10
+ Author: Tobias Lekman
11
+ License-Expression: MIT
12
+ License-File: src/qualify/themes/lekman/fonts/OFL.txt
13
+ License-File: src/qualify/themes/plain/fonts/OFL.txt
14
+ Keywords: evidence,pytest,qualification,traceability,validation
15
+ Classifier: Development Status :: 3 - Alpha
16
+ Classifier: Framework :: Pytest
17
+ Classifier: Intended Audience :: Developers
18
+ Classifier: Operating System :: OS Independent
19
+ Classifier: Programming Language :: Python :: 3
20
+ Classifier: Programming Language :: Python :: 3 :: Only
21
+ Classifier: Topic :: Software Development :: Quality Assurance
22
+ Classifier: Topic :: Software Development :: Testing
23
+ Classifier: Typing :: Typed
24
+ Requires-Python: >=3.10
25
+ Requires-Dist: fpdf2>=2.8.3
26
+ Requires-Dist: jsonschema>=4.18
27
+ Requires-Dist: pytest>=8.3
28
+ Requires-Dist: pyyaml>=6.0.1
29
+ Description-Content-Type: text/markdown
30
+
31
+ # lekman-qualify
32
+
33
+ > Qualification tests (IQ, OQ, PQ, intended use) for pytest that leave redacted
34
+ > JSON evidence and a PDF report a reviewer can read without the code.
35
+
36
+ This is the Python implementation of the
37
+ [qualify specification](../../spec/qualify/README.md), as a pytest plugin. It
38
+ writes the same evidence files as the TypeScript reference
39
+ [@lekman/qualify](../qualify-ts/README.md) and passes the same conformance
40
+ scenarios. It is not published yet.
41
+
42
+ ## Why
43
+
44
+ A passing test run says that the tests passed. It does not say what each test
45
+ claimed, what it checked, which requirement it covers, or where it ran, and it
46
+ cannot be read six months later without the code. A qualification test here is
47
+ an ordinary pytest test with an ID, a narrative (given, when, then) and
48
+ recorded checks. When it runs, the process that ran it writes one JSON record,
49
+ with secrets already removed. `qualify merge` builds one run file from the
50
+ records, and `qualify report` renders it as a PDF to read, sign and file.
51
+
52
+ pytest still runs the tests. There is no wrapper command: select tests with
53
+ pytest's own `-k` and `-m`.
54
+
55
+ ## Install
56
+
57
+ ```bash
58
+ pip install lekman-qualify
59
+ uv add --dev lekman-qualify
60
+ ```
61
+
62
+ Python 3.10 or later and pytest 8.3 or later. The plugin registers itself
63
+ through the `pytest11` entry point; there is nothing to add to `conftest.py`.
64
+
65
+ ## A Test
66
+
67
+ ```python
68
+ import qualify
69
+
70
+
71
+ @qualify.oq(
72
+ "OQ-EXPORT-003",
73
+ "The bank file never changes an amount",
74
+ given="a day of payments in three currencies",
75
+ when="the export writes the bank file",
76
+ then="every amount in the file equals the amount in the ledger",
77
+ requirements=["EXPORT-003"],
78
+ risk="high",
79
+ scope="read",
80
+ tags=["export"],
81
+ )
82
+ def test_bank_file_keeps_amounts(ctx: qualify.Context, ledger) -> None:
83
+ file = export_bank_file(ledger) # ledger is an ordinary fixture
84
+ ctx.check("bank file", "EUR keeps two decimals").that(file.amount("EUR")).equals("100.50")
85
+ ```
86
+
87
+ `qualify.iq`, `qualify.oq`, `qualify.pq` and `qualify.iu` declare the level.
88
+ `scope` is required for OQ and PQ (`read` or `read-write`); IQ is always
89
+ `read` and IU always `read-write`. Further clauses after `then` go in `and_`,
90
+ because `and` is a Python keyword. The function must still be named `test_...`
91
+ for pytest to collect it; the plugin warns when a declared function is not.
92
+
93
+ ```bash
94
+ pytest -k OQ-EXPORT-003 # one test, by ID
95
+ pytest -m oq # every OQ test
96
+ pytest -m "oq and not slow" # by level and tag
97
+ pytest --qualify-merge # also build the run file at the end
98
+ ```
99
+
100
+ A declaration error, such as an OQ test with no `scope` or an ID that does
101
+ not match its level, fails collection with one error per test and writes no
102
+ evidence. Parametrised qualification tests are refused in the same way,
103
+ because one ID must map to one record; declare each case with its own ID.
104
+
105
+ ## The Context
106
+
107
+ | Call | Does |
108
+ | --------------------------------------------------- | -------------------------------------------------------------------- |
109
+ | `ctx.check(context, description).that(v).equals(x)` | A hard check: records, then raises `CheckFailed` if it fails |
110
+ | `ctx.check(...).soft().that(v)...` | A soft check: records and continues; the test fails at the end |
111
+ | `ctx.delegate(context, description, fn)` | Runs `fn`, which has its own assertions, and records it as one check |
112
+ | `ctx.attach(filename, body, content_type)` | An inline attachment; `application/json` also takes a value |
113
+ | `ctx.secret(name)` | Reads `QUALIFY_SECRET_<NAME>`, registers it, returns a `Secret` |
114
+ | `ctx.redact(value)` | Registers a value for redaction |
115
+ | `ctx.skip(reason)` | Skips the test at runtime (`pytest.skip`), with the reason recorded |
116
+ | `ctx.run_context(extra)` | Attaches `run-context.json` with the run's source, actor and target |
117
+ | `ctx.id`, `ctx.level`, `ctx.environment` | Identity, and where the test runs |
118
+
119
+ Matchers: `equals`, `not_equals`, `contains`, `matches`, `is_true`,
120
+ `is_false`, `is_null`, `is_not_null`, `has_length`, `at_least`, `at_most`,
121
+ `one_of`. The evidence records the shared camel-case names (`notEquals`,
122
+ `isTrue`). Values are compared after the normalisation in section 7.1 of the
123
+ specification, so `NaN` equals `NaN`, `1` equals `1.0`, and key order does not
124
+ matter.
125
+
126
+ `qualify.Secret` is a `str` subclass whose `repr()` hides the value and which
127
+ registers the value for redaction when it is created. Use it for any secret
128
+ your own fixtures read:
129
+
130
+ ```python
131
+ @pytest.fixture
132
+ def api_token() -> qualify.Secret:
133
+ return qualify.Secret(vault.read("api-token"))
134
+ ```
135
+
136
+ ### Outcomes
137
+
138
+ The outcome comes from a `pytest_runtest_makereport` hook wrapper that reads
139
+ the exception of the phase that ended the test.
140
+
141
+ | The test ends with | Outcome |
142
+ | ----------------------------------------------------------------- | ------------------------------------- |
143
+ | Nothing, and every check passed | `pass` |
144
+ | A failed check, an `AssertionError`, or `pytest.fail()` | `fail` |
145
+ | Any other exception, also from a fixture during setup | `error`, with the type and message |
146
+ | `ctx.skip()`, `pytest.skip()`, a skip marker, or `pytest.xfail()` | `skip`, with the reason |
147
+ | A write-protection or "not applicable" rule | `skip`, before any fixture runs |
148
+ | The process dies (a crash, a killed worker) | `error`, `DidNotFinish`, from `merge` |
149
+
150
+ A hard check that fails raises `CheckFailed`. If the body catches it, the test
151
+ still fails at the end, because a record says `pass` only when every check
152
+ passed. An `xfail` marker does not change the record: a test that fails is
153
+ recorded as failing, whatever pytest reports. A teardown error after the body
154
+ does not change the record either.
155
+
156
+ ## Settings
157
+
158
+ | Variable | pytest ini option | Default | Meaning |
159
+ | -------------------------- | -------------------------- | ------------------------ | ------------------------------------------------------------ |
160
+ | `QUALIFY_ENVIRONMENT` | `qualify_environment` | `local` | The environment name |
161
+ | `QUALIFY_PRODUCTION` | `qualify_production` | `production,prod` | Environments where `read-write` tests are skipped |
162
+ | `QUALIFY_ALLOW_WRITE` | `qualify_allow_write` | none | `ID=reason;ID=reason`: tests allowed to write in production |
163
+ | `QUALIFY_NO_OVERRIDES` | `qualify_no_overrides` | off | `1` refuses every override and stops the run at start |
164
+ | `QUALIFY_EVIDENCE_DIR` | `qualify_evidence_dir` | `qualification/evidence` | Where records and the run file go |
165
+ | `QUALIFY_ATTACHMENT_LIMIT` | `qualify_attachment_limit` | `1048576` | Bytes per attachment; a larger one ends the test as an error |
166
+ | `QUALIFY_RUN_ID` | | created per run | Share one run across jobs, machines or languages |
167
+ | `QUALIFY_TARGET_<NAME>` | | none | Facts about the target, recorded as `run.target.<name>` |
168
+
169
+ The environment variable wins over the ini option. `QUALIFY_EVIDENCE_DIR` is
170
+ relative to the working directory; `qualify_evidence_dir` is relative to
171
+ pytest's rootdir. A setting that cannot be used stops the run before any test
172
+ starts, with a usage error.
173
+
174
+ Code can give the same settings, and three that only code can give, in
175
+ `conftest.py`. The variables and ini options still win.
176
+
177
+ ```python
178
+ # conftest.py
179
+ import qualify
180
+
181
+ qualify.configure(
182
+ resolve_environment=lambda: deploy_target(), # if it raises: production
183
+ secret_source=lambda name: vault.read(name), # behind ctx.secret
184
+ not_applicable=[qualify.NotApplicable("no GPU in CI", tags=["gpu"], environments=["ci"])],
185
+ )
186
+ ```
187
+
188
+ ## pytest-xdist
189
+
190
+ Each worker writes its own records. The controller creates one run ID (or
191
+ takes `QUALIFY_RUN_ID`) and hands it to every worker, so all records of one
192
+ `pytest -n 4` run share it. With `--qualify-merge`, the controller merges once
193
+ the workers have finished. The conformance suite runs scenarios 1, 2 and 13 on
194
+ two workers, twice, and checks both run files are identical.
195
+
196
+ ## The CLI
197
+
198
+ ```text
199
+ qualify merge [--evidence-dir DIR] [--allow-runs ID,ID] [--out FILE]
200
+ qualify validate FILE...
201
+ qualify report RUNFILE --pdf FILE [--protocol protocol.yaml] [--map trace.yaml]
202
+ [--theme NAME|PATH] [--title TEXT] [--subtitle TEXT]
203
+ [--document-id TEXT] [--classification TEXT]
204
+ [--signatures ROLE;ROLE;...] [--no-signatures]
205
+ ```
206
+
207
+ `python -m qualify` is the same command. Exit codes are those of section 8 of
208
+ the specification: 0 pass, 1 a test failed or errored, 2 invalid input (no
209
+ file written), 3 incomplete, 64 usage error. `merge` writes
210
+ `test-evidence.json` and its `.sha256` sidecar. `report` exits 0 whenever it
211
+ writes the PDF, because a failed run needs its report too.
212
+
213
+ `qualify trace` is in the TypeScript package only. The Python command prints
214
+ where to find it and exits 64:
215
+
216
+ ```bash
217
+ npx @lekman/qualify trace qualification/evidence/test-evidence.json \
218
+ --protocol protocol.yaml --md trace.md --json trace.json
219
+ ```
220
+
221
+ ## The PDF Report
222
+
223
+ ```bash
224
+ qualify report qualification/evidence/test-evidence.json --pdf report.pdf \
225
+ --protocol protocol.yaml --theme plain --document-id PROJ-123-OQ
226
+ ```
227
+
228
+ `report` renders a run file as a PDF a reviewer can read, sign and file
229
+ without the code: a cover page with the verdict and the counts, the run
230
+ details with the run file's SHA-256, the summary and trace table, one section
231
+ per test with its narrative, checks and attachments, and blank signature
232
+ lines. The layout is section 9 of the specification; how it looks comes from a
233
+ theme. Two themes ship, byte for byte the ones in
234
+ [spec/qualify/report/](../../spec/qualify/report/README.md): `lekman`, the
235
+ default, and `plain`, with no logo and Noto Sans. `--theme` takes a name or the
236
+ path of your own `theme.json`, and `QUALIFY_REPORT_THEME` does the same; the
237
+ flag wins. Set `QUALIFY_REPORT_TIME` to an ISO 8601 UTC instant to reproduce a
238
+ report with a fixed generation time.
239
+
240
+ The first three pages of [a sample report](docs/sample-report-lekman.pdf) with
241
+ the default theme, rendered from the conformance run of scenarios 1, 2 and 6:
242
+
243
+ ![Cover page of a sample report, lekman theme](docs/report-lekman-page-1.png)
244
+
245
+ ![Run details, summary and trace table, lekman theme](docs/report-lekman-page-2.png)
246
+
247
+ ![The first two test sections, lekman theme](docs/report-lekman-page-3.png)
248
+
249
+ The same run with `--theme plain` ([PDF](docs/sample-report-plain.pdf)):
250
+
251
+ ![Cover page of a sample report, plain theme](docs/report-plain-page-1.png)
252
+
253
+ ![Run details, summary and trace table, plain theme](docs/report-plain-page-2.png)
254
+
255
+ ![The first two test sections, plain theme](docs/report-plain-page-3.png)
256
+
257
+ ## What It Costs
258
+
259
+ - **Three runtime dependencies besides pytest, about 31 MB installed.** A
260
+ correct draft 2020-12 validator and a PDF writer with font subsetting are not
261
+ worth writing again.
262
+
263
+ | Package | Installed | For |
264
+ | -------------------- | --------- | ------------------------------------------------------------------------------------- |
265
+ | `fpdf2` 2.8 or later | 28 MB | Writing the PDF. Most of it is `fontTools` (14 MB) and `Pillow` (12 MB), both needed. |
266
+ | `jsonschema` 4.18 | 2 MB | Schema validation, with `attrs`, `referencing` and `rpds-py` (a compiled extension). |
267
+ | `PyYAML` 6 | 0.6 MB | Reading `protocol.yaml` and `trace.yaml`, as YAML 1.2 reads them. |
268
+
269
+ The plugin imports none of the report's dependencies; only `qualify report`
270
+ does. `fpdf2` is under the LGPL-3.0, used unmodified as a library.
271
+
272
+ - **Two bundled themes, 1.7 MB.** The wheel is about 900 KB, most of it the
273
+ fonts: Geist for `lekman` and Noto Sans for `plain`, under the SIL Open Font
274
+ License. A report embeds only the glyphs it uses, so a short one is about
275
+ 20 KB (`plain`) to 65 KB (`lekman`, with its logo). A character the theme's
276
+ font cannot draw prints as `?`; the evidence keeps the exact text. The
277
+ `lekman` logo and name are the author's marks and not under the MIT licence;
278
+ use `plain`, or your own theme, for your reports.
279
+ - **The plugin loads in every pytest run** in an environment that has it
280
+ installed. Without a qualification test it writes nothing and runs no git
281
+ command. It still checks the `QUALIFY_*` settings when the session starts.
282
+ - **Sites come from the stack.** A check's site is the first frame outside
283
+ qualify's own code, so a helper that wraps `ctx.check` records the helper's
284
+ line.
285
+ - **Attachments are inline**, base64 in the JSON, 1 MiB each by default. A
286
+ larger attachment ends the test as an error; it is never truncated.
287
+ - **Redaction is process-wide.** A value registered in one test is redacted in
288
+ every later record of the same process, and in pytest's own failure output
289
+ for those tests. Values shorter than 6 characters are refused with a
290
+ `RedactionWarning`, because they would redact ordinary words.
291
+ - **`delegate` runs synchronous functions only.** For async code, await it in
292
+ the test and check its result.
293
+ - **Python types with no JSON counterpart.** `Decimal`, `UUID` and paths are
294
+ recorded as strings, an `Enum` as its value, a dataclass as a map of its
295
+ fields, a naive `datetime` as UTC, and anything else as
296
+ `[unserialisable: <type>]`.
297
+ - **`timeout_ms` needs pytest-timeout.** The declaration turns it into a
298
+ `pytest.mark.timeout` marker; without that plugin, the marker does nothing.
299
+
300
+ ## Layout and Contributing
301
+
302
+ ```text
303
+ src/qualify/
304
+ api.py the iq, oq, pq and iu decorators
305
+ plugin.py the pytest hooks and the ctx fixture
306
+ context.py ctx, checks, matchers, Secret
307
+ recorder.py the started marker and the redacted record of one test
308
+ declaration.py sections 2.1 and 4: declarations and IDs
309
+ settings.py section 5: settings, write protection, not-applicable rules
310
+ envelope.py section 6.4: the run envelope, git, ULIDs
311
+ evidence.py section 6: file names and the atomic writer
312
+ values.py section 7.1: normalisation, and numbers as JavaScript prints them
313
+ jsontext.py section 6.3: how every evidence file is written
314
+ redaction.py section 7.2
315
+ merge.py section 6.5
316
+ schema.py validation against schemas/, copies of spec/qualify/*.schema.json
317
+ cli.py section 8: merge, validate and report
318
+ trace.py the trace rows the report shows, and protocol.yaml and trace.yaml
319
+ theme.py section 9.3: loading and checking themes, and their templates
320
+ report/ section 9: content.py what it says, layout.py where, pdf.py drawn with fpdf2
321
+ themes/ the bundled themes, copies of spec/qualify/report/themes/
322
+ tests/ unit tests, and the plugin through pytester
323
+ conformance/ one test file per scenario, run.py (the harness), and pdf_text.py
324
+ scripts/ render_pages.py: PDF pages to PNG with pypdfium2, for the images above
325
+ ```
326
+
327
+ ```bash
328
+ uv sync
329
+ uv run ruff format --check . && uv run ruff check .
330
+ uv run mypy # strict
331
+ uv run pytest --cov # fails under 80 % line coverage
332
+ bun run build # at the repository root, for the trace scenarios
333
+ uv run python conformance/run.py # every scenario, diffed against the expected files
334
+ ```
335
+
336
+ The harness runs each scenario's test file through pytest in a fresh evidence
337
+ directory, with every other `QUALIFY_*` and `PYTEST_*` variable removed, then
338
+ runs this package's `qualify merge`, masks the run file and compares it byte
339
+ for byte with `spec/qualify/conformance/expected/`. Scenarios 12, 19 and 20 run
340
+ `node packages/qualify-ts/dist/cli.js trace` on the Python run file. Scenarios 21
341
+ and 22 render reports, extract their text with `pypdf`, and find every line of
342
+ the outline files in order. The harness never writes an expected file; they
343
+ belong to the TypeScript reference.
344
+
345
+ Text extraction cannot see a glyph drawn wrong, so look at the pages too.
346
+ `uv run python conformance/run.py --only report --sample docs` keeps the two
347
+ sample PDFs, and `uv run python scripts/render_pages.py
348
+ docs/sample-report-lekman.pdf docs/report-lekman` writes the page images.
349
+
350
+ ## Related
351
+
352
+ - [spec/qualify/README.md](../../spec/qualify/README.md): the contract every
353
+ implementation follows.
354
+ - [practices/qualification-tests/README.md](../../practices/qualification-tests/README.md):
355
+ the problem, the pattern, and when not to use it.