pagetrace 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- pagetrace-0.1.0/PKG-INFO +316 -0
- pagetrace-0.1.0/README.md +294 -0
- pagetrace-0.1.0/pyproject.toml +122 -0
- pagetrace-0.1.0/pyproject.toml.orig +107 -0
- pagetrace-0.1.0/src/pagetrace/__init__.py +3 -0
- pagetrace-0.1.0/src/pagetrace/__main__.py +5 -0
- pagetrace-0.1.0/src/pagetrace/checks/__init__.py +69 -0
- pagetrace-0.1.0/src/pagetrace/checks/base.py +50 -0
- pagetrace-0.1.0/src/pagetrace/checks/continuation.py +129 -0
- pagetrace-0.1.0/src/pagetrace/checks/coverage.py +38 -0
- pagetrace-0.1.0/src/pagetrace/checks/cursors.py +195 -0
- pagetrace-0.1.0/src/pagetrace/checks/items.py +64 -0
- pagetrace-0.1.0/src/pagetrace/checks/sequence.py +180 -0
- pagetrace-0.1.0/src/pagetrace/checks/totals.py +124 -0
- pagetrace-0.1.0/src/pagetrace/cli.py +135 -0
- pagetrace-0.1.0/src/pagetrace/errors.py +7 -0
- pagetrace-0.1.0/src/pagetrace/inputs/__init__.py +106 -0
- pagetrace-0.1.0/src/pagetrace/inputs/har.py +185 -0
- pagetrace-0.1.0/src/pagetrace/inputs/jsonl.py +145 -0
- pagetrace-0.1.0/src/pagetrace/limits.py +16 -0
- pagetrace-0.1.0/src/pagetrace/model.py +296 -0
- pagetrace-0.1.0/src/pagetrace/normalize.py +346 -0
- pagetrace-0.1.0/src/pagetrace/py.typed +0 -0
- pagetrace-0.1.0/src/pagetrace/render/__init__.py +52 -0
- pagetrace-0.1.0/src/pagetrace/render/json.py +72 -0
- pagetrace-0.1.0/src/pagetrace/render/junit.py +79 -0
- pagetrace-0.1.0/src/pagetrace/render/terminal.py +69 -0
- pagetrace-0.1.0/src/pagetrace/selectors.py +154 -0
pagetrace-0.1.0/PKG-INFO
ADDED
|
@@ -0,0 +1,316 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: pagetrace
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Offline checker for API pagination traces. Never claims more than the trace proves.
|
|
5
|
+
Author: PageTrace maintainers
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
Classifier: Development Status :: 2 - Pre-Alpha
|
|
8
|
+
Classifier: Environment :: Console
|
|
9
|
+
Classifier: Intended Audience :: Developers
|
|
10
|
+
Classifier: Programming Language :: Python :: 3
|
|
11
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
12
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
13
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
15
|
+
Classifier: Topic :: Software Development :: Testing
|
|
16
|
+
Classifier: Typing :: Typed
|
|
17
|
+
Requires-Dist: python-jsonpath>=2.2.1
|
|
18
|
+
Requires-Python: >=3.11
|
|
19
|
+
Project-URL: Homepage, https://github.com/OWNER/pagetrace
|
|
20
|
+
Project-URL: Changelog, https://github.com/OWNER/pagetrace/blob/main/CHANGELOG.md
|
|
21
|
+
Description-Content-Type: text/markdown
|
|
22
|
+
|
|
23
|
+
<div align="center">
|
|
24
|
+
|
|
25
|
+
<img src="https://raw.githubusercontent.com/Corner324/PageTrace/main/docs/assets/header.svg" alt="PageTrace" width="720">
|
|
26
|
+
|
|
27
|
+
**English** · [Русский](https://github.com/Corner324/PageTrace/blob/main/README.ru.md) · [中文](https://github.com/Corner324/PageTrace/blob/main/README.zh.md)
|
|
28
|
+
|
|
29
|
+
**Prove what your API pagination actually did. Offline, from a trace you already have.**
|
|
30
|
+
|
|
31
|
+
[](https://github.com/Corner324/PageTrace/actions/workflows/ci.yml)
|
|
32
|
+
[](https://github.com/Corner324/PageTrace/blob/main/docs/testing.md)
|
|
33
|
+
[](https://github.com/Corner324/PageTrace/blob/main/docs/testing.md)
|
|
34
|
+
[](https://github.com/Corner324/PageTrace/blob/main/docs/security.md)
|
|
35
|
+
[](https://github.com/Corner324/PageTrace/blob/main/docs/compatibility.md)
|
|
36
|
+
[](https://github.com/Corner324/PageTrace/blob/main/LICENSE)
|
|
37
|
+
|
|
38
|
+
[⚡ Install](#-install) · [🔍 How it works](#-how-it-works) · [🎯 Presets](#-presets) · [🧭 What it will not tell you](#-what-it-will-not-tell-you) · [🔒 Security](#-security) · [📐 Invariants](https://github.com/Corner324/PageTrace/blob/main/specs/invariants.md)
|
|
39
|
+
|
|
40
|
+
</div>
|
|
41
|
+
|
|
42
|
+
---
|
|
43
|
+
|
|
44
|
+
## 💡 Why this exists
|
|
45
|
+
|
|
46
|
+
Pagination fails quietly. Every page returns `200 OK`, the loop terminates, nothing is logged as
|
|
47
|
+
an error, and the export is short by four hundred rows. Nobody notices until someone reconciles
|
|
48
|
+
a total a month later, and by then the capture is gone.
|
|
49
|
+
|
|
50
|
+
The usual tools are the wrong shape for it. An integration test hits the live API and goes flaky.
|
|
51
|
+
Logs show you individual requests, never the relationship between them - and the bug always lives
|
|
52
|
+
in the relationship: page 3 asked for a token page 2 never handed out. Reading a HAR by hand works
|
|
53
|
+
right up to the twelfth page.
|
|
54
|
+
|
|
55
|
+
PageTrace takes the recording you already have and reports what it proves.
|
|
56
|
+
|
|
57
|
+
| You have heard | PageTrace answers, from the trace alone |
|
|
58
|
+
| -------------- | -------------------------------------- |
|
|
59
|
+
| "The export is missing rows" | whether a page was requested with a token the previous page never returned, whether the sequence revisited a state, whether the capture just stopped early |
|
|
60
|
+
| "The job runs forever" | whether the cursor stalled on one value, or the request sequence returned to a state it had already visited |
|
|
61
|
+
| "We get duplicates downstream" | which item identity appeared on which two pages |
|
|
62
|
+
| "Their API is broken" | which side the trace can actually attribute it to, which is often neither |
|
|
63
|
+
|
|
64
|
+
> [!IMPORTANT]
|
|
65
|
+
> **The one rule: never claim more than the trace proves.**
|
|
66
|
+
> Every finding carries a certainty and the exact observations behind it. There are no confidence
|
|
67
|
+
> scores anywhere in this tool. When the evidence is not there, it says
|
|
68
|
+
> `INSUFFICIENT_EVIDENCE` and names the flag you need to set, instead of guessing.
|
|
69
|
+
|
|
70
|
+
Three places it earns its keep: pinned to a regression test in CI, where a proven defect fails the
|
|
71
|
+
build; in your hand during an incident, where a colleague's HAR is the only evidence left; and
|
|
72
|
+
pointed at a vendor before you integrate, where you need the claim to survive being quoted back
|
|
73
|
+
at you.
|
|
74
|
+
|
|
75
|
+
---
|
|
76
|
+
|
|
77
|
+
## ⚡ See it in 30 seconds
|
|
78
|
+
|
|
79
|
+
```bash
|
|
80
|
+
uv run pagetrace check examples/broken-cursor.jsonl
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
```
|
|
84
|
+
broken-cursor.jsonl: 3 pages, preset cursor
|
|
85
|
+
|
|
86
|
+
PROVEN ERROR CURSOR_CONTINUATION_MISMATCH [INV-CURSOR-001] pages 1, 2
|
|
87
|
+
Page 2 was requested with a continuation token that page 1 did not return.
|
|
88
|
+
page[1].response.next_cursor = "eyJvIjo0fQ"
|
|
89
|
+
page[2].request.cursor = "eyJvIjozfQ"
|
|
90
|
+
|
|
91
|
+
PROVEN WARNING DUPLICATE_ITEM_ACROSS_PAGES [INV-ITEM-001] pages 1, 2
|
|
92
|
+
An item observed on page 1 was observed again on page 2; its identity is in the evidence below. That the same item appears twice is a fact; whether it means overlapping pages depends on a uniqueness contract this trace does not contain.
|
|
93
|
+
page[1].items[0].id = 3
|
|
94
|
+
page[2].items[0].id = 3
|
|
95
|
+
|
|
96
|
+
2 proven, 0 suspicious, 0 insufficient evidence
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
Exit code 1. Note what the second finding refuses to do: it reports the duplicate as a fact and
|
|
100
|
+
declines to call it a pagination bug, because no trace can carry a uniqueness contract. That is
|
|
101
|
+
the whole design in one message.
|
|
102
|
+
|
|
103
|
+
`examples/clean-cursor.jsonl` is the same run without the defects, and exits 0.
|
|
104
|
+
|
|
105
|
+
---
|
|
106
|
+
|
|
107
|
+
## 🔍 How it works
|
|
108
|
+
|
|
109
|
+
<div align="center">
|
|
110
|
+
<picture>
|
|
111
|
+
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/Corner324/PageTrace/main/docs/assets/flow.en.dark.svg">
|
|
112
|
+
<img alt="Recorded trace, parser, normalizer, invariant engine, report. No network call at any step." src="https://raw.githubusercontent.com/Corner324/PageTrace/main/docs/assets/flow.en.light.svg" width="820">
|
|
113
|
+
</picture>
|
|
114
|
+
</div>
|
|
115
|
+
|
|
116
|
+
The important part is structural, not a promise. A detector receives normalized observations and
|
|
117
|
+
nothing else: `Page` has no field for a raw body or an `authorization` header, so the engine
|
|
118
|
+
physically cannot read one. See [docs/architecture.md](https://github.com/Corner324/PageTrace/blob/main/docs/architecture.md).
|
|
119
|
+
|
|
120
|
+
---
|
|
121
|
+
|
|
122
|
+
## ⚡ Install
|
|
123
|
+
|
|
124
|
+
> [!NOTE]
|
|
125
|
+
> Installing from PyPI is not available yet. The release workflow publishes through Trusted
|
|
126
|
+
> Publishing over OIDC with no token fallback, so nothing can be published until a trusted
|
|
127
|
+
> publisher is configured for the project and the `pypi` environment
|
|
128
|
+
> ([docs/compatibility.md](https://github.com/Corner324/PageTrace/blob/main/docs/compatibility.md)).
|
|
129
|
+
|
|
130
|
+
From a clone, with [uv](https://docs.astral.sh/uv/):
|
|
131
|
+
|
|
132
|
+
```bash
|
|
133
|
+
uv sync
|
|
134
|
+
uv run pagetrace check examples/broken-cursor.jsonl
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
With pip into an environment of your own, or `pipx install .` for a standalone command:
|
|
138
|
+
|
|
139
|
+
```bash
|
|
140
|
+
pip install .
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
---
|
|
144
|
+
|
|
145
|
+
## 📼 Record a trace
|
|
146
|
+
|
|
147
|
+
A JSONL trace is one JSON object per line, one per request/response pair, in capture order:
|
|
148
|
+
|
|
149
|
+
```json
|
|
150
|
+
{"v": 1, "seq": 0, "request": {"method": "GET", "url": "https://api.example.com/v1/items?limit=2"}, "response": {"status": 200, "headers": {"content-type": "application/json"}, "body": {"items": [{"id": 1, "name": "item-1"}, {"id": 2, "name": "item-2"}], "next_cursor": "eyJvIjoyfQ", "has_more": true, "total": 5}}}
|
|
151
|
+
```
|
|
152
|
+
|
|
153
|
+
`v` and `response.status` are required; `seq` is optional and line order is authoritative. Full
|
|
154
|
+
rules: [specs/formats/jsonl-v1.md](https://github.com/Corner324/PageTrace/blob/main/specs/formats/jsonl-v1.md).
|
|
155
|
+
|
|
156
|
+
<details>
|
|
157
|
+
<summary><b>Or export a HAR from the browser</b> - no code to write, works against anyone's API</summary>
|
|
158
|
+
|
|
159
|
+
<br>
|
|
160
|
+
|
|
161
|
+
Open DevTools, go to the Network tab, reproduce the pagination, export the request list as HAR.
|
|
162
|
+
A browser capture holds every asset the page loaded, so filter it down to the API calls:
|
|
163
|
+
|
|
164
|
+
```bash
|
|
165
|
+
uv run pagetrace check your-capture.har --url-filter /v1/items
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
`OPTIONS` and `HEAD` requests are dropped before pages are numbered, so a CORS preflight never
|
|
169
|
+
looks like a client that failed to advance.
|
|
170
|
+
|
|
171
|
+
</details>
|
|
172
|
+
|
|
173
|
+
> [!WARNING]
|
|
174
|
+
> A HAR holds live credentials. PageTrace keeps exactly one response header and never renders a
|
|
175
|
+
> body, but the file on your disk still contains your `Authorization` header - treat it like a
|
|
176
|
+
> password, and do not attach it to a public issue.
|
|
177
|
+
|
|
178
|
+
---
|
|
179
|
+
|
|
180
|
+
## 🎯 Presets
|
|
181
|
+
|
|
182
|
+
Selectors are [RFC 9535](https://www.rfc-editor.org/rfc/rfc9535.html) JSONPath. A preset default
|
|
183
|
+
that does not match resolves to "not found" and proves nothing, rather than becoming a guess.
|
|
184
|
+
|
|
185
|
+
| Preset | Default selectors | Fits |
|
|
186
|
+
| ------ | ----------------- | ---- |
|
|
187
|
+
| `cursor` <sub>default</sub> | `--cursor $.query.cursor` `--next $.next_cursor` `--items $.items` `--id $.id` `--total $.total` `--has-more $.has_more` | APIs that hand back an opaque continuation token |
|
|
188
|
+
| `page` | `--page $.query.page`, plus the same items / id / total / has-more defaults | page numbers, 0-based or 1-based |
|
|
189
|
+
| `offset` | `--offset $.query.offset`, plus the same items / id / total / has-more defaults | an offset and a limit |
|
|
190
|
+
| `link` | items / id / total / has-more only; continuation comes from the HTTP `Link` header | `rel="next"` in a Link header (RFC 8288) |
|
|
191
|
+
|
|
192
|
+
> [!TIP]
|
|
193
|
+
> Every default is overridable by flag, and `uv run pagetrace check --help` prints all of them
|
|
194
|
+
> with the preset each belongs to. If a finding says `INSUFFICIENT_EVIDENCE`, the flag to set is
|
|
195
|
+
> named in the message.
|
|
196
|
+
|
|
197
|
+
---
|
|
198
|
+
|
|
199
|
+
## 📊 Certainty, severity, exit codes
|
|
200
|
+
|
|
201
|
+
Two independent axes. Certainty is what the evidence supports; severity is how much it matters if
|
|
202
|
+
true. A finding is never a probability.
|
|
203
|
+
|
|
204
|
+
| Certainty | Meaning |
|
|
205
|
+
| --------- | ------- |
|
|
206
|
+
| `PROVEN` | the trace itself shows it, with the preconditions satisfied |
|
|
207
|
+
| `SUSPICIOUS` | consistent with a defect **and** with an innocent explanation |
|
|
208
|
+
| `INSUFFICIENT_EVIDENCE` | the selector or the trace did not supply what the check needs |
|
|
209
|
+
|
|
210
|
+
| Exit | Meaning |
|
|
211
|
+
| ---- | ------- |
|
|
212
|
+
| `0` | nothing both proven and severe |
|
|
213
|
+
| `1` | at least one `PROVEN` finding of severity `ERROR` |
|
|
214
|
+
| `2` | invalid input or configuration |
|
|
215
|
+
|
|
216
|
+
Exit 1 needs both axes. A duplicate item across pages is `PROVEN WARNING` and does not fail a
|
|
217
|
+
build: the finding declines to say whether overlapping windows are a bug, so failing on it would
|
|
218
|
+
be the tool claiming what its own message refuses to. A build that goes red on "this might be a
|
|
219
|
+
cache" trains people to ignore the tool.
|
|
220
|
+
|
|
221
|
+
For CI, take the machine formats rather than the terminal text, which is not a contract:
|
|
222
|
+
|
|
223
|
+
```bash
|
|
224
|
+
uv run pagetrace check examples/broken-cursor.jsonl --format json --output report.json
|
|
225
|
+
```
|
|
226
|
+
|
|
227
|
+
---
|
|
228
|
+
|
|
229
|
+
## 🧭 What it will not tell you
|
|
230
|
+
|
|
231
|
+
This section is the point of the project, not a disclaimer.
|
|
232
|
+
|
|
233
|
+
- **That pagination never terminates.** A finite trace can prove the sequence revisited a state,
|
|
234
|
+
and that is what `INV-CURSOR-003` says. A cycle back to the page-0 state after a `null` cursor
|
|
235
|
+
drops to `SUSPICIOUS`, because two concatenated runs look exactly the same.
|
|
236
|
+
- **That records are missing.** `INV-TOTAL-001` compares a declared total against observed
|
|
237
|
+
identities and is never worded as "N records are missing": PageTrace does not know that `total`
|
|
238
|
+
counts the same population, that the dataset held still, or that the same filters applied.
|
|
239
|
+
- **That the API is at fault.** `INV-CONT-001` and `INV-TRACE-001` report that the trace ended
|
|
240
|
+
while the server was still offering more, and name neither the API, the client, nor the recorder.
|
|
241
|
+
- **Anything about a duplicate beyond the duplicate itself**, without a uniqueness contract the
|
|
242
|
+
trace does not contain.
|
|
243
|
+
- **Anything from a page it could not read.** Non-2xx responses are excluded from every
|
|
244
|
+
body-derived check and reported once by `INV-TRACE-002`, so a trace nobody could read never
|
|
245
|
+
looks clean.
|
|
246
|
+
- **Anything a selector did not resolve.** Matching nothing, matching twice, or landing on an
|
|
247
|
+
object is `INSUFFICIENT_EVIDENCE` naming the flag, never a guess. JSON `null` is a different
|
|
248
|
+
state: an observation meaning "the server declared no continuation".
|
|
249
|
+
- **That `5` and `"5"` differ** - comparison is on the canonical text form, because a query
|
|
250
|
+
parameter can only carry text - or that a truthy `1` or `"true"` means `has_more`. Only a
|
|
251
|
+
boolean counts.
|
|
252
|
+
|
|
253
|
+
Every claim and every non-claim condition is in [specs/invariants.md](https://github.com/Corner324/PageTrace/blob/main/specs/invariants.md), the
|
|
254
|
+
normative contract for the tool. Eleven invariants, each with written preconditions.
|
|
255
|
+
|
|
256
|
+
---
|
|
257
|
+
|
|
258
|
+
## 🔒 Security
|
|
259
|
+
|
|
260
|
+
A HAR or JSONL trace is an untrusted, secret-bearing file. Parsers keep exactly one response
|
|
261
|
+
header, `link`, because exactly one is read; `authorization`, `cookie` and everything else is
|
|
262
|
+
dropped before the model is built. Raw bodies never reach a detector or a renderer, so only the
|
|
263
|
+
scalars you selected are displayed, with control characters escaped and lengths truncated. Input
|
|
264
|
+
is bounded per line and per decoded body, exceeding a limit is a usage error rather than a
|
|
265
|
+
traceback, and the package contains no networking dependency or code path at all. Threat table
|
|
266
|
+
and limits: [docs/security.md](https://github.com/Corner324/PageTrace/blob/main/docs/security.md).
|
|
267
|
+
|
|
268
|
+
---
|
|
269
|
+
|
|
270
|
+
## 📈 Performance
|
|
271
|
+
|
|
272
|
+
Measured 2026-09-11, one run per shape on a Windows 11 developer machine with CPython 3.12.6;
|
|
273
|
+
memory is peak Python allocation from `tracemalloc`. The shape of the numbers is the finding, not
|
|
274
|
+
the third digit.
|
|
275
|
+
|
|
276
|
+
| Shape | Pages | Items | Seconds | Peak MiB |
|
|
277
|
+
| ----- | ----: | ----: | ------: | -------: |
|
|
278
|
+
| wide | 1 000 | 100 000 | 2.6 | 16.1 |
|
|
279
|
+
| very wide | 1 000 | 1 000 000 | 20.9 | 156.8 |
|
|
280
|
+
| ten thousand pages | 10 000 | 1 000 000 | 28.3 | 142.8 |
|
|
281
|
+
| hundred thousand pages | 100 000 | 5 000 000 | 120.8 | 715.4 |
|
|
282
|
+
|
|
283
|
+
> [!NOTE]
|
|
284
|
+
> Three time targets are missed and stay written down as targets rather than quietly relaxed:
|
|
285
|
+
> 1k pages / 100k ids wants 2 s and takes 2.6 s; 10k / 1M wants 10 s and takes 28.3 s, about 3x
|
|
286
|
+
> over; the 100k / 5M stretch target wants 60 s and takes 120.8 s, about 2x over. Both memory
|
|
287
|
+
> targets are met with room to spare. Full table and reasoning:
|
|
288
|
+
> [docs/performance.md](https://github.com/Corner324/PageTrace/blob/main/docs/performance.md).
|
|
289
|
+
|
|
290
|
+
---
|
|
291
|
+
|
|
292
|
+
## 🛠 Contributing
|
|
293
|
+
|
|
294
|
+
```bash
|
|
295
|
+
uv run ruff check . && uv run ruff format --check . && uv run pyright && uv run pytest
|
|
296
|
+
```
|
|
297
|
+
|
|
298
|
+
425 tests pass locally at 98.97% branch coverage, on 3.11, 3.12, 3.13 and 3.14 on Windows. The
|
|
299
|
+
property suite in `tests/property/` is the main correctness layer: a valid generated trace must
|
|
300
|
+
produce no `PROVEN` finding, and each mutation operator must wake exactly one detector. Mutation
|
|
301
|
+
score is 77.9% against the unit tests as of 2026-09-12, with the survivors sorted by hand in
|
|
302
|
+
[docs/testing.md](https://github.com/Corner324/PageTrace/blob/main/docs/testing.md); mutmut refuses to run on Windows, so that run came from a
|
|
303
|
+
Linux container.
|
|
304
|
+
|
|
305
|
+
Before a new detector: an entry in [specs/invariants.md](https://github.com/Corner324/PageTrace/blob/main/specs/invariants.md) with its non-claim
|
|
306
|
+
conditions, then the test oracle, then the code. [CONTRIBUTING.md](https://github.com/Corner324/PageTrace/blob/main/CONTRIBUTING.md) has the
|
|
307
|
+
workflow and the definition of done, [AGENTS.md](https://github.com/Corner324/PageTrace/blob/main/AGENTS.md) the same for coding agents, and the
|
|
308
|
+
rest is in [docs/architecture.md](https://github.com/Corner324/PageTrace/blob/main/docs/architecture.md), [docs/testing.md](https://github.com/Corner324/PageTrace/blob/main/docs/testing.md),
|
|
309
|
+
[docs/compatibility.md](https://github.com/Corner324/PageTrace/blob/main/docs/compatibility.md), [docs/adr/](https://github.com/Corner324/PageTrace/tree/main/docs/adr/) and
|
|
310
|
+
[CHANGELOG.md](https://github.com/Corner324/PageTrace/blob/main/CHANGELOG.md).
|
|
311
|
+
|
|
312
|
+
<div align="center">
|
|
313
|
+
|
|
314
|
+
**MIT** · [LICENSE](https://github.com/Corner324/PageTrace/blob/main/LICENSE)
|
|
315
|
+
|
|
316
|
+
</div>
|
|
@@ -0,0 +1,294 @@
|
|
|
1
|
+
<div align="center">
|
|
2
|
+
|
|
3
|
+
<img src="https://raw.githubusercontent.com/Corner324/PageTrace/main/docs/assets/header.svg" alt="PageTrace" width="720">
|
|
4
|
+
|
|
5
|
+
**English** · [Русский](https://github.com/Corner324/PageTrace/blob/main/README.ru.md) · [中文](https://github.com/Corner324/PageTrace/blob/main/README.zh.md)
|
|
6
|
+
|
|
7
|
+
**Prove what your API pagination actually did. Offline, from a trace you already have.**
|
|
8
|
+
|
|
9
|
+
[](https://github.com/Corner324/PageTrace/actions/workflows/ci.yml)
|
|
10
|
+
[](https://github.com/Corner324/PageTrace/blob/main/docs/testing.md)
|
|
11
|
+
[](https://github.com/Corner324/PageTrace/blob/main/docs/testing.md)
|
|
12
|
+
[](https://github.com/Corner324/PageTrace/blob/main/docs/security.md)
|
|
13
|
+
[](https://github.com/Corner324/PageTrace/blob/main/docs/compatibility.md)
|
|
14
|
+
[](https://github.com/Corner324/PageTrace/blob/main/LICENSE)
|
|
15
|
+
|
|
16
|
+
[⚡ Install](#-install) · [🔍 How it works](#-how-it-works) · [🎯 Presets](#-presets) · [🧭 What it will not tell you](#-what-it-will-not-tell-you) · [🔒 Security](#-security) · [📐 Invariants](https://github.com/Corner324/PageTrace/blob/main/specs/invariants.md)
|
|
17
|
+
|
|
18
|
+
</div>
|
|
19
|
+
|
|
20
|
+
---
|
|
21
|
+
|
|
22
|
+
## 💡 Why this exists
|
|
23
|
+
|
|
24
|
+
Pagination fails quietly. Every page returns `200 OK`, the loop terminates, nothing is logged as
|
|
25
|
+
an error, and the export is short by four hundred rows. Nobody notices until someone reconciles
|
|
26
|
+
a total a month later, and by then the capture is gone.
|
|
27
|
+
|
|
28
|
+
The usual tools are the wrong shape for it. An integration test hits the live API and goes flaky.
|
|
29
|
+
Logs show you individual requests, never the relationship between them - and the bug always lives
|
|
30
|
+
in the relationship: page 3 asked for a token page 2 never handed out. Reading a HAR by hand works
|
|
31
|
+
right up to the twelfth page.
|
|
32
|
+
|
|
33
|
+
PageTrace takes the recording you already have and reports what it proves.
|
|
34
|
+
|
|
35
|
+
| You have heard | PageTrace answers, from the trace alone |
|
|
36
|
+
| -------------- | -------------------------------------- |
|
|
37
|
+
| "The export is missing rows" | whether a page was requested with a token the previous page never returned, whether the sequence revisited a state, whether the capture just stopped early |
|
|
38
|
+
| "The job runs forever" | whether the cursor stalled on one value, or the request sequence returned to a state it had already visited |
|
|
39
|
+
| "We get duplicates downstream" | which item identity appeared on which two pages |
|
|
40
|
+
| "Their API is broken" | which side the trace can actually attribute it to, which is often neither |
|
|
41
|
+
|
|
42
|
+
> [!IMPORTANT]
|
|
43
|
+
> **The one rule: never claim more than the trace proves.**
|
|
44
|
+
> Every finding carries a certainty and the exact observations behind it. There are no confidence
|
|
45
|
+
> scores anywhere in this tool. When the evidence is not there, it says
|
|
46
|
+
> `INSUFFICIENT_EVIDENCE` and names the flag you need to set, instead of guessing.
|
|
47
|
+
|
|
48
|
+
Three places it earns its keep: pinned to a regression test in CI, where a proven defect fails the
|
|
49
|
+
build; in your hand during an incident, where a colleague's HAR is the only evidence left; and
|
|
50
|
+
pointed at a vendor before you integrate, where you need the claim to survive being quoted back
|
|
51
|
+
at you.
|
|
52
|
+
|
|
53
|
+
---
|
|
54
|
+
|
|
55
|
+
## ⚡ See it in 30 seconds
|
|
56
|
+
|
|
57
|
+
```bash
|
|
58
|
+
uv run pagetrace check examples/broken-cursor.jsonl
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
```
|
|
62
|
+
broken-cursor.jsonl: 3 pages, preset cursor
|
|
63
|
+
|
|
64
|
+
PROVEN ERROR CURSOR_CONTINUATION_MISMATCH [INV-CURSOR-001] pages 1, 2
|
|
65
|
+
Page 2 was requested with a continuation token that page 1 did not return.
|
|
66
|
+
page[1].response.next_cursor = "eyJvIjo0fQ"
|
|
67
|
+
page[2].request.cursor = "eyJvIjozfQ"
|
|
68
|
+
|
|
69
|
+
PROVEN WARNING DUPLICATE_ITEM_ACROSS_PAGES [INV-ITEM-001] pages 1, 2
|
|
70
|
+
An item observed on page 1 was observed again on page 2; its identity is in the evidence below. That the same item appears twice is a fact; whether it means overlapping pages depends on a uniqueness contract this trace does not contain.
|
|
71
|
+
page[1].items[0].id = 3
|
|
72
|
+
page[2].items[0].id = 3
|
|
73
|
+
|
|
74
|
+
2 proven, 0 suspicious, 0 insufficient evidence
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
Exit code 1. Note what the second finding refuses to do: it reports the duplicate as a fact and
|
|
78
|
+
declines to call it a pagination bug, because no trace can carry a uniqueness contract. That is
|
|
79
|
+
the whole design in one message.
|
|
80
|
+
|
|
81
|
+
`examples/clean-cursor.jsonl` is the same run without the defects, and exits 0.
|
|
82
|
+
|
|
83
|
+
---
|
|
84
|
+
|
|
85
|
+
## 🔍 How it works
|
|
86
|
+
|
|
87
|
+
<div align="center">
|
|
88
|
+
<picture>
|
|
89
|
+
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/Corner324/PageTrace/main/docs/assets/flow.en.dark.svg">
|
|
90
|
+
<img alt="Recorded trace, parser, normalizer, invariant engine, report. No network call at any step." src="https://raw.githubusercontent.com/Corner324/PageTrace/main/docs/assets/flow.en.light.svg" width="820">
|
|
91
|
+
</picture>
|
|
92
|
+
</div>
|
|
93
|
+
|
|
94
|
+
The important part is structural, not a promise. A detector receives normalized observations and
|
|
95
|
+
nothing else: `Page` has no field for a raw body or an `authorization` header, so the engine
|
|
96
|
+
physically cannot read one. See [docs/architecture.md](https://github.com/Corner324/PageTrace/blob/main/docs/architecture.md).
|
|
97
|
+
|
|
98
|
+
---
|
|
99
|
+
|
|
100
|
+
## ⚡ Install
|
|
101
|
+
|
|
102
|
+
> [!NOTE]
|
|
103
|
+
> Installing from PyPI is not available yet. The release workflow publishes through Trusted
|
|
104
|
+
> Publishing over OIDC with no token fallback, so nothing can be published until a trusted
|
|
105
|
+
> publisher is configured for the project and the `pypi` environment
|
|
106
|
+
> ([docs/compatibility.md](https://github.com/Corner324/PageTrace/blob/main/docs/compatibility.md)).
|
|
107
|
+
|
|
108
|
+
From a clone, with [uv](https://docs.astral.sh/uv/):
|
|
109
|
+
|
|
110
|
+
```bash
|
|
111
|
+
uv sync
|
|
112
|
+
uv run pagetrace check examples/broken-cursor.jsonl
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
With pip into an environment of your own, or `pipx install .` for a standalone command:
|
|
116
|
+
|
|
117
|
+
```bash
|
|
118
|
+
pip install .
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
---
|
|
122
|
+
|
|
123
|
+
## 📼 Record a trace
|
|
124
|
+
|
|
125
|
+
A JSONL trace is one JSON object per line, one per request/response pair, in capture order:
|
|
126
|
+
|
|
127
|
+
```json
|
|
128
|
+
{"v": 1, "seq": 0, "request": {"method": "GET", "url": "https://api.example.com/v1/items?limit=2"}, "response": {"status": 200, "headers": {"content-type": "application/json"}, "body": {"items": [{"id": 1, "name": "item-1"}, {"id": 2, "name": "item-2"}], "next_cursor": "eyJvIjoyfQ", "has_more": true, "total": 5}}}
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
`v` and `response.status` are required; `seq` is optional and line order is authoritative. Full
|
|
132
|
+
rules: [specs/formats/jsonl-v1.md](https://github.com/Corner324/PageTrace/blob/main/specs/formats/jsonl-v1.md).
|
|
133
|
+
|
|
134
|
+
<details>
|
|
135
|
+
<summary><b>Or export a HAR from the browser</b> - no code to write, works against anyone's API</summary>
|
|
136
|
+
|
|
137
|
+
<br>
|
|
138
|
+
|
|
139
|
+
Open DevTools, go to the Network tab, reproduce the pagination, export the request list as HAR.
|
|
140
|
+
A browser capture holds every asset the page loaded, so filter it down to the API calls:
|
|
141
|
+
|
|
142
|
+
```bash
|
|
143
|
+
uv run pagetrace check your-capture.har --url-filter /v1/items
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
`OPTIONS` and `HEAD` requests are dropped before pages are numbered, so a CORS preflight never
|
|
147
|
+
looks like a client that failed to advance.
|
|
148
|
+
|
|
149
|
+
</details>
|
|
150
|
+
|
|
151
|
+
> [!WARNING]
|
|
152
|
+
> A HAR holds live credentials. PageTrace keeps exactly one response header and never renders a
|
|
153
|
+
> body, but the file on your disk still contains your `Authorization` header - treat it like a
|
|
154
|
+
> password, and do not attach it to a public issue.
|
|
155
|
+
|
|
156
|
+
---
|
|
157
|
+
|
|
158
|
+
## 🎯 Presets
|
|
159
|
+
|
|
160
|
+
Selectors are [RFC 9535](https://www.rfc-editor.org/rfc/rfc9535.html) JSONPath. A preset default
|
|
161
|
+
that does not match resolves to "not found" and proves nothing, rather than becoming a guess.
|
|
162
|
+
|
|
163
|
+
| Preset | Default selectors | Fits |
|
|
164
|
+
| ------ | ----------------- | ---- |
|
|
165
|
+
| `cursor` <sub>default</sub> | `--cursor $.query.cursor` `--next $.next_cursor` `--items $.items` `--id $.id` `--total $.total` `--has-more $.has_more` | APIs that hand back an opaque continuation token |
|
|
166
|
+
| `page` | `--page $.query.page`, plus the same items / id / total / has-more defaults | page numbers, 0-based or 1-based |
|
|
167
|
+
| `offset` | `--offset $.query.offset`, plus the same items / id / total / has-more defaults | an offset and a limit |
|
|
168
|
+
| `link` | items / id / total / has-more only; continuation comes from the HTTP `Link` header | `rel="next"` in a Link header (RFC 8288) |
|
|
169
|
+
|
|
170
|
+
> [!TIP]
|
|
171
|
+
> Every default is overridable by flag, and `uv run pagetrace check --help` prints all of them
|
|
172
|
+
> with the preset each belongs to. If a finding says `INSUFFICIENT_EVIDENCE`, the flag to set is
|
|
173
|
+
> named in the message.
|
|
174
|
+
|
|
175
|
+
---
|
|
176
|
+
|
|
177
|
+
## 📊 Certainty, severity, exit codes
|
|
178
|
+
|
|
179
|
+
Two independent axes. Certainty is what the evidence supports; severity is how much it matters if
|
|
180
|
+
true. A finding is never a probability.
|
|
181
|
+
|
|
182
|
+
| Certainty | Meaning |
|
|
183
|
+
| --------- | ------- |
|
|
184
|
+
| `PROVEN` | the trace itself shows it, with the preconditions satisfied |
|
|
185
|
+
| `SUSPICIOUS` | consistent with a defect **and** with an innocent explanation |
|
|
186
|
+
| `INSUFFICIENT_EVIDENCE` | the selector or the trace did not supply what the check needs |
|
|
187
|
+
|
|
188
|
+
| Exit | Meaning |
|
|
189
|
+
| ---- | ------- |
|
|
190
|
+
| `0` | nothing both proven and severe |
|
|
191
|
+
| `1` | at least one `PROVEN` finding of severity `ERROR` |
|
|
192
|
+
| `2` | invalid input or configuration |
|
|
193
|
+
|
|
194
|
+
Exit 1 needs both axes. A duplicate item across pages is `PROVEN WARNING` and does not fail a
|
|
195
|
+
build: the finding declines to say whether overlapping windows are a bug, so failing on it would
|
|
196
|
+
be the tool claiming what its own message refuses to. A build that goes red on "this might be a
|
|
197
|
+
cache" trains people to ignore the tool.
|
|
198
|
+
|
|
199
|
+
For CI, take the machine formats rather than the terminal text, which is not a contract:
|
|
200
|
+
|
|
201
|
+
```bash
|
|
202
|
+
uv run pagetrace check examples/broken-cursor.jsonl --format json --output report.json
|
|
203
|
+
```
|
|
204
|
+
|
|
205
|
+
---
|
|
206
|
+
|
|
207
|
+
## 🧭 What it will not tell you
|
|
208
|
+
|
|
209
|
+
This section is the point of the project, not a disclaimer.
|
|
210
|
+
|
|
211
|
+
- **That pagination never terminates.** A finite trace can prove the sequence revisited a state,
|
|
212
|
+
and that is what `INV-CURSOR-003` says. A cycle back to the page-0 state after a `null` cursor
|
|
213
|
+
drops to `SUSPICIOUS`, because two concatenated runs look exactly the same.
|
|
214
|
+
- **That records are missing.** `INV-TOTAL-001` compares a declared total against observed
|
|
215
|
+
identities and is never worded as "N records are missing": PageTrace does not know that `total`
|
|
216
|
+
counts the same population, that the dataset held still, or that the same filters applied.
|
|
217
|
+
- **That the API is at fault.** `INV-CONT-001` and `INV-TRACE-001` report that the trace ended
|
|
218
|
+
while the server was still offering more, and name neither the API, the client, nor the recorder.
|
|
219
|
+
- **Anything about a duplicate beyond the duplicate itself**, without a uniqueness contract the
|
|
220
|
+
trace does not contain.
|
|
221
|
+
- **Anything from a page it could not read.** Non-2xx responses are excluded from every
|
|
222
|
+
body-derived check and reported once by `INV-TRACE-002`, so a trace nobody could read never
|
|
223
|
+
looks clean.
|
|
224
|
+
- **Anything a selector did not resolve.** Matching nothing, matching twice, or landing on an
|
|
225
|
+
object is `INSUFFICIENT_EVIDENCE` naming the flag, never a guess. JSON `null` is a different
|
|
226
|
+
state: an observation meaning "the server declared no continuation".
|
|
227
|
+
- **That `5` and `"5"` differ** - comparison is on the canonical text form, because a query
|
|
228
|
+
parameter can only carry text - or that a truthy `1` or `"true"` means `has_more`. Only a
|
|
229
|
+
boolean counts.
|
|
230
|
+
|
|
231
|
+
Every claim and every non-claim condition is in [specs/invariants.md](https://github.com/Corner324/PageTrace/blob/main/specs/invariants.md), the
|
|
232
|
+
normative contract for the tool. Eleven invariants, each with written preconditions.
|
|
233
|
+
|
|
234
|
+
---
|
|
235
|
+
|
|
236
|
+
## 🔒 Security
|
|
237
|
+
|
|
238
|
+
A HAR or JSONL trace is an untrusted, secret-bearing file. Parsers keep exactly one response
|
|
239
|
+
header, `link`, because exactly one is read; `authorization`, `cookie` and everything else is
|
|
240
|
+
dropped before the model is built. Raw bodies never reach a detector or a renderer, so only the
|
|
241
|
+
scalars you selected are displayed, with control characters escaped and lengths truncated. Input
|
|
242
|
+
is bounded per line and per decoded body, exceeding a limit is a usage error rather than a
|
|
243
|
+
traceback, and the package contains no networking dependency or code path at all. Threat table
|
|
244
|
+
and limits: [docs/security.md](https://github.com/Corner324/PageTrace/blob/main/docs/security.md).
|
|
245
|
+
|
|
246
|
+
---
|
|
247
|
+
|
|
248
|
+
## 📈 Performance
|
|
249
|
+
|
|
250
|
+
Measured 2026-09-11, one run per shape on a Windows 11 developer machine with CPython 3.12.6;
|
|
251
|
+
memory is peak Python allocation from `tracemalloc`. The shape of the numbers is the finding, not
|
|
252
|
+
the third digit.
|
|
253
|
+
|
|
254
|
+
| Shape | Pages | Items | Seconds | Peak MiB |
|
|
255
|
+
| ----- | ----: | ----: | ------: | -------: |
|
|
256
|
+
| wide | 1 000 | 100 000 | 2.6 | 16.1 |
|
|
257
|
+
| very wide | 1 000 | 1 000 000 | 20.9 | 156.8 |
|
|
258
|
+
| ten thousand pages | 10 000 | 1 000 000 | 28.3 | 142.8 |
|
|
259
|
+
| hundred thousand pages | 100 000 | 5 000 000 | 120.8 | 715.4 |
|
|
260
|
+
|
|
261
|
+
> [!NOTE]
|
|
262
|
+
> Three time targets are missed and stay written down as targets rather than quietly relaxed:
|
|
263
|
+
> 1k pages / 100k ids wants 2 s and takes 2.6 s; 10k / 1M wants 10 s and takes 28.3 s, about 3x
|
|
264
|
+
> over; the 100k / 5M stretch target wants 60 s and takes 120.8 s, about 2x over. Both memory
|
|
265
|
+
> targets are met with room to spare. Full table and reasoning:
|
|
266
|
+
> [docs/performance.md](https://github.com/Corner324/PageTrace/blob/main/docs/performance.md).
|
|
267
|
+
|
|
268
|
+
---
|
|
269
|
+
|
|
270
|
+
## 🛠 Contributing
|
|
271
|
+
|
|
272
|
+
```bash
|
|
273
|
+
uv run ruff check . && uv run ruff format --check . && uv run pyright && uv run pytest
|
|
274
|
+
```
|
|
275
|
+
|
|
276
|
+
425 tests pass locally at 98.97% branch coverage, on 3.11, 3.12, 3.13 and 3.14 on Windows. The
|
|
277
|
+
property suite in `tests/property/` is the main correctness layer: a valid generated trace must
|
|
278
|
+
produce no `PROVEN` finding, and each mutation operator must wake exactly one detector. Mutation
|
|
279
|
+
score is 77.9% against the unit tests as of 2026-09-12, with the survivors sorted by hand in
|
|
280
|
+
[docs/testing.md](https://github.com/Corner324/PageTrace/blob/main/docs/testing.md); mutmut refuses to run on Windows, so that run came from a
|
|
281
|
+
Linux container.
|
|
282
|
+
|
|
283
|
+
Before a new detector: an entry in [specs/invariants.md](https://github.com/Corner324/PageTrace/blob/main/specs/invariants.md) with its non-claim
|
|
284
|
+
conditions, then the test oracle, then the code. [CONTRIBUTING.md](https://github.com/Corner324/PageTrace/blob/main/CONTRIBUTING.md) has the
|
|
285
|
+
workflow and the definition of done, [AGENTS.md](https://github.com/Corner324/PageTrace/blob/main/AGENTS.md) the same for coding agents, and the
|
|
286
|
+
rest is in [docs/architecture.md](https://github.com/Corner324/PageTrace/blob/main/docs/architecture.md), [docs/testing.md](https://github.com/Corner324/PageTrace/blob/main/docs/testing.md),
|
|
287
|
+
[docs/compatibility.md](https://github.com/Corner324/PageTrace/blob/main/docs/compatibility.md), [docs/adr/](https://github.com/Corner324/PageTrace/tree/main/docs/adr/) and
|
|
288
|
+
[CHANGELOG.md](https://github.com/Corner324/PageTrace/blob/main/CHANGELOG.md).
|
|
289
|
+
|
|
290
|
+
<div align="center">
|
|
291
|
+
|
|
292
|
+
**MIT** · [LICENSE](https://github.com/Corner324/PageTrace/blob/main/LICENSE)
|
|
293
|
+
|
|
294
|
+
</div>
|