chunkedte 0.2.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- chunkedte-0.2.0/.gitattributes +1 -0
- chunkedte-0.2.0/.github/FUNDING.yml +1 -0
- chunkedte-0.2.0/.gitignore +16 -0
- chunkedte-0.2.0/CHANGELOG.md +42 -0
- chunkedte-0.2.0/CONTRIBUTING.md +19 -0
- chunkedte-0.2.0/IMPROVEMENTS-round.md +144 -0
- chunkedte-0.2.0/LICENSE +21 -0
- chunkedte-0.2.0/PKG-INFO +198 -0
- chunkedte-0.2.0/README.md +162 -0
- chunkedte-0.2.0/pyproject.toml +71 -0
- chunkedte-0.2.0/requirements-dev.txt +3 -0
- chunkedte-0.2.0/scripts/check.ps1 +16 -0
- chunkedte-0.2.0/scripts/fetch_upstream.py +37 -0
- chunkedte-0.2.0/scripts/release.ps1 +28 -0
- chunkedte-0.2.0/src/chunkedte/__init__.py +22 -0
- chunkedte-0.2.0/src/chunkedte/__main__.py +9 -0
- chunkedte-0.2.0/src/chunkedte/_cli.py +87 -0
- chunkedte-0.2.0/src/chunkedte/_decode.py +177 -0
- chunkedte-0.2.0/src/chunkedte/_encode.py +82 -0
- chunkedte-0.2.0/src/chunkedte/_errors.py +11 -0
- chunkedte-0.2.0/src/chunkedte/_framing.py +66 -0
- chunkedte-0.2.0/src/chunkedte/_grammar.py +85 -0
- chunkedte-0.2.0/src/chunkedte/_limits.py +33 -0
- chunkedte-0.2.0/src/chunkedte/py.typed +0 -0
- chunkedte-0.2.0/tests/conftest.py +36 -0
- chunkedte-0.2.0/tests/reference.py +123 -0
- chunkedte-0.2.0/tests/test_cli.py +113 -0
- chunkedte-0.2.0/tests/test_contract.py +40 -0
- chunkedte-0.2.0/tests/test_decode.py +121 -0
- chunkedte-0.2.0/tests/test_docs.py +85 -0
- chunkedte-0.2.0/tests/test_e2e_wheel.py +85 -0
- chunkedte-0.2.0/tests/test_encode.py +133 -0
- chunkedte-0.2.0/tests/test_framing.py +105 -0
- chunkedte-0.2.0/tests/test_fuzz.py +98 -0
- chunkedte-0.2.0/tests/test_limits.py +142 -0
- chunkedte-0.2.0/tests/test_strict.py +161 -0
- chunkedte-0.2.0/tests/test_typing.py +77 -0
- chunkedte-0.2.0/tests/test_upstream.py +158 -0
- chunkedte-0.2.0/tests/upstream/MANIFEST.json +20 -0
|
@@ -0,0 +1 @@
|
|
|
1
|
+
* text=auto eol=lf
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
buy_me_a_coffee: jlianesglrs
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
__pycache__/
|
|
2
|
+
*.py[cod]
|
|
3
|
+
.venv/
|
|
4
|
+
venv/
|
|
5
|
+
build/
|
|
6
|
+
dist/
|
|
7
|
+
*.egg-info/
|
|
8
|
+
.pytest_cache/
|
|
9
|
+
.mypy_cache/
|
|
10
|
+
.ruff_cache/
|
|
11
|
+
.coverage
|
|
12
|
+
htmlcov/
|
|
13
|
+
|
|
14
|
+
# Third-party reference files, fetched locally by scripts/fetch_upstream.py; only the manifest is tracked.
|
|
15
|
+
tests/upstream/*
|
|
16
|
+
!tests/upstream/MANIFEST.json
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project are documented here. The format follows
|
|
4
|
+
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the project uses
|
|
5
|
+
[Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
6
|
+
|
|
7
|
+
## [Unreleased]
|
|
8
|
+
|
|
9
|
+
## [0.2.0]
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
|
|
13
|
+
- `Limits` and `LimitError`: bounds on one size line, all chunk extensions of a message,
|
|
14
|
+
the chunk size (below 2**63 by default), the body and the trailer section.
|
|
15
|
+
- `Decoder.trailers` returns the trailer fields in arrival order; `Decoder.unused_data`
|
|
16
|
+
holds the bytes after the message (a pipelined request, for example).
|
|
17
|
+
- `Encoder` streams chunks and closes the message with optional trailer fields; `encode`
|
|
18
|
+
takes `trailers` too. Trailer names, values and framing fields are validated.
|
|
19
|
+
- `is_chunked` decides from Transfer-Encoding and Content-Length whether a body is chunked
|
|
20
|
+
(RFC 9112 sections 6.1 and 6.3) and rejects the combinations used for smuggling.
|
|
21
|
+
- `chunkedte decode` and `chunkedte encode` on the command line, also `python -m chunkedte`.
|
|
22
|
+
- `decode`, `Decoder.feed` and `encode` accept `bytearray` and `memoryview`.
|
|
23
|
+
|
|
24
|
+
### Changed
|
|
25
|
+
|
|
26
|
+
- The decoder follows RFC 9112 section 7.1 strictly: the size is hex digits only (no `+`,
|
|
27
|
+
`0x`, `_` or whitespace), extensions must match the grammar, only CRLF ends a line (bare
|
|
28
|
+
LF and bare CR are errors), exactly CRLF must follow chunk data, and trailer lines must be
|
|
29
|
+
valid field lines without obs-fold.
|
|
30
|
+
- `decode` raises when bytes follow the message; bytes fed to a finished `Decoder` go to
|
|
31
|
+
`unused_data` instead of raising.
|
|
32
|
+
- A `Decoder` that raised keeps raising on every later `feed`.
|
|
33
|
+
- Lines split over many reads are scanned once, so decoding is linear in the input.
|
|
34
|
+
- `encode` rejects a non-int `chunk_size` and non bytes-like data with `TypeError`.
|
|
35
|
+
|
|
36
|
+
## [0.1.0]
|
|
37
|
+
|
|
38
|
+
- `encode` frames a byte string as a chunked body split into fixed-size pieces.
|
|
39
|
+
- `Decoder` incrementally decodes chunked bytes fed in any split, ignoring chunk
|
|
40
|
+
extensions and trailer fields (RFC 7230 section 4.1).
|
|
41
|
+
- `decode` decodes a complete message in one call.
|
|
42
|
+
- `ChunkedError` is raised on a malformed size line or broken framing.
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
# Contributing
|
|
2
|
+
|
|
3
|
+
Thanks for taking the time. This project keeps a small, predictable shape.
|
|
4
|
+
|
|
5
|
+
## Setup
|
|
6
|
+
|
|
7
|
+
```
|
|
8
|
+
pip install -r requirements-dev.txt
|
|
9
|
+
```
|
|
10
|
+
|
|
11
|
+
## Before you open a pull request
|
|
12
|
+
|
|
13
|
+
- Keep public API in `src/chunkedte/__init__.py`; internals live in `_`-prefixed modules.
|
|
14
|
+
- Add a test for the behaviour you change. Tests inject time, disk and network; they
|
|
15
|
+
never touch the real thing.
|
|
16
|
+
- Run `scripts/check.ps1` (format, lint, types, tests) and make sure it passes.
|
|
17
|
+
- One logical change per commit, imperative subject line.
|
|
18
|
+
|
|
19
|
+
Open an issue first if the change is large or shifts the public API.
|
|
@@ -0,0 +1,144 @@
|
|
|
1
|
+
# chunkedte improvement round
|
|
2
|
+
|
|
3
|
+
Progress:
|
|
4
|
+
- [x] Brainstorm ranked improvements (this file)
|
|
5
|
+
- [x] Strict chunk-size line: HEXDIG only, chunk-ext grammar, CRLF only, size cap
|
|
6
|
+
- [x] Strict framing after chunk data and in the trailer section (field-line grammar, no obs-fold)
|
|
7
|
+
- [x] Limits (`Limits`, `LimitError`): line, total extensions, chunk size, body, trailer section
|
|
8
|
+
- [x] Linear time on hostile input (no rescans of buffered bytes)
|
|
9
|
+
- [x] Trailers exposed, bytes after the message kept apart (`unused_data`), `decode` rejects them
|
|
10
|
+
- [x] Encoder: incremental `Encoder`, trailers on `encode`, field validation
|
|
11
|
+
- [x] `is_chunked` for Transfer-Encoding and Content-Length (RFC 9112 section 6.1 and 6.3)
|
|
12
|
+
- [x] Command line tool (`python -m chunkedte`)
|
|
13
|
+
- [x] Contracts: fetched RFC grammar, advisory payloads, differential fuzz, typing, README, wheel e2e
|
|
14
|
+
- [x] Mutation run (guards removed one at a time, each must fail the suite)
|
|
15
|
+
- [x] Docs: README, CHANGELOG, docstrings, packaging metadata
|
|
16
|
+
- [x] Gates: ruff format, ruff check, mypy --strict, pytest
|
|
17
|
+
- [x] Independent review findings fixed with regression tests
|
|
18
|
+
- [x] Pull request opened (#2), guard clean
|
|
19
|
+
|
|
20
|
+
## Starting point
|
|
21
|
+
|
|
22
|
+
`0.1.0` (not on PyPI) ships `encode`, `decode`, `Decoder` and `ChunkedError`. Read as an
|
|
23
|
+
HTTP server or proxy author who feeds it attacker bytes, it has these gaps:
|
|
24
|
+
|
|
25
|
+
- The size is parsed with `int(token.strip(), 16)`. Python's `int` takes `+5`, `0x5`, `5_0`
|
|
26
|
+
and surrounding whitespace, none of which is `1*HEXDIG`. This is the exact bug class of
|
|
27
|
+
the waitress advisory below: two parsers that disagree on a chunk size disagree on where
|
|
28
|
+
the next request starts.
|
|
29
|
+
- Everything after `;` is dropped unchecked, so a size line such as `5;\nxx` or `5;a b`
|
|
30
|
+
passes. Lines are split on CRLF only, so a bare LF inside an extension is swallowed, the
|
|
31
|
+
pattern of the aiohttp and netty advisories.
|
|
32
|
+
- Trailer lines are read but never checked: obs-fold, whitespace before the colon, a line
|
|
33
|
+
without a colon and control bytes all pass. Trailers are thrown away, with no way to read
|
|
34
|
+
them.
|
|
35
|
+
- No limits: an endless size line, endless extensions or an endless trailer section are
|
|
36
|
+
buffered without bound (the Node.js advisory below), and the chunk size has no cap.
|
|
37
|
+
- A line split over many small feeds is searched from the start on every feed: quadratic.
|
|
38
|
+
- Bytes after the terminator stay in the buffer silently; `decode(b"0\r\n\r\nGET /")`
|
|
39
|
+
returns `b""` with no error.
|
|
40
|
+
- The encoder cannot send trailers or stream; there is no help for the Transfer-Encoding
|
|
41
|
+
and Content-Length decision that decides whether chunked applies at all.
|
|
42
|
+
- Packaging: no keywords or Python classifiers, README examples never run, no wheel test.
|
|
43
|
+
|
|
44
|
+
## Sources (fetched 2026-10-13)
|
|
45
|
+
|
|
46
|
+
- RFC 9112, https://www.rfc-editor.org/rfc/rfc9112.txt (sha256 `e4f426ba...`): section 7.1
|
|
47
|
+
(chunked-body, chunk, chunk-size = 1*HEXDIG, last-chunk = 1*("0"), recipients must cope
|
|
48
|
+
with large numerals), 7.1.1 (chunk-ext with BWS around `;` and `=`, token or quoted-string
|
|
49
|
+
values, a server ought to limit the total extension length), 7.1.2 (trailer-section =
|
|
50
|
+
*( field-line CRLF )), 7.1.3 (decoding pseudo-code), 2.2 (CRLF; a recipient MAY accept a
|
|
51
|
+
bare LF, a bare CR is invalid), 5 and 5.1 (field-line, no whitespace before the colon),
|
|
52
|
+
5.2 (obs-fold), 6.1 and 6.3 (Transfer-Encoding overrides Content-Length, both together
|
|
53
|
+
ought to be an error, chunked must be the final coding of a request).
|
|
54
|
+
- RFC 9112 errata, https://www.rfc-editor.org/errata/rfc9112: two verified editorial errata
|
|
55
|
+
(obs-text reference, a missing changes note), none on section 7. The page also lists
|
|
56
|
+
RFC 9931 (March 2026) as updating RFC 9112; fetched from
|
|
57
|
+
https://www.rfc-editor.org/rfc/rfc9931.txt, it covers optimistic protocol upgrades, not
|
|
58
|
+
the chunked coding.
|
|
59
|
+
- RFC 9110, https://www.rfc-editor.org/rfc/rfc9110.txt (sha256 `21c1cdce...`): 5.5
|
|
60
|
+
(field-value, field-vchar, obs-text), 5.6.2 (token, tchar), 5.6.3 (OWS, BWS), 5.6.4
|
|
61
|
+
(quoted-string, qdtext, quoted-pair), 6.5 and 6.5.1 (trailer fields; fields for framing
|
|
62
|
+
and routing cannot be sent as trailers).
|
|
63
|
+
- GHSA-4f7p-27jc-3c36 (waitress, CVE-2022-24761): `int()` accepted `+10` and `0x01` as
|
|
64
|
+
sizes; extensions were dropped without validation.
|
|
65
|
+
- GHSA-vqfr-h8mv-ghfj (h11, CVE-2025-43859): any two bytes accepted after chunk data instead
|
|
66
|
+
of CRLF; the advisory's payload `5\r\nAAAAAXX2\r\n45\r\n0\r\n\r\n` smuggles a request.
|
|
67
|
+
- GHSA-8495-4g3g-x7pr (aiohttp, CVE-2024-52304) and GHSA-fghv-69vj-qj49 (netty,
|
|
68
|
+
CVE-2025-58056): a bare LF inside a chunk extension ended the size line.
|
|
69
|
+
- GHSA-g9pc-8g42-g6vq (Go net/http via RoadRunner, CVE-2025-22871): a bare LF accepted as the
|
|
70
|
+
size line terminator.
|
|
71
|
+
- GHSA-prhj-8562-p8gj (Node.js, CVE-2024-22019): unbounded chunk extension bytes across a
|
|
72
|
+
message exhausted CPU and bandwidth.
|
|
73
|
+
- GHSA-cggh-pq45-6h9x (llhttp, CVE-2023-30589): a bare CR accepted as a line terminator.
|
|
74
|
+
|
|
75
|
+
The RFC text is third-party: `scripts/fetch_upstream.py` fetches it into a gitignored
|
|
76
|
+
`tests/upstream/` folder checked against a tracked `MANIFEST.json` (URL, size, sha256,
|
|
77
|
+
licence); the tests that read it skip when it is absent. Advisory payloads are short test
|
|
78
|
+
vectors, cited with their URL next to them.
|
|
79
|
+
|
|
80
|
+
## Ranked brainstorm
|
|
81
|
+
|
|
82
|
+
1. **Strict size line** (correctness, smuggling). Size must fullmatch `1*HEXDIG`; extensions
|
|
83
|
+
must match the RFC 9112 grammar exactly (BWS, token, quoted-string); the line ends only at
|
|
84
|
+
CRLF; bare LF and bare CR are errors. No lenient switch: the advisories show that every
|
|
85
|
+
leniency here is a desync with some other parser.
|
|
86
|
+
2. **Strict data terminator and trailer section.** Exactly CRLF after chunk data, failing on
|
|
87
|
+
the first wrong byte. Trailer lines must be `token ":" OWS field-value OWS`; a line that
|
|
88
|
+
starts with SP or HTAB (obs-fold) is rejected, as are control bytes.
|
|
89
|
+
3. **Limits** with safe defaults and a distinct `LimitError` (maps to 413/431 rather than
|
|
90
|
+
400): one size line, total extension bytes per message, chunk size (default below
|
|
91
|
+
2**63 so a value never wraps in a downstream 64-bit parser), body size, trailer section.
|
|
92
|
+
4. **Linear time.** Keep a scan offset so a line split over n feeds costs O(n), and fail as
|
|
93
|
+
soon as the buffered line passes its limit instead of waiting for a CRLF.
|
|
94
|
+
5. **Trailers and pipelining.** `Decoder.trailers` returns the parsed fields in order;
|
|
95
|
+
bytes after the final CRLF go to `Decoder.unused_data` (the next pipelined message), and
|
|
96
|
+
`decode` raises if there are any.
|
|
97
|
+
6. **Encoder.** `Encoder.write` and `Encoder.finish(trailers)` for streaming; `encode` takes
|
|
98
|
+
trailers. Names must be tokens, values field-values, and framing or routing fields
|
|
99
|
+
(Content-Length, Transfer-Encoding, Trailer, Host) are refused as trailers.
|
|
100
|
+
7. **`is_chunked(transfer_encoding, content_length, request=True)`** applies RFC 9112 6.1
|
|
101
|
+
and 6.3: both fields together is an error, chunked twice or not last in a request is an
|
|
102
|
+
error, chunked with parameters is an error, a response with chunked not last is read to
|
|
103
|
+
close (returns False).
|
|
104
|
+
8. **Typed API**: bytes-like input, `Limits` frozen dataclass, `py.typed`, mypy snippet test.
|
|
105
|
+
9. **CLI**: `python -m chunkedte decode|encode` over stdin and stdout, trailers on stderr.
|
|
106
|
+
10. **Contracts**: grammar classes checked against the fetched RFC ABNF, advisory payloads,
|
|
107
|
+
a differential fuzz against an independent whole-message reference parser over random
|
|
108
|
+
mutations and random splits, README examples executed, installed-wheel e2e, mutation run.
|
|
109
|
+
11. **Packaging**: keywords, classifiers, console script, CHANGELOG, version 0.2.0.
|
|
110
|
+
|
|
111
|
+
Out of scope: content codings (gzip inside chunked), HTTP header parsing beyond the two
|
|
112
|
+
framing fields, and any lenient mode.
|
|
113
|
+
|
|
114
|
+
## Mutation run
|
|
115
|
+
|
|
116
|
+
66 hand-written mutants (one guard, bound or grammar class changed at a time) over every
|
|
117
|
+
module, each run against the whole suite with bytecode writes off, `__pycache__` purged
|
|
118
|
+
and a per-mutant timeout; the suite was green before and after. 63 were killed at once.
|
|
119
|
+
Of the three survivors, one exposed a weak test (`unused_data` before completion was
|
|
120
|
+
checked with an empty buffer) and is now killed; one was a redundant type check in
|
|
121
|
+
`is_chunked` that the regex already enforces, now removed; the last (`\Z` to `$` in the
|
|
122
|
+
Transfer-Encoding list parser) is equivalent, because the zero-length-match guard rejects
|
|
123
|
+
the trailing newline either way.
|
|
124
|
+
|
|
125
|
+
## Independent review
|
|
126
|
+
|
|
127
|
+
One medium and eight low findings, all handled:
|
|
128
|
+
|
|
129
|
+
- `is_chunked` backtracked quadratically on long whitespace runs (two adjacent OWS runs in
|
|
130
|
+
the list regex). Fixed by moving the trailing OWS inside the optional coding group;
|
|
131
|
+
regression tests with 64 KiB of spaces and tabs.
|
|
132
|
+
- Transfer-Encoding errors echoed the whole field; now cut to 64 characters, with a test.
|
|
133
|
+
- The encoder refused only framing and routing trailers; it now refuses every RFC 9110
|
|
134
|
+
section 6.5.1 category (authentication, request modifiers, response controls, content
|
|
135
|
+
format), tested.
|
|
136
|
+
- The CLI failed on a latin-1 trailer with an ASCII terminal; trailer bytes outside ASCII
|
|
137
|
+
are now escaped, tested.
|
|
138
|
+
- Documented: received trailers must stay apart from the header section, `feed` drops the
|
|
139
|
+
pieces of the call that raises, the CLI may leave a truncated body on stdout on failure,
|
|
140
|
+
and `is_chunked` leaves HEAD, 1xx, 204, 304, CONNECT and HTTP/1.0 to the caller.
|
|
141
|
+
- Kept on purpose: two wall-clock tests with generous bounds (they are the only way to
|
|
142
|
+
catch a quadratic regression, and the mutation run showed they bite), CLI tests that use
|
|
143
|
+
`tmp_path`, and Python 3.10 as the floor (the code uses `X | Y` type aliases).
|
|
144
|
+
|
chunkedte-0.2.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Javier Lianes
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
chunkedte-0.2.0/PKG-INFO
ADDED
|
@@ -0,0 +1,198 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: chunkedte
|
|
3
|
+
Version: 0.2.0
|
|
4
|
+
Summary: Strict, incremental, typed, zero-dependency codec for HTTP/1.1 chunked transfer coding
|
|
5
|
+
Project-URL: Homepage, https://lianes8studio.com
|
|
6
|
+
Project-URL: Repository, https://github.com/JaviLianes8/chunkedte
|
|
7
|
+
Project-URL: Issues, https://github.com/JaviLianes8/chunkedte/issues
|
|
8
|
+
Project-URL: Changelog, https://github.com/JaviLianes8/chunkedte/blob/main/CHANGELOG.md
|
|
9
|
+
Project-URL: LinkedIn, https://www.linkedin.com/in/jlianes/
|
|
10
|
+
Project-URL: Funding, https://buymeacoffee.com/jlianesglrs
|
|
11
|
+
Author-email: Javier Lianes <javier@lianes8studio.com>
|
|
12
|
+
License-Expression: MIT
|
|
13
|
+
License-File: LICENSE
|
|
14
|
+
Keywords: chunked,codec,http,request-smuggling,rfc9112,trailers,transfer-encoding
|
|
15
|
+
Classifier: Development Status :: 4 - Beta
|
|
16
|
+
Classifier: Intended Audience :: Developers
|
|
17
|
+
Classifier: Operating System :: OS Independent
|
|
18
|
+
Classifier: Programming Language :: Python :: 3
|
|
19
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
21
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
22
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
23
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
24
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
25
|
+
Classifier: Topic :: Internet :: WWW/HTTP
|
|
26
|
+
Classifier: Topic :: Software Development :: Libraries :: Python Modules
|
|
27
|
+
Classifier: Typing :: Typed
|
|
28
|
+
Requires-Python: >=3.10
|
|
29
|
+
Provides-Extra: dev
|
|
30
|
+
Requires-Dist: build; extra == 'dev'
|
|
31
|
+
Requires-Dist: mypy; extra == 'dev'
|
|
32
|
+
Requires-Dist: pytest; extra == 'dev'
|
|
33
|
+
Requires-Dist: ruff; extra == 'dev'
|
|
34
|
+
Requires-Dist: twine; extra == 'dev'
|
|
35
|
+
Description-Content-Type: text/markdown
|
|
36
|
+
|
|
37
|
+
# chunkedte
|
|
38
|
+
|
|
39
|
+
Strict, incremental, typed, zero-dependency codec for HTTP/1.1 chunked transfer coding
|
|
40
|
+
(RFC 9112 section 7.1).
|
|
41
|
+
|
|
42
|
+
## Install
|
|
43
|
+
|
|
44
|
+
```
|
|
45
|
+
pip install chunkedte
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
Python 3.10 or newer. No dependencies.
|
|
49
|
+
|
|
50
|
+
## Usage
|
|
51
|
+
|
|
52
|
+
Decode a complete message in one call. `decode` raises `ChunkedError` if the message is
|
|
53
|
+
malformed, unfinished, or followed by more bytes:
|
|
54
|
+
|
|
55
|
+
```python
|
|
56
|
+
from chunkedte import ChunkedError, decode
|
|
57
|
+
|
|
58
|
+
print(decode(b"5\r\nhello\r\n0\r\n\r\n"))
|
|
59
|
+
try:
|
|
60
|
+
decode(b"0x5\r\nhello\r\n0\r\n\r\n")
|
|
61
|
+
except ChunkedError as error:
|
|
62
|
+
print(error)
|
|
63
|
+
# b'hello'
|
|
64
|
+
# malformed chunk-size line: b'0x5'
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
Decode incrementally as bytes arrive off a socket. `Decoder.feed` takes any split and
|
|
68
|
+
returns the body pieces available so far; trailer fields and whatever follows the message
|
|
69
|
+
(a pipelined request, say) are kept apart:
|
|
70
|
+
|
|
71
|
+
```python
|
|
72
|
+
from chunkedte import Decoder
|
|
73
|
+
|
|
74
|
+
decoder = Decoder()
|
|
75
|
+
body = b""
|
|
76
|
+
for part in [b"4\r\nwi", b"ki\r\n5;sig=ab\r\npedia\r\n0\r\nDigest: x\r\n", b"\r\nGET /"]:
|
|
77
|
+
for piece in decoder.feed(part):
|
|
78
|
+
body += piece
|
|
79
|
+
print(body, decoder.is_complete)
|
|
80
|
+
print(decoder.trailers, decoder.unused_data)
|
|
81
|
+
# b'wikipedia' True
|
|
82
|
+
# [('Digest', 'x')] b'GET /'
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
Bound what a peer can make you buffer. Passing a bound raises `LimitError`, a subclass of
|
|
86
|
+
`ChunkedError`, so a server can answer 413 or 431 instead of 400:
|
|
87
|
+
|
|
88
|
+
```python
|
|
89
|
+
from chunkedte import Decoder, LimitError, Limits
|
|
90
|
+
|
|
91
|
+
decoder = Decoder(Limits(max_body_size=1024, max_trailer_size=4096))
|
|
92
|
+
try:
|
|
93
|
+
decoder.feed(b"10000\r\n")
|
|
94
|
+
except LimitError as error:
|
|
95
|
+
print(error)
|
|
96
|
+
print(Limits())
|
|
97
|
+
# body exceeds max_body_size
|
|
98
|
+
# Limits(max_line_size=4096, max_extensions_size=16384, max_chunk_size=9223372036854775807, max_body_size=None, max_trailer_size=16384)
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
Encode in one call, or stream with `Encoder` and close with optional trailer fields:
|
|
102
|
+
|
|
103
|
+
```python
|
|
104
|
+
from chunkedte import Encoder, encode
|
|
105
|
+
|
|
106
|
+
print(encode(b"abcd", chunk_size=2))
|
|
107
|
+
encoder = Encoder()
|
|
108
|
+
print(encoder.write(b"hello") + encoder.finish({"X-Checksum": "42"}))
|
|
109
|
+
# b'2\r\nab\r\n2\r\ncd\r\n0\r\n\r\n'
|
|
110
|
+
# b'5\r\nhello\r\n0\r\nX-Checksum: 42\r\n\r\n'
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
Decide whether a message body is chunked at all from its Transfer-Encoding and
|
|
114
|
+
Content-Length values (`None` when absent). `is_chunked` applies RFC 9112 sections 6.1 and
|
|
115
|
+
6.3 and raises `ChunkedError` on the combinations used for request smuggling:
|
|
116
|
+
|
|
117
|
+
```python
|
|
118
|
+
from chunkedte import ChunkedError, is_chunked
|
|
119
|
+
|
|
120
|
+
print(is_chunked("gzip, chunked"), is_chunked(None, "12"))
|
|
121
|
+
print(is_chunked("gzip", request=False))
|
|
122
|
+
for te, cl in [("chunked", "12"), ("chunked, chunked", None), ("chunked, gzip", None)]:
|
|
123
|
+
try:
|
|
124
|
+
is_chunked(te, cl)
|
|
125
|
+
except ChunkedError as error:
|
|
126
|
+
print(error)
|
|
127
|
+
# True False
|
|
128
|
+
# False
|
|
129
|
+
# both Transfer-Encoding and Content-Length are present
|
|
130
|
+
# chunked is applied more than once
|
|
131
|
+
# chunked is not the final transfer coding of a request
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
### Command line
|
|
135
|
+
|
|
136
|
+
```
|
|
137
|
+
$ printf '5\r\nhello\r\n0\r\nX-Sum: 1\r\n\r\n' | chunkedte decode
|
|
138
|
+
hello
|
|
139
|
+
X-Sum: 1
|
|
140
|
+
$ printf 'abc' | chunkedte encode --chunk-size 2 --trailer "X-Sum: 1"
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
`decode` writes the body to stdout and the trailer fields to stderr; `encode` frames stdin
|
|
144
|
+
(or a file) as a chunked body. Both exit with status 1 and a message on bad input; body
|
|
145
|
+
bytes decoded before the error may already be on stdout, so treat the output of a failed
|
|
146
|
+
run as truncated. Trailer bytes outside printable ASCII are written as escapes.
|
|
147
|
+
`python -m chunkedte` works too.
|
|
148
|
+
|
|
149
|
+
## What "strict" means here
|
|
150
|
+
|
|
151
|
+
Every leniency in a chunked parser is a way for two HTTP processors to disagree on where a
|
|
152
|
+
message ends, which is how request smuggling works. So the decoder rejects:
|
|
153
|
+
|
|
154
|
+
- a chunk size that is not only hex digits (`+5`, `0x5`, `5_0`, ` 5`, `5 `);
|
|
155
|
+
- chunk extensions that do not follow the grammar (whitespace is allowed only around `;`
|
|
156
|
+
and `=`, values are tokens or quoted strings);
|
|
157
|
+
- a bare LF or a bare CR anywhere a CRLF is expected, including inside an extension;
|
|
158
|
+
- anything other than CRLF right after the chunk data;
|
|
159
|
+
- trailer lines that are not `name: value`, with whitespace before the colon, with control
|
|
160
|
+
bytes, or folded onto a continuation line (obs-fold).
|
|
161
|
+
|
|
162
|
+
Extensions are validated and then ignored, as RFC 9112 asks. The default `Limits` cap one
|
|
163
|
+
size line at 4 KiB, all extensions of a message at 16 KiB, the trailer section at 16 KiB and
|
|
164
|
+
a chunk size below 2**63 (so a forwarded size never wraps in a 64-bit parser); the body is
|
|
165
|
+
unbounded unless you set `max_body_size`. A line split over many small reads is scanned
|
|
166
|
+
once, so decoding stays linear in the input. When `feed` raises, the pieces decoded in that
|
|
167
|
+
call are dropped with it. Received trailers are returned as they came: RFC 9110 section
|
|
168
|
+
6.5.2 says not to merge them into the header section, so keep them apart (a `Content-Length`
|
|
169
|
+
trailer, say, must never reach your framing logic). The encoder refuses trailer names that
|
|
170
|
+
are not tokens, values with control bytes or edge whitespace, and the fields RFC 9110
|
|
171
|
+
section 6.5.1 needs before the content: framing and routing (Content-Length,
|
|
172
|
+
Transfer-Encoding, Trailer, TE, Host, Connection, Keep-Alive, Upgrade, Max-Forwards),
|
|
173
|
+
authentication (Authorization, Proxy-Authorization, WWW-Authenticate, Proxy-Authenticate,
|
|
174
|
+
Cookie, Set-Cookie), request modifiers and response controls (Expect, Range,
|
|
175
|
+
Cache-Control) and content format (Content-Type, Content-Encoding, Content-Range).
|
|
176
|
+
`is_chunked` leaves the rest of RFC 9112 section 6.3 to you: responses to HEAD, 1xx, 204,
|
|
177
|
+
304 and 2xx to CONNECT carry no body, and an HTTP/1.0 message with Transfer-Encoding has
|
|
178
|
+
faulty framing. There is no lenient mode.
|
|
179
|
+
|
|
180
|
+
## Why
|
|
181
|
+
|
|
182
|
+
Chunked transfer coding is simple to frame and easy to get subtly wrong: the terminating
|
|
183
|
+
`0` chunk, the closing blank line, extensions and trailers are routinely mishandled, and
|
|
184
|
+
the security advisories of the last few years (waitress, h11, aiohttp, netty, Go net/http,
|
|
185
|
+
Node.js) are all variations on a parser accepting bytes another parser reads differently.
|
|
186
|
+
The existing Python options are tied to one HTTP stack (`h11`, `aiohttp`) or to one server.
|
|
187
|
+
This is a standalone, typed, zero-dependency codec with the strict behaviour by default:
|
|
188
|
+
reach for it when you move chunked framing yourself and do not want a whole HTTP stack.
|
|
189
|
+
|
|
190
|
+
## License
|
|
191
|
+
|
|
192
|
+
MIT. See [LICENSE](LICENSE).
|
|
193
|
+
|
|
194
|
+
## Links
|
|
195
|
+
|
|
196
|
+
- Website: https://lianes8studio.com
|
|
197
|
+
- GitHub: https://github.com/JaviLianes8
|
|
198
|
+
- LinkedIn: https://www.linkedin.com/in/jlianes/
|
|
@@ -0,0 +1,162 @@
|
|
|
1
|
+
# chunkedte
|
|
2
|
+
|
|
3
|
+
Strict, incremental, typed, zero-dependency codec for HTTP/1.1 chunked transfer coding
|
|
4
|
+
(RFC 9112 section 7.1).
|
|
5
|
+
|
|
6
|
+
## Install
|
|
7
|
+
|
|
8
|
+
```
|
|
9
|
+
pip install chunkedte
|
|
10
|
+
```
|
|
11
|
+
|
|
12
|
+
Python 3.10 or newer. No dependencies.
|
|
13
|
+
|
|
14
|
+
## Usage
|
|
15
|
+
|
|
16
|
+
Decode a complete message in one call. `decode` raises `ChunkedError` if the message is
|
|
17
|
+
malformed, unfinished, or followed by more bytes:
|
|
18
|
+
|
|
19
|
+
```python
|
|
20
|
+
from chunkedte import ChunkedError, decode
|
|
21
|
+
|
|
22
|
+
print(decode(b"5\r\nhello\r\n0\r\n\r\n"))
|
|
23
|
+
try:
|
|
24
|
+
decode(b"0x5\r\nhello\r\n0\r\n\r\n")
|
|
25
|
+
except ChunkedError as error:
|
|
26
|
+
print(error)
|
|
27
|
+
# b'hello'
|
|
28
|
+
# malformed chunk-size line: b'0x5'
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
Decode incrementally as bytes arrive off a socket. `Decoder.feed` takes any split and
|
|
32
|
+
returns the body pieces available so far; trailer fields and whatever follows the message
|
|
33
|
+
(a pipelined request, say) are kept apart:
|
|
34
|
+
|
|
35
|
+
```python
|
|
36
|
+
from chunkedte import Decoder
|
|
37
|
+
|
|
38
|
+
decoder = Decoder()
|
|
39
|
+
body = b""
|
|
40
|
+
for part in [b"4\r\nwi", b"ki\r\n5;sig=ab\r\npedia\r\n0\r\nDigest: x\r\n", b"\r\nGET /"]:
|
|
41
|
+
for piece in decoder.feed(part):
|
|
42
|
+
body += piece
|
|
43
|
+
print(body, decoder.is_complete)
|
|
44
|
+
print(decoder.trailers, decoder.unused_data)
|
|
45
|
+
# b'wikipedia' True
|
|
46
|
+
# [('Digest', 'x')] b'GET /'
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
Bound what a peer can make you buffer. Passing a bound raises `LimitError`, a subclass of
|
|
50
|
+
`ChunkedError`, so a server can answer 413 or 431 instead of 400:
|
|
51
|
+
|
|
52
|
+
```python
|
|
53
|
+
from chunkedte import Decoder, LimitError, Limits
|
|
54
|
+
|
|
55
|
+
decoder = Decoder(Limits(max_body_size=1024, max_trailer_size=4096))
|
|
56
|
+
try:
|
|
57
|
+
decoder.feed(b"10000\r\n")
|
|
58
|
+
except LimitError as error:
|
|
59
|
+
print(error)
|
|
60
|
+
print(Limits())
|
|
61
|
+
# body exceeds max_body_size
|
|
62
|
+
# Limits(max_line_size=4096, max_extensions_size=16384, max_chunk_size=9223372036854775807, max_body_size=None, max_trailer_size=16384)
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
Encode in one call, or stream with `Encoder` and close with optional trailer fields:
|
|
66
|
+
|
|
67
|
+
```python
|
|
68
|
+
from chunkedte import Encoder, encode
|
|
69
|
+
|
|
70
|
+
print(encode(b"abcd", chunk_size=2))
|
|
71
|
+
encoder = Encoder()
|
|
72
|
+
print(encoder.write(b"hello") + encoder.finish({"X-Checksum": "42"}))
|
|
73
|
+
# b'2\r\nab\r\n2\r\ncd\r\n0\r\n\r\n'
|
|
74
|
+
# b'5\r\nhello\r\n0\r\nX-Checksum: 42\r\n\r\n'
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
Decide whether a message body is chunked at all from its Transfer-Encoding and
|
|
78
|
+
Content-Length values (`None` when absent). `is_chunked` applies RFC 9112 sections 6.1 and
|
|
79
|
+
6.3 and raises `ChunkedError` on the combinations used for request smuggling:
|
|
80
|
+
|
|
81
|
+
```python
|
|
82
|
+
from chunkedte import ChunkedError, is_chunked
|
|
83
|
+
|
|
84
|
+
print(is_chunked("gzip, chunked"), is_chunked(None, "12"))
|
|
85
|
+
print(is_chunked("gzip", request=False))
|
|
86
|
+
for te, cl in [("chunked", "12"), ("chunked, chunked", None), ("chunked, gzip", None)]:
|
|
87
|
+
try:
|
|
88
|
+
is_chunked(te, cl)
|
|
89
|
+
except ChunkedError as error:
|
|
90
|
+
print(error)
|
|
91
|
+
# True False
|
|
92
|
+
# False
|
|
93
|
+
# both Transfer-Encoding and Content-Length are present
|
|
94
|
+
# chunked is applied more than once
|
|
95
|
+
# chunked is not the final transfer coding of a request
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
### Command line
|
|
99
|
+
|
|
100
|
+
```
|
|
101
|
+
$ printf '5\r\nhello\r\n0\r\nX-Sum: 1\r\n\r\n' | chunkedte decode
|
|
102
|
+
hello
|
|
103
|
+
X-Sum: 1
|
|
104
|
+
$ printf 'abc' | chunkedte encode --chunk-size 2 --trailer "X-Sum: 1"
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
`decode` writes the body to stdout and the trailer fields to stderr; `encode` frames stdin
|
|
108
|
+
(or a file) as a chunked body. Both exit with status 1 and a message on bad input; body
|
|
109
|
+
bytes decoded before the error may already be on stdout, so treat the output of a failed
|
|
110
|
+
run as truncated. Trailer bytes outside printable ASCII are written as escapes.
|
|
111
|
+
`python -m chunkedte` works too.
|
|
112
|
+
|
|
113
|
+
## What "strict" means here
|
|
114
|
+
|
|
115
|
+
Every leniency in a chunked parser is a way for two HTTP processors to disagree on where a
|
|
116
|
+
message ends, which is how request smuggling works. So the decoder rejects:
|
|
117
|
+
|
|
118
|
+
- a chunk size that is not only hex digits (`+5`, `0x5`, `5_0`, ` 5`, `5 `);
|
|
119
|
+
- chunk extensions that do not follow the grammar (whitespace is allowed only around `;`
|
|
120
|
+
and `=`, values are tokens or quoted strings);
|
|
121
|
+
- a bare LF or a bare CR anywhere a CRLF is expected, including inside an extension;
|
|
122
|
+
- anything other than CRLF right after the chunk data;
|
|
123
|
+
- trailer lines that are not `name: value`, with whitespace before the colon, with control
|
|
124
|
+
bytes, or folded onto a continuation line (obs-fold).
|
|
125
|
+
|
|
126
|
+
Extensions are validated and then ignored, as RFC 9112 asks. The default `Limits` cap one
|
|
127
|
+
size line at 4 KiB, all extensions of a message at 16 KiB, the trailer section at 16 KiB and
|
|
128
|
+
a chunk size below 2**63 (so a forwarded size never wraps in a 64-bit parser); the body is
|
|
129
|
+
unbounded unless you set `max_body_size`. A line split over many small reads is scanned
|
|
130
|
+
once, so decoding stays linear in the input. When `feed` raises, the pieces decoded in that
|
|
131
|
+
call are dropped with it. Received trailers are returned as they came: RFC 9110 section
|
|
132
|
+
6.5.2 says not to merge them into the header section, so keep them apart (a `Content-Length`
|
|
133
|
+
trailer, say, must never reach your framing logic). The encoder refuses trailer names that
|
|
134
|
+
are not tokens, values with control bytes or edge whitespace, and the fields RFC 9110
|
|
135
|
+
section 6.5.1 needs before the content: framing and routing (Content-Length,
|
|
136
|
+
Transfer-Encoding, Trailer, TE, Host, Connection, Keep-Alive, Upgrade, Max-Forwards),
|
|
137
|
+
authentication (Authorization, Proxy-Authorization, WWW-Authenticate, Proxy-Authenticate,
|
|
138
|
+
Cookie, Set-Cookie), request modifiers and response controls (Expect, Range,
|
|
139
|
+
Cache-Control) and content format (Content-Type, Content-Encoding, Content-Range).
|
|
140
|
+
`is_chunked` leaves the rest of RFC 9112 section 6.3 to you: responses to HEAD, 1xx, 204,
|
|
141
|
+
304 and 2xx to CONNECT carry no body, and an HTTP/1.0 message with Transfer-Encoding has
|
|
142
|
+
faulty framing. There is no lenient mode.
|
|
143
|
+
|
|
144
|
+
## Why
|
|
145
|
+
|
|
146
|
+
Chunked transfer coding is simple to frame and easy to get subtly wrong: the terminating
|
|
147
|
+
`0` chunk, the closing blank line, extensions and trailers are routinely mishandled, and
|
|
148
|
+
the security advisories of the last few years (waitress, h11, aiohttp, netty, Go net/http,
|
|
149
|
+
Node.js) are all variations on a parser accepting bytes another parser reads differently.
|
|
150
|
+
The existing Python options are tied to one HTTP stack (`h11`, `aiohttp`) or to one server.
|
|
151
|
+
This is a standalone, typed, zero-dependency codec with the strict behaviour by default:
|
|
152
|
+
reach for it when you move chunked framing yourself and do not want a whole HTTP stack.
|
|
153
|
+
|
|
154
|
+
## License
|
|
155
|
+
|
|
156
|
+
MIT. See [LICENSE](LICENSE).
|
|
157
|
+
|
|
158
|
+
## Links
|
|
159
|
+
|
|
160
|
+
- Website: https://lianes8studio.com
|
|
161
|
+
- GitHub: https://github.com/JaviLianes8
|
|
162
|
+
- LinkedIn: https://www.linkedin.com/in/jlianes/
|