chunkedte 0.2.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (39) hide show
  1. chunkedte-0.2.0/.gitattributes +1 -0
  2. chunkedte-0.2.0/.github/FUNDING.yml +1 -0
  3. chunkedte-0.2.0/.gitignore +16 -0
  4. chunkedte-0.2.0/CHANGELOG.md +42 -0
  5. chunkedte-0.2.0/CONTRIBUTING.md +19 -0
  6. chunkedte-0.2.0/IMPROVEMENTS-round.md +144 -0
  7. chunkedte-0.2.0/LICENSE +21 -0
  8. chunkedte-0.2.0/PKG-INFO +198 -0
  9. chunkedte-0.2.0/README.md +162 -0
  10. chunkedte-0.2.0/pyproject.toml +71 -0
  11. chunkedte-0.2.0/requirements-dev.txt +3 -0
  12. chunkedte-0.2.0/scripts/check.ps1 +16 -0
  13. chunkedte-0.2.0/scripts/fetch_upstream.py +37 -0
  14. chunkedte-0.2.0/scripts/release.ps1 +28 -0
  15. chunkedte-0.2.0/src/chunkedte/__init__.py +22 -0
  16. chunkedte-0.2.0/src/chunkedte/__main__.py +9 -0
  17. chunkedte-0.2.0/src/chunkedte/_cli.py +87 -0
  18. chunkedte-0.2.0/src/chunkedte/_decode.py +177 -0
  19. chunkedte-0.2.0/src/chunkedte/_encode.py +82 -0
  20. chunkedte-0.2.0/src/chunkedte/_errors.py +11 -0
  21. chunkedte-0.2.0/src/chunkedte/_framing.py +66 -0
  22. chunkedte-0.2.0/src/chunkedte/_grammar.py +85 -0
  23. chunkedte-0.2.0/src/chunkedte/_limits.py +33 -0
  24. chunkedte-0.2.0/src/chunkedte/py.typed +0 -0
  25. chunkedte-0.2.0/tests/conftest.py +36 -0
  26. chunkedte-0.2.0/tests/reference.py +123 -0
  27. chunkedte-0.2.0/tests/test_cli.py +113 -0
  28. chunkedte-0.2.0/tests/test_contract.py +40 -0
  29. chunkedte-0.2.0/tests/test_decode.py +121 -0
  30. chunkedte-0.2.0/tests/test_docs.py +85 -0
  31. chunkedte-0.2.0/tests/test_e2e_wheel.py +85 -0
  32. chunkedte-0.2.0/tests/test_encode.py +133 -0
  33. chunkedte-0.2.0/tests/test_framing.py +105 -0
  34. chunkedte-0.2.0/tests/test_fuzz.py +98 -0
  35. chunkedte-0.2.0/tests/test_limits.py +142 -0
  36. chunkedte-0.2.0/tests/test_strict.py +161 -0
  37. chunkedte-0.2.0/tests/test_typing.py +77 -0
  38. chunkedte-0.2.0/tests/test_upstream.py +158 -0
  39. chunkedte-0.2.0/tests/upstream/MANIFEST.json +20 -0
@@ -0,0 +1 @@
1
+ * text=auto eol=lf
@@ -0,0 +1 @@
1
+ buy_me_a_coffee: jlianesglrs
@@ -0,0 +1,16 @@
1
+ __pycache__/
2
+ *.py[cod]
3
+ .venv/
4
+ venv/
5
+ build/
6
+ dist/
7
+ *.egg-info/
8
+ .pytest_cache/
9
+ .mypy_cache/
10
+ .ruff_cache/
11
+ .coverage
12
+ htmlcov/
13
+
14
+ # Third-party reference files, fetched locally by scripts/fetch_upstream.py; only the manifest is tracked.
15
+ tests/upstream/*
16
+ !tests/upstream/MANIFEST.json
@@ -0,0 +1,42 @@
1
+ # Changelog
2
+
3
+ All notable changes to this project are documented here. The format follows
4
+ [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the project uses
5
+ [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
6
+
7
+ ## [Unreleased]
8
+
9
+ ## [0.2.0]
10
+
11
+ ### Added
12
+
13
+ - `Limits` and `LimitError`: bounds on one size line, all chunk extensions of a message,
14
+ the chunk size (below 2**63 by default), the body and the trailer section.
15
+ - `Decoder.trailers` returns the trailer fields in arrival order; `Decoder.unused_data`
16
+ holds the bytes after the message (a pipelined request, for example).
17
+ - `Encoder` streams chunks and closes the message with optional trailer fields; `encode`
18
+ takes `trailers` too. Trailer names, values and framing fields are validated.
19
+ - `is_chunked` decides from Transfer-Encoding and Content-Length whether a body is chunked
20
+ (RFC 9112 sections 6.1 and 6.3) and rejects the combinations used for smuggling.
21
+ - `chunkedte decode` and `chunkedte encode` on the command line, also `python -m chunkedte`.
22
+ - `decode`, `Decoder.feed` and `encode` accept `bytearray` and `memoryview`.
23
+
24
+ ### Changed
25
+
26
+ - The decoder follows RFC 9112 section 7.1 strictly: the size is hex digits only (no `+`,
27
+ `0x`, `_` or whitespace), extensions must match the grammar, only CRLF ends a line (bare
28
+ LF and bare CR are errors), exactly CRLF must follow chunk data, and trailer lines must be
29
+ valid field lines without obs-fold.
30
+ - `decode` raises when bytes follow the message; bytes fed to a finished `Decoder` go to
31
+ `unused_data` instead of raising.
32
+ - A `Decoder` that raised keeps raising on every later `feed`.
33
+ - Lines split over many reads are scanned once, so decoding is linear in the input.
34
+ - `encode` rejects a non-int `chunk_size` and non bytes-like data with `TypeError`.
35
+
36
+ ## [0.1.0]
37
+
38
+ - `encode` frames a byte string as a chunked body split into fixed-size pieces.
39
+ - `Decoder` incrementally decodes chunked bytes fed in any split, ignoring chunk
40
+ extensions and trailer fields (RFC 7230 section 4.1).
41
+ - `decode` decodes a complete message in one call.
42
+ - `ChunkedError` is raised on a malformed size line or broken framing.
@@ -0,0 +1,19 @@
1
+ # Contributing
2
+
3
+ Thanks for taking the time. This project keeps a small, predictable shape.
4
+
5
+ ## Setup
6
+
7
+ ```
8
+ pip install -r requirements-dev.txt
9
+ ```
10
+
11
+ ## Before you open a pull request
12
+
13
+ - Keep public API in `src/chunkedte/__init__.py`; internals live in `_`-prefixed modules.
14
+ - Add a test for the behaviour you change. Tests inject time, disk and network; they
15
+ never touch the real thing.
16
+ - Run `scripts/check.ps1` (format, lint, types, tests) and make sure it passes.
17
+ - One logical change per commit, imperative subject line.
18
+
19
+ Open an issue first if the change is large or shifts the public API.
@@ -0,0 +1,144 @@
1
+ # chunkedte improvement round
2
+
3
+ Progress:
4
+ - [x] Brainstorm ranked improvements (this file)
5
+ - [x] Strict chunk-size line: HEXDIG only, chunk-ext grammar, CRLF only, size cap
6
+ - [x] Strict framing after chunk data and in the trailer section (field-line grammar, no obs-fold)
7
+ - [x] Limits (`Limits`, `LimitError`): line, total extensions, chunk size, body, trailer section
8
+ - [x] Linear time on hostile input (no rescans of buffered bytes)
9
+ - [x] Trailers exposed, bytes after the message kept apart (`unused_data`), `decode` rejects them
10
+ - [x] Encoder: incremental `Encoder`, trailers on `encode`, field validation
11
+ - [x] `is_chunked` for Transfer-Encoding and Content-Length (RFC 9112 section 6.1 and 6.3)
12
+ - [x] Command line tool (`python -m chunkedte`)
13
+ - [x] Contracts: fetched RFC grammar, advisory payloads, differential fuzz, typing, README, wheel e2e
14
+ - [x] Mutation run (guards removed one at a time, each must fail the suite)
15
+ - [x] Docs: README, CHANGELOG, docstrings, packaging metadata
16
+ - [x] Gates: ruff format, ruff check, mypy --strict, pytest
17
+ - [x] Independent review findings fixed with regression tests
18
+ - [x] Pull request opened (#2), guard clean
19
+
20
+ ## Starting point
21
+
22
+ `0.1.0` (not on PyPI) ships `encode`, `decode`, `Decoder` and `ChunkedError`. Read as an
23
+ HTTP server or proxy author who feeds it attacker bytes, it has these gaps:
24
+
25
+ - The size is parsed with `int(token.strip(), 16)`. Python's `int` takes `+5`, `0x5`, `5_0`
26
+ and surrounding whitespace, none of which is `1*HEXDIG`. This is the exact bug class of
27
+ the waitress advisory below: two parsers that disagree on a chunk size disagree on where
28
+ the next request starts.
29
+ - Everything after `;` is dropped unchecked, so a size line such as `5;\nxx` or `5;a b`
30
+ passes. Lines are split on CRLF only, so a bare LF inside an extension is swallowed, the
31
+ pattern of the aiohttp and netty advisories.
32
+ - Trailer lines are read but never checked: obs-fold, whitespace before the colon, a line
33
+ without a colon and control bytes all pass. Trailers are thrown away, with no way to read
34
+ them.
35
+ - No limits: an endless size line, endless extensions or an endless trailer section are
36
+ buffered without bound (the Node.js advisory below), and the chunk size has no cap.
37
+ - A line split over many small feeds is searched from the start on every feed: quadratic.
38
+ - Bytes after the terminator stay in the buffer silently; `decode(b"0\r\n\r\nGET /")`
39
+ returns `b""` with no error.
40
+ - The encoder cannot send trailers or stream; there is no help for the Transfer-Encoding
41
+ and Content-Length decision that decides whether chunked applies at all.
42
+ - Packaging: no keywords or Python classifiers, README examples never run, no wheel test.
43
+
44
+ ## Sources (fetched 2026-10-13)
45
+
46
+ - RFC 9112, https://www.rfc-editor.org/rfc/rfc9112.txt (sha256 `e4f426ba...`): section 7.1
47
+ (chunked-body, chunk, chunk-size = 1*HEXDIG, last-chunk = 1*("0"), recipients must cope
48
+ with large numerals), 7.1.1 (chunk-ext with BWS around `;` and `=`, token or quoted-string
49
+ values, a server ought to limit the total extension length), 7.1.2 (trailer-section =
50
+ *( field-line CRLF )), 7.1.3 (decoding pseudo-code), 2.2 (CRLF; a recipient MAY accept a
51
+ bare LF, a bare CR is invalid), 5 and 5.1 (field-line, no whitespace before the colon),
52
+ 5.2 (obs-fold), 6.1 and 6.3 (Transfer-Encoding overrides Content-Length, both together
53
+ ought to be an error, chunked must be the final coding of a request).
54
+ - RFC 9112 errata, https://www.rfc-editor.org/errata/rfc9112: two verified editorial errata
55
+ (obs-text reference, a missing changes note), none on section 7. The page also lists
56
+ RFC 9931 (March 2026) as updating RFC 9112; fetched from
57
+ https://www.rfc-editor.org/rfc/rfc9931.txt, it covers optimistic protocol upgrades, not
58
+ the chunked coding.
59
+ - RFC 9110, https://www.rfc-editor.org/rfc/rfc9110.txt (sha256 `21c1cdce...`): 5.5
60
+ (field-value, field-vchar, obs-text), 5.6.2 (token, tchar), 5.6.3 (OWS, BWS), 5.6.4
61
+ (quoted-string, qdtext, quoted-pair), 6.5 and 6.5.1 (trailer fields; fields for framing
62
+ and routing cannot be sent as trailers).
63
+ - GHSA-4f7p-27jc-3c36 (waitress, CVE-2022-24761): `int()` accepted `+10` and `0x01` as
64
+ sizes; extensions were dropped without validation.
65
+ - GHSA-vqfr-h8mv-ghfj (h11, CVE-2025-43859): any two bytes accepted after chunk data instead
66
+ of CRLF; the advisory's payload `5\r\nAAAAAXX2\r\n45\r\n0\r\n\r\n` smuggles a request.
67
+ - GHSA-8495-4g3g-x7pr (aiohttp, CVE-2024-52304) and GHSA-fghv-69vj-qj49 (netty,
68
+ CVE-2025-58056): a bare LF inside a chunk extension ended the size line.
69
+ - GHSA-g9pc-8g42-g6vq (Go net/http via RoadRunner, CVE-2025-22871): a bare LF accepted as the
70
+ size line terminator.
71
+ - GHSA-prhj-8562-p8gj (Node.js, CVE-2024-22019): unbounded chunk extension bytes across a
72
+ message exhausted CPU and bandwidth.
73
+ - GHSA-cggh-pq45-6h9x (llhttp, CVE-2023-30589): a bare CR accepted as a line terminator.
74
+
75
+ The RFC text is third-party: `scripts/fetch_upstream.py` fetches it into a gitignored
76
+ `tests/upstream/` folder checked against a tracked `MANIFEST.json` (URL, size, sha256,
77
+ licence); the tests that read it skip when it is absent. Advisory payloads are short test
78
+ vectors, cited with their URL next to them.
79
+
80
+ ## Ranked brainstorm
81
+
82
+ 1. **Strict size line** (correctness, smuggling). Size must fullmatch `1*HEXDIG`; extensions
83
+ must match the RFC 9112 grammar exactly (BWS, token, quoted-string); the line ends only at
84
+ CRLF; bare LF and bare CR are errors. No lenient switch: the advisories show that every
85
+ leniency here is a desync with some other parser.
86
+ 2. **Strict data terminator and trailer section.** Exactly CRLF after chunk data, failing on
87
+ the first wrong byte. Trailer lines must be `token ":" OWS field-value OWS`; a line that
88
+ starts with SP or HTAB (obs-fold) is rejected, as are control bytes.
89
+ 3. **Limits** with safe defaults and a distinct `LimitError` (maps to 413/431 rather than
90
+ 400): one size line, total extension bytes per message, chunk size (default below
91
+ 2**63 so a value never wraps in a downstream 64-bit parser), body size, trailer section.
92
+ 4. **Linear time.** Keep a scan offset so a line split over n feeds costs O(n), and fail as
93
+ soon as the buffered line passes its limit instead of waiting for a CRLF.
94
+ 5. **Trailers and pipelining.** `Decoder.trailers` returns the parsed fields in order;
95
+ bytes after the final CRLF go to `Decoder.unused_data` (the next pipelined message), and
96
+ `decode` raises if there are any.
97
+ 6. **Encoder.** `Encoder.write` and `Encoder.finish(trailers)` for streaming; `encode` takes
98
+ trailers. Names must be tokens, values field-values, and framing or routing fields
99
+ (Content-Length, Transfer-Encoding, Trailer, Host) are refused as trailers.
100
+ 7. **`is_chunked(transfer_encoding, content_length, request=True)`** applies RFC 9112 6.1
101
+ and 6.3: both fields together is an error, chunked twice or not last in a request is an
102
+ error, chunked with parameters is an error, a response with chunked not last is read to
103
+ close (returns False).
104
+ 8. **Typed API**: bytes-like input, `Limits` frozen dataclass, `py.typed`, mypy snippet test.
105
+ 9. **CLI**: `python -m chunkedte decode|encode` over stdin and stdout, trailers on stderr.
106
+ 10. **Contracts**: grammar classes checked against the fetched RFC ABNF, advisory payloads,
107
+ a differential fuzz against an independent whole-message reference parser over random
108
+ mutations and random splits, README examples executed, installed-wheel e2e, mutation run.
109
+ 11. **Packaging**: keywords, classifiers, console script, CHANGELOG, version 0.2.0.
110
+
111
+ Out of scope: content codings (gzip inside chunked), HTTP header parsing beyond the two
112
+ framing fields, and any lenient mode.
113
+
114
+ ## Mutation run
115
+
116
+ 66 hand-written mutants (one guard, bound or grammar class changed at a time) over every
117
+ module, each run against the whole suite with bytecode writes off, `__pycache__` purged
118
+ and a per-mutant timeout; the suite was green before and after. 63 were killed at once.
119
+ Of the three survivors, one exposed a weak test (`unused_data` before completion was
120
+ checked with an empty buffer) and is now killed; one was a redundant type check in
121
+ `is_chunked` that the regex already enforces, now removed; the last (`\Z` to `$` in the
122
+ Transfer-Encoding list parser) is equivalent, because the zero-length-match guard rejects
123
+ the trailing newline either way.
124
+
125
+ ## Independent review
126
+
127
+ One medium and eight low findings, all handled:
128
+
129
+ - `is_chunked` backtracked quadratically on long whitespace runs (two adjacent OWS runs in
130
+ the list regex). Fixed by moving the trailing OWS inside the optional coding group;
131
+ regression tests with 64 KiB of spaces and tabs.
132
+ - Transfer-Encoding errors echoed the whole field; now cut to 64 characters, with a test.
133
+ - The encoder refused only framing and routing trailers; it now refuses every RFC 9110
134
+ section 6.5.1 category (authentication, request modifiers, response controls, content
135
+ format), tested.
136
+ - The CLI failed on a latin-1 trailer with an ASCII terminal; trailer bytes outside ASCII
137
+ are now escaped, tested.
138
+ - Documented: received trailers must stay apart from the header section, `feed` drops the
139
+ pieces of the call that raises, the CLI may leave a truncated body on stdout on failure,
140
+ and `is_chunked` leaves HEAD, 1xx, 204, 304, CONNECT and HTTP/1.0 to the caller.
141
+ - Kept on purpose: two wall-clock tests with generous bounds (they are the only way to
142
+ catch a quadratic regression, and the mutation run showed they bite), CLI tests that use
143
+ `tmp_path`, and Python 3.10 as the floor (the code uses `X | Y` type aliases).
144
+
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Javier Lianes
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,198 @@
1
+ Metadata-Version: 2.5
2
+ Name: chunkedte
3
+ Version: 0.2.0
4
+ Summary: Strict, incremental, typed, zero-dependency codec for HTTP/1.1 chunked transfer coding
5
+ Project-URL: Homepage, https://lianes8studio.com
6
+ Project-URL: Repository, https://github.com/JaviLianes8/chunkedte
7
+ Project-URL: Issues, https://github.com/JaviLianes8/chunkedte/issues
8
+ Project-URL: Changelog, https://github.com/JaviLianes8/chunkedte/blob/main/CHANGELOG.md
9
+ Project-URL: LinkedIn, https://www.linkedin.com/in/jlianes/
10
+ Project-URL: Funding, https://buymeacoffee.com/jlianesglrs
11
+ Author-email: Javier Lianes <javier@lianes8studio.com>
12
+ License-Expression: MIT
13
+ License-File: LICENSE
14
+ Keywords: chunked,codec,http,request-smuggling,rfc9112,trailers,transfer-encoding
15
+ Classifier: Development Status :: 4 - Beta
16
+ Classifier: Intended Audience :: Developers
17
+ Classifier: Operating System :: OS Independent
18
+ Classifier: Programming Language :: Python :: 3
19
+ Classifier: Programming Language :: Python :: 3 :: Only
20
+ Classifier: Programming Language :: Python :: 3.10
21
+ Classifier: Programming Language :: Python :: 3.11
22
+ Classifier: Programming Language :: Python :: 3.12
23
+ Classifier: Programming Language :: Python :: 3.13
24
+ Classifier: Programming Language :: Python :: 3.14
25
+ Classifier: Topic :: Internet :: WWW/HTTP
26
+ Classifier: Topic :: Software Development :: Libraries :: Python Modules
27
+ Classifier: Typing :: Typed
28
+ Requires-Python: >=3.10
29
+ Provides-Extra: dev
30
+ Requires-Dist: build; extra == 'dev'
31
+ Requires-Dist: mypy; extra == 'dev'
32
+ Requires-Dist: pytest; extra == 'dev'
33
+ Requires-Dist: ruff; extra == 'dev'
34
+ Requires-Dist: twine; extra == 'dev'
35
+ Description-Content-Type: text/markdown
36
+
37
+ # chunkedte
38
+
39
+ Strict, incremental, typed, zero-dependency codec for HTTP/1.1 chunked transfer coding
40
+ (RFC 9112 section 7.1).
41
+
42
+ ## Install
43
+
44
+ ```
45
+ pip install chunkedte
46
+ ```
47
+
48
+ Python 3.10 or newer. No dependencies.
49
+
50
+ ## Usage
51
+
52
+ Decode a complete message in one call. `decode` raises `ChunkedError` if the message is
53
+ malformed, unfinished, or followed by more bytes:
54
+
55
+ ```python
56
+ from chunkedte import ChunkedError, decode
57
+
58
+ print(decode(b"5\r\nhello\r\n0\r\n\r\n"))
59
+ try:
60
+ decode(b"0x5\r\nhello\r\n0\r\n\r\n")
61
+ except ChunkedError as error:
62
+ print(error)
63
+ # b'hello'
64
+ # malformed chunk-size line: b'0x5'
65
+ ```
66
+
67
+ Decode incrementally as bytes arrive off a socket. `Decoder.feed` takes any split and
68
+ returns the body pieces available so far; trailer fields and whatever follows the message
69
+ (a pipelined request, say) are kept apart:
70
+
71
+ ```python
72
+ from chunkedte import Decoder
73
+
74
+ decoder = Decoder()
75
+ body = b""
76
+ for part in [b"4\r\nwi", b"ki\r\n5;sig=ab\r\npedia\r\n0\r\nDigest: x\r\n", b"\r\nGET /"]:
77
+ for piece in decoder.feed(part):
78
+ body += piece
79
+ print(body, decoder.is_complete)
80
+ print(decoder.trailers, decoder.unused_data)
81
+ # b'wikipedia' True
82
+ # [('Digest', 'x')] b'GET /'
83
+ ```
84
+
85
+ Bound what a peer can make you buffer. Passing a bound raises `LimitError`, a subclass of
86
+ `ChunkedError`, so a server can answer 413 or 431 instead of 400:
87
+
88
+ ```python
89
+ from chunkedte import Decoder, LimitError, Limits
90
+
91
+ decoder = Decoder(Limits(max_body_size=1024, max_trailer_size=4096))
92
+ try:
93
+ decoder.feed(b"10000\r\n")
94
+ except LimitError as error:
95
+ print(error)
96
+ print(Limits())
97
+ # body exceeds max_body_size
98
+ # Limits(max_line_size=4096, max_extensions_size=16384, max_chunk_size=9223372036854775807, max_body_size=None, max_trailer_size=16384)
99
+ ```
100
+
101
+ Encode in one call, or stream with `Encoder` and close with optional trailer fields:
102
+
103
+ ```python
104
+ from chunkedte import Encoder, encode
105
+
106
+ print(encode(b"abcd", chunk_size=2))
107
+ encoder = Encoder()
108
+ print(encoder.write(b"hello") + encoder.finish({"X-Checksum": "42"}))
109
+ # b'2\r\nab\r\n2\r\ncd\r\n0\r\n\r\n'
110
+ # b'5\r\nhello\r\n0\r\nX-Checksum: 42\r\n\r\n'
111
+ ```
112
+
113
+ Decide whether a message body is chunked at all from its Transfer-Encoding and
114
+ Content-Length values (`None` when absent). `is_chunked` applies RFC 9112 sections 6.1 and
115
+ 6.3 and raises `ChunkedError` on the combinations used for request smuggling:
116
+
117
+ ```python
118
+ from chunkedte import ChunkedError, is_chunked
119
+
120
+ print(is_chunked("gzip, chunked"), is_chunked(None, "12"))
121
+ print(is_chunked("gzip", request=False))
122
+ for te, cl in [("chunked", "12"), ("chunked, chunked", None), ("chunked, gzip", None)]:
123
+ try:
124
+ is_chunked(te, cl)
125
+ except ChunkedError as error:
126
+ print(error)
127
+ # True False
128
+ # False
129
+ # both Transfer-Encoding and Content-Length are present
130
+ # chunked is applied more than once
131
+ # chunked is not the final transfer coding of a request
132
+ ```
133
+
134
+ ### Command line
135
+
136
+ ```
137
+ $ printf '5\r\nhello\r\n0\r\nX-Sum: 1\r\n\r\n' | chunkedte decode
138
+ hello
139
+ X-Sum: 1
140
+ $ printf 'abc' | chunkedte encode --chunk-size 2 --trailer "X-Sum: 1"
141
+ ```
142
+
143
+ `decode` writes the body to stdout and the trailer fields to stderr; `encode` frames stdin
144
+ (or a file) as a chunked body. Both exit with status 1 and a message on bad input; body
145
+ bytes decoded before the error may already be on stdout, so treat the output of a failed
146
+ run as truncated. Trailer bytes outside printable ASCII are written as escapes.
147
+ `python -m chunkedte` works too.
148
+
149
+ ## What "strict" means here
150
+
151
+ Every leniency in a chunked parser is a way for two HTTP processors to disagree on where a
152
+ message ends, which is how request smuggling works. So the decoder rejects:
153
+
154
+ - a chunk size that is not only hex digits (`+5`, `0x5`, `5_0`, ` 5`, `5 `);
155
+ - chunk extensions that do not follow the grammar (whitespace is allowed only around `;`
156
+ and `=`, values are tokens or quoted strings);
157
+ - a bare LF or a bare CR anywhere a CRLF is expected, including inside an extension;
158
+ - anything other than CRLF right after the chunk data;
159
+ - trailer lines that are not `name: value`, with whitespace before the colon, with control
160
+ bytes, or folded onto a continuation line (obs-fold).
161
+
162
+ Extensions are validated and then ignored, as RFC 9112 asks. The default `Limits` cap one
163
+ size line at 4 KiB, all extensions of a message at 16 KiB, the trailer section at 16 KiB and
164
+ a chunk size below 2**63 (so a forwarded size never wraps in a 64-bit parser); the body is
165
+ unbounded unless you set `max_body_size`. A line split over many small reads is scanned
166
+ once, so decoding stays linear in the input. When `feed` raises, the pieces decoded in that
167
+ call are dropped with it. Received trailers are returned as they came: RFC 9110 section
168
+ 6.5.2 says not to merge them into the header section, so keep them apart (a `Content-Length`
169
+ trailer, say, must never reach your framing logic). The encoder refuses trailer names that
170
+ are not tokens, values with control bytes or edge whitespace, and the fields RFC 9110
171
+ section 6.5.1 needs before the content: framing and routing (Content-Length,
172
+ Transfer-Encoding, Trailer, TE, Host, Connection, Keep-Alive, Upgrade, Max-Forwards),
173
+ authentication (Authorization, Proxy-Authorization, WWW-Authenticate, Proxy-Authenticate,
174
+ Cookie, Set-Cookie), request modifiers and response controls (Expect, Range,
175
+ Cache-Control) and content format (Content-Type, Content-Encoding, Content-Range).
176
+ `is_chunked` leaves the rest of RFC 9112 section 6.3 to you: responses to HEAD, 1xx, 204,
177
+ 304 and 2xx to CONNECT carry no body, and an HTTP/1.0 message with Transfer-Encoding has
178
+ faulty framing. There is no lenient mode.
179
+
180
+ ## Why
181
+
182
+ Chunked transfer coding is simple to frame and easy to get subtly wrong: the terminating
183
+ `0` chunk, the closing blank line, extensions and trailers are routinely mishandled, and
184
+ the security advisories of the last few years (waitress, h11, aiohttp, netty, Go net/http,
185
+ Node.js) are all variations on a parser accepting bytes another parser reads differently.
186
+ The existing Python options are tied to one HTTP stack (`h11`, `aiohttp`) or to one server.
187
+ This is a standalone, typed, zero-dependency codec with the strict behaviour by default:
188
+ reach for it when you move chunked framing yourself and do not want a whole HTTP stack.
189
+
190
+ ## License
191
+
192
+ MIT. See [LICENSE](LICENSE).
193
+
194
+ ## Links
195
+
196
+ - Website: https://lianes8studio.com
197
+ - GitHub: https://github.com/JaviLianes8
198
+ - LinkedIn: https://www.linkedin.com/in/jlianes/
@@ -0,0 +1,162 @@
1
+ # chunkedte
2
+
3
+ Strict, incremental, typed, zero-dependency codec for HTTP/1.1 chunked transfer coding
4
+ (RFC 9112 section 7.1).
5
+
6
+ ## Install
7
+
8
+ ```
9
+ pip install chunkedte
10
+ ```
11
+
12
+ Python 3.10 or newer. No dependencies.
13
+
14
+ ## Usage
15
+
16
+ Decode a complete message in one call. `decode` raises `ChunkedError` if the message is
17
+ malformed, unfinished, or followed by more bytes:
18
+
19
+ ```python
20
+ from chunkedte import ChunkedError, decode
21
+
22
+ print(decode(b"5\r\nhello\r\n0\r\n\r\n"))
23
+ try:
24
+ decode(b"0x5\r\nhello\r\n0\r\n\r\n")
25
+ except ChunkedError as error:
26
+ print(error)
27
+ # b'hello'
28
+ # malformed chunk-size line: b'0x5'
29
+ ```
30
+
31
+ Decode incrementally as bytes arrive off a socket. `Decoder.feed` takes any split and
32
+ returns the body pieces available so far; trailer fields and whatever follows the message
33
+ (a pipelined request, say) are kept apart:
34
+
35
+ ```python
36
+ from chunkedte import Decoder
37
+
38
+ decoder = Decoder()
39
+ body = b""
40
+ for part in [b"4\r\nwi", b"ki\r\n5;sig=ab\r\npedia\r\n0\r\nDigest: x\r\n", b"\r\nGET /"]:
41
+ for piece in decoder.feed(part):
42
+ body += piece
43
+ print(body, decoder.is_complete)
44
+ print(decoder.trailers, decoder.unused_data)
45
+ # b'wikipedia' True
46
+ # [('Digest', 'x')] b'GET /'
47
+ ```
48
+
49
+ Bound what a peer can make you buffer. Passing a bound raises `LimitError`, a subclass of
50
+ `ChunkedError`, so a server can answer 413 or 431 instead of 400:
51
+
52
+ ```python
53
+ from chunkedte import Decoder, LimitError, Limits
54
+
55
+ decoder = Decoder(Limits(max_body_size=1024, max_trailer_size=4096))
56
+ try:
57
+ decoder.feed(b"10000\r\n")
58
+ except LimitError as error:
59
+ print(error)
60
+ print(Limits())
61
+ # body exceeds max_body_size
62
+ # Limits(max_line_size=4096, max_extensions_size=16384, max_chunk_size=9223372036854775807, max_body_size=None, max_trailer_size=16384)
63
+ ```
64
+
65
+ Encode in one call, or stream with `Encoder` and close with optional trailer fields:
66
+
67
+ ```python
68
+ from chunkedte import Encoder, encode
69
+
70
+ print(encode(b"abcd", chunk_size=2))
71
+ encoder = Encoder()
72
+ print(encoder.write(b"hello") + encoder.finish({"X-Checksum": "42"}))
73
+ # b'2\r\nab\r\n2\r\ncd\r\n0\r\n\r\n'
74
+ # b'5\r\nhello\r\n0\r\nX-Checksum: 42\r\n\r\n'
75
+ ```
76
+
77
+ Decide whether a message body is chunked at all from its Transfer-Encoding and
78
+ Content-Length values (`None` when absent). `is_chunked` applies RFC 9112 sections 6.1 and
79
+ 6.3 and raises `ChunkedError` on the combinations used for request smuggling:
80
+
81
+ ```python
82
+ from chunkedte import ChunkedError, is_chunked
83
+
84
+ print(is_chunked("gzip, chunked"), is_chunked(None, "12"))
85
+ print(is_chunked("gzip", request=False))
86
+ for te, cl in [("chunked", "12"), ("chunked, chunked", None), ("chunked, gzip", None)]:
87
+ try:
88
+ is_chunked(te, cl)
89
+ except ChunkedError as error:
90
+ print(error)
91
+ # True False
92
+ # False
93
+ # both Transfer-Encoding and Content-Length are present
94
+ # chunked is applied more than once
95
+ # chunked is not the final transfer coding of a request
96
+ ```
97
+
98
+ ### Command line
99
+
100
+ ```
101
+ $ printf '5\r\nhello\r\n0\r\nX-Sum: 1\r\n\r\n' | chunkedte decode
102
+ hello
103
+ X-Sum: 1
104
+ $ printf 'abc' | chunkedte encode --chunk-size 2 --trailer "X-Sum: 1"
105
+ ```
106
+
107
+ `decode` writes the body to stdout and the trailer fields to stderr; `encode` frames stdin
108
+ (or a file) as a chunked body. Both exit with status 1 and a message on bad input; body
109
+ bytes decoded before the error may already be on stdout, so treat the output of a failed
110
+ run as truncated. Trailer bytes outside printable ASCII are written as escapes.
111
+ `python -m chunkedte` works too.
112
+
113
+ ## What "strict" means here
114
+
115
+ Every leniency in a chunked parser is a way for two HTTP processors to disagree on where a
116
+ message ends, which is how request smuggling works. So the decoder rejects:
117
+
118
+ - a chunk size that is not only hex digits (`+5`, `0x5`, `5_0`, ` 5`, `5 `);
119
+ - chunk extensions that do not follow the grammar (whitespace is allowed only around `;`
120
+ and `=`, values are tokens or quoted strings);
121
+ - a bare LF or a bare CR anywhere a CRLF is expected, including inside an extension;
122
+ - anything other than CRLF right after the chunk data;
123
+ - trailer lines that are not `name: value`, with whitespace before the colon, with control
124
+ bytes, or folded onto a continuation line (obs-fold).
125
+
126
+ Extensions are validated and then ignored, as RFC 9112 asks. The default `Limits` cap one
127
+ size line at 4 KiB, all extensions of a message at 16 KiB, the trailer section at 16 KiB and
128
+ a chunk size below 2**63 (so a forwarded size never wraps in a 64-bit parser); the body is
129
+ unbounded unless you set `max_body_size`. A line split over many small reads is scanned
130
+ once, so decoding stays linear in the input. When `feed` raises, the pieces decoded in that
131
+ call are dropped with it. Received trailers are returned as they came: RFC 9110 section
132
+ 6.5.2 says not to merge them into the header section, so keep them apart (a `Content-Length`
133
+ trailer, say, must never reach your framing logic). The encoder refuses trailer names that
134
+ are not tokens, values with control bytes or edge whitespace, and the fields RFC 9110
135
+ section 6.5.1 needs before the content: framing and routing (Content-Length,
136
+ Transfer-Encoding, Trailer, TE, Host, Connection, Keep-Alive, Upgrade, Max-Forwards),
137
+ authentication (Authorization, Proxy-Authorization, WWW-Authenticate, Proxy-Authenticate,
138
+ Cookie, Set-Cookie), request modifiers and response controls (Expect, Range,
139
+ Cache-Control) and content format (Content-Type, Content-Encoding, Content-Range).
140
+ `is_chunked` leaves the rest of RFC 9112 section 6.3 to you: responses to HEAD, 1xx, 204,
141
+ 304 and 2xx to CONNECT carry no body, and an HTTP/1.0 message with Transfer-Encoding has
142
+ faulty framing. There is no lenient mode.
143
+
144
+ ## Why
145
+
146
+ Chunked transfer coding is simple to frame and easy to get subtly wrong: the terminating
147
+ `0` chunk, the closing blank line, extensions and trailers are routinely mishandled, and
148
+ the security advisories of the last few years (waitress, h11, aiohttp, netty, Go net/http,
149
+ Node.js) are all variations on a parser accepting bytes another parser reads differently.
150
+ The existing Python options are tied to one HTTP stack (`h11`, `aiohttp`) or to one server.
151
+ This is a standalone, typed, zero-dependency codec with the strict behaviour by default:
152
+ reach for it when you move chunked framing yourself and do not want a whole HTTP stack.
153
+
154
+ ## License
155
+
156
+ MIT. See [LICENSE](LICENSE).
157
+
158
+ ## Links
159
+
160
+ - Website: https://lianes8studio.com
161
+ - GitHub: https://github.com/JaviLianes8
162
+ - LinkedIn: https://www.linkedin.com/in/jlianes/