json-compress 1.0.3__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- json_compress-1.0.3/.gitignore +60 -0
- json_compress-1.0.3/CHANGELOG.md +47 -0
- json_compress-1.0.3/LICENSE +21 -0
- json_compress-1.0.3/PKG-INFO +495 -0
- json_compress-1.0.3/README.md +441 -0
- json_compress-1.0.3/examples/benchmark.py +275 -0
- json_compress-1.0.3/pyproject.toml +91 -0
- json_compress-1.0.3/src/json_compress/__init__.py +242 -0
- json_compress-1.0.3/src/json_compress/__main__.py +5 -0
- json_compress-1.0.3/src/json_compress/cli.py +243 -0
- json_compress-1.0.3/src/json_compress/column_order.py +82 -0
- json_compress-1.0.3/src/json_compress/constants.py +50 -0
- json_compress-1.0.3/src/json_compress/decode.py +224 -0
- json_compress-1.0.3/src/json_compress/encode.py +325 -0
- json_compress-1.0.3/src/json_compress/jsontext.py +249 -0
- json_compress-1.0.3/src/json_compress/locate.py +358 -0
- json_compress-1.0.3/src/json_compress/normalize.py +151 -0
- json_compress-1.0.3/src/json_compress/options.py +112 -0
- json_compress-1.0.3/src/json_compress/pack.py +134 -0
- json_compress-1.0.3/src/json_compress/py.typed +0 -0
- json_compress-1.0.3/src/json_compress/sizes.py +48 -0
- json_compress-1.0.3/src/json_compress/table.py +170 -0
- json_compress-1.0.3/src/json_compress/tokens.py +125 -0
- json_compress-1.0.3/tests/generator.py +97 -0
- json_compress-1.0.3/tests/interop/verify-interop.mjs +114 -0
- json_compress-1.0.3/tests/test_api.py +348 -0
- json_compress-1.0.3/tests/test_cli.py +143 -0
- json_compress-1.0.3/tests/test_conformance.py +94 -0
- json_compress-1.0.3/tests/test_fuzz.py +101 -0
- json_compress-1.0.3/tests/test_locate.py +92 -0
- json_compress-1.0.3/tests/test_pack.py +107 -0
- json_compress-1.0.3/tests/test_roundtrip.py +280 -0
- json_compress-1.0.3/tests/test_tokens.py +103 -0
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
# Working notes and the internal code this library was repurposed from.
|
|
2
|
+
concept.txt
|
|
3
|
+
results-helper.ts
|
|
4
|
+
|
|
5
|
+
# Dependencies
|
|
6
|
+
node_modules/
|
|
7
|
+
*/node_modules/
|
|
8
|
+
|
|
9
|
+
# javascript/dist is deliberately committed: jsDelivr serves the browser bundle
|
|
10
|
+
# straight out of the repository, so the built files have to be in it.
|
|
11
|
+
|
|
12
|
+
# .NET build output. Scoped to dotnet/, because javascript/bin holds the CLI
|
|
13
|
+
# entry point and is meant to be committed.
|
|
14
|
+
dotnet/**/bin/
|
|
15
|
+
dotnet/**/obj/
|
|
16
|
+
dotnet/**/TestResults/
|
|
17
|
+
*.user
|
|
18
|
+
|
|
19
|
+
# Python build output and caches.
|
|
20
|
+
__pycache__/
|
|
21
|
+
*.py[cod]
|
|
22
|
+
python/build/
|
|
23
|
+
python/dist/
|
|
24
|
+
python/.venv/
|
|
25
|
+
python/*.egg-info/
|
|
26
|
+
.pytest_cache/
|
|
27
|
+
.ruff_cache/
|
|
28
|
+
|
|
29
|
+
# PHP dependencies and caches. composer.lock is not committed: this is a
|
|
30
|
+
# library, and the lock file of a library binds nobody.
|
|
31
|
+
php/vendor/
|
|
32
|
+
php/composer.lock
|
|
33
|
+
php/.phpunit.cache/
|
|
34
|
+
.phpunit.result.cache
|
|
35
|
+
|
|
36
|
+
# Go build output, and the corpus `go test -fuzz` writes as it works. The module
|
|
37
|
+
# has no dependencies, so there is no go.sum and nothing vendored.
|
|
38
|
+
go/json-compress
|
|
39
|
+
go/json-compress.exe
|
|
40
|
+
go/**/testdata/fuzz/
|
|
41
|
+
go/**/*.test
|
|
42
|
+
go/coverage.out
|
|
43
|
+
|
|
44
|
+
# Editor and OS noise
|
|
45
|
+
.DS_Store
|
|
46
|
+
Thumbs.db
|
|
47
|
+
.vscode/
|
|
48
|
+
.idea/
|
|
49
|
+
|
|
50
|
+
# Logs
|
|
51
|
+
npm-debug.log*
|
|
52
|
+
*.log
|
|
53
|
+
|
|
54
|
+
# Rust build output. Cargo.lock is not committed: this is a library, and the
|
|
55
|
+
# lock file of a library binds nobody - the CI matrix wants to resolve fresh.
|
|
56
|
+
rust/target/
|
|
57
|
+
rust/Cargo.lock
|
|
58
|
+
rust/fuzz/target/
|
|
59
|
+
rust/fuzz/corpus/
|
|
60
|
+
rust/fuzz/artifacts/
|
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to `json-compress` are recorded here. The format follows
|
|
4
|
+
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project uses
|
|
5
|
+
[semantic versioning](https://semver.org/spec/v2.0.0.html).
|
|
6
|
+
|
|
7
|
+
## [1.0.3] - 2026-10-13
|
|
8
|
+
|
|
9
|
+
First release of the Python implementation. Its version number matches the other
|
|
10
|
+
implementations; 1.0.0 was never published.
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
|
|
14
|
+
- `compress` and `decompress`, a lossless round trip through a JSON-only wire
|
|
15
|
+
format: a key catalogue, columnar tables for lists of like-shaped dicts,
|
|
16
|
+
constant and dictionary columns, and a catalogue of repeated string values.
|
|
17
|
+
- Key order is preserved, including inside packed tables. Where no single column
|
|
18
|
+
order can satisfy every row, the list is left as dicts rather than reordered.
|
|
19
|
+
- A missing key and a null one stay distinct through the round trip.
|
|
20
|
+
- `stringify` and `parse`, the same pair at the string level.
|
|
21
|
+
- `decompress_if_needed` and `is_compressed`, for boundaries carrying both forms.
|
|
22
|
+
- `measure`, reporting minified, compressed and gzipped sizes before you commit.
|
|
23
|
+
- `create_codec` and `JsonCompressCodec`, binding a set of options once.
|
|
24
|
+
- `locate` and `validate`, reporting the exact line, column and cause of a
|
|
25
|
+
malformed JSON document rather than relaying the parser's message. The scan is
|
|
26
|
+
iterative, so a pathologically nested document is reported as too deep rather
|
|
27
|
+
than exhausting the interpreter stack.
|
|
28
|
+
- `minify` and `beautify`.
|
|
29
|
+
- `byte_length`, `gzip_bytes`, `gunzip` and `gzip_size`, over the standard
|
|
30
|
+
library with no dependency and a zeroed gzip timestamp so sizes are stable.
|
|
31
|
+
- `pack`, `unpack`, `pack_all`, `unpack_all` and `text_to_array`, the columnar
|
|
32
|
+
transform on its own for consumers outside Python.
|
|
33
|
+
- A command line covering `compress`, `decompress`, `stats`, `minify`,
|
|
34
|
+
`beautify` and `validate`, reading a file or stdin, also reachable as
|
|
35
|
+
`python -m json_compress`.
|
|
36
|
+
- Type hints throughout, and a `py.typed` marker.
|
|
37
|
+
- Conversion of the standard library types a JSON writer is expected to handle -
|
|
38
|
+
`datetime`, `date`, `time`, `Decimal`, `UUID`, `Enum`, dataclasses and tuples -
|
|
39
|
+
and a `for_json()` hook for anything else.
|
|
40
|
+
- Float formatting that follows the ECMAScript rule rather than `repr`, so this
|
|
41
|
+
implementation writes the same bytes as the JavaScript and .NET ones. Integers
|
|
42
|
+
keep Python's exact precision.
|
|
43
|
+
- A conformance suite run against the 144-case corpus written by the JavaScript
|
|
44
|
+
encoder, asserting both that every envelope is restored and that every
|
|
45
|
+
envelope is reproduced byte for byte.
|
|
46
|
+
|
|
47
|
+
[1.0.0]: https://github.com/eharain/JSON-Compress/releases/tag/python-v1.0.0
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Tech Style Ltd
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,495 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: json-compress
|
|
3
|
+
Version: 1.0.3
|
|
4
|
+
Summary: Lossless, dependency-free JSON compression for Python. Hoists repeated object keys into a catalogue, packs lists of dicts into columnar tables, and de-duplicates repeated string values - typically 40-80% smaller JSON, still valid JSON, and an exact round trip.
|
|
5
|
+
Project-URL: Homepage, https://tech-style.co/product-json-compress.html
|
|
6
|
+
Project-URL: Documentation, https://github.com/eharain/JSON-Compress/tree/main/python#readme
|
|
7
|
+
Project-URL: Repository, https://github.com/eharain/JSON-Compress
|
|
8
|
+
Project-URL: Issues, https://github.com/eharain/JSON-Compress/issues
|
|
9
|
+
Project-URL: Changelog, https://github.com/eharain/JSON-Compress/blob/main/python/CHANGELOG.md
|
|
10
|
+
Project-URL: Specification, https://github.com/eharain/JSON-Compress/blob/main/SPEC.md
|
|
11
|
+
Author-email: Ejaz Hussain Arain <hello@tech-style.co>
|
|
12
|
+
Maintainer-email: Tech Style Ltd <hello@tech-style.co>
|
|
13
|
+
License: MIT License
|
|
14
|
+
|
|
15
|
+
Copyright (c) 2026 Tech Style Ltd
|
|
16
|
+
|
|
17
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
18
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
19
|
+
in the Software without restriction, including without limitation the rights
|
|
20
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
21
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
22
|
+
furnished to do so, subject to the following conditions:
|
|
23
|
+
|
|
24
|
+
The above copyright notice and this permission notice shall be included in all
|
|
25
|
+
copies or substantial portions of the Software.
|
|
26
|
+
|
|
27
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
28
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
29
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
30
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
31
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
32
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
33
|
+
SOFTWARE.
|
|
34
|
+
License-File: LICENSE
|
|
35
|
+
Keywords: api-response,bandwidth,columnar,compress,compression,decompress,deduplicate,dictionary,gzip,json,key-catalogue,lossless,minify,pack,payload,serialization,shrink,unpack,zero-dependencies
|
|
36
|
+
Classifier: Development Status :: 5 - Production/Stable
|
|
37
|
+
Classifier: Intended Audience :: Developers
|
|
38
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
39
|
+
Classifier: Operating System :: OS Independent
|
|
40
|
+
Classifier: Programming Language :: Python :: 3
|
|
41
|
+
Classifier: Programming Language :: Python :: 3.9
|
|
42
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
43
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
44
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
45
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
46
|
+
Classifier: Programming Language :: Python :: Implementation :: CPython
|
|
47
|
+
Classifier: Programming Language :: Python :: Implementation :: PyPy
|
|
48
|
+
Classifier: Topic :: Internet :: WWW/HTTP
|
|
49
|
+
Classifier: Topic :: Software Development :: Libraries :: Python Modules
|
|
50
|
+
Classifier: Topic :: System :: Archiving :: Compression
|
|
51
|
+
Classifier: Typing :: Typed
|
|
52
|
+
Requires-Python: >=3.9
|
|
53
|
+
Description-Content-Type: text/markdown
|
|
54
|
+
|
|
55
|
+
# json-compress
|
|
56
|
+
|
|
57
|
+
**Lossless JSON compression that is still JSON.** No dependencies, no binary
|
|
58
|
+
framing, no C extension. Typically 45–75% smaller, and an exact round trip.
|
|
59
|
+
|
|
60
|
+
[](https://pypi.org/project/json-compress/)
|
|
61
|
+
[](https://pypi.org/project/json-compress/)
|
|
62
|
+
[](https://github.com/eharain/JSON-Compress/blob/main/LICENSE)
|
|
63
|
+
[](https://github.com/eharain/JSON-Compress#one-format-six-languages-the-same-bytes)
|
|
64
|
+
|
|
65
|
+
**One of six implementations of one wire format** — JavaScript, .NET, Python,
|
|
66
|
+
PHP, Go and Rust. Compress in any of them, restore in any other, byte for byte.
|
|
67
|
+
[How that is guaranteed ↓](#the-same-format-in-six-languages)
|
|
68
|
+
|
|
69
|
+
```bash
|
|
70
|
+
pip install json-compress
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
```python
|
|
74
|
+
from json_compress import compress, decompress
|
|
75
|
+
|
|
76
|
+
users = [
|
|
77
|
+
{"id": 1, "name": "Ada", "role": "admin", "country": "United Kingdom"},
|
|
78
|
+
{"id": 2, "name": "Bob", "role": "admin", "country": "United Kingdom"},
|
|
79
|
+
{"id": 3, "name": "Cy", "role": "user", "country": "United Kingdom"},
|
|
80
|
+
]
|
|
81
|
+
|
|
82
|
+
small = compress(users)
|
|
83
|
+
# {"jc": 1,
|
|
84
|
+
# "k": ["id", "name", "role", "country"],
|
|
85
|
+
# "d": {"~t": [0, 1, 2, 3],
|
|
86
|
+
# "~e": [0, 0, 0, 2],
|
|
87
|
+
# "~cv": ["United Kingdom"],
|
|
88
|
+
# "~r": [[1, "Ada", "admin"], [2, "Bob", "admin"], [3, "Cy", "user"]]}}
|
|
89
|
+
# 'country' never changes, so it is stored once. Three rows is too few for the
|
|
90
|
+
# string catalogue to pay off on 'role' as well - at a thousand rows it would.
|
|
91
|
+
|
|
92
|
+
decompress(small) # exactly the list you started with
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
This is the Python implementation of the
|
|
96
|
+
[json-compress format](https://github.com/eharain/JSON-Compress/blob/main/SPEC.md).
|
|
97
|
+
A document written here reads correctly in the
|
|
98
|
+
[JavaScript](https://github.com/eharain/JSON-Compress/tree/main/javascript) and
|
|
99
|
+
[.NET](https://github.com/eharain/JSON-Compress/tree/main/dotnet)
|
|
100
|
+
implementations, and the other way round — which is not a claim, but a
|
|
101
|
+
[test](tests/test_conformance.py) run against a corpus the JavaScript encoder
|
|
102
|
+
produced.
|
|
103
|
+
|
|
104
|
+
---
|
|
105
|
+
|
|
106
|
+
## Why
|
|
107
|
+
|
|
108
|
+
JSON spends most of its bytes saying the same things over and over. A thousand
|
|
109
|
+
records is a thousand copies of every key name; a `status` column with three
|
|
110
|
+
possible values is a thousand copies of three short strings; a `currency` field
|
|
111
|
+
that is always `"GBP"` is a thousand copies of one.
|
|
112
|
+
|
|
113
|
+
This writes each of those down once.
|
|
114
|
+
|
|
115
|
+
| | |
|
|
116
|
+
|---|---|
|
|
117
|
+
| **Key catalogue** | Every key name is written once and referenced by a one- or two-character token. |
|
|
118
|
+
| **Columnar tables** | A list of like-shaped dicts becomes a column list and a rows matrix — the keys leave the rows entirely. |
|
|
119
|
+
| **Constant columns** | A column whose value never changes is lifted out of the rows and stored once. |
|
|
120
|
+
| **Dictionary columns** | A column drawn from a small set of strings becomes integer indices. |
|
|
121
|
+
| **String catalogue** | Any string value that repeats often enough to pay for itself is written once. |
|
|
122
|
+
|
|
123
|
+
The output is still ordinary JSON, so it goes anywhere JSON goes: an HTTP body,
|
|
124
|
+
a WebSocket frame, a `jsonb` column, a queue message, a log line. Nothing in
|
|
125
|
+
your stack needs to learn a new format.
|
|
126
|
+
|
|
127
|
+
## What it measures
|
|
128
|
+
|
|
129
|
+
From `python examples/benchmark.py`, which round-trips every case before
|
|
130
|
+
reporting it:
|
|
131
|
+
|
|
132
|
+
| Scenario | Minified | Compressed | Saved | + gzip | Saved |
|
|
133
|
+
|---|---:|---:|---:|---:|---:|
|
|
134
|
+
| API record set (1,000 users) | 206.2 KB | 54.8 KB | **73%** | 13.4 KB | 20% |
|
|
135
|
+
| Nested API response (orders with lines) | 96.5 KB | 39.9 KB | **59%** | 9.5 KB | 5% |
|
|
136
|
+
| Time series (2,000 readings) | 166.7 KB | 44.3 KB | **73%** | 11.5 KB | 19% |
|
|
137
|
+
| Structured logs (1,500 lines) | 237.3 KB | 57.3 KB | **76%** | 21.8 KB | 14% |
|
|
138
|
+
| GeoJSON feature collection (400 points) | 58.9 KB | 31.6 KB | **46%** | 7.1 KB | 5% |
|
|
139
|
+
| Config document (deeply nested, few repeats) | 3.1 KB | 2.2 KB | **29%** | 0.5 KB | −21% |
|
|
140
|
+
|
|
141
|
+
The last two columns are the honest ones. Most JSON meets gzip on the way out,
|
|
142
|
+
and gzip already removes a lot of this redundancy — so the question is what is
|
|
143
|
+
left. On record-shaped data, a further 5–20% on top of gzip. On a small config
|
|
144
|
+
file with little repetition, the envelope costs more than it saves, and the
|
|
145
|
+
table says so.
|
|
146
|
+
|
|
147
|
+
The first four columns match the other five implementations' benchmarks figure
|
|
148
|
+
for figure, because all six build the same documents from the same generator and
|
|
149
|
+
all six encoders write the same bytes. The gzip columns are a little
|
|
150
|
+
different: zlib and the browser `CompressionStream` do not tune deflate
|
|
151
|
+
identically.
|
|
152
|
+
|
|
153
|
+
**Use it for record sets, logs, time series and API responses.** For a small
|
|
154
|
+
document you already gzip, it will not earn its keep — measure with
|
|
155
|
+
`measure()` before you commit.
|
|
156
|
+
|
|
157
|
+
## Where it helps most
|
|
158
|
+
|
|
159
|
+
- **API responses.** Less to serialise, less to send, less to parse.
|
|
160
|
+
- **Cache entries.** Redis and friends charge by the byte.
|
|
161
|
+
- **WebSocket and Channels traffic**, where per-message compression is often off.
|
|
162
|
+
- **Queue and task payloads** with per-message size limits.
|
|
163
|
+
- **JSON columns in a database.** Postgres `jsonb`, SQLite `TEXT`: smaller rows,
|
|
164
|
+
smaller indexes, smaller backups.
|
|
165
|
+
- **Logs and telemetry**, which are almost entirely repeated keys.
|
|
166
|
+
|
|
167
|
+
---
|
|
168
|
+
|
|
169
|
+
## Install
|
|
170
|
+
|
|
171
|
+
```bash
|
|
172
|
+
pip install json-compress
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
Python 3.9 and later. No dependencies — everything it needs is in the standard
|
|
176
|
+
library. The command line comes with the package:
|
|
177
|
+
|
|
178
|
+
```bash
|
|
179
|
+
json-compress stats orders.json
|
|
180
|
+
```
|
|
181
|
+
|
|
182
|
+
---
|
|
183
|
+
|
|
184
|
+
## API
|
|
185
|
+
|
|
186
|
+
Everything is importable from the top-level package.
|
|
187
|
+
|
|
188
|
+
### `compress(value, options=None, **overrides) -> dict`
|
|
189
|
+
|
|
190
|
+
Compress a document. The result is an ordinary dict of JSON-shaped values, so it
|
|
191
|
+
can be handed to `json.dumps`, stored, posted or logged anywhere plain JSON
|
|
192
|
+
goes. The input is left untouched.
|
|
193
|
+
|
|
194
|
+
### `decompress(envelope) -> Any`
|
|
195
|
+
|
|
196
|
+
Restore it. Raises `DecodeError` if the envelope is not readable, rather than
|
|
197
|
+
returning something nearly right. `DecodeError` is a `ValueError`, so it is
|
|
198
|
+
caught by the handler you already have.
|
|
199
|
+
|
|
200
|
+
### `stringify(value, options=None, indent=None)` and `parse(text)`
|
|
201
|
+
|
|
202
|
+
The same pair at the string level.
|
|
203
|
+
|
|
204
|
+
```python
|
|
205
|
+
text = stringify(orders) # compress, then serialise
|
|
206
|
+
back = parse(text) # parse, then restore
|
|
207
|
+
```
|
|
208
|
+
|
|
209
|
+
### `decompress_if_needed(value)` and `is_compressed(value)`
|
|
210
|
+
|
|
211
|
+
Restores a compressed value, passes anything else straight through. Useful at a
|
|
212
|
+
boundary you do not fully control — an endpoint mid-migration, a cache holding
|
|
213
|
+
entries written by two versions of the same service.
|
|
214
|
+
|
|
215
|
+
### `measure(value, options=None, source=None, gzip=True) -> SizeReport`
|
|
216
|
+
|
|
217
|
+
What compression would buy, before you commit to it — here on the 1,000-user
|
|
218
|
+
record set the benchmark builds:
|
|
219
|
+
|
|
220
|
+
```python
|
|
221
|
+
report = measure(users)
|
|
222
|
+
report.minified # 211180
|
|
223
|
+
report.compressed # 56136
|
|
224
|
+
report.percent # 73.4
|
|
225
|
+
report.gzip_minified # 17168
|
|
226
|
+
report.gzip_compressed # 13740
|
|
227
|
+
```
|
|
228
|
+
|
|
229
|
+
`ratio` and `percent` are measured against **minified** JSON, not against
|
|
230
|
+
whatever whitespace the input arrived with. Pass `source=` if you want
|
|
231
|
+
`original` to reflect the real input, and `gzip=False` to skip the gzip
|
|
232
|
+
measurements.
|
|
233
|
+
|
|
234
|
+
### `create_codec(options=None, **overrides)`
|
|
235
|
+
|
|
236
|
+
Bind options once — the natural unit for a transport layer or a storage adapter.
|
|
237
|
+
It holds nothing but its frozen options, so one instance is safe to share across
|
|
238
|
+
threads:
|
|
239
|
+
|
|
240
|
+
```python
|
|
241
|
+
CODEC = create_codec(min_rows=10)
|
|
242
|
+
|
|
243
|
+
CODEC.stringify(payload)
|
|
244
|
+
CODEC.parse(text)
|
|
245
|
+
```
|
|
246
|
+
|
|
247
|
+
### `validate(text)` and `locate(text)`
|
|
248
|
+
|
|
249
|
+
Check a JSON document and say exactly where it breaks — line, column, the line
|
|
250
|
+
itself, and what is wrong in plain words:
|
|
251
|
+
|
|
252
|
+
```python
|
|
253
|
+
locate('{\n "a": 1,\n "b": oops\n}')
|
|
254
|
+
# JsonLocation(valid=False, error='expected a value, found "o"',
|
|
255
|
+
# position=19, line=3, column=8, excerpt=' "b": oops')
|
|
256
|
+
```
|
|
257
|
+
|
|
258
|
+
Parser messages are written for parser authors and often name the wrong culprit,
|
|
259
|
+
so this scans the document itself against RFC 8259. It knows about single
|
|
260
|
+
quotes, trailing commas, `True`/`None`, leading zeros, unterminated strings and
|
|
261
|
+
bad escapes. A `JsonLocation` is truthy when the document is valid, so
|
|
262
|
+
`if validate(text):` reads the way you would expect.
|
|
263
|
+
|
|
264
|
+
### `minify(text)` and `beautify(text, indent=2)`
|
|
265
|
+
|
|
266
|
+
Whitespace only, nothing else touched. Both write line feeds whatever the
|
|
267
|
+
platform, so a document beautified on Windows matches the same document
|
|
268
|
+
beautified anywhere else.
|
|
269
|
+
|
|
270
|
+
### `byte_length`, `gzip_bytes`, `gunzip`, `gzip_size`
|
|
271
|
+
|
|
272
|
+
UTF-8 byte counts — what actually goes over the wire — and gzip with a zeroed
|
|
273
|
+
timestamp, so the same input always measures the same.
|
|
274
|
+
|
|
275
|
+
### `pack(records)` and `unpack(packed)`
|
|
276
|
+
|
|
277
|
+
The columnar idea on its own, with no envelope and no catalogue:
|
|
278
|
+
|
|
279
|
+
```python
|
|
280
|
+
pack([{"id": 1, "name": "Ada"}, {"id": 2, "name": "Bob"}])
|
|
281
|
+
# {'fields': ['id', 'name'], 'data': [[1, 'Ada'], [2, 'Bob']], 'count': 2}
|
|
282
|
+
```
|
|
283
|
+
|
|
284
|
+
Useful when the other end is not Python, or when you want something a human can
|
|
285
|
+
read straight off a network panel. `pack_all` / `unpack_all` do the same for a
|
|
286
|
+
list of result sets, which is what multi-statement database drivers return. Both
|
|
287
|
+
leave anything that is not the expected shape alone, so they can be dropped into
|
|
288
|
+
a pipeline without a shape check first.
|
|
289
|
+
|
|
290
|
+
---
|
|
291
|
+
|
|
292
|
+
## Options
|
|
293
|
+
|
|
294
|
+
```python
|
|
295
|
+
from json_compress import CompressOptions, compress
|
|
296
|
+
|
|
297
|
+
compress(value, CompressOptions(
|
|
298
|
+
strings=True, # de-duplicate repeated string values
|
|
299
|
+
tables=True, # pack lists of like-shaped dicts into tables
|
|
300
|
+
constants=True, # hoist unchanging columns out of the rows
|
|
301
|
+
dict_columns=True, # encode low-cardinality string columns as indices
|
|
302
|
+
min_rows=2, # smallest list that may become a table
|
|
303
|
+
max_hole_ratio=0.3, # largest share of missing cells a table may carry
|
|
304
|
+
min_string_count=2, # times a string must occur before it can be catalogued
|
|
305
|
+
min_string_length=1, # shortest string that may be catalogued
|
|
306
|
+
))
|
|
307
|
+
```
|
|
308
|
+
|
|
309
|
+
Options may equally be a plain mapping or keyword arguments, whichever reads
|
|
310
|
+
better at the call site:
|
|
311
|
+
|
|
312
|
+
```python
|
|
313
|
+
compress(value, {"tables": False})
|
|
314
|
+
compress(value, min_rows=10)
|
|
315
|
+
```
|
|
316
|
+
|
|
317
|
+
Every default is safe on arbitrary input, and every field validates on
|
|
318
|
+
construction, so a nonsense value surfaces at the call site rather than as a
|
|
319
|
+
mysteriously large output. The two worth reaching for are `min_rows`, if you
|
|
320
|
+
have many tiny lists and want them left alone, and `max_hole_ratio`, if your
|
|
321
|
+
records are sparse and you would rather pack them anyway.
|
|
322
|
+
|
|
323
|
+
---
|
|
324
|
+
|
|
325
|
+
## What survives the round trip
|
|
326
|
+
|
|
327
|
+
Everything JSON can express, exactly:
|
|
328
|
+
|
|
329
|
+
- **Key order**, including in packed tables. Where one column order cannot
|
|
330
|
+
satisfy every row, the list is left as dicts rather than quietly reordered.
|
|
331
|
+
- **A missing key versus a null one.** `{"a": 1}` and `{"a": 1, "b": None}` in
|
|
332
|
+
the same list stay different.
|
|
333
|
+
- **Any string**, including ones starting with a tilde, containing control
|
|
334
|
+
characters, made of astral-plane characters, or holding an unpaired surrogate.
|
|
335
|
+
- **Keys that look like numbers**, which most catalogue schemes reorder — and
|
|
336
|
+
which would be silently reordered by a JavaScript reader if the tokens were
|
|
337
|
+
not letter-first.
|
|
338
|
+
- **A key literally named `__proto__`**, which is ordinary data here and a trap
|
|
339
|
+
in some of the languages this format travels to.
|
|
340
|
+
|
|
341
|
+
Values JSON cannot carry are treated the way JSON treats them: non-finite
|
|
342
|
+
numbers become `None`, negative zero becomes zero, and a cycle raises rather
|
|
343
|
+
than hanging. Types the standard library can describe are converted the obvious
|
|
344
|
+
way — `datetime`, `date` and `time` to ISO 8601 text, `Decimal` to a number,
|
|
345
|
+
`UUID` and `Enum` to their value, dataclasses to their fields, tuples to lists.
|
|
346
|
+
Anything else may define a `for_json()` method, which is the nearest thing
|
|
347
|
+
Python has to JavaScript's `toJSON`.
|
|
348
|
+
|
|
349
|
+
This is checked by 2,000 randomly generated documents on every test run, each
|
|
350
|
+
compared character for character against what it started as, plus every
|
|
351
|
+
combination of the options above, plus a corpus written by the JavaScript
|
|
352
|
+
implementation.
|
|
353
|
+
|
|
354
|
+
The one bound worth knowing is nesting depth. Compressing walks the document
|
|
355
|
+
recursively, so a document nested deeper than the interpreter's recursion limit
|
|
356
|
+
raises `RecursionError` rather than being silently truncated — around 1,000
|
|
357
|
+
levels by default, which `sys.setrecursionlimit` will raise if you genuinely
|
|
358
|
+
have such a document. `locate` is iterative and has no such limit.
|
|
359
|
+
|
|
360
|
+
### Numbers are written the way every implementation writes them
|
|
361
|
+
|
|
362
|
+
The one place this package deliberately parts company with `json.dumps` is
|
|
363
|
+
float formatting. `repr` switches to exponent form at 1e16 and below 1e-4, so
|
|
364
|
+
`json.dumps` writes `1e+18` and `1e-05`; the format follows the ECMAScript rule
|
|
365
|
+
instead — positional between 1e-7 and 1e21 — because that is what the wire
|
|
366
|
+
format is defined against and what every document in the conformance corpus was
|
|
367
|
+
written with.
|
|
368
|
+
|
|
369
|
+
The digits themselves are identical: both languages print the shortest decimal
|
|
370
|
+
that reads back as the same double. Integers keep Python's exact precision
|
|
371
|
+
rather than being folded through a float, so a value too large for a double
|
|
372
|
+
survives here where it would not in JavaScript.
|
|
373
|
+
|
|
374
|
+
---
|
|
375
|
+
|
|
376
|
+
## Command line
|
|
377
|
+
|
|
378
|
+
```bash
|
|
379
|
+
json-compress stats orders.json
|
|
380
|
+
json-compress compress orders.json -o orders.jc.json
|
|
381
|
+
json-compress decompress orders.jc.json --pretty
|
|
382
|
+
curl -s https://api.example.com/orders | json-compress compress > orders.jc.json
|
|
383
|
+
```
|
|
384
|
+
|
|
385
|
+
`stats` changes nothing and reports what compression would save, with and
|
|
386
|
+
without gzip. `validate`, `minify` and `beautify` are there too. Run
|
|
387
|
+
`json-compress --help` for the full list, or `python -m json_compress` if you
|
|
388
|
+
would rather not put the script on your PATH.
|
|
389
|
+
|
|
390
|
+
---
|
|
391
|
+
|
|
392
|
+
## The format
|
|
393
|
+
|
|
394
|
+
The wire format is fully specified in
|
|
395
|
+
[`SPEC.md`](https://github.com/eharain/JSON-Compress/blob/main/SPEC.md) — enough
|
|
396
|
+
to write an implementation from, in any language, without reading this one. In
|
|
397
|
+
outline:
|
|
398
|
+
|
|
399
|
+
```json
|
|
400
|
+
{
|
|
401
|
+
"jc": 1,
|
|
402
|
+
"k": ["the", "key", "catalogue"],
|
|
403
|
+
"s": ["repeated", "string", "values"],
|
|
404
|
+
"d": "the encoded document"
|
|
405
|
+
}
|
|
406
|
+
```
|
|
407
|
+
|
|
408
|
+
Keys become short letter-first tokens; a tilde marks a catalogue reference or an
|
|
409
|
+
escaped literal; a `~t` member marks a columnar table. That is the whole of it.
|
|
410
|
+
|
|
411
|
+
---
|
|
412
|
+
|
|
413
|
+
## The same format, in six languages
|
|
414
|
+
|
|
415
|
+
This is one of six implementations of the same wire format. They do not merely
|
|
416
|
+
interoperate: given the same document they emit **the same bytes**. A payload
|
|
417
|
+
compressed by any one of them is restored exactly by any other, in any
|
|
418
|
+
direction.
|
|
419
|
+
|
|
420
|
+
| Language | Package | Install | |
|
|
421
|
+
|---|---|---|---|
|
|
422
|
+
| JavaScript / TypeScript | [`@tech-style/json-compress`](https://www.npmjs.com/package/@tech-style/json-compress) on npm | `npm install @tech-style/json-compress` | |
|
|
423
|
+
| .NET | [`TechStyle.JsonCompress`](https://www.nuget.org/packages/TechStyle.JsonCompress) on NuGet | `dotnet add package TechStyle.JsonCompress` | |
|
|
424
|
+
| **Python** | [`json-compress`](https://pypi.org/project/json-compress/) on PyPI | `pip install json-compress` | **this one** |
|
|
425
|
+
| PHP | [`tech-style/json-compress`](https://packagist.org/packages/tech-style/json-compress) on Packagist | `composer require tech-style/json-compress` | |
|
|
426
|
+
| Go | [`github.com/eharain/JSON-Compress/go`](https://pkg.go.dev/github.com/eharain/JSON-Compress/go) | `go get github.com/eharain/JSON-Compress/go` | |
|
|
427
|
+
| Rust | [`json-compress`](https://crates.io/crates/json-compress) on crates.io | `cargo add json-compress` | |
|
|
428
|
+
|
|
429
|
+
Java is planned, against the same specification.
|
|
430
|
+
|
|
431
|
+
Every one of them ships the same command line, called `json-compress`, so the
|
|
432
|
+
guarantee is something you can watch happen:
|
|
433
|
+
|
|
434
|
+
```bash
|
|
435
|
+
json-compress compress orders.json > orders.jc.json # in any language
|
|
436
|
+
json-compress decompress orders.jc.json > restored.json # in any other
|
|
437
|
+
diff orders.json restored.json # no output
|
|
438
|
+
```
|
|
439
|
+
|
|
440
|
+
### How that is kept true
|
|
441
|
+
|
|
442
|
+
A corpus of **144 documents** — the awkward cases from the specification, plus
|
|
443
|
+
120 generated from a seeded source every suite reproduces independently — is
|
|
444
|
+
checked in once at
|
|
445
|
+
[`dotnet/tests/.../conformance.json`](https://github.com/eharain/JSON-Compress/blob/main/dotnet/tests/TechStyle.JsonCompress.Tests/fixtures/conformance.json),
|
|
446
|
+
each paired with the envelope the reference encoder wrote for it.
|
|
447
|
+
|
|
448
|
+
Every implementation asserts **both directions** against that one shared file on
|
|
449
|
+
every CI run: it must restore all 144 envelopes to the original value, and it
|
|
450
|
+
must write those same 144 envelopes itself. The specification only requires the
|
|
451
|
+
first. Doing the second as well is what turns "compatible" into "identical".
|
|
452
|
+
|
|
453
|
+
The cases are the awkward ones on purpose — the tilde escape, holes standing
|
|
454
|
+
against real nulls, `__proto__` as a key name, key names shaped like tokens,
|
|
455
|
+
catalogues large enough to need two-character tokens, rows whose key orders
|
|
456
|
+
disagree, a raw column whose last cell is `-1`. Each of those has broken an
|
|
457
|
+
implementation at some point.
|
|
458
|
+
|
|
459
|
+
The six agree on every document JavaScript could have written, and diverge only
|
|
460
|
+
where the languages' own number models do — on JSON text `JSON.stringify` could
|
|
461
|
+
not have produced in the first place. The [root
|
|
462
|
+
README](https://github.com/eharain/JSON-Compress#where-the-languages-are-allowed-to-differ) lists those cases
|
|
463
|
+
precisely. Each implementation still restores its own documents exactly, which
|
|
464
|
+
is all the format promises.
|
|
465
|
+
|
|
466
|
+
---
|
|
467
|
+
|
|
468
|
+
## Building and testing
|
|
469
|
+
|
|
470
|
+
```bash
|
|
471
|
+
python -m unittest discover tests # 110 tests, including the JavaScript corpus
|
|
472
|
+
python examples/benchmark.py
|
|
473
|
+
python -m build
|
|
474
|
+
```
|
|
475
|
+
|
|
476
|
+
`pytest` runs the same suite unchanged if you prefer it.
|
|
477
|
+
|
|
478
|
+
The conformance corpus lives at
|
|
479
|
+
`dotnet/tests/TechStyle.JsonCompress.Tests/fixtures/conformance.json` and is
|
|
480
|
+
shared with the .NET suite rather than duplicated, so the Python tests prove
|
|
481
|
+
this implementation agrees with the reference without Node being installed. To
|
|
482
|
+
check the other direction as well:
|
|
483
|
+
|
|
484
|
+
```bash
|
|
485
|
+
node tests/interop/verify-interop.mjs # every case through both implementations
|
|
486
|
+
```
|
|
487
|
+
|
|
488
|
+
---
|
|
489
|
+
|
|
490
|
+
## Licence
|
|
491
|
+
|
|
492
|
+
MIT. Free for commercial use.
|
|
493
|
+
|
|
494
|
+
Built by [Tech Style Ltd](https://tech-style.co/) — issues and pull requests
|
|
495
|
+
welcome at [github.com/eharain/JSON-Compress](https://github.com/eharain/JSON-Compress).
|