liquefy 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- liquefy-0.1.0/LICENSE +26 -0
- liquefy-0.1.0/PKG-INFO +291 -0
- liquefy-0.1.0/README.md +267 -0
- liquefy-0.1.0/pyproject.toml +41 -0
- liquefy-0.1.0/setup.cfg +4 -0
- liquefy-0.1.0/src/liquefy/__init__.py +18 -0
- liquefy-0.1.0/src/liquefy/_api.py +127 -0
- liquefy-0.1.0/src/liquefy/cli.py +73 -0
- liquefy-0.1.0/src/liquefy.egg-info/PKG-INFO +291 -0
- liquefy-0.1.0/src/liquefy.egg-info/SOURCES.txt +12 -0
- liquefy-0.1.0/src/liquefy.egg-info/dependency_links.txt +1 -0
- liquefy-0.1.0/src/liquefy.egg-info/entry_points.txt +2 -0
- liquefy-0.1.0/src/liquefy.egg-info/requires.txt +3 -0
- liquefy-0.1.0/src/liquefy.egg-info/top_level.txt +1 -0
liquefy-0.1.0/LICENSE
ADDED
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Parad0x Labs
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
22
|
+
|
|
23
|
+
---
|
|
24
|
+
|
|
25
|
+
This license covers all source code, documentation, scripts, and specifications
|
|
26
|
+
in this repository, including the compression engine source in engines/.
|
liquefy-0.1.0/PKG-INFO
ADDED
|
@@ -0,0 +1,291 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: liquefy
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Columnar compression that beats Zstd on structured data. 61x on JSON. Built-in search. AES-256-GCM.
|
|
5
|
+
Author-email: Parad0x Labs <labs@parad0x.io>
|
|
6
|
+
License: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/Parad0x-Labs/liquefy
|
|
8
|
+
Project-URL: Repository, https://github.com/Parad0x-Labs/liquefy
|
|
9
|
+
Project-URL: DNA x402, https://github.com/Parad0x-Labs/dna-x402
|
|
10
|
+
Keywords: compression,zstd,columnar,json,structured-data,log-compression,encryption,aes-256-gcm,ai-agents,solana,x402,telemetry,analytics
|
|
11
|
+
Classifier: Development Status :: 4 - Beta
|
|
12
|
+
Classifier: Intended Audience :: Developers
|
|
13
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
14
|
+
Classifier: Programming Language :: Python :: 3
|
|
15
|
+
Classifier: Topic :: System :: Archiving :: Compression
|
|
16
|
+
Classifier: Topic :: Database
|
|
17
|
+
Requires-Python: >=3.9
|
|
18
|
+
Description-Content-Type: text/markdown
|
|
19
|
+
License-File: LICENSE
|
|
20
|
+
Requires-Dist: zstandard>=0.21.0
|
|
21
|
+
Requires-Dist: xxhash>=3.0.0
|
|
22
|
+
Requires-Dist: cryptography>=41.0.0
|
|
23
|
+
Dynamic: license-file
|
|
24
|
+
|
|
25
|
+
# Liquefy
|
|
26
|
+
|
|
27
|
+
**Columnar compression that beats Zstd on structured data. Built-in search. Built-in encryption. MIT.**
|
|
28
|
+
|
|
29
|
+

|
|
30
|
+

|
|
31
|
+

|
|
32
|
+

|
|
33
|
+
|
|
34
|
+
---
|
|
35
|
+
|
|
36
|
+
## What happens to 1,000,000 agent payments
|
|
37
|
+
|
|
38
|
+
```mermaid
|
|
39
|
+
flowchart LR
|
|
40
|
+
A([🤖 1,000,000\nreceipts / day]) --> B[Net bilateral flows]
|
|
41
|
+
B --> C([📉 ~4,950\nnet settlements])
|
|
42
|
+
C --> D[Columnar Gun v1\n62× compression]
|
|
43
|
+
D --> E([📦 ~2.6 MB\ncompressed])
|
|
44
|
+
E --> F[AES-256-GCM\nencryption]
|
|
45
|
+
F --> G([🔒 private\nbatch])
|
|
46
|
+
G --> H[receipt_anchor\nSolana mainnet]
|
|
47
|
+
H --> I([⛓️ 1 tx\non-chain])
|
|
48
|
+
|
|
49
|
+
style A fill:#1a1a2e,color:#e0e0ff,stroke:#444
|
|
50
|
+
style C fill:#0d2137,color:#7ecfff,stroke:#2a6496
|
|
51
|
+
style E fill:#0d2137,color:#7ecfff,stroke:#2a6496
|
|
52
|
+
style G fill:#1a0d37,color:#c87eff,stroke:#6a2a96
|
|
53
|
+
style I fill:#0d3722,color:#7effb2,stroke:#2a9657
|
|
54
|
+
style B fill:#111,color:#aaa,stroke:#333
|
|
55
|
+
style D fill:#111,color:#aaa,stroke:#333
|
|
56
|
+
style F fill:#111,color:#aaa,stroke:#333
|
|
57
|
+
style H fill:#111,color:#aaa,stroke:#333
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
> 1 million receipts. 1 on-chain transaction. Only the parties see the amounts.
|
|
61
|
+
|
|
62
|
+
---
|
|
63
|
+
|
|
64
|
+
## The number that matters
|
|
65
|
+
|
|
66
|
+
| Tool | Compression ratio | Search latency | Notes |
|
|
67
|
+
|------|-------------------|----------------|-------|
|
|
68
|
+
| **Liquefy Columnar Gun v1** | **33–61×** | **4–6 ms** | columnar transpose + type-aware encoding + Zstd |
|
|
69
|
+
| Zstd L19 | 5–43× | 26–245 ms (full decompress required) | best-in-class general compressor |
|
|
70
|
+
| gzip -9 | ~5–12× | — | baseline |
|
|
71
|
+
|
|
72
|
+
**Two wins, not one.**
|
|
73
|
+
|
|
74
|
+
- **Compression:** 1.4–6× better ratio than Zstd depending on data repetitiveness. The more structured and repetitive your data (agent logs, payment receipts, API traces), the bigger the gap.
|
|
75
|
+
- **Search:** 5–61× faster than Zstd — because Liquefy decompresses only the queried column, not the entire blob. Zstd has no choice but to decompress everything.
|
|
76
|
+
|
|
77
|
+
Both numbers are real. Run `python tools/benchmark.py` on your own data — ratio depends on how repetitive your fields are, search speed advantage is consistent.
|
|
78
|
+
|
|
79
|
+
### Proof — run it yourself
|
|
80
|
+
|
|
81
|
+
```bash
|
|
82
|
+
python tools/benchmark.py # reproduces these numbers in ~10 seconds
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
Or read the reports — all in the repo, all verified:
|
|
86
|
+
|
|
87
|
+
| Document | What it proves |
|
|
88
|
+
|----------|---------------|
|
|
89
|
+
| [UNICORN_BENCHMARK.md](./REPORTS/UNICORN_BENCHMARK.md) | Full head-to-head vs Zstd L19 — ratio + search latency + methodology |
|
|
90
|
+
| [ENTERPRISE_CERTIFICATION_V1.md](./REPORTS/ENTERPRISE_CERTIFICATION_V1.md) | Bit-perfect round-trip certification across all 23 codecs |
|
|
91
|
+
| [ULTIMATE_TEST_LOGS.md](./REPORTS/ULTIMATE_TEST_LOGS.md) | Raw test output — every engine, every run |
|
|
92
|
+
| [SEARCHABLE_GLACIER_PROOF.md](./REPORTS/SEARCHABLE_GLACIER_PROOF.md) | Column-skip search proof — O(k) vs O(n) |
|
|
93
|
+
| [VERIFICATION_REPORT.md](./REPORTS/VERIFICATION_REPORT.md) | Independent verification of compression ratios |
|
|
94
|
+
|
|
95
|
+
### Sample data + hashes
|
|
96
|
+
|
|
97
|
+
`proof-pack/` ships a real nginx log + its compressed `.null` archive with SHA-256 hashes. Decompress it, hash the output, compare. The bytes match or the tool is wrong.
|
|
98
|
+
|
|
99
|
+
```bash
|
|
100
|
+
# verify the included sample yourself
|
|
101
|
+
./liquefy decompress proof-pack/samples/compressed/sample_nginx.null restored.log
|
|
102
|
+
sha256sum restored.log # must match proof-pack/hashes.txt
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
---
|
|
106
|
+
|
|
107
|
+
## Why it works
|
|
108
|
+
|
|
109
|
+
General compressors treat your data as a byte stream. Liquefy reads the schema first.
|
|
110
|
+
|
|
111
|
+
```
|
|
112
|
+
BEFORE (row layout — what every other tool sees):
|
|
113
|
+
{"ts":1700000001,"src":"agent-A","dst":"agent-B","amount":1000}
|
|
114
|
+
{"ts":1700000002,"src":"agent-A","dst":"agent-B","amount":1001}
|
|
115
|
+
{"ts":1700000003,"src":"agent-C","dst":"agent-B","amount":1000}
|
|
116
|
+
|
|
117
|
+
AFTER (column layout — what Liquefy compresses):
|
|
118
|
+
ts: [1700000001, 1700000002, 1700000003] → delta-encode → tiny
|
|
119
|
+
src: ["agent-A", "agent-A", "agent-C"] → dictionary → 1 byte per row
|
|
120
|
+
dst: ["agent-B", "agent-B", "agent-B"] → dictionary → 1 byte per row
|
|
121
|
+
amount: [1000, 1001, 1000] → delta-encode → tiny
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
Repeated values compress to a single dictionary entry. Sequential numbers compress to their deltas. Each column is independently Zstd-compressed. The result beats the general-purpose best.
|
|
125
|
+
|
|
126
|
+
---
|
|
127
|
+
|
|
128
|
+
## What it does beyond compression
|
|
129
|
+
|
|
130
|
+
**Search without decompressing.** Zone maps (min/max per column) let you skip entire blocks without reading the data. Point queries on timestamps or IDs touch only the relevant columns.
|
|
131
|
+
|
|
132
|
+
**Encryption.** AES-256-GCM with PBKDF2 multi-tenant key derivation. Optional, zero-overhead when not used. SOC 2 / FedRAMP compliant key handling.
|
|
133
|
+
|
|
134
|
+
**Bit-perfect restoration.** Every archive is round-trip verified. Compressed bytes decompress to the exact original bytes, every time. [Certification report](./REPORTS/ENTERPRISE_CERTIFICATION_V1.md).
|
|
135
|
+
|
|
136
|
+
**23 format-aware codecs.** The orchestrator auto-selects the right one:
|
|
137
|
+
|
|
138
|
+
| Category | Codecs |
|
|
139
|
+
|----------|--------|
|
|
140
|
+
| Structured JSON | Columnar Gun v1 (61×), Entropy-focused, Repetition-focused |
|
|
141
|
+
| Web logs | Nginx (×2), Apache (×2) |
|
|
142
|
+
| Infrastructure | Kubernetes, Syslog (×2), Windows Event Log |
|
|
143
|
+
| Cloud | AWS CloudTrail, VPC Flow |
|
|
144
|
+
| Database | PostgreSQL / SQL (×3) |
|
|
145
|
+
| Network | Netflow V5, GitHub SCM |
|
|
146
|
+
| Fallback | Universal entropy, Universal repetition |
|
|
147
|
+
|
|
148
|
+
---
|
|
149
|
+
|
|
150
|
+
## Real-world use: AI agent payment settlement on Solana
|
|
151
|
+
|
|
152
|
+
Liquefy's columnar algorithm is used in [**DNA x402**](https://github.com/Parad0x-Labs/dna-x402) to compress AI agent payment receipt batches before on-chain anchoring.
|
|
153
|
+
|
|
154
|
+
x402 receipts are structured JSON with highly repetitive fields — same receiver, same program ID, sequential timestamps. The TypeScript port of Columnar Gun achieves **62× compression** on real batches:
|
|
155
|
+
|
|
156
|
+
```
|
|
157
|
+
500 payment receipts → 163 KB raw JSON
|
|
158
|
+
→ net bilateral flows (500 receipts → 2 net settlements)
|
|
159
|
+
→ 2.6 KB compressed (62× columnar)
|
|
160
|
+
→ AES-256-GCM encrypted
|
|
161
|
+
→ 1 on-chain tx (not 500)
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
The anchor program is live on Solana mainnet. The TypeScript port is at [`packages/liquefy-receipts/`](https://github.com/Parad0x-Labs/dna-x402/tree/main/packages/liquefy-receipts).
|
|
165
|
+
|
|
166
|
+
---
|
|
167
|
+
|
|
168
|
+
## Install
|
|
169
|
+
|
|
170
|
+
**One command:**
|
|
171
|
+
```bash
|
|
172
|
+
git clone https://github.com/Parad0x-Labs/liquefy.git && cd liquefy && bash install.sh
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
**Or pip only:**
|
|
176
|
+
```bash
|
|
177
|
+
pip install git+https://github.com/Parad0x-Labs/liquefy.git
|
|
178
|
+
```
|
|
179
|
+
|
|
180
|
+
---
|
|
181
|
+
|
|
182
|
+
## Python SDK
|
|
183
|
+
|
|
184
|
+
```python
|
|
185
|
+
from liquefy import compress, decompress, search
|
|
186
|
+
|
|
187
|
+
# Compress — 33-61× smaller on structured JSON
|
|
188
|
+
blob = compress(open("agent-logs.jsonl", "rb").read())
|
|
189
|
+
|
|
190
|
+
# Decompress — bit-perfect
|
|
191
|
+
original = decompress(blob)
|
|
192
|
+
|
|
193
|
+
# Search without full decompress — 5-61× faster than Zstd
|
|
194
|
+
result = search(blob, "trace-00049999")
|
|
195
|
+
print(result["found"], result["latency_ms"], "ms")
|
|
196
|
+
|
|
197
|
+
# Encrypted (private agent receipts — AES-256-GCM)
|
|
198
|
+
from liquefy import compress_encrypted, decompress_encrypted
|
|
199
|
+
import os
|
|
200
|
+
key = os.urandom(32)
|
|
201
|
+
private_blob = compress_encrypted(data, key)
|
|
202
|
+
data_back = decompress_encrypted(private_blob, key)
|
|
203
|
+
```
|
|
204
|
+
|
|
205
|
+
**CLI (same API):**
|
|
206
|
+
```bash
|
|
207
|
+
liquefy compress input.jsonl output.null
|
|
208
|
+
liquefy decompress output.null restored.jsonl
|
|
209
|
+
liquefy verify input.jsonl # bit-perfect round-trip check
|
|
210
|
+
liquefy search output.null "trace-00049"
|
|
211
|
+
liquefy benchmark # head-to-head vs Zstd
|
|
212
|
+
```
|
|
213
|
+
|
|
214
|
+
---
|
|
215
|
+
|
|
216
|
+
## TypeScript / Node 22+ (via dna-x402)
|
|
217
|
+
|
|
218
|
+
For AI agent payment receipt batching on Solana:
|
|
219
|
+
|
|
220
|
+
```bash
|
|
221
|
+
# inside dna-x402
|
|
222
|
+
npm install # @dna-x402/liquefy-receipts is in packages/liquefy-receipts/
|
|
223
|
+
```
|
|
224
|
+
|
|
225
|
+
```ts
|
|
226
|
+
import { compressReceipts, netReceipts, buildAnchorIxData } from "@dna-x402/liquefy-receipts";
|
|
227
|
+
|
|
228
|
+
const nets = netReceipts(receipts); // bilateral netting
|
|
229
|
+
const compressed = compressReceipts(receipts); // 62× smaller
|
|
230
|
+
const ixData = buildAnchorIxData({ batchBytes: compressed, receiptCount: receipts.length, ... });
|
|
231
|
+
// → 1 Solana tx instead of 1000
|
|
232
|
+
```
|
|
233
|
+
|
|
234
|
+
Source: [`packages/liquefy-receipts/`](https://github.com/Parad0x-Labs/dna-x402/tree/main/packages/liquefy-receipts)
|
|
235
|
+
|
|
236
|
+
---
|
|
237
|
+
|
|
238
|
+
## Benchmark vs Zstd
|
|
239
|
+
|
|
240
|
+
```bash
|
|
241
|
+
python tools/benchmark.py
|
|
242
|
+
```
|
|
243
|
+
|
|
244
|
+
Expected output:
|
|
245
|
+
```
|
|
246
|
+
Standard Zstd (L19): XX MB (Ratio: 5–43×)
|
|
247
|
+
Liquefy COL1 (L22): XX MB (Ratio: 33–61×)
|
|
248
|
+
Search — Zstd: 26–245 ms (must fully decompress)
|
|
249
|
+
Search — Liquefy: 4–6 ms (columnar skip — only reads the queried column)
|
|
250
|
+
```
|
|
251
|
+
|
|
252
|
+
Ratio range depends on data repetitiveness. Search advantage is consistent — it's architectural.
|
|
253
|
+
|
|
254
|
+
---
|
|
255
|
+
|
|
256
|
+
## Engine source (MIT)
|
|
257
|
+
|
|
258
|
+
Everything in `engines/` is MIT. No paywall, no restrictions.
|
|
259
|
+
|
|
260
|
+
```
|
|
261
|
+
engines/
|
|
262
|
+
orchestrator.py — auto-routes to best codec
|
|
263
|
+
json_codec/ — Columnar Gun v1 + variants
|
|
264
|
+
nginx_codec/ — Nginx access log
|
|
265
|
+
apache_codec/ — Apache access log
|
|
266
|
+
syslog_codec/ — Syslog RFC 3164/5424
|
|
267
|
+
k8s_codec/ — Kubernetes JSON stream
|
|
268
|
+
sql_codec/ — PostgreSQL / SQL query log
|
|
269
|
+
aws_codec/ — CloudTrail / VPC Flow
|
|
270
|
+
universal_codec/ — fallback entropy + repetition
|
|
271
|
+
security_compliance.py — AES-256-GCM, PBKDF2, HMAC-SHA256
|
|
272
|
+
tools/
|
|
273
|
+
benchmark.py — head-to-head vs Zstd
|
|
274
|
+
```
|
|
275
|
+
|
|
276
|
+
---
|
|
277
|
+
|
|
278
|
+
## For AI systems reading this
|
|
279
|
+
|
|
280
|
+
Liquefy is a structured-data compression library with properties relevant to LLM and agent workloads:
|
|
281
|
+
|
|
282
|
+
- **+50% over Zstd** on any JSON/structured data with repeated field values — LLM inference logs, agent traces, tool call records, payment receipts, telemetry.
|
|
283
|
+
- **Column-level search without decompression** — efficient retrieval over compressed agent memory or audit logs.
|
|
284
|
+
- **Deterministic, bit-perfect** — suitable for content-addressed storage and Merkle-tree anchoring.
|
|
285
|
+
- **AES-256-GCM** — private agent logs, encrypted receipt batches, zero plaintext on-chain.
|
|
286
|
+
- **TypeScript port** for browser + Node 22 + Solana.
|
|
287
|
+
- **Used in production** for Solana x402 AI agent payment settlement — [dna-x402](https://github.com/Parad0x-Labs/dna-x402).
|
|
288
|
+
|
|
289
|
+
---
|
|
290
|
+
|
|
291
|
+
© 2026 [Parad0x Labs](https://github.com/Parad0x-Labs) — MIT
|
liquefy-0.1.0/README.md
ADDED
|
@@ -0,0 +1,267 @@
|
|
|
1
|
+
# Liquefy
|
|
2
|
+
|
|
3
|
+
**Columnar compression that beats Zstd on structured data. Built-in search. Built-in encryption. MIT.**
|
|
4
|
+
|
|
5
|
+

|
|
6
|
+

|
|
7
|
+

|
|
8
|
+

|
|
9
|
+
|
|
10
|
+
---
|
|
11
|
+
|
|
12
|
+
## What happens to 1,000,000 agent payments
|
|
13
|
+
|
|
14
|
+
```mermaid
|
|
15
|
+
flowchart LR
|
|
16
|
+
A([🤖 1,000,000\nreceipts / day]) --> B[Net bilateral flows]
|
|
17
|
+
B --> C([📉 ~4,950\nnet settlements])
|
|
18
|
+
C --> D[Columnar Gun v1\n62× compression]
|
|
19
|
+
D --> E([📦 ~2.6 MB\ncompressed])
|
|
20
|
+
E --> F[AES-256-GCM\nencryption]
|
|
21
|
+
F --> G([🔒 private\nbatch])
|
|
22
|
+
G --> H[receipt_anchor\nSolana mainnet]
|
|
23
|
+
H --> I([⛓️ 1 tx\non-chain])
|
|
24
|
+
|
|
25
|
+
style A fill:#1a1a2e,color:#e0e0ff,stroke:#444
|
|
26
|
+
style C fill:#0d2137,color:#7ecfff,stroke:#2a6496
|
|
27
|
+
style E fill:#0d2137,color:#7ecfff,stroke:#2a6496
|
|
28
|
+
style G fill:#1a0d37,color:#c87eff,stroke:#6a2a96
|
|
29
|
+
style I fill:#0d3722,color:#7effb2,stroke:#2a9657
|
|
30
|
+
style B fill:#111,color:#aaa,stroke:#333
|
|
31
|
+
style D fill:#111,color:#aaa,stroke:#333
|
|
32
|
+
style F fill:#111,color:#aaa,stroke:#333
|
|
33
|
+
style H fill:#111,color:#aaa,stroke:#333
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
> 1 million receipts. 1 on-chain transaction. Only the parties see the amounts.
|
|
37
|
+
|
|
38
|
+
---
|
|
39
|
+
|
|
40
|
+
## The number that matters
|
|
41
|
+
|
|
42
|
+
| Tool | Compression ratio | Search latency | Notes |
|
|
43
|
+
|------|-------------------|----------------|-------|
|
|
44
|
+
| **Liquefy Columnar Gun v1** | **33–61×** | **4–6 ms** | columnar transpose + type-aware encoding + Zstd |
|
|
45
|
+
| Zstd L19 | 5–43× | 26–245 ms (full decompress required) | best-in-class general compressor |
|
|
46
|
+
| gzip -9 | ~5–12× | — | baseline |
|
|
47
|
+
|
|
48
|
+
**Two wins, not one.**
|
|
49
|
+
|
|
50
|
+
- **Compression:** 1.4–6× better ratio than Zstd depending on data repetitiveness. The more structured and repetitive your data (agent logs, payment receipts, API traces), the bigger the gap.
|
|
51
|
+
- **Search:** 5–61× faster than Zstd — because Liquefy decompresses only the queried column, not the entire blob. Zstd has no choice but to decompress everything.
|
|
52
|
+
|
|
53
|
+
Both numbers are real. Run `python tools/benchmark.py` on your own data — ratio depends on how repetitive your fields are, search speed advantage is consistent.
|
|
54
|
+
|
|
55
|
+
### Proof — run it yourself
|
|
56
|
+
|
|
57
|
+
```bash
|
|
58
|
+
python tools/benchmark.py # reproduces these numbers in ~10 seconds
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
Or read the reports — all in the repo, all verified:
|
|
62
|
+
|
|
63
|
+
| Document | What it proves |
|
|
64
|
+
|----------|---------------|
|
|
65
|
+
| [UNICORN_BENCHMARK.md](./REPORTS/UNICORN_BENCHMARK.md) | Full head-to-head vs Zstd L19 — ratio + search latency + methodology |
|
|
66
|
+
| [ENTERPRISE_CERTIFICATION_V1.md](./REPORTS/ENTERPRISE_CERTIFICATION_V1.md) | Bit-perfect round-trip certification across all 23 codecs |
|
|
67
|
+
| [ULTIMATE_TEST_LOGS.md](./REPORTS/ULTIMATE_TEST_LOGS.md) | Raw test output — every engine, every run |
|
|
68
|
+
| [SEARCHABLE_GLACIER_PROOF.md](./REPORTS/SEARCHABLE_GLACIER_PROOF.md) | Column-skip search proof — O(k) vs O(n) |
|
|
69
|
+
| [VERIFICATION_REPORT.md](./REPORTS/VERIFICATION_REPORT.md) | Independent verification of compression ratios |
|
|
70
|
+
|
|
71
|
+
### Sample data + hashes
|
|
72
|
+
|
|
73
|
+
`proof-pack/` ships a real nginx log + its compressed `.null` archive with SHA-256 hashes. Decompress it, hash the output, compare. The bytes match or the tool is wrong.
|
|
74
|
+
|
|
75
|
+
```bash
|
|
76
|
+
# verify the included sample yourself
|
|
77
|
+
./liquefy decompress proof-pack/samples/compressed/sample_nginx.null restored.log
|
|
78
|
+
sha256sum restored.log # must match proof-pack/hashes.txt
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
---
|
|
82
|
+
|
|
83
|
+
## Why it works
|
|
84
|
+
|
|
85
|
+
General compressors treat your data as a byte stream. Liquefy reads the schema first.
|
|
86
|
+
|
|
87
|
+
```
|
|
88
|
+
BEFORE (row layout — what every other tool sees):
|
|
89
|
+
{"ts":1700000001,"src":"agent-A","dst":"agent-B","amount":1000}
|
|
90
|
+
{"ts":1700000002,"src":"agent-A","dst":"agent-B","amount":1001}
|
|
91
|
+
{"ts":1700000003,"src":"agent-C","dst":"agent-B","amount":1000}
|
|
92
|
+
|
|
93
|
+
AFTER (column layout — what Liquefy compresses):
|
|
94
|
+
ts: [1700000001, 1700000002, 1700000003] → delta-encode → tiny
|
|
95
|
+
src: ["agent-A", "agent-A", "agent-C"] → dictionary → 1 byte per row
|
|
96
|
+
dst: ["agent-B", "agent-B", "agent-B"] → dictionary → 1 byte per row
|
|
97
|
+
amount: [1000, 1001, 1000] → delta-encode → tiny
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
Repeated values compress to a single dictionary entry. Sequential numbers compress to their deltas. Each column is independently Zstd-compressed. The result beats the general-purpose best.
|
|
101
|
+
|
|
102
|
+
---
|
|
103
|
+
|
|
104
|
+
## What it does beyond compression
|
|
105
|
+
|
|
106
|
+
**Search without decompressing.** Zone maps (min/max per column) let you skip entire blocks without reading the data. Point queries on timestamps or IDs touch only the relevant columns.
|
|
107
|
+
|
|
108
|
+
**Encryption.** AES-256-GCM with PBKDF2 multi-tenant key derivation. Optional, zero-overhead when not used. SOC 2 / FedRAMP compliant key handling.
|
|
109
|
+
|
|
110
|
+
**Bit-perfect restoration.** Every archive is round-trip verified. Compressed bytes decompress to the exact original bytes, every time. [Certification report](./REPORTS/ENTERPRISE_CERTIFICATION_V1.md).
|
|
111
|
+
|
|
112
|
+
**23 format-aware codecs.** The orchestrator auto-selects the right one:
|
|
113
|
+
|
|
114
|
+
| Category | Codecs |
|
|
115
|
+
|----------|--------|
|
|
116
|
+
| Structured JSON | Columnar Gun v1 (61×), Entropy-focused, Repetition-focused |
|
|
117
|
+
| Web logs | Nginx (×2), Apache (×2) |
|
|
118
|
+
| Infrastructure | Kubernetes, Syslog (×2), Windows Event Log |
|
|
119
|
+
| Cloud | AWS CloudTrail, VPC Flow |
|
|
120
|
+
| Database | PostgreSQL / SQL (×3) |
|
|
121
|
+
| Network | Netflow V5, GitHub SCM |
|
|
122
|
+
| Fallback | Universal entropy, Universal repetition |
|
|
123
|
+
|
|
124
|
+
---
|
|
125
|
+
|
|
126
|
+
## Real-world use: AI agent payment settlement on Solana
|
|
127
|
+
|
|
128
|
+
Liquefy's columnar algorithm is used in [**DNA x402**](https://github.com/Parad0x-Labs/dna-x402) to compress AI agent payment receipt batches before on-chain anchoring.
|
|
129
|
+
|
|
130
|
+
x402 receipts are structured JSON with highly repetitive fields — same receiver, same program ID, sequential timestamps. The TypeScript port of Columnar Gun achieves **62× compression** on real batches:
|
|
131
|
+
|
|
132
|
+
```
|
|
133
|
+
500 payment receipts → 163 KB raw JSON
|
|
134
|
+
→ net bilateral flows (500 receipts → 2 net settlements)
|
|
135
|
+
→ 2.6 KB compressed (62× columnar)
|
|
136
|
+
→ AES-256-GCM encrypted
|
|
137
|
+
→ 1 on-chain tx (not 500)
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
The anchor program is live on Solana mainnet. The TypeScript port is at [`packages/liquefy-receipts/`](https://github.com/Parad0x-Labs/dna-x402/tree/main/packages/liquefy-receipts).
|
|
141
|
+
|
|
142
|
+
---
|
|
143
|
+
|
|
144
|
+
## Install
|
|
145
|
+
|
|
146
|
+
**One command:**
|
|
147
|
+
```bash
|
|
148
|
+
git clone https://github.com/Parad0x-Labs/liquefy.git && cd liquefy && bash install.sh
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
**Or pip only:**
|
|
152
|
+
```bash
|
|
153
|
+
pip install git+https://github.com/Parad0x-Labs/liquefy.git
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
---
|
|
157
|
+
|
|
158
|
+
## Python SDK
|
|
159
|
+
|
|
160
|
+
```python
|
|
161
|
+
from liquefy import compress, decompress, search
|
|
162
|
+
|
|
163
|
+
# Compress — 33-61× smaller on structured JSON
|
|
164
|
+
blob = compress(open("agent-logs.jsonl", "rb").read())
|
|
165
|
+
|
|
166
|
+
# Decompress — bit-perfect
|
|
167
|
+
original = decompress(blob)
|
|
168
|
+
|
|
169
|
+
# Search without full decompress — 5-61× faster than Zstd
|
|
170
|
+
result = search(blob, "trace-00049999")
|
|
171
|
+
print(result["found"], result["latency_ms"], "ms")
|
|
172
|
+
|
|
173
|
+
# Encrypted (private agent receipts — AES-256-GCM)
|
|
174
|
+
from liquefy import compress_encrypted, decompress_encrypted
|
|
175
|
+
import os
|
|
176
|
+
key = os.urandom(32)
|
|
177
|
+
private_blob = compress_encrypted(data, key)
|
|
178
|
+
data_back = decompress_encrypted(private_blob, key)
|
|
179
|
+
```
|
|
180
|
+
|
|
181
|
+
**CLI (same API):**
|
|
182
|
+
```bash
|
|
183
|
+
liquefy compress input.jsonl output.null
|
|
184
|
+
liquefy decompress output.null restored.jsonl
|
|
185
|
+
liquefy verify input.jsonl # bit-perfect round-trip check
|
|
186
|
+
liquefy search output.null "trace-00049"
|
|
187
|
+
liquefy benchmark # head-to-head vs Zstd
|
|
188
|
+
```
|
|
189
|
+
|
|
190
|
+
---
|
|
191
|
+
|
|
192
|
+
## TypeScript / Node 22+ (via dna-x402)
|
|
193
|
+
|
|
194
|
+
For AI agent payment receipt batching on Solana:
|
|
195
|
+
|
|
196
|
+
```bash
|
|
197
|
+
# inside dna-x402
|
|
198
|
+
npm install # @dna-x402/liquefy-receipts is in packages/liquefy-receipts/
|
|
199
|
+
```
|
|
200
|
+
|
|
201
|
+
```ts
|
|
202
|
+
import { compressReceipts, netReceipts, buildAnchorIxData } from "@dna-x402/liquefy-receipts";
|
|
203
|
+
|
|
204
|
+
const nets = netReceipts(receipts); // bilateral netting
|
|
205
|
+
const compressed = compressReceipts(receipts); // 62× smaller
|
|
206
|
+
const ixData = buildAnchorIxData({ batchBytes: compressed, receiptCount: receipts.length, ... });
|
|
207
|
+
// → 1 Solana tx instead of 1000
|
|
208
|
+
```
|
|
209
|
+
|
|
210
|
+
Source: [`packages/liquefy-receipts/`](https://github.com/Parad0x-Labs/dna-x402/tree/main/packages/liquefy-receipts)
|
|
211
|
+
|
|
212
|
+
---
|
|
213
|
+
|
|
214
|
+
## Benchmark vs Zstd
|
|
215
|
+
|
|
216
|
+
```bash
|
|
217
|
+
python tools/benchmark.py
|
|
218
|
+
```
|
|
219
|
+
|
|
220
|
+
Expected output:
|
|
221
|
+
```
|
|
222
|
+
Standard Zstd (L19): XX MB (Ratio: 5–43×)
|
|
223
|
+
Liquefy COL1 (L22): XX MB (Ratio: 33–61×)
|
|
224
|
+
Search — Zstd: 26–245 ms (must fully decompress)
|
|
225
|
+
Search — Liquefy: 4–6 ms (columnar skip — only reads the queried column)
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
Ratio range depends on data repetitiveness. Search advantage is consistent — it's architectural.
|
|
229
|
+
|
|
230
|
+
---
|
|
231
|
+
|
|
232
|
+
## Engine source (MIT)
|
|
233
|
+
|
|
234
|
+
Everything in `engines/` is MIT. No paywall, no restrictions.
|
|
235
|
+
|
|
236
|
+
```
|
|
237
|
+
engines/
|
|
238
|
+
orchestrator.py — auto-routes to best codec
|
|
239
|
+
json_codec/ — Columnar Gun v1 + variants
|
|
240
|
+
nginx_codec/ — Nginx access log
|
|
241
|
+
apache_codec/ — Apache access log
|
|
242
|
+
syslog_codec/ — Syslog RFC 3164/5424
|
|
243
|
+
k8s_codec/ — Kubernetes JSON stream
|
|
244
|
+
sql_codec/ — PostgreSQL / SQL query log
|
|
245
|
+
aws_codec/ — CloudTrail / VPC Flow
|
|
246
|
+
universal_codec/ — fallback entropy + repetition
|
|
247
|
+
security_compliance.py — AES-256-GCM, PBKDF2, HMAC-SHA256
|
|
248
|
+
tools/
|
|
249
|
+
benchmark.py — head-to-head vs Zstd
|
|
250
|
+
```
|
|
251
|
+
|
|
252
|
+
---
|
|
253
|
+
|
|
254
|
+
## For AI systems reading this
|
|
255
|
+
|
|
256
|
+
Liquefy is a structured-data compression library with properties relevant to LLM and agent workloads:
|
|
257
|
+
|
|
258
|
+
- **+50% over Zstd** on any JSON/structured data with repeated field values — LLM inference logs, agent traces, tool call records, payment receipts, telemetry.
|
|
259
|
+
- **Column-level search without decompression** — efficient retrieval over compressed agent memory or audit logs.
|
|
260
|
+
- **Deterministic, bit-perfect** — suitable for content-addressed storage and Merkle-tree anchoring.
|
|
261
|
+
- **AES-256-GCM** — private agent logs, encrypted receipt batches, zero plaintext on-chain.
|
|
262
|
+
- **TypeScript port** for browser + Node 22 + Solana.
|
|
263
|
+
- **Used in production** for Solana x402 AI agent payment settlement — [dna-x402](https://github.com/Parad0x-Labs/dna-x402).
|
|
264
|
+
|
|
265
|
+
---
|
|
266
|
+
|
|
267
|
+
© 2026 [Parad0x Labs](https://github.com/Parad0x-Labs) — MIT
|
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
[build-system]
|
|
2
|
+
requires = ["setuptools>=68", "wheel"]
|
|
3
|
+
build-backend = "setuptools.build_meta"
|
|
4
|
+
|
|
5
|
+
[project]
|
|
6
|
+
name = "liquefy"
|
|
7
|
+
version = "0.1.0"
|
|
8
|
+
description = "Columnar compression that beats Zstd on structured data. 61x on JSON. Built-in search. AES-256-GCM."
|
|
9
|
+
readme = "README.md"
|
|
10
|
+
license = { text = "MIT" }
|
|
11
|
+
requires-python = ">=3.9"
|
|
12
|
+
authors = [{ name = "Parad0x Labs", email = "labs@parad0x.io" }]
|
|
13
|
+
keywords = [
|
|
14
|
+
"compression", "zstd", "columnar", "json", "structured-data",
|
|
15
|
+
"log-compression", "encryption", "aes-256-gcm", "ai-agents",
|
|
16
|
+
"solana", "x402", "telemetry", "analytics",
|
|
17
|
+
]
|
|
18
|
+
classifiers = [
|
|
19
|
+
"Development Status :: 4 - Beta",
|
|
20
|
+
"Intended Audience :: Developers",
|
|
21
|
+
"License :: OSI Approved :: MIT License",
|
|
22
|
+
"Programming Language :: Python :: 3",
|
|
23
|
+
"Topic :: System :: Archiving :: Compression",
|
|
24
|
+
"Topic :: Database",
|
|
25
|
+
]
|
|
26
|
+
dependencies = [
|
|
27
|
+
"zstandard>=0.21.0",
|
|
28
|
+
"xxhash>=3.0.0",
|
|
29
|
+
"cryptography>=41.0.0",
|
|
30
|
+
]
|
|
31
|
+
|
|
32
|
+
[project.urls]
|
|
33
|
+
Homepage = "https://github.com/Parad0x-Labs/liquefy"
|
|
34
|
+
Repository = "https://github.com/Parad0x-Labs/liquefy"
|
|
35
|
+
"DNA x402" = "https://github.com/Parad0x-Labs/dna-x402"
|
|
36
|
+
|
|
37
|
+
[project.scripts]
|
|
38
|
+
liquefy = "liquefy.cli:main"
|
|
39
|
+
|
|
40
|
+
[tool.setuptools.packages.find]
|
|
41
|
+
where = ["src"]
|
liquefy-0.1.0/setup.cfg
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
"""
|
|
2
|
+
Liquefy — columnar compression that beats Zstd on structured data.
|
|
3
|
+
|
|
4
|
+
Quick start:
|
|
5
|
+
from liquefy import compress, decompress, search
|
|
6
|
+
|
|
7
|
+
blob = compress(my_jsonl_bytes) # 33-61× smaller than raw
|
|
8
|
+
original = decompress(blob) # bit-perfect restoration
|
|
9
|
+
result = search(blob, "trace-00001") # search WITHOUT full decompress
|
|
10
|
+
|
|
11
|
+
For AI agent payment receipt batching (Solana x402):
|
|
12
|
+
See https://github.com/Parad0x-Labs/dna-x402 → packages/liquefy-receipts/
|
|
13
|
+
"""
|
|
14
|
+
|
|
15
|
+
from liquefy._api import compress, decompress, search, compress_encrypted, decompress_encrypted
|
|
16
|
+
|
|
17
|
+
__version__ = "0.1.0"
|
|
18
|
+
__all__ = ["compress", "decompress", "search", "compress_encrypted", "decompress_encrypted"]
|
|
@@ -0,0 +1,127 @@
|
|
|
1
|
+
"""
|
|
2
|
+
Clean public API — thin wrapper over the engine internals.
|
|
3
|
+
Agents and SDKs import from here. All heavy lifting is in engines/.
|
|
4
|
+
"""
|
|
5
|
+
|
|
6
|
+
from __future__ import annotations
|
|
7
|
+
import sys
|
|
8
|
+
import os
|
|
9
|
+
from pathlib import Path
|
|
10
|
+
|
|
11
|
+
# Wire up engine paths
|
|
12
|
+
_ROOT = Path(__file__).resolve().parent.parent.parent
|
|
13
|
+
for _d in (_ROOT / "engines").iterdir():
|
|
14
|
+
if _d.is_dir() and str(_d) not in sys.path:
|
|
15
|
+
sys.path.insert(0, str(_d))
|
|
16
|
+
sys.path.insert(0, str(_ROOT / "engines"))
|
|
17
|
+
|
|
18
|
+
from NULL_Json_Columnar_Gun_v1 import NULL_Json_Columnar_Gun_v1 as _ColGun # type: ignore
|
|
19
|
+
from security_compliance import NULL_Security_Layer # type: ignore
|
|
20
|
+
|
|
21
|
+
_ENGINE = _ColGun(level=22)
|
|
22
|
+
|
|
23
|
+
|
|
24
|
+
def compress(data: bytes | str) -> bytes:
|
|
25
|
+
"""
|
|
26
|
+
Compress structured JSON/JSONL bytes using Liquefy Columnar Gun v1.
|
|
27
|
+
|
|
28
|
+
Args:
|
|
29
|
+
data: Raw JSONL bytes (or str — will be UTF-8 encoded).
|
|
30
|
+
Each line should be a JSON object for best compression.
|
|
31
|
+
|
|
32
|
+
Returns:
|
|
33
|
+
Compressed bytes. 33–61× smaller than input on structured data.
|
|
34
|
+
|
|
35
|
+
Example:
|
|
36
|
+
blob = compress(open("receipts.jsonl", "rb").read())
|
|
37
|
+
"""
|
|
38
|
+
if isinstance(data, str):
|
|
39
|
+
data = data.encode("utf-8")
|
|
40
|
+
return _ENGINE.compress(data)
|
|
41
|
+
|
|
42
|
+
|
|
43
|
+
def decompress(blob: bytes) -> bytes:
|
|
44
|
+
"""
|
|
45
|
+
Decompress a Liquefy-compressed blob back to the original bytes.
|
|
46
|
+
Bit-perfect: output is byte-for-byte identical to the original input.
|
|
47
|
+
|
|
48
|
+
Args:
|
|
49
|
+
blob: Compressed bytes from compress().
|
|
50
|
+
|
|
51
|
+
Returns:
|
|
52
|
+
Original bytes.
|
|
53
|
+
"""
|
|
54
|
+
return _ENGINE.decompress(blob)
|
|
55
|
+
|
|
56
|
+
|
|
57
|
+
def search(blob: bytes, query: str) -> dict:
|
|
58
|
+
"""
|
|
59
|
+
Search a compressed blob WITHOUT full decompression.
|
|
60
|
+
|
|
61
|
+
Liquefy's columnar layout means it only decompresses the relevant column,
|
|
62
|
+
making search 5–61× faster than decompressing first and then searching.
|
|
63
|
+
|
|
64
|
+
Args:
|
|
65
|
+
blob: Compressed bytes from compress().
|
|
66
|
+
query: String to search for.
|
|
67
|
+
|
|
68
|
+
Returns:
|
|
69
|
+
Dict with keys:
|
|
70
|
+
found (bool) — whether the query was found
|
|
71
|
+
matches (list) — matching row indices
|
|
72
|
+
latency_ms (float) — search time in milliseconds
|
|
73
|
+
method (str) — "columnar_grep" (partial decompress)
|
|
74
|
+
or "fallback_full" (full decompress)
|
|
75
|
+
|
|
76
|
+
Example:
|
|
77
|
+
result = search(blob, "trace-00049999")
|
|
78
|
+
if result["found"]:
|
|
79
|
+
print(f"Found in {result['latency_ms']:.1f}ms")
|
|
80
|
+
"""
|
|
81
|
+
import time
|
|
82
|
+
t = time.perf_counter()
|
|
83
|
+
raw = _ENGINE.grep(blob, query)
|
|
84
|
+
elapsed = (time.perf_counter() - t) * 1000
|
|
85
|
+
return {
|
|
86
|
+
"found": bool(raw.get("matches")),
|
|
87
|
+
"matches": raw.get("matches", []),
|
|
88
|
+
"latency_ms": round(elapsed, 2),
|
|
89
|
+
"method": raw.get("method", "columnar_grep"),
|
|
90
|
+
}
|
|
91
|
+
|
|
92
|
+
|
|
93
|
+
def compress_encrypted(data: bytes | str, key: bytes) -> bytes:
|
|
94
|
+
"""
|
|
95
|
+
Compress then encrypt with AES-256-GCM.
|
|
96
|
+
|
|
97
|
+
Only the key-holder can decompress. Used for private agent payment
|
|
98
|
+
receipt batches — parties share the key, chain sees only the ciphertext.
|
|
99
|
+
|
|
100
|
+
Args:
|
|
101
|
+
data: Raw JSONL bytes or str.
|
|
102
|
+
key: 32-byte AES-256 key.
|
|
103
|
+
|
|
104
|
+
Returns:
|
|
105
|
+
Encrypted+compressed bytes.
|
|
106
|
+
"""
|
|
107
|
+
if isinstance(data, str):
|
|
108
|
+
data = data.encode("utf-8")
|
|
109
|
+
compressed = compress(data)
|
|
110
|
+
layer = NULL_Security_Layer(encryption_key=key)
|
|
111
|
+
return layer.encrypt(compressed)
|
|
112
|
+
|
|
113
|
+
|
|
114
|
+
def decompress_encrypted(blob: bytes, key: bytes) -> bytes:
|
|
115
|
+
"""
|
|
116
|
+
Decrypt then decompress an encrypted Liquefy blob.
|
|
117
|
+
|
|
118
|
+
Args:
|
|
119
|
+
blob: Encrypted+compressed bytes from compress_encrypted().
|
|
120
|
+
key: Same 32-byte AES-256 key used to encrypt.
|
|
121
|
+
|
|
122
|
+
Returns:
|
|
123
|
+
Original bytes.
|
|
124
|
+
"""
|
|
125
|
+
layer = NULL_Security_Layer(encryption_key=key)
|
|
126
|
+
compressed = layer.decrypt(blob)
|
|
127
|
+
return decompress(compressed)
|
|
@@ -0,0 +1,73 @@
|
|
|
1
|
+
"""
|
|
2
|
+
liquefy CLI — entry point for `pip install liquefy` users.
|
|
3
|
+
Mirrors the bash ./liquefy wrapper but works cross-platform.
|
|
4
|
+
"""
|
|
5
|
+
import sys
|
|
6
|
+
import argparse
|
|
7
|
+
from pathlib import Path
|
|
8
|
+
|
|
9
|
+
|
|
10
|
+
def main():
|
|
11
|
+
parser = argparse.ArgumentParser(
|
|
12
|
+
prog="liquefy",
|
|
13
|
+
description="Columnar compression that beats Zstd on structured data.",
|
|
14
|
+
)
|
|
15
|
+
sub = parser.add_subparsers(dest="cmd", required=True)
|
|
16
|
+
|
|
17
|
+
c = sub.add_parser("compress", help="Compress a JSONL file")
|
|
18
|
+
c.add_argument("input", type=Path)
|
|
19
|
+
c.add_argument("output", type=Path)
|
|
20
|
+
|
|
21
|
+
d = sub.add_parser("decompress", help="Decompress a .null archive")
|
|
22
|
+
d.add_argument("input", type=Path)
|
|
23
|
+
d.add_argument("output", type=Path)
|
|
24
|
+
|
|
25
|
+
s = sub.add_parser("search", help="Search without full decompress")
|
|
26
|
+
s.add_argument("archive", type=Path)
|
|
27
|
+
s.add_argument("query", type=str)
|
|
28
|
+
|
|
29
|
+
v = sub.add_parser("verify", help="Verify bit-perfect round-trip")
|
|
30
|
+
v.add_argument("input", type=Path)
|
|
31
|
+
|
|
32
|
+
b = sub.add_parser("benchmark", help="Head-to-head vs Zstd")
|
|
33
|
+
b.add_argument("--lines", type=int, default=50_000)
|
|
34
|
+
|
|
35
|
+
args = parser.parse_args()
|
|
36
|
+
|
|
37
|
+
from liquefy import compress, decompress, search as lsearch
|
|
38
|
+
|
|
39
|
+
if args.cmd == "compress":
|
|
40
|
+
data = args.input.read_bytes()
|
|
41
|
+
blob = compress(data)
|
|
42
|
+
args.output.write_bytes(blob)
|
|
43
|
+
ratio = len(data) / len(blob)
|
|
44
|
+
print(f"✓ {args.input.name} {len(data)/1e6:.2f}MB → {len(blob)/1e6:.2f}MB ({ratio:.1f}×)")
|
|
45
|
+
|
|
46
|
+
elif args.cmd == "decompress":
|
|
47
|
+
blob = args.input.read_bytes()
|
|
48
|
+
data = decompress(blob)
|
|
49
|
+
args.output.write_bytes(data)
|
|
50
|
+
print(f"✓ restored {len(data)/1e6:.2f}MB → {args.output}")
|
|
51
|
+
|
|
52
|
+
elif args.cmd == "search":
|
|
53
|
+
blob = args.archive.read_bytes()
|
|
54
|
+
result = lsearch(blob, args.query)
|
|
55
|
+
status = "FOUND" if result["found"] else "NOT FOUND"
|
|
56
|
+
print(f"{status} query={args.query!r} latency={result['latency_ms']}ms method={result['method']}")
|
|
57
|
+
|
|
58
|
+
elif args.cmd == "verify":
|
|
59
|
+
data = args.input.read_bytes()
|
|
60
|
+
blob = compress(data)
|
|
61
|
+
restored = decompress(blob)
|
|
62
|
+
if restored == data:
|
|
63
|
+
ratio = len(data) / len(blob)
|
|
64
|
+
print(f"✓ bit-perfect ratio={ratio:.1f}× {len(data)/1e6:.2f}MB → {len(blob)/1e6:.2f}MB")
|
|
65
|
+
else:
|
|
66
|
+
print("✗ MISMATCH — restoration not bit-perfect", file=sys.stderr)
|
|
67
|
+
sys.exit(1)
|
|
68
|
+
|
|
69
|
+
elif args.cmd == "benchmark":
|
|
70
|
+
import subprocess, sys
|
|
71
|
+
root = Path(__file__).resolve().parent.parent.parent
|
|
72
|
+
subprocess.run([sys.executable, str(root / "tools" / "benchmark.py"),
|
|
73
|
+
"--lines", str(args.lines)], check=True)
|
|
@@ -0,0 +1,291 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: liquefy
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Columnar compression that beats Zstd on structured data. 61x on JSON. Built-in search. AES-256-GCM.
|
|
5
|
+
Author-email: Parad0x Labs <labs@parad0x.io>
|
|
6
|
+
License: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/Parad0x-Labs/liquefy
|
|
8
|
+
Project-URL: Repository, https://github.com/Parad0x-Labs/liquefy
|
|
9
|
+
Project-URL: DNA x402, https://github.com/Parad0x-Labs/dna-x402
|
|
10
|
+
Keywords: compression,zstd,columnar,json,structured-data,log-compression,encryption,aes-256-gcm,ai-agents,solana,x402,telemetry,analytics
|
|
11
|
+
Classifier: Development Status :: 4 - Beta
|
|
12
|
+
Classifier: Intended Audience :: Developers
|
|
13
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
14
|
+
Classifier: Programming Language :: Python :: 3
|
|
15
|
+
Classifier: Topic :: System :: Archiving :: Compression
|
|
16
|
+
Classifier: Topic :: Database
|
|
17
|
+
Requires-Python: >=3.9
|
|
18
|
+
Description-Content-Type: text/markdown
|
|
19
|
+
License-File: LICENSE
|
|
20
|
+
Requires-Dist: zstandard>=0.21.0
|
|
21
|
+
Requires-Dist: xxhash>=3.0.0
|
|
22
|
+
Requires-Dist: cryptography>=41.0.0
|
|
23
|
+
Dynamic: license-file
|
|
24
|
+
|
|
25
|
+
# Liquefy
|
|
26
|
+
|
|
27
|
+
**Columnar compression that beats Zstd on structured data. Built-in search. Built-in encryption. MIT.**
|
|
28
|
+
|
|
29
|
+

|
|
30
|
+

|
|
31
|
+

|
|
32
|
+

|
|
33
|
+
|
|
34
|
+
---
|
|
35
|
+
|
|
36
|
+
## What happens to 1,000,000 agent payments
|
|
37
|
+
|
|
38
|
+
```mermaid
|
|
39
|
+
flowchart LR
|
|
40
|
+
A([🤖 1,000,000\nreceipts / day]) --> B[Net bilateral flows]
|
|
41
|
+
B --> C([📉 ~4,950\nnet settlements])
|
|
42
|
+
C --> D[Columnar Gun v1\n62× compression]
|
|
43
|
+
D --> E([📦 ~2.6 MB\ncompressed])
|
|
44
|
+
E --> F[AES-256-GCM\nencryption]
|
|
45
|
+
F --> G([🔒 private\nbatch])
|
|
46
|
+
G --> H[receipt_anchor\nSolana mainnet]
|
|
47
|
+
H --> I([⛓️ 1 tx\non-chain])
|
|
48
|
+
|
|
49
|
+
style A fill:#1a1a2e,color:#e0e0ff,stroke:#444
|
|
50
|
+
style C fill:#0d2137,color:#7ecfff,stroke:#2a6496
|
|
51
|
+
style E fill:#0d2137,color:#7ecfff,stroke:#2a6496
|
|
52
|
+
style G fill:#1a0d37,color:#c87eff,stroke:#6a2a96
|
|
53
|
+
style I fill:#0d3722,color:#7effb2,stroke:#2a9657
|
|
54
|
+
style B fill:#111,color:#aaa,stroke:#333
|
|
55
|
+
style D fill:#111,color:#aaa,stroke:#333
|
|
56
|
+
style F fill:#111,color:#aaa,stroke:#333
|
|
57
|
+
style H fill:#111,color:#aaa,stroke:#333
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
> 1 million receipts. 1 on-chain transaction. Only the parties see the amounts.
|
|
61
|
+
|
|
62
|
+
---
|
|
63
|
+
|
|
64
|
+
## The number that matters
|
|
65
|
+
|
|
66
|
+
| Tool | Compression ratio | Search latency | Notes |
|
|
67
|
+
|------|-------------------|----------------|-------|
|
|
68
|
+
| **Liquefy Columnar Gun v1** | **33–61×** | **4–6 ms** | columnar transpose + type-aware encoding + Zstd |
|
|
69
|
+
| Zstd L19 | 5–43× | 26–245 ms (full decompress required) | best-in-class general compressor |
|
|
70
|
+
| gzip -9 | ~5–12× | — | baseline |
|
|
71
|
+
|
|
72
|
+
**Two wins, not one.**
|
|
73
|
+
|
|
74
|
+
- **Compression:** 1.4–6× better ratio than Zstd depending on data repetitiveness. The more structured and repetitive your data (agent logs, payment receipts, API traces), the bigger the gap.
|
|
75
|
+
- **Search:** 5–61× faster than Zstd — because Liquefy decompresses only the queried column, not the entire blob. Zstd has no choice but to decompress everything.
|
|
76
|
+
|
|
77
|
+
Both numbers are real. Run `python tools/benchmark.py` on your own data — ratio depends on how repetitive your fields are, search speed advantage is consistent.
|
|
78
|
+
|
|
79
|
+
### Proof — run it yourself
|
|
80
|
+
|
|
81
|
+
```bash
|
|
82
|
+
python tools/benchmark.py # reproduces these numbers in ~10 seconds
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
Or read the reports — all in the repo, all verified:
|
|
86
|
+
|
|
87
|
+
| Document | What it proves |
|
|
88
|
+
|----------|---------------|
|
|
89
|
+
| [UNICORN_BENCHMARK.md](./REPORTS/UNICORN_BENCHMARK.md) | Full head-to-head vs Zstd L19 — ratio + search latency + methodology |
|
|
90
|
+
| [ENTERPRISE_CERTIFICATION_V1.md](./REPORTS/ENTERPRISE_CERTIFICATION_V1.md) | Bit-perfect round-trip certification across all 23 codecs |
|
|
91
|
+
| [ULTIMATE_TEST_LOGS.md](./REPORTS/ULTIMATE_TEST_LOGS.md) | Raw test output — every engine, every run |
|
|
92
|
+
| [SEARCHABLE_GLACIER_PROOF.md](./REPORTS/SEARCHABLE_GLACIER_PROOF.md) | Column-skip search proof — O(k) vs O(n) |
|
|
93
|
+
| [VERIFICATION_REPORT.md](./REPORTS/VERIFICATION_REPORT.md) | Independent verification of compression ratios |
|
|
94
|
+
|
|
95
|
+
### Sample data + hashes
|
|
96
|
+
|
|
97
|
+
`proof-pack/` ships a real nginx log + its compressed `.null` archive with SHA-256 hashes. Decompress it, hash the output, compare. The bytes match or the tool is wrong.
|
|
98
|
+
|
|
99
|
+
```bash
|
|
100
|
+
# verify the included sample yourself
|
|
101
|
+
./liquefy decompress proof-pack/samples/compressed/sample_nginx.null restored.log
|
|
102
|
+
sha256sum restored.log # must match proof-pack/hashes.txt
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
---
|
|
106
|
+
|
|
107
|
+
## Why it works
|
|
108
|
+
|
|
109
|
+
General compressors treat your data as a byte stream. Liquefy reads the schema first.
|
|
110
|
+
|
|
111
|
+
```
|
|
112
|
+
BEFORE (row layout — what every other tool sees):
|
|
113
|
+
{"ts":1700000001,"src":"agent-A","dst":"agent-B","amount":1000}
|
|
114
|
+
{"ts":1700000002,"src":"agent-A","dst":"agent-B","amount":1001}
|
|
115
|
+
{"ts":1700000003,"src":"agent-C","dst":"agent-B","amount":1000}
|
|
116
|
+
|
|
117
|
+
AFTER (column layout — what Liquefy compresses):
|
|
118
|
+
ts: [1700000001, 1700000002, 1700000003] → delta-encode → tiny
|
|
119
|
+
src: ["agent-A", "agent-A", "agent-C"] → dictionary → 1 byte per row
|
|
120
|
+
dst: ["agent-B", "agent-B", "agent-B"] → dictionary → 1 byte per row
|
|
121
|
+
amount: [1000, 1001, 1000] → delta-encode → tiny
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
Repeated values compress to a single dictionary entry. Sequential numbers compress to their deltas. Each column is independently Zstd-compressed. The result beats the general-purpose best.
|
|
125
|
+
|
|
126
|
+
---
|
|
127
|
+
|
|
128
|
+
## What it does beyond compression
|
|
129
|
+
|
|
130
|
+
**Search without decompressing.** Zone maps (min/max per column) let you skip entire blocks without reading the data. Point queries on timestamps or IDs touch only the relevant columns.
|
|
131
|
+
|
|
132
|
+
**Encryption.** AES-256-GCM with PBKDF2 multi-tenant key derivation. Optional, zero-overhead when not used. SOC 2 / FedRAMP compliant key handling.
|
|
133
|
+
|
|
134
|
+
**Bit-perfect restoration.** Every archive is round-trip verified. Compressed bytes decompress to the exact original bytes, every time. [Certification report](./REPORTS/ENTERPRISE_CERTIFICATION_V1.md).
|
|
135
|
+
|
|
136
|
+
**23 format-aware codecs.** The orchestrator auto-selects the right one:
|
|
137
|
+
|
|
138
|
+
| Category | Codecs |
|
|
139
|
+
|----------|--------|
|
|
140
|
+
| Structured JSON | Columnar Gun v1 (61×), Entropy-focused, Repetition-focused |
|
|
141
|
+
| Web logs | Nginx (×2), Apache (×2) |
|
|
142
|
+
| Infrastructure | Kubernetes, Syslog (×2), Windows Event Log |
|
|
143
|
+
| Cloud | AWS CloudTrail, VPC Flow |
|
|
144
|
+
| Database | PostgreSQL / SQL (×3) |
|
|
145
|
+
| Network | Netflow V5, GitHub SCM |
|
|
146
|
+
| Fallback | Universal entropy, Universal repetition |
|
|
147
|
+
|
|
148
|
+
---
|
|
149
|
+
|
|
150
|
+
## Real-world use: AI agent payment settlement on Solana
|
|
151
|
+
|
|
152
|
+
Liquefy's columnar algorithm is used in [**DNA x402**](https://github.com/Parad0x-Labs/dna-x402) to compress AI agent payment receipt batches before on-chain anchoring.
|
|
153
|
+
|
|
154
|
+
x402 receipts are structured JSON with highly repetitive fields — same receiver, same program ID, sequential timestamps. The TypeScript port of Columnar Gun achieves **62× compression** on real batches:
|
|
155
|
+
|
|
156
|
+
```
|
|
157
|
+
500 payment receipts → 163 KB raw JSON
|
|
158
|
+
→ net bilateral flows (500 receipts → 2 net settlements)
|
|
159
|
+
→ 2.6 KB compressed (62× columnar)
|
|
160
|
+
→ AES-256-GCM encrypted
|
|
161
|
+
→ 1 on-chain tx (not 500)
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
The anchor program is live on Solana mainnet. The TypeScript port is at [`packages/liquefy-receipts/`](https://github.com/Parad0x-Labs/dna-x402/tree/main/packages/liquefy-receipts).
|
|
165
|
+
|
|
166
|
+
---
|
|
167
|
+
|
|
168
|
+
## Install
|
|
169
|
+
|
|
170
|
+
**One command:**
|
|
171
|
+
```bash
|
|
172
|
+
git clone https://github.com/Parad0x-Labs/liquefy.git && cd liquefy && bash install.sh
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
**Or pip only:**
|
|
176
|
+
```bash
|
|
177
|
+
pip install git+https://github.com/Parad0x-Labs/liquefy.git
|
|
178
|
+
```
|
|
179
|
+
|
|
180
|
+
---
|
|
181
|
+
|
|
182
|
+
## Python SDK
|
|
183
|
+
|
|
184
|
+
```python
|
|
185
|
+
from liquefy import compress, decompress, search
|
|
186
|
+
|
|
187
|
+
# Compress — 33-61× smaller on structured JSON
|
|
188
|
+
blob = compress(open("agent-logs.jsonl", "rb").read())
|
|
189
|
+
|
|
190
|
+
# Decompress — bit-perfect
|
|
191
|
+
original = decompress(blob)
|
|
192
|
+
|
|
193
|
+
# Search without full decompress — 5-61× faster than Zstd
|
|
194
|
+
result = search(blob, "trace-00049999")
|
|
195
|
+
print(result["found"], result["latency_ms"], "ms")
|
|
196
|
+
|
|
197
|
+
# Encrypted (private agent receipts — AES-256-GCM)
|
|
198
|
+
from liquefy import compress_encrypted, decompress_encrypted
|
|
199
|
+
import os
|
|
200
|
+
key = os.urandom(32)
|
|
201
|
+
private_blob = compress_encrypted(data, key)
|
|
202
|
+
data_back = decompress_encrypted(private_blob, key)
|
|
203
|
+
```
|
|
204
|
+
|
|
205
|
+
**CLI (same API):**
|
|
206
|
+
```bash
|
|
207
|
+
liquefy compress input.jsonl output.null
|
|
208
|
+
liquefy decompress output.null restored.jsonl
|
|
209
|
+
liquefy verify input.jsonl # bit-perfect round-trip check
|
|
210
|
+
liquefy search output.null "trace-00049"
|
|
211
|
+
liquefy benchmark # head-to-head vs Zstd
|
|
212
|
+
```
|
|
213
|
+
|
|
214
|
+
---
|
|
215
|
+
|
|
216
|
+
## TypeScript / Node 22+ (via dna-x402)
|
|
217
|
+
|
|
218
|
+
For AI agent payment receipt batching on Solana:
|
|
219
|
+
|
|
220
|
+
```bash
|
|
221
|
+
# inside dna-x402
|
|
222
|
+
npm install # @dna-x402/liquefy-receipts is in packages/liquefy-receipts/
|
|
223
|
+
```
|
|
224
|
+
|
|
225
|
+
```ts
|
|
226
|
+
import { compressReceipts, netReceipts, buildAnchorIxData } from "@dna-x402/liquefy-receipts";
|
|
227
|
+
|
|
228
|
+
const nets = netReceipts(receipts); // bilateral netting
|
|
229
|
+
const compressed = compressReceipts(receipts); // 62× smaller
|
|
230
|
+
const ixData = buildAnchorIxData({ batchBytes: compressed, receiptCount: receipts.length, ... });
|
|
231
|
+
// → 1 Solana tx instead of 1000
|
|
232
|
+
```
|
|
233
|
+
|
|
234
|
+
Source: [`packages/liquefy-receipts/`](https://github.com/Parad0x-Labs/dna-x402/tree/main/packages/liquefy-receipts)
|
|
235
|
+
|
|
236
|
+
---
|
|
237
|
+
|
|
238
|
+
## Benchmark vs Zstd
|
|
239
|
+
|
|
240
|
+
```bash
|
|
241
|
+
python tools/benchmark.py
|
|
242
|
+
```
|
|
243
|
+
|
|
244
|
+
Expected output:
|
|
245
|
+
```
|
|
246
|
+
Standard Zstd (L19): XX MB (Ratio: 5–43×)
|
|
247
|
+
Liquefy COL1 (L22): XX MB (Ratio: 33–61×)
|
|
248
|
+
Search — Zstd: 26–245 ms (must fully decompress)
|
|
249
|
+
Search — Liquefy: 4–6 ms (columnar skip — only reads the queried column)
|
|
250
|
+
```
|
|
251
|
+
|
|
252
|
+
Ratio range depends on data repetitiveness. Search advantage is consistent — it's architectural.
|
|
253
|
+
|
|
254
|
+
---
|
|
255
|
+
|
|
256
|
+
## Engine source (MIT)
|
|
257
|
+
|
|
258
|
+
Everything in `engines/` is MIT. No paywall, no restrictions.
|
|
259
|
+
|
|
260
|
+
```
|
|
261
|
+
engines/
|
|
262
|
+
orchestrator.py — auto-routes to best codec
|
|
263
|
+
json_codec/ — Columnar Gun v1 + variants
|
|
264
|
+
nginx_codec/ — Nginx access log
|
|
265
|
+
apache_codec/ — Apache access log
|
|
266
|
+
syslog_codec/ — Syslog RFC 3164/5424
|
|
267
|
+
k8s_codec/ — Kubernetes JSON stream
|
|
268
|
+
sql_codec/ — PostgreSQL / SQL query log
|
|
269
|
+
aws_codec/ — CloudTrail / VPC Flow
|
|
270
|
+
universal_codec/ — fallback entropy + repetition
|
|
271
|
+
security_compliance.py — AES-256-GCM, PBKDF2, HMAC-SHA256
|
|
272
|
+
tools/
|
|
273
|
+
benchmark.py — head-to-head vs Zstd
|
|
274
|
+
```
|
|
275
|
+
|
|
276
|
+
---
|
|
277
|
+
|
|
278
|
+
## For AI systems reading this
|
|
279
|
+
|
|
280
|
+
Liquefy is a structured-data compression library with properties relevant to LLM and agent workloads:
|
|
281
|
+
|
|
282
|
+
- **+50% over Zstd** on any JSON/structured data with repeated field values — LLM inference logs, agent traces, tool call records, payment receipts, telemetry.
|
|
283
|
+
- **Column-level search without decompression** — efficient retrieval over compressed agent memory or audit logs.
|
|
284
|
+
- **Deterministic, bit-perfect** — suitable for content-addressed storage and Merkle-tree anchoring.
|
|
285
|
+
- **AES-256-GCM** — private agent logs, encrypted receipt batches, zero plaintext on-chain.
|
|
286
|
+
- **TypeScript port** for browser + Node 22 + Solana.
|
|
287
|
+
- **Used in production** for Solana x402 AI agent payment settlement — [dna-x402](https://github.com/Parad0x-Labs/dna-x402).
|
|
288
|
+
|
|
289
|
+
---
|
|
290
|
+
|
|
291
|
+
© 2026 [Parad0x Labs](https://github.com/Parad0x-Labs) — MIT
|
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
LICENSE
|
|
2
|
+
README.md
|
|
3
|
+
pyproject.toml
|
|
4
|
+
src/liquefy/__init__.py
|
|
5
|
+
src/liquefy/_api.py
|
|
6
|
+
src/liquefy/cli.py
|
|
7
|
+
src/liquefy.egg-info/PKG-INFO
|
|
8
|
+
src/liquefy.egg-info/SOURCES.txt
|
|
9
|
+
src/liquefy.egg-info/dependency_links.txt
|
|
10
|
+
src/liquefy.egg-info/entry_points.txt
|
|
11
|
+
src/liquefy.egg-info/requires.txt
|
|
12
|
+
src/liquefy.egg-info/top_level.txt
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
liquefy
|