storetle 0.2.0__tar.gz

storetle-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Davis Brief
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

storetle-0.2.0/PKG-INFO

@@ -0,0 +1,161 @@
+Metadata-Version: 2.4
+Name: storetle
+Version: 0.2.0
+Summary: HTML-aware compression for document corpora — solid-archive ratios with random access
+Author-email: Davis Brief <davis@team8.co>
+License: MIT
+Project-URL: Homepage, https://github.com/adventurelands/storetle
+Project-URL: Specification, https://github.com/adventurelands/storetle/blob/main/FORMAT.md
+Keywords: html,compression,warc,web-archive,corpus,dataset,streaming
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Topic :: Internet :: WWW/HTTP
+Classifier: Topic :: System :: Archiving :: Compression
+Classifier: Programming Language :: Python :: 3
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: fast
+Requires-Dist: lxml; extra == "fast"
+Dynamic: license-file
+
+# Storetle
+
+**HTML-aware compression for document corpora — solid-archive ratios with random access.**
+
+Storetle stores large collections of HTML (web crawls, academic corpora,
+training datasets) in a format that is ~46% smaller than the per-record
+gzip WARC files the web-archiving world ships today, while still letting
+you pull any single document out of a multi-gigabyte archive without
+decompressing the rest — locally, or straight off object storage.
+
+```
+pip: storetle (Python, read/write)  ·  rust/: storetle-rs (Rust, read)  ·  web/: read .storetle in the browser
+```
+
+## The honest benchmark
+
+Two different questions, two tables. Corpus: 10 real pages (Wikipedia,
+arXiv abstracts, PLOS articles), 1.75 MB raw HTML, measured June 2026.
+Reproduce with `storetle bench <folder>`.
+
+**1. Among formats with random access** (you can extract one doc without
+decompressing everything before it — this is how WARC is actually deployed):
+
+| method | bytes | vs deployed standard |
+|---|---|---|
+| per-record gzip -9 (standard WARC) | 373,626 | — |
+| per-record zstd -19 | 325,807 | −12.8% |
+| per-record zstd -19 + trained dict | 274,226 | −26.6% |
+| **storetle** | **200,598** | **−46.3%** |
+
+**2. Against solid archives** (maximum compression, no random access):
+
+| method | bytes |
+|---|---|
+| tar + gzip -9 | 370,307 |
+| tar + zstd -19 | 220,512 |
+| tar + zstd -22 --long | 220,422 |
+| tar + zstd -22 + trained dict | 204,386 |
+| **storetle (keeps random access)** | **200,598** |
+
+Storetle matches solid zstd-22 while remaining randomly accessible. The
+margin comes from three things: HTML-aware encoding (tags/attributes become
+1-byte IDs from a shared vocabulary, structure and text compressed as
+separate streams), a 1 MB dictionary trained on the binary encoding, and
+256-document chunks that capture cross-page template redundancy.
+
+On larger corpora measured against gzip WARC: 28.4% smaller on 3,000 live
+Common Crawl docs (348.6 MB), 27–82% on same-domain collections (191 pages,
+20 domains) where template sharing is strongest. Round-trip verified on all
+of the above. Stream it yourself: `python3 bench_cc.py --docs 3000`.
+
+## Install
+
+```bash
+brew install zstd        # macOS   (Ubuntu: apt install libzstd-dev)
+pip install storetle
+```
+
+No Python dependencies — stdlib plus system libzstd via ctypes (brotli
+fallback if zstd is missing). `lxml` is optional but strongly recommended
+for encoding speed.
+
+## CLI
+
+```bash
+storetle pack      my_crawl/ archive.storetle     # folder of .html → archive
+storetle unpack    archive.storetle out/          # archive → .html files
+storetle info      archive.storetle               # stats
+storetle get       archive.storetle 42            # one doc to stdout, O(1)
+storetle bench     my_crawl/                      # benchmark on YOUR data
+storetle from-warc CC-MAIN.warc.gz archive.storetle
+storetle to-warc   archive.storetle out.warc.gz
+storetle train     my_corpus/ --output my.bin     # domain-specific dictionary
+```
+
+## Python API
+
+```python
+import storetle
+
+with storetle.StreamWriter('archive.storetle', workers=8) as w:
+    for html in crawl:
+        w.append(html)
+
+with storetle.StreamReader('archive.storetle') as r:
+    print(r.doc_count)
+    doc   = r[42]          # random access: decompresses one ~2MB chunk
+    batch = r[100:200]
+    for doc in r:          # sequential
+        ...
+```
+
+## Rust reader
+
+A read-only Rust implementation lives in [`rust/`](rust/) — library plus a
+`storetle-rs` CLI (`ls` / `get` / `unpack`), differentially tested
+byte-for-byte against the Python decoder.
+
+## In the browser
+
+[`web/`](web/) has a zero-dependency demo page: the Rust reader compiled to
+WebAssembly. Drop a `.storetle` file onto the page and browse its documents.
+
+## How it works
+
+1. **Parse** — HTML is tokenized to a node stream (lxml fast path, pure-Python fallback).
+2. **Encode** — tags and attribute names become 1-byte IDs from a fixed
+   vocabulary (130 tags, 163 attributes, 1,394 shared strings).
+   `class="flex items-center gap-4"` is split into per-token vocabulary
+   lookups. Structure and text go to separate streams.
+3. **Chunk** — up to 256 docs / 2 MiB are concatenated, preserving
+   cross-document redundancy.
+4. **Compress** — zstd-22 with a 1 MB dictionary trained on the binary
+   encoding (ships with the codec).
+5. **Index** — a footer index maps documents to chunks, so readers seek
+   instead of scanning. Works over HTTP range requests against plain
+   object storage.
+
+Full byte-level spec: [FORMAT.md](FORMAT.md).
+
+## Limitations — read these
+
+- **Structural, not byte-exact.** Reconstructed HTML preserves every tag,
+  attribute, text node, comment, and script/style body, but is
+  re-serialized (indentation and inter-tag whitespace differ). Fine for
+  corpora and ML pipelines; **wrong for byte-exact archival** — if you need
+  forensic fidelity, use WARC.
+- **HTML only.** `from-warc` keeps HTML response records and skips
+  everything else. A raw passthrough mode for JSON/text is on the roadmap.
+- **Encoding speed** ~3.5 MB/s per core (Python). Parallel via
+  `workers=N`. Decoding is zstd-bound and fast. A native encoder is on the
+  roadmap.
+- **Alpha.** Format version 2. Validated on 150k+ Common Crawl documents,
+  but expect rough edges.
+
+## License
+
+MIT © 2026 Davis Brief

storetle-0.2.0/README.md

@@ -0,0 +1,138 @@
+# Storetle
+
+**HTML-aware compression for document corpora — solid-archive ratios with random access.**
+
+Storetle stores large collections of HTML (web crawls, academic corpora,
+training datasets) in a format that is ~46% smaller than the per-record
+gzip WARC files the web-archiving world ships today, while still letting
+you pull any single document out of a multi-gigabyte archive without
+decompressing the rest — locally, or straight off object storage.
+
+```
+pip: storetle (Python, read/write)  ·  rust/: storetle-rs (Rust, read)  ·  web/: read .storetle in the browser
+```
+
+## The honest benchmark
+
+Two different questions, two tables. Corpus: 10 real pages (Wikipedia,
+arXiv abstracts, PLOS articles), 1.75 MB raw HTML, measured June 2026.
+Reproduce with `storetle bench <folder>`.
+
+**1. Among formats with random access** (you can extract one doc without
+decompressing everything before it — this is how WARC is actually deployed):
+
+| method | bytes | vs deployed standard |
+|---|---|---|
+| per-record gzip -9 (standard WARC) | 373,626 | — |
+| per-record zstd -19 | 325,807 | −12.8% |
+| per-record zstd -19 + trained dict | 274,226 | −26.6% |
+| **storetle** | **200,598** | **−46.3%** |
+
+**2. Against solid archives** (maximum compression, no random access):
+
+| method | bytes |
+|---|---|
+| tar + gzip -9 | 370,307 |
+| tar + zstd -19 | 220,512 |
+| tar + zstd -22 --long | 220,422 |
+| tar + zstd -22 + trained dict | 204,386 |
+| **storetle (keeps random access)** | **200,598** |
+
+Storetle matches solid zstd-22 while remaining randomly accessible. The
+margin comes from three things: HTML-aware encoding (tags/attributes become
+1-byte IDs from a shared vocabulary, structure and text compressed as
+separate streams), a 1 MB dictionary trained on the binary encoding, and
+256-document chunks that capture cross-page template redundancy.
+
+On larger corpora measured against gzip WARC: 28.4% smaller on 3,000 live
+Common Crawl docs (348.6 MB), 27–82% on same-domain collections (191 pages,
+20 domains) where template sharing is strongest. Round-trip verified on all
+of the above. Stream it yourself: `python3 bench_cc.py --docs 3000`.
+
+## Install
+
+```bash
+brew install zstd        # macOS   (Ubuntu: apt install libzstd-dev)
+pip install storetle
+```
+
+No Python dependencies — stdlib plus system libzstd via ctypes (brotli
+fallback if zstd is missing). `lxml` is optional but strongly recommended
+for encoding speed.
+
+## CLI
+
+```bash
+storetle pack      my_crawl/ archive.storetle     # folder of .html → archive
+storetle unpack    archive.storetle out/          # archive → .html files
+storetle info      archive.storetle               # stats
+storetle get       archive.storetle 42            # one doc to stdout, O(1)
+storetle bench     my_crawl/                      # benchmark on YOUR data
+storetle from-warc CC-MAIN.warc.gz archive.storetle
+storetle to-warc   archive.storetle out.warc.gz
+storetle train     my_corpus/ --output my.bin     # domain-specific dictionary
+```
+
+## Python API
+
+```python
+import storetle
+
+with storetle.StreamWriter('archive.storetle', workers=8) as w:
+    for html in crawl:
+        w.append(html)
+
+with storetle.StreamReader('archive.storetle') as r:
+    print(r.doc_count)
+    doc   = r[42]          # random access: decompresses one ~2MB chunk
+    batch = r[100:200]
+    for doc in r:          # sequential
+        ...
+```
+
+## Rust reader
+
+A read-only Rust implementation lives in [`rust/`](rust/) — library plus a
+`storetle-rs` CLI (`ls` / `get` / `unpack`), differentially tested
+byte-for-byte against the Python decoder.
+
+## In the browser
+
+[`web/`](web/) has a zero-dependency demo page: the Rust reader compiled to
+WebAssembly. Drop a `.storetle` file onto the page and browse its documents.
+
+## How it works
+
+1. **Parse** — HTML is tokenized to a node stream (lxml fast path, pure-Python fallback).
+2. **Encode** — tags and attribute names become 1-byte IDs from a fixed
+   vocabulary (130 tags, 163 attributes, 1,394 shared strings).
+   `class="flex items-center gap-4"` is split into per-token vocabulary
+   lookups. Structure and text go to separate streams.
+3. **Chunk** — up to 256 docs / 2 MiB are concatenated, preserving
+   cross-document redundancy.
+4. **Compress** — zstd-22 with a 1 MB dictionary trained on the binary
+   encoding (ships with the codec).
+5. **Index** — a footer index maps documents to chunks, so readers seek
+   instead of scanning. Works over HTTP range requests against plain
+   object storage.
+
+Full byte-level spec: [FORMAT.md](FORMAT.md).
+
+## Limitations — read these
+
+- **Structural, not byte-exact.** Reconstructed HTML preserves every tag,
+  attribute, text node, comment, and script/style body, but is
+  re-serialized (indentation and inter-tag whitespace differ). Fine for
+  corpora and ML pipelines; **wrong for byte-exact archival** — if you need
+  forensic fidelity, use WARC.
+- **HTML only.** `from-warc` keeps HTML response records and skips
+  everything else. A raw passthrough mode for JSON/text is on the roadmap.
+- **Encoding speed** ~3.5 MB/s per core (Python). Parallel via
+  `workers=N`. Decoding is zstd-bound and fast. A native encoder is on the
+  roadmap.
+- **Alpha.** Format version 2. Validated on 150k+ Common Crawl documents,
+  but expect rough edges.
+
+## License
+
+MIT © 2026 Davis Brief

storetle-0.2.0/pyproject.toml

@@ -0,0 +1,42 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "storetle"
+version = "0.2.0"
+description = "HTML-aware compression for document corpora — solid-archive ratios with random access"
+readme = "README.md"
+requires-python = ">=3.8"
+license = { text = "MIT" }
+authors = [{ name = "Davis Brief", email = "davis@team8.co" }]
+keywords = ["html", "compression", "warc", "web-archive", "corpus", "dataset", "streaming"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Topic :: Internet :: WWW/HTTP",
+    "Topic :: System :: Archiving :: Compression",
+    "Programming Language :: Python :: 3",
+]
+
+# No hard Python dependencies — stdlib + system libzstd via ctypes.
+# lxml is optional but strongly recommended for encoding throughput.
+dependencies = []
+
+[project.optional-dependencies]
+fast = ["lxml"]
+
+[project.urls]
+Homepage = "https://github.com/adventurelands/storetle"
+Specification = "https://github.com/adventurelands/storetle/blob/main/FORMAT.md"
+
+[project.scripts]
+storetle = "storetle.cli:main"
+
+[tool.setuptools]
+packages = ["storetle"]
+
+[tool.setuptools.package-data]
+storetle = ["cube_dict_v10.bin"]

storetle-0.2.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

storetle-0.2.0/storetle/__init__.py

@@ -0,0 +1,111 @@
+# storetle — HTML-aware streaming compression for large document collections
+#
+# Primary API:
+#   StreamWriter   — append HTML documents to a .storetle file
+#   StreamReader   — read/iterate/random-access a .storetle file
+#   pack           — compress a folder of HTML files → .storetle
+#   unpack         — decompress .storetle → folder of HTML files
+#   benchmark      — compare storetle vs gzip on your own data
+
+from .stream import StreamWriter, StreamReader
+from .folder import pack, unpack
+
+__version__ = '0.2.0'
+__all__ = ['StreamWriter', 'StreamReader', 'pack', 'unpack', 'benchmark']
+
+
+def benchmark(folder, quiet=False):
+    """Benchmark storetle vs gzip WARC on a folder of HTML files.
+
+    Returns a dict with size comparisons. Prints a table unless quiet=True.
+
+    Example:
+        import storetle
+        results = storetle.benchmark('my_crawl_data/')
+    """
+    import gzip, os, tempfile, time
+    from pathlib import Path
+
+    files = sorted(Path(folder).glob('**/*.html'))
+    if not files:
+        raise ValueError(f'No .html files found in {folder}')
+
+    docs = [f.read_bytes() for f in files]
+    total_html = sum(len(d) for d in docs)
+
+    # gzip WARC (industry standard)
+    warc_raw = b''.join(
+        'WARC/1.0\r\nContent-Length: {}\r\n\r\n'.format(len(d)).encode()
+        + d + b'\r\n\r\n'
+        for d in docs
+    )
+    warc_gz = len(gzip.compress(warc_raw, compresslevel=9))
+
+    # gzip per-file
+    gz_pf = sum(len(gzip.compress(d, compresslevel=9)) for d in docs)
+
+    # storetle
+    with tempfile.NamedTemporaryFile(suffix='.storetle', delete=False) as tf:
+        tmp = tf.name
+    try:
+        t0 = time.time()
+        with StreamWriter(tmp) as w:
+            for d in docs:
+                w.append(d)
+        write_time = time.time() - t0
+
+        t1 = time.time()
+        with StreamReader(tmp) as r:
+            recovered = list(r)
+        read_time = time.time() - t1
+
+        cube_size = os.path.getsize(tmp)
+    finally:
+        os.unlink(tmp)
+
+    rt_ok = len(recovered) == len(docs)
+
+    result = {
+        'files':          len(files),
+        'original_bytes': total_html,
+        'gzip_warc':      warc_gz,
+        'gzip_per_file':  gz_pf,
+        'storetle':     cube_size,
+        'savings_vs_gzip_warc_pct': round((warc_gz - cube_size) / warc_gz * 100, 1),
+        'write_kbps':     int(total_html / 1024 / max(write_time, 0.001)),
+        'read_kbps':      int(total_html / 1024 / max(read_time, 0.001)),
+        'roundtrip_ok':   rt_ok,
+    }
+
+    if not quiet:
+        _print_benchmark(result)
+
+    return result
+
+
+def _print_benchmark(r):
+    def fmt(n):
+        if n < 1024: return f'{n}B'
+        if n < 1048576: return f'{n/1024:.1f}KB'
+        return f'{n/1048576:.2f}MB'
+
+    def pct(a, b):
+        return f'{100*(1-a/b):.1f}%'
+
+    orig = r['original_bytes']
+    print(f'\n  storetle benchmark — {r["files"]} files, {fmt(orig)} original\n')
+    print(f'  {"Format":<28} {"Size":>10}  {"Savings":>8}')
+    print(f'  {"─"*50}')
+    print(f'  {"Original HTML":<28} {fmt(orig):>10}')
+    print(f'  {"gzip per-file (current)":<28} {fmt(r["gzip_per_file"]):>10}  {pct(r["gzip_per_file"], orig):>8}')
+    print(f'  {"gzip WARC (Common Crawl std)":<28} {fmt(r["gzip_warc"]):>10}  {pct(r["gzip_warc"], orig):>8}')
+    print(f'  {"storetle":<28} {fmt(r["storetle"]):>10}  {pct(r["storetle"], orig):>8}')
+    print()
+
+    saved = r["gzip_warc"] - r["storetle"]
+    sign  = '+' if saved > 0 else ''
+    print(f'  vs gzip WARC:  {sign}{r["savings_vs_gzip_warc_pct"]}% smaller  ({fmt(abs(saved))} {"saved" if saved > 0 else "larger"})')
+    print(f'  Write speed:   {r["write_kbps"]:,} KB/s')
+    print(f'  Read speed:    {r["read_kbps"]:,} KB/s')
+    print(f'  Round-trip:    {"✓ all documents verified" if r["roundtrip_ok"] else "✗ FAILED"}')
+    print()

storetle-0.2.0/storetle/brotli_compat.py

@@ -0,0 +1,96 @@
+# brotli_compat.py — thin ctypes wrapper around the system brotli library
+#
+# Exposes compress(data, quality=11) and decompress(data) with the same
+# calling convention as zlib.compress / zlib.decompress so callers can
+# swap one for the other with a single import change.
+#
+# Falls back to zlib if the brotli dylib is not present (other machines).
+
+import ctypes, zlib, os
+
+_BROTLI_OK   = False
+_enc = None
+_dec = None
+
+def _try_load():
+    global _BROTLI_OK, _enc, _dec
+    candidates = [
+        '/usr/local/lib/libbrotlienc.dylib',
+        '/usr/local/Cellar/brotli/1.2.0/lib/libbrotlienc.dylib',
+        'libbrotlienc.so',
+        'libbrotlienc.dylib',
+    ]
+    dec_candidates = [p.replace('enc', 'dec') for p in candidates]
+
+    for ep, dp in zip(candidates, dec_candidates):
+        if os.path.exists(ep) and os.path.exists(dp):
+            try:
+                e = ctypes.CDLL(ep)
+                d = ctypes.CDLL(dp)
+                # wire up encoder
+                e.BrotliEncoderMaxCompressedSize.restype  = ctypes.c_size_t
+                e.BrotliEncoderMaxCompressedSize.argtypes = [ctypes.c_size_t]
+                e.BrotliEncoderCompress.restype  = ctypes.c_int
+                e.BrotliEncoderCompress.argtypes = [
+                    ctypes.c_int, ctypes.c_int, ctypes.c_int,
+                    ctypes.c_size_t, ctypes.c_char_p,
+                    ctypes.POINTER(ctypes.c_size_t), ctypes.c_char_p,
+                ]
+                # wire up decoder
+                d.BrotliDecoderDecompress.restype  = ctypes.c_int
+                d.BrotliDecoderDecompress.argtypes = [
+                    ctypes.c_size_t, ctypes.c_char_p,
+                    ctypes.POINTER(ctypes.c_size_t), ctypes.c_char_p,
+                ]
+                _enc = e; _dec = d; _BROTLI_OK = True
+                return
+            except Exception:
+                continue
+
+_try_load()
+
+
+def compress(data: bytes, quality: int = 11, lgwin: int = 24, mode: int = 0) -> bytes:
+    """Compress data with brotli (quality 0-11, mode 0=generic 1=text 2=font).
+    Falls back to zlib level 9."""
+    if not _BROTLI_OK:
+        return zlib.compress(data, level=9)
+
+    max_out  = _enc.BrotliEncoderMaxCompressedSize(len(data))
+    out_buf  = ctypes.create_string_buffer(max_out)
+    out_size = ctypes.c_size_t(max_out)
+
+    ok = _enc.BrotliEncoderCompress(
+        quality, lgwin, mode,
+        len(data), data,
+        ctypes.byref(out_size), out_buf,
+    )
+    if not ok:
+        raise RuntimeError('brotli compression failed')
+    return out_buf.raw[:out_size.value]
+
+
+def decompress(data: bytes) -> bytes:
+    """Decompress brotli data. Falls back to zlib if brotli unavailable."""
+    if not _BROTLI_OK:
+        return zlib.decompress(data)
+
+    # Grow output buffer until it fits (decompressed size is unknown).
+    max_out = max(len(data) * 10, 1 << 20)   # start at 10× or 1 MB
+    while True:
+        out_buf  = ctypes.create_string_buffer(max_out)
+        out_size = ctypes.c_size_t(max_out)
+        result   = _dec.BrotliDecoderDecompress(
+            len(data), data,
+            ctypes.byref(out_size), out_buf,
+        )
+        if result == 1:   # BROTLI_DECODER_RESULT_SUCCESS
+            return out_buf.raw[:out_size.value]
+        if result == 3:   # BROTLI_DECODER_RESULT_NEEDS_MORE_OUTPUT
+            max_out *= 4
+            continue
+        raise RuntimeError(f'brotli decompression failed (result={result})')
+
+
+def available() -> bool:
+    return _BROTLI_OK