lmzip 1.0.0__tar.gz

lmzip-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Fanxin Sun
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

lmzip-1.0.0/PKG-INFO

@@ -0,0 +1,135 @@
+Metadata-Version: 2.4
+Name: lmzip
+Version: 1.0.0
+Summary: Lossless compression for model weights: 34.7% off BF16, byte for byte, with a store you can read through
+Author: Fanxin Sun
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/FanxinSun/lmz
+Project-URL: Repository, https://github.com/FanxinSun/lmz
+Project-URL: Issues, https://github.com/FanxinSun/lmz/issues
+Project-URL: Funding, https://buymeacoffee.com/fanxinsun
+Keywords: compression,lossless,safetensors,gguf,llm,model-weights,bfloat16,fp8,quantization,zstd,rans,entropy-coding,fuse,filesystem,model-store,checkpoint,machine-learning
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Operating System :: MacOS
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Programming Language :: C
+Classifier: Topic :: System :: Archiving :: Compression
+Classifier: Topic :: System :: Filesystems
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: zstd
+Requires-Dist: zstandard>=0.21; extra == "zstd"
+Dynamic: license-file
+
+# lmz
+
+[![tests](https://github.com/FanxinSun/lmz/actions/workflows/tests.yml/badge.svg)](https://github.com/FanxinSun/lmz/actions/workflows/tests.yml)
+[![PyPI](https://img.shields.io/pypi/v/lmzip)](https://pypi.org/project/lmzip/)
+[![Python](https://img.shields.io/pypi/pyversions/lmzip)](https://pypi.org/project/lmzip/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
+
+**Smaller checkpoints. Byte for byte.**
+
+Lossless compression built for model weights. `zstd -1` takes 22.7% off a
+Llama-3.1-8B BF16 checkpoint. lmz takes **34.7%** — and **64.6%** off the
+directory as Hugging Face actually ships it, which is **13 GB more than zstd
+on one 8B model**.
+
+Nothing is approximated. Every byte comes back.
+
+```
+pip install lmzip
+lmz compress ./Llama-3.1-8B-Instruct/
+```
+
+## What it saves
+
+Real checkpoints, every round-trip verified byte-identical.
+
+| | size | after | saved | zstd -1 |
+|---|---|---|---|---|
+| **Llama-3.1-8B, whole HF directory** | 32.13 GB | 11.38 GB | **64.6%** | 22.7% |
+| **Llama-3.1-8B, 4 BF16 shards** | 16.06 GB | 10.49 GB | **34.7%** | 22.7% |
+| Ministral-8B, whole directory | 32.11 GB | 11.55 GB | **64.0%** | 22.7% |
+| Pythia-160m, 3 training checkpoints | 1.81 GiB | 644 MiB | **65.3%** | 22.7% |
+| bge-m3 directory (FP32 container) | 4.59 GB | 2.45 GB | **46.5%** | — |
+| 8-bit AdamW optimizer state ×2 | 161 MiB | 119 MiB | **26.1%** | — |
+
+On BF16 weights lmz beats the published state of the art, and sits 0.3 points
+off the bound no lossless coder of any kind can pass:
+
+| on real Llama BF16 | saved |
+|---|---|
+| **lmz** | **34.7%** |
+| [ZipNN](https://github.com/zipnn/zipnn) (published, same model) | 33.6% |
+| [DFloat11](https://arxiv.org/pdf/2504.11651) (published) | ~30% |
+| bzip2 -9 | 30.7% |
+| xz -6 | 29.9% |
+| zstd -19 | 23.6% |
+| theoretical joint-entropy bound | 35.0% |
+
+**Three things a general compressor structurally cannot do**, which is where
+most of the margin comes from: store a tensor once when a directory ships it
+twice, code a checkpoint as the difference from the one before it, and split
+on a float's own bit-fields instead of byte boundaries.
+
+## Where it is not worth it
+
+Stated plainly, because a compressor that only advertises its wins should not
+be believed:
+
+| | lmz | best alternative | verdict |
+|---|---|---|---|
+| Quantised GGUF (Q8_0 / Q4_K_M) | 6.7% / 5.1% | 5.5% / 2.5% | the quantiser already took it |
+| FP8 safetensors | 17.14% | zstd -3, 17.11% | **just use zstd**, the gap is 0.03 points |
+| Text, code, JSON, binaries | = zstd | zstd | lmz *is* zstd here, by design |
+| Read speed | slower | a plain file | see below |
+
+Reading a compressed file transparently can never beat a plain one by more
+than `1/(1−saved)` — you still have to read the archive. That is 1.5× on BF16
+and 1.05× on Q4_K, so on a fast SSD the mount is *slower*. It buys disk, not
+speed.
+
+## Also included
+
+```
+lmz add ./my-model/ && lmz mount ~/models   # read a compressed model as ordinary
+                                            #   files; llama.cpp needs no patch
+lmz fs ~/.lmz/data ~/data                   # a read-write compressed filesystem;
+                                            #   32.1% where btrfs+zstd gets 18.9%
+```
+
+## Buy me a coffee
+
+lmz is free, MIT-licensed and unfunded. If it saved you disk or bandwidth —
+
+### [☕ **Buy me a coffee**](https://buymeacoffee.com/fanxinsun)
+
+or [Alipay](assets/alipay.jpg) (打开支付宝，扫一扫). Thank you.
+
+## Documentation
+
+- [**How it works**](docs/how-it-works.md) — why a float array defeats a
+  general-purpose compressor, and the bit-level choices that close the gap
+- [**Measured results**](docs/results.md) — every number, with its conditions
+- [**Using lmz**](docs/usage.md) — command line, Python API, the mount and the
+  filesystem
+- [**Limitations**](docs/limitations.md) — where it does not pay, and what the
+  81 tests check
+
+Python 3.10+, no runtime dependencies. zstd comes from the standard library on
+3.14+; a C compiler, if present, is used once to build the SIMD kernel into the
+package directory — nothing is installed system-wide. Runs straight from a
+checkout with `./lmz-cli` if you would rather not install it at all.
+
+Check what is active with `lmz doctor`.

lmzip-1.0.0/README.md

@@ -0,0 +1,102 @@
+# lmz
+
+[![tests](https://github.com/FanxinSun/lmz/actions/workflows/tests.yml/badge.svg)](https://github.com/FanxinSun/lmz/actions/workflows/tests.yml)
+[![PyPI](https://img.shields.io/pypi/v/lmzip)](https://pypi.org/project/lmzip/)
+[![Python](https://img.shields.io/pypi/pyversions/lmzip)](https://pypi.org/project/lmzip/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
+
+**Smaller checkpoints. Byte for byte.**
+
+Lossless compression built for model weights. `zstd -1` takes 22.7% off a
+Llama-3.1-8B BF16 checkpoint. lmz takes **34.7%** — and **64.6%** off the
+directory as Hugging Face actually ships it, which is **13 GB more than zstd
+on one 8B model**.
+
+Nothing is approximated. Every byte comes back.
+
+```
+pip install lmzip
+lmz compress ./Llama-3.1-8B-Instruct/
+```
+
+## What it saves
+
+Real checkpoints, every round-trip verified byte-identical.
+
+| | size | after | saved | zstd -1 |
+|---|---|---|---|---|
+| **Llama-3.1-8B, whole HF directory** | 32.13 GB | 11.38 GB | **64.6%** | 22.7% |
+| **Llama-3.1-8B, 4 BF16 shards** | 16.06 GB | 10.49 GB | **34.7%** | 22.7% |
+| Ministral-8B, whole directory | 32.11 GB | 11.55 GB | **64.0%** | 22.7% |
+| Pythia-160m, 3 training checkpoints | 1.81 GiB | 644 MiB | **65.3%** | 22.7% |
+| bge-m3 directory (FP32 container) | 4.59 GB | 2.45 GB | **46.5%** | — |
+| 8-bit AdamW optimizer state ×2 | 161 MiB | 119 MiB | **26.1%** | — |
+
+On BF16 weights lmz beats the published state of the art, and sits 0.3 points
+off the bound no lossless coder of any kind can pass:
+
+| on real Llama BF16 | saved |
+|---|---|
+| **lmz** | **34.7%** |
+| [ZipNN](https://github.com/zipnn/zipnn) (published, same model) | 33.6% |
+| [DFloat11](https://arxiv.org/pdf/2504.11651) (published) | ~30% |
+| bzip2 -9 | 30.7% |
+| xz -6 | 29.9% |
+| zstd -19 | 23.6% |
+| theoretical joint-entropy bound | 35.0% |
+
+**Three things a general compressor structurally cannot do**, which is where
+most of the margin comes from: store a tensor once when a directory ships it
+twice, code a checkpoint as the difference from the one before it, and split
+on a float's own bit-fields instead of byte boundaries.
+
+## Where it is not worth it
+
+Stated plainly, because a compressor that only advertises its wins should not
+be believed:
+
+| | lmz | best alternative | verdict |
+|---|---|---|---|
+| Quantised GGUF (Q8_0 / Q4_K_M) | 6.7% / 5.1% | 5.5% / 2.5% | the quantiser already took it |
+| FP8 safetensors | 17.14% | zstd -3, 17.11% | **just use zstd**, the gap is 0.03 points |
+| Text, code, JSON, binaries | = zstd | zstd | lmz *is* zstd here, by design |
+| Read speed | slower | a plain file | see below |
+
+Reading a compressed file transparently can never beat a plain one by more
+than `1/(1−saved)` — you still have to read the archive. That is 1.5× on BF16
+and 1.05× on Q4_K, so on a fast SSD the mount is *slower*. It buys disk, not
+speed.
+
+## Also included
+
+```
+lmz add ./my-model/ && lmz mount ~/models   # read a compressed model as ordinary
+                                            #   files; llama.cpp needs no patch
+lmz fs ~/.lmz/data ~/data                   # a read-write compressed filesystem;
+                                            #   32.1% where btrfs+zstd gets 18.9%
+```
+
+## Buy me a coffee
+
+lmz is free, MIT-licensed and unfunded. If it saved you disk or bandwidth —
+
+### [☕ **Buy me a coffee**](https://buymeacoffee.com/fanxinsun)
+
+or [Alipay](assets/alipay.jpg) (打开支付宝，扫一扫). Thank you.
+
+## Documentation
+
+- [**How it works**](docs/how-it-works.md) — why a float array defeats a
+  general-purpose compressor, and the bit-level choices that close the gap
+- [**Measured results**](docs/results.md) — every number, with its conditions
+- [**Using lmz**](docs/usage.md) — command line, Python API, the mount and the
+  filesystem
+- [**Limitations**](docs/limitations.md) — where it does not pay, and what the
+  81 tests check
+
+Python 3.10+, no runtime dependencies. zstd comes from the standard library on
+3.14+; a C compiler, if present, is used once to build the SIMD kernel into the
+package directory — nothing is installed system-wide. Runs straight from a
+checkout with `./lmz-cli` if you would rather not install it at all.
+
+Check what is active with `lmz doctor`.

lmzip-1.0.0/lmz/__init__.py

@@ -0,0 +1,40 @@
+"""lmz - fast lossless compression for large model weights.
+
+Model weights compress poorly with general-purpose tools because a float
+array looks like noise byte for byte. It is not: in a BF16 tensor the
+sign-and-exponent byte of every element takes only a few dozen distinct
+values, while the mantissa byte really is close to random. Separating those
+into distinct planes lets the exponents be entropy coded properly and the
+mantissas be passed through untouched, which is both smaller and faster than
+compressing the interleaved bytes.
+
+    import lmz
+    lmz.compress("model.safetensors", "model.lmz")
+    lmz.decompress("model.lmz", "restored.safetensors")
+
+Output is byte-for-byte identical to the input.
+
+A model can also be kept in the store and read in place, so nothing is ever
+expanded onto disk:
+
+    import lmz
+    lmz.Store().add("./my-model", "my-model")   # compressed, once
+
+    lmz mount ~/models                          # every runtime reads it as
+                                                #   ordinary model files
+"""
+
+__version__ = "1.0.0"
+
+from .api import (DEFAULT_CHUNK_SIZE, DEFAULT_LEVEL, Stats, backends, compress,
+                  decompress, info, read_tensor, verify)
+from .api import MappedArchive, append, extract  # noqa: F401
+from .format import FormatError
+from .lmzfs import LmzFS  # noqa: F401
+from .store import Store, mount  # noqa: F401
+
+__all__ = [
+    "compress", "decompress", "verify", "info", "read_tensor", "backends",
+    "MappedArchive", "append", "extract", "Store", "mount", "LmzFS",
+    "Stats", "FormatError", "DEFAULT_LEVEL", "DEFAULT_CHUNK_SIZE", "__version__",
+]

lmzip-1.0.0/lmz/__main__.py

@@ -0,0 +1,5 @@
+import sys
+
+from .cli import main
+
+sys.exit(main())