polypress 0.2.0__py3-none-any.whl

polypress-0.2.0.dist-info/METADATA

@@ -0,0 +1,121 @@
+Metadata-Version: 2.4
+Name: polypress
+Version: 0.2.0
+Summary: Lossless compression for data tables, smaller than xz, zstd, brotli and Parquet on coded survey and administrative data
+Author: Mahdi Akbarin
+License-Expression: MIT
+Project-URL: Homepage, https://www.polypressapp.com
+Project-URL: Repository, https://github.com/potJim80/PolyPressApp
+Keywords: compression,csv,parquet,columnar,survey data
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: System :: Archiving :: Compression
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.20
+Provides-Extra: parquet
+Requires-Dist: pyarrow>=8; extra == "parquet"
+Dynamic: license-file
+
+# Polypress
+
+**A lossless compressor for data tables.** CSV, Parquet, and anything else
+shaped like rows and columns.
+
+General-purpose compressors see a table as a stream of bytes. Columnar formats
+see it as columns, and compress each one on its own. Polypress is built on the
+fact that **the columns of a real table are not independent of each other** —
+`City` is nearly determined by `Postal Code`, `latitude` is repeated inside
+`location`, adjacent sensor columns hold nearly the same number — and models
+those relationships directly.
+
+The original table comes back **exactly**: same columns, same column order,
+same row order, every cell as the exact string it was.
+
+```sh
+pip install polypress
+polypress compress data.csv        # -> data.csv.ppz
+polypress restore  data.csv.ppz    # -> data.csv
+polypress info     data.csv.ppz    # plan, shape, how much was reordered
+```
+
+## The benchmark, and how it was chosen
+
+The obvious objection to any compression result is *"you picked the files"*, so
+the main benchmark does not pick. It asks the Socrata open-data catalog for its
+datasets **in descending order of page views** and takes the first 500 that
+pass four mechanical filters (the CSV downloads; at least 2 columns and 20
+rows; at least 50 KB; not a duplicate). Rank order is public and fixed, so the
+list reproduces, and every rejection is recorded with its reason.
+
+**500 tables, 3.57 GB of CSV, 14.3 million rows, 10,557 columns**, against 17
+competing codecs:
+
+| | |
+|---|---|
+| Round-trips exactly | **500 of 500** |
+| Smaller than the best of all 17 competitors | **478 of 500** |
+| Margin over the best other tool | median **1.25x**, best 2.73x, worst 0.65x |
+| Compression vs raw CSV | median **14.35x** |
+
+Head to head: it beats `xz -9e`, `zstd -22`, `gzip -9` and `lz4 -9` on **all
+500**; `parquet+zstd` on 449 of 450; `bzip2 -9` on 491 of 500; `brotli -q 11`
+on 487 of 500.
+
+**Two things belong next to those numbers.** Parquet, read with type inference,
+reproduced the exact printed text on only 81 of the 500 tables — turning
+`"1.50"` into `1.5` makes a smaller file for reasons that have nothing to do
+with compression, and Polypress guarantees the printed cell. And the margin is
+not a better final compressor: re-finishing the same modelled streams with the
+competitor's own entropy coder still wins 449 of 450 against `parquet+zstd`.
+
+## How it works
+
+1. **Predict down a column.** Fit a low-degree polynomial to the last few
+   values, extrapolate one step, store the error. Re-centring the fit at every
+   cell means the coefficients never have to be stored — order-*k*
+   extrapolation is exactly the *k*-th finite difference.
+2. **The same thing in two dimensions.** Where adjacent numeric columns are
+   *commensurable*, a cell is predicted from its left, upper and upper-left
+   neighbours. This is what wins on matrix-shaped tables, and no shipped
+   columnar codec does it — Gorilla, DoubleDelta, T64, zfp and fpzip all
+   predict down a single column.
+3. **Reorder the rows so a column collapses into runs.** Rather than model that
+   `City` depends on `Postal Code`, sort by the parent: equal parents become
+   adjacent and the child collapses. **The permutation is free** — the decoder
+   has already rebuilt the parent and recomputes the same stable sort. Your
+   table comes back in its original row order.
+
+## What it is honest about
+
+- **It is slow to compress.** Median 2.0 MB/s encode, 134 MB/s decode. The
+  never-worse guarantee is what costs it: eligible tables are encoded twice and
+  the smaller result wins.
+- **The core idea is not novel.** US 8,312,026 B2 (Vo, AT&T, filed 2009)
+  discloses it. It was arrived at here independently and the novelty claim was
+  withdrawn. Both relevant patents have expired.
+- **The 2x results are matrix-shaped tables.** 1.3–1.5x is typical, and
+  text-heavy tables are the weak genre.
+- **All 22 losses out of 500 are published**, including the one genuine
+  undiagnosed loss.
+- **Every benchmark dataset is a government open-data table.** There is no
+  reason to assume the result transfers to other genres.
+- **Nobody outside the project has run it.**
+
+## Requirements
+
+Python 3.9+ and numpy. A C compiler is optional — the accelerator compiles
+itself on first import and falls back to numpy without one. `pyarrow` is
+needed only for `.parquet` input and output: `pip install 'polypress[parquet]'`.
+
+## More
+
+Source, tests, the full benchmark harness and the committed sweep results:
+**https://github.com/potJim80/PolyPressApp** — including `memory/LAWS.md`, the
+findings that govern what gets built, and the README section listing everything
+that was tried and failed.
+
+MIT licensed.

polypress-0.2.0.dist-info/RECORD

@@ -0,0 +1,15 @@
+polypress/__init__.py,sha256=wo-1RDT5el4Ra_iF0azwVAhQDZnu9myJs9LUp756rhE,477
+polypress/caccel.py,sha256=7l7LR22be8GpIYpa-13Nx6B54V5XRtf43h83dXWSAuU,5471
+polypress/cli.py,sha256=vNF7hlmdUfFx4Uxb9rA_FijC_kV1ONu-SfqaWy8lD14,9922
+polypress/codec.py,sha256=z8DZ5LWA-Ri67OL6nfXMdVaahNhH1WVH1wagLSdEg8o,9904
+polypress/dtz.py,sha256=sNYYPYbPt_Ky0tGydLWt1lOSzEBFAbPwYunp33DfVqg,34800
+polypress/fast.py,sha256=cD0m7K7lPZQGO0zXIY_gVICgpaTBLgR6OzDPqNOou-o,58551
+polypress/stream.py,sha256=CLXWqfvDW-r0LaG7jCwdoFU8pK3wnVfGtiZJAgYBQVw,15832
+polypress/tcz.c,sha256=sNlSit2eYzZPdFfPTHGM55Rl-0Zi0xYk3krvX70DI50,7357
+polypress/turbo.py,sha256=xzvYgXBX9D4DzKTqcxVmq4YRD2AmYsKo9HEnrs-hlPI,63160
+polypress-0.2.0.dist-info/licenses/LICENSE,sha256=dzEqVzuWEUSE2lOUbeaCgxHUMtohs4O_WeBLe6hrB70,1070
+polypress-0.2.0.dist-info/METADATA,sha256=LdjKaprJRFF8l-qnLpOrPWyRWdxs8Pl78pqKiSn4l9U,5561
+polypress-0.2.0.dist-info/WHEEL,sha256=aeYiig01lYGDzBgS8HxWXOg3uV61G9ijOsup-k9o1sk,91
+polypress-0.2.0.dist-info/entry_points.txt,sha256=DI8n7uEWagVuaUdc_eb0OyK-wRyRtrjkQAIL91ur01s,49
+polypress-0.2.0.dist-info/top_level.txt,sha256=j-bQ0QOAfEB1hzvrb0gVFUTicSSGS89znrvQ2zI0Lfc,10
+polypress-0.2.0.dist-info/RECORD,,

polypress-0.2.0.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (82.0.1)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

polypress-0.2.0.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+polypress = polypress.cli:main

polypress-0.2.0.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Mahdi Akbarin
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

polypress-0.2.0.dist-info/top_level.txt

@@ -0,0 +1 @@
+polypress