splitgrid 0.1.0__tar.gz → 0.1.2__tar.gz

{splitgrid-0.1.0 → splitgrid-0.1.2}/Makefile

@@ -1,6 +1,6 @@
 # SplitGrid
 
-.PHONY: test test-all test-slow verify install-test
+.PHONY: test test-all test-slow verify install-test bench bench-child profile-pack
 
 # Prefer a local venv so `make test` / `make verify` work without activating it.
 PYTEST := $(if $(wildcard .venv/bin/pytest),.venv/bin/pytest,python3 -m pytest)
@@ -22,3 +22,16 @@ test-slow:
 # deal contracts + hypothesis round-trips (excludes CrossHair subprocess).
 verify:
 	$(PYTEST) tests/test_serialization_verification.py tests/test_payload_codec_policy_verification.py -m "not slow" -q
+
+PYTHON := $(if $(wildcard .venv/bin/python),.venv/bin/python,python3)
+
+# Asymmetric host/child serialization bench (needs numpy).
+bench:
+	$(PYTHON) scripts/bench_serialization.py --direction both
+
+bench-child:
+	$(PYTHON) scripts/bench_serialization.py --child-only
+
+profile-pack:
+	$(PYTHON) scripts/profile_pack.py
+	$(PYTHON) scripts/profile_nones.py

{splitgrid-0.1.0/src/splitgrid.egg-info → splitgrid-0.1.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: splitgrid
-Version: 0.1.0
+Version: 0.1.2
 Summary: Asymmetric split-grid serialization: host stdlib flatten + optional Cython, child NumPy unpack
 Author-email: Keith Cu <keithcu@gmail.com>
 License: GPL-3.0-or-later
@@ -35,23 +35,33 @@ Dynamic: license-file
 
 # SplitGrid
 
+[![PyPI](https://img.shields.io/pypi/v/splitgrid.svg)](https://pypi.org/project/splitgrid/)
+
 Asymmetric **split-grid** serialization for rectangular numeric and mixed-type grids.
 
-This package was **pulled out of [WriterAgent](https://github.com/KeithCu/writeragent)** (LibreOffice Writer/Calc/Draw AI extension). It is the same host flatten + child unpack codec that lived in `plugin.scripting.payload_codec`, plus the optional Cython flatten accelerator from `native/writeragent_vec`. WriterAgent can later depend on `splitgrid` without changing the wire dict.
+This package was **pulled out of [WriterAgent](https://github.com/KeithCu/writeragent)** (LibreOffice Writer/Calc/Draw AI extension). It is the host flatten + child unpack codec that lived in `plugin.scripting.payload_codec`, plus the optional Cython flatten accelerator from `native/writeragent_vec`.
+
+## Why this exists
+
+A spreadsheet host often cannot import NumPy: LibreOffice’s embedded Python is a different interpreter from the user’s venv, and loading the user’s C extensions into the host is an ABI footgun. Heavy compute therefore runs in a **child** process that *does* have NumPy. The range still has to cross that boundary.
+
+On the wire, a Calc-style range is a nested list: `list[list[float | int | str | None]]`. Standard pickle walks **one heap object per cell**. On a 20,000 × 5 grid that is ~12 ms for dump + load. Pack the same numbers into a contiguous `float64` buffer and Pickle Protocol 5 moves the bytes in ~0.015 ms; the child can `np.frombuffer` in ~0.002 ms.
+
+Almost all of the remaining time is **host flatten** — turning nested Python objects into that buffer without NumPy (~8.3 ms pure Python on that shape; ~3 ms with the Cython helper). SplitGrid is that flatten/unpack. Length-prefixed Pickle 5 framing stays in the application; this package is the codec only.
 
-Repo: [github.com/KeithCu/writeragent](https://github.com/KeithCu/writeragent)
+Column-wise blobs and JSON + Base64 were tried first. Transposing columns on the host builds extra object graphs; Base64 and per-column reconstructs lose the `frombuffer` win. One row-major float64 buffer plus a sparse string map is faster and simpler.
 
-## What split-grid is
+## How it works
 
-The compute path is **asymmetric by design**:
+The path is **asymmetric**:
 
-- **Host pack** (LibreOffice’s embedded Python, or any NumPy-free interpreter) flattens a 1D or 2D grid into a contiguous `float64` buffer plus a sparse **integer-keyed** `strings` map. Empty cells become `NaN`. Zip codes like `"02138"` stay strings — they are never parsed as floats.
-- **Child unpack** (venv with NumPy) materializes a **pure-numeric** grid (`strings == {}`) with `np.frombuffer` (ndarray). Mixed grids use a vectorized object-masking path and return nested lists, restoring `None` for NaN holes.
-- **Host unpack** preserves `float('nan')` from the buffer (does **not** coerce holes to `None`). That is the locked egress policy: NaN becomes a Calc error, not a silent blank.
+- **Host pack** (stdlib only) flattens a 1D or 2D rectangular grid into a contiguous `float64` buffer plus a sparse **integer-keyed** `strings` map. Empty cells become `NaN`. Zip codes like `"02138"` stay strings — they are never parsed as floats.
+- **Child unpack** (NumPy) materializes a **pure-numeric** grid (`strings == {}`) with `np.frombuffer` (ndarray). Mixed grids use a vectorized object-masking path and return nested lists, restoring `None` for NaN holes.
+- **Host unpack** preserves `float('nan')` from the buffer (does **not** coerce holes to `None`). That is the locked egress policy: NaN becomes a spreadsheet error, not a silent blank.
 
 Grids with fewer than **100 cells** (`BINARY_MIN_CELLS`) stay nested Python lists. Force `"always"` / `"never"` overrides that threshold (used by A/B tests).
 
-Wire envelope (Pickle5-friendly dict, same keys as WriterAgent):
+Wire envelope (Pickle5-friendly dict):
 
 ```python
 {
@@ -64,9 +74,30 @@ Wire envelope (Pickle5-friendly dict, same keys as WriterAgent):
 }
 ```
 
+| Cell value | `buffer` (float64) | `strings` |
+|------------|--------------------|-----------|
+| `None` (empty cell) | `NaN` | — |
+| `int` / `float` | numeric value | — |
+| `bool` | `0.0` / `1.0` | — |
+| `str` (including `"02138"`) | `NaN` | text by flat index |
+
 There is **no datetime lane** on the float64 buffer. Python `datetime` objects stringify into `strings`. Do not add a `'date'` column kind.
 
-Jagged 2D grids raise `ValueError` (Calc ranges are rectangular).
+Jagged 2D grids raise `ValueError` (spreadsheet ranges are rectangular).
+
+## Numbers
+
+Median timings from an asymmetric bench (host = stdlib pack; child = deserialize + materialize).
+
+Ingress, **20,000 × 5** (100k cells):
+
+| Format | Pack | Dump | Load | Materialize | Total |
+|--------|------|------|------|-------------|-------|
+| JSON nested lists | 6.2 ms | 22.8 ms | 14.6 ms | 2.1 ms | 45.7 ms |
+| Pickle 5 nested lists | 6.1 ms | 1.4 ms | 2.9 ms | 2.0 ms | 12.4 ms |
+| **Pickle 5 + split-grid** | 8.3 ms | **0.013 ms** | **0.015 ms** | **0.002 ms** | **8.3 ms** |
+
+Child materialize of a **100 × 100** numeric grid: nested-list pickle then `np.array` ~0.6 ms; split-grid `frombuffer` ~**0.016 ms**. Below 100 cells the envelope is not worth it — that is why `BINARY_MIN_CELLS` exists. Host pack still dominates large ingress; Cython only speeds that loop.
 
 ## Python vs Cython
 
@@ -78,7 +109,7 @@ Host flatten is an optimized **pure-Python** loop:
 - lazy column-state upgrades
 - rectangular validation before the hot loop
 
-The optional **Cython** module `splitgrid.pack` exposes `fast_flatten_grid_1d` / `fast_flatten_grid_2d`. It is loaded dynamically and **canary-tested** at import. If the extension is missing or fails the canary, the codec uses pure Python. Importing `splitgrid` never requires a compiler.
+The optional **Cython** module `splitgrid.pack` exposes `fast_flatten_grid_1d` / `fast_flatten_grid_2d`. It is loaded dynamically and **canary-tested** at import. If the extension is missing or fails the canary, the codec uses pure Python. Importing `splitgrid` never requires a compiler. Extension builds use release flags (`-O3 -DNDEBUG -g0` on Unix, `/O2 /DNDEBUG` on Windows).
 
 ## Install
 
@@ -108,7 +139,21 @@ make verify             # deal contracts + hypothesis round-trips, no CrossHair
 
 Default pytest **does not require** the compiled extension. When `splitgrid.pack` is built, `tests/test_cython_parity.py` compares Cython and Python flatten components on the same grids.
 
-## Public API (WriterAgent-compatible)
+## Bench
+
+Requires NumPy (and the optional Cython extension if you want the accelerated pack row). From a source checkout:
+
+```bash
+python scripts/bench_serialization.py --direction both
+python scripts/bench_serialization.py --child-only
+python scripts/bench_unpacking_opt.py
+python scripts/profile_pack.py
+python scripts/profile_nones.py
+python scripts/run_serialization_ab.py --all
+make bench
+```
+
+## Public API
 
 ```python
 from splitgrid import (
@@ -129,14 +174,6 @@ from splitgrid import (
 
 `host_pack_data(..., force="auto"|"always"|"never")` chooses split-grid vs nested list. `host_pack_multi_data` is a thin `multi_data` wrapper over the same per-grid packing.
 
-## How WriterAgent will consume this
-
-In [WriterAgent](https://github.com/KeithCu/writeragent), replace `from plugin.scripting.payload_codec import host_pack_data, ...` with `from splitgrid import host_pack_data, ...`. The envelope tag stays `"split_grid"`, dtype `"float64"`, integer-keyed `strings`, and `column_kinds` `int`/`float`/`bool`. Pickle Protocol 5 framing stays in WriterAgent’s `ipc.py` — this package is the codec only.
-
-## Releasing
-
-Tags matching `v*` (for example `v0.1.0`) run `.github/workflows/publish.yml`: cibuildwheel + sdist, then trusted publishing to PyPI (GitHub environment `pypi`, no API token). The workflow file must already be on `main` before you push the tag.
-
 ## License
 
-GPL-3.0-or-later (same as WriterAgent).
+GPL-3.0-or-later.

splitgrid-0.1.2/README.md

@@ -0,0 +1,144 @@
+# SplitGrid
+
+[![PyPI](https://img.shields.io/pypi/v/splitgrid.svg)](https://pypi.org/project/splitgrid/)
+
+Asymmetric **split-grid** serialization for rectangular numeric and mixed-type grids.
+
+This package was **pulled out of [WriterAgent](https://github.com/KeithCu/writeragent)** (LibreOffice Writer/Calc/Draw AI extension). It is the host flatten + child unpack codec that lived in `plugin.scripting.payload_codec`, plus the optional Cython flatten accelerator from `native/writeragent_vec`.
+
+## Why this exists
+
+A spreadsheet host often cannot import NumPy: LibreOffice’s embedded Python is a different interpreter from the user’s venv, and loading the user’s C extensions into the host is an ABI footgun. Heavy compute therefore runs in a **child** process that *does* have NumPy. The range still has to cross that boundary.
+
+On the wire, a Calc-style range is a nested list: `list[list[float | int | str | None]]`. Standard pickle walks **one heap object per cell**. On a 20,000 × 5 grid that is ~12 ms for dump + load. Pack the same numbers into a contiguous `float64` buffer and Pickle Protocol 5 moves the bytes in ~0.015 ms; the child can `np.frombuffer` in ~0.002 ms.
+
+Almost all of the remaining time is **host flatten** — turning nested Python objects into that buffer without NumPy (~8.3 ms pure Python on that shape; ~3 ms with the Cython helper). SplitGrid is that flatten/unpack. Length-prefixed Pickle 5 framing stays in the application; this package is the codec only.
+
+Column-wise blobs and JSON + Base64 were tried first. Transposing columns on the host builds extra object graphs; Base64 and per-column reconstructs lose the `frombuffer` win. One row-major float64 buffer plus a sparse string map is faster and simpler.
+
+## How it works
+
+The path is **asymmetric**:
+
+- **Host pack** (stdlib only) flattens a 1D or 2D rectangular grid into a contiguous `float64` buffer plus a sparse **integer-keyed** `strings` map. Empty cells become `NaN`. Zip codes like `"02138"` stay strings — they are never parsed as floats.
+- **Child unpack** (NumPy) materializes a **pure-numeric** grid (`strings == {}`) with `np.frombuffer` (ndarray). Mixed grids use a vectorized object-masking path and return nested lists, restoring `None` for NaN holes.
+- **Host unpack** preserves `float('nan')` from the buffer (does **not** coerce holes to `None`). That is the locked egress policy: NaN becomes a spreadsheet error, not a silent blank.
+
+Grids with fewer than **100 cells** (`BINARY_MIN_CELLS`) stay nested Python lists. Force `"always"` / `"never"` overrides that threshold (used by A/B tests).
+
+Wire envelope (Pickle5-friendly dict):
+
+```python
+{
+    "__wa_payload__": "split_grid",
+    "dtype": "float64",
+    "column_kinds": ["int", "float"],  # per column: int / float / bool
+    "shape": [rows, cols],             # or [n] for 1D
+    "buffer": b"...",                  # row-major float64 bytes
+    "strings": {7: "banana"},          # integer keys, not str(idx)
+}
+```
+
+| Cell value | `buffer` (float64) | `strings` |
+|------------|--------------------|-----------|
+| `None` (empty cell) | `NaN` | — |
+| `int` / `float` | numeric value | — |
+| `bool` | `0.0` / `1.0` | — |
+| `str` (including `"02138"`) | `NaN` | text by flat index |
+
+There is **no datetime lane** on the float64 buffer. Python `datetime` objects stringify into `strings`. Do not add a `'date'` column kind.
+
+Jagged 2D grids raise `ValueError` (spreadsheet ranges are rectangular).
+
+## Numbers
+
+Median timings from an asymmetric bench (host = stdlib pack; child = deserialize + materialize).
+
+Ingress, **20,000 × 5** (100k cells):
+
+| Format | Pack | Dump | Load | Materialize | Total |
+|--------|------|------|------|-------------|-------|
+| JSON nested lists | 6.2 ms | 22.8 ms | 14.6 ms | 2.1 ms | 45.7 ms |
+| Pickle 5 nested lists | 6.1 ms | 1.4 ms | 2.9 ms | 2.0 ms | 12.4 ms |
+| **Pickle 5 + split-grid** | 8.3 ms | **0.013 ms** | **0.015 ms** | **0.002 ms** | **8.3 ms** |
+
+Child materialize of a **100 × 100** numeric grid: nested-list pickle then `np.array` ~0.6 ms; split-grid `frombuffer` ~**0.016 ms**. Below 100 cells the envelope is not worth it — that is why `BINARY_MIN_CELLS` exists. Host pack still dominates large ingress; Cython only speeds that loop.
+
+## Python vs Cython
+
+Host flatten is an optimized **pure-Python** loop:
+
+- identity type checks (`type(val) is float`, `val is None`)
+- bound-method capture (`buf_append = buf.append`)
+- `None` as NaN on the fast path
+- lazy column-state upgrades
+- rectangular validation before the hot loop
+
+The optional **Cython** module `splitgrid.pack` exposes `fast_flatten_grid_1d` / `fast_flatten_grid_2d`. It is loaded dynamically and **canary-tested** at import. If the extension is missing or fails the canary, the codec uses pure Python. Importing `splitgrid` never requires a compiler. Extension builds use release flags (`-O3 -DNDEBUG -g0` on Unix, `/O2 /DNDEBUG` on Windows).
+
+## Install
+
+```bash
+pip install splitgrid
+```
+
+PyPI wheels include the compiled Cython flatten accelerator. From a source checkout (Cython is built if a compiler is available):
+
+```bash
+pip install .
+pip install -e ".[test]"      # editable + pytest, hypothesis, deal, numpy
+pip install -e ".[numpy]"     # child unpack/pack
+pip install -e ".[verify]"    # crosshair-tool
+```
+
+Host pack stays NumPy-free. Child `child_unpack_*` / `child_pack_*` import NumPy locally.
+
+## Test
+
+```bash
+pytest                  # default: -m "not slow"
+pytest -m "not slow"    # same
+pytest -m slow          # CrossHair subprocess checks (optional)
+make verify             # deal contracts + hypothesis round-trips, no CrossHair
+```
+
+Default pytest **does not require** the compiled extension. When `splitgrid.pack` is built, `tests/test_cython_parity.py` compares Cython and Python flatten components on the same grids.
+
+## Bench
+
+Requires NumPy (and the optional Cython extension if you want the accelerated pack row). From a source checkout:
+
+```bash
+python scripts/bench_serialization.py --direction both
+python scripts/bench_serialization.py --child-only
+python scripts/bench_unpacking_opt.py
+python scripts/profile_pack.py
+python scripts/profile_nones.py
+python scripts/run_serialization_ab.py --all
+make bench
+```
+
+## Public API
+
+```python
+from splitgrid import (
+    BINARY_MIN_CELLS,
+    host_pack_split_grid,
+    host_unpack_split_grid,
+    child_unpack_split_grid,
+    child_pack_split_grid,
+    host_pack_data,
+    host_unpack_data,
+    child_unpack_data,
+    child_pack_result,
+    is_split_grid,
+    load_cython_accelerator,
+    get_cython_status_info,
+)
+```
+
+`host_pack_data(..., force="auto"|"always"|"never")` chooses split-grid vs nested list. `host_pack_multi_data` is a thin `multi_data` wrapper over the same per-grid packing.
+
+## License
+
+GPL-3.0-or-later.

{splitgrid-0.1.0 → splitgrid-0.1.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "splitgrid"
-version = "0.1.0"
+version = "0.1.2"
 description = "Asymmetric split-grid serialization: host stdlib flatten + optional Cython, child NumPy unpack"
 readme = "README.md"
 requires-python = ">=3.11"

{splitgrid-0.1.0 → splitgrid-0.1.2}/setup.py

@@ -10,9 +10,10 @@ system = platform.system()
 machine = platform.machine().lower()
 
 if system == "Windows":
-    extra_compile_args.append("/O2")
+    extra_compile_args.extend(["/O2", "/DNDEBUG"])
 else:
-    extra_compile_args.append("-O3")
+    # -g0 overrides manylinux/CPython default -g so wheels are not full of debug symbols.
+    extra_compile_args.extend(["-O3", "-DNDEBUG", "-g0"])
 
 # Only apply SPLITGRID_ARCH / WRITERAGENT_ARCH logic on Linux x86_64.
 # Generic x86-64 (not v3): flatten is memory-bound; SIMD floors buy ~1%.

{splitgrid-0.1.0 → splitgrid-0.1.2}/src/splitgrid/__init__.py

@@ -55,7 +55,7 @@ from splitgrid.codec import (
     wire_cell_count,
 )
 
-__version__ = "0.1.0"
+__version__ = "0.1.2"
 
 
 def __getattr__(name: str):

{splitgrid-0.1.0 → splitgrid-0.1.2}/src/splitgrid/codec.py

@@ -813,6 +813,10 @@ def _flatten_update_column_state(column_states: list[int], c: int, val: Any) ->
             column_states[c] = 2
     elif tname.startswith("float"):
         column_states[c] = 3
+    else:
+        # Decimal/Fraction/etc. already survived float(val). Default to float,
+        # matching Cython _update_column_state — not int (state 0).
+        column_states[c] = 3
 
 
 def _flatten_append_cell_slow(
@@ -1421,7 +1425,16 @@ def _child_unpack_single_data(wire: Any) -> Any:
         else:
             grid = list(unpacked)
         if is_numeric_grid(grid):
-            arr = np.array(grid, dtype=np.float64)
+            # is_numeric_coercible treats whitespace/"" as Calc blanks, but
+            # np.float64 cannot convert those strings (ValueError). Keep the list.
+            try:
+                arr = np.array(grid, dtype=np.float64)
+            except ValueError:
+                log.debug(
+                    "splitgrid child_unpack json_list as-is (non-floatable blanks) %s",
+                    describe_wire_value(unpacked),
+                )
+                return grid
             log.debug(
                 "splitgrid child_unpack json_list -> ndarray shape=%s",
                 arr.shape,

{splitgrid-0.1.0 → splitgrid-0.1.2}/src/splitgrid/pack.c

@@ -6,6 +6,8 @@
         "depends": [],
         "extra_compile_args": [
             "-O3",
+            "-DNDEBUG",
+            "-g0",
             "-march=x86-64"
         ],
         "name": "splitgrid.pack",

{splitgrid-0.1.0 → splitgrid-0.1.2/src/splitgrid.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: splitgrid
-Version: 0.1.0
+Version: 0.1.2
 Summary: Asymmetric split-grid serialization: host stdlib flatten + optional Cython, child NumPy unpack
 Author-email: Keith Cu <keithcu@gmail.com>
 License: GPL-3.0-or-later
@@ -35,23 +35,33 @@ Dynamic: license-file
 
 # SplitGrid
 
+[![PyPI](https://img.shields.io/pypi/v/splitgrid.svg)](https://pypi.org/project/splitgrid/)
+
 Asymmetric **split-grid** serialization for rectangular numeric and mixed-type grids.
 
-This package was **pulled out of [WriterAgent](https://github.com/KeithCu/writeragent)** (LibreOffice Writer/Calc/Draw AI extension). It is the same host flatten + child unpack codec that lived in `plugin.scripting.payload_codec`, plus the optional Cython flatten accelerator from `native/writeragent_vec`. WriterAgent can later depend on `splitgrid` without changing the wire dict.
+This package was **pulled out of [WriterAgent](https://github.com/KeithCu/writeragent)** (LibreOffice Writer/Calc/Draw AI extension). It is the host flatten + child unpack codec that lived in `plugin.scripting.payload_codec`, plus the optional Cython flatten accelerator from `native/writeragent_vec`.
+
+## Why this exists
+
+A spreadsheet host often cannot import NumPy: LibreOffice’s embedded Python is a different interpreter from the user’s venv, and loading the user’s C extensions into the host is an ABI footgun. Heavy compute therefore runs in a **child** process that *does* have NumPy. The range still has to cross that boundary.
+
+On the wire, a Calc-style range is a nested list: `list[list[float | int | str | None]]`. Standard pickle walks **one heap object per cell**. On a 20,000 × 5 grid that is ~12 ms for dump + load. Pack the same numbers into a contiguous `float64` buffer and Pickle Protocol 5 moves the bytes in ~0.015 ms; the child can `np.frombuffer` in ~0.002 ms.
+
+Almost all of the remaining time is **host flatten** — turning nested Python objects into that buffer without NumPy (~8.3 ms pure Python on that shape; ~3 ms with the Cython helper). SplitGrid is that flatten/unpack. Length-prefixed Pickle 5 framing stays in the application; this package is the codec only.
 
-Repo: [github.com/KeithCu/writeragent](https://github.com/KeithCu/writeragent)
+Column-wise blobs and JSON + Base64 were tried first. Transposing columns on the host builds extra object graphs; Base64 and per-column reconstructs lose the `frombuffer` win. One row-major float64 buffer plus a sparse string map is faster and simpler.
 
-## What split-grid is
+## How it works
 
-The compute path is **asymmetric by design**:
+The path is **asymmetric**:
 
-- **Host pack** (LibreOffice’s embedded Python, or any NumPy-free interpreter) flattens a 1D or 2D grid into a contiguous `float64` buffer plus a sparse **integer-keyed** `strings` map. Empty cells become `NaN`. Zip codes like `"02138"` stay strings — they are never parsed as floats.
-- **Child unpack** (venv with NumPy) materializes a **pure-numeric** grid (`strings == {}`) with `np.frombuffer` (ndarray). Mixed grids use a vectorized object-masking path and return nested lists, restoring `None` for NaN holes.
-- **Host unpack** preserves `float('nan')` from the buffer (does **not** coerce holes to `None`). That is the locked egress policy: NaN becomes a Calc error, not a silent blank.
+- **Host pack** (stdlib only) flattens a 1D or 2D rectangular grid into a contiguous `float64` buffer plus a sparse **integer-keyed** `strings` map. Empty cells become `NaN`. Zip codes like `"02138"` stay strings — they are never parsed as floats.
+- **Child unpack** (NumPy) materializes a **pure-numeric** grid (`strings == {}`) with `np.frombuffer` (ndarray). Mixed grids use a vectorized object-masking path and return nested lists, restoring `None` for NaN holes.
+- **Host unpack** preserves `float('nan')` from the buffer (does **not** coerce holes to `None`). That is the locked egress policy: NaN becomes a spreadsheet error, not a silent blank.
 
 Grids with fewer than **100 cells** (`BINARY_MIN_CELLS`) stay nested Python lists. Force `"always"` / `"never"` overrides that threshold (used by A/B tests).
 
-Wire envelope (Pickle5-friendly dict, same keys as WriterAgent):
+Wire envelope (Pickle5-friendly dict):
 
 ```python
 {
@@ -64,9 +74,30 @@ Wire envelope (Pickle5-friendly dict, same keys as WriterAgent):
 }
 ```
 
+| Cell value | `buffer` (float64) | `strings` |
+|------------|--------------------|-----------|
+| `None` (empty cell) | `NaN` | — |
+| `int` / `float` | numeric value | — |
+| `bool` | `0.0` / `1.0` | — |
+| `str` (including `"02138"`) | `NaN` | text by flat index |
+
 There is **no datetime lane** on the float64 buffer. Python `datetime` objects stringify into `strings`. Do not add a `'date'` column kind.
 
-Jagged 2D grids raise `ValueError` (Calc ranges are rectangular).
+Jagged 2D grids raise `ValueError` (spreadsheet ranges are rectangular).
+
+## Numbers
+
+Median timings from an asymmetric bench (host = stdlib pack; child = deserialize + materialize).
+
+Ingress, **20,000 × 5** (100k cells):
+
+| Format | Pack | Dump | Load | Materialize | Total |
+|--------|------|------|------|-------------|-------|
+| JSON nested lists | 6.2 ms | 22.8 ms | 14.6 ms | 2.1 ms | 45.7 ms |
+| Pickle 5 nested lists | 6.1 ms | 1.4 ms | 2.9 ms | 2.0 ms | 12.4 ms |
+| **Pickle 5 + split-grid** | 8.3 ms | **0.013 ms** | **0.015 ms** | **0.002 ms** | **8.3 ms** |
+
+Child materialize of a **100 × 100** numeric grid: nested-list pickle then `np.array` ~0.6 ms; split-grid `frombuffer` ~**0.016 ms**. Below 100 cells the envelope is not worth it — that is why `BINARY_MIN_CELLS` exists. Host pack still dominates large ingress; Cython only speeds that loop.
 
 ## Python vs Cython
 
@@ -78,7 +109,7 @@ Host flatten is an optimized **pure-Python** loop:
 - lazy column-state upgrades
 - rectangular validation before the hot loop
 
-The optional **Cython** module `splitgrid.pack` exposes `fast_flatten_grid_1d` / `fast_flatten_grid_2d`. It is loaded dynamically and **canary-tested** at import. If the extension is missing or fails the canary, the codec uses pure Python. Importing `splitgrid` never requires a compiler.
+The optional **Cython** module `splitgrid.pack` exposes `fast_flatten_grid_1d` / `fast_flatten_grid_2d`. It is loaded dynamically and **canary-tested** at import. If the extension is missing or fails the canary, the codec uses pure Python. Importing `splitgrid` never requires a compiler. Extension builds use release flags (`-O3 -DNDEBUG -g0` on Unix, `/O2 /DNDEBUG` on Windows).
 
 ## Install
 
@@ -108,7 +139,21 @@ make verify             # deal contracts + hypothesis round-trips, no CrossHair
 
 Default pytest **does not require** the compiled extension. When `splitgrid.pack` is built, `tests/test_cython_parity.py` compares Cython and Python flatten components on the same grids.
 
-## Public API (WriterAgent-compatible)
+## Bench
+
+Requires NumPy (and the optional Cython extension if you want the accelerated pack row). From a source checkout:
+
+```bash
+python scripts/bench_serialization.py --direction both
+python scripts/bench_serialization.py --child-only
+python scripts/bench_unpacking_opt.py
+python scripts/profile_pack.py
+python scripts/profile_nones.py
+python scripts/run_serialization_ab.py --all
+make bench
+```
+
+## Public API
 
 ```python
 from splitgrid import (
@@ -129,14 +174,6 @@ from splitgrid import (
 
 `host_pack_data(..., force="auto"|"always"|"never")` chooses split-grid vs nested list. `host_pack_multi_data` is a thin `multi_data` wrapper over the same per-grid packing.
 
-## How WriterAgent will consume this
-
-In [WriterAgent](https://github.com/KeithCu/writeragent), replace `from plugin.scripting.payload_codec import host_pack_data, ...` with `from splitgrid import host_pack_data, ...`. The envelope tag stays `"split_grid"`, dtype `"float64"`, integer-keyed `strings`, and `column_kinds` `int`/`float`/`bool`. Pickle Protocol 5 framing stays in WriterAgent’s `ipc.py` — this package is the codec only.
-
-## Releasing
-
-Tags matching `v*` (for example `v0.1.0`) run `.github/workflows/publish.yml`: cibuildwheel + sdist, then trusted publishing to PyPI (GitHub environment `pypi`, no API token). The workflow file must already be on `main` before you push the tag.
-
 ## License
 
-GPL-3.0-or-later (same as WriterAgent).
+GPL-3.0-or-later.

{splitgrid-0.1.0 → splitgrid-0.1.2}/tests/test_payload_codec.py

@@ -15,6 +15,7 @@ from __future__ import annotations
 
 import ast
 import math
+from decimal import Decimal
 from pathlib import Path
 from unittest.mock import patch
 
@@ -41,6 +42,7 @@ from splitgrid.codec import (
     should_use_binary_envelope,
     wire_cell_count,
 )
+from tests.serialization_ab_support import cython_accelerator_context
 from tests.payload_codec_test_support import (
     MIXED_LABEL_GRID,
     MIXED_WITH_ZIP,
@@ -512,6 +514,43 @@ def test_mixed_grid_preserves_non_numeric_string() -> None:
     assert out[0][1] == "hello"
 
 
+def test_whitespace_only_cell_does_not_crash_child_unpack() -> None:
+    """Pasted '   ' is numeric-coercible for Calc but np.float64 cannot convert it."""
+    pytest.importorskip("numpy")
+    grid = [[1.0, "   ", 3.0, 4.0], [5.0, 6.0, 7.0, 8.0], [9.0, 10.0, 11.0, 12.0]]
+    out = child_unpack_data(host_pack_data(grid, force="always"))
+    assert isinstance(out, list)
+    assert out[0][1] == "   "
+
+
+def test_empty_string_mixed_split_grid_does_not_crash_child_unpack() -> None:
+    """Bare '' on mixed split_grid must not raise (Calc usually maps '' to None first)."""
+    pytest.importorskip("numpy")
+    grid = [[1.0, "", 3.0, 4.0], [5.0, 6.0, 7.0, 8.0], [9.0, 10.0, 11.0, 12.0]]
+    out = child_unpack_data(host_pack_data(grid, force="always"))
+    assert isinstance(out, list)
+    assert out[0][1] == ""
+
+
+def test_mixed_grid_real_nan_becomes_none_on_child() -> None:
+    """Documented: mixed-grid ingress has no blank-vs-NaN wire bit."""
+    pytest.importorskip("numpy")
+    grid = [[1.0, float("nan")], ["label", 4.0]]
+    out = child_unpack_data(host_pack_data(grid, force="always"))
+    assert out[0][1] is None
+    assert out[1][0] == "label"
+
+
+def test_decimal_split_grid_stays_float_not_truncated_int() -> None:
+    """stdlib flatten must label Decimal columns float (Cython already did)."""
+    pytest.importorskip("numpy")
+    grid = [[Decimal("1.5"), Decimal("2.25")], [Decimal("3.0"), Decimal("4.75")]]
+    with cython_accelerator_context(enabled=False):
+        out = child_unpack_data(host_pack_data(grid, force="always"))
+    assert out[0][0] == pytest.approx(1.5)
+    assert out[0][1] == pytest.approx(2.25)
+
+
 def test_bool_cells_round_trip_in_numeric_grid() -> None:
     """Calc booleans in an all-numeric grid become 0.0/1.0 in child ndarray (float64 lane)."""
     np = pytest.importorskip("numpy")

splitgrid-0.1.0/README.md

@@ -1,107 +0,0 @@
-# SplitGrid
-
-Asymmetric **split-grid** serialization for rectangular numeric and mixed-type grids.
-
-This package was **pulled out of [WriterAgent](https://github.com/KeithCu/writeragent)** (LibreOffice Writer/Calc/Draw AI extension). It is the same host flatten + child unpack codec that lived in `plugin.scripting.payload_codec`, plus the optional Cython flatten accelerator from `native/writeragent_vec`. WriterAgent can later depend on `splitgrid` without changing the wire dict.
-
-Repo: [github.com/KeithCu/writeragent](https://github.com/KeithCu/writeragent)
-
-## What split-grid is
-
-The compute path is **asymmetric by design**:
-
-- **Host pack** (LibreOffice’s embedded Python, or any NumPy-free interpreter) flattens a 1D or 2D grid into a contiguous `float64` buffer plus a sparse **integer-keyed** `strings` map. Empty cells become `NaN`. Zip codes like `"02138"` stay strings — they are never parsed as floats.
-- **Child unpack** (venv with NumPy) materializes a **pure-numeric** grid (`strings == {}`) with `np.frombuffer` (ndarray). Mixed grids use a vectorized object-masking path and return nested lists, restoring `None` for NaN holes.
-- **Host unpack** preserves `float('nan')` from the buffer (does **not** coerce holes to `None`). That is the locked egress policy: NaN becomes a Calc error, not a silent blank.
-
-Grids with fewer than **100 cells** (`BINARY_MIN_CELLS`) stay nested Python lists. Force `"always"` / `"never"` overrides that threshold (used by A/B tests).
-
-Wire envelope (Pickle5-friendly dict, same keys as WriterAgent):
-
-```python
-{
-    "__wa_payload__": "split_grid",
-    "dtype": "float64",
-    "column_kinds": ["int", "float"],  # per column: int / float / bool
-    "shape": [rows, cols],             # or [n] for 1D
-    "buffer": b"...",                  # row-major float64 bytes
-    "strings": {7: "banana"},          # integer keys, not str(idx)
-}
-```
-
-There is **no datetime lane** on the float64 buffer. Python `datetime` objects stringify into `strings`. Do not add a `'date'` column kind.
-
-Jagged 2D grids raise `ValueError` (Calc ranges are rectangular).
-
-## Python vs Cython
-
-Host flatten is an optimized **pure-Python** loop:
-
-- identity type checks (`type(val) is float`, `val is None`)
-- bound-method capture (`buf_append = buf.append`)
-- `None` as NaN on the fast path
-- lazy column-state upgrades
-- rectangular validation before the hot loop
-
-The optional **Cython** module `splitgrid.pack` exposes `fast_flatten_grid_1d` / `fast_flatten_grid_2d`. It is loaded dynamically and **canary-tested** at import. If the extension is missing or fails the canary, the codec uses pure Python. Importing `splitgrid` never requires a compiler.
-
-## Install
-
-```bash
-pip install splitgrid
-```
-
-PyPI wheels include the compiled Cython flatten accelerator. From a source checkout (Cython is built if a compiler is available):
-
-```bash
-pip install .
-pip install -e ".[test]"      # editable + pytest, hypothesis, deal, numpy
-pip install -e ".[numpy]"     # child unpack/pack
-pip install -e ".[verify]"    # crosshair-tool
-```
-
-Host pack stays NumPy-free. Child `child_unpack_*` / `child_pack_*` import NumPy locally.
-
-## Test
-
-```bash
-pytest                  # default: -m "not slow"
-pytest -m "not slow"    # same
-pytest -m slow          # CrossHair subprocess checks (optional)
-make verify             # deal contracts + hypothesis round-trips, no CrossHair
-```
-
-Default pytest **does not require** the compiled extension. When `splitgrid.pack` is built, `tests/test_cython_parity.py` compares Cython and Python flatten components on the same grids.
-
-## Public API (WriterAgent-compatible)
-
-```python
-from splitgrid import (
-    BINARY_MIN_CELLS,
-    host_pack_split_grid,
-    host_unpack_split_grid,
-    child_unpack_split_grid,
-    child_pack_split_grid,
-    host_pack_data,
-    host_unpack_data,
-    child_unpack_data,
-    child_pack_result,
-    is_split_grid,
-    load_cython_accelerator,
-    get_cython_status_info,
-)
-```
-
-`host_pack_data(..., force="auto"|"always"|"never")` chooses split-grid vs nested list. `host_pack_multi_data` is a thin `multi_data` wrapper over the same per-grid packing.
-
-## How WriterAgent will consume this
-
-In [WriterAgent](https://github.com/KeithCu/writeragent), replace `from plugin.scripting.payload_codec import host_pack_data, ...` with `from splitgrid import host_pack_data, ...`. The envelope tag stays `"split_grid"`, dtype `"float64"`, integer-keyed `strings`, and `column_kinds` `int`/`float`/`bool`. Pickle Protocol 5 framing stays in WriterAgent’s `ipc.py` — this package is the codec only.
-
-## Releasing
-
-Tags matching `v*` (for example `v0.1.0`) run `.github/workflows/publish.yml`: cibuildwheel + sdist, then trusted publishing to PyPI (GitHub environment `pypi`, no API token). The workflow file must already be on `main` before you push the tag.
-
-## License
-
-GPL-3.0-or-later (same as WriterAgent).