fastconstmap 0.9.0__tar.gz → 0.11.0__tar.gz

{fastconstmap-0.9.0/fastconstmap.egg-info → fastconstmap-0.11.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: fastconstmap
-Version: 0.9.0
+Version: 0.11.0
 Summary: Fast, immutable, compact map from strings to uint64 (binary fuse filter).
 Author-email: Daniel Lemire <daniel@lemire.me>
 License: Apache-2.0
@@ -8,6 +8,7 @@ Project-URL: Homepage, https://github.com/lemire/fastconstmap
 Project-URL: Source, https://github.com/lemire/fastconstmap
 Classifier: License :: OSI Approved :: Apache Software License
 Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: Free Threading :: 3 - Stable
 Classifier: Programming Language :: C
 Classifier: Operating System :: OS Independent
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
@@ -29,7 +30,8 @@ Given a `dict[str, int]` at build time, you get back a lookup object that:
 
 - uses **~9 bytes per key** (or ~18 with missing-key detection),
 - answers a lookup in **one xxhash call plus three array reads**,
-- is **immutable** and **serializable** to bytes / a file,
+- is **immutable** and **serializable** to bytes / a file, in a format the
+  Go and Rust implementations read too,
 - exposes both a single-key API (`m[key]`) and a **batched** API
   (`m.get_many([...])`, `m.get_many_into([...], buf)`) that amortises Python-C
   call overhead and overlaps the memory accesses of a whole block of keys.
@@ -50,7 +52,7 @@ dependencies.
 ## Usage
 
 ```python
-from fastconstmap import ConstMap, VerifiedConstMap
+from fastconstmap import ConstMap, VerifiedConstMap, PairedVerifiedConstMap
 
 d = {"apple": 100, "banana": 200, "cherry": 300}
 
@@ -69,13 +71,19 @@ vm.get("grape", -1)         # -> -1
 vm["grape"]                 # raises KeyError
 vm.get_many(["banana", "grape"], default=-1)  # -> [200, -1]
 
+# Variant 3: same as VerifiedConstMap, laid out for fewer cache misses when
+# the keys you look up are usually present.
+pm = PairedVerifiedConstMap(d)
+pm["apple"]                 # -> 100
+pm.get("grape")             # -> None
+
 # Batches can also be written straight into a buffer you own — no list of
 # Python ints is built.
 import array
 out = array.array("Q", [0]) * 3
 m.get_many_into(["apple", "banana", "cherry"], out)   # -> 3; out == [100, 200, 300]
 
-# Either kind can be saved and loaded.
+# Any kind can be saved and loaded.
 m.save("mymap.cmap")
 m2 = ConstMap.load("mymap.cmap")
 
@@ -84,18 +92,48 @@ blob = m.to_bytes()
 m3 = ConstMap.from_bytes(blob)
 ```
 
-### Choosing between `ConstMap` and `VerifiedConstMap`
+### Choosing between `ConstMap`, `VerifiedConstMap` and `PairedVerifiedConstMap`
 
-| | `ConstMap` | `VerifiedConstMap` |
-|--|--|--|
-| Bytes per key | ~9 | ~18 |
-| Lookup of present key | value | value |
-| Lookup of missing key | **undefined garbage** | `KeyError` / `default` / `None` |
-| Best if | you always look up known keys | you need dict-like semantics |
+| | `ConstMap` | `VerifiedConstMap` | `PairedVerifiedConstMap` |
+|--|--|--|--|
+| Bytes per key | ~9 | ~18 | ~18 |
+| Lookup of present key | value | value | value |
+| Lookup of missing key | **undefined garbage** | `KeyError` / `default` / `None` | same |
+| Cache lines touched per lookup | 3 | 6 (3 on a miss) | 3 |
+| Best if | you always look up known keys | you need dict-like semantics and most lookups miss | you need dict-like semantics and most lookups hit |
 
-The false-positive rate of `VerifiedConstMap` (a missing key wrongly
+The false-positive rate of the verified maps (a missing key wrongly
 reported as present) is roughly 2⁻⁶⁴, which is negligible in practice.
 
+`VerifiedConstMap` keeps its values and its 64-bit check words in two
+separate arrays. A lookup reads three check words, and if they match, three
+value words: six cache lines, in two different arrays. `PairedVerifiedConstMap`
+is the same map with each value stored next to its check word, so the three
+reads bring in both at once (three cache lines), and on x86-64 and AArch64
+each slot is loaded and XORed as one 128-bit word (SSE2 / NEON).
+
+The trade-off is what happens on a **miss**. With the split layout, a
+one-at-a-time lookup of an absent key reads only the check array, which is
+half the size of the whole map and so far more likely to be sitting in cache.
+The paired layout always pulls in the value alongside the check, so its
+working set for misses is twice as large. In batched lookups
+(`get_many`, `get_many_into`) the split layout gathers both arrays
+unconditionally, so there the paired layout wins whether keys hit or miss.
+
+Measured at the C level (`fcm_*_lookup`, one key at a time, random order
+over the whole key set, best of five), ns per lookup, *hit* = key present,
+*miss* = key absent:
+
+| | keys | `VerifiedConstMap` hit | `PairedVerifiedConstMap` hit | `VerifiedConstMap` miss | `PairedVerifiedConstMap` miss |
+|--|--|--|--|--|--|
+| Apple M4 Max | 1M | 9.9 | 9.5 | **6.4** | 8.8 |
+| | 10M | 20.9 | **17.1** | **15.5** | 17.1 |
+| Xeon Gold 6548N | 1M | 17.3 | **12.8** | 13.8 | 13.0 |
+| | 10M | 44.4 | **37.3** | **31.8** | 37.4 |
+
+The two serialised formats are the same size but not interchangeable:
+loading one class's bytes with the other raises `ValueError`.
+
 ## Batched lookups
 
 `get_many` takes an iterable of keys and returns a list of values, in the same
@@ -140,14 +178,25 @@ dominates instead.
 
 | | | loop over lookup | `lookup_many` | |
 |---|---|---|---|---|
-| **Apple M4 Max** | `ConstMap` cold | 5.5 | **3.8** | 30% |
-| | `ConstMap` hot | 4.2 | **3.1** | 27% |
-| | `VerifiedConstMap` cold | 9.6 | **7.8** | 19% |
-| | `VerifiedConstMap` hot | 5.4 | **5.2** | 4% |
-| **Xeon Gold 6548N** | `ConstMap` cold | 12.6 | **10.4** | 18% |
-| | `ConstMap` hot | 6.3 | **5.0** | 21% |
-| | `VerifiedConstMap` cold | 15.9 | 16.6 | −5% |
-| | `VerifiedConstMap` hot | 7.9 | **6.7** | 15% |
+| **Apple M4 Max** | `ConstMap` cold | 7.1 | **5.5** | 23% |
+| | `ConstMap` hot | 6.1 | **4.6** | 24% |
+| | `VerifiedConstMap` cold | 11.9 | **8.0** | 33% |
+| | `VerifiedConstMap` hot | 7.4 | **5.8** | 21% |
+| | `PairedVerifiedConstMap` cold | 10.8 | **7.3** | 32% |
+| | `PairedVerifiedConstMap` hot | 6.6 | **4.8** | 27% |
+| **Xeon Gold 6548N** | `ConstMap` cold | 19.1 | **13.6** | 29% |
+| | `ConstMap` hot | 10.7 | **8.5** | 21% |
+| | `VerifiedConstMap` cold | 21.8 | 21.9 | ~0% |
+| | `VerifiedConstMap` hot | 12.6 | **10.5** | 17% |
+| | `PairedVerifiedConstMap` cold | 21.0 | **15.0** | 29% |
+| | `PairedVerifiedConstMap` hot | 12.5 | **9.9** | 21% |
+
+These are with the default key hash, XXH64, which is what makes the files
+interchangeable with the Go and Rust implementations (see
+[Interoperability](#interoperability-with-the-go-and-rust-implementations)).
+XXH3, the hash of 0.9 and earlier, is about 1.3 ns faster per 20-byte key on
+the M4 Max and 2 ns on the Xeon; `ConstMap(d, hash="xxh3")` keeps it, at the
+price of files only fastconstmap can read.
 
 The one case where batching does not pay is `VerifiedConstMap` in the cold
 regime on the Xeon. That map probes two arrays per lookup, so a fresh batch is
@@ -155,7 +204,9 @@ bound by a memory latency the overlapping cannot hide. The Go original reports
 the same result on the same processor.
 
 Through the Python API, ns/key on the same batches — 0.8.0 is the previous
-per-key loop, 0.9.0 the batched path:
+per-key loop, 0.9.0 the batched path (both measured with XXH3; with the
+default XXH64 of 0.10, add roughly 1-4 ns/key to the batched columns, and
+1-5 ns to a single `m[k]` that costs 50-120 ns):
 
 | | | 0.8.0 | 0.9.0 | `get_many_into` |
 |---|---|---|---|---|
@@ -247,7 +298,7 @@ Notes and constraints:
   use. Drop the map (`del cm`) before calling `shm.close()`.
 - The map is **immutable** — the intended pattern is *write once in the
   producer, then only read in every process*. Concurrent readers need no
-  locking.
+  locking (see [Threads](#threads-and-free-threaded-python)).
 - `from_buffer` verifies the magic bytes and the FNV-1a checksum, so a
   truncated or corrupt block raises `ValueError` rather than returning
   garbage.
@@ -256,12 +307,82 @@ Notes and constraints:
   all satisfy the alignment requirement; an offset slice of a buffer may
   not, in which case `from_buffer` raises `ValueError` — use
   `from_bytes()` (which copies) instead.
-- `VerifiedConstMap` supports the same three methods.
+- `VerifiedConstMap` and `PairedVerifiedConstMap` support the same three
+  methods.
 
 `from_bytes()` remains available when you *want* an owned copy (or need to
 load on a big-endian host): it copies the data and the resulting map owns
 its memory independently of the source buffer.
 
+## Threads and free-threaded Python
+
+The maps are immutable, so any number of threads can look up keys in the
+same map at once, without locking. The extension supports free-threaded
+Python (3.13t, 3.14t): it does not re-enable the GIL on import, and lookups
+from several threads then run in parallel.
+
+```python
+from concurrent.futures import ThreadPoolExecutor
+
+m = ConstMap(d)
+with ThreadPoolExecutor(8) as pool:
+    results = list(pool.map(m.get_many, batches))
+```
+
+Building a map from a dict that another thread is modifying is safe (on a
+free-threaded build the map is built from a snapshot of the dict), and so is
+a batched lookup over a list that another thread is modifying. `__init__` can only be called once on a
+map; calling it again raises `TypeError` rather than replacing the data
+under other readers.
+
+## Interoperability with the Go and Rust implementations
+
+A map saved by any of [`constmap`][constmap] (Go), [`rsconstmap`][rsconstmap]
+(Rust) and fastconstmap loads in the other two, on a little-endian host (which
+is every mainstream one: x86-64, ARM64, RISC-V, Apple silicon). The three share
+the key hash (XXH64), the mixing, the seed sequence and the layout, so for the
+same input Go and Rust write byte-identical files, and fastconstmap writes the
+same bytes apart from one header word:
+
+| class | magic | notes |
+|---|---|---|
+| `ConstMap` | `CMAP0003` | Go and Rust write `CMAP0001`, the same format minus a 4-byte key count; all three read both |
+| `VerifiedConstMap` | `VMAP0001` | identical; the header word Go and Rust leave zero holds the key count here |
+| `PairedVerifiedConstMap` | `PMAP0001` | likewise |
+
+```python
+m = ConstMap.load("built-by-go.cmap")     # from constmap.SaveToFile / save_to_file
+vm = VerifiedConstMap.load("built-by-rust.vmap")
+```
+
+Two things to know about a map that came from Go or Rust:
+
+- **`len(m)` is 0**, because those writers do not record the key count. Every
+  lookup is unaffected.
+- **A Go or Rust `ConstMap` file (`CMAP0001`) cannot be opened with
+  `from_buffer`**: its 28-byte header leaves the array 4 bytes off an 8-byte
+  boundary in any aligned buffer, so `from_buffer` raises `ValueError`. `load`
+  and `from_bytes`, which copy, work as usual. The other two formats have a
+  32-byte header and view fine.
+
+**Files from fastconstmap 0.9 and earlier** (`CMAP0002` / `VCMP0002`) still
+load and answer correctly: they were built with XXH3, and a map keeps the hash
+its file was built with (`m.hash` tells you which). Re-saving such a map keeps
+its legacy magic, since its table only answers to XXH3; Go and Rust name these
+files in their error message. To get a file the other implementations read,
+rebuild the map from its keys.
+
+**Keeping XXH3.** XXH3 is faster than XXH64 on short keys, by about 1-2 ns per
+lookup at the C level. If you do not need the shared format, build with
+`ConstMap(d, hash="xxh3")` (likewise for the other two classes); such a map
+serializes with the legacy magic, readable by fastconstmap only.
+
+The interoperability is tested: `tests/interop/` holds files written by the Go
+package from a fixed input, which the test suite loads and checks, and the Go
+and Rust repositories hold files written by this package.
+
+[rsconstmap]: https://github.com/lemire/rsconstmap
+
 ## Benchmark
 
 On an Apple M4 Max, with 1,000,000 string keys (`key-{i}-{hex}` shaped
@@ -315,7 +436,7 @@ python benchmarks/benchmark.py 1000000
 
 Given *n* (key, value) pairs the algorithm:
 
-1. Hashes each key with xxhash3 to a 64-bit value.
+1. Hashes each key with XXH64 (or XXH3 if asked) to a 64-bit value.
 2. Maps each hashed key to three positions `h0, h1, h2` in an array of
    size ~1.125·*n*, using overlapping segments.
 3. Finds, via [peeling][bff], an ordering in which each key has an

{fastconstmap-0.9.0 → fastconstmap-0.11.0}/README.md

@@ -11,7 +11,8 @@ Given a `dict[str, int]` at build time, you get back a lookup object that:
 
 - uses **~9 bytes per key** (or ~18 with missing-key detection),
 - answers a lookup in **one xxhash call plus three array reads**,
-- is **immutable** and **serializable** to bytes / a file,
+- is **immutable** and **serializable** to bytes / a file, in a format the
+  Go and Rust implementations read too,
 - exposes both a single-key API (`m[key]`) and a **batched** API
   (`m.get_many([...])`, `m.get_many_into([...], buf)`) that amortises Python-C
   call overhead and overlaps the memory accesses of a whole block of keys.
@@ -32,7 +33,7 @@ dependencies.
 ## Usage
 
 ```python
-from fastconstmap import ConstMap, VerifiedConstMap
+from fastconstmap import ConstMap, VerifiedConstMap, PairedVerifiedConstMap
 
 d = {"apple": 100, "banana": 200, "cherry": 300}
 
@@ -51,13 +52,19 @@ vm.get("grape", -1)         # -> -1
 vm["grape"]                 # raises KeyError
 vm.get_many(["banana", "grape"], default=-1)  # -> [200, -1]
 
+# Variant 3: same as VerifiedConstMap, laid out for fewer cache misses when
+# the keys you look up are usually present.
+pm = PairedVerifiedConstMap(d)
+pm["apple"]                 # -> 100
+pm.get("grape")             # -> None
+
 # Batches can also be written straight into a buffer you own — no list of
 # Python ints is built.
 import array
 out = array.array("Q", [0]) * 3
 m.get_many_into(["apple", "banana", "cherry"], out)   # -> 3; out == [100, 200, 300]
 
-# Either kind can be saved and loaded.
+# Any kind can be saved and loaded.
 m.save("mymap.cmap")
 m2 = ConstMap.load("mymap.cmap")
 
@@ -66,18 +73,48 @@ blob = m.to_bytes()
 m3 = ConstMap.from_bytes(blob)
 ```
 
-### Choosing between `ConstMap` and `VerifiedConstMap`
+### Choosing between `ConstMap`, `VerifiedConstMap` and `PairedVerifiedConstMap`
 
-| | `ConstMap` | `VerifiedConstMap` |
-|--|--|--|
-| Bytes per key | ~9 | ~18 |
-| Lookup of present key | value | value |
-| Lookup of missing key | **undefined garbage** | `KeyError` / `default` / `None` |
-| Best if | you always look up known keys | you need dict-like semantics |
+| | `ConstMap` | `VerifiedConstMap` | `PairedVerifiedConstMap` |
+|--|--|--|--|
+| Bytes per key | ~9 | ~18 | ~18 |
+| Lookup of present key | value | value | value |
+| Lookup of missing key | **undefined garbage** | `KeyError` / `default` / `None` | same |
+| Cache lines touched per lookup | 3 | 6 (3 on a miss) | 3 |
+| Best if | you always look up known keys | you need dict-like semantics and most lookups miss | you need dict-like semantics and most lookups hit |
 
-The false-positive rate of `VerifiedConstMap` (a missing key wrongly
+The false-positive rate of the verified maps (a missing key wrongly
 reported as present) is roughly 2⁻⁶⁴, which is negligible in practice.
 
+`VerifiedConstMap` keeps its values and its 64-bit check words in two
+separate arrays. A lookup reads three check words, and if they match, three
+value words: six cache lines, in two different arrays. `PairedVerifiedConstMap`
+is the same map with each value stored next to its check word, so the three
+reads bring in both at once (three cache lines), and on x86-64 and AArch64
+each slot is loaded and XORed as one 128-bit word (SSE2 / NEON).
+
+The trade-off is what happens on a **miss**. With the split layout, a
+one-at-a-time lookup of an absent key reads only the check array, which is
+half the size of the whole map and so far more likely to be sitting in cache.
+The paired layout always pulls in the value alongside the check, so its
+working set for misses is twice as large. In batched lookups
+(`get_many`, `get_many_into`) the split layout gathers both arrays
+unconditionally, so there the paired layout wins whether keys hit or miss.
+
+Measured at the C level (`fcm_*_lookup`, one key at a time, random order
+over the whole key set, best of five), ns per lookup, *hit* = key present,
+*miss* = key absent:
+
+| | keys | `VerifiedConstMap` hit | `PairedVerifiedConstMap` hit | `VerifiedConstMap` miss | `PairedVerifiedConstMap` miss |
+|--|--|--|--|--|--|
+| Apple M4 Max | 1M | 9.9 | 9.5 | **6.4** | 8.8 |
+| | 10M | 20.9 | **17.1** | **15.5** | 17.1 |
+| Xeon Gold 6548N | 1M | 17.3 | **12.8** | 13.8 | 13.0 |
+| | 10M | 44.4 | **37.3** | **31.8** | 37.4 |
+
+The two serialised formats are the same size but not interchangeable:
+loading one class's bytes with the other raises `ValueError`.
+
 ## Batched lookups
 
 `get_many` takes an iterable of keys and returns a list of values, in the same
@@ -122,14 +159,25 @@ dominates instead.
 
 | | | loop over lookup | `lookup_many` | |
 |---|---|---|---|---|
-| **Apple M4 Max** | `ConstMap` cold | 5.5 | **3.8** | 30% |
-| | `ConstMap` hot | 4.2 | **3.1** | 27% |
-| | `VerifiedConstMap` cold | 9.6 | **7.8** | 19% |
-| | `VerifiedConstMap` hot | 5.4 | **5.2** | 4% |
-| **Xeon Gold 6548N** | `ConstMap` cold | 12.6 | **10.4** | 18% |
-| | `ConstMap` hot | 6.3 | **5.0** | 21% |
-| | `VerifiedConstMap` cold | 15.9 | 16.6 | −5% |
-| | `VerifiedConstMap` hot | 7.9 | **6.7** | 15% |
+| **Apple M4 Max** | `ConstMap` cold | 7.1 | **5.5** | 23% |
+| | `ConstMap` hot | 6.1 | **4.6** | 24% |
+| | `VerifiedConstMap` cold | 11.9 | **8.0** | 33% |
+| | `VerifiedConstMap` hot | 7.4 | **5.8** | 21% |
+| | `PairedVerifiedConstMap` cold | 10.8 | **7.3** | 32% |
+| | `PairedVerifiedConstMap` hot | 6.6 | **4.8** | 27% |
+| **Xeon Gold 6548N** | `ConstMap` cold | 19.1 | **13.6** | 29% |
+| | `ConstMap` hot | 10.7 | **8.5** | 21% |
+| | `VerifiedConstMap` cold | 21.8 | 21.9 | ~0% |
+| | `VerifiedConstMap` hot | 12.6 | **10.5** | 17% |
+| | `PairedVerifiedConstMap` cold | 21.0 | **15.0** | 29% |
+| | `PairedVerifiedConstMap` hot | 12.5 | **9.9** | 21% |
+
+These are with the default key hash, XXH64, which is what makes the files
+interchangeable with the Go and Rust implementations (see
+[Interoperability](#interoperability-with-the-go-and-rust-implementations)).
+XXH3, the hash of 0.9 and earlier, is about 1.3 ns faster per 20-byte key on
+the M4 Max and 2 ns on the Xeon; `ConstMap(d, hash="xxh3")` keeps it, at the
+price of files only fastconstmap can read.
 
 The one case where batching does not pay is `VerifiedConstMap` in the cold
 regime on the Xeon. That map probes two arrays per lookup, so a fresh batch is
@@ -137,7 +185,9 @@ bound by a memory latency the overlapping cannot hide. The Go original reports
 the same result on the same processor.
 
 Through the Python API, ns/key on the same batches — 0.8.0 is the previous
-per-key loop, 0.9.0 the batched path:
+per-key loop, 0.9.0 the batched path (both measured with XXH3; with the
+default XXH64 of 0.10, add roughly 1-4 ns/key to the batched columns, and
+1-5 ns to a single `m[k]` that costs 50-120 ns):
 
 | | | 0.8.0 | 0.9.0 | `get_many_into` |
 |---|---|---|---|---|
@@ -229,7 +279,7 @@ Notes and constraints:
   use. Drop the map (`del cm`) before calling `shm.close()`.
 - The map is **immutable** — the intended pattern is *write once in the
   producer, then only read in every process*. Concurrent readers need no
-  locking.
+  locking (see [Threads](#threads-and-free-threaded-python)).
 - `from_buffer` verifies the magic bytes and the FNV-1a checksum, so a
   truncated or corrupt block raises `ValueError` rather than returning
   garbage.
@@ -238,12 +288,82 @@ Notes and constraints:
   all satisfy the alignment requirement; an offset slice of a buffer may
   not, in which case `from_buffer` raises `ValueError` — use
   `from_bytes()` (which copies) instead.
-- `VerifiedConstMap` supports the same three methods.
+- `VerifiedConstMap` and `PairedVerifiedConstMap` support the same three
+  methods.
 
 `from_bytes()` remains available when you *want* an owned copy (or need to
 load on a big-endian host): it copies the data and the resulting map owns
 its memory independently of the source buffer.
 
+## Threads and free-threaded Python
+
+The maps are immutable, so any number of threads can look up keys in the
+same map at once, without locking. The extension supports free-threaded
+Python (3.13t, 3.14t): it does not re-enable the GIL on import, and lookups
+from several threads then run in parallel.
+
+```python
+from concurrent.futures import ThreadPoolExecutor
+
+m = ConstMap(d)
+with ThreadPoolExecutor(8) as pool:
+    results = list(pool.map(m.get_many, batches))
+```
+
+Building a map from a dict that another thread is modifying is safe (on a
+free-threaded build the map is built from a snapshot of the dict), and so is
+a batched lookup over a list that another thread is modifying. `__init__` can only be called once on a
+map; calling it again raises `TypeError` rather than replacing the data
+under other readers.
+
+## Interoperability with the Go and Rust implementations
+
+A map saved by any of [`constmap`][constmap] (Go), [`rsconstmap`][rsconstmap]
+(Rust) and fastconstmap loads in the other two, on a little-endian host (which
+is every mainstream one: x86-64, ARM64, RISC-V, Apple silicon). The three share
+the key hash (XXH64), the mixing, the seed sequence and the layout, so for the
+same input Go and Rust write byte-identical files, and fastconstmap writes the
+same bytes apart from one header word:
+
+| class | magic | notes |
+|---|---|---|
+| `ConstMap` | `CMAP0003` | Go and Rust write `CMAP0001`, the same format minus a 4-byte key count; all three read both |
+| `VerifiedConstMap` | `VMAP0001` | identical; the header word Go and Rust leave zero holds the key count here |
+| `PairedVerifiedConstMap` | `PMAP0001` | likewise |
+
+```python
+m = ConstMap.load("built-by-go.cmap")     # from constmap.SaveToFile / save_to_file
+vm = VerifiedConstMap.load("built-by-rust.vmap")
+```
+
+Two things to know about a map that came from Go or Rust:
+
+- **`len(m)` is 0**, because those writers do not record the key count. Every
+  lookup is unaffected.
+- **A Go or Rust `ConstMap` file (`CMAP0001`) cannot be opened with
+  `from_buffer`**: its 28-byte header leaves the array 4 bytes off an 8-byte
+  boundary in any aligned buffer, so `from_buffer` raises `ValueError`. `load`
+  and `from_bytes`, which copy, work as usual. The other two formats have a
+  32-byte header and view fine.
+
+**Files from fastconstmap 0.9 and earlier** (`CMAP0002` / `VCMP0002`) still
+load and answer correctly: they were built with XXH3, and a map keeps the hash
+its file was built with (`m.hash` tells you which). Re-saving such a map keeps
+its legacy magic, since its table only answers to XXH3; Go and Rust name these
+files in their error message. To get a file the other implementations read,
+rebuild the map from its keys.
+
+**Keeping XXH3.** XXH3 is faster than XXH64 on short keys, by about 1-2 ns per
+lookup at the C level. If you do not need the shared format, build with
+`ConstMap(d, hash="xxh3")` (likewise for the other two classes); such a map
+serializes with the legacy magic, readable by fastconstmap only.
+
+The interoperability is tested: `tests/interop/` holds files written by the Go
+package from a fixed input, which the test suite loads and checks, and the Go
+and Rust repositories hold files written by this package.
+
+[rsconstmap]: https://github.com/lemire/rsconstmap
+
 ## Benchmark
 
 On an Apple M4 Max, with 1,000,000 string keys (`key-{i}-{hex}` shaped
@@ -297,7 +417,7 @@ python benchmarks/benchmark.py 1000000
 
 Given *n* (key, value) pairs the algorithm:
 
-1. Hashes each key with xxhash3 to a 64-bit value.
+1. Hashes each key with XXH64 (or XXH3 if asked) to a 64-bit value.
 2. Maps each hashed key to three positions `h0, h1, h2` in an array of
    size ~1.125·*n*, using overlapping segments.
 3. Finds, via [peeling][bff], an ordering in which each key has an

{fastconstmap-0.9.0 → fastconstmap-0.11.0}/benchmarks/bench_batch.c

@@ -113,14 +113,18 @@ int main(int argc, char **argv) {
     if (!values) { fprintf(stderr, "oom\n"); return 1; }
     for (size_t i = 0; i < map_n; i++) values[i] = i;
 
-    fcm_constmap_t          cm;
-    fcm_verified_constmap_t vm;
+    fcm_constmap_t                 cm;
+    fcm_verified_constmap_t        vm;
+    fcm_paired_verified_constmap_t pm;
     if (fcm_constmap_new(&cm, all.keys, values, map_n) != FCM_OK) {
         fprintf(stderr, "construction failed\n"); return 1;
     }
     if (fcm_verified_constmap_new(&vm, all.keys, values, map_n) != FCM_OK) {
         fprintf(stderr, "verified construction failed\n"); return 1;
     }
+    if (fcm_paired_verified_constmap_new(&pm, all.keys, values, map_n) != FCM_OK) {
+        fprintf(stderr, "paired construction failed\n"); return 1;
+    }
 
     keyset_t pools[NUM_POOLS];
     for (int p = 0; p < NUM_POOLS; p++) {
@@ -215,11 +219,51 @@ int main(int argc, char **argv) {
     report("lookup_many (hot)", now_sec() - t0, total,
            loop_ns * 1e9 / (double)total);
 
+    /* ---- PairedVerifiedConstMap, cold ---- */
+    printf("\nPairedVerifiedConstMap:\n");
+    t0 = now_sec();
+    for (int i = 0; i < REPEATS; i++) {
+        const keyset_t *q = &pools[i % NUM_POOLS];
+        for (size_t j = 0; j < q->n; j++)
+            out[j] = fcm_paired_verified_constmap_lookup(&pm, q->keys[j].bytes, q->keys[j].len);
+        sink = out[0];
+    }
+    loop_ns = now_sec() - t0;
+    report("loop over lookup (cold)", loop_ns, total, 0.0);
+
+    t0 = now_sec();
+    for (int i = 0; i < REPEATS; i++) {
+        const keyset_t *q = &pools[i % NUM_POOLS];
+        fcm_paired_verified_constmap_lookup_many(&pm, q->keys, q->n, out);
+        sink = out[0];
+    }
+    report("lookup_many (cold)", now_sec() - t0, total,
+           loop_ns * 1e9 / (double)total);
+
+    /* ---- PairedVerifiedConstMap, hot ---- */
+    t0 = now_sec();
+    for (int i = 0; i < REPEATS; i++) {
+        for (size_t j = 0; j < hot->n; j++)
+            out[j] = fcm_paired_verified_constmap_lookup(&pm, hot->keys[j].bytes, hot->keys[j].len);
+        sink = out[0];
+    }
+    loop_ns = now_sec() - t0;
+    report("loop over lookup (hot)", loop_ns, total, 0.0);
+
+    t0 = now_sec();
+    for (int i = 0; i < REPEATS; i++) {
+        fcm_paired_verified_constmap_lookup_many(&pm, hot->keys, hot->n, out);
+        sink = out[0];
+    }
+    report("lookup_many (hot)", now_sec() - t0, total,
+           loop_ns * 1e9 / (double)total);
+
     for (int p = 0; p < NUM_POOLS; p++) keyset_free(&pools[p]);
     keyset_free(&all);
     free(values);
     free(out);
     fcm_constmap_free(&cm);
     fcm_verified_constmap_free(&vm);
+    fcm_paired_verified_constmap_free(&pm);
     return 0;
 }

{fastconstmap-0.9.0 → fastconstmap-0.11.0}/benchmarks/benchmark.py

@@ -22,7 +22,7 @@ import sys
 import time
 from typing import Callable
 
-from fastconstmap import ConstMap, VerifiedConstMap
+from fastconstmap import ConstMap, PairedVerifiedConstMap, VerifiedConstMap
 
 
 def make_data(n: int, seed: int = 0xC0FFEE) -> tuple[list[str], dict[str, int]]:
@@ -118,6 +118,10 @@ def main() -> None:
     vm = VerifiedConstMap(d)
     print(f"  VerifiedConstMap.__init__             {time.perf_counter()-t0:7.3f} s")
 
+    t0 = time.perf_counter()
+    pm = PairedVerifiedConstMap(d)
+    print(f"  PairedVerifiedConstMap.__init__       {time.perf_counter()-t0:7.3f} s")
+
     t0 = time.perf_counter()
     pyd = dict(d)
     print(f"  dict(d)                               {time.perf_counter()-t0:7.3f} s")
@@ -130,6 +134,7 @@ def main() -> None:
     print("\nMemory:")
     report_size("ConstMap.nbytes",         cm.nbytes(), n)
     report_size("VerifiedConstMap.nbytes", vm.nbytes(), n)
+    report_size("PairedVerifiedConstMap.nbytes", pm.nbytes(), n)
     dict_total = deep_sizeof(pyd)
     report_size("dict (table+keys+values)", dict_total, n)
     print(f"  {'ratio dict / ConstMap':<38} {dict_total / cm.nbytes():.1f}x")
@@ -140,6 +145,7 @@ def main() -> None:
     time_loop("dict[k]",             n, lambda i: pyd[query[i]])
     time_loop("ConstMap[k]",         n, lambda i: cm[query[i]])
     time_loop("VerifiedConstMap[k]", n, lambda i: vm[query[i]])
+    time_loop("PairedVerifiedConstMap[k]", n, lambda i: pm[query[i]])
 
     # ---- Batched lookup ----
     # Cold rotates through many distinct random batches, so the cache lines a
@@ -175,6 +181,10 @@ def main() -> None:
                lambda: vm.get_many(cold()))
     time_batch("VerifiedConstMap.get_many_into (cold)", args.batches, args.batch_size,
                lambda: vm.get_many_into(cold(), out))
+    time_batch("PairedVerifiedConstMap.get_many (cold)", args.batches, args.batch_size,
+               lambda: pm.get_many(cold()))
+    time_batch("PairedVerifiedConstMap.get_many_into (cold)", args.batches, args.batch_size,
+               lambda: pm.get_many_into(cold(), out))
 
     # ---- Serialization ----
     print("\nSerialization:")

{fastconstmap-0.9.0 → fastconstmap-0.11.0}/fastconstmap/__init__.py

@@ -3,7 +3,7 @@
 Build a compact lookup structure from any ``dict[str, int]`` (or
 ``dict[bytes, int]``). Lookup costs one xxhash and three array reads.
 
-Two classes are exposed:
+Three classes are exposed:
 
 * :class:`ConstMap` — minimal, ~9 bytes/key. Returns undefined values for
   keys that were not in the original mapping. Use when you know upstream
@@ -12,9 +12,18 @@ Two classes are exposed:
   probability ~1 − 2⁻⁶⁴. Behaves like a normal ``dict``: ``m[k]`` raises
   :class:`KeyError`, ``m.get(k, default)`` returns ``default``, ``k in m``
   works.
+* :class:`PairedVerifiedConstMap` — same size and behaviour as
+  :class:`VerifiedConstMap`, but stores each value next to its check word so
+  a lookup of a present key touches three cache lines instead of six (and
+  reads each as one 128-bit word on x86-64 and AArch64). Faster when most
+  lookups hit; slower when most miss. Its serialised form is not
+  interchangeable with :class:`VerifiedConstMap`'s.
 
-Both types are immutable after construction and can be serialised to bytes
-or to a file path. Both also take batches: ``get_many`` returns a list, and
+All three are immutable after construction and can be serialised to bytes
+or to a file path, in a format shared with the Go (``constmap``) and Rust
+(``rsconstmap``) implementations: a file saved by any of the three loads in
+the other two, on a little-endian host. Pass ``hash="xxh3"`` to a constructor
+for a faster key hash whose files only fastconstmap reads. All also take batches: ``get_many`` returns a list, and
 ``get_many_into`` writes 64-bit words straight into a buffer you own (an
 ``array("Q")``, a numpy ``uint64`` array, a ``SharedMemory`` block), which
 allocates nothing per key. A batch is faster than a loop because the map
@@ -23,7 +32,7 @@ whole block overlap instead of being paid one after another. They can also be pl
 ``SharedMemory`` block and opened by many processes with no per-process
 copy — see ``serialized_size``, ``write_into`` and ``from_buffer``.
 """
-from ._fastconstmap import ConstMap, VerifiedConstMap, NOT_FOUND
+from ._fastconstmap import ConstMap, VerifiedConstMap, PairedVerifiedConstMap, NOT_FOUND
 
-__all__ = ["ConstMap", "VerifiedConstMap", "NOT_FOUND"]
-__version__ = "0.9.0"
+__all__ = ["ConstMap", "VerifiedConstMap", "PairedVerifiedConstMap", "NOT_FOUND"]
+__version__ = "0.10.0"