fastconstmap 0.1.0__tar.gz → 0.7.0__tar.gz

{fastconstmap-0.1.0 → fastconstmap-0.7.0}/MANIFEST.in

@@ -5,3 +5,6 @@ include setup.py
 recursive-include src *.c *.h
 recursive-include src/third_party *
 recursive-include fastconstmap *.py *.pyi py.typed
+recursive-include tests *.py
+recursive-include examples *.py
+recursive-include benchmarks *.py

{fastconstmap-0.1.0/fastconstmap.egg-info → fastconstmap-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: fastconstmap
-Version: 0.1.0
+Version: 0.7.0
 Summary: Fast, immutable, compact map from strings to uint64 (binary fuse filter).
 Author-email: Daniel Lemire <daniel@lemire.me>
 License: Apache-2.0
@@ -19,6 +19,7 @@ Dynamic: license-file
 # fastconstmap
 
 [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)
+[![Test](https://github.com/lemire/fastconstmap/actions/workflows/test.yml/badge.svg)](https://github.com/lemire/fastconstmap/actions/workflows/test.yml)
 
 Fast, immutable, compact map from strings to 64-bit integers — for Python.
 
@@ -101,6 +102,78 @@ Keys must be unique (Python dict semantics already guarantee this).
 Construction raises `ValueError` in the extremely unlikely event of an
 xxhash collision (~2⁻⁶⁴ per key pair).
 
+## Sharing a map across processes (zero-copy)
+
+A map's serialized form *is* its in-memory lookup array (plus a small
+header). That means it can live in a
+[`multiprocessing.shared_memory`](https://docs.python.org/3/library/multiprocessing.shared_memory.html)
+block and be opened by any number of processes **without copying** — every
+process reads the same physical pages.
+
+Three methods make this work:
+
+| Method | Purpose |
+|--|--|
+| `m.serialized_size()` | bytes needed to hold the serialized map |
+| `m.write_into(buffer)` | serialize straight into a writable buffer (no intermediate `bytes`) |
+| `ConstMap.from_buffer(buffer)` | open a **zero-copy** map that reads directly from `buffer` |
+
+Producer — build once, publish into a **named** shared-memory block:
+
+```python
+from multiprocessing.shared_memory import SharedMemory
+from fastconstmap import ConstMap
+
+SHM_NAME = "fastconstmap_demo"
+
+cm = ConstMap({f"key-{i}": i for i in range(1_000_000)})
+
+shm = SharedMemory(create=True, size=cm.serialized_size(), name=SHM_NAME)
+cm.write_into(shm.buf)
+# keep `shm` alive (do not close/unlink) while consumers are running
+```
+
+Consumer — attach to the same name with no copy:
+
+```python
+from multiprocessing.shared_memory import SharedMemory
+from fastconstmap import ConstMap
+
+SHM_NAME = "fastconstmap_demo"
+
+shm = SharedMemory(name=SHM_NAME)           # attach by name, no `create=`
+cm = ConstMap.from_buffer(shm.buf)          # zero-copy: no per-process copy
+cm["key-42"]                                # reads straight from shared memory
+```
+
+Choosing the name yourself (rather than letting `SharedMemory` generate
+one) means consumers can hard-code it or read it from config — no need to
+pass the auto-generated name around. Pick a unique name; creating a block
+whose name already exists raises `FileExistsError`.
+
+Notes and constraints:
+
+- **`from_buffer` does not copy.** The returned map holds a reference to the
+  buffer; the buffer (and, for shared memory, the `SharedMemory` object)
+  must stay alive and **must not be closed or mutated** while the map is in
+  use. Drop the map (`del cm`) before calling `shm.close()`.
+- The map is **immutable** — the intended pattern is *write once in the
+  producer, then only read in every process*. Concurrent readers need no
+  locking.
+- `from_buffer` verifies the magic bytes and the FNV-1a checksum, so a
+  truncated or corrupt block raises `ValueError` rather than returning
+  garbage.
+- Requirements: a **little-endian** host (x86-64, ARM64, …) and an
+  **8-byte-aligned** buffer. `SharedMemory.buf`, `bytes`, and `bytearray`
+  all satisfy the alignment requirement; an offset slice of a buffer may
+  not, in which case `from_buffer` raises `ValueError` — use
+  `from_bytes()` (which copies) instead.
+- `VerifiedConstMap` supports the same three methods.
+
+`from_bytes()` remains available when you *want* an owned copy (or need to
+load on a big-endian host): it copies the data and the resulting map owns
+its memory independently of the source buffer.
+
 ## Benchmark
 
 On an Apple M-series CPU, with 1,000,000 string keys

{fastconstmap-0.1.0 → fastconstmap-0.7.0}/README.md

@@ -1,6 +1,7 @@
 # fastconstmap
 
 [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)
+[![Test](https://github.com/lemire/fastconstmap/actions/workflows/test.yml/badge.svg)](https://github.com/lemire/fastconstmap/actions/workflows/test.yml)
 
 Fast, immutable, compact map from strings to 64-bit integers — for Python.
 
@@ -83,6 +84,78 @@ Keys must be unique (Python dict semantics already guarantee this).
 Construction raises `ValueError` in the extremely unlikely event of an
 xxhash collision (~2⁻⁶⁴ per key pair).
 
+## Sharing a map across processes (zero-copy)
+
+A map's serialized form *is* its in-memory lookup array (plus a small
+header). That means it can live in a
+[`multiprocessing.shared_memory`](https://docs.python.org/3/library/multiprocessing.shared_memory.html)
+block and be opened by any number of processes **without copying** — every
+process reads the same physical pages.
+
+Three methods make this work:
+
+| Method | Purpose |
+|--|--|
+| `m.serialized_size()` | bytes needed to hold the serialized map |
+| `m.write_into(buffer)` | serialize straight into a writable buffer (no intermediate `bytes`) |
+| `ConstMap.from_buffer(buffer)` | open a **zero-copy** map that reads directly from `buffer` |
+
+Producer — build once, publish into a **named** shared-memory block:
+
+```python
+from multiprocessing.shared_memory import SharedMemory
+from fastconstmap import ConstMap
+
+SHM_NAME = "fastconstmap_demo"
+
+cm = ConstMap({f"key-{i}": i for i in range(1_000_000)})
+
+shm = SharedMemory(create=True, size=cm.serialized_size(), name=SHM_NAME)
+cm.write_into(shm.buf)
+# keep `shm` alive (do not close/unlink) while consumers are running
+```
+
+Consumer — attach to the same name with no copy:
+
+```python
+from multiprocessing.shared_memory import SharedMemory
+from fastconstmap import ConstMap
+
+SHM_NAME = "fastconstmap_demo"
+
+shm = SharedMemory(name=SHM_NAME)           # attach by name, no `create=`
+cm = ConstMap.from_buffer(shm.buf)          # zero-copy: no per-process copy
+cm["key-42"]                                # reads straight from shared memory
+```
+
+Choosing the name yourself (rather than letting `SharedMemory` generate
+one) means consumers can hard-code it or read it from config — no need to
+pass the auto-generated name around. Pick a unique name; creating a block
+whose name already exists raises `FileExistsError`.
+
+Notes and constraints:
+
+- **`from_buffer` does not copy.** The returned map holds a reference to the
+  buffer; the buffer (and, for shared memory, the `SharedMemory` object)
+  must stay alive and **must not be closed or mutated** while the map is in
+  use. Drop the map (`del cm`) before calling `shm.close()`.
+- The map is **immutable** — the intended pattern is *write once in the
+  producer, then only read in every process*. Concurrent readers need no
+  locking.
+- `from_buffer` verifies the magic bytes and the FNV-1a checksum, so a
+  truncated or corrupt block raises `ValueError` rather than returning
+  garbage.
+- Requirements: a **little-endian** host (x86-64, ARM64, …) and an
+  **8-byte-aligned** buffer. `SharedMemory.buf`, `bytes`, and `bytearray`
+  all satisfy the alignment requirement; an offset slice of a buffer may
+  not, in which case `from_buffer` raises `ValueError` — use
+  `from_bytes()` (which copies) instead.
+- `VerifiedConstMap` supports the same three methods.
+
+`from_bytes()` remains available when you *want* an owned copy (or need to
+load on a big-endian host): it copies the data and the resulting map owns
+its memory independently of the source buffer.
+
 ## Benchmark
 
 On an Apple M-series CPU, with 1,000,000 string keys

fastconstmap-0.7.0/benchmarks/benchmark.py

@@ -0,0 +1,155 @@
+"""Benchmark fastconstmap against the built-in dict.
+
+Run as:
+    python benchmarks/benchmark.py [N]
+
+Defaults to N = 1,000,000 keys.
+"""
+from __future__ import annotations
+import argparse
+import random
+import sys
+import time
+from typing import Callable
+
+from fastconstmap import ConstMap, VerifiedConstMap
+
+
+def make_data(n: int, seed: int = 0xC0FFEE) -> tuple[list[str], dict[str, int]]:
+    rng = random.Random(seed)
+    keys = [f"key-{i}-{rng.randint(0, 1 << 30):x}" for i in range(n)]
+    return keys, {k: i for i, k in enumerate(keys)}
+
+
+def time_loop(label: str, iterations: int, fn: Callable[[int], None]) -> float:
+    # Warm-up so the JIT-equivalent (branch predictor, page faults) settles.
+    for i in range(min(iterations, 1000)):
+        fn(i)
+    t0 = time.perf_counter()
+    for i in range(iterations):
+        fn(i)
+    elapsed = time.perf_counter() - t0
+    per_op_ns = elapsed * 1e9 / iterations
+    print(f"  {label:<28} {per_op_ns:8.1f} ns/op  ({elapsed:.3f} s total)")
+    return per_op_ns
+
+
+def time_batch(label: str, batches: int, batch_size: int, fn: Callable[[], None]) -> float:
+    for _ in range(min(batches, 5)):
+        fn()
+    t0 = time.perf_counter()
+    for _ in range(batches):
+        fn()
+    elapsed = time.perf_counter() - t0
+    per_op_ns = elapsed * 1e9 / (batches * batch_size)
+    print(f"  {label:<28} {per_op_ns:8.1f} ns/op  "
+          f"({elapsed*1000/batches:.2f} ms/batch of {batch_size})")
+    return per_op_ns
+
+
+def report_size(label: str, value: int, n: int) -> None:
+    print(f"  {label:<28} {value:12,} bytes  ({value/n:.2f} bytes/key)")
+
+
+def deep_sizeof(obj, seen: set[int] | None = None) -> int:
+    """Recursive sizeof that walks dict / list / tuple / set, deduping by id.
+
+    Returns the total bytes attributable to ``obj`` and everything it
+    transitively references. Small interned ints (-5..256) and interned
+    strings are counted once because of the id() dedup, which matches
+    reality — they're shared with the rest of the process.
+    """
+    if seen is None:
+        seen = set()
+    obj_id = id(obj)
+    if obj_id in seen:
+        return 0
+    seen.add(obj_id)
+    size = sys.getsizeof(obj)
+    if isinstance(obj, dict):
+        for k, v in obj.items():
+            size += deep_sizeof(k, seen)
+            size += deep_sizeof(v, seen)
+    elif isinstance(obj, (list, tuple, set, frozenset)):
+        for item in obj:
+            size += deep_sizeof(item, seen)
+    return size
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("n", type=int, nargs="?", default=1_000_000)
+    parser.add_argument("--lookups", type=int, default=2_000_000,
+                        help="number of individual lookups to time")
+    parser.add_argument("--batch-size", type=int, default=1024)
+    parser.add_argument("--batches",    type=int, default=2000)
+    args = parser.parse_args()
+    n = args.n
+
+    print(f"=== fastconstmap benchmark — n = {n:,} keys, "
+          f"python {sys.version.split()[0]} ===\n")
+
+    keys, d = make_data(n)
+    # Lookup index pattern: shuffled to avoid trivial cache wins.
+    rng = random.Random(1234)
+    lookup_idx = [rng.randrange(n) for _ in range(args.lookups)]
+
+    # ---- Construction time ----
+    print("Construction:")
+    t0 = time.perf_counter()
+    cm = ConstMap(d)
+    print(f"  ConstMap.__init__              {time.perf_counter()-t0:7.3f} s")
+
+    t0 = time.perf_counter()
+    vm = VerifiedConstMap(d)
+    print(f"  VerifiedConstMap.__init__      {time.perf_counter()-t0:7.3f} s")
+
+    t0 = time.perf_counter()
+    pyd = dict(d)
+    print(f"  dict(d)                        {time.perf_counter()-t0:7.3f} s")
+
+    # ---- Memory ----
+    # The ConstMap retains no string objects: cm.nbytes() is its entire
+    # footprint. To compare honestly we walk the dict and add up the
+    # dict-table size *plus* every str key and int value it references
+    # (deduped by id, so interned objects count once).
+    print("\nMemory:")
+    report_size("ConstMap.nbytes",         cm.nbytes(), n)
+    report_size("VerifiedConstMap.nbytes", vm.nbytes(), n)
+    dict_total = deep_sizeof(pyd)
+    report_size("dict (table+keys+values)", dict_total, n)
+    print(f"  ratio dict / ConstMap          {dict_total / cm.nbytes():.1f}x")
+
+    # ---- Single lookup ----
+    print(f"\nSingle lookup, {args.lookups:,} ops:")
+
+    time_loop("dict[k]",                     args.lookups, lambda i: pyd[keys[lookup_idx[i]]])
+    time_loop("ConstMap[k]",                 args.lookups, lambda i: cm[keys[lookup_idx[i]]])
+    time_loop("VerifiedConstMap[k]",         args.lookups, lambda i: vm[keys[lookup_idx[i]]])
+
+    # ---- Batched lookup ----
+    batch_keys = [keys[rng.randrange(n)] for _ in range(args.batch_size)]
+    print(f"\nBatched lookup, {args.batches} × {args.batch_size}:")
+
+    # dict batch: pure Python list comprehension is the natural comparison.
+    time_batch("dict comprehension",         args.batches, args.batch_size,
+               lambda: [pyd[k] for k in batch_keys])
+    time_batch("ConstMap.get_many",          args.batches, args.batch_size,
+               lambda: cm.get_many(batch_keys))
+    time_batch("VerifiedConstMap.get_many",  args.batches, args.batch_size,
+               lambda: vm.get_many(batch_keys))
+
+    # ---- Serialization ----
+    print("\nSerialization:")
+    t0 = time.perf_counter()
+    blob = cm.to_bytes()
+    print(f"  ConstMap.to_bytes              {time.perf_counter()-t0:7.3f} s  ({len(blob):,} bytes)")
+    t0 = time.perf_counter()
+    cm2 = ConstMap.from_bytes(blob)
+    print(f"  ConstMap.from_bytes            {time.perf_counter()-t0:7.3f} s")
+    # Verify round-trip.
+    assert cm2[keys[0]] == d[keys[0]]
+
+
+if __name__ == "__main__":
+    main()

fastconstmap-0.7.0/examples/example.py

@@ -0,0 +1,25 @@
+"""Tiny end-to-end example: build, look up, save, reload."""
+from fastconstmap import ConstMap, VerifiedConstMap
+
+d = {"apple": 100, "banana": 200, "cherry": 300}
+
+# Minimal, no missing-key detection.
+m = ConstMap(d)
+print("apple   ->", m["apple"])
+print("batched ->", m.get_many(["banana", "cherry"]))
+
+# Dict-like, detects missing keys.
+vm = VerifiedConstMap(d)
+print("vm grape?", vm.get("grape"))
+print("vm has banana?", "banana" in vm)
+
+# Round-trip through bytes.
+blob = m.to_bytes()
+m2 = ConstMap.from_bytes(blob)
+assert m2["banana"] == 200
+
+# Round-trip through a file.
+m.save("example.cmap")
+m3 = ConstMap.load("example.cmap")
+assert m3["cherry"] == 300
+print("save/load OK")

{fastconstmap-0.1.0 → fastconstmap-0.7.0}/fastconstmap/__init__.py

@@ -14,9 +14,11 @@ Two classes are exposed:
   works.
 
 Both types are immutable after construction and can be serialised to bytes
-or to a file path.
+or to a file path. They can also be placed in a ``multiprocessing``
+``SharedMemory`` block and opened by many processes with no per-process
+copy — see ``serialized_size``, ``write_into`` and ``from_buffer``.
 """
 from ._fastconstmap import ConstMap, VerifiedConstMap
 
 __all__ = ["ConstMap", "VerifiedConstMap"]
-__version__ = "0.1.0"
+__version__ = "0.7.0"

{fastconstmap-0.1.0 → fastconstmap-0.7.0}/fastconstmap/_fastconstmap.pyi

@@ -3,8 +3,9 @@ from __future__ import annotations
 from os import PathLike
 from typing import Iterable, Mapping, Optional, Union
 
-KeyT  = Union[str, bytes]
-PathT = Union[str, bytes, PathLike[str], PathLike[bytes]]
+KeyT    = Union[str, bytes]
+PathT   = Union[str, bytes, PathLike[str], PathLike[bytes]]
+BufferT = Union[bytes, bytearray, memoryview]
 
 
 class ConstMap:
@@ -21,7 +22,11 @@ class ConstMap:
     def get_many(self, keys: Iterable[KeyT]) -> list[int]: ...
     def to_bytes(self) -> bytes: ...
     @classmethod
-    def from_bytes(cls, data: bytes) -> "ConstMap": ...
+    def from_bytes(cls, data: BufferT) -> "ConstMap": ...
+    def serialized_size(self) -> int: ...
+    def write_into(self, buffer: BufferT) -> int: ...
+    @classmethod
+    def from_buffer(cls, buffer: BufferT) -> "ConstMap": ...
     def save(self, path: PathT) -> None: ...
     @classmethod
     def load(cls, path: PathT) -> "ConstMap": ...
@@ -46,7 +51,11 @@ class VerifiedConstMap:
     def get_many(self, keys: Iterable[KeyT], default: Optional[int] = None) -> list[Optional[int]]: ...
     def to_bytes(self) -> bytes: ...
     @classmethod
-    def from_bytes(cls, data: bytes) -> "VerifiedConstMap": ...
+    def from_bytes(cls, data: BufferT) -> "VerifiedConstMap": ...
+    def serialized_size(self) -> int: ...
+    def write_into(self, buffer: BufferT) -> int: ...
+    @classmethod
+    def from_buffer(cls, buffer: BufferT) -> "VerifiedConstMap": ...
     def save(self, path: PathT) -> None: ...
     @classmethod
     def load(cls, path: PathT) -> "VerifiedConstMap": ...

{fastconstmap-0.1.0 → fastconstmap-0.7.0/fastconstmap.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: fastconstmap
-Version: 0.1.0
+Version: 0.7.0
 Summary: Fast, immutable, compact map from strings to uint64 (binary fuse filter).
 Author-email: Daniel Lemire <daniel@lemire.me>
 License: Apache-2.0
@@ -19,6 +19,7 @@ Dynamic: license-file
 # fastconstmap
 
 [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)
+[![Test](https://github.com/lemire/fastconstmap/actions/workflows/test.yml/badge.svg)](https://github.com/lemire/fastconstmap/actions/workflows/test.yml)
 
 Fast, immutable, compact map from strings to 64-bit integers — for Python.
 
@@ -101,6 +102,78 @@ Keys must be unique (Python dict semantics already guarantee this).
 Construction raises `ValueError` in the extremely unlikely event of an
 xxhash collision (~2⁻⁶⁴ per key pair).
 
+## Sharing a map across processes (zero-copy)
+
+A map's serialized form *is* its in-memory lookup array (plus a small
+header). That means it can live in a
+[`multiprocessing.shared_memory`](https://docs.python.org/3/library/multiprocessing.shared_memory.html)
+block and be opened by any number of processes **without copying** — every
+process reads the same physical pages.
+
+Three methods make this work:
+
+| Method | Purpose |
+|--|--|
+| `m.serialized_size()` | bytes needed to hold the serialized map |
+| `m.write_into(buffer)` | serialize straight into a writable buffer (no intermediate `bytes`) |
+| `ConstMap.from_buffer(buffer)` | open a **zero-copy** map that reads directly from `buffer` |
+
+Producer — build once, publish into a **named** shared-memory block:
+
+```python
+from multiprocessing.shared_memory import SharedMemory
+from fastconstmap import ConstMap
+
+SHM_NAME = "fastconstmap_demo"
+
+cm = ConstMap({f"key-{i}": i for i in range(1_000_000)})
+
+shm = SharedMemory(create=True, size=cm.serialized_size(), name=SHM_NAME)
+cm.write_into(shm.buf)
+# keep `shm` alive (do not close/unlink) while consumers are running
+```
+
+Consumer — attach to the same name with no copy:
+
+```python
+from multiprocessing.shared_memory import SharedMemory
+from fastconstmap import ConstMap
+
+SHM_NAME = "fastconstmap_demo"
+
+shm = SharedMemory(name=SHM_NAME)           # attach by name, no `create=`
+cm = ConstMap.from_buffer(shm.buf)          # zero-copy: no per-process copy
+cm["key-42"]                                # reads straight from shared memory
+```
+
+Choosing the name yourself (rather than letting `SharedMemory` generate
+one) means consumers can hard-code it or read it from config — no need to
+pass the auto-generated name around. Pick a unique name; creating a block
+whose name already exists raises `FileExistsError`.
+
+Notes and constraints:
+
+- **`from_buffer` does not copy.** The returned map holds a reference to the
+  buffer; the buffer (and, for shared memory, the `SharedMemory` object)
+  must stay alive and **must not be closed or mutated** while the map is in
+  use. Drop the map (`del cm`) before calling `shm.close()`.
+- The map is **immutable** — the intended pattern is *write once in the
+  producer, then only read in every process*. Concurrent readers need no
+  locking.
+- `from_buffer` verifies the magic bytes and the FNV-1a checksum, so a
+  truncated or corrupt block raises `ValueError` rather than returning
+  garbage.
+- Requirements: a **little-endian** host (x86-64, ARM64, …) and an
+  **8-byte-aligned** buffer. `SharedMemory.buf`, `bytes`, and `bytearray`
+  all satisfy the alignment requirement; an offset slice of a buffer may
+  not, in which case `from_buffer` raises `ValueError` — use
+  `from_bytes()` (which copies) instead.
+- `VerifiedConstMap` supports the same three methods.
+
+`from_bytes()` remains available when you *want* an owned copy (or need to
+load on a big-endian host): it copies the data and the resulting map owns
+its memory independently of the source buffer.
+
 ## Benchmark
 
 On an Apple M-series CPU, with 1,000,000 string keys

{fastconstmap-0.1.0 → fastconstmap-0.7.0}/fastconstmap.egg-info/SOURCES.txt

@@ -3,6 +3,8 @@ MANIFEST.in
 README.md
 pyproject.toml
 setup.py
+benchmarks/benchmark.py
+examples/example.py
 fastconstmap/__init__.py
 fastconstmap/_fastconstmap.pyi
 fastconstmap/py.typed

{fastconstmap-0.1.0 → fastconstmap-0.7.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "fastconstmap"
-version = "0.1.0"
+version = "0.7.0"
 description = "Fast, immutable, compact map from strings to uint64 (binary fuse filter)."
 readme = "README.md"
 license = { text = "Apache-2.0" }
@@ -38,7 +38,10 @@ test-command = "pytest {project}/tests"
 archs = ["x86_64", "universal2", "arm64"]
 
 [tool.cibuildwheel.linux]
-archs = ["x86_64"]
+# "auto64" = the runner's native 64-bit arch (no 32-bit i686). The wheels
+# workflow runs this on both an x86_64 runner and an aarch64 runner, so
+# this produces x86_64 and aarch64 manylinux wheels — each built natively.
+archs = ["auto64"]
 
 [tool.cibuildwheel.windows]
 archs = ["AMD64"]

{fastconstmap-0.1.0 → fastconstmap-0.7.0}/src/constmap.c

@@ -563,15 +563,17 @@ uint64_t fcm_verified_constmap_lookup(const fcm_verified_constmap_t *vm,
 /*   [4] segment_length                                                      */
 /*   [4] segment_count                                                       */
 /*   [4] data_len                                                            */
+/*   [4] reserved (zero) — pads the header to 32 bytes so the data array      */
+/*       starts 8-byte aligned, enabling zero-copy uint64 views               */
 /*   [8 * data_len] data                                                     */
 /*   (verified only) [8 * data_len] checks                                   */
 /*   [8] FNV-1a-64 checksum of all preceding bytes                           */
 /* ------------------------------------------------------------------------- */
 
-static const uint8_t fcm_magic_constmap[8]          = {'C','M','A','P','0','0','0','1'};
-static const uint8_t fcm_magic_verified_constmap[8] = {'V','C','M','P','0','0','0','1'};
+static const uint8_t fcm_magic_constmap[8]          = {'C','M','A','P','0','0','0','2'};
+static const uint8_t fcm_magic_verified_constmap[8] = {'V','C','M','P','0','0','0','2'};
 
-#define FCM_HEADER_SIZE 28u  /* 8 + 8 + 4 + 4 + 4 */
+#define FCM_HEADER_SIZE 32u  /* 8 magic + 8 seed + 4 seglen + 4 segcount + 4 datalen + 4 pad */
 #define FCM_TRAILER_SIZE 8u  /* checksum */
 
 static inline void fcm_write_u32(uint8_t *p, uint32_t v) {
@@ -629,6 +631,7 @@ int fcm_constmap_write(const fcm_constmap_t *cm, void *buf) {
     fcm_write_u32(p, cm->segment_length); p += 4;
     fcm_write_u32(p, cm->segment_count);  p += 4;
     fcm_write_u32(p, cm->data_len);       p += 4;
+    fcm_write_u32(p, 0);                  p += 4;  /* reserved padding */
     for (uint32_t i = 0; i < cm->data_len; i++) {
         fcm_write_u64(p, cm->data[i]); p += 8;
     }
@@ -688,6 +691,7 @@ int fcm_verified_constmap_write(const fcm_verified_constmap_t *vm, void *buf) {
     fcm_write_u32(p, vm->segment_length); p += 4;
     fcm_write_u32(p, vm->segment_count);  p += 4;
     fcm_write_u32(p, vm->data_len);       p += 4;
+    fcm_write_u32(p, 0);                  p += 4;  /* reserved padding */
     for (uint32_t i = 0; i < vm->data_len; i++) {
         fcm_write_u64(p, vm->data[i]); p += 8;
     }
@@ -744,3 +748,78 @@ int fcm_verified_constmap_read(fcm_verified_constmap_t *out, const void *buf, si
     out->checks               = checks;
     return FCM_OK;
 }
+
+/* ------------------------------------------------------------------------- */
+/* Zero-copy views                                                           */
+/* ------------------------------------------------------------------------- */
+
+static inline int fcm_host_is_little_endian(void) {
+    const uint16_t x = 1;
+    return *(const uint8_t *)&x == 1;
+}
+
+int fcm_constmap_view(fcm_constmap_t *out, const void *buf, size_t buf_len) {
+    if (!out) return FCM_E_INVALID_FORMAT;
+    memset(out, 0, sizeof(*out));
+    if (!fcm_host_is_little_endian()) return FCM_E_INVALID_FORMAT;
+    if (buf_len < FCM_HEADER_SIZE + FCM_TRAILER_SIZE) return FCM_E_SHORT_BUFFER;
+
+    const uint8_t *p = (const uint8_t *)buf;
+    if (memcmp(p, fcm_magic_constmap, 8) != 0) return FCM_E_INVALID_FORMAT;
+
+    uint32_t data_len = fcm_read_u32(p + 24);
+    size_t expected = FCM_HEADER_SIZE + (size_t)data_len * 8u + FCM_TRAILER_SIZE;
+    if (buf_len < expected) return FCM_E_SHORT_BUFFER;
+
+    uint64_t got_sum      = fcm_read_u64(p + expected - 8);
+    uint64_t expected_sum = fcm_fnv1a64(p, expected - 8);
+    if (got_sum != expected_sum) return FCM_E_CHECKSUM;
+
+    const uint8_t *dp = p + FCM_HEADER_SIZE;
+    if (((uintptr_t)dp & 7u) != 0) return FCM_E_UNALIGNED;
+
+    uint32_t segment_length   = fcm_read_u32(p + 16);
+    out->seed                 = fcm_read_u64(p + 8);
+    out->segment_length       = segment_length;
+    out->segment_length_mask  = segment_length ? segment_length - 1 : 0;
+    out->segment_count        = fcm_read_u32(p + 20);
+    out->segment_count_length = out->segment_count * segment_length;
+    out->data_len             = data_len;
+    /* Borrowed pointer into `buf`; the integer round-trip launders away the
+     * source const-ness. Lookups only read this array. */
+    out->data = (uint64_t *)(uintptr_t)dp;
+    return FCM_OK;
+}
+
+int fcm_verified_constmap_view(fcm_verified_constmap_t *out, const void *buf, size_t buf_len) {
+    if (!out) return FCM_E_INVALID_FORMAT;
+    memset(out, 0, sizeof(*out));
+    if (!fcm_host_is_little_endian()) return FCM_E_INVALID_FORMAT;
+    if (buf_len < FCM_HEADER_SIZE + FCM_TRAILER_SIZE) return FCM_E_SHORT_BUFFER;
+
+    const uint8_t *p = (const uint8_t *)buf;
+    if (memcmp(p, fcm_magic_verified_constmap, 8) != 0) return FCM_E_INVALID_FORMAT;
+
+    uint32_t data_len = fcm_read_u32(p + 24);
+    size_t expected = FCM_HEADER_SIZE + (size_t)data_len * 16u + FCM_TRAILER_SIZE;
+    if (buf_len < expected) return FCM_E_SHORT_BUFFER;
+
+    uint64_t got_sum      = fcm_read_u64(p + expected - 8);
+    uint64_t expected_sum = fcm_fnv1a64(p, expected - 8);
+    if (got_sum != expected_sum) return FCM_E_CHECKSUM;
+
+    const uint8_t *dp = p + FCM_HEADER_SIZE;
+    const uint8_t *cp = dp + (size_t)data_len * 8u;
+    if (((uintptr_t)dp & 7u) != 0) return FCM_E_UNALIGNED;
+
+    uint32_t segment_length   = fcm_read_u32(p + 16);
+    out->seed                 = fcm_read_u64(p + 8);
+    out->segment_length       = segment_length;
+    out->segment_length_mask  = segment_length ? segment_length - 1 : 0;
+    out->segment_count        = fcm_read_u32(p + 20);
+    out->segment_count_length = out->segment_count * segment_length;
+    out->data_len             = data_len;
+    out->data   = (uint64_t *)(uintptr_t)dp;
+    out->checks = (uint64_t *)(uintptr_t)cp;
+    return FCM_OK;
+}

{fastconstmap-0.1.0 → fastconstmap-0.7.0}/src/constmap.h

@@ -25,6 +25,7 @@ extern "C" {
 #define FCM_E_INVALID_FORMAT     -5
 #define FCM_E_CHECKSUM           -6
 #define FCM_E_SHORT_BUFFER       -7
+#define FCM_E_UNALIGNED          -8
 
 #define FCM_NOT_FOUND ((uint64_t)0xFFFFFFFFFFFFFFFFULL)
 
@@ -85,6 +86,24 @@ size_t fcm_verified_constmap_serialized_size(const fcm_verified_constmap_t *vm);
 int    fcm_verified_constmap_write(const fcm_verified_constmap_t *vm, void *buf);
 int    fcm_verified_constmap_read (fcm_verified_constmap_t *out, const void *buf, size_t buf_len);
 
+/* Zero-copy views over a serialized buffer.
+ *
+ * Unlike `*_read`, these do NOT copy: `out->data` (and `out->checks`) point
+ * directly into `buf`. This makes a map openable from shared memory with no
+ * per-process copy.
+ *
+ * Requirements: a little-endian host and a buffer whose start is 8-byte
+ * aligned (so the embedded uint64 arrays are aligned). The serialized
+ * format pads its header to a multiple of 8 to guarantee this.
+ *
+ * Returns FCM_OK, or FCM_E_UNALIGNED / FCM_E_INVALID_FORMAT (e.g. on a
+ * big-endian host) / FCM_E_CHECKSUM / FCM_E_SHORT_BUFFER.
+ *
+ * The caller MUST keep `buf` alive for the lifetime of `out`, and MUST NOT
+ * call `fcm_*_free` on a view (the data is borrowed, not owned). */
+int fcm_constmap_view(fcm_constmap_t *out, const void *buf, size_t buf_len);
+int fcm_verified_constmap_view(fcm_verified_constmap_t *out, const void *buf, size_t buf_len);
+
 #ifdef __cplusplus
 }
 #endif

{fastconstmap-0.1.0 → fastconstmap-0.7.0}/src/fastconstmap_module.c

@@ -153,8 +153,22 @@ fail:
 typedef struct {
     PyObject_HEAD
     fcm_constmap_t cm;
+    /* When `view.obj` is non-NULL the map is a zero-copy view: `cm.data`
+     * points into `view`'s buffer and must NOT be freed. */
+    Py_buffer view;
 } PyConstMap;
 
+/* Release whatever storage the map currently holds (owned heap data or a
+ * borrowed buffer). Safe to call on a zero-initialised struct. */
+static void pyconstmap_release(PyConstMap *self) {
+    if (self->view.obj != NULL) {
+        /* Borrowed: data lives inside the buffer; just drop our reference. */
+        PyBuffer_Release(&self->view);   /* sets view.obj = NULL */
+        self->cm.data = NULL;
+    }
+    fcm_constmap_free(&self->cm);        /* frees owned data; no-op if NULL */
+}
+
 static int PyConstMap_init(PyConstMap *self, PyObject *args, PyObject *kwargs) {
     PyObject *dict;
     static char *kwlist[] = {"mapping", NULL};
@@ -166,7 +180,7 @@ static int PyConstMap_init(PyConstMap *self, PyObject *args, PyObject *kwargs) {
     PyObject  *keep = NULL;
     if (fcm_dict_to_arrays(dict, &n, &keys, &vals, &keep) < 0) return -1;
 
-    fcm_constmap_free(&self->cm);
+    pyconstmap_release(self);
 
     int rc;
     Py_BEGIN_ALLOW_THREADS
@@ -182,7 +196,7 @@ static int PyConstMap_init(PyConstMap *self, PyObject *args, PyObject *kwargs) {
 }
 
 static void PyConstMap_dealloc(PyConstMap *self) {
-    fcm_constmap_free(&self->cm);
+    pyconstmap_release(self);
     Py_TYPE(self)->tp_free((PyObject *)self);
 }
 
@@ -363,20 +377,74 @@ static PyObject *PyConstMap_data_bytes(PyConstMap *self, PyObject *Py_UNUSED(ign
     return PyLong_FromUnsignedLongLong((unsigned long long)self->cm.data_len * 8ULL);
 }
 
+static PyObject *PyConstMap_serialized_size(PyConstMap *self, PyObject *Py_UNUSED(ignored)) {
+    return PyLong_FromSize_t(fcm_constmap_serialized_size(&self->cm));
+}
+
+/* Serialize directly into a caller-provided writable buffer (e.g. a
+ * SharedMemory block), avoiding the intermediate bytes object. */
+static PyObject *PyConstMap_write_into(PyConstMap *self, PyObject *arg) {
+    Py_buffer view;
+    if (PyObject_GetBuffer(arg, &view, PyBUF_WRITABLE) < 0) return NULL;
+    size_t need = fcm_constmap_serialized_size(&self->cm);
+    if (view.len < 0 || (size_t)view.len < need) {
+        PyErr_Format(PyExc_ValueError,
+                     "buffer too small: need %zu bytes, got %zd",
+                     need, view.len);
+        PyBuffer_Release(&view);
+        return NULL;
+    }
+    fcm_constmap_write(&self->cm, view.buf);
+    PyBuffer_Release(&view);
+    return PyLong_FromSize_t(need);
+}
+
+/* Zero-copy: build a map that reads directly out of `buffer`. The buffer
+ * is kept alive by the returned object; lookups touch it directly. */
+static PyObject *PyConstMap_from_buffer(PyTypeObject *type, PyObject *arg) {
+    Py_buffer view;
+    if (PyObject_GetBuffer(arg, &view, PyBUF_SIMPLE) < 0) return NULL;
+    PyConstMap *self = (PyConstMap *)type->tp_alloc(type, 0);
+    if (!self) { PyBuffer_Release(&view); return NULL; }
+    /* tp_alloc zeroed the struct, so self->view.obj == NULL and cm is clear. */
+    int rc = fcm_constmap_view(&self->cm, view.buf, (size_t)view.len);
+    if (rc != FCM_OK) {
+        PyBuffer_Release(&view);
+        Py_DECREF(self);
+        if (rc == FCM_E_UNALIGNED) {
+            PyErr_SetString(PyExc_ValueError,
+                "buffer is not 8-byte aligned; cannot create a zero-copy view "
+                "(use from_bytes() to copy instead)");
+        } else {
+            fcm_set_python_error(rc);
+        }
+        return NULL;
+    }
+    self->view = view;  /* keep the buffer alive for the map's lifetime */
+    return (PyObject *)self;
+}
+
 static PyMethodDef PyConstMap_methods[] = {
-    {"get",         (PyCFunction)PyConstMap_get,         METH_O,
+    {"get",             (PyCFunction)PyConstMap_get,             METH_O,
      "Return the value for `key`. If `key` was not in the original mapping, the return value is undefined."},
-    {"get_many",    (PyCFunction)PyConstMap_get_many,    METH_O,
+    {"get_many",        (PyCFunction)PyConstMap_get_many,        METH_O,
      "Look up an iterable of keys. Returns a list of ints in the same order."},
-    {"to_bytes",    (PyCFunction)PyConstMap_serialize,   METH_NOARGS,
+    {"to_bytes",        (PyCFunction)PyConstMap_serialize,       METH_NOARGS,
      "Serialize the map to bytes."},
-    {"from_bytes",  (PyCFunction)PyConstMap_deserialize, METH_O   | METH_CLASS,
-     "Deserialize a map from a bytes-like object."},
-    {"save",        (PyCFunction)PyConstMap_save,        METH_O,
+    {"from_bytes",      (PyCFunction)PyConstMap_deserialize,     METH_O   | METH_CLASS,
+     "Deserialize a map from a bytes-like object (copies the data)."},
+    {"serialized_size", (PyCFunction)PyConstMap_serialized_size, METH_NOARGS,
+     "Number of bytes that to_bytes()/write_into() will produce."},
+    {"write_into",      (PyCFunction)PyConstMap_write_into,      METH_O,
+     "Serialize directly into a writable buffer (e.g. SharedMemory.buf). Returns bytes written."},
+    {"from_buffer",     (PyCFunction)PyConstMap_from_buffer,     METH_O   | METH_CLASS,
+     "Create a zero-copy map that reads directly from a buffer (e.g. shared memory). "
+     "The buffer is kept alive by the returned map and must not be modified while in use."},
+    {"save",            (PyCFunction)PyConstMap_save,            METH_O,
      "Save the map to a file path."},
-    {"load",        (PyCFunction)PyConstMap_load,        METH_O   | METH_CLASS,
+    {"load",            (PyCFunction)PyConstMap_load,            METH_O   | METH_CLASS,
      "Load a map from a file path."},
-    {"nbytes",      (PyCFunction)PyConstMap_data_bytes,  METH_NOARGS,
+    {"nbytes",          (PyCFunction)PyConstMap_data_bytes,      METH_NOARGS,
      "Heap size of the lookup array in bytes."},
     {NULL, NULL, 0, NULL}
 };
@@ -407,8 +475,19 @@ static PyTypeObject PyConstMapType = {
 typedef struct {
     PyObject_HEAD
     fcm_verified_constmap_t vm;
+    /* Non-NULL view.obj => zero-copy view: data/checks point into the buffer. */
+    Py_buffer view;
 } PyVerifiedConstMap;
 
+static void pyverified_release(PyVerifiedConstMap *self) {
+    if (self->view.obj != NULL) {
+        PyBuffer_Release(&self->view);
+        self->vm.data   = NULL;
+        self->vm.checks = NULL;
+    }
+    fcm_verified_constmap_free(&self->vm);
+}
+
 static int PyVerifiedConstMap_init(PyVerifiedConstMap *self, PyObject *args, PyObject *kwargs) {
     PyObject *dict;
     static char *kwlist[] = {"mapping", NULL};
@@ -420,7 +499,7 @@ static int PyVerifiedConstMap_init(PyVerifiedConstMap *self, PyObject *args, PyO
     PyObject  *keep = NULL;
     if (fcm_dict_to_arrays(dict, &n, &keys, &vals, &keep) < 0) return -1;
 
-    fcm_verified_constmap_free(&self->vm);
+    pyverified_release(self);
 
     int rc;
     Py_BEGIN_ALLOW_THREADS
@@ -436,7 +515,7 @@ static int PyVerifiedConstMap_init(PyVerifiedConstMap *self, PyObject *args, PyO
 }
 
 static void PyVerifiedConstMap_dealloc(PyVerifiedConstMap *self) {
-    fcm_verified_constmap_free(&self->vm);
+    pyverified_release(self);
     Py_TYPE(self)->tp_free((PyObject *)self);
 }
 
@@ -655,20 +734,69 @@ static PyObject *PyVerifiedConstMap_nbytes(PyVerifiedConstMap *self, PyObject *P
     return PyLong_FromUnsignedLongLong((unsigned long long)self->vm.data_len * 16ULL);
 }
 
+static PyObject *PyVerifiedConstMap_serialized_size(PyVerifiedConstMap *self, PyObject *Py_UNUSED(ignored)) {
+    return PyLong_FromSize_t(fcm_verified_constmap_serialized_size(&self->vm));
+}
+
+static PyObject *PyVerifiedConstMap_write_into(PyVerifiedConstMap *self, PyObject *arg) {
+    Py_buffer view;
+    if (PyObject_GetBuffer(arg, &view, PyBUF_WRITABLE) < 0) return NULL;
+    size_t need = fcm_verified_constmap_serialized_size(&self->vm);
+    if (view.len < 0 || (size_t)view.len < need) {
+        PyErr_Format(PyExc_ValueError,
+                     "buffer too small: need %zu bytes, got %zd",
+                     need, view.len);
+        PyBuffer_Release(&view);
+        return NULL;
+    }
+    fcm_verified_constmap_write(&self->vm, view.buf);
+    PyBuffer_Release(&view);
+    return PyLong_FromSize_t(need);
+}
+
+static PyObject *PyVerifiedConstMap_from_buffer(PyTypeObject *type, PyObject *arg) {
+    Py_buffer view;
+    if (PyObject_GetBuffer(arg, &view, PyBUF_SIMPLE) < 0) return NULL;
+    PyVerifiedConstMap *self = (PyVerifiedConstMap *)type->tp_alloc(type, 0);
+    if (!self) { PyBuffer_Release(&view); return NULL; }
+    int rc = fcm_verified_constmap_view(&self->vm, view.buf, (size_t)view.len);
+    if (rc != FCM_OK) {
+        PyBuffer_Release(&view);
+        Py_DECREF(self);
+        if (rc == FCM_E_UNALIGNED) {
+            PyErr_SetString(PyExc_ValueError,
+                "buffer is not 8-byte aligned; cannot create a zero-copy view "
+                "(use from_bytes() to copy instead)");
+        } else {
+            fcm_set_python_error(rc);
+        }
+        return NULL;
+    }
+    self->view = view;
+    return (PyObject *)self;
+}
+
 static PyMethodDef PyVerifiedConstMap_methods[] = {
-    {"get",        (PyCFunction)PyVerifiedConstMap_get,                       METH_VARARGS,
+    {"get",             (PyCFunction)PyVerifiedConstMap_get,                       METH_VARARGS,
      "Return value for `key`, or `default` (default None) if not present."},
-    {"get_many",   (PyCFunction)(void *)PyVerifiedConstMap_get_many,          METH_VARARGS | METH_KEYWORDS,
+    {"get_many",        (PyCFunction)(void *)PyVerifiedConstMap_get_many,          METH_VARARGS | METH_KEYWORDS,
      "Look up an iterable of keys; missing keys yield `default` (default None)."},
-    {"to_bytes",   (PyCFunction)PyVerifiedConstMap_serialize,                 METH_NOARGS,
+    {"to_bytes",        (PyCFunction)PyVerifiedConstMap_serialize,                 METH_NOARGS,
      "Serialize the map to bytes."},
-    {"from_bytes", (PyCFunction)PyVerifiedConstMap_deserialize,               METH_O | METH_CLASS,
-     "Deserialize a map from a bytes-like object."},
-    {"save",       (PyCFunction)PyVerifiedConstMap_save,                      METH_O,
+    {"from_bytes",      (PyCFunction)PyVerifiedConstMap_deserialize,               METH_O | METH_CLASS,
+     "Deserialize a map from a bytes-like object (copies the data)."},
+    {"serialized_size", (PyCFunction)PyVerifiedConstMap_serialized_size,           METH_NOARGS,
+     "Number of bytes that to_bytes()/write_into() will produce."},
+    {"write_into",      (PyCFunction)PyVerifiedConstMap_write_into,                METH_O,
+     "Serialize directly into a writable buffer (e.g. SharedMemory.buf). Returns bytes written."},
+    {"from_buffer",     (PyCFunction)PyVerifiedConstMap_from_buffer,               METH_O | METH_CLASS,
+     "Create a zero-copy map that reads directly from a buffer (e.g. shared memory). "
+     "The buffer is kept alive by the returned map and must not be modified while in use."},
+    {"save",            (PyCFunction)PyVerifiedConstMap_save,                      METH_O,
      "Save the map to a file path."},
-    {"load",       (PyCFunction)PyVerifiedConstMap_load,                      METH_O | METH_CLASS,
+    {"load",            (PyCFunction)PyVerifiedConstMap_load,                      METH_O | METH_CLASS,
      "Load a map from a file path."},
-    {"nbytes",     (PyCFunction)PyVerifiedConstMap_nbytes,                    METH_NOARGS,
+    {"nbytes",          (PyCFunction)PyVerifiedConstMap_nbytes,                    METH_NOARGS,
      "Heap size of the data + checks arrays in bytes."},
     {NULL, NULL, 0, NULL}
 };