msgspec-serde 0.1.0__tar.gz → 0.1.2__tar.gz

msgspec_serde-0.1.2/PKG-INFO

@@ -0,0 +1,170 @@
+Metadata-Version: 2.4
+Name: msgspec-serde
+Version: 0.1.2
+Requires-Dist: flatbuffers>=24.3.25
+Requires-Dist: msgspec>=0.19
+Requires-Dist: numpy>=2 ; python_full_version < '3.15'
+Requires-Dist: numpy>=2.5 ; python_full_version >= '3.15'
+License-File: LICENSE
+License-File: src/msgspec_serde/_reflection/LICENSE.txt
+Summary: Performant serialization helpers for msgspec and flatbuffers
+License-Expression: Apache-2.0
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
+
+# msgspec-serde
+
+`msgspec-serde` brings together the performance and flexibility of `msgspec` with other
+common data formats:
+  * Representation of gridded data via `NumPy`-like arrays and views.
+  * The cross-language and schema/IDL support of `FlatBuffers`.
+
+It can generate typed `msgspec.Struct` models and read-only FlatBuffers
+views from `.fbs` schemas.
+
+Its JSON, MessagePack, and FlatBuffers codecs also support very fast encoding/decoding of
+`msgspec.Struct` types containing NumPy arrays (with or without a FlatBuffers schema).
+
+For example, in the [recorded benchmarks](benchmarks.md), the API encodes
+FlatBuffers about 20–22 times faster and decodes them into complete models about
+11–13 times faster than the official Python FlatBuffers API.
+
+## Installation
+
+In a Python 3.12 or newer environment, install the package:
+
+```shell
+python -m pip install msgspec-serde
+```
+
+Generating modules from IDL also requires `flatc`, the FlatBuffers compiler. Install it
+separately and make sure `flatc --version` works in your shell. It is **not** required to:
+  * Import, use, and ser/de of IDL-generated modules.
+  * Import and use JSON/MessagePack ser/de of regular `Struct` containing NumPy arrays.
+
+## Quick start
+
+This takes you through generating `msgspec.Struct` from a FlatBuffers IDL, through ser/de.
+If you're interested in efficient ser/de of regular `Struct` with NumPy arrays, you can skip
+the generation step.
+
+### Generating and subclassing `msgspec.Struct` from IDL
+
+Save this schema as `reading.fbs`:
+
+```fbs
+namespace Example;
+
+table Reading {
+  name:string (required);
+  values:[float] (required);
+}
+
+root_type Reading;
+```
+
+Generate the Python module:
+
+```shell
+msgspec_flatc generate reading.fbs -o generated
+```
+
+This creates `generated/example/reading.py`. The example below subclasses
+`Reading` to reject blank names. Save it as `demo.py` in the same directory as
+`reading.fbs`:
+
+```python
+import numpy as np
+
+from example.reading import Reading, ReadingView
+from msgspec_serde import flatbuffer, json, msgpack
+
+class ValidatedReading(Reading):
+    def __post_init__(self) -> None:
+        if not self.name.strip():
+            raise ValueError("name must not be blank")
+
+
+reading = ValidatedReading(
+    name="latency", values=np.array([1.25, 2.5], dtype=np.float32)
+)
+
+buffer = flatbuffer.encode(reading)
+view = flatbuffer.decode(buffer, type=ReadingView)
+print(view.name, view.values.tolist())
+
+model = view.to_model(ValidatedReading)
+assert isinstance(model, ValidatedReading)
+model.values[0] = 3.0
+assert view.values[0] == 1.25
+
+encoded_json = json.encode(model)
+from_json = json.decode(encoded_json, type=ValidatedReading)
+assert isinstance(from_json, ValidatedReading)
+
+encoded_msgpack = msgpack.encode(model)
+from_msgpack = msgpack.decode(encoded_msgpack, type=ValidatedReading)
+assert isinstance(from_msgpack, ValidatedReading)
+```
+
+Run it from that directory:
+
+```shell
+PYTHONPATH=generated python demo.py
+```
+
+The script prints `latency [1.25, 2.5]`.
+
+A view reads fields on demand from the serialized buffer. Numeric vectors are
+read-only NumPy arrays over that buffer. Calling `to_model(ValidatedReading)`
+copies the data into a mutable model whose arrays own their storage. To decode
+directly into the subclass, use
+`flatbuffer.decode(buffer, type=ValidatedReading)`.
+
+Pass the subclass to `to_model()` or the decoder to run its `__post_init__()`
+validator when constructing the model. Lazy views do not run model validation.
+
+## Features
+
+- Encode NumPy arrays without custom conversion hooks through the
+  [JSON and MessagePack codecs](TUTORIAL.md#8-encode-json-and-messagepack).
+  Reuse `Encoder` and `Decoder` instances when processing many values.
+- [Decode into application subclasses](TUTORIAL.md#7-add-application-validation)
+  and run their `__post_init__()` validation.
+- Vectors of tables with a `(key)` field become dictionaries in models and
+  read-only `TableMap` mappings in views. See
+  [keyed table vectors](TUTORIAL.md#keyed-table-vectors).
+- Read [typed nested FlatBuffers](TUTORIAL.md#10-put-a-typed-flatbuffer-inside-a-byte-vector).
+- [Register additional dynamic payload types](TUTORIAL.md#11-add-open-ended-dynamic-payloads)
+  without changing the envelope schema.
+- Generate models and views for tables, structs, enums, fixed arrays, and
+  unions. Schema `include` statements are supported. See
+  [supported schemas and limitations](TUTORIAL.md#supported-schemas-and-limitations).
+
+## Benchmarks
+
+Selected-field reads are about 1.2 times faster than the official Python
+FlatBuffers API in the recorded workloads. This measures reading a few fields,
+separately from decoding the complete model.
+
+In the recursive 1,000-object benchmark, MessagePack encodes fastest with
+16-value vectors, while FlatBuffers decodes complete models fastest. With
+256-value vectors, FlatBuffers is fastest in both directions.
+
+See [benchmarks.md](benchmarks.md) for charts, methodology, environment details,
+and reproducible commands.
+
+## Tutorial
+
+The [tutorial](TUTORIAL.md) walks through schema generation and the generated
+APIs. The complete runnable example is in
+[examples/tutorial](examples/tutorial).
+
+## Backward compatibility
+
+We aim to keep code generated by an earlier release working with later runtimes
+in the same major version. This is a goal rather than a guarantee.
+Major-version mismatches raise `GeneratedCodeVersionError`. Code generated by a
+later release may not work with an earlier runtime. See
+[runtime version checks and warning controls](TUTORIAL.md#generated-code-compatibility).
+

msgspec_serde-0.1.2/README.md

@@ -0,0 +1,155 @@
+# msgspec-serde
+
+`msgspec-serde` brings together the performance and flexibility of `msgspec` with other
+common data formats:
+  * Representation of gridded data via `NumPy`-like arrays and views.
+  * The cross-language and schema/IDL support of `FlatBuffers`.
+
+It can generate typed `msgspec.Struct` models and read-only FlatBuffers
+views from `.fbs` schemas.
+
+Its JSON, MessagePack, and FlatBuffers codecs also support very fast encoding/decoding of
+`msgspec.Struct` types containing NumPy arrays (with or without a FlatBuffers schema).
+
+For example, in the [recorded benchmarks](benchmarks.md), the API encodes
+FlatBuffers about 20–22 times faster and decodes them into complete models about
+11–13 times faster than the official Python FlatBuffers API.
+
+## Installation
+
+In a Python 3.12 or newer environment, install the package:
+
+```shell
+python -m pip install msgspec-serde
+```
+
+Generating modules from IDL also requires `flatc`, the FlatBuffers compiler. Install it
+separately and make sure `flatc --version` works in your shell. It is **not** required to:
+  * Import, use, and ser/de of IDL-generated modules.
+  * Import and use JSON/MessagePack ser/de of regular `Struct` containing NumPy arrays.
+
+## Quick start
+
+This takes you through generating `msgspec.Struct` from a FlatBuffers IDL, through ser/de.
+If you're interested in efficient ser/de of regular `Struct` with NumPy arrays, you can skip
+the generation step.
+
+### Generating and subclassing `msgspec.Struct` from IDL
+
+Save this schema as `reading.fbs`:
+
+```fbs
+namespace Example;
+
+table Reading {
+  name:string (required);
+  values:[float] (required);
+}
+
+root_type Reading;
+```
+
+Generate the Python module:
+
+```shell
+msgspec_flatc generate reading.fbs -o generated
+```
+
+This creates `generated/example/reading.py`. The example below subclasses
+`Reading` to reject blank names. Save it as `demo.py` in the same directory as
+`reading.fbs`:
+
+```python
+import numpy as np
+
+from example.reading import Reading, ReadingView
+from msgspec_serde import flatbuffer, json, msgpack
+
+class ValidatedReading(Reading):
+    def __post_init__(self) -> None:
+        if not self.name.strip():
+            raise ValueError("name must not be blank")
+
+
+reading = ValidatedReading(
+    name="latency", values=np.array([1.25, 2.5], dtype=np.float32)
+)
+
+buffer = flatbuffer.encode(reading)
+view = flatbuffer.decode(buffer, type=ReadingView)
+print(view.name, view.values.tolist())
+
+model = view.to_model(ValidatedReading)
+assert isinstance(model, ValidatedReading)
+model.values[0] = 3.0
+assert view.values[0] == 1.25
+
+encoded_json = json.encode(model)
+from_json = json.decode(encoded_json, type=ValidatedReading)
+assert isinstance(from_json, ValidatedReading)
+
+encoded_msgpack = msgpack.encode(model)
+from_msgpack = msgpack.decode(encoded_msgpack, type=ValidatedReading)
+assert isinstance(from_msgpack, ValidatedReading)
+```
+
+Run it from that directory:
+
+```shell
+PYTHONPATH=generated python demo.py
+```
+
+The script prints `latency [1.25, 2.5]`.
+
+A view reads fields on demand from the serialized buffer. Numeric vectors are
+read-only NumPy arrays over that buffer. Calling `to_model(ValidatedReading)`
+copies the data into a mutable model whose arrays own their storage. To decode
+directly into the subclass, use
+`flatbuffer.decode(buffer, type=ValidatedReading)`.
+
+Pass the subclass to `to_model()` or the decoder to run its `__post_init__()`
+validator when constructing the model. Lazy views do not run model validation.
+
+## Features
+
+- Encode NumPy arrays without custom conversion hooks through the
+  [JSON and MessagePack codecs](TUTORIAL.md#8-encode-json-and-messagepack).
+  Reuse `Encoder` and `Decoder` instances when processing many values.
+- [Decode into application subclasses](TUTORIAL.md#7-add-application-validation)
+  and run their `__post_init__()` validation.
+- Vectors of tables with a `(key)` field become dictionaries in models and
+  read-only `TableMap` mappings in views. See
+  [keyed table vectors](TUTORIAL.md#keyed-table-vectors).
+- Read [typed nested FlatBuffers](TUTORIAL.md#10-put-a-typed-flatbuffer-inside-a-byte-vector).
+- [Register additional dynamic payload types](TUTORIAL.md#11-add-open-ended-dynamic-payloads)
+  without changing the envelope schema.
+- Generate models and views for tables, structs, enums, fixed arrays, and
+  unions. Schema `include` statements are supported. See
+  [supported schemas and limitations](TUTORIAL.md#supported-schemas-and-limitations).
+
+## Benchmarks
+
+Selected-field reads are about 1.2 times faster than the official Python
+FlatBuffers API in the recorded workloads. This measures reading a few fields,
+separately from decoding the complete model.
+
+In the recursive 1,000-object benchmark, MessagePack encodes fastest with
+16-value vectors, while FlatBuffers decodes complete models fastest. With
+256-value vectors, FlatBuffers is fastest in both directions.
+
+See [benchmarks.md](benchmarks.md) for charts, methodology, environment details,
+and reproducible commands.
+
+## Tutorial
+
+The [tutorial](TUTORIAL.md) walks through schema generation and the generated
+APIs. The complete runnable example is in
+[examples/tutorial](examples/tutorial).
+
+## Backward compatibility
+
+We aim to keep code generated by an earlier release working with later runtimes
+in the same major version. This is a goal rather than a guarantee.
+Major-version mismatches raise `GeneratedCodeVersionError`. Code generated by a
+later release may not work with an earlier runtime. See
+[runtime version checks and warning controls](TUTORIAL.md#generated-code-compatibility).

{msgspec_serde-0.1.0 → msgspec_serde-0.1.2}/pyproject.toml

@@ -1,7 +1,7 @@
 [project]
 name = "msgspec-serde"
-version = "0.1.0"
-description = "Generate lazy FlatBuffers views and msgspec models from FlatBuffers schemas"
+version = "0.1.2"
+description = "Performant serialization helpers for msgspec and flatbuffers"
 readme = "README.md"
 license = "Apache-2.0"
 license-files = ["LICENSE", "src/msgspec_serde/_reflection/LICENSE.txt"]
@@ -9,14 +9,15 @@ requires-python = ">=3.12"
 dependencies = [
     "flatbuffers>=24.3.25",
     "msgspec>=0.19",
-    "numpy>=2",
+    "numpy>=2; python_version < '3.15'",
+    "numpy>=2.5; python_version >= '3.15'",
 ]
 
 [project.scripts]
 msgspec_flatc = "msgspec_serde.cli:main"
 
 [build-system]
-requires = ["maturin>=1.10,<2"]
+requires = ["maturin>=1.15,<2"]
 build-backend = "maturin"
 
 [tool.maturin]
@@ -33,7 +34,7 @@ benchmark = [
     "pyperf>=2.10",
 ]
 upstream-benchmark = [
-    "orjson==3.11.4",
+    "orjson==3.11.6",
     "ormsgpack==1.11.0",
 ]
 dev = [

{msgspec_serde-0.1.0 → msgspec_serde-0.1.2}/rust/Cargo.lock

@@ -186,7 +186,7 @@ dependencies = [
 
 [[package]]
 name = "msgspec-serde-native"
-version = "0.1.0"
+version = "0.1.2"
 dependencies = [
  "bytemuck",
  "flatbuffers",
@@ -264,8 +264,7 @@ dependencies = [
 [[package]]
 name = "numpy"
 version = "0.29.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6a5b15d63a5ff39e378daed0e1340d3a5964703ea9712eb09a0dc66fade996f4"
+source = "git+https://github.com/PyO3/rust-numpy?rev=9df402373716a60f0e7825b6043568abe4e63b46#9df402373716a60f0e7825b6043568abe4e63b46"
 dependencies = [
  "libc",
  "ndarray",

{msgspec_serde-0.1.0 → msgspec_serde-0.1.2}/rust/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "msgspec-serde-native"
-version = "0.1.0"
+version = "0.1.2"
 edition = "2024"
 license = "Apache-2.0"
 description = "Rust serialization backend for msgspec-serde"
@@ -12,8 +12,9 @@ crate-type = ["cdylib"]
 [dependencies]
 bytemuck = { version = "1", features = ["derive"] }
 flatbuffers = "25.12.19"
-numpy = "0.29"
-pyo3 = { version = "0.29", features = ["abi3-py312"] }
+# ABI3T support: https://github.com/PyO3/rust-numpy/pull/556
+numpy = { git = "https://github.com/PyO3/rust-numpy", rev = "9df402373716a60f0e7825b6043568abe4e63b46" }
+pyo3 = { version = "0.29", features = ["abi3-py312", "abi3t-py315"] }
 rmp = "0.8"
 rmp-serde = "1.3"
 rmpv = { version = "1.3", features = ["with-serde"] }

msgspec_serde-0.1.2/rust/src/plan/buffer.rs

@@ -0,0 +1,159 @@
+use super::*;
+
+#[pyclass(module = "msgspec_serde._native", frozen)]
+pub(super) struct NativeBuffer {
+    pub(super) data: Vec<u8>,
+    pub(super) start: usize,
+}
+
+impl NativeBuffer {
+    fn finished_bytes(&self) -> &[u8] {
+        &self.data[self.start..]
+    }
+}
+
+fn includes_flags(flags: c_int, required: c_int) -> bool {
+    flags & required == required
+}
+
+#[pymethods]
+impl NativeBuffer {
+    #[getter]
+    fn _allocation_size(&self) -> usize {
+        self.data.len()
+    }
+
+    unsafe fn __getbuffer__(
+        slf: Bound<'_, Self>,
+        view: *mut ffi::Py_buffer,
+        flags: c_int,
+    ) -> PyResult<()> {
+        if view.is_null() {
+            return Err(PyBufferError::new_err("buffer view is null"));
+        }
+        if includes_flags(flags, ffi::PyBUF_WRITABLE) {
+            return Err(PyBufferError::new_err("native FlatBuffer is read-only"));
+        }
+
+        let owner = slf.borrow();
+        let bytes = owner.finished_bytes();
+        let (data_ptr, length) = (bytes.as_ptr(), bytes.len());
+        drop(owner);
+        // SAFETY: PyO3 provides a valid writable Py_buffer pointer. `slf` is
+        // transferred to `view.obj`, keeping the immutable Vec allocation alive
+        // until CPython releases the exported buffer.
+        unsafe {
+            (*view).obj = slf.into_any().into_ptr();
+            (*view).buf = data_ptr.cast_mut().cast::<c_void>();
+            (*view).len = length as isize;
+            (*view).readonly = 1;
+            (*view).itemsize = 1;
+            (*view).format = if includes_flags(flags, ffi::PyBUF_FORMAT) {
+                c"B".as_ptr().cast_mut()
+            } else {
+                ptr::null_mut()
+            };
+            (*view).ndim = 1;
+            (*view).shape = if includes_flags(flags, ffi::PyBUF_ND) {
+                &raw mut (*view).len
+            } else {
+                ptr::null_mut()
+            };
+            (*view).strides = if includes_flags(flags, ffi::PyBUF_STRIDES) {
+                &raw mut (*view).itemsize
+            } else {
+                ptr::null_mut()
+            };
+            (*view).suboffsets = ptr::null_mut();
+            (*view).internal = ptr::null_mut();
+        }
+        Ok(())
+    }
+
+    unsafe fn __releasebuffer__(&self, _view: *mut ffi::Py_buffer) {}
+}
+
+pub(super) fn extract_bytes(value: &Bound<'_, PyAny>) -> PyResult<Vec<u8>> {
+    if let Ok(bytes) = value.cast::<PyBytes>() {
+        return Ok(bytes.as_bytes().to_vec());
+    }
+    let bytes = value
+        .py()
+        .import("builtins")?
+        .getattr("bytes")?
+        .call1((value,))?;
+    Ok(bytes.cast::<PyBytes>()?.as_bytes().to_vec())
+}
+
+pub(super) fn nonnegative_usize(value: isize, name: &str) -> PyResult<usize> {
+    usize::try_from(value)
+        .map_err(|_| PyValueError::new_err(format!("{name} must be greater than or equal to zero")))
+}
+
+pub(super) fn validate_identifier(identifier: Option<&str>) -> PyResult<()> {
+    if let Some(value) = identifier
+        && (value.len() != 4 || !value.is_ascii())
+    {
+        return Err(PyValueError::new_err(
+            "FlatBuffers file identifiers must contain four ASCII bytes",
+        ));
+    }
+    Ok(())
+}
+
+pub(super) fn buffer_byte_length(value: &Bound<'_, PyAny>) -> PyResult<usize> {
+    Ok(PyUntypedBuffer::get(value)?.len_bytes())
+}
+
+pub(super) fn checked_byte_buffer(value: &Bound<'_, PyAny>) -> PyResult<PyBuffer<u8>> {
+    let buffer = PyBuffer::<u8>::get(value)?;
+    check_byte_buffer(&buffer)?;
+    Ok(buffer)
+}
+
+pub(super) fn check_byte_buffer(buffer: &PyBuffer<u8>) -> PyResult<()> {
+    if buffer.dimensions() != 1 || !buffer.is_c_contiguous() {
+        return Err(PyTypeError::new_err(
+            "byte vector fields must be one-dimensional and C-contiguous",
+        ));
+    }
+    Ok(())
+}
+
+pub(super) fn immutable_buffer_subslice<'a>(
+    buffer: &PyBuffer<u8>,
+    owner: &'a [u8],
+) -> PyResult<&'a [u8]> {
+    let start = (buffer.buf_ptr() as usize)
+        .checked_sub(owner.as_ptr() as usize)
+        .ok_or_else(|| PyBufferError::new_err("buffer lies outside its immutable owner"))?;
+    let end = start
+        .checked_add(buffer.len_bytes())
+        .filter(|&end| end <= owner.len())
+        .ok_or_else(|| PyBufferError::new_err("buffer lies outside its immutable owner"))?;
+    Ok(&owner[start..end])
+}
+
+#[inline]
+pub(super) fn with_input_bytes<T>(
+    value: &Bound<'_, PyAny>,
+    decode: impl Fn(&[u8]) -> PyResult<T>,
+) -> PyResult<T> {
+    if let Ok(value) = value.cast::<PyBytes>() {
+        return decode(value.as_bytes());
+    }
+    if let Ok(view) = value.cast::<PyMemoryView>() {
+        let owner = view.getattr("obj")?;
+        if let Ok(owner) = owner.cast::<PyBytes>() {
+            let buffer = checked_byte_buffer(value)?;
+            return decode(immutable_buffer_subslice(&buffer, owner.as_bytes())?);
+        }
+        if let Ok(owner) = owner.cast::<NativeBuffer>() {
+            let buffer = checked_byte_buffer(value)?;
+            let owner = owner.borrow();
+            return decode(immutable_buffer_subslice(&buffer, owner.finished_bytes())?);
+        }
+    }
+    let data = extract_bytes(value)?;
+    decode(&data)
+}