lightstream-io 0.5.0__tar.gz

lightstream_io-0.5.0/PKG-INFO

@@ -0,0 +1,268 @@
+Metadata-Version: 2.4
+Name: lightstream-io
+Version: 0.5.0
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Rust
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: Software Development :: Libraries
+Requires-Dist: minarrow>=0.16
+Requires-Dist: pytest>=7.0 ; extra == 'dev'
+Requires-Dist: pyarrow>=14.0 ; extra == 'dev'
+Requires-Dist: polars>=1.0 ; extra == 'dev'
+Requires-Dist: duckdb>=1.0 ; extra == 'dev'
+Provides-Extra: dev
+Summary: Streaming Arrow I/O for Python - files, sockets, and network transports with zero-copy minarrow interop.
+Keywords: arrow,streaming,minarrow,ipc,parquet,quic
+Home-Page: https://github.com/SpaceCell/lightstream
+Author: Peter Bower
+License-Expression: MPL-2.0
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
+Project-URL: Homepage, https://github.com/SpaceCell/lightstream
+Project-URL: Repository, https://github.com/SpaceCell/lightstream
+
+# lightstream
+
+Move Arrow tables between processes, services and storage from Python without adding a gRPC stack or writing transport-specific framing.
+
+`lightstream` provides one streaming API across files, memory maps, sockets and network transports. Readers return [`minarrow`](https://github.com/pbower/minarrow) objects and implement the Arrow PyCapsule protocol, allowing PyArrow, Polars, DuckDB and other Arrow-compatible libraries to consume data **straight off the wire** without an intermediate conversion.
+
+## Installation
+
+```bash
+pip install lightstream-io
+```
+
+## Usage
+
+Everything is `read` or `write`.
+
+The URI selects the transport, `protocol` selects the wire framing, and file extensions select the storage format.
+
+```python
+import lightstream as ls
+```
+
+### Files
+
+Read Arrow IPC, Parquet, CSV and JSON through the same interface.
+
+```python
+# Read an entire dataset.
+table = ls.read("quotes.arrow").read_all()
+
+# Stream batches without loading the complete dataset into memory.
+for batch in ls.read("large.parquet"):
+    process(batch)
+
+# Write any Arrow-compatible object.
+with ls.write("output.parquet", compression="zstd") as writer:
+    writer.write(table)
+```
+
+Arrow IPC readers use memory mapping and out-of-core routing where appropriate.
+
+Readers yield `minarrow.Table` batches. `read_all()` returns a `Table` when the result is contiguous or a `ChunkedTable` when multiple chunks must be preserved.
+
+### Network transports
+
+The same interface streams raw Arrow IPC over TCP, Unix domain sockets, WebSocket, HTTP, QUIC, WebTransport and standard I/O.
+
+```python
+for batch in ls.read("tcp://feed.example.com:9000"):
+    process(batch)
+```
+
+Supported URI schemes include:
+
+| Transport             | URI                     |
+| --------------------- | ----------------------- |
+| TCP                   | `tcp://host:port`       |
+| Unix domain socket    | `uds:///path/to/socket` |
+| WebSocket             | `ws://host/path`        |
+| Secure WebSocket      | `wss://host/path`       |
+| HTTP                  | `http://host/path`      |
+| HTTPS                 | `https://host/path`     |
+| QUIC                  | `quic://host:port`      |
+| WebTransport          | `wt://host/path`        |
+| Standard input/output | `stdio://`              |
+
+### Accepting connections
+
+Either endpoint can accept the connection.
+
+Set `accept=True` to bind the endpoint on first use and block until the connecting peer arrives.
+
+```python
+writer = ls.write(
+    "uds:///tmp/feed.sock",
+    accept=True,
+)
+
+writer.write(table)
+writer.close()
+```
+
+The listener remains available for the lifetime of the process. Connections arriving while a serving loop is active wait in the listener backlog, allowing the accepting endpoint to operate as a persistent server.
+
+### TLS
+
+QUIC and WebTransport include TLS at the protocol level. The accepting endpoint presents a PEM certificate and private key, while the connecting endpoint verifies the certificate using PEM roots.
+
+```python
+writer = ls.write(
+    "quic://0.0.0.0:4433",
+    accept=True,
+    tls_cert="server.pem",
+    tls_key="server-key.pem",
+)
+
+for batch in ls.read(
+    "quic://feed.example.com:4433",
+    tls_ca="roots.pem",
+):
+    process(batch)
+```
+
+Secure WebSocket and HTTPS transports use their corresponding TLS-enabled URI schemes.
+
+## Arrow interoperability
+
+Every reader implements the Arrow PyCapsule stream protocol.
+
+Arrow-compatible libraries can therefore consume a Lightstream reader directly.
+
+```python
+import duckdb
+import lightstream as ls
+
+reader = ls.read("quotes.arrow")
+
+result = duckdb.sql(
+    "SELECT SUM(qty) FROM reader"
+)
+```
+
+The same reader can be passed to libraries such as Polars without first converting its batches into Python objects.
+
+## Lightstream protocol
+
+The Lightstream protocol multiplexes named Arrow tables and opaque messages over one connection.
+
+Opaque payloads can carry formats such as Protobuf or MessagePack, while table channels retain their Arrow schema and batch representation.
+
+```python
+reader = ls.read(
+    "uds:///tmp/feed.sock",
+    protocol="lightstream",
+)
+
+reader.register_table(
+    "quotes",
+    representative_table,
+)
+reader.register_message("health")
+
+for frame in reader:
+    if frame.is_table():
+        on_quotes(frame.table)
+    else:
+        on_health(frame.payload)
+```
+
+`frame.table` is a `minarrow.Table`. Message payloads are returned as `bytes`.
+
+Both peers must register compatible channels before exchanging frames.
+
+The protocol is transport-independent, so the same table and message definitions can be used over TCP, Unix domain sockets, WebSocket, HTTP, QUIC or WebTransport.
+
+## Standard I/O pipelines
+
+The standard I/O transport can stream Arrow IPC or line-oriented text.
+
+Set `format="csv"` to write CSV records or `format="json"` to write NDJSON. Text emitted by another process is decoded back into table batches on read.
+
+This allows Unix tools, command-line programs and agents to participate directly in a table pipeline.
+
+```bash
+python examples/run-example.py sed
+python examples/run-example.py jq
+python examples/run-example.py claude
+python examples/run-example.py sql
+python examples/run-example.py debug
+```
+
+The pipeline examples demonstrate:
+
+* `sed` rewriting rows in flight
+* `jq` filtering NDJSON and returning CSV
+* an agent diagnosing invalid batches
+* rolling DuckDB SQL over a live stream
+* `tail -f`-style table inspection and pretty-printing
+
+## Transport examples
+
+Worked examples are provided under `examples/transport/`.
+
+Each transport includes:
+
+* a Python server
+* a Rust server
+* a Python client
+* the same million-row input table
+
+Use the example router to select the transport and backend:
+
+```bash
+# Python backend over TCP.
+python examples/run-example.py tcp
+
+# Rust backend over QUIC.
+python examples/run-example.py quic --rust
+```
+
+Available transports are:
+
+```text
+tcp
+uds
+ws
+wss
+http
+https
+quic
+wt
+stdio
+```
+
+TLS examples generate a private root certificate and a server certificate for local execution.
+
+## Building from source
+
+Lightstream currently requires the Rust nightly toolchain. The repository selects the required toolchain automatically through `rust-toolchain.toml`.
+
+```bash
+pip install maturin
+maturin develop
+pytest tests/
+```
+
+## Licence
+
+Copyright © 2025–2026 Peter Garfield Bower.
+
+Licensed under the Mozilla Public License 2.0. See `LICENSE` for the standard terms.
+
+See [MPL 2.0 FAQ](https://www.mozilla.org/en-US/MPL/2.0/FAQ/) if you are unfamiliar with this open-source license.
+
+## Affiliation notice
+
+Lightstream is not affiliated with Apache Arrow or the Apache Software Foundation.
+
+It implements public Arrow formats through Minarrow and interoperates with the Arrow ecosystem through the Arrow PyCapsule protocol.
+
+`lightstream` is maintained by [SpaceCell](https://spacecell.com) and forms part of its open-source foundation for high-performance data computing.
+

lightstream_io-0.5.0/README.md

@@ -0,0 +1,241 @@
+# lightstream
+
+Move Arrow tables between processes, services and storage from Python without adding a gRPC stack or writing transport-specific framing.
+
+`lightstream` provides one streaming API across files, memory maps, sockets and network transports. Readers return [`minarrow`](https://github.com/pbower/minarrow) objects and implement the Arrow PyCapsule protocol, allowing PyArrow, Polars, DuckDB and other Arrow-compatible libraries to consume data **straight off the wire** without an intermediate conversion.
+
+## Installation
+
+```bash
+pip install lightstream-io
+```
+
+## Usage
+
+Everything is `read` or `write`.
+
+The URI selects the transport, `protocol` selects the wire framing, and file extensions select the storage format.
+
+```python
+import lightstream as ls
+```
+
+### Files
+
+Read Arrow IPC, Parquet, CSV and JSON through the same interface.
+
+```python
+# Read an entire dataset.
+table = ls.read("quotes.arrow").read_all()
+
+# Stream batches without loading the complete dataset into memory.
+for batch in ls.read("large.parquet"):
+    process(batch)
+
+# Write any Arrow-compatible object.
+with ls.write("output.parquet", compression="zstd") as writer:
+    writer.write(table)
+```
+
+Arrow IPC readers use memory mapping and out-of-core routing where appropriate.
+
+Readers yield `minarrow.Table` batches. `read_all()` returns a `Table` when the result is contiguous or a `ChunkedTable` when multiple chunks must be preserved.
+
+### Network transports
+
+The same interface streams raw Arrow IPC over TCP, Unix domain sockets, WebSocket, HTTP, QUIC, WebTransport and standard I/O.
+
+```python
+for batch in ls.read("tcp://feed.example.com:9000"):
+    process(batch)
+```
+
+Supported URI schemes include:
+
+| Transport             | URI                     |
+| --------------------- | ----------------------- |
+| TCP                   | `tcp://host:port`       |
+| Unix domain socket    | `uds:///path/to/socket` |
+| WebSocket             | `ws://host/path`        |
+| Secure WebSocket      | `wss://host/path`       |
+| HTTP                  | `http://host/path`      |
+| HTTPS                 | `https://host/path`     |
+| QUIC                  | `quic://host:port`      |
+| WebTransport          | `wt://host/path`        |
+| Standard input/output | `stdio://`              |
+
+### Accepting connections
+
+Either endpoint can accept the connection.
+
+Set `accept=True` to bind the endpoint on first use and block until the connecting peer arrives.
+
+```python
+writer = ls.write(
+    "uds:///tmp/feed.sock",
+    accept=True,
+)
+
+writer.write(table)
+writer.close()
+```
+
+The listener remains available for the lifetime of the process. Connections arriving while a serving loop is active wait in the listener backlog, allowing the accepting endpoint to operate as a persistent server.
+
+### TLS
+
+QUIC and WebTransport include TLS at the protocol level. The accepting endpoint presents a PEM certificate and private key, while the connecting endpoint verifies the certificate using PEM roots.
+
+```python
+writer = ls.write(
+    "quic://0.0.0.0:4433",
+    accept=True,
+    tls_cert="server.pem",
+    tls_key="server-key.pem",
+)
+
+for batch in ls.read(
+    "quic://feed.example.com:4433",
+    tls_ca="roots.pem",
+):
+    process(batch)
+```
+
+Secure WebSocket and HTTPS transports use their corresponding TLS-enabled URI schemes.
+
+## Arrow interoperability
+
+Every reader implements the Arrow PyCapsule stream protocol.
+
+Arrow-compatible libraries can therefore consume a Lightstream reader directly.
+
+```python
+import duckdb
+import lightstream as ls
+
+reader = ls.read("quotes.arrow")
+
+result = duckdb.sql(
+    "SELECT SUM(qty) FROM reader"
+)
+```
+
+The same reader can be passed to libraries such as Polars without first converting its batches into Python objects.
+
+## Lightstream protocol
+
+The Lightstream protocol multiplexes named Arrow tables and opaque messages over one connection.
+
+Opaque payloads can carry formats such as Protobuf or MessagePack, while table channels retain their Arrow schema and batch representation.
+
+```python
+reader = ls.read(
+    "uds:///tmp/feed.sock",
+    protocol="lightstream",
+)
+
+reader.register_table(
+    "quotes",
+    representative_table,
+)
+reader.register_message("health")
+
+for frame in reader:
+    if frame.is_table():
+        on_quotes(frame.table)
+    else:
+        on_health(frame.payload)
+```
+
+`frame.table` is a `minarrow.Table`. Message payloads are returned as `bytes`.
+
+Both peers must register compatible channels before exchanging frames.
+
+The protocol is transport-independent, so the same table and message definitions can be used over TCP, Unix domain sockets, WebSocket, HTTP, QUIC or WebTransport.
+
+## Standard I/O pipelines
+
+The standard I/O transport can stream Arrow IPC or line-oriented text.
+
+Set `format="csv"` to write CSV records or `format="json"` to write NDJSON. Text emitted by another process is decoded back into table batches on read.
+
+This allows Unix tools, command-line programs and agents to participate directly in a table pipeline.
+
+```bash
+python examples/run-example.py sed
+python examples/run-example.py jq
+python examples/run-example.py claude
+python examples/run-example.py sql
+python examples/run-example.py debug
+```
+
+The pipeline examples demonstrate:
+
+* `sed` rewriting rows in flight
+* `jq` filtering NDJSON and returning CSV
+* an agent diagnosing invalid batches
+* rolling DuckDB SQL over a live stream
+* `tail -f`-style table inspection and pretty-printing
+
+## Transport examples
+
+Worked examples are provided under `examples/transport/`.
+
+Each transport includes:
+
+* a Python server
+* a Rust server
+* a Python client
+* the same million-row input table
+
+Use the example router to select the transport and backend:
+
+```bash
+# Python backend over TCP.
+python examples/run-example.py tcp
+
+# Rust backend over QUIC.
+python examples/run-example.py quic --rust
+```
+
+Available transports are:
+
+```text
+tcp
+uds
+ws
+wss
+http
+https
+quic
+wt
+stdio
+```
+
+TLS examples generate a private root certificate and a server certificate for local execution.
+
+## Building from source
+
+Lightstream currently requires the Rust nightly toolchain. The repository selects the required toolchain automatically through `rust-toolchain.toml`.
+
+```bash
+pip install maturin
+maturin develop
+pytest tests/
+```
+
+## Licence
+
+Copyright © 2025–2026 Peter Garfield Bower.
+
+Licensed under the Mozilla Public License 2.0. See `LICENSE` for the standard terms.
+
+See [MPL 2.0 FAQ](https://www.mozilla.org/en-US/MPL/2.0/FAQ/) if you are unfamiliar with this open-source license.
+
+## Affiliation notice
+
+Lightstream is not affiliated with Apache Arrow or the Apache Software Foundation.
+
+It implements public Arrow formats through Minarrow and interoperates with the Arrow ecosystem through the Arrow PyCapsule protocol.
+
+`lightstream` is maintained by [SpaceCell](https://spacecell.com) and forms part of its open-source foundation for high-performance data computing.

lightstream_io-0.5.0/pyproject.toml

@@ -0,0 +1,35 @@
+[build-system]
+requires = ["maturin>=1.0,<2.0"]
+build-backend = "maturin"
+
+[project]
+name = "lightstream-io"
+version = "0.5.0"
+description = "Streaming Arrow I/O for Python - files, sockets, and network transports with zero-copy minarrow interop."
+readme = "README.md"
+requires-python = ">=3.9"
+license = "MPL-2.0"
+authors = [{ name = "Peter Bower" }]
+keywords = ["arrow", "streaming", "minarrow", "ipc", "parquet", "quic"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Rust",
+    "Topic :: Scientific/Engineering",
+    "Topic :: Software Development :: Libraries",
+]
+dependencies = ["minarrow>=0.16"]
+
+[project.urls]
+Homepage = "https://github.com/SpaceCell/lightstream"
+Repository = "https://github.com/SpaceCell/lightstream"
+
+[project.optional-dependencies]
+dev = ["pytest>=7.0", "pyarrow>=14.0", "polars>=1.0", "duckdb>=1.0"]
+
+[tool.maturin]
+features = ["pyo3/extension-module"]
+module-name = "lightstream"
+manifest-path = "python/Cargo.toml"