omextra 0.0.0.dev503__tar.gz → 0.0.0.dev505__tar.gz

{omextra-0.0.0.dev503/omextra.egg-info → omextra-0.0.0.dev505}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: omextra
-Version: 0.0.0.dev503
+Version: 0.0.0.dev505
 Summary: omextra
 Author: wrmsr
 License-Expression: BSD-3-Clause
@@ -14,7 +14,7 @@ Classifier: Programming Language :: Python :: 3.13
 Requires-Python: >=3.13
 Description-Content-Type: text/markdown
 License-File: LICENSE
-Requires-Dist: omlish==0.0.0.dev503
+Requires-Dist: omlish==0.0.0.dev505
 Dynamic: license-file
 
 # Overview

omextra-0.0.0.dev505/omextra/io/buffers/DESIGN.md

@@ -0,0 +1,253 @@
+# Buffer System Design Notes — Historical Architecture Summary
+
+This document captures the design rationale, constraints, and evolution of a **general-purpose byte buffer subsystem**
+intended to support a **Netty-like, pipeline-oriented protocol toolkit in Python**. It is meant to serve as durable
+architectural context for future development and onboarding, not as API documentation.
+
+---
+
+## 1. Problem Statement & Goals
+
+The goal was to design a **foundational buffer layer** suitable for building protocol stacks in Python with the
+following constraints:
+
+- **Pure Python, no external dependencies**, compatible with Python 3.8+
+- **Embeddable**: usable inside arbitrary runtimes (sync, async, fork-only, threaded, event-loop driven, or none at all)
+- **Independent of asyncio / async-await**, but adaptable to them
+- **Composable** and **pipeline-friendly**, favoring small chained transforms over large monolithic objects
+- **Correctness and resilience first** (OOM avoidance, bounded growth, clear error signaling)
+- **Low copy where possible**, but never at the expense of clarity or safety
+- Explicitly **not performance-maximal**; Python-appropriate efficiency is sufficient
+
+The buffer layer is treated as *infrastructure*: it must be robust enough that higher-level protocol code never needs to
+second-guess its behavior.
+
+---
+
+## 2. Design Philosophy
+
+### Composition over inheritance Rather than building large stateful readers/writers, the design favors:
+- Small buffer objects with explicit operations (`find`, `split_to`, `coalesce`, etc.)
+- Stateless helper functions layered atop buffers (binary reads, framing, decoding)
+- Pipeline codecs that operate purely on buffers + views
+
+### Explicit boundaries
+- **Copy boundaries are explicit** (`tobytes`, `read_bytes`)
+- **Mutation vs. presentation** is clearly distinguished
+- **Ephemeral vs. stable views** are explicitly defined
+
+### “Everything needs a timeout / limit” All operations that can grow memory unboundedly must be limitable:
+- Per-buffer `max_bytes`
+- Per-frame `max_size`
+- Clear error types when limits are exceeded
+
+---
+
+## 3. Core Abstractions
+
+### BytesBuffer (conceptual) A readable byte container with:
+- Logical length (`__len__`)
+- Search (`find`, `rfind`) — *stream-correct*, across internal segmentation
+- Non-consuming inspection (`peek`, `segments`)
+- Consuming operations (`advance`, `split_to`)
+- Contiguity guarantee (`coalesce(n)`)
+
+Buffers are **not sequences** and are **not random-access containers** in the general sense:
+- They may support indexed access incidentally
+- But they are optimized for *prefix-oriented, streaming access*
+- They are not intended for arbitrary slicing or mutation
+
+---
+
+## 4. Two Concrete Buffer Backends
+
+### SegmentedBytesBuffer A list-of-segments design (bytes / bytearray chunks):
+
+**Strengths**
+- Avoids pathological “large buffer pinned by tiny tail”
+- Stable zero-copy views for `split_to`
+- Natural fit for network chunking and streaming
+
+**Tradeoffs**
+- Search and coalescing require careful logic
+- Segment count must be managed (coalescing / heuristics)
+
+### LinearBytesBuffer A single `bytearray` + read/write indices:
+
+**Strengths**
+- Fast scanning and header parsing
+- Naturally contiguous prefix
+- Efficient `reserve` / `commit`
+
+**Tradeoffs**
+- `split_to` must copy to keep views stable
+- Needs compaction heuristics to avoid growth from head advancement
+
+These two backends intentionally cover different workload shapes; both conform to the same conceptual interface.
+
+---
+
+## 5. Views & Lifetime Rules
+
+### BytesView / SegmentedBytesView Objects returned from `split_to`:
+- Represent bytes **removed** from the buffer
+- Must remain **stable forever**
+- May internally reference original segments or copies
+
+### Ephemeral views Returned from:
+- `peek`
+- `segments`
+- `coalesce`
+
+Rules:
+- Valid only until the next buffer mutation
+- Never safe to hold across writes / advances
+- Always exposed as `memoryview`
+
+This mirrors real-world behavior in systems like Netty (retained slices) but without refcounting.
+
+---
+
+## 6. Coalescing Semantics (Key Design Point)
+
+### `coalesce(n)` Guarantees the first `n` readable bytes are contiguous.
+
+- **Non-consuming**
+- May restructure internal storage
+- Copies *only if necessary*, and only the minimal prefix
+- Disallowed while a reservation is outstanding
+
+Rationale:
+- Python lacks efficient per-byte iteration
+- Many operations (binary parsing, decoding) require contiguity
+- Copying should happen *close to storage*, not in higher-level code
+
+Unlike Netty or Tokio (where coalescing is implicit or pattern-based), this is an explicit primitive.
+
+---
+
+## 7. Reservation Model (`reserve` / `commit`)
+
+Designed to support:
+- Zero-copy reads from drivers
+- Predictable memory behavior
+
+Rules:
+- Only one outstanding reservation at a time
+- While reserved:
+  - No reshaping operations allowed
+  - No coalescing, splitting, advancing, or writing
+- `commit(n)` appends exactly `n` bytes
+- Reservation buffers are **temporary**, not views into live storage
+  - Avoids Python `BufferError` pinning issues
+
+This design was informed directly by pitfalls discovered when using `BytesIO.getbuffer()`.
+
+---
+
+## 8. Limits & Error Taxonomy
+
+### Core error types
+- `NeedMoreData`: insufficient bytes, retry later
+- `BufferTooLarge`: buffer growth exceeded cap
+- `FrameTooLarge`: single frame exceeded size limit
+- `OutstandingReserve` / `NoOutstandingReserve`: invalid state transitions
+
+Design choice:
+- Limit errors subclass `ValueError`
+- State errors subclass `RuntimeError`
+- Preserves compatibility while enabling semantic distinction
+
+### Limits
+- `max_bytes` on buffers (optional, default None)
+- `max_size` on framers/codecs
+
+These limits are enforced eagerly to prevent memory exhaustion.
+
+---
+
+## 9. Framing & Search
+
+### Longest-match delimiter framing A dedicated codec layer implements:
+- Overlapping delimiters (`\r` vs `\r\n`)
+- Longest-match semantics
+- Deferred emission when ambiguity exists
+- Explicit `final=True` flush at EOF
+
+Key insight: > Delimiter resolution must live *above* the buffer but *below* protocol logic.
+
+The buffer’s `find/rfind` remain simple, single-needle primitives; framing logic resolves ambiguity.
+
+---
+
+## 10. Binary Read Helpers
+
+Implemented as **pure functions atop buffers**, not methods:
+- `peek_exact`
+- `take`
+- `read_bytes`
+- `read_u8`, `read_u16_be`, `read_u32_le`, etc.
+
+All rely on:
+- `coalesce(n)` for contiguity
+- `advance(n)` for consumption
+- `NeedMoreData` for retry signaling
+
+This keeps the buffer surface small while enabling rich protocol parsing.
+
+---
+
+## 11. File-Like Adapters & Interop
+
+Adapters exist to bridge:
+- File-like objects → buffers
+- Buffers → file-like readers/writers
+
+Key properties:
+- Policy-driven behavior (`raise`, `return_partial`, `block`)
+- No reliance on `io` abstractions in the core
+- Explicit handling of `BytesIO.getbuffer()` pinning hazards
+
+Interop is intentionally *ugly but isolated*; the core remains clean.
+
+---
+
+## 12. Relation to Other Ecosystems
+
+### Netty
+- Netty’s buffer complexity is split across many types (heap, direct, composite)
+- This design captures the *behavioral essence* (segmentation, coalescing, stable views) without refcounting
+
+### Tokio / Rust
+- Similar semantics exist implicitly (`Buf::chunk`, `advance`)
+- Python benefits from explicit coalescing due to higher call overhead
+
+### Twisted / asyncio
+- Older designs rely on callbacks and file-like objects
+- This system is explicitly designed *post-async/await*, but not dependent on it
+
+---
+
+## 13. What This Enables Next
+
+With the buffer layer stabilized, higher-level work can proceed safely:
+- Length-prefixed framing
+- HTTP/1 streaming parsing
+- Binary protocol codecs
+- Pipeline lifecycle (EOF, errors, close)
+- Transport drivers (async, sync, custom)
+
+The buffer layer is now considered **foundationally complete**: additional features should be justified by concrete
+protocol needs, not speculation.
+
+---
+
+## 14. Key Takeaway
+
+This buffer system is intentionally:
+- **Explicit**, not magical
+- **Predictable**, not clever
+- **Composable**, not monolithic
+
+It trades a small amount of convenience for long-term correctness and clarity — exactly what is needed to support a
+robust, Netty-like protocol toolkit in Python.

omextra-0.0.0.dev505/omextra/io/buffers/adapters.py

@@ -0,0 +1,294 @@
+# ruff: noqa: UP006 UP007 UP045
+# @omlish-lite
+import io
+import time
+import typing as ta
+
+from .errors import NeedMoreData
+from .segmented import SegmentedBytesView
+from .types import BytesLike
+from .types import BytesView
+from .types import BytesViewLike
+
+
+##
+
+
+class FileLikeRawBytesReader:
+    """
+    Adapter: file-like object -> RawBytesReader-style `read1`.
+
+    This is intentionally permissive: it duck-types common file-like APIs.
+
+    Notes:
+      - If the object has `read1`, we use it.
+      - Otherwise we fall back to `read`.
+      - This is a *raw* reader: it makes no buffering guarantees beyond what the wrapped object provides.
+    """
+
+    def __init__(self, f: ta.Any) -> None:
+        super().__init__()
+
+        self._f = f
+
+    def read1(self, n: int = -1, /) -> bytes:
+        f = self._f
+        if hasattr(f, 'read1'):
+            return ta.cast(bytes, f.read1(n))
+        return ta.cast(bytes, f.read(n))
+
+
+class FileLikeBufferedBytesReader(FileLikeRawBytesReader):
+    """
+    Adapter: file-like object -> BufferedBytesReader-style `read/readall`.
+
+    Notes:
+      - Uses `readall` if present; otherwise uses `read()` with `-1`.
+      - Does not impose additional buffering; it reflects the wrapped object's behavior.
+    """
+
+    def read(self, n: int = -1, /) -> bytes:
+        return ta.cast(bytes, self._f.read(n))
+
+    def readall(self) -> bytes:
+        f = self._f
+        if hasattr(f, 'readall'):
+            return ta.cast(bytes, f.readall())
+        return ta.cast(bytes, f.read())
+
+
+class BytesBufferReaderAdapter:
+    """
+    Adapter: BytesBuffer -> file-like reader methods (`read1`, `read`, `readall`).
+
+    This adapter is policy-driven for how it behaves when insufficient bytes are available. The core buffer is
+    intentionally non-blocking; blocking behavior (if desired) must be provided via a `fill` callback that supplies more
+    bytes into the buffer.
+
+    `policy`:
+      - 'raise': raise NeedMoreData if fewer than `n` bytes are available (for n >= 0).
+      - 'return_partial': return whatever is available (possibly b'') up to `n`.
+      - 'block': repeatedly call `fill()` until satisfied or until `fill()` signals EOF.
+
+    `fill`:
+      - Callable that writes more bytes into the underlying MutableBytesBuffer and returns:
+          * True  -> made progress / more data may be available
+          * False -> EOF / no more data will arrive
+      - Only used when policy == 'block'.
+
+    This adapter exists for interop with legacy code that expects file-like objects, but the core design remains
+    independent from `io` and blocking semantics.
+    """
+
+    def __init__(
+            self,
+            buf: ta.Any,
+            *,
+            policy: ta.Literal['raise', 'return_partial', 'block'] = 'raise',
+            fill: ta.Optional[ta.Callable[[], bool]] = None,
+            block_sleep: ta.Union[ta.Callable[[], None], float, None] = None,
+    ) -> None:
+        super().__init__()
+
+        self._buf = buf
+        self._policy = policy
+        self._fill = fill
+        self._block_sleep = block_sleep
+
+        if self._policy == 'block' and self._fill is None:
+            raise ValueError('policy=block requires fill')
+
+    def _on_block(self) -> None:
+        if (bs := self._block_sleep) is None:
+            return
+        elif callable(bs):
+            bs()
+        else:
+            time.sleep(bs)
+
+    def read1(self, n: int = -1, /) -> bytes:
+        return self.read(n)
+
+    def read(self, n: int = -1, /) -> bytes:
+        buf = self._buf
+
+        if n is None or n < 0:
+            return self.readall()
+
+        if n == 0:
+            return b''
+
+        while True:
+            ln = len(buf)
+            if ln >= n:
+                v = buf.split_to(n)
+                return ta.cast(bytes, v.tobytes())
+
+            if self._policy == 'return_partial':
+                if ln == 0:
+                    return b''
+                v = buf.split_to(ln)
+                return ta.cast(bytes, v.tobytes())
+
+            if self._policy == 'raise':
+                raise NeedMoreData
+
+            # block
+            if not ta.cast('ta.Callable[[], bool]', self._fill)():
+                # EOF
+                if ln == 0:
+                    return b''
+                v = buf.split_to(ln)
+                return ta.cast(bytes, v.tobytes())
+
+            self._on_block()
+
+    def readall(self) -> bytes:
+        buf = self._buf
+        parts: ta.List[bytes] = []
+
+        while True:
+            ln = len(buf)
+            if ln:
+                v = buf.split_to(ln)
+                parts.append(ta.cast(bytes, v.tobytes()))
+                continue
+
+            if self._policy == 'block':
+                if not ta.cast('ta.Callable[[], bool]', self._fill)():
+                    break
+                self._on_block()
+                continue
+
+            break
+
+        return b''.join(parts)
+
+
+class BytesBufferWriterAdapter:
+    """
+    Adapter: file-like writer sink <- BytesBuffer / bytes-like.
+
+    This is intentionally small and dumb: it exists to bridge into code expecting a `.write(...)`
+    method on an object.
+
+    If given a BytesView-like object, it writes segment-by-segment to avoid materializing copies
+    when the sink can accept multiple writes efficiently.
+    """
+
+    def __init__(self, f: ta.Any) -> None:
+        super().__init__()
+
+        self._f = f
+
+    def write(self, data: ta.Any) -> int:
+        f = self._f
+
+        if isinstance(data, (bytes, bytearray, memoryview)):
+            b = data.tobytes() if isinstance(data, memoryview) else bytes(data)
+            return ta.cast(int, f.write(b))
+
+        if isinstance(data, BytesViewLike):
+            total = 0
+            for mv in data.segments():
+                total += ta.cast(int, f.write(bytes(mv)))
+            return total
+
+        if isinstance(data, BytesView):
+            b = data.tobytes()
+            return ta.cast(int, f.write(b))
+
+        raise TypeError(data)
+
+
+class BytesIoBytesBuffer:
+    """
+    BytesBuffer/MutableBytesBuffer implementation backed by `io.BytesIO`, using `getbuffer()`.
+
+    This exists primarily for interoperability with code that already produces/consumes `BytesIO`,
+    and to demonstrate how `getbuffer()` can expose a non-copying `memoryview` of internal storage.
+
+    Caveat (important):
+      - Any exported `memoryview` from `BytesIO.getbuffer()` can pin the BytesIO against resizing.
+      - If a caller holds onto a view and we attempt to grow/resize internally, `BytesIO` may raise
+        `BufferError`. We surface that as a RuntimeError.
+
+    This backing is therefore best suited for controlled scenarios; it is *not* the recommended
+    default buffer backend for pynetty (segmented/bytearray backends are more predictable).
+    """
+
+    def __init__(self) -> None:
+        super().__init__()
+
+        self._bio = io.BytesIO()
+        self._rpos = 0
+
+        # reserve/commit state
+        self._resv: ta.Optional[bytearray] = None
+        self._resv_len = 0
+
+    def __len__(self) -> int:
+        return self._bio.getbuffer().nbytes - self._rpos
+
+    def peek(self) -> memoryview:
+        mv = self._bio.getbuffer()
+        return mv[self._rpos:]
+
+    def segments(self) -> ta.Sequence[memoryview]:
+        mv = self._bio.getbuffer()
+        seg = mv[self._rpos:]
+        return (seg,) if len(seg) else ()
+
+    def advance(self, n: int, /) -> None:
+        if n < 0 or n > len(self):
+            raise ValueError(n)
+        self._rpos += n
+        # Optional compaction heuristic: if we've consumed a lot, rebuild a smaller BytesIO.
+        # This may fail if someone holds a getbuffer() view (BufferError).
+        if self._rpos and self._rpos >= 65536 and self._rpos >= (self._bio.getbuffer().nbytes // 2):
+            try:
+                remaining = self._bio.getbuffer()[self._rpos:].tobytes()
+                self._bio = io.BytesIO(remaining)
+                self._rpos = 0
+            except BufferError as e:
+                raise RuntimeError('BytesIO buffer is pinned by an exported view') from e
+
+    def split_to(self, n: int, /) -> SegmentedBytesView:
+        if n < 0 or n > len(self):
+            raise ValueError(n)
+        mv = self._bio.getbuffer()
+        out = mv[self._rpos:self._rpos + n]
+        self._rpos += n
+        return SegmentedBytesView((out,))
+
+    def write(self, data: BytesLike, /) -> None:
+        if not data:
+            return
+        if isinstance(data, memoryview):
+            data = data.tobytes()
+        try:
+            self._bio.seek(0, io.SEEK_END)
+            self._bio.write(ta.cast(bytes, data))
+        except BufferError as e:
+            raise RuntimeError('BytesIO buffer is pinned by an exported view') from e
+
+    def reserve(self, n: int, /) -> memoryview:
+        if n < 0:
+            raise ValueError(n)
+        if self._resv is not None:
+            raise RuntimeError('outstanding reserve')
+        b = bytearray(n)
+        self._resv = b
+        self._resv_len = n
+        return memoryview(b)
+
+    def commit(self, n: int, /) -> None:
+        if self._resv is None:
+            raise RuntimeError('no outstanding reserve')
+        if n < 0 or n > self._resv_len:
+            raise ValueError(n)
+        b = self._resv
+        self._resv = None
+        self._resv_len = 0
+        if n:
+            self.write(memoryview(b)[:n])

omextra-0.0.0.dev505/omextra/io/buffers/all.py

@@ -0,0 +1,63 @@
+from .adapters import (  # noqa
+    FileLikeRawBytesReader,
+    FileLikeBufferedBytesReader,
+    BytesBufferReaderAdapter,
+    BytesBufferWriterAdapter,
+    BytesIoBytesBuffer,
+)
+
+from .errors import (  # noqa
+    BuffersError,
+    NeedMoreData,
+    BufferLimitError,
+    BufferTooLarge,
+    FrameTooLarge,
+    BufferStateError,
+    OutstandingReserve,
+    NoOutstandingReserve,
+)
+
+from .framing import (  # noqa
+    LongestMatchDelimiterFramer,
+)
+
+from .linear import (  # noqa
+    LinearBytesBuffer,
+)
+
+from .reading import (  # noqa
+    peek_u8,
+    read_u8,
+    peek_u16_be,
+    read_u16_be,
+    peek_u16_le,
+    read_u16_le,
+    peek_u32_be,
+    read_u32_be,
+    peek_u32_le,
+    read_u32_le,
+)
+
+from .scanning import (  # noqa
+    ScanningBytesBuffer,
+)
+
+from .segmented import (  # noqa
+    SegmentedBytesView,
+    SegmentedBytesBuffer,
+)
+
+from .types import (  # noqa
+    BytesLike,
+
+    BytesView,
+    BytesBuffer,
+    MutableBytesBuffer,
+)
+
+from .utils import (  # noqa
+    can_bytes,
+    iter_bytes_segments,
+    to_bytes,
+    bytes_len,
+)

omextra-0.0.0.dev505/omextra/io/buffers/errors.py

@@ -0,0 +1,54 @@
+# @omlish-lite
+
+
+##
+
+
+class BuffersError(Exception):
+    pass
+
+
+class NeedMoreData(BuffersError):  # noqa
+    """
+    Raised when an operation cannot complete because insufficient bytes are currently buffered.
+
+    This is intentionally distinct from EOF: it means "try again after feeding more bytes".
+    """
+
+
+class BufferLimitError(ValueError, BuffersError):  # noqa
+    """
+    Base class for buffer/framing limit violations.
+
+    Subclasses inherit from ValueError so existing tests expecting ValueError continue to pass.
+    """
+
+
+class BufferTooLarge(BufferLimitError):  # noqa
+    """
+    Buffered data exceeded a configured cap without finding a boundary that would allow progress.
+
+    Typically indicates an unframed stream, a missing delimiter, or an upstream not enforcing limits.
+    """
+
+
+class FrameTooLarge(BufferLimitError):  # noqa
+    """
+    A single decoded frame (payload before its boundary delimiter/length) exceeded a configured max size.
+    """
+
+
+class BufferStateError(RuntimeError, BuffersError):  # noqa
+    """
+    Base class for invalid buffer state transitions (e.g., coalescing while a reservation is outstanding).
+
+    Subclasses inherit from RuntimeError so existing tests expecting RuntimeError continue to pass.
+    """
+
+
+class OutstandingReserve(BufferStateError):  # noqa
+    """A reserve() is outstanding; an operation requiring stable storage cannot proceed."""
+
+
+class NoOutstandingReserve(BufferStateError):  # noqa
+    """commit() was called without a preceding reserve()."""