comio 0.1.0__tar.gz

comio-0.1.0/.claude/projects/-Users-sungdongkim-works-cio/memory/MEMORY.md

@@ -0,0 +1 @@
+- [Import convention](feedback_import_convention.md) — `import cio as io`, Go-like DX is a core design goal

comio-0.1.0/.claude/projects/-Users-sungdongkim-works-cio/memory/feedback_import_convention.md

@@ -0,0 +1,11 @@
+---
+name: import convention
+description: User uses "import cio as io" as the standard import — Go-like DX (io.Reader, io.Writer) is a core design goal
+type: feedback
+---
+
+Always use `import cio as io` in examples and documentation. The goal is Go-like DX where protocols are accessed as `io.Reader`, `io.Writer`, `io.scroll()`, etc.
+
+**Why:** The user explicitly wants a Go-inspired developer experience for Python I/O primitives.
+
+**How to apply:** Use this convention in all code examples, README, and documentation. When suggesting usage patterns, frame them as `io.X` not `cio.X`.

comio-0.1.0/.claude/settings.local.json

@@ -0,0 +1,10 @@
+{
+  "permissions": {
+    "allow": [
+      "Bash(pip install:*)",
+      "Bash(python:*)",
+      "Bash(pip index:*)",
+      "Bash(twine upload:*)"
+    ]
+  }
+}

comio-0.1.0/PKG-INFO

@@ -0,0 +1,109 @@
+Metadata-Version: 2.4
+Name: comio
+Version: 0.1.0
+Summary: Composable I/O primitives for async Python
+Project-URL: Repository, https://github.com/sungdongkim/cio
+License-Expression: MIT
+Classifier: Framework :: AsyncIO
+Classifier: Programming Language :: Python :: 3
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Requires-Dist: anyio>=4.0
+Provides-Extra: dev
+Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# cio – Composable I/O
+
+I/O primitives for async Python. Designed to be used as:
+
+```python
+import comio as io
+```
+
+Giving you `io.Reader`, `io.Writer`, `io.Listener`, etc. — a Go-like DX for Python.
+
+## Install
+
+```bash
+pip install comio
+```
+
+## Usage
+
+### Implement a Reader / Writer
+
+Any object with the right method is a valid `io.Reader` or `io.Writer` — no base class needed.
+
+```python
+import comio as io
+
+class JsonL:
+    def __init__(self, f):
+        self.f = f
+
+    async def read(self, *, cursor=None, n=None) -> io.Page:
+        self.f.seek(cursor or 0)
+        line = self.f.readline()
+        if line == "":
+            return Page([], io.EOF)
+        return Page(items=[json.loads(line)], next_cursor=self.f.tell())
+
+    async def write(self, item: dict) -> None:
+        self.f.write(json.dumps(item) + "\n")
+```
+
+### Read pages with `scroll`
+
+```python
+reader = JsonL(open("data.jsonl"))
+
+async for page in io.scroll(reader):
+    print(page.items)
+```
+
+### Drain everything with `read_all`
+
+```python
+items = await io.read_all(reader)
+```
+
+### Resume from a cursor
+
+```python
+items = await io.read_all(reader, cursor=saved_cursor)
+```
+
+### Normalize a Reader into a stream with `as_listener`
+
+```python
+async for item in io.as_listener(reader):
+    process(item)
+```
+
+### Buffered copy: Listener → Batcher
+
+```python
+await io.copy(listener, batcher, n=100)  # flush every 100 items
+```
+
+### Streaming pipe with backpressure
+
+```python
+from cio import pipe
+
+async def transform(item):
+    return {**item, "processed": True}
+
+await pipe(listener, writer, transform)
+```
+
+## Protocols
+
+| Protocol     | Method   | Description                      |
+|--------------|----------|----------------------------------|
+| `Reader[T]`  | `read`   | Pull a page of items             |
+| `Listener[T]`| `listen` | Push items as an async iterator  |
+| `Writer[T]`  | `write`  | Accept a single item             |
+| `Batcher[T]` | `batch`  | Accept a sequence of items       |

comio-0.1.0/README.md

@@ -0,0 +1,93 @@
+# cio – Composable I/O
+
+I/O primitives for async Python. Designed to be used as:
+
+```python
+import comio as io
+```
+
+Giving you `io.Reader`, `io.Writer`, `io.Listener`, etc. — a Go-like DX for Python.
+
+## Install
+
+```bash
+pip install comio
+```
+
+## Usage
+
+### Implement a Reader / Writer
+
+Any object with the right method is a valid `io.Reader` or `io.Writer` — no base class needed.
+
+```python
+import comio as io
+
+class JsonL:
+    def __init__(self, f):
+        self.f = f
+
+    async def read(self, *, cursor=None, n=None) -> io.Page:
+        self.f.seek(cursor or 0)
+        line = self.f.readline()
+        if line == "":
+            return Page([], io.EOF)
+        return Page(items=[json.loads(line)], next_cursor=self.f.tell())
+
+    async def write(self, item: dict) -> None:
+        self.f.write(json.dumps(item) + "\n")
+```
+
+### Read pages with `scroll`
+
+```python
+reader = JsonL(open("data.jsonl"))
+
+async for page in io.scroll(reader):
+    print(page.items)
+```
+
+### Drain everything with `read_all`
+
+```python
+items = await io.read_all(reader)
+```
+
+### Resume from a cursor
+
+```python
+items = await io.read_all(reader, cursor=saved_cursor)
+```
+
+### Normalize a Reader into a stream with `as_listener`
+
+```python
+async for item in io.as_listener(reader):
+    process(item)
+```
+
+### Buffered copy: Listener → Batcher
+
+```python
+await io.copy(listener, batcher, n=100)  # flush every 100 items
+```
+
+### Streaming pipe with backpressure
+
+```python
+from cio import pipe
+
+async def transform(item):
+    return {**item, "processed": True}
+
+await pipe(listener, writer, transform)
+```
+
+## Protocols
+
+| Protocol     | Method   | Description                      |
+|--------------|----------|----------------------------------|
+| `Reader[T]`  | `read`   | Pull a page of items             |
+| `Listener[T]`| `listen` | Push items as an async iterator  |
+| `Writer[T]`  | `write`  | Accept a single item             |
+| `Batcher[T]` | `batch`  | Accept a sequence of items       |

comio-0.1.0/comio/__init__.py

@@ -0,0 +1,37 @@
+"""comio – Composable I/O primitives for async Python."""
+
+from .io import (
+    Batcher,
+    Cursor,
+    EOF,
+    Listener,
+    Page,
+    Reader,
+    Writer,
+    copy,
+    as_listener,
+    read_all,
+    scroll,
+)
+from .pipe import Handler, PipeConfig, pipe
+
+__all__ = [
+    # Primitives
+    "Cursor",
+    "EOF",
+    # Protocols
+    "Page",
+    "Reader",
+    "Listener",
+    "Writer",
+    "Batcher",
+    "Handler",
+    # Functions
+    "scroll",
+    "read_all",
+    "copy",
+    "as_listener",
+    "pipe",
+    # Config
+    "PipeConfig",
+]

comio-0.1.0/comio/adapters/jsonl.py

@@ -0,0 +1,28 @@
+import json
+import io
+from dataclasses import dataclass
+
+from ..io import EOF, Cursor
+
+@dataclass
+class Page:
+    items: list[dict]
+    next_cursor: object
+
+class JsonL:
+
+    def __init__(self, f: io.TextIOWrapper):
+        self.f = f
+
+    async def read(self, cursor: Cursor | None = None, n: int | None =None) -> Page:
+        self.f.seek(cursor or 0)
+        l = self.f.readline()
+        if l == "":
+            return Page([], EOF)
+        return Page(items=[json.loads(l)], next_cursor=self.f.tell())
+    
+
+    async def write(self, item: dict) -> None:
+        self.f.write(json.dumps(item) + "\n")
+        self.f.flush()
+

comio-0.1.0/comio/io.py

@@ -0,0 +1,193 @@
+"""Composable I/O primitives.
+
+Defines four protocol interfaces for asynchronous data flow:
+
+    Reader: pull-based, paginated source  (read → Page)
+    Writer: single-item sink              (write)
+    Batch: multi-item sink               (batch)
+    Listener: push-based, streaming source  (listen → AsyncIterator)
+
+And composable functions that wire them together:
+
+    scroll:  iterate pages from a Reader
+    read_all: drain a Reader into a list
+    copy:     connect any source to any sink
+"""
+
+from __future__ import annotations
+
+import typing as t
+
+T = t.TypeVar("T")
+T_co = t.TypeVar("T_co", covariant=True)
+T_contra = t.TypeVar("T_contra", contravariant=True)
+
+Cursor: t.TypeAlias = t.Any
+"""An opaque position marker within a data source.
+
+Can be any type the data source uses to track position:
+an integer offset, a string token, a complex object, etc.
+Each Reader implementation defines how to interpret its cursors.
+"""
+
+
+class _EOF:
+    """Sentinel indicating the end of a data source."""
+    pass
+
+
+EOF = _EOF()
+"""Singleton sentinel. Return this as ``next_cursor`` to signal
+that there is no more data to read."""
+
+
+class Page(t.Protocol, t.Generic[T_co]):
+    """A single page of results returned by a Reader.
+
+    Attributes:
+        items: The items contained in this page.
+        next_cursor: The cursor pointing to the next page,
+            or ``EOF`` if this is the last page.
+    """
+
+    @property
+    def items(self) -> t.Sequence[T_co]: ...
+
+    @property
+    def next_cursor(self) -> Cursor: ...
+
+
+class Reader(t.Protocol, t.Generic[T_co]):
+    """Pull-based, paginated data source.
+
+    Implementors decide:
+    - How ``cursor`` maps to a position in the underlying store.
+    - What page size ``n`` means (item count, byte count, etc.).
+    - When to return ``EOF`` as ``next_cursor`` to signal completion.
+    """
+
+    async def read(self, *, cursor: Cursor = None, n: int | None = None) -> Page[T_co]:
+        """Read one page of data starting at ``cursor``.
+
+        Args:
+            cursor: Position to read from. ``None`` means the beginning.
+            n: Requested page size. ``None`` lets the implementation choose.
+
+        Returns:
+            A Page whose ``next_cursor`` is ``EOF`` when no more data remains.
+        """
+        ...
+
+
+class Listener(t.Protocol, t.Generic[T_co]):
+    """Push-based, streaming data source.
+
+    Unlike Reader which requires the caller to pull pages,
+    a Listener yields items as they become available.
+    """
+
+    async def listen(self) -> t.AsyncIterator[T_co]:
+        """Start listening and yield items as they arrive.
+
+        The iterator completes when the source is exhausted or closed.
+        """
+        ...
+
+
+class Writer(t.Protocol, t.Generic[T_contra]):
+    """Single-item sink.
+
+    Accepts one item at a time via ``write()``.
+    """
+
+    async def write(self, item: T_contra) -> None:
+        """Write a single item to the destination."""
+        ...
+
+
+class Batcher(t.Protocol, t.Generic[T_contra]):
+    """Multi-item sink.
+
+    Accepts a sequence of items at once via ``batch()``,
+    enabling bulk inserts, buffered writes, etc.
+    """
+
+    async def batch(self, items: t.Sequence[T_contra]) -> None:
+        """Write a batch of items to the destination."""
+        ...
+
+
+async def scroll(r: Reader[T_co], *, cursor: Cursor = None, n: int | None = None) -> t.AsyncIterator[Page[T_co]]:
+    """Iterate pages from a Reader until the source is exhausted.
+
+    Args:
+        r: The reader to pull pages from.
+        cursor: Starting position. ``None`` begins at the start.
+        n: Requested page size per read. ``None`` lets the reader choose.
+
+    Yields:
+        Each non-empty Page. Stops when the page is empty
+        or ``next_cursor`` is ``EOF``.
+    """
+    while True:
+        page = await r.read(cursor=cursor, n=n)
+        if not page.items:
+            break
+        yield page
+        if page.next_cursor is EOF:
+            break
+        cursor = page.next_cursor
+
+
+async def read_all(r: Reader[T_co], *, cursor: Cursor = None, n: int | None = None) -> t.Sequence[T_co]:
+    """Drain a Reader and collect every item into a single sequence.
+
+    Args:
+        r: The reader to drain.
+        cursor: Starting position. ``None`` begins at the start.
+        n: Requested page size per read. ``None`` lets the reader choose.
+
+    Returns:
+        All items concatenated across every page.
+    """
+    items: t.List[T_co] = []
+    async for page in scroll(r, cursor=cursor, n=n):
+        items.extend(page.items)
+    return items
+
+
+async def copy(src: Listener[T_co], dst: Batcher[T_co], n: int) -> None:
+    """Buffer items from a Listener and flush to a Batcher every ``n`` items.
+
+    Args:
+        src: Push-based source to consume from.
+        dst: Batch sink to flush into.
+        n: Buffer size. Flushes every ``n`` items, plus any remainder at the end.
+    """
+    buf: t.List[T_co] = []
+    async for item in await src.listen():
+        buf.append(item)
+        if len(buf) >= n:
+            await dst.batch(buf)
+            buf = []
+    if buf:
+        await dst.batch(buf)
+
+
+async def as_listener(r: Reader[T_co], *, cursor: Cursor = None, n: int | None = None) -> t.AsyncIterator[T_co]:
+    """Convert a Reader into an item-level AsyncIterator.
+
+    Normalizes a pull-based Reader into the same shape as Listener.listen(),
+    so downstream code can treat both sources uniformly.
+
+    Args:
+        r: The reader to convert.
+        cursor: Starting position.
+        n: Page size passed to the reader.
+
+    Yields:
+        Individual items from each page.
+    """
+    async for page in scroll(r, cursor=cursor, n=n):
+        for item in page.items:
+            yield item

comio-0.1.0/comio/pipe.py

@@ -0,0 +1,106 @@
+"""Streaming pipe: Listener → Handler → Writer.
+
+Connects a push-based source to a sink through a transformation handler,
+using in-memory streams with configurable backpressure.
+
+    src ──→ [in_stream] ──→ handler ──→ [out_stream] ──→ dest
+
+Three concurrent tasks (ingress / process / egress) are managed
+by an anyio task group. When the source is exhausted, streams close
+in order and the pipe shuts down gracefully.
+"""
+
+from __future__ import annotations
+
+import typing as t
+
+import anyio
+
+from .io import Listener, Writer
+
+
+
+In = t.TypeVar("In", covariant=True)
+Out = t.TypeVar("Out", contravariant=True)
+H_In = t.TypeVar("H_In", contravariant=True)
+H_Out = t.TypeVar("H_Out", covariant=True)
+
+
+class Handler(t.Protocol, t.Generic[H_In, H_Out]):
+    """Transforms a single input item into a single output item."""
+
+    async def __call__(self, item: H_In) -> H_Out: ...
+
+
+class PipeConfig(t.TypedDict):
+    buffer: t.NotRequired[int]
+    """Buffer size for the memory streams between stages.
+    Default is 0 (rendezvous channel = strongest backpressure)."""
+
+    on_boot: t.NotRequired[t.Sequence[t.Callable[[], t.Awaitable[None]]]]
+    """Hooks executed sequentially before starting the pipe."""
+
+    on_close: t.NotRequired[t.Sequence[t.Callable[[], t.Awaitable[None]]]]
+    """Hooks executed sequentially after shutdown (even on failure)."""
+
+
+async def pipe(
+    src: Listener[In],
+    dest: Writer[Out],
+    h: Handler[In, Out],
+    *,
+    cfg: PipeConfig | None = None,
+) -> None:
+    """Run a streaming pipe: ``src → h → dest``.
+
+    Args:
+        src: Push-based source that yields input items.
+        dest: Sink that receives transformed output items.
+        h: Handler that transforms each input into an output.
+        cfg: Optional configuration for buffer size and lifecycle hooks.
+    """
+    if cfg is None:
+        cfg = PipeConfig()
+
+    for boot in cfg.get("on_boot", []):
+        await boot()
+    try:
+        await _run(src, dest, h, buffer=cfg.get("buffer", 0))
+    finally:
+        for close in cfg.get("on_close", []):
+            await close()
+
+
+async def _run(
+    src: Listener[In],
+    dest: Writer[Out],
+    h: Handler[In, Out],
+    *,
+    buffer: int,
+) -> None:
+    in_send, in_recv = anyio.create_memory_object_stream[In](buffer)
+    out_send, out_recv = anyio.create_memory_object_stream[Out](buffer)
+
+    async def ingress() -> None:
+        """src → in_stream. Closes in_send when the source is exhausted."""
+        async with in_send:
+            async for item in await src.listen():
+                await in_send.send(item)
+
+    async def process() -> None:
+        """in_stream → handler → out_stream.
+        Closes out_send when in_recv is exhausted."""
+        async with out_send, in_recv:
+            async for item in in_recv:
+                await out_send.send(await h(item))
+
+    async def egress() -> None:
+        """out_stream → dest. Finishes when out_recv is closed."""
+        async with out_recv:
+            async for item in out_recv:
+                await dest.write(item)
+
+    async with anyio.create_task_group() as tg:
+        tg.start_soon(egress)
+        tg.start_soon(process)
+        tg.start_soon(ingress)

comio-0.1.0/pyproject.toml

@@ -0,0 +1,31 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "comio"
+version = "0.1.0"
+description = "Composable I/O primitives for async Python"
+readme = "README.md"
+requires-python = ">=3.11"
+license = "MIT"
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "Framework :: AsyncIO",
+    "Typing :: Typed",
+]
+dependencies = [
+    "anyio>=4.0",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=8.0",
+    "pytest-asyncio>=0.24",
+]
+
+[project.urls]
+Repository = "https://github.com/sungdongkim/cio"
+
+[tool.pytest.ini_options]
+asyncio_mode = "auto"

comio-0.1.0/tests/test_jsonl.py

@@ -0,0 +1,83 @@
+import io
+import json
+import pytest
+
+from comio import scroll, read_all, EOF
+from comio.adapters.jsonl import JsonL
+
+
+@pytest.fixture
+def sample_file():
+    """Create an in-memory JSONL file with 3 lines."""
+    buf = io.StringIO()
+    for i in range(3):
+        buf.write(json.dumps({"id": i}) + "\n")
+    buf.seek(0)
+    return buf
+
+
+@pytest.mark.asyncio
+async def test_read_single_page(sample_file):
+    reader = JsonL(sample_file)
+    page = await reader.read()
+    assert page.items == [{"id": 0}]
+    assert page.next_cursor is not EOF
+
+
+@pytest.mark.asyncio
+async def test_read_with_cursor(sample_file):
+    reader = JsonL(sample_file)
+    first = await reader.read()
+    second = await reader.read(cursor=first.next_cursor)
+    assert second.items == [{"id": 1}]
+
+
+@pytest.mark.asyncio
+async def test_read_eof(sample_file):
+    reader = JsonL(sample_file)
+    cursor = None
+    for _ in range(3):
+        page = await reader.read(cursor=cursor)
+        cursor = page.next_cursor
+    last = await reader.read(cursor=cursor)
+    assert last.items == []
+    assert last.next_cursor is EOF
+
+
+@pytest.mark.asyncio
+async def test_scroll(sample_file):
+    reader = JsonL(sample_file)
+    pages = []
+    async for page in scroll(reader):
+        pages.append(page)
+    assert len(pages) == 3
+    assert [p.items[0]["id"] for p in pages] == [0, 1, 2]
+
+
+@pytest.mark.asyncio
+async def test_read_all(sample_file):
+    reader = JsonL(sample_file)
+    items = await read_all(reader)
+    assert items == [{"id": 0}, {"id": 1}, {"id": 2}]
+
+
+@pytest.mark.asyncio
+async def test_write(tmp_path):
+    path = tmp_path / "out.jsonl"
+    with open(path, "w") as f:
+        writer = JsonL(f)
+        await writer.write({"a": 1})
+        await writer.write({"b": 2})
+
+    lines = path.read_text().strip().split("\n")
+    assert len(lines) == 2
+    assert json.loads(lines[0]) == {"a": 1}
+    assert json.loads(lines[1]) == {"b": 2}
+
+
+@pytest.mark.asyncio
+async def test_read_all_from_cursor(sample_file):
+    reader = JsonL(sample_file)
+    first = await reader.read()
+    items = await read_all(reader, cursor=first.next_cursor)
+    assert items == [{"id": 1}, {"id": 2}]