data-ingestor 0.2.0__py3-none-any.whl

data_ingestor-0.2.0.dist-info/METADATA

@@ -0,0 +1,162 @@
+Metadata-Version: 2.5
+Name: data-ingestor
+Version: 0.2.0
+Summary: Data ingestion (streaming, batch, micro-batch) with a ports-and-adapters architecture
+Requires-Python: >=3.14
+Requires-Dist: confluent-kafka>=2.15.0
+Provides-Extra: sse
+Requires-Dist: httpx>=0.27; extra == 'sse'
+Provides-Extra: ws
+Requires-Dist: websocket-client>=1.8; extra == 'ws'
+Description-Content-Type: text/markdown
+
+# Ingestor
+
+A Python **data ingestion** app that reads records from a source (**Source**) and writes them to a destination (**Sink**). The same logic serves several modes: **streaming** (Kafka), **batch** (file) and **micro-batch** (by limit/window).
+
+The architecture is **ports and adapters**: the logic depends only on interfaces (`ingestor.ports`), and each tool lives isolated in its own adapter (`ingestor.adapters.*`). Swapping source or destination - or using in-memory fakes in tests - does not touch the logic.
+
+## Requirements
+
+- Python 3.14 or newer.
+- `uv` installed to manage dependencies.
+- Docker with Docker Compose and Docker Desktop running (only for the Kafka mode).
+- Port `9092` free for local Kafka.
+
+## How to run
+
+### 1. Clone the project
+
+```bash
+git clone https://github.com/rikemorais/ingestor.git
+cd ingestor
+```
+
+### 2. Install dependencies
+
+```bash
+uv sync
+```
+
+### 3. Start Kafka
+
+```bash
+docker compose up -d
+```
+
+Wait for the broker to start. To follow the logs:
+
+```bash
+docker compose logs -f kafka
+```
+
+Press `Ctrl+C` to leave the logs. Kafka keeps running.
+
+### 4. Run the app
+
+```bash
+uv run ingestor
+```
+
+> `ingestor` is the command defined in `pyproject.toml` (`[project.scripts]`). Equivalent alternative: `uv run python -m ingestor`.
+
+The demo seeds a few records into the topic and then **ingests from Kafka to the console**. With local Kafka available, the expected output is:
+
+```text
+Connected to Kafka: 1 broker(s).
+Ingested: origin=demo-messages key=1 value=Hello Kafka #1
+Ingested: origin=demo-messages key=2 value=Hello Kafka #2
+Ingested: origin=demo-messages key=3 value=Hello Kafka #3
+Ingested: origin=demo-messages key=4 value=Hello Kafka #4
+Ingested: origin=demo-messages key=5 value=Hello Kafka #5
+Ingested 5 records.
+```
+
+The connection uses `localhost:9092` by default and waits up to 10 seconds for a response.
+
+> Note: `KafkaSource` consumes in the `ingestor` group and, after a few consecutive empty polls (`max_empty_polls`), stops - that's why the demo terminates instead of waiting forever. Running again with the same group won't re-read already-read messages (the offset is saved). To re-consume everything, change the group in `src/ingestor/adapters/kafka/connection.py` or recreate the topic.
+
+## Use another address
+
+Set `KAFKA_BOOTSTRAP_SERVERS` when running:
+
+```bash
+KAFKA_BOOTSTRAP_SERVERS=server:9092 uv run ingestor
+```
+
+Replace `server:9092` with your broker's address. The current setup uses an unauthenticated connection; SASL/TLS clusters need adjustments in the adapter.
+
+## Run the tests (no Kafka)
+
+The tests use in-memory implementations (fakes) and a file source, so they **need neither Docker nor a broker**:
+
+```bash
+uv run python -m unittest discover -v
+```
+
+They exercise the same `ingest` function with a `FakeConnection` and with a `FileSource` (batch) - proving in practice that the logic does not depend on the Kafka lib.
+
+> Tip: when adding/moving files under `src/`, if `uv run` complains about a module not found, run once `uv sync --reinstall-package data-ingestor`.
+
+## Stop Kafka
+
+```bash
+docker compose stop
+```
+
+To start the same container again:
+
+```bash
+docker compose start
+```
+
+## If the connection fails
+
+Check that Docker is running and inspect the broker's state and logs:
+
+```bash
+docker compose ps
+docker compose logs --tail=50 kafka
+```
+
+If Kafka is still starting up, wait and run again. Also check the address and that port `9092` is free. On failure, the app prints the error and exits with code `1`.
+
+## Structure
+
+`src/` layout, with the code in an `ingestor` package organized as ports × adapters:
+
+```text
+ingestor/
+├── src/
+│   └── ingestor/
+│       ├── __init__.py
+│       ├── app.py              # use case: ingest(source, sink, limit)
+│       ├── cli.py              # Composition Root: picks and injects the adapters
+│       ├── __main__.py         # enables `python -m ingestor`
+│       ├── ports/              # interfaces + domain (tool-agnostic)
+│       │   ├── record.py       # Record (value, key, origin)
+│       │   ├── source.py       # Source: read() -> Iterator[Record]
+│       │   ├── sink.py         # Sink: write(record)
+│       │   └── connection.py   # Connection: source() / sink()
+│       └── adapters/           # concrete implementations (one folder per tool)
+│           ├── kafka/          # streaming: KafkaSource, KafkaSink, KafkaConnection, KafkaInfoCluster
+│           ├── console/        # generic destination: ConsoleSink
+│           └── batch/          # batch: FileSource (reads a file line by line)
+├── tests/
+│   ├── __init__.py
+│   ├── fakes.py                # ListSource, MemorySink, FakeConnection
+│   └── test_ingestor.py        # unittest tests
+├── docker-compose.yml          # local Kafka
+├── pyproject.toml              # dependencies, build and the `ingestor` command
+└── uv.lock
+```
+
+**How the layers relate:** `app.py` depends only on `ingestor.ports` (Source, Sink, Connection); the adapters in `adapters/` implement those ports; and only `cli.py` knows the concrete classes, wiring everything together. This way no specific lib (like `confluent_kafka`) "leaks" into the logic or the tests.
+
+**The three modes, the same logic:**
+
+| Mode | Source | Does `ingest` change? |
+|---|---|---|
+| Streaming | `KafkaSource` | no |
+| Batch | `FileSource` | no |
+| Micro-batch | any Source + `limit` | no |

data_ingestor-0.2.0.dist-info/RECORD

@@ -0,0 +1,31 @@
+ingestor/__init__.py,sha256=POOTDxaaJ7b0Mz_8DhMLthIYafbCKOA8MxjbMwMF83U,557
+ingestor/__main__.py,sha256=h6fAzlUnDiT9KRd2Qtv32nGtI_CvXdrzST7JejjdC5A,134
+ingestor/app.py,sha256=9ORxW4a9LAWcji9cYAJf041jXVqj466qva0N2UkDqoI,938
+ingestor/cli.py,sha256=bS0cCkI2c9prr35BqDV1UorgaWvwvMudweVis2acF3g,1800
+ingestor/testing.py,sha256=Lxlz9iaYvmRmoKETawSV1qVn9s1CYou18RmaiCfneE0,1381
+ingestor/adapters/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+ingestor/adapters/batch/__init__.py,sha256=qm5F0mKGnyiJUF57Vm7_tvshwOpjnVVL643I6-61LJY,145
+ingestor/adapters/batch/file_source.py,sha256=s7D0zsv6goapZ80dMC3s8LzA4ACjK7W3inIIOrKGV3E,1320
+ingestor/adapters/console/__init__.py,sha256=ZOF7AyOaebGjjd8xcsa1jT8JpYiBQLTbBkN7cKwplrY,177
+ingestor/adapters/console/json_sink.py,sha256=LzZ5GLMPLN-hir0wYz-8XaZxh7dQjJwK5vYxqQqXvgQ,691
+ingestor/adapters/console/sink.py,sha256=EcW-eXgF_xoyDWCbldkvyzvq8GPAzgUarJfeKMymi0U,806
+ingestor/adapters/kafka/__init__.py,sha256=UulxZ9vQi-00n1tVBdN_9VI-iuATNsVbKhvRzz4Yaug,431
+ingestor/adapters/kafka/connection.py,sha256=dL8e7YiWOlqCjZZyCYol61RKOR_Qx2RFmRxiLzvJSLI,2579
+ingestor/adapters/kafka/info.py,sha256=JCVrleU2lvRydkqZJeM4ukF5UChi0cbPAdHk4yCyHJc,1012
+ingestor/adapters/kafka/sink.py,sha256=DnfGhcTGjTgAc_dzZqu0hI8V2R-UfCH5kOcQcP_wA28,1566
+ingestor/adapters/kafka/source.py,sha256=KqHhXg3n6Ql56jyl2L1vsX_VrdYNA5CNxDP-kOScHDs,2683
+ingestor/adapters/sse/__init__.py,sha256=JCl5ZKznpXDS1AoHwfE5f-n3jLSoENMETVkDwy1sk4M,140
+ingestor/adapters/sse/source.py,sha256=aEZhFvf7Gke9sSQmQILBeFPP_1wC557bvezGBGc7d4o,4111
+ingestor/adapters/tee/__init__.py,sha256=mXAJr-tzlFHLEFMmYFSSEtmU-zZfzqub4NUrv67pmtU,126
+ingestor/adapters/tee/sink.py,sha256=KRHAl57Z1McXtkEs8V8pFsos6YBr-7gl9Hm-Bs-KlYY,541
+ingestor/adapters/ws/__init__.py,sha256=W36UK_HbnItNFGYDX0N2NyCGFL-7bWWgw7t5EWnmgj0,160
+ingestor/adapters/ws/source.py,sha256=o36RF7AEdw_jQr9ASckwITOpBrLQdrJSIEY_jKBV_Dc,3635
+ingestor/ports/__init__.py,sha256=5G-T8Ob8LfH6Cg4PJrZgTFI4xbBqskd7jm_whCiZnv0,326
+ingestor/ports/connection.py,sha256=UQ4M9J-6PB2OqyJuCt4z183eTOzcKt2gYBw8zG3QeqM,850
+ingestor/ports/record.py,sha256=VCv3q5gT_LaxgxsnURetCSa41JKXz8iW-0zwLIvmEAI,524
+ingestor/ports/sink.py,sha256=xoGDM54bIXflJAd4Z61EQt3UsWzBclTymOeRh4hCFQs,570
+ingestor/ports/source.py,sha256=xeJvd7OznLkCw_yHRGCUApumSIfwZyaFiJr2MKOHjl4,674
+data_ingestor-0.2.0.dist-info/METADATA,sha256=cSU6rSi357BLZduD_2FAHIbZWQN8d1dxu5EDdr_Bjho,5927
+data_ingestor-0.2.0.dist-info/WHEEL,sha256=W3fkpkm7-wf9vBI5Z-7s0eWkeM-spu78I8Neb98DeEg,87
+data_ingestor-0.2.0.dist-info/entry_points.txt,sha256=hS0XLj0TGKr7OZY3k7au0TDjPOTZv8IgYj8gaKwU8JE,47
+data_ingestor-0.2.0.dist-info/RECORD,,

data_ingestor-0.2.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.32.4
+Root-Is-Purelib: true
+Tag: py3-none-any

data_ingestor-0.2.0.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+ingestor = ingestor.cli:main

ingestor/__init__.py

@@ -0,0 +1,15 @@
+"""Ingestor: read records from sources (streaming, batch, micro-batch) and write them to sinks.
+
+Organized as ports & adapters:
+- `ports`: Source, Sink, Connection and the Record domain model.
+- `adapters`: concrete implementations (kafka, console, ...).
+- `app`: the `ingest` use case (Source -> Sink).
+- `cli`: the Composition Root.
+"""
+
+from importlib.metadata import PackageNotFoundError, version
+
+try:
+    __version__ = version("data-ingestor")
+except PackageNotFoundError:  # pragma: no cover - only when running uninstalled
+    __version__ = "0.0.0"

ingestor/__main__.py

@@ -0,0 +1,6 @@
+"""Entry point for ``python -m ingestor``."""
+
+from ingestor.cli import main
+
+if __name__ == "__main__":
+    raise SystemExit(main())

ingestor/adapters/batch/__init__.py

@@ -0,0 +1,5 @@
+"""Batch adapters: finite sources (files, tables, ...)."""
+
+from ingestor.adapters.batch.file_source import FileSource
+
+__all__ = ["FileSource"]

ingestor/adapters/batch/file_source.py

@@ -0,0 +1,43 @@
+"""Batch source: reads a text file, one record per line.
+
+Shows batch entering the system without touching ``ingest``: just another Source.
+"""
+
+from collections.abc import Iterator
+from pathlib import Path
+from typing import TextIO
+
+from ingestor.ports.record import Record
+
+
+class FileSource:
+    """A finite Source that reads a text file line by line."""
+
+    def __init__(self, path: str | Path, encoding: str = "utf-8") -> None:
+        """Store the file path.
+
+        Args:
+            path: Path to the text file to read.
+            encoding: Text encoding used to open the file.
+        """
+        self._path = Path(path)
+        self._encoding = encoding
+        self._file: TextIO | None = None
+
+    def read(self) -> Iterator[Record]:
+        """Open the file and yield one record per line.
+
+        Yields:
+            A record whose ``value`` is the line (newline stripped) and whose
+            ``origin`` is the file name.
+        """
+        self._file = self._path.open("r", encoding=self._encoding)
+        origin = self._path.name
+        for line in self._file:
+            yield Record(value=line.rstrip("\n"), origin=origin)
+
+    def close(self) -> None:
+        """Close the file if it is open."""
+        if self._file is not None:
+            self._file.close()
+            self._file = None

ingestor/adapters/console/__init__.py

@@ -0,0 +1,4 @@
+from ingestor.adapters.console.json_sink import JsonFieldsConsoleSink
+from ingestor.adapters.console.sink import ConsoleSink
+
+__all__ = ["ConsoleSink", "JsonFieldsConsoleSink"]

ingestor/adapters/console/json_sink.py

@@ -0,0 +1,24 @@
+"""Console sink for displaying selected fields from JSON records."""
+
+import json
+from collections.abc import Iterable
+
+from ingestor.ports.record import Record
+
+
+class JsonFieldsConsoleSink:
+    """Print selected JSON fields from each record."""
+
+    def __init__(self, fields: Iterable[str]) -> None:
+        self._fields = tuple(fields)
+
+    def write(self, record: Record) -> None:
+        try:
+            payload = json.loads(record.value)
+            values = " ".join(f"{field}={payload.get(field, '-')}" for field in self._fields)
+            print(values)
+        except (TypeError, json.JSONDecodeError):
+            print(record.value)
+
+    def close(self) -> None:
+        pass

ingestor/adapters/console/sink.py

@@ -0,0 +1,28 @@
+"""A simple sink that prints records - handy for demos and debugging."""
+
+from ingestor.ports.record import Record
+
+
+class ConsoleSink:
+    """A Sink that writes each record to standard output.
+
+    Depends on no tool, showing that the Sink port is truly agnostic.
+    """
+
+    def __init__(self, include_metadata: bool = True) -> None:
+        self._include_metadata = include_metadata
+
+    def write(self, record: Record) -> None:
+        """Print a record to stdout.
+
+        Args:
+            record: The record to print.
+        """
+        if self._include_metadata:
+            print(f"Ingested: Origin({record.origin}) Key({record.key}) Value: {record.value}")
+        else:
+            print(record.value)
+
+    def close(self) -> None:
+        """No-op: there is nothing to flush."""
+        pass

ingestor/adapters/kafka/__init__.py

@@ -0,0 +1,14 @@
+"""Kafka adapter: all Kafka-specific code lives here."""
+
+from ingestor.adapters.kafka.connection import KafkaConnection, KafkaConnectionInfo
+from ingestor.adapters.kafka.info import KafkaInfoCluster
+from ingestor.adapters.kafka.sink import KafkaSink
+from ingestor.adapters.kafka.source import KafkaSource
+
+__all__ = [
+    "KafkaConnection",
+    "KafkaConnectionInfo",
+    "KafkaSource",
+    "KafkaSink",
+    "KafkaInfoCluster",
+]

ingestor/adapters/kafka/connection.py

@@ -0,0 +1,81 @@
+"""Kafka connection: builds KafkaSource and KafkaSink.
+
+The only place that joins the Kafka config, the concrete adapters, the
+AdminClient and the metadata translation. The rest of the system sees only
+the Connection interface.
+"""
+
+from dataclasses import dataclass, field
+
+from confluent_kafka.admin import AdminClient
+
+from ingestor.adapters.kafka.info import KafkaInfoCluster
+from ingestor.adapters.kafka.sink import KafkaSink
+from ingestor.adapters.kafka.source import KafkaSource
+from ingestor.ports.connection import Connection
+from ingestor.ports.sink import Sink
+from ingestor.ports.source import Source
+
+
+@dataclass(frozen=True)
+class KafkaConnectionInfo:
+    """Kafka-specific connection config.
+
+    Attributes:
+        brokers: Broker addresses to connect to.
+        topics: Topics the source subscribes to; the first is the sink default.
+        group: Consumer group id.
+        from_beginning: Whether the source reads from the start of the topic.
+    """
+
+    brokers: list[str]
+    topics: list[str] = field(default_factory=list)
+    group: str = "ingestor"
+    from_beginning: bool = True
+
+
+class KafkaConnection(Connection):
+    """Builds Kafka adapters from a KafkaConnectionInfo."""
+
+    def __init__(self, info: KafkaConnectionInfo) -> None:
+        """Store the config and derive the broker string.
+
+        Args:
+            info: The Kafka connection config.
+        """
+        self._info = info
+        self._servers = ",".join(info.brokers)
+
+    def connect(self, timeout: float = 10.0) -> KafkaInfoCluster:
+        """Fetch cluster metadata (a Kafka-only extra, not part of the port).
+
+        Args:
+            timeout: Seconds to wait for the metadata response.
+
+        Returns:
+            A summary of the cluster.
+        """
+        cluster = AdminClient({"bootstrap.servers": self._servers}).list_topics(timeout=timeout)
+        return KafkaInfoCluster.from_cluster(cluster)
+
+    def source(self) -> Source:
+        """Build a KafkaSource for the configured topics.
+
+        Returns:
+            A streaming Source.
+        """
+        return KafkaSource(
+            bootstrap_servers=self._servers,
+            topics=self._info.topics,
+            group=self._info.group,
+            from_beginning=self._info.from_beginning,
+        )
+
+    def sink(self) -> Sink:
+        """Build a KafkaSink targeting the first configured topic.
+
+        Returns:
+            An output Sink.
+        """
+        topic = self._info.topics[0] if self._info.topics else None
+        return KafkaSink(bootstrap_servers=self._servers, default_topic=topic)

ingestor/adapters/kafka/info.py

@@ -0,0 +1,37 @@
+"""Kafka-specific cluster summary (metadata)."""
+
+from dataclasses import dataclass
+from typing import Any, Self
+
+
+@dataclass(frozen=True)
+class KafkaInfoCluster:
+    """Cluster metadata translated from confluent's ClusterMetadata.
+
+    Attributes:
+        brokers: Broker addresses discovered in the cluster.
+        topics: Topic names discovered in the cluster, sorted.
+    """
+
+    brokers: list[str]
+    topics: list[str]
+
+    @property
+    def broker_count(self) -> int:
+        """Number of brokers in the cluster."""
+        return len(self.brokers)
+
+    @classmethod
+    def from_cluster(cls, cluster: Any) -> Self:
+        """Build a summary from confluent_kafka's ClusterMetadata.
+
+        Args:
+            cluster: The ClusterMetadata returned by ``list_topics``.
+
+        Returns:
+            A domain-level summary of the cluster.
+        """
+        return cls(
+            brokers=[str(broker) for broker in cluster.brokers.values()],
+            topics=sorted(cluster.topics.keys()),
+        )

ingestor/adapters/kafka/sink.py

@@ -0,0 +1,48 @@
+"""Kafka sink: writes records by producing to a topic."""
+
+from confluent_kafka import Producer
+
+from ingestor.ports.record import Record
+
+
+class KafkaSink:
+    """An output Sink backed by a confluent-kafka Producer."""
+
+    def __init__(
+        self,
+        bootstrap_servers: str = "localhost:9092",
+        default_topic: str | None = None,
+    ):
+        """Create the producer.
+
+        Args:
+            bootstrap_servers: Comma-separated broker addresses.
+            default_topic: Topic used when a record has no ``origin``.
+        """
+        self._producer = Producer({"bootstrap.servers": bootstrap_servers})
+        self._default_topic = default_topic
+
+    def write(self, record: Record) -> None:
+        """Produce a record to Kafka.
+
+        Args:
+            record: The record to produce. Its ``origin`` is used as the topic,
+                falling back to ``default_topic``.
+
+        Raises:
+            ValueError: If no destination topic can be determined.
+        """
+        topic = record.origin or self._default_topic
+        if topic is None:
+            raise ValueError("No destination topic: set record.origin or default_topic.")
+        self._producer.produce(topic, value=record.value, key=record.key)
+        self._producer.poll(0)
+
+    def close(self, timeout: float = 10.0) -> None:
+        """Flush pending messages to the broker.
+
+        Args:
+            timeout: Maximum seconds to wait for delivery before giving up, so a
+                dead broker cannot block forever.
+        """
+        self._producer.flush(timeout)

ingestor/adapters/kafka/source.py

@@ -0,0 +1,80 @@
+"""Kafka source: consumes messages and yields them as records."""
+
+from collections.abc import Iterator
+
+from confluent_kafka import Consumer, KafkaError, KafkaException
+
+from ingestor.ports.record import Record
+
+
+class KafkaSource:
+    """A streaming Source backed by a confluent-kafka Consumer."""
+
+    def __init__(
+        self,
+        bootstrap_servers: str = "localhost:9092",
+        topics: list[str] | None = None,
+        group: str = "ingestor",
+        from_beginning: bool = True,
+        max_empty_polls: int = 10,
+        timeout: float = 1.0,
+    ):
+        """Create the consumer.
+
+        Args:
+            bootstrap_servers: Comma-separated broker addresses.
+            topics: Topics to subscribe to.
+            group: Consumer group id (controls committed offsets).
+            from_beginning: Read from the start of the topic when the group has
+                no saved offset; otherwise read only new messages.
+            max_empty_polls: Stop after this many consecutive empty polls, so
+                demos and batch-style runs terminate.
+            timeout: Seconds to wait on each poll.
+        """
+        self._topics = topics or []
+        self._max_empty_polls = max_empty_polls
+        self._timeout = timeout
+        self._consumer = Consumer(
+            {
+                "bootstrap.servers": bootstrap_servers,
+                "group.id": group,
+                "auto.offset.reset": "earliest" if from_beginning else "latest",
+            }
+        )
+
+    def read(self) -> Iterator[Record]:
+        """Subscribe and yield records until the topic goes quiet.
+
+        Yields:
+            Each consumed message translated into a Record.
+
+        Raises:
+            KafkaException: If the broker returns a non-EOF error.
+        """
+        self._consumer.subscribe(self._topics)
+        empty = 0
+        while empty < self._max_empty_polls:
+            message = self._consumer.poll(self._timeout)
+            if message is None:
+                empty += 1
+                continue
+
+            error = message.error()
+            if error is not None:
+                if error.code() == KafkaError._PARTITION_EOF:
+                    empty += 1
+                    continue
+                raise KafkaException(error)
+
+            empty = 0
+            value = message.value()
+            key = message.key()
+            yield Record(
+                value=value.decode("utf-8") if value is not None else "",
+                key=key.decode("utf-8") if key is not None else None,
+                origin=message.topic(),
+            )
+
+    def close(self) -> None:
+        """Close the underlying consumer."""
+        self._consumer.close()

ingestor/adapters/sse/__init__.py

@@ -0,0 +1,5 @@
+"""SSE adapters: streaming sources over Server-Sent Events."""
+
+from ingestor.adapters.sse.source import SSESource
+
+__all__ = ["SSESource"]

ingestor/adapters/sse/source.py

@@ -0,0 +1,106 @@
+"""Server-Sent Events (SSE) source: consumes any text/event-stream endpoint.
+
+True streaming: it holds the HTTP connection open and yields a Record for each
+event the server pushes, as it arrives. Reconnects automatically on drops so it
+can run continuously. Example: Wikimedia EventStreams.
+"""
+
+import sys
+import time
+from collections.abc import Iterator
+
+import httpx
+
+from ingestor.ports.record import Record
+
+
+class SSESource:
+    """A streaming Source for any SSE (text/event-stream) endpoint."""
+
+    def __init__(
+        self,
+        url: str,
+        origin: str | None = None,
+        connect_timeout: float = 10.0,
+        user_agent: str = "ingestor/0.1 (https://github.com/rikemorais/ingestor)",
+        reconnect: bool = True,
+        reconnect_delay: float = 3.0,
+    ):
+        """Configure the stream.
+
+        Args:
+            url: The SSE endpoint to connect to.
+            origin: Value used as each record's ``origin``; defaults to the URL.
+            connect_timeout: Seconds to wait for the connection to open (reads
+                have no timeout, since the stream stays open between events).
+            user_agent: Sent as the ``User-Agent`` header. Some providers (such
+                as Wikimedia) reject requests without a descriptive one.
+            reconnect: Reconnect automatically if the connection drops or ends.
+            reconnect_delay: Seconds to wait before each reconnection attempt.
+        """
+        self._url = url
+        self._origin = origin or url
+        self._connect_timeout = connect_timeout
+        self._user_agent = user_agent
+        self._reconnect = reconnect
+        self._reconnect_delay = reconnect_delay
+        self._running = True
+        self._client: httpx.Client | None = None
+
+    def read(self) -> Iterator[Record]:
+        """Yield one Record per SSE event, reconnecting on drops.
+
+        Yields:
+            A record whose ``value`` is the event's ``data`` payload, ``key`` is
+            the event id (if sent) and ``origin`` is the configured origin.
+        """
+        while self._running:
+            try:
+                yield from self._stream_once()
+            except httpx.HTTPError as error:
+                if not self._reconnect or not self._running:
+                    raise
+                print(
+                    f"[SSESource] connection lost ({error}); "
+                    f"reconnecting in {self._reconnect_delay}s",
+                    file=sys.stderr,
+                )
+            if not self._reconnect:
+                break
+            if self._running:
+                time.sleep(self._reconnect_delay)
+
+    def _stream_once(self) -> Iterator[Record]:
+        headers = {
+            "Accept": "text/event-stream",
+            "User-Agent": self._user_agent,
+        }
+        with httpx.Client(timeout=httpx.Timeout(self._connect_timeout, read=None)) as client:
+            self._client = client
+            with client.stream("GET", self._url, headers=headers) as response:
+                response.raise_for_status()
+                data: list[str] = []
+                event_id: str | None = None
+                for line in response.iter_lines():
+                    if line == "":
+                        if data:
+                            yield Record(
+                                value="\n".join(data),
+                                key=event_id,
+                                origin=self._origin,
+                            )
+                            data = []
+                            event_id = None
+                    elif line.startswith("data:"):
+                        data.append(line[len("data:") :].lstrip())
+                    elif line.startswith("id:"):
+                        event_id = line[len("id:") :].strip()
+                    # comment (":...") and other fields (event:, retry:) fall through
+        self._client = None
+
+    def close(self) -> None:
+        """Stop reconnecting and close the HTTP connection."""
+        self._running = False
+        if self._client is not None:
+            self._client.close()
+            self._client = None

ingestor/adapters/tee/__init__.py

@@ -0,0 +1,5 @@
+"""Adapters that fan records out to multiple sinks."""
+
+from ingestor.adapters.tee.sink import TeeSink
+
+__all__ = ["TeeSink"]

ingestor/adapters/tee/sink.py

@@ -0,0 +1,21 @@
+"""Sink adapter that writes each record to multiple sinks."""
+
+from collections.abc import Iterable
+
+from ingestor.ports.record import Record
+from ingestor.ports.sink import Sink
+
+
+class TeeSink:
+    """Forward every record to all configured sinks."""
+
+    def __init__(self, sinks: Iterable[Sink]) -> None:
+        self._sinks = tuple(sinks)
+
+    def write(self, record: Record) -> None:
+        for sink in self._sinks:
+            sink.write(record)
+
+    def close(self) -> None:
+        for sink in self._sinks:
+            sink.close()

ingestor/adapters/ws/__init__.py

@@ -0,0 +1,5 @@
+"""WebSocket adapters: streaming sources over WebSocket connections."""
+
+from ingestor.adapters.ws.source import WebSocketSource
+
+__all__ = ["WebSocketSource"]

ingestor/adapters/ws/source.py

@@ -0,0 +1,99 @@
+"""WebSocket source: consumes any WebSocket endpoint as a stream of records.
+
+True streaming: it keeps the socket open and yields a Record for each message
+the server pushes. Reconnects automatically on drops so it can run continuously.
+Example: the Coinbase market-data feed.
+"""
+
+import json
+import sys
+import time
+from collections.abc import Iterator
+
+from websocket import WebSocket, WebSocketException, create_connection
+
+from ingestor.ports.record import Record
+
+
+class WebSocketSource:
+    """A streaming Source for any WebSocket endpoint."""
+
+    def __init__(
+        self,
+        url: str,
+        subscribe: dict | str | None = None,
+        origin: str | None = None,
+        open_timeout: float = 10.0,
+        reconnect: bool = True,
+        reconnect_delay: float = 3.0,
+    ):
+        """Configure the stream.
+
+        Args:
+            url: The ``wss://`` endpoint to connect to.
+            subscribe: Message sent right after connecting (a dict is JSON
+                encoded). Used by feeds that need a subscription, like Coinbase.
+            origin: Value used as each record's ``origin``; defaults to the URL.
+            open_timeout: Seconds to wait for the connection to open.
+            reconnect: Reconnect automatically if the connection drops.
+            reconnect_delay: Seconds to wait before each reconnection attempt.
+        """
+        self._url = url
+        self._subscribe = subscribe
+        self._origin = origin or url
+        self._open_timeout = open_timeout
+        self._reconnect = reconnect
+        self._reconnect_delay = reconnect_delay
+        self._running = True
+        self._ws: WebSocket | None = None
+
+    def read(self) -> Iterator[Record]:
+        """Yield one Record per message, reconnecting on drops.
+
+        Yields:
+            A record whose ``value`` is the raw message text and ``origin`` is
+            the configured origin.
+        """
+        while self._running:
+            try:
+                yield from self._stream_once()
+            except (WebSocketException, OSError) as error:
+                if not self._reconnect or not self._running:
+                    raise
+                print(
+                    f"[WebSocketSource] connection lost ({error}); "
+                    f"reconnecting in {self._reconnect_delay}s",
+                    file=sys.stderr,
+                )
+            if not self._reconnect:
+                break  # pragma: no cover - _stream_once only returns by raising
+            if self._running:
+                time.sleep(self._reconnect_delay)
+
+    def _stream_once(self) -> Iterator[Record]:
+        self._ws = create_connection(self._url, timeout=self._open_timeout)
+        try:
+            if self._subscribe is not None:
+                message = (
+                    self._subscribe
+                    if isinstance(self._subscribe, str)
+                    else json.dumps(self._subscribe)
+                )
+                self._ws.send(message)
+            self._ws.settimeout(None)  # block waiting for pushed messages
+            while True:
+                frame = self._ws.recv()
+                if frame:
+                    text = frame if isinstance(frame, str) else frame.decode("utf-8", "replace")
+                    yield Record(value=text, origin=self._origin)
+        finally:
+            if self._ws is not None:
+                self._ws.close()
+                self._ws = None
+
+    def close(self) -> None:
+        """Stop reconnecting and close the WebSocket connection."""
+        self._running = False
+        if self._ws is not None:
+            self._ws.close()
+            self._ws = None

ingestor/app.py

@@ -0,0 +1,32 @@
+"""Core ingestion use case: read from a Source and write to a Sink.
+
+Talks only to the ports. The same ``ingest`` serves streaming, batch and
+micro-batch - only the injected Source changes (and ``limit``, for micro-batch).
+"""
+
+from ingestor.ports import Sink, Source
+
+
+def ingest(source: Source, sink: Sink, limit: int | None = None) -> int:
+    """Read every record from ``source`` and write it to ``sink``.
+
+    Args:
+        source: Where records are read from.
+        sink: Where records are written to.
+        limit: Stop after this many records; ``None`` reads until the source is
+            exhausted.
+
+    Returns:
+        The number of records ingested.
+    """
+    total = 0
+    try:
+        for record in source.read():
+            sink.write(record)
+            total += 1
+            if limit is not None and total >= limit:
+                break
+    finally:
+        source.close()
+        sink.close()
+    return total

ingestor/cli.py

@@ -0,0 +1,52 @@
+"""Composition Root: picks the concrete tools and injects them into the logic.
+
+Demo: seeds a few records into Kafka and then ingests from Kafka to the console.
+To change source/destination, just build another Source/Sink here - app.py stays.
+"""
+
+import os
+import sys
+
+from confluent_kafka import KafkaException
+
+from ingestor.adapters.console import ConsoleSink
+from ingestor.adapters.kafka import KafkaConnection, KafkaConnectionInfo
+from ingestor.app import ingest
+from ingestor.ports import Record
+
+TOPIC = "demo-messages"
+
+
+def main() -> int:
+    """Run the Kafka-to-console demo.
+
+    Seeds a few records into the topic, then ingests them to the console.
+
+    Returns:
+        Process exit code: ``0`` on success, ``1`` if Kafka cannot be reached.
+    """
+    server = os.getenv("KAFKA_BOOTSTRAP_SERVERS", "localhost:9092")
+    connection = KafkaConnection(KafkaConnectionInfo(brokers=server.split(","), topics=[TOPIC]))
+    try:
+        info = connection.connect()
+        if info.broker_count == 0:
+            print(
+                "No Kafka brokers reachable. Is Kafka running? Start it with: docker compose up -d",
+                file=sys.stderr,
+            )
+            return 1
+        print(f"Connected to Kafka: {info.broker_count} broker(s).")
+
+        # (demo) seed a few records into the topic
+        kafka_sink = connection.sink()
+        for i in range(1, 6):
+            kafka_sink.write(Record(value=f"Hello Kafka #{i}", key=str(i), origin=TOPIC))
+        kafka_sink.close()
+
+        # ingest from Kafka to the console
+        total = ingest(connection.source(), ConsoleSink(), limit=5)
+        print(f"Ingested {total} records.")
+        return 0
+    except KafkaException as error:
+        print(f"Failed to connect to Kafka: {error}", file=sys.stderr)
+        return 1

ingestor/ports/__init__.py

@@ -0,0 +1,8 @@
+"""The ports (interfaces) and the domain model, all in one place."""
+
+from ingestor.ports.connection import Connection, ConnectionInfo
+from ingestor.ports.record import Record
+from ingestor.ports.sink import Sink
+from ingestor.ports.source import Source
+
+__all__ = ["Record", "Source", "Sink", "Connection", "ConnectionInfo"]

ingestor/ports/connection.py

@@ -0,0 +1,36 @@
+"""Connection port: builds a tool's Source and Sink.
+
+Generic - it knows nothing about Kafka. Each tool implements this interface by
+returning its own Source/Sink, which satisfy the abstract ports.
+"""
+
+from typing import Protocol
+
+from ingestor.ports.sink import Sink
+from ingestor.ports.source import Source
+
+
+class ConnectionInfo(Protocol):
+    """Marker for a tool's connection config (Kafka, Rabbit, file, ...)."""
+
+    ...
+
+
+class Connection(Protocol):
+    """Factory that builds a Source and a Sink for one tool."""
+
+    def source(self) -> Source:
+        """Build the input adapter.
+
+        Returns:
+            A Source that yields records from the tool.
+        """
+        ...
+
+    def sink(self) -> Sink:
+        """Build the output adapter.
+
+        Returns:
+            A Sink that writes records to the tool.
+        """
+        ...

ingestor/ports/record.py

@@ -0,0 +1,21 @@
+"""Domain record that flows through ingestion.
+
+Tool-agnostic: it knows nothing about Kafka, files or any other source.
+"""
+
+from dataclasses import dataclass
+
+
+@dataclass(frozen=True)
+class Record:
+    """A single unit of data read from a source or written to a sink.
+
+    Attributes:
+        value: The record payload.
+        key: Optional partitioning/identity key.
+        origin: Where the record came from (topic, file name, table...).
+    """
+
+    value: str
+    key: str | None = None
+    origin: str | None = None

ingestor/ports/sink.py

@@ -0,0 +1,24 @@
+"""Output port: a destination that receives records (data lake, Kafka, console...)."""
+
+from typing import Protocol
+
+from ingestor.ports.record import Record
+
+
+class Sink(Protocol):
+    """Contract for anything that consumes records."""
+
+    def write(self, record: Record) -> None:
+        """Write a single record to the destination.
+
+        Args:
+            record: The record to persist.
+        """
+        ...
+
+    def close(self) -> None:
+        """Flush and release resources.
+
+        Make sure everything was written before closing.
+        """
+        ...

ingestor/ports/source.py

@@ -0,0 +1,26 @@
+"""Input port: a source that yields records.
+
+Streaming, batch and micro-batch are all Sources - only how ``read`` iterates
+changes: continuous for streaming, finite for batch, windowed for micro-batch.
+"""
+
+from collections.abc import Iterator
+from typing import Protocol
+
+from ingestor.ports.record import Record
+
+
+class Source(Protocol):
+    """Contract for anything that produces records."""
+
+    def read(self) -> Iterator[Record]:
+        """Yield records one by one.
+
+        Yields:
+            The next record read from the underlying system.
+        """
+        ...
+
+    def close(self) -> None:
+        """Release any resources held by the source."""
+        ...

ingestor/testing.py

@@ -0,0 +1,52 @@
+"""In-memory test doubles of the ports.
+
+These satisfy the Source, Sink and Connection ports without any real tool, so you
+can test an ingestion pipeline without Kafka or Docker, in this project or in your
+own. Import them as ``from ingestor.testing import ListSource, MemorySink``.
+"""
+
+from collections.abc import Iterator
+
+from ingestor.ports import Record
+
+
+class ListSource:
+    """Finite source: yields a ready list of records (batch-like)."""
+
+    def __init__(self, records: list[Record]) -> None:
+        self._records = records
+        self.closed = False
+
+    def read(self) -> Iterator[Record]:
+        yield from self._records
+
+    def close(self) -> None:
+        self.closed = True
+
+
+class MemorySink:
+    """Sink that collects records in memory, for inspection."""
+
+    def __init__(self) -> None:
+        self.records: list[Record] = []
+        self.closed = False
+
+    def write(self, record: Record) -> None:
+        self.records.append(record)
+
+    def close(self) -> None:
+        self.closed = True
+
+
+class FakeConnection:
+    """Connection without any Kafka - proves tool swappability."""
+
+    def __init__(self, records: list[Record]) -> None:
+        self._records = records
+        self.used_sink = MemorySink()
+
+    def source(self) -> ListSource:
+        return ListSource(self._records)
+
+    def sink(self) -> MemorySink:
+        return self.used_sink