kafka-component 0.1.0__tar.gz

kafka_component-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,23 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+  workflow_call:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.11", "3.12", "3.13", "3.14"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v3
+        with:
+          python-version: ${{ matrix.python-version }}
+      - run: uv sync --all-groups
+      - run: uv run pytest
+      - run: uv run ruff format --check .
+      - run: uv run ruff check .

kafka_component-0.1.0/.github/workflows/publish-pypi.yml

@@ -0,0 +1,31 @@
+name: Publish to PyPI
+
+on:
+  release:
+    types: [published]
+  workflow_dispatch:
+    inputs:
+      target:
+        description: "PyPI target"
+        required: true
+        default: "pypi"
+        type: choice
+        options: ["pypi", "testpypi"]
+
+jobs:
+  ci:
+    uses: ./.github/workflows/ci.yml
+
+  publish:
+    needs: ci
+    runs-on: ubuntu-latest
+    environment: ${{ github.event.inputs.target || 'pypi' }}
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v3
+      - run: uv build
+      - uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          repository-url: ${{ (github.event.inputs.target == 'testpypi') && 'https://test.pypi.org/legacy/' || '' }}

kafka_component-0.1.0/.gitignore

@@ -0,0 +1,6 @@
+.superpowers/
+*.egg-info/
+__pycache__/
+.venv/
+dist/
+uv.lock

kafka_component-0.1.0/.python-version

@@ -0,0 +1 @@
+3.13

kafka_component-0.1.0/CHANGELOG.md

@@ -0,0 +1,5 @@
+# Changelog
+
+## 0.1.0
+
+- Initial release: `KafkaProducerComponent`, `KafkaConsumerComponent`, `ErrorPolicy` (`SkipAndLogPolicy`, `DeadLetterPolicy`).

kafka_component-0.1.0/CONTEXT.md

@@ -0,0 +1,25 @@
+# kafka-component
+
+Component library for Kafka producer/consumer lifecycle, built on python-components' `Component` contract. Provides async producer and consumer components with explicit, no-silent-failure delivery semantics.
+
+## Language
+
+**KafkaProducerComponent**:
+A `Component` that publishes JSON-encoded message values to Kafka, waiting for broker acknowledgment before returning.
+_Avoid_: Producer, Kafka publisher
+
+**KafkaConsumerComponent**:
+A `Component` that consumes JSON-encoded message values from Kafka, committing offsets only after the message handler succeeds.
+_Avoid_: Consumer, Kafka subscriber
+
+**ErrorPolicy**:
+The strategy a `KafkaConsumerComponent` invokes when a message handler raises, deciding what happens to the failed message.
+_Avoid_: failure handler, retry handler
+
+**SkipAndLogPolicy**:
+The default `ErrorPolicy` — logs the failure and advances past the message without retrying.
+_Avoid_: default policy
+
+**DeadLetterPolicy**:
+An `ErrorPolicy` that republishes the failed message and its error metadata to a dead-letter topic, defaulting to `{topic}.DLQ` when the caller doesn't supply one explicitly.
+_Avoid_: DLQ handler

kafka_component-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Fabricio Lima
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

kafka_component-0.1.0/PKG-INFO

@@ -0,0 +1,97 @@
+Metadata-Version: 2.5
+Name: kafka-component
+Version: 0.1.0
+Summary: Async Kafka producer/consumer components for python-components
+Project-URL: Homepage, https://github.com/fabriciooml/kafka-component
+Project-URL: Repository, https://github.com/fabriciooml/kafka-component
+Project-URL: Issues, https://github.com/fabriciooml/kafka-component/issues
+Project-URL: Changelog, https://github.com/fabriciooml/kafka-component/blob/main/CHANGELOG.md
+Author: Fabricio Lima
+License-Expression: MIT
+License-File: LICENSE
+Keywords: aiokafka,asyncio,kafka,python-components
+Classifier: Development Status :: 3 - Alpha
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Requires-Dist: aiokafka<1,>=0.14
+Requires-Dist: fastapi>=0.115
+Requires-Dist: python-components<0.5,>=0.4.0
+Description-Content-Type: text/markdown
+
+# kafka-component
+
+Async Kafka producer/consumer components for [python-components](https://github.com/lucassant95/python-components).
+
+## Install
+
+```bash
+uv add kafka-component
+```
+
+## Requirements
+
+- Python >= 3.11
+- A running Kafka broker (KRaft or ZooKeeper mode)
+
+## Usage
+
+```python
+import asyncio
+
+from python_components import System
+from kafka_component import KafkaProducerComponent, KafkaConsumerComponent, DeadLetterPolicy
+
+producer = KafkaProducerComponent(bootstrap_servers="localhost:9092")
+
+async def handle_order(value):
+    print("received order", value)
+
+consumer = KafkaConsumerComponent(
+    bootstrap_servers="localhost:9092",
+    group_id="orders-service",
+    topics=["orders"],
+    handler=handle_order,
+    error_policy=DeadLetterPolicy(producer, dlq_topic="orders.DLQ"),
+).using(["producer"])
+
+system = System({"producer": producer, "consumer": consumer})
+
+async def main():
+    async with system:
+        await producer.send("orders", {"order_id": 1})
+
+asyncio.run(main())
+```
+
+The consumer's `.using(["producer"])` declares a start-order dependency on the
+producer so `System`'s topological sort always starts the producer first —
+it is not read as an injected `self.producer` attribute by
+`KafkaConsumerComponent`; the `DeadLetterPolicy` above already holds a
+direct reference to the `producer` instance.
+
+## Semantics and caveats
+
+- **At-least-once delivery.** The consumer commits offsets only after the handler succeeds. A crash between a successful handler call and the commit can redeliver a message — handlers should be idempotent if that matters.
+- **Sequential processing.** One message is handled at a time, in partition order. There is no built-in concurrency; scale via more consumer instances/partitions, not intra-instance parallelism.
+- **Not a supervisor.** On handler failure, the configured `ErrorPolicy` runs (default `SkipAndLogPolicy`, or `DeadLetterPolicy`) and the loop continues — the component itself does not retry, crash-loop, or restart. Health is exposed via `routes()`; process supervision is the caller's job.
+- **JSON value-only.** Message values are `json.dumps`/`json.loads`. Keys and headers pass through untouched.
+- **Graceful shutdown drains the in-flight message.** `shutdown()` waits for the currently-processing message's handler (and its `ErrorPolicy`, if it fails) to finish before stopping — it does not abandon work mid-message.
+
+## Development
+
+```bash
+uv sync --all-groups
+uv run pytest
+uv run ruff format --check .
+uv run ruff check .
+```
+
+Integration tests spin up a real Kafka broker via [testcontainers](https://testcontainers.com/modules/kafka/). For local development against a persistent broker instead, run `docker compose up -d`.
+
+## License
+
+MIT

kafka_component-0.1.0/README.md

@@ -0,0 +1,73 @@
+# kafka-component
+
+Async Kafka producer/consumer components for [python-components](https://github.com/lucassant95/python-components).
+
+## Install
+
+```bash
+uv add kafka-component
+```
+
+## Requirements
+
+- Python >= 3.11
+- A running Kafka broker (KRaft or ZooKeeper mode)
+
+## Usage
+
+```python
+import asyncio
+
+from python_components import System
+from kafka_component import KafkaProducerComponent, KafkaConsumerComponent, DeadLetterPolicy
+
+producer = KafkaProducerComponent(bootstrap_servers="localhost:9092")
+
+async def handle_order(value):
+    print("received order", value)
+
+consumer = KafkaConsumerComponent(
+    bootstrap_servers="localhost:9092",
+    group_id="orders-service",
+    topics=["orders"],
+    handler=handle_order,
+    error_policy=DeadLetterPolicy(producer, dlq_topic="orders.DLQ"),
+).using(["producer"])
+
+system = System({"producer": producer, "consumer": consumer})
+
+async def main():
+    async with system:
+        await producer.send("orders", {"order_id": 1})
+
+asyncio.run(main())
+```
+
+The consumer's `.using(["producer"])` declares a start-order dependency on the
+producer so `System`'s topological sort always starts the producer first —
+it is not read as an injected `self.producer` attribute by
+`KafkaConsumerComponent`; the `DeadLetterPolicy` above already holds a
+direct reference to the `producer` instance.
+
+## Semantics and caveats
+
+- **At-least-once delivery.** The consumer commits offsets only after the handler succeeds. A crash between a successful handler call and the commit can redeliver a message — handlers should be idempotent if that matters.
+- **Sequential processing.** One message is handled at a time, in partition order. There is no built-in concurrency; scale via more consumer instances/partitions, not intra-instance parallelism.
+- **Not a supervisor.** On handler failure, the configured `ErrorPolicy` runs (default `SkipAndLogPolicy`, or `DeadLetterPolicy`) and the loop continues — the component itself does not retry, crash-loop, or restart. Health is exposed via `routes()`; process supervision is the caller's job.
+- **JSON value-only.** Message values are `json.dumps`/`json.loads`. Keys and headers pass through untouched.
+- **Graceful shutdown drains the in-flight message.** `shutdown()` waits for the currently-processing message's handler (and its `ErrorPolicy`, if it fails) to finish before stopping — it does not abandon work mid-message.
+
+## Development
+
+```bash
+uv sync --all-groups
+uv run pytest
+uv run ruff format --check .
+uv run ruff check .
+```
+
+Integration tests spin up a real Kafka broker via [testcontainers](https://testcontainers.com/modules/kafka/). For local development against a persistent broker instead, run `docker compose up -d`.
+
+## License
+
+MIT

kafka_component-0.1.0/docker-compose.yml

@@ -0,0 +1,14 @@
+services:
+  kafka:
+    image: apache/kafka:latest
+    ports:
+      - "9092:9092"
+    environment:
+      KAFKA_NODE_ID: 1
+      KAFKA_PROCESS_ROLES: broker,controller
+      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
+      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
+      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
+      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
+      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
+      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

kafka_component-0.1.0/docs/adr/0001-aiokafka-client-and-delivery-semantics.md

@@ -0,0 +1,16 @@
+---
+status: accepted
+---
+
+# aiokafka client with explicit delivery semantics
+
+kafka-component needs a Kafka client that fits the family's async-first, no-supervisor stance (components expose health, they don't self-heal). We chose **aiokafka** over confluent-kafka because confluent's asyncio support wraps a threaded producer/consumer rather than being asyncio-native, which fights the async-first design; we accept aiokafka's lower raw throughput and community (not enterprise-SLA) maintenance pace as the trade-off.
+
+On top of that, `KafkaConsumerComponent` commits offsets manually — only after the handler succeeds (at-least-once, no auto-commit) — and `KafkaProducerComponent` defaults to wait-for-ack (`acks=all`) rather than fire-and-forget. Per-message failure handling is pluggable via `ErrorPolicy` (default `SkipAndLogPolicy`, optional `DeadLetterPolicy`) instead of a built-in retry loop, keeping the component from silently losing or endlessly retrying messages.
+
+## Considered Options
+
+- confluent-kafka (rejected: threaded async wrapper, not asyncio-native)
+- auto-commit consumer (rejected: risks silent data loss on crash mid-processing)
+- fire-and-forget producer, `acks=0`/`1` (rejected as default: conflicts with family's explicit-failure stance; could be exposed later as opt-in)
+- built-in retry-N policy (rejected for v1: real design surface — backoff, ordering guarantees — punted; pluggable `ErrorPolicy` leaves room to add without breaking the API)