flechtwerk 0.1.0__tar.gz

flechtwerk-0.1.0/.github/workflows/ci.yaml

@@ -0,0 +1,9 @@
+jobs:
+  test:
+    uses: ./.github/workflows/test.yaml
+name: CI
+on:
+  pull_request:
+    branches: [ main ]
+  push:
+    branches: [ '**' ]

flechtwerk-0.1.0/.github/workflows/publish.yaml

@@ -0,0 +1,30 @@
+jobs:
+  test:
+    uses: ./.github/workflows/test.yaml
+    with:
+      run-coverage: false
+  publish:
+    environment: pypi
+    name: Publish to PyPI
+    needs: test
+    permissions:
+      contents: read # job-level permissions reset all unlisted scopes to none
+      id-token: write # required for trusted publishing
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v5
+        with:
+          fetch-depth: 0 # hatch-vcs derives the version from the tag
+      - uses: astral-sh/setup-uv@v8.3.2
+        with:
+          python-version: '3.14'
+      - name: Install dependencies
+        run: uv sync
+      - name: Build package
+        run: uv build
+      - name: Publish to PyPI
+        run: uv publish
+name: Publish to PyPI
+on:
+  push:
+    tags: [ 'v*.*.*' ]

flechtwerk-0.1.0/.github/workflows/test.yaml

@@ -0,0 +1,35 @@
+jobs:
+  test:
+    name: Test Python ${{ matrix.python-version }}
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v5
+        with:
+          fetch-depth: 0 # hatch-vcs derives the version from tags
+      - uses: astral-sh/setup-uv@v8.3.2
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install dependencies
+        run: uv sync
+      - name: Run tests without coverage
+        if: '!inputs.run-coverage'
+        run: uv run pytest -m 'integration or not integration'
+      - name: Run tests with coverage
+        if: inputs.run-coverage
+        run: |
+          uv run coverage run -m pytest -m 'integration or not integration'
+          uv run coverage report
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ${{ fromJson(inputs.python-versions) }}
+name: Reusable Test Workflow
+on:
+  workflow_call:
+    inputs:
+      python-versions:
+        type: string
+        default: '["3.14"]'
+      run-coverage:
+        type: boolean
+        default: true

flechtwerk-0.1.0/.gitignore

@@ -0,0 +1,16 @@
+*.egg-info
+.coverage
+.coverage.*
+.hypothesis/
+.idea/
+.pytest_cache/
+.venv
+.vscode/
+/.claude/settings.local.json
+/CLAUDE.local.md
+/graphify-out/
+__pycache__/
+build/
+coverage.xml
+dist/
+htmlcov/

flechtwerk-0.1.0/CLAUDE.md

@@ -0,0 +1,192 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Development Practices
+
+### Git Operations
+
+- **Never push without permission**: Only push commits when explicitly asked to do so
+- Commits can be made freely, but pushing requires explicit user request
+- **Don't commit code automatically**
+- Always use PGP signatures when committing (`git commit -S`)
+
+### Code Style
+
+- **Alphabetical ordering**: Keep dictionary keys, constants, imports, and other collections in alphabetical order where possible
+- **Comprehensions over accumulator loops**: Build lists, dicts, and sets with comprehensions or generator expressions — not by starting with an empty container and mutating it in a `for` loop. Exception: loop bodies with real side effects (logging, I/O, external mutation)
+- **Markdown formatting**: Always include empty lines after headings in Markdown files
+- **File endings**: All text files must end with a newline
+- **Typing**: Use `X | None` instead of `Optional[X]` for typing
+
+## Commands
+
+```bash
+uv sync                        # venv + all dependencies
+uv run pytest                  # unit tier — Docker-free
+uv run pytest -m integration   # integration tier — ephemeral Kafka/Mosquitto via testcontainers; skips when Docker is unreachable
+uv run coverage run -m pytest -m "integration or not integration" && uv run coverage report   # both tiers, matching CI
+uv build                       # sdist + wheel; version derived from the latest vX.Y.Z tag via hatch-vcs
+```
+
+### Releasing
+
+Tag `vX.Y.Z` and push the tag; `.github/workflows/publish.yaml` runs the test suite, builds the package with the tag-derived version (hatch-vcs), and publishes it to PyPI via trusted publishing. Nothing else sets the version.
+
+## Architecture
+
+Flechtwerk is an async stream processing framework for Kafka with hexagonal architecture (ports and adapters), packaged under `src/flechtwerk/`. Applications build stages in two shapes:
+
+1. **Extractor** (abstract — construct via `Extractor.of(...)` or subclass): Async polls external sources based on Kafka configuration, maintains state in RocksDB for incremental processing
+2. **Transformer** (abstract — construct via `Transformer.of(...)` or subclass): Consumes partitioned input topics and publishes to Kafka with exactly-once delivery
+
+Module by module:
+
+- **`attribute/`**: `Attribute[V]` is a single, type-safe handle on a dict key carrying an explicit `Codec[V]` — `Attribute(name, codec)` declares a required field, `Attribute(name, codec, optional=True)` (keyword-only) one that may be absent or `None`. Codecs are compositional: atoms (`STR`, `INT`, `BOOL`, `DATE`, `FLOAT`, `DATETIME`, `TIME`, `RECORD`, `ANY`) and constructors (`LIST(V)`, `SET(V)`, `TUPLE(V)`, `DICT(V)`). `Record` wraps a `dict[str, Any]` (`.raw`) and accepts only `Attribute` keys for indexing — codecs run on every write to enforce that `.raw` stays JSON-native. **Two constructor paths:** `Record(source)` accepts another `Record` (copy via `raw.copy()`) or `dict[Attribute, Any]` (typed literal — runs each Attribute's `write_to`); `Record.wrap(raw_dict)` is the wire-format entry point that accepts `dict[str, Any]` (raw JSON, `.raw` payloads, pickle restore) and runs every value through the recursive `_encode_any` walker. Pick the path that matches the input shape — typed literals and Record-spread go through `__init__`, raw JSON goes through `.wrap`. Subclasses (`Event`, `State`, `Config`) inherit both. A required attribute (the default) rejects `None` writes so a missing value can't land silently as JSON `null`; `optional=True` accepts `None` (stored as JSON `null`, encoder bypassed). This write-side guard is the *only* behavior the flag drives — the read distinction (`V` vs `V | None`) is carried by the method, not the attribute (`[]` reads-or-raises, `.get` / `.pop` tolerate absence), and `required` is a derived bool property (the inverse of `optional`). `ANY` is the escape hatch: a recursive walker over JSON-native primitives + `Record` + `datetime` + `date` + `time` + `set` + `tuple` that raises `TypeError` on anything else. Dict-spread (`Record({**other, NEW_ATTR: value})`) is supported — `Record.keys()` yields `ViewAttribute` handles whose overridden `read_from` / `write_to` round-trip raw wire values through the standard Record paths, so the spread itself stays in the typed-`__init__` path. **Polymorphic dispatch over isinstance branches:** every `Record` dict-op (`__getitem__`, `__setitem__`, `__contains__`, `__delitem__`, `get`, `pop`) is a one-line delegation to a method on the Attribute (`read_from`, `write_to`, `present_in`, `delete_from`, `get_from`, `pop_from`) — new Attribute kinds (e.g. `ViewAttribute`, declared in `attribute.py` but not re-exported from the package) override the primitives and inherit the derived ops via default impls on the base. The hierarchy is sealed (`__init_subclass__` rejects subclasses outside `attribute.py`), so kind dispatch is exhaustive and cross-method invariants only need policing in that one module. `IDENTITY` codec lives in `codecs.py` for `ViewAttribute`'s wire-form pass-through but is similarly non-exported. Pickle reconstructs via `__reduce__` = `(cls.wrap, (raw,))`, landing back in `wrap`; `copy.copy`/`copy.deepcopy` use the dedicated `__copy__`/`__deepcopy__` instead.
+- **`types.py`**: `Config`, `Event`, `State` are `Record` subclasses + `IncomingMessage`/`Message` dataclasses (timestamps are `datetime`, not millis). `Stage` (non-exported) is the common base of `Extractor`/`Transformer`: it owns the `config_topics: list[str]` declaration (default `[]`) and the `enrich(config)` hook — one-time enrichment applied by the config machinery once per config record (bootstrap compacts first: once per surviving entry), never per poll tick or lookup. Partitioned `input_topics: list[str]` is declared by `Transformer` alone — a plain `Extractor` consumes only config topics.
+- **`extractor.py`**: `Extractor` (ABC, `poll` is `@abstractmethod`) + `ExtractorRunner`. Async polling with `asyncio.gather` for concurrent config processing. Build via `Extractor.of(config_topics=..., poll=...)` (returns a private `_FunctionalExtractor` concrete subclass) or subclass directly for lifecycle needs. Config handling rides the config machinery (`configs.py`): the `group_id=None` consumer is assigned all partitions of every config topic, bootstrapped to the end offsets captured at startup (compacted by wire key, empty values are tombstones, `enrich` once per surviving entry), then drained non-blocking each poll cycle; the runner's per-key config entries are read back from the (already enriched) store. `poll(config, state)` yields `Message | State`; the runner only persists `State` when it differs from current. Yielding an empty/falsy `State` deletes the entry. **Re-entry contract** (pinned by test): for a given config, `poll()` is re-entered only after the previous invocation's messages were sent and the producer flushed — push sources that defer upstream ACKs until Kafka durability depend on this ordering. **Wakeup**: `Extractor.wakeup` (an optional `asyncio.Event`, default `None`) lets a push-driven stage end the runner's between-cycles wait early; `poll_interval_seconds` then degrades to the idle/config-drain cadence.
+- **`transformer.py`**: `Transformer` (ABC, `transform` is `@abstractmethod`) + `TransformerRunner` + `Task` + `TaskRebalanceListener`. Build via `Transformer.of(input_topics=..., transform=...)` (returns a private `_FunctionalTransformer` concrete subclass) or subclass directly. Work is partitioned into per-input-partition **tasks**, each owning a transactional producer (static ID `{application_id}-{partition}` — EOS-v1 fencing) and a partition-scoped `ChangelogStateStore`. Exactly-once delivery: one Kafka transaction per task per `getmany()` batch covering that task's output messages, state changes (deduped to one final write per key), and offset commits; task transactions commit concurrently and independently. Within a batch, records are bucketed by *(task, state key)*; same-bucket records run serially (so each one sees the previous one's yielded state), and buckets run concurrently via `asyncio.gather` so I/O-bound `transform()` calls overlap. Cross-bucket ordering is not preserved. Within a bucket, records appear in `input_topics` order then Kafka offset order. Each `transform` call receives a defensive deepcopy of the running state so in-place mutation without a yield can't leak. Stateless transformers simply never yield `State`. Rebalances (eager only in aiokafka) tear down all tasks under the batch lock and rebuild assigned ones in the main loop (producer start = fencing point → changelog restore → resume); the listener records failures on `runner.fatal` because aiokafka swallows callback exceptions. A transformer may additionally declare `config_topics` and look entries up via `self.configs.get(wire_key)`: the runner bootstraps the store before the subscribe joins the group, injects `configs` onto the transformer before `__aenter__`, and drains updates once per loop iteration outside the batch lock — every record of a batch sees one consistent config snapshot, and rebalances never touch the instance-level store. Lookups are eventually consistent and NOT part of any task transaction (Kafka Streams' GlobalKTable caveat). `Transformer.of(...)` accepts `enrich` (machinery-applied, works without `self`); store lookups need a subclass.
+- **`kafka.py`**: `parse_message()` (JSON → `Event`), `encode_json()` (`Record` → UTF-8 bytes), `restore_changelog()` (rebuilds state from a compacted topic, optionally restricted to a partition subset for task restore; reads to the end offsets captured at entry — the LSO under `read_committed` — instead of treating an empty poll as end-of-log; primes aiokafka's metadata cache via `consumer._client.set_topics()` when discovering partitions — no fully public API exists for this, integration tests lock down the coupling). `read_to_end()` (the assign/seek/read-until-captured-end loop), `is_tombstone()`, `decode_key()`/`decode_event()` are the shared building blocks of `restore_changelog` and the config machinery. Runners type-hint `aiokafka.AIOKafkaConsumer`/`AIOKafkaProducer` directly — no wrapper classes.
+- **`configs.py`**: `ConfigStore` (exported) + `bootstrap_config_store` + `drain_config_updates` — Kafka Streams' **GlobalKTable** pattern, specialized to configuration. Config topics are read in full (`group_id=None`, all partitions) into ONE per-process latest-value store keyed by **wire key** — a single key namespace merged across all of a stage's config topics (a tombstone on any topic deletes the key). The source topics are their own changelog (must be compacted, should stay small — the store lives in RAM per instance), no offsets are committed, and everything is re-read on every startup. `Stage.enrich` is applied here, once per record (bootstrap: once per surviving entry); Kafka Streams forbids transforming records into global stores (KIP-813) because a checkpoint restore would bypass the transformation — Flechtwerk has no checkpoints, every boot re-reads through the same enrich path, so the store can't diverge. Values are kept as wire bytes and parsed lazily on `get()` — each call returns a fresh `Config`; malformed values decode to an empty `Config` with a warning so they hit the caller's validation instead of masquerading as missing. Tests seed via `ConfigStore.of({key: config})`.
+- **`state.py`**: `StateStore` port + `RocksDBStateStore` and `ChangelogStateStore` adapters. Storage primitive is `put_bytes(key, raw)`; `put(key, state)` is `serialize`-then-`put_bytes`. `deserialize` is JSON-only — undecodable bytes are an unrecoverable data error (crash, then reset the affected state). `ChangelogStateStore` writes the same wire bytes to both inner store and Kafka; its optional `partition` pins changelog writes to one explicit partition (transformer tasks) while `None` (extractors) uses key hashing; the producer is shared with the owning runner/task via DI so transformer `put()` calls join the task's open transaction. `restore` replays the changelog (optionally one partition) through `inner.put_bytes` (raw bytes pass through; deserialization is deferred to first `get()` per key). `get()` returns a protective copy. `partition_counts()` reads per-topic partition counts for the startup co-partitioning validation. `rocksdict` is imported lazily on first RocksDB open — stateless stages never load it.
+- **`module.py`**: `Flechtwerk` — the narrow application-facing handle (an ABC exposing only `of(...)` / `run()` / the async context manager), plus the private `_FlechtwerkModule` reactor-di DI container that lazily creates and shares all Kafka resources. `Flechtwerk.of(...)` returns a `_FlechtwerkModule` typed as `Flechtwerk`, so an application never sees the wiring (same idiom as `Extractor.of` / `Transformer.of`). To embed as a child of a larger reactor-di module, declare `make[Flechtwerk, _FlechtwerkModule]` and let the parent wire every `lookup` field. The `Flechtwerk` base stays annotation-free — `@module` walks `get_type_hints` over the MRO, so any annotated attribute on it would leak onto the public type; `test_public_handle_exposes_no_container_internals` pins this. Extractors get a single shared non-transactional producer and a global `ChangelogStateStore`; transformers get factory methods (`create_task_producer`, `create_task_store`, `create_restore_consumer`) from which the runner builds per-task resources. The runners consume the caller's stage through the `configured_stage` factory, which completes an MQTT-sourced stage with the broker settings and the observer — the module stays pure factory + mediation, no behavioral methods. All consumers run `read_committed`; the transformer input consumer uses the Range assignor. Transformers with config topics get a dedicated group-less `config_consumer` (client id suffix `-config`); extractors reuse their main consumer (already group-less); both get the `config_store`. Async context manager — Prometheus scrape HTTP server is the outermost layer; startup runs `validate_topics` (transformer needs ≥1 input topic, extractor needs ≥1 config topic, the lists must be disjoint), validates input/changelog partition counts for transformers (config topics are exempt — their counts are unconstrained), and existence-checks config topics (a missing one must fail fast: the assign-based bootstrap would never discover a topic created later). `compression_type` defaults to `"zstd"` (JSON compresses ~13×), which is why the package depends on `aiokafka[zstd]`.
+- **`mqtt.py`**: the MQTT→Kafka bridge for push-driven extractors — the only framework module importing paho-mqtt eagerly (`module.py` reaches it only via a lazy import; the dependency ships as the `flechtwerk[mqtt]` extra). `MqttConnection` owns ONE paho client per process, driven entirely by the asyncio event loop via socket callbacks (no threads); `clean_session=False` + a stable client_id (the module-wide `client_id`) + `manual_ack=True` give at-least-once for QoS ≥ 1 publishers. Inbound messages route by topic filter to per-topic `MqttSubscription` views (buffer + pending-ACK list each); unmatched QoS 0 messages are warned and dropped, unmatched QoS ≥ 1 messages are **held un-ACKed** and re-routed the moment a matching subscription registers — the persistent session replays its backlog right after CONNACK, *before* the Kafka config bootstrap has subscribed anything, so ACKing (or dropping) there would silently destroy the very backlog `clean_session=False` protects. Held messages for a genuinely stale session subscription stall the session's inflight window — the documented cost of deferring unsubscribe-on-config-removal (remediation: a fresh `client_id` or broker-side session cleanup). No in-process reconnect: an unexpected disconnect surfaces as a `ConnectionError` from `drain()` once the buffer empties → crash → orchestrator restart. `MqttExtractor` opens the connection eagerly in `__aenter__`, sets the runner `wakeup` event (`on_message` fires it after buffering → sub-second delivery), and owns a template `poll()`: ACK-previous-batch (safe by the runner's re-entry contract) → `drain(drain_limit)` → per message `relay(config, topic, payload)` with JSON decoding. The `relay` hook returns a `Message` (forward + ACK with next batch), `None` (drop + immediate ACK), or raises (poison-drop: warn + ACK + counter — crashing would redeliver the poison forever at QoS 1). Build via `MqttExtractor.of(config_topics=..., relay=...)` or subclass; the `topic` config attribute (`flechtwerk.mqtt.TOPIC`) is framework-owned (one config record = one MQTT topic filter = one subscription). Sources that don't fit (stateful, 1:N, non-JSON) override `poll()` — the connection layer works without the template. Broker settings are injected via `Flechtwerk.of(mqtt=MqttBrokerConfig(...))`; the dataclass lives in `module.py` (reactor-di resolves annotations at decoration time, and the container must stay paho-free).
+- **`metrics.py` / `observer.py`**: Runners emit observer events (`message_in`, `message_out`, `transaction_committed`, `active_configs`, `config_message_in`, `config_store_entries` — the "did my config arrive?" gauge — `config_store_restored`, `state_restored`, `tasks_assigned`, `dispatch_scope`, `batch_scope`, `poll_cycle_scope`; MQTT: `mqtt_connected`, `mqtt_disconnected`, `mqtt_message_in`, `mqtt_message_dropped` with reason `filtered`/`poison`, `mqtt_buffered` — the MQTT `topic` label is always the subscription filter, bounded cardinality); `PrometheusObserver` translates these into prometheus-client metrics. Label *names* are caller-provided via `metrics_labels` — the framework itself is application-agnostic and knows nothing about what the labels are called. `metrics_port == 0` disables Prometheus and uses the no-op `Observer`.
+- **`testing.py`**: `FakeKafkaConsumer`/`FakeKafkaProducer` (duck-typed aiokafka subset), `make_record()` factory (real `aiokafka.ConsumerRecord` instances), `RecordingObserver`, `InMemoryStateStore`, and MQTT doubles (`FakeMqttConnection`/`FakeMqttSubscription` with an `acked` record, `make_mqtt_message()`; paho imports deferred so importing `testing` never loads paho). Pre-set `extractor.connection = FakeMqttConnection()` to test relays without a broker.
+
+The framework has no CLI, no module-level `os.getenv`, and no `load_dotenv` — all configuration is injected by the caller.
+
+## Key Architectural Patterns
+
+**Stateful Processing**: State lives in ephemeral RocksDB instances backed by a compacted Kafka changelog topic (Kafka Streams pattern — no PVC, pods are ephemeral). Generators yield `State` to persist; the runner only writes when state differs from current. Yielding empty/falsy `State` deletes the entry (Kafka tombstone). Stateless stages never open a RocksDB file. For transformers, state identity is *(input partition, extract_key)*: each task owns its own RocksDB store, writes its changelog entries to its own changelog partition (explicit-partition produce, not key hashing), and restores exactly that partition on assignment. The framework makes no assumptions about what `extract_key()` returns — a key produced from several partitions yields independent per-task entries (with the default `extract_key = msg.key`, Kafka's key partitioning makes this indistinguishable from a global key). Extractors keep a single global store restored once at startup.
+
+**Co-Partitioning Trap**: If one logical state entry must see records from *several* input topics, those topics must be co-partitioned by key — same key bytes, same partitioner, same partition count. Only the partition count is validated at startup; key/partitioner alignment cannot be checked and is the application's responsibility (exactly as in Kafka Streams). Get it wrong and the same `extract_key` arrives on different partition numbers, yielding independent state shards owned by different tasks — possibly on different instances. This is a silent split, not an error. Kafka Streams' DSL protects against the related case (key changed inside the topology) by auto-inserting a repartition topic; Flechtwerk is a Processor-API-level framework and has no equivalent — a mid-pipeline key change requires an explicit intermediate topic, i.e. another transformer hop. Output keys and changelog keys impose no constraints: output partitioning is decoupled from task identity, and changelog placement is explicit-partition, ignoring keys entirely.
+
+**The escape hatch for config lookups is a config topic**: when one side of the "join" is a config table rather than a keyed stream, declare it in `config_topics` instead of `input_topics` (Kafka Streams' GlobalKTable, specialized to configuration). Every instance reads it in full into the per-process `ConfigStore`; lookups go through `self.configs.get(wire_key)`; partition placement and count are irrelevant, so any producer — Kafka UI included — can write to it. The trade: the source topic must be compacted and stay small, the wire key is authoritative (tombstones carry no body), and lookups are eventually consistent, outside the task transaction.
+
+**Exactly-Once Delivery & Load Balancing**: Transformer work is split into **tasks** — one per input partition number, spanning that partition of every input topic (the consumer uses the Range assignor, which co-assigns same-numbered partitions). Each task owns a transactional producer with the static transactional ID `{application_id}-{partition}`; one Kafka transaction per task per batch covers that task's output messages, state changelog writes (the task's `ChangelogStateStore` shares its producer), and offset commits. Multiple instances are safe: when a partition moves, the new owner's `InitProducerId` fences the previous owner's producer and aborts its in-flight transaction (Kafka Streams EOS-v1 — aiokafka has no KIP-447 generation fencing), and state is re-restored from the changelog's last stable offset before processing resumes. On rebalance, all tasks are torn down and rebuilt — never retained, since a missed rebalance would make retained producers/stores silently stale. All framework consumers run `read_committed`. Constraints: all input topics of a transformer must have equal partition counts (validated at startup, matching changelog created); the partition count is frozen once state exists (repartitioning requires a state migration); instances beyond the partition count sit idle.
+
+**Extractors Are Single-Instance**: The multi-instance safety story above is transformer-only. Nothing fences concurrent extractor instances: each one reads *all* configs (`group_id=None`, no partition assignment), polls every external API redundantly, and writes the same state keys to a key-hashed changelog — last writer wins, so a slow instance can overwrite an advanced cursor with a stale one (silent re-import, not just duplicates). Extractor output is deliberately non-transactional (at-least-once — polling an external API can't be atomic with anything), so the transformer's fencing primitive doesn't exist here. Run exactly one replica per extractor; orchestrator restart latency is invisible at poll-interval timescales, so replicas buy no availability either. Sharding extractors along config-topic partitions was considered and rejected: extractor concurrency is already handled by `asyncio.gather` on one event loop, throughput is bounded by external API rate limits, and a single config's epoch backfill is a sequential cursor walk that partitioning can't split. The signal that would reopen this decision is a poll *cycle* duration approaching the poll *interval* (watch `poll_cycle_scope`); the cheap fix at that point is config-level sharding across deployments, not framework machinery.
+
+**"Let It Crash" Error Strategy**: No framework-level retry logic. Errors propagate; recovery is infrastructure: orchestrator restarts (e.g. Kubernetes `CrashLoopBackOff`), changelog replay restores state, transformer transactions catch any duplicates from partial writes. The key distinction is recoverable vs non-recoverable: only use try/except when the catch block can actually *remedy* the problem (e.g. refresh an expired token, skip a 400 on an endpoint that doesn't exist for this tenant). For transient errors like timeouts or 5xx, crash — sleeping and retrying in-process is reimplementing `CrashLoopBackOff` poorly. Never catch-and-skip data errors (silent data loss).
+
+## Invariant: config topics never participate in a Kafka transaction
+
+In a transformer, config topics must have no contact with any task
+transaction. This holds by construction, through three independent
+mechanisms — keep all of them intact:
+
+- **Separate, group-less consumer.** A transformer's config topics are read
+  by a dedicated `config_consumer` with `group_id=None` (`module.py`). No
+  consumer group means no committed offsets — config-topic offsets can never
+  appear in `send_offsets_to_transaction`. The offsets that DO enter a task
+  transaction are built exclusively from the main consumer's input-topic
+  batch, and `validate_topics` keeps `config_topics` disjoint from
+  `input_topics`, closing that path too.
+- **Updates land outside the transaction boundary.** `check_config_updates`
+  runs once per loop iteration, outside the batch lock, and only mutates the
+  in-memory `ConfigStore` — no task, producer, or transaction involved. The
+  fetch-then-drain-then-process sequencing gives every record of a batch one
+  consistent config snapshot; that is a scheduling courtesy, not
+  transactional coupling.
+- **No write path through the task producers.** The framework never produces
+  to a config topic; the store is fed only by `bootstrap_config_store` and
+  `drain_config_updates`.
+
+Lookups via `self.configs.get(...)` are therefore eventually consistent —
+Kafka Streams' GlobalKTable caveat, stated on `Transformer.configs` and in
+`configs.py`.
+
+### Why the config consumer still runs read_committed
+
+The isolation level is a consumption-side filter; it does not enroll config
+reads in any transaction, so it cannot violate the invariant above. For the
+normal case — non-transactional producers (ops tooling, Kafka UI) writing
+config — it makes no difference at all: records are visible immediately
+either way. It matters only when a *transactional* producer writes to a
+config topic (nothing forbids a transformer emitting an output `Message`
+onto one): `read_uncommitted` would apply records from aborted transactions
+to the store — and a startup bootstrap would compact them in until the next
+boot — while `read_committed` merely delays visibility until commit, which
+the eventually-consistent contract already absorbs. `read_committed` also
+gives `bootstrap_config_store` / `read_to_end` a well-defined end offset
+(the LSO). Switching to `read_uncommitted` buys nothing and opens the
+aborted-write hole — keep `read_committed`, matching every other framework
+consumer.
+
+## Invariant: the extractor runner's re-entry contract
+
+For any given config, `ExtractorRunner` re-enters `poll()` only after the
+previous invocation's yielded messages were sent to Kafka and the producer
+was flushed (`poll_one` awaits `send_batch` before returning; a send failure
+crashes the process). The MQTT template's ACK-the-previous-batch-at-the-top-
+of-the-next-poll pattern is correct *only* because of this ordering — do not
+weaken it. `test_reentry_contract_flush_strictly_precedes_next_poll` pins it.
+
+## Invariant: paho-mqtt stays confined to flechtwerk.mqtt
+
+- `flechtwerk.mqtt` is the only framework module that imports paho eagerly.
+  `module.py` must never import `.mqtt` at module level — the lazy import
+  inside the `configured_stage` factory is both what keeps `mqtt → module`
+  acyclic and what makes the `flechtwerk[mqtt]` optional extra work (an
+  application that never configures MQTT never loads paho).
+  `testing.py`'s MQTT doubles defer their paho imports for the same reason.
+- `MqttBrokerConfig` lives in `module.py`, not `mqtt.py`: reactor-di's
+  `@module` decorator resolves all class annotations at decoration time, so
+  the `mqtt: lookup[MqttBrokerConfig | None]` slot needs a runtime-importable,
+  paho-free name.
+- The framework reads no environment and does no identity defaulting: broker
+  settings arrive fully resolved through `Flechtwerk.of(mqtt=...)` (or
+  parent-module wiring), and the session identity is the module-wide
+  `client_id` (injected onto the stage by `configured_stage`; applications
+  typically pass a per-instance stable identity — e.g. the pod name in
+  Kubernetes). `MqttExtractor` rejects an empty `client_id` at startup
+  (MQTT 3.1.1 forbids one with a persistent session).
+
+## Boundary rule: which transport adapters belong in the framework
+
+Flechtwerk may own a transport adapter when its *correctness depends on runner
+delivery semantics* — MQTT qualifies because manual-ACK-after-Kafka-durable
+leans on the re-entry contract above. It must never own payload semantics,
+source-specific parsing, or per-source config schemas (those stay in
+application code); an adapter that would work identically as application code
+stays application code. Outbound MQTT (Kafka→MQTT command publishing) is
+explicitly out of scope for now: an MQTT publish can never join a Kafka
+transaction, so any future sink is at-least-once by construction and needs
+its own design. `MqttConnection` is deliberately direction-neutral so a sink
+can ride the same connection later.
+
+## Known limitation: no subscription lifecycle for removed configs
+
+Removing (tombstoning) or suspending a config deletes its runner entry, but
+nothing unsubscribes the MQTT topic: the broker session keeps the
+subscription, the in-process view keeps buffering, and — at QoS ≥ 1 — its
+un-ACKed messages occupy the shared session's inflight window until every
+slot is taken and the broker pauses delivery for ALL topics of this client.
+Messages held for a genuinely stale session subscription (a topic subscribed
+under this client_id in an earlier deployment and never unsubscribed) stall
+the window the same way. This is a known, deliberately deferred limitation:
+a correct fix needs a runner→stage config-removal hook, a broker
+UNSUBSCRIBE, and a decision about un-ACKed buffered messages (they were
+never written to Kafka). Until then: stop the publisher before removing a
+config, and recover a wedged session with a fresh `client_id` (a new broker
+session) or broker-side session cleanup.
+
+## graphify (optional, local)
+
+[graphify](https://pypi.org/project/graphifyy/) builds a queryable knowledge graph of this codebase (god nodes, community structure, cross-file relationships). It is a per-developer convenience, not part of the project: `graphify-out/` is gitignored, nothing in the build or CI depends on it, and its Claude Code hooks live in the machine-local `.claude/settings.local.json`. Skip this section entirely if `graphify-out/graph.json` does not exist.
+
+Initialization (once per clone, opt-in):
+
+```bash
+uv tool install graphifyy     # or: pip install graphifyy
+```
+
+Then run `/graphify .` in a Claude Code session to build the graph, and optionally `graphify hook install` to add git post-commit/post-checkout hooks that rebuild it automatically (AST-only, no API cost).
+
+Rules (only when graphify-out/graph.json exists):
+
+- For codebase questions, first run `graphify query "<question>"`. Use `graphify path "<A>" "<B>"` for relationships and `graphify explain "<concept>"` for focused concepts. These return a scoped subgraph, usually much smaller than GRAPH_REPORT.md or raw grep output.
+- If graphify-out/wiki/index.md exists, use it for broad navigation instead of raw source browsing.
+- Read graphify-out/GRAPH_REPORT.md only for broad architecture review or when query/path/explain do not surface enough context.
+- After modifying code, run `graphify update .` to keep the graph current (AST-only, no API cost).

flechtwerk-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 b.sure GmbH
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

flechtwerk-0.1.0/PKG-INFO

@@ -0,0 +1,292 @@
+Metadata-Version: 2.4
+Name: flechtwerk
+Version: 0.1.0
+Summary: Truly async Python stream processing with real Kafka transactions for exactly-once delivery, and an MQTT→Kafka bridge that ACKs only after Kafka has the data
+Project-URL: Homepage, https://github.com/bsure-analytics/flechtwerk
+Project-URL: Issues, https://github.com/bsure-analytics/flechtwerk/issues
+Project-URL: Repository, https://github.com/bsure-analytics/flechtwerk
+Author: Christian Schlichtherle, Viktor Penev
+Maintainer: b.sure GmbH
+License: MIT
+License-File: LICENSE
+Keywords: aiokafka,asyncio,exactly-once,kafka,kafka-streams,mqtt,stream-processing
+Classifier: Development Status :: 4 - Beta
+Classifier: Framework :: AsyncIO
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: System :: Distributed Computing
+Requires-Python: >=3.14
+Requires-Dist: aiokafka[zstd]~=0.13
+Requires-Dist: prometheus-client~=0.21
+Requires-Dist: reactor-di~=0.5
+Requires-Dist: rocksdict~=0.3
+Provides-Extra: mqtt
+Requires-Dist: paho-mqtt~=2.1; extra == 'mqtt'
+Description-Content-Type: text/markdown
+
+# Flechtwerk
+
+[![CI](https://github.com/bsure-analytics/flechtwerk/actions/workflows/ci.yaml/badge.svg)](https://github.com/bsure-analytics/flechtwerk/actions/workflows/ci.yaml)
+[![PyPI version](https://img.shields.io/pypi/v/flechtwerk.svg)](https://pypi.org/project/flechtwerk/)
+[![Python versions](https://img.shields.io/pypi/pyversions/flechtwerk.svg)](https://pypi.org/project/flechtwerk/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+Truly async Python stream processing with real Kafka transactions for exactly-once delivery, and an MQTT→Kafka bridge that ACKs only after Kafka has the data
+
+## What it is
+
+Flechtwerk is a small async stream processing framework for Kafka. It takes the operational design that Kafka Streams nailed a decade ago — consumer groups for partition assignment, compacted changelog topics as the durable state of record, Kafka transactions tying state writes, output messages, and offset commits into a single atomic unit — and ports it to modern async Python.
+
+If you've run Kafka Streams in production, the model is immediately familiar: stateful operators backed by RocksDB, recovery via changelog replay, exactly-once delivery via transactions, ephemeral compute that can be killed and rescheduled freely because all durable state lives in Kafka.
+
+## Why it exists
+
+Existing Python options each fail one of the constraints that matter for I/O-bound, transactional, multi-instance stream processing:
+
+- **Faust**: stateful but RocksDB + multi-instance recovery is fragile, and "exactly-once" is idempotent-producer-plus-careful-offsets rather than real Kafka transactions spanning state and output.
+- **Quix Streams**: pleasant API, but the core loop is synchronous — fatal for workloads driven by concurrent async I/O (HTTP polling, MQTT subscriptions, etc.).
+- **Bytewax**: a Rust dataflow engine with Python bindings; excellent for CPU-bound partitioned dataflow, awkward for async I/O and heavier than the operational model needs.
+- **Apache Beam (on Flink)**: the Python SDK runs in a separate worker process and shuttles data to JVM operators over gRPC via the Beam portability framework. Setup is a maze of portable runners and SDK harnesses; failures span two runtimes and produce errors that are hard to localize. If Flink is the right answer, Java or Scala is a saner way to reach it.
+
+Flechtwerk assumes Python 3.14, `asyncio`, `aiokafka`, and `uvloop` are the right primitives and builds directly on them.
+
+## Installation
+
+```bash
+pip install flechtwerk            # or: uv add flechtwerk
+pip install "flechtwerk[mqtt]"    # with the MQTT→Kafka bridge (paho-mqtt)
+```
+
+Python 3.14+. Runtime dependencies: `aiokafka[zstd]`, `prometheus-client`, `reactor-di`, and `rocksdict`. Run it on `uvloop` for best throughput — the framework works on stock `asyncio` (and therefore on Windows), but the event loop is the application's choice.
+
+## The API
+
+The whole contract is two `yield` statements inside an async generator:
+
+- `yield Message(...)` to emit an output record.
+- `yield State(...)` to persist state for the current key.
+  - `yield <falsy State>` to tombstone the key.
+
+That's it. There are no agents, no tables, no DSL, no `@app.topic` decorators, no fluent builders. An async generator is already the right shape — pull-based, backpressure-friendly, naturally composable — and every Python developer already knows how to read one.
+
+```python
+from collections.abc import AsyncIterator
+
+from flechtwerk import Event, IncomingMessage, Message, State, Transformer
+from flechtwerk.attribute import Attribute, DATETIME, INT
+
+SEEN = Attribute("seen", INT)
+"""How many events this key has produced so far."""
+TIMESTAMP = Attribute("timestamp", DATETIME)
+"""When the event happened at the source."""
+
+
+async def transform(msg: IncomingMessage, state: State) -> AsyncIterator[Message | State]:
+    seen = (state.get(SEEN) or 0) + 1
+    yield Message(key=msg.key, topic="my-output", value=Event({**msg.value, SEEN: seen}))
+    yield State({SEEN: seen, TIMESTAMP: msg.value[TIMESTAMP]})
+
+
+stage = Transformer.of(input_topics=["my-input"], transform=transform)
+```
+
+The typed `Attribute` handles indexing those records are explained in [Typed records, not bare dicts](#typed-records-not-bare-dicts) below.
+
+An `Extractor` is the same two-yield contract driven from the other end: `poll(config, state)` runs once per config record per poll cycle, pulls from the external source, and uses `State` as its resume cursor:
+
+```python
+from collections.abc import AsyncIterator
+from datetime import datetime, timezone
+
+from flechtwerk import Config, Event, Extractor, Message, State
+from flechtwerk.attribute import Attribute, DATETIME, INT, STR
+
+CYCLE = Attribute("cycle", INT)
+"""Resume cursor — stands in for whatever your source pages by."""
+NAME = Attribute("name", STR)
+POLLED_AT = Attribute("polled_at", DATETIME)
+
+
+async def poll(config: Config, state: State) -> AsyncIterator[Message | State]:
+    cycle = (state.get(CYCLE) or 0) + 1               # your API call goes here
+    yield Message(
+        key=config[NAME],
+        topic="my-extract",
+        value=Event({CYCLE: cycle, POLLED_AT: datetime.now(timezone.utc)}),
+    )
+    yield State({CYCLE: cycle})
+
+
+stage = Extractor.of(config_topics=["my-config"], poll=poll)
+```
+
+`Extractor` and `Transformer` are ABCs. Use the `.of(...)` factory for stateless or simply-stateful stages, or subclass directly when you need lifecycle management (HTTP clients, dedup instances, etc.) via `__aenter__` / `__aexit__`. Stateless stages simply never yield `State` and never open a RocksDB file.
+
+Running a stage is one call — all configuration is injected, nothing is read from the environment:
+
+```python
+import asyncio
+
+from flechtwerk import Flechtwerk
+
+
+async def main() -> None:
+    await Flechtwerk.of(
+        application_id="my-transformer",
+        bootstrap_servers="localhost:9092",
+        client_id="my-transformer-0",   # process identity: unique per instance, stable across restarts
+        poll_interval_seconds=60,
+        stage=stage,                    # from above
+    ).run()
+
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+This plus one stage definition from above is the whole program — point it at any Kafka broker.
+
+### Typed records, not bare dicts
+
+You met `Attribute` above. It exists because a stream processor lives on the JSON boundary: every input is a dict decoded from the wire, and every output and every state write goes back through `json.dumps`. Handled as bare dicts, that boundary leaks into everything — each read re-checks presence and re-parses timestamps, a `datetime` assigned three hops earlier blows up only when the record is finally serialized, and a field that silently became `null` surfaces as a `KeyError` in some consumer far from the code that dropped it.
+
+The `flechtwerk.attribute` library moves all of that to the write site. Each field is declared exactly once, as a typed handle pairing a wire name with an explicit `Codec[V]`:
+
+```python
+from datetime import datetime, timezone
+
+from flechtwerk import Event
+from flechtwerk.attribute import Attribute, DATETIME, LIST, STR
+
+DEVICE = Attribute("device", STR)
+LAST_SEEN = Attribute("last_seen", DATETIME)
+TAGS = Attribute("tags", LIST(STR), optional=True)
+
+event = Event({
+    DEVICE: "sensor-1",
+    LAST_SEEN: datetime(2026, 7, 12, 9, 30, tzinfo=timezone.utc),
+    TAGS: ["a", "b"],
+})
+
+event[LAST_SEEN]   # datetime(2026, 7, 12, 9, 30, tzinfo=timezone.utc) — a real datetime
+event.raw          # {'device': 'sensor-1', 'last_seen': '2026-07-12T09:30:00Z', 'tags': ['a', 'b']}
+
+# Both of these raise at the write site, not at serialization time:
+#   event[DEVICE] = 42     # expected str, got int
+#   event[DEVICE] = None   # cannot assign None to required Attribute('device')
+```
+
+`Event`, `State`, and `Config` are `Record` subclasses — dict-like containers indexed by these handles rather than string keys. (`Message` is a frozen dataclass envelope carrying a key, topic, `Event` value, and optional timestamp.) The codec runs on **every write**, so the underlying `.raw` payload stays JSON-native by construction: wire encoding is a straight `json.dumps(event.raw)`, decoding is a straight `Event.wrap(raw)`, and nothing in between ever needs to re-validate. A required attribute (the default) rejects `None` so a dropped value can't silently land as JSON `null`; declare fields where absence is legal with `optional=True`. The read distinction is carried by the method, not the declaration — `event[LAST_SEEN]` reads-or-raises, `event.get(TAGS)` tolerates absence and returns `V | None`.
+
+Codecs compose: atoms (`STR`, `INT`, `BOOL`, `DATE`, `FLOAT`, `DATETIME`, `TIME`, `RECORD`, `ANY`) plus constructors (`LIST(V)`, `SET(V)`, `TUPLE(V)`, `DICT(V)`). And records spread like dicts — `Event({**event, LAST_SEEN: later})` — so enrichment never has to mutate its input.
+
+The point isn't ceremony; it's that the boundary between "Python object graph" and "JSON on the wire" is enforced at assignment time, once per Attribute declaration, rather than re-derived on every serialize/deserialize.
+
+### Config topics — shared lookup tables
+
+A stage declares two kinds of topics. `input_topics` (transformers only) are partitioned: their records drive `transform()` and define the task model. `config_topics` are read **in full by every instance** into one per-process `ConfigStore` keyed by wire key — Kafka Streams' GlobalKTable, specialized to configuration:
+
+```python
+from collections.abc import AsyncIterator
+
+from flechtwerk import Extractor, IncomingMessage, Message, State, Transformer
+
+
+class MyExtractor(Extractor):
+    config_topics = ["my-config"]          # an extractor's inputs ARE config topics
+    ...                                    # plus your poll()
+
+
+class RequestDriven(Transformer):
+    input_topics = ["my-requests"]         # partitioned, keyed stream
+    config_topics = ["my-config"]          # config table, joined by key
+
+    async def transform(self, msg: IncomingMessage, state: State) -> AsyncIterator[Message | State]:
+        config = self.configs.get(msg.key)  # eventually consistent lookup
+        if config is None:
+            return                          # no config for this key (yet)
+        yield Message(key=msg.key, topic="my-results", value=msg.value)
+
+
+stage = RequestDriven()
+```
+
+For extractors this is not an extra mechanism but the baseline: config topics are the only Kafka input an extractor has. For transformers it is the escape hatch from the co-partitioning requirement: a config topic's partition placement and count are irrelevant, so any producer (Kafka UI included) can write configs without routing them to the "right" partition. The source topics are their own changelog — compacted, small, re-read on every startup — and lookups are eventually consistent, outside the task transaction (the GlobalKTable caveat).
+
+`Stage.enrich(config)` hooks one-time derivation (e.g. an API lookup) into the config path: the framework applies it once per config record — never per poll tick or lookup — and both stage kinds inherit it. Kafka Streams forbids transforming records on their way into a global store (KIP-813) because a checkpoint-based restore would bypass the transformation; Flechtwerk re-reads the topics through the same enrich path on every startup, so the enriched store cannot diverge.
+
+### MQTT sources — push into the poll loop
+
+`flechtwerk.mqtt` bridges a push-driven MQTT source into the extractor model out of the box. The framework owns everything protocol-shaped — one shared paho connection per process driven by the asyncio event loop (no threads), persistent sessions with a stable client id, manual-ACK at-least-once (a batch is ACKed to the broker only once it is provably durable in Kafka — at the top of the next poll, per the runner's re-entry contract), per-topic subscriptions fed by config records, an arrival wakeup so delivery latency is sub-second rather than poll-interval-bound, and Prometheus metrics. An application writes one pure function:
+
+```python
+from datetime import datetime, timezone
+
+from flechtwerk import Config, Event, Message
+from flechtwerk.attribute import Attribute, DATETIME, RECORD, Record, STR
+from flechtwerk.mqtt import MqttExtractor
+
+DATA = Attribute("data", RECORD)
+DEVICE_ID = Attribute("device_id", STR)
+PROCESSING_TIME = Attribute("processing_time", DATETIME)
+
+
+def relay(config: Config, topic: str, payload: Record) -> Message | None:
+    return Message(
+        key=payload[DEVICE_ID],             # missing → the framework poison-drops
+        topic="my-extract",
+        value=Event({DATA: payload, PROCESSING_TIME: datetime.now(timezone.utc)}),
+    )
+
+
+stage = MqttExtractor.of(config_topics=["my-config"], relay=relay)
+```
+
+Return a `Message` to forward, `None` to drop (ACKed immediately), or raise to poison-drop (logged, ACKed, counted — never a crash loop on a broken payload). Sources that don't fit the one-in-at-most-one-out shape override `poll()`; the connection layer works without the template. Broker settings are injected via `Flechtwerk.of(mqtt=MqttBrokerConfig(...))`, and paho stays confined to `flechtwerk.mqtt` — `import flechtwerk` never loads it, and the dependency ships as the optional `flechtwerk[mqtt]` extra.
+
+## Operational model
+
+- **Consumer groups** drive partition assignment and rebalancing — standard Kafka semantics, no custom coordination.
+- **Changelog topics** (compacted) are the durable state of record. RocksDB is a local cache rebuilt by replay on startup. Pods are ephemeral; no PVCs.
+- **Kafka transactions** span all output messages, all state changelog writes, and offset commits for a single processing batch. Transformer work is split into per-input-partition tasks; each task owns a transactional producer (static transactional ID — EOS-v1 fencing) shared with its changelog state store via DI — closing the gap that lets duplicates leak in frameworks that treat these as separate concerns.
+- **Per-batch parallelism by state key.** Within a `getmany()` batch, records are bucketed by state key within each task. Buckets run concurrently via `asyncio.gather` so I/O-bound `transform` calls overlap, while records sharing a key run serially inside their bucket — each one sees the previous one's yielded state. Each `transform` call receives a defensive deepcopy of the running state, so in-place mutation without a `yield State(...)` can't leak. Cross-key ordering is not preserved; within a bucket, records appear in `input_topics` order then Kafka offset order.
+- **"Let it crash"** error strategy. The line is recoverable vs non-recoverable, not transient vs persistent: catch only when the handler can actually *remedy* the problem (refresh an expired token, skip a 400 on an endpoint that doesn't exist for this tenant). Timeouts and 5xx crash — sleeping and retrying in-process is reimplementing `CrashLoopBackOff` poorly. Never catch-and-skip data errors; that's silent data loss. Recovery is infrastructure: Kubernetes restart, changelog replay, transaction abort.
+
+## Architecture
+
+Hexagonal: ports and adapters. The core (`Extractor`, `Transformer`, their runners) depends only on abstract ports (`StateStore`, `Observer`, `aiokafka` consumer/producer interfaces). Adapters (`RocksDBStateStore`, `ChangelogStateStore`, `PrometheusObserver`, the `flechtwerk.mqtt` transport, `FakeKafkaConsumer`/`FakeKafkaProducer`/`FakeMqttConnection` for tests) plug in via DI through `reactor-di`. Transport adapters earn a place in the framework only when their correctness depends on runner delivery semantics — MQTT's manual-ACK protocol does; a plain HTTP poller does not.
+
+The framework has no CLI, no module-level `os.getenv`, no `load_dotenv`, and no opinions about how applications are packaged or deployed. All configuration is injected by the caller.
+
+## Local development falls out for free
+
+Because all durable state lives in Kafka — input topics, output topics, and compacted changelog topics — replicating production state on a developer laptop is just a matter of mirroring the relevant topics and committed consumer-group offsets into a local Kafka cluster. A locally-run stage then starts up against the local cluster, replays the changelog into a fresh RocksDB, and resumes processing exactly where its production counterpart left off.
+
+No PVCs to snapshot, no opaque local state directories to copy, no VPN into a shared dev cluster. The same property that makes Kubernetes pods disposable in production makes them reproducible on a laptop. Frameworks that hide state in framework-managed local stores cannot offer this cleanly; the Kafka Streams model can, and Flechtwerk inherits it.
+
+This is a property of the model, not a feature of the framework — building a small mirror script is straightforward and lives in application code.
+
+## What's deliberately not here
+
+- **Event-time windowing with watermarks** — if you need this, use Flink. Flechtwerk targets the much larger class of problems where processing-time semantics are sufficient.
+- **Stream-stream joins, complex topologies** — operators compose by writing to and reading from intermediate Kafka topics, not by chaining method calls.
+- **Savepoints / state migrations** — recovery is changelog replay; schema evolution is the application's responsibility.
+- **A DSL** — the stream-processing vocabulary is `Extractor`, `Transformer`, `Message`, `State`, `Event`, `Config`, `ConfigStore`, plus the typed-record handles of `flechtwerk.attribute`. That's it.
+
+## Development
+
+```bash
+uv sync                        # venv + all dependencies
+uv run pytest                  # unit tier — Docker-free
+uv run pytest -m integration   # integration tier — ephemeral Kafka/Mosquitto via testcontainers
+uv run coverage run -m pytest -m "integration or not integration" && uv run coverage report
+```
+
+Releases are cut by tagging: pushing a `vX.Y.Z` tag runs the test suite, builds the package with the tag-derived version (hatch-vcs), and publishes it to PyPI via trusted publishing.
+
+## Status
+
+Flechtwerk was extracted from the data integration platform it was developed in, where it runs every stage in production. The API is small and settled in shape, but pre-1.0 — minor releases may still move things around. One design rule carries over from its origin as an embedded framework: framework code references no application paths, modules, or environment variables; all configuration is injected by the caller.