sqllocks-shape-kafka 0.9.1__tar.gz

sqllocks_shape_kafka-0.9.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025-2026 SQLLocks (Jonathan Stewart)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

sqllocks_shape_kafka-0.9.1/PKG-INFO

@@ -0,0 +1,125 @@
+Metadata-Version: 2.4
+Name: sqllocks-shape-kafka
+Version: 0.9.1
+Summary: Shape plugin: Kafka stream source and event emitter
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/sqllocks/shape
+Project-URL: Repository, https://github.com/sqllocks/shape
+Project-URL: Issues, https://github.com/sqllocks/shape/issues
+Project-URL: Documentation, https://github.com/sqllocks/shape/tree/main/plugins/shape-kafka
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Database
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: sqllocks-shape==0.9.1
+Requires-Dist: confluent-kafka<4,>=2.5
+Provides-Extra: avro
+Requires-Dist: fastavro>=1.9; extra == "avro"
+Provides-Extra: protobuf
+Requires-Dist: protobuf>=5; extra == "protobuf"
+Dynamic: license-file
+
+# sqllocks-shape-kafka
+
+Shape plugin: a Kafka stream source (`shape.stream_sources`, scheme `kafka://`).
+
+```
+pip install sqllocks-shape-kafka          # or: pip install 'sqllocks-shape[kafka]'
+shape stream-profile kafka://broker:9092/orders -o orders.json
+```
+
+On Alpine/musllinux, `confluent-kafka` has no ARM64 or x86_64 wheels and builds from
+source. Installing this plugin, `sqllocks-shape[kafka]` or `sqllocks-shape[all]` needs
+librdkafka **2.16 or newer**, development headers and C build tools. Alpine's
+`librdkafka-dev` is currently older. Build the library first using the
+[Alpine installation steps](../../docs/INSTALL.md#alpine-and-musllinux), which match
+`scripts/release_smoke_alpine.sh`, or
+[Confluent's build instructions](https://github.com/confluentinc/librdkafka#building).
+
+`kafka://host1:9092,host2:9092/TOPIC` reads one topic as Arrow micro-batches. Message bodies are
+JSON objects; their fields become columns, and `_shape_event_time` carries the event time (the
+body's `_shape_event_time`, or the message timestamp). Shape keeps its own position: the consumer
+is *assigned* every partition at explicit offsets, nothing is committed to the broker, and a
+`StreamOffset` is `{partition: next offset}`, so a checkpoint resumes exactly where it stopped.
+
+Options, offsets, delivery guarantees and the checkpoint format are described in
+`docs/plugins/streaming.md`. Extra `confluent-kafka` settings (SASL, TLS) go in through the
+`config` source option, preferably from a file (`--options-file`).
+
+## Emitting events: `kafka://`
+
+```
+shape emit retail --realtime --rate 5000 --sink kafka://broker:9092/retail-events
+```
+
+`shape.emitters` `kafka`: one message per event, body = the event JSON (flat, or `--envelope
+cloudevents`), **message key = the idempotency key `<table>/<seq>`**, header `shape-table`. The
+producer uses `acks=all` and `enable.idempotence=true` (overridable through the `config` option, for
+SASL/TLS and the like). `emit` returns after the broker acknowledged every message, so the
+runtime's checkpoint never passes an undelivered event; a failed delivery is retried by the runtime
+(a message may arrive twice: keep the first of each key). A full local queue is waited for.
+See `docs/EMIT.md`.
+
+### Registry formats: Avro, Protobuf, JSON Schema
+
+```
+pip install 'sqllocks-shape-kafka[avro]'      # fastavro; or [protobuf]; json-schema needs nothing
+shape emit retail --sink kafka://broker:9092/orders --event-format avro \
+    --sink-config kafka.schema_registry_url=https://registry:8081
+```
+
+`--event-format json|avro|protobuf|json-schema` (default `json`, today's bytes). The emitter
+derives the schema from the table's Arrow schema, registers it with a Confluent-compatible registry
+(`kafka.schema_registry_url`; `kafka.schema_registry_username` and
+`kafka.schema_registry_password=env://NAME`, a credential reference; `kafka.subject_strategy` =
+`topic` (default, `<topic>-value`), `record` or `topic_record`) and writes the Confluent wire format:
+byte 0, the 4-byte big-endian schema id, the payload (Protobuf: a zero byte for the message index
+`[0]` first). The record (message) name is the table name in the namespace/package `shape.events`;
+the table and its columns must be valid identifiers for Avro and Protobuf. The key stays
+`<table>/<seq>`. The encoders are imported on first use; a missing one stops the run (exit 2) with the
+`pip install` command above.
+
+Type mapping (a nullable Arrow field is a nullable field: an Avro union with `null`, a Protobuf
+`optional`, a JSON `null`; `_shape_table` and `_shape_seq` are never null; a float is always nullable
+because NaN and infinities are sent as `null`, as in the flat JSON event):
+
+| Arrow | Avro | Protobuf | JSON Schema |
+|---|---|---|---|
+| bool | `boolean` | `bool` | `boolean` |
+| int8, int16, int32 | `int` | `int32` | `integer` |
+| uint8, uint16 | `int` | `uint32` | `integer` |
+| int64, uint32 | `long` | `int64`, `uint32` | `integer` |
+| uint64 | `long` (a value above 2^63-1 cannot be encoded) | `uint64` | `integer` |
+| float32, float64 | `float`, `double` | `float`, `double` | `number` |
+| string | `string` | `string` | `string` |
+| uuid (the Arrow extension type) | `string`, logical type `uuid` | `string` | `string`, format `uuid` |
+| binary | `bytes` | `bytes` | `string`, `contentEncoding: base64` |
+| decimal(p, s) | `bytes`, logical type `decimal(p, s)` | `string` (canonical text) | `string` (canonical text) |
+| date | `int`, logical type `date` | `int32` (days since 1970-01-01) | `string`, format `date` |
+| time | `time-millis` (s, ms) or `time-micros` | `int64` (microseconds) | `string` |
+| timestamp, no time zone | `local-timestamp-millis` (s, ms) or `-micros` (us, ns) | `int64` (wall-clock microseconds since the epoch) | `string`, "timestamp without time zone" |
+| timestamp with time zone | `timestamp-millis` (s, ms) or `-micros` (us, ns) | `int64` (UTC microseconds since the epoch) | `string`, format `date-time` |
+
+A nanosecond value is cut to microseconds. A dictionary column is its value type. Lists, structs and
+maps are not mapped (the table is refused with exit 2; use `json`). Protobuf field numbers follow the
+column order. Tests decode every message back from the registry's copy of the schema and compare it
+with the flat event (`shape_kafka.testing.FakeRegistry`, `decode_messages`).
+
+Tests: contract tests run on every PR against in-memory fakes
+(`shape_kafka.testing.FakeBroker`, `FakeProducer`; the emitter contract is
+`shape.streaming.emit.contract`); `pytest -m emulator plugins/shape-kafka/tests` runs the
+end-to-end tests against a real broker (`docker compose -f ci/emulators/docker-compose.yml up -d
+--wait kafka`), nightly in CI.
+
+Its version always equals core's (`sqllocks-shape`), and it is released together with core.
+How plugins are written: `docs/plugins/authoring.md` in the repository.
+
+Live tests (`pytest -m live plugins/shape-kafka/tests`) need `KAFKA_SERVERS`, `KAFKA_TOPIC` and,
+optionally, `KAFKA_CONFIG` (JSON); the nightly job runs them only where those secrets exist.

sqllocks_shape_kafka-0.9.1/README.md

@@ -0,0 +1,97 @@
+# sqllocks-shape-kafka
+
+Shape plugin: a Kafka stream source (`shape.stream_sources`, scheme `kafka://`).
+
+```
+pip install sqllocks-shape-kafka          # or: pip install 'sqllocks-shape[kafka]'
+shape stream-profile kafka://broker:9092/orders -o orders.json
+```
+
+On Alpine/musllinux, `confluent-kafka` has no ARM64 or x86_64 wheels and builds from
+source. Installing this plugin, `sqllocks-shape[kafka]` or `sqllocks-shape[all]` needs
+librdkafka **2.16 or newer**, development headers and C build tools. Alpine's
+`librdkafka-dev` is currently older. Build the library first using the
+[Alpine installation steps](../../docs/INSTALL.md#alpine-and-musllinux), which match
+`scripts/release_smoke_alpine.sh`, or
+[Confluent's build instructions](https://github.com/confluentinc/librdkafka#building).
+
+`kafka://host1:9092,host2:9092/TOPIC` reads one topic as Arrow micro-batches. Message bodies are
+JSON objects; their fields become columns, and `_shape_event_time` carries the event time (the
+body's `_shape_event_time`, or the message timestamp). Shape keeps its own position: the consumer
+is *assigned* every partition at explicit offsets, nothing is committed to the broker, and a
+`StreamOffset` is `{partition: next offset}`, so a checkpoint resumes exactly where it stopped.
+
+Options, offsets, delivery guarantees and the checkpoint format are described in
+`docs/plugins/streaming.md`. Extra `confluent-kafka` settings (SASL, TLS) go in through the
+`config` source option, preferably from a file (`--options-file`).
+
+## Emitting events: `kafka://`
+
+```
+shape emit retail --realtime --rate 5000 --sink kafka://broker:9092/retail-events
+```
+
+`shape.emitters` `kafka`: one message per event, body = the event JSON (flat, or `--envelope
+cloudevents`), **message key = the idempotency key `<table>/<seq>`**, header `shape-table`. The
+producer uses `acks=all` and `enable.idempotence=true` (overridable through the `config` option, for
+SASL/TLS and the like). `emit` returns after the broker acknowledged every message, so the
+runtime's checkpoint never passes an undelivered event; a failed delivery is retried by the runtime
+(a message may arrive twice: keep the first of each key). A full local queue is waited for.
+See `docs/EMIT.md`.
+
+### Registry formats: Avro, Protobuf, JSON Schema
+
+```
+pip install 'sqllocks-shape-kafka[avro]'      # fastavro; or [protobuf]; json-schema needs nothing
+shape emit retail --sink kafka://broker:9092/orders --event-format avro \
+    --sink-config kafka.schema_registry_url=https://registry:8081
+```
+
+`--event-format json|avro|protobuf|json-schema` (default `json`, today's bytes). The emitter
+derives the schema from the table's Arrow schema, registers it with a Confluent-compatible registry
+(`kafka.schema_registry_url`; `kafka.schema_registry_username` and
+`kafka.schema_registry_password=env://NAME`, a credential reference; `kafka.subject_strategy` =
+`topic` (default, `<topic>-value`), `record` or `topic_record`) and writes the Confluent wire format:
+byte 0, the 4-byte big-endian schema id, the payload (Protobuf: a zero byte for the message index
+`[0]` first). The record (message) name is the table name in the namespace/package `shape.events`;
+the table and its columns must be valid identifiers for Avro and Protobuf. The key stays
+`<table>/<seq>`. The encoders are imported on first use; a missing one stops the run (exit 2) with the
+`pip install` command above.
+
+Type mapping (a nullable Arrow field is a nullable field: an Avro union with `null`, a Protobuf
+`optional`, a JSON `null`; `_shape_table` and `_shape_seq` are never null; a float is always nullable
+because NaN and infinities are sent as `null`, as in the flat JSON event):
+
+| Arrow | Avro | Protobuf | JSON Schema |
+|---|---|---|---|
+| bool | `boolean` | `bool` | `boolean` |
+| int8, int16, int32 | `int` | `int32` | `integer` |
+| uint8, uint16 | `int` | `uint32` | `integer` |
+| int64, uint32 | `long` | `int64`, `uint32` | `integer` |
+| uint64 | `long` (a value above 2^63-1 cannot be encoded) | `uint64` | `integer` |
+| float32, float64 | `float`, `double` | `float`, `double` | `number` |
+| string | `string` | `string` | `string` |
+| uuid (the Arrow extension type) | `string`, logical type `uuid` | `string` | `string`, format `uuid` |
+| binary | `bytes` | `bytes` | `string`, `contentEncoding: base64` |
+| decimal(p, s) | `bytes`, logical type `decimal(p, s)` | `string` (canonical text) | `string` (canonical text) |
+| date | `int`, logical type `date` | `int32` (days since 1970-01-01) | `string`, format `date` |
+| time | `time-millis` (s, ms) or `time-micros` | `int64` (microseconds) | `string` |
+| timestamp, no time zone | `local-timestamp-millis` (s, ms) or `-micros` (us, ns) | `int64` (wall-clock microseconds since the epoch) | `string`, "timestamp without time zone" |
+| timestamp with time zone | `timestamp-millis` (s, ms) or `-micros` (us, ns) | `int64` (UTC microseconds since the epoch) | `string`, format `date-time` |
+
+A nanosecond value is cut to microseconds. A dictionary column is its value type. Lists, structs and
+maps are not mapped (the table is refused with exit 2; use `json`). Protobuf field numbers follow the
+column order. Tests decode every message back from the registry's copy of the schema and compare it
+with the flat event (`shape_kafka.testing.FakeRegistry`, `decode_messages`).
+
+Tests: contract tests run on every PR against in-memory fakes
+(`shape_kafka.testing.FakeBroker`, `FakeProducer`; the emitter contract is
+`shape.streaming.emit.contract`); `pytest -m emulator plugins/shape-kafka/tests` runs the
+end-to-end tests against a real broker (`docker compose -f ci/emulators/docker-compose.yml up -d
+--wait kafka`), nightly in CI.
+
+Its version always equals core's (`sqllocks-shape`), and it is released together with core.
+How plugins are written: `docs/plugins/authoring.md` in the repository.
+
+Live tests (`pytest -m live plugins/shape-kafka/tests`) need `KAFKA_SERVERS`, `KAFKA_TOPIC` and,
+optionally, `KAFKA_CONFIG` (JSON); the nightly job runs them only where those secrets exist.

sqllocks_shape_kafka-0.9.1/pyproject.toml

@@ -0,0 +1,43 @@
+[build-system]
+requires = ["setuptools>=77"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "sqllocks-shape-kafka"
+# Lockstep with core (T-09): scripts/check_plugin_skeletons.py fails when this drifts.
+version = "0.9.1"
+description = "Shape plugin: Kafka stream source and event emitter"
+readme = "README.md"
+requires-python = ">=3.11"
+license = "MIT"
+license-files = ["LICENSE"]
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3 :: Only",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
+    "Operating System :: OS Independent",
+    "Topic :: Database",
+]
+dependencies = ["sqllocks-shape==0.9.1", "confluent-kafka>=2.5,<4"]
+
+[project.urls]
+Homepage = "https://github.com/sqllocks/shape"
+Repository = "https://github.com/sqllocks/shape"
+Issues = "https://github.com/sqllocks/shape/issues"
+Documentation = "https://github.com/sqllocks/shape/tree/main/plugins/shape-kafka"
+
+[project.optional-dependencies]
+avro = ["fastavro>=1.9"]
+protobuf = ["protobuf>=5"]
+
+[project.entry-points."shape.stream_sources"]
+kafka = "shape_kafka:KafkaStreamSource"
+
+[project.entry-points."shape.emitters"]
+kafka = "shape_kafka:KafkaEmitter"
+
+[tool.setuptools.packages.find]
+where = ["src"]

sqllocks_shape_kafka-0.9.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

sqllocks_shape_kafka-0.9.1/src/shape_kafka/__init__.py

@@ -0,0 +1,17 @@
+"""Shape plugin: Kafka.
+
+* ``shape.stream_sources``: ``kafka`` reads a topic (``kafka://host:9092/topic``) as Arrow
+  micro-batches with ``{partition: next offset}`` offsets, for ``shape stream-profile``.
+
+* ``shape.emitters``: ``kafka`` sends events to a topic (``kafka://host:9092/topic``), one
+  message per event keyed by the idempotency key ``<table>/<seq>``, for ``shape emit``.
+
+``confluent-kafka`` loads only when a read or a send starts.
+"""
+
+from .emitter import KafkaEmitter
+from .source import KafkaStreamSource
+
+SHAPE_API = "1.0"
+
+__all__ = ["SHAPE_API", "KafkaEmitter", "KafkaStreamSource"]

sqllocks_shape_kafka-0.9.1/src/shape_kafka/emitter.py

@@ -0,0 +1,361 @@
+"""The ``kafka://`` emitter (P5-02).
+
+``kafka://broker1:9092,broker2:9092/topic`` sends each event as one message: the body is the
+event's JSON (flat, or the CloudEvents envelope), the **message key is the D-12 idempotency key**
+``<table>/<seq>`` (so a replay of a row lands in the same partition with the same key, and a
+log-compacted topic keeps one message per row), and the header ``shape-table`` names the table.
+
+Delivery is at-least-once. ``emit`` returns after the broker has acknowledged every message of
+its batches (``acks=all``, idempotent producer): a failed delivery raises ``ConnectionError``,
+which the runtime retries, so a repeated message is possible and a lost one is not. A full local
+queue is waited for (the producer is polled until it drains), which is the backpressure the
+runtime passes back to the generator. A message the broker did not acknowledge within
+``flush_timeout`` seconds raises ``TimeoutError``.
+
+Options of :meth:`KafkaEmitter.emit`:
+
+``envelope``       ``"flat"`` (default) or ``"cloudevents"`` (JSON only).
+``event_format``   ``"json"`` (default: the event's JSON, unchanged), ``"avro"``, ``"protobuf"`` or
+                   ``"json-schema"``: the schema is derived from the table's Arrow schema,
+                   registered with the registry at ``schema_registry_url``, and the message value
+                   is the Confluent wire format (see :mod:`shape_kafka.formats`).
+``schema_registry_url``, ``subject_strategy`` (``topic``, ``record``, ``topic_record``),
+``schema_registry_username`` and ``schema_registry_password``
+                   the registry (``--sink-config kafka.KEY=VALUE``; the password is a credential
+                   reference resolved before it reaches the emitter).
+``config``         extra ``confluent-kafka`` producer settings (security, SASL, compression, ...);
+                   ``bootstrap.servers`` comes from the URI. Defaults: ``acks=all``,
+                   ``enable.idempotence=true``, ``linger.ms=5``.
+``flush_timeout``  seconds to wait for acknowledgements (default 60).
+``resuming``       accepted and ignored (a topic has no file to continue).
+
+Rejections: a non-retryable error about one message (too large, an invalid record) or an event the
+chosen ``event_format`` cannot hold does not fail the batch. Every other event is delivered and
+acknowledged, then :class:`shape.streaming.emit.RejectedEvents` is raised with the keys and
+reasons (``shape emit --dead-letter`` routes them). A dead-letter record (batch column
+``_shape_dead_letter_reason``) also carries the header ``shape-dead-letter-reason``.
+
+``confluent-kafka`` is imported when the first message is sent, never at plugin load.
+"""
+
+from __future__ import annotations
+
+import json
+from collections.abc import Callable, Iterable, Mapping
+from functools import partial
+from typing import Any
+from urllib.parse import unquote, urlsplit
+
+import pyarrow as pa
+
+from shape.errors import ShapeError
+from shape.streaming.emit import RejectedEvents, Rejection
+from shape.streaming.emit.formats import (
+    ENVELOPES,
+    FIELD_DEAD_REASON,
+    FIELD_POISON,
+    FIELD_SEQ,
+    FIELD_TABLE,
+    EncodedEvent,
+    encode_events,
+    poison_body,
+)
+
+from .formats import EVENT_FORMATS, EncodeError, TableCodec, make_codec, wire
+from .registry import SUBJECT_STRATEGIES, RegistryClient, Transport, subject_name
+
+DEFAULT_CONFIG: dict[str, Any] = {"acks": "all", "enable.idempotence": True, "linger.ms": 5}
+HEADER_TABLE = "shape-table"
+HEADER_SYNTHETIC = "shape-synthetic"
+HEADER_DEAD_LETTER_REASON = "shape-dead-letter-reason"
+# Errors about one message, not about the cluster or the topic: the broker or the client refuses
+# this message for good. Such a message is a rejection (dead-lettered when there is a destination
+# for them); every other failure is the destination's and stops the run after the retries.
+PER_MESSAGE_ERRORS = frozenset(
+    {
+        "MSG_SIZE_TOO_LARGE",
+        "_MSG_SIZE_TOO_LARGE",
+        "INVALID_MSG",
+        "INVALID_MSG_SIZE",
+        "INVALID_RECORD",
+        "CORRUPT_MESSAGE",
+        "RECORD_LIST_TOO_LARGE",
+    }
+)
+_POLL = 0.05  # seconds one wait for a full queue to drain
+
+
+def parse_uri(uri: str) -> tuple[str, str]:
+    """``(bootstrap servers, topic)`` of a ``kafka://`` URI."""
+    parts = urlsplit(uri)
+    if parts.scheme != "kafka" or not parts.netloc:
+        raise ShapeError(f"not a kafka URI: {uri!r} (kafka://host:9092[,host2:9092]/topic)")
+    topic = unquote(parts.path.lstrip("/"))
+    if not topic or "/" in topic:
+        raise ShapeError(f"the kafka URI needs one topic name: {uri!r}")
+    return parts.netloc, topic
+
+
+def _per_message_error(err: Any) -> str | None:
+    """The name of ``err`` when it is a non-retryable error about one message, else ``None``."""
+    name = getattr(err, "name", None)
+    if not callable(name):
+        return None
+    retriable = getattr(err, "retriable", None)
+    if callable(retriable) and retriable():
+        return None
+    text = str(name())
+    return text if text in PER_MESSAGE_ERRORS else None
+
+
+def _confluent_producer(config: dict[str, Any]) -> Any:
+    try:
+        from confluent_kafka import Producer
+    except ImportError as exc:
+        raise ShapeError(
+            "emitting to Kafka needs the confluent-kafka package (pip install sqllocks-shape-kafka)"
+        ) from exc
+    return Producer(config)
+
+
+class KafkaEmitter:
+    """Events to a Kafka topic, keyed by the idempotency key.
+
+    ``producer_factory(config)`` is for tests: it stands in for ``confluent_kafka.Producer``.
+    """
+
+    name = "kafka"
+    accepts_poison = True  # one JSON text per message: a cut-off body is a poison message
+    supports_synthetic = True  # the `synthetic` option marks every message with a header
+    schemes = ("kafka",)
+    event_formats = EVENT_FORMATS  # `--event-format`
+    sink_config_keys = (  # `--sink-config kafka.KEY=VALUE`
+        "schema_registry_url",
+        "subject_strategy",
+        "schema_registry_username",
+        "schema_registry_password",
+    )
+
+    def __init__(
+        self,
+        producer_factory: Callable[[dict[str, Any]], Any] | None = None,
+        registry_transport: Transport | None = None,
+    ) -> None:
+        self._factory = producer_factory or _confluent_producer
+        self._registry_transport = registry_transport
+        self._producers: dict[str, Any] = {}
+        self._registries: dict[tuple[Any, ...], RegistryClient] = {}
+        self._codecs: dict[tuple[Any, ...], tuple[TableCodec, int]] = {}
+
+    def _registry(self, url: str, user: str | None, password: str | None) -> RegistryClient:
+        key = (url, user, password)
+        client = self._registries.get(key)
+        if client is None:
+            client = self._registries[key] = RegistryClient(
+                url, username=user, password=password, transport=self._registry_transport
+            )
+        return client
+
+    def _codec(
+        self,
+        fmt: str,
+        topic: str,
+        table: str,
+        schema: pa.Schema,
+        strategy: str,
+        registry: RegistryClient,
+    ) -> tuple[TableCodec, int]:
+        """The codec of ``table`` and the id its schema got from the registry (registered on
+        the table's first batch, once per run). A table whose columns change during the run (a
+        drift plan that adds or drops one) gets a new schema, registered under the same subject:
+        whether that is a compatible new version is the registry's decision."""
+        shape = tuple((f.name, str(f.type), f.nullable) for f in schema if f.name != FIELD_POISON)
+        key = (fmt, topic, strategy, table, shape)
+        hit = self._codecs.get(key)
+        if hit is None:
+            codec = make_codec(fmt, table, schema)
+            subject = subject_name(strategy, topic, codec.record)
+            hit = self._codecs[key] = (
+                codec,
+                registry.register(subject, codec.schema_text, codec.schema_type),
+            )
+        return hit
+
+    def _registry_events(
+        self,
+        batch: pa.RecordBatch,
+        fmt: str,
+        topic: str,
+        strategy: str,
+        registry: RegistryClient,
+    ) -> tuple[list[EncodedEvent], list[Rejection]]:
+        """``batch`` as wire-format events, in order, for ``fmt``; an event the format cannot
+        hold is a :class:`Rejection` (the original flat JSON is its body) and is not in the
+        list."""
+        marks = (
+            [bool(v) for v in batch.column(FIELD_POISON).to_pylist()]
+            if FIELD_POISON in batch.schema.names
+            else None
+        )
+        if marks is not None:
+            batch = batch.select([n for n in batch.schema.names if n != FIELD_POISON])
+        tables = batch.column(FIELD_TABLE).to_pylist()
+        seqs = batch.column(FIELD_SEQ).to_pylist()
+        times = (
+            batch.column("_shape_event_time").cast(pa.string()).to_pylist()
+            if "_shape_event_time" in batch.schema.names
+            else [None] * len(tables)
+        )
+        bodies: list[bytes | EncodeError | None] = [None] * len(tables)
+        for table in dict.fromkeys(tables):
+            idx = [i for i, t in enumerate(tables) if t == table]
+            part = batch if len(idx) == len(tables) else batch.take(pa.array(idx))
+            codec, schema_id = self._codec(fmt, topic, table, part.schema, strategy, registry)
+            for i, payload in zip(idx, codec.encode_batch(part), strict=True):
+                bodies[i] = (
+                    payload if isinstance(payload, EncodeError) else wire(schema_id, payload, fmt)
+                )
+        events: list[EncodedEvent] = []
+        rejected: list[Rejection] = []
+        for i, body in enumerate(bodies):
+            key = f"{tables[i]}/{seqs[i]}"
+            if isinstance(body, EncodeError):
+                rejected.append(Rejection(key, f"cannot encode as {fmt}: {body}"))
+                continue
+            assert body is not None
+            if marks is not None and marks[i]:
+                body = poison_body(body)
+            events.append(EncodedEvent(key, tables[i], seqs[i], times[i], body))
+        return events, rejected
+
+    def _producer(self, servers: str, extra: Mapping[str, Any] | None) -> Any:
+        config = {**DEFAULT_CONFIG, **(extra or {}), "bootstrap.servers": servers}
+        key = json.dumps(config, sort_keys=True, default=str)
+        producer = self._producers.get(key)
+        if producer is None:
+            producer = self._producers[key] = self._factory(config)
+        return producer
+
+    def emit(
+        self,
+        uri: str,
+        batches: Iterable[pa.RecordBatch],
+        *,
+        envelope: str = "flat",
+        resuming: bool = False,
+        config: Mapping[str, Any] | None = None,
+        flush_timeout: float = 60.0,
+        synthetic: bool = False,
+        event_format: str = "json",
+        schema_registry_url: str | None = None,
+        subject_strategy: str = "topic",
+        schema_registry_username: str | None = None,
+        schema_registry_password: str | None = None,
+        **options: Any,
+    ) -> int:
+        """Send every batch; return the number of events, after the broker acknowledged them."""
+        if options:
+            raise ShapeError(f"unknown kafka emitter options: {sorted(options)}")
+        if envelope not in ENVELOPES:
+            raise ShapeError(f"unknown envelope {envelope!r}; choose from {', '.join(ENVELOPES)}")
+        if event_format not in EVENT_FORMATS:
+            raise ShapeError(
+                f"unknown event format {event_format!r}; choose from {', '.join(EVENT_FORMATS)}"
+            )
+        registry: RegistryClient | None = None
+        if event_format != "json":
+            if envelope != "flat":
+                raise ShapeError(
+                    f"--envelope {envelope} is JSON only: it cannot be combined with "
+                    f"--event-format {event_format}"
+                )
+            if subject_strategy not in SUBJECT_STRATEGIES:
+                raise ShapeError(
+                    f"unknown subject strategy {subject_strategy!r}; "
+                    f"choose from {', '.join(SUBJECT_STRATEGIES)}"
+                )
+            if not schema_registry_url:
+                raise ShapeError(
+                    f"--event-format {event_format} needs a schema registry: "
+                    "--sink-config kafka.schema_registry_url=URL"
+                )
+            registry = self._registry(
+                schema_registry_url, schema_registry_username, schema_registry_password
+            )
+        servers, topic = parse_uri(uri)
+        producer = self._producer(servers, config)
+        failures: list[Any] = []
+        rejected: list[Rejection] = []
+
+        def reported(key: str, body: bytes, err: Any, _msg: Any = None) -> None:
+            if err is None:
+                return
+            name = _per_message_error(err)
+            if name is not None:
+                rejected.append(Rejection(key, name, body))
+            else:
+                failures.append(err)
+
+        sent = 0
+        marker = [(HEADER_SYNTHETIC, b"true")] if synthetic else []
+        for batch in batches:
+            reasons = (
+                batch.column(FIELD_DEAD_REASON).to_pylist()
+                if FIELD_DEAD_REASON in batch.schema.names
+                else None
+            )
+            if registry is None:
+                events = encode_events(batch, envelope)
+            else:
+                events, refused = self._registry_events(
+                    batch, event_format, topic, subject_strategy, registry
+                )
+                rejected.extend(refused)
+            for i, event in enumerate(events):
+                headers = [(HEADER_TABLE, event.table.encode("utf-8")), *marker]
+                if reasons is not None:
+                    headers.append((HEADER_DEAD_LETTER_REASON, str(reasons[i]).encode("utf-8")))
+                while True:
+                    try:
+                        producer.produce(
+                            topic,
+                            value=event.body,
+                            key=event.key.encode("utf-8"),
+                            headers=headers,
+                            on_delivery=partial(reported, event.key, event.body),
+                        )
+                        break
+                    except BufferError:
+                        # The local queue is full: serve delivery reports until it drains.
+                        producer.poll(_POLL)
+                    except Exception as exc:  # a KafkaException: the client refuses the message
+                        name = _per_message_error(exc.args[0] if exc.args else None)
+                        if name is None:
+                            raise
+                        rejected.append(Rejection(event.key, name, event.body))
+                        break
+                sent += 1
+            producer.poll(0)
+        pending = producer.flush(flush_timeout)
+        if failures:
+            raise ConnectionError(
+                f"kafka: {len(failures)} of {sent} messages were not delivered to {topic!r}: "
+                f"{failures[0]}"
+            )
+        if pending:
+            raise TimeoutError(
+                f"kafka: {pending} messages to {topic!r} not acknowledged in {flush_timeout}s"
+            )
+        if rejected:
+            raise RejectedEvents(rejected)
+        return sent
+
+    def flush(self) -> None:
+        for producer in self._producers.values():
+            producer.flush(60.0)
+
+    def close(self) -> None:
+        self.flush()
+        self._producers.clear()
+        self._codecs.clear()
+        self._registries.clear()