log-foundry 0.10.2.dev49__tar.gz → 0.10.2.dev51__tar.gz

{log_foundry-0.10.2.dev49 → log_foundry-0.10.2.dev51}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: log-foundry
-Version: 0.10.2.dev49
+Version: 0.10.2.dev51
 Summary: Generate logs for your console and JSON events for downstream consumption.
 License-Expression: MIT
 License-File: LICENSE
@@ -274,9 +274,9 @@ the sink that is already live is a no-op: no drain, no close. The previous sink
 do not hand it back to a later call.
 
 The 5 s covers the **whole** call — both drains and the previous sink's `close()`. `Sink.close()`
-takes no timeout of its own (`KafkaSink.close()` flushes its producer, so an unreachable broker
-blocks it), so that close runs on its own daemon thread and is joined for whatever is left of the
-budget. A hung `close()` therefore costs you the budget, once, and the close carries on in the
+takes no timeout of its own — though `KafkaSink` bounds its own flush at `flush_timeout=10.0`
+and counts whatever is still queued — so that close runs on its own daemon thread and is joined
+for whatever is left of the budget. A hung `close()` therefore costs you the budget, once, and the close carries on in the
 background afterwards.
 
 Nothing is *reported* when that join expires — a slow close is not a failed swap, and a counter
@@ -541,9 +541,9 @@ A few conventions hold across every sink below:
 | Sink | Import from | Configure |
 |---|---|---|
 | `StdoutSink` | `log_foundry.sinks.stdout` | `StdoutSink(stream=sys.stdout)` — one JSON line per event; the zero-config default |
-| `StderrSink` | `log_foundry.sinks.util` | `StderrSink(stream=sys.stderr)` — same, on stderr (twelve-factor) |
-| `NullSink` | `log_foundry.sinks.util` | `NullSink()` — discard everything; `.dropped` counts events |
-| `MemorySink` | `log_foundry.sinks.util` | `MemorySink(maxlen=None)` — collect into `.events` (a bounded ring when `maxlen` is set) |
+| `StderrSink` | `log_foundry.sinks.stdout` | `StderrSink(stream=sys.stderr)` — same, on stderr (twelve-factor) |
+| `NullSink` | `log_foundry.sinks.null` | `NullSink()` — discard everything; `.dropped` counts events |
+| `MemorySink` | `log_foundry.sinks.memory` | `MemorySink(maxlen=None)` — collect into `.events` (a bounded ring when `maxlen` is set) |
 
 ```python
 from log_foundry.sinks.stdout import StdoutSink
@@ -592,7 +592,7 @@ attached to each record; the sink never configures or tears down logging itself.
 | Sink | Import from | Configure |
 |---|---|---|
 | `FileSink` | `log_foundry.sinks.file` | `FileSink(path, *, encoding="utf-8")` — append NDJSON to one file |
-| `RotatingFileSink` | `log_foundry.sinks.file` | `RotatingFileSink(path, *, max_bytes=0, backup_count=0, when=None, interval=1)` — rotate by size and/or time, keeping `backup_count` numbered backups |
+| `RotatingFileSink` | `log_foundry.sinks.file` | `RotatingFileSink(path, *, max_bytes=0, backup_count=1, when=None, interval=1)` — rotate by size and/or time, keeping `backup_count` numbered backups. **`backup_count=0` truncates**: every event since the last rotation is destroyed. The default keeps one generation, costing 2 × `max_bytes` on disk under a size trigger, or one full rollover period under a time-only one — and `max_bytes` defaults to `0`, which bounds nothing |
 | `SQLiteSink` | `log_foundry.sinks.sqlite` | `SQLiteSink(database, *, table="log_events", create_table=True)` — batch-insert into an embedded SQLite DB |
 
 `RotatingFileSink`'s time trigger uses a `when` unit code — `"S"`/`"M"`/`"H"`/`"D"` — times `interval`
@@ -647,7 +647,7 @@ completed, not that every chunk of it landed.
 | `OpenSearchSink` | `log_foundry.sinks.elasticsearch` | same signature as `ElasticsearchSink` (identical bulk protocol) |
 | `LokiSink` | `log_foundry.sinks.loki` | `LokiSink(url, *, labels=("service", "env", "level"), **http_kwargs)` — Grafana Loki push API |
 | `LogstashSink` | `log_foundry.sinks.logstash` | `LogstashSink(url=…, **http_kwargs)` for HTTP, **or** `LogstashSink(host=…, port=…, transport="tcp")` for a raw TCP/UDP socket |
-| `SyslogSink` | `log_foundry.sinks.syslog` | `SyslogSink(host, port=514, *, transport="udp", facility="user", app_name="log-foundry")` — RFC 5424 over UDP/TCP |
+| `SyslogSink` | `log_foundry.sinks.syslog` | `SyslogSink(host, port=514, *, transport="udp", facility="user", app_name="log-foundry", max_datagram_bytes=65507)` — RFC 5424 over UDP/TCP. A UDP frame over the limit is dropped and counted rather than sent, retried and abandoned; TCP is a stream and is unaffected |
 
 ```python
 from log_foundry.sinks.elasticsearch import ElasticsearchSink
@@ -753,9 +753,9 @@ Each needs its own extra (lazy-imported). All publish + retry within a bound and
 
 | Sink | Import from | Extra | Configure |
 |---|---|---|---|
-| `KafkaSink` | `log_foundry.sinks.kafka` | `kafka` | `KafkaSink(topic, *, bootstrap_servers="…", key_field="trace_id")` |
-| `RedisStreamsSink` | `log_foundry.sinks.redis` | `redis` | `RedisStreamsSink(stream, *, url=None)` — `XADD` |
-| `RedisListSink` | `log_foundry.sinks.redis` | `redis` | `RedisListSink(key, *, url=None)` — `RPUSH` |
+| `KafkaSink` | `log_foundry.sinks.kafka` | `kafka` | `KafkaSink(topic, *, flush_timeout=10.0, bootstrap_servers="…", key_field="trace_id")` |
+| `RedisStreamsSink` | `log_foundry.sinks.redis` | `redis` | `RedisStreamsSink(stream, *, url=None, maxlen=None)` — `XADD`. `maxlen` caps the stream (`approximate=True`); trimming happens **at Redis**, after delivery, so it is invisible to `health()` — which is why the default is unbounded |
+| `RedisListSink` | `log_foundry.sinks.redis` | `redis` | `RedisListSink(key, *, url=None, maxlen=None)` — `RPUSH` + `LTRIM` to the newest `maxlen`; same destination-side trimming caveat |
 | `RabbitMQSink` | `log_foundry.sinks.rabbitmq` | `amqp` | `RabbitMQSink(*, exchange, routing_key, url=None)` — persistent messages |
 | `NATSSink` | `log_foundry.sinks.nats` | `nats` | `NATSSink(subject, *, jetstream=False, servers=None)` |
 | `GooglePubSubSink` | `log_foundry.sinks.pubsub` | `gcp-pubsub` | `GooglePubSubSink(topic)` |

{log_foundry-0.10.2.dev49 → log_foundry-0.10.2.dev51}/README.md

@@ -238,9 +238,9 @@ the sink that is already live is a no-op: no drain, no close. The previous sink
 do not hand it back to a later call.
 
 The 5 s covers the **whole** call — both drains and the previous sink's `close()`. `Sink.close()`
-takes no timeout of its own (`KafkaSink.close()` flushes its producer, so an unreachable broker
-blocks it), so that close runs on its own daemon thread and is joined for whatever is left of the
-budget. A hung `close()` therefore costs you the budget, once, and the close carries on in the
+takes no timeout of its own — though `KafkaSink` bounds its own flush at `flush_timeout=10.0`
+and counts whatever is still queued — so that close runs on its own daemon thread and is joined
+for whatever is left of the budget. A hung `close()` therefore costs you the budget, once, and the close carries on in the
 background afterwards.
 
 Nothing is *reported* when that join expires — a slow close is not a failed swap, and a counter
@@ -505,9 +505,9 @@ A few conventions hold across every sink below:
 | Sink | Import from | Configure |
 |---|---|---|
 | `StdoutSink` | `log_foundry.sinks.stdout` | `StdoutSink(stream=sys.stdout)` — one JSON line per event; the zero-config default |
-| `StderrSink` | `log_foundry.sinks.util` | `StderrSink(stream=sys.stderr)` — same, on stderr (twelve-factor) |
-| `NullSink` | `log_foundry.sinks.util` | `NullSink()` — discard everything; `.dropped` counts events |
-| `MemorySink` | `log_foundry.sinks.util` | `MemorySink(maxlen=None)` — collect into `.events` (a bounded ring when `maxlen` is set) |
+| `StderrSink` | `log_foundry.sinks.stdout` | `StderrSink(stream=sys.stderr)` — same, on stderr (twelve-factor) |
+| `NullSink` | `log_foundry.sinks.null` | `NullSink()` — discard everything; `.dropped` counts events |
+| `MemorySink` | `log_foundry.sinks.memory` | `MemorySink(maxlen=None)` — collect into `.events` (a bounded ring when `maxlen` is set) |
 
 ```python
 from log_foundry.sinks.stdout import StdoutSink
@@ -556,7 +556,7 @@ attached to each record; the sink never configures or tears down logging itself.
 | Sink | Import from | Configure |
 |---|---|---|
 | `FileSink` | `log_foundry.sinks.file` | `FileSink(path, *, encoding="utf-8")` — append NDJSON to one file |
-| `RotatingFileSink` | `log_foundry.sinks.file` | `RotatingFileSink(path, *, max_bytes=0, backup_count=0, when=None, interval=1)` — rotate by size and/or time, keeping `backup_count` numbered backups |
+| `RotatingFileSink` | `log_foundry.sinks.file` | `RotatingFileSink(path, *, max_bytes=0, backup_count=1, when=None, interval=1)` — rotate by size and/or time, keeping `backup_count` numbered backups. **`backup_count=0` truncates**: every event since the last rotation is destroyed. The default keeps one generation, costing 2 × `max_bytes` on disk under a size trigger, or one full rollover period under a time-only one — and `max_bytes` defaults to `0`, which bounds nothing |
 | `SQLiteSink` | `log_foundry.sinks.sqlite` | `SQLiteSink(database, *, table="log_events", create_table=True)` — batch-insert into an embedded SQLite DB |
 
 `RotatingFileSink`'s time trigger uses a `when` unit code — `"S"`/`"M"`/`"H"`/`"D"` — times `interval`
@@ -611,7 +611,7 @@ completed, not that every chunk of it landed.
 | `OpenSearchSink` | `log_foundry.sinks.elasticsearch` | same signature as `ElasticsearchSink` (identical bulk protocol) |
 | `LokiSink` | `log_foundry.sinks.loki` | `LokiSink(url, *, labels=("service", "env", "level"), **http_kwargs)` — Grafana Loki push API |
 | `LogstashSink` | `log_foundry.sinks.logstash` | `LogstashSink(url=…, **http_kwargs)` for HTTP, **or** `LogstashSink(host=…, port=…, transport="tcp")` for a raw TCP/UDP socket |
-| `SyslogSink` | `log_foundry.sinks.syslog` | `SyslogSink(host, port=514, *, transport="udp", facility="user", app_name="log-foundry")` — RFC 5424 over UDP/TCP |
+| `SyslogSink` | `log_foundry.sinks.syslog` | `SyslogSink(host, port=514, *, transport="udp", facility="user", app_name="log-foundry", max_datagram_bytes=65507)` — RFC 5424 over UDP/TCP. A UDP frame over the limit is dropped and counted rather than sent, retried and abandoned; TCP is a stream and is unaffected |
 
 ```python
 from log_foundry.sinks.elasticsearch import ElasticsearchSink
@@ -717,9 +717,9 @@ Each needs its own extra (lazy-imported). All publish + retry within a bound and
 
 | Sink | Import from | Extra | Configure |
 |---|---|---|---|
-| `KafkaSink` | `log_foundry.sinks.kafka` | `kafka` | `KafkaSink(topic, *, bootstrap_servers="…", key_field="trace_id")` |
-| `RedisStreamsSink` | `log_foundry.sinks.redis` | `redis` | `RedisStreamsSink(stream, *, url=None)` — `XADD` |
-| `RedisListSink` | `log_foundry.sinks.redis` | `redis` | `RedisListSink(key, *, url=None)` — `RPUSH` |
+| `KafkaSink` | `log_foundry.sinks.kafka` | `kafka` | `KafkaSink(topic, *, flush_timeout=10.0, bootstrap_servers="…", key_field="trace_id")` |
+| `RedisStreamsSink` | `log_foundry.sinks.redis` | `redis` | `RedisStreamsSink(stream, *, url=None, maxlen=None)` — `XADD`. `maxlen` caps the stream (`approximate=True`); trimming happens **at Redis**, after delivery, so it is invisible to `health()` — which is why the default is unbounded |
+| `RedisListSink` | `log_foundry.sinks.redis` | `redis` | `RedisListSink(key, *, url=None, maxlen=None)` — `RPUSH` + `LTRIM` to the newest `maxlen`; same destination-side trimming caveat |
 | `RabbitMQSink` | `log_foundry.sinks.rabbitmq` | `amqp` | `RabbitMQSink(*, exchange, routing_key, url=None)` — persistent messages |
 | `NATSSink` | `log_foundry.sinks.nats` | `nats` | `NATSSink(subject, *, jetstream=False, servers=None)` |
 | `GooglePubSubSink` | `log_foundry.sinks.pubsub` | `gcp-pubsub` | `GooglePubSubSink(topic)` |

{log_foundry-0.10.2.dev49 → log_foundry-0.10.2.dev51}/pyproject.toml

@@ -20,7 +20,7 @@ dependencies = [
 ]
 
 # Optional features. Install with: pip install log-foundry[aws]
-version = "0.10.2.dev49"
+version = "0.10.2.dev51"
 
 [project.optional-dependencies]
 aws = ["boto3>=1.43.61"]              # SQSSink, SNSSink, KinesisSink, FirehoseSink

{log_foundry-0.10.2.dev49 → log_foundry-0.10.2.dev51}/src/log_foundry/sinks/_socket.py

@@ -2,6 +2,7 @@
 
 from __future__ import annotations
 
+import errno
 import socket
 import threading
 
@@ -13,6 +14,26 @@ __all__ = ["SocketTransport"]
 
 _BACKOFF_BASE = 0.1
 
+DEFAULT_MAX_DATAGRAM_BYTES = 65507
+"""Bytes one UDP datagram may carry, the IPv4 payload maximum (SPEC-038 FR-007).
+
+65,535 less the 20-byte IP header and the 8-byte UDP header. That is the **IPv4** figure, and
+it is used for IPv6 too: IPv6 does not count its own header in the payload length, so its true
+maximum is 65,527 — a 20-byte discrepancy no real path can reach, since no MTU carries either.
+It is the ceiling the *protocol* imposes; a path's real limit is often far lower, which is why
+it is configurable. TCP is a stream and has no such limit, so this bounds nothing there.
+"""
+
+_PERMANENT_ERRNOS = frozenset({errno.EMSGSIZE})
+"""Socket errnos that describe the message rather than the destination (FR-007 AC-3).
+
+``EMSGSIZE`` alone. Every other errno this transport can raise — ``ECONNREFUSED``,
+``EHOSTUNREACH``, ``ENETDOWN``, ``EPIPE``, ``ETIMEDOUT`` — describes a destination that may come
+back, and treating any of those as permanent would turn a transient outage into silent loss. A
+set stated here rather than a condition inferred at the call site, so adding to it is a decision
+someone makes deliberately.
+"""
+
 
 def _make_tcp(host: str, port: int, timeout: float) -> socket.socket:
     """Opens a connected TCP socket.
@@ -95,6 +116,8 @@ class SocketTransport:
 
     Attributes:
       failed: Messages abandoned past the reconnect-retry bound.
+      dropped_oversized: UDP datagrams discarded before any send for exceeding
+        ``max_datagram_bytes``.
     """
 
     def __init__(
@@ -105,6 +128,7 @@ class SocketTransport:
         transport: str = "tcp",
         timeout: float = 5.0,
         max_retries: int = 3,
+        max_datagram_bytes: int = DEFAULT_MAX_DATAGRAM_BYTES,
     ) -> None:
         """Configures the destination and retry bound without opening a socket yet.
 
@@ -116,6 +140,8 @@ class SocketTransport:
           max_retries: Reconnect retries per message, floored at zero for the reason
             ``Worker._emit`` floors its own (SPEC-021) — a negative value made no attempt at all
             and abandoned the message without moving ``failed``.
+          max_datagram_bytes: The largest UDP datagram to attempt, defaulting to
+            :data:`DEFAULT_MAX_DATAGRAM_BYTES`. Ignored for TCP, which is a stream.
 
         Returns:
           None.
@@ -130,8 +156,10 @@ class SocketTransport:
         self._transport = transport
         self._timeout = timeout
         self._max_retries = max(max_retries, 0)
+        self._max_datagram_bytes = max_datagram_bytes
         self._sock: socket.socket | None = None
         self.failed = 0
+        self.dropped_oversized = 0
         self._counter_lock = threading.Lock()
         self.log_foundry_stop_signal: threading.Event | None = None
         self._lock = threading.Lock()
@@ -170,15 +198,62 @@ class SocketTransport:
                     f"SocketTransport delivered none of {len(messages)} message(s): "
                     f"the transport is closed"
                 )
+            sendable = self._sendable(messages)
             delivered = 0
-            for message in messages:
+            for message in sendable:
                 if self._send_one(message):
                     delivered += 1
-        if delivered == 0:
+        if sendable and delivered == 0:
             raise SinkDeliveryError(
-                f"SocketTransport delivered none of {len(messages)} message(s)"
+                f"SocketTransport delivered none of {len(sendable)} message(s)"
             )
 
+    def _sendable(self, messages: list[bytes]) -> list[bytes]:
+        """Discards UDP datagrams too large to send, before any attempt (FR-007 AC-1).
+
+        A 70 KB event produced ``OSError errno=40`` (EMSGSIZE), which the retry loop treated as
+        transient: four sends with backoff, counted as ``failed``, then a raise that sent the
+        worker round for three more rounds — roughly sixteen futile sends and seconds of backoff
+        on the single drain thread, never converging. The size is knowable *before* sending, and
+        every other size-limited sink here drops and counts first, so this one does too.
+
+        Dropping is right rather than raising: the datagram is permanently unsendable on this
+        path, so there is nothing for a retry to fix, and reporting it as a delivery failure
+        would have the worker re-send the rest of the batch alongside it.
+
+        **``dropped_oversized`` inflates in one case, and it is recorded rather than hidden.**
+        If a batch holds an oversized frame *and* its sendable remainder then fails totally, the
+        emit raises, the worker retries the whole batch, and this filter re-frames and re-drops
+        the same event once per attempt — up to four times for one unsendable frame. Everywhere
+        else ``dropped`` is an exact count, so the exception matters. Nothing here can fix it:
+        the worker owns the retry and hands back the original events, so the sink cannot know it
+        has seen them before. ``FirehoseSink._records`` and ``KinesisSink._records`` have the
+        same shape and predate this, which is why the fix belongs one level up if it is ever
+        taken.
+
+        Args:
+          messages: The framed messages, in order.
+
+        Returns:
+          Those within the datagram limit. All of them under TCP, which is a stream.
+
+        Raises:
+          None.
+        """
+        if self._transport != "udp":
+            return messages
+        sendable = [m for m in messages if len(m) <= self._max_datagram_bytes]
+        dropped = len(messages) - len(sendable)
+        if dropped:
+            with self._counter_lock:
+                self.dropped_oversized += dropped
+            _diag.lost(
+                "message",
+                dropped,
+                f"SocketTransport, over the {self._max_datagram_bytes}-byte datagram limit",
+            )
+        return sendable
+
     def losses(self) -> SinkLosses:
         """Reports messages abandoned past the reconnect-retry bound (SPEC-026 FR-002).
 
@@ -192,7 +267,7 @@ class SocketTransport:
           None.
         """
         with self._counter_lock:
-            return SinkLosses(dropped=0, failed=self.failed)
+            return SinkLosses(dropped=self.dropped_oversized, failed=self.failed)
 
     def close(self) -> None:
         """Closes the held socket, if any (FR-005, FR-012).
@@ -224,6 +299,12 @@ class SocketTransport:
         refused" from "host unknown", and the code is an integer from the OS rather than caller
         data.
 
+        An errno in ``_PERMANENT_ERRNOS`` skips the remaining attempts (FR-007 AC-3): a re-send
+        of identical bytes to the same destination can only earn the same answer, so retrying
+        buys nothing and costs the drain thread its backoff. :meth:`_sendable` normally catches
+        the oversized case first; this is the backstop for a path MTU smaller than the datagram
+        limit, which no local check can know.
+
         Args:
           message: The exact bytes to put on the wire.
 
@@ -242,7 +323,7 @@ class SocketTransport:
                 return True
             except OSError as err:
                 self._reset()
-                if attempt < self._max_retries:
+                if attempt < self._max_retries and err.errno not in _PERMANENT_ERRNOS:
                     wait(_BACKOFF_BASE * (2**attempt), self.log_foundry_stop_signal)
                     continue
                 with self._counter_lock:

{log_foundry-0.10.2.dev49 → log_foundry-0.10.2.dev51}/src/log_foundry/sinks/file.py

@@ -120,9 +120,18 @@ class RotatingFileSink:
     Rotation renames the active file through numbered backups, prunes any beyond the backup
     count, and opens a fresh active file — a backup count of zero keeps none, simply replacing
     the active file. Backups are numbered rather than timestamped, so no filename derives from
-    a clock at all and the monotonic deadline has no naming consequence. No event is lost across
-    a rotation, because the rotate happens before the pending event is written and the event
-    lands in the fresh file.
+    a clock at all and the monotonic deadline has no naming consequence. **The pending event is
+    not lost across a rotation**, because the rotate happens before it is written and it lands in
+    the fresh file (SPEC-038 FR-012 AC-5). That is the whole of the claim: it says nothing about
+    events *already written*, which retention governs — at ``backup_count=0`` every one of them
+    is destroyed at each rollover, which is why that is no longer the default.
+
+    No counter is added for what retention discards, at any ``backup_count``. This sink is a
+    bounded ring buffer, retention *is* the configuration, and discarding the oldest generation
+    is that configuration working — the precedent being ``MemorySink(maxlen)``, which behaves as
+    a bounded ring, counts nothing and implements no ``losses()``. Neither ``dropped`` (defined
+    as discarded *before* attempting delivery) nor ``failed`` fits an event that was written and
+    flushed to disk.
 
     A rotation rebinds the active stream, so it is the sink where concurrent writers did real
     damage: a second thread mid-``emit`` could write to the handle rotation had just closed, or
@@ -135,7 +144,7 @@ class RotatingFileSink:
         path: str,
         *,
         max_bytes: int = 0,
-        backup_count: int = 0,
+        backup_count: int = 1,
         when: str | None = None,
         interval: int = 1,
     ) -> None:
@@ -148,7 +157,15 @@ class RotatingFileSink:
         Args:
           path: The active file to append to.
           max_bytes: The size trigger, or 0 to disable it.
-          backup_count: How many numbered backups to retain.
+          backup_count: How many numbered backups to retain. **The default is 1** (SPEC-038
+            FR-012): at ``0`` a rotation calls ``os.remove`` on the active file, so
+            ``RotatingFileSink("app.log", max_bytes=10_000_000)`` silently threw away 10 MB at
+            every rollover. ``0`` still truncates, unchanged, for a caller who wants that.
+
+            The cost of the new default is disk: **2 x max_bytes under a size trigger, or one
+            full rollover period under a time-only one** — and ``max_bytes`` defaults to ``0``,
+            which bounds nothing, so a time-triggered sink's ceiling is whatever one period
+            writes.
           when: The time-trigger unit code, matched case-insensitively, or ``None`` to disable
             it. The vocabulary mirrors a subset of the stdlib ``TimedRotatingFileHandler``'s.
           interval: How many units make up one rollover period.

{log_foundry-0.10.2.dev49 → log_foundry-0.10.2.dev51}/src/log_foundry/sinks/firehose.py

@@ -44,7 +44,12 @@ class FirehoseSink:
 
     MAX_RECORDS = 500
     MAX_REQUEST_BYTES = 4 * 1024 * 1024
-    MAX_RECORD_BYTES = 1024 * 1024
+    MAX_RECORD_BYTES = 1_024_000
+    """Firehose's documented per-record ceiling: 1,000 KiB, which is **not** ``1024 * 1024``.
+
+    The 24,576-byte gap between the two is a band of records this sink used to pass through and
+    the service then rejected (SPEC-038 FR-009).
+    """
 
     def __init__(self, delivery_stream: str, *, client: Any = None, max_retries: int = 3) -> None:
         """Binds the sink to a delivery stream.
@@ -155,6 +160,13 @@ class FirehoseSink:
     def _records(self, batch: list[dict[str, object]]) -> list[dict[str, Any]]:
         """Builds the request entries, dropping any record too large to ever fit (FR-011).
 
+        Each record ends with a newline (SPEC-038 FR-005). Firehose concatenates record payloads
+        verbatim into the delivery buffer and the *producer* must supply the separator, so
+        without it the S3 objects read ``{"a":1}{"b":2}`` — unparseable by Athena, Glue and
+        OpenSearch ingest, and inconsistent with the NDJSON every other sink here emits. The
+        newline is part of ``Data``, so it is charged to both the per-record limit checked below
+        and the per-request budget :meth:`emit` chunks by.
+
         Args:
           batch: The events to convert.
 
@@ -166,7 +178,7 @@ class FirehoseSink:
         """
         records: list[dict[str, Any]] = []
         for event in batch:
-            data = json.dumps(event).encode("utf-8")
+            data = json.dumps(event).encode("utf-8") + b"\n"
             if len(data) > self.MAX_RECORD_BYTES:
                 with self._counter_lock:
                     self.dropped_oversized += 1

{log_foundry-0.10.2.dev49 → log_foundry-0.10.2.dev51}/src/log_foundry/sinks/kafka.py

@@ -12,6 +12,39 @@ from log_foundry.sinks.base import SinkDeliveryError, SinkLosses
 __all__ = ["KafkaSink"]
 
 
+def _usable_timeout(value: float) -> float:
+    """Returns a flush timeout that is actually a bound, or the default.
+
+    ``0`` is the value that switched this sink's exit delivery off, and ``inf`` is the unbounded
+    wait FR-006 exists to remove, so neither is accepted from a caller. The test is
+    ``not (0 < value < inf)`` rather than a pair of comparisons, because ``NaN`` compares
+    ``False`` to everything and would otherwise slip through (SPEC-027 FR-001's reasoning for
+    ``Retry-After``).
+
+    Args:
+      value: The caller's timeout.
+
+    Returns:
+      The value if it bounds anything, else :data:`DEFAULT_FLUSH_TIMEOUT`.
+
+    Raises:
+      None.
+    """
+    return value if 0 < value < float("inf") else DEFAULT_FLUSH_TIMEOUT
+
+DEFAULT_FLUSH_TIMEOUT = 10.0
+"""Seconds :meth:`KafkaSink.close` waits for the producer to drain (SPEC-038 FR-006).
+
+Ten rather than ``message.timeout.ms``'s five minutes.
+
+It is spent **beside** ``shutdown()``'s budget, not from it: ``Worker.shutdown`` joins the drain
+thread against its deadline and *then* closes the sink inline, with no remaining-budget argument
+(arch §13, since ``Sink.close`` takes no timeout). Measured, ``shutdown(timeout=2.0)`` against a
+dead broker takes 10.01 s. Ten seconds is chosen to keep that overrun small rather than to fit
+inside a budget it cannot see — five minutes is what it replaces.
+"""
+
+
 class KafkaSink:
     """A :class:`~log_foundry.sinks.base.Sink` that produces events to a Kafka topic.
 
@@ -20,8 +53,17 @@ class KafkaSink:
     returns without blocking, while the delivery result arrives asynchronously on a callback
     serviced by ``poll()`` and ``flush()``.
 
+    The worst case (SPEC-027 FR-005) is not a retry loop — this sink has none — but its
+    ``close()``: one ``flush_timeout`` wait, 10 s at the default, spent **beside**
+    ``shutdown()``'s budget rather than from it. ``Worker.shutdown`` joins the drain thread
+    against its deadline and *then* closes the sink inline with no remaining-budget argument
+    (arch §13), so the two add up: measured, ``shutdown(timeout=2.0)`` against a dead broker
+    takes 10.01 s.
+
     Attributes:
-      failed: Messages whose delivery callback reported an error.
+      flush_timeout: Seconds :meth:`close` waits for the producer to drain.
+      failed: Messages whose delivery callback reported an error, plus any still queued when
+        that close flush expired — the count ``flush()`` returns.
       rejected: Messages ``produce()`` itself refused — a full local queue, a serialization fault
         — which never reached the producer's batch at all.
 
@@ -43,6 +85,7 @@ class KafkaSink:
         producer: Any = None,
         bootstrap_servers: str | None = None,
         key_field: str = "trace_id",
+        flush_timeout: float = DEFAULT_FLUSH_TIMEOUT,
     ) -> None:
         """Binds the sink to a topic and a producer.
 
@@ -51,6 +94,13 @@ class KafkaSink:
           producer: A ``confluent-kafka``-shaped producer, or ``None`` to build one.
           bootstrap_servers: The broker list, required when no producer is injected.
           key_field: The event key used as the message key, or empty for no key.
+          flush_timeout: Seconds :meth:`close` waits for the producer to drain before counting
+            what is left as lost (SPEC-038 FR-006). Unbounded, an unreachable broker held
+            process exit for ``message.timeout.ms`` — five minutes by default — despite
+            ``shutdown(timeout=30)``. Non-positive or non-finite values fall back to the
+            default, as ``Worker._emit`` floors its own retries (SPEC-021): ``0`` reinstates
+            exactly the ``flush(0)`` that switched this sink's exit delivery off, and ``inf``
+            reinstates the unbounded wait.
 
         Returns:
           None.
@@ -70,6 +120,7 @@ class KafkaSink:
         self.topic = topic
         self.producer = producer
         self.key_field = key_field
+        self.flush_timeout = _usable_timeout(flush_timeout)
         self.failed = 0
         self.rejected = 0
         self._counter_lock = threading.Lock()
@@ -175,7 +226,50 @@ class KafkaSink:
             if self._closed:
                 return
             self._closed = True
-            self.producer.flush()
+            self._flush_bounded()
+
+    def _flush_bounded(self) -> None:
+        """Flushes the producer within a bound, counting whatever it could not deliver (FR-006).
+
+        ``Producer.flush()`` with no timeout waits for ``message.timeout.ms`` — five minutes by
+        default — and ``Worker.shutdown`` closes the live sink **inline and unbounded** (arch
+        §13), so an unreachable broker held process exit for five minutes despite
+        ``shutdown(timeout=30)``. The wait is now capped by ``flush_timeout``.
+
+        **The stop signal is deliberately not consulted here.** A revision cut the timeout to
+        zero while a shutdown was in progress, reasoning that the events were lost either way.
+        They were not: ``produce()`` is a local hand-off and ``flush()`` is the only thing that
+        drains the producer's batch, so with the stop event set — which it always is by the time
+        ``close()`` runs, since ``Worker.shutdown`` sets it *before* the join — Kafka's exit
+        delivery was switched off entirely. Measured through a real ``shutdown()``: eleven buffered
+        messages — nine ``info()`` calls plus the span's start and end — ``flush(0)``, **zero
+        delivered**, all eleven booked as ``failed``. That is worse
+        than the hang this bound exists to fix, and it is the same mistake FR-001 AC-4a records
+        for ``HTTPSink`` — skipping *work* during the exit drain, rather than skipping a *wait*.
+
+        ``flush()`` *returns* the number of messages still queued, which is exactly the count
+        lost at exit — the old call discarded it, so the loss was silent. It is counted into
+        ``failed`` and announced once, with a count rather than one line per message.
+
+        Args:
+          None.
+
+        Returns:
+          None.
+
+        Raises:
+          Exception: Whatever the producer raises, which ``close``'s caller already handles.
+        """
+        remaining = self.producer.flush(self.flush_timeout)
+        if type(remaining) is not int or remaining <= 0:
+            return
+        with self._counter_lock:
+            self.failed += remaining
+        _diag.lost(
+            "message",
+            remaining,
+            f"KafkaSink, still queued when the {self.flush_timeout}s close flush expired",
+        )
 
     def _key(self, event: dict[str, object]) -> bytes | None:
         """Derives one message's partition key from the configured field.

{log_foundry-0.10.2.dev49 → log_foundry-0.10.2.dev51}/src/log_foundry/sinks/kinesis.py

@@ -116,7 +116,7 @@ class KinesisSink:
             records,
             max_count=self.MAX_RECORDS,
             max_bytes=self.MAX_REQUEST_BYTES,
-            size_of=lambda record: len(record["Data"]),
+            size_of=_record_size,
         ):
             chunks += 1
             outcome = self._send(chunk)
@@ -242,3 +242,26 @@ class KinesisSink:
                 )
                 return sent - len(records)
         return 0
+
+
+def _record_size(record: dict[str, Any]) -> int:
+    """Measures one request entry, partition key included (SPEC-038 FR-009).
+
+    ``PutRecords`` charges the partition key against the 5 MiB request limit, and a key may be up
+    to 256 bytes, so a 500-record request could understate itself by ~128 KB — enough to have the
+    service reject a chunk this sink believed was inside the budget. ``SQSSink`` charges its FIFO
+    ids for the same reason and records the same rationale.
+
+    This applies to Kinesis alone: a Firehose record is ``{"Data": data}`` and that API has no
+    partition key, so there is nothing there to charge.
+
+    Args:
+      record: One ``PutRecords`` entry.
+
+    Returns:
+      The bytes it contributes to the request.
+
+    Raises:
+      None.
+    """
+    return len(record["Data"]) + len(record["PartitionKey"])

{log_foundry-0.10.2.dev49 → log_foundry-0.10.2.dev51}/src/log_foundry/sinks/logstash.py

@@ -8,7 +8,7 @@ from typing import TYPE_CHECKING
 if TYPE_CHECKING:
     import threading
 
-from log_foundry.sinks._socket import SocketTransport
+from log_foundry.sinks._socket import DEFAULT_MAX_DATAGRAM_BYTES, SocketTransport
 from log_foundry.sinks.base import SinkLosses
 from log_foundry.sinks.http import HTTPSink
 
@@ -54,6 +54,7 @@ class LogstashSink:
         transport: str = "tcp",
         timeout: float = 5.0,
         max_retries: int = 3,
+        max_datagram_bytes: int = DEFAULT_MAX_DATAGRAM_BYTES,
         **http_kwargs: object,
     ) -> None:
         """Selects and builds exactly one backend.
@@ -65,6 +66,9 @@ class LogstashSink:
           transport: ``"tcp"`` or ``"udp"``, in socket mode.
           timeout: Seconds allowed per request or connection.
           max_retries: Retries the chosen backend makes.
+          max_datagram_bytes: In UDP socket mode, the largest datagram to attempt; a frame over
+            it is dropped and counted rather than sent, retried and abandoned (SPEC-038 FR-007).
+            Ignored in HTTP mode and over TCP, which is a stream.
           **http_kwargs: Forwarded to :class:`~log_foundry.sinks.http.HTTPSink` in HTTP mode.
 
         Returns:
@@ -82,7 +86,12 @@ class LogstashSink:
         elif host is not None and port is not None:
             self._http = None
             self._socket = SocketTransport(
-                host, port, transport=transport, timeout=timeout, max_retries=max_retries
+                host,
+                port,
+                transport=transport,
+                timeout=timeout,
+                max_retries=max_retries,
+                max_datagram_bytes=max_datagram_bytes,
             )
         else:
             raise ValueError("LogstashSink requires either url= (HTTP) or host= + port= (socket)")