log-foundry 0.10.2.dev71__tar.gz → 0.10.2.dev73__tar.gz

{log_foundry-0.10.2.dev71 → log_foundry-0.10.2.dev73}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: log-foundry
-Version: 0.10.2.dev71
+Version: 0.10.2.dev73
 Summary: Generate logs for your console and JSON events for downstream consumption.
 License-Expression: MIT
 License-File: LICENSE
@@ -694,7 +694,7 @@ completed, not that every chunk of it landed.
 | `ElasticsearchSink` | `log_foundry.sinks.elasticsearch` | `ElasticsearchSink(url, *, index, auth=None, **http_kwargs)` — POST to `_bulk`, parsing per-item errors (`.item_errors`) |
 | `OpenSearchSink` | `log_foundry.sinks.elasticsearch` | same signature as `ElasticsearchSink` (identical bulk protocol) |
 | `LokiSink` | `log_foundry.sinks.loki` | `LokiSink(url, *, labels=("service", "env", "level"), **http_kwargs)` — Grafana Loki push API |
-| `LogstashSink` | `log_foundry.sinks.logstash` | `LogstashSink(url=…, **http_kwargs)` for HTTP, **or** `LogstashSink(host=…, port=…, transport="tcp")` for a raw TCP/UDP socket |
+| `LogstashSink` | `log_foundry.sinks.logstash` | `LogstashSink(url=…, body_format="json_array", **http_kwargs)` for HTTP, **or** `LogstashSink(host=…, port=…, transport="tcp")` for a raw TCP/UDP socket. HTTP mode posts a JSON array as `application/json`, which a **stock** `http` input parses into one event per element; pass `body_format="ndjson"` for an input configured with `additional_codecs => {"application/x-ndjson" => "json_lines"}`, which that setting *replaces* the default map to provide |
 | `SyslogSink` | `log_foundry.sinks.syslog` | `SyslogSink(host, port=514, *, transport="udp", facility="user", app_name="log-foundry", max_datagram_bytes=65507)` — RFC 5424 over UDP/TCP. A UDP frame over the limit is dropped and counted rather than sent, retried and abandoned; TCP is a stream and is unaffected |
 
 ```python
@@ -797,7 +797,16 @@ Two things worth knowing:
 
 #### Queue & stream
 
-Each needs its own extra (lazy-imported). All publish + retry within a bound and close cleanly.
+Each needs its own extra (lazy-imported). All publish within a bound and close cleanly. **Where the
+retry lives differs, and it was measured rather than assumed** (SPEC-041 FR-004): the Redis,
+RabbitMQ and Event Hubs sinks retry through `sinks/_retry`, so their backoff is bounded *and* cut
+short by a shutdown. `KafkaSink`, `NATSSink` and `GooglePubSubSink` add no retry loop and need
+none — each hands off locally and returns without waiting, so nothing of theirs holds the single
+drain thread. Their clients retry on their own threads within their own bounds:
+`message.timeout.ms` (5 min default) for Kafka, a 600 s deadline for Pub/Sub, and for NATS a
+JetStream publish bounded by its 5 s ack timeout with no retry at all. `NATSSink` refuses a batch
+outright while its client reports itself disconnected, so a sustained outage moves
+`health().failed_batches` instead of being absorbed.
 
 | Sink | Import from | Extra | Configure |
 |---|---|---|---|
@@ -821,7 +830,7 @@ Write-only inserts (querying is the downstream tool's job); each needs its own e
 | Sink | Import from | Extra | Configure |
 |---|---|---|---|
 | `MongoDBSink` | `log_foundry.sinks.mongodb` | `mongo` | `MongoDBSink(*, uri="…", database="…", collection="…")` |
-| `PostgresSink` | `log_foundry.sinks.postgres` | `postgres` | `PostgresSink(table, *, dsn="…", create_table=False)` — JSONB `event` column + extracted columns |
+| `PostgresSink` | `log_foundry.sinks.postgres` | `postgres` | `PostgresSink(table, *, dsn="…", create_table=False, connect_timeout=5)` — JSONB `event` column + extracted columns. Reconnects an **owned** connection the server has closed; a `connection=` you inject is never reopened. `connect_timeout` is passed to libpq explicitly, so it **overrides** any `connect_timeout` in your DSN |
 | `ClickHouseSink` | `log_foundry.sinks.clickhouse` | `clickhouse` | `ClickHouseSink(table, *, dsn="…", create_table=False)` — MergeTree, columnar insert |
 
 `PostgresSink` / `ClickHouseSink` default `create_table=False` (you own the schema and indexes); set

{log_foundry-0.10.2.dev71 → log_foundry-0.10.2.dev73}/README.md

@@ -658,7 +658,7 @@ completed, not that every chunk of it landed.
 | `ElasticsearchSink` | `log_foundry.sinks.elasticsearch` | `ElasticsearchSink(url, *, index, auth=None, **http_kwargs)` — POST to `_bulk`, parsing per-item errors (`.item_errors`) |
 | `OpenSearchSink` | `log_foundry.sinks.elasticsearch` | same signature as `ElasticsearchSink` (identical bulk protocol) |
 | `LokiSink` | `log_foundry.sinks.loki` | `LokiSink(url, *, labels=("service", "env", "level"), **http_kwargs)` — Grafana Loki push API |
-| `LogstashSink` | `log_foundry.sinks.logstash` | `LogstashSink(url=…, **http_kwargs)` for HTTP, **or** `LogstashSink(host=…, port=…, transport="tcp")` for a raw TCP/UDP socket |
+| `LogstashSink` | `log_foundry.sinks.logstash` | `LogstashSink(url=…, body_format="json_array", **http_kwargs)` for HTTP, **or** `LogstashSink(host=…, port=…, transport="tcp")` for a raw TCP/UDP socket. HTTP mode posts a JSON array as `application/json`, which a **stock** `http` input parses into one event per element; pass `body_format="ndjson"` for an input configured with `additional_codecs => {"application/x-ndjson" => "json_lines"}`, which that setting *replaces* the default map to provide |
 | `SyslogSink` | `log_foundry.sinks.syslog` | `SyslogSink(host, port=514, *, transport="udp", facility="user", app_name="log-foundry", max_datagram_bytes=65507)` — RFC 5424 over UDP/TCP. A UDP frame over the limit is dropped and counted rather than sent, retried and abandoned; TCP is a stream and is unaffected |
 
 ```python
@@ -761,7 +761,16 @@ Two things worth knowing:
 
 #### Queue & stream
 
-Each needs its own extra (lazy-imported). All publish + retry within a bound and close cleanly.
+Each needs its own extra (lazy-imported). All publish within a bound and close cleanly. **Where the
+retry lives differs, and it was measured rather than assumed** (SPEC-041 FR-004): the Redis,
+RabbitMQ and Event Hubs sinks retry through `sinks/_retry`, so their backoff is bounded *and* cut
+short by a shutdown. `KafkaSink`, `NATSSink` and `GooglePubSubSink` add no retry loop and need
+none — each hands off locally and returns without waiting, so nothing of theirs holds the single
+drain thread. Their clients retry on their own threads within their own bounds:
+`message.timeout.ms` (5 min default) for Kafka, a 600 s deadline for Pub/Sub, and for NATS a
+JetStream publish bounded by its 5 s ack timeout with no retry at all. `NATSSink` refuses a batch
+outright while its client reports itself disconnected, so a sustained outage moves
+`health().failed_batches` instead of being absorbed.
 
 | Sink | Import from | Extra | Configure |
 |---|---|---|---|
@@ -785,7 +794,7 @@ Write-only inserts (querying is the downstream tool's job); each needs its own e
 | Sink | Import from | Extra | Configure |
 |---|---|---|---|
 | `MongoDBSink` | `log_foundry.sinks.mongodb` | `mongo` | `MongoDBSink(*, uri="…", database="…", collection="…")` |
-| `PostgresSink` | `log_foundry.sinks.postgres` | `postgres` | `PostgresSink(table, *, dsn="…", create_table=False)` — JSONB `event` column + extracted columns |
+| `PostgresSink` | `log_foundry.sinks.postgres` | `postgres` | `PostgresSink(table, *, dsn="…", create_table=False, connect_timeout=5)` — JSONB `event` column + extracted columns. Reconnects an **owned** connection the server has closed; a `connection=` you inject is never reopened. `connect_timeout` is passed to libpq explicitly, so it **overrides** any `connect_timeout` in your DSN |
 | `ClickHouseSink` | `log_foundry.sinks.clickhouse` | `clickhouse` | `ClickHouseSink(table, *, dsn="…", create_table=False)` — MergeTree, columnar insert |
 
 `PostgresSink` / `ClickHouseSink` default `create_table=False` (you own the schema and indexes); set

{log_foundry-0.10.2.dev71 → log_foundry-0.10.2.dev73}/pyproject.toml

@@ -20,7 +20,7 @@ dependencies = [
 ]
 
 # Optional features. Install with: pip install log-foundry[aws]
-version = "0.10.2.dev71"
+version = "0.10.2.dev73"
 
 [project.optional-dependencies]
 aws = ["boto3>=1.43.61"]              # SQSSink, SNSSink, KinesisSink, FirehoseSink
@@ -178,6 +178,19 @@ ignore = [
 # trip this rule.
 "src/log_foundry/sinks/postgres.py" = ["S608"]
 "src/log_foundry/sinks/sqlite.py" = ["S608"]
+# The integration tests build DDL and SELECTs around a table name they generate themselves --
+# a `uuid4().hex` slice, so the table can be created and dropped per test without two runs
+# colliding. Scoped to these two files for the same reason the two sinks above are scoped
+# individually rather than the rule being switched off for `tests/**`: a future test that
+# interpolates something it did not generate should still trip it.
+"tests/integration/test_postgres.py" = ["S608"]
+"tests/integration/test_clickhouse.py" = ["S608"]
+# The NATS outage test stops and restarts its own compose service, because "the server went
+# away" is the condition under test and no fake reproduces what the real client does about it.
+# Every argument is a literal or a path this module derives from `__file__`; there is no shell
+# and no caller input. `docker` is resolved from PATH deliberately -- CI and a developer laptop
+# put it in different places.
+"tests/integration/test_nats.py" = ["S603", "S607"]
 # make-sbom.py exists to drive two virtualenvs it creates itself, so subprocess is the whole job.
 # Every argv element is either a literal, a path under a `tempfile.TemporaryDirectory` this script
 # just made, or a value read from the repository's own pyproject.toml / poetry.lock. There is no

{log_foundry-0.10.2.dev71 → log_foundry-0.10.2.dev73}/src/log_foundry/sinks/kafka.py

@@ -53,6 +53,15 @@ class KafkaSink:
     returns without blocking, while the delivery result arrives asynchronously on a callback
     serviced by ``poll()`` and ``flush()``.
 
+    **Retry (SPEC-041 FR-004).** This sink adds none and needs none: ``produce()`` is a local
+    hand-off that returns without waiting — measured at 0.0000 s for three messages against a
+    dead broker — so it never holds the worker's single drain thread, and librdkafka's own retry
+    is bounded by ``message.timeout.ms`` (five minutes by default). Measured: with that set to
+    1500 ms the delivery callback fired at 2.01 s, and at 4000 ms it fired at 4.01 s. SPEC-027's
+    "cut short by a shutdown" governs a wait *between attempts*, and this sink has none to cut —
+    which is also why it deliberately exposes no ``log_foundry_stop_signal`` attribute, since
+    ``_lifecycle.offer_stop_signal`` probes by ``hasattr`` and that absence is the opt-out.
+
     The worst case (SPEC-027 FR-005) is not a retry loop — this sink has none — but its
     ``close()``: one ``flush_timeout`` wait, 10 s at the default, spent **beside**
     ``shutdown()``'s budget rather than from it. ``Worker.shutdown`` joins the drain thread

{log_foundry-0.10.2.dev71 → log_foundry-0.10.2.dev73}/src/log_foundry/sinks/logstash.py

@@ -15,16 +15,44 @@ from log_foundry.sinks.http import HTTPSink
 
 __all__ = ["LogstashSink"]
 
+_BODY_FORMATS = frozenset({"json_array", "ndjson"})
+"""The two HTTP wire forms, validated rather than passed through.
+
+``HTTPSink._body`` branches on ``== "json_array"`` and treats everything else as NDJSON, so a
+typo -- ``"json-array"`` -- would silently ship the exact body FR-003 exists to stop shipping,
+against a destination that cannot parse it. A parameter documented as taking one of two values
+refuses the third.
+"""
+
 
 class LogstashSink:
     """A :class:`~log_foundry.sinks.base.Sink` that ships JSON lines to Logstash (FR-005).
 
-    Two mutually-exclusive modes are chosen at construction: with a URL the batch goes as JSON
-    lines over HTTP, reusing the ``HTTPSink`` core, and with a host and port each event goes as
-    one newline-terminated line over a raw TCP or UDP socket, reusing
+    Two mutually-exclusive modes are chosen at construction: with a URL the batch goes over HTTP,
+    reusing the ``HTTPSink`` core, and with a host and port each event goes as one
+    newline-terminated line over a raw TCP or UDP socket, reusing
     :class:`~log_foundry.sinks._socket.SocketTransport`. Either backend handles its own bounded
     retry and raises on total failure of its own accord, so ``emit`` needs no rule of its own.
 
+    **What the Logstash side must be configured as** (SPEC-041 FR-003). In HTTP mode the default
+    ``body_format="json_array"`` posts a JSON array as ``application/json``, which a **stock**
+    ``http`` input parses into one event per element with no configuration at all. That is a
+    change: this sink used to post NDJSON as ``application/x-ndjson``, which the ``http`` input's
+    default ``additional_codecs`` map (``{"application/json" => "json"}``) does not cover, so the
+    body fell through to the ``plain`` codec. Measured against Logstash 8.15 — a batch of three
+    arrived as **one** event with the whole payload as text in ``message`` and every structured
+    field gone.
+
+    ``body_format="ndjson"`` restores the old wire form, and it is not merely legacy: an input
+    configured ``additional_codecs => {"application/x-ndjson" => "json_lines"}`` -- the documented
+    workaround for the behaviour above -- needs it. That setting **replaces** the default map
+    rather than merging with it, so such an input parses NDJSON correctly and a JSON array
+    incorrectly. The two configurations are mutually exclusive unless the input lists both, which
+    is why the wire form is a parameter rather than a silent change.
+
+    Socket mode is unaffected: it is newline-delimited either way, and a ``tcp``/``udp`` input
+    uses the ``line``/``json_lines`` codec rather than ``additional_codecs``.
+
     In socket mode both IPv4 and IPv6 destinations are supported, over either transport
     (SPEC-031 FR-002): the UDP socket's address family is resolved from ``host`` rather than
     assumed, which is what an unconditional ``AF_INET`` used to make impossible.
@@ -58,6 +86,7 @@ class LogstashSink:
         host: str | None = None,
         port: int | None = None,
         transport: str = "tcp",
+        body_format: str = "json_array",
         timeout: float = 5.0,
         max_retries: int = 3,
         max_datagram_bytes: int = DEFAULT_MAX_DATAGRAM_BYTES,
@@ -70,6 +99,12 @@ class LogstashSink:
           host: The destination host, selecting socket mode.
           port: The destination port, selecting socket mode.
           transport: ``"tcp"`` or ``"udp"``, in socket mode.
+          body_format: ``"json_array"`` (the default) or ``"ndjson"``. It selects the HTTP wire
+            form and has **no effect** in socket mode, which is always newline-delimited — but it
+            is *validated* in both, because a value this class silently ignores in one mode and
+            silently misreads in the other is worse than a refusal. See the class docstring for
+            what each one requires of the Logstash side; ``"ndjson"`` is the escape hatch for an
+            input already configured with ``additional_codecs``.
           timeout: Seconds allowed per request or connection.
           max_retries: Retries the chosen backend makes.
           max_datagram_bytes: In UDP socket mode, the largest datagram to attempt; a frame over
@@ -81,11 +116,17 @@ class LogstashSink:
           None.
 
         Raises:
-          ValueError: If neither a URL nor a host and port were given.
+          ValueError: If neither a URL nor a host and port were given, or if ``body_format`` is
+            neither ``"json_array"`` nor ``"ndjson"``.
         """
+        if body_format not in _BODY_FORMATS:
+            raise ValueError(
+                f"LogstashSink body_format must be one of {sorted(_BODY_FORMATS)}, "
+                f"not {body_format!r}"
+            )
         if url is not None:
             self._http: HTTPSink | None = HTTPSink(
-                url, body_format="ndjson", timeout=timeout, max_retries=max_retries,
+                url, body_format=body_format, timeout=timeout, max_retries=max_retries,
                 **http_kwargs,  # type: ignore[arg-type]
             )
             self._socket: SocketTransport | None = None

{log_foundry-0.10.2.dev71 → log_foundry-0.10.2.dev73}/src/log_foundry/sinks/nats.py

@@ -21,6 +21,31 @@ class NATSSink:
     completion from the synchronous ``emit``, which therefore returns only after the batch is
     handed off. With JetStream enabled it publishes through JetStream for durable
     acknowledgement.
+
+    **Retry and the worst-case delay** (SPEC-041 FR-004). This sink adds no retry loop and needs
+    none: a core ``publish()`` writes into the client's outbound buffer and returns without
+    waiting — measured at 0.00 s for fifty publishes — so it never holds the worker's single
+    drain thread and there is no backoff for a shutdown to cut short. SPEC-027's guarantee is met
+    because there is no wait, not because a wait is bounded. Under JetStream ``publish()`` awaits
+    an ack bounded by the driver's own timeout (5 s by default) and does not retry.
+
+    **A disconnected client is reported, not absorbed** (FR-004 AC-5). That non-blocking publish
+    is exactly what made this sink report success for events that had not left the process: with
+    the server stopped, five successive emits each returned in 0.00 s with ``losses()`` reading
+    all zeros, and when the client's reconnect budget (60 attempts × 2 s by default) ran out
+    first, **one of six events reached the destination with every counter still at zero**. That
+    is SPEC-026 FR-001's shape — a sink the worker believes, so its retry never engages and
+    ``failed_batches`` never moves. ``emit`` now refuses a batch while the client reports itself
+    disconnected.
+
+    The limit is stated rather than overclaimed: ``is_connected`` does not flip the instant the
+    server dies, so the first batch in the window before the client notices is still accepted and
+    still buffered — measured, one of five emits landed in that window. It is the same
+    check-then-act window ``MongoDBSink`` and ``KafkaSink`` already document for their own flags;
+    what the guard ends is the far larger case of an outage that has been going on for any
+    appreciable time. Refusing moves no ``losses()`` counter, per SPEC-032: it is a failure
+    *reported* to the worker rather than one absorbed, and counting both would report one loss
+    twice.
     """
 
     def __init__(
@@ -86,8 +111,38 @@ class NATSSink:
                 raise SinkDeliveryError(
                     f"NATSSink published none of {len(batch)} event(s): the sink is closed"
                 )
+            if not self._is_connected():
+                raise SinkDeliveryError(
+                    f"NATSSink published none of {len(batch)} event(s): "
+                    "the client is disconnected"
+                )
             self._loop.run_until_complete(self._publish_all(batch))
 
+    def _is_connected(self) -> bool:
+        """Reports whether the client can currently put anything on the wire (FR-004 AC-5).
+
+        Probed by name, as ``drain`` and ``flush`` are, because this sink is written against a
+        driver it does not own and accepts an injected ``client=``. A client that does not
+        publish the attribute is assumed connected: the guard exists to convert a *known*
+        non-delivery into a reported one, and inventing a refusal for a client that never
+        claimed to be disconnected would fail batches that were going to succeed.
+
+        Args:
+          None.
+
+        Returns:
+          True when the client reports itself connected, or says nothing about it.
+
+        Raises:
+          None. A driver whose attribute access raises is treated as connected, so a probe can
+            never be the reason a batch fails.
+        """
+        try:
+            connected = getattr(self._client, "is_connected", True)
+        except Exception:
+            return True
+        return bool(connected)
+
     def losses(self) -> SinkLosses:
         """Reports events whose publish raised (SPEC-026 FR-002).
 

{log_foundry-0.10.2.dev71 → log_foundry-0.10.2.dev73}/src/log_foundry/sinks/postgres.py

@@ -15,6 +15,22 @@ __all__ = ["PostgresSink"]
 
 _BACKOFF_BASE = 0.1
 
+DEFAULT_CONNECT_TIMEOUT = 5
+"""Seconds libpq may spend opening one connection (SPEC-041 FR-002).
+
+``psycopg.connect()`` takes no timeout unless one is given, and libpq's default is to wait
+indefinitely. That call runs on the worker's single drain thread, holding this sink's emit lock,
+and it consults no stop signal -- so against a host that blackholes packets rather than refusing
+them it is exactly the unbounded, uninterruptible wait SPEC-027 exists to remove. Measured
+against an unroutable address: **75.01 s** with no timeout, **2.03 s** with ``connect_timeout=2``.
+
+Floored at libpq's own minimum of 2 rather than accepted verbatim, because ``0`` means "wait
+forever" -- reinstating the defect -- the same reason ``KafkaSink._usable_timeout`` refuses its
+own degenerate values (SPEC-038 FR-006).
+"""
+
+_MIN_CONNECT_TIMEOUT = 2
+
 _COLUMNS = ("timestamp", "level", "trace_id", "span_id", "function", "service")
 
 
@@ -23,8 +39,15 @@ class PostgresSink:
 
     Each event is stored as a ``JSONB`` ``event`` column plus a few extracted columns for
     indexing. ``psycopg`` v3 is the optional ``postgres`` extra, imported lazily. The sink is
-    write-only, and the worst-case delay (SPEC-027 FR-005) is ``max_retries`` interruptible waits
-    per batch, 0.7 s at the defaults.
+    write-only. The worst-case delay (SPEC-027 FR-005) has two halves, and only one of them is
+    interruptible. The backoffs are ``max_retries`` waits per batch — 0.7 s at the defaults —
+    taken through ``_retry.wait`` on the worker's stop event, so a shutdown cuts them short. The
+    reconnects are up to ``max_retries + 1`` connects of ``connect_timeout`` each, 20 s more at
+    the defaults, and they are **bounded but not interruptible**: the wait is inside libpq, which
+    consults nothing. That is the settled line rather than a gap — a shutdown shortens a *wait*
+    and never skips *work*, and a reconnect is the work an in-flight batch needs (SPEC-038
+    FR-001 AC-4a). Both halves sit inside the existing retry budget rather than a loop of their
+    own, which is what bounds them (FR-002 AC-3).
 
     The driver requirement satisfied (SPEC-028 FR-002): a ``psycopg`` connection carries one
     transaction, and this sink's unit of work is a ``cursor`` / ``commit`` / ``rollback`` sequence
@@ -48,6 +71,7 @@ class PostgresSink:
         create_table: bool = False,
         chunk_size: int = 1000,
         max_retries: int = 3,
+        connect_timeout: int = DEFAULT_CONNECT_TIMEOUT,
     ) -> None:
         """Connects to the database and prepares the insert statement.
 
@@ -61,6 +85,13 @@ class PostgresSink:
           max_retries: Retries per batch, floored at zero as ``Worker._emit`` floors its own
             (SPEC-021) — a negative value returned having attempted no insert at all, and
             reported success.
+          connect_timeout: Seconds libpq may spend opening a connection. Defaults to
+            :data:`DEFAULT_CONNECT_TIMEOUT` (5) and is floored at libpq's own minimum of 2, since
+            ``0`` means "wait forever" and would reinstate the unbounded connect this argument
+            exists to remove. It is passed explicitly and therefore **overrides any
+            ``connect_timeout`` in the DSN** — a DSN asking for 30 gets this value instead, so
+            set it here rather than there. Applies to the connection opened at construction and
+            to every reconnect.
 
         Returns:
           None.
@@ -78,11 +109,12 @@ class PostgresSink:
         self._lock = threading.Lock()
         self._counter_lock = threading.Lock()
         self._owns_connection = connection is None
+        self._dsn = dsn
+        self.connect_timeout = max(connect_timeout, _MIN_CONNECT_TIMEOUT)
         if connection is None:
-            import psycopg  # type: ignore[import-not-found]
-
-            connection = psycopg.connect(dsn)
+            connection = self._connect()
         self._conn = connection
+        self._reconnect_announced = False
         columns = ", ".join((*_COLUMNS, "event"))
         placeholders = ", ".join(["%s"] * len(_COLUMNS) + ["%s::jsonb"])
         self._insert_sql = f"INSERT INTO {self._table} ({columns}) VALUES ({placeholders})"
@@ -153,6 +185,7 @@ class PostgresSink:
         """
         for attempt in range(self.max_retries + 1):
             try:
+                self._reconnect_if_broken()
                 with self._conn.cursor() as cur:
                     for chunk in chunk_list(batch, self._chunk_size):
                         cur.executemany(self._insert_sql, [self._row(event) for event in chunk])
@@ -168,12 +201,126 @@ class PostgresSink:
                 _diag.lost(
                     "event",
                     len(batch),
-                    f"PostgresSink, {self.max_retries + 1} attempts, {type(err).__name__}",
+                    f"PostgresSink, {self.max_retries + 1} attempts, {type(err).__name__}"
+                    + (
+                        ", borrowed connection is broken and this sink may not reopen it"
+                        if not self._owns_connection and self._is_broken()
+                        else ""
+                    ),
                 )
                 raise SinkDeliveryError(
                     f"PostgresSink inserted none of {len(batch)} event(s)"
                 ) from None
 
+    def _connect(self) -> Any:
+        """Opens a connection to the configured DSN, bounded by :attr:`connect_timeout`.
+
+        The bound is passed as a keyword rather than left to the DSN, so it holds **regardless of**
+        what the caller's connection string says — which is the same fact ``__init__`` states from
+        the caller's side, that this argument overrides a ``connect_timeout`` in the DSN. See
+        :data:`DEFAULT_CONNECT_TIMEOUT` for why an unbounded connect here is the defect rather
+        than the default.
+
+        Args:
+          None.
+
+        Returns:
+          The new connection.
+
+        Raises:
+          ImportError: If the ``postgres`` extra is not installed.
+          Exception: Whatever the driver raises when connecting.
+        """
+        import psycopg  # type: ignore[import-not-found]
+
+        return psycopg.connect(self._dsn, connect_timeout=self.connect_timeout)
+
+    def _reconnect_if_broken(self) -> None:
+        """Replaces an unusable **owned** connection before an insert attempt (FR-002).
+
+        A ``psycopg`` connection is permanently unusable once the server closes it, and this sink
+        opened one in ``__init__`` and never reopened it — so a single restart, failover or idle
+        timeout ended log delivery for the life of the process, with every in-batch retry running
+        against the same dead handle. Measured against a real Postgres: one
+        ``pg_terminate_backend`` and three subsequent batches were lost, one row delivered.
+        Every sibling already recovers (``SocketTransport._reset``, boto3, clickhouse-connect's
+        pool, pymongo's pool).
+
+        **It runs at the top of each attempt, not in the retry branch.** ``max_retries`` is a
+        public argument floored at zero, so a reconnect placed where a retry remains never runs
+        at all at ``max_retries=0`` and the defect would survive at that setting forever. At the
+        top of the attempt, the first emit after a failure reopens — which is FR-002 AC-1
+        literally — at every value. It also avoids spending a connect immediately before the
+        raise, where it can only cost time.
+
+        **A borrowed connection is never reopened**, per ``architecture.md`` §13's borrowed-client
+        constraint: the caller owns that object and its lifetime, and replacing it here would
+        leak theirs and reconnect a session they may be sharing. The exhausted batch's existing
+        ``_diag.lost`` line gains a clause rather than a new stderr site, and it is conditioned on
+        the connection actually being broken — appending it to every borrowed-connection failure
+        would put "not reopened" on a constraint violation, where reopening was never the
+        question.
+
+        The state is read from the connection rather than inferred: ``closed`` and ``broken`` are
+        both ``False`` before the first failure and both ``True`` after it, so an ordinary SQL
+        error — a constraint violation, a full disk — does not churn the connection. Both are
+        probed by name because this sink accepts any ``psycopg``-shaped object it does not own.
+
+Both diagnostics here are **announced once per outage, not once per attempt**. Unthrottled
+        they fire on every attempt of every batch, so a down server turned one stderr line into
+        five per batch, indefinitely — a diagnostic that floods is one an operator stops reading,
+        and the batch's own ``_diag.lost`` line already records the loss. The flag covers the
+        failed ``close()`` as well as the failed connect, because a failed reconnect leaves the
+        old object in place and the next attempt closes it again. It clears on a successful
+        reconnect, so a later outage is announced again.
+
+        Args:
+          None.
+
+        Returns:
+          None.
+
+        Raises:
+          None. A failed reconnect is absorbed and left to the next attempt, which is what keeps
+            this inside the existing retry budget rather than adding a loop of its own (AC-3):
+            the batch's bound is unchanged, and a connect that cannot succeed simply spends an
+            attempt visibly, through the counters.
+        """
+        if not self._owns_connection or not self._is_broken():
+            return
+        announced, self._reconnect_announced = self._reconnect_announced, True
+        try:
+            self._conn.close()
+        except Exception as err:
+            if not announced:
+                _diag.absorbed("PostgresSink.close of a broken connection", err)
+        try:
+            self._conn = self._connect()
+            self._reconnect_announced = False
+        except Exception as err:
+            if not announced:
+                _diag.absorbed("PostgresSink.reconnect", err)
+
+    def _is_broken(self) -> bool:
+        """Reports whether the held connection can no longer be used.
+
+        Args:
+          None.
+
+        Returns:
+          True when the driver reports the connection closed or broken.
+
+        Raises:
+          None. A driver whose attribute access raises is treated as usable, so a probe can
+            never be the reason a batch fails.
+        """
+        try:
+            return bool(getattr(self._conn, "closed", False)) or bool(
+                getattr(self._conn, "broken", False)
+            )
+        except Exception:
+            return False
+
     def _rollback(self) -> None:
         """Discards the failed transaction, absorbing a rollback that itself fails (FR-002).
 
@@ -210,6 +357,13 @@ class PostgresSink:
         Idempotent, and takes the emit lock so the final commit never lands in the middle of
         another thread's transaction (SPEC-028 FR-002).
 
+        **The final commit is guarded** (SPEC-041 FR-002). It was unconditional, so a connection
+        the server had closed made ``close()`` raise — reachable without any exotic timing, since
+        a broken connection is only repaired inside an emit attempt and a process that breaks and
+        then shuts down never has another. The release still has to happen, and a commit that
+        cannot reach the server has nothing to publish, so the failure is announced by type and
+        the close proceeds.
+
         Args:
           None.
 
@@ -217,13 +371,16 @@ class PostgresSink:
           None.
 
         Raises:
-          Exception: Whatever the driver raises on commit or close.
+          Exception: Whatever the driver raises on close. The commit no longer escapes.
         """
         with self._lock:
             if self._closed:
                 return
             self._closed = True
-            self._conn.commit()
+            try:
+                self._conn.commit()
+            except Exception as err:
+                _diag.absorbed("PostgresSink.close commit", err)
             if self._owns_connection:
                 self._conn.close()
 

{log_foundry-0.10.2.dev71 → log_foundry-0.10.2.dev73}/src/log_foundry/sinks/pubsub.py

@@ -59,6 +59,14 @@ class GooglePubSubSink:
     imported lazily. ``publish()`` returns a future that resolves asynchronously, so the sink
     accumulates the batch's futures and resolves them on :meth:`close`.
 
+    **Retry (SPEC-041 FR-004).** The client's own retry is bounded and runs on the client's
+    threads, never the worker's drain thread: the generated ``publish`` carries
+    ``Retry(initial=0.1, maximum=60.0, multiplier=4, deadline=600.0)`` with a 60 s per-call
+    timeout, so a publish gives up after ten minutes at the outside. What this sink contributes
+    is the only wait the drain thread ever takes — :meth:`_await_overflow`'s
+    ``overflow_timeout``, bounded and interruptible per SPEC-027. Measured with the destination
+    stopped and ``overflow_timeout=5.0``: every over-bound emit returned in 5.00 s exactly.
+
     The driver requirement satisfied (SPEC-028 FR-002): this sink takes **no** transport lock —
     the publisher client owns its own batching and threading, and ``publish()`` is a local
     hand-off. What it does hold is the pending-futures list, which is genuinely shared between