log-foundry 0.10.2.dev90__tar.gz → 0.10.2.dev92__tar.gz

{log_foundry-0.10.2.dev90 → log_foundry-0.10.2.dev92}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: log-foundry
-Version: 0.10.2.dev90
+Version: 0.10.2.dev92
 Summary: Generate logs for your console and JSON events for downstream consumption.
 License-Expression: MIT
 License-File: LICENSE
@@ -818,23 +818,37 @@ Two things worth knowing:
 #### Queue & stream
 
 Each needs its own extra (lazy-imported). All publish within a bound and close cleanly. **Where the
-retry lives differs, and it was measured rather than assumed** (SPEC-041 FR-004): the Redis,
-RabbitMQ and Event Hubs sinks retry through `sinks/_retry`, so their backoff is bounded *and* cut
-short by a shutdown. `KafkaSink`, `NATSSink` and `GooglePubSubSink` add no retry loop and need
-none — each hands off locally and returns without waiting, so nothing of theirs holds the single
-drain thread. Their clients retry on their own threads within their own bounds:
-`message.timeout.ms` (5 min default) for Kafka, a 600 s deadline for Pub/Sub, and for NATS a
-JetStream publish bounded by its 5 s ack timeout with no retry at all. `NATSSink` refuses a batch
-outright while its client reports itself disconnected, so a sustained outage moves
-`health().failed_batches` instead of being absorbed.
+bound lives differs, and it was measured rather than assumed** (SPEC-041 FR-004, SPEC-047): the
+Redis, RabbitMQ and Event Hubs sinks retry through `sinks/_retry`, so their backoff is bounded
+*and* cut short by a shutdown.
+
+`KafkaSink` and `GooglePubSubSink` add no retry loop and need none — each hands off locally and
+returns without waiting, and their clients retry on their own threads within their own bounds.
+(`KafkaSink` holds nothing at all; `GooglePubSubSink` does take one wait on the drain thread, its
+`overflow_timeout`, when a batch exceeds `max_pending` — bounded and interruptible per SPEC-027.) For Kafka that is `message.timeout.ms`, five
+minutes by default (measured: the delivery callback fires at 300.18 s), reachable through
+`producer_config=`; for Pub/Sub, a 600 s deadline.
+
+> ~~and for NATS a JetStream publish bounded by its 5 s ack timeout with no retry at all~~ —
+> **superseded by SPEC-047 FR-001.** That was true per *event* and false per *batch*: the awaits
+> were sequential and nothing bounded how many there were, so a stalled server cost `n × 5 s` on
+> the single drain thread (measured: 25.01 s for five events), and `_final_drain` hands the exit
+> backlog over as one batch.
+
+`NATSSink` now bounds a whole `emit` with one `publish_timeout` (10 s by default), giving each
+JetStream publish the lesser of the driver's ack timeout and the budget remaining; a core publish
+takes no timeout and is bounded between events. Its connect, reconnect and drain timeouts are
+reachable from the constructor — at the driver's defaults, construction against an unreachable
+server blocks for a measured 120.17 s. It refuses a batch outright while its client reports itself
+disconnected, so a sustained outage moves `health().failed_batches` instead of being absorbed.
 
 | Sink | Import from | Extra | Configure |
 |---|---|---|---|
-| `KafkaSink` | `log_foundry.sinks.kafka` | `kafka` | `KafkaSink(topic, *, flush_timeout=10.0, bootstrap_servers="…", key_field="trace_id")` |
+| `KafkaSink` | `log_foundry.sinks.kafka` | `kafka` | `KafkaSink(topic, *, flush_timeout=10.0, bootstrap_servers="…", key_field="trace_id", producer_config=None)` — `producer_config` is merged **beneath** the sink's own keys, so it reaches `message.timeout.ms` and friends without displacing `bootstrap.servers`; passing it with `producer=` is a `ValueError` |
 | `RedisStreamsSink` | `log_foundry.sinks.redis` | `redis` | `RedisStreamsSink(stream, *, url=None, maxlen=None)` — `XADD`. `maxlen` caps the stream (`approximate=True`); trimming happens **at Redis**, after delivery, so it is invisible to `health()` — which is why the default is unbounded |
 | `RedisListSink` | `log_foundry.sinks.redis` | `redis` | `RedisListSink(key, *, url=None, maxlen=None)` — `RPUSH` + `LTRIM` to the newest `maxlen`; same destination-side trimming caveat |
 | `RabbitMQSink` | `log_foundry.sinks.rabbitmq` | `amqp` | `RabbitMQSink(*, exchange, routing_key, url=None)` — persistent messages |
-| `NATSSink` | `log_foundry.sinks.nats` | `nats` | `NATSSink(subject, *, jetstream=False, servers=None)` |
+| `NATSSink` | `log_foundry.sinks.nats` | `nats` | `NATSSink(subject, *, jetstream=False, servers=None, publish_timeout=10.0, connect_timeout=None, max_reconnect_attempts=None, reconnect_time_wait=None, drain_timeout=None)` — `publish_timeout` bounds one whole `emit` and applies to an injected `client=` too; the four `None` timeouts are forwarded to `nats.connect` only when set, and passing one with `client=` is a `ValueError` |
 | `GooglePubSubSink` | `log_foundry.sinks.pubsub` | `gcp-pubsub` | `GooglePubSubSink(topic)` |
 | `AzureEventHubsSink` | `log_foundry.sinks.eventhubs` | `azure-eventhubs` | `AzureEventHubsSink(*, connection_str="…", eventhub=None)` |
 

{log_foundry-0.10.2.dev90 → log_foundry-0.10.2.dev92}/README.md

@@ -782,23 +782,37 @@ Two things worth knowing:
 #### Queue & stream
 
 Each needs its own extra (lazy-imported). All publish within a bound and close cleanly. **Where the
-retry lives differs, and it was measured rather than assumed** (SPEC-041 FR-004): the Redis,
-RabbitMQ and Event Hubs sinks retry through `sinks/_retry`, so their backoff is bounded *and* cut
-short by a shutdown. `KafkaSink`, `NATSSink` and `GooglePubSubSink` add no retry loop and need
-none — each hands off locally and returns without waiting, so nothing of theirs holds the single
-drain thread. Their clients retry on their own threads within their own bounds:
-`message.timeout.ms` (5 min default) for Kafka, a 600 s deadline for Pub/Sub, and for NATS a
-JetStream publish bounded by its 5 s ack timeout with no retry at all. `NATSSink` refuses a batch
-outright while its client reports itself disconnected, so a sustained outage moves
-`health().failed_batches` instead of being absorbed.
+bound lives differs, and it was measured rather than assumed** (SPEC-041 FR-004, SPEC-047): the
+Redis, RabbitMQ and Event Hubs sinks retry through `sinks/_retry`, so their backoff is bounded
+*and* cut short by a shutdown.
+
+`KafkaSink` and `GooglePubSubSink` add no retry loop and need none — each hands off locally and
+returns without waiting, and their clients retry on their own threads within their own bounds.
+(`KafkaSink` holds nothing at all; `GooglePubSubSink` does take one wait on the drain thread, its
+`overflow_timeout`, when a batch exceeds `max_pending` — bounded and interruptible per SPEC-027.) For Kafka that is `message.timeout.ms`, five
+minutes by default (measured: the delivery callback fires at 300.18 s), reachable through
+`producer_config=`; for Pub/Sub, a 600 s deadline.
+
+> ~~and for NATS a JetStream publish bounded by its 5 s ack timeout with no retry at all~~ —
+> **superseded by SPEC-047 FR-001.** That was true per *event* and false per *batch*: the awaits
+> were sequential and nothing bounded how many there were, so a stalled server cost `n × 5 s` on
+> the single drain thread (measured: 25.01 s for five events), and `_final_drain` hands the exit
+> backlog over as one batch.
+
+`NATSSink` now bounds a whole `emit` with one `publish_timeout` (10 s by default), giving each
+JetStream publish the lesser of the driver's ack timeout and the budget remaining; a core publish
+takes no timeout and is bounded between events. Its connect, reconnect and drain timeouts are
+reachable from the constructor — at the driver's defaults, construction against an unreachable
+server blocks for a measured 120.17 s. It refuses a batch outright while its client reports itself
+disconnected, so a sustained outage moves `health().failed_batches` instead of being absorbed.
 
 | Sink | Import from | Extra | Configure |
 |---|---|---|---|
-| `KafkaSink` | `log_foundry.sinks.kafka` | `kafka` | `KafkaSink(topic, *, flush_timeout=10.0, bootstrap_servers="…", key_field="trace_id")` |
+| `KafkaSink` | `log_foundry.sinks.kafka` | `kafka` | `KafkaSink(topic, *, flush_timeout=10.0, bootstrap_servers="…", key_field="trace_id", producer_config=None)` — `producer_config` is merged **beneath** the sink's own keys, so it reaches `message.timeout.ms` and friends without displacing `bootstrap.servers`; passing it with `producer=` is a `ValueError` |
 | `RedisStreamsSink` | `log_foundry.sinks.redis` | `redis` | `RedisStreamsSink(stream, *, url=None, maxlen=None)` — `XADD`. `maxlen` caps the stream (`approximate=True`); trimming happens **at Redis**, after delivery, so it is invisible to `health()` — which is why the default is unbounded |
 | `RedisListSink` | `log_foundry.sinks.redis` | `redis` | `RedisListSink(key, *, url=None, maxlen=None)` — `RPUSH` + `LTRIM` to the newest `maxlen`; same destination-side trimming caveat |
 | `RabbitMQSink` | `log_foundry.sinks.rabbitmq` | `amqp` | `RabbitMQSink(*, exchange, routing_key, url=None)` — persistent messages |
-| `NATSSink` | `log_foundry.sinks.nats` | `nats` | `NATSSink(subject, *, jetstream=False, servers=None)` |
+| `NATSSink` | `log_foundry.sinks.nats` | `nats` | `NATSSink(subject, *, jetstream=False, servers=None, publish_timeout=10.0, connect_timeout=None, max_reconnect_attempts=None, reconnect_time_wait=None, drain_timeout=None)` — `publish_timeout` bounds one whole `emit` and applies to an injected `client=` too; the four `None` timeouts are forwarded to `nats.connect` only when set, and passing one with `client=` is a `ValueError` |
 | `GooglePubSubSink` | `log_foundry.sinks.pubsub` | `gcp-pubsub` | `GooglePubSubSink(topic)` |
 | `AzureEventHubsSink` | `log_foundry.sinks.eventhubs` | `azure-eventhubs` | `AzureEventHubsSink(*, connection_str="…", eventhub=None)` |
 

{log_foundry-0.10.2.dev90 → log_foundry-0.10.2.dev92}/pyproject.toml

@@ -20,7 +20,7 @@ dependencies = [
 ]
 
 # Optional features. Install with: pip install log-foundry[aws]
-version = "0.10.2.dev90"
+version = "0.10.2.dev92"
 
 [project.optional-dependencies]
 aws = ["boto3>=1.43.61"]              # SQSSink, SNSSink, KinesisSink, FirehoseSink

{log_foundry-0.10.2.dev90 → log_foundry-0.10.2.dev92}/src/log_foundry/sinks/kafka.py

@@ -54,7 +54,7 @@ class KafkaSink:
     serviced by ``poll()`` and ``flush()``.
 
     **Retry (SPEC-041 FR-004).** This sink adds none and needs none: ``produce()`` is a local
-    hand-off that returns without waiting — measured at 0.0000 s for three messages against a
+    hand-off that returns without waiting — measured at 0.0001 s for three messages against a
     dead broker — so it never holds the worker's single drain thread, and librdkafka's own retry
     is bounded by ``message.timeout.ms`` (five minutes by default). Measured: with that set to
     1500 ms the delivery callback fired at 2.01 s, and at 4000 ms it fired at 4.01 s. SPEC-027's
@@ -62,6 +62,14 @@ class KafkaSink:
     which is also why it deliberately exposes no ``log_foundry_stop_signal`` attribute, since
     ``_lifecycle.offer_stop_signal`` probes by ``hasattr`` and that absence is the opt-out.
 
+    **The bound is ``librdkafka``'s, and since SPEC-047 FR-003 it is reachable.** ``message.
+    timeout.ms`` is what decides how long an accepted message is retried, five minutes by
+    default — measured, the delivery callback fires at 300.18 s — and before ``producer_config``
+    nothing could change it without abandoning ``bootstrap_servers=`` and injecting a whole
+    producer. No log-foundry retry was added instead: ``produce()`` having returned means the
+    producer owns delivery, so re-sending duplicates whatever it eventually lands (SPEC-018), and
+    a second bound over a five-minute one multiplies the worst case rather than shortening it.
+
     The worst case (SPEC-027 FR-005) is not a retry loop — this sink has none — but its
     ``close()``: one ``flush_timeout`` wait, 10 s at the default, spent **beside**
     ``shutdown()``'s budget rather than from it. ``Worker.shutdown`` joins the drain thread
@@ -95,6 +103,7 @@ class KafkaSink:
         bootstrap_servers: str | None = None,
         key_field: str = "trace_id",
         flush_timeout: float = DEFAULT_FLUSH_TIMEOUT,
+        producer_config: dict[str, object] | None = None,
     ) -> None:
         """Binds the sink to a topic and a producer.
 
@@ -110,22 +119,43 @@ class KafkaSink:
             default, as ``Worker._emit`` floors its own retries (SPEC-021): ``0`` reinstates
             exactly the ``flush(0)`` that switched this sink's exit delivery off, and ``inf``
             reinstates the unbounded wait.
+          producer_config: ``librdkafka`` configuration merged **beneath** this sink's own keys
+            when it builds the producer (SPEC-047 FR-003). It is the route to the bound that
+            actually governs delivery here: this sink adds no retry — ``produce()`` is a local
+            hand-off, measured at 0.0001 s for three messages against a dead broker — and
+            ``librdkafka`` retries on its own thread within ``message.timeout.ms``, whose
+            five-minute default was measured at a 300.18 s delivery callback. Before this
+            argument nothing reached it, so a caller working to a shorter deadline could not fit
+            the sink inside one. The default is deliberately left alone: lowering it would drop
+            messages that a process surviving a two-minute broker outage delivers today, which
+            is the durable-buffer role ``architecture.md`` section 8 gives this sink.
 
         Returns:
           None.
 
         Raises:
-          ValueError: If no producer is injected and no broker list was given.
+          ValueError: If no producer is injected and no broker list was given, or if
+            ``producer_config`` is passed alongside ``producer=`` — it can only be applied to a
+            producer this sink builds, and ignoring it would silently discard the caller's bound
+            (SPEC-043's rule).
           ImportError: If the ``kafka`` extra is not installed.
         """
-        if producer is None:
+        if producer is not None:
+            if producer_config is not None:
+                raise ValueError(
+                    "KafkaSink cannot apply producer_config to an injected producer, which is "
+                    "already built; pass the settings where the producer is constructed"
+                )
+        else:
             if bootstrap_servers is None:
                 raise ValueError(
                     "KafkaSink requires bootstrap_servers when no producer is injected"
                 )
             from confluent_kafka import Producer  # type: ignore[import-not-found]
 
-            producer = Producer({"bootstrap.servers": bootstrap_servers})
+            producer = Producer(
+                {**(producer_config or {}), "bootstrap.servers": bootstrap_servers}
+            )
         self.topic = topic
         self.producer = producer
         self.key_field = key_field

{log_foundry-0.10.2.dev90 → log_foundry-0.10.2.dev92}/src/log_foundry/sinks/nats.py

@@ -87,8 +87,11 @@ class NATSSink:
     disconnected.
 
     The limit is stated rather than overclaimed: ``is_connected`` does not flip the instant the
-    server dies, so the first batch in the window before the client notices is still accepted and
-    still buffered — measured, one of five emits landed in that window. It is the same
+    server dies, so a batch in the window before the client notices is still accepted and still
+    buffered. ~~Measured, one of five emits landed in that window.~~ **Corrected by SPEC-047
+    FR-004:** the window is wider than that reading suggests — the flag was still ``True``
+    **40 s after the server was stopped**, which is a lower bound on the window rather than the
+    window, because ``nats-py`` notices through a 120 s ping interval or a failed write. It is the same
     check-then-act window ``MongoDBSink`` and ``KafkaSink`` already document for their own flags;
     what the guard ends is the far larger case of an outage that has been going on for any
     appreciable time. Refusing moves no ``losses()`` counter, per SPEC-032: it is a failure
@@ -104,6 +107,10 @@ class NATSSink:
         jetstream: bool = False,
         servers: str | None = None,
         publish_timeout: float = DEFAULT_PUBLISH_TIMEOUT,
+        connect_timeout: float | None = None,
+        max_reconnect_attempts: int | None = None,
+        reconnect_time_wait: float | None = None,
+        drain_timeout: float | None = None,
     ) -> None:
         """Binds the sink to a subject and connects if no client was injected.
 
@@ -116,14 +123,52 @@ class NATSSink:
             :func:`~log_foundry.sinks._retry.usable_timeout` as ``KafkaSink`` floors its flush
             (SPEC-047 FR-001). It applies to an injected ``client=`` too: it is this sink's own
             bound over its own loop, not a request made of the driver at connect time.
+          connect_timeout: Seconds the driver may spend on one connection attempt, or ``None``
+            to pass nothing and leave the driver's own default (SPEC-047 FR-002).
+          max_reconnect_attempts: Attempts the driver makes before giving up, or ``None`` to
+            pass nothing. This is what governs the *initial* connect too, which is why the
+            constructor blocked for a measured 120.17 s against a dead server at the driver's
+            defaults (60 attempts x 2 s) before this argument existed. **``0`` and negative
+            values do not mean "give up immediately" — they block forever.** ``nats-py`` retires
+            a server from its pool only under ``if max_reconnect_attempts > 0``, so a
+            non-positive value never retires one and the connect loop does not terminate;
+            measured, both ``0`` and ``-1`` were still blocking at 30 s and one probe of ``0``
+            ran past 400 s. The smallest value that bounds anything is ``1`` (measured 2.02 s
+            against a dead server). Passed through rather than corrected here, because
+            overriding a driver's documented behaviour would surprise a caller who knows it.
+          reconnect_time_wait: Seconds between reconnect attempts, or ``None`` to pass nothing.
+          drain_timeout: Seconds ``Client.drain`` may spend, or ``None`` to pass nothing. It is
+            forwarded because it does bound the driver's subscription drain, but it is **not**
+            what bounds :meth:`close`: ``drain()`` ends with a ``flush()`` carrying the driver's
+            own 10 s default, and that is what fired in both measured cases (10.00 s against a
+            stalled server, 0.00 s against a stopped one). Recorded in ``architecture.md``
+            section 12.
 
         Returns:
           None.
 
         Raises:
+          ValueError: If a connect-time argument is passed alongside ``client=``, which cannot
+            consume it — SPEC-043's rule that an argument no backend can use is an error rather
+            than a silent ignore. ``publish_timeout`` is deliberately outside that set: it is
+            this sink's own bound and applies to any client.
           ImportError: If the ``nats`` extra is not installed.
           Exception: Whatever the driver raises when connecting.
         """
+        options: dict[str, object] = {
+            "connect_timeout": connect_timeout,
+            "max_reconnect_attempts": max_reconnect_attempts,
+            "reconnect_time_wait": reconnect_time_wait,
+            "drain_timeout": drain_timeout,
+        }
+        supplied = {name: value for name, value in options.items() if value is not None}
+        if client is not None and supplied:
+            raise ValueError(
+                "NATSSink cannot apply "
+                + ", ".join(sorted(supplied))
+                + " to an injected client, which is already connected; "
+                "pass them where the client is built, or drop client="
+            )
         self._subject = subject
         self._jetstream = jetstream
         self.publish_timeout = usable_timeout(publish_timeout, DEFAULT_PUBLISH_TIMEOUT)
@@ -135,7 +180,7 @@ class NATSSink:
             import nats  # type: ignore[import-not-found]
 
             client = self._loop.run_until_complete(
-                nats.connect(servers or "nats://localhost:4222")
+                nats.connect(servers or "nats://localhost:4222", **supplied)
             )
         self._client = client