log-foundry 0.10.2.dev23__tar.gz → 0.10.2.dev24__tar.gz

{log_foundry-0.10.2.dev23 → log_foundry-0.10.2.dev24}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: log-foundry
-Version: 0.10.2.dev23
+Version: 0.10.2.dev24
 Summary: Generate logs for your console and JSON events for downstream consumption.
 License-Expression: MIT
 License-File: LICENSE
@@ -669,7 +669,16 @@ in `CallbackSink`.
 #### Writing your own sink
 
 `Sink` is two required methods, `emit(batch)` and `close()`, plus two rules about *how* `emit`
-fails. They are not stylistic — the library's whole loss-reporting apparatus is built on them:
+fails and one about *when* it is called. They are not stylistic — the library's whole
+loss-reporting apparatus is built on them:
+
+- **Tolerate concurrent calls.** `emit` may run on more than one thread at once, and `close` may
+  be called while an `emit` is in flight. The worker drains on its own thread, but a level call
+  made with no active span emits synchronously on the *caller's* thread — which is any thread of
+  your application. If your sink holds mutable transport state (a stream it rebinds, a socket it
+  reuses, a connection with transaction scope), guard it with a `threading.Lock` held for the
+  whole operation that assumes exclusivity. If it holds none, you need do nothing. The library
+  cannot serialize this for you: it does not own the calling thread.
 
 - **Raise when you delivered none of the batch**, after your own retries are spent. That is the
   signal the worker's bounded retry and `health().failed_batches` depend on, and the one case where
@@ -685,29 +694,38 @@ never engages, `failed_batches` stays at zero, and `flush()` returns `True` whil
 lost.
 
 Optionally add `losses()` to report what you absorbed. It must never raise and must be safe to call
-while `emit` is running (`health()` is a poll):
+while `emit` is running (`health()` is a poll) — which means the counters need their own lock, kept
+separate from the transport one so a poll never waits on an in-flight send:
 
 ```python
+import threading
 from log_foundry.sinks.base import SinkDeliveryError, SinkLosses
 
 class MySink:
     def __init__(self) -> None:
         self._dropped = self._failed = 0
+        self._lock = threading.Lock()          # transport state
+        self._counter_lock = threading.Lock()  # counters only, never held across I/O
 
     def emit(self, batch: list[dict[str, object]]) -> None:
         delivered = 0
-        for chunk in self._chunks(batch):
-            if self._send(chunk):          # your own bounded retry
-                delivered += len(chunk)
-            else:
-                self._failed += len(chunk)
+        with self._lock:                       # your connection, socket or stream
+            for chunk in self._chunks(batch):
+                if self._send(chunk):          # your own bounded retry
+                    delivered += len(chunk)
+                else:
+                    with self._counter_lock:   # transport -> counter, never the reverse
+                        self._failed += len(chunk)
         if batch and not delivered:
             raise SinkDeliveryError(f"MySink delivered none of {len(batch)} event(s)")
 
     def losses(self) -> SinkLosses:
-        return SinkLosses(dropped=self._dropped, failed=self._failed)
+        with self._counter_lock:               # both fields from one instant
+            return SinkLosses(dropped=self._dropped, failed=self._failed)
 
-    def close(self) -> None: ...
+    def close(self) -> None:
+        with self._lock:                       # never release under an active writer
+            ...
 ```
 
 `losses()` is optional and probed by name, so a sink written before it existed keeps working and

{log_foundry-0.10.2.dev23 → log_foundry-0.10.2.dev24}/README.md

@@ -633,7 +633,16 @@ in `CallbackSink`.
 #### Writing your own sink
 
 `Sink` is two required methods, `emit(batch)` and `close()`, plus two rules about *how* `emit`
-fails. They are not stylistic — the library's whole loss-reporting apparatus is built on them:
+fails and one about *when* it is called. They are not stylistic — the library's whole
+loss-reporting apparatus is built on them:
+
+- **Tolerate concurrent calls.** `emit` may run on more than one thread at once, and `close` may
+  be called while an `emit` is in flight. The worker drains on its own thread, but a level call
+  made with no active span emits synchronously on the *caller's* thread — which is any thread of
+  your application. If your sink holds mutable transport state (a stream it rebinds, a socket it
+  reuses, a connection with transaction scope), guard it with a `threading.Lock` held for the
+  whole operation that assumes exclusivity. If it holds none, you need do nothing. The library
+  cannot serialize this for you: it does not own the calling thread.
 
 - **Raise when you delivered none of the batch**, after your own retries are spent. That is the
   signal the worker's bounded retry and `health().failed_batches` depend on, and the one case where
@@ -649,29 +658,38 @@ never engages, `failed_batches` stays at zero, and `flush()` returns `True` whil
 lost.
 
 Optionally add `losses()` to report what you absorbed. It must never raise and must be safe to call
-while `emit` is running (`health()` is a poll):
+while `emit` is running (`health()` is a poll) — which means the counters need their own lock, kept
+separate from the transport one so a poll never waits on an in-flight send:
 
 ```python
+import threading
 from log_foundry.sinks.base import SinkDeliveryError, SinkLosses
 
 class MySink:
     def __init__(self) -> None:
         self._dropped = self._failed = 0
+        self._lock = threading.Lock()          # transport state
+        self._counter_lock = threading.Lock()  # counters only, never held across I/O
 
     def emit(self, batch: list[dict[str, object]]) -> None:
         delivered = 0
-        for chunk in self._chunks(batch):
-            if self._send(chunk):          # your own bounded retry
-                delivered += len(chunk)
-            else:
-                self._failed += len(chunk)
+        with self._lock:                       # your connection, socket or stream
+            for chunk in self._chunks(batch):
+                if self._send(chunk):          # your own bounded retry
+                    delivered += len(chunk)
+                else:
+                    with self._counter_lock:   # transport -> counter, never the reverse
+                        self._failed += len(chunk)
         if batch and not delivered:
             raise SinkDeliveryError(f"MySink delivered none of {len(batch)} event(s)")
 
     def losses(self) -> SinkLosses:
-        return SinkLosses(dropped=self._dropped, failed=self._failed)
+        with self._counter_lock:               # both fields from one instant
+            return SinkLosses(dropped=self._dropped, failed=self._failed)
 
-    def close(self) -> None: ...
+    def close(self) -> None:
+        with self._lock:                       # never release under an active writer
+            ...
 ```
 
 `losses()` is optional and probed by name, so a sink written before it existed keeps working and

{log_foundry-0.10.2.dev23 → log_foundry-0.10.2.dev24}/pyproject.toml

@@ -20,7 +20,7 @@ dependencies = [
 ]
 
 # Optional features. Install with: pip install log-foundry[aws]
-version = "0.10.2.dev23"
+version = "0.10.2.dev24"
 
 [project.optional-dependencies]
 aws = ["boto3>=1.43.61"]              # SQSSink, SNSSink, KinesisSink, FirehoseSink

{log_foundry-0.10.2.dev23 → log_foundry-0.10.2.dev24}/src/log_foundry/sinks/_socket.py

@@ -108,6 +108,7 @@ class SocketTransport:
         self._max_retries = max(max_retries, 0)
         self._sock: socket.socket | None = None
         self.failed = 0
+        self._counter_lock = threading.Lock()
         self.stop_signal: threading.Event | None = None
         self._lock = threading.Lock()
 
@@ -153,7 +154,8 @@ class SocketTransport:
         Raises:
           None.
         """
-        return SinkLosses(dropped=0, failed=self.failed)
+        with self._counter_lock:
+            return SinkLosses(dropped=0, failed=self.failed)
 
     def close(self) -> None:
         """Closes the held socket, if any (FR-005, FR-012).
@@ -205,7 +207,8 @@ class SocketTransport:
                 if attempt < self._max_retries:
                     wait(_BACKOFF_BASE * (2**attempt), self.stop_signal)
                     continue
-                self.failed += 1
+                with self._counter_lock:
+                    self.failed += 1
                 _diag.lost(
                     "message",
                     1,

{log_foundry-0.10.2.dev23 → log_foundry-0.10.2.dev24}/src/log_foundry/sinks/base.py

@@ -57,6 +57,15 @@ class Sink(Protocol):
     ``None`` is the same answer as having no method at all, which is what keeps a third-party
     sink written against the pre-SPEC-026 interface satisfying this one. :func:`read_losses` is
     the probe.
+
+    "Safe to call during an emit" is a concurrency requirement once :meth:`emit` is (SPEC-028
+    FR-003). The shipped sinks keep their loss counters under a **dedicated** lock, separate
+    from whatever guards their transport: an increment is a read-modify-write that Python does
+    not promise is atomic, and the pair read must come from one instant rather than straddling
+    another thread's increments. It is deliberately not the transport lock — ``health()`` is
+    the call an operator makes when a destination is already hanging, and sharing one lock
+    would make that poll wait for an in-flight emit and its retry backoff. Where a sink holds
+    both, the order is always transport then counter, never the reverse.
     """
 
     def emit(self, batch: list[dict[str, object]]) -> None:

{log_foundry-0.10.2.dev23 → log_foundry-0.10.2.dev24}/src/log_foundry/sinks/clickhouse.py

@@ -3,10 +3,8 @@
 from __future__ import annotations
 
 import json
-from typing import TYPE_CHECKING, Any
-
-if TYPE_CHECKING:
-    import threading
+import threading
+from typing import Any
 
 from log_foundry import _diag
 from log_foundry.sinks._chunk import chunk_list, valid_identifier
@@ -42,6 +40,12 @@ class ClickHouseSink:
     columns plus the full event as a ``String`` column, inserted in a single call per chunk. The
     sink is write-only, and the worst-case delay (SPEC-027 FR-005) is ``max_retries``
     interruptible waits per chunk, 0.7 s at the defaults.
+
+    The driver requirement satisfied (SPEC-028 FR-002): a ``clickhouse-connect`` client holds
+    per-session state across an insert and the project does not publish it as safe to share
+    between threads, so this sink serializes its use rather than assuming otherwise. A lock is
+    the conservative reading — one client per thread would be the alternative, and that is the
+    connection-pool design FR-002 puts out of scope.
     """
 
     def __init__(
@@ -81,6 +85,8 @@ class ClickHouseSink:
         self.stop_signal: threading.Event | None = None
         self.failed = 0
         self._closed = False
+        self._lock = threading.Lock()
+        self._counter_lock = threading.Lock()
         self._owns_client = client is None
         if client is None:
             import clickhouse_connect  # type: ignore[import-not-found]
@@ -93,6 +99,9 @@ class ClickHouseSink:
     def losses(self) -> SinkLosses:
         """Reports rows in a chunk abandoned past the retry bound (SPEC-026 FR-002).
 
+        Reads under the counter lock rather than the emit lock (SPEC-028 FR-003), so a poll
+        never waits on an in-flight insert and its backoff.
+
         Args:
           None.
 
@@ -102,7 +111,8 @@ class ClickHouseSink:
         Raises:
           None.
         """
-        return SinkLosses(dropped=0, failed=self.failed)
+        with self._counter_lock:
+            return SinkLosses(dropped=0, failed=self.failed)
 
     def emit(self, batch: list[dict[str, object]]) -> None:
         """Inserts each chunk as one columnar call, retrying on failure (FR-002).
@@ -122,16 +132,18 @@ class ClickHouseSink:
         if not batch:
             return
         chunks = inserted = 0
-        for chunk in chunk_list(batch, self._chunk_size):
-            chunks += 1
-            inserted += self._insert([self._row(event) for event in chunk])
+        with self._lock:
+            for chunk in chunk_list(batch, self._chunk_size):
+                chunks += 1
+                inserted += self._insert([self._row(event) for event in chunk])
         if chunks and not inserted:
             raise SinkDeliveryError(f"ClickHouseSink inserted none of {chunks} chunk(s)")
 
     def close(self) -> None:
         """Closes the client only if the sink owns it (FR-005).
 
-        Idempotent.
+        Idempotent, and takes the emit lock so the client is never closed mid-insert
+        (SPEC-028 FR-002).
 
         Args:
           None.
@@ -142,11 +154,12 @@ class ClickHouseSink:
         Raises:
           Exception: Whatever the client raises on close.
         """
-        if self._closed:
-            return
-        if self._owns_client:
-            self.client.close()
-        self._closed = True
+        with self._lock:
+            if self._closed:
+                return
+            if self._owns_client:
+                self.client.close()
+            self._closed = True
 
     def _row(self, event: dict[str, object]) -> list[object]:
         """Builds one row: the extracted columns, then the whole event as JSON.
@@ -182,7 +195,8 @@ class ClickHouseSink:
                 if attempt < self.max_retries:
                     wait(_BACKOFF_BASE * (2**attempt), self.stop_signal)
                     continue
-                self.failed += len(rows)
+                with self._counter_lock:
+                    self.failed += len(rows)
                 _diag.lost(
                     "row",
                     len(rows),

{log_foundry-0.10.2.dev23 → log_foundry-0.10.2.dev24}/src/log_foundry/sinks/elasticsearch.py

@@ -98,10 +98,11 @@ class ElasticsearchSink(HTTPSink):
         Raises:
           None.
         """
-        return SinkLosses(
-            dropped=self.dropped_oversized,
-            failed=self.failed + self.item_errors + self.dropped_unadjudicated,
-        )
+        with self._counter_lock:
+            return SinkLosses(
+                dropped=self.dropped_oversized,
+                failed=self.failed + self.item_errors + self.dropped_unadjudicated,
+            )
 
     def _parse_bulk_response(self, payload: bytes, sent: int) -> bool:
         """Counts rejected items and reports whether the response proves nothing was indexed.
@@ -139,7 +140,8 @@ class ElasticsearchSink(HTTPSink):
         items = usable_results(data.get("items"))
         errors = sum(1 for item in items if _has_error(item))
         if len(items) != sent or not errors:
-            self.dropped_unadjudicated += sent
+            with self._counter_lock:
+                self.dropped_unadjudicated += sent
             _diag.lost(
                 "event",
                 sent,
@@ -148,7 +150,8 @@ class ElasticsearchSink(HTTPSink):
                 f"not retried",
             )
             return False
-        self.item_errors += errors
+        with self._counter_lock:
+            self.item_errors += errors
         _diag.lost("bulk item", errors, f"{type(self).__name__}, rejected by the server")
         return errors == sent
 

{log_foundry-0.10.2.dev23 → log_foundry-0.10.2.dev24}/src/log_foundry/sinks/eventhubs.py

@@ -3,10 +3,8 @@
 from __future__ import annotations
 
 import json
-from typing import TYPE_CHECKING, Any
-
-if TYPE_CHECKING:
-    import threading
+import threading
+from typing import Any
 
 from log_foundry import _diag
 from log_foundry.sinks._retry import wait
@@ -69,6 +67,7 @@ class AzureEventHubsSink:
         self.stop_signal: threading.Event | None = None
         self.failed = 0
         self.dropped_oversized = 0
+        self._counter_lock = threading.Lock()
 
     def losses(self) -> SinkLosses:
         """Reports oversized drops and events in a batch abandoned past the bound (FR-002).
@@ -82,7 +81,8 @@ class AzureEventHubsSink:
         Raises:
           None.
         """
-        return SinkLosses(dropped=self.dropped_oversized, failed=self.failed)
+        with self._counter_lock:
+            return SinkLosses(dropped=self.dropped_oversized, failed=self.failed)
 
     def emit(self, batch: list[dict[str, object]]) -> None:
         """Packs events into batches within the size limit and sends each (FR-009).
@@ -118,7 +118,8 @@ class AzureEventHubsSink:
                 delivered += self._send(current)
             current = self.producer.create_batch()
             if not _try_add(current, data):
-                self.dropped_oversized += 1
+                with self._counter_lock:
+                    self.dropped_oversized += 1
                 _diag.lost("event", 1, "AzureEventHubsSink, too large for an empty 1 MB batch")
         if len(current) > 0:
             attempted += 1
@@ -167,7 +168,8 @@ class AzureEventHubsSink:
                 if attempt < self.max_retries:
                     wait(_BACKOFF_BASE * (2**attempt), self.stop_signal)
                     continue
-                self.failed += len(event_batch)
+                with self._counter_lock:
+                    self.failed += len(event_batch)
                 _diag.lost(
                     "event",
                     len(event_batch),

{log_foundry-0.10.2.dev23 → log_foundry-0.10.2.dev24}/src/log_foundry/sinks/firehose.py

@@ -3,10 +3,8 @@
 from __future__ import annotations
 
 import json
-from typing import TYPE_CHECKING, Any
-
-if TYPE_CHECKING:
-    import threading
+import threading
+from typing import Any
 
 from log_foundry import _diag
 from log_foundry.sinks._batch import adjudicate_positional, usable_results
@@ -67,6 +65,7 @@ class FirehoseSink:
         self.failed = 0
         self.dropped_oversized = 0
         self.dropped_unadjudicated = 0
+        self._counter_lock = threading.Lock()
 
     def emit(self, batch: list[dict[str, object]]) -> None:
         """Re-chunks to the request limits and sends each chunk, retrying failures (FR-004).
@@ -126,10 +125,11 @@ class FirehoseSink:
         Raises:
           None.
         """
-        return SinkLosses(
-            dropped=self.dropped_oversized,
-            failed=self.failed + self.dropped_unadjudicated,
-        )
+        with self._counter_lock:
+            return SinkLosses(
+                dropped=self.dropped_oversized,
+                failed=self.failed + self.dropped_unadjudicated,
+            )
 
     def close(self) -> None:
         """Does nothing, since the sink buffers nothing internally (FR-001).
@@ -160,7 +160,8 @@ class FirehoseSink:
         for event in batch:
             data = json.dumps(event).encode("utf-8")
             if len(data) > self.MAX_RECORD_BYTES:
-                self.dropped_oversized += 1
+                with self._counter_lock:
+                    self.dropped_oversized += 1
                 _diag.lost(
                     "event",
                     1,
@@ -201,7 +202,8 @@ class FirehoseSink:
             results = usable_results(response.get("RequestResponses"))
             verdict = adjudicate_positional(records, results)
             if verdict.unadjudicated:
-                self.dropped_unadjudicated += verdict.unadjudicated
+                with self._counter_lock:
+                    self.dropped_unadjudicated += verdict.unadjudicated
                 _diag.lost(
                     "record",
                     verdict.unadjudicated,
@@ -216,7 +218,8 @@ class FirehoseSink:
             if attempt < self.max_retries:
                 wait(_BACKOFF_BASE * (2**attempt), self.stop_signal)
             if attempt >= self.max_retries:
-                self.failed += len(records)
+                with self._counter_lock:
+                    self.failed += len(records)
                 _diag.lost(
                     "record",
                     len(records),

{log_foundry-0.10.2.dev23 → log_foundry-0.10.2.dev24}/src/log_foundry/sinks/http.py

@@ -4,6 +4,7 @@ from __future__ import annotations
 
 import gzip as _gzip
 import json
+import threading
 import urllib.error
 import urllib.request
 from base64 import b64encode
@@ -14,7 +15,6 @@ from log_foundry.sinks._retry import clamp_server_delay, wait
 from log_foundry.sinks.base import SinkDeliveryError, SinkLosses
 
 if TYPE_CHECKING:
-    import threading
     from collections.abc import Callable
 
 __all__ = ["HTTPSink", "merge_headers"]
@@ -129,6 +129,7 @@ class HTTPSink:
         self._opener = opener if opener is not None else urllib.request.urlopen
         self.failed = 0
         self.dropped_oversized = 0
+        self._counter_lock = threading.Lock()
 
     def emit(self, batch: list[dict[str, object]]) -> None:
         """Serializes the batch per the configured body format and POSTs it (FR-001).
@@ -161,7 +162,8 @@ class HTTPSink:
         Raises:
           None.
         """
-        return SinkLosses(dropped=self.dropped_oversized, failed=self.failed)
+        with self._counter_lock:
+            return SinkLosses(dropped=self.dropped_oversized, failed=self.failed)
 
     def close(self) -> None:
         """Does nothing, since ``urllib`` opens a fresh connection per request (FR-012).
@@ -377,7 +379,8 @@ class HTTPSink:
         Raises:
           SinkDeliveryError: Always.
         """
-        self.failed += 1
+        with self._counter_lock:
+            self.failed += 1
         detail = f"{type(self).__name__}, {self.max_retries + 1} attempt(s), {reason}"
         _diag.lost("request", 1, detail)
         raise SinkDeliveryError(detail)

{log_foundry-0.10.2.dev23 → log_foundry-0.10.2.dev24}/src/log_foundry/sinks/kafka.py

@@ -3,6 +3,7 @@
 from __future__ import annotations
 
 import json
+import threading
 from typing import Any
 
 from log_foundry import _diag
@@ -61,6 +62,7 @@ class KafkaSink:
         self.key_field = key_field
         self.failed = 0
         self.rejected = 0
+        self._counter_lock = threading.Lock()
 
     def losses(self) -> SinkLosses:
         """Reports refused and undelivered messages (SPEC-026 FR-002).
@@ -82,7 +84,8 @@ class KafkaSink:
         Raises:
           None.
         """
-        return SinkLosses(dropped=self.rejected, failed=self.failed)
+        with self._counter_lock:
+            return SinkLosses(dropped=self.rejected, failed=self.failed)
 
     def emit(self, batch: list[dict[str, object]]) -> None:
         """Produces one message per event and serves delivery callbacks without blocking.
@@ -108,7 +111,8 @@ class KafkaSink:
             try:
                 self.producer.produce(self.topic, value=body, key=key, callback=self._on_delivery)
             except Exception as err:
-                self.rejected += 1
+                with self._counter_lock:
+                    self.rejected += 1
                 _diag.lost("message", 1, f"KafkaSink produce, {type(err).__name__}")
                 continue
             accepted += 1
@@ -167,7 +171,8 @@ class KafkaSink:
           None.
         """
         if err is not None:
-            self.failed += 1
+            with self._counter_lock:
+                self.failed += 1
             _diag.lost("message", 1, f"KafkaSink delivery, {type(err).__name__}{_code(err)}")
 
 

{log_foundry-0.10.2.dev23 → log_foundry-0.10.2.dev24}/src/log_foundry/sinks/kinesis.py

@@ -3,10 +3,8 @@
 from __future__ import annotations
 
 import json
-from typing import TYPE_CHECKING, Any
-
-if TYPE_CHECKING:
-    import threading
+import threading
+from typing import Any
 
 from log_foundry import _diag
 from log_foundry.sinks._batch import adjudicate_positional, usable_results
@@ -76,6 +74,7 @@ class KinesisSink:
         self.failed = 0
         self.dropped_oversized = 0
         self.dropped_unadjudicated = 0
+        self._counter_lock = threading.Lock()
 
     def emit(self, batch: list[dict[str, object]]) -> None:
         """Re-chunks to the request limits and sends each chunk, retrying failures (FR-003).
@@ -135,10 +134,11 @@ class KinesisSink:
         Raises:
           None.
         """
-        return SinkLosses(
-            dropped=self.dropped_oversized,
-            failed=self.failed + self.dropped_unadjudicated,
-        )
+        with self._counter_lock:
+            return SinkLosses(
+                dropped=self.dropped_oversized,
+                failed=self.failed + self.dropped_unadjudicated,
+            )
 
     def close(self) -> None:
         """Does nothing, since the sink buffers nothing internally (FR-001).
@@ -169,7 +169,8 @@ class KinesisSink:
         for event in batch:
             data = json.dumps(event).encode("utf-8")
             if len(data) > self.MAX_RECORD_BYTES:
-                self.dropped_oversized += 1
+                with self._counter_lock:
+                    self.dropped_oversized += 1
                 _diag.lost(
                     "event",
                     1,
@@ -209,7 +210,8 @@ class KinesisSink:
             results = usable_results(response.get("Records"))
             verdict = adjudicate_positional(records, results)
             if verdict.unadjudicated:
-                self.dropped_unadjudicated += verdict.unadjudicated
+                with self._counter_lock:
+                    self.dropped_unadjudicated += verdict.unadjudicated
                 _diag.lost(
                     "record",
                     verdict.unadjudicated,
@@ -223,7 +225,8 @@ class KinesisSink:
             if attempt < self.max_retries:
                 wait(_BACKOFF_BASE * (2**attempt), self.stop_signal)
             if attempt >= self.max_retries:
-                self.failed += len(records)
+                with self._counter_lock:
+                    self.failed += len(records)
                 _diag.lost(
                     "record",
                     len(records),