log-foundry 0.10.2.dev48__tar.gz → 0.10.2.dev49__tar.gz

{log_foundry-0.10.2.dev48 → log_foundry-0.10.2.dev49}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: log-foundry
-Version: 0.10.2.dev48
+Version: 0.10.2.dev49
 Summary: Generate logs for your console and JSON events for downstream consumption.
 License-Expression: MIT
 License-File: LICENSE
@@ -609,11 +609,40 @@ lf.configure(sink=RotatingFileSink("app.log.jsonl", max_bytes=10_000_000, backup
 
 All build on `HTTPSink` (stdlib `urllib`): they POST batches with bounded `429`/`5xx` retry
 (honoring `Retry-After`) and need **no** extra. On the specialized sinks, `**http_kwargs` forwards
-to `HTTPSink` (`headers=`, `auth=`, `gzip=`, `timeout=`, `max_retries=`).
+to `HTTPSink` (`headers=`, `auth=`, `gzip=`, `timeout=`, `max_retries=`, `max_retry_after=`,
+`max_batch_count=`, `max_batch_bytes=`, `opener=`).
+
+Each sink **re-chunks** a batch to its destination's limits, so one large `emit` — the whole
+pending backlog at process exit, for instance — becomes several requests the destination will
+accept rather than one it rejects whole. Every subclass sets its own defaults from its vendor's
+documentation where there is one; override with `max_batch_count=` / `max_batch_bytes=`. An event
+too large to be sent on its own is dropped and counted in `health().sink.dropped`.
+
+| Sink | `max_batch_count` | `max_batch_bytes` | Where the figure comes from |
+|---|---|---|---|
+| `DatadogSink` | 1,000 | 5,000,000 | both documented by Datadog for the logs intake |
+| `NewRelicSink` | 1,000 | 1,000,000 | the Log API's documented 1 MB (10⁶ bytes) per POST |
+| `HoneycombSink` | 1,000 | 1,000,000 | Honeycomb's documented 1 MB of uncompressed JSON |
+| `ElasticsearchSink` / `OpenSearchSink` | 1,000 | 10,000,000 | chosen below the documented 100 MB `http.max_content_length` |
+| `LokiSink` | 1,000 | 4,000,000 | chosen; Loki's own cap is an operator-tunable server setting |
+| `SplunkHECSink` | 1,000 | 1,000,000 | chosen; Splunk publishes no fixed HEC payload limit |
+| `LogstashSink` (HTTP mode) | 1,000 | 5,000,000 | inherits the generic defaults |
+| `HTTPSink` (generic) | 1,000 | 5,000,000 | chosen, for an endpoint the library knows nothing about |
+
+A count of 1,000 is Datadog's published array limit and a conservative default elsewhere — no
+destination in this family documents a smaller one. `DatadogSink` additionally enforces a
+1,000,000-byte limit on a *single* log, which is the one case where a destination's per-event cap
+is stricter than its per-request one.
+
+Two consequences worth knowing. An event too large to be sent on its own is **dropped**, not
+attempted — including in `LogstashSink`'s HTTP mode, which previously put any size on the wire.
+And because a request now carries one chunk rather than the whole batch, a partial failure is
+reported through `health().sink` rather than raised: `flush()` returning `True` means the drain
+completed, not that every chunk of it landed.
 
 | Sink | Import from | Configure |
 |---|---|---|
-| `HTTPSink` | `log_foundry.sinks.http` | `HTTPSink(url, *, method="POST", headers=None, auth=None, body_format="ndjson", timeout=5.0, gzip=False, max_retries=3)` — generic POST. `auth` is a bearer-token `str` or `(user, pass)` for basic; `body_format` is `"ndjson"` or `"json_array"` |
+| `HTTPSink` | `log_foundry.sinks.http` | `HTTPSink(url, *, method="POST", headers=None, auth=None, body_format="ndjson", timeout=5.0, gzip=False, max_retries=3, max_retry_after=30.0, max_batch_count=None, max_batch_bytes=None, opener=None)` — generic POST. `auth` is a bearer-token `str` or `(user, pass)` for basic; `body_format` is `"ndjson"` or `"json_array"`; the two `max_batch_*` default to `None`, meaning "use this class's limits" from the table below (1,000 / 5,000,000 here); `opener` injects a `urlopen`-shaped callable for tests |
 | `ElasticsearchSink` | `log_foundry.sinks.elasticsearch` | `ElasticsearchSink(url, *, index, auth=None, **http_kwargs)` — POST to `_bulk`, parsing per-item errors (`.item_errors`) |
 | `OpenSearchSink` | `log_foundry.sinks.elasticsearch` | same signature as `ElasticsearchSink` (identical bulk protocol) |
 | `LokiSink` | `log_foundry.sinks.loki` | `LokiSink(url, *, labels=("service", "env", "level"), **http_kwargs)` — Grafana Loki push API |

{log_foundry-0.10.2.dev48 → log_foundry-0.10.2.dev49}/README.md

@@ -573,11 +573,40 @@ lf.configure(sink=RotatingFileSink("app.log.jsonl", max_bytes=10_000_000, backup
 
 All build on `HTTPSink` (stdlib `urllib`): they POST batches with bounded `429`/`5xx` retry
 (honoring `Retry-After`) and need **no** extra. On the specialized sinks, `**http_kwargs` forwards
-to `HTTPSink` (`headers=`, `auth=`, `gzip=`, `timeout=`, `max_retries=`).
+to `HTTPSink` (`headers=`, `auth=`, `gzip=`, `timeout=`, `max_retries=`, `max_retry_after=`,
+`max_batch_count=`, `max_batch_bytes=`, `opener=`).
+
+Each sink **re-chunks** a batch to its destination's limits, so one large `emit` — the whole
+pending backlog at process exit, for instance — becomes several requests the destination will
+accept rather than one it rejects whole. Every subclass sets its own defaults from its vendor's
+documentation where there is one; override with `max_batch_count=` / `max_batch_bytes=`. An event
+too large to be sent on its own is dropped and counted in `health().sink.dropped`.
+
+| Sink | `max_batch_count` | `max_batch_bytes` | Where the figure comes from |
+|---|---|---|---|
+| `DatadogSink` | 1,000 | 5,000,000 | both documented by Datadog for the logs intake |
+| `NewRelicSink` | 1,000 | 1,000,000 | the Log API's documented 1 MB (10⁶ bytes) per POST |
+| `HoneycombSink` | 1,000 | 1,000,000 | Honeycomb's documented 1 MB of uncompressed JSON |
+| `ElasticsearchSink` / `OpenSearchSink` | 1,000 | 10,000,000 | chosen below the documented 100 MB `http.max_content_length` |
+| `LokiSink` | 1,000 | 4,000,000 | chosen; Loki's own cap is an operator-tunable server setting |
+| `SplunkHECSink` | 1,000 | 1,000,000 | chosen; Splunk publishes no fixed HEC payload limit |
+| `LogstashSink` (HTTP mode) | 1,000 | 5,000,000 | inherits the generic defaults |
+| `HTTPSink` (generic) | 1,000 | 5,000,000 | chosen, for an endpoint the library knows nothing about |
+
+A count of 1,000 is Datadog's published array limit and a conservative default elsewhere — no
+destination in this family documents a smaller one. `DatadogSink` additionally enforces a
+1,000,000-byte limit on a *single* log, which is the one case where a destination's per-event cap
+is stricter than its per-request one.
+
+Two consequences worth knowing. An event too large to be sent on its own is **dropped**, not
+attempted — including in `LogstashSink`'s HTTP mode, which previously put any size on the wire.
+And because a request now carries one chunk rather than the whole batch, a partial failure is
+reported through `health().sink` rather than raised: `flush()` returning `True` means the drain
+completed, not that every chunk of it landed.
 
 | Sink | Import from | Configure |
 |---|---|---|
-| `HTTPSink` | `log_foundry.sinks.http` | `HTTPSink(url, *, method="POST", headers=None, auth=None, body_format="ndjson", timeout=5.0, gzip=False, max_retries=3)` — generic POST. `auth` is a bearer-token `str` or `(user, pass)` for basic; `body_format` is `"ndjson"` or `"json_array"` |
+| `HTTPSink` | `log_foundry.sinks.http` | `HTTPSink(url, *, method="POST", headers=None, auth=None, body_format="ndjson", timeout=5.0, gzip=False, max_retries=3, max_retry_after=30.0, max_batch_count=None, max_batch_bytes=None, opener=None)` — generic POST. `auth` is a bearer-token `str` or `(user, pass)` for basic; `body_format` is `"ndjson"` or `"json_array"`; the two `max_batch_*` default to `None`, meaning "use this class's limits" from the table below (1,000 / 5,000,000 here); `opener` injects a `urlopen`-shaped callable for tests |
 | `ElasticsearchSink` | `log_foundry.sinks.elasticsearch` | `ElasticsearchSink(url, *, index, auth=None, **http_kwargs)` — POST to `_bulk`, parsing per-item errors (`.item_errors`) |
 | `OpenSearchSink` | `log_foundry.sinks.elasticsearch` | same signature as `ElasticsearchSink` (identical bulk protocol) |
 | `LokiSink` | `log_foundry.sinks.loki` | `LokiSink(url, *, labels=("service", "env", "level"), **http_kwargs)` — Grafana Loki push API |

{log_foundry-0.10.2.dev48 → log_foundry-0.10.2.dev49}/pyproject.toml

@@ -20,7 +20,7 @@ dependencies = [
 ]
 
 # Optional features. Install with: pip install log-foundry[aws]
-version = "0.10.2.dev48"
+version = "0.10.2.dev49"
 
 [project.optional-dependencies]
 aws = ["boto3>=1.43.61"]              # SQSSink, SNSSink, KinesisSink, FirehoseSink

{log_foundry-0.10.2.dev48 → log_foundry-0.10.2.dev49}/src/log_foundry/sinks/datadog.py

@@ -20,8 +20,20 @@ class DatadogSink(HTTPSink):
     It takes **no** transport lock (SPEC-028 FR-002) and **adds no post-close guard**
     (SPEC-032 FR-003), for the reasons :class:`~log_foundry.sinks.http.HTTPSink` records: there
     is no transport held and ``close()`` releases nothing.
+
+    Attributes:
+      MAX_BATCH_COUNT: 1,000 — Datadog's documented maximum array size for the logs intake.
+      MAX_BATCH_BYTES: 5,000,000 — its documented maximum uncompressed payload.
+      MAX_EVENT_BYTES: 1,000,000 — its documented maximum for a *single* log. This is the one
+        sink in the family whose per-event limit is stricter than its request limit, so without
+        it a 2 MB event passes the 5 MB request budget and is rejected by a limit the budget
+        cannot see. All three are the vendor's own figures, from the Logs API's send-logs limits.
     """
 
+    MAX_BATCH_COUNT = 1000
+    MAX_EVENT_BYTES = 1_000_000
+    MAX_BATCH_BYTES = 5_000_000
+
     def __init__(
         self,
         api_key: str,
@@ -54,22 +66,19 @@ class DatadogSink(HTTPSink):
             headers=headers, body_format="json_array", **http_kwargs,  # type: ignore[arg-type]
         )
 
-    def emit(self, batch: list[dict[str, object]]) -> None:
-        """POSTs each enriched event as one JSON array (FR-007).
+    def _render(self, event: dict[str, object]) -> str:
+        """Serializes one enriched entry for the JSON array (FR-007).
 
         Args:
-          batch: The events to ship. An empty batch is a no-op.
+          event: The event to enrich and serialize.
 
         Returns:
-          None.
+          The serialized entry.
 
         Raises:
-          SinkDeliveryError: If the request was abandoned past the retry bound.
+          TypeError: If the event is not JSON-serializable, which ``sanitize`` prevents.
         """
-        if not batch:
-            return
-        body = json.dumps([self._entry(event) for event in batch]).encode("utf-8")
-        self._send(body, content_type="application/json")
+        return json.dumps(self._entry(event))
 
     def _entry(self, event: dict[str, object]) -> dict[str, object]:
         """Copies an event and applies the configured Datadog enrichment.

{log_foundry-0.10.2.dev48 → log_foundry-0.10.2.dev49}/src/log_foundry/sinks/elasticsearch.py

@@ -6,8 +6,8 @@ import json
 
 from log_foundry import _diag
 from log_foundry.sinks._batch import usable_results
-from log_foundry.sinks.base import SinkDeliveryError, SinkLosses
-from log_foundry.sinks.http import HTTPSink
+from log_foundry.sinks.base import SinkLosses
+from log_foundry.sinks.http import HTTPSink, _Item
 
 __all__ = ["ElasticsearchSink", "OpenSearchSink"]
 
@@ -29,8 +29,20 @@ class ElasticsearchSink(HTTPSink):
     It takes **no** transport lock (SPEC-028 FR-002) and **adds no post-close guard**
     (SPEC-032 FR-003), for the reasons :class:`~log_foundry.sinks.http.HTTPSink` records: there
     is no transport held and ``close()`` releases nothing.
+
+    Attributes:
+      MAX_BATCH_COUNT: 1,000 — this library's conservative default; Elasticsearch bounds a bulk
+        request by size, not by document count.
+      MAX_BATCH_BYTES: 10,000,000 — chosen *below* the documented hard limit rather than at it.
+        Elasticsearch "limits the maximum size of a HTTP request to 100mb by default", but its
+        bulk guidance is to find a working size by experiment rather than to send the largest
+        request the server will accept, and a 100 MB bulk is a poor default for a log shipper.
+        Raise it with ``max_batch_bytes=``.
     """
 
+    MAX_BATCH_COUNT = 1000
+    MAX_BATCH_BYTES = 10_000_000
+
     def __init__(self, url: str, *, index: str, auth: str | tuple[str, str] | None = None,
                  **http_kwargs: object) -> None:
         """Points the sink at a cluster's ``_bulk`` endpoint.
@@ -54,38 +66,44 @@ class ElasticsearchSink(HTTPSink):
         self.item_errors = 0
         self.dropped_unadjudicated = 0
 
-    def emit(self, batch: list[dict[str, object]]) -> None:
-        """Builds the ``_bulk`` NDJSON payload, POSTs it, and parses the response items.
+    def _render(self, event: dict[str, object]) -> str:
+        """Serializes one event as its ``_bulk`` action line and source line.
 
-        An abandoned request raises out of ``_send`` (SPEC-026 FR-001): nothing was indexed, so
-        there is no response to parse and nothing downstream to duplicate.
+        Both lines are one item so the pair can never be split across two requests, and so the
+        chunker's byte budget charges the action line the wire actually carries.
 
         Args:
-          batch: The events to index. An empty batch is a no-op.
+          event: The event to index.
 
         Returns:
-          None.
+          The two newline-joined lines, without the trailing newline :meth:`_body` adds.
 
         Raises:
-          SinkDeliveryError: If the request was abandoned, or if every item carried an error so
-            the ``200`` indexed nothing — a total failure like any other, which must reach the
-            worker (FR-001). A retry cannot duplicate, and where the cause is permanent the
-            worker abandons the batch after its bound and records it, which beats a silent
-            success. A response rejecting some items stays partial and is reported through
-            :meth:`losses`.
+          TypeError: If the event is not JSON-serializable, which ``sanitize`` prevents.
         """
-        if not batch:
-            return
-        lines: list[str] = []
-        for event in batch:
-            lines.append(json.dumps({"index": {"_index": self._index}}))
-            lines.append(json.dumps(event))
-        body = ("\n".join(lines) + "\n").encode("utf-8")
-        payload = self._send(body, content_type="application/x-ndjson")
-        if self._parse_bulk_response(payload, len(batch)):
-            raise SinkDeliveryError(
-                f"{type(self).__name__} indexed none of {len(batch)} event(s)"
-            )
+        action = json.dumps({"index": {"_index": self._index}})
+        return f"{action}\n{json.dumps(event)}"
+
+    def _handle_response(self, payload: bytes, items: list[_Item]) -> bool:
+        """Parses the chunk's ``_bulk`` items and reports whether anything was indexed.
+
+        Returning ``False`` rather than raising is what lets a sibling chunk that succeeded
+        stand: :meth:`~log_foundry.sinks.http.HTTPSink.emit` raises only when *every* chunk came
+        back empty-handed, which is the total failure the worker's retry exists for, while a
+        batch that indexed some of its chunks must not be re-sent whole (SPEC-026 FR-001).
+
+        Args:
+          payload: The response body.
+          items: The chunk the response describes.
+
+        Returns:
+          False when every item in the chunk carried an error, True otherwise — including for a
+          body this sink cannot read, since the request itself succeeded.
+
+        Raises:
+          None.
+        """
+        return not self._parse_bulk_response(payload, len(items))
 
     def losses(self) -> SinkLosses:
         """Reports abandoned requests plus server-rejected bulk items (SPEC-026 FR-002).

{log_foundry-0.10.2.dev48 → log_foundry-0.10.2.dev49}/src/log_foundry/sinks/honeycomb.py

@@ -18,8 +18,17 @@ class HoneycombSink(HTTPSink):
     It takes **no** transport lock (SPEC-028 FR-002) and **adds no post-close guard**
     (SPEC-032 FR-003), for the reasons :class:`~log_foundry.sinks.http.HTTPSink` records: there
     is no transport held and ``close()`` releases nothing.
+
+    Attributes:
+      MAX_BATCH_COUNT: 1,000 — this library's conservative default, since Honeycomb documents no
+        maximum event count for the batch endpoint.
+      MAX_BATCH_BYTES: 1,000,000 — Honeycomb's documented 1 MB of uncompressed JSON for the
+        Create Events endpoint.
     """
 
+    MAX_BATCH_COUNT = 1000
+    MAX_BATCH_BYTES = 1_000_000
+
     def __init__(
         self,
         api_key: str,
@@ -48,19 +57,16 @@ class HoneycombSink(HTTPSink):
             headers=headers, body_format="json_array", **http_kwargs,  # type: ignore[arg-type]
         )
 
-    def emit(self, batch: list[dict[str, object]]) -> None:
-        """POSTs the batch in Honeycomb's ``[{"data": event}, ...]`` shape (FR-010).
+    def _render(self, event: dict[str, object]) -> str:
+        """Serializes one event in Honeycomb's ``{"data": event}`` batch shape (FR-010).
 
         Args:
-          batch: The events to ship. An empty batch is a no-op.
+          event: The event to wrap and serialize.
 
         Returns:
-          None.
+          The serialized entry.
 
         Raises:
-          SinkDeliveryError: If the request was abandoned past the retry bound.
+          TypeError: If the event is not JSON-serializable, which ``sanitize`` prevents.
         """
-        if not batch:
-            return
-        body = json.dumps([{"data": event} for event in batch]).encode("utf-8")
-        self._send(body, content_type="application/json")
+        return json.dumps({"data": event})