log-foundry 0.4.1.dev3__tar.gz → 0.4.1.dev4__tar.gz

{log_foundry-0.4.1.dev3 → log_foundry-0.4.1.dev4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: log-foundry
-Version: 0.4.1.dev3
+Version: 0.4.1.dev4
 Summary: Generate logs for your console and JSON events for downstream consumption.
 License-Expression: MIT
 License-File: LICENSE
@@ -512,7 +512,7 @@ limits, retries partial failures, and drops any single event too large to ever f
 
 | Sink | Import from | Configure |
 |---|---|---|
-| `SQSSink` | `log_foundry.sinks.sqs` | `SQSSink(queue_url, *, max_retries=3)` — the headline production path: a durable buffer in front of ELK, absorbing downstream spikes/outages |
+| `SQSSink` | `log_foundry.sinks.sqs` | `SQSSink(queue_url, *, max_retries=3, fifo=None, message_group_id=None, message_deduplication_id=None)` — the headline production path: a durable buffer in front of ELK, absorbing downstream spikes/outages. Standard **and** FIFO queues |
 | `SNSSink` | `log_foundry.sinks.sns` | `SNSSink(topic_arn, *, max_retries=3)` |
 | `KinesisSink` | `log_foundry.sinks.kinesis` | `KinesisSink(stream_name, *, partition_key_field="trace_id", max_retries=3)` |
 | `FirehoseSink` | `log_foundry.sinks.firehose` | `FirehoseSink(delivery_stream, *, max_retries=3)` |
@@ -525,6 +525,51 @@ lf.configure(service="payments",
 
 Consuming from the buffer and indexing into ELK is a separate component, outside this library.
 
+`SQSSink` does not retry a message SQS rejects as a **sender fault** — the retry would re-send it
+byte-identical, so it can only fail the same way. Those are counted on `.failed` immediately and
+the SQS error code is named on stderr. Throttles and internal errors are still retried up to
+`max_retries`.
+
+##### FIFO queues
+
+A queue URL ending in `.fifo` switches `SQSSink` into FIFO mode automatically — AWS requires the
+suffix on every FIFO queue, so nothing needs configuring:
+
+```python
+SQSSink(queue_url="https://sqs.us-east-1.amazonaws.com/123456789012/logs.fifo")
+```
+
+Each message then carries a **`MessageGroupId`**, which defaults to the event's own `trace_id`.
+SQS guarantees ordering *within* a group, and a trace is exactly the unit whose events should stay
+ordered — while separate traces land in separate groups, so the queue delivers them in parallel
+instead of serializing your whole process behind one group. (`KinesisSink` partitions on `trace_id`
+by default for the same reason.) The **`MessageDeduplicationId`** defaults to the event's `log_id`,
+already a per-event UUID, so SQS's five-minute deduplication window never collapses two distinct
+records.
+
+Override the group with a constant or a callable:
+
+```python
+# One group for the whole process — strict global ordering, capped at ~300 msg/s.
+SQSSink(queue_url=FIFO_URL, message_group_id="payments")
+
+# Group by anything on the event. Baggage lands in `fields`, so this groups by tenant
+# and falls back to per-trace when unset:
+SQSSink(queue_url=FIFO_URL,
+        message_group_id=lambda e: str(e["fields"].get("tenant_id") or e["trace_id"]))
+```
+
+Pass `fifo=True` or `fifo=False` to override the URL-based detection. Standard queues are entirely
+unaffected — their messages carry neither parameter.
+
+Two things worth knowing:
+
+- **Ordering is best-effort across a retry.** If one message fails and a same-group message ahead
+  of it succeeded, the retry lands after it. Holding a whole group back on a single failure would
+  trade log delivery for ordering you can rebuild from `timestamp`, so the sink doesn't.
+- **FIFO queues cap throughput** at 300 messages/second (3,000 with batching), or higher in
+  high-throughput mode. That's queue-side configuration, not something the library sets.
+
 #### Queue & stream
 
 Each needs its own extra (lazy-imported). All publish + retry within a bound and close cleanly.

{log_foundry-0.4.1.dev3 → log_foundry-0.4.1.dev4}/README.md

@@ -476,7 +476,7 @@ limits, retries partial failures, and drops any single event too large to ever f
 
 | Sink | Import from | Configure |
 |---|---|---|
-| `SQSSink` | `log_foundry.sinks.sqs` | `SQSSink(queue_url, *, max_retries=3)` — the headline production path: a durable buffer in front of ELK, absorbing downstream spikes/outages |
+| `SQSSink` | `log_foundry.sinks.sqs` | `SQSSink(queue_url, *, max_retries=3, fifo=None, message_group_id=None, message_deduplication_id=None)` — the headline production path: a durable buffer in front of ELK, absorbing downstream spikes/outages. Standard **and** FIFO queues |
 | `SNSSink` | `log_foundry.sinks.sns` | `SNSSink(topic_arn, *, max_retries=3)` |
 | `KinesisSink` | `log_foundry.sinks.kinesis` | `KinesisSink(stream_name, *, partition_key_field="trace_id", max_retries=3)` |
 | `FirehoseSink` | `log_foundry.sinks.firehose` | `FirehoseSink(delivery_stream, *, max_retries=3)` |
@@ -489,6 +489,51 @@ lf.configure(service="payments",
 
 Consuming from the buffer and indexing into ELK is a separate component, outside this library.
 
+`SQSSink` does not retry a message SQS rejects as a **sender fault** — the retry would re-send it
+byte-identical, so it can only fail the same way. Those are counted on `.failed` immediately and
+the SQS error code is named on stderr. Throttles and internal errors are still retried up to
+`max_retries`.
+
+##### FIFO queues
+
+A queue URL ending in `.fifo` switches `SQSSink` into FIFO mode automatically — AWS requires the
+suffix on every FIFO queue, so nothing needs configuring:
+
+```python
+SQSSink(queue_url="https://sqs.us-east-1.amazonaws.com/123456789012/logs.fifo")
+```
+
+Each message then carries a **`MessageGroupId`**, which defaults to the event's own `trace_id`.
+SQS guarantees ordering *within* a group, and a trace is exactly the unit whose events should stay
+ordered — while separate traces land in separate groups, so the queue delivers them in parallel
+instead of serializing your whole process behind one group. (`KinesisSink` partitions on `trace_id`
+by default for the same reason.) The **`MessageDeduplicationId`** defaults to the event's `log_id`,
+already a per-event UUID, so SQS's five-minute deduplication window never collapses two distinct
+records.
+
+Override the group with a constant or a callable:
+
+```python
+# One group for the whole process — strict global ordering, capped at ~300 msg/s.
+SQSSink(queue_url=FIFO_URL, message_group_id="payments")
+
+# Group by anything on the event. Baggage lands in `fields`, so this groups by tenant
+# and falls back to per-trace when unset:
+SQSSink(queue_url=FIFO_URL,
+        message_group_id=lambda e: str(e["fields"].get("tenant_id") or e["trace_id"]))
+```
+
+Pass `fifo=True` or `fifo=False` to override the URL-based detection. Standard queues are entirely
+unaffected — their messages carry neither parameter.
+
+Two things worth knowing:
+
+- **Ordering is best-effort across a retry.** If one message fails and a same-group message ahead
+  of it succeeded, the retry lands after it. Holding a whole group back on a single failure would
+  trade log delivery for ordering you can rebuild from `timestamp`, so the sink doesn't.
+- **FIFO queues cap throughput** at 300 messages/second (3,000 with batching), or higher in
+  high-throughput mode. That's queue-side configuration, not something the library sets.
+
 #### Queue & stream
 
 Each needs its own extra (lazy-imported). All publish + retry within a bound and close cleanly.

{log_foundry-0.4.1.dev3 → log_foundry-0.4.1.dev4}/pyproject.toml

@@ -20,7 +20,7 @@ dependencies = [
 ]
 
 # Optional features. Install with: pip install log-foundry[aws]
-version = "0.4.1.dev3"
+version = "0.4.1.dev4"
 
 [project.optional-dependencies]
 aws = ["boto3>=1.34"]              # SQSSink, SNSSink, KinesisSink, FirehoseSink

{log_foundry-0.4.1.dev3 → log_foundry-0.4.1.dev4}/src/log_foundry/sinks/sqs.py

@@ -194,6 +194,11 @@ class SQSSink:
         are counted (``failed``) and logged, not silently dropped (FR-003). The FIFO parameters
         are attached only when the queue is FIFO, so a standard queue's entries are exactly
         ``Id`` + ``MessageBody`` (SPEC-016 FR-004).
+
+        Entries SQS marks ``SenderFault`` are abandoned rather than retried (SPEC-016 FR-006):
+        a retry re-sends them byte-identical, so a fault in the request itself can only fail
+        the same way. Throttles and internal errors carry ``SenderFault: false`` and are still
+        retried under the bound.
         """
         entries: list[dict[str, str]] = []
         for i, item in enumerate(prepared):
@@ -208,8 +213,22 @@ class SQSSink:
             failed = response.get("Failed", [])
             if not failed:
                 return
-            failed_ids = {entry["Id"] for entry in failed}
-            entries = [entry for entry in entries if entry["Id"] in failed_ids]
+            # Abandon sender faults immediately and name the code: the retry would re-send the
+            # entry byte-identical, and the code is the only thing that makes a rejection
+            # diagnosable from the log line alone. A missing flag is treated as retryable, so
+            # an unfamiliar response shape degrades to the old behaviour rather than dropping.
+            sender_faults = [item for item in failed if item.get("SenderFault")]
+            if sender_faults:
+                self.failed += len(sender_faults)
+                sys.stderr.write(
+                    f"log-foundry: {len(sender_faults)} SQS message(s) rejected as invalid "
+                    f"(first code: {sender_faults[0].get('Code', 'unknown')}); not retried\n"
+                )
+
+            retryable_ids = {item["Id"] for item in failed if not item.get("SenderFault")}
+            if not retryable_ids:
+                return
+            entries = [entry for entry in entries if entry["Id"] in retryable_ids]
             if attempt >= self.max_retries:
                 self.failed += len(entries)
                 sys.stderr.write(