omdnotificationforwarder 4.5__tar.gz → 4.6__tar.gz

{omdnotificationforwarder-4.5 → omdnotificationforwarder-4.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: omdnotificationforwarder
-Version: 4.5
+Version: 4.6
 Summary: A framework for notification scripts for OMD
 Project-URL: Homepage, https://github.com/lausser/noteventificationforhandlerwarder
 Project-URL: Bug Tracker, https://github.com/lausser/noteventificationforhandlerwarder/issues
@@ -521,10 +521,23 @@ define command{
 
 When forwarding to a recipient fails, the event is saved in a local sqlite database. The framework automatically attempts to re-submit these spooled events on every subsequent call to notificationforwarder for the same forwarder.
 
-The flush mechanism runs inside `forward_formatted()` for **every event** — not just heartbeats or special cases. Before submitting the current event, the framework checks if there are spooled events waiting. If there are, it attempts to flush them by calling `submit()` for each one. Events that are successfully submitted are deleted from the spool. Events that still fail stay in the queue for the next attempt.
+The flush mechanism runs inside `forward_formatted()` for **every event** — not just heartbeats or special cases. Before submitting the current event, the framework checks if there are spooled events waiting. If there are, it attempts to flush them by calling `submit()` for each one, oldest first. Events that are successfully submitted are deleted from the spool. The first event that fails again ends the flush: it and all later events stay in the queue for the next attempt. A receiver that does not accept one event will most likely not accept the next ones either, so a flush costs at most one submit timeout instead of one per spooled event.
+
+If the receiver rejects a spooled event for good (for example because of an invalid payload), that event must not block the queue until `max_spool_minutes` have passed. When a flush stopped at an event and the current event is then accepted by the receiver, the receiver is evidently up. The stopped event is moved to the end of the queue and logged as "spooled event moved to the end of the spool". It keeps its original time, so it is still dropped when `max_spool_minutes` have passed, and the next flush can deliver the events behind it.
 
 This means that in a busy monitoring environment where notifications flow regularly, spooled events get drained naturally — each new notification triggers a flush attempt on its way through.
 
+### Delivery Order
+
+Spooled events are stored and re-submitted in the order in which they failed. As long as every delivery succeeds, the receiver gets the events in the order in which they were created. After delivery failures, this is not guaranteed. A newer event can reach the receiver before an older one, for example a RECOVERY before its PROBLEM:
+
+- While one notificationforwarder process is flushing the spool, another process submits its own new event directly, ahead of the spooled ones.
+- A spooled event was rejected while the receiver accepted the current event. The rejected event is moved to the end of the spool and is delivered, if at all, after newer events.
+- The flush stopped early, or `probe()` returned False. The process still submits its own new event directly.
+- A submit ran into the timeout, but the receiver got the event anyway. The event stays in the spool, and the receiver gets it a second time later, after newer events.
+
+If the receiver depends on the order, let the formatter include the event's creation time (`omd_originating_timestamp`, set for every event) in the payload. The receiver or its integration can then ignore an event that is older than the state it already has.
+
 ### Probe Method
 
 Forwarders can implement an optional `probe()` method to optimize the flush behavior. The `probe()` method checks whether the receiving system is reachable *before* the framework attempts to flush spooled events via `submit()`.

{omdnotificationforwarder-4.5 → omdnotificationforwarder-4.6}/README.md

@@ -500,10 +500,23 @@ define command{
 
 When forwarding to a recipient fails, the event is saved in a local sqlite database. The framework automatically attempts to re-submit these spooled events on every subsequent call to notificationforwarder for the same forwarder.
 
-The flush mechanism runs inside `forward_formatted()` for **every event** — not just heartbeats or special cases. Before submitting the current event, the framework checks if there are spooled events waiting. If there are, it attempts to flush them by calling `submit()` for each one. Events that are successfully submitted are deleted from the spool. Events that still fail stay in the queue for the next attempt.
+The flush mechanism runs inside `forward_formatted()` for **every event** — not just heartbeats or special cases. Before submitting the current event, the framework checks if there are spooled events waiting. If there are, it attempts to flush them by calling `submit()` for each one, oldest first. Events that are successfully submitted are deleted from the spool. The first event that fails again ends the flush: it and all later events stay in the queue for the next attempt. A receiver that does not accept one event will most likely not accept the next ones either, so a flush costs at most one submit timeout instead of one per spooled event.
+
+If the receiver rejects a spooled event for good (for example because of an invalid payload), that event must not block the queue until `max_spool_minutes` have passed. When a flush stopped at an event and the current event is then accepted by the receiver, the receiver is evidently up. The stopped event is moved to the end of the queue and logged as "spooled event moved to the end of the spool". It keeps its original time, so it is still dropped when `max_spool_minutes` have passed, and the next flush can deliver the events behind it.
 
 This means that in a busy monitoring environment where notifications flow regularly, spooled events get drained naturally — each new notification triggers a flush attempt on its way through.
 
+### Delivery Order
+
+Spooled events are stored and re-submitted in the order in which they failed. As long as every delivery succeeds, the receiver gets the events in the order in which they were created. After delivery failures, this is not guaranteed. A newer event can reach the receiver before an older one, for example a RECOVERY before its PROBLEM:
+
+- While one notificationforwarder process is flushing the spool, another process submits its own new event directly, ahead of the spooled ones.
+- A spooled event was rejected while the receiver accepted the current event. The rejected event is moved to the end of the spool and is delivered, if at all, after newer events.
+- The flush stopped early, or `probe()` returned False. The process still submits its own new event directly.
+- A submit ran into the timeout, but the receiver got the event anyway. The event stays in the spool, and the receiver gets it a second time later, after newer events.
+
+If the receiver depends on the order, let the formatter include the event's creation time (`omd_originating_timestamp`, set for every event) in the payload. The receiver or its integration can then ignore an event that is older than the state it already has.
+
 ### Probe Method
 
 Forwarders can implement an optional `probe()` method to optimize the flush behavior. The `probe()` method checks whether the receiving system is reachable *before* the framework attempts to flush spooled events via `submit()`.

{omdnotificationforwarder-4.5 → omdnotificationforwarder-4.6}/bin/notificationforwarder

@@ -78,7 +78,7 @@ Example for an HTTP-based reporter:
                       help='Logger type: text (default) or json, or custom logger module',
                       default='text')
     parser.add_argument('--version', action='version',
-                      version=f'%(prog)s 4.5')
+                      version=f'%(prog)s 4.6')
 
     args = parser.parse_args()
     if not hasattr(args, 'formatter'):

{omdnotificationforwarder-4.5 → omdnotificationforwarder-4.6}/pyproject.toml

@@ -21,7 +21,7 @@ packages = ["src/notificationforwarder"]
 
 [project]
 name = "omdnotificationforwarder"
-version = "4.5"
+version = "4.6"
 authors = [
   { name="Gerhard Lausser", email="lausser@yahoo.com" },
 ]

{omdnotificationforwarder-4.5 → omdnotificationforwarder-4.6}/src/notificationforwarder/baseclass.py

@@ -330,6 +330,7 @@ class NotificationForwarder(object):
         return enrich_raw_event(raw_event)
 
     def forward_formatted(self, formatted_event):
+        self.flush_stopped_at = None
         try:
             """probe() checks if a forwarder is principally capable to submit
             an event. It is mostly used to contact an api and confirm that
@@ -376,6 +377,11 @@ class NotificationForwarder(object):
                     'formatted_event': formatted_event,
                     'status': 'success'
                 })
+            if formatted_event is not None and self.flush_stopped_at is not None:
+                # the receiver accepted this event, so it is up. the spooled
+                # event at which the flush stopped was rejected for its own
+                # reasons and must not block the spool until it expires.
+                self.move_stopped_event_to_end()
             return success
         return False
 
@@ -411,7 +417,38 @@ class NotificationForwarder(object):
     def acquire_lock_with_retry(self, lock_file, max_attempts=3, base_delay=0.1):
         return acquire_lock_with_retry(lock_file, logger, max_attempts=max_attempts, base_delay=base_delay)
 
+    def move_stopped_event_to_end(self):
+        event_id = self.flush_stopped_at
+        self.flush_stopped_at = None
+        try:
+            with open(self.db_lock_file, "w") as lock_file:
+                # a running flush could hold the row in memory, so move it
+                # only while no other process is flushing
+                if not self.acquire_lock_with_retry(lock_file, max_attempts=1):
+                    logger.debug("missed the flush lock, rejected event stays in place", {
+                        'event_id': event_id,
+                    })
+                    return
+                try:
+                    new_event_id = self.spool_store.move_to_end(event_id)
+                finally:
+                    import fcntl
+                    fcntl.lockf(lock_file, fcntl.LOCK_UN)
+            if new_event_id is not None:
+                logger.critical("spooled event moved to the end of the spool", {
+                    'event_id': event_id,
+                    'new_event_id': new_event_id,
+                    'action': 'moved_to_end',
+                })
+        except Exception as e:
+            logger.critical("could not move rejected spooled event", {
+                'event_id': event_id,
+                'exception': e,
+            })
+
     def flush(self):
+        # id of the spooled event at which the flush stopped, see move_stopped_event_to_end()
+        self.flush_stopped_at = None
         with open(self.db_lock_file, "w") as lock_file:
             locked = self.acquire_lock_with_retry(lock_file)
             if locked:
@@ -429,8 +466,10 @@ class NotificationForwarder(object):
                         })
                     # walk through the rows which existed when the flush started.
                     # rows spooled meanwhile by concurrent processes are left for the
-                    # next flush and each row is tried at most once, so the flush lock
-                    # is not held forever while the destination keeps failing.
+                    # next flush and each row is tried at most once. the first failed
+                    # submit ends the flush, because a receiver which does not accept
+                    # one event will most likely not accept the next ones either and
+                    # every further attempt would cost another submit timeout.
                     last_id = 0
                     snapshot_max_id = self.spool_store.max_id()
                     while True:
@@ -442,7 +481,6 @@ class NotificationForwarder(object):
                             'spooled_count': len(id_events),
                             'action': 'resend'
                         })
-                        progress = False
                         for id, text in id_events:
                             last_id = id
                             raw_event = self.spool_store.decode(text)
@@ -454,7 +492,6 @@ class NotificationForwarder(object):
                                     'action': 'discard_during_replay'
                                 })
                                 self.spool_store.delete(id)
-                                progress = True
                             elif formatted_event:
                                 replay_attempted += 1
                                 try:
@@ -465,7 +502,6 @@ class NotificationForwarder(object):
                                     error_message = str(e)
                                 if success:
                                     self.spool_store.delete(id)
-                                    progress = True
                                     replayed += 1
                                     logger.info("delete spooled event", {
                                         'spooled_count': 1,
@@ -481,6 +517,8 @@ class NotificationForwarder(object):
                                     if error_message:
                                         context['exception'] = error_message
                                     logger.critical("event stays in spool", context)
+                                    self.flush_stopped_at = id
+                                    break
                             else:
                                 deleted_trash += 1
                                 logger.critical("could not format spooled event", {
@@ -490,12 +528,11 @@ class NotificationForwarder(object):
                                     'action': 'could_not_format'
                                 })
                                 self.spool_store.delete(id)
-                                progress = True
                                 logger.info("delete trash event", {
                                     'event_id': id,
                                     'action': 'delete_trash'
                                 })
-                        if not progress:
+                        if self.flush_stopped_at is not None:
                             # only the rows this flush was responsible for, not
                             # the ones spooled concurrently after the snapshot
                             logger.critical("spooled events could not be submitted", {

{omdnotificationforwarder-4.5 → omdnotificationforwarder-4.6}/src/notificationforwarder/json/logger.py

@@ -121,6 +121,8 @@ class JsonLogger(NotificationLogger):
             msg_data["split_count"] = context['split_count']
         if 'event_id' in context:
             msg_data["event_id"] = context['event_id']
+        if 'new_event_id' in context:
+            msg_data["new_event_id"] = context['new_event_id']
 
         # Add action type
         if 'action' in context:

{omdnotificationforwarder-4.5 → omdnotificationforwarder-4.6}/src/notificationforwarder/spool.py

@@ -80,6 +80,23 @@ class SpoolStore:
             self.cursor.execute("DELETE FROM " + self.table_name + " WHERE id = ?", (event_id,))
             self.connection.commit()
 
+    def move_to_end(self, event_id):
+        # re-insert the row behind all others, keeping payload and timestamp,
+        # so it still expires on schedule. a row which is gone or already the
+        # last one (this includes a reused id) is left alone.
+        sql_copy = ("INSERT INTO " + self.table_name + "(payload, timestamp)"
+                    " SELECT payload, timestamp FROM " + self.table_name +
+                    " WHERE id = ? AND id < (SELECT MAX(id) FROM " + self.table_name + ")")
+        with self._lock:
+            self.cursor.execute(sql_copy, (event_id,))
+            if self.cursor.rowcount != 1:
+                self.connection.commit()
+                return None
+            new_id = self.cursor.lastrowid
+            self.cursor.execute("DELETE FROM " + self.table_name + " WHERE id = ?", (event_id,))
+            self.connection.commit()
+            return new_id
+
     def decode(self, text):
         return json.loads(text)
 

{omdnotificationforwarder-4.5 → omdnotificationforwarder-4.6}/src/notificationforwarder/text/logger.py

@@ -93,6 +93,11 @@ class TextLogger(NotificationLogger):
         if 'split_error' in context and 'raw_event' in context:
             return f"error split_events failed for {context['raw_event']}"
 
+        # Handle a rejected spooled event which was moved behind the others
+        if context.get('action') == 'moved_to_end':
+            return (f"event {context.get('event_id', '')} was rejected while the receiver accepts "
+                    f"other events and moved to the end of the spool as {context.get('new_event_id', '')}")
+
         # Handle spooled count messages
         if 'spooled_count' in context and 'action' in context:
             action = context['action']

{omdnotificationforwarder-4.5 → omdnotificationforwarder-4.6}/tests/TESTS.md

@@ -167,13 +167,32 @@ Integration tests for `load_forwarder()` using the `pythonpath/lib/python` test
 | Test | What it verifies | Intention |
 |------|------------------|-----------|
 | `test_discarded_event_deleted_with_trash_log` | Spooled event returning a discarded `FormattedEvent` during replay is skipped and deleted, not submitted | The flush loop checks `is_discarded` and treats discarded events as trash. |
-| `test_persistent_failure_keeps_row_and_terminates` | `submit()` always returning `False` → event stays in spool, stuck detection fires | Persistent failures are not silently dropped; the system logs and stops retrying. |
+| `test_persistent_failure_keeps_row_and_terminates` | `submit()` always returning `False` → event stays in spool, stuck detection fires | Persistent failures are not silently dropped; the system logs and stops at the first failed replay. |
 | `test_spool_init_failure_returns_false` | `SpoolStore.enqueue` raising `sqlite3.OperationalError` → returns `False` with "database error" log | Database errors are caught and logged, not raised. |
-| `test_flush_beyond_batch_limit_stuck_detection` | 12 spooled events (exceeds batch limit of 10) with all submits failing → stuck detection, all events stay | The stuck detection logic handles batch boundaries correctly. |
-| `test_flush_terminates_while_other_processes_spool` | All submits fail while a second connection spools a new event after each submit → flush stops after the first batch, events 0–9 submitted once each, all 22 rows kept, stuck log reports `spooled_count` 12 (snapshot rows, not the live queue) | A flush is bounded to the rows present at its start; concurrent spooling cannot keep it (and the flush lock) alive. |
+| `test_flush_beyond_batch_limit_stuck_detection` | 12 spooled events (exceeds batch limit of 10), the first 11 replays succeed and the 12th fails → the flush continues into the second batch, stops at the 12th event, summary attempted=12, recovered=11, stays_in_spool=1 | Batch boundaries do not end a flush; only a failed replay does, and the counters stay accurate. |
+| `test_flush_terminates_while_other_processes_spool` | All submits fail while a second connection spools a new event after each submit → flush stops at the first failed submit, only event 0 submitted, all 13 rows kept, stuck log reports `spooled_count` 12 (snapshot rows, not the live queue) | A flush is bounded to the rows present at its start; concurrent spooling cannot keep it (and the flush lock) alive. |
 | `test_formatter_returns_none_during_replay_deletes_trash` | `format_event` returning `None` during replay → event deleted as trash | Un-formattable events are cleaned up, not retried forever. |
 | `test_summary_logging_resumes_after_flush` | `baseclass_logs_summary` is reset to `True` at flush start | The no_more_logging flag is reset so operators see summary logs after recovery. |
 
+### class TestFlushStopAndMoveToEnd
+
+Spooled events 0, 1, 2 (ids 1, 2, 3) are replayed through a mocked `submit()`; the live event is forwarded with `forward()`.
+
+| Test | What it verifies | Intention |
+|------|------------------|-----------|
+| `test_flush_stops_at_first_failed_replay` | All replays fail → only event 0 submitted, all three stay, `flush_stopped_at` = 1, critical "event stays in spool" and "spooled events could not be submitted" | During an outage a flush costs one submit timeout, not one per spooled event. |
+| `test_flush_stops_after_successful_replays` | Event 0 succeeds, event 1 fails → 0 removed, 2 not submitted, 1 and 2 stay | The flush stops at the first failure, also after successful replays. |
+| `test_flush_stops_on_submit_timeout` | `submit()` raises `ForwarderTimeoutError` → only event 0 submitted | A timeout counts as a failed replay. |
+| `test_flush_continues_after_trash` | Event 0 discarded during replay, event 1 succeeds → spool empty, `flush_stopped_at` is `None` | Removed trash does not stop the flush. |
+| `test_rejected_event_moves_to_end` | Event 0 rejected, live event accepted → spool order 1, 2, 0; the moved row (id 4) keeps the original timestamp; critical "spooled event moved to the end of the spool" with `event_id` 1, `new_event_id` 4; the next forward delivers 1 and 2 and does not move 0 again (now last) | A permanently rejected event cannot block the backlog. |
+| `test_no_move_when_live_submit_fails` | All submits fail → spool 0, 1, 2, live; no move | During an outage the order of the backlog stays as it is. |
+| `test_no_move_without_live_submit` | `forward_formatted(None)` → flush stops at 0, nothing submitted afterwards, no move | Only a successful submit proves that the receiver is up. |
+| `test_heartbeat_success_moves_rejected_event` | Live event is a heartbeat and is accepted → spool order 1, 2, 0 | Heartbeats also prove that the receiver is up. |
+| `test_no_move_while_another_process_flushes` | A child process takes the flush lock while the live event is submitted → no move, debug "missed the flush lock, rejected event stays in place"; after the lock is released the next forward moves 0 | A running flush may hold the row in memory, so the move happens only under the flush lock. |
+| `test_no_move_when_event_is_gone` | Another connection deletes event 0 while the live event is submitted → spool 1, 2, no move, no duplicate | A row delivered by another process is not copied back. |
+| `test_no_move_when_event_is_already_last` | The rejected event is the only one → stays with id 1, no move log | Moving the last event would change nothing. |
+| `test_moved_event_still_expires` | Event 0 spooled 4 minutes ago is moved, then `prune_expired(3)` drops it | Retention is measured from the original spool time. |
+
 ### class TestSpoolStore
 
 Low-level `SpoolStore` operations against a real SQLite database.
@@ -191,7 +210,11 @@ Low-level `SpoolStore` operations against a real SQLite database.
 | `test_enqueue_non_serializable_data` | Enqueuing a `set` raises `TypeError` | Non-JSON-serializable data is rejected. |
 | `test_decode_round_trip` | JSON encode → decode produces the original dict | Serialization preserves data. |
 | `test_concurrent_enqueue` | Two threads each enqueuing 10 events concurrently → 20 rows, no corruption | Thread safety of concurrent writes. |
-| `test_concurrent_enqueue_shared_instance` | Shared SpoolStore instance across threads documents SQLite cursor limitation | SQLite doesn't allow recursive cursor use; the real forwarder uses file locks. |
+| `test_concurrent_enqueue_shared_instance` | Two threads each enqueue 10 events on one shared SpoolStore instance → 20 rows, no errors | The per-instance thread lock serializes access to the shared cursor; without it the test fails or crashes the interpreter. |
+| `test_move_to_end_keeps_payload_and_timestamp` | `move_to_end(1)` with 3 rows → returns 4, order (2, 1), (3, 2), (4, 0), timestamp of the moved row unchanged | The moved row still expires on schedule. |
+| `test_move_to_end_skips_last_and_missing_rows` | `move_to_end()` of the last row or a missing id → `None`, rows unchanged | A reused id is always the last one, so it is never moved. |
+| `test_move_to_end_does_not_hold_write_lock` | After a move and after a skipped move, a second connection with a 0.1 s busy timeout can insert | The move commits, other processes can keep spooling. |
+| `test_concurrent_mixed_operations_shared_instance` | One thread enqueues 200 events while another fetches (by `after_id`), counts and deletes every other row on the same instance → no errors, fetched rows in id order and never twice, every event either fetched or still spooled, `count()` = enqueued − deleted | Mixed operations on a shared instance stay consistent. SQLite reuses the id of a deleted last row, so the test does not assume ids grow monotonically. |
 
 ---
 
@@ -371,6 +394,7 @@ Smoke tests for every logging call path in `baseclass.py`. Each test calls a spe
 | `test_concurrent_flush_suppressed` | `info("concurrent flush suppressed", ...)` renders | Concurrent flush suppression path is covered. |
 | `test_spooled_events_to_be_resent` | `info("spooled events to be re-sent", ...)` renders | Spool replay initiation path is covered. |
 | `test_spooled_events_could_not_be_submitted` | `critical("spooled events could not be submitted", ...)` renders | Persistent submission failure path is covered. |
+| `test_spooled_event_moved_to_end` | `action: moved_to_end` renders "event 7 was rejected while the receiver accepts other events and moved to the end of the spool as 31" | The text log names the old and the new id of a moved event. |
 | `test_delivery_failed_could_not_persist` | `critical("delivery failed and event could not be persisted", ...)` renders | Unrecoverable failure path is covered. |
 | `test_formatted_event_incomplete` | `critical("formatted event incomplete", ...)` renders | Incomplete event path is covered. |
 
@@ -379,6 +403,7 @@ Smoke tests for every logging call path in `baseclass.py`. Each test calls a spe
 | Test | What it verifies | Intention |
 |------|------------------|-----------|
 | `test_success_structured_fields` | JsonLogger `info("forwarded", ...)` with event, status, forwarder_name, split_count renders structured JSON | Success path JSON structure is correct. |
+| `test_moved_to_end_structured_fields` | JsonLogger output of a moved event contains `event_id`, `new_event_id` and `action` | Both ids reach the JSON log. |
 | `test_failure_structured_fields` | JsonLogger `critical("forward failed", ...)` with exception, event, spooled, forwarder_name, attempt renders structured JSON | Failure path JSON structure is correct. |
 
 ---
@@ -503,6 +528,7 @@ Smoke tests for every logging call path in `baseclass.py`. Each test calls a spe
 | `test_flush_replays_spooled_events_on_recovery` | New forwarder (no `fail`) triggers flush → spooled events replayed and deleted | Recovery path: flush + replay. |
 | `test_flush_drops_expired_spooled_events` | Spooled events older than `max_spool_minutes` are dropped during flush | Expired events are cleaned up. |
 | `test_flush_logs_when_concurrent_lock_is_unavailable` | Flush lock not acquired → "concurrent flush suppressed" log | Concurrent flush prevention works. |
+| `test_flush_is_suppressed_while_another_process_holds_the_lock` | A child process holds the real fcntl flush lock → flush tries 3 times, logs "concurrent flush suppressed", spooled event stays; after the child releases the lock the next flush replays it | The cross-process flush lock and its retry work against a real competing process. |
 | `test_timeout_decorator_raises_forwarder_timeout` | `@timeout` decorator raises `ForwarderTimeoutError` on timeout | Timeout enforcement works. |
 | `test_timeout_decorator_preserves_underlying_exception` | Non-timeout exceptions propagate normally through `@timeout` | Timeout doesn't swallow other exceptions. |
 

{omdnotificationforwarder-4.5 → omdnotificationforwarder-4.6}/tests/test_delivery_resilience.py

@@ -1,6 +1,7 @@
 import logging
 import os
 import shutil
+import subprocess
 import sys
 import time
 
@@ -131,6 +132,40 @@ def test_flush_logs_when_concurrent_lock_is_unavailable(resilience_setup, monkey
     assert "DEBUG - missed the flush lock" in log
 
 
+def test_flush_is_suppressed_while_another_process_holds_the_lock(resilience_setup):
+    forwarder = new_example_forwarder(fail="1")
+    forwarder.forward({"description": "spool me", "signature": "sig-locked"})
+    assert forwarder.num_spooled_events() == 1
+
+    # fcntl locks are per process, so the competing flush must be a real process
+    holder = subprocess.Popen(
+        [sys.executable, "-c",
+         "import fcntl, sys\n"
+         "f = open(sys.argv[1], 'w')\n"
+         "fcntl.lockf(f, fcntl.LOCK_EX | fcntl.LOCK_NB)\n"
+         "print('locked', flush=True)\n"
+         "sys.stdin.read()\n",
+         forwarder.db_lock_file],
+        stdin=subprocess.PIPE, stdout=subprocess.PIPE, text=True)
+    try:
+        assert holder.stdout.readline().strip() == "locked"
+        recovering_forwarder = new_example_forwarder()
+        recovering_forwarder.flush()
+        assert recovering_forwarder.num_spooled_events() == 1
+        log = open(get_logfile(recovering_forwarder)).read()
+        assert log.count("DEBUG - flush lock failed") == 3
+        assert "INFO - concurrent flush suppressed" in log
+    finally:
+        holder.stdin.close()
+        holder.wait(timeout=10)
+
+    # once the other process has released the lock, the next flush replays the event
+    recovering_forwarder.flush()
+    assert recovering_forwarder.num_spooled_events() == 0
+    signatures = [line.strip() for line in open(recovering_forwarder.signaturefile).readlines()]
+    assert signatures == ["sig-locked"]
+
+
 def test_timeout_decorator_raises_forwarder_timeout():
     @timeout(0.01, error_message="submit ran into a timeout")
     def slow_call():

{omdnotificationforwarder-4.5 → omdnotificationforwarder-4.6}/tests/test_loggers.py

@@ -417,6 +417,20 @@ class TestTextLoggerMessageBranches:
         finally:
             pl.removeHandler(h)
 
+    def test_spooled_event_moved_to_end(self, setup):
+        """'spooled event moved to the end of the spool' names the old and new id."""
+        logger, pl, h = self._make_logger("test_moved_to_end")
+        try:
+            message = logger._build_message("spooled event moved to the end of the spool", {
+                'event_id': 7,
+                'new_event_id': 31,
+                'action': 'moved_to_end',
+            })
+        finally:
+            pl.removeHandler(h)
+        assert message == ("event 7 was rejected while the receiver accepts other events "
+                           "and moved to the end of the spool as 31")
+
     def test_delivery_failed_could_not_persist(self, setup):
         """'delivery failed and event could not be persisted' message."""
         logger, pl, h = self._make_logger("test_could_not_persist")
@@ -473,6 +487,29 @@ class TestJsonLoggerStructuredFields:
         finally:
             pl.removeHandler(h)
 
+    def test_moved_to_end_structured_fields(self, setup):
+        """JSON output of a moved event includes both ids and the action."""
+        import io
+        import json as json_mod
+        logger, pl, h = self._make_logger("test_json_moved_to_end")
+        stream = io.StringIO()
+        capture = logging.StreamHandler(stream)
+        pl.addHandler(capture)
+        try:
+            logger.critical("spooled event moved to the end of the spool", {
+                'event_id': 7,
+                'new_event_id': 31,
+                'action': 'moved_to_end',
+            })
+        finally:
+            pl.removeHandler(capture)
+            pl.removeHandler(h)
+        entry = json_mod.loads(stream.getvalue().strip().splitlines()[-1])
+        text = json_mod.dumps(entry)
+        assert '"event_id": 7' in text
+        assert '"new_event_id": 31' in text
+        assert '"action": "moved_to_end"' in text
+
     def test_failure_structured_fields(self, setup):
         """JSON failure path includes spool and exception fields."""
         import json as json_mod

{omdnotificationforwarder-4.5 → omdnotificationforwarder-4.6}/tests/test_resilience_spool.py

@@ -546,7 +546,7 @@ class TestSpoolReplayEdges:
         assert any("database error" in str(c) for c in critical_calls)
 
     def test_flush_beyond_batch_limit_stuck_detection(self, setup, monkeypatch):
-        """Flush with more than 10 events triggers stuck detection correctly."""
+        """Flush continues into the next batch and stops at the first failed replay."""
         forwarder = _new_forwarder(
             "example",
             forwarder_name="webhook",
@@ -559,11 +559,14 @@ class TestSpoolReplayEdges:
 
         assert forwarder.num_spooled_events() == 12
 
-        # All submits fail to trigger stuck detection
-        def always_fail(formatted_event):
-            return False
+        # the first 11 replays succeed, the 12th (in the second batch) fails
+        submitted = []
 
-        monkeypatch.setattr(forwarder, "submit", always_fail)
+        def fail_twelfth(formatted_event):
+            submitted.append(formatted_event.eventopts["event_num"])
+            return formatted_event.eventopts["event_num"] != 11
+
+        monkeypatch.setattr(forwarder, "submit", fail_twelfth)
 
         def format_ok(raw):
             fe = _formatted_event(raw)
@@ -579,8 +582,8 @@ class TestSpoolReplayEdges:
         original_logger = baseclass.logger
 
         class MockLogger:
-            def info(self, msg, *args, **kwargs):
-                info_calls.append(msg)
+            def info(self, msg, context=None, *args, **kwargs):
+                info_calls.append((msg, context))
             def critical(self, msg, *args, **kwargs):
                 critical_calls.append(msg)
             def warning(self, msg, *args, **kwargs):
@@ -594,14 +597,19 @@ class TestSpoolReplayEdges:
         finally:
             baseclass.logger = original_logger
 
-        # All 12 events should still be in spool (all failed)
-        assert forwarder.num_spooled_events() == 12
-
-        # Stuck detection should have triggered after the first batch (10 fetched, none replayed)
+        # the flush went on into the second batch and stopped at the 12th event
+        assert submitted == list(range(12))
+        assert forwarder.num_spooled_events() == 1
+        assert forwarder.flush_stopped_at == 12
         assert any("could not be submitted" in str(c) for c in critical_calls)
-
-        # spool replay summary should be logged
-        assert any("spool replay summary" in str(c) for c in info_calls)
+        summary = [ctx for msg, ctx in info_calls if msg == "spool replay summary"]
+        assert summary == [{
+            'attempted': 12,
+            'recovered_count': 11,
+            'stays_in_spool_count': 1,
+            'deleted_trash_count': 0,
+            'dropped_count': 0,
+        }]
 
     def test_flush_terminates_while_other_processes_spool(self, setup, monkeypatch):
         """Flush must not loop forever when concurrent processes keep spooling.
@@ -663,10 +671,10 @@ class TestSpoolReplayEdges:
             baseclass.logger = original_logger
             other_process.close()
 
-        # one batch failed completely -> stop, every event tried at most once
-        assert submitted == list(range(10))
-        # nothing lost: 12 original + 10 spooled concurrently
-        assert forwarder.num_spooled_events() == 22
+        # the first failed submit ends the flush
+        assert submitted == [0]
+        # nothing lost: 12 original + 1 spooled concurrently
+        assert forwarder.num_spooled_events() == 13
         # the stuck log counts only the rows of this flush's snapshot
         stuck = [ctx for msg, ctx in critical_calls if msg == "spooled events could not be submitted"]
         assert stuck == [{'spooled_count': 12, 'action': 'could_not_submit'}]
@@ -743,6 +751,256 @@ class TestSpoolReplayEdges:
         assert forwarder.baseclass_logs_summary is True
 
 
+# ============================================================================
+# Flush stops at the first failure, rejected events move to the end
+# ============================================================================
+
+class _RecordingLogger:
+    """Records (level, message, context) of every log call."""
+    def __init__(self):
+        self.calls = []
+
+    def __getattr__(self, level):
+        def log(msg, context=None, *args, **kwargs):
+            self.calls.append((level, msg, context))
+        return log
+
+    def messages(self, level):
+        return [msg for lvl, msg, _ctx in self.calls if lvl == level]
+
+    def moved(self):
+        return [ctx for _lvl, msg, ctx in self.calls if msg == "spooled event moved to the end of the spool"]
+
+
+class TestFlushStopAndMoveToEnd:
+    """Spooled events are numbered 0, 1, 2, ... (ids 1, 2, 3, ...); the live event is "live"."""
+
+    LIVE = {"HOSTNAME": "h1", "event_num": "live"}
+
+    def _forwarder(self, monkeypatch, submit, num_spooled=3, heartbeat=False):
+        forwarder = _new_forwarder(
+            "example",
+            forwarder_name="webhook",
+            forwarder_opts={"url": "http://example.invalid/api"},
+        )
+        for i in range(num_spooled):
+            forwarder.spool_store.enqueue({"HOSTNAME": "h1", "event_num": i})
+
+        def format_ok(raw):
+            fe = _formatted_event(raw)
+            fe.payload = {"data": "test"}
+            fe.summary = "test"
+            if raw.get("event_num") == "live":
+                fe.is_heartbeat = heartbeat
+            return fe
+
+        monkeypatch.setattr(forwarder, "format_event", format_ok)
+        monkeypatch.setattr(forwarder, "submit", submit)
+        # new() has set the module logger, so record from here on
+        self.log = _RecordingLogger()
+        monkeypatch.setattr(baseclass, "logger", self.log)
+        return forwarder
+
+    def _spooled(self, forwarder):
+        rows = forwarder.spool_store.fetch_batch(limit=100)
+        return [forwarder.spool_store.decode(text)["event_num"] for _id, text in rows]
+
+    def _timestamp(self, forwarder, event_id):
+        forwarder.dbcurs.execute(
+            "SELECT timestamp FROM {table} WHERE id = ?".format(table=forwarder.table_name), (event_id,))
+        return forwarder.dbcurs.fetchone()[0]
+
+    @staticmethod
+    def _reject_first(formatted_event):
+        return formatted_event.eventopts["event_num"] != 0
+
+    def test_flush_stops_at_first_failed_replay(self, setup, monkeypatch):
+        submitted = []
+
+        def always_fail(formatted_event):
+            submitted.append(formatted_event.eventopts["event_num"])
+            return False
+
+        forwarder = self._forwarder(monkeypatch, always_fail)
+        forwarder.flush()
+
+        assert submitted == [0]
+        assert self._spooled(forwarder) == [0, 1, 2]
+        assert forwarder.flush_stopped_at == 1
+        assert self.log.messages("critical") == ["event stays in spool", "spooled events could not be submitted"]
+
+    def test_flush_stops_after_successful_replays(self, setup, monkeypatch):
+        submitted = []
+
+        def fail_second(formatted_event):
+            submitted.append(formatted_event.eventopts["event_num"])
+            return formatted_event.eventopts["event_num"] != 1
+
+        forwarder = self._forwarder(monkeypatch, fail_second)
+        forwarder.flush()
+
+        assert submitted == [0, 1]
+        assert self._spooled(forwarder) == [1, 2]
+
+    def test_flush_stops_on_submit_timeout(self, setup, monkeypatch):
+        submitted = []
+
+        def time_out(formatted_event):
+            submitted.append(formatted_event.eventopts["event_num"])
+            raise baseclass.ForwarderTimeoutError("submit ran into a timeout")
+
+        forwarder = self._forwarder(monkeypatch, time_out)
+        forwarder.flush()
+
+        assert submitted == [0]
+        assert self._spooled(forwarder) == [0, 1, 2]
+
+    def test_flush_continues_after_trash(self, setup, monkeypatch):
+        submitted = []
+
+        def succeed(formatted_event):
+            submitted.append(formatted_event.eventopts["event_num"])
+            return True
+
+        forwarder = self._forwarder(monkeypatch, succeed, num_spooled=2)
+        format_ok = forwarder.format_event
+
+        def discard_first(raw):
+            fe = format_ok(raw)
+            if raw.get("event_num") == 0:
+                fe.discard()
+            return fe
+
+        monkeypatch.setattr(forwarder, "format_event", discard_first)
+        forwarder.flush()
+
+        assert submitted == [1]
+        assert self._spooled(forwarder) == []
+        assert forwarder.flush_stopped_at is None
+
+    def test_rejected_event_moves_to_end(self, setup, monkeypatch):
+        forwarder = self._forwarder(monkeypatch, self._reject_first)
+        forwarder.dbcurs.execute(
+            "UPDATE {table} SET timestamp = datetime('now', '-1 minutes') WHERE id = 1".format(
+                table=forwarder.table_name))
+        forwarder.dbconn.commit()
+        original_timestamp = self._timestamp(forwarder, 1)
+
+        # the flush stops at 0, the live event is accepted -> 0 moves behind 1 and 2
+        forwarder.forward(dict(self.LIVE))
+
+        assert self._spooled(forwarder) == [1, 2, 0]
+        assert self._timestamp(forwarder, 4) == original_timestamp
+        assert self.log.moved() == [{'event_id': 1, 'new_event_id': 4, 'action': 'moved_to_end'}]
+
+        # the next flush delivers 1 and 2 and stops at 0 again, which is now last
+        forwarder.forward(dict(self.LIVE))
+        assert self._spooled(forwarder) == [0]
+        assert len(self.log.moved()) == 1
+
+    def test_no_move_when_live_submit_fails(self, setup, monkeypatch):
+        def always_fail(formatted_event):
+            return False
+
+        forwarder = self._forwarder(monkeypatch, always_fail)
+        forwarder.forward(dict(self.LIVE))
+
+        assert self._spooled(forwarder) == [0, 1, 2, "live"]
+        assert self.log.moved() == []
+
+    def test_no_move_without_live_submit(self, setup, monkeypatch):
+        forwarder = self._forwarder(monkeypatch, self._reject_first)
+        # forward_formatted() without an event: flush only, nothing submitted afterwards
+        forwarder.forward_formatted(None)
+
+        assert self._spooled(forwarder) == [0, 1, 2]
+        assert self.log.moved() == []
+
+    def test_heartbeat_success_moves_rejected_event(self, setup, monkeypatch):
+        live_heartbeats = []
+
+        def reject_first(formatted_event):
+            if formatted_event.eventopts["event_num"] == "live":
+                live_heartbeats.append(formatted_event.is_heartbeat)
+            return self._reject_first(formatted_event)
+
+        forwarder = self._forwarder(monkeypatch, reject_first, heartbeat=True)
+        forwarder.forward(dict(self.LIVE))
+
+        assert live_heartbeats == [True]
+        assert self._spooled(forwarder) == [1, 2, 0]
+
+    def test_no_move_while_another_process_flushes(self, setup, monkeypatch):
+        import subprocess
+        holder = []
+
+        def reject_first_and_lock(formatted_event):
+            if formatted_event.eventopts["event_num"] == "live" and not holder:
+                # another process takes the flush lock between the flush and the move
+                holder.append(subprocess.Popen(
+                    [sys.executable, "-c",
+                     "import fcntl, sys\n"
+                     "f = open(sys.argv[1], 'w')\n"
+                     "fcntl.lockf(f, fcntl.LOCK_EX | fcntl.LOCK_NB)\n"
+                     "print('locked', flush=True)\n"
+                     "sys.stdin.read()\n",
+                     forwarder.db_lock_file],
+                    stdin=subprocess.PIPE, stdout=subprocess.PIPE, text=True))
+                assert holder[0].stdout.readline().strip() == "locked"
+            return self._reject_first(formatted_event)
+
+        forwarder = self._forwarder(monkeypatch, reject_first_and_lock)
+        try:
+            forwarder.forward(dict(self.LIVE))
+            assert self._spooled(forwarder) == [0, 1, 2]
+            assert self.log.moved() == []
+            assert "missed the flush lock, rejected event stays in place" in self.log.messages("debug")
+        finally:
+            holder[0].stdin.close()
+            holder[0].wait(timeout=10)
+
+        # the next flush stops at 0 again and, with the lock free, moves it
+        forwarder.forward(dict(self.LIVE))
+        assert self._spooled(forwarder) == [1, 2, 0]
+
+    def test_no_move_when_event_is_gone(self, setup, monkeypatch):
+        def reject_first_then_gone(formatted_event):
+            if formatted_event.eventopts["event_num"] == "live":
+                # another process delivered and removed event 0 meanwhile
+                other_process = SpoolStore(forwarder.db_file, forwarder.table_name)
+                other_process.open()
+                other_process.delete(1)
+                other_process.close()
+            return self._reject_first(formatted_event)
+
+        forwarder = self._forwarder(monkeypatch, reject_first_then_gone)
+        forwarder.forward(dict(self.LIVE))
+
+        assert self._spooled(forwarder) == [1, 2]
+        assert self.log.moved() == []
+
+    def test_no_move_when_event_is_already_last(self, setup, monkeypatch):
+        forwarder = self._forwarder(monkeypatch, self._reject_first, num_spooled=1)
+        forwarder.forward(dict(self.LIVE))
+
+        assert [event_id for event_id, _text in forwarder.spool_store.fetch_batch()] == [1]
+        assert self.log.moved() == []
+
+    def test_moved_event_still_expires(self, setup, monkeypatch):
+        forwarder = self._forwarder(monkeypatch, self._reject_first)
+        forwarder.dbcurs.execute(
+            "UPDATE {table} SET timestamp = datetime('now', '-4 minutes') WHERE id = 1".format(
+                table=forwarder.table_name))
+        forwarder.dbconn.commit()
+
+        forwarder.forward(dict(self.LIVE))
+        assert self._spooled(forwarder) == [1, 2, 0]
+
+        # a retention of 3 minutes, measured from the original spool time, drops 0
+        assert forwarder.spool_store.prune_expired(3) == 1
+        assert self._spooled(forwarder) == [1, 2]
+
+
 # ============================================================================
 # SpoolStore direct tests
 # ============================================================================
@@ -997,3 +1255,118 @@ class TestSpoolStore:
         assert errors == []
         assert store.count() == 20
         store.close()
+
+    def test_concurrent_mixed_operations_shared_instance(self, setup):
+        """One thread enqueues while another fetches, counts and deletes on the
+        same SpoolStore instance. Fetches see only fully stored rows in id order,
+        no row twice, and the final count matches enqueued minus deleted."""
+        import threading
+
+        db_path = os.path.join(OMD_ROOT, "tmp", "test_spool_shared_mixed.db")
+        store = SpoolStore(db_path, "test_events")
+        store.open()
+        store.init_db()
+
+        num_events = 200
+        errors = []
+        fetched = []
+        deleted = []
+        writer_done = threading.Event()
+
+        def writer():
+            try:
+                for i in range(num_events):
+                    store.enqueue({"seq": i})
+            except Exception as e:
+                errors.append(e)
+            finally:
+                writer_done.set()
+
+        def reader():
+            try:
+                last_id = 0
+                while True:
+                    finished = writer_done.is_set()
+                    batch = store.fetch_batch(limit=10, after_id=last_id)
+                    store.count()
+                    store.max_id()
+                    for event_id, text in batch:
+                        fetched.append((event_id, store.decode(text)["seq"]))
+                        last_id = event_id
+                        # delete every other row, keep the rest
+                        if event_id % 2 == 0:
+                            store.delete(event_id)
+                            deleted.append(event_id)
+                    if finished and not batch:
+                        break
+            except Exception as e:
+                errors.append(e)
+
+        threads = [threading.Thread(target=writer), threading.Thread(target=reader)]
+        for t in threads:
+            t.start()
+        for t in threads:
+            t.join(timeout=30)
+        assert not any(t.is_alive() for t in threads)
+
+        assert errors == []
+        ids = [event_id for event_id, _seq in fetched]
+        assert ids == sorted(set(ids))
+        # without AUTOINCREMENT sqlite reuses the id of a deleted last row, so
+        # an event can land behind the reader's after_id. It must then still
+        # be in the spool: every event was either fetched once or is kept.
+        fetched_seqs = [seq for _event_id, seq in fetched]
+        assert fetched_seqs == sorted(set(fetched_seqs))
+        remaining = [store.decode(text)["seq"] for _id, text in store.fetch_batch(limit=num_events)]
+        kept_seqs = [seq for event_id, seq in fetched if event_id not in deleted]
+        assert sorted(set(fetched_seqs) | set(remaining)) == list(range(num_events))
+        assert set(kept_seqs) <= set(remaining)
+        assert store.count() == num_events - len(deleted)
+        store.close()
+
+    def _store_with_rows(self, name, num_rows):
+        db_path = os.path.join(OMD_ROOT, "tmp", name)
+        store = SpoolStore(db_path, "test_events")
+        store.open()
+        store.init_db()
+        for i in range(num_rows):
+            store.enqueue({"seq": i})
+        return store
+
+    def test_move_to_end_keeps_payload_and_timestamp(self, setup):
+        """move_to_end() puts the row behind all others with its payload and timestamp."""
+        store = self._store_with_rows("test_spool_move.db", 3)
+        store.cursor.execute("UPDATE test_events SET timestamp = '2020-01-01 00:00:00' WHERE id = 1")
+        store.connection.commit()
+
+        assert store.move_to_end(1) == 4
+        rows = store.fetch_batch()
+        assert [(event_id, store.decode(text)["seq"]) for event_id, text in rows] == [(2, 1), (3, 2), (4, 0)]
+        store.cursor.execute("SELECT timestamp FROM test_events WHERE id = 4")
+        assert store.cursor.fetchone()[0] == "2020-01-01 00:00:00"
+        store.close()
+
+    def test_move_to_end_skips_last_and_missing_rows(self, setup):
+        """A row that is already last, or gone, is not moved."""
+        store = self._store_with_rows("test_spool_move_skip.db", 2)
+
+        assert store.move_to_end(2) is None
+        assert store.move_to_end(99) is None
+        assert [event_id for event_id, _text in store.fetch_batch()] == [1, 2]
+        store.close()
+
+    def test_move_to_end_does_not_hold_write_lock(self, setup):
+        """After move_to_end() and after a skipped move, other connections can insert."""
+        store = self._store_with_rows("test_spool_move_lock.db", 2)
+        other = sqlite3.connect(store.db_file, timeout=0.1)
+        try:
+            store.move_to_end(1)
+            other.execute("INSERT INTO test_events(payload) VALUES ('{}')")
+            other.commit()
+            store.move_to_end(99)
+            other.execute("INSERT INTO test_events(payload) VALUES ('{}')")
+            other.commit()
+        finally:
+            other.close()
+        assert store.count() == 4
+        store.close()