omdnotificationforwarder 4.4__tar.gz → 4.6__tar.gz

{omdnotificationforwarder-4.4 → omdnotificationforwarder-4.6}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: omdnotificationforwarder
-Version: 4.4
+Version: 4.6
 Summary: A framework for notification scripts for OMD
 Project-URL: Homepage, https://github.com/lausser/noteventificationforhandlerwarder
 Project-URL: Bug Tracker, https://github.com/lausser/noteventificationforhandlerwarder/issues
@@ -276,13 +276,18 @@ Install the optional Nostr support with `pip install notificationforwarder[nostr
 |nsec|bech32 private key for signing|required|
 |tags|extra Nostr tags to include|`monitoring` plus host/service/state tags|
 
-Example:
+The recipient is identified by a `p` tag containing the recipient's public key (npub). The `p` tag must be present in the tags list — the forwarder will raise an error if no recipient is found.
+
+#### Static recipient
+
+If all notifications go to the same recipient, pass the `p` tag via `--forwarderopt tags`:
 
 ```bash
 $USER1$/notificationforwarder \
     --forwarder nostr \
     --forwarderopt relays='wss://relay.damus.io,wss://nostr-pub.wellorder.net' \
     --forwarderopt nsec='$NSEC$' \
+    --forwarderopt tags='[["p", "npub1xyz..."]]' \
     --formatter nostr \
     --eventopt HOSTNAME='$HOSTNAME$' \
     --eventopt SERVICEDESC='$SERVICEDESC$' \
@@ -290,7 +295,55 @@ $USER1$/notificationforwarder \
     --eventopt SERVICEOUTPUT='$SERVICEOUTPUT$'
 ```
 
-The recipient `p` tag can be provided through the `tags` forwarder option or custom formatter output. The message content is encrypted before relay publication, so it shows up as a DM in clients that support NIP-04 direct messages.
+#### Per-contact recipient
+
+In a real monitoring setup, different contacts may have different Nostr public keys. Use a custom contact variable (e.g., `$CONTACTNOSTRNPUB$`) and pass it as an eventopt. The formatter then adds it as the `p` tag.
+
+**Contact definition:**
+
+```
+define contact {
+    contact_name    admin-team
+    ...
+    _NOSTRNPUB      npub1xyz...
+}
+```
+
+**Command definition:**
+
+```
+define command {
+    command_name    notify-service-nostr
+    command_line    $USER1$/notificationforwarder \
+                        --forwarder nostr \
+                        --forwarderopt relays='wss://relay.damus.io' \
+                        --forwarderopt nsec='$NSEC$' \
+                        --formatter nostr_custom \
+                        --eventopt HOSTNAME='$HOSTNAME$' \
+                        --eventopt SERVICEDESC='$SERVICEDESC$' \
+                        --eventopt SERVICESTATE='$SERVICESTATE$' \
+                        --eventopt SERVICEOUTPUT='$SERVICEOUTPUT$' \
+                        --eventopt NOSTR_PUBKEY='$_CONTACTNOSTRNPUB$' \
+                    >> $USER4$/var/log/notificationforwarder_errors.log 2>&1
+}
+```
+
+**Custom formatter** in `~/local/lib/python/notificationforwarder/nostr_custom/formatter.py`:
+
+```python
+from notificationforwarder.nostr.formatter import NostrFormatter
+
+class NostrCustomFormatter(NostrFormatter):
+    def format_event(self, event):
+        super().format_event(event)
+        pubkey = event.eventopts.get("NOSTR_PUBKEY", "")
+        if pubkey:
+            event.payload["tags"].append(["p", pubkey])
+```
+
+The `p` tag is appended to the tags list that the default Nostr formatter already builds with monitoring tags. The forwarder picks it up and uses it as the recipient for the encrypted DM.
+
+The message content is encrypted before relay publication, so it shows up as a DM in clients that support NIP-04 direct messages.
 
 ## Loggers
 
@@ -463,3 +516,117 @@ define command{
                     >> $USER4$/var/log/notificationforwarder_errors.log 2>&1
 }
 ```
+
+## Spool Flushing
+
+When forwarding to a recipient fails, the event is saved in a local sqlite database. The framework automatically attempts to re-submit these spooled events on every subsequent call to notificationforwarder for the same forwarder.
+
+The flush mechanism runs inside `forward_formatted()` for **every event** — not just heartbeats or special cases. Before submitting the current event, the framework checks if there are spooled events waiting. If there are, it attempts to flush them by calling `submit()` for each one, oldest first. Events that are successfully submitted are deleted from the spool. The first event that fails again ends the flush: it and all later events stay in the queue for the next attempt. A receiver that does not accept one event will most likely not accept the next ones either, so a flush costs at most one submit timeout instead of one per spooled event.
+
+If the receiver rejects a spooled event for good (for example because of an invalid payload), that event must not block the queue until `max_spool_minutes` have passed. When a flush stopped at an event and the current event is then accepted by the receiver, the receiver is evidently up. The stopped event is moved to the end of the queue and logged as "spooled event moved to the end of the spool". It keeps its original time, so it is still dropped when `max_spool_minutes` have passed, and the next flush can deliver the events behind it.
+
+This means that in a busy monitoring environment where notifications flow regularly, spooled events get drained naturally — each new notification triggers a flush attempt on its way through.
+
+### Delivery Order
+
+Spooled events are stored and re-submitted in the order in which they failed. As long as every delivery succeeds, the receiver gets the events in the order in which they were created. After delivery failures, this is not guaranteed. A newer event can reach the receiver before an older one, for example a RECOVERY before its PROBLEM:
+
+- While one notificationforwarder process is flushing the spool, another process submits its own new event directly, ahead of the spooled ones.
+- A spooled event was rejected while the receiver accepted the current event. The rejected event is moved to the end of the spool and is delivered, if at all, after newer events.
+- The flush stopped early, or `probe()` returned False. The process still submits its own new event directly.
+- A submit ran into the timeout, but the receiver got the event anyway. The event stays in the spool, and the receiver gets it a second time later, after newer events.
+
+If the receiver depends on the order, let the formatter include the event's creation time (`omd_originating_timestamp`, set for every event) in the payload. The receiver or its integration can then ignore an event that is older than the state it already has.
+
+### Probe Method
+
+Forwarders can implement an optional `probe()` method to optimize the flush behavior. The `probe()` method checks whether the receiving system is reachable *before* the framework attempts to flush spooled events via `submit()`.
+
+The framework uses `probe()` as a pre-check with three possible scenarios:
+
+|Forwarder has probe()?|probe() result|What happens|Why|
+|---|---|---|---|
+|No|—|flush proceeds|No way to check availability, must try|
+|Yes|True|flush proceeds|Receiver is confirmed reachable|
+|Yes|False|flush skipped|Receiver is known to be down, avoid wasted submit() calls|
+
+Without `probe()`, every flush attempt would call `submit()` for each spooled event even when the receiver is known to be down — wasting time and producing noisy error logs. With `probe()`, the framework can short-circuit: if the receiver is unreachable, the flush is skipped entirely and spooled events wait in the database until the next call.
+
+Example implementation:
+
+```python
+def probe(self):
+    r = requests.head(self.url)
+    return r.status_code == 200
+```
+
+### Heartbeat Events
+
+Formatters can mark an event as a heartbeat by setting `event.is_heartbeat = True` on the `FormattedEvent` object. Heartbeat events have two special properties:
+
+1. **They skip spooling on failure.** If a heartbeat event cannot be delivered, it is NOT saved to the spool database. This prevents periodic check-in events from accumulating in the retry queue.
+
+2. **They guarantee regular flush attempts.** While every event triggers the flush phase, heartbeat events ensure this happens at regular intervals — but only when scheduled. A cronjob or Nagios service using notificationforwarder as its check command provides this scheduling, creating a steady pulse of heartbeat events. Without such a schedule, heartbeats don't exist and spooled events could sit in the queue for hours waiting for the next real notification to trigger a flush.
+
+Example formatter that sets the heartbeat flag:
+
+```python
+class MyFormatter(NotificationFormatter):
+    def format_event(self, event):
+        notification_type = event.eventopts.get("NOTIFICATIONTYPE", "")
+        event.is_heartbeat = notification_type == "HEARTBEAT"
+
+        event.payload = {
+            "hostname": event.eventopts.get("HOSTNAME", ""),
+            "heartbeat": event.is_heartbeat,
+        }
+        event.summary = "heartbeat from {}".format(event.eventopts.get("HOSTNAME", ""))
+```
+
+### Scheduling Heartbeats in Nagios/Naemon
+
+Since notificationforwarder always exits with code 0, it can be used as a Nagios check command to schedule periodic heartbeats. The service will always be in OK status — it's using Nagios purely as a scheduling mechanism.
+
+**Command definition:**
+
+```
+define command {
+    command_name    notificationforwarder_heartbeat
+    command_line    $USER1$/notificationforwarder \
+                        --forwarder webhook \
+                        --forwarderopt url=https://xxx.service-now.com/api/now/import/u_notification \
+                        --formatter myformatter \
+                        --eventopt HOSTNAME='$HOSTNAME$' \
+                        --eventopt SERVICEDESC='$SERVICEDESC$' \
+                        --eventopt NOTIFICATIONTYPE=HEARTBEAT \
+                        --eventopt SERVICEOUTPUT='heartbeat from $HOSTNAME$' \
+                    >> $USER4$/var/log/notificationforwarder_errors.log 2>&1
+}
+```
+
+**Service definition:**
+
+```
+define service {
+    host_name               monitoring-server
+    service_description     notificationforwarder-heartbeat
+    check_command           notificationforwarder_heartbeat
+    max_check_attempts      1
+    check_interval          5
+    retry_interval          1
+    notification_interval   0
+    notification_options    n
+    notification_period     24x7
+}
+```
+
+The key settings:
+- `check_interval 5` — runs the heartbeat every 5 minutes
+- `notification_interval 0` — no Nagios-level notifications
+- `notification_options n` — don't send notifications on state changes
+
+The heartbeat goes through the normal notificationforwarder flow: the formatter sets `is_heartbeat = True`, the flush phase checks for spooled events (and calls `probe()` if available), and the heartbeat itself is submitted to the receiver. If the receiver accepts it, the receiving system knows the monitoring infrastructure is alive (dead-man switch). If the receiver is down, the heartbeat is simply dropped — it won't add to the spool queue.
+
+The two use cases for periodic heartbeats:
+- **Dead-man switch**: The receiving system (e.g., ServiceNow) gets a regular "I'm alive" signal. If heartbeats stop, it knows the monitoring system is down.
+- **Spool draining**: Even when nothing breaks, the heartbeat triggers flush attempts at regular intervals, so spooled events are delivered quickly once the receiver recovers — without waiting for the next real notification.

{omdnotificationforwarder-4.4 → omdnotificationforwarder-4.6}/README.md

@@ -255,13 +255,18 @@ Install the optional Nostr support with `pip install notificationforwarder[nostr
 |nsec|bech32 private key for signing|required|
 |tags|extra Nostr tags to include|`monitoring` plus host/service/state tags|
 
-Example:
+The recipient is identified by a `p` tag containing the recipient's public key (npub). The `p` tag must be present in the tags list — the forwarder will raise an error if no recipient is found.
+
+#### Static recipient
+
+If all notifications go to the same recipient, pass the `p` tag via `--forwarderopt tags`:
 
 ```bash
 $USER1$/notificationforwarder \
     --forwarder nostr \
     --forwarderopt relays='wss://relay.damus.io,wss://nostr-pub.wellorder.net' \
     --forwarderopt nsec='$NSEC$' \
+    --forwarderopt tags='[["p", "npub1xyz..."]]' \
     --formatter nostr \
     --eventopt HOSTNAME='$HOSTNAME$' \
     --eventopt SERVICEDESC='$SERVICEDESC$' \
@@ -269,7 +274,55 @@ $USER1$/notificationforwarder \
     --eventopt SERVICEOUTPUT='$SERVICEOUTPUT$'
 ```
 
-The recipient `p` tag can be provided through the `tags` forwarder option or custom formatter output. The message content is encrypted before relay publication, so it shows up as a DM in clients that support NIP-04 direct messages.
+#### Per-contact recipient
+
+In a real monitoring setup, different contacts may have different Nostr public keys. Use a custom contact variable (e.g., `$CONTACTNOSTRNPUB$`) and pass it as an eventopt. The formatter then adds it as the `p` tag.
+
+**Contact definition:**
+
+```
+define contact {
+    contact_name    admin-team
+    ...
+    _NOSTRNPUB      npub1xyz...
+}
+```
+
+**Command definition:**
+
+```
+define command {
+    command_name    notify-service-nostr
+    command_line    $USER1$/notificationforwarder \
+                        --forwarder nostr \
+                        --forwarderopt relays='wss://relay.damus.io' \
+                        --forwarderopt nsec='$NSEC$' \
+                        --formatter nostr_custom \
+                        --eventopt HOSTNAME='$HOSTNAME$' \
+                        --eventopt SERVICEDESC='$SERVICEDESC$' \
+                        --eventopt SERVICESTATE='$SERVICESTATE$' \
+                        --eventopt SERVICEOUTPUT='$SERVICEOUTPUT$' \
+                        --eventopt NOSTR_PUBKEY='$_CONTACTNOSTRNPUB$' \
+                    >> $USER4$/var/log/notificationforwarder_errors.log 2>&1
+}
+```
+
+**Custom formatter** in `~/local/lib/python/notificationforwarder/nostr_custom/formatter.py`:
+
+```python
+from notificationforwarder.nostr.formatter import NostrFormatter
+
+class NostrCustomFormatter(NostrFormatter):
+    def format_event(self, event):
+        super().format_event(event)
+        pubkey = event.eventopts.get("NOSTR_PUBKEY", "")
+        if pubkey:
+            event.payload["tags"].append(["p", pubkey])
+```
+
+The `p` tag is appended to the tags list that the default Nostr formatter already builds with monitoring tags. The forwarder picks it up and uses it as the recipient for the encrypted DM.
+
+The message content is encrypted before relay publication, so it shows up as a DM in clients that support NIP-04 direct messages.
 
 ## Loggers
 
@@ -442,3 +495,117 @@ define command{
                     >> $USER4$/var/log/notificationforwarder_errors.log 2>&1
 }
 ```
+
+## Spool Flushing
+
+When forwarding to a recipient fails, the event is saved in a local sqlite database. The framework automatically attempts to re-submit these spooled events on every subsequent call to notificationforwarder for the same forwarder.
+
+The flush mechanism runs inside `forward_formatted()` for **every event** — not just heartbeats or special cases. Before submitting the current event, the framework checks if there are spooled events waiting. If there are, it attempts to flush them by calling `submit()` for each one, oldest first. Events that are successfully submitted are deleted from the spool. The first event that fails again ends the flush: it and all later events stay in the queue for the next attempt. A receiver that does not accept one event will most likely not accept the next ones either, so a flush costs at most one submit timeout instead of one per spooled event.
+
+If the receiver rejects a spooled event for good (for example because of an invalid payload), that event must not block the queue until `max_spool_minutes` have passed. When a flush stopped at an event and the current event is then accepted by the receiver, the receiver is evidently up. The stopped event is moved to the end of the queue and logged as "spooled event moved to the end of the spool". It keeps its original time, so it is still dropped when `max_spool_minutes` have passed, and the next flush can deliver the events behind it.
+
+This means that in a busy monitoring environment where notifications flow regularly, spooled events get drained naturally — each new notification triggers a flush attempt on its way through.
+
+### Delivery Order
+
+Spooled events are stored and re-submitted in the order in which they failed. As long as every delivery succeeds, the receiver gets the events in the order in which they were created. After delivery failures, this is not guaranteed. A newer event can reach the receiver before an older one, for example a RECOVERY before its PROBLEM:
+
+- While one notificationforwarder process is flushing the spool, another process submits its own new event directly, ahead of the spooled ones.
+- A spooled event was rejected while the receiver accepted the current event. The rejected event is moved to the end of the spool and is delivered, if at all, after newer events.
+- The flush stopped early, or `probe()` returned False. The process still submits its own new event directly.
+- A submit ran into the timeout, but the receiver got the event anyway. The event stays in the spool, and the receiver gets it a second time later, after newer events.
+
+If the receiver depends on the order, let the formatter include the event's creation time (`omd_originating_timestamp`, set for every event) in the payload. The receiver or its integration can then ignore an event that is older than the state it already has.
+
+### Probe Method
+
+Forwarders can implement an optional `probe()` method to optimize the flush behavior. The `probe()` method checks whether the receiving system is reachable *before* the framework attempts to flush spooled events via `submit()`.
+
+The framework uses `probe()` as a pre-check with three possible scenarios:
+
+|Forwarder has probe()?|probe() result|What happens|Why|
+|---|---|---|---|
+|No|—|flush proceeds|No way to check availability, must try|
+|Yes|True|flush proceeds|Receiver is confirmed reachable|
+|Yes|False|flush skipped|Receiver is known to be down, avoid wasted submit() calls|
+
+Without `probe()`, every flush attempt would call `submit()` for each spooled event even when the receiver is known to be down — wasting time and producing noisy error logs. With `probe()`, the framework can short-circuit: if the receiver is unreachable, the flush is skipped entirely and spooled events wait in the database until the next call.
+
+Example implementation:
+
+```python
+def probe(self):
+    r = requests.head(self.url)
+    return r.status_code == 200
+```
+
+### Heartbeat Events
+
+Formatters can mark an event as a heartbeat by setting `event.is_heartbeat = True` on the `FormattedEvent` object. Heartbeat events have two special properties:
+
+1. **They skip spooling on failure.** If a heartbeat event cannot be delivered, it is NOT saved to the spool database. This prevents periodic check-in events from accumulating in the retry queue.
+
+2. **They guarantee regular flush attempts.** While every event triggers the flush phase, heartbeat events ensure this happens at regular intervals — but only when scheduled. A cronjob or Nagios service using notificationforwarder as its check command provides this scheduling, creating a steady pulse of heartbeat events. Without such a schedule, heartbeats don't exist and spooled events could sit in the queue for hours waiting for the next real notification to trigger a flush.
+
+Example formatter that sets the heartbeat flag:
+
+```python
+class MyFormatter(NotificationFormatter):
+    def format_event(self, event):
+        notification_type = event.eventopts.get("NOTIFICATIONTYPE", "")
+        event.is_heartbeat = notification_type == "HEARTBEAT"
+
+        event.payload = {
+            "hostname": event.eventopts.get("HOSTNAME", ""),
+            "heartbeat": event.is_heartbeat,
+        }
+        event.summary = "heartbeat from {}".format(event.eventopts.get("HOSTNAME", ""))
+```
+
+### Scheduling Heartbeats in Nagios/Naemon
+
+Since notificationforwarder always exits with code 0, it can be used as a Nagios check command to schedule periodic heartbeats. The service will always be in OK status — it's using Nagios purely as a scheduling mechanism.
+
+**Command definition:**
+
+```
+define command {
+    command_name    notificationforwarder_heartbeat
+    command_line    $USER1$/notificationforwarder \
+                        --forwarder webhook \
+                        --forwarderopt url=https://xxx.service-now.com/api/now/import/u_notification \
+                        --formatter myformatter \
+                        --eventopt HOSTNAME='$HOSTNAME$' \
+                        --eventopt SERVICEDESC='$SERVICEDESC$' \
+                        --eventopt NOTIFICATIONTYPE=HEARTBEAT \
+                        --eventopt SERVICEOUTPUT='heartbeat from $HOSTNAME$' \
+                    >> $USER4$/var/log/notificationforwarder_errors.log 2>&1
+}
+```
+
+**Service definition:**
+
+```
+define service {
+    host_name               monitoring-server
+    service_description     notificationforwarder-heartbeat
+    check_command           notificationforwarder_heartbeat
+    max_check_attempts      1
+    check_interval          5
+    retry_interval          1
+    notification_interval   0
+    notification_options    n
+    notification_period     24x7
+}
+```
+
+The key settings:
+- `check_interval 5` — runs the heartbeat every 5 minutes
+- `notification_interval 0` — no Nagios-level notifications
+- `notification_options n` — don't send notifications on state changes
+
+The heartbeat goes through the normal notificationforwarder flow: the formatter sets `is_heartbeat = True`, the flush phase checks for spooled events (and calls `probe()` if available), and the heartbeat itself is submitted to the receiver. If the receiver accepts it, the receiving system knows the monitoring infrastructure is alive (dead-man switch). If the receiver is down, the heartbeat is simply dropped — it won't add to the spool queue.
+
+The two use cases for periodic heartbeats:
+- **Dead-man switch**: The receiving system (e.g., ServiceNow) gets a regular "I'm alive" signal. If heartbeats stop, it knows the monitoring system is down.
+- **Spool draining**: Even when nothing breaks, the heartbeat triggers flush attempts at regular intervals, so spooled events are delivered quickly once the receiver recovers — without waiting for the next real notification.

{omdnotificationforwarder-4.4 → omdnotificationforwarder-4.6}/bin/notificationforwarder

@@ -78,7 +78,7 @@ Example for an HTTP-based reporter:
                       help='Logger type: text (default) or json, or custom logger module',
                       default='text')
     parser.add_argument('--version', action='version',
-                      version=f'%(prog)s 4.4')
+                      version=f'%(prog)s 4.6')
 
     args = parser.parse_args()
     if not hasattr(args, 'formatter'):

{omdnotificationforwarder-4.4 → omdnotificationforwarder-4.6}/pyproject.toml

@@ -21,7 +21,7 @@ packages = ["src/notificationforwarder"]
 
 [project]
 name = "omdnotificationforwarder"
-version = "4.4"
+version = "4.6"
 authors = [
   { name="Gerhard Lausser", email="lausser@yahoo.com" },
 ]

{omdnotificationforwarder-4.4 → omdnotificationforwarder-4.6}/src/notificationforwarder/baseclass.py

@@ -330,6 +330,7 @@ class NotificationForwarder(object):
         return enrich_raw_event(raw_event)
 
     def forward_formatted(self, formatted_event):
+        self.flush_stopped_at = None
         try:
             """probe() checks if a forwarder is principally capable to submit
             an event. It is mostly used to contact an api and confirm that
@@ -376,6 +377,11 @@ class NotificationForwarder(object):
                     'formatted_event': formatted_event,
                     'status': 'success'
                 })
+            if formatted_event is not None and self.flush_stopped_at is not None:
+                # the receiver accepted this event, so it is up. the spooled
+                # event at which the flush stopped was rejected for its own
+                # reasons and must not block the spool until it expires.
+                self.move_stopped_event_to_end()
             return success
         return False
 
@@ -411,7 +417,38 @@ class NotificationForwarder(object):
     def acquire_lock_with_retry(self, lock_file, max_attempts=3, base_delay=0.1):
         return acquire_lock_with_retry(lock_file, logger, max_attempts=max_attempts, base_delay=base_delay)
 
+    def move_stopped_event_to_end(self):
+        event_id = self.flush_stopped_at
+        self.flush_stopped_at = None
+        try:
+            with open(self.db_lock_file, "w") as lock_file:
+                # a running flush could hold the row in memory, so move it
+                # only while no other process is flushing
+                if not self.acquire_lock_with_retry(lock_file, max_attempts=1):
+                    logger.debug("missed the flush lock, rejected event stays in place", {
+                        'event_id': event_id,
+                    })
+                    return
+                try:
+                    new_event_id = self.spool_store.move_to_end(event_id)
+                finally:
+                    import fcntl
+                    fcntl.lockf(lock_file, fcntl.LOCK_UN)
+            if new_event_id is not None:
+                logger.critical("spooled event moved to the end of the spool", {
+                    'event_id': event_id,
+                    'new_event_id': new_event_id,
+                    'action': 'moved_to_end',
+                })
+        except Exception as e:
+            logger.critical("could not move rejected spooled event", {
+                'event_id': event_id,
+                'exception': e,
+            })
+
     def flush(self):
+        # id of the spooled event at which the flush stopped, see move_stopped_event_to_end()
+        self.flush_stopped_at = None
         with open(self.db_lock_file, "w") as lock_file:
             locked = self.acquire_lock_with_retry(lock_file)
             if locked:
@@ -427,75 +464,82 @@ class NotificationForwarder(object):
                             'spooled_count': dropped,
                             'action': 'dropped'
                         })
-                    last_events_to_flush = 0
+                    # walk through the rows which existed when the flush started.
+                    # rows spooled meanwhile by concurrent processes are left for the
+                    # next flush and each row is tried at most once. the first failed
+                    # submit ends the flush, because a receiver which does not accept
+                    # one event will most likely not accept the next ones either and
+                    # every further attempt would cost another submit timeout.
+                    last_id = 0
+                    snapshot_max_id = self.spool_store.max_id()
                     while True:
-                        events_to_flush = self.num_spooled_events()
-                        if events_to_flush:
-                            logger.info("spooled events to be re-sent", {
-                                'spooled_count': events_to_flush,
-                                'action': 'resend'
-                            })
-                        else:
+                        id_events = self.spool_store.fetch_batch(after_id=last_id, max_id=snapshot_max_id)
+                        if not id_events:
                             logger.debug("nothing left to flush", {})
                             break
-                        if last_events_to_flush == events_to_flush:
-                            if events_to_flush != 0:
-                                logger.critical("spooled events could not be submitted", {
-                                    'spooled_count': last_events_to_flush,
-                                    'action': 'could_not_submit'
+                        logger.info("spooled events to be re-sent", {
+                            'spooled_count': len(id_events),
+                            'action': 'resend'
+                        })
+                        for id, text in id_events:
+                            last_id = id
+                            raw_event = self.spool_store.decode(text)
+                            formatted_event = self.format_event(raw_event)
+                            if formatted_event and formatted_event.is_discarded:
+                                deleted_trash += 1
+                                logger.info("discard spooled event during replay", {
+                                    'event_id': id,
+                                    'action': 'discard_during_replay'
                                 })
-                            break
-                        else:
-                            id_events = self.spool_store.fetch_batch()
-                            for id, text in id_events:
-                                raw_event = self.spool_store.decode(text)
-                                formatted_event = self.format_event(raw_event)
-                                if formatted_event and formatted_event.is_discarded:
-                                    deleted_trash += 1
-                                    logger.info("discard spooled event during replay", {
-                                        'event_id': id,
-                                        'action': 'discard_during_replay'
-                                    })
+                                self.spool_store.delete(id)
+                            elif formatted_event:
+                                replay_attempted += 1
+                                try:
+                                    result = self.submit(formatted_event)
+                                    success, _report_payload, error_message = apply_forward_result(result)
+                                except Exception as e:
+                                    success = False
+                                    error_message = str(e)
+                                if success:
                                     self.spool_store.delete(id)
-                                elif formatted_event:
-                                    replay_attempted += 1
-                                    try:
-                                        result = self.submit(formatted_event)
-                                        success, _report_payload, error_message = apply_forward_result(result)
-                                    except Exception as e:
-                                        success = False
-                                        error_message = str(e)
-                                    if success:
-                                        self.spool_store.delete(id)
-                                        replayed += 1
-                                        logger.info("delete spooled event", {
-                                            'spooled_count': 1,
-                                            'event_id': id,
-                                            'action': 'delete'
-                                        })
-                                    else:
-                                        stayed_in_spool += 1
-                                        context = {
-                                            'event_id': id,
-                                            'action': 'stays_in_spool'
-                                        }
-                                        if error_message:
-                                            context['exception'] = error_message
-                                        logger.critical("event stays in spool", context)
-                                else:
-                                    deleted_trash += 1
-                                    logger.critical("could not format spooled event", {
-                                        'raw_event': raw_event,
-                                        'event_id': id,
+                                    replayed += 1
+                                    logger.info("delete spooled event", {
                                         'spooled_count': 1,
-                                        'action': 'could_not_format'
-                                    })
-                                    self.spool_store.delete(id)
-                                    logger.info("delete trash event", {
                                         'event_id': id,
-                                        'action': 'delete_trash'
+                                        'action': 'delete'
                                     })
-                            last_events_to_flush = events_to_flush
+                                else:
+                                    stayed_in_spool += 1
+                                    context = {
+                                        'event_id': id,
+                                        'action': 'stays_in_spool'
+                                    }
+                                    if error_message:
+                                        context['exception'] = error_message
+                                    logger.critical("event stays in spool", context)
+                                    self.flush_stopped_at = id
+                                    break
+                            else:
+                                deleted_trash += 1
+                                logger.critical("could not format spooled event", {
+                                    'raw_event': raw_event,
+                                    'event_id': id,
+                                    'spooled_count': 1,
+                                    'action': 'could_not_format'
+                                })
+                                self.spool_store.delete(id)
+                                logger.info("delete trash event", {
+                                    'event_id': id,
+                                    'action': 'delete_trash'
+                                })
+                        if self.flush_stopped_at is not None:
+                            # only the rows this flush was responsible for, not
+                            # the ones spooled concurrently after the snapshot
+                            logger.critical("spooled events could not be submitted", {
+                                'spooled_count': self.spool_store.count(max_id=snapshot_max_id),
+                                'action': 'could_not_submit'
+                            })
+                            break
                     logger.info("spool replay summary", {
                         'attempted': replay_attempted,
                         'recovered_count': replayed,

{omdnotificationforwarder-4.4 → omdnotificationforwarder-4.6}/src/notificationforwarder/json/logger.py

@@ -121,6 +121,8 @@ class JsonLogger(NotificationLogger):
             msg_data["split_count"] = context['split_count']
         if 'event_id' in context:
             msg_data["event_id"] = context['event_id']
+        if 'new_event_id' in context:
+            msg_data["new_event_id"] = context['new_event_id']
 
         # Add action type
         if 'action' in context: