omdnotificationforwarder 4.4__tar.gz → 4.5__tar.gz

{omdnotificationforwarder-4.4 → omdnotificationforwarder-4.5}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: omdnotificationforwarder
-Version: 4.4
+Version: 4.5
 Summary: A framework for notification scripts for OMD
 Project-URL: Homepage, https://github.com/lausser/noteventificationforhandlerwarder
 Project-URL: Bug Tracker, https://github.com/lausser/noteventificationforhandlerwarder/issues
@@ -276,13 +276,18 @@ Install the optional Nostr support with `pip install notificationforwarder[nostr
 |nsec|bech32 private key for signing|required|
 |tags|extra Nostr tags to include|`monitoring` plus host/service/state tags|
 
-Example:
+The recipient is identified by a `p` tag containing the recipient's public key (npub). The `p` tag must be present in the tags list — the forwarder will raise an error if no recipient is found.
+
+#### Static recipient
+
+If all notifications go to the same recipient, pass the `p` tag via `--forwarderopt tags`:
 
 ```bash
 $USER1$/notificationforwarder \
     --forwarder nostr \
     --forwarderopt relays='wss://relay.damus.io,wss://nostr-pub.wellorder.net' \
     --forwarderopt nsec='$NSEC$' \
+    --forwarderopt tags='[["p", "npub1xyz..."]]' \
     --formatter nostr \
     --eventopt HOSTNAME='$HOSTNAME$' \
     --eventopt SERVICEDESC='$SERVICEDESC$' \
@@ -290,7 +295,55 @@ $USER1$/notificationforwarder \
     --eventopt SERVICEOUTPUT='$SERVICEOUTPUT$'
 ```
 
-The recipient `p` tag can be provided through the `tags` forwarder option or custom formatter output. The message content is encrypted before relay publication, so it shows up as a DM in clients that support NIP-04 direct messages.
+#### Per-contact recipient
+
+In a real monitoring setup, different contacts may have different Nostr public keys. Use a custom contact variable (e.g., `$CONTACTNOSTRNPUB$`) and pass it as an eventopt. The formatter then adds it as the `p` tag.
+
+**Contact definition:**
+
+```
+define contact {
+    contact_name    admin-team
+    ...
+    _NOSTRNPUB      npub1xyz...
+}
+```
+
+**Command definition:**
+
+```
+define command {
+    command_name    notify-service-nostr
+    command_line    $USER1$/notificationforwarder \
+                        --forwarder nostr \
+                        --forwarderopt relays='wss://relay.damus.io' \
+                        --forwarderopt nsec='$NSEC$' \
+                        --formatter nostr_custom \
+                        --eventopt HOSTNAME='$HOSTNAME$' \
+                        --eventopt SERVICEDESC='$SERVICEDESC$' \
+                        --eventopt SERVICESTATE='$SERVICESTATE$' \
+                        --eventopt SERVICEOUTPUT='$SERVICEOUTPUT$' \
+                        --eventopt NOSTR_PUBKEY='$_CONTACTNOSTRNPUB$' \
+                    >> $USER4$/var/log/notificationforwarder_errors.log 2>&1
+}
+```
+
+**Custom formatter** in `~/local/lib/python/notificationforwarder/nostr_custom/formatter.py`:
+
+```python
+from notificationforwarder.nostr.formatter import NostrFormatter
+
+class NostrCustomFormatter(NostrFormatter):
+    def format_event(self, event):
+        super().format_event(event)
+        pubkey = event.eventopts.get("NOSTR_PUBKEY", "")
+        if pubkey:
+            event.payload["tags"].append(["p", pubkey])
+```
+
+The `p` tag is appended to the tags list that the default Nostr formatter already builds with monitoring tags. The forwarder picks it up and uses it as the recipient for the encrypted DM.
+
+The message content is encrypted before relay publication, so it shows up as a DM in clients that support NIP-04 direct messages.
 
 ## Loggers
 
@@ -463,3 +516,104 @@ define command{
                     >> $USER4$/var/log/notificationforwarder_errors.log 2>&1
 }
 ```
+
+## Spool Flushing
+
+When forwarding to a recipient fails, the event is saved in a local sqlite database. The framework automatically attempts to re-submit these spooled events on every subsequent call to notificationforwarder for the same forwarder.
+
+The flush mechanism runs inside `forward_formatted()` for **every event** — not just heartbeats or special cases. Before submitting the current event, the framework checks if there are spooled events waiting. If there are, it attempts to flush them by calling `submit()` for each one. Events that are successfully submitted are deleted from the spool. Events that still fail stay in the queue for the next attempt.
+
+This means that in a busy monitoring environment where notifications flow regularly, spooled events get drained naturally — each new notification triggers a flush attempt on its way through.
+
+### Probe Method
+
+Forwarders can implement an optional `probe()` method to optimize the flush behavior. The `probe()` method checks whether the receiving system is reachable *before* the framework attempts to flush spooled events via `submit()`.
+
+The framework uses `probe()` as a pre-check with three possible scenarios:
+
+|Forwarder has probe()?|probe() result|What happens|Why|
+|---|---|---|---|
+|No|—|flush proceeds|No way to check availability, must try|
+|Yes|True|flush proceeds|Receiver is confirmed reachable|
+|Yes|False|flush skipped|Receiver is known to be down, avoid wasted submit() calls|
+
+Without `probe()`, every flush attempt would call `submit()` for each spooled event even when the receiver is known to be down — wasting time and producing noisy error logs. With `probe()`, the framework can short-circuit: if the receiver is unreachable, the flush is skipped entirely and spooled events wait in the database until the next call.
+
+Example implementation:
+
+```python
+def probe(self):
+    r = requests.head(self.url)
+    return r.status_code == 200
+```
+
+### Heartbeat Events
+
+Formatters can mark an event as a heartbeat by setting `event.is_heartbeat = True` on the `FormattedEvent` object. Heartbeat events have two special properties:
+
+1. **They skip spooling on failure.** If a heartbeat event cannot be delivered, it is NOT saved to the spool database. This prevents periodic check-in events from accumulating in the retry queue.
+
+2. **They guarantee regular flush attempts.** While every event triggers the flush phase, heartbeat events ensure this happens at regular intervals — but only when scheduled. A cronjob or Nagios service using notificationforwarder as its check command provides this scheduling, creating a steady pulse of heartbeat events. Without such a schedule, heartbeats don't exist and spooled events could sit in the queue for hours waiting for the next real notification to trigger a flush.
+
+Example formatter that sets the heartbeat flag:
+
+```python
+class MyFormatter(NotificationFormatter):
+    def format_event(self, event):
+        notification_type = event.eventopts.get("NOTIFICATIONTYPE", "")
+        event.is_heartbeat = notification_type == "HEARTBEAT"
+
+        event.payload = {
+            "hostname": event.eventopts.get("HOSTNAME", ""),
+            "heartbeat": event.is_heartbeat,
+        }
+        event.summary = "heartbeat from {}".format(event.eventopts.get("HOSTNAME", ""))
+```
+
+### Scheduling Heartbeats in Nagios/Naemon
+
+Since notificationforwarder always exits with code 0, it can be used as a Nagios check command to schedule periodic heartbeats. The service will always be in OK status — it's using Nagios purely as a scheduling mechanism.
+
+**Command definition:**
+
+```
+define command {
+    command_name    notificationforwarder_heartbeat
+    command_line    $USER1$/notificationforwarder \
+                        --forwarder webhook \
+                        --forwarderopt url=https://xxx.service-now.com/api/now/import/u_notification \
+                        --formatter myformatter \
+                        --eventopt HOSTNAME='$HOSTNAME$' \
+                        --eventopt SERVICEDESC='$SERVICEDESC$' \
+                        --eventopt NOTIFICATIONTYPE=HEARTBEAT \
+                        --eventopt SERVICEOUTPUT='heartbeat from $HOSTNAME$' \
+                    >> $USER4$/var/log/notificationforwarder_errors.log 2>&1
+}
+```
+
+**Service definition:**
+
+```
+define service {
+    host_name               monitoring-server
+    service_description     notificationforwarder-heartbeat
+    check_command           notificationforwarder_heartbeat
+    max_check_attempts      1
+    check_interval          5
+    retry_interval          1
+    notification_interval   0
+    notification_options    n
+    notification_period     24x7
+}
+```
+
+The key settings:
+- `check_interval 5` — runs the heartbeat every 5 minutes
+- `notification_interval 0` — no Nagios-level notifications
+- `notification_options n` — don't send notifications on state changes
+
+The heartbeat goes through the normal notificationforwarder flow: the formatter sets `is_heartbeat = True`, the flush phase checks for spooled events (and calls `probe()` if available), and the heartbeat itself is submitted to the receiver. If the receiver accepts it, the receiving system knows the monitoring infrastructure is alive (dead-man switch). If the receiver is down, the heartbeat is simply dropped — it won't add to the spool queue.
+
+The two use cases for periodic heartbeats:
+- **Dead-man switch**: The receiving system (e.g., ServiceNow) gets a regular "I'm alive" signal. If heartbeats stop, it knows the monitoring system is down.
+- **Spool draining**: Even when nothing breaks, the heartbeat triggers flush attempts at regular intervals, so spooled events are delivered quickly once the receiver recovers — without waiting for the next real notification.

{omdnotificationforwarder-4.4 → omdnotificationforwarder-4.5}/README.md

@@ -255,13 +255,18 @@ Install the optional Nostr support with `pip install notificationforwarder[nostr
 |nsec|bech32 private key for signing|required|
 |tags|extra Nostr tags to include|`monitoring` plus host/service/state tags|
 
-Example:
+The recipient is identified by a `p` tag containing the recipient's public key (npub). The `p` tag must be present in the tags list — the forwarder will raise an error if no recipient is found.
+
+#### Static recipient
+
+If all notifications go to the same recipient, pass the `p` tag via `--forwarderopt tags`:
 
 ```bash
 $USER1$/notificationforwarder \
     --forwarder nostr \
     --forwarderopt relays='wss://relay.damus.io,wss://nostr-pub.wellorder.net' \
     --forwarderopt nsec='$NSEC$' \
+    --forwarderopt tags='[["p", "npub1xyz..."]]' \
     --formatter nostr \
     --eventopt HOSTNAME='$HOSTNAME$' \
     --eventopt SERVICEDESC='$SERVICEDESC$' \
@@ -269,7 +274,55 @@ $USER1$/notificationforwarder \
     --eventopt SERVICEOUTPUT='$SERVICEOUTPUT$'
 ```
 
-The recipient `p` tag can be provided through the `tags` forwarder option or custom formatter output. The message content is encrypted before relay publication, so it shows up as a DM in clients that support NIP-04 direct messages.
+#### Per-contact recipient
+
+In a real monitoring setup, different contacts may have different Nostr public keys. Use a custom contact variable (e.g., `$CONTACTNOSTRNPUB$`) and pass it as an eventopt. The formatter then adds it as the `p` tag.
+
+**Contact definition:**
+
+```
+define contact {
+    contact_name    admin-team
+    ...
+    _NOSTRNPUB      npub1xyz...
+}
+```
+
+**Command definition:**
+
+```
+define command {
+    command_name    notify-service-nostr
+    command_line    $USER1$/notificationforwarder \
+                        --forwarder nostr \
+                        --forwarderopt relays='wss://relay.damus.io' \
+                        --forwarderopt nsec='$NSEC$' \
+                        --formatter nostr_custom \
+                        --eventopt HOSTNAME='$HOSTNAME$' \
+                        --eventopt SERVICEDESC='$SERVICEDESC$' \
+                        --eventopt SERVICESTATE='$SERVICESTATE$' \
+                        --eventopt SERVICEOUTPUT='$SERVICEOUTPUT$' \
+                        --eventopt NOSTR_PUBKEY='$_CONTACTNOSTRNPUB$' \
+                    >> $USER4$/var/log/notificationforwarder_errors.log 2>&1
+}
+```
+
+**Custom formatter** in `~/local/lib/python/notificationforwarder/nostr_custom/formatter.py`:
+
+```python
+from notificationforwarder.nostr.formatter import NostrFormatter
+
+class NostrCustomFormatter(NostrFormatter):
+    def format_event(self, event):
+        super().format_event(event)
+        pubkey = event.eventopts.get("NOSTR_PUBKEY", "")
+        if pubkey:
+            event.payload["tags"].append(["p", pubkey])
+```
+
+The `p` tag is appended to the tags list that the default Nostr formatter already builds with monitoring tags. The forwarder picks it up and uses it as the recipient for the encrypted DM.
+
+The message content is encrypted before relay publication, so it shows up as a DM in clients that support NIP-04 direct messages.
 
 ## Loggers
 
@@ -442,3 +495,104 @@ define command{
                     >> $USER4$/var/log/notificationforwarder_errors.log 2>&1
 }
 ```
+
+## Spool Flushing
+
+When forwarding to a recipient fails, the event is saved in a local sqlite database. The framework automatically attempts to re-submit these spooled events on every subsequent call to notificationforwarder for the same forwarder.
+
+The flush mechanism runs inside `forward_formatted()` for **every event** — not just heartbeats or special cases. Before submitting the current event, the framework checks if there are spooled events waiting. If there are, it attempts to flush them by calling `submit()` for each one. Events that are successfully submitted are deleted from the spool. Events that still fail stay in the queue for the next attempt.
+
+This means that in a busy monitoring environment where notifications flow regularly, spooled events get drained naturally — each new notification triggers a flush attempt on its way through.
+
+### Probe Method
+
+Forwarders can implement an optional `probe()` method to optimize the flush behavior. The `probe()` method checks whether the receiving system is reachable *before* the framework attempts to flush spooled events via `submit()`.
+
+The framework uses `probe()` as a pre-check with three possible scenarios:
+
+|Forwarder has probe()?|probe() result|What happens|Why|
+|---|---|---|---|
+|No|—|flush proceeds|No way to check availability, must try|
+|Yes|True|flush proceeds|Receiver is confirmed reachable|
+|Yes|False|flush skipped|Receiver is known to be down, avoid wasted submit() calls|
+
+Without `probe()`, every flush attempt would call `submit()` for each spooled event even when the receiver is known to be down — wasting time and producing noisy error logs. With `probe()`, the framework can short-circuit: if the receiver is unreachable, the flush is skipped entirely and spooled events wait in the database until the next call.
+
+Example implementation:
+
+```python
+def probe(self):
+    r = requests.head(self.url)
+    return r.status_code == 200
+```
+
+### Heartbeat Events
+
+Formatters can mark an event as a heartbeat by setting `event.is_heartbeat = True` on the `FormattedEvent` object. Heartbeat events have two special properties:
+
+1. **They skip spooling on failure.** If a heartbeat event cannot be delivered, it is NOT saved to the spool database. This prevents periodic check-in events from accumulating in the retry queue.
+
+2. **They guarantee regular flush attempts.** While every event triggers the flush phase, heartbeat events ensure this happens at regular intervals — but only when scheduled. A cronjob or Nagios service using notificationforwarder as its check command provides this scheduling, creating a steady pulse of heartbeat events. Without such a schedule, heartbeats don't exist and spooled events could sit in the queue for hours waiting for the next real notification to trigger a flush.
+
+Example formatter that sets the heartbeat flag:
+
+```python
+class MyFormatter(NotificationFormatter):
+    def format_event(self, event):
+        notification_type = event.eventopts.get("NOTIFICATIONTYPE", "")
+        event.is_heartbeat = notification_type == "HEARTBEAT"
+
+        event.payload = {
+            "hostname": event.eventopts.get("HOSTNAME", ""),
+            "heartbeat": event.is_heartbeat,
+        }
+        event.summary = "heartbeat from {}".format(event.eventopts.get("HOSTNAME", ""))
+```
+
+### Scheduling Heartbeats in Nagios/Naemon
+
+Since notificationforwarder always exits with code 0, it can be used as a Nagios check command to schedule periodic heartbeats. The service will always be in OK status — it's using Nagios purely as a scheduling mechanism.
+
+**Command definition:**
+
+```
+define command {
+    command_name    notificationforwarder_heartbeat
+    command_line    $USER1$/notificationforwarder \
+                        --forwarder webhook \
+                        --forwarderopt url=https://xxx.service-now.com/api/now/import/u_notification \
+                        --formatter myformatter \
+                        --eventopt HOSTNAME='$HOSTNAME$' \
+                        --eventopt SERVICEDESC='$SERVICEDESC$' \
+                        --eventopt NOTIFICATIONTYPE=HEARTBEAT \
+                        --eventopt SERVICEOUTPUT='heartbeat from $HOSTNAME$' \
+                    >> $USER4$/var/log/notificationforwarder_errors.log 2>&1
+}
+```
+
+**Service definition:**
+
+```
+define service {
+    host_name               monitoring-server
+    service_description     notificationforwarder-heartbeat
+    check_command           notificationforwarder_heartbeat
+    max_check_attempts      1
+    check_interval          5
+    retry_interval          1
+    notification_interval   0
+    notification_options    n
+    notification_period     24x7
+}
+```
+
+The key settings:
+- `check_interval 5` — runs the heartbeat every 5 minutes
+- `notification_interval 0` — no Nagios-level notifications
+- `notification_options n` — don't send notifications on state changes
+
+The heartbeat goes through the normal notificationforwarder flow: the formatter sets `is_heartbeat = True`, the flush phase checks for spooled events (and calls `probe()` if available), and the heartbeat itself is submitted to the receiver. If the receiver accepts it, the receiving system knows the monitoring infrastructure is alive (dead-man switch). If the receiver is down, the heartbeat is simply dropped — it won't add to the spool queue.
+
+The two use cases for periodic heartbeats:
+- **Dead-man switch**: The receiving system (e.g., ServiceNow) gets a regular "I'm alive" signal. If heartbeats stop, it knows the monitoring system is down.
+- **Spool draining**: Even when nothing breaks, the heartbeat triggers flush attempts at regular intervals, so spooled events are delivered quickly once the receiver recovers — without waiting for the next real notification.

{omdnotificationforwarder-4.4 → omdnotificationforwarder-4.5}/bin/notificationforwarder

@@ -78,7 +78,7 @@ Example for an HTTP-based reporter:
                       help='Logger type: text (default) or json, or custom logger module',
                       default='text')
     parser.add_argument('--version', action='version',
-                      version=f'%(prog)s 4.4')
+                      version=f'%(prog)s 4.5')
 
     args = parser.parse_args()
     if not hasattr(args, 'formatter'):

{omdnotificationforwarder-4.4 → omdnotificationforwarder-4.5}/pyproject.toml

@@ -21,7 +21,7 @@ packages = ["src/notificationforwarder"]
 
 [project]
 name = "omdnotificationforwarder"
-version = "4.4"
+version = "4.5"
 authors = [
   { name="Gerhard Lausser", email="lausser@yahoo.com" },
 ]

{omdnotificationforwarder-4.4 → omdnotificationforwarder-4.5}/src/notificationforwarder/baseclass.py

@@ -427,75 +427,82 @@ class NotificationForwarder(object):
                             'spooled_count': dropped,
                             'action': 'dropped'
                         })
-                    last_events_to_flush = 0
+                    # walk through the rows which existed when the flush started.
+                    # rows spooled meanwhile by concurrent processes are left for the
+                    # next flush and each row is tried at most once, so the flush lock
+                    # is not held forever while the destination keeps failing.
+                    last_id = 0
+                    snapshot_max_id = self.spool_store.max_id()
                     while True:
-                        events_to_flush = self.num_spooled_events()
-                        if events_to_flush:
-                            logger.info("spooled events to be re-sent", {
-                                'spooled_count': events_to_flush,
-                                'action': 'resend'
-                            })
-                        else:
+                        id_events = self.spool_store.fetch_batch(after_id=last_id, max_id=snapshot_max_id)
+                        if not id_events:
                             logger.debug("nothing left to flush", {})
                             break
-                        if last_events_to_flush == events_to_flush:
-                            if events_to_flush != 0:
-                                logger.critical("spooled events could not be submitted", {
-                                    'spooled_count': last_events_to_flush,
-                                    'action': 'could_not_submit'
+                        logger.info("spooled events to be re-sent", {
+                            'spooled_count': len(id_events),
+                            'action': 'resend'
+                        })
+                        progress = False
+                        for id, text in id_events:
+                            last_id = id
+                            raw_event = self.spool_store.decode(text)
+                            formatted_event = self.format_event(raw_event)
+                            if formatted_event and formatted_event.is_discarded:
+                                deleted_trash += 1
+                                logger.info("discard spooled event during replay", {
+                                    'event_id': id,
+                                    'action': 'discard_during_replay'
                                 })
-                            break
-                        else:
-                            id_events = self.spool_store.fetch_batch()
-                            for id, text in id_events:
-                                raw_event = self.spool_store.decode(text)
-                                formatted_event = self.format_event(raw_event)
-                                if formatted_event and formatted_event.is_discarded:
-                                    deleted_trash += 1
-                                    logger.info("discard spooled event during replay", {
-                                        'event_id': id,
-                                        'action': 'discard_during_replay'
-                                    })
+                                self.spool_store.delete(id)
+                                progress = True
+                            elif formatted_event:
+                                replay_attempted += 1
+                                try:
+                                    result = self.submit(formatted_event)
+                                    success, _report_payload, error_message = apply_forward_result(result)
+                                except Exception as e:
+                                    success = False
+                                    error_message = str(e)
+                                if success:
                                     self.spool_store.delete(id)
-                                elif formatted_event:
-                                    replay_attempted += 1
-                                    try:
-                                        result = self.submit(formatted_event)
-                                        success, _report_payload, error_message = apply_forward_result(result)
-                                    except Exception as e:
-                                        success = False
-                                        error_message = str(e)
-                                    if success:
-                                        self.spool_store.delete(id)
-                                        replayed += 1
-                                        logger.info("delete spooled event", {
-                                            'spooled_count': 1,
-                                            'event_id': id,
-                                            'action': 'delete'
-                                        })
-                                    else:
-                                        stayed_in_spool += 1
-                                        context = {
-                                            'event_id': id,
-                                            'action': 'stays_in_spool'
-                                        }
-                                        if error_message:
-                                            context['exception'] = error_message
-                                        logger.critical("event stays in spool", context)
-                                else:
-                                    deleted_trash += 1
-                                    logger.critical("could not format spooled event", {
-                                        'raw_event': raw_event,
-                                        'event_id': id,
+                                    progress = True
+                                    replayed += 1
+                                    logger.info("delete spooled event", {
                                         'spooled_count': 1,
-                                        'action': 'could_not_format'
-                                    })
-                                    self.spool_store.delete(id)
-                                    logger.info("delete trash event", {
                                         'event_id': id,
-                                        'action': 'delete_trash'
+                                        'action': 'delete'
                                     })
-                            last_events_to_flush = events_to_flush
+                                else:
+                                    stayed_in_spool += 1
+                                    context = {
+                                        'event_id': id,
+                                        'action': 'stays_in_spool'
+                                    }
+                                    if error_message:
+                                        context['exception'] = error_message
+                                    logger.critical("event stays in spool", context)
+                            else:
+                                deleted_trash += 1
+                                logger.critical("could not format spooled event", {
+                                    'raw_event': raw_event,
+                                    'event_id': id,
+                                    'spooled_count': 1,
+                                    'action': 'could_not_format'
+                                })
+                                self.spool_store.delete(id)
+                                progress = True
+                                logger.info("delete trash event", {
+                                    'event_id': id,
+                                    'action': 'delete_trash'
+                                })
+                        if not progress:
+                            # only the rows this flush was responsible for, not
+                            # the ones spooled concurrently after the snapshot
+                            logger.critical("spooled events could not be submitted", {
+                                'spooled_count': self.spool_store.count(max_id=snapshot_max_id),
+                                'action': 'could_not_submit'
+                            })
+                            break
                     logger.info("spool replay summary", {
                         'attempted': replay_attempted,
                         'recovered_count': replayed,

{omdnotificationforwarder-4.4 → omdnotificationforwarder-4.5}/src/notificationforwarder/spool.py

@@ -31,9 +31,14 @@ class SpoolStore:
             self.cursor.execute(sql_create)
             self.connection.commit()
 
-    def count(self):
+    def count(self, max_id=None):
+        sql_count = "SELECT COUNT(*) FROM " + self.table_name
+        params = []
+        if max_id is not None:
+            sql_count += " WHERE id <= ?"
+            params.append(max_id)
         with self._lock:
-            self.cursor.execute("SELECT COUNT(*) FROM " + self.table_name)
+            self.cursor.execute(sql_count, params)
             return self.cursor.fetchone()[0]
 
     def enqueue(self, raw_event):
@@ -47,12 +52,27 @@ class SpoolStore:
         sql_delete = "DELETE FROM " + self.table_name + " WHERE CAST(STRFTIME('%s', timestamp) AS INTEGER) < ?"
         with self._lock:
             self.cursor.execute(sql_delete, (outdated,))
-            return self.cursor.rowcount
+            dropped = self.cursor.rowcount
+            # commit immediately, otherwise the implicit transaction keeps
+            # the database write-locked for other processes during a flush
+            self.connection.commit()
+            return dropped
+
+    def max_id(self):
+        with self._lock:
+            self.cursor.execute("SELECT MAX(id) FROM " + self.table_name)
+            return self.cursor.fetchone()[0] or 0
 
-    def fetch_batch(self, limit=10):
-        sql_select = "SELECT id, payload FROM " + self.table_name + " ORDER BY id LIMIT ?"
+    def fetch_batch(self, limit=10, after_id=0, max_id=None):
+        sql_select = "SELECT id, payload FROM " + self.table_name + " WHERE id > ?"
+        params = [after_id]
+        if max_id is not None:
+            sql_select += " AND id <= ?"
+            params.append(max_id)
+        sql_select += " ORDER BY id LIMIT ?"
+        params.append(limit)
         with self._lock:
-            self.cursor.execute(sql_select, (limit,))
+            self.cursor.execute(sql_select, params)
             return self.cursor.fetchall()
 
     def delete(self, event_id):

{omdnotificationforwarder-4.4 → omdnotificationforwarder-4.5}/tests/TESTS.md

@@ -170,6 +170,7 @@ Integration tests for `load_forwarder()` using the `pythonpath/lib/python` test
 | `test_persistent_failure_keeps_row_and_terminates` | `submit()` always returning `False` → event stays in spool, stuck detection fires | Persistent failures are not silently dropped; the system logs and stops retrying. |
 | `test_spool_init_failure_returns_false` | `SpoolStore.enqueue` raising `sqlite3.OperationalError` → returns `False` with "database error" log | Database errors are caught and logged, not raised. |
 | `test_flush_beyond_batch_limit_stuck_detection` | 12 spooled events (exceeds batch limit of 10) with all submits failing → stuck detection, all events stay | The stuck detection logic handles batch boundaries correctly. |
+| `test_flush_terminates_while_other_processes_spool` | All submits fail while a second connection spools a new event after each submit → flush stops after the first batch, events 0–9 submitted once each, all 22 rows kept, stuck log reports `spooled_count` 12 (snapshot rows, not the live queue) | A flush is bounded to the rows present at its start; concurrent spooling cannot keep it (and the flush lock) alive. |
 | `test_formatter_returns_none_during_replay_deletes_trash` | `format_event` returning `None` during replay → event deleted as trash | Un-formattable events are cleaned up, not retried forever. |
 | `test_summary_logging_resumes_after_flush` | `baseclass_logs_summary` is reset to `True` at flush start | The no_more_logging flag is reset so operators see summary logs after recovery. |
 
@@ -186,6 +187,7 @@ Low-level `SpoolStore` operations against a real SQLite database.
 | `test_fetch_batch_limit_zero` | `fetch_batch(limit=0)` returns `[]` | Zero limit is handled. |
 | `test_fetch_batch_limit_exceeds_total` | Limit exceeding total returns all rows | No error when limit is too high. |
 | `test_prune_expired_boundary` | `prune_expired(5)` deletes only events strictly older than 5 minutes | Boundary conditions are correct. |
+| `test_prune_expired_does_not_hold_write_lock` | After `prune_expired()`, a second connection with a 0.1 s busy timeout can insert; `count(max_id=1)` counts only rows up to that id | Pruning commits immediately, so other processes can spool while a flush replays. |
 | `test_enqueue_non_serializable_data` | Enqueuing a `set` raises `TypeError` | Non-JSON-serializable data is rejected. |
 | `test_decode_round_trip` | JSON encode → decode produces the original dict | Serialization preserves data. |
 | `test_concurrent_enqueue` | Two threads each enqueuing 10 events concurrently → 20 rows, no corruption | Thread safety of concurrent writes. |

{omdnotificationforwarder-4.4 → omdnotificationforwarder-4.5}/tests/test_resilience_spool.py

@@ -98,6 +98,41 @@ class TestHeartbeatSpoolPolicy:
         # Heartbeat events should not be spooled
         assert len(spooled) == 0
 
+    def test_heartbeat_alternating_success_and_failure_does_not_grow_queue(self, setup, monkeypatch):
+        """Heartbeat events must not increase the spool queue, even when delivery fails."""
+        forwarder = _new_forwarder(
+            "example",
+            forwarder_name="webhook",
+            forwarder_opts={"url": "http://example.invalid/api"},
+        )
+
+        results = iter([True, False, True, False])
+        monkeypatch.setattr(forwarder, "submit", lambda fe: next(results))
+
+        def make_heartbeat_event(seq):
+            event = _formatted_event({"description": f"heartbeat {seq}"})
+            event.is_heartbeat = True
+            event.payload = {"seq": seq}
+            event.summary = f"heartbeat {seq}"
+            return event
+
+        events = [
+            {"description": "heartbeat 1"},
+            {"description": "heartbeat 2"},
+            {"description": "heartbeat 3"},
+            {"description": "heartbeat 4"},
+        ]
+
+        monkeypatch.setattr(
+            forwarder,
+            "format_event",
+            lambda raw: make_heartbeat_event(raw["description"]),
+        )
+
+        for event in events:
+            forwarder.forward(event)
+            assert forwarder.num_spooled_events() == 0
+
     def test_webhook_non_heartbeat_failure_spools(self, setup, monkeypatch):
         """Non-heartbeat events are spooled on failure."""
         forwarder = _new_forwarder(
@@ -562,12 +597,80 @@ class TestSpoolReplayEdges:
         # All 12 events should still be in spool (all failed)
         assert forwarder.num_spooled_events() == 12
 
-        # Stuck detection should have triggered after 2 iterations (10 fetched, then 2 remaining = unchanged)
+        # Stuck detection should have triggered after the first batch (10 fetched, none replayed)
         assert any("could not be submitted" in str(c) for c in critical_calls)
 
         # spool replay summary should be logged
         assert any("spool replay summary" in str(c) for c in info_calls)
 
+    def test_flush_terminates_while_other_processes_spool(self, setup, monkeypatch):
+        """Flush must not loop forever when concurrent processes keep spooling.
+
+        Regression: stuck detection compared the total spool count, which
+        other processes change by spooling, so the same failing head batch
+        was retried endlessly while the flush lock was held.
+        """
+        forwarder = _new_forwarder(
+            "example",
+            forwarder_name="webhook",
+            forwarder_opts={"url": "http://example.invalid/api"},
+        )
+        for i in range(12):
+            forwarder.spool_store.enqueue({"HOSTNAME": "h1", "event_num": i})
+
+        # a second connection plays a concurrent notificationforwarder process
+        other_process = SpoolStore(forwarder.db_file, forwarder.table_name)
+        other_process.open()
+
+        def format_ok(raw):
+            fe = _formatted_event(raw)
+            fe.payload = {"data": "test"}
+            fe.summary = "test"
+            return fe
+
+        submitted = []
+
+        def fail_while_others_spool(formatted_event):
+            submitted.append(formatted_event.eventopts["event_num"])
+            other_process.enqueue({"HOSTNAME": "h1", "event_num": 1000 + len(submitted)})
+            if len(submitted) > 100:
+                raise SystemExit("flush does not terminate")
+            return False
+
+        monkeypatch.setattr(forwarder, "format_event", format_ok)
+        monkeypatch.setattr(forwarder, "submit", fail_while_others_spool)
+
+        critical_calls = []
+        original_logger = baseclass.logger
+
+        class MockLogger:
+            def critical(self, msg, context=None, *args, **kwargs):
+                critical_calls.append((msg, context))
+
+            def info(self, msg, *args, **kwargs):
+                pass
+
+            def warning(self, msg, *args, **kwargs):
+                pass
+
+            def debug(self, msg, *args, **kwargs):
+                pass
+
+        baseclass.logger = MockLogger()
+        try:
+            forwarder.flush()
+        finally:
+            baseclass.logger = original_logger
+            other_process.close()
+
+        # one batch failed completely -> stop, every event tried at most once
+        assert submitted == list(range(10))
+        # nothing lost: 12 original + 10 spooled concurrently
+        assert forwarder.num_spooled_events() == 22
+        # the stuck log counts only the rows of this flush's snapshot
+        stuck = [ctx for msg, ctx in critical_calls if msg == "spooled events could not be submitted"]
+        assert stuck == [{'spooled_count': 12, 'action': 'could_not_submit'}]
+
     def test_formatter_returns_none_during_replay_deletes_trash(self, setup, monkeypatch):
         """Formatter returning None during replay deletes event as trash."""
         forwarder = _new_forwarder(
@@ -773,6 +876,31 @@ class TestSpoolStore:
 
         store.close()
 
+    def test_prune_expired_does_not_hold_write_lock(self, setup):
+        """After prune_expired another process must be able to spool.
+
+        Regression: the DELETE was not committed, so the implicit transaction
+        kept the database write-locked during the whole flush.
+        """
+        db_path = os.path.join(OMD_ROOT, "tmp", "test_spool_prune_lock.db")
+        store = SpoolStore(db_path, "test_events")
+        store.open()
+        store.init_db()
+        store.enqueue({"event": "keep"})
+
+        store.prune_expired(5)
+
+        other_process = sqlite3.connect(db_path, timeout=0.1)
+        try:
+            other_process.execute("INSERT INTO test_events(payload) VALUES (?)", ('{"event": "other"}',))
+            other_process.commit()
+        finally:
+            other_process.close()
+
+        assert store.count() == 2
+        assert store.count(max_id=1) == 1
+        store.close()
+
     def test_enqueue_non_serializable_data(self, setup):
         """SpoolStore.enqueue with non-serializable data raises."""
         import json as json_mod

{omdnotificationforwarder-4.4 → omdnotificationforwarder-4.5}/tests/test_runtime_foundation.py

@@ -77,6 +77,23 @@ def test_runtime_config_normalizes_forwarder_options(runtime_setup):
     del os.environ["NOTIFICATIONFORWARDER_MAX_SPOOL_MINUTES"]
 
 
+def test_env_var_max_spool_minutes_used_as_fallback(runtime_setup):
+    os.environ["NOTIFICATIONFORWARDER_MAX_SPOOL_MINUTES"] = "15"
+
+    config = RuntimeConfig.from_inputs(
+        "example",
+        None,
+        "example",
+        verbose=False,
+        debug=False,
+        forwarder_opts={},
+    )
+
+    assert config.max_spool_minutes == 15
+
+    del os.environ["NOTIFICATIONFORWARDER_MAX_SPOOL_MINUTES"]
+
+
 def test_resolve_component_supports_derived_and_explicit_names():
     assert resolve_component("json", "Logger") == ("json", "JsonLogger", "derived-class")
     assert resolve_component("custom.module.LoggerClass", "Logger") == (