omdnotificationforwarder 4.3__tar.gz → 4.5__tar.gz

{omdnotificationforwarder-4.3 → omdnotificationforwarder-4.5}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: omdnotificationforwarder
-Version: 4.3
+Version: 4.5
 Summary: A framework for notification scripts for OMD
 Project-URL: Homepage, https://github.com/lausser/noteventificationforhandlerwarder
 Project-URL: Bug Tracker, https://github.com/lausser/noteventificationforhandlerwarder/issues
@@ -276,13 +276,18 @@ Install the optional Nostr support with `pip install notificationforwarder[nostr
 |nsec|bech32 private key for signing|required|
 |tags|extra Nostr tags to include|`monitoring` plus host/service/state tags|
 
-Example:
+The recipient is identified by a `p` tag containing the recipient's public key (npub). The `p` tag must be present in the tags list — the forwarder will raise an error if no recipient is found.
+
+#### Static recipient
+
+If all notifications go to the same recipient, pass the `p` tag via `--forwarderopt tags`:
 
 ```bash
 $USER1$/notificationforwarder \
     --forwarder nostr \
     --forwarderopt relays='wss://relay.damus.io,wss://nostr-pub.wellorder.net' \
     --forwarderopt nsec='$NSEC$' \
+    --forwarderopt tags='[["p", "npub1xyz..."]]' \
     --formatter nostr \
     --eventopt HOSTNAME='$HOSTNAME$' \
     --eventopt SERVICEDESC='$SERVICEDESC$' \
@@ -290,7 +295,55 @@ $USER1$/notificationforwarder \
     --eventopt SERVICEOUTPUT='$SERVICEOUTPUT$'
 ```
 
-The recipient `p` tag can be provided through the `tags` forwarder option or custom formatter output. The message content is encrypted before relay publication, so it shows up as a DM in clients that support NIP-04 direct messages.
+#### Per-contact recipient
+
+In a real monitoring setup, different contacts may have different Nostr public keys. Use a custom contact variable (e.g., `$CONTACTNOSTRNPUB$`) and pass it as an eventopt. The formatter then adds it as the `p` tag.
+
+**Contact definition:**
+
+```
+define contact {
+    contact_name    admin-team
+    ...
+    _NOSTRNPUB      npub1xyz...
+}
+```
+
+**Command definition:**
+
+```
+define command {
+    command_name    notify-service-nostr
+    command_line    $USER1$/notificationforwarder \
+                        --forwarder nostr \
+                        --forwarderopt relays='wss://relay.damus.io' \
+                        --forwarderopt nsec='$NSEC$' \
+                        --formatter nostr_custom \
+                        --eventopt HOSTNAME='$HOSTNAME$' \
+                        --eventopt SERVICEDESC='$SERVICEDESC$' \
+                        --eventopt SERVICESTATE='$SERVICESTATE$' \
+                        --eventopt SERVICEOUTPUT='$SERVICEOUTPUT$' \
+                        --eventopt NOSTR_PUBKEY='$_CONTACTNOSTRNPUB$' \
+                    >> $USER4$/var/log/notificationforwarder_errors.log 2>&1
+}
+```
+
+**Custom formatter** in `~/local/lib/python/notificationforwarder/nostr_custom/formatter.py`:
+
+```python
+from notificationforwarder.nostr.formatter import NostrFormatter
+
+class NostrCustomFormatter(NostrFormatter):
+    def format_event(self, event):
+        super().format_event(event)
+        pubkey = event.eventopts.get("NOSTR_PUBKEY", "")
+        if pubkey:
+            event.payload["tags"].append(["p", pubkey])
+```
+
+The `p` tag is appended to the tags list that the default Nostr formatter already builds with monitoring tags. The forwarder picks it up and uses it as the recipient for the encrypted DM.
+
+The message content is encrypted before relay publication, so it shows up as a DM in clients that support NIP-04 direct messages.
 
 ## Loggers
 
@@ -463,3 +516,104 @@ define command{
                     >> $USER4$/var/log/notificationforwarder_errors.log 2>&1
 }
 ```
+
+## Spool Flushing
+
+When forwarding to a recipient fails, the event is saved in a local sqlite database. The framework automatically attempts to re-submit these spooled events on every subsequent call to notificationforwarder for the same forwarder.
+
+The flush mechanism runs inside `forward_formatted()` for **every event** — not just heartbeats or special cases. Before submitting the current event, the framework checks if there are spooled events waiting. If there are, it attempts to flush them by calling `submit()` for each one. Events that are successfully submitted are deleted from the spool. Events that still fail stay in the queue for the next attempt.
+
+This means that in a busy monitoring environment where notifications flow regularly, spooled events get drained naturally — each new notification triggers a flush attempt on its way through.
+
+### Probe Method
+
+Forwarders can implement an optional `probe()` method to optimize the flush behavior. The `probe()` method checks whether the receiving system is reachable *before* the framework attempts to flush spooled events via `submit()`.
+
+The framework uses `probe()` as a pre-check with three possible scenarios:
+
+|Forwarder has probe()?|probe() result|What happens|Why|
+|---|---|---|---|
+|No|—|flush proceeds|No way to check availability, must try|
+|Yes|True|flush proceeds|Receiver is confirmed reachable|
+|Yes|False|flush skipped|Receiver is known to be down, avoid wasted submit() calls|
+
+Without `probe()`, every flush attempt would call `submit()` for each spooled event even when the receiver is known to be down — wasting time and producing noisy error logs. With `probe()`, the framework can short-circuit: if the receiver is unreachable, the flush is skipped entirely and spooled events wait in the database until the next call.
+
+Example implementation:
+
+```python
+def probe(self):
+    r = requests.head(self.url)
+    return r.status_code == 200
+```
+
+### Heartbeat Events
+
+Formatters can mark an event as a heartbeat by setting `event.is_heartbeat = True` on the `FormattedEvent` object. Heartbeat events have two special properties:
+
+1. **They skip spooling on failure.** If a heartbeat event cannot be delivered, it is NOT saved to the spool database. This prevents periodic check-in events from accumulating in the retry queue.
+
+2. **They guarantee regular flush attempts.** While every event triggers the flush phase, heartbeat events ensure this happens at regular intervals — but only when scheduled. A cronjob or Nagios service using notificationforwarder as its check command provides this scheduling, creating a steady pulse of heartbeat events. Without such a schedule, heartbeats don't exist and spooled events could sit in the queue for hours waiting for the next real notification to trigger a flush.
+
+Example formatter that sets the heartbeat flag:
+
+```python
+class MyFormatter(NotificationFormatter):
+    def format_event(self, event):
+        notification_type = event.eventopts.get("NOTIFICATIONTYPE", "")
+        event.is_heartbeat = notification_type == "HEARTBEAT"
+
+        event.payload = {
+            "hostname": event.eventopts.get("HOSTNAME", ""),
+            "heartbeat": event.is_heartbeat,
+        }
+        event.summary = "heartbeat from {}".format(event.eventopts.get("HOSTNAME", ""))
+```
+
+### Scheduling Heartbeats in Nagios/Naemon
+
+Since notificationforwarder always exits with code 0, it can be used as a Nagios check command to schedule periodic heartbeats. The service will always be in OK status — it's using Nagios purely as a scheduling mechanism.
+
+**Command definition:**
+
+```
+define command {
+    command_name    notificationforwarder_heartbeat
+    command_line    $USER1$/notificationforwarder \
+                        --forwarder webhook \
+                        --forwarderopt url=https://xxx.service-now.com/api/now/import/u_notification \
+                        --formatter myformatter \
+                        --eventopt HOSTNAME='$HOSTNAME$' \
+                        --eventopt SERVICEDESC='$SERVICEDESC$' \
+                        --eventopt NOTIFICATIONTYPE=HEARTBEAT \
+                        --eventopt SERVICEOUTPUT='heartbeat from $HOSTNAME$' \
+                    >> $USER4$/var/log/notificationforwarder_errors.log 2>&1
+}
+```
+
+**Service definition:**
+
+```
+define service {
+    host_name               monitoring-server
+    service_description     notificationforwarder-heartbeat
+    check_command           notificationforwarder_heartbeat
+    max_check_attempts      1
+    check_interval          5
+    retry_interval          1
+    notification_interval   0
+    notification_options    n
+    notification_period     24x7
+}
+```
+
+The key settings:
+- `check_interval 5` — runs the heartbeat every 5 minutes
+- `notification_interval 0` — no Nagios-level notifications
+- `notification_options n` — don't send notifications on state changes
+
+The heartbeat goes through the normal notificationforwarder flow: the formatter sets `is_heartbeat = True`, the flush phase checks for spooled events (and calls `probe()` if available), and the heartbeat itself is submitted to the receiver. If the receiver accepts it, the receiving system knows the monitoring infrastructure is alive (dead-man switch). If the receiver is down, the heartbeat is simply dropped — it won't add to the spool queue.
+
+The two use cases for periodic heartbeats:
+- **Dead-man switch**: The receiving system (e.g., ServiceNow) gets a regular "I'm alive" signal. If heartbeats stop, it knows the monitoring system is down.
+- **Spool draining**: Even when nothing breaks, the heartbeat triggers flush attempts at regular intervals, so spooled events are delivered quickly once the receiver recovers — without waiting for the next real notification.

{omdnotificationforwarder-4.3 → omdnotificationforwarder-4.5}/README.md

@@ -255,13 +255,18 @@ Install the optional Nostr support with `pip install notificationforwarder[nostr
 |nsec|bech32 private key for signing|required|
 |tags|extra Nostr tags to include|`monitoring` plus host/service/state tags|
 
-Example:
+The recipient is identified by a `p` tag containing the recipient's public key (npub). The `p` tag must be present in the tags list — the forwarder will raise an error if no recipient is found.
+
+#### Static recipient
+
+If all notifications go to the same recipient, pass the `p` tag via `--forwarderopt tags`:
 
 ```bash
 $USER1$/notificationforwarder \
     --forwarder nostr \
     --forwarderopt relays='wss://relay.damus.io,wss://nostr-pub.wellorder.net' \
     --forwarderopt nsec='$NSEC$' \
+    --forwarderopt tags='[["p", "npub1xyz..."]]' \
     --formatter nostr \
     --eventopt HOSTNAME='$HOSTNAME$' \
     --eventopt SERVICEDESC='$SERVICEDESC$' \
@@ -269,7 +274,55 @@ $USER1$/notificationforwarder \
     --eventopt SERVICEOUTPUT='$SERVICEOUTPUT$'
 ```
 
-The recipient `p` tag can be provided through the `tags` forwarder option or custom formatter output. The message content is encrypted before relay publication, so it shows up as a DM in clients that support NIP-04 direct messages.
+#### Per-contact recipient
+
+In a real monitoring setup, different contacts may have different Nostr public keys. Use a custom contact variable (e.g., `$CONTACTNOSTRNPUB$`) and pass it as an eventopt. The formatter then adds it as the `p` tag.
+
+**Contact definition:**
+
+```
+define contact {
+    contact_name    admin-team
+    ...
+    _NOSTRNPUB      npub1xyz...
+}
+```
+
+**Command definition:**
+
+```
+define command {
+    command_name    notify-service-nostr
+    command_line    $USER1$/notificationforwarder \
+                        --forwarder nostr \
+                        --forwarderopt relays='wss://relay.damus.io' \
+                        --forwarderopt nsec='$NSEC$' \
+                        --formatter nostr_custom \
+                        --eventopt HOSTNAME='$HOSTNAME$' \
+                        --eventopt SERVICEDESC='$SERVICEDESC$' \
+                        --eventopt SERVICESTATE='$SERVICESTATE$' \
+                        --eventopt SERVICEOUTPUT='$SERVICEOUTPUT$' \
+                        --eventopt NOSTR_PUBKEY='$_CONTACTNOSTRNPUB$' \
+                    >> $USER4$/var/log/notificationforwarder_errors.log 2>&1
+}
+```
+
+**Custom formatter** in `~/local/lib/python/notificationforwarder/nostr_custom/formatter.py`:
+
+```python
+from notificationforwarder.nostr.formatter import NostrFormatter
+
+class NostrCustomFormatter(NostrFormatter):
+    def format_event(self, event):
+        super().format_event(event)
+        pubkey = event.eventopts.get("NOSTR_PUBKEY", "")
+        if pubkey:
+            event.payload["tags"].append(["p", pubkey])
+```
+
+The `p` tag is appended to the tags list that the default Nostr formatter already builds with monitoring tags. The forwarder picks it up and uses it as the recipient for the encrypted DM.
+
+The message content is encrypted before relay publication, so it shows up as a DM in clients that support NIP-04 direct messages.
 
 ## Loggers
 
@@ -442,3 +495,104 @@ define command{
                     >> $USER4$/var/log/notificationforwarder_errors.log 2>&1
 }
 ```
+
+## Spool Flushing
+
+When forwarding to a recipient fails, the event is saved in a local sqlite database. The framework automatically attempts to re-submit these spooled events on every subsequent call to notificationforwarder for the same forwarder.
+
+The flush mechanism runs inside `forward_formatted()` for **every event** — not just heartbeats or special cases. Before submitting the current event, the framework checks if there are spooled events waiting. If there are, it attempts to flush them by calling `submit()` for each one. Events that are successfully submitted are deleted from the spool. Events that still fail stay in the queue for the next attempt.
+
+This means that in a busy monitoring environment where notifications flow regularly, spooled events get drained naturally — each new notification triggers a flush attempt on its way through.
+
+### Probe Method
+
+Forwarders can implement an optional `probe()` method to optimize the flush behavior. The `probe()` method checks whether the receiving system is reachable *before* the framework attempts to flush spooled events via `submit()`.
+
+The framework uses `probe()` as a pre-check with three possible scenarios:
+
+|Forwarder has probe()?|probe() result|What happens|Why|
+|---|---|---|---|
+|No|—|flush proceeds|No way to check availability, must try|
+|Yes|True|flush proceeds|Receiver is confirmed reachable|
+|Yes|False|flush skipped|Receiver is known to be down, avoid wasted submit() calls|
+
+Without `probe()`, every flush attempt would call `submit()` for each spooled event even when the receiver is known to be down — wasting time and producing noisy error logs. With `probe()`, the framework can short-circuit: if the receiver is unreachable, the flush is skipped entirely and spooled events wait in the database until the next call.
+
+Example implementation:
+
+```python
+def probe(self):
+    r = requests.head(self.url)
+    return r.status_code == 200
+```
+
+### Heartbeat Events
+
+Formatters can mark an event as a heartbeat by setting `event.is_heartbeat = True` on the `FormattedEvent` object. Heartbeat events have two special properties:
+
+1. **They skip spooling on failure.** If a heartbeat event cannot be delivered, it is NOT saved to the spool database. This prevents periodic check-in events from accumulating in the retry queue.
+
+2. **They guarantee regular flush attempts.** While every event triggers the flush phase, heartbeat events ensure this happens at regular intervals — but only when scheduled. A cronjob or Nagios service using notificationforwarder as its check command provides this scheduling, creating a steady pulse of heartbeat events. Without such a schedule, heartbeats don't exist and spooled events could sit in the queue for hours waiting for the next real notification to trigger a flush.
+
+Example formatter that sets the heartbeat flag:
+
+```python
+class MyFormatter(NotificationFormatter):
+    def format_event(self, event):
+        notification_type = event.eventopts.get("NOTIFICATIONTYPE", "")
+        event.is_heartbeat = notification_type == "HEARTBEAT"
+
+        event.payload = {
+            "hostname": event.eventopts.get("HOSTNAME", ""),
+            "heartbeat": event.is_heartbeat,
+        }
+        event.summary = "heartbeat from {}".format(event.eventopts.get("HOSTNAME", ""))
+```
+
+### Scheduling Heartbeats in Nagios/Naemon
+
+Since notificationforwarder always exits with code 0, it can be used as a Nagios check command to schedule periodic heartbeats. The service will always be in OK status — it's using Nagios purely as a scheduling mechanism.
+
+**Command definition:**
+
+```
+define command {
+    command_name    notificationforwarder_heartbeat
+    command_line    $USER1$/notificationforwarder \
+                        --forwarder webhook \
+                        --forwarderopt url=https://xxx.service-now.com/api/now/import/u_notification \
+                        --formatter myformatter \
+                        --eventopt HOSTNAME='$HOSTNAME$' \
+                        --eventopt SERVICEDESC='$SERVICEDESC$' \
+                        --eventopt NOTIFICATIONTYPE=HEARTBEAT \
+                        --eventopt SERVICEOUTPUT='heartbeat from $HOSTNAME$' \
+                    >> $USER4$/var/log/notificationforwarder_errors.log 2>&1
+}
+```
+
+**Service definition:**
+
+```
+define service {
+    host_name               monitoring-server
+    service_description     notificationforwarder-heartbeat
+    check_command           notificationforwarder_heartbeat
+    max_check_attempts      1
+    check_interval          5
+    retry_interval          1
+    notification_interval   0
+    notification_options    n
+    notification_period     24x7
+}
+```
+
+The key settings:
+- `check_interval 5` — runs the heartbeat every 5 minutes
+- `notification_interval 0` — no Nagios-level notifications
+- `notification_options n` — don't send notifications on state changes
+
+The heartbeat goes through the normal notificationforwarder flow: the formatter sets `is_heartbeat = True`, the flush phase checks for spooled events (and calls `probe()` if available), and the heartbeat itself is submitted to the receiver. If the receiver accepts it, the receiving system knows the monitoring infrastructure is alive (dead-man switch). If the receiver is down, the heartbeat is simply dropped — it won't add to the spool queue.
+
+The two use cases for periodic heartbeats:
+- **Dead-man switch**: The receiving system (e.g., ServiceNow) gets a regular "I'm alive" signal. If heartbeats stop, it knows the monitoring system is down.
+- **Spool draining**: Even when nothing breaks, the heartbeat triggers flush attempts at regular intervals, so spooled events are delivered quickly once the receiver recovers — without waiting for the next real notification.

{omdnotificationforwarder-4.3 → omdnotificationforwarder-4.5}/bin/notificationforwarder

@@ -78,7 +78,7 @@ Example for an HTTP-based reporter:
                       help='Logger type: text (default) or json, or custom logger module',
                       default='text')
     parser.add_argument('--version', action='version',
-                      version=f'%(prog)s 4.3')
+                      version=f'%(prog)s 4.5')
 
     args = parser.parse_args()
     if not hasattr(args, 'formatter'):

{omdnotificationforwarder-4.3 → omdnotificationforwarder-4.5}/pyproject.toml

@@ -21,7 +21,7 @@ packages = ["src/notificationforwarder"]
 
 [project]
 name = "omdnotificationforwarder"
-version = "4.3"
+version = "4.5"
 authors = [
   { name="Gerhard Lausser", email="lausser@yahoo.com" },
 ]

{omdnotificationforwarder-4.3 → omdnotificationforwarder-4.5}/src/notificationforwarder/baseclass.py

@@ -415,6 +415,7 @@ class NotificationForwarder(object):
         with open(self.db_lock_file, "w") as lock_file:
             locked = self.acquire_lock_with_retry(lock_file)
             if locked:
+                self.baseclass_logs_summary = True
                 try:
                     dropped = self.spool_store.prune_expired(self.max_spool_minutes)
                     replay_attempted = 0
@@ -426,68 +427,82 @@ class NotificationForwarder(object):
                             'spooled_count': dropped,
                             'action': 'dropped'
                         })
-                    last_events_to_flush = 0
+                    # walk through the rows which existed when the flush started.
+                    # rows spooled meanwhile by concurrent processes are left for the
+                    # next flush and each row is tried at most once, so the flush lock
+                    # is not held forever while the destination keeps failing.
+                    last_id = 0
+                    snapshot_max_id = self.spool_store.max_id()
                     while True:
-                        events_to_flush = self.num_spooled_events()
-                        if events_to_flush:
-                            logger.info("spooled events to be re-sent", {
-                                'spooled_count': events_to_flush,
-                                'action': 'resend'
-                            })
-                        else:
+                        id_events = self.spool_store.fetch_batch(after_id=last_id, max_id=snapshot_max_id)
+                        if not id_events:
                             logger.debug("nothing left to flush", {})
                             break
-                        if last_events_to_flush == events_to_flush:
-                            if events_to_flush != 0:
-                                logger.critical("spooled events could not be submitted", {
-                                    'spooled_count': last_events_to_flush,
-                                    'action': 'could_not_submit'
+                        logger.info("spooled events to be re-sent", {
+                            'spooled_count': len(id_events),
+                            'action': 'resend'
+                        })
+                        progress = False
+                        for id, text in id_events:
+                            last_id = id
+                            raw_event = self.spool_store.decode(text)
+                            formatted_event = self.format_event(raw_event)
+                            if formatted_event and formatted_event.is_discarded:
+                                deleted_trash += 1
+                                logger.info("discard spooled event during replay", {
+                                    'event_id': id,
+                                    'action': 'discard_during_replay'
                                 })
-                            break
-                        else:
-                            id_events = self.spool_store.fetch_batch()
-                            for id, text in id_events:
-                                raw_event = self.spool_store.decode(text)
-                                formatted_event = self.format_event(raw_event)
-                                if formatted_event:
-                                    replay_attempted += 1
-                                    try:
-                                        result = self.submit(formatted_event)
-                                        success, _report_payload, error_message = apply_forward_result(result)
-                                    except Exception as e:
-                                        success = False
-                                        error_message = str(e)
-                                    if success:
-                                        self.spool_store.delete(id)
-                                        replayed += 1
-                                        logger.info("delete spooled event", {
-                                            'spooled_count': 1,
-                                            'event_id': id,
-                                            'action': 'delete'
-                                        })
-                                    else:
-                                        stayed_in_spool += 1
-                                        context = {
-                                            'event_id': id,
-                                            'action': 'stays_in_spool'
-                                        }
-                                        if error_message:
-                                            context['exception'] = error_message
-                                        logger.critical("event stays in spool", context)
-                                else:
-                                    deleted_trash += 1
-                                    logger.critical("could not format spooled event", {
-                                        'raw_event': raw_event,
-                                        'event_id': id,
-                                        'spooled_count': 1,
-                                        'action': 'could_not_format'
-                                    })
+                                self.spool_store.delete(id)
+                                progress = True
+                            elif formatted_event:
+                                replay_attempted += 1
+                                try:
+                                    result = self.submit(formatted_event)
+                                    success, _report_payload, error_message = apply_forward_result(result)
+                                except Exception as e:
+                                    success = False
+                                    error_message = str(e)
+                                if success:
                                     self.spool_store.delete(id)
-                                    logger.info("delete trash event", {
+                                    progress = True
+                                    replayed += 1
+                                    logger.info("delete spooled event", {
+                                        'spooled_count': 1,
                                         'event_id': id,
-                                        'action': 'delete_trash'
+                                        'action': 'delete'
                                     })
-                            last_events_to_flush = events_to_flush
+                                else:
+                                    stayed_in_spool += 1
+                                    context = {
+                                        'event_id': id,
+                                        'action': 'stays_in_spool'
+                                    }
+                                    if error_message:
+                                        context['exception'] = error_message
+                                    logger.critical("event stays in spool", context)
+                            else:
+                                deleted_trash += 1
+                                logger.critical("could not format spooled event", {
+                                    'raw_event': raw_event,
+                                    'event_id': id,
+                                    'spooled_count': 1,
+                                    'action': 'could_not_format'
+                                })
+                                self.spool_store.delete(id)
+                                progress = True
+                                logger.info("delete trash event", {
+                                    'event_id': id,
+                                    'action': 'delete_trash'
+                                })
+                        if not progress:
+                            # only the rows this flush was responsible for, not
+                            # the ones spooled concurrently after the snapshot
+                            logger.critical("spooled events could not be submitted", {
+                                'spooled_count': self.spool_store.count(max_id=snapshot_max_id),
+                                'action': 'could_not_submit'
+                            })
+                            break
                     logger.info("spool replay summary", {
                         'attempted': replay_attempted,
                         'recovered_count': replayed,

{omdnotificationforwarder-4.3 → omdnotificationforwarder-4.5}/src/notificationforwarder/email/forwarder.py

@@ -5,6 +5,8 @@ from email.mime.text import MIMEText
 from email.mime.multipart import MIMEMultipart
 from notificationforwarder.baseclass import NotificationForwarder, NotificationFormatter, timeout
 
+logger = logging.getLogger("notificationforwarder.email")
+
 class EmailForwarder(NotificationForwarder):
     def __init__(self, opts):
         super(self.__class__, self).__init__(opts)
@@ -31,6 +33,6 @@ class EmailForwarder(NotificationForwarder):
             server.quit()
             return True
         except Exception as e:
-            logging.critical("sending mail failed: {}".format(str(e)))
+            logger.critical("sending mail failed: {}".format(str(e)))
             return False
 

omdnotificationforwarder-4.5/src/notificationforwarder/spool.py

@@ -0,0 +1,113 @@
+try:
+    import simplejson as json
+except ImportError:
+    import json
+import random
+import sqlite3
+import threading
+import time
+import fcntl
+
+
+class SpoolStore:
+    def __init__(self, db_file, table_name):
+        self.db_file = db_file
+        self.table_name = table_name
+        self.connection = None
+        self.cursor = None
+        self._lock = threading.Lock()
+
+    def open(self):
+        self.connection = sqlite3.connect(self.db_file, check_same_thread=False)
+        self.cursor = self.connection.cursor()
+
+    def init_db(self):
+        sql_create = """CREATE TABLE IF NOT EXISTS """ + self.table_name + """ (
+                id INTEGER PRIMARY KEY,
+                payload TEXT NOT NULL,
+                timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL
+            )"""
+        with self._lock:
+            self.cursor.execute(sql_create)
+            self.connection.commit()
+
+    def count(self, max_id=None):
+        sql_count = "SELECT COUNT(*) FROM " + self.table_name
+        params = []
+        if max_id is not None:
+            sql_count += " WHERE id <= ?"
+            params.append(max_id)
+        with self._lock:
+            self.cursor.execute(sql_count, params)
+            return self.cursor.fetchone()[0]
+
+    def enqueue(self, raw_event):
+        sql_insert = "INSERT INTO " + self.table_name + "(payload) VALUES (?)"
+        with self._lock:
+            self.cursor.execute(sql_insert, (json.dumps(raw_event),))
+            self.connection.commit()
+
+    def prune_expired(self, max_spool_minutes):
+        outdated = int(time.time() - 60 * max_spool_minutes)
+        sql_delete = "DELETE FROM " + self.table_name + " WHERE CAST(STRFTIME('%s', timestamp) AS INTEGER) < ?"
+        with self._lock:
+            self.cursor.execute(sql_delete, (outdated,))
+            dropped = self.cursor.rowcount
+            # commit immediately, otherwise the implicit transaction keeps
+            # the database write-locked for other processes during a flush
+            self.connection.commit()
+            return dropped
+
+    def max_id(self):
+        with self._lock:
+            self.cursor.execute("SELECT MAX(id) FROM " + self.table_name)
+            return self.cursor.fetchone()[0] or 0
+
+    def fetch_batch(self, limit=10, after_id=0, max_id=None):
+        sql_select = "SELECT id, payload FROM " + self.table_name + " WHERE id > ?"
+        params = [after_id]
+        if max_id is not None:
+            sql_select += " AND id <= ?"
+            params.append(max_id)
+        sql_select += " ORDER BY id LIMIT ?"
+        params.append(limit)
+        with self._lock:
+            self.cursor.execute(sql_select, params)
+            return self.cursor.fetchall()
+
+    def delete(self, event_id):
+        with self._lock:
+            self.cursor.execute("DELETE FROM " + self.table_name + " WHERE id = ?", (event_id,))
+            self.connection.commit()
+
+    def decode(self, text):
+        return json.loads(text)
+
+    def commit(self):
+        with self._lock:
+            self.connection.commit()
+
+    def close(self):
+        with self._lock:
+            if self.cursor:
+                self.cursor.close()
+            if self.connection:
+                self.connection.commit()
+                self.connection.close()
+
+
+def acquire_lock_with_retry(lock_file, app_logger, max_attempts=3, base_delay=0.1):
+    for attempt in range(max_attempts):
+        try:
+            fcntl.lockf(lock_file, fcntl.LOCK_EX | fcntl.LOCK_NB)
+            app_logger.debug("flush lock set", {})
+            return True
+        except IOError as exc:
+            app_logger.debug("flush lock failed", {
+                'attempt': attempt + 1,
+                'exception': exc,
+            })
+            if attempt < max_attempts - 1:
+                delay = base_delay * (2 ** attempt) + random.uniform(0, 0.1)
+                time.sleep(delay)
+    return False