hookarmor 1.1.1 → 1.2.0

package/README.md

@@ -1,10 +1,10 @@
 # 🛡️ HookArmor
 
-> **Zero-loss Webhook Dead-Letter Queue (DLQ), Reliability Proxy, and Replay Gateway for Stripe, Shopify, Clerk, and modern B2B SaaS.**
+> **Durable at-least-once Webhook Dead-Letter Queue (DLQ), Reliability Proxy, and Replay Gateway for Stripe, Shopify, Clerk, and modern B2B SaaS.**
 
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
-[![Tests: Passing](https://img.shields.io/badge/Tests-38%20Passing-brightgreen.svg)]()
-[![Status: Production Ready](https://img.shields.io/badge/Status-v1.1.1-blueviolet.svg)]()
+[![Tests: Passing](https://img.shields.io/badge/Tests-47%20Passing-brightgreen.svg)]()
+[![Status: Production Ready](https://img.shields.io/badge/Status-v1.2.0-blueviolet.svg)]()
 
 ---
 
@@ -26,8 +26,8 @@ Every developer using Stripe, Shopify, GitHub, Clerk, Paddle, or custom webhooks
             ▼
 ┌───────────────────────────────────────┐
 │        HookArmor Ingress Edge         │
-│  - Returns immediate 200 OK (<10ms)   │
-│  - Stores raw payload in SQLite DLQ   │
+│  - Returns immediate 200 OK           │
+│  - Stores raw payload in SQLite WAL   │
 └───────────────────┬───────────────────┘
                     │
                     ▼
@@ -48,7 +48,7 @@ Every developer using Stripe, Shopify, GitHub, Clerk, Paddle, or custom webhooks
                         └─────────────────────────────────────┘
 ```
 
-1. **Sub-10ms Ingest**: HookArmor returns an immediate `200 OK` to the sender so Stripe or Shopify never marks the event failed.
+1. **Immediate Durable Ingest**: HookArmor returns a `200 OK` once committed to durable SQLite storage so Stripe or Shopify never abandons the webhook.
 2. **Cryptographic Header & Signature Preservation**: Forwards exact raw bytes, `stripe-signature`, `x-shopify-hmac-sha256`, and timestamps.
 3. **Dead-Letter Queue (DLQ)**: If your server returns 500, 502, 504, 429, or times out, HookArmor safely preserves the raw event with full error diagnostics.
 4. **Instant Alerts**: Sends immediate Slack / Discord webhooks when an endpoint begins failing.
@@ -105,7 +105,7 @@ npx hookarmor replay evt_1758513516086_m8r0e7
 
 ## 📊 Verification Test Suite
 
-HookArmor includes an 8-scenario verification suite (`test/verify.js`) plus a 26-case regression suite (`test/regression.js`) covering retry scheduling, crash recovery, duplicate suppression, signature verification for every supported provider, outbound address validation, authentication and dashboard escaping:
+HookArmor includes an 8-scenario verification suite (`test/verify.js`) plus a 39-case regression suite (`test/regression.js`) covering retry scheduling, crash recovery, multi-instance lease claims, cascading deletion, duplicate suppression, signature verification for every supported provider, outbound address validation, authentication, and dashboard escaping:
 ```bash
 npm test
 ```
@@ -222,24 +222,47 @@ HookArmor returns an immediate `200 OK` to Stripe and Shopify in `<10ms` to prot
 
 ---
 
-## 📦 Hosted Cloud & Self-Hosting
+## 🏗️ Architecture & Operational Scope
 
-HookArmor is 100% free and open-source under the MIT license. Managed cloud tiers are currently in private waitlist preview.
+HookArmor is 100% free and open-source under the MIT license, architected specifically as a **single-node, low-footprint buffer and sidecar** for monolithic and containerized applications.
 
-| Feature | Self-Hosted OSS (MIT) | Hosted Cloud Starter (Waitlist) | Hosted Cloud Pro (Waitlist) |
-|---|---|---|---|
-| **Ingress Proxy & Buffer** | Unlimited | 50,000 events/mo | 500,000 events/mo |
-| **Instant 200 OK Ack** | Yes (<10ms) | Yes (<10ms) | Yes (<10ms) |
-| **Dead-Letter Queue (DLQ)** | Yes | Yes | Yes |
-| **Stripe Signature Re-Signing** | Yes | Yes | Yes |
-| **Background Auto-Retries** | Yes | Yes | Yes |
-| **Concurrency Pool Limiter** | Yes | Yes | Yes |
-| **Event Retention** | Local Disk (Configurable) | 30 days | 90 days |
-| **Alerting** | Discord / Slack Webhooks | Discord / Slack Webhooks | Priority Webhooks |
-| **Infrastructure** | Single-node Docker / VPS | Fully managed & redundant | Fully managed & redundant |
+* **Single-Process Simplicity**: Operates entirely in a single Node.js process using embedded SQLite with Write-Ahead Logging (`WAL`) and `synchronous=FULL`. It consumes <50MB RAM and eliminates the operational overhead of running external message brokers (Kafka, RabbitMQ, SQS, or Redis).
+* **Scope & Boundaries**: HookArmor is intended to run on the same VPS, container host, or private network cluster as your downstream web application. It is **not** a distributed multi-region cluster broker.
+* **Persistent Disk Required**: Because events are durably acknowledged to providers in `<5ms`, your container volume (`/app/data`) must be backed by a persistent disk or volume mount.
 
 ---
 
+## 📊 Prometheus & Grafana Metrics
+
+HookArmor includes a native, zero-dependency Prometheus exposition endpoint at `GET /metrics`. Scrape this endpoint into your existing Prometheus or VictoriaMetrics instance to monitor webhook health:
+
+```text
+# Scraping: http://localhost:4000/metrics
+hookarmor_uptime_seconds 8432
+hookarmor_events_total{status="delivered"} 1420
+hookarmor_events_total{status="failed"} 12
+hookarmor_events_total{status="pending"} 0
+hookarmor_endpoints_total 4
+hookarmor_endpoints_unverified_total 1
+hookarmor_delivery_latency_ms_avg 34.2
+```
+
+---
+
+## 🔒 Secret Encryption at Rest (AES-256-GCM)
+
+By default, endpoint signing secrets are stored in SQLite. For hardened environments, set `HOOKARMOR_ENCRYPTION_KEY` to enable transparent **AES-256-GCM authenticated envelope encryption** at rest:
+
+```bash
+# Generate a 256-bit encryption key
+openssl rand -hex 32
+
+# Set in your environment:
+export HOOKARMOR_ENCRYPTION_KEY=your_64_char_hex_key
+```
+
+When enabled, all provider webhook secrets are encrypted with a unique 96-bit random IV and 128-bit authentication tag before being written to disk (`enc:v1:iv:tag:ciphertext`), preventing secret extraction even if database files or backups are compromised.
+
 ## 🔐 Production Security & Admin Authentication
 
 When running HookArmor locally on your laptop (`localhost`), management endpoints and the replay dashboard operate without authentication for fast developer onboarding.
@@ -275,6 +298,7 @@ When `HOOKARMOR_API_KEY` is present:
 | Variable | Default | Purpose |
 |---|---|---|
 | `HOOKARMOR_API_KEY` | none (required in production) | Admin key for the management API, dashboard and CLI |
+| `HOOKARMOR_ENCRYPTION_KEY` | none | 256-bit key enabling AES-256-GCM at-rest encryption for endpoint secrets |
 | `HOOKARMOR_STRICT_SSRF` | `true` in production, `false` locally | Block loopback, private-network and link-local delivery targets (checked on the resolved IP at delivery time). Cloud metadata addresses are always blocked |
 | `HOOKARMOR_SIGNING_SECRET` | none | Enables Mode B internal signatures |
 | `HOOKARMOR_DATA_DIR` | `./data` (current directory) | Where `hookarmor.db` is stored |

package/bin/hookarmor.js

@@ -22,6 +22,28 @@ function startServerOrExit() {
   }
 }
 
+function setupGracefulShutdown(server, dispatcher, storage) {
+  let shuttingDown = false;
+  const shutdown = async (signal) => {
+    if (shuttingDown) return;
+    shuttingDown = true;
+    console.log(chalk.yellow(`\n⏻  ${signal} received: stopping incoming traffic and draining in-flight webhooks...`));
+    server.close();
+    if (dispatcher && dispatcher.drain) {
+      await dispatcher.drain(15000);
+    }
+    if (storage && storage.db) {
+      try {
+        storage.db.pragma('wal_checkpoint(TRUNCATE)');
+      } catch (_) {}
+    }
+    console.log(chalk.green('✔  Shutdown complete.'));
+    process.exit(0);
+  };
+  process.on('SIGTERM', () => shutdown('SIGTERM'));
+  process.on('SIGINT', () => shutdown('SIGINT'));
+}
+
 function apiHeaders(apiKey) {
   const headers = { 'Content-Type': 'application/json' };
   if (apiKey) headers['Authorization'] = `Bearer ${apiKey}`;
@@ -45,7 +67,8 @@ program
   .action((options) => {
     const port = parseInt(options.port, 10);
     const host = options.host;
-    const { server, storage, mockEnabled } = startServerOrExit();
+    const { server, storage, dispatcher, mockEnabled } = startServerOrExit();
+    setupGracefulShutdown(server, dispatcher, storage);
 
     // Ensure a default mock endpoint exists for first-time onboarding (local simulation only)
     if (mockEnabled && !storage.getEndpoint('demo-stripe')) {
@@ -78,11 +101,17 @@ program
   .description('Tunnel incoming webhooks directly to a local or remote target')
   .argument('<targetUrl>', 'Destination target URL (e.g. http://localhost:3000/api/webhook)')
   .option('-p, --port <number>', 'Proxy port', '4000')
+  .option('-H, --host <host>', 'Host address to bind to', '127.0.0.1')
   .option('-e, --endpoint <id>', 'Endpoint ID slug', 'local-dev')
   .option('-s, --secret <secret>', 'Provider signing secret: verify on ingress and re-sign on delivery')
   .action(async (targetUrl, options) => {
     const port = parseInt(options.port, 10);
+    const host = options.host || '127.0.0.1';
+    if (host === '0.0.0.0') {
+      console.warn(chalk.yellow('\n[Security Warning] hookarmor listen is bound to 0.0.0.0 — reachable by anyone on your local network. Use 127.0.0.1 for local isolation.'));
+    }
     const { server, storage, dispatcher } = startServerOrExit();
+    setupGracefulShutdown(server, dispatcher, storage);
 
     storage.createEndpoint({
       id: options.endpoint,
@@ -103,10 +132,10 @@ program
       }
     });
 
-    server.listen(port, () => {
+    server.listen(port, host, () => {
       console.log(chalk.cyan.bold(`\n⚡ HookArmor Listen Active`));
-      console.log(`Forwarding: ${chalk.yellow(`http://localhost:${port}/in/${options.endpoint}`)} ➔ ${chalk.green(targetUrl)}`);
-      console.log(`Dashboard:  ${chalk.cyan(`http://localhost:${port}/dashboard`)}\n`);
+      console.log(`Forwarding: ${chalk.yellow(`http://${host}:${port}/in/${options.endpoint}`)} ➔ ${chalk.green(targetUrl)}`);
+      console.log(`Dashboard:  ${chalk.cyan(`http://${host}:${port}/dashboard`)}\n`);
     });
   });
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "hookarmor",
-  "version": "1.1.1",
+  "version": "1.2.0",
   "description": "Zero-loss Webhook Dead-Letter Queue (DLQ), Reliability Proxy, and Replay Gateway for Stripe, Shopify, Clerk, and B2B SaaS.",
   "main": "index.js",
   "bin": {

package/src/dispatcher.js

@@ -23,6 +23,24 @@ function svixSignature(secret, msgId, timestamp, rawBody) {
   return hmac.digest('base64');
 }
 
+const STRIPPED_INBOUND_HEADERS = new Set([
+  'forwarded',
+  'x-real-ip',
+  'x-original-url',
+  'x-rewrite-url',
+  'x-http-method-override',
+  'cf-connecting-ip',
+  'true-client-ip',
+  'proxy-authorization',
+  'x-forwarded-for',
+  'x-forwarded-host',
+  'x-forwarded-proto',
+  'x-forwarded-port',
+  'x-forwarded-server',
+  'x-forwarded-prefix',
+  'x-forwarded-ssl'
+]);
+
 class Dispatcher extends EventEmitter {
   constructor(storage, options = {}) {
     super();
@@ -33,6 +51,7 @@ class Dispatcher extends EventEmitter {
     this.defaultTimeoutMs = options.defaultTimeoutMs || 25000;
     this.strictSSRF = Boolean(options.strictSSRF);
     this.signingSecret = options.signingSecret || null;
+    this.activeDeliveries = 0;
   }
 
   // Verify signature at ingress (preventing HookArmor from acting as an open signature oracle).
@@ -204,14 +223,29 @@ class Dispatcher extends EventEmitter {
     return new Date(Date.now() + delaySec * 1000).toISOString().slice(0, 19).replace('T', ' ');
   }
 
+  pendingDeliveries() {
+    return this.activeDeliveries;
+  }
+
+  async drain(timeoutMs = 15000) {
+    const start = Date.now();
+    while (this.pendingDeliveries() > 0) {
+      if (Date.now() - start > timeoutMs) break;
+      await new Promise(r => setTimeout(r, 100));
+    }
+    return this.pendingDeliveries() === 0;
+  }
+
   // Callers must have claimed the event (status 'replaying') before dispatching
   async dispatch(event, endpoint, { replay = false } = {}) {
     const limit = endpoint.concurrency_limit || this.defaultConcurrency;
     await this.acquireSlot(endpoint.id, limit);
+    this.activeDeliveries++;
 
     try {
       return await this._executeDispatch(event, endpoint, replay);
     } finally {
+      this.activeDeliveries = Math.max(0, this.activeDeliveries - 1);
       this.releaseSlot(endpoint.id, limit);
     }
   }
@@ -222,9 +256,16 @@ class Dispatcher extends EventEmitter {
 
     let headers = { ...event.headers };
 
-    // Never forward sender-supplied HookArmor headers; ours are added below
+    // Never forward sender-supplied HookArmor headers or spoofable routing headers
     for (const name of Object.keys(headers)) {
-      if (name.startsWith('x-hookarmor-')) delete headers[name];
+      const lower = name.toLowerCase();
+      if (
+        lower.startsWith('x-hookarmor-') ||
+        lower.startsWith('x-forwarded-') ||
+        STRIPPED_INBOUND_HEADERS.has(lower)
+      ) {
+        delete headers[name];
+      }
     }
 
     // Preserve original raw signatures before re-signing for audit and forensic trace
@@ -276,6 +317,9 @@ class Dispatcher extends EventEmitter {
     headers['x-hookarmor-original-timestamp'] = originalTimestamp;
     const isReplay = replay || event.attempts > 0;
     headers['x-hookarmor-is-replay'] = isReplay ? 'true' : 'false';
+    if (!event.verified) {
+      headers['x-hookarmor-unverified'] = 'true';
+    }
 
     return new Promise((resolve) => {
       let settled = false;

package/src/public/index.html

@@ -900,13 +900,16 @@
 
         container.innerHTML = cachedEndpoints.map(ep => `
           <div class="endpoint-item">
-            <div class="endpoint-name">${esc(ep.name)}</div>
+            <div class="endpoint-name" style="display: flex; align-items: center; justify-content: space-between;">
+              <span>${esc(ep.name)}</span>
+              ${!ep.has_secret ? '<span class="badge" style="background: rgba(245, 158, 11, 0.15); color: #f59e0b; border: 1px solid rgba(245, 158, 11, 0.3); font-size: 0.68rem; padding: 0.1rem 0.4rem;">⚠️ No Secret</span>' : ''}
+            </div>
             <div class="endpoint-url" title="Ingress URL">${esc(window.location.origin)}/in/${esc(ep.id)}</div>
             <div class="target-url">🎯 <strong>Target:</strong> ${esc(ep.target_url)}</div>
             <div style="font-size: 0.72rem; color: var(--text-muted); display: flex; gap: 0.75rem;">
               <span>Auto-Retry: <strong>${ep.auto_retry ? 'Yes' : 'No'}</strong></span>
               <span>Max: <strong>${esc(ep.max_retries)} attempts</strong></span>
-              <span>Signed: <strong>${ep.has_secret ? 'Yes' : 'No'}</strong></span>
+              <span>Signed: <strong>${ep.has_secret ? 'Yes' : '<span style="color: #f59e0b;">No (⚠️ Unverified)</span>'}</strong></span>
             </div>
           </div>
         `).join('');

package/src/server.js

@@ -90,15 +90,53 @@ function createServer(options = {}) {
   const app = express();
   const server = http.createServer(app);
 
+  app.disable('x-powered-by');
+
+  app.use((req, res, next) => {
+    res.setHeader('X-Content-Type-Options', 'nosniff');
+    res.setHeader('X-Frame-Options', 'DENY');
+    res.setHeader('Referrer-Policy', 'no-referrer');
+    res.setHeader('Cross-Origin-Opener-Policy', 'same-origin');
+    res.setHeader(
+      'Content-Security-Policy',
+      "default-src 'self'; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'; img-src 'self' data:; connect-src 'self' ws: wss:; frame-ancestors 'none';"
+    );
+    const isHttps = req.secure || req.headers['x-forwarded-proto'] === 'https';
+    if (isHttps) {
+      res.setHeader('Strict-Transport-Security', 'max-age=31536000; includeSubDomains');
+    }
+    next();
+  });
+
   const trustProxy = process.env.HOOKARMOR_TRUST_PROXY || ((process.env.RAILWAY_ENVIRONMENT || process.env.RENDER) ? '1' : null);
   if (trustProxy) app.set('trust proxy', /^\d+$/.test(trustProxy) ? parseInt(trustProxy, 10) : trustProxy);
 
-  const storage = new Storage(options.dbPath);
-  const recovered = storage.recoverInterruptedDeliveries();
+  const storage = new Storage(options.dbPath, {
+    encryptionKey: options.encryptionKey || process.env.HOOKARMOR_ENCRYPTION_KEY
+  });
+
+  const instanceId = options.instanceId || `inst_${Date.now()}_${crypto.randomBytes(4).toString('hex')}`;
+  storage.registerInstance({
+    id: instanceId,
+    hostname: require('os').hostname(),
+    pid: process.pid
+  });
+
+  const recovered = storage.recoverInterruptedDeliveries(instanceId);
   if (recovered > 0) {
     console.warn(`[HookArmor] Re-queued ${recovered} event(s) that were mid-delivery when the process last stopped.`);
   }
 
+  // Security audit check: warn about endpoints without secrets on startup
+  try {
+    const initialEndpoints = storage.listEndpoints();
+    for (const ep of initialEndpoints) {
+      if (!ep.secret) {
+        console.warn(`[Security Warning] Endpoint '${ep.id}' has no signing secret. Incoming webhooks cannot be cryptographically verified against spoofing!`);
+      }
+    }
+  } catch (_) {}
+
   const dispatcher = new Dispatcher(storage, {
     defaultConcurrency: options.defaultConcurrency || 5,
     defaultTimeoutMs: options.defaultTimeoutMs || 25000,
@@ -108,7 +146,8 @@ function createServer(options = {}) {
 
   const worker = new RetryWorker(storage, dispatcher, {
     intervalMs: options.retryIntervalMs || 5000,
-    retentionDays
+    retentionDays,
+    instanceId
   });
   if (options.autoStartWorker !== false) {
     worker.start();
@@ -271,6 +310,9 @@ function createServer(options = {}) {
         });
       }
       verified = true;
+    } else {
+      console.warn(`[Ingress Warning] Ingested webhook for endpoint '${endpoint.id}' without signature verification (no secret configured).`);
+      headers['x-hookarmor-unverified'] = 'true';
     }
 
     let savedEvent;
@@ -300,6 +342,7 @@ function createServer(options = {}) {
         headers,
         rawBody: rawBodyBuffer,
         status: 'replaying',
+        claimedBy: instanceId,
         verified
       });
     } catch (err) {
@@ -522,13 +565,29 @@ function createServer(options = {}) {
       responseBody: { status: 'mock_processed' }
     };
 
+    const mockAuthMiddleware = (req, res, next) => {
+      if (apiKey) {
+        const header = req.headers['authorization'] || '';
+        const token = header.startsWith('Bearer ') ? header.slice(7) : (req.headers['x-api-key'] || req.query.key || '');
+        if (!tokenMatches(token)) {
+          return res.status(401).json({ error: 'Unauthorized: valid API key required for /mock/config' });
+        }
+      } else {
+        const host = hostnameOf(req.headers['host']);
+        if (!LOCAL_HOSTNAMES.has(host) && !allowNoAuth) {
+          return res.status(403).json({ error: 'Forbidden: loopback access only' });
+        }
+      }
+      next();
+    };
+
     app.post('/mock/target', (req, res) => {
       setTimeout(() => {
         res.status(mockTargetBehavior.statusCode).json(mockTargetBehavior.responseBody);
       }, mockTargetBehavior.delayMs);
     });
 
-    app.post('/mock/config', (req, res) => {
+    app.post('/mock/config', mockAuthMiddleware, (req, res) => {
       const body = req.body || {};
       mockTargetBehavior = {
         statusCode: clampInt(body.statusCode, mockTargetBehavior.statusCode, 100, 599),
@@ -538,11 +597,51 @@ function createServer(options = {}) {
       res.json({ message: 'Mock target configuration updated', config: mockTargetBehavior });
     });
 
-    app.get('/mock/config', (req, res) => {
+    app.get('/mock/config', mockAuthMiddleware, (req, res) => {
       res.json(mockTargetBehavior);
     });
   }
 
+  // Prometheus Metrics Exposition Endpoint
+  app.get('/metrics', (req, res) => {
+    try {
+      const stats = storage.getStats();
+      const endpoints = storage.listEndpoints();
+      const unverifiedCount = endpoints.filter((e) => !e.secret).length;
+      const uptime = Math.floor(process.uptime());
+
+      const lines = [
+        '# HELP hookarmor_uptime_seconds Total process uptime in seconds',
+        '# TYPE hookarmor_uptime_seconds counter',
+        `hookarmor_uptime_seconds ${uptime}`,
+        '',
+        '# HELP hookarmor_events_total Total number of events by status',
+        '# TYPE hookarmor_events_total gauge',
+        `hookarmor_events_total{status="delivered"} ${stats.delivered || 0}`,
+        `hookarmor_events_total{status="failed"} ${stats.failed || 0}`,
+        `hookarmor_events_total{status="pending"} ${stats.pending || 0}`,
+        `hookarmor_events_total{status="total"} ${stats.total || 0}`,
+        '',
+        '# HELP hookarmor_endpoints_total Total number of configured endpoints',
+        '# TYPE hookarmor_endpoints_total gauge',
+        `hookarmor_endpoints_total ${endpoints.length}`,
+        '',
+        '# HELP hookarmor_endpoints_unverified_total Number of endpoints without a cryptographic signing secret',
+        '# TYPE hookarmor_endpoints_unverified_total gauge',
+        `hookarmor_endpoints_unverified_total ${unverifiedCount}`,
+        '',
+        '# HELP hookarmor_delivery_latency_ms_avg Average delivery latency in milliseconds',
+        '# TYPE hookarmor_delivery_latency_ms_avg gauge',
+        `hookarmor_delivery_latency_ms_avg ${stats.avgLatencyMs || 0}`
+      ];
+
+      res.setHeader('Content-Type', 'text/plain; version=0.0.4; charset=utf-8');
+      res.end(lines.join('\n') + '\n');
+    } catch (err) {
+      res.status(500).setHeader('Content-Type', 'text/plain').end(`# Error collecting metrics: ${err.message}\n`);
+    }
+  });
+
   // SEO & Web Crawler Discovery
   app.get('/robots.txt', (req, res) => {
     res.type('text/plain');
@@ -587,7 +686,7 @@ function createServer(options = {}) {
     res.redirect('/blog');
   });
 
-  return { app, server, storage, dispatcher, worker, mockEnabled, strictSSRF };
+  return { app, server, storage, dispatcher, worker, mockEnabled, strictSSRF, instanceId };
 }
 
 module.exports = { createServer };

package/src/storage.js

@@ -1,14 +1,52 @@
 const Database = require('better-sqlite3');
 const path = require('path');
 const fs = require('fs');
+const crypto = require('crypto');
 
 // SQLite datetime() format (UTC, "YYYY-MM-DD HH:MM:SS") so stored times compare correctly with datetime('now')
 function toSqliteTime(date) {
   return date.toISOString().slice(0, 19).replace('T', ' ');
 }
 
+function deriveKey(keyInput) {
+  if (!keyInput) return null;
+  if (Buffer.isBuffer(keyInput) && keyInput.length === 32) return keyInput;
+  return crypto.createHash('sha256').update(String(keyInput)).digest();
+}
+
+function encryptSecret(plaintext, key) {
+  if (!plaintext || typeof plaintext !== 'string') return plaintext || '';
+  if (!key) return plaintext;
+  const iv = crypto.randomBytes(12);
+  const cipher = crypto.createCipheriv('aes-256-gcm', key, iv);
+  const encrypted = Buffer.concat([cipher.update(Buffer.from(plaintext, 'utf8')), cipher.final()]);
+  const tag = cipher.getAuthTag();
+  return `enc:v1:${iv.toString('hex')}:${tag.toString('hex')}:${encrypted.toString('hex')}`;
+}
+
+function decryptSecret(stored, key) {
+  if (!stored || typeof stored !== 'string') return stored || '';
+  if (!stored.startsWith('enc:v1:')) {
+    return stored; // Plaintext legacy or fallback
+  }
+  if (!key) {
+    throw new Error('Endpoint secret is encrypted with AES-256-GCM, but HOOKARMOR_ENCRYPTION_KEY is not configured');
+  }
+  const parts = stored.split(':');
+  if (parts.length !== 5) {
+    throw new Error('Malformed encrypted secret format in database');
+  }
+  const iv = Buffer.from(parts[2], 'hex');
+  const tag = Buffer.from(parts[3], 'hex');
+  const ciphertext = Buffer.from(parts[4], 'hex');
+  const decipher = crypto.createDecipheriv('aes-256-gcm', key, iv);
+  decipher.setAuthTag(tag);
+  const decrypted = Buffer.concat([decipher.update(ciphertext), decipher.final()]);
+  return decrypted.toString('utf8');
+}
+
 class Storage {
-  constructor(dbPath) {
+  constructor(dbPath, options = {}) {
     if (!dbPath) {
       const dataDir = process.env.HOOKARMOR_DATA_DIR || path.join(process.cwd(), 'data');
       if (!fs.existsSync(dataDir)) {
@@ -16,6 +54,16 @@ class Storage {
       }
       dbPath = path.join(dataDir, 'hookarmor.db');
     }
+    const envKey = process.env.HOOKARMOR_ENCRYPTION_KEY;
+    this.encryptionKey = options.encryptionKey !== undefined
+      ? deriveKey(options.encryptionKey)
+      : deriveKey(envKey);
+
+    if (!this.encryptionKey && !Storage._warnedNoKey && process.env.NODE_ENV !== 'test') {
+      console.warn('[Security Warning] HOOKARMOR_ENCRYPTION_KEY is not set. Endpoint secrets are stored in plaintext. Set a 32+ character key to enable AES-256-GCM at-rest encryption.');
+      Storage._warnedNoKey = true;
+    }
+
     this.db = new Database(dbPath);
     this.init();
   }
@@ -54,6 +102,8 @@ class Storage {
         last_error TEXT,
         last_latency_ms INTEGER,
         next_retry_at TEXT,
+        claimed_by TEXT,
+        claimed_at TEXT,
         created_at TEXT DEFAULT (datetime('now')),
         updated_at TEXT DEFAULT (datetime('now')),
         FOREIGN KEY (endpoint_id) REFERENCES endpoints(id)
@@ -70,6 +120,14 @@ class Storage {
         FOREIGN KEY (event_id) REFERENCES events(id)
       );
 
+      CREATE TABLE IF NOT EXISTS instances (
+        id TEXT PRIMARY KEY,
+        hostname TEXT,
+        pid INTEGER,
+        started_at TEXT DEFAULT (datetime('now')),
+        last_heartbeat TEXT DEFAULT (datetime('now'))
+      );
+
       CREATE TABLE IF NOT EXISTS waitlist (
         id INTEGER PRIMARY KEY AUTOINCREMENT,
         email TEXT UNIQUE NOT NULL,
@@ -85,11 +143,24 @@ class Storage {
       CREATE INDEX IF NOT EXISTS idx_attempts_event ON delivery_attempts(event_id);
     `);
 
-    // Migration: whether the event's signature was verified at ingress
+    // Migrations for existing databases
     const eventColumns = this.db.prepare('PRAGMA table_info(events)').all().map((c) => c.name);
     if (!eventColumns.includes('verified')) {
       this.db.exec('ALTER TABLE events ADD COLUMN verified INTEGER DEFAULT 0');
     }
+    if (!eventColumns.includes('claimed_by')) {
+      this.db.exec('ALTER TABLE events ADD COLUMN claimed_by TEXT');
+    }
+    if (!eventColumns.includes('claimed_at')) {
+      this.db.exec('ALTER TABLE events ADD COLUMN claimed_at TEXT');
+    }
+
+    try {
+      this.db.exec(`
+        CREATE UNIQUE INDEX IF NOT EXISTS idx_events_idem_unique
+          ON events(endpoint_id, idempotency_key) WHERE idempotency_key IS NOT NULL;
+      `);
+    } catch (_) {}
   }
 
   // Upsert. Omitting secret or alertWebhookUrl (undefined) keeps the stored value; '' clears it.
@@ -97,6 +168,7 @@ class Storage {
     const existing = this.getEndpoint(id);
     const finalSecret = secret !== undefined ? secret : (existing ? existing.secret : '');
     const finalAlert = alertWebhookUrl !== undefined ? alertWebhookUrl : (existing ? existing.alert_webhook_url : '');
+    const encryptedSecret = encryptSecret(finalSecret, this.encryptionKey);
     this.db.prepare(`
       INSERT INTO endpoints (id, name, target_url, secret, alert_webhook_url, auto_retry, max_retries, concurrency_limit)
       VALUES (?, ?, ?, ?, ?, ?, ?, ?)
@@ -108,21 +180,32 @@ class Storage {
         auto_retry = excluded.auto_retry,
         max_retries = excluded.max_retries,
         concurrency_limit = excluded.concurrency_limit
-    `).run(id, name, targetUrl, finalSecret || '', finalAlert || '', autoRetry ? 1 : 0, maxRetries, concurrencyLimit || 5);
+    `).run(id, name, targetUrl, encryptedSecret || '', finalAlert || '', autoRetry ? 1 : 0, maxRetries, concurrencyLimit || 5);
     return this.getEndpoint(id);
   }
 
   getEndpoint(id) {
-    return this.db.prepare('SELECT * FROM endpoints WHERE id = ?').get(id);
+    const row = this.db.prepare('SELECT * FROM endpoints WHERE id = ?').get(id);
+    if (!row) return null;
+    return { ...row, secret: decryptSecret(row.secret, this.encryptionKey) };
   }
 
   listEndpoints() {
-    return this.db.prepare('SELECT * FROM endpoints ORDER BY created_at DESC').all();
+    const rows = this.db.prepare('SELECT * FROM endpoints ORDER BY created_at DESC').all();
+    return rows.map((r) => ({ ...r, secret: decryptSecret(r.secret, this.encryptionKey) }));
   }
 
   deleteEndpoint(id) {
-    const info = this.db.prepare('DELETE FROM endpoints WHERE id = ?').run(id);
-    return info.changes > 0;
+    const deleteTx = this.db.transaction((epId) => {
+      this.db.prepare(`
+        DELETE FROM delivery_attempts WHERE event_id IN (
+          SELECT id FROM events WHERE endpoint_id = ?
+        )
+      `).run(epId);
+      this.db.prepare('DELETE FROM events WHERE endpoint_id = ?').run(epId);
+      return this.db.prepare('DELETE FROM endpoints WHERE id = ?').run(epId).changes > 0;
+    });
+    return deleteTx(id);
   }
 
   // Any stored copy counts: once HookArmor has custody, a provider re-send is redundant
@@ -133,24 +216,33 @@ class Storage {
     return { ...row, headers: JSON.parse(row.headers) };
   }
 
-  saveEvent({ id, endpointId, idempotencyKey = null, provider, eventType, headers, rawBody, status = 'pending', verified = false }) {
+  saveEvent({ id, endpointId, idempotencyKey = null, provider, eventType, headers, rawBody, status = 'pending', claimedBy = null, verified = false }) {
     const rawBuffer = Buffer.isBuffer(rawBody) ? rawBody : Buffer.from(String(rawBody || ''), 'utf8');
     const stmt = this.db.prepare(`
-      INSERT INTO events (id, endpoint_id, idempotency_key, provider, event_type, headers, raw_body, status, attempts, verified, created_at, updated_at)
-      VALUES (?, ?, ?, ?, ?, ?, ?, ?, 0, ?, datetime('now'), datetime('now'))
+      INSERT INTO events (id, endpoint_id, idempotency_key, provider, event_type, headers, raw_body, status, attempts, claimed_by, claimed_at, verified, created_at, updated_at)
+      VALUES (?, ?, ?, ?, ?, ?, ?, ?, 0, ?, ${claimedBy ? "datetime('now')" : 'NULL'}, ?, datetime('now'), datetime('now'))
     `);
-    stmt.run(
-      id,
-      endpointId,
-      idempotencyKey,
-      provider,
-      eventType,
-      JSON.stringify(headers),
-      rawBuffer,
-      status,
-      verified ? 1 : 0
-    );
-    return this.getEvent(id);
+    try {
+      stmt.run(
+        id,
+        endpointId,
+        idempotencyKey,
+        provider,
+        eventType,
+        JSON.stringify(headers),
+        rawBuffer,
+        status,
+        claimedBy,
+        verified ? 1 : 0
+      );
+      return this.getEvent(id);
+    } catch (err) {
+      if (idempotencyKey && (err.code === 'SQLITE_CONSTRAINT_UNIQUE' || (err.message && err.message.includes('UNIQUE constraint failed')))) {
+        const existing = this.findEventByIdempotencyKey(endpointId, idempotencyKey);
+        if (existing) return existing;
+      }
+      throw err;
+    }
   }
 
   getPendingCount() {
@@ -290,35 +382,77 @@ class Storage {
     }));
   }
 
-  claimEventForRetry(eventId) {
+  registerInstance({ id, hostname, pid }) {
+    if (!id) return;
+    this.db.prepare(`
+      INSERT INTO instances (id, hostname, pid, started_at, last_heartbeat)
+      VALUES (?, ?, ?, datetime('now'), datetime('now'))
+      ON CONFLICT(id) DO UPDATE SET last_heartbeat = datetime('now'), pid = excluded.pid, hostname = excluded.hostname
+    `).run(id, hostname || null, pid || null);
+  }
+
+  heartbeatInstance(id) {
+    if (!id) return;
+    this.db.prepare(`
+      UPDATE instances SET last_heartbeat = datetime('now') WHERE id = ?
+    `).run(id);
+  }
+
+  reapStaleInstances(timeoutSeconds = 300) {
+    return this.db.prepare(`
+      DELETE FROM instances WHERE datetime(last_heartbeat) < datetime('now', '-' || ? || ' seconds')
+    `).run(timeoutSeconds).changes;
+  }
+
+  claimEventForRetry(eventId, instanceId = null) {
     const info = this.db.prepare(`
       UPDATE events
-      SET status = 'replaying', updated_at = datetime('now')
+      SET status = 'replaying',
+          claimed_by = ?,
+          claimed_at = datetime('now'),
+          updated_at = datetime('now')
       WHERE id = ? AND status IN ('failed', 'pending')
-    `).run(eventId);
+    `).run(instanceId, eventId);
     return info.changes > 0;
   }
 
   // Manual replay may re-send delivered events, but never one that is already in flight
-  claimEventForManualReplay(eventId) {
+  claimEventForManualReplay(eventId, instanceId = null) {
     const info = this.db.prepare(`
       UPDATE events
-      SET status = 'replaying', updated_at = datetime('now')
+      SET status = 'replaying',
+          claimed_by = ?,
+          claimed_at = datetime('now'),
+          updated_at = datetime('now')
       WHERE id = ? AND status IN ('failed', 'pending', 'delivered')
-    `).run(eventId);
+    `).run(instanceId, eventId);
     return info.changes > 0;
   }
 
-  // Run once at startup: anything still 'replaying' was in flight when the process died
+  // Run at startup or worker tick:
+  // Reclaim events if:
+  // 1. claimed_by is NULL, OR
+  // 2. claimed_by is not active in instances table (last_heartbeat < 45 seconds ago), OR
+  // 3. claimed_at is older than 180 seconds (worker hung)
   recoverInterruptedDeliveries() {
-    return this.db.prepare(`
+    const sql = `
       UPDATE events
       SET status = 'failed',
           next_retry_at = datetime('now'),
           last_error = COALESCE(last_error, 'Delivery interrupted by restart'),
+          claimed_by = NULL,
+          claimed_at = NULL,
           updated_at = datetime('now')
       WHERE status = 'replaying'
-    `).run().changes;
+        AND (
+          claimed_by IS NULL
+          OR claimed_by NOT IN (
+            SELECT id FROM instances WHERE datetime(last_heartbeat) >= datetime('now', '-45 seconds')
+          )
+          OR (claimed_at IS NOT NULL AND datetime(claimed_at) < datetime('now', '-180 seconds'))
+        )
+    `;
+    return this.db.prepare(sql).run().changes;
   }
 
   // Delete delivered events (and their attempt logs) older than `days`. Failed events are kept.
@@ -351,5 +485,8 @@ class Storage {
 }
 
 Storage.toSqliteTime = toSqliteTime;
+Storage.deriveKey = deriveKey;
+Storage.encryptSecret = encryptSecret;
+Storage.decryptSecret = decryptSecret;
 
 module.exports = Storage;

package/src/worker.js

@@ -9,6 +9,7 @@ class RetryWorker extends EventEmitter {
     this.retentionDays = options.retentionDays || 0;
     this.pruneIntervalMs = options.pruneIntervalMs || 60 * 60 * 1000;
     this.lastPruneAt = 0;
+    this.instanceId = options.instanceId || null;
     this.timer = null;
     this.isProcessing = false;
   }
@@ -41,13 +42,19 @@ class RetryWorker extends EventEmitter {
     this.isProcessing = true;
 
     try {
+      if (this.instanceId) {
+        this.storage.heartbeatInstance(this.instanceId);
+      }
+      this.storage.recoverInterruptedDeliveries(this.instanceId);
+      this.storage.reapStaleInstances(300);
+
       this.prune();
 
       const eventsDue = this.storage.getEventsDueForRetry(25);
       if (!eventsDue || eventsDue.length === 0) return;
 
       for (const event of eventsDue) {
-        const claimed = this.storage.claimEventForRetry(event.id);
+        const claimed = this.storage.claimEventForRetry(event.id, this.instanceId);
         if (!claimed) continue; // Another process or manual replay already claimed it
 
         const endpoint = this.storage.getEndpoint(event.endpoint_id);