@jetit/publisher 5.6.3 → 6.0.1

package/README.md

@@ -19,6 +19,7 @@
     - [Event Filtering](#event-filtering)
     - [Performance Monitoring](#performance-monitoring)
     - [Prometheus Integration](#prometheus-integration)
+  - [Publisher vs PublisherLite](#publisher-vs-publisherlite)
   - [Advanced Features](#advanced-features)
     - [Content-Based Deduplication](#content-based-deduplication)
     - [Multiple Event Subscriptions](#multiple-event-subscriptions)
@@ -72,6 +73,33 @@ publisher.listen('my-event').subscribe(event => {
 });
 ```
 
+### PublisherLite Usage
+
+For scenarios where a single stream per event type is preferred (allowing multiple consumer groups on that single stream), use `PublisherLite`:
+
+```typescript
+import { PublisherLite, EventData } from '@jetit/publisher'; // Import PublisherLite
+
+// Create an instance of PublisherLite
+const publisherLite = new PublisherLite('MyServiceLite');
+
+// Publish an event (same as Publisher)
+const eventData: EventData<{ message: string }> = {
+    eventName: 'my-lite-event',
+    data: { message: 'Hello, Lite world!' }
+};
+await publisherLite.publish(eventData);
+
+// Subscribe to an event (same as Publisher)
+publisherLite.listen('my-lite-event').subscribe(event => {
+    console.log(`Received lite event: ${event.eventName}`, event.data);
+});
+
+// Configuration is also similar, just pass it to PublisherLite constructor
+const liteConfig: Partial<IStreamsConfig> = { /* ... your config ... */ };
+const configuredPublisherLite = new PublisherLite('MyConfiguredServiceLite', liteConfig, 'my-redis-connection');
+```
+
 ### Configuration
 
 The `Publisher` class can be configured with various options, including Circuit Breaker and Backpressure handling:
@@ -97,6 +125,7 @@ const config: Partial<IStreamsConfig> = {
         halfOpenStateMaxAttempts: 10,
         maxStoredEvents: 5000,
     },
+    maxPendingTasks: 1000, // Max concurrent pending retry tasks per event per consumer (default: 1000, 0 = unbounded)
 };
 
 const publisher = new Publisher('MyService', config);
@@ -240,6 +269,43 @@ app.listen(3000, () => {
 });
 ```
 
+## Publisher vs PublisherLite
+
+This library offers two main publisher implementations: `Publisher` and `PublisherLite`. Choose the one that best fits your architecture and scaling needs.
+
+**`Publisher` (Default - Multi-Stream)**
+
+*   **Mechanism:** Creates a separate Redis Stream for *each* consumer group subscribing to an event type (e.g., `my-event:cg-serviceA`, `my-event:cg-serviceB`).
+*   **Pros:**
+    *   Provides strong isolation between consumer groups. Issues or heavy load in one group's stream don't directly impact others.
+    *   Potentially simpler cleanup logic per stream.
+    *   May offer better performance distribution if consumer groups have vastly different processing speeds or volumes.
+*   **Cons:**
+    *   Can lead to a large number of streams in Redis, especially with many event types and consumer groups, increasing memory usage and management overhead.
+    *   Requires more complex ID generation (like the adaptive Lua script) to handle potential ID collisions across streams when publishing.
+*   **Use When:**
+    *   You have a manageable number of consumer groups per event type.
+    *   Strong isolation between consumer groups is critical.
+    *   You anticipate significant differences in load or processing characteristics between consumer groups for the same event.
+
+**`PublisherLite` (Single-Stream)**
+
+*   **Mechanism:** Uses a *single* Redis Stream per event type (prefixed with `sl:`, e.g., `sl:my-event`). All consumer groups for that event read from this single stream.
+*   **Pros:**
+    *   Significantly reduces the number of streams in Redis, lowering memory usage and simplifying management.
+    *   Simplifies the publishing logic (no need for complex multi-stream ID generation).
+    *   Aligns more closely with the standard Redis Streams consumer group model.
+*   **Cons:**
+    *   Less isolation between consumer groups; a very slow or problematic consumer group could potentially impact the processing lag for others on the same stream (though Redis handles much of this internally).
+    *   Cleanup logic (`XTRIM`) is slightly more complex as it needs to consider the progress of *all* consumer groups on the stream before trimming messages.
+*   **Use When:**
+    *   You have a large number of event types or expect many consumer groups per event.
+    *   Reducing Redis resource consumption (memory, keyspace) is a priority.
+    *   You prefer a simpler publishing mechanism and are comfortable with the standard Redis consumer group behavior on a shared stream.
+    *   **Important:** `PublisherLite` currently focuses on the core publish/listen flow and **does not support scheduled publishing** (`scheduledPublish` method). Use the standard `Publisher` if you require scheduled events.
+
+**Note:** Both `Publisher` and `PublisherLite` share the same configuration options (`IStreamsConfig`) and core features like DLQ, Circuit Breaker, Metrics, etc. The primary difference lies in the underlying Redis Stream structure they use and the support for scheduled publishing.
+
 ## Advanced Features
 
 ### Content-Based Deduplication
@@ -293,6 +359,8 @@ The Circuit Breaker has three states:
 - Retry logic with exponential backoff for failed operations
 - Circuit Breaker to prevent overwhelming failed services
 - Dead Letter Queue (DLQ) for handling subscription failures
+- **Pending Retry Task Cap**: Limits the number of concurrent in-flight pending message retry tasks per event per consumer instance. Prevents unbounded memory growth and Redis saturation under sustained load. Configurable via `maxPendingTasks` (default: 1000). Set to `0` to disable the cap (unbounded, pre-6.0.1 behavior). When the cap is reached, a rate-limited warning is logged and skipped messages are retried on the next Pub/Sub trigger.
+- **Adaptive Redis Stream ID Generation (Publisher only)**: The default `Publisher` automatically switches to an optimized ID generation strategy using a Lua script when publishing to many consumer groups (>10 by default) or when ID conflicts are detected. This prevents `XADD` errors related to non-monotonic IDs in high-throughput scenarios. Configurable via `optimizationThreshold` and `optimizationDurationMs`. (`PublisherLite` does not include this complex logic).
   
 ## Cleanup and Graceful Shutdown
 

package/package.json

@@ -1,12 +1,14 @@
 {
   "name": "@jetit/publisher",
-  "version": "5.6.3",
+  "version": "6.0.1",
   "type": "commonjs",
+  "peerDependencies": {
+    "@jetit/id": ">=0.0.14",
+    "ioredis": ">=5.0.0",
+    "rxjs": ">=7.0.0"
+  },
   "dependencies": {
-    "@jetit/id": "^0.0.13",
-    "ioredis": "^5.3.0",
-    "rxjs": "^7.8.0",
-    "tslib": "2.7.0"
+    "tslib": "2.8.1"
   },
   "types": "./src/index.d.ts",
   "main": "./src/index.js"

package/src/lib/publisher.d.ts

@@ -1,5 +1,6 @@
 export { setRedisConnectionSettings as setRedisConfig } from './redis/registry';
 export { Streams as Publisher } from './redis/streams';
+export { StreamsLite as PublisherLite } from './redis/streams-lite';
 export { ScheduledProcessor as __SCHEDULER_INTERNALS__ } from './redis/scheduler';
 export { UTILS as StreamUtilityFunctions } from './redis/utils';
 export { PrometheusAdapter } from './monitoring/adapters/prom';

package/src/lib/publisher.js

@@ -1,10 +1,12 @@
 "use strict";
 Object.defineProperty(exports, "__esModule", { value: true });
-exports.publishScheduledBatch = exports.publishBatch = exports.PrometheusAdapter = exports.StreamUtilityFunctions = exports.__SCHEDULER_INTERNALS__ = exports.Publisher = exports.setRedisConfig = void 0;
+exports.publishScheduledBatch = exports.publishBatch = exports.PrometheusAdapter = exports.StreamUtilityFunctions = exports.__SCHEDULER_INTERNALS__ = exports.PublisherLite = exports.Publisher = exports.setRedisConfig = void 0;
 var registry_1 = require("./redis/registry");
 Object.defineProperty(exports, "setRedisConfig", { enumerable: true, get: function () { return registry_1.setRedisConnectionSettings; } });
 var streams_1 = require("./redis/streams");
 Object.defineProperty(exports, "Publisher", { enumerable: true, get: function () { return streams_1.Streams; } });
+var streams_lite_1 = require("./redis/streams-lite");
+Object.defineProperty(exports, "PublisherLite", { enumerable: true, get: function () { return streams_lite_1.StreamsLite; } });
 var scheduler_1 = require("./redis/scheduler");
 Object.defineProperty(exports, "__SCHEDULER_INTERNALS__", { enumerable: true, get: function () { return scheduler_1.ScheduledProcessor; } });
 var utils_1 = require("./redis/utils");

package/src/lib/redis/scheduler.js

@@ -63,7 +63,7 @@ class ScheduledProcessor {
                     .xadd(streamName, key, 'data', JSON.stringify(eventData))
                     .catch((e) => logger_1.PUBLISHER_LOGGER.error(`PUBLISHER: Publishing event ${eventData.eventName} to consumer groups: ${consumerGroups.join(', ')} failed with data ${JSON.stringify(eventData)}, ${e} `));
                 if (key === '*')
-                    key = `${generatedKey ?? key}`;
+                    key = generatedKey || key;
             }
             if (eventData.repeatInterval) {
                 const nextEventTime = currentTime + eventData.repeatInterval;

package/src/lib/redis/streams-lite.d.ts

@@ -0,0 +1,187 @@
+import { Observable } from 'rxjs';
+import { IAggregatedMetrics, TQueryableMetrics } from '../monitoring/types';
+import { CircuitState } from '../performance/circuit_breaker';
+import { EventData, IListenOptions, IStreamsConfig, PublishData } from './types';
+export declare class StreamsLite {
+    private redisConnectionId;
+    private _redisPublisher?;
+    private _redisGroups?;
+    private config;
+    private dlq;
+    private metricsCollector;
+    private consumerGroupName;
+    private instanceId;
+    private instanceUniqueId;
+    private cleanUpTimer;
+    private eventsListened;
+    private subscriptions;
+    private duplicateChecker;
+    private circuitBreaker;
+    private get redisPublisher();
+    private get redisGroups();
+    private DEFAULT_STREAMS_CONFIG;
+    /**
+     * Creates a new Streams instance for a given service.
+     *
+     * The constructor initializes the Redis connections for publishers, subscribers, and consumer groups.
+     * It also sets up an interval timer for clearing expired messages from Redis and another interval timer
+     * for processing scheduled events at regular intervals.
+     *
+     * @param serviceName - A unique name for the service that will be using this Streams instance.
+     *
+     * @example
+     *
+     * // Create a new Streams instance for the "POS" service
+     * const streams = new Streams('POS');
+     */
+    constructor(serviceName: string, config?: Partial<IStreamsConfig>, redisConnectionId?: string);
+    private setupCircuitBreakerListeners;
+    private runClear;
+    /**
+     * Publishes an event with the given data to the Redis event stream.
+     *
+     * The method generates a unique event ID for each event, and adds it to the event data to handle message
+     * deduplication.
+     *
+     * @param data - An EventData<T> object containing the data for the event.
+     *
+     * @returns A Promise that resolves when the event has been published to the Redis stream.
+     *
+     * @example
+     *
+     * // Publish an "order.created" event with the given order data
+     * const orderData = { id: '123', customerName: 'John Doe', amount: 100 };
+     * const eventData = { eventName: 'order.created', data: orderData };
+     * await publisher.publish(eventData);
+     */
+    publish<TData = unknown, TName extends string = string>(data: PublishData<TData, TName>, multicast?: boolean): Promise<string>;
+    /**
+     * Listens for events with the given name and returns an Observable that emits an EventData<T> object
+     * each time a new event is received.
+     *
+     * The method uses a BehaviorSubject to emit the events as Observables. The BehaviorSubject ensures
+     * that new subscribers receive the last emitted event, even if they subscribe after the event has been emitted.
+     *
+     * If an error occurs while subscribing, the method logs the error to the PUBLISHER_LOGGER and throws
+     * an error. This is done to prevent the service from continuing without a proper event subscription.
+     *
+     * There is retry logic with exponential backoff to handle error cases. These are also controllable by the
+     * calling service
+     *
+     * @param eventName - The name of the event to listen for.
+     *
+     * @returns An Observable that emits an EventData<T> object each time a new event is received.
+     *
+     * @example
+     *
+     * // Listen for "order.created" events
+     * const orderCreated = streams.listen<OrderCreatedEvent>('order.created');
+     *
+     * // Subscribe to the Observable and log each new event
+     * orderCreated.subscribe((event) => {
+     *   PUBLISHER_LOGGER.log('New order created:', event.data);
+     * });
+     */
+    listen<T = unknown, const TName extends string = string>(eventName: TName, listenerOptions?: IListenOptions<T>): Observable<EventData<T, TName>>;
+    private createConsumerAndRegister;
+    private listenInternals;
+    /**
+     * This method allows the possibility of a graceful shutdown by cleaning up the
+     * redis connections.
+     *
+     * In all services where the library is used, its better to implement this method
+     *
+     * process.on('SIGTERM', shutdown);
+     * process.on('SIGINT', shutdown);
+     *
+     * async function shutdown(): Promise<void> {
+     *   PUBLISHER_LOGGER.log('Graceful shutdown initiated.');
+     *   try {
+     *     await streams.close();
+     *     PUBLISHER_LOGGER.log('Resources and connections successfully closed.');
+     *   } catch (error) {
+     *     PUBLISHER_LOGGER.error('Error during graceful shutdown:', error);
+     *   }
+     *   process.exit(0);
+     * }
+     */
+    close(): Promise<void>;
+    private cleanupAcknowledgedMessages;
+    getDiagnosticData(events: string[]): Promise<{
+        status: string;
+        message: string;
+        data?: undefined;
+    } | {
+        status: string;
+        data: {
+            eventName: string;
+            streamName: string;
+            consumerGroupMap: {
+                consumerGroup: string;
+                diagnostics: Generator<Promise<unknown[]> | {
+                    count: number;
+                    consumers: {
+                        consumerName: string;
+                        pendingCount: string;
+                    }[];
+                }, void, [number, string, string, string[][]]>;
+            }[];
+        }[];
+        message: string;
+    }>;
+    private logPerformance;
+    /**
+     * @description
+     * This method is use to retry an event that has ended in the dead letter queue,
+     * which happens after the first retry.
+     */
+    retryFromDLQ(eventId: string): Promise<boolean>;
+    /**
+     * @description
+     * This returns the number of items and the rate at which events are added
+     * to the queue. The queue is global and hence remains as is
+     */
+    getDLQStats(): Promise<{
+        size: number;
+        additionRate: number;
+    }>;
+    private removeSubscription;
+    /**
+     * @description
+     * This is a simple helper utility that can be used externally to create alerts based
+     * on thresholds that can be provided into the function. It returns true/false for each
+     * key that is provided. Not all keys are required
+     */
+    checkThresholds(thresholds: Partial<TQueryableMetrics>): Promise<Record<string, boolean>>;
+    /**
+     * @description
+     * This will return you the stats of the publisher for the last 6 hours after cleaning
+     */
+    getMetrics(startTime: number, endTime: number): Promise<IAggregatedMetrics[]>;
+    /**
+     * @description
+     * This will return you the latest stats of the publisher
+     */
+    getLatestMetrics(): Promise<IAggregatedMetrics | null>;
+    /**
+     * @description
+     * This returns the status of the performance control setup. This includes
+     * the circuit breaker
+     */
+    getPerformanceControlStatus(): Promise<{
+        circuitBreakerState: CircuitState;
+    }>;
+    /**
+     * @description
+     * This is a manual control to process stored events in case the
+     * circuit is OPEN
+     */
+    processStoredEvents(): Promise<void>;
+    /**
+     * Acknowledges a message and updates the last acknowledged message ID for the specific consumer group.
+     * This is used to track cleanup progress and ensure we don't delete unprocessed messages.
+     */
+    acknowledgeMessage(ackKey: string): Promise<void>;
+    private frameMessageKey;
+    private demergeMessageKey;
+}