@cap-js-community/event-queue 0.2.4 → 0.2.5

package/README.md

@@ -5,22 +5,24 @@
 [![REUSE status](https://api.reuse.software/badge/github.com/cap-js-community/event-queue)](https://api.reuse.software/info/github.com/cap-js-community/event-queue)
 [![CI Main](https://github.com/cap-js-community/event-queue/actions/workflows/main-ci.yml/badge.svg)](https://github.com/cap-js-community/event-queue/commits/main)
 
-The Event-Queue is a framework built on top of CAP Node.js, specifically designed to enable efficient and streamlined
-asynchronous event processing in a multi-tenancy environment. With a strong emphasis on load balancing, this package
-ensures optimal distribution of workload across all available application instances. By offering managed tenant-specific
-transactions, similar to CAP handlers, the Event-Queue framework simplifies event and asynchronous processing, thereby
-enhancing the overall performance of your application.
+The Event-Queue is a framework built on top of CAP Node.js, designed specifically for efficient and
+streamlined asynchronous event processing. With a focus on load balancing, this package ensures optimal
+event distribution across all available application instances. By providing managed transactions similar to CAP
+handlers, the Event-Queue framework simplifies event processing, enhancing the overall performance of your application.
+
+Additionally, Event-Queue provides support for [periodic events](https://cap-js-community.github.io/event-queue/configure-event/#periodic-events),
+allowing for processing at defined intervals. This feature further extends its capabilities in load balancing and
+transaction management, ensuring that even regularly occurring tasks are handled efficiently and effectively without
+overloading any single instance. This makes it an ideal solution for applications needing consistent, reliable event processing.
 
 ## Getting started
 
 - Run `npm add @cap-js-community/event-queue` in `@sap/cds` project
-- Activate the cds-plugin in the cds section of the package.sjon
+- Activate the cds-plugin in the cds section of the package.json.
 
 ### As cds-plugin
 
-Extend the cds section of your package.json. Reference to the cds-plugin section in the capire documentation about the
-cds-plugin concept.
-https://cap.cloud.sap/docs/releases/march23#new-cds-plugin-technique
+For detailed information check out the [documentation](https://cap-js-community.github.io/event-queue/setup).
 
 ```json
 {
@@ -35,13 +37,16 @@ https://cap.cloud.sap/docs/releases/march23#new-cds-plugin-technique
 
 ## Features
 
-- load balancing of event processing throughout app instances
-- control concurrency in app instances
-- managed transactions for event processing
+Learn more about features in the [documentation](https://cap-js-community.github.io/event-queue/#functionality-and-problem-solutions). To compare the
+event-queue with other SAP products head over to [Distinction from other solutions](https://cap-js-community.github.io/event-queue/diff-to-outbox/).
+
+- [load balancing](https://cap-js-community.github.io/event-queue/load-balancing) and concurrency control of event processing across app instances
+- [periodic events](https://cap-js-community.github.io/event-queue/configure-event/#periodic-events) similar to running cron jobs for business processes
+- [managed transactions](https://cap-js-community.github.io/event-queue/transaction-handling) for event processing
 - async processing of processing intensive tasks for better UI responsiveness
 - push/pull mechanism for reducing delay between publish an event and processing
 - cluster published events during processing (e.g. for combining multiple E-Mail events to one E-Mail)
-- plug and play via cds-plugin
+- [plug and play](https://cap-js-community.github.io/event-queue/setup) via cds-plugin
 
 ## Documentation
 
@@ -50,7 +55,7 @@ Head over to our [Documentation](https://cap-js-community.github.io/event-queue/
 ## Support, Feedback, Contributing
 
 This project is open to feature requests/suggestions, bug reports etc.
-via [GitHub issues](https://github.com/cap-js-community/<your-project>/issues). Contribution and feedback are encouraged
+via [GitHub issues](https://github.com/cap-js-communityevent-queue/issues). Contribution and feedback are encouraged
 and always welcome. For more information about how to contribute, the project structure, as well as additional
 contribution information, see our [Contribution Guidelines](CONTRIBUTING.md).
 
@@ -62,7 +67,7 @@ times.
 
 ## Licensing
 
-Copyright 2023 SAP SE or an SAP affiliate company and `@cap-js-community/event-queue contributors`. Please see
+Copyright 2023 SAP SE or an SAP affiliate company and `@cap-js-community/event-queue` contributors. Please see
 our [LICENSE](LICENSE) for copyright and license information. Detailed information including third-party components and
 their licensing/copyright information is
 available [via the REUSE tool](https://api.reuse.software/info/github.com/cap-js-community/<your-project>).

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@cap-js-community/event-queue",
-  "version": "0.2.4",
+  "version": "0.2.5",
   "description": "An event queue that enables secure transactional processing of asynchronous events, featuring instant event processing with Redis Pub/Sub and load distribution across all application instances.",
   "main": "src/index.js",
   "files": [
@@ -31,7 +31,7 @@
     "lint:ci": "npm run eslint:ci && npm run prettier:ci",
     "eslint": "eslint --fix .",
     "eslint:ci": "eslint .",
-    "prettier": "prettier --write --log-level error .",
+    "prettier": "prettier --write --loglevel error .",
     "prettier:ci": "prettier --check .",
     "prepareRelease": "npm prune --production",
     "docs": "cd docs && bundle exec jekyll serve",
@@ -42,14 +42,14 @@
     "node": ">=16"
   },
   "dependencies": {
-    "redis": "4.6.10",
+    "redis": "4.6.11",
     "verror": "1.10.1",
     "yaml": "2.3.4"
   },
   "devDependencies": {
-    "@sap/cds": "7.3.1",
-    "@sap/cds-dk": "7.3.1",
-    "eslint": "8.52.0",
+    "@sap/cds": "7.4.0",
+    "@sap/cds-dk": "7.4.0",
+    "eslint": "8.54.0",
     "eslint-config-prettier": "9.0.0",
     "eslint-plugin-jest": "27.6.0",
     "eslint-plugin-node": "11.1.0",

package/src/EventQueueProcessorBase.js

@@ -534,7 +534,7 @@ class EventQueueProcessorBase {
    */
   async getQueueEntriesAndSetToInProgress() {
     let result = [];
-    const refDateStartAfter = new Date(Date.now() + this.#config.runInterval);
+    const refDateStartAfter = new Date(Date.now() + this.#config.runInterval * 1.2);
     await executeInNewTransaction(this.__baseContext, "eventQueue-getQueueEntriesAndSetToInProgress", async (tx) => {
       const entries = await tx.run(
         SELECT.from(this.#config.tableNameEventQueue)
@@ -586,7 +586,7 @@ class EventQueueProcessorBase {
       this.#handleDelayedEvents(delayedEvents);
 
       result = openEvents;
-      this.logger.info("Selected event queue entries for processing", {
+      this.logger[eventsForProcessing.length ? "info" : "debug"]("Selected event queue entries for processing", {
         openEvents: openEvents.length,
         ...(delayedEvents.length && { delayedEvents: delayedEvents.length }),
         ...(exceededTries.length && { exceededTries: exceededTries.length }),
@@ -639,7 +639,7 @@ class EventQueueProcessorBase {
   }
 
   #clusterEvents(events, refDateStartAfter) {
-    const refDate = new Date(refDateStartAfter.getTime() - this.#config.runInterval + EVENT_START_AFTER_HEADROOM);
+    const refDate = new Date(refDateStartAfter.getTime() - this.#config.runInterval * 1.2 + EVENT_START_AFTER_HEADROOM);
     return events.reduce(
       (result, event) => {
         if (event.attempts === this.__retryAttempts + TRIES_FOR_EXCEEDED_EVENTS) {
@@ -858,11 +858,11 @@ class EventQueueProcessorBase {
   }
 
   async scheduleNextPeriodEvent(queueEntry) {
-    const interval = this.#eventConfig.interval;
+    const intervalInSec = this.#eventConfig.interval * 1000;
     const newEvent = {
       type: this.#eventType,
       subType: this.#eventSubType,
-      startAfter: new Date(new Date(queueEntry.startAfter).getTime() + interval * 1000),
+      startAfter: new Date(new Date(queueEntry.startAfter).getTime() + intervalInSec),
     };
     const { relative } = this.#eventSchedulerInstance.calculateOffset(
       this.#eventType,
@@ -871,7 +871,7 @@ class EventQueueProcessorBase {
     );
 
     // more than one interval behind - shift tick to keep up
-    if (relative < 0 && Math.abs(relative) >= this.#eventConfig.interval * 1000) {
+    if (relative < 0 && Math.abs(relative) >= intervalInSec) {
       newEvent.startAfter = new Date(Date.now() + 5 * 1000);
       this.logger.info("interval adjusted because shifted more than one interval", {
         eventType: this.#eventType,
@@ -883,7 +883,7 @@ class EventQueueProcessorBase {
     this.tx._skipEventQueueBroadcase = true;
     await this.tx.run(INSERT.into(this.#config.tableNameEventQueue).entries({ ...newEvent }));
     this.tx._skipEventQueueBroadcase = false;
-    if (interval < this.#config.runInterval) {
+    if (intervalInSec < this.#config.runInterval * 1.5) {
       this.#handleDelayedEvents([newEvent]);
       const { relative: relativeAfterSchedule } = this.#eventSchedulerInstance.calculateOffset(
         this.#eventType,
@@ -941,6 +941,10 @@ class EventQueueProcessorBase {
     return this.__logger ?? this.__baseLogger;
   }
 
+  set logger(value) {
+    this.__logger = value;
+  }
+
   get queueEntriesWithPayloadMap() {
     return this.#queueEntriesWithPayloadMap;
   }

package/src/config.js

@@ -21,7 +21,7 @@ class Config {
   #runInterval;
   #redisEnabled;
   #initialized;
-  #parallelTenantProcessing;
+  #instanceLoadLimit;
   #tableNameEventQueue;
   #tableNameEventLock;
   #isRunnerDeactivated;
@@ -42,7 +42,7 @@ class Config {
     this.#runInterval = null;
     this.#redisEnabled = null;
     this.#initialized = false;
-    this.#parallelTenantProcessing = null;
+    this.#instanceLoadLimit = null;
     this.#tableNameEventQueue = null;
     this.#tableNameEventLock = null;
     this.#isRunnerDeactivated = false;
@@ -214,12 +214,12 @@ class Config {
     this.#initialized = value;
   }
 
-  get parallelTenantProcessing() {
-    return this.#parallelTenantProcessing;
+  get instanceLoadLimit() {
+    return this.#instanceLoadLimit;
   }
 
-  set parallelTenantProcessing(value) {
-    this.#parallelTenantProcessing = value;
+  set instanceLoadLimit(value) {
+    this.#instanceLoadLimit = value;
   }
 
   get tableNameEventQueue() {

package/src/initialize.js

@@ -29,7 +29,7 @@ const CONFIG_VARS = [
   ["processEventsAfterPublish", true],
   ["isRunnerDeactivated", false],
   ["runInterval", 5 * 60 * 1000],
-  ["parallelTenantProcessing", 5],
+  ["instanceLoadLimit", 20],
   ["tableNameEventQueue", BASE_TABLES.EVENT],
   ["tableNameEventLock", BASE_TABLES.LOCK],
   ["disableRedis", false],
@@ -43,7 +43,7 @@ const initialize = async ({
   processEventsAfterPublish,
   isRunnerDeactivated,
   runInterval,
-  parallelTenantProcessing,
+  instanceLoadLimit,
   tableNameEventQueue,
   tableNameEventLock,
   disableRedis,
@@ -52,7 +52,7 @@ const initialize = async ({
 } = {}) => {
   // TODO: initialize check:
   // - content of yaml check
-  // - betweenRuns and parallelTenantProcessing
+  // - betweenRuns and instanceLoadLimit
 
   if (config.initialized) {
     return;
@@ -65,7 +65,7 @@ const initialize = async ({
     processEventsAfterPublish,
     isRunnerDeactivated,
     runInterval,
-    parallelTenantProcessing,
+    instanceLoadLimit,
     tableNameEventQueue,
     tableNameEventLock,
     disableRedis,
@@ -92,7 +92,7 @@ const initialize = async ({
     multiTenancyEnabled: config.isMultiTenancy,
     redisEnabled: config.redisEnabled,
     runInterval: config.runInterval,
-    config: config.parallelTenantProcessing,
+    config: config.instanceLoadLimit,
   });
 };
 

package/src/periodicEvents.js

@@ -7,6 +7,7 @@ const { processChunkedSync } = require("./shared/common");
 const eventConfig = require("./config");
 
 const COMPONENT_NAME = "eventQueue/periodicEvents";
+const CHUNK_SIZE_INSERT_PERIODIC_EVENTS = 4;
 
 const checkAndInsertPeriodicEvents = async (context) => {
   const tx = cds.tx(context);
@@ -84,13 +85,17 @@ const checkAndInsertPeriodicEvents = async (context) => {
 
 const insertPeriodEvents = async (tx, events) => {
   const startAfter = new Date();
-  processChunkedSync(events, 4, (chunk) => {
-    cds.log(COMPONENT_NAME).info("inserting changed or new periodic events", {
+  let counter = 1;
+  const chunks = Math.ceil(events.length / CHUNK_SIZE_INSERT_PERIODIC_EVENTS);
+  const logger = cds.log(COMPONENT_NAME);
+  processChunkedSync(events, CHUNK_SIZE_INSERT_PERIODIC_EVENTS, (chunk) => {
+    logger.info(`${counter}/${chunks} | inserting chunk of changed or new periodic events`, {
       events: chunk.map(({ type, subType }) => {
         const { interval } = eventConfig.getEventConfig(type, subType);
         return { type, subType, interval };
       }),
     });
+    counter++;
   });
   const periodEventsInsert = events.map((periodicEvent) => ({
     type: periodicEvent.type,

package/src/shared/WorkerQueue.js

@@ -82,7 +82,7 @@ class WorkerQueue {
    **/
   static get instance() {
     if (!WorkerQueue.#instance) {
-      WorkerQueue.#instance = new WorkerQueue(config.parallelTenantProcessing);
+      WorkerQueue.#instance = new WorkerQueue(config.instanceLoadLimit);
     }
     return WorkerQueue.#instance;
   }

package/src/shared/eventScheduler.js

@@ -19,7 +19,7 @@ class EventScheduler {
       return; // event combination already scheduled
     }
     this.#scheduledEvents[key] = true;
-    cds.log(COMPONENT_NAME).info("scheduling event queue run for delayed event", {
+    cds.log(COMPONENT_NAME).debug("scheduling event queue run for delayed event", {
       type,
       subType,
       delaySeconds: (date.getTime() - Date.now()) / 1000,