@cap-js-community/event-queue 0.2.3 → 0.2.5

package/README.md

@@ -5,22 +5,24 @@
 [![REUSE status](https://api.reuse.software/badge/github.com/cap-js-community/event-queue)](https://api.reuse.software/info/github.com/cap-js-community/event-queue)
 [![CI Main](https://github.com/cap-js-community/event-queue/actions/workflows/main-ci.yml/badge.svg)](https://github.com/cap-js-community/event-queue/commits/main)
 
-The Event-Queue is a framework built on top of CAP Node.js, specifically designed to enable efficient and streamlined
-asynchronous event processing in a multi-tenancy environment. With a strong emphasis on load balancing, this package
-ensures optimal distribution of workload across all available application instances. By offering managed tenant-specific
-transactions, similar to CAP handlers, the Event-Queue framework simplifies event and asynchronous processing, thereby
-enhancing the overall performance of your application.
+The Event-Queue is a framework built on top of CAP Node.js, designed specifically for efficient and
+streamlined asynchronous event processing. With a focus on load balancing, this package ensures optimal
+event distribution across all available application instances. By providing managed transactions similar to CAP
+handlers, the Event-Queue framework simplifies event processing, enhancing the overall performance of your application.
+
+Additionally, Event-Queue provides support for [periodic events](https://cap-js-community.github.io/event-queue/configure-event/#periodic-events),
+allowing for processing at defined intervals. This feature further extends its capabilities in load balancing and
+transaction management, ensuring that even regularly occurring tasks are handled efficiently and effectively without
+overloading any single instance. This makes it an ideal solution for applications needing consistent, reliable event processing.
 
 ## Getting started
 
 - Run `npm add @cap-js-community/event-queue` in `@sap/cds` project
-- Activate the cds-plugin in the cds section of the package.sjon
+- Activate the cds-plugin in the cds section of the package.json.
 
 ### As cds-plugin
 
-Extend the cds section of your package.json. Reference to the cds-plugin section in the capire documentation about the
-cds-plugin concept.
-https://cap.cloud.sap/docs/releases/march23#new-cds-plugin-technique
+For detailed information check out the [documentation](https://cap-js-community.github.io/event-queue/setup).
 
 ```json
 {
@@ -35,13 +37,16 @@ https://cap.cloud.sap/docs/releases/march23#new-cds-plugin-technique
 
 ## Features
 
-- load balancing of event processing throughout app instances
-- control concurrency in app instances
-- managed transactions for event processing
+Learn more about features in the [documentation](https://cap-js-community.github.io/event-queue/#functionality-and-problem-solutions). To compare the
+event-queue with other SAP products head over to [Distinction from other solutions](https://cap-js-community.github.io/event-queue/diff-to-outbox/).
+
+- [load balancing](https://cap-js-community.github.io/event-queue/load-balancing) and concurrency control of event processing across app instances
+- [periodic events](https://cap-js-community.github.io/event-queue/configure-event/#periodic-events) similar to running cron jobs for business processes
+- [managed transactions](https://cap-js-community.github.io/event-queue/transaction-handling) for event processing
 - async processing of processing intensive tasks for better UI responsiveness
 - push/pull mechanism for reducing delay between publish an event and processing
 - cluster published events during processing (e.g. for combining multiple E-Mail events to one E-Mail)
-- plug and play via cds-plugin
+- [plug and play](https://cap-js-community.github.io/event-queue/setup) via cds-plugin
 
 ## Documentation
 
@@ -50,7 +55,7 @@ Head over to our [Documentation](https://cap-js-community.github.io/event-queue/
 ## Support, Feedback, Contributing
 
 This project is open to feature requests/suggestions, bug reports etc.
-via [GitHub issues](https://github.com/cap-js-community/<your-project>/issues). Contribution and feedback are encouraged
+via [GitHub issues](https://github.com/cap-js-communityevent-queue/issues). Contribution and feedback are encouraged
 and always welcome. For more information about how to contribute, the project structure, as well as additional
 contribution information, see our [Contribution Guidelines](CONTRIBUTING.md).
 
@@ -62,7 +67,7 @@ times.
 
 ## Licensing
 
-Copyright 2023 SAP SE or an SAP affiliate company and `@cap-js-community/event-queue contributors`. Please see
+Copyright 2023 SAP SE or an SAP affiliate company and `@cap-js-community/event-queue` contributors. Please see
 our [LICENSE](LICENSE) for copyright and license information. Detailed information including third-party components and
 their licensing/copyright information is
 available [via the REUSE tool](https://api.reuse.software/info/github.com/cap-js-community/<your-project>).

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@cap-js-community/event-queue",
-  "version": "0.2.3",
+  "version": "0.2.5",
   "description": "An event queue that enables secure transactional processing of asynchronous events, featuring instant event processing with Redis Pub/Sub and load distribution across all application instances.",
   "main": "src/index.js",
   "files": [
@@ -31,7 +31,7 @@
     "lint:ci": "npm run eslint:ci && npm run prettier:ci",
     "eslint": "eslint --fix .",
     "eslint:ci": "eslint .",
-    "prettier": "prettier --write --log-level error .",
+    "prettier": "prettier --write --loglevel error .",
     "prettier:ci": "prettier --check .",
     "prepareRelease": "npm prune --production",
     "docs": "cd docs && bundle exec jekyll serve",
@@ -42,14 +42,14 @@
     "node": ">=16"
   },
   "dependencies": {
-    "redis": "4.6.10",
+    "redis": "4.6.11",
     "verror": "1.10.1",
     "yaml": "2.3.4"
   },
   "devDependencies": {
-    "@sap/cds": "7.3.1",
-    "@sap/cds-dk": "7.3.1",
-    "eslint": "8.52.0",
+    "@sap/cds": "7.4.0",
+    "@sap/cds-dk": "7.4.0",
+    "eslint": "8.54.0",
     "eslint-config-prettier": "9.0.0",
     "eslint-plugin-jest": "27.6.0",
     "eslint-plugin-node": "11.1.0",

package/src/EventQueueError.js

@@ -16,6 +16,7 @@ const ERROR_CODES = {
   MISSING_IMPL: "MISSING_IMPL",
   DUPLICATE_EVENT_REGISTRATION: "DUPLICATE_EVENT_REGISTRATION",
   NO_MANUEL_INSERT_OF_PERIODIC: "NO_MANUEL_INSERT_OF_PERIODIC",
+  LOAD_HIGHER_THAN_LIMIT: "LOAD_HIGHER_THAN_LIMIT",
 };
 
 const ERROR_CODES_META = {
@@ -59,6 +60,9 @@ const ERROR_CODES_META = {
   [ERROR_CODES.NO_MANUEL_INSERT_OF_PERIODIC]: {
     message: "Periodic events are managed by the framework and are not allowed to insert manually.",
   },
+  [ERROR_CODES.LOAD_HIGHER_THAN_LIMIT]: {
+    message: "The defined load of an event is higher than the maximum defined limit. Check your configuration!",
+  },
 };
 
 class EventQueueError extends VError {
@@ -206,6 +210,16 @@ class EventQueueError extends VError {
       message
     );
   }
+  static loadHigherThanLimit(load) {
+    const { message } = ERROR_CODES_META[ERROR_CODES.LOAD_HIGHER_THAN_LIMIT];
+    return new EventQueueError(
+      {
+        name: ERROR_CODES.LOAD_HIGHER_THAN_LIMIT,
+        info: { load },
+      },
+      message
+    );
+  }
 }
 
 module.exports = EventQueueError;

package/src/EventQueueProcessorBase.js

@@ -98,6 +98,20 @@ class EventQueueProcessorBase {
     throw new Error(IMPLEMENT_ERROR_MESSAGE);
   }
 
+  /**
+   * Process one periodic event
+   * @param processContext the context valid for the event processing. This context is associated with a valid transaction
+   *                       Access to the context is also possible with this.getContextForEventProcessing(key).
+   *                       The associated tx can be accessed with this.getTxForEventProcessing(key).
+   * @param {string} key cluster key generated during the clustering step. By default, this is ID of the event queue entry
+   * @param {Object} queueEntry this is the queueEntry which should be processed
+   * @returns {Promise<undefined>}
+   */
+  // eslint-disable-next-line no-unused-vars
+  async processPeriodicEvent(processContext, key, queueEntry) {
+    throw new Error(IMPLEMENT_ERROR_MESSAGE);
+  }
+
   startPerformanceTracerEvents() {
     this.__performanceLoggerEvents = new PerformanceTracer(this.logger, "Processing events");
   }
@@ -254,7 +268,8 @@ class EventQueueProcessorBase {
         this.#determineAndAddEventStatusToMap(queueEntry.ID, EventProcessingStatus.Error, statusMap)
       );
       this.logger.error(
-        `The supplied status tuple doesn't have the required structure. Setting all entries to error. Error: ${error.toString()}`,
+        "The supplied status tuple doesn't have the required structure. Setting all entries to error.",
+        error,
         {
           eventType: this.#eventType,
           eventSubType: this.#eventSubType,
@@ -295,7 +310,8 @@ class EventQueueProcessorBase {
   handleErrorDuringProcessing(error, queueEntries) {
     queueEntries = Array.isArray(queueEntries) ? queueEntries : [queueEntries];
     this.logger.error(
-      `Caught error during event processing - setting queue entry to error. Please catch your promises/exceptions. Error: ${error}`,
+      "Caught error during event processing - setting queue entry to error. Please catch your promises/exceptions",
+      error,
       {
         eventType: this.#eventType,
         eventSubType: this.#eventSubType,
@@ -309,14 +325,11 @@ class EventQueueProcessorBase {
   }
 
   handleErrorDuringPeriodicEventProcessing(error, queueEntry) {
-    this.logger.error(
-      `Caught error during event periodic processing. Please catch your promises/exceptions. Error: ${error}`,
-      {
-        eventType: this.#eventType,
-        eventSubType: this.#eventSubType,
-        queueEntryId: queueEntry.ID,
-      }
-    );
+    this.logger.error("Caught error during event periodic processing. Please catch your promises/exceptions.", error, {
+      eventType: this.#eventType,
+      eventSubType: this.#eventSubType,
+      queueEntryId: queueEntry.ID,
+    });
   }
 
   async setPeriodicEventStatus(queueEntryIds) {
@@ -491,7 +504,7 @@ class EventQueueProcessorBase {
   }
 
   handleErrorDuringClustering(error) {
-    this.logger.error(`Error during clustering of events - setting all queue entries to error. Error: ${error}`, {
+    this.logger.error("Error during clustering of events - setting all queue entries to error.", error, {
       eventType: this.#eventType,
       eventSubType: this.#eventSubType,
     });
@@ -521,7 +534,7 @@ class EventQueueProcessorBase {
    */
   async getQueueEntriesAndSetToInProgress() {
     let result = [];
-    const refDateStartAfter = new Date(Date.now() + this.#config.runInterval);
+    const refDateStartAfter = new Date(Date.now() + this.#config.runInterval * 1.2);
     await executeInNewTransaction(this.__baseContext, "eventQueue-getQueueEntriesAndSetToInProgress", async (tx) => {
       const entries = await tx.run(
         SELECT.from(this.#config.tableNameEventQueue)
@@ -573,7 +586,7 @@ class EventQueueProcessorBase {
       this.#handleDelayedEvents(delayedEvents);
 
       result = openEvents;
-      this.logger.info("Selected event queue entries for processing", {
+      this.logger[eventsForProcessing.length ? "info" : "debug"]("Selected event queue entries for processing", {
         openEvents: openEvents.length,
         ...(delayedEvents.length && { delayedEvents: delayedEvents.length }),
         ...(exceededTries.length && { exceededTries: exceededTries.length }),
@@ -626,7 +639,7 @@ class EventQueueProcessorBase {
   }
 
   #clusterEvents(events, refDateStartAfter) {
-    const refDate = new Date(refDateStartAfter.getTime() - this.#config.runInterval + EVENT_START_AFTER_HEADROOM);
+    const refDate = new Date(refDateStartAfter.getTime() - this.#config.runInterval * 1.2 + EVENT_START_AFTER_HEADROOM);
     return events.reduce(
       (result, event) => {
         if (event.attempts === this.__retryAttempts + TRIES_FOR_EXCEEDED_EVENTS) {
@@ -677,7 +690,8 @@ class EventQueueProcessorBase {
             await this.#persistEventQueueStatusForExceeded(this.tx, [exceededEvent], EventProcessingStatus.Exceeded);
           } catch (err) {
             this.logger.error(
-              `Caught error during hook for exceeded events - setting queue entry to error. Please catch your promises/exceptions. Error: ${err}`,
+              "Caught error during hook for exceeded events - setting queue entry to error. Please catch your promises/exceptions.",
+              err,
               {
                 eventType: this.#eventType,
                 eventSubType: this.#eventSubType,
@@ -822,6 +836,10 @@ class EventQueueProcessorBase {
       [this.#eventType, this.#eventSubType].join("##")
     );
     if (!lockAcquired) {
+      this.logger.debug("no lock available, exit processing", {
+        type: this.#eventType,
+        subType: this.#eventSubType,
+      });
       return false;
     }
     this.__lockAcquired = true;
@@ -835,16 +853,16 @@ class EventQueueProcessorBase {
     try {
       await distributedLock.releaseLock(this.context, [this.#eventType, this.#eventSubType].join("##"));
     } catch (err) {
-      this.logger.error("Releasing distributed lock failed. Error:", err.toString());
+      this.logger.error("Releasing distributed lock failed.", err);
     }
   }
 
   async scheduleNextPeriodEvent(queueEntry) {
-    const interval = this.#eventConfig.interval;
+    const intervalInSec = this.#eventConfig.interval * 1000;
     const newEvent = {
       type: this.#eventType,
       subType: this.#eventSubType,
-      startAfter: new Date(new Date(queueEntry.startAfter).getTime() + interval * 1000),
+      startAfter: new Date(new Date(queueEntry.startAfter).getTime() + intervalInSec),
     };
     const { relative } = this.#eventSchedulerInstance.calculateOffset(
       this.#eventType,
@@ -853,7 +871,7 @@ class EventQueueProcessorBase {
     );
 
     // more than one interval behind - shift tick to keep up
-    if (relative < 0 && Math.abs(relative) >= this.#eventConfig.interval * 1000) {
+    if (relative < 0 && Math.abs(relative) >= intervalInSec) {
       newEvent.startAfter = new Date(Date.now() + 5 * 1000);
       this.logger.info("interval adjusted because shifted more than one interval", {
         eventType: this.#eventType,
@@ -865,7 +883,7 @@ class EventQueueProcessorBase {
     this.tx._skipEventQueueBroadcase = true;
     await this.tx.run(INSERT.into(this.#config.tableNameEventQueue).entries({ ...newEvent }));
     this.tx._skipEventQueueBroadcase = false;
-    if (interval < this.#config.runInterval) {
+    if (intervalInSec < this.#config.runInterval * 1.5) {
       this.#handleDelayedEvents([newEvent]);
       const { relative: relativeAfterSchedule } = this.#eventSchedulerInstance.calculateOffset(
         this.#eventType,
@@ -923,6 +941,10 @@ class EventQueueProcessorBase {
     return this.__logger ?? this.__baseLogger;
   }
 
+  set logger(value) {
+    this.__logger = value;
+  }
+
   get queueEntriesWithPayloadMap() {
     return this.#queueEntriesWithPayloadMap;
   }

package/src/config.js

@@ -11,6 +11,7 @@ const GLOBAL_TX_TIMEOUT = 30 * 60 * 1000;
 const REDIS_CONFIG_CHANNEL = "EVENT_QUEUE_CONFIG_CHANNEL";
 const COMPONENT_NAME = "eventQueue/config";
 const MIN_INTERVAL_SEC = 10;
+const DEFAULT_LOAD = 1;
 
 class Config {
   #logger;
@@ -20,13 +21,14 @@ class Config {
   #runInterval;
   #redisEnabled;
   #initialized;
-  #parallelTenantProcessing;
+  #instanceLoadLimit;
   #tableNameEventQueue;
   #tableNameEventLock;
   #isRunnerDeactivated;
   #configFilePath;
   #processEventsAfterPublish;
   #skipCsnCheck;
+  #registerAsEventProcessor;
   #disableRedis;
   #env;
   #eventMap;
@@ -40,7 +42,7 @@ class Config {
     this.#runInterval = null;
     this.#redisEnabled = null;
     this.#initialized = false;
-    this.#parallelTenantProcessing = null;
+    this.#instanceLoadLimit = null;
     this.#tableNameEventQueue = null;
     this.#tableNameEventLock = null;
     this.#isRunnerDeactivated = false;
@@ -106,11 +108,13 @@ class Config {
     config.events = config.events ?? [];
     config.periodicEvents = config.periodicEvents ?? [];
     this.#eventMap = config.events.reduce((result, event) => {
+      event.load = event.load ?? DEFAULT_LOAD;
       this.validateAdHocEvents(result, event);
       result[[event.type, event.subType].join("##")] = event;
       return result;
     }, {});
     this.#eventMap = config.periodicEvents.reduce((result, event) => {
+      event.load = event.load ?? DEFAULT_LOAD;
       const SUFFIX_PERIODIC = "_PERIODIC";
       event.type = `${event.type}${SUFFIX_PERIODIC}`;
       event.isPeriodic = true;
@@ -210,12 +214,12 @@ class Config {
     this.#initialized = value;
   }
 
-  get parallelTenantProcessing() {
-    return this.#parallelTenantProcessing;
+  get instanceLoadLimit() {
+    return this.#instanceLoadLimit;
   }
 
-  set parallelTenantProcessing(value) {
-    this.#parallelTenantProcessing = value;
+  set instanceLoadLimit(value) {
+    this.#instanceLoadLimit = value;
   }
 
   get tableNameEventQueue() {
@@ -274,6 +278,14 @@ class Config {
     return this.#updatePeriodicEvents;
   }
 
+  set registerAsEventProcessor(value) {
+    this.#registerAsEventProcessor = value;
+  }
+
+  get registerAsEventProcessor() {
+    return this.#registerAsEventProcessor;
+  }
+
   get isMultiTenancy() {
     return !!cds.requires.multitenancy;
   }

package/src/initialize.js

@@ -29,7 +29,7 @@ const CONFIG_VARS = [
   ["processEventsAfterPublish", true],
   ["isRunnerDeactivated", false],
   ["runInterval", 5 * 60 * 1000],
-  ["parallelTenantProcessing", 5],
+  ["instanceLoadLimit", 20],
   ["tableNameEventQueue", BASE_TABLES.EVENT],
   ["tableNameEventLock", BASE_TABLES.LOCK],
   ["disableRedis", false],
@@ -43,7 +43,7 @@ const initialize = async ({
   processEventsAfterPublish,
   isRunnerDeactivated,
   runInterval,
-  parallelTenantProcessing,
+  instanceLoadLimit,
   tableNameEventQueue,
   tableNameEventLock,
   disableRedis,
@@ -52,7 +52,7 @@ const initialize = async ({
 } = {}) => {
   // TODO: initialize check:
   // - content of yaml check
-  // - betweenRuns and parallelTenantProcessing
+  // - betweenRuns and instanceLoadLimit
 
   if (config.initialized) {
     return;
@@ -65,7 +65,7 @@ const initialize = async ({
     processEventsAfterPublish,
     isRunnerDeactivated,
     runInterval,
-    parallelTenantProcessing,
+    instanceLoadLimit,
     tableNameEventQueue,
     tableNameEventLock,
     disableRedis,
@@ -88,10 +88,11 @@ const initialize = async ({
   registerCdsShutdown();
   logger.info("event queue initialized", {
     registerAsEventProcessor: config.registerAsEventProcessor,
+    processEventsAfterPublish: config.processEventsAfterPublish,
     multiTenancyEnabled: config.isMultiTenancy,
     redisEnabled: config.redisEnabled,
     runInterval: config.runInterval,
-    config: config.parallelTenantProcessing,
+    config: config.instanceLoadLimit,
   });
 };
 

package/src/periodicEvents.js

@@ -7,6 +7,7 @@ const { processChunkedSync } = require("./shared/common");
 const eventConfig = require("./config");
 
 const COMPONENT_NAME = "eventQueue/periodicEvents";
+const CHUNK_SIZE_INSERT_PERIODIC_EVENTS = 4;
 
 const checkAndInsertPeriodicEvents = async (context) => {
   const tx = cds.tx(context);
@@ -63,12 +64,15 @@ const checkAndInsertPeriodicEvents = async (context) => {
     cds.log(COMPONENT_NAME).info("deleting periodic events because they have changed", {
       changedEvents: exitingWithNotMatchingInterval.map(({ type, subType }) => ({ type, subType })),
     });
-  await tx.run(
-    DELETE.from(eventConfig.tableNameEventQueue).where(
-      "ID IN",
-      exitingWithNotMatchingInterval.map(({ ID }) => ID)
-    )
-  );
+
+  if (exitingWithNotMatchingInterval.length) {
+    await tx.run(
+      DELETE.from(eventConfig.tableNameEventQueue).where(
+        "ID IN",
+        exitingWithNotMatchingInterval.map(({ ID }) => ID)
+      )
+    );
+  }
 
   const newOrChangedEvents = newEvents.concat(exitingWithNotMatchingInterval);
 
@@ -81,13 +85,17 @@ const checkAndInsertPeriodicEvents = async (context) => {
 
 const insertPeriodEvents = async (tx, events) => {
   const startAfter = new Date();
-  processChunkedSync(events, 4, (chunk) => {
-    cds.log(COMPONENT_NAME).info("inserting changed or new periodic events", {
+  let counter = 1;
+  const chunks = Math.ceil(events.length / CHUNK_SIZE_INSERT_PERIODIC_EVENTS);
+  const logger = cds.log(COMPONENT_NAME);
+  processChunkedSync(events, CHUNK_SIZE_INSERT_PERIODIC_EVENTS, (chunk) => {
+    logger.info(`${counter}/${chunks} | inserting chunk of changed or new periodic events`, {
       events: chunk.map(({ type, subType }) => {
         const { interval } = eventConfig.getEventConfig(type, subType);
         return { type, subType, interval };
       }),
     });
+    counter++;
   });
   const periodEventsInsert = events.map((periodicEvent) => ({
     type: periodicEvent.type,

package/src/processEventQueue.js

@@ -6,23 +6,13 @@ const cds = require("@sap/cds");
 
 const config = require("./config");
 const { TransactionMode } = require("./constants");
-const { limiter, Funnel } = require("./shared/common");
+const { limiter } = require("./shared/common");
 
 const { executeInNewTransaction, TriggerRollback } = require("./shared/cdsHelper");
 
 const COMPONENT_NAME = "eventQueue/processEventQueue";
 const MAX_EXECUTION_TIME = 5 * 60 * 1000;
 
-const eventQueueRunner = async (context, events) => {
-  const startTime = new Date();
-  const funnel = new Funnel();
-  await Promise.allSettled(
-    events.map((event) =>
-      funnel.run(event.load, async () => processEventQueue(context, event.type, event.subType, startTime))
-    )
-  );
-};
-
 const processEventQueue = async (context, eventType, eventSubType, startTime = new Date()) => {
   let iterationCounter = 0;
   let shouldContinue = true;
@@ -111,7 +101,7 @@ const processEventQueue = async (context, eventType, eventSubType, startTime = n
       }
     }
   } catch (err) {
-    cds.log(COMPONENT_NAME).error("Processing event queue failed with unexpected error. Error:", err, {
+    cds.log(COMPONENT_NAME).error("Processing event queue failed with unexpected error.", err, {
       eventType,
       eventSubType,
     });
@@ -170,7 +160,7 @@ const processPeriodicEvent = async (eventTypeInstance) => {
           eventTypeInstance.processEventContext = tx.context;
           eventTypeInstance.setTxForEventProcessing(queueEntry.ID, cds.tx(tx.context));
           try {
-            await eventTypeInstance.processEvent(tx.context, queueEntry.ID, [queueEntry]);
+            await eventTypeInstance.processPeriodicEvent(tx.context, queueEntry.ID, queueEntry);
           } catch (err) {
             eventTypeInstance.handleErrorDuringPeriodicEventProcessing(err, queueEntry);
             throw new TriggerRollback();
@@ -194,7 +184,7 @@ const processPeriodicEvent = async (eventTypeInstance) => {
       );
     }
   } catch (err) {
-    cds.log(COMPONENT_NAME).error("Processing periodic events failed with unexpected error. Error:", err, {
+    cds.log(COMPONENT_NAME).error("Processing periodic events failed with unexpected error.", err, {
       eventType: eventTypeInstance?.eventType,
       eventSubType: eventTypeInstance?.eventSubType,
     });
@@ -278,5 +268,4 @@ const resilientRequire = (path) => {
 
 module.exports = {
   processEventQueue,
-  eventQueueRunner,
 };

package/src/redisPubSub.js

@@ -48,7 +48,10 @@ const broadcastEvent = async (tenantId, type, subType) => {
       [type, subType].join("##")
     );
     if (result) {
-      logger.info("skip publish redis event as no lock is available");
+      logger.info("skip publish redis event as no lock is available", {
+        type,
+        subType,
+      });
       return;
     }
     logger.debug("publishing redis event", {
@@ -58,7 +61,7 @@ const broadcastEvent = async (tenantId, type, subType) => {
     });
     await redis.publishMessage(EVENT_MESSAGE_CHANNEL, JSON.stringify({ tenantId, type, subType }));
   } catch (err) {
-    logger.error(`publish event failed with error: ${err.toString()}`, {
+    logger.error("publish event failed!", err, {
       tenantId,
       type,
       subType,

package/src/runner.js

@@ -3,8 +3,8 @@
 const { randomUUID } = require("crypto");
 
 const eventQueueConfig = require("./config");
-const { eventQueueRunner, processEventQueue } = require("./processEventQueue");
-const { getWorkerPoolInstance } = require("./shared/WorkerQueue");
+const { processEventQueue } = require("./processEventQueue");
+const WorkerQueue = require("./shared/WorkerQueue");
 const cdsHelper = require("./shared/cdsHelper");
 const distributedLock = require("./shared/distributedLock");
 const SetIntervalDriftSafe = require("./shared/SetIntervalDriftSafe");
@@ -21,7 +21,7 @@ const OFFSET_FIRST_RUN = 10 * 1000;
 let tenantIdHash;
 let singleRunDone;
 
-const singleTenant = () => _scheduleFunction(_checkPeriodicEventsSingleTenant, _executeRunForTenant);
+const singleTenant = () => _scheduleFunction(_checkPeriodicEventsSingleTenant, _singleTenantDb);
 
 const multiTenancyDb = () => _scheduleFunction(_multiTenancyPeriodicEvents, _multiTenancyDb);
 
@@ -75,7 +75,7 @@ const _multiTenancyRedis = async () => {
     return;
   }
 
-  _executeAllTenants(tenantIds, runId);
+  return _executeEventsAllTenants(tenantIds, runId);
 };
 
 const _checkAndTriggerPeriodicEventUpdate = (tenantIds) => {
@@ -87,15 +87,43 @@ const _checkAndTriggerPeriodicEventUpdate = (tenantIds) => {
   if (tenantIdHash && tenantIdHash !== hash) {
     cds.log(COMPONENT_NAME).info("tenant id hash changed, triggering updating periodic events!");
     _multiTenancyPeriodicEvents().catch((err) => {
-      cds.log(COMPONENT_NAME).error("Error during triggering updating periodic events! Error:", err);
+      cds.log(COMPONENT_NAME).error("Error during triggering updating periodic events!", err);
     });
   }
 };
 
-const _executeAllTenantsGeneric = (tenantIds, runId, fn) => {
-  const workerQueueInstance = getWorkerPoolInstance();
+const _executeEventsAllTenants = (tenantIds, runId) => {
+  const events = eventQueueConfig.allEvents;
+  const promises = [];
   tenantIds.forEach((tenantId) => {
-    workerQueueInstance.addToQueue(async () => {
+    events.forEach((event) => {
+      promises.push(
+        WorkerQueue.instance.addToQueue(event.load, async () => {
+          try {
+            const lockId = `${runId}_${event.type}_${event.subType}`;
+            const tenantContext = new cds.EventContext({ tenant: tenantId });
+            const couldAcquireLock = await distributedLock.acquireLock(tenantContext, lockId, {
+              expiryTime: eventQueueConfig.runInterval * 0.95,
+            });
+            if (!couldAcquireLock) {
+              return;
+            }
+            await runEventCombinationForTenant(tenantId, event.type, event.subType, true);
+          } catch (err) {
+            cds.log(COMPONENT_NAME).error("executing event-queue run for tenant failed", {
+              tenantId,
+            });
+          }
+        })
+      );
+    });
+  });
+  return promises;
+};
+
+const _executePeriodicEventsAllTenants = (tenantIds, runId) => {
+  tenantIds.forEach((tenantId) => {
+    WorkerQueue.instance.addToQueue(1, async () => {
       try {
         const tenantContext = new cds.EventContext({ tenant: tenantId });
         const couldAcquireLock = await distributedLock.acquireLock(tenantContext, runId, {
@@ -104,7 +132,7 @@ const _executeAllTenantsGeneric = (tenantIds, runId, fn) => {
         if (!couldAcquireLock) {
           return;
         }
-        await fn(tenantId, runId);
+        await _checkPeriodicEventsSingleTenant(tenantId);
       } catch (err) {
         cds.log(COMPONENT_NAME).error("executing event-queue run for tenant failed", {
           tenantId,
@@ -114,34 +142,20 @@ const _executeAllTenantsGeneric = (tenantIds, runId, fn) => {
   });
 };
 
-const _executeAllTenants = (tenantIds, runId) => _executeAllTenantsGeneric(tenantIds, runId, _executeRunForTenant);
-
-const _executePeriodicEventsAllTenants = (tenantIds, runId) =>
-  _executeAllTenantsGeneric(tenantIds, runId, _checkPeriodicEventsSingleTenant);
-
-const _executeRunForTenant = async (tenantId, runId) => {
-  const logger = cds.log(COMPONENT_NAME);
-  try {
-    const eventsForAutomaticRun = eventQueueConfig.allEvents;
-    const subdomain = await cdsHelper.getSubdomainForTenantId(tenantId);
-    const context = new cds.EventContext({
-      tenant: tenantId,
-      // NOTE: we need this because of logging otherwise logs would not contain the subdomain
-      http: { req: { authInfo: { getSubdomain: () => subdomain } } },
-    });
-    cds.context = context;
-    logger.info("executing eventQueue run", {
-      tenantId,
-      subdomain,
-      ...(runId ? { runId } : null),
-    });
-    await eventQueueRunner(context, eventsForAutomaticRun);
-  } catch (err) {
-    logger.error(`Couldn't process eventQueue for tenant! Next try after defined interval. Error: ${err}`, {
-      tenantId,
-      redisEnabled: eventQueueConfig.redisEnabled,
+const _singleTenantDb = async (tenantId) => {
+  const events = eventQueueConfig.allEvents;
+  events.forEach((event) => {
+    WorkerQueue.instance.addToQueue(event.load, async () => {
+      try {
+        await runEventCombinationForTenant(tenantId, event.type, event.subType, true);
+      } catch (err) {
+        cds.log(COMPONENT_NAME).error("executing event-queue run for tenant failed", {
+          tenantId,
+          redisEnabled: eventQueueConfig.redisEnabled,
+        });
+      }
     });
-  }
+  });
 };
 
 const _acquireRunId = async (context) => {
@@ -196,15 +210,12 @@ const _calculateOffsetForFirstRun = async () => {
   } catch (err) {
     cds
       .log(COMPONENT_NAME)
-      .error(
-        "calculating offset for first run failed, falling back to default. Runs might be out-of-sync. Error:",
-        err
-      );
+      .error("calculating offset for first run failed, falling back to default. Runs might be out-of-sync.", err);
   }
   return offsetDependingOnLastRun;
 };
 
-const runEventCombinationForTenant = async (tenantId, type, subType) => {
+const runEventCombinationForTenant = async (tenantId, type, subType, skipWorkerPool) => {
   try {
     const subdomain = await getSubdomainForTenantId(tenantId);
     const context = new cds.EventContext({
@@ -213,7 +224,15 @@ const runEventCombinationForTenant = async (tenantId, type, subType) => {
       http: { req: { authInfo: { getSubdomain: () => subdomain } } },
     });
     cds.context = context;
-    getWorkerPoolInstance().addToQueue(async () => await processEventQueue(context, type, subType));
+    if (skipWorkerPool) {
+      return await processEventQueue(context, type, subType);
+    } else {
+      const config = eventQueueConfig.getEventConfig(type, subType);
+      return await WorkerQueue.instance.addToQueue(
+        config.load,
+        async () => await processEventQueue(context, type, subType)
+      );
+    }
   } catch (err) {
     const logger = cds.log(COMPONENT_NAME);
     logger.error("error executing event combination for tenant", err, {
@@ -230,11 +249,9 @@ const _multiTenancyDb = async () => {
     logger.info("executing event queue run for single instance and multi tenant");
     const tenantIds = await cdsHelper.getAllTenantIds();
     _checkAndTriggerPeriodicEventUpdate(tenantIds);
-    _executeAllTenants(tenantIds, EVENT_QUEUE_RUN_ID);
+    return _executeEventsAllTenants(tenantIds, EVENT_QUEUE_RUN_ID);
   } catch (err) {
-    logger.error(
-      `Couldn't fetch tenant ids for event queue processing! Next try after defined interval. Error: ${err}`
-    );
+    logger.error("Couldn't fetch tenant ids for event queue processing! Next try after defined interval.", err);
   }
 };
 
@@ -245,14 +262,18 @@ const _multiTenancyPeriodicEvents = async () => {
     const tenantIds = await cdsHelper.getAllTenantIds();
     _executePeriodicEventsAllTenants(tenantIds, EVENT_QUEUE_RUN_PERIODIC_EVENT);
   } catch (err) {
-    logger.error(`Couldn't fetch tenant ids for updating periodic event processing! Error: ${err}`);
+    logger.error("Couldn't fetch tenant ids for updating periodic event processing!", err);
   }
 };
 
 const _checkPeriodicEventsSingleTenant = async (tenantId) => {
   const logger = cds.log(COMPONENT_NAME);
-  if (!eventQueueConfig.updatePeriodicEvents) {
-    logger.info("updating of periodic events is disabled");
+  if (!eventQueueConfig.updatePeriodicEvents || !eventQueueConfig.periodicEvents.length) {
+    logger.info("updating of periodic events is disabled or no periodic events configured", {
+      updateEnabled: eventQueueConfig.updatePeriodicEvents,
+      events: eventQueueConfig.periodicEvents.length,
+    });
+    return;
   }
   try {
     const subdomain = await cdsHelper.getSubdomainForTenantId(tenantId);
@@ -270,7 +291,7 @@ const _checkPeriodicEventsSingleTenant = async (tenantId) => {
       await periodicEvents.checkAndInsertPeriodicEvents(tx.context);
     });
   } catch (err) {
-    logger.error(`Couldn't process eventQueue for tenant! Next try after defined interval. Error: ${err}`, {
+    logger.error("Couldn't update periodic events for tenant! Next try after defined interval.", err, {
       tenantId,
       redisEnabled: eventQueueConfig.redisEnabled,
     });

package/src/shared/WorkerQueue.js

@@ -3,64 +3,106 @@
 const cds = require("@sap/cds");
 
 const config = require("../config");
+const EventQueueError = require("../EventQueueError");
 
 const COMPONENT_NAME = "eventQueue/WorkerQueue";
-
-let instance = null;
+const NANO_TO_MS = 1e6;
+const THRESHOLD = {
+  INFO: 5 * 1000,
+  WARN: 10 * 1000,
+  ERROR: 15 * 1000,
+};
 
 class WorkerQueue {
+  #concurrencyLimit;
+  #runningPromises;
+  #runningLoad;
+  #queue;
+  static #instance;
+
   constructor(concurrency) {
     if (Number.isNaN(concurrency) || concurrency <= 0) {
-      this.__concurrencyLimit = 1;
+      this.#concurrencyLimit = 1;
     } else {
-      this.__concurrencyLimit = concurrency;
+      this.#concurrencyLimit = concurrency;
     }
-    this.__runningPromises = [];
-    this.__queue = [];
+    this.#runningPromises = [];
+    this.#runningLoad = 0;
+    this.#queue = [];
   }
 
-  addToQueue(cb) {
+  addToQueue(load, cb) {
+    if (load > this.#concurrencyLimit) {
+      throw EventQueueError.loadHigherThanLimit(load);
+    }
+
+    const startTime = process.hrtime.bigint();
     const p = new Promise((resolve, reject) => {
-      this.__queue.push([cb, resolve, reject]);
+      this.#queue.push([load, cb, resolve, reject, startTime]);
     });
     this._checkForNext();
     return p;
   }
 
-  _executeFunction(cb, resolve, reject) {
+  _executeFunction(load, cb, resolve, reject, startTime) {
+    this.checkAndLogWaitingTime(startTime);
     const promise = Promise.resolve().then(() => cb());
-    this.__runningPromises.push(promise);
+    this.#runningPromises.push(promise);
+    this.#runningLoad = this.#runningLoad + load;
     promise
       .finally(() => {
-        this.__runningPromises.splice(this.__runningPromises.indexOf(promise), 1);
+        this.#runningLoad = this.#runningLoad - load;
+        this.#runningPromises.splice(this.#runningPromises.indexOf(promise), 1);
         this._checkForNext();
       })
       .then((...results) => {
         resolve(...results);
       })
       .catch((err) => {
-        cds.log(COMPONENT_NAME).error("Error happened in WorkQueue. Errors should be caught before! Error:", err);
+        cds.log(COMPONENT_NAME).error("Error happened in WorkQueue. Errors should be caught before!", err);
         reject(err);
       });
   }
 
   _checkForNext() {
-    if (!this.__queue.length || this.__runningPromises.length >= this.__concurrencyLimit) {
+    const load = this.#queue[0]?.[0];
+    if (!this.#queue.length || this.#runningLoad + load > this.#concurrencyLimit) {
       return;
     }
-    const [cb, resolve, reject] = this.__queue.shift();
-    this._executeFunction(cb, resolve, reject);
+    const args = this.#queue.shift();
+    this._executeFunction(...args);
+  }
+
+  get runningPromises() {
+    return this.#runningPromises;
   }
-}
 
-module.exports = {
-  getWorkerPoolInstance: () => {
-    if (!instance) {
-      instance = new WorkerQueue(config.parallelTenantProcessing);
+  /**
+   @return { WorkerQueue }
+   **/
+  static get instance() {
+    if (!WorkerQueue.#instance) {
+      WorkerQueue.#instance = new WorkerQueue(config.instanceLoadLimit);
     }
-    return instance;
-  },
-  _: {
-    WorkerQueue,
-  },
-};
+    return WorkerQueue.#instance;
+  }
+
+  checkAndLogWaitingTime(startTime) {
+    const diffMs = Math.round(Number(process.hrtime.bigint() - startTime) / NANO_TO_MS);
+    let logLevel;
+    if (diffMs >= THRESHOLD.ERROR) {
+      logLevel = "error";
+    } else if (diffMs >= THRESHOLD.WARN) {
+      logLevel = "warn";
+    } else if (diffMs >= THRESHOLD.INFO) {
+      logLevel = "info";
+    } else {
+      logLevel = "debug";
+    }
+    cds.log(COMPONENT_NAME)[logLevel]("Waiting time in worker queue", {
+      diffMs,
+    });
+  }
+}
+
+module.exports = WorkerQueue;

package/src/shared/common.js

@@ -1,9 +1,6 @@
 "use strict";
 
 const crypto = require("crypto");
-
-const { floor, abs, min } = Math;
-
 const arrayToFlatMap = (array, key = "ID") => {
   return array.reduce((result, element) => {
     result[element[key]] = element;
@@ -11,74 +8,6 @@ const arrayToFlatMap = (array, key = "ID") => {
   }, {});
 };
 
-/**
- * Establish a "Funnel" instance to limit how much
- * load can be processed in parallel. This is somewhat
- * similar to the limiter function however it has some
- * distinctintly different features. The Funnel will
- * not know in advance which functions and how many
- * loads it will have to process.
- */
-class Funnel {
-  /**
-   * Create a funnel with specified capacity
-   * @param capacity - the capacity of the funnel (integer, sign will be ignored)
-   */
-  constructor(capacity = 100) {
-    this.runningPromises = [];
-    this.capacity = floor(abs(capacity));
-  }
-
-  /**
-   * Asynchronously run a function that will put a specified load to the funnel.
-   * The total amount of load of all running functions shall not
-   * exceed the capacity of the funnel. If the desired load exceeds the capacity
-   * the funnel will wait until sufficient capacity is available.
-   * If a function requires a load >= capacity, then it will run
-   * exclusively.
-   * @param load - the load (integer, sign will be ignored)
-   * @param f
-   * @param args
-   * @return {Promise<unknown>}
-   */
-  async run(load, f, ...args) {
-    load = min(floor(abs(load)), Number.MAX_SAFE_INTEGER);
-
-    // wait for sufficient capacity
-    while (this.capacity < load && this.runningPromises.length > 0) {
-      try {
-        await Promise.race(this.runningPromises);
-      } catch {
-        // Yes, we must ignore exceptions here. The
-        // caller expects exceptions from f and no
-        // exceptions from other workloads.
-        // Other exceptions must be handled by the
-        // other callers. See (*) below.
-      }
-    }
-
-    // map function call to promise
-    const p = f.constructor.name === "AsyncFunction" ? f(...args) : Promise.resolve().then(() => f(...args));
-
-    // create promise for book keeping
-    const workload = p.finally(() => {
-      // remove workload
-      this.runningPromises.splice(this.runningPromises.indexOf(workload), 1);
-      // and reclaim its capacity
-      this.capacity += load;
-    });
-
-    // claim the capacity and schedule workload
-    this.capacity -= load;
-    this.runningPromises.push(workload);
-
-    // make the caller wait for the workload
-    // this also establish the seemingly missing
-    // exception handling. See (*) above.
-    return workload;
-  }
-}
-
 /**
  * Defines a promise that resolves when all payloads are processed by the iterator, but limits
  * the number concurrent executions.
@@ -132,4 +61,4 @@ const processChunkedSync = (inputs, chunkSize, chunkHandler) => {
 
 const hashStringTo32Bit = (value) => crypto.createHash("sha256").update(String(value)).digest("base64").slice(0, 32);
 
-module.exports = { arrayToFlatMap, Funnel, limiter, isValidDate, processChunkedSync, hashStringTo32Bit };
+module.exports = { arrayToFlatMap, limiter, isValidDate, processChunkedSync, hashStringTo32Bit };

package/src/shared/distributedLock.js

@@ -4,6 +4,8 @@ const redis = require("./redis");
 const config = require("../config");
 const cdsHelper = require("./cdsHelper");
 
+const KEY_PREFIX = "EVENT_QUEUE";
+
 const acquireLock = async (context, key, { tenantScoped = true, expiryTime = config.globalTxTimeout } = {}) => {
   const fullKey = _generateKey(context, tenantScoped, key);
   if (config.redisEnabled) {
@@ -128,7 +130,7 @@ const _generateKey = (context, tenantScoped, key) => {
   const keyParts = [];
   tenantScoped && keyParts.push(context.tenant);
   keyParts.push(key);
-  return keyParts.join("##");
+  return `${KEY_PREFIX}_${keyParts.join("##")}`;
 };
 
 module.exports = {

package/src/shared/eventScheduler.js

@@ -19,7 +19,7 @@ class EventScheduler {
       return; // event combination already scheduled
     }
     this.#scheduledEvents[key] = true;
-    cds.log(COMPONENT_NAME).info("scheduling event queue run for delayed event", {
+    cds.log(COMPONENT_NAME).debug("scheduling event queue run for delayed event", {
       type,
       subType,
       delaySeconds: (date.getTime() - Date.now()) / 1000,