ti2 1.0.95 → 1.0.96

package/controllers/README.md

@@ -1 +1,70 @@
-Place your controllers in this directory.
+# Bookings Controller Caching and Locking Strategy
+
+The `$bookingsProductSearch` function in `controllers/bookings.js` implements a sophisticated caching and locking strategy to optimize product searches, manage stale data, and prevent redundant operations. This document outlines its core flow and the locking mechanisms involved.
+
+## Core Flow of `$bookingsProductSearch`
+
+1.  **Initialization**:
+    *   Retrieves user, application (`appKey`), and token details (including `hint`).
+    *   Determines the specific plugin function to call for product search (e.g., `searchProducts` or `searchProductsForItinerary`).
+    *   Calculates a `cacheKey` based on `userId`, `hint`, and a static `operationId` ('bookingsProductSearch').
+    *   Defines two distinct lock keys derived from this `cacheKey`:
+        *   `pluginExecutionLockKey` (resolves to `${cacheKey}:lock`): Used to serialize direct calls to the plugin.
+        *   `jobQueueLockKey` (resolves to `${cacheKey}:jobLock`): Used to prevent multiple submissions of background refresh jobs.
+    *   Fetches the current cache content (`initialActualCacheContent`) and its `lastUpdated` timestamp.
+    *   Determines if the cache is stale (`isStaleByTTR`) based on the Time-To-Refresh (`ttr`) value from the token or plugin settings.
+    *   Checks for a `doNotCallPluginForProducts` flag (from token or plugin settings) and whether a `pluginExecutionLockKey` is currently active.
+
+2.  **Request Handling Logic (Simplified Order):**
+
+    *   **Condition 1: `doNotCallPluginForProducts` is true AND NOT `forceRefresh`**:
+        *   If this flag is set and the request is not a forced refresh, the system serves data directly from the cache if available.
+        *   If the cache is empty, it returns an empty product list.
+        *   No plugin calls are made, and no background jobs are queued in this path.
+
+    *   **Condition 2: `forceRefresh` is true**:
+        *   The system attempts to fetch fresh data directly from the plugin.
+        *   The `fetchFromPluginAndCache` helper function is invoked. This function:
+            *   Sets the `pluginExecutionLockKey` before calling the plugin to prevent other concurrent direct calls.
+            *   Calls the plugin's product search method.
+            *   If the plugin returns valid products, these are saved to the cache (both `cacheKey` for data and `${cacheKey}:lastUpdated` for timestamp).
+            *   Drops the `pluginExecutionLockKey` after completion.
+        *   The (potentially filtered) results from the plugin are returned to the client.
+
+    *   **Condition 3: Cache Exists AND NOT `forceRefresh`**:
+        *   **Sub-condition 3a: Cache is fresh OR `pluginExecutionLockKey` is active**:
+            *   If the cache is not stale according to its TTR, or if a direct plugin execution (like a `forceRefresh` or initial population) is already in progress (indicated by an active `pluginExecutionLockKey`), the system serves data from the existing cache.
+        *   **Sub-condition 3b: Cache is stale AND no `pluginExecutionLockKey` is active**:
+            *   The system checks for an active `jobQueueLockKey`.
+            *   If `jobQueueLockKey` IS active: It implies that another request very recently detected the stale cache and has already queued a background refresh job. The current request serves the stale data from the cache without queueing another job.
+            *   If `jobQueueLockKey` is NOT active: This request is the first (or among the first) to find the stale cache without an ongoing direct refresh or a recently queued job. It will:
+                1.  Set the `jobQueueLockKey` with a short TTL (e.g., 60 seconds).
+                2.  Queue a background job using `addJob`. This job will eventually call the plugin to refresh the data and then use `$updateProductSearchCache` to update the cache.
+                3.  Serve the stale data from the cache to the current client.
+
+    *   **Condition 4: No Cache Content AND NOT `forceRefresh` AND NOT `doNotCallPluginForProducts`**:
+        *   If there's no existing cache content and none of the preceding conditions (like `forceRefresh` or `doNotCallPluginForProducts`) were met, the system needs to populate the cache.
+        *   It calls `fetchFromPluginAndCache` (which sets `pluginExecutionLockKey`, calls the plugin, caches results, and drops the lock) to get initial data.
+        *   The (potentially filtered) results are returned to the client.
+
+## Locking Mechanisms Explained
+
+The system uses two types of locks, both based on the primary `cacheKey`, to manage concurrency and prevent redundant operations:
+
+1.  **`pluginExecutionLockKey` (derived from `${cacheKey}:lock`)**:
+    *   **Purpose**: To prevent multiple simultaneous *direct calls* to the external plugin for the same product search parameters. This is crucial during `forceRefresh` scenarios or when the cache is being populated for the first time by concurrent requests.
+    *   **Behavior**:
+        *   This lock is set by the `fetchFromPluginAndCache` helper function immediately before it makes an actual call to the plugin's `searchProducts` (or equivalent) method.
+        *   It is configured with a TTL (e.g., 120 seconds) to ensure it automatically expires if the process holding the lock crashes or fails to release it.
+        *   The lock is explicitly dropped by `fetchFromPluginAndCache` after the plugin call completes (whether successfully or with an error).
+        *   Other parts of the main logic (e.g., in Condition 3a) check for the presence of this lock (`hasPluginExecutionLock`). If active, it signals that a direct plugin data fetch is already in progress, prompting the current request to, for example, serve stale data or wait, rather than initiating another direct plugin call.
+
+2.  **`jobQueueLockKey` (derived from `${cacheKey}:jobLock`)**:
+    *   **Purpose**: To prevent the submission of multiple identical *background refresh jobs* by nearly simultaneous requests when the cache is found to be stale and no direct plugin execution (covered by `pluginExecutionLockKey`) is active.
+    *   **Behavior**:
+        *   This lock is checked specifically when the cache is determined to be effectively stale (`isEffectivelyStale`) and no `pluginExecutionLockKey` is currently active (Condition 3b).
+        *   If the `jobQueueLockKey` is NOT found in the cache, the current request assumes responsibility for queuing the refresh job. It sets this lock with a short TTL (e.g., 60 seconds) *before* calling `addJob`.
+        *   If the `jobQueueLockKey` IS found, it indicates that another request has very recently detected the stale state and has already taken action to queue the refresh job. The current request will then proceed to serve stale data without attempting to queue another job.
+        *   This mechanism ensures that even if numerous requests detect stale data at virtually the same moment, only one of them will succeed in setting the `jobQueueLockKey` and thereby be responsible for queueing the single background refresh task.
+
+These two locks work in tandem: `pluginExecutionLockKey` manages contention for direct, immediate plugin interactions, while `jobQueueLockKey` manages contention for initiating background refresh tasks when stale data is being served. This dual-lock strategy helps maintain system performance and avoids overwhelming external plugin services or the background job queue.

package/controllers/__tests__/bookings-searchProducts.js

@@ -0,0 +1,533 @@
+/* globals beforeAll describe it expect jest beforeEach */
+
+const chance = require('chance').Chance();
+const hash = require('object-hash');
+const cache = require('../../cache');
+// Remove direct import of listJobs, jobStatus as we'll mock addJob
+// const { listJobs, jobStatus } = require('../../worker/queue');
+
+// Mock the worker/queue module
+jest.mock('../../worker/queue', () => ({
+  ...jest.requireActual('../../worker/queue'), // Import and retain default behavior
+  addJob: jest.fn().mockResolvedValue({ id: 'mockJobId' }), // Mock addJob
+  // Provide mock for jobStatus as it might be called by other parts of the code or tests
+  jobStatus: jest.fn().mockResolvedValue({ status: 'completed', progress: 100, returnValue: null }),
+}));
+const { addJob, jobStatus: mockJobStatus } = require('../../worker/queue'); // Now addJob is the mock
+
+const testUtils = require('../../test/utils'); // Require the module itself
+
+// Global setup for the entire test file
+let globalDoApiPost, globalPlugins, globalUtils, globalSqldb;
+
+beforeAll(async () => {
+  // Initialize utils once for the entire test file
+  // Ensure all plugins needed across different describe blocks are listed here.
+  globalUtils = await testUtils({ plugins: ['lockTestPlugin', 'travelgate'] });
+  globalDoApiPost = globalUtils.doApiPost;
+  globalPlugins = globalUtils.plugins; // Array of instantiated plugins from the app instance
+  globalSqldb = globalUtils.sqldb; 
+});
+
+afterAll(async () => {
+  // Close DB connection after all tests in this file have run
+  if (globalSqldb && globalSqldb.sequelize && globalSqldb.sequelize.close) {
+    await globalSqldb.sequelize.close();
+  }
+});
+
+describe('user: bookings controller - searchProducts', () => {
+  let doApiGet;
+  let doApiPost;
+  let travelgatePlugin; // Specific plugin instance for this suite
+  let userToken;
+  let testAppName;
+  let testUserId;
+  let testPluginToken;
+  let testHint;
+
+  beforeAll(async () => {
+    // Use globally initialized utilities and plugins
+    doApiGet = globalUtils.doApiGet;
+    doApiPost = globalUtils.doApiPost;
+    travelgatePlugin = globalPlugins.find(p => p.name === 'travelgate');
+    if (!travelgatePlugin) {
+      throw new Error("travelgatePlugin not found. Ensure it's included in global testUtils setup.");
+    }
+
+    // Setup a new app, user, and integration using globalUtils.appSetup
+    // This appName ('travelgate') should match the plugin name we want to test.
+    const setupData = await globalUtils.appSetup({
+      appName: 'travelgate',
+      // userId will be generated by appSetup if not provided
+      // token and tokenHint will be default from appSetup unless specified
+    });
+    testAppName = setupData.newApp.name; // Should be 'travelgate'
+    testUserId = setupData.userId;
+    testPluginToken = setupData.token; // Token config for the 'travelgate' integration
+    testHint = setupData.hint;         // Hint for this 'travelgate' integration
+
+    userToken = globalUtils.createUserToken(testUserId); // JWT for API calls
+    expect(userToken).toBeTruthy();
+
+    // Drop cache keys for this specific context
+    const cacheKey = hash({
+      appKey: testAppName,
+      userId: testUserId,
+      hint: testHint, // Use the hint from the default integration
+      operationId: 'bookingsProductSearch',
+    });
+    await cache.drop({
+      pluginName: testAppName,
+      key: cacheKey,
+    });
+    await cache.drop({
+      pluginName: testAppName,
+      key: `${cacheKey}:lock`,
+    });
+  });
+
+  beforeEach(async () => {
+    // Clear mocks relevant to this suite.
+    if (travelgatePlugin && travelgatePlugin.searchProducts && travelgatePlugin.searchProducts.mockClear) {
+      travelgatePlugin.searchProducts.mockClear();
+    }
+    if (addJob && addJob.mockClear) { // Ensure addJob is cleared if tests in this suite might call it
+        addJob.mockClear();
+    }
+  });
+
+  describe('searchProducts', () => {
+    it('should be able to get bookings products for most users without special setup: no cache', async () => {
+      const payload = {};
+      const { products } = await doApiPost({
+        url: `/products/${testAppName}/${testUserId}/${testHint}/search`,
+        token: userToken,
+        payload,
+      });
+      expect(travelgatePlugin.searchProducts).toHaveBeenCalled();
+      expect(Array.isArray(products)).toBeTruthy();
+      expect(products.length).toBe(2);
+      expect(products[0].options.length).toBe(1);
+      expect(products[1].options.length).toBe(2);
+    });
+    it('should be able to get booking products: no cache, forceRefresh', async () => {
+      // NOTE: we SHOULD NOT need to remove the cache first, since we are forceRefreshing, we are testing the endpoint get's called while having a cache created
+      const payload = {
+        forceRefresh: true,
+      };
+      const { products } = await doApiPost({
+        url: `/products/${testAppName}/${testUserId}/${testHint}/search`,
+        token: userToken,
+        payload,
+      });
+      expect(travelgatePlugin.searchProducts).toHaveBeenCalled();
+      expect(Array.isArray(products)).toBeTruthy();
+      expect(products.length).toBe(2);
+      expect(products[0].options.length).toBe(1);
+      expect(products[1].options.length).toBe(2);
+      expect(travelgatePlugin.searchProducts.mock.calls[0][0].token).toEqual(testPluginToken);
+    });
+    describe('cache exists and is valid', () => {
+      it('should be able to get booking products: using cache', async () => {
+        const payload = {};
+        const { products } = await doApiPost({
+          url: `/products/${testAppName}/${testUserId}/${testHint}/search`,
+          token: userToken,
+          payload,
+        });
+        expect(travelgatePlugin.searchProducts).not.toHaveBeenCalled();
+        expect(Array.isArray(products)).toBeTruthy();
+        expect(products.length).toBe(2);
+        expect(products[0].options.length).toBe(1);
+        expect(products[1].options.length).toBe(2);
+      });
+      it('should be able to get booking products with searchInput', async () => {
+
+        const payload = {
+          searchInput: 'Transfer from Sydney Harbor to Hilton Hotel',
+        };
+        const { products } = await doApiPost({
+          url: `/products/${testAppName}/${testUserId}/${testHint}/search`,
+          token: userToken,
+          payload,
+        });
+        expect(travelgatePlugin.searchProducts).not.toHaveBeenCalled();
+        expect(Array.isArray(products)).toBeTruthy();
+        expect(products.length).toBe(1);
+        expect(products[0].productName).toBe('Davids');
+        expect(products[0].options.length).toBe(1);
+        expect(products[0].options[0].optionName).toBe('Transfer from Sydney Harbor Bridge to Hilton Hotel');
+      });
+    });
+    describe('doNotCallPluginForProducts flag', () => {
+      const doNotCallHint = 'hint_for_doNotCallPluginForProducts';
+      beforeAll(async () => {
+        // Create a specific integration for this test suite
+        const newIntegrationContent = {
+          endpoint: 'https://api.travelgatex.com', // Can be any valid URL
+          apiKey: chance.guid(),
+          client: 'tourconnect', // Or any other fields your plugin might expect
+          doNotCallPluginForProducts: true,
+        };
+        // Use globalUtils.appSetup to create this specific integration (UserAppKey)
+        await globalUtils.appSetup({
+            appName: testAppName, // Still 'travelgate' context for the plugin
+            userId: testUserId,   // Same user
+            tokenHint: doNotCallHint,
+            token: newIntegrationContent,
+        });
+        // Clear mocks that might have been called during appSetup
+        if (travelgatePlugin.searchProducts.mockClear) travelgatePlugin.searchProducts.mockClear();
+      });
+
+      it('productSearch should not call the plugin', async () => {
+        // first time call, no cache expect no plugin call and empty products
+        let products;
+        await doApiPost({
+          url: `/products/${testAppName}/${testUserId}/${doNotCallHint}/search`,
+          token: userToken,
+          payload: {},
+        }).then(({ products: p }) => {
+          products = p;
+        });
+        expect(travelgatePlugin.searchProducts).not.toHaveBeenCalled();
+        expect(Array.isArray(products)).toBeTruthy();
+        expect(products.length).toBe(0);
+
+      });
+      it('a forceRefresh should trigger the call to the plugin', async() => {
+        // second time call, forceRefresh and still expect plugin call
+        let products; // Define products here to ensure it's in scope for the then block
+        await doApiPost({
+          url: `/products/${testAppName}/${testUserId}/${doNotCallHint}/search`,
+          token: userToken,
+          payload: { forceRefresh: true },
+        }).then(({ products: p }) => {
+          products = p; // Assign to the outer scoped 'products'
+        });
+        expect(travelgatePlugin.searchProducts).toHaveBeenCalled();
+        expect(Array.isArray(products)).toBeTruthy();
+        expect(products.length).toBe(2);
+
+      });
+    });
+    describe('cache TTR and lock mechanism', () => {
+      const ttrTestHint = 'ttr-test'; // This hint is for UserAppKey within 'travelgate' appName
+      const shortTTRToken = {
+        endpoint: 'https://api.travelgatex.com', // Can be any valid URL
+        apiKey: chance.guid(),
+        client: 'tourconnect', // Or any other fields
+        ttlForProducts: 2, // 2 seconds TTR
+      };
+      beforeAll(async () => {
+        // Create/Update UserAppKey for testAppName ('travelgate') with this specific hint and TTR token
+        await globalUtils.appSetup({
+            appName: testAppName, // 'travelgate'
+            userId: testUserId,
+            tokenHint: ttrTestHint,
+            token: shortTTRToken,
+        });
+        if (travelgatePlugin.searchProducts.mockClear) travelgatePlugin.searchProducts.mockClear();
+      });
+      describe('inside of the TTR period', () => {
+        it('first call should create the cache', async ()=> {
+          await doApiPost({
+            url: `/products/${testAppName}/${testUserId}/${ttrTestHint}/search`, // appName is 'travelgate'
+            token: userToken,
+            payload: {},
+          });
+          expect(travelgatePlugin.searchProducts).toHaveBeenCalledTimes(1);
+        });
+        it('inmediate call should not call the plugin mehthod', async () => {
+          // Second immediate call should use cache
+          await doApiPost({
+            url: `/products/${testAppName}/${testUserId}/${ttrTestHint}/search`, // appName is 'travelgate'
+            token: userToken,
+            payload: {},
+          });
+          expect(travelgatePlugin.searchProducts).not.toHaveBeenCalled();
+        });
+      });
+      describe('outside of the TTR period', () => {
+        it('wait for the TTR to expire', async () => {
+          // Wait for TTR to expire (2 seconds + buffer)
+          await new Promise(resolve => {
+            setTimeout(resolve, 2100);
+          });
+        });
+        it('call outside of TTR should serve stale data and queue background refresh with correct parameters', async () => {
+          travelgatePlugin.searchProducts.mockClear(); // Clear before action
+          addJob.mockClear(); // Clear addJob mock before this action that should trigger it
+
+          const { products } = await doApiPost({
+            url: `/products/${testAppName}/${testUserId}/${ttrTestHint}/search`, // appName is 'travelgate'
+            token: userToken,
+            payload: {},
+          });
+          // This synchronous call should serve stale data.
+          // The plugin should NOT be called by *this* request directly as a background job is queued.
+          expect(travelgatePlugin.searchProducts).not.toHaveBeenCalled();
+          expect(Array.isArray(products)).toBeTruthy();
+          expect(products.length).toBe(2); // Assuming stale data (from initial cache population) is available and has 2 products
+          
+          // Check that addJob was called and verify its parameters
+          expect(addJob).toHaveBeenCalledTimes(1);
+          
+          const jobCall = addJob.mock.calls[0]; // Get the first (and only expected) call to addJob
+          const jobData = jobCall[0]; // First argument to addJob is the job payload
+          const jobParams = jobCall[1]; // Second argument is params like removeOnComplete
+
+          const expectedPluginMethod = travelgatePlugin.searchProducts ? 'searchProducts' : 'searchProductsForItinerary';
+
+          expect(jobData.type).toBe('plugin');
+          expect(jobData.pluginName).toBe(testAppName); // 'travelgate'
+          expect(jobData.method).toBe(expectedPluginMethod);
+          expect(jobData.token).toBeDefined(); // Token passed to the job
+          expect(jobData.payload.userId).toBe(testUserId);
+          expect(jobData.payload.payload).toEqual({}); // Original payload for the plugin method was empty
+          expect(jobData.postProcess.controller).toBe('bookings');
+          expect(jobData.postProcess.action).toBe('$updateProductSearchCache');
+          expect(jobData.postProcess.args.appKey).toBe(testAppName);
+          expect(jobData.postProcess.args.userId).toBe(testUserId);
+          expect(jobData.postProcess.args.hint).toBe(ttrTestHint);
+          expect(jobParams).toEqual({ removeOnComplete: true });
+        });
+      });
+      // The 'lock mechanism (job queuing on stale cache)' describe block has been moved to the top level.
+    });
+  });
+
+  describe('bookingsProductSearch caching - stale cache on TTR expiry', () => {
+    // This suite tests behavior when TTR expires and a refresh yields empty results,
+    // expecting stale cache to be served. It uses a real cache with a short TTL.
+    // This suite also uses the global utils and travelgatePlugin. testAppName and testUserId are from parent suite.
+
+    const staleCacheTestHint = 'stale-cache-expiry-test-hint'; // For 'travelgate' appName (testAppName)
+    const shortTtlForProducts = 2; // 2 seconds
+    const staleCacheTokenConfig = {
+      endpoint: 'https://api.travelgatex.com/stale-test', // Unique endpoint for clarity
+      apiKey: chance.guid(),
+      client: 'tourconnect-stale-test',
+      ttlForProducts: shortTtlForProducts,
+    };
+
+    beforeEach(async () => {
+      // Clear mocks
+      if (travelgatePlugin && travelgatePlugin.searchProducts && travelgatePlugin.searchProducts.mockClear) {
+        travelgatePlugin.searchProducts.mockClear();
+      }
+      if (addJob && addJob.mockClear) {
+        addJob.mockClear();
+      }
+
+      // Ensure a clean cache state for this specific hint before each test run
+      // testAppName here is 'travelgate' from the outer describe block's setup
+      const cacheKeyForTest = hash({
+        userId: testUserId, // User from the 'user: bookings controller' suite
+        hint: staleCacheTestHint,
+        operationId: 'bookingsProductSearch',
+      });
+      await cache.drop({ pluginName: testAppName, key: cacheKeyForTest });
+      await cache.drop({ pluginName: testAppName, key: `${cacheKeyForTest}:lastUpdated` });
+      await cache.drop({ pluginName: testAppName, key: `${cacheKeyForTest}:lock` });
+      
+      // Setup the UserAppKey (integration) with the short TTL for products
+      // This uses the testUserId and testAppName ('travelgate') from the parent describe
+      await globalUtils.appSetup({
+          appName: testAppName,
+          userId: testUserId,
+          tokenHint: staleCacheTestHint,
+          token: staleCacheTokenConfig,
+      });
+      // Clear mocks that might have been called during appSetup
+      if (travelgatePlugin.searchProducts.mockClear) travelgatePlugin.searchProducts.mockClear();
+      if (addJob.mockClear) addJob.mockClear();
+    });
+
+    it('should return stale (non-empty) products when TTR expires and refresh yields empty products', async () => {
+      const initialProductsInCache = [{ productId: 'staleProd1', name: 'Stale Product One', optionId: 'optStale1' }];
+      const productsFromPluginRefresh = []; // Simulate plugin returning empty on refresh
+
+      // 1. First call: Populate the cache with initialProductsInCache
+      travelgatePlugin.searchProducts.mockResolvedValueOnce({ products: initialProductsInCache });
+      const { products: firstCallResult } = await doApiPost({
+        url: `/products/${testAppName}/${testUserId}/${staleCacheTestHint}/search`,
+        token: userToken,
+        payload: { searchInput: '' }, 
+      });
+
+      expect(firstCallResult).toEqual(initialProductsInCache);
+      expect(travelgatePlugin.searchProducts).toHaveBeenCalledTimes(1);
+      expect(addJob).not.toHaveBeenCalled(); // No job on initial population
+      travelgatePlugin.searchProducts.mockClear();
+
+      // 2. Wait for TTL to expire (shortTtlForProducts is 2s, wait 3s)
+      await new Promise(resolve => setTimeout(resolve, (shortTtlForProducts + 1) * 1000));
+
+      // 3. Second call: TTR has expired. A background job will be queued.
+      //    The plugin method for the *background job* will return empty results.
+      //    This mock is for the plugin call that the *worker* would make.
+      travelgatePlugin.searchProducts.mockResolvedValueOnce({ products: productsFromPluginRefresh });
+      
+      const { products: secondCallResult } = await doApiPost({
+        url: `/products/${testAppName}/${testUserId}/${staleCacheTestHint}/search`,
+        token: userToken,
+        payload: { searchInput: '' },
+      });
+
+      // Assert that the stale data (initialProductsInCache) is returned by this synchronous call.
+      expect(secondCallResult).toEqual(initialProductsInCache);
+      expect(secondCallResult.length).toBeGreaterThan(0);
+      
+      // Verify that addJob was called to queue the background refresh.
+      expect(addJob).toHaveBeenCalledTimes(1);
+      // The travelgatePlugin.searchProducts mock was for the *job*, so it shouldn't be called by the API directly here.
+      // The controller serves stale and queues job; it doesn't call plugin directly in this path.
+      expect(travelgatePlugin.searchProducts).not.toHaveBeenCalled();
+
+      // Note: This test verifies the immediate response serves stale data and a job is queued.
+      // It does not verify the cache state *after* the job (which would use the empty refresh)
+      // because $updateProductSearchCache currently caches empty results.
+    });
+  });
+});
+
+describe('Bookings Product Search Lock Mechanism (Job Queuing on Stale Cache)', () => {
+  let doApiPost;
+  let lockTestPlugin; // Specific plugin instance for this suite
+  let userToken;
+  let testAppName;
+  let testUserId;
+  let ttrTestHint;
+  const shortTTRTokenConfig = {
+    endpoint: 'https://api.travelgatex.com/lock-test',
+    apiKey: chance.guid(),
+    client: 'tourconnect-lock-test',
+    ttlForProducts: 1, // 1 second TTR for faster testing
+  };
+
+  beforeAll(async () => {
+    doApiPost = globalDoApiPost;
+    lockTestPlugin = globalPlugins.find(p => p.name === 'lockTestPlugin');
+    if (!lockTestPlugin) {
+      throw new Error("lockTestPlugin not found. Ensure it's included in global testUtils setup.");
+    }
+
+    ttrTestHint = 'lock-mechanism-hint';
+
+    // Use globalUtils.appSetup to create the app, user, integration, and userAppKey
+    // This ensures the Integration 'lockTestPlugin' is created before UserAppKey.
+    // The appName here must match an actual plugin name.
+    // A unique userId is generated by appSetup if not provided.
+    const setupData = await globalUtils.appSetup({
+      appName: 'lockTestPlugin',    // This will be the integrationId and plugin name
+      token: shortTTRTokenConfig,   // Pass the specific token config for UserAppKey
+      tokenHint: ttrTestHint,       // Pass the specific hint for UserAppKey
+    });
+
+    testAppName = setupData.newApp.name; // Should be 'lockTestPlugin'
+    testUserId = setupData.userId;
+    userToken = globalUtils.createUserToken(testUserId); // JWT for API calls
+
+    // The UserAppKey with shortTTRTokenConfig and ttrTestHint is now created by appSetup.
+    // No need for an additional doApiPost to create/update it here.
+
+    // Initial cache clear for this specific context. testAppName is 'lockTestPlugin'.
+    const cacheKey = hash({
+      userId: testUserId,
+      hint: ttrTestHint,
+      operationId: 'bookingsProductSearch',
+    });
+    await cache.drop({ pluginName: testAppName, key: cacheKey });
+    await cache.drop({ pluginName: testAppName, key: `${cacheKey}:lastUpdated` });
+    await cache.drop({ pluginName: testAppName, key: `${cacheKey}:lock` }); // pluginExecutionLock
+    await cache.drop({ pluginName: testAppName, key: `${cacheKey}:jobLock` }); // jobQueueLock
+  });
+
+  beforeEach(async () => {
+    // Clear mocks before each test in this suite
+    addJob.mockClear(); // Reset addJob mock calls
+    if (lockTestPlugin && lockTestPlugin.searchProducts && lockTestPlugin.searchProducts.mockClear) {
+      lockTestPlugin.searchProducts.mockClear();
+    }
+    // Clear other global mocks if they were used by this suite and need resetting
+    if (mockJobStatus && mockJobStatus.mockClear) mockJobStatus.mockClear();
+  });
+
+  it('multiple concurrent requests to stale cache should serve stale data and queue only one new refresh job', async () => {
+    // 1. First call: Populate the cache using the specific lockTestPlugin
+    lockTestPlugin.searchProducts.mockResolvedValueOnce({ products: [{ id: 'prod1', name: 'Initial Product' }] });
+
+    await doApiPost({
+      url: `/products/${testAppName}/${testUserId}/${ttrTestHint}/search`, // testAppName is 'lockTestApp'
+      token: userToken,
+      payload: {},
+    });
+    expect(lockTestPlugin.searchProducts).toHaveBeenCalledTimes(1);
+    expect(addJob).not.toHaveBeenCalled(); // No job queued on initial population
+    lockTestPlugin.searchProducts.mockClear(); // Clear for next phase
+
+    // 2. Wait for TTR to expire (ttlForProducts is 1s, wait 1.5s)
+    await new Promise(resolve => setTimeout(resolve, 1500));
+
+    // 3. Make multiple concurrent requests to the now stale cache
+    // Mock plugin response for the background refresh. This mock is for the job that addJob is supposed to queue.
+    lockTestPlugin.searchProducts.mockResolvedValueOnce({ products: [{ id: 'prod2', name: 'Refreshed Product' }] });
+
+    const makeRequest = () => doApiPost({
+      url: `/products/${testAppName}/${testUserId}/${ttrTestHint}/search`,
+      token: userToken,
+      payload: {},
+    });
+
+    const requestPromises = [];
+    requestPromises.push(makeRequest()); // First request hits stale cache, should queue job
+    await global.sleep(50); // Small delay to simulate near concurrency
+    requestPromises.push(makeRequest()); // Second request should also hit stale cache, but not queue another job
+    await global.sleep(50);
+    requestPromises.push(makeRequest()); // Third request
+
+    const results = await Promise.all(requestPromises);
+
+    // Assertions:
+    // a. All requests should serve stale data (the "Initial Product")
+    results.forEach(result => {
+      expect(result.products).toEqual([{ id: 'prod1', name: 'Initial Product' }]);
+    });
+
+    // b. The plugin's searchProducts method (on lockTestPlugin) should NOT have been called directly by these API requests
+    expect(lockTestPlugin.searchProducts).not.toHaveBeenCalled();
+
+    // c. addJob should have been called exactly once
+    expect(addJob).toHaveBeenCalledTimes(1);
+
+    // d. Verify arguments of addJob
+    if (addJob.mock.calls.length > 0) {
+      const expectedPluginMethodName = lockTestPlugin.searchProducts ? 'searchProducts' : 'searchProductsForItinerary';
+      expect(addJob).toHaveBeenCalledWith(
+        expect.objectContaining({
+          type: 'plugin',
+          pluginName: testAppName, // This should be 'lockTestPlugin'
+          method: expectedPluginMethodName,
+          token: expect.objectContaining({ ttlForProducts: shortTTRTokenConfig.ttlForProducts }),
+          payload: expect.objectContaining({
+            payload: {}, // originalRequestBody was empty for the product search
+            userId: testUserId,
+          }),
+          postProcess: expect.objectContaining({
+            controller: 'bookings',
+            action: '$updateProductSearchCache',
+            args: expect.objectContaining({
+              appKey: testAppName, // 'lockTestPlugin'
+              userId: testUserId,
+              hint: ttrTestHint,
+            }),
+          }),
+        }),
+        { removeOnComplete: true }
+      );
+    }
+  });
+});
+