ti2 1.0.99 → 1.0.102

package/.windsurfrules

@@ -0,0 +1,9 @@
+This is a Node.js project using Sequelize for database operations and Express.js for the web server.
+The API is defined in api.yml using OpenAPI 3.0.2
+The database is defined in models/index.js
+The models are defined in models/
+to run tests you have to run them on the ti2 named docker container, and consider this codebase is part of node_modules/ti2 path for example : 
+`` $ docker exec ti2 bash -c "cd /ti2 && npx jest lib/__tests__/callback.js --forceExit"``
+When writing / editing tests, make use of test/utils.js and review their implementation to understand how to mock the environment.
+Do not make any changes until you 95% confident of the changes you want to make, ask follow up questions until you have that confidence.
+when sending jobs to the background worker using the queue, asssume a worker is always runing to execute it but is on a different instance thread, so to review / wait for jobs we have to use the wokrer/queue.js listJobs fn to check for it

package/CLAUDE.md

@@ -0,0 +1,159 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+Ti2 (Tourism Information Interchange) is an open-source integration framework for the tourism industry. It provides standardized functions for bookings, content, and rates management through a plugin-based architecture.
+
+## Critical Development Rules
+
+1. **Docker Container Execution**: ALL commands must be run inside the `ti2` Docker container
+   - The codebase is located at `/ti2` path inside the container
+   - This project is part of `node_modules/ti2` in the container environment
+   - **NEVER run npm, node, or npx commands on the host machine**
+   - Always use: `docker exec ti2 bash -c "cd /ti2 && <command>"`
+   - Example for npm install: `docker exec ti2 bash -c "cd /ti2 && npm install <package>"`
+
+2. **Testing Rules**:
+   - Tests MUST be run inside the Docker container
+   - Example: `docker exec ti2 bash -c "cd /ti2 && npx jest lib/__tests__/callback.js --forceExit"`
+   - Use `test/utils.js` for mocking the environment
+   - Review test utility implementations before writing/editing tests
+
+3. **Background Jobs**:
+   - Jobs sent to the queue run on a separate worker thread/instance
+   - Use `worker/queue.js` `listJobs` function to check job status
+   - Workers are always assumed to be running
+
+4. **Code Confidence**: Do not make changes until 95% confident. Ask follow-up questions if uncertain.
+
+## Common Development Commands
+
+```bash
+# Run tests (inside Docker container)
+docker exec ti2 bash -c "cd /ti2 && npm test"
+docker exec ti2 bash -c "cd /ti2 && npx jest [test-file] --forceExit"
+
+# Run specific test file
+docker exec ti2 bash -c "cd /ti2 && npx jest controllers/__tests__/bookings.js --forceExit"
+
+# Database migrations
+docker exec ti2 bash -c "cd /ti2 && npm run sequelize db:migrate"
+
+# Generate documentation
+docker exec ti2 bash -c "cd /ti2 && npm run doc"
+```
+
+## Architecture
+
+### Core Components
+
+1. **Main Entry Point** (`index.js`):
+   - Initializes Express server on port 10010 (default)
+   - Sets up Swagger documentation at `/api-docs`
+   - Handles plugin instantiation and middleware
+   - Implements caching mechanism
+   - Manages background job processing via `worker/queue.js`
+
+2. **API Definition** (`api.yml`):
+   - OpenAPI 3.0.0 specification
+   - Defines all endpoints and schemas
+   - JWT Bearer authentication
+
+3. **Plugin System**:
+   - Two types: Integration Plugins (connect systems) and App Plugins (value-added tools)
+   - Plugins are instantiated with cache, axios, events, and configuration
+   - Plugin schemas are merged with main API schema
+   - Each plugin gets its own routes under `/apps/{pluginName}`
+
+4. **Controllers** (`controllers/`):
+   - `app.js`: Core application controller
+   - `bookings.js`: Booking operations (searchProducts, createBooking, etc.)
+   - `admin.js`: Administrative functions
+   - `user.js`: User management
+   - `ping.js`: Health check endpoints
+   - `allotment.js`: Allotment management
+
+5. **Database Models** (`models/`):
+   - Sequelize ORM with MySQL
+   - Key models: Integration, User, UserAppKey, CronJobs, ApiCronJobs
+   - Migrations in `migrations/` directory
+
+6. **Worker System** (`worker/`):
+   - Bull queue for background job processing
+   - Redis-backed job queue
+   - Supports plugin jobs, API jobs, and callback jobs
+   - Multi-worker support with throng
+
+7. **Authentication** (`auth/authHandler.js`):
+   - JWT-based authentication
+   - Bearer token scheme
+
+### Key Features
+
+- **Caching**: Redis-based caching with configurable TTL
+- **Event System**: EventEmitter2 for plugin communication
+- **Background Jobs**: Async job processing with Bull/Redis
+- **Cron Jobs**: Scheduled task execution
+- **Multi-tenancy**: Support for multiple integrations/plugins
+
+### Plugin Development
+
+Plugins must implement standard methods:
+- `validateToken`: Token validation
+- `tokenTemplate`: Token configuration template
+- Booking methods: `searchProducts`, `searchAvailability`, `createBooking`, `cancelBooking`
+- Content methods: `getProducts`, `getProduct`, `createProduct`, `updateProduct`
+
+### Testing Strategy
+
+- Jest testing framework
+- Test fixtures in `__fixtures__/`
+- Mock plugins in `__mocks__/`
+- Use `test/utils.js` for creating test environments
+- Tests organized by controller in `controllers/__tests__/`
+
+## Database
+
+- Sequelize ORM with MySQL/MariaDB
+- Connection managed through `models/db.js`
+- Migrations handled via sequelize-cli
+- Models auto-loaded from `models/` directory
+
+## Environment Variables
+
+Key variables:
+- `PORT`: Server port (default: 10010)
+- `REDIS_URL`: Redis connection (default: redis://redis:6379)
+- `jwtSecret`: JWT signing secret
+- `adminKey`: Admin authentication key
+- `WEB_CONCURRENCY`: Number of workers
+- `MAX_JOBS_PER_WORKER`: Job concurrency per worker
+- `SSL_INSECURE_ALLOWED_DOMAINS`: Pipe-separated list of domains where SSL certificate verification is skipped (e.g., `localhost|staging.example.com|192.168.1.100`)
+
+Plugin-specific variables follow pattern: `ti2_{pluginName}_{setting}`
+
+### SSL Configuration
+
+The `SSL_INSECURE_ALLOWED_DOMAINS` environment variable allows you to specify domains where SSL certificate verification should be skipped. This is useful for:
+- Development environments with self-signed certificates
+- Testing against staging servers with invalid certificates
+- Local development with HTTPS
+
+Example usage:
+```bash
+# Skip SSL verification for specific domains
+export SSL_INSECURE_ALLOWED_DOMAINS="localhost|staging.api.com|192.168.1.100"
+
+# Multiple domains separated by pipe
+export SSL_INSECURE_ALLOWED_DOMAINS="dev.example.com|test.example.com"
+```
+
+**Important Implementation Notes**:
+- SSL configuration is ONLY applied to plugin axios instances (for external API calls) and worker axios instances
+- Internal communication (callbacks, per-request axios) do NOT use SSL configuration  
+- SSL configuration uses a per-request interceptor approach for better isolation
+- The configuration checks domains dynamically on each request
+
+**Warning**: Only use this in development/testing environments. Never skip SSL verification in production for security reasons.

package/__tests__/cache.js

@@ -0,0 +1,76 @@
+/* globals describe it expect beforeAll afterAll afterEach */
+
+const cache = require('../cache');
+
+describe('cache', () => {
+  const testPluginName = 'testPlugin';
+  const testKeys = [];
+
+  afterEach(async () => {
+    // Clean up test keys
+    for (const key of testKeys) {
+      await cache.drop({ pluginName: testPluginName, key });
+    }
+    testKeys.length = 0;
+  });
+
+
+  describe('scan', () => {
+    it('should return keys matching the pattern without plugin prefix', async () => {
+      // Save some test keys
+      const keys = ['product:123', 'product:456', 'booking:789'];
+      for (const key of keys) {
+        testKeys.push(key);
+        await cache.save({ pluginName: testPluginName, key, value: { test: true } });
+      }
+
+      // Scan for product keys
+      const result = await cache.scan({ pluginName: testPluginName, pattern: 'product:*' });
+
+      expect(result).toHaveLength(2);
+      expect(result).toContain('product:123');
+      expect(result).toContain('product:456');
+      expect(result).not.toContain('booking:789');
+    });
+
+    it('should return empty array when no keys match', async () => {
+      const result = await cache.scan({ pluginName: testPluginName, pattern: 'nonexistent:*' });
+
+      expect(result).toEqual([]);
+    });
+
+    it('should not include keys from other plugins', async () => {
+      const otherPluginName = 'otherPlugin';
+
+      // Save keys for both plugins
+      await cache.save({ pluginName: testPluginName, key: 'shared:key1', value: { test: true } });
+      await cache.save({ pluginName: otherPluginName, key: 'shared:key2', value: { test: true } });
+      testKeys.push('shared:key1');
+
+      // Scan should only return keys for the specified plugin
+      const result = await cache.scan({ pluginName: testPluginName, pattern: 'shared:*' });
+
+      expect(result).toHaveLength(1);
+      expect(result).toContain('shared:key1');
+      expect(result).not.toContain('shared:key2');
+
+      // Cleanup other plugin key
+      await cache.drop({ pluginName: otherPluginName, key: 'shared:key2' });
+    });
+
+    it('should handle wildcard patterns correctly', async () => {
+      const keys = ['cache:v1:product:1', 'cache:v1:product:2', 'cache:v2:product:1'];
+      for (const key of keys) {
+        testKeys.push(key);
+        await cache.save({ pluginName: testPluginName, key, value: { test: true } });
+      }
+
+      // Scan with nested wildcard
+      const result = await cache.scan({ pluginName: testPluginName, pattern: 'cache:v1:*' });
+
+      expect(result).toHaveLength(2);
+      expect(result).toContain('cache:v1:product:1');
+      expect(result).toContain('cache:v1:product:2');
+    });
+  });
+});

package/cache.js

@@ -93,6 +93,26 @@ const drop = async ({
 
 const keys = async () => cache.keys('*');
 
+const scan = ({
+  pluginName,
+  pattern,
+  count = 1000,
+}) => new Promise((resolve, reject) => {
+  const fullPattern = `${pluginName}:${pattern}`;
+  const foundKeys = [];
+  const prefixLength = pluginName.length + 1; // +1 for the colon
+
+  const stream = cache.scanStream({ match: fullPattern, count });
+  stream.on('data', keys => {
+    keys.forEach(key => {
+      // Return keys without the plugin prefix
+      foundKeys.push(key.slice(prefixLength));
+    });
+  });
+  stream.on('end', () => resolve(foundKeys));
+  stream.on('error', reject);
+});
+
 module.exports = {
   cache,
   save,
@@ -100,4 +120,5 @@ module.exports = {
   drop,
   get,
   keys,
+  scan,
 };

package/controllers/__tests__/bookings-searchProducts.js

@@ -285,7 +285,7 @@ describe('user: bookings controller - searchProducts', () => {
           expect(jobData.method).toBe(expectedPluginMethod);
           expect(jobData.token).toBeDefined(); // Token passed to the job
           expect(jobData.payload.userId).toBe(testUserId);
-          expect(jobData.payload.payload).toEqual({}); // Original payload for the plugin method was empty
+          expect(jobData.payload.payload).toEqual({ forceRefresh: false }); // Original payload for the plugin method
           expect(jobData.postProcess.controller).toBe('bookings');
           expect(jobData.postProcess.action).toBe('$updateProductSearchCache');
           expect(jobData.postProcess.args.appKey).toBe(testAppName);
@@ -512,7 +512,7 @@ describe('Bookings Product Search Lock Mechanism (Job Queuing on Stale Cache)',
           method: expectedPluginMethodName,
           token: expect.objectContaining({ ttlForProducts: shortTTRTokenConfig.ttlForProducts }),
           payload: expect.objectContaining({
-            payload: {}, // originalRequestBody was empty for the product search
+            payload: { forceRefresh: false }, // originalRequestBody was empty for the product search
             userId: testUserId,
           }),
           postProcess: expect.objectContaining({

package/controllers/__tests__/bookings.js

@@ -168,4 +168,86 @@ describe('user: bookings controller', () => {
     expect(plugins[0].createBooking.mock.calls[0][0].payload).toEqual(payload);
     expect(plugins[0].createBooking.mock.calls[0][0].token).toEqual(token);
   });
+
+  describe('product search with empty cache', () => {
+    const productSearchCacheKey = hash({
+      userId,
+      hint: 'testingToken',
+      operationId: 'bookingsProductSearch',
+    });
+
+    beforeEach(async () => {
+      // Clear cache before each test
+      await cache.drop({ pluginName: appKey, key: productSearchCacheKey });
+      await cache.drop({ pluginName: appKey, key: `${productSearchCacheKey}:lock` });
+      await cache.drop({ pluginName: appKey, key: `${productSearchCacheKey}:lastUpdated` });
+      await cache.drop({ pluginName: appKey, key: `${productSearchCacheKey}:jobLock` });
+      jest.clearAllMocks();
+    });
+
+    it('should fetch from plugin when cache is empty and no search filter', async () => {
+      // First call: set up empty cache
+      plugins[0].searchProducts.mockImplementationOnce(() => ({ products: [] }));
+      await doApiPost({
+        url: `/products/${appKey}/${userId}/testingToken/search`,
+        token: userToken,
+        payload: {},
+      });
+
+      // Second call: cache is empty, no filter - should call plugin again
+      plugins[0].searchProducts.mockImplementationOnce(() => ({
+        products: [{ productId: '999', productName: 'Fresh Product' }],
+      }));
+      const { products } = await doApiPost({
+        url: `/products/${appKey}/${userId}/testingToken/search`,
+        token: userToken,
+        payload: {},
+      });
+
+      expect(plugins[0].searchProducts).toHaveBeenCalledTimes(2);
+      expect(products).toHaveLength(1);
+      expect(products[0].productName).toBe('Fresh Product');
+    });
+
+    it('should return empty when searching with filter on empty cache', async () => {
+      // Set up empty cache first
+      plugins[0].searchProducts.mockImplementationOnce(() => ({ products: [] }));
+      await doApiPost({
+        url: `/products/${appKey}/${userId}/testingToken/search`,
+        token: userToken,
+        payload: {},
+      });
+
+      // Search with filter on empty cache - should return empty (filter finds nothing)
+      const { products } = await doApiPost({
+        url: `/products/${appKey}/${userId}/testingToken/search`,
+        token: userToken,
+        payload: { searchInput: 'nonexistent' },
+      });
+
+      expect(products).toHaveLength(0);
+    });
+
+    it('should return cached products when cache has data', async () => {
+      // Set up cache with products
+      plugins[0].searchProducts.mockImplementationOnce(() => ({
+        products: [{ productId: '123', productName: 'Cached Product' }],
+      }));
+      await doApiPost({
+        url: `/products/${appKey}/${userId}/testingToken/search`,
+        token: userToken,
+        payload: {},
+      });
+
+      // Second call - should return cached data
+      const { products } = await doApiPost({
+        url: `/products/${appKey}/${userId}/testingToken/search`,
+        token: userToken,
+        payload: {},
+      });
+
+      expect(products).toHaveLength(1);
+      expect(products[0].productName).toBe('Cached Product');
+    });
+  });
 });

package/controllers/bookings.js

@@ -138,8 +138,8 @@ const $bookingsProductSearch = plugins => async ({
     forceRefresh = false,
   } = originalRequestBody;
 
-  // Payload for the plugin function (func) - omit controller-specific flags
-  const payloadForPlugin = R.omit(['forceRefresh'], originalRequestBody);
+  // Payload for the plugin function (func) - pass forceRefresh so plugins can trigger background cache rebuilds
+  const payloadForPlugin = { ...R.omit(['forceRefresh'], originalRequestBody), forceRefresh };
 
   const app = plugins.find(({ name }) => name === appKey);
   assert(userId, 'userId is required');
@@ -212,48 +212,47 @@ const $bookingsProductSearch = plugins => async ({
 
   // 3. Cache exists (initialActualCacheContent) and not forceRefresh:
   if (initialActualCacheContent && initialActualCacheContent.products) {
-    // Effective staleness check for deciding if a background refresh is needed.
-    const isEffectivelyStale = isStaleByTTR && !doNotCallPluginForProducts;
-
-    if (!isEffectivelyStale || hasPluginExecutionLock) {
-      // Cache is fresh OR (is stale BUT a plugin execution is already in progress for refresh): Serve from cache.
-      const searchResults = $searchProductList(initialActualCacheContent.products, searchInput, optionId);
-      return { ...initialActualCacheContent, products: searchResults, ...(token.configuration || {}) };
-    } else {
-      // Cache is effectively stale AND no plugin execution is currently in progress.
-      // Try to queue a background job.
-      const hasJobQueueLock = await app.cache.get({ key: jobQueueLockKey });
-
-      if (hasJobQueueLock) {
-        // A background job has recently been queued by another request. Serve stale data.
+    const cacheIsEmpty = !initialActualCacheContent.products.length;
+    const trimmedSearch = (searchInput || '').trim();
+    const searchFilterIsEmpty = (!trimmedSearch || trimmedSearch === '*') && !(optionId && optionId.length);
+
+    // If cache is empty and no search filter, skip to case 4 to fetch fresh data
+    const shouldSkipEmptyCache = cacheIsEmpty && searchFilterIsEmpty;
+    if (!shouldSkipEmptyCache) {
+      const returnCachedResults = () => {
         const searchResults = $searchProductList(initialActualCacheContent.products, searchInput, optionId);
         return { ...initialActualCacheContent, products: searchResults, ...(token.configuration || {}) };
-      } else {
-        // No job queue lock. Set one, then queue the job, then serve stale data.
-        await app.cache.save({ key: jobQueueLockKey, value: true, ttl: 60 }); // Lock for 60 seconds
+      };
 
-        const searchResults = $searchProductList(initialActualCacheContent.products, searchInput, optionId);
-        
-        const pluginMethodPayload = {
-          payload: payloadForPlugin,
-          userId,
-        };
-        const jobData = {
-          type: 'plugin',
-          pluginName: appKey,
-          method: app.searchProducts ? 'searchProducts' : 'searchProductsForItinerary',
-          token,
-          payload: pluginMethodPayload,
-          postProcess: {
-            controller: 'bookings',
-            action: '$updateProductSearchCache',
-            args: { appKey, userId, hint },
-          },
-        };
-        await addJob(jobData, { removeOnComplete: true });
-        
-        return { ...initialActualCacheContent, products: searchResults, ...(token.configuration || {}) };
+      const isEffectivelyStale = isStaleByTTR && !doNotCallPluginForProducts;
+
+      // Cache is fresh or plugin execution in progress: serve from cache
+      if (!isEffectivelyStale || hasPluginExecutionLock) {
+        return returnCachedResults();
+      }
+
+      // Cache is stale - check if background job already queued
+      const hasJobQueueLock = await app.cache.get({ key: jobQueueLockKey });
+      if (hasJobQueueLock) {
+        return returnCachedResults();
       }
+
+      // Queue background refresh job and serve stale data
+      await app.cache.save({ key: jobQueueLockKey, value: true, ttl: 60 });
+      await addJob({
+        type: 'plugin',
+        pluginName: appKey,
+        method: app.searchProducts ? 'searchProducts' : 'searchProductsForItinerary',
+        token,
+        payload: { payload: payloadForPlugin, userId },
+        postProcess: {
+          controller: 'bookings',
+          action: '$updateProductSearchCache',
+          args: { appKey, userId, hint },
+        },
+      }, { removeOnComplete: true });
+
+      return returnCachedResults();
     }
   }
 

package/index.js

@@ -89,6 +89,7 @@ module.exports = async ({
         get: args => cache.get({ ...args, pluginName }),
         getOrExec: args => cache.getOrExec({ ...args, pluginName }),
         save: args => cache.save({ ...args, pluginName }),
+        scan: args => cache.scan({ ...args, pluginName }),
       },
       axios: pluginAxios,
       events: ti2Events,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ti2",
-  "version": "1.0.99",
+  "version": "1.0.102",
   "description": "Tourist Industry Exchange (TI2)",
   "main": "index.js",
   "scripts": {