qtools-library-dot-d 1.0.9 → 1.0.11

package/README.md

@@ -229,6 +229,8 @@ This pattern is used extensively in the qtools ecosystem for building scalable,
 
 ## HISTORY
 
+v1.11 - clarified error message "Ignoring invalid file".
+v1.10 - there was a reason I did not return the exception list. Undone.
 v1.09 - fixed broken error handling when library file path is bad; added demonstration application
 v1.08 - updated README (several times)
 v1.05 - cleaned up toString() report, returns exception list (or err if callback) if needed, added qtools-functional-library

package/README2.md

@@ -0,0 +1,418 @@
+## NAME
+
+**qtools-library-dot-d** is system for creating dynamic, modular libraries in Node.js. The dotD (dot-D) pattern provides a powerful way to build extensible applications through automatic module discovery and loading.
+
+## What is qtools-library-dot-d?
+
+**qtools-library-dot-d** is a Node.js library that enables dynamic module loading from directories. It automatically discovers JavaScript files in specified directories and makes them available as methods on a library object, creating a plugin-like architecture without the complexity of traditional plugin systems.
+
+## The dotD Pattern
+
+The "dotD" naming convention (short for "dot-D") refers to directories that contain dynamically loadable modules:
+
+- `modules.d/` - Directory containing module files
+- `endpoints.d/` - Directory containing API endpoint modules  
+- `accessPoints.d/` - Directory containing data access modules
+- `mappers.d/` - Directory containing data mapping modules
+- `processors.d/` - Directory containing processing modules
+
+Each module, `.js` file or directory with package.json, in a dotD directory becomes available through the library object, with the filename determining the module identifier.
+
+## Two Primary Usage Patterns
+
+### Pattern 1: Function Registration (Classic dotD)
+
+Modules register functions that become methods on the library object:
+
+```javascript
+// In dotD module: myFunction.js
+const moduleFunction = ({ moduleName } = {}) => ({ dotD, passThroughParameters } = {}) => {
+    const workingFunction = (params) => {
+        // Function logic here
+        return result;
+    };
+
+    // Register the function
+    dotD.library.add(moduleName, workingFunction);
+};
+module.exports = moduleFunction({ moduleName });
+```
+
+```javascript
+// Usage after loading:
+myLibrary.myFunction(params); // Calls the registered function
+```
+
+### Pattern 2: Initialization Scripts (Modern dotD)
+
+Modules are initialization scripts that run once during `loadModules()`, performing setup work and optionally returning results:
+
+```javascript
+// In dotD module: setupWatcher.js
+const moduleFunction = ({ dotD: myLibraryDotD, passThroughParameters }) => {
+    const { xLog, config, userVisibleProcessLogger } = passThroughParameters;
+
+    // Do initialization work
+    const result = initializeWatcher(config.watchPath);
+
+    // Track what was done
+    myLibraryDotD.logList.push(`${moduleName}: ${result.status}`);
+
+    // Return result (optional)
+    return result;
+};
+module.exports = moduleFunction;
+```
+
+This pattern is particularly useful for:
+
+- Setting up file watchers, database connections, or other persistent resources
+- Configuring Express routes or middleware
+- Initializing services that run in the background
+- One-time setup tasks that don't need to be called repeatedly
+
+## Why Use Dynamic dotD Libraries?
+
+### 1. **Automatic Module Discovery**
+
+No need to manually import or register modules. Simply add a `.js` file to the dotD directory and it's automatically available.
+
+```javascript
+// Traditional approach - manual imports
+const module1 = require('./modules/module1');
+const module2 = require('./modules/module2');
+const module3 = require('./modules/module3');
+
+// dotD approach - automatic discovery
+const myLibrary = qtLibDotD({ libraryName: 'myLibrary' }); //libraryName is used as a label in reports
+myLibrary.setLibraryPath('./modules.d');
+myLibrary.loadModules();
+// All modules in modules.d/ are now processed/available
+```
+
+### 2. **Plugin Architecture**
+
+Create extensible applications where new functionality can be added by simply dropping files into directories.
+
+```javascript
+// Add new functionality by creating a new file
+// modules.d/newFeature.js -> automatically processed during loadModules()
+```
+
+### 3. **Shared Resources**
+
+Pass common resources (configuration, database connections, utilities) to all modules through `passThroughParameters`. These are given, by that name, to the module when it is instantiated during the loadModules() process.
+
+```javascript
+const passThroughParameters = {
+    config: getConfig(),
+    database: dbConnection,
+    logger: logger,
+    utilities: commonUtils,
+    queue: jobQueue,
+    eventHandler: eventHandler
+};
+myLibrary.loadModules({ passThroughParameters });
+```
+
+### 4. **Clean Separation of Concerns**
+
+Each module is self-contained and focused on a single responsibility, making code easier to maintain and test.
+
+### 5. **Runtime Flexibility**
+
+Modules can be loaded synchronously or asynchronously, and the library can be sealed to prevent modifications after loading.
+
+## Real-World Use Cases
+
+### API Endpoint Management (Initialization Pattern)
+
+```javascript
+// endpoints.d/userEndpoints.js - Sets up Express routes
+const moduleFunction = ({ dotD: endpointsDotD, passThroughParameters }) => {
+    const { expressApp, routingPrefix } = passThroughParameters;
+
+    // Set up routes
+    expressApp.get(`${routingPrefix}users`, getUserHandler);
+    expressApp.post(`${routingPrefix}users`, createUserHandler);
+
+    endpointsDotD.logList.push('User endpoints configured');
+    return { routesAdded: 2 };
+};
+```
+
+### Service Initialization (Initialization Pattern)
+
+```javascript
+// processors.d/fileWatcher.js - Sets up file system monitoring
+const moduleFunction = ({ dotD: processorsDotD, passThroughParameters }) => {
+    const { config, eventHandler, logger } = passThroughParameters;
+
+    // Initialize file watcher
+    const watcher = setupFileWatcher({
+        path: config.watchedPath,
+        onFileDetected: (file) => eventHandler({ filePath: file })
+    });
+
+    logger(`File watcher initialized: ${config.watchedPath}`);
+    processorsDotD.logList.push(`fileWatcher: active`);
+
+    return { watcher, status: 'active' };
+};
+```
+
+### Data Access Layer (Function Registration Pattern)
+
+```javascript
+// accessPoints.d/getUserData.js
+const moduleFunction = ({ moduleName } = {}) => ({ dotD, passThroughParameters } = {}) => {
+    const { database } = passThroughParameters;
+
+    const getUserData = (userId, callback) => {
+        // Database access logic
+        database.query('SELECT * FROM users WHERE id = ?', [userId], callback);
+    };
+
+    dotD.library.add(moduleName, getUserData);
+    // Later called as dotD.getUserData(userId, callback)
+};
+```
+
+### Business Logic Modules (Function Registration with Init Pattern)
+
+```javascript
+// processors.d/validateOrder.js
+const moduleFunction = ({ dotD, passThroughParameters }) => {
+    const { xLog, getConfig } = process.global;
+    const localConfig = getConfig(moduleName);
+
+    let newInstanceData;
+
+    const init = () => {
+        // Initialization logic called after dotD loading completes
+        xLog.status(`${moduleName} validator initialized`);
+        // This is where you'd set up the module's initial state
+        return {
+            toolName: moduleName,
+            status: 'ready',
+            type: 'validator'
+        };
+    };
+
+    const validateOrder = (orderData) => {
+        // Validation logic using newInstanceData and orderData
+        return { valid: true, errors: [] };
+    };
+
+    // Register both the main function and the init function
+    dotD.library.add(moduleName, { validateOrder, init });
+    // Later called as dotD.validateOrder.validateOrder(orderData)
+    // Init called as dotD.validateOrder.init()
+
+    dotD.logList.push(`${moduleName}: loaded`);
+};
+```
+
+## How It Works
+
+### Basic Setup (Both Patterns)
+
+```javascript
+const qtLibDotD = require('qtools-library-dot-d');
+const myLibrary = qtLibDotD({ libraryName: 'myLibrary' });
+```
+
+### Async Pipeline Integration (Recommended)
+
+```javascript
+const {pipeRunner, taskListPlus} = new require('qtools-asynchronous-pipe-plus')();
+
+const moduleFunction = ({ config, services }, callback) => {
+    const myLibraryDotD = qtLibDotD({ libraryName: 'MyLibrary' });
+    const taskList = new taskListPlus();
+
+    // Set library path
+    taskList.push((args, next) => {
+        const { myLibraryDotD } = args;
+        myLibraryDotD.setLibraryPath(
+            path.join(__dirname, 'modules.d'),
+            (err, libraryPath) => next(err, { ...args, libraryPath })
+        );
+    });
+
+    // Load modules
+    taskList.push((args, next) => {
+        const { myLibraryDotD } = args;
+        const passThroughParameters = { config, services };
+
+        myLibraryDotD.loadModules({ passThroughParameters }, (err, result) => {
+            next(err, { ...args, result });
+        });
+    });
+
+    // Execute pipeline
+    pipeRunner(taskList.getList(), { myLibraryDotD }, (err, args) => {
+        if (err) return callback(err);
+
+        myLibraryDotD.seal();
+        callback(null, myLibraryDotD);
+    });
+};
+
+// Usage in parent application with init activation:
+const parentApp = ({ dependencies }, callback) => {
+    moduleFunction({ config, services }, (err, { myLibraryDotD }) => {
+        if (err) return callback(err);
+
+        // Activate all modules that have init functions
+        // or, activate specific ones with specific arguments
+        Object.keys(myLibraryDotD).forEach(name => {
+            if (myLibraryDotD[name].init) {
+                myLibraryDotD[name].init(); //could pass in args
+            }
+        });
+
+        callback(null, { myLibraryDotD });
+    });
+};
+```
+
+### Simple Setup (Legacy)
+
+```javascript
+// Synchronous
+myLibrary.setLibraryPath('./modules.d');
+myLibrary.loadModules({ passThroughParameters });
+
+// Asynchronous
+myLibrary.setLibraryPath('./modules.d', (err, path) => {
+    if (err) return callback(err);
+    myLibrary.loadModules({ passThroughParameters }, callback);
+});
+```
+
+## Module Structure Patterns
+
+### Function Registration Module
+
+```javascript
+const moduleName = __filename.replace(__dirname + '/', '').replace(/.js$/, '');
+
+const moduleFunction = ({ moduleName } = {}) => ({ dotD, passThroughParameters } = {}) => {
+    const { config, logger } = passThroughParameters;
+
+    const workingFunction = (/* your parameters */) => {
+        // Your module logic here
+        return result;
+    };
+
+    // Register the function
+    dotD.library.add(moduleName, workingFunction);
+};
+
+module.exports = moduleFunction({ moduleName });
+```
+
+### Initialization Script Module
+
+```javascript
+const moduleName = __filename.replace(__dirname + '/', '').replace(/.js$/, '');
+
+const moduleFunction = ({ dotD: libraryDotD, passThroughParameters }) => {
+    const { xLog, getConfig } = process.global;
+    const localConfig = getConfig(moduleName);
+
+    const { config, services } = passThroughParameters;
+
+    // Do initialization work
+    const result = initializeService(config);
+
+    // Track what was done
+    libraryDotD.logList.push(`${moduleName}: ${result.status}`);
+
+    // Return result (optional)
+    return result;
+};
+
+module.exports = moduleFunction;
+```
+
+## Pattern Selection Guidelines
+
+**Use Function Registration Pattern when:**
+
+- You need callable functions/methods on the library object
+- Building a traditional API or service layer
+- Modules provide stateless utility functions
+- You want `myLibrary.functionName()` style access
+
+**Use Initialization Script Pattern when:**
+
+- Setting up persistent services (file watchers, database connections)
+- Configuring application routes or middleware  
+- One-time setup tasks during application startup
+- Building plugin architectures where modules do setup work
+- Working with qtools-asynchronous-pipe-plus pipelines
+
+## Benefits Over Traditional Approaches
+
+| Traditional           | dotD Libraries              |
+| --------------------- | --------------------------- |
+| Manual imports        | Automatic discovery         |
+| Tight coupling        | Loose coupling              |
+| Hard to extend        | Easy to extend              |
+| Configuration in code | Configuration through files |
+| Monolithic            | Modular                     |
+| Static structure      | Dynamic structure           |
+
+## Getting Started
+
+1. **Install the dependency:**
+   
+   ```bash
+   npm install qtools-library-dot-d
+   ```
+
+2. **Create your dotD directory:**
+   
+   ```bash
+   mkdir myModules.d
+   ```
+
+3. **Choose your pattern and create modules:**
+   
+   - For function registration: Follow the classic pattern
+   - For initialization scripts: Follow the modern pattern
+   - Access shared resources via `passThroughParameters`
+
+## Advanced Features
+
+- **Async module loading** for modules that need initialization
+- **Error handling** during module loading with detailed reporting
+- **Library introspection** with `toString()` and `qtdProcessLog()`
+- **Multiple libraries** in the same application
+- **Nested module structures** for complex applications
+- **Pipeline integration** with qtools-asynchronous-pipe-plus
+- **Logging and tracking** via `dotD.logList` for debugging
+
+## Integration with Larger Systems
+
+The dotD pattern works particularly well in:
+
+- **Web frameworks** (Express, Koa) for route organization
+- **API systems** for endpoint management
+- **CLI tools** for command organization  
+- **Microservices** for feature modularity
+- **Plugin systems** for extensible applications
+- **Queue systems** for dynamic job processor loading
+- **File processing pipelines** for modular processing sources
+
+This pattern is used extensively in the qtools ecosystem for building scalable, maintainable Node.js applications.
+
+## HISTORY
+
+v1.10 - there was a reason I did not return the exception list. Undone.
+v1.09 - fixed broken error handling when library file path is bad; added demonstration application
+v1.08 - updated README (several times)
+v1.05 - cleaned up toString() report, returns exception list (or err if callback) if needed, added qtools-functional-library
+v1.04 - Now tolerates (and reports) directories and files that are not modules.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "qtools-library-dot-d",
-  "version": "1.0.9",
+  "version": "1.0.11",
   "description": "Scan a directory for JS modules that can execute and addMe() that attaches it to some form of name/value list, eg, a Map() with functions as values or an xPressJS route/endpoint list.",
   "main": "qtools-library-dot-d.js",
   "scripts": {

package/qtools-library-dot-d.js

@@ -14,7 +14,7 @@ const qt = require('qtools-functional-library');
 //START OF moduleFunction() ============================================================
 const moduleFunction =
 	({ moduleName } = {}) =>
-	({ libraryName='libraryDotD' }) => {
+	({ libraryName = 'libraryDotD' }) => {
 		// ======================================================================================
 		// SETTERS
 
@@ -47,7 +47,7 @@ const moduleFunction =
 				require(item.endpointPath)(item.moduleArgs); //calls module with dotD object, including library.add()
 			} catch (err) {
 				item.exceptionList.push(
-					`${path.basename(item.endpointPath)}: ${err.toString().split('\n').qtFirst('n/a')}`,
+					`ignoring invlaid file: ${path.basename(item.endpointPath)}: ${err.toString().split('\n').qtFirst('n/a')}`,
 				);
 			}
 
@@ -207,17 +207,10 @@ const moduleFunction =
 			// INIT AND EXECUTE THE PIPELINE
 
 			if (typeof callback == 'function') {
-				if (exceptionList.length) {
-					callback(exceptionList);
-				} else {
-					callback();
-				}
+				callback();
 			} else {
-				if (exceptionList.length) {
-					return exceptionList;
-				} else {
-					return;
-				}}
+				return;
+			}
 		};
 
 		const externalElements = {