codehooks-js 1.3.10 → 1.3.12

package/README.md

@@ -1,19 +1,28 @@
 # codehooks-js 
 
-Codehooks.io official development library - provides Express.JS-like syntax for using codehooks.io.
+The official JavaScript/TypeScript library for [Codehooks.io](https://codehooks.io) - a serverless backend platform that provides everything you need to build and deploy backend applications.
 
-Read the [Quickstart for developers](https://codehooks.io/docs/quickstart-cli).
+## What is Codehooks.io?
 
-For more details navigate to official docs: [https://codehooks.io/docs](https://codehooks.io/docs).
+Codehooks.io is a complete serverless backend platform that combines:
+- **Database** - NoSQL document storage with MongoDB-like queries
+- **API Server** - Express.js-style routing and middleware
+- **Background Jobs** - Queues, workers, and scheduled tasks
+- **Real-time** - Server-sent events for live updates
+- **File Storage** - Static file serving and blob storage
+- **Workflows** - Step-based application logic
 
-## Install
+All in one platform with zero infrastructure management.
+
+## Quick Start
 
 Install the [Codehooks CLI](https://codehooks.io/docs/cli): 
 
 ```shell
 $ npm install -g codehooks
 ```
-Next install the codehooks-js NPM package in your project.
+
+Install the codehooks-js library in your project:
 
 ```shell
 npm install codehooks-js
@@ -114,12 +123,200 @@ Correcting the errors should ultimately show a successfull compile output, ready
 $ coho compile
 
 OK 🙌
-
 ```
 
+Your backend application is now available at your project's endpoint URL. For example:
+
+`https://myproject-ff00.codehooks.io/dev/*`
+
 ## Deploy
 
 From the project directory run: 
 ```shell
 $ codehooks deploy
-```
+```
+
+## Documentation
+
+For complete documentation, visit [https://codehooks.io/docs](https://codehooks.io/docs).
+
+## API
+
+## Codehooks Class API Reference
+
+The Codehooks class provides a comprehensive backend application framework with the following APIs:
+
+### **HTTP Routing APIs**
+- **`post(path, ...hook)`** - Register POST route handlers
+- **`get(path, ...hook)`** - Register GET route handlers  
+- **`put(path, ...hook)`** - Register PUT route handlers
+- **`patch(path, ...hook)`** - Register PATCH route handlers
+- **`delete(path, ...hook)`** - Register DELETE route handlers
+- **`all(path, ...hook)`** - Register handlers for all HTTP methods
+
+### **Middleware & Authentication APIs**
+- **`use(...hook)`** - Register global middleware (supports string paths, RegExp, or function)
+- **`useRoute(route, ...hook)`** - Register route-specific middleware
+- **`auth(path, ...hook)`** - Register authentication middleware for specific paths
+
+### **Background Processing APIs**
+- **`queue(topic, ...hook)`** - Register queue handlers for background processing
+- **`worker(name, ...hook)`** - Register worker functions (also adds to queues for legacy support)
+- **`job(cronExpression, ...hook)`** - Register scheduled cron jobs
+
+### **Static File Serving APIs**
+- **`static(options, hook)`** - Serve static files from source code directory
+- **`storage(options, hook)`** - Serve files from blob storage directory
+
+### **Template & Configuration APIs**
+- **`set(key, val)`** - Set application configuration settings
+- **`render(view, data, cb)`** - Render templates with data
+- **`crudlify(schema, options)`** - Auto-generate CRUD REST API endpoints
+
+### **Real-time Communication APIs**
+- **`realtime(path, ...hook)`** - Set up server-sent events (SSE) channels for real-time communication
+
+### **Workflow Management APIs**
+- **`createWorkflow(name, description, steps, options)`** - Create and register a new workflow instance
+
+### **Application Lifecycle APIs**
+- **`init(hook)`** - Initialize the application and return manifest
+- **`start(hook)`** - Alias for `init()` method
+
+
+
+The Codehooks class serves as a comprehensive backend application framework that combines HTTP routing, background processing, real-time communication, and workflow management capabilities in a single, cohesive API.
+
+## Datastore API Reference
+
+The Datastore provides a unified interface for both NoSQL document storage and Key-Value operations. It supports MongoDB-like query syntax and provides streaming capabilities for large datasets.
+
+### **Connection & Collection Management**
+- **`open()`** - Connect to the Datastore and return the API interface
+- **`collection(name)`** - Get a NoSQL collection by name for document operations
+
+### **NoSQL Document Operations**
+
+#### **Read Operations**
+- **`getOne(collection, query)`** - Get a single document by ID or query
+- **`findOne(collection, query)`** - Alias for getOne
+- **`getMany(collection, query?, options?)`** - Get a stream of documents matching query
+- **`find(collection, query?, options?)`** - Alias for getMany
+
+#### **Write Operations**
+- **`insertOne(collection, document)`** - Insert a new document into a collection
+- **`updateOne(collection, query, document, updateOperators?, options?)`** - Update one document (patches existing data)
+- **`updateMany(collection, query, document, updateOperators?)`** - Update multiple documents
+- **`replaceOne(collection, query, document, options?)`** - Replace one document completely
+- **`replaceMany(collection, query, document, options?)`** - Replace multiple documents
+
+#### **Delete Operations**
+- **`removeOne(collection, query)`** - Remove one document by ID or query
+- **`removeMany(collection, query)`** - Remove multiple documents matching query
+
+#### **Schema Management**
+- **`createSchema(collection, schema)`** - Validate collection data against JSON-Schema
+- **`setSchema(collection, schema)`** - Alias for createSchema
+- **`removeSchema(collection)`** - Remove JSON-schema validation for collection
+- **`getSchema(collection)`** - Get JSON-schema for collection
+
+#### **Utility Operations**
+- **`count(collection)`** - Count documents in a collection
+
+### **Key-Value Operations**
+
+#### **Basic Key-Value Operations**
+- **`set(key, value, options?)`** - Set a key-value pair
+- **`get(key, options?)`** - Get a value by key
+- **`getAll(keyPattern, options?)`** - Get all key-value pairs matching pattern
+- **`del(key, value)`** - Delete a key-value pair
+- **`delAll(keyPattern, options?)`** - Delete all key-value pairs matching pattern
+
+#### **Numeric Operations**
+- **`incr(key, value, options?)`** - Increment a numeric value
+- **`decr(key, value, options?)`** - Decrement a numeric value
+
+### **Queue Operations**
+- **`enqueue(topic, document, options?)`** - Add a job to a queue for background processing
+- **`enqueueFromQuery(collection, query, topic, options?)`** - Queue each item from a database query
+
+### **DataStream Interface**
+
+When using `getMany()` or `find()`, you get a DataStream object with these methods:
+
+- **`on(event, callback)`** - Listen for data events (e.g., `'data'`, `'end'`)
+- **`json(response)`** - Pipe data directly to HTTP response as JSON
+- **`toArray()`** - Convert stream to array of objects
+- **`forEach(callback)`** - Iterate over each object in the stream
+
+### **Usage Examples**
+
+```javascript
+import { datastore } from 'codehooks-js';
+
+// Connect to datastore
+const conn = await datastore.open();
+
+// NoSQL operations
+const doc = await conn.insertOne('users', { name: 'John', email: 'john@example.com' });
+const user = await conn.getOne('users', { _id: doc._id });
+const users = await conn.getMany('users', { active: true }).toArray();
+await conn.updateOne('users', { _id: doc._id }, { lastLogin: new Date() });
+
+// Key-Value operations
+await conn.set('user:123:session', { token: 'abc123', expires: new Date() });
+const session = await conn.get('user:123:session');
+await conn.incr('visits', 1);
+
+// Queue operations
+await conn.enqueue('emailWorker', { to: 'user@example.com', template: 'welcome' });
+```
+
+### **Query Syntax**
+
+The Datastore supports MongoDB-like query syntax:
+
+```javascript
+// Simple equality
+{ status: 'active' }
+
+// Comparison operators
+{ age: { $gt: 18 } }
+{ price: { $lte: 100 } }
+
+// Logical operators
+{ $and: [{ status: 'active' }, { age: { $gte: 18 } }] }
+{ $or: [{ category: 'A' }, { category: 'B' }] }
+
+// Array operations
+{ tags: { $in: ['javascript', 'nodejs'] } }
+{ tags: { $all: ['javascript', 'nodejs'] } }
+
+// Regular expressions
+{ name: { $regex: /john/i } }
+```
+
+### **Options**
+
+Many operations accept an options object for additional configuration:
+
+```javascript
+// Upsert option for updates
+await conn.updateOne('users', { email: 'john@example.com' }, 
+  { lastLogin: new Date() }, {}, { upsert: true });
+
+// Key-Value options
+await conn.set('key', 'value', { ttl: 3600000 }); // 1 hour TTL
+```
+
+### **Key Features:**
+1. **Express-style routing** with support for all HTTP methods
+2. **Middleware system** with global and route-specific middleware
+3. **Background processing** with queues, workers, and scheduled jobs
+4. **Static file serving** from both source and blob storage
+5. **Real-time communication** via Server-Sent Events
+6. **Workflow engine** for complex step-based applications
+7. **Auto-generated CRUD APIs** with schema validation
+8. **Template rendering** system
+9. **Singleton pattern** for global application state
+10. **Datastore** with MongoDB-like query syntax, key-value operations, and queue management

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "codehooks-js",
-  "version": "1.3.10",
+  "version": "1.3.12",
   "type": "module",
   "description": "Codehooks.io official library - provides express.JS like syntax",
   "main": "index.js",

package/types/index.d.ts

@@ -1080,10 +1080,10 @@ export type WorkflowEvents = {
   'stateUpdated': { workflowName: string; state: any; instanceId: string };
   
   /**
-   * Emitted when a step is waiting for input
+   * Emitted when a step is enqueued for execution
    * @event
    */
-  'stepWaiting': { workflowName: string; step: string; instanceId: string };
+  'stepEnqueued': { workflowName: string; step: string; state: any; instanceId: string };
   
   /**
    * Emitted when a workflow instance is continued after waiting
@@ -1113,7 +1113,11 @@ export type WorkflowEvents = {
 /**
  * Definition of a workflow step function
  */
-export type WorkflowDefinition = Record<string, (state: any, callback: (nextStep: string | string[] | null, newState: any, options?: any) => void) => Promise<void>>;
+export type WorkflowDefinition = Record<string, (
+    state: any, 
+    callback: (nextStep: string | string[] | null, newState: any, options?: any) => void,
+    waiterFunction?: (waiter?: any) => void
+) => Promise<void>>;
 
 /**
  * Configuration options for a workflow step
@@ -1141,20 +1145,6 @@ export type WorkflowConfig = {
   steps?: Record<string, StepOptions>;
 };
 
-/**
- * Engine for managing workflow-based applications
- * @extends Workflow
- */
-export type WorkflowEngine = Workflow & {
-  /**
-   * Register a new workflow
-   * @param name - Unique identifier for the workflow
-   * @param description - Human-readable description
-   * @param steps - Step definitions
-   */
-  register: (name: string, description: string, steps: WorkflowDefinition) => Promise<string>;
-};
-
 /**
  * Workflow engine for managing step-based applications
  * @extends EventEmitter
@@ -1185,48 +1175,118 @@ export type Workflow = {
   configure: (options: WorkflowConfig) => void;
 
   /**
-   * Register a new workflow
+   * Get the collection name for storing workflow data
+   * @returns The collection name
+   */
+  getCollectionName: () => string;
+
+  /**
+   * Set the collection name for storing workflow data
+   * @param name - Collection name
+   */
+  setCollectionName: (name: string) => void;
+
+  /**
+   * Get the queue prefix for workflow jobs
+   * @returns The queue prefix
+   */
+  getQueuePrefix: () => string;
+
+  /**
+   * Set the queue prefix for workflow jobs
+   * @param prefix - Queue prefix
+   */
+  setQueuePrefix: (prefix: string) => void;
+
+  /**
+   * Get the timeout for workflow steps
+   * @returns The timeout in milliseconds
+   */
+  getTimeout: () => number;
+
+  /**
+   * Set the timeout for workflow steps
+   * @param timeout - Timeout in milliseconds
+   */
+  setTimeout: (timeout: number) => void;
+
+  /**
+   * Get the maximum step count
+   * @returns The maximum step count
+   */
+  getMaxStepCount: () => number;
+
+  /**
+   * Set the maximum step count
+   * @param maxStepCount - Maximum step count
+   */
+  setMaxStepCount: (maxStepCount: number) => void;
+
+  /**
+   * Get the steps configuration
+   * @returns The steps configuration
+   */
+  getStepsConfig: () => Record<string, StepOptions>;
+
+  /**
+   * Set the steps configuration
+   * @param steps - Steps configuration
+   */
+  setStepsConfig: (steps: Record<string, StepOptions>) => void;
+
+  /**
+   * Get the step definition for a specific step
+   * @param workflowName - Name of the workflow
+   * @param stepName - Name of the step
+   * @returns The step function
+   */
+  getDefinition: (workflowName: string, stepName: string) => Function;
+
+  /**
+   * Register a new workflow with a Codehooks application
    * @param app - Codehooks application instance
-   * @param name - Unique identifier for the workflow
-   * @param description - Human-readable description
-   * @param definition - Object containing step definitions
    * @returns Promise with the registered workflow name
    */
-  register: (app: Codehooks, name: string, description: string, definition: Record<string, Function>) => Promise<string>;
+  register: (app: Codehooks) => Promise<string>;
   
   /**
    * Start a new workflow instance
-   * @param name - Name of the workflow to start
    * @param initialState - Initial state for the workflow instance
-   * @returns Promise with the workflow instance ID
+   * @returns Promise with the workflow instance
    */
-  start: (name: string, initialState: any) => Promise<{ id: string }>;
+  start: (initialState: any) => Promise<any>;
   
   /**
    * Update the state of a workflow instance
-   * @param workflowName - Name of the workflow
    * @param instanceId - ID of the workflow instance
    * @param state - New state to update with
    * @param options - Options for the update, { continue: false } to avoid continuing the step
    * @returns Promise with the updated state
    */
-  updateState: (workflowName: string, instanceId: string, state: any, options?: UpdateOptions) => Promise<any>;
+  updateState: (instanceId: string, state: any, options?: UpdateOptions) => Promise<any>;
+  
+  /**
+   * Set the complete state of a workflow instance
+   * @param instanceId - ID of the workflow instance
+   * @param stateData - Object containing _id and state
+   * @returns Promise<void>
+   */
+  setState: (instanceId: string, stateData: { _id: string; state: any }) => Promise<void>;
   
   /**
    * Continue a paused workflow instance
-   * @param workflowName - Name of the workflow
    * @param instanceId - ID of the workflow instance
    * @param reset - Whether to reset all step counts (true) or just the current step (false)
    * @returns Promise with the queue ID for the continued step
    */
-  continue: (workflowName: string, instanceId: string, reset?: boolean) => Promise<{ qId: string }>;
+  continue: (instanceId: string, reset?: boolean) => Promise<{ instanceId: string }>;
   
   /**
    * Get the status of a workflow instance
    * @param id - ID of the workflow instance
    * @returns Promise with the workflow status
    */
-  getWorkflowStatus: (id: string) => Promise<any>;
+  getStepsStatus: (id: string) => Promise<any>;
   
   /**
    * Get all workflow instances matching a filter
@@ -1240,7 +1300,7 @@ export type Workflow = {
    * @param id - ID of the workflow instance to cancel
    * @returns Promise with the cancellation result
    */
-  cancelWorkflow: (id: string) => Promise<any>;
+  cancelSteps: (id: string) => Promise<any>;
 
   /**
    * Register an event listener
@@ -1278,7 +1338,7 @@ export type Workflow = {
    * Continue all timed out workflow instances
    * @returns Promise with array of results containing queue IDs for continued workflows
    */
-  continueAllTimedOut: () => Promise<Array<{qId: string}>>;
+  continueAllTimedOut: () => Promise<Array<{instanceId: string}>>;
 
   /**
    * Check if a specific step in a workflow instance has timed out
@@ -1297,6 +1357,7 @@ export type Workflow = {
    * //   finishTime?: string,    // if step has finished
    * //   currentTime?: string,   // if step is still running
    * //   timeoutSource: 'stepConfig' | 'defaultOptions' | 'globalTimeout'
+   * //   reason?: string         // if step not found
    * // }
    */
   isStepTimedOut: (workflow: any) => {
@@ -1374,11 +1435,14 @@ export type WorkflowEvent =
  * Data structure for workflow events
  */
 export type WorkflowEventData = {
-  workflowName: string;
+  workflowName?: string;
+  name?: string;
+  description?: string;
   step?: string;
   state?: any;
   instanceId?: string;
   error?: Error;
+  id?: string;
   [key: string]: any;
 };
 

package/workflow/engine.mjs

@@ -261,6 +261,18 @@ class Workflow extends EventEmitter {
                     return;
                 }
                 try {
+                    const waiterFunction = async (state) => {
+                        if (state) {
+                            console.debug('waiter', state);
+                            await connection.updateOne(this.#collectionName,
+                                { _id: instanceId },
+                                { $set: { ...state, updatedAt: new Date().toISOString() } });                            
+                            resolve();
+                            return;
+                        } else {
+                            resolve(); // no state update
+                        }
+                    }
                     await func({...newState, instanceId: newState._id}, async (nextStep, userState, options) => {
                         try {
                             // Protect system-level properties
@@ -370,8 +382,8 @@ class Workflow extends EventEmitter {
                             console.error('error', error.message);
                             reject(error);
                         }
-                    });
-                    resolve();
+                    }, waiterFunction);
+                    //resolve();
                 } catch (error) {
                     console.error('Error executing step function:', error);
                     reject(error);
@@ -411,17 +423,7 @@ class Workflow extends EventEmitter {
      * @private
      */
     async registerWithApp(app, name, description, definition) {
-        this.emit('workflowCreated', { name, description });
-
-        // Log initial state of definitions Map
-        console.debug('Before registration - Current definitions Map:', {
-            size: this.#definitions.size,
-            keys: Array.from(this.#definitions.keys()),
-            entries: Array.from(this.#definitions.entries()).map(([key, value]) => ({
-                key,
-                stepNames: Object.keys(value)
-            }))
-        });
+        this.emit('workflowCreated', { name, description });        
 
         // Validate each step in the definition
         for (const [stepName, step] of Object.entries(definition)) {
@@ -456,63 +458,33 @@ class Workflow extends EventEmitter {
         }
 
         // Store the definition
-        this.#definitions.set(name, definition);
-        
-        // Log state after registration
-        console.debug('After registration - Updated definitions Map:', {
-            size: this.#definitions.size,
-            keys: Array.from(this.#definitions.keys()),
-            entries: Array.from(this.#definitions.entries()).map(([key, value]) => ({
-                key,
-                stepNames: Object.keys(value)
-            }))
-        });
+        this.#definitions.set(name, definition);        
 
         return name;
     }
 
     /**
      * Start a new steps instance
-     * @param {string} name - Name of the steps workflow to start
      * @param {Object} initialState - Initial state for the steps instance
      * @returns {Promise<{id: string}>} The steps instance ID
      * @throws {Error} If starting steps fails
      */
-    async start(name, initialState) {
-        this.emit('workflowStarted', { name, initialState });
-
-        // Log definitions Map state at start
-        console.debug('At workflow start - Current definitions Map:', {
-            size: this.#definitions.size,
-            keys: Array.from(this.#definitions.keys()),
-            entries: Array.from(this.#definitions.entries()).map(([key, value]) => ({
-                key,
-                stepNames: Object.keys(value)
-            }))
-        });
+    async start(initialState) {
+        this.emit('workflowStarted', { name: this.#name, initialState });        
 
         return new Promise(async (resolve, reject) => {
             try {
-                console.debug('Starting workflow', name);
-                const funcs = this.#definitions.get(name);
-                console.debug('Retrieved workflow definition:', {
-                    exists: !!funcs,
-                    stepNames: funcs ? Object.keys(funcs) : []
-                });
-
+                console.debug('Starting workflow', this.#name);
+                const funcs = this.#definitions.get(this.#name);
+                
                 if (!funcs) {
-                    reject(new Error(`No workflow definition found for: ${name}`));
+                    reject(new Error(`No workflow definition found for: ${this.#name}`));
                     return;
                 }
 
                 const firstStepName = Object.keys(funcs)[0];
                 const firstStep = funcs[firstStepName];
-                console.debug('First step details:', {
-                    name: firstStepName,
-                    exists: !!firstStep,
-                    type: firstStep ? typeof firstStep : 'undefined'
-                });
-
+                
                 if (!firstStep) {
                     reject(new Error('No start step defined in workflow'));
                     return;
@@ -521,9 +493,9 @@ class Workflow extends EventEmitter {
                 const connection = await Datastore.open();
                 // Create a new workflow state in the database
                 const newState = await connection.insertOne(this.#collectionName,
-                    { ...initialState, nextStep: firstStepName, createdAt: new Date().toISOString(), workflowName: name, stepCount: {  } });
-                const { _id: ID } = await connection.enqueue(`${this.#queuePrefix}_${name}_${firstStepName}`, {
-                    stepsName: name,
+                    { ...initialState, nextStep: firstStepName, createdAt: new Date().toISOString(), workflowName: this.#name, stepCount: {  } });
+                const { _id: ID } = await connection.enqueue(`${this.#queuePrefix}_${this.#name}_${firstStepName}`, {
+                    stepsName: this.#name,
                     goto: firstStepName,
                     state: newState,
                     instanceId: newState._id,
@@ -539,21 +511,20 @@ class Workflow extends EventEmitter {
 
     /**
      * Update the state of a workflow instance
-     * @param {string} stepsName - Name of the workflow
      * @param {string} instanceId - ID of the workflow instance
      * @param {Object} state - New state to update with
      * @param {Object} options - Options for the update, { continue: false } to avoid continuing the the step
      * @returns {Promise<Object>} The updated state
      */
-    async updateState(stepsName, instanceId, state, options={continue: true}) {
-        this.emit('stepsStateUpdating', { stepsName, instanceId, state });
+    async updateState(instanceId, state, options={continue: true}) {
+        this.emit('stepsStateUpdating', { name: this.#name, instanceId, state });
         const connection = await Datastore.open();
         return new Promise(async (resolve, reject) => {
             const doc = await connection.updateOne(this.#collectionName, 
                 { _id: instanceId }, 
                 { $set: { ...state, updatedAt: new Date().toISOString() } });
             if (options.continue) {
-                await this.continue(stepsName, instanceId, false);
+                await this.continue(instanceId, false);
             } 
             resolve({ ...doc });                       
         });
@@ -572,12 +543,11 @@ class Workflow extends EventEmitter {
 
     /**
      * Continue a paused steps instance
-     * @param {string} stepsName - Name of the steps workflow
      * @param {string} instanceId - ID of the steps instance
      * @returns {Promise<{qId: string}>} Queue ID for the continued step
      * @throws {Error} If steps instance not found
      */
-    async continue(stepsName, instanceId, reset=false) {
+    async continue(instanceId, reset=false) {
         const connection = await Datastore.open();
         const state = await connection.findOne(this.#collectionName, { _id: instanceId });
         if (!state) {
@@ -596,11 +566,11 @@ class Workflow extends EventEmitter {
         
         await connection.updateOne(this.#collectionName, { _id: instanceId }, { $set: { stepCount: state.stepCount } });
         console.debug('continue state', state);
-        this.emit('workflowContinued', { stepsName, step: state.nextStep, instanceId });                    
+        this.emit('workflowContinued', { name: this.#name, step: state.nextStep, instanceId });                    
 
         return new Promise(async (resolve, reject) => {
-            const { _id: ID } = await connection.enqueue(`${this.#queuePrefix}_${stepsName}_${state.nextStep}`, {
-                stepsName,
+            const { _id: ID } = await connection.enqueue(`${this.#queuePrefix}_${this.#name}_${state.nextStep}`, {
+                stepsName: this.#name,
                 goto: state.nextStep,
                 state: state,
                 options: {},
@@ -626,7 +596,7 @@ class Workflow extends EventEmitter {
             if (diffMillis > this.#timeout) {
                 const diffMinutes = diffMillis / (1000 * 60);
                 console.log('Timed out:', workflow._id, workflow.nextStep, `(${diffMinutes.toFixed(1)} minutes old)`);
-                const result = await this.continue(workflow.workflowName, workflow._id, true);    
+                const result = await this.continue(workflow._id, true);    
                 console.log('Continued:', result._id);
                 results.push(result);
             }