data-primals-engine 1.6.5 → 1.7.1

package/doc/AI-assistance.md

@@ -0,0 +1,93 @@
+# AI Assistance: Leverage Artificial Intelligence
+
+`data-primals-engine` includes a powerful, built-in AI assistant named **Prior**. This assistant is designed to understand natural language queries, allowing you to interact with your data, generate insights, and perform actions without writing complex code or API requests.
+
+## Enabling the AI Assistant
+
+To use the AI assistant, you must first provide API keys for one or more supported AI providers. The engine natively supports providers like **OpenAI, Google (Gemini), DeepSeek, and Anthropic**.
+
+You can configure your keys in two ways:
+
+1.  **Environment Variables**: Set the appropriate variable in your `.env` file. This is the recommended approach for production.
+    ```env
+    # Choose one or more providers
+    OPENAI_API_KEY=sk-xxxxxxxxxxxxxxxxxxxx
+    GOOGLE_API_KEY=AIzaSyxxxxxxxxxxxxxxxxxxxx
+    DEEPSEEK_API_KEY=sk-xxxxxxxxxxxxxxxxxxxx
+    ```
+
+2.  **`env` Model**: For user-specific keys, you can store them in the `env` model within the application. The assistant will automatically look for a key belonging to the current user.
+
+Once a key is configured, the AI assistant chat interface becomes available in the UI.
+
+## Core Capabilities
+
+The assistant, "Prior," can perform a wide range of actions based on your natural language commands.
+
+### 1. Data Querying and Visualization
+
+You can ask the assistant to find, filter, and display your data. It can present the results in various formats.
+
+-   **Simple Search**:
+    > "Show me the latest 5 orders from French customers."
+
+-   **Chart Generation**: Create charts on the fly.
+    > "Create a bar chart of user sign-ups per month for the last year."
+
+-   **Custom HTML Views**: Generate sophisticated, styled views of your data using Handlebars templates.
+    > "Build a dashboard of my active projects with their status and due dates, using a modern dark theme."
+
+### 2. Data Modification (with Confirmation)
+
+The assistant can create, update, or delete data, but it will **always ask for your confirmation** before executing a modifying action. This provides a critical safety layer.
+
+-   **Create Data**:
+    > "Create a new task to 'Follow up with Client X' due tomorrow."
+
+-   **Update Data**:
+    > "Update the status of all tickets in the 'Support' category to 'resolved'."
+
+-   **Delete Data**:
+    > "Delete all draft products created before last month."
+
+When you issue such a command, the assistant will respond with a summary of the action it intends to perform and a "Confirm" button. The action is only executed after you click it.
+
+### 3. Answering Questions
+
+The assistant can answer questions about the data it has access to.
+
+> "What was our total revenue in the last quarter?"
+
+## How It Works: The Reasoning Loop
+
+When you send a message, the assistant follows a strict reasoning process:
+
+1.  **Analyze Intent**: It first analyzes your request to understand what you want to do.
+2.  **Search Models (Internal Step)**: Its first action is always to call the `search_models` tool internally. This gives it the exact structure, field names, and types for the relevant data models. This step is crucial for preventing errors and "hallucinations."
+3.  **Formulate Action**: Based on your intent and the model structures it found, it formulates a final action, such as `search`, `generateChart`, or a `post` request.
+4.  **Execute or Confirm**:
+    -   If it's a read-only action (like `search` or `generateChart`), it executes it and displays the result.
+    -   If it's a write action (like `post` or `delete`), it presents the action to you for confirmation.
+
+## AI in Workflows: The `GenerateAIContent` Action
+
+Beyond the chat interface, you can leverage AI directly within your automation **Workflows**. The `GenerateAIContent` action allows you to call an AI model as a step in a workflow.
+
+This is perfect for tasks like:
+-   Summarizing a new support ticket.
+-   Generating a product description when a new product is added.
+-   Classifying incoming data based on its content.
+-   Translating text.
+
+**Example `workflowAction`:**
+```json
+{
+    "name": "Summarize Ticket",
+    "type": "GenerateAIContent",
+    "aiProvider": "OpenAI",
+    "aiModel": "gpt-4-turbo",
+    "prompt": "Summarize the following support ticket in one sentence: {triggerData.description}"
+}
+```
+
+This integration of AI both as an interactive assistant and as an automation component makes `data-primals-engine` a powerful platform for building intelligent applications.

package/doc/Advanced-workflows.md

@@ -0,0 +1,90 @@
+# Advanced Workflows: Custom Logic with Scripts
+
+While the standard workflow actions in `data-primals-engine` cover many common use cases, the `ExecuteScript` action provides ultimate flexibility by allowing you to run custom server-side JavaScript code as part of any workflow. This enables complex data transformations, conditional logic, and dynamic interactions that go beyond pre-defined actions.
+
+## The `ExecuteScript` Action
+
+The `ExecuteScript` action type within a `workflowAction` document lets you write and execute a JavaScript snippet in a secure, sandboxed environment.
+
+### Key Features:
+
+-   **Full JavaScript Support**: Write modern JavaScript (async/await) to implement your logic.
+-   **Access to Workflow Context**: Your script can read from and write to the `contextData` object, allowing you to pass data between workflow steps.
+-   **Database Interaction**: Use a sandboxed `db` object to perform CRUD operations.
+-   **Error Handling**: Throwing an error in your script will fail the current workflow step and can trigger the `onFailureStep`.
+
+### Action Configuration
+
+When creating a `workflowAction` of type `ExecuteScript`, the most important field is `script`:
+
+-   **`script`** (code - JavaScript): The JavaScript code to be executed.
+
+**Example `workflowAction` document:**
+```json
+{
+    "name": "Calculate Order Discount",
+    "type": "ExecuteScript",
+    "script": "const orderTotal = contextData.triggerData.totalAmount;\nif (orderTotal > 100) {\n  contextData.discountAmount = orderTotal * 0.1;\n  contextData.needsManagerApproval = false;\n} else if (orderTotal > 500) {\n  contextData.discountAmount = orderTotal * 0.2;\n  contextData.needsManagerApproval = true;\n}\nreturn contextData;"
+}
+```
+
+## The Execution Environment
+
+Your script runs in an `async` function and has access to several globally-injected objects:
+
+-   **`contextData`** (object): This is the heart of your workflow's state. It contains:
+    -   `triggerData`: The data from the event that initiated the workflow (e.g., the newly created document in a `DataAdded` trigger).
+    -   Any data added by previous steps.
+    -   Your script can read from and write to `contextData`. The returned object from your script will become the new `contextData` for subsequent steps.
+
+-   **`db`** (object): A secure API to interact with the database. All methods are `async` and must be `await`-ed. They automatically respect the permissions of the user who triggered the workflow.
+    -   `await db.create(modelName, dataObject)`
+    -   `await db.find(modelName, filter)`
+    -   `await db.findOne(modelName, filter)`
+    -   `await db.update(modelName, filter, updateObject)`
+    -   `await db.delete(modelName, filter)`
+
+-   **`logger`** (object): A safe logging utility to help with debugging.
+    -   `logger.info(...)`
+    -   `logger.warn(...)`
+    -   `logger.error(...)`
+
+-   **`env`** (object): Provides access to user-defined variables stored in the `env` model.
+    -   `await env.get(variableName)`
+    -   `await env.getAll()`
+
+## Example: Conditional Logic and Data Creation
+
+Imagine a workflow triggered when a `ticket` is created. If the ticket priority is "high", we want to create a new `task` for a manager.
+
+1.  **Trigger**: `onEvent: DataAdded`, `targetModel: ticket`.
+2.  **Step 1**: "Check Priority and Create Task"
+    -   **Action**: `ExecuteScript`
+    -   **Script**:
+        ```javascript
+        // Check if the priority is high
+        if (contextData.triggerData.priority !== 'high') {
+            logger.info('Ticket priority is not high, skipping task creation.');
+            return contextData; // Exit script
+        }
+
+        // Find the manager's user ID (assuming a 'manager' role exists)
+        const manager = await db.findOne('user', { role: { $find: { name: 'manager' } } });
+
+        if (manager) {
+            // Create a new task and assign it to the manager
+            await db.create('task', {
+                title: `High-priority ticket: ${contextData.triggerData.subject}`,
+                assignedTo: manager._id,
+                relatedTicket: contextData.triggerData._id,
+                status: 'todo'
+            });
+            logger.info(`Task created for manager ${manager.username}.`);
+        } else {
+            logger.warn('No manager found to assign the task to.');
+        }
+
+        return contextData; // Always return the context
+        ```
+
+By leveraging the `ExecuteScript` action, you can build sophisticated, stateful, and dynamic workflows that precisely match your application's business rules.

package/doc/Event-system.md

@@ -0,0 +1,79 @@
+# Event System: Extending Core Functionality
+
+The `data-primals-engine` is built on a powerful event-driven architecture. The `Event` system allows you to hook into core operations, enabling you to extend and customize the engine's behavior without modifying its source code. This is the primary mechanism for creating custom modules and advanced middleware.
+
+## Listening to Events
+
+You can register a callback function that will be executed whenever a specific event is triggered. This is done using the `Event.Listen()` method.
+
+### Syntax
+
+```javascript
+Event.Listen(eventName, callback, scope, type);
+```
+
+-   **`eventName`** (string): The name of the event to listen to (e.g., `"OnDataAdded"`).
+-   **`callback`** (function): The function to execute when the event fires. The arguments it receives depend on the event.
+-   **`scope`** (string): Must be `"event"`.
+-   **`type`** (string): Determines the callback signature.
+    -   `"user"`: The callback receives arguments relevant to a user action (e.g., `(data)`).
+    -   `"system"`: The callback receives the full `engine` instance as its first argument, followed by other arguments (e.g., `(engine, data)`).
+
+### Example: Logging New Data
+
+```javascript
+import { Event } from 'data-primals-engine';
+
+Event.Listen(
+    "OnDataAdded",
+    (engine, insertedDocs) => {
+        const logger = engine.getComponent('Logger');
+        logger.info(`New documents were added:`, insertedDocs);
+    },
+    "event",
+    "system"
+);
+```
+
+## Triggering Custom Events
+
+You can also define and trigger your own custom events. This is useful for making your own modules extensible.
+
+```javascript
+import { Event } from 'data-primals-engine';
+
+async function myCustomFunction(data) {
+    // ... some logic ...
+
+    // Trigger a custom event and pass data to listeners
+    const results = await Event.Trigger("OnMyCustomEvent", "event", "user", data);
+
+    // ... do something with results from listeners ...
+}
+```
+
+When multiple listeners are registered for an event, their return values are merged:
+-   **strings** are concatenated.
+-   **numbers** are added.
+-   **booleans** are logical AND-ed.
+-   **arrays** are concatenated.
+-   **objects** are merged (spread operator).
+
+## Core Events Table
+
+Here is a list of the most important events you can listen to:
+
+| Event              | Description                                                      | Scope  | Triggered by                 | Arguments (Payload)                                                                |
+|:-------------------|:-----------------------------------------------------------------|:-------|:-----------------------------|:-----------------------------------------------------------------------------------|
+| `OnServerStart`    | Triggered once the HTTP server is started and listening.         | System | `engine.start()`             | `engine`                                                                           |
+| `OnDatabaseLoaded` | Triggered once the lient is ready                                | System |                              | `engine`                                                                           |
+| `OnModelsLoaded`   | Triggered after the initial models are loaded at startup.        | System | `setupInitialModels()`       | `engine`, `dbModels`                                                               |
+| `OnModelEdited`    | Triggered after a model definition has been modified.            | System | `editModel()`                | `newModel`                                                                         |
+| `OnDataAdded`      | Triggered after new data has been inserted.                      | System | `insertData()`               | `engine`, `insertedDocs`                                                           |
+| `OnDataEdited`     | Triggered after data has been edited.                            | System | `editData()` / `patchData()` | `engine`, `{modelName, before, after}`                                             |
+| `OnDataDeleted`    | Triggered just after data is actually deleted.                   | System | `deleteData()`               | `engine`, `{model, filter}`                                                        |
+| `OnDataSearched`   | Triggered after a data search.                                   | System | `searchData()`               | `engine`, `{data, count}`                                                          |
+| `OnDataInsert`     | Triggered just before data insertion. Allows modifying the data. | System | internal                     | `data`                                                                             |
+| `OnDataValidate`   | Triggered to override validation checks.                         | System | internal                     | `value`, `field`, `data`                                                           |
+| `OnChatAction`     | Triggered when an AI assistant decides on an action.             | User   | `handleChatRequest`          | `action`, `params`, `parsedResponse`, `command`, `llmOptions`, `user`, `reqParams` |
+| `OnSystemPrompt`   | Triggered to override the AI assistant's system prompt.          | User   | `handleChatRequest`          | `user`                                                                             |

package/doc/Packs-gallery.md

@@ -0,0 +1,73 @@
+# Packs Gallery: Discover and Install Ready-to-Use Configurations
+
+**Packs** are pre-built configurations that allow you to quickly set up your `data-primals-engine` application with ready-to-use models, data, workflows, and more. They are designed to bootstrap common use cases like a CRM, an e-commerce backoffice, or a project management tool, saving you significant development time.
+
+The **Pack Gallery** is the centralized place within the UI where you can discover, preview, and install these packs.
+
+## What is a Pack?
+
+A pack is a JSON object that can contain:
+
+-   **`models`**: A list of model definitions (schemas) to be created.
+-   **`data`**: Sample or initial data for the models in the pack. Data can be generic or language-specific.
+-   **`workflows`**: Pre-configured automation workflows, including triggers, steps, and actions.
+-   **`dashboards`** and **`kpis`**: Ready-to-use dashboards and KPIs for monitoring.
+
+## Installing a Pack
+
+You can install a pack in two ways: by its ID from the gallery or by providing a custom JSON structure directly.
+
+### 1. Installing from the Gallery
+
+Each pack in the gallery has a unique ID. You can use the `installPack` function with this ID to install it.
+
+```javascript
+import { installPack } from 'data-primals-engine';
+
+// Assuming 'currentUser' is your authenticated user object
+const packId = "61d1f1a9e3f1a9e3f1a9e3f1"; // Example ID for an e-commerce pack
+const installationSummary = await installPack(packId, currentUser, "en");
+
+console.log(installationSummary);
+// Returns a summary of created models, inserted data, etc.
+```
+
+### 2. Installing a Custom Pack
+
+You can also define a pack on-the-fly and install it. This is useful for migrating configurations between environments or for programmatic setup.
+
+The `installPack` function accepts an array containing the pack's JSON definition.
+
+```javascript
+import { installPack } from 'data-primals-engine';
+
+const myCustomPack = [
+    {
+        "name": "Simple Blog Pack",
+        "description": "A basic setup for a blog with posts and categories.",
+        "tags": ["blog", "cms"],
+        "models": [
+            {
+                "name": "post",
+                "fields": [
+                    { "name": "title", "type": "string_t", "required": true },
+                    { "name": "content", "type": "richtext_t" },
+                    { "name": "status", "type": "enum", "items": ["draft", "published"] }
+                ]
+            }
+        ],
+        "data": {
+            "all": { // Data for all languages
+                "post": [
+                    { "title": { "key": "first_post_title", "value": "My First Post" }, "status": "draft" }
+                ]
+            }
+        }
+    }
+];
+
+const summary = await installPack(myCustomPack, currentUser, "en");
+console.log(summary);
+```
+
+By using packs, you can dramatically accelerate your project setup and ensure consistency across different instances of your application.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "data-primals-engine",
-  "version": "1.6.5",
+  "version": "1.7.1",
   "description": "data-primals-engine is a package responsible from handling large amount of data using MongoDB in a practical and performant way. It can also get workflow models working (for automation), and fully supports internationalisation. It also has integrated AI assistant.",
   "main": "src/engine.js",
   "type": "module",
@@ -28,16 +28,24 @@
     "@rollup/rollup-linux-x64-gnu": "4.6.1"
   },
   "resolutions": {
-    "on-headers": "1.1.0",
-    "brace-expansion": "2.0.2",
-    "prismjs": "1.30.0",
+    "flatted": ">=3.4.2",
+    "rollup": ">=4.59.0",
+    "on-headers": ">=1.1.0",
+    "brace-expansion": ">=2.0.2",
+    "prismjs": ">=1.30.0",
     "xml2js": ">=0.5.0",
     "tar-fs": ">=3.1.1",
-    "vite": ">=7.0.8",
-    "node-forge": ">=1.3.2",
+    "vite": ">=7.3.2",
+    "node-forge": ">=1.4.0",
     "js-yaml": ">=4.1.1",
     "mdast-util-to-hast": ">=13.2.1",
-    "jws": ">=3.2.3"
+    "jws": ">=3.2.3",
+    "undici": ">=6.23.0",
+    "@xmldom/xmldom": ">=0.8.12",
+    "picomatch": ">=4.0.4",
+    "langsmith": ">=0.5.19",
+    "qs": ">=6.14.2",
+    "fast-xml-parser": ">=4.5.5"
   },
   "overrides": {
     "react-syntax-highlighter": {
@@ -48,11 +56,17 @@
     },
     "refractor": {
       "prismjs": "1.30.0"
+    },
+    "body-parser": {
+      "qs": ">=6.14.1"
+    },
+    "stripe": {
+      "qs": ">=6.14.1"
     }
   },
   "repository": {
     "type": "git",
-    "url": "https://github.com/anonympins/data-primals-engine.git"
+    "url": "git+https://github.com/anonympins/data-primals-engine.git"
   },
   "exports": {
     ".": "./src/index.js",
@@ -63,7 +77,7 @@
     "./*": "./src/*.js"
   },
   "dependencies": {
-    "@langchain/core": "^0.3.79",
+    "@langchain/core": "^0.3.80",
     "archiver": "^7.0.1",
     "aws-sdk": "^2.1692.0",
     "bcrypt": "^6.0.0",
@@ -78,17 +92,17 @@
     "express-csrf-double-submit-cookie": "^2.0.0",
     "express-formidable": "^1.2.0",
     "express-mongo-sanitize": "^2.2.0",
-    "express-rate-limit": "^8.0.1",
+    "express-rate-limit": "8.2.2",
     "express-session": "^1.18.2",
-    "handlebars": "^4.7.8",
+    "handlebars": "4.7.9",
     "i18next-browser-languagedetector": "^8.2.0",
     "isolated-vm": "^5.0.4",
     "juice": "^11.0.3",
-    "mathjs": "^15.1.0",
+    "mathjs": "15.2.0",
     "mongodb": "^6.18.0",
     "node-cache": "^5.1.2",
     "node-schedule": "^2.1.1",
-    "nodemailer": "^7.0.11",
+    "nodemailer": "8.0.5",
     "openai": "^6.9.1",
     "passport": "^0.7.0",
     "passport-saml-encrypted": "^0.1.13",
@@ -103,11 +117,11 @@
     "sirv": "^3.0.2",
     "stripe": "^20.0.0",
     "swagger-ui-express": "^5.0.1",
-    "tar": "^7.5.2",
+    "tar": "7.5.11",
     "tinycolor2": "^1.6.0",
     "uniqid": "^5.4.0",
-    "vitest": "^3.2.4",
-    "yaml": "^2.8.1"
+    "vitest": "3.2.4",
+    "yaml": "2.8.3"
   },
   "peerDependencies": {
     "express": "^5.2.1",

package/src/constants.js

@@ -277,7 +277,7 @@ metaModels['erp'] = { load: [ 'accountingExercise', 'accountingLineItem', 'accou
  * Available model field attributes
  * @type {string[]}
  */
-export const allowedFields = ['locked', 'hiddenable', 'anonymized', 'condition', 'color', 'index', 'indexType', 'type', 'required', 'hint', 'default', 'validate', 'unique', 'name', 'placeholder', 'asMain'];
+export const allowedFields = ['locked', 'hiddenable', 'encrypted', 'anonymized', 'condition', 'color', 'index', 'indexType', 'type', 'required', 'hint', 'default', 'validate', 'unique', 'name', 'placeholder', 'asMain'];