data-primals-engine 1.6.3 → 1.7.0

package/doc/AI-assistance.md

@@ -0,0 +1,93 @@
+# AI Assistance: Leverage Artificial Intelligence
+
+`data-primals-engine` includes a powerful, built-in AI assistant named **Prior**. This assistant is designed to understand natural language queries, allowing you to interact with your data, generate insights, and perform actions without writing complex code or API requests.
+
+## Enabling the AI Assistant
+
+To use the AI assistant, you must first provide API keys for one or more supported AI providers. The engine natively supports providers like **OpenAI, Google (Gemini), DeepSeek, and Anthropic**.
+
+You can configure your keys in two ways:
+
+1.  **Environment Variables**: Set the appropriate variable in your `.env` file. This is the recommended approach for production.
+    ```env
+    # Choose one or more providers
+    OPENAI_API_KEY=sk-xxxxxxxxxxxxxxxxxxxx
+    GOOGLE_API_KEY=AIzaSyxxxxxxxxxxxxxxxxxxxx
+    DEEPSEEK_API_KEY=sk-xxxxxxxxxxxxxxxxxxxx
+    ```
+
+2.  **`env` Model**: For user-specific keys, you can store them in the `env` model within the application. The assistant will automatically look for a key belonging to the current user.
+
+Once a key is configured, the AI assistant chat interface becomes available in the UI.
+
+## Core Capabilities
+
+The assistant, "Prior," can perform a wide range of actions based on your natural language commands.
+
+### 1. Data Querying and Visualization
+
+You can ask the assistant to find, filter, and display your data. It can present the results in various formats.
+
+-   **Simple Search**:
+    > "Show me the latest 5 orders from French customers."
+
+-   **Chart Generation**: Create charts on the fly.
+    > "Create a bar chart of user sign-ups per month for the last year."
+
+-   **Custom HTML Views**: Generate sophisticated, styled views of your data using Handlebars templates.
+    > "Build a dashboard of my active projects with their status and due dates, using a modern dark theme."
+
+### 2. Data Modification (with Confirmation)
+
+The assistant can create, update, or delete data, but it will **always ask for your confirmation** before executing a modifying action. This provides a critical safety layer.
+
+-   **Create Data**:
+    > "Create a new task to 'Follow up with Client X' due tomorrow."
+
+-   **Update Data**:
+    > "Update the status of all tickets in the 'Support' category to 'resolved'."
+
+-   **Delete Data**:
+    > "Delete all draft products created before last month."
+
+When you issue such a command, the assistant will respond with a summary of the action it intends to perform and a "Confirm" button. The action is only executed after you click it.
+
+### 3. Answering Questions
+
+The assistant can answer questions about the data it has access to.
+
+> "What was our total revenue in the last quarter?"
+
+## How It Works: The Reasoning Loop
+
+When you send a message, the assistant follows a strict reasoning process:
+
+1.  **Analyze Intent**: It first analyzes your request to understand what you want to do.
+2.  **Search Models (Internal Step)**: Its first action is always to call the `search_models` tool internally. This gives it the exact structure, field names, and types for the relevant data models. This step is crucial for preventing errors and "hallucinations."
+3.  **Formulate Action**: Based on your intent and the model structures it found, it formulates a final action, such as `search`, `generateChart`, or a `post` request.
+4.  **Execute or Confirm**:
+    -   If it's a read-only action (like `search` or `generateChart`), it executes it and displays the result.
+    -   If it's a write action (like `post` or `delete`), it presents the action to you for confirmation.
+
+## AI in Workflows: The `GenerateAIContent` Action
+
+Beyond the chat interface, you can leverage AI directly within your automation **Workflows**. The `GenerateAIContent` action allows you to call an AI model as a step in a workflow.
+
+This is perfect for tasks like:
+-   Summarizing a new support ticket.
+-   Generating a product description when a new product is added.
+-   Classifying incoming data based on its content.
+-   Translating text.
+
+**Example `workflowAction`:**
+```json
+{
+    "name": "Summarize Ticket",
+    "type": "GenerateAIContent",
+    "aiProvider": "OpenAI",
+    "aiModel": "gpt-4-turbo",
+    "prompt": "Summarize the following support ticket in one sentence: {triggerData.description}"
+}
+```
+
+This integration of AI both as an interactive assistant and as an automation component makes `data-primals-engine` a powerful platform for building intelligent applications.

package/doc/Advanced-workflows.md

@@ -0,0 +1,90 @@
+# Advanced Workflows: Custom Logic with Scripts
+
+While the standard workflow actions in `data-primals-engine` cover many common use cases, the `ExecuteScript` action provides ultimate flexibility by allowing you to run custom server-side JavaScript code as part of any workflow. This enables complex data transformations, conditional logic, and dynamic interactions that go beyond pre-defined actions.
+
+## The `ExecuteScript` Action
+
+The `ExecuteScript` action type within a `workflowAction` document lets you write and execute a JavaScript snippet in a secure, sandboxed environment.
+
+### Key Features:
+
+-   **Full JavaScript Support**: Write modern JavaScript (async/await) to implement your logic.
+-   **Access to Workflow Context**: Your script can read from and write to the `contextData` object, allowing you to pass data between workflow steps.
+-   **Database Interaction**: Use a sandboxed `db` object to perform CRUD operations.
+-   **Error Handling**: Throwing an error in your script will fail the current workflow step and can trigger the `onFailureStep`.
+
+### Action Configuration
+
+When creating a `workflowAction` of type `ExecuteScript`, the most important field is `script`:
+
+-   **`script`** (code - JavaScript): The JavaScript code to be executed.
+
+**Example `workflowAction` document:**
+```json
+{
+    "name": "Calculate Order Discount",
+    "type": "ExecuteScript",
+    "script": "const orderTotal = contextData.triggerData.totalAmount;\nif (orderTotal > 100) {\n  contextData.discountAmount = orderTotal * 0.1;\n  contextData.needsManagerApproval = false;\n} else if (orderTotal > 500) {\n  contextData.discountAmount = orderTotal * 0.2;\n  contextData.needsManagerApproval = true;\n}\nreturn contextData;"
+}
+```
+
+## The Execution Environment
+
+Your script runs in an `async` function and has access to several globally-injected objects:
+
+-   **`contextData`** (object): This is the heart of your workflow's state. It contains:
+    -   `triggerData`: The data from the event that initiated the workflow (e.g., the newly created document in a `DataAdded` trigger).
+    -   Any data added by previous steps.
+    -   Your script can read from and write to `contextData`. The returned object from your script will become the new `contextData` for subsequent steps.
+
+-   **`db`** (object): A secure API to interact with the database. All methods are `async` and must be `await`-ed. They automatically respect the permissions of the user who triggered the workflow.
+    -   `await db.create(modelName, dataObject)`
+    -   `await db.find(modelName, filter)`
+    -   `await db.findOne(modelName, filter)`
+    -   `await db.update(modelName, filter, updateObject)`
+    -   `await db.delete(modelName, filter)`
+
+-   **`logger`** (object): A safe logging utility to help with debugging.
+    -   `logger.info(...)`
+    -   `logger.warn(...)`
+    -   `logger.error(...)`
+
+-   **`env`** (object): Provides access to user-defined variables stored in the `env` model.
+    -   `await env.get(variableName)`
+    -   `await env.getAll()`
+
+## Example: Conditional Logic and Data Creation
+
+Imagine a workflow triggered when a `ticket` is created. If the ticket priority is "high", we want to create a new `task` for a manager.
+
+1.  **Trigger**: `onEvent: DataAdded`, `targetModel: ticket`.
+2.  **Step 1**: "Check Priority and Create Task"
+    -   **Action**: `ExecuteScript`
+    -   **Script**:
+        ```javascript
+        // Check if the priority is high
+        if (contextData.triggerData.priority !== 'high') {
+            logger.info('Ticket priority is not high, skipping task creation.');
+            return contextData; // Exit script
+        }
+
+        // Find the manager's user ID (assuming a 'manager' role exists)
+        const manager = await db.findOne('user', { role: { $find: { name: 'manager' } } });
+
+        if (manager) {
+            // Create a new task and assign it to the manager
+            await db.create('task', {
+                title: `High-priority ticket: ${contextData.triggerData.subject}`,
+                assignedTo: manager._id,
+                relatedTicket: contextData.triggerData._id,
+                status: 'todo'
+            });
+            logger.info(`Task created for manager ${manager.username}.`);
+        } else {
+            logger.warn('No manager found to assign the task to.');
+        }
+
+        return contextData; // Always return the context
+        ```
+
+By leveraging the `ExecuteScript` action, you can build sophisticated, stateful, and dynamic workflows that precisely match your application's business rules.

package/doc/Event-system.md

@@ -0,0 +1,79 @@
+# Event System: Extending Core Functionality
+
+The `data-primals-engine` is built on a powerful event-driven architecture. The `Event` system allows you to hook into core operations, enabling you to extend and customize the engine's behavior without modifying its source code. This is the primary mechanism for creating custom modules and advanced middleware.
+
+## Listening to Events
+
+You can register a callback function that will be executed whenever a specific event is triggered. This is done using the `Event.Listen()` method.
+
+### Syntax
+
+```javascript
+Event.Listen(eventName, callback, scope, type);
+```
+
+-   **`eventName`** (string): The name of the event to listen to (e.g., `"OnDataAdded"`).
+-   **`callback`** (function): The function to execute when the event fires. The arguments it receives depend on the event.
+-   **`scope`** (string): Must be `"event"`.
+-   **`type`** (string): Determines the callback signature.
+    -   `"user"`: The callback receives arguments relevant to a user action (e.g., `(data)`).
+    -   `"system"`: The callback receives the full `engine` instance as its first argument, followed by other arguments (e.g., `(engine, data)`).
+
+### Example: Logging New Data
+
+```javascript
+import { Event } from 'data-primals-engine';
+
+Event.Listen(
+    "OnDataAdded",
+    (engine, insertedDocs) => {
+        const logger = engine.getComponent('Logger');
+        logger.info(`New documents were added:`, insertedDocs);
+    },
+    "event",
+    "system"
+);
+```
+
+## Triggering Custom Events
+
+You can also define and trigger your own custom events. This is useful for making your own modules extensible.
+
+```javascript
+import { Event } from 'data-primals-engine';
+
+async function myCustomFunction(data) {
+    // ... some logic ...
+
+    // Trigger a custom event and pass data to listeners
+    const results = await Event.Trigger("OnMyCustomEvent", "event", "user", data);
+
+    // ... do something with results from listeners ...
+}
+```
+
+When multiple listeners are registered for an event, their return values are merged:
+-   **strings** are concatenated.
+-   **numbers** are added.
+-   **booleans** are logical AND-ed.
+-   **arrays** are concatenated.
+-   **objects** are merged (spread operator).
+
+## Core Events Table
+
+Here is a list of the most important events you can listen to:
+
+| Event              | Description                                                      | Scope  | Triggered by                 | Arguments (Payload)                                                                |
+|:-------------------|:-----------------------------------------------------------------|:-------|:-----------------------------|:-----------------------------------------------------------------------------------|
+| `OnServerStart`    | Triggered once the HTTP server is started and listening.         | System | `engine.start()`             | `engine`                                                                           |
+| `OnDatabaseLoaded` | Triggered once the lient is ready                                | System |                              | `engine`                                                                           |
+| `OnModelsLoaded`   | Triggered after the initial models are loaded at startup.        | System | `setupInitialModels()`       | `engine`, `dbModels`                                                               |
+| `OnModelEdited`    | Triggered after a model definition has been modified.            | System | `editModel()`                | `newModel`                                                                         |
+| `OnDataAdded`      | Triggered after new data has been inserted.                      | System | `insertData()`               | `engine`, `insertedDocs`                                                           |
+| `OnDataEdited`     | Triggered after data has been edited.                            | System | `editData()` / `patchData()` | `engine`, `{modelName, before, after}`                                             |
+| `OnDataDeleted`    | Triggered just after data is actually deleted.                   | System | `deleteData()`               | `engine`, `{model, filter}`                                                        |
+| `OnDataSearched`   | Triggered after a data search.                                   | System | `searchData()`               | `engine`, `{data, count}`                                                          |
+| `OnDataInsert`     | Triggered just before data insertion. Allows modifying the data. | System | internal                     | `data`                                                                             |
+| `OnDataValidate`   | Triggered to override validation checks.                         | System | internal                     | `value`, `field`, `data`                                                           |
+| `OnChatAction`     | Triggered when an AI assistant decides on an action.             | User   | `handleChatRequest`          | `action`, `params`, `parsedResponse`, `command`, `llmOptions`, `user`, `reqParams` |
+| `OnSystemPrompt`   | Triggered to override the AI assistant's system prompt.          | User   | `handleChatRequest`          | `user`                                                                             |

package/doc/Packs-gallery.md

@@ -0,0 +1,73 @@
+# Packs Gallery: Discover and Install Ready-to-Use Configurations
+
+**Packs** are pre-built configurations that allow you to quickly set up your `data-primals-engine` application with ready-to-use models, data, workflows, and more. They are designed to bootstrap common use cases like a CRM, an e-commerce backoffice, or a project management tool, saving you significant development time.
+
+The **Pack Gallery** is the centralized place within the UI where you can discover, preview, and install these packs.
+
+## What is a Pack?
+
+A pack is a JSON object that can contain:
+
+-   **`models`**: A list of model definitions (schemas) to be created.
+-   **`data`**: Sample or initial data for the models in the pack. Data can be generic or language-specific.
+-   **`workflows`**: Pre-configured automation workflows, including triggers, steps, and actions.
+-   **`dashboards`** and **`kpis`**: Ready-to-use dashboards and KPIs for monitoring.
+
+## Installing a Pack
+
+You can install a pack in two ways: by its ID from the gallery or by providing a custom JSON structure directly.
+
+### 1. Installing from the Gallery
+
+Each pack in the gallery has a unique ID. You can use the `installPack` function with this ID to install it.
+
+```javascript
+import { installPack } from 'data-primals-engine';
+
+// Assuming 'currentUser' is your authenticated user object
+const packId = "61d1f1a9e3f1a9e3f1a9e3f1"; // Example ID for an e-commerce pack
+const installationSummary = await installPack(packId, currentUser, "en");
+
+console.log(installationSummary);
+// Returns a summary of created models, inserted data, etc.
+```
+
+### 2. Installing a Custom Pack
+
+You can also define a pack on-the-fly and install it. This is useful for migrating configurations between environments or for programmatic setup.
+
+The `installPack` function accepts an array containing the pack's JSON definition.
+
+```javascript
+import { installPack } from 'data-primals-engine';
+
+const myCustomPack = [
+    {
+        "name": "Simple Blog Pack",
+        "description": "A basic setup for a blog with posts and categories.",
+        "tags": ["blog", "cms"],
+        "models": [
+            {
+                "name": "post",
+                "fields": [
+                    { "name": "title", "type": "string_t", "required": true },
+                    { "name": "content", "type": "richtext_t" },
+                    { "name": "status", "type": "enum", "items": ["draft", "published"] }
+                ]
+            }
+        ],
+        "data": {
+            "all": { // Data for all languages
+                "post": [
+                    { "title": { "key": "first_post_title", "value": "My First Post" }, "status": "draft" }
+                ]
+            }
+        }
+    }
+];
+
+const summary = await installPack(myCustomPack, currentUser, "en");
+console.log(summary);
+```
+
+By using packs, you can dramatically accelerate your project setup and ensure consistency across different instances of your application.

package/doc/automation-workflows.md

@@ -0,0 +1,102 @@
+# Automation with Workflows: Create Automated Processes
+
+The `data-primals-engine` features a powerful **Workflow** system that enables you to define and automate complex business processes. Workflows are sequences of steps and actions that can be triggered by various events or on a schedule, allowing for sophisticated automation directly within your backend.
+
+## Core Workflow Models
+
+The workflow system is composed of several interconnected data models:
+
+### 1. `workflow`
+
+The `workflow` model is the top-level definition of an automated process. It outlines the overall flow and its starting point.
+
+-   **`name`** (string, required): A unique, descriptive name for the workflow (e.g., "Order Validation", "Low Stock Notification").
+-   **`description`** (richtext, optional): A detailed explanation of the workflow's purpose.
+-   **`startStep`** (relation to `workflowStep`, optional): The first step to execute when the workflow is initiated.
+
+### 2. `workflowTrigger`
+
+A `workflowTrigger` defines an event or schedule that initiates a workflow.
+
+-   **`workflow`** (relation to `workflow`, required): The workflow that this trigger belongs to.
+-   **`name`** (string, required, unique): A descriptive name for the trigger (e.g., "New Order Created", "Stock < 5", "Monday 9 AM Report").
+-   **`type`** (enum, required): How the workflow is initiated:
+    -   `manual`: Triggered by a data event.
+    -   `scheduled`: Triggered by a cron schedule.
+-   **`onEvent`** (enum, for `manual` type): The data event that triggers the workflow: `DataAdded`, `DataEdited`, `DataDeleted`, `ModelAdded`, `ModelEdited`, `ModelDeleted`.
+-   **`targetModel`** (string, for `manual` type): The name of the model targeted by the `onEvent`.
+-   **`dataFilter`** (code - JSON, optional, for `manual` type): Optional MongoDB filter conditions checked against the `triggerData` before executing the workflow.
+-   **`cronExpression`** (string, for `scheduled` type): A cron expression (e.g., `'0 9 * * 1'` for Monday 9 AM) to schedule the workflow.
+-   **`isActive`** (boolean): Whether the trigger is currently active.
+-   **`env`** (code - JSON, optional): Environment variables (JSON key/value pairs) specific to this trigger.
+
+### 3. `workflowStep`
+
+A `workflowStep` represents a single stage within a workflow process. It can contain conditions, actions, and define the next steps based on success or failure.
+
+-   **`workflow`** (relation to `workflow`, required): The workflow this step belongs to.
+-   **`name`** (string, optional): A descriptive name for the step (e.g., "Check Inventory", "Send Confirmation Email").
+-   **`conditions`** (code - JSON, optional): Optional conditions (MongoDB filter syntax) that must be met before the step's actions are executed. These can reference `contextData`.
+-   **`actions`** (multiple relation to `workflowAction`, required): The main operations performed by this step.
+-   **`onSuccessStep`** (relation to `workflowStep`, optional): The next step to execute if this step's conditions are met and actions succeed.
+-   **`onFailureStep`** (relation to `workflowStep`, optional): The next step if conditions fail or any action within this step fails.
+-   **`isTerminal`** (boolean, default: `false`): Indicates if this step marks the end of a workflow path.
+
+### 4. `workflowAction`
+
+A `workflowAction` defines a specific operation to be performed by a `workflowStep`. This is where the actual work of the workflow happens.
+
+-   **`name`** (string, required): Name of the action (e.g., "Update Order Status", "Send Email", "Call Payment API").
+-   **`type`** (enum, required): The type of operation to perform:
+    -   `UpdateData`: Modify existing data in a `targetModel`.
+    -   `CreateData`: Add new data to a `targetModel`.
+    -   `DeleteData`: Remove data from a `targetModel`.
+    -   `ExecuteScript`: Run custom JavaScript code.
+    -   `HttpRequest`: Make an HTTP request to an external service.
+    -   `SendEmail`: Send an email.
+    -   `Wait`: Pause the workflow for a specified duration.
+    -   `GenerateAIContent`: Generate content using an AI model (e.g., OpenAI, Google Gemini).
+    -   `ExecuteServiceFunction`: Call a function from a registered internal service (e.g., 'stripe').
+-   **`targetModel`** (string, for data operations): The model to target.
+-   **`targetSelector`** (code - JSON, for `UpdateData`/`DeleteData`): Expression to filter the target document(s).
+-   **`fieldsToUpdate`** (code - JSON, for `UpdateData`): Key-value pairs of fields to update.
+-   **`dataToCreate`** (code - JSON, for `CreateData`): Object template for the new document.
+-   **`script`** (code - JavaScript, for `ExecuteScript`): The JavaScript code to execute.
+-   **`url`, `method`, `headers`, `body`** (for `HttpRequest`): Details for the HTTP request.
+-   **`emailRecipients`, `emailSubject`, `emailContent`** (for `SendEmail`): Email details.
+-   **`duration`, `durationUnit`** (for `Wait`): How long to pause.
+-   **`aiProvider`, `aiModel`, `prompt`** (for `GenerateAIContent`): AI model and prompt details.
+-   **`serviceName`, `functionName`, `args`** (for `ExecuteServiceFunction`): Service call details.
+
+## Workflow Execution (`workflowRun`)
+
+Each time a workflow is triggered, a `workflowRun` document is created to track its execution.
+
+-   **`workflow`** (relation to `workflow`): The workflow definition that was executed.
+-   **`contextData`** (code - JSON): A snapshot of the data or event that triggered this run, and any data generated during execution.
+-   **`status`** (enum): The current status (`pending`, `running`, `completed`, `failed`, `waiting`, `cancelled`).
+-   **`history`** (array of objects): Detailed execution history of each step and action.
+-   **`startedAt`, `completedAt`**: Timestamps for the run.
+-   **`error`**: Error message if the workflow run failed.
+
+## Example Workflow Scenario
+
+Consider a workflow to "Send a Welcome Email to New Users":
+
+1.  **`workflow`**: "New User Onboarding"
+2.  **`workflowTrigger`**:
+    -   `name`: "On New User Added"
+    -   `type`: `manual`
+    -   `onEvent`: `DataAdded`
+    -   `targetModel`: `user`
+3.  **`workflowStep`**: "Send Welcome Email"
+    -   `workflow`: "New User Onboarding"
+    -   `actions`: [ID of "Send Welcome Email Action"]
+    -   `onSuccessStep`: (optional) "Create Onboarding Task"
+4.  **`workflowAction`**: "Send Welcome Email Action"
+    -   `type`: `SendEmail`
+    -   `emailRecipients`: `{triggerData.contact.email}`
+    -   `emailSubject`: "Welcome to Our Platform, {triggerData.contact.firstName}!"
+    -   `emailContent`: "Hello {triggerData.contact.firstName}, welcome aboard..."
+
+This powerful system allows you to automate virtually any process, from simple notifications to complex data transformations and integrations with external services.

package/doc/core-concepts.md

@@ -0,0 +1,33 @@
+# Core Concepts: Data Modeling Fundamentals
+
+The `data-primals-engine` is built around a flexible and powerful data modeling system that allows you to define and manage your application's data structures without complex migrations. This section introduces the fundamental concepts of data modeling within the engine.
+
+## Dynamic and Schema-less Nature
+
+At its core, `data-primals-engine` leverages **MongoDB**, a NoSQL database. This provides inherent flexibility, allowing for a schema-less approach. You can define and update your data models either through a user interface or by providing JSON definitions, and the system adapts dynamically without requiring traditional database migrations. This significantly accelerates development and iteration cycles.
+
+## Models
+
+A **Model** is the blueprint for a collection of data. It defines the structure and characteristics of the documents you store. For example, a `User` model would define what properties a user object has (e.g., `username`, `email`, `roles`).
+
+Models are defined by a `name`, an optional `description`, an `icon`, and a list of `fields`. The engine comes with a set of default models for common entities like `user`, `product`, `workflow`, and more.
+
+## Fields
+
+**Fields** are the individual attributes that make up a model. Each field has a `name`, a `type`, and various optional properties that define its behavior and constraints.
+
+Common field types include:
+-   `string`: For text data.
+-   `number`: For numerical data.
+-   `boolean`: For true/false values.
+-   `datetime`: For date and time values.
+-   `relation`: To establish relationships between different models (e.g., a `product` model having a `category` field that relates to a `taxonomy` model).
+-   `enum`: For fields with a predefined set of allowed values.
+-   `file`: For handling file uploads (e.g., `profilePicture` in the `user` model).
+-   `code`: For storing code snippets (e.g., `script` in an `endpoint` model).
+
+Fields can also have properties like `required`, `unique`, `min`, `max`, `default` values, and `hint` for user guidance.
+
+This dynamic and field-based approach to data modeling provides the flexibility needed for rapidly building complex data-driven applications.
+
+**Next: Data Models**

package/doc/custom-api-endpoints.md

@@ -0,0 +1,40 @@
+# Custom API Endpoints: Create Dynamic HTTP Routes
+
+The `data-primals-engine` allows you to extend its API by defining **Custom API Endpoints**. These endpoints are dynamic HTTP routes that execute server-side JavaScript code directly from the backend, providing immense flexibility for custom logic and integrations.
+
+## The `endpoint` Model
+
+Custom API Endpoints are managed through the built-in `endpoint` data model. Each document in this model represents a unique API route.
+
+### Key Fields of the `endpoint` Model:
+
+-   **`name`** (string, required): A human-readable name for the endpoint.
+-   **`path`** (string, required, unique): The URL path segment that comes after `/api/actions/`. For example, if `path` is `send-welcome-email`, the full endpoint URL would be `/api/actions/send-welcome-email`.
+-   **`method`** (enum, required): The HTTP method (GET, POST, PUT, PATCH, DELETE) that this endpoint responds to.
+-   **`code`** (code, required): The JavaScript script that will be executed when this endpoint is called. This script has access to various utilities and the request context.
+-   **`isActive`** (boolean, default: `true`): If checked, the endpoint is active and can be called.
+-   **`isPublic`** (boolean, default: `false`): If checked, this endpoint will be accessible without authentication. Use with caution.
+
+### Script Execution Environment
+
+The JavaScript `code` field provides a powerful environment for your custom logic:
+
+-   **`db`**: An object providing methods for interacting with the database (e.g., `db.find`, `db.create`, `db.update`, `db.delete`).
+-   **`logger`**: A logging utility (e.g., `logger.info`, `logger.error`).
+-   **`env`**: Access to environment variables defined in the `env` model.
+-   **`request`**: The incoming HTTP request object, containing `body`, `query`, `params`, and `headers`.
+-   **`user`**: The authenticated user object (if `isPublic` is false).
+
+The script's return value will be sent as the JSON response to the client.
+
+### Example `endpoint` Definition (from `defaultModels.js`):
+
+```javascript
+// The default code for a new endpoint
+logger.info('Custom endpoint executed with body:', request.body);
+return { success: true, message: 'Endpoint executed!', received: request.body };
+```
+
+This feature allows developers to build highly customized backend logic directly within the `data-primals-engine`, making it a versatile tool for various application needs.
+
+**Next: Data Models**

package/doc/dashboards-kpis-charts.md

@@ -0,0 +1,49 @@
+# Dashboards, KPIs, and Charts: Track Your Key Metrics
+
+The `data-primals-engine` provides robust features for visualizing and tracking your application's performance and key metrics through customizable Dashboards and Key Performance Indicators (KPIs).
+
+## Dashboards
+
+A **Dashboard** serves as a personalized overview, allowing users to arrange and display various KPIs and charts relevant to their needs.
+
+### Key Fields of the `dashboard` Model:
+
+-   **`name`** (string, required): The customizable display name of the dashboard.
+-   **`description`** (string, optional): Provides additional context for the dashboard.
+-   **`layout`** (code - JSON, required): A JSON structure describing the organization of KPIs and charts. This defines how elements are arranged (e.g., in columns).
+    -   **Example Layout**: `"{ \"type\": \"columns\", \"columns\": [ [\"kpi_id_1\"], [\"kpi_id_2\", \"kpi_id_3\"] ] }"`
+-   **`settings`** (code - JSON, optional): JSON settings for the dashboard, such as a `defaultTimeRange` (e.g., 'last_7_days') or a `refreshInterval` in seconds.
+-   **`isDefault`** (boolean, default: `false`): If `true`, this dashboard is displayed by default for the user.
+
+## Key Performance Indicators (KPIs)
+
+**KPIs** are quantifiable measures used to evaluate the success of an organization, employee, etc., in meeting objectives. In `data-primals-engine`, KPIs are highly configurable to extract meaningful insights from your data.
+
+### Key Fields of the `kpi` Model:
+
+-   **`name`** (string, required): The display name of the KPI (e.g., "Total Revenue", "Active Users").
+-   **`description`** (string, optional): Additional information about the KPI.
+-   **`targetModel`** (string, required): The name of the data model on which to calculate the KPI (e.g., `order`, `user`).
+-   **`aggregationType`** (enum, required): The type of calculation to perform:
+    -   `count`: Count the number of documents.
+    -   `sum`: Sum a numeric field.
+    -   `avg`: Calculate the average of a numeric field.
+    -   `min`: Find the minimum value of a numeric field.
+    -   `max`: Find the maximum value of a numeric field.
+-   **`aggregationField`** (string, optional): The name of the numeric field to apply the aggregation to (e.g., `totalAmount`). Not required for `count` type.
+-   **`matchFormula`** (code - JSON, optional): A MongoDB `$match` JSON filter to apply before aggregation (e.g., `{ "status": "delivered" }`). This allows you to filter the data used for the KPI calculation.
+-   **`showTotal`** (boolean): Whether to display a grand total alongside the KPI.
+-   **`showPercentTotal`** (boolean): Whether to display the KPI value as a percentage of a total.
+-   **`totalMatchFormula`** (code - JSON, optional): A MongoDB `$match` JSON filter for calculating the total when `showPercentTotal` is enabled.
+-   **`unit`** (string, optional): The unit to display with the KPI value (e.g., "€", "$", "users").
+-   **`icon`** (string, optional): An icon name (e.g., `FaUsers`, `FaShoppingCart`) to visually represent the KPI.
+-   **`order`** (number, optional): The display order of the KPI on a dashboard.
+-   **`color`** (string, optional): A color associated with the KPI for visual distinction.
+
+## Charts
+
+While the `kpi` model focuses on single aggregated values, the engine also supports the integration of charts to visualize data trends and distributions over time or across categories. These are typically rendered based on underlying KPI data or direct queries, offering a richer analytical experience.
+
+By combining Dashboards, configurable KPIs, and charting capabilities, `data-primals-engine` empowers users to effectively monitor and understand their key business metrics.
+
+**Next: Users**