data-primals-engine 1.6.3 → 1.7.0

package/doc/data-management.md

@@ -0,0 +1,120 @@
+# Data Management: Create, Read, Update, and Delete Entries
+
+The `data-primals-engine` provides a powerful and flexible API for performing standard CRUD (Create, Read, Update, Delete) operations on all defined data models. These generic endpoints allow you to interact with your data programmatically.
+
+## Authentication
+
+All data management operations require authentication. You must provide a valid `_user` query parameter (your username) and a `Bearer` token in the `Authorization` header for most operations.
+
+## API Endpoints Overview
+
+The following endpoints are available for generic data management:
+
+### 1. Create Data: `POST /api/data`
+
+This endpoint allows you to create one or more new documents within a specified model.
+
+-   **Method**: `POST`
+-   **Summary**: Create one or more documents
+-   **Description**: Creates one or more new documents in a specified model.
+-   **Security**: `BearerAuth`
+-   **Parameters**:
+    -   `_user` (query, required): Your username for authentication.
+    -   `model` (body, required): The name of the model in which to create the documents (e.g., 'user', 'product').
+    -   `data` (body, required): The data to insert. Can be a single object or an array of objects. Using the `$find` operator is recommended for relational data.
+    -   `lang` (query, optional, default: 'en'): Language used for error messages.
+-   **Example Request Body**:
+    ```json
+    {
+      "model": "product",
+      "data": {
+        "name": "New Product",
+        "price": 99.99,
+        "currency": "60d0fe4f5311236168a109cb"
+      }
+    }
+    ```
+-   **Responses**: `201` (Document(s) successfully created), `400` (Invalid data), `401` (Unauthorized).
+
+### 2. Search Data: `POST /api/data/search`
+
+This endpoint allows you to search and retrieve documents from a specified model. It supports powerful filtering, sorting, pagination, and relation population.
+
+-   **Method**: `POST`
+-   **Summary**: Search among data
+-   **Description**: Search across all data of the specified model.
+-   **Security**: `BearerAuth`
+-   **Parameters**:
+    -   `_user` (query, required): Your username for authentication.
+    -   `model` (body, required): The name of the data model (e.g., 'user', 'product').
+    -   `filter` (body, optional): MongoDB filter JSON object for the search.
+    -   `sort` (query, optional): Sort parameter by field (e.g., `fieldName:ASC;fieldName2:DESC`).
+    -   `limit` (query, optional, default: 1000): Maximum number of documents to return.
+    -   `offset` (query, optional, default: 0): Number of documents to skip (for pagination).
+    -   `depth` (query, optional, default: 1): Population depth for 'relation' type fields.
+    -   `lang` (query, optional, default: 'en'): Language used for error messages.
+-   **Example Request Body (Filter)**:
+    ```json
+    {
+      "model": "product",
+      "filter": {
+        "price": { "$gt": 50 },
+        "category": "60d0fe4f5311236168a109cc"
+      }
+    }
+    ```
+-   **Responses**: `200` (Success with returned data), `401` (Unauthorized).
+
+### 3. Update Data: `PUT /api/data/{id}` or `PUT /api/data` (Bulk)
+
+You can update a single document by its ID or perform bulk updates using a filter.
+
+-   **Method**: `PUT`
+-   **Summary**: Update a document (by ID) or Bulk update documents
+-   **Description**: Updates an existing document using its ID, or multiple documents using a filter.
+-   **Security**: `BearerAuth`
+-   **Parameters**:
+    -   `_user` (query, required): Your username for authentication.
+    -   `id` (path, required for single update): The unique identifier (`_id`) of the document to update.
+    -   `model` (body, required): The name of the data model.
+    -   `data` (body, required): The data to edit. For bulk updates, this object will contain the fields to update.
+    -   `filter` (body, optional, for bulk update): MongoDB filter JSON object to select documents for bulk update.
+    -   `lang` (query, optional, default: 'en'): Language used for error messages.
+-   **Example Request Body (Single Update)**:
+    ```json
+    {
+      "model": "product",
+      "data": {
+        "price": 120.00
+      }
+    }
+    ```
+-   **Responses**: `200` (Document successfully updated), `400` (Invalid data), `401` (Unauthorized), `404` (Document not found).
+
+### 4. Delete Data: `DELETE /api/data`
+
+This endpoint allows you to permanently delete one or more documents.
+
+-   **Method**: `DELETE`
+-   **Summary**: Delete one or more document(s)
+-   **Description**: Permanently deletes a document using its ID, or multiple documents using a filter.
+-   **Security**: `BearerAuth`
+-   **Parameters**:
+    -   `_user` (query, required): Your username for authentication.
+    -   `ids` (body, optional): An array of identifiers of the documents to delete.
+    -   `filter` (body, optional): The MongoDB JSON filter to apply for bulk deletion.
+    -   `lang` (query, optional, default: 'en'): Language used for error messages.
+-   **Example Request Body (Bulk Delete)**:
+    ```json
+    {
+      "model": "product",
+      "filter": {
+        "status": "draft"
+      }
+    }
+    ```
+-   **Responses**: `200` (Document successfully deleted), `401` (Unauthorized), `404` (Document not found).
+
+These generic CRUD endpoints provide a flexible and powerful way to interact with all your data models within the `data-primals-engine`.
+
+**Next: Dashboards, KPIs, and Charts**

package/doc/data-models.md

@@ -0,0 +1,75 @@
+# Data Models: Structuring Your Information
+
+Data Models are the fundamental building blocks in `data-primals-engine`, defining how your application's data is structured and stored. They are essentially blueprints for collections of documents, enabling you to organize and manage diverse types of information.
+
+## What is a Data Model?
+
+In `data-primals-engine`, a data model is a flexible definition of a data entity. Unlike traditional relational databases, models here are schema-less (thanks to MongoDB), meaning you can evolve their structure over time without rigid migration processes. Each model corresponds to a collection in your MongoDB database.
+
+## Defining a Model
+
+Models can be defined either through the platform's user interface or by providing a JSON schema. Key attributes of a model include:
+
+-   **`name`** (string, unique): The technical name of the model (e.g., `product`, `user`, `order`). This is used for API interactions and internal references.
+-   **`description`** (string): A brief explanation of the model's purpose.
+-   **`icon`** (string): An icon (e.g., a Font Awesome class) to visually represent the model in the UI.
+-   **`tags`** (array of strings): Keywords for categorization and filtering.
+-   **`locked`** (boolean): Indicates if the model is a system model and cannot be modified or deleted via the standard UI (e.g., `user`, `permission`).
+-   **`fields`** (array of objects): The core of the model, defining its attributes.
+
+## Fields: The Attributes of a Model
+
+Each field within a model defines a specific piece of data. Fields have various properties to control their type, behavior, and validation:
+
+-   **`name`** (string, unique within model): The name of the attribute (e.g., `title`, `price`, `email`).
+-   **`type`** (string, required): The data type of the field. Common types include:
+    -   `string`, `string_t` (translatable string)
+    -   `number`
+    -   `boolean`
+    -   `datetime`, `date`
+    -   `email`, `url`, `phone`, `password`
+    -   `richtext`, `richtext_t` (translatable rich text)
+    -   `enum` (predefined list of values)
+    -   `file` (for file uploads)
+    -   `relation` (to link to another model)
+    -   `array` (for lists of values or sub-documents)
+    -   `code` (for storing code snippets, e.g., JSON, JavaScript)
+-   **`required`** (boolean): If `true`, the field must have a value.
+-   **`unique`** (boolean): If `true`, the field's value must be unique across all documents in the collection.
+-   **`default`**: A default value for the field.
+-   **`min`, `max`**: Minimum and maximum values for `number` fields, or length for `string` fields.
+-   **`relation`** (string, for `relation` type): The name of the model this field relates to.
+-   **`multiple`** (boolean, for `relation` type): If `true`, the field can relate to multiple documents in the target model.
+-   **`hint`** (string): A helpful description displayed in the UI.
+
+## Example: The `product` Model
+
+The `product` model, defined in `defaultModels.js`, illustrates how fields are used to structure information about a product:
+
+```javascript
+product: {
+    name: 'product',
+    "icon": "FaShoppingBag",
+    "description": "",
+    "tags": ["ecommerce", "products"],
+    fields: [
+        { name: 'name', type: 'string_t', required: true },
+        { name: 'image', type: 'array', itemsType: 'file', mimeTypes: ['image/jpeg', 'image/png', 'image/gif', 'image/webp'] },
+        { name: 'description', type: 'richtext_t' },
+        { name: 'price', type: 'number', required: true },
+        { name: 'currency', type: 'relation', relation: 'currency', required: true },
+        { name: 'billingFrequency', type: 'enum', items: ['none', 'monthly', 'yearly'] },
+        { name: 'slug', type: 'string', required: true, unique: true },
+        { name: 'brand', type: 'relation', relation: 'brand' },
+        { name: 'category', type: 'relation', relation: 'taxonomy' },
+        { name: 'seoTitle', type: 'string_t' },
+        { name: 'seoDescription', type: 'string_t' }
+    ]
+},
+```
+
+This example shows how a `product` can have a translatable `name`, multiple `image` files, a `price` with a `currency` relation, and be linked to a `brand` and `category` (taxonomy).
+
+By defining models and their fields, you create a robust and adaptable data foundation for your `data-primals-engine` application.
+
+**Next: Data Management**

package/doc/index.md

@@ -0,0 +1,14 @@
+# data-primals-engine Documentation
+
+Welcome to the `data-primals-engine` documentation! This guide will help you understand the core concepts, features, and how to effectively use the platform to build powerful data-driven applications.
+
+## Table of Contents
+
+-   Core Concepts: Data Modeling Fundamentals
+-   Data Models: Structuring Your Information
+-   Data Management: Create, Read, Update, and Delete Entries
+-   Custom API Endpoints: Create Dynamic HTTP Routes
+-   Dashboards, KPIs, and Charts: Track Your Key Metrics
+-   Users: Manage Access to the Platform
+-   Roles and Permissions: Define Who Can See and Do What
+-   Automation with Workflows: Create Automated Processes

package/doc/roles-permissions.md

@@ -0,0 +1,43 @@
+# Roles and Permissions: Define Who Can See and Do What
+
+The `data-primals-engine` implements a robust Role-Based Access Control (RBAC) system to manage user authorizations. This system relies on two core models: `role` and `permission`, allowing for granular control over what users can access and perform within the platform.
+
+## Permissions
+
+A **Permission** defines a specific action or access right within the system. Permissions can be very broad (e.g., "admin.full_access") or highly specific (e.g., "product.edit").
+
+### Key Fields of the `permission` Model:
+
+-   **`name`** (string, required): A unique identifier for the permission (e.g., `model.create`, `product.read`, `user.delete`).
+-   **`description`** (richtext, optional): A detailed explanation of what the permission grants.
+-   **`filter`** (code - JSON, optional): A JSON filter that restricts the scope of this permission. This is a powerful feature for implementing granular access control. For example, a `product.edit` permission could have a filter `{ "owner": "{_user}" }` to allow a user to only edit products they own. The target model is typically deduced from the permission name (e.g., `product.edit` implies the `product` model).
+
+## Roles
+
+A **Role** is a collection of permissions. Instead of assigning individual permissions to each user, you assign roles, which simplifies management. A user can have multiple roles.
+
+### Key Fields of the `role` Model:
+
+-   **`name`** (string, required, unique): The name of the role (e.g., `Administrator`, `Editor`, `Viewer`).
+-   **`permissions`** (multiple relation to `permission` model): A list of `permission` documents associated with this role. Any user assigned this role will inherit all its permissions.
+
+## How RBAC Works
+
+1.  **Define Permissions**: Create specific `permission` documents for every action or resource you want to control. Use the `filter` field for fine-grained control.
+2.  **Create Roles**: Group relevant permissions into `role` documents.
+3.  **Assign Roles to Users**: Assign one or more `role` documents to each `user` document.
+
+When a user attempts an action (e.g., accessing an API endpoint or modifying a data entry), the system checks if the user's assigned roles grant them the necessary permissions. If a permission has a `filter`, that filter is applied to the data access query, ensuring the user only interacts with allowed subsets of data.
+
+### User Permissions (Exceptions)
+
+The `userPermission` model allows for exceptions to the standard role-based permissions. This model can grant or revoke specific permissions for an individual user, either permanently or temporarily.
+
+-   **`user`** (relation to `user` model): The user for whom the exception applies.
+-   **`permission`** (relation to `permission` model): The specific permission being granted or revoked.
+-   **`isGranted`** (boolean, required): `true` to grant the permission, `false` to explicitly revoke it.
+-   **`expiresAt`** (datetime, optional): If set, the exception is temporary.
+
+This comprehensive RBAC system ensures that your `data-primals-engine` application remains secure and that users only have access to the functionalities and data they are authorized to use.
+
+**Next: Automation with Workflows**

package/doc/users.md

@@ -0,0 +1,30 @@
+# Users: Manage Access to the Platform
+
+The `data-primals-engine` includes a robust user management system, allowing you to control access to your data and functionalities. The core of this system is the `user` model, which stores all necessary information about platform users.
+
+## The `user` Model
+
+The `user` model is a fundamental, system-locked model that defines the attributes of each user account.
+
+### Key Fields of the `user` Model:
+
+-   **`username`** (string, required, unique): The unique identifier for the user, used for login.
+-   **`password`** (password): The user's hashed password.
+-   **`gender`** (enum): Optional field for gender, with options like `male`, `female`, `other`, `prefer_not_to_say`.
+-   **`contact`** (relation to `contact` model): Links to a `contact` document, which can store detailed personal information like `firstName`, `lastName`, `email`, and `phone`.
+-   **`roles`** (multiple relation to `role` model): Assigns one or more roles to the user, determining their permissions within the system.
+-   **`lang`** (relation to `lang` model): Specifies the preferred language for the user interface and content.
+-   **`profilePicture`** (file): Allows users to upload a profile image (supports JPEG and PNG).
+-   **`tokens`** (multiple relation to `token` model): Stores authentication tokens associated with the user.
+
+### User Authentication
+
+The engine provides mechanisms for user authentication, typically involving username/password credentials to issue JWT (JSON Web Tokens). These tokens are then used to secure API requests, ensuring that only authorized users can access or modify data.
+
+### User-Specific Data
+
+Each document created within the `data-primals-engine` is associated with a user via the `_user` field. This ensures data isolation and allows for granular access control, where users primarily interact with data they own or have been granted access to.
+
+Effective user management is crucial for maintaining security and organizing access within your `data-primals-engine` application.
+
+**Next: Roles and Permissions**

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "data-primals-engine",
-  "version": "1.6.3",
+  "version": "1.7.0",
   "description": "data-primals-engine is a package responsible from handling large amount of data using MongoDB in a practical and performant way. It can also get workflow models working (for automation), and fully supports internationalisation. It also has integrated AI assistant.",
   "main": "src/engine.js",
   "type": "module",
@@ -28,14 +28,18 @@
     "@rollup/rollup-linux-x64-gnu": "4.6.1"
   },
   "resolutions": {
-    "on-headers": "1.1.0",
-    "brace-expansion": "2.0.2",
-    "prismjs": "1.30.0",
+    "on-headers": ">=1.1.0",
+    "brace-expansion": ">=2.0.2",
+    "prismjs": ">=1.30.0",
     "xml2js": ">=0.5.0",
     "tar-fs": ">=3.1.1",
     "vite": ">=7.0.8",
     "node-forge": ">=1.3.2",
-    "js-yaml": ">=4.1.1"
+    "js-yaml": ">=4.1.1",
+    "mdast-util-to-hast": ">=13.2.1",
+    "jws": ">=3.2.3",
+    "qs": ">=6.14.1",
+    "undici": ">=6.23.0"
   },
   "overrides": {
     "react-syntax-highlighter": {
@@ -46,6 +50,12 @@
     },
     "refractor": {
       "prismjs": "1.30.0"
+    },
+    "body-parser": {
+      "qs": ">=6.14.1"
+    },
+    "stripe": {
+      "qs": ">=6.14.1"
     }
   },
   "repository": {
@@ -61,11 +71,11 @@
     "./*": "./src/*.js"
   },
   "dependencies": {
-    "@langchain/core": "^0.3.79",
+    "@langchain/core": "^0.3.80",
     "archiver": "^7.0.1",
     "aws-sdk": "^2.1692.0",
     "bcrypt": "^6.0.0",
-    "body-parser": "^2.2.1",
+    "body-parser": ">=2.2.1",
     "chalk": "^5.4.1",
     "check-disk-space": "^3.4.0",
     "compression": "^1.8.1",
@@ -86,7 +96,7 @@
     "mongodb": "^6.18.0",
     "node-cache": "^5.1.2",
     "node-schedule": "^2.1.1",
-    "nodemailer": "^7.0.10",
+    "nodemailer": "^7.0.11",
     "openai": "^6.9.1",
     "passport": "^0.7.0",
     "passport-saml-encrypted": "^0.1.13",
@@ -101,14 +111,14 @@
     "sirv": "^3.0.2",
     "stripe": "^20.0.0",
     "swagger-ui-express": "^5.0.1",
-    "tar": "^7.5.2",
+    "tar": "7.5.4",
     "tinycolor2": "^1.6.0",
     "uniqid": "^5.4.0",
     "vitest": "^3.2.4",
     "yaml": "^2.8.1"
   },
   "peerDependencies": {
-    "express": "^5.1.0",
+    "express": "^5.2.1",
     "passport-azure-ad": "^4.3.5",
     "passport-google-oauth20": "^2.0.0",
     "react": "18.3.1",

package/src/client.js

@@ -1,4 +1,6 @@
-// data-primals-engine/client
-// Ce fichier sert de point d'entrée pour les modules qui peuvent être utilisés en toute sécurité côté client (navigateur).
-
-export { Event } from './events.js';
+// data-primals-engine/client
+// Ce fichier sert de point d'entrée pour les modules qui peuvent être utilisés en toute sécurité côté client (navigateur).
+
+export { Event } from './events.js';
+export { GameObject, Logger, BenchmarkTool } from './gameObject.js';
+export { Config } from './config.js';

package/src/constants.js

@@ -277,7 +277,7 @@ metaModels['erp'] = { load: [ 'accountingExercise', 'accountingLineItem', 'accou
  * Available model field attributes
  * @type {string[]}
  */
-export const allowedFields = ['locked', 'hiddenable', 'anonymized', 'condition', 'color', 'index', 'indexType', 'type', 'required', 'hint', 'default', 'validate', 'unique', 'name', 'placeholder', 'asMain'];
+export const allowedFields = ['locked', 'hiddenable', 'encrypted', 'anonymized', 'condition', 'color', 'index', 'indexType', 'type', 'required', 'hint', 'default', 'validate', 'unique', 'name', 'placeholder', 'asMain'];
 
 
 

package/src/core.js

@@ -40,7 +40,10 @@ export const isUnsecureKey = (key) => {
 export const parseSafeJSON = (json) => JSON.parse(json, (key, value) => isUnsecureKey(key) ? undefined : value);
 
 
-export const isDate = dt => String(new Date(dt)) !== 'Invalid Date'
+export const isDate = dt => {
+    if (dt === null || typeof dt === 'undefined') return false;
+    return String(new Date(dt)) !== 'Invalid Date';
+};
 
 export const safeAssignObject = (obj, key, value) => {
     if( !isUnsecureKey(key)){
@@ -157,11 +160,11 @@ export function isPathRelativeTo(dir, parent) {
  * @param value string value
  */
 export function isGUID(value) {
-    return typeof(value) === 'string' && value.match(/^[a-zA-Z0-9]{8}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{12}$/);
+    return !!(typeof(value) === 'string' && value.match(/^[a-zA-Z0-9]{8}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{12}$/));
 }
 
 export function isPlainObject(obj) {
-    return typeof obj === 'object' && obj !== null;
+    return typeof obj === 'object' && obj !== null && !Array.isArray(obj) && !(obj instanceof Date);
 }
 
 export function escapeRegExp(string) {
@@ -210,7 +213,11 @@ function splitmix32(a) {
 }
 
 
-export const getRand = () =>splitmix32(seed);
+export const getRand = () => {
+    const result = splitmix32(seed);
+    seed++; // Simple increment to change the state for the next call
+    return result;
+};
 
 const MAX_RANGE_SIZE = 2n ** 64n
 const buffer = new BigUint64Array(512)
@@ -371,14 +378,26 @@ export const event_off = (name, callback) => {
 };
 
 
-export function slugify(str,replacer='-', replaceUnicode=false) {
-    str = str.replace(/^\s+|\s+$/g, ''); // trim leading/trailing white space
-    str = str.toLowerCase(); // convert string to lowercase
-    if( replaceUnicode)
-        str = str.replace(/[^a-z0-9 -]/g, ''); // remove any non-alphanumeric characters
-    str = str.replace(/\s+/g, replacer) // replace spaces with hyphens
-        .replace(/-+/g, replacer); // remove consecutive hyphens
-    return str;
+export function slugify(str,replacer='-', removeNonAscii=false) {
+
+    if (!str) return '';
+
+    // 1. Convert to string, lowercase, and trim whitespace.
+    str = str.toString().toLowerCase().trim();
+
+    // 2. Normalize Unicode characters (e.g., è -> e). This is now always done.
+    str = str.normalize('NFD').replace(/[\u0300-\u036f]/g, '');
+
+    // 3. Remove any remaining non-ASCII characters if the flag is set.
+    if (removeNonAscii) {
+        str = str.replace(/[^a-z0-9\s-]/g, '');
+    }
+
+    // 4. Replace spaces with the replacer and clean up consecutive/trailing replacers.
+    return str
+        .replace(/\s+/g, replacer) // Replace spaces with the replacer.
+        .replace(new RegExp(`[^a-z0-9${replacer}]`, 'g'), '') // Remove any character that is not a letter, a number, or the replacer itself.
+        .replace(new RegExp(`${replacer}+`, 'g'), replacer); // Replace multiple replacers with a single one.
 }
 
 // resource : https://stackoverflow.com/questions/190852/how-can-i-get-file-extensions-with-javascript

package/src/defaultModels.js

@@ -303,7 +303,7 @@ export const defaultModels = {
         "icon": "FaTags",
         "description": "",
         "tags": ["cms", "organization"],
-        fields: [
+        "fields": [
             { name: 'name', type: 'string_t', required: true, color: '#71A314' },
             { name: 'parent', type: 'relation', relation: 'taxonomy', color: '#233607' }, // Relation vers la taxonomie parente
             { name: 'type', type: 'enum', items: ['keyword', 'category'], color: '#BFBFBF' },