data-primals-engine 1.0.0

package/LICENSE.md

@@ -0,0 +1,7 @@
+Copyright 2025 - 2026, data-primals-engine (https://npm.io/package/data-primals-engine)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

package/README.md

@@ -0,0 +1,169 @@
+# Data Primals Engine
+
+**data-primals-engine** est un moteur backend **Node.js** puissant et flexible, conçu pour accélérer le développement d'applications complexes basées sur les données. Construit sur **Express.js** et **MongoDB**, il offre une architecture modulaire, un système de modélisation de données dynamique, des workflows d'automatisation, une gestion avancée des utilisateurs, et bien plus encore.
+
+> Que vous construisiez un CRM, une boutique e-commerce, un CMS ou une application SaaS complexe, **data-primals-engine** vous fournit les fondations robustes pour vous concentrer sur ce qui rend votre application unique.
+
+---
+
+## 🚀 Caractéristiques Principales
+
+- **Modélisation de données dynamique** : Créez des schémas de données via JSON sans migration de code.
+- **API RESTful puissante** : Opérations CRUD avancées, recherche et filtrage inclus.
+- **Architecture modulaire** : Ajoutez vos propres modules dynamiquement.
+- **Workflows d'automatisation** : Créez des processus métier automatisés.
+- **Authentification & autorisation** : Fournisseurs d'utilisateurs extensibles, rôles et permissions granulaires.
+- **📃 Packs de démarrage** : CRM, e-commerce, site vitrine, etc.
+- **🧠 Intégration IA** : Support de OpenAI, Google Gemini via LangChain.
+- **i18n complet** : Multilingue avec traduction des interfaces et validations.
+- **📄 Documentation automatique** : Swagger disponible par défaut sur `/api-docs`.
+
+---
+
+## 🚀 Prérequisa
+
+- Node.js ≥ 18
+- MongoDB installé ou accessible
+- NPM ou Yarn pour la gestion des dépendances
+
+---
+
+## ⚡ Installation et Lancement Rapide
+
+```bash
+git clone https://votre-repository/data-primals-engine.git
+cd data-primals-engine
+npm install
+```
+
+Créez un fichier `.env` à la racine :
+```env
+MONGO_DB_URL=mongodb://127.0.0.1:27017
+```
+
+Démarrer le serveur :
+```bash
+# Mode développement
+tnpm run devserver
+
+# Mode production
+npm run server
+```
+
+Par défaut, l'application tourne sur le port **7633**.
+
+---
+
+## 📊 Concepts Clés
+
+### 1. 📂 Modèles
+Définis en JSON, les modèles structurent vos données.
+
+```json
+{
+  "name": "product",
+  "description": "Fiche produit e-commerce",
+  "fields": [
+    { "name": "name", "type": "string", "required": true },
+    { "name": "price", "type": "number", "required": true },
+    { "name": "stock", "type": "number", "default": 0 },
+    { "name": "category", "type": "relation", "relation": "taxonomy" }
+  ]
+}
+```
+
+### 2. 📊 Modules
+Fonctionnalités activables via configuration :
+- `mongodb`, `data`, `user`, `workflow`, `file`, `assistant`, `swagger`
+
+### 3. 🏘️ Packs de Démarrage
+- **E-commerce** : Produits, commandes, KPIs...
+- **CRM** : Contacts, opportunités, interactions
+- **Site web / blog** : Pages, articles, multi-langue
+
+### 4. ⚒ Workflows
+Automatisez vos processus avec :
+- **Triggers** : DataAdded, DataUpdated, Scheduled, Manual
+- **Actions** : CreateData, UpdateData, SendEmail, ApiCall
+
+Exemple :
+> Lors d'une nouvelle commande : Envoyer un mail au client, mettre à jour le stock, notifier la logistique.
+
+---
+
+## 🔧 API : Exemple avec `curl`
+
+### Créer un produit
+```bash
+curl -X POST http://localhost:7633/api/data \
+     -H "Content-Type: application/json" \
+     -H "_user: demouser" \
+     -H "Authorization: Bearer demotoken" \
+     -d '{
+           "model": "product",
+           "data": {
+             "name": "Widget Extraordinaire",
+             "price": 99.99,
+             "stock": 150,
+             "published": true
+           }
+         }'
+```
+
+### Rechercher des produits
+```bash
+curl -X POST http://localhost:7633/api/data/search \
+     -H "Content-Type: application/json" \
+     -H "_user: demouser" \
+     -H "Authorization: Bearer demotoken" \
+     -d '{
+           "model": "product",
+           "filter": {
+             "price": {
+               "$gt": 50
+             },
+             "published": true
+           },
+           "sort": "price:ASC",
+           "limit": 10,
+           "depth": 1
+         }'
+```
+
+---
+
+## 📁 Structure du Projet
+```
+data-primals-engine/
+├── src/
+│   ├── engine.js
+│   ├── modules/
+│   ├── providers/
+│   ├── constants.js
+│   ├── defaultModels.js
+├── config/
+│   ├── models/
+│   └── packs/
+└── server.js
+```
+
+---
+
+## 🌟 Comment Contribuer ?
+
+1. Forkez le repo
+2. Créez une branche : `git checkout -b feature/nom`
+3. Commitez : `git commit -m "Ajout de ..."`
+4. Push : `git push origin feature/nom`
+5. Ouvrez une pull request
+
+Merci de laisser une ⭐ au repo si vous aimez le projet !
+
+---
+
+## 📚 Licence
+Distribué sous licence **MIT**. Voir le fichier `LICENSE`.
+
+---
+
+## 🔼 Retour en haut

package/package.json

@@ -0,0 +1,72 @@
+{
+  "name": "data-primals-engine",
+  "version": "1.0.0",
+  "description": "datas-primals-engine is the package responsible from handling large amount of data in a practical and performant way. It can handle large amount of data in a practical and performant way. It can also get workflow models working (for automation), and fully supports internationalisation.",
+  "main": "src/engine.js",
+  "type": "module",
+  "scripts": {
+    "devserver": "cross-env NODE_ENV=development node server.js",
+    "server": "cross-env NODE_ENV=production node server.js",
+    "migrate:create": "node src/migrate.js create",
+    "migrate:up": "node src/migrate.js up",
+    "migrate:down": "node src/migrate.js down",
+    "migrate:revert": "node src/migrate.js revert",
+    "migrate:status": "node src/migrate.js status"
+  },
+  "exports": {
+    ".": "./src/index.js",
+    "./modules/*": "./src/modules/*.js",
+    "./*": "./src/*.js"
+  },
+  "dependencies": {
+    "@langchain/core": "^0.3.62",
+    "@langchain/google-genai": "^0.2.14",
+    "@langchain/openai": "^0.5.18",
+    "archiver": "^7.0.1",
+    "aws-sdk": "^2.1692.0",
+    "body-parser": "^1.20.3",
+    "chalk": "^5.4.1",
+    "check-disk-space": "^3.4.0",
+    "compression": "^1.8.0",
+    "connect-mongo": "^5.1.0",
+    "cookie-parser": "^1.4.7",
+    "csv-parse": "^5.6.0",
+    "csv-parser": "^3.2.0",
+    "express": "^4.21.2",
+    "express-csrf-double-submit-cookie": "^1.2.1",
+    "express-session": "^1.18.1",
+    "express-sitemap-xml": "^3.1.0",
+    "juice": "^11.0.1",
+    "mathjs": "^14.4.0",
+    "mongodb": "^6.12.0",
+    "node-schedule": "^2.1.1",
+    "nodemailer": "^6.10.0",
+    "rate-limiter-flexible": "^5.0.5",
+    "swagger-ui-express": "^5.0.1",
+    "sirv": "^3.0.1",
+    "tar": "^7.4.3",
+    "tar-fs": "^3.0.8",
+    "uniqid": "^5.4.0",
+    "yaml": "^2.8.0"
+  },
+  "devDependencies": {
+    "@eslint/js": "^9.17.0",
+    "cross-env": "^7.0.3",
+    "eslint": "^9.17.0",
+    "globals": "^15.14.0",
+    "process": "^0.11.10"
+  },
+  "keywords": [
+    "mongodb",
+    "data",
+    "primals",
+    "automation",
+    "aws",
+    "bucket", "S3"
+  ],
+  "author": "anonympins",
+  "license": "MIT",
+  "engines": {
+    "node": ">=18.0.0"
+  }
+}

package/server.js

@@ -0,0 +1,85 @@
+
+
+// ===============
+
+//set ES modules to be loaded by the script
+import process from "node:process";
+import {Config} from "../data-primals-engine/src/config.js";
+import {Engine} from "../data-primals-engine/src/engine.js";
+import {BenchmarkTool, GameObject, Logger} from "../data-primals-engine/src/gameObject.js";
+import {
+    createModel, DATAS_API_TOKEN,
+    getModels, getResource,
+    searchData,
+    validateModelStructure
+} from "../data-primals-engine/src/modules/data.js";
+import {availableLangs, install, langs, maxBytesPerSecondThrottleFile} from "../data-primals-engine/src/constants.js";
+import fs from "node:fs";
+import {
+    event_on, getFileExtension,
+    shuffle,
+    uuidv4
+} from "../data-primals-engine/src/core.js";
+import {datasCollection, filesCollection, modelsCollection, packsCollection} from "../data-primals-engine/src/modules/mongodb.js";
+import {translations} from "../data-primals-engine/src/i18n.js";
+import {getFieldValueHash} from "../data-primals-engine/src/data.js";
+
+import swaggerUi from 'swagger-ui-express';
+
+import YAML from 'yaml';
+import util from "node:util";
+import path from "node:path";
+import { getAllPacks} from "../data-primals-engine/src/packs.js";
+import {middlewareAuthenticator, userInitiator} from "../data-primals-engine/src/modules/user.js";
+import {defaultModels} from "../data-primals-engine/src/defaultModels.js";
+Config.Set("modules", ["mongodb", "data", "file", "bucket", "workflow","user", "assistant"])
+Config.Set("middlewares", []);
+
+const bench = GameObject.Create("Benchmark");
+const timer = bench.addComponent(BenchmarkTool);
+timer.start();
+
+const swaggerDocument = YAML.parse(fs.readFileSync(process.cwd()+'/swagger-en.yml', 'utf8'));
+
+const engine = Engine.Create();
+
+const models = defaultModels;
+
+const port = process.env.PORT || 7633;
+engine.start(port, async (r) => {
+    const logger = engine.getComponent(Logger);
+    logger.info("Loading and validating models...");
+    const ms = Object.values(models);
+
+    let dbModels = await getModels();
+
+    for(let i = 0; i < ms.length; ++i){
+        const model = ms[i];
+        validateModelStructure(model);
+        // Création des modèles
+        if( !dbModels.find(m =>m.name === model.name) )
+        {
+            model.locked = true;
+            const r = await createModel(model);
+            dbModels.push({...model, _id: r.insertedId });
+        }
+        else
+            logger.info('Model loaded (' + model.name + ')', {});
+    }
+
+    engine.use('/doc', swaggerUi.serve, swaggerUi.setup(swaggerDocument));
+
+    if( install )
+        await installAllPacks();
+
+    timer.stop();
+});
+
+const installAllPacks = async () => {
+
+    const packs = await getAllPacks();
+
+    console.log(util.inspect(packs, false, 20, true));
+    await packsCollection.deleteMany({ _user: { $exists : false }});
+    await packsCollection.insertMany(packs);
+}

package/src/config.js

@@ -0,0 +1,16 @@
+
+
+
+const config = [];
+
+export const Config = {
+    Get: (name, defaultValue) => {
+        if( config[name] === undefined ){
+            return defaultValue;
+        }
+        return config[name];
+    },
+    Set: (name, value) => {
+        config[name] = value;
+    }
+}

package/src/constants.js

@@ -0,0 +1,210 @@
+export const dbName = "engine";
+
+export const cookiesSecret = 'hoaivuymzovyoznllmafivpzaovphlejvalwjvelfhqochakfesv';
+
+
+export const langs = {
+    fr: "Français",
+    en: "English",
+    ar: "عربي",
+    fa: "فارسی",
+    it: "Italiano",
+    es: "Español",
+    el: "Ελληνικά",
+    de: "Deutsch",
+    cs: "Čeština",
+    sv: "Svenska",
+    pt: "Português",
+    ja: "日本語",
+    zh: "简体中文",
+    ru: "Русский",
+};
+
+export const availableLangs = [
+    "en",
+    "fa",
+    "ar",
+    "fr",
+    "it",
+    "es",
+    "pt",
+    "de",
+    "el",
+    "ru",
+    "cs",
+    "sv",
+];
+
+export const awsDefaultConfig = {
+    bucketName : 'bucket-primals',
+    region: 'eu-north-1',
+}
+
+// 10000 tiny users
+// 1000 modern users
+// 100 mega utilisateurs potentiality
+// 250 000 entrées par utilisateur
+export const maxModelsPerUser = 1000;
+export const maxTotalDataPerUser = 500000;
+export const maxDataPerModelPerUser = 10000;
+export const maxStringLength = 4096;
+export const maxPasswordLength = 100000;
+export const maxRichTextLength = 100000;
+export const maxExportCount = 50000;
+export const maxMagnetsDataPerModel = 100;
+export const maxMagnetsModels = 20;
+export const defaultMaxRequestData = 500;
+export const maxRequestData = 2500;
+export const maxPostData = 500;
+export const maxRelationsPerData = 1500;
+export const install = true;
+export const maxFilterDepth = 8;
+export const elementsPerPage = 30;
+
+
+export const maxAlertsPerUser = 15;
+export const storageSafetyMargin = 0.95;
+export const kilobytes = 1024;
+export const megabytes = 1024*1024;
+
+export const maxBytesPerSecondThrottleFile = 800*kilobytes;  // 800ko/s
+export const maxBytesPerSecondThrottleData = 200*kilobytes; // 200Ko/s
+
+export const searchRequestTimeout = 15000;
+
+export const maxModelsInCache = 100000;
+export const maxModelNameLength = 150;
+
+export const maxDataSize = '20mb';
+export const maxFileSize = 20 * 1024 * 1024; // 20 Mo
+
+export const mainFieldsTypes = ['string_t', 'string', 'url', 'enum', 'email', 'phone', 'date', 'datetime'];
+
+export const maxExecutionsByStep = 5;
+export const maxWorkflowSteps = 15;
+
+export const maxPrivateFileSize = 20 * megabytes; // Taille max par fichier privé (20 Mo)
+export const maxTotalPrivateFilesSizeFree = 125 * megabytes; // Stockage total pour les comptes gratuits (250 Mo)
+export const maxTotalPrivateFilesSizeStandard = 250 * megabytes; // Stockage total pour les comptes standard (250 Mo)
+export const maxTotalPrivateFilesSizePremium = 20000 * kilobytes * kilobytes; //
+
+export const plans = {
+    free:{
+        maxModelsPerUser,
+        maxTotalDataPerUser,
+        maxDataPerModelPerUser,
+        requestLimitPerHour: 7200,
+    },
+    premium: {
+        requestLimitPerHour: 250000,
+    }
+}
+//
+export const optionsSanitizer = {
+    allowedTags: [
+        "img",
+        "address", "article", "aside", "footer", "header", "h1", "h2", "h3", "h4",
+        "h5", "h6", "hgroup", "main", "nav", "section", "blockquote", "dd", "div",
+        "dl", "dt", "figcaption", "figure", "hr", "li", "main", "ol", "p", "pre",
+        "ul", "a", "abbr", "b", "bdi", "bdo", "br", "cite", "code", "data", "dfn",
+        "em", "i", "kbd", "mark", "q", "rb", "rp", "rt", "rtc", "ruby", "s", "samp",
+        "small", "span", "strong", "sub", "sup", "time", "u", "var", "wbr", "caption",
+        "col", "colgroup", "table", "tbody", "td", "tfoot", "th", "thead", "tr"
+    ],
+    disallowedTagsMode: 'discard',
+    allowedAttributes: {
+        a: [ 'href', 'name', 'target' ],
+        code: ['class'],
+        img: [ 'src', "alt","width","height","style" ]
+    },
+// Lots of these won't come up by default because we don't allow them
+    selfClosing: [ 'img', 'br', 'hr', 'area', 'base', 'basefont', 'input', 'link', 'meta' ],
+// URL schemes we permit
+    allowedSchemes: [ 'http', 'https', 'ftp', 'mailto', 'tel' ],
+    allowedSchemesByTag: {},
+    allowedSchemesAppliedToAttributes: [ 'href', 'src', 'cite' ],
+    allowProtocolRelative: true,
+    enforceHtmlBoundary: false
+}
+
+
+export const metaModels = {};
+
+metaModels['common'] = { load: ['contact', 'location', 'request'] };
+metaModels['personal'] = { load: ['budget', 'imageGallery'] };
+metaModels['users'] = { load: ['permission', 'role', 'user', 'token'], 'require': ['i18n', 'common'] };
+metaModels['i18n'] = { load: ['translation','lang']};
+metaModels['website'] = { load: ['webpage', 'content', 'taxonomy', 'contact', 'resource'], 'require': ['i18n'] };
+metaModels['messaging'] = { load: ['alert','ticket', 'message', 'channel'], 'require': ['i18n'] };
+metaModels['eshopping'] = { load: [
+        'order', 'currency', 'product', 'productVariant', 'discount', 'cart', 'cartItem',
+        'brand', 'return', 'review', 'stock', 'returnItem', 'userSubscription',
+        'warehouse', 'shipment', 'campaign', 'stockAlert', 'invoice'],
+    'require': ['i18n', 'users', 'messaging'] };
+metaModels['workflow'] = { load: ['env', 'workflow', 'workflowRun', 'workflowAction', 'workflowStep', 'workflowTrigger']};
+metaModels['erp'] = { load: [ 'accountingExercise', 'accountingLineItem', 'accountingEntry', 'employee', 'dashboard', 'kpi'] };
+
+export const profiles = {
+    'personal': ['contact', 'location', 'imageGallery', 'budget', 'currency', 'taxonomy'], // budget,
+    'developer': ['alert','request','webpage', 'content', 'taxonomy', 'resource', 'translation', 'contact', 'location', 'channel', 'lang', 'token', 'message', 'ticket', 'user', 'permission', 'role'],
+    'company': ['alert','request','location', 'campaign', 'order', 'currency', 'product', 'cart', 'cartItem', 'invoice', 'messaging', 'user', 'role', 'permission', 'token','translation', 'lang', 'webpage', 'content', 'taxonomy', 'contact', 'resource', 'accountingExercise', 'accountingLineItem', 'accountingEntry', 'employee', 'kpi', 'dashboard'],
+    'engineer': ['alert','request','dashboard', 'kpi', 'user', 'role', 'token', 'permission', 'workflow', 'workflowRun', 'workflowStep', "channel", "message", 'workflowAction', 'workflowTrigger']
+}
+
+// Dans ConditionBuilder.jsx ou un fichier de constantes partagé
+export const mongoOperators = [
+    { value: '$eq', label: '==', inputType: 'text' }, // Égal (pour string, number, boolean...)
+    { value: '$ne', label: '!=', inputType: 'text' }, // Différent
+    { value: '$gt', label: '>', inputType: 'number' }, // Supérieur (nombre, date)
+    { value: '$gte', label: '>=', inputType: 'number' }, // Supérieur ou égal à
+    { value: '$lt', label: '<', inputType: 'number' }, // Inférieur à
+    { value: '$lte', label: '<=', inputType: 'number' }, // Inférieur ou égal à
+    { value: '$regex', label: 'contient (regex)', inputType: 'text' }, // Correspondance Regex (pour string)
+    { value: '$in', label: 'est dans (liste)', inputType: 'csv' }, // Dans un tableau (valeurs séparées par des virgules)
+    { value: '$nin', label: "n'est pas dans (liste)", inputType: 'csv' }, // Pas dans un tableau
+    { value: '$exists', label: 'existe', inputType: 'boolean' }, // Le champ existe (true/false)
+    // { value: '$not', label: 'NON (condition)', inputType: 'condition' }, // Pourrait encapsuler une autre condition
+    // Ajoutez d'autres opérateurs MongoDB pertinents si besoin ($size, $type, $elemMatch...)
+];
+
+export const allowedFields = ['locked', 'hiddenable', 'anonymized', 'condition', 'color', 'index', 'type', 'required', 'hint', 'default', 'validate', 'unique', 'name', 'placeholder', 'asMain'];
+
+
+
+export const MONGO_OPERATORS = {
+    ADD: { mongo: '$add', label: '+', type: 'numeric', multi: true },
+    SUBTRACT: { mongo: '$subtract', label: '-', type: 'numeric', multi: false },
+    MULTIPLY: { mongo: '$multiply', label: '*', type: 'numeric', multi: true },
+    DIVIDE: { mongo: '$divide', label: '/', type: 'numeric', multi: false },
+    MODULO: { mongo: '$mod', label: '%', type: 'numeric', multi: false },
+    POW: { mongo: '$pow', label: 'pow', type: 'numeric', multi: false },
+    SQRT: { mongo: '$sqrt', label: '√', type: 'numeric', multi: false, unary: true },
+    ABS: { mongo: '$abs', label: 'abs', type: 'numeric', multi: false, unary: true },
+    CEIL: { mongo: '$ceil', label: 'ceil', type: 'numeric', multi: false, unary: true },
+    FLOOR: { mongo: '$floor', label: 'floor', type: 'numeric', multi: false, unary: true },
+    COS: { mongo: '$cos', label: 'cos', type: 'numeric', multi: false, unary: true },
+    ACOS: { mongo: '$acos', label: 'acos', type: 'numeric', multi: false, unary: true },
+    SIN: { mongo: '$sin', label: 'sin', type: 'numeric', multi: false, unary: true },
+    ASIN: { mongo: '$asin', label: 'asin', type: 'numeric', multi: false, unary: true },
+    TAN: { mongo: '$tan', label: 'tan', type: 'numeric', multi: false, unary: true },
+    ATAN: { mongo: '$atan', label: 'atan', type: 'numeric', multi: false, unary: true },
+    LN: { mongo: '$ln', label: 'ln', type: 'numeric', multi: false, unary: true },
+    LOG: { mongo: '$log10', label: 'log', type: 'numeric', multi: false, unary: true },
+
+    CONCAT: { mongo: '$concat', label: 'concat', type: 'string', multi: true },
+
+    YEAR: { mongo: '$year', label: 'year', type: 'date', unary: true },
+    MONTH: { mongo: '$month', label: 'month', type: 'date', unary: true },
+    WEEK: { mongo: '$week', label: 'week', type: 'date', unary: true },
+    DAY_OF_MONTH: { mongo: '$dayOfMonth', label: 'day', type: 'date', unary: true },
+    HOUR: { mongo: '$hour', label: 'hour', type: 'date', unary: true },
+    MINUTE: { mongo: '$minute', label: 'min', type: 'date', unary: true },
+    SECOND: { mongo: '$second', label: 'sec', type: 'date', unary: true },
+};
+
+export const OPERAND_TYPES = {
+    FIELD: 'field',
+    CONSTANT: 'constant',
+    PREVIOUS_STEP: 'previousStep',
+};
+