data-primals-engine 1.7.1 → 1.7.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (36) hide show
  1. package/README.md +160 -160
  2. package/client/package-lock.json +484 -175
  3. package/client/package.json +7 -4
  4. package/client/src/AssistantChat.jsx +1 -3
  5. package/client/src/DataLayout.jsx +19 -20
  6. package/client/src/DataTable.jsx +2 -2
  7. package/client/src/DocumentationPageLayout.scss +1 -1
  8. package/client/src/ViewSwitcher.jsx +1 -1
  9. package/client/vite.config.js +31 -30
  10. package/package.json +27 -17
  11. package/src/ai.jobs.js +135 -0
  12. package/src/constants.js +561 -545
  13. package/src/data.js +2 -0
  14. package/src/engine.js +50 -42
  15. package/src/modules/assistant/assistant.js +782 -763
  16. package/src/modules/assistant/constants.js +23 -16
  17. package/src/modules/assistant/providers.js +77 -37
  18. package/src/modules/bucket.js +4 -0
  19. package/src/modules/data/data.cluster.js +191 -0
  20. package/src/modules/data/data.core.js +11 -8
  21. package/src/modules/data/data.js +311 -311
  22. package/src/modules/data/data.operations.js +186 -106
  23. package/src/modules/data/data.relations.js +1 -0
  24. package/src/modules/data/data.replication.js +83 -0
  25. package/src/modules/data/data.routes.js +2183 -1879
  26. package/src/modules/mongodb.js +76 -73
  27. package/src/modules/user.js +7 -1
  28. package/src/modules/worker-script-runner.js +97 -0
  29. package/src/modules/workflow.js +177 -52
  30. package/src/packs.js +5701 -5701
  31. package/src/providers.js +298 -297
  32. package/test/assistant.test.js +207 -206
  33. package/test/data.integration.test.js +1425 -1416
  34. package/test/import_export.integration.test.js +210 -210
  35. package/test/workflow.actions.integration.test.js +487 -475
  36. package/test/workflow.integration.test.js +332 -329
package/README.md CHANGED
@@ -1,161 +1,161 @@
1
- # data-primals-engine
2
- [![Node.js CI](https://github.com/anonympins/data-primals-engine/actions/workflows/node.js.yml/badge.svg?branch=main)](https://github.com/anonympins/data-primals-engine/actions/workflows/node.js.yml)
3
- ![](https://img.shields.io/npm/dw/data-primals-engine)
4
- ![](https://img.shields.io/npm/last-update/data-primals-engine)
5
- ![](https://img.shields.io/github/v/release/anonympins/data-primals-engine)
6
- ![](https://img.shields.io/github/license/anonympins/data-primals-engine)
7
-
8
- **data-primals-engine** is a powerful and flexible **Node.js** backend framework designed to accelerate the development of complex data-driven applications. Built on **Express.js** and **MongoDB**, it provides a solid foundation so you can focus on what makes your application unique.
9
-
10
- > Whether you're building a CRM, e-commerce site, CMS, or SaaS platform, **data-primals-engine** provides the strong foundations so you can focus on what makes your application unique.
11
-
12
- ---
13
-
14
- <p align="center">
15
- <a href="https://data.primals.net/prez1.jpg" target="_blank"><img alt="Light" src="https://data.primals.net/prez1.jpg" width="35%"></a>
16
- <a href="https://data.primals.net/prez6.jpg" target="_blank"><img alt="Light" src="https://data.primals.net/prez6.jpg" width="35%"></a>
17
- <a href="https://data.primals.net/api-docs" target="_blank"><img alt="Dark" src="https://data.primals.net/prez5.jpg" width="25%"></a>
18
- </p>
19
- ---
20
-
21
- ## 🚀 Key Features
22
-
23
- - **Visual Data Modeling**: Define and update schemas using a powerful UI Model Creator, or directly with JSON. No migrations required.
24
- - **Custom API Endpoints**: Create server-side logic and new API endpoints directly from the UI in a secure, sandboxed environment.
25
- - **Automation Workflows**: Trigger complex actions based on data events (create, update, delete) or schedules (cron).
26
- - **Advanced Querying & Aggregation**: Go beyond simple filters with a visual Condition Builder, deep relation expansion, complex lookups, and dynamic calculated fields.
27
- - **Rich UI Data Views**: Ready-to-use React components to display your data, including a powerful and configurable Data Table, a Kanban board, and a Calendar view.
28
- - **Integrated Backup & Restore**: Secure, encrypted user data backups with rotation policies, supporting both local and AWS S3 storage.
29
- - **Automatic Data Auditing**: Automatically tracks all changes (create, update, delete) for every record, providing a complete version history for auditing and traceability.
30
- - **Event-Driven & Extensible**: A core event system allows for deep customization and the easy creation of new modules or plugins.
31
- - **Authentication & Authorization**: Robust role-based access control (RBAC) and pluggable user providers.
32
- - **Built-in File Management**: Handle file uploads seamlessly with integrated support for AWS S3 storage.
33
- - **🧠 AI Integration**: Natively supports OpenAI, DeepSeek and Google Gemini models via LangChain for content generation, analysis, and more.
34
- - **🌐 Internationalization (i18n)**: Fully supports multilingual interfaces and user-specific translated data.
35
- - **📦 Starter Packs**: Quickly bootstrap applications with pre-built data packs for CRM, e-commerce, and more.
36
- - **📄Auto-Generated API Documentation**: Interactive API documentation available via the interface or at `/api-docs`.
37
-
38
- ## 🌟 Why Choose data-primals-engine?
39
-
40
- - **Zero Boilerplate**: Focus on your business logic, not infrastructure
41
- - **Scalability**: Architecture designed for rapidly growing applications
42
- - **Modularity**: Enable/disable features as needed
43
- - **Batteries Included**: Everything you need to get started quickly
44
- - **Proven Performance**: Handles 50k+ documents efficiently
45
- - **AI Ready**: Built-in LangChain integration for main providers (OpenAI,Gemini,Anthropic,DeepSeek)
46
-
47
- ---
48
-
49
- ## 📊 Proven Performance
50
-
51
- The engine has been rigorously tested to ensure stability and scalability. Load tests simulating complex, multi-step user journeys (including model creation, data import, and API interactions) show excellent results:
52
- - **Zero Failures**: 100% success rate under sustained concurrent load.
53
- - **Excellent Responsiveness**: Median response times as low as 15-30ms.
54
- - **Linear Scaling**: Predictable performance as user load increases.
55
-
56
- For detailed reports, see the Performance Testing Documentation.
57
-
58
- ---
59
-
60
- ## ⚙️ Requirements
61
-
62
- - Node.js ≥ 20
63
- - MongoDB (local or remote), see installation guide
64
- - NPM or Yarn
65
-
66
- ---
67
-
68
- ## ⚡ Quick Start
69
-
70
- ### 1. Prerequisites
71
- - **Node.js** ≥ 20
72
- - **MongoDB** (local or remote)
73
-
74
- ### check
75
- ```bash
76
- # Verify required versions
77
- node -v # Must show ≥ v20
78
- mongod --version # Must be installed
79
- ```
80
-
81
- ### 2. Installation
82
-
83
- ```bash
84
- npm i data-primals-engine
85
- ```
86
- or
87
- ```bash
88
- git clone https://github.com/anonympins/data-primals-engine.git
89
- cd data-primals-engine
90
- npm install
91
- ```
92
-
93
- ### 3. Configuration
94
-
95
- Create a `.env` file in the project root to configure the connection to your database :
96
-
97
- ```env
98
- MONGO_DB_URL=mongodb://127.0.0.1:27017/my_database
99
- JWT_SECRET=a_long_and_random_secret
100
- ```
101
-
102
- To discover all available environment variables (AWS, SMTP, AI, etc.), consult the **configuration documentation**.
103
-
104
- ### 4. Launch
105
-
106
- ```bash
107
- # Development mode with automatic reloading
108
- npm run devserver
109
-
110
- # Production mode
111
- npm run server
112
- ```
113
-
114
- By default, the application is available at `http://localhost:7633`.
115
-
116
- ---
117
-
118
-
119
- ## 🧭 Explore the Platform
120
- Discover the core features to get started building and managing your data.
121
- Link to the official documentation: https://data.primals.net/en/documentation/
122
-
123
- - 🧠 [Core Concepts](https://github.com/anonympins/data-primals-engine/wiki/Concepts): Explore the fundamentals of data modeling
124
- - 🔌 [Custom API Endpoints](https://github.com/anonympins/data-primals-engine/wiki/Custom-Endpoints): Create dynamic HTTP routes directly from the backend
125
- - 🏗️ [Data Models](https://data.primals.net/en/documentation/create-models): Structure your information.
126
- - 🗃️ [Data Management](https://data.primals.net/en/documentation/manage-data): Create, read, update, and delete your entries.
127
- - 📊 [Views (Table, Kanban)](Views): Visualize your data in different ways. *(Page to be created)*
128
- - 📈 [Dashboards, KPIs, and Charts](https://data.primals.net/en/documentation/dashboards): Track your key metrics.
129
- - 👥 [Users](https://data.primals.net/en/documentation/users): Manage access to the platform.
130
- - 🔐 [Roles and Permissions](https://data.primals.net/en/documentation/roles-permissions): Define who can see and do what.
131
- - ⚙️ [Automation with Workflows](https://data.primals.net/en/documentation/automation-workflows): Create automated processes.
132
-
133
- ## 🔌 Integrate with your tools
134
- Connect the platform to your external applications and services via our API.
135
-
136
- - 📡 [API Requests](https://data.primals.net/en/documentation/api): The basics for interacting with the REST API.
137
- - 🚀 [Advanced Requests](https://data.primals.net/en/documentation/advanced-requests): Filters, sorts, projections, and more.
138
- - 📦 [Bulk Operations](https://data.primals.net/en/documentation/bulk-operations): Perform operations on large volumes of data.
139
-
140
- ## 🧩 Extend Capabilities
141
- Go further by developing your own features and business logic.
142
-
143
- - ⚡ [Event System](Event-system): React to events in real time. *(Page to create)*
144
- - 🧠 [Advanced Workflows](Advanced-workflows): Create complex logic with custom scripts. *(Page to create)*
145
- - 📦 [Modules](Modules): Package and share your developments. *(Page to create)*
146
- - ❤️ [Contribute to the project (CONTRIBUTING.md)](https://github.com/anonympins/data-primals-engine/blob/main/CONTRIBUTING.md): Join the developer community.
147
-
148
- ## 💻 For Developers
149
- Essential tools for a seamless development experience.
150
-
151
- - 📖 [Explore the API with Swagger](https://data.primals.net/api-docs/): Test API access points directly from your browser.
152
- - 📮 [Postman Collection](https://data.primals.net/doc/API.postman_collection.json): Import our collection to start querying in just a few clicks.
153
-
154
- ## 🤝 Contribution
155
-
156
- Contributions are welcome! Check out the open issues to get started.
157
-
158
- ---
159
-
160
- ## 📄 License
1
+ # data-primals-engine
2
+ [![Node.js CI](https://github.com/anonympins/data-primals-engine/actions/workflows/node.js.yml/badge.svg?branch=main)](https://github.com/anonympins/data-primals-engine/actions/workflows/node.js.yml)
3
+ ![](https://img.shields.io/npm/dw/data-primals-engine)
4
+ ![](https://img.shields.io/npm/last-update/data-primals-engine)
5
+ ![](https://img.shields.io/github/v/release/anonympins/data-primals-engine)
6
+ ![](https://img.shields.io/github/license/anonympins/data-primals-engine)
7
+
8
+ **data-primals-engine** is a powerful and flexible **Node.js** backend framework designed to accelerate the development of complex data-driven applications. Built on **Express.js** and **MongoDB**, it provides a solid foundation so you can focus on what makes your application unique.
9
+
10
+ > Whether you're building a CRM, e-commerce site, CMS, or SaaS platform, **data-primals-engine** provides the strong foundations so you can focus on what makes your application unique.
11
+
12
+ ---
13
+
14
+ <p align="center">
15
+ <a href="https://data.primals.net/prez1.jpg" target="_blank"><img alt="Light" src="https://data.primals.net/prez1.jpg" width="35%"></a>
16
+ <a href="https://data.primals.net/prez6.jpg" target="_blank"><img alt="Light" src="https://data.primals.net/prez6.jpg" width="35%"></a>
17
+ <a href="https://data.primals.net/api-docs" target="_blank"><img alt="Dark" src="https://data.primals.net/prez5.jpg" width="25%"></a>
18
+ </p>
19
+ ---
20
+
21
+ ## 🚀 Key Features
22
+
23
+ - **Visual Data Modeling**: Define and update schemas using a powerful UI Model Creator, or directly with JSON. No migrations required.
24
+ - **Custom API Endpoints**: Create server-side logic and new API endpoints directly from the UI in a secure, sandboxed environment.
25
+ - **Automation Workflows**: Trigger complex actions based on data events (create, update, delete) or schedules (cron).
26
+ - **Advanced Querying & Aggregation**: Go beyond simple filters with a visual Condition Builder, deep relation expansion, complex lookups, and dynamic calculated fields.
27
+ - **Rich UI Data Views**: Ready-to-use React components to display your data, including a powerful and configurable Data Table, a Kanban board, and a Calendar view.
28
+ - **Integrated Backup & Restore**: Secure, encrypted user data backups with rotation policies, supporting both local and AWS S3 storage.
29
+ - **Automatic Data Auditing**: Automatically tracks all changes (create, update, delete) for every record, providing a complete version history for auditing and traceability.
30
+ - **Event-Driven & Extensible**: A core event system allows for deep customization and the easy creation of new modules or plugins.
31
+ - **Authentication & Authorization**: Robust role-based access control (RBAC) and pluggable user providers.
32
+ - **Built-in File Management**: Handle file uploads seamlessly with integrated support for AWS S3 storage.
33
+ - **🧠 AI Integration**: Natively supports OpenAI, DeepSeek and Google Gemini models via LangChain for content generation, analysis, and more.
34
+ - **🌐 Internationalization (i18n)**: Fully supports multilingual interfaces and user-specific translated data.
35
+ - **📦 Starter Packs**: Quickly bootstrap applications with pre-built data packs for CRM, e-commerce, and more.
36
+ - **📄Auto-Generated API Documentation**: Interactive API documentation available via the interface or at `/api-docs`.
37
+
38
+ ## 🌟 Why Choose data-primals-engine?
39
+
40
+ - **Zero Boilerplate**: Focus on your business logic, not infrastructure
41
+ - **Scalability**: Architecture designed for rapidly growing applications
42
+ - **Modularity**: Enable/disable features as needed
43
+ - **Batteries Included**: Everything you need to get started quickly
44
+ - **Proven Performance**: Handles 50k+ documents efficiently
45
+ - **AI Ready**: Built-in LangChain integration for main providers (OpenAI,Gemini,Anthropic,DeepSeek)
46
+
47
+ ---
48
+
49
+ ## 📊 Proven Performance
50
+
51
+ The engine has been rigorously tested to ensure stability and scalability. Load tests simulating complex, multi-step user journeys (including model creation, data import, and API interactions) show excellent results:
52
+ - **Zero Failures**: 100% success rate under sustained concurrent load.
53
+ - **Excellent Responsiveness**: Median response times as low as 15-30ms.
54
+ - **Linear Scaling**: Predictable performance as user load increases.
55
+
56
+ For detailed reports, see the Performance Testing Documentation.
57
+
58
+ ---
59
+
60
+ ## ⚙️ Requirements
61
+
62
+ - Node.js ≥ 20
63
+ - MongoDB (local or remote), see installation guide
64
+ - NPM or Yarn
65
+
66
+ ---
67
+
68
+ ## ⚡ Quick Start
69
+
70
+ ### 1. Prerequisites
71
+ - **Node.js** ≥ 20
72
+ - **MongoDB** (local or remote)
73
+
74
+ ### check
75
+ ```bash
76
+ # Verify required versions
77
+ node -v # Must show ≥ v20
78
+ mongod --version # Must be installed
79
+ ```
80
+
81
+ ### 2. Installation
82
+
83
+ ```bash
84
+ npm i data-primals-engine
85
+ ```
86
+ or
87
+ ```bash
88
+ git clone https://github.com/anonympins/data-primals-engine.git
89
+ cd data-primals-engine
90
+ npm install
91
+ ```
92
+
93
+ ### 3. Configuration
94
+
95
+ Create a `.env` file in the project root to configure the connection to your database :
96
+
97
+ ```env
98
+ MONGO_DB_URL=mongodb://127.0.0.1:27017/my_database
99
+ JWT_SECRET=a_long_and_random_secret
100
+ ```
101
+
102
+ To discover all available environment variables (AWS, SMTP, AI, etc.), consult the **configuration documentation**.
103
+
104
+ ### 4. Launch
105
+
106
+ ```bash
107
+ # Development mode with automatic reloading
108
+ npm run devserver
109
+
110
+ # Production mode
111
+ npm run server
112
+ ```
113
+
114
+ By default, the application is available at `http://localhost:7633`.
115
+
116
+ ---
117
+
118
+
119
+ ## 🧭 Explore the Platform
120
+ Discover the core features to get started building and managing your data.
121
+ Link to the official documentation: https://data.primals.net/en/documentation/
122
+
123
+ - 🧠 [Core Concepts](https://github.com/anonympins/data-primals-engine/wiki/Concepts): Explore the fundamentals of data modeling
124
+ - 🔌 [Custom API Endpoints](https://github.com/anonympins/data-primals-engine/wiki/Custom-Endpoints): Create dynamic HTTP routes directly from the backend
125
+ - 🏗️ [Data Models](https://data.primals.net/en/documentation/create-models): Structure your information.
126
+ - 🗃️ [Data Management](https://data.primals.net/en/documentation/manage-data): Create, read, update, and delete your entries.
127
+ - 📊 [Views (Table, Kanban)](Views): Visualize your data in different ways. *(Page to be created)*
128
+ - 📈 [Dashboards, KPIs, and Charts](https://data.primals.net/en/documentation/dashboards): Track your key metrics.
129
+ - 👥 [Users](https://data.primals.net/en/documentation/users): Manage access to the platform.
130
+ - 🔐 [Roles and Permissions](https://data.primals.net/en/documentation/roles-permissions): Define who can see and do what.
131
+ - ⚙️ [Automation with Workflows](https://data.primals.net/en/documentation/automation-workflows): Create automated processes.
132
+
133
+ ## 🔌 Integrate with your tools
134
+ Connect the platform to your external applications and services via our API.
135
+
136
+ - 📡 [API Requests](https://data.primals.net/en/documentation/api): The basics for interacting with the REST API.
137
+ - 🚀 [Advanced Requests](https://data.primals.net/en/documentation/advanced-requests): Filters, sorts, projections, and more.
138
+ - 📦 [Bulk Operations](https://data.primals.net/en/documentation/bulk-operations): Perform operations on large volumes of data.
139
+
140
+ ## 🧩 Extend Capabilities
141
+ Go further by developing your own features and business logic.
142
+
143
+ - ⚡ [Event System](Event-system): React to events in real time. *(Page to create)*
144
+ - 🧠 [Advanced Workflows](Advanced-workflows): Create complex logic with custom scripts. *(Page to create)*
145
+ - 📦 [Modules](Modules): Package and share your developments. *(Page to create)*
146
+ - ❤️ [Contribute to the project (CONTRIBUTING.md)](https://github.com/anonympins/data-primals-engine/blob/main/CONTRIBUTING.md): Join the developer community.
147
+
148
+ ## 💻 For Developers
149
+ Essential tools for a seamless development experience.
150
+
151
+ - 📖 [Explore the API with Swagger](https://data.primals.net/api-docs/): Test API access points directly from your browser.
152
+ - 📮 [Postman Collection](https://data.primals.net/doc/API.postman_collection.json): Import our collection to start querying in just a few clicks.
153
+
154
+ ## 🤝 Contribution
155
+
156
+ Contributions are welcome! Check out the open issues to get started.
157
+
158
+ ---
159
+
160
+ ## 📄 License
161
161
  Distributed under the **MIT License**. See the `LICENSE` file.