data-primals-engine 1.7.0 → 1.7.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +160 -160
- package/client/package-lock.json +1080 -212
- package/client/package.json +12 -6
- package/client/src/AssistantChat.jsx +1 -3
- package/client/src/DataLayout.jsx +19 -20
- package/client/src/DataTable.jsx +2 -2
- package/client/src/DocumentationPageLayout.scss +1 -1
- package/client/src/ViewSwitcher.jsx +1 -1
- package/client/vite.config.js +31 -30
- package/package.json +39 -23
- package/src/ai.jobs.js +135 -0
- package/src/constants.js +561 -545
- package/src/core.js +487 -477
- package/src/data.js +2 -0
- package/src/email.js +0 -2
- package/src/engine.js +50 -42
- package/src/filter.js +348 -343
- package/src/modules/assistant/assistant.js +782 -763
- package/src/modules/assistant/constants.js +23 -16
- package/src/modules/assistant/providers.js +77 -37
- package/src/modules/bucket.js +4 -0
- package/src/modules/data/data.cluster.js +191 -0
- package/src/modules/data/data.core.js +11 -8
- package/src/modules/data/data.js +13 -4
- package/src/modules/data/data.operations.js +186 -106
- package/src/modules/data/data.relations.js +1 -0
- package/src/modules/data/data.replication.js +83 -0
- package/src/modules/data/data.routes.js +2183 -1879
- package/src/modules/mongodb.js +76 -73
- package/src/modules/user.js +7 -1
- package/src/modules/worker-script-runner.js +97 -0
- package/src/modules/workflow.js +1953 -1815
- package/src/packs.js +5701 -5697
- package/src/providers.js +298 -297
- package/test/assistant.test.js +207 -206
- package/test/data.integration.test.js +1425 -1416
- package/test/import_export.integration.test.js +210 -210
- package/test/workflow.actions.integration.test.js +487 -475
- package/test/workflow.integration.test.js +332 -329
package/README.md
CHANGED
|
@@ -1,161 +1,161 @@
|
|
|
1
|
-
# data-primals-engine
|
|
2
|
-
[](https://github.com/anonympins/data-primals-engine/actions/workflows/node.js.yml)
|
|
3
|
-

|
|
4
|
-

|
|
5
|
-

|
|
6
|
-

|
|
7
|
-
|
|
8
|
-
**data-primals-engine** is a powerful and flexible **Node.js** backend framework designed to accelerate the development of complex data-driven applications. Built on **Express.js** and **MongoDB**, it provides a solid foundation so you can focus on what makes your application unique.
|
|
9
|
-
|
|
10
|
-
> Whether you're building a CRM, e-commerce site, CMS, or SaaS platform, **data-primals-engine** provides the strong foundations so you can focus on what makes your application unique.
|
|
11
|
-
|
|
12
|
-
---
|
|
13
|
-
|
|
14
|
-
<p align="center">
|
|
15
|
-
<a href="https://data.primals.net/prez1.jpg" target="_blank"><img alt="Light" src="https://data.primals.net/prez1.jpg" width="35%"></a>
|
|
16
|
-
<a href="https://data.primals.net/prez6.jpg" target="_blank"><img alt="Light" src="https://data.primals.net/prez6.jpg" width="35%"></a>
|
|
17
|
-
<a href="https://data.primals.net/api-docs" target="_blank"><img alt="Dark" src="https://data.primals.net/prez5.jpg" width="25%"></a>
|
|
18
|
-
</p>
|
|
19
|
-
---
|
|
20
|
-
|
|
21
|
-
## 🚀 Key Features
|
|
22
|
-
|
|
23
|
-
- **Visual Data Modeling**: Define and update schemas using a powerful UI Model Creator, or directly with JSON. No migrations required.
|
|
24
|
-
- **Custom API Endpoints**: Create server-side logic and new API endpoints directly from the UI in a secure, sandboxed environment.
|
|
25
|
-
- **Automation Workflows**: Trigger complex actions based on data events (create, update, delete) or schedules (cron).
|
|
26
|
-
- **Advanced Querying & Aggregation**: Go beyond simple filters with a visual Condition Builder, deep relation expansion, complex lookups, and dynamic calculated fields.
|
|
27
|
-
- **Rich UI Data Views**: Ready-to-use React components to display your data, including a powerful and configurable Data Table, a Kanban board, and a Calendar view.
|
|
28
|
-
- **Integrated Backup & Restore**: Secure, encrypted user data backups with rotation policies, supporting both local and AWS S3 storage.
|
|
29
|
-
- **Automatic Data Auditing**: Automatically tracks all changes (create, update, delete) for every record, providing a complete version history for auditing and traceability.
|
|
30
|
-
- **Event-Driven & Extensible**: A core event system allows for deep customization and the easy creation of new modules or plugins.
|
|
31
|
-
- **Authentication & Authorization**: Robust role-based access control (RBAC) and pluggable user providers.
|
|
32
|
-
- **Built-in File Management**: Handle file uploads seamlessly with integrated support for AWS S3 storage.
|
|
33
|
-
- **🧠 AI Integration**: Natively supports OpenAI, DeepSeek and Google Gemini models via LangChain for content generation, analysis, and more.
|
|
34
|
-
- **🌐 Internationalization (i18n)**: Fully supports multilingual interfaces and user-specific translated data.
|
|
35
|
-
- **📦 Starter Packs**: Quickly bootstrap applications with pre-built data packs for CRM, e-commerce, and more.
|
|
36
|
-
- **📄Auto-Generated API Documentation**: Interactive API documentation available via the interface or at `/api-docs`.
|
|
37
|
-
|
|
38
|
-
## 🌟 Why Choose data-primals-engine?
|
|
39
|
-
|
|
40
|
-
- **Zero Boilerplate**: Focus on your business logic, not infrastructure
|
|
41
|
-
- **Scalability**: Architecture designed for rapidly growing applications
|
|
42
|
-
- **Modularity**: Enable/disable features as needed
|
|
43
|
-
- **Batteries Included**: Everything you need to get started quickly
|
|
44
|
-
- **Proven Performance**: Handles 50k+ documents efficiently
|
|
45
|
-
- **AI Ready**: Built-in LangChain integration for main providers (OpenAI,Gemini,Anthropic,DeepSeek)
|
|
46
|
-
|
|
47
|
-
---
|
|
48
|
-
|
|
49
|
-
## 📊 Proven Performance
|
|
50
|
-
|
|
51
|
-
The engine has been rigorously tested to ensure stability and scalability. Load tests simulating complex, multi-step user journeys (including model creation, data import, and API interactions) show excellent results:
|
|
52
|
-
- **Zero Failures**: 100% success rate under sustained concurrent load.
|
|
53
|
-
- **Excellent Responsiveness**: Median response times as low as 15-30ms.
|
|
54
|
-
- **Linear Scaling**: Predictable performance as user load increases.
|
|
55
|
-
|
|
56
|
-
For detailed reports, see the Performance Testing Documentation.
|
|
57
|
-
|
|
58
|
-
---
|
|
59
|
-
|
|
60
|
-
## ⚙️ Requirements
|
|
61
|
-
|
|
62
|
-
- Node.js ≥ 20
|
|
63
|
-
- MongoDB (local or remote), see installation guide
|
|
64
|
-
- NPM or Yarn
|
|
65
|
-
|
|
66
|
-
---
|
|
67
|
-
|
|
68
|
-
## ⚡ Quick Start
|
|
69
|
-
|
|
70
|
-
### 1. Prerequisites
|
|
71
|
-
- **Node.js** ≥ 20
|
|
72
|
-
- **MongoDB** (local or remote)
|
|
73
|
-
|
|
74
|
-
### check
|
|
75
|
-
```bash
|
|
76
|
-
# Verify required versions
|
|
77
|
-
node -v # Must show ≥ v20
|
|
78
|
-
mongod --version # Must be installed
|
|
79
|
-
```
|
|
80
|
-
|
|
81
|
-
### 2. Installation
|
|
82
|
-
|
|
83
|
-
```bash
|
|
84
|
-
npm i data-primals-engine
|
|
85
|
-
```
|
|
86
|
-
or
|
|
87
|
-
```bash
|
|
88
|
-
git clone https://github.com/anonympins/data-primals-engine.git
|
|
89
|
-
cd data-primals-engine
|
|
90
|
-
npm install
|
|
91
|
-
```
|
|
92
|
-
|
|
93
|
-
### 3. Configuration
|
|
94
|
-
|
|
95
|
-
Create a `.env` file in the project root to configure the connection to your database :
|
|
96
|
-
|
|
97
|
-
```env
|
|
98
|
-
MONGO_DB_URL=mongodb://127.0.0.1:27017/my_database
|
|
99
|
-
JWT_SECRET=a_long_and_random_secret
|
|
100
|
-
```
|
|
101
|
-
|
|
102
|
-
To discover all available environment variables (AWS, SMTP, AI, etc.), consult the **configuration documentation**.
|
|
103
|
-
|
|
104
|
-
### 4. Launch
|
|
105
|
-
|
|
106
|
-
```bash
|
|
107
|
-
# Development mode with automatic reloading
|
|
108
|
-
npm run devserver
|
|
109
|
-
|
|
110
|
-
# Production mode
|
|
111
|
-
npm run server
|
|
112
|
-
```
|
|
113
|
-
|
|
114
|
-
By default, the application is available at `http://localhost:7633`.
|
|
115
|
-
|
|
116
|
-
---
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
## 🧭 Explore the Platform
|
|
120
|
-
Discover the core features to get started building and managing your data.
|
|
121
|
-
Link to the official documentation: https://data.primals.net/en/documentation/
|
|
122
|
-
|
|
123
|
-
- 🧠 [Core Concepts](https://github.com/anonympins/data-primals-engine/wiki/Concepts): Explore the fundamentals of data modeling
|
|
124
|
-
- 🔌 [Custom API Endpoints](https://github.com/anonympins/data-primals-engine/wiki/Custom-Endpoints): Create dynamic HTTP routes directly from the backend
|
|
125
|
-
- 🏗️ [Data Models](https://data.primals.net/en/documentation/create-models): Structure your information.
|
|
126
|
-
- 🗃️ [Data Management](https://data.primals.net/en/documentation/manage-data): Create, read, update, and delete your entries.
|
|
127
|
-
- 📊 [Views (Table, Kanban)](Views): Visualize your data in different ways. *(Page to be created)*
|
|
128
|
-
- 📈 [Dashboards, KPIs, and Charts](https://data.primals.net/en/documentation/dashboards): Track your key metrics.
|
|
129
|
-
- 👥 [Users](https://data.primals.net/en/documentation/users): Manage access to the platform.
|
|
130
|
-
- 🔐 [Roles and Permissions](https://data.primals.net/en/documentation/roles-permissions): Define who can see and do what.
|
|
131
|
-
- ⚙️ [Automation with Workflows](https://data.primals.net/en/documentation/automation-workflows): Create automated processes.
|
|
132
|
-
|
|
133
|
-
## 🔌 Integrate with your tools
|
|
134
|
-
Connect the platform to your external applications and services via our API.
|
|
135
|
-
|
|
136
|
-
- 📡 [API Requests](https://data.primals.net/en/documentation/api): The basics for interacting with the REST API.
|
|
137
|
-
- 🚀 [Advanced Requests](https://data.primals.net/en/documentation/advanced-requests): Filters, sorts, projections, and more.
|
|
138
|
-
- 📦 [Bulk Operations](https://data.primals.net/en/documentation/bulk-operations): Perform operations on large volumes of data.
|
|
139
|
-
|
|
140
|
-
## 🧩 Extend Capabilities
|
|
141
|
-
Go further by developing your own features and business logic.
|
|
142
|
-
|
|
143
|
-
- ⚡ [Event System](Event-system): React to events in real time. *(Page to create)*
|
|
144
|
-
- 🧠 [Advanced Workflows](Advanced-workflows): Create complex logic with custom scripts. *(Page to create)*
|
|
145
|
-
- 📦 [Modules](Modules): Package and share your developments. *(Page to create)*
|
|
146
|
-
- ❤️ [Contribute to the project (CONTRIBUTING.md)](https://github.com/anonympins/data-primals-engine/blob/main/CONTRIBUTING.md): Join the developer community.
|
|
147
|
-
|
|
148
|
-
## 💻 For Developers
|
|
149
|
-
Essential tools for a seamless development experience.
|
|
150
|
-
|
|
151
|
-
- 📖 [Explore the API with Swagger](https://data.primals.net/api-docs/): Test API access points directly from your browser.
|
|
152
|
-
- 📮 [Postman Collection](https://data.primals.net/doc/API.postman_collection.json): Import our collection to start querying in just a few clicks.
|
|
153
|
-
|
|
154
|
-
## 🤝 Contribution
|
|
155
|
-
|
|
156
|
-
Contributions are welcome! Check out the open issues to get started.
|
|
157
|
-
|
|
158
|
-
---
|
|
159
|
-
|
|
160
|
-
## 📄 License
|
|
1
|
+
# data-primals-engine
|
|
2
|
+
[](https://github.com/anonympins/data-primals-engine/actions/workflows/node.js.yml)
|
|
3
|
+

|
|
4
|
+

|
|
5
|
+

|
|
6
|
+

|
|
7
|
+
|
|
8
|
+
**data-primals-engine** is a powerful and flexible **Node.js** backend framework designed to accelerate the development of complex data-driven applications. Built on **Express.js** and **MongoDB**, it provides a solid foundation so you can focus on what makes your application unique.
|
|
9
|
+
|
|
10
|
+
> Whether you're building a CRM, e-commerce site, CMS, or SaaS platform, **data-primals-engine** provides the strong foundations so you can focus on what makes your application unique.
|
|
11
|
+
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
<p align="center">
|
|
15
|
+
<a href="https://data.primals.net/prez1.jpg" target="_blank"><img alt="Light" src="https://data.primals.net/prez1.jpg" width="35%"></a>
|
|
16
|
+
<a href="https://data.primals.net/prez6.jpg" target="_blank"><img alt="Light" src="https://data.primals.net/prez6.jpg" width="35%"></a>
|
|
17
|
+
<a href="https://data.primals.net/api-docs" target="_blank"><img alt="Dark" src="https://data.primals.net/prez5.jpg" width="25%"></a>
|
|
18
|
+
</p>
|
|
19
|
+
---
|
|
20
|
+
|
|
21
|
+
## 🚀 Key Features
|
|
22
|
+
|
|
23
|
+
- **Visual Data Modeling**: Define and update schemas using a powerful UI Model Creator, or directly with JSON. No migrations required.
|
|
24
|
+
- **Custom API Endpoints**: Create server-side logic and new API endpoints directly from the UI in a secure, sandboxed environment.
|
|
25
|
+
- **Automation Workflows**: Trigger complex actions based on data events (create, update, delete) or schedules (cron).
|
|
26
|
+
- **Advanced Querying & Aggregation**: Go beyond simple filters with a visual Condition Builder, deep relation expansion, complex lookups, and dynamic calculated fields.
|
|
27
|
+
- **Rich UI Data Views**: Ready-to-use React components to display your data, including a powerful and configurable Data Table, a Kanban board, and a Calendar view.
|
|
28
|
+
- **Integrated Backup & Restore**: Secure, encrypted user data backups with rotation policies, supporting both local and AWS S3 storage.
|
|
29
|
+
- **Automatic Data Auditing**: Automatically tracks all changes (create, update, delete) for every record, providing a complete version history for auditing and traceability.
|
|
30
|
+
- **Event-Driven & Extensible**: A core event system allows for deep customization and the easy creation of new modules or plugins.
|
|
31
|
+
- **Authentication & Authorization**: Robust role-based access control (RBAC) and pluggable user providers.
|
|
32
|
+
- **Built-in File Management**: Handle file uploads seamlessly with integrated support for AWS S3 storage.
|
|
33
|
+
- **🧠 AI Integration**: Natively supports OpenAI, DeepSeek and Google Gemini models via LangChain for content generation, analysis, and more.
|
|
34
|
+
- **🌐 Internationalization (i18n)**: Fully supports multilingual interfaces and user-specific translated data.
|
|
35
|
+
- **📦 Starter Packs**: Quickly bootstrap applications with pre-built data packs for CRM, e-commerce, and more.
|
|
36
|
+
- **📄Auto-Generated API Documentation**: Interactive API documentation available via the interface or at `/api-docs`.
|
|
37
|
+
|
|
38
|
+
## 🌟 Why Choose data-primals-engine?
|
|
39
|
+
|
|
40
|
+
- **Zero Boilerplate**: Focus on your business logic, not infrastructure
|
|
41
|
+
- **Scalability**: Architecture designed for rapidly growing applications
|
|
42
|
+
- **Modularity**: Enable/disable features as needed
|
|
43
|
+
- **Batteries Included**: Everything you need to get started quickly
|
|
44
|
+
- **Proven Performance**: Handles 50k+ documents efficiently
|
|
45
|
+
- **AI Ready**: Built-in LangChain integration for main providers (OpenAI,Gemini,Anthropic,DeepSeek)
|
|
46
|
+
|
|
47
|
+
---
|
|
48
|
+
|
|
49
|
+
## 📊 Proven Performance
|
|
50
|
+
|
|
51
|
+
The engine has been rigorously tested to ensure stability and scalability. Load tests simulating complex, multi-step user journeys (including model creation, data import, and API interactions) show excellent results:
|
|
52
|
+
- **Zero Failures**: 100% success rate under sustained concurrent load.
|
|
53
|
+
- **Excellent Responsiveness**: Median response times as low as 15-30ms.
|
|
54
|
+
- **Linear Scaling**: Predictable performance as user load increases.
|
|
55
|
+
|
|
56
|
+
For detailed reports, see the Performance Testing Documentation.
|
|
57
|
+
|
|
58
|
+
---
|
|
59
|
+
|
|
60
|
+
## ⚙️ Requirements
|
|
61
|
+
|
|
62
|
+
- Node.js ≥ 20
|
|
63
|
+
- MongoDB (local or remote), see installation guide
|
|
64
|
+
- NPM or Yarn
|
|
65
|
+
|
|
66
|
+
---
|
|
67
|
+
|
|
68
|
+
## ⚡ Quick Start
|
|
69
|
+
|
|
70
|
+
### 1. Prerequisites
|
|
71
|
+
- **Node.js** ≥ 20
|
|
72
|
+
- **MongoDB** (local or remote)
|
|
73
|
+
|
|
74
|
+
### check
|
|
75
|
+
```bash
|
|
76
|
+
# Verify required versions
|
|
77
|
+
node -v # Must show ≥ v20
|
|
78
|
+
mongod --version # Must be installed
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
### 2. Installation
|
|
82
|
+
|
|
83
|
+
```bash
|
|
84
|
+
npm i data-primals-engine
|
|
85
|
+
```
|
|
86
|
+
or
|
|
87
|
+
```bash
|
|
88
|
+
git clone https://github.com/anonympins/data-primals-engine.git
|
|
89
|
+
cd data-primals-engine
|
|
90
|
+
npm install
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
### 3. Configuration
|
|
94
|
+
|
|
95
|
+
Create a `.env` file in the project root to configure the connection to your database :
|
|
96
|
+
|
|
97
|
+
```env
|
|
98
|
+
MONGO_DB_URL=mongodb://127.0.0.1:27017/my_database
|
|
99
|
+
JWT_SECRET=a_long_and_random_secret
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
To discover all available environment variables (AWS, SMTP, AI, etc.), consult the **configuration documentation**.
|
|
103
|
+
|
|
104
|
+
### 4. Launch
|
|
105
|
+
|
|
106
|
+
```bash
|
|
107
|
+
# Development mode with automatic reloading
|
|
108
|
+
npm run devserver
|
|
109
|
+
|
|
110
|
+
# Production mode
|
|
111
|
+
npm run server
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
By default, the application is available at `http://localhost:7633`.
|
|
115
|
+
|
|
116
|
+
---
|
|
117
|
+
|
|
118
|
+
|
|
119
|
+
## 🧭 Explore the Platform
|
|
120
|
+
Discover the core features to get started building and managing your data.
|
|
121
|
+
Link to the official documentation: https://data.primals.net/en/documentation/
|
|
122
|
+
|
|
123
|
+
- 🧠 [Core Concepts](https://github.com/anonympins/data-primals-engine/wiki/Concepts): Explore the fundamentals of data modeling
|
|
124
|
+
- 🔌 [Custom API Endpoints](https://github.com/anonympins/data-primals-engine/wiki/Custom-Endpoints): Create dynamic HTTP routes directly from the backend
|
|
125
|
+
- 🏗️ [Data Models](https://data.primals.net/en/documentation/create-models): Structure your information.
|
|
126
|
+
- 🗃️ [Data Management](https://data.primals.net/en/documentation/manage-data): Create, read, update, and delete your entries.
|
|
127
|
+
- 📊 [Views (Table, Kanban)](Views): Visualize your data in different ways. *(Page to be created)*
|
|
128
|
+
- 📈 [Dashboards, KPIs, and Charts](https://data.primals.net/en/documentation/dashboards): Track your key metrics.
|
|
129
|
+
- 👥 [Users](https://data.primals.net/en/documentation/users): Manage access to the platform.
|
|
130
|
+
- 🔐 [Roles and Permissions](https://data.primals.net/en/documentation/roles-permissions): Define who can see and do what.
|
|
131
|
+
- ⚙️ [Automation with Workflows](https://data.primals.net/en/documentation/automation-workflows): Create automated processes.
|
|
132
|
+
|
|
133
|
+
## 🔌 Integrate with your tools
|
|
134
|
+
Connect the platform to your external applications and services via our API.
|
|
135
|
+
|
|
136
|
+
- 📡 [API Requests](https://data.primals.net/en/documentation/api): The basics for interacting with the REST API.
|
|
137
|
+
- 🚀 [Advanced Requests](https://data.primals.net/en/documentation/advanced-requests): Filters, sorts, projections, and more.
|
|
138
|
+
- 📦 [Bulk Operations](https://data.primals.net/en/documentation/bulk-operations): Perform operations on large volumes of data.
|
|
139
|
+
|
|
140
|
+
## 🧩 Extend Capabilities
|
|
141
|
+
Go further by developing your own features and business logic.
|
|
142
|
+
|
|
143
|
+
- ⚡ [Event System](Event-system): React to events in real time. *(Page to create)*
|
|
144
|
+
- 🧠 [Advanced Workflows](Advanced-workflows): Create complex logic with custom scripts. *(Page to create)*
|
|
145
|
+
- 📦 [Modules](Modules): Package and share your developments. *(Page to create)*
|
|
146
|
+
- ❤️ [Contribute to the project (CONTRIBUTING.md)](https://github.com/anonympins/data-primals-engine/blob/main/CONTRIBUTING.md): Join the developer community.
|
|
147
|
+
|
|
148
|
+
## 💻 For Developers
|
|
149
|
+
Essential tools for a seamless development experience.
|
|
150
|
+
|
|
151
|
+
- 📖 [Explore the API with Swagger](https://data.primals.net/api-docs/): Test API access points directly from your browser.
|
|
152
|
+
- 📮 [Postman Collection](https://data.primals.net/doc/API.postman_collection.json): Import our collection to start querying in just a few clicks.
|
|
153
|
+
|
|
154
|
+
## 🤝 Contribution
|
|
155
|
+
|
|
156
|
+
Contributions are welcome! Check out the open issues to get started.
|
|
157
|
+
|
|
158
|
+
---
|
|
159
|
+
|
|
160
|
+
## 📄 License
|
|
161
161
|
Distributed under the **MIT License**. See the `LICENSE` file.
|