data-primals-engine 1.1.7 → 1.1.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +96 -14
- package/client/src/RelationField.jsx +59 -47
- package/client/src/translations.js +12 -11
- package/package.json +9 -10
- package/src/core.js +10 -5
- package/src/engine.js +27 -9
- package/src/middlewares/middleware-mongodb.js +1 -1
- package/src/modules/data.js +108 -83
- package/test/data.backup.integration.test.js +3 -1
- package/test/data.integration.test.js +102 -3
package/README.md
CHANGED
|
@@ -1,4 +1,9 @@
|
|
|
1
1
|
# Data Primals Engine
|
|
2
|
+
[](https://github.com/anonympins/data-primals-engine/actions/workflows/node.js.yml)
|
|
3
|
+

|
|
4
|
+

|
|
5
|
+

|
|
6
|
+

|
|
2
7
|
|
|
3
8
|
**data-primals-engine** is a powerful and flexible **Node.js** backend framework designed to accelerate development of complex data-driven applications. Built on **Express.js** and **MongoDB**, it offers dynamic data modeling, automation workflows, advanced user management, and more.
|
|
4
9
|
|
|
@@ -35,7 +40,7 @@ npm i data-primals-engine
|
|
|
35
40
|
```
|
|
36
41
|
or
|
|
37
42
|
```bash
|
|
38
|
-
git clone https://
|
|
43
|
+
git clone https://github.com/anonympins/data-primals-engine.git
|
|
39
44
|
cd data-primals-engine
|
|
40
45
|
npm install
|
|
41
46
|
```
|
|
@@ -78,7 +83,7 @@ By default, the app runs on port **7633**.
|
|
|
78
83
|
|
|
79
84
|
## 🧠 Concepts
|
|
80
85
|
|
|
81
|
-
###
|
|
86
|
+
### Models
|
|
82
87
|
Define schemas using JSON:
|
|
83
88
|
```json
|
|
84
89
|
{
|
|
@@ -115,23 +120,15 @@ Define schemas using JSON:
|
|
|
115
120
|
| model | Stores a model by name | – |
|
|
116
121
|
| modelField | Stores a model field path | – |
|
|
117
122
|
|
|
118
|
-
###
|
|
123
|
+
### Modules
|
|
119
124
|
Activatable features:
|
|
120
125
|
- `mongodb`, `data`, `user`, `workflow`, `file`, `assistant`, `swagger`
|
|
121
126
|
|
|
122
|
-
###
|
|
127
|
+
### Starter Packs
|
|
123
128
|
- **E-commerce**: Products, orders, KPIs
|
|
124
129
|
- **CRM**: Contacts, leads, interactions
|
|
125
130
|
- **Website/blog**: Pages, posts, i18n
|
|
126
131
|
|
|
127
|
-
### 4. Workflows
|
|
128
|
-
Automate business processes:
|
|
129
|
-
- **Triggers**: `DataAdded`, `DataUpdated`, `Scheduled`, `Manual`
|
|
130
|
-
- **Actions**: `CreateData`, `UpdateData`, `SendEmail`, `ApiCall`
|
|
131
|
-
|
|
132
|
-
Example:
|
|
133
|
-
> When a new order is created, email the customer, update stock, and notify logistics.
|
|
134
|
-
|
|
135
132
|
---
|
|
136
133
|
|
|
137
134
|
## 🔌 API Examples (using `curl`)
|
|
@@ -440,7 +437,7 @@ data-primals-engine/
|
|
|
440
437
|
│ ├── engine.js // The Express engine that serves the API
|
|
441
438
|
│ ├── constants.js // The inner-application constants definitions
|
|
442
439
|
│ ├── packs.js // The packs that will be loaded and available with installPack() method
|
|
443
|
-
│ ├── defaultModels.js // The default models available
|
|
440
|
+
│ ├── defaultModels.js // The default models available at startup.
|
|
444
441
|
│ ├── ...
|
|
445
442
|
└── server.js
|
|
446
443
|
```
|
|
@@ -473,8 +470,93 @@ A workflow is composed of two main parts: **Triggers** and **Actions**.
|
|
|
473
470
|
|
|
474
471
|
See the details of the workflow models for more details.
|
|
475
472
|
|
|
473
|
+
## ⚡ Dynamic API Endpoints
|
|
474
|
+
Beyond standard CRUD operations, data-primals-engine allows you to create your own custom API endpoints directly from the UI. This feature acts like a built-in serverless function environment, enabling you to write custom business logic, integrate with third-party services, or create complex data aggregations on the fly.
|
|
475
|
+
|
|
476
|
+
Your code is executed in a secure, isolated sandbox, with access to the core data functions and the incoming request context.
|
|
477
|
+
|
|
478
|
+
### How It Works
|
|
479
|
+
Define an Endpoint: You create a new document in the endpoint model.
|
|
480
|
+
Write Your Logic: In the code field, you write the JavaScript that will be executed.
|
|
481
|
+
Activate: The engine automatically listens for requests on /api/actions/:path that match your endpoint's definition.
|
|
482
|
+
|
|
483
|
+
### The endpoint Model
|
|
484
|
+
To create a custom endpoint, you need to define a document with the following structure:
|
|
485
|
+
```json
|
|
486
|
+
{
|
|
487
|
+
"name": "GetContactPostCount",
|
|
488
|
+
"path": "postCount/:name",
|
|
489
|
+
"method": "GET",
|
|
490
|
+
"code": "const posts = await db.find('content', { author: { $find: { $eq: ['$lastName', request.params.name]}}}); return { postCount: posts.length };",
|
|
491
|
+
"isActive": true
|
|
492
|
+
}
|
|
493
|
+
```
|
|
494
|
+
| Field | Type | Description |
|
|
495
|
+
|:---------|:--------|:---------------------------------------------------------------------|
|
|
496
|
+
| name | string | A descriptive name for your endpoint (e.g., "Calculate User Stats"). |
|
|
497
|
+
| path | string | The URL path. It can include parameters like :id. |
|
|
498
|
+
| method | enum | The HTTP method: GET, POST, PUT, PATCH, or DELETE. |
|
|
499
|
+
| code | code | The JavaScript code to execute when the endpoint is called. |
|
|
500
|
+
| isActive | boolean | A flag to enable or disable the endpoint without deleting it. |
|
|
476
501
|
---
|
|
477
502
|
|
|
503
|
+
### The Execution Context
|
|
504
|
+
Your JavaScript code runs in an async context and has access to several global objects that are securely injected into the sandbox:
|
|
505
|
+
|
|
506
|
+
#### The context Object
|
|
507
|
+
> This object contains all the information about the incoming HTTP request.
|
|
508
|
+
- context.request.**body**: The parsed request body (for POST, PUT, PATCH).
|
|
509
|
+
- context.request.**query**: The URL query parameters as an object.
|
|
510
|
+
- context.request.**params**: The URL path parameters (e.g., username from /user-summary/:username).
|
|
511
|
+
- context.request.**headers**: The incoming request headers.
|
|
512
|
+
|
|
513
|
+
#### The db Object
|
|
514
|
+
> A secure API to interact with the database. All methods are async and must be awaited. They automatically operate within the authenticated user's permissions.
|
|
515
|
+
- await db.**create**(modelName, dataObject): Inserts a new document.
|
|
516
|
+
- await db.**find**(modelName, filter): Finds multiple documents. Returns an array.
|
|
517
|
+
- await db.**findOne**(modelName, filter): Finds a single document. Returns an object or null.
|
|
518
|
+
- await db.**update**(modelName, filter, updateObject): Partially updates documents matching the filter (similar to a PATCH).
|
|
519
|
+
- await db.**delete**(modelName, filter): Deletes documents matching the filter.
|
|
520
|
+
|
|
521
|
+
#### The logger Object
|
|
522
|
+
> A safe way to log messages from your script. These logs will be collected and can be returned in the API response if an error occurs, which is very useful for debugging.
|
|
523
|
+
- logger.**info**(...args)
|
|
524
|
+
- logger.**warn**(...args)
|
|
525
|
+
- logger.**error**(...args)
|
|
526
|
+
|
|
527
|
+
#### The env Object
|
|
528
|
+
> Provides access to the user-defined variables stored in the env model, not the server's process.env.
|
|
529
|
+
- await env.**get**(variableName): Retrieves a single environment variable's value.
|
|
530
|
+
- await env.**getAll**(): Retrieves all user environment variables as an object.
|
|
531
|
+
|
|
532
|
+
#### Example: Creating a User Summary Endpoint
|
|
533
|
+
Let's create an endpoint that fetches a user's profile and counts how many posts they have published.
|
|
534
|
+
1. Create the Endpoint Document
|
|
535
|
+
Create a new document in the endpoint model with the following data:
|
|
536
|
+
```json
|
|
537
|
+
{
|
|
538
|
+
"name": "Get User Summary",
|
|
539
|
+
"path": "user-summary/:username",
|
|
540
|
+
"method": "GET",
|
|
541
|
+
"isActive": true,
|
|
542
|
+
"code": "const { username } = context.request.params;\n\n if (!username) {\n logger.error('Username parameter is required.');\n return { success: false, error: 'Username is required.' };\n }\n\n logger.info(`Fetching summary for user: ${username}`);\n\n // Fetch the user profile using the sandboxed db API\n const userProfile = await db.findOne('userProfile', { username: username });\n\n if (!userProfile) {\n return { success: false, error: 'User not found' };\n }\n\n // Count the user's published posts\n const posts = await db.find('post', { \n authorId: userProfile._id.toString(), \n status: 'published' \n });\n\n return {\n profile: userProfile,\n publishedPosts: posts.length\n };\n});",
|
|
543
|
+
}
|
|
544
|
+
```
|
|
545
|
+
2. Call the New Endpoint
|
|
546
|
+
You can now call this custom endpoint like any other API route:
|
|
547
|
+
|
|
548
|
+
Expected response :
|
|
549
|
+
```json
|
|
550
|
+
{
|
|
551
|
+
"profile": {
|
|
552
|
+
"_id": "60d0fe4f5311236168a109ca",
|
|
553
|
+
"username": "demo",
|
|
554
|
+
"bio": "A demo user profile.",
|
|
555
|
+
"...": "..."
|
|
556
|
+
},
|
|
557
|
+
"publishedPosts": 15
|
|
558
|
+
}
|
|
559
|
+
```
|
|
478
560
|
## 🤝 Contributing
|
|
479
561
|
|
|
480
562
|
1. Fork the repo
|
|
@@ -492,4 +574,4 @@ Distributed under the **MIT License**. See `LICENSE` file.
|
|
|
492
574
|
|
|
493
575
|
---
|
|
494
576
|
|
|
495
|
-
## 🔼 Back to Top
|
|
577
|
+
## [🔼](#) Back to Top
|
|
@@ -1,16 +1,16 @@
|
|
|
1
|
-
import React, { useState, useEffect, useRef
|
|
1
|
+
import React, { useState, useEffect, useRef } from 'react';
|
|
2
2
|
import { useQuery, useQueryClient } from 'react-query';
|
|
3
3
|
import { useModelContext } from './contexts/ModelContext.jsx';
|
|
4
4
|
import { TextField } from './Field.jsx';
|
|
5
5
|
import { useAuthContext } from './contexts/AuthContext.jsx';
|
|
6
|
-
import {
|
|
6
|
+
import { getDataAsString, getUserId } from 'data-primals-engine/data';
|
|
7
7
|
import { FaEdit, FaTrash } from 'react-icons/fa';
|
|
8
8
|
import Button from './Button.jsx';
|
|
9
|
-
import {mainFieldsTypes} from "../../src/constants.js";
|
|
9
|
+
import { mainFieldsTypes } from "../../src/constants.js";
|
|
10
10
|
import Draggable from "./Draggable.jsx";
|
|
11
|
-
import {useTranslation} from "react-i18next";
|
|
11
|
+
import { useTranslation } from "react-i18next";
|
|
12
12
|
|
|
13
|
-
const RelationField = ({ field, help, onFocus, onBlur, onChange, value=null }) => {
|
|
13
|
+
const RelationField = ({ field, help, onFocus, onBlur, onChange, value = null }) => {
|
|
14
14
|
const { models, dataByModel, setOnSuccessCallbacks, relationIds, setRelationIds } = useModelContext();
|
|
15
15
|
const { name, relation: modelName } = field;
|
|
16
16
|
const queryClient = useQueryClient();
|
|
@@ -19,45 +19,67 @@ const RelationField = ({ field, help, onFocus, onBlur, onChange, value=null }) =
|
|
|
19
19
|
const [searchValue, setSearchValue] = useState('');
|
|
20
20
|
const [history, setHistory] = useState([]);
|
|
21
21
|
const { me } = useAuthContext();
|
|
22
|
-
const tr = useTranslation()
|
|
23
|
-
const {t, i18n} = tr
|
|
22
|
+
const tr = useTranslation();
|
|
23
|
+
const { t, i18n } = tr;
|
|
24
24
|
const model = models?.find(f => f.name === modelName && f._user === me?.username);
|
|
25
25
|
|
|
26
26
|
// Fetch related data based on search value
|
|
27
27
|
const { data: results = [], isError, refetch } = useQuery(
|
|
28
|
-
|
|
28
|
+
// La clé de la requête inclut maintenant le filtre de relation pour une mise en cache correcte
|
|
29
|
+
['api/search', model?.name, field?.name, searchValue, JSON.stringify(field.relationFilter)],
|
|
29
30
|
async ({ signal }) => {
|
|
30
31
|
if (!model) return [];
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
32
|
+
|
|
33
|
+
// --- DÉBUT DE LA NOUVELLE LOGIQUE DE FILTRAGE ---
|
|
34
|
+
|
|
35
|
+
// 1. On récupère le filtre permanent défini dans le modèle.
|
|
36
|
+
// S'il n'y en a pas, on utilise un objet vide qui n'aura aucun effet.
|
|
37
|
+
const permanentFilter = field.relationFilter || {};
|
|
38
|
+
|
|
39
|
+
// 2. On construit le filtre basé sur la recherche de l'utilisateur.
|
|
40
|
+
const searchFilter = {};
|
|
41
|
+
if (searchValue) {
|
|
42
|
+
const orConditions = [];
|
|
43
|
+
// Recherche sur les champs principaux (asMain)
|
|
36
44
|
model.fields.forEach(f => {
|
|
37
|
-
if (f.asMain) {
|
|
38
|
-
|
|
39
|
-
orFilter.push({"$regexMatch": { input: '$'+f.name, regex: searchValue}});
|
|
45
|
+
if (f.asMain && mainFieldsTypes.includes(f.type)) {
|
|
46
|
+
orConditions.push({ [f.name]: { "$regex": searchValue, "$options": "i" } });
|
|
40
47
|
}
|
|
41
|
-
})
|
|
42
|
-
|
|
48
|
+
});
|
|
49
|
+
// Si aucun champ principal, recherche sur les champs texte
|
|
50
|
+
if (orConditions.length === 0) {
|
|
43
51
|
model.fields.forEach(f => {
|
|
44
|
-
if (["string", "string_t", "richtext", "url"]
|
|
45
|
-
|
|
52
|
+
if (["string", "string_t", "richtext", "url"].includes(f.type)) {
|
|
53
|
+
orConditions.push({ [f.name]: { "$regex": searchValue, "$options": "i" } });
|
|
46
54
|
}
|
|
47
55
|
});
|
|
48
|
-
if (!orFilter.length) {
|
|
49
|
-
orFilter.push({"$eq": ['_id', searchValue]});
|
|
50
|
-
}
|
|
51
56
|
}
|
|
52
|
-
|
|
57
|
+
// Si toujours rien, on cherche sur l'ID (utile pour les développeurs)
|
|
58
|
+
if (orConditions.length === 0) {
|
|
59
|
+
orConditions.push({ "_id": searchValue });
|
|
60
|
+
}
|
|
61
|
+
searchFilter['$or'] = orConditions;
|
|
53
62
|
}
|
|
63
|
+
|
|
64
|
+
// 3. On combine les deux filtres avec un opérateur $and.
|
|
65
|
+
// Un document devra correspondre au filtre permanent ET au filtre de recherche.
|
|
66
|
+
const finalFilter = {
|
|
67
|
+
"$and": [
|
|
68
|
+
permanentFilter,
|
|
69
|
+
searchFilter
|
|
70
|
+
]
|
|
71
|
+
};
|
|
72
|
+
|
|
73
|
+
// --- FIN DE LA NOUVELLE LOGIQUE DE FILTRAGE ---
|
|
74
|
+
|
|
54
75
|
const params = new URLSearchParams();
|
|
55
|
-
params.append('_user', getUserId(me));
|
|
56
76
|
params.append('model', field.relation);
|
|
57
|
-
params.append('limit', '
|
|
58
|
-
params.append('depth', '2');
|
|
77
|
+
params.append('limit', '100'); // Limite raisonnable pour les suggestions
|
|
78
|
+
params.append('depth', '2');
|
|
79
|
+
|
|
59
80
|
return fetch(`/api/data/search?${params.toString()}`, {
|
|
60
|
-
|
|
81
|
+
// On envoie le filtre final et complet
|
|
82
|
+
body: JSON.stringify({ filter: finalFilter }),
|
|
61
83
|
method: 'POST',
|
|
62
84
|
signal: signal,
|
|
63
85
|
headers: { 'Content-Type': 'application/json' },
|
|
@@ -65,9 +87,11 @@ const RelationField = ({ field, help, onFocus, onBlur, onChange, value=null }) =
|
|
|
65
87
|
.then(e => e.json())
|
|
66
88
|
.then(e => e.data);
|
|
67
89
|
},
|
|
68
|
-
{ enabled: !!model }
|
|
90
|
+
{ enabled: !!model && showResults } // La requête ne s'exécute que si le panneau de résultats est visible
|
|
69
91
|
);
|
|
70
92
|
|
|
93
|
+
// ... (le reste du composant reste identique) ...
|
|
94
|
+
|
|
71
95
|
useEffect(() => {
|
|
72
96
|
setSearchValue('');
|
|
73
97
|
setHistory([]);
|
|
@@ -100,7 +124,6 @@ const RelationField = ({ field, help, onFocus, onBlur, onChange, value=null }) =
|
|
|
100
124
|
}
|
|
101
125
|
}, [value]);
|
|
102
126
|
|
|
103
|
-
//console.log(field, value);
|
|
104
127
|
const updateValue = () => {
|
|
105
128
|
if (!field.multiple) {
|
|
106
129
|
const v = history.find(d => d._id === value);
|
|
@@ -108,7 +131,7 @@ const RelationField = ({ field, help, onFocus, onBlur, onChange, value=null }) =
|
|
|
108
131
|
setSearchValue(getDataAsString(model, v, tr, models) || '');
|
|
109
132
|
onChange({ name, value });
|
|
110
133
|
} else {
|
|
111
|
-
//
|
|
134
|
+
// Ne rien faire si la valeur n'est pas dans l'historique pour éviter d'effacer
|
|
112
135
|
}
|
|
113
136
|
} else {
|
|
114
137
|
if (Array.isArray(value)) {
|
|
@@ -133,7 +156,7 @@ const RelationField = ({ field, help, onFocus, onBlur, onChange, value=null }) =
|
|
|
133
156
|
return value;
|
|
134
157
|
});
|
|
135
158
|
} else {
|
|
136
|
-
onChange({ name, value });
|
|
159
|
+
onChange({ name, value: selectedValues });
|
|
137
160
|
}
|
|
138
161
|
}
|
|
139
162
|
setResultsVisible(false);
|
|
@@ -141,10 +164,6 @@ const RelationField = ({ field, help, onFocus, onBlur, onChange, value=null }) =
|
|
|
141
164
|
e.preventDefault();
|
|
142
165
|
};
|
|
143
166
|
|
|
144
|
-
useEffect(() => {
|
|
145
|
-
if (showResults) refetch();
|
|
146
|
-
}, [showResults]);
|
|
147
|
-
|
|
148
167
|
const handleRemove = (element) => {
|
|
149
168
|
if (!field.multiple) return;
|
|
150
169
|
setSelectedValues(values => {
|
|
@@ -162,6 +181,7 @@ const RelationField = ({ field, help, onFocus, onBlur, onChange, value=null }) =
|
|
|
162
181
|
inputRef.current.ref.setAttribute('autocomplete', 'off');
|
|
163
182
|
}
|
|
164
183
|
}, [inputRef]);
|
|
184
|
+
|
|
165
185
|
return (
|
|
166
186
|
<div onFocus={onFocus} onBlur={onBlur} className="field field-relation flex flex-row flex-start flex-1">
|
|
167
187
|
{help && <div className={"flex help"}>{help}</div>}
|
|
@@ -176,17 +196,15 @@ const RelationField = ({ field, help, onFocus, onBlur, onChange, value=null }) =
|
|
|
176
196
|
id={field.name}
|
|
177
197
|
onFocus={e => {
|
|
178
198
|
setResultsVisible(true);
|
|
179
|
-
queryClient.invalidateQueries(['api/search', field.name, searchValue]);
|
|
180
199
|
}}
|
|
181
200
|
value={searchValue}
|
|
182
201
|
onChange={e => {
|
|
183
|
-
if (!e.target.value) onChange({ name, value:
|
|
184
|
-
else onChange({ name, value: '' });
|
|
202
|
+
if (!e.target.value) onChange({ name, value: null });
|
|
185
203
|
setResultsVisible(true);
|
|
186
204
|
setSearchValue(e.target.value);
|
|
187
205
|
}}
|
|
188
206
|
onBlur={(e) => {
|
|
189
|
-
setResultsVisible(false);
|
|
207
|
+
setTimeout(() => setResultsVisible(false), 150); // Léger délai pour permettre le clic sur un résultat
|
|
190
208
|
}}
|
|
191
209
|
/>
|
|
192
210
|
<Button
|
|
@@ -200,11 +218,6 @@ const RelationField = ({ field, help, onFocus, onBlur, onChange, value=null }) =
|
|
|
200
218
|
<div className="results" onKeyDown={e => {
|
|
201
219
|
if( e.key === 'Escape' )
|
|
202
220
|
setResultsVisible(false);
|
|
203
|
-
}} onBlur={e => {
|
|
204
|
-
const t = e.target.parentNode.children[e.target.parentNode.childElementCount-1];
|
|
205
|
-
if(t === e.target){
|
|
206
|
-
setResultsVisible(false);
|
|
207
|
-
}
|
|
208
221
|
}} >
|
|
209
222
|
{!isError &&
|
|
210
223
|
(results || []).map(r => {
|
|
@@ -224,7 +237,7 @@ const RelationField = ({ field, help, onFocus, onBlur, onChange, value=null }) =
|
|
|
224
237
|
{field.multiple && selectedValues.length > 0 && (
|
|
225
238
|
<div ref={ref} tabIndex={0} className="selected-values flex flex-border flex-row flex-no-gap flex-start flex-1">
|
|
226
239
|
<Draggable items={selectedValues} renderItem={(id,i) =>{
|
|
227
|
-
const val = history.find(f => f._id === id) || dataByModel[modelName]
|
|
240
|
+
const val = history.find(f => f._id === id) || dataByModel[modelName]?.find(f => f._id === id);
|
|
228
241
|
if (!val) {
|
|
229
242
|
return <div className="flex" key={id}>data non chargée<FaTrash onClick={() => handleRemove(id)} /></div>;
|
|
230
243
|
}
|
|
@@ -239,7 +252,6 @@ const RelationField = ({ field, help, onFocus, onBlur, onChange, value=null }) =
|
|
|
239
252
|
}} />
|
|
240
253
|
</div>
|
|
241
254
|
)}
|
|
242
|
-
|
|
243
255
|
</div>
|
|
244
256
|
);
|
|
245
257
|
};
|
|
@@ -523,7 +523,7 @@ export const websiteTranslations = {
|
|
|
523
523
|
"prez1.5": "Support des traductions et langues",
|
|
524
524
|
"links.deploy_api": "Déployez votre API",
|
|
525
525
|
"links.documentation": "Documentation",
|
|
526
|
-
"links.demo": "
|
|
526
|
+
"links.demo": "Démarrer la démo",
|
|
527
527
|
"filterstringfield.placeholder.regex": 'Expression régulière'
|
|
528
528
|
|
|
529
529
|
}
|
|
@@ -1483,7 +1483,7 @@ export const websiteTranslations = {
|
|
|
1483
1483
|
"dataimporter.title": "Import data into {{model}}",
|
|
1484
1484
|
"dataimporter.hasCsvHeaders": "Does the CSV file contain a header row?",
|
|
1485
1485
|
"dataimporter.info": "Import your JSON or CSV file containing the required data for your model. Your data will be copied to your personal database. The transfer rate is {{constante}}",
|
|
1486
|
-
"links.demo": "
|
|
1486
|
+
"links.demo": "Start the demo",
|
|
1487
1487
|
"filterstringfield.placeholder.regex": 'Regular expression',
|
|
1488
1488
|
"cgu.h1": "Terms of Use (TOU)",
|
|
1489
1489
|
"cgu.h1.1": "1. Introduction",
|
|
@@ -2887,7 +2887,7 @@ export const websiteTranslations = {
|
|
|
2887
2887
|
"dataimporter.title": "Importar datos a {{model}}",
|
|
2888
2888
|
"dataimporter.hasCsvHeaders": "¿El archivo CSV contiene una fila de encabezado?",
|
|
2889
2889
|
"dataimporter.info": "Importa tu archivo JSON o CSV con los datos necesarios para tu modelo. Tus datos se copiarán a tu base de datos personal. La velocidad de transferencia es {{constante}}",
|
|
2890
|
-
"links.demo": "
|
|
2890
|
+
"links.demo": "Iniciar la demostración",
|
|
2891
2891
|
"filterstringfield.placeholder.regex": 'Expresión regular',
|
|
2892
2892
|
"cgu.h1": "Condiciones Generales de Uso (CGU)",
|
|
2893
2893
|
"cgu.h1.1": "1. Introducción",
|
|
@@ -4298,7 +4298,7 @@ export const websiteTranslations = {
|
|
|
4298
4298
|
"dataimporter.title": "Importar dados para {{model}}",
|
|
4299
4299
|
"dataimporter.hasCsvHeaders": "O ficheiro CSV contém uma linha de cabeçalho?",
|
|
4300
4300
|
"dataimporter.info": "Importe o seu ficheiro JSON ou CSV contendo os dados necessários para o seu modelo. Os seus dados serão copiados para a sua base de dados pessoal. A taxa de transferência é {{constante}}",
|
|
4301
|
-
"links.demo": "
|
|
4301
|
+
"links.demo": "Iniciar a demonstração",
|
|
4302
4302
|
"filterstringfield.placeholder.regex": 'Expressão regular',
|
|
4303
4303
|
"cgu.h1": "Termos e Condições Gerais de Uso (TCGU)",
|
|
4304
4304
|
"cgu.h1.1": "1. Introdução",
|
|
@@ -5681,7 +5681,7 @@ export const websiteTranslations = {
|
|
|
5681
5681
|
"dataimporter.title": "Daten in {{model}} importieren",
|
|
5682
5682
|
"dataimporter.hasCsvHeaders": "Enthält die CSV-Datei eine Kopfzeile?",
|
|
5683
5683
|
"dataimporter.info": "Importieren Sie Ihre JSON- oder CSV-Datei mit den benötigten Daten für Ihr Modell. Ihre Daten werden in Ihre persönliche Datenbank kopiert. Die Übertragungsrate beträgt {{constante}}",
|
|
5684
|
-
"links.demo": "
|
|
5684
|
+
"links.demo": "Demo starten",
|
|
5685
5685
|
"filterstringfield.placeholder.regex": 'Regulärer Ausdruck',
|
|
5686
5686
|
"cgu.h1": "Nutzungsbedingungen (TOU)",
|
|
5687
5687
|
"cgu.h1.1": "1. Einleitung",
|
|
@@ -7087,7 +7087,7 @@ export const websiteTranslations = {
|
|
|
7087
7087
|
"dataimporter.title": "Importa dati in {{model}}",
|
|
7088
7088
|
"dataimporter.hasCsvHeaders": "Il file CSV contiene una riga di intestazione?",
|
|
7089
7089
|
"dataimporter.info": "Importa il tuo file JSON o CSV contenente i dati necessari per il tuo modello. I tuoi dati verranno copiati nel tuo database personale. La velocità di trasferimento è {{constante}}",
|
|
7090
|
-
"links.demo": "
|
|
7090
|
+
"links.demo": "Avvia la demo",
|
|
7091
7091
|
"filterstringfield.placeholder.regex": 'Espressione regolare',
|
|
7092
7092
|
"cgu.h1": "Termini di Utilizzo (TOU)",
|
|
7093
7093
|
"cgu.h1.1": "1. Introduzione",
|
|
@@ -8483,7 +8483,7 @@ export const websiteTranslations = {
|
|
|
8483
8483
|
"dataimporter.title": "Importovat data do {{model}}",
|
|
8484
8484
|
"dataimporter.hasCsvHeaders": "Obsahuje soubor CSV řádek záhlaví?",
|
|
8485
8485
|
"dataimporter.info": "Importujte soubor JSON nebo CSV obsahující požadovaná data pro váš model. Vaše data budou zkopírována do vaší osobní databáze. Přenosová rychlost je {{constante}}",
|
|
8486
|
-
"links.demo": "
|
|
8486
|
+
"links.demo": "Spustit demo",
|
|
8487
8487
|
"filterstringfield.placeholder.regex": 'Regulární výraz',
|
|
8488
8488
|
"cgu.h1": "Podmínky použití (TOU)",
|
|
8489
8489
|
"cgu.h1.1": "1. Úvod",
|
|
@@ -9887,7 +9887,7 @@ export const websiteTranslations = {
|
|
|
9887
9887
|
"dataimporter.title": "Импортировать данные в {{model}}",
|
|
9888
9888
|
"dataimporter.hasCsvHeaders": "Содержит ли CSV-файл строку заголовка?",
|
|
9889
9889
|
"dataimporter.info": "Импортируйте файл JSON или CSV, содержащий необходимые данные для вашей модели. Ваши данные будут скопированы в вашу личную базу данных. Скорость передачи составляет {{constante}}",
|
|
9890
|
-
"links.demo": "
|
|
9890
|
+
"links.demo": "Запустить демо",
|
|
9891
9891
|
"filterstringfield.placeholder.regex": 'Регулярное выражение',
|
|
9892
9892
|
"cgu.h1": "Условия использования (TOU)",
|
|
9893
9893
|
"cgu.h1.1": "1. Введение",
|
|
@@ -11311,7 +11311,7 @@ export const websiteTranslations = {
|
|
|
11311
11311
|
"dataimporter.title": "استيراد البيانات إلى {{model}}",
|
|
11312
11312
|
"dataimporter.hasCsvHeaders": "هل يحتوي ملف CSV على صف رأس؟",
|
|
11313
11313
|
"dataimporter.info": "استورد ملف JSON أو CSV الذي يحتوي على البيانات المطلوبة لنموذجك. سيتم نسخ بياناتك إلى قاعدة بياناتك الشخصية. معدل النقل ثابت ({{constante}}).",
|
|
11314
|
-
"links.demo": "
|
|
11314
|
+
"links.demo": "ابدأ العرض التوضيحي",
|
|
11315
11315
|
"filterstringfield.placeholder.regex": 'تعبير عادي',
|
|
11316
11316
|
"cgu.h1": "الشروط العامة للاستخدام (GTU)",
|
|
11317
11317
|
"cgu.h1.1": "1. المقدمة",
|
|
@@ -12711,7 +12711,7 @@ export const websiteTranslations = {
|
|
|
12711
12711
|
"dataimporter.title": "Importera data till {{model}}",
|
|
12712
12712
|
"dataimporter.hasCsvHeaders": "Innehåller CSV-filen en rubrikrad?",
|
|
12713
12713
|
"dataimporter.info": "Importera din JSON- eller CSV-fil som innehåller den data som krävs för din modell. Dina data kommer att kopieras till din personliga databas. Överföringshastigheten är {{constante}}",
|
|
12714
|
-
"links.demo": "
|
|
12714
|
+
"links.demo": "Starta demon",
|
|
12715
12715
|
"filterstringfield.placeholder.regex": 'Reguljärt uttryck',
|
|
12716
12716
|
"cgu.h1": "Användarvillkor (Användarvillkor)",
|
|
12717
12717
|
"cgu.h1.1": "1. Introduktion",
|
|
@@ -14112,7 +14112,7 @@ export const websiteTranslations = {
|
|
|
14112
14112
|
"dataimporter.title": "Εισαγωγή δεδομένων στο {{model}}",
|
|
14113
14113
|
"dataimporter.hasCsvHeaders": "Περιέχει το αρχείο CSV μια γραμμή κεφαλίδας;",
|
|
14114
14114
|
"dataimporter.info": "Εισαγάγετε το αρχείο JSON που περιέχει τα απαιτούμενα δεδομένα για το μοντέλο σας. Τα δεδομένα σας θα αντιγραφούν στην προσωπική σας βάση δεδομένων. Ο ρυθμός μεταφοράς είναι {{constante}}",
|
|
14115
|
-
"links.demo": "
|
|
14115
|
+
"links.demo": "Έναρξη της επίδειξης",
|
|
14116
14116
|
"filterstringfield.placeholder.regex": "Κανονική έκφραση",
|
|
14117
14117
|
"cgu.h1": "Γενικοί όροι χρήσης (GTU)",
|
|
14118
14118
|
"cgu.h1.1": "1. Εισαγωγή",
|
|
@@ -14561,6 +14561,7 @@ export const websiteTranslations = {
|
|
|
14561
14561
|
},
|
|
14562
14562
|
fa: {
|
|
14563
14563
|
"translation": {
|
|
14564
|
+
"links.demo": "شروع نسخهٔ نمایشی",
|
|
14564
14565
|
"cb.searchOperator": "جستجوی یک عملگر...",
|
|
14565
14566
|
"cb.changeOperator": "تغییر عملگر",
|
|
14566
14567
|
"cb.deleteBlock": "حذف این بلوک",
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "data-primals-engine",
|
|
3
|
-
"version": "1.1.
|
|
3
|
+
"version": "1.1.8",
|
|
4
4
|
"description": "data-primals-engine is the package responsible from handling large amount of data in a practical and performant way. It can handle large amount of data in a practical and performant way. It can also get workflow models working (for automation), and fully supports internationalisation.",
|
|
5
5
|
"main": "src/engine.js",
|
|
6
6
|
"type": "module",
|
|
@@ -43,9 +43,9 @@
|
|
|
43
43
|
"./*": "./src/*.js"
|
|
44
44
|
},
|
|
45
45
|
"peerDependencies": {
|
|
46
|
+
"express": "^5.1.0",
|
|
46
47
|
"react": ">=18.0.0",
|
|
47
|
-
"react-query": ">=3.0.0"
|
|
48
|
-
"express": "^5.1.0"
|
|
48
|
+
"react-query": ">=3.0.0"
|
|
49
49
|
},
|
|
50
50
|
"dependencies": {
|
|
51
51
|
"@langchain/core": "^0.3.66",
|
|
@@ -69,6 +69,7 @@
|
|
|
69
69
|
"express-rate-limit": "^8.0.1",
|
|
70
70
|
"express-session": "^1.18.2",
|
|
71
71
|
"i18next-browser-languagedetector": "^8.2.0",
|
|
72
|
+
"isolated-vm": "^4.7.2",
|
|
72
73
|
"juice": "^11.0.1",
|
|
73
74
|
"mathjs": "^14.6.0",
|
|
74
75
|
"mongodb": "^6.18.0",
|
|
@@ -88,7 +89,6 @@
|
|
|
88
89
|
"tar": "^7.4.3",
|
|
89
90
|
"uniqid": "^5.4.0",
|
|
90
91
|
"vitest": "^3.2.4",
|
|
91
|
-
"isolated-vm": "^4.7.2",
|
|
92
92
|
"yaml": "^2.8.0"
|
|
93
93
|
},
|
|
94
94
|
"devDependencies": {
|
|
@@ -102,13 +102,12 @@
|
|
|
102
102
|
"client"
|
|
103
103
|
],
|
|
104
104
|
"keywords": [
|
|
105
|
-
"
|
|
106
|
-
"
|
|
107
|
-
"
|
|
105
|
+
"data-driven engine",
|
|
106
|
+
"headless CMS",
|
|
107
|
+
"backend",
|
|
108
108
|
"automation",
|
|
109
|
-
"
|
|
110
|
-
"
|
|
111
|
-
"S3"
|
|
109
|
+
"AWS S3",
|
|
110
|
+
"MongoDB"
|
|
112
111
|
],
|
|
113
112
|
"author": "anonympins",
|
|
114
113
|
"license": "MIT",
|
package/src/core.js
CHANGED
|
@@ -340,12 +340,13 @@ export const event_off = (name, callback) => {
|
|
|
340
340
|
};
|
|
341
341
|
|
|
342
342
|
|
|
343
|
-
export function slugify(str) {
|
|
343
|
+
export function slugify(str,replacer='-', replaceUnicode=false) {
|
|
344
344
|
str = str.replace(/^\s+|\s+$/g, ''); // trim leading/trailing white space
|
|
345
345
|
str = str.toLowerCase(); // convert string to lowercase
|
|
346
|
-
|
|
347
|
-
.replace(
|
|
348
|
-
|
|
346
|
+
if( replaceUnicode)
|
|
347
|
+
str = str.replace(/[^a-z0-9 -]/g, ''); // remove any non-alphanumeric characters
|
|
348
|
+
str = str.replace(/\s+/g, replacer) // replace spaces with hyphens
|
|
349
|
+
.replace(/-+/g, replacer); // remove consecutive hyphens
|
|
349
350
|
return str;
|
|
350
351
|
}
|
|
351
352
|
|
|
@@ -388,4 +389,8 @@ export function object_equals( x, y ) {
|
|
|
388
389
|
// allows x[ p ] to be set to undefined
|
|
389
390
|
|
|
390
391
|
return true;
|
|
391
|
-
}
|
|
392
|
+
}
|
|
393
|
+
|
|
394
|
+
export const isValidPath = (path) =>{
|
|
395
|
+
return /^(?:[a-z]:)?[\/\\]{0,2}(?:[.\/\\ ](?![.\/\\\n])|[^<>:"|?*.\/\\ \n])+$/gmi.test(path);
|
|
396
|
+
}
|
package/src/engine.js
CHANGED
|
@@ -41,17 +41,35 @@ const secureContext = tls.createSecureContext({
|
|
|
41
41
|
ca: caFile, cert: certFile, key: keyFile
|
|
42
42
|
});
|
|
43
43
|
|
|
44
|
+
export const dbUrl = process.env.CI ? 'mongodb://mongodb:27017' : (process.env.MONGO_DB_URL || 'mongodb://127.0.0.1:27017');
|
|
45
|
+
|
|
44
46
|
const isTlsActive = !(!process.env.TLS || ["0", "false"].includes(process.env.TLS.toLowerCase()));
|
|
45
47
|
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
48
|
+
const clientOptions = {
|
|
49
|
+
maxPoolSize: databasePoolSize
|
|
50
|
+
};
|
|
51
|
+
// On ajoute les options TLS si elles sont activées
|
|
52
|
+
if (isTlsActive) {
|
|
53
|
+
clientOptions.tls = true;
|
|
54
|
+
// Chemin vers le certificat de l'autorité de certification (pour faire confiance au serveur)
|
|
55
|
+
if (process.env.CA_CERT) {
|
|
56
|
+
clientOptions.tlsCAFile = process.env.CA_CERT;
|
|
57
|
+
}
|
|
58
|
+
// Chemin vers le certificat et la clé du CLIENT (pour que le serveur vous fasse confiance)
|
|
59
|
+
if (process.env.CERT_KEY) {
|
|
60
|
+
clientOptions.tlsCertificateKeyFile = process.env.CERT_KEY;
|
|
61
|
+
}
|
|
62
|
+
// Options pour le développement (à utiliser avec prudence)
|
|
63
|
+
if (tlsAllowInvalidCertificates) {
|
|
64
|
+
clientOptions.tlsAllowInvalidCertificates = true;
|
|
65
|
+
}
|
|
66
|
+
if (tlsAllowInvalidHostnames) {
|
|
67
|
+
clientOptions.tlsAllowInvalidHostnames = true;
|
|
68
|
+
}
|
|
69
|
+
}
|
|
70
|
+
|
|
71
|
+
export const MongoClient = new InternalMongoClient(dbUrl, clientOptions);
|
|
72
|
+
|
|
55
73
|
|
|
56
74
|
// Database Name
|
|
57
75
|
export const MongoDatabase = MongoClient.db(dbName);
|
|
@@ -112,7 +112,7 @@ function sanitize(target, options = {}) {
|
|
|
112
112
|
export function middleware(options = {}) {
|
|
113
113
|
const hasOnSanitize = typeof options.onSanitize === 'function';
|
|
114
114
|
return function (req, res, next) {
|
|
115
|
-
['body', 'params', 'headers', 'query'].forEach(function (key) {
|
|
115
|
+
['body', 'fields', 'params', 'headers', 'query'].forEach(function (key) {
|
|
116
116
|
if (req[key]) {
|
|
117
117
|
// The _sanitize function mutates the req[key] object in-place.
|
|
118
118
|
// We only need to capture whether it was sanitized to call the hook.
|
package/src/modules/data.js
CHANGED
|
@@ -3,7 +3,7 @@ import {BSON, ObjectId} from "mongodb";
|
|
|
3
3
|
import * as util from 'node:util';
|
|
4
4
|
import {promisify} from 'node:util';
|
|
5
5
|
import crypto from "node:crypto";
|
|
6
|
-
import {exec} from 'node:child_process';
|
|
6
|
+
import {exec,execFile} from 'node:child_process';
|
|
7
7
|
import sanitizeHtml from 'sanitize-html';
|
|
8
8
|
import * as tar from "tar";
|
|
9
9
|
import process from "node:process";
|
|
@@ -105,6 +105,7 @@ const delay = ms => new Promise(res => setTimeout(res, ms));
|
|
|
105
105
|
|
|
106
106
|
const getBackupDir = () => process.env.BACKUP_DIR || './backups'; // Répertoire de stockage des sauvegardes
|
|
107
107
|
const execAsync = promisify(exec);
|
|
108
|
+
const execFileAsync = promisify(execFile);
|
|
108
109
|
|
|
109
110
|
let importJobs = {};
|
|
110
111
|
const IMPORT_CHUNK_SIZE = 100; // Nombre d'enregistrements à traiter par lot
|
|
@@ -3489,6 +3490,24 @@ async function insertAndResolveRelations(doc, model, collection, me, idMap) {
|
|
|
3489
3490
|
return existingDoc._id;
|
|
3490
3491
|
}
|
|
3491
3492
|
|
|
3493
|
+
for (const field of model.fields) {
|
|
3494
|
+
if (field.type === 'relation' && field.relationFilter && docToProcess[field.name]) {
|
|
3495
|
+
const relatedIds = Array.isArray(docToProcess[field.name]) ? docToProcess[field.name] : [docToProcess[field.name]];
|
|
3496
|
+
for (const id of relatedIds) {
|
|
3497
|
+
const targetCollection = await getCollectionForUser(me, field.targetModel);
|
|
3498
|
+
const validationQuery = {
|
|
3499
|
+
_id: new ObjectId(id), // L'ID doit correspondre
|
|
3500
|
+
...field.relationFilter // ET le document doit respecter le filtre
|
|
3501
|
+
};
|
|
3502
|
+
const relatedDoc = await targetCollection.findOne(validationQuery);
|
|
3503
|
+
if (!relatedDoc) {
|
|
3504
|
+
// Si on ne trouve rien, c'est que l'ID est invalide ou ne respecte pas le filtre.
|
|
3505
|
+
throw new Error(`La valeur '${id}' pour le champ '${field.name}' ne respecte pas le filtre de relation défini.`);
|
|
3506
|
+
}
|
|
3507
|
+
}
|
|
3508
|
+
}
|
|
3509
|
+
}
|
|
3510
|
+
|
|
3492
3511
|
// Insertion en conservant éventuellement l'ID original
|
|
3493
3512
|
const result = docToProcess._id
|
|
3494
3513
|
? await collection.insertOne(docToProcess)
|
|
@@ -3708,6 +3727,33 @@ const internalEditOrPatchData = async (modelName, filter, data, files, user, isP
|
|
|
3708
3727
|
}
|
|
3709
3728
|
}
|
|
3710
3729
|
|
|
3730
|
+
for (const field of model.fields) {
|
|
3731
|
+
// On ne vérifie que si un champ de relation avec un filtre est en cours de modification.
|
|
3732
|
+
if (field.type === 'relation' && field.relationFilter && updateData[field.name] !== undefined) {
|
|
3733
|
+
const relatedIds = Array.isArray(updateData[field.name])
|
|
3734
|
+
? updateData[field.name]
|
|
3735
|
+
: (updateData[field.name] ? [updateData[field.name]] : []);
|
|
3736
|
+
|
|
3737
|
+
for (const id of relatedIds) {
|
|
3738
|
+
if (!id || !isObjectId(id)) continue; // Ignorer les valeurs null/invalides
|
|
3739
|
+
|
|
3740
|
+
const targetCollection = await getCollectionForUser(user, field.relation);
|
|
3741
|
+
|
|
3742
|
+
const validationQuery = {
|
|
3743
|
+
_id: new ObjectId(id),
|
|
3744
|
+
...field.relationFilter
|
|
3745
|
+
};
|
|
3746
|
+
|
|
3747
|
+
const relatedDoc = await targetCollection.findOne(validationQuery);
|
|
3748
|
+
|
|
3749
|
+
if (!relatedDoc) {
|
|
3750
|
+
// Si on ne trouve rien, c'est que l'ID est invalide ou ne respecte pas le filtre.
|
|
3751
|
+
throw new Error(`La valeur '${id}' pour le champ '${field.name}' ne respecte pas le filtre de relation défini.`);
|
|
3752
|
+
}
|
|
3753
|
+
}
|
|
3754
|
+
}
|
|
3755
|
+
}
|
|
3756
|
+
|
|
3711
3757
|
// 8. Calcul du nouveau hash et préparation des données finales
|
|
3712
3758
|
const finalStateForHash = { ...existingDocs[0], ...updateData };
|
|
3713
3759
|
const newHash = getFieldValueHash(model, finalStateForHash);
|
|
@@ -5172,6 +5218,7 @@ const handleFields = async (model, data, user, isRecursiveCall = false) => {
|
|
|
5172
5218
|
}
|
|
5173
5219
|
};
|
|
5174
5220
|
|
|
5221
|
+
let restoreRequests = {};
|
|
5175
5222
|
export const validateRestoreRequest = (username, token) => {
|
|
5176
5223
|
const request = restoreRequests[username];
|
|
5177
5224
|
if (!request) {
|
|
@@ -5202,20 +5249,21 @@ export const loadFromDump = async (user, options = {}) => {
|
|
|
5202
5249
|
throw new Error("No encryption key found for this user. Cannot restore.");
|
|
5203
5250
|
}
|
|
5204
5251
|
|
|
5252
|
+
const userId = getObjectHash({user: user.username});
|
|
5205
5253
|
// --- La logique pour trouver le fichier de backup (local ou S3) reste la même ---
|
|
5206
5254
|
// (le code existant pour trouver/télécharger le backupFilePath va ici)
|
|
5207
5255
|
// ...
|
|
5208
5256
|
let backupFilePath; // Assurez-vous que cette variable est bien définie avec le chemin du fichier .tar.gz
|
|
5209
5257
|
// Exemple simplifié :
|
|
5210
5258
|
const backupDir = getBackupDir();
|
|
5211
|
-
const backupFilenameRegex = new RegExp(`^backup_${
|
|
5259
|
+
const backupFilenameRegex = new RegExp(`^backup_${userId}_(\\d+)\\.tar\\.gz$`);
|
|
5212
5260
|
const backupFiles = fs.readdirSync(backupDir).filter(filename => backupFilenameRegex.test(filename));
|
|
5213
5261
|
if (backupFiles.length === 0) throw new Error(`Aucun fichier de sauvegarde local trouvé pour l'utilisateur ${user.username}.`);
|
|
5214
5262
|
const latestBackupFile = backupFiles.sort((a, b) => parseInt(b.match(backupFilenameRegex)[1], 10) - parseInt(a.match(backupFilenameRegex)[1], 10))[0];
|
|
5215
5263
|
backupFilePath = path.join(backupDir, latestBackupFile);
|
|
5216
5264
|
// --- Fin de la logique de recherche de fichier ---
|
|
5217
5265
|
|
|
5218
|
-
const tmpRestoreDir = path.join(backupDir, `tmp_restore_${
|
|
5266
|
+
const tmpRestoreDir = path.join(backupDir, `tmp_restore_${userId}_${Date.now()}`);
|
|
5219
5267
|
|
|
5220
5268
|
try {
|
|
5221
5269
|
await runCryptoWorkerTask('decrypt', { filePath: backupFilePath, password: encryptedKey });
|
|
@@ -5252,17 +5300,25 @@ export const loadFromDump = async (user, options = {}) => {
|
|
|
5252
5300
|
}
|
|
5253
5301
|
|
|
5254
5302
|
let command;
|
|
5303
|
+
const args = [
|
|
5304
|
+
'--uri', dbUrl,
|
|
5305
|
+
'--db', dbName
|
|
5306
|
+
];
|
|
5307
|
+
|
|
5255
5308
|
if (modelsOnly) {
|
|
5256
|
-
|
|
5257
|
-
command = `mongorestore --uri="${dbUrl}" --db=${dbName} --nsInclude="${dbName}.models" "${restoreSourceDir}"`;
|
|
5309
|
+
args.push('--nsInclude', `${dbName}.models`);
|
|
5258
5310
|
} else {
|
|
5259
|
-
//
|
|
5260
|
-
|
|
5311
|
+
// mongorestore accepte plusieurs fois l'option --nsInclude
|
|
5312
|
+
args.push('--nsInclude', `${dbName}.datas`);
|
|
5313
|
+
args.push('--nsInclude', `${dbName}.models`);
|
|
5261
5314
|
}
|
|
5315
|
+
// Le répertoire source est le dernier argument
|
|
5316
|
+
args.push(restoreSourceDir);
|
|
5262
5317
|
|
|
5263
|
-
logger.info(`[${action}] Executing restore command: ${command}`);
|
|
5264
|
-
await execAsync(command);
|
|
5265
5318
|
|
|
5319
|
+
logger.info(`[${action}] Executing restore command: ${command}`);
|
|
5320
|
+
await execFileAsync('mongorestore', args);
|
|
5321
|
+
|
|
5266
5322
|
// --- Tâches Post-Restauration ---
|
|
5267
5323
|
await scheduleAlerts();
|
|
5268
5324
|
await scheduleWorkflowTriggers();
|
|
@@ -5301,121 +5357,90 @@ const readKeyFromFile = (user) => {
|
|
|
5301
5357
|
return null;
|
|
5302
5358
|
};
|
|
5303
5359
|
|
|
5360
|
+
// C:/Dev/data-primals-engine/src/modules/data.js
|
|
5361
|
+
|
|
5304
5362
|
export const dumpUserData = async (user) => {
|
|
5305
|
-
|
|
5306
|
-
// Pour cet exemple, on simule la config S3. Remplace par la vraie récupération.
|
|
5307
|
-
const s3Config = user.configS3; // Supposons que l'objet 'user' passé contient déjà 'configS3'
|
|
5363
|
+
const s3Config = user.configS3;
|
|
5308
5364
|
const backupDir = getBackupDir();
|
|
5365
|
+
const userId = getObjectHash({ user: user.username });
|
|
5366
|
+
const backupFilename = `backup_${userId}`;
|
|
5367
|
+
const timestamp = Date.now();
|
|
5368
|
+
|
|
5369
|
+
// Déclarer les chemins ici pour qu'ils soient accessibles dans tout le scope de la fonction
|
|
5370
|
+
const localTempDumpDir = path.join(backupDir, `${backupFilename}_${timestamp}_temp`);
|
|
5371
|
+
const finalArchiveName = `${backupFilename}_${timestamp}.tar.gz`;
|
|
5372
|
+
const localArchivePath = path.join(backupDir, finalArchiveName);
|
|
5309
5373
|
|
|
5310
5374
|
let encryptedKey = readKeyFromFile(user);
|
|
5311
5375
|
if (!encryptedKey) {
|
|
5312
5376
|
encryptedKey = generateAndStoreKey(user);
|
|
5313
|
-
logger.info('Encryption key générée.');
|
|
5314
|
-
} else {
|
|
5315
|
-
logger.info('Encryption key chargée.');
|
|
5316
5377
|
}
|
|
5317
5378
|
|
|
5318
5379
|
try {
|
|
5319
|
-
// Déterminer la fréquence de la sauvegarde
|
|
5320
5380
|
const backupFrequency = await engine.userProvider.getBackupFrequency(user);
|
|
5321
|
-
|
|
5322
5381
|
logger.info(`Fréquence de sauvegarde : ${backupFrequency}.`);
|
|
5323
5382
|
|
|
5324
|
-
// Définir le nom du fichier de sauvegarde et les chemins
|
|
5325
|
-
const backupFilename = `backup_${user.username}`; //nom corrigé
|
|
5326
|
-
const backupFileBasePath = path.join(backupDir, backupFilename); //chemin corrigé
|
|
5327
|
-
const backupFilePath = `${backupFileBasePath}_${Date.now()}`;
|
|
5328
|
-
const backupFilenameBase = `backup_${user.username}`;
|
|
5329
|
-
const timestamp = Date.now();
|
|
5330
|
-
const localTempDumpDir = path.join(backupDir, `${backupFilenameBase}_${timestamp}_temp`); // Répertoire temporaire local
|
|
5331
|
-
const finalArchiveName = `${backupFilenameBase}_${timestamp}.tar.gz`; // Nom de l'archive finale
|
|
5332
|
-
const localArchivePath = path.join(backupDir, finalArchiveName); // Chemin
|
|
5333
|
-
|
|
5334
|
-
logger.info(`Chemin du fichier de sauvegarde : ${backupFilePath}.`);
|
|
5335
|
-
|
|
5336
|
-
// Ajoute les filtres sur les collections et l'utilisateur
|
|
5337
5383
|
const collections = await MongoDatabase.listCollections().toArray();
|
|
5338
|
-
let query;
|
|
5339
|
-
let col;
|
|
5340
5384
|
for (const collection of collections) {
|
|
5341
|
-
|
|
5342
|
-
|
|
5343
|
-
|
|
5344
|
-
|
|
5345
|
-
|
|
5346
|
-
|
|
5347
|
-
|
|
5348
|
-
|
|
5349
|
-
|
|
5350
|
-
|
|
5351
|
-
|
|
5352
|
-
|
|
5353
|
-
await execAsync(command);
|
|
5385
|
+
const collsToBackup = [await getUserCollectionName(user), 'models'];
|
|
5386
|
+
if (collsToBackup.includes(collection.name)) {
|
|
5387
|
+
const query = { _user: user.username };
|
|
5388
|
+
const args = [
|
|
5389
|
+
'--uri', dbUrl,
|
|
5390
|
+
'--db', dbName,
|
|
5391
|
+
'--out', localTempDumpDir,
|
|
5392
|
+
'--collection', collection.name,
|
|
5393
|
+
'--query', JSON.stringify(query)
|
|
5394
|
+
];
|
|
5395
|
+
logger.info(`Exécution de la commande : mongodump ${args.join(' ')}`);
|
|
5396
|
+
await execFileAsync('mongodump', args);
|
|
5354
5397
|
}
|
|
5355
|
-
|
|
5356
5398
|
}
|
|
5357
5399
|
|
|
5358
|
-
|
|
5359
|
-
// Zipper le contenu du répertoire de dump temporaire
|
|
5360
|
-
// Le répertoire zippé sera `localTempDumpDir/${dbName}` si mongodump a créé ce sous-dossier
|
|
5361
|
-
const dumpSourceDir = path.join(localTempDumpDir, dbName); // Le répertoire que mongodump a créé
|
|
5400
|
+
const dumpSourceDir = path.join(localTempDumpDir, dbName);
|
|
5362
5401
|
if (fs.existsSync(dumpSourceDir)) {
|
|
5363
5402
|
await tar.create({ gzip: true, file: localArchivePath, C: localTempDumpDir }, [dbName]);
|
|
5364
5403
|
logger.info(`Archive de sauvegarde locale créée : ${localArchivePath}`);
|
|
5365
5404
|
} else {
|
|
5366
|
-
|
|
5405
|
+
logger.warn(`Le répertoire de dump ${dumpSourceDir} était vide. Aucune archive n'a été créée.`);
|
|
5406
|
+
// On s'arrête ici car il n'y a rien à traiter
|
|
5407
|
+
return Promise.resolve();
|
|
5367
5408
|
}
|
|
5368
5409
|
|
|
5369
|
-
|
|
5370
|
-
await encryptFile(localArchivePath, encryptedKey); // Ta fonction existante
|
|
5410
|
+
await encryptFile(localArchivePath, encryptedKey);
|
|
5371
5411
|
|
|
5372
5412
|
if (s3Config && s3Config.bucketName) {
|
|
5373
|
-
logger.info(`Téléversement de la sauvegarde vers S3 pour l'utilisateur ${user.username}. Bucket: ${s3Config.bucketName}`);
|
|
5374
5413
|
await uploadToS3(s3Config, localArchivePath, finalArchiveName);
|
|
5375
|
-
|
|
5376
|
-
// Optionnel: supprimer l'archive locale après l'upload S3 réussi
|
|
5377
|
-
fs.unlinkSync(localArchivePath);
|
|
5378
|
-
logger.info(`Archive locale ${localArchivePath} supprimée après l'upload S3.`);
|
|
5414
|
+
fs.unlinkSync(localArchivePath); // Supprime l'archive locale après l'upload
|
|
5379
5415
|
} else {
|
|
5380
|
-
logger.info(`Aucune configuration S3 trouvée
|
|
5381
|
-
// Si pas de S3, le fichier chiffré localement (si encryptFile a été appelé) reste.
|
|
5382
|
-
// Si tu ne chiffres pas localement et pas de S3, c'est l'archive .tar.gz qui reste.
|
|
5383
|
-
}
|
|
5384
|
-
|
|
5385
|
-
// Supprimer le répertoire de dump temporaire
|
|
5386
|
-
if (fs.existsSync(localTempDumpDir)) {
|
|
5387
|
-
fs.rmSync(localTempDumpDir, { recursive: true, force: true });
|
|
5388
|
-
logger.info(`Répertoire de dump temporaire ${localTempDumpDir} supprimé.`);
|
|
5416
|
+
logger.info(`Aucune configuration S3 trouvée. La sauvegarde reste locale : ${localArchivePath}.`);
|
|
5389
5417
|
}
|
|
5390
5418
|
|
|
5391
5419
|
logger.info(`Sauvegarde réussie pour l'utilisateur ${user.username}.`);
|
|
5392
|
-
|
|
5393
|
-
// si les sauvegardes sont sur S3 (lister depuis S3, supprimer depuis S3).
|
|
5394
|
-
await manageBackupRotation(user, backupFrequency, s3Config); // Passer s3Config
|
|
5420
|
+
await manageBackupRotation(user, backupFrequency, s3Config);
|
|
5395
5421
|
|
|
5396
|
-
return Promise.resolve();
|
|
5397
5422
|
} catch (error) {
|
|
5398
5423
|
logger.error(`Erreur lors de la sauvegarde pour l'utilisateur ${user.username}:`, error);
|
|
5399
|
-
// Nettoyage
|
|
5400
|
-
const localTempDumpDir = path.join(backupDir, `backup_${user.username}_${Date.now()}_temp`); // Recalculer pour être sûr
|
|
5401
|
-
if (fs.existsSync(localTempDumpDir)) {
|
|
5402
|
-
fs.rmSync(localTempDumpDir, { recursive: true, force: true });
|
|
5403
|
-
}
|
|
5404
|
-
const localArchivePath = path.join(backupDir, `backup_${user.username}_${Date.now()}.tar.gz`); // Recalculer
|
|
5424
|
+
// Nettoyage de l'archive si elle a été créée avant l'erreur
|
|
5405
5425
|
if (fs.existsSync(localArchivePath)) {
|
|
5406
5426
|
fs.unlinkSync(localArchivePath);
|
|
5407
5427
|
}
|
|
5408
|
-
throw error; // Relancer l'erreur
|
|
5428
|
+
throw error; // Relancer l'erreur pour que l'appelant soit informé
|
|
5429
|
+
} finally {
|
|
5430
|
+
// --- NETTOYAGE GARANTI ---
|
|
5431
|
+
// Ce bloc s'exécute toujours, que la sauvegarde réussisse ou échoue.
|
|
5432
|
+
if (fs.existsSync(localTempDumpDir)) {
|
|
5433
|
+
fs.rmSync(localTempDumpDir, { recursive: true, force: true });
|
|
5434
|
+
logger.info(`Répertoire de dump temporaire ${localTempDumpDir} supprimé.`);
|
|
5435
|
+
}
|
|
5409
5436
|
}
|
|
5410
|
-
}
|
|
5411
|
-
|
|
5437
|
+
};
|
|
5412
5438
|
async function manageBackupRotation(user, backupFrequency, s3Config = null) { // Accepter s3Config
|
|
5413
|
-
const userId = user.username;
|
|
5414
|
-
|
|
5439
|
+
const userId = getObjectHash({user:user.username});
|
|
5415
5440
|
let filesToManage = [];
|
|
5416
5441
|
|
|
5417
5442
|
if (s3Config && s3Config.bucketName) {
|
|
5418
|
-
logger.info(`Gestion de la rotation des sauvegardes S3 pour ${
|
|
5443
|
+
logger.info(`Gestion de la rotation des sauvegardes S3 pour ${userid}.`);
|
|
5419
5444
|
const s3Backups = await listS3Backups(s3Config);
|
|
5420
5445
|
// Filtrer pour ne garder que les backups de cet utilisateur et trier
|
|
5421
5446
|
filesToManage = s3Backups
|
|
@@ -86,7 +86,7 @@ afterAll(async () => {
|
|
|
86
86
|
fs.readdirSync(backupDir).forEach(file => {
|
|
87
87
|
fs.unlinkSync(path.join(backupDir, file));
|
|
88
88
|
});
|
|
89
|
-
|
|
89
|
+
fs.rmdirSync(backupDir); // Remove the directory itself
|
|
90
90
|
}
|
|
91
91
|
});
|
|
92
92
|
|
|
@@ -132,5 +132,7 @@ describe('Data Backup and Restore Integration', () => {
|
|
|
132
132
|
expect(docAfterRestore.testField).toBe('Initial Value');
|
|
133
133
|
expect(docAfterRestore.optionalField).toBe(123);
|
|
134
134
|
|
|
135
|
+
console.log("Tests passed for backup and restore");
|
|
136
|
+
|
|
135
137
|
}, 15000); // Timeout augmenté pour les opérations de fichiers
|
|
136
138
|
});
|
|
@@ -1,13 +1,13 @@
|
|
|
1
1
|
// __tests__/data.integration.test.js
|
|
2
2
|
import { ObjectId } from 'mongodb';
|
|
3
|
-
import {expect, describe, it, beforeEach, beforeAll} from 'vitest';
|
|
3
|
+
import {expect, describe, it, beforeEach, beforeAll, afterAll} from 'vitest';
|
|
4
4
|
import { Config } from '../src/config.js';
|
|
5
5
|
|
|
6
6
|
import {
|
|
7
7
|
insertData,
|
|
8
8
|
editData,
|
|
9
9
|
deleteData,
|
|
10
|
-
searchData, installPack
|
|
10
|
+
searchData, installPack, deleteModels, createModel
|
|
11
11
|
} from 'data-primals-engine/modules/data';
|
|
12
12
|
|
|
13
13
|
import {
|
|
@@ -120,11 +120,12 @@ async function setupTestContext() {
|
|
|
120
120
|
};
|
|
121
121
|
}
|
|
122
122
|
|
|
123
|
+
let engine;
|
|
123
124
|
describe('Intégration des fonctions CRUD de données avec validation complète', () => {
|
|
124
125
|
|
|
125
126
|
beforeAll(async () =>{
|
|
126
127
|
Config.Set("modules", ["mongodb", "data", "file", "bucket", "workflow","user", "assistant"]);
|
|
127
|
-
await initEngine();
|
|
128
|
+
engine = await initEngine();
|
|
128
129
|
|
|
129
130
|
})
|
|
130
131
|
|
|
@@ -788,5 +789,103 @@ describe('Intégration des fonctions CRUD de données avec validation complète'
|
|
|
788
789
|
expect(validModel).not.toBeNull();
|
|
789
790
|
});
|
|
790
791
|
});
|
|
792
|
+
// In test/data.integration.test.js
|
|
793
|
+
|
|
794
|
+
describe('relationFilter validation', () => {
|
|
795
|
+
let user;
|
|
796
|
+
let activeProductId, inactiveProductId;
|
|
797
|
+
|
|
798
|
+
// Model names are kept in French to match the database, but variables are in English.
|
|
799
|
+
const productModel = {
|
|
800
|
+
name: 'produitTestFiltre',
|
|
801
|
+
description: '',
|
|
802
|
+
fields: [
|
|
803
|
+
{ name: 'name', type: 'string' },
|
|
804
|
+
{ name: 'actif', type: 'boolean', default: false }
|
|
805
|
+
]
|
|
806
|
+
};
|
|
807
|
+
const orderModel = {
|
|
808
|
+
name: 'commandeTestFiltre',
|
|
809
|
+
description: '',
|
|
810
|
+
fields: [
|
|
811
|
+
{ name: 'ref', type: 'string' },
|
|
812
|
+
{
|
|
813
|
+
name: 'produit',
|
|
814
|
+
type: 'relation',
|
|
815
|
+
relation: 'produitTestFiltre',
|
|
816
|
+
relationFilter: { actif: true } // Only link active products
|
|
817
|
+
}
|
|
818
|
+
]
|
|
819
|
+
};
|
|
820
|
+
|
|
821
|
+
// Set up the context once for all tests in this describe block.
|
|
822
|
+
beforeAll(async () => {
|
|
823
|
+
user = await engine.userProvider.findUserByUsername('demo');
|
|
824
|
+
// Cleanup before starting to ensure a clean state
|
|
825
|
+
await deleteModels({ name: productModel.name, _user: user.username });
|
|
826
|
+
await deleteModels({ name: orderModel.name, _user: user.username });
|
|
827
|
+
await deleteData(productModel.name, [], {}, user);
|
|
828
|
+
await deleteData(orderModel.name, [], {}, user);
|
|
829
|
+
|
|
830
|
+
// Create models
|
|
831
|
+
await createModel({ ...productModel, _user: user.username });
|
|
832
|
+
await createModel({ ...orderModel, _user: user.username });
|
|
833
|
+
|
|
834
|
+
// Create test data
|
|
835
|
+
const activeProduct = await insertData(productModel.name, { name: 'Active Product', actif: true }, {}, user);
|
|
836
|
+
const inactiveProduct = await insertData(productModel.name, { name: 'Inactive Product', actif: false }, {}, user);
|
|
837
|
+
|
|
838
|
+
activeProductId = activeProduct.insertedIds[0];
|
|
839
|
+
inactiveProductId = inactiveProduct.insertedIds[0];
|
|
840
|
+
});
|
|
841
|
+
|
|
842
|
+
// Clean up everything after all tests in this block have run.
|
|
843
|
+
afterAll(async () => {
|
|
844
|
+
await deleteModels({ name: productModel.name, _user: user.username });
|
|
845
|
+
await deleteModels({ name: orderModel.name, _user: user.username });
|
|
846
|
+
await deleteData(productModel.name, [], {}, user);
|
|
847
|
+
await deleteData(orderModel.name, [], {}, user);
|
|
848
|
+
});
|
|
849
|
+
|
|
850
|
+
it('should ALLOW inserting data with a valid relation', async () => {
|
|
851
|
+
const result = await insertData(orderModel.name, { ref: 'CMD-OK', produit: activeProductId }, {}, user);
|
|
852
|
+
expect(result.success).toBe(true);
|
|
853
|
+
expect(result.insertedIds).toHaveLength(1);
|
|
854
|
+
// Cleanup the created order for test isolation
|
|
855
|
+
await deleteData(orderModel.name, result.insertedIds, {}, user);
|
|
856
|
+
});
|
|
857
|
+
|
|
858
|
+
it('should REJECT inserting data with a relation that does not respect the filter', async () => {
|
|
859
|
+
const result = await insertData(orderModel.name, { ref: 'CMD-FAIL', produit: inactiveProductId }, {}, user);
|
|
860
|
+
expect(result.success).toBe(false);
|
|
861
|
+
expect(result.error).toContain('produit');
|
|
862
|
+
});
|
|
863
|
+
|
|
864
|
+
it('should ALLOW updating data with a valid relation', async () => {
|
|
865
|
+
// First, create a valid order
|
|
866
|
+
const initialOrder = await insertData(orderModel.name, { ref: 'CMD-TO-EDIT', produit: activeProductId }, {}, user);
|
|
791
867
|
|
|
868
|
+
// Update it (even with the same value, this tests the code path)
|
|
869
|
+
const result = await editData(orderModel.name, { _id: initialOrder.insertedIds[0] }, { produit: activeProductId }, {}, user);
|
|
870
|
+
|
|
871
|
+
expect(result.success).toBe(true);
|
|
872
|
+
// The hash might not change if only metadata like _updatedAt changes, so modifiedCount can be 0 or 1.
|
|
873
|
+
expect(result.modifiedCount).toBeGreaterThanOrEqual(0);
|
|
874
|
+
|
|
875
|
+
// Cleanup
|
|
876
|
+
await deleteData(orderModel.name, initialOrder.insertedIds, {}, user);
|
|
877
|
+
});
|
|
878
|
+
|
|
879
|
+
it('should REJECT updating data with a relation that does not respect the filter', async () => {
|
|
880
|
+
// First, create a valid order
|
|
881
|
+
const initialOrder = await insertData(orderModel.name, { ref: 'CMD-TO-EDIT-FAIL', produit: activeProductId }, {}, user);
|
|
882
|
+
|
|
883
|
+
// Attempt to link it to an inactive product, expecting it to fail.
|
|
884
|
+
const result = await editData(orderModel.name, { _id: initialOrder.insertedIds[0] }, { produit: inactiveProductId }, {}, user);
|
|
885
|
+
expect(result.success).toBe(false);
|
|
886
|
+
expect(result.error).toContain('produit');
|
|
887
|
+
// Cleanup
|
|
888
|
+
await deleteData(orderModel.name, initialOrder.insertedIds, {}, user);
|
|
889
|
+
});
|
|
890
|
+
});
|
|
792
891
|
});
|