meadow-integration 1.0.1 → 1.0.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CONTRIBUTING.md +50 -0
- package/README.md +223 -7
- package/docs/README.md +107 -7
- package/docs/_sidebar.md +38 -0
- package/docs/_topbar.md +7 -0
- package/docs/cli-reference.md +242 -0
- package/docs/comprehensions.md +98 -0
- package/docs/cover.md +11 -0
- package/docs/css/docuserve.css +73 -0
- package/docs/examples-walkthrough.md +138 -0
- package/docs/index.html +37 -20
- package/docs/integration-adapter.md +109 -0
- package/docs/mapping-files.md +140 -0
- package/docs/programmatic-api.md +173 -0
- package/docs/rest-api-reference.md +731 -0
- package/docs/retold-catalog.json +153 -0
- package/docs/retold-keyword-index.json +4828 -0
- package/examples/Example-001-CSV-Check.sh +29 -0
- package/examples/Example-002-CSV-Transform-Implicit.sh +31 -0
- package/examples/Example-003-CSV-Transform-CLI-Options.sh +39 -0
- package/examples/Example-004-CSV-Transform-Mapping-File.sh +41 -0
- package/examples/Example-005-Multi-Entity-Bookstore.sh +60 -0
- package/examples/Example-006-Multi-CSV-Intersect.sh +74 -0
- package/examples/Example-007-Comprehension-To-Array.sh +41 -0
- package/examples/Example-008-Comprehension-To-CSV.sh +51 -0
- package/examples/Example-009-JSON-Array-Transform.sh +46 -0
- package/examples/Example-010-Programmatic-API.js +138 -0
- package/examples/README.md +44 -0
- package/examples/output/.gitignore +2 -0
- package/package.json +7 -4
- package/source/Meadow-Integration.js +3 -1
- package/source/cli/Meadow-Integration-CLI-Program.js +4 -1
- package/source/cli/commands/Meadow-Integration-Command-ObjectArrayToCSV.js +49 -32
- package/source/cli/commands/Meadow-Integration-Command-Serve.js +51 -0
- package/source/restserver/Meadow-Integration-Server-Endpoints.js +83 -0
- package/source/restserver/Meadow-Integration-Server.js +86 -0
- package/source/restserver/endpoints/Endpoint-CSVCheck.js +91 -0
- package/source/restserver/endpoints/Endpoint-CSVTransform.js +189 -0
- package/source/restserver/endpoints/Endpoint-ComprehensionArray.js +121 -0
- package/source/restserver/endpoints/Endpoint-ComprehensionIntersect.js +166 -0
- package/source/restserver/endpoints/Endpoint-ComprehensionPush.js +209 -0
- package/source/restserver/endpoints/Endpoint-EntityFromTabularFolder.js +252 -0
- package/source/restserver/endpoints/Endpoint-JSONArrayTransform.js +238 -0
- package/source/restserver/endpoints/Endpoint-ObjectArrayToCSV.js +231 -0
- package/source/restserver/endpoints/Endpoint-TSVCheck.js +93 -0
- package/source/restserver/endpoints/Endpoint-TSVTransform.js +191 -0
- package/test/Meadow-Integration-Server_test.js +1170 -0
- package/test/data/test-comprehension-secondary.json +8 -0
- package/test/data/test-comprehension.json +8 -0
- package/test/data/test-small.csv +6 -0
- package/test/data/test-small.json +7 -0
- package/test/data/test-small.tsv +6 -0
|
@@ -0,0 +1,731 @@
|
|
|
1
|
+
# REST API Reference
|
|
2
|
+
|
|
3
|
+
The Meadow Integration Server exposes all CLI functionality as REST endpoints. Start it with:
|
|
4
|
+
|
|
5
|
+
```shell
|
|
6
|
+
# Via CLI command
|
|
7
|
+
npx meadow-integration serve
|
|
8
|
+
npx meadow-integration serve -p 3000
|
|
9
|
+
|
|
10
|
+
# Or from the repository
|
|
11
|
+
npm start -- serve
|
|
12
|
+
npm start -- serve -p 3000
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
The server defaults to port `8086`. The `MEADOW_INTEGRATION_PORT` environment variable is also respected.
|
|
16
|
+
|
|
17
|
+
All data endpoints accept `POST` with a JSON body (`Content-Type: application/json`).
|
|
18
|
+
|
|
19
|
+
The example `curl` commands below assume the server is running at `http://localhost:8086` and that file paths reference the example data shipped with this module (under `docs/examples/data/`).
|
|
20
|
+
|
|
21
|
+
---
|
|
22
|
+
|
|
23
|
+
## GET /1.0/Status
|
|
24
|
+
|
|
25
|
+
Server health check and endpoint listing.
|
|
26
|
+
|
|
27
|
+
```shell
|
|
28
|
+
curl http://localhost:8086/1.0/Status
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
**Response:**
|
|
32
|
+
|
|
33
|
+
```json
|
|
34
|
+
{
|
|
35
|
+
"Product": "Meadow-Integration-Server",
|
|
36
|
+
"Version": "1.0.2",
|
|
37
|
+
"Status": "Running",
|
|
38
|
+
"Endpoints": [
|
|
39
|
+
"POST /1.0/CSV/Check",
|
|
40
|
+
"POST /1.0/CSV/Transform",
|
|
41
|
+
"POST /1.0/TSV/Check",
|
|
42
|
+
"POST /1.0/TSV/Transform",
|
|
43
|
+
"POST /1.0/JSONArray/Transform",
|
|
44
|
+
"POST /1.0/JSONArray/TransformRecords",
|
|
45
|
+
"POST /1.0/Comprehension/Intersect",
|
|
46
|
+
"POST /1.0/Comprehension/IntersectFiles",
|
|
47
|
+
"POST /1.0/Comprehension/ToArray",
|
|
48
|
+
"POST /1.0/Comprehension/ToArrayFromFile",
|
|
49
|
+
"POST /1.0/Comprehension/ToCSV",
|
|
50
|
+
"POST /1.0/Comprehension/ToCSVFromFile",
|
|
51
|
+
"POST /1.0/Comprehension/Push",
|
|
52
|
+
"POST /1.0/Comprehension/PushFile",
|
|
53
|
+
"POST /1.0/Entity/FromTabularFolder"
|
|
54
|
+
]
|
|
55
|
+
}
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
---
|
|
59
|
+
|
|
60
|
+
## CSV Operations
|
|
61
|
+
|
|
62
|
+
### POST /1.0/CSV/Check
|
|
63
|
+
|
|
64
|
+
Analyze a CSV file for column statistics. Equivalent to CLI: `csvcheck <file>`.
|
|
65
|
+
|
|
66
|
+
**Request Body:**
|
|
67
|
+
|
|
68
|
+
| Field | Type | Required | Description |
|
|
69
|
+
|-------|------|----------|-------------|
|
|
70
|
+
| `File` | string | yes | Absolute path to the CSV file |
|
|
71
|
+
| `Records` | boolean | no | Include all parsed records in the output (default: `false`) |
|
|
72
|
+
| `QuoteDelimiter` | string | no | Quote character (default: `"`) |
|
|
73
|
+
|
|
74
|
+
**Example** *(corresponds to [Example 001](examples-walkthrough.md))*:
|
|
75
|
+
|
|
76
|
+
```shell
|
|
77
|
+
curl -X POST http://localhost:8086/1.0/CSV/Check \
|
|
78
|
+
-H "Content-Type: application/json" \
|
|
79
|
+
-d '{
|
|
80
|
+
"File": "/path/to/docs/examples/data/books.csv"
|
|
81
|
+
}'
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
**Response:**
|
|
85
|
+
|
|
86
|
+
```json
|
|
87
|
+
{
|
|
88
|
+
"DataSet": "/path/to/docs/examples/data/books.csv",
|
|
89
|
+
"FirstRow": { "id": "1", "book_id": "2767052", "title": "The Hunger Games (The Hunger Games, #1)", "..." : "..." },
|
|
90
|
+
"RowCount": 9999,
|
|
91
|
+
"LastRow": { "..." : "..." },
|
|
92
|
+
"Headers": ["id", "book_id", "best_book_id", "work_id", "books_count", "isbn", "isbn13", "authors", "original_publication_year", "original_title", "title", "language_code", "..."],
|
|
93
|
+
"ColumnCount": 23,
|
|
94
|
+
"ColumnStatistics": {
|
|
95
|
+
"id": { "Count": 9999, "EmptyCount": 0, "NumericCount": 9999, "FirstValue": "1", "LastValue": "10000" },
|
|
96
|
+
"title": { "Count": 9999, "EmptyCount": 0, "NumericCount": 0, "FirstValue": "The Hunger Games (The Hunger Games, #1)", "LastValue": "..." },
|
|
97
|
+
"language_code": { "Count": 9999, "EmptyCount": 1084, "NumericCount": 0, "FirstValue": "eng", "LastValue": "eng" }
|
|
98
|
+
},
|
|
99
|
+
"Records": null
|
|
100
|
+
}
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
---
|
|
104
|
+
|
|
105
|
+
### POST /1.0/CSV/Transform
|
|
106
|
+
|
|
107
|
+
Transform a CSV file into a comprehension. Equivalent to CLI: `csvtransform <file>`.
|
|
108
|
+
|
|
109
|
+
**Request Body:**
|
|
110
|
+
|
|
111
|
+
| Field | Type | Required | Description |
|
|
112
|
+
|-------|------|----------|-------------|
|
|
113
|
+
| `File` | string | yes | Absolute path to the CSV file |
|
|
114
|
+
| `Entity` | string | no | Entity name (auto-detected from filename if omitted) |
|
|
115
|
+
| `GUIDName` | string | no | GUID column name (default: `GUID{Entity}`) |
|
|
116
|
+
| `GUIDTemplate` | string | no | Pict template for GUID generation |
|
|
117
|
+
| `Mappings` | object | no | Column mappings: `{ "OutputCol": "{~D:Record.inputCol~}" }` |
|
|
118
|
+
| `MappingConfiguration` | object | no | Full mapping config (same format as a mapping JSON file) |
|
|
119
|
+
| `IncomingComprehension` | object | no | Existing comprehension to merge new records into |
|
|
120
|
+
| `Extended` | boolean | no | Return full operation state instead of just the comprehension |
|
|
121
|
+
| `QuoteDelimiter` | string | no | Quote character (default: `"`) |
|
|
122
|
+
|
|
123
|
+
**Example -- implicit transform** *(corresponds to [Example 002](examples-walkthrough.md))*:
|
|
124
|
+
|
|
125
|
+
```shell
|
|
126
|
+
curl -X POST http://localhost:8086/1.0/CSV/Transform \
|
|
127
|
+
-H "Content-Type: application/json" \
|
|
128
|
+
-d '{
|
|
129
|
+
"File": "/path/to/docs/examples/data/books.csv"
|
|
130
|
+
}'
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
Entity name, GUID template, and column mappings are auto-detected from the filename and first row.
|
|
134
|
+
|
|
135
|
+
**Example -- with entity and GUID options** *(corresponds to [Example 003](examples-walkthrough.md))*:
|
|
136
|
+
|
|
137
|
+
```shell
|
|
138
|
+
curl -X POST http://localhost:8086/1.0/CSV/Transform \
|
|
139
|
+
-H "Content-Type: application/json" \
|
|
140
|
+
-d '{
|
|
141
|
+
"File": "/path/to/docs/examples/data/books.csv",
|
|
142
|
+
"Entity": "Book",
|
|
143
|
+
"GUIDName": "GUIDBook",
|
|
144
|
+
"GUIDTemplate": "Book_{~D:Record.id~}"
|
|
145
|
+
}'
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
**Example -- with a mapping configuration** *(corresponds to [Example 004](examples-walkthrough.md))*:
|
|
149
|
+
|
|
150
|
+
```shell
|
|
151
|
+
curl -X POST http://localhost:8086/1.0/CSV/Transform \
|
|
152
|
+
-H "Content-Type: application/json" \
|
|
153
|
+
-d '{
|
|
154
|
+
"File": "/path/to/docs/examples/data/books.csv",
|
|
155
|
+
"MappingConfiguration": {
|
|
156
|
+
"Entity": "Book",
|
|
157
|
+
"GUIDTemplate": "Book_{~D:Record.id~}",
|
|
158
|
+
"Mappings": {
|
|
159
|
+
"Title": "{~D:Record.title~}",
|
|
160
|
+
"Language": "{~D:Record.language_code~}",
|
|
161
|
+
"PublicationYear": "{~D:Fable.Math.roundPrecise(Record.original_publication_year,0)~}",
|
|
162
|
+
"ISBN": "{~D:Record.isbn~}",
|
|
163
|
+
"Genre": "Unknown",
|
|
164
|
+
"Type": "Book",
|
|
165
|
+
"ImageURL": "{~D:Record.image_url~}"
|
|
166
|
+
}
|
|
167
|
+
}
|
|
168
|
+
}'
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
**Example -- multi-entity bookstore** *(corresponds to [Example 005](examples-walkthrough.md))*:
|
|
172
|
+
|
|
173
|
+
Step 1: Create Book records.
|
|
174
|
+
|
|
175
|
+
```shell
|
|
176
|
+
curl -X POST http://localhost:8086/1.0/CSV/Transform \
|
|
177
|
+
-H "Content-Type: application/json" \
|
|
178
|
+
-d '{
|
|
179
|
+
"File": "/path/to/docs/examples/data/books.csv",
|
|
180
|
+
"MappingConfiguration": {
|
|
181
|
+
"Entity": "Book",
|
|
182
|
+
"GUIDTemplate": "Book_{~D:Record.id~}",
|
|
183
|
+
"Mappings": {
|
|
184
|
+
"Title": "{~D:Record.title~}",
|
|
185
|
+
"Language": "{~D:Record.language_code~}",
|
|
186
|
+
"ISBN": "{~D:Record.isbn~}"
|
|
187
|
+
}
|
|
188
|
+
}
|
|
189
|
+
}'
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
Step 2: Take the response from step 1 and pass it as `IncomingComprehension` to add Author records from the same CSV.
|
|
193
|
+
|
|
194
|
+
```shell
|
|
195
|
+
curl -X POST http://localhost:8086/1.0/CSV/Transform \
|
|
196
|
+
-H "Content-Type: application/json" \
|
|
197
|
+
-d '{
|
|
198
|
+
"File": "/path/to/docs/examples/data/books.csv",
|
|
199
|
+
"MappingConfiguration": {
|
|
200
|
+
"Entity": "Author",
|
|
201
|
+
"MultipleGUIDUniqueness": true,
|
|
202
|
+
"Solvers": ["NewRecordsGUIDUniqueness = STRINGGETSEGMENTS(IncomingRecord.authors,\",\")"],
|
|
203
|
+
"GUIDTemplate": "Author_{~PascalCaseIdentifier:Record._GUIDUniqueness~}",
|
|
204
|
+
"Mappings": {
|
|
205
|
+
"Name": "{~D:Record._GUIDUniqueness~}"
|
|
206
|
+
}
|
|
207
|
+
},
|
|
208
|
+
"IncomingComprehension": { "Book": { "...step 1 results..." : {} } }
|
|
209
|
+
}'
|
|
210
|
+
```
|
|
211
|
+
|
|
212
|
+
**Response** (single-entity example):
|
|
213
|
+
|
|
214
|
+
```json
|
|
215
|
+
{
|
|
216
|
+
"Book": {
|
|
217
|
+
"Book_1": {
|
|
218
|
+
"GUIDBook": "Book_1",
|
|
219
|
+
"Title": "The Hunger Games (The Hunger Games, #1)",
|
|
220
|
+
"Language": "eng",
|
|
221
|
+
"ISBN": "439023483"
|
|
222
|
+
},
|
|
223
|
+
"Book_2": {
|
|
224
|
+
"GUIDBook": "Book_2",
|
|
225
|
+
"Title": "Harry Potter and the Sorcerer's Stone (Harry Potter, #1)",
|
|
226
|
+
"Language": "eng",
|
|
227
|
+
"ISBN": "439554934"
|
|
228
|
+
}
|
|
229
|
+
}
|
|
230
|
+
}
|
|
231
|
+
```
|
|
232
|
+
|
|
233
|
+
---
|
|
234
|
+
|
|
235
|
+
## TSV Operations
|
|
236
|
+
|
|
237
|
+
### POST /1.0/TSV/Check
|
|
238
|
+
|
|
239
|
+
Analyze a TSV file for column statistics. Equivalent to CLI: `tsvcheck <file>`.
|
|
240
|
+
|
|
241
|
+
**Request Body:**
|
|
242
|
+
|
|
243
|
+
Same fields as `/1.0/CSV/Check`. The delimiter is automatically set to tab.
|
|
244
|
+
|
|
245
|
+
**Example:**
|
|
246
|
+
|
|
247
|
+
```shell
|
|
248
|
+
curl -X POST http://localhost:8086/1.0/TSV/Check \
|
|
249
|
+
-H "Content-Type: application/json" \
|
|
250
|
+
-d '{
|
|
251
|
+
"File": "/path/to/airports.tsv"
|
|
252
|
+
}'
|
|
253
|
+
```
|
|
254
|
+
|
|
255
|
+
---
|
|
256
|
+
|
|
257
|
+
### POST /1.0/TSV/Transform
|
|
258
|
+
|
|
259
|
+
Transform a TSV file into a comprehension. Equivalent to CLI: `tsvtransform <file>`.
|
|
260
|
+
|
|
261
|
+
**Request Body:**
|
|
262
|
+
|
|
263
|
+
Same fields as `/1.0/CSV/Transform`. The delimiter is automatically set to tab.
|
|
264
|
+
|
|
265
|
+
**Example** *(matches the TSV airport mapping in the [Mapping Files](mapping-files.md) docs)*:
|
|
266
|
+
|
|
267
|
+
```shell
|
|
268
|
+
curl -X POST http://localhost:8086/1.0/TSV/Transform \
|
|
269
|
+
-H "Content-Type: application/json" \
|
|
270
|
+
-d '{
|
|
271
|
+
"File": "/path/to/airports.tsv",
|
|
272
|
+
"MappingConfiguration": {
|
|
273
|
+
"Entity": "Airport",
|
|
274
|
+
"GUIDTemplate": "Airport-{~D:iata~}",
|
|
275
|
+
"Mappings": {
|
|
276
|
+
"Code": "{~D:iata~}",
|
|
277
|
+
"Name": "{~D:name~}",
|
|
278
|
+
"City": "{~D:city~}",
|
|
279
|
+
"State": "{~D:state~}",
|
|
280
|
+
"Country": "{~D:country~}",
|
|
281
|
+
"Latitude": "{~D:lat~}",
|
|
282
|
+
"Longitude": "{~D:long~}"
|
|
283
|
+
}
|
|
284
|
+
}
|
|
285
|
+
}'
|
|
286
|
+
```
|
|
287
|
+
|
|
288
|
+
---
|
|
289
|
+
|
|
290
|
+
## JSON Array Operations
|
|
291
|
+
|
|
292
|
+
### POST /1.0/JSONArray/Transform
|
|
293
|
+
|
|
294
|
+
Transform a JSON array file into a comprehension. Equivalent to CLI: `jsonarraytransform <file>`.
|
|
295
|
+
|
|
296
|
+
**Request Body:**
|
|
297
|
+
|
|
298
|
+
Same fields as `/1.0/CSV/Transform`.
|
|
299
|
+
|
|
300
|
+
**Example** *(corresponds to [Example 009](examples-walkthrough.md))*:
|
|
301
|
+
|
|
302
|
+
```shell
|
|
303
|
+
curl -X POST http://localhost:8086/1.0/JSONArray/Transform \
|
|
304
|
+
-H "Content-Type: application/json" \
|
|
305
|
+
-d '{
|
|
306
|
+
"File": "/path/to/books-array.json",
|
|
307
|
+
"MappingConfiguration": {
|
|
308
|
+
"Entity": "Book",
|
|
309
|
+
"GUIDTemplate": "Book_{~D:Record.id~}",
|
|
310
|
+
"Mappings": {
|
|
311
|
+
"Title": "{~D:Record.title~}",
|
|
312
|
+
"Language": "{~D:Record.language_code~}",
|
|
313
|
+
"ISBN": "{~D:Record.isbn~}"
|
|
314
|
+
}
|
|
315
|
+
}
|
|
316
|
+
}'
|
|
317
|
+
```
|
|
318
|
+
|
|
319
|
+
---
|
|
320
|
+
|
|
321
|
+
### POST /1.0/JSONArray/TransformRecords
|
|
322
|
+
|
|
323
|
+
Transform an in-memory JSON array into a comprehension. No file needed -- records are passed directly in the request body.
|
|
324
|
+
|
|
325
|
+
**Request Body:**
|
|
326
|
+
|
|
327
|
+
| Field | Type | Required | Description |
|
|
328
|
+
|-------|------|----------|-------------|
|
|
329
|
+
| `Records` | array | yes | Array of record objects |
|
|
330
|
+
| `Entity` | string | no | Entity name |
|
|
331
|
+
| `GUIDName` | string | no | GUID column name |
|
|
332
|
+
| `GUIDTemplate` | string | no | Pict template for GUID generation |
|
|
333
|
+
| `Mappings` | object | no | Column mappings |
|
|
334
|
+
| `MappingConfiguration` | object | no | Full mapping config |
|
|
335
|
+
| `IncomingComprehension` | object | no | Existing comprehension to merge into |
|
|
336
|
+
| `Extended` | boolean | no | Return full operation state |
|
|
337
|
+
|
|
338
|
+
**Example:**
|
|
339
|
+
|
|
340
|
+
```shell
|
|
341
|
+
curl -X POST http://localhost:8086/1.0/JSONArray/TransformRecords \
|
|
342
|
+
-H "Content-Type: application/json" \
|
|
343
|
+
-d '{
|
|
344
|
+
"Records": [
|
|
345
|
+
{ "id": "1", "title": "The Hunger Games", "language_code": "eng", "isbn": "439023483" },
|
|
346
|
+
{ "id": "2", "title": "Harry Potter and the Sorcerers Stone", "language_code": "eng", "isbn": "439554934" },
|
|
347
|
+
{ "id": "3", "title": "Twilight", "language_code": "eng", "isbn": "316015849" }
|
|
348
|
+
],
|
|
349
|
+
"Entity": "Book",
|
|
350
|
+
"GUIDTemplate": "Book_{~D:Record.id~}",
|
|
351
|
+
"Mappings": {
|
|
352
|
+
"Title": "{~D:Record.title~}",
|
|
353
|
+
"Language": "{~D:Record.language_code~}",
|
|
354
|
+
"ISBN": "{~D:Record.isbn~}"
|
|
355
|
+
}
|
|
356
|
+
}'
|
|
357
|
+
```
|
|
358
|
+
|
|
359
|
+
**Response:**
|
|
360
|
+
|
|
361
|
+
```json
|
|
362
|
+
{
|
|
363
|
+
"Book": {
|
|
364
|
+
"Book_1": { "GUIDBook": "Book_1", "Title": "The Hunger Games", "Language": "eng", "ISBN": "439023483" },
|
|
365
|
+
"Book_2": { "GUIDBook": "Book_2", "Title": "Harry Potter and the Sorcerers Stone", "Language": "eng", "ISBN": "439554934" },
|
|
366
|
+
"Book_3": { "GUIDBook": "Book_3", "Title": "Twilight", "Language": "eng", "ISBN": "316015849" }
|
|
367
|
+
}
|
|
368
|
+
}
|
|
369
|
+
```
|
|
370
|
+
|
|
371
|
+
---
|
|
372
|
+
|
|
373
|
+
## Comprehension Operations
|
|
374
|
+
|
|
375
|
+
### POST /1.0/Comprehension/Intersect
|
|
376
|
+
|
|
377
|
+
Merge two comprehension objects. Equivalent to CLI: `comprehensionintersect`.
|
|
378
|
+
|
|
379
|
+
Records from the secondary comprehension are merged into the primary. Matching GUIDs are overwritten by the secondary values.
|
|
380
|
+
|
|
381
|
+
**Request Body:**
|
|
382
|
+
|
|
383
|
+
| Field | Type | Required | Description |
|
|
384
|
+
|-------|------|----------|-------------|
|
|
385
|
+
| `PrimaryComprehension` | object | yes | The base comprehension |
|
|
386
|
+
| `SecondaryComprehension` | object | yes | Comprehension to merge in |
|
|
387
|
+
| `Entity` | string | no | Entity to merge (auto-detected if omitted) |
|
|
388
|
+
|
|
389
|
+
**Example** *(corresponds to [Example 006](examples-walkthrough.md) -- merging Seattle neighborhood data)*:
|
|
390
|
+
|
|
391
|
+
```shell
|
|
392
|
+
curl -X POST http://localhost:8086/1.0/Comprehension/Intersect \
|
|
393
|
+
-H "Content-Type: application/json" \
|
|
394
|
+
-d '{
|
|
395
|
+
"PrimaryComprehension": {
|
|
396
|
+
"Neighborhood": {
|
|
397
|
+
"Ballard": { "GUIDNeighborhood": "Ballard", "HousingUnits": "15234" },
|
|
398
|
+
"Capitol Hill": { "GUIDNeighborhood": "Capitol Hill", "HousingUnits": "18721" }
|
|
399
|
+
}
|
|
400
|
+
},
|
|
401
|
+
"SecondaryComprehension": {
|
|
402
|
+
"Neighborhood": {
|
|
403
|
+
"Ballard": { "GUIDNeighborhood": "Ballard", "MedianRent": "1850" },
|
|
404
|
+
"Capitol Hill": { "GUIDNeighborhood": "Capitol Hill", "MedianRent": "1650" }
|
|
405
|
+
}
|
|
406
|
+
},
|
|
407
|
+
"Entity": "Neighborhood"
|
|
408
|
+
}'
|
|
409
|
+
```
|
|
410
|
+
|
|
411
|
+
**Response:**
|
|
412
|
+
|
|
413
|
+
```json
|
|
414
|
+
{
|
|
415
|
+
"Neighborhood": {
|
|
416
|
+
"Ballard": { "GUIDNeighborhood": "Ballard", "HousingUnits": "15234", "MedianRent": "1850" },
|
|
417
|
+
"Capitol Hill": { "GUIDNeighborhood": "Capitol Hill", "HousingUnits": "18721", "MedianRent": "1650" }
|
|
418
|
+
}
|
|
419
|
+
}
|
|
420
|
+
```
|
|
421
|
+
|
|
422
|
+
---
|
|
423
|
+
|
|
424
|
+
### POST /1.0/Comprehension/IntersectFiles
|
|
425
|
+
|
|
426
|
+
Merge two comprehension JSON files. File-based version of the above.
|
|
427
|
+
|
|
428
|
+
**Request Body:**
|
|
429
|
+
|
|
430
|
+
| Field | Type | Required | Description |
|
|
431
|
+
|-------|------|----------|-------------|
|
|
432
|
+
| `File` | string | yes | Absolute path to the primary comprehension file |
|
|
433
|
+
| `IntersectFile` | string | yes | Absolute path to the secondary comprehension file |
|
|
434
|
+
| `Entity` | string | no | Entity to merge |
|
|
435
|
+
|
|
436
|
+
**Example:**
|
|
437
|
+
|
|
438
|
+
```shell
|
|
439
|
+
curl -X POST http://localhost:8086/1.0/Comprehension/IntersectFiles \
|
|
440
|
+
-H "Content-Type: application/json" \
|
|
441
|
+
-d '{
|
|
442
|
+
"File": "/path/to/seattle-housing-chars.json",
|
|
443
|
+
"IntersectFile": "/path/to/seattle-housing-costs.json",
|
|
444
|
+
"Entity": "Neighborhood"
|
|
445
|
+
}'
|
|
446
|
+
```
|
|
447
|
+
|
|
448
|
+
---
|
|
449
|
+
|
|
450
|
+
### POST /1.0/Comprehension/ToArray
|
|
451
|
+
|
|
452
|
+
Convert an object-keyed comprehension into a JSON array. Equivalent to CLI: `comprehensionarray`.
|
|
453
|
+
|
|
454
|
+
**Request Body:**
|
|
455
|
+
|
|
456
|
+
| Field | Type | Required | Description |
|
|
457
|
+
|-------|------|----------|-------------|
|
|
458
|
+
| `Comprehension` | object | yes | The comprehension object |
|
|
459
|
+
| `Entity` | string | no | Entity to extract (auto-detected if omitted) |
|
|
460
|
+
|
|
461
|
+
**Example** *(corresponds to [Example 007](examples-walkthrough.md))*:
|
|
462
|
+
|
|
463
|
+
```shell
|
|
464
|
+
curl -X POST http://localhost:8086/1.0/Comprehension/ToArray \
|
|
465
|
+
-H "Content-Type: application/json" \
|
|
466
|
+
-d '{
|
|
467
|
+
"Comprehension": {
|
|
468
|
+
"Book": {
|
|
469
|
+
"Book_1": { "GUIDBook": "Book_1", "Title": "The Hunger Games", "Language": "eng" },
|
|
470
|
+
"Book_2": { "GUIDBook": "Book_2", "Title": "Harry Potter", "Language": "eng" }
|
|
471
|
+
}
|
|
472
|
+
},
|
|
473
|
+
"Entity": "Book"
|
|
474
|
+
}'
|
|
475
|
+
```
|
|
476
|
+
|
|
477
|
+
**Response:**
|
|
478
|
+
|
|
479
|
+
```json
|
|
480
|
+
[
|
|
481
|
+
{ "GUIDBook": "Book_1", "Title": "The Hunger Games", "Language": "eng" },
|
|
482
|
+
{ "GUIDBook": "Book_2", "Title": "Harry Potter", "Language": "eng" }
|
|
483
|
+
]
|
|
484
|
+
```
|
|
485
|
+
|
|
486
|
+
---
|
|
487
|
+
|
|
488
|
+
### POST /1.0/Comprehension/ToArrayFromFile
|
|
489
|
+
|
|
490
|
+
File-based version of the above.
|
|
491
|
+
|
|
492
|
+
**Request Body:**
|
|
493
|
+
|
|
494
|
+
| Field | Type | Required | Description |
|
|
495
|
+
|-------|------|----------|-------------|
|
|
496
|
+
| `File` | string | yes | Absolute path to the comprehension file |
|
|
497
|
+
| `Entity` | string | no | Entity to extract |
|
|
498
|
+
|
|
499
|
+
**Example:**
|
|
500
|
+
|
|
501
|
+
```shell
|
|
502
|
+
curl -X POST http://localhost:8086/1.0/Comprehension/ToArrayFromFile \
|
|
503
|
+
-H "Content-Type: application/json" \
|
|
504
|
+
-d '{
|
|
505
|
+
"File": "/path/to/books-comprehension.json",
|
|
506
|
+
"Entity": "Book"
|
|
507
|
+
}'
|
|
508
|
+
```
|
|
509
|
+
|
|
510
|
+
---
|
|
511
|
+
|
|
512
|
+
### POST /1.0/Comprehension/ToCSV
|
|
513
|
+
|
|
514
|
+
Convert a comprehension or array of objects to CSV. Equivalent to CLI: `objectarraytocsv`.
|
|
515
|
+
|
|
516
|
+
Returns `text/csv` content. Nested objects are flattened using dot notation (e.g. `address.city`).
|
|
517
|
+
|
|
518
|
+
**Request Body (array form):**
|
|
519
|
+
|
|
520
|
+
| Field | Type | Required | Description |
|
|
521
|
+
|-------|------|----------|-------------|
|
|
522
|
+
| `Records` | array | yes* | Array of record objects |
|
|
523
|
+
|
|
524
|
+
**Request Body (comprehension form):**
|
|
525
|
+
|
|
526
|
+
| Field | Type | Required | Description |
|
|
527
|
+
|-------|------|----------|-------------|
|
|
528
|
+
| `Comprehension` | object | yes* | Comprehension object |
|
|
529
|
+
| `Entity` | string | no | Entity to extract (auto-detected for single-entity comprehensions) |
|
|
530
|
+
|
|
531
|
+
*Provide either `Records` or `Comprehension`.
|
|
532
|
+
|
|
533
|
+
**Example -- from array** *(corresponds to [Example 008](examples-walkthrough.md))*:
|
|
534
|
+
|
|
535
|
+
```shell
|
|
536
|
+
curl -X POST http://localhost:8086/1.0/Comprehension/ToCSV \
|
|
537
|
+
-H "Content-Type: application/json" \
|
|
538
|
+
-d '{
|
|
539
|
+
"Records": [
|
|
540
|
+
{ "GUIDBook": "Book_1", "Title": "The Hunger Games", "Language": "eng", "ISBN": "439023483" },
|
|
541
|
+
{ "GUIDBook": "Book_2", "Title": "Harry Potter", "Language": "eng", "ISBN": "439554934" }
|
|
542
|
+
]
|
|
543
|
+
}'
|
|
544
|
+
```
|
|
545
|
+
|
|
546
|
+
**Response** (`Content-Type: text/csv`):
|
|
547
|
+
|
|
548
|
+
```csv
|
|
549
|
+
GUIDBook,ISBN,Language,Title
|
|
550
|
+
Book_1,439023483,eng,The Hunger Games
|
|
551
|
+
Book_2,439554934,eng,Harry Potter
|
|
552
|
+
```
|
|
553
|
+
|
|
554
|
+
**Example -- from comprehension:**
|
|
555
|
+
|
|
556
|
+
```shell
|
|
557
|
+
curl -X POST http://localhost:8086/1.0/Comprehension/ToCSV \
|
|
558
|
+
-H "Content-Type: application/json" \
|
|
559
|
+
-d '{
|
|
560
|
+
"Comprehension": {
|
|
561
|
+
"Book": {
|
|
562
|
+
"Book_1": { "GUIDBook": "Book_1", "Title": "The Hunger Games", "Language": "eng" },
|
|
563
|
+
"Book_2": { "GUIDBook": "Book_2", "Title": "Harry Potter", "Language": "eng" }
|
|
564
|
+
}
|
|
565
|
+
},
|
|
566
|
+
"Entity": "Book"
|
|
567
|
+
}'
|
|
568
|
+
```
|
|
569
|
+
|
|
570
|
+
---
|
|
571
|
+
|
|
572
|
+
### POST /1.0/Comprehension/ToCSVFromFile
|
|
573
|
+
|
|
574
|
+
File-based version of the above.
|
|
575
|
+
|
|
576
|
+
**Request Body:**
|
|
577
|
+
|
|
578
|
+
| Field | Type | Required | Description |
|
|
579
|
+
|-------|------|----------|-------------|
|
|
580
|
+
| `File` | string | yes | Absolute path to a JSON file (array or comprehension) |
|
|
581
|
+
| `Entity` | string | no | Entity to extract |
|
|
582
|
+
|
|
583
|
+
**Example:**
|
|
584
|
+
|
|
585
|
+
```shell
|
|
586
|
+
curl -X POST http://localhost:8086/1.0/Comprehension/ToCSVFromFile \
|
|
587
|
+
-H "Content-Type: application/json" \
|
|
588
|
+
-d '{
|
|
589
|
+
"File": "/path/to/books-array.json"
|
|
590
|
+
}'
|
|
591
|
+
```
|
|
592
|
+
|
|
593
|
+
---
|
|
594
|
+
|
|
595
|
+
### POST /1.0/Comprehension/Push
|
|
596
|
+
|
|
597
|
+
Push a comprehension to Meadow REST APIs via the Integration Adapter. Equivalent to CLI: `load_comprehension`.
|
|
598
|
+
|
|
599
|
+
Each entity in the comprehension gets its own adapter. Records are marshaled to Meadow format and upserted to the target server.
|
|
600
|
+
|
|
601
|
+
**Request Body:**
|
|
602
|
+
|
|
603
|
+
| Field | Type | Required | Description |
|
|
604
|
+
|-------|------|----------|-------------|
|
|
605
|
+
| `Comprehension` | object | yes | The comprehension to push |
|
|
606
|
+
| `GUIDPrefix` | string | no | GUID marshal prefix (default: `INTG-DEF`) |
|
|
607
|
+
| `EntityGUIDPrefix` | string | no | Per-entity GUID prefix |
|
|
608
|
+
| `ServerURL` | string | no | Target Meadow API URL (default: `http://localhost:8086/1.0/`) |
|
|
609
|
+
|
|
610
|
+
**Example:**
|
|
611
|
+
|
|
612
|
+
```shell
|
|
613
|
+
curl -X POST http://localhost:8086/1.0/Comprehension/Push \
|
|
614
|
+
-H "Content-Type: application/json" \
|
|
615
|
+
-d '{
|
|
616
|
+
"Comprehension": {
|
|
617
|
+
"Book": {
|
|
618
|
+
"Book_1": { "GUIDBook": "Book_1", "Title": "The Hunger Games", "Language": "eng" },
|
|
619
|
+
"Book_2": { "GUIDBook": "Book_2", "Title": "Harry Potter", "Language": "eng" }
|
|
620
|
+
}
|
|
621
|
+
},
|
|
622
|
+
"GUIDPrefix": "IMPORT-2024",
|
|
623
|
+
"EntityGUIDPrefix": "BK",
|
|
624
|
+
"ServerURL": "http://my-meadow-server:8080/1.0/"
|
|
625
|
+
}'
|
|
626
|
+
```
|
|
627
|
+
|
|
628
|
+
**Response:**
|
|
629
|
+
|
|
630
|
+
```json
|
|
631
|
+
{
|
|
632
|
+
"Success": true,
|
|
633
|
+
"EntitiesPushed": ["Book"],
|
|
634
|
+
"Message": "Pushed comprehension for 1 entity(ies)."
|
|
635
|
+
}
|
|
636
|
+
```
|
|
637
|
+
|
|
638
|
+
---
|
|
639
|
+
|
|
640
|
+
### POST /1.0/Comprehension/PushFile
|
|
641
|
+
|
|
642
|
+
File-based version of the above.
|
|
643
|
+
|
|
644
|
+
**Request Body:**
|
|
645
|
+
|
|
646
|
+
| Field | Type | Required | Description |
|
|
647
|
+
|-------|------|----------|-------------|
|
|
648
|
+
| `File` | string | yes | Absolute path to the comprehension file |
|
|
649
|
+
| `GUIDPrefix` | string | no | GUID marshal prefix |
|
|
650
|
+
| `EntityGUIDPrefix` | string | no | Per-entity GUID prefix |
|
|
651
|
+
| `ServerURL` | string | no | Target Meadow API URL |
|
|
652
|
+
|
|
653
|
+
**Example:**
|
|
654
|
+
|
|
655
|
+
```shell
|
|
656
|
+
curl -X POST http://localhost:8086/1.0/Comprehension/PushFile \
|
|
657
|
+
-H "Content-Type: application/json" \
|
|
658
|
+
-d '{
|
|
659
|
+
"File": "/path/to/bookstore-comprehension.json",
|
|
660
|
+
"GUIDPrefix": "IMPORT-2024",
|
|
661
|
+
"ServerURL": "http://my-meadow-server:8080/1.0/"
|
|
662
|
+
}'
|
|
663
|
+
```
|
|
664
|
+
|
|
665
|
+
---
|
|
666
|
+
|
|
667
|
+
## Entity Generation
|
|
668
|
+
|
|
669
|
+
### POST /1.0/Entity/FromTabularFolder
|
|
670
|
+
|
|
671
|
+
Generate comprehensions from all tabular files (CSV, TSV, JSON) in a folder. Equivalent to CLI: `entitycomprehensionsfromtabularfolders`.
|
|
672
|
+
|
|
673
|
+
Each file produces its own entity (inferred from filename) unless `Entity` is specified to force all files into one.
|
|
674
|
+
|
|
675
|
+
**Request Body:**
|
|
676
|
+
|
|
677
|
+
| Field | Type | Required | Description |
|
|
678
|
+
|-------|------|----------|-------------|
|
|
679
|
+
| `Folder` | string | yes | Absolute path to the folder |
|
|
680
|
+
| `Entity` | string | no | Force all files to this entity name |
|
|
681
|
+
| `MappingConfiguration` | object | no | Mapping hints applied to all files |
|
|
682
|
+
|
|
683
|
+
**Example** *(corresponds to the Seattle neighborhoods scenario in [Example 006](examples-walkthrough.md))*:
|
|
684
|
+
|
|
685
|
+
```shell
|
|
686
|
+
curl -X POST http://localhost:8086/1.0/Entity/FromTabularFolder \
|
|
687
|
+
-H "Content-Type: application/json" \
|
|
688
|
+
-d '{
|
|
689
|
+
"Folder": "/path/to/docs/examples/data/seattle_neighborhoods/",
|
|
690
|
+
"Entity": "Neighborhood",
|
|
691
|
+
"MappingConfiguration": {
|
|
692
|
+
"GUIDTemplate": "{~D:Record.Neighborhood Name~}",
|
|
693
|
+
"Mappings": {
|
|
694
|
+
"Name": "{~D:Record.Neighborhood Name~}"
|
|
695
|
+
}
|
|
696
|
+
}
|
|
697
|
+
}'
|
|
698
|
+
```
|
|
699
|
+
|
|
700
|
+
**Response:**
|
|
701
|
+
|
|
702
|
+
```json
|
|
703
|
+
{
|
|
704
|
+
"Neighborhood": {
|
|
705
|
+
"Ballard": { "GUIDNeighborhood": "Ballard", "Name": "Ballard", "..." : "..." },
|
|
706
|
+
"Fremont": { "GUIDNeighborhood": "Fremont", "Name": "Fremont", "..." : "..." }
|
|
707
|
+
}
|
|
708
|
+
}
|
|
709
|
+
```
|
|
710
|
+
|
|
711
|
+
---
|
|
712
|
+
|
|
713
|
+
## CLI-to-REST Equivalence
|
|
714
|
+
|
|
715
|
+
Every CLI command has a corresponding REST endpoint. The table below maps between the two:
|
|
716
|
+
|
|
717
|
+
| CLI Command | REST Endpoint | Notes |
|
|
718
|
+
|-------------|---------------|-------|
|
|
719
|
+
| `csvcheck <file>` | `POST /1.0/CSV/Check` | `File` in body |
|
|
720
|
+
| `csvtransform <file>` | `POST /1.0/CSV/Transform` | `-m` becomes `MappingConfiguration`, `-i` becomes `IncomingComprehension` |
|
|
721
|
+
| `tsvcheck <file>` | `POST /1.0/TSV/Check` | `File` in body |
|
|
722
|
+
| `tsvtransform <file>` | `POST /1.0/TSV/Transform` | Same mapping as csvtransform |
|
|
723
|
+
| `jsonarraytransform <file>` | `POST /1.0/JSONArray/Transform` | Or use `/TransformRecords` for in-memory |
|
|
724
|
+
| `comprehensionintersect <file> -i <file2>` | `POST /1.0/Comprehension/Intersect` | Or `/IntersectFiles` for file paths |
|
|
725
|
+
| `comprehensionarray <file>` | `POST /1.0/Comprehension/ToArray` | Or `/ToArrayFromFile` for file paths |
|
|
726
|
+
| `objectarraytocsv <file>` | `POST /1.0/Comprehension/ToCSV` | Or `/ToCSVFromFile` for file paths |
|
|
727
|
+
| `load_comprehension <file>` | `POST /1.0/Comprehension/Push` | Or `/PushFile` for file paths |
|
|
728
|
+
| `entc_ftf <folder>` | `POST /1.0/Entity/FromTabularFolder` | `Folder` in body |
|
|
729
|
+
| *(new)* | `POST /1.0/JSONArray/TransformRecords` | In-memory only, no CLI equivalent |
|
|
730
|
+
|
|
731
|
+
For file-based CLI commands, the REST API offers both an in-memory variant (pass data directly in the request body) and a file variant (pass a `File` path). The in-memory variants are useful when chaining operations in a pipeline without writing intermediate files to disk.
|