meadow-integration 1.0.4 → 1.0.6

package/docs/rest/overview.md

@@ -0,0 +1,120 @@
+# Meadow Integration REST API Overview
+
+The Meadow Integration REST API server provides HTTP endpoints for data transformation, comprehension operations, and entity generation. It is built on the Orator/Restify service stack.
+
+## Starting the Server
+
+### Via the CLI
+
+```bash
+# Start on the default port (8086)
+mdwint serve
+
+# Start on a custom port
+mdwint serve --port 9000
+```
+
+### Via Environment Variable
+
+```bash
+MEADOW_INTEGRATION_PORT=9000 mdwint serve
+```
+
+### Programmatically
+
+```javascript
+const MeadowIntegrationServer = require('meadow-integration').IntegrationServer;
+
+const server = new MeadowIntegrationServer({ APIServerPort: 8086 });
+
+server.start((pError) =>
+{
+    if (pError)
+    {
+        console.error('Failed to start server:', pError);
+    }
+    else
+    {
+        console.log('Server is running.');
+    }
+});
+```
+
+## Base URL
+
+All endpoints are versioned under `/1.0/`. When the server runs on the default port, the base URL is:
+
+```
+http://localhost:8086/1.0/
+```
+
+## Content-Type
+
+- **Requests**: All POST endpoints expect `Content-Type: application/json`.
+- **Responses**: All endpoints return `application/json` unless otherwise noted (the `ToCSV` endpoints return `text/csv`).
+
+## Available Endpoints
+
+| Method | Path | Description |
+|--------|------|-------------|
+| GET | `/1.0/Status` | Server status and endpoint list |
+| POST | `/1.0/CSV/Check` | Analyze a CSV file for statistics |
+| POST | `/1.0/CSV/Transform` | Transform a CSV file into a comprehension |
+| POST | `/1.0/TSV/Check` | Analyze a TSV file for statistics |
+| POST | `/1.0/TSV/Transform` | Transform a TSV file into a comprehension |
+| POST | `/1.0/JSONArray/Transform` | Transform a JSON array file into a comprehension |
+| POST | `/1.0/JSONArray/TransformRecords` | Transform an in-memory JSON array into a comprehension |
+| POST | `/1.0/Comprehension/Intersect` | Merge two comprehension objects (in-memory) |
+| POST | `/1.0/Comprehension/IntersectFiles` | Merge two comprehension files |
+| POST | `/1.0/Comprehension/ToArray` | Convert a comprehension to an array (in-memory) |
+| POST | `/1.0/Comprehension/ToArrayFromFile` | Convert a comprehension file to an array |
+| POST | `/1.0/Comprehension/ToCSV` | Convert a comprehension or array to CSV (in-memory) |
+| POST | `/1.0/Comprehension/ToCSVFromFile` | Convert a comprehension or array file to CSV |
+| POST | `/1.0/Comprehension/Push` | Push a comprehension to Meadow REST APIs (in-memory) |
+| POST | `/1.0/Comprehension/PushFile` | Push a comprehension file to Meadow REST APIs |
+| POST | `/1.0/Entity/FromTabularFolder` | Generate comprehensions from a folder of tabular files |
+
+## Status Endpoint
+
+A quick way to verify the server is running:
+
+```bash
+curl http://localhost:8086/1.0/Status
+```
+
+See `status.md` for full details.
+
+## Error Handling
+
+All endpoints return structured error responses with an `Error` property:
+
+```json
+{
+    "Error": "No valid File path provided in request body."
+}
+```
+
+Common HTTP status codes:
+
+| Code | Meaning |
+|------|---------|
+| 200 | Success |
+| 400 | Bad request (missing or invalid parameters) |
+| 404 | File or resource not found |
+| 500 | Internal server error |
+
+## File Paths
+
+Endpoints that accept file paths (the `File` or `Folder` properties) expect absolute paths on the server filesystem. Relative paths are resolved from the server's working directory.
+
+## Detailed Documentation
+
+Each endpoint group has its own documentation file:
+
+- [Status](status.md)
+- [CSV Operations](csv.md)
+- [TSV Operations](tsv.md)
+- [JSON Array Operations](json-array.md)
+- [Comprehension Operations](comprehension.md)
+- [Comprehension Push](comprehension-push.md)
+- [Entity Generation](entity-generation.md)

package/docs/rest/status.md

@@ -0,0 +1,63 @@
+# GET /1.0/Status
+
+Returns the server status, product version, and a list of all available endpoints.
+
+## Request
+
+- **Method**: GET
+- **Path**: `/1.0/Status`
+- **Body**: None
+
+## Response
+
+```json
+{
+    "Product": "Meadow-Integration-Server",
+    "Version": "1.0.5",
+    "Status": "Running",
+    "Endpoints": [
+        "POST /1.0/CSV/Check",
+        "POST /1.0/CSV/Transform",
+        "POST /1.0/TSV/Check",
+        "POST /1.0/TSV/Transform",
+        "POST /1.0/JSONArray/Transform",
+        "POST /1.0/JSONArray/TransformRecords",
+        "POST /1.0/Comprehension/Intersect",
+        "POST /1.0/Comprehension/IntersectFiles",
+        "POST /1.0/Comprehension/ToArray",
+        "POST /1.0/Comprehension/ToArrayFromFile",
+        "POST /1.0/Comprehension/ToCSV",
+        "POST /1.0/Comprehension/ToCSVFromFile",
+        "POST /1.0/Comprehension/Push",
+        "POST /1.0/Comprehension/PushFile",
+        "POST /1.0/Entity/FromTabularFolder"
+    ]
+}
+```
+
+## curl Example
+
+```bash
+curl -s http://localhost:8086/1.0/Status
+```
+
+## JavaScript Example (dependency-free)
+
+```javascript
+const http = require('http');
+
+const options = {
+    hostname: 'localhost',
+    port: 8086,
+    path: '/1.0/Status',
+    method: 'GET'
+};
+
+const req = http.request(options, (res) => {
+    let data = '';
+    res.on('data', (chunk) => { data += chunk; });
+    res.on('end', () => { console.log(JSON.parse(data)); });
+});
+req.on('error', (err) => { console.error('Request error:', err.message); });
+req.end();
+```

package/docs/rest/tsv.md

@@ -0,0 +1,241 @@
+# TSV Endpoints
+
+The TSV endpoints are functionally identical to the CSV endpoints but operate on tab-separated value files. The parser delimiter is automatically set to `\t`.
+
+## POST /1.0/TSV/Check
+
+Analyze a TSV file and return statistics about its structure and content.
+
+### Request Body
+
+| Property | Type | Required | Default | Description |
+|----------|------|----------|---------|-------------|
+| `File` | string | Yes | - | Absolute path to the TSV file to analyze |
+| `Records` | boolean | No | `false` | When `true`, include all parsed records in the response |
+| `QuoteDelimiter` | string | No | `"` | Character used for quoting fields |
+
+### Response
+
+```json
+{
+    "File": "/data/products.tsv",
+    "RowCount": 150,
+    "ColumnCount": 4,
+    "Columns": ["sku", "name", "price", "category"]
+}
+```
+
+### Error Responses
+
+| Status | Condition |
+|--------|-----------|
+| 400 | No valid `File` path provided |
+| 404 | File does not exist |
+| 500 | Error reading the TSV file |
+
+### curl Example
+
+```bash
+curl -s -X POST http://localhost:8086/1.0/TSV/Check \
+  -H "Content-Type: application/json" \
+  -d '{
+    "File": "/data/products.tsv"
+  }'
+```
+
+With records:
+
+```bash
+curl -s -X POST http://localhost:8086/1.0/TSV/Check \
+  -H "Content-Type: application/json" \
+  -d '{
+    "File": "/data/products.tsv",
+    "Records": true
+  }'
+```
+
+### JavaScript Example (dependency-free)
+
+```javascript
+const http = require('http');
+
+const requestBody = JSON.stringify({
+    File: '/data/products.tsv'
+});
+
+const options = {
+    hostname: 'localhost',
+    port: 8086,
+    path: '/1.0/TSV/Check',
+    method: 'POST',
+    headers: {
+        'Content-Type': 'application/json',
+        'Content-Length': Buffer.byteLength(requestBody)
+    }
+};
+
+const req = http.request(options, (res) => {
+    let data = '';
+    res.on('data', (chunk) => { data += chunk; });
+    res.on('end', () => {
+        const stats = JSON.parse(data);
+        console.log(`Rows: ${stats.RowCount}, Columns: ${stats.ColumnCount}`);
+        console.log('Column names:', stats.Columns);
+    });
+});
+req.on('error', (err) => { console.error('Request error:', err.message); });
+req.write(requestBody);
+req.end();
+```
+
+---
+
+## POST /1.0/TSV/Transform
+
+Transform a TSV file into a comprehension object.
+
+### Request Body
+
+| Property | Type | Required | Default | Description |
+|----------|------|----------|---------|-------------|
+| `File` | string | Yes | - | Absolute path to the TSV file to transform |
+| `Entity` | string | No | Derived from filename | Entity name for the comprehension |
+| `GUIDName` | string | No | `GUID{Entity}` | Name of the GUID column in the output |
+| `GUIDTemplate` | string | No | Auto-generated | Pict template expression for generating GUIDs |
+| `Mappings` | object | No | - | Column-level mapping overrides |
+| `MappingConfiguration` | object | No | - | Full explicit mapping configuration object |
+| `IncomingComprehension` | object | No | `{}` | Existing comprehension to merge new records into |
+| `Extended` | boolean | No | `false` | Return full mapping outcome state |
+| `QuoteDelimiter` | string | No | `"` | Character used for quoting fields |
+
+### Response
+
+Returns a comprehension object (same structure as CSV/Transform):
+
+```json
+{
+    "Product": {
+        "SKU-001": {
+            "GUIDProduct": "SKU-001",
+            "sku": "SKU-001",
+            "name": "Widget",
+            "price": "9.99",
+            "category": "Hardware"
+        }
+    }
+}
+```
+
+When `Extended` is `true`, returns the full mapping outcome.
+
+### Error Responses
+
+| Status | Condition |
+|--------|-----------|
+| 400 | No valid `File` path provided |
+| 404 | File does not exist |
+| 500 | Error reading the TSV file |
+
+### curl Example
+
+Basic transform:
+
+```bash
+curl -s -X POST http://localhost:8086/1.0/TSV/Transform \
+  -H "Content-Type: application/json" \
+  -d '{
+    "File": "/data/products.tsv"
+  }'
+```
+
+With entity name and GUID template:
+
+```bash
+curl -s -X POST http://localhost:8086/1.0/TSV/Transform \
+  -H "Content-Type: application/json" \
+  -d '{
+    "File": "/data/products.tsv",
+    "Entity": "Product",
+    "GUIDTemplate": "{~D:Record.sku~}"
+  }'
+```
+
+### JavaScript Example (dependency-free)
+
+```javascript
+const http = require('http');
+
+const requestBody = JSON.stringify({
+    File: '/data/products.tsv',
+    Entity: 'Product',
+    GUIDTemplate: '{~D:Record.sku~}'
+});
+
+const options = {
+    hostname: 'localhost',
+    port: 8086,
+    path: '/1.0/TSV/Transform',
+    method: 'POST',
+    headers: {
+        'Content-Type': 'application/json',
+        'Content-Length': Buffer.byteLength(requestBody)
+    }
+};
+
+const req = http.request(options, (res) => {
+    let data = '';
+    res.on('data', (chunk) => { data += chunk; });
+    res.on('end', () => {
+        const comprehension = JSON.parse(data);
+        const entityKeys = Object.keys(comprehension);
+        console.log('Entities:', entityKeys);
+        entityKeys.forEach((entity) => {
+            const recordCount = Object.keys(comprehension[entity]).length;
+            console.log(`  ${entity}: ${recordCount} records`);
+        });
+    });
+});
+req.on('error', (err) => { console.error('Request error:', err.message); });
+req.write(requestBody);
+req.end();
+```
+
+### Merging into an Existing Comprehension
+
+```javascript
+const http = require('http');
+
+const requestBody = JSON.stringify({
+    File: '/data/products_update.tsv',
+    Entity: 'Product',
+    GUIDTemplate: '{~D:Record.sku~}',
+    IncomingComprehension: {
+        Product: {
+            'SKU-001': { GUIDProduct: 'SKU-001', sku: 'SKU-001', name: 'Widget', price: '9.99' }
+        }
+    }
+});
+
+const options = {
+    hostname: 'localhost',
+    port: 8086,
+    path: '/1.0/TSV/Transform',
+    method: 'POST',
+    headers: {
+        'Content-Type': 'application/json',
+        'Content-Length': Buffer.byteLength(requestBody)
+    }
+};
+
+const req = http.request(options, (res) => {
+    let data = '';
+    res.on('data', (chunk) => { data += chunk; });
+    res.on('end', () => {
+        const merged = JSON.parse(data);
+        console.log('Merged record count:', Object.keys(merged.Product).length);
+    });
+});
+req.on('error', (err) => { console.error('Request error:', err.message); });
+req.write(requestBody);
+req.end();
+```

package/docs/retold-catalog.json

@@ -1,5 +1,5 @@
 {
-	"Generated": "2026-02-18T03:35:30.874Z",
+	"Generated": "2026-03-04T15:46:38.412Z",
 	"GitHubOrg": "stevenvelozo",
 	"DefaultBranch": "master",
 	"Groups": [
@@ -25,6 +25,77 @@
 			"Key": "docs",
 			"Description": "",
 			"Modules": [
+				{
+					"Name": "api",
+					"Repo": "api",
+					"Group": "docs",
+					"Branch": "master",
+					"HasDocs": true,
+					"HasCover": false,
+					"Sidebar": [],
+					"DocFiles": [
+						"api/clone-rest-client.md",
+						"api/connection-manager.md",
+						"api/guid-map.md",
+						"api/integration-adapter.md",
+						"api/operation.md",
+						"api/sync-entity-initial.md",
+						"api/sync-entity-ongoing.md",
+						"api/sync.md",
+						"api/tabular-check.md",
+						"api/tabular-transform.md"
+					]
+				},
+				{
+					"Name": "cli",
+					"Repo": "cli",
+					"Group": "docs",
+					"Branch": "master",
+					"HasDocs": true,
+					"HasCover": false,
+					"Sidebar": [],
+					"DocFiles": [
+						"cli/comprehensionarray.md",
+						"cli/comprehensionintersect.md",
+						"cli/csvcheck.md",
+						"cli/csvtransform.md",
+						"cli/data-clone.md",
+						"cli/jsonarraytransform.md",
+						"cli/load-comprehension.md",
+						"cli/objectarraytocsv.md",
+						"cli/overview.md",
+						"cli/serve.md",
+						"cli/tsvtransform.md"
+					]
+				},
+				{
+					"Name": "css",
+					"Repo": "css",
+					"Group": "docs",
+					"Branch": "master",
+					"HasDocs": true,
+					"HasCover": false,
+					"Sidebar": [],
+					"DocFiles": [
+						"css/docuserve.css"
+					]
+				},
+				{
+					"Name": "data-clone",
+					"Repo": "data-clone",
+					"Group": "docs",
+					"Branch": "master",
+					"HasDocs": true,
+					"HasCover": false,
+					"Sidebar": [],
+					"DocFiles": [
+						"data-clone/configuration.md",
+						"data-clone/connection-manager.md",
+						"data-clone/docker.md",
+						"data-clone/overview.md",
+						"data-clone/sync-modes.md"
+					]
+				},
 				{
 					"Name": "examples",
 					"Repo": "examples",
@@ -37,9 +108,9 @@
 						"examples/bookstore/.gitignore",
 						"examples/bookstore/BookData.md",
 						"examples/bookstore/Generate_Bookstore_Comprehension.sh",
-						"examples/bookstore/mapping_books_Author.json",
-						"examples/bookstore/mapping_books_Book.json",
 						"examples/bookstore/mapping_books_BookAuthorJoin.json",
+						"examples/bookstore/mapping_books_author.json",
+						"examples/bookstore/mapping_books_book.json",
 						"examples/data/books.csv",
 						"examples/data/seattle_neighborhoods/Seattle_Data_Information.md",
 						"examples/data/seattle_neighborhoods/housing_characteristics_Neighborhoods.csv",
@@ -50,6 +121,25 @@
 						"examples/multi_set_integration/part_1_getting_data.md"
 					]
 				},
+				{
+					"Name": "rest",
+					"Repo": "rest",
+					"Group": "docs",
+					"Branch": "master",
+					"HasDocs": true,
+					"HasCover": false,
+					"Sidebar": [],
+					"DocFiles": [
+						"rest/comprehension-push.md",
+						"rest/comprehension.md",
+						"rest/csv.md",
+						"rest/entity-generation.md",
+						"rest/json-array.md",
+						"rest/overview.md",
+						"rest/status.md",
+						"rest/tsv.md"
+					]
+				},
 				{
 					"Name": "vocabulary",
 					"Repo": "vocabulary",