meadow-integration 1.0.1 → 1.0.4

package/CONTRIBUTING.md

@@ -0,0 +1,50 @@
+# Contributing to Retold
+
+We welcome contributions to Retold and its modules. This guide covers the expectations and process for contributing.
+
+## Code of Conduct
+
+The Retold community values **empathy**, **equity**, **kindness**, and **thoughtfulness**. We expect all participants to treat each other with respect, assume good intent, and engage constructively. These values apply to all interactions: pull requests, issues, discussions, and code review.
+
+## How to Contribute
+
+### Pull Requests
+
+Pull requests are the preferred method for contributing changes. To submit one:
+
+1. Fork the module repository you want to change
+2. Create a branch for your work
+3. Make your changes, following the code style of the module you are editing
+4. Ensure your changes have test coverage (see below)
+5. Open a pull request against the module's main branch
+
+**Submitting a pull request does not guarantee it will be accepted.** Maintainers review contributions for fit, quality, and alignment with the project's direction. A PR may be declined, or you may be asked to revise it. This is normal and not a reflection on the quality of your effort.
+
+### Reporting Issues
+
+If you find a bug or have a feature suggestion, open an issue on the relevant module's repository. Include enough detail to reproduce the problem or understand the proposal.
+
+## Test Coverage
+
+Every commit must include test coverage for the changes it introduces. Retold modules use Mocha in TDD style. Before submitting:
+
+- **Write tests** for any new functionality or bug fixes
+- **Run the existing test suite** with `npm test` and confirm all tests pass
+- **Check coverage** with `npm run coverage` if the module supports it
+
+Pull requests that break existing tests or lack coverage for new code will not be merged.
+
+## Code Style
+
+Follow the conventions of the module you are working in. The general Retold style is:
+
+- **Tabs** for indentation, never spaces
+- **Plain JavaScript** only (no TypeScript)
+- **Allman-style braces** (opening brace on its own line)
+- **Variable naming:** `pVariable` for parameters, `tmpVariable` for temporaries, `libSomething` for imports
+
+When in doubt, match what the surrounding code does.
+
+## Repository Structure
+
+Each module is its own git repository. The [retold](https://github.com/stevenvelozo/retold) repository tracks module organization but does not contain module source code. Direct your pull request to the specific module repository where your change belongs.

package/README.md

@@ -1,15 +1,231 @@
 # Meadow Integration
 
-A suite of tools for managing data into a centralized non-specific schema format.
+A suite of tools for managing data into a centralized non-specific schema format called a **Comprehension**.
 
-These tools are built to be usable from the command-line, as a web service, or within your own codebase.  This code repository presents these behaviors both as a suite of externally usable fable services, a command-line utility to leverage them and a set of web service behaviors.
+These tools are built to be usable from the command-line, as a web service, or within your own codebase.  This module presents these behaviors both as a suite of externally usable fable services, a command-line utility to leverage them and a set of web service behaviors.
 
-## CSV Stuff
+## What is a Comprehension?
 
-So you like the CSV format?  So does this utility.
+A Comprehension is a JSON object that stores entity records keyed by their GUID.  It acts as an intermediate format for integrating records from external systems (CSV, TSV, JSON) into Meadow entities.
 
-You can try out what it can do to provide stats on CSVs by running the following from the repository root:
+```json
+{
+  "Book": {
+    "Book_1": { "GUIDBook": "Book_1", "Title": "The Hunger Games", "Language": "eng" },
+    "Book_2": { "GUIDBook": "Book_2", "Title": "Harry Potter", "Language": "eng" }
+  }
+}
+```
+
+## Quick Start
+
+```shell
+npm install
+```
+
+### Analyze a CSV
+
+```shell
+npm start -- csvcheck ./docs/examples/data/books.csv -o stats.json
+```
+
+### Transform a CSV into a Comprehension
+
+```shell
+npm start -- csvtransform ./docs/examples/data/books.csv \
+  -e "Book" -n "GUIDBook" -g "Book_{~D:Record.id~}" \
+  -o books.json
+```
+
+### Use a Mapping File
+
+```shell
+npm start -- csvtransform ./docs/examples/data/books.csv \
+  -m ./docs/examples/bookstore/mapping_books_Book.json \
+  -o books.json
+```
+
+### Merge Multiple Data Sources
+
+```shell
+npm start -- comprehensionintersect Set1.json -i Set2.json -e "MyEntity" -o merged.json
+```
+
+## CLI Commands
+
+| Command | Description |
+|---------|-------------|
+| `csvcheck` | Analyze a CSV file for statistics |
+| `csvtransform` | Transform a CSV into a comprehension |
+| `tsvtransform` | Transform a TSV into a comprehension |
+| `jsonarraytransform` | Transform a JSON array into a comprehension |
+| `comprehensionintersect` | Merge two comprehension files |
+| `comprehensionarray` | Convert object comprehension to array |
+| `objectarraytocsv` | Export a JSON array to CSV |
+| `load_comprehension` | Push a comprehension to Meadow REST APIs |
+| `serve` | Start the REST API server |
+
+## REST API Server
+
+Start the integration server to access all commands as HTTP endpoints:
+
+```shell
+npm start -- serve
+npm start -- serve -p 3000
+```
+
+Then call any endpoint with `curl` or your HTTP client of choice:
+
+```shell
+# Analyze a CSV
+curl -X POST http://localhost:8086/1.0/CSV/Check \
+  -H "Content-Type: application/json" \
+  -d '{ "File": "/absolute/path/to/books.csv" }'
+
+# Transform records in-memory (no file needed)
+curl -X POST http://localhost:8086/1.0/JSONArray/TransformRecords \
+  -H "Content-Type: application/json" \
+  -d '{
+    "Records": [
+      { "id": "1", "title": "The Hunger Games", "isbn": "439023483" },
+      { "id": "2", "title": "Harry Potter", "isbn": "439554934" }
+    ],
+    "Entity": "Book",
+    "GUIDTemplate": "Book_{~D:Record.id~}",
+    "Mappings": { "Title": "{~D:Record.title~}", "ISBN": "{~D:Record.isbn~}" }
+  }'
+
+# Merge two comprehensions
+curl -X POST http://localhost:8086/1.0/Comprehension/Intersect \
+  -H "Content-Type: application/json" \
+  -d '{
+    "PrimaryComprehension": { "Book": { "Book_1": { "GUIDBook": "Book_1", "Title": "The Hunger Games" } } },
+    "SecondaryComprehension": { "Book": { "Book_1": { "GUIDBook": "Book_1", "Rating": "4.3" } } }
+  }'
+```
+
+See [REST API Reference](docs/rest-api-reference.md) for full endpoint documentation with sample request bodies for every operation.
+
+## Examples
+
+The `examples/` folder contains 10 runnable scripts demonstrating all major features:
 
 ```shell
- npm start -- csvcheck ./documentation/examples/data/housing_costs_Neighborhoods_-8848403750169343217.csv
- ```
+cd examples
+
+# Analyze CSV structure
+./Example-001-CSV-Check.sh
+
+# Transform with auto-detection
+./Example-002-CSV-Transform-Implicit.sh
+
+# Transform with CLI options
+./Example-003-CSV-Transform-CLI-Options.sh
+
+# Transform with mapping file
+./Example-004-CSV-Transform-Mapping-File.sh
+
+# Multi-entity bookstore (Book, Author, BookAuthorJoin)
+./Example-005-Multi-Entity-Bookstore.sh
+
+# Merge three Seattle neighborhood CSVs
+./Example-006-Multi-CSV-Intersect.sh
+
+# Convert comprehension to array
+./Example-007-Comprehension-To-Array.sh
+
+# Export comprehension to CSV
+./Example-008-Comprehension-To-CSV.sh
+
+# Transform from JSON array
+./Example-009-JSON-Array-Transform.sh
+
+# Programmatic API usage
+node Example-010-Programmatic-API.js
+```
+
+## Mapping Files
+
+Mapping files define how source columns become comprehension fields:
+
+```json
+{
+  "Entity": "Book",
+  "GUIDTemplate": "Book_{~D:Record.id~}",
+  "Mappings": {
+    "Title": "{~D:Record.title~}",
+    "Language": "{~D:Record.language_code~}",
+    "ISBN": "{~D:Record.isbn~}",
+    "Genre": "Unknown"
+  }
+}
+```
+
+For multi-record generation (e.g. splitting comma-separated authors), use Solvers:
+
+```json
+{
+  "Entity": "Author",
+  "MultipleGUIDUniqueness": true,
+  "Solvers": ["NewRecordsGUIDUniqueness = STRINGGETSEGMENTS(IncomingRecord.authors,\",\")"],
+  "GUIDTemplate": "Author_{~PascalCaseIdentifier:Record._GUIDUniqueness~}",
+  "Mappings": {
+    "Name": "{~D:Record._GUIDUniqueness~}"
+  }
+}
+```
+
+## Architecture
+
+```
+External Data (CSV / TSV / JSON)
+        |
+        v
+   TabularTransform Service  -- column mapping via Pict templates
+        |
+        v
+   Comprehension Object      -- entity records keyed by GUID
+        |
+        v
+   Integration Adapter       -- marshal to Meadow schema
+        |
+        v
+   GUID Map                  -- track external <-> Meadow IDs
+        |
+        v
+   Meadow REST API           -- batch upsert / single upsert
+```
+
+## Documentation
+
+Full documentation is available at `docs/index.html` (powered by pict-docuserve):
+
+- [CLI Reference](docs/cli-reference.md)
+- [REST API Reference](docs/rest-api-reference.md)
+- [Mapping Files](docs/mapping-files.md)
+- [Comprehensions](docs/comprehensions.md)
+- [Programmatic API](docs/programmatic-api.md)
+- [Integration Adapter](docs/integration-adapter.md)
+- [Examples Walkthrough](docs/examples-walkthrough.md)
+- [Vocabulary](docs/vocabulary/)
+
+## Testing
+
+```shell
+npm test
+```
+
+## Related Packages
+
+- [meadow](https://github.com/stevenvelozo/meadow) - Data access and ORM
+- [meadow-endpoints](https://github.com/stevenvelozo/meadow-endpoints) - Auto-generated REST endpoints
+- [orator](https://github.com/stevenvelozo/orator) - API server abstraction
+- [fable](https://github.com/stevenvelozo/fable) - Application services framework
+
+## License
+
+MIT
+
+## Contributing
+
+Pull requests are welcome. For details on our code of conduct, contribution process, and testing requirements, see the [Retold Contributing Guide](https://github.com/stevenvelozo/retold/blob/main/docs/contributing.md).

package/docs/README.md

@@ -1,15 +1,115 @@
 # Meadow Integration
 
-A suite of tools for managing data into a centralized non-specific schema format.
+A suite of tools for managing data into a centralized non-specific schema format called a **Comprehension**.
 
-These tools are built to be usable from the command-line, as a web service, or within your own codebase.  This code repository presents these behaviors both as a suite of externally usable fable services, a command-line utility to leverage them and a set of web service behaviors.
+These tools are built to be usable from the command-line, as a web service, or within your own codebase.  This module presents these behaviors both as a suite of externally usable fable services, a command-line utility to leverage them and a set of web service behaviors.
 
-## CSV Stuff
+## What is a Comprehension?
 
-So you like the CSV format?  So does this utility.
+A **Comprehension** is a JSON object that stores entity records keyed by their GUID.  It acts as an intermediate data format for integrating records from external systems (CSV, TSV, JSON) into Meadow entities.
 
-You can try out what it can do to provide stats on CSVs by running the following from the repository root:
+```json
+{
+  "Book": {
+    "Book_1": { "GUIDBook": "Book_1", "Title": "The Hunger Games", "Language": "eng" },
+    "Book_2": { "GUIDBook": "Book_2", "Title": "Harry Potter", "Language": "eng" }
+  },
+  "Author": {
+    "Author_SuzanneCollins": { "GUIDAuthor": "Author_SuzanneCollins", "Name": "Suzanne Collins" }
+  }
+}
+```
+
+A single comprehension can hold multiple entity types, making it easy to model related data from the same source.
+
+## Installation
+
+```shell
+npm install meadow-integration
+```
+
+Or for CLI usage:
+
+```shell
+npx meadow-integration --help
+```
+
+## Quick Start
+
+### 1. Analyze a CSV file
+
+```shell
+npx meadow-integration csvcheck ./my-data.csv -o stats.json
+```
+
+This produces a JSON file with row/column counts, headers, and per-column statistics.
+
+### 2. Transform a CSV into a Comprehension
+
+```shell
+npx meadow-integration csvtransform ./my-data.csv \
+  -e "MyEntity" \
+  -n "GUIDMyEntity" \
+  -g "MyEntity_{~D:Record.id~}" \
+  -o my-comprehension.json
+```
+
+### 3. Use a Mapping File for precise control
+
+```json
+{
+  "Entity": "Book",
+  "GUIDTemplate": "Book_{~D:Record.id~}",
+  "Mappings": {
+    "Title": "{~D:Record.title~}",
+    "Language": "{~D:Record.language_code~}",
+    "ISBN": "{~D:Record.isbn~}"
+  }
+}
+```
+
+```shell
+npx meadow-integration csvtransform ./books.csv -m mapping.json -o books.json
+```
+
+### 4. Merge multiple comprehensions
 
 ```shell
- npm start -- csvcheck ./documentation/examples/data/housing_costs_Neighborhoods_-8848403750169343217.csv
- ```
+npx meadow-integration comprehensionintersect Set1.json -i Set2.json -e "MyEntity" -o merged.json
+```
+
+## Architecture
+
+```
+External Data (CSV / TSV / JSON)
+        |
+        v
+   TabularTransform Service
+   (column mapping via Pict templates)
+        |
+        v
+   Comprehension Object
+   (Entity records keyed by GUID)
+        |
+        v
+   Integration Adapter
+   (marshal to Meadow schema)
+        |
+        v
+   GUID Map
+   (track external <-> Meadow IDs)
+        |
+        v
+   Meadow REST API
+   (batch upsert / single upsert)
+```
+
+## Next Steps
+
+- [CLI Reference](cli-reference.md) -- All commands and their options
+- [REST API Reference](rest-api-reference.md) -- All REST endpoints with curl examples
+- [Mapping Files](mapping-files.md) -- How to write column mapping configurations
+- [Comprehensions](comprehensions.md) -- The comprehension data format in detail
+- [Programmatic API](programmatic-api.md) -- Using services directly in your code
+- [Integration Adapter](integration-adapter.md) -- Pushing data to Meadow REST APIs
+- [Examples](examples-walkthrough.md) -- Walkthrough of all runnable examples

package/docs/_sidebar.md

@@ -0,0 +1,38 @@
+- Getting Started
+
+  - [Introduction](/)
+  - [Quick Start](README.md)
+
+- Concepts
+
+  - [Comprehensions](comprehensions.md)
+  - [Mapping Files](mapping-files.md)
+
+- Reference
+
+  - [CLI Reference](cli-reference.md)
+  - [REST API Reference](rest-api-reference.md)
+  - [Programmatic API](programmatic-api.md)
+  - [Integration Adapter](integration-adapter.md)
+
+- Examples
+
+  - [Examples Walkthrough](examples-walkthrough.md)
+  - [Bookstore Example](examples/bookstore/BookData.md)
+  - [Seattle Multi-Set](examples/multi_set_integration/multi_set_step_by_step.md)
+
+- Vocabulary
+
+  - [Comprehension](vocabulary/Comprehension.md)
+  - [Entity](vocabulary/Entity.md)
+  - [GUID](vocabulary/GUID.md)
+  - [Schema](vocabulary/Schema.md)
+  - [Set](vocabulary/Set.md)
+  - [Source](vocabulary/Source.md)
+  - [Column](vocabulary/Column.md)
+  - [Record](vocabulary/Record.md)
+  - [Format](vocabulary/Format.md)
+  - [Join](vocabulary/Join.md)
+  - [Snapshot](vocabulary/Snapshot.md)
+  - [Version](vocabulary/Version.md)
+  - [Confidence](vocabulary/Confidence.md)

package/docs/_topbar.md

@@ -0,0 +1,7 @@
+# Meadow Integration
+
+- [Overview](README.md)
+- [CLI Reference](cli-reference.md)
+- [API Reference](programmatic-api.md)
+- [Examples](examples-walkthrough.md)
+- [GitHub](https://github.com/stevenvelozo/meadow-integration)

package/docs/cli-reference.md

@@ -0,0 +1,242 @@
+# CLI Reference
+
+The `meadow-integration` CLI (also available as `mdwint` when installed globally) provides commands for transforming, analyzing, and merging tabular data.
+
+## Global Usage
+
+```shell
+npx meadow-integration [command] [options]
+
+# Or from the repository:
+npm start -- [command] [options]
+```
+
+---
+
+## csvcheck
+
+Analyze a CSV file and produce column-level statistics.
+
+```shell
+npx meadow-integration csvcheck <file> [options]
+```
+
+**Arguments:**
+| Argument | Description |
+|----------|-------------|
+| `<file>` | Path to the CSV file to analyze |
+
+**Options:**
+| Option | Description |
+|--------|-------------|
+| `-f, --file <filepath>` | Alternate way to specify the CSV file |
+| `-o, --output <filepath>` | Output file path. Default: `./CSV-Stats-[filename].json` |
+| `-r, --records` | Include all parsed records in the output |
+
+**Output:** A JSON file containing:
+- `RowCount` / `ColumnCount` -- Number of data rows and columns
+- `Headers` -- Array of column names
+- `FirstRow` / `LastRow` -- The first and last records
+- `ColumnStatistics` -- Per-column: Count, EmptyCount, NumericCount, FirstValue, LastValue
+
+**Example:**
+```shell
+npx meadow-integration csvcheck ./data/books.csv -o book-stats.json
+```
+
+---
+
+## csvtransform
+
+Transform a CSV file into a Comprehension JSON file.
+
+```shell
+npx meadow-integration csvtransform <file> [options]
+```
+
+**Arguments:**
+| Argument | Description |
+|----------|-------------|
+| `<file>` | Path to the CSV file to transform |
+
+**Options:**
+| Option | Description |
+|--------|-------------|
+| `-e, --entity <name>` | Entity name in the comprehension |
+| `-n, --guidname <name>` | GUID column name (e.g. `GUIDBook`) |
+| `-g, --guidtemplate <template>` | Pict template for GUID values (e.g. `Book_{~D:Record.id~}`) |
+| `-c, --columns <mappings>` | Inline column mappings: `Col1={~D:col1~},Col2={~D:col2~}` |
+| `-m, --mappingfile <path>` | Path to a JSON mapping file |
+| `-i, --incoming <path>` | Existing comprehension file to merge into |
+| `-o, --output <path>` | Output file path |
+| `-x, --extended` | Output full operation state, not just the comprehension |
+| `-q, --quotedelimiter <char>` | Quote delimiter character (default: `"`) |
+
+**Priority of configuration:**
+1. Implicit -- auto-detected from the first CSV row
+2. Explicit -- loaded from a mapping file (`-m`)
+3. User -- command-line options (`-e`, `-g`, `-c`, etc.)
+
+Each layer overrides the previous.
+
+**Example:**
+```shell
+# With CLI options
+npx meadow-integration csvtransform ./books.csv \
+  -e Book -n GUIDBook -g "Book_{~D:Record.id~}" \
+  -o books.json
+
+# With mapping file
+npx meadow-integration csvtransform ./books.csv \
+  -m mapping_Book.json -o books.json
+
+# Merging into existing comprehension
+npx meadow-integration csvtransform ./books.csv \
+  -m mapping_Author.json \
+  -i books.json -o books.json
+```
+
+---
+
+## tsvtransform
+
+Transform a TSV file into a Comprehension.  Same interface as `csvtransform` but uses tab delimiters.
+
+```shell
+npx meadow-integration tsvtransform <file> [options]
+```
+
+All options are the same as `csvtransform`.  The delimiter is automatically set to tab (`\t`).
+
+---
+
+## jsonarraytransform
+
+Transform a JSON array file into a Comprehension.
+
+```shell
+npx meadow-integration jsonarraytransform <file> [options]
+```
+
+All options are the same as `csvtransform`.  The input file must contain a valid JSON array.
+
+---
+
+## comprehensionintersect
+
+Merge two Comprehension files together.  Records with the same GUID are merged (later values overwrite earlier ones).
+
+```shell
+npx meadow-integration comprehensionintersect <primary_file> [options]
+```
+
+**Options:**
+| Option | Description |
+|--------|-------------|
+| `-i, --intersect <path>` | Secondary comprehension file to merge |
+| `-e, --entity <name>` | Entity name to merge (auto-detected if omitted) |
+| `-o, --output <path>` | Output file path |
+
+**Example:**
+```shell
+npx meadow-integration comprehensionintersect Set1.json \
+  -i Set2.json -e Document -o merged.json
+```
+
+---
+
+## comprehensionarray
+
+Convert an object-keyed Comprehension into a JSON array.
+
+```shell
+npx meadow-integration comprehensionarray <file> [options]
+```
+
+**Options:**
+| Option | Description |
+|--------|-------------|
+| `-e, --entity <name>` | Entity to extract (auto-detected if omitted) |
+| `-o, --output <path>` | Output file path |
+
+**Example:**
+```shell
+npx meadow-integration comprehensionarray books.json -e Book -o books-array.json
+```
+
+---
+
+## objectarraytocsv
+
+Convert a JSON array or entity comprehension to CSV format.
+
+```shell
+npx meadow-integration objectarraytocsv <file> [options]
+```
+
+**Options:**
+| Option | Description |
+|--------|-------------|
+| `-e, --entity <name>` | Entity key to extract from the file (optional; expects a plain array if omitted) |
+| `-o, --output <path>` | Output CSV file path |
+
+Nested objects are flattened using dot notation (e.g. `address.city`).
+
+---
+
+## load_comprehension
+
+Push a Comprehension to Meadow REST APIs via the Integration Adapter.
+
+```shell
+npx meadow-integration load_comprehension <file> [options]
+```
+
+**Options:**
+| Option | Description |
+|--------|-------------|
+| `-p, --prefix <prefix>` | GUID prefix for the push |
+| `-e, --entityguidprefix <prefix>` | Per-entity GUID prefix |
+
+This command automatically creates Integration Adapters for each entity in the comprehension and pushes records via upsert operations to the configured Meadow server.
+
+---
+
+## entitycomprehensionsfromtabularfolders
+
+Generate entity comprehensions from a folder of tabular data files.
+
+```shell
+npx meadow-integration entitycomprehensionsfromtabularfolders <folder> [options]
+```
+
+**Options:**
+| Option | Description |
+|--------|-------------|
+| `-e, --entity <name>` | Force all files to a specific entity |
+| `-m, --mapping <path>` | Mapping hints file |
+| `-o, --output <path>` | Output file path |
+
+---
+
+## serve
+
+Start the Meadow Integration REST API server.  See [REST API Reference](rest-api-reference.md) for full endpoint documentation.
+
+```shell
+npx meadow-integration serve [options]
+```
+
+**Aliases:** `server`, `rest`
+
+**Options:**
+| Option | Description |
+|--------|-------------|
+| `-p, --port <port>` | Port to listen on (default: `8086`) |
+
+The `MEADOW_INTEGRATION_PORT` environment variable is also respected.
+
+**Example:**
+```shell
+npx meadow-integration serve -p 3000
+```