meadow-integration 1.0.5 → 1.0.7

package/docs/cli/jsonarraytransform.md

@@ -0,0 +1,166 @@
+# jsonarraytransform
+
+Transform a JSON array file into a Comprehension JSON file. This command provides the same mapping and configuration capabilities as [csvtransform](csvtransform.md), but reads from a JSON array instead of a CSV file.
+
+**Aliases:** `jsonarray_t`, `jsonarray_transform`
+
+## Usage
+
+```shell
+mdwint jsonarraytransform <file> [options]
+```
+
+## Arguments
+
+| Argument | Required | Description |
+|----------|----------|-------------|
+| `<file>` | Yes | Path to the JSON array file to transform |
+
+## Options
+
+| Option | Description | Default |
+|--------|-------------|---------|
+| `-e, --entity <name>` | Entity name in the Comprehension | Auto-detected from filename |
+| `-n, --guidname <name>` | Name of the GUID column in the generated Comprehension | `GUID<Entity>` |
+| `-g, --guidtemplate <template>` | Pict template for generating the entity GUID | `GUID-<Entity>-{~Data:Record.<first_key>~}` |
+| `-c, --columns <mappings>` | Inline column mappings in `Key={~D:col~}` format | Auto-detected 1:1 mapping |
+| `-m, --mappingfile <path>` | Path to a JSON mapping file | -- |
+| `-i, --incoming <path>` | Existing Comprehension file to merge into | -- |
+| `-o, --output <path>` | Output file path | `./JSON-Comprehension-<filename>.json` |
+| `-x, --extended` | Output full operation state, not just the Comprehension | `false` |
+
+## Input Format
+
+The input file must contain a valid JSON array of objects:
+
+```json
+[
+  { "id": 1, "title": "The Hunger Games", "author": "Suzanne Collins" },
+  { "id": 2, "title": "Harry Potter", "author": "J.K. Rowling" },
+  { "id": 3, "title": "To Kill a Mockingbird", "author": "Harper Lee" }
+]
+```
+
+The command validates that:
+- The file contains valid JSON
+- The parsed value is an array
+- The array contains at least one record
+
+## Configuration Priority
+
+The same three-layer configuration priority applies as with `csvtransform`:
+
+1. **Implicit** -- Auto-detected from the first record's keys
+2. **Explicit** -- Loaded from a mapping file via `-m`
+3. **User** -- Command-line options (`-e`, `-n`, `-g`, `-c`)
+
+Each layer overrides the previous.
+
+## Output Format
+
+The output is a standard Comprehension JSON object:
+
+```json
+{
+  "Book": {
+    "Book_1": {
+      "GUIDBook": "Book_1",
+      "Title": "The Hunger Games",
+      "Author": "Suzanne Collins"
+    },
+    "Book_2": {
+      "GUIDBook": "Book_2",
+      "Title": "Harry Potter",
+      "Author": "J.K. Rowling"
+    }
+  }
+}
+```
+
+## Examples
+
+### Transform with a mapping file
+
+```shell
+mdwint jsonarraytransform ./data/books-array.json \
+  -m mapping_Book.json \
+  -o books-comprehension.json
+```
+
+### Transform with CLI options
+
+```shell
+mdwint jsonarraytransform ./data/books-array.json \
+  -e Book \
+  -n GUIDBook \
+  -g "Book_{~D:Record.id~}" \
+  -o books.json
+```
+
+### Implicit transform
+
+```shell
+mdwint jsonarraytransform ./data/books-array.json -o books.json
+```
+
+Entity name, GUID template, and mappings are all auto-detected from the filename and the keys of the first record in the array.
+
+### Merge into an existing Comprehension
+
+```shell
+mdwint jsonarraytransform ./data/authors.json \
+  -m mapping_Author.json \
+  -i existing-store.json \
+  -o existing-store.json
+```
+
+### Inline column mappings
+
+```shell
+mdwint jsonarraytransform ./data/books-array.json \
+  -e Book \
+  -g "Book_{~D:Record.id~}" \
+  -c "Title={~D:Record.title~},Author={~D:Record.author~}" \
+  -o books.json
+```
+
+### Extended output for debugging
+
+```shell
+mdwint jsonarraytransform ./data/books-array.json \
+  -m mapping_Book.json \
+  -x \
+  -o debug-output.json
+```
+
+### Round-trip workflow: CSV to JSON array to Comprehension
+
+```shell
+# 1. Transform CSV to a Comprehension
+mdwint csvtransform ./data/books.csv -e Book -o books-object.json
+
+# 2. Convert to array format
+mdwint comprehensionarray books-object.json -e Book -o books-array.json
+
+# 3. Re-transform from JSON array with a different mapping
+mdwint jsonarraytransform books-array.json -m new-mapping.json -o remapped.json
+```
+
+## Tips
+
+- This command loads the entire JSON file into memory at once, unlike `csvtransform` which streams line by line. For very large datasets, consider converting to CSV first.
+- The input file must be a JSON array at the top level (starting with `[`). If your data is nested inside an object, extract the array first.
+- All features available in `csvtransform` work here, including Solvers, `MultipleGUIDUniqueness`, and mapping files.
+- The default output filename uses a `JSON-Comprehension-` prefix rather than `CSV-Comprehension-`.
+
+## Notes
+
+- Unlike `csvtransform` and `tsvtransform`, this command does not have a `-q` (quotedelimiter) option since JSON parsing handles quoting natively.
+- The command processes records synchronously (not streamed), so memory usage is proportional to file size.
+
+## See Also
+
+- [csvtransform](csvtransform.md) -- CSV equivalent
+- [tsvtransform](tsvtransform.md) -- TSV equivalent
+- [comprehensionarray](comprehensionarray.md) -- Convert Comprehension objects to arrays
+- [Mapping Files](../mapping-files.md) -- Full mapping file specification

package/docs/cli/load-comprehension.md

@@ -0,0 +1,129 @@
+# load_comprehension
+
+Push a Comprehension JSON file to Meadow REST APIs via the Integration Adapter. This command automatically creates an Integration Adapter for each entity found in the Comprehension and performs upsert operations against the configured Meadow server endpoints.
+
+**Aliases:** `load`, `push`
+
+## Usage
+
+```shell
+mdwint load_comprehension <comprehension_file> [options]
+```
+
+## Arguments
+
+| Argument | Required | Description |
+|----------|----------|-------------|
+| `<comprehension_file>` | Yes | Path to the Comprehension JSON file to push |
+
+## Options
+
+| Option | Description | Default |
+|--------|-------------|---------|
+| `-p, --prefix <guid_prefix>` | GUID prefix applied to the entire Comprehension push. Used to namespace records within the Integration Adapter's set management. | -- |
+| `-e, --entityguidprefix <prefix>` | GUID prefix applied per entity. Useful for distinguishing records from different import batches. | -- |
+
+## How It Works
+
+1. The Comprehension file is loaded from disk.
+2. For each entity key in the Comprehension (e.g., `Book`, `Author`, `BookAuthorJoin`), an Integration Adapter is instantiated.
+3. Each Integration Adapter is configured with the entity name and optional GUID prefixes.
+4. All records for each entity are added to the adapter's source record set.
+5. The adapter's `integrateRecords` method is called, which marshals each record and pushes it to the Meadow REST API via upsert (create or update).
+
+The push operation uses the `SimpleMarshal` and `ForceMarshal` modes, meaning records are pushed as-is without complex field transformation.
+
+## Prerequisites
+
+The Meadow server must be running and accessible. The server URL and credentials are configured through the Fable settings (typically via a `meadow-integration-config.json` or environment variables).
+
+## Examples
+
+### Basic push
+
+```shell
+mdwint load_comprehension ./bookstore-comprehension.json
+```
+
+Pushes all entities (Book, Author, BookAuthorJoin, etc.) found in the Comprehension to the configured Meadow API.
+
+### Push with a GUID prefix
+
+```shell
+mdwint load_comprehension ./bookstore-comprehension.json -p "Import-2024-Q1"
+```
+
+The prefix is applied to the Integration Adapter's set GUID marshal prefix, helping to namespace this import batch.
+
+### Push with entity GUID prefix
+
+```shell
+mdwint load_comprehension ./bookstore-comprehension.json \
+  -e "BookstoreDemo"
+```
+
+### Push with both prefixes
+
+```shell
+mdwint push ./bookstore-comprehension.json \
+  -p "Import-2024" \
+  -e "Demo"
+```
+
+### Using the alias
+
+```shell
+mdwint push ./data/comprehension.json
+mdwint load ./data/comprehension.json
+```
+
+### Full workflow: CSV to API
+
+```shell
+# 1. Transform CSV to Comprehension
+mdwint csvtransform ./data/books.csv \
+  -m mapping_Book.json \
+  -o bookstore.json
+
+# 2. Add Author entities
+mdwint csvtransform ./data/books.csv \
+  -m mapping_Author.json \
+  -i bookstore.json \
+  -o bookstore.json
+
+# 3. Push to Meadow API
+mdwint load_comprehension ./bookstore.json -p "BookImport-001"
+```
+
+## Console Output
+
+The command logs progress throughout the push operation:
+
+```
+Pushing comprehension file [./bookstore.json] to the Meadow Endpoints APIs.
+Initializing and configuring data integration adapters...
+Loading Comprehension File...
+Wiring up Integration Adapters...
+Finished importing comprehension file.
+```
+
+Errors during the push (e.g., network failures, API errors) are logged with details.
+
+## Tips
+
+- Ensure the Meadow server is running and reachable before running this command.
+- The command processes entities sequentially using an anticipation chain. Large Comprehensions with many entities will take proportionally longer.
+- The GUID prefix options are useful when running multiple import batches against the same Meadow server, helping to identify and manage records from different sources.
+- The entity abbreviation for the Integration Adapter is automatically derived from the capital letters of the entity name (e.g., `BookAuthorJoin` becomes `BAJ`).
+
+## Notes
+
+- This command requires a configured Meadow server. Without proper server configuration, the adapters will fail to connect.
+- The `SimpleMarshal` mode means records are pushed with their field values as-is. No additional field transformation or validation is performed beyond what the Meadow API enforces.
+- The command does not call back on completion of the server process -- it waits for all integration operations to finish before exiting.
+
+## See Also
+
+- [Integration Adapter](../integration-adapter.md) -- Detailed adapter configuration and behavior
+- [data-clone](data-clone.md) -- Clone data from a Meadow API to a local database
+- [Comprehensions](../comprehensions.md) -- Core data structure documentation

package/docs/cli/objectarraytocsv.md

@@ -0,0 +1,159 @@
+# objectarraytocsv
+
+Convert a JSON array of objects or an entity Comprehension into a CSV file. Nested objects are automatically flattened using dot notation. This command is useful for exporting data to spreadsheets, sharing with tools that expect CSV input, or round-tripping data through the integration pipeline.
+
+**Aliases:** `object_array_to_csv`, `array_to_csv`
+
+## Usage
+
+```shell
+mdwint objectarraytocsv <file> [options]
+```
+
+## Arguments
+
+| Argument | Required | Description |
+|----------|----------|-------------|
+| `<file>` | Yes | Path to the JSON file containing the data to export |
+
+## Options
+
+| Option | Description | Default |
+|--------|-------------|---------|
+| `-e, --entity <name>` | Entity key to extract from a Comprehension object. Omit if the file is a plain JSON array. | `false` (expects array) |
+| `-o, --output <path>` | Output CSV file path | `./Flattened-Object-<filename>.csv` or `./Flattened-Object-<filename>-Entity-<entity>.csv` |
+
+## Input Formats
+
+The command accepts several input formats:
+
+### Plain JSON Array
+
+A top-level array of objects (no `-e` flag needed):
+
+```json
+[
+  { "id": 1, "name": "Alice", "email": "alice@example.com" },
+  { "id": 2, "name": "Bob", "email": "bob@example.com" }
+]
+```
+
+### Comprehension with Entity Flag
+
+An object-keyed Comprehension, where you specify which entity to extract with `-e`:
+
+```json
+{
+  "User": {
+    "User_1": { "GUIDUser": "User_1", "Name": "Alice" },
+    "User_2": { "GUIDUser": "User_2", "Name": "Bob" }
+  }
+}
+```
+
+### Single-Entity Comprehension (Auto-Detected)
+
+If the input file is an object with a single key whose value is an object (not an array), the entity is auto-detected:
+
+```json
+{
+  "Book": {
+    "Book_1": { "Title": "The Hunger Games" },
+    "Book_2": { "Title": "Harry Potter" }
+  }
+}
+```
+
+## Output Format
+
+The output is a standard CSV file with:
+
+- A header row containing all unique field names across all records, sorted alphabetically
+- One data row per record
+- Nested objects flattened with dot notation (e.g., `address.city`, `address.zip`)
+- Values containing commas, double quotes, or newlines are properly escaped
+
+Example output:
+
+```csv
+GUIDBook,ISBN,Language,Title
+Book_1,0439023483,eng,The Hunger Games
+Book_2,0439554934,eng,Harry Potter
+```
+
+## Examples
+
+### Export a plain JSON array to CSV
+
+```shell
+mdwint objectarraytocsv ./data/books-array.json -o books.csv
+```
+
+### Export from a Comprehension with entity flag
+
+```shell
+mdwint objectarraytocsv ./store.json -e Book -o books.csv
+```
+
+### Auto-detect single entity
+
+```shell
+mdwint objectarraytocsv ./single-entity-comprehension.json -o export.csv
+```
+
+### Using the alias
+
+```shell
+mdwint array_to_csv ./data.json -o export.csv
+```
+
+### Full pipeline: CSV to Comprehension and back to CSV
+
+```shell
+# 1. Transform CSV to Comprehension
+mdwint csvtransform ./raw-books.csv -m mapping_Book.json -o books.json
+
+# 2. Convert Comprehension to array
+mdwint comprehensionarray ./books.json -e Book -o books-array.json
+
+# 3. Export array to CSV
+mdwint objectarraytocsv ./books-array.json -o books-export.csv
+```
+
+### Export with nested objects
+
+Given input:
+
+```json
+[
+  { "name": "Alice", "address": { "city": "Seattle", "state": "WA" } },
+  { "name": "Bob", "address": { "city": "Portland", "state": "OR" } }
+]
+```
+
+The output CSV will have flattened column headers:
+
+```csv
+address.city,address.state,name
+Seattle,WA,Alice
+Portland,OR,Bob
+```
+
+## Tips
+
+- If your Comprehension has multiple entity keys and you do not specify `-e`, the command will error and list the available entity keys. Always use `-e` for multi-entity Comprehensions.
+- Column headers in the CSV are sorted alphabetically. This means the column order may differ from the original data or Comprehension.
+- For Comprehensions in object format (not array format), use `comprehensionarray` first to convert to an array, then pipe to this command. Alternatively, pass the Comprehension directly with `-e` to extract the entity.
+- The command handles `null` and `undefined` values by outputting empty strings in the CSV.
+
+## Notes
+
+- Arrays within records are serialized as their string representation in the CSV output.
+- The output file uses streaming writes, making it efficient for large datasets.
+- If no records are found in the input, the command will error.
+
+## See Also
+
+- [comprehensionarray](comprehensionarray.md) -- Convert object Comprehension to array format
+- [csvtransform](csvtransform.md) -- Transform CSV files into Comprehensions
+- [Comprehensions](../comprehensions.md) -- Data structure documentation

package/docs/cli/overview.md

@@ -0,0 +1,96 @@
+# CLI Overview
+
+The `meadow-integration` CLI provides commands for transforming, analyzing, merging, and loading tabular data into the Meadow ecosystem. It is available as the `mdwint` binary when installed globally, or can be run directly with `npx`.
+
+## Installation
+
+```shell
+# Global install
+npm install -g meadow-integration
+
+# Or run directly with npx
+npx meadow-integration [command] [options]
+
+# Or from within the repository
+npm start -- [command] [options]
+```
+
+## Commands
+
+| Command | Aliases | Description |
+|---------|---------|-------------|
+| [csvcheck](csvcheck.md) | `csv_c`, `csv_check` | Analyze a CSV file and produce column-level statistics |
+| [csvtransform](csvtransform.md) | `csv_t`, `csv_transform` | Transform a CSV file into a Comprehension JSON file |
+| [tsvtransform](tsvtransform.md) | `tsv_t`, `tsv_transform` | Transform a TSV file into a Comprehension JSON file |
+| [jsonarraytransform](jsonarraytransform.md) | `jsonarray_t`, `jsonarray_transform` | Transform a JSON array file into a Comprehension |
+| [comprehensionintersect](comprehensionintersect.md) | `intersect` | Merge two Comprehension JSON files together |
+| [comprehensionarray](comprehensionarray.md) | `comprehension_to_array`, `array` | Convert an object-keyed Comprehension into a JSON array |
+| [objectarraytocsv](objectarraytocsv.md) | `object_array_to_csv`, `array_to_csv` | Convert a JSON array or Comprehension to CSV format |
+| [load_comprehension](load-comprehension.md) | `load`, `push` | Push a Comprehension to Meadow REST APIs |
+| [data-clone](data-clone.md) | `clone`, `sync` | Clone data from a Meadow API source to a local database |
+| [serve](serve.md) | `server`, `rest` | Start the Meadow Integration REST API server |
+
+## Typical Workflows
+
+### Analyze and Transform CSV Data
+
+```shell
+# 1. Inspect the CSV structure
+mdwint csvcheck data.csv -o stats.json
+
+# 2. Transform to a Comprehension
+mdwint csvtransform data.csv -m mapping.json -o comprehension.json
+```
+
+### Build a Multi-Entity Comprehension
+
+```shell
+# 1. Create the first entity
+mdwint csvtransform data.csv -m mapping_Book.json -o store.json
+
+# 2. Add another entity to the same file
+mdwint csvtransform data.csv -m mapping_Author.json -i store.json -o store.json
+
+# 3. Add join records
+mdwint csvtransform data.csv -m mapping_Join.json -i store.json -o store.json
+```
+
+### Merge Data from Multiple Sources
+
+```shell
+# 1. Transform each source file
+mdwint csvtransform source_a.csv -e Item -o set_a.json
+mdwint csvtransform source_b.csv -e Item -o set_b.json
+
+# 2. Merge by matching GUIDs
+mdwint comprehensionintersect set_a.json -i set_b.json -e Item -o merged.json
+```
+
+### Export Comprehension to CSV
+
+```shell
+# 1. Convert from object to array format
+mdwint comprehensionarray comprehension.json -e MyEntity -o array.json
+
+# 2. Export array to CSV
+mdwint objectarraytocsv array.json -o export.csv
+```
+
+### Push Data to a Meadow Server
+
+```shell
+mdwint load_comprehension comprehension.json -p "Import-2024"
+```
+
+### Clone API Data to a Local Database
+
+```shell
+mdwint data-clone -a https://api.example.com -d MySQL -dh localhost -dn mydb
+```
+
+## Related Documentation
+
+- [Comprehensions](../comprehensions.md) -- Core data structure concepts
+- [Mapping Files](../mapping-files.md) -- Detailed mapping file reference
+- [REST API Reference](../rest-api-reference.md) -- Server endpoint documentation
+- [Integration Adapter](../integration-adapter.md) -- Programmatic push/pull operations

package/docs/cli/serve.md

@@ -0,0 +1,102 @@
+# serve
+
+Start the Meadow Integration REST API server. This provides HTTP endpoints for performing CSV checks, CSV transforms, comprehension operations, and other integration tasks programmatically over a network.
+
+**Aliases:** `server`, `rest`
+
+## Usage
+
+```shell
+mdwint serve [options]
+```
+
+## Options
+
+| Option | Description | Default |
+|--------|-------------|---------|
+| `-p, --port <port>` | Port number for the server to listen on | `8086` |
+
+## Environment Variables
+
+| Variable | Description |
+|----------|-------------|
+| `MEADOW_INTEGRATION_PORT` | Override the default port. Takes precedence over the default but is overridden by the `-p` CLI flag. |
+
+## Port Resolution Priority
+
+The port is resolved in the following order (highest priority first):
+
+1. `-p` CLI flag
+2. `MEADOW_INTEGRATION_PORT` environment variable
+3. Default value: `8086`
+
+## Examples
+
+### Start on the default port
+
+```shell
+mdwint serve
+```
+
+Starts the server on port 8086.
+
+### Start on a custom port
+
+```shell
+mdwint serve -p 3000
+```
+
+### Using an environment variable
+
+```shell
+MEADOW_INTEGRATION_PORT=9090 mdwint serve
+```
+
+### Using the alias
+
+```shell
+mdwint server -p 3000
+mdwint rest -p 3000
+```
+
+### Run in the background
+
+```shell
+mdwint serve -p 8086 &
+```
+
+## Console Output
+
+```
+Starting Meadow Integration REST server on port 8086...
+```
+
+The server process stays alive after starting. It does not exit on its own -- use `Ctrl+C` or send a termination signal to stop it.
+
+## API Endpoints
+
+Once the server is running, the REST API endpoints are available for HTTP requests. See the [REST API Reference](../rest-api-reference.md) for full endpoint documentation.
+
+The server exposes endpoints for the same operations available via the CLI:
+
+- CSV check
+- CSV transform
+- Comprehension array conversion
+- Comprehension intersection
+- Object array to CSV conversion
+
+## Tips
+
+- The server is built on Orator (Restify). It supports standard HTTP methods and JSON request/response bodies.
+- For production deployments, consider running the server behind a reverse proxy (e.g., nginx) and using process management (e.g., PM2, systemd, or Ultravisor).
+- The server does not implement authentication by default. If exposing it beyond localhost, add appropriate network-level access controls.
+
+## Notes
+
+- The server process intentionally does not call back after starting, keeping the Node.js process alive indefinitely.
+- If the server fails to start (e.g., port already in use), an error is logged and the process exits.
+
+## See Also
+
+- [REST API Reference](../rest-api-reference.md) -- Full endpoint documentation
+- [Programmatic API](../programmatic-api.md) -- Using meadow-integration as a library