meadow-integration 1.0.4 → 1.0.6

package/docs/architecture.md

@@ -0,0 +1,423 @@
+# Architecture
+
+This document describes the architectural design of Meadow Integration, covering both the data transformation and data synchronization pipelines.
+
+## High-Level System Architecture
+
+Meadow Integration sits between external data sources and the Meadow data access layer. It provides three interfaces (CLI, REST, Programmatic) that share a common set of services.
+
+```mermaid
+flowchart TB
+    subgraph External["External Data Sources"]
+        CSV["CSV Files"]
+        TSV["TSV Files"]
+        JSON["JSON Arrays"]
+        API["Remote Meadow API"]
+    end
+
+    subgraph Interfaces["Interfaces"]
+        CLI["CLI Program\n(pict-service-commandlineutility)"]
+        REST["REST Server\n(Orator + Restify)"]
+        PROG["Programmatic API\n(require meadow-integration)"]
+    end
+
+    subgraph Services["Core Services"]
+        TC["TabularCheck"]
+        TT["TabularTransform"]
+        IA["IntegrationAdapter"]
+        GM["GUIDMap"]
+        CM["ConnectionManager"]
+        CRC["CloneRestClient"]
+        SYNC["Sync"]
+        SEI["SyncEntityInitial"]
+        SEO["SyncEntityOngoing"]
+        OP["Operation"]
+    end
+
+    subgraph Targets["Targets"]
+        MAPI["Meadow REST API\n(write)"]
+        DB["Local Database\n(MySQL / MSSQL)"]
+    end
+
+    CSV --> CLI
+    TSV --> CLI
+    JSON --> CLI
+    CSV --> REST
+    TSV --> REST
+    JSON --> REST
+
+    CLI --> TC
+    CLI --> TT
+    CLI --> IA
+    CLI --> SYNC
+    REST --> TC
+    REST --> TT
+    REST --> IA
+    PROG --> TC
+    PROG --> TT
+    PROG --> IA
+    PROG --> SYNC
+
+    TT --> IA
+    IA --> GM
+    IA --> MAPI
+
+    API --> CRC
+    CRC --> SYNC
+    SYNC --> SEI
+    SYNC --> SEO
+    SEI --> CM
+    SEO --> CM
+    CM --> DB
+
+    SYNC --> OP
+    SEI --> OP
+    SEO --> OP
+```
+
+The system divides cleanly into two pipelines that share the `Operation` utility for timing and progress tracking.
+
+## Data Transformation Pipeline
+
+The transformation pipeline converts tabular data into Meadow entity records. Each stage is a discrete service that can be used independently.
+
+```mermaid
+flowchart LR
+    subgraph Input["1. Parse"]
+        FILE["Source File\n(CSV / TSV / JSON)"]
+        PARSE["Stream Parser\n(csv-parser / JSON.parse)"]
+    end
+
+    subgraph Transform["2. Transform"]
+        RECORD["Raw Record"]
+        MAPPING["Mapping Config\n(Implicit + Explicit + User)"]
+        TEMPLATE["Pict Template Engine\n({~D:Record.col~})"]
+        SOLVER["Solvers\n(Expression Parser)"]
+    end
+
+    subgraph Collect["3. Collect"]
+        COMP["Comprehension\n{Entity: {GUID: Record}}"]
+        MERGE["Intersect\n(merge by GUID)"]
+    end
+
+    subgraph Push["4. Push"]
+        ADAPTER["IntegrationAdapter"]
+        MARSHAL["Marshal Record\n(schema validation,\nstring truncation,\nGUID prefixing)"]
+        GUIDMAP["GUIDMap\n(external <-> Meadow IDs)"]
+        UPSERT["Upsert\n(single or bulk)"]
+    end
+
+    FILE --> PARSE --> RECORD
+    MAPPING --> TEMPLATE
+    RECORD --> TEMPLATE --> COMP
+    RECORD --> SOLVER --> COMP
+    COMP --> MERGE --> COMP
+    COMP --> ADAPTER --> MARSHAL --> GUIDMAP --> UPSERT
+```
+
+### Stage Details
+
+**Parse** -- CSV and TSV files are streamed through a parser that emits one record per row. JSON Array files are loaded and iterated. The `TabularCheck` service can analyze records without transforming them, producing column statistics.
+
+**Transform** -- The `TabularTransform` service applies a three-layer configuration cascade to each record:
+
+1. **Implicit** -- Auto-generated from the first record's keys (column names become field names, the first column is used for GUID generation)
+2. **Explicit** -- Loaded from a mapping file that specifies entity name, GUID template, and column-to-field mappings
+3. **User** -- Command-line overrides for entity name, GUID name, GUID template, and inline column mappings
+
+Each layer merges on top of the previous one using `Object.assign`, so User settings always win.
+
+Pict template expressions resolve column values at transformation time. Solvers (powered by the Fable Expression Parser) enable multi-entity extraction from a single source row by dynamically generating multiple GUID uniqueness entries.
+
+**Collect** -- Transformed records accumulate in a Comprehension object. Records with duplicate GUIDs within the same parse are merged. Records can also be merged with an existing Comprehension loaded from disk.
+
+**Push** -- The `IntegrationAdapter` marshals comprehension records into Meadow-compatible format. It fetches the target entity schema from the Meadow API, validates field types, truncates strings that exceed schema-defined sizes, and strips reserved columns (`CreateDate`, `UpdateDate`, `Deleted`, `DeleteDate`). Cross-entity GUID references are resolved through the `GUIDMap`. Records are pushed via upsert -- individually for small sets, or in configurable bulk batches (default threshold: 1000 records, batch size: 100) for large sets.
+
+## Data Synchronization Pipeline
+
+The Data Clone pipeline replicates entity data from a remote Meadow API into a local relational database.
+
+```mermaid
+flowchart TB
+    subgraph Config["Configuration"]
+        MCFG[".meadow.config.json"]
+        SCHEMA["Extended Schema JSON\n(from Stricture)"]
+        CLIOPTS["CLI Overrides\n(--api_server, --db_host, etc.)"]
+    end
+
+    subgraph Auth["1. Authenticate"]
+        CRC["CloneRestClient"]
+        SESSION["Session Management\n(cookie / token)"]
+    end
+
+    subgraph Connect["2. Connect"]
+        CM["ConnectionManager"]
+        POOL["Connection Pool\n(MySQL or MSSQL)"]
+    end
+
+    subgraph Init["3. Initialize Schema"]
+        LOAD["Load Extended Schema"]
+        CREATE["Create Tables\n(if not exist)"]
+        INDEX["Create Indexes\n(GUID unique, Deleted)"]
+    end
+
+    subgraph Sync["4. Sync Entities"]
+        COMPARE["Compare\nlocal vs. remote\n(max ID, count, UpdateDate)"]
+        DOWNLOAD["Download Pages\n(filtered + sorted)"]
+        WRITE["Marshal + Write\n(create or update)"]
+        PROGRESS["Progress Tracking\n(Operation service)"]
+    end
+
+    MCFG --> CRC
+    CLIOPTS --> CRC
+    MCFG --> CM
+    CLIOPTS --> CM
+    SCHEMA --> LOAD
+
+    CRC --> SESSION --> DOWNLOAD
+    CM --> POOL --> WRITE
+    LOAD --> CREATE --> INDEX
+
+    COMPARE --> DOWNLOAD --> WRITE
+    WRITE --> PROGRESS
+```
+
+### Stage Details
+
+**Authenticate** -- The `CloneRestClient` authenticates with the remote Meadow API by posting credentials to `/Authenticate`. The resulting session cookie or token is attached to all subsequent requests. If no credentials are configured, authentication is skipped (for unauthenticated APIs). HTTP keep-alive is enabled for connection reuse.
+
+**Connect** -- The `ConnectionManager` establishes a connection pool to the local database. It supports MySQL (via `meadow-connection-mysql`) and MSSQL (via `meadow-connection-mssql`). The provider is selected by the `Destination.Provider` configuration key.
+
+**Initialize Schema** -- The Meadow extended schema JSON (produced by Stricture's `build` command) is loaded. For each entity in the schema (or a configured subset), the sync service uses the Meadow provider to create the table if it does not exist. It then creates a unique index on the GUID column and a non-unique index on the Deleted column using the `ConnectionManager`.
+
+**Sync Entities** -- Entities are synced sequentially in the order defined by `SyncEntityList` (or schema order if the list is empty). Two sync strategies are available:
+
+- **Initial** -- Queries the local max ID and the remote max ID and record count. Generates paginated URL partials filtered to records with IDs greater than the local maximum. Downloads each page and creates records locally with identity insert enabled so primary keys match the remote system.
+- **Ongoing** -- Extends Initial sync with `UpdateDate` comparison. After identifying new records by ID, it also compares `UpdateDate` timestamps. Records where the remote `UpdateDate` differs from the local `UpdateDate` by more than 5 milliseconds are updated. This handles both new records and modifications.
+
+## Service Dependency Diagram
+
+All services extend `fable-serviceproviderbase` and register with a Fable instance. The diagram below shows the dependency relationships.
+
+```mermaid
+classDiagram
+    class FableServiceProviderBase {
+        +fable
+        +options
+        +log
+        +serviceType
+    }
+
+    class TabularCheck {
+        +serviceType: TabularCheck
+        +newStatisticsObject()
+        +collectStatistics()
+    }
+
+    class TabularTransform {
+        +serviceType: TabularTransform
+        +newMappingOutcomeObject()
+        +transformRecord()
+        +addRecordToComprehension()
+        +createRecordFromMapping()
+    }
+
+    class IntegrationAdapter {
+        +serviceType: IntegrationAdapter
+        +Entity
+        +addSourceRecord()
+        +integrateRecords()
+        +marshalRecord()
+        +pushRecordsToServer()
+    }
+
+    class GUIDMap {
+        +serviceType: MeadowGUIDMap
+        +mapGUIDToID()
+        +getIDFromGUID()
+        +mapExternalGUIDtoMeadowGUID()
+        +getMeadowIDFromExternalGUID()
+    }
+
+    class ConnectionManager {
+        +serviceType: MeadowConnectionManager
+        +Provider
+        +ConnectionPool
+        +connect()
+        +createIndex()
+    }
+
+    class CloneRestClient {
+        +serviceType: MeadowCloneRestClient
+        +serverURL
+        +authenticate()
+        +deauthenticate()
+        +getJSON()
+        +upsertEntity()
+        +getEntitySet()
+    }
+
+    class Sync {
+        +serviceType: MeadowSync
+        +SyncMode
+        +SyncEntityList
+        +loadMeadowSchema()
+        +syncEntity()
+        +syncAll()
+    }
+
+    class SyncEntityInitial {
+        +serviceType: MeadowSyncEntityInitial
+        +EntitySchema
+        +PageSize
+        +initialize()
+        +sync()
+        +marshalRecord()
+    }
+
+    class SyncEntityOngoing {
+        +serviceType: MeadowSyncEntityOngoing
+        +EntitySchema
+        +PageSize
+        +initialize()
+        +sync()
+        +marshalRecord()
+    }
+
+    class Operation {
+        +timeStamps
+        +progressTrackers
+        +createTimeStamp()
+        +createProgressTracker()
+        +printProgressTrackerStatus()
+    }
+
+    FableServiceProviderBase <|-- TabularCheck
+    FableServiceProviderBase <|-- TabularTransform
+    FableServiceProviderBase <|-- IntegrationAdapter
+    FableServiceProviderBase <|-- GUIDMap
+    FableServiceProviderBase <|-- ConnectionManager
+    FableServiceProviderBase <|-- CloneRestClient
+    FableServiceProviderBase <|-- Sync
+    FableServiceProviderBase <|-- SyncEntityInitial
+    FableServiceProviderBase <|-- SyncEntityOngoing
+
+    IntegrationAdapter --> GUIDMap : uses
+    Sync --> SyncEntityInitial : creates
+    Sync --> SyncEntityOngoing : creates
+    SyncEntityInitial --> Operation : uses
+    SyncEntityOngoing --> Operation : uses
+    SyncEntityInitial ..> CloneRestClient : reads from
+    SyncEntityOngoing ..> CloneRestClient : reads from
+    SyncEntityInitial ..> ConnectionManager : writes to
+    SyncEntityOngoing ..> ConnectionManager : writes to
+```
+
+## Configuration Cascade
+
+Configuration for the CLI flows through multiple layers, each overriding the previous.
+
+```mermaid
+flowchart LR
+    DEF["Default Configuration\n(Default-Meadow-Integration-\nConfiguration.json)"]
+    FILE[".meadow.config.json\n(working directory)"]
+    CLI["Command-Line Flags\n(--api_server, --db_host, etc.)"]
+
+    DEF -->|"base"| MERGED["Resolved Configuration"]
+    FILE -->|"overrides"| MERGED
+    CLI -->|"overrides"| MERGED
+```
+
+For data transformation, the mapping configuration has its own three-layer cascade:
+
+```mermaid
+flowchart LR
+    IMP["Implicit\n(auto-detected from\nfirst record)"]
+    EXP["Explicit\n(mapping file\nvia -m flag)"]
+    USR["User\n(CLI flags:\n-e, -g, -n, -c)"]
+
+    IMP -->|"base"| FINAL["Final Mapping Config"]
+    EXP -->|"overrides"| FINAL
+    USR -->|"overrides"| FINAL
+```
+
+## Sync Mode Comparison
+
+The two sync modes serve different purposes and have different performance characteristics.
+
+```mermaid
+flowchart TB
+    subgraph Initial["Initial Sync"]
+        I1["Query local max ID"]
+        I2["Query remote max ID + count"]
+        I3["Generate paginated URL partials\n(filter: ID > local max)"]
+        I4["Download each page"]
+        I5["For each record:\nRead local by ID"]
+        I6{"Record\nexists?"}
+        I7["Skip"]
+        I8["Create with\nidentity insert"]
+
+        I1 --> I2 --> I3 --> I4 --> I5 --> I6
+        I6 -->|"yes"| I7
+        I6 -->|"no"| I8
+    end
+
+    subgraph Ongoing["Ongoing Sync"]
+        O1["Query local max ID + UpdateDate"]
+        O2["Query remote max ID + UpdateDate + count"]
+        O3["Iterate all records\n(paginated, ID ascending)"]
+        O4["For each record:\nRead local by ID"]
+        O5{"Record\nexists?"}
+        O6{"UpdateDate\ndifference\n> 5ms?"}
+        O7["Update record"]
+        O8["Skip"]
+        O9["Create with\nidentity insert"]
+
+        O1 --> O2 --> O3 --> O4 --> O5
+        O5 -->|"yes"| O6
+        O5 -->|"no"| O9
+        O6 -->|"yes"| O7
+        O6 -->|"no"| O8
+    end
+```
+
+| Aspect | Initial | Ongoing |
+|--------|---------|---------|
+| **Purpose** | First-time clone or catch-up | Incremental sync of changes |
+| **Strategy** | Only downloads records with IDs above local max | Walks all records and compares timestamps |
+| **Handles new records** | Yes | Yes |
+| **Handles updates** | No | Yes (by UpdateDate comparison) |
+| **Performance** | Faster for first clone (skips existing) | Slower per run but keeps data current |
+| **Typical usage** | Run once, then switch to Ongoing | Run on a schedule (cron, Docker) |
+
+## Docker Deployment
+
+The included Dockerfile builds a production image for running Data Clone as a containerized service. The image is based on `node:20-bookworm` and expects a `.meadow.config.json` to be provided at runtime (via volume mount or baked into a derived image).
+
+```mermaid
+flowchart LR
+    subgraph Build["Docker Build"]
+        BASE["node:20-bookworm"]
+        DEPS["npm install --omit=dev"]
+        SRC["Copy source + scripts"]
+    end
+
+    subgraph Runtime["Docker Runtime"]
+        CFG[".meadow.config.json\n(volume mount)"]
+        SCHEMA["Extended Schema\n(volume mount)"]
+        RUN["scripts/run.sh"]
+    end
+
+    subgraph External["External"]
+        REMOTE["Remote Meadow API"]
+        LOCAL["Local Database"]
+    end
+
+    BASE --> DEPS --> SRC --> RUN
+    CFG --> RUN
+    SCHEMA --> RUN
+    RUN --> REMOTE
+    RUN --> LOCAL
+```
+
+The `docker-compose.yml` can be used to run the Data Clone alongside a local MySQL or MSSQL container for development and testing.

package/docs/cli/comprehensionarray.md

@@ -0,0 +1,111 @@
+# comprehensionarray
+
+Convert an object-keyed Comprehension into a JSON array. Comprehensions store records as `{ GUID: record }` objects for fast lookup and merging, but sometimes you need a plain array for export, UI consumption, or further processing.
+
+**Aliases:** `comprehension_to_array`, `array`
+
+## Usage
+
+```shell
+mdwint comprehensionarray <file> [options]
+```
+
+## Arguments
+
+| Argument | Required | Description |
+|----------|----------|-------------|
+| `<file>` | Yes | Path to the Comprehension file to convert |
+
+## Options
+
+| Option | Description | Default |
+|--------|-------------|---------|
+| `-e, --entity <name>` | Entity name to extract from the Comprehension | Auto-detected from the first key |
+| `-o, --output <path>` | Output file path | `./Array-Comprehension-<filename>.json` |
+
+## Input Format
+
+The input is a standard object-keyed Comprehension:
+
+```json
+{
+  "Book": {
+    "Book_1": { "GUIDBook": "Book_1", "Title": "The Hunger Games", "Language": "eng" },
+    "Book_2": { "GUIDBook": "Book_2", "Title": "Harry Potter", "Language": "eng" }
+  }
+}
+```
+
+## Output Format
+
+The output is a JSON array of the record objects:
+
+```json
+[
+  { "GUIDBook": "Book_1", "Title": "The Hunger Games", "Language": "eng" },
+  { "GUIDBook": "Book_2", "Title": "Harry Potter", "Language": "eng" }
+]
+```
+
+The GUID keys are discarded; only the record values are included in the array.
+
+## Examples
+
+### Basic conversion with explicit entity
+
+```shell
+mdwint comprehensionarray ./books.json -e Book -o books-array.json
+```
+
+### Auto-detect entity name
+
+```shell
+mdwint comprehensionarray ./books.json -o books-array.json
+```
+
+If `-e` is omitted, the entity name is inferred from the first key in the Comprehension. If the Comprehension has only one entity, this works automatically.
+
+### Using the alias
+
+```shell
+mdwint array ./books.json -e Book -o books-array.json
+```
+
+### Pipeline: Comprehension to CSV export
+
+This command is commonly used as an intermediate step when exporting to CSV:
+
+```shell
+# Step 1: Convert object Comprehension to array
+mdwint comprehensionarray ./store.json -e Book -o books-array.json
+
+# Step 2: Export array to CSV
+mdwint objectarraytocsv ./books-array.json -o books.csv
+```
+
+### Extract one entity from a multi-entity Comprehension
+
+```shell
+# Extract just the Author records from a multi-entity file
+mdwint comprehensionarray ./bookstore.json -e Author -o authors-array.json
+
+# Extract just the BookAuthorJoin records
+mdwint comprehensionarray ./bookstore.json -e BookAuthorJoin -o joins-array.json
+```
+
+## Tips
+
+- When working with multi-entity Comprehensions, always specify the `-e` flag to select which entity to extract. Without it, only the first entity key is used.
+- This command is non-destructive to the input file. The original Comprehension is not modified.
+- The output array preserves the order in which object keys were enumerated, which is generally insertion order in modern JavaScript engines.
+
+## Notes
+
+- If the specified entity does not exist in the Comprehension, the output will be an empty array.
+- If no entities are found in the Comprehension file, the command will error.
+
+## See Also
+
+- [objectarraytocsv](objectarraytocsv.md) -- Convert a JSON array to CSV format
+- [csvtransform](csvtransform.md) -- Create Comprehensions from CSV files
+- [Comprehensions](../comprehensions.md) -- Object vs. array format documentation

package/docs/cli/comprehensionintersect.md

@@ -0,0 +1,132 @@
+# comprehensionintersect
+
+Merge two Comprehension JSON files together. Records with the same GUID in both files are merged, with values from the secondary file overwriting values in the primary file. This is useful when the same entities have data spread across multiple source files.
+
+**Aliases:** `intersect`
+
+## Usage
+
+```shell
+mdwint comprehensionintersect <primary_file> [options]
+```
+
+## Arguments
+
+| Argument | Required | Description |
+|----------|----------|-------------|
+| `<primary_file>` | Yes | Path to the primary Comprehension file |
+
+## Options
+
+| Option | Description | Default |
+|--------|-------------|---------|
+| `-i, --intersect <path>` | Path to the secondary Comprehension file to merge with the primary | (required) |
+| `-e, --entity <name>` | Entity name to merge | Auto-detected from the first key of the primary Comprehension |
+| `-o, --output <path>` | Output file path | `./Intersected-Comprehension-<filename>.json` |
+
+## How Merging Works
+
+The intersect operation iterates over every record in the secondary Comprehension and matches it to the primary Comprehension by GUID:
+
+- **Matching GUID found in primary**: The record fields are merged. Fields from the secondary file overwrite fields in the primary file. Fields that exist only in the primary are preserved.
+- **No matching GUID in primary**: The record from the secondary file is added to the result.
+- **GUID exists only in primary**: The record is preserved as-is in the output.
+
+This behavior makes `comprehensionintersect` ideal for enriching records with data from additional sources.
+
+## Output Format
+
+The output is a standard Comprehension JSON object containing the merged records:
+
+```json
+{
+  "Neighborhood": {
+    "Capitol Hill": {
+      "GUIDNeighborhood": "Capitol Hill",
+      "MedianHomeValue": "625000",
+      "MedianRent": "1850",
+      "Population": "32000",
+      "MedianAge": "33"
+    }
+  }
+}
+```
+
+## Examples
+
+### Basic intersection
+
+```shell
+mdwint comprehensionintersect set1.json \
+  -i set2.json \
+  -e Document \
+  -o merged.json
+```
+
+### Auto-detect entity name
+
+```shell
+mdwint comprehensionintersect set1.json \
+  -i set2.json \
+  -o merged.json
+```
+
+If `-e` is omitted, the entity name is inferred from the first key in the primary Comprehension file.
+
+### Chain multiple intersections
+
+When you have data spread across three or more source files, chain the intersect operations:
+
+```shell
+# Step 1: Transform each source to a Comprehension
+mdwint csvtransform housing_chars.csv \
+  -e Neighborhood -n GUIDNeighborhood \
+  -g "{~D:Record.Neighborhood Name~}" \
+  -o set_chars.json
+
+mdwint csvtransform housing_costs.csv \
+  -e Neighborhood -n GUIDNeighborhood \
+  -g "{~D:Record.Neighborhood Name~}" \
+  -o set_costs.json
+
+mdwint csvtransform demographics.csv \
+  -e Neighborhood -n GUIDNeighborhood \
+  -g "{~D:Record.Neighborhood Name~}" \
+  -o set_demographics.json
+
+# Step 2: Merge the first two
+mdwint comprehensionintersect set_chars.json \
+  -i set_costs.json \
+  -e Neighborhood \
+  -o merged.json
+
+# Step 3: Merge the third into the result
+mdwint comprehensionintersect merged.json \
+  -i set_demographics.json \
+  -e Neighborhood \
+  -o merged.json
+```
+
+### Using the alias
+
+```shell
+mdwint intersect primary.json -i secondary.json -o result.json
+```
+
+## Tips
+
+- The GUID template must be identical across the source Comprehensions for records to match. Use the same `-g` value or the same mapping file GUID template when generating the source Comprehensions.
+- The secondary file's values overwrite the primary file's values for matching fields. If you want the primary's values to take priority, swap the file arguments.
+- You can intersect Comprehensions that were generated from different file formats (e.g., one from CSV, one from TSV, one from JSON array).
+- For large merge operations with many source files, consider writing a shell script that chains the intersect calls sequentially.
+
+## Notes
+
+- The `-i` option is required. The command will error if no secondary Comprehension file is specified.
+- If no entity is specified and the primary Comprehension has multiple entity keys, the first key is used.
+- The output file can be the same as one of the input files, allowing in-place merge operations.
+
+## See Also
+
+- [csvtransform](csvtransform.md) -- Generate Comprehensions from CSV files
+- [Comprehensions](../comprehensions.md) -- Core data structure and merging concepts