meadow-integration 1.0.1 → 1.0.4

package/docs/comprehensions.md

@@ -0,0 +1,98 @@
+# Comprehensions
+
+A Comprehension is the core data structure in meadow-integration.  It is a JSON object that stores entity records keyed by their GUID, providing fast lookup and easy merging across data sources.
+
+## Object Format
+
+The standard comprehension format stores records as properties of an entity object:
+
+```json
+{
+  "EntityName": {
+    "GUID-1": { "GUIDEntityName": "GUID-1", "Field1": "value", "Field2": "value" },
+    "GUID-2": { "GUIDEntityName": "GUID-2", "Field1": "value", "Field2": "value" }
+  }
+}
+```
+
+### Benefits of Object Format
+
+- **O(1) lookup** by GUID -- no scanning required
+- **Natural deduplication** -- duplicate GUIDs merge automatically
+- **Easy merging** -- `Object.assign()` combines records from multiple sources
+- **Multi-entity support** -- one file can hold Books, Authors, and Joins
+
+## Array Format
+
+For export or consumption by other tools, comprehensions can be converted to arrays:
+
+```json
+[
+  { "GUIDEntityName": "GUID-1", "Field1": "value", "Field2": "value" },
+  { "GUIDEntityName": "GUID-2", "Field1": "value", "Field2": "value" }
+]
+```
+
+Convert between formats using:
+
+```shell
+# Object -> Array
+npx meadow-integration comprehensionarray input.json -e MyEntity -o output.json
+
+# Array -> CSV
+npx meadow-integration objectarraytocsv input.json -o output.csv
+```
+
+## Multi-Entity Comprehensions
+
+A single comprehension file can contain records for multiple entity types.  This is created by running `csvtransform` multiple times with different mapping files, passing the `-i` flag to merge into the existing comprehension:
+
+```shell
+# Create Book entities
+npx meadow-integration csvtransform books.csv -m mapping_Book.json -o store.json
+
+# Add Author entities to the same file
+npx meadow-integration csvtransform books.csv -m mapping_Author.json -i store.json -o store.json
+
+# Add BookAuthorJoin entities
+npx meadow-integration csvtransform books.csv -m mapping_Join.json -i store.json -o store.json
+```
+
+Result:
+
+```json
+{
+  "Book": { "Book_1": {...}, "Book_2": {...} },
+  "Author": { "Author_SuzanneCollins": {...}, "Author_JKRowling": {...} },
+  "BookAuthorJoin": { "BAJ_A_SuzanneCollins_B_1": {...} }
+}
+```
+
+## Merging Comprehensions
+
+The `comprehensionintersect` command merges two comprehension files.  Records with matching GUIDs have their fields merged (later values overwrite earlier ones):
+
+```shell
+npx meadow-integration comprehensionintersect file1.json -i file2.json -e MyEntity -o merged.json
+```
+
+This is particularly useful when the same entities have data spread across multiple source files (e.g. housing characteristics and housing costs for the same neighborhoods).
+
+## GUID Design
+
+GUIDs are the primary key for comprehension records.  Good GUID design ensures:
+
+- **Uniqueness** -- Each record gets a distinct key
+- **Determinism** -- The same source data always generates the same GUID
+- **Mergeability** -- Related data from different sources can be matched
+
+### GUID Template Patterns
+
+| Pattern | Use Case |
+|---------|----------|
+| `Entity_{~D:Record.id~}` | Single-column natural key |
+| `{~D:Record.date~}_{~D:Record.seq~}` | Composite key |
+| `{~PascalCaseIdentifier:Record.name~}` | Name-based key (normalized) |
+| `{~D:Record.shared_key~}` | Cross-file merge key |
+
+When the same GUID template is used across multiple csvtransform runs on different source files, the records are automatically merged in the comprehension.

package/docs/cover.md

@@ -0,0 +1,11 @@
+# Meadow Integration <small>1.0</small>
+
+> Data integration toolkit for transforming CSV, TSV, and JSON into Meadow entity comprehensions
+
+- Transform tabular data into structured entity records
+- Merge multiple data sources by shared keys
+- Multi-entity extraction from single sources using Solvers
+- CLI utility, programmatic API, and REST integration
+
+[GitHub](https://github.com/stevenvelozo/meadow-integration)
+[Get Started](#meadow-integration)

package/docs/css/docuserve.css

@@ -0,0 +1,73 @@
+/* ============================================================================
+   Pict Docuserve - Base Styles
+   ============================================================================ */
+
+/* Reset and base */
+*, *::before, *::after {
+	box-sizing: border-box;
+}
+
+html, body {
+	margin: 0;
+	padding: 0;
+	font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif;
+	font-size: 16px;
+	line-height: 1.5;
+	color: #423D37;
+	background-color: #fff;
+	-webkit-font-smoothing: antialiased;
+	-moz-osx-font-smoothing: grayscale;
+}
+
+/* Typography */
+h1, h2, h3, h4, h5, h6 {
+	margin-top: 0;
+	line-height: 1.3;
+}
+
+a {
+	color: #2E7D74;
+	text-decoration: none;
+}
+
+a:hover {
+	color: #256861;
+}
+
+/* Application container */
+#Docuserve-Application-Container {
+	min-height: 100vh;
+}
+
+/* Utility: scrollbar styling */
+::-webkit-scrollbar {
+	width: 8px;
+}
+
+::-webkit-scrollbar-track {
+	background: #F5F0E8;
+}
+
+::-webkit-scrollbar-thumb {
+	background: #D4CCBE;
+	border-radius: 4px;
+}
+
+::-webkit-scrollbar-thumb:hover {
+	background: #B5AA9A;
+}
+
+/* Responsive adjustments */
+@media (max-width: 768px) {
+	html {
+		font-size: 14px;
+	}
+
+	#Docuserve-Sidebar-Container {
+		display: none;
+	}
+
+	.docuserve-body {
+		flex-direction: column;
+	}
+}

package/docs/examples-walkthrough.md

@@ -0,0 +1,138 @@
+# Examples Walkthrough
+
+The `examples/` folder contains runnable scripts that demonstrate meadow-integration features.  Each script is self-contained and writes output to `examples/output/`.
+
+## Data Sources
+
+The examples use data from `docs/examples/data/`:
+
+- **books.csv** -- 10,000 book records (id, title, author, isbn, ratings, etc.)
+- **seattle_neighborhoods/** -- Three Seattle census datasets sharing a Neighborhood Name column
+
+## Example 001: CSV Check
+
+Analyzes a CSV file to understand its structure before transforming it.
+
+```shell
+./examples/Example-001-CSV-Check.sh
+```
+
+**What it does:** Runs `csvcheck` on books.csv and produces a statistics file showing row/column counts, headers, and per-column analysis (empty counts, numeric counts, first/last values).
+
+**When to use:** As the first step with any new data source to understand what you are working with.
+
+## Example 002: CSV Transform (Implicit)
+
+Transforms a CSV with auto-detected settings.
+
+```shell
+./examples/Example-002-CSV-Transform-Implicit.sh
+```
+
+**What it does:** Runs `csvtransform` with no mapping file or options.  The entity name is derived from the filename, the GUID is generated from the first column, and all columns are mapped 1:1.
+
+**When to use:** Quick exploration of what a comprehension looks like before writing a mapping file.
+
+## Example 003: CSV Transform (CLI Options)
+
+Controls the entity name and GUID template via command-line flags.
+
+```shell
+./examples/Example-003-CSV-Transform-CLI-Options.sh
+```
+
+**What it does:** Uses `-e Book`, `-n GUIDBook`, and `-g "Book_{~D:Record.id~}"` to explicitly define the entity and GUID structure.
+
+**When to use:** One-off transforms where you want specific entity naming but do not need column filtering.
+
+## Example 004: CSV Transform (Mapping File)
+
+Uses a JSON mapping file for precise column control.
+
+```shell
+./examples/Example-004-CSV-Transform-Mapping-File.sh
+```
+
+**What it does:** Reads `mapping_books_Book.json` which maps only 7 fields from the 23-column CSV.  Demonstrates computed fields (`PublicationYear` using `Math.roundPrecise`) and static values (`Genre: "Unknown"`).
+
+**When to use:** Production data integration where you want to control exactly which fields are included and how they are named.
+
+## Example 005: Multi-Entity Bookstore
+
+Builds a complete relational data set from a single CSV.
+
+```shell
+./examples/Example-005-Multi-Entity-Bookstore.sh
+```
+
+**What it does:** Runs three `csvtransform` passes on books.csv with different mapping files to create:
+1. **Book** entities (one per book)
+2. **Author** entities (unique authors, split from comma-separated values using Solvers)
+3. **BookAuthorJoin** entities (many-to-many relationships)
+
+Each pass uses `-i` to merge into the same comprehension file.
+
+**When to use:** When a single data source needs to be decomposed into multiple related entity types.
+
+## Example 006: Multi-CSV Intersect
+
+Merges three CSV files that share a common key.
+
+```shell
+./examples/Example-006-Multi-CSV-Intersect.sh
+```
+
+**What it does:** Transforms three Seattle neighborhood CSVs into separate comprehensions keyed on Neighborhood Name, then uses `comprehensionintersect` to merge them into one unified dataset.
+
+**When to use:** When the same entities have data spread across multiple source files (e.g. different database tables or API responses).
+
+## Example 007: Comprehension to Array
+
+Converts from object-keyed format to a JSON array.
+
+```shell
+./examples/Example-007-Comprehension-To-Array.sh
+```
+
+**What it does:** Creates a Book comprehension, then converts it from `{ "Book_1": {...} }` format to `[ {...}, {...} ]` format.
+
+**When to use:** When downstream consumers need an array (UI tables, further processing, export).
+
+## Example 008: Comprehension to CSV
+
+Full round-trip from CSV through comprehension back to CSV.
+
+```shell
+./examples/Example-008-Comprehension-To-CSV.sh
+```
+
+**What it does:** CSV -> Comprehension -> Array -> CSV export.
+
+**When to use:** Reviewing transformed data in a spreadsheet, or creating filtered/cleaned CSVs.
+
+## Example 009: JSON Array Transform
+
+Transforms a JSON array file into a comprehension.
+
+```shell
+./examples/Example-009-JSON-Array-Transform.sh
+```
+
+**What it does:** Creates a JSON array from CSV data, then uses `jsonarraytransform` with a mapping file to create a new comprehension.
+
+**When to use:** When your source data comes as JSON (e.g. API responses) instead of CSV.
+
+## Example 010: Programmatic API
+
+Uses meadow-integration services directly in Node.js code.
+
+```shell
+node examples/Example-010-Programmatic-API.js
+```
+
+**What it does:** Demonstrates three services without the CLI:
+1. **TabularCheck** -- collecting statistics on parsed CSV records
+2. **TabularTransform** -- building a comprehension with explicit configuration
+3. **GUIDMap** -- tracking bidirectional GUID-to-ID mappings
+
+**When to use:** When you are building data integration into a larger application and need programmatic control.

package/docs/index.html

@@ -1,22 +1,39 @@
-<!DOCTYPE html>
+<!doctype html>
 <html lang="en">
-<head>
-  <meta charset="UTF-8">
-  <title>Document</title>
-  <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" />
-  <meta name="description" content="Description">
-  <meta name="viewport" content="width=device-width, initial-scale=1.0, minimum-scale=1.0">
-  <link rel="stylesheet" href="//cdn.jsdelivr.net/npm/docsify@4/lib/themes/vue.css">
-</head>
-<body>
-  <div id="app"></div>
-  <script>
-    window.$docsify = {
-      name: '',
-      repo: ''
-    }
-  </script>
-  <!-- Docsify v4 -->
-  <script src="//cdn.jsdelivr.net/npm/docsify@4"></script>
-</body>
+	<head>
+		<meta charset="utf-8">
+		<meta http-equiv="X-UA-Compatible" content="IE=edge">
+		<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
+		<meta name="description" content="Documentation powered by pict-docuserve">
+
+		<title>Documentation</title>
+
+		<!-- Application Stylesheet -->
+		<link href="css/docuserve.css" rel="stylesheet">
+		<!-- KaTeX stylesheet for LaTeX equation rendering -->
+		<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@0.16.21/dist/katex.min.css">
+		<!-- PICT Dynamic View CSS Container -->
+		<style id="PICT-CSS"></style>
+
+		<!-- Load the PICT library from jsDelivr CDN -->
+		<script src="https://cdn.jsdelivr.net/npm/pict@1/dist/pict.min.js" type="text/javascript"></script>
+		<!-- Bootstrap the Application -->
+		<script type="text/javascript">
+//<![CDATA[
+		Pict.safeOnDocumentReady(() => { Pict.safeLoadPictApplication(PictDocuserve, 2)});
+//]]>
+		</script>
+	</head>
+	<body>
+		<!-- The root container for the Pict application -->
+		<div id="Docuserve-Application-Container"></div>
+
+		<!-- Mermaid diagram rendering -->
+		<script src="https://cdn.jsdelivr.net/npm/mermaid@11/dist/mermaid.min.js"></script>
+		<script>mermaid.initialize({ startOnLoad: false, theme: 'default' });</script>
+		<!-- KaTeX for LaTeX equation rendering -->
+		<script defer src="https://cdn.jsdelivr.net/npm/katex@0.16.21/dist/katex.min.js"></script>
+		<!-- Load the Docuserve PICT Application Bundle from jsDelivr CDN -->
+		<script src="https://cdn.jsdelivr.net/npm/pict-docuserve@0/dist/pict-docuserve.min.js" type="text/javascript"></script>
+	</body>
 </html>

package/docs/integration-adapter.md

@@ -0,0 +1,109 @@
+# Integration Adapter
+
+The Integration Adapter is the bridge between comprehension data and a running Meadow REST API.  It handles record marshaling, GUID generation, schema validation, batch operations, and retry logic.
+
+## Overview
+
+```
+Source Records (from comprehension or external system)
+        |
+        v
+    addSourceRecord()
+        |
+        v
+    marshalRecord()  -- map external GUIDs to Meadow GUIDs
+                     -- validate against Meadow schema
+                     -- truncate strings to schema limits
+                     -- strip reserved fields (CreateDate, etc.)
+        |
+        v
+    pushRecordsToServer()  -- single or bulk upsert
+                           -- automatic retry on failure
+        |
+        v
+    GUIDMap updated  -- Meadow IDs stored for cross-entity lookups
+```
+
+## GUID Marshaling
+
+External system GUIDs are transformed into deterministic Meadow GUIDs using this pattern:
+
+```
+{AdapterSetGUIDMarshalPrefix}-{EntityGUIDMarshalPrefix}-{ExternalGUID}
+```
+
+For example, with default settings and entity "Book":
+
+```
+External GUID: "12345"
+Meadow GUID:   "INTG-DEF-E-Book-12345"
+```
+
+This ensures GUIDs are unique across integration sets and entities.
+
+### Cross-Entity GUID Resolution
+
+When a source record contains GUID fields for other entities (e.g. `GUIDAuthor` on a BookAuthorJoin record), the adapter looks up the corresponding Meadow ID via the GUIDMap:
+
+```javascript
+// Source record: { GUIDBookAuthorJoin: "BAJ_1", GUIDBook: "Book_1", GUIDAuthor: "Author_5" }
+// Adapter marshals:
+//   - GUIDBookAuthorJoin -> "INTG-DEF-E-BookAuthorJoin-BAJ_1"
+//   - GUIDBook -> looks up Meadow ID for external GUID "Book_1" -> IDBook: 42
+//   - GUIDAuthor -> looks up Meadow ID for external GUID "Author_5" -> IDAuthor: 17
+```
+
+This is why entity integration order matters -- entities that are referenced by other entities should be integrated first so their IDs are available in the GUIDMap.
+
+## Batch Processing
+
+The adapter automatically switches between single and bulk upsert modes:
+
+- **Below threshold** (`< RecordThresholdForBulkUpsert`, default 1000): Records are upserted one at a time
+- **Above threshold**: Records are batched into groups of `BulkUpsertBatchSize` (default 100) and sent as bulk upserts
+
+## Retry Logic
+
+Failed upsert operations are retried up to `RecordPushRetryThreshold` times (default 5), with a hard cap of 50 retries.  The adapter validates server responses to confirm:
+
+1. The response contains the entity ID field
+2. The ID is a positive number
+3. The response GUID matches the sent GUID
+
+If validation fails, the record is retried.
+
+## Schema Validation
+
+When `integrateRecords()` is called, the adapter fetches the entity schema from the Meadow API (`GET /Entity/Schema`).  During marshaling:
+
+- String fields are truncated to their schema-defined `size`
+- Non-string fields are passed through
+- Unknown fields (not in schema) are dropped
+- Reserved fields (`CreateDate`, `UpdateDate`, `Deleted`, `DeleteDate`) are always stripped
+
+## Delete Operations
+
+Records with `Deleted: true` in the source are queued for deletion.  The adapter looks up each record by GUID via the API, then issues a DELETE request using the Meadow ID.
+
+## CLI Usage
+
+The `load_comprehension` command wraps the Integration Adapter for CLI use:
+
+```shell
+npx meadow-integration load_comprehension ./my-comprehension.json \
+  -p "MY-PREFIX" \
+  -e "MY-ENTITY-PREFIX"
+```
+
+This automatically creates adapters for every entity in the comprehension and processes them in sequence.
+
+## Static Helper
+
+The `getAdapter()` static method provides a convenient way to get or create an adapter:
+
+```javascript
+const libAdapter = require('meadow-integration/source/Meadow-Service-Integration-Adapter.js');
+
+// Gets existing adapter for 'Book' or creates a new one
+let tmpAdapter = libAdapter.getAdapter(myFable, 'Book', 'BK');
+```

package/docs/mapping-files.md

@@ -0,0 +1,140 @@
+# Mapping Files
+
+Mapping files give you precise control over how source data columns become comprehension fields.  They are JSON files that define the entity name, GUID generation template, and field-by-field mappings using Pict templates.
+
+## Basic Structure
+
+```json
+{
+  "Entity": "Book",
+  "GUIDTemplate": "Book_{~D:Record.id~}",
+  "Mappings": {
+    "Title": "{~D:Record.title~}",
+    "Language": "{~D:Record.language_code~}",
+    "ISBN": "{~D:Record.isbn~}",
+    "Genre": "Unknown"
+  }
+}
+```
+
+### Fields
+
+| Field | Required | Description |
+|-------|----------|-------------|
+| `Entity` | Yes | The entity name.  Determines the top-level key in the comprehension. |
+| `GUIDTemplate` | Yes | Pict template that generates a unique GUID for each record. |
+| `GUIDName` | No | Name of the GUID column (default: `GUID{Entity}`). |
+| `Mappings` | Yes | Object mapping output field names to Pict template expressions. |
+| `Solvers` | No | Array of fable expression strings to run before mapping. |
+| `MultipleGUIDUniqueness` | No | When `true`, a single source row produces multiple output records. |
+| `ManyfestAddresses` | No | When `true`, use Manyfest dot-notation for nested field assignment. |
+
+## Pict Template Syntax
+
+Templates use the `{~D:...~}` syntax to reference values from the current source record.  Each record is available as `Record.<column>`.
+
+| Template | Description |
+|----------|-------------|
+| `{~D:Record.title~}` | Direct column reference |
+| `{~D:Record.first~} {~D:Record.last~}` | String concatenation |
+| `"Unknown"` | Static/literal value |
+| `{~D:Fable.Math.roundPrecise(Record.year,0)~}` | Function call on the value |
+| `{~PascalCaseIdentifier:Record.name~}` | Template with format modifier |
+
+The full Pict template engine is available, including format modifiers and function calls through the Fable service container.
+
+## Example: Simple Column Mapping
+
+Given a CSV with columns `id, name, email, age`:
+
+```json
+{
+  "Entity": "User",
+  "GUIDTemplate": "User_{~D:Record.id~}",
+  "Mappings": {
+    "DisplayName": "{~D:Record.name~}",
+    "EmailAddress": "{~D:Record.email~}",
+    "Age": "{~D:Record.age~}"
+  }
+}
+```
+
+This produces records like:
+
+```json
+{
+  "GUIDUser": "User_42",
+  "DisplayName": "Alice Smith",
+  "EmailAddress": "alice@example.com",
+  "Age": "30"
+}
+```
+
+## Example: Computed GUID from Multiple Columns
+
+```json
+{
+  "Entity": "Transaction",
+  "GUIDTemplate": "TXN_{~D:Record.date~}_{~D:Record.account_id~}_{~D:Record.seq~}",
+  "Mappings": {
+    "Amount": "{~D:Record.amount~}",
+    "AccountID": "{~D:Record.account_id~}",
+    "TransactionDate": "{~D:Record.date~}"
+  }
+}
+```
+
+Composite GUIDs are useful when the source data has no single unique key.
+
+## Example: Multi-Record Generation with Solvers
+
+When a single source row needs to produce multiple comprehension records, use `MultipleGUIDUniqueness` with a Solver expression that splits a value.
+
+Given a books CSV where the `authors` column contains comma-separated names:
+
+```json
+{
+  "Entity": "Author",
+  "MultipleGUIDUniqueness": true,
+  "Solvers": [
+    "NewRecordsGUIDUniqueness = STRINGGETSEGMENTS(IncomingRecord.authors,\",\")"
+  ],
+  "GUIDTemplate": "Author_{~PascalCaseIdentifier:Record._GUIDUniqueness~}",
+  "Mappings": {
+    "Name": "{~D:Record._GUIDUniqueness~}"
+  }
+}
+```
+
+The Solver splits the comma-separated author names into an array stored in `NewRecordsGUIDUniqueness`.  For each entry, a record is created with `_GUIDUniqueness` set to that entry's value.
+
+## Example: Join Table Mapping
+
+To create many-to-many join records between Books and Authors:
+
+```json
+{
+  "Entity": "BookAuthorJoin",
+  "MultipleGUIDUniqueness": true,
+  "Solvers": [
+    "NewRecordsGUIDUniqueness = STRINGGETSEGMENTS(IncomingRecord.authors,\",\")"
+  ],
+  "GUIDTemplate": "BAJ_A_{~PascalCaseIdentifier:Record._GUIDUniqueness~}_B_{~D:Record.id~}",
+  "Mappings": {
+    "GUIDBook": "Book_{~D:Record.id~}",
+    "GUIDAuthor": "Author_{~PascalCaseIdentifier:Record._GUIDUniqueness~}"
+  }
+}
+```
+
+This generates one join record per book-author pair, with cross-reference GUIDs that match the Book and Author entities.
+
+## Configuration Priority
+
+When running a transform, three layers of configuration are merged:
+
+1. **Implicit** -- Auto-detected from the first record's columns (entity name from filename, 1:1 column mappings)
+2. **Explicit** -- Loaded from a mapping file via `-m`
+3. **User** -- Command-line options (`-e`, `-n`, `-g`, `-c`)
+
+Each layer overrides the previous, so CLI options always win.