meadow-integration 1.0.4 → 1.0.6

package/docs/implementation-reference.md

@@ -0,0 +1,311 @@
+# Implementation Reference
+
+This document covers the internal structure of Meadow Integration, including module layout, service patterns, configuration resolution, error handling, and extension points.
+
+## Module Structure
+
+```
+meadow-integration/
+├── package.json
+├── Dockerfile
+├── docker-compose.yml
+├── scripts/
+│   └── run.sh
+├── source/
+│   ├── Meadow-Integration.js                          # Main export (programmatic API)
+│   ├── Meadow-Service-Integration-Adapter.js          # IntegrationAdapter service
+│   ├── Meadow-Service-Integration-GUIDMap.js          # GUIDMap service
+│   ├── cli/
+│   │   ├── Meadow-Integration-CLI-Program.js          # CLI program setup
+│   │   ├── Meadow-Integration-CLI-Run.js              # CLI entry point
+│   │   ├── Default-Meadow-Integration-Configuration.json
+│   │   └── commands/
+│   │       ├── Meadow-Integration-Command-CSVCheck.js
+│   │       ├── Meadow-Integration-Command-CSVTransform.js
+│   │       ├── Meadow-Integration-Command-TSVCheck.js
+│   │       ├── Meadow-Integration-Command-TSVTransform.js
+│   │       ├── Meadow-Integration-Command-JSONArrayTransform.js
+│   │       ├── Meadow-Integration-Command-ComprehensionIntersect.js
+│   │       ├── Meadow-Integration-Command-ComprehensionArray.js
+│   │       ├── Meadow-Integration-Command-ComprehensionPush.js
+│   │       ├── Meadow-Integration-Command-ObjectArrayToCSV.js
+│   │       ├── Meadow-Integration-Command-EntityFromTabularFolder.js
+│   │       ├── Meadow-Integration-Command-DataClone.js
+│   │       ├── Meadow-Integration-Command-ConvertXLSMToXLSX.js
+│   │       └── Meadow-Integration-Command-Serve.js
+│   ├── restserver/
+│   │   ├── Meadow-Integration-Server.js               # REST server class
+│   │   ├── Meadow-Integration-Server-Endpoints.js     # Endpoint registration
+│   │   └── endpoints/
+│   │       ├── Endpoint-CSVCheck.js
+│   │       ├── Endpoint-CSVTransform.js
+│   │       ├── Endpoint-TSVCheck.js
+│   │       ├── Endpoint-TSVTransform.js
+│   │       ├── Endpoint-JSONArrayTransform.js
+│   │       ├── Endpoint-ComprehensionIntersect.js
+│   │       ├── Endpoint-ComprehensionArray.js
+│   │       ├── Endpoint-ComprehensionPush.js
+│   │       ├── Endpoint-ObjectArrayToCSV.js
+│   │       └── Endpoint-EntityFromTabularFolder.js
+│   └── services/
+│       ├── clone/
+│       │   ├── Meadow-Service-ConnectionManager.js    # Database connection pooling
+│       │   ├── Meadow-Service-RestClient.js           # Authenticated REST client
+│       │   ├── Meadow-Service-Sync.js                 # Sync orchestrator
+│       │   ├── Meadow-Service-Sync-Entity-Initial.js  # Initial sync strategy
+│       │   ├── Meadow-Service-Sync-Entity-Ongoing.js  # Ongoing sync strategy
+│       │   └── Meadow-Service-Operation.js            # Timing and progress tracking
+│       └── tabular/
+│           ├── Service-TabularCheck.js                # Statistics collection
+│           └── Service-TabularTransform.js            # Record transformation
+├── test/
+├── examples/
+└── docs/
+```
+
+## Service Registration Patterns
+
+All services in Meadow Integration extend `fable-serviceproviderbase`. They follow the Fable service provider pattern for registration and instantiation.
+
+### Registration and Instantiation
+
+Services are registered with a service type name and then instantiated. This gives them access to `this.fable` (the Fable instance), `this.options` (merged configuration), and `this.log` (the Fable logger).
+
+```javascript
+// Register the service type
+this.fable.serviceManager.addServiceType('MeadowConnectionManager', libMeadowConnectionManager);
+
+// Instantiate with options and an optional service hash (name)
+this.fable.serviceManager.instantiateServiceProvider('MeadowConnectionManager', tmpConfig.Destination);
+```
+
+After instantiation, the service is available on the Fable instance by its type name:
+
+```javascript
+this.fable.MeadowConnectionManager.connect(fCallback);
+```
+
+### Lazy Service Initialization
+
+Several services check whether a dependency is already registered before adding it. This prevents duplicate registration when multiple services share a dependency.
+
+```javascript
+if (!this.fable.hasOwnProperty('MeadowGUIDMap'))
+{
+    this.fable.addAndInstantiateServiceType('MeadowGUIDMap', libGUIDMap);
+}
+```
+
+### Named Service Instances
+
+When multiple instances of the same service type are needed (e.g., one `SyncEntity` per table), the third parameter to `instantiateServiceProvider` acts as a service hash:
+
+```javascript
+this.fable.serviceManager.instantiateServiceProvider(
+    'MeadowSyncEntityInitial',
+    tmpSyncEntityOptions,
+    `SyncEntity-${tmpEntitySchema.TableName}`
+);
+```
+
+These are stored in `this.fable.servicesMap.MeadowSyncEntityInitial` keyed by the hash.
+
+## Configuration Resolution Order
+
+### CLI Program Configuration
+
+The CLI program (`pict-service-commandlineutility`) resolves configuration in this order:
+
+1. **Default** -- `Default-Meadow-Integration-Configuration.json` baked into the module
+2. **File** -- `.meadow.config.json` in the current working directory (auto-gathered by the CLI framework)
+3. **Command-line** -- Flags passed to the specific command (e.g., `--api_server`, `--db_host`)
+
+The result is available as `this.fable.ProgramConfiguration`.
+
+### Data Clone Configuration
+
+The `data-clone` command has its own `_resolveConfig()` method that builds the final configuration:
+
+```javascript
+const tmpConfig = JSON.parse(JSON.stringify(this.fable.ProgramConfiguration));
+
+// Apply CLI overrides for Source
+if (this.CommandOptions.api_server) tmpConfig.Source.ServerURL = this.CommandOptions.api_server;
+
+// Apply CLI overrides for Destination
+if (this.CommandOptions.db_host) tmpConfig.Destination[tmpProvider].server = this.CommandOptions.db_host;
+```
+
+### Transformation Mapping Configuration
+
+The `TabularTransform` service uses a three-layer cascade for mapping configuration:
+
+1. **Implicit** -- Auto-generated from the first incoming record. Entity name is derived from the filename, GUID template uses the first column, and all columns are mapped 1:1.
+2. **Explicit** -- Loaded from a mapping file (JSON). Specifies entity name, GUID template, and selective column mappings with Pict template expressions.
+3. **User** -- CLI flags (`-e`, `-g`, `-n`, `-c`) override individual values.
+
+The layers merge with `Object.assign`:
+
+```javascript
+tmpMappingOutcome.Configuration = Object.assign(
+    {},
+    tmpMappingOutcome.ImplicitConfiguration,
+    tmpMappingOutcome.ExplicitConfiguration,
+    tmpMappingOutcome.UserConfiguration
+);
+```
+
+## Error Handling Patterns
+
+### Retry with Threshold
+
+The `IntegrationAdapter` uses a retry pattern for upsert operations. If a push fails, it retries up to `RecordPushRetryThreshold` times (default: 5, hard cap: 50):
+
+```javascript
+upsertSingleRecord(fCallback, pRecordGUID, pRetryCount)
+{
+    let tmpRetryCount = (typeof(pRetryCount) === 'undefined') ? 0 : pRetryCount;
+
+    if ((tmpRetryCount > this._RecordPushRetryThreshold) || (tmpRetryCount > 50))
+    {
+        this.log.error(`Retry threshold reached for ${this.Entity}.${pRecordGUID}`);
+        return fCallback();
+    }
+
+    // ... attempt upsert, on failure:
+    this.upsertSingleRecord(fCallback, pRecordGUID, tmpRetryCount++);
+}
+```
+
+### Non-Fatal Continuation
+
+Sync operations log errors but continue processing remaining records and entities. This prevents a single bad record from stopping an entire clone operation:
+
+```javascript
+if (pCreateError)
+{
+    this.log.error(`Error creating record ${this.EntitySchema.TableName}: ${pCreateError}`);
+    return fEntitySyncComplete();  // Continue to next record
+}
+```
+
+### Connection Failure
+
+The `ConnectionManager` wraps database driver loading in try/catch blocks to handle cases where the driver package is not installed:
+
+```javascript
+try
+{
+    const libMeadowConnectionMySQL = require('meadow-connection-mysql');
+    // ... connect
+}
+catch (pError)
+{
+    this.log.error(`Failed to load MySQL provider. Ensure meadow-connection-mysql is installed.`);
+    return fCallback(pError);
+}
+```
+
+### Validation at Construction
+
+Sync entity services validate their options in the constructor, throwing errors for missing or invalid configuration rather than failing silently later:
+
+```javascript
+if (!this.options.hasOwnProperty('MeadowEntitySchema'))
+{
+    throw new Error('MeadowSyncEntityInitial requires a valid MeadowEntitySchema option.');
+}
+```
+
+## Extension Points
+
+### Custom Record Marshaling
+
+The `IntegrationAdapter.integrateRecords` method accepts an optional `fMarshalExtraData` callback that runs after each record is marshaled. This allows injecting additional data or performing side effects:
+
+```javascript
+adapter.integrateRecords(fCallback,
+    (pSourceRecord, pMarshaledRecord) =>
+    {
+        // Add computed fields, log specific records, etc.
+        pMarshaledRecord.ComputedField = computeValue(pSourceRecord);
+    });
+```
+
+### Custom REST Client Preparation
+
+The `CloneRestClient` exposes a `prepareRequestOptions` method that can be overridden to add custom headers, authentication tokens, or request modifications:
+
+```javascript
+cloneClient.prepareRequestOptions = (pOptions) =>
+{
+    pOptions.headers = { 'X-Custom-Header': 'value' };
+    return pOptions;
+};
+```
+
+### Custom Sync Entity Lists
+
+The `Sync` service accepts a `SyncEntityList` array in its options or via `ProgramConfiguration`. When provided, only the listed entities are synced, in the specified order. When empty, all entities in the loaded schema are synced:
+
+```json
+{
+    "SyncEntityList": ["User", "Role", "Permission"]
+}
+```
+
+### Per-Entity Sync Options
+
+The `SyncEntityOptions` configuration key allows per-entity overrides:
+
+```json
+{
+    "SyncEntityOptions": {
+        "AuditLog": { "PageSize": 500 }
+    }
+}
+```
+
+### Solver-Based Multi-Entity Extraction
+
+The `TabularTransform` service supports Solvers in mapping configurations. Solvers use the Fable Expression Parser to run arbitrary expressions during transformation, enabling extraction of multiple entity types from a single source row:
+
+```json
+{
+    "Entity": "BookAuthor",
+    "MultipleGUIDUniqueness": true,
+    "GUIDTemplate": "BookAuthor_{~D:Record.book_id~}_{~D:_GUIDUniqueness~}",
+    "Solvers": [
+        "// Custom solver expressions that populate NewRecordsGUIDUniqueness"
+    ]
+}
+```
+
+### MappingOutcome Lifecycle Hooks
+
+The `TabularTransform` service provides overridable hooks for the mapping outcome lifecycle:
+
+- `onBeforeInitializeMappingOutcomeObject(pMappingOutcomeObject)` -- Called before the mapping configuration is resolved
+- `onAfterInitializeMappingOutcomeObject(pMappingOutcomeObject)` -- Called after configuration is merged and the comprehension entity container is created
+
+These can be overridden in a subclass to inject custom initialization logic.
+
+## Testing Approach
+
+Tests are written with Mocha in TDD style and run via Quackage:
+
+```shell
+npm test          # Run tests
+npm run coverage  # Run with nyc coverage
+```
+
+Test files are located in the `test/` directory. The test suite covers:
+
+- **Tabular parsing** -- Verifying CSV, TSV, and JSON array parsing with known input files
+- **Transformation** -- Testing mapping configurations, GUID generation, and comprehension output
+- **Comprehension operations** -- Intersect, array conversion, and CSV export
+- **Integration Adapter** -- Record marshaling, GUID prefixing, and cross-entity ID resolution
+- **REST endpoints** -- Starting the server and exercising each endpoint with HTTP requests
+- **Data Clone services** -- ConnectionManager, CloneRestClient, and Sync service integration
+
+Test data files are stored in the `debug/testdata/` directory and include sample CSV, TSV, and JSON files along with mapping configurations and expected outputs.

package/docs/overview.md

@@ -0,0 +1,156 @@
+# Overview
+
+Meadow Integration is a toolkit for bridging external data sources and the Meadow data access layer. It handles two complementary workflows: **data transformation** (getting data _into_ Meadow) and **data synchronization** (replicating data _from_ a Meadow API to a local database).
+
+## Why Meadow Integration Exists
+
+Applications built on the Retold stack use Meadow as their data access layer, with entities defined by Stricture schemas and exposed through auto-generated REST APIs. Meadow Integration fills the gaps around that core:
+
+- **Ingestion** -- External data (CSV exports, TSV dumps, JSON feeds) needs to be converted into the entity format that Meadow expects before it can be loaded.
+- **Merging** -- Data often comes from multiple sources that describe the same entities. These sources need to be reconciled by shared keys before loading.
+- **Cloning** -- Teams running analytics, reporting, or development work need local copies of production data without direct database access. Data Clone pulls entity records through the Meadow REST API and writes them into a local MySQL or MSSQL database.
+
+## Three Modes of Operation
+
+Meadow Integration exposes its functionality through three interfaces. All three share the same underlying service implementations.
+
+### CLI
+
+The `meadow-integration` command-line program (also available as `mdwint` when installed globally) provides commands for every operation. It is built on `pict-service-commandlineutility` and reads configuration from a `.meadow.config.json` file if one exists in the working directory.
+
+```shell
+npx meadow-integration csvtransform ./books.csv -m mapping.json -o books.json
+npx meadow-integration data-clone --api_server https://api.example.com/1.0/ --schema_path ./schema.json
+```
+
+### REST API
+
+The `serve` command starts an HTTP server (powered by Orator and Restify) that exposes every transformation and merge operation as a POST endpoint. This lets other services call Meadow Integration over the network.
+
+```shell
+npx meadow-integration serve -p 8086
+```
+
+Endpoints are grouped under `/1.0/` and include CSV, TSV, JSON Array, and Comprehension operations.
+
+### Programmatic API
+
+The module exports its services directly for use in your own Node.js code:
+
+```javascript
+const MeadowIntegration = require('meadow-integration');
+
+// TabularCheck, IntegrationServer, ConnectionManager,
+// CloneRestClient, Sync, SyncEntityInitial,
+// SyncEntityOngoing, Operation
+```
+
+Services follow the Fable service provider pattern: instantiate them with a Fable instance and they get access to logging, configuration, and dependency injection automatically.
+
+## Data Transformation Pipeline
+
+The transformation pipeline converts external tabular data into Meadow-compatible entity records through a format called a **Comprehension**.
+
+```
+External Data (CSV / TSV / JSON Array)
+        |
+        v
+  TabularTransform Service
+  (column mapping via Pict templates)
+        |
+        v
+  Comprehension Object
+  (entity records keyed by GUID)
+        |
+        v
+  Integration Adapter
+  (marshal to Meadow schema, resolve cross-entity GUIDs)
+        |
+        v
+  GUID Map
+  (track external <-> Meadow IDs)
+        |
+        v
+  Meadow REST API
+  (single upsert or bulk upsert)
+```
+
+**Key steps:**
+
+1. **Parse** -- The source file is parsed into individual records (rows).
+2. **Map** -- Each record is transformed using a mapping configuration. Pict template expressions (`{~D:Record.column_name~}`) resolve column values. A GUID is generated for each record from a template.
+3. **Collect** -- Transformed records are gathered into a Comprehension object, keyed by their generated GUID. Duplicate GUIDs are merged automatically.
+4. **Marshal** -- The Integration Adapter converts comprehension records into Meadow entity format, prefixing GUIDs and resolving cross-entity ID references through the GUID Map.
+5. **Push** -- Records are upserted to the target Meadow REST API, either individually or in bulk batches (configurable threshold).
+
+## Data Synchronization Pipeline
+
+The Data Clone pipeline replicates entities from a remote Meadow REST API into a local relational database.
+
+```
+Remote Meadow REST API
+        |
+        v
+  CloneRestClient
+  (authenticated HTTP, session management, caching)
+        |
+        v
+  Sync Service
+  (orchestrates per-entity sync in schema order)
+        |
+        v
+  SyncEntity (Initial or Ongoing)
+  (paginated download, record comparison, create/update)
+        |
+        v
+  ConnectionManager
+  (MySQL or MSSQL connection pool)
+        |
+        v
+  Local Database
+  (tables auto-created from Meadow schema, indexes on GUID and Deleted columns)
+```
+
+**Key steps:**
+
+1. **Connect** -- The ConnectionManager establishes a connection pool to the local database. The CloneRestClient authenticates with the remote API (if credentials are configured).
+2. **Schema** -- The Meadow extended schema JSON is loaded. Tables are created locally if they do not exist. Indexes are added on GUID and Deleted columns.
+3. **Compare** -- For each entity, the sync service compares local and remote max IDs and record counts to determine what needs to be synchronized.
+4. **Download** -- Records are fetched from the remote API in configurable page sizes using filtered, sorted endpoint calls.
+5. **Write** -- Each downloaded record is marshaled into the local schema format and either created or updated in the local database. Identity insert is enabled so that primary keys match the remote system.
+
+Two sync modes are available:
+
+- **Initial** -- Downloads all records with IDs greater than the local maximum. Intended for first-time clones or catch-up operations.
+- **Ongoing** -- Compares `UpdateDate` timestamps between local and remote records and updates any that have changed. Handles both new records and modifications.
+
+## Key Concepts
+
+### Comprehension
+
+A JSON object that stores entity records keyed by their GUID. A single comprehension can hold multiple entity types. This is the intermediate data format that connects all transformation operations.
+
+### GUID
+
+A deterministic, template-generated identifier that uniquely identifies a record across systems. GUIDs are composed from a configurable template (e.g., `Book_{~D:Record.isbn~}`), ensuring the same source data always produces the same key.
+
+### Entity
+
+A named type of record corresponding to a Meadow schema table (e.g., `Book`, `Author`, `Airport`). Entities have a standard identity column (`IDEntityName`), a GUID column (`GUIDEntityName`), and timestamping columns (`CreateDate`, `UpdateDate`).
+
+### Mapping
+
+A configuration object that describes how to transform source columns into entity fields. It includes the entity name, a GUID template, and a dictionary of field-to-template mappings. Optional Solvers allow multi-entity extraction from a single source row.
+
+## When to Use Which Tool
+
+| Scenario | Tool |
+|----------|------|
+| One-off CSV/TSV analysis | `csvcheck` CLI command |
+| Transform a file into entity records | `csvtransform` / `tsvtransform` / `jsonarraytransform` CLI commands |
+| Combine multiple data sources | `comprehensionintersect` CLI command |
+| Push transformed data to a Meadow API | `load_comprehension` CLI command or Comprehension Push REST endpoint |
+| Expose transformations as a service | `serve` CLI command (REST API) |
+| Replicate a Meadow API to a local database | `data-clone` CLI command |
+| Run data clone in a container | Docker deployment with `.meadow.config.json` |
+| Use services in your own code | Programmatic API via `require('meadow-integration')` |

package/docs/quickstart.md

@@ -0,0 +1,233 @@
+# Quick Start
+
+This guide walks through the core workflows of Meadow Integration, from analyzing a CSV file to cloning a remote Meadow API into a local database.
+
+## Installation
+
+Install the package from npm:
+
+```shell
+npm install meadow-integration
+```
+
+For CLI usage without a global install:
+
+```shell
+npx meadow-integration --help
+```
+
+If you install globally (or add it to your project), the `mdwint` shorthand is also available:
+
+```shell
+npm install -g meadow-integration
+mdwint --help
+```
+
+## Your First CSV Analysis
+
+Start by inspecting a CSV file to understand its structure. The `csvcheck` command produces statistics about rows, columns, and per-column data quality.
+
+```shell
+npx meadow-integration csvcheck ./data/books.csv -o book-stats.json
+```
+
+This writes a JSON file containing:
+
+- Row and column counts
+- Header names
+- First and last rows as sample data
+- Per-column statistics: total count, empty count, numeric count, first value, and last value
+
+Use this to verify your data before transforming it.
+
+## Your First CSV Transformation
+
+Transform a CSV into a Comprehension -- the intermediate JSON format used across all Meadow Integration operations.
+
+```shell
+npx meadow-integration csvtransform ./data/books.csv \
+  -e "Book" \
+  -n "GUIDBook" \
+  -g "Book_{~D:Record.id~}" \
+  -o books-comprehension.json
+```
+
+This produces a JSON file where each record is keyed by its generated GUID:
+
+```json
+{
+  "Book": {
+    "Book_1": { "GUIDBook": "Book_1", "id": "1", "title": "The Hunger Games", ... },
+    "Book_2": { "GUIDBook": "Book_2", "id": "2", "title": "Harry Potter", ... }
+  }
+}
+```
+
+**Flags explained:**
+
+- `-e Book` -- The entity name in the comprehension
+- `-n GUIDBook` -- The name of the GUID column on each record
+- `-g "Book_{~D:Record.id~}"` -- A Pict template that generates each record's GUID from its `id` column
+- `-o books-comprehension.json` -- Where to write the output
+
+## Using Mapping Files
+
+For production use, mapping files give you precise control over which columns map to which fields and how GUIDs are generated.
+
+Create a file called `mapping_Book.json`:
+
+```json
+{
+  "Entity": "Book",
+  "GUIDTemplate": "Book_{~D:Record.id~}",
+  "Mappings": {
+    "Title": "{~D:Record.title~}",
+    "Language": "{~D:Record.language_code~}",
+    "ISBN": "{~D:Record.isbn~}",
+    "AverageRating": "{~D:Record.average_rating~}"
+  }
+}
+```
+
+Then run the transform with the mapping file:
+
+```shell
+npx meadow-integration csvtransform ./data/books.csv \
+  -m mapping_Book.json \
+  -o books-comprehension.json
+```
+
+The mapping file acts as the "Explicit" configuration layer. Any CLI flags you pass (like `-e` or `-g`) override the mapping file as the "User" layer.
+
+## Merging Comprehensions
+
+When data comes from multiple files or sources, merge them with `comprehensionintersect`. Records with the same GUID are combined, with later values overwriting earlier ones.
+
+First, create a second comprehension from a different source:
+
+```shell
+npx meadow-integration csvtransform ./data/book-ratings.csv \
+  -m mapping_BookRatings.json \
+  -o ratings-comprehension.json
+```
+
+Then merge them:
+
+```shell
+npx meadow-integration comprehensionintersect books-comprehension.json \
+  -i ratings-comprehension.json \
+  -e Book \
+  -o merged-books.json
+```
+
+You can also merge during transformation by passing an existing comprehension with `-i`:
+
+```shell
+npx meadow-integration csvtransform ./data/authors.csv \
+  -m mapping_Author.json \
+  -i books-comprehension.json \
+  -o full-comprehension.json
+```
+
+This adds `Author` records to the same comprehension that already contains `Book` records.
+
+## Pushing to a Meadow API
+
+Once your comprehension is ready, push it to a running Meadow REST API:
+
+```shell
+npx meadow-integration load_comprehension ./full-comprehension.json \
+  -p "IMPORT-2024"
+```
+
+This creates an Integration Adapter for each entity type in the comprehension. For each record, the adapter:
+
+1. Generates a Meadow-compatible GUID using the configured prefix
+2. Resolves cross-entity ID references through the GUID Map
+3. Upserts the record to the Meadow API (single or bulk, based on record count)
+
+The `-p` flag sets the adapter set GUID marshal prefix, which namespaces the import.
+
+## Setting Up Data Clone
+
+Data Clone replicates a remote Meadow API into a local database. Configuration is stored in a `.meadow.config.json` file in your working directory.
+
+Create `.meadow.config.json`:
+
+```json
+{
+  "Source": {
+    "ServerURL": "https://api.example.com/1.0/",
+    "UserID": "sync-user",
+    "Password": "sync-password"
+  },
+  "Destination": {
+    "Provider": "MySQL",
+    "MySQL": {
+      "server": "127.0.0.1",
+      "port": 3306,
+      "user": "root",
+      "password": "localpass",
+      "database": "meadow_clone",
+      "connectionLimit": 20
+    }
+  },
+  "SchemaPath": "./schema/Model-Extended.json",
+  "Sync": {
+    "DefaultSyncMode": "Initial",
+    "PageSize": 100,
+    "SyncEntityList": [],
+    "SyncEntityOptions": {}
+  }
+}
+```
+
+**Configuration sections:**
+
+- **Source** -- The remote Meadow REST API to read from. `UserID` and `Password` are optional; if omitted, authentication is skipped.
+- **Destination** -- The local database to write to. Supports `MySQL` and `MSSQL` providers.
+- **SchemaPath** -- Path to the Meadow extended schema JSON (generated by Stricture). This defines which tables and columns to create locally.
+- **Sync** -- Controls the sync behavior. Leave `SyncEntityList` empty to sync all entities in the schema, or list specific entity names to sync a subset.
+
+## Running a Data Clone
+
+With the configuration in place, run the clone:
+
+```shell
+npx meadow-integration data-clone
+```
+
+This will:
+
+1. Authenticate with the source API (if credentials are configured)
+2. Connect to the local database
+3. Load the Meadow schema and create any missing tables
+4. Add indexes on GUID and Deleted columns
+5. Download and insert all records for each entity
+
+For subsequent runs, switch to Ongoing mode to only sync changes:
+
+```shell
+npx meadow-integration data-clone -s Ongoing
+```
+
+You can also override configuration from the command line:
+
+```shell
+npx meadow-integration data-clone \
+  --api_server https://api.example.com/1.0/ \
+  --db_host 127.0.0.1 \
+  --db_name meadow_clone \
+  --schema_path ./schema/Model-Extended.json \
+  --sync_mode Initial
+```
+
+## Next Steps
+
+- [Overview](overview.md) -- Full feature overview and when to use each tool
+- [Architecture](architecture.md) -- System design with diagrams
+- [Mapping Files](mapping-files.md) -- Detailed mapping configuration reference
+- [Comprehensions](comprehensions.md) -- The comprehension data format in depth
+- [CLI Reference](cli-reference.md) -- All commands and their options
+- [REST API Reference](rest-api-reference.md) -- All REST endpoints
+- [Examples](examples-walkthrough.md) -- Walkthrough of all runnable examples