meadow-integration 1.0.5 → 1.0.6

package/docs/data-clone/docker.md

@@ -0,0 +1,290 @@
+# Docker Deployment
+
+The meadow-integration module includes Docker support for running data clone operations in containerized environments.
+
+## Dockerfiles
+
+### Production Dockerfile
+
+The primary `Dockerfile` builds a production-ready image based on `node:20-bookworm`.
+
+#### Build Stages
+
+The Dockerfile uses a multi-stage build:
+
+**Base stage**:
+1. Starts from `node:20-bookworm`.
+2. Installs system utilities: `curl`, `vim`, `nano`, `less`, `tmux`, `uuid-runtime`.
+3. Installs `nodemon` globally.
+4. Copies `package.json` and runs `npm install --omit=dev` (production dependencies only).
+5. Copies `source/` and `scripts/` directories.
+6. Cleans up development artifacts (`package-lock.json`, `.git`, `test`).
+7. If a `Meadow-Config-Docker.json` file exists, it is moved to `source/cli/Default-Meadow-Integration-Configuration.json` to serve as the default configuration inside the container.
+
+**Production stage**:
+1. Extends the base stage.
+2. Records the build timestamp in `build.date`.
+3. Runs `scripts/run.sh` as the default command.
+
+#### Building the Image
+
+```bash
+docker build -t retold/meadow-integration:latest .
+```
+
+Or use the provided helper script:
+
+```bash
+./Docker-Build.sh
+```
+
+#### Custom Configuration at Build Time
+
+To bake a default configuration into the image, create a `Meadow-Config-Docker.json` file in the project root before building:
+
+```json
+{
+    "Source": {
+        "ServerURL": "https://api.production.example.com/1.0/",
+        "UserID": "sync_service",
+        "Password": "service_password"
+    },
+    "Destination": {
+        "Provider": "MySQL",
+        "MySQL": {
+            "server": "mysql-host",
+            "port": 3306,
+            "user": "clone_user",
+            "password": "clone_password",
+            "database": "production_clone",
+            "connectionLimit": 20
+        }
+    },
+    "SchemaPath": "/service_root/schema/Model-Extended.json",
+    "Sync": {
+        "DefaultSyncMode": "Initial",
+        "PageSize": 100,
+        "SyncEntityList": []
+    }
+}
+```
+
+This file is automatically picked up during the build and used as the default configuration.
+
+### Development Dockerfile (Dockerfile_LUXURYCode)
+
+The `Dockerfile_LUXURYCode` builds a development image based on `codercom/code-server:latest`. It provides a browser-based VS Code environment for developing and debugging the meadow-integration module.
+
+#### What It Includes
+
+- **Base**: code-server (VS Code in the browser)
+- **Runtime**: Node.js 20 via NVM
+- **System tools**: vim, curl, tmux
+- **VS Code extensions**:
+  - Mocha Test Adapter
+  - Test Explorer
+  - Indent Rainbow
+  - ESLint
+  - GitLens
+
+#### Volumes
+
+| Volume | Purpose |
+|--------|---------|
+| `/home/coder/.config` | code-server configuration |
+| `/home/coder/.vscode` | VS Code settings |
+| `/home/coder/meadow-integration` | Project source (mount your local checkout here) |
+
+#### Building the Development Image
+
+```bash
+docker build -f Dockerfile_LUXURYCode -t retold/meadow-integration-dev:latest .
+```
+
+#### Running the Development Image
+
+```bash
+docker run -d \
+  -p 8443:8080 \
+  -v "$(pwd):/home/coder/meadow-integration" \
+  retold/meadow-integration-dev:latest
+```
+
+Access the development environment at `http://localhost:8443`.
+
+## Docker Compose
+
+The `docker-compose.yml` file defines a service for running the data clone:
+
+```yaml
+version: '2'
+
+services:
+  meadow-integration-clone:
+    image: retold/meadow-integration:latest
+    volumes:
+      - "${RETOLD_DIR:-./}:/service_root"
+    environment:
+      - RUN_LOCAL_DEV=true
+    networks:
+      - back
+
+networks:
+  back:
+    external:
+      name: meadow_backend
+```
+
+### Running with Docker Compose
+
+```bash
+docker-compose up -d meadow-integration-clone
+```
+
+Or use the provided helper script:
+
+```bash
+./Docker-Compose.sh
+```
+
+### Environment Variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `RETOLD_DIR` | `./` | Host directory to mount as `/service_root` |
+| `RUN_LOCAL_DEV` | - | Set to `true` for local development mode |
+| `MEADOW_INTEGRATION_PORT` | `8086` | Port override for the REST server (when running the `serve` command) |
+
+### Network
+
+The compose file expects an external Docker network named `meadow_backend`. Create it before running:
+
+```bash
+docker network create meadow_backend
+```
+
+## Volume Mounts for Configuration
+
+### Mounting a Config File
+
+To provide a `.meadow.config.json` at runtime without baking it into the image:
+
+```bash
+docker run \
+  -v "/path/to/.meadow.config.json:/service_root/.meadow.config.json" \
+  -v "/path/to/schema:/service_root/schema" \
+  retold/meadow-integration:latest
+```
+
+### Mounting a Schema Directory
+
+The schema file referenced by `SchemaPath` must be accessible inside the container. Mount it as a volume:
+
+```bash
+docker run \
+  -v "/host/path/schema:/service_root/schema" \
+  retold/meadow-integration:latest
+```
+
+## Production Deployment Tips
+
+### Running an Initial Clone
+
+```bash
+docker run --rm \
+  --network meadow_backend \
+  -v "/config/.meadow.config.json:/service_root/.meadow.config.json" \
+  -v "/data/schema:/service_root/schema" \
+  retold/meadow-integration:latest \
+  node source/cli/Meadow-Integration-CLI-Run.js data-clone --sync_mode Initial
+```
+
+### Running an Ongoing Sync on a Schedule
+
+Use cron or a container orchestrator to run ongoing syncs periodically:
+
+```bash
+docker run --rm \
+  --network meadow_backend \
+  -v "/config/.meadow.config.json:/service_root/.meadow.config.json" \
+  -v "/data/schema:/service_root/schema" \
+  retold/meadow-integration:latest \
+  node source/cli/Meadow-Integration-CLI-Run.js data-clone --sync_mode Ongoing
+```
+
+### Using the Post-Run Delay
+
+The `--post_run_delay` flag keeps the container alive for a specified number of minutes after the sync completes. This is useful in orchestrated environments where you want to inspect logs before the container exits:
+
+```bash
+docker run --rm \
+  --network meadow_backend \
+  -v "/config/.meadow.config.json:/service_root/.meadow.config.json" \
+  -v "/data/schema:/service_root/schema" \
+  retold/meadow-integration:latest \
+  node source/cli/Meadow-Integration-CLI-Run.js data-clone --post_run_delay 5
+```
+
+### Running the REST Server in Docker
+
+To run the integration REST API server inside Docker:
+
+```bash
+docker run -d \
+  -p 8086:8086 \
+  --name meadow-integration-api \
+  retold/meadow-integration:latest \
+  node source/cli/Meadow-Integration-CLI-Run.js serve --port 8086
+```
+
+### Tagging and Pushing
+
+Helper scripts are provided for image management:
+
+```bash
+# Tag the image
+./Docker-Tag.sh
+
+# Push to a registry
+./Docker-Push.sh
+```
+
+### Health Checks
+
+When running the REST server, use the Status endpoint as a health check:
+
+```yaml
+services:
+  meadow-integration-api:
+    image: retold/meadow-integration:latest
+    command: node source/cli/Meadow-Integration-CLI-Run.js serve
+    ports:
+      - "8086:8086"
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:8086/1.0/Status"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+```
+
+### Database Connectivity
+
+When running inside Docker, make sure the container can reach the database server. Common patterns:
+
+- **Same Docker network**: Use the container name as the hostname (e.g., `mysql-server`).
+- **Host machine**: Use `host.docker.internal` (Docker Desktop) or `172.17.0.1` (Linux).
+- **External database**: Use the external hostname or IP.
+
+Update the `Destination` configuration accordingly:
+
+```json
+{
+    "Destination": {
+        "Provider": "MySQL",
+        "MySQL": {
+            "server": "mysql-server",
+            "database": "production_clone"
+        }
+    }
+}
+```

package/docs/data-clone/overview.md

@@ -0,0 +1,173 @@
+# Data Clone System Overview
+
+The data clone system synchronizes data from a remote Meadow API server to a local relational database. It is designed for offline access, reporting, data warehousing, and building local replicas of Meadow-managed datasets.
+
+## What It Does
+
+The data clone system connects to a source Meadow REST API, reads entity records using paginated queries, and writes them into a local MySQL or MSSQL database. Tables are auto-created from the Meadow schema, and indexes are built on GUID and Deleted columns for query performance.
+
+## Why Use It
+
+- **Offline access**: Work with your data without a persistent connection to the source API.
+- **Reporting**: Run complex SQL queries against a local database without impacting the production API.
+- **Data warehousing**: Aggregate data from Meadow APIs into a central database for analytics.
+- **Development**: Maintain a local copy of production data for development and testing.
+
+## Key Components
+
+### ConnectionManager
+
+Manages database connections for MySQL and MSSQL providers. Handles connection pooling, table creation, and index management.
+
+See [connection-manager.md](connection-manager.md) for details.
+
+### RestClient
+
+A REST client service (`MeadowCloneRestClient`) that communicates with the source Meadow API. Handles authentication, session management, caching, and paginated entity retrieval via keep-alive HTTP connections.
+
+Key capabilities:
+- Session-based authentication (`authenticate` / `deauthenticate`)
+- CRUD operations (`getEntity`, `createEntity`, `updateEntity`, `upsertEntity`, `deleteEntity`)
+- Bulk retrieval with automatic pagination (`getEntitySet`)
+- Built-in LRU object cache per entity type (30-second TTL, 10,000 entries)
+
+### Sync
+
+The `MeadowSync` service orchestrates the overall synchronization. It loads a Meadow schema, creates `SyncEntity` instances for each entity, and runs them in sequence.
+
+### SyncEntity (Initial and Ongoing)
+
+Two sync strategies are available:
+
+- **MeadowSyncEntityInitial**: Full clone using ID-based pagination. Only creates new records.
+- **MeadowSyncEntityOngoing**: Incremental sync using UpdateDate comparison. Creates new records and updates changed ones.
+
+See [sync-modes.md](sync-modes.md) for a detailed comparison.
+
+## Supported Providers
+
+| Provider | Module | Default Port |
+|----------|--------|-------------|
+| MySQL | `meadow-connection-mysql` | 3306 |
+| MSSQL | `meadow-connection-mssql` | 1433 |
+
+## Sync Modes
+
+| Mode | Strategy | Creates | Updates | Use Case |
+|------|----------|---------|---------|----------|
+| Initial | ID-based pagination from max local ID | Yes | No | First-time full clone |
+| Ongoing | UpdateDate comparison across all records | Yes | Yes | Incremental sync after initial clone |
+
+## Running a Data Clone
+
+### Via CLI
+
+```bash
+mdwint data-clone \
+  --api_server "https://api.example.com/1.0/" \
+  --api_username "admin" \
+  --api_password "secret" \
+  --db_host "127.0.0.1" \
+  --db_name "local_clone" \
+  --schema_path "./schema/Model-Extended.json" \
+  --sync_mode "Initial"
+```
+
+### Via Configuration File
+
+Create a `.meadow.config.json` file (see [configuration.md](configuration.md)) and run:
+
+```bash
+mdwint data-clone
+```
+
+### Programmatically
+
+```javascript
+const meadowIntegration = require('meadow-integration');
+const libFable = require('fable');
+
+const fable = new libFable({});
+
+// Register services
+fable.serviceManager.addServiceType('MeadowCloneRestClient', meadowIntegration.CloneRestClient);
+fable.serviceManager.instantiateServiceProvider('MeadowCloneRestClient', {
+    ServerURL: 'https://api.example.com/1.0/',
+    UserID: 'admin',
+    Password: 'secret'
+});
+
+fable.serviceManager.addServiceType('MeadowConnectionManager', meadowIntegration.ConnectionManager);
+fable.serviceManager.instantiateServiceProvider('MeadowConnectionManager', {
+    Provider: 'MySQL',
+    MySQL: {
+        server: '127.0.0.1',
+        port: 3306,
+        user: 'root',
+        password: '',
+        database: 'local_clone',
+        connectionLimit: 20
+    }
+});
+
+fable.serviceManager.addServiceType('MeadowSync', meadowIntegration.Sync);
+
+// Authenticate, connect, load schema, sync
+fable.MeadowCloneRestClient.authenticate((pAuthError) => {
+    fable.MeadowConnectionManager.connect((pConnectError, pPool) => {
+        fable.serviceManager.instantiateServiceProvider('MeadowSync', {
+            ConnectionPool: pPool,
+            PageSize: 100
+        });
+        fable.MeadowSync.SyncMode = 'Initial';
+        const schema = require('./schema/Model-Extended.json');
+        fable.MeadowSync.loadMeadowSchema(schema, (pLoadError) => {
+            fable.MeadowSync.syncAll((pSyncError) => {
+                console.log('Sync complete!');
+            });
+        });
+    });
+});
+```
+
+## Architecture Diagram
+
+```
++-------------------+       HTTPS/HTTP       +-------------------+
+|  Source Meadow    |  <-------------------> |  RestClient       |
+|  API Server       |    GET /Entity/...     |  (MeadowClone     |
++-------------------+    Session Auth        |   RestClient)     |
+                                             +--------+----------+
+                                                      |
+                                                      v
+                                             +-------------------+
+                                             |  MeadowSync       |
+                                             |  (Orchestrator)   |
+                                             +--------+----------+
+                                                      |
+                                         +------------+------------+
+                                         |                         |
+                                  +------+------+          +-------+-------+
+                                  | SyncEntity  |          | SyncEntity    |
+                                  | Initial     |          | Ongoing       |
+                                  +------+------+          +-------+-------+
+                                         |                         |
+                                         v                         v
+                                  +------+-------------------------+------+
+                                  |       ConnectionManager               |
+                                  |       (MySQL / MSSQL)                 |
+                                  +---------------------------------------+
+                                                      |
+                                                      v
+                                             +-------------------+
+                                             |  Local Database   |
+                                             |  (MySQL / MSSQL)  |
+                                             +-------------------+
+```
+
+## Related Documentation
+
+- [Connection Manager](connection-manager.md) -- Database connection setup and index creation
+- [Sync Modes](sync-modes.md) -- Detailed comparison of Initial vs Ongoing sync
+- [Configuration](configuration.md) -- Full `.meadow.config.json` reference
+- [Docker Deployment](docker.md) -- Running data-clone in Docker

package/docs/data-clone/sync-modes.md

@@ -0,0 +1,186 @@
+# Sync Modes
+
+The data clone system supports two sync modes: **Initial** and **Ongoing**. Each mode uses a different strategy for determining which records to fetch and how to write them to the local database.
+
+## Initial Sync
+
+**Purpose**: Perform a full clone of data from the source API to the local database.
+
+### Strategy
+
+1. **Read local max ID**: Query the local database for the maximum value of the entity's `DefaultIdentifier` (primary key).
+2. **Read local count**: Count the total records in the local database.
+3. **Read server max ID**: Query the source API for the maximum ID via `GET /{Entity}/Max/{DefaultIdentifier}`.
+4. **Read server count**: Query the source API for the total count via `GET /{Entity}s/Count`.
+5. **Estimate work**: Calculate the estimated number of new records as `Server.RecordCount - Local.RecordCount`.
+6. **Generate paginated URLs**: Build a list of URL partials using the filter `FBV~{ID}~GT~{LocalMaxID}~FSF~{ID}~ASC~ASC` with pagination offsets.
+7. **Fetch and insert**: For each page of records:
+   - Check if the record exists locally by its ID.
+   - If the record does not exist, marshal and create it.
+   - If the record already exists, skip it.
+
+### Key Characteristics
+
+| Aspect | Behavior |
+|--------|----------|
+| Record creation | Yes |
+| Record updates | No |
+| Pagination | All pages generated upfront based on server count |
+| Filter | Records with ID greater than local max |
+| Sort | Ascending by DefaultIdentifier |
+| Concurrency | 1 page at a time, 5 records in parallel per page |
+| Progress tracking | Tracks estimated vs completed record count |
+
+### When to Use
+
+- First-time data cloning from a source API.
+- Rebuilding a local database from scratch.
+- When you only need new records and do not need to detect changes to existing records.
+
+### Create Behavior
+
+When creating records during Initial sync, the following flags are set on the Meadow query:
+
+- `setDisableAutoIdentity(true)` -- Preserves the original ID from the source.
+- `setDisableAutoDateStamp(true)` -- Preserves original CreateDate/UpdateDate values.
+- `setDisableAutoUserStamp(true)` -- Preserves original CreatingIDUser/UpdatingIDUser values.
+- `setDisableDeleteTracking(true)` -- Preserves the original Deleted flag.
+- `AllowIdentityInsert = true` -- Allows inserting records with explicit ID values.
+
+---
+
+## Ongoing Sync
+
+**Purpose**: Incrementally sync new and changed records from the source API.
+
+### Strategy
+
+1. **Check for UpdateDate column**: Verify that the entity schema contains an `UpdateDate` column.
+2. **Read local max ID**: Same as Initial.
+3. **Read local max UpdateDate**: Query for the most recent `UpdateDate` in the local database.
+4. **Read local count**: Same as Initial.
+5. **Read server max ID**: Same as Initial.
+6. **Read server max UpdateDate**: Query the source API via `GET /{Entity}/Max/UpdateDate`.
+7. **Read server count**: Same as Initial.
+8. **Estimate request count**: Calculate `Math.ceil(Server.RecordCount / PageSize)`.
+9. **Recursive fetch**: Using a recursive `anticipate` pattern:
+   - Fetch a page of records sorted by ID ascending, starting from `LastRequestedID`.
+   - For each record:
+     - If it does not exist locally, create it.
+     - If it exists, compare `UpdateDate` values. If the difference exceeds 5 milliseconds, update the local record.
+     - If the dates are within 5ms, skip the record.
+   - After processing the page, recursively add the next page fetch to the anticipate chain.
+   - Continue until all estimated pages have been processed.
+
+### Key Characteristics
+
+| Aspect | Behavior |
+|--------|----------|
+| Record creation | Yes |
+| Record updates | Yes |
+| Pagination | Recursive, one page at a time |
+| Filter | Records with ID greater than `LastRequestedID` (starts at 0) |
+| Sort | Ascending by DefaultIdentifier |
+| UpdateDate threshold | 5 milliseconds (changes smaller than 5ms are ignored) |
+| Concurrency | Sequential pages, sequential records within each page |
+| Progress tracking | Tracks request count against estimated total |
+
+### Recursive Anticipate Pattern
+
+The Ongoing sync uses a recursive pattern where each page fetch, upon completion, adds another page fetch to the `anticipate` queue. This ensures that the `LastRequestedID` advances as records are processed:
+
+```
+Page 1 (IDs 1-100) -> process -> update LastRequestedID to 100
+  -> add Page 2 (IDs > 100) to anticipate queue
+    Page 2 (IDs 101-200) -> process -> update LastRequestedID to 200
+      -> add Page 3 (IDs > 200) to anticipate queue
+        ...
+```
+
+This approach handles cases where the server record count changes during the sync process.
+
+### When to Use
+
+- After an Initial sync has been completed.
+- When running periodic sync jobs to keep the local database up to date.
+- When you need to detect and apply changes to existing records.
+
+### Create and Update Behavior
+
+**Creating** new records uses the same flags as Initial sync (disabled auto-identity, auto-date, auto-user, and delete tracking with `AllowIdentityInsert`).
+
+**Updating** existing records also uses the same disabled flags to preserve the original metadata from the source.
+
+---
+
+## Comparison Table
+
+| Feature | Initial | Ongoing |
+|---------|---------|---------|
+| Creates new records | Yes | Yes |
+| Updates existing records | No | Yes |
+| Starting point | Local max ID | ID 0 (scans all records) |
+| Pagination strategy | Pre-computed URL list | Recursive anticipate |
+| UpdateDate comparison | No | Yes (5ms threshold) |
+| Schema requirement | DefaultIdentifier | DefaultIdentifier, UpdateDate |
+| Typical use | First-time clone | Incremental updates |
+| Performance | Fast (skips existing) | Thorough (checks every record) |
+
+## Setting the Sync Mode
+
+### Via CLI
+
+```bash
+mdwint data-clone --sync_mode Initial
+mdwint data-clone --sync_mode Ongoing
+```
+
+### Via Configuration
+
+In `.meadow.config.json`:
+
+```json
+{
+    "Sync": {
+        "DefaultSyncMode": "Initial"
+    }
+}
+```
+
+### Programmatically
+
+```javascript
+fable.MeadowSync.SyncMode = 'Ongoing';
+```
+
+The sync mode must be set **before** calling `loadMeadowSchema()`, because the mode determines which `SyncEntity` class (Initial or Ongoing) is instantiated for each entity.
+
+## Record Marshaling
+
+Both sync modes use the same `marshalRecord` method to prepare source records for local database insertion:
+
+- Columns defined in the entity schema are mapped from the source record.
+- `null` and `undefined` values are skipped.
+- Object values are JSON-stringified.
+- `DateTime` values are reformatted to `YYYY-MM-DD HH:mm:ss.SSS` in UTC.
+- Empty string values are skipped for non-DateTime fields.
+- Columns ending in `JSON` are auto-populated from matching object properties (e.g., `ConfigJSON` is populated from `Config` if `Config` is an object).
+
+## Table and Index Creation
+
+Both sync modes automatically:
+
+1. Create the database table if it does not exist (via Meadow provider's `createTable`).
+2. Create a unique index on the GUID column (if present).
+3. Create a non-unique index on the Deleted column (if present).
+
+This happens during the `initialize()` phase before any records are synced.
+
+## Progress Tracking
+
+Both modes use the `MeadowOperation` utility class for progress tracking:
+
+- **Initial**: Tracks `FullSync-{TableName}` with estimated record count.
+- **Ongoing**: Tracks `UpdateSync-{TableName}` with estimated request count.
+
+Progress includes: percent complete, average operation time, and estimated completion time.