meadow-integration 1.0.4 → 1.0.6

package/docs/cli/tsvtransform.md

@@ -0,0 +1,144 @@
+# tsvtransform
+
+Transform a TSV (tab-separated values) file into a Comprehension JSON file. This command works identically to [csvtransform](csvtransform.md) but uses a tab character as the column delimiter instead of a comma.
+
+**Aliases:** `tsv_t`, `tsv_transform`
+
+## Usage
+
+```shell
+mdwint tsvtransform <file> [options]
+```
+
+## Arguments
+
+| Argument | Required | Description |
+|----------|----------|-------------|
+| `<file>` | Yes | Path to the TSV file to transform |
+
+## Options
+
+| Option | Description | Default |
+|--------|-------------|---------|
+| `-e, --entity <name>` | Entity name in the Comprehension | Auto-detected from filename |
+| `-n, --guidname <name>` | Name of the GUID column in the generated Comprehension | `GUID<Entity>` |
+| `-g, --guidtemplate <template>` | Pict template for generating the entity GUID | `GUID-<Entity>-{~Data:Record.<first_column>~}` |
+| `-c, --columns <mappings>` | Inline column mappings in `Key={~D:col~}` format | Auto-detected 1:1 mapping |
+| `-m, --mappingfile <path>` | Path to a JSON mapping file | -- |
+| `-i, --incoming <path>` | Existing Comprehension file to merge into | -- |
+| `-o, --output <path>` | Output file path | `./TSV-Comprehension-<filename>.json` |
+| `-x, --extended` | Output full operation state, not just the Comprehension | `false` |
+| `-q, --quotedelimiter <char>` | Quote delimiter character for TSV parsing | `"` |
+
+## Configuration Priority
+
+The same three-layer configuration priority applies as with `csvtransform`:
+
+1. **Implicit** -- Auto-detected from the first TSV row
+2. **Explicit** -- Loaded from a mapping file via `-m`
+3. **User** -- Command-line options (`-e`, `-n`, `-g`, `-c`)
+
+Each layer overrides the previous.
+
+## Output Format
+
+The output is a standard Comprehension JSON object, identical in format to `csvtransform` output:
+
+```json
+{
+  "Airport": {
+    "Airport-SEA": {
+      "GUIDAirport": "Airport-SEA",
+      "Code": "SEA",
+      "Name": "Seattle Tacoma International",
+      "City": "Seattle",
+      "State": "WA"
+    }
+  }
+}
+```
+
+## Examples
+
+### Basic TSV transform with a mapping file
+
+Given a TSV file `airports.tsv` with columns `iata`, `name`, `city`, `state`, `country`, `lat`, `long`:
+
+```shell
+mdwint tsvtransform ./data/airports.tsv \
+  -m mapping_Airport.json \
+  -o airports.json
+```
+
+Where `mapping_Airport.json` contains:
+
+```json
+{
+  "Entity": "Airport",
+  "GUIDTemplate": "Airport-{~D:Record.iata~}",
+  "Mappings": {
+    "Code": "{~D:Record.iata~}",
+    "Name": "{~D:Record.name~}",
+    "Description": "{~D:Record.name~} airport in {~D:Record.city~}",
+    "City": "{~D:Record.city~}",
+    "State": "{~D:Record.state~}",
+    "Country": "{~D:Record.country~}",
+    "Latitude": "{~D:Record.lat~}",
+    "Longitude": "{~D:Record.long~}"
+  }
+}
+```
+
+### Transform with CLI options
+
+```shell
+mdwint tsvtransform ./data/airports.tsv \
+  -e Airport \
+  -n GUIDAirport \
+  -g "Airport-{~D:Record.iata~}" \
+  -c "Code={~D:Record.iata~},Name={~D:Record.name~},City={~D:Record.city~},State={~D:Record.state~},Country={~D:Record.country~},Latitude={~D:Record.lat~},Longitude={~D:Record.long~}" \
+  -o airports.json
+```
+
+### Implicit transform (auto-detect everything)
+
+```shell
+mdwint tsvtransform ./data/airports.tsv -o airports.json
+```
+
+### Merge into an existing Comprehension
+
+```shell
+mdwint tsvtransform ./data/airports.tsv \
+  -m mapping_Airport.json \
+  -i existing-comprehension.json \
+  -o existing-comprehension.json
+```
+
+### Extended output for debugging
+
+```shell
+mdwint tsvtransform ./data/airports.tsv \
+  -m mapping_Airport.json \
+  -x \
+  -o debug-airports.json
+```
+
+## Tips
+
+- TSV files are common exports from spreadsheet applications and database tools. The tab delimiter avoids issues with commas embedded in field values.
+- The quote delimiter defaults to `"` but can be changed with `-q`. TSV files often do not use quote delimiters at all; if your file has no quoting, you can leave the default.
+- All other behavior, including duplicate GUID merging, multi-entity support, and Solver expressions, works identically to `csvtransform`.
+- Inline column mappings (`-c`) do not support commas or equal signs within template values. Use a mapping file for complex mappings.
+
+## Notes
+
+- The default output filename pattern is `./TSV-Comprehension-<filename>.json` (note the `TSV-` prefix rather than `CSV-`).
+- The delimiter is set to `\t` internally. You do not need to specify it.
+
+## See Also
+
+- [csvtransform](csvtransform.md) -- Comma-delimited equivalent
+- [jsonarraytransform](jsonarraytransform.md) -- JSON array equivalent
+- [Mapping Files](../mapping-files.md) -- Full mapping file specification
+- [Comprehensions](../comprehensions.md) -- Core data structure documentation

package/docs/data-clone/configuration.md

@@ -0,0 +1,357 @@
+# Configuration Reference
+
+The data clone system is configured through a `.meadow.config.json` file and/or CLI flags. This document covers the full configuration schema, cascading resolution behavior, and complete working examples.
+
+## Configuration File
+
+The configuration file is named `.meadow.config.json`. The CLI toolkit (`pict-service-commandlineutility`) automatically searches for this file using a cascading strategy.
+
+### Cascading Resolution Order
+
+The configuration file is resolved in the following order (first match wins):
+
+1. **Current working directory** -- `.meadow.config.json` in the directory where the command is run.
+2. **Parent directories** -- Walks up the directory tree searching for `.meadow.config.json`.
+3. **Home directory** -- `~/.meadow.config.json` as a fallback.
+
+Settings from the configuration file are merged with the built-in defaults. CLI flags override configuration file values.
+
+## Full Configuration Schema
+
+```json
+{
+    "Source": {
+        "ServerURL": "https://api.example.com/1.0/",
+        "UserID": false,
+        "Password": false
+    },
+    "Destination": {
+        "Provider": "MySQL",
+        "MySQL": {
+            "server": "127.0.0.1",
+            "port": 3306,
+            "user": "root",
+            "password": "",
+            "database": "meadow",
+            "connectionLimit": 20
+        },
+        "MSSQL": {
+            "server": "127.0.0.1",
+            "port": 1433,
+            "user": "sa",
+            "password": "",
+            "database": "meadow",
+            "ConnectionPoolLimit": 20
+        }
+    },
+    "SchemaPath": "./schema/Model-Extended.json",
+    "Sync": {
+        "DefaultSyncMode": "Initial",
+        "PageSize": 100,
+        "SyncEntityList": [],
+        "SyncEntityOptions": {}
+    }
+}
+```
+
+## Section Reference
+
+### Source
+
+Configuration for the remote Meadow API server to clone data from.
+
+| Property | Type | Default | Description |
+|----------|------|---------|-------------|
+| `ServerURL` | string | `"https://localhost:8080/1.0/"` | Base URL of the Meadow API (must include trailing `/1.0/`) |
+| `UserID` | string or false | `false` | Username for API authentication; set to `false` to skip auth |
+| `Password` | string or false | `false` | Password for API authentication; set to `false` to skip auth |
+
+When `UserID` and `Password` are both `false`, the system skips authentication and makes unauthenticated requests.
+
+### Destination
+
+Configuration for the local database where cloned data is stored.
+
+| Property | Type | Default | Description |
+|----------|------|---------|-------------|
+| `Provider` | string | `"MySQL"` | Database provider: `"MySQL"` or `"MSSQL"` |
+| `MySQL` | object | See below | MySQL-specific connection settings |
+| `MSSQL` | object | See below | MSSQL-specific connection settings |
+
+Only the section matching the selected `Provider` is used.
+
+#### MySQL Settings
+
+| Property | Type | Default | Description |
+|----------|------|---------|-------------|
+| `server` | string | `"127.0.0.1"` | MySQL server hostname |
+| `port` | number | `3306` | MySQL server port |
+| `user` | string | `"root"` | Database username |
+| `password` | string | `""` | Database password |
+| `database` | string | `"meadow"` | Database name |
+| `connectionLimit` | number | `20` | Connection pool size |
+
+#### MSSQL Settings
+
+| Property | Type | Default | Description |
+|----------|------|---------|-------------|
+| `server` | string | `"127.0.0.1"` | MSSQL server hostname |
+| `port` | number | `1433` | MSSQL server port |
+| `user` | string | `"sa"` | Database username |
+| `password` | string | `""` | Database password |
+| `database` | string | `"meadow"` | Database name |
+| `ConnectionPoolLimit` | number | `20` | Connection pool size |
+
+### SchemaPath
+
+| Property | Type | Default | Description |
+|----------|------|---------|-------------|
+| `SchemaPath` | string | `"./schema/Model-Extended.json"` | Path to the Meadow extended schema JSON file (resolved relative to the config file or CWD) |
+
+The schema file is a compiled Stricture model containing table definitions, column metadata, and Meadow schema information. It is generated by `stricture` during the build process.
+
+### Sync
+
+| Property | Type | Default | Description |
+|----------|------|---------|-------------|
+| `DefaultSyncMode` | string | `"Initial"` | Default sync mode: `"Initial"` or `"Ongoing"` |
+| `PageSize` | number | `100` | Number of records to fetch per API request |
+| `SyncEntityList` | array | `[]` | List of entity (table) names to sync. Empty array means sync all entities in the schema |
+| `SyncEntityOptions` | object | `{}` | Per-entity configuration overrides (keyed by entity name) |
+
+#### SyncEntityList
+
+When `SyncEntityList` is an empty array `[]`, the system syncs every entity found in the loaded schema. To sync only specific entities, list them by their table name:
+
+```json
+{
+    "Sync": {
+        "SyncEntityList": ["Book", "Author", "BookAuthorJoin"]
+    }
+}
+```
+
+Entities are synced in the order they appear in this list.
+
+#### SyncEntityOptions
+
+Per-entity overrides, keyed by entity name:
+
+```json
+{
+    "Sync": {
+        "SyncEntityOptions": {
+            "AuditLog": {
+                "PageSize": 500
+            }
+        }
+    }
+}
+```
+
+## CLI Flag Overrides
+
+All configuration values can be overridden via CLI flags. CLI flags take precedence over values from `.meadow.config.json`.
+
+### Source (API) Flags
+
+| Flag | Long Form | Description |
+|------|-----------|-------------|
+| `-a` | `--api_server` | Source API server URL |
+| `-u` | `--api_username` | API username |
+| `-p` | `--api_password` | API password |
+
+### Destination (Database) Flags
+
+| Flag | Long Form | Description |
+|------|-----------|-------------|
+| `-d` | `--db_provider` | Database provider (`MySQL` or `MSSQL`) |
+| `-dh` | `--db_host` | Database server hostname |
+| `-dp` | `--db_port` | Database server port |
+| `-du` | `--db_username` | Database username |
+| `-dw` | `--db_password` | Database password |
+| `-dn` | `--db_name` | Database name |
+
+### Other Flags
+
+| Flag | Long Form | Default | Description |
+|------|-----------|---------|-------------|
+| `-sp` | `--schema_path` | From config | Path to Meadow extended schema JSON |
+| `-s` | `--sync_mode` | `"Initial"` | Sync mode: `Initial` or `Ongoing` |
+| `-w` | `--post_run_delay` | `0` | Minutes to wait after sync before exiting |
+
+### Example with Full CLI Overrides
+
+```bash
+mdwint data-clone \
+  --api_server "https://api.prod.example.com/1.0/" \
+  --api_username "sync_user" \
+  --api_password "s3cret" \
+  --db_provider "MySQL" \
+  --db_host "db.local" \
+  --db_port 3306 \
+  --db_username "clone_user" \
+  --db_password "db_pass" \
+  --db_name "prod_clone" \
+  --schema_path "./schema/Model-Extended.json" \
+  --sync_mode "Initial"
+```
+
+## Complete Working Examples
+
+### Example 1: MySQL Initial Clone
+
+`.meadow.config.json`:
+
+```json
+{
+    "Source": {
+        "ServerURL": "https://api.example.com/1.0/",
+        "UserID": "admin",
+        "Password": "admin_password"
+    },
+    "Destination": {
+        "Provider": "MySQL",
+        "MySQL": {
+            "server": "127.0.0.1",
+            "port": 3306,
+            "user": "root",
+            "password": "root_password",
+            "database": "app_clone",
+            "connectionLimit": 20
+        }
+    },
+    "SchemaPath": "./schema/Model-Extended.json",
+    "Sync": {
+        "DefaultSyncMode": "Initial",
+        "PageSize": 200,
+        "SyncEntityList": []
+    }
+}
+```
+
+Run:
+
+```bash
+mdwint data-clone
+```
+
+### Example 2: MSSQL Ongoing Sync with Specific Entities
+
+`.meadow.config.json`:
+
+```json
+{
+    "Source": {
+        "ServerURL": "https://api.example.com/1.0/",
+        "UserID": "sync_service",
+        "Password": "service_password"
+    },
+    "Destination": {
+        "Provider": "MSSQL",
+        "MSSQL": {
+            "server": "sql-server.local",
+            "port": 1433,
+            "user": "sa",
+            "password": "YourStrong!Passw0rd",
+            "database": "warehouse",
+            "ConnectionPoolLimit": 10
+        }
+    },
+    "SchemaPath": "/opt/schemas/Model-Extended.json",
+    "Sync": {
+        "DefaultSyncMode": "Ongoing",
+        "PageSize": 100,
+        "SyncEntityList": [
+            "Customer",
+            "Order",
+            "OrderItem",
+            "Product"
+        ],
+        "SyncEntityOptions": {}
+    }
+}
+```
+
+Run:
+
+```bash
+mdwint data-clone
+```
+
+### Example 3: No Authentication (Public API)
+
+`.meadow.config.json`:
+
+```json
+{
+    "Source": {
+        "ServerURL": "http://public-api.example.com/1.0/",
+        "UserID": false,
+        "Password": false
+    },
+    "Destination": {
+        "Provider": "MySQL",
+        "MySQL": {
+            "server": "127.0.0.1",
+            "port": 3306,
+            "user": "root",
+            "password": "",
+            "database": "public_data",
+            "connectionLimit": 20
+        }
+    },
+    "SchemaPath": "./schema/PublicModel-Extended.json",
+    "Sync": {
+        "DefaultSyncMode": "Initial",
+        "PageSize": 500,
+        "SyncEntityList": []
+    }
+}
+```
+
+### Example 4: CLI Overrides on Top of Config File
+
+Given a minimal `.meadow.config.json`:
+
+```json
+{
+    "SchemaPath": "./schema/Model-Extended.json",
+    "Sync": {
+        "DefaultSyncMode": "Initial",
+        "PageSize": 100
+    }
+}
+```
+
+Override the source and destination at runtime:
+
+```bash
+mdwint data-clone \
+  --api_server "https://staging.example.com/1.0/" \
+  --api_username "dev" \
+  --api_password "dev_pass" \
+  --db_host "localhost" \
+  --db_name "staging_clone" \
+  --sync_mode "Ongoing"
+```
+
+## Configuration Validation
+
+The `data-clone` command validates the following before starting:
+
+1. `Source.ServerURL` must be set (via config or `--api_server`).
+2. `SchemaPath` must be set (via config or `--schema_path`).
+3. The destination database configuration for the selected provider must include at least `server` and `database`.
+
+If any of these are missing, the command exits with an error message indicating which configuration is required.
+
+## Using the Explanation Command
+
+The CLI includes a built-in command to display the resolved configuration:
+
+```bash
+mdwint config
+```
+
+This shows the merged result of default values, configuration file values, and CLI overrides, helping you verify what settings will be used before running a data clone.

package/docs/data-clone/connection-manager.md

@@ -0,0 +1,206 @@
+# ConnectionManager
+
+The `MeadowConnectionManager` service manages database connections for the data clone system. It supports MySQL and MSSQL providers, handles connection pooling, and provides automatic index creation on synced tables.
+
+## Provider Selection
+
+Set the `Provider` property to choose the database backend:
+
+| Provider | Value | Required Module |
+|----------|-------|-----------------|
+| MySQL | `"MySQL"` | `meadow-connection-mysql` |
+| MSSQL | `"MSSQL"` | `meadow-connection-mssql` |
+
+Both modules are included as dependencies in `meadow-integration`.
+
+## MySQL Configuration
+
+### Options
+
+| Property | Type | Default | Description |
+|----------|------|---------|-------------|
+| `server` | string | `"127.0.0.1"` | MySQL server hostname or IP address |
+| `port` | number | `3306` | MySQL server port |
+| `user` | string | `"root"` | Database username |
+| `password` | string | `""` | Database password |
+| `database` | string | `"meadow"` | Database name |
+| `connectionLimit` | number | `20` | Maximum number of connections in the pool |
+
+### Example
+
+```javascript
+const meadowIntegration = require('meadow-integration');
+const libFable = require('fable');
+
+const fable = new libFable({});
+
+fable.serviceManager.addServiceType('MeadowConnectionManager', meadowIntegration.ConnectionManager);
+fable.serviceManager.instantiateServiceProvider('MeadowConnectionManager', {
+    Provider: 'MySQL',
+    MySQL: {
+        server: '127.0.0.1',
+        port: 3306,
+        user: 'app_user',
+        password: 'secure_password',
+        database: 'meadow_clone',
+        connectionLimit: 20
+    }
+});
+
+fable.MeadowConnectionManager.connect((pError, pConnectionPool) => {
+    if (pError) {
+        console.error('MySQL connection failed:', pError);
+        return;
+    }
+    console.log('Connected to MySQL. Pool ready.');
+    // pConnectionPool is the mysql2 pool instance
+});
+```
+
+### How MySQL Connection Works
+
+1. The `meadow-connection-mysql` module is loaded dynamically.
+2. The `connectionLimit` is propagated to `fable.settings` for the Meadow provider.
+3. MySQL settings are applied to `fable.settings.MySQL`.
+4. The provider is registered as `MeadowMySQLProvider` and connected via `connectAsync`.
+5. The resulting connection pool is stored in `ConnectionManager.ConnectionPool`.
+
+## MSSQL Configuration
+
+### Options
+
+| Property | Type | Default | Description |
+|----------|------|---------|-------------|
+| `server` | string | `"127.0.0.1"` | MSSQL server hostname or IP address |
+| `port` | number | `1433` | MSSQL server port |
+| `user` | string | `"sa"` | Database username |
+| `password` | string | `""` | Database password |
+| `database` | string | `"meadow"` | Database name |
+| `ConnectionPoolLimit` | number | `20` | Maximum number of connections in the pool |
+
+### Example
+
+```javascript
+const meadowIntegration = require('meadow-integration');
+const libFable = require('fable');
+
+const fable = new libFable({});
+
+fable.serviceManager.addServiceType('MeadowConnectionManager', meadowIntegration.ConnectionManager);
+fable.serviceManager.instantiateServiceProvider('MeadowConnectionManager', {
+    Provider: 'MSSQL',
+    MSSQL: {
+        server: 'sql-server.local',
+        port: 1433,
+        user: 'sa',
+        password: 'YourStrong!Passw0rd',
+        database: 'meadow_clone',
+        ConnectionPoolLimit: 20
+    }
+});
+
+fable.MeadowConnectionManager.connect((pError, pConnectionPool) => {
+    if (pError) {
+        console.error('MSSQL connection failed:', pError);
+        return;
+    }
+    console.log('Connected to MSSQL. Pool ready.');
+});
+```
+
+### How MSSQL Connection Works
+
+1. The `meadow-connection-mssql` module is loaded dynamically.
+2. MSSQL settings are applied to `fable.settings.MSSQL`.
+3. The provider is registered as `MeadowMSSQLProvider` and connected via `connectAsync`.
+4. The resulting connection pool is stored in `ConnectionManager.ConnectionPool`.
+
+## Automatic Index Creation
+
+During sync initialization, the `SyncEntity` (both Initial and Ongoing) automatically creates indexes on two column types:
+
+### GUID Column Index
+
+A **unique** index is created on any column with `DataType: "GUID"` in the Meadow schema. The index is named `AK_{TableName}_{ColumnName}`.
+
+### Deleted Column Index
+
+A **non-unique** index is created on the `Deleted` column if it exists. This speeds up queries that filter on soft-deleted records.
+
+### Index Creation Behavior
+
+- **MySQL**: Uses `INFORMATION_SCHEMA.STATISTICS` to check if the index already exists before creating it. Duplicate key errors (`ER_DUP_KEYNAME`) are silently ignored.
+- **MSSQL**: Uses `sys.indexes` with `IF NOT EXISTS` to conditionally create the index.
+- If no GUID or Deleted columns exist on a table, index creation is skipped entirely.
+- If no `ConnectionManager` or connection pool is available, index creation is skipped with a log message.
+
+### Programmatic Index Creation
+
+You can also create indexes manually:
+
+```javascript
+const entitySchema = {
+    TableName: 'Book',
+    Columns: [
+        { Column: 'GUIDBook', DataType: 'GUID' }
+    ]
+};
+
+fable.MeadowConnectionManager.createIndex(
+    entitySchema,
+    { Column: 'GUIDBook' },
+    true,  // unique
+    (pError) => {
+        if (pError) {
+            console.error('Index creation failed:', pError);
+        } else {
+            console.log('Index created successfully.');
+        }
+    }
+);
+```
+
+## Connection Pooling
+
+Both MySQL and MSSQL use connection pooling to manage database connections efficiently:
+
+- **MySQL**: Uses the `connectionLimit` property (default 20) via the `mysql2` pool.
+- **MSSQL**: Uses the `ConnectionPoolLimit` property (default 20) via the `mssql` pool.
+
+The pool is created once during `connect()` and reused for all subsequent queries within the sync process.
+
+## Checking Connection State
+
+```javascript
+if (fable.MeadowConnectionManager.connected) {
+    console.log('Database is connected.');
+} else {
+    console.log('Database is not connected.');
+}
+```
+
+## Default Configuration
+
+The default configuration is exported as `MeadowConnectionManager.default_configuration`:
+
+```json
+{
+    "Provider": "MySQL",
+    "MySQL": {
+        "server": "127.0.0.1",
+        "port": 3306,
+        "user": "root",
+        "password": "",
+        "database": "meadow",
+        "connectionLimit": 20
+    },
+    "MSSQL": {
+        "server": "127.0.0.1",
+        "port": 1433,
+        "user": "sa",
+        "password": "",
+        "database": "meadow",
+        "ConnectionPoolLimit": 20
+    }
+}
+```