meadow-integration 1.0.4 → 1.0.6

package/docs/cli/csvcheck.md

@@ -0,0 +1,111 @@
+# csvcheck
+
+Analyze a CSV file and produce column-level statistics. This is typically the first step when working with a new data set, helping you understand the structure, column names, and data characteristics before writing mapping files.
+
+**Aliases:** `csv_c`, `csv_check`
+
+## Usage
+
+```shell
+mdwint csvcheck <file> [options]
+```
+
+## Arguments
+
+| Argument | Required | Description |
+|----------|----------|-------------|
+| `<file>` | Yes | Path to the CSV file to analyze |
+
+## Options
+
+| Option | Description | Default |
+|--------|-------------|---------|
+| `-f, --file <filepath>` | Alternate way to specify the CSV file path | -- |
+| `-o, --output <filepath>` | Output file path for the statistics JSON | `./CSV-Stats-<filename>.json` |
+| `-r, --records` | Include all parsed records in the output statistics object | `false` |
+| `-q, --quotedelimiter <char>` | The quote delimiter character for CSV parsing | `"` |
+
+## Output Format
+
+The command writes a JSON file containing the following structure:
+
+```json
+{
+  "RowCount": 100,
+  "ColumnCount": 8,
+  "Headers": ["id", "name", "email", "age", "city", "state", "zip", "joined"],
+  "FirstRow": { "id": "1", "name": "Alice Smith", "..." : "..." },
+  "LastRow": { "id": "100", "name": "Bob Jones", "..." : "..." },
+  "ColumnStatistics": {
+    "id": { "Count": 100, "EmptyCount": 0, "NumericCount": 100, "FirstValue": "1", "LastValue": "100" },
+    "name": { "Count": 100, "EmptyCount": 2, "NumericCount": 0, "FirstValue": "Alice Smith", "LastValue": "Bob Jones" }
+  }
+}
+```
+
+| Field | Description |
+|-------|-------------|
+| `RowCount` | Total number of data rows (excluding header) |
+| `ColumnCount` | Number of columns detected |
+| `Headers` | Array of column header names |
+| `FirstRow` | The first data record as a key-value object |
+| `LastRow` | The last data record as a key-value object |
+| `ColumnStatistics` | Per-column statistics including count, empty count, numeric count, first value, and last value |
+
+When the `-r` flag is used, an additional `Records` property is included containing every parsed row.
+
+## Console Output
+
+In addition to writing the statistics file, the command prints a summary to the console:
+
+```
+Parsing CSV file [./data/books.csv]...
+...CSV parser completed, examined 100 rows of data.
+...Found 8 columns in the CSV file.
+...Writing statistics to file [./CSV-Stats-books.csv.json]...
+...Statistics written.
+Summary: 100 rows, 8 columns in [./data/books.csv].
+  Headers: id, name, email, age, city, state, zip, joined
+  First Row: {"id":"1","name":"Alice Smith",...}
+  Last Row: {"id":"100","name":"Bob Jones",...}
+  Column Statistics:
+    -> [id]: {"Count":100,"EmptyCount":0,"NumericCount":100,...}
+    -> [name]: {"Count":100,"EmptyCount":2,"NumericCount":0,...}
+```
+
+## Examples
+
+### Basic CSV analysis
+
+```shell
+mdwint csvcheck ./data/books.csv
+```
+
+Writes statistics to `./CSV-Stats-books.csv.json`.
+
+### Specify an output file
+
+```shell
+mdwint csvcheck ./data/books.csv -o ./output/book-stats.json
+```
+
+### Include all records in the output
+
+```shell
+mdwint csvcheck ./data/users.csv -r -o ./output/user-stats-full.json
+```
+
+This is useful for debugging or when you want to inspect the fully parsed data alongside the statistics.
+
+### Using the alias
+
+```shell
+mdwint csv_check ./data/products.csv -o product-stats.json
+```
+
+## Tips
+
+- Run `csvcheck` before writing a mapping file. The `Headers` array in the output tells you the exact column names to reference in your `{~D:Record.<column>~}` templates.
+- The `ColumnStatistics` can help identify which columns contain numeric data, which have empty values, and which might be good candidates for GUID generation.
+- Use the `-r` flag sparingly on large files, as it embeds every record in the output JSON.
+- If your CSV uses a non-standard quote character, pass it with `-q`. For example, `-q "'"` for single-quoted fields.

package/docs/cli/csvtransform.md

@@ -0,0 +1,170 @@
+# csvtransform
+
+Transform a CSV file into a Comprehension JSON file. This is the primary command for converting tabular data into the Meadow Comprehension format. It supports implicit mapping (auto-detected from column headers), explicit mapping files, and command-line configuration options.
+
+**Aliases:** `csv_t`, `csv_transform`
+
+## Usage
+
+```shell
+mdwint csvtransform <file> [options]
+```
+
+## Arguments
+
+| Argument | Required | Description |
+|----------|----------|-------------|
+| `<file>` | Yes | Path to the CSV file to transform |
+
+## Options
+
+| Option | Description | Default |
+|--------|-------------|---------|
+| `-e, --entity <name>` | Entity name in the Comprehension | Auto-detected from filename |
+| `-n, --guidname <name>` | Name of the GUID column in the generated Comprehension | `GUID<Entity>` |
+| `-g, --guidtemplate <template>` | Pict template for generating the entity GUID | `GUID-<Entity>-{~Data:Record.<first_column>~}` |
+| `-c, --columns <mappings>` | Inline column mappings in `Key={~D:col~}` format | Auto-detected 1:1 mapping |
+| `-m, --mappingfile <path>` | Path to a JSON mapping file | -- |
+| `-i, --incoming <path>` | Existing Comprehension file to merge into | -- |
+| `-o, --output <path>` | Output file path | `./CSV-Comprehension-<filename>.json` |
+| `-x, --extended` | Output full operation state, not just the Comprehension | `false` |
+| `-q, --quotedelimiter <char>` | Quote delimiter character for CSV parsing | `"` |
+
+## Configuration Priority
+
+When running a transform, three layers of configuration are merged. Each layer overrides the previous:
+
+1. **Implicit** -- Auto-detected from the first CSV row. The entity name is derived from the filename, and a 1:1 column mapping is generated.
+2. **Explicit** -- Loaded from a mapping file via `-m`. See [Mapping Files](../mapping-files.md) for the full specification.
+3. **User** -- Command-line options (`-e`, `-n`, `-g`, `-c`). These always take highest priority.
+
+## Output Format
+
+The standard output is a Comprehension JSON object:
+
+```json
+{
+  "Book": {
+    "Book_1": {
+      "GUIDBook": "Book_1",
+      "Title": "The Hunger Games",
+      "Language": "eng",
+      "ISBN": "0439023483"
+    },
+    "Book_2": {
+      "GUIDBook": "Book_2",
+      "Title": "Harry Potter",
+      "Language": "eng",
+      "ISBN": "0439554934"
+    }
+  }
+}
+```
+
+When the `-x` flag is used, the output includes the full operation state with configuration details, implicit/explicit configurations, and parsed row counts.
+
+## Examples
+
+### Implicit transform (auto-detect everything)
+
+```shell
+mdwint csvtransform ./data/books.csv -o books.json
+```
+
+The entity name, GUID template, and column mappings are all auto-detected from the filename and CSV headers.
+
+### Transform with CLI options
+
+```shell
+mdwint csvtransform ./data/books.csv \
+  -e Book \
+  -n GUIDBook \
+  -g "Book_{~D:Record.id~}" \
+  -o books.json
+```
+
+### Transform with inline column mappings
+
+```shell
+mdwint csvtransform ./data/books.csv \
+  -e Book \
+  -g "Book_{~D:Record.id~}" \
+  -c "Title={~D:Record.title~},Language={~D:Record.language_code~},ISBN={~D:Record.isbn~}" \
+  -o books.json
+```
+
+### Transform with a mapping file
+
+```shell
+mdwint csvtransform ./data/books.csv \
+  -m mapping_Book.json \
+  -o books.json
+```
+
+Where `mapping_Book.json` contains:
+
+```json
+{
+  "Entity": "Book",
+  "GUIDTemplate": "Book_{~D:Record.id~}",
+  "Mappings": {
+    "Title": "{~D:Record.title~}",
+    "Language": "{~D:Record.language_code~}",
+    "ISBN": "{~D:Record.isbn~}"
+  }
+}
+```
+
+### Merge into an existing Comprehension
+
+```shell
+# First pass: create Book entities
+mdwint csvtransform ./data/books.csv \
+  -m mapping_Book.json \
+  -o store.json
+
+# Second pass: add Author entities to the same file
+mdwint csvtransform ./data/books.csv \
+  -m mapping_Author.json \
+  -i store.json \
+  -o store.json
+```
+
+### Extended output for debugging
+
+```shell
+mdwint csvtransform ./data/books.csv \
+  -m mapping_Book.json \
+  -x \
+  -o debug-output.json
+```
+
+### Custom quote delimiter
+
+```shell
+mdwint csvtransform ./data/pipe-quoted.csv \
+  -m mapping.json \
+  -q "'" \
+  -o output.json
+```
+
+## Tips
+
+- Use `csvcheck` first to inspect your CSV and get the exact column header names for your mapping templates.
+- When building multi-entity Comprehensions from a single CSV, use the `-i` flag on subsequent passes to merge entities into one file.
+- Inline column mappings (`-c`) do not support commas or equal signs within template values. Use a mapping file (`-m`) for complex mappings.
+- Duplicate GUIDs within the same transform run are merged automatically, with later values overwriting earlier ones.
+- The `-x` flag is useful for debugging mapping issues. It outputs the full operation state including which configuration layer was used.
+
+## Notes
+
+- If no output file is specified, the default is `./CSV-Comprehension-<filename>.json`.
+- If no incoming comprehension file is specified, the default path is checked; if it exists, it is loaded and merged into.
+- The command uses streaming line-by-line parsing, so it can handle large CSV files without loading the entire file into memory.
+
+## See Also
+
+- [Mapping Files](../mapping-files.md) -- Full mapping file specification
+- [Comprehensions](../comprehensions.md) -- Core data structure documentation
+- [tsvtransform](tsvtransform.md) -- Same command for tab-delimited files
+- [jsonarraytransform](jsonarraytransform.md) -- Same command for JSON array files

package/docs/cli/data-clone.md

@@ -0,0 +1,277 @@
+# data-clone
+
+Clone data from a Meadow API source to a local MySQL or MSSQL database. This command connects to a Meadow REST API server, reads entity data according to an extended schema definition, and writes it into a local relational database. It supports both initial full clones and ongoing incremental synchronization.
+
+**Aliases:** `clone`, `sync`
+
+## Usage
+
+```shell
+mdwint data-clone [options]
+```
+
+## Options
+
+### API Connection
+
+| Option | Description | Default |
+|--------|-------------|---------|
+| `-a, --api_server <url>` | Source Meadow API server URL | -- |
+| `-u, --api_username <username>` | API authentication username | -- |
+| `-p, --api_password <password>` | API authentication password | -- |
+
+### Database Connection
+
+| Option | Description | Default |
+|--------|-------------|---------|
+| `-d, --db_provider <provider>` | Database provider: `MySQL` or `MSSQL` | `MySQL` |
+| `-dh, --db_host <host>` | Database server hostname or IP address | -- |
+| `-dp, --db_port <port>` | Database server port | Provider default (3306 for MySQL, 1433 for MSSQL) |
+| `-du, --db_username <username>` | Database authentication username | -- |
+| `-dw, --db_password <password>` | Database authentication password | -- |
+| `-dn, --db_name <name>` | Target database name | -- |
+
+### Schema and Sync
+
+| Option | Description | Default |
+|--------|-------------|---------|
+| `-sp, --schema_path <path>` | Path to a Meadow extended schema JSON file defining the entities and fields to clone | -- |
+| `-s, --sync_mode <mode>` | Synchronization mode: `Initial` for a full clone, `Ongoing` for incremental updates | `Initial` |
+| `-w, --post_run_delay <minutes>` | Minutes to wait after a sync run completes before exiting (or re-running in ongoing mode) | `0` |
+
+## Configuration File
+
+Instead of passing all options on the command line, you can create a `.meadow.config.json` file in your working directory. The command reads this file automatically and merges its values with any CLI flags provided (CLI flags take precedence).
+
+### MySQL Configuration
+
+```json
+{
+  "api_server": "https://api.example.com",
+  "api_username": "integration-user",
+  "api_password": "secret-token",
+
+  "db_provider": "MySQL",
+  "db_host": "localhost",
+  "db_port": 3306,
+  "db_username": "root",
+  "db_password": "mysql-password",
+  "db_name": "meadow_clone",
+
+  "schema_path": "./schema/extended-schema.json",
+  "sync_mode": "Initial",
+  "post_run_delay": 0
+}
+```
+
+### MSSQL Configuration
+
+```json
+{
+  "api_server": "https://api.example.com",
+  "api_username": "integration-user",
+  "api_password": "secret-token",
+
+  "db_provider": "MSSQL",
+  "db_host": "sql-server.local",
+  "db_port": 1433,
+  "db_username": "sa",
+  "db_password": "mssql-password",
+  "db_name": "MeadowClone",
+
+  "schema_path": "./schema/extended-schema.json",
+  "sync_mode": "Initial"
+}
+```
+
+### With Entity List
+
+You can scope the clone to specific entities by including an `entities` array in the configuration:
+
+```json
+{
+  "api_server": "https://api.example.com",
+  "api_username": "integration-user",
+  "api_password": "secret-token",
+
+  "db_provider": "MySQL",
+  "db_host": "localhost",
+  "db_port": 3306,
+  "db_username": "root",
+  "db_password": "mysql-password",
+  "db_name": "meadow_clone",
+
+  "schema_path": "./schema/extended-schema.json",
+  "sync_mode": "Initial",
+
+  "entities": ["Book", "Author", "BookAuthorJoin"]
+}
+```
+
+### Ongoing Sync with Delay
+
+For continuous synchronization, set the sync mode to `Ongoing` and configure a delay between runs:
+
+```json
+{
+  "api_server": "https://api.example.com",
+  "api_username": "integration-user",
+  "api_password": "secret-token",
+
+  "db_provider": "MySQL",
+  "db_host": "localhost",
+  "db_port": 3306,
+  "db_username": "root",
+  "db_password": "mysql-password",
+  "db_name": "meadow_sync",
+
+  "schema_path": "./schema/extended-schema.json",
+  "sync_mode": "Ongoing",
+  "post_run_delay": 15
+}
+```
+
+This configuration waits 15 minutes between each sync cycle.
+
+### Minimal Configuration
+
+If you are using environment variables or default values for most settings, a minimal config might look like:
+
+```json
+{
+  "api_server": "https://api.example.com",
+  "api_username": "admin",
+  "api_password": "password",
+
+  "db_provider": "MySQL",
+  "db_host": "localhost",
+  "db_name": "local_clone",
+
+  "schema_path": "./schema.json"
+}
+```
+
+## Sync Modes
+
+### Initial
+
+Performs a full clone of all entity data from the API source. The target database tables are created (or recreated) based on the schema definition, and all records are fetched and inserted.
+
+Use this mode when:
+- Setting up a new local database for the first time
+- You need a complete refresh of all data
+- The schema has changed and tables need to be rebuilt
+
+### Ongoing
+
+Performs incremental synchronization, fetching only records that have been created or modified since the last sync. This mode is much faster than a full clone for large datasets.
+
+Use this mode when:
+- You have already performed an initial clone
+- You want to keep the local database up to date with the API source
+- You are running the clone as a scheduled or continuous process
+
+## Examples
+
+### Full clone with CLI flags
+
+```shell
+mdwint data-clone \
+  -a https://api.example.com \
+  -u admin \
+  -p secret123 \
+  -d MySQL \
+  -dh localhost \
+  -dp 3306 \
+  -du root \
+  -dw rootpass \
+  -dn meadow_clone \
+  -sp ./schema/extended-schema.json \
+  -s Initial
+```
+
+### Clone using a config file
+
+```shell
+# With .meadow.config.json in the current directory:
+mdwint data-clone
+```
+
+### Override config file values with CLI flags
+
+```shell
+# Use config file for most settings, override the sync mode
+mdwint data-clone -s Ongoing -w 10
+```
+
+### Clone to MSSQL
+
+```shell
+mdwint data-clone \
+  -a https://api.example.com \
+  -u admin \
+  -p secret123 \
+  -d MSSQL \
+  -dh sql-server.local \
+  -dp 1433 \
+  -du sa \
+  -dw sapassword \
+  -dn MeadowClone \
+  -sp ./schema.json
+```
+
+### Ongoing sync with delay
+
+```shell
+mdwint data-clone \
+  -a https://api.example.com \
+  -u admin \
+  -p secret123 \
+  -d MySQL \
+  -dh localhost \
+  -dn meadow_sync \
+  -sp ./schema.json \
+  -s Ongoing \
+  -w 15
+```
+
+This runs a sync, waits 15 minutes, then repeats.
+
+### Using aliases
+
+```shell
+mdwint clone -a https://api.example.com -sp ./schema.json
+mdwint sync -s Ongoing -w 5
+```
+
+## Schema File
+
+The `--schema_path` option points to a Meadow extended schema JSON file. This file defines the entities and their fields that should be cloned. The schema determines:
+
+- Which entities to fetch from the API
+- What database tables and columns to create
+- Field types and constraints for the local database
+
+Refer to the Meadow [Schema](../vocabulary/Schema.md) documentation for the schema file format.
+
+## Tips
+
+- Always run an `Initial` sync before switching to `Ongoing` mode. The ongoing mode expects the database tables and baseline data to already exist.
+- Use the `.meadow.config.json` file for settings you reuse across runs. Use CLI flags for one-off overrides (e.g., switching between Initial and Ongoing mode).
+- For production deployments, avoid putting passwords directly in the config file. Use environment variables or a secrets manager.
+- The `post_run_delay` option is useful for running the clone as a long-lived process that keeps data in sync. Set it to `0` for a single run that exits after completion.
+- When scoping to specific entities via the `entities` array in the config file, ensure that any entity dependencies (e.g., join tables referencing parent entities) are also included.
+
+## Notes
+
+- The database must already exist before running the clone. The command creates tables but does not create the database itself.
+- The command requires the `meadow-connection-mysql` or `meadow-connection-mssql` packages, which are included as dependencies of `meadow-integration`.
+- The API server must be a Meadow server with REST endpoints for the entities defined in the schema.
+- Configuration file values and CLI flags are merged, with CLI flags taking precedence.
+
+## See Also
+
+- [load_comprehension](load-comprehension.md) -- Push local Comprehension data to a Meadow API
+- [serve](serve.md) -- Start a local Meadow Integration REST API server
+- [Schema](../vocabulary/Schema.md) -- Meadow schema file format
+- [Integration Adapter](../integration-adapter.md) -- Programmatic data integration