meadow-integration 1.0.4 → 1.0.6

package/docs/api/tabular-check.md

@@ -0,0 +1,213 @@
+# Service-TabularCheck (MeadowIntegrationTabularCheck)
+
+Statistical analysis service for tabular data records. Collects column-level statistics including row counts, empty value counts, numeric detection, and first/last values for data quality inspection before transformation or integration.
+
+**Source:** `source/services/tabular/Service-TabularCheck.js`
+
+**Extends:** `fable-serviceproviderbase`
+
+## Constructor
+
+```js
+const tabularCheck = fable.serviceManager.instantiateServiceProvider('TabularCheck', pOptions);
+```
+
+No custom options are required.
+
+## Methods
+
+### `newStatisticsObject(pTabularDatasetName)`
+
+Creates a new, empty statistics container object for tracking dataset metrics.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `pTabularDatasetName` | `string` | A name identifying the dataset (e.g. `'airports.csv'`). |
+
+**Returns:** `object` -- A statistics object with the following shape:
+
+```js
+{
+	DataSet: 'airports.csv',       // Name of the dataset
+	FirstRow: null,                // Reference to the first record processed
+	RowCount: 0,                   // Total number of rows processed
+	LastRow: null,                 // Reference to the most recent record processed
+	Headers: [],                   // Array of column name strings, in discovery order
+	ColumnCount: 0,                // Total unique columns discovered
+	ColumnStatistics: {},          // Per-column statistics (see below)
+	Records: null                  // Array of all records (only if pStoreFullRecord is true)
+}
+```
+
+### `collectStatistics(pRecord, pStatisticsObject, pStoreFullRecord)`
+
+Processes a single record and updates the statistics object with its data.
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `pRecord` | `object` | -- | A single data record (key-value object). |
+| `pStatisticsObject` | `object` | *(auto-created)* | The statistics object to update. If not a valid object, a new one is created automatically. |
+| `pStoreFullRecord` | `boolean` | `false` | If `true`, pushes the full record into `Records` array for later inspection. |
+
+**Returns:** `object` -- The updated statistics object.
+
+**Per-column statistics shape** (each entry in `ColumnStatistics`):
+
+```js
+{
+	Count: 0,           // Number of records that have this column
+	EmptyCount: 0,      // Number of records where this column is null or empty string
+	NumericCount: 0,    // Number of records where this column's value is numeric
+	FirstValue: null,   // The first non-null value seen for this column
+	LastValue: null      // The most recent value seen for this column
+}
+```
+
+**Behavior notes:**
+- Each call increments `RowCount` by 1.
+- The first record processed is stored as `FirstRow`; every record updates `LastRow`.
+- New columns are added to `Headers` and `ColumnStatistics` as they are discovered.
+- A value is considered numeric if `fable.Math.parsePrecise(value, NaN)` returns a valid number.
+- A value is considered empty if it is strictly `null` or an empty string `''`.
+- Callers are responsible for ensuring each record is only sent through once.
+
+## Statistics Object Shape (Complete)
+
+```js
+{
+	DataSet: 'my-dataset',
+	FirstRow: { Name: 'Alice', Age: '30', City: 'Portland' },
+	RowCount: 1000,
+	LastRow: { Name: 'Zoe', Age: '25', City: 'Seattle' },
+	Headers: ['Name', 'Age', 'City'],
+	ColumnCount: 3,
+	ColumnStatistics:
+	{
+		Name:
+		{
+			Count: 1000,
+			EmptyCount: 5,
+			NumericCount: 0,
+			FirstValue: 'Alice',
+			LastValue: 'Zoe'
+		},
+		Age:
+		{
+			Count: 1000,
+			EmptyCount: 12,
+			NumericCount: 988,
+			FirstValue: '30',
+			LastValue: '25'
+		},
+		City:
+		{
+			Count: 1000,
+			EmptyCount: 50,
+			NumericCount: 0,
+			FirstValue: 'Portland',
+			LastValue: 'Seattle'
+		}
+	},
+	Records: null
+}
+```
+
+## Usage Examples
+
+### Basic Dataset Statistics
+
+```js
+const libFable = require('fable');
+const libTabularCheck = require('meadow-integration/source/services/tabular/Service-TabularCheck');
+
+const fable = new libFable({ Product: 'DataInspector' });
+
+fable.serviceManager.addServiceType('TabularCheck', libTabularCheck);
+const checker = fable.serviceManager.instantiateServiceProvider('TabularCheck');
+
+// Create a statistics container
+const stats = checker.newStatisticsObject('customers.csv');
+
+// Process records one by one (e.g. from a CSV stream)
+const records =
+[
+	{ Name: 'Alice', Email: 'alice@example.com', Age: '30' },
+	{ Name: 'Bob', Email: '', Age: '25' },
+	{ Name: '', Email: 'charlie@example.com', Age: 'unknown' },
+	{ Name: 'Diana', Email: 'diana@example.com', Age: '35' }
+];
+
+for (const record of records)
+{
+	checker.collectStatistics(record, stats);
+}
+
+console.log(`Dataset: ${stats.DataSet}`);
+console.log(`Rows: ${stats.RowCount}`);           // 4
+console.log(`Columns: ${stats.ColumnCount}`);      // 3
+console.log(`Headers: ${stats.Headers.join(', ')}`); // Name, Email, Age
+
+console.log('Name empty count:', stats.ColumnStatistics.Name.EmptyCount);     // 1
+console.log('Email empty count:', stats.ColumnStatistics.Email.EmptyCount);   // 1
+console.log('Age numeric count:', stats.ColumnStatistics.Age.NumericCount);   // 3 (30, 25, 35)
+console.log('Age first value:', stats.ColumnStatistics.Age.FirstValue);       // '30'
+```
+
+### Storing Full Records for Inspection
+
+```js
+const stats = checker.newStatisticsObject('products.csv');
+stats.Records = []; // Must initialize the Records array before enabling storage
+
+for (const record of productRecords)
+{
+	checker.collectStatistics(record, stats, true);
+}
+
+console.log(`Stored ${stats.Records.length} records for inspection.`);
+console.log('First stored record:', stats.Records[0]);
+```
+
+### Auto-Created Statistics Object
+
+```js
+// If you pass a non-object, a statistics object is created automatically
+const stats = checker.collectStatistics({ Name: 'Alice', Age: '30' }, null);
+console.log(stats.DataSet); // 'Unknown-{uuid}'
+console.log(stats.RowCount); // 1
+```
+
+### Analyzing Column Quality
+
+```js
+const stats = checker.newStatisticsObject('orders.csv');
+
+for (const record of orderRecords)
+{
+	checker.collectStatistics(record, stats);
+}
+
+// Report columns with high empty rates
+for (const header of stats.Headers)
+{
+	const colStats = stats.ColumnStatistics[header];
+	const emptyRate = (colStats.EmptyCount / colStats.Count * 100).toFixed(1);
+
+	if (colStats.EmptyCount > 0)
+	{
+		console.log(`Column "${header}": ${emptyRate}% empty (${colStats.EmptyCount}/${colStats.Count})`);
+	}
+
+	// Detect likely numeric columns
+	const numericRate = (colStats.NumericCount / colStats.Count * 100).toFixed(1);
+	if (colStats.NumericCount > colStats.Count * 0.9)
+	{
+		console.log(`Column "${header}": likely numeric (${numericRate}% numeric values)`);
+	}
+}
+```
+
+## Related Services
+
+- [Service-TabularTransform](./tabular-transform.md) -- Transforms tabular records using mapping configurations; often used after TabularCheck validates the data shape.
+- [MeadowIntegrationAdapter](./integration-adapter.md) -- Integrates transformed records into a Meadow data store.

package/docs/api/tabular-transform.md

@@ -0,0 +1,316 @@
+# Service-TabularTransform (MeadowIntegrationTabularTransform)
+
+Transformation service for mapping tabular data records into Meadow entity comprehensions. Uses configurable column mapping templates to convert external data (CSV rows, API results, etc.) into entity records with auto-generated GUIDs, template-based field values, and optional solver-based transformations.
+
+**Source:** `source/services/tabular/Service-TabularTransform.js`
+
+**Extends:** `fable-serviceproviderbase`
+
+## Constructor
+
+```js
+const transform = fable.serviceManager.instantiateServiceProvider('TabularTransform', pOptions);
+```
+
+No custom options are required.
+
+## Mapping Configuration
+
+A mapping configuration defines how incoming data fields map to entity record fields. It can be provided explicitly or generated automatically from the first record.
+
+### Mapping Configuration Shape
+
+```js
+{
+	"Entity": "Airport",
+	"GUIDTemplate": "Airport-{~D:iata~}",
+	"GUIDName": "GUIDAirport",              // Auto-derived as GUID{Entity} if omitted
+	"Mappings":
+	{
+		"Code": "{~D:iata~}",
+		"Name": "{~D:name~}",
+		"Description": "{~D:name~} airport in {~D:city~}",
+		"City": "{~D:city~}",
+		"State": "{~D:state~}",
+		"Country": "{~D:country~}",
+		"Latitude": "{~D:lat~}",
+		"Longitude": "{~D:long~}"
+	},
+	"Solvers": [],                           // Optional array of expression solver definitions
+	"MultipleGUIDUniqueness": false,         // If true, supports multiple GUID uniqueness entries per record
+	"ManyfestAddresses": false               // If true, uses Manyfest setValueAtAddress for nested paths
+}
+```
+
+Template expressions use Fable's template engine syntax (e.g., `{~D:fieldname~}` or `{~Data:Record.fieldname~}`).
+
+## Mapping Outcome Object
+
+The mapping outcome object tracks the state and results of a transformation operation.
+
+### `newMappingOutcomeObject()`
+
+Creates a new, empty mapping outcome object.
+
+**Returns:**
+
+```js
+{
+	Comprehension: {},             // Generated records keyed by entity name, then by GUID
+	ExistingComprehension: false,  // Optional existing comprehension for merging with previous data
+
+	ImplicitConfiguration: false,  // Auto-generated configuration from first record
+	ExplicitConfiguration: false,  // User-provided mapping configuration file
+	UserConfiguration: {},         // Runtime overrides (e.g. different entity name)
+	Configuration: false,          // Final merged configuration (Implicit + Explicit + User)
+
+	ParsedRowCount: 0,             // Number of rows processed
+	BadRecords: []                 // Records that failed validation (no GUID, invalid data)
+}
+```
+
+## Methods
+
+### `newMappingOutcomeObject()`
+
+Creates a fresh mapping outcome container for a new transformation session.
+
+**Returns:** `object` -- An empty mapping outcome object (see shape above).
+
+### `generateMappingConfigurationPrototype(pRepresentativeString, pRecord)`
+
+Auto-generates a mapping configuration from a representative string (typically a filename) and a sample record.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `pRepresentativeString` | `string` | A name used to derive the entity name (e.g. `'my favorite cats.csv'` becomes `'MyFavoriteCats'`). |
+| `pRecord` | `object` | A sample record whose keys become the mapping fields. |
+
+**Returns:** `object` -- A mapping configuration with auto-generated `Entity`, `GUIDTemplate`, and `Mappings`.
+
+**Example:**
+
+```js
+const config = transform.generateMappingConfigurationPrototype('airport data', { iata: 'PDX', name: 'Portland' });
+// Returns:
+// {
+//   Entity: 'AirportData',
+//   GUIDTemplate: 'GUID-AirportData-{~Data:Record.iata~}',
+//   Mappings: {
+//     iata: '{~Data:Record.iata~}',
+//     name: '{~Data:Record.name~}'
+//   }
+// }
+```
+
+### `createRecordFromMapping(pRecord, pMapping, pRecordPrototype)`
+
+Creates a single entity record by applying a mapping configuration to an incoming data record.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `pRecord` | `object` | The incoming data record (used as template data context). |
+| `pMapping` | `object` | The mapping configuration with `GUIDName`, `GUIDTemplate`, and `Mappings`. |
+| `pRecordPrototype` | `object\|null` | Optional base object to merge onto (deep-cloned). Defaults to `{}`. |
+
+**Returns:** `object` -- The generated entity record with GUID and mapped fields.
+
+**Behavior:**
+- Sets the GUID field using `fable.parseTemplate(GUIDTemplate, pRecord)`.
+- For each key in `Mappings`, resolves the template expression against `pRecord`.
+- If `pMapping.ManyfestAddresses` is `true`, uses `fable.manifest.setValueAtAddress()` for nested property paths.
+
+### `addRecordToComprehension(pIncomingRecord, pMappingOutcome, pNewRecordPrototype, pGUIDUniquenessString)`
+
+Creates a record from mapping and adds it to the comprehension within the mapping outcome, handling duplicates and merging with existing data.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `pIncomingRecord` | `object` | The raw incoming data record. |
+| `pMappingOutcome` | `object` | The mapping outcome object containing the comprehension. |
+| `pNewRecordPrototype` | `object` | Optional record prototype for base values. |
+| `pGUIDUniquenessString` | `string` | Optional uniqueness string injected as `_GUIDUniqueness` into the record before mapping. |
+
+**Duplicate handling:**
+- If the generated GUID already exists in the current `Comprehension`, the new record is merged onto the existing one via `Object.assign`.
+- If the GUID exists in `ExistingComprehension` (previous run), it is pulled in and merged.
+- If the record has no valid GUID, it is added to `BadRecords`.
+
+### `transformRecord(pIncomingRecord, pMappingOutcomeObject)`
+
+The primary entry point for transforming a single incoming record. Handles initialization, solver execution, and comprehension insertion.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `pIncomingRecord` | `object` | The raw incoming data record. |
+| `pMappingOutcomeObject` | `object` | The mapping outcome object (created via `newMappingOutcomeObject()`). |
+
+**Behavior:**
+1. Initializes the mapping outcome if not already done (merges Implicit, Explicit, and User configurations).
+2. Increments `ParsedRowCount`.
+3. Creates a solution context object with `IncomingRecord`, `MappingConfiguration`, `RowIndex`, `Fable`, and `AppData`.
+4. If `Configuration.Solvers` is defined, executes each solver expression via `fable.ExpressionParser.solve()`. Solvers can modify `NewRecordPrototype` and `NewRecordsGUIDUniqueness`.
+5. If `MultipleGUIDUniqueness` is enabled and uniqueness entries exist, creates one record per uniqueness entry.
+6. Otherwise, creates a single record and adds it to the comprehension.
+
+### Lifecycle Hooks
+
+Two empty methods are available for subclass overrides:
+
+- **`onBeforeInitializeMappingOutcomeObject(pMappingOutcomeObject)`** -- Called before configuration merging.
+- **`onAfterInitializeMappingOutcomeObject(pMappingOutcomeObject)`** -- Called after configuration merging and GUID name setup.
+
+## Usage Examples
+
+### Basic CSV-to-Entity Transformation
+
+```js
+const libFable = require('fable');
+const libTabularTransform = require('meadow-integration/source/services/tabular/Service-TabularTransform');
+
+const fable = new libFable({ Product: 'DataImporter' });
+
+fable.serviceManager.addServiceType('TabularTransform', libTabularTransform);
+const transform = fable.serviceManager.instantiateServiceProvider('TabularTransform');
+
+// Define the mapping configuration
+const mappingConfig =
+{
+	Entity: 'Airport',
+	GUIDTemplate: 'Airport-{~Data:Record.iata~}',
+	Mappings:
+	{
+		Code: '{~Data:Record.iata~}',
+		Name: '{~Data:Record.name~}',
+		City: '{~Data:Record.city~}',
+		State: '{~Data:Record.state~}',
+		Country: '{~Data:Record.country~}'
+	}
+};
+
+// Create a mapping outcome and set the explicit configuration
+const outcome = transform.newMappingOutcomeObject();
+outcome.ExplicitConfiguration = mappingConfig;
+
+// Transform each CSV row
+const csvRows =
+[
+	{ iata: 'PDX', name: 'Portland International', city: 'Portland', state: 'OR', country: 'US' },
+	{ iata: 'SEA', name: 'Seattle-Tacoma International', city: 'Seattle', state: 'WA', country: 'US' },
+	{ iata: 'SFO', name: 'San Francisco International', city: 'San Francisco', state: 'CA', country: 'US' }
+];
+
+for (const row of csvRows)
+{
+	transform.transformRecord(row, outcome);
+}
+
+console.log(`Parsed ${outcome.ParsedRowCount} rows.`);
+console.log(`Bad records: ${outcome.BadRecords.length}`);
+console.log('Airport records:', Object.keys(outcome.Comprehension.Airport));
+// ['Airport-PDX', 'Airport-SEA', 'Airport-SFO']
+
+const pdxRecord = outcome.Comprehension.Airport['Airport-PDX'];
+console.log(pdxRecord);
+// {
+//   GUIDAirport: 'Airport-PDX',
+//   Code: 'PDX',
+//   Name: 'Portland International',
+//   City: 'Portland',
+//   State: 'OR',
+//   Country: 'US'
+// }
+```
+
+### Auto-Generated Configuration from Data
+
+```js
+const transform = fable.serviceManager.instantiateServiceProvider('TabularTransform');
+
+// Auto-generate a mapping from a sample record
+const sampleRecord = { iata: 'PDX', name: 'Portland', city: 'Portland', state: 'OR' };
+const autoConfig = transform.generateMappingConfigurationPrototype('airports', sampleRecord);
+
+console.log(autoConfig.Entity);        // 'Airports'
+console.log(autoConfig.GUIDTemplate);  // 'GUID-Airports-{~Data:Record.iata~}'
+console.log(autoConfig.Mappings);
+// {
+//   iata: '{~Data:Record.iata~}',
+//   name: '{~Data:Record.name~}',
+//   city: '{~Data:Record.city~}',
+//   state: '{~Data:Record.state~}'
+// }
+```
+
+### Creating a Single Record from Mapping
+
+```js
+const mapping =
+{
+	GUIDName: 'GUIDProduct',
+	GUIDTemplate: 'PROD-{~Data:Record.sku~}',
+	Mappings:
+	{
+		Name: '{~Data:Record.name~}',
+		Price: '{~Data:Record.price~}'
+	}
+};
+
+const record = transform.createRecordFromMapping(
+	{ sku: 'ABC-123', name: 'Widget', price: '19.99' },
+	mapping);
+
+console.log(record);
+// {
+//   GUIDProduct: 'PROD-ABC-123',
+//   Name: 'Widget',
+//   Price: '19.99'
+// }
+```
+
+### Merging with Existing Comprehension
+
+```js
+const outcome = transform.newMappingOutcomeObject();
+outcome.ExplicitConfiguration = mappingConfig;
+
+// Provide existing data from a previous run
+outcome.ExistingComprehension =
+{
+	Airport:
+	{
+		'Airport-PDX': { GUIDAirport: 'Airport-PDX', Code: 'PDX', Name: 'Portland Intl', Rating: 5 }
+	}
+};
+
+// Transform a record that matches an existing GUID
+transform.transformRecord(
+	{ iata: 'PDX', name: 'Portland International (Updated)', city: 'Portland', state: 'OR', country: 'US' },
+	outcome);
+
+// The existing record is merged with new data
+const merged = outcome.Comprehension.Airport['Airport-PDX'];
+console.log(merged.Rating);  // 5 (preserved from existing)
+console.log(merged.Name);    // 'Portland International (Updated)' (overwritten by new data)
+```
+
+### Using User Configuration Overrides
+
+```js
+const outcome = transform.newMappingOutcomeObject();
+outcome.ExplicitConfiguration = mappingConfig;
+
+// Override the entity name at runtime
+outcome.UserConfiguration = { Entity: 'InternationalAirport' };
+
+transform.transformRecord(csvRows[0], outcome);
+
+// Records are stored under the overridden entity name
+console.log(Object.keys(outcome.Comprehension)); // ['InternationalAirport']
+```
+
+## Related Services
+
+- [Service-TabularCheck](./tabular-check.md) -- Statistical analysis for tabular data; often used before transformation to validate the data shape.
+- [MeadowIntegrationAdapter](./integration-adapter.md) -- Integrates the comprehension output from TabularTransform into a Meadow data store.
+- [MeadowGUIDMap](./guid-map.md) -- Maintains GUID-to-ID mappings that may be needed during integration of transformed records.