meadow-integration 1.0.41 → 1.0.43

package/BUILDING-AND-PUBLISHING.md

@@ -39,7 +39,7 @@ the plain `release:patch` ships to npm and skips the image rebuild.
   `npm login`.
 - **Git remote configured** — `git remote get-url origin` should print
   the GitHub HTTPS or SSH URL. If not, `git remote add origin
-  git@github.com:stevenvelozo/meadow-integration.git`.
+  git@github.com:fable-retold/meadow-integration.git`.
 - **Push access to the repo** — required so `postversion` /
   `postpublish` can push the tag. The GHCR workflow runs under
   `GITHUB_TOKEN` so no extra registry auth needed for image pushing.
@@ -187,7 +187,7 @@ After `release:patch` completes:
 1. **npm**: `npm view meadow-integration version` should print the new
    version (may take ~30s for the registry to update).
 2. **GHCR workflow**: visit
-   `https://github.com/stevenvelozo/meadow-integration/actions` and
+   `https://github.com/fable-retold/meadow-integration/actions` and
    confirm the "Publish container image" run succeeded.
 3. **Image**: `docker pull ghcr.io/stevenvelozo/meadow-integration:<version>`
    should succeed. The image is also tagged as `latest`, `<major>`, and

package/README.md

@@ -1,8 +1,10 @@
 # Meadow Integration
 
+> **[Read the Meadow-Integration Documentation](https://fable-retold.github.io/meadow-integration/)** - interactive docs with the full API reference.
+
 A suite of tools for managing data into a centralized non-specific schema format called a **Comprehension**.
 
-These tools are built to be usable from the command-line, as a web service, or within your own codebase.  This module presents these behaviors both as a suite of externally usable fable services, a command-line utility to leverage them and a set of web service behaviors.
+These tools are built to be usable from the command-line, as a web service, or within your own codebase.  This module presents these behaviors both as a suite of externally usable fable services, a command-line utility to use them and a set of web service behaviors.
 
 ## What is a Comprehension?
 
@@ -230,6 +232,23 @@ The `data-clone` command synchronizes entity data from a remote Meadow REST API
 |------|-------------|
 | **Initial** | Compares the max entity ID on the server against the local database and downloads all records with IDs greater than the local maximum. Designed for first-time bulk population. |
 | **Ongoing** | Walks through all server records page by page, comparing `UpdateDate` timestamps. Creates missing records locally and updates records whose server timestamp is newer than the local copy. |
+| **OngoingEventualConsistency** | Time-budgeted variant of Ongoing: bisects the id range newest-first within `BackSyncTimeLimit`, then pulls new tail records. Reconciles deleted records separately (see below). |
+| **TrueUp** | Forward keyset walk of the whole server range to repair gaps/drift. |
+
+### Deleted-record reconciliation
+
+When `SyncDeletedRecords` is enabled, `OngoingEventualConsistency` reconciles server-side deletions into the local clone. Each deleted server row is matched to the local row by **identity only** — never by GUID, because GUIDs are not guaranteed unique in cloned data and a GUID match could soft-delete the wrong record — and flagged via `doDelete`. Rows whose id is not present locally are counted and skipped (not created). Processing is newest-first by the indexed id, within `BackSyncTimeLimit`.
+
+#### Resumable cursor options
+
+By default each run sweeps the deleted set newest-first from the top, which on a large backlog never reaches older records within the time budget. These options make the sweep resumable:
+
+| Option | Description |
+|--------|-------------|
+| `DeleteCursorStatePath` | (Optional) Path to a JSON file persisting per-table head/tail id marks so each run resumes the deleted-record sweep instead of re-walking from the newest record. Unset = disabled. The file must persist between runs to be useful; a missing/corrupt file degrades safely to a full sweep. Holds only a few integers per table — no record data. |
+| `DeleteResweepIntervalHours` | (Optional, default `168`) Once the backlog is drained, hours between full re-sweeps that catch deletions of older records landing in already-swept id ranges. |
+
+With the cursor enabled, each run runs a cheap **head pass** (`id > headID`, new deletions since last run) then continues the **tail** (`id < tailID`) within budget, using keyset paging (no growing `OFFSET`). Options propagate via `MeadowSync` options / per-entity `SyncEntityOptions`, and the strategy also reads matching keys from `fable.settings`.
 
 ### CLI Options
 
@@ -328,13 +347,13 @@ npm test
 
 ## Related Packages
 
-- [meadow](https://github.com/stevenvelozo/meadow) - Data access and ORM
-- [meadow-endpoints](https://github.com/stevenvelozo/meadow-endpoints) - Auto-generated REST endpoints
-- [meadow-connection-mysql](https://github.com/stevenvelozo/meadow-connection-mysql) - MySQL database provider for Meadow
-- [meadow-connection-mssql](https://github.com/stevenvelozo/meadow-connection-mssql) - MSSQL database provider for Meadow
-- [orator](https://github.com/stevenvelozo/orator) - API server abstraction
-- [fable](https://github.com/stevenvelozo/fable) - Application services framework
-- [pict-service-commandlineutility](https://github.com/stevenvelozo/pict-service-commandlineutility) - CLI framework with cascading configuration
+- [meadow](https://github.com/fable-retold/meadow) - Data access and ORM
+- [meadow-endpoints](https://github.com/fable-retold/meadow-endpoints) - Auto-generated REST endpoints
+- [meadow-connection-mysql](https://github.com/fable-retold/meadow-connection-mysql) - MySQL database provider for Meadow
+- [meadow-connection-mssql](https://github.com/fable-retold/meadow-connection-mssql) - MSSQL database provider for Meadow
+- [orator](https://github.com/fable-retold/orator) - API server abstraction
+- [fable](https://github.com/fable-retold/fable) - Application services framework
+- [pict-service-commandlineutility](https://github.com/fable-retold/pict-service-commandlineutility) - CLI framework with cascading configuration
 
 ## License
 

package/docs/README.md

@@ -2,7 +2,7 @@
 
 A suite of tools for managing data into a centralized non-specific schema format called a **Comprehension**.
 
-These tools are built to be usable from the command-line, as a web service, or within your own codebase.  This module presents these behaviors both as a suite of externally usable fable services, a command-line utility to leverage them and a set of web service behaviors.
+These tools are built to be usable from the command-line, as a web service, or within your own codebase.  This module presents these behaviors both as a suite of externally usable fable services, a command-line utility to use them and a set of web service behaviors.
 
 In addition to data transformation, Meadow Integration includes a **Data Clone** pipeline for replicating entity data from a remote Meadow REST API into a local MySQL or MSSQL database.
 

package/docs/_cover.md

@@ -8,5 +8,5 @@
 - Synchronize remote Meadow APIs to local databases
 - CLI utility, programmatic API, and REST integration
 
-[GitHub](https://github.com/stevenvelozo/meadow-integration)
+[GitHub](https://github.com/fable-retold/meadow-integration)
 [Get Started](#meadow-integration)

package/docs/_topbar.md

@@ -6,4 +6,4 @@
 - [API Reference](programmatic-api.md)
 - [Data Clone](data-clone/overview.md)
 - [Examples](examples-walkthrough.md)
-- [GitHub](https://github.com/stevenvelozo/meadow-integration)
+- [GitHub](https://github.com/fable-retold/meadow-integration)

package/docs/_version.json

@@ -1,7 +1,7 @@
 {
 	"Name": "meadow-integration",
-	"Version": "1.0.40",
+	"Version": "1.0.41",
 	"Description": "Meadow Data Integration",
-	"GeneratedAt": "2026-05-21T17:36:27.541Z",
-	"GitCommit": "278227d"
+	"GeneratedAt": "2026-05-26T21:12:33.932Z",
+	"GitCommit": "170b8fe"
 }

package/docs/architecture.md

@@ -6,74 +6,8 @@ This document describes the architectural design of Meadow Integration, covering
 
 Meadow Integration sits between external data sources and the Meadow data access layer. It provides three interfaces (CLI, REST, Programmatic) that share a common set of services.
 
-```mermaid
-flowchart TB
-    subgraph External["External Data Sources"]
-        CSV["CSV Files"]
-        TSV["TSV Files"]
-        JSON["JSON Arrays"]
-        API["Remote Meadow API"]
-    end
-
-    subgraph Interfaces["Interfaces"]
-        CLI["CLI Program\n(pict-service-commandlineutility)"]
-        REST["REST Server\n(Orator + Restify)"]
-        PROG["Programmatic API\n(require meadow-integration)"]
-    end
-
-    subgraph Services["Core Services"]
-        TC["TabularCheck"]
-        TT["TabularTransform"]
-        IA["IntegrationAdapter"]
-        GM["GUIDMap"]
-        CM["ConnectionManager"]
-        CRC["CloneRestClient"]
-        SYNC["Sync"]
-        SEI["SyncEntityInitial"]
-        SEO["SyncEntityOngoing"]
-        OP["Operation"]
-    end
-
-    subgraph Targets["Targets"]
-        MAPI["Meadow REST API\n(write)"]
-        DB["Local Database\n(MySQL / MSSQL)"]
-    end
-
-    CSV --> CLI
-    TSV --> CLI
-    JSON --> CLI
-    CSV --> REST
-    TSV --> REST
-    JSON --> REST
-
-    CLI --> TC
-    CLI --> TT
-    CLI --> IA
-    CLI --> SYNC
-    REST --> TC
-    REST --> TT
-    REST --> IA
-    PROG --> TC
-    PROG --> TT
-    PROG --> IA
-    PROG --> SYNC
-
-    TT --> IA
-    IA --> GM
-    IA --> MAPI
-
-    API --> CRC
-    CRC --> SYNC
-    SYNC --> SEI
-    SYNC --> SEO
-    SEI --> CM
-    SEO --> CM
-    CM --> DB
-
-    SYNC --> OP
-    SEI --> OP
-    SEO --> OP
-```
+<!-- bespoke diagram: edit diagrams/high-level-system-architecture.mmd or .hints.json, then: npx pict-renderer-graph build modules/meadow/meadow-integration/docs -->
+![High-Level System Architecture](diagrams/high-level-system-architecture.svg)
 
 The system divides cleanly into two pipelines that share the `Operation` utility for timing and progress tracking.
 
@@ -81,39 +15,8 @@ The system divides cleanly into two pipelines that share the `Operation` utility
 
 The transformation pipeline converts tabular data into Meadow entity records. Each stage is a discrete service that can be used independently.
 
-```mermaid
-flowchart LR
-    subgraph Input["1. Parse"]
-        FILE["Source File\n(CSV / TSV / JSON)"]
-        PARSE["Stream Parser\n(csv-parser / JSON.parse)"]
-    end
-
-    subgraph Transform["2. Transform"]
-        RECORD["Raw Record"]
-        MAPPING["Mapping Config\n(Implicit + Explicit + User)"]
-        TEMPLATE["Pict Template Engine\n({~D:Record.col~})"]
-        SOLVER["Solvers\n(Expression Parser)"]
-    end
-
-    subgraph Collect["3. Collect"]
-        COMP["Comprehension\n{Entity: {GUID: Record}}"]
-        MERGE["Intersect\n(merge by GUID)"]
-    end
-
-    subgraph Push["4. Push"]
-        ADAPTER["IntegrationAdapter"]
-        MARSHAL["Marshal Record\n(schema validation,\nstring truncation,\nGUID prefixing)"]
-        GUIDMAP["GUIDMap\n(external <-> Meadow IDs)"]
-        UPSERT["Upsert\n(single or bulk)"]
-    end
-
-    FILE --> PARSE --> RECORD
-    MAPPING --> TEMPLATE
-    RECORD --> TEMPLATE --> COMP
-    RECORD --> SOLVER --> COMP
-    COMP --> MERGE --> COMP
-    COMP --> ADAPTER --> MARSHAL --> GUIDMAP --> UPSERT
-```
+<!-- bespoke diagram: edit diagrams/data-transformation-pipeline.mmd or .hints.json, then: npx pict-renderer-graph build modules/meadow/meadow-integration/docs -->
+![Data Transformation Pipeline](diagrams/data-transformation-pipeline.svg)
 
 ### Stage Details
 
@@ -137,50 +40,8 @@ Pict template expressions resolve column values at transformation time. Solvers
 
 The Data Clone pipeline replicates entity data from a remote Meadow API into a local relational database.
 
-```mermaid
-flowchart TB
-    subgraph Config["Configuration"]
-        MCFG[".meadow.config.json"]
-        SCHEMA["Extended Schema JSON\n(from Stricture)"]
-        CLIOPTS["CLI Overrides\n(--api_server, --db_host, etc.)"]
-    end
-
-    subgraph Auth["1. Authenticate"]
-        CRC["CloneRestClient"]
-        SESSION["Session Management\n(cookie / token)"]
-    end
-
-    subgraph Connect["2. Connect"]
-        CM["ConnectionManager"]
-        POOL["Connection Pool\n(MySQL or MSSQL)"]
-    end
-
-    subgraph Init["3. Initialize Schema"]
-        LOAD["Load Extended Schema"]
-        CREATE["Create Tables\n(if not exist)"]
-        INDEX["Create Indexes\n(GUID unique, Deleted)"]
-    end
-
-    subgraph Sync["4. Sync Entities"]
-        COMPARE["Compare\nlocal vs. remote\n(max ID, count, UpdateDate)"]
-        DOWNLOAD["Download Pages\n(filtered + sorted)"]
-        WRITE["Marshal + Write\n(create or update)"]
-        PROGRESS["Progress Tracking\n(Operation service)"]
-    end
-
-    MCFG --> CRC
-    CLIOPTS --> CRC
-    MCFG --> CM
-    CLIOPTS --> CM
-    SCHEMA --> LOAD
-
-    CRC --> SESSION --> DOWNLOAD
-    CM --> POOL --> WRITE
-    LOAD --> CREATE --> INDEX
-
-    COMPARE --> DOWNLOAD --> WRITE
-    WRITE --> PROGRESS
-```
+<!-- bespoke diagram: edit diagrams/data-synchronization-pipeline.mmd or .hints.json, then: npx pict-renderer-graph build modules/meadow/meadow-integration/docs -->
+![Data Synchronization Pipeline](diagrams/data-synchronization-pipeline.svg)
 
 ### Stage Details
 
@@ -317,69 +178,20 @@ classDiagram
 
 Configuration for the CLI flows through multiple layers, each overriding the previous.
 
-```mermaid
-flowchart LR
-    DEF["Default Configuration\n(Default-Meadow-Integration-\nConfiguration.json)"]
-    FILE[".meadow.config.json\n(working directory)"]
-    CLI["Command-Line Flags\n(--api_server, --db_host, etc.)"]
-
-    DEF -->|"base"| MERGED["Resolved Configuration"]
-    FILE -->|"overrides"| MERGED
-    CLI -->|"overrides"| MERGED
-```
+<!-- bespoke diagram: edit diagrams/configuration-cascade.mmd or .hints.json, then: npx pict-renderer-graph build modules/meadow/meadow-integration/docs -->
+![Configuration Cascade](diagrams/configuration-cascade.svg)
 
 For data transformation, the mapping configuration has its own three-layer cascade:
 
-```mermaid
-flowchart LR
-    IMP["Implicit\n(auto-detected from\nfirst record)"]
-    EXP["Explicit\n(mapping file\nvia -m flag)"]
-    USR["User\n(CLI flags:\n-e, -g, -n, -c)"]
-
-    IMP -->|"base"| FINAL["Final Mapping Config"]
-    EXP -->|"overrides"| FINAL
-    USR -->|"overrides"| FINAL
-```
+<!-- bespoke diagram: edit diagrams/configuration-cascade-2.mmd or .hints.json, then: npx pict-renderer-graph build modules/meadow/meadow-integration/docs -->
+![Configuration Cascade](diagrams/configuration-cascade-2.svg)
 
 ## Sync Mode Comparison
 
 The two sync modes serve different purposes and have different performance characteristics.
 
-```mermaid
-flowchart TB
-    subgraph Initial["Initial Sync"]
-        I1["Query local max ID"]
-        I2["Query remote max ID + count"]
-        I3["Generate paginated URL partials\n(filter: ID > local max)"]
-        I4["Download each page"]
-        I5["For each record:\nRead local by ID"]
-        I6{"Record\nexists?"}
-        I7["Skip"]
-        I8["Create with\nidentity insert"]
-
-        I1 --> I2 --> I3 --> I4 --> I5 --> I6
-        I6 -->|"yes"| I7
-        I6 -->|"no"| I8
-    end
-
-    subgraph Ongoing["Ongoing Sync"]
-        O1["Query local max ID + UpdateDate"]
-        O2["Query remote max ID + UpdateDate + count"]
-        O3["Iterate all records\n(paginated, ID ascending)"]
-        O4["For each record:\nRead local by ID"]
-        O5{"Record\nexists?"}
-        O6{"UpdateDate\ndifference\n> 5ms?"}
-        O7["Update record"]
-        O8["Skip"]
-        O9["Create with\nidentity insert"]
-
-        O1 --> O2 --> O3 --> O4 --> O5
-        O5 -->|"yes"| O6
-        O5 -->|"no"| O9
-        O6 -->|"yes"| O7
-        O6 -->|"no"| O8
-    end
-```
+<!-- bespoke diagram: edit diagrams/sync-mode-comparison.mmd or .hints.json, then: npx pict-renderer-graph build modules/meadow/meadow-integration/docs -->
+![Sync Mode Comparison](diagrams/sync-mode-comparison.svg)
 
 | Aspect | Initial | Ongoing |
 |--------|---------|---------|
@@ -394,30 +206,7 @@ flowchart TB
 
 The included Dockerfile builds a production image for running Data Clone as a containerized service. The image is based on `node:20-bookworm` and expects a `.meadow.config.json` to be provided at runtime (via volume mount or baked into a derived image).
 
-```mermaid
-flowchart LR
-    subgraph Build["Docker Build"]
-        BASE["node:20-bookworm"]
-        DEPS["npm install --omit=dev"]
-        SRC["Copy source + scripts"]
-    end
-
-    subgraph Runtime["Docker Runtime"]
-        CFG[".meadow.config.json\n(volume mount)"]
-        SCHEMA["Extended Schema\n(volume mount)"]
-        RUN["scripts/run.sh"]
-    end
-
-    subgraph External["External"]
-        REMOTE["Remote Meadow API"]
-        LOCAL["Local Database"]
-    end
-
-    BASE --> DEPS --> SRC --> RUN
-    CFG --> RUN
-    SCHEMA --> RUN
-    RUN --> REMOTE
-    RUN --> LOCAL
-```
+<!-- bespoke diagram: edit diagrams/docker-deployment.mmd or .hints.json, then: npx pict-renderer-graph build modules/meadow/meadow-integration/docs -->
+![Docker Deployment](diagrams/docker-deployment.svg)
 
 The `docker-compose.yml` can be used to run the Data Clone alongside a local MySQL or MSSQL container for development and testing.