retold-facto 0.0.4 → 0.1.1

package/README.md

@@ -0,0 +1,142 @@
+# Retold Facto
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+---
+
+A data warehouse and knowledge graph for the Retold ecosystem. Facto ingests records from arbitrary sources, tracks their provenance and certainty, compiles them into projections via declarative mappings, and deploys those projections to any Meadow-supported backend. It runs as a standalone REST server, a Pict web application, and -- optionally -- as an [Ultravisor](https://github.com/stevenvelozo/ultravisor) beacon exposing its ingest / transform / deploy operations as workflow capabilities.
+
+## Features
+
+- **Records + Certainty** -- Every ingested record carries source provenance, schema version, ingest-job lineage, and a configurable `CertaintyIndex` entry so downstream queries can filter on confidence
+- **Ingest Engine** -- Batch ingests from CSV, JSON, folder scans, or direct API calls; tracks `IngestJob` status, dedupes with content signatures, and auto-increments dataset versions
+- **Projection Engine** -- Compiles raw records into flat, denormalized projections using declarative `Mappings` JSON and five built-in merge strategies (WriteAll, FirstWriteWins, ReliabilityOverwrite, MergeAndReinforce, FieldFillOnly)
+- **Connection Manager** -- First-class support for SQLite, MySQL, PostgreSQL, and MSSQL projection targets via Meadow connectors; masked-password safe API
+- **Mapping DSL** -- `Entity + GUIDTemplate + Mappings` JSON descriptors drive `meadow-integration`'s `TabularTransform` for flattening, comprehension, and de-duplication
+- **Source Catalog** -- Research-grade catalog with `SourceCatalogEntry`, `CatalogDatasetDefinition`, and a folder scanner that discovers datasets from `README.md` files
+- **Multi-Entity Web UI** -- Two Pict browser applications (`pict-app` and `pict-app-full`) provide source, dataset, record, projection, mapping, and connection management
+- **Ultravisor Beacon** -- Optional beacon mode exposes three capabilities (`FactoData`, `FactoTransform`, `FactoDeploy`) so workflows can orchestrate Facto remotely
+- **Meadow Native** -- Schema is defined in a single stricture JSON file; REST endpoints for every entity come for free via `meadow-endpoints`
+- **Orator Native** -- Built on the standard Retold Orator + Restify stack; every subsystem exposes its own REST surface
+
+## Installation
+
+```bash
+npm install retold-facto
+# or globally for the CLI
+npm install -g retold-facto
+```
+
+## Quick Start
+
+```bash
+# Initialize the default SQLite schema
+retold-facto init
+
+# Start the server (default :8386)
+retold-facto serve
+
+# Start on a custom port with a custom database
+retold-facto serve --port 9000 --db /var/data/facto.sqlite
+
+# Scan a folder of README-based dataset definitions
+retold-facto scan ./my-data
+```
+
+Open `http://localhost:8386/` for the web UI, or hit the REST API at `/1.0/*` (auto-generated Meadow CRUD) and `/facto/*` (subsystem endpoints).
+
+## CLI
+
+```bash
+retold-facto <command> [options]
+
+Commands:
+  serve [default]              Start the REST API + Pict web UI
+  init                         Create the default schema (21 tables)
+  ingest <file> [dataset-id] [source-id] [type]
+                               Parse and ingest a CSV/JSON file
+  source list | add            Source CRUD shortcuts
+  dataset list | add           Dataset CRUD shortcuts
+  scan <folder>                Discover datasets from README-annotated folders
+  scan provision <folder>      Provision discovered datasets into Facto
+  scan ingest <folder>         Ingest discovered datasets end-to-end
+
+Options:
+  -c, --config <file>          JSON configuration file
+  -p, --port <port>            API server port (default 8386)
+  -d, --db <path>              SQLite database path (default ./data/facto.sqlite)
+  -s, --scan-path <path>       Add a scan path (repeatable)
+  -l, --log [path]             Write a log file
+```
+
+## Subsystems
+
+Facto is composed of twelve service managers layered over Meadow. Each one owns a subset of the schema and a subset of the REST surface:
+
+| Subsystem | Purpose | Docs |
+|---|---|---|
+| **Recordset** | Records, CertaintyIndex, IngestJob lifecycle | [docs/subsystems/recordset.md](docs/subsystems/recordset.md) |
+| **Projection** | MultiSetProjection, ProjectionStore, merge strategies, deployment | [docs/subsystems/projection.md](docs/subsystems/projection.md) |
+| **Mapping** | `Entity + GUIDTemplate + Mappings` transform descriptors | [docs/subsystems/mapping.md](docs/subsystems/mapping.md) |
+| **Connection** | External database connections for projection targets | [docs/subsystems/connection.md](docs/subsystems/connection.md) |
+| **Audit** | Timestamped CRUD columns, ingest job logs, certainty logs | [docs/subsystems/audit.md](docs/subsystems/audit.md) |
+
+## Ultravisor Integration
+
+Facto can register as an [Ultravisor](https://github.com/stevenvelozo/ultravisor) beacon and expose its operations as workflow capabilities:
+
+- **`FactoData`** -- Source / Dataset / Record / IngestJob / ProjectionStore CRUD
+- **`FactoTransform`** -- Apply a mapping to a batch of records (pure function, no side effects)
+- **`FactoDeploy`** -- Deploy a projection schema to an external store
+
+In a typical Retold deployment, Ultravisor orchestrates pipelines that dispatch these capabilities to one or more Facto beacons running close to their data sources. See [docs/ultravisor-integration.md](docs/ultravisor-integration.md) for the full beacon contract and workflow patterns.
+
+Facto also runs perfectly well without Ultravisor -- beacon mode is optional.
+
+## Documentation
+
+- [Overview](docs/README.md)
+- [Quick Start](docs/quickstart.md)
+- [Architecture](docs/architecture.md)
+- [API Reference](docs/api-reference.md)
+- [Ultravisor Integration](docs/ultravisor-integration.md)
+- **Subsystems**
+  - [Recordset](docs/subsystems/recordset.md)
+  - [Projection](docs/subsystems/projection.md)
+  - [Connection](docs/subsystems/connection.md)
+  - [Mapping](docs/subsystems/mapping.md)
+  - [Audit](docs/subsystems/audit.md)
+
+## Testing
+
+```bash
+npm test                 # Mocha TDD unit tests
+npm run test-browser     # Puppeteer headless browser tests
+```
+
+## Building
+
+```bash
+npm run build
+npm run build-codemirror
+```
+
+## Related Packages
+
+- [meadow](https://github.com/stevenvelozo/meadow) -- ORM / query DSL
+- [meadow-endpoints](https://github.com/stevenvelozo/meadow-endpoints) -- auto-generated REST CRUD
+- [meadow-integration](https://github.com/stevenvelozo/meadow-integration) -- `TabularTransform` and `CertaintyAccumulator` used by the projection engine
+- [orator](https://github.com/stevenvelozo/orator) -- REST server framework
+- [pict](https://github.com/stevenvelozo/pict) -- MVC framework for the web UI
+- [stricture](https://github.com/stevenvelozo/stricture) -- schema definition language (MicroDDL)
+- [ultravisor](https://github.com/stevenvelozo/ultravisor) -- workflow orchestrator (optional beacon target)
+- [ultravisor-beacon](https://github.com/stevenvelozo/ultravisor-beacon) -- beacon protocol client
+- [bibliograph](https://github.com/stevenvelozo/bibliograph) -- (dependency; reserved for richer audit logging)
+
+## License
+
+MIT
+
+## Contributing
+
+Pull requests welcome. See the [Retold Contributing Guide](https://github.com/stevenvelozo/retold/blob/main/docs/contributing.md) for the code of conduct, contribution process, and testing requirements.

package/docs/README.md

@@ -0,0 +1,73 @@
+# Retold Facto
+
+> A data warehouse and knowledge graph for the Retold ecosystem
+
+Retold Facto is the persistence and transform layer that sits between raw data sources and the analytical stores, search indexes, and reporting systems that consume them. It is built as a Fable service that hosts an Orator REST server, a Meadow-backed data access layer, a pair of Pict web applications for browser-based management, and an optional Ultravisor beacon for workflow-driven orchestration.
+
+Facto is the answer to a specific problem: **you have a lot of data from a lot of sources with different schemas, different reliability, different update cadences, and different downstream consumers, and you need a single place to store it, audit its provenance, transform it, and publish it.** The core design decisions follow from that problem:
+
+- **Raw before refined.** Every record lands in the warehouse verbatim, tagged with the source it came from, the ingest job that produced it, and a certainty value. Projections are computed downstream from the raw records, not in place of them.
+- **Mappings as data.** How you flatten raw records into an analytical shape is itself a first-class entity -- a row in the `ProjectionMapping` table with a `MappingConfiguration` JSON blob. You can edit mappings through the web UI, version them, export them, and re-run them.
+- **Everything is Meadow.** The schema is a stricture JSON file. Every entity gets automatic CRUD endpoints, queryable filters, and connector flexibility. Swap SQLite for MySQL for PostgreSQL without touching the business logic.
+- **Workflows, not cron jobs.** Facto can register as an Ultravisor beacon and expose its ingest / transform / deploy operations as workflow capabilities. Ultravisor handles the scheduling, retry, and distribution; Facto handles the work.
+
+The twelve service managers inside Facto each own a small, focused piece of the system:
+
+1. **RetoldFactoSourceManager** -- source CRUD and activation
+2. **RetoldFactoDatasetManager** -- dataset CRUD and type filtering
+3. **RetoldFactoRecordManager** -- record ingest and certainty assignment
+4. **RetoldFactoIngestEngine** -- file-based and batch-based ingest with job tracking
+5. **RetoldFactoProjectionEngine** -- projection compilation, deployment, and querying (the largest subsystem)
+6. **RetoldFactoCatalogManager** -- catalog entries and provisioning helpers
+7. **RetoldFactoStoreConnectionManager** -- external database connections with masked passwords
+8. **RetoldFactoSchemaManager** -- schema versioning and MicroDDL compilation
+9. **RetoldFactoSourceFolderScanner** -- README-driven folder discovery
+10. **RetoldFactoDataLakeService** -- offline data lake primitives
+11. **ThroughputMonitor** -- per-stage performance metrics
+12. **RetoldFactoBeaconProvider** -- Ultravisor beacon registration (optional)
+
+## Features
+
+- **Raw Warehouse** -- 21 tables covering sources, datasets, records, ingest jobs, binary attachments, and certainty metadata
+- **Batch Ingest** -- `IngestEngine` with file scanning, job tracking, content-signature dedupe, and automatic dataset versioning
+- **Projection Compilation** -- `ProjectionEngine` applies `Mappings` JSON to raw records and produces de-duped comprehensions
+- **Merge Strategies** -- Five built-in strategies (`WriteAll`, `FirstWriteWins`, `ReliabilityOverwrite`, `MergeAndReinforce`, `FieldFillOnly`) for combining records from multiple sources
+- **Projection Deployment** -- Flat output materialized to any Meadow-supported backend (SQLite, MySQL, PostgreSQL, MSSQL)
+- **Web UI** -- Two Pict browser applications: a compact `pict-app` and a full-featured `pict-app-full` with per-entity editors and projection flow diagrams
+- **Source Catalog** -- Research-grade catalog entries, folder scanning via README conventions, and automatic provisioning
+- **Optional Beacon** -- Register as an Ultravisor beacon and expose `FactoData`, `FactoTransform`, and `FactoDeploy` capabilities
+- **Path Safety** -- All filesystem operations use sanitized paths relative to the configured data directory
+
+## When to Use It
+
+Reach for Facto when you need:
+
+- A single warehouse for records from heterogeneous sources with provenance tracking
+- A declarative transform pipeline that produces analytical projections without writing ETL code
+- A REST surface and a web UI for managing sources, datasets, records, projections, and mappings
+- An Ultravisor-integratable target for workflow-driven data operations
+- A place to experiment with multi-source merge strategies and certainty-weighted joins
+
+Skip it if your data pipeline is a single-source ETL from one CSV to one database -- you can do that with Meadow directly and skip the projection/certainty/catalog overhead.
+
+## Relationship with Ultravisor
+
+Facto and Ultravisor are designed to work together, but neither requires the other:
+
+- **Facto standalone** -- `retold-facto serve` starts the REST server, loads the schema, and opens the web UI. Ingest runs when you POST to it, projections compile when you invoke the compile endpoint, and deployment happens on demand.
+- **Facto + Ultravisor** -- `retold-facto serve --beacon <url>` additionally registers the beacon with an Ultravisor coordinator. Workflow operations can then dispatch `FactoData`, `FactoTransform`, and `FactoDeploy` capabilities to the beacon instead of calling REST endpoints directly.
+
+The beacon mode is most useful in distributed deployments where several Facto instances run close to their data sources and an Ultravisor coordinator orchestrates the pipeline that flows through them. See [Ultravisor Integration](ultravisor-integration.md) for the full contract.
+
+## Learn More
+
+- [Quick Start](quickstart.md) -- install, init, serve, and run your first ingest
+- [Architecture](architecture.md) -- process layout, class hierarchy, and data-flow diagrams
+- [API Reference](api-reference.md) -- every REST endpoint, grouped by subsystem
+- [Ultravisor Integration](ultravisor-integration.md) -- beacon capabilities and workflow patterns
+- **Subsystem guides:**
+  - [Recordset](subsystems/recordset.md) -- records, certainty, and ingest jobs
+  - [Projection](subsystems/projection.md) -- compilation, merge strategies, deployment
+  - [Mapping](subsystems/mapping.md) -- the transform DSL and how to write one
+  - [Connection](subsystems/connection.md) -- external projection store management
+  - [Audit](subsystems/audit.md) -- what is tracked and how to query it

package/docs/_cover.md

@@ -0,0 +1,18 @@
+# Retold Facto
+
+> A data warehouse and knowledge graph for the Retold ecosystem
+
+Ingest records from anywhere, track their provenance and certainty, compile them into projections via declarative mappings, and deploy those projections to any Meadow-supported backend. Run it standalone, as a Pict web application, or as an Ultravisor beacon exposing ingest / transform / deploy as workflow capabilities.
+
+- **Records + Certainty** -- Every record carries source provenance, ingest-job lineage, and a configurable certainty index
+- **Ingest Engine** -- CSV, JSON, folder-scan, and API ingest with dedupe, versioning, and job tracking
+- **Projection Engine** -- Compile raw records into flat projections via declarative mappings and five merge strategies
+- **Connection Manager** -- SQLite, MySQL, PostgreSQL, and MSSQL projection targets via Meadow connectors
+- **Source Catalog** -- Research-grade catalog with README-based folder discovery
+- **Ultravisor Beacon** -- Optional beacon mode exposes FactoData, FactoTransform, and FactoDeploy as workflow capabilities
+
+[Overview](README.md)
+[Quick Start](quickstart.md)
+[Architecture](architecture.md)
+[Ultravisor Integration](ultravisor-integration.md)
+[GitHub](https://github.com/stevenvelozo/retold-facto)

package/docs/_sidebar.md

@@ -0,0 +1,30 @@
+- Getting Started
+
+  - [Overview](README.md)
+  - [Quick Start](quickstart.md)
+
+- Reference
+
+  - [Architecture](architecture.md)
+  - [API Reference](api-reference.md)
+  - [Ultravisor Integration](ultravisor-integration.md)
+
+- Subsystems
+
+  - [Recordset](subsystems/recordset.md)
+  - [Projection](subsystems/projection.md)
+  - [Mapping](subsystems/mapping.md)
+  - [Connection](subsystems/connection.md)
+  - [Audit](subsystems/audit.md)
+
+- Retold Ecosystem
+
+  - [Meadow](/meadow/meadow/)
+  - [Meadow Endpoints](/meadow/meadow-endpoints/)
+  - [Meadow Integration](/meadow/meadow-integration/)
+  - [Orator](/orator/orator/)
+  - [Pict](/pict/pict/)
+  - [Stricture](/utility/stricture/)
+  - [Ultravisor](/apps/ultravisor/)
+  - [Ultravisor Beacon](/utility/ultravisor-beacon/)
+  - [Fable](/fable/fable/)

package/docs/_topbar.md

@@ -0,0 +1,13 @@
+# Retold Facto
+
+- [Overview](README.md)
+- [Quick Start](quickstart.md)
+- [Architecture](architecture.md)
+- [API Reference](api-reference.md)
+- [Ultravisor](ultravisor-integration.md)
+- [Recordset](subsystems/recordset.md)
+- [Projection](subsystems/projection.md)
+- [Mapping](subsystems/mapping.md)
+- [Connection](subsystems/connection.md)
+- [Audit](subsystems/audit.md)
+- [GitHub](https://github.com/stevenvelozo/retold-facto)

package/docs/_version.json

@@ -0,0 +1,7 @@
+{
+	"Name": "retold-facto",
+	"Version": "0.1.0",
+	"Description": "Data warehouse and knowledge graph storage for the Retold ecosystem.",
+	"GeneratedAt": "2026-04-10T17:27:53.362Z",
+	"GitCommit": "94d77a9"
+}

package/docs/api-reference.md

@@ -0,0 +1,245 @@
+# API Reference
+
+Facto exposes two parallel REST surfaces:
+
+1. **`/1.0/<Entity>`** -- auto-generated Meadow CRUD endpoints for every entity in the schema (created by `meadow-endpoints` from the stricture JSON)
+2. **`/facto/*`** -- subsystem-specific endpoints registered by each service manager
+
+This reference groups endpoints by subsystem. For the Meadow CRUD conventions (filtering, sorting, bulk operations, page size, etc.) see the [meadow-endpoints](https://github.com/stevenvelozo/meadow-endpoints) documentation.
+
+## Meadow CRUD
+
+For every entity in the schema, `meadow-endpoints` generates:
+
+| Method | Path | Purpose |
+|---|---|---|
+| `GET` | `/1.0/<Entity>` | List (supports `FBV~Column~Op~Value` filter expressions, page / pageSize, sort) |
+| `GET` | `/1.0/<Entity>/:ID` | Read a single row |
+| `POST` | `/1.0/<Entity>` | Create a row |
+| `PUT` | `/1.0/<Entity>/:ID` | Update a row |
+| `DELETE` | `/1.0/<Entity>/:ID` | Soft-delete a row |
+| `GET` | `/1.0/<Entity>/Count` | Row count (with optional filter) |
+
+Entities: `Source`, `SourceDocumentation`, `Dataset`, `DatasetSource`, `Record`, `RecordBinary`, `CertaintyIndex`, `IngestJob`, `SourceCatalogEntry`, `CatalogDatasetDefinition`, `MultiSetProjection`, `ProjectionStore`, `ProjectionMapping`, `ProjectionCertaintyLog`, `StoreConnection`, `FactoSchema`, `SchemaDocumentation`, `SchemaVersion`, `ThroughputEvent`.
+
+## Source Manager (`/facto/source/*`)
+
+| Method | Path | Purpose |
+|---|---|---|
+| `GET` | `/facto/source/by-hash/:Hash` | Look up a source by its human-readable hash |
+| `GET` | `/facto/sources/active` | List only active (not soft-deleted, not deactivated) sources |
+| `PUT` | `/facto/source/:IDSource/activate` | Mark a source as active |
+| `PUT` | `/facto/source/:IDSource/deactivate` | Mark a source as inactive |
+
+## Dataset Manager (`/facto/dataset/*`)
+
+| Method | Path | Purpose |
+|---|---|---|
+| `GET` | `/facto/dataset/:IDDataset` | Fetch a dataset with its full metadata |
+| `GET` | `/facto/dataset/by-hash/:Hash` | Look up a dataset by hash |
+| `GET` | `/facto/datasets/by-type/:Type` | List datasets filtered by type (`Raw`, `Compositional`, `Projection`, `Derived`) |
+| `GET` | `/facto/datasets/types` | List the allowed dataset types |
+
+## Record Manager (`/facto/record/*`)
+
+| Method | Path | Purpose |
+|---|---|---|
+| `POST` | `/facto/record/ingest` | Batch-ingest an array of records into a dataset. Body: `{ Records, IDDataset, IDSource, Type, DefaultCertainty }`. Returns `{ Ingested, Errors }`. |
+
+See the [Recordset subsystem](subsystems/recordset.md) for the record shape and certainty semantics.
+
+## Ingest Engine (`/facto/ingest/*`)
+
+| Method | Path | Purpose |
+|---|---|---|
+| `GET` | `/facto/ingest/jobs` | List all ingest jobs, optionally filtered by dataset, source, or status |
+| `GET` | `/facto/ingest/job/:IDIngestJob` | Fetch a job's full metadata and log |
+| `GET` | `/facto/ingest/statuses` | List the allowed ingest statuses |
+| `POST` | `/facto/ingest/file` | Ingest a file uploaded as multipart/form-data or by server-side path |
+| `POST` | `/facto/ingest/job` | Create a new empty ingest job (for subsequent record-level posts) |
+
+## Projection Engine (`/facto/projection*` and `/facto/projections*`)
+
+The projection engine has the largest REST surface because it owns compilation, mapping CRUD, deployment, and querying.
+
+### Discovery and Summary
+
+| Method | Path | Purpose |
+|---|---|---|
+| `GET` | `/facto/projections` | List every dataset of type `Projection` |
+| `GET` | `/facto/projections/summary` | Aggregate counts (projections, mappings, stores) |
+| `GET` | `/facto/datasets/by-type/Projection` | Same as `/facto/projections` |
+
+### Query
+
+| Method | Path | Purpose |
+|---|---|---|
+| `POST` | `/facto/projections/query` | Run a parameterized query against a projection store |
+| `POST` | `/facto/projections/aggregate` | Run an aggregation query |
+| `POST` | `/facto/projections/certainty` | Analyze certainty distribution for a projection |
+| `POST` | `/facto/projections/compare` | Compare two projections side-by-side |
+
+### Schema and Store
+
+| Method | Path | Purpose |
+|---|---|---|
+| `GET` | `/facto/projection/:IDDataset/schema` | Fetch the current schema for a projection |
+| `POST` | `/facto/projection/:IDDataset/save-schema` | Save a schema edit |
+| `GET` | `/facto/projection/:IDDataset/stores` | List projection stores configured for a dataset |
+| `POST` | `/facto/projection/:IDDataset/deploy` | Deploy the projection to an external store. Body: `{ IDStoreConnection, TargetTableName }` |
+
+### Mappings
+
+| Method | Path | Purpose |
+|---|---|---|
+| `GET` | `/facto/projection/:IDDataset/mappings` | List mappings for a dataset |
+| `GET` | `/facto/projection/mapping/:IDProjectionMapping` | Fetch a single mapping |
+| `POST` | `/facto/projection/:IDDataset/mapping` | Create a new mapping |
+| `POST` | `/facto/projection/mapping/:IDProjectionMapping/update` | Update an existing mapping |
+| `POST` | `/facto/projection/:IDDataset/discover-fields` | Auto-discover the projection schema from a sample of records |
+| `POST` | `/facto/projection/:IDDataset/import` | Import a mapping configuration from JSON |
+
+### Comprehensions
+
+| Method | Path | Purpose |
+|---|---|---|
+| `GET` | `/facto/projection/:IDDataset/comprehensions` | List the compiled comprehensions for a dataset |
+| `GET` | `/facto/projection/:IDDataset/comprehension/:Filename` | Fetch a single compiled comprehension |
+
+See the [Projection subsystem](subsystems/projection.md) and [Mapping subsystem](subsystems/mapping.md) for the semantics.
+
+## Catalog Manager (`/facto/catalog/*`)
+
+| Method | Path | Purpose |
+|---|---|---|
+| `GET` | `/facto/catalog/entries` | List catalog entries |
+| `GET` | `/facto/catalog/entry/:ID` | Fetch a single catalog entry |
+| `GET` | `/facto/catalog/export` | Export the full catalog as JSON |
+| `GET` | `/facto/catalog/search` | Search catalog entries |
+| `GET` | `/facto/catalog/source-links` | List source relationships for a catalog entry |
+| `POST` | `/facto/catalog/entry` | Create a catalog entry |
+| `POST` | `/facto/catalog/entry/:ID/update` | Update an existing catalog entry |
+| `POST` | `/facto/catalog/dataset/:IDEntry` | Provision the dataset described by a catalog entry |
+| `POST` | `/facto/catalog/import` | Import a catalog from JSON |
+
+## Store Connection Manager (`/facto/connection*`)
+
+| Method | Path | Purpose |
+|---|---|---|
+| `GET` | `/facto/connections` | List all store connections (passwords masked) |
+| `GET` | `/facto/connection/:IDStoreConnection` | Fetch a single connection |
+| `GET` | `/facto/connection/available-types` | List the supported connector types |
+| `POST` | `/facto/connection` | Create a new connection |
+| `POST` | `/facto/connection/test` | Test connectivity against an existing or prospective connection |
+| `POST` | `/facto/connection/:IDStoreConnection` | Update a connection (password preserved if client sends `***`) |
+| `DELETE` | `/facto/connection/:IDStoreConnection` | Soft-delete a connection |
+
+See the [Connection subsystem](subsystems/connection.md) for the full configuration shape and security model.
+
+## Source Folder Scanner (`/facto/scanner/*`)
+
+| Method | Path | Purpose |
+|---|---|---|
+| `GET` | `/facto/scanner/paths` | List configured scan paths |
+| `GET` | `/facto/scanner/datasets` | List datasets discovered by scanning |
+| `GET` | `/facto/scanner/dataset/:Name` | Fetch a specific discovered dataset |
+| `POST` | `/facto/scanner/dataset/:Name/provision` | Provision a discovered dataset (create Source and Dataset rows) |
+| `POST` | `/facto/scanner/dataset/:Name/ingest` | Provision and run the full ingest pipeline for a discovered dataset |
+
+## Schema Manager (`/facto/schema*`)
+
+| Method | Path | Purpose |
+|---|---|---|
+| `GET` | `/facto/schemas/active` | List active schemas |
+| `GET` | `/facto/schema/:IDSchema` | Fetch a schema |
+| `GET` | `/facto/schema/:IDSchema/documentation` | Fetch schema documentation |
+| `POST` | `/facto/schema` | Create a schema |
+| `POST` | `/facto/schema/:IDSchema/update` | Update a schema |
+| `POST` | `/facto/schema/:IDSchema/documentation` | Attach documentation to a schema |
+| `POST` | `/facto/schema/compile` | Compile MicroDDL text into a Meadow schema |
+| `POST` | `/facto/schema/preview-sql` | Preview the DDL that would be produced for a given Meadow schema |
+
+## Application Class (`RetoldFacto`)
+
+`RetoldFacto` extends `fable-serviceproviderbase` and exposes the following developer-facing methods:
+
+### Model Management
+
+| Method | Purpose |
+|---|---|
+| `loadModel(pModelName, pModelObject, pStorageProvider, fCallback)` | Load a schema model from a JavaScript object |
+| `loadModelFromFile(pModelName, pModelPath, pModelFilename, fCallback)` | Load a schema model from a stricture JSON file |
+| `rebuildFullModel()` | Rebuild the merged schema after a model load |
+
+### Schema
+
+| Method | Purpose |
+|---|---|
+| `createSchema(fCallback)` | Execute `FACTO_SCHEMA_SQL` against the current database |
+| `isEndpointGroupEnabled(pGroupName)` | Returns `true` if a named endpoint group is enabled in `options.Endpoints` |
+
+### Lifecycle
+
+| Method | Purpose |
+|---|---|
+| `onBeforeInitialize(fCallback)` | Lifecycle hook, inherited from Fable |
+| `onInitialize(fCallback)` | Lifecycle hook, inherited from Fable |
+| `onAfterInitialize(fCallback)` | Lifecycle hook, inherited from Fable |
+| `initializeService(fCallback)` | Initialize Meadow, Orator, and all subsystem managers |
+| `initializePersistenceEngine(fCallback)` | Initialize Meadow connectors and load the schema |
+| `startService(fCallback)` | Start the Orator HTTP server |
+| `stopService(fCallback)` | Stop the Orator HTTP server and disconnect the beacon (if registered) |
+
+### Utility
+
+| Method | Purpose |
+|---|---|
+| `generateHash(pInput)` | Generate a stable slug from a string |
+
+## Subsystem Manager Methods
+
+Most subsystem managers expose only `connectRoutes(pOratorServiceServer)` as their public API (the routes themselves are the interface). The three that expose rich internal methods worth calling from code:
+
+### `RetoldFactoRecordManager`
+
+- `ingestSingleRecord(pRecordData, fCallback)` -- create a single `Record` with its `CertaintyIndex` entry
+
+### `RetoldFactoIngestEngine`
+
+- `ingestFile(pFilePath, pIDDataset, pIDSource, pOptions, fCallback)`
+- `ingestJSON(pData, pIDDataset, pIDSource, pOptions, fCallback)`
+- `ingestCSV(pCSVContent, pIDDataset, pIDSource, pOptions, fCallback)`
+- `getNextDatasetVersion(pIDDataset, fCallback)`
+- `computeContentSignature(pContent)` -- SHA-256 of ingested content used for dedupe
+- `checkDuplicateSignature(pIDDataset, pSignature, fCallback)`
+- `appendJobLog(pIDIngestJob, pMessage, fCallback)`
+
+### `RetoldFactoProjectionEngine`
+
+- `compileProjection(pIDDataset, pConfig, fCallback)`
+- `deploySchema(pIDDataset, pIDStoreConnection, pTargetTableName, fCallback)`
+- `discoverProjectionFields(pIDDataset, fCallback)`
+- `importProjectionMapping(pIDDataset, pMappingJSON, fCallback)`
+
+### `RetoldFactoBeaconProvider`
+
+- `connectBeacon(pBeaconConfig, fCallback)` -- register with an Ultravisor coordinator
+- `disconnectBeacon(fCallback)` -- unregister cleanly
+
+See [Ultravisor Integration](ultravisor-integration.md) for the beacon config shape and the capabilities registered.
+
+## Response Conventions
+
+Every Facto subsystem endpoint follows the same response shape:
+
+```json
+{ "Success": true, "Data": { /* ... */ } }
+```
+
+Or on failure:
+
+```json
+{ "Success": false, "Error": "Human-readable message" }
+```
+
+Meadow CRUD endpoints use their own (longer-standing) conventions -- see the meadow-endpoints docs for the full reference.