retold 1.0.6 → 4.0.1

package/docs/meadow.md

@@ -0,0 +1,211 @@
+# Meadow — Data Access Layer
+
+Meadow provides provider-agnostic data access for Retold applications. Define your entities once, and Meadow handles CRUD operations, query generation, schema management, and audit tracking across MySQL, MSSQL, SQLite, or in-browser ALASQL.
+
+## How the Pieces Fit
+
+```mermaid
+graph LR
+  app["Application Code"] --> meadow["Meadow<br/>(ORM)"]
+  meadow --> foxhound["FoxHound<br/>(Query DSL)"]
+  foxhound --> conn["Connection<br/>(mysql/mssql/sqlite)"]
+  conn --> db[("Database")]
+  meadow --> stricture["Stricture<br/>(Schema)"]
+  stricture --> output["JSON Schema<br/>DDL<br/>Documentation"]
+
+  style app fill:#e8f5e9,stroke:#43a047,color:#333
+  style meadow fill:#fff3e0,stroke:#ffa726,color:#333
+  style foxhound fill:#fff3e0,stroke:#ffcc80,color:#333
+  style conn fill:#fff3e0,stroke:#ffcc80,color:#333
+  style stricture fill:#fff3e0,stroke:#ffcc80,color:#333
+  style db fill:#ffebee,stroke:#ef5350,color:#333
+  style output fill:#f5f5f5,stroke:#bdbdbd,color:#666
+```
+
+**Meadow** is the data broker — it exposes CRUD methods, handles audit columns, and delegates query building and execution to FoxHound and the connection provider.
+
+**FoxHound** generates SQL from a fluent API. Write one query definition; get correct SQL for any supported dialect.
+
+**Stricture** defines schemas in a simple MicroDDL format and generates JSON, SQL DDL, documentation, and Meadow schema files from a single source of truth.
+
+## Core Modules
+
+### [Meadow](/meadow/meadow/)
+
+The central data broker. Each Meadow instance represents one data entity (table).
+
+```javascript
+const libFable = require('fable');
+const libMeadow = require('meadow');
+
+let _Fable = new libFable({
+    Product: 'BookStore',
+    MySQL: { Server: 'localhost', User: 'root', Database: 'bookstore' }
+});
+
+let _BookMeadow = _Fable.instantiateServiceProvider('Meadow',
+{
+    Scope: 'Book',
+    DefaultSchema: require('./Book-Schema.json')
+});
+```
+
+**Built-in operations:**
+
+| Operation | Method | Description |
+|-----------|--------|-------------|
+| Create | `.doCreate(record, callback)` | Insert a new record |
+| Read | `.doRead(id, callback)` | Get one record by ID |
+| Reads | `.doReads(query, callback)` | Get multiple records with filtering |
+| Update | `.doUpdate(record, callback)` | Update an existing record |
+| Delete | `.doDelete(id, callback)` | Soft-delete a record |
+| Undelete | `.doUndelete(id, callback)` | Restore a soft-deleted record |
+| Count | `.doCount(query, callback)` | Count matching records |
+
+**Automatic features:** audit columns (CreatingIDUser, UpdatingIDUser, CreateDate, UpdateDate), soft deletes (Deleted flag + DeleteDate), GUID uniqueness (GUIDBook column), and data marshalling.
+
+**npm:** `meadow` · **Version:** 2.0.x
+
+---
+
+### [FoxHound](/meadow/foxhound/)
+
+A fluent query DSL that generates dialect-specific SQL.
+
+```javascript
+let tmpQuery = _Fable.instantiateServiceProvider('FoxHound')
+    .setScope('Book')
+    .addFilter('Author', 'Tolkien')
+    .addFilter('YearPublished', '1954', '>=')
+    .setSort('Title')
+    .setBegin(0).setCap(10)
+    .buildReadQuery();
+
+// MySQL:  SELECT * FROM Book WHERE Author = 'Tolkien'
+//         AND YearPublished >= '1954' ORDER BY Title LIMIT 0,10
+// MSSQL:  SELECT TOP 10 * FROM Book WHERE Author = 'Tolkien' ...
+// SQLite: SELECT * FROM Book WHERE Author = 'Tolkien' ... LIMIT 10 OFFSET 0
+```
+
+**Supported dialects:** MySQL, MSSQL, SQLite, ALASQL (in-browser)
+
+**Operations:** Create, Read, Update, Delete, Count — each generates the correct SQL for the configured dialect with parameterized queries to prevent injection.
+
+**npm:** `foxhound` · **Version:** 2.0.x
+
+---
+
+### [Stricture](/meadow/stricture/)
+
+A MicroDDL for rapid data model definition.
+
+```
+# Book entity
+Book
+	IDBook           | int    | identity | primary
+	GUIDBook         | guid   | unique
+	Title            | string(200)
+	Author           | string(200)
+	YearPublished    | int
+	ISBN             | string(20)
+```
+
+From this definition, Stricture generates:
+
+- **JSON schema** — For Meadow entity configuration
+- **MySQL CREATE TABLE** — Database DDL with indexes
+- **Meadow schema** — Column definitions for the data broker
+- **Documentation** — Human-readable data model docs
+
+**npm:** `stricture` · **Version:** 1.0.x
+
+---
+
+### [Meadow-Endpoints](/meadow/meadow-endpoints/)
+
+Auto-generates a full REST API from a Meadow entity. Given a "Book" entity, you get:
+
+| Route | Method | Operation |
+|-------|--------|-----------|
+| `GET /Books` | Reads | List with filtering and pagination |
+| `GET /Books/Count` | Count | Record count |
+| `GET /Book/:id` | Read | Single record by ID |
+| `GET /Book/Schema` | Schema | Entity schema definition |
+| `POST /Book` | Create | Insert new record |
+| `PUT /Book` | Update | Modify existing record |
+| `DEL /Book/:id` | Delete | Soft-delete record |
+| `DEL /Book/:id/Undelete` | Undelete | Restore record |
+
+**Behavior injection** lets you add hooks at any lifecycle point:
+
+```javascript
+_Endpoints.BehaviorModifications.setBehavior('Read-Authorize',
+    function(pRequest, fCallback)
+    {
+        // Check permissions before allowing the read
+        if (!pRequest.UserSession.authenticated)
+        {
+            pRequest.CommonServices.log.warn('Unauthorized access attempt');
+            return fCallback('Unauthorized');
+        }
+        return fCallback();
+    });
+```
+
+**npm:** `meadow-endpoints` · **Version:** 4.0.x
+
+---
+
+### [Retold-Data-Service](/meadow/retold-data-service/)
+
+An all-in-one Fable service that assembles a Stricture schema into a complete REST API — Meadow entity, endpoints, and connection — in a single service instantiation.
+
+```javascript
+let _BookService = _Fable.instantiateServiceProvider('RetoldDataService',
+{
+    Scope: 'Book',
+    Schema: BookSchema
+});
+
+// That's it — full CRUD REST API is ready to be wired to Orator
+```
+
+**npm:** `retold-data-service` · **Version:** 2.0.x
+
+## Connection Modules
+
+| Module | Database | npm |
+|--------|----------|-----|
+| [meadow-connection-mysql](/meadow/meadow-connection-mysql/) | MySQL / MariaDB | `meadow-connection-mysql` |
+| [meadow-connection-mssql](/meadow/meadow-connection-mssql/) | Microsoft SQL Server | `meadow-connection-mssql` |
+| [meadow-connection-sqlite](/meadow/meadow-connection-sqlite/) | SQLite (via better-sqlite3) | `meadow-connection-sqlite` |
+
+Each connection module provides a pooled database connection as a Fable service. Swap between databases by changing which connection module you install — no application code changes needed.
+
+## Supporting Modules
+
+| Module | Purpose | npm |
+|--------|---------|-----|
+| [bibliograph](/meadow/bibliograph/) | Key-value record comprehension for change tracking in ingestion pipelines | `bibliograph` |
+| [parime](/meadow/parime/) | Generic data lake behaviors and services | `parime` |
+| [retold-harness](/meadow/retold-harness/) | Pre-built API harness with a bookstore demo (8 entities, 10,000+ records) | `retold-harness` |
+| [meadow-integration](/meadow/meadow-integration/) | Data integration tools for CSV import and schema mapping | `meadow-integration` |
+| [meadow-graph-client](/meadow/meadow-graph-client/) | Client for pulling related record sets from relational database graphs | `meadow-graph-client` |
+
+## All Meadow Modules
+
+| Module | Description |
+|--------|-------------|
+| [meadow](/meadow/meadow/) | Provider-agnostic data broker and ORM |
+| [foxhound](/meadow/foxhound/) | Fluent query DSL with multi-dialect SQL generation |
+| [stricture](/meadow/stricture/) | MicroDDL schema definition and code generation |
+| [meadow-endpoints](/meadow/meadow-endpoints/) | Automatic RESTful CRUD endpoint generation |
+| [meadow-connection-mysql](/meadow/meadow-connection-mysql/) | MySQL connection provider |
+| [meadow-connection-mssql](/meadow/meadow-connection-mssql/) | MSSQL connection provider |
+| [meadow-connection-sqlite](/meadow/meadow-connection-sqlite/) | SQLite connection provider |
+| [retold-data-service](/meadow/retold-data-service/) | All-in-one schema-to-REST-API service |
+| [retold-harness](/meadow/retold-harness/) | Demo API harness with bookstore dataset |
+| [bibliograph](/meadow/bibliograph/) | Record-level change tracking |
+| [parime](/meadow/parime/) | Data lake behaviors |
+| [meadow-integration](/meadow/meadow-integration/) | Data integration and CSV import |
+| [meadow-graph-client](/meadow/meadow-graph-client/) | Relational graph record fetching |

package/docs/modules.md

@@ -0,0 +1,96 @@
+# All Modules
+
+An exhaustive list of every repository in the Retold suite, organized by group. Each module is its own git repository hosted at `github.com/stevenvelozo/<module-name>`.
+
+## Fable — Core Ecosystem (6 modules)
+
+| Module | npm | Description |
+|--------|-----|-------------|
+| [fable](/fable/fable/) | `fable` | Service dependency injection, configuration, and logging library — the foundation of every Retold application |
+| [fable-serviceproviderbase](/fable/fable-serviceproviderbase/) | `fable-serviceproviderbase` | Base classes for Fable services providing registration, DI, and lifecycle |
+| [fable-settings](/fable/fable-settings/) | `fable-settings` | Tolerant configuration chain for loading and merging application settings |
+| [fable-log](/fable/fable-log/) | `fable-log` | Flexible logging wrapper with six levels and extensible output streams |
+| [fable-uuid](/fable/fable-uuid/) | `fable-uuid` | UUID generator supporting RFC 4122 v4 and configurable random strings |
+| [fable-log-logger-bunyan](/fable/fable-log-logger-bunyan/) | `fable-log-logger-bunyan` | Bunyan structured logging provider for Fable-Log |
+
+## Meadow — Data Access Layer (13 modules)
+
+| Module | npm | Description |
+|--------|-----|-------------|
+| [meadow](/meadow/meadow/) | `meadow` | Provider-agnostic data broker with CRUD operations, audit tracking, and soft deletes |
+| [foxhound](/meadow/foxhound/) | `foxhound` | Fluent query DSL generating dialect-specific SQL for MySQL, MSSQL, SQLite, and ALASQL |
+| [stricture](/meadow/stricture/) | `stricture` | MicroDDL schema definition language generating JSON, SQL DDL, and documentation |
+| [meadow-endpoints](/meadow/meadow-endpoints/) | `meadow-endpoints` | Automatic RESTful CRUD endpoint generation from Meadow entities with behavior injection |
+| [meadow-connection-mysql](/meadow/meadow-connection-mysql/) | `meadow-connection-mysql` | MySQL/MariaDB pooled connection provider for Meadow |
+| [meadow-connection-mssql](/meadow/meadow-connection-mssql/) | `meadow-connection-mssql` | Microsoft SQL Server connection provider for Meadow |
+| [meadow-connection-sqlite](/meadow/meadow-connection-sqlite/) | `meadow-connection-sqlite` | SQLite connection provider for Meadow via better-sqlite3 |
+| [retold-data-service](/meadow/retold-data-service/) | `retold-data-service` | All-in-one Fable service assembling schema → entity → endpoints → REST API |
+| [retold-harness](/meadow/retold-harness/) | `retold-harness` | Pre-built API harness with a bookstore demo (8 entities, 10,000+ records) |
+| [bibliograph](/meadow/bibliograph/) | `bibliograph` | Key-value record comprehension for change tracking in data ingestion pipelines |
+| [parime](/meadow/parime/) | `parime` | Generic data lake behaviors and services |
+| [meadow-integration](/meadow/meadow-integration/) | `meadow-integration` | Data integration tools for CSV import, schema mapping, and centralized formats |
+| [meadow-graph-client](/meadow/meadow-graph-client/) | `meadow-graph-client` | Client for pulling related record sets from relational database graphs |
+
+## Orator — API Server (6 modules)
+
+| Module | npm | Description |
+|--------|-----|-------------|
+| [orator](/orator/orator/) | `orator` | Unopinionated HTTP server abstraction with lifecycle hooks and route management |
+| [orator-serviceserver-restify](/orator/orator-serviceserver-restify/) | `orator-serviceserver-restify` | Production HTTP server implementation powered by Restify |
+| [orator-static-server](/orator/orator-static-server/) | `orator-static-server` | Static file serving with MIME detection, default files, and subdomain routing |
+| [orator-http-proxy](/orator/orator-http-proxy/) | `orator-http-proxy` | HTTP reverse proxy for forwarding requests to backend services |
+| [tidings](/orator/tidings/) | `tidings` | Extensible reporting system for generating HTML, PDF, and other format reports |
+| [orator-endpoint](/orator/orator-endpoint/) | `orator-endpoint` | Pluggable API endpoint base class for reusable route handlers |
+
+## Pict — MVC Tools (15 modules)
+
+| Module | npm | Description |
+|--------|-----|-------------|
+| [pict](/pict/pict/) | `pict` | Non-opinionated MVC module with template expression engine for text-based UIs |
+| [pict-view](/pict/pict-view/) | `pict-view` | View base class with full lifecycle (init, render, solve, marshal), renderables, and CSS |
+| [pict-template](/pict/pict-template/) | `pict-template` | Template handler base class for custom expression types |
+| [pict-provider](/pict/pict-provider/) | `pict-provider` | Data provider base class for delivering data to views |
+| [pict-application](/pict/pict-application/) | `pict-application` | Application base class coordinating views, state, and lifecycle |
+| [pict-section-form](/pict/pict-section-form/) | `pict-section-form` | Configuration-driven dynamic forms with 13+ input types and data marshaling |
+| [pict-section-recordset](/pict/pict-section-recordset/) | `pict-section-recordset` | CRUD record management views from Meadow endpoint schemas |
+| [pict-section-tuigrid](/pict/pict-section-tuigrid/) | `pict-section-tuigrid` | Toast UI Grid integration for tabular data display and editing |
+| [pict-section-content](/pict/pict-section-content/) | `pict-section-content` | Content display sections for static and dynamic content blocks |
+| [pict-panel](/pict/pict-panel/) | `pict-panel` | Hot-loadable control panel component for browser applications |
+| [pict-router](/pict/pict-router/) | `pict-router` | Hash-based URL routing via Navigo with template string route functions |
+| [informary](/pict/informary/) | `informary` | Dependency-free browser form marshaling with undo/redo and field-level deltas |
+| [cryptbrau](/pict/cryptbrau/) | `cryptbrau` | Simple in-browser symmetric encryption |
+| [pict-serviceproviderbase](/pict/pict-serviceproviderbase/) | `pict-serviceproviderbase` | Base classes for Pict services with pre-initialization support |
+| [pict-service-commandlineutility](/pict/pict-service-commandlineutility/) | `pict-service-commandlineutility` | CLI utility module built on Commander for Pict-based command-line tools |
+
+## Utility — Build & Documentation Tools (5 modules)
+
+| Module | npm | Description |
+|--------|-----|-------------|
+| [indoctrinate](/utility/indoctrinate/) | `indoctrinate` | Documentation scaffolding with content cataloging, label-based filtering, and multi-format output |
+| [manyfest](/utility/manyfest/) | `manyfest` | JSON manifest for consistent data description, validation, and address-based access across layers |
+| [quackage](/utility/quackage/) | `quackage` | Standardized build tool for browser bundles, transpilation, testing, and packaging |
+| [choreographic](/utility/choreographic/) | `choreographic` | Scaffolding for single-run data processing scripts with organized output |
+| [ultravisor](/utility/ultravisor/) | `ultravisor` | Process supervision with scheduled tasks, distributed nodes, and LLM integration |
+
+## Summary
+
+| Group | Count | Focus |
+|-------|-------|-------|
+| Fable | 6 | Core ecosystem, DI, configuration, logging |
+| Meadow | 13 | Data access, ORM, query DSL, schema, connectors |
+| Orator | 6 | API server, HTTP, static files, proxy, reporting |
+| Pict | 15 | MVC, views, templates, forms, grids, routing |
+| Utility | 5 | Build tools, manifests, docs, process supervision |
+| **Total** | **45** | |
+
+## GitHub Repositories
+
+All modules are hosted at `github.com/stevenvelozo/<module-name>`:
+
+- [github.com/stevenvelozo/retold](https://github.com/stevenvelozo/retold) — This meta-repository
+- [github.com/stevenvelozo/fable](https://github.com/stevenvelozo/fable) — Core ecosystem
+- [github.com/stevenvelozo/meadow](https://github.com/stevenvelozo/meadow) — Data access
+- [github.com/stevenvelozo/orator](https://github.com/stevenvelozo/orator) — API server
+- [github.com/stevenvelozo/pict](https://github.com/stevenvelozo/pict) — MVC tools
+
+Each module follows the same structure: `package.json`, `source/`, `test/` (Mocha TDD), and optionally `docs/` (Docsify).

package/docs/orator.md

@@ -0,0 +1,164 @@
+# Orator — API Server
+
+Orator provides an unopinionated HTTP server abstraction for Retold applications. It wraps service server implementations (Restify for production, IPC for testing) behind a consistent interface, so your application code does not depend on any specific HTTP library.
+
+## How It Works
+
+```mermaid
+graph TB
+  app["Your Application<br/>(routes, middleware, etc.)"]
+  app --> core["Orator Core<br/>Lifecycle, content types,<br/>route registration"]
+  core --> restify["Restify Server<br/>(Production)"]
+  core --> ipc["IPC Server<br/>(Testing)"]
+  core --> other["(Other servers)"]
+
+  style app fill:#e8f5e9,stroke:#43a047,color:#333
+  style core fill:#e3f2fd,stroke:#42a5f5,color:#333
+  style restify fill:#fff,stroke:#90caf9,color:#333
+  style ipc fill:#fff,stroke:#90caf9,color:#333
+  style other fill:#f5f5f5,stroke:#bdbdbd,color:#666
+```
+
+Your application registers routes, middleware, and lifecycle hooks with Orator. Orator delegates to whichever service server implementation is configured — Restify for real HTTP, IPC for in-process testing.
+
+## Core Modules
+
+### [Orator](/orator/orator/)
+
+The main server abstraction. Manages lifecycle, route registration, and service server delegation.
+
+```javascript
+const libFable = require('fable');
+const libOrator = require('orator');
+
+let _Fable = new libFable({
+    Product: 'MyAPI',
+    APIServerPort: 8080
+});
+
+let _Orator = _Fable.instantiateServiceProvider('Orator');
+
+// Register routes
+_Orator.addRoute('GET', '/health', function(pRequest, pResponse, fNext)
+{
+    pResponse.send({ status: 'ok' });
+    return fNext();
+});
+
+// Start the server
+_Orator.startService(function(pError)
+{
+    _Fable.log.info('Server listening on port 8080');
+});
+```
+
+**Lifecycle hooks** let you inject behavior at server startup, before/after route handling, and at shutdown:
+
+```javascript
+_Orator.addPreStartFunction(function(fCallback) { /* ... */ return fCallback(); });
+_Orator.addPreRouteFunction(function(pRequest, pResponse, fNext) { /* ... */ return fNext(); });
+```
+
+**npm:** `orator` · **Version:** 5.0.x
+
+---
+
+### [Orator-ServiceServer-Restify](/orator/orator-serviceserver-restify/)
+
+Production HTTP server powered by Restify. Provides full HTTP support, body parsing, CORS, middleware, and configurable listen options.
+
+This is the server implementation used for real deployments. Install it alongside Orator and it registers itself automatically.
+
+```javascript
+const libOrator = require('orator');
+require('orator-serviceserver-restify');
+
+// Restify server is now the active service server
+```
+
+**npm:** `orator-serviceserver-restify` · **Version:** 2.0.x
+
+---
+
+### [Orator-Static-Server](/orator/orator-static-server/)
+
+Static file serving for Orator with MIME type detection, default file support, subdomain-based folder routing, and configurable route prefix stripping.
+
+```javascript
+_Orator.addStaticRoute('/app', '/var/www/app/dist');
+```
+
+**npm:** `orator-static-server` · **Version:** 1.0.x
+
+---
+
+### [Orator-HTTP-Proxy](/orator/orator-http-proxy/)
+
+Reverse proxy for forwarding requests to backend services. Supports multiple HTTP verbs and handles HTTP-to-HTTPS translation.
+
+```javascript
+_Orator.addProxyRoute('/api/external', 'https://backend.example.com/api');
+```
+
+**npm:** `orator-http-proxy` · **Version:** 1.0.x
+
+---
+
+### [Tidings](/orator/tidings/)
+
+An extensible reporting system for generating HTML, PDF, and other format reports. Provides asset collection, templating, and rasterization in a micro-service-friendly pattern.
+
+**npm:** `tidings` · **Version:** 1.0.x
+
+---
+
+### [Orator-Endpoint](/orator/orator-endpoint/)
+
+A base class for creating reusable, pluggable API endpoint handlers that can be shared across Orator applications.
+
+**npm:** `orator-endpoint` · **Version:** 1.0.x
+
+## Typical Setup
+
+A standard API server combines Orator with Meadow-Endpoints:
+
+```javascript
+const libFable = require('fable');
+const libOrator = require('orator');
+const libMeadowEndpoints = require('meadow-endpoints');
+require('orator-serviceserver-restify');
+
+let _Fable = new libFable({
+    Product: 'BookAPI',
+    APIServerPort: 8086,
+    MySQL: { Server: 'localhost', User: 'root', Database: 'bookstore' }
+});
+
+let _Orator = _Fable.instantiateServiceProvider('Orator');
+
+// Create a data entity and wire its endpoints to Orator
+let _BookEntity = _Fable.instantiateServiceProvider('Meadow',
+    { Scope: 'Book', DefaultSchema: BookSchema });
+let _BookEndpoints = _Fable.instantiateServiceProvider('MeadowEndpoints',
+    { Entity: _BookEntity });
+_BookEndpoints.connectRoutes(_Orator);
+
+// Start serving
+_Orator.startService((pError) =>
+{
+    _Fable.log.info('BookAPI running on :8086');
+});
+```
+
+This gives you a full REST API with Create, Read, Reads, Update, Delete, Undelete, Count, and Schema endpoints for the Book entity — all auto-generated.
+
+## All Orator Modules
+
+| Module | Description |
+|--------|-------------|
+| [orator](/orator/orator/) | HTTP server abstraction with lifecycle hooks |
+| [orator-serviceserver-restify](/orator/orator-serviceserver-restify/) | Production HTTP server via Restify |
+| [orator-static-server](/orator/orator-static-server/) | Static file serving with MIME detection |
+| [orator-http-proxy](/orator/orator-http-proxy/) | HTTP reverse proxy pass-through |
+| [tidings](/orator/tidings/) | Reporting scaffolding for HTML/PDF output |
+| [orator-endpoint](/orator/orator-endpoint/) | Reusable endpoint base class |