meadow 2.0.22 → 2.0.26

package/docs/providers/rocksdb.md

@@ -0,0 +1,297 @@
+# RocksDB Provider
+
+> Embedded key-value store for high-throughput scenarios with in-memory filtering
+
+The RocksDB provider connects Meadow to [RocksDB](https://rocksdb.org/), a high-performance embedded key-value store. Unlike SQL-based providers, RocksDB does not use a FoxHound dialect for query generation. Instead, records are stored as key-value pairs and all filtering is performed in-memory after a full scan. Direct key lookups are O(1), making this provider ideal for high-throughput scenarios where reads are primarily by key.
+
+## Setup
+
+### Install Dependencies
+
+```bash
+npm install meadow meadow-connection-rocksdb
+```
+
+### Register the Connection
+
+```javascript
+const libFable = require('fable').new(
+	{
+		RocksDB:
+		{
+			RocksDBFolder: './data/rocksdb',
+			KeyMode: 'GUID'
+		}
+	});
+
+const libMeadowConnectionRocksDB = require('meadow-connection-rocksdb');
+
+// Register the connection service
+libFable.serviceManager.addServiceType('MeadowRocksDBProvider', libMeadowConnectionRocksDB);
+libFable.serviceManager.instantiateServiceProvider('MeadowRocksDBProvider');
+```
+
+### Create a Meadow DAL
+
+```javascript
+const libMeadow = require('meadow');
+
+const meadow = libMeadow.new(libFable, 'Book')
+	.setProvider('RocksDB')
+	.setDefaultIdentifier('IDBook')
+	.setSchema([
+		{ Column: 'IDBook', Type: 'AutoIdentity' },
+		{ Column: 'GUIDBook', Type: 'AutoGUID' },
+		{ Column: 'Title', Type: 'String', Size: '255' },
+		{ Column: 'Author', Type: 'String', Size: '128' },
+		{ Column: 'CreateDate', Type: 'CreateDate' },
+		{ Column: 'CreatingIDUser', Type: 'CreateIDUser' },
+		{ Column: 'UpdateDate', Type: 'UpdateDate' },
+		{ Column: 'UpdatingIDUser', Type: 'UpdateIDUser' },
+		{ Column: 'DeleteDate', Type: 'DeleteDate' },
+		{ Column: 'DeletingIDUser', Type: 'DeleteIDUser' },
+		{ Column: 'Deleted', Type: 'Deleted' }
+	]);
+```
+
+## Connection Management
+
+### Connection Configuration
+
+| Setting | Type | Default | Description |
+|---------|------|---------|-------------|
+| `RocksDB.RocksDBFolder` | string | -- | Filesystem path to the RocksDB data directory |
+| `RocksDB.KeyMode` | string | `'GUID'` | Key generation strategy: `'GUID'` or `'ID'` |
+
+### Key Modes
+
+The `KeyMode` setting determines how records are keyed in the store:
+
+| Mode | Key Format | Description |
+|------|------------|-------------|
+| `GUID` | `M-E-{Scope}-{GUID}` | Keys use the record's GUID (default) |
+| `ID` | `M-EBI-{Scope}-{ID}` | Keys use the record's integer identity |
+
+For example, with a `Book` scope:
+
+```
+GUID mode:  M-E-Book-a1b2c3d4-e5f6-7890-abcd-ef1234567890
+ID mode:    M-EBI-Book-42
+```
+
+## Storage Model
+
+### Key-Value Architecture
+
+RocksDB is a key-value store, not a relational database. Each Meadow record is serialized as a JSON value stored under a structured key. This has important implications:
+
+- **No SQL dialect** -- The provider does not use FoxHound for query generation
+- **No JOINs** -- Cross-entity queries are not supported
+- **No aggregations** -- Beyond COUNT, no aggregate functions are available
+- **In-memory filtering** -- All filter operations require scanning records and evaluating filters in memory
+
+### Direct Key Lookup
+
+When reading a record by its primary key (GUID or ID depending on KeyMode), the provider performs a direct key lookup against RocksDB. This is an O(1) operation and is extremely fast regardless of the total number of records in the store.
+
+### In-Memory Filter Engine
+
+For queries with filters, sorts, or pagination, the provider:
+
+1. Scans all records in the scope prefix range
+2. Deserializes each record from JSON
+3. Evaluates filter conditions against each record in memory
+4. Applies sorting in memory
+5. Applies pagination (begin/cap) in memory
+6. Returns the matching result set
+
+This means that filtered reads scale linearly with the total number of records in the scope. For large datasets with complex filter requirements, consider a SQL-based provider instead.
+
+## CRUD Operations
+
+### Create
+
+Serializes the record as JSON and stores it under the generated key.
+
+```javascript
+meadow.doCreate(
+	meadow.query.addRecord({ Title: 'Dune', Author: 'Frank Herbert' }),
+	(pError, pCreateQuery, pReadQuery, pRecord) =>
+	{
+		console.log('New ID:', pRecord.IDBook);
+		console.log('Stored with key: M-E-Book-' + pRecord.GUIDBook);
+	});
+```
+
+### Read
+
+Performs a direct key lookup when filtering by the primary key, or an in-memory scan for other filters.
+
+```javascript
+// Direct key lookup (O(1))
+meadow.doRead(
+	meadow.query.addFilter('IDBook', 42),
+	(pError, pQuery, pRecord) =>
+	{
+		console.log('Title:', pRecord.Title);
+	});
+
+// In-memory filtered scan
+meadow.doReads(
+	meadow.query.setCap(25).setBegin(0).addFilter('Author', 'Frank Herbert', '='),
+	(pError, pQuery, pRecords) =>
+	{
+		pRecords.forEach((pBook) => console.log(pBook.Title));
+	});
+```
+
+### Update
+
+Reads the existing record by key, merges the updated fields, and writes the updated record back.
+
+```javascript
+meadow.doUpdate(
+	meadow.query
+		.addFilter('IDBook', 42)
+		.addRecord({ Title: 'Updated Title' }),
+	(pError, pUpdateQuery, pReadQuery, pRecord) =>
+	{
+		console.log('Updated:', pRecord.Title);
+	});
+```
+
+### Delete
+
+Executes a soft delete by setting the Deleted flag on the stored record.
+
+```javascript
+meadow.doDelete(
+	meadow.query.addFilter('IDBook', 42),
+	(pError, pQuery, pResult) =>
+	{
+		console.log('Deleted records:', pResult);
+	});
+```
+
+### Undelete
+
+Reverses a soft delete by clearing the Deleted flag.
+
+```javascript
+meadow.doUndelete(
+	meadow.query.addFilter('IDBook', 42),
+	(pError, pQuery, pResult) =>
+	{
+		console.log('Restored records:', pResult);
+	});
+```
+
+### Count
+
+Scans the scope prefix and counts matching records in memory.
+
+```javascript
+meadow.doCount(
+	meadow.query,
+	(pError, pQuery, pCount) =>
+	{
+		console.log('Total books:', pCount);
+	});
+```
+
+## Limitations
+
+| Feature | Supported | Notes |
+|---------|-----------|-------|
+| Direct key lookup | Yes | O(1) performance |
+| Filtered reads | Yes | In-memory post-scan filtering |
+| Sorting | Yes | In-memory after scan |
+| Pagination (begin/cap) | Yes | In-memory after scan |
+| COUNT | Yes | In-memory count after scan |
+| JOINs | No | Not supported in key-value model |
+| Aggregations (SUM, AVG) | No | Only COUNT is available |
+| FoxHound dialect | No | Provider uses its own filter engine |
+
+## When to Use RocksDB
+
+| Use Case | Recommendation |
+|----------|---------------|
+| High-throughput key lookups | Excellent -- O(1) reads by key |
+| Embedded applications | Excellent -- no server process required |
+| Write-heavy workloads | Good -- RocksDB is optimized for writes |
+| Complex filtered queries | Consider SQL providers instead |
+| JOIN-heavy data models | Use MySQL, MSSQL, or PostgreSQL |
+| Browser applications | Use ALASQL provider instead |
+
+## Full Setup Example
+
+A complete working example using RocksDB for local key-value storage:
+
+```javascript
+const libFable = require('fable').new(
+	{
+		RocksDB:
+		{
+			RocksDBFolder: './data/bookstore',
+			KeyMode: 'GUID'
+		}
+	});
+
+const libMeadow = require('meadow');
+const libMeadowConnectionRocksDB = require('meadow-connection-rocksdb');
+
+// Register the connection service
+libFable.serviceManager.addServiceType('MeadowRocksDBProvider', libMeadowConnectionRocksDB);
+libFable.serviceManager.instantiateServiceProvider('MeadowRocksDBProvider');
+
+// Create the DAL
+const meadow = libMeadow.new(libFable, 'Book')
+	.setProvider('RocksDB')
+	.setDefaultIdentifier('IDBook')
+	.setSchema([
+		{ Column: 'IDBook', Type: 'AutoIdentity' },
+		{ Column: 'GUIDBook', Type: 'AutoGUID' },
+		{ Column: 'Title', Type: 'String', Size: '255' },
+		{ Column: 'Author', Type: 'String', Size: '128' },
+		{ Column: 'CreateDate', Type: 'CreateDate' },
+		{ Column: 'CreatingIDUser', Type: 'CreateIDUser' },
+		{ Column: 'UpdateDate', Type: 'UpdateDate' },
+		{ Column: 'UpdatingIDUser', Type: 'UpdateIDUser' },
+		{ Column: 'DeleteDate', Type: 'DeleteDate' },
+		{ Column: 'DeletingIDUser', Type: 'DeleteIDUser' },
+		{ Column: 'Deleted', Type: 'Deleted' }
+	]);
+
+// Insert a record
+meadow.doCreate(
+	meadow.query.addRecord({ Title: 'Dune', Author: 'Frank Herbert' }),
+	(pError, pCreateQuery, pReadQuery, pRecord) =>
+	{
+		console.log('Created book with ID:', pRecord.IDBook);
+		console.log('GUID:', pRecord.GUIDBook);
+		console.log('Key: M-E-Book-' + pRecord.GUIDBook);
+
+		// Direct key lookup
+		meadow.doRead(
+			meadow.query.addFilter('IDBook', pRecord.IDBook),
+			(pError, pQuery, pRecord) =>
+			{
+				console.log('Read back:', pRecord.Title, 'by', pRecord.Author);
+			});
+	});
+```
+
+## Error Handling
+
+All operations follow the same error handling pattern:
+
+- Storage errors are stored in `pQuery.parameters.result.error`
+- Key generation or serialization failures are logged as warnings
+- File system errors (e.g., missing RocksDBFolder) bubble through the callback
+
+## Related Documentation
+
+- [Providers Overview](providers/README.md) -- Comparison of all providers
+- [SQLite Provider](providers/sqlite.md) -- Embedded SQL alternative
+- [ALASQL Provider](providers/alasql.md) -- In-memory SQL alternative
+- [meadow-connection-rocksdb](https://github.com/stevenvelozo/meadow-connection-rocksdb) -- Connection module source

package/docs/query-dsl.md

@@ -0,0 +1,269 @@
+# Query DSL
+
+Meadow uses [FoxHound](https://github.com/stevenvelozo/foxhound) as its query DSL. FoxHound provides a fluent, chainable API for building database queries without writing raw SQL. Meadow wraps FoxHound and manages query lifecycle, cloning, and schema injection.
+
+## Getting a Query
+
+Access a new query through the `meadow.query` property:
+
+```javascript
+var tmpQuery = tmpBookDAL.query;
+```
+
+Every access to `.query` returns a fresh, independent clone of the internal FoxHound instance. The clone comes pre-configured with the Meadow entity scope and schema.
+
+## Query Cloning and Isolation
+
+Each call to `meadow.query` produces an isolated query object. Configuring one query never affects another:
+
+```javascript
+var tmpQueryA = tmpBookDAL.query
+	.addFilter('Author', 'Asimov')
+	.setCap(10);
+
+var tmpQueryB = tmpBookDAL.query
+	.addFilter('Author', 'Herbert')
+	.setCap(50);
+
+// tmpQueryA and tmpQueryB are completely independent.
+// tmpQueryA filters for Asimov with cap 10.
+// tmpQueryB filters for Herbert with cap 50.
+```
+
+This isolation is critical for concurrent operations. You can safely build multiple queries in parallel without state leakage.
+
+## Filtering
+
+Add filters to constrain which records are returned. The `addFilter` method accepts a column name, a value, and an optional operator.
+
+### Basic Syntax
+
+```javascript
+// Equality (default operator)
+tmpQuery.addFilter('Author', 'Frank Herbert');
+
+// With explicit operator
+tmpQuery.addFilter('YearPublished', 1960, '>=');
+```
+
+### Supported Operators
+
+| Operator | Description | Example |
+|----------|-------------|---------|
+| `=` | Equal to (default) | `addFilter('Status', 'Active')` |
+| `!=` | Not equal to | `addFilter('Status', 'Deleted', '!=')` |
+| `>` | Greater than | `addFilter('Price', 10, '>')` |
+| `<` | Less than | `addFilter('Price', 100, '<')` |
+| `>=` | Greater than or equal | `addFilter('YearPublished', 2000, '>=')` |
+| `<=` | Less than or equal | `addFilter('YearPublished', 2020, '<=')` |
+| `LIKE` | Pattern match | `addFilter('Title', '%Dune%', 'LIKE')` |
+| `IN` | Value in set | `addFilter('IDBook', '1,2,3', 'IN')` |
+
+### Combining Filters
+
+Multiple filters are combined with `AND` logic:
+
+```javascript
+var tmpQuery = tmpBookDAL.query
+	.addFilter('Author', 'Frank Herbert')
+	.addFilter('YearPublished', 1965, '>=')
+	.addFilter('YearPublished', 1985, '<=');
+```
+
+This generates a query equivalent to:
+
+```sql
+WHERE Author = 'Frank Herbert'
+  AND YearPublished >= 1965
+  AND YearPublished <= 1985
+  AND Deleted = 0
+```
+
+Note that `Deleted = 0` is added automatically when the schema contains a `Deleted` column. See [Soft Deletes](soft-deletes.md) for details.
+
+## Pagination
+
+Control result set size and offset with `setCap` and `setBegin`.
+
+### setCap(n)
+
+Limits the number of records returned:
+
+```javascript
+// Return at most 25 records
+var tmpQuery = tmpBookDAL.query
+	.setCap(25);
+```
+
+### setBegin(n)
+
+Sets the starting offset for pagination:
+
+```javascript
+// Skip the first 50 records, return the next 25
+var tmpQuery = tmpBookDAL.query
+	.setCap(25)
+	.setBegin(50);
+```
+
+### Pagination Example
+
+```javascript
+var tmpPageSize = 20;
+var tmpPageNumber = 3;
+
+var tmpQuery = tmpBookDAL.query
+	.setCap(tmpPageSize)
+	.setBegin(tmpPageSize * tmpPageNumber);
+
+tmpBookDAL.doReads(tmpQuery,
+	function (pError, pQuery, pRecords)
+	{
+		console.log('Page', tmpPageNumber, ':', pRecords.length, 'records');
+	});
+```
+
+## Sorting
+
+Add sort directives with `addSort`. Each sort directive is an object with `Column` and `Direction` properties.
+
+```javascript
+var tmpQuery = tmpBookDAL.query
+	.addSort({ Column: 'Author', Direction: 'ASC' })
+	.addSort({ Column: 'Title', Direction: 'DESC' });
+```
+
+Sort directives are applied in the order they are added.
+
+## Column Selection
+
+Restrict which columns are returned with `setDataElements`. This is useful for performance when you only need specific fields.
+
+```javascript
+var tmpQuery = tmpBookDAL.query
+	.setDataElements(['IDBook', 'Title', 'Author']);
+
+tmpBookDAL.doReads(tmpQuery,
+	function (pError, pQuery, pRecords)
+	{
+		// Each record only contains IDBook, Title, and Author
+	});
+```
+
+## Distinct
+
+Return only unique rows with `setDistinct`:
+
+```javascript
+var tmpQuery = tmpBookDAL.query
+	.setDataElements(['Author'])
+	.setDistinct(true);
+
+tmpBookDAL.doReads(tmpQuery,
+	function (pError, pQuery, pRecords)
+	{
+		// Returns unique authors only
+	});
+```
+
+## Record Attachment
+
+Attach a record to the query for create and update operations using `addRecord`:
+
+```javascript
+// For doCreate
+var tmpQuery = tmpBookDAL.query
+	.addRecord(
+		{
+			Title: 'Neuromancer',
+			Author: 'William Gibson',
+			YearPublished: 1984
+		});
+
+tmpBookDAL.doCreate(tmpQuery,
+	function (pError, pCreateQuery, pReadQuery, pRecord)
+	{
+		console.log('Created:', pRecord.IDBook);
+	});
+
+// For doUpdate (must include the default identifier)
+var tmpQuery = tmpBookDAL.query
+	.addRecord(
+		{
+			IDBook: 5,
+			Title: 'Neuromancer (Anniversary Edition)'
+		});
+
+tmpBookDAL.doUpdate(tmpQuery,
+	function (pError, pUpdateQuery, pReadQuery, pRecord)
+	{
+		console.log('Updated:', pRecord.Title);
+	});
+```
+
+For updates, the record must include the default identifier column (e.g. `IDBook`). Meadow automatically adds a filter on this column to ensure only the intended record is updated.
+
+## Delete Tracking Control
+
+By default, queries exclude records where `Deleted = 1`. To include deleted records in your results, disable delete tracking on the query:
+
+```javascript
+var tmpQuery = tmpBookDAL.query
+	.setDisableDeleteTracking(true);
+
+tmpBookDAL.doReads(tmpQuery,
+	function (pError, pQuery, pRecords)
+	{
+		// pRecords includes both active and soft-deleted records
+	});
+```
+
+## User Identity
+
+The user identity attached to a query is used for auto-stamping `CreateIDUser`, `UpdateIDUser`, and `DeleteIDUser` columns.
+
+### Setting User Identity on the DAL
+
+The preferred approach is to set the user ID on the Meadow instance, which applies to all queries:
+
+```javascript
+tmpBookDAL.setIDUser(42);
+```
+
+### Setting User Identity Per Query
+
+Override the user identity on an individual query:
+
+```javascript
+var tmpQuery = tmpBookDAL.query;
+tmpQuery.query.IDUser = 99;
+```
+
+The per-query value takes precedence over the DAL-level value.
+
+### Disabling Auto-Stamps
+
+To prevent automatic date and user stamps on a query, set the disable flags:
+
+```javascript
+var tmpQuery = tmpBookDAL.query
+	.addRecord(
+		{
+			IDBook: 5,
+			Title: 'Manual Update'
+		});
+
+// Prevent automatic UpdateDate stamping
+tmpQuery.query.disableAutoDateStamp = true;
+
+// Prevent automatic UpdatingIDUser stamping
+tmpQuery.query.disableAutoUserStamp = true;
+
+tmpBookDAL.doUpdate(tmpQuery,
+	function (pError, pUpdateQuery, pReadQuery, pRecord)
+	{
+		// UpdateDate and UpdatingIDUser were not changed
+	});
+```
+
+See [Audit Tracking](audit-tracking.md) for more details on how auto-stamping works.