@karmaniverous/entity-manager 6.13.3 → 6.14.1

package/README.md

@@ -1,34 +1,348 @@
-<!-- TYPEDOC_EXCLUDE -->
+# entity-manager
 
-> [API Documentation](https://docs.karmanivero.us/entity-manager/) • [CHANGELOG](https://github.com/karmaniverous/entity-manager/tree/main/CHANGELOG.md)
+[![npm version](https://img.shields.io/npm/v/@karmaniverous/entity-manager.svg)](https://www.npmjs.com/package/@karmaniverous/entity-manager) ![Node Current](https://img.shields.io/node/v/@karmaniverous/entity-manager) <!-- TYPEDOC_EXCLUDE --> [![docs](https://img.shields.io/badge/docs-website-blue)](https://docs.karmanivero.us/entity-manager) [![changelog](https://img.shields.io/badge/changelog-latest-blue.svg)](https://github.com/karmaniverous/entity-manager/tree/main/CHANGELOG.md)<!-- /TYPEDOC_EXCLUDE --> [![license](https://img.shields.io/badge/license-BSD--3--Clause-blue.svg)](https://github.com/karmaniverous/entity-manager/tree/main/LICENSE.md)
 
-<!-- /TYPEDOC_EXCLUDE -->
+EntityManager implements rational indexing & cross-shard querying at scale in your NoSQL database so you can focus on your application logic.
 
-# entity-manager
+If you have any questions, please [start a discussion](https://github.com/karmaniverous/entity-manager/discussions). Otherwise stay tuned!
 
-**EntityManager implements rational indexing & cross-shard querying at scale in your NoSQL database so you can focus on your application logic.**
+## What is this?
 
-> The Typescript refactor is still in flux, but stabilizing! Still fleshing out the [demo](https://github.com/karmaniverous/entity-manager-demo) & [documentation](https://karmanivero.us/projects/entity-manager/intro/).
+Entity Manager is a TypeScript-first library that applies a provider‑agnostic, highly opinionated single‑table design to your NoSQL data. It lets you:
 
-If you have any questions, please [start a discussion](https://github.com/karmaniverous/entity-manager/discussions). Otherwise stay tuned!
+- Define a global hash key and range key, plus additional generated properties used by your indexes.
+- Encode/decode indexable elements via transcodes so strings sort like their original types.
+- Configure a time‑based sharding strategy (shard bumps) that grows as you scale.
+- Query across many shards and indexes in parallel through injected “shard query” functions — results are combined, de‑duplicated, sorted, and returned with a compact, dehydrated page key for the next request.
+
+It is designed to work with stores like DynamoDB but keeps the orchestration provider‑neutral.
+
+Key links:
+
+- API: https://docs.karmanivero.us/entity-manager
+- Requirements: see .stan/system/stan.requirements.md (authoritative for v6.14.0)
+- Example test configuration: see test/config.ts
+
+## Features
+
+- Global model for generated properties, indexes, and property transcodes
+  - generatedProperties.sharded and generatedProperties.unsharded
+  - Global indexes: indexToken → { hashKey, rangeKey, projections? }
+  - Global propertyTranscodes: property → transcodeName
+- Deterministic sharding
+  - Time‑windowed shard bumps: { timestamp, charBits, chars }
+  - Full shard space assignment per bump (uses radix\*\*chars placeholders)
+  - Cross‑bump query enumeration over all applicable shards
+- Page key dehydration/rehydration
+  - Compact string arrays and lz‑string compression for transport
+  - Rehydrate back to pageKey objects for each index+shard
+- Provider‑agnostic parallel query orchestration
+  - Inject shard query functions for each index
+  - Parallel fan‑out with configurable throttle
+  - Combine, dedupe by unique property, and sort results
+- Strong typing + runtime validation
+  - Zod‑validated config parsing
+  - Robust TypeScript surface for config, items, keys, queries
+
+## Install
+
+```bash
+npm install @karmaniverous/entity-manager
+# optional: testing support used in the repo
+npm install --save-dev @karmaniverous/mock-db
+```
+
+TypeScript is strongly recommended. The library will validate configuration at runtime for JavaScript users, but you lose compile‑time guarantees.
+
+## Usage overview
+
+The pattern has three parts:
+
+1. Define your entity types and a config map
+
+- Each entity type lists all properties that exist on your records.
+- Entity‑level behaviors live in the config’s entities block (timestamp property, unique property, shard bumps).
+- Generated properties, indexes, and transcodes are defined globally.
+
+2. Create an EntityManager instance
+
+- Pass your config (validated with Zod).
+- Optionally inject a logger with debug/error methods (defaults to console).
+
+3. Use EntityManager helpers
+
+- addKeys / getPrimaryKey / removeKeys
+- encodeGeneratedProperty / decodeGeneratedProperty
+- getIndexComponents / unwrapIndex / dehydrateIndexItem / rehydrateIndexItem
+- dehydratePageKeyMap / rehydratePageKeyMap
+- query(options) to orchestrate cross‑shard multi‑index queries
+
+## Quick start (TypeScript)
+
+Below is a minimal end‑to‑end example showing shape and intent. It mirrors the current implementation’s global config model.
+
+```ts
+import {
+  defaultTranscodes,
+  type ConfigMap,
+} from '@karmaniverous/entity-manager';
+import { EntityManager } from '@karmaniverous/entity-manager';
+
+// 1) Entity definitions (Typescript types)
+interface User {
+  userId: string; // unique property
+  created: number; // timestamp property
+  updated: number;
+  firstNameCanonical: string;
+  lastNameCanonical: string;
+  // Generated properties exist on stored items but are configured globally:
+  // e.g., firstNameRK (unsharded), lastNameRK (unsharded), userPK (sharded)
+}
+
+type MyConfigMap = ConfigMap<{
+  EntityMap: { user: User };
+  HashKey: 'hashKey'; // defaults are 'hashKey' / 'rangeKey' if omitted
+  RangeKey: 'rangeKey';
+  ShardedKeys: 'userPK'; // token(s) for sharded generated properties
+  UnshardedKeys: 'firstNameRK' | 'lastNameRK';
+  TranscodedProperties:
+    | 'userId'
+    | 'created'
+    | 'updated'
+    | 'firstNameCanonical'
+    | 'lastNameCanonical';
+}>;
+
+// 2) Build a config with the global model
+const now = Date.now();
+const manager = new EntityManager<MyConfigMap>({
+  // Per-entity: unique + timestamp + shard schedule (+ optional defaults)
+  entities: {
+    user: {
+      uniqueProperty: 'userId',
+      timestampProperty: 'created',
+      shardBumps: [
+        // records with timestamp < now → effectively unsharded
+        { timestamp: now, charBits: 1, chars: 0 },
+        // records with timestamp ≥ now → 1 char at radix 4 (2^2) gives 4 shards
+        { timestamp: now, charBits: 2, chars: 1 },
+      ],
+      // optional defaults (used by query if omitted in options)
+      defaultLimit: 10,
+      defaultPageSize: 10,
+    },
+  },
+
+  // Global generated properties (tokens → element lists)
+  generatedProperties: {
+    sharded: {
+      userPK: ['userId'], // atomic (all required)
+    },
+    unsharded: {
+      firstNameRK: ['firstNameCanonical', 'lastNameCanonical', 'created'],
+      lastNameRK: ['lastNameCanonical', 'firstNameCanonical', 'created'],
+    },
+  },
+
+  // Global key tokens
+  hashKey: 'hashKey',
+  rangeKey: 'rangeKey',
+
+  // Global indexes (hashKey, rangeKey must match allowed token sets)
+  indexes: {
+    created: { hashKey: 'hashKey', rangeKey: 'created' },
+    updated: { hashKey: 'hashKey', rangeKey: 'updated' },
+    firstName: { hashKey: 'hashKey', rangeKey: 'firstNameRK' },
+    lastName: { hashKey: 'hashKey', rangeKey: 'lastNameRK' },
+    userCreated: { hashKey: 'userPK', rangeKey: 'created' }, // sharded alt hash
+  },
+
+  // Transcode mapping for scalar/unsharded elements and properties
+  propertyTranscodes: {
+    userId: 'string',
+    created: 'timestamp',
+    updated: 'timestamp',
+    firstNameCanonical: 'string',
+    lastNameCanonical: 'string',
+  },
+
+  // Transcodes (can override/extend defaultTranscodes)
+  transcodes: defaultTranscodes,
+
+  // Delimiters and query throttle (defaults shown)
+  generatedKeyDelimiter: '|',
+  generatedValueDelimiter: '#',
+  shardKeyDelimiter: '!',
+  throttle: 10,
+});
+```
+
+### Generate keys on items
+
+```ts
+// A partial item (no keys yet)
+const user = {
+  userId: 'u123',
+  created: Date.now(),
+  updated: Date.now(),
+  firstNameCanonical: 'lee',
+  lastNameCanonical: 'zhang',
+};
+
+// Add hashKey, rangeKey, and generated properties
+const record = manager.addKeys('user', user); // returns EntityRecord<...>
+
+// Get just the primary key
+const keyOnly = manager.getPrimaryKey('user', user); // { hashKey, rangeKey }
+
+// Remove generated keys from a stored record
+const pruned = manager.removeKeys('user', record);
+```
+
+### Encode/decode generated property strings
+
+```ts
+// Encode an unsharded generated property (always returns a string)
+const fn = manager.encodeGeneratedProperty('firstNameRK', record);
+// e.g. "firstNameCanonical#lee|lastNameCanonical#zhang|created#000001711234567"
+
+// Decode back into an object fragment
+import { decodeGeneratedProperty } from '@karmaniverous/entity-manager';
+const decoded = decodeGeneratedProperty(manager, fn); // { firstNameCanonical: 'lee', ... }
+```
+
+### Query across shards and indexes
+
+Entity Manager relies on injected shard query functions to perform provider‑specific queries on each shard/index page. The library orchestrates:
+
+- page‑key rehydration → parallel shard queries → de‑duplication and sorting → page‑key dehydration.
+
+```ts
+import type {
+  QueryOptions,
+  ShardQueryFunction,
+} from '@karmaniverous/entity-manager';
+
+// Example shard query using a made-up client (see @karmaniverous/mock-db in repo tests)
+const firstNameQuery: ShardQueryFunction<MyConfigMap> = async (
+  hashKey,
+  pageKey,
+  pageSize,
+) => {
+  // Return { count, items, pageKey? } for this shard+index page
+  // pageKey is a partial item object with necessary index components
+  // ... perform provider-specific work here ...
+  return { count: 0, items: [], pageKey };
+};
+
+// Invoke query with shardQueryMap
+const result = await manager.query({
+  entityToken: 'user',
+  item: {}, // often used to supply elements for alternate hash keys
+  shardQueryMap: { firstName: firstNameQuery },
+  limit: 50,
+  pageSize: 10,
+  // optional: pageKeyMap: previousResult.pageKeyMap,
+  // optional: timestampFrom / timestampTo for shard-space windowing
+});
+```
+
+Notes:
+
+- The result includes a compressed pageKeyMap string for the next call.
+- Entity Manager enumerates the hash key space for the time window, rehydrates the prior page keys (if any), and fans out queries across all shard+index pairs in `shardQueryMap` (up to `throttle`).
+- Items are deduplicated by the entity’s unique property and sorted by `sortOrder` (if provided).
+
+## Configuration reference (current model, v6.14.0)
+
+- entities: Record<entityToken, { timestampProperty, uniqueProperty, shardBumps?, defaultLimit?, defaultPageSize? }>
+- generatedProperties:
+  - sharded: Record<ShardedKey, TranscodedProperties[]>
+  - unsharded: Record<UnshardedKey, TranscodedProperties[]>
+- indexes: Record<indexToken, {
+  - hashKey: HashKey | ShardedKey
+  - rangeKey: RangeKey | UnshardedKey | TranscodedProperties
+  - projections?: string[]
+    }>
+- propertyTranscodes: Record<TranscodedProperties, keyof TranscodeMap>
+- transcodes: Record<transcodeName, { encode, decode }> (defaults to defaultTranscodes)
+- hashKey: HashKey (e.g., 'hashKey')
+- rangeKey: RangeKey (e.g., 'rangeKey')
+- generatedKeyDelimiter: string (default '|', must match /\W+/)
+- generatedValueDelimiter: string (default '#', must match /\W+/)
+- shardKeyDelimiter: string (default '!', must match /\W+/)
+- throttle: number (default 10)
+
+Validation highlights:
+
+- Delimiters must not contain each other.
+- Keys and tokens must be mutually exclusive as required.
+- Generated property element lists are non‑empty and have no duplicates.
+- propertyTranscodes values must exist in transcodes.
+- Index hashKey/rangeKey must use valid token sets.
+- shardBumps are sorted, include a zero‑timestamp bump if missing, and chars must increase monotonically with timestamp.
+
+Sharding:
+
+- For assignment: a record always uses all placeholders for its applicable bump; suffix space is (2**charBits) ** chars.
+- For queries: hash key space spans all bumps overlapping [timestampFrom, timestampTo].
+
+## ESM / CJS
+
+```ts
+// ESM
+import {
+  EntityManager,
+  defaultTranscodes,
+} from '@karmaniverous/entity-manager';
+
+// CJS
+const {
+  EntityManager,
+  defaultTranscodes,
+} = require('@karmaniverous/entity-manager');
+```
+
+## Logging
+
+All helpers log debug context and error detail via the injected logger (defaults to `console`). In tests, you may supply a quiet logger:
+
+```ts
+const logger = { debug: () => undefined, error: console.error };
+const manager = new EntityManager(config, logger);
+```
+
+## Delimiter safety
+
+Generated key/value delimiters and the shard key delimiter are used when composing strings:
 
-## Why?
+- generatedKeyDelimiter: '|' (between pairs)
+- generatedValueDelimiter: '#' (between key and value)
+- shardKeyDelimiter: '!' (between entity token and shard suffix)
 
-Traditional relational database systems like MySQL or SQL Server implement indexing & scaling strategies at a platform level based on schemas defined at design time.
+Your scalar property values used in generated properties should not include these delimiters. If they must, set custom delimiters (must match /\W+/ and not contain each other).
 
-NoSQL platforms like DynamoDB offer far better performance at scale, but structured index & shard keys must be defined as data elements and exploited by application logic in data retrieval & cross-shard queries. **They shift the burden of complexity from the database platform to the developer!**
+## Types you’ll use most
 
-EntityManager encapsulates a provider-agnostic, highly opinionated approach to the [single-table design pattern](https://aws.amazon.com/blogs/compute/creating-a-single-table-design-with-amazon-dynamodb/).
+- ConfigMap<M>
+- EntityItem<C>, EntityRecord<C>, EntityKey<C>, EntityToken<C>
+- QueryOptions<C>, QueryResult<C>
+- PageKey<C>, PageKeyMap<C>
+- ShardQueryFunction<C>, ShardQueryMap<C>, ShardBump
 
-With EntityManager, you can:
+See the full API: https://docs.karmanivero.us/entity-manager
 
-- Define related data entities & structured keys wth a simple, declarative configuration format.
+## Scripts (repo)
 
-- Specify a partition sharding strategy that maximizes query performance while permitting planned, staged scaling over time.
+- build: rollup outputs ESM/CJS + .d.ts
+- test: vitest with coverage
+- lint: ESLint (type‑aware) + Prettier integration
+- docs: TypeDoc (links to external type docs for shared utility packages)
+- typecheck: tsc + tsd (type‑level tests)
 
-- Add or remove structured index keys from entity data objects with a single method call.
+## License
 
-- Perform paged, cross-shard, multi-index queries with a single method call.
+BSD‑3‑Clause (see package.json).
 
 ---
 

package/dist/cjs/{EntityManager.js → EntityManager/EntityManager.js}

@@ -1,10 +1,10 @@
 'use strict';
 
 var tslib = require('tslib');
-var radash = require('radash');
 var addKeys = require('./addKeys.js');
 var encodeGeneratedProperty = require('./encodeGeneratedProperty.js');
 var findIndexToken = require('./findIndexToken.js');
+var getPrimaryKey = require('./getPrimaryKey.js');
 var ParsedConfig = require('./ParsedConfig.js');
 var query = require('./query.js');
 var removeKeys = require('./removeKeys.js');
@@ -52,59 +52,33 @@ class EntityManager {
     /**
      * Encode a generated property value. Returns a string or undefined if atomicity requirement of sharded properties not met.
      *
-     * @param property - {@link Config.generatedProperties | Generated property} key.
+     * @param property - {@link Config | Config} `generatedProperties` key.
      * @param item - {@link EntityItem | `EntityItem`} object.
      *
      * @returns Encoded generated property value.
      *
-     * @throws `Error` if `property` is not a {@link Config.generatedProperties | generated property}.
+     * @throws `Error` if `property` is not a {@link Config | Config} `generatedProperties` key.
      */
     encodeGeneratedProperty(property, item) {
         return encodeGeneratedProperty.encodeGeneratedProperty(this, property, item);
     }
-    /**
-     * Update generated properties, hash key, and range key on an {@link EntityItem | `EntityItem`} object.
-     *
-     * @param entityToken - {@link Config | `Config`} `entities` key.
-     * @param item - {@link EntityItem | `EntityItem`} object.
-     * @param overwrite - Overwrite existing properties (default `false`).
-     *
-     * @returns {@link EntityRecord | `EntityRecord`} object with updated properties.
-     *
-     * @throws `Error` if `entityToken` is invalid.
-     */
-    addKeys(entityToken, item, overwrite = false) {
-        return addKeys.addKeys(this, entityToken, item, overwrite);
+    addKeys(entityToken, i, overwrite = false) {
+        if (Array.isArray(i)) {
+            return i.map((item) => addKeys.addKeys(this, entityToken, item, overwrite));
+        }
+        return addKeys.addKeys(this, entityToken, i, overwrite);
     }
-    /**
-     * Convert an {@link EntityItem | `EntityItem`} into an {@link EntityKey | `EntityKey`}.
-     *
-     * @param entityToken - {@link Config | `Config`} `entities` key.
-     * @param item - {@link EntityItem | `EntityItem`} object.
-     * @param overwrite - Overwrite existing properties (default `false`).
-     *
-     * @returns {@link EntityKey | `EntityKey`} extracted from shallow clone of `item` with updated properties.
-     *
-     * @throws `Error` if `entityToken` is invalid.
-     */
-    getPrimaryKey(entityToken, item, overwrite = false) {
-        const { hashKey, rangeKey } = this.config;
-        return radash.pick(!overwrite && item[hashKey] && item[rangeKey]
-            ? item
-            : addKeys.addKeys(this, entityToken, item, overwrite), [this.config.hashKey, this.config.rangeKey]);
+    getPrimaryKey(entityToken, i, overwrite = false) {
+        if (Array.isArray(i)) {
+            return i.map((item) => getPrimaryKey.getPrimaryKey(this, entityToken, item, overwrite));
+        }
+        return getPrimaryKey.getPrimaryKey(this, entityToken, i, overwrite);
     }
-    /**
-     * Strips generated properties, hash key, and range key from an {@link EntityRecord | `EntityRecord`} object.
-     *
-     * @param entityToken - {@link Config | `Config`} `entities` key.
-     * @param item - {@link EntityRecord | `EntityRecord`} object.
-     *
-     * @returns {@link EntityItem | `EntityItem`} with generated properties, hash key & range key removed.
-     *
-     * @throws `Error` if `entityToken` is invalid.
-     */
-    removeKeys(entityToken, item) {
-        return removeKeys.removeKeys(this, entityToken, item);
+    removeKeys(entityToken, i) {
+        if (Array.isArray(i)) {
+            return i.map((item) => removeKeys.removeKeys(this, entityToken, item));
+        }
+        return removeKeys.removeKeys(this, entityToken, i);
     }
     /**
      * Find an index token in a {@link Config | `Config`} object based on the index `hashKey` and `rangeKey`.