@memberjunction/version-history 0.0.1 → 4.1.0

package/README.md

@@ -1,45 +1,270 @@
 # @memberjunction/version-history
 
-## ⚠️ IMPORTANT NOTICE ⚠️
+Version labeling, snapshot capture, diff, and restore for MemberJunction records. Provides point-in-time versioning with full entity dependency graph awareness.
 
-**This package is created solely for the purpose of setting up OIDC (OpenID Connect) trusted publishing with npm.**
+## Overview
 
-This is **NOT** a functional package and contains **NO** code or functionality beyond the OIDC setup configuration.
+The `@memberjunction/version-history` package enables developers to create named version labels that capture record state at specific points in time, compare changes between labels, and restore records to previous states while respecting entity dependency ordering.
 
-## Purpose
+```mermaid
+graph TD
+    A["VersionHistoryEngine<br/>(Facade)"] --> B["LabelManager"]
+    A --> C["SnapshotBuilder"]
+    A --> D["DiffEngine"]
+    A --> E["RestoreEngine"]
+    A --> F["DependencyGraphWalker"]
 
-This package exists to:
-1. Configure OIDC trusted publishing for the package name `@memberjunction/version-history`
-2. Enable secure, token-less publishing from CI/CD workflows
-3. Establish provenance for packages published under this name
+    B --> G["Version Labels"]
+    C --> H["Version Label Items"]
+    D --> I["DiffResult"]
+    E --> J["RestoreResult"]
+    F --> K["DependencyNode Tree"]
 
-## What is OIDC Trusted Publishing?
+    style A fill:#2d6a9f,stroke:#1a4971,color:#fff
+    style B fill:#7c5295,stroke:#563a6b,color:#fff
+    style C fill:#7c5295,stroke:#563a6b,color:#fff
+    style D fill:#7c5295,stroke:#563a6b,color:#fff
+    style E fill:#7c5295,stroke:#563a6b,color:#fff
+    style F fill:#7c5295,stroke:#563a6b,color:#fff
+    style G fill:#2d8659,stroke:#1a5c3a,color:#fff
+    style H fill:#2d8659,stroke:#1a5c3a,color:#fff
+    style I fill:#b8762f,stroke:#8a5722,color:#fff
+    style J fill:#b8762f,stroke:#8a5722,color:#fff
+    style K fill:#b8762f,stroke:#8a5722,color:#fff
+```
 
-OIDC trusted publishing allows package maintainers to publish packages directly from their CI/CD workflows without needing to manage npm access tokens. Instead, it uses OpenID Connect to establish trust between the CI/CD provider (like GitHub Actions) and npm.
+## Installation
 
-## Setup Instructions
+```bash
+npm install @memberjunction/version-history
+```
 
-To properly configure OIDC trusted publishing for this package:
+## Quick Start
 
-1. Go to [npmjs.com](https://www.npmjs.com/) and navigate to your package settings
-2. Configure the trusted publisher (e.g., GitHub Actions)
-3. Specify the repository and workflow that should be allowed to publish
-4. Use the configured workflow to publish your actual package
+```typescript
+import { VersionHistoryEngine } from '@memberjunction/version-history';
 
-## DO NOT USE THIS PACKAGE
+const engine = new VersionHistoryEngine();
 
-This package is a placeholder for OIDC configuration only. It:
-- Contains no executable code
-- Provides no functionality
-- Should not be installed as a dependency
-- Exists only for administrative purposes
+// Create a label capturing a record and its dependencies
+const { Label, CaptureResult } = await engine.CreateLabel({
+  Name: 'Before Refactor',
+  Scope: 'Record',
+  EntityName: 'AI Prompts',
+  RecordKey: promptKey,
+  IncludeDependencies: true,
+}, contextUser);
 
-## More Information
+// Later: see what changed since the label
+const diff = await engine.DiffLabelToCurrentState(Label.ID, contextUser);
 
-For more details about npm's trusted publishing feature, see:
-- [npm Trusted Publishing Documentation](https://docs.npmjs.com/generating-provenance-statements)
-- [GitHub Actions OIDC Documentation](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect)
+// Restore if needed
+const result = await engine.RestoreToLabel(Label.ID, {}, contextUser);
+```
 
----
+## Label Scopes
 
-**Maintained for OIDC setup purposes only**
+| Scope | Description | Use Case |
+|-------|-------------|----------|
+| `Record` | Single record and its dependencies | Safe point before editing a specific record |
+| `Entity` | All records of a specific entity | Checkpoint before bulk updates |
+| `System` | All tracked entities | Full system snapshot before a release |
+
+## Architecture
+
+### Sub-Engines
+
+| Sub-Engine | Responsibility |
+|-----------|----------------|
+| **LabelManager** | Label CRUD and lifecycle management |
+| **SnapshotBuilder** | Captures record state into label items with batched queries |
+| **DependencyGraphWalker** | Traverses entity relationships to discover dependent records |
+| **DiffEngine** | Compares snapshots between labels or between a label and current state |
+| **RestoreEngine** | Applies labeled state back to records in dependency order |
+
+### Snapshot and Restore Flow
+
+```mermaid
+sequenceDiagram
+    participant User
+    participant VHE as VersionHistoryEngine
+    participant SB as SnapshotBuilder
+    participant DGW as DependencyGraphWalker
+    participant DB as Database
+
+    User->>VHE: CreateLabel(params)
+    VHE->>SB: CaptureRecord(labelId, entity, key)
+    SB->>DGW: WalkDependents(entity, key)
+    DGW->>DB: Query related records
+    DGW-->>SB: DependencyNode tree
+    SB->>DB: Save VersionLabelItem for each record
+    SB-->>VHE: CaptureResult
+
+    Note over User,DB: Time passes, records are modified
+
+    User->>VHE: RestoreToLabel(labelId, options)
+    VHE->>VHE: Create safety Pre-Restore label
+    VHE->>DB: Load snapshots, apply in dependency order
+    VHE-->>User: RestoreResult
+```
+
+## Dependency Graph Walker
+
+The walker discovers all records that should be included in a version label. It traverses both **reverse relationships** (child records that belong to the root) and **forward references** (records the root or its children point to).
+
+### Two Key Mechanisms
+
+#### 1. EntityRelationship-Driven Reverse Walking
+
+Instead of scanning every entity for foreign keys that point to the current entity, the walker uses **EntityRelationship metadata** that MemberJunction already maintains. Only explicitly registered children are walked.
+
+```mermaid
+graph TD
+    A["AI Agents"] -->|"EntityRelationship"| B["AI Agent Prompts"]
+    A -->|"EntityRelationship"| C["AI Agent Actions"]
+    A -->|"EntityRelationship"| D["AI Agent Relationships"]
+    A -->|"EntityRelationship"| E["AI Agent Models"]
+    A -.-x|"NOT walked"| G["Random Table with FK"]
+
+    style A fill:#2d6a9f,stroke:#1a4971,color:#fff
+    style B fill:#2d8659,stroke:#1a5c3a,color:#fff
+    style C fill:#2d8659,stroke:#1a5c3a,color:#fff
+    style D fill:#2d8659,stroke:#1a5c3a,color:#fff
+    style E fill:#2d8659,stroke:#1a5c3a,color:#fff
+    style G fill:#b8762f,stroke:#8a5722,color:#fff
+```
+
+#### 2. Ancestor Stack -- Prevents Backtracking
+
+The walker maintains a stack of entity type names representing the path from root to the current node. When evaluating any relationship, if the target entity type is already on the ancestor stack, it is skipped. This surgically prevents graph explosion without arbitrary depth limits.
+
+### Walk Algorithm
+
+```mermaid
+flowchart TD
+    Start(["walkChildren called"]) --> DepthCheck{"Depth >= MaxDepth?"}
+    DepthCheck -->|Yes| Stop(["Return"])
+    DepthCheck -->|No| Reverse["Walk Reverse Relationships"]
+
+    Reverse --> ReverseLoop{"For each EntityRelationship"}
+    ReverseLoop -->|Next| AncestorR{"On ancestor stack?"}
+    AncestorR -->|Yes| SkipR(["Skip"])
+    AncestorR -->|No| LoadChildren["Load child records"]
+    LoadChildren --> RecurseR["Recurse walkChildren"]
+    RecurseR --> ReverseLoop
+
+    ReverseLoop -->|Done| Forward["Walk Forward References"]
+    Forward --> ForwardLoop{"For each FK field"}
+    ForwardLoop -->|Next| SystemCheck{"System FK?"}
+    SystemCheck -->|Yes| SkipF(["Skip"])
+    SystemCheck -->|No| AncestorF{"On ancestor stack?"}
+    AncestorF -->|Yes| SkipF2(["Skip"])
+    AncestorF -->|No| LoadTarget["Load referenced record"]
+    LoadTarget --> RecurseF["Recurse walkChildren"]
+    RecurseF --> ForwardLoop
+    ForwardLoop -->|Done| Stop
+
+    style SkipR fill:#b8762f,stroke:#8a5722,color:#fff
+    style SkipF fill:#b8762f,stroke:#8a5722,color:#fff
+    style SkipF2 fill:#b8762f,stroke:#8a5722,color:#fff
+    style Start fill:#2d6a9f,stroke:#1a4971,color:#fff
+    style Stop fill:#2d8659,stroke:#1a5c3a,color:#fff
+```
+
+### Design Rationale
+
+| Concern | Solution |
+|---------|----------|
+| Which children to walk? | **EntityRelationship** -- admin-controlled, CodeGen-maintained |
+| Preventing graph explosion? | **Ancestor stack** -- blocks backtracking to any entity type on current path |
+| Infrastructure FKs (UserID, etc.)? | **System FK skip list** -- regex patterns for known infrastructure fields |
+| Cycle detection? | **Visited set** -- `entityName::recordID` prevents revisiting any record |
+| Sub-agent recursion? | Ancestor stack is **path-based** -- pops on backtrack, allowing re-entry from a different branch |
+
+### Forward FK Skip Patterns
+
+The following FK field name patterns are never followed during forward walking, as they reference system infrastructure rather than business data:
+
+- `CreatedByUserID`, `UpdatedByUserID`, `UserID`
+- `ContextUserID`, `ModifiedByUserID`
+- `CreatedBy`, `UpdatedBy`
+- `OwnerID`, `OwnerUserID`
+- `AssignedToID`, `AssignedToUserID`
+- `EntityID` (polymorphic reference)
+
+## Snapshot Builder -- Batched Capture
+
+When capturing records into a version label, the SnapshotBuilder uses **batched queries** to minimize database round trips:
+
+```mermaid
+sequenceDiagram
+    participant SB as SnapshotBuilder
+    participant DB as Database
+
+    Note over SB: Receive flat list of N nodes
+
+    SB->>SB: Group nodes by EntityID
+    loop For each entity group
+        SB->>DB: ONE RunView: RecordChanges<br/>WHERE EntityID = X<br/>AND RecordID IN (a, b, c, ...)
+        DB-->>SB: Latest changes for all records in group
+    end
+
+    Note over SB: Build lookup map
+
+    loop For each node without a RecordChange
+        SB->>DB: Create synthetic snapshot
+    end
+
+    loop For each node
+        SB->>DB: Create VersionLabelItem
+    end
+```
+
+**Before batching**: N individual RunView calls.
+**After batching**: ~5-10 RunView calls (one per unique entity type in the graph).
+
+## API Reference
+
+### VersionHistoryEngine
+
+```typescript
+const engine = new VersionHistoryEngine();
+
+// Create a label with dependency walking
+const { Label, CaptureResult } = await engine.CreateLabel({
+    Name: 'Before Refactor v2',
+    Scope: 'Record',
+    EntityName: 'AI Agents',
+    RecordKey: agentKey,
+    IncludeDependencies: true,
+    MaxDepth: 10,
+    ExcludeEntities: ['AI Agent Runs'],
+}, contextUser);
+
+// Diff against current state
+const diff = await engine.DiffLabelToCurrentState(Label.ID, contextUser);
+
+// Restore if needed
+const result = await engine.RestoreToLabel(Label.ID, { DryRun: true }, contextUser);
+```
+
+### WalkOptions
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `MaxDepth` | `10` | Maximum recursion depth |
+| `EntityFilter` | `[]` | Only include these entities (empty = all) |
+| `ExcludeEntities` | `[]` | Skip these entities entirely |
+| `IncludeDeleted` | `false` | Include soft-deleted records |
+
+## Dependencies
+
+| Package | Purpose |
+|---------|---------|
+| `@memberjunction/core` | Entity system, metadata, and CompositeKey |
+| `@memberjunction/core-entities` | VersionLabel entity types |
+| `@memberjunction/global` | Global state management |
+
+## License
+
+ISC

package/dist/DependencyGraphWalker.d.ts

@@ -0,0 +1,165 @@
+import { CompositeKey, UserInfo } from '@memberjunction/core';
+import { DependencyNode, WalkOptions } from './types.js';
+/**
+ * Walks entity relationship graphs to discover records for version labeling.
+ *
+ * ## Two-direction traversal
+ *
+ * **Reverse**: Uses the entity's `RelatedEntities` (EntityRelationship metadata)
+ * to find child records. This is curated by CodeGen and admins — only explicitly
+ * registered One-To-Many relationships are walked.
+ *
+ * **Forward**: Scans the current entity's own FK fields to find records it
+ * references (e.g., AI Agent Prompt → AI Prompt via PromptID). Skips
+ * system/infrastructure fields (UserID, CreatedBy, etc.).
+ *
+ * ## Discovery mode — prevents explosion from forward references
+ *
+ * Records found via **reverse walk** are "owned children" and get full treatment
+ * (both reverse + forward walks). Records found via **forward walk** are
+ * "referenced records" and only get forward-only treatment (their own FK refs
+ * are followed, but their EntityRelationship children are NOT discovered).
+ *
+ * This prevents the classic explosion pattern:
+ * ```
+ * Agent → AgentPrompt → Prompt → [reverse to ALL PromptModels] → Model
+ *   → [reverse to ALL AgentModels across ALL agents] → explosion!
+ * ```
+ *
+ * With discovery mode:
+ * ```
+ * Agent → AgentPrompt(reverse,full) → Prompt(forward,forward-only)
+ *   → Model(forward,forward-only) → STOP (no reverse walk on Model)
+ * ```
+ *
+ * ## Ancestor stack — prevents backtracking
+ *
+ * A set of entity names on the current path from root to the current node is
+ * maintained. When evaluating any relationship, if the target entity type is
+ * already on the ancestor stack, it is skipped. This is a secondary safety net
+ * that prevents cycles even within full-mode walks.
+ *
+ * ## Filters
+ *
+ * - Only walks entities with `TrackRecordChanges === true`
+ * - Skips system FK fields via regex patterns (UserID, CreatedBy, etc.)
+ * - Respects `ExcludeEntities` and `EntityFilter` options
+ * - Uses a global `visited` set for record-level cycle detection
+ */
+export declare class DependencyGraphWalker {
+    /** Cache: entityID → reverse relationships from EntityRelationship metadata */
+    private reverseRelCache;
+    /** Cache: entityID → forward FK references from field metadata */
+    private forwardRefCache;
+    /**
+     * Walk from a root record through its relationships, building a tree of
+     * all records that should be included in a version label.
+     *
+     * @param entityName - The starting entity name
+     * @param recordKey  - The starting record's primary key
+     * @param options    - Controls depth, filtering, etc.
+     * @param contextUser - Server-side user context
+     * @returns The root DependencyNode with all descendants populated
+     */
+    WalkDependents(entityName: string, recordKey: CompositeKey, options: WalkOptions, contextUser: UserInfo): Promise<DependencyNode>;
+    /**
+     * Flatten a dependency tree into a topologically sorted list.
+     * Parents appear before their children, ensuring safe restore ordering.
+     */
+    FlattenTopological(root: DependencyNode): DependencyNode[];
+    /**
+     * Recursively discover and attach child nodes for a given parent node.
+     *
+     * The `discoveryMode` controls which directions are walked:
+     * - `'full'`: both reverse (EntityRelationship) and forward (FK refs)
+     * - `'forward-only'`: only forward FK refs — no reverse children discovered
+     *
+     * Records found via reverse walk are recursed with `'full'` mode.
+     * Records found via forward walk are recursed with `'forward-only'` mode.
+     */
+    private walkChildren;
+    /**
+     * Walk reverse relationships: for each EntityRelationship on the parent
+     * entity, load matching child records and recurse into them.
+     *
+     * Skips any child entity type that is already on the ancestor stack,
+     * preventing backtracking (e.g., from AI Prompt back to AI Agent Prompts).
+     *
+     * Children found via reverse walk are recursed with 'full' discovery mode,
+     * since they are "owned children" that may have their own children.
+     */
+    private walkReverseRelationships;
+    /**
+     * Walk forward FK references: for each FK field on the parent entity that
+     * points to another tracked entity, load the referenced record and recurse.
+     *
+     * Skips system/infrastructure FKs (UserID, CreatedBy, etc.) and any target
+     * entity type already on the ancestor stack.
+     *
+     * Targets found via forward walk are recursed with 'forward-only' discovery
+     * mode — they won't discover their own reverse children, preventing graph
+     * explosion from "hub" entities like AI Models.
+     */
+    private walkForwardReferences;
+    /**
+     * Discover reverse relationships for an entity using EntityRelationship
+     * metadata. Only includes One-To-Many relationships to entities that have
+     * TrackRecordChanges enabled.
+     *
+     * Uses EntityRelationship (already loaded in memory on EntityInfo) rather
+     * than scanning all entities' FK fields. This means only relationships
+     * that CodeGen or admins have explicitly registered are walked.
+     */
+    private discoverReverseRelationships;
+    /**
+     * Discover forward FK references on an entity — fields that point TO other
+     * entities. Only includes references to entities with TrackRecordChanges
+     * enabled, and skips system/infrastructure FK fields.
+     */
+    private discoverForwardReferences;
+    /**
+     * Load child records for a reverse relationship.
+     * Queries the child entity where the join field matches the parent's key value.
+     */
+    private loadChildRecords;
+    /**
+     * Load a single record's data by primary key.
+     */
+    private loadRecordData;
+    /**
+     * Build a DependencyNode from record data, register it in the visited set,
+     * and attach it as a child of the parent. Returns null if already visited.
+     */
+    private registerNode;
+    /** Create an empty stats accumulator. */
+    private createEmptyStats;
+    /** Increment the per-entity record counter. */
+    private incrementEntityCount;
+    /**
+     * Log a detailed summary of the walk: total records, per-entity counts,
+     * skips, and suppression stats.
+     */
+    private logWalkSummary;
+    /** Build a unique key for the visited set: "EntityName::recordID" */
+    private visitKey;
+    /**
+     * Determine the parent key field from an EntityRelationship record.
+     * Uses EntityKeyField if specified, otherwise falls back to the first PK.
+     */
+    private resolveParentKeyFieldFromRelationship;
+    /**
+     * Determine the target key field from a FK field definition.
+     * Uses RelatedEntityFieldName if specified, otherwise falls back to the
+     * target entity's first PK.
+     */
+    private resolveParentKeyFieldFromFK;
+    /** Check if a field name matches a system/infrastructure FK pattern. */
+    private isSystemFKField;
+    /** Check if an entity should be skipped based on filter options. */
+    private shouldSkipEntity;
+    /** Check if an entity has a soft delete field. */
+    private entityHasSoftDelete;
+    /** Apply defaults to walk options. */
+    private resolveDefaults;
+}
+//# sourceMappingURL=DependencyGraphWalker.d.ts.map

package/dist/DependencyGraphWalker.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"DependencyGraphWalker.d.ts","sourceRoot":"","sources":["../src/DependencyGraphWalker.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,YAAY,EAA0E,QAAQ,EAAuB,MAAM,sBAAsB,CAAC;AAC3J,OAAO,EAAE,cAAc,EAAE,WAAW,EAAE,MAAM,SAAS,CAAC;AA6FtD;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA6CG;AACH,qBAAa,qBAAqB;IAC9B,+EAA+E;IAC/E,OAAO,CAAC,eAAe,CAA4C;IACnE,kEAAkE;IAClE,OAAO,CAAC,eAAe,CAAyC;IAMhE;;;;;;;;;OASG;IACU,cAAc,CACvB,UAAU,EAAE,MAAM,EAClB,SAAS,EAAE,YAAY,EACvB,OAAO,EAAE,WAAW,EACpB,WAAW,EAAE,QAAQ,GACtB,OAAO,CAAC,cAAc,CAAC;IA2C1B;;;OAGG;IACI,kBAAkB,CAAC,IAAI,EAAE,cAAc,GAAG,cAAc,EAAE;IAiBjE;;;;;;;;;OASG;YACW,YAAY;IA4B1B;;;;;;;;;OASG;YACW,wBAAwB;IAyCtC;;;;;;;;;;OAUG;YACW,qBAAqB;IAwDnC;;;;;;;;OAQG;IACH,OAAO,CAAC,4BAA4B;IA6BpC;;;;OAIG;IACH,OAAO,CAAC,yBAAyB;IA+BjC;;;OAGG;YACW,gBAAgB;IAsC9B;;OAEG;YACW,cAAc;IA0B5B;;;OAGG;IACH,OAAO,CAAC,YAAY;IAqCpB,yCAAyC;IACzC,OAAO,CAAC,gBAAgB;IAUxB,+CAA+C;IAC/C,OAAO,CAAC,oBAAoB;IAK5B;;;OAGG;IACH,OAAO,CAAC,cAAc;IA0BtB,qEAAqE;IACrE,OAAO,CAAC,QAAQ;IAIhB;;;OAGG;IACH,OAAO,CAAC,qCAAqC;IAU7C;;;;OAIG;IACH,OAAO,CAAC,2BAA2B;IAUnC,wEAAwE;IACxE,OAAO,CAAC,eAAe;IAIvB,oEAAoE;IACpE,OAAO,CAAC,gBAAgB;IAUxB,kDAAkD;IAClD,OAAO,CAAC,mBAAmB;IAI3B,sCAAsC;IACtC,OAAO,CAAC,eAAe;CAQ1B"}