@memberjunction/version-history 0.0.1 → 4.0.0

package/README.md

@@ -1,45 +1,229 @@
 # @memberjunction/version-history
 
-## ⚠️ IMPORTANT NOTICE ⚠️
+Version labeling, snapshot capture, diff, and restore for MemberJunction records.
+
+## Architecture Overview
+
+```
+VersionHistoryEngine (facade)
+├── LabelManager         — label CRUD and lifecycle
+├── SnapshotBuilder      — captures record state into label items (batched)
+├── DiffEngine           — compares snapshots between labels
+├── RestoreEngine        — applies labeled state back to records
+└── DependencyGraphWalker — traverses entity relationships
+```
+
+## Dependency Graph Walker
+
+The walker discovers all records that should be included in a version label. It
+traverses both **reverse relationships** (child records that belong to the root)
+and **forward references** (records the root or its children point to).
+
+### Two Key Mechanisms
+
+#### 1. EntityRelationship-Driven Reverse Walking
+
+Instead of scanning every entity in the system for foreign keys that point to the
+current entity (expensive O(N*M) scan), the walker uses the
+**EntityRelationship metadata** that MemberJunction already maintains.
+
+Each entity's `RelatedEntities` array defines which child entities are
+meaningful. These are auto-generated by CodeGen when FK relationships are
+detected, and admins can add/remove them. This means only explicitly registered
+children are walked — not every table that happens to have a matching FK.
+
+```mermaid
+graph TD
+    A[AI Agents] -->|"EntityRelationship<br/>RelatedEntities"| B[AI Agent Prompts]
+    A -->|EntityRelationship| C[AI Agent Actions]
+    A -->|EntityRelationship| D[AI Agent Relationships]
+    A -->|EntityRelationship| E[AI Agent Models]
+    A -->|EntityRelationship| F[AI Agent Configurations]
+    A -.-x|"NOT walked<br/>(no EntityRelationship)"| G[Some Random Table<br/>with AgentID FK]
+
+    style G fill:#fee,stroke:#c00,stroke-dasharray: 5 5
+```
+
+#### 2. Ancestor Stack — Prevents Backtracking
+
+The walker maintains a **stack of entity type names** representing the path from
+root to the current node. When evaluating any relationship (reverse or forward),
+if the target entity type is already on the ancestor stack, it is **skipped**.
+
+This surgically prevents graph explosion without arbitrary depth limits:
+
+```mermaid
+graph TD
+    Agent["AI Agents<br/>(root)"] --> AP["AI Agent Prompts<br/>(reverse)"]
+    AP --> Prompt["AI Prompts<br/>(forward via PromptID)"]
+    Prompt -.->|"BLOCKED<br/>AI Agent Prompts<br/>is on ancestor stack"| AP2["Other AI Agent Prompts<br/>referencing same prompt"]
+    Prompt --> Model["AI Models<br/>(forward via DefaultModelID)"]
+    Model -.->|"BLOCKED<br/>AI Prompts<br/>is on ancestor stack"| Prompt2["Other AI Prompts<br/>using same model"]
+
+    style AP2 fill:#fee,stroke:#c00,stroke-dasharray: 5 5
+    style Prompt2 fill:#fee,stroke:#c00,stroke-dasharray: 5 5
+```
+
+### Walk Algorithm — Step by Step
+
+```mermaid
+flowchart TD
+    Start([walkChildren called]) --> DepthCheck{Depth >= MaxDepth?}
+    DepthCheck -->|Yes| Stop([Return])
+    DepthCheck -->|No| Reverse[Walk Reverse Relationships]
+
+    Reverse --> ReverseLoop{For each EntityRelationship<br/>on current entity}
+    ReverseLoop -->|Next rel| AncestorCheckR{Child entity<br/>on ancestor stack?}
+    AncestorCheckR -->|Yes| SkipR([Skip — would backtrack])
+    AncestorCheckR -->|No| LoadChildren[Load child records<br/>via RunView]
+    LoadChildren --> RegisterR[Register nodes + push<br/>child entity onto ancestors]
+    RegisterR --> RecurseR[Recurse walkChildren<br/>for each child]
+    RecurseR --> PopR[Pop child entity<br/>from ancestors]
+    PopR --> ReverseLoop
+
+    ReverseLoop -->|Done| Forward[Walk Forward References]
+    Forward --> ForwardLoop{For each FK field<br/>on current entity}
+    ForwardLoop -->|Next FK| SystemCheck{System FK?<br/>UserID, CreatedBy, etc.}
+    SystemCheck -->|Yes| SkipF([Skip — infrastructure field])
+    SystemCheck -->|No| AncestorCheckF{Target entity<br/>on ancestor stack?}
+    AncestorCheckF -->|Yes| SkipF2([Skip — would backtrack])
+    AncestorCheckF -->|No| LoadTarget[Load referenced record<br/>via RunView]
+    LoadTarget --> RegisterF[Register node + push<br/>target entity onto ancestors]
+    RegisterF --> RecurseF[Recurse walkChildren<br/>for referenced record]
+    RecurseF --> PopF[Pop target entity<br/>from ancestors]
+    PopF --> ForwardLoop
+    ForwardLoop -->|Done| Stop
+
+    SkipR --> ReverseLoop
+    SkipF --> ForwardLoop
+    SkipF2 --> ForwardLoop
+
+    style SkipR fill:#fee,stroke:#c00
+    style SkipF fill:#fee,stroke:#c00
+    style SkipF2 fill:#fee,stroke:#c00
+```
+
+### Concrete Example — Labeling an AI Agent
+
+Given an AI Agent with 2 prompts, 1 action, and 1 sub-agent (via AI Agent
+Relationships), the walker produces:
+
+```mermaid
+graph TD
+    Root["AI Agent (root)<br/>ancestors: [AI Agents]"]
+
+    Root --> AP1["AI Agent Prompt #1<br/>(reverse)<br/>ancestors: [..., AI Agent Prompts]"]
+    Root --> AP2["AI Agent Prompt #2<br/>(reverse)"]
+    Root --> AA["AI Agent Action<br/>(reverse)<br/>ancestors: [..., AI Agent Actions]"]
+    Root --> AR["AI Agent Relationship<br/>(reverse)<br/>ancestors: [..., AI Agent Relationships]"]
+
+    AP1 --> P1["AI Prompt #1<br/>(forward via PromptID)<br/>ancestors: [..., AI Prompts]"]
+    AP2 --> P2["AI Prompt #2<br/>(forward)"]
+    AA --> Act["Action<br/>(forward via ActionID)<br/>ancestors: [..., Actions]"]
+    AR --> SubAgent["Sub-Agent<br/>(forward via SubAgentID)<br/>ancestors: [..., AI Agents]"]
+
+    P1 --> M1["AI Model<br/>(forward via DefaultModelID)"]
+    P2 --> M1
+
+    SubAgent --> SAP["Sub-Agent's Prompt<br/>(reverse)"]
+    SAP --> SP["AI Prompt #3<br/>(forward)"]
+
+    P1 -.->|"BLOCKED: AI Agent Prompts on stack"| X1["Other Agent Prompts"]
+    M1 -.->|"BLOCKED: AI Prompts on stack"| X2["Other Prompts"]
+    Act -.->|"BLOCKED: skipped via EntityRel filter"| X3["Action Params (403 rows)"]
+
+    style X1 fill:#fee,stroke:#c00,stroke-dasharray: 5 5
+    style X2 fill:#fee,stroke:#c00,stroke-dasharray: 5 5
+    style X3 fill:#fee,stroke:#c00,stroke-dasharray: 5 5
+```
+
+**Result**: ~15-25 targeted records instead of 1,363 from the naive approach.
+
+### Why This Design
+
+| Concern | Solution |
+|---|---|
+| Which children to walk? | **EntityRelationship** — admin-controlled, CodeGen-maintained |
+| Preventing graph explosion? | **Ancestor stack** — blocks backtracking to any entity type on current path |
+| Infrastructure FKs (UserID, etc.)? | **System FK skip list** — regex patterns for known infrastructure fields |
+| Cycle detection? | **Visited set** — `entityName::recordID` prevents revisiting any record |
+| Sub-agent recursion? | Ancestor stack is **path-based** — pops on backtrack, allowing re-entry from a different branch |
+
+### Forward FK Skip Patterns
+
+The following FK field name patterns are never followed during forward walking,
+as they reference system infrastructure (Users, audit fields) rather than
+business data:
+
+- `CreatedByUserID`, `UpdatedByUserID`, `UserID`
+- `ContextUserID`, `ModifiedByUserID`
+- `CreatedBy`, `UpdatedBy`
+- `OwnerID`, `OwnerUserID`
+- `AssignedToID`, `AssignedToUserID`
+- `EntityID` (polymorphic reference)
+
+## Snapshot Builder — Batched Capture
+
+When capturing records into a version label, the SnapshotBuilder uses **batched
+queries** to minimize database round trips:
+
+```mermaid
+sequenceDiagram
+    participant SB as SnapshotBuilder
+    participant DB as Database
+
+    Note over SB: Receive flat list of N nodes
+
+    SB->>SB: Group nodes by EntityID
+    loop For each entity group
+        SB->>DB: ONE RunView: RecordChanges<br/>WHERE EntityID = X<br/>AND RecordID IN (a, b, c, ...)
+        DB-->>SB: Latest changes for all records in group
+    end
+
+    Note over SB: Build lookup map:<br/>entityId::recordId → RecordChange
+
+    loop For each node without a RecordChange
+        SB->>DB: Create synthetic snapshot (Save)
+    end
+
+    loop For each node
+        SB->>DB: Create VersionLabelItem (Save)
+    end
+```
 
-**This package is created solely for the purpose of setting up OIDC (OpenID Connect) trusted publishing with npm.**
+**Before batching**: N individual RunView calls (946 for a 1363-record label).
+**After batching**: ~5-10 RunView calls (one per unique entity type in the graph).
 
-This is **NOT** a functional package and contains **NO** code or functionality beyond the OIDC setup configuration.
+## API
 
-## Purpose
+### VersionHistoryEngine (main facade)
 
-This package exists to:
-1. Configure OIDC trusted publishing for the package name `@memberjunction/version-history`
-2. Enable secure, token-less publishing from CI/CD workflows
-3. Establish provenance for packages published under this name
+```typescript
+const engine = new VersionHistoryEngine();
 
-## What is OIDC Trusted Publishing?
+// Create a label with dependency walking
+const { Label, CaptureResult } = await engine.CreateLabel({
+    Name: 'Before Refactor v2',
+    Scope: 'Record',
+    EntityName: 'AI Agents',
+    RecordKey: agentKey,
+    IncludeDependencies: true,
+    MaxDepth: 10,
+    ExcludeEntities: ['AI Agent Runs'],  // skip run history
+}, contextUser);
 
-OIDC trusted publishing allows package maintainers to publish packages directly from their CI/CD workflows without needing to manage npm access tokens. Instead, it uses OpenID Connect to establish trust between the CI/CD provider (like GitHub Actions) and npm.
+// Diff against current state
+const diff = await engine.DiffLabelToCurrentState(Label.ID, contextUser);
 
-## Setup Instructions
+// Restore if needed
+const result = await engine.RestoreToLabel(Label.ID, { DryRun: true }, contextUser);
+```
 
-To properly configure OIDC trusted publishing for this package:
+### WalkOptions
 
-1. Go to [npmjs.com](https://www.npmjs.com/) and navigate to your package settings
-2. Configure the trusted publisher (e.g., GitHub Actions)
-3. Specify the repository and workflow that should be allowed to publish
-4. Use the configured workflow to publish your actual package
-
-## DO NOT USE THIS PACKAGE
-
-This package is a placeholder for OIDC configuration only. It:
-- Contains no executable code
-- Provides no functionality
-- Should not be installed as a dependency
-- Exists only for administrative purposes
-
-## More Information
-
-For more details about npm's trusted publishing feature, see:
-- [npm Trusted Publishing Documentation](https://docs.npmjs.com/generating-provenance-statements)
-- [GitHub Actions OIDC Documentation](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect)
-
----
-
-**Maintained for OIDC setup purposes only**
+| Option | Default | Description |
+|---|---|---|
+| `MaxDepth` | `10` | Maximum recursion depth |
+| `EntityFilter` | `[]` | Only include these entities (empty = all) |
+| `ExcludeEntities` | `[]` | Skip these entities entirely |
+| `IncludeDeleted` | `false` | Include soft-deleted records |

package/dist/DependencyGraphWalker.d.ts

@@ -0,0 +1,165 @@
+import { CompositeKey, UserInfo } from '@memberjunction/core';
+import { DependencyNode, WalkOptions } from './types.js';
+/**
+ * Walks entity relationship graphs to discover records for version labeling.
+ *
+ * ## Two-direction traversal
+ *
+ * **Reverse**: Uses the entity's `RelatedEntities` (EntityRelationship metadata)
+ * to find child records. This is curated by CodeGen and admins — only explicitly
+ * registered One-To-Many relationships are walked.
+ *
+ * **Forward**: Scans the current entity's own FK fields to find records it
+ * references (e.g., AI Agent Prompt → AI Prompt via PromptID). Skips
+ * system/infrastructure fields (UserID, CreatedBy, etc.).
+ *
+ * ## Discovery mode — prevents explosion from forward references
+ *
+ * Records found via **reverse walk** are "owned children" and get full treatment
+ * (both reverse + forward walks). Records found via **forward walk** are
+ * "referenced records" and only get forward-only treatment (their own FK refs
+ * are followed, but their EntityRelationship children are NOT discovered).
+ *
+ * This prevents the classic explosion pattern:
+ * ```
+ * Agent → AgentPrompt → Prompt → [reverse to ALL PromptModels] → Model
+ *   → [reverse to ALL AgentModels across ALL agents] → explosion!
+ * ```
+ *
+ * With discovery mode:
+ * ```
+ * Agent → AgentPrompt(reverse,full) → Prompt(forward,forward-only)
+ *   → Model(forward,forward-only) → STOP (no reverse walk on Model)
+ * ```
+ *
+ * ## Ancestor stack — prevents backtracking
+ *
+ * A set of entity names on the current path from root to the current node is
+ * maintained. When evaluating any relationship, if the target entity type is
+ * already on the ancestor stack, it is skipped. This is a secondary safety net
+ * that prevents cycles even within full-mode walks.
+ *
+ * ## Filters
+ *
+ * - Only walks entities with `TrackRecordChanges === true`
+ * - Skips system FK fields via regex patterns (UserID, CreatedBy, etc.)
+ * - Respects `ExcludeEntities` and `EntityFilter` options
+ * - Uses a global `visited` set for record-level cycle detection
+ */
+export declare class DependencyGraphWalker {
+    /** Cache: entityID → reverse relationships from EntityRelationship metadata */
+    private reverseRelCache;
+    /** Cache: entityID → forward FK references from field metadata */
+    private forwardRefCache;
+    /**
+     * Walk from a root record through its relationships, building a tree of
+     * all records that should be included in a version label.
+     *
+     * @param entityName - The starting entity name
+     * @param recordKey  - The starting record's primary key
+     * @param options    - Controls depth, filtering, etc.
+     * @param contextUser - Server-side user context
+     * @returns The root DependencyNode with all descendants populated
+     */
+    WalkDependents(entityName: string, recordKey: CompositeKey, options: WalkOptions, contextUser: UserInfo): Promise<DependencyNode>;
+    /**
+     * Flatten a dependency tree into a topologically sorted list.
+     * Parents appear before their children, ensuring safe restore ordering.
+     */
+    FlattenTopological(root: DependencyNode): DependencyNode[];
+    /**
+     * Recursively discover and attach child nodes for a given parent node.
+     *
+     * The `discoveryMode` controls which directions are walked:
+     * - `'full'`: both reverse (EntityRelationship) and forward (FK refs)
+     * - `'forward-only'`: only forward FK refs — no reverse children discovered
+     *
+     * Records found via reverse walk are recursed with `'full'` mode.
+     * Records found via forward walk are recursed with `'forward-only'` mode.
+     */
+    private walkChildren;
+    /**
+     * Walk reverse relationships: for each EntityRelationship on the parent
+     * entity, load matching child records and recurse into them.
+     *
+     * Skips any child entity type that is already on the ancestor stack,
+     * preventing backtracking (e.g., from AI Prompt back to AI Agent Prompts).
+     *
+     * Children found via reverse walk are recursed with 'full' discovery mode,
+     * since they are "owned children" that may have their own children.
+     */
+    private walkReverseRelationships;
+    /**
+     * Walk forward FK references: for each FK field on the parent entity that
+     * points to another tracked entity, load the referenced record and recurse.
+     *
+     * Skips system/infrastructure FKs (UserID, CreatedBy, etc.) and any target
+     * entity type already on the ancestor stack.
+     *
+     * Targets found via forward walk are recursed with 'forward-only' discovery
+     * mode — they won't discover their own reverse children, preventing graph
+     * explosion from "hub" entities like AI Models.
+     */
+    private walkForwardReferences;
+    /**
+     * Discover reverse relationships for an entity using EntityRelationship
+     * metadata. Only includes One-To-Many relationships to entities that have
+     * TrackRecordChanges enabled.
+     *
+     * Uses EntityRelationship (already loaded in memory on EntityInfo) rather
+     * than scanning all entities' FK fields. This means only relationships
+     * that CodeGen or admins have explicitly registered are walked.
+     */
+    private discoverReverseRelationships;
+    /**
+     * Discover forward FK references on an entity — fields that point TO other
+     * entities. Only includes references to entities with TrackRecordChanges
+     * enabled, and skips system/infrastructure FK fields.
+     */
+    private discoverForwardReferences;
+    /**
+     * Load child records for a reverse relationship.
+     * Queries the child entity where the join field matches the parent's key value.
+     */
+    private loadChildRecords;
+    /**
+     * Load a single record's data by primary key.
+     */
+    private loadRecordData;
+    /**
+     * Build a DependencyNode from record data, register it in the visited set,
+     * and attach it as a child of the parent. Returns null if already visited.
+     */
+    private registerNode;
+    /** Create an empty stats accumulator. */
+    private createEmptyStats;
+    /** Increment the per-entity record counter. */
+    private incrementEntityCount;
+    /**
+     * Log a detailed summary of the walk: total records, per-entity counts,
+     * skips, and suppression stats.
+     */
+    private logWalkSummary;
+    /** Build a unique key for the visited set: "EntityName::recordID" */
+    private visitKey;
+    /**
+     * Determine the parent key field from an EntityRelationship record.
+     * Uses EntityKeyField if specified, otherwise falls back to the first PK.
+     */
+    private resolveParentKeyFieldFromRelationship;
+    /**
+     * Determine the target key field from a FK field definition.
+     * Uses RelatedEntityFieldName if specified, otherwise falls back to the
+     * target entity's first PK.
+     */
+    private resolveParentKeyFieldFromFK;
+    /** Check if a field name matches a system/infrastructure FK pattern. */
+    private isSystemFKField;
+    /** Check if an entity should be skipped based on filter options. */
+    private shouldSkipEntity;
+    /** Check if an entity has a soft delete field. */
+    private entityHasSoftDelete;
+    /** Apply defaults to walk options. */
+    private resolveDefaults;
+}
+//# sourceMappingURL=DependencyGraphWalker.d.ts.map

package/dist/DependencyGraphWalker.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"DependencyGraphWalker.d.ts","sourceRoot":"","sources":["../src/DependencyGraphWalker.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,YAAY,EAA0E,QAAQ,EAAuB,MAAM,sBAAsB,CAAC;AAC3J,OAAO,EAAE,cAAc,EAAE,WAAW,EAAE,MAAM,SAAS,CAAC;AA6FtD;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA6CG;AACH,qBAAa,qBAAqB;IAC9B,+EAA+E;IAC/E,OAAO,CAAC,eAAe,CAA4C;IACnE,kEAAkE;IAClE,OAAO,CAAC,eAAe,CAAyC;IAMhE;;;;;;;;;OASG;IACU,cAAc,CACvB,UAAU,EAAE,MAAM,EAClB,SAAS,EAAE,YAAY,EACvB,OAAO,EAAE,WAAW,EACpB,WAAW,EAAE,QAAQ,GACtB,OAAO,CAAC,cAAc,CAAC;IA2C1B;;;OAGG;IACI,kBAAkB,CAAC,IAAI,EAAE,cAAc,GAAG,cAAc,EAAE;IAiBjE;;;;;;;;;OASG;YACW,YAAY;IA4B1B;;;;;;;;;OASG;YACW,wBAAwB;IAyCtC;;;;;;;;;;OAUG;YACW,qBAAqB;IAwDnC;;;;;;;;OAQG;IACH,OAAO,CAAC,4BAA4B;IA6BpC;;;;OAIG;IACH,OAAO,CAAC,yBAAyB;IA+BjC;;;OAGG;YACW,gBAAgB;IAsC9B;;OAEG;YACW,cAAc;IA0B5B;;;OAGG;IACH,OAAO,CAAC,YAAY;IAqCpB,yCAAyC;IACzC,OAAO,CAAC,gBAAgB;IAUxB,+CAA+C;IAC/C,OAAO,CAAC,oBAAoB;IAK5B;;;OAGG;IACH,OAAO,CAAC,cAAc;IA0BtB,qEAAqE;IACrE,OAAO,CAAC,QAAQ;IAIhB;;;OAGG;IACH,OAAO,CAAC,qCAAqC;IAU7C;;;;OAIG;IACH,OAAO,CAAC,2BAA2B;IAUnC,wEAAwE;IACxE,OAAO,CAAC,eAAe;IAIvB,oEAAoE;IACpE,OAAO,CAAC,gBAAgB;IAUxB,kDAAkD;IAClD,OAAO,CAAC,mBAAmB;IAI3B,sCAAsC;IACtC,OAAO,CAAC,eAAe;CAQ1B"}