masterrecord 0.3.37 → 0.3.39

package/.claude/settings.local.json

@@ -62,7 +62,8 @@
       "Bash(masterrecord update-database:*)",
       "Bash(npx mocha:*)",
       "Bash(masterrecord add-migration:*)",
-      "Bash(masterrecord:*)"
+      "Bash(masterrecord:*)",
+      "Bash(git -C /Users/alexanderrich/Documents/development/bookbaghq/bookbag-training log --oneline --all -- *MASTERRECORD_ISSUE*)"
     ],
     "deny": [],
     "ask": []

package/FOREIGN_KEY_STRING_FIX.md

@@ -0,0 +1,288 @@
+# Foreign Key String Value Bug - FIXED in v0.3.39
+
+## Problem Summary
+
+When assigning string values to foreign key fields defined via `belongsTo()`, MasterRecord silently excluded them from INSERT statements, causing NOT NULL constraint failures.
+
+## Example of the Bug
+
+```javascript
+// Entity with belongsTo relationship
+class UserOrganizationRole extends EntityModel {
+    User(db) {
+        db.belongsTo('User', 'user_id');  // Creates foreignKey 'user_id'
+    }
+
+    Organization(db) {
+        db.belongsTo('Organization', 'organization_id');
+    }
+}
+
+// User code
+const orgRole = new UserOrganizationRole();
+orgRole.user_id = currentUser.id;  // currentUser.id is STRING "2"
+orgRole.organization_id = newOrg.id;  // newOrg.id is NUMBER 8
+orgRole.role = 'org_admin';
+
+await userContext.saveChanges();
+// ❌ BEFORE: SqliteError: NOT NULL constraint failed: UserOrganizationRole.user_id
+// ✅ AFTER: Works perfectly!
+```
+
+## Root Cause
+
+1. `belongsTo('User', 'user_id')` creates an entity property named `User` with metadata `foreignKey: 'user_id'`
+2. The INSERT builder looked for `fields['User']` (navigation property name)
+3. But users set `fields['user_id']` (foreign key field name)
+4. Field not found → silently skipped → INSERT statement missing the field
+
+**Generated SQL (BEFORE):**
+```sql
+INSERT INTO [UserOrganizationRole] ([organization_id], [role])
+VALUES (8, 'org_admin')
+-- ❌ user_id is missing!
+```
+
+**Generated SQL (AFTER):**
+```sql
+INSERT INTO [UserOrganizationRole] ([user_id], [organization_id], [role])
+VALUES (2, 8, 'org_admin')
+-- ✅ user_id is included and auto-converted from "2" to 2
+```
+
+## The Fix
+
+Modified `_buildSQLInsertObjectParameterized()` in all three database engines to check BOTH the navigation property name AND the foreign key field name:
+
+```javascript
+// SQLLiteEngine.js, mySQLEngine.js, postgresEngine.js
+for (const column in modelEntity) {
+    if (column.indexOf("__") === -1) {
+        let fieldColumn = fields[column];  // Check navigation property (e.g., 'User')
+
+        // 🔥 NEW: Also check the foreignKey field name (e.g., 'user_id')
+        if ((fieldColumn === undefined || fieldColumn === null) &&
+            modelEntity[column].relationshipType === "belongsTo" &&
+            modelEntity[column].foreignKey) {
+            fieldColumn = fields[modelEntity[column].foreignKey];
+        }
+
+        if ((fieldColumn !== undefined && fieldColumn !== null) && typeof(fieldColumn) !== "object") {
+            // Existing validation and type coercion logic...
+            // This already handles string-to-integer conversion!
+        }
+    }
+}
+```
+
+## Auto-Conversion
+
+The existing `_validateAndCoerceFieldType()` method already handled string-to-integer conversion:
+
+```javascript
+case "integer":
+    if (actualType === 'string') {
+        const parsed = parseInt(value, 10);
+        if (isNaN(parsed)) {
+            throw new Error(`Type mismatch for ${entityName}.${fieldName}: Expected integer, got string "${value}" which cannot be converted`);
+        }
+        console.warn(`⚠️  Field ${entityName}.${fieldName}: Auto-converting string "${value}" to integer ${parsed}`);
+        return parsed;
+    }
+```
+
+So once the field is found, the conversion happens automatically!
+
+## Why String IDs Happen in Real Apps
+
+### 1. authService Returns String IDs
+```javascript
+// authService.js line 167:
+res.id = String(obj.user.id);  // Explicit string conversion
+
+// Returns:
+{
+  id: "2",  // ← STRING
+  email: "customer1@bookbag.ai",
+  system_role: "system_user"
+}
+```
+
+### 2. HTTP Requests (JSON)
+```javascript
+// Client sends:
+{ "user_id": "2", "organization_id": "8" }
+
+// Express parses as strings
+req.body.user_id  // "2" (string)
+```
+
+### 3. JWT Tokens
+```javascript
+const token = jwt.decode(req.headers.authorization);
+token.userId  // "2" (string from JWT claim)
+```
+
+### 4. Database Returns Numbers
+```javascript
+const newOrg = new Organization();
+// ... set properties ...
+await organizationContext.saveChanges();
+
+newOrg.id  // 8 (NUMBER from auto-increment)
+```
+
+**Result:** Mixed types are common in real-world apps!
+
+## Both Patterns Now Work
+
+### Pattern 1: Navigation Property (OLD, still works)
+```javascript
+const user = await ctx.User.where(u => u.id == 2).first();
+orgRole.User = user;  // Set navigation property
+```
+
+### Pattern 2: Foreign Key Field (NEW, now works!)
+```javascript
+orgRole.user_id = currentUser.id;  // "2" (string)
+// Auto-converted to 2 (integer) ✅
+```
+
+### Pattern 3: Direct Integer (always worked)
+```javascript
+orgRole.user_id = 2;  // Already a number
+```
+
+## Test Coverage
+
+**8 comprehensive tests** verify:
+1. ✅ String foreign key value included in INSERT
+2. ✅ Number foreign key value still works
+3. ✅ Mixed string and number foreign keys
+4. ✅ String with leading zeros ("007" → 7)
+5. ✅ Invalid strings throw clear error
+6. ✅ Empty strings throw clear error
+7. ✅ Backward compatible (navigation property)
+8. ✅ Prefers navigation property if both set
+
+Run tests:
+```bash
+node test/foreign-key-string-value-test.js
+```
+
+## Affected Database Engines
+
+All three engines fixed:
+- ✅ SQLiteEngine (`SQLLiteEngine.js`)
+- ✅ MySQLEngine (`mySQLEngine.js`)
+- ✅ PostgresEngine (`postgresEngine.js`)
+
+## Upgrade Path
+
+```bash
+npm install -g masterrecord@0.3.39
+```
+
+**No code changes needed!** The fix is automatic.
+
+### If You Have Workarounds
+
+If you added `parseInt()` workarounds like this:
+```javascript
+orgRole.user_id = parseInt(currentUser.id, 10);
+```
+
+You can now remove them (but leaving them is harmless):
+```javascript
+orgRole.user_id = currentUser.id;  // Works directly now!
+```
+
+## Error Messages (Before vs After)
+
+### Before v0.3.39
+```
+SqliteError: NOT NULL constraint failed: UserOrganizationRole.user_id
+```
+**Confusing!** The field WAS set, but silently skipped.
+
+### After v0.3.39
+If you pass an invalid string:
+```
+INSERT failed: Type mismatch for UserOrganizationRole.User: Expected integer, got string "invalid" which cannot be converted to a number
+```
+**Clear!** Tells you exactly what's wrong.
+
+## Impact
+
+- ✅ **Auto-converts** string foreign keys to integers
+- ✅ **Clear errors** for invalid values (not silent failures)
+- ✅ **Backward compatible** - all existing code works
+- ✅ **Matches real-world usage** where IDs are often strings
+- ✅ **Works across all databases** (SQLite, MySQL, PostgreSQL)
+
+## Files Changed
+
+1. **SQLLiteEngine.js** - Lines 1127-1137
+2. **mySQLEngine.js** - Lines 654-664
+3. **postgresEngine.js** - Lines 601-611
+4. **test/foreign-key-string-value-test.js** (NEW)
+5. **package.json** - Version 0.3.39
+6. **readme.md** - Changelog entry
+7. **MEMORY.md** - Implementation notes
+
+## Related Issues
+
+This bug was reported by Bookbag.ai Engineering Team and affects any application that:
+- Uses JWT tokens for authentication (IDs as strings)
+- Receives data from HTTP requests (JSON has string values)
+- Mixes data from different sources (auth service + database)
+- Uses authService pattern (converts IDs to strings for consistency)
+
+## Verification
+
+To verify the fix works with your code:
+
+```javascript
+const UserOrganizationRole = require('./models/userOrganizationRole');
+const userContext = require('./models/userContext');
+
+async function testFix() {
+    const ctx = new userContext();
+
+    const orgRole = new UserOrganizationRole();
+    orgRole.user_id = "2";  // ← STRING VALUE
+    orgRole.organization_id = 1;
+    orgRole.role = "org_admin";
+    orgRole.created_at = Date.now().toString();
+    orgRole.updated_at = Date.now().toString();
+
+    ctx.UserOrganizationRole.add(orgRole);
+
+    try {
+        await ctx.saveChanges();
+        console.log('✅ SUCCESS! String foreign key value was auto-converted.');
+        console.log('Inserted record:', orgRole);
+    } catch (err) {
+        console.error('❌ FAILED:', err.message);
+    }
+}
+
+testFix();
+```
+
+Expected output:
+```
+⚠️  Field UserOrganizationRole.User: Auto-converting string "2" to integer 2
+✅ SUCCESS! String foreign key value was auto-converted.
+Inserted record: { id: 1, user_id: 2, organization_id: 1, role: 'org_admin', ... }
+```
+
+## Conclusion
+
+The bug is **completely fixed** in v0.3.39. String values assigned to foreign key fields are now:
+1. **Detected** (checked in both navigation property and foreign key field name)
+2. **Validated** (throws error for invalid strings)
+3. **Converted** (auto-converts to integer)
+4. **Included** in INSERT statements (no more silent failures)
+
+Upgrade to v0.3.39 and enjoy hassle-free foreign key handling! 🎉

package/GLOBAL_REGISTRY_VERIFICATION.md

@@ -0,0 +1,375 @@
+# Global Model Registry - Verification Document
+
+## Issue Fixed
+MasterRecord v0.3.38 eliminates confusing warnings during CLI operations when the same context class is instantiated multiple times.
+
+## The Problem (v0.3.36/v0.3.37)
+When users ran `masterrecord add-migration`, they saw warnings:
+```
+Warning: dbset() called multiple times for table 'User' - updating existing registration
+Warning: dbset() called multiple times for table 'Auth' - updating existing registration
+...
+```
+
+These warnings appeared during **normal operation** because:
+1. The CLI creates 2-3 instances of the context class to inspect schema
+2. Each instance runs the constructor
+3. Each constructor calls `dbset()` for each entity
+4. The duplicate detection (added in v0.3.36) triggered warnings
+
+## The Solution (v0.3.38)
+Added a global model registry that tracks which models have been registered per context class:
+- First instance of a context class: Warns about genuine duplicates in constructor
+- Subsequent instances: Silent (expected CLI behavior)
+
+## Technical Implementation
+
+### 1. Static Global Registry
+```javascript
+// context.js - Line 180
+static _globalModelRegistry = {};
+// Structure: { 'userContext': Set(['User', 'Auth', 'Settings']), 'qaContext': Set([...]) }
+```
+
+### 2. Instance-Level First Instance Flag
+```javascript
+// context.js - Constructor (Line 192)
+const globalRegistry = context._globalModelRegistry[this.__name];
+this.__isFirstInstance = !globalRegistry || globalRegistry.size === 0;
+```
+
+### 3. Conditional Warning in dbset()
+```javascript
+// context.js - dbset() method (Line 1050)
+if (existingIndex !== -1) {
+    // Duplicate detected in THIS instance
+    if (this.__isFirstInstance) {
+        // Only warn on first instance
+        console.warn(`Warning: dbset() called multiple times for table '${tableName}'...`);
+    }
+    // Update registration
+    this.__entities[existingIndex] = validModel;
+} else {
+    // New entity - add to arrays
+    this.__entities.push(validModel);
+}
+
+// Always mark as globally seen
+const globalRegistry = context._globalModelRegistry[this.__name];
+globalRegistry.add(tableName);
+```
+
+## Verification Tests
+
+### Test 1: Multiple Context Instances (CLI Pattern)
+```javascript
+class userContext extends context {
+    constructor() {
+        super();
+        this.dbset(User);
+        this.dbset(Auth);
+        this.dbset(Settings);
+    }
+}
+
+// CLI behavior simulation
+const ctx1 = new userContext();  // First instance
+const ctx2 = new userContext();  // Second instance
+const ctx3 = new userContext();  // Third instance
+
+// RESULT: Zero warnings (all instances work correctly)
+```
+
+### Test 2: Genuine Duplicate in Constructor
+```javascript
+class buggyContext extends context {
+    constructor() {
+        super();
+        this.dbset(User);    // Line 5
+        this.dbset(User);    // Line 6 - DUPLICATE!
+    }
+}
+
+const ctx1 = new buggyContext();  // First instance - WARNS
+const ctx2 = new buggyContext();  // Second instance - Silent
+
+// RESULT:
+// - First instance: Warns about duplicate (helps user fix their code)
+// - Subsequent instances: Silent (user already warned)
+```
+
+### Test 3: Different Context Classes
+```javascript
+class userContext extends context {
+    constructor() {
+        super();
+        this.dbset(User);
+    }
+}
+
+class adminContext extends context {
+    constructor() {
+        super();
+        this.dbset(User);  // Same model, different context
+    }
+}
+
+const userCtx = new userContext();
+const adminCtx = new adminContext();
+
+// RESULT: Zero warnings (different context classes have separate registries)
+```
+
+## Real-World Scenario: User's qaContext
+
+### User's Code Pattern (BEFORE)
+```javascript
+class qaContext extends context {
+    constructor() {
+        super();
+        // Line 58
+        this.dbset(TaxonomyTemplate);
+
+        // ... 150 lines of other code ...
+
+        // Line 207 - Common pattern: seed data
+        this.dbset(TaxonomyTemplate).seed(templates);
+    }
+}
+```
+
+### What Happened Before v0.3.38
+```bash
+$ masterrecord add-migration InitialCreate qaContext
+Warning: dbset() called multiple times for table 'TaxonomyTemplate' - updating existing registration
+Warning: dbset() called multiple times for table 'TaxonomyTemplate' - updating existing registration
+Warning: dbset() called multiple times for table 'TaxonomyTemplate' - updating existing registration
+✓ Migration 'InitialCreate' created successfully
+```
+- User confused: "Did I do something wrong?"
+- In reality: CLI just instantiated context 3 times (normal behavior)
+
+### What Happens With v0.3.38
+```bash
+$ masterrecord add-migration InitialCreate qaContext
+Warning: dbset() called multiple times for table 'TaxonomyTemplate' in constructor - updating existing registration
+✓ Migration 'InitialCreate' created successfully
+```
+- **One warning** (first instance) - Alerts user to duplicate `dbset()` in their code
+- User can fix: Remove line 58 or line 207 (depends on pattern)
+- After fix: Zero warnings on all future CLI operations
+
+### User's Fixed Code
+```javascript
+class qaContext extends context {
+    constructor() {
+        super();
+        // Only call dbset() once, with seed data attached
+        this.dbset(TaxonomyTemplate).seed(templates);
+    }
+}
+```
+
+```bash
+$ masterrecord add-migration InitialCreate qaContext
+✓ Migration 'InitialCreate' created successfully
+```
+✅ Clean output, no warnings!
+
+## Test Results Summary
+
+### Global Model Registry Tests (test/global-model-registry-test.js)
+- **15 tests** - All passing ✅
+  1. Multiple instances should not warn (CLI pattern)
+  2. Models should be added to global registry on first instance
+  3. Global registry should not have duplicates after multiple instances
+  4. Genuine duplicate in constructor should warn
+  5. Duplicate should warn only on first instance
+  6. Entity should be registered once despite duplicate in constructor
+  7. Same model in different context classes should not warn
+  8. Different context classes should have separate registries
+  9. Multiple instances of different contexts should not warn
+  10. qaContext pattern (dbset then dbset.seed) should warn about duplicate
+  11. Mixed registration should warn only about duplicates
+  12. Empty context should not warn
+  13. Large context with 50 models should not warn on multiple instances
+  14. Registry should not pollute other context classes
+  15. Many context classes should work independently
+
+### Integration with Existing Tests
+- **Entity Deduplication Tests** (test/entity-deduplication-test.js): 5/5 passing ✅
+- **Seed Deduplication Tests** (test/seed-deduplication-test.js): 8/8 passing ✅
+- **qaContext Pattern Tests** (test/qa-context-pattern-test.js): 7/7 passing ✅
+
+## Edge Cases Handled
+
+### 1. Empty Context
+```javascript
+class emptyContext extends context {
+    constructor() {
+        super();
+        // No entities
+    }
+}
+
+const ctx1 = new emptyContext();
+const ctx2 = new emptyContext();
+// RESULT: Zero warnings, registry exists but empty
+```
+
+### 2. Large Context (50+ Models)
+```javascript
+class largeContext extends context {
+    constructor() {
+        super();
+        for (let i = 0; i < 50; i++) {
+            this.dbset(models[i]);
+        }
+    }
+}
+
+const ctx1 = new largeContext();
+const ctx2 = new largeContext();
+// RESULT: Zero warnings, all 50 models registered correctly
+```
+
+### 3. Mixed Registration
+```javascript
+class mixedContext extends context {
+    constructor() {
+        super();
+        this.dbset(User);      // New
+        this.dbset(Auth);      // New
+        this.dbset(User);      // Duplicate
+        this.dbset(Settings);  // New
+        this.dbset(Auth);      // Duplicate
+    }
+}
+
+const ctx = new mixedContext();
+// RESULT: 2 warnings (User and Auth duplicates)
+// ctx.__entities.length === 3 (User, Auth, Settings)
+```
+
+## Memory Considerations
+
+### Registry Size
+- **Per context class**: One Set object
+- **Per model**: One string (table name)
+- **Typical application**: 3-10 context classes, 5-20 models each
+- **Memory footprint**: ~1-5 KB total (negligible)
+
+### Lifetime
+- Registry persists for application lifetime (intentional caching)
+- Not a memory leak - limited by number of context classes (fixed at compile time)
+- Cleared only on process restart or explicit call to `context.clearModelRegistry()` (if needed for testing)
+
+## Backward Compatibility
+
+### Existing Code Works Unchanged
+```javascript
+// v0.3.36 code
+class userContext extends context {
+    constructor() {
+        super();
+        this.dbset(User);
+        this.dbset(Auth);
+    }
+}
+
+// Still works in v0.3.38, no code changes needed
+```
+
+### Warning Messages Still Appear for Genuine Bugs
+```javascript
+// This still warns (on first instance)
+class buggyContext extends context {
+    constructor() {
+        super();
+        this.dbset(User);
+        this.dbset(User);  // Still warns: "Warning: dbset() called multiple times..."
+    }
+}
+```
+
+## Industry Standard Comparison
+
+### TypeORM Pattern
+```typescript
+@Entity()
+class User { ... }
+
+// Multiple data sources use same entity definition
+const ds1 = new DataSource({ entities: [User] });
+const ds2 = new DataSource({ entities: [User] });
+// No warnings, no re-registration
+```
+
+### Sequelize Pattern
+```javascript
+const UserModel = (sequelize) => sequelize.define('User', { ... });
+
+const db1 = new Sequelize();
+const User1 = UserModel(db1);
+
+const db2 = new Sequelize();
+const User2 = UserModel(db2);
+// No warnings, each connection gets its own model instance
+```
+
+### Mongoose Pattern
+```javascript
+const User = mongoose.model('User', userSchema);
+
+// Subsequent calls return cached model (no warning)
+const User2 = mongoose.model('User');
+```
+
+**MasterRecord v0.3.38 Now Matches This Pattern:**
+- First instance registers models, adds to global registry
+- Subsequent instances expected, no warnings
+- Genuine duplicates within same constructor still warn
+
+## Upgrade Path
+
+### For MasterRecord Users
+
+1. **Update to v0.3.38:**
+   ```bash
+   npm install -g masterrecord@0.3.38
+   ```
+
+2. **No code changes needed** - CLI warnings automatically cleaned up
+
+3. **If you see warnings** (on first instance only):
+   - Check your context constructor for duplicate `dbset()` calls
+   - Common pattern: `dbset(Entity)` + later `dbset(Entity).seed(data)`
+   - Fix: Remove one of the `dbset()` calls
+
+4. **After fixing duplicates:**
+   ```bash
+   masterrecord add-migration YourMigration yourContext
+   ```
+   Should see clean output with zero warnings.
+
+## Success Criteria Checklist
+
+✅ CLI commands produce clean output (no spurious warnings)
+✅ Multiple context instances work without warnings
+✅ Genuine duplicates in constructor still emit warnings (first instance only)
+✅ Different context classes maintain separate registries
+✅ All existing tests still pass (entity deduplication, seed deduplication, etc.)
+✅ 15 new tests pass (global registry functionality)
+✅ Memory usage remains acceptable (<5 KB overhead)
+✅ Backward compatible - existing user code continues to work
+✅ Matches industry-standard ORM patterns (TypeORM, Sequelize, Mongoose)
+
+## Conclusion
+
+MasterRecord v0.3.38 successfully eliminates confusing CLI warnings while preserving the ability to detect genuine bugs in user code. The global model registry provides a clean, intuitive developer experience that matches industry-standard ORM patterns.
+
+**User Impact:**
+- ✅ Clean CLI output during migration generation
+- ✅ Clear guidance when actual bugs exist (warns once on first instance)
+- ✅ No code changes required (automatic improvement)
+- ✅ Better developer experience overall

package/SQLLiteEngine.js

@@ -1126,6 +1126,14 @@ class SQLLiteEngine {
             if(column.indexOf("__") === -1 ){
                 var fieldColumn = fields[column];
 
+                // 🔥 FIX: For belongsTo relationships, also check the foreignKey field name
+                // Users can set either orgRole.User = obj OR orgRole.user_id = 2
+                if((fieldColumn === undefined || fieldColumn === null) &&
+                   modelEntity[column].relationshipType === "belongsTo" &&
+                   modelEntity[column].foreignKey) {
+                    fieldColumn = fields[modelEntity[column].foreignKey];
+                }
+
                 if((fieldColumn !== undefined && fieldColumn !== null ) && typeof(fieldColumn) !== "object"){
                     // 🔥 Apply toDatabase transformer before validation
                     try {