@reldens/storage 0.88.0 → 0.90.0

package/CLAUDE.md

@@ -2,6 +2,24 @@
 
 This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
 
+## ⚠️ CRITICAL: The Library Works - Fix The Tests
+
+**This package is battle-tested in hundreds of production projects handling thousands of API calls successfully.**
+
+**Default Assumption: Tests are failing because TEST CODE is incorrect, NOT because library code is broken.**
+
+**Your Primary Task:**
+- Fix test files in `tests/` directory to properly use the library
+- Study working examples from reldens-cms and reldens main project
+- Make all tests pass by correcting how they call the library APIs
+
+**If You Suspect a Library Bug:**
+- Present complete proof with production evidence
+- Wait for user confirmation before proposing any library changes
+- Do NOT modify `lib/` code without explicit approval
+
+---
+
 ## Package Overview
 
 **@reldens/storage** is the database abstraction layer for Reldens. It provides:
@@ -52,6 +70,7 @@ npx reldens-storage-prisma --host=<host> --database=<db> --user=<user> --passwor
 **BaseDriver** (`lib/base-driver.js`):
 - Abstract base class for ORM drivers
 - Provides common interface for all database operations
+- Stores operator mapping for all drivers (`this.operatorsMap`)
 - Key methods (all must be implemented by drivers):
   - CRUD: `create()`, `update()`, `delete()`, `upsert()`
   - Read: `load()`, `loadById()`, `loadAll()`, `loadOne()`
@@ -59,6 +78,9 @@ npx reldens-storage-prisma --host=<host> --database=<db> --user=<user> --passwor
   - Count: `count()`, `countWithRelations()`
   - Query: `rawQuery()`, `executeCustomQuery()`
   - Helpers: `parseRelationsString()`, `isJsonField()`
+- **Operator Mapping**: Maps string operators to SQL operators
+  - `GT` → `>`, `GTE` → `>=`, `LT` → `<`, `LTE` → `<=`, `NE` → `!=`, `EQ` → `=`
+  - Available to all drivers via `this.operatorsMap`
 
 **BaseDataServer** (`lib/base-data-server.js`):
 - Abstract base class for data servers
@@ -93,8 +115,11 @@ npx reldens-storage-prisma --host=<host> --database=<db> --user=<user> --passwor
   - Uses `withGraphFetched()` for eager loading
   - Supports relation modifiers (orderBy, limit)
   - JSON field handling with `castText()` for LIKE queries
-  - Filter operators: OR, IN, NOT, LIKE
+  - **Filter operators** (case-insensitive): OR, IN, NOT, LIKE, GT, GTE, LT, LTE, NE, EQ
+  - **Operator conversion**: String operators converted to uppercase before processing
+  - **Nested filtering**: AND/OR operators support nested conditions with proper SQL grouping
   - Methods: `appendFilters()`, `appendRelationsToQuery()`
+  - **Upsert behavior**: Check if record exists → update if found, create if not
 
 **MikroORM Driver** (`lib/mikro-orm/`):
 - `mikro-orm-driver.js`: Driver implementation
@@ -122,6 +147,8 @@ npx reldens-storage-prisma --host=<host> --database=<db> --user=<user> --passwor
   - Binary targets configuration
   - Data proxy support
   - Windows permission error handling
+  - **Prisma.DbNull handling**: PrismaDataServer passes `Prisma.DbNull` to driver, which passes it to type caster
+  - **Type caster isolation**: PrismaTypeCaster never requires `@prisma/client` directly, receives `prismaDbNull` as prop
 
 ### Generators
 
@@ -164,7 +191,14 @@ npx reldens-storage-prisma --host=<host> --database=<db> --user=<user> --passwor
 
 **EntitiesTranslationsGeneration** (`lib/generators/entities-translations-generation.js`):
 - Generates `entities-translations.js` file
-- Creates i18n translation keys for entities
+- Creates i18n translation keys for entities and fields
+- Generates two types of translations:
+  - Table labels: Human-readable entity names
+  - Entity-specific field translations: Field labels per entity
+- Methods:
+  - `getTranslationLabels()`: Generates table name translations
+  - `getFieldTranslations()`: Generates entity-specific field translations
+  - `formatFieldName()`: Converts field names to human-readable labels (e.g., `owner_id` → `Owner ID`)
 
 **BaseGenerator** (`lib/generators/base-generator.js`):
 - Base class for all generators
@@ -228,7 +262,9 @@ All generated files are created in the **generated-entities/** directory:
 
 **Configuration Files:**
 - entities-config.js (entity relation configuration)
-- entities-translations.js (translation keys)
+- entities-translations.js (translation keys with two sections):
+  - `labels`: Table name translations (e.g., `'skills_class_path': 'Class Paths'`)
+  - `fields`: Entity-specific field translations (e.g., `'skills_class_path': {'id': 'ID', 'key': 'Key'}`)
 
 ## Relation Keys Pattern
 
@@ -472,3 +508,96 @@ if(!prismaClient){
 **Used By:**
 - `bin/reldens-storage.js`: CLI entity generator
 - External packages: `@reldens/cms` CLI tools (update-password, generate-entities, generate-sitemap)
+
+## Test Suite Architecture
+
+**CRITICAL:** For detailed test suite architecture, execution flow, and lifecycle hooks, see: `.claude/test-architecture.md`
+
+### The Golden Rule: Connect Once, Test Many
+
+**DO NOT** create database connections, tables, or entities in `beforeEach()` hooks. These operations happen **ONCE per driver** in `before()` hooks.
+
+**Correct test lifecycle:**
+1. `before()` hook - RUNS ONCE per driver:
+   - Connect to database
+   - Create tables via SQL
+   - Generate entities (introspects database, creates models)
+   - For Prisma: Generate schema → Generate client → Reconnect
+   - Get repository references
+
+2. `beforeEach()` hook - RUNS BEFORE EACH TEST:
+   - DELETE data from tables (fast cleanup)
+   - Never drop/recreate tables
+   - Never reconnect to database
+
+3. `after()` hook - RUNS ONCE per driver:
+   - DROP all tables
+   - Disconnect from database
+
+### Test Database Operations
+
+- **cleanDatabase()**: Uses DELETE statements to clear data between tests (fast)
+- **dropTestTables()**: Uses DROP TABLE statements for final cleanup (slow, runs once)
+- **executeRawSQL()**: Creates tables from SQL schema file (slow, runs once)
+
+### DataServer Flow
+
+All three drivers follow this pattern:
+
+```
+1. new DataServer({config, rawEntities})
+   ↓
+2. await dataServer.connect()
+   ↓
+3. await executeRawSQL(dataServer, schemaSql)  ← Tables must exist before next step
+   ↓
+4. await generateTestEntities(dataServer, driverName)
+   ├─ Introspects database schema
+   ├─ Creates models from tables
+   ├─ [Prisma only]: Generate schema + client + reconnect
+   └─ Calls dataServer.generateEntities()
+   ↓
+5. dataServer.getEntity('entityName')  ← Returns repository
+```
+
+### Why Tables Must Exist Before Entity Generation
+
+- `generateTestEntities()` calls `fetchEntitiesFromDatabase()`
+- This queries MySQL `information_schema` for actual table structures
+- Without tables, introspection returns empty/null
+- Entity generation fails without table metadata
+
+### Prisma Special Handling
+
+Prisma requires additional subprocess steps during entity generation:
+
+1. Generate `schema.prisma` file from database (subprocess: `npx reldens-storage-prisma`)
+2. Generate PrismaClient code (subprocess: `npx prisma generate`)
+3. Disconnect old client
+4. Clear Node.js module cache
+5. Reconnect with newly generated client
+6. Now PrismaClient has models for the current database schema
+
+This happens ONCE during the `before()` hook inside `generateTestEntities()`.
+
+**For critical patterns including:**
+- Why tests use dynamic models (not generated models)
+- Prisma client loading pattern
+- Common pitfalls and solutions
+
+**See:** `.claude/test-architecture.md` - Critical Patterns section
+
+### Test Files
+
+**Integration tests:**
+- `tests/integration/test-cross-driver-compatibility.js`: Full CRUD cycle for all 3 drivers
+- `tests/integration/test-nested-filters.js`: Complex filter syntax (AND/OR/NOT/IN/LIKE)
+- `tests/integration/test-relations.js`: Relation loading and nested relations
+
+**Test utilities:**
+- `tests/utils/test-helpers.js`: Database setup, entity generation, cleanup utilities
+- `tests/utils/test-runner.js`: Test framework with suite/group/test methods, uses Logger for output
+
+**All integration test files use the correct lifecycle pattern as of 2026-01-18.**
+
+For complete details on test architecture, common issues, performance comparison, and troubleshooting, see `.claude/test-architecture.md`.

package/bin/reldens-storage.js

@@ -1,4 +1,4 @@
-#!/usr/bin/env node
+#!/usr/bin/env node
 
 /**
  *

package/lib/base-driver.js

@@ -21,6 +21,14 @@ class BaseDriver
         this.sortBy = sc.get(props, 'sortBy', false);
         this.sortDirection = sc.get(props, 'sortDirection', 'ASC');
         this.entityConfig = sc.get(props, 'entityConfig', {});
+        this.operatorsMap = {
+            'GT': '>',
+            'GTE': '>=',
+            'LT': '<',
+            'LTE': '<=',
+            'NE': '!=',
+            'EQ': '='
+        };
     }
 
     databaseName()

package/lib/entities-generator.js

@@ -402,9 +402,7 @@ class EntitiesGenerator
             Logger.critical('Failed to fetch driver key.');
             return false;
         }
-        if('objection-js' === driverKey){
-            this.modelsGeneration.setAllTablesData(tables);
-        }
+        this.modelsGeneration.setAllTablesData(tables);
         let tablesToGenerate = this.filterTablesToGenerate(tables, driverKey);
         if(0 === Object.keys(tablesToGenerate).length && !this.isOverride){
             return true;

package/lib/entity-templates/entities-translations.template

@@ -7,5 +7,8 @@
 module.exports.entitiesTranslations = {
     labels: {
         {{labels}}
+    },
+    fields: {
+        {{fields}}
     }
 };

package/lib/entity-templates/mikro-orm-model.template

@@ -25,11 +25,12 @@ class {{modelClassName}}
 
 const schema = new EntitySchema({
     class: {{modelClassName}},
+    tableName: '{{tableName}}',
     properties: {
         {{entityPropertiesDefinition}}
     },
 });
-
+{{fkMappingsAttachment}}
 module.exports = {
     {{modelClassName}},
     entity: {{modelClassName}},

package/lib/generators/entities-generation.js

@@ -75,15 +75,26 @@ class EntitiesGeneration extends BaseGenerator
             return false;
         }
         Logger.info('Generated entity file: '+fileName);
+        let properties = this.buildPropertiesObject(tableData.columns);
         this.generatedEntities[tableName] = {
             entityClassName,
             entityFileName: fileName,
             tableName,
-            titleProperty
+            titleProperty,
+            properties
         };
         return true;
     }
 
+    buildPropertiesObject(columns)
+    {
+        let properties = {};
+        for(let columnName of Object.keys(columns)){
+            properties[columnName] = true;
+        }
+        return properties;
+    }
+
     detectPrimaryKeyColumn(columns)
     {
         for(let columnName of Object.keys(columns)){

package/lib/generators/entities-translations-generation.js

@@ -37,10 +37,10 @@ class EntitiesTranslationsGeneration extends BaseGenerator
             Logger.critical('Failed to read entities translations template file: '+this.templatePath);
             return false;
         }
-        let translationsContent = translationsTemplateContent.replace(
-            /{{labels}}/g,
-            this.getTranslationLabels(Object.assign({}, existingEntities, generatedEntities))
-        );
+        let allEntities = Object.assign({}, existingEntities, generatedEntities);
+        let translationsContent = translationsTemplateContent
+            .replace(/{{labels}}/g, this.getTranslationLabels(allEntities))
+            .replace(/{{fields}}/g, this.getFieldTranslations(allEntities));
         if(!FileHandler.writeFile(this.entitiesTranslationsPath, translationsContent)){
             Logger.critical('Failed to write entities translations file: '+this.entitiesTranslationsPath);
             return false;
@@ -130,6 +130,42 @@ class EntitiesTranslationsGeneration extends BaseGenerator
         return labels.join(',\n        ');
     }
 
+    getFieldTranslations(allEntities)
+    {
+        let fieldsTranslations = [];
+        for(let tableName of Object.keys(allEntities)){
+            let entity = allEntities[tableName];
+            if(!sc.hasOwn(entity, 'properties')){
+                continue;
+            }
+            let entityFields = [];
+            for(let fieldName of Object.keys(entity.properties)){
+                let formattedName = this.formatFieldName(fieldName);
+                entityFields.push('\''+fieldName+'\': \''+formattedName+'\'');
+            }
+            if(0 === entityFields.length){
+                continue;
+            }
+            fieldsTranslations.push(
+                '\''+tableName+'\': {\n            '+entityFields.join(',\n            ')+'\n        }'
+            );
+        }
+        return fieldsTranslations.join(',\n        ');
+    }
+
+    formatFieldName(fieldName)
+    {
+        let formatted = fieldName
+            .split('_')
+            .map(word => word.charAt(0).toUpperCase()+word.slice(1))
+            .join(' ');
+        formatted = formatted.replace(/\sId(\s|$)/g, ' ID$1');
+        formatted = formatted.replace(/\sAt(\s|$)/g, ' At$1');
+        formatted = formatted.replace(/\sKey(\s|$)/g, ' Key$1');
+        formatted = formatted.replace(/^Id(\s|$)/g, 'ID$1');
+        return formatted;
+    }
+
 }
 
 module.exports.EntitiesTranslationsGeneration = EntitiesTranslationsGeneration;

package/lib/generators/models-generation.js

@@ -20,6 +20,12 @@ class ModelsGeneration extends BaseGenerator
         this.generatedModels = {};
         this.allTablesData = {};
         this.removeIdFromMultipleRelations = sc.get(props, 'removeIdFromMultipleRelations', true);
+        this.mikroOrmRelationKindMap = {
+            'BelongsToOneRelation': 'm:1',
+            'HasOneRelation': '1:1',
+            'HasManyRelation': '1:m',
+            'ManyToManyRelation': 'm:n'
+        };
     }
 
     async generateModelFile(tableName, tableData, driverKey, entityInfo, relationMetadata = {})
@@ -40,7 +46,12 @@ class ModelsGeneration extends BaseGenerator
             .map(columnName => 'this.'+columnName+' = '+columnName+';')
             .join('\n        ');
         let modelRelations = this.generateModelRelations(tableName, tableData, relationMetadata, driverKey);
-        let entityPropertiesDefinition = this.getEntityPropertiesDefinition(tableData.columns, driverKey);
+        let entityPropertiesDefinition = this.getEntityPropertiesDefinition(tableData.columns, driverKey, tableName, tableData);
+        let fkMappingsJson = this.generateFkMappings(tableName, tableData, driverKey);
+        let fkMappingsAttachment = '';
+        if('mikro-orm' === driverKey && fkMappingsJson){
+            fkMappingsAttachment = 'schema._fkMappings = '+fkMappingsJson+';';
+        }
         let idColumn = this.generateIdColumn(tableData.columns, driverKey);
         let replacements = {
             modelClassName,
@@ -49,6 +60,7 @@ class ModelsGeneration extends BaseGenerator
             modelPropertiesConstructor,
             modelRelations,
             entityPropertiesDefinition,
+            fkMappingsAttachment,
             idColumn
         };
         let modelContent = this.applyReplacements(modelTemplateContent, replacements);
@@ -461,24 +473,85 @@ class ModelsGeneration extends BaseGenerator
         return 'HasManyRelation';
     }
 
-    getEntityPropertiesDefinition(columns, driverKey)
+    getEntityPropertiesDefinition(columns, driverKey, tableName = null, tableData = null)
     {
         if('mikro-orm' === driverKey){
             let entityProps = [];
+            if(!tableName || !tableData){
+                for(let columnName of Object.keys(columns)){
+                    let column = columns[columnName];
+                    let isPrimary = 'PRI' === column.key;
+                    let type = TypeMapper.mapDbTypeToJsType(column.type);
+                    let hasDefault = null !== column.default && undefined !== column.default;
+                    let propDef = columnName+': { type: \''+type+'\'';
+                    if(isPrimary){
+                        propDef += ', primary: true';
+                    }
+                    if(column.nullable || hasDefault){
+                        propDef += ', nullable: true';
+                    }
+                    propDef += ' }';
+                    entityProps.push(propDef);
+                }
+                return entityProps.join(',\n        ');
+            }
+            let relations = this.detectObjectionJsRelations(tableName, tableData);
+            let fkColumnsUsedByRelations = {};
+            for(let relationKey of Object.keys(relations)){
+                let relation = relations[relationKey];
+                if(relation.fromColumn){
+                    fkColumnsUsedByRelations[relation.fromColumn] = true;
+                }
+            }
             for(let columnName of Object.keys(columns)){
                 let column = columns[columnName];
+                if(fkColumnsUsedByRelations[columnName]){
+                    let type = TypeMapper.mapDbTypeToJsType(column.type);
+                    let propDef = columnName+': { type: \''+type+'\', persist: false }';
+                    entityProps.push(propDef);
+                    continue;
+                }
                 let isPrimary = 'PRI' === column.key;
                 let type = TypeMapper.mapDbTypeToJsType(column.type);
+                let hasDefault = null !== column.default && undefined !== column.default;
                 let propDef = columnName+': { type: \''+type+'\'';
                 if(isPrimary){
                     propDef += ', primary: true';
                 }
-                if(column.nullable){
+                if(column.nullable || hasDefault){
                     propDef += ', nullable: true';
                 }
                 propDef += ' }';
                 entityProps.push(propDef);
             }
+            for(let relationKey of Object.keys(relations)){
+                let relation = relations[relationKey];
+                let relatedModelClass = sc.capitalizedCamelCase(relation.referencedTable)+'Model';
+                let kind = sc.get(this.mikroOrmRelationKindMap, relation.relationType, 'm:1');
+                let propDef = relationKey+': {\n';
+                propDef += '            kind: \''+kind+'\',\n';
+                propDef += '            entity: \''+relatedModelClass+'\'';
+                if(relation.fromColumn){
+                    propDef += ',\n            joinColumn: \''+relation.fromColumn+'\'';
+                }
+                propDef += '\n        }';
+                entityProps.push(propDef);
+            }
+            let reverseRelations = this.detectReverseObjectionJsRelations(tableName);
+            for(let relationKey of Object.keys(reverseRelations)){
+                let relation = reverseRelations[relationKey];
+                let relatedModelClass = sc.capitalizedCamelCase(relation.referencingTable)+'Model';
+                let kind = sc.get(this.mikroOrmRelationKindMap, relation.relationType, '1:m');
+                let mappedBy = this.findMappedByKey(tableName, relation.referencingTable, relation);
+                let propDef = relationKey+': {\n';
+                propDef += '            kind: \''+kind+'\',\n';
+                propDef += '            entity: \''+relatedModelClass+'\'';
+                if(mappedBy){
+                    propDef += ',\n            mappedBy: \''+mappedBy+'\'';
+                }
+                propDef += '\n        }';
+                entityProps.push(propDef);
+            }
             return entityProps.join(',\n        ');
         }
         if('prisma' === driverKey){
@@ -502,6 +575,52 @@ class ModelsGeneration extends BaseGenerator
         return '';
     }
 
+    findMappedByKey(currentTable, referencingTable, reverseRelation)
+    {
+        if(!this.allTablesData[referencingTable]){
+            return 'related_'+currentTable;
+        }
+        let referencingTableData = this.allTablesData[referencingTable];
+        let forwardRelations = this.detectObjectionJsRelations(referencingTable, referencingTableData);
+        for(let forwardKey of Object.keys(forwardRelations)){
+            let forward = forwardRelations[forwardKey];
+            if(forward.referencedTable === currentTable && forward.toColumn === reverseRelation.fromColumn){
+                return forwardKey;
+            }
+        }
+        return 'related_'+currentTable;
+    }
+
+    generateFkMappings(tableName, tableData, driverKey)
+    {
+        if('mikro-orm' !== driverKey){
+            return '';
+        }
+        if(!tableName || !tableData){
+            return '';
+        }
+        let fkMappings = {};
+        let referenceCounts = this.countReferencesPerTable(tableData);
+        for(let columnName of Object.keys(tableData.columns)){
+            let column = tableData.columns[columnName];
+            if(!sc.hasOwn(column, 'referencedTable')){
+                continue;
+            }
+            let relationKey = this.generateForwardRelationKey(column.referencedTable, columnName, referenceCounts);
+            let relatedModelClass = sc.capitalizedCamelCase(column.referencedTable)+'Model';
+            fkMappings[columnName] = {
+                relationKey: relationKey,
+                entityName: relatedModelClass,
+                referencedColumn: column.referencedColumn,
+                nullable: column.nullable || false
+            };
+        }
+        if(0 === Object.keys(fkMappings).length){
+            return '';
+        }
+        return JSON.stringify(fkMappings, null, 4);
+    }
+
     generateRegisteredModelsFile(entitiesInfo, existingModels, driverKey, templatePath)
     {
         if(!FileHandler.exists(templatePath)){
@@ -541,8 +660,9 @@ class ModelsGeneration extends BaseGenerator
             }
             let modelClassName = entity.modelClassName || sc.capitalizedCamelCase(tableName)+'Model';
             let modelFileName = entity.modelFileName || sc.kebabCase(tableName)+'-model.js';
+            let variableName = 'mikro-orm' !== driverKey ? '{ '+modelClassName+' }' : sc.camelCase(tableName)+'Model';
             registeredModels.push(
-                'const { '+modelClassName+' } = require(\'./'+modelFileName.replace('.js', '')+'\');'
+                'const '+variableName+' = require(\'./'+modelFileName.replace('.js', '')+'\');'
             );
         }
         return registeredModels.join('\n');
@@ -559,7 +679,8 @@ class ModelsGeneration extends BaseGenerator
                 }
             }
             let modelClassName = entity.modelClassName || sc.capitalizedCamelCase(tableName)+'Model';
-            registeredEntitiesObject.push(sc.camelCase(tableName)+': '+modelClassName);
+            let variableName = 'mikro-orm' === driverKey ? sc.camelCase(tableName)+'Model' : modelClassName;
+            registeredEntitiesObject.push(sc.camelCase(tableName)+': '+variableName);
         }
         return registeredEntitiesObject.join(',\n    ');
     }

package/lib/mikro-orm/mikro-orm-data-server.js

@@ -10,7 +10,7 @@ const { MySQLTablesProvider } = require('../mysql-tables-provider');
 const { MikroORM } = require('@mikro-orm/core');
 const { MySqlDriver } = require('@mikro-orm/mysql');
 const { MongoDriver } = require('@mikro-orm/mongodb');
-const { ErrorManager, Logger, sc } = require('@reldens/utils');
+const { Logger, sc } = require('@reldens/utils');
 
 class MikroOrmDataServer extends BaseDataServer
 {
@@ -28,6 +28,27 @@ class MikroOrmDataServer extends BaseDataServer
         this.warnWhenNoEntities = Boolean(props.warnWhenNoEntities);
     }
 
+    getEntitiesToRegister()
+    {
+        if(this.entitiesPath){
+            //Logger.debug('MikroORM using entitiesPath');
+            return this.entitiesPath;
+        }
+        if(0 < Object.keys(this.entities).length){
+            //Logger.debug('MikroORM using this.entities: '+Object.keys(this.entities).join(', '));
+            return this.entities;
+        }
+        if(!this.rawEntities){
+            //Logger.debug('MikroORM has no entities!');
+            return [];
+        }
+        let entities = Object.values(this.rawEntities);
+        //Logger.debug('MikroORM extracting from rawEntities, count: '+entities.length);
+        return entities.map(rawEntity => {
+            return (rawEntity && rawEntity.schema) ? rawEntity.schema : rawEntity;
+        });
+    }
+
     async connect()
     {
         if(this.initialized){
@@ -39,15 +60,7 @@ class MikroOrmDataServer extends BaseDataServer
             clientUrl: this.connectString,
             allowGlobalContext: true,
         };
-        if(this.rawEntities){
-            providedSetup.entities = Object.values(this.rawEntities);
-        }
-        if(0 < Object.keys(this.entities).length){
-            providedSetup.entities = this.entities;
-        }
-        if(this.entitiesPath){
-            providedSetup.entities = this.entitiesPath;
-        }
+        providedSetup.entities = this.getEntitiesToRegister();
         providedSetup.discovery = {warnWhenNoEntities: this.warnWhenNoEntities};
         this.orm = await MikroORM.init(providedSetup);
         this.initialized = Date.now();
@@ -60,6 +73,7 @@ class MikroOrmDataServer extends BaseDataServer
             return false;
         }
         await this.orm.close();
+        this.initialized = null;
         return true;
     }