@friggframework/core 2.0.0--canary.461.8cf93ae.0 → 2.0.0--canary.474.aa465e4.0

package/database/MONGODB_TRANSACTION_FIX.md

@@ -0,0 +1,198 @@
+# MongoDB Transaction Namespace Fix
+
+## Problem
+
+The encryption health check was failing with the following error:
+
+```
+Cannot create namespace frigg.Credential in multi-document transaction.
+Error code: 263
+```
+
+### Root Cause
+
+MongoDB does not allow creating collections (namespaces) inside multi-document transactions. When Prisma tries to create a document in a collection that doesn't exist yet, MongoDB needs to implicitly create the collection. If this happens inside a transaction context, MongoDB throws error code 263.
+
+### Technical Details
+
+- **MongoDB Constraint**: Collections must exist before being used in multi-document transactions
+- **Prisma Behavior**: Prisma may implicitly use transactions for certain operations
+- **Impact**: Health checks fail on fresh databases or when collections haven't been created yet
+
+## Solution
+
+**Implemented a comprehensive schema initialization system that ensures all collections exist at application startup.**
+
+### Architectural Approach
+
+Rather than checking before each individual database operation, we take a **systematic, fail-fast approach**:
+
+1. **Parse Prisma Schema**: Extract all collection names from the Prisma schema definition
+2. **Initialize at Startup**: Create all collections when the database connection is established
+3. **Fail Fast**: If there are database issues, the application fails immediately at startup rather than during runtime operations
+4. **Idempotent**: Safe to run multiple times - only creates collections that don't exist
+
+This follows the **"fail fast"** principle and ensures consistent state across all application instances.
+
+### Changes Made
+
+1. **Created MongoDB Schema Initialization** (`packages/core/database/utils/mongodb-schema-init.js`)
+   - `initializeMongoDBSchema()` - Ensures all Prisma collections exist at startup
+   - `getPrismaCollections()` - Returns list of all Prisma collection names
+   - `PRISMA_COLLECTIONS` - Constant array of all 13 Prisma collections
+   - Only runs for MongoDB (skips PostgreSQL)
+   - Fails fast if database not connected
+
+2. **Created MongoDB Collection Utilities** (`packages/core/database/utils/mongodb-collection-utils.js`)
+   - `ensureCollectionExists(collectionName)` - Ensures a single collection exists
+   - `ensureCollectionsExist(collectionNames)` - Batch creates multiple collections
+   - `collectionExists(collectionName)` - Checks if a collection exists
+   - Handles race conditions gracefully (NamespaceExists errors)
+
+3. **Integrated into Database Connection** (`packages/core/database/prisma.js`)
+   - Modified `connectPrisma()` to call `initializeMongoDBSchema()` after connection
+   - Ensures all collections exist before application handles requests
+
+4. **Updated Health Check Repository** (`packages/core/database/repositories/health-check-repository-mongodb.js`)
+   - Removed per-operation collection existence checks
+   - Added documentation noting schema is initialized at startup
+
+5. **Added Comprehensive Tests**
+   - `mongodb-schema-init.test.js` - Tests schema initialization system
+   - `mongodb-collection-utils.test.js` - Tests collection utility functions
+   - Tests error handling, race conditions, and edge cases
+
+### Implementation Flow
+
+```javascript
+// 1. Application startup - connect to database
+await connectPrisma();
+  └─> await initializeMongoDBSchema();
+       └─> await ensureCollectionsExist([
+            'User', 'Token', 'Credential', 'Entity',
+            'Integration', 'IntegrationMapping', 'Process',
+            'Sync', 'DataIdentifier', 'Association',
+            'AssociationObject', 'State', 'WebsocketConnection'
+           ]);
+
+// 2. Now all collections exist - safe to handle requests
+// No per-operation checks needed!
+await prisma.credential.create({ data: {...} }); // Works without namespace error
+```
+
+## Best Practices Followed
+
+1. **Domain-Driven Design**: Created reusable utility module for MongoDB-specific concerns
+2. **Hexagonal Architecture**: Infrastructure concerns (schema initialization) handled in infrastructure layer
+3. **Test-Driven Development**: Added comprehensive tests for all utility functions
+4. **Fail Fast Principle**: Database issues discovered at startup, not during runtime
+5. **Idempotency**: Safe to run multiple times across multiple instances
+6. **Error Handling**: Graceful degradation on race conditions and errors
+7. **Documentation**: Inline comments, JSDoc, and comprehensive documentation
+
+## Benefits
+
+### Immediate Benefits
+- ✅ Fixes encryption health check failures on fresh databases
+- ✅ Prevents transaction namespace errors across **all** Prisma operations
+- ✅ No per-operation overhead - collections created once at startup
+- ✅ Fail fast - database issues discovered immediately at startup
+- ✅ Idempotent - safe to run multiple times and across multiple instances
+
+### Architectural Benefits
+- ✅ **Clean separation of concerns**: Schema initialization is infrastructure concern, handled at startup
+- ✅ **Follows DDD/Hexagonal Architecture**: Infrastructure layer handles database setup, repositories focus on business operations
+- ✅ **Consistent across all environments**: Dev, test, staging, production all follow same pattern
+- ✅ **No repository-level checks needed**: All repositories benefit automatically
+- ✅ **Well-tested and documented**: Comprehensive test coverage and documentation
+
+### Operational Benefits
+- ✅ **Predictable startup**: Clear logging of schema initialization
+- ✅ **Zero runtime overhead**: Collections created once, not on every operation
+- ✅ **Production-ready**: Handles race conditions, errors, and edge cases gracefully
+
+## Design Decisions
+
+### Why Initialize at Startup?
+
+We considered two approaches:
+
+**❌ Per-Operation Checks (Initial approach)**
+```javascript
+async createCredential(data) {
+    await ensureCollectionExists('Credential'); // Check every time
+    return await prisma.credential.create({ data });
+}
+```
+- Pros: Guarantees collection exists before each operation
+- Cons: Runtime overhead, repeated checks, scattered logic
+
+**✅ Startup Initialization (Final approach)**
+```javascript
+// Once at startup
+await connectPrisma(); // Initializes all collections
+
+// All operations just work
+async createCredential(data) {
+    return await prisma.credential.create({ data }); // No checks needed
+}
+```
+- Pros: Zero runtime overhead, centralized logic, fail fast, consistent
+- Cons: Requires database connection at startup (already required)
+
+### Benefits of Startup Approach
+
+1. **Performance**: Collections created once vs. checking before every operation
+2. **Simplicity**: No conditional logic in repositories
+3. **Reliability**: Fail fast at startup if database has issues
+4. **Maintainability**: Single source of truth for schema initialization
+5. **DDD Alignment**: Infrastructure concerns handled in infrastructure layer
+
+## Logging Output
+
+When the application starts, you'll see clear logging:
+
+```
+Initializing MongoDB schema - ensuring all collections exist...
+Created MongoDB collection: Credential
+MongoDB schema initialization complete - 13 collections verified (45ms)
+```
+
+On subsequent startups (collections already exist):
+```
+Initializing MongoDB schema - ensuring all collections exist...
+MongoDB schema initialization complete - 13 collections verified (12ms)
+```
+
+## References
+
+- [Prisma Issue #8305](https://github.com/prisma/prisma/issues/8305) - MongoDB "Cannot create namespace" error
+- [Mongoose Issue #6699](https://github.com/Automattic/mongoose/issues/6699) - Similar issue in Mongoose
+- [MongoDB Transactions Documentation](https://www.mongodb.com/docs/manual/core/transactions/#transactions-and-operations) - Operations allowed in transactions
+- [Prisma MongoDB Guide](https://www.prisma.io/docs/guides/database/mongodb) - Using Prisma with MongoDB
+
+## Future Considerations
+
+### Automatic Schema Sync
+Consider enhancing the system to:
+- Parse Prisma schema file dynamically to extract collection names
+- Auto-detect schema changes and create new collections
+- Provide CLI command for manual schema initialization
+
+### Migration Support
+For production deployments with existing data:
+- Document migration procedures for new collections
+- Consider pre-migration scripts for blue-green deployments
+- Add health check for schema initialization status
+
+### Multi-Database Support
+The system already handles:
+- ✅ MongoDB - Full schema initialization
+- ✅ PostgreSQL - Skips initialization (uses Prisma migrations)
+- Consider adding explicit migration support for DocumentDB-specific features
+
+### Index Creation
+Future enhancement could also create indexes at startup:
+- Parse Prisma schema for `@@index` directives
+- Create indexes if they don't exist
+- Provide index health checks

package/database/utils/prisma-runner.js

@@ -18,12 +18,9 @@ const chalk = require('chalk');
 function getPrismaSchemaPath(dbType, projectRoot = process.cwd()) {
     // Try multiple locations for the schema file
     // Priority order:
-    // 1. Lambda layer path (where the schema actually exists in deployed Lambda)
-    // 2. Local node_modules (where @friggframework/core is installed - production scenario)
-    // 3. Parent node_modules (workspace/monorepo setup)
+    // 1. Local node_modules (where @friggframework/core is installed - production scenario)
+    // 2. Parent node_modules (workspace/monorepo setup)
     const possiblePaths = [
-        // Lambda layer path - this is where the schema actually exists in deployed Lambda
-        `/opt/nodejs/node_modules/generated/prisma-${dbType}/schema.prisma`,
         // Check where Frigg is installed via npm (production scenario)
         path.join(projectRoot, 'node_modules', '@friggframework', 'core', `prisma-${dbType}`, 'schema.prisma'),
         path.join(projectRoot, '..', 'node_modules', '@friggframework', 'core', `prisma-${dbType}`, 'schema.prisma')

package/package.json

@@ -1,7 +1,7 @@
 {
     "name": "@friggframework/core",
     "prettier": "@friggframework/prettier-config",
-    "version": "2.0.0--canary.461.8cf93ae.0",
+    "version": "2.0.0--canary.474.aa465e4.0",
     "dependencies": {
         "@aws-sdk/client-apigatewaymanagementapi": "^3.588.0",
         "@aws-sdk/client-kms": "^3.588.0",
@@ -38,9 +38,9 @@
         }
     },
     "devDependencies": {
-        "@friggframework/eslint-config": "2.0.0--canary.461.8cf93ae.0",
-        "@friggframework/prettier-config": "2.0.0--canary.461.8cf93ae.0",
-        "@friggframework/test": "2.0.0--canary.461.8cf93ae.0",
+        "@friggframework/eslint-config": "2.0.0--canary.474.aa465e4.0",
+        "@friggframework/prettier-config": "2.0.0--canary.474.aa465e4.0",
+        "@friggframework/test": "2.0.0--canary.474.aa465e4.0",
         "@prisma/client": "^6.17.0",
         "@types/lodash": "4.17.15",
         "@typescript-eslint/eslint-plugin": "^8.0.0",
@@ -80,5 +80,5 @@
     "publishConfig": {
         "access": "public"
     },
-    "gitHead": "8cf93ae0a6e42ae09247d20d4073495aac4971cf"
+    "gitHead": "aa465e49207b1725bc36827e65efdd54ad879337"
 }