@solidstarters/solid-core 1.2.160 → 1.2.163

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@solidstarters/solid-core",
-  "version": "1.2.160",
+  "version": "1.2.163",
   "description": "This module is a NestJS module containing all the required core providers required by a Solid application",
   "main": "dist/index.js",
   "types": "dist/index.d.ts",
@@ -68,6 +68,7 @@
     "pg": "^8.11.3",
     "pluralize": "^8.0.0",
     "puppeteer": "^23.2.0",
+    "r2r-js": "^0.4.43",
     "reflect-metadata": "^0.1.13",
     "rxjs": "^7.8.1",
     "swagger-ui-express": "^5.0.0",

package/src/commands/ingest-rag-chunking-strategy-for.md

@@ -0,0 +1,224 @@
+# Optimized RAG Chunking Strategy for Solid No-Code Platform Metadata
+
+## Executive Summary
+
+This document outlines an intelligent chunking strategy for ingesting Solid no-code platform metadata into R2R vector database to enable effective retrieval for LLM-powered MCP tools.
+
+## System Architecture Context
+
+**Platform**: Solid no-code platform with metadata-driven code generation
+**RAG System**: R2R (https://r2r-docs.sciphi.ai/) 
+**Integration**: MCP client → MCP server → LLM tools (createModule, createModel, addField, etc.)
+**Workflow**: User chat → tool call → RAG retrieval → LLM context → metadata JSON generation → database seeding → code generation
+
+## Chunking Strategy
+
+### 1. Hierarchical Semantic Chunking
+
+**Primary Strategy**: Create chunks based on logical metadata boundaries while maintaining semantic relationships.
+
+#### Level 1: Module-Level Chunks
+```
+Chunk Type: "module_overview"
+Content: Module metadata + high-level model summaries
+Size: ~500-800 tokens
+Purpose: Provide context for module-level operations
+```
+
+#### Level 2: Model-Level Chunks  
+```
+Chunk Type: "model_definition"
+Content: Complete model with all fields + relationships
+Size: ~800-1200 tokens per model
+Purpose: Support model creation/modification operations
+```
+
+#### Level 3: Field-Level Chunks
+```
+Chunk Type: "field_specification"
+Content: Individual field + its configuration + usage patterns
+Size: ~200-400 tokens per field
+Purpose: Enable precise field-level modifications
+```
+
+#### Level 4: Cross-Reference Chunks
+```
+Chunk Type: "relationship_mapping"
+Content: Relations between models + dependency chains
+Size: ~300-600 tokens
+Purpose: Maintain referential integrity during modifications
+```
+
+### 2. Functional Domain Chunks
+
+#### UI/UX Configuration Chunks
+```
+Chunk Type: "ui_configuration"
+Content: Views + layouts + actions + menus (grouped by feature)
+Size: ~600-1000 tokens
+Purpose: Support UI generation and modification
+```
+
+#### Security & Permissions Chunks
+```
+Chunk Type: "security_rules"
+Content: Roles + permissions + security rules (grouped)
+Size: ~400-800 tokens
+Purpose: Maintain security context during operations
+```
+
+#### Integration Chunks
+```
+Chunk Type: "system_integration"
+Content: Email/SMS templates + media providers + scheduled jobs
+Size: ~300-600 tokens per integration type
+Purpose: Support system-level configurations
+```
+
+### 3. Chunk Metadata Enhancement
+
+Each chunk should include structured metadata:
+
+```json
+{
+  "chunk_id": "unique_identifier",
+  "chunk_type": "model_definition|field_specification|ui_configuration|etc",
+  "module_name": "fees-portal",
+  "model_name": "institute|feeType|etc", 
+  "component_names": ["field1", "field2"],
+  "relationships": ["related_model1", "related_model2"],
+  "last_modified": "timestamp",
+  "version": "semantic_version",
+  "dependencies": ["required_chunks"],
+  "keywords": ["domain_specific_terms"]
+}
+```
+
+### 4. Overlap Strategy for Context Preservation
+
+- **Boundary Overlap**: 50-100 token overlap between related chunks
+- **Reference Preservation**: Include parent/child context in each chunk
+- **Relationship Chains**: Maintain connection information across chunk boundaries
+
+## R2R Integration Implementation
+
+### Document Ingestion API Usage
+
+Based on R2R documentation, use these endpoints:
+
+#### Initial Ingestion
+```bash
+POST /v1/documents
+Content-Type: multipart/form-data
+
+# Ingest chunked documents with metadata
+curl -X POST "http://localhost:7272/v1/documents" \
+  -H "Authorization: Bearer <token>" \
+  -F "file=@chunk_module_overview.json" \
+  -F "metadata={\"chunk_type\":\"module_overview\",\"module\":\"fees-portal\"}"
+```
+
+#### Update Existing Documents
+```bash
+PUT /v1/documents/{document_id}
+Content-Type: multipart/form-data
+
+# Update specific chunks when metadata changes
+curl -X PUT "http://localhost:7272/v1/documents/{doc_id}" \
+  -H "Authorization: Bearer <token>" \
+  -F "file=@updated_chunk.json" \
+  -F "metadata={\"version\":\"1.1.0\",\"last_modified\":\"2025-01-15T10:30:00Z\"}"
+```
+
+### CLI Ingestion Command Structure
+
+Create a CLI command for automated ingestion:
+
+```bash
+# Initial ingestion
+solid-cli rag:ingest --metadata-file ./metadata.json --strategy hierarchical
+
+# Update ingestion (incremental)
+solid-cli rag:update --changed-components "institute.fields,views" --strategy differential
+
+# Full re-ingestion
+solid-cli rag:reingest --metadata-file ./metadata.json --force
+```
+
+## Retrieval Optimization
+
+### Query Enhancement Strategies
+
+1. **Semantic Search**: Use embedding-based retrieval for concept matching
+2. **Hybrid Search**: Combine semantic + keyword search for precision
+3. **Contextual Filtering**: Apply metadata filters based on operation type
+
+### MCP Tool Integration
+
+Each MCP tool should specify its retrieval requirements:
+
+```python
+# Example for addField tool
+def addField(model_name: str, field_spec: dict):
+    # Retrieve relevant chunks
+    query = f"model definition {model_name} field types relationships"
+    
+    context_chunks = r2r_client.search(
+        query=query,
+        filters={
+            "chunk_type": ["model_definition", "field_specification"],
+            "model_name": model_name,
+            "module_name": "fees-portal"
+        },
+        limit=5
+    )
+    
+    # Pass context to LLM for field generation
+    return generate_field_json(context_chunks, field_spec)
+```
+
+## Versioning & Change Management
+
+### Differential Updates
+- Track changes at component level (model, field, view, etc.)
+- Update only affected chunks + dependents
+- Maintain version history for rollback capability
+
+### Consistency Checks
+- Validate cross-references after updates
+- Ensure relationship integrity
+- Verify permission consistency
+
+## Performance Considerations
+
+### Chunk Size Optimization
+- Target 200-1200 tokens per chunk for optimal retrieval
+- Balance between context completeness and search precision
+- Monitor retrieval latency and adjust sizes accordingly
+
+### Indexing Strategy
+- Create composite indexes on frequently queried metadata fields
+- Optimize for common query patterns (model + field combinations)
+- Regular index maintenance and optimization
+
+## Implementation Checklist
+
+- [ ] Implement hierarchical chunking algorithm
+- [ ] Create metadata extraction and enhancement pipeline
+- [ ] Set up R2R ingestion endpoints integration
+- [ ] Build CLI commands for ingestion management
+- [ ] Develop retrieval optimization for each MCP tool
+- [ ] Implement versioning and change tracking
+- [ ] Create monitoring and performance metrics
+- [ ] Test with real metadata updates and tool operations
+
+## Success Metrics
+
+1. **Retrieval Accuracy**: >95% relevant context retrieval for tool operations
+2. **Update Efficiency**: <30s for incremental metadata updates
+3. **Context Quality**: LLM tools produce correct JSON with minimal hallucination
+4. **System Performance**: <200ms average retrieval time per tool operation
+
+---
+
+*This strategy ensures your RAG pipeline provides precise, contextual information to your MCP tools while maintaining system performance and update efficiency.*

package/src/commands/ingest.command.ts

@@ -0,0 +1,36 @@
+import { Logger } from '@nestjs/common';
+import { Command, CommandRunner, Option } from 'nest-commander';
+import { SolidRegistry } from 'src/helpers/solid-registry';
+import { IngestMetadataService } from 'src/services/genai/ingest-metadata.service';
+
+interface IngestCommandOptions {
+  module?: string;
+  // seeder?: string;
+}
+
+@Command({ name: 'ingest', description: 'Ingests solid metadata json for all modules deployed in the consuming project' })
+export class IngestCommand extends CommandRunner {
+  private readonly logger = new Logger(IngestCommand.name);
+
+  constructor(
+    private readonly solidRegistry: SolidRegistry,
+    private readonly ingestMetadataService: IngestMetadataService,
+  ) {
+    super();
+  }
+
+  async run(passedParam: string[], options?: IngestCommandOptions): Promise<void> {
+    this.logger.log(`Running the solid ingest for module ${options.module} at ${new Date()}`);
+    await this.ingestMetadataService.ingest();
+  }
+
+  @Option({
+    flags: '-m, --module [module name]',
+    description: 'The seeder to run.',
+    required: false,
+    defaultValue: ''
+  })
+  parseString(val: string): string {
+    return val;
+  }
+}

package/src/commands/refresh-module.command.ts

@@ -35,7 +35,6 @@ export class RefreshModuleCommand extends CommandRunner {
     await this.moduleMetadataService.generateCode(codeGenerationOptions);
   }
 
-
   @Option({
     flags: '-i, --id [module ID]',
     description: 'Module ID from the ss_module_metadata table',

package/src/controllers/service.controller.ts

@@ -1,20 +1,27 @@
 import { Body, Controller, Get, Logger, Post, UseGuards } from '@nestjs/common';
-import { DiscoveryService, MetadataScanner, Reflector } from '@nestjs/core';
 import { Public } from 'src/decorators/public.decorator';
 import { SolidRegistry } from '../helpers/solid-registry';
-import { ApiTags } from '@nestjs/swagger';
+import { ApiBearerAuth, ApiTags } from '@nestjs/swagger';
 import { ThrottlerGuard, SkipThrottle } from '@nestjs/throttler';
+import { ActiveUserData } from 'src/interfaces/active-user-data.interface';
+import { ActiveUser } from 'src/decorators/active-user.decorator';
+import { AiInteractionService } from 'src/services/ai-interaction.service';
+import { MqMessageService } from 'src/services/mq-message.service';
+import { ErrorMapperService } from 'src/helpers/error-mapper.service';
 
 
 @Controller('')
 @ApiTags("Common")
 @UseGuards(ThrottlerGuard)
-@SkipThrottle({ short: true, login: true, burst: true, sustained: true }) //Skip all
+@SkipThrottle({ short: true, login: true, burst: true, sustained: true }) // Skip all
 export class ServiceController {
     private readonly logger = new Logger(ServiceController.name);
 
     constructor(
         private readonly solidRegistry: SolidRegistry,
+        private readonly aiInteractionService: AiInteractionService,
+        private readonly mqMessageService: MqMessageService,
+        private readonly errorMapper: ErrorMapperService
     ) { }
 
     @Public()
@@ -23,6 +30,62 @@ export class ServiceController {
         return { pong: 'v1.0.2' };
     }
 
+    @ApiBearerAuth("jwt")
+    @Get('mcp/ping')
+    async mcpPingPong(@ActiveUser() activeUser: ActiveUserData) {
+        // TODO: do a MCP client invocation, wait for response and return.
+        // If failure then decide shape to return.
+
+        const threadId = `pingPongTxn-${activeUser.sub}`;
+
+        const { queueMessageId, aiInteractionId } = await this.aiInteractionService.triggerMcpClientJob(
+            `Can you do 1 + 1`,
+            activeUser.sub,
+            true,
+            threadId
+        );
+
+        this.logger.debug(`mcp ping pong job triggered: queueMessageId=${queueMessageId}, aiInteractionId=${aiInteractionId}`);
+
+        // Wait up to 2 minutes, start at 500ms poll, back off to max 2s, throw if failed:
+        const result = await this.mqMessageService.waitForTerminalStatus(queueMessageId, {
+            timeoutMs: 2 * 60 * 1000,
+            intervalMs: 500,
+            maxIntervalMs: 2000,
+            throwOnFailure: false,
+        });
+
+        this.logger.debug(`mcp ping pong job finished with stage=${result.stage}`)
+
+        this.logger.debug(`mcp ping pong trying to find genai (child) interaction for aiInteraction for id=${aiInteractionId}`)
+
+        // @ts-ignore
+        const genAiInteractions = await this.aiInteractionService.find({
+            filters: {
+                parentInteraction: {
+                    id: {
+                        $eq: aiInteractionId
+                    }
+                }
+            }
+        });
+
+        const genAiInteraction = genAiInteractions['records'][0];
+        this.logger.debug(genAiInteraction.message);
+
+        this.logger.debug(`identified gen-ai interaction with id=${genAiInteraction.id}`);
+        this.logger.debug(`proceeding with applying the gen-ai interaction`)
+
+        return {
+            mcpPong: 'v1.0.2',
+            genAiInteraction: {
+                status: genAiInteraction.status,
+                errorCode: genAiInteraction.status === 'failed' ? this.errorMapper.mapMessage(genAiInteraction.errorMessage, genAiInteraction.metadata) : '',
+                errorMessage: genAiInteraction.errorMessage,
+            }
+        };
+    }
+
     @Public()
     @SkipThrottle({ short: false, login: true, burst: true, sustained: true }) //Enable the short throttle only
     @Post('seed')

package/src/controllers/test.controller.ts

@@ -1,20 +1,23 @@
 import { Body, Controller, Logger, Post, UploadedFile, UploadedFiles, UseInterceptors } from "@nestjs/common";
 import { AnyFilesInterceptor, FileInterceptor } from "@nestjs/platform-express";
-import { ApiTags } from "@nestjs/swagger";
+import { ApiBearerAuth, ApiTags } from "@nestjs/swagger";
 import { SolidRegistry } from "src/helpers/solid-registry";
 import { Auth } from "src/decorators/auth.decorator";
 import { AuthType } from "src/enums/auth-type.enum";
+import { IngestMetadataService } from "src/services/genai/ingest-metadata.service";
 
 export class SeedData {
   seeder: string;
 }
 
-@Auth(AuthType.None)
 @Controller('test')
 @ApiTags("App Builder")
 export class TestController {
   private readonly logger = new Logger(TestController.name);
-  constructor(private readonly solidRegistry: SolidRegistry) { }
+  constructor(
+    private readonly solidRegistry: SolidRegistry,
+    private readonly ingestMetadataService: IngestMetadataService,
+  ) { }
 
   @Auth(AuthType.None)
   @UseInterceptors(AnyFilesInterceptor())
@@ -23,6 +26,7 @@ export class TestController {
     return { message: 'file uploaded', fileNames: files.map(f => f.originalname), formData: body };
   }
 
+  @Auth(AuthType.None)
   @Post('upload')
   @UseInterceptors(FileInterceptor('file')) // 'file' here is the name of the field in the form
   uploadFile(@UploadedFile() file: Express.Multer.File, @Body() body: any) {
@@ -33,6 +37,14 @@ export class TestController {
     return { filename: file.originalname };
   }
 
+  @ApiBearerAuth("jwt")
+  @Post('mcp/ingest')
+  @UseInterceptors(FileInterceptor('file'))
+  async mcpIngest(@UploadedFile() file: Express.Multer.File, @Body() body: any) {
+    await this.ingestMetadataService.ingest();
+    return { ok: true };
+  }
+
   // @Public()
   // @Post('seed')
   // async seedData(@Body() seedData: any) {

package/src/entities/common.entity.ts

@@ -1,7 +1,10 @@
 import { Column, CreateDateColumn, DeleteDateColumn, ManyToOne, PrimaryGeneratedColumn, UpdateDateColumn } from "typeorm";
 import type { User } from "./user.entity";
+import { Exclude, Expose, Type } from "class-transformer";
 
+@Exclude()
 export abstract class CommonEntity {
+    @Expose()
     @PrimaryGeneratedColumn({ type: 'integer' })
     id: number
 
@@ -17,18 +20,25 @@ export abstract class CommonEntity {
     @Column({ name: "deletedTracker", default: "not-deleted" })
     deletedTracker: string;
 
+    @Expose()
     @Column({ type: "timestamp", name: 'published_at', default: null ,nullable: true})
     publishedAt: Date;
 
+    @Expose()
     @Column({ type: "varchar", name: 'locale_name', default: null })
     localeName: string;
 
+    @Expose()
     @Column({ type: "int", name: 'default_entity_locale_id', default: null })
     defaultEntityLocaleId: number;
 
+    @Expose()
+    @Type( () => require('./user.entity').User?.default ?? require('./user.entity').User )
     @ManyToOne(() => require('./user.entity').User?.default ?? require('./user.entity').User, { onDelete: 'SET NULL', nullable: true })
     createdBy: User;
 
+    @Expose()
+    @Type( () => require('./user.entity').User?.default ?? require('./user.entity').User )
     @ManyToOne(() => require('./user.entity').User?.default ?? require('./user.entity').User, { onDelete: 'SET NULL', nullable: true })
     updatedBy: User;    
 }

package/src/entities/user.entity.ts

@@ -2,72 +2,104 @@ import { CommonEntity } from "src/entities/common.entity"
 import { Entity, Column, Index, JoinTable, ManyToMany, OneToMany, TableInheritance } from "typeorm";
 import { RoleMetadata } from 'src/entities/role-metadata.entity';
 import { UserViewMetadata } from 'src/entities/user-view-metadata.entity'
+import { Exclude, Expose } from "class-transformer";
 
 @Entity("ss_user")
 @TableInheritance({ column: { type: "varchar", name: "type", default: "User" } })
+@Exclude()
 export class User extends CommonEntity {
     @Column({ type: "varchar", nullable: true })
-    fullName: string;
+    @Expose()
+    fullName: string; 
     @Index({ unique: true })
     @Column({ type: "varchar" })
+    @Expose()
     username: string;
     @Index({ unique: true })
     @Column({ type: "varchar", nullable: true })
+    @Expose()
     email: string;
     @Index({ unique: true })
     @Column({ type: "varchar", nullable: true })
+    @Expose()
     mobile: string;
     @Column({ type: "varchar", nullable: true })
+    // don't send to client
     password: string;
     @Column({ type: "boolean", nullable: true, default: true })
+    @Expose()
     forcePasswordChange: boolean = true;
     @Column({ type: "varchar", default: "local" })
+    // don't send to client
     lastLoginProvider: string = "local";
     @Column({ type: "varchar", nullable: true })
+    // don't send to client (test)
     accessCode: string;
     @Column({ type: "varchar", nullable: true })
+    // don't send to client
     googleAccessToken: string;
     @Column({ type: "varchar", nullable: true })
+    // don't send to client
     googleId: string;
     @Column({ type: "varchar", nullable: true })
+    // don't send to client
     googleProfilePicture: string;
     @Column({ type: "boolean", default: true })
+    @Expose()
     active: boolean = true;
     @Column({ type: "timestamp", nullable: true })
+    // don't send to client
     forgotPasswordConfirmedAt: Date;
     @Column({ type: "varchar", nullable: true })
+    // don't send to client
     verificationTokenOnForgotPassword: string;
     @Column({ type: "timestamp", nullable: true })
+    // don't send to client
     verificationTokenOnForgotPasswordExpiresAt: Date;
     @Column({ type: "timestamp", nullable: true })
+    // don't send to client
     emailVerifiedOnRegistrationAt: Date;
     @Column({ type: "varchar", nullable: true })
+    // don't send to client
     emailVerificationTokenOnRegistration: string;
     @Column({ type: "timestamp", nullable: true })
+    // don't send to client
     emailVerificationTokenOnRegistrationExpiresAt: Date;
     @Column({ type: "timestamp", nullable: true })
+    // don't send to client
     mobileVerifiedOnRegistrationAt: Date;
     @Column({ type: "varchar", nullable: true })
+    // don't send to client
     mobileVerificationTokenOnRegistration: string;
     @Column({ type: "timestamp", nullable: true })
+    // don't send to client
     mobileVerificationTokenOnRegistrationExpiresAt: Date;
     @Column({ type: "timestamp", nullable: true })
+    // don't send to client
     emailVerifiedOnLoginAt: Date;
     @Column({ type: "varchar", nullable: true })
+    // don't send to client
     emailVerificationTokenOnLogin: string;
     @Column({ type: "timestamp", nullable: true })
+    // don't send to client
     emailVerificationTokenOnLoginExpiresAt: Date;
     @Column({ type: "timestamp", nullable: true })
+    // don't send to client
     mobileVerifiedOnLoginAt: Date;
     @Column({ type: "varchar", nullable: true })
+    // don't send to client
     mobileVerificationTokenOnLogin: string;
     @Column({ type: "timestamp", nullable: true })
+    // don't send to client
     mobileVerificationTokenOnLoginExpiresAt: Date;
     @Column({ type: "varchar", nullable: true })
+    @Expose()
     customPayload: string;
     @ManyToMany(() => RoleMetadata, roleMetadata => roleMetadata.users, { cascade: true })
     @JoinTable()
+    @Expose()
     roles: RoleMetadata[];
     @OneToMany(() => UserViewMetadata, userViewMetadata => userViewMetadata.user, { cascade: true })
+    // don't send to client
     userViewMetadata: UserViewMetadata[];
 }