@exulu/backend 1.46.0 → 1.47.0

package/mintlify-docs/backend/introduction.mdx

@@ -0,0 +1,218 @@
+---
+title: "Introduction"
+description: "Fully featured platform as a service stack for creating and managing AI agents with semantic search, background workers, and multi-provider support"
+---
+
+## What is Exulu IMP?
+
+Exulu IMP (Intelligence Management Platform) is a fully featured PaaS framework for building production-ready multi- or single-agent applications. It provides the infrastructure you need to manage agents, semantic search contexts, background job processing, and tool integrations.
+
+## Key features
+
+<CardGroup cols={2}>
+  <Card title="Multi-provider agents" icon="users">
+    Support for 60+ providers through Vercel AI SDK, plus flexible integration with custom LLM providers via vLLM or Ollama
+  </Card>
+  <Card title="Semantic search" icon="scan-search">
+    Vector-powered context system for RAG implementations with pgvector. Out-of-the-box generation of tooling for semantic search and agentic search for agents
+  </Card>
+  <Card title="Background workers" icon="biceps-flexed">
+    BullMQ-based job queue for long-running tasks and scheduled operations
+  </Card>
+  <Card title="Express API" icon="server">
+    Ready-to-use GraphQL and REST APIs with authentication and file uploads
+  </Card>
+</CardGroup>
+
+## Quick start
+
+Get started by cloning the example repository or installing the package:
+
+```bash
+# Option 1: Clone the example project (recommended)
+git clone https://github.com/Qventu/exulu-example
+
+# Option 2: Install the NPM package
+npm install @exulu/backend
+```
+
+Create your first Exulu IMP instance using the factory pattern:
+
+```typescript
+import { ExuluApp, ExuluProvider } from "@exulu/backend";
+import { createAnthropic } from "@ai-sdk/anthropic";
+
+let instance: ExuluApp | null = null;
+
+export const exulu = async (): Promise<ExuluApp> => {
+  if (instance) {
+    return instance;
+  }
+
+  instance = new ExuluApp();
+  instance = await instance.create({
+    config: {
+      workers: { enabled: true },
+      MCP: { enabled: false },
+      telemetry: { enabled: false }
+    },
+    contexts: [],
+    rerankers: [],
+    tools: [],
+    agents: [
+      new ExuluProvider({
+        id: "assistant",
+        name: "Assistant",
+        provider: "anthropic",
+        description: "A helpful AI assistant",
+        type: "agent",
+        capabilities: {
+          text: true,
+          images: [".png", ".jpg", ".jpeg", ".webp"],
+          files: [".pdf", ".docx"],
+          audio: [],
+          video: []
+        },
+        maxContextLength: 200000,
+        config: {
+          name: "Assistant",
+          instructions: "You are a helpful assistant.",
+          model: {
+            create: ({ apiKey }) => {
+              const anthropic = createAnthropic({ apiKey });
+              return anthropic.languageModel("claude-sonnet-4-5");
+            }
+          }
+        }
+      })
+    ]
+  });
+
+  const expressApp = await instance.express.init();
+  expressApp.listen(3000);
+
+  return instance;
+};
+```
+
+<Card
+  title="Full quickstart guide"
+  icon="rocket"
+  href="/quickstart"
+  horizontal
+>
+  Follow the complete setup guide with database initialization and configuration
+</Card>
+
+## Core concepts
+
+<CardGroup cols={2}>
+  <Card
+    title="ExuluApp"
+    icon="cube"
+    href="/core/exulu-app/introduction"
+  >
+    The main application class that orchestrates all components
+  </Card>
+  <Card
+    title="Contexts"
+    icon="database"
+    href="/core/exulu-context/introduction"
+  >
+    Semantic search indices with vector embeddings for RAG
+  </Card>
+  <Card
+    title="Agents"
+    icon="robot"
+    href="/core/exulu-agent/introduction"
+  >
+    AI agents that process requests and use tools
+  </Card>
+  <Card
+    title="Tools"
+    icon="wrench"
+    href="/core/exulu-tool/introduction"
+  >
+    Functions that extend agent capabilities
+  </Card>
+</CardGroup>
+
+## Architecture
+
+Exulu IMP follows a modular architecture:
+
+1. **ExuluApp** - Central orchestrator that manages all components
+2. **Contexts** - Semantic search indices with automatic tool generation
+3. **Agents** - LLM-powered execution units with tool calling
+4. **Workers** - Background job processors for async operations
+5. **Express API** - GraphQL and REST endpoints for client access
+
+<Note>
+  The platform is designed to be deployed as an npm package in your Node.js application, giving you full control over hosting and customization.
+</Note>
+
+## Use cases
+
+<AccordionGroup>
+  <Accordion title="RAG applications">
+    Build retrieval-augmented generation systems with semantic search contexts. Ingest documents, generate embeddings, and let agents query relevant information automatically.
+  </Accordion>
+
+  <Accordion title="Multi-agent systems">
+    Create complex workflows with multiple specialized agents. Each agent can have different capabilities, tools, and LLM providers.
+  </Accordion>
+
+  <Accordion title="Background processing">
+    Offload long-running tasks like embeddings generation, document processing, and scheduled data sync to BullMQ workers.
+  </Accordion>
+
+  <Accordion title="Tool-augmented AI">
+    Extend agent capabilities with custom tools for API calls, database queries, file operations, and external service integrations.
+  </Accordion>
+</AccordionGroup>
+
+## What's included
+
+When you install `@exulu/backend`, you get:
+
+- **Default agents** for GPT-5, GPT-4.1, GPT-4o, Claude Opus 4, Claude Sonnet 4/4.5, Gemini 2.5, Llama 3.3, and more
+- **Built-in tools** for todo management and web search
+- **Database utilities** for PostgreSQL and vector storage
+- **Authentication** with NextAuth integration
+- **Job queues** with Redis and BullMQ
+- **Logging** with Winston
+- **Telemetry** with OpenTelemetry
+- **Type definitions** for TypeScript
+
+## Next steps
+
+<CardGroup cols={2}>
+  <Card
+    title="Quickstart"
+    icon="rocket"
+    href="/quickstart"
+  >
+    Set up your first Exulu IMP instance
+  </Card>
+  <Card
+    title="ExuluApp reference"
+    icon="cube"
+    href="/core/exulu-app/introduction"
+  >
+    Learn about the main application class
+  </Card>
+  <Card
+    title="Configuration"
+    icon="gear"
+    href="/core/exulu-app/configuration"
+  >
+    Configure logging, workers, and integrations
+  </Card>
+  <Card
+    title="API reference"
+    icon="code"
+    href="/api-reference/introduction"
+  >
+    Explore the GraphQL and REST APIs
+  </Card>
+</CardGroup>

package/mintlify-docs/changelog.mdx

@@ -0,0 +1,293 @@
+---
+title: "Changelog"
+description: "Product updates and release notes for Exulu IMP"
+---
+
+<Update label="March 2026">
+  **Release Date:** March 5, 2026
+
+  We're excited to share the latest updates to Exulu IMP! This release focuses on intelligent retrieval capabilities, code quality improvements, and expanded AI model support.
+
+  ### Versions
+  - **Backend NPM package:** 1.47.0
+
+  <Info>This is a development release. Please make sure you are using the latest compatible frontend and backend versions.</Info>
+
+  ## What's New
+
+  ### Agentic Retrieval System
+  We've introduced a powerful AI-powered retrieval system that revolutionizes how your agents search through knowledge bases:
+
+  - **Multi-step Retrieval Agent:** Implements an intelligent tool loop architecture that dynamically selects the best search strategy
+  - **Smart Search Strategy Selection:** Automatically chooses between hybrid, semantic, or keyword search based on query context
+  - **Multilingual Support:** Built-in language detection and reasoning support for 8+ languages
+  - **Intelligent Reranking:** Improves search relevance by reordering results based on semantic similarity
+  - **Resilient API Handling:** Retry logic with exponential backoff ensures reliable operations
+
+  ### NLP Query Preprocessing
+  Enhanced search capabilities with advanced natural language processing:
+
+  - **Language Detection:** Automatic detection using the franc library with support for multiple languages
+  - **Language-Specific Stemming:** Implements the natural NLP library for better query understanding
+  - **Query Variations:** Generates multilingual query variations for improved search coverage
+  - **PostgreSQL FTS Integration:** Maps ISO 639-3 language codes to PostgreSQL full-text search configurations
+
+  ### Expanded Default AI Model Support
+
+  #### New Cerebras Default Models
+  High-performance AI models with impressive context windows now available via `ExuluDefaultAgents.cerebras.*`:
+  - **GPT-OSS-120B** (128k context)
+  - **LLAMA-3.1-8B** (32k context)
+  - **LLAMA-3.3-70B** (32k context)
+
+  ### Perplexity Internet Search
+  Real-time web search capabilities directly in your agents:
+
+  - **Live Web Search:** Query the internet in real-time via Perplexity API
+  - **Advanced Filtering:** Support for domain filtering and recency filtering
+  - **Automatic Citations:** Properly formatted citations in search results
+  - **Rate Limit Handling:** Built-in protection against API rate limits
+
+  ## Code Quality & Developer Experience
+
+  ### Refactored Codebase Architecture
+  - **Modular Organization:** Refactored `classes.ts` into individual modules for better maintainability
+  - **Fixed Import References:** Corrected all source import paths for consistency
+  - **Decorator Folder Fix:** Corrected typo in registry decorator folder name (`decoraters` → `decorators`) and updated all import paths
+
+  ### Enhanced Development Tooling
+  For contributors working on the Exulu IMP source code, we've significantly improved the development experience with comprehensive tooling:
+
+  - **ESLint Integration:** Configured with TypeScript support for consistent code quality
+  - **Prettier Formatting:** Automatic code formatting for a clean codebase
+  - **Jest Testing Framework:** Set up with ts-jest for robust testing
+  - **Code Duplication Detection:** Added jscpd for identifying and managing code duplication
+  - **npm Scripts:** New commands for testing, linting, formatting, and validation
+
+  <Note>These tooling improvements are for source contributors only and do not affect users of the `@exulu/backend` npm package.</Note>
+
+  Available scripts for contributors:
+  ```bash
+  npm test           # Run tests
+  npm run lint       # Check code quality
+  npm run format     # Format code
+  npm run validate   # Type check + lint + test
+  npm run cpd        # Check code duplication
+  ```
+
+  ### Stricter TypeScript Configuration
+  Improved type safety across the codebase:
+  - Enabled `noImplicitAny` and `strictNullChecks`
+  - Added `noUnusedLocals` and `noUnusedParameters` checks
+  - Updated target from ESNext to ES2022 for better compatibility
+
+  ## Bug Fixes & Improvements
+
+  ### Security & Stability
+  - **npm Audit Vulnerabilities:** Fixed all identified security vulnerabilities
+  - **Removed Debug Code:** Cleaned up development artifacts and debug file writes from agentic-retrieval.ts
+  - **Agent ID Conflicts:** Fixed duplicate agent IDs in Claude Sonnet 4 and 4.5 agents
+
+  ### Code Cleanup
+  - **Linting Errors:** Resolved all linting errors across the codebase
+  - **Prettier Formatting:** Applied consistent code formatting
+  - **Documentation Organization:** Moved file storage documentation to proper folder structure
+  - **Removed Unused Code:** Cleaned up commented console.log statements and unnecessary imports
+  - **Math Tools Removed:** Removed unused math tools (arithmetic, statistics, trigonometric)
+
+  ## Dependency Updates
+
+  ### Major Upgrades
+  - **Vercel AI SDK:** Upgraded from v5 to v6 for the latest AI capabilities
+  - **AI Provider SDKs:** Updated Anthropic, Google Vertex, and OpenAI SDKs
+
+  ### New Dependencies
+  - `@ai-sdk/cerebras` - Cerebras model support
+  - `@perplexity-ai/perplexity_ai` - Web search integration
+  - `franc` - Language detection
+  - `natural` - NLP and stemming
+  - `wink-nlp` - Additional NLP capabilities
+  - `zod-to-json-schema` - Schema conversion utilities
+  - `jscpd` - Code duplication detection
+
+  ## What This Means For You
+
+  - **Better Code Quality:** Enhanced tooling and stricter TypeScript checks catch issues earlier
+  - **Easier Debugging:** Cleaner codebase with removed debug artifacts
+  - **More Maintainable:** Modular architecture makes it easier to understand and extend
+  - **Smarter Search:** Agentic retrieval and NLP preprocessing deliver more relevant results
+  - **Multilingual Support:** Your agents can now handle queries in multiple languages
+  - **More AI Options:** Choose from expanded model offerings including Cerebras and Perplexity
+  - **Live Web Search:** Agents can now access real-time information from the internet
+  - **More Reliable:** Bug fixes and security updates ensure stable operations
+  - **Better Performance:** Optimized search tools and consolidated architecture
+  - **Enhanced Security:** Resolved npm audit vulnerabilities
+
+  ## Getting Started
+
+  To upgrade to the latest version:
+
+  ```bash
+  npm install @exulu/backend@latest
+  ```
+</Update>
+
+<Update label="January 2026">
+  **Release Date:** January 10, 2026
+
+  We're excited to announce a major platform update bringing significant workflow enhancements, a significantly improved chat experience, and substantial architectural improvements. This release represents our most comprehensive update to date, with a focus on developer productivity, system performance, and user experience.
+
+  ### Versions
+  - **Backend NPM package:** 1.45.0
+  - **Frontend Docker image:** latest | 1.27.0
+  - **Frontend NPM package:** 1.16.1
+
+  <Info>Please make sure you are using the latest compatible frontend and backend versions.</Info>
+
+  ## What's New
+
+  ### Enhanced Workflow Automation
+  We've significantly enhanced our workflow system with powerful new scheduling and monitoring capabilities:
+
+  - **CRON Scheduling:** Automate workflow execution with flexible CRON-based scheduling featuring built-in presets (hourly, daily, weekly, monthly) or custom CRON expressions with cron-validator integration for pattern validation
+  - **Simplified Workflow Object:** Streamlined workflow schema with direct agent references via UUID, removing unnecessary complexity around ownership, variables, and RBAC fields
+  - **Run History Tracking:** Comprehensive history view for every workflow execution with full result, error, and metadata inspection
+  - **Real-time Monitoring:** Live status indicators showing active, completed, and failed workflow runs with instant updates
+  - **Visual Feedback:** Enhanced UI with status badges, tooltips, and at-a-glance health indicators for better workflow visibility
+  - **Improved UX:** Added persistent "Save as Workflow" banner in chat interface for easier workflow template creation, and users can now edit individual chat messages that are part of a workflow
+  - **Workflow Queue Configuration:** Workflows are created using a specific Agent instance. It is now possible to define a queue on an ExuluProvider class (in the new workflows property), which if provided enables you to define concurrency limits (in code) and automated schedules (in the UI and API)
+
+  These improvements make it easier than ever to automate recurring tasks and monitor their execution at scale.
+
+  ### Chat Experience Overhaul
+  We've completely refactored the chat architecture to deliver a faster, more intuitive conversational experience:
+
+  - **Server Components Migration:** Chat pages now leverage Next.js server components for dramatically improved initial load performance
+  - **Session Auto-creation:** Cleaner flow when starting new conversations with automatic session management
+  - **Improved Session UI:** Streamlined the Chat Session UI by hiding metadata such as created date and edit options behind hover states
+  - **Chat Search:** New dedicated search page to find specific messages or sessions across your entire chat history
+  - **Save as Workflow:** Persistent banner in chat interface making it easy to convert conversations into reusable workflow templates
+  - **Automatic Citations:** Agents using context search tools now automatically format inline citations with item metadata, chunk IDs, and context references, these are automatically processed, showing citation badges in the chat including a dialog window showing the original chunk the citation references
+  - **User Personalization:** Enhanced system prompts with user firstname/lastname fields for better agent personalization (configurable via privacy settings)
+
+  ### Agent & Prompt Management
+  Managing agents and prompts is now significantly more powerful and organized:
+
+  - **Prompt Library:** Browse prompts by tags with grouped category views at `/prompts/[tag]`
+  - **Agent Prompt Templates Browser:** Intuitive sheet interface for browsing and assigning prompts to agents with folder organization and tag filtering directly from the agent configuration screen
+  - **Quick Agent Duplication:** Copy existing agents with one click to accelerate agent creation using the new `<entity>CopyOneById` GraphQL mutation
+  - **Increased Agent Tool Execution Steps:** Extended default agent step count from 2 to 5 across chat and function calling flows for more complex reasoning tasks
+  - **Enhanced Tool Configuration:** Collapsible categories and search functionality in tool selection
+  - **Separated Instructions:** Clear distinction between system instructions and custom instructions, now fully exposed in GraphQL schema and shown in the UI if provided
+
+  ### Knowledge Base Data Processing Architecture
+  A fundamental refactor of our data processing system improves consistency and reduces complexity:
+
+  - **Unified Items Filtering:** New centralized items-filter.tsx component provides consistent filtering experience across items list view, embedding job trigger creation, and processor job filter creation with enhanced filter options and real-time result preview
+  - **Item-level Processing:** Migrated from field-level to item-level processors (`ExuluContextFieldProcessor` → `ExuluContextProcessor`) to reduce complexity
+  - **Bulk Operations:** Process multiple items at once with new `ProcessItems` GraphQL mutation supporting filter and limit parameters
+  - **Processor Visibility:** New processors view in the knowledge base section showing configuration details, queue status, and execution history
+  - **Timestamp Tracking:** Added `last_processed_at` field on items for better processing visibility
+  - **Queue Management Integration:** Direct access to queue controls from processor configuration with configurable timeouts (default: 180s, up to 30 minutes for long operations)
+  - **Enhanced Concurrency:** Separate worker and queue-level concurrency settings for granular control over job processing
+  - **Improved Database Pool:** Increased connection pool from 20 to 50 connections with statement timeouts to handle concurrent processor jobs
+  - **Automatic JSON Handling:** JSON fields now automatically serialize during create and update operations
+
+  ### Project Context Integration
+  Agents can now automatically leverage project-specific context for enhanced retrieval capabilities:
+
+  - **Project-aware Sessions:** Chat sessions can be associated with projects for contextual tool access with automatic project tool injection
+  - **Project-scoped Retrieval:** New `createProjectRetrievalTool` enables project-specific information search across multiple contexts with hybrid search and filtering
+  - **Enhanced Metadata:** Project cards display creation date, character count, and chunk information
+  - **Automatic Tool Ingestion:** Agents automatically gain access to project-specific retrieval tools when sessions have associated projects
+  - **Project Caching:** Optimized repeated project queries with intelligent caching mechanisms
+
+  ### Vector Search & Context System
+  Major improvements to our semantic search and retrieval capabilities:
+
+  - **Adaptive Filtering:** Automatic threshold filtering (60-70% of best match score) removes low-quality results
+  - **Chunk Expansion:** Configurable option to retrieve surrounding chunks (before/after) for better context continuity
+  - **Enhanced Hybrid Search:** Improved RRF weighting and scoring with better cosine distance, tsvector, and hybrid cutoff thresholds
+  - **Increased Capacity:** Maximum retrieval limit increased from 50 to 250 results
+  - **Improved Text Search:** tsvector search now uses OR logic for better partial matching of technical terms
+  - **Archive Filtering:** Archived items automatically excluded from all search methods
+  - **Context Metadata:** New `chunks_count` and `embeddings_updated_at` fields for better visibility
+  - **Chunk Management:** Added limit parameters to `generateChunks` and `deleteChunks` mutations with proper counting support
+  - **Direct Search API:** Added search method directly to `ExuluContext` class for easier integration
+
+  <Warning>Depending on your current Exulu IMP version you might need to update your database schema to add missing fields.</Warning>
+
+  ### Access Control & Security
+  Enhanced security and access control mechanisms across the platform:
+
+  - **Refined RBAC Logic:** More explicit and maintainable role-based access control with improved conditionals
+  - **Non-admin Chunk Limits:** Non-super-admins can generate/delete chunks with configurable limits for safety
+  - **User-scoped File Access:** Proper folder prefixing and user-scoped access control in file upload routes
+  - **Global File Management:** Enhanced download access control to support both user-scoped and global file access
+  - **Record Privacy Reset:** Copied records automatically reset to private `rights_mode` with new timestamps
+  - **Improved Authentication:** Better handling of unauthenticated requests with proper fallback behavior
+  - **Context-specific Validation:** Enhanced role validation for context-specific operations
+
+  ## Technical Improvements
+
+  ### Performance & Architecture
+  - **Next.js 16 Upgrade:** Migrated entire application to Next.js 16 for latest features and performance
+  - **GraphiQL v4:** Updated to latest GraphiQL version with improved developer experience
+  - **Server-side Data Fetching:** New `fetchGraphQLServerSide` moves certain fetch operations to server components
+  - **Optimistic Updates:** Better perceived performance with optimistic UI updates throughout
+  - **Query Optimization:** Full-text search now uses `ts_rank` instead of `ts_rank_cd` for better performance
+  - **GraphQL Filter Extensions:** Added `lte` and `gte` operators for Float filter types
+  - **Eval Worker Scaling:** Increased eval runs queue concurrency from 1 to 10 workers for faster evaluation processing
+
+  ### Developer Experience
+  - **Toast Notifications:** Standardized on Sonner library for consistent notification styling
+  - **Type Safety:** Enhanced TypeScript definitions across agent sessions, items, and workflows
+  - **Better Error Handling:** Improved API error responses and user-facing error states with proper null handling in job results
+  - **S3 Integration:** Simplified bucket handling with slash-based key notation (bucket/key) for better consistency
+  - **Global File Uploads:** New global file upload support outside user directories with enhanced access control (used primarily for agent visuals)
+
+  ### UI/UX Polish
+  - **Navigation Improvements:** Streamlined navigation with better visual hierarchy
+  - **Layout Refinements:** Fixed numerous spacing and alignment issues across pages
+  - **Status Indicators:** Consistent status badges and tooltips for better information density
+
+  ## Bug Fixes & Refinements
+
+  ### Minor Fixes
+  - Resolved duplicate todo list message rendering
+  - Fixed command item disabled styling and duplicate keys in tool lists
+  - Corrected search results layout from multi-column to single-column for readability
+  - Added automatic data refresh after chunk deletion
+  - Fixed missing spaces in chat session layout
+  - Improved retry functionality for embeddings and processor jobs
+  - Fixed `hasNextPage` logic in pagination queries
+  - Fixed access control to properly handle unauthenticated requests with `super_admin` bypass
+  - Corrected S3 key prefix ordering (user prefix before general prefix) and proper bucket name handling
+  - Fixed file deletion by properly stripping bucket prefix
+  - Fixed `created_by` type handling (string vs number) in access control checks
+  - Fixed eval run deletion to properly clean up associated `job_results` entries
+  - Improved chunk deletion safety by checking table existence before deletion
+  - Fixed image handling in chat API for proper multimodal support
+
+  ## Cleanup & Deprecations
+  This release includes significant cleanup to maintain codebase health:
+
+  - Removed deprecated `jobs-status-area` component
+  - Removed obsolete `chat-empty-state` component
+  - Removed deprecated `agent-nav` component
+  - Consolidated redundant layout files
+  - Removed `recent-jobs` component in favor of unified queue management
+  - Removed deprecated `ExuluDefaultContexts` export
+  - Deleted unused template contexts (`code-standards.ts`, `outputs.ts`)
+  - Removed hardcoded context registrations from `ExuluApp`
+  - Removed excessive debug `console.log` statements from auth, postgres, and redis clients
+  - Cleaned up redundant code across multiple modules
+
+  ## What's Next
+  Looking ahead, we're focusing on:
+
+  - Advanced workflow conditions and branching
+  - Enhanced eval capabilities with more flexible test case management
+  - Real-time collaboration features for shared agents and workflows
+  - Extended monitoring and observability for production deployments
+</Update>