@ixo/common 1.1.0

package/docs/services.md

@@ -0,0 +1,168 @@
+# Services Documentation
+
+## Overview
+
+The Services module in `@ixo/common` provides essential services for managing Matrix rooms and chat sessions. It integrates with the Matrix protocol through `@ixo/matrix` and provides type-safe operations for room and session management.
+
+## Core Services
+
+### Room Manager Service
+
+A service for managing Matrix rooms with support for creation, retrieval, and access control.
+
+```typescript
+import { RoomManagerService } from '@ixo/common/services';
+
+const roomManager = new RoomManagerService();
+
+// Create or get existing room
+const roomId = await roomManager.getOrCreateRoom({
+  did: 'user-did',
+  oracleName: 'oracle-name',
+  userAccessToken: 'matrix-access-token',
+});
+
+// Get room by DID and oracle name
+const room = await roomManager.getRoom({
+  did: 'user-did',
+  oracleName: 'oracle-name',
+});
+```
+
+#### Room Manager Capabilities
+
+- Create new Matrix rooms
+- Retrieve existing rooms by DID and oracle name
+- Get or create rooms (idempotent operation)
+
+#### Room Manager Types
+
+```typescript
+interface CreateRoomDto {
+  did: string;
+  oracleName: string;
+  userAccessToken: string;
+}
+
+interface GetRoomDto {
+  did: string;
+  oracleName: string;
+}
+```
+
+### Session Manager Service
+
+A service for managing chat sessions with support for persistence in Matrix rooms.
+
+```typescript
+import { SessionManagerService } from '@ixo/common/services';
+
+const sessionManager = new SessionManagerService();
+
+// Create new chat session
+const session = await sessionManager.createSession({
+  did: 'user-did',
+  oracleName: 'oracle-name',
+  matrixAccessToken: 'access-token',
+});
+
+// List user's sessions
+const { sessions } = await sessionManager.listSessions({
+  did: 'user-did',
+  matrixAccessToken: 'access-token',
+});
+
+// Delete a session
+await sessionManager.deleteSession({
+  did: 'user-did',
+  sessionId: 'session-uuid',
+  matrixAccessToken: 'access-token',
+});
+```
+
+#### Session Manager Capabilities
+
+- Create new chat sessions
+- List existing sessions for a user
+- Delete sessions
+- Automatic session title generation using AI
+- Session state persistence in Matrix rooms
+- Type-safe operations with DTOs
+
+#### Session Types
+
+```typescript
+interface ChatSession {
+  sessionId: string;
+  oracleName: string;
+  title: string;
+  lastUpdatedAt: string;
+  createdAt: string;
+}
+
+interface CreateChatSessionDto {
+  did: string;
+  oracleName: string;
+  matrixAccessToken: string;
+}
+
+interface ListChatSessionsDto {
+  did: string;
+  matrixAccessToken: string;
+}
+
+interface DeleteChatSessionDto {
+  did: string;
+  sessionId: string;
+  matrixAccessToken: string;
+}
+```
+
+## Error Handling
+
+The services provide specific error types for common scenarios:
+
+```typescript
+import {
+  NoUserRoomsFoundError,
+  RoomNotFoundError,
+  UserNotInRoomError,
+} from '@ixo/common/services';
+
+try {
+  await operation();
+} catch (error) {
+  if (error instanceof NoUserRoomsFoundError) {
+    // Handle case where user has no rooms
+  } else if (error instanceof RoomNotFoundError) {
+    // Handle case where specific room not found
+  } else if (error instanceof UserNotInRoomError) {
+    // Handle case where user doesn't have access
+  }
+}
+```
+
+## Integration with Matrix
+
+Both services integrate with the Matrix protocol through `@ixo/matrix`:
+
+- Uses Matrix rooms for persistence
+- Leverages Matrix state events for session storage
+- Handles Matrix authentication and access control
+- Provides type-safe Matrix operations
+
+## Best Practices
+
+### Room Management
+
+- Store room IDs for frequent access
+- Handle room creation idempotently
+- Validate user access tokens
+- Use appropriate error handling
+
+### Session Management
+
+- Use UUIDs for session IDs
+- Handle session state updates atomically
+- Implement proper cleanup for deleted sessions
+- Validate session access before operations

package/docs/tools.md

@@ -0,0 +1,325 @@
+# Tools and Utilities Documentation
+
+This document provides detailed information about the tools and utilities available in the `@ixo/common` package.
+
+## Tools
+
+### Ask IXO Guru Tool
+
+A LangChain tool for interacting with the IXO Guru AI, which provides access to IXO's internal knowledge base.
+
+```typescript
+import { askIXOGuruTool } from '@ixo/common/ai/tools';
+
+// Ask a question
+const response = await askIXOGuruTool.invoke({
+  question: "What is IXO's approach to impact tokens?",
+  sessionId: 'uuid-v4-string', // To save the conversation to matrix
+});
+```
+
+#### Configuration
+
+Requires the following environment variables:
+
+```env
+IXO_GURU_QUERY_ENDPOINT=your_endpoint_url
+GURU_ASSISTANCE_API_TOKEN=your_api_token
+ORACLE_DID=your_oracle_did
+```
+
+#### Response Format
+
+```yaml
+answer: "The AI's response to your question"
+sessionId: 'conversation-session-uuid'
+```
+
+#### Error Handling
+
+The tool handles various error scenarios:
+
+- Missing environment variables
+- API connection issues
+- Invalid session IDs
+- Authentication failures
+
+### Retriever Tool
+
+A LangChain tool factory for creating document retrieval tools that can search and filter documents from a vector database with relevance checking.
+
+```typescript
+import { retrieverToolFactory } from '@ixo/common/ai/tools';
+import { VectorDBDataStore } from '@ixo/data-store';
+
+// Create a retriever tool
+const retriever = retrieverToolFactory({
+  store: vectorStore, // Your VectorDBDataStore instance
+  similarThreshold: 0.3, // Optional similarity threshold
+  filters: { category: 'tech' }, // Optional filters
+  map: new Map(), // Optional map to store metadata
+  requestId: 'unique-id', // Optional request identifier
+});
+
+// Use the tool
+const docs = await retriever.invoke({
+  query: "What's new in AI?",
+});
+```
+
+#### Features
+
+- Integrates with vector databases via VectorDBDataStore
+- Configurable similarity threshold for initial filtering
+- Additional AI-powered relevance checking for better results
+- Optional metadata mapping for tracking used documents
+- Support for custom filters and OpenAI models
+- Returns LangChain Document objects
+
+#### Configuration Options
+
+```typescript
+type RetrieverToolFactoryArgs = {
+  model?: BaseChatModel; // Optional OpenAI model
+  filters?: Record<string, unknown>; // Vector store filters
+  similarThreshold?: number; // Similarity cutoff (default: 0.3)
+  store: VectorDBDataStore; // Required vector store
+  map?: Map<string, unknown>; // Optional metadata map
+  requestId?: string; // Optional request tracking
+};
+```
+
+#### Behavior
+
+- Queries vector store with similarity search
+- For threshold ≥ 0.3, performs additional AI relevance check
+- Stores document metadata in provided map if configured
+- Returns undefined if no documents found
+- Handles errors gracefully with logging
+
+### Web Search Tool
+
+A LangChain tool that provides web search capabilities using the Tavily API. This tool is optimized for comprehensive, accurate, and trusted search results.
+
+```typescript
+import { webSearchTool } from '@ixo/common/ai/tools';
+
+// Use in LangChain
+const result = await webSearchTool.invoke({
+  input: "What's new in AI technology?",
+});
+```
+
+#### Features and Behavior
+
+- Uses Tavily's AI-powered search engine
+- Returns structured search results including:
+  - Query summary
+  - Top 3 most relevant results
+  - Each result includes title, URL, content, and publish date
+- Results are formatted in YAML for better readability for AI and ovid parsing errors
+- Built-in input validation using Zod
+
+#### Web Search Configuration
+
+Requires the `TAVILY_API_KEY` environment variable to be set:
+
+```env
+TAVILY_API_KEY=your_api_key_here
+```
+
+#### Web Search Response Format
+
+```yaml
+query: 'your search query'
+summary: 'AI-generated summary of search results'
+results:
+  - title: 'Result Title'
+    url: 'https://result.url'
+    content: 'Result content snippet'
+    publishedAt: '2024-01-28'
+```
+
+## Utilities
+
+### File Loading Utility
+
+A versatile utility for loading and processing various types of files, supporting both local files and URLs. It automatically handles different file formats and returns processed LangChain Document objects.
+
+```typescript
+import { loadFile } from '@ixo/common/ai/utils';
+
+// Load a local file
+const pdfDocs = await loadFile('path/to/document.pdf');
+
+// Load a file from URL with optional fetch options
+const docsDocs = await loadFile('https://example.com/document.docx', {
+  headers: {
+    Authorization: 'Bearer token', // user OAuth token from Slack
+  },
+});
+```
+
+#### Supported File Types
+
+- PDF (.pdf)
+- Microsoft Word (.doc, .docx)
+- Markdown (.md, .markdown)
+- HTML (.html, .htm)
+- Plain Text (.txt)
+
+#### File Loading Features
+
+- Automatic file type detection based on extension and MIME type
+- Support for both local files and URLs
+- Built-in error handling and logging
+- Automatic text splitting using RecursiveCharacterTextSplitter
+- HTML to text conversion for web content
+- Returns LangChain Document objects ready for further processing
+
+### Document Splitter
+
+A utility for splitting text content into manageable chunks using LangChain's RecursiveCharacterTextSplitter.
+
+```typescript
+import { docSplitter } from '@ixo/common/ai/utils';
+
+// Split a single string
+const chunks = await docSplitter('Long text content...');
+
+// Split multiple strings
+const multiChunks = await docSplitter(['Text 1...', 'Text 2...']);
+```
+
+#### Document Splitter Features
+
+- Input validation using Zod
+- Supports both single string and string array inputs
+- Returns array of LangChain Document objects
+- Automatic error handling for empty or invalid inputs
+
+### JSON to YAML Converter
+
+A utility for converting JSON objects to YAML format, useful for creating human-readable configurations and outputs.
+
+```typescript
+import { jsonToYaml } from '@ixo/common/ai/utils';
+
+const yaml = jsonToYaml({
+  name: 'example',
+  config: {
+    enabled: true,
+    values: [1, 2, 3],
+  },
+});
+```
+
+### Document Stringifier
+
+A utility for converting LangChain Document objects to string format.
+
+```typescript
+import { stringifyDocs } from '@ixo/common/ai/utils';
+
+const stringified = stringifyDocs(documents);
+```
+
+### Array Chunking
+
+A utility for splitting arrays into smaller chunks of a specified size.
+
+```typescript
+import { chunkArr } from '@ixo/common/ai/utils';
+
+const chunks = chunkArr([1, 2, 3, 4, 5], 2);
+// Result: [[1, 2], [3, 4], [5]]
+```
+
+### Document Relevance Checker
+
+A utility that uses LLMs to determine if a document is relevant to a given query. It's particularly useful for filtering search results and ensuring content relevance.
+
+```typescript
+import checkDocRelevance from '@ixo/common/ai/utils';
+
+// Check document relevance
+const isRelevant = await checkDocRelevance({
+  doc: 'Document content or LangChain Document', // Can be string or Document
+  query: 'What technologies does IXO use?',
+  model: customModel, // Optional: defaults to OpenAI
+});
+```
+
+#### Relevance Checker Capabilities
+
+- Accepts both string content and LangChain Document objects
+- Uses structured LLM output with Zod validation
+- Customizable LLM model support
+- Semantic relevance checking based on content meaning
+- Built-in prompt template for consistent evaluation
+
+#### Relevance Checker Behavior
+
+The checker evaluates relevance based on three criteria:
+
+1. Identifies documents completely unrelated to the query
+2. Considers documents relevant if they contain related keywords or semantic meaning
+3. Accepts partially relevant documents if they contain any related information
+
+#### Usage Example with Custom Model
+
+```typescript
+import { ChatOpenAI } from '@langchain/openai';
+import checkDocRelevance from '@ixo/common/ai/utils';
+
+const customModel = new ChatOpenAI({
+  modelName: 'gpt-4',
+  temperature: 0,
+});
+
+const isRelevant = await checkDocRelevance({
+  doc: new Document({ pageContent: 'content here' }),
+  query: 'your query',
+  model: customModel,
+});
+```
+
+### Similarity Search Filter
+
+A utility for filtering vector similarity search results based on a similarity threshold. This is useful for ensuring high-quality search results by removing low-similarity matches.
+
+```typescript
+import { filterSimilaritySearchResults } from '@ixo/common/ai/utils';
+
+// Filter search results
+const results: [Document, number][] = [
+  [doc1, 0.8],
+  [doc2, 0.4],
+  [doc3, 0.9],
+];
+
+const filteredDocs = filterSimilaritySearchResults(results, 0.7);
+// Returns only docs with similarity >= 0.7
+```
+
+#### Filter Capabilities
+
+- Type-safe implementation with generics
+- Works with LangChain Document objects
+- Filters based on configurable threshold
+- Preserves document order
+- Simple and efficient filtering
+
+#### Usage with Vector Search
+
+```typescript
+import { filterSimilaritySearchResults } from '@ixo/common/ai/utils';
+import { type Document } from '@langchain/core/documents';
+
+// Example with vector search results
+type SearchResult = [Document, number];
+const searchResults: SearchResult[] = await vectorStore.similaritySearch(query);
+
+// Filter results with similarity >= 0.8
+const highQualityResults = filterSimilaritySearchResults(searchResults, 0.8);
+```

package/jest.config.js

@@ -0,0 +1,6 @@
+const baseConfig = require('@ixo/jest-config/nest');
+
+module.exports = {
+  ...baseConfig,
+  displayName: 'common',
+};

package/package.json

@@ -0,0 +1,81 @@
+{
+  "name": "@ixo/common",
+  "version": "1.1.0",
+  "private": false,
+  "publishConfig": {
+    "access": "public"
+  },
+  "type": "module",
+  "main": "dist/index.js",
+  "types": "dist/index.d.ts",
+  "exports": {
+    ".": {
+      "types": "./dist/index.d.ts",
+      "import": "./dist/index.js",
+      "default": "./dist/index.js"
+    },
+    "./ai": {
+      "types": "./dist/ai/index.d.ts",
+      "import": "./dist/ai/index.js"
+    },
+    "./*": {
+      "types": "./dist/*.d.ts",
+      "import": "./dist/*.js"
+    }
+  },
+  "devDependencies": {
+    "@jest/types": "^29.6.3",
+    "@langchain/langgraph-checkpoint-validation": "^1.0.0",
+    "@testcontainers/postgresql": "^10.16.0",
+    "@types/better-sqlite3": "^7.6.13",
+    "@types/jest": "^29.5.14",
+    "@types/node": "^22.7.3",
+    "@types/store": "^2.0.5",
+    "better-sqlite3": "^12.4.6",
+    "jest": "^29.7.0",
+    "testcontainers": "^10.16.0",
+    "ts-jest": "^29.2.5",
+    "typescript": "^5.3.3",
+    "@ixo/eslint-config": "1.0.0",
+    "@ixo/data-store": "^1.0.1",
+    "@ixo/jest-config": "0.0.0",
+    "@ixo/typescript-config": "1.0.0",
+    "@ixo/oracles-events": "^1.0.1"
+  },
+  "dependencies": {
+    "@cosmjs/proto-signing": "^0.33.0",
+    "@cosmjs/stargate": "^0.33.0",
+    "@ixo/impactxclient-sdk": "^2.3.1",
+    "@langchain/community": "^1.0.4",
+    "@langchain/core": "1.0.6",
+    "@langchain/langgraph": "1.0.2",
+    "@langchain/langgraph-checkpoint": "^1.0.0",
+    "@langchain/langgraph-checkpoint-sqlite": "^1.0.0",
+    "@langchain/openai": "^1.1.2",
+    "@langchain/textsplitters": "^1.0.0",
+    "@tavily/core": "0.0.2",
+    "axios": "^1.7.9",
+    "axios-retry": "^4.5.0",
+    "class-transformer": "^0.5.1",
+    "class-validator": "^0.14.1",
+    "dotenv": "^16.4.5",
+    "duck-duck-scrape": "^2.2.5",
+    "html-to-text": "9.0.5",
+    "langchain": "1.0.6",
+    "langfuse": "^3.38.5",
+    "langsmith": "^0.1.61",
+    "mammoth": "1.8.0",
+    "openai": "^6.9.1",
+    "pdf-parse": "^1.1.1",
+    "playwright": "^1.51.0",
+    "store": "^2.0.12",
+    "zod": "^4.1.12",
+    "@ixo/logger": "0.0.2",
+    "@ixo/matrix": "1.1.0",
+    "@ixo/oracles-chain-client": "1.2.0"
+  },
+  "scripts": {
+    "build": "tsc && tsc-alias",
+    "test": "jest"
+  }
+}

package/src/ai/checkpointer/index.ts

@@ -0,0 +1 @@
+export * from '@langchain/langgraph-checkpoint-sqlite';

package/src/ai/index.ts

@@ -0,0 +1,8 @@
+export * from './checkpointer/index.js';
+export * from './models/index.js';
+export * from './nodes/index.js';
+export * from './semantic-router-factory/index.js';
+export * from './tools/index.js';
+export * from './types.js';
+export * from './utils/index.js';
+export { zodResponseFormat } from 'openai/helpers/zod';

package/src/ai/models/index.ts

@@ -0,0 +1 @@
+export * from './openai.js';

package/src/ai/models/openai.test.ts

@@ -0,0 +1,72 @@
+import { ChatOpenAI, OpenAIEmbeddings } from '@langchain/openai';
+import { getChatOpenAiModel, getOpenAiEmbeddings } from './openai.js';
+
+// Mock environment variables
+const originalEnv = process.env;
+
+describe('OpenAI Models', () => {
+  beforeEach(() => {
+    jest.resetModules();
+    process.env = { ...originalEnv };
+    process.env.OPENAI_API_KEY = 'test-api-key';
+  });
+
+  afterEach(() => {
+    process.env = originalEnv;
+  });
+
+  describe('getChatOpenAiModel', () => {
+    it('should create a ChatOpenAI instance with default parameters', () => {
+      const model = getChatOpenAiModel();
+
+      expect(model).toBeInstanceOf(ChatOpenAI);
+      expect(model).toHaveProperty('temperature', 0.2);
+      expect(model).toHaveProperty('modelName', 'gpt-4o-mini');
+      expect(model).toHaveProperty('apiKey', 'test-api-key');
+    });
+
+    it('should override default parameters when provided', () => {
+      const customParams = {
+        temperature: 0.8,
+        model: 'gpt-4',
+        apiKey: 'custom-api-key',
+      };
+
+      const model = getChatOpenAiModel(customParams);
+
+      expect(model).toBeInstanceOf(ChatOpenAI);
+      expect(model).toHaveProperty('temperature', 0.8);
+      expect(model).toHaveProperty('modelName', 'gpt-4');
+      expect(model).toHaveProperty('apiKey', 'custom-api-key');
+    });
+  });
+
+  describe('getOpenAiEmbeddings', () => {
+    it('should create an OpenAIEmbeddings instance with default parameters', () => {
+      const embeddings = getOpenAiEmbeddings();
+      expect(embeddings).toBeInstanceOf(OpenAIEmbeddings);
+      expect(embeddings.model).toEqual('text-embedding-3-small');
+    });
+
+    it('should override default parameters when provided', () => {
+      const customParams = {
+        model: 'text-embedding-ada-002',
+        apiKey: 'custom-api-key',
+      };
+
+      const embeddings = getOpenAiEmbeddings(customParams);
+
+      expect(embeddings).toBeInstanceOf(OpenAIEmbeddings);
+      expect(embeddings.model).toEqual('text-embedding-ada-002');
+    });
+  });
+
+  describe('Error handling', () => {
+    it('should throw an error when OPENAI_API_KEY is not set', () => {
+      delete process.env.OPENAI_API_KEY;
+
+      expect(() => getChatOpenAiModel()).toThrow();
+      expect(() => getOpenAiEmbeddings()).toThrow();
+    });
+  });
+});