@superatomai/sdk-node 0.0.4 → 0.0.6

package/README.md

@@ -5,13 +5,17 @@ A TypeScript/Node.js SDK for building AI-powered data applications with Superato
 ## Features
 
 - **WebSocket Communication** - Real-time bidirectional messaging with automatic reconnection
+- **AI-Powered Component Generation** - Automatically generate dashboard components and visualizations from user questions
+- **Intelligent Text Responses** - LLM-powered conversational responses with SQL query execution and retry logic
+- **Component Matching & Classification** - Smart component selection based on question type and visualization needs
 - **Collection Handlers** - Register custom data operation handlers (CRUD, queries, mutations)
 - **User Management** - Built-in authentication and user storage with file-based persistence
 - **Dashboard & Report Management** - Create and manage dashboards and reports with DSL-based rendering
-- **LLM Integration** - Unified interface for Anthropic Claude and Groq models with streaming support
+- **Multi-Provider LLM Integration** - Unified interface for Anthropic Claude and Groq models with automatic fallback
 - **Thread & UI Block Management** - Organize conversations and UI components with automatic cleanup
-- **Log Collection** - Capture and send runtime logs to the UI
-- **Prompt Loader** - Load custom prompts from the file system
+- **Log Collection** - Capture and send runtime logs to the UI with configurable log levels
+- **Prompt Loader** - Load and cache custom prompts from the file system with template variable support
+- **Database Schema Management** - Automatic schema documentation generation for LLM context
 - **Cleanup Service** - Automatic memory management and old data removal
 - **TypeScript First** - Full type safety with comprehensive type definitions
 
@@ -79,6 +83,7 @@ interface SuperatomSDKConfig {
   ANTHROPIC_API_KEY?: string;        // Optional: Anthropic API key for LLM
   GROQ_API_KEY?: string;             // Optional: Groq API key for LLM
   LLM_PROVIDERS?: LLMProvider[];     // Optional: Custom LLM providers
+  logLevel?: LogLevel;               // Optional: Log level (errors, warnings, info, verbose) (default: 'info')
 }
 ```
 
@@ -87,6 +92,7 @@ interface SuperatomSDKConfig {
 - `SA_WEBSOCKET_URL` - WebSocket URL (default: `wss://ws.superatom.ai/websocket`)
 - `ANTHROPIC_API_KEY` - Anthropic API key for Claude models
 - `GROQ_API_KEY` - Groq API key for open-source models
+- `SUPERATOM_LOG_LEVEL` - Log level for SDK logging (errors, warnings, info, verbose) (default: 'info')
 
 ## Core Features
 
@@ -329,7 +335,216 @@ const jsonResult = await LLM.stream(
 - With provider: `anthropic/model-name` or `groq/model-name`
 - Without provider: Defaults to Anthropic
 
-### 7. Thread & UI Block Management
+### 7. AI-Powered User Response System
+
+The SDK includes a powerful AI system for generating intelligent responses to user questions with two modes: **component generation** and **text responses**.
+
+#### Component Generation Mode
+
+Automatically match and generate dashboard components based on user questions:
+
+```typescript
+import { get_user_response } from '@superatomai/sdk-node/userResponse';
+
+const components = [
+  { id: 'sales-chart', name: 'SalesChart', type: 'BarChart', /* ... */ },
+  { id: 'kpi-card', name: 'RevenueKPI', type: 'KPICard', /* ... */ },
+  // ... more components
+];
+
+const result = await get_user_response(
+  'Show me sales by region',
+  components,
+  anthropicApiKey,
+  groqApiKey,
+  ['anthropic', 'groq'], // Provider fallback order
+  logCollector,
+  conversationHistory,
+  'component' // Component generation mode
+);
+
+if (result.success) {
+  console.log('Generated component:', result.data.component);
+  console.log('Reasoning:', result.data.reasoning);
+}
+```
+
+**Features:**
+- **Question Classification** - Automatically determines question type (analytical, data_modification, general)
+- **Visualization Type Detection** - Identifies required visualization types (charts, tables, KPIs)
+- **Multi-Component Dashboards** - Generates multiple components for comprehensive analysis
+- **Props Modification** - Intelligently modifies component props including SQL queries
+- **SQL Query Validation** - Ensures queries have proper LIMIT clauses and fixes scalar subqueries
+
+#### Text Response Mode
+
+Generate conversational text responses with SQL query execution:
+
+```typescript
+const result = await get_user_response(
+  'What were the top 5 selling products last month?',
+  components,
+  anthropicApiKey,
+  groqApiKey,
+  ['anthropic', 'groq'],
+  logCollector,
+  conversationHistory,
+  'text', // Text response mode
+  (chunk) => {
+    // Stream text chunks in real-time
+    process.stdout.write(chunk);
+  },
+  collections // Required for query execution
+);
+
+if (result.success) {
+  console.log('Text response:', result.data.text);
+  console.log('Matched components:', result.data.matchedComponents);
+  console.log('Container component:', result.data.component);
+}
+```
+
+**Features:**
+- **SQL Query Execution** - Automatically generates and executes SQL queries via tool calling
+- **Automatic Retry Logic** - Up to 6 retry attempts with query correction on errors
+- **Streaming Responses** - Real-time text streaming with query execution status updates
+- **Component Suggestions** - Parses component suggestions from text and matches with available components
+- **Layout Discovery** - Automatically selects appropriate dashboard layouts based on component metadata
+- **Performance Tracking** - Measures and logs total time taken for request processing
+
+#### Using BaseLLM Classes Directly
+
+For more control, use the BaseLLM implementations directly:
+
+```typescript
+import { anthropicLLM, groqLLM } from '@superatomai/sdk-node/userResponse';
+
+// Classify user question
+const classification = await anthropicLLM.classifyUserQuestion(
+  'Show me sales trends',
+  apiKey,
+  logCollector,
+  conversationHistory
+);
+
+console.log('Question type:', classification.questionType);
+console.log('Visualizations needed:', classification.visualizations);
+
+// Generate analytical component
+const result = await anthropicLLM.generateAnalyticalComponent(
+  'Show me sales by region',
+  components,
+  'BarChart', // Preferred visualization type
+  apiKey,
+  logCollector,
+  conversationHistory
+);
+
+// Match existing component
+const matchResult = await anthropicLLM.matchComponent(
+  'Update the sales dashboard',
+  components,
+  apiKey,
+  logCollector,
+  conversationHistory
+);
+
+// Generate next questions
+const nextQuestions = await anthropicLLM.generateNextQuestions(
+  originalPrompt,
+  generatedComponent,
+  componentData,
+  apiKey,
+  logCollector,
+  conversationHistory
+);
+```
+
+**BaseLLM Methods:**
+- `handleUserRequest()` - Main orchestration method (supports both component and text modes)
+- `classifyUserQuestion()` - Classify question type and identify visualizations
+- `generateAnalyticalComponent()` - Generate single analytical component
+- `generateMultipleAnalyticalComponents()` - Generate multiple components in parallel
+- `matchComponent()` - Match and modify existing component
+- `validateAndModifyProps()` - Validate and modify component props
+- `generateTextResponse()` - Generate text with tool calling support
+- `matchComponentsFromTextResponse()` - Match components from text suggestions
+- `generateNextQuestions()` - Generate follow-up question suggestions
+
+### 8. Prompt Loader System
+
+The SDK includes a sophisticated prompt loading system with caching and template variable support:
+
+```typescript
+import { promptLoader } from '@superatomai/sdk-node/userResponse';
+
+// Initialize with custom directory (default: .prompts)
+const sdk = new SuperatomSDK({
+  apiKey: 'your-api-key',
+  projectId: 'your-project-id',
+  promptsDir: './my-custom-prompts',
+});
+
+// Prompts are automatically loaded and cached on initialization
+// Access prompt cache size
+const cacheSize = promptLoader.getCacheSize();
+console.log(`Loaded ${cacheSize} prompts`);
+
+// Load specific prompts with variables
+const prompts = await promptLoader.loadPrompts('classify', {
+  USER_PROMPT: userQuestion,
+  CONVERSATION_HISTORY: history || 'No previous conversation'
+});
+
+console.log('System prompt:', prompts.system);
+console.log('User prompt:', prompts.user);
+```
+
+**Prompt Directory Structure:**
+```
+.prompts/
+  ├── classify/
+  │   ├── system.md     # System prompt for classification
+  │   └── user.md       # User prompt template
+  ├── match-component/
+  │   ├── system.md
+  │   └── user.md
+  ├── modify-props/
+  │   ├── system.md
+  │   └── user.md
+  ├── text-response/
+  │   ├── system.md
+  │   └── user.md
+  └── match-text-components/
+      ├── system.md
+      └── user.md
+```
+
+**Template Variables:**
+Variables in prompts are replaced using the `{{VARIABLE_NAME}}` syntax:
+
+```markdown
+# system.md
+You are analyzing this question: {{USER_PROMPT}}
+
+Previous conversation:
+{{CONVERSATION_HISTORY}}
+
+Available components:
+{{AVAILABLE_COMPONENTS}}
+```
+
+**Built-in Prompt Types:**
+- `classify` - Question classification and visualization type detection
+- `match-component` - Component matching and selection
+- `modify-props` - Props validation and modification
+- `single-component` - Single analytical component generation
+- `text-response` - Text response with tool calling
+- `match-text-components` - Component matching from text suggestions
+- `container-metadata` - Container title and description generation
+- `actions` - Next question generation
+
+### 9. Thread & UI Block Management
 
 Organize conversations and UI components:
 
@@ -379,45 +594,92 @@ threadManager.deleteThread(thread.getId());
 - `form` - Interactive forms
 - Custom types as needed
 
-### 8. Log Collection
+### 10. Logging System
 
-Capture and send logs to the UI:
+The SDK includes a comprehensive logging system with environment-based log levels for both terminal and UI log collection.
 
-```typescript
-import { UILogCollector } from '@superatomai/sdk-node';
+#### Log Levels
+
+The SDK supports hierarchical log levels:
 
-const collector = new UILogCollector();
+- **`errors`** - Only error logs are shown
+- **`warnings`** - Warning and error logs are shown
+- **`info`** - Info, warning, and error logs are shown (default)
+- **`verbose`** - All logs including debug messages are shown
 
-// Start collecting logs
-collector.start();
+#### Configuring Log Level
 
-// Logs are automatically captured
-console.log('This will be captured');
-console.error('Error messages too');
+You can set the log level in three ways:
+
+**1. Via environment variable (recommended for deployment):**
+
+```bash
+export SUPERATOM_LOG_LEVEL=verbose
+node your-app.js
+```
+
+**2. Via SDK configuration (overrides environment variable):**
+
+```typescript
+import { SuperatomSDK } from '@superatomai/sdk-node';
 
-// Add custom logs
-collector.addLog({
-  level: 'info',
-  message: 'Processing data...',
-  type: 'explanation',
-  data: { progress: 50 },
+const sdk = new SuperatomSDK({
+  apiKey: 'your-api-key',
+  projectId: 'your-project-id',
+  logLevel: 'verbose', // errors | warnings | info | verbose
 });
+```
 
-// Get captured logs
-const logs = collector.getLogs();
+**3. Programmatically at runtime:**
+
+```typescript
+import { logger } from '@superatomai/sdk-node';
+
+// Change log level at runtime
+logger.setLogLevel('verbose');
 
-// Send logs to UI (via SDK)
-const message = collector.createMessage('uiblock-123');
-sdk.send(message);
+// Get current log level
+const currentLevel = logger.getLogLevel(); // returns 'verbose'
+```
 
-// Stop collecting
-collector.stop();
+#### Using the Logger
 
-// Clear logs
-collector.clear();
+```typescript
+import { logger } from '@superatomai/sdk-node';
+
+// These will be filtered based on the configured log level
+logger.error('Critical error occurred');        // Shown at all levels
+logger.warn('Something might be wrong');        // Hidden when level is 'errors'
+logger.info('Processing completed');            // Hidden when level is 'errors' or 'warnings'
+logger.debug('Detailed debug information');     // Only shown when level is 'verbose'
 ```
 
-### 9. Cleanup Service
+#### UI Log Collection
+
+The `UILogCollector` also respects the log level configuration. Logs are sent to the frontend and displayed in the terminal based on the configured level.
+
+```typescript
+import { UILogCollector } from '@superatomai/sdk-node';
+
+const collector = new UILogCollector(
+  clientId,
+  (msg) => sdk.send(msg),
+  'uiblock-123'
+);
+
+// These logs are filtered based on log level before being sent to UI
+collector.error('Error message');      // Always sent
+collector.warn('Warning message');     // Sent unless level is 'errors'
+collector.info('Info message');        // Sent unless level is 'errors' or 'warnings'
+collector.debug('Debug message');      // Only sent when level is 'verbose'
+
+// Get all collected logs
+const logs = collector.getLogs();
+```
+
+**Note:** The log level affects both terminal output and UI log collection, ensuring consistent logging behavior across your application.
+
+### 11. Cleanup Service
 
 Automatic memory management for threads and UI blocks:
 
@@ -458,31 +720,6 @@ STORAGE_CONFIG.UIBLOCK_RETENTION_DAYS = 14;
 STORAGE_CONFIG.MAX_COMPONENT_DATA_SIZE = 200 * 1024;
 ```
 
-### 10. Prompt Loader
-
-Load custom prompts from the file system:
-
-```typescript
-// Prompts are automatically loaded from .prompts/ directory
-// Or specify custom directory in config:
-const sdk = new SuperatomSDK({
-  apiKey: 'your-api-key',
-  projectId: 'your-project-id',
-  promptsDir: './my-prompts',
-});
-
-// Prompts are loaded and cached on initialization
-// They can be accessed by handlers internally
-```
-
-**Prompt File Structure:**
-```
-.prompts/
-  ├── system.txt          # System prompts
-  ├── user-greeting.txt   # User greeting prompts
-  └── analysis.txt        # Analysis prompts
-```
-
 ## API Reference
 
 ### SuperatomSDK
@@ -629,9 +866,35 @@ import type {
   IncomingMessage,
   SuperatomSDKConfig,
   CollectionHandler,
+  CollectionOperation,
   User,
-  DSLRendererProps,
+  UsersData,
+  Component,
+  T_RESPONSE,
+  LLMProvider,
+  LogLevel,
+  CapturedLog,
+  Action,
 } from '@superatomai/sdk-node';
+
+// Import LLM and utility classes
+import {
+  LLM,
+  UserManager,
+  UILogCollector,
+  Thread,
+  UIBlock,
+  ThreadManager,
+  CleanupService,
+  logger,
+} from '@superatomai/sdk-node';
+
+// Import user response utilities
+import {
+  get_user_response,
+  anthropicLLM,
+  groqLLM,
+} from '@superatomai/sdk-node/userResponse';
 ```
 
 ## Examples
@@ -640,9 +903,13 @@ Check out the [examples](./examples) directory for complete examples:
 - Basic WebSocket communication
 - Data collection handlers
 - User authentication flow
-- LLM integration
+- LLM integration with streaming
+- AI-powered component generation
+- Text response with query execution
+- Component matching and classification
 - Thread management
 - Dashboard creation
+- Prompt customization
 
 ## Development