@probelabs/probe-chat 0.6.0-rc102 → 0.6.0-rc103

package/LOCAL_IMAGE_SUPPORT.md

@@ -0,0 +1,195 @@
+# Local Image Support in Probe Agent
+
+The probe agent now supports reading local image files directly from file paths mentioned in user messages and **automatically loads images when mentioned during the agentic loop**.
+
+## Features Added
+
+### Automatic Local File Detection
+- Detects local image file paths in user messages
+- Supports both relative and absolute paths  
+- Recognizes common image extensions: `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.bmp`, `.svg`
+
+### 🚀 NEW: Agentic Loop Image Loading
+- **Automatic detection**: Agent automatically detects when it mentions image files in its internal thinking
+- **Smart loading**: Images are loaded and added to the AI context for subsequent iterations
+- **Persistent context**: Loaded images remain available throughout the conversation
+- **Tool result processing**: Images mentioned in tool outputs are also automatically loaded
+- **Caching**: Prevents reloading the same images multiple times
+
+### Security Features
+- Path validation to prevent directory traversal attacks
+- Restricts file access to allowed directories (respects `ALLOWED_FOLDERS` environment variable)
+- Validates file existence and readability before processing
+
+### Supported Path Formats
+```
+./image.png                    # Relative path from current directory
+../assets/screenshot.jpg       # Relative path with directory traversal
+/absolute/path/to/image.webp   # Absolute path
+image.gif                      # File in current directory
+```
+
+### Automatic Conversion
+- Local files are automatically converted to base64 data URLs
+- Maintains original MIME type based on file extension
+- Seamlessly integrates with existing URL and base64 image support
+
+## Usage Examples
+
+### Basic Usage
+```javascript
+import { ProbeChat } from './probeChat.js';
+
+const chat = new ProbeChat({ debug: true });
+
+// The agent will automatically detect and process the local image
+const response = await chat.chat('Analyze this screenshot: ./screenshot.png');
+```
+
+### Mixed Content
+```javascript
+// Mix local files with URLs
+const message = `
+  Compare this local image ./local.png 
+  with this remote image https://example.com/remote.jpg
+`;
+
+const response = await chat.chat(message);
+```
+
+### Direct Function Usage
+```javascript
+import { extractImageUrls } from './probeChat.js';
+
+const message = 'Please review this diagram: ./architecture.png';
+const result = await extractImageUrls(message, true);
+
+console.log(`Found ${result.urls.length} images`);
+console.log(`Cleaned message: "${result.cleanedMessage}"`);
+```
+
+## 🤖 Agentic Loop Integration
+
+The most powerful feature is automatic image loading during the agent's internal reasoning process.
+
+### How It Works
+
+When the probe agent is working through a task, it can now:
+
+1. **Mention an image file** in its reasoning: "I need to check ./screenshot.png"
+2. **Automatically load the image** before the next AI iteration
+3. **Use visual context** for enhanced analysis and problem-solving
+
+### Agentic Flow Example
+
+```
+👤 USER: "Analyze the system architecture"
+
+🤖 AGENT: "Let me search for architecture documentation..."
+   🔍 Tool: search "architecture design"
+   📊 Result: "Found ./docs/system-diagram.png"
+   
+🤖 AGENT: "I found a system diagram at ./docs/system-diagram.png. Let me analyze it."
+   🖼️  AUTO: Image ./docs/system-diagram.png loaded into context
+   
+🤖 AGENT: "Based on the diagram I can see..." 
+   💭 AI now has visual access to the diagram and can analyze it
+```
+
+### Trigger Patterns
+
+The agent automatically loads images when it mentions:
+
+- **Direct paths**: `./screenshot.png`, `/path/to/image.jpg`
+- **Contextual references**: "the file diagram.png shows", "looking at chart.gif"  
+- **Tool results**: When tools return paths to image files
+- **Generated content**: "saved visualization as ./output.png"
+
+### Benefits
+
+- **🧠 Enhanced reasoning**: Agent gains visual understanding of referenced images
+- **🔄 Seamless workflow**: No manual image loading required
+- **⚡ Performance**: Intelligent caching prevents reloading
+- **🔒 Security**: Same security validations as manual loading
+- **📱 Persistence**: Images remain available throughout the conversation
+
+## Security Considerations
+
+### Path Restrictions
+- Files must be within the allowed directory structure
+- Prevents access to system files (e.g., `/etc/passwd`)
+- Respects the `ALLOWED_FOLDERS` environment variable
+
+### File Validation
+- Verifies file existence before attempting to read
+- Validates file extensions against supported image formats
+- Handles file reading errors gracefully
+
+### Error Handling
+- Failed file reads are logged but don't interrupt processing
+- Invalid paths are silently ignored
+- Maintains functionality for valid images even if some fail
+
+## Implementation Details
+
+### Pattern Matching
+The system uses an enhanced regex pattern to detect:
+```javascript
+/(?:data:image\/[a-zA-Z]*;base64,[A-Za-z0-9+/=]+|https?:\/\/(?:(?:private-user-images\.githubusercontent\.com|github\.com\/user-attachments\/assets)\/[^\s"'<>]+|[^\s"'<>]+\.(?:png|jpg|jpeg|webp|gif)(?:\?[^\s"'<>]*)?)|(?:\.?\.?\/)?[^\s"'<>]*\.(?:png|jpg|jpeg|webp|gif))/gi
+```
+
+### Processing Pipeline
+1. **Pattern Detection** - Find all potential image references in text
+2. **Classification** - Distinguish between URLs, base64 data, and local paths
+3. **Validation** - Verify local file paths for security and existence
+4. **Conversion** - Read local files and convert to base64 data URLs
+5. **Integration** - Pass processed images to AI models
+
+### File Size Limitations
+- No explicit file size limits implemented
+- Memory usage scales with image size
+- Large images may impact performance
+
+## Testing
+
+Run the test suite to verify functionality:
+```bash
+cd examples/chat
+node test-local-image-reading.js
+```
+
+The test covers:
+- Basic local file detection and conversion
+- Mixed URL and local file processing
+- Relative path handling
+- Security validation
+- Error handling for missing files
+
+## Backward Compatibility
+
+This enhancement is fully backward compatible:
+- Existing URL-based image handling unchanged
+- Base64 data URL support maintained
+- No breaking changes to existing APIs
+
+## Environment Configuration
+
+Set allowed folders to restrict file access:
+```bash
+export ALLOWED_FOLDERS="/path/to/project,/path/to/assets"
+```
+
+If no `ALLOWED_FOLDERS` is set, defaults to current working directory.
+
+## Error Handling
+
+The system gracefully handles various error conditions:
+- **File not found**: Logged and ignored
+- **Permission denied**: Logged and ignored  
+- **Invalid format**: Logged and ignored
+- **Path traversal attempts**: Blocked by security validation
+
+Enable debug mode to see detailed logging:
+```javascript
+const chat = new ProbeChat({ debug: true });
+```

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@probelabs/probe-chat",
-  "version": "0.6.0-rc102",
+  "version": "0.6.0-rc103",
   "description": "CLI and web interface for Probe code search (formerly @probelabs/probe-web and @probelabs/probe-chat)",
   "main": "index.js",
   "type": "module",
@@ -96,6 +96,7 @@
     "logo.png",
     "README.md",
     "TRACING.md",
+    "LOCAL_IMAGE_SUPPORT.md",
     "LICENSE"
   ]
 }

package/probeChat.js

@@ -2,7 +2,8 @@ import 'dotenv/config';
 import { ProbeAgent } from '@probelabs/probe/agent';
 import { TokenUsageDisplay } from './tokenUsageDisplay.js';
 import { writeFileSync, existsSync } from 'fs';
-import { join } from 'path';
+import { readFile, stat } from 'fs/promises';
+import { join, resolve, isAbsolute } from 'path';
 import { TelemetryConfig } from './telemetry.js';
 import { trace } from '@opentelemetry/api';
 import { appTracer } from './appTracer.js';
@@ -39,24 +40,128 @@ if (typeof process !== 'undefined' && !process.env.PROBE_CHAT_SKIP_FOLDER_VALIDA
   validateFolders();
 }
 
+// Maximum image file size (20MB) to prevent OOM attacks
+const MAX_IMAGE_FILE_SIZE = 20 * 1024 * 1024;
+
+/**
+ * Security validation for local file paths
+ * @param {string} filePath - The file path to validate
+ * @param {string} baseDir - The base directory to restrict access to
+ * @returns {boolean} - Whether the path is safe to access
+ */
+function isSecureFilePath(filePath, baseDir = process.cwd()) {
+  try {
+    // Resolve the absolute path
+    const absolutePath = isAbsolute(filePath) ? filePath : resolve(baseDir, filePath);
+    const normalizedBase = resolve(baseDir);
+    
+    // Ensure the resolved path is within the allowed directory
+    return absolutePath.startsWith(normalizedBase);
+  } catch (error) {
+    return false;
+  }
+}
+
 /**
- * Extract image URLs from message text
+ * Convert local image file to base64 data URL
+ * @param {string} filePath - Path to the image file
+ * @param {boolean} debug - Whether to log debug information
+ * @returns {Promise<string|null>} - Base64 data URL or null if failed
+ */
+async function convertImageFileToBase64(filePath, debug = false) {
+  try {
+    // Security check: validate the file path against all allowed directories
+    const allowedDirs = allowedFolders.length > 0 ? allowedFolders : [process.cwd()];
+    const isPathAllowed = allowedDirs.some(dir => isSecureFilePath(filePath, dir));
+    
+    if (!isPathAllowed) {
+      if (debug) {
+        console.log(`[DEBUG] Security check failed for path: ${filePath}`);
+      }
+      return null;
+    }
+
+    // Resolve the path - for relative paths, use the first allowed directory as base
+    const baseDir = allowedDirs[0];
+    const absolutePath = isAbsolute(filePath) ? filePath : resolve(baseDir, filePath);
+    
+    // Check if file exists and get file stats
+    let fileStats;
+    try {
+      fileStats = await stat(absolutePath);
+    } catch (error) {
+      if (debug) {
+        console.log(`[DEBUG] File not found: ${absolutePath}`);
+      }
+      return null;
+    }
+
+    // Validate file size to prevent OOM attacks
+    if (fileStats.size > MAX_IMAGE_FILE_SIZE) {
+      if (debug) {
+        console.log(`[DEBUG] Image file too large: ${absolutePath} (${fileStats.size} bytes, max: ${MAX_IMAGE_FILE_SIZE})`);
+      }
+      return null;
+    }
+
+    // Determine MIME type based on file extension
+    const extension = absolutePath.toLowerCase().split('.').pop();
+    const mimeTypes = {
+      'png': 'image/png',
+      'jpg': 'image/jpeg', 
+      'jpeg': 'image/jpeg',
+      'webp': 'image/webp',
+      'gif': 'image/gif'
+    };
+    
+    const mimeType = mimeTypes[extension];
+    if (!mimeType) {
+      if (debug) {
+        console.log(`[DEBUG] Unsupported image format: ${extension}`);
+      }
+      return null;
+    }
+
+    // Read file and convert to base64 asynchronously
+    const fileBuffer = await readFile(absolutePath);
+    const base64Data = fileBuffer.toString('base64');
+    const dataUrl = `data:${mimeType};base64,${base64Data}`;
+    
+    if (debug) {
+      console.log(`[DEBUG] Successfully converted ${absolutePath} to base64 (${fileBuffer.length} bytes)`);
+    }
+    
+    return dataUrl;
+  } catch (error) {
+    if (debug) {
+      console.log(`[DEBUG] Error converting file to base64: ${error.message}`);
+    }
+    return null;
+  }
+}
+
+// Export the extractImageUrls function for testing
+export { extractImageUrls };
+
+/**
+ * Extract image URLs and local file paths from message text
  * @param {string} message - The message text to analyze
  * @param {boolean} debug - Whether to log debug information
- * @returns {Array} Array of { url: string, cleanedMessage: string }
+ * @returns {Promise<Object>} Promise resolving to { urls: Array, cleanedMessage: string }
  */
-function extractImageUrls(message, debug = false) {
+async function extractImageUrls(message, debug = false) {
   // This function should be called within the session context, so it will inherit the trace ID
   const tracer = trace.getTracer('probe-chat', '1.0.0');
-  return tracer.startActiveSpan('content.image.extract', (span) => {
+  return tracer.startActiveSpan('content.image.extract', async (span) => {
     try {
-      // Pattern to match image URLs and base64 data:
+      // Pattern to match image URLs, base64 data, and local file paths:
       // 1. GitHub private-user-images URLs (always images, regardless of extension)
       // 2. GitHub user-attachments/assets URLs (always images, regardless of extension)
       // 3. URLs with common image extensions (PNG, JPG, JPEG, WebP, GIF)
       // 4. Base64 data URLs (data:image/...)
+      // 5. Local file paths with image extensions (relative and absolute)
       // Updated to stop at quotes, spaces, or common HTML/XML delimiters
-      const imageUrlPattern = /(?:data:image\/[a-zA-Z]*;base64,[A-Za-z0-9+/=]+|https?:\/\/(?:(?:private-user-images\.githubusercontent\.com|github\.com\/user-attachments\/assets)\/[^\s"'<>]+|[^\s"'<>]+\.(?:png|jpg|jpeg|webp|gif)(?:\?[^\s"'<>]*)?))/gi;
+      const imageUrlPattern = /(?:data:image\/[a-zA-Z]*;base64,[A-Za-z0-9+/=]+|https?:\/\/(?:(?:private-user-images\.githubusercontent\.com|github\.com\/user-attachments\/assets)\/[^\s"'<>]+|[^\s"'<>]+\.(?:png|jpg|jpeg|webp|gif)(?:\?[^\s"'<>]*)?)|(?:\.?\.?\/)?[^\s"'<>]*\.(?:png|jpg|jpeg|webp|gif))/gi;
 
       span.setAttributes({
         'message.length': message.length,
@@ -69,31 +174,57 @@ function extractImageUrls(message, debug = false) {
       }
 
       const urls = [];
+      const foundPatterns = [];
       let match;
 
       while ((match = imageUrlPattern.exec(message)) !== null) {
-        urls.push(match[0]);
+        foundPatterns.push(match[0]);
         if (debug) {
-          console.log(`[DEBUG] Found image URL: ${match[0]}`);
+          console.log(`[DEBUG] Found image pattern: ${match[0]}`);
+        }
+      }
+
+      // Process each found pattern - convert local files to base64, keep URLs as-is
+      for (const pattern of foundPatterns) {
+        // Check if it's already a URL or base64 data
+        if (pattern.startsWith('http') || pattern.startsWith('data:image/')) {
+          urls.push(pattern);
+          if (debug) {
+            console.log(`[DEBUG] Using URL/base64 as-is: ${pattern.substring(0, 50)}...`);
+          }
+        } else {
+          // It's a local file path - convert to base64
+          const base64Data = await convertImageFileToBase64(pattern, debug);
+          if (base64Data) {
+            urls.push(base64Data);
+            if (debug) {
+              console.log(`[DEBUG] Converted local file ${pattern} to base64`);
+            }
+          } else {
+            if (debug) {
+              console.log(`[DEBUG] Failed to convert local file: ${pattern}`);
+            }
+          }
         }
       }
 
-      // Clean the message by removing found URLs
+      // Clean the message by removing found patterns
       let cleanedMessage = message;
-      urls.forEach(url => {
-        cleanedMessage = cleanedMessage.replace(url, '').trim();
+      foundPatterns.forEach(pattern => {
+        cleanedMessage = cleanedMessage.replace(pattern, '').trim();
       });
 
       // Clean up any remaining extra whitespace
       cleanedMessage = cleanedMessage.replace(/\s+/g, ' ').trim();
 
       span.setAttributes({
-        'images.found': urls.length,
+        'patterns.found': foundPatterns.length,
+        'images.processed': urls.length,
         'message.cleaned_length': cleanedMessage.length
       });
 
       if (debug) {
-        console.log(`[DEBUG] Extracted ${urls.length} image URLs`);
+        console.log(`[DEBUG] Found ${foundPatterns.length} patterns, processed ${urls.length} images`);
         console.log(`[DEBUG] Cleaned message length: ${cleanedMessage.length}`);
       }
 
@@ -163,7 +294,7 @@ export class ProbeChat {
     let cleanedMessage = message;
 
     if (!images.length) {
-      const extracted = extractImageUrls(message, this.debug);
+      const extracted = await extractImageUrls(message, this.debug);
       images = extracted.urls;
       cleanedMessage = extracted.cleanedMessage;