ragdoll-cli 0.0.2

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: f19f7c89cad761c7da655010fdc868b7124ab164d0481cf2d97dea485df58317
+  data.tar.gz: c2c7be844cc5addcd4386c4ea0596c75abe9cbbb29167b218ee95826a9c98ace
+SHA512:
+  metadata.gz: 27c20b905c6fa11a5941a8c1343b01b37affd6d84fd4e3884733988031c2e91f7336c351fc7da90474c68bc4b951cae37b3b3d2d6f85e25f010d19d3110ef43a
+  data.tar.gz: 121b9df6c28f2ed96ddfaf2f00468b7f53aced41b04ed77dca015ec8690e801e3fd10dfc8567737b3448835bb7010234b11630e53a68a494064b8f089a3654ba

data/README.md

@@ -0,0 +1,296 @@
+<div align="center" style="background-color: yellow; color: black; padding: 20px; margin: 20px 0; border: 2px solid black; font-size: 48px; font-weight: bold;">
+  ⚠️ CAUTION ⚠️<br />
+  Software Under Development by a Crazy Man
+</div>
+<br />
+<div align="center">
+  <table>
+    <tr>
+      <td width="50%">
+        <a href="https://research.ibm.com/blog/retrieval-augmented-generation-RAG" target="_blank">
+          <img src="ragdoll-cli.png" alt="Ragdoll CLI Putting the Puzzle Together" width="800">
+        </a>
+      </td>
+      <td width="50%" valign="top">
+        <p>Multi-modal RAG (Retrieval-Augmented Generation) is an architecture that integrates multiple data types (such as text, images, and audio) to enhance AI response generation. It combines retrieval-based methods, which fetch relevant information from a knowledge base, with generative large language models (LLMs) that create coherent and contextually appropriate outputs. This approach allows for more comprehensive and engaging user interactions, such as chatbots that respond with both text and images or educational tools that incorporate visual aids into learning materials. By leveraging various modalities, multi-modal RAG systems improve context understanding and user experience.</p>
+      </td>
+    </tr>
+  </table>
+</div>
+
+# Ragdoll::CLI
+
+Standalone command-line interface for the Ragdoll RAG (Retrieval-Augmented Generation) system. Provides document import, search, and management capabilities through a simple CLI.
+
+## Installation
+
+```bash
+gem install ragdoll-cli
+```
+
+This will install the `ragdoll` command-line tool.
+
+## Quick Start
+
+1. **Initialize configuration:**
+   ```bash
+   ragdoll init
+   ```
+
+2. **Set your API key:**
+   ```bash
+   export OPENAI_API_KEY=your_api_key_here
+   ```
+
+3. **Import documents:**
+   ```bash
+   ragdoll import "docs/*.pdf" --recursive
+   ```
+
+4. **Search for content:**
+   ```bash
+   ragdoll search "What is machine learning?"
+   ```
+
+## Commands
+
+### Configuration
+
+```bash
+# Initialize configuration
+ragdoll init
+
+# Show current configuration
+ragdoll config show
+
+# Set configuration values
+ragdoll config set llm_provider openai
+ragdoll config set chunk_size 1000
+
+# Get configuration values
+ragdoll config get embedding_model
+
+# Show config file path
+ragdoll config path
+
+# Show database configuration and status
+ragdoll config database
+```
+
+### Document Import
+
+```bash
+# Import files matching a pattern
+ragdoll import "documents/*.pdf"
+
+# Import recursively from directory
+ragdoll import "docs/**/*" --recursive
+
+# Filter by document type
+ragdoll import "files/*" --type pdf
+
+# Available types: pdf, docx, txt, md, html
+```
+
+### Search
+
+```bash
+# Basic search
+ragdoll search "machine learning concepts"
+
+# Limit number of results
+ragdoll search "AI algorithms" --limit 5
+
+# Different output formats
+ragdoll search "deep learning" --format json
+ragdoll search "AI" --format plain
+ragdoll search "ML" --format table  # default
+```
+
+### Document Management
+
+```bash
+# Add a single document
+ragdoll add <path>
+
+# List all documents
+ragdoll list
+
+# Limit number of documents shown
+ragdoll list --limit 10
+
+# Different output formats
+ragdoll list --format json
+ragdoll list --format plain
+
+# Check document status
+ragdoll status <id>
+
+# Update document metadata
+ragdoll update <id> --title "New Title"
+
+# Delete a document
+ragdoll delete <id>
+ragdoll delete <id> --force  # Bypass confirmation
+
+# Show system statistics
+ragdoll stats
+ragdoll stats --format json
+ragdoll stats --format plain
+```
+
+### Retrieval Utilities
+
+```bash
+# Get context for RAG applications
+ragdoll context "<query>" --limit 5
+
+# Enhance a prompt with context
+ragdoll enhance "<prompt>" --context_limit 5
+```
+
+### Utilities
+
+```bash
+# Show version information
+ragdoll version
+
+# Show help
+ragdoll help
+ragdoll help import  # Help for specific command
+
+# Check system health
+ragdoll health
+```
+
+## Configuration
+
+The CLI uses a YAML configuration file located at `~/.ragdoll/config.yml`. You can customize various settings:
+
+```yaml
+llm_provider: openai
+embedding_provider: openai
+embedding_model: text-embedding-3-small
+chunk_size: 1000
+chunk_overlap: 200
+search_similarity_threshold: 0.7
+max_search_results: 10
+storage_backend: file
+storage_config:
+  directory: "~/.ragdoll"
+api_keys:
+  openai: your_key_here
+  anthropic: your_key_here
+```
+
+### Environment Variables
+
+API keys can be set via environment variables (recommended):
+
+```bash
+export OPENAI_API_KEY=your_key_here
+export ANTHROPIC_API_KEY=your_key_here
+export GOOGLE_API_KEY=your_key_here
+export AZURE_OPENAI_API_KEY=your_key_here
+export HUGGINGFACE_API_KEY=your_key_here
+export OLLAMA_ENDPOINT=http://localhost:11434
+```
+
+### Custom Configuration Location
+
+```bash
+export RAGDOLL_CONFIG=/path/to/custom/config.yml
+```
+
+## Storage
+
+Documents and embeddings are stored in a PostgreSQL database managed by the `ragdoll-core` gem for production performance. Configuration and log files are stored locally in `~/.ragdoll/`:
+
+- `~/.ragdoll/config.yml` - Configuration settings
+- `~/.ragdoll/ragdoll.log` - Log file (if configured)
+
+## Supported Document Types
+
+- **PDF files** (`.pdf`) - Extracts text and metadata
+- **Microsoft Word** (`.docx`) - Extracts text, tables, and metadata
+- **Text files** (`.txt`) - Plain text import
+- **Markdown** (`.md`, `.markdown`) - Markdown document import
+- **HTML** (`.html`, `.htm`) - Strips HTML tags and imports text
+
+## Examples
+
+### Import a directory of documentation
+
+```bash
+# Import all markdown files from a docs directory
+ragdoll import "docs/**/*.md" --recursive
+
+# Import mixed document types
+ragdoll import "knowledge-base/*" --recursive
+```
+
+### Search and get enhanced prompts
+
+```bash
+# Basic search
+ragdoll search "How to configure SSL certificates?"
+
+# Get detailed results
+ragdoll search "database optimization" --format plain --limit 3
+```
+
+### Manage your knowledge base
+
+```bash
+# See what's in your knowledge base
+ragdoll stats
+ragdoll list --limit 20
+
+# Check status of a specific document
+ragdoll status 123
+
+# Update document title
+ragdoll update 123 --title "Updated Document Title"
+
+# Delete a document
+ragdoll delete 123
+```
+
+## Integration with Other Tools
+
+The CLI is designed to work well with other command-line tools:
+
+```bash
+# Search and pipe to jq for JSON processing
+ragdoll search "API documentation" --format json | jq '.results[0].content'
+
+# Import files found by find command
+find ./docs -name "*.pdf" -exec ragdoll import {} \;
+
+# Use with xargs for batch processing
+ls *.md | xargs -I {} ragdoll import {}
+```
+
+## Troubleshooting
+
+### Common Issues
+
+1. **No API key configured:**
+   ```
+   Error: Missing API key
+   Solution: Set OPENAI_API_KEY environment variable or add to config
+   ```
+
+2. **No documents found:**
+   ```
+   ragdoll stats  # Check if documents are imported
+   ragdoll list   # See what documents exist
+   ```
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/MadBomber/ragdoll-cli.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/Rakefile

@@ -0,0 +1,21 @@
+# frozen_string_literal: true
+
+require 'simplecov'
+SimpleCov.start
+
+# Suppress bundler/rubygems warnings
+$VERBOSE = nil
+
+require "bundler/gem_tasks"
+require "rake/testtask"
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << "test"
+  t.libs << "lib"
+  t.test_files = FileList["test/**/*_test.rb"]
+end
+
+# Load annotate tasks
+Dir.glob("lib/tasks/*.rake").each { |r| load r }
+
+task default: :test

data/bin/ragdoll

@@ -0,0 +1,7 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "ragdoll"
+require_relative "../lib/ragdoll/cli"
+
+Ragdoll::CLI::Main.start(ARGV)

data/lib/ragdoll/cli/commands/config.rb

@@ -0,0 +1,152 @@
+# frozen_string_literal: true
+
+require 'yaml'
+
+module Ragdoll
+  module CLI
+    class Config < Thor
+      desc 'init', 'Initialize Ragdoll configuration'
+      def init
+        loader = ConfigurationLoader.new
+
+        if loader.config_exists?
+          puts "Configuration file already exists at: #{loader.config_path}"
+          if yes?('Overwrite existing configuration?')
+            loader.create_default_config
+            puts "Configuration file created at: #{loader.config_path}"
+          else
+            puts 'Configuration unchanged.'
+            return
+          end
+        else
+          loader.create_default_config
+          puts "Configuration file created at: #{loader.config_path}"
+        end
+
+        puts "\nDefault configuration created with PostgreSQL database."
+        puts 'You may need to:'
+        puts '1. Ensure PostgreSQL is installed and running'
+        puts '2. Create the database: createdb ragdoll_development'
+        puts '3. Set your API keys in environment variables:'
+        puts '   export OPENAI_API_KEY=your_key_here'
+        puts "4. Or add them to the config file under 'api_keys' section"
+        puts '5. For production, update the database configuration:'
+        puts '   ragdoll config set database_config.database ragdoll_production'
+        puts "6. Edit #{loader.config_path} to customize settings"
+      end
+
+      desc 'show', 'Show current configuration'
+      def show
+        loader = ConfigurationLoader.new
+
+        unless loader.config_exists?
+          puts "No configuration file found. Run 'ragdoll config init' to create one."
+          return
+        end
+
+        config = YAML.load_file(loader.config_path)
+        puts "Configuration from: #{loader.config_path}"
+        puts
+        puts YAML.dump(config)
+      end
+
+      desc 'set KEY VALUE', 'Set a configuration value'
+      def set(key, value)
+        loader = ConfigurationLoader.new
+
+        unless loader.config_exists?
+          puts "No configuration file found. Run 'ragdoll config init' to create one."
+          return
+        end
+
+        config = YAML.load_file(loader.config_path)
+
+        # Parse numeric values
+        if value.match?(/^\d+$/)
+          value = value.to_i
+        elsif value.match?(/^\d+\.\d+$/)
+          value = value.to_f
+        elsif value == 'true'
+          value = true
+        elsif value == 'false'
+          value = false
+        end
+
+        # Support nested keys with dot notation
+        keys = key.split('.')
+        current = config
+        keys[0..-2].each do |k|
+          current[k] ||= {}
+          current = current[k]
+        end
+        current[keys.last] = value
+
+        File.write(loader.config_path, YAML.dump(config))
+        puts "Set #{key} = #{value}"
+        puts 'Note: Restart the CLI or reload configuration for changes to take effect.'
+      end
+
+      desc 'get KEY', 'Get a configuration value'
+      def get(key)
+        loader = ConfigurationLoader.new
+
+        unless loader.config_exists?
+          puts "No configuration file found. Run 'ragdoll config init' to create one."
+          return
+        end
+
+        config = YAML.load_file(loader.config_path)
+
+        # Support nested keys with dot notation
+        keys = key.split('.')
+        value = config
+        keys.each do |k|
+          value = value[k] if value.is_a?(Hash)
+        end
+
+        puts "#{key} = #{value}"
+      end
+
+      desc 'path', 'Show configuration file path'
+      def path
+        loader = ConfigurationLoader.new
+        puts loader.config_path
+      end
+
+      desc 'database', 'Show database configuration and status'
+      def database
+        loader = ConfigurationLoader.new
+
+        unless loader.config_exists?
+          puts "No configuration file found. Run 'ragdoll config init' to create one."
+          return
+        end
+
+        config = YAML.load_file(loader.config_path)
+        db_config = config['database_config']
+
+        puts 'Database Configuration:'
+        puts "  Adapter: #{db_config['adapter']}"
+        puts "  Database: #{db_config['database']}"
+        puts "  Auto-migrate: #{db_config['auto_migrate']}"
+
+        if db_config['adapter'] == 'postgresql'
+          puts "  Host: #{db_config['host'] || 'localhost'}"
+          puts "  Port: #{db_config['port'] || 5432}"
+          puts "  Username: #{db_config['username']}"
+        end
+
+        begin
+          client = StandaloneClient.new
+          if client.healthy?
+            puts "\nDatabase Status: ✓ Connected"
+          else
+            puts "\nDatabase Status: ✗ Connection failed"
+          end
+        rescue StandardError => e
+          puts "\nDatabase Status: ✗ Error - #{e.message}"
+        end
+      end
+    end
+  end
+end

data/lib/ragdoll/cli/commands/delete.rb

@@ -0,0 +1,37 @@
+# frozen_string_literal: true
+
+module Ragdoll
+  module CLI
+    class Delete
+      def call(id, options)
+        client = StandaloneClient.new
+
+        puts "Deleting document ID: #{id}"
+        puts "Options: #{options.to_h}" unless options.to_h.empty?
+        puts
+
+        unless options[:force]
+          puts "Are you sure you want to delete document ID #{id}? This action cannot be undone."
+          return unless yes?('Confirm deletion?')
+        end
+
+        result = client.delete_document(id: id)
+
+        if result[:success]
+          puts "Document ID #{id} deleted successfully."
+          puts result[:message] if result[:message]
+        else
+          puts "Failed to delete document ID #{id}."
+          puts result[:message] if result[:message]
+        end
+      end
+
+      private
+
+      def yes?(question)
+        require 'highline/import'
+        agree("#{question} (y/n) ")
+      end
+    end
+  end
+end

data/lib/ragdoll/cli/commands/health.rb

@@ -0,0 +1,22 @@
+# frozen_string_literal: true
+
+module Ragdoll
+  module CLI
+    class Health
+      def call(_options)
+        client = StandaloneClient.new
+
+        puts 'Checking system health'
+        puts
+
+        if client.healthy?
+          puts 'System Status: ✓ Healthy'
+          puts 'The Ragdoll system is operational.'
+        else
+          puts 'System Status: ✗ Unhealthy'
+          puts 'There may be issues with the database or configuration.'
+        end
+      end
+    end
+  end
+end

data/lib/ragdoll/cli/commands/list.rb

@@ -0,0 +1,57 @@
+# frozen_string_literal: true
+
+require 'json'
+
+module Ragdoll
+  module CLI
+    class List
+      def call(options)
+        client = StandaloneClient.new
+
+        puts 'Listing documents'
+        puts "Options: #{options.to_h}" unless options.to_h.empty?
+        puts
+
+        list_options = {}
+        list_options[:limit] = options[:limit] if options[:limit]
+
+        documents = client.list_documents(**list_options)
+
+        if documents.empty?
+          puts 'No documents found.'
+          return
+        end
+
+        case options[:format]
+        when 'json'
+          puts JSON.pretty_generate(documents)
+        when 'plain'
+          documents.each_with_index do |doc, index|
+            puts "#{index + 1}. #{doc[:title] || 'Untitled'}"
+            puts "   ID: #{doc[:id]}"
+            puts "   Status: #{doc[:status] || 'N/A'}"
+            puts
+          end
+        else
+          # Table format (default)
+          puts "Found #{documents.length} documents:"
+          puts
+          puts 'Rank'.ljust(5) + 'Title'.ljust(30) + 'ID'.ljust(10) + 'Status'
+          puts '-' * 60
+
+          documents.each_with_index do |doc, index|
+            rank = (index + 1).to_s.ljust(5)
+            title = (doc[:title] || 'Untitled')[0..29].ljust(30)
+            id = doc[:id].to_s.ljust(10)
+            status = doc[:status] || 'N/A'
+
+            puts "#{rank}#{title}#{id}#{status}"
+          end
+
+          puts
+          puts 'Use --format=json for complete results or --format=plain for detailed view'
+        end
+      end
+    end
+  end
+end

data/lib/ragdoll/cli/commands/search.rb

@@ -0,0 +1,88 @@
+# frozen_string_literal: true
+
+require 'json'
+
+module Ragdoll
+  module CLI
+    class Search
+      def call(query, options)
+        client = StandaloneClient.new
+
+        puts "Searching for: #{query}"
+        puts "Options: #{options.to_h}" unless options.to_h.empty?
+        puts
+
+        search_options = {}
+        search_options[:limit] = options[:limit] if options[:limit]
+        search_options[:content_type] = options[:content_type] if options[:content_type]
+        search_options[:classification] = options[:classification] if options[:classification]
+        search_options[:keywords] = options[:keywords].split(',').map(&:strip) if options[:keywords]
+        search_options[:tags] = options[:tags].split(',').map(&:strip) if options[:tags]
+
+        search_response = client.search(query, **search_options)
+        
+        # Extract the actual results array from the response
+        results = search_response[:results] || search_response['results'] || []
+
+        if results.empty?
+          total = search_response[:total_results] || search_response['total_results'] || 0
+          puts "No results found for '#{query}'"
+          puts "(Total documents in system: #{total})" if total > 0
+          puts "Try adjusting your search terms or check if documents have been processed."
+          return
+        end
+
+        case options[:format]
+        when 'json'
+          puts JSON.pretty_generate(search_response)
+        when 'plain'
+          results.each_with_index do |result, index|
+            title = safe_string_value(result, [:title, :document_title], 'Untitled')
+            content = safe_string_value(result, [:content, :text], '')
+            puts "#{index + 1}. #{title}"
+            puts "   ID: #{result[:document_id] || result[:id]}"
+            puts "   Similarity: #{result[:similarity]&.round(3) || 'N/A'}"
+            puts "   Content: #{content[0..200]}..."
+            puts
+          end
+        else
+          # Table format (default)
+          puts "Found #{results.length} results:"
+          puts
+          puts 'Rank'.ljust(5) + 'Title'.ljust(30) + 'Similarity'.ljust(12) + 'Content Preview'
+          puts '-' * 80
+
+          results.each_with_index do |result, index|
+            rank = (index + 1).to_s.ljust(5)
+            title = safe_string_value(result, [:title, :document_title], 'Untitled')[0..29].ljust(30)
+            similarity = (result[:similarity]&.round(3) || 'N/A').to_s.ljust(12)
+            content = safe_string_value(result, [:content, :text], '')[0..50]
+            content += '...' if content.length == 50
+
+            puts "#{rank}#{title}#{similarity}#{content}"
+          end
+
+          puts
+          puts 'Use --format=json for complete results or --format=plain for detailed view'
+        end
+      end
+
+      private
+
+      def safe_string_value(obj, keys, default)
+        return default.to_s unless obj.respond_to?(:[])
+        
+        keys.each do |key|
+          begin
+            value = obj[key] || obj[key.to_s]
+            return value.to_s if value
+          rescue TypeError, NoMethodError
+            # Skip this key if access fails
+            next
+          end
+        end
+        default.to_s
+      end
+    end
+  end
+end