universal_document_processor 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 35c7ce3be00b697398ed7be3f364c2b9b29002cc763265c9a9b29519dceb2439
+  data.tar.gz: 0c9636fdf3323e5629da41e62747dbabdf0af369e359bf1096cceed5642ee248
+SHA512:
+  metadata.gz: a9f33acf6694f950ae5e4dff14a1adc2b40a51b8fe95311598c51fd5cdf2fa762700cbe078fa8589b39f170a1a98136a065b0f9fa94dd92e832a497d1c817249
+  data.tar.gz: facc2f1555429de4aa5076c570d8547022b791a34e55bd54350225dea73c17f968eae063bc83520f89702c97c8bfec150b1c4175f39322bad1071aa3ed09bfea

data/AI_USAGE_GUIDE.md

@@ -0,0 +1,404 @@
+# 🤖 Universal Document Processor - AI Agent Usage Guide
+
+## Overview
+
+The Universal Document Processor gem includes powerful AI-powered document analysis capabilities through its built-in **Agentic AI** features. Once you've installed the gem, you can leverage AI to analyze, summarize, extract information, and interact with your documents intelligently.
+
+## 🚀 Quick Setup
+
+### 1. Install the Gem
+
+```bash
+gem install universal_document_processor
+```
+
+### 2. Set Up Your OpenAI API Key
+
+```bash
+# Set environment variable
+export OPENAI_API_KEY="your-openai-api-key-here"
+```
+
+Or pass it directly in your code:
+
+```ruby
+options = { api_key: 'your-openai-api-key-here' }
+```
+
+### 3. Basic AI Usage
+
+```ruby
+require 'universal_document_processor'
+
+# Basic AI analysis
+result = UniversalDocumentProcessor.ai_analyze('document.pdf')
+puts result
+```
+
+## 🧠 AI Features Overview
+
+### Available AI Methods
+
+1. **`ai_analyze`** - Comprehensive document analysis
+2. **`ai_summarize`** - Generate summaries of different lengths
+3. **`ai_extract_info`** - Extract specific information categories
+4. **`ai_translate`** - Translate document content
+5. **`ai_classify`** - Classify document type and purpose
+6. **`ai_insights`** - Generate insights and recommendations
+7. **`ai_action_items`** - Extract actionable items
+8. **`ai_compare`** - Compare multiple documents
+9. **`ai_chat`** - Interactive chat about documents
+
+## 📝 Detailed Usage Examples
+
+### 1. Document Analysis
+
+#### General Analysis
+```ruby
+# Analyze any document comprehensively
+analysis = UniversalDocumentProcessor.ai_analyze('report.pdf')
+puts analysis
+```
+
+#### Specific Query Analysis
+```ruby
+# Ask specific questions about the document
+analysis = UniversalDocumentProcessor.ai_analyze('contract.pdf', {
+  query: "What are the key terms and conditions?"
+})
+puts analysis
+```
+
+### 2. Document Summarization
+
+```ruby
+# Short summary (2-3 sentences)
+summary = UniversalDocumentProcessor.ai_summarize('document.pdf', length: :short)
+
+# Medium summary (1-2 paragraphs) - default
+summary = UniversalDocumentProcessor.ai_summarize('document.pdf', length: :medium)
+
+# Detailed summary
+summary = UniversalDocumentProcessor.ai_summarize('document.pdf', length: :long)
+
+puts summary
+```
+
+### 3. Information Extraction
+
+```ruby
+# Extract default categories
+info = UniversalDocumentProcessor.ai_extract_info('meeting_notes.pdf')
+
+# Extract specific categories
+info = UniversalDocumentProcessor.ai_extract_info('contract.pdf', [
+  'parties', 'dates', 'financial_terms', 'obligations', 'deadlines'
+])
+
+puts info
+```
+
+### 4. Document Translation
+
+```ruby
+# Translate to different languages
+spanish_content = UniversalDocumentProcessor.ai_translate('document.pdf', 'Spanish')
+japanese_content = UniversalDocumentProcessor.ai_translate('document.pdf', 'Japanese')
+french_content = UniversalDocumentProcessor.ai_translate('document.pdf', 'French')
+
+puts spanish_content
+```
+
+### 5. Document Classification
+
+```ruby
+# Classify document type and purpose
+classification = UniversalDocumentProcessor.ai_classify('unknown_document.pdf')
+
+# Returns structured information about document type
+puts classification
+```
+
+### 6. Generate Insights
+
+```ruby
+# Get AI-powered insights and recommendations
+insights = UniversalDocumentProcessor.ai_insights('business_plan.pdf')
+
+# Returns analysis of key themes, recommendations, etc.
+puts insights
+```
+
+### 7. Extract Action Items
+
+```ruby
+# Extract actionable items from documents
+action_items = UniversalDocumentProcessor.ai_action_items('meeting_minutes.pdf')
+
+# Returns structured list of tasks, deadlines, assignments
+puts action_items
+```
+
+### 8. Compare Documents
+
+```ruby
+# Compare multiple documents
+comparison = UniversalDocumentProcessor.ai_compare([
+  'version1.pdf',
+  'version2.pdf',
+  'version3.pdf'
+], :content)
+
+puts comparison
+```
+
+## 🎯 Advanced Usage with Document Objects
+
+### Using Document Objects for More Control
+
+```ruby
+# Create document object for advanced operations
+doc = UniversalDocumentProcessor::Document.new('complex_document.pdf')
+
+# Use AI methods on the document object
+summary = doc.ai_summarize(length: :medium)
+insights = doc.ai_insights
+action_items = doc.ai_action_items
+
+# Interactive chat about the document
+response = doc.ai_chat("What are the main risks mentioned in this document?")
+puts response
+```
+
+### Creating and Reusing AI Agent
+
+```ruby
+# Create an AI agent with custom configuration
+ai_agent = UniversalDocumentProcessor.create_ai_agent({
+  model: 'gpt-4',
+  temperature: 0.7,
+  api_key: 'your-api-key'
+})
+
+# Process document
+doc_result = UniversalDocumentProcessor.process('document.pdf')
+
+# Use AI agent for multiple operations
+summary = ai_agent.summarize_document(doc_result, length: :short)
+insights = ai_agent.generate_insights(doc_result)
+classification = ai_agent.classify_document(doc_result)
+
+# Interactive chat
+response = ai_agent.chat("Tell me about the financial projections", doc_result)
+```
+
+## 🛠️ Configuration Options
+
+### AI Agent Configuration
+
+```ruby
+options = {
+  api_key: 'your-openai-api-key',      # OpenAI API key
+  model: 'gpt-4',                       # AI model to use
+  temperature: 0.7,                     # Response creativity (0.0-1.0)
+  max_history: 10,                      # Conversation history limit
+  base_url: 'https://api.openai.com/v1' # API endpoint
+}
+
+# Use with any AI method
+result = UniversalDocumentProcessor.ai_analyze('document.pdf', options)
+```
+
+## 💡 Use Case Examples
+
+### 1. Legal Document Analysis
+
+```ruby
+# Analyze legal contracts
+contract_analysis = UniversalDocumentProcessor.ai_analyze('contract.pdf', {
+  query: "Extract all key terms, obligations, and potential risks"
+})
+
+# Extract specific legal information
+legal_info = UniversalDocumentProcessor.ai_extract_info('contract.pdf', [
+  'parties', 'effective_date', 'termination_clauses', 'payment_terms', 'liabilities'
+])
+```
+
+### 2. Business Report Processing
+
+```ruby
+# Summarize quarterly reports
+summary = UniversalDocumentProcessor.ai_summarize('q4_report.pdf', length: :medium)
+
+# Extract key business metrics
+metrics = UniversalDocumentProcessor.ai_extract_info('q4_report.pdf', [
+  'revenue', 'expenses', 'profit_margins', 'growth_metrics', 'forecasts'
+])
+
+# Get strategic insights
+insights = UniversalDocumentProcessor.ai_insights('q4_report.pdf')
+```
+
+### 3. Meeting Minutes Processing
+
+```ruby
+# Extract action items from meeting notes
+action_items = UniversalDocumentProcessor.ai_action_items('meeting_notes.pdf')
+
+# Summarize meeting outcomes
+summary = UniversalDocumentProcessor.ai_summarize('meeting_notes.pdf', length: :short)
+
+# Extract key decisions and follow-ups
+decisions = UniversalDocumentProcessor.ai_extract_info('meeting_notes.pdf', [
+  'decisions_made', 'action_items', 'deadlines', 'assigned_people'
+])
+```
+
+### 4. Research Paper Analysis
+
+```ruby
+# Analyze research papers
+analysis = UniversalDocumentProcessor.ai_analyze('research_paper.pdf', {
+  query: "What are the main findings and methodology used?"
+})
+
+# Extract research data
+research_info = UniversalDocumentProcessor.ai_extract_info('research_paper.pdf', [
+  'hypothesis', 'methodology', 'results', 'conclusions', 'future_work'
+])
+```
+
+## 🔄 Interactive Document Chat
+
+```ruby
+# Create document object
+doc = UniversalDocumentProcessor::Document.new('document.pdf')
+
+# Start interactive chat session
+puts "Chat with your document (type 'exit' to quit):"
+
+loop do
+  print "> "
+  user_input = gets.chomp
+  break if user_input.downcase == 'exit'
+  
+  response = doc.ai_chat(user_input)
+  puts "AI: #{response}\n\n"
+end
+```
+
+## 📊 Batch AI Processing
+
+```ruby
+# Process multiple documents with AI
+documents = ['doc1.pdf', 'doc2.docx', 'doc3.xlsx']
+
+# Batch summarization
+summaries = documents.map do |file|
+  {
+    file: file,
+    summary: UniversalDocumentProcessor.ai_summarize(file, length: :short)
+  }
+end
+
+# Batch classification
+classifications = documents.map do |file|
+  {
+    file: file,
+    classification: UniversalDocumentProcessor.ai_classify(file)
+  }
+end
+```
+
+## 🚨 Error Handling
+
+```ruby
+begin
+  result = UniversalDocumentProcessor.ai_analyze('document.pdf')
+  puts result
+rescue ArgumentError => e
+  puts "Configuration error: #{e.message}"
+  puts "Please check your OpenAI API key"
+rescue UniversalDocumentProcessor::ProcessingError => e
+  puts "Processing error: #{e.message}"
+rescue StandardError => e
+  puts "Unexpected error: #{e.message}"
+end
+```
+
+## 🎛️ Environment Variables
+
+Set these environment variables for seamless operation:
+
+```bash
+# Required
+export OPENAI_API_KEY="your-openai-api-key"
+
+# Optional
+export OPENAI_MODEL="gpt-4"
+export OPENAI_TEMPERATURE="0.7"
+export OPENAI_BASE_URL="https://api.openai.com/v1"
+```
+
+## 🔧 Troubleshooting
+
+### Common Issues and Solutions
+
+1. **Missing API Key**
+   ```ruby
+   # Error: ArgumentError: OpenAI API key is required
+   # Solution: Set OPENAI_API_KEY environment variable or pass api_key in options
+   ```
+
+2. **API Rate Limits**
+   ```ruby
+   # Add delays between requests for large batch operations
+   documents.each_with_index do |doc, index|
+     result = UniversalDocumentProcessor.ai_analyze(doc)
+     sleep(1) if index % 10 == 0  # Pause every 10 requests
+   end
+   ```
+
+3. **Large Documents**
+   ```ruby
+   # For very large documents, consider processing in chunks
+   options = { max_content_length: 10000 }
+   result = UniversalDocumentProcessor.ai_analyze('large_doc.pdf', options)
+   ```
+
+## 📚 Best Practices
+
+1. **Optimize API Usage**
+   - Cache results for repeated analysis
+   - Use appropriate summary lengths
+   - Batch similar operations
+
+2. **Security**
+   - Store API keys securely
+   - Don't log sensitive document content
+   - Use environment variables for configuration
+
+3. **Performance**
+   - Process documents in parallel when possible
+   - Use specific queries rather than general analysis
+   - Consider document size when choosing AI operations
+
+## 🎯 Next Steps
+
+1. **Explore Advanced Features**: Try different AI models and temperature settings
+2. **Integrate with Your Application**: Build AI-powered document workflows
+3. **Customize for Your Domain**: Create domain-specific extraction categories
+4. **Scale Your Usage**: Implement batch processing for large document sets
+
+## 📞 Support
+
+For issues with AI functionality:
+1. Check your OpenAI API key and credits
+2. Verify document format compatibility
+3. Review error messages for specific guidance
+4. Consult the main gem documentation for additional features
+
+---
+
+*This guide covers the AI capabilities of the Universal Document Processor gem. The AI features require an OpenAI API key and internet connection to function.* 

data/CHANGELOG.md

@@ -0,0 +1,67 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [1.0.0] - 2025-06-23
+
+### Added
+- Initial release of Universal Document Processor
+- Support for multiple document formats (PDF, DOC, DOCX, XLS, XLSX, PPT, PPTX, TXT, HTML, XML, CSV, Images, Archives)
+- Unified API for document processing across all formats
+- Text extraction from all supported formats
+- Metadata extraction (author, creation date, file size, etc.)
+- Image extraction from documents
+- Table detection and extraction
+- Character validation and encoding support
+- Multi-language support including Japanese (日本語)
+- Invalid character detection and cleaning
+- Japanese filename handling with proper encoding
+- Archive processing (ZIP, RAR, 7Z)
+- Document conversion capabilities
+- Batch processing support
+- Production-ready error handling and fallbacks
+
+### AI Features Added
+- **AI-Powered Document Analysis**: Comprehensive document analysis using OpenAI GPT models
+- **Document Summarization**: Generate summaries of different lengths (short, medium, long)
+- **Information Extraction**: Extract specific categories of information from documents
+- **Document Translation**: Translate document content to any language
+- **Document Classification**: Classify document type and purpose automatically
+- **Insights Generation**: Generate AI-powered insights and recommendations
+- **Action Items Extraction**: Extract actionable items and tasks from documents
+- **Document Comparison**: Compare multiple documents intelligently
+- **Interactive Chat**: Chat with documents using natural language
+- **Conversation Memory**: Maintain conversation history for context
+- **Custom AI Configuration**: Configurable AI models, temperature, and settings
+
+### Features
+- **Processor Classes**: Specialized processors for each document type
+- **Japanese Support**: Full support for Japanese text and filenames
+- **Character Validation**: Detect and clean invalid characters
+- **File Detection**: Intelligent MIME type detection
+- **Extensible Architecture**: Easy to add new processors and formats
+- **Comprehensive Error Handling**: Graceful fallbacks and error recovery
+- **Memory Efficient**: Optimized for large document processing
+- **Thread Safe**: Safe for concurrent processing
+
+### Dependencies
+- activesupport (~> 7.0)
+- marcel (~> 1.0) - MIME type detection
+- nokogiri (~> 1.13) - XML/HTML parsing
+- rubyzip (~> 2.3) - Archive processing
+
+### Optional Dependencies
+- pdf-reader (~> 2.0) - Enhanced PDF processing
+- prawn (~> 2.4) - PDF generation
+- docx (~> 0.8) - Word document processing
+- roo (~> 2.8) - Excel/Spreadsheet processing
+- mini_magick (~> 4.11) - Image processing
+- yomu (~> 0.2) - Universal text extraction fallback
+
+[Unreleased]: https://github.com/vikas-vpatil/universal_document_processor/compare/v1.0.0...HEAD
+[1.0.0]: https://github.com/vikas-vpatil/universal_document_processor/releases/tag/v1.0.0