legal_summariser 0.2.0 → 0.3.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 99da5ab12240efdb658eafc5b3e76ef46834f7a7d76bf86edfe1958ea75c4f58
-  data.tar.gz: aa0ee6b2406771e99c22af8d5ab00145eeee8666a8ccbda9b96c48ed87e0e408
+  metadata.gz: 17172c652ce52a7b5707e22bcec192c56e3538255cc579cdf4d331de5caf2d6e
+  data.tar.gz: 16383b701c48ce0e674edfccd9eece8fa31b6659ca29ff4f073939bf735b7564
 SHA512:
-  metadata.gz: 20d58233629912675fd4fa7a44c0813d1267e25bc0004df18d37c60ed069906f31d8a68cc165c337809ff040e874d41053b347eb4b3df46f98bf85451a1f654d
-  data.tar.gz: f7bc3b2feab8929485a5387e93ecc0762b32d18903460ba5e33ea1a7c3dd010102c8cbe10feff018694c0ea2a9641c6919904da574a041a226d1de0f1134122b
+  metadata.gz: aa1b6fd284af5525e90b8cf15333eb2cfddecc1179142b7f8ea0df306e7a04f92a637e208e985de6517414ec60aa565ba8baba37f993952969a565641ad45d6e
+  data.tar.gz: 638343a4a36085b27561abe93e0374e1e034ad3dd701646d1d3d9d38537c41bb3bef243a32b191526b5237e16a8f933ee9055e51b5dc62b21411752bf40958ca

data/CHANGELOG.md

@@ -5,6 +5,37 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.3.1] - 2025-01-09
+
+### Updated
+- **Gem Metadata**: Updated author information and contact details
+- **RubyGems Page**: Enhanced description and metadata for better discoverability
+- **Links**: Updated all GitHub repository links to correct owner
+- **Documentation**: Added additional metadata links (bug tracker, wiki, documentation)
+
+## [0.3.0] - 2025-01-09
+
+### Added
+- **Plain Language Generator**: AI-powered legal text simplification with fine-tuned models
+- **Model Training System**: Complete training pipeline for custom legal language models
+- **Advanced Multilingual Support**: Enhanced processing for 8 languages with cultural adaptations
+- **PDF Annotation System**: Rich PDF output with highlighting, comments, and risk indicators
+- **AI/ML Integration**: Support for external AI APIs and local model training
+- **Advanced NLP Features**: Readability scoring, complexity reduction metrics, and text analysis
+
+### Enhanced
+- **Legal Text Processing**: 30+ legal jargon mappings and sentence pattern simplification
+- **Cross-Language Translation**: Legal term mapping across multiple languages
+- **Document Analysis**: Enhanced with plain language generation and multilingual processing
+- **Performance Monitoring**: Extended metrics for new AI/ML operations
+- **Error Handling**: Comprehensive error management for AI operations and model training
+
+### Technical Improvements
+- **Model Architecture**: Pattern-based, statistical, and neural model support
+- **Caching System**: Extended for translation and model results
+- **API Integration**: Support for external translation and AI services
+- **Cultural Adaptations**: Legal system-specific processing for different countries
+
 ## [0.2.0] - 2025-01-09
 
 ### Added

data/CONTRIBUTING.md

@@ -0,0 +1,231 @@
+# Contributing to Legal Summariser
+
+Thank you for your interest in contributing to Legal Summariser! This document provides guidelines for contributing to the project.
+
+## 🚀 Getting Started
+
+### Prerequisites
+
+- Ruby 2.6 or higher
+- Bundler gem
+- Git
+
+### Development Setup
+
+1. Fork the repository
+2. Clone your fork:
+   ```bash
+   git clone https://github.com/your-username/legal_summariser.git
+   cd legal_summariser
+   ```
+
+3. Install dependencies:
+   ```bash
+   bundle install
+   ```
+
+4. Run tests to ensure everything works:
+   ```bash
+   bundle exec rspec
+   ```
+
+## 🧪 Testing
+
+We maintain comprehensive test coverage. Please ensure all tests pass before submitting a PR:
+
+```bash
+# Run all tests
+bundle exec rspec
+
+# Run tests with coverage
+bundle exec rspec --format documentation
+
+# Run specific test file
+bundle exec rspec spec/text_extractor_spec.rb
+
+# Run linter
+bundle exec rubocop
+```
+
+### Writing Tests
+
+- Write tests for all new functionality
+- Follow existing test patterns and naming conventions
+- Use descriptive test names that explain the expected behavior
+- Include both positive and negative test cases
+- Test edge cases and error conditions
+
+## 📝 Code Style
+
+We follow Ruby community standards:
+
+- Use 2 spaces for indentation
+- Keep lines under 120 characters
+- Use descriptive variable and method names
+- Add comments for complex logic
+- Follow RuboCop guidelines
+
+Run the linter before submitting:
+```bash
+bundle exec rubocop
+```
+
+## 🔧 Development Guidelines
+
+### Architecture
+
+The gem follows a modular architecture:
+
+- `TextExtractor`: Document parsing and text extraction
+- `Summariser`: Text summarization and key point extraction
+- `ClauseDetector`: Legal clause identification
+- `RiskAnalyzer`: Risk assessment and compliance checking
+- `Formatter`: Output formatting (JSON, Markdown, Text)
+- `Cache`: Result caching system
+- `PerformanceMonitor`: Performance tracking
+- `Configuration`: Gem configuration management
+
+### Adding New Features
+
+1. **Create an issue** describing the feature
+2. **Write tests** for the new functionality
+3. **Implement the feature** following existing patterns
+4. **Update documentation** including README and code comments
+5. **Add examples** if applicable
+6. **Ensure all tests pass**
+
+### Adding New Document Types
+
+To add support for a new document type:
+
+1. Add detection patterns in `detect_document_type` method
+2. Update supported formats documentation
+3. Add test cases for the new format
+4. Update CLI help text if needed
+
+### Adding New Risk Patterns
+
+To add new risk detection patterns:
+
+1. Add patterns to `RiskAnalyzer` class
+2. Include severity levels and recommendations
+3. Add corresponding test cases
+4. Update documentation
+
+## 📚 Documentation
+
+- Update README.md for user-facing changes
+- Add inline documentation for new methods
+- Include examples for new features
+- Update CHANGELOG.md following semantic versioning
+
+## 🐛 Bug Reports
+
+When reporting bugs, please include:
+
+- Ruby version
+- Gem version
+- Operating system
+- Steps to reproduce
+- Expected vs actual behavior
+- Sample files (if applicable and not confidential)
+
+## 💡 Feature Requests
+
+Feature requests should include:
+
+- Clear description of the feature
+- Use case and motivation
+- Proposed implementation approach
+- Potential impact on existing functionality
+
+## 🔄 Pull Request Process
+
+1. **Create a feature branch** from `main`:
+   ```bash
+   git checkout -b feature/your-feature-name
+   ```
+
+2. **Make your changes** following the guidelines above
+
+3. **Commit with descriptive messages**:
+   ```bash
+   git commit -m "Add support for new document format"
+   ```
+
+4. **Push to your fork**:
+   ```bash
+   git push origin feature/your-feature-name
+   ```
+
+5. **Create a Pull Request** with:
+   - Clear title and description
+   - Reference to related issues
+   - List of changes made
+   - Test results
+
+### PR Requirements
+
+- [ ] All tests pass
+- [ ] Code follows style guidelines
+- [ ] Documentation is updated
+- [ ] CHANGELOG.md is updated
+- [ ] No breaking changes (or clearly documented)
+
+## 🏷️ Release Process
+
+Releases follow semantic versioning:
+
+- **MAJOR**: Breaking changes
+- **MINOR**: New features (backward compatible)
+- **PATCH**: Bug fixes (backward compatible)
+
+## 🤝 Code of Conduct
+
+- Be respectful and inclusive
+- Focus on constructive feedback
+- Help others learn and grow
+- Maintain professionalism
+
+## 📞 Getting Help
+
+- Create an issue for bugs or feature requests
+- Join discussions in existing issues
+- Contact maintainers for questions
+
+## 🎯 Areas for Contribution
+
+We welcome contributions in these areas:
+
+### High Priority
+- Additional document format support (ODT, RTF, HTML)
+- Enhanced clause detection patterns
+- Multi-language support improvements
+- Performance optimizations
+
+### Medium Priority
+- Additional risk assessment rules
+- Better error handling and recovery
+- Enhanced caching strategies
+- CLI improvements
+
+### Documentation
+- More usage examples
+- Video tutorials
+- API documentation improvements
+- Translation to other languages
+
+## 🙏 Recognition
+
+Contributors will be:
+- Listed in the README.md
+- Mentioned in release notes
+- Given credit in commit messages
+
+## 📄 License
+
+By contributing, you agree that your contributions will be licensed under the MIT License.
+
+---
+
+Thank you for contributing to Legal Summariser! Your efforts help make legal document analysis more accessible to everyone. 🚀

data/README.md

@@ -6,7 +6,8 @@
 
 [![Ruby](https://img.shields.io/badge/Ruby-2.6+-red.svg)](https://ruby-lang.org)
 [![License](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
-[![Tests](https://img.shields.io/badge/Tests-26%20passing-green.svg)](#testing)
+[![Tests](https://img.shields.io/badge/Tests-100+%20passing-green.svg)](#testing)
+[![Version](https://img.shields.io/badge/Version-0.3.0-blue.svg)](https://rubygems.org/gems/legal_summariser)
 
 ---
 
@@ -27,7 +28,8 @@
 
 ## Features
 
-- **Document Processing**: Supports PDF, DOCX, and plain text files
+### 🚀 Core Analysis
+- **Document Processing**: Supports PDF, DOCX, RTF, and plain text files
 - **Smart Summarisation**: Converts legal documents into concise plain English
 - **Clause Detection**: Automatically identifies key legal clauses including:
   - Data Processing & Privacy (GDPR/KVKK compliance)
@@ -40,8 +42,29 @@
   - Governing Law
 - **Risk Analysis**: Flags potential legal risks and unfair terms
 - **Compliance Checking**: Identifies gaps in GDPR, KVKK, and other regulations
-- **Multiple Output Formats**: JSON, Markdown, and plain text
-- **CLI Interface**: Command-line tool for batch processing
+
+### 🤖 AI/ML Features (v0.3.0)
+- **Plain Language Generator**: AI-powered legal text simplification with 30+ jargon mappings
+- **Model Training System**: Train custom legal language models (pattern-based, statistical, neural)
+- **Readability Scoring**: Calculate complexity reduction and readability metrics
+- **Fine-tuning Support**: Customize models for specific legal domains
+
+### 🌍 Multilingual Support
+- **8 Languages Supported**: English, Spanish, French, German, Italian, Portuguese, Turkish, Dutch
+- **Legal Term Translation**: Cross-language legal terminology mapping
+- **Cultural Adaptations**: Legal system-specific processing for different countries
+- **AI Translation Integration**: Support for external translation APIs
+
+### 📄 Advanced Output
+- **PDF Annotations**: Rich PDF output with highlighting, comments, and risk indicators
+- **Multiple Formats**: JSON, Markdown, plain text, and annotated PDF
+- **Batch Processing**: Process multiple documents simultaneously
+- **Performance Monitoring**: Built-in metrics and caching system
+
+### 🛠️ Developer Tools
+- **CLI Interface**: Comprehensive command-line tool
+- **Configuration System**: Flexible configuration with validation
+- **Caching System**: Result caching with TTL and size management
 - **Offline Processing**: Works without internet for sensitive documents
 
 ## Installation
@@ -73,41 +96,76 @@ require "legal_summariser"
 
 # Basic usage
 summary = LegalSummariser.summarise("contracts/nda.pdf")
-puts summary[:plain_text]
-# => "This Non-Disclosure Agreement establishes confidentiality, valid for 2 years. The company may terminate at any time..."
+puts summary[:plain_text]        # AI-generated plain language version
+puts summary[:summary]           # Original summary
+puts summary[:multilingual]      # Multi-language processing results
 
-# With options
+# Advanced AI features
 result = LegalSummariser.summarise("contract.pdf", {
   format: 'markdown',
-  max_sentences: 3
+  max_sentences: 3,
+  language: 'es',              # Process in Spanish
+  plain_language: true,        # Enable AI plain language generation
+  generate_annotations: true   # Create PDF annotations
 })
 
-# Access different parts of the analysis
+# Access AI-enhanced analysis
 puts result[:key_points]        # Key contract points
 puts result[:clauses]           # Detected legal clauses
 puts result[:risks]             # Risk analysis
-puts result[:metadata]          # Document metadata
+puts result[:plain_text]        # AI-simplified version
+puts result[:multilingual]      # Multi-language results
+puts result[:metadata]          # Enhanced metadata with AI metrics
+
+# Plain Language Generator
+generator = LegalSummariser::PlainLanguageGenerator.new
+plain_result = generator.generate("The party of the first part shall indemnify...")
+puts plain_result[:text]        # "The first party will compensate..."
+puts plain_result[:readability_score]  # Readability improvement metrics
+
+# Model Training
+trainer = LegalSummariser::ModelTrainer.new
+trainer.train_model('contract_model', training_data, type: 'statistical')
+trainer.fine_tune_model('contract_model', fine_tuning_data)
+
+# Multilingual Processing
+processor = LegalSummariser::MultilingualProcessor.new
+result = processor.process_multilingual("contract.pdf", source: 'en', target: 'es')
+
+# PDF Annotations
+annotator = LegalSummariser::PDFAnnotator.new
+annotator.create_annotated_pdf("contract.pdf", analysis_results, "annotated_contract.pdf")
 ```
 
 ### Command Line Interface
 
 ```bash
-# Analyze a document
+# Basic analysis
 legal_summariser analyze contract.pdf
 
-# Specify output format
-legal_summariser analyze contract.pdf --format markdown
+# AI-enhanced analysis with plain language
+legal_summariser analyze contract.pdf --plain-language --format markdown
 
-# Save to file
-legal_summariser analyze contract.pdf --output summary.md --format markdown
+# Multilingual processing
+legal_summariser analyze contract.pdf --language es --translate-to en
 
-# Run demo
-legal_summariser demo
+# Generate annotated PDF
+legal_summariser analyze contract.pdf --annotate --output annotated_contract.pdf
 
-# Show supported formats
-legal_summariser supported_formats
+# Batch processing with AI features
+legal_summariser batch contracts/ --plain-language --multilingual
+
+# Configuration and stats
+legal_summariser config --set language=es
+legal_summariser stats
 
-# Show version
+# Model management
+legal_summariser train-model --type statistical --data training_data.json
+legal_summariser list-models
+
+# Utility commands
+legal_summariser demo
+legal_summariser supported_formats
 legal_summariser version
 ```
 
@@ -176,11 +234,16 @@ The system automatically detects and optimizes analysis for:
 - **Pattern matching**: For compliance gap identification
 
 ### Key Components
-- **TextExtractor**: Multi-format document parsing
-- **Summariser**: Plain English conversion engine
-- **ClauseDetector**: Legal clause identification
-- **RiskAnalyzer**: Risk assessment and flagging
-- **Formatter**: Multi-format output generation
+- **TextExtractor**: Multi-format document parsing (PDF, DOCX, RTF, TXT)
+- **Summariser**: Enhanced plain English conversion engine
+- **ClauseDetector**: Advanced legal clause identification
+- **RiskAnalyzer**: Comprehensive risk assessment and flagging
+- **PlainLanguageGenerator**: AI-powered legal text simplification
+- **ModelTrainer**: Custom model training and fine-tuning system
+- **MultilingualProcessor**: Cross-language processing and translation
+- **PDFAnnotator**: Rich PDF annotation and highlighting
+- **Formatter**: Multi-format output generation (JSON, Markdown, PDF)
+- **Cache & Performance**: Advanced caching and performance monitoring
 
 ## Development
 
@@ -206,9 +269,9 @@ gem install ./legal_summariser-*.gem
 ## Roadmap
 
 - **v0.1** ✅ Text extraction + basic summarisation
-- **v0.2** ✅ Clause detection + risk flagging  
-- **v0.3** 🔄 Plain language generator (fine-tuned models)
-- **v1.0** 📋 Multi-language support + PDF annotation output
+- **v0.2** ✅ Clause detection + risk flagging + performance enhancements
+- **v0.3** ✅ AI/ML features + multilingual support + PDF annotations
+- **v1.0** 📋 Advanced neural models + enterprise features + API service
 
 ## Contributing
 
@@ -234,7 +297,7 @@ This tool is designed to assist with legal document analysis but should not repl
 This project leverages my expertise in:
 
 - **Ruby Development**: Gem architecture, modular design patterns
-- **AI & NLP**: Rule-based text analysis, pattern recognition
+- **AI & NLP**: Advanced machine learning, neural networks, multilingual processing
 - **Cybersecurity**: Compliance frameworks (GDPR, KVKK), risk assessment
 - **Digital Forensics**: Legal document analysis, evidence extraction
 - **Software Engineering**: Test-driven development, CLI tools

data/examples/advanced_configuration.rb

@@ -0,0 +1,195 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+# Example: Advanced configuration and customization
+require 'legal_summariser'
+require 'logger'
+
+puts "=== Advanced Legal Summariser Configuration ==="
+
+# Example 1: Custom logging configuration
+puts "\n1. Custom Logging Setup"
+custom_logger = Logger.new('legal_analysis.log')
+custom_logger.level = Logger::DEBUG
+custom_logger.formatter = proc do |severity, datetime, progname, msg|
+  "[#{datetime.strftime('%Y-%m-%d %H:%M:%S')}] #{severity}: #{msg}\n"
+end
+
+LegalSummariser.configure do |config|
+  config.logger = custom_logger
+  config.language = 'en'
+  config.max_file_size = 20 * 1024 * 1024 # 20MB
+  config.timeout = 60 # 60 seconds
+  config.enable_caching = true
+  config.cache_dir = './custom_cache'
+end
+
+puts "Configuration applied successfully!"
+
+# Example 2: Multi-language support
+puts "\n2. Multi-language Configuration"
+LegalSummariser.configure do |config|
+  config.language = 'tr' # Turkish
+end
+
+puts "Language set to Turkish (TR)"
+puts "Supported languages: #{LegalSummariser.configuration.supported_languages.join(', ')}"
+
+# Example 3: Performance monitoring
+puts "\n3. Performance Monitoring"
+monitor = LegalSummariser.performance_monitor
+
+# Simulate some operations for demonstration
+monitor.start_timer(:demo_operation)
+sleep(0.1) # Simulate work
+monitor.end_timer(:demo_operation)
+
+monitor.record(:demo_metric, 42.5)
+monitor.record(:demo_metric, 38.2)
+
+puts "Performance Report:"
+puts monitor.report
+
+# Example 4: Cache management
+puts "\n4. Cache Management"
+cache = LegalSummariser::Cache.new
+
+# Show cache statistics
+cache_stats = cache.stats
+puts "Cache Status: #{cache_stats[:enabled] ? 'Enabled' : 'Disabled'}"
+
+if cache_stats[:enabled]
+  puts "Cache Directory: #{cache_stats[:cache_dir]}"
+  puts "Cached Files: #{cache_stats[:file_count]}"
+  puts "Cache Size: #{cache_stats[:total_size_mb]} MB"
+end
+
+# Example 5: Error handling and validation
+puts "\n5. Configuration Validation"
+begin
+  LegalSummariser.configure do |config|
+    config.language = 'invalid_language'
+  end
+rescue LegalSummariser::Error => e
+  puts "Configuration error caught: #{e.message}"
+end
+
+# Reset to valid configuration
+LegalSummariser.configure do |config|
+  config.language = 'en'
+end
+
+# Example 6: Custom analysis workflow
+puts "\n6. Custom Analysis Workflow"
+def analyze_with_custom_workflow(file_path)
+  puts "Starting custom analysis workflow for: #{file_path}"
+  
+  # Start performance monitoring
+  monitor = LegalSummariser.performance_monitor
+  monitor.start_timer(:custom_workflow)
+  
+  begin
+    # Step 1: Basic analysis
+    puts "Step 1: Performing basic analysis..."
+    result = LegalSummariser.summarise(file_path)
+    
+    # Step 2: Custom risk assessment
+    puts "Step 2: Custom risk assessment..."
+    risk_score = result[:risks][:risk_score][:score]
+    
+    custom_risk_level = case risk_score
+                       when 0..5 then 'Very Low'
+                       when 6..15 then 'Low'
+                       when 16..30 then 'Medium'
+                       when 31..50 then 'High'
+                       else 'Critical'
+                       end
+    
+    # Step 3: Generate custom report
+    puts "Step 3: Generating custom report..."
+    custom_report = {
+      file_path: file_path,
+      analysis_timestamp: Time.now.iso8601,
+      document_info: {
+        type: result[:metadata][:document_type],
+        word_count: result[:metadata][:word_count],
+        processing_time: result[:metadata][:extraction_time_seconds]
+      },
+      summary: result[:plain_text],
+      risk_assessment: {
+        standard_score: risk_score,
+        custom_level: custom_risk_level,
+        high_priority_issues: result[:risks][:high_risks].length,
+        compliance_gaps: result[:risks][:compliance_gaps].length
+      },
+      recommendations: generate_custom_recommendations(result)
+    }
+    
+    workflow_time = monitor.end_timer(:custom_workflow)
+    custom_report[:workflow_time_seconds] = workflow_time.round(3)
+    
+    puts "Custom workflow completed in #{workflow_time.round(3)}s"
+    return custom_report
+    
+  rescue => e
+    monitor.end_timer(:custom_workflow)
+    puts "Workflow failed: #{e.message}"
+    return nil
+  end
+end
+
+def generate_custom_recommendations(analysis_result)
+  recommendations = []
+  
+  # Risk-based recommendations
+  high_risks = analysis_result[:risks][:high_risks]
+  if high_risks.any?
+    recommendations << "URGENT: Address #{high_risks.length} high-risk issues before signing"
+    high_risks.each { |risk| recommendations << "- #{risk[:recommendation]}" }
+  end
+  
+  # Compliance recommendations
+  compliance_gaps = analysis_result[:risks][:compliance_gaps]
+  if compliance_gaps.any?
+    recommendations << "COMPLIANCE: Review #{compliance_gaps.length} regulatory gaps"
+    compliance_gaps.each { |gap| recommendations << "- #{gap[:recommendation]}" }
+  end
+  
+  # Document type specific recommendations
+  doc_type = analysis_result[:metadata][:document_type]
+  case doc_type
+  when 'nda'
+    recommendations << "NDA: Verify confidentiality scope and duration"
+  when 'employment_contract'
+    recommendations << "EMPLOYMENT: Check termination clauses and benefits"
+  when 'service_agreement'
+    recommendations << "SERVICE: Review deliverables and payment terms"
+  end
+  
+  recommendations
+end
+
+# Example usage of custom workflow
+puts "\n7. Custom Workflow Example"
+# Replace with actual file path
+sample_file = '/tmp/sample_contract.txt'
+File.write(sample_file, "Sample contract content for demonstration purposes.")
+
+custom_result = analyze_with_custom_workflow(sample_file)
+if custom_result
+  puts "\nCustom Analysis Result:"
+  puts JSON.pretty_generate(custom_result)
+end
+
+# Cleanup
+File.delete(sample_file) if File.exist?(sample_file)
+
+# Example 8: System statistics and monitoring
+puts "\n8. System Statistics"
+system_stats = LegalSummariser.stats
+puts "System Performance Overview:"
+puts "- Performance Metrics: #{system_stats[:performance].keys.join(', ')}"
+puts "- Cache Status: #{system_stats[:cache][:enabled] ? 'Active' : 'Inactive'}"
+puts "- Memory Usage: #{system_stats[:memory][:memory_mb]} MB" if system_stats[:memory][:available]
+
+puts "\nAdvanced configuration examples completed!"