UrlCategorise 0.1.3 → 0.1.7

data/docs/v0.1.4-features.md

@@ -0,0 +1,215 @@
+# UrlCategorise v0.1.4 Release Notes
+
+## New Features Added
+
+### 🆕 Dataset-Specific Helper Methods
+
+Added dedicated helper methods for tracking dataset-only statistics separately from DNS blocklists:
+
+```ruby
+client = UrlCategorise::Client.new(dataset_config: { kaggle: {} })
+client.load_kaggle_dataset('owner', 'dataset-name')
+
+# New dataset-specific methods
+client.count_of_dataset_hosts      # Returns hosts from datasets only
+client.count_of_dataset_categories # Returns categories from datasets only
+
+# Existing methods still work and count both
+client.count_of_hosts     # All hosts (DNS lists + datasets)
+client.count_of_categories # All categories (DNS lists + datasets)
+```
+
+**Implementation Details:**
+- `count_of_dataset_hosts` - Sums hosts from categories tracked in `@dataset_categories` Set
+- `count_of_dataset_categories` - Returns size of `@dataset_categories` Set
+- Gracefully handles nil categories with safe navigation (`&.size || 0`)
+- Both methods return 0 when no datasets are loaded
+
+### 🆕 IAB Content Taxonomy Compliance
+
+Full support for IAB (Interactive Advertising Bureau) Content Taxonomy standards:
+
+```ruby
+# Enable IAB v3.0 compliance (recommended)
+client = UrlCategorise::Client.new(
+  iab_compliance: true,
+  iab_version: :v3
+)
+
+# Enable IAB v2.0 compliance
+client = UrlCategorise::Client.new(
+  iab_compliance: true,
+  iab_version: :v2
+)
+
+# Categorization returns IAB codes instead of custom categories
+categories = client.categorise("badsite.com")
+puts categories # => ["626"] (IAB v3 code for illegal content)
+
+# Helper methods
+client.iab_compliant?            # => true/false
+client.get_iab_mapping(:malware) # => "626" (v3) or "IAB25" (v2)
+```
+
+**IAB Category Mappings:**
+
+**IAB Content Taxonomy v3.0:**
+- Security threats (`malware`, `phishing`, `illegal`) → `626` (Illegal Content)
+- Advertising (`advertising`, `mobile_ads`) → `3` (Advertising)
+- Gambling → `7-39` (Gambling subcategory)
+- Adult content (`pornography`) → `626` (Adult Content)
+- Social platforms → `14` (Society)
+- Technology → `19` (Technology & Computing)
+
+**IAB Content Taxonomy v2.0:**
+- Security threats → `IAB25` (Non-Standard Content)
+- Advertising → `IAB3` (Advertising)
+- Gambling → `IAB7-39` (Gambling subcategory)
+- Adult content → `IAB25-3` (Pornography)
+
+**Implementation Details:**
+- New `IabCompliance` module with comprehensive mappings
+- Support for both v2.0 and v3.0 standards
+- IAB compliance affects all categorization methods (`categorise`, `categorise_ip`, `resolve_and_categorise`)
+- Automatic deduplication of IAB codes
+- Graceful handling of unknown categories (returns 'Unknown')
+
+### 🧪 Comprehensive Test Coverage
+
+Added extensive test suites for new features:
+
+**Dataset Methods Tests:**
+- `client_dataset_methods_test.rb` (9 tests)
+- Tests for empty, populated, and mixed dataset scenarios
+- Integration tests with existing methods
+- Edge case handling (nil categories, empty arrays)
+
+**IAB Compliance Tests:**
+- `iab_compliance_test.rb` (14 tests) - Module functionality
+- `client_iab_compliance_test.rb` (19 tests) - Client integration
+- Comprehensive mapping validation for v2 and v3
+- Integration with all categorization methods
+- DNS resolution with IAB compliance
+- Error handling and edge cases
+
+**Total Test Stats:**
+- **316 tests, 2455 assertions, 0 failures, 0 errors**
+- **94.69% line coverage** (660/697 lines)
+- All new features fully tested with edge cases
+
+### 🔧 Technical Implementation
+
+**New Files:**
+- `lib/url_categorise/iab_compliance.rb` - IAB mapping module
+- `test/url_categorise/client_dataset_methods_test.rb` - Dataset helper tests
+- `test/url_categorise/iab_compliance_test.rb` - IAB module tests
+- `test/url_categorise/client_iab_compliance_test.rb` - IAB client integration tests
+
+**Updated Files:**
+- `lib/url_categorise/client.rb` - Added dataset helpers and IAB support
+- `lib/url_categorise.rb` - Required new IAB module
+- `lib/url_categorise/version.rb` - Bumped to 0.1.4
+
+**New Client Attributes:**
+- `iab_compliance_enabled` - Boolean flag for IAB compliance
+- `iab_version` - IAB taxonomy version (:v2 or :v3)
+
+**New Client Methods:**
+- `count_of_dataset_hosts` - Dataset-specific host count
+- `count_of_dataset_categories` - Dataset-specific category count
+- `iab_compliant?` - Check IAB compliance status
+- `get_iab_mapping(category)` - Get IAB code for category
+
+### 🚀 Usage Examples
+
+**Dataset Statistics:**
+```ruby
+client = UrlCategorise::Client.new(dataset_config: { kaggle: {} })
+client.load_kaggle_dataset('owner', 'dataset')
+
+puts "Total hosts: #{client.count_of_hosts}"           # All sources
+puts "Dataset hosts: #{client.count_of_dataset_hosts}" # Datasets only
+puts "DNS list hosts: #{client.count_of_hosts - client.count_of_dataset_hosts}"
+```
+
+**IAB Compliance:**
+```ruby
+# Production environment with IAB compliance
+client = UrlCategorise::Client.new(
+  iab_compliance: true,
+  iab_version: :v3,
+  dataset_config: { kaggle: { username: 'user', api_key: 'key' } }
+)
+
+# All methods return IAB codes
+domain_cats = client.categorise("example.com")        # => ["3", "626"]
+ip_cats = client.categorise_ip("192.168.1.100")      # => ["626"]
+resolved_cats = client.resolve_and_categorise("site.com") # => ["3"]
+
+# Check compliance
+puts "IAB compliant: #{client.iab_compliant?}"       # => true
+puts "Using version: #{client.iab_version}"          # => :v3
+```
+
+**Rails Service Integration:**
+```ruby
+class UrlCategorizerService
+  def initialize
+    @client = UrlCategorise::ActiveRecordClient.new(
+      iab_compliance: Rails.env.production?,
+      iab_version: :v3,
+      dataset_config: {
+        kaggle: {
+          username: ENV['KAGGLE_USERNAME'],
+          api_key: ENV['KAGGLE_API_KEY']
+        }
+      }
+    )
+  end
+
+  def stats
+    {
+      total_hosts: @client.count_of_hosts,
+      dataset_hosts: @client.count_of_dataset_hosts,
+      dns_list_hosts: @client.count_of_hosts - @client.count_of_dataset_hosts,
+      iab_compliant: @client.iab_compliant?,
+      iab_version: @client.iab_version
+    }
+  end
+end
+```
+
+### 🔄 Migration Guide
+
+**From v0.1.3 to v0.1.4:**
+
+1. **No Breaking Changes** - All existing code continues to work
+2. **Optional New Features** - IAB compliance and dataset helpers are opt-in
+3. **Enhanced Statistics** - Use new helper methods for better insights
+
+**Recommended Updates:**
+```ruby
+# Before (still works)
+client = UrlCategorise::Client.new
+puts "Total: #{client.count_of_hosts}"
+
+# After (enhanced)
+client = UrlCategorise::Client.new(
+  iab_compliance: true,  # Optional IAB compliance
+  iab_version: :v3       # Optional version selection
+)
+puts "Total hosts: #{client.count_of_hosts}"
+puts "Dataset hosts: #{client.count_of_dataset_hosts}"
+puts "IAB compliant: #{client.iab_compliant?}"
+```
+
+### 📊 Quality Assurance
+
+- ✅ All 316 tests pass with 0 failures/errors
+- ✅ 94.69% line coverage maintained
+- ✅ Code style enforced with rubocop
+- ✅ Comprehensive edge case testing
+- ✅ Memory-efficient implementation
+- ✅ Backward compatibility preserved
+
+This release adds powerful new features while maintaining the reliability and performance standards established in previous versions.

data/docs/video-url-detection.md

@@ -0,0 +1,353 @@
+# Video URL Detection Feature
+
+## Overview
+
+The UrlCategorise gem now includes advanced video URL detection capabilities that allow you to determine if a URL is a direct link to video content, rather than just a homepage, profile page, or other non-video resource on a video hosting domain.
+
+## Key Features
+
+### 🎬 Direct Video URL Detection
+
+New `video_url?` method provides precise detection of video content URLs:
+
+```ruby
+client = UrlCategorise::Client.new(regex_categorization: true)
+
+# Direct video content URLs return true
+client.video_url?("https://youtube.com/watch?v=dQw4w9WgXcQ")  # => true
+client.video_url?("https://vimeo.com/123456789")            # => true
+client.video_url?("https://tiktok.com/@user/video/123")     # => true
+client.video_url?("https://dailymotion.com/video/x7abc123") # => true
+
+# Non-video URLs return false
+client.video_url?("https://youtube.com")                    # => false
+client.video_url?("https://youtube.com/@channel")           # => false
+client.video_url?("https://google.com/search?q=cats")       # => false
+```
+
+### 📡 Comprehensive Video Hosting Lists
+
+- **3,500+ video hosting domains** extracted from yt-dlp (YouTube-dl fork) extractors
+- **50+ regex patterns** for identifying video content URLs
+- **Automatic generation** using `bin/generate_video_lists` script
+- **Remote list fetching** from GitHub repository
+
+### 🔗 Remote Pattern Files
+
+Video patterns are automatically fetched from remote GitHub repository:
+
+- **Video domains**: `https://raw.githubusercontent.com/TRex22/url_categorise/refs/heads/main/lists/video_hosting_domains.hosts`
+- **URL patterns**: `https://raw.githubusercontent.com/TRex22/url_categorise/refs/heads/main/lists/video_url_patterns.txt`
+
+### 🧠 Enhanced Categorization
+
+Regex categorization provides more specific categorization for video URLs:
+
+```ruby
+client = UrlCategorise::Client.new(regex_categorization: true)
+
+# Basic domain categorization
+client.categorise('https://youtube.com') # => [:video_hosting]
+
+# Enhanced content detection for actual video URLs
+client.categorise('https://youtube.com/watch?v=abc123') # => [:video_hosting, :video_hosting_content]
+```
+
+## Technical Implementation
+
+### Constants Updates
+
+New constants in `UrlCategorise::Constants`:
+
+```ruby
+# Remote video URL patterns file
+VIDEO_URL_PATTERNS_FILE = 'https://raw.githubusercontent.com/TRex22/url_categorise/refs/heads/main/lists/video_url_patterns.txt'.freeze
+
+# Updated video hosting category to use remote list
+video_hosting: ['https://raw.githubusercontent.com/TRex22/url_categorise/refs/heads/main/lists/video_hosting_domains.hosts']
+```
+
+### Client Enhancements
+
+New `Client` class features:
+
+```ruby
+# Default regex patterns file now uses remote URL
+attribute :regex_patterns_file, default: -> { VIDEO_URL_PATTERNS_FILE }
+
+# New video URL detection method
+def video_url?(url)
+  # 1. Check if URL is valid and not empty
+  # 2. Ensure regex categorization is enabled
+  # 3. Verify URL is from a video hosting domain
+  # 4. Check if URL matches video content patterns
+  # 5. Return true only if all conditions are met
+end
+
+# Enhanced pattern fetching supports remote URLs
+def fetch_regex_patterns_content
+  # Supports HTTP/HTTPS, file://, and direct file paths
+  # Graceful error handling for network failures
+end
+```
+
+### Pattern Generation
+
+The `bin/generate_video_lists` script has been enhanced:
+
+- **Fixed terminal output** - Clean progress display without character overlap
+- **Suppressed regex warnings** - No more nested operator warnings
+- **Improved domain extraction** - More comprehensive pattern matching
+- **Manual high-priority patterns** - Curated YouTube, Vimeo, TikTok patterns
+- **3,500+ domains extracted** - Up from 96 domains previously
+
+## Usage Examples
+
+### Basic Video URL Detection
+
+```ruby
+require 'url_categorise'
+
+# Initialize client with regex categorization
+client = UrlCategorise::Client.new(regex_categorization: true)
+
+# Test various video URLs
+urls = [
+  'https://youtube.com/watch?v=dQw4w9WgXcQ',  # YouTube video
+  'https://youtube.com',                      # YouTube homepage
+  'https://youtube.com/@pewdiepie',           # YouTube channel
+  'https://vimeo.com/123456789',             # Vimeo video
+  'https://tiktok.com/@user/video/123',      # TikTok video
+  'https://google.com/search?q=cats'         # Non-video site
+]
+
+urls.each do |url|
+  is_video = client.video_url?(url)
+  categories = client.categorise(url)
+  puts "#{url} -> video: #{is_video}, categories: #{categories}"
+end
+```
+
+### Content Filtering Application
+
+```ruby
+class ContentFilter
+  def initialize
+    @client = UrlCategorise::Client.new(regex_categorization: true)
+  end
+
+  def filter_video_content(urls)
+    results = {
+      video_content: [],
+      video_sites: [],
+      other: []
+    }
+
+    urls.each do |url|
+      if @client.video_url?(url)
+        results[:video_content] << url
+      elsif @client.categorise(url).include?(:video_hosting)
+        results[:video_sites] << url
+      else
+        results[:other] << url
+      end
+    end
+
+    results
+  end
+end
+
+# Usage
+filter = ContentFilter.new
+results = filter.filter_video_content([
+  'https://youtube.com/watch?v=abc',   # -> :video_content
+  'https://youtube.com/@channel',      # -> :video_sites
+  'https://example.com'                # -> :other
+])
+```
+
+### Rails Integration
+
+```ruby
+# app/services/video_url_service.rb
+class VideoUrlService
+  include Singleton
+
+  def initialize
+    @client = UrlCategorise::Client.new(
+      regex_categorization: true,
+      cache_dir: Rails.root.join('tmp', 'url_cache')
+    )
+  end
+
+  def video_content?(url)
+    Rails.cache.fetch("video_content_#{Digest::MD5.hexdigest(url)}", expires_in: 1.hour) do
+      @client.video_url?(url)
+    end
+  end
+
+  def categorize_video_url(url)
+    categories = @client.categorise(url)
+    is_video_content = @client.video_url?(url)
+    
+    {
+      url: url,
+      categories: categories,
+      is_video_content: is_video_content,
+      risk_level: calculate_risk_level(categories)
+    }
+  end
+
+  private
+
+  def calculate_risk_level(categories)
+    return 'safe' if categories.empty?
+    return 'blocked' if (categories & [:malware, :phishing]).any?
+    return 'restricted' if (categories & [:pornography, :gambling]).any?
+    'allowed'
+  end
+end
+
+# Usage in controllers
+class VideosController < ApplicationController
+  def check
+    url = params[:url]
+    result = VideoUrlService.instance.categorize_video_url(url)
+    render json: result
+  end
+end
+```
+
+## Testing
+
+Comprehensive test suite with 15 new tests covering:
+
+### Core Functionality Tests
+
+- Video URL detection when regex categorization is disabled/enabled
+- Detection of video vs non-video domains  
+- Video content URLs vs homepage/profile URLs
+- Multiple video hosting categories (`video`, `video_hosting`, etc.)
+- Graceful handling of invalid URLs
+
+### Network and Error Handling Tests
+
+- Remote pattern file fetching with mocked HTTP requests
+- Graceful failure when remote files are not accessible
+- Timeout and network error handling
+- Invalid regex pattern handling
+
+### Test Results
+
+- ✅ **383 tests, 2,729 assertions, 0 failures, 0 errors**
+- ✅ **92.88% line coverage** (926/997 lines)
+- ✅ All video URL detection functionality fully tested
+
+## Performance Considerations
+
+### Caching
+
+- Pattern files are cached locally when `cache_dir` is specified
+- Remote patterns are only fetched once per session
+- Client-side caching recommended for high-traffic applications
+
+### Memory Usage
+
+- Regex patterns are compiled once during client initialization
+- Minimal memory overhead for video URL detection
+- Efficient Set-based domain lookups
+
+### Network Optimization
+
+- Remote pattern files are small (~50KB total)
+- Graceful fallback to local files if remote fetch fails
+- HTTP timeout configuration supported
+
+## Migration Guide
+
+### From Previous Versions
+
+No breaking changes - video URL detection is entirely optional:
+
+```ruby
+# Existing code continues to work unchanged
+client = UrlCategorise::Client.new
+categories = client.categorise('youtube.com')
+
+# Enable video URL detection when needed
+client_with_video = UrlCategorise::Client.new(regex_categorization: true)
+is_video = client_with_video.video_url?('https://youtube.com/watch?v=abc')
+```
+
+### Enabling Video Detection
+
+To use video URL detection in existing applications:
+
+1. **Enable regex categorization**: `regex_categorization: true`
+2. **Use the new method**: `client.video_url?(url)`
+3. **Optional configuration**: Custom pattern files, caching
+
+### Configuration Options
+
+```ruby
+# Default configuration (recommended)
+client = UrlCategorise::Client.new(regex_categorization: true)
+
+# Custom pattern file (local)
+client = UrlCategorise::Client.new(
+  regex_categorization: true,
+  regex_patterns_file: 'path/to/custom/patterns.txt'
+)
+
+# Custom pattern file (remote)
+client = UrlCategorise::Client.new(
+  regex_categorization: true,
+  regex_patterns_file: 'https://example.com/patterns.txt'
+)
+
+# With caching
+client = UrlCategorise::Client.new(
+  regex_categorization: true,
+  cache_dir: './cache'
+)
+```
+
+## Maintenance
+
+### Updating Video Lists
+
+The video hosting lists are automatically maintained in the GitHub repository. To generate updated lists locally:
+
+```bash
+# Generate fresh video hosting lists
+ruby bin/generate_video_lists
+
+# This creates/updates:
+# - lists/video_hosting_domains.hosts (3,500+ domains)
+# - lists/video_url_patterns.txt (50+ patterns)
+```
+
+### Manual Pattern Curation
+
+High-priority patterns are manually curated in the generation script:
+
+- YouTube video and Shorts URLs
+- Vimeo video URLs  
+- Dailymotion video URLs
+- Twitch video URLs
+- TikTok video URLs
+
+### List Health Monitoring
+
+Use the existing health monitoring for video hosting lists:
+
+```ruby
+client = UrlCategorise::Client.new
+report = client.check_all_lists
+
+# Check video hosting category specifically
+video_status = report[:successful_lists][:video_hosting]
+puts "Video hosting lists: #{video_status&.length || 0} URLs working"
+```
+
+This feature provides enterprise-grade video URL detection while maintaining the gem's focus on performance, reliability, and ease of use.

data/lib/url_categorise/active_record_client.rb

@@ -118,7 +118,7 @@ module UrlCategorise
       return unless UrlCategorise::Models.available?
 
       # Store list metadata
-      @host_urls.each do |category, urls|
+      (host_urls || {}).each do |category, urls|
         urls.each do |url|
           next unless url.is_a?(String)