UrlCategorise 0.1.3 → 0.1.7

data/README.md

@@ -5,6 +5,10 @@ A comprehensive Ruby gem for categorizing URLs and domains based on various secu
 ## Features
 
 - **Comprehensive Coverage**: 60+ high-quality categories including security, content, and specialized lists
+- **Video Content Detection**: Advanced regex-based categorization with `video_url?` method to distinguish video content from other website resources
+- **Custom Video Lists**: Generate and maintain comprehensive video hosting domain lists using yt-dlp extractors
+- **Kaggle Dataset Integration**: Automatic loading and processing of machine learning datasets from Kaggle
+- **Multiple Data Sources**: Supports blocklists, CSV datasets, and Kaggle ML datasets  
 - **Multiple List Formats**: Supports hosts files, pfSense, AdSense, uBlock Origin, dnsmasq, and plain text formats
 - **Intelligent Caching**: Hash-based file update detection with configurable local cache
 - **DNS Resolution**: Resolve domains to IPs and check against IP-based blocklists  
@@ -14,6 +18,10 @@ A comprehensive Ruby gem for categorizing URLs and domains based on various secu
 - **Metadata Tracking**: Track last update times, ETags, and content hashes
 - **Health Monitoring**: Automatic detection and removal of broken blocklist sources
 - **List Validation**: Built-in tools to verify all configured URLs are accessible
+- **Auto-Loading Datasets**: Automatic processing of predefined datasets during client initialization
+- **ActiveAttr Settings**: In-memory modification of client settings using attribute setters
+- **Data Export**: Export categorized data as hosts files per category or comprehensive CSV exports
+- **CLI Commands**: Command-line utilities for data export and list checking
 
 ## Installation
 
@@ -44,6 +52,15 @@ puts "Total hosts: #{client.count_of_hosts}"
 puts "Categories: #{client.count_of_categories}"  
 puts "Data size: #{client.size_of_data} MB"
 
+# Get detailed size breakdown
+puts "Total data size: #{client.size_of_data} MB (#{client.size_of_data_bytes} bytes)"
+puts "Blocklist data size: #{client.size_of_blocklist_data} MB (#{client.size_of_blocklist_data_bytes} bytes)"
+puts "Dataset data size: #{client.size_of_dataset_data} MB (#{client.size_of_dataset_data_bytes} bytes)"
+
+# Get dataset-specific statistics (if datasets are loaded)
+puts "Dataset hosts: #{client.count_of_dataset_hosts}"
+puts "Dataset categories: #{client.count_of_dataset_categories}"
+
 # Categorize a URL or domain
 categories = client.categorise("badsite.com")
 puts "Categories: #{categories}" # => [:malware, :phishing]
@@ -57,6 +74,116 @@ ip_categories = client.categorise_ip("192.168.1.100")
 puts "IP categories: #{ip_categories}"
 ```
 
+## New Features
+
+### Dynamic Settings with ActiveAttr
+
+The Client class now supports in-memory modification of settings using ActiveAttr:
+
+```ruby
+client = UrlCategorise::Client.new
+
+# Modify settings dynamically
+client.smart_categorization_enabled = true
+client.iab_compliance_enabled = true
+client.iab_version = :v2
+client.request_timeout = 30
+client.dns_servers = ['8.8.8.8', '8.8.4.4']
+
+# Settings take effect immediately - no need to recreate the client
+categories = client.categorise('reddit.com') # Uses new smart categorization rules
+```
+
+### Data Export Features
+
+#### Hosts File Export
+
+Export all categorized domains as separate hosts files per category:
+
+```ruby
+# Export to default location
+result = client.export_hosts_files
+
+# Export to custom location
+result = client.export_hosts_files('/custom/export/path')
+
+# Result includes file information and summary
+puts "Exported #{result[:_summary][:total_categories]} categories"
+puts "Total domains: #{result[:_summary][:total_domains]}"
+puts "Files saved to: #{result[:_summary][:export_directory]}"
+```
+
+Each category gets its own hosts file (e.g., `malware.hosts`, `advertising.hosts`) with proper headers and sorted domains.
+
+#### CSV Data Export
+
+Export all data as a single comprehensive CSV file for AI training and analysis:
+
+```ruby
+# Export to default location
+result = client.export_csv_data
+
+# Export to custom location with IAB compliance
+client.iab_compliance_enabled = true
+result = client.export_csv_data('/custom/export/path')
+
+# Returns information about created files:
+# {
+#   csv_file: '/path/url_categorise_comprehensive_export_20231201_143022.csv',
+#   summary_file: '/path/export_summary_20231201_143022.json',
+#   total_entries: 50000,
+#   summary: { ... },
+#   export_directory: '/path'
+# }
+```
+
+**Single comprehensive CSV file contains:**
+
+- **Domain Categorization Data**: All processed domains with categories, source types, IAB mappings
+- **Raw Dataset Content**: Original dataset entries with titles, descriptions, text, summaries, and all available fields
+- **Dynamic Headers**: Automatically adapts to include all available data fields
+- **Data Type Column**: Distinguishes between 'domain_categorization', 'raw_dataset_content', etc.
+
+**Key Features:**
+- Everything in one file for easy analysis and AI/ML training
+- Rich textual content from original datasets
+- IAB Content Taxonomy compliance mapping
+- Smart categorization metadata
+- Source type tracking (dataset vs blocklist)
+
+#### CLI Commands
+
+Command-line utilities for data export:
+
+```bash
+# Export hosts files
+$ bundle exec export_hosts --output /tmp/hosts --verbose
+
+# Export CSV data with all features enabled
+$ bundle exec export_csv --output /tmp/csv --iab-compliance --smart-categorization --auto-load-datasets --verbose
+
+# Generate updated video hosting lists
+$ ruby bin/generate_video_lists
+
+# Check health of all blocklist URLs
+$ bundle exec check_lists
+
+# Export with custom Kaggle credentials
+$ bundle exec export_csv --auto-load-datasets --kaggle-credentials ~/my-kaggle.json --verbose
+
+# Basic export (domains only)
+$ bundle exec export_csv --output /tmp/csv
+
+# Check URL health (existing command)
+$ bundle exec check_lists
+```
+
+**Key CLI Options:**
+- `--auto-load-datasets`: Load datasets from constants to include rich text content
+- `--kaggle-credentials FILE`: Specify custom Kaggle credentials file
+- `--iab-compliance`: Enable IAB Content Taxonomy mapping
+- `--smart-categorization`: Enable intelligent category filtering
+
 ## Advanced Configuration
 
 ### File Caching
@@ -113,7 +240,11 @@ client = UrlCategorise::Client.new(
   cache_dir: "./url_cache",                                # Enable local caching
   force_download: false,                                   # Use cache when available
   dns_servers: ['1.1.1.1', '1.0.0.1'],                   # Cloudflare DNS servers
-  request_timeout: 15                                      # 15 second HTTP timeout
+  request_timeout: 15,                                     # 15 second HTTP timeout
+  iab_compliance: true,                                    # Enable IAB compliance
+  iab_version: :v3,                                        # Use IAB Content Taxonomy v3.0
+  auto_load_datasets: false,                               # Disable automatic dataset loading (default)
+  smart_categorization: false                              # Disable smart post-processing (default)
 )
 ```
 
@@ -132,6 +263,248 @@ host_urls = {
 client = UrlCategorise::Client.new(host_urls: host_urls)
 ```
 
+### Video Content Detection
+
+The gem includes advanced regex-based categorization specifically for video hosting platforms. This helps distinguish between actual video content URLs and other resources like homepages, user profiles, playlists, or community content.
+
+#### Video Hosting Domains
+
+The gem maintains a comprehensive list of video hosting domains extracted from yt-dlp (YouTube-dl fork) extractors:
+
+```ruby
+# Generate/update video hosting lists
+system("ruby bin/generate_video_lists")
+
+# Use video hosting categorization
+client = UrlCategorise::Client.new
+categories = client.categorise("youtube.com")
+# => [:video_hosting]
+```
+
+#### Video Content vs Other Resources
+
+Enable regex categorization to distinguish video content from other resources:
+
+```ruby
+client = UrlCategorise::Client.new(
+  regex_categorization: true  # Uses remote video patterns by default
+)
+
+# Regular homepage gets basic category
+client.categorise("https://youtube.com")
+# => [:video_hosting]
+
+# Actual video URL gets enhanced categorization
+client.categorise("https://youtube.com/watch?v=dQw4w9WgXcQ") 
+# => [:video_hosting, :video_hosting_content]
+
+# User profile page - no content enhancement
+client.categorise("https://youtube.com/@username")
+# => [:video_hosting]
+```
+
+#### Direct Video URL Detection
+
+Use the `video_url?` method to check if a URL is a direct link to video content:
+
+```ruby
+client = UrlCategorise::Client.new(regex_categorization: true)
+
+# Check if URLs are direct video content links
+client.video_url?("https://youtube.com/watch?v=dQw4w9WgXcQ")  # => true
+client.video_url?("https://youtube.com")                     # => false
+client.video_url?("https://youtube.com/@channel")            # => false
+client.video_url?("https://vimeo.com/123456789")            # => true
+client.video_url?("https://tiktok.com/@user/video/123")     # => true
+
+# Works with various video hosting platforms
+client.video_url?("https://dailymotion.com/video/x7abc123") # => true
+client.video_url?("https://twitch.tv/videos/1234567890")    # => true
+
+# Returns false for non-video domains
+client.video_url?("https://google.com/search?q=cats")       # => false
+```
+
+**How it works:**
+1. First checks if the URL is from a known video hosting domain
+2. Then uses regex patterns to determine if it's a direct video content URL
+3. Returns `true` only if both conditions are met
+4. Handles invalid URLs gracefully (returns `false`)
+
+#### Maintaining Video Lists
+
+The gem includes a script to generate and maintain comprehensive video hosting lists:
+
+```bash
+# Generate updated video hosting lists
+ruby bin/generate_video_lists
+
+# This creates:
+# - lists/video_hosting_domains.hosts (PiHole compatible)  
+# - lists/video_url_patterns.txt (Regex patterns for content detection)
+```
+
+The script fetches data from yt-dlp extractors and combines it with manually curated major platforms to ensure comprehensive coverage.
+
+### Smart Categorization (Post-Processing)
+
+Smart categorization solves the problem of overly broad domain-level categorization. For example, `reddit.com` might appear in health & fitness blocklists, but not all Reddit content is health-related.
+
+#### The Problem
+
+```ruby
+# Without smart categorization
+client.categorise("reddit.com")
+# => [:reddit, :social_media, :health_and_fitness, :forums]  # Too broad!
+
+client.categorise("reddit.com/r/technology") 
+# => [:reddit, :social_media, :health_and_fitness, :forums]  # Still wrong!
+```
+
+#### The Solution
+
+```ruby
+# Enable smart categorization
+client = UrlCategorise::Client.new(
+  smart_categorization: true  # Remove overly broad categories
+)
+
+client.categorise("reddit.com")
+# => [:reddit, :social_media]  # Much more accurate!
+```
+
+#### How It Works
+
+Smart categorization automatically removes overly broad categories for known platforms:
+
+- **Social Media Platforms** (Reddit, Facebook, Twitter, etc.): Removes categories like `:health_and_fitness`, `:forums`, `:news`, `:technology`, `:education`
+- **Search Engines** (Google, Bing, etc.): Removes categories like `:news`, `:shopping`, `:travel`
+- **Video Platforms** (YouTube, Vimeo, etc.): Removes categories like `:education`, `:entertainment`, `:music`
+
+#### Custom Smart Rules
+
+You can define custom rules for specific domains or URL patterns:
+
+```ruby
+custom_rules = {
+  reddit_subreddits: {
+    domains: ['reddit.com'],
+    remove_categories: [:health_and_fitness, :forums],
+    add_categories_by_path: {
+      /\/r\/fitness/ => [:health_and_fitness],      # Add back for /r/fitness
+      /\/r\/technology/ => [:technology],           # Add technology for /r/technology 
+      /\/r\/programming/ => [:technology, :programming]
+    }
+  },
+  my_company_domains: {
+    domains: ['mycompany.com'],
+    allowed_categories_only: [:business, :technology]  # Only allow specific categories
+  }
+}
+
+client = UrlCategorise::Client.new(
+  smart_categorization: true,
+  smart_rules: custom_rules
+)
+
+# Now path-based categorization works
+client.categorise('reddit.com')           # => [:reddit, :social_media]
+client.categorise('reddit.com/r/fitness') # => [:reddit, :social_media, :health_and_fitness]
+client.categorise('reddit.com/r/technology') # => [:reddit, :social_media, :technology]
+```
+
+#### Available Rule Types
+
+- **`remove_categories`**: Remove specific categories for domains
+- **`keep_primary_only`**: Keep only specified categories, remove others
+- **`allowed_categories_only`**: Only allow specific categories, block all others
+- **`add_categories_by_path`**: Add categories based on URL path patterns
+
+#### Smart Rules with IAB Compliance
+
+Smart categorization works seamlessly with IAB compliance:
+
+```ruby
+client = UrlCategorise::Client.new(
+  smart_categorization: true,
+  iab_compliance: true,
+  iab_version: :v3
+)
+
+# Returns clean IAB codes after smart processing
+categories = client.categorise("reddit.com")  # => ["14"] (Society - Social Media)
+```
+
+## IAB Content Taxonomy Compliance
+
+UrlCategorise supports IAB (Interactive Advertising Bureau) Content Taxonomy compliance for standardized content categorization:
+
+### Basic IAB Compliance
+
+```ruby
+# Enable IAB v3.0 compliance (default)
+client = UrlCategorise::Client.new(
+  iab_compliance: true,
+  iab_version: :v3
+)
+
+# Enable IAB v2.0 compliance
+client = UrlCategorise::Client.new(
+  iab_compliance: true,
+  iab_version: :v2
+)
+
+# Categorization returns IAB codes instead of custom categories
+categories = client.categorise("badsite.com")
+puts categories # => ["626"] (IAB v3 code for illegal content)
+
+# Check IAB compliance status
+puts client.iab_compliant? # => true
+
+# Get IAB mapping for a specific category
+puts client.get_iab_mapping(:malware) # => "626" (v3) or "IAB25" (v2)
+```
+
+### IAB Category Mappings
+
+The gem maps security and content categories to appropriate IAB codes:
+
+**IAB Content Taxonomy v3.0 (recommended):**
+- `malware`, `phishing`, `illegal` → `626` (Illegal Content)
+- `advertising`, `mobile_ads` → `3` (Advertising)
+- `gambling` → `7-39` (Gambling)
+- `pornography` → `626` (Adult Content)
+- `social_media` → `14` (Society)
+- `technology` → `19` (Technology & Computing)
+
+**IAB Content Taxonomy v2.0:**
+- `malware`, `phishing` → `IAB25` (Non-Standard Content)
+- `advertising` → `IAB3` (Advertising)
+- `gambling` → `IAB7-39` (Gambling)
+- `pornography` → `IAB25-3` (Pornography)
+
+### Integration with Datasets
+
+IAB compliance works seamlessly with dataset processing:
+
+```ruby
+client = UrlCategorise::Client.new(
+  iab_compliance: true,
+  iab_version: :v3,
+  dataset_config: {
+    kaggle: { username: 'user', api_key: 'key' }
+  },
+  auto_load_datasets: true  # Automatically load predefined datasets with IAB mapping
+)
+
+# Load additional datasets - categories will be mapped to IAB codes
+client.load_kaggle_dataset('owner', 'dataset-name')
+client.load_csv_dataset('https://example.com/data.csv')
+
+# All categorization methods return IAB codes
+categories = client.categorise("example.com") # => ["3", "626"]
+```
+
 ## Available Categories
 
 ### Security & Threat Intelligence
@@ -194,9 +567,43 @@ ruby bin/check_lists
 
 ## Dataset Processing
 
-UrlCategorise now supports processing external datasets from Kaggle and CSV files to expand categorization data:
+UrlCategorise supports processing external datasets from Kaggle and CSV files to expand categorization data beyond traditional blocklists. This allows integration of machine learning datasets and custom URL classification data:
+
+### Automatic Dataset Loading
+
+Enable automatic loading of predefined datasets during client initialization:
 
-### Kaggle Dataset Integration
+```ruby
+# Enable automatic dataset loading from constants
+client = UrlCategorise::Client.new(
+  dataset_config: {
+    kaggle: {
+      username: ENV['KAGGLE_USERNAME'], 
+      api_key: ENV['KAGGLE_API_KEY']
+    },
+    cache_path: './dataset_cache',
+    download_path: './downloads'
+  },
+  auto_load_datasets: true  # Automatically loads all predefined datasets
+)
+
+# Datasets are now automatically integrated and ready for use
+categories = client.categorise('https://example.com')
+puts "Dataset categories loaded: #{client.count_of_dataset_categories}"
+puts "Dataset hosts: #{client.count_of_dataset_hosts}"
+```
+
+The gem includes predefined high-quality datasets in constants:
+- **`shaurov/website-classification-using-url`** - Comprehensive URL classification dataset
+- **`hetulmehta/website-classification`** - Website categorization with cleaned text data  
+- **`shawon10/url-classification-dataset-dmoz`** - DMOZ-based URL classification
+- **Data.world CSV dataset** - Additional URL categorization data
+
+### Manual Dataset Loading
+
+You can also load datasets manually for more control over the process:
+
+#### Kaggle Dataset Integration
 
 Load datasets directly from Kaggle using three authentication methods:
 
@@ -244,7 +651,7 @@ client.load_kaggle_dataset('owner', 'dataset-name', {
 categories = client.categorise('https://example.com')
 ```
 
-### CSV Dataset Processing
+#### CSV Dataset Processing
 
 Load datasets from direct CSV URLs:
 
@@ -286,7 +693,10 @@ dataset_config = {
   timeout: 30                       # HTTP timeout for downloads
 }
 
-client = UrlCategorise::Client.new(dataset_config: dataset_config)
+client = UrlCategorise::Client.new(
+  dataset_config: dataset_config,
+  auto_load_datasets: true          # Enable automatic loading of predefined datasets
+)
 ```
 
 ### Disabling Kaggle Functionality
@@ -485,7 +895,18 @@ class UrlCategorizerService
       cache_dir: Rails.root.join('tmp', 'url_cache'),
       use_database: true,
       force_download: Rails.env.development?,
-      request_timeout: Rails.env.production? ? 30 : 10  # Longer timeout in production
+      request_timeout: Rails.env.production? ? 30 : 10,  # Longer timeout in production
+      iab_compliance: Rails.env.production?,              # Enable IAB compliance in production
+      iab_version: :v3,                                   # Use IAB Content Taxonomy v3.0
+      auto_load_datasets: Rails.env.production?,          # Auto-load datasets in production
+      dataset_config: {
+        kaggle: {
+          username: ENV['KAGGLE_USERNAME'],
+          api_key: ENV['KAGGLE_API_KEY']
+        },
+        cache_path: Rails.root.join('tmp', 'dataset_cache'),
+        download_path: Rails.root.join('tmp', 'dataset_downloads')
+      }
     )
   end
 
@@ -508,12 +929,34 @@ class UrlCategorizerService
   end
 
   def stats
-    @client.database_stats
+    base_stats = @client.database_stats
+    base_stats.merge({
+      dataset_hosts: @client.count_of_dataset_hosts,
+      dataset_categories: @client.count_of_dataset_categories,
+      iab_compliant: @client.iab_compliant?,
+      iab_version: @client.iab_version
+    })
   end
 
   def refresh_lists!
     @client.update_database
   end
+
+  def load_dataset(type, identifier, options = {})
+    case type.to_s
+    when 'kaggle'
+      owner, dataset = identifier.split('/')
+      @client.load_kaggle_dataset(owner, dataset, options)
+    when 'csv'
+      @client.load_csv_dataset(identifier, options)
+    else
+      raise ArgumentError, "Unsupported dataset type: #{type}"
+    end
+  end
+
+  def get_iab_mapping(category)
+    @client.get_iab_mapping(category)
+  end
 end
 ```
 

data/bin/export_csv

@@ -0,0 +1,120 @@
+#!/usr/bin/env ruby
+
+require 'bundler/setup'
+require 'optparse'
+require_relative '../lib/url_categorise'
+
+options = {
+  output_path: nil,
+  cache_dir: nil,
+  verbose: false,
+  iab_compliance: false,
+  smart_categorization: false,
+  auto_load_datasets: false,
+  kaggle_credentials_file: nil
+}
+
+OptionParser.new do |opts|
+  opts.banner = "Usage: #{$0} [options]"
+  opts.separator ""
+  opts.separator "Export all categorized domains and dataset content as a single CSV file for AI training"
+  opts.separator ""
+  
+  opts.on("-o", "--output PATH", "Output directory path (default: cache_dir/exports/csv or ./exports/csv)") do |path|
+    options[:output_path] = path
+  end
+  
+  opts.on("-c", "--cache-dir PATH", "Cache directory path for client initialization") do |path|
+    options[:cache_dir] = path
+  end
+  
+  opts.on("--iab-compliance", "Enable IAB compliance for category mapping") do
+    options[:iab_compliance] = true
+  end
+  
+  opts.on("--smart-categorization", "Enable smart categorization") do
+    options[:smart_categorization] = true
+  end
+  
+  opts.on("--auto-load-datasets", "Auto-load datasets from constants for rich content export") do
+    options[:auto_load_datasets] = true
+  end
+  
+  opts.on("--kaggle-credentials FILE", "Path to Kaggle credentials file (default: ~/.kaggle/kaggle.json)") do |file|
+    options[:kaggle_credentials_file] = file
+  end
+  
+  opts.on("-v", "--verbose", "Verbose output") do
+    options[:verbose] = true
+  end
+  
+  opts.on("-h", "--help", "Show this help message") do
+    puts opts
+    exit
+  end
+end.parse!
+
+puts "=== UrlCategorise CSV Data Export ===" if options[:verbose]
+puts "Initializing client..." if options[:verbose]
+
+begin
+  # Build dataset config if datasets should be loaded
+  dataset_config = {}
+  if options[:auto_load_datasets]
+    dataset_config = {
+      cache_path: options[:cache_dir] ? File.join(options[:cache_dir], 'datasets') : './url_cache/datasets',
+      download_path: options[:cache_dir] ? File.join(options[:cache_dir], 'downloads') : './url_cache/downloads'
+    }
+    
+    # Add Kaggle credentials if provided
+    if options[:kaggle_credentials_file]
+      dataset_config[:kaggle] = { credentials_file: options[:kaggle_credentials_file] }
+    elsif File.exist?(File.expand_path('~/.kaggle/kaggle.json'))
+      dataset_config[:kaggle] = { credentials_file: '~/.kaggle/kaggle.json' }
+    end
+  end
+
+  client = UrlCategorise::Client.new(
+    cache_dir: options[:cache_dir],
+    iab_compliance: options[:iab_compliance],
+    smart_categorization: options[:smart_categorization],
+    auto_load_datasets: options[:auto_load_datasets],
+    dataset_config: dataset_config
+  )
+  
+  if options[:verbose] && options[:auto_load_datasets]
+    puts "Client initialized with dataset loading enabled"
+    puts "Dataset statistics:"
+    puts "  Dataset categories: #{client.count_of_dataset_categories}"
+    puts "  Dataset hosts: #{client.count_of_dataset_hosts.to_s.reverse.gsub(/(\\d{3})(?=\\d)/, '\\1,').reverse}"
+    puts "  Dataset data size: #{client.size_of_dataset_data.round(2)} MB" if client.respond_to?(:size_of_dataset_data)
+  end
+  
+  puts "Exporting CSV data..." if options[:verbose]
+  
+  result = client.export_csv_data(options[:output_path])
+  
+  puts "\n✅ Export completed successfully!"
+  puts "📁 Export directory: #{result[:export_directory]}"
+  puts "📄 CSV file: #{result[:csv_file]}"
+  puts "📄 Summary file: #{result[:summary_file]}"
+  
+  puts "\n📊 Data Summary:"
+  puts "  Total entries: #{result[:total_entries]}"
+  puts "  Domain categorizations: #{result[:summary][:domain_categorization_entries]}"
+  puts "  Dataset content entries: #{result[:summary][:dataset_content_entries]}"
+  puts "  Total categories: #{result[:summary][:total_categories]}"
+  puts "  Has dataset content: #{result[:summary][:has_dataset_content]}"
+  
+  if options[:verbose]
+    puts "\n🏷️  Categories included:"
+    result[:summary][:categories].each do |category|
+      puts "    - #{category}"
+    end
+  end
+  
+rescue StandardError => e
+  puts "❌ Error: #{e.message}"
+  puts e.backtrace if options[:verbose]
+  exit 1
+end