UrlCategorise 0.1.6 → 0.1.7

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c1271c16022cb3fb5e9efe9af22deca404204ee54e1b23243c4ebd862565c76b
-  data.tar.gz: '082814ad46484b87027e0a1d0042f3f37ab20a3d491f4a8e39a585e14fb40ac8'
+  metadata.gz: e4725733a26240e3bc23bfe9292d3ed0222db4e0308c5228da12a9ede1347e4b
+  data.tar.gz: a315a03357b48260c543459fab91ac8ec0601a0d7001b120bdc6f72e543e3671
 SHA512:
-  metadata.gz: 8f74b667ba1d993237e224d3db7fcb0323816716961280e5fd27fbf162ecb162f83fd126af89ce812e98fbf8fe48a15c47697b656e345d23465a23ab9913581e
-  data.tar.gz: 57ed94a3e9a9be6db60c3d48dd9d4ac4da52836951b06a9e6937c44968e5509cda43193491c9ec20039649e01d5bb4a6214f769aab2d6c27c8d047d7b9bd5273
+  metadata.gz: 74c8be653a2ae97ad74300105a95c326685d9669defd5c980cf16636e96965080a587453c21d5af6014874145f8f93d66c444574142f1568588f82b9577a7440
+  data.tar.gz: 450ae86237aa6a743a289e9b1165b9bae49f78068af8d727114763629b863567dc37de1b2ba38522e93a0ae08dd1683b79a062df2e2d18e76884fd4f8415c1db

data/.claude/settings.local.json

@@ -15,7 +15,8 @@
       "Bash(timeout 30 ruby dataset_loading_example.rb)",
       "Bash(timeout:*)",
       "Bash(DEBUG=1 timeout 300 ruby correct_usage_example.rb)",
-      "Bash(chmod:*)"
+      "Bash(chmod:*)",
+      "Bash(bundle exec bin/export_csv:*)"
     ],
     "deny": []
   }

data/.gitignore

@@ -55,3 +55,4 @@ build-iPhoneSimulator/
 # Used by RuboCop. Remote config files pulled in from inherit_from directive.
 # .rubocop-https?--*
 url_cache/*
+exports/*

data/CLAUDE.md

@@ -103,6 +103,34 @@ The gem includes automatic monitoring and cleanup of broken URLs:
 
 ### New Features (Latest Version)
 
+#### Video Hosting Detection and Regex Categorization
+Advanced video content detection system with:
+
+- **Comprehensive Video Hosting Lists**: Generate PiHole-compatible hosts files from yt-dlp extractors  
+- **Regex-Based Content Detection**: Distinguish between video content URLs vs homepages/profiles/playlists
+- **Direct Video URL Detection**: `video_url?` method to check if URLs are direct video content links
+- **Automatic List Generation**: `bin/generate_video_lists` script fetches and processes yt-dlp data
+- **Video Hosting Category**: Separate `video_hosting` category with 3,500+ domains
+- **Smart Content Categorization**: URLs matching video patterns get `*_content` suffix categories
+- **Remote Pattern Files**: Automatically downloads video patterns from GitHub repository
+
+```ruby
+# Enable regex categorization for video content detection (uses remote patterns by default)
+client = UrlCategorise::Client.new(regex_categorization: true)
+
+# Basic domain categorization
+client.categorise('https://youtube.com') # => [:video_hosting]
+
+# Enhanced content detection  
+client.categorise('https://youtube.com/watch?v=abc123') # => [:video_hosting, :video_hosting_content]
+
+# Direct video URL detection
+client.video_url?('https://youtube.com/watch?v=abc123')  # => true
+client.video_url?('https://youtube.com')                # => false
+client.video_url?('https://vimeo.com/123456789')       # => true
+client.video_url?('https://tiktok.com/@user/video/123') # => true
+```
+
 #### Dynamic Settings with ActiveAttr
 The Client class now uses ActiveAttr to provide dynamic attribute modification:
 
@@ -141,7 +169,7 @@ result = client.export_hosts_files('/custom/export/path')
 ```
 
 ##### CSV Data Export
-Export all data as a single CSV file for AI training and analysis:
+Export all data as a single comprehensive CSV file for AI training and analysis:
 
 ```ruby
 # Export to default location (cache_dir/exports/csv or ./exports/csv)
@@ -150,33 +178,68 @@ result = client.export_csv_data
 # Export to custom location  
 result = client.export_csv_data('/custom/export/path')
 
-# CSV includes: domain, category, source_type, is_dataset_category, iab_category_v2, iab_category_v3, export_timestamp
-# Metadata file includes: export info, client settings, data summary, dataset metadata
+# Returns:
+# {
+#   csv_file: 'url_categorise_comprehensive_export_TIMESTAMP.csv',
+#   summary_file: 'export_summary_TIMESTAMP.json',
+#   total_entries: 75000,
+#   summary: { domain_categorization_entries: 50000, dataset_content_entries: 25000 },
+#   export_directory: '/export/path'
+# }
 ```
 
+**Comprehensive Export Features:**
+- **Everything in One File**: Combined domains, categories, and raw dataset content
+- **Rich Dataset Content**: Original titles, descriptions, summaries, and text from datasets
+- **Dynamic Headers**: Automatically includes all available fields from any dataset
+- **Data Type Tracking**: Distinguishes between processed domains and raw dataset entries
+- **Perfect for AI/ML**: Single file with both structured categorization and rich textual features
+
 #### CLI Commands
-New command-line utilities for data export:
+Command-line utilities for comprehensive data export:
 
 ```bash
-# Export hosts files
+# Export hosts files per category
 $ bundle exec export_hosts --output /tmp/hosts --verbose
 
-# Export CSV data with IAB compliance
-$ bundle exec export_csv --output /tmp/csv --iab-compliance --verbose
+# Full CSV export with datasets and all features
+$ bundle exec export_csv --auto-load-datasets --iab-compliance --smart-categorization --verbose
+
+# Custom configuration export
+$ bundle exec export_csv --cache-dir ./custom_cache --kaggle-credentials ~/kaggle.json --output /tmp/export
 
-# Check URL health (existing)
+# Basic domain categorization only
+$ bundle exec export_csv --output /tmp/basic
+
+# Health check for all blocklist URLs
 $ bundle exec check_lists
+
+# Generate updated video hosting lists
+$ ruby bin/generate_video_lists
 ```
 
+**Enhanced CLI Features:**
+- `--auto-load-datasets`: Automatically load datasets from constants for rich content export
+- `--kaggle-credentials FILE`: Custom Kaggle API credentials file path
+- Full integration with all client features (IAB compliance, smart categorization, etc.)
+- Verbose output shows dataset statistics and loading progress
+- Video list generation from yt-dlp extractors with manual curation
+
 ### List Sources
 Primary sources include:
 - The Block List Project
 - hagezi/dns-blocklists  
 - StevenBlack/hosts
 - Various specialized security lists
+- **yt-dlp video extractors**: Comprehensive video hosting domain detection (3,500+ domains)
+- **GitHub-hosted video patterns**: Remote video URL detection patterns with manual curation
 - **Kaggle datasets**: Public URL classification datasets
 - **Custom CSV files**: Direct CSV dataset URLs with flexible column mapping
 
+**Video hosting lists are now automatically fetched from:**
+- Video domains: `https://raw.githubusercontent.com/TRex22/url_categorise/refs/heads/main/lists/video_hosting_domains.hosts`
+- URL patterns: `https://raw.githubusercontent.com/TRex22/url_categorise/refs/heads/main/lists/video_url_patterns.txt`
+
 ### Testing Guidelines
 - Mock all HTTP requests using WebMock
 - Test both success and failure scenarios

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    UrlCategorise (0.1.6)
+    UrlCategorise (0.1.7)
       active_attr (>= 0.17.1, < 1.0)
       api_pattern (>= 0.0.6, < 1.0)
       csv (>= 3.3.0, < 4.0)
@@ -11,6 +11,7 @@ PATH
       json (>= 2.7.0, < 3.0)
       kaggle (>= 0.0.3, < 1.0)
       nokogiri (>= 1.18.9, < 2.0)
+      reline (>= 0.6.2)
       resolv (>= 0.4.0, < 1.0)
       rubyzip (>= 2.3.0, < 3.0)
 
@@ -88,6 +89,7 @@ GEM
       multi_xml (>= 0.5.2)
     i18n (1.14.7)
       concurrent-ruby (~> 1.0)
+    io-console (0.8.1)
     json (2.13.2)
     kaggle (0.0.3)
       csv (>= 3.3)
@@ -137,6 +139,8 @@ GEM
       loofah (~> 2.21)
       nokogiri (>= 1.15.7, != 1.16.7, != 1.16.6, != 1.16.5, != 1.16.4, != 1.16.3, != 1.16.2, != 1.16.1, != 1.16.0.rc1, != 1.16.0)
     rake (13.3.0)
+    reline (0.6.2)
+      io-console (~> 0.5)
     resolv (0.6.2)
     rexml (3.4.1)
     ruby-progressbar (1.13.0)

data/README.md

@@ -5,6 +5,8 @@ A comprehensive Ruby gem for categorizing URLs and domains based on various secu
 ## Features
 
 - **Comprehensive Coverage**: 60+ high-quality categories including security, content, and specialized lists
+- **Video Content Detection**: Advanced regex-based categorization with `video_url?` method to distinguish video content from other website resources
+- **Custom Video Lists**: Generate and maintain comprehensive video hosting domain lists using yt-dlp extractors
 - **Kaggle Dataset Integration**: Automatic loading and processing of machine learning datasets from Kaggle
 - **Multiple Data Sources**: Supports blocklists, CSV datasets, and Kaggle ML datasets  
 - **Multiple List Formats**: Supports hosts files, pfSense, AdSense, uBlock Origin, dnsmasq, and plain text formats
@@ -115,7 +117,7 @@ Each category gets its own hosts file (e.g., `malware.hosts`, `advertising.hosts
 
 #### CSV Data Export
 
-Export all data as a single CSV file for AI training and analysis:
+Export all data as a single comprehensive CSV file for AI training and analysis:
 
 ```ruby
 # Export to default location
@@ -125,30 +127,63 @@ result = client.export_csv_data
 client.iab_compliance_enabled = true
 result = client.export_csv_data('/custom/export/path')
 
-# CSV includes comprehensive data:
-# - domain, category, source_type, is_dataset_category
-# - iab_category_v2, iab_category_v3, export_timestamp
-# - smart_categorization_enabled
-
-# Metadata file includes:
-# - Export info, client settings, data summary, dataset metadata
+# Returns information about created files:
+# {
+#   csv_file: '/path/url_categorise_comprehensive_export_20231201_143022.csv',
+#   summary_file: '/path/export_summary_20231201_143022.json',
+#   total_entries: 50000,
+#   summary: { ... },
+#   export_directory: '/path'
+# }
 ```
 
+**Single comprehensive CSV file contains:**
+
+- **Domain Categorization Data**: All processed domains with categories, source types, IAB mappings
+- **Raw Dataset Content**: Original dataset entries with titles, descriptions, text, summaries, and all available fields
+- **Dynamic Headers**: Automatically adapts to include all available data fields
+- **Data Type Column**: Distinguishes between 'domain_categorization', 'raw_dataset_content', etc.
+
+**Key Features:**
+- Everything in one file for easy analysis and AI/ML training
+- Rich textual content from original datasets
+- IAB Content Taxonomy compliance mapping
+- Smart categorization metadata
+- Source type tracking (dataset vs blocklist)
+
 #### CLI Commands
 
-New command-line utilities for data export:
+Command-line utilities for data export:
 
 ```bash
 # Export hosts files
 $ bundle exec export_hosts --output /tmp/hosts --verbose
 
-# Export CSV data with IAB compliance
-$ bundle exec export_csv --output /tmp/csv --iab-compliance --verbose
+# Export CSV data with all features enabled
+$ bundle exec export_csv --output /tmp/csv --iab-compliance --smart-categorization --auto-load-datasets --verbose
+
+# Generate updated video hosting lists
+$ ruby bin/generate_video_lists
+
+# Check health of all blocklist URLs
+$ bundle exec check_lists
+
+# Export with custom Kaggle credentials
+$ bundle exec export_csv --auto-load-datasets --kaggle-credentials ~/my-kaggle.json --verbose
+
+# Basic export (domains only)
+$ bundle exec export_csv --output /tmp/csv
 
 # Check URL health (existing command)
 $ bundle exec check_lists
 ```
 
+**Key CLI Options:**
+- `--auto-load-datasets`: Load datasets from constants to include rich text content
+- `--kaggle-credentials FILE`: Specify custom Kaggle credentials file
+- `--iab-compliance`: Enable IAB Content Taxonomy mapping
+- `--smart-categorization`: Enable intelligent category filtering
+
 ## Advanced Configuration
 
 ### File Caching
@@ -228,6 +263,89 @@ host_urls = {
 client = UrlCategorise::Client.new(host_urls: host_urls)
 ```
 
+### Video Content Detection
+
+The gem includes advanced regex-based categorization specifically for video hosting platforms. This helps distinguish between actual video content URLs and other resources like homepages, user profiles, playlists, or community content.
+
+#### Video Hosting Domains
+
+The gem maintains a comprehensive list of video hosting domains extracted from yt-dlp (YouTube-dl fork) extractors:
+
+```ruby
+# Generate/update video hosting lists
+system("ruby bin/generate_video_lists")
+
+# Use video hosting categorization
+client = UrlCategorise::Client.new
+categories = client.categorise("youtube.com")
+# => [:video_hosting]
+```
+
+#### Video Content vs Other Resources
+
+Enable regex categorization to distinguish video content from other resources:
+
+```ruby
+client = UrlCategorise::Client.new(
+  regex_categorization: true  # Uses remote video patterns by default
+)
+
+# Regular homepage gets basic category
+client.categorise("https://youtube.com")
+# => [:video_hosting]
+
+# Actual video URL gets enhanced categorization
+client.categorise("https://youtube.com/watch?v=dQw4w9WgXcQ") 
+# => [:video_hosting, :video_hosting_content]
+
+# User profile page - no content enhancement
+client.categorise("https://youtube.com/@username")
+# => [:video_hosting]
+```
+
+#### Direct Video URL Detection
+
+Use the `video_url?` method to check if a URL is a direct link to video content:
+
+```ruby
+client = UrlCategorise::Client.new(regex_categorization: true)
+
+# Check if URLs are direct video content links
+client.video_url?("https://youtube.com/watch?v=dQw4w9WgXcQ")  # => true
+client.video_url?("https://youtube.com")                     # => false
+client.video_url?("https://youtube.com/@channel")            # => false
+client.video_url?("https://vimeo.com/123456789")            # => true
+client.video_url?("https://tiktok.com/@user/video/123")     # => true
+
+# Works with various video hosting platforms
+client.video_url?("https://dailymotion.com/video/x7abc123") # => true
+client.video_url?("https://twitch.tv/videos/1234567890")    # => true
+
+# Returns false for non-video domains
+client.video_url?("https://google.com/search?q=cats")       # => false
+```
+
+**How it works:**
+1. First checks if the URL is from a known video hosting domain
+2. Then uses regex patterns to determine if it's a direct video content URL
+3. Returns `true` only if both conditions are met
+4. Handles invalid URLs gracefully (returns `false`)
+
+#### Maintaining Video Lists
+
+The gem includes a script to generate and maintain comprehensive video hosting lists:
+
+```bash
+# Generate updated video hosting lists
+ruby bin/generate_video_lists
+
+# This creates:
+# - lists/video_hosting_domains.hosts (PiHole compatible)  
+# - lists/video_url_patterns.txt (Regex patterns for content detection)
+```
+
+The script fetches data from yt-dlp extractors and combines it with manually curated major platforms to ensure comprehensive coverage.
+
 ### Smart Categorization (Post-Processing)
 
 Smart categorization solves the problem of overly broad domain-level categorization. For example, `reddit.com` might appear in health & fitness blocklists, but not all Reddit content is health-related.

data/bin/export_csv

@@ -9,13 +9,15 @@ options = {
   cache_dir: nil,
   verbose: false,
   iab_compliance: false,
-  smart_categorization: false
+  smart_categorization: false,
+  auto_load_datasets: false,
+  kaggle_credentials_file: nil
 }
 
 OptionParser.new do |opts|
   opts.banner = "Usage: #{$0} [options]"
   opts.separator ""
-  opts.separator "Export all categorized domains and metadata as a single CSV file for AI training"
+  opts.separator "Export all categorized domains and dataset content as a single CSV file for AI training"
   opts.separator ""
   
   opts.on("-o", "--output PATH", "Output directory path (default: cache_dir/exports/csv or ./exports/csv)") do |path|
@@ -34,6 +36,14 @@ OptionParser.new do |opts|
     options[:smart_categorization] = true
   end
   
+  opts.on("--auto-load-datasets", "Auto-load datasets from constants for rich content export") do
+    options[:auto_load_datasets] = true
+  end
+  
+  opts.on("--kaggle-credentials FILE", "Path to Kaggle credentials file (default: ~/.kaggle/kaggle.json)") do |file|
+    options[:kaggle_credentials_file] = file
+  end
+  
   opts.on("-v", "--verbose", "Verbose output") do
     options[:verbose] = true
   end
@@ -48,12 +58,38 @@ puts "=== UrlCategorise CSV Data Export ===" if options[:verbose]
 puts "Initializing client..." if options[:verbose]
 
 begin
+  # Build dataset config if datasets should be loaded
+  dataset_config = {}
+  if options[:auto_load_datasets]
+    dataset_config = {
+      cache_path: options[:cache_dir] ? File.join(options[:cache_dir], 'datasets') : './url_cache/datasets',
+      download_path: options[:cache_dir] ? File.join(options[:cache_dir], 'downloads') : './url_cache/downloads'
+    }
+    
+    # Add Kaggle credentials if provided
+    if options[:kaggle_credentials_file]
+      dataset_config[:kaggle] = { credentials_file: options[:kaggle_credentials_file] }
+    elsif File.exist?(File.expand_path('~/.kaggle/kaggle.json'))
+      dataset_config[:kaggle] = { credentials_file: '~/.kaggle/kaggle.json' }
+    end
+  end
+
   client = UrlCategorise::Client.new(
     cache_dir: options[:cache_dir],
     iab_compliance: options[:iab_compliance],
-    smart_categorization: options[:smart_categorization]
+    smart_categorization: options[:smart_categorization],
+    auto_load_datasets: options[:auto_load_datasets],
+    dataset_config: dataset_config
   )
   
+  if options[:verbose] && options[:auto_load_datasets]
+    puts "Client initialized with dataset loading enabled"
+    puts "Dataset statistics:"
+    puts "  Dataset categories: #{client.count_of_dataset_categories}"
+    puts "  Dataset hosts: #{client.count_of_dataset_hosts.to_s.reverse.gsub(/(\\d{3})(?=\\d)/, '\\1,').reverse}"
+    puts "  Dataset data size: #{client.size_of_dataset_data.round(2)} MB" if client.respond_to?(:size_of_dataset_data)
+  end
+  
   puts "Exporting CSV data..." if options[:verbose]
   
   result = client.export_csv_data(options[:output_path])
@@ -61,13 +97,14 @@ begin
   puts "\n✅ Export completed successfully!"
   puts "📁 Export directory: #{result[:export_directory]}"
   puts "📄 CSV file: #{result[:csv_file]}"
-  puts "📄 Metadata file: #{result[:metadata_file]}"
+  puts "📄 Summary file: #{result[:summary_file]}"
   
   puts "\n📊 Data Summary:"
-  puts "  Total domains: #{result[:summary][:total_domains]}"
+  puts "  Total entries: #{result[:total_entries]}"
+  puts "  Domain categorizations: #{result[:summary][:domain_categorization_entries]}"
+  puts "  Dataset content entries: #{result[:summary][:dataset_content_entries]}"
   puts "  Total categories: #{result[:summary][:total_categories]}"
-  puts "  Dataset categories: #{result[:summary][:dataset_categories_count]}"
-  puts "  Blocklist categories: #{result[:summary][:blocklist_categories_count]}"
+  puts "  Has dataset content: #{result[:summary][:has_dataset_content]}"
   
   if options[:verbose]
     puts "\n🏷️  Categories included:"