UrlCategorise 0.1.3 → 0.1.7

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: dcb05d79b6bc09b5b338183d412cd309d9634a342c95b14ea25df5926d8609fb
-  data.tar.gz: effa4c7a010ee574fe6a41653af553a68710ceca46ebdb9dd5352096af5fa7e1
+  metadata.gz: e4725733a26240e3bc23bfe9292d3ed0222db4e0308c5228da12a9ede1347e4b
+  data.tar.gz: a315a03357b48260c543459fab91ac8ec0601a0d7001b120bdc6f72e543e3671
 SHA512:
-  metadata.gz: a527c801cbf6318305d640925dd75922c2ac1bcc76a5de75c75e3ad24698305c3d7d885d7da8dd280e61ceb1fe91a57eac185c5c209e11685f2ddb6833b120b9
-  data.tar.gz: de81765d20a0b36c54b71b935928777140f86f6e0a130c71ec7804ed28c1b3d8f12ca30fff5cf8c93952aaed7a9c279fe35a0d32bc83a851aad2556b55fd7942
+  metadata.gz: 74c8be653a2ae97ad74300105a95c326685d9669defd5c980cf16636e96965080a587453c21d5af6014874145f8f93d66c444574142f1568588f82b9577a7440
+  data.tar.gz: 450ae86237aa6a743a289e9b1165b9bae49f78068af8d727114763629b863567dc37de1b2ba38522e93a0ae08dd1683b79a062df2e2d18e76884fd4f8415c1db

data/.claude/settings.local.json

@@ -7,10 +7,16 @@
       "Bash(bundle exec ruby:*)",
       "Bash(find:*)",
       "Bash(grep:*)",
-      "Read(//Users/trex22/development/rubygems/kaggle/**)",
       "Bash(for file in test/url_categorise/*dataset*test.rb)",
       "Bash(do echo \"Checking $file...\")",
-      "Bash(done)"
+      "Bash(done)",
+      "Bash(bundle exec rubocop:*)",
+      "Bash(rubocop:*)",
+      "Bash(timeout 30 ruby dataset_loading_example.rb)",
+      "Bash(timeout:*)",
+      "Bash(DEBUG=1 timeout 300 ruby correct_usage_example.rb)",
+      "Bash(chmod:*)",
+      "Bash(bundle exec bin/export_csv:*)"
     ],
     "deny": []
   }

data/.gitignore

@@ -54,3 +54,5 @@ build-iPhoneSimulator/
 
 # Used by RuboCop. Remote config files pulled in from inherit_from directive.
 # .rubocop-https?--*
+url_cache/*
+exports/*

data/CLAUDE.md

@@ -84,14 +84,146 @@ The gem includes automatic monitoring and cleanup of broken URLs:
 - **Data Hashing**: SHA256 content hashing for dataset change detection
 - **Category Mapping**: Flexible column detection and category mapping for datasets
 - **Credential Warnings**: Helpful warnings when Kaggle credentials are missing but functionality continues
+- **IAB Compliance**: Full support for IAB Content Taxonomy v2.0 and v3.0 standards
+- **Dataset-Specific Metrics**: Separate counting methods for dataset vs DNS list categorization
+- **Enhanced Statistics**: Extended helper methods for comprehensive data insights
+- **ActiveAttr Settings**: In-memory modification of client settings using attribute setters
+- **Data Export**: Multiple export formats including hosts files per category and CSV data exports
+- **CLI Commands**: Command-line utilities for data export and list checking
 
 ### Architecture
-- `Client` class: Main interface for categorization
+- `Client` class: Main interface for categorization with IAB compliance support and ActiveAttr attributes
 - `DatasetProcessor` class: Handles Kaggle and CSV dataset processing
+- `IabCompliance` module: Maps categories to IAB Content Taxonomy v2.0/v3.0 standards
 - `Constants` module: Contains default list URLs and categories
 - `ActiveRecordClient` class: Database-backed client with dataset history
 - Modular design allows extending with new list sources and datasets
-- Support for custom list directories, caching, and dataset integration
+- Support for custom list directories, caching, dataset integration, IAB compliance, and data export
+- ActiveAttr integration for dynamic setting modification and attribute validation
+
+### New Features (Latest Version)
+
+#### Video Hosting Detection and Regex Categorization
+Advanced video content detection system with:
+
+- **Comprehensive Video Hosting Lists**: Generate PiHole-compatible hosts files from yt-dlp extractors  
+- **Regex-Based Content Detection**: Distinguish between video content URLs vs homepages/profiles/playlists
+- **Direct Video URL Detection**: `video_url?` method to check if URLs are direct video content links
+- **Automatic List Generation**: `bin/generate_video_lists` script fetches and processes yt-dlp data
+- **Video Hosting Category**: Separate `video_hosting` category with 3,500+ domains
+- **Smart Content Categorization**: URLs matching video patterns get `*_content` suffix categories
+- **Remote Pattern Files**: Automatically downloads video patterns from GitHub repository
+
+```ruby
+# Enable regex categorization for video content detection (uses remote patterns by default)
+client = UrlCategorise::Client.new(regex_categorization: true)
+
+# Basic domain categorization
+client.categorise('https://youtube.com') # => [:video_hosting]
+
+# Enhanced content detection  
+client.categorise('https://youtube.com/watch?v=abc123') # => [:video_hosting, :video_hosting_content]
+
+# Direct video URL detection
+client.video_url?('https://youtube.com/watch?v=abc123')  # => true
+client.video_url?('https://youtube.com')                # => false
+client.video_url?('https://vimeo.com/123456789')       # => true
+client.video_url?('https://tiktok.com/@user/video/123') # => true
+```
+
+#### Dynamic Settings with ActiveAttr
+The Client class now uses ActiveAttr to provide dynamic attribute modification:
+
+```ruby
+client = UrlCategorise::Client.new
+
+# Modify settings in-memory
+client.smart_categorization_enabled = true
+client.iab_compliance_enabled = true
+client.iab_version = :v2
+client.request_timeout = 30
+client.dns_servers = ['8.8.8.8', '8.8.4.4']
+
+# Settings take effect immediately
+categories = client.categorise('reddit.com') # Uses new smart categorization rules
+```
+
+#### Data Export Features
+
+##### Hosts File Export
+Export all categorized domains as separate hosts files per category:
+
+```ruby
+# Export to default location (cache_dir/exports/hosts or ./exports/hosts)
+result = client.export_hosts_files
+
+# Export to custom location
+result = client.export_hosts_files('/custom/export/path')
+
+# Returns hash with file information:
+# {
+#   malware: { path: '/path/malware.hosts', filename: 'malware.hosts', count: 1500 },
+#   advertising: { path: '/path/advertising.hosts', filename: 'advertising.hosts', count: 25000 },
+#   _summary: { total_categories: 15, total_domains: 50000, export_directory: '/path' }
+# }
+```
+
+##### CSV Data Export
+Export all data as a single comprehensive CSV file for AI training and analysis:
+
+```ruby
+# Export to default location (cache_dir/exports/csv or ./exports/csv)
+result = client.export_csv_data
+
+# Export to custom location  
+result = client.export_csv_data('/custom/export/path')
+
+# Returns:
+# {
+#   csv_file: 'url_categorise_comprehensive_export_TIMESTAMP.csv',
+#   summary_file: 'export_summary_TIMESTAMP.json',
+#   total_entries: 75000,
+#   summary: { domain_categorization_entries: 50000, dataset_content_entries: 25000 },
+#   export_directory: '/export/path'
+# }
+```
+
+**Comprehensive Export Features:**
+- **Everything in One File**: Combined domains, categories, and raw dataset content
+- **Rich Dataset Content**: Original titles, descriptions, summaries, and text from datasets
+- **Dynamic Headers**: Automatically includes all available fields from any dataset
+- **Data Type Tracking**: Distinguishes between processed domains and raw dataset entries
+- **Perfect for AI/ML**: Single file with both structured categorization and rich textual features
+
+#### CLI Commands
+Command-line utilities for comprehensive data export:
+
+```bash
+# Export hosts files per category
+$ bundle exec export_hosts --output /tmp/hosts --verbose
+
+# Full CSV export with datasets and all features
+$ bundle exec export_csv --auto-load-datasets --iab-compliance --smart-categorization --verbose
+
+# Custom configuration export
+$ bundle exec export_csv --cache-dir ./custom_cache --kaggle-credentials ~/kaggle.json --output /tmp/export
+
+# Basic domain categorization only
+$ bundle exec export_csv --output /tmp/basic
+
+# Health check for all blocklist URLs
+$ bundle exec check_lists
+
+# Generate updated video hosting lists
+$ ruby bin/generate_video_lists
+```
+
+**Enhanced CLI Features:**
+- `--auto-load-datasets`: Automatically load datasets from constants for rich content export
+- `--kaggle-credentials FILE`: Custom Kaggle API credentials file path
+- Full integration with all client features (IAB compliance, smart categorization, etc.)
+- Verbose output shows dataset statistics and loading progress
+- Video list generation from yt-dlp extractors with manual curation
 
 ### List Sources
 Primary sources include:
@@ -99,9 +231,15 @@ Primary sources include:
 - hagezi/dns-blocklists  
 - StevenBlack/hosts
 - Various specialized security lists
+- **yt-dlp video extractors**: Comprehensive video hosting domain detection (3,500+ domains)
+- **GitHub-hosted video patterns**: Remote video URL detection patterns with manual curation
 - **Kaggle datasets**: Public URL classification datasets
 - **Custom CSV files**: Direct CSV dataset URLs with flexible column mapping
 
+**Video hosting lists are now automatically fetched from:**
+- Video domains: `https://raw.githubusercontent.com/TRex22/url_categorise/refs/heads/main/lists/video_hosting_domains.hosts`
+- URL patterns: `https://raw.githubusercontent.com/TRex22/url_categorise/refs/heads/main/lists/video_url_patterns.txt`
+
 ### Testing Guidelines
 - Mock all HTTP requests using WebMock
 - Test both success and failure scenarios

data/Gemfile.lock

@@ -1,14 +1,17 @@
 PATH
   remote: .
   specs:
-    UrlCategorise (0.1.3)
+    UrlCategorise (0.1.7)
+      active_attr (>= 0.17.1, < 1.0)
       api_pattern (>= 0.0.6, < 1.0)
       csv (>= 3.3.0, < 4.0)
       digest (>= 3.1.0, < 4.0)
       fileutils (>= 1.7.0, < 2.0)
       httparty (>= 0.22.0, < 1.0)
       json (>= 2.7.0, < 3.0)
+      kaggle (>= 0.0.3, < 1.0)
       nokogiri (>= 1.18.9, < 2.0)
+      reline (>= 0.6.2)
       resolv (>= 0.4.0, < 1.0)
       rubyzip (>= 2.3.0, < 3.0)
 
@@ -86,7 +89,14 @@ GEM
       multi_xml (>= 0.5.2)
     i18n (1.14.7)
       concurrent-ruby (~> 1.0)
+    io-console (0.8.1)
     json (2.13.2)
+    kaggle (0.0.3)
+      csv (>= 3.3)
+      fileutils (>= 1.7)
+      httparty (>= 0.23)
+      oj (= 3.16.11)
+      rubyzip (>= 2.0)
     logger (1.7.0)
     loofah (2.24.1)
       crass (~> 1.0.2)
@@ -107,6 +117,10 @@ GEM
       bigdecimal (~> 3.1)
     nokogiri (1.18.9-arm64-darwin)
       racc (~> 1.4)
+    oj (3.16.11)
+      bigdecimal (>= 3.0)
+      ostruct (>= 0.2)
+    ostruct (0.6.3)
     pry (0.15.2)
       coderay (~> 1.1)
       method_source (~> 1.0)
@@ -125,6 +139,8 @@ GEM
       loofah (~> 2.21)
       nokogiri (>= 1.15.7, != 1.16.7, != 1.16.6, != 1.16.5, != 1.16.4, != 1.16.3, != 1.16.2, != 1.16.1, != 1.16.0.rc1, != 1.16.0)
     rake (13.3.0)
+    reline (0.6.2)
+      io-console (~> 0.5)
     resolv (0.6.2)
     rexml (3.4.1)
     ruby-progressbar (1.13.0)