ligamagic-scraper 0.6.0

data/README.md

@@ -0,0 +1,614 @@
+# Liga Magic Scraper
+
+A Ruby gem for scraping product data from Liga Magic's card listings using Capybara and Selenium.
+
+  - 💎 Can be used as a gem or standalone CLI tool
+  - 🖥️ Supports headed (visible) and headless browser modes
+  - 📝 Saves results to JSON files in the `scrapped/` directory
+## Roadmap
+
+- [Global Ligamagic product scrapping](#global-ligamagic-product-scrapping)
+- [Store specific scrapping](#store-specific-scrapping)
+- [Alert System](#alert-system)
+
+## Features
+
+### Global Ligamagic product scrapping
+  - [X] Extracts product name, ID, and pricing information
+  - [X] Automatically clicks "Load More" to paginate through all products
+  - [X] Stops when unavailable products are encountered
+  - [X] Uses a global-search flag (`-g` or `--global`)
+
+### Store specific scrapping
+  - [X] Extracts product name and pricing information from any Liga Magic store
+  - [X] Accepts store domain name only (e.g., `test-store`)
+  - [X] Store search with search term support (`-u STORE -s TERM`)
+  - [X] Store listing with max pages limit (`-u STORE -p N`)
+  - [X] Automatic pagination with safety limits
+  - [X] Automatically orders by price (most expensive first)
+  - [X] Filters only in-stock items automatically
+  - [X] Automatically builds clean URLs with only necessary parameters
+  - [X] Uses `--store` or `-u` flag
+  - [X] Extracts store name and ID automatically
+  - [X] Corrected selectors for card-desc structure
+  - [!] **Limitation**: Store searches with search terms cannot extract price/qty due to CSS obfuscation (see docs)
+
+### Alert System
+  - [X] Detects changes between scrapes (new products, removed products, price changes, availability changes)
+  - [X] Compares current scrape with most recent previous scrape
+  - [X] Multiple alert types supported: file, telegram
+  - [X] CLI flag to enable alerts: `-a` or `--alerts`
+  - [X] Configurable alert types via `--alert-types`
+  - [X] File alert fully implemented with organized directory structure
+  - [ ] Telegram alert action - currently stub only
+
+## Requirements
+
+- Ruby 2.7 or higher
+- Chrome browser installed
+- ChromeDriver (automatically managed by selenium-webdriver)
+
+## Installation
+
+### As a Global Command
+
+Install the gem globally to use the `ligamagic-scraper` command anywhere:
+
+```bash
+# Build and install the gem
+gem build ligamagic-scraper.gemspec
+gem install ./ligamagic-scraper-0.1.0.gem
+```
+
+### As a Library in Your Project
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'ligamagic-scraper', path: '/path/to/ligamagic-scrapper'
+```
+
+Or install locally:
+
+```bash
+bundle install
+```
+
+## Usage
+
+### Command Line Interface (Global Installation)
+
+After installing the gem globally, you can use the `ligamagic-scraper` command from anywhere:
+
+```bash
+ligamagic-scraper -s "booster box"
+```
+
+**Search Options:**
+- `-s`, `--search SEARCH` - Search term (global search or store search with -u)
+- `-u`, `--store DOMAIN` - Store domain (e.g., `test-store`)
+- `-p`, `--pages N` - Pages to scrape (max: 5, required with -u if no -s)
+
+**Options:**
+- `-g`, `--global` - Use global Liga Magic search (default, optional)
+- `-b`, `--browser-mode MODE` - Browser mode: `headed` (default, visible browser) or `headless` (no UI)
+- `-a`, `--alerts` - Enable alert system (detects changes from previous scrapes)
+- `--alert-types TYPES` - Alert types: `file`, `email`, `telegram`, `webhook` (comma-separated, default: file)
+- `-h`, `--help` - Show help message
+- `--version` - Show version
+
+**Usage Examples:**
+
+| Use Case | Command | Pages | Browser |
+|----------|---------|-------|---------|
+| **Global Search** | | | |
+| Basic global search | `ligamagic-scraper -s "booster box"` | Unlimited | Visible |
+| Global search headless | `ligamagic-scraper -s "dual land" -b headless` | Unlimited | Headless |
+| Global search with alerts | `ligamagic-scraper -s "commander" -a` | Unlimited | Visible |
+| **Store Listing** | | | |
+| List 2 pages | `ligamagic-scraper -u test-store -p 2` | 2 | Visible |
+| List 5 pages (max) | `ligamagic-scraper -u test-store -p 5` | 5 | Visible |
+| List with 10 (capped at 5) | `ligamagic-scraper -u test-store -p 10` | 5 (max) | Visible |
+| List store headless | `ligamagic-scraper -u test-store -p 3 -b headless` | 3 | Headless |
+| List store with alerts | `ligamagic-scraper -u test-store -p 4 -a` | 4 | Visible |
+| **Store Search** | | | |
+| Search within store | `ligamagic-scraper -u test-store -s "volcanic"` | Unlimited | Visible |
+| Store search headless | `ligamagic-scraper -u test-store -s "force of will" -b headless` | Unlimited | Headless |
+| Store search with alerts | `ligamagic-scraper -u test-store -s "mox" -a` | Unlimited | Visible |
+| **Advanced** | | | |
+| Multiple alert types | `ligamagic-scraper -s "lotus" -a --alert-types file,telegram` | Unlimited | Visible |
+| Store with custom alerts | `ligamagic-scraper -u test-store -s "power nine" -a --alert-types file,email` | Unlimited | Visible |
+
+**Search Modes:**
+
+The gem supports three search modes:
+
+| Mode | Flags | Pages | When to Use |
+|------|-------|-------|-------------|
+| **Global Search** | `-s TERM` | Unlimited | Search across all Liga Magic stores |
+| **Store Listing** | `-u STORE -p N` | 1-5 (max) | List products from a specific store |
+| **Store Search** | `-u STORE -s TERM` | Unlimited | Search for specific products within a store |
+
+**Notes:**
+- **Maximum 5 pages** allowed for store listings (values above 5 are capped)
+  - `-p 2` uses 2 pages
+  - `-p 5` uses 5 pages
+  - `-p 10` is capped to 5 pages
+- Store searches with search term have unlimited pagination
+- **Automatic filtering**: Store results are automatically:
+  - Ordered by price (most expensive first)
+  - Filtered to show only in-stock items
+- **Limitation**: Store searches with search terms (`-u STORE -s TERM`) cannot extract price/qty
+  - Liga Magic uses CSS sprite obfuscation for anti-scraping protection
+  - Only card name and ID are extracted when searching
+  - Store listings (`-u STORE -p N`) work normally with full price/qty data
+- Use `-b headless` for background scraping without browser UI
+
+### Using as a Library
+
+You can also import and use the scraper in your Ruby projects:
+
+#### Global Search
+
+```ruby
+require 'ligamagic_scraper'
+
+# Create a global scraper instance
+scraper = LigaMagicScraper::GlobalScraper.new(
+  search_term: "booster box",
+  browser_mode: 'headless'
+)
+
+# Run the scrape
+results = scraper.scrape
+
+# Save results to JSON
+scraper.save_to_json(results)
+
+# Access the results
+results.each do |product|
+  puts "#{product[:name]}: R$ #{product[:min_price]}"
+end
+
+# Access execution logs
+scraper.formatted_logs.each { |log| puts log }
+```
+
+#### Store-Specific Search
+
+```ruby
+require 'ligamagic_scraper'
+
+# Create a store scraper instance for listing (with max pages)
+scraper = LigaMagicScraper::StoreScraper.new(
+  store_domain: "test-store",
+  max_pages: 5,
+  browser_mode: 'headless'
+)
+
+# Run the scrape
+results = scraper.scrape
+
+# Save results to JSON
+scraper.save_to_json(results)
+
+# Access the results
+results.each do |product|
+  puts "[#{product[:card_id]}] #{product[:name]}: R$ #{product[:price]} (#{product[:qtd]} available)"
+end
+
+# Access execution logs
+scraper.formatted_logs.each { |log| puts log }
+```
+
+### Development Usage (Without Installing)
+
+If you're developing or testing, you can run the executable directly from the project:
+
+```bash
+# With visible browser (default)
+./bin/ligamagic-scraper -s "booster box"
+
+# With headless mode
+./bin/ligamagic-scraper -s "commander deck" -b headless
+
+# Or with bundle exec
+bundle exec ./bin/ligamagic-scraper -s "commander deck" -b headless
+```
+
+### Programmatic Logging
+
+All scrapers collect execution logs that can be accessed programmatically:
+
+```ruby
+scraper = LigaMagicScraper::GlobalScraper.new(search_term: "test")
+results = scraper.scrape
+
+# Access full log entries (with timestamp, level, source)
+scraper.logs
+# => [
+#   { timestamp: 2025-10-27 12:00:00, level: :info, message: "🚀 Starting...", source: "LigaMagicScraper::GlobalScraper" },
+#   ...
+# ]
+
+# Access just the messages (for display)
+scraper.formatted_logs
+# => ["🚀 Starting Liga Magic global search scraper...", "🔍 Search term: test", ...]
+
+# Filter by level
+error_logs = scraper.logs.select { |l| l[:level] == :error }
+
+# Display logs
+scraper.formatted_logs.each { |msg| puts msg }
+```
+
+**Log Levels:**
+- `:info` - Major milestones (starting, completing, saving)
+- `:debug` - Detailed progress (pagination clicks, individual product extraction)
+- `:warning` - Non-fatal issues (skipped items, limits reached)
+- `:error` - Errors and exceptions
+
+**Note:** The CLI automatically displays all logs during execution. When using as a library, logs are collected silently and can be accessed via `scraper.logs` or `scraper.formatted_logs`
+
+## How it Works
+
+The scraper will:
+1. Visit the Liga Magic search page with your search term
+2. Automatically click "Load More" until all available products are loaded
+3. Extract product names, IDs, slugs, and prices
+4. Display results in a formatted output
+5. Save results to a JSON file in `scrapped/YYYYMMDD_search_slug.json`
+6. Return a Ruby array of hashes with the data
+
+## Output Format
+
+### Global Search Output
+
+The global scraper returns an array of hashes with product details:
+
+```ruby
+[
+  {
+    id: "48851",
+    slug: "caixa_de_booster_amonkhet",
+    name: "Caixa de Booster - Amonkhet",
+    min_price: 1499.0,
+    avg_price: 1499.0,
+    max_price: 1499.0
+  },
+  {
+    id: "80494",
+    slug: "caixa_de_booster_dominaria",
+    name: "Caixa de Booster - Dominaria",
+    min_price: 1399.99,
+    avg_price: 1482.66,
+    max_price: 1499.95
+  },
+  # ...
+]
+```
+
+**Global Search Fields:**
+- `id`: Unique product code from Liga Magic (pcode), e.g., "48851" (can be null)
+- `slug`: URL-friendly version of product name (normalized, no accents, always present)
+- `name`: Full product name in Portuguese
+- `min_price`: Minimum price across all stores (float)
+- `avg_price`: Average price across all stores (float)
+- `max_price`: Maximum price across all stores (float)
+
+### Store-Specific Output
+
+The store scraper returns an array with different fields:
+
+```ruby
+[
+  {
+    card_id: "16149",
+    name: "Volcanic Island",
+    slug: "volcanic_island",
+    price: 2500.0,
+    qtd: 1,
+    available: true
+  },
+  # ...
+]
+```
+
+**Store-Specific Fields:**
+- `card_id`: Unique card identifier from Liga Magic (extracted from link URL)
+- `name`: Full product name
+- `slug`: URL-friendly version of product name
+- `price`: Product price at this store (float)
+- `qtd`: Quantity available (integer)
+- `available`: Whether the product is available (boolean based on qtd > 0)
+
+### JSON File Format
+
+Results are automatically saved to organized directories:
+
+**Global Searches:**
+- Directory: `scrapped/global/`
+- Format: `YYYYMMDD_HHMMSS__search_slug.json`
+- Examples:
+  - `scrapped/global/20251027_143022__booster_box.json`
+  - `scrapped/global/20251027_151530__commander_deck.json`
+
+**Store Searches:**
+- Directory: `scrapped/stores/{store_domain}/`
+- Format (without search): `YYYYMMDD_HHMMSS.json`
+- Format (with search): `YYYYMMDD_HHMMSS__search_slug.json`
+- Examples:
+  - `scrapped/stores/test-store/20251027_143022.json` (store listing)
+  - `scrapped/stores/test-store/20251027_151530__volcanic_island.json` (store search)
+
+The datetime format allows multiple scrapes per day, which is useful for price tracking and change detection.
+
+JSON structure:
+```json
+{
+  "search_term": "booster box",
+  "scraped_at": "2025-10-26T14:30:00-03:00",
+  "total_products": 42,
+  "products": [
+    {
+      "id": "48851",
+      "slug": "caixa_de_booster_amonkhet",
+      "name": "Caixa de Booster - Amonkhet",
+      "min_price": 1499.0,
+      "avg_price": 1499.0,
+      "max_price": 1499.0
+    }
+  ]
+}
+```
+
+## Alert System
+
+The alert system automatically detects changes between scrapes and can notify you through multiple channels.
+
+### How It Works
+
+1. **Enable alerts** with the `-a` or `--alerts` flag
+2. The system **automatically finds** the most recent previous scrape file
+3. **Compares** the current scrape with the previous one
+4. **Detects changes**:
+   - 🆕 New products added
+   - ❌ Products removed
+   - 💰 Price changes (with percentage change)
+   - 📊 Quantity changes (for store scraper)
+   - 📦 Availability changes
+5. **Sends notifications** via configured alert types
+
+### Alert Types
+
+- **file**: Save alert to JSON file in `alerts_json/` directory (implemented)
+- **telegram**: Send Telegram message (stub - not implemented yet)
+
+### CLI Usage
+
+```bash
+# Basic alert (uses file alert)
+ligamagic-scraper -s "commander deck" -a
+
+# Multiple alert types
+ligamagic-scraper -u test-store -s "volcanic" -a --alert-types file,email,telegram
+
+# Alerts with headless mode
+ligamagic-scraper -s "booster box" -a -b headless
+```
+
+### Library Usage
+
+```ruby
+require 'ligamagic_scraper'
+
+# Configure alerts
+alert_config = {
+  enabled: true,
+  alert_types: [:file, :email],
+  compare_previous: true
+}
+
+# Create scraper with alerts
+scraper = LigaMagicScraper::GlobalScraper.new(
+  search_term: "commander deck",
+  alert_config: alert_config
+)
+
+results = scraper.scrape
+scraper.save_to_json(results)  # Automatically processes alerts
+```
+
+### Detected Changes Format
+
+```ruby
+{
+  has_changes: true,
+  timestamp: "2025-10-27T...",
+  search_type: "global",
+  search_term: "commander deck",
+  new_products: [...],           # Products that weren't in previous scrape
+  removed_products: [...],       # Products that are no longer available
+  price_changes: [               # Products with price changes
+    {
+      id: "12345",
+      name: "Product Name",
+      previous_min: 100.0,
+      current_min: 85.0,
+      change: -15.0,
+      change_percent: -15.0
+    }
+  ],
+  quantity_changes: [            # Quantity changes (store scraper only)
+    {
+      id: "16149",
+      name: "Volcanic Island",
+      previous_qtd: 3,
+      current_qtd: 1,
+      change: -2
+    }
+  ],
+  availability_changes: [...]    # Products that became available/unavailable
+}
+```
+
+### Alert Output
+
+**File Alert** saves changes to organized directories:
+
+**Global Searches:**
+- Directory: `alerts_json/global/`
+- Example: `alerts_json/global/20251027_143022.json`
+
+**Store Searches:**
+- Directory: `alerts_json/stores/{store_domain}/`
+- Example: `alerts_json/stores/kamm-store/20251027_143022.json`
+
+Each alert file contains full change details as JSON.
+
+**Example Alert File:**
+```json
+{
+  "has_changes": true,
+  "timestamp": "2025-10-27T14:30:22-03:00",
+  "search_type": "global",
+  "search_term": "commander deck",
+  "new_products": [...],
+  "removed_products": [...],
+  "price_changes": [...],
+  "quantity_changes": [...],
+  "availability_changes": [...]
+}
+```
+
+## Configuration
+
+The scraper uses:
+- **Driver**: Selenium Chrome (headed mode by default, headless mode available)
+- **Browser Modes**: 
+  - `headed` (default): Visible browser window for monitoring
+  - `headless`: No UI, ideal for servers and automation
+- **Wait Time**: 10 seconds for dynamic content
+- **Base URL**: `https://www.ligamagic.com.br/?view=cards%2Fsearch&tipo=1`
+- **Max Pagination**: 50 clicks maximum (configurable via `MAX_CLICKS` constant)
+- **Output Directory**: `scrapped/` (created automatically)
+
+## Development
+
+After checking out the repo, run `bundle install` to install dependencies.
+
+### Manual Build and Install
+
+To build and install the gem manually:
+
+```bash
+gem build ligamagic-scraper.gemspec
+gem install ./ligamagic-scraper-0.1.0.gem
+```
+
+To uninstall:
+
+```bash
+gem uninstall ligamagic-scraper
+```
+
+## Version Management & Releases
+
+The gem includes automated Rake tasks for version management following [Semantic Versioning](https://semver.org/).
+
+### Quick Release (Recommended)
+
+The easiest way to release a new version is using the automated release tasks:
+
+```bash
+# For bug fixes (0.1.0 -> 0.1.1)
+rake release_patch
+
+# For new features (0.1.0 -> 0.2.0)
+rake release_minor
+
+# For breaking changes (0.1.0 -> 1.0.0)
+rake release_major
+```
+
+These commands will automatically:
+1. Bump the version number
+2. Clean old gem files
+3. Build the new gem
+4. Install it locally
+
+### Check Current Version
+
+```bash
+rake version
+# Output: Current version: 0.1.0
+```
+
+### Manual Version Bumping
+
+If you just want to bump the version without building:
+
+```bash
+# Bump patch version (0.1.0 -> 0.1.1)
+rake bump_patch
+
+# Bump minor version (0.1.0 -> 0.2.0)
+rake bump_minor
+
+# Bump major version (0.1.0 -> 1.0.0)
+rake bump_major
+```
+
+### Build & Install Tasks
+
+```bash
+# Build and install locally (recommended)
+rake install_local
+
+# Just build (without installing)
+rake build
+
+# Clean old gem files
+rake clean
+
+# Uninstall the gem
+rake uninstall
+```
+
+### Semantic Versioning Guide
+
+- **Patch** (X.X.1): Bug fixes, small tweaks, no new features
+- **Minor** (X.1.0): New features, backward compatible changes
+- **Major** (1.0.0): Breaking changes, major redesigns
+
+### Example Workflow
+
+```bash
+# 1. Make your code changes
+# 2. Update CHANGELOG.md with your changes
+# 3. Release the new version
+rake release_patch
+
+# Output:
+# 📈 Version bumped: 0.1.0 -> 0.1.1
+# 🧹 Cleaning old gem files...
+# 🔨 Building gem...
+# 📦 Installing ligamagic-scraper-0.1.1.gem...
+# ✅ Gem installed successfully!
+```
+
+### All Available Rake Tasks
+
+To see all available tasks:
+
+```bash
+rake -T
+```
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](LICENSE).
+
+## Contributing
+
+Bug reports and pull requests are welcome!