source_monitor 0.3.0 → 0.3.2

data/.claude/skills/sm-host-setup/reference/setup-checklist.md

@@ -0,0 +1,134 @@
+# Host App Setup Checklist
+
+Step-by-step checklist for integrating SourceMonitor into a host Rails application.
+
+## Phase 1: Prerequisites
+
+- [ ] **Ruby 3.4+** installed (`ruby -v`)
+- [ ] **Rails 8.0+** host application (`bin/rails about`)
+- [ ] **PostgreSQL 14+** running and accessible
+- [ ] **Node.js 18+** installed (for Tailwind/esbuild assets)
+- [ ] **Solid Queue** in host Gemfile (`gem "solid_queue"`)
+- [ ] **Solid Cable** or Redis available for Action Cable
+
+## Phase 2: Install the Gem
+
+```bash
+# Option A: Released version
+bundle add source_monitor --version "~> 0.3.0"
+
+# Option B: GitHub edge
+# Add to Gemfile: gem "source_monitor", github: "dchuk/source_monitor"
+
+bundle install
+```
+
+- [ ] Gem added to Gemfile
+- [ ] `bundle install` succeeds
+
+## Phase 3: Run the Generator
+
+```bash
+# Guided (recommended)
+bin/source_monitor install
+
+# Manual
+bin/rails generate source_monitor:install --mount-path=/source_monitor
+```
+
+Verify after running:
+- [ ] `config/routes.rb` contains `mount SourceMonitor::Engine, at: "/source_monitor"`
+- [ ] `config/initializers/source_monitor.rb` exists
+
+## Phase 4: Database Setup
+
+```bash
+# Copy engine migrations to host
+bin/rails railties:install:migrations FROM=source_monitor
+
+# Apply all pending migrations
+bin/rails db:migrate
+```
+
+- [ ] Engine migrations copied (check `db/migrate/` for `sourcemon_*` tables)
+- [ ] `bin/rails db:migrate` succeeds
+- [ ] Tables created: `sourcemon_sources`, `sourcemon_items`, `sourcemon_fetch_logs`, etc.
+
+## Phase 5: Configure Authentication
+
+Edit `config/initializers/source_monitor.rb`:
+
+```ruby
+SourceMonitor.configure do |config|
+  # Devise example
+  config.authentication.authenticate_with :authenticate_user!
+  config.authentication.authorize_with ->(controller) {
+    controller.current_user&.admin?
+  }
+  config.authentication.current_user_method = :current_user
+  config.authentication.user_signed_in_method = :user_signed_in?
+end
+```
+
+- [ ] Authentication hook configured
+- [ ] Authorization hook configured (if needed)
+
+## Phase 6: Configure Workers
+
+Ensure `config/solid_queue.yml` (or equivalent) includes the SourceMonitor queues:
+
+```yaml
+# config/solid_queue.yml
+production:
+  dispatchers:
+    - polling_interval: 1
+      batch_size: 500
+  workers:
+    - queues: "source_monitor_fetch"
+      threads: 2
+      processes: 1
+    - queues: "source_monitor_scrape"
+      threads: 2
+      processes: 1
+```
+
+```bash
+bin/rails solid_queue:start
+```
+
+- [ ] Queue configuration includes `source_monitor_fetch` and `source_monitor_scrape`
+- [ ] Workers started and processing
+
+## Phase 7: Verify Installation
+
+```bash
+bin/source_monitor verify
+```
+
+- [ ] Verification passes (exit code 0)
+- [ ] Dashboard loads at mount path (e.g., `http://localhost:3000/source_monitor`)
+- [ ] Create a test source, trigger "Fetch Now", confirm items appear
+
+## Phase 8: Optional Configuration
+
+- [ ] HTTP client tuned (timeouts, proxy, retries)
+- [ ] Fetching intervals configured for your workload
+- [ ] Health thresholds adjusted
+- [ ] Retention policy set
+- [ ] Custom scraper adapters registered
+- [ ] Event callbacks wired for host integration
+- [ ] Realtime adapter confirmed (Solid Cable or Redis)
+- [ ] Mission Control integration enabled (if desired)
+
+## Troubleshooting
+
+| Problem | Solution |
+|---|---|
+| `bin/source_monitor` not found | Run `bundle install`, ensure gem is loaded |
+| Migrations fail | Check for duplicate Solid Queue migrations, remove dupes |
+| Dashboard 404 | Verify mount in `config/routes.rb`, restart server |
+| Jobs not processing | Start Solid Queue workers, check queue names match config |
+| Action Cable errors | Verify Solid Cable or Redis is configured in `cable.yml` |
+| Auth redirect loop | Check `authenticate_with` matches your auth system |
+
+See `docs/troubleshooting.md` for comprehensive fixes.

data/.claude/skills/sm-job/SKILL.md

@@ -0,0 +1,263 @@
+---
+name: sm-job
+description: Solid Queue job conventions for the SourceMonitor engine. Use when creating new background jobs, modifying existing jobs, configuring queues, or working with job scheduling and retry policies.
+allowed-tools: Read, Write, Edit, Bash, Glob, Grep
+---
+
+# SourceMonitor Job Development
+
+## Overview
+
+SourceMonitor uses Solid Queue (Rails 8 default) for background processing. All jobs inherit from `SourceMonitor::ApplicationJob` and use engine-namespaced queues.
+
+## Queue Architecture
+
+| Queue Role | Default Name | Jobs |
+|------------|-------------|------|
+| `:fetch` | `source_monitor_fetch` | FetchFeedJob, ScheduleFetchesJob, ItemCleanupJob, LogCleanupJob, SourceHealthCheckJob, ImportOpmlJob, ImportSessionHealthCheckJob |
+| `:scrape` | `source_monitor_scrape` | ScrapeItemJob |
+
+Queue names respect the host app's `ActiveJob::Base.queue_name_prefix` and `queue_name_delimiter`.
+
+## Existing Jobs
+
+| Job | Queue | Purpose | Pattern |
+|-----|-------|---------|---------|
+| `FetchFeedJob` | `:fetch` | Fetches a single source's feed | Delegates to `FetchRunner` |
+| `ScheduleFetchesJob` | `:fetch` | Batch-enqueues due fetches | Delegates to `Scheduler.run` |
+| `ScrapeItemJob` | `:scrape` | Scrapes a single item's URL | Delegates to `Scraping::ItemScraper` |
+| `ItemCleanupJob` | `:fetch` | Prunes items by retention policy | Delegates to `RetentionPruner` |
+| `LogCleanupJob` | `:fetch` | Removes old fetch/scrape logs | Direct SQL batches |
+| `SourceHealthCheckJob` | `:fetch` | Runs health check on a source | Delegates to `Health::SourceHealthCheck` |
+| `ImportOpmlJob` | `:fetch` | Imports sources from OPML | Delegates to source creation |
+| `ImportSessionHealthCheckJob` | `:fetch` | Health-checks import candidates | Delegates to `Health::ImportSourceHealthCheck` |
+
+## Key Conventions
+
+### 1. Shallow Jobs
+
+Jobs contain **only** deserialization + delegation. No business logic lives in job classes.
+
+```ruby
+# CORRECT -- shallow delegation
+def perform(source_id)
+  source = SourceMonitor::Source.find_by(id: source_id)
+  return unless source
+  SourceMonitor::Fetching::FetchRunner.new(source: source).run
+end
+
+# WRONG -- business logic in job
+def perform(source_id)
+  source = SourceMonitor::Source.find(source_id)
+  response = Faraday.get(source.feed_url)  # Don't do this
+  feed = Feedjira.parse(response.body)      # Business logic belongs elsewhere
+end
+```
+
+### 2. Queue Declaration
+
+Use the `source_monitor_queue` class method (not `queue_as`):
+
+```ruby
+class MyJob < SourceMonitor::ApplicationJob
+  source_monitor_queue :fetch  # Uses SourceMonitor.queue_name(:fetch)
+end
+```
+
+This ensures the queue name respects engine configuration and host app prefixes.
+
+### 3. ID-Based Arguments
+
+Pass record IDs, not Active Record objects. Guard against missing records:
+
+```ruby
+def perform(source_id)
+  source = SourceMonitor::Source.find_by(id: source_id)
+  return unless source  # Silently skip if deleted
+  # ...
+end
+```
+
+### 4. Error Handling
+
+Use ActiveJob's built-in error handling:
+
+```ruby
+discard_on ActiveJob::DeserializationError  # Record deleted between enqueue and perform
+retry_on SomeTransientError, wait: 30.seconds, attempts: 5
+```
+
+### 5. Logging Pattern
+
+Use structured logging with a consistent format:
+
+```ruby
+def log(stage, **extra)
+  return unless defined?(Rails) && Rails.respond_to?(:logger) && Rails.logger
+  payload = { stage: "SourceMonitor::MyJob##{stage}", **extra }.compact
+  Rails.logger.info("[SourceMonitor::MyJob] #{payload.to_json}")
+rescue StandardError
+  nil
+end
+```
+
+## Creating a New Job
+
+### Template
+
+```ruby
+# app/jobs/source_monitor/my_new_job.rb
+# frozen_string_literal: true
+
+module SourceMonitor
+  class MyNewJob < ApplicationJob
+    source_monitor_queue :fetch  # or :scrape
+
+    discard_on ActiveJob::DeserializationError
+
+    def perform(record_id)
+      record = SourceMonitor::Source.find_by(id: record_id)
+      return unless record
+
+      # Delegate to a service/model method
+      SourceMonitor::MyService.new(record: record).call
+    end
+  end
+end
+```
+
+### Steps
+
+1. Create file at `app/jobs/source_monitor/my_new_job.rb`
+2. Inherit from `SourceMonitor::ApplicationJob`
+3. Call `source_monitor_queue` with `:fetch` or `:scrape`
+4. Add `discard_on ActiveJob::DeserializationError`
+5. Accept IDs as arguments, guard with `find_by`
+6. Delegate to service/model -- no business logic in the job
+7. Write tests in `test/jobs/source_monitor/my_new_job_test.rb`
+
+## Queue Configuration
+
+### Engine Configuration
+
+```ruby
+SourceMonitor.configure do |config|
+  config.queue_namespace = "source_monitor"        # Base namespace
+  config.fetch_queue_name = "source_monitor_fetch"  # Fetch queue name
+  config.scrape_queue_name = "source_monitor_scrape" # Scrape queue name
+  config.fetch_queue_concurrency = 2                # Concurrent fetch workers
+  config.scrape_queue_concurrency = 2               # Concurrent scrape workers
+end
+```
+
+### Queue Name Resolution
+
+```ruby
+SourceMonitor.queue_name(:fetch)  # => "source_monitor_fetch"
+# With host app prefix "myapp":   => "myapp_source_monitor_fetch"
+```
+
+### Recurring Jobs
+
+ScheduleFetchesJob is typically configured as a recurring job in `config/recurring.yml`:
+
+```yaml
+production:
+  schedule_fetches:
+    class: SourceMonitor::ScheduleFetchesJob
+    schedule: every 1 minute
+    queue: source_monitor_fetch
+```
+
+## Retry Policies
+
+FetchFeedJob uses a custom retry strategy via `RetryPolicy`:
+
+| Error Type | Retry Attempts | Wait | Circuit Breaker |
+|------------|---------------|------|-----------------|
+| Timeout | 2 | 2 min | 1 hour |
+| Connection | 3 | 5 min | 1 hour |
+| HTTP 429 | 2 | 15 min | 90 min |
+| HTTP 5xx | 2 | 10 min | 90 min |
+| HTTP 4xx | 1 | 45 min | 2 hours |
+| Parsing | 1 | 30 min | 2 hours |
+| Unexpected | 1 | 30 min | 2 hours |
+
+## CleanupOptions Helper
+
+`SourceMonitor::Jobs::CleanupOptions` normalizes job arguments for cleanup jobs:
+
+```ruby
+options = CleanupOptions.normalize(options)  # Symbolize keys, handle nil
+now = CleanupOptions.resolve_time(options[:now])  # Parse time strings
+ids = CleanupOptions.extract_ids(options[:source_ids])  # Flatten/parse IDs
+batch_size = CleanupOptions.batch_size(options, default: 100)  # Safe integer
+```
+
+## Testing
+
+### Test Template
+
+```ruby
+# test/jobs/source_monitor/my_new_job_test.rb
+# frozen_string_literal: true
+
+require "test_helper"
+
+module SourceMonitor
+  class MyNewJobTest < ActiveJob::TestCase
+    setup do
+      @source = create_source!
+    end
+
+    test "performs work for valid source" do
+      # Stub external calls
+      MyService.any_instance.expects(:call).once
+
+      MyNewJob.perform_now(@source.id)
+    end
+
+    test "silently skips missing source" do
+      assert_nothing_raised do
+        MyNewJob.perform_now(-1)
+      end
+    end
+
+    test "enqueues on correct queue" do
+      assert_enqueued_with(job: MyNewJob, queue: SourceMonitor.queue_name(:fetch).to_s) do
+        MyNewJob.perform_later(@source.id)
+      end
+    end
+  end
+end
+```
+
+### Testing Enqueue from Models
+
+```ruby
+test "fetching enqueues via FetchRunner.enqueue" do
+  with_inline_jobs do
+    stub_request(:get, source.feed_url).to_return(status: 200, body: feed_xml)
+    SourceMonitor::Fetching::FetchRunner.enqueue(source)
+  end
+end
+```
+
+## Checklist
+
+- [ ] Job inherits from `SourceMonitor::ApplicationJob`
+- [ ] Uses `source_monitor_queue` (not `queue_as`)
+- [ ] Accepts IDs, not AR objects
+- [ ] Guards with `find_by` + early return
+- [ ] No business logic in the job class
+- [ ] `discard_on ActiveJob::DeserializationError`
+- [ ] Error handling with `retry_on` where appropriate
+- [ ] Test covers perform, missing record, and queue assignment
+- [ ] All tests GREEN
+
+## References
+
+- `app/jobs/source_monitor/` -- All engine jobs
+- `lib/source_monitor/jobs/` -- Job support classes (CleanupOptions, Visibility, SolidQueueMetrics)
+- `lib/source_monitor/configuration.rb` -- Queue configuration
+- `test/jobs/source_monitor/` -- Job tests

data/.claude/skills/sm-job/reference/job-conventions.md

@@ -0,0 +1,245 @@
+# Job Conventions Reference
+
+## ApplicationJob Base Class
+
+All engine jobs inherit from `SourceMonitor::ApplicationJob`:
+
+```ruby
+# app/jobs/source_monitor/application_job.rb
+module SourceMonitor
+  parent_job = defined?(::ApplicationJob) ? ::ApplicationJob : ActiveJob::Base
+
+  class ApplicationJob < parent_job
+    class << self
+      def source_monitor_queue(role)
+        queue_as SourceMonitor.queue_name(role)
+      end
+    end
+  end
+end
+```
+
+Key behaviors:
+- Inherits from host app's `ApplicationJob` if available, otherwise `ActiveJob::Base`
+- Provides `source_monitor_queue` class method for engine-aware queue naming
+- Host app job middleware (logging, error tracking) applies automatically
+
+## Queue Naming
+
+### Configuration Chain
+
+```
+SourceMonitor.queue_name(:fetch)
+  -> config.queue_name_for(:fetch)
+     -> config.fetch_queue_name  # "source_monitor_fetch"
+     -> prepend ActiveJob::Base.queue_name_prefix if set
+```
+
+### Default Names
+
+| Role | Queue Name |
+|------|-----------|
+| `:fetch` | `source_monitor_fetch` |
+| `:scrape` | `source_monitor_scrape` |
+
+### With Host App Prefix
+
+If the host app sets `ActiveJob::Base.queue_name_prefix = "myapp"`:
+- Fetch queue becomes `myapp_source_monitor_fetch`
+- Scrape queue becomes `myapp_source_monitor_scrape`
+
+## Job Patterns by Type
+
+### Fetch Job (FetchFeedJob)
+
+The most complex job, demonstrating retry strategy integration:
+
+```ruby
+class FetchFeedJob < ApplicationJob
+  FETCH_CONCURRENCY_RETRY_WAIT = 30.seconds
+  EARLY_EXECUTION_LEEWAY = 30.seconds
+
+  source_monitor_queue :fetch
+
+  discard_on ActiveJob::DeserializationError
+  retry_on FetchRunner::ConcurrencyError, wait: 30.seconds, attempts: 5
+
+  def perform(source_id, force: false)
+    source = Source.find_by(id: source_id)
+    return unless source
+    return unless should_run?(source, force: force)
+    FetchRunner.new(source: source, force: force).run
+  rescue FetchError => error
+    handle_transient_error(source, error)
+  end
+end
+```
+
+Notable patterns:
+- `should_run?` guard prevents premature execution
+- `ConcurrencyError` uses ActiveJob `retry_on` (another worker holds the lock)
+- `FetchError` uses custom retry logic via `RetryPolicy`
+- `force: false` keyword argument for manual vs scheduled fetches
+
+### Cleanup Job (ItemCleanupJob)
+
+Demonstrates options normalization pattern:
+
+```ruby
+class ItemCleanupJob < ApplicationJob
+  DEFAULT_BATCH_SIZE = 100
+  source_monitor_queue :fetch
+
+  def perform(options = nil)
+    options = Jobs::CleanupOptions.normalize(options)
+    scope = resolve_scope(options)
+    batch_size = Jobs::CleanupOptions.batch_size(options, default: DEFAULT_BATCH_SIZE)
+    now = Jobs::CleanupOptions.resolve_time(options[:now])
+    strategy = resolve_strategy(options)
+
+    scope.find_in_batches(batch_size:) do |batch|
+      batch.each { |source| RetentionPruner.call(source:, now:, strategy:) }
+    end
+  end
+end
+```
+
+Notable patterns:
+- Accepts flexible `options` hash (works with both manual and recurring invocation)
+- Uses `CleanupOptions` helper for safe normalization
+- Batched processing with configurable batch size
+
+### Worker Job (ScrapeItemJob)
+
+Demonstrates lifecycle logging:
+
+```ruby
+class ScrapeItemJob < ApplicationJob
+  source_monitor_queue :scrape
+  discard_on ActiveJob::DeserializationError
+
+  def perform(item_id)
+    log("job:start", item_id: item_id)
+    item = Item.includes(:source).find_by(id: item_id)
+    return unless item
+
+    source = item.source
+    unless source&.scraping_enabled?
+      log("job:skipped_scraping_disabled", item: item)
+      Scraping::State.clear_inflight!(item)
+      return
+    end
+
+    Scraping::State.mark_processing!(item)
+    Scraping::ItemScraper.new(item:, source:).call
+    log("job:completed", item: item, status: item.scrape_status)
+  rescue StandardError => error
+    log("job:error", item: item, error: error.message)
+    Scraping::State.mark_failed!(item)
+    raise
+  ensure
+    Scraping::State.clear_inflight!(item) if item
+  end
+end
+```
+
+Notable patterns:
+- `includes(:source)` prevents N+1 query
+- Lifecycle state management (`mark_processing!`, `clear_inflight!`)
+- Error re-raise after state cleanup
+- Structured JSON logging at each stage
+
+### Scheduling Job (ScheduleFetchesJob)
+
+Simplest pattern -- pure delegation:
+
+```ruby
+class ScheduleFetchesJob < ApplicationJob
+  source_monitor_queue :fetch
+
+  def perform(options = nil)
+    limit = extract_limit(options)
+    Scheduler.run(limit:)
+  end
+end
+```
+
+### Broadcast Job (SourceHealthCheckJob)
+
+Demonstrates result broadcasting:
+
+```ruby
+class SourceHealthCheckJob < ApplicationJob
+  source_monitor_queue :fetch
+  discard_on ActiveJob::DeserializationError
+
+  def perform(source_id)
+    source = Source.find_by(id: source_id)
+    return unless source
+
+    result = Health::SourceHealthCheck.new(source: source).call
+    broadcast_outcome(source, result)
+    result
+  rescue StandardError => error
+    record_unexpected_failure(source, error) if source
+    broadcast_outcome(source, nil, error) if source
+    nil
+  end
+end
+```
+
+Notable patterns:
+- Always broadcasts UI update (success or failure)
+- Creates log record even for unexpected failures
+- Returns nil on error instead of re-raising (health checks are non-critical)
+
+## _later / _now Naming Convention
+
+Models and services should expose `_later` methods for async work:
+
+```ruby
+# On the model or service
+def self.fetch_later(source_or_id, force: false)
+  FetchRunner.enqueue(source_or_id, force: force)
+end
+
+def self.fetch_now(source, force: false)
+  FetchRunner.run(source: source, force: force)
+end
+```
+
+Jobs are the mechanism, not the API. Callers should use model/service methods, not enqueue jobs directly.
+
+## Job Support Classes
+
+### CleanupOptions
+
+**File:** `lib/source_monitor/jobs/cleanup_options.rb`
+
+Normalizes job arguments for cleanup jobs:
+
+| Method | Purpose |
+|--------|---------|
+| `normalize(options)` | Symbolize keys, handle nil/non-Hash |
+| `resolve_time(value)` | Parse Time/String/nil to Time |
+| `extract_ids(value)` | Flatten arrays, split CSV, convert to integers |
+| `integer(value)` | Safe Integer conversion |
+| `batch_size(options, default:)` | Extract positive batch size |
+
+### FetchFailureSubscriber
+
+**File:** `lib/source_monitor/jobs/fetch_failure_subscriber.rb`
+
+Subscribes to Solid Queue failure events for fetch queue jobs. Used for metrics and alerting.
+
+### Visibility
+
+**File:** `lib/source_monitor/jobs/visibility.rb`
+
+Tracks queue depth and timing metrics per queue.
+
+### SolidQueueMetrics
+
+**File:** `lib/source_monitor/jobs/solid_queue_metrics.rb`
+
+Queries Solid Queue tables for dashboard metrics: pending count, failed count, paused queues, oldest job age.