source_monitor 0.3.0 → 0.3.2

data/.claude/skills/sm-health-rule/reference/health-system.md

@@ -0,0 +1,269 @@
+# Health System Reference
+
+## System Registration
+
+The health system registers itself during engine initialization via `Health.setup!`:
+
+```ruby
+module SourceMonitor
+  module Health
+    module_function
+
+    def setup!
+      register_fetch_callback
+    end
+
+    def fetch_callback
+      @fetch_callback ||= lambda do |event|
+        source = event&.source
+        next unless source
+        SourceHealthMonitor.new(source: source).call
+      rescue StandardError => error
+        log_error(source, error)
+      end
+    end
+
+    def register_fetch_callback
+      callbacks = SourceMonitor.config.events.callbacks_for(:after_fetch_completed)
+      return if callbacks.include?(fetch_callback)
+      SourceMonitor.config.events.after_fetch_completed(fetch_callback)
+    end
+  end
+end
+```
+
+Key behaviors:
+- Uses a memoized lambda to prevent duplicate registration
+- Checks existing callbacks before adding
+- Errors in the health callback are logged but don't crash the fetch pipeline
+
+## SourceHealthMonitor Details
+
+### Rolling Success Rate Calculation
+
+```ruby
+def calculate_success_rate(logs)
+  successes = logs.count { |log| log.success? }
+  total = logs.size
+  return 0.0 if total.zero?
+  (successes.to_f / total).round(4)
+end
+```
+
+The rate is stored as a float (0.0 to 1.0) on `source.rolling_success_rate`.
+
+### Minimum Sample Size
+
+Thresholds only apply when the number of logs equals the configured `window_size`. This prevents premature auto-pausing of new sources:
+
+```ruby
+def thresholds_applicable?(sample_size)
+  sample_size >= minimum_sample_size
+end
+
+def minimum_sample_size
+  [config.window_size.to_i, 1].max
+end
+```
+
+### Auto-Pause Algorithm
+
+```
+IF rate < auto_pause_threshold AND thresholds_active:
+  new_until = now + auto_pause_cooldown_minutes
+  IF existing_until > new_until:
+    keep existing_until (don't shorten pause)
+  ELSE:
+    use new_until
+
+  SET auto_paused_until = new_until
+  SET auto_paused_at = now (or keep existing)
+  PUSH next_fetch_at past pause window
+  PUSH backoff_until past pause window
+```
+
+### Auto-Resume Algorithm
+
+```
+IF auto_paused_until IS NOT NULL AND rate >= auto_resume_threshold:
+  CLEAR auto_paused_until
+  CLEAR auto_paused_at
+  CLEAR backoff_until
+```
+
+The resume threshold defaults to `max(auto_resume_threshold, auto_pause_threshold)` to prevent flapping.
+
+### Consecutive Failure Detection
+
+```ruby
+def consecutive_failures(logs)
+  logs.take_while { |log| !log_success?(log) }.size
+end
+```
+
+Logs are ordered by `started_at DESC`, so `take_while` counts the most recent consecutive failures.
+
+### Improving Streak Detection
+
+```ruby
+def improving_streak?(logs)
+  success_streak = 0
+  failure_seen = false
+  logs.each do |log|
+    if log_success?(log)
+      success_streak += 1
+    else
+      failure_seen = true
+      break
+    end
+  end
+  success_streak >= 2 && failure_seen
+end
+```
+
+A source is "improving" when it has >= 2 consecutive recent successes AND at least one failure exists in the window.
+
+### Fixed Interval Enforcement
+
+For non-adaptive sources, the monitor ensures `backoff_until` doesn't persist beyond what's needed:
+
+```ruby
+def enforce_fixed_interval(attrs, auto_paused_until)
+  return if source.adaptive_fetching_enabled?
+  return if auto_paused_active?(auto_paused_until)
+
+  backoff_value = attrs.key?(:backoff_until) ? attrs[:backoff_until] : source.backoff_until
+  return if backoff_value.blank?
+
+  fixed_minutes = [source.fetch_interval_minutes.to_i, 1].max
+  attrs[:next_fetch_at] = now + fixed_minutes.minutes
+  attrs[:backoff_until] = nil
+end
+```
+
+## Circuit Breaker vs Auto-Pause
+
+These are two separate protection mechanisms:
+
+| Feature | Circuit Breaker | Auto-Pause |
+|---------|----------------|------------|
+| **Scope** | Single fetch attempt | Rolling window |
+| **Trigger** | RetryPolicy exhaustion | Success rate threshold |
+| **Duration** | Error-type specific (1-2 hours) | Configurable cooldown (60 min default) |
+| **Fields** | `fetch_circuit_*` | `auto_paused_*` |
+| **Managed by** | RetryPolicy + SourceUpdater | SourceHealthMonitor |
+| **Resets on** | Successful fetch | Success rate recovery |
+
+### Circuit Breaker Flow
+
+```
+Error occurs -> RetryPolicy.decision
+  -> retry? (attempts remaining)
+     -> schedule retry with wait
+  -> open_circuit? (attempts exhausted)
+     -> set fetch_circuit_until
+     -> FetchRunner skips fetch while circuit open
+```
+
+### Auto-Pause Flow
+
+```
+After every fetch -> SourceHealthMonitor.call
+  -> calculate rolling success rate
+  -> IF rate < threshold AND window full
+     -> set auto_paused_until
+     -> push next_fetch_at past pause window
+```
+
+## HealthCheckLog Model
+
+```ruby
+class HealthCheckLog < ApplicationRecord
+  include SourceMonitor::Loggable
+
+  belongs_to :source
+  has_one :log_entry, as: :loggable, dependent: :destroy
+
+  attribute :http_response_headers, default: -> { {} }
+  validates :source, presence: true
+
+  after_save :sync_log_entry  # Creates unified LogEntry record
+end
+```
+
+Fields:
+- `source_id` -- associated source
+- `success` -- boolean
+- `started_at`, `completed_at` -- timing
+- `duration_ms` -- request duration
+- `http_status` -- response status code
+- `http_response_headers` -- response headers hash
+- `error_class`, `error_message` -- error details
+
+## Source Model Health Fields
+
+From migration `20251012090000_add_health_fields_to_sources`:
+
+```ruby
+# Health status tracking
+:health_status            # string, default: "healthy"
+:health_status_changed_at # datetime
+:rolling_success_rate     # float
+:health_auto_pause_threshold # float (per-source override)
+
+# Auto-pause state
+:auto_paused_at           # datetime
+:auto_paused_until        # datetime
+
+# Circuit breaker state (fetch-level)
+:fetch_retry_attempt      # integer, default: 0
+:fetch_circuit_opened_at  # datetime
+:fetch_circuit_until      # datetime
+
+# Existing fields used by health system
+:failure_count            # integer
+:last_error               # string
+:last_error_at            # datetime
+:backoff_until            # datetime
+:next_fetch_at            # datetime
+:fetch_status             # string
+```
+
+## Source Model Health Methods
+
+```ruby
+def fetch_circuit_open?
+  fetch_circuit_until.present? && fetch_circuit_until.future?
+end
+
+def auto_paused?
+  auto_paused_until.present? && auto_paused_until.future?
+end
+
+def fetch_retry_attempt
+  value = super
+  value.present? ? value : 0
+end
+```
+
+## Health Check Controllers
+
+| Controller | Actions | Purpose |
+|------------|---------|---------|
+| `SourceHealthChecksController` | `create` | Trigger on-demand health check |
+| `SourceHealthResetsController` | `create` | Reset all health state |
+| `HealthController` | `show` | Engine health endpoint |
+
+## Import Source Health Check
+
+Used during OPML import to validate feed URLs before importing:
+
+```ruby
+result = Health::ImportSourceHealthCheck.new(feed_url: "https://example.com/feed.xml").call
+```
+
+- No Source record needed (works with raw URL)
+- No conditional headers (no prior state)
+- Returns simple Result: `status`, `error_message`, `http_status`
+- Used by `ImportSessionHealthCheckJob` for batch validation
+- Results stored in `import_session.parsed_sources[n]["health_status"]`

data/.claude/skills/sm-host-setup/SKILL.md

@@ -0,0 +1,223 @@
+---
+name: sm-host-setup
+description: Use when setting up SourceMonitor in a host Rails application, including gem installation, engine mounting, migration copying, initializer creation, and verifying the install works.
+allowed-tools: Read, Write, Edit, Bash, Glob, Grep
+---
+
+# sm-host-setup: Host Application Setup
+
+Guides integration of the SourceMonitor engine into a host Rails application.
+
+## When to Use
+
+- Adding SourceMonitor to a new or existing Rails 8 host app
+- Troubleshooting a broken installation
+- Re-running setup after upgrading the gem
+- Rolling back the engine from a host app
+
+## Prerequisites
+
+| Requirement | Minimum | How to Check |
+|---|---|---|
+| Ruby | 3.4+ | `ruby -v` |
+| Rails | 8.0+ | `bin/rails about` |
+| PostgreSQL | 14+ | `psql --version` |
+| Node.js | 18+ | `node -v` |
+| Solid Queue | >= 0.3, < 3.0 | Check host Gemfile |
+| Solid Cable or Redis | Solid Cable >= 3.0 | Check host Gemfile |
+
+## Installation Methods
+
+### Method 1: Guided CLI (Recommended)
+
+The engine ships a guided installer that handles every step interactively.
+
+```bash
+# From the host app root:
+bundle add source_monitor --version "~> 0.3.0"
+bundle install
+bin/source_monitor install
+```
+
+The guided workflow:
+1. Checks prerequisites via `DependencyChecker`
+2. Prompts for mount path (default: `/source_monitor`)
+3. Ensures `gem "source_monitor"` in Gemfile, runs `bundle install`
+4. Runs `npm install` if `package.json` exists
+5. Executes `bin/rails generate source_monitor:install --mount-path=...`
+6. Copies and deduplicates migrations, runs `bin/rails db:migrate`
+7. Patches the initializer with navigation hint and optional Devise hooks
+8. Runs verification and prints a report
+
+Non-interactive mode:
+```bash
+bin/source_monitor install --yes
+```
+
+### Method 2: Manual Step-by-Step
+
+```bash
+# 1. Add the gem
+gem "source_monitor", "~> 0.3.0"  # in Gemfile
+bundle install
+
+# 2. Run the install generator
+bin/rails generate source_monitor:install --mount-path=/source_monitor
+
+# 3. Copy engine migrations
+bin/rails railties:install:migrations FROM=source_monitor
+
+# 4. Apply migrations
+bin/rails db:migrate
+
+# 5. Start background workers
+bin/rails solid_queue:start
+
+# 6. Verify
+bin/source_monitor verify
+```
+
+### Method 3: GitHub Edge
+
+```ruby
+# Gemfile
+gem "source_monitor", github: "dchuk/source_monitor"
+```
+
+## What the Install Generator Does
+
+The generator (`SourceMonitor::Generators::InstallGenerator`) performs two actions:
+
+1. **Mounts the engine** in `config/routes.rb`:
+   ```ruby
+   mount SourceMonitor::Engine, at: "/source_monitor"
+   ```
+   Skips if already mounted. The mount path is configurable via `--mount-path`.
+
+2. **Creates the initializer** at `config/initializers/source_monitor.rb`:
+   Uses the template at `lib/generators/source_monitor/install/templates/source_monitor.rb.tt`. Skips if the file already exists.
+
+Re-running the generator is safe and idempotent.
+
+## Post-Install Configuration
+
+After installation, review and customize the initializer. Key areas:
+
+| Section | Purpose |
+|---|---|
+| Queue settings | Queue names, concurrency, namespace |
+| Authentication | `authenticate_with`, `authorize_with` hooks |
+| HTTP client | Timeouts, proxy, retries |
+| Fetching | Adaptive scheduling intervals and factors |
+| Health | Auto-pause/resume thresholds |
+| Scrapers | Register custom scraper adapters |
+| Retention | Item age limits and pruning strategy |
+| Events | Lifecycle callbacks for integration |
+| Models | Table prefix, concerns, validations |
+| Realtime | Action Cable adapter (Solid Cable/Redis) |
+
+See the `sm-configure` skill for full configuration reference.
+
+## Verification
+
+```bash
+# CLI verification
+bin/source_monitor verify
+
+# Rake task verification
+bin/rails source_monitor:setup:verify
+
+# Prerequisite check only
+bin/rails source_monitor:setup:check
+```
+
+Verification checks Solid Queue workers and Action Cable health. Non-zero exit on failure.
+
+Enable telemetry logging:
+```bash
+SOURCE_MONITOR_SETUP_TELEMETRY=true bin/source_monitor verify
+# Logs to log/source_monitor_setup.log
+```
+
+## Devise Integration
+
+When Devise is detected, the guided installer offers to wire authentication hooks:
+
+```ruby
+config.authentication.authenticate_with :authenticate_user!
+config.authentication.authorize_with ->(controller) {
+  controller.current_user&.respond_to?(:admin?) ? controller.current_user.admin? : true
+}
+config.authentication.current_user_method = :current_user
+config.authentication.user_signed_in_method = :user_signed_in?
+```
+
+## Rollback
+
+1. Remove `gem "source_monitor"` from Gemfile, `bundle install`
+2. Delete `config/initializers/source_monitor.rb`
+3. Remove `mount SourceMonitor::Engine` from `config/routes.rb`
+4. Drop tables: `bin/rails db:migrate:down VERSION=<timestamp>` for each engine migration
+5. Remove Solid Queue/Cable migrations only if no other components need them
+
+## Host Compatibility
+
+| Scenario | Status |
+|---|---|
+| Rails 8 full-stack app | Supported |
+| Rails 8 API-only app | Supported |
+| Dedicated Solid Queue database | Supported |
+| Redis-backed Action Cable | Supported |
+
+## Key Source Files
+
+| File | Purpose |
+|---|---|
+| `lib/source_monitor/setup/workflow.rb` | Guided installer orchestration |
+| `lib/generators/source_monitor/install/install_generator.rb` | Rails generator |
+| `lib/generators/source_monitor/install/templates/source_monitor.rb.tt` | Initializer template |
+| `lib/source_monitor/setup/initializer_patcher.rb` | Post-install patching |
+| `lib/source_monitor/setup/verification/runner.rb` | Verification runner |
+| `lib/source_monitor/engine.rb` | Engine configuration and initializers |
+| `docs/setup.md` | Full setup documentation |
+| `docs/troubleshooting.md` | Common fixes |
+
+## References
+
+- `docs/setup.md` -- Complete setup workflow documentation
+- `docs/configuration.md` -- Configuration reference
+- `docs/troubleshooting.md` -- Common issues and fixes
+
+## Testing
+
+After setup, verify with:
+1. `bin/source_monitor verify` -- Checks Solid Queue and Action Cable
+2. Visit the mount path in browser -- Dashboard should load
+3. Create a source and trigger "Fetch Now" -- Validates end-to-end
+
+Optional system test for host apps using Devise:
+```ruby
+# test/system/source_monitor_setup_test.rb
+require "application_system_test_case"
+
+class SourceMonitorSetupTest < ApplicationSystemTestCase
+  test "signed in admin can reach SourceMonitor" do
+    user = users(:admin)
+    sign_in user
+    visit "/source_monitor"
+    assert_text "SourceMonitor Dashboard"
+  end
+end
+```
+
+## Checklist
+
+- [ ] Ruby 3.4+, Rails 8.0+, PostgreSQL 14+ available
+- [ ] `gem "source_monitor"` added to Gemfile
+- [ ] `bundle install` completed
+- [ ] Install generator ran (`bin/rails generate source_monitor:install`)
+- [ ] Engine migrations copied and applied
+- [ ] Solid Queue workers started
+- [ ] Authentication hooks configured in initializer
+- [ ] `bin/source_monitor verify` passes
+- [ ] Dashboard accessible at mount path

data/.claude/skills/sm-host-setup/reference/initializer-template.md

@@ -0,0 +1,195 @@
+# Initializer Template Reference
+
+Complete annotated initializer for `config/initializers/source_monitor.rb`.
+
+The install generator creates this file from `lib/generators/source_monitor/install/templates/source_monitor.rb.tt`. This reference expands on every option.
+
+## Full Template
+
+```ruby
+# frozen_string_literal: true
+
+# SourceMonitor engine configuration.
+#
+# These values default to conservative settings that work for most hosts.
+# Tweak them here instead of monkey-patching the engine so upgrades remain easy.
+# Restart the application after changes.
+SourceMonitor.configure do |config|
+
+  # ===========================================================================
+  # Queue & Worker Settings
+  # ===========================================================================
+
+  # Namespace prefix for queue names and instrumentation keys.
+  # Combined with ActiveJob.queue_name_prefix automatically.
+  config.queue_namespace = "source_monitor"
+
+  # Dedicated queue names. Must match entries in config/solid_queue.yml.
+  config.fetch_queue_name = "source_monitor_fetch"
+  config.scrape_queue_name = "source_monitor_scrape"
+
+  # Worker concurrency per queue (advisory for Solid Queue).
+  config.fetch_queue_concurrency = 2
+  config.scrape_queue_concurrency = 2
+
+  # Override the job class Solid Queue uses for recurring "command" tasks.
+  # config.recurring_command_job_class = "MyRecurringCommandJob"
+
+  # Toggle lightweight queue metrics on the dashboard.
+  config.job_metrics_enabled = true
+
+  # ===========================================================================
+  # Mission Control Integration
+  # ===========================================================================
+
+  # Show "Open Mission Control" link on the SourceMonitor dashboard.
+  config.mission_control_enabled = false
+
+  # String path, route helper lambda, or nil.
+  # config.mission_control_dashboard_path = "/mission_control"
+  # config.mission_control_dashboard_path = -> {
+  #   Rails.application.routes.url_helpers.mission_control_jobs_path
+  # }
+  config.mission_control_dashboard_path = nil
+
+  # ===========================================================================
+  # Authentication
+  # ===========================================================================
+  # Handlers: Symbol (invoked on controller) or callable (receives controller).
+
+  # Authenticate before accessing any SourceMonitor page.
+  # config.authentication.authenticate_with :authenticate_user!
+
+  # Authorize after authentication (return false or raise to deny).
+  # config.authentication.authorize_with ->(controller) {
+  #   controller.current_user&.admin?
+  # }
+
+  # Method names SourceMonitor uses to access current user info.
+  # config.authentication.current_user_method = :current_user
+  # config.authentication.user_signed_in_method = :user_signed_in?
+
+  # ===========================================================================
+  # HTTP Client
+  # ===========================================================================
+  # Maps to Faraday middleware options.
+
+  config.http.timeout = 15          # Total request timeout (seconds)
+  config.http.open_timeout = 5      # Connection open timeout (seconds)
+  config.http.max_redirects = 5     # Max redirects to follow
+  # config.http.user_agent = "SourceMonitor/#{SourceMonitor::VERSION}"
+  # config.http.proxy = ENV["SOURCE_MONITOR_HTTP_PROXY"]
+  # config.http.headers = { "X-Request-ID" => -> { SecureRandom.uuid } }
+
+  # Retry settings (mapped to faraday-retry)
+  # config.http.retry_max = 4
+  # config.http.retry_interval = 0.5
+  # config.http.retry_interval_randomness = 0.5
+  # config.http.retry_backoff_factor = 2
+  # config.http.retry_statuses = [429, 500, 502, 503, 504]
+
+  # ===========================================================================
+  # Adaptive Fetch Scheduling
+  # ===========================================================================
+
+  # config.fetching.min_interval_minutes = 5       # Floor (default: 5 min)
+  # config.fetching.max_interval_minutes = 1440    # Ceiling (default: 24 hrs)
+  # config.fetching.increase_factor = 1.25         # Multiplier when no new items
+  # config.fetching.decrease_factor = 0.75         # Multiplier when items arrive
+  # config.fetching.failure_increase_factor = 1.5  # Multiplier on errors
+  # config.fetching.jitter_percent = 0.1           # Random jitter (+/-10%)
+
+  # ===========================================================================
+  # Source Health Monitoring
+  # ===========================================================================
+
+  config.health.window_size = 20                   # Fetch attempts to evaluate
+  config.health.healthy_threshold = 0.8            # Ratio for "healthy" badge
+  config.health.warning_threshold = 0.5            # Ratio for "warning" badge
+  config.health.auto_pause_threshold = 0.2         # Auto-pause below this
+  config.health.auto_resume_threshold = 0.6        # Auto-resume above this
+  config.health.auto_pause_cooldown_minutes = 60   # Grace period before re-enable
+
+  # ===========================================================================
+  # Scraper Adapters
+  # ===========================================================================
+  # Must inherit from SourceMonitor::Scrapers::Base.
+
+  # config.scrapers.register(:custom, "MyApp::Scrapers::CustomAdapter")
+  # config.scrapers.unregister(:readability)  # Remove built-in
+
+  # ===========================================================================
+  # Retention Defaults
+  # ===========================================================================
+  # Sources inherit these when their retention fields are blank.
+
+  config.retention.items_retention_days = nil   # nil = retain forever
+  config.retention.max_items = nil              # nil = unlimited
+  # config.retention.strategy = :destroy        # or :soft_delete
+
+  # ===========================================================================
+  # Scraping Controls
+  # ===========================================================================
+
+  # config.scraping.max_in_flight_per_source = 25  # Concurrent scrapes per source
+  # config.scraping.max_bulk_batch_size = 100       # Max bulk enqueue size
+
+  # ===========================================================================
+  # Event Callbacks
+  # ===========================================================================
+  # Handlers receive a single event struct.
+
+  # config.events.after_item_created do |event|
+  #   # event.item, event.source, event.entry, event.result, event.status
+  #   NewItemNotifier.publish(event.item, source: event.source)
+  # end
+
+  # config.events.after_item_scraped do |event|
+  #   # event.item, event.source, event.result, event.log, event.status
+  #   SearchIndexer.reindex(event.item) if event.success?
+  # end
+
+  # config.events.after_fetch_completed do |event|
+  #   # event.source, event.result, event.status
+  #   Rails.logger.info "Fetch for #{event.source.name}: #{event.status}"
+  # end
+
+  # config.events.register_item_processor ->(context) {
+  #   # context.item, context.source, context.entry, context.result, context.status
+  #   ItemIndexer.index(context.item)
+  # }
+
+  # ===========================================================================
+  # Model Extensions
+  # ===========================================================================
+
+  # Override default table name prefix (default: "sourcemon_").
+  # config.models.table_name_prefix = "sourcemon_"
+
+  # Include concerns to extend engine models.
+  # config.models.source.include_concern "MyApp::SourceMonitor::SourceExtensions"
+  # config.models.item.include_concern "MyApp::SourceMonitor::ItemExtensions"
+
+  # Register custom validations.
+  # config.models.source.validate :enforce_custom_rules
+  # config.models.source.validate ->(record) {
+  #   record.errors.add(:base, "custom error") unless record.valid_for_my_app?
+  # }
+
+  # ===========================================================================
+  # Realtime (Action Cable) Adapter
+  # ===========================================================================
+
+  config.realtime.adapter = :solid_cable
+  # config.realtime.adapter = :redis
+  # config.realtime.redis_url = ENV.fetch("SOURCE_MONITOR_REDIS_URL", nil)
+
+  # Solid Cable tuning:
+  # config.realtime.solid_cable.polling_interval = "0.1.seconds"
+  # config.realtime.solid_cable.message_retention = "1.day"
+  # config.realtime.solid_cable.autotrim = true
+  # config.realtime.solid_cable.silence_polling = true
+  # config.realtime.solid_cable.use_skip_locked = true
+  # config.realtime.solid_cable.connects_to = { database: { writing: :cable } }
+end
+```