source_monitor 0.2.1 → 0.3.1

data/.claude/skills/sm-configuration-setting/reference/settings-catalog.md

@@ -0,0 +1,248 @@
+# Configuration Settings Catalog
+
+All configuration sections with their attributes, defaults, and types.
+
+## Top-Level Attributes
+
+**File:** `lib/source_monitor/configuration.rb`
+
+| Attribute | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `queue_namespace` | String | `"source_monitor"` | Namespace prefix for queue names |
+| `fetch_queue_name` | String | `"source_monitor_fetch"` | Queue name for fetch jobs |
+| `scrape_queue_name` | String | `"source_monitor_scrape"` | Queue name for scrape jobs |
+| `fetch_queue_concurrency` | Integer | `2` | Max concurrent fetch workers |
+| `scrape_queue_concurrency` | Integer | `2` | Max concurrent scrape workers |
+| `recurring_command_job_class` | Class/nil | `nil` | Custom recurring job class |
+| `job_metrics_enabled` | Boolean | `true` | Enable job metrics tracking |
+| `mission_control_enabled` | Boolean | `false` | Enable Mission Control integration |
+| `mission_control_dashboard_path` | String/Proc/nil | `nil` | Path or callable for Mission Control |
+
+**Methods:**
+- `queue_name_for(:fetch)` / `queue_name_for(:scrape)` -- Returns prefixed queue name
+- `concurrency_for(:fetch)` / `concurrency_for(:scrape)` -- Returns concurrency limit
+
+---
+
+## HTTPSettings
+
+**File:** `lib/source_monitor/configuration/http_settings.rb`
+
+| Attribute | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `timeout` | Integer | `15` | Total request timeout (seconds) |
+| `open_timeout` | Integer | `5` | Connection open timeout (seconds) |
+| `max_redirects` | Integer | `5` | Max HTTP redirects to follow |
+| `user_agent` | String | `"SourceMonitor/<version>"` | User-Agent header value |
+| `proxy` | String/nil | `nil` | HTTP proxy URL |
+| `headers` | Hash | `{}` | Default HTTP headers |
+| `retry_max` | Integer | `4` | Max retry attempts |
+| `retry_interval` | Float | `0.5` | Base retry interval (seconds) |
+| `retry_interval_randomness` | Float | `0.5` | Retry interval randomness factor |
+| `retry_backoff_factor` | Integer | `2` | Exponential backoff multiplier |
+| `retry_statuses` | Array/nil | `nil` | HTTP statuses to retry on |
+
+Has `reset!` method.
+
+---
+
+## FetchingSettings
+
+**File:** `lib/source_monitor/configuration/fetching_settings.rb`
+
+| Attribute | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `min_interval_minutes` | Integer | `5` | Minimum fetch interval |
+| `max_interval_minutes` | Integer | `1440` (24h) | Maximum fetch interval |
+| `increase_factor` | Float | `1.25` | Multiplier when content unchanged |
+| `decrease_factor` | Float | `0.75` | Multiplier when content changed |
+| `failure_increase_factor` | Float | `1.5` | Multiplier on fetch failure |
+| `jitter_percent` | Float | `0.1` | Random jitter (10%) |
+
+Has `reset!` method. All attributes are plain `attr_accessor`.
+
+---
+
+## HealthSettings
+
+**File:** `lib/source_monitor/configuration/health_settings.rb`
+
+| Attribute | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `window_size` | Integer | `20` | Rolling window of fetches for health calc |
+| `healthy_threshold` | Float | `0.8` | Success rate above = healthy |
+| `warning_threshold` | Float | `0.5` | Success rate above = warning |
+| `auto_pause_threshold` | Float | `0.2` | Success rate below = auto-pause |
+| `auto_resume_threshold` | Float | `0.6` | Success rate above = auto-resume |
+| `auto_pause_cooldown_minutes` | Integer | `60` | Cooldown before auto-resume check |
+
+Has `reset!` method. All attributes are plain `attr_accessor`.
+
+---
+
+## ScrapingSettings
+
+**File:** `lib/source_monitor/configuration/scraping_settings.rb`
+
+| Attribute | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `max_in_flight_per_source` | Integer/nil | `25` | Max concurrent scrape jobs per source |
+| `max_bulk_batch_size` | Integer/nil | `100` | Max items in a bulk scrape batch |
+
+Has `reset!` method. Custom setters normalize values:
+- `nil` -> `nil`
+- `""` -> `nil`
+- `0` or negative -> `nil`
+- String -> parsed integer (if positive)
+- Positive integer -> kept as-is
+
+---
+
+## RetentionSettings
+
+**File:** `lib/source_monitor/configuration/retention_settings.rb`
+
+| Attribute | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `items_retention_days` | Integer/nil | `nil` | Days to keep items (nil = forever) |
+| `max_items` | Integer/nil | `nil` | Max items per source (nil = unlimited) |
+| `strategy` | Symbol | `:destroy` | `:destroy` or `:soft_delete` |
+
+No `reset!` method (defaults set in `initialize`). The `strategy=` setter validates against allowed values and raises `ArgumentError` for invalid input. Setting `nil` resets to `:destroy`.
+
+---
+
+## RealtimeSettings
+
+**File:** `lib/source_monitor/configuration/realtime_settings.rb`
+
+| Attribute | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `adapter` | Symbol | `:solid_cable` | One of `:solid_cable`, `:redis`, `:async` |
+| `redis_url` | String/nil | `nil` | Redis URL (for `:redis` adapter) |
+| `solid_cable` | SolidCableOptions | (nested object) | Solid Cable options |
+
+Has `reset!` method. The `adapter=` setter validates against `VALID_ADAPTERS`.
+
+**SolidCableOptions:**
+
+| Attribute | Type | Default |
+|-----------|------|---------|
+| `polling_interval` | String | `"0.1.seconds"` |
+| `message_retention` | String | `"1.day"` |
+| `autotrim` | Boolean | `true` |
+| `silence_polling` | Boolean | `true` |
+| `use_skip_locked` | Boolean | `true` |
+| `trim_batch_size` | Integer/nil | `nil` |
+| `connects_to` | Hash/nil | `nil` |
+
+**Methods:**
+- `action_cable_config` -- Returns hash suitable for ActionCable configuration
+- `solid_cable=(hash)` -- Bulk-assign SolidCable options from a hash
+
+---
+
+## AuthenticationSettings
+
+**File:** `lib/source_monitor/configuration/authentication_settings.rb`
+
+| Attribute | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `authenticate_handler` | Handler/nil | `nil` | Authentication handler |
+| `authorize_handler` | Handler/nil | `nil` | Authorization handler |
+| `current_user_method` | Symbol/nil | `nil` | Method name for current user |
+| `user_signed_in_method` | Symbol/nil | `nil` | Method name for signed-in check |
+
+Has `reset!` method.
+
+**Methods:**
+- `authenticate_with(handler = nil, &block)` -- Register authentication handler
+- `authorize_with(handler = nil, &block)` -- Register authorization handler
+
+Handler types: `:symbol` (method name), `:callable` (lambda/proc/block).
+
+---
+
+## Events
+
+**File:** `lib/source_monitor/configuration/events.rb`
+
+| Callback Key | Description |
+|-------------|-------------|
+| `after_item_created` | Fires after a new item is created from a feed entry |
+| `after_item_scraped` | Fires after an item's content is scraped |
+| `after_fetch_completed` | Fires after a feed fetch completes (success or failure) |
+
+**Methods:**
+- `after_item_created(handler = nil, &block)` -- Register callback
+- `after_item_scraped(handler = nil, &block)` -- Register callback
+- `after_fetch_completed(handler = nil, &block)` -- Register callback
+- `register_item_processor(processor = nil, &block)` -- Register item processor
+- `callbacks_for(name)` -- Returns array of callbacks (dup'd)
+- `item_processors` -- Returns array of processors (dup'd)
+- `reset!` -- Clears all callbacks and processors
+
+All handlers must respond to `#call`. Raises `ArgumentError` otherwise.
+
+---
+
+## ScraperRegistry
+
+**File:** `lib/source_monitor/configuration/scraper_registry.rb`
+
+Enumerable registry of scraper adapters.
+
+**Methods:**
+- `register(name, adapter)` -- Register adapter class by name
+- `unregister(name)` -- Remove adapter
+- `adapter_for(name)` -- Look up adapter class by name
+- `each` -- Iterate over `{name => adapter}` pairs
+
+Names are normalized to lowercase alphanumeric + underscores. Adapter classes must inherit from `SourceMonitor::Scrapers::Base`.
+
+---
+
+## Models
+
+**File:** `lib/source_monitor/configuration/models.rb`
+
+| Attribute | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `table_name_prefix` | String | `"sourcemon_"` | Database table prefix |
+
+**Model Keys:** `source`, `item`, `fetch_log`, `scrape_log`, `health_check_log`, `item_content`, `log_entry`
+
+Each key returns a `ModelDefinition` instance.
+
+**Methods:**
+- `for(name)` -- Returns `ModelDefinition` by name (raises on unknown)
+
+---
+
+## ModelDefinition
+
+**File:** `lib/source_monitor/configuration/model_definition.rb`
+
+**Methods:**
+- `include_concern(concern = nil, &block)` -- Register a concern module
+- `each_concern` -- Iterate over registered concerns (yields `[signature, resolved_module]`)
+- `validate(handler = nil, **options, &block)` -- Register a custom validation
+- `validations` -- Returns array of `ValidationDefinition` instances
+
+Concerns can be: Module instance, String constant name, or anonymous block.
+Validations can be: Symbol method name, String, lambda, or block.
+
+---
+
+## ValidationDefinition
+
+**File:** `lib/source_monitor/configuration/validation_definition.rb`
+
+| Attribute | Type | Description |
+|-----------|------|-------------|
+| `handler` | Symbol/Proc | The validation handler |
+| `options` | Hash | Options hash (e.g., `{ if: :active? }`) |
+
+**Methods:**
+- `signature` -- Returns deduplication key
+- `symbol?` -- Returns true if handler is a Symbol or String

data/.claude/skills/sm-configuration-setting/reference/settings-pattern.md

@@ -0,0 +1,297 @@
+# Step-by-Step: Adding a Configuration Setting
+
+## Scenario A: Add to Existing Section
+
+### Example: Add `stale_threshold_hours` to FetchingSettings
+
+**Step 1: Edit the settings file**
+
+```ruby
+# lib/source_monitor/configuration/fetching_settings.rb
+class FetchingSettings
+  attr_accessor :min_interval_minutes,
+    :max_interval_minutes,
+    :increase_factor,
+    :decrease_factor,
+    :failure_increase_factor,
+    :jitter_percent,
+    :stale_threshold_hours        # ADD: new accessor
+
+  def reset!
+    @min_interval_minutes = 5
+    @max_interval_minutes = 24 * 60
+    @increase_factor = 1.25
+    @decrease_factor = 0.75
+    @failure_increase_factor = 1.5
+    @jitter_percent = 0.1
+    @stale_threshold_hours = 48   # ADD: default value
+  end
+end
+```
+
+**Step 2: Write tests**
+
+```ruby
+# test/lib/source_monitor/configuration_test.rb
+
+test "stale_threshold_hours has correct default" do
+  assert_equal 48, SourceMonitor.config.fetching.stale_threshold_hours
+end
+
+test "stale_threshold_hours can be overridden" do
+  SourceMonitor.configure do |config|
+    config.fetching.stale_threshold_hours = 72
+  end
+  assert_equal 72, SourceMonitor.config.fetching.stale_threshold_hours
+end
+
+test "stale_threshold_hours resets with configuration" do
+  SourceMonitor.configure do |config|
+    config.fetching.stale_threshold_hours = 72
+  end
+  SourceMonitor.reset_configuration!
+  assert_equal 48, SourceMonitor.config.fetching.stale_threshold_hours
+end
+```
+
+**Step 3: Run tests**
+
+```bash
+PARALLEL_WORKERS=1 bin/rails test test/lib/source_monitor/configuration_test.rb
+```
+
+---
+
+## Scenario B: Add Setting with Validation
+
+### Example: Add validated `strategy` to a section
+
+**Step 1: Edit the settings file with custom setter**
+
+```ruby
+class MySettings
+  VALID_MODES = %i[fast balanced thorough].freeze
+
+  attr_reader :mode
+
+  def initialize
+    reset!
+  end
+
+  def mode=(value)
+    normalized = value&.to_sym
+    unless VALID_MODES.include?(normalized)
+      raise ArgumentError, "Invalid mode #{value.inspect}. Must be one of: #{VALID_MODES.join(', ')}"
+    end
+    @mode = normalized
+  end
+
+  def reset!
+    @mode = :balanced
+  end
+end
+```
+
+**Step 2: Write tests for all paths**
+
+```ruby
+test "mode defaults to balanced" do
+  assert_equal :balanced, SourceMonitor.config.my_section.mode
+end
+
+test "mode accepts valid values" do
+  %i[fast balanced thorough].each do |mode|
+    SourceMonitor.configure do |config|
+      config.my_section.mode = mode
+    end
+    assert_equal mode, SourceMonitor.config.my_section.mode
+  end
+end
+
+test "mode accepts string values" do
+  SourceMonitor.configure do |config|
+    config.my_section.mode = "fast"
+  end
+  assert_equal :fast, SourceMonitor.config.my_section.mode
+end
+
+test "mode rejects invalid values" do
+  assert_raises(ArgumentError, /Invalid mode/) do
+    SourceMonitor.configure do |config|
+      config.my_section.mode = :invalid
+    end
+  end
+end
+```
+
+---
+
+## Scenario C: Add Setting with Normalization
+
+### Example: Numeric setting that normalizes edge cases
+
+Follow the `ScrapingSettings` pattern:
+
+```ruby
+class MySettings
+  DEFAULT_LIMIT = 50
+
+  attr_reader :limit
+
+  def initialize
+    reset!
+  end
+
+  def limit=(value)
+    @limit = normalize_numeric(value)
+  end
+
+  def reset!
+    @limit = DEFAULT_LIMIT
+  end
+
+  private
+
+  def normalize_numeric(value)
+    return nil if value.nil?
+    return nil if value == ""
+    integer = value.respond_to?(:to_i) ? value.to_i : value
+    integer.positive? ? integer : nil
+  end
+end
+```
+
+**Test normalization edge cases:**
+
+```ruby
+test "limit normalizes string to integer" do
+  SourceMonitor.configure { |c| c.my_section.limit = "10" }
+  assert_equal 10, SourceMonitor.config.my_section.limit
+end
+
+test "limit normalizes nil to nil" do
+  SourceMonitor.configure { |c| c.my_section.limit = nil }
+  assert_nil SourceMonitor.config.my_section.limit
+end
+
+test "limit normalizes empty string to nil" do
+  SourceMonitor.configure { |c| c.my_section.limit = "" }
+  assert_nil SourceMonitor.config.my_section.limit
+end
+
+test "limit normalizes zero to nil" do
+  SourceMonitor.configure { |c| c.my_section.limit = 0 }
+  assert_nil SourceMonitor.config.my_section.limit
+end
+
+test "limit normalizes negative to nil" do
+  SourceMonitor.configure { |c| c.my_section.limit = -5 }
+  assert_nil SourceMonitor.config.my_section.limit
+end
+```
+
+---
+
+## Scenario D: Create a New Settings Section
+
+### Step 1: Create the settings file
+
+```ruby
+# lib/source_monitor/configuration/notifications_settings.rb
+# frozen_string_literal: true
+
+module SourceMonitor
+  class Configuration
+    class NotificationsSettings
+      attr_accessor :enabled, :channels, :throttle_seconds
+
+      def initialize
+        reset!
+      end
+
+      def reset!
+        @enabled = true
+        @channels = []
+        @throttle_seconds = 60
+      end
+    end
+  end
+end
+```
+
+### Step 2: Add require and reader to Configuration
+
+```ruby
+# lib/source_monitor/configuration.rb
+
+# At the top, add require:
+require "source_monitor/configuration/notifications_settings"
+
+# In the class:
+attr_reader :http, :scrapers, :retention, :events, :models,
+  :realtime, :fetching, :health, :authentication, :scraping,
+  :notifications
+
+# In initialize:
+def initialize
+  # ... existing ...
+  @notifications = NotificationsSettings.new
+end
+```
+
+### Step 3: Write tests
+
+```ruby
+# test/lib/source_monitor/configuration_test.rb
+
+test "notifications settings have correct defaults" do
+  settings = SourceMonitor.config.notifications
+  assert_equal true, settings.enabled
+  assert_equal [], settings.channels
+  assert_equal 60, settings.throttle_seconds
+end
+
+test "notifications settings can be configured" do
+  SourceMonitor.configure do |config|
+    config.notifications.enabled = false
+    config.notifications.channels = [:email]
+    config.notifications.throttle_seconds = 30
+  end
+
+  settings = SourceMonitor.config.notifications
+  assert_equal false, settings.enabled
+  assert_equal [:email], settings.channels
+  assert_equal 30, settings.throttle_seconds
+end
+
+test "notifications reset restores defaults" do
+  SourceMonitor.configure do |config|
+    config.notifications.enabled = false
+  end
+  SourceMonitor.reset_configuration!
+  assert_equal true, SourceMonitor.config.notifications.enabled
+end
+```
+
+### Step 4: Run tests
+
+```bash
+PARALLEL_WORKERS=1 bin/rails test test/lib/source_monitor/configuration_test.rb
+```
+
+---
+
+## Checklist for Any New Setting
+
+- [ ] Attribute added to settings class (attr_accessor or custom setter)
+- [ ] Default value set in `reset!` (or `initialize` for classes without `reset!`)
+- [ ] require statement added (if new file)
+- [ ] Reader method exposed on Configuration (if new section)
+- [ ] Initialization in Configuration#initialize (if new section)
+- [ ] Tests: default value is correct
+- [ ] Tests: value can be overridden
+- [ ] Tests: reset restores default
+- [ ] Tests: validation raises ArgumentError (if applicable)
+- [ ] Tests: edge cases for normalization (if applicable)
+- [ ] All tests pass: `PARALLEL_WORKERS=1 bin/rails test test/lib/source_monitor/configuration_test.rb`
+- [ ] RuboCop clean: `bin/rubocop lib/source_monitor/configuration/`

data/.claude/skills/sm-configure/SKILL.md

@@ -0,0 +1,153 @@
+---
+name: sm-configure
+description: Use when configuring SourceMonitor engine settings via the DSL, including queue settings, HTTP client, fetching, health, scrapers, retention, scraping controls, events, model extensions, realtime, and authentication.
+allowed-tools: Read, Write, Edit, Bash, Glob, Grep
+---
+
+# sm-configure: Engine Configuration DSL
+
+Comprehensive reference for configuring SourceMonitor via `SourceMonitor.configure`.
+
+## When to Use
+
+- Adding or modifying settings in `config/initializers/source_monitor.rb`
+- Understanding what configuration options are available
+- Debugging configuration-related issues
+- Setting up environment-specific overrides
+
+## Configuration Entry Point
+
+All configuration lives inside the `configure` block in the host app's initializer:
+
+```ruby
+SourceMonitor.configure do |config|
+  # settings here
+end
+```
+
+After the block executes, `ModelExtensions.reload!` runs automatically to apply any model changes. Restart web and worker processes after changes.
+
+## Configuration Sections
+
+The `config` object (`SourceMonitor::Configuration`) has 10 sub-sections plus top-level queue/job settings:
+
+| Section | Accessor | Class |
+|---|---|---|
+| Top-level | `config.*` | `Configuration` |
+| HTTP | `config.http` | `HTTPSettings` |
+| Fetching | `config.fetching` | `FetchingSettings` |
+| Health | `config.health` | `HealthSettings` |
+| Scrapers | `config.scrapers` | `ScraperRegistry` |
+| Retention | `config.retention` | `RetentionSettings` |
+| Scraping | `config.scraping` | `ScrapingSettings` |
+| Events | `config.events` | `Events` |
+| Models | `config.models` | `Models` |
+| Realtime | `config.realtime` | `RealtimeSettings` |
+| Authentication | `config.authentication` | `AuthenticationSettings` |
+
+See `reference/configuration-reference.md` for every setting with types, defaults, and examples.
+
+## Quick Examples
+
+### Queue Configuration
+```ruby
+config.queue_namespace = "source_monitor"
+config.fetch_queue_name = "source_monitor_fetch"
+config.fetch_queue_concurrency = 4
+```
+
+### HTTP Client
+```ruby
+config.http.timeout = 30
+config.http.proxy = ENV["HTTP_PROXY"]
+config.http.retry_max = 3
+```
+
+### Authentication (Devise)
+```ruby
+config.authentication.authenticate_with :authenticate_user!
+config.authentication.authorize_with ->(c) { c.current_user&.admin? }
+```
+
+### Events
+```ruby
+config.events.after_item_created { |e| Notifier.new_item(e.item) }
+config.events.register_item_processor ->(ctx) { Indexer.index(ctx.item) }
+```
+
+### Model Extensions
+```ruby
+config.models.table_name_prefix = "sm_"
+config.models.source.include_concern "MyApp::SourceExtension"
+config.models.item.validate :custom_check
+```
+
+### Realtime
+```ruby
+config.realtime.adapter = :redis
+config.realtime.redis_url = ENV["REDIS_URL"]
+```
+
+## Helper APIs
+
+```ruby
+SourceMonitor.config                         # Current configuration
+SourceMonitor.configure { |c| ... }          # Set configuration
+SourceMonitor.reset_configuration!           # Revert to defaults (for tests)
+SourceMonitor.events                         # Shortcut to config.events
+SourceMonitor.queue_name(:fetch)             # Resolved queue name
+SourceMonitor.queue_concurrency(:scrape)     # Resolved concurrency
+SourceMonitor.mission_control_dashboard_path # Resolved MC path or nil
+```
+
+## Key Source Files
+
+| File | Purpose |
+|---|---|
+| `lib/source_monitor/configuration.rb` | Main Configuration class |
+| `lib/source_monitor/configuration/http_settings.rb` | HTTP client settings |
+| `lib/source_monitor/configuration/fetching_settings.rb` | Adaptive scheduling |
+| `lib/source_monitor/configuration/health_settings.rb` | Health monitoring |
+| `lib/source_monitor/configuration/scraper_registry.rb` | Scraper adapter registry |
+| `lib/source_monitor/configuration/retention_settings.rb` | Item retention |
+| `lib/source_monitor/configuration/scraping_settings.rb` | Scraping controls |
+| `lib/source_monitor/configuration/events.rb` | Event callbacks |
+| `lib/source_monitor/configuration/models.rb` | Model extensions config |
+| `lib/source_monitor/configuration/model_definition.rb` | Per-model definition |
+| `lib/source_monitor/configuration/realtime_settings.rb` | Action Cable settings |
+| `lib/source_monitor/configuration/authentication_settings.rb` | Auth settings |
+| `lib/source_monitor/configuration/validation_definition.rb` | Validation wrapper |
+
+## References
+
+- `reference/configuration-reference.md` -- Complete settings reference
+- `docs/configuration.md` -- Official configuration documentation
+- `lib/generators/source_monitor/install/templates/source_monitor.rb.tt` -- Initializer template
+
+## Testing
+
+Reset configuration between tests:
+```ruby
+setup do
+  SourceMonitor.reset_configuration!
+end
+```
+
+Test custom configuration:
+```ruby
+test "custom queue name" do
+  SourceMonitor.configure do |config|
+    config.fetch_queue_name = "custom_fetch"
+  end
+  assert_equal "custom_fetch", SourceMonitor.queue_name(:fetch)
+end
+```
+
+## Checklist
+
+- [ ] Initializer exists at `config/initializers/source_monitor.rb`
+- [ ] Queue names match `config/solid_queue.yml` entries
+- [ ] Authentication hooks configured for host auth system
+- [ ] HTTP timeouts appropriate for target feeds
+- [ ] Retention policy set for production
+- [ ] Workers restarted after configuration changes