source_monitor 0.3.0 → 0.3.2

data/.claude/skills/sm-configuration-setting/SKILL.md

@@ -0,0 +1,264 @@
+---
+name: sm-configuration-setting
+description: How to add or modify configuration settings in the Source Monitor engine. Use when adding a new config option, modifying defaults, creating a new settings section, or understanding the configuration architecture.
+allowed-tools: Read, Write, Edit, Bash, Glob, Grep
+---
+
+# Source Monitor Configuration Settings
+
+## Architecture Overview
+
+The `SourceMonitor::Configuration` class was refactored from 655 lines into a lean 87-line orchestrator plus 12 extracted settings files. Each settings file follows a consistent pattern.
+
+```
+lib/source_monitor/configuration.rb          # Main class (87 lines)
+lib/source_monitor/configuration/
+  authentication_settings.rb                  # Auth handlers
+  events.rb                                   # Callbacks and processors
+  fetching_settings.rb                        # Adaptive interval tuning
+  health_settings.rb                          # Health monitoring thresholds
+  http_settings.rb                            # HTTP client defaults
+  model_definition.rb                         # Concern/validation injection
+  models.rb                                   # Model definitions registry
+  realtime_settings.rb                        # ActionCable adapter config
+  retention_settings.rb                       # Item retention policies
+  scraper_registry.rb                         # Scraper adapter registry
+  scraping_settings.rb                        # Scraping concurrency limits
+  validation_definition.rb                    # Validation DSL
+```
+
+## How Configuration Works
+
+```ruby
+# In host app initializer
+SourceMonitor.configure do |config|
+  config.fetching.min_interval_minutes = 10
+  config.http.timeout = 30
+  config.retention.strategy = :soft_delete
+end
+
+# Access at runtime
+SourceMonitor.config.fetching.min_interval_minutes  # => 10
+```
+
+The `config` object is a `SourceMonitor::Configuration` instance. Sub-sections are accessed via reader methods that return settings objects.
+
+## Adding a Setting to an Existing Section
+
+### Step 1: Identify the Settings File
+
+| Setting Category | File | Class |
+|-----------------|------|-------|
+| HTTP client | `http_settings.rb` | `HTTPSettings` |
+| Adaptive fetching | `fetching_settings.rb` | `FetchingSettings` |
+| Health monitoring | `health_settings.rb` | `HealthSettings` |
+| Scraping limits | `scraping_settings.rb` | `ScrapingSettings` |
+| Item retention | `retention_settings.rb` | `RetentionSettings` |
+| Realtime/cable | `realtime_settings.rb` | `RealtimeSettings` |
+| Authentication | `authentication_settings.rb` | `AuthenticationSettings` |
+| Events/callbacks | `events.rb` | `Events` |
+| Scraper adapters | `scraper_registry.rb` | `ScraperRegistry` |
+| Model extensions | `models.rb` / `model_definition.rb` | `Models` / `ModelDefinition` |
+
+### Step 2: Add the Attribute
+
+```ruby
+# lib/source_monitor/configuration/fetching_settings.rb
+class FetchingSettings
+  attr_accessor :min_interval_minutes,
+    :max_interval_minutes,
+    :increase_factor,
+    :decrease_factor,
+    :failure_increase_factor,
+    :jitter_percent,
+    :my_new_setting              # <-- Add here
+
+  def reset!
+    @min_interval_minutes = 5
+    @max_interval_minutes = 24 * 60
+    @increase_factor = 1.25
+    @decrease_factor = 0.75
+    @failure_increase_factor = 1.5
+    @jitter_percent = 0.1
+    @my_new_setting = "default"  # <-- Set default here
+  end
+end
+```
+
+### Step 3: Write Tests
+
+```ruby
+# test/lib/source_monitor/configuration_test.rb
+test "my_new_setting has correct default" do
+  assert_equal "default", SourceMonitor.config.fetching.my_new_setting
+end
+
+test "my_new_setting can be overridden" do
+  SourceMonitor.configure do |config|
+    config.fetching.my_new_setting = "custom"
+  end
+  assert_equal "custom", SourceMonitor.config.fetching.my_new_setting
+end
+```
+
+### Step 4: Verify Reset
+
+Ensure `reset!` restores the default. The test suite calls `SourceMonitor.reset_configuration!` in setup, which recreates the entire Configuration object.
+
+## Adding a Setting with Validation
+
+For settings that need input normalization or validation, use custom setters:
+
+```ruby
+class ScrapingSettings
+  attr_accessor :max_in_flight_per_source, :max_bulk_batch_size
+
+  # Custom setter with normalization
+  def max_in_flight_per_source=(value)
+    @max_in_flight_per_source = normalize_numeric(value)
+  end
+
+  private
+
+  def normalize_numeric(value)
+    return nil if value.nil?
+    return nil if value == ""
+    integer = value.respond_to?(:to_i) ? value.to_i : value
+    integer.positive? ? integer : nil
+  end
+end
+```
+
+For enum-style settings with strict validation:
+
+```ruby
+class RetentionSettings
+  def strategy=(value)
+    normalized = normalize_strategy(value)
+    @strategy = normalized unless normalized.nil?
+  end
+
+  private
+
+  def normalize_strategy(value)
+    return :destroy if value.nil?
+    if value.respond_to?(:to_sym)
+      candidate = value.to_sym
+      raise ArgumentError, "Invalid retention strategy #{value.inspect}" unless %i[destroy soft_delete].include?(candidate)
+      candidate
+    else
+      raise ArgumentError, "Invalid retention strategy #{value.inspect}"
+    end
+  end
+end
+```
+
+## Creating a New Settings Section
+
+### Step 1: Create the Settings Class
+
+```ruby
+# lib/source_monitor/configuration/notifications_settings.rb
+# frozen_string_literal: true
+
+module SourceMonitor
+  class Configuration
+    class NotificationsSettings
+      attr_accessor :enabled, :channels, :throttle_seconds
+
+      def initialize
+        reset!
+      end
+
+      def reset!
+        @enabled = true
+        @channels = []
+        @throttle_seconds = 60
+      end
+    end
+  end
+end
+```
+
+### Step 2: Register in Configuration
+
+```ruby
+# lib/source_monitor/configuration.rb
+require "source_monitor/configuration/notifications_settings"
+
+class Configuration
+  attr_reader :http, :scrapers, :retention, :events, :models,
+    :realtime, :fetching, :health, :authentication, :scraping,
+    :notifications  # <-- Add reader
+
+  def initialize
+    # ... existing initialization ...
+    @notifications = NotificationsSettings.new  # <-- Initialize
+  end
+end
+```
+
+### Step 3: Write Tests
+
+```ruby
+test "notifications settings have correct defaults" do
+  settings = SourceMonitor.config.notifications
+  assert_equal true, settings.enabled
+  assert_equal [], settings.channels
+  assert_equal 60, settings.throttle_seconds
+end
+
+test "notifications settings can be configured" do
+  SourceMonitor.configure do |config|
+    config.notifications.enabled = false
+    config.notifications.channels = [:email, :slack]
+    config.notifications.throttle_seconds = 30
+  end
+
+  settings = SourceMonitor.config.notifications
+  assert_equal false, settings.enabled
+  assert_equal [:email, :slack], settings.channels
+  assert_equal 30, settings.throttle_seconds
+end
+```
+
+## Patterns by Section Type
+
+### Simple Accessor Pattern (FetchingSettings, HealthSettings)
+
+Plain `attr_accessor` with defaults in `reset!`. No validation.
+
+### Normalized Setter Pattern (ScrapingSettings)
+
+Custom setter that normalizes input (strings to integers, negatives to nil).
+
+### Enum Setter Pattern (RetentionSettings, RealtimeSettings)
+
+Custom setter that validates against an allowed list and raises `ArgumentError`.
+
+### Handler/Callback Pattern (AuthenticationSettings, Events)
+
+Registration methods that accept symbols, lambdas, or blocks.
+
+### Registry Pattern (ScraperRegistry)
+
+Named registration with lookup, unregistration, and enumeration.
+
+### Nested Object Pattern (RealtimeSettings::SolidCableOptions)
+
+Sub-objects with their own `reset!` and `to_h` methods.
+
+## Testing Checklist
+
+- [ ] Default value is correct
+- [ ] Value can be overridden via `SourceMonitor.configure`
+- [ ] `reset!` restores the default (tested via `SourceMonitor.reset_configuration!`)
+- [ ] Validation raises `ArgumentError` for invalid values (if applicable)
+- [ ] String/nil normalization works correctly (if applicable)
+- [ ] Test file: `test/lib/source_monitor/configuration_test.rb`
+- [ ] Run: `PARALLEL_WORKERS=1 bin/rails test test/lib/source_monitor/configuration_test.rb`
+
+## References
+
+- [reference/settings-catalog.md](reference/settings-catalog.md) -- All settings sections with their attributes
+- [reference/settings-pattern.md](reference/settings-pattern.md) -- Step-by-step pattern for adding settings

data/.claude/skills/sm-configuration-setting/reference/settings-catalog.md

@@ -0,0 +1,248 @@
+# Configuration Settings Catalog
+
+All configuration sections with their attributes, defaults, and types.
+
+## Top-Level Attributes
+
+**File:** `lib/source_monitor/configuration.rb`
+
+| Attribute | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `queue_namespace` | String | `"source_monitor"` | Namespace prefix for queue names |
+| `fetch_queue_name` | String | `"source_monitor_fetch"` | Queue name for fetch jobs |
+| `scrape_queue_name` | String | `"source_monitor_scrape"` | Queue name for scrape jobs |
+| `fetch_queue_concurrency` | Integer | `2` | Max concurrent fetch workers |
+| `scrape_queue_concurrency` | Integer | `2` | Max concurrent scrape workers |
+| `recurring_command_job_class` | Class/nil | `nil` | Custom recurring job class |
+| `job_metrics_enabled` | Boolean | `true` | Enable job metrics tracking |
+| `mission_control_enabled` | Boolean | `false` | Enable Mission Control integration |
+| `mission_control_dashboard_path` | String/Proc/nil | `nil` | Path or callable for Mission Control |
+
+**Methods:**
+- `queue_name_for(:fetch)` / `queue_name_for(:scrape)` -- Returns prefixed queue name
+- `concurrency_for(:fetch)` / `concurrency_for(:scrape)` -- Returns concurrency limit
+
+---
+
+## HTTPSettings
+
+**File:** `lib/source_monitor/configuration/http_settings.rb`
+
+| Attribute | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `timeout` | Integer | `15` | Total request timeout (seconds) |
+| `open_timeout` | Integer | `5` | Connection open timeout (seconds) |
+| `max_redirects` | Integer | `5` | Max HTTP redirects to follow |
+| `user_agent` | String | `"SourceMonitor/<version>"` | User-Agent header value |
+| `proxy` | String/nil | `nil` | HTTP proxy URL |
+| `headers` | Hash | `{}` | Default HTTP headers |
+| `retry_max` | Integer | `4` | Max retry attempts |
+| `retry_interval` | Float | `0.5` | Base retry interval (seconds) |
+| `retry_interval_randomness` | Float | `0.5` | Retry interval randomness factor |
+| `retry_backoff_factor` | Integer | `2` | Exponential backoff multiplier |
+| `retry_statuses` | Array/nil | `nil` | HTTP statuses to retry on |
+
+Has `reset!` method.
+
+---
+
+## FetchingSettings
+
+**File:** `lib/source_monitor/configuration/fetching_settings.rb`
+
+| Attribute | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `min_interval_minutes` | Integer | `5` | Minimum fetch interval |
+| `max_interval_minutes` | Integer | `1440` (24h) | Maximum fetch interval |
+| `increase_factor` | Float | `1.25` | Multiplier when content unchanged |
+| `decrease_factor` | Float | `0.75` | Multiplier when content changed |
+| `failure_increase_factor` | Float | `1.5` | Multiplier on fetch failure |
+| `jitter_percent` | Float | `0.1` | Random jitter (10%) |
+
+Has `reset!` method. All attributes are plain `attr_accessor`.
+
+---
+
+## HealthSettings
+
+**File:** `lib/source_monitor/configuration/health_settings.rb`
+
+| Attribute | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `window_size` | Integer | `20` | Rolling window of fetches for health calc |
+| `healthy_threshold` | Float | `0.8` | Success rate above = healthy |
+| `warning_threshold` | Float | `0.5` | Success rate above = warning |
+| `auto_pause_threshold` | Float | `0.2` | Success rate below = auto-pause |
+| `auto_resume_threshold` | Float | `0.6` | Success rate above = auto-resume |
+| `auto_pause_cooldown_minutes` | Integer | `60` | Cooldown before auto-resume check |
+
+Has `reset!` method. All attributes are plain `attr_accessor`.
+
+---
+
+## ScrapingSettings
+
+**File:** `lib/source_monitor/configuration/scraping_settings.rb`
+
+| Attribute | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `max_in_flight_per_source` | Integer/nil | `25` | Max concurrent scrape jobs per source |
+| `max_bulk_batch_size` | Integer/nil | `100` | Max items in a bulk scrape batch |
+
+Has `reset!` method. Custom setters normalize values:
+- `nil` -> `nil`
+- `""` -> `nil`
+- `0` or negative -> `nil`
+- String -> parsed integer (if positive)
+- Positive integer -> kept as-is
+
+---
+
+## RetentionSettings
+
+**File:** `lib/source_monitor/configuration/retention_settings.rb`
+
+| Attribute | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `items_retention_days` | Integer/nil | `nil` | Days to keep items (nil = forever) |
+| `max_items` | Integer/nil | `nil` | Max items per source (nil = unlimited) |
+| `strategy` | Symbol | `:destroy` | `:destroy` or `:soft_delete` |
+
+No `reset!` method (defaults set in `initialize`). The `strategy=` setter validates against allowed values and raises `ArgumentError` for invalid input. Setting `nil` resets to `:destroy`.
+
+---
+
+## RealtimeSettings
+
+**File:** `lib/source_monitor/configuration/realtime_settings.rb`
+
+| Attribute | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `adapter` | Symbol | `:solid_cable` | One of `:solid_cable`, `:redis`, `:async` |
+| `redis_url` | String/nil | `nil` | Redis URL (for `:redis` adapter) |
+| `solid_cable` | SolidCableOptions | (nested object) | Solid Cable options |
+
+Has `reset!` method. The `adapter=` setter validates against `VALID_ADAPTERS`.
+
+**SolidCableOptions:**
+
+| Attribute | Type | Default |
+|-----------|------|---------|
+| `polling_interval` | String | `"0.1.seconds"` |
+| `message_retention` | String | `"1.day"` |
+| `autotrim` | Boolean | `true` |
+| `silence_polling` | Boolean | `true` |
+| `use_skip_locked` | Boolean | `true` |
+| `trim_batch_size` | Integer/nil | `nil` |
+| `connects_to` | Hash/nil | `nil` |
+
+**Methods:**
+- `action_cable_config` -- Returns hash suitable for ActionCable configuration
+- `solid_cable=(hash)` -- Bulk-assign SolidCable options from a hash
+
+---
+
+## AuthenticationSettings
+
+**File:** `lib/source_monitor/configuration/authentication_settings.rb`
+
+| Attribute | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `authenticate_handler` | Handler/nil | `nil` | Authentication handler |
+| `authorize_handler` | Handler/nil | `nil` | Authorization handler |
+| `current_user_method` | Symbol/nil | `nil` | Method name for current user |
+| `user_signed_in_method` | Symbol/nil | `nil` | Method name for signed-in check |
+
+Has `reset!` method.
+
+**Methods:**
+- `authenticate_with(handler = nil, &block)` -- Register authentication handler
+- `authorize_with(handler = nil, &block)` -- Register authorization handler
+
+Handler types: `:symbol` (method name), `:callable` (lambda/proc/block).
+
+---
+
+## Events
+
+**File:** `lib/source_monitor/configuration/events.rb`
+
+| Callback Key | Description |
+|-------------|-------------|
+| `after_item_created` | Fires after a new item is created from a feed entry |
+| `after_item_scraped` | Fires after an item's content is scraped |
+| `after_fetch_completed` | Fires after a feed fetch completes (success or failure) |
+
+**Methods:**
+- `after_item_created(handler = nil, &block)` -- Register callback
+- `after_item_scraped(handler = nil, &block)` -- Register callback
+- `after_fetch_completed(handler = nil, &block)` -- Register callback
+- `register_item_processor(processor = nil, &block)` -- Register item processor
+- `callbacks_for(name)` -- Returns array of callbacks (dup'd)
+- `item_processors` -- Returns array of processors (dup'd)
+- `reset!` -- Clears all callbacks and processors
+
+All handlers must respond to `#call`. Raises `ArgumentError` otherwise.
+
+---
+
+## ScraperRegistry
+
+**File:** `lib/source_monitor/configuration/scraper_registry.rb`
+
+Enumerable registry of scraper adapters.
+
+**Methods:**
+- `register(name, adapter)` -- Register adapter class by name
+- `unregister(name)` -- Remove adapter
+- `adapter_for(name)` -- Look up adapter class by name
+- `each` -- Iterate over `{name => adapter}` pairs
+
+Names are normalized to lowercase alphanumeric + underscores. Adapter classes must inherit from `SourceMonitor::Scrapers::Base`.
+
+---
+
+## Models
+
+**File:** `lib/source_monitor/configuration/models.rb`
+
+| Attribute | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `table_name_prefix` | String | `"sourcemon_"` | Database table prefix |
+
+**Model Keys:** `source`, `item`, `fetch_log`, `scrape_log`, `health_check_log`, `item_content`, `log_entry`
+
+Each key returns a `ModelDefinition` instance.
+
+**Methods:**
+- `for(name)` -- Returns `ModelDefinition` by name (raises on unknown)
+
+---
+
+## ModelDefinition
+
+**File:** `lib/source_monitor/configuration/model_definition.rb`
+
+**Methods:**
+- `include_concern(concern = nil, &block)` -- Register a concern module
+- `each_concern` -- Iterate over registered concerns (yields `[signature, resolved_module]`)
+- `validate(handler = nil, **options, &block)` -- Register a custom validation
+- `validations` -- Returns array of `ValidationDefinition` instances
+
+Concerns can be: Module instance, String constant name, or anonymous block.
+Validations can be: Symbol method name, String, lambda, or block.
+
+---
+
+## ValidationDefinition
+
+**File:** `lib/source_monitor/configuration/validation_definition.rb`
+
+| Attribute | Type | Description |
+|-----------|------|-------------|
+| `handler` | Symbol/Proc | The validation handler |
+| `options` | Hash | Options hash (e.g., `{ if: :active? }`) |
+
+**Methods:**
+- `signature` -- Returns deduplication key
+- `symbol?` -- Returns true if handler is a Symbol or String