source_monitor 0.3.0 → 0.3.2

data/.claude/skills/sm-event-handler/reference/events-api.md

@@ -0,0 +1,229 @@
+# Events API Reference
+
+Complete reference for SourceMonitor's event system.
+
+Source: `lib/source_monitor/events.rb` and `lib/source_monitor/configuration/events.rb`
+
+## Registration API
+
+All registration happens on `config.events` inside the configure block:
+
+```ruby
+SourceMonitor.configure do |config|
+  config.events.after_item_created { |event| ... }
+  config.events.after_item_scraped(handler)
+  config.events.after_fetch_completed(MyHandler.new)
+  config.events.register_item_processor(->(ctx) { ... })
+end
+```
+
+### `after_item_created(handler = nil, &block)`
+
+Register a callback for new item creation.
+
+**Handler requirements:** Must respond to `#call`. Receives an `ItemCreatedEvent`.
+
+**Returns:** The registered callable.
+
+### `after_item_scraped(handler = nil, &block)`
+
+Register a callback for item scrape completion.
+
+**Handler requirements:** Must respond to `#call`. Receives an `ItemScrapedEvent`.
+
+**Returns:** The registered callable.
+
+### `after_fetch_completed(handler = nil, &block)`
+
+Register a callback for feed fetch completion.
+
+**Handler requirements:** Must respond to `#call`. Receives a `FetchCompletedEvent`.
+
+**Returns:** The registered callable.
+
+### `register_item_processor(processor = nil, &block)`
+
+Register an item processor for post-entry processing.
+
+**Handler requirements:** Must respond to `#call`. Receives an `ItemProcessorContext`.
+
+**Returns:** The registered callable.
+
+### `callbacks_for(name) -> Array`
+
+Retrieve a copy of registered callbacks for a given event name.
+
+### `item_processors -> Array`
+
+Retrieve a copy of registered item processors.
+
+### `reset!`
+
+Clear all callbacks and item processors. Used in tests.
+
+## Event Structs
+
+### `ItemCreatedEvent`
+
+Fired by `Events.after_item_created` after a new item is created from a feed entry.
+
+```ruby
+ItemCreatedEvent = Struct.new(
+  :item,        # SourceMonitor::Item - the newly created item
+  :source,      # SourceMonitor::Source - the owning source
+  :entry,       # Object - raw feed entry from Feedjira
+  :result,      # Object - creation result
+  :status,      # String - result status (e.g., "created")
+  :occurred_at, # Time - when the event fired
+  keyword_init: true
+)
+```
+
+**Helper methods:**
+- `created?` -- returns `true` when `status.to_s == "created"`
+
+**Dispatched from:** `SourceMonitor::Events.after_item_created` (called by `EntryProcessor`)
+
+### `ItemScrapedEvent`
+
+Fired by `Events.after_item_scraped` after content scraping completes.
+
+```ruby
+ItemScrapedEvent = Struct.new(
+  :item,        # SourceMonitor::Item - the scraped item
+  :source,      # SourceMonitor::Source - the owning source
+  :result,      # Object - scrape result
+  :log,         # SourceMonitor::ScrapeLog - the scrape log record
+  :status,      # String - result status
+  :occurred_at, # Time - when the event fired
+  keyword_init: true
+)
+```
+
+**Helper methods:**
+- `success?` -- returns `true` when `status.to_s != "failed"`
+
+**Dispatched from:** `SourceMonitor::Events.after_item_scraped` (called by `ItemScraper`)
+
+### `FetchCompletedEvent`
+
+Fired by `Events.after_fetch_completed` after a feed fetch finishes.
+
+```ruby
+FetchCompletedEvent = Struct.new(
+  :source,      # SourceMonitor::Source - the fetched source
+  :result,      # Object - fetch result
+  :status,      # String - result status
+  :occurred_at, # Time - when the event fired
+  keyword_init: true
+)
+```
+
+**Dispatched from:** `SourceMonitor::Events.after_fetch_completed` (called by `Completion::EventPublisher`)
+
+### `ItemProcessorContext`
+
+Passed to item processors registered via `register_item_processor`.
+
+```ruby
+ItemProcessorContext = Struct.new(
+  :item,        # SourceMonitor::Item - the processed item
+  :source,      # SourceMonitor::Source - the owning source
+  :entry,       # Object - raw feed entry
+  :result,      # Object - processing result
+  :status,      # String - result status
+  :occurred_at, # Time - when processing occurred
+  keyword_init: true
+)
+```
+
+**Dispatched from:** `SourceMonitor::Events.run_item_processors` (called by `EntryProcessor`)
+
+## Dispatch Mechanics
+
+### `Events.dispatch(event_name, event)`
+
+Iterates all callbacks for the event name and calls each one:
+
+```ruby
+def dispatch(event_name, event)
+  SourceMonitor.config.events.callbacks_for(event_name).each do |callback|
+    invoke(callback, event)
+  rescue StandardError => error
+    log_handler_error(event_name, callback, error)
+  end
+end
+```
+
+### `Events.invoke(callable, event)`
+
+Handles zero-arity and single-arity callables:
+
+```ruby
+def invoke(callable, event)
+  if callable.respond_to?(:arity) && callable.arity.zero?
+    callable.call
+  else
+    callable.call(event)
+  end
+end
+```
+
+### Error Logging
+
+Handler errors are logged but never propagated:
+
+```
+[SourceMonitor] after_item_created handler #<Proc:0x...> failed: RuntimeError: boom
+```
+
+Logged via `Rails.logger.error` with `warn` as fallback.
+
+## Handler Types
+
+| Type | Example | Notes |
+|---|---|---|
+| Block | `after_item_created { \|e\| ... }` | Most common for simple handlers |
+| Lambda | `after_item_created ->( e) { ... }` | Strict arity checking |
+| Proc | `after_item_created proc { \|e\| ... }` | Relaxed arity |
+| Object | `after_item_created(MyHandler.new)` | Must define `#call` |
+| Zero-arity | `after_item_created -> { ... }` | Called without event argument |
+
+## Multiple Handlers
+
+Multiple handlers can be registered for the same event. They execute in registration order:
+
+```ruby
+config.events.after_item_created { |e| log(e) }           # runs first
+config.events.after_item_created { |e| notify(e) }        # runs second
+config.events.after_item_created { |e| index(e) }         # runs third
+```
+
+If handler 2 raises, handlers 1 and 3 still execute (error is caught after each).
+
+## Callback Keys
+
+The `CALLBACK_KEYS` constant defines valid event names:
+
+```ruby
+CALLBACK_KEYS = %i[after_item_created after_item_scraped after_fetch_completed].freeze
+```
+
+Registering an unknown event raises `ArgumentError`.
+
+## Pipeline Integration Points
+
+| Event | Triggered By | File |
+|---|---|---|
+| `after_item_created` | `EntryProcessor` after creating item | `lib/source_monitor/fetching/feed_fetcher/entry_processor.rb` |
+| `after_item_scraped` | `ItemScraper` after scraping | `lib/source_monitor/scraping/item_scraper.rb` |
+| `after_fetch_completed` | `EventPublisher` after fetch | `lib/source_monitor/fetching/completion/event_publisher.rb` |
+| Item processors | `EntryProcessor` after item created | `lib/source_monitor/fetching/feed_fetcher/entry_processor.rb` |
+
+## Best Practices
+
+1. **Keep handlers lightweight** -- heavy work should be enqueued as background jobs
+2. **Handlers should be idempotent** -- they may be retried or run multiple times
+3. **Never raise in handlers** -- errors are caught but indicate problems
+4. **Use item processors for normalization** -- they run close to creation
+5. **Use event callbacks for side effects** -- notifications, indexing, analytics

data/.claude/skills/sm-health-rule/SKILL.md

@@ -0,0 +1,327 @@
+---
+name: sm-health-rule
+description: Health status rules, circuit breaker, and auto-pause logic for SourceMonitor sources. Use when working with health checks, health status transitions, auto-pause thresholds, circuit breaker behavior, or adding new health rules.
+allowed-tools: Read, Write, Edit, Bash, Glob, Grep
+disable-model-invocation: true
+---
+
+# SourceMonitor Health Rule Development
+
+## Overview
+
+The health system monitors source reliability by tracking fetch success rates and automatically pausing unreliable sources. It consists of three main components:
+
+1. **SourceHealthMonitor** -- evaluates rolling success rate, determines health status, triggers auto-pause/resume
+2. **SourceHealthCheck** -- performs on-demand HTTP health checks
+3. **SourceHealthReset** -- resets all health state for a source
+
+## Architecture
+
+```
+Health Module (setup!)
+  |
+  +-- Registers callback: after_fetch_completed -> SourceHealthMonitor
+  |
+  +-- SourceHealthMonitor (per-fetch evaluation)
+  |     +-- Reads recent fetch_logs
+  |     +-- Calculates rolling_success_rate
+  |     +-- Determines health_status
+  |     +-- Triggers auto-pause / auto-resume
+  |
+  +-- SourceHealthCheck (on-demand)
+  |     +-- HTTP GET to feed_url
+  |     +-- Creates HealthCheckLog record
+  |     +-- Returns Result struct
+  |
+  +-- SourceHealthReset (manual reset)
+        +-- Clears all health state
+        +-- Resets to "healthy" status
+```
+
+## Key Files
+
+| File | Purpose | Lines |
+|------|---------|-------|
+| `lib/source_monitor/health.rb` | Module entry point, callback registration | 47 |
+| `lib/source_monitor/health/source_health_monitor.rb` | Rolling success rate + status + auto-pause | 210 |
+| `lib/source_monitor/health/source_health_check.rb` | On-demand HTTP health check | 100 |
+| `lib/source_monitor/health/source_health_reset.rb` | Reset all health state | 68 |
+| `lib/source_monitor/health/import_source_health_check.rb` | Health check for import candidates | 55 |
+| `lib/source_monitor/configuration/health_settings.rb` | Configuration defaults | 27 |
+| `app/models/source_monitor/health_check_log.rb` | Health check log record | 28 |
+| `app/jobs/source_monitor/source_health_check_job.rb` | Background health check job | 77 |
+
+## Health Status Values
+
+| Status | Meaning | Trigger |
+|--------|---------|---------|
+| `healthy` | Source is reliable | success_rate >= healthy_threshold (0.8) |
+| `warning` | Some failures occurring | success_rate >= warning_threshold (0.5) but < healthy |
+| `critical` | High failure rate | success_rate < warning_threshold |
+| `declining` | Consecutive failures | >= 3 consecutive failures in recent logs |
+| `improving` | Recovery in progress | >= 2 consecutive successes after a failure |
+| `auto_paused` | Automatically paused | success_rate < auto_pause_threshold (0.2) |
+
+### Status Priority (highest to lowest)
+
+```
+auto_paused > declining > improving > healthy > warning > critical
+```
+
+Determination logic:
+
+```ruby
+def determine_status(rate, auto_paused_until, logs)
+  if auto_paused_active?(auto_paused_until)
+    "auto_paused"
+  elsif consecutive_failures(logs) >= 3
+    "declining"
+  elsif improving_streak?(logs)
+    "improving"
+  elsif rate >= healthy_threshold
+    "healthy"
+  elsif rate >= warning_threshold
+    "warning"
+  else
+    "critical"
+  end
+end
+```
+
+## Health Configuration
+
+### Default Settings
+
+| Setting | Default | Purpose |
+|---------|---------|---------|
+| `window_size` | 20 | Number of recent fetch logs to evaluate |
+| `healthy_threshold` | 0.8 | Success rate for "healthy" status |
+| `warning_threshold` | 0.5 | Success rate for "warning" status |
+| `auto_pause_threshold` | 0.2 | Below this, source is auto-paused |
+| `auto_resume_threshold` | 0.6 | Above this, auto-pause is lifted |
+| `auto_pause_cooldown_minutes` | 60 | Minimum pause duration |
+
+### Per-Source Override
+
+Sources can override `health_auto_pause_threshold` (validated 0-1 range):
+
+```ruby
+source.health_auto_pause_threshold = 0.3  # More tolerant than default 0.2
+```
+
+### Configuration Access
+
+```ruby
+SourceMonitor.configure do |config|
+  config.health.window_size = 30
+  config.health.auto_pause_threshold = 0.15
+  config.health.auto_pause_cooldown_minutes = 120
+end
+```
+
+## SourceHealthMonitor
+
+### How It Works
+
+The monitor runs automatically after every fetch via the `after_fetch_completed` event callback.
+
+**Step 1: Gather Data**
+```ruby
+logs = source.fetch_logs.order(started_at: :desc).limit(window_size)
+rate = successes.to_f / total
+```
+
+**Step 2: Check Thresholds**
+
+Thresholds only apply when `logs.size >= window_size` (minimum sample size).
+
+**Step 3: Auto-Resume Check**
+
+If source is currently auto-paused and success rate >= `auto_resume_threshold`:
+- Clear `auto_paused_until` and `auto_paused_at`
+- Clear `backoff_until`
+
+**Step 4: Auto-Pause Check**
+
+If success rate < `auto_pause_threshold`:
+- Set `auto_paused_until` to `now + cooldown_minutes`
+- Set `auto_paused_at` to now (or keep existing)
+- Push `next_fetch_at` and `backoff_until` past the pause window
+
+**Step 5: Fixed Interval Enforcement**
+
+For non-adaptive sources that are not paused, clear `backoff_until` and reset `next_fetch_at` to the fixed interval.
+
+**Step 6: Apply Status**
+
+Only updates `health_status` and `health_status_changed_at` when the status actually changes.
+
+### Source Fields Updated
+
+| Field | Type | Purpose |
+|-------|------|---------|
+| `health_status` | string | Current health status |
+| `health_status_changed_at` | datetime | When status last changed |
+| `rolling_success_rate` | float | Current success rate (0.0-1.0) |
+| `auto_paused_at` | datetime | When auto-pause was triggered |
+| `auto_paused_until` | datetime | When auto-pause expires |
+| `health_auto_pause_threshold` | float | Per-source override |
+
+## Circuit Breaker (Fetch-Level)
+
+Separate from health status, the fetch pipeline has its own circuit breaker via `RetryPolicy`:
+
+| Field | Purpose |
+|-------|---------|
+| `fetch_retry_attempt` | Current retry count |
+| `fetch_circuit_opened_at` | When circuit was opened |
+| `fetch_circuit_until` | When circuit closes |
+
+```ruby
+def fetch_circuit_open?
+  fetch_circuit_until.present? && fetch_circuit_until.future?
+end
+```
+
+Circuit breaker is managed by `RetryPolicy` in `SourceUpdater` and `FetchFeedJob`. It is distinct from the health system's auto-pause.
+
+## SourceHealthCheck
+
+On-demand HTTP check that creates a `HealthCheckLog`:
+
+```ruby
+result = SourceMonitor::Health::SourceHealthCheck.new(source: source).call
+result.success?  # => true/false
+result.log       # => HealthCheckLog record
+result.error     # => exception if failed
+```
+
+Features:
+- Uses source's custom headers and conditional request headers (ETag, If-Modified-Since)
+- HTTP 200-399 is considered successful
+- Creates a HealthCheckLog record regardless of outcome
+- Does NOT update health status (that's the monitor's job)
+
+## SourceHealthReset
+
+Manually resets all health state to defaults:
+
+```ruby
+SourceMonitor::Health::SourceHealthReset.call(source: source)
+```
+
+Resets:
+- `health_status` -> "healthy"
+- `auto_paused_at`, `auto_paused_until` -> nil
+- `rolling_success_rate` -> nil
+- `failure_count` -> 0
+- `last_error`, `last_error_at` -> nil
+- `backoff_until` -> nil
+- `fetch_status` -> "idle"
+- `fetch_retry_attempt` -> 0
+- `fetch_circuit_opened_at`, `fetch_circuit_until` -> nil
+- `next_fetch_at` -> calculated from fetch_interval_minutes
+
+Uses `with_lock` for concurrency safety.
+
+## ImportSourceHealthCheck
+
+Lightweight health check for import candidates (no Source record needed):
+
+```ruby
+result = Health::ImportSourceHealthCheck.new(feed_url: url).call
+result.status        # => "healthy" or "unhealthy"
+result.error_message # => nil or error description
+result.http_status   # => HTTP status code
+```
+
+## Adding a New Health Rule
+
+To add a new condition that affects health status:
+
+### Step 1: Define the Rule
+
+Add a method to `SourceHealthMonitor`:
+
+```ruby
+def my_custom_condition?(logs)
+  # Evaluate logs or source state
+  # Return true if condition is met
+end
+```
+
+### Step 2: Integrate into Status Determination
+
+Add the condition to `determine_status`:
+
+```ruby
+def determine_status(rate, auto_paused_until, logs)
+  if auto_paused_active?(auto_paused_until)
+    "auto_paused"
+  elsif my_custom_condition?(logs)  # Add here
+    "my_custom_status"
+  elsif consecutive_failures(logs) >= 3
+    "declining"
+  # ...
+end
+```
+
+### Step 3: Add Configuration
+
+If the rule needs a threshold, add it to `HealthSettings`:
+
+```ruby
+class HealthSettings
+  attr_accessor :my_threshold
+  def reset!
+    @my_threshold = 0.5  # default
+    # ...
+  end
+end
+```
+
+### Step 4: Update Source Model
+
+If adding a new status value, ensure views and helpers handle it.
+
+### Step 5: Write Tests
+
+```ruby
+test "source enters my_custom_status when condition met" do
+  source = create_source!
+  # Create fetch logs that trigger the condition
+  monitor = SourceMonitor::Health::SourceHealthMonitor.new(source: source)
+  monitor.call
+  assert_equal "my_custom_status", source.reload.health_status
+end
+```
+
+## Testing
+
+- Health monitor tests: `test/lib/source_monitor/health/source_health_monitor_test.rb`
+- Health check tests: `test/lib/source_monitor/health/source_health_check_test.rb`
+- Health reset tests: `test/lib/source_monitor/health/source_health_reset_test.rb`
+- Health module tests: `test/lib/source_monitor/health/health_module_test.rb`
+- Health check job tests: `test/jobs/source_monitor/source_health_check_job_test.rb`
+- Controller tests: `test/controllers/source_monitor/source_health_checks_controller_test.rb`
+
+Use `PARALLEL_WORKERS=1` for single test files to avoid PG segfault.
+
+## Checklist
+
+- [ ] New health rules evaluate from `fetch_logs` (rolling window)
+- [ ] Thresholds are configurable via `HealthSettings`
+- [ ] Per-source overrides supported where appropriate
+- [ ] Status transitions only fire when status actually changes
+- [ ] Auto-pause cooldown prevents flapping
+- [ ] Tests cover threshold boundaries and edge cases
+- [ ] Health status values handled in views/helpers
+
+## References
+
+- `lib/source_monitor/health/` -- All health system code
+- `lib/source_monitor/configuration/health_settings.rb` -- Configuration
+- `app/models/source_monitor/source.rb` -- Source health fields
+- `app/models/source_monitor/health_check_log.rb` -- Health check log model
+- `app/jobs/source_monitor/source_health_check_job.rb` -- Background health check
+- `test/lib/source_monitor/health/` -- Health system tests