source_monitor 0.3.0 → 0.3.2

data/.claude/skills/sm-pipeline-stage/SKILL.md

@@ -0,0 +1,254 @@
+---
+name: sm-pipeline-stage
+description: How to add or modify fetch and scrape pipeline stages in SourceMonitor. Use when working on FeedFetcher, EntryProcessor, ItemCreator, completion handlers, or adding new processing steps to the feed ingestion pipeline.
+allowed-tools: Read, Write, Edit, Bash, Glob, Grep
+---
+
+# SourceMonitor Pipeline Stage Development
+
+## Overview
+
+The SourceMonitor fetch pipeline transforms RSS/Atom/JSON feeds into persisted `Item` records. The pipeline has two main phases: **fetching** (HTTP + parsing) and **item processing** (entry parsing + content extraction + persistence).
+
+## Pipeline Architecture
+
+```
+FetchRunner (orchestrator)
+  |
+  +-- AdvisoryLock (PG advisory lock per source)
+  |
+  +-- FeedFetcher (HTTP fetch + parse + process)
+  |     |
+  |     +-- AdaptiveInterval (next_fetch_at calculation)
+  |     +-- SourceUpdater (source record updates + fetch logs)
+  |     +-- EntryProcessor (iterates feed entries)
+  |           |
+  |           +-- ItemCreator (per-entry)
+  |                 |
+  |                 +-- EntryParser (attribute extraction)
+  |                 |     +-- MediaExtraction (enclosures, thumbnails)
+  |                 |
+  |                 +-- ContentExtractor (readability processing)
+  |
+  +-- Completion Handlers (post-fetch)
+        +-- RetentionHandler (prune old items)
+        +-- FollowUpHandler (enqueue scrape jobs)
+        +-- EventPublisher (dispatch callbacks)
+```
+
+## Key Files
+
+| File | Purpose | Lines |
+|------|---------|-------|
+| `lib/source_monitor/fetching/fetch_runner.rb` | Orchestrator: lock, fetch, completion handlers | 142 |
+| `lib/source_monitor/fetching/feed_fetcher.rb` | HTTP request, response routing, error handling | 285 |
+| `lib/source_monitor/fetching/feed_fetcher/adaptive_interval.rb` | Dynamic fetch interval calculation | 141 |
+| `lib/source_monitor/fetching/feed_fetcher/source_updater.rb` | Persists source state + creates fetch logs | 200 |
+| `lib/source_monitor/fetching/feed_fetcher/entry_processor.rb` | Iterates feed entries, calls ItemCreator | 89 |
+| `lib/source_monitor/fetching/completion/retention_handler.rb` | Post-fetch item retention pruning | 30 |
+| `lib/source_monitor/fetching/completion/follow_up_handler.rb` | Enqueues scrape jobs for new items | 37 |
+| `lib/source_monitor/fetching/completion/event_publisher.rb` | Dispatches `after_fetch_completed` event | 22 |
+| `lib/source_monitor/fetching/retry_policy.rb` | Per-error-type retry/circuit-breaker decisions | 85 |
+| `lib/source_monitor/fetching/advisory_lock.rb` | PG advisory lock wrapper | 54 |
+| `lib/source_monitor/items/item_creator.rb` | Find-or-create items by GUID/fingerprint | 174 |
+| `lib/source_monitor/items/item_creator/entry_parser.rb` | Extracts all attributes from feed entries | 294 |
+| `lib/source_monitor/items/item_creator/content_extractor.rb` | Readability-based content processing | 113 |
+| `lib/source_monitor/items/item_creator/entry_parser/media_extraction.rb` | Enclosures, thumbnails, media content | 96 |
+
+## Adding a New Pipeline Stage
+
+### Option 1: Add a Completion Handler
+
+Completion handlers run after every fetch, inside the `FetchRunner`. Best for cross-cutting post-fetch logic.
+
+**Step 1:** Create the handler class:
+
+```ruby
+# lib/source_monitor/fetching/completion/my_handler.rb
+# frozen_string_literal: true
+
+module SourceMonitor
+  module Fetching
+    module Completion
+      class MyHandler
+        def initialize(**deps)
+          # Accept dependencies for testability
+        end
+
+        def call(source:, result:)
+          return unless should_run?(source:, result:)
+          # Your logic here
+        end
+
+        private
+
+        def should_run?(source:, result:)
+          result&.status == :fetched
+        end
+      end
+    end
+  end
+end
+```
+
+**Step 2:** Wire it into `FetchRunner#initialize`:
+
+```ruby
+# In FetchRunner#initialize, add parameter:
+def initialize(source:, ..., my_handler: nil)
+  @my_handler = my_handler || Completion::MyHandler.new
+end
+```
+
+**Step 3:** Call it in `FetchRunner#run` (inside the lock block):
+
+```ruby
+def run
+  lock.with_lock do
+    mark_fetching!
+    result = fetcher_class.new(source: source).call
+    retention_handler.call(source:, result:)
+    follow_up_handler.call(source:, result:)
+    my_handler.call(source:, result:)  # <-- Add here
+    schedule_retry_if_needed(result)
+    mark_complete!(result)
+  end
+  event_publisher.call(source:, result:)
+  result
+end
+```
+
+### Option 2: Add an Entry Processor Hook
+
+Use `SourceMonitor::Events.run_item_processors` to add per-item processing without modifying the pipeline core.
+
+```ruby
+# In an initializer or engine setup:
+SourceMonitor.configure do |config|
+  config.events.on_item_processed do |source:, entry:, result:|
+    # Custom per-item logic
+  end
+end
+```
+
+### Option 3: Add an EntryParser Extension
+
+To extract new fields from feed entries, extend `EntryParser`:
+
+```ruby
+# Add a new extract method to EntryParser
+def extract_my_field
+  return unless entry.respond_to?(:my_field)
+  string_or_nil(entry.my_field)
+end
+```
+
+Then add it to the `parse` method's return hash.
+
+### Option 4: Add a New Retention Strategy
+
+```ruby
+# lib/source_monitor/items/retention_strategies/archive.rb
+module SourceMonitor
+  module Items
+    module RetentionStrategies
+      class Archive
+        def initialize(source:)
+          @source = source
+        end
+
+        def apply(batch:, now: Time.current)
+          # Your archival logic
+          count = 0
+          batch.each do |item|
+            item.update!(archived_at: now)
+            count += 1
+          end
+          count
+        end
+
+        private
+        attr_reader :source
+      end
+    end
+  end
+end
+```
+
+Register in `RetentionPruner::STRATEGY_CLASSES`.
+
+## Data Flow Details
+
+See `reference/` for detailed documentation:
+- `reference/feed-fetcher-architecture.md` -- FeedFetcher module structure
+- `reference/completion-handlers.md` -- Completion handler patterns
+- `reference/entry-processing.md` -- Entry processing pipeline
+
+## Error Handling
+
+The pipeline uses a typed error hierarchy rooted at `FetchError`:
+
+| Error Class | Code | Trigger |
+|-------------|------|---------|
+| `TimeoutError` | `timeout` | Request timeout |
+| `ConnectionError` | `connection` | Connection/SSL failure |
+| `HTTPError` | `http_error` | Non-200/304 HTTP status |
+| `ParsingError` | `parsing` | Feedjira parse failure |
+| `UnexpectedResponseError` | `unexpected_response` | Any other StandardError |
+
+Each error type maps to a `RetryPolicy` with configurable attempts, wait times, and circuit-breaker thresholds.
+
+## Result Types
+
+**FeedFetcher::Result** -- returned from `FeedFetcher#call`:
+- `status` -- `:fetched`, `:not_modified`, or `:failed`
+- `feed` -- parsed Feedjira feed object
+- `response` -- HTTP response
+- `body` -- raw response body
+- `error` -- FetchError (on failure)
+- `item_processing` -- EntryProcessingResult
+- `retry_decision` -- RetryPolicy::Decision
+
+**ItemCreator::Result** -- returned from `ItemCreator.call`:
+- `item` -- the Item record
+- `status` -- `:created` or `:updated`
+- `matched_by` -- `:guid` or `:fingerprint` (for updates)
+
+## Testing
+
+- Test helpers: `create_source!`, `with_inline_jobs`
+- WebMock blocks all external HTTP; stub responses manually
+- Use `PARALLEL_WORKERS=1` for single test files
+- Inject dependencies (client, lock_factory) for isolation
+
+```ruby
+test "processes new feed entries" do
+  source = create_source!(feed_url: "https://example.com/feed.xml")
+  stub_request(:get, source.feed_url).to_return(
+    status: 200,
+    body: File.read("test/fixtures/files/sample_feed.xml")
+  )
+
+  result = SourceMonitor::Fetching::FeedFetcher.new(source: source).call
+
+  assert_equal :fetched, result.status
+  assert result.item_processing.created.positive?
+end
+```
+
+## Checklist
+
+- [ ] New stage follows dependency injection pattern (accept collaborators in initialize)
+- [ ] Stage has a `call(source:, result:)` interface (for completion handlers)
+- [ ] Error handling returns gracefully (don't crash the pipeline)
+- [ ] Instrumentation payload updated if stage adds metrics
+- [ ] Tests cover success, failure, and skip conditions
+- [ ] No N+1 queries (use `includes`/`preload`)
+- [ ] Documented in this skill's reference files
+
+## References
+
+- `lib/source_monitor/fetching/` -- All fetching pipeline code
+- `lib/source_monitor/items/` -- Item creation and retention
+- `test/lib/source_monitor/fetching/` -- Fetching tests
+- `test/lib/source_monitor/items/` -- Item processing tests

data/.claude/skills/sm-pipeline-stage/reference/completion-handlers.md

@@ -0,0 +1,152 @@
+# Completion Handlers
+
+## Overview
+
+Completion handlers are post-fetch processing steps managed by `FetchRunner`. They execute inside the advisory lock (except `EventPublisher`, which runs after the lock is released).
+
+## Execution Order
+
+```
+lock.with_lock do
+  mark_fetching!
+  result = fetcher.call
+  1. RetentionHandler   -- prune old items
+  2. FollowUpHandler    -- enqueue scrape jobs for new items
+  3. schedule_retry     -- re-enqueue on transient failure
+  mark_complete!
+end
+4. EventPublisher       -- dispatch after_fetch_completed callback (outside lock)
+```
+
+## RetentionHandler
+
+**File:** `lib/source_monitor/fetching/completion/retention_handler.rb`
+
+Applies item retention pruning after every fetch.
+
+```ruby
+class RetentionHandler
+  def initialize(pruner: SourceMonitor::Items::RetentionPruner)
+  def call(source:, result:)
+    pruner.call(source:, strategy: SourceMonitor.config.retention.strategy)
+  end
+end
+```
+
+- Delegates to `RetentionPruner` which supports age-based and count-based limits
+- Rescues errors gracefully (logs and returns nil)
+- Strategy comes from global config (`:destroy` or `:soft_delete`)
+
+### RetentionPruner Details
+
+Two pruning modes:
+1. **Age-based** (`items_retention_days`): Removes items older than N days (uses `COALESCE(published_at, created_at)`)
+2. **Count-based** (`max_items`): Keeps only the N most recent items
+
+Strategies:
+- `Destroy` -- calls `item.destroy!` per record
+- `SoftDelete` -- sets `deleted_at` timestamp, adjusts `items_count` counter
+
+Configuration priority: source-level setting > global config.
+
+## FollowUpHandler
+
+**File:** `lib/source_monitor/fetching/completion/follow_up_handler.rb`
+
+Enqueues scrape jobs for newly created items when auto-scraping is enabled.
+
+```ruby
+class FollowUpHandler
+  def initialize(enqueuer_class:, job_class:)
+  def call(source:, result:)
+    return unless should_enqueue?(source:, result:)
+    result.item_processing.created_items.each do |item|
+      next unless item.present? && item.scraped_at.nil?
+      enqueuer_class.enqueue(item:, source:, job_class:, reason: :auto)
+    end
+  end
+end
+```
+
+Guard conditions:
+- Result status must be `:fetched`
+- Source must have `scraping_enabled?` and `auto_scrape?`
+- At least one item was created
+- Item must not already be scraped (`scraped_at.nil?`)
+
+## EventPublisher
+
+**File:** `lib/source_monitor/fetching/completion/event_publisher.rb`
+
+Dispatches the `after_fetch_completed` callback to all registered listeners.
+
+```ruby
+class EventPublisher
+  def initialize(dispatcher: SourceMonitor::Events)
+  def call(source:, result:)
+    dispatcher.after_fetch_completed(source:, result:)
+  end
+end
+```
+
+- Runs **outside** the advisory lock to prevent long-running callbacks from holding the lock
+- The health monitoring system (`SourceMonitor::Health`) registers a callback here
+- Callbacks are registered via `SourceMonitor.config.events.after_fetch_completed(lambda)`
+
+## Adding a New Completion Handler
+
+### Pattern
+
+```ruby
+# lib/source_monitor/fetching/completion/my_handler.rb
+module SourceMonitor
+  module Fetching
+    module Completion
+      class MyHandler
+        def initialize(**deps)
+          @dependency = deps[:dependency] || DefaultDependency
+        end
+
+        def call(source:, result:)
+          return unless should_run?(source:, result:)
+          # Your logic
+        rescue StandardError => error
+          Rails.logger.error("[SourceMonitor] MyHandler failed: #{error.message}")
+          nil
+        end
+
+        private
+
+        attr_reader :dependency
+
+        def should_run?(source:, result:)
+          result&.status == :fetched
+        end
+      end
+    end
+  end
+end
+```
+
+### Wiring
+
+1. Add require in `fetch_runner.rb`
+2. Add parameter to `FetchRunner#initialize` with default instance
+3. Add `attr_reader` for the handler
+4. Call `my_handler.call(source:, result:)` in `#run`
+5. Decide placement: inside lock (for data consistency) or outside (for non-critical work)
+
+### Testing
+
+```ruby
+test "my_handler is called on successful fetch" do
+  handler = Minitest::Mock.new
+  handler.expect(:call, nil, source: source, result: anything)
+
+  runner = FetchRunner.new(source: source, my_handler: handler)
+  stub_successful_fetch(source)
+  runner.run
+
+  handler.verify
+end
+```

data/.claude/skills/sm-pipeline-stage/reference/entry-processing.md

@@ -0,0 +1,191 @@
+# Entry Processing Pipeline
+
+## Overview
+
+The entry processing pipeline transforms raw feed entries into persisted `Item` records. The pipeline was refactored from a 601-line `ItemCreator` monolith into three focused classes.
+
+```
+EntryProcessor (89 lines)
+  |
+  +-- ItemCreator (174 lines) -- find-or-create orchestrator
+        |
+        +-- EntryParser (294 lines) -- attribute extraction
+        |     +-- MediaExtraction (96 lines) -- media-specific extraction
+        |
+        +-- ContentExtractor (113 lines) -- readability processing
+```
+
+## EntryProcessor
+
+**File:** `lib/source_monitor/fetching/feed_fetcher/entry_processor.rb`
+
+Iterates `feed.entries`, calling `ItemCreator.call` for each. Individual entry failures are caught and counted without stopping the batch.
+
+Returns `EntryProcessingResult` with:
+- `created` / `updated` / `failed` counts
+- `items` -- all processed Item records
+- `created_items` / `updated_items` -- separate lists
+- `errors` -- normalized error details for failed entries
+
+## ItemCreator
+
+**File:** `lib/source_monitor/items/item_creator.rb`
+
+Orchestrates finding or creating an Item record for a single feed entry.
+
+### Deduplication Strategy
+
+Items are matched in priority order:
+
+1. **GUID match** (case-insensitive) -- if the entry has a `entry_id`
+2. **Content fingerprint** -- SHA256 of `title + url + content`
+
+```ruby
+def existing_item_for(attributes, raw_guid_present:)
+  if raw_guid_present
+    existing = find_item_by_guid(guid)
+    return [existing, :guid] if existing
+  end
+  if fingerprint.present?
+    existing = find_item_by_fingerprint(fingerprint)
+    return [existing, :fingerprint] if existing
+  end
+  [nil, nil]
+end
+```
+
+### Concurrent Duplicate Handling
+
+When `ActiveRecord::RecordNotUnique` is raised during creation, the code falls back to finding and updating the conflicting record:
+
+```ruby
+def create_new_item(attributes, raw_guid_present:)
+  new_item = source.items.new
+  apply_attributes(new_item, attributes)
+  new_item.save!
+  Result.new(item: new_item, status: :created)
+rescue ActiveRecord::RecordNotUnique
+  handle_concurrent_duplicate(attributes, raw_guid_present:)
+end
+```
+
+### Result Struct
+
+```ruby
+Result = Struct.new(:item, :status, :matched_by) do
+  def created? = status == :created
+  def updated? = status == :updated
+end
+```
+
+## EntryParser
+
+**File:** `lib/source_monitor/items/item_creator/entry_parser.rb`
+
+Extracts all item attributes from a Feedjira entry object. Handles RSS 2.0, Atom, and JSON Feed formats.
+
+### Extracted Fields
+
+| Field | Method | Notes |
+|-------|--------|-------|
+| `guid` | `extract_guid` | `entry_id` preferred; falls back to `id` if not same as URL |
+| `url` | `extract_url` | Tries `url`, `link_nodes` (alternate), `links` |
+| `title` | -- | Direct from entry |
+| `author` | `extract_author` | Single author string |
+| `authors` | `extract_authors` | Aggregates from rss_authors, dc_creators, author_nodes, JSON Feed |
+| `summary` | `extract_summary` | Entry summary/description |
+| `content` | `extract_content` | Tries `content`, `content_encoded`, `summary` |
+| `published_at` | `extract_timestamp` | First of `published`, `updated` |
+| `updated_at_source` | `extract_updated_timestamp` | Entry `updated` field |
+| `categories` | `extract_categories` | From `categories`, `tags`, JSON Feed tags |
+| `tags` | `extract_tags` | Subset of categories |
+| `keywords` | `extract_keywords` | From `media_keywords_raw`, `itunes_keywords_raw` |
+| `enclosures` | `extract_enclosures` | RSS enclosures, Atom links, JSON attachments |
+| `media_thumbnail_url` | `extract_media_thumbnail_url` | Media RSS thumbnails, entry image |
+| `media_content` | `extract_media_content` | Media RSS content nodes |
+| `language` | `extract_language` | Entry or JSON Feed language |
+| `copyright` | `extract_copyright` | Entry or JSON Feed copyright |
+| `comments_url` | `extract_comments_url` | RSS comments element |
+| `comments_count` | `extract_comments_count` | slash:comments or comments_count |
+| `metadata` | `extract_metadata` | Full entry hash under `feedjira_entry` key |
+| `content_fingerprint` | `generate_fingerprint` | SHA256 of title+url+content |
+
+### Feed Format Detection
+
+```ruby
+def json_entry?
+  defined?(Feedjira::Parser::JSONFeedItem) && entry.is_a?(Feedjira::Parser::JSONFeedItem)
+end
+
+def atom_entry?
+  defined?(Feedjira::Parser::AtomEntry) && entry.is_a?(Feedjira::Parser::AtomEntry)
+end
+```
+
+### Helper Methods
+
+- `string_or_nil(value)` -- strips and returns nil for blank strings
+- `sanitize_string_array(values)` -- deduplicates and compacts
+- `split_keywords(value)` -- splits on `,` or `;`
+- `safe_integer(value)` -- safe Integer conversion
+- `normalize_metadata(value)` -- JSON round-trip for serializable hash
+
+## ContentExtractor
+
+**File:** `lib/source_monitor/items/item_creator/content_extractor.rb`
+
+Processes HTML content through readability parsing when enabled on the source.
+
+### Processing Flow
+
+```
+process_feed_content(raw_content, title:)
+  -> should_process_feed_content?(raw_content)
+     -> source.feed_content_readability_enabled?
+     -> raw_content.present?
+     -> html_fragment?(raw_content)
+  -> wrap_content_for_readability(raw_content, title:)
+     -> builds full HTML document with title
+  -> ReadabilityParser.new.parse(html:, readability:)
+  -> build_feed_content_metadata(result:, raw_content:, processed_content:)
+  -> returns [processed_content, metadata]
+```
+
+### Guard Conditions
+
+Content processing only runs when:
+1. Source has `feed_content_readability_enabled?`
+2. Content is present (not blank)
+3. Content looks like HTML (`html_fragment?` checks for `<tag` pattern)
+
+### Metadata
+
+Processing metadata is stored under `feed_content_processing` key:
+- `strategy` -- always "readability"
+- `status` -- parser result status
+- `applied` -- whether processed content was used
+- `changed` -- whether content differs from raw
+- `readability_text_length` -- extracted text length
+- `title` -- extracted title
+
+## MediaExtraction
+
+**File:** `lib/source_monitor/items/item_creator/entry_parser/media_extraction.rb`
+
+Mixed into `EntryParser` to handle media-specific fields.
+
+### Enclosure Sources
+
+| Format | Source | Key Fields |
+|--------|--------|------------|
+| RSS 2.0 | `enclosure_nodes` | url, type, length |
+| Atom | `link_nodes` with `rel="enclosure"` | url, type, length |
+| JSON Feed | `json["attachments"]` | url, mime_type, size_in_bytes, duration |
+
+### Media Content
+
+From Media RSS `media_content_nodes`: url, type, medium, height, width, file_size, duration, expression.
+
+### Thumbnails
+
+Priority: `media_thumbnail_nodes` first, then `entry.image`.