source_monitor 0.3.0 → 0.3.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: '0794aede84cb7a711fd2a57e1ecf83a54a45167e05aa6523f01726a148d0882e'
-  data.tar.gz: 006314ffd72138c4ce3b0e2edfa3aa8272bf43da74252fce6965198eeb9c1f5f
+  metadata.gz: b8348a9154458260c8b6a3c3dfdeea65e8327b9d7a48dd6c1a2fbe36f9f1bf46
+  data.tar.gz: 2d56c477bc1191f4f505645f9f9f23fceaa1f15d1fd065e5032ac2d40ae3c0d3
 SHA512:
-  metadata.gz: ed74136275e5c4e5a353660b6eb5ccc069cdcb6bf96b8dd052ad5ef77da1de21fe42e0a2e06b07ed4fc8014b094c59c6c7b0e2782756d0abbd086fd8a3a65795
-  data.tar.gz: c359365443bc9eb0f598ad06f9c55c0b82898439d560b3e67a54cd5277131bba53f85a12c9c5740cefa585172bd25930a79478e382fd031484bf9924f497e4e0
+  metadata.gz: 5f875810a4a5e53aae8ec0609e0ab86a3a97ba8661826de6dcd8cf3abd3091f3e4838d8c301ed0f962da0c5849123f8a1fa7c2614e5341126753a8556a0d6fce
+  data.tar.gz: 50a7007476a9b989af9bfe989bc758a9febbc80478c541995044e1a6fd55fe54736520f643ca00845de57a4d5fc5ccfe1d83dd7994affff7530a3cd692f6fb96

data/.claude/skills/sm-architecture/SKILL.md

@@ -0,0 +1,233 @@
+---
+name: sm-architecture
+description: Provides SourceMonitor engine architecture context. Use when working with engine internals, lib/ module structure, autoload organization, configuration DSL, pipelines, or any structural/organizational code in the source_monitor namespace.
+allowed-tools: Read, Glob, Grep
+user-invocable: false
+---
+
+# SourceMonitor Architecture
+
+## Overview
+
+SourceMonitor is a Rails 8 mountable engine (`SourceMonitor::Engine`). Code is split between:
+- **`app/`** -- Rails conventions (models, controllers, views, jobs, concerns)
+- **`lib/source_monitor/`** -- Domain logic, configuration, pipelines, utilities
+
+The engine uses **Ruby autoload** (not Zeitwerk) for `lib/` modules, with explicit `require` only for critical boot-time modules.
+
+## Boot Sequence
+
+`lib/source_monitor.rb` loads in this order:
+
+1. **Optional gems** (rescue LoadError): `solid_queue`, `solid_cable`, `turbo-rails`, `ransack`
+2. **Table name prefix** setup via `redefine_method`
+3. **Explicit requires** (11 files -- must load at boot):
+   - `version`, `engine`, `configuration`, `model_extensions`
+   - `events`, `instrumentation`, `metrics`
+   - `health`, `realtime`, `feedjira_extensions`
+4. **Autoload declarations** (71 modules) organized by namespace
+
+## Engine Configuration
+
+`SourceMonitor::Engine` (`lib/source_monitor/engine.rb`):
+- `isolate_namespace SourceMonitor`
+- Table name prefix from `config.models.table_name_prefix`
+- Initializers: assets, metrics subscribers, dashboard streams, jobs/Solid Queue setup
+
+## Module Tree
+
+```
+SourceMonitor (top-level)
+|-- HTTP                    # Faraday HTTP client factory
+|-- Scheduler               # Fetch scheduling coordinator
+|-- Assets                  # Asset path helpers
+|
+|-- Analytics/              # Dashboard analytics queries
+|   |-- SourceFetchIntervalDistribution
+|   |-- SourceActivityRates
+|   |-- SourcesIndexMetrics
+|
+|-- Dashboard/              # Dashboard UI support
+|   |-- QuickAction, QuickActionsPresenter
+|   |-- RecentActivity, RecentActivityPresenter
+|   |-- Queries, TurboBroadcaster
+|   |-- UpcomingFetchSchedule
+|
+|-- Fetching/               # Feed fetch pipeline
+|   |-- FeedFetcher          # Main orchestrator
+|   |   |-- AdaptiveInterval # Interval calculation
+|   |   |-- SourceUpdater    # Source state updates
+|   |   |-- EntryProcessor   # Entry iteration + item creation
+|   |-- FetchRunner          # Job-level fetch coordinator
+|   |-- RetryPolicy          # Retry/circuit-breaker decisions
+|   |-- StalledFetchReconciler
+|   |-- AdvisoryLock         # PG advisory locks
+|   |-- FetchError (+ subclasses)
+|
+|-- Items/                  # Item management
+|   |-- ItemCreator          # Create/update items from entries
+|   |   |-- EntryParser      # Parse feed entries to attributes
+|   |   |-- ContentExtractor # Process content through readability
+|   |-- RetentionPruner      # Age/count-based item cleanup
+|   |-- RetentionStrategies/ # Destroy vs SoftDelete
+|
+|-- ImportSessions/         # OPML import support
+|   |-- EntryNormalizer
+|   |-- HealthCheckBroadcaster
+|
+|-- Jobs/                   # Job infrastructure
+|   |-- CleanupOptions
+|   |-- Visibility           # Queue visibility setup
+|   |-- SolidQueueMetrics
+|   |-- FetchFailureSubscriber
+|
+|-- Logs/                   # Unified log system
+|   |-- EntrySync            # Sync log records to LogEntry
+|   |-- FilterSet, Query, TablePresenter
+|
+|-- Models/                 # Model concerns
+|   |-- Sanitizable          # String/hash sanitization
+|   |-- UrlNormalizable      # URL normalization
+|
+|-- Scrapers/               # Content scraping adapters
+|   |-- Base                 # Scraper interface
+|   |-- Readability          # Default readability adapter
+|   |-- Fetchers/HttpFetcher
+|   |-- Parsers/ReadabilityParser
+|
+|-- Scraping/               # Scraping orchestration
+|   |-- Enqueuer, Scheduler
+|   |-- ItemScraper (+ AdapterResolver, Persistence)
+|   |-- BulkSourceScraper, BulkResultPresenter
+|   |-- State
+|
+|-- Configuration/          # Configuration DSL (12 settings files)
+|   |-- HTTPSettings, FetchingSettings, HealthSettings
+|   |-- ScrapingSettings, RealtimeSettings, RetentionSettings
+|   |-- AuthenticationSettings, ScraperRegistry
+|   |-- Events, ValidationDefinition
+|   |-- ModelDefinition, Models
+|
+|-- Security/               # Security layer
+|   |-- ParameterSanitizer
+|   |-- Authentication
+|
+|-- Setup/                  # Install/setup wizard
+|   |-- CLI, Workflow, Requirements
+|   |-- Detectors, DependencyChecker
+|   |-- GemfileEditor, BundleInstaller, NodeInstaller
+|   |-- InstallGenerator, MigrationInstaller, InitializerPatcher
+|   |-- Verification/ (Result, Runner, Printer, etc.)
+|
+|-- Pagination/Paginator
+|-- Release/ (Changelog, Runner)
+|-- Sources/ (Params, TurboStreamPresenter)
+|-- TurboStreams/StreamResponder
+```
+
+## Key Architectural Patterns
+
+### 1. Configuration DSL
+
+The `Configuration` class composes 12 settings objects:
+
+```ruby
+SourceMonitor.configure do |config|
+  config.http.timeout = 30
+  config.fetching.min_interval_minutes = 5
+  config.health.auto_pause_threshold = 0.3
+  config.retention.strategy = :soft_delete
+  config.scraping.concurrency = 3
+  config.models.table_name_prefix = "sm_"
+end
+```
+
+Each settings class is a standalone PORO with defaults. Configuration is resettable via `reset_configuration!`.
+
+### 2. ModelExtensions Registry
+
+Models register themselves at class load time:
+```ruby
+SourceMonitor::ModelExtensions.register(self, :source)
+```
+
+This enables:
+- Dynamic table name prefixing
+- Host-app concern injection
+- Host-app validation injection
+- Full reload on config change
+
+### 3. Event System
+
+Three lifecycle events dispatched through `SourceMonitor::Events`:
+
+| Event | Fired When | Payload |
+|-------|-----------|---------|
+| `after_item_created` | New item saved | ItemCreatedEvent |
+| `after_item_scraped` | Scrape completed | ItemScrapedEvent |
+| `after_fetch_completed` | Fetch finished | FetchCompletedEvent |
+
+Plus `item_processors` -- callbacks run for every item (created or updated).
+
+Events are registered via `config.events` and dispatched with error isolation per handler.
+
+### 4. Instrumentation (ActiveSupport::Notifications)
+
+| Event | Purpose |
+|-------|---------|
+| `source_monitor.fetch.start` | Fetch beginning |
+| `source_monitor.fetch.finish` | Fetch completed |
+| `source_monitor.items.duplicate` | Duplicate item detected |
+| `source_monitor.items.retention` | Retention pruning |
+
+`Metrics` module subscribes to these and maintains counters/gauges.
+
+### 5. Pipeline Architecture
+
+**Fetch Pipeline:**
+```
+FetchRunner
+  -> AdvisoryLock (PG lock per source)
+    -> FeedFetcher.call
+      -> HTTP request (Faraday)
+      -> Parse feed (Feedjira)
+      -> EntryProcessor.process_feed_entries
+        -> ItemCreator.call (per entry)
+          -> EntryParser.parse
+          -> ContentExtractor.process_feed_content
+        -> Events.run_item_processors
+        -> Events.after_item_created
+      -> SourceUpdater.update_source_for_success
+        -> AdaptiveInterval.apply_adaptive_interval!
+      -> SourceUpdater.create_fetch_log
+      -> Events.after_fetch_completed
+    -> Completion handlers (retention, follow-up scraping)
+```
+
+**Scrape Pipeline:**
+```
+Scraping::Enqueuer
+  -> ItemScraper
+    -> AdapterResolver (select scraper)
+    -> Scrapers::Base subclass
+      -> Fetchers::HttpFetcher
+      -> Parsers::ReadabilityParser
+    -> Persistence (save to ItemContent)
+    -> Events.after_item_scraped
+```
+
+### 6. Health Monitoring
+
+Health module hooks into `after_fetch_completed`:
+- `SourceHealthMonitor` -- calculates rolling success rate
+- `SourceHealthCheck` -- HTTP health probe
+- Auto-pause sources below threshold
+- `SourceHealthReset` -- manual health reset
+
+## References
+
+- [Module Map](reference/module-map.md) -- Full module tree with responsibilities
+- [Extraction Patterns](reference/extraction-patterns.md) -- Refactoring patterns from Phase 3/4
+- Main entry: `lib/source_monitor.rb`
+- Engine: `lib/source_monitor/engine.rb`
+- Configuration: `lib/source_monitor/configuration.rb` + `lib/source_monitor/configuration/*.rb`

data/.claude/skills/sm-architecture/reference/extraction-patterns.md

@@ -0,0 +1,192 @@
+# SourceMonitor Extraction Patterns
+
+Patterns used during Phase 3 and Phase 4 refactoring to decompose large classes into focused sub-modules.
+
+## Pattern 1: Sub-Module Extraction (FeedFetcher)
+
+**Before:** `FeedFetcher` was 627 lines handling HTTP requests, response parsing, source state updates, adaptive interval calculation, and entry processing.
+
+**After:** 285 lines in the main class + 3 sub-modules.
+
+### Structure
+
+```
+lib/source_monitor/fetching/
+  feed_fetcher.rb                    # 285 lines - orchestrator
+  feed_fetcher/
+    adaptive_interval.rb             # Interval calculation logic
+    source_updater.rb                # Source state updates + fetch log creation
+    entry_processor.rb               # Feed entry iteration + ItemCreator calls
+```
+
+### Technique
+
+1. **Create sub-directory** matching the parent class name
+2. **Extract cohesive responsibilities** into separate classes
+3. **Pass dependencies via constructor** (source, adaptive_interval)
+4. **Lazy accessor pattern** in parent:
+   ```ruby
+   def source_updater
+     @source_updater ||= SourceUpdater.new(source: source, adaptive_interval: adaptive_interval)
+   end
+   ```
+5. **Forwarding methods** for backward compatibility with tests:
+   ```ruby
+   def process_feed_entries(feed) = entry_processor.process_feed_entries(feed)
+   def jitter_offset(interval_seconds) = adaptive_interval.jitter_offset(interval_seconds)
+   ```
+6. **Require in parent file** (not autoloaded -- explicit require):
+   ```ruby
+   require "source_monitor/fetching/feed_fetcher/adaptive_interval"
+   require "source_monitor/fetching/feed_fetcher/source_updater"
+   require "source_monitor/fetching/feed_fetcher/entry_processor"
+   ```
+
+### Key Design Decisions
+
+- `AdaptiveInterval` is a pure calculator -- no side effects, receives source for reading config
+- `SourceUpdater` handles all `source.update!` calls and fetch log creation
+- `EntryProcessor` iterates entries and fires events (item_processors, after_item_created)
+- Parent `FeedFetcher` remains the public API (`#call`) and coordinates the pipeline
+
+---
+
+## Pattern 2: Configuration Decomposition
+
+**Before:** `Configuration` was 655 lines with all settings inline.
+
+**After:** 87 lines composing 12 standalone settings objects.
+
+### Structure
+
+```
+lib/source_monitor/configuration.rb          # 87 lines - composer
+lib/source_monitor/configuration/
+  http_settings.rb
+  fetching_settings.rb
+  health_settings.rb
+  scraping_settings.rb
+  realtime_settings.rb
+  retention_settings.rb
+  authentication_settings.rb
+  scraper_registry.rb
+  events.rb
+  validation_definition.rb
+  model_definition.rb
+  models.rb
+```
+
+### Technique
+
+1. **One settings class per domain** (HTTP, fetching, health, etc.)
+2. **Composition via attr_reader** in parent:
+   ```ruby
+   attr_reader :http, :scrapers, :retention, :events, :models,
+               :realtime, :fetching, :health, :authentication, :scraping
+   ```
+3. **Initialize all in constructor**:
+   ```ruby
+   def initialize
+     @http = HTTPSettings.new
+     @fetching = FetchingSettings.new
+     # ...
+   end
+   ```
+4. **Each settings class is a PORO** with `attr_accessor` and sensible defaults
+5. **Explicit require** (not autoloaded) since Configuration is boot-critical
+
+### Key Design Decisions
+
+- Settings objects are simple POROs, not ActiveModel objects
+- No validation at settings level -- validated at usage point
+- Host app accesses via `config.http.timeout = 30` (dot-chain)
+- Reset via `@config = Configuration.new` (new object, not clearing fields)
+
+---
+
+## Pattern 3: Controller Concern Extraction (ImportSessionsController)
+
+**Before:** `ImportSessionsController` was 792 lines with wizard logic, health checks, and OPML parsing.
+
+**After:** 295 lines + 4 concerns.
+
+### Structure
+
+```
+app/controllers/source_monitor/import_sessions_controller.rb   # 295 lines
+app/controllers/concerns/source_monitor/import_sessions/
+  step_navigation.rb          # Wizard step logic
+  health_checking.rb          # Health check actions
+  source_selection.rb         # Source selection/deselection
+  import_execution.rb         # Final import execution
+```
+
+### Technique
+
+1. **Group by wizard step/feature** -- each concern handles a coherent set of actions
+2. **Include in controller**:
+   ```ruby
+   include ImportSessions::StepNavigation
+   include ImportSessions::HealthChecking
+   ```
+3. **Share state via controller methods** (e.g., `@import_session`, `current_user`)
+4. **Before-action filters** stay in main controller for clarity
+
+---
+
+## Pattern 4: Processor Extraction (ItemCreator)
+
+**Before:** `ItemCreator` was 601 lines handling entry parsing, content extraction, readability processing, and item persistence.
+
+**After:** 174 lines + EntryParser (390 lines) + ContentExtractor (113 lines).
+
+### Structure
+
+```
+lib/source_monitor/items/
+  item_creator.rb                        # 174 lines - orchestrator
+  item_creator/
+    entry_parser.rb                      # 390 lines - parse feed entries
+    entry_parser/media_extraction.rb     # Media parsing concern
+    content_extractor.rb                 # 113 lines - readability processing
+```
+
+### Technique
+
+1. **EntryParser** handles all Feedjira entry field extraction:
+   - URL extraction, timestamp parsing, author normalization
+   - GUID generation, fingerprint calculation
+   - Category/tag/keyword parsing, media extraction
+   - Includes `MediaExtraction` concern for media-specific parsing
+
+2. **ContentExtractor** handles readability content processing:
+   - Decision logic for when to process content
+   - HTML wrapping for readability parser
+   - Result metadata building
+
+3. **Parent ItemCreator** remains the public API (`#call`, `.call`) and handles:
+   - Duplicate detection (by GUID or fingerprint)
+   - Create vs update decision
+   - Concurrent duplicate handling (rescue RecordNotUnique)
+
+4. **Forwarding methods** for backward compatibility (same as FeedFetcher pattern)
+
+---
+
+## Common Principles Across All Extractions
+
+1. **Public API stays on the parent** -- callers don't need to change
+2. **Backward-compatible forwarding** -- old test callsites keep working
+3. **Constructor injection** -- dependencies passed in, not looked up globally
+4. **Lazy accessors** -- sub-modules created on first use
+5. **Explicit require** for sub-modules (not autoloaded) since parent requires them
+6. **Cohesion over size** -- extract by responsibility, not arbitrary line count
+7. **No inheritance** -- composition via delegation, not subclassing
+
+## When to Apply
+
+Use sub-module extraction when a class has:
+- 3+ distinct responsibilities that can be named
+- Methods that cluster into groups with different collaborators
+- Test files that are hard to navigate due to mixed concerns
+- A clear "orchestrator" role that coordinates the extracted pieces

data/.claude/skills/sm-architecture/reference/module-map.md

@@ -0,0 +1,194 @@
+# SourceMonitor Module Map
+
+Complete module tree with each module's responsibility.
+
+## Top-Level Modules (Explicit Require)
+
+| Module | File | Responsibility |
+|--------|------|----------------|
+| `SourceMonitor` | `lib/source_monitor.rb` | Namespace root, configure/reset API, autoload declarations |
+| `Engine` | `lib/source_monitor/engine.rb` | Rails engine setup, isolate_namespace, initializers |
+| `Configuration` | `lib/source_monitor/configuration.rb` | Central config object, composes 12 settings objects |
+| `ModelExtensions` | `lib/source_monitor/model_extensions.rb` | Dynamic table names, concern/validation injection |
+| `Events` | `lib/source_monitor/events.rb` | Lifecycle event dispatch (item created, scraped, fetch completed) |
+| `Instrumentation` | `lib/source_monitor/instrumentation.rb` | ActiveSupport::Notifications wrapper |
+| `Metrics` | `lib/source_monitor/metrics.rb` | Counter/gauge tracking, notification subscribers |
+| `Health` | `lib/source_monitor/health.rb` | Health monitoring setup, fetch callback registration |
+| `Realtime` | `lib/source_monitor/realtime.rb` | ActionCable/Turbo Streams adapter and broadcaster setup |
+| `FeedJiraExtensions` | `lib/source_monitor/feedjira_extensions.rb` | Feedjira monkey-patches/extensions |
+
+## Autoloaded Modules
+
+### SourceMonitor (Root Level)
+
+| Module | File | Responsibility |
+|--------|------|----------------|
+| `HTTP` | `http.rb` | Faraday client factory with configurable timeouts, user-agent, headers |
+| `Scheduler` | `scheduler.rb` | Coordinates scheduled fetch jobs |
+| `Assets` | `assets.rb` | Asset path resolution helpers |
+
+### Analytics
+
+| Module | File | Responsibility |
+|--------|------|----------------|
+| `SourceFetchIntervalDistribution` | `analytics/source_fetch_interval_distribution.rb` | Distribution stats for fetch intervals |
+| `SourceActivityRates` | `analytics/source_activity_rates.rb` | Item creation rates per source |
+| `SourcesIndexMetrics` | `analytics/sources_index_metrics.rb` | Aggregate metrics for sources index |
+
+### Dashboard
+
+| Module | File | Responsibility |
+|--------|------|----------------|
+| `QuickAction` | `dashboard/quick_action.rb` | Quick action data object |
+| `QuickActionsPresenter` | `dashboard/quick_actions_presenter.rb` | Format quick actions for view |
+| `RecentActivity` | `dashboard/recent_activity.rb` | Recent activity query |
+| `RecentActivityPresenter` | `dashboard/recent_activity_presenter.rb` | Format activity for view |
+| `Queries` | `dashboard/queries.rb` | Dashboard aggregate queries |
+| `TurboBroadcaster` | `dashboard/turbo_broadcaster.rb` | Broadcast dashboard updates |
+| `UpcomingFetchSchedule` | `dashboard/upcoming_fetch_schedule.rb` | Next-fetch schedule display |
+
+### Fetching (Feed Fetch Pipeline)
+
+| Module | File | Responsibility |
+|--------|------|----------------|
+| `FeedFetcher` | `fetching/feed_fetcher.rb` | Main fetch orchestrator: request, parse, process entries, update source |
+| `FeedFetcher::AdaptiveInterval` | `fetching/feed_fetcher/adaptive_interval.rb` | Compute next fetch interval based on content changes |
+| `FeedFetcher::SourceUpdater` | `fetching/feed_fetcher/source_updater.rb` | Update source record after fetch (success/failure/not-modified) |
+| `FeedFetcher::EntryProcessor` | `fetching/feed_fetcher/entry_processor.rb` | Iterate feed entries, call ItemCreator, fire events |
+| `FetchRunner` | `fetching/fetch_runner.rb` | Job-level coordinator: acquire lock, run FeedFetcher, handle completion |
+| `RetryPolicy` | `fetching/retry_policy.rb` | Retry/circuit-breaker decision logic |
+| `StalledFetchReconciler` | `fetching/stalled_fetch_reconciler.rb` | Reset sources stuck in "fetching" status |
+| `AdvisoryLock` | `fetching/advisory_lock.rb` | PostgreSQL advisory lock wrapper |
+| `FetchError` | `fetching/fetch_error.rb` | Error hierarchy (TimeoutError, ConnectionError, HTTPError, ParsingError, UnexpectedResponseError) |
+
+### Items
+
+| Module | File | Responsibility |
+|--------|------|----------------|
+| `ItemCreator` | `items/item_creator.rb` | Create or update Item from feed entry |
+| `ItemCreator::EntryParser` | `items/item_creator/entry_parser.rb` | Parse Feedjira entry into attribute hash |
+| `ItemCreator::ContentExtractor` | `items/item_creator/content_extractor.rb` | Process content through readability parser |
+| `RetentionPruner` | `items/retention_pruner.rb` | Prune items by age/count per source |
+| `RetentionStrategies` | `items/retention_strategies.rb` | Strategy pattern for retention |
+| `RetentionStrategies::Destroy` | `items/retention_strategies/destroy.rb` | Hard-delete retention strategy |
+| `RetentionStrategies::SoftDelete` | `items/retention_strategies/soft_delete.rb` | Soft-delete retention strategy |
+
+### ImportSessions
+
+| Module | File | Responsibility |
+|--------|------|----------------|
+| `EntryNormalizer` | `import_sessions/entry_normalizer.rb` | Normalize OPML entries to standard format |
+| `HealthCheckBroadcaster` | `import_sessions/health_check_broadcaster.rb` | Broadcast health check progress via Turbo Streams |
+
+### Jobs
+
+| Module | File | Responsibility |
+|--------|------|----------------|
+| `CleanupOptions` | `jobs/cleanup_options.rb` | Options for job cleanup tasks |
+| `Visibility` | `jobs/visibility.rb` | Configure queue visibility for Solid Queue |
+| `SolidQueueMetrics` | `jobs/solid_queue_metrics.rb` | Extract metrics from Solid Queue tables |
+| `FetchFailureSubscriber` | `jobs/fetch_failure_subscriber.rb` | ActiveJob error subscriber for fetch failures |
+
+### Logs
+
+| Module | File | Responsibility |
+|--------|------|----------------|
+| `EntrySync` | `logs/entry_sync.rb` | Sync FetchLog/ScrapeLog/HealthCheckLog to unified LogEntry |
+| `FilterSet` | `logs/filter_set.rb` | Log filtering parameters |
+| `Query` | `logs/query.rb` | Log query builder |
+| `TablePresenter` | `logs/table_presenter.rb` | Format log entries for table display |
+
+### Models (Shared Concerns)
+
+| Module | File | Responsibility |
+|--------|------|----------------|
+| `Sanitizable` | `models/sanitizable.rb` | `sanitizes_string_attributes`, `sanitizes_hash_attributes` class methods |
+| `UrlNormalizable` | `models/url_normalizable.rb` | `normalizes_urls`, `validates_url_format` class methods |
+
+### Scrapers (Scraper Adapters)
+
+| Module | File | Responsibility |
+|--------|------|----------------|
+| `Base` | `scrapers/base.rb` | Abstract scraper interface |
+| `Readability` | `scrapers/readability.rb` | Default readability-based scraper |
+| `Fetchers::HttpFetcher` | `scrapers/fetchers/http_fetcher.rb` | HTTP content fetcher for scrapers |
+| `Parsers::ReadabilityParser` | `scrapers/parsers/readability_parser.rb` | Parse HTML to readable content |
+
+### Scraping (Scraping Orchestration)
+
+| Module | File | Responsibility |
+|--------|------|----------------|
+| `Enqueuer` | `scraping/enqueuer.rb` | Queue scrape jobs for items |
+| `Scheduler` | `scraping/scheduler.rb` | Schedule scraping across sources |
+| `ItemScraper` | `scraping/item_scraper.rb` | Scrape a single item |
+| `ItemScraper::AdapterResolver` | `scraping/item_scraper/adapter_resolver.rb` | Select scraper adapter for a source |
+| `ItemScraper::Persistence` | `scraping/item_scraper/persistence.rb` | Save scrape results to ItemContent |
+| `BulkSourceScraper` | `scraping/bulk_source_scraper.rb` | Scrape all pending items for a source |
+| `BulkResultPresenter` | `scraping/bulk_result_presenter.rb` | Format bulk scrape results |
+| `State` | `scraping/state.rb` | Track scraping state per source |
+
+### Configuration (12 Settings Files)
+
+| Module | File | Responsibility |
+|--------|------|----------------|
+| `HTTPSettings` | `configuration/http_settings.rb` | HTTP timeouts, user-agent, proxy |
+| `FetchingSettings` | `configuration/fetching_settings.rb` | Adaptive interval params, retry config |
+| `HealthSettings` | `configuration/health_settings.rb` | Health check thresholds, auto-pause config |
+| `ScrapingSettings` | `configuration/scraping_settings.rb` | Scraping concurrency, timeouts |
+| `RealtimeSettings` | `configuration/realtime_settings.rb` | ActionCable/Turbo Streams config |
+| `RetentionSettings` | `configuration/retention_settings.rb` | Item retention strategy, defaults |
+| `AuthenticationSettings` | `configuration/authentication_settings.rb` | Auth callbacks for host app |
+| `ScraperRegistry` | `configuration/scraper_registry.rb` | Register custom scraper adapters |
+| `Events` | `configuration/events.rb` | Event callback storage |
+| `ValidationDefinition` | `configuration/validation_definition.rb` | Host-app validation definitions |
+| `ModelDefinition` | `configuration/model_definition.rb` | Per-model extension definitions |
+| `Models` | `configuration/models.rb` | Model registry and table prefix config |
+
+### Health
+
+| Module | File | Responsibility |
+|--------|------|----------------|
+| `SourceHealthMonitor` | `health/source_health_monitor.rb` | Calculate rolling success rate, update health_status |
+| `SourceHealthCheck` | `health/source_health_check.rb` | Perform HTTP health check on a source |
+| `SourceHealthReset` | `health/source_health_reset.rb` | Reset health state for a source |
+| `ImportSourceHealthCheck` | `health/import_source_health_check.rb` | Health check for import session sources |
+
+### Security
+
+| Module | File | Responsibility |
+|--------|------|----------------|
+| `ParameterSanitizer` | `security/parameter_sanitizer.rb` | Sanitize controller parameters |
+| `Authentication` | `security/authentication.rb` | Authentication helper callbacks |
+
+### Setup (Install Wizard)
+
+| Module | File | Responsibility |
+|--------|------|----------------|
+| `CLI` | `setup/cli.rb` | Command-line interface for setup |
+| `Workflow` | `setup/workflow.rb` | Step-by-step setup orchestration |
+| `Requirements` | `setup/requirements.rb` | System requirements checking |
+| `Detectors` | `setup/detectors.rb` | Detect existing config/gems |
+| `DependencyChecker` | `setup/dependency_checker.rb` | Check gem dependencies |
+| `GemfileEditor` | `setup/gemfile_editor.rb` | Edit host app Gemfile |
+| `BundleInstaller` | `setup/bundle_installer.rb` | Run bundle install |
+| `NodeInstaller` | `setup/node_installer.rb` | Install Node.js dependencies |
+| `InstallGenerator` | `setup/install_generator.rb` | Rails generator for install |
+| `MigrationInstaller` | `setup/migration_installer.rb` | Copy and run migrations |
+| `InitializerPatcher` | `setup/initializer_patcher.rb` | Patch host app initializer |
+| `Verification::Result` | `setup/verification/result.rb` | Verification result + summary |
+| `Verification::Runner` | `setup/verification/runner.rb` | Run all verification checks |
+| `Verification::Printer` | `setup/verification/printer.rb` | Print verification results |
+| `Verification::SolidQueueVerifier` | `setup/verification/solid_queue_verifier.rb` | Verify Solid Queue setup |
+| `Verification::ActionCableVerifier` | `setup/verification/action_cable_verifier.rb` | Verify Action Cable setup |
+| `Verification::TelemetryLogger` | `setup/verification/telemetry_logger.rb` | Log setup telemetry |
+
+### Other
+
+| Module | File | Responsibility |
+|--------|------|----------------|
+| `Pagination::Paginator` | `pagination/paginator.rb` | Offset-based pagination helper |
+| `Release::Changelog` | `release/changelog.rb` | Generate changelog from git history |
+| `Release::Runner` | `release/runner.rb` | Coordinate gem release process |
+| `Sources::Params` | `sources/params.rb` | Strong parameter definitions |
+| `Sources::TurboStreamPresenter` | `sources/turbo_stream_presenter.rb` | Source Turbo Stream formatting |
+| `TurboStreams::StreamResponder` | `turbo_streams/stream_responder.rb` | Turbo Stream response builder |