source_monitor 0.7.1 → 0.8.1

data/.vbw-planning/codebase/INDEX.md

@@ -1,86 +0,0 @@
-# Index
-
-Cross-referenced index of key findings across all mapping documents.
-
-## Quick Reference
-
-| Document | Focus | Key Finding |
-|----------|-------|-------------|
-| [STACK.md](STACK.md) | Technology choices | Rails 8.1.1 engine, Ruby 3.4+, PostgreSQL, Solid Queue, Tailwind 3 |
-| [DEPENDENCIES.md](DEPENDENCIES.md) | Dependency analysis | 14 runtime gems, PG-only, optional deps loaded silently |
-| [ARCHITECTURE.md](ARCHITECTURE.md) | System design | 10 domain modules, event-driven, pluggable scrapers |
-| [STRUCTURE.md](STRUCTURE.md) | Directory layout | ~324 Ruby files, 124 tests, 24 migrations |
-| [CONVENTIONS.md](CONVENTIONS.md) | Code style | Rails omakase, frozen strings, Struct-based results |
-| [TESTING.md](TESTING.md) | Test infrastructure | Minitest, parallel, SimpleCov branch coverage, nightly profiling |
-| [CONCERNS.md](CONCERNS.md) | Risks & debt | Large files, PG lock-in, coverage gaps, no default auth |
-| [PATTERNS.md](PATTERNS.md) | Recurring patterns | Service objects, adapter pattern, event callbacks, Turbo Streams |
-
-## Key Entry Points
-
-| Purpose | File | Notes |
-|---------|------|-------|
-| Gem entry point | `lib/source_monitor.rb` | 102+ require statements, module definition |
-| Engine definition | `lib/source_monitor/engine.rb` | Initializers, asset registration |
-| Configuration DSL | `lib/source_monitor/configuration.rb` | 12 nested settings classes |
-| Routes | `config/routes.rb` | 24 lines, RESTful resources |
-| Main model | `app/models/source_monitor/source.rb` | Core domain entity |
-| Dashboard | `app/controllers/source_monitor/dashboard_controller.rb` | Landing page |
-| Fetch pipeline | `lib/source_monitor/fetching/feed_fetcher.rb` | Core data ingestion |
-| Scrape pipeline | `lib/source_monitor/scraping/item_scraper.rb` | Content extraction orchestrator |
-| Scheduler | `lib/source_monitor/scheduler.rb` | Periodic fetch scheduling |
-| JS entry | `app/assets/javascripts/source_monitor/application.js` | Stimulus app setup |
-| CSS entry | `app/assets/stylesheets/source_monitor/application.tailwind.css` | Tailwind input |
-| Test entry | `test/test_helper.rb` | Test infrastructure setup |
-
-## Data Model Reference
-
-| Model | Table | Key Relationships |
-|-------|-------|-------------------|
-| `Source` | `sourcemon_sources` | has_many: items, fetch_logs, scrape_logs, health_check_logs, log_entries |
-| `Item` | `sourcemon_items` | belongs_to: source; has_one: item_content; has_many: scrape_logs, log_entries |
-| `ItemContent` | `sourcemon_item_contents` | belongs_to: item (separate table for large scraped content) |
-| `FetchLog` | `sourcemon_fetch_logs` | belongs_to: source; has_one: log_entry (polymorphic) |
-| `ScrapeLog` | `sourcemon_scrape_logs` | belongs_to: item, source; has_one: log_entry (polymorphic) |
-| `HealthCheckLog` | `sourcemon_health_check_logs` | belongs_to: source; has_one: log_entry (polymorphic) |
-| `LogEntry` | `sourcemon_log_entries` | delegated_type: loggable (FetchLog/ScrapeLog/HealthCheckLog) |
-| `ImportSession` | `sourcemon_import_sessions` | JSONB state for wizard flow |
-| `ImportHistory` | `sourcemon_import_histories` | Records completed imports |
-
-## Job Reference
-
-| Job Class | Queue | Schedule | Purpose |
-|-----------|-------|----------|---------|
-| `ScheduleFetchesJob` | fetch | Recurring | Triggers scheduler to find due sources |
-| `FetchFeedJob` | fetch | On-demand | Fetches one source's feed |
-| `ScrapeItemJob` | scrape | On-demand | Scrapes one item's content |
-| `SourceHealthCheckJob` | fetch | On-demand | Health check for one source |
-| `ImportSessionHealthCheckJob` | fetch | On-demand | Health check during OPML import |
-| `ImportOpmlJob` | fetch | On-demand | Bulk creates sources from OPML |
-| `LogCleanupJob` | fetch | Recurring | Prunes old log entries |
-| `ItemCleanupJob` | fetch | Recurring | Prunes items per retention policy |
-
-## Configuration Surface Area
-
-| Section | Key Settings | Defaults |
-|---------|-------------|----------|
-| Queues | `fetch_queue_name`, `scrape_queue_name`, concurrency | `source_monitor_fetch`, `source_monitor_scrape`, 2 each |
-| HTTP | timeout, retries, user agent, proxy, headers | 15s/5s timeout, 4 retries |
-| Fetching | adaptive interval params, jitter | 5min-24hr, 1.25x increase, 0.75x decrease |
-| Health | window size, thresholds, auto-pause | 20 window, 0.8/0.5/0.2 thresholds |
-| Scraping | max_in_flight, max_bulk_batch | 25, 100 |
-| Retention | days, max_items, strategy | nil (no auto-cleanup), :destroy |
-| Realtime | adapter (solid_cable/redis/async) | solid_cable |
-| Authentication | handlers, current_user_method | nil (no auth by default) |
-| Models | table_name_prefix, concerns, validations | `sourcemon_` |
-
-## Critical Cross-Cutting Concerns
-
-1. **PG-only** (ARCHITECTURE + CONCERNS): `FOR UPDATE SKIP LOCKED` and `NULLS FIRST/LAST` SQL are PostgreSQL-specific. No other DB supported.
-
-2. **No default auth** (ARCHITECTURE + CONCERNS): Engine mounts without authentication unless host app configures it. Import wizard has a `create_guest_user` fallback.
-
-3. **Eager loading** (STRUCTURE + CONCERNS): All 102+ require statements in `lib/source_monitor.rb` load at boot time.
-
-4. **Coverage debt** (TESTING + CONCERNS): `config/coverage_baseline.json` lists 2329 lines of known uncovered code, particularly in `FeedFetcher`, `ItemCreator`, `Configuration`, and `Dashboard::Queries`.
-
-5. **Large files** (STRUCTURE + CONCERNS): `FeedFetcher` (627 lines), `Configuration` (655 lines), and `ImportSessionsController` (792 lines) are candidates for extraction.

data/.vbw-planning/codebase/META.md

@@ -1,42 +0,0 @@
-# META
-
-## Mapping Metadata
-
-| Field | Value |
-|-------|-------|
-| mapped_at | 2026-02-09T00:00:00Z |
-| git_hash | 1fb781a35575c2d46bfb8cd96f90c64ddf6ed210 |
-| file_count | 530 |
-| mode | full |
-| monorepo | false |
-| mapping_tier | solo (inline) |
-
-## Documents
-
-| File | Domain | Description |
-|------|--------|-------------|
-| `STACK.md` | Tech Stack | Technology choices, framework versions, build pipeline |
-| `DEPENDENCIES.md` | Tech Stack | Runtime, dev, and test dependency analysis with coupling assessment |
-| `ARCHITECTURE.md` | Architecture | System overview, domain modules, data model, job architecture |
-| `STRUCTURE.md` | Architecture | Full directory tree with file counts and organization |
-| `CONVENTIONS.md` | Quality | Naming conventions, code style, frontend patterns |
-| `TESTING.md` | Quality | Test framework, CI pipeline, coverage strategy, profiling |
-| `CONCERNS.md` | Concerns | Technical debt, security risks, performance, operational risks |
-| `INDEX.md` | Synthesis | Cross-referenced index with key findings and quick references |
-| `PATTERNS.md` | Synthesis | 14 recurring patterns across the codebase |
-| `META.md` | Meta | This file -- mapping metadata and document inventory |
-
-## Language Breakdown
-
-| Language | File Count | Notes |
-|----------|-----------|-------|
-| Ruby (.rb) | ~324 | Models, controllers, jobs, lib modules, tests |
-| ERB (.erb) | ~48 | View templates |
-| Markdown (.md) | ~45 | Documentation |
-| YAML (.yml) | ~16 | Configuration files |
-| JavaScript (.js) | ~14 | Stimulus controllers, build entry points |
-| CSS (.css) | ~2 | Tailwind input and build output |
-
-## Project Summary
-
-SourceMonitor is a mountable Rails 8 engine (v0.2.1) that ingests RSS/Atom/JSON feeds, scrapes full article content via pluggable adapters, and provides Solid Queue-powered dashboards for monitoring and remediation. It is distributed as a RubyGem, requires PostgreSQL and Ruby >= 3.4.0, and integrates with host Rails applications via the standard engine mounting pattern with an isolated namespace.

data/.vbw-planning/codebase/PATTERNS.md

@@ -1,262 +0,0 @@
-# Patterns
-
-Recurring patterns observed across the SourceMonitor codebase.
-
-## 1. Service Object Pattern
-
-**Where**: `lib/source_monitor/fetching/`, `lib/source_monitor/scraping/`, `lib/source_monitor/health/`, `lib/source_monitor/items/`
-
-Service objects encapsulate domain operations. They follow a consistent structure:
-
-```ruby
-class SomeService
-  def initialize(source:, **deps)
-    @source = source
-  end
-
-  def call
-    # orchestrate operation
-    # return a Result struct
-  end
-end
-```
-
-**Examples**:
-- `Fetching::FeedFetcher` -- `#call` returns `Result` struct
-- `Scraping::ItemScraper` -- `#call` returns `Result` struct
-- `Health::SourceHealthMonitor` -- `#call` updates source health
-- `Health::SourceHealthCheck` -- `#call` probes source URL
-- `Items::RetentionPruner` -- `#call` prunes old items
-- `Items::ItemCreator` -- `.call(source:, entry:)` class method
-
-## 2. Struct-Based Result Objects
-
-**Where**: Throughout all service objects
-
-Operations return typed `Struct` instances rather than raw hashes or arrays:
-
-```ruby
-Result = Struct.new(:status, :item, :log, :message, :error, keyword_init: true) do
-  def success?
-    status.to_s != "failed"
-  end
-end
-```
-
-**Examples**:
-- `Scraping::ItemScraper::Result` -- `:status, :item, :log, :message, :error`
-- `Scrapers::Base::Result` -- `:status, :html, :content, :metadata`
-- `Fetching::FeedFetcher::Result` -- `:status, :feed, :response, :body, :error, :item_processing, :retry_decision`
-- `Fetching::FeedFetcher::EntryProcessingResult` -- `:created, :updated, :failed, :items, :errors`
-- `Events::ItemCreatedEvent`, `Events::ItemScrapedEvent`, `Events::FetchCompletedEvent`
-- `Setup::Verification::Result` -- verification outcome
-
-## 3. Adapter/Strategy Pattern
-
-**Where**: `lib/source_monitor/scrapers/`, `lib/source_monitor/realtime/`, `lib/source_monitor/items/retention_strategies/`
-
-Pluggable behavior via abstract base class with `#call` contract:
-
-```ruby
-class Scrapers::Base
-  def call
-    raise NotImplementedError
-  end
-end
-```
-
-**Instances**:
-- **Scraper adapters**: `Scrapers::Base` -> `Scrapers::Readability` (registered in `ScraperRegistry`)
-- **Realtime adapters**: `solid_cable`, `redis`, `async` (configured in `RealtimeSettings`)
-- **Retention strategies**: `:destroy`, `:soft_delete` (in `Items::RetentionStrategies/`)
-
-## 4. Event/Callback System
-
-**Where**: `lib/source_monitor/events.rb`, `lib/source_monitor/configuration.rb`
-
-Event-driven communication between engine components:
-
-```ruby
-# Registration
-SourceMonitor.config.events.after_item_created { |event| ... }
-SourceMonitor.config.events.after_item_scraped { |event| ... }
-SourceMonitor.config.events.after_fetch_completed { |event| ... }
-
-# Dispatch
-SourceMonitor::Events.after_item_created(item:, source:, entry:, result:)
-```
-
-- Typed event structs carry context
-- Error isolation: each handler failure is logged, does not stop other handlers
-- Item processor pipeline: `Events.run_item_processors` runs all registered processors
-- Used by `Health` module to register fetch completion callback
-
-## 5. Configuration DSL with Nested Settings Objects
-
-**Where**: `lib/source_monitor/configuration.rb`
-
-Deeply nested configuration with domain-specific settings classes:
-
-```ruby
-SourceMonitor.configure do |config|
-  config.http.timeout = 30
-  config.fetching.min_interval_minutes = 10
-  config.health.window_size = 50
-  config.scrapers.register(:custom, MyCustomScraper)
-  config.models.source.include_concern SomeConcern
-  config.authentication.authenticate_with :authenticate_admin!
-end
-```
-
-**Pattern traits**:
-- Each settings class has `reset!` for test isolation
-- Constants for defaults (e.g., `DEFAULT_QUEUE_NAMESPACE`)
-- Callable values supported (procs/lambdas) for dynamic resolution
-- Validation in setters (e.g., `RealtimeSettings#adapter=` checks `VALID_ADAPTERS`)
-
-## 6. Model Extension System
-
-**Where**: `lib/source_monitor/model_extensions.rb`, `lib/source_monitor/configuration.rb`
-
-Host apps can dynamically inject concerns and validations into engine models:
-
-```ruby
-config.models.source.include_concern "MyApp::SourceExtensions"
-config.models.source.validate :custom_validator
-config.models.source.validate { |record| record.errors.add(:base, "invalid") if ... }
-```
-
-- `ModelExtensions.register(model_class, key)` called in each model class body
-- `ModelExtensions.reload!` re-applies all extensions on configuration change
-- Manages table name prefix assignment
-- Tracks applied concerns/validations to prevent duplicates
-- Removes old extension validations before re-applying
-
-## 7. Turbo Stream Response Pattern
-
-**Where**: `lib/source_monitor/turbo_streams/stream_responder.rb`, controllers
-
-Controllers build Turbo Stream responses via a `StreamResponder` builder:
-
-```ruby
-responder = SourceMonitor::TurboStreams::StreamResponder.new
-presenter = SourceMonitor::Sources::TurboStreamPresenter.new(source:, responder:)
-presenter.render_deletion(metrics:, query:, ...)
-responder.toast(message:, level: :success)
-render turbo_stream: responder.render(view_context)
-```
-
-- Accumulates stream actions as an array
-- Supports toast notifications, redirects, and custom actions
-- Pairs with Stimulus controllers on the frontend
-
-## 8. Defensive Logging Guard
-
-**Where**: All jobs and service objects
-
-Consistent pattern for safe logging:
-
-```ruby
-def log(stage, **extra)
-  return unless defined?(Rails) && Rails.respond_to?(:logger) && Rails.logger
-  Rails.logger.info("[SourceMonitor::...] #{payload.to_json}")
-rescue StandardError
-  nil
-end
-```
-
-This three-part guard (`defined?`, `respond_to?`, truthy check) prevents errors when running outside Rails (e.g., in tests or standalone scripts).
-
-## 9. ActiveSupport::Notifications Instrumentation
-
-**Where**: `lib/source_monitor/instrumentation.rb`, `lib/source_monitor/metrics.rb`
-
-Standard Rails instrumentation pattern:
-
-```ruby
-# Emit events
-SourceMonitor::Instrumentation.fetch_start(payload)
-SourceMonitor::Instrumentation.fetch_finish(payload)
-
-# Subscribe to events
-ActiveSupport::Notifications.subscribe("source_monitor.fetch.finish") do |...|
-  SourceMonitor::Metrics.increment(:fetch_finished_total)
-end
-```
-
-- Events namespaced as `source_monitor.*`
-- Monotonic clock for duration measurement
-- Metrics module aggregates counters and gauges in memory
-
-## 10. Search/Filter with Ransack
-
-**Where**: Models (`ransackable_attributes`, `ransackable_associations`), `SanitizesSearchParams` concern
-
-```ruby
-class Source < ApplicationRecord
-  def self.ransackable_attributes(_auth_object = nil)
-    %w[name feed_url website_url created_at ...]
-  end
-end
-```
-
-- Explicit whitelisting of searchable attributes (required by Ransack 4+)
-- `SanitizesSearchParams` controller concern sanitizes search inputs
-- Used in `SourcesController#index` and `LogsController#index`
-
-## 11. Circuit Breaker / Retry Policy
-
-**Where**: `lib/source_monitor/fetching/retry_policy.rb`, `lib/source_monitor/fetching/feed_fetcher.rb`, `app/jobs/source_monitor/fetch_feed_job.rb`
-
-Fetch failures trigger an escalating retry policy:
-
-1. **Retry with backoff**: Exponential wait, up to N attempts
-2. **Circuit open**: After exhausting retries, block fetches for a cooldown period
-3. **Circuit close**: Scheduler recovers after cooldown expires
-
-State stored on `Source` model: `fetch_retry_attempt`, `fetch_circuit_opened_at`, `fetch_circuit_until`, `backoff_until`.
-
-## 12. Wizard State Machine (OPML Import)
-
-**Where**: `app/models/source_monitor/import_session.rb`, `app/controllers/source_monitor/import_sessions_controller.rb`
-
-Multi-step wizard with explicit step ordering:
-
-```ruby
-STEP_ORDER = %w[upload preview health_check configure confirm].freeze
-```
-
-- State persisted in `ImportSession` model with JSONB columns
-- Each step has dedicated `handle_*_step` and `prepare_*_context` methods
-- Navigation via `next_step`/`previous_step` model methods
-- Step transitions guarded by validation (e.g., "select at least one source")
-
-## 13. Soft Delete Pattern
-
-**Where**: `app/models/source_monitor/item.rb`
-
-Items use soft deletion via `deleted_at` timestamp rather than physical deletion:
-
-```ruby
-scope :active, -> { where(deleted_at: nil) }
-scope :with_deleted, -> { unscope(where: :deleted_at) }
-scope :only_deleted, -> { where.not(deleted_at: nil) }
-
-def soft_delete!(timestamp: Time.current)
-  update_columns(deleted_at: timestamp, updated_at: timestamp)
-  Source.decrement_counter(:items_count, source_id)
-end
-```
-
-No `default_scope` is used (explicitly noted as avoiding anti-pattern).
-
-## 14. Separate Content Table
-
-**Where**: `app/models/source_monitor/item.rb`, `app/models/source_monitor/item_content.rb`
-
-Large scraped content is stored in a separate `ItemContent` model rather than on the `Item` directly:
-
-- Lazy-loaded via `has_one :item_content`
-- Auto-created when content is assigned, auto-destroyed when both fields become blank
-- Delegates `scraped_html` and `scraped_content` to `ItemContent`
-- Prevents bloating the items table with large text columns

data/.vbw-planning/codebase/STACK.md

@@ -1,101 +0,0 @@
-# Tech Stack
-
-## Core Platform
-
-| Layer | Technology | Version |
-|-------|-----------|---------|
-| Language | Ruby | >= 3.4.0 (CI uses 3.4.4) |
-| Framework | Rails | >= 8.0.3, < 9.0 (locked at 8.1.1) |
-| Database | PostgreSQL | 15 (CI service image) |
-| Background Jobs | Solid Queue | >= 0.3, < 3.0 (locked at 1.2.4) |
-| WebSocket/Realtime | Solid Cable | >= 3.0, < 4.0 (locked at 3.0.12) |
-| Frontend Interactivity | Turbo Rails | ~> 2.0 (locked at 2.0.20) |
-| JS Framework | Stimulus (Hotwired) | ^3.2.2 |
-| CSS Framework | Tailwind CSS | ^3.4.10 |
-| Web Server | Puma | 7.1.0 |
-| Asset Pipeline | Propshaft | 1.3.1 |
-
-## Project Type
-
-Mountable Rails 8 Engine gem (`source_monitor.gemspec`), distributed as a RubyGem. The engine uses `isolate_namespace SourceMonitor` and provides its own models, controllers, views, jobs, and frontend assets.
-
-- **Version**: 0.2.1
-- **Required Ruby**: >= 3.4.0
-- **License**: MIT
-
-## Feed Parsing & HTTP
-
-| Purpose | Gem | Version |
-|---------|-----|---------|
-| RSS/Atom/JSON feed parsing | Feedjira | >= 3.2, < 5.0 (locked 4.0.1) |
-| HTTP client | Faraday | ~> 2.9 (locked 2.14.0) |
-| HTTP retry middleware | faraday-retry | ~> 2.2 |
-| HTTP redirect following | faraday-follow_redirects | ~> 0.4 |
-| HTTP gzip compression | faraday-gzip | ~> 3.0 |
-
-## Content Scraping & Parsing
-
-| Purpose | Gem | Version |
-|---------|-----|---------|
-| HTML parsing (fast, C-based) | Nokolexbor | ~> 0.5 (locked 0.6.2) |
-| HTML parsing (standard) | Nokogiri | 1.18.10 (transitive) |
-| Article content extraction | ruby-readability | ~> 0.7 |
-
-## Search & Querying
-
-| Purpose | Gem | Version |
-|---------|-----|---------|
-| Search/filter forms | Ransack | ~> 4.2 (locked 4.4.1) |
-
-## Frontend Build Pipeline
-
-| Tool | Purpose | Version |
-|------|---------|---------|
-| esbuild | JS bundling | ^0.23.0 |
-| Tailwind CSS | Utility-first CSS | ^3.4.10 |
-| PostCSS | CSS processing | ^8.4.45 |
-| Autoprefixer | CSS vendor prefixes | ^10.4.20 |
-| ESLint | JS linting | ^9.11.0 |
-| Stylelint | CSS linting | ^16.8.0 |
-
-Build orchestration via `package.json` scripts:
-- `npm run build` -- builds both CSS (tailwindcss) and JS (esbuild)
-- `cssbundling-rails` (~> 1.4) and `jsbundling-rails` (~> 1.3) bridge npm builds into the Rails asset pipeline
-
-## JS Dependencies (Runtime)
-
-| Package | Purpose |
-|---------|---------|
-| `@hotwired/stimulus` ^3.2.2 | Stimulus controllers for UI interactions |
-| `stimulus-use` ^0.52.0 | Stimulus composable behaviors library |
-
-## Testing Stack
-
-| Tool | Purpose |
-|------|---------|
-| Minitest | Test framework (Rails default) |
-| Capybara | System/integration test driver |
-| Selenium WebDriver | Browser automation for system tests |
-| WebMock | HTTP request stubbing |
-| VCR | HTTP interaction recording/playback |
-| SimpleCov | Code coverage (branch coverage enabled) |
-| test-prof | Test profiling (TagProf, EventProf) |
-| StackProf | Sampling profiler for performance analysis |
-
-## Code Quality & Security
-
-| Tool | Purpose |
-|------|---------|
-| RuboCop (rails-omakase) | Ruby/Rails linting (omakase style) |
-| Brakeman | Static security analysis |
-| ESLint | JavaScript linting |
-| Stylelint | CSS linting |
-
-## CI/CD
-
-- **GitHub Actions** with 5 jobs: `lint`, `security`, `test`, `release_verification`, `profiling` (scheduled nightly)
-- Ruby 3.4.4, Node 20
-- PostgreSQL 15 as service container
-- Diff coverage enforcement via custom `bin/check-diff-coverage`
-- Test profiling guardrails via `bin/check-test-prof-metrics`
-- Parallel test execution (configurable via `SOURCE_MONITOR_TEST_WORKERS`)