source_monitor 0.7.1 → 0.8.1

data/.vbw-planning/ROADMAP.md

@@ -1,53 +0,0 @@
-# Roadmap
-
-## Milestone: aia-ssl-fix
-
-### Phases
-
-1. [x] **AIA Certificate Resolution** -- Fix SSL failures for feeds with missing intermediate certificates by implementing AIA (Authority Information Access) resolution
-2. [x] **Test Performance** -- Reduce test suite runtime from ~133s to ~50s by splitting monolithic test classes, enabling parallelism, reducing log IO, and adopting before_all
-
-### Phase Details
-
-#### Phase 1: AIA Certificate Resolution
-
-**Goal:** Implement automatic AIA intermediate certificate fetching so feeds like netflixtechblog.com (served via Medium/AWS with wrong intermediates) succeed without manual cert configuration.
-
-**Requirements:**
-- REQ-AIA-01: Create AIAResolver module with thread-safe cache and 1-hour TTL
-- REQ-AIA-02: Add cert_store: parameter to HTTP.client for custom cert stores
-- REQ-AIA-03: On Faraday::SSLError, attempt AIA resolution before failing
-- REQ-AIA-04: Best-effort only -- never make things worse (rescue StandardError -> nil)
-
-**Success Criteria:**
-- [ ] AIAResolver.resolve(hostname) fetches leaf cert, extracts AIA URL, downloads intermediate
-- [ ] HTTP.client(cert_store:) accepts and uses custom cert stores
-- [ ] FeedFetcher retries once with AIA-resolved cert store on SSL failure
-- [ ] All existing tests pass (1003+), new tests cover AIA paths
-- [ ] RuboCop zero offenses, Brakeman zero warnings
-
-#### Phase 2: Test Performance
-
-**Goal:** Reduce test suite wall-clock time from ~133s to ~50s through structural optimizations. The 3-agent investigation identified that FeedFetcherTest (71 tests, 84.8s, 64% of total) is a monolithic class that cannot be parallelized, integration tests add 31s, and 95MB of debug logging adds 5-15s.
-
-**Requirements:**
-- REQ-PERF-01: Split FeedFetcherTest into 5+ smaller classes by concern (success paths, error handling, adaptive interval, dirty-check, content fingerprint, utilities)
-- REQ-PERF-02: Set test log level to :warn in test/dummy/config/environments/test.rb (eliminates 95MB log IO)
-- REQ-PERF-03: Tag integration tests (host_install_flow, release_packaging) so they can be excluded during dev with --exclude-pattern
-- REQ-PERF-04: Switch default parallelism from forks to threads (avoids PG segfault, enables splitting benefit)
-- REQ-PERF-05: Adopt before_all/setup_once in top DB-heavy test files (dashboard/queries_test.rb, etc.)
-
-**Success Criteria:**
-- [ ] FeedFetcherTest split into 5+ files, each independently runnable
-- [ ] All 1031+ tests pass with PARALLEL_WORKERS=1 and default workers
-- [ ] Test suite completes in <70s locally (down from 133s)
-- [ ] `bin/rails test --exclude-pattern="**/integration/**"` runs <50s
-- [ ] RuboCop zero offenses, Brakeman zero warnings
-- [ ] No test isolation regressions (parallel runs still green)
-
-### Progress
-
-| Phase | Status | Plans | Completed |
-|-------|--------|-------|-----------|
-| 1. AIA Certificate Resolution | Complete | 3 | 3 |
-| 2. Test Performance | Complete | 4 | 4 |

data/.vbw-planning/SHIPPED.md

@@ -1,63 +0,0 @@
-# Shipped Milestones
-
-## default (2026-02-09 to 2026-02-10)
-
-**Core value:** Drop-in Rails engine for feed monitoring, content scraping, and operational dashboards.
-
-### Metrics
-
-| Metric | Value |
-|--------|-------|
-| Phases | 4 |
-| Plans completed | 14 |
-| Commits | 18 |
-| Requirements satisfied | 15/15 |
-| Test runs | 841 (up from 473) |
-| Coverage | 86.97% line (510 uncovered, down from 2117) |
-
-### Phases
-
-1. Coverage Analysis & Quick Wins (2 plans)
-2. Critical Path Test Coverage (5 plans)
-3. Large File Refactoring (4 plans)
-4. Code Quality & Conventions Cleanup (3 plans)
-
-### Archive
-
-Location: `.vbw-planning/milestones/default/`
-Tag: `milestone/default`
-
----
-
-## upgrade-assurance (2026-02-12 to 2026-02-13)
-
-**Goal:** Give host app developers confidence that gem updates go smoothly -- automated migration detection, upgrade command, config validation, and AI-assisted upgrade guidance.
-
-### Metrics
-
-| Metric | Value |
-|--------|-------|
-| Phases | 3 |
-| Plans completed | 3 |
-| Tasks completed | 14 |
-| Commits | 12 |
-| Requirements satisfied | 5/5 (REQ-26 through REQ-30) |
-| Tests | 1003 (up from 973) |
-
-### Phases
-
-1. Upgrade Command & Migration Verifier (1 plan, 5 tasks)
-2. Configuration Deprecation Framework (1 plan, 4 tasks)
-3. Upgrade Skill & Documentation (1 plan, 5 tasks)
-
-### Key Decisions
-
-- 3 phases: command, config, skill -- each independently valuable
-- `.source_monitor_version` marker file for version tracking
-- Deprecation registry with :warning and :error severities
-- sm-upgrade as a consumer skill (installed by default)
-
-### Archive
-
-Location: `.vbw-planning/milestones/upgrade-assurance/`
-Tag: `milestone/upgrade-assurance`

data/.vbw-planning/STATE.md

@@ -1,27 +0,0 @@
-# State
-
-## Current Position
-
-- **Milestone:** aia-ssl-fix
-- **Phase:** 2 -- Test Performance
-- **Status:** Complete
-- **Progress:** 100%
-
-## Decisions
-
-| Decision | Date | Context |
-|----------|------|---------|
-| Single-phase milestone for AIA fix | 2026-02-17 | Complete plan already validated; no scoping needed |
-| 3 plans with wave parallelism | 2026-02-17 | Plans 01+02 (wave 1, disjoint files), Plan 03 (wave 2, integration) |
-
-## Todos
-
-## Metrics
-
-- **Started:** 2026-02-17
-- **Phases:** 1
-- **Plans:** 3
-- **Tests at start:** 1003
-- **Tests at end:** 1025
-- **Commits:** 4 (f60e9bf, 4c9568a, 9c38bc3, e68a6b0)
-- **Plans completed:** 3/3

data/.vbw-planning/codebase/ARCHITECTURE.md

@@ -1,147 +0,0 @@
-# Architecture
-
-## System Overview
-
-SourceMonitor is a mountable Rails 8 engine that ingests RSS, Atom, and JSON feeds, scrapes full article content, and surfaces Solid Queue-powered dashboards for monitoring and remediation. It is packaged as a RubyGem and mounted into a host Rails application.
-
-## Architecture Pattern
-
-**Mountable Rails Engine** with `isolate_namespace SourceMonitor`. The engine:
-- Provides its own models, controllers, views, jobs, and frontend assets
-- Uses its own `ApplicationRecord`, `ApplicationController`, and `ApplicationJob` base classes
-- Namespaces all database tables under a configurable prefix (default: `sourcemon_`)
-- Registers its own routes, asset paths, and initializers
-
-## Core Domain Modules
-
-### 1. Feed Fetching (`lib/source_monitor/fetching/`)
-The primary data ingestion pipeline:
-- `FeedFetcher` -- Orchestrates HTTP request, feed parsing via Feedjira, item creation, adaptive interval scheduling, and retry policies
-- `FetchRunner` -- Entry point for enqueuing fetch jobs; handles concurrency control
-- `FetchError` hierarchy -- Typed error classes (TimeoutError, ConnectionError, HTTPError, ParsingError)
-- `RetryPolicy` -- Exponential backoff with circuit breaker pattern
-- `StalledFetchReconciler` -- Recovers stalled fetch jobs
-
-### 2. Content Scraping (`lib/source_monitor/scraping/` + `lib/source_monitor/scrapers/`)
-Pluggable content extraction system:
-- `Scrapers::Base` -- Abstract adapter contract; subclasses implement `#call` returning a `Result` struct
-- `Scrapers::Readability` -- Default adapter using ruby-readability
-- `Scraping::ItemScraper` -- Orchestrator that resolves adapter, executes scrape, persists results
-- `Scraping::BulkSourceScraper` -- Batch scraping across all items for a source
-- `Scraping::Enqueuer` -- Manages scrape job queuing with in-flight throttling
-- `Scraping::State` -- Tracks in-flight scrape state via cache/memory
-
-### 3. Health Monitoring (`lib/source_monitor/health/`)
-Source health tracking system:
-- `SourceHealthMonitor` -- Computes health status from recent fetch history (sliding window)
-- `SourceHealthCheck` -- One-off health probe for a source URL
-- `ImportSourceHealthCheck` -- Variant for import wizard health checks
-- `SourceHealthReset` -- Resets health status for a source
-- Configurable thresholds: healthy (0.8), warning (0.5), auto-pause (0.2)
-
-### 4. Scheduling (`lib/source_monitor/scheduler.rb`)
-- `Scheduler` -- Periodic job that finds sources due for fetch using `FOR UPDATE SKIP LOCKED`
-- Adaptive fetch interval algorithm (increase/decrease factors, jitter)
-- Integrates with `StalledFetchReconciler` for recovery
-
-### 5. Event System (`lib/source_monitor/events.rb`)
-- Typed event structs: `ItemCreatedEvent`, `ItemScrapedEvent`, `FetchCompletedEvent`
-- Callback-based dispatch with error isolation per handler
-- Item processor pipeline for custom host-app processing
-- Registration via `SourceMonitor.config.events.after_*` DSL
-
-### 6. Configuration (`lib/source_monitor/configuration.rb`)
-Rich nested configuration object with sub-configs:
-- `HTTPSettings` -- timeout, retries, user agent, proxy, headers
-- `ScraperRegistry` -- pluggable adapter registration
-- `RetentionSettings` -- item retention days, max items, strategy (destroy/soft_delete)
-- `RealtimeSettings` -- adapter selection (solid_cable/redis/async)
-- `FetchingSettings` -- adaptive interval tuning
-- `HealthSettings` -- health window and threshold configuration
-- `AuthenticationSettings` -- pluggable authentication/authorization handlers
-- `ScrapingSettings` -- concurrency limits
-- `Events` -- callback registration
-- `Models` -- concern injection and custom validation registration
-
-### 7. Model Extensions (`lib/source_monitor/model_extensions.rb`)
-Dynamic model customization system:
-- Host apps can inject concerns and validations into engine models at configuration time
-- `ModelExtensions.register(model_class, key)` -- called in each model class body
-- `ModelExtensions.reload!` -- re-applies all extensions (called on configuration change)
-- Manages table name assignment from configurable prefix
-
-### 8. Real-time Broadcasting (`lib/source_monitor/realtime/`)
-- `Realtime::Adapter` -- Configures Action Cable based on selected adapter
-- `Realtime::Broadcaster` -- Broadcasts source/item updates and toast notifications
-- `Dashboard::TurboBroadcaster` -- Wires dashboard stat updates to Turbo Streams
-
-### 9. Setup/Installation System (`lib/source_monitor/setup/`)
-Comprehensive host-app installation workflow:
-- `Setup::CLI` -- Command-line interface for setup
-- `Setup::Workflow` -- Orchestrates multi-step installation
-- `Setup::Requirements` / `Setup::Detectors` -- System requirement checks
-- `Setup::GemfileEditor` / `Setup::BundleInstaller` / `Setup::NodeInstaller` -- Dependency installation
-- `Setup::InstallGenerator` -- Rails generator for migrations, routes, initializer
-- `Setup::Verification::Runner` -- Post-install verification (Solid Queue, Action Cable)
-
-### 10. OPML Import Wizard (`app/controllers/source_monitor/import_sessions_controller.rb`)
-Multi-step wizard for bulk feed import:
-- 5-step flow: upload -> preview -> health_check -> configure -> confirm
-- State persisted in `ImportSession` model with JSONB columns
-- Health check jobs run asynchronously with Turbo Stream progress updates
-- Bulk settings applied to all imported sources
-
-## Data Model
-
-```
-Source (sourcemon_sources)
-  |-- has_many Item (sourcemon_items)
-  |     |-- has_one ItemContent (sourcemon_item_contents) [separate table for large content]
-  |     |-- has_many ScrapeLog (sourcemon_scrape_logs)
-  |     +-- has_many LogEntry (sourcemon_log_entries) [polymorphic]
-  |-- has_many FetchLog (sourcemon_fetch_logs)
-  |-- has_many HealthCheckLog (sourcemon_health_check_logs)
-  +-- has_many LogEntry (sourcemon_log_entries)
-
-LogEntry (sourcemon_log_entries)
-  |-- delegated_type :loggable -> FetchLog | ScrapeLog | HealthCheckLog
-
-ImportSession (sourcemon_import_sessions)
-  +-- JSONB columns for wizard state
-
-ImportHistory (sourcemon_import_histories)
-  +-- Records completed imports
-```
-
-## Job Architecture
-
-All jobs inherit from `SourceMonitor::ApplicationJob` which inherits from the host app's `ApplicationJob` (or `ActiveJob::Base`).
-
-| Job | Queue | Purpose |
-|-----|-------|---------|
-| `ScheduleFetchesJob` | fetch | Recurring: triggers Scheduler to find and enqueue due sources |
-| `FetchFeedJob` | fetch | Fetches a single source's feed, creates items |
-| `ScrapeItemJob` | scrape | Scrapes content for a single item |
-| `SourceHealthCheckJob` | fetch | Runs health check for a source |
-| `ImportSessionHealthCheckJob` | fetch | Health checks during OPML import wizard |
-| `ImportOpmlJob` | fetch | Bulk-creates sources from OPML import |
-| `LogCleanupJob` | fetch | Recurring: prunes old log entries |
-| `ItemCleanupJob` | fetch | Recurring: prunes old items per retention policy |
-
-Queue names are configurable via `config.fetch_queue_name` and `config.scrape_queue_name`.
-
-## Security Architecture
-
-- `Security::Authentication` -- Pluggable authentication via handler callbacks (symbol method names or callables)
-- `Security::ParameterSanitizer` -- HTML sanitization of all user inputs via `ActionView::Base.full_sanitizer`
-- `Models::Sanitizable` -- Concern that sanitizes string and hash model attributes before validation
-- `Models::UrlNormalizable` -- URL normalization and validation concern
-- `SanitizesSearchParams` -- Controller concern for search parameter sanitization
-- CSRF protection enabled (`protect_from_forgery with: :exception`)
-- No built-in user model -- delegates to host app
-
-## Instrumentation & Metrics
-
-- `Instrumentation` -- Emits `ActiveSupport::Notifications` events for fetch lifecycle
-- `Metrics` -- In-memory counters and gauges, populated via notification subscribers
-- Events: `source_monitor.fetch.start`, `source_monitor.fetch.finish`, `source_monitor.scheduler.run`, `source_monitor.items.duplicate`, `source_monitor.items.retention`

data/.vbw-planning/codebase/CONCERNS.md

@@ -1,99 +0,0 @@
-# Concerns
-
-## Technical Debt
-
-### 1. Large Files
-- `lib/source_monitor/fetching/feed_fetcher.rb` (627 lines) -- The core fetch pipeline handles HTTP, parsing, item creation, adaptive intervals, retry strategies, and source updates all in one class. This is the largest single-responsibility violation.
-- `lib/source_monitor/configuration.rb` (655 lines) -- Contains ~12 nested configuration classes in a single file. Each settings class could be extracted.
-- `app/controllers/source_monitor/import_sessions_controller.rb` (792 lines) -- The OPML import wizard controller handles all 5 wizard steps, file parsing, health checks, selection management, and bulk operations.
-- `lib/source_monitor/items/item_creator.rb` -- Large item creation file (based on coverage baseline entry count).
-- `config/coverage_baseline.json` (2329 lines) -- This file itself is very large, indicating significant uncovered code.
-
-### 2. Coverage Baseline Gaps
-The `config/coverage_baseline.json` catalogs uncovered lines. Particularly notable gaps:
-- `lib/source_monitor/items/item_creator.rb` -- Hundreds of uncovered branch lines
-- `lib/source_monitor/fetching/feed_fetcher.rb` -- Extensive uncovered branches
-- `lib/source_monitor/configuration.rb` -- Many configuration edge cases untested
-- `lib/source_monitor/dashboard/queries.rb` -- Dashboard query logic largely uncovered
-- `lib/source_monitor/realtime/broadcaster.rb` -- Broadcasting logic has gaps
-- `lib/source_monitor/scraping/bulk_source_scraper.rb` -- Bulk scraping coverage gaps
-- `lib/source_monitor/analytics/sources_index_metrics.rb` -- Analytics largely uncovered
-
-### 3. LogEntry Hard-coded Table Name
-In `app/models/source_monitor/log_entry.rb` line 6: `self.table_name = "sourcemon_log_entries"` -- This bypasses the configurable table name prefix system that all other models use via `ModelExtensions.register`. The table name is hard-coded despite the engine supporting custom prefixes.
-
-### 4. `lib/source_monitor.rb` Entry Point Complexity
-The main entry point has 102+ require statements loaded eagerly. This means all engine code is loaded at boot regardless of what features the host app uses. No autoloading or lazy-loading strategy.
-
-## Architectural Risks
-
-### 1. PostgreSQL Lock-in
-The `Scheduler` uses `FOR UPDATE SKIP LOCKED` (PostgreSQL-specific locking), and queries use `NULLS FIRST`/`NULLS LAST` ordering. The engine will not work with MySQL or SQLite. This is documented nowhere in the gemspec constraints.
-
-### 2. In-Memory Metrics Without Persistence
-`SourceMonitor::Metrics` stores counters and gauges in module-level instance variables (`@counters`, `@gauges`). These are:
-- Lost on process restart
-- Not shared across workers/processes
-- Not suitable for production monitoring
-
-### 3. Scraping State Management
-`SourceMonitor::Scraping::State` tracks in-flight scrapes but the mechanism (cache/memory) is fragile. In multi-process deployments, this could lead to over-enqueuing or under-enqueuing of scrape jobs.
-
-### 4. Optional Dependency Loading
-Solid Queue, Solid Cable, Turbo, and Ransack are loaded with `rescue LoadError`. This means the engine could silently fail to load core functionality without clear error messages to the host app developer.
-
-### 5. No Database Index Verification
-24 migration files create the schema incrementally. There is no single-schema check to verify all expected indexes exist after migration.
-
-## Security Considerations
-
-### 1. No Built-in Authentication
-The engine relies entirely on the host app for authentication. The `Security::Authentication` module provides hooks but no default protection. An unconfigured engine is accessible to anyone who can reach the mount path.
-
-### 2. Fallback User Creation in Import Controller
-`ImportSessionsController#create_guest_user` (lines 416-429) can create a `User` record in the host app's database when no authentication is configured. This is a defensive fallback but could be unexpected behavior.
-
-### 3. Parameter Sanitization Scope
-`Security::ParameterSanitizer` uses `ActionView::Base.full_sanitizer` which strips all HTML. This is broad but may not catch all injection vectors (e.g., URL-based attacks, header injection via custom_headers).
-
-### 4. Custom Headers Passthrough
-Source `custom_headers` are stored as JSONB and passed to HTTP requests (`FeedFetcher#request_headers`). Malicious header values could potentially be used for SSRF or header injection if not properly validated.
-
-## Performance Considerations
-
-### 1. N+1 Query Risk in Dashboard
-`Dashboard::Queries` performs multiple database queries for stats, recent activity, quick actions, job metrics, and fetch schedules. The controller assembles all of these synchronously.
-
-### 2. Large OPML File Handling
-The OPML import parses entire files in memory (`Nokogiri::XML(content)`). Large OPML files with thousands of entries could cause memory pressure.
-
-### 3. Bulk Scrape Operations
-`BulkSourceScraper` enqueues individual `ScrapeItemJob` for each item. For sources with thousands of items, this could flood the job queue.
-
-### 4. Coverage Baseline File
-The `config/coverage_baseline.json` at 2329 lines is parsed at test time and represents significant technical debt in test coverage.
-
-## Operational Risks
-
-### 1. No Health Check Endpoint Documentation
-The `/health` endpoint exists but its response format and behavior are not documented.
-
-### 2. Stalled Fetch Recovery
-The `StalledFetchReconciler` runs within the `Scheduler`, but if the scheduler itself stalls, there is no external recovery mechanism.
-
-### 3. Circuit Breaker State in Database
-Fetch circuit breaker state (`fetch_circuit_opened_at`, `fetch_circuit_until`) is stored on the `Source` model. This means circuit breaker resets require database writes, and a failing database connection prevents circuit recovery.
-
-## Dependency Risks
-
-### 1. ruby-readability (~> 0.7)
-This gem appears to be minimally maintained. The `0.7.x` line is old. It depends on Nokogiri which receives frequent security updates, creating a transitive dependency management burden.
-
-### 2. Nokolexbor (~> 0.5)
-Native extension gem with C bindings to the Lexbor HTML parser. Platform-specific builds could cause deployment issues, especially on less common architectures.
-
-### 3. Feedjira Wide Version Range
-`>= 3.2, < 5.0` allows major version bumps that could introduce breaking API changes.
-
-### 4. Ruby >= 3.4.0 Minimum
-This is an aggressive minimum that excludes many production Ruby installations still on 3.2.x or 3.3.x.

data/.vbw-planning/codebase/CONVENTIONS.md

@@ -1,97 +0,0 @@
-# Conventions
-
-## Ruby Code Style
-
-- **Style Guide**: Rails Omakase via `rubocop-rails-omakase` gem
-- **Frozen String Literals**: Consistently used across all app/ and lib/ Ruby files (`# frozen_string_literal: true`)
-- **No frozen_string_literal**: Absent from some test files and the main `lib/source_monitor.rb` entry point
-
-## Naming Conventions
-
-### Module/Class Naming
-- All engine code namespaced under `SourceMonitor::`
-- Sub-modules use descriptive domain names: `Fetching`, `Scraping`, `Health`, `Security`, `Dashboard`, `Realtime`
-- Service objects named as nouns (e.g., `FeedFetcher`, `ItemScraper`, `Scheduler`, `RetentionPruner`)
-- Presenters suffixed with `Presenter` (e.g., `BulkResultPresenter`, `RecentActivityPresenter`, `TurboStreamPresenter`)
-- Query objects in `analytics/` and `dashboard/` (e.g., `Queries`, `SourcesIndexMetrics`)
-
-### File Organization
-- One class per file, file path mirrors class namespace
-- Nested classes extracted to sub-directories (e.g., `item_scraper/adapter_resolver.rb`, `item_scraper/persistence.rb`)
-- Concerns placed in `concerns/` subdirectories
-
-### Database Naming
-- Table prefix: `sourcemon_` (configurable via `config.models.table_name_prefix`)
-- Tables: `sourcemon_sources`, `sourcemon_items`, `sourcemon_fetch_logs`, etc.
-- Foreign keys follow Rails conventions: `source_id`, `item_id`
-- Timestamps use `started_at`/`completed_at` pattern for log records
-
-### Method Naming
-- Bang methods (`!`) for operations that raise on failure: `soft_delete!`, `mark_processing!`
-- Predicate methods: `fetch_circuit_open?`, `auto_paused?`, `deleted?`, `success?`
-- `call` method pattern for service objects and scraper adapters
-- Private `set_*` for controller before_actions: `set_source`, `set_import_session`
-
-### Job Naming
-- Jobs suffixed with `Job`: `FetchFeedJob`, `ScrapeItemJob`, `ScheduleFetchesJob`
-- Queue assignment via `source_monitor_queue :fetch` class method
-
-### Controller Naming
-- Resource controllers follow Rails conventions: `SourcesController`, `ItemsController`
-- Singular nested resource controllers: `SourceFetchesController`, `SourceRetriesController`
-- Action-specific controllers for non-RESTful actions: `SourceBulkScrapesController`, `SourceHealthChecksController`
-
-## Configuration Patterns
-
-- Configuration via `SourceMonitor.configure { |config| ... }` block DSL
-- Nested configuration objects with `reset!` methods for testing
-- Settings classes with explicit defaults defined as constants
-- Callable values supported (procs/lambdas) for dynamic configuration
-
-## Error Handling
-
-- Custom error hierarchy: `FetchError` base with typed subclasses (`TimeoutError`, `ConnectionError`, `HTTPError`, `ParsingError`)
-- Errors carry structured data: `http_status`, `response`, `original_error`
-- Defensive `rescue StandardError` wrappers around logging calls
-- `Rails.logger` usage guarded with `defined?(Rails) && Rails.respond_to?(:logger) && Rails.logger`
-
-## Struct Usage
-
-- `Struct.new(..., keyword_init: true)` extensively used for result/event objects
-- Named structs: `Result`, `ResponseWrapper`, `EntryProcessingResult`
-- Event structs: `ItemCreatedEvent`, `ItemScrapedEvent`, `FetchCompletedEvent`
-
-## Frontend Conventions
-
-- CSS scoped under `.fm-admin` class with Tailwind `important` selector
-- Stimulus controllers registered on `window.SourceMonitorStimulus`
-- Controller naming: kebab-case in HTML (`async-submit`, `confirm-navigation`)
-- ERB partials prefixed with underscore, organized per resource
-- Turbo Stream responses via `SourceMonitor::TurboStreams::StreamResponder`
-
-## View Conventions
-
-- Layouts: Engine uses host app layout; views are ERB
-- Partials extensively used: `_row.html.erb`, `_details.html.erb`, `_form_fields.html.erb`
-- Wizard steps as separate partials: `steps/_upload.html.erb`, `steps/_preview.html.erb`
-- Toast notifications broadcast via realtime system
-
-## Test Conventions
-
-- Minitest with Rails test helpers (not RSpec)
-- Test files mirror source structure: `test/lib/source_monitor/scraping/item_scraper_test.rb`
-- Test helper methods: `create_source!` factory method in `ActiveSupport::TestCase`
-- Parallel test execution by default
-- Configuration reset in `setup` block: `SourceMonitor.reset_configuration!`
-- WebMock disables external connections
-- VCR for HTTP interaction recording
-- test-prof `before_all` for expensive shared setup
-- `with_inline_jobs` and `with_queue_adapter` helpers for job testing
-
-## Documentation Conventions
-
-- YARD-style comments in some service classes
-- Inline comments explain "why" not "what"
-- `# :nocov:` markers for untestable/defensive code paths
-- CHANGELOG maintained in conventional format
-- CONTRIBUTING.md present

data/.vbw-planning/codebase/DEPENDENCIES.md

@@ -1,100 +0,0 @@
-# Dependencies
-
-## Runtime Dependencies (gemspec)
-
-These are declared in `source_monitor.gemspec` and required by any host app using the engine.
-
-| Gem | Constraint | Purpose | Risk |
-|-----|-----------|---------|------|
-| rails | >= 8.0.3, < 9.0 | Core framework | Major version lock |
-| cssbundling-rails | ~> 1.4 | CSS asset bundling bridge | Low |
-| jsbundling-rails | ~> 1.3 | JS asset bundling bridge | Low |
-| turbo-rails | ~> 2.0 | Hotwire Turbo for real-time updates | Medium - optional but recommended |
-| feedjira | >= 3.2, < 5.0 | Feed parsing (RSS/Atom) | Core functionality |
-| faraday | ~> 2.9 | HTTP client | Core functionality |
-| faraday-retry | ~> 2.2 | Retry middleware | Low |
-| faraday-follow_redirects | ~> 0.4 | Redirect handling | Low |
-| faraday-gzip | ~> 3.0 | Compression support | Low |
-| nokolexbor | ~> 0.5 | Fast HTML parsing (lexbor engine) | Medium - native extension |
-| ruby-readability | ~> 0.7 | Article content extraction | Medium - older gem |
-| solid_queue | >= 0.3, < 3.0 | Background job processing | Core functionality |
-| solid_cable | >= 3.0, < 4.0 | Action Cable adapter | Real-time features |
-| ransack | ~> 4.2 | Search/filter form builder | Medium |
-
-## Optional Runtime Dependencies
-
-Loaded with rescue from LoadError in `lib/source_monitor.rb`:
-- `solid_queue` -- optional if host uses different Active Job backend
-- `solid_cable` -- optional if host uses Redis or another Action Cable adapter
-- `turbo-rails` -- optional but recommended for real-time updates
-- `ransack` -- powers search forms when available
-
-## Development Dependencies (Gemfile)
-
-| Gem | Purpose |
-|-----|---------|
-| puma | Development web server |
-| pg | PostgreSQL adapter |
-| propshaft | Asset pipeline |
-| rubocop-rails-omakase | Linting (Rails omakase style) |
-| brakeman | Security scanning |
-
-## Test Dependencies (Gemfile)
-
-| Gem | Purpose |
-|-----|---------|
-| simplecov | Code coverage reporting |
-| test-prof | Test profiling toolkit |
-| stackprof | Stack sampling profiler |
-| capybara | System test framework |
-| webmock | HTTP stubbing |
-| vcr | HTTP recording/playback |
-| selenium-webdriver | Browser driver for system tests |
-
-## JavaScript Dependencies (package.json)
-
-### Runtime
-
-| Package | Version | Purpose |
-|---------|---------|---------|
-| @hotwired/stimulus | ^3.2.2 | Stimulus JS framework |
-| stimulus-use | ^0.52.0 | Composable Stimulus behaviors |
-
-### Development
-
-| Package | Version | Purpose |
-|---------|---------|---------|
-| esbuild | ^0.23.0 | JS bundling |
-| tailwindcss | ^3.4.10 | CSS framework |
-| postcss | ^8.4.45 | CSS transformation |
-| autoprefixer | ^10.4.20 | CSS vendor prefixes |
-| eslint | ^9.11.0 | JS linting |
-| @eslint/js | ^9.11.0 | ESLint core config |
-| stylelint | ^16.8.0 | CSS linting |
-| stylelint-config-standard | ^36.0.0 | Stylelint standard config |
-
-## Dependency Coupling Analysis
-
-### Tight Coupling (hard to replace)
-- **Rails 8.x** -- entire engine is built on Rails conventions
-- **Feedjira** -- core feed parsing logic, deeply integrated in `Fetching::FeedFetcher`
-- **Faraday** -- HTTP client used throughout `SourceMonitor::HTTP`, configurable via middleware stack
-- **Solid Queue** -- integrated in engine initializer, scheduler, and job visibility system
-- **PostgreSQL** -- uses `FOR UPDATE SKIP LOCKED`, `NULLS FIRST/LAST` SQL syntax
-
-### Moderate Coupling (replaceable with effort)
-- **Nokolexbor/Nokogiri** -- HTML parsing in scrapers and OPML import
-- **Ransack** -- used in model `ransackable_attributes` declarations and search forms
-- **Tailwind CSS** -- all views use Tailwind utility classes scoped under `.fm-admin`
-
-### Loose Coupling (easily replaceable)
-- **Solid Cable** -- configurable via `config.realtime.adapter`, supports `:solid_cable`, `:redis`, `:async`
-- **ruby-readability** -- wrapped in `Scrapers::Readability` adapter behind pluggable adapter interface
-- **Turbo Rails** -- optional, loaded conditionally
-
-## Version Constraints of Note
-
-- Ruby >= 3.4.0 is a relatively aggressive minimum requirement
-- Rails >= 8.0.3 pins to the latest major, narrowing host app compatibility
-- Solid Queue has a wide range (0.3 to 3.0), suggesting early adoption with forward-looking flexibility
-- PostgreSQL is the only supported database (uses PG-specific SQL features)